Activation functions: diagnose saturation and dead units

Activation functions shape both forward signals and backward gradients. Sigmoid can saturate at large positive or negative inputs, producing derivatives near zero. ReLU returns zero for negative inputs, so those examples produce zero local gradient. These mechanisms can slow learning, but a diagnosis needs measurements across units, batches and training steps.

Inspect values and derivatives

The neural lab evaluates inputs [-20, -5, -1, 0, 1, 5, 20].

from deep_learning_cases import activation_case

result = activation_case()
assert result["sigmoid_near_zero_gradients"] == 2
assert result["relu_dead_negative_inputs"] == 3
print(result["sigmoid_gradients"][0], result["relu_gradients"][:4])

The sigmoid derivative at -20 is about 2.06e-9, and it is equally tiny at 20. ReLU gradients are zero at the three negative inputs; this lab defines the derivative at zero as zero too. Leaky ReLU maps the negative examples to [-0.2, -0.05, -0.01] with slope 0.01 rather than flattening them.

“Dead” needs a time window

A negative ReLU input on one example does not make the unit dead. Track the fraction of zero activations by layer and unit across a representative dataset. A unit is operationally concerning when it remains inactive across relevant inputs and updates. Also monitor gradient norms and weight changes.

Poor input scaling, an excessive learning rate and unfavorable bias initialization can push activations into unhelpful regions. Normalize inputs using training statistics, inspect pre-activation distributions and compare a controlled activation change. Do not swap several architecture and optimizer choices at once.

Match output activation and loss

Hidden activations and output activations have different jobs. Binary classification commonly combines a logit with a numerically stable binary cross-entropy implementation. Multi-class classification uses logits with cross-entropy. Applying sigmoid and then a loss that expects logits can duplicate a transformation and reduce numerical stability.

For regression, output constraints should follow the target. A ReLU output cannot predict a negative value. A sigmoid output caps predictions between zero and one unless values are transformed. State that target contract explicitly.

Activation histograms alone do not establish model quality. Compare validation behavior, calibration and error slices. The Data Science course connects activation mechanics with gradient checks and controlled training experiments.

Exercise

Log per-layer zero fractions, activation percentiles and gradient norms for ReLU, leaky ReLU and tanh under identical initialization. Repeat after multiplying the learning rate by ten and explain which mechanism changed.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Backpropagation with a small numerical gradient check.
• Continue with Choose a learning rate using training evidence.

References: scikit-learn activation definitions and PyTorch ReLU API.