What are activation functions in a neural network?

In this article

1. Before you start
2. Why interviewers ask this
3. Activation functions from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A candidate who can list ReLU, sigmoid and tanh but cannot say what breaks without them has memorised names instead of reasons. Activation functions show up in deep learning screens as "why do we need non-linearities", "ReLU versus sigmoid and the vanishing gradient", "what is wrong with ReLU", "what do modern transformers use", and as a coding task: write a numerically stable softmax. Reported in interviews at AI labs and large tech companies, the stable softmax appears often because it is short and easy to get quietly wrong.

A weak answer recites a list. A strong answer links each function to a failure it prevents or causes, and writes the softmax with the maximum subtracted.

Activation functions from first principles

In plain words

ReLU

ReLU, short for rectified linear unit, is the activation function that returns its input when the input is positive and zero otherwise, written as the larger of zero and x.

Softmax

Softmax is a function that turns a vector of raw scores into probabilities by exponentiating each score and dividing by the sum of all the exponentials, so every output is positive and they add to 1.

Log-sum-exp trick

The log-sum-exp trick is a way to compute the logarithm of a sum of exponentials without overflow, by subtracting the largest input before exponentiating and adding it back afterwards.

Think of a row of light dimmer switches. If every switch is a smooth linear dial, chaining a hundred of them is no more expressive than one dial with a different setting. Put a hard bend between the dials, such as "no light below this point", and the chain can draw complicated shapes.

The analogy breaks because a network's bends are learned jointly with its weights, not placed by hand.

In one precise sentence, an activation function is the bend between linear layers.

Step 1: Without a non-linearity, layers collapse into one

Two linear layers in a row are exactly one linear layer, so a network of any depth built only from matrix multiplications can learn nothing a single matrix cannot. If the first layer has weights $W_1$ and the second $W_2$, then $W_2(W_{1}x)=(W_2{W}_1)x$, and $W_2{W}_1$ is just another matrix.

A check on random matrices confirms it: the deep and single-matrix outputs differ by about $\approx {10}^{-16}$, which is rounding error, and putting a ReLU between the layers makes them differ¹. The linear algebra for machine learning page explains why products of matrices stay linear.

The input is a vector and two weight matrices. The output shows that a bend between them is what adds expressive power.

Step 2: ReLU keeps the gradient alive where sigmoid shrinks it

ReLU mitigates the vanishing gradient better than sigmoid because its slope is exactly 1 for any positive input, while the sigmoid's slope never exceeds 0.25. A vanishing gradient is a gradient that shrinks toward zero as it is passed back through many layers, so early layers stop learning. The PyTorch 2.14 reference defines ReLU as applying the rectified linear unit function element-wise, the larger of zero and the input.

The sigmoid, $\sigma (z)=1/(1+e^{-z})$, has a derivative $\sigma (z)(1-\sigma (z))$ with a maximum of 0.25 at $z=0$. Backpropagation multiplies the gradient at each layer by this slope and by the layer's weights. The slopes alone, each at its maximum, multiply to 0.000977 after 5 layers and 0.00000095 after 10¹, so with weights no larger than 1 the gradient shrinks at least that fast. A ReLU path through active units multiplies by a slope of 1 instead.

ReLU has a weakness. A unit whose input is negative for every example has slope 0 and receives no gradient. In an illustrative unit with a large negative bias, none of 1,000 random inputs activates it, so its weight update is exactly 0¹. That is the dying ReLU problem.

Step 3: Smooth and gated variants soften the cut

The Gaussian error linear unit (GELU), the sigmoid linear unit (SiLU) and SwiGLU replace ReLU's hard cut with a smooth one, and many modern transformers use one of them. Hendrycks and Gimpel (2016, v5 2023) define the Gaussian error linear unit as $x\mathrm{\Phi }(x)$, where $\mathrm{\Phi }$ is the standard Gaussian cumulative distribution function, so each input is weighted by its value instead of gated by its sign. SiLU, also called Swish, is $x$ times the sigmoid of $x$.

At an input of −1, ReLU outputs 0, GELU outputs −0.1587 and SiLU outputs −0.2689, so both let a small negative signal and its gradient through¹. SwiGLU is a gated variant: the layer computes two projections of the input and multiplies one, passed through SiLU, by the other. Shazeer (2020) studies gated linear units in the transformer feed-forward layer and reports that some variants improve quality over ReLU or GELU. The transformers hub shows where that layer sits.

1 · Layers collapse2 · ReLU keeps slope3 · Smooth the cut4 · Stay finiteReplay

Step-by-step description

1. Without a bend, stacked linear layers equal one layer.
2. ReLU passes gradients; a sigmoid shrinks them.
3. GELU and SwiGLU let small negatives through.
4. Subtract the max score before exponentiating.

• Two linear layers in a row are exactly one linear layer.
• ReLU has a slope of 1 for positive inputs while the sigmoid slope never exceeds 0.25.
• At an input of -1 ReLU outputs 0, GELU outputs 0.1587 below zero and SiLU 0.2689 below zero.
• Stable softmax of the scores 1000, 999 and 998 gives 0.6652, 0.2447 and 0.0900.

Step 4: The log-sum-exp trick keeps softmax from overflowing

The log-sum-exp trick makes softmax safe by subtracting the largest score before exponentiating, which changes nothing mathematically and removes overflow. Softmax is $e^{z_i}/{\sum }_j{e}^{z_j}$. Each raw score $z_i$ is a logit: a logit is a score before softmax turns it into a probability. Dividing numerator and denominator by $e^m$, where $m$ is the largest score, gives the same probabilities with every exponent at most 0.

Overflow is real. The exponential of 710 exceeds the largest 64-bit float, and of 89 exceeds the largest 32-bit float, with thresholds near 709.78 and 88.72¹. For scores of 1000, 999 and 998, the naive formula returns not-a-number for every class, while the stable formula returns 0.6652, 0.2447 and 0.0900¹.

The same idea computes the log of a sum of exponentials: $\log \sum e^{z_j}=m+\log \sum e^{z_j-m}$. For these scores it is 1000.4076. The PyTorch 2.14 logsumexp reference says only that the computation is numerically stabilized. In PyTorch, CrossEntropyLoss expects unnormalized logits and applies the stable form itself, while the Keras cross-entropy losses assume probabilities unless from_logits=True is set. The cross-entropy loss page explains the loss that consumes these logits.

Worked example

The sigmoid column is $(0.25{)}^n$, so each layer removes ${\log }_{10}4\approx 0.60$ decimal digits: 3.01 digits after 5 layers, 6.02 after 10 and 12.04 after 20, against 0 for the ReLU path¹.

Text description of the diagram

• Sigmoid slopes at their maximum lose 3.01 decimal digits after 5 layers, 6.02 after 10 and 12.04 after 20.
• An active ReLU path loses 0 digits.

Plotted from −3 to 3, ReLU is flat at 0 for every negative input, while GELU and SiLU dip below zero, to −0.1587 and −0.2689 at −1, and approach the input again by 3, where they give 2.9960 and 2.8577¹.

Text description of the diagram

• At an input of -1, ReLU outputs 0, GELU outputs 0.1587 below zero and SiLU 0.2689 below zero.
• At an input of 3, GELU gives 2.9960 and SiLU 2.8577.

A gated layer has three weight matrices instead of two, which matters when you size a transformer block. At a model width of 4,096, a hidden size of 16,384 with two matrices holds 134,217,728 weights, and a gated layer with a hidden size of 10,923 holds 134,221,824, almost the same¹.

A reader can redo the stable softmax by hand. With scores 1000, 999 and 998, subtract 1000 to get 0, −1 and −2. The exponentials are 1, 0.3679 and 0.1353, which sum to 1.5032, so the probabilities are 0.6652, 0.2447 and 0.0900. The script and its saved output hold the rest¹.

Trade-offs and edge cases

No activation function is best everywhere; each trades gradient flow, cost and smoothness. The choice depends on depth, the layer type and the hardware.

• ReLU. Cheapest and simple, but units can die. Variants such as the leaky version keep a small slope for negative inputs.
• Sigmoid and tanh. Still the right output for a probability or a gate, but poor hidden-layer choices in deep networks because of the shrinking slope.
• GELU and SwiGLU. Smoother and common in transformer feed-forward layers, at slightly higher cost per element and, for gated units, a different matrix count.
• Softmax in lower precision. Half-precision floats overflow far sooner than 32-bit ones, so the max subtraction matters even more; see 16-bit floating point (FP16) versus brain floating point (BF16).
• Fused kernels. Because softmax and normalization read and write a lot of memory for little arithmetic, fusing them helps; FlashAttention applies this idea to attention, and graphics processing unit (GPU) fundamentals explains why memory traffic dominates.

How to answer it in an interview

The spoken structure is: say what collapses, name the bend, compare slopes, mention the modern variants, then give the max-subtraction detail.

Common mistakes

• "ReLU is differentiable everywhere" (wrong). It has a kink at 0, and frameworks pick a value there, usually 0 (right: say the gradient is defined by convention at the kink).
• "Sigmoid vanishes because it is slow to compute" (wrong). It vanishes because its slope is at most 0.25 and shrinks toward 0 for large inputs (right: name the slope).
• "Softmax of large scores just gives a one-hot output" (wrong). Softmax depends only on the gaps between scores, so 1000, 999 and 998 give 0.6652, 0.2447 and 0.0900; only the naive formula fails, with not-a-number¹ (right: large gaps, not large values, push softmax toward one-hot).
• "Subtracting the max changes the probabilities" (wrong). Softmax depends only on differences between scores (right: it is exactly the same function).
• "Dying ReLU means the network is broken" (wrong). Some units go silent; the fixes are a smaller learning rate, careful initialization or a leaky variant (right: describe it as units with zero gradient).

Follow-up questions

What is the purpose of the non-linearities in a neural network?

Non-linearities stop consecutive linear layers from collapsing into a single linear map. Without them a hundred layers learn the same family of functions as one.

Which activation function is better at mitigating the vanishing gradient problem, ReLU or sigmoid?

ReLU, because its slope is 1 for positive inputs while the sigmoid's slope is at most 0.25, so with weights no larger than 1 the gradient of a deep sigmoid network shrinks by at least a factor of four per layer, as the table shows. Tanh has a slope of at most 1 but still saturates for large inputs.

What is the drawback of ReLU?

Units whose input stays negative output 0 and receive zero gradient, so they can stop learning permanently. Leaky variants and careful learning rates reduce the problem, and the deep learning hub lists related failure modes.

Why do transformers use GELU or SwiGLU instead of ReLU?

Reported results favor the smooth or gated forms; Shazeer's 2020 study found that some gated variants improve transformer quality over ReLU or GELU. The reason is empirical more than theoretical, so say "reported to improve" rather than "proven".

Why does softmax need numerical stabilization?

Exponentials of large scores overflow, in 32-bit floats beyond about 88.72, and the result is not-a-number. Subtracting the maximum score first keeps every exponent at most 0 without changing the output.

Sources

1. torch.nn.ReLU, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
2. torch.logsumexp, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
3. Gaussian Error Linear Units (GELUs) (Hendrycks and Gimpel), arXiv 1606.08415 v5 — arXiv (v1 submitted 2016-06-27), 6 Jun 2023
4. GLU Variants Improve Transformer (Shazeer), arXiv 2002.05202 — arXiv, 12 Feb 2020
5. torch.nn.CrossEntropyLoss, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
6. Probabilistic losses (CategoricalCrossentropy, from_logits), Keras 3 API documentation — Keras team (page undated; read 2026-10-24), 24 Oct 2026