What this activation function visualizer shows
Tick up to eight functions in the list under the graph. Solid lines are the functions themselves, dashed lines in the same colour are their derivatives, and the Show menu switches between both, f(x) only and f′(x) only. Move the pointer over the graph (on a phone, drag sideways) and the table underneath gives f(x) and f′(x) for every plotted function at that x. x range and the + and − buttons zoom; Same scale on both axes makes a slope of 1 look like 45°, which is the honest way to compare ReLU with a sigmoid; and Shade where |f′| < 0.01 marks the flat tails of the function chosen in One function in detail.
That panel prints the formula and its derivative, measures the slope at 0, the largest slope, the lowest value and where the flat tails begin, and copies both formulas as LaTeX. The table below it is the list view: all 16 functions with formula, derivative, output range, peak slope, whether they are zero-centred, where they saturate, and a well-known place each one is used. Every peak and minimum in that table was recomputed from the plotted functions by this page’s verification script.
The functions, grouped by shape
Squashing functions: sigmoid, tanh, softsign, hard sigmoid
These map every input into a bounded interval. The sigmoid σ(x) = 1/(1 + e⁻ˣ) has derivative σ(x)(1 − σ(x)), which is never larger than 0.25, reached at x = 0. Tanh is the same curve stretched to (−1, 1) — tanh(x) = 2σ(2x) − 1 — so its peak slope is 1. Both go flat quickly: |f′| drops below 0.01 for |x| ≥ 2.99 with tanh and |x| ≥ 4.58 with the sigmoid. Softsign, x/(1 + |x|), has the same range as tanh but approaches ±1 only like 1/|x|, so its slope stays above 0.01 until |x| = 9. The hard sigmoid used in MobileNetV3 replaces the curve with three straight pieces and has slope exactly 1/6 between −3 and 3 and exactly 0 outside.
Rectifiers: ReLU, Leaky ReLU, PReLU, ELU, SELU
ReLU, max(0, x), has slope exactly 1 for positive inputs and exactly 0 for negative ones; at the kink the derivative is undefined, and this page uses 0 there, as PyTorch does. Leaky ReLU and PReLU keep a small slope α (or a learned a) on the left so a unit can never stop receiving gradient. ELU bends smoothly towards −α instead, and is only smooth at 0 when α = 1. SELU scales ELU with λ ≈ 1.0507 and α ≈ 1.6733, constants chosen so that a standard-normal input keeps mean 0 and variance 1: integrating numerically, this page gets E[f(z)²] = 1.000000000 and a mean below 10⁻¹⁴.
Smooth gates: GELU, SiLU/Swish, Mish, softplus, hard swish
These multiply x by a soft switch. GELU is x·Φ(x), with Φ the normal distribution function; it dips to −0.1700 at x = −0.752, and its slope peaks at 1.1289 at x = √2 — above 1, which ReLU never is. The tanh approximation used in GPT-2’s original code differs from the exact erf form by at most 4.73 × 10⁻⁴ (at x = ±2.70), and its slope by at most 8.68 × 10⁻⁴ (at x = ±2.02), measured on [−10, 10]. Swish is x·σ(βx) and β = 1 is SiLU. Press Swish β sweep to watch β run from 0.1 to 10: as β → 0 it becomes the straight line x/2, and its largest distance from ReLU is 0.2785/β, so it closes in on ReLU as β grows. Mish, x·tanh(softplus(x)), has slope 0.6 at 0 (exactly tanh(ln 2)) and a minimum of −0.3088. Softplus is the smooth ReLU whose derivative is the sigmoid itself.
The binary step
Rosenblatt’s perceptron used a step. Its derivative is 0 wherever it exists, so backpropagation has nothing to pass back — in the depth panel every layer of a step network below the output layer gets a gradient of exactly 0. That is the whole reason smooth or piecewise-linear functions replaced it.
Why the derivative decides what a deep network can learn
Backpropagation multiplies one factor per layer on the way back to the input: the transposed weight matrix, then the activation’s derivative at that layer’s inputs. If the derivative is at most 0.25, then with weights scaled so that the matrices roughly preserve size — Xavier initialisation does that for a square layer — the gradient shrinks by about a factor of four per layer, every layer.
The Japanese write-up behind this page measured exactly that. It trained width-32 MLPs on two-moons (1000 points, noise 0.25, 700 for training and 300 for testing) with Adam at learning rate 0.01, giving each function its own best initialisation: He for ReLU and GELU, Xavier for tanh and sigmoid. At initialisation the first-layer gradient of the sigmoid network fell from 1.14 × 10⁻² at depth 1 to 2.05 × 10⁻¹⁷ at depth 24 — the “15 orders of magnitude” in its title (log₁₀ of the ratio is 14.7). ReLU, GELU and tanh stayed between 10⁻² and 10⁻³. After training, the sigmoid network still reached 0.803 test accuracy at depth 12 and then fell to 0.490 at depth 16 and 24, which is exactly 147/300: it predicted class 1 for every test point. ReLU and GELU stayed between 0.89 and 0.95 at every depth; tanh dropped to 0.80 at depth 24.
The Gradient through depth panel recomputes the first half of that experiment in your browser: the same architecture, the same loss, the same metric (the mean absolute value of each weight matrix’s gradient), on 700 two-moons points generated with the same recipe. It does not train, so it cannot reproduce the accuracy numbers, and it draws its own random weights, so its numbers are its own. With the default ReLU vs sigmoid, 24 layers (seed 1) the sigmoid line falls from 7.00 × 10⁻³ at the output layer to 3.22 × 10⁻¹⁷ at layer 1: 14.3 orders of magnitude, or −0.637 orders per layer. ReLU with He goes from 2.21 × 10⁻² to 3.08 × 10⁻³, 0.9 orders in total. Across 20 random seeds the sigmoid loses 14.3 to 15.9 orders and shrinks by a factor of 0.224 to 0.238 per layer — just under the 0.25 ceiling set by σ′ — while ReLU’s total stays between −0.3 and 1.3 orders.
Switch the plot to Layer-1 gradient vs depth, or press The article’s four, and the panel builds a separate network for each depth from 1 to 24, as the article did. Over 20 seeds the median sigmoid values are 1.51 × 10⁻², 3.32 × 10⁻³, 2.04 × 10⁻⁴, 4.63 × 10⁻⁷, 1.30 × 10⁻⁹, 4.48 × 10⁻¹² and 2.94 × 10⁻¹⁷ at depths 1, 2, 4, 8, 12, 16 and 24, against the article’s 1.14 × 10⁻², 3.07 × 10⁻³, 2.14 × 10⁻⁴, 5.54 × 10⁻⁷, 2.40 × 10⁻⁹, 2.34 × 10⁻¹² and 2.05 × 10⁻¹⁷; the median drop from depth 1 to 24 is 14.7 orders, the same as the article’s.
Initialisation is the other half
A flat line in the per-layer view is not the whole story. ReLU with the wrong init compares He and Xavier at 48 layers. Both lines are flat, but with Xavier the whole profile sits about seven orders lower: 7.84 × 10⁻¹⁰ at layer 1 against 1.14 × 10⁻² with He (seed 1). The problem is in the forward pass: ReLU discards half of its input, and only He’s factor of 2 makes up for it. Measured directly, the output of the 48th layer has a root-mean-square of 8 × 10⁻¹⁰ to 6 × 10⁻⁹ with Xavier and 0.04 to 0.28 with He, over five seeds.
Smooth modern functions shows the same effect where it is less expected. With He initialisation at 48 layers, the median layer-1 gradient over 20 seeds is 3.4 × 10⁻⁷ for GELU, 2.2 × 10⁻⁸ for SiLU and 7.4 × 10⁻⁵ for Mish, against 4.2 × 10⁻² for SELU with LeCun initialisation, whose forward signal stays at a root-mean-square of 0.88 to 1.2. He’s factor of 2 was derived for ReLU; deep plain stacks of other functions need their own scaling (or the normalisation layers that real architectures add).
Can one ReLU imitate a sigmoid?
The article’s first check fitted scale·ReLU(x + shift) + offset to the sigmoid on [−8, 8] and got scale 0.1009, shift 4.5522, offset 0.0029 and R² = 0.922 — a respectable fit. The same line evaluated on [−30, 30] misses by up to 2.49, and at x = 30 it is at 3.49 while the sigmoid is at 1.0000. The last panel on this page solves the same least-squares problem exactly and reproduces all six numbers. Move the window to see the trade-off: at ±4 the fit inside is better (R² = 0.976) and the miss on [−30, 30] is worse (4.41); at ±16 R² drops to 0.863 and the miss shrinks to 1.07. A good fit inside a window says nothing about what happens outside it, and a saturating function and a non-saturating one part ways exactly there.
Limits
- The depth panel measures gradients at initialisation, from one forward and one backward pass with zero biases. It says which networks can start learning, not how well they end up.
- The metric is the mean |∂loss/∂W| per weight matrix, as in the article, not a matrix norm. Layer 1 (2 × width) and the output layer (width × 1) have other shapes, so the per-layer change is fitted over the hidden-to-hidden layers only.
- At kinks the derivative is undefined and the readout shows one side of it.
- The “where it is used” column lists well-known examples, not a survey.