Interactive notes

Neural networks, hands-on

Neural networks build functions by composing simple layers. This page lets you train small ones and look inside them: what each unit computes, how gradient descent trains them, what they can approximate, and what is known about how they generalize. Nothing is precomputed: the forward pass, backpropagation and stochastic gradient descent run in the page itself.

Setting. We observe a sample $S = ((x_1, y_1), \dots, (x_n, y_n))$ of independent draws from an unknown distribution $P$ on $\mathcal{X} \times \mathcal{Y}$. A predictor $f$ is judged by its risk $R(f) = \mathbb{E}_{(x, y) \sim P}[\ell(y, f(x))]$ for a loss $\ell$, which cannot be computed since $P$ is unknown, and is trained on its empirical risk $\widehat{R}_S(f) = \frac{1}{n} \sum_{i \le n} \ell(y_i, f(x_i))$. In the playground, the test risk is the empirical risk on 1000 fresh points, an estimate of $R(f)$.

How to use it. The playground trains a network on the dataset of your choice. Below it, guided experiments follow the four parts of this page. After the playground, each part has its own section, where interactive figures illustrate its key results.

Speed

Data

Network

Activation $\sigma$
Hidden layers

Training

Depth $L$ = · $P$ = parameters

Each square shows what a unit computes, as a function of the input: hover over it for details. A dashed square is a dead ReLU unit. Edges are the weights, thicker when larger: positive, negative. Under the network, the norm of the gradient of $F$ in the weights of each layer.

training risk $\widehat{R}_S(f_\theta)$ test risk log scale, against iterations
Training risk $\widehat{R}_S(f_\theta)$
Test risk, on 1000 fresh points, estimating $R(f_\theta)$
Gap $R - \widehat{R}_S$
Training error / test error /
Penalty $\Omega(\theta) = \lambda \sum_l \|W_l\|_F^2$
Affine pieces of $f_\theta$ on the plot
$P$, $n$ and $\sqrt{P/n}$, ,

Part I. Architectures

A network of depth $L$ composes $L$ parametrized maps, $f_{(\theta_1, \dots, \theta_L)}(x) = \rho^{(L)}_{\theta_L}(\cdots \rho^{(1)}_{\theta_1}(x))$. In the playground, the first $L - 1$ are non-linear layers $\rho^{(i)}(h) = \sigma(W_i h + b_i)$ and the last one is affine, $\rho^{(L)}(h) = W_L h + b_L$, followed by the softmax in classification. Layer $i$ has $d_i$ units and the parameters $\theta_i = (W_i, b_i)$, so that $P = \sum_{i \le L} d_i (d_{i-1} + 1)$, the count shown above the network.

Two facts organize this part. A composition of affine maps is affine, so without the non-linearity depth is useless (Experiment I.1, I.2). And each unit computes a simple function of the layer before it, which the playground draws in its square: half-planes and ramps for the first layer, combinations of them further on (I.3, I.4).

The activation function

The non-linearity $\sigma$ is a fixed function, applied coordinatewise. Its shape matters twice. In the forward pass, it shapes the functions the network can represent: ReLU networks are continuous and piecewise affine, with kinks where units switch on or off (Experiment I.4). In the backward pass, $\sigma'$ multiplies the gradient at every layer (backpropagation), and a small $\sigma'$ repeated over many layers makes the gradient vanish (Experiment II.5).

$\sigma(u)$ $\sigma'(u)$

$\sigma(u) = (u)_+ = \max(0, u)$ and $\sigma'(u) = \mathbf{1}\{u > 0\}$, a subgradient being chosen at $0$. Cheap, unbounded, and its derivative is $1$ on active units, so it does not shrink the gradient. A unit that is negative on every data point gets no gradient at all: it is dead.

The softmax and the cross-entropy

With $d$ classes, the last layer outputs logits $z \in \mathbb{R}^d$, and the softmax turns them into a probability vector, $\mathrm{sm}(z)_j = e^{z_j} / \sum_{k \le d} e^{z_k}$, a point of the simplex $\Delta_{d-1} = \{u \in \mathbb{R}^d : u \ge 0, \ \sum_k u_k = 1\}$. Try the buttons: adding the same constant to every logit changes nothing, since the softmax only sees their differences; doubling them sharpens the distribution towards the largest one.

With a one-hot label $e_c$, the loss is $-\log \mathrm{sm}(z)_c$, and its gradient in $z$ is $\mathrm{sm}(z) - e_c$, the predicted distribution minus the truth, since $\partial\, \mathrm{sm}(z)_j / \partial z_i = \mathrm{sm}(z)_j \, (\delta_{ij} - \mathrm{sm}(z)_i)$. It vanishes only in the limit $\mathrm{sm}(z) \to e_c$: on separable data, training keeps increasing the logits, and the loss tends to $0$ without reaching it. On one-hot labels, the cross-entropy is the negative log-likelihood of the model $\mathbb{P}(y = c \mid x) = \mathrm{sm}(z)_c$: training is maximum likelihood. See Experiment I.5.

True class $c$
$\sum_k \mathrm{sm}(z)_k$ = · loss $-\log \mathrm{sm}(z)_c$ =

The cross-entropy is a divergence

For probability vectors $p$ (the label) and $q$ (the prediction), $\ell(p, q) = -\sum_i p_i \log q_i = H(p) + \mathrm{KL}(p \,\|\, q)$, where $H(p) = \ell(p, p)$ is the entropy of $p$. Only the divergence depends on $q$, so $q \mapsto \ell(p, q)$ is minimized exactly at $q = p$, by Jensen's inequality.

But its minimum is $H(p)$, not $0$: for $p = q = (1/2, 1/2)$ the loss is $\log 2 \approx 0.693$, and $\ell(p, q) = 0$ does not characterize $q = p$. On one-hot labels, $H(p) = 0$ and both statements hold. The figure is the case of two classes, with $p$ and $q$ the probabilities of class 1. Drag on the plot to move $q$.

$\ell(p, q)$ = = $H(p)$ + $\mathrm{KL}(p \,\|\, q)$

Part II. Optimization and the chain rule

Training minimizes the regularized empirical risk $F(\theta) = \frac{1}{n} \sum_{i \le n} \ell(y_i, f_\theta(x_i)) + \Omega(\theta)$ by stochastic gradient descent, $\theta_t = \theta_{t-1} - \gamma_t \nabla_\theta F^{(B)}(\theta_{t-1})$, where $F^{(B)}$ averages the loss over a batch $B$. In the playground, $\Omega(\theta) = \lambda \sum_l \|W_l\|_F^2$, the step $\gamma$ is constant, and the batches are drawn without replacement: one epoch is one pass over the data.

Backpropagation, step by step

The gradient is computed by the chain rule, ordered so that a single backward traversal gives every partial derivative. Take a node $N = N(\mathrm{input}, \mathrm{params})$ of the network, and write the loss as a function $\ell(N)$ of its output, everything after the node included. Then

$$\frac{\partial \ell}{\partial\, \mathrm{params}} = \frac{\partial \ell}{\partial N} \, \frac{\partial N}{\partial\, \mathrm{params}} \quad (\ast) \qquad \text{and} \qquad \frac{\partial \ell}{\partial\, \mathrm{input}} = \frac{\partial \ell}{\partial N} \, \frac{\partial N}{\partial\, \mathrm{input}} \quad (\ast\ast).$$

Both need $\partial \ell / \partial N$, the derivative of the loss with respect to the output of the node. It travels backwards: (∗∗) turns it into the same derivative for the node before, and (∗) reads off the gradient in the parameters of the node on the way. Below, a network with an input $x \in \mathbb{R}^2$, two hidden units and a scalar output, on one data point with the quadratic loss. Run the forward pass, then the backward pass, and follow which identity each step uses.

input $x$
$W_1 \cdot + \, b_1$ $W_1$ $b_1$ $\partial \ell / \partial W_1$ $\partial \ell / \partial b_1$
$a_1$ $\partial \ell / \partial a_1$
$\sigma$, coordinatewise $\sigma'(a_1)$
$h_1$ $\partial \ell / \partial h_1$
$W_2 \cdot + \, b_2$ $W_2$ $b_2$ $\partial \ell / \partial W_2$ $\partial \ell / \partial b_2$
$a_2 = f_\theta(x)$ $\partial \ell / \partial a_2$
$(\cdot - y)^2$ $y$
loss $\ell$

Start. The input is $x = (1, 0.5)$, the target $y = 1$, and the parameters $\theta = (W_1, b_1, W_2, b_2)$ are $P = 9$ numbers. Press Next to run the forward pass.

loss:

What gradient descent does, and does not, do

Nothing guarantees that SGD finds a minimizer. $F$ is not convex as soon as $L \ge 2$, it has many critical points, and its minimizers come in orbits of size at least $\prod_l (d_l)!$ because permuting hidden units, with their weights, does not change the function. Under a step size condition, gradient descent converges to a critical point, and nothing more is proved in general. The experiments of this part make each of these facts visible:

  • II.1, the learning rate, including a step so large that the network dies at a critical point;
  • II.2, gradient descent against SGD, per iteration and per epoch;
  • II.3, different initializations, different minimizers, different test risks;
  • II.4, the permutation symmetry;
  • II.5, vanishing gradients in a deep sigmoid network.

Part III. Universal approximation

Theorem. Let $g \in C^0([a, b], \mathbb{R})$ and $\epsilon > 0$. There is a network $f(x) = \sum_{j \le m} \eta_j (w_j x + b_j)_+$ with one hidden layer of finitely many ReLU units such that $\sup_{x \in [a, b]} |g(x) - f(x)| \le \epsilon$.

The proof has two steps, both visible in the figure. First, $g$ is uniformly continuous on the compact $[a, b]$ (Heine's theorem): there is $\eta > 0$ such that $|x - x'| \le \eta$ implies $|g(x) - g(x')| \le \epsilon$. Take knots $a = x_1 < \dots < x_N = b$ of spacing at most $\eta$, and let $g_\eta$ be the piecewise affine interpolant of $g$ on them. On $[x_k, x_{k+1}]$, $g_\eta(x)$ is a convex combination of $g(x_k)$ and $g(x_{k+1})$, both within $\epsilon$ of $g(x)$, so $\sup |g - g_\eta| \le \epsilon$. Second, $g_\eta$ is exactly a one hidden layer network, with one unit per knot:

$$g_\eta(x) = \sum_{i \ge 0} (S_i - S_{i-1})(x - x_i)_+ \quad \text{on } [a, b],$$

where $S_i$ is the slope of $g_\eta$ on $[x_i, x_{i+1}]$, and the extra knot $x_0 = a - 1$, with $S_{-1} = 0$ and $S_0 = g(a)$, produces the value at $a$. The units are the kinks, and the coefficients are the jumps of the slope.

Move Units added to build the sum one unit at a time. With the units up to the knot $x_k$, the partial sum is exactly $g_\eta$ on $[a, x_{k+1}]$ and continues affinely beyond. This is the telescoping argument of the proof: on $[x_k, x_{k+1}]$ the slopes of the active units add up to $\sum_{i = 0}^{k} (S_i - S_{i-1}) = S_k$, and at $a$ only the unit of $x_0$ is active, with value $S_0 (a - x_0) = g(a)$. The plot on the right gives the uniform error against the number of units $N$. For a $G$-Lipschitz target, the first step works with $\epsilon = G \eta$, so the proof guarantees $G (b - a)/(N - 1)$ (dashed), an upper bound: on smooth targets the interpolant does better, while $\sqrt{|x|}$, continuous but not Lipschitz, is approximated more slowly, though the theorem still applies (its error zigzags, depending on whether a knot falls on the cusp at $0$). Choose Draw to draw your own $g$ with the mouse.

target $g$ network (the partial sum) one term per unit right: $\sup |g - g_\eta|$, guarantee $G\eta$
sup error on $[a, b]$ = · spacing $\eta$ = · guarantee $G \eta$ =

In several dimensions, the same holds for any continuous activation that is not a polynomial (Cybenko, 1989; Hornik, 1991; Leshno, Lin, Pinkus and Schocken, 1993). It is an existence statement: it says that a good $f$ is in $\mathcal{F}$, which makes the approximation error small, but neither that SGD finds it nor that finitely many samples identify it. In the playground: III.1 (one unit, one kink), III.2 (SGD misses a network that exists) and III.3 (two spirals).

Part IV. Generalization

Write $R^\star$ for the smallest risk of any predictor, and $\mathcal{F}$ for the networks of a given architecture. The excess risk of a trained network $\hat f \in \mathcal{F}$ splits into two terms,

$$R(\hat f) - R^\star = \underbrace{R(\hat f) - \inf_{f \in \mathcal{F}} R(f)}_{\text{estimation error}} + \underbrace{\inf_{f \in \mathcal{F}} R(f) - R^\star}_{\text{approximation error}}.$$

Universal approximation makes the approximation error small for wide enough networks, and leaves the estimation error. Networks of a fixed architecture form a parametric class with $P$ parameters, and the classical tools for such classes (covering numbers, Rademacher complexity, concentration) give the following. With bounded inputs and weights and a Lipschitz, bounded loss, with probability at least $1 - \delta$, every $f \in \mathcal{F}$ satisfies

$$R(f) \le \widehat{R}_S(f) + O\Big(\sqrt{P/n}\Big),$$

with a constant exponential in the depth, and up to a term in $\sqrt{\log(1/\delta)/n}$ (Anthony and Bartlett, Neural Network Learning: Theoretical Foundations, 1999). Counting through the VC dimension of ReLU networks, of order $P L \log P$, gives a similar bound (Bartlett, Harvey, Liaw and Mehrabian, 2019). The bound is uniform over $\mathcal{F}$, so it holds for whatever weights SGD returns, minimizer or not. The playground shows both sides of the story. Overfitting is real when $P \gg n$, and the gap does shrink with $n$ (Experiment IV.1). But the bound is vacuous long before the network stops generalizing: in that experiment, the gap is already about $0.01$ at $n = 100$, where $\sqrt{P/n} \approx 3.4$, and it decreases much faster than $\sqrt{P/n}$ in between. A network can even memorize random labels (Experiment IV.2):

Setting$P$$n$$\sqrt{P/n}$observed gap (test error − training error)
ResNet-18 on CIFAR-10$1.1 \times 10^7$$5 \times 10^4$150.05
Experiment IV.2, true labels2274407.50.035
Experiment IV.2, random labels2274407.50.4

A gap between error rates cannot exceed $1$: these bounds are not loose but vacuous. With random labels, the same network on the same points reaches zero training error and a test error at chance level. It shatters those points, so no bound depending only on the class $\mathcal{F}$, through $P$ or the VC dimension, can be below $1$ for it. Since the same class generalizes on the true labels, the explanation has to involve the data and the algorithm: SGD does not explore all of $\mathcal{F}$ (Zhang, Bengio, Hardt, Recht and Vinyals, 2017). Why deep networks generalize is mostly an open question.