Machine Learning & Neural Networks

Computers that learn from experience!

Junior level — plain language, no maths

Think about teaching a dog to sit. You don't hand it a rulebook - you show it, reward it when it gets there, gently correct it when it doesn't, and after enough tries it just knows. Machine learning works almost exactly like this, except the "dog" is a program and the "treats" are numbers - scores that go up when it's right and down when it's wrong.

Ordinary programming means spelling out every rule by hand: "if the email says 'lottery', call it spam." But spam that dodges the word slips right through. A machine learning system takes the opposite approach - feed it thousands of real spam and real emails and it hunts down patterns no human would ever think to write down, then builds its own rules from scratch.

Today's most powerful version stacks neural networks - layers of tiny maths units loosely modelled on brain cells. Each one takes in numbers, multiplies them by weights it has learned, and passes the result along. Wire millions of these together, train them on billions of examples, and you get the thing that unlocks your phone with your face, translates a hundred languages on the fly, and paints pictures from a sentence. In the simulation below, watch a small network learn to tell two kinds of dots apart, live.

Gradient descent, backpropagation, and the universal approximation theorem

Student level — the core equations

A network of \(L\) layers is just one big nested function, \(f(x) = W_L\,\sigma(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots) + b_{L-1}) + b_L\), where each \(\sigma\) is a nonlinearity - ReLU \(\sigma(z)=\max(0,z)\) or sigmoid \(\sigma(z)=1/(1+e^{-z})\) - and the weights \(W_i\) and biases \(b_i\) are what the network learns. Without the \(\sigma\)'s the whole stack would collapse to a single linear map; the nonlinearity is what lets depth buy you anything. Training means tuning \(\theta=\{W_i,b_i\}\) to shrink a loss, say the cross-entropy \(L = -\sum_i y_i \log \hat{y}_i\).

Gradient descent nudges every parameter downhill on that loss: \(\theta \leftarrow \theta - \eta\,\nabla_\theta L\), with \(\eta\) the learning rate. The trick that makes it feasible on billions of parameters is backpropagation - the chain rule run backwards through the network, reusing each layer's work so one sweep computes every gradient. In practice you estimate \(\nabla_\theta L\) on small random mini-batches (stochastic gradient descent), noisy but cheap, and adaptive optimisers like Adam keep a separate step size per parameter to converge faster.

Why should this ever work? The Universal Approximation Theorem (Cybenko 1989, Hornik 1991) proves a single hidden layer, given enough units, can mimic any continuous function as closely as you like. But "enough" can be astronomically many; depth is what makes it practical, representing compactly what a shallow net would need exponentially many neurons for. The real fight isn't fitting the training data - it's generalising to new data, and tools like dropout, weight decay and batch normalisation exist to stop a network from simply memorising its examples.

Optimisation landscape, generalisation theory, and transformers

Scholar level — full mathematical depth

01The loss landscape isn't the trap we feared

A deep network's loss \(L(\theta)\) is wildly non-convex in millions of dimensions, and for years the fear was that gradient descent would get stuck in bad local minima. High dimensions turned out to be kinder than expected. Almost every critical point is a saddle, not a minimum - to be a true minimum, all million-plus Hessian eigenvalues must happen to be positive, which is vanishingly unlikely - and the local minima that do exist mostly sit near the global one (Dauphin et al., 2014). The optimiser slides off saddles rather than drowning in traps.

02Flat minima generalise; sharp ones don't

Not all minima are equal. The sharpness of a solution - the largest Hessian eigenvalue \(\lambda_{\max}(\partial^2 L/\partial\theta^2)\) - tracks how well it generalises: broad, flat basins are forgiving of small perturbations and tend to transfer to new data, while sharp spikes overfit. Sharpness-Aware Minimisation makes this explicit, optimising the worst loss in a neighbourhood, \(\min_\theta \max_{\|\varepsilon\|\le\rho} L(\theta+\varepsilon)\), and reliably nudging test accuracy up. You want the widest valley, not merely the lowest point.

03The overparameterisation puzzle

Classical learning theory says a model with far more parameters than data points should overfit catastrophically. Deep networks do the opposite - they fit the training set perfectly and still generalise, a state now called benign overfitting. PAC-Bayes bounds like \(L(f) \le \hat{L}(f) + \sqrt{\tfrac{\mathrm{KL}(Q\|P) + \ln(n/\delta)}{2n}}\) can stay non-vacuous even here when the prior is chosen well, but the honest summary is that we have a working theory only in pieces. One of the field's most successful technologies isn't yet fully explained.

04Double descent and the bias of SGD

Push past the point where a model can exactly memorise the data - the interpolation threshold - and something strange happens: test error, having peaked, falls again. This double descent curve quietly overturns the old bias-variance story. Part of the answer is that stochastic gradient descent has an implicit bias: among the infinitely many parameter settings that fit the data, it drifts toward low-norm, "simple" ones. The regulariser was hiding in the optimiser all along.

05The Transformer and self-attention

Nearly all of today's frontier models are Transformers (Vaswani et al., 2017), which threw out recurrence in favour of self-attention: \(\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\tfrac{QK^\top}{\sqrt{d_k}}\right)V\). Every token compares itself to every other and pulls in a weighted mix of their values - quadratic \(O(n^2)\) cost, but only \(O(1)\) sequential depth, so it parallelises across a whole sequence at once. Stack many such heads, each learning a different pattern of relationships, add positional encodings for order, and you have the engine behind modern language models.

06Scaling laws: intelligence you can budget

The strangest empirical fact about deep learning is how predictable it is at scale. Test loss falls as a clean power law in model size, data and compute across seven orders of magnitude. The Chinchilla analysis (Hoffmann et al., 2022) pinned the optimal recipe: for a compute budget \(C\), grow parameters and training tokens together, roughly \(N \propto C^{0.5}\) and \(D \propto C^{0.5}\). That capability can be dialled up by spending more - rather than waiting for a new idea - is exactly what turned deep learning from a research curiosity into an industry.