Deep Learning, from First Principles

A neural network is just a function

Strip away the jargon and a neural network is a plain function: numbers in, numbers out. Feed it the 784 pixel values of a handwritten digit, and it returns 10 numbers scoring how much the image looks like a "0", a "1", … a "9".

What makes it special is that the function has millions of tunable constants inside it, called parameters (or weights). You never write the recognition logic yourself. Instead:

As a senior engineer, think of it as the inverse of normal programming. Normally you write code and the data flows through it. Here you fix the code's shape and let the data write the constants. Training is a search through parameter-space for the constants that make the function behave.

Everything below — nn.Linear, nn.ReLU, logits, cross-entropy, SGD — is just machinery for those three steps. Let's build it up from a single neuron.

The neuron: a weighted sum with an attitude

A single artificial neuron does something you learned in school: it computes a line.

x is the input, w is the weight (how much the input matters, and in which direction), and b is the bias (a baseline offset, like the intercept of a line). That's it. A neuron with one input is the equation of a straight line, and w and b are the two knobs the optimiser will later turn.

Play with the knobs below and watch the function change. This is literally what "learning" will adjust — nothing more mystical than this.

With several inputs, the neuron just extends the sum — one weight per input:

Read it as a relevance-weighted vote: each input votes, its weight says how much (and whether) its vote counts, and the bias sets the default disposition. A spam-filter neuron might learn a big positive weight on the "contains FREE!!!" feature and a negative weight on "sender is in contacts".

nn.Linear: a whole layer of neurons at once

One neuron gives one output. A layer is simply many neurons reading the same inputs in parallel, each with its own weights and bias, each producing one output. Stack their weight rows into a matrix W and their biases into a vector b, and the whole layer becomes one matrix multiplication:

That is all nn.Linear(in_features, out_features) is: a matrix of shape (out, in), a bias vector of shape (out,), and one matmul. nn.Linear(4, 3) holds 4×3 + 3 = 15 parameters. A layer isn't a new concept — it's the neuron from Part 1, vectorised.

Every line in the animation is one weight — one multiply. Each output node sums everything arriving at it, adds its bias, done. When people say a model "has 7 billion parameters", they mean the total count of these connection weights across all its layers.

In PyTorch

import torch
import torch.nn as nn

layer = nn.Linear(4, 3)      # W: (3,4) matrix, b: (3,) vector — created with random values
x = torch.randn(4)          # a fake input vector
y = layer(x)                # y = W @ x + b   → 3 numbers out

print(layer.weight.shape)   # torch.Size([3, 4]) ← these are the knobs training will turn
print(layer.bias.shape)     # torch.Size([3])

nn.ReLU: why networks need non-linearity

Here's a trap worth understanding deeply. Suppose you stack two linear layers:

Multiply it out and you get y = (W₂W₁)·x + (W₂b₁ + b₂) — which is just another single linear layer with a different matrix. Stacking linear layers buys you nothing. A 100-layer purely-linear network collapses to a 1-layer one. It can still only draw straight lines.

The fix is almost comically simple: between layers, apply a non-linear "kink" to every number. The most popular kink is the Rectified Linear Unit:

Negative in → 0 out. Positive in → unchanged. That's the entire function. It has no parameters, nothing to learn — it's a fixed, dumb bend. But because the composition of "linear → bend → linear → bend…" can no longer be flattened into one linear map, the network gains the power to build curves out of straight pieces, the way a polygon approximates a circle.

Try 2 hidden neurons — the fit stays crude, only a couple of bends available. Try 24 — the curve hugs the sine. This is the universal approximation idea in miniature: with enough hinged pieces, a network can approximate essentially any continuous function. Depth and width buy you more pieces.

In PyTorch

model = nn.Sequential(
    nn.Linear(1, 8),   # 1 input → 8 hidden values
    nn.ReLU(),         # bend each of the 8 (no parameters!)
    nn.Linear(8, 1),   # blend the 8 bent pieces into 1 output
)

nn.Flatten: reshaping, not learning

nn.Linear expects a flat vector of numbers. But an image arrives as a grid — say 28×28 pixels. nn.Flatten is nothing but the adapter between the two: it unrolls the grid, row by row, into one long vector. A 28×28 image becomes a 784-vector. No parameters, no maths, no learning — pure reshaping, like array.flat() in JavaScript or reshaping a 2D array in C.

Worth noticing what's lost: after flattening, the network no longer knows that pixel 29 sat directly below pixel 1. Spatial neighbourhood information is discarded — the linear layer just sees 784 independent numbers. Simple models accept this loss; convolutional networks (a later topic) were invented precisely to keep it.

In PyTorch

flatten = nn.Flatten()
x = torch.randn(32, 1, 28, 28)  # a batch of 32 grayscale images
x = flatten(x)
print(x.shape)                 # torch.Size([32, 784]) — batch dim kept, rest unrolled

Note the first dimension (the batch of 32 images processed together) survives — Flatten only unrolls everything after it. Batching exists purely for efficiency: GPUs are happiest multiplying big matrices, so we push many samples through at once.

The forward pass: composing the pipeline

You now know every part. A classic digit-classifier is just these parts composed, function-style — data flows left to right through the whole thing:

The same picture as code

model = nn.Sequential(
    nn.Flatten(),          # 28×28 image  → 784 vector        (plumbing)
    nn.Linear(784, 128),   # 784 → 128    (100,480 knobs)     (learned)
    nn.ReLU(),             # bend                             (fixed)
    nn.Linear(128, 10),    # 128 → 10     (1,290 knobs)       (learned)
)
logits = model(image)      # the "forward pass": just calling the function

Running input through the model is called the forward pass — it really is nothing more than calling a composed function. The 10 numbers that come out the far end are called logits, and they deserve their own section, because they confuse everyone at first.

Logits and softmax: from raw scores to probabilities

The final linear layer outputs raw, unbounded real numbers — maybe [2.3, −1.1, 0.4]. These are logits: unnormalised evidence scores, one per class. Bigger = more confident, but they aren't probabilities. They can be negative, they don't sum to 1, only their relative differences carry meaning.

To turn logits into a proper probability distribution we apply softmax:

Exponentiate every logit (making everything positive and amplifying gaps), then divide by the total (making them sum to 1). It's a "soft" version of picking the max: the winner gets most of the probability mass, but the losers keep a share proportional to how close they were.

Two things to try above. First, slide all three logits up by the same amount — the probabilities don't change (only differences matter). Second, push one logit far above the rest — softmax saturates towards certainty. This exponential amplification is why a logit gap of just 3–4 already means >95% confidence.

The loss function: quantifying "wrong"

To improve the model automatically, we need wrongness as a single number — a loss — that we can minimise. The choice depends on the task:

For regression (predicting a quantity, like a house price), the classic is mean squared error: average of (prediction − truth)². Simple, punishes big misses quadratically.

For classification, the standard is cross-entropy, and its core idea fits in one sentence: the loss is the negative log of the probability the model assigned to the correct answer.

If the image is a cat and the model said "cat: 100%", the loss is −log(1) = 0. Said "cat: 50%"? loss ≈ 0.69. Said "cat: 1%"? loss ≈ 4.6 — and it keeps exploding towards infinity as the model gets confidently wrong. Drag the slider to feel the shape:

Notice the asymmetry — the crucial design feature. Going from 90% → 99% confident saves you a little loss. Going from 10% → 1% on the true class costs you enormously. Cross-entropy therefore spends the model's capacity stamping out confident mistakes first. It also gives large gradients exactly where the model is most wrong, which (next section) means the biggest corrective nudges happen where they're needed most.

In PyTorch

loss_fn = nn.CrossEntropyLoss()    # fuses softmax + log + pick-correct-class, stably

logits = model(images)             # (batch, 10) raw scores — NOT softmaxed
loss   = loss_fn(logits, labels)   # labels: (batch,) integer class ids, e.g. tensor([3, 7, ...])
# loss is ONE scalar: the average -log p(correct) over the batch

Gradient descent: how the knobs actually get turned

Now the centrepiece. We have a loss that depends on every one of the model's parameters. Picture the loss as a landscape: each point on the ground is one particular setting of all the knobs, and the altitude is how wrong the model is there. Training = finding a low valley.

You can't survey the whole landscape (with a million knobs it's a million-dimensional space). But at your current position you can compute, cheaply, the local slope in every direction — the gradient. The gradient of the loss with respect to each weight answers: "if I nudge this knob up a hair, does the loss rise or fall, and how steeply?"

The algorithm is then embarrassingly simple — gradient descent:

Take a small step downhill, everywhere at once. lr is the learning rate: the step size, the single most important hyperparameter in deep learning. Feel why below:

Experiments to run: with a tiny learning rate the ball inches along and can stall in the shallow local dip. With a huge one it overshoots and ricochets — loss can even diverge. A moderate rate sometimes carries enough momentum-of-step to hop the small bump and reach the deeper valley. Real training is exactly this, in a million dimensions simultaneously.

Where do gradients come from? Backpropagation

The network is a chain of simple operations, so the chain rule of calculus lets us compute ∂loss/∂w for every weight in one backwards sweep. PyTorch records every operation during the forward pass into a graph; calling loss.backward() walks that graph in reverse, depositing each weight's gradient into weight.grad. You never derive anything by hand — this is autograd, and it's the reason frameworks like PyTorch exist at all.

In PyTorch

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

loss.backward()        # backprop: fills w.grad for every parameter w
optimizer.step()       # w ← w - lr * (processed) w.grad, for all params
optimizer.zero_grad()  # reset grads to 0 (they accumulate by default — classic gotcha)

The training loop: the whole story in 10 lines

Every piece is now on the table. Deep learning training, in its entirety, is this loop:

for epoch in range(num_epochs):            # an epoch = one full pass over the dataset
    for images, labels in dataloader:      # mini-batches, e.g. 32 samples at a time

        logits = model(images)             # 1. FORWARD: run the function
        loss   = loss_fn(logits, labels)   # 2. LOSS: one number, "how wrong?"

        optimizer.zero_grad()              # 3. clear old gradients
        loss.backward()                    # 4. BACKWARD: slopes for every knob
        optimizer.step()                   # 5. UPDATE: nudge every knob downhill

Forward, loss, backward, step. Millions of tiny nudges, each making the model infinitesimally less wrong on the current batch, and structure gradually crystallises out of the initially-random weights. There is no other magic — GPT and a digit classifier both train with exactly this loop.

Playground: train a real network, right here

Below is a genuine neural network — Linear(2, h) → ReLU → Linear(h, 2) — implemented in JavaScript with hand-written backprop, training live in your browser. The task: classify 2D points (orange vs blue). The shaded background shows the model's current decision boundary; watch it start random and morph to wrap the data as the loss falls.

Things worth trying, mapped back to the theory:

Hidden = 2 on the circle dataset: too few ReLU hinges — the boundary stays crude (Part 3). Crank the learning rate to 1.0: watch the loss curve go jagged or explode (Part 8). Two spirals with 48 neurons: hard shapes need capacity and patience. And notice the boundary is always made of straight-ish segments joined at angles — you can literally see the ReLU pieces.

Glossary: the ideas, one line each

Where to go next

You now have the full mental model behind the PyTorch beginner tutorial — revisit it and every line should feel obvious. Natural next concepts, in order: train/validation split and overfitting (why low training loss isn't the goal), convolutions (linear layers that keep the spatial structure Flatten throws away), and embeddings & attention (how the same machinery scales to language models).