Watch one prediction become a weight update.

Define the example

The prompt is “Pick the blue …” and the target token is part. The candidate vocabulary contains only part, pallet and tool. We supply the feature vector x = [0.8, 0.2, −0.4] and a small weight matrix with one row for each candidate.

This is a deliberately limited output-head example. The features are fixed illustrative values; no tokenizer, attention layers or full transformer are hidden behind the diagram.

Compute a distribution

Each candidate receives a logit from the dot product of its weight row and the input vector: z = Wx. Softmax converts the logits into a probability distribution. The subtraction of the largest logit in the implementation improves numerical stability without changing the result.

For the target part, cross-entropy is L = −log p(part). A confident prediction for the wrong candidate can therefore have a high loss even if the response looks plausible to a reader.

Differentiate the loss

For this softmax and cross-entropy combination, the logit gradient is p − y, where y is the one-hot target. The gradient for a weight is (p − y) × x.

Inspect the three components in the apparatus. The sign tells us the direction of the local update; the magnitude depends on both the prediction and the input feature. All three candidate rows are updated, not just the target row.

Apply one update

The apparatus uses one plain gradient-descent step: W′ = W − η∇W L. The learning-rate control changes η. During playback, the interpolation reveals the update continuously; it is not a series of extra training steps.

We then recompute logits, probabilities and loss with the updated weights. The displayed improvement is specific to this one synthetic example. It says nothing about generalisation to a held-out dataset.

What a real programme adds

A training programme introduces batches, changing inputs, larger architectures, an optimiser, data splits and repeated evaluation. The system also needs to preserve data permissions, experimental configuration and checkpoint state.

The same discipline still applies: define the target, inspect the learning signal and compare the result under an explicit evaluation protocol. See our model engineering approach and expert data programmes.