📰 Dev.to · Gary Jackson

Run several attention heads in parallel on embedding slices, add a two-layer MLP for per-position computation, and assemble a transformer block.

Build causal self-attention with Q/K/V projections, scaled dot-product scoring, softmax weights, and a KV cache for sequential processing.

Add two stabilisation patterns deep networks need: RMSNorm to keep activations bounded, and residual connections to give gradients a highway.

Assemble a full training loop: forward, loss, backward, and Adam parameter updates with momentum, adaptive scaling, and learning rate decay.

Give tokens and positions learned vector identities, assemble a minimal forward pass to logits, and compute cross-entropy loss.

Implement a counting-based bigram model to pin down the next-token prediction task and establish a loss baseline before neural networks enter.

Build a character-level tokenizer with a BOS delimiter that converts between characters and integer IDs.

Implement Backward() on Value: topologically sort the graph, then walk it in reverse using the chain rule to fill every .Grad.