Giles' blog

LLM from scratch

Between December 2024 and December 2025, I worked through Sebastian Raschka's book "Build a Large Language Model (from Scratch)", writing up the things I found interesting and/or surprising, and filling in some of the gaps. The book is a brilliant explanation of how to build an LLM, but sometimes glosses over why we do particular things -- for example, why certain sequences of matrix multiplications result in a working, trainable attention mechanism.

After that, I wanted to flesh things out a bit, so spent six months doing in-depth studies into hyperparameter tweaks, training upwards of twenty GPT-2-small sized models, and reimplementing the code from my notes in JAX.

Here are the posts in this epic series:

• Writing an LLM from scratch, part 1 (22 December 2024). Kicking off -- covering the introductory first chapter in the book. Adorably, I thought I might be able to finish the book over my Christmas break :-)
• Writing an LLM from scratch, part 2 (23 December 2024). The basics of embeddings and tokenisation.
• Writing an LLM from scratch, part 3 (26 December 2024). Wrangling data so that it can be handled by an LLM, plus token and position embeddings.
• Writing an LLM from scratch, part 4 (28 December 2024). Starting to try to understand attention by looking at the history.
• Writing an LLM from scratch, part 5 -- more on self-attention (11 January 2025). More on attention -- the history, and a little more detail on what it actually is.
• Writing an LLM from scratch, part 6 -- starting to code self-attention (21 January 2025). Our first cut at an attention mechanism -- a non-trainable one.
• Writing an LLM from scratch, part 6b -- a correction (28 January 2025). You can skip this one, it was a post to alert people who were reading along that the previous one had a (since-corrected) error.
• Writing an LLM from scratch, part 7 -- wrapping up non-trainable self-attention (7 February 2025). Still digging into the example of non-trainable attention.
• Writing an LLM from scratch, part 8 -- trainable self-attention (4 March 2025). Finally, an explanation of self-attention that makes sense! (At least to me...)
• Writing an LLM from scratch, part 9 -- causal attention (9 March 2025). Causal, or masked self-attention: when we're considering a token, we don't pay attention to later ones.
• Writing an LLM from scratch, part 10 -- dropout (19 March 2025). Adding dropout to the LLM's training is pretty simple, though it does raise one interesting question...
• Writing an LLM from scratch, part 11 -- batches (19 April 2025). Batching speeds up training and inference, but for LLMs we can't just use matrices for it -- we need higher-order tensors.
• Writing an LLM from scratch, part 12 -- multi-head attention (21 April 2025). Finally getting to the end of chapter 3! This time it’s multi-head attention: what it is, how it works, and why the code does what it does.
• Writing an LLM from scratch, part 13 -- the 'why' of attention, or: attention heads are dumb (8 May 2025). Realising that attention heads are simpler than I thought explained why we do the calculations we do.
• Writing an LLM from scratch, part 14 -- the complexity of self-attention at scale (14 May 2025). Starting to build intuition on how self-attention scales (and why the simple version doesn't).
• Writing an LLM from scratch, part 15 -- from context vectors to logits; or, can it really be that simple?! (31 May 2025). The way we get from context vectors to next-word prediction turns out to be simpler than I imagined -- but understanding why it works took a bit of thought.
• Writing an LLM from scratch, part 16 -- layer normalisation (8 July 2025). Working through layer normalisation -- why do we do it, how does it work, and why doesn't it break everything?
• Writing an LLM from scratch, part 17 -- the feed-forward network (12 August 2025). The feed-forward network is one of the easiest parts of an LLM in terms of implementation -- but when I thought about it I realised it was one of the most important.
• Writing an LLM from scratch, part 18 -- residuals, shortcut connections, and the Talmud (18 August 2025). The book's description of shortcut connections is overly-simplified, I think -- residuals are more than just shortcuts to help with gradients.
• Writing an LLM from scratch, part 19 -- wrapping up Chapter 4 (29 August 2025). A state-of-play update after finishing Chapter 4, with a roadmap of what’s coming next.
• Writing an LLM from scratch, part 20 -- starting training, and cross entropy loss (2 October 2025). Starting training our LLM requires a loss function, which is called cross entropy loss. What is this and why does it work?
• Writing an LLM from scratch, part 21 -- perplexed by perplexity (7 October 2025). Raschka calls out perplexity in a sidebar, but I wanted to understand it in a little more depth.
• Writing an LLM from scratch, part 22 -- finally training our LLM! (15 October 2025). Finally, we train an LLM! The final part of Chapter 5 runs the model on real text, then loads OpenAI’s GPT-2 weights for comparison.
• Writing an LLM from scratch, part 23 -- fine-tuning for classification (22 October 2025). After all the hard work, chapter 6 is a nice easy one -- how do we take a next-token predictor and turn it into a classifier?
• Writing an LLM from scratch, part 24 -- the transcript hack (28 October 2025). Back when I started playing with LLMs, I found that you could build a (very basic) chatbot with a base model -- no instruction fine-tuning at all! Does that work with GPT-2?
• Writing an LLM from scratch, part 25 -- instruction fine-tuning (29 October 2025). Some notes on the first part of chapter 7: instruction fine-tuning.
• Writing an LLM from scratch, part 26 -- evaluating the fine-tuned model (3 November 2025). Coming to the end of the book! We evaluate the quality of the responses our model produces, manually and automatically.
• Writing an LLM from scratch, part 27 -- what's left, and what's next? (4 November 2025). Having finished the main body of the book, it's time to think about what I need to do to treat this project as fully done.
• Writing an LLM from scratch, part 28 -- training a base model from scratch on an RTX 3090 (2 December 2025). I felt like it should be possible to train a GPT-2 small level model on my own hardware using modern tools and open datasets from scratch. It was!
• Writing an LLM from scratch, part 29 -- using DistributedDataParallel to train a base model from scratch in the cloud (7 January 2026). Having trained a base model from scratch on my own machine over 48 hours, I wanted to make it faster by training with multiple GPUs in the cloud.
• Writing an LLM from scratch, part 30 -- digging into the LLM-as-a-judge results (9 January 2026). I was unhappy with the LLM-as-a-judge instruction fine-tuning results I got when comparing my various base models. Could I make them any better?
• Writing an LLM from scratch, part 31 -- the models are now on Hugging Face (17 January 2026). I've trained seven models using the GPT-2 architecture: let's share them on Hugging Face!
• Writing an LLM from scratch, part 32a -- Interventions: training a baseline model (4 February 2026). I want to try a bunch of interventions like gradient clipping and removing dropout to see if my models get better. I need a baseline train without them so that I can be sure of the results.
• Writing an LLM from scratch, part 32b -- Interventions: gradient clipping (5 February 2026). Does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected.
• Writing an LLM from scratch, part 32c -- Interventions: removing dropout (5 February 2026). Does removing dropout improve our baseline model's test loss? Yes, absolutely, and even more than gradient clipping did.
• Writing an LLM from scratch, part 32d -- Interventions: adding attention bias (6 February 2026). Having bias terms for the query, key, and value attention weights is apparently no longer used because it doesn't help. Let's check that it really doesn't for our model!
• Writing an LLM from scratch, part 32e -- Interventions: the learning rate (10 March 2026). The learning rate is an essential hyperparameter for training models with gradient descent, but what actual values should you use, and what does it mean to schedule it?
• Writing an LLM from scratch, part 32f -- Interventions: weight decay (23 March 2026). What is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?
• Writing an LLM from scratch, part 32g -- Interventions: weight tying (24 March 2026). Weight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?
• Writing an LLM from scratch, part 32h -- Interventions: full fat float32 (3 April 2026). I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead to a worse test loss at the end?
• Writing an LLM from scratch, part 32i -- Interventions: what is in the noise? (7 April 2026). How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.
• Writing an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloud (9 April 2026). Now that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?
• Writing an LLM from scratch, part 32k -- Interventions: training a better model locally with gradient accumulation (15 April 2026). Having worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about gradient accumulation.
• Writing an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning results (20 April 2026). I wanted to revisit the instruction fine-tuning tests I used to do, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!
• Writing an LLM from scratch, part 32m -- Interventions: conclusion (21 April 2026). Wrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!
• Writing an LLM from scratch, part 33 -- what I learned from finally getting round to the appendices (22 April 2026). I've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.
• Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run (30 June 2026). I'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.
• Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX) (8 July 2026). The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.