Semester 3 · Language models
Building and Training LLMs from Scratch
https://epochschool.com/subjects/llm-from-scratch
Build a language model the way large labs do, at a size one CPU core can train: a production byte-level BPE tokeniser, a cleaned and deduplicated pretraining corpus, a GPT-2-exact model and a resumable training loop.
You then account for every FLOP, byte of activation memory and unit of cost of a Transformer run, write the FlashAttention backward pass, plan parallel layouts, use scaling laws to spend a compute budget, and look inside your model with probes, the logit lens and activation patching. Finally you modernise it with RoPE, RMSNorm, SwiGLU and grouped-query attention, make it cheaper with speculative decoding and int8, and evaluate it honestly as a base model.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Implement a production byte-level BPE tokeniser (deterministic training with incremental pair counts, rank-order encoding, special tokens, lossless decoding and vocabulary extension) and quantify how vocabulary size and training data change compression, sequence length and cost across languages and scripts.
- LO2Build and measure a pretraining data pipeline: text extraction, language identification, heuristic and model-based quality filtering, PII masking, exact and MinHash-LSH near-duplicate removal, and mixture weighting.
- LO3Implement a GPT-2-style decoder-only language model exactly (loadable module names and weight layout, tying and initialisation), derive its exact parameter count, and justify each choice.
- LO4Write, monitor and debug a reproducible language-model pretraining loop (token files and the data position, AdamW parameter groups, schedules and budgets in tokens, checkpoint and resume) and diagnose loss-curve pathologies.
- LO5Account exactly for the FLOPs, activation memory (including the T^2 attention term), KV cache, and time and dollar cost of Transformer training and inference, building on the general accounting of Optimisation & Training at Scale.
- LO6Derive and implement in miniature the IO-aware (Flash) attention backward pass, and plan sequence, context, expert, tensor, pipeline and data-parallel layouts for a Transformer of a given size on a given cluster.
- LO7Fit scaling laws to training runs, use them to choose compute-optimal model and data sizes, adjust for inference cost and scarce data, predict the loss of larger runs, and judge when an extrapolation can be trusted.
- LO8Implement and compare modern architecture and inference techniques: RMSNorm, rotary embeddings with cache offsets, SwiGLU, grouped-query attention with its KV cache, mixture-of-experts routing, speculative decoding and weight-only int8 quantisation.
- LO9Evaluate base models with tokeniser-independent likelihood metrics and likelihood-based benchmarks, detect train-test contamination, and report results with honest uncertainty.
- LO10Look inside a small GPT with linear probes and controls, the logit lens, attention-pattern analysis of induction heads and activation patching, and state what each method can and cannot show.
Chapters
The chapter and lab list for this subject is being finalised and will be published here.
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Optimisation & Training at Scale
Neural Networks & Deep Learning
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
NLP & Language Model Foundations
Where it leads
Subject 2 of 4 in the Language models track.
Previous: NLP & Language Model FoundationsNext: Post-training, Alignment and Evaluation of LLMs
Post-training, Alignment and Evaluation of LLMs
LLM Applications: Retrieval, RAG and Agents
Capstone Project
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.