The Training Pipeline
Building a modern LLM is not one step — it's a multi-phase pipeline. This chapter covers Phase 1. The next chapter covers Phases 2 and 3.
Next-Token Prediction: The Simplest Powerful Idea
Predict the next word. That's it. This objective trained GPT-4, Claude, and every modern chatbot.
Interactive: Watch next-token prediction in action
The loss function
Where x_1, x_2, ..., x_T are the tokens in a single training sequence (e.g., x_1="The", x_2="cat", x_3="sat") — not the entire dataset. At each position t, the model predicts the next token x_{t+1} given all previous tokens in that sequence. The total training loss averages this across all sequences in the dataset.
Cross-entropy loss: for each position, how surprised is the model by the correct next token? Lower surprise = better model. Summed over every position in every training document.
Teacher forcing
During training, the model always sees the ground-truth previous tokens, not its own predictions. This enables parallel training (all positions computed at once) but creates "exposure bias" — at inference, the model conditions on its own (possibly wrong) outputs.
Why does "just predicting the next word" produce intelligence?
Language is a compressed encoding of human knowledge. To predict the next token well across billions of documents — science papers, code, conversations, stories — the model must build internal representations of logic, facts, causality, and social conventions. It doesn't "understand" in the human sense, but it approximates the statistical structure of all the knowledge encoded in text.
Scaling Laws: Bigger vs Better-Trained
Two landmark papers that changed how the industry trains models.
Kaplan et al. (2020, OpenAI)
"Scaling Laws for Neural Language Models"
Found that loss scales as a power law with model size, data, and compute — across 7+ orders of magnitude. Concluded: bigger models are more sample-efficient, so train very large models on "modest" data.
This led to GPT-3 (175B params, only 300B tokens) — huge model, relatively little data.
Hoffmann et al. (2022, DeepMind)
"Chinchilla" — Training Compute-Optimal Models
Overturned Kaplan's conclusion: models were undertrained, not undersized. For compute-optimal training, use ~20 tokens per parameter. Double the model size? Double the training data too.
The Chinchilla effect on the industry
After Chinchilla, the industry shifted toward training on much more data. For open-weight models especially, training smaller models much longer became the dominant strategy. Some labs (OpenAI, Google) also scaled up model size using MoE architectures, but the universal trend was more tokens:
15T tokens (1,875:1)
8T tokens (889:1)
18T tokens (250:1)
2T tokens (30:1)
3.5T tokens (19:1)
~7T tokens (est. ~1,000:1)
Chinchilla's optimal ratio was ~20 tokens per parameter. Today's small models train at ratios of 250:1 to 1,875:1 — up to ~94× the Chinchilla optimum — deliberately investing more in training to make inference cheaper.
What Comes After Pretraining?
Pretraining produces a powerful base model — but it just autocompletes text. It won't answer questions or follow instructions.
A pretrained model has learned language, facts, and reasoning patterns from trillions of tokens. But without further training, it's like a brilliant encyclopedia that can only continue whatever text you start — it has no concept of "user" and "assistant."
The next chapter covers post-training: how SFT, RLHF, DPO, GRPO, and other techniques turn this raw capability into a useful, safe, and reasoning-capable assistant.