What is nanochat?
The "best ChatGPT that $100 can buy" — a complete, readable LLM pipeline from raw text to interactive chatbot.
nanochat is Andrej Karpathy's capstone project for his LLM101n course. It is the successor to nanoGPT (now deprecated) and goes far beyond it — where nanoGPT stopped at pretraining, nanochat implements the full pipeline from tokenizer training all the way to an interactive chat UI with tool use.
The entire project is roughly ~8,000 lines of code across Python, Rust, and C/CUDA. Every line is meant to be read and understood. There are no hidden abstractions — no HuggingFace Trainer, no transformers library. Just raw PyTorch, a custom tokenizer in Rust, and clear, educational code.
The result? For about $100, a model that reaches ~31% on MMLU (well above GPT-2's chance-level ~25%) and approaches GPT-2-grade quality on the CORE benchmark — the $100 depth-20 speedrun scores CORE ~0.22 (just under GPT-2's ~0.26), and a slightly larger run matches it. It can hold conversations, follow instructions, and even execute Python code.
nanochat at a Glance
Why nanochat matters for learning
End-to-end visibility
Most LLM frameworks hide the details behind abstractions. nanochat exposes everything: the tokenizer is written from scratch in Rust, the model is raw PyTorch, training loops are explicit, and there is no magic.
Every chapter, one codebase
Tokenization (Ch 1), embeddings (Ch 2), transformers (Ch 3), pretraining (Ch 4), normalization and activations (Ch 5), prompting (Ch 8), fine-tuning (Ch 13), decoding (Ch 14), GPU inference (Ch 15) — all represented in nanochat's code.
Modern architecture choices
nanochat does not replicate the original 2017 transformer. It uses RoPE, RMSNorm, ReLU-squared, QK-norm, untied embeddings, and other modern improvements — the same family of choices powering LLaMA, Gemma, and other frontier models.
Successor to nanoGPT
nanoGPT only covered pretraining a base model. nanochat adds the entire post-training pipeline: midtraining on conversations, supervised fine-tuning, reinforcement learning, and a chat interface with tool use.
The Full Pipeline
nanochat's training has 5 stages, each corresponding to concepts from earlier chapters. Click each stage to explore what happens, what data is used, and why.
Interactive Training Pipeline
Click each stage to see its details. The flow follows the exact order nanochat trains a model from nothing to a working chatbot.
Training
Tokenizer Training
Ch 1Remember from Chapter 1: before a model can process text, it needs a tokenizer to convert characters into integer token IDs. nanochat trains its own BPE tokenizer from scratch.
A subset of FineWeb-EDU — a curated, education-focused web corpus. The tokenizer sees raw text to learn common byte-pair patterns.
BPE (Byte Pair Encoding) iteratively merges the most frequent adjacent byte pairs into new tokens. Starting from 256 raw bytes, it builds up to a 65,536-token vocabulary.
Written in Rust for speed. Tokenization must process billions of tokens, so Python would be too slow. The Rust tokenizer achieves ~1.8M tokens/sec.
It is a power of 2 (216), which is efficient for GPU memory alignment. Larger vocabularies reduce sequence length (good) but increase embedding table size (trade-off). 64K is a sweet spot used by many modern models.
Pretraining
Ch 3, 4This is the most expensive stage — and the foundation of everything. As we covered in Chapter 4, pretraining teaches the model language by predicting the next token on massive text data.
~11 billion tokens from FineWeb-EDU (drawn from a ~100B-token downloaded pool), streamed during training. The data is pre-tokenized and stored in binary shard files for fast loading. No data is ever repeated (one epoch).
Next-token prediction (causal language modeling). Given tokens [1, 2, ..., N], predict token [N+1]. Cross-entropy loss between predicted logits and actual next tokens.
Muon optimizer for the matrix (2D) weights, plus AdamW for the embedding and unembedding layers, cosine learning rate schedule with warmup, gradient clipping at 1.0, weight decay of 0.1. Distributed across 8 GPUs using DDP (DistributedDataParallel).
The model transforms from random weights to a capable text completer. It learns grammar, facts, reasoning patterns, and style — but it cannot hold a conversation or follow instructions yet.
Midtraining
Ch 8A relatively new concept in the LLM pipeline. Midtraining bridges the gap between raw pretraining and fine-tuning. As we discussed in Chapter 8, models need to learn conversational formats and special tokens for tool use.
A mix of conversational data formatted with special turn-taking tokens (<|user|>, <|assistant|>), plus continued general text. This teaches the model the chat format without forgetting general knowledge.
The model learns to recognize conversation structure, tool use markers (for Python code execution), and the distinction between user and assistant turns. Still next-token prediction, but on formatted data.
Supervised Fine-Tuning (SFT)
Ch 13Now we apply the techniques from Chapter 13. SFT teaches the model to be a helpful assistant by training on curated instruction-response pairs.
Conversational and task data: SmolTalk (general conversations, the main component) plus MMLU (knowledge), GSM8K (grade-school math), and spelling tasks. Much smaller than pretraining data — roughly 800K examples, not billions.
Same cross-entropy loss, but only on the assistant's response tokens. The user's prompt tokens are masked from the loss — we do not want to teach the model to generate user messages, only to respond to them.
Pretraining uses a lower learning rate and processes massive data. SFT uses a much smaller learning rate (to avoid catastrophic forgetting) and tiny, high-quality datasets. It is essentially "steering" an already capable model.
The model goes from "can complete text" to "can follow instructions." It learns to answer questions, solve problems step-by-step, and format responses appropriately.
Reinforcement Learning (GRPO)
Ch 4The final stage uses a simplified form of RL — GRPO (Group Relative Policy Optimization) — to improve math reasoning. This connects back to the reward-based optimization concepts introduced in Chapter 4.
GSM8K math problems. The model generates multiple candidate answers for each problem. Correct answers (verifiable by checking the final number) get a reward of 1; incorrect answers get 0.
For each problem, generate a group of responses. Compute a reward for each (correct = 1, wrong = 0). Each response's reward relative to the group average (its advantage) is used to update the policy — no separate reward model or critic needed. Karpathy's simplified version reduces essentially to REINFORCE.
Math has a verifiable correct answer, which makes it easy to compute rewards automatically. For creative or open-ended tasks, you would need human labelers or a trained reward model — nanochat keeps it simple.
The model improves at multi-step mathematical reasoning. It learns to show its work and arrive at correct answers more consistently, without losing its general conversational ability.
Architecture Choices
nanochat does not use the original 2017 transformer. It adopts the same modern design improvements found in LLaMA, Gemma, and other recent models. Each choice connects back to concepts from Chapter 3.
nanochat vs. "Textbook" Transformer
The left column shows the classic 2017 "Attention Is All You Need" choice. The right column shows what nanochat actually uses. The explanation column describes why.
RoPE Intuition
RoPE rotates the query and key vectors by angles proportional to their position in the sequence. When you compute Q · K, the rotation angles subtract, leaving a function of the relative distance between tokens. The model never sees absolute positions — only how far apart tokens are.
ReLU² Sparsity
Standard ReLU zeros out negatives but leaves positives unchanged. ReLU² zeros out negatives and squares positives, so values near zero are suppressed while large values are amplified. Because it zeros all negative pre-activations, a large fraction of FFN activations end up at zero — only a subset of neurons fire strongly for any given token.
The "One Dial" Design
nanochat scales via a single hyperparameter: --depth. Every other architectural dimension is derived from it. Move the slider to see how the model scales.
Interactive Model Scaling
The depth parameter controls the number of transformer layers. All other dimensions — model width, number of heads, head dimension — are computed from it using fixed ratios. This is a deliberate design choice: by reducing the search space to a single axis, Karpathy avoids the combinatorial explosion of hyperparameter tuning.
Model Size vs Depth
Why "one dial"?
When you have multiple independent hyperparameters (layers, width, heads, head_dim, FFN multiplier), you face a combinatorial search space. By coupling everything to a single parameter, nanochat guarantees that each model size is well-proportioned. The ratios are inspired by scaling laws research and practical experience from the LLaMA model family.
How dimensions are derived
Given a depth d, the model dimension is d * 64, number of attention heads is d_model / 128, and each head has dimension 128 (required by Flash Attention 3). The FFN hidden dimension is 4 * d_model (following the standard 4x expansion). This means the model grows in both depth and width simultaneously.
Training Data & Pipeline
How nanochat gets its data, tokenizes it at 1.8M tokens/sec, and distributes training across 8 GPUs — connecting back to Chapter 1 and Chapter 15.
Data Pipeline Overview
web corpus
quality scoring
Rust, 1.8M tok/s
.bin files
streaming
FineWeb-EDU
A dataset of web pages filtered for educational quality, created by HuggingFace. Unlike using all of Common Crawl (which is full of spam, ads, and low-quality content), FineWeb-EDU selects pages that would be useful in an educational context. This quality filtering is crucial — remember from Chapter 4 that data quality matters more than quantity.
Distributed Training
Training runs across 8 H100 GPUs using PyTorch's DistributedDataParallel (DDP). As we covered in Chapter 15, DDP replicates the model on each GPU, gives each one a different data shard, and synchronizes gradients with all-reduce after each backward pass.
shard 0
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
Streaming Tokenization
nanochat's Rust tokenizer processes text into binary token files at ~1.8 million tokens per second. This is not a bottleneck — tokenizing the data is a one-time cost, and the resulting .bin files contain packed 16-bit token IDs that the PyTorch data loader can memory-map and read without any parsing overhead during training.
From Base Model to Chatbot
Each training stage transforms the model's behavior. This visual pipeline shows the progression from "word salad" to "helpful assistant" — connecting Chapter 4 and Chapter 13.
Behavior Transformation
Click each stage to see an example of how the model would respond to: "What is 15 + 28?"
Inference & KV Cache
How nanochat generates text at inference time — the prefill/decode split and KV cache optimization from Chapter 3, Chapter 14, and Chapter 15.
Two-Phase Generation
As we explored in Chapter 14, text generation has two distinct phases with very different computational profiles.
Phase 1: Prefill
Process the entire prompt in parallel. All tokens attend to all previous tokens, filling the KV cache with computed key/value pairs for every layer and position.
Phase 2: Decode
Generate one token at a time. Each new token only needs to compute attention against the cached keys and values, not reprocess the entire sequence.
KV Cache: Why It Matters
Without a KV cache, generating the N-th token would require reprocessing all N-1 previous tokens through every attention layer — an O(N²) operation. The KV cache stores the K and V projections for every previous token at every layer, reducing each decoding step to O(N).
nanochat uses full Multi-Head Attention in the speedrun, but the code is written to support grouped-query attention (GQA) — the technique that lets production models shrink the KV cache. In standard MHA with, say, 16 heads, you store 16 separate K and V tensors per layer. With GQA/MQA, multiple query heads share fewer K/V heads — collapsing to a single shared K/V (MQA) cuts the cache to 1/16th. This trade-off — slightly less model quality for dramatically better inference efficiency — is why nearly all production models now use GQA.
Multi-Head Attention (what nanochat uses)
Example: 16 layers, 16 heads, d_head=64, seq=2048 = 128 MB
Multi-Query Attention (the GQA extreme)
Example: 16 layers, 1 KV head, d_head=64, seq=2048 = 8 MB (16x smaller)
Tool Use: Python Code Execution
nanochat can generate Python code blocks during a conversation, execute them in a sandboxed environment, and incorporate the results into its response. This is the practical application of the tool use concepts from Chapter 8. During midtraining, the model learned special tokens that mark code blocks for execution. At inference time, when the model emits a code-execution marker, the runtime intercepts the output, runs the code, and feeds the result back as context for continued generation.
The Big Picture
What nanochat teaches us about frontier models.
From $100 to $100 million
nanochat proves that building a ChatGPT-class model is no longer a billion-dollar mystery. The entire pipeline — from raw bytes to interactive conversation — fits in ~8,000 lines of readable code and costs about $100 to train.
What makes frontier models like GPT-4 or Claude different is not architectural magic: it is scale (trillions of tokens, thousands of GPUs, months of training), data curation (carefully curated, multilingual, diverse training mixtures), alignment (extensive RLHF with human labelers), and engineering (custom kernels, mixed precision, fault tolerance across massive clusters).
But the core ideas? They are exactly what you have learned in these 17 chapters. Every concept in nanochat — every BPE merge, every attention head, every gradient update — maps directly to a section in this book. You now understand, at a mechanical level, how large language models work.