Why Transformers Changed Everything
Before transformers (2017), sequence models processed tokens one at a time. Transformers process them all at once.
Before: RNNs / LSTMs
Sequential: must finish word 1 before starting word 2. Long-range dependencies fade as the chain grows. Training is slow — can't parallelize.
After: Transformers
Parallel: every token can attend to every other token simultaneously. Long-range dependencies are direct. Training is massively parallel on GPUs.
The breakthrough was both algorithmic (attention replaces recurrence) and practical (scales to modern hardware). This is why every major LLM — GPT, Claude, LLaMA, Gemini — is built on transformers.
Inside a Transformer Block
Every transformer is a stack of identical blocks. Each block has two main stages: self-attention (mix information across tokens) and feed-forward (transform each token independently).
What each component does:
End-to-End: Watch Vectors Flow Through a Tiny LLM
A 2-layer, 2-head transformer with 8-dimensional vectors. Step through each stage from raw text to predicted next token — every number is real and traceable.
Self-Attention: The Core Mechanism
The single idea that makes transformers work. Best understood through a concrete example.
The problem attention solves
Consider this sentence:
The word "bank" is ambiguous — financial institution or riverbank? The token embedding for "bank" is the same either way (it's just a lookup table). So how does the model figure out which meaning is intended?
Answer: it looks at the other words. "steep" and "mud" are strong signals for riverbank. "closed" and "account" would signal financial. Attention is the mechanism that lets "bank" look at every other token and absorb the relevant context.
The core idea in one sentence: Each token creates a search query ("what context do I need?"), every other token advertises what it has, the best matches share their information. That's attention.
Q, K, V: The library analogy
Think of attention as a library search. Every token plays two roles simultaneously: it's both a visitor searching for books and a book on the shelf.
Query (Q)
"What am I searching for?"
The visitor's search request. "bank" might generate a query that means "I'm an ambiguous noun — I need context about my environment."
Q = token_state * W_QA learned projection that extracts "what I need" from each token's current state.
Key (K)
"What's written on my spine?"
The book's label/title. "steep" generates a key that says "I'm an adjective describing physical terrain." The search engine matches queries to keys.
K = token_state * W_KA learned projection that extracts "what I have to offer" from each token.
Value (V)
"Here's the book's actual content."
The full content of the book. Once "bank" finds that "steep" is relevant, it reads the value of "steep" to absorb its meaning. Keys are for matching; values are for reading.
V = token_state * W_VA learned projection that extracts "the content I share" from each token.
Interactive: Follow "bank" through attention
Click each step to see how "bank" resolves its ambiguity by attending to other tokens.
Where do Q, K, V come from?
Every token produces all three — Q, K, and V — by multiplying its embedding by three different learned weight matrices. Same input, three different projections, three different roles:
When generating "on" (the current token), the model needs to decide what context is relevant. Q comes from the current token. K and V come from all previous tokens (including the current one):
The formula (now that you have the intuition)
This shows bidirectional attention (as in BERT-style encoder models). In causal/decoder models (GPT, Claude), tokens can only attend to previous tokens — the upper-right triangle would be masked to zero.
Multi-Head Attention
One attention head can only focus on one type of relationship at a time. Multiple heads let the model look for many patterns simultaneously.
How it works
Instead of one big attention computation, the model splits the embedding into h smaller pieces (heads), runs attention on each independently, then concatenates the results. By convention, d_head = d_model / n_heads so the concat reconstructs the full dimension — but this is a design choice, not a hard requirement.
where head_i = Attention(Q * W_Qi, K * W_Ki, V * W_Vi)
Each head has its own learned W_Q, W_K, W_V projection matrices, so each head learns to look for different things. Researchers have identified specific head types like induction heads (pattern completion) and copy suppression heads (preventing repetition). See section 10 for the full picture with interactive examples.
Head dimensions across real models
| Model | Q heads | KV heads | d_model | d_head |
|---|---|---|---|---|
| GPT-2 Small | 12 | 12 | 768 | 64 |
| GPT-2 XL | 25 | 25 | 1600 | 64 |
| LLaMA 3 8B | 32 | 8 (GQA) | 4096 | 128 |
| LLaMA 3 70B | 64 | 8 (GQA) | 8192 | 128 |
Key relationship: Q heads × d_head = d_model always (e.g., 32 × 128 = 4096). In GQA models, KV heads < Q heads — each KV head is shared by a group of Q heads (e.g., 32 Q heads ÷ 8 KV heads = groups of 4), reducing KV cache size by 4×.
Residual Connections & Layer Norm
The unsung heroes that make deep transformers (80+ layers) trainable at all.
Residual Connections
LayerNorm subtracts the mean and divides by standard deviation per vector (not per batch), then applies learned scale (γ) and shift (β) parameters. This keeps values in a stable range so deep layers don't explode or collapse.
The residual connection (Add) lets the original signal pass through unchanged, so each layer only needs to learn a small refinement:
Pre-LN vs Post-LN
LN(x + Attn(x))
Normalize after adding. Used in original Transformer. Can be unstable in very deep models — needs careful warmup.
x + Attn(LN(x))
Normalize before the sublayer. Much more stable training. Used by GPT-2+, LLaMA, and most modern LLMs.
RMSNorm (Root Mean Square Layer Norm) is a simplified variant used by LLaMA and Mistral — it only normalizes the scale, not the mean, making it ~10-15% faster.
Positional Encoding
Attention is permutation-invariant — "dog bites man" and "man bites dog" look the same without position info. Positional encoding fixes this.
Sinusoidal (original Transformer)
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Fixed, deterministic waves at different frequencies. Each position gets a unique pattern. Can generalize to unseen sequence lengths because sin/cos are continuous functions.
Applied: once, before layer 1. Added per-dimension to the token embedding (same shape: 768 or 4096 dims).
Used by: Original Transformer (Vaswani et al., 2017)
Learned Embeddings
Just add a learnable vector for each position. The model discovers the best position representation during training. Simple but limited to the maximum training sequence length.
Applied: once, before layer 1. Added per-dimension, same shape as token embedding.
Used by: GPT-2, BERT
RoPE (Rotary Position Embeddings)
Instead of adding position info to the embeddings, RoPE rotates pairs of dimensions in the Q and K vectors based on their position.
Take dimensions d0 and d1 of a Q vector. RoPE treats them as a 2D point and rotates by an angle that depends on the token's position:
q'_d1 = q_d0 × sin(θ) + q_d1 × cos(θ)
where θ = position / 10000^(2i/d)
Say Q has d0=0.5, d1=0.3 at position 5 with θ=0.5 radians:
The magic: When Q at position 5 dot-products with K at position 3, the rotation angles partially cancel — the dot product depends only on the difference (5-3=2), not the absolute positions. Two tokens 2 apart always interact the same way, whether they're at positions 5,3 or 100,98.
Each dimension pair rotates at a different frequency (the circles above): low dims rotate fast (fine position detail), high dims rotate slowly (coarse position). Like a clock with second, minute, and hour hands.
Applied: at every layer, inside attention. Rotates dimension pairs (d0-d1, d2-d3, ...) in Q and K before the dot product. Does NOT touch V or the input embedding.
Used by: LLaMA, Mistral, Gemma, most modern LLMs
ALiBi (Attention with Linear Biases)
Doesn't modify embeddings or Q/K vectors at all. Instead, adds a scalar penalty to the attention score based on distance: farther tokens get lower scores. Each head uses a different decay rate.
Extremely simple and extrapolates well to longer sequences than seen during training.
Applied: at every layer, as a bias on attention scores. No modification to embeddings or Q/K/V vectors — just a distance penalty added after Q*K^T.
Used by: BLOOM, MPT
Where does each method inject position info?
| Method | What it modifies | When | Granularity |
|---|---|---|---|
| Sinusoidal | Input embedding | Once (before layer 1) | Per dimension (sin/cos wave per dim pair) |
| Learned | Input embedding | Once (before layer 1) | Per dimension (learned value per dim) |
| RoPE | Q and K vectors | Every layer (inside attention) | Per dimension pair (rotates d0-d1, d2-d3, ...) |
| ALiBi | Attention scores | Every layer (after Q*K^T) | Per token pair (scalar bias by distance) |
Interactive: How sinusoidal PE gets added to a token
The token embedding is a vector of numbers. The positional encoding is another vector of the same size. They get added element-wise. Drag the position slider to see how the PE changes — the token stays the same, but the result shifts.
Showing first 16 of 64 dimensions. Bar heights represent values. Notice how changing the position shifts the green PE bars (especially the low-frequency left side stays similar for nearby positions) while the blue token embedding stays constant.
Positional encoding visualizations
Each row is a position (0-39), each column is a dimension. Colors show the sin/cos values — lower dimensions (left) change slowly, higher dimensions (right) change rapidly.
Architectures & Modern Variants
The attention mask controls the architecture. Causal masking → decoder (generation). Bidirectional → encoder (understanding). Plus modern efficiency variants.
Bidirectional (Encoder models)
Every token can attend to every other token — past AND future. The model sees the full context. Used for understanding tasks: classification, NER, retrieval.
Models: BERT, RoBERTa, DeBERTa
Causal / Autoregressive (Decoder models)
Each token can only attend to itself and previous tokens — never the future. The model generates autoregressively: one token at a time, feeding each output back as input for the next step ("The" → "The cat" → "The cat sat" → ...). Auto (self) + regressive (predicting from prior values).
Models: GPT, Claude, LLaMA, Mistral
Encoder-Only
bidirectional
Every token attends to every other token — past AND future. Produces rich representations but can't generate text autoregressively.
Decoder-Only most LLMs today
causal (left-to-right)
Each token can only see tokens before it (causal mask). Generates text one token at a time. This is how ChatGPT, Claude, etc. work.
Encoder-Decoder
Encoder reads the full input bidirectionally. Decoder generates output autoregressively but can also attend to the encoder's output via cross-attention — a second attention where Q comes from the decoder but K,V come from the encoder.
Multi-Query Attention (MQA)
All heads share the same K and V projections. Only Q differs per head. Cuts KV cache by ~Nx (where N = number of heads).
Used by: PaLM, Falcon
Grouped-Query Attention (GQA)
A middle ground: heads are divided into groups, and each group shares K and V. Better quality than MQA, smaller cache than full MHA.
Used by: LLaMA 2/3, Mistral, Gemma 2+ — the modern default
Sliding Window Attention
Each token only attends to the last W tokens (e.g., 4096). Reduces O(n^2) to O(n*W). Information still flows across the full sequence through stacked layers.
Used by: Mistral 7B (W=4096)
Interactive: How Q and KV heads connect
Toggle between MHA, GQA, and MQA to see how query heads map to key/value heads. Each Q head needs a K,V pair — the question is whether each Q gets its own K,V or shares.
Mixture of Experts (MoE)
Not an attention variant, but a transformer scaling trick. Replace the single FFN with multiple "expert" FFNs. A router network picks the top-k experts (usually 2) per token. The model has many more parameters but only activates a fraction for each token — better quality at lower compute cost.
Used by: Mixtral (8x7B = 46.7B params, 12.9B active), GPT-4 (rumored), Switch Transformer
Efficiency: KV Cache & Cost
The optimizations that make long-context inference practical — and why they're necessary.
Interactive: Step through generation with KV Cache
Click Generate next token to watch the KV cache grow as the model generates "The cat sat on." At each step, see exactly what gets computed vs reused.
The trade-off: KV cache saves massive compute but costs memory. For LLaMA 2 70B (80 layers, GQA with 8 KV heads, fp16) at 128k context, the KV cache consumes ~40 GB per request. Full MHA (64 KV heads) would require ~320 GB — an 8x blowup. This is why techniques like GQA, MQA, and quantized KV caches are essential.
Attention complexity: O(n^2 * d)
Compute and memory grow quadratically with sequence length. Doubling context from 4k to 8k roughly quadruples attention cost.
| Sequence Length | Attention FLOPs (relative) | KV Cache (70B, 8 KV heads, fp16) | Real-world note |
|---|---|---|---|
| 2,048 | 1x | ~0.6 GB | GPT-3 original context |
| 4,096 | 4x | ~1.2 GB | LLaMA 2 default |
| 32,768 | 256x | ~10 GB | Many current models |
| 128,000 | 3,900x | ~39 GB | GPT-4 Turbo, Claude |
| 1,000,000 | 238,000x | ~305 GB | Gemini 1.5 Pro |
What Do Heads & Neurons Actually Learn?
Mechanistic interpretability: reverse-engineering the circuits inside transformers to understand why they produce specific outputs.
Attention heads specialize
Not all attention heads do the same thing. During training, heads develop distinct roles — some track syntax, some copy tokens, some suppress predictions. Researchers have identified several recurring head types:
Previous Token Heads
Always attend to position i-1. They feed the "what came right before" signal to downstream layers. Found in early layers.
Induction Heads
Pattern-match: if [A][B] appeared before, and [A] appears again, predict [B]. The core mechanism behind in-context learning. Found in mid layers. (Olsson et al., 2022)
Duplicate Token Heads
Detect when the same token has appeared earlier in context. They signal "I've seen this before" to help the model track references and repetition.
Copy Suppression Heads
When an upstream layer naively predicts a token, these heads attend to where that token appeared and write an opposing signal to suppress the prediction. They prevent the model from blindly copying. (e.g., GPT-2 L10H7)
Interactive: See what each head type looks like
Click a head type to see its idealized attention pattern on a sample sentence. These are illustrative patterns showing the characteristic shape of each head type, not outputs from a specific model. The heatmap shows how much each token (row) attends to every other token (column). Bright = high attention.
Interactive: Induction head in action
An induction head learns the pattern: "if [A][B] appeared before, and [A] appears again, predict [B]." Click tokens in the sequence to see which earlier tokens the induction head attends to and why.
Neurons, superposition & polysemanticity
Individual neurons are hard to interpret because of superposition: models pack more concepts than they have dimensions, so a single neuron activates (produces a high output value after the activation function, e.g., GELU or SwiGLU) for multiple unrelated things (polysemanticity). A neuron might activate for both "dog photos" and "the concept of democracy."
The problem: polysemantic neurons
One neuron activates for multiple unrelated concepts. You can't point to a single neuron and say "this represents X." The model stores ~100x more features than it has dimensions.
The solution: sparse autoencoders
SAEs decompose activations into a sparse, interpretable basis. Anthropic extracted 34M interpretable features from Claude 3 Sonnet — 70% rated genuinely interpretable by humans. (Anthropic, 2024)
How SAEs work
Decode: ĥ = W_dec · z (reconstruction)
Loss: ||h - ĥ||² + λ · ||z||₁ (reconstruct + encourage sparsity)
The key insight: z is sparse — only a few features activate for any given input. Each active feature corresponds to one interpretable concept (a language, a topic, an entity, a syntactic role). Features are multimodal and multilingual: the "Golden Gate Bridge" feature activates for English text, Japanese text, and images of the bridge.
Circuits: how heads & neurons compose
Heads and neurons don't work alone — they form circuits through the residual stream. Information flows: head writes to residual → neuron reads from residual → next head reads the updated residual. Researchers have mapped complete circuits for specific tasks:
Interactive: Indirect Object Identification (IOI) circuit
Given "Mary and John went to the store. Mary gave a book to ___", the model should predict "John." This requires tracking who is who and who already acted. Click the circuit stages to see how information flows:
Why this matters: Understanding circuits lets researchers steer model behavior. By clamping or amplifying specific SAE features during inference, you can increase truthfulness, reduce toxicity, or suppress sycophancy — more targeted than RLHF. This is a core technique for AI safety research.
Transformer Failure Modes
Understanding what can go wrong helps you design better systems around the model.
Attention Dilution
On very long contexts, attention weights spread thin across thousands of tokens. Important tokens may receive too little weight. This is why "lost in the middle" is a real phenomenon — models often perform worse on information placed in the middle of long prompts.
Positional Degradation
Models trained on shorter sequences may perform poorly when extended to longer ones, even with position encoding extrapolation. RoPE NTK-scaling and YaRN help but don't fully solve this.
Hallucination
The model generates plausible-sounding but factually wrong content. This is partly because the training objective (predict next token) rewards fluency, not factual accuracy. RAG and grounding help mitigate this.
Decoding Instability
Temperature, top-k, and top-p settings can dramatically change output quality. Too high = incoherent. Too low = repetitive. The model itself is deterministic; the randomness is in the sampling strategy.