Why tokenize at all?
AI models don't read words — they read numbers. Tokenizers convert text into numbered pieces.
The Tokenization Spectrum
There are three levels to split text. Modern models use subword methods — the sweet spot in the middle.
Word-level
1 token per word. Vocab = every word ever seen (500k+). Problem: can't handle new/misspelled words ("unkown" = ???)
Subword (BPE, WordPiece, Unigram)
Splits into reusable pieces. Vocab ~30k-100k. Sweet spot: handles any word, compact sequences
Character-level
1 token per character. Tiny vocab (~256). Problem: sequences are very long, model must learn spelling from scratch
BPE: Byte-Pair Encoding
A method — a recipe for merging the most frequent character pairs, step by step.
Byte-level BPE
What modern LLMs actually use — BPE on raw bytes, not characters. No "unknown token" ever.
The Problem with Character BPE
Character-level BPE (the demo above) works great for English. But what happens with:
Unicode has 150,000+ characters. You'd need a massive base vocabulary just for individual characters, and rare scripts would still be "unknown."
The Byte-level Solution
Instead of starting with characters, start with the 256 possible byte values. Every piece of text — in any language, any script, any emoji — is just bytes underneath.
Then BPE merges run on these bytes. Common byte sequences (like ASCII letters) merge quickly into familiar pieces. Rare characters stay as byte sequences — ugly but never "unknown."
How GPT-2 made it work
"Hello"
word boundaries
[48, 65, 6C, 6C, 6F]
to visible chars
["Hello"]
GPT-2's trick: map each of the 256 byte values to a visible Unicode character (so bytes look like printable text), then run standard BPE on those mapped characters.
The result: a single algorithm that handles every language, every emoji, every encoding — with a base vocab of just 256.
Used by: GPT-2/3 (50K vocab), GPT-3.5/4 via tiktoken (100K vocab), GPT-4o (200K vocab), Claude, and most modern LLMs. Same byte-level BPE technique, but each model trains its own vocabulary.
WordPiece
A method — like BPE, but picks merges by likelihood instead of raw frequency. Used by BERT.
BPE: Merge the most frequent pair
Simple rule: whichever pair appears most often, merge it. Fast and effective.
WordPiece: Merge the pair that maximizes likelihood
Smarter rule: prefer pairs that always go together over pairs that are just individually common. "q" and "u" almost never appear apart, so merging them is very informative.
WordPiece's ## prefix
WordPiece marks continuation tokens with ##. This tells you which pieces start a word vs. continue one:
Used by: BERT, DistilBERT, Electra, MobileBERT, and other models from Google's NLP line.
SentencePiece: The Toolbox
A software tool by Google — it can run BPE or Unigram inside it.
Raw Text Input
"Hello world"
No spaces needed!
Key Superpower
Treats the input as a stream of bytes — no assumptions about spaces or word boundaries. Works for every language.
SentencePiece
Pick your algorithm:
BPE vs Unigram inside SentencePiece
Bottom-up: start with characters, greedily merge the most frequent pair. Same algorithm as standalone BPE.
-> h e ll o
-> h ell o
-> hell o
-> hello
Top-down: start with a huge vocabulary, prune the least useful pieces. Picks the most probable segmentation.
X Remove least useful...
X Keep pruning...
-> ["hel", "lo"] (highest probability split)
Interactive BPE Trainer
Write a training corpus, train BPE from scratch, then test the learned tokenizer on new text.
The Full Picture
Methods are recipes. Tools are kitchens. Models pick a combination of both.
| BPE method | Byte-level BPE method | WordPiece method | Unigram method | SentencePiece tool | |
|---|---|---|---|---|---|
| What is it? | Algorithm | Algorithm (BPE variant) | Algorithm | Algorithm | Software library (Google) |
| Direction | Bottom-up merges | Bottom-up merges | Bottom-up merges | Top-down pruning | Runs BPE or Unigram |
| Merge criterion | Most frequent pair | Most frequent pair | Highest likelihood gain | Remove least useful piece | Depends on algorithm chosen |
| Starts from | Characters | 256 byte values | Characters | All substrings (huge) | Raw text (no pre-splitting) |
| Unknown tokens? | Possible (rare chars) | Never (any byte encodable) | Possible (uses [UNK]) | Rare (falls back to chars) | Depends on algorithm |
| Word boundary | Pre-tokenize by spaces | Regex pre-split | Pre-tokenize, ## prefix | No pre-tokenization needed | Uses ▁ (space marker) |
| Typical vocab | 30k-100k | 50k-100k | 30k-110k | 32k-64k | You choose |
| Used by | Original GPT | GPT-2/3/4, Claude | BERT, DistilBERT | T5, ALBERT, XLNet | LLaMA 1/2, T5, ALBERT, mBART |
Why Tokens Are Not Words
The same "amount" of text can produce wildly different token counts depending on the content.
Efficient (few tokens)
Expensive (many tokens)
Token-expensive content (watch out for these)
Special Tokens
Hidden structural markers that control how the model interprets the sequence. You rarely see them, but they shape everything.
Common special tokens
Chat/instruction tokens
Modern chat models use special tokens to mark role boundaries. Getting these wrong causes subtle bugs:
Each model family has its own format (ChatML, LLaMA's format, etc.). Mismatched turn markers during fine-tuning is a common source of degraded quality.
Context Windows
The maximum number of tokens a model can process at once. Everything — instructions, history, retrieved docs, AND the generated output — must fit.
The context window is a working desk, not a filing cabinet
Drop tokens from the front or back. Cheapest but riskiest — you might cut the instruction the model needs most.
Process long text in overlapping chunks. Preserves local detail but can't do global reasoning across the full document.
Compress older context into a summary. Preserves gist but loses exact wording. Used in long chat sessions.
| Model | Context Window | ~Word Equivalent | Note |
|---|---|---|---|
| GPT-3 (original) | 2,048 | ~1,500 words | Could barely fit a long email |
| GPT-3.5 Turbo | 4,096 / 16,384 | ~3K / ~12K words | Original 4K; 16K variant added later |
| GPT-4 Turbo | 128,000 | ~96,000 words | A full novel |
| Claude 3 (200K) / Claude 4 (1M) | 200,000 - 1,000,000 | ~150K - 750K words | Multiple books |
| Gemini 1.5 Pro | 1,000,000 | ~750,000 words | An entire codebase |
Token Budgeting in Production
Tokens = compute = money = latency. Smart budgeting is a first-class engineering problem.
Why tokens matter for cost
LLM APIs bill per token. More tokens = more $$$, more compute, higher latency. Two prompts that look similar to a human can differ dramatically in token count.
Input (prefill) — all tokens hit the GPU in one parallel batch. High hardware utilization, very efficient.
Output (decode) — tokens are generated one at a time, sequentially (can't produce token N+1 until token N is known). Each step runs the new token through all layers, attending to the entire KV cache. The actual math per token is small, but the GPU must read all model weights + cached K/V from memory each time — making it memory-bandwidth bound (compute cores sit idle waiting for data). 100 output tokens = 100 serial passes.
Budget backward from the limit
Start from the maximum window and subtract fixed costs. What remains is your retrieval budget.
In practice, measure average AND worst-case (P99) token usage — the "tail." If your average prompt is 4k tokens but 1% of prompts hit 12k, those outliers will overflow and silently truncate. Budget for the tail, not the average.