Model Families & Training Objectives

The Evolution of Language Models

From counting word pairs to predicting the next token with 400 billion parameters.

1990s-2000s

N-gram Models

Count how often word sequences appear. "I want to" is often followed by "go" or "eat." Fast and simple, but can only look at 2-5 words back, and can't generalize — "I'd like to" is treated as completely unrelated.

2013-14

Word2Vec (2013) / GloVe (2014)

Learn a fixed vector per word from co-occurrence patterns. "king" and "queen" get similar vectors. But each word has only ONE vector — "bank" is the same whether it means riverbank or financial institution.

2014-2017

RNNs / LSTMs

Process tokens sequentially, maintaining a hidden state. Can model variable-length context. But sequential processing = slow training, and long-range dependencies still fade. Dominated NLP for ~4 years.

2017+

Transformers

"Attention Is All You Need" (Vaswani et al., 2017). Parallel processing, direct long-range attention, scales to billions of parameters on modern hardware. Every major LLM today is a transformer.

2018-now

Foundation Models

Pretrain once on massive data, then adapt to many tasks. BERT (2018) for understanding, GPT-2 (2019) for generation, GPT-3 (2020) for few-shot learning, ChatGPT/Claude (2022-23) for instruction following.

The Three Pretraining Objectives

The training objective is the most important design choice. It determines what the model learns to do well.

Next-Token Prediction

(Autoregressive)

            The cat sat on the ???
          

Given all previous tokens, predict the next one. Trained left-to-right.

Best for: Generation, chat, code, creative writing

Models: GPT, Claude, LLaMA, Mistral

Masked Language Modeling

(Bidirectional)

            The [MASK] sat on the mat
          

Hide random tokens, predict them from both left AND right context.

Best for: Classification, search, NER, understanding

Models: BERT, RoBERTa, DeBERTa

Span Corruption / Seq2Seq

(Encoder-Decoder)

            The <X> the mat -> cat sat on
          

Replace spans with sentinels, generate the missing content.

Best for: Translation, summarization, structured tasks

Models: T5, BART, mBART, Flan-T5

Masked Language Modeling: BERT's Objective

Hide 15% of the tokens, predict them from both sides. This is how BERT learns deep understanding.

BERT's 15% masking strategy

Original: "The cat sat on the mat"
15% of tokens selected for prediction
Of those selected tokens:

              80% -> replaced with [MASK]
            
              10% -> replaced with random token
            
              10% -> left unchanged

Why this weird 80/10/10 split?

Without it, the model would only learn to predict [MASK] tokens — a symbol that never appears in real text. The split forces the model to:

80% [MASK]: The main training signal. Model learns to predict from context.

10% random: Replaces with a random token — teaches the model that what it sees might be wrong, building robustness to noise.

10% unchanged: Keeps the original token — bridges the gap between pretraining (which uses [MASK]) and fine-tuning (which never does), so the model maintains good representations for actual tokens.

What RoBERTa changed (and why it matters)

BERT (2018)

Static masking (same masks every epoch)
Next Sentence Prediction (NSP) loss
Trained on ~16 GB of text
Batch size 256

RoBERTa (2019)

Dynamic masking (different random masks each epoch)
Removed NSP (turned out to hurt)
Trained on 160 GB of text (10x more)
Batch size 8,000 (31x larger)

Same architecture. All gains from training recipe.

Seq2Seq: Text-to-Text (T5's Approach)

Convert every NLP task into "text in, text out." Classification, translation, summarization — all the same format.

T5's unified framing: everything is text-to-text

Classification

            Input: "mnli premise: A man is eating. hypothesis: A person is dining."

            Output: "entailment"

Translation

            Input: "translate English to German: That is good."

            Output: "Das ist gut."

Summarization

            Input: "summarize: [long article text...]"

            Output: "[summary]"

Question Answering

            Input: "question: What is the capital? context: France's capital is Paris."

            Output: "Paris"

Span corruption (T5's pretraining objective)

Instead of masking individual tokens (BERT), T5 corrupts contiguous spans and replaces them with sentinel tokens. The decoder generates only the missing spans.

Original: "The cat sat on the fluffy mat"
Input: "The <X> on the <Y> mat"
Target: "<X> cat sat <Y> fluffy"

More efficient than MLM because replacing spans with single sentinels produces shorter target sequences.

T5 model sizes

Variant	Parameters
Small	60M
Base	220M
Large	770M
3B	3B
11B	11B

Flan-T5 = T5 instruction-tuned on 1,800+ tasks. Same architecture, double-digit accuracy gains, converges 2-5x faster on downstream tasks.

Model Families: The Specs

Exact parameter counts, training data, and context lengths for every major model family.

Encoder models (understanding)

Model	Year	Params	Layers	Hidden	Heads	Context	Objective
BERT base	2018	110M	12	768	12	512	MLM + NSP
BERT large	2018	340M	24	1,024	16	512	MLM + NSP
RoBERTa	2019	125/355M	12/24	768/1,024	12/16	512	MLM only
DeBERTa v3	2021	86-304M	12-24	768-1,024	12-16	512	RTD (replaces MLM)

Decoder models (generation)

Model	Year	Params	Layers	Hidden	Heads	Context	Training Tokens
GPT-2 Small	2019	124M	12	768	12	1,024	~10B (WebText, 40GB text)
GPT-2 XL	2019	1.5B	48	1,600	25	1,024	~10B (same WebText)
GPT-3	2020	175B	96	12,288	96	2,048	300B
LLaMA 1 65B	2023	65B	80	8,192	64	2,048	1.4T
LLaMA 2 70B	2023	70B	80	8,192	64	4,096	2.0T
LLaMA 3 8B	2024	8B	32	4,096	32	8,192	15T+
LLaMA 3.1 405B	2024	405B	126	16,384	128	128K	15T+
Mistral 7B v0.1	2023	7.3B	32	4,096	32	8,192 (4K SW)	undisclosed
Mixtral 8x7B	2023	46.7B (12.9B active)	32	4,096	32	32K	undisclosed
LLaMA 4 Scout	2025	109B (17B active, MoE 16E)	undisclosed	undisclosed	undisclosed	10M	undisclosed
LLaMA 4 Maverick	2025	400B (17B active, MoE 128E)	undisclosed	undisclosed	undisclosed	1M	undisclosed
Mistral Large 3	2025	675B (41B active, MoE)	undisclosed	undisclosed	undisclosed	256K	undisclosed
Closed-source models — architecture details not publicly disclosed
GPT-5.5	2026	undisclosed	undisclosed	undisclosed	undisclosed	1M	undisclosed
Claude Opus 4	2025	undisclosed	undisclosed	undisclosed	undisclosed	200K	undisclosed
Gemini 3.5	2026	undisclosed	undisclosed	undisclosed	undisclosed	1M	undisclosed

Generative vs Discriminative Models

Two fundamentally different approaches — and why the line between them is blurring.

Generative

Models the distribution of the data itself — can generate new samples. Learns P(x) or P(x|condition).

Strengths: Flexible — chat, code, creative writing, reasoning. Can do classification via prompting.

Weaknesses: Expensive, slower, harder to calibrate for narrow tasks.

Examples: GPT-4, Claude, LLaMA

Discriminative

Maps inputs directly to labels or decisions. Learns P(label|input) without modeling how the input was generated.

Strengths: Efficient, fast, easy to calibrate. Great for narrow, high-volume tasks.

Weaknesses: Can't generate text. Needs labeled data per task.

Examples: BERT + classifier head, logistic regression, XGBoost

The line is blurring: GPT-4 can do classification via prompting (generative model doing discriminative tasks), and BERT can generate text with masked token infilling. Modern practice: use generative models for flexibility, discriminative for efficiency.

Foundation Models vs Task-Specific Models

One model for everything, or a specialized model per task?

Foundation Model

Pretrained broadly on diverse data. Adapted via prompting, fine-tuning, or RAG.

+ One model, many tasks

+ Few-shot / zero-shot capability

+ Amortizes training cost

- Expensive to run

- Harder to control

- May hallucinate

Task-Specific Model

Trained or fine-tuned for one narrow job. Optimized for that specific task.

+ Fast and cheap to run

+ Well-calibrated for its task

+ Easier to validate

- Needs labeled data per task

- Can't generalize

- Must retrain for changes