Post-Training: Alignment, Tuning & Reasoning

The Training Pipeline

Pretraining teaches the model to predict text. Alignment teaches it to follow instructions and be helpful. But why split training into phases at all?

Why not train everything at once?

Each training phase uses different data, different objectives, and different scales. Combining them would create conflicting goals — the model can't simultaneously learn grammar from trillions of web pages AND learn to refuse harmful requests from a few thousand examples. The phases build on each other:

PHASE 1: PRETRAINING

Learn language

Predict the next token on trillions of tokens from the internet. Learns grammar, facts, reasoning patterns, code, and world knowledge.

→

PHASE 2: MIDTRAINING (increasingly standard)

Extend capabilities

Continue training on curated, high-quality data: long documents, code, math, multilingual text. Extends context length and deepens specific skills. Not all models include this phase — some go directly from pretraining to post-training.

→

PHASE 3: POST-TRAINING

Align & teach reasoning

Three sub-steps, typically in this order:

1. SFT (Supervised Fine-Tuning)

Train on human-written instruction/response pairs (10K-1M+ examples)

2. Reasoning RL (GRPO, PPO, or similar)

RL on math/code with verifiable answers — teaches step-by-step thinking

3. Alignment RL (RLHF / DPO)

RL from human preference — helpful, harmless, honest

Reasoning is now standard across frontier labs: OpenAI (o-series), Anthropic (extended thinking), Google (deep think), DeepSeek (R1), and open-source models (Qwen3, Llama-Nemotron)

Why this order matters — ceiling vs accessibility: A useful way to think about it: pretraining sets the ceiling of what the model can do — its knowledge, reasoning ability, and language fluency. Post-training determines the accessibility — whether users can actually get the model to do useful things. A pretrained model just continues whatever text you give it; it won't answer questions or follow instructions. The InstructGPT result (below) is striking: a 1.3B-parameter aligned model was preferred by humans over an unaligned 175B model — not because post-training is "more important," but because without it, the larger model's massive capability was inaccessible. Midtraining sits between: it sharpens specific skills (math, code, long context) on curated data before alignment begins.

Why not combine them? Each phase uses fundamentally different data: pretraining learns from trillions of tokens with a simple objective (predict the next token), while post-training learns from a small set of human-labeled examples (what's helpful, what's harmful). Mixing them would drown the alignment examples in a sea of web text — the model would barely notice them. Training also needs to be tuned differently for each phase; if you optimized for alignment too aggressively during pretraining, the model would forget the language skills it just learned.

	Pretraining	Midtraining	SFT	Reasoning RL	Alignment RL
Loss / objective	Cross-entropy (next-token prediction)	Cross-entropy (same, curated data)	Cross-entropy (typically on response tokens only)	Policy gradient with verifiable reward (correct / incorrect)	Policy gradient with reward model (PPO) or supervised preference loss (DPO)
What provides the signal	The next token in the training text	The next token (higher-quality text)	Human-written gold responses	Ground-truth answers (math answers, code tests)	Human raters or AI judges ranking outputs
Algorithm	AdamW (most common; also Adafactor, Muon)	AdamW (lower learning rate)	AdamW (fine-tuning LR)	GRPO, PPO, or REINFORCE	PPO, DPO, or GRPO
Learning rate	0.0001 – 0.0003 aggressive — big steps	0.00001 – 0.00005 10× slower — careful refinement	0.00001 – 0.00002 gentle — don't break what pretraining built	0.000001 – 0.000005 100× slower — nudge, don't overwrite	0.000001 – 0.000005 same — tiny adjustments to behavior
Parameters trained	All weights initialized randomly	All weights continuing from pretrained checkpoint	All weights frontier labs use full fine-tuning; LoRA is used for API/customer fine-tuning only	All weights full model update (confirmed by DeepSeek-R1, LLaMA 3/4)	All weights full fine-tuning for flagship models
Data scale	1-40T+ tokens	100B-5T tokens	10K-1M+ examples	~50K-100K problems	~50K-1M+ comparisons
Cost (estimated)	$100M-$1B+	$10M-$50M	$10M-$100M+ for frontier models (large human-annotation + synthetic-data pipelines)
What it changes	Everything — builds the model from scratch	Sharpens skills, extends context	Activates the Q&A format (capability already exists in base model)	Teaches "think before answering"	Steers toward helpful, harmless, honest

Not shown: Distillation — training a smaller model to mimic a larger one (e.g., DeepSeek-R1-Distill, Llama 3.2 1B/3B, Gemma 2 9B, Gemini 3.5 Flash). Can happen during pretraining or post-training. Now standard for deploying efficient versions of frontier models.

The Alignment Pipeline

The InstructGPT / ChatGPT three-step process that launched the alignment era.

The three-step alignment pipeline (InstructGPT / ChatGPT)

STEP 1

SFT

Supervised Fine-Tuning

Humans write ideal responses to prompts. Fine-tune on ~13,000 (prompt, response) pairs.

"Explain quantum computing" → [expert-written answer]

↓

STEP 2

Reward Model

Humans rank multiple outputs best → worst (~33K ranked prompts, 4-9 outputs each). A separate model learns to predict human preferences.

Response A > Response B > Response C → RM learns: score(A) > score(B) > score(C)

↓

STEP 3

PPO (Reinforcement Learning)

The SFT model generates responses; the reward model scores them; PPO updates the model to produce higher-scoring responses.

Generate → Score with RM → Update policy → repeat ×1000s

Key result: On instruction-following prompts, InstructGPT (1.3B params) was preferred by humans over GPT-3 (175B params). Alignment was more valuable than 100x more parameters.

Modern Post-Training Approaches

The InstructGPT pipeline was the 2022 breakthrough. Since then, several approaches have largely supplanted PPO for post-training.

DPO (Direct Preference Optimization)

Eliminates the reward model entirely. Reparameterizes the RLHF objective so the preference loss is computed directly from the policy model. Uses simple binary cross-entropy on preferred vs. dispreferred responses.

Used by: LLaMA 3/4 (iterative DPO over 6 rounds), Zephyr, many open-source models.

GRPO (Group Relative Policy Optimization)

Generates multiple responses per prompt, scores them with verifiable rewards (math/code correctness), and updates the model using group-relative advantages. Eliminates the critic model, halving memory vs PPO.

Used by: DeepSeek-R1, Qwen3, Llama-Nemotron. The dominant algorithm for reasoning RL.

Constitutional AI (Claude)

Two phases: (1) SFT with AI self-critique guided by a written "constitution" of principles. (2) RLAIF — the AI itself generates preference rankings, replacing human labelers.

Used by: Anthropic (Claude). Scales better than human feedback. The constitution makes values explicit and auditable.

Full Fine-Tuning vs PEFT

Full fine-tuning updates every weight. PEFT methods update only a tiny fraction. The visual difference is dramatic.

Interactive: What gets updated?

Each cell represents a block of model parameters. Green = updated during training. Blue/dim = frozen.

Aspect	Full Fine-Tuning	PEFT (LoRA, etc.)
Params updated	All (100%)	0.1-2% of total
GPU memory	Very high (full model + optimizer states)	Low (base frozen, only adapters in optimizer)
Training speed	Slower (more gradients)	Faster (fewer gradients)
Task adaptation	Maximum	Very good (surprisingly close)
Catastrophic forgetting	Higher risk	Lower risk (base weights preserved)
Multi-task serving	One model per task	Swap adapters (same base model)
Storage per variant	Full model copy (~7B = ~14GB in FP16)	~10-50 MB adapter file

LoRA Explained Visually

Low-Rank Adaptation freezes the original weight matrix and learns a small bypass: two tiny matrices that approximate a weight update.

The LoRA bypass: W + BA

Original Weight W

4096 x 4096 = 16.8M params

FROZEN

LoRA Bypass

4096 x r

TRAINABLE

r x 4096

TRAINABLE

With r=16: only 131K params (0.8%!)

h = W*x + (B*A)*x where rank r << d

Why "low rank" works

Research shows that the weight updates during fine-tuning have low intrinsic dimensionality. The change in weights can be well-approximated by a rank-16 or rank-64 matrix, even though the weight matrix itself is 4096x4096. LoRA exploits this by only learning the low-rank factors.

Interactive: Rank trade-off

Rank (r): 16

Which layers to target?

LoRA is typically applied to the attention projection matrices. The most common configuration targets q_proj and v_proj, though targeting all projection layers (q_proj, k_proj, v_proj, o_proj) can yield better results.

q_proj (always)

v_proj (always)

k_proj (optional)

o_proj (optional)

gate_proj / up_proj (FFN, less common)

QLoRA

QLoRA combines quantized base weights with LoRA adapters, making it possible to fine-tune 65B+ models on a single GPU.

LoRA

Base model: FP16 (16-bit)

7B params x 2 bytes = ~14 GB

14 GB

LoRA adapters: FP16

~0.1% = ~14 MB

14 MB

Total GPU memory: ~16-24 GB (frozen base + adapter optimizer + activations)

QLoRA

Base model: NF4 (4-bit)

7B params x 0.5 bytes = ~3.5 GB

3.5 GB

LoRA adapters: FP16

~0.1% = ~14 MB (still full precision!)

14 MB

Total GPU memory: ~6-10 GB (fits on consumer GPUs!)

Key insight: QLoRA quantizes the frozen base model to 4-bit but keeps the LoRA adapters in full precision (FP16). During training, the 4-bit weights are temporarily dequantized for computation. The gradients only flow through the small LoRA matrices. This is why QLoRA can fine-tune a 65B model on a single 48GB GPU (Dettmers et al., 2023).

Model Distillation

Train a smaller student to imitate a larger teacher. The student learns from the teacher's soft probabilities, not just hard labels.

Teacher-Student Pipeline

Teacher

GPT-4 / Claude / 70B model

Large, slow, expensive

Soft probabilities

              "cat": 0.45

              "kitten": 0.30

              "feline": 0.15

              "dog": 0.05

              "pet": 0.03

              ...

Rich signal: "kitten" is similar to "cat"

Knowledge

Transfer

Student

Llama-3.1-8B / 1B model

Small, fast, cheap

Learns to match teacher's distribution

              "cat": 0.42

              "kitten": 0.28

              "feline": 0.17

              ...

Captures the teacher's knowledge structure

Why soft labels are better than hard labels

Hard label: "cat" = 1, everything else = 0.
Loses all information about word relationships.

Soft label: "cat" = 0.45, "kitten" = 0.30, "feline" = 0.15...
Preserves semantic similarity -- "kitten" is close to "cat."

Distillation vs PEFT: Distillation creates a new, smaller model entirely. PEFT adapts an existing model more cheaply. Distillation is for deployment efficiency; PEFT is for task adaptation.

When Fine-Tuning Is Worth It

Fine-tuning is not automatically the next step after a weak prompt. Show restraint.

Decision Framework

Fine-tune when:

1 Prompting and retrieval have plateaued -- you've tried hard and measured

2 The task is stable -- the desired behavior won't change next month

3 You have labeled data that represents what you want

4 The behavior must be repeated at scale -- consistent format, style, or knowledge

5 The business case rewards tighter consistency or lower cost per request

Don't fine-tune when:

1 The task changes rapidly -- monthly updates invalidate the fine-tuned model

2 The dataset is weak -- noisy, small, or inconsistent labels

3 The problem is knowledge retrieval -- RAG is the right tool, not fine-tuning

4 Better prompting or tool use can solve it (cheaper and more flexible)

5 You haven't measured the baseline -- you don't know what you're improving over

Key insight: Fine-tuning is not automatically the next step after a weak prompt. Sometimes the real issue is poor data, bad chunking, or missing tools.

Dataset Quality Checklist

A small, clean dataset is often more valuable than a large noisy one. The model will faithfully learn your inconsistencies too.

Interactive Checklist: Is your dataset ready?

Click each item to check it off. A good dataset should pass all of these.

Clear and unambiguous labels

Each example has one obvious correct label. No "it depends" cases without clear resolution rules.

Representative of production distribution

The data covers the same topics, styles, and edge cases the model will see in production -- not just "easy" examples.

Diverse across edge cases

Includes rare but important cases: short inputs, long inputs, multilingual, adversarial, out-of-scope queries.

Consistent formatting

All examples follow the same format template. The model learns format from examples -- inconsistency teaches inconsistency.

No data leakage from eval set

Training and evaluation sets are truly separate. Near-duplicates and paraphrases of eval examples are removed from training.

Aligned to production behavior

The outputs in training match what you actually want the model to say in production -- not just technically correct but also in the right style, length, and tone.

Sufficient volume for the task complexity

Simple format tasks: ~100-500 examples. Complex reasoning: 1000+. More data helps but returns diminish after the model has seen the pattern.

Readiness:

0/7

The golden rule: Data quality defines the ceiling. Fine-tuning amplifies the pattern in the data; it does not invent a better target than the one you provide.

Catastrophic Forgetting

When fine-tuning pushes the model so hard toward a new task that it loses its general capabilities.

Visual: Capabilities over fine-tuning time

--- Target task performance --- General reasoning --- Code generation --- Safety behavior

Training epochs: 3

Symptoms

Model becomes excellent at the fine-tuned task

But loses ability to do things it could before

General reasoning degrades

Safety guardrails become unreliable

Output style becomes narrow and repetitive

Mitigations

PEFT Use LoRA instead of full fine-tuning -- base weights stay frozen

Data mix Include general-purpose data alongside task data (e.g., 20% general)

Early stop Monitor both task AND general metrics; stop when general degrades

Low LR Use a very small learning rate to limit weight changes

Evaluation Before Release

Evaluate both target-task gains and unintended regressions. Compare against the baseline and the cheapest non-fine-tuned alternative.

Evaluation Matrix

Task Performance

Must test

Accuracy on target task

Format compliance rate

Latency (did fine-tuning change inference speed?)

Edge case handling

Regression Checks

Must test

General reasoning benchmark (MMLU, etc.)

Safety evaluations (refusal of harmful prompts)

Code generation (if relevant)

Math / factual accuracy

Comparison Baselines

Must compare

Base model (before fine-tuning)

Best prompt-only approach

RAG-augmented baseline

Cost per correct response

Fine-tuned
Model

Task Eval
accuracy, format

Regression
Suite
safety, general

A/B Test
vs baseline

Ship or
Iterate

Key principle: A mature evaluation compares the fine-tuned model against the baseline model and the cheapest non-fine-tuned alternative. Otherwise you cannot tell whether fine-tuning truly earned its complexity.

Cost Tradeoffs & When to Avoid Fine-Tuning

Fine-tuning has both upfront and lifecycle costs. Teams sometimes focus on training cost and forget the long-term burden.

Interactive: Total Cost of Fine-Tuning

Dataset size: 5,000

Model size (B params): 7B

Iterations/year: 4

The full cost picture

Upfront Costs

Data creation/curation

Training compute (GPU hours)

Evaluation infrastructure

Engineering time

Per-Iteration Costs

Data refresh and QA

Re-training runs

Regression testing

Deployment and validation

Ongoing Costs

Model versioning and storage

A/B testing infrastructure

Monitoring for drift

Team knowledge maintenance

The discipline of not fine-tuning

A team should avoid fine-tuning when the task changes rapidly, the dataset is weak, the behavior is mostly knowledge retrieval rather than skill adaptation, or the product can be solved with better prompting, retrieval, or tool use. Fine-tuning can add complexity without solving the actual bottleneck. Good engineers do not optimize the wrong layer of the stack.