Text Generation, Decoding & Serving

Temperature, Top-k, and Top-p Sampling

Three knobs that control how "creative" or "focused" the model's output is. They all modify the probability distribution before sampling the next token.

Interactive: Adjust the parameters

Watch how the token probability distribution changes as you move the sliders.

Temperature: 1.00

Top-k: 20 (off)

Top-p (nucleus): 1.00 (off)

Temperature

p_i = exp(z_i / T) / sum(exp(z_j / T))

T < 1: sharpens (more confident, less diverse). T > 1: flattens (more random, more creative). T -> 0: becomes greedy (always picks highest).

Top-k

Keep only the k most probable tokens, zero out the rest, then renormalize. Simple cutoff. k=1 is greedy decoding. k=40 is a common default.

Top-p (Nucleus)

Keep the smallest set of tokens whose cumulative probability exceeds p. Adapts to the shape of the distribution — includes more tokens when the model is uncertain, fewer when confident.

Greedy vs Beam Search vs Sampling

Three fundamentally different approaches to choosing which tokens to generate. Each makes a different tradeoff between quality and diversity.

Greedy Decoding

Always pick the highest-probability token. Fast, deterministic, but can get stuck in repetitive or suboptimal sequences.

Best for: factual answers, code generation, deterministic outputs.

Speed: fastest

Beam Search

Maintain k "beams" (partial sequences) simultaneously. At each step, expand all beams and keep the top-k total scores. Finds higher-probability sequences.

Best for: translation, summarization, when you need the "best" output.

Speed: k times slower

Sampling

Randomly sample from the probability distribution (optionally modified by temperature, top-k, top-p). Produces diverse, natural-sounding text.

Best for: creative writing, chat, brainstorming, when diversity matters.

Speed: fast, non-deterministic

Strategy	Deterministic?	Diverse?	Finds global best?	Typical use
Greedy	Yes	No	No (locally optimal)	Classification, extraction
Beam (k=4)	Yes	Somewhat	Approximately	Translation, summarization
Sampling (T=0.7)	No	Yes	No (stochastic)	Chat, creative writing

Streaming Generation

Why waiting for the full response is bad UX — and how streaming fixes it by sending tokens as they're generated.

Visual timeline: Streaming vs Non-Streaming

Without streaming

User waits for the entire response to be generated. Long silence, then everything appears at once. Feels slow even if total time is the same.

With streaming (SSE / WebSocket)

Tokens arrive as they're generated. User sees the response building in real time. Feels fast and interactive. Users can cancel early if the direction is wrong.

Live streaming demo

Click "Stream" to see tokens appear one by one, simulating real streaming generation.

Systems implications of streaming:

Error handling changes — Errors mid-stream mean partial output is already visible. Need graceful degradation.

Cancellation — User can stop generation early, saving compute. Requires proper connection lifecycle management.

Batching & Concurrency

How serving systems pack multiple requests onto the GPU simultaneously to maximize throughput — and the tradeoffs involved.

Visual: Requests packing into GPU compute

Without batching

One request at a time. GPU is mostly idle between requests. Low utilization, high per-request latency for queued requests.

With continuous batching

Multiple requests processed simultaneously. New requests join the batch as old ones finish. GPU stays busy. Higher throughput.

Key serving concepts

Static batching Wait until batch is full, process all at once. Simple but wastes time waiting and short sequences pad to max length.

Continuous batching Requests join and leave the batch dynamically. A finishing request immediately frees its slot for a new one. Much better GPU utilization.

Iteration-level scheduling At each decoding step, decide which requests to process. Can preempt low-priority requests or pause long generations.

The tradeoff Larger batches = higher throughput but higher per-request latency. The optimal batch size depends on the SLA.

The KV Cache

The single most important optimization for autoregressive generation. Without it, every new token would recompute attention over the entire sequence from scratch.

Cached vs recomputed: what happens at each step

Without KV cache (naive)

Step 1: process [The] -> compute K,V for "The"
Step 2: process [The, cat] -> recompute K,V for "The" AND "cat"
Step 3: process [The, cat, sat] -> recompute ALL three
Step 4: process [The, cat, sat, on] -> recompute ALL four

Cost per step grows linearly. Total cost is O(n^2).

With KV cache

Step 1: process [The] -> store K,V for "The"
Step 2: process [cat] only -> reuse stored K,V, add new K,V
Step 3: process [sat] only -> reuse all stored K,V, add new
Step 4: process [on] only -> reuse all stored K,V, add new

Each step only computes the new token's K,V projection (not all previous tokens'). Attention is still O(n) per step, but avoiding redundant K,V recomputation is a massive speedup.

Visual: KV cache growth

Each column is a decoding step. Green blocks are cached (reused). Orange blocks are newly computed. Without caching, every block would be orange.

KV cache memory budget

Model	Layers	Heads	d_head	KV cache per token	128K ctx
LLaMA 3 8B	32	8 (GQA)	128	~128 KB	~16 GB
LLaMA 3 70B	80	8 (GQA)	128	~320 KB	~40 GB
GPT-4 class (estimated)	~120?	~16?	128?	~1 MB?	~120 GB?

The KV cache can consume more GPU memory than the model weights themselves for long sequences. This is why techniques like GQA (Grouped Query Attention), paged attention (vLLM), and KV cache compression exist.

Quantization: FP16, INT8, INT4

Reducing the numerical precision of weights to save memory and speed up inference — the key technique for running large models on consumer hardware.

Precision comparison

FP32

32 bits per value. Full precision. Baseline quality.

70B model: ~280 GB

FP16 / BF16

16 bits per value. Negligible quality loss. Standard for training and inference.

70B model: ~140 GB

INT8

8 bits per value. Small quality loss on most tasks. Great for inference.

70B model: ~70 GB

INT4

4 bits per value. Noticeable quality loss on hard tasks. Enables consumer GPUs.

70B model: ~35 GB

Precision vs Model Size vs Quality

Post-training quantization (PTQ) Quantize after training. Simple, no retraining needed. May degrade quality on sensitive tasks. GPTQ, AWQ are popular methods.

Quantization-aware training (QAT) Simulate quantization during the forward pass so the model learns to be robust to reduced precision. Better quality but requires full training compute.

QLoRA (different from QAT) Quantizes the frozen base model to 4-bit (NF4) but trains LoRA adapters in full precision (FP16). The base weights are never updated through quantization — it's a memory-saving technique for fine-tuning, not QAT.

Throughput vs Latency Tradeoff

The fundamental tension in serving: serving many users efficiently (throughput) vs serving each user quickly (latency). Optimizing one often hurts the other.

Interactive: Batch size vs Metrics

Drag the batch size slider to see how throughput and latency respond.

Batch size: 8

Throughput

140 tok/s

P50 Latency

51 ms/tok

Which to optimize?

Workload	Priority	Why
Interactive chat	Latency (TTFT + TPS)	Users notice delay. First-token latency and streaming speed matter most.
Batch processing	Throughput	Processing 10M documents — total time matters, individual speed doesn't.
Code completion	Latency (P99)	Must feel instant. High P99 latency kills the feature — users turn it off.
Copilot agents	Both	Need fast responses AND efficient multi-turn. Balance with tiered routing.

Long-Context Serving Challenges

Models now support 128K-1M token contexts, but using all of it on every request is expensive and sometimes counterproductive.

Challenges

Memory: KV cache explosion At 128K tokens, the KV cache alone can be 4-30 GB per request. Limits concurrency.

Compute: O(n^2) attention Standard attention scales quadratically with sequence length. 128K tokens = 16 billion attention interactions per layer.

Quality: context dilution Models struggle with "lost in the middle" — information in the middle of very long contexts is less likely to be used than information at the start or end.

Cost: $$$ per request A 100K-token prompt is 50x more expensive than a 2K-token prompt. Retrieval can often provide the same information at 1/50th the cost.

Solutions

Retrieval instead of cramming Use RAG to find the relevant 2K tokens from 100K documents instead of sending everything.

Sparse attention patterns Only attend to nearby tokens + selected distant tokens. Reduces O(n^2) to O(n*sqrt(n)) or O(n*log(n)).

KV cache compression Evict old or low-attention KV pairs. Quantize cached values. Multi-query / grouped-query attention (GQA).

Context design Place critical info at the start and end. Summarize long sections. Use structured formats that models handle well.

Safety & Moderation Pipeline

Safety is not a single classifier — it's a pipeline of controls applied before, during, and after generation.

The full safety pipeline

User input
prompt + context

Input filter
block harmful prompts

Tool gating
permission checks

Model
RLHF-aligned

Output filter
content moderation

Response
or refusal

Pre-generation

Input classifiers detect jailbreak attempts, PII, or policy-violating prompts. System prompts enforce behavior boundaries. Tool permissions limit what the model can do.

During generation

Constrained decoding can prevent certain token sequences. Watermarking embeds invisible signals. RLHF alignment means the model itself has learned to refuse unsafe requests.

Post-generation

Output classifiers check for harmful content. PII redaction removes leaked data. Citation verification checks factual claims. Human review for high-stakes decisions.

Key insight: No single mechanism catches everything. Layered defense ("defense in depth") is essential. The model's alignment (RLHF/RLAIF) is the foundation, but external guardrails add robustness.

Scalable LLM Service Architecture

What it takes to turn "I have a model" into "I have a production service that handles 10,000 requests per second."

Full system diagram

Web client

Mobile app

API consumer

API Gateway / Load Balancer
Auth, rate limiting, routing

Orchestration Layer
Prompt assembly, tool dispatch, retrieval

Vector DB
RAG retrieval

Tool APIs
search, compute

Cache
prompt + response

Model Serving Cluster (autoscaled)

GPU Node 1
Model shard

GPU Node 2
Model shard

GPU Node N
Model shard

Logging
traces, metrics

Evaluation
quality monitoring

Guardrails
safety, PII

Production components most teams underestimate

Evaluation pipeline Continuous monitoring of output quality. Regression detection. A/B testing between model versions.

Tiered model routing Route simple queries to cheaper/faster models. Reserve expensive models for hard tasks. Saves 60-80% compute.

Rollback strategy When a new model version degrades quality, you need instant rollback. Model versioning + prompt versioning = complex matrix.