GPU Architecture: The Hardware
A GPU is a massively parallel processor with thousands of small cores organized into Streaming Multiprocessors (SMs), each with dedicated Tensor Cores for matrix math, all connected to high-bandwidth memory (HBM).
Interactive GPU Die Map
Hover over SMs to highlight them. This represents a simplified GPU with 132 SMs (like an H100). Each SM contains Tensor Cores for matrix operations.
Inside One SM
(128)
(4)
CPU vs GPU
Memory Hierarchy: Speed vs Capacity
~33 MB total (132 SMs × 256 KB)
~33 MB total (132 SMs × 256 KB)
~50 MB • ~4 TB/s
80 GB • 3.35 TB/s (H100)
512+ GB • ~64 GB/s via PCIe Gen5 x16
Each level trades capacity for speed. LLM inference is dominated by HBM reads — HBM is off-chip DRAM stacked on the package, connected via a silicon interposer. It is the largest and slowest memory the GPU accesses.
Streaming Multiprocessor (SM)
The fundamental compute unit. Each SM has 128 CUDA cores for general math, 4 Tensor Cores for matrix multiply-accumulate, shared memory, and a register file. An H100 has 132 SMs.
Tensor Cores
Specialized hardware for matrix multiplication. H100's 4th-gen Tensor Cores process large matrix tiles (up to 64×256 via a single warp-group matrix instruction, executed asynchronously over multiple cycles), providing massive throughput for the linear algebra that dominates transformer inference (matmuls for Q/K/V projections, FFN layers).
HBM (High Bandwidth Memory)
Stacked DRAM chips bonded directly to the GPU die. Provides massive bandwidth (3.35 TB/s on H100) but is still the primary bottleneck for LLM inference. This is your "VRAM."
SRAM (On-chip)
Tiny but ultra-fast memory within each SM (256 KB combined shared memory + L1, up to 228 KB allocatable as shared memory). Used for intermediate computations. FlashAttention works by tiling attention computations to keep data in SRAM instead of reading/writing HBM repeatedly.
Where Things Live in Memory
GPU VRAM must hold the model weights, KV cache, activations, and I/O buffers simultaneously. The stacked bar below shows how VRAM is allocated during inference.
Interactive VRAM Allocation
Adjust model size, context length, and batch size to see how GPU memory is consumed.
Model Weights
The learned parameters. 70B params at FP16 = 140 GB. Largest consumer of VRAM. Loaded once, read every token generation step.
KV Cache
Stores computed key/value pairs for all previous tokens so they don't need recomputing. Grows with sequence length and batch size. Can consume more than weights for long contexts.
Activations
Temporary intermediate values during the forward pass. Reused per layer, so only one layer's activations live in memory at a time. Relatively small for inference.
I/O Buffers & Overhead
Input token embeddings, output logits, CUDA context, memory allocator overhead. Typically tiny (<1 GB) but always present.
One Token Generation: Step by Step
Click through each step to see exactly what happens on the GPU when generating a single output token during autoregressive decoding.
The Memory Bandwidth Bottleneck
LLM inference is almost always memory-bound, not compute-bound. The GPU's Tensor Cores sit idle waiting for data to arrive from HBM.
Why Inference is Memory-Bound
Arithmetic Intensity = FLOPs performed per byte loaded from memory. When this ratio is low, the GPU spends most of its time waiting for memory reads.
For a single request generating one token: each weight (2 bytes in FP16) is loaded from HBM and used for just 2 FLOPs (one multiply + one add), then discarded. That gives an arithmetic intensity of 2/2 = 1 FLOP/byte.
The H100 can do 989 dense FP16 TFLOPS but only load 3.35 TB/s from HBM. To saturate compute, you need ~295 FLOPs per byte loaded. At batch=1, you get ~1 FLOP/byte. The Tensor Cores are ~99.7% idle.
Interactive: Bytes vs FLOPs
Adjust batch size to see how arithmetic intensity changes.
Attention: Memory-Bound
For each token, attention reads the entire KV cache (past keys and values). With long contexts, this dominates runtime. FlashAttention helps by keeping data in SRAM.
FFN: Also Memory-Bound (batch=1)
The FFN (feed-forward network) has 2/3 of the model's parameters. At batch=1, each weight is loaded once and used once. At higher batches, FFN becomes compute-bound.
Batching: The Key to Efficiency
Batching multiple requests together amortizes HBM reads across more FLOPs, transforming the workload from memory-bound to compute-bound.
Same Weights, More Work
With batching, each weight loaded from HBM is multiplied against multiple input vectors instead of just one. This increases arithmetic intensity proportionally to the batch size.
Static Batching
Wait until batch is full, process all together, return all together. Simple but wastes GPU time — short sequences pad to the longest, and finished sequences wait for the slowest.
Wasted compute on padding
Continuous Batching
Iteration-level scheduling: as soon as one sequence finishes, a new one takes its slot immediately. No padding, no waiting. Used by vLLM, TGI, TensorRT-LLM.
2-3x higher throughput
The uncomfortable truth about GPU utilization
A $30,000+ H100 GPU sits ~89% idle even with 32 concurrent users. At batch=32, only 10.8% of Tensor Core capacity is used — the rest waits for data to arrive from HBM. You'd need batch ≈ 295 to fully saturate compute, which is impractical because each request's KV cache eats VRAM.
This is why the entire LLM serving industry exists: vLLM, TensorRT-LLM, continuous batching, PagedAttention, speculative decoding — all trying to squeeze more useful work out of hardware that's fundamentally bottlenecked by memory bandwidth, not compute.
NVIDIA recognized this: the A100→H100 jump tripled compute but only grew bandwidth 68%, making the memory wall worse. So the H200 was a bandwidth-only upgrade (same compute, +43% BW), and the B200 grew bandwidth proportionally more than compute. The memory pipe is the bottleneck, and recent generations focus on widening it.
Multi-GPU: Tensor & Pipeline Parallelism
When a model doesn't fit on one GPU, you split it across multiple GPUs. The two main strategies are tensor parallelism (split each layer's weights) and pipeline parallelism (put different layers on different GPUs).
Interactive: Splitting a 70B Model
Choose a parallelism strategy and number of GPUs to see how the model is distributed.
Tensor Parallelism (TP)
Split weight matrices across GPUs. Each GPU computes a portion of every layer, then GPUs synchronize via all-reduce after each layer. Requires fast interconnect (NVLink: 900 GB/s).
Lower latency High bandwidth needed
Pipeline Parallelism (PP)
Assign groups of consecutive layers to different GPUs. Data flows through them sequentially. Only activations (small) are communicated between GPUs. Can use slower interconnects.
Less communication Pipeline bubbles
GPU Comparison for LLM Inference
Key specifications that determine inference performance. Memory bandwidth matters most for latency; VRAM capacity determines maximum model + batch size.
| GPU | VRAM | Memory BW | FP16 Dense | FP16 Sparse | Max 70B Batch* |
|---|---|---|---|---|---|
| A100 40GB | 40 GB | 1.6 TB/s | 312 | 624 | ~4 |
| A100 80GB | 80 GB | 2.0 TB/s | 312 | 624 | ~12 |
| H100 SXM | 80 GB | 3.35 TB/s | 989 | 1,979 | ~12 |
| H200 | 141 GB | 4.8 TB/s | 989 | 1,979 | ~30 |
| B200 | 192 GB | 8.0 TB/s | 2,250 | 4,500 | ~50 |
* Conservative real-world max batch for a 70B INT4-quantized model (~35 GB weights) with 2K context and GQA, on a single GPU. These are deliberately below the theoretical maximum — they reserve headroom for FP16 KV cache, activations, memory fragmentation, and CUDA graphs, so real deployments don't OOM. The theoretical ceiling (FP16 KV, no overhead) is several times higher. FP16 weights (140 GB) require multi-GPU; multi-GPU setups scale further.
Why These Numbers Matter
~42 ms/token
needs multi-GPU (140 GB > 80 GB VRAM)
~7K tok/s max
~0.34% utilized