Table of Contents
Part I — Architecture
Tokens, Tokenization & Context Windows
BPE, WordPiece, SentencePiece, byte-level encoding, special tokens, context budgeting, and why output tokens cost more.
Embeddings & Semantic Representations
Token/sentence/document embeddings, cosine vs dot product vs euclidean, L2 normalization, bi-encoder vs cross-encoder, CLIP.
Transformer Architecture & Attention
Self-attention (Q/K/V), multi-head attention, positional encoding (sinusoidal/RoPE/ALiBi), KV cache, causal masking, Flash Attention.
Part II — Training
Model Families & Training Objectives
From BERT to GPT to LLaMA — MLM, seq2seq, generative vs discriminative, foundation vs task-specific, model specs and architecture comparison.
Pretraining & Scaling Laws
Next-token prediction deep dive, interactive loss function demo, teacher forcing, scaling laws (Kaplan vs Chinchilla), compute-optimal training.
Post-Training: Alignment, Tuning & Reasoning
How SFT, RLHF, DPO, and GRPO turn a base model into a useful assistant — plus LoRA, distillation, and when to fine-tune.
Math Foundations & Number Formats
Softmax, cross-entropy, backpropagation, activation functions (ReLU/GELU/SwiGLU), FP32/FP16/BF16/INT8/INT4, mixed precision training.
Part III — Applications
Classification with LLMs
Zero-shot/few-shot classification, prompt vs fine-tune decision framework, metrics (precision/recall/F1), human-in-the-loop pipelines.
Topic Modeling & Theme Discovery
Embedding-based clustering, UMAP/t-SNE/PCA, K-means vs DBSCAN vs hierarchical, LLM-powered cluster naming, topic drift.
Prompting, In-Context Learning & Tool Use
System/user/assistant roles, chain-of-thought, structured output, function calling, prompt injection attacks, versioning.
Multimodal Large Language Models
Vision encoders (VLM), CLIP, visual grounding, OCR vs native vision, audio/video pipelines, architecture patterns (LLaVA, GPT-4V).
Part IV — Retrieval & RAG
Retrieval Foundations
Lexical vs dense retrieval, hybrid search (RRF), chunking strategies, vector databases, HNSW/IVF, reranking, HyDE, offline metrics.
Production RAG Architectures
Naive vs production RAG, multi-hop retrieval, hallucination reduction, citations, agentic RAG, caching, permissions/ACL.
Custom Embeddings & Retrieval Optimization
Domain adaptation, hard negatives, training losses (contrastive/triplet/InfoNCE), product quantization, drift monitoring.
Part V — Inference & Hardware
Text Generation, Decoding & Serving
Temperature/top-k/top-p, beam search, streaming, batching, KV cache, quantization (FP16/INT8/INT4), scalable LLM architecture.
GPU Inference: How LLMs Run on Hardware
GPU architecture, VRAM allocation, token generation on hardware, memory bandwidth bottleneck, batching efficiency, tensor parallelism.
Part VI — Frontiers
Practical: nanochat
Karpathy's $100 ChatGPT — full pipeline from BPE tokenizer to RL, connecting every concept from chapters 1-16 in ~8,000 lines of code.
Advanced Topics & Extensions
Mixture of Experts, knowledge graphs, adaptive softmax, model ecosystems, bias/fairness, interpretability, privacy, deployment optimization.