Architectures, Extensions & Deployment

Mixture of Experts (MoE) Deep Dive

A sparse architecture that scales parameter count without proportionally scaling compute. The key idea: not every parameter needs to fire for every token.

How MoE routing works

Token
hidden state

Router
learned gate network

Produces a score for
each expert (softmax)

Top-k selection (k=2 typical)

MoE architecture overview

Input Embeddings

Repeated N times

Multi-Head Self-Attention

Add & Layer Norm

MoE Layer (replaces dense FFN)

Router

8 experts total, 2 active per token (highlighted). Each expert is a standard FFN.

Add & Layer Norm

Output Head

Advantages

- Scale to trillions of parameters with constant compute per token

- Different experts can specialize in different skills/languages

- Train faster per step (less compute per forward pass)

Failure modes

- Router collapse: most tokens go to the same 1-2 experts

- Load imbalance: some experts overloaded, others idle

- Harder to debug: quality issues may be routing issues

- Memory: all experts must be in memory even if only 2 are active

Model	Total params	Active params	Experts	Active per token
Mixtral 8x7B	46.7B	~12.9B	8	2
Mixtral 8x22B	141B	~39B	8	2
Switch Transformer	1.6T	~7B	2048	1
GPT-4 (unverified leak)	~1.8T?	~220B?	16?	2?

Knowledge Graphs + LLMs

When structured knowledge complements unstructured text retrieval — and when it doesn't.

Interactive: Knowledge Graph Structure

Nodes are entities. Edges are typed relationships. Click "New Layout" to rearrange.

When to use what

Need	Best tool
"Find docs about X"	Vector retrieval (RAG)
"What owns what?"	Knowledge graph
"Who reports to whom?"	Knowledge graph
"Similar products to X"	Vector retrieval
"All entities related to X within 3 hops"	Knowledge graph
"Semantic search over docs"	Vector retrieval
"Complex multi-hop reasoning"	Both (graph + RAG)

Best practice:

Many strong production systems use both. Vector retrieval for broad evidence discovery, knowledge graphs for entity-grounded logic and traceability. The choice is empirical, not ideological.

KG strengths Explicit facts, typed relations, multi-hop traversal, entity disambiguation, regulatory traceability.

KG weaknesses Expensive to build and maintain. Incomplete by nature. Requires schema design. Doesn't handle fuzzy/ambiguous queries well.

Adaptive Softmax

How to make the output layer efficient when your vocabulary has 128,000+ tokens — most of which are rare.

The problem: standard softmax is expensive

The output layer maps hidden_dim -> vocab_size. For a model with 128K vocabulary and 4096-dim hidden state, that's a 4096 x 128,256 matrix multiplication at every decoding step for every token. Most of that compute goes to rare tokens that almost never get selected. Note: modern LLMs (GPT-4, LLaMA 3, Claude) compute the full softmax rather than using adaptive softmax — the technique below is historically important and still relevant for resource-constrained settings.

Token frequency follows a power law. The top 1% of tokens account for ~80% of usage.

The solution: hierarchical clusters

Adaptive softmax groups the vocabulary into clusters by frequency. Frequent tokens get the full-dimensional projection. Rare tokens use a cheaper, lower-dimensional projection.

Cluster 1: Top 2K tokens Full dim (4096)

"the", "a", "is", "and" ... Computed every step. These cover ~80% of all outputs.

Cluster 2: Next 10K tokens Reduced dim (1024)

Less common words. Only compute full softmax over these if Cluster 1 doesn't contain the answer.

Cluster 3: Remaining 116K tokens Minimal dim (256)

Rare tokens, special characters. Rarely needed. Cheapest to compute when invoked.

Claude vs GPT Ecosystem Comparison

Both are frontier systems. The differences are in packaging, defaults, and ecosystem ergonomics — not "which is better."

Dimension	Claude (Anthropic)	GPT (OpenAI)	What matters in practice
Context window	up to 1M tokens (Opus 4.x / Sonnet 4.6)	up to 400K (GPT-5 Pro)	Long context is a capability, not a default mode. Retrieval often beats cramming.
Tool use	Function calling, MCP	Function calling, MCP, GPT Actions	MCP is now a cross-vendor standard (Anthropic, OpenAI, Google, Microsoft).
Reasoning	Extended thinking (visible chain)	GPT-5 series reasoning (5.2 / 5.5)	Different approaches to "slow thinking." Evaluate on your specific task.
Safety approach	Constitutional AI, RLHF	RLHF, system prompt controls	Both invest heavily. Specific refusal behavior differs by model version.
API ergonomics	Messages API, streaming	Chat Completions / Responses API	Very similar patterns. Switching cost is low.
Ecosystem	Claude Code, Artifacts, MCP	ChatGPT, GPTs, Actions, Codex	OpenAI has broader consumer reach. Anthropic focuses on developer/enterprise.

Practical note:

Stay current-aware and avoid hard-coding assumptions that change by release. The right comparison is always empirical: benchmark the candidate models on your workload, your prompts, and your latency budget instead of relying on brand-level folklore. Strong candidates separate research novelty from deployability.

Hyperparameters Beyond Learning Rate

Every knob the engineer sets (rather than the model learns) is a hyperparameter. Good teams treat them as system design, not afterthoughts.

Training hyperparameters

Batch size Affects gradient noise, memory, and training speed. Larger batches = less noisy gradients but need more memory and may generalize worse.

Weight decay Regularization that penalizes large weights. Prevents overfitting. AdamW decouples it from the optimizer update.

Warmup steps Start with a tiny learning rate, ramp up. Prevents early instability before the optimizer's adaptive state has calibrated.

Sequence length Longer sequences = quadratically more attention compute + more KV cache memory. But training on short sequences limits what the model learns about long-range dependencies.

Serving & RAG hyperparameters

Temperature / top-p Control output diversity. Task-dependent. Factual Q&A wants low temperature; brainstorming wants high.

Chunk size (RAG) How big each retrieved passage is. Too small = missing context. Too large = noise dilutes signal. 256-1024 tokens is typical.

Reranker depth (RAG) How many candidates to retrieve before reranking. Retrieve 100, rerank to top 5. Higher depth = better recall but more compute.

Max tokens / timeout Limits on generation length and time. Critical for cost control and user experience. Often forgotten until the bill arrives.

Interactive: Learning rate schedule

Adjust warmup fraction and decay type to see how the learning rate changes over training.

Warmup %: 5%

Bias & Fairness in LLM Outputs

Making bias concrete and measurable — not a slogan, but an engineering process.

Where bias enters

Training data

Internet text reflects historical biases. Underrepresentation of minority perspectives. Toxic content patterns.

Model architecture

Attention patterns may amplify high-frequency patterns. Rare groups get less representation in learned features.

RLHF / fine-tuning

Annotator demographics and guidelines shape what the model considers "good." Can both fix and introduce biases.

Deployment context

System prompts, retrieval corpus, user interface framing all shape how biases manifest in practice.

Engineering responses

Disaggregated evaluation Don't just measure average accuracy. Break down by demographic group, topic, language. Surface where the model fails unevenly.

Representative edge cases Curate test sets that cover minority perspectives, ambiguous scenarios, and culturally sensitive topics. Test what matters, not just what's easy.

Multi-layer mitigation Data curation + RLHF alignment + retrieval constraints + output validation + human escalation. No single fix resolves all bias.

Escalation policies Define when the model should refuse, disclaim uncertainty, or route to a human. "I don't know" is better than a confidently biased answer.

Interpretability Challenges

Understanding why a model produced a specific output is one of the hardest open problems in AI. Here's what works, what doesn't, and why attention maps can mislead.

Attention visualization: useful but misleading

Attention weights show what tokens the model "looked at," but NOT what it learned from them. High attention does not mean high influence on the output. The value vectors determine what information actually flows through.

Common interpretability pitfalls

Attention != explanation Attention weights are intermediate computations, not reasons. The model may attend to a token for syntactic reasons while the semantic decision happens elsewhere.

Multiple heads, multiple stories A 32-head model has 32 different attention patterns per layer. Which one do you visualize? They often contradict each other. Aggregation hides this.

Post-hoc vs faithful Most explanation methods (LIME, SHAP, attention) are post-hoc: they create a plausible story, but the model didn't actually reason that way. Faithful interpretability (mechanistic) is much harder.

What actually works Probing classifiers, activation patching, sparse autoencoders on internal representations. These reveal what information is encoded, not just what tokens were attended to.

Privacy in LLM Deployment

Privacy is not a feature to add later — it's an architectural constraint that shapes the entire system from day one.

Privacy threat model for LLM systems

User sends
sensitive prompt

Retrieval fetches
confidential docs

Model processes
combined context

Response may
leak PII/secrets

Logs capture
everything

Attack surfaces

Training data extraction Models can memorize and regurgitate training data, including PII, code, and private communications.

Prompt leakage Retrieved documents in the context may contain sensitive information that the model exposes in its response.

Log exposure Observability systems (logs, traces, metrics) capture prompts and responses. Who has access to these logs?

Controls

Access control on retrieval Only retrieve documents the user is authorized to see. Enforce at query time, not just at indexing time.

PII redaction Detect and remove PII from inputs, retrieved context, and outputs. Multi-layer: before model, and after model.

Data minimization Don't log full prompts/responses unless necessary. Retention limits. Separate PII from telemetry.

Common Deployment Bottlenecks

The things teams consistently underestimate. Click to mark items you've addressed in your system.

Operational

Evaluation maintenance
Eval sets go stale. New edge cases emerge. Maintaining quality benchmarks is ongoing work, not a one-time setup.

Prompt drift
Prompts are code. They need versioning, testing, and regression checks. A prompt that works on GPT-4 may fail on GPT-4o.

Model version drift
Model updates change behavior. You need automated regression detection and rollback capability.

Observability
Can you trace a bad output back to its retrieval results, prompt, model version, and user context? If not, debugging is guesswork.

Quality & Governance

Access control complexity
When retrieval, tools, and model all have different permission models, ensuring the response respects all of them is hard.

Long-tail latency
P50 looks great; P99 is terrible. Long inputs, complex tool calls, and cold caches create unpredictable spikes.

Cross-layer debugging
Failures span retrieval + permissions + model + tools. No single team owns the whole path. Distributed debugging across teams is the bottleneck.

Cost at scale
Compute cost is obvious. The hidden costs: evaluation compute, data labeling, prompt engineering time, incident response for AI-specific failures.

Human escalation paths
When the model fails, where does the user go? Graceful degradation and human handoff are features, not afterthoughts.

Deployment readiness:

0/9