Custom vs Off-the-Shelf Embeddings
The most common mistake: fine-tuning embeddings when the real problem is bad chunking, missing metadata, or poor prompts. Use this decision tree.
Decision Tree: Do you need custom embeddings?
(e.g., text-embedding-3-large, E5)?
Click each node to expand. Custom embeddings are valuable when domain-specific vocabulary or task mismatch causes retrieval failures, not when the real bottleneck is pipeline design.
When custom embeddings clearly win
Strong signals for custom
Fix these first (not embeddings)
Domain Adaptation Approaches
Three strategies for adapting embeddings to your domain, from lightest to heaviest.
Continued Pretraining
Keep training the base model's masked-language objective on your domain corpus. Teaches the model new vocabulary and domain patterns.
Contrastive Fine-Tuning
Train on (query, positive_doc) pairs with hard negatives. Directly optimizes the embedding space for your retrieval task.
Full Retraining
Train a new embedding model from scratch on your domain. Maximum control but extreme cost. Rarely justified unless you are building an embedding product.
baseline
Pretraining
domain vocab
Fine-Tuning
task-specific
on your data
Common pipeline: pretrain first for vocabulary, then fine-tune for task alignment.
Hard Negatives
The single most impactful ingredient in embedding fine-tuning. Hard negatives teach the model what "close but wrong" looks like.
Visual: Easy vs Hard Negatives
Easy negatives: Obviously wrong. The model already knows "cat care" is not related to "quantum physics." No learning signal.
Hard negatives: Close but wrong. "Cat breeds" looks similar to "cat health" but is not the right answer. Forces the model to learn subtle distinctions.
Concrete example
How to mine hard negatives
Run BM25 search for each query. Top results that aren't labeled positive are hard negatives -- keyword overlap but semantically wrong.
Other queries' positives in the same batch become negatives. Free and effective with large batch sizes.
Use a cross-encoder to score many candidates. Those scored medium-high but known-irrelevant are the hardest negatives.
Training Losses for Embeddings
The loss function determines what the embedding space optimizes for. Three dominant approaches.
Contrastive Loss (Pair-based)
Operates on pairs: (a, b) with a label y (1=similar, 0=dissimilar). Pulls similar pairs together, pushes dissimilar apart.
Pros: Simple, works on pairs.
Cons: Needs margin tuning. No relative ranking between negatives.
Triplet Loss
Triplets: (anchor, positive, negative). Requires d(a,p) + margin < d(a,n). More explicit separation constraint.
Pros: Intuitive, enforces relative ordering.
Cons: Triplet mining is expensive. Many triplets are uninformative.
InfoNCE (used by most SOTA models)
One positive vs many negatives. Softmax over similarities -- the positive should get the highest probability among all candidates.
Pros: Scales with batch size, no margin tuning. State of the art.
Cons: Needs large batches for enough negatives.
Key insight: InfoNCE dominates modern embedding training (CLIP, E5, GTE, Sentence Transformers). The key insight is that it naturally uses in-batch negatives, so larger batches = more negatives = better representations. Temperature t controls how sharply the model discriminates.
Long Document Strategies
Most embedding models have a 512-token limit. Documents are much longer. Three strategies to bridge the gap.
Fixed-Size Chunking
Split into fixed-size windows, embed each. Simple but can split thoughts mid-sentence.
Overlap: Add 50-100 token overlap between chunks to preserve context at boundaries.
Semantic Chunking
Split at natural boundaries: paragraphs, sections, headers. Each chunk is a coherent thought.
Better: Preserves meaning but chunk sizes vary widely.
Hierarchical Embeddings
Create embeddings at multiple levels: doc summary + section + chunk. Search coarse-to-fine.
Best: Most accurate but complex to build and maintain.
| Strategy | Complexity | Retrieval Quality | When to Use |
|---|---|---|---|
| Fixed Chunking | Low | Moderate | Prototyping, homogeneous docs |
| Semantic Chunking | Medium | Good | Structured docs with clear sections |
| Hierarchical | High | Best | Large corpora needing precision at scale |
Multilingual Embedding Considerations
Making embeddings work across languages requires specific architectural and training choices.
Cross-Lingual Alignment
A multilingual embedding model maps "cat" (English), "gato" (Spanish), and "chat" (French) to nearby vectors because they mean the same thing.
Encoder
Vector Space
Encoder
Vector!
Key Challenges
Production tip: Models like multilingual-e5-large, BGE-M3, and Cohere's embed-multilingual-v3 are strong baselines. Always test with real queries in your target languages -- MTEB scores are averages that can hide per-language weaknesses.
Index Compression
At scale, raw float32 vectors are too expensive to store and search. Compression makes large-scale retrieval practical.
Visual: How quantization compresses vectors
Product Quantization (PQ)
Splits the vector into sub-vectors, then replaces each sub-vector with the ID of its nearest centroid from a learned codebook. Lossy but dramatic compression.
32x compression!
Scalar Quantization (SQ)
Converts each float32 value to a smaller integer type (int8 or int4). Simpler than PQ, less compression but higher accuracy.
4x compression with minimal recall loss.
| Method | Compression | Recall Loss | Speed Impact | Best For |
|---|---|---|---|---|
| No compression (float32) | 1x | None | Baseline | Small datasets (<100K) |
| Scalar Quantization (int8) | 4x | Very Low | Faster (SIMD) | Default choice for most |
| Product Quantization | 10-32x | Moderate | Faster (smaller index) | Billions of vectors |
| Binary Quantization | 32x | Higher | Much faster (hamming) | First-pass filtering |
Similarity Threshold Tuning
The threshold at which you consider a retrieval result "relevant" directly trades precision for recall. There is no universal right answer.
Interactive: Precision/Recall Tradeoff
Drag the threshold slider. A higher threshold means fewer results but more precise; lower means more results but noisier.
Of the results returned, what fraction is actually relevant?
Of all relevant documents, how many were returned?
The dot on the curve tracks your current threshold setting.
Use for: safety-critical retrieval (medical, legal), RAG where false positives cause hallucination. Fewer results, but each one is highly likely correct.
Use for: exploratory search, creative brainstorming, when missing results is costlier than noise. More results, but includes marginal matches.
Key insight: There is no universal "good" threshold. It depends on the embedding model, the domain, and the cost of false positives vs false negatives. Always calibrate with labeled data from your actual task.
Monitoring Retrieval Drift
Embeddings work today but can degrade silently as your data, queries, or user behavior changes. Monitoring catches this before users notice.
What can go wrong over time
Embeddings work great
New topics in corpus
Query patterns shift
Silent quality degradation
What to Monitor
Drift Detection Signals
Embedding Migration Planning
Switching embedding models is a full data migration. Planning prevents costly downtime and silent quality regression.
Migration Pipeline
Model
on your data
Index
parallel to prod
dual-read
swap primary
Old Index
after bake
Why you can't just swap models
Migration checklist
Cost estimation: Re-embedding 10M documents at $0.00002/1K tokens (OpenAI text-embedding-3-small) with avg 200 tokens/doc = ~$40 in API costs plus compute time. The real cost is the engineering effort to run shadow indexing, A/B testing, and threshold recalibration safely.