What is an embedding?
An embedding is a list of numbers (a vector) that represents a word, sentence, or document. The trick: similar meanings get similar numbers.
hundreds to thousands of numbers
Think of it like GPS coordinates for meaning. Just like latitude/longitude tell you where a place is on Earth, an embedding tells you where a piece of text sits in "meaning space."
Before embeddings
Computers compared text mainly by matching exact words. "doctor" and "physician" were much harder to connect — methods like thesauri and TF-IDF helped, but required manual effort or missed semantic nuance.
With embeddings
"doctor" and "physician" get similar vectors because they appear in similar contexts during training. The computer now "knows" they mean similar things.
Embedding Space (2D visualization)
Real embeddings have hundreds to thousands of dimensions (384-3072 is typical). Here's a simplified 2D view — similar words cluster together.
Hover over points to see labels. Click to add your own word.
Note: This shows proximity (Euclidean closeness), not cosine angle. In practice, most embedding models L2-normalize their vectors, which makes cosine similarity and Euclidean distance directly equivalent — close points = small angle from origin. This is a 2D simplification; real high-dimensional spaces have hundreds of dimensions that are not individually interpretable.
Measuring Similarity: Cosine vs Dot Product
Once you have vectors, how do you tell if two are "close"? Three common methods.
Cosine Similarity
Measures the angle between two vectors. Normalizes away magnitude — focuses on direction.
Range: -1 (opposite) to +1 (identical)
Best when: vector magnitudes vary, you want pure semantic direction.
Dot Product
Multiplies matching dimensions and sums them up. Bigger vectors get higher scores even if they point the same direction.
Range: -inf to +inf
Best when: vectors are already L2-normalized (then it equals cosine).
Euclidean Distance
Measures the straight-line distance between two points in vector space. Smaller = more similar.
Range: 0 (identical) to +inf
Best when: absolute position matters, not just direction.
Note: Some models intentionally encode information in vector magnitude (e.g., confidence or specificity). Cosine discards this. The right metric depends on how the embedding model was trained — always match the metric to the model's documentation.
Visual Intuition: Drag the vectors!
Drag the tips of vector A (red) and B (blue) to see how each metric changes. The three panels show what each metric actually measures geometrically.
How each metric behaves in 3 scenarios
These graphs show the key differences. Each scenario keeps one thing constant and varies another — watch how the metrics respond differently.
B points same direction as A. X-axis: B's length increases.
Both vectors length=1. X-axis: angle between them (0-180).
Both tips stay close but move far from origin.
Numeric Calculator: Compare two vectors
L2 Normalization
Why engineers shrink all vectors to the same length before comparing them.
The Problem Without Normalization
Imagine two documents about "cats." One has a vector with large magnitudes [5.0, 3.0, 4.0] and another with small magnitudes [0.5, 0.3, 0.4]. They point in the same direction (same meaning!), but dot product gives wildly different scores.
Doc A dominates just because its numbers are bigger, not because it's more relevant.
After L2 Normalization
Divide each vector by its length. Now every vector has length = 1. Only the direction (meaning) matters.
Now identical! Same direction = same meaning. Dot product and cosine give the same result.
Caveat: Normalization is a design choice, not magic. Whether it helps depends on how your embedding model was trained and how your vector index computes similarity. Some models intentionally encode useful information in magnitude — normalizing it away can hurt performance. Always check the model's recommended metric.
Token Embeddings: The First Step Inside Every LLM
Before a model can do anything — attention, reasoning, generation — it must convert each token into a vector. That's what the token embedding layer does.
What happens when you type "The cat sat"
table[3797] ->
table[3332] ->
[0.87, 0.21, ...]
[0.44, 0.73, ...]
The Embedding Table: A Giant Lookup Dictionary
The core of a token embedding layer is a table — one row per token in the vocabulary, one column per dimension. For GPT-2, that's 50,257 rows (tokens) x 768 columns (dimensions).
| Token | d0 | d1 | d2 | d3 | d4 |
| ID 0 "!" | 0.02 | -0.15 | 0.33 | 0.07 | -0.21 |
| ID 1 "\"" | 0.11 | 0.08 | -0.22 | 0.14 | 0.31 |
| ... | 50,257 rows total | ||||
| ID 464 "The" | 0.12 | -0.34 | 0.56 | 0.89 | -0.12 |
| ID 3797 "cat" | 0.87 | 0.21 | -0.43 | 0.15 | 0.68 |
| ... | |||||
Given token ID 464 ("The"), the model just looks up row 464 and returns that 768-dimensional vector. No computation — just a table lookup. These values are learned during training.
Key facts about token embeddings
Token Embeddings vs Contextual Embeddings: The Journey Through the Model
Token embeddings are just the starting point. As they pass through each transformer layer, they get transformed by attention and feed-forward networks. By the time they exit the final layer, they're completely different — they're now contextual representations that encode the meaning of that token in this specific sentence.
every time, no context
other tokens each layer
different from "bank" + "loan"
Example: "river bank" vs "bank account"
Why this matters for search
Sentence & Document Embeddings
Taking the contextual outputs from the final layer and compressing them into a single vector for the whole text.
Sentence Embeddings
One vector for the whole sentence/paragraph. Created by pooling the final-layer token outputs:
Use for: Semantic search, RAG, clustering, duplicate detection
Document Embeddings
One vector for an entire document. Two approaches:
Use for: Document retrieval, recommendations, topic clustering
Dense vs Sparse Representations
Two fundamentally different ways to represent text — and why modern systems use both.
Dense Embeddings
Every dimension has a value — 768 numbers, all meaningful
Strength: Captures semantic meaning. "doctor" and "physician" are close even with zero word overlap.
Weakness: Can retrieve conceptually related but wrong content. May miss exact terms.
Sparse Retrieval (TF-IDF vectors, BM25 ranking)
TF-IDF produces explicit sparse vectors. BM25 is a ranking function (not a vector) that shares the same keyword-matching philosophy.
Strength: Exact keyword matching. Won't miss "product-ID-12345."
Weakness: No semantic understanding. "doctor" and "physician" are completely unrelated.
Hybrid Retrieval: Best of Both
Both searches run simultaneously on the same query. Dense finds semantically relevant docs ("physician" matches "doctor"); sparse ensures exact terms aren't missed ("product-ID-12345"). Results are merged and reranked before returning.
Bi-Encoder vs Cross-Encoder
Two architectures for comparing text — one is fast, the other is accurate. Production uses both.
Bi-Encoder (the fast librarian)
Encodes query and document separately. Document vectors can be pre-computed and stored in a vector database. Comparison = fast vector similarity.
Speed: Milliseconds over millions of docs
Accuracy: Moderate (no cross-attention)
Cross-Encoder (the careful reviewer)
Processes query and document together through the full model. Every word can attend to every other word. Much richer comparison but can't pre-compute.
Speed: Slow (must run model per pair)
Accuracy: High (full cross-attention)
The Production Pattern: Two-Stage Retrieval
recall top 100
rerank to top 10
Hubness & Anisotropy
Two problems that can silently break your vector search even when the embeddings look fine.
Hubness
Some vectors become "hubs" — they show up as nearest neighbors for way too many queries, even unrelated ones. Like a popular kid who gets picked for every team.
Query: "quantum physics intro" -> Hub doc
Query: "cat grooming tips" -> Hub doc
Same doc keeps showing up!
Red hub point is nearest neighbor to many unrelated queries (arrows).
Anisotropy
Vectors crowd into a narrow cone instead of spreading evenly across the space. Like everyone at a party standing in one corner.
Everything looks "similar" to everything!
Hard to distinguish truly relevant from irrelevant.
Left: healthy (spread out). Right: anisotropic (crowded into a cone).
Fixes: Better negative sampling during training, L2-normalization, post-hoc whitening transforms, adding a reranking stage, or switching to a better-trained embedding model.
Embedding Dimension: The Trade-off
More dimensions = richer meaning but higher cost. How to pick the right size.
| Model | Dimensions | Storage per 1M docs | Use case |
|---|---|---|---|
| text-embedding-3-small | 512-1536 | ~2-6 GB | General search, budget-friendly |
| text-embedding-3-large | 256-3072 | ~1-12 GB | High-fidelity retrieval |
| all-MiniLM-L6-v2 | 384 | ~1.5 GB | Fast, lightweight search |
| BERT base | 768 | ~3 GB | General NLP tasks |
| E5-large-v2 | 1024 | ~4 GB | High-quality retrieval |
Evaluating Embeddings
How do you know if your embeddings are actually good before putting them in production?
For Retrieval
For Clustering
For Similarity / Classification
The golden rule:
Never evaluate embeddings in isolation. Test them inside the full pipeline they'll power — the same embedding model can perform very differently depending on the chunking strategy, query rewriting, reranking, and downstream task. MTEB is the standard leaderboard for comparing embedding models across tasks.
The Full Semantic Search Pipeline
How embeddings fit into a real retrieval system from end to end.
Indexing Phase (offline, once)
(FAISS, Pinecone, etc.)
Query Phase (online, per request)
(ANN lookup)
(cross-encoder)
answer + citations
How Are Embeddings Trained?
Embeddings don't appear by magic. Models learn them through clever training objectives.
Contrastive Learning
The most common approach for modern embedding models. The idea is simple:
This is how Sentence-BERT (Reimers & Gurevych, 2019) made sentence-level similarity search practical — by training BERT with contrastive objectives instead of using raw BERT outputs.
Negative Sampling
The quality of the "negative" examples matters enormously. Two strategies:
Poor negative sampling is one of the main causes of hubness and anisotropy problems.
Domain Mismatch: Not All Embeddings Are Interchangeable
An embedding model trained on Wikipedia won't necessarily work well for legal contracts or medical records.
The Problem
Embedding models learn from their training data. A general model understands everyday language but may not capture the specialized meaning of domain-specific terms — placing them near common-language meanings instead of domain-relevant ones.
The Fix
Match your embedding model to your domain:
How Image Embeddings Work
Before we get to multimodal, let's understand how a model turns raw pixels into a vector — the same way tokenization turns text into vectors.
From pixels to vector: step by step
3 channels (RGB)
= 150,528 numbers
= 196 patches
(like "tokens")
via linear projection
+ position encoding
all 196 patches
(same as text!)
= one vector
for the whole image
The key insight: patches are tokens
The Vision Transformer (ViT) treats image patches exactly like text tokens. A 224x224 image split into 16x16 patches gives 196 "tokens." Each gets embedded, position-encoded, and fed through transformer layers — the same architecture as GPT or BERT.
Before ViT: CNNs produced image embeddings too
Before Vision Transformers (2020), Convolutional Neural Networks (CNNs) like ResNet were the standard. They slide small filters across the image, building up feature maps layer by layer — edges -> textures -> parts -> objects.
The final layer's output (before the classification head) is the image embedding. ResNet-50 produces a 2048-dim vector. That vector captures "what's in this image" in a way that similar images get similar vectors.
ViT replaced CNNs for most tasks because transformers scale better and transfer-learn more effectively, but the core idea — compress an image into a fixed-size vector — is the same.
CLIP: How Multimodal Models Are Actually Trained
CLIP (Contrastive Language-Image Pretraining) by OpenAI taught models to understand both text and images by training them together. Here's exactly how.
The Training Setup
CLIP was trained on 400 million (image, text) pairs scraped from the internet — photos with their captions, alt-text, titles, etc. The training objective is elegantly simple:
in shared space
Contrastive training: the NxN matrix trick
In each training batch of N (image, text) pairs, CLIP computes the similarity between every image and every text — an NxN matrix. The diagonal (matching pairs) should be high; everything else should be low.
| "dog on beach" |
"red car" |
"sunset sky" |
"coffee cup" |
|
| dog pic | 0.92 | 0.05 | 0.12 | 0.03 |
| car pic | 0.08 | 0.89 | 0.11 | 0.06 |
| sunset pic | 0.14 | 0.07 | 0.95 | 0.09 |
| coffee pic | 0.02 | 0.04 | 0.06 | 0.91 |
Off-diagonal = non-matching (push low)
With batch size of 32,768 (CLIP's actual setting), each image is contrasted against 32,767 wrong captions. This massive scale is what makes the learned space so robust.
What CLIP's shared space looks like
After training, text and images live in the same 512/768-dim vector space. You can compare any text to any image (or image to image, or text to text) using cosine similarity.
Circles = images. Diamonds = text. Same color = matching pair. They cluster together in the shared space.
Zero-shot classification (no training needed!)
CLIP can classify images it has never seen during training. You just embed the class names as text and find which one is closest to the image:
"a photo of a dog" -> [0.3, -0.1, ...] cos=0.92
"a photo of a cat" -> [0.1, 0.5, ...] cos=0.31
"a photo of a car" -> [-0.2, 0.8, ...] cos=0.08
Winner: dog (no fine-tuning needed)
Why this was a breakthrough
Before CLIP, image classifiers needed labeled datasets per task. Want to classify dogs vs cats? Train on a dog/cat dataset. Want to classify cars? Train on a car dataset.
CLIP changed this: one model, trained once on image-text pairs, can do any classification task just by changing the text prompts. No retraining. This is the image equivalent of what GPT-3 did for text with few-shot prompting.
Beyond CLIP: The Multimodal Embedding Zoo
CLIP opened the door. Now there are specialized models for every modality combination.
| Model | Modalities | Training approach | Key use case |
|---|---|---|---|
| CLIP | Text + Image | Contrastive (NxN matching) | Zero-shot image classification, text-to-image search |
| SigLIP | Text + Image | Sigmoid loss (per-pair, not NxN) | Same as CLIP but scales to larger batches more efficiently |
| ALIGN | Text + Image | Contrastive on 1.8B noisy pairs | Shows scale > curation for multimodal learning |
| ImageBind | Text + Image + Audio + Depth + Thermal + IMU | Bind all modalities to images as anchor | Cross-modal retrieval (audio -> image, text -> depth, etc.) |
| CLAP | Text + Audio | Contrastive (like CLIP for audio) | Audio search, music recommendation, sound classification |
| ONE-PEACE | Text + Image + Audio | Shared transformer backbone | Unified multimodal understanding |
CLIP vs GPT-4V: Encode-only vs Encode+Decode
CLIP can only compare images and text (which is closer?). To actually reason about an image and generate an answer, you need to wire a vision encoder into an LLM decoder. Here's the key architectural difference:
CLIP (encode only — no decoder)
Output: a similarity score (0-1). Can answer "is this a dog or cat?" by comparing to text labels. Cannot generate text, explain reasoning, or answer open-ended questions.
GPT-4V / LLaVA (encoder + LLM decoder)
(frozen CLIP)
MLP bridge
GPT / LLaMA
answer
The CLIP vision encoder is reused (often frozen) but its output goes through a projection layer into an LLM that can reason and generate. Can answer "what breed is this dog and why do you think so?"
Inside the LLM decoder: image patches become tokens
The critical insight: the vision encoder doesn't produce one vector for the whole image. It produces one vector per patch — hundreds of them. These get projected and concatenated with text tokens, so the LLM "reads" the image the same way it reads text.
Image (384x384) -> ViT splits into 24x24 = 576 patches of 16x16 pixels. Each patch becomes a 1024-dim vector. The ViT is often a frozen CLIP encoder — reusing what CLIP already learned about images.
A learned MLP maps each 1024-dim patch vector to the LLM's hidden size (e.g., 4096-dim). This is the "bridge" — often the only new component trained when building a multimodal LLM.
The LLM's self-attention runs over all 582 tokens. Text tokens attend to image patches ("breed" attends to the fur texture patches) and generate: "This is a golden retriever."
This is why image inputs cost more — a single 384x384 image consumes 576 tokens. Higher resolution = more patches = more tokens = higher cost and latency.
Audio and video work the same way
Audio Embeddings
Audio is converted to a spectrogram (a 2D image of frequency vs time), then processed like an image through a ViT or CNN. Models like Whisper, CLAP, and AudioMAE produce audio vectors that can be compared to text or other audio.
Video Embeddings
Sample frames at regular intervals (e.g., 1 fps), embed each frame independently with ViT, then aggregate across time. Some models (ViViT, VideoMAE) process 3D patch tubes (space + time) in a single pass.