What are embeddings? Vector representations of text, images, and code, in 2026

The short answer

An embedding is a learned vector of numbers that represents a piece of data, most often a chunk of text but increasingly also images, code, or audio. The vector lives in a high-dimensional space and is trained so that semantically similar inputs end up close together and unrelated inputs end up far apart. Distance in the space is a proxy for meaning.

The aggregated source consensus, across the original word2vec paper, ¹ the GloVe paper, ² the Sentence-BERT paper, ³ and the documentation from current commercial providers, ^{4 5 6} characterises embeddings as the workhorse representation behind semantic search, retrieval-augmented generation, clustering, recommendation, classification, and anomaly detection. The shift from 2013-era word-level embeddings to 2026-era sentence and multimodal embeddings is a shift in granularity and training objective, not in the underlying idea.

For a developer choosing an embedding model today, the source consensus weights three axes: the task (general English retrieval, multilingual, code, multimodal), the quality as measured by the MTEB benchmark, ^{7 8} and the cost and deployment model (closed-API per-token billing versus self-hosted open weights).

What “vector representation” actually means

A vector is just an ordered list of numbers. An embedding vector for the sentence “the cat sat on the mat” might be a list of 1,024 floating-point numbers; each individual number carries no human-readable meaning, but the pattern across all 1,024 dimensions, taken together, encodes the sentence’s meaning in a way the embedding model has learned during training.

Two embeddings can be compared. If the angle between the two vectors is small (or equivalently, their cosine similarity is close to 1), the model considers the two inputs semantically similar. If the angle is large, the model considers them unrelated. The geometry of the space is the meaning.

The dimension of the vector is a model-specific choice. Word2vec used 300 dimensions in its widely-cited 2013 configuration. ¹ Sentence-BERT models typically use 384 or 768 dimensions. OpenAI’s text-embedding-3-large produces 3,072 dimensions by default and can be truncated to a smaller size at API request time. ⁴ Cohere’s embed-english-v3.0 produces 1,024 dimensions. ⁵ Higher dimensions usually mean more capacity to distinguish fine-grained meaning but also more storage and slower distance computation.

Image: thumbnail from the Hugging Face MTEB launch blog post, used for editorial coverage of the benchmark.

Distance metrics: cosine, dot product, Euclidean

Three distance measures dominate practical work.

Cosine similarity measures the angle between two vectors and ignores their magnitude. For vectors $\mathbf{a}$ and $\mathbf{b}$:

$\mathrm{cos\_sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert}$

Cosine similarity returns a value between $-1$ and $1. Identical vectors give \1; orthogonal vectors give \0$; opposite vectors give$-1$. Sentence-BERT was explicitly designed to produce embeddings that work well under cosine similarity, and the paper trains the siamese network so that semantically equivalent sentences sit close in cosine space. ³

Dot product is the numerator of the cosine formula, $\mathbf{a} \cdot \mathbf{b}$, with no normalisation. When the model is trained to produce unit-norm vectors (length 1), dot product and cosine similarity are identical. Most modern embedding APIs return unit-norm vectors precisely so dot product becomes a fast drop-in for cosine.

Euclidean (L2) distance is the straight-line distance between two points in the space:

$d_{L2}(\mathbf{a}, \mathbf{b}) = \sqrt{\sum_{i=1}^{n} (a_i - b_i)^2}$

For unit-norm vectors, L2 distance and cosine similarity are monotonically related; ranking results by either gives the same order. The choice between the three is largely a matter of which the vector database is optimised for, not which encodes “real” similarity.

How embeddings are trained

The training objective shapes what “similar” ends up meaning to the model. Three families dominate.

Predictive objectives, the word2vec family. Word2vec trains a shallow neural network to predict either the surrounding words from a target word (Skip-gram) or the target word from the surrounding words (Continuous Bag of Words), using a 1.6-billion-word corpus. ¹ The vectors that fall out as a byproduct turn out to encode striking analogical structure (the famous king - man + woman ≈ queen result).

Co-occurrence factorisation, the GloVe family. GloVe trains a model directly on the matrix of word co-occurrence counts, factorising it so that the dot product of two word vectors approximates the logarithm of their co-occurrence probability. ²

Contrastive learning, the modern dominant approach. Sentence-BERT and its successors train on pairs (or triplets) of sentences where one pair is labelled “semantically similar” and another labelled “dissimilar”. The objective pushes similar pairs together and dissimilar pairs apart in the vector space. ³ Modern commercial embeddings (OpenAI, Cohere, Voyage) and modern open-source models (BGE, Nomic, E5) are all contrastive-trained at scale on large pair datasets mined from the web. The contrastive objective is what lets a single model serve retrieval, clustering, and classification — the geometry it learns generalises across all three.

The 2026 commercial and open-source landscape

The publicly documented models cluster into a small set of practical choices. The figures below cite vendor pricing pages; prices fluctuate, so verify before purchase on the day of integration.

Model	Dimensions	Max input tokens	Pricing per 1M tokens	Source
OpenAI text-embedding-3-large	3,072 (truncatable)	8,191	$0.13	4
OpenAI text-embedding-3-small	1,536 (truncatable)	8,191	$0.02	4
Cohere embed-english-v3.0	1,024	512	$0.10	5
Voyage AI voyage-3-large	1,024–2,048	32,000	$0.18	6
Voyage AI voyage-3	1,024	32,000	$0.06	6
BAAI bge-m3 (open-weights)	1,024	8,192	self-host	10
Nomic nomic-embed-text-v1.5 (open-weights)	768 (truncatable to 64 via Matryoshka)	8,192	self-host	9

Pricing as of 2026-05-19; verify on the vendor’s pricing page before integration.

The Massive Text Embedding Benchmark (MTEB) evaluates models across a published taxonomy of tasks (classification, clustering, retrieval, semantic textual similarity, reranking, summarisation, bitext mining, pair classification) and across 112 languages. ^{7 8} The leaderboard is the most-cited cross-vendor comparison surface; per current rankings reported by aggregator sources tracking the leaderboard, Cohere embed-v4, OpenAI text-embedding-3-large, and Voyage’s voyage-3-large cluster at the top of the English-retrieval table, with open-weight bge-m3 and nomic-embed-text-v1.5 within a few benchmark points behind. Sources flag that MTEB scores have known limitations: a model can be over-optimised against the benchmark’s specific tasks and still underperform on a downstream production workload, so the leaderboard is a starting filter rather than the final word.

What embeddings are used for

Five workflows account for most of the production activity.

Semantic search. Index a corpus by embedding every document, embed an incoming query the same way, return the documents whose vectors are closest to the query vector. The advantage over keyword search is that the query “how do I cancel my subscription” matches a document titled “ending your paid plan” even with zero word overlap.

Retrieval-augmented generation. The retrieval step inside a RAG pipeline is usually a vector search over a chunked corpus, using an embedding model. The top-$k$ chunks get inserted into the prompt of a generation model. Embedding quality directly affects RAG answer quality; a weak embedding model surfaces irrelevant chunks and the generator confabulates over them.

Clustering and topic discovery. Embed a large corpus, run a clustering algorithm (k-means, HDBSCAN) over the vectors, inspect the clusters. The clusters correspond approximately to topics the embedding model considered coherent.

Classification on small data. Embed a small labelled dataset, train a lightweight classifier (logistic regression, k-nearest-neighbours) on top of the embeddings. This works well when fine-tuning the whole model is overkill and few-shot prompting is too brittle.

Anomaly and duplicate detection. Items whose embedding is unusually far from any cluster center are anomalies; items whose embeddings are unusually close to another item are near-duplicates.

Multimodal and code embeddings

Text is the most-cited modality but the same idea extends. Image embeddings from models like OpenCLIP and SigLIP map an image to a vector in the same space as text embeddings, so a text query can retrieve images. Code embeddings from Voyage’s voyage-code-3 and OpenAI’s embedding models trained on code map source files to vectors so a natural-language description can retrieve relevant code, and a code snippet can find similar snippets in a repository. ^{6 4} The training objective differs in detail (typically a contrastive image-text or code-text pairing) but the downstream usage pattern is identical to text embeddings.

Common pitfalls

Five things trip up first-time users.

First, chunk size matters more than model choice for most RAG deployments. A high-end embedding model on naively-chunked data underperforms a mid-tier embedding model on well-chunked data. Most embedding models have a maximum input length; exceeding it silently truncates or fails. Cohere’s embed-english-v3.0 caps at 512 tokens, which is short enough that careless chunking discards meaningful context. ⁵

Second, distance is not meaning. The space the model learned encodes whatever pattern the training data optimised for. A model trained on web text will rank “the cat sat on the mat” and “a feline rested on the rug” as similar because the training data correlates them; on a domain the model didn’t see in training, the geometry can collapse.

Third, embeddings are not interchangeable. A query embedded by OpenAI cannot be searched against a corpus embedded by Cohere; the vector spaces are unrelated. Switching models means re-embedding the entire corpus.

Fourth, dimension truncation is not free. Modern models like text-embedding-3-large and nomic-embed-text-v1.5 support truncating the output vector via Matryoshka representation learning, which preserves most quality at smaller dimensions. ^{4 9} Older models do not have this property and cannot be safely truncated.

Fifth, MTEB is not a guarantee. A model that tops the English retrieval task on MTEB can underperform on a domain-specific corpus (medical records, legal contracts, internal documentation). Always evaluate on a held-out sample of the actual target data before committing.

Honest caveats

This explainer covers the embedding families that dominate the 2026 production landscape. It does not cover every variant: ColBERT-style late-interaction embeddings, instruction-tuned embeddings, hyperbolic embeddings, and graph embeddings each have their own literature and use cases.

The pricing figures cite each vendor’s pricing surface as of 2026-05-19, surfaced via the OpenAI embeddings guide, the Cohere model documentation, and the Voyage AI pricing page. The OpenAI and Cohere pages were not directly fetchable in the research session; the figures reflect the vendor pages as reported by aggregator coverage of those surfaces, and the cited vendor URLs in the footnotes are the canonical references readers should verify against on the day of integration. Vendor pricing for embeddings drifts on a roughly quarterly cadence.

MTEB scores cited here come from the leaderboard rather than from an independent reproduction. The leaderboard is community-curated and individual scores can move when a model card is updated.