Text Embedding

Definition

A text embedding is a list of numbers that represents a piece of text so that texts with similar meaning have vectors that lie close together. Search systems use embeddings to match by meaning, but the numbers are hard to interpret and can blur exact details such as names and negation.

In plain terms

A model learns, from a very large amount of text, to place words or sentences in a space where neighbours tend to be related. Comparing two embeddings, usually with Cosine Similarity, then gives a measure of how related the texts are, even when they share no words.

Strengths: paraphrases, synonyms and related concepts. Weaknesses: negation (“works” vs “does not work”) can end up close, exact names and numbers can blur, behaviour on a new domain is hard to predict, and the reason a result was returned is hard to explain. Embeddings from different models cannot be compared, so changing the model means re-embedding everything.

Example

A toy example with two numbers per word (real embeddings use hundreds):

A word-matching search for “car” never returns a page that says only “automobile”. An embedding search puts it next to “car”. The same method could also put “the engine does not start” close to “the engine starts”, because the topic words are alike.

Common misunderstandings

• “Embeddings understand text the way people do.” They capture statistical patterns of use, which usually track meaning but not always.
• “Embeddings from two models can be mixed.” Each model defines its own space; mixing them gives meaningless distances.
• “Meaning-based search always beats keyword search.” It often needs both, and the better one is an empirical question (see Normalized Discounted Cumulative Gain).