Subject
Relevance
Matching, ranking and judging how well results fit what was asked.
Tutorials
- Keyword Match vs. Semantic Relevance: How to Tell Whether a Ranking Is Good
Two ways to decide that a result fits a query, and a simple method for judging which ranking is better: measure it against people's judgments.
- Counting Stories, Not Copies: Near-Duplicates and Syndicated News
News travels by copy. To count what was reported, rather than how often it was reprinted, group near-identical texts. A walk-through of shingles, similarity scores and the limits of the method.
- Designing Scores You Can Explain: Rules, Weights, Unknowns and Reasons
A score that cannot say why is a number to be argued about. How to build a scoring scheme with hard rules, weights, a proper place for unknown values, reasons with every result and a check against human judgments.
Glossary terms
- BM25
A ranking function that scores documents for a query from the query words they contain, with a cap on the credit for repetition and an adjustment for length.
- Cohen's Kappa
How much two raters agree beyond what chance alone would produce: 1 is perfect, 0 is no better than chance.
- Cosine Similarity
How closely two vectors point in the same direction, as the cosine of the angle between them. It ignores length.
- Explainable Scoring
Returning, with every score, the reasons behind it: which rules fired, what each criterion contributed and what data were missing.
- Inverted Index
A table mapping each word to the documents that contain it, so a search engine can find matches without scanning every document.
- Mean Reciprocal Rank (MRR)
The average, over many queries, of one divided by the position of the first correct result. It rewards finding one right answer early.
- Named Entity Recognition (NER)
Finding mentions of real-world things in text, such as people, organisations and places, and labelling their type. Linking each mention to one identifier is a separate step.
- Near-duplicate Detection
Finding documents that are almost, but not exactly, the same, such as a news story republished with a new headline, by comparing fingerprints of their word sequences.
- Normalized Discounted Cumulative Gain (nDCG)
A score for a ranked list that adds the graded relevance of results with lower positions discounted, divided by the best score possible. It runs from 0 to 1.
- Position Bias
People click results partly because they are high on the page. Click counts therefore reflect the ranking as well as the relevance.
- Precision and Recall
Two measures of search and classification quality: how clean the returned results are, and how complete they are.
- Relevance Judgments
Human assessments of how well each result answers a query, recorded as grades. They are the answer key against which ranking methods are scored.
- Search Intent
What the person behind a query is trying to achieve: find a place, learn something, or do something such as buy. Relevance depends on it.
- Sentiment Analysis
Automatically classifying text by the attitude it expresses, usually positive, negative or neutral. It reads wording, not truth, and has well-known blind spots.
- Text Embedding
A list of numbers representing a text so that texts with similar meaning lie close together. Useful for matching by meaning; hard to interpret.
- TF-IDF
A weight for a word in a document: how often it occurs there, multiplied by how rare it is across all documents.
- Three-valued Logic
Logic with a third value, unknown, besides true and false. A rule that depends on a missing value is unknown, not false.
- Tokenization
Splitting text into the units a search or analysis system works with, usually words. It decides what can match, and in Chinese finding the word boundaries is a task of its own.
- Weighted Scoring Model
Rates each candidate on several criteria, multiplies each rating by an agreed weight and adds them up. Trade-offs become explicit, and the answer depends on the weights.