Evidlake
Menu

Entity Resolution

Also known as: record linkage, data matching, deduplication of entities.

Deciding which records, in one source or many, refer to the same real-world thing, and linking them under one identifier.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Definition

Entity resolution decides which records, from one source or many, refer to the same real-world thing, such as the same product or company listed under different names, and links them under one identifier. Good practice links rather than merges, so a wrong match can be undone.

In plain terms

Names are messy: “Acme Widget 12 oz”, “ACME Widget (12oz)”. Identifiers are not always shared between sources. Entity resolution normalises the descriptions, compares candidate pairs on several fields, and decides match, non-match or “needs a look”.

Two mistakes are possible and they pull in opposite directions. A false merge joins two different things and contaminates both histories; a missed match leaves one thing split in two. Most systems tune the threshold to the cost of each and send the uncertain middle to a person.

Example

Four records after lower-casing and removing punctuation:

RecordNormalisedVerdict against the first
A: “Acme Widget 12 oz”acme widget 12 oz(reference)
B: “ACME Widget (12oz)”acme widget 12oz → acme widget 12 ozMatch
C: “Acme Widget 16 oz”acme widget 16 ozDifferent size: not a match
D: “Acme Gadget 12 oz”acme gadget 12 ozDifferent product: not a match

A rule that compared only brand and product name would merge A, B and C, and the 16 oz size would then inherit the 12 oz history.

Common misunderstandings

  • “An exact name match is enough.” Names vary in spelling and format; and identical names can belong to different things.
  • “Once merged, the decision is final.” Keep the original records and the link, so that a mistake can be reversed.
  • “A model can decide every case.” The uncertain middle needs a threshold and, in practice, human review.
  • Data Lineage

    The recorded path of a piece of data: where it came from, how it was changed, and what depends on it.

  • Data Quality Dimensions

    Named aspects — accuracy, completeness, consistency, timeliness, uniqueness — that turn the vague idea of clean data into checks you can measure.

References

  1. Peter Christen. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection . Springer, 2012. ISBN 978-3642311635.
  2. Ivan P. Fellegi, Alan B. Sunter. A Theory of Record Linkage . Journal of the American Statistical Association, 1969. DOI: 10.1080/01621459.1969.10501049