Counting Stories, Not Copies: Near-Duplicates and Syndicated News

One story from a news agency can appear in forty newspapers, three aggregators and a dozen blogs, each a few words different from the next. Search for a company and you find forty-five “articles”. Count them and you may conclude that the company dominated the news. In reality, one report was reprinted. Telling the two apart is the job of near-duplicate detection, and it is the foundation of any honest count of coverage (Mention Volume).

Why copies exist

• Wire copy: agencies write once and sell to many outlets, which publish it unchanged or lightly edited.
• Republishing: sites repost articles from partners, with a note of the original source or without one.
• Aggregators and mirrors: pages that gather snippets or whole stories automatically.
• Templates: press releases, earnings summaries and sports results arrive as near-identical text with a few numbers changed.

Copies matter for exposure (how many places carried it) and do not matter for independence (how many sources stand behind it). A claim repeated forty times is not forty confirmations (Corroboration, Syndicated Content).

Exact duplicates are easy, near-duplicates are not

A checksum of the text (Cryptographic Hash) finds exact copies instantly. But change one word, add a byline, or append “Read more” and the checksum is completely different. Real copies are nearly identical, so we need a measure of how much two texts overlap.

Shingles and the Jaccard score

The standard approach, from Andrei Broder’s 1997 paper, cuts each text into overlapping runs of consecutive words called shingles. With runs of three words, the sentence “the central bank raised rates on tuesday citing persistent inflation” becomes:

the central bank · central bank raised · bank raised rates · raised rates on · rates on tuesday · on tuesday citing · tuesday citing persistent · citing persistent inflation

That is 8 shingles. Two texts are then compared by the Jaccard similarity: shingles they share, divided by all distinct shingles in either. Identical texts score 1, texts with nothing in common score 0.

Compare that sentence (A) with four others:

Notice what the numbers do. The copy with an added clause (B) scores high; a one-word rewrite (C) scores about 0.45 because one changed word breaks three shingles; D, a genuine second report of the same event, scores 0, which is right: it is not a copy. E shares a little boilerplate with A but is a different story.

Choosing the threshold

A threshold turns scores into decisions: “similar enough to count as the same story”. Somewhere between 0.4 and 0.7 is typical for news, but it must be chosen on your own texts. Too low, and different stories get merged (a company’s earnings follow the same template every quarter); too high, and lightly edited copies slip through as separate stories. The way to choose is to take a sample of pairs, judge each by hand, and see which threshold agrees best with you (Precision and Recall). Check again when the sources change.

Doing it at scale

Comparing every pair of texts is impractical with millions of documents. Techniques such as MinHash and SimHash compress each text’s shingles into a short signature so that similar texts can be found quickly, without comparing everything with everything. The ideas are the same; the speed is what changes. The references below describe both.

From pairs to clusters

Once pairs are linked, group them into clusters: a cluster is a story. For each, keep:

• the original (the earliest or the most complete version, if you can tell),
• the number of copies and the list of outlets (for exposure),
• the first publication time (for timing),
• the link to every member, so a mistaken merge can be reversed.

Never delete the copies. Counting stories is a view over the stored mentions, not a deletion of them.

Where it goes wrong

• Boilerplate: headers, footers, cookie notices and “related stories” boxes make unrelated pages look alike. Strip them before comparing.
• Short items: a headline or a tweet has too few shingles for a stable score.
• Translations and heavy rewrites: word overlap vanishes, so these appear as independent reports even when they are copies. Embedding-based similarity can help but brings its own errors (Text Embedding).
• Templates: daily weather, results and filings differ only in numbers and look alike by design; treat them with a separate rule.
• Chinese and other unspaced languages: shingles of words need word segmentation first, or shingles of characters instead (Tokenization).

What to report

Report both measures and name them:

A statement such as “the company was reported in 6 stories, carried by 45 outlets” is more accurate, and more informative, than either number alone.