Event Detection

Definition

Event detection is finding, in a stream of documents, the moments when something new has happened, typically by spotting a sudden burst of documents or terms about the same thing. Grouping similar documents into stories first separates “a lot of coverage” from “a lot of copies”.

In plain terms

Two steps do the work. First, group documents that are about the same thing, so that a story reprinted fifty times counts once (see Near-duplicate Detection). Second, watch how fast stories arrive and flag the times when the rate is far above its usual level. A well-known model, by Jon Kleinberg, treats a stream as switching between a low-rate and a high-rate state and finds the bursts that way.

Practical care goes into the baseline (weekdays differ from weekends), the threshold, and changes in what is being collected, which can look like events.

Example

Stories per day mentioning a topic. The first seven days set the baseline: 10, 9, 11, 10, 12, 9, 11, with a mean of 10.3 and a standard deviation of 1.1. A simple rule flags a day as a burst when its count exceeds the mean plus three standard deviations, here 13.6.

The event runs from day 8 to day 10. If day 8’s 48 were mostly copies of one story, deduplicating first would show whether it was one event or many.

Common misunderstandings

• “Every burst is a real-world event.” Bursts also come from reprints, bots, a change in collection and scheduled releases.
• “One threshold fits every topic.” Baselines and variability differ widely between topics and sources.
• “An event is the same as a topic.” A topic persists for months; an event is a time-bounded happening within it.