Late-arriving Data

Definition

Late-arriving data are records that reach a pipeline after the period they belong to has already been processed or reported. They are common, through delayed reports, corrections and offline devices, so a pipeline needs an explicit rule: ignore them, reprocess, or record a revision.

In plain terms

The moment something happened and the moment you learn about it are different times, and the gap is not constant. A store uploads Monday’s sales on Wednesday. A source corrects last month’s figure. A sensor was offline for a day and sends its backlog.

Ignoring late data silently makes totals too low. Overwriting yesterday’s report silently makes it impossible to say what it showed when it was published. The careful approach keeps both: the revised value, and the fact that it became known later.

Example

Monday’s sales of a store group, as the system learned them:

Both numbers are right as of their time. A weekly review written on Tuesday used 1,000; recomputing it from today’s table would use 1,040. Keeping both, with the time each became known, is what lets you reproduce the Tuesday report and also see the revision.

Common misunderstandings

• “Late data is bad data.” It is real data that arrived late; discarding it biases totals downwards.
• “Just reprocess the old period.” Do it as a revision that keeps the earlier version, or past reports cannot be reproduced.
• “A fixed delay solves it.” Waiting longer reduces the problem but never removes it; a rule for stragglers is still needed.