Watermark

Definition

A watermark marks how far a pipeline has progressed, typically the latest timestamp or sequence number it has fully processed. Incremental jobs ask the source only for records beyond it, and records that arrive behind it need a separate rule or they are missed.

In plain terms

Re-reading everything on every run is wasteful, so incremental jobs remember a position: “I have everything up to 10:02”. The next run asks for records after 10:02. This works perfectly when records arrive in order.

In real systems they do not. A record about 09:58 can reach the source at 10:07. If the watermark has already moved past 09:58, that record is never requested. The two usual remedies are to look back a safety margin on every run (and rely on idempotent writes to ignore repeats) or to key the watermark on arrival time rather than event time. In stream processing the word has a related meaning: an estimate of how complete the data is up to a given event time.

Example

A job runs every 5 minutes and asks for records with event time after its watermark.

A record with event time 09:58 reaches the source at 10:07. By the 10:10 run the watermark is 10:02, so the record (09:58) is skipped for good. With a 15-minute lookback the 10:10 run asks for everything after 09:47, finds it, and an idempotent write keeps the records it has already seen from being counted twice.

Common misunderstandings

• “The watermark is just the time of the last record.” In incremental loads it is the position you have safely reached, and late records can fall behind it.
• “A watermark guarantees completeness.” It only says how far you have read; completeness needs a rule for late data.
• “Lookback means double counting.” It does not, provided writes are idempotent.