Heartbeat

Definition

A heartbeat is a small periodic signal showing that a process or node was recently able to communicate. Missing several heartbeats may make the receiver suspect failure, but cannot prove it: delay, overload and network partition look like a crash. Heartbeats provide liveness evidence, not complete health.

In plain terms

A worker regularly says “I am still here.” If the coordinator stops hearing that message, it can mark the worker as suspect and eventually stop sending new work. The waiting period prevents one delayed packet from turning a healthy machine into a false failure.

A heartbeat should carry an observation time and, where useful, a small status summary. It should not become a large monitoring upload: the more expensive it is, the more likely overload will delay the very signal used to detect overload.

Example

A node sends a heartbeat every 15 seconds. The coordinator marks it suspect after 45 seconds without one and offline after 90 seconds.

The node might still be running at 95 seconds. Jobs need their own Lease and stale-result rules; node status alone should not decide which attempt owns a result.

Common misunderstandings

• “No heartbeat means the process is dead.” It means communication was not observed within the threshold.
• “Heartbeat received means the worker is healthy.” A process can send heartbeats while a disk, executor or dependency is failing.
• “Use one timeout everywhere.” Networks and workloads differ; thresholds should reflect observed delay and recovery goals.