When a Worker Disappears: Leases, Retries and Safe Job Execution

A background job seems simple when everything works: put work in a queue, let a worker take it, then record the result. Failure makes the hidden questions visible. What if the worker loses its connection after doing the work but before acknowledging it? What if it pauses for a minute and returns after another worker has taken over? What if a retry repeats an external side effect?

Reliable execution starts by accepting that a coordinator cannot perfectly distinguish a dead worker from a slow or disconnected one. It uses time limits, attempt records and repeat-safe operations to make that uncertainty manageable.

Separate the job from the attempt

A job describes the desired outcome: resize image 42, rebuild report 17, or calculate a score for batch 9. An attempt is one particular try by one worker. The distinction matters because one job may need several attempts.

Give the job a stable identifier and each attempt a new identifier:

Record	Identity	Typical state
Job	Stable across retries	queued, running, succeeded, failed, cancelled
Attempt	New for every claim	claimed, active, timed out, completed, rejected as stale

Keeping attempts makes failure inspectable. Instead of overwriting “running” with “failed” and losing the path, the record can show that attempt A timed out, attempt B was claimed later, and B produced the accepted result.

A claim needs a time limit

If claiming a job creates a permanent lock, one crashed worker can strand the job forever. A Lease makes the claim temporary. The worker receives the job for a fixed period and renews that lease while it is healthy. If renewals stop, the coordinator eventually allows a new attempt.

The lease duration is a trade-off:

Too long, and recovery after a failed worker is slow.
Too short, and normal pauses or network delays cause false expiry.
Renew too often, and the coordinator spends unnecessary effort on heartbeats.
Renew too late, and a brief delay can make a healthy attempt look stale.

A common shape is to renew well before expiry and add a little randomness so thousands of workers do not renew in the same instant. The actual values should come from observed job times and network behaviour, not a copied default.

Expiry is permission to retry, not proof of death

Suppose attempt A has a lease until 10:01. Its network connection breaks, but it keeps computing. At 10:01 the coordinator creates attempt B. At 10:02 both finish.

Nothing in the lease can stop A’s process by magic. The system must decide which result is acceptable. Useful defences include:

Check the attempt at completion. Accept a result only if its attempt is still the current one.
Use a fencing number. Every new lease receives a larger number; the destination rejects writes carrying an older number.
Make the operation idempotent. Repeating it produces no extra effect (Idempotency).
Keep immutable attempt records. Preserve late results for diagnosis even when they are not accepted as the job’s result.

The right combination depends on where side effects happen. A database under your control can check a fencing number in the same transaction as the write. An e-mail or payment sent to an external service may require that service’s idempotency key. A command that changes an unmanaged file may not be safely retryable at all.

Acknowledgements close the loop

In a Work Queue, removing a job as soon as a worker receives it risks losing the job if the worker then fails. Reliable queues normally wait for an acknowledgement. The worker claims, performs the work, stores the result durably and only then acknowledges completion.

There is still a narrow failure window: the result can be stored while the acknowledgement is lost. The queue then offers the job again. This is why At-least-once Delivery and idempotent result handling belong together. Redelivery is not an exceptional bug; it is one possible outcome of choosing not to lose work.

Large outputs often need a separate hand-off. A worker can upload a file to an Artifact Staging Area, report its content hash, and let the consumer acknowledge that it has safely copied and verified the file. The staging area can then expire the temporary copy. This keeps transport separate from long-term evidence storage and makes incomplete transfers detectable.

Retry only failures that may improve

Not every failure deserves another attempt. Separate them before retrying:

Failure	Example	Sensible response
Temporary infrastructure failure	Worker disconnects; service times out	Retry with a limit and Exponential Backoff
Invalid job	Required field is missing	Fail and report; a retry sees the same input
Policy rejection	Worker lacks permission or capability	Do not run there; find a qualified worker or stop
Unsafe side effect	Outcome is unknown and no idempotency key exists	Pause for review rather than guessing
Deadline passed	The result is no longer useful	Stop and record the deadline outcome

Bound every retry policy by attempts, elapsed time or a job deadline. Infinite retry turns a permanent error into permanent load and hides the real failure behind a queue that never empties.

Placement is part of correctness

A free worker is not necessarily a suitable worker. A job may require a particular operating system, enough memory, a local dataset or exclusive access to a resource. Express those needs as explicit capabilities and constraints. Do not rely on a machine name that happens to imply them today.

When no worker satisfies the constraints, say so. “Queued” is misleading if the system already knows the job cannot run anywhere. A useful status explains which requirement is unmet and applies a deadline so the job does not wait forever.

What to record

Good execution history answers operational questions without reconstructing them from scattered logs:

Who submitted the job, and with what stable idempotency key?
Which constraints determined placement?
Which worker and attempt held the lease at each time?
When were the job, attempt and acknowledgement created?
Why did an attempt stop, and is the error safe to retry?
Which attempt produced the accepted result?
Was a late result rejected, retained for diagnosis or accepted because no newer attempt existed?

Use timestamps with one time standard and keep the worker’s observations separate from the coordinator’s receipt times. This makes clock differences and delayed messages visible rather than silently rewriting history.

Test the uncomfortable paths

A happy-path test proves little about a distributed job system. Add tests that deliberately create ambiguity:

Stop a worker after it claims a job but before it starts.
Stop it after the side effect but before acknowledgement.
Delay renewal until the lease expires, then let the old attempt return.
Deliver the same job twice and verify one logical effect.
Submit a job for which no worker qualifies and verify the explanation and deadline.
Exhaust the retry limit and verify that the final error keeps the earlier attempts.

The aim is not to remove uncertainty from a network. It is to make every uncertain transition explicit, bounded and reviewable: a stable job, a sequence of attempts, a time-limited claim, a recorded result and a retry policy that never assumes repetition is harmless.