When the Queue Grows: Backpressure, Admission and Dead Letters

A Work Queue lets producers submit work without waiting for a worker to finish. That separation absorbs short bursts and smooths uneven workloads. It does not remove the relationship between arrivals and capacity. If work arrives faster than workers complete it for long enough, the queue grows, jobs wait longer, deadlines expire and recovery becomes harder.

The safe response is to measure the gap, bound it and tell producers when the system cannot accept more.

A queue is delayed demand

Queue depth alone is easy to read incorrectly. One thousand waiting jobs may be harmless if workers finish ten thousand per minute, or disastrous if they finish ten. Pair depth with rates and age:

arrival rate: jobs accepted per minute;
completion rate: jobs reaching a terminal state per minute;
oldest waiting age;
time-to-start percentiles;
number of jobs already past their useful deadline;
retry traffic as a share of all claims.

Suppose arrivals average 120 jobs per minute and workers finish 80. The backlog grows by roughly 40 per minute. After half an hour, about 1,200 additional jobs are waiting. Adding a larger queue changes where they wait, not the 40-job gap.

Queueing relationships such as Little’s Law connect average items in a stable system, throughput and average time in the system. The practical lesson is simple: more items waiting usually means more waiting time unless completion capacity rises or arrivals fall.

Bound the waiting room

Every queue needs explicit bounds. Useful limits include:

maximum waiting jobs;
maximum bytes of queued payloads;
maximum age or a per-job deadline;
per-customer or per-namespace quotas;
a cap on concurrent claims for a shared dependency.

When a bound is reached, the system needs an admission decision. It can reject new work, ask the producer to slow down, accept only higher-priority classes, or direct work to a different controlled path. Silently accepting work that cannot start before its deadline is not kindness; it hides failure until later.

This signal is Backpressure. It moves capacity information upstream while the producer can still choose what to do.

Make the response actionable

A useful refusal says why it happened and whether a later retry may help. Compare:

“Error” gives the producer no safe action.
“Queue at limit” explains the condition.
“Queue at limit; retry after 30 seconds” gives a temporary recovery instruction.
“Deadline cannot be met” says retrying the same request is pointless.

Producers that retry should use Exponential Backoff with jitter and an attempt or elapsed-time limit. An immediate synchronized retry turns controlled backpressure into a new traffic spike.

Admission also needs fairness. One noisy producer should not consume every waiting slot. Separate quotas or weighted sharing can preserve room for smaller or higher-value workloads, but the policy should be visible rather than hidden in arbitrary queue order.

Distinguish slow, stuck and impossible work

A growing queue can have different causes:

Pattern	Likely cause	Response
Arrivals briefly exceed completions, then recover	Normal burst	Let bounded queue absorb it
Completions fall across all workers	Shared dependency or worker failure	Reduce admission; investigate dependency
One job type retries repeatedly	Persistent input or executor defect	Stop retry loop; isolate failures
Jobs wait with no eligible worker	Placement mismatch	Explain missing capability; correct supply or request
Oldest age rises while depth looks flat	New work may be jumping the line	Inspect priority and fairness policy

Depth is an outcome. Diagnose which flow changed before scaling workers. More workers can make a shared database, external service or rate limit fail faster.

Use dead letters deliberately

Some work should leave the normal queue. After bounded retries, expiry or a permanent validation error, move the item to a Dead-letter Queue with its error and attempt history. This prevents one poison item from cycling forever and preserves it for inspection.

A dead-letter queue needs four policies:

Entry: which terminal conditions send an item there?
Retention: how long is the item kept?
Ownership: who reviews growth and individual failures?
Recovery: is the item corrected, replaced, replayed or discarded with a reason?

Blind replay is dangerous. If the original attempt partly completed an external side effect, replay may duplicate it. Confirm Idempotency or use the destination’s stable idempotency key before putting work back.

Protect downstream services too

Workers often call the same database or external service. The job system may have spare CPU while that dependency is saturated. Concurrency limits, Rate Limiting and a Circuit Breaker keep worker parallelism from overwhelming the slower part.

These controls belong close to the constrained resource. A global worker count may be too coarse: 100 jobs can run safely when they use different services, while three concurrent jobs may be too many for one exclusive account.

Plan recovery before the incident

When arrivals return to normal, the backlog still exists. Decide how recovery works:

reserve some capacity for fresh, time-sensitive work;
expire jobs whose result is no longer useful;
process high-value classes first without starving the rest;
increase workers only after confirming downstream capacity;
watch retry rate so old failures do not consume the new capacity;
estimate clearance time from backlog and net completion rate.

If 6,000 jobs are waiting, arrivals continue at 100 per minute and expanded workers complete 160, the backlog shrinks by 60 per minute. Clearing it takes about 100 minutes, not 37.5, because new work continues to arrive.

Test overload as a normal state

Run a controlled test in which arrivals exceed capacity. Verify that the queue stops at its bound, refusal is explicit, producers back off, deadlines are enforced, and monitoring shows oldest age as well as depth. Then inject a permanently failing job and confirm that it reaches the dead-letter path after the stated limit without blocking healthy work.

A queue earns trust not when it can hold an unlimited number of jobs, but when it behaves predictably at its limit: bounded memory, honest admission, fair waiting, safe retries and a visible place for work that cannot succeed.