Choosing Where Work Runs: Capabilities, Capacity and Placement

When several workers are idle, sending a job to the next one in a list can look fair and simple. It is correct only when every worker is interchangeable and every job costs the same. Real work has requirements: an operating system, a tool version, enough memory, a local device, a licensed account or exclusive access to a profile. Workers also change while the queue is waiting. A Job Scheduler has to make those facts explicit.

Start with hard requirements

A hard requirement decides whether a worker is eligible at all. Examples include:

• operating system and processor architecture;
• an installed, approved executor version;
• access to a named local resource;
• enough memory or disk for the minimum safe run;
• permission to serve the job’s namespace;
• an exclusive resource that is not already reserved.

Write these as data, not as implications hidden in a hostname such as fast-mac-2. Names drift; explicit capabilities can be checked. A worker either satisfies every hard requirement or it does not enter the candidate set.

This filtering step protects correctness. A job that needs a local credential is not merely slower on a node without it; it cannot complete there. A process built for another architecture does not become valid because that node has spare memory.

Then rank the eligible workers

After filtering, the scheduler may have several valid choices. Preferences can rank them without pretending they are requirements:

• keep some spare capacity for urgent jobs;
• prefer a worker that already has a large input cached;
• spread long jobs across failure domains;
• reduce transfer distance;
• keep related work together when locality matters;
• avoid repeatedly selecting the same worker.

Mixing preference with eligibility creates confusing failures. If cached data is helpful but not required, a cache miss should lower a score rather than make the job impossible to place. If a licence truly binds the job to one machine, model it as a requirement.

Capacity is more than a worker count

“Four workers online” says little about how much work can start. One machine may offer eight small slots; another may safely run only one memory-heavy task. Model the resources that actually constrain execution.

Consider three waiting jobs and two nodes:

Job B cannot run on Node 1 because its memory minimum is not met, and it cannot run on Node 2 because profile X is absent. The correct result is not to force placement. It is to keep B waiting with an explanation while A and C use capacity that they can safely consume.

Requests should describe what a job needs; limits should protect the worker from runaway use. Historical measurements can improve estimates, but should not silently turn a safety minimum into an optimistic average.

Exclusive resources need names and scope

Some resources cannot be shared even when CPU and memory are free: a browser profile, camera, hardware token, output directory or rate-limited account. Give each resource a stable identity and define its scope.

An exclusivity rule might apply:

• on one worker;
• across all workers attached to one account;
• across the whole job namespace;
• only during one phase of execution.

A plain “maximum concurrency = 1” is ambiguous without that scope. It can accidentally serialize unrelated work or allow two nodes to use what should be one global resource.

Treat observations as dated evidence

Worker capacity and health are observations, not timeless facts. A Heartbeat received five seconds ago is stronger evidence than one received five minutes ago. Display when CPU, memory, disk and capability data were observed. When a worker becomes suspect, stop placing new work there even if its last capacity report looked excellent.

Do not use node status as a substitute for job ownership. A worker can look offline while its process still runs. Existing attempts need their own Lease and stale-result rule.

Make waiting explainable

“Queued” combines several very different situations:

An explanation turns placement from scheduler folklore into a reviewable decision. It also separates a temporary capacity wait from an impossible request.

Every wait needs a bound. A deadline, maximum queue age or operator policy should end a job that can no longer produce useful work. Leaving it queued forever hides a configuration error and distorts queue health.

Test placement as a pure decision

Most placement logic can be tested without starting real work. Given a job, a set of worker snapshots and active reservations, ask for the eligible set, the chosen worker and the reasons others were excluded.

Include uncomfortable cases:

1. every worker is missing a different requirement;
2. capacity changes between selection and claim;
3. two schedulers race for one exclusive resource;
4. a worker turns suspect after selection;
5. a high-priority job arrives while lower-priority work is waiting;
6. the deadline passes with no eligible worker.

The scheduler’s job is not to keep every machine busy at any cost. It is to place work only where the declared requirements, current evidence and concurrency rules say it can run safely—and to explain when no such place exists.