Fleet job queues

Engineering account · Reviewed 2026-10-23

Suppose you have several computers available and a software project to test. You want a computer with room to do the work, and a clear result when it finishes. Having the computers is only part of the problem: something still has to decide which work goes where.

A fleet is the group of computers. A job queue is their shared to-do list. In Corvis, a runner, the program responsible for carrying out work, claims a job that its computer is equipped for and has room to run. For software checks, the result is attached to the exact version of the code that was checked.

This is a first-party account of the Corvis queue design, reviewed against implementation sources. It follows software checks, then explains the boundary with agent sessions. Public references support the underlying techniques; they are not independent verification of the Corvis implementation. The diagrams are explanatory, not performance measurements.

Getting work to a suitable computer

The earlier dispatcher took checks through a shared runner while other machines could sit idle. Long checks and session-management operations competed for that path. The response was to give different kinds of work separate lanes and let node-specific runners claim checks independently. A lane is a category served by its own runner path.

Parallel execution also needs a definition of suitable. A machine can be available but lack a required tool, platform, or enough free memory. Corvis filters candidates by lane, capabilities, any requested node, and the projects that fit the runner's available resources before handing it work.

This ordering came from a concrete failure: a runner claimed an oversized job and then refused it, even though another machine could have run it. Checking fit before ownership avoids that particular dead end. Lanes separate dispatch responsibilities; they do not create additional CPU or memory.

Follow one job

Imagine the browser check in Figure A. Its path through the design is:

Claim and start are separate operations. On an initial claim, the database records ownership before the execution node and run identifier are reported. A process can also start before its start report reaches the database. That gap is why recovery must inspect the execution record instead of treating every claimed job as never launched.

Concurrent claims

The next-job claim combines selection and update using FOR UPDATE SKIP LOCKED. A transaction can skip a candidate row another transaction has locked and look for another. PostgreSQL documents this use for queue consumers, while warning that skipped locks produce an inconsistent view. An empty claim result therefore need not mean the queue is empty. This option applies to row locks; it does not eliminate every kind of waiting.¹

Priority and fairness

Corvis's next-job ordering puts release-verification service jobs first, designated release-gate projects next, and other jobs after them. Older jobs come first within those groups. Priority is derived from the service identity and project rather than a caller-selected priority number.

That is a priority policy, not a per-project fair-share scheduler. The reviewed ordering has no age-based promotion across priority groups. Sustained higher-priority arrivals can delay lower-priority work. Queue wait by project and priority class is the measurement needed to assess how much this matters in practice.

When a worker disappears

A lease is time-limited ownership. The queue can reassign eligible claimed or running jobs after a finite lease expires. Heartbeats extend leases. Claims without an expiry are not recovered by this timeout rule.

The runner also has active recovery paths:

Why cancellation is not fencing

Consider this possible sequence. Worker A starts a job, loses database connectivity, and keeps executing. Its lease expires. Worker B claims the job and begins another execution. A later reconciliation may stop A, but there can be an overlap before that succeeds.

Fencing addresses a different boundary: the resource receiving a write rejects requests from an obsolete owner. A common design assigns each new owner a monotonically increasing token and makes the receiving service reject older tokens. The service must enforce it; generating a token alone is insufficient. Hazelcast's FencedLock documentation illustrates this stale-owner problem.²

The reviewed Corvis dispatch path uses claimant identity and lifecycle checks, plus process reconciliation. This review did not establish an end-to-end fencing contract enforced by every external resource a job could modify. It therefore does not establish that general side-effecting jobs are safe to run under overlapping ownership.

Idempotency answers another question: can the same intended operation be retried without an additional effect? A stable operation identifier lets the receiving service recognize a retry. Recording that identifier must be coordinated with performing the effect; otherwise a crash between them can still produce duplication. AWS's public account describes this API contract.³

A job identifier or commit hash is not, by itself, that contract. Even a test can write to a shared database or call an external service. Whether duplicate execution is acceptable depends on the job's actual effects and isolation.

Time and retry boundaries

The explicit lost-run retry budget and lease expiry are different mechanisms. The reviewed expiry predicate makes an expired job eligible without applying the lost-run event counter. A bounded retry in one recovery path must not be described as a universal attempt limit.

Lease timing also depends on the database clock expression: PostgreSQL's now() is fixed at transaction start, while clock_timestamp() reads the advancing clock. A lease calculated using now() can spend some of its lifetime inside the transaction before the caller receives it.⁴

What a passing check establishes

The request names a source commit, and the runner exports that revision into a staging tree. An early exported-tree check failed because its repository index had not been populated. A source snapshot still has to satisfy the tools' environment assumptions.

The result reader selects evidence by project and commit. It distinguishes a live job, a completed verdict, and work refused or cancelled. For task-linked jobs, terminal outcome writers also record the task-thread notification in the closing database transaction. That joins result publication to the durable outcome; it does not make external job effects transactional.

The useful conclusion is: this check reported this outcome for this project at this revision. Revision pinning identifies the main source input. It does not alone pin companion repositories, packages, toolchains, service state, hardware, or external responses. Fully reproducible execution needs those inputs controlled and recorded too.

Where agent sessions fit

The queue is already used for more than software checks: separate lanes carry control and session-management operations. Corvis also has task handoff and agent-session resume machinery outside the software-check runner. Treating the entire platform as a test queue would miss those layers.

Reassigning a queue job is not equivalent to recovering an agent's interrupted computation. A resume operation depends on the session state and runtime it targets. A fresh handoff uses the task context. Neither operation, by its presence alone, proves that arbitrary workers are disposable or that every external action can be replayed safely.

For extending this pattern to longer workflows, the concrete questions are where progress is saved, which effects can be retried, how stale owners are rejected, and what evidence proves an old execution has stopped. Those requirements belong to each workload, not just to the queue table.

Evidence and measurements

This page describes mechanisms and their limits. It does not publish a throughput benchmark, availability percentage, or measured recovery-time distribution. Figures A and B are conceptual illustrations.

A useful performance study would publish its observation window, job mix, machine capacity, and treatment of cancellations and retries. It would then report queue wait by project and priority, preparation time separately from execution time, completed jobs per interval, recovery delay, and duplicate-execution incidents. Without those definitions, a completion count or a busy-machine screenshot cannot establish fleet reliability.

All citations below lead to public primary material. They explain the general mechanisms discussed here. Corvis-specific behavior is this page's first-party engineering account, not an independently reproduced benchmark or a claim that the public references audit Corvis.