Census: the claims-on update gate and recover's generation census

Status: PROPOSED. Every row carries rev3=PROPOSED-claims-on-gate-recover. The ruling is asked of the census owner on #280.

The branch is claims-on-gate-recover, from main 3d444a1b. It fixes two defects a deployment harness found on a box running with CARLOSHOSTCLAIM on: (a) the update gate rolled every update back, and its rollback too, because a claims-on edge takes about 22 s to listen and the gate waited 20 s; (b) carlos-system-recover's root generation census refused racily and, since the edge's guard drop-in Requires= the recover unit, kept the edge down.

It adds one census overlay, internal/provision/testdata/plan14-root-reads-claims-on-gate-recover.tsv, loaded last in TestPlan14RootReadCensus, after scan-demand-revival. Every row is authority. There is no legacy row and no frozen declaration, so the seal stands at a21dba12…f0cc262. The internal/host envelope is re-pinned. The record-write census (TestPlan9B8RecordWriteCensus) does not change: no record write is added or moved.

Ruling 6

No-census boxes see no change, and claims-off boxes see none in the gate. A claims-off box that retains a census (claims were on and were turned off) still uses root participation, so it receives the recover settlement changes below exactly as a claims-on box does; that is the existing retained-census rule, not a new one.

• The gate's new window and fail-fast are keyed on CARLOSHOSTCLAIM, read with hostwrite.Enabled from the update and recover units' environment (both load host.env). Claims off, the gate is exactly the 20 s one and reads no systemctl show inside its wait (TestClaimsOffGateIsUnchangedByAnEdgeThatListensAt22s, TestClaimsOffGateWaitsItsFullWindowThroughACrash).
• The recover deferral applies only where root participation is required (a census). A no-census box's refused settlement fails recover as on main (TestRuling6NoCensusRecoverIsUnchanged).
• The generation census itself still runs only where it ran before: it is reached only through the ledger's launch verifier, which a no-census box never consults.

The units

(a) The update gate

| unit | change | executing identity | disposition | |---|---|---|---| | internal/host/systemapply.go RunSystemApply | re-pinned, no read of its own: picks the gate window through systemApplyGate (CARLOSHOSTCLAIM by hostwrite.Enabled) | root apply and recovery, uid 0 | authority, D,F,J4, unchanged |

The gate (systemApplier.gate, gateCheck, gateFailFast, systemGateWaitFor) reads only systemctl is-active and systemctl show answers it already read, and has no row.

(b) Recover's generation census

| unit | change | executing identity | disposition | |---|---|---|---| | internal/host/guardtransaction.go systemApplier.checkGuardRecovery | re-pinned: its reads are unchanged; the two no-journal settlements call settleWithoutJournal, which on a census box wraps a refused controllersRecovered in errSettlementDeferred | root system update and recovery, uid 0 | authority, D, unchanged | | internal/host/systemapply.go systemApplier.Run | re-pinned: its lock-file read is unchanged; under --recover a deferred settlement is logged and not fatal, and is not tried again at the end of the run | root apply and recovery, uid 0 | authority, D,F,J4, unchanged | | internal/host/systemparticipation.go systemApplier.checkUnarmedParticipationRecovery | re-pinned, no read of its own: with no journal open its settlement goes through settleWithoutJournal | root system update and recovery, uid 0 | authority, G,F, unchanged | | internal/host/rootgeneration_kernel_linux.go generationKernel.absent | re-pinned: same reads; a task refusal is typed (generationTaskRefusal, same text) so the retry can tell task-level refusals from structural ones | root census, uid 0 | authority, E2, unchanged | | internal/host/rootgeneration_kernel_linux.go generationKernel.task | re-pinned: same reads; a start time that moves mid-inspection refuses as errGenerationTaskReplaced (still os.ErrPermission); every reference check still runs on every accepted service task | root census, uid 0 | authority, E2, unchanged | | internal/host/rootgeneration_census_linux.go generationCensusRetry | re-pinned, no read of its own: whole passes; every pass and its blocking reads run under one 20 s deadline, and a pass that ends past it refuses; task-level refusals a later pass can clear retry with backoff from 250 ms inside it; reference, namespace, mount and structural refusals refuse at once | root census, uid 0 | authority, E2, unchanged | | internal/host/rootgeneration_services_linux.go generationServiceReader | re-pinned, now a binding with no read: the production generationServiceEnv | root census, uid 0 | authority, E2, unchanged | | internal/host/rootgeneration_services_linux.go generationServiceEnv.reader | new: generationServiceReader's body. The exe Readlink moves first, for diagnostics only. Every refusal names check, unit, cgroup and exe. A non-root task, or a root task of a unit with a selected image (the guarded edge), whose exe reads <path> (deleted) goes to deletedProgram | root census, uid 0 | authority, E2 (its caller's) | | internal/host/rootgeneration_services_linux.go generationServiceShow | new: the systemctl show moved out unchanged | root census, uid 0 | authority, E2 (its caller's) | | internal/host/rootgeneration_services_linux.go generationServiceTrust | new: the unit-file EvalSymlinks and OpenTrustedFile moved out unchanged | root census, uid 0 | authority, E2 (its caller's) | | internal/host/rootgeneration_services_linux.go generationServiceEnv.deletedProgram | new root read: opens a service task's image through /proc/<pid>/exe, Fstats it and hashes it | root census, uid 0, in carlos-system-recover, carlos-system-update, instance-control's start and the guard-exec prelude | authority, F,J4 |

Why the deleted-exe read is F,J4

deletedProgram makes the proof the guard makes before it execs a binary, on the same kind of input. The guard's reads are F,J4 root-input: hostfloor OpenProtected, Disk.VerifyFile and ExecutingHash (which opens /proc/self/exe and hashes it), and in this package readRootFloorBinary. deletedProgram opens the running image of another task the same way, requires a regular file owned by root, with no links left and no group or other write, hashes at most 512 MiB of it, and accepts only a digest equal to the signed router build of <opt>/carlos or <opt>/carlos.prev, which readRootFloorBinary proves from the identity cache the box holds. Classifying it at E2 with its caller would understate what it decides: a match is what lets an unselected inode count as the unit's program.

It is reached for a non-root task, and for a root task of a unit with a selected image (the guarded edge, whose selected program is <opt>/carlos), when the unit selects <opt>/carlos and the exe link reads <opt>/carlos (deleted). A root task of a unit with no selected image keeps main's path walk, which a deleted exe fails, so the strict root scan applies to it (TestRecoverCensusDeletedExeOfARootTask). Anything else refuses as before. Acceptance only lets the census go on to the reference checks: exe, cwd, root, descriptors, maps and mounts are still compared against the retired generations (TestRecoverCensusStillChecksReferencesOfAnAcceptedDeletedExe).

Fix round 1 (Astra's review of 8eed5c19)

| unit | change | disposition | |---|---|---| | systemApplier.checkGuardRecovery | every no-journal settlement (controllers census, completion or withdrawal, the census again inside EndParticipation) in the marker, no-marker and already-completed branches goes through settleWithoutJournal, which now defers only a result that is a census refusal and nothing else (onlyCensusRefusal); an unfinished transaction's marker is removed inside the settlement, after its census. Round 0's deferral of any controller failure is withdrawn | authority, D, re-pinned | | systemApplier.checkUnarmedParticipationRecovery | the same census-only rule | authority, G,F, re-pinned | | productionGenerationKernelCensus | every refusal typed generationCensusRefusal; each pass's lookup is built on the census deadline context and wires its recheck | authority, E2, re-pinned at main's class | | productionGenerationAbsence, generationRetiredController, type generationKernel | wire or carry the lookup's recheck | authority, E2, re-pinned at main's class | | generationKernel.task | holds a trusted task to its proven image: recheck before each use of the trust and at the end; no recheck wired refuses every trusted task | authority, E2, re-pinned | | generationCensusRetry | one 20 s deadline shared by every pass and its blocking reads; a pass ending past it refuses | authority, E2, re-pinned | | generationServiceReader, generationServiceEnv.reader, generationServiceEnv.deletedProgram | the lookup returns classify and recheck; the hashed descriptor is kept to bind the task; the hash stops at the deadline | re-pinned at their classes | | generationServiceEnv.recheck (new root read) | re-opens /proc/<pid>/exe and requires the hashed inode (device, inode, mount ID, file handle), still root-owned, unlinked, not group or other writable | authority, F,J4, as deletedProgram | | generationExeIdentity (new) | Fstat, Statx and name_to_handle_at of an open image descriptor | authority, F,J4, as its callers | | generationContextReader.Read (new) | the hash's reader over the already open image; stops at the deadline | authority, F,J4, as its caller |

No execution exclusion exists for a service task's own exec: root's launch exclusion (participationRelease, Ledger.ExcludeLaunches) covers claim launches only. The rechecks are sampled identity checks at observation points (before each use of the trust and at the end of the inspection), not execution exclusion: a replacement visible at a recheck refuses, but a task that execs an unsigned helper after the final /proc/<pid>/exe open keeps its classification until the next pass. Main's live-exe path (the selected-program walk and plain non-root services) has the same race after its one observation; this branch extends that exposure to the deleted, root-owned, protected inodes whose bytes match the signed current or previous router build, which main refused outright. It does not admit an unsigned helper on its own. The binding applies only to deleted-image acceptance; live-image rechecks return at once. Main's path is unchanged here and needs a ruling. Still no legacy row, so the seal stands; internal/host's envelope is re-pinned again.

Fix round 2 (Astra's round 2 on ec057b9b)

• Post-journal settlement (finding 1a). closeGuardJournal durably removes the journal, then settles (endGuard, clearUpdateBegun). That settlement ran unwrapped, so a census refusal there failed recover and the edge's dependency job with the files whole and the journal gone: the Run 11b/11d shape. It now goes through settleWithoutJournal, forward and backward, guarded and unarmed, and Run under --recover consumes the deferral from recoverIfNeeded without keeping the fatal error or retrying settlement. Non-census failures stay fatal.
• Acquisition barrier (finding 1b). With no update record (a prepared transaction never begun), EndParticipation's BeginUpdate withdrew the record it wrote when its census refused, so a deferral left Acquire open. settleWithoutJournal now calls Ledger.HoldUpdate first, under the transaction lock: root's update record kept, or written under the transaction's id, with no census of its own. Acquire reads the ledger census and update.json, so it stays refused while recover defers. A barrier that cannot be written leaves the refusal fatal. An already-completed transaction with no record left replays at once and runs no census. Ledger.HoldUpdate is a new unit at its siblings' class (D).
• The signed-image proof under the deadline (finding 3). readRootFloorBinary becomes a wrapper (no read) over readRootFloorBinaryContext, whose hash stops between reads and whose identity scan stops before each identity once the census context ends. generationContextReader moves beside it. The two helpers are witnessed directly; the lookup is witnessed to hand the deadline to the proof; the plumbing from readRootFloorBinaryContext into the helpers is not witnessed by a non-root test (the proof needs root-owned files), so a mutant that drops the context inside it survives.
• Prose (finding 4). The rows and this note now describe the rechecks as sampled identity checks, name the final-open race as shared with main's live-exe path, state the deadline's coverage and its gaps, and no longer say the three restored tests expect exit 0.

Fix round 3 (Astra's round 3 on d16cb5f0)

• The legacy marker's settlement (finding 1b, unarmed). With root participation, no armed floor, a legacy update-begun.json and neither an update record nor a journal, checkUnarmedParticipationRecovery returns without settling and the run's end settled the marker through clearUpdateBegun directly, logging any error: its BeginUpdate was refused by the census, which withdrew the record it wrote, so recover exited 0 with acquisitions open. settleUpdateBegun now settles through settleWithoutJournal with the recover-legacy barrier, and settleMarker decides what the outcome means for the run: a deferral under --recover is logged and left to a later run; a barrier that could not be written (errAcquisitionBarrier) joins the run's error in every mode; every other marker failure is logged as on main. TestRecoverHoldsAcquisitionsForALegacyMarker, TestRecoverFailsALegacyMarkerWhoseBarrierCannotBeWritten.
• Prose (findings 2 and 3 of round 3). The deadline is described as cooperative cancellation at checkpoints with the uncancelled work named, and the boot-delay figures as estimates; every row that said a trusted task is "held to" its image now says it is rechecked at observation points and names what a recheck cannot see.

Fix round 4 (Astra's round 4 on 2a823aef)

• A barrier prerequisite failure is a barrier failure. In settleWithoutJournal, when ParticipationRequired itself failed (the store did not open, a retained census did not read) the original census refusal was returned bare, so the legacy marker's settlement logged it and recover exited 0 without a barrier. It now returns errAcquisitionBarrier with both errors kept, which settleMarker makes fatal in every mode; a box where participation is not required still falls back as before. TestRecoverFailsALegacyMarkerWhoseBarrierPrerequisiteFails injects the failure after BeginUpdate has withdrawn its record.
• Ruling 6 wording qualified: a claims-off box with a retained census still uses participation and receives the settlement changes.

The census window and the worst-case boot delay

Each call of root's kernel census runs under one deadline, 20 s from its start (generationCensusRetryFor). Every pass gets a context that ends at the deadline. The deadline is cooperative cancellation at the implemented checkpoints, plus the rejection of late results: systemctl show runs through exec.CommandContext; the deleted-image hash checks the context between reads; (fix round 2) the signed-image proof, readRootFloorBinaryContext, checks it between reads of <opt>/carlos and <opt>/carlos.prev and before each cached identity. A pass that ends past the deadline refuses even if it succeeded, and no pass starts after it. Between checkpoints the work is not cancelled: the opens and metadata operations, the identity cache's directory enumeration (Names, a ReadDir), a whole identity record's read (io.ReadAll), a systemctl show already started, one Read of an image, and the versions-tree walk and mount checks before the first pass (bounded at 8192 entries). The figures below are therefore estimates, not bounds.

On the recover path, with the census refusing every time, the edge's start waits at most for, in order:

1. the guard-exec recover prelude's census (rootFloorDowngradeIdle), when a moves floor is armed: 20 s;
2. recover's first controllers census: 20 s;
3. root's pending-controller recovery (recoverRootExecution), which runs under its own 30 s context, plus two cleanups of up to 30 s each;
4. recover's second controllers census: 20 s.

Then recover defers and exits 0. That is about 90 s, and about 150 s when both cleanups run to their bounds, as estimates: each 20 s window can overrun by the uncancelled work named above. The recover unit's TimeoutStartSec is 10 min. instance-control's start runs its own census under the same 20 s window.

Tests named in the rows

• (a): TestSystemGateWaitIsKeyedOnClaimsMode, TestSystemApplyReadsClaimsModeFromTheUnitEnvironment, TestClaimsOnGatePassesAnEdgeThatListensAt22s, TestClaimsOnGateWaitsUpTo120s, TestClaimsOnGateFailsFastOnACrashRestart, TestClaimsOnGateFailsFastOnAFailedOrInactiveEdge, TestClaimsOnRollbackGateUsesTheSameRule, TestClaimsOnRollbackGateFailsFastToo, and the two claims-off tests above.
• (b) fix 1: TestRecoverDefersACensusRefusalWithNoJournalOpen, TestRecoverFailsACensusRefusalWithAJournalOpen, TestRecoverDefersACensusRefusalWithoutAnArmedFloor, TestApplyStillFailsACensusRefusalWithNoJournalOpen, TestRuling6NoCensusRecoverIsUnchanged; TestGuardRecoveryRefusesControllersAfterBegin, TestPlan9M8GuardRecoveryRecoversPendingWithoutJournal and TestSystemGuardRegionHAKeepsFailedCompletionEvidence are unchanged from main: deferral is census-only (fix round 1), so a live or unreadable controller and a failed completion still fail as they did.
• (b) fix 2: TestRecoverCensusAcceptsADeletedExeOfTheCurrentOrPreviousBuild, TestRecoverCensusAcceptsTheEdgesDeletedExeInSupervise, TestRecoverCensusRefusesOtherDeletedExes, TestRecoverCensusRefusesADeletedExeNotOwnedByRoot, TestRecoverCensusStillChecksReferencesOfAnAcceptedDeletedExe.
• (b) fix 3: TestRecoverCensusRetriesATransientRefusalWithinTheWindow, TestRecoverCensusRetriesOnlyTaskLevelTransients, TestPlan14GenerationCensusRetriesWholeObservation (unchanged).
• (b) fix 4: TestRecoverCensusRefusalsNameCheckUnitCgroupAndExe.

The census tests drive the production service lookup over a fake /proc, replacing only systemctl, the unit files' ownership check, the signed-binary proof and the owner uid, which a non-root test cannot provide. No root-runner test was run for this branch.