| 1 | # Census: the claims-on update gate and recover's generation census |
| 2 | |
| 3 | Status: PROPOSED. Every row carries `rev3=PROPOSED-claims-on-gate-recover`. |
| 4 | The ruling is asked of the census owner on #280. |
| 5 | |
| 6 | The branch is `claims-on-gate-recover`, from main 3d444a1b. It fixes two |
| 7 | defects a deployment harness found on a box running with CARLOS_HOST_CLAIM |
| 8 | on: (a) the update gate rolled every update back, and its rollback too, |
| 9 | because a claims-on edge takes about 22 s to listen and the gate waited 20 s; |
| 10 | (b) carlos-system-recover's root generation census refused racily and, since |
| 11 | the edge's guard drop-in `Requires=` the recover unit, kept the edge down. |
| 12 | |
| 13 | It adds one census overlay, |
| 14 | `internal/provision/testdata/plan14-root-reads-claims-on-gate-recover.tsv`, |
| 15 | loaded last in `TestPlan14RootReadCensus`, after scan-demand-revival. Every |
| 16 | row is authority. There is no legacy row and no frozen declaration, so the |
| 17 | seal stands at `a21dba12…f0cc262`. The `internal/host` envelope is re-pinned. |
| 18 | The record-write census (`TestPlan9B8RecordWriteCensus`) does not change: no |
| 19 | record write is added or moved. |
| 20 | |
| 21 | ## Ruling 6 |
| 22 | |
| 23 | No-census boxes see no change, and claims-off boxes see none in the gate. A |
| 24 | claims-off box that retains a census (claims were on and were turned off) |
| 25 | still uses root participation, so it receives the recover settlement |
| 26 | changes below exactly as a claims-on box does; that is the existing |
| 27 | retained-census rule, not a new one. |
| 28 | |
| 29 | - The gate's new window and fail-fast are keyed on CARLOS_HOST_CLAIM, read |
| 30 | with `hostwrite.Enabled` from the update and recover units' environment |
| 31 | (both load host.env). Claims off, the gate is exactly the 20 s one and |
| 32 | reads no `systemctl show` inside its wait |
| 33 | (`TestClaimsOffGateIsUnchangedByAnEdgeThatListensAt22s`, |
| 34 | `TestClaimsOffGateWaitsItsFullWindowThroughACrash`). |
| 35 | - The recover deferral applies only where root participation is required |
| 36 | (a census). A no-census box's refused settlement fails recover as on main |
| 37 | (`TestRuling6NoCensusRecoverIsUnchanged`). |
| 38 | - The generation census itself still runs only where it ran before: it is |
| 39 | reached only through the ledger's launch verifier, which a no-census box |
| 40 | never consults. |
| 41 | |
| 42 | ## The units |
| 43 | |
| 44 | ### (a) The update gate |
| 45 | |
| 46 | | unit | change | executing identity | disposition | |
| 47 | |---|---|---|---| |
| 48 | | `internal/host/systemapply.go` `RunSystemApply` | re-pinned, no read of its own: picks the gate window through `systemApplyGate` (CARLOS_HOST_CLAIM by `hostwrite.Enabled`) | root apply and recovery, uid 0 | authority, D,F,J4, unchanged | |
| 49 | |
| 50 | The gate (`systemApplier.gate`, `gateCheck`, `gateFailFast`, `systemGateWaitFor`) |
| 51 | reads only `systemctl is-active` and `systemctl show` answers it already read, |
| 52 | and has no row. |
| 53 | |
| 54 | ### (b) Recover's generation census |
| 55 | |
| 56 | | unit | change | executing identity | disposition | |
| 57 | |---|---|---|---| |
| 58 | | `internal/host/guardtransaction.go` `systemApplier.checkGuardRecovery` | re-pinned: its reads are unchanged; the two no-journal settlements call `settleWithoutJournal`, which on a census box wraps a refused `controllersRecovered` in `errSettlementDeferred` | root system update and recovery, uid 0 | authority, D, unchanged | |
| 59 | | `internal/host/systemapply.go` `systemApplier.Run` | re-pinned: its lock-file read is unchanged; under `--recover` a deferred settlement is logged and not fatal, and is not tried again at the end of the run | root apply and recovery, uid 0 | authority, D,F,J4, unchanged | |
| 60 | | `internal/host/systemparticipation.go` `systemApplier.checkUnarmedParticipationRecovery` | re-pinned, no read of its own: with no journal open its settlement goes through `settleWithoutJournal` | root system update and recovery, uid 0 | authority, G,F, unchanged | |
| 61 | | `internal/host/rootgeneration_kernel_linux.go` `generationKernel.absent` | re-pinned: same reads; a task refusal is typed (`generationTaskRefusal`, same text) so the retry can tell task-level refusals from structural ones | root census, uid 0 | authority, E2, unchanged | |
| 62 | | `internal/host/rootgeneration_kernel_linux.go` `generationKernel.task` | re-pinned: same reads; a start time that moves mid-inspection refuses as `errGenerationTaskReplaced` (still `os.ErrPermission`); every reference check still runs on every accepted service task | root census, uid 0 | authority, E2, unchanged | |
| 63 | | `internal/host/rootgeneration_census_linux.go` `generationCensusRetry` | re-pinned, no read of its own: whole passes; every pass and its blocking reads run under one 20 s deadline, and a pass that ends past it refuses; task-level refusals a later pass can clear retry with backoff from 250 ms inside it; reference, namespace, mount and structural refusals refuse at once | root census, uid 0 | authority, E2, unchanged | |
| 64 | | `internal/host/rootgeneration_services_linux.go` `generationServiceReader` | re-pinned, now a binding with no read: the production `generationServiceEnv` | root census, uid 0 | authority, E2, unchanged | |
| 65 | | `internal/host/rootgeneration_services_linux.go` `generationServiceEnv.reader` | new: `generationServiceReader`'s body. The exe `Readlink` moves first, for diagnostics only. Every refusal names check, unit, cgroup and exe. A non-root task, or a root task of a unit with a selected image (the guarded edge), whose exe reads `<path> (deleted)` goes to `deletedProgram` | root census, uid 0 | authority, E2 (its caller's) | |
| 66 | | `internal/host/rootgeneration_services_linux.go` `generationServiceShow` | new: the `systemctl show` moved out unchanged | root census, uid 0 | authority, E2 (its caller's) | |
| 67 | | `internal/host/rootgeneration_services_linux.go` `generationServiceTrust` | new: the unit-file `EvalSymlinks` and `OpenTrustedFile` moved out unchanged | root census, uid 0 | authority, E2 (its caller's) | |
| 68 | | `internal/host/rootgeneration_services_linux.go` `generationServiceEnv.deletedProgram` | new root read: opens a service task's image through `/proc/<pid>/exe`, `Fstat`s it and hashes it | root census, uid 0, in carlos-system-recover, carlos-system-update, instance-control's start and the guard-exec prelude | authority, F,J4 | |
| 69 | |
| 70 | ### Why the deleted-exe read is F,J4 |
| 71 | |
| 72 | `deletedProgram` makes the proof the guard makes before it execs a binary, |
| 73 | on the same kind of input. The guard's reads are F,J4 root-input: |
| 74 | `hostfloor` `OpenProtected`, `Disk.VerifyFile` and `ExecutingHash` (which |
| 75 | opens `/proc/self/exe` and hashes it), and in this package |
| 76 | `readRootFloorBinary`. `deletedProgram` opens the running image of another |
| 77 | task the same way, requires a regular file owned by root, with no links left |
| 78 | and no group or other write, hashes at most 512 MiB of it, and accepts only a |
| 79 | digest equal to the signed router build of `<opt>/carlos` or |
| 80 | `<opt>/carlos.prev`, which `readRootFloorBinary` proves from the identity |
| 81 | cache the box holds. Classifying it at E2 with its caller would understate |
| 82 | what it decides: a match is what lets an unselected inode count as the |
| 83 | unit's program. |
| 84 | |
| 85 | It is reached for a non-root task, and for a root task of a unit with a |
| 86 | selected image (the guarded edge, whose selected program is |
| 87 | `<opt>/carlos`), when the unit selects `<opt>/carlos` and the exe link reads |
| 88 | `<opt>/carlos (deleted)`. A root task of a unit with no selected image keeps |
| 89 | main's path walk, which a deleted exe fails, so the strict root scan |
| 90 | applies to it (`TestRecoverCensusDeletedExeOfARootTask`). Anything else |
| 91 | refuses as before. Acceptance only lets the census go on to the |
| 92 | reference checks: exe, cwd, root, descriptors, maps and mounts are still |
| 93 | compared against the retired generations |
| 94 | (`TestRecoverCensusStillChecksReferencesOfAnAcceptedDeletedExe`). |
| 95 | |
| 96 | ## Fix round 1 (Astra's review of 8eed5c19) |
| 97 | |
| 98 | | unit | change | disposition | |
| 99 | |---|---|---| |
| 100 | | `systemApplier.checkGuardRecovery` | every no-journal settlement (controllers census, completion or withdrawal, the census again inside `EndParticipation`) in the marker, no-marker and already-completed branches goes through `settleWithoutJournal`, which now defers only a result that is a census refusal and nothing else (`onlyCensusRefusal`); an unfinished transaction's marker is removed inside the settlement, after its census. Round 0's deferral of any controller failure is withdrawn | authority, D, re-pinned | |
| 101 | | `systemApplier.checkUnarmedParticipationRecovery` | the same census-only rule | authority, G,F, re-pinned | |
| 102 | | `productionGenerationKernelCensus` | every refusal typed `generationCensusRefusal`; each pass's lookup is built on the census deadline context and wires its recheck | authority, E2, re-pinned at main's class | |
| 103 | | `productionGenerationAbsence`, `generationRetiredController`, `type generationKernel` | wire or carry the lookup's recheck | authority, E2, re-pinned at main's class | |
| 104 | | `generationKernel.task` | holds a trusted task to its proven image: recheck before each use of the trust and at the end; no recheck wired refuses every trusted task | authority, E2, re-pinned | |
| 105 | | `generationCensusRetry` | one 20 s deadline shared by every pass and its blocking reads; a pass ending past it refuses | authority, E2, re-pinned | |
| 106 | | `generationServiceReader`, `generationServiceEnv.reader`, `generationServiceEnv.deletedProgram` | the lookup returns classify and recheck; the hashed descriptor is kept to bind the task; the hash stops at the deadline | re-pinned at their classes | |
| 107 | | `generationServiceEnv.recheck` (new root read) | re-opens `/proc/<pid>/exe` and requires the hashed inode (device, inode, mount ID, file handle), still root-owned, unlinked, not group or other writable | authority, F,J4, as `deletedProgram` | |
| 108 | | `generationExeIdentity` (new) | `Fstat`, `Statx` and `name_to_handle_at` of an open image descriptor | authority, F,J4, as its callers | |
| 109 | | `generationContextReader.Read` (new) | the hash's reader over the already open image; stops at the deadline | authority, F,J4, as its caller | |
| 110 | |
| 111 | No execution exclusion exists for a service task's own `exec`: root's |
| 112 | launch exclusion (`participationRelease`, `Ledger.ExcludeLaunches`) covers |
| 113 | claim launches only. The rechecks are sampled identity checks at observation |
| 114 | points (before each use of the trust and at the end of the inspection), not |
| 115 | execution exclusion: a replacement visible at a recheck refuses, but a task |
| 116 | that execs an unsigned helper after the final `/proc/<pid>/exe` open keeps |
| 117 | its classification until the next pass. Main's live-exe path (the |
| 118 | selected-program walk and plain non-root services) has the same race after |
| 119 | its one observation; this branch extends that exposure to the deleted, |
| 120 | root-owned, protected inodes whose bytes match the signed current or |
| 121 | previous router build, which main refused outright. It does not admit an |
| 122 | unsigned helper on its own. The binding applies only to deleted-image |
| 123 | acceptance; live-image rechecks return at once. Main's path is unchanged |
| 124 | here and needs a ruling. Still no legacy row, so the |
| 125 | seal stands; `internal/host`'s envelope is re-pinned again. |
| 126 | |
| 127 | ## Fix round 2 (Astra's round 2 on ec057b9b) |
| 128 | |
| 129 | - **Post-journal settlement (finding 1a).** `closeGuardJournal` durably |
| 130 | removes the journal, then settles (`endGuard`, `clearUpdateBegun`). That |
| 131 | settlement ran unwrapped, so a census refusal there failed recover and the |
| 132 | edge's dependency job with the files whole and the journal gone: the Run |
| 133 | 11b/11d shape. It now goes through `settleWithoutJournal`, forward and |
| 134 | backward, guarded and unarmed, and `Run` under `--recover` consumes the |
| 135 | deferral from `recoverIfNeeded` without keeping the fatal error or |
| 136 | retrying settlement. Non-census failures stay fatal. |
| 137 | - **Acquisition barrier (finding 1b).** With no update record (a prepared |
| 138 | transaction never begun), `EndParticipation`'s `BeginUpdate` withdrew the |
| 139 | record it wrote when its census refused, so a deferral left `Acquire` |
| 140 | open. `settleWithoutJournal` now calls `Ledger.HoldUpdate` first, under the |
| 141 | transaction lock: root's update record kept, or written under the |
| 142 | transaction's id, with no census of its own. `Acquire` reads the ledger |
| 143 | census and `update.json`, so it stays refused while recover defers. A |
| 144 | barrier that cannot be written leaves the refusal fatal. An |
| 145 | already-completed transaction with no record left replays at once and |
| 146 | runs no census. `Ledger.HoldUpdate` is a new unit at its siblings' class |
| 147 | (D). |
| 148 | - **The signed-image proof under the deadline (finding 3).** |
| 149 | `readRootFloorBinary` becomes a wrapper (no read) over |
| 150 | `readRootFloorBinaryContext`, whose hash stops between reads and whose |
| 151 | identity scan stops before each identity once the census context ends. |
| 152 | `generationContextReader` moves beside it. The two helpers are witnessed |
| 153 | directly; the lookup is witnessed to hand the deadline to the proof; the |
| 154 | plumbing from `readRootFloorBinaryContext` into the helpers is not |
| 155 | witnessed by a non-root test (the proof needs root-owned files), so a |
| 156 | mutant that drops the context inside it survives. |
| 157 | - **Prose (finding 4).** The rows and this note now describe the rechecks |
| 158 | as sampled identity checks, name the final-open race as shared with main's |
| 159 | live-exe path, state the deadline's coverage and its gaps, and no longer |
| 160 | say the three restored tests expect exit 0. |
| 161 | |
| 162 | ## Fix round 3 (Astra's round 3 on d16cb5f0) |
| 163 | |
| 164 | - **The legacy marker's settlement (finding 1b, unarmed).** With root |
| 165 | participation, no armed floor, a legacy `update-begun.json` and neither an |
| 166 | update record nor a journal, `checkUnarmedParticipationRecovery` returns |
| 167 | without settling and the run's end settled the marker through |
| 168 | `clearUpdateBegun` directly, logging any error: its `BeginUpdate` was |
| 169 | refused by the census, which withdrew the record it wrote, so recover |
| 170 | exited 0 with acquisitions open. `settleUpdateBegun` now settles through |
| 171 | `settleWithoutJournal` with the `recover-legacy` barrier, and `settleMarker` |
| 172 | decides what the outcome means for the run: a deferral under `--recover` |
| 173 | is logged and left to a later run; a barrier that could not be written |
| 174 | (`errAcquisitionBarrier`) joins the run's error in every mode; every |
| 175 | other marker failure is logged as on main. |
| 176 | `TestRecoverHoldsAcquisitionsForALegacyMarker`, |
| 177 | `TestRecoverFailsALegacyMarkerWhoseBarrierCannotBeWritten`. |
| 178 | - **Prose (findings 2 and 3 of round 3).** The deadline is described as |
| 179 | cooperative cancellation at checkpoints with the uncancelled work named, |
| 180 | and the boot-delay figures as estimates; every row that said a trusted |
| 181 | task is "held to" its image now says it is rechecked at observation points |
| 182 | and names what a recheck cannot see. |
| 183 | |
| 184 | ## Fix round 4 (Astra's round 4 on 2a823aef) |
| 185 | |
| 186 | - **A barrier prerequisite failure is a barrier failure.** In |
| 187 | `settleWithoutJournal`, when `ParticipationRequired` itself failed (the |
| 188 | store did not open, a retained census did not read) the original census |
| 189 | refusal was returned bare, so the legacy marker's settlement logged it and |
| 190 | recover exited 0 without a barrier. It now returns `errAcquisitionBarrier` |
| 191 | with both errors kept, which `settleMarker` makes fatal in every mode; a |
| 192 | box where participation is not required still falls back as before. |
| 193 | `TestRecoverFailsALegacyMarkerWhoseBarrierPrerequisiteFails` injects the |
| 194 | failure after `BeginUpdate` has withdrawn its record. |
| 195 | - **Ruling 6 wording** qualified: a claims-off box with a retained census |
| 196 | still uses participation and receives the settlement changes. |
| 197 | |
| 198 | ## The census window and the worst-case boot delay |
| 199 | |
| 200 | Each call of root's kernel census runs under one deadline, 20 s from its |
| 201 | start (`generationCensusRetryFor`). Every pass gets a context that ends at |
| 202 | the deadline. The deadline is cooperative cancellation at the implemented |
| 203 | checkpoints, plus the rejection of late results: `systemctl show` runs |
| 204 | through `exec.CommandContext`; the deleted-image hash checks the context |
| 205 | between reads; (fix round 2) the signed-image proof, |
| 206 | `readRootFloorBinaryContext`, checks it between reads of `<opt>/carlos` and |
| 207 | `<opt>/carlos.prev` and before each cached identity. A pass that ends past |
| 208 | the deadline refuses even if it succeeded, and no pass starts after it. |
| 209 | Between checkpoints the work is not cancelled: the opens and metadata |
| 210 | operations, the identity cache's directory enumeration (`Names`, a |
| 211 | `ReadDir`), a whole identity record's read (`io.ReadAll`), a `systemctl |
| 212 | show` already started, one `Read` of an image, and the versions-tree walk and |
| 213 | mount checks before the first pass (bounded at 8192 entries). The figures |
| 214 | below are therefore estimates, not bounds. |
| 215 | |
| 216 | On the recover path, with the census refusing every time, the edge's start |
| 217 | waits at most for, in order: |
| 218 | |
| 219 | 1. the guard-exec recover prelude's census (`rootFloorDowngradeIdle`), when |
| 220 | a moves floor is armed: 20 s; |
| 221 | 2. recover's first controllers census: 20 s; |
| 222 | 3. root's pending-controller recovery (`recoverRootExecution`), which runs |
| 223 | under its own 30 s context, plus two cleanups of up to 30 s each; |
| 224 | 4. recover's second controllers census: 20 s. |
| 225 | |
| 226 | Then recover defers and exits 0. That is about 90 s, and about 150 s when |
| 227 | both cleanups run to their bounds, as estimates: each 20 s window can |
| 228 | overrun by the uncancelled work named above. The recover unit's TimeoutStartSec is |
| 229 | 10 min. instance-control's start runs its own census under the same 20 s |
| 230 | window. |
| 231 | |
| 232 | ## Tests named in the rows |
| 233 | |
| 234 | - (a): `TestSystemGateWaitIsKeyedOnClaimsMode`, |
| 235 | `TestSystemApplyReadsClaimsModeFromTheUnitEnvironment`, |
| 236 | `TestClaimsOnGatePassesAnEdgeThatListensAt22s`, |
| 237 | `TestClaimsOnGateWaitsUpTo120s`, `TestClaimsOnGateFailsFastOnACrashRestart`, |
| 238 | `TestClaimsOnGateFailsFastOnAFailedOrInactiveEdge`, |
| 239 | `TestClaimsOnRollbackGateUsesTheSameRule`, |
| 240 | `TestClaimsOnRollbackGateFailsFastToo`, and the two claims-off tests above. |
| 241 | - (b) fix 1: `TestRecoverDefersACensusRefusalWithNoJournalOpen`, |
| 242 | `TestRecoverFailsACensusRefusalWithAJournalOpen`, |
| 243 | `TestRecoverDefersACensusRefusalWithoutAnArmedFloor`, |
| 244 | `TestApplyStillFailsACensusRefusalWithNoJournalOpen`, |
| 245 | `TestRuling6NoCensusRecoverIsUnchanged`; `TestGuardRecoveryRefusesControllersAfterBegin`, |
| 246 | `TestPlan9M8GuardRecoveryRecoversPendingWithoutJournal` and |
| 247 | `TestSystemGuardRegionHAKeepsFailedCompletionEvidence` are unchanged from |
| 248 | main: deferral is census-only (fix round 1), so a live or unreadable |
| 249 | controller and a failed completion still fail as they did. |
| 250 | - (b) fix 2: `TestRecoverCensusAcceptsADeletedExeOfTheCurrentOrPreviousBuild`, |
| 251 | `TestRecoverCensusAcceptsTheEdgesDeletedExeInSupervise`, |
| 252 | `TestRecoverCensusRefusesOtherDeletedExes`, |
| 253 | `TestRecoverCensusRefusesADeletedExeNotOwnedByRoot`, |
| 254 | `TestRecoverCensusStillChecksReferencesOfAnAcceptedDeletedExe`. |
| 255 | - (b) fix 3: `TestRecoverCensusRetriesATransientRefusalWithinTheWindow`, |
| 256 | `TestRecoverCensusRetriesOnlyTaskLevelTransients`, |
| 257 | `TestPlan14GenerationCensusRetriesWholeObservation` (unchanged). |
| 258 | - (b) fix 4: `TestRecoverCensusRefusalsNameCheckUnitCgroupAndExe`. |
| 259 | |
| 260 | The census tests drive the production service lookup over a fake `/proc`, |
| 261 | replacing only `systemctl`, the unit files' ownership check, the |
| 262 | signed-binary proof and the owner uid, which a non-root test cannot provide. |
| 263 | No root-runner test was run for this branch. |
| 264 | |
amadan