carlos / platform Public
File too large to show in full — download it above (showing a truncated preview below).
| 1 | # Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 19) |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| 6 | |
| 7 | **Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 15 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| 8 | |
| 9 | **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| 12 | |
| 13 | ## Rev 19: round 18 dispositions |
| 14 | |
| 15 | Astra's round 18 (`CODEX-PLAN-REGION-HA-1BN-R18.md`) resolved all three round 17 findings, accepted rulings 1 to 27, and raised two P2 test gaps. Both were checked against the plan's tests and accepted; none is declined. Neither exposed a production defect: the new events and oracle found no failing seed in the unchanged code. Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, and then each lifecycle guard was removed in turn. |
| 16 | |
| 17 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 18 | |---|---|---|---|---|---| |
| 19 | | 1 (P2) | A | The property's events only started and completed operations. Its recovery oracle wanted every relocation to be ready again on this box. So it could not express a box excluded from an operation that goes on, and it would have rejected correct recovery there. | accepted | New events: `replaced` (this box, the target, is replaced within the same operation by `sp2`, and the relocation continues) and `probeFailedAndReleased`. In every fourth seed, a script follows round 18's trace: target installed, probe fails, release journaled, replaced, cleanup blocked, a pass, an agent crash, then the random events and recovery. The oracle now asks a participating box to be ready, and an excluded box to have finished its cleanup: no controller, no hostexec record, no journaled participation, with the operation still running. New invariant (i): nothing for an operation opens again on a box excluded from it. Invariants (a), cleanup before clear, and (h), persistence across a crash, stay. | Task 7 (`TestRelocationControllerLifecycleProperties`) | |
| 20 | | 2 (P2) | A | `TestExecutorChooserSpreadsThenBalancesThenHashes` took the first choice as given and derived its later expectations from it. A chooser that hashed a constant instead of the record's host passed. | accepted | `TestExecutorChooserHashesTheRecordsHostOnEqualLoads` uses equal loads. It computes the winner independently (the greatest SHA-256 of host + "\x00" + box id, claim P5-D2) for `shop.bab.oncarlos.com` (box-b) and `cafe.bab.oncarlos.com` (box-a). The test itself asserts that the two winners differ, and requires `Choose` to return each one. Substituting a constant hostname in `ExecutorChooser.Choose` fails it (cafe chose box-b). | Task 2 | |
| 21 | |
| 22 | Mutation check (property at 200 seeds, `-failfast`): `Sweep` finishing a participation whose box is excluded, killed at seed 4 (the excluded box never finishes its cleanup). Rev 14's guards, rechecked under the new events, were all killed again: the cleanup before a clear (seed 1, (a)), the verified forget (seed 3, (b)), lost-controller recovery (seed 10), the journal before the open (seed 1, (b)), the close before the clear (seed 6), and the restart path's verification (seed 1, (b)). |
| 23 | |
| 24 | ## Rev 18: round 17 dispositions |
| 25 | |
| 26 | Astra's round 17 (`CODEX-PLAN-REGION-HA-1BN-R17.md`) rated F1 to F3 as only partly closed, with one P1 and two P2s. It accepted ruling 26 and its cost. All three findings were checked against the rev 17 code, reproduced, and accepted; none is declined. |
| 27 | |
| 28 | Stage A was dry-run on a clean 3407d114 in task order, and the lifecycle property passed 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. |
| 29 | |
| 30 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 31 | |---|---|---|---|---|---| |
| 32 | | 1 (P1, F2) | A, spec | Rev 17's `filesPin` returned the first manifest it found before checking any evidence. §5.6's pre-start copy can lack the `instance.key` the app mints after starting, so a nonempty database was pinned with no key. | accepted | `Pin` computes `needKey`: a writer was admitted and the pinned database is nonempty. `filesPin` then refuses with `ErrFilesUnevidenced` unless the pinned manifest has a live `instance.key` entry (`InstanceKeyPath`, `liveEntry`), and the worker marks the operation `stuck` (rev 17's path). Ruling 25 is amended. The durability window is recorded as the open request L21, with both candidate contracts, for Paul. | Task 5 (`TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase`: keyless with a nonempty database refuses; keyed pins; keyless with an empty database at a release of 0 pins); Requests (L21) | |
| 33 | | 2 (P2, F1) | A | `Worker.Step` sends a fenced target to `replace` before any phase check. `replace` returned the chooser's error for ever, so a target terminated after its release, with no spare, bypassed rev 17's bound. | accepted | New fence-consuming `hostclaim.FenceTargetWithoutSpare`, sharing `FenceExhaustedTarget`'s body (`fenceTargetStuck`): it retains the loss, puts the box on `lost`, revokes a dead executor and records `stuck` with its reason. `replace` calls it `ReleaseDeadline` after the fence's `observed_at` when no eligible box exists. | Task 1; Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare/target_terminated_after_its_release_true`: waiting before the deadline; after it, `stuck` with the fence consumed, `dst` lost and `op.exec` "none") | |
| 34 | | 3 (P2, F3) | A | `ReplaceTarget`'s default branch (`probe-failed`, or after P's admission) deleted `prepared` without recording an unopened P. That happens when the probe deadline passes before any admission and the target is then terminated. | accepted | `abandonPrepared(c)` records the current P when no writer opened it. `install` and `ReplaceTarget`'s default branch call it before they replace or drop P. `FenceExhaustedTarget` and `FenceTargetWithoutSpare` keep an unopened P for the operator, so they drop nothing. | Task 1 (`TestRelocationParksAPDroppedFromProbeFailed`: probe deadline before admission, target terminated, replaced, nested relocation completed; P1 is parked) | |
| 35 | |
| 36 | Guards removed, each failing its test: the keyless refusal; `needKey` only for a nonempty database; the dead-target bound; the bound consuming the fence; `ReplaceTarget` recording the P it drops. That is five. |
| 37 | |
| 38 | ## Rev 17: two-box model findings |
| 39 | |
| 40 | The joint model (`origin/region-ha-hostlife-two-box`, model plan rev 2 at 6d25c54e) reported three gaps in rev 16 that review rounds 1 to 16 missed. Each comes with a direct test (`TestFailoverBeforeTheFirstManifest`, `TestProbeFailedInATwoBoxRegion`, `TestAbandonedPreparedGenerationParked`). I checked each against this plan's code first, and all three reproduce. All are accepted; none is declined. |
| 41 | |
| 42 | Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed its 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. |
| 43 | |
| 44 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 45 | |---|---|---|---|---|---| |
| 46 | | F2 (data loss) | A, B | `StorePinner.filesPin` pinned an empty files set whenever no manifest existed in the admitted chain, whatever the holder's writer history. An install from it starts with no `instance.key`, and the app mints a new one over data sealed with the old (model INV-4). | accepted (ruling 25) | `filesPin` refuses with `ErrFilesUnevidenced` unless no writer was ever admitted (session 0) or the journaled release flushed nothing (`manifest_seq` 0). `Worker.pinUnderLease` marks the operation `stuck` on that error. Stage B: 6b's cold admission writes its manifest copy before the app starts (§5.6), with a named test. | Task 5 (`TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter`, `TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerMarksAnUnevidencedFilesPinStuck`); stage B re-plan | |
| 47 | | F1 (liveness) | A | In `probe-failed` with the release journaled, `Worker.Step` returned the chooser's error forever ("a later pass may find one"). With two boxes no spare exists, and `replaceTargetSlot` refuses the source. | accepted (ruling 26: stuck, not the source) | `ReleaseDeadline` after `probe_failed_since` with still no eligible box, the worker marks the operation `stuck`. Before that it waits as before. | Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare`: waiting before the deadline, `stuck` after it) | |
| 48 | | F3 (leak) | A | `CompleteRelocation` queued `park-local` chores only. A P replaced while unopened (at `installing`, or a deadline `probe-failed` before any admission) stayed in `holder.location` for the at-rest sweep alone. | accepted (ruling 27) | `install` journals every unopened P it replaces in `abandoned_prepared`. The serving CAS queues a `park-generation` for each in the region (`Owner` `region:<r>`), deduplicated, never for an opened P. | Task 1 (`TestRelocationParksEveryAbandonedPreparedGeneration`: replaced at `installing`, completed on P2. P1 is parked and P2 is not.) | |
| 49 | |
| 50 | Guards removed, each failing its test: `install` journaling the abandoned P; the serving CAS queuing its park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; and waiting before the bound. That is eight. |
| 51 | |
| 52 | Checked and unchanged: a target terminated with no spare in `starting` or `releasing` also waits for a spare. The live source still serves there (it is `op.exec` until its admission fence closes), so that is not an outage. The model's finding is the `probe-failed` case only. |
| 53 | |
| 54 | ## Rev 16: round 15 dispositions |
| 55 | |
| 56 | Astra's round 15 (`CODEX-PLAN-REGION-HA-1BN-R15.md`) resolved all four round 14 items and raised two P2s. I checked both against spec §5.8 (line 847, "a box that finds its receipt missing after that point … the op goes to `stuck`") and against the rev 15 property code. Both are accepted; none is declined. |
| 57 | |
| 58 | Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, in about four and a half minutes. Each new guard was then removed in turn. |
| 59 | |
| 60 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 61 | |---|---|---|---|---|---| |
| 62 | | 1 (P2) | A, B | Rev 15's `Step` called `EnsureLocal` only while P was unopened. After P opened it went straight to the next attempt, so a lost receipt never made the operation `stuck` as §5.8 requires. | accepted | Receipt verification is now separate from restoration. A new effect, `VerifyReceipt`, runs before every attempt once P is opened. It compares this box's durable receipt (the new `Receipt` provider: the install journal under its `install_stamp`) with the journaled `local_install`, and reads nothing else. If the receipt is missing or different, `Step` marks the operation `stuck` (`MarkStuck`, allowed in `probing` once P is opened) and makes no admission and no reinstall. Files, env and timers changed by the admitted writers are never compared or reverted. Stage B: the production `Receipt`, with named tests. | Task 6 (`TestAParticipantIsStuckWhenItsReceiptIsLostAfterPOpens`: P is opened and the agent restarts. With the receipt lost, the claim goes `stuck` and no admission follows. With it intact, the next attempt goes ahead and nothing is reinstalled or verified against content.) Task 7 (`TestVerifyReceiptChecksTheReceiptOnly`). Stage B re-plan | |
| 63 | | 2 (P2) | A | Rev 15's `restart` repaired the faults it was meant to survive: it unblocked and un-failed the cleanup, and closed the old controllers in the background, so their hooks ran. | accepted | `restart` becomes `crash`, an abrupt death. No controller is closed and no hook runs or completes. A dead generation's cleanups, including ones waiting when it died, never return. Its clock never fires again, and the next generation uses fresh lock files: the kernel releases a dead process's locks. Durable records and every external fault persist; recovery stays its own later event. New invariant (h): a record goes only through a successful clear or a successor's open over it, never through a crash or any other step, and while a record exists root's §5.10 check refuses an executable update. | Task 7 (the property, 200 seeds) | |
| 64 | | C1 | A | Found by the new crash event. In-process, a dead generation's hostexec goroutines still run: `publish` retries its "lost" write through a directory fd it already holds open. After a repair it recreated a record a live generation had just cleared (seed 75). | fixed in the test | Each generation reaches the data directory through its own symlink, and a crash removes the dead one, so newly opened paths fail. The repair waits 300 ms so a write already in flight through an open fd lands as part of the repair. A real dead process writes nothing, so production code is unchanged. | Task 7 (the property's `dataLink`, `recoverAll`) | |
| 65 | |
| 66 | Mutation check, with the property at 200 seeds and `-failfast`, plus targeted tests: |
| 67 | |
| 68 | | Guard removed | Result | |
| 69 | |---|---| |
| 70 | | `Step` marking `stuck` on a lost receipt after P opens | killed: `TestAParticipantIsStuck…` | |
| 71 | | `VerifyReceipt` comparing the receipt exactly | killed: `TestVerifyReceiptChecksTheReceiptOnly` | |
| 72 | | both `EnsureLocal` calls | killed, seed 182: (e) | |
| 73 | | `EnsureLocal` before an attempt alone / before entering probing alone | each survived: the other covers it | |
| 74 | | rev 14's ten lifecycle guards | all killed again; the crash event now kills "journal before the open" at (b) and the handover at (h) | |
| 75 | |
| 76 | While building C1 I also tried an in-flight-open guard: no forgetting while an abandoned open might still write. It survived 200 seeds, so I removed it. `ClearStale` already refuses while such an open holds the host's lock, and `recordIsMine` refuses on anything unclear. |
| 77 | |
| 78 | Astra's readings of rulings 1 to 24 stand. |
| 79 | |
| 80 | ## Rev 15: round 14 dispositions |
| 81 | |
| 82 | Astra's round 14 (`CODEX-PLAN-REGION-HA-1BN-R14.md`) resolved the lease-recovery finding. It left round 13's finding 2 partly resolved and raised three new P2s. All four are accepted; none is declined. |
| 83 | |
| 84 | Stage A was dry-run again on a clean 3407d114, applied in task order. The lifecycle property passed its 200 seeds (about three minutes, niced) and was re-run for every mutation. Each new guard was removed in turn; the results are below. |
| 85 | |
| 86 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 87 | |---|---|---|---|---|---| |
| 88 | | R13-2 (partly) | A | Rev 14's `recordIsMine` decoded the JSON itself. A parseable record without its enrollment or incarnation, which hostexec's own reader refuses (`State.validate`, `Binding.valid`), was taken for another binding, and the participation was forgotten. | accepted | New `hostexec.ReadLocal(dataDir, host)` (`internal/hostexec/readlocal.go`) is hostexec's validated reader: the same checks `Open`, `CheckLocal` and `ClearStale` apply. `recordIsMine` uses it. Only an absent record, or a valid record of another binding, is "not mine"; anything hostexec refuses is an error, and the participation is kept. | Task 7 (`TestARecordHostexecRefusesKeepsItsParticipation`, for a missing enrollment and a missing incarnation: refused and kept, still kept after a restart, then the record is repaired and a cleanup and clear succeed) | |
| 89 | | 1 (P2) | A, B | Claim rev 6.30 R18-1 and moves rev 7.22 §7.6 require the same thing: a create-only prepared manifest at sequence 0, bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. Rev 14's `ToInstalled` required P's sequence to equal the pin's, and `CopyPinnedManifest` kept the source's sequence. | accepted | `CopyPinnedManifest` writes P's prepared manifest at sequence 0, keeping the pinned entries by their content-addressed objects. `PreparedAuthority` separates the manifest's identity from its provenance: `FilesSeq` must be 0, `SourceFilesSeq` must be the pin's sequence, and `InstallStamp` is bound by `ToInstalled` from `install_stamp`. Ruling 13 is corrected. Stage B: R1 has the first admission consume that manifest by its journaled digest as the publisher's baseline, and the re-plan names the tests. | Task 1 (`TestRelocationPreparedManifestIsSequenceZero`: a prepared manifest claimed at the source's sequence 7 is refused, one built from another source revision is refused, and the accepted one is bound to the stamp); Task 5 (`TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne`: source at 7, prepared at 0, first session publication at 1, inherited entries intact); stage B re-plan | |
| 90 | | 2 (P2) | A, B | `Step` in `installed` with `local_install` journaled went straight to probing. Nothing checked that the install still existed after a lost lease's scoped cleanup retired the unopened P's local install. The property never reached installation. | accepted | `hostrelocate.Effects.EnsureLocal` runs before probing and before every attempt until P is opened. `host.RelocationEffects.EnsureLocal` verifies the receipt's install (`VerifyLocal`). If the install is retired, it restores it with the same `Install` under the same `install_stamp`, refuses a restore that is not the receipt's, and verifies again. It never touches an opened P. The property now also covers the target's role: half the seeds start with the local install journaled, and a new event hands the installation to this box. Its effects are stateful: `Install` makes the install, the scoped cleanup retires it unless P is opened, and `Admit` opens P. Two invariants are added: (f) P's first admission never meets a missing local install, and (g) an opened P is never reinstalled. Stage B: the production `VerifyLocal` and restore, with named tests. | Task 6 (`TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission`), Task 7 (`TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens`, the property), stage B re-plan | |
| 91 | | 3 (P2) | B | 6b's `runtime.Prerequisites` returns `ClaimDependencyError` for any probe or non-serving claim (6b plan, `if p.Probe \|\| o.Claim.Op.Kind != hostclaim.KindServing`). So a source that claim §4 and Task 1 let wake in `starting` and `releasing` could not, even after R1. | accepted (contract) | R1 now also asks for the source's cold-admission dispatch in `starting` and `releasing` while its admission fence is open, refused once the fence closes or the claim no longer names the box. A target's ordinary wake stays refused. The stage B checklist names the tests. | Requests (R1), stage B re-plan | |
| 92 | |
| 93 | Mutation check, with the lifecycle property at 200 seeds and `-failfast`, plus targeted tests: |
| 94 | |
| 95 | | Guard removed | Result | |
| 96 | |---|---| |
| 97 | | both `EnsureLocal` calls | killed, seed 32: (f) | |
| 98 | | `EnsureLocal` before an attempt while P is unopened | killed, seed 142: (f) | |
| 99 | | `EnsureLocal` before entering probing | survived: the attempt-time check covers it (defence in depth) | |
| 100 | | `EnsureLocal` restoring a retired install / refusing a restore under another stamp | killed: `TestEnsureLocal…` | |
| 101 | | `recordIsMine` through `hostexec.ReadLocal` | killed: `TestARecordHostexecRefusesKeepsItsParticipation` | |
| 102 | | `recordIsMine` comparing the binding | killed, seed 3: (b) | |
| 103 | | `ToInstalled` requiring sequence 0 / the pinned source sequence / binding the stamp | killed: `TestRelocationPreparedManifestIsSequenceZero` (each) | |
| 104 | | `CopyPinnedManifest` writing sequence 0 | killed: `TestCopyPinnedManifestEnforcesThePin` | |
| 105 | | rev 14's lifecycle guards (`finish`'s cleanup before a clear, verified forget, lost recovery, bounded open, journal before open, handover, close before clear, `ErrInactive` superseded, verified restart forget) | all killed again (seeds 2 to 46) | |
| 106 | |
| 107 | The `EnsureLocal` short-circuit for an opened P also survives, because the participant never calls `EnsureLocal` once P is opened. It is kept so the effect is safe on its own. |
| 108 | |
| 109 | Astra's readings of rulings 1 to 24 stand. Ruling 13 is corrected as finding 1 requires. |
| 110 | |
| 111 | ## Rev 14: round 13 dispositions |
| 112 | |
| 113 | Astra's round 13 (`CODEX-PLAN-REGION-HA-1BN-R13.md`) resolved both findings from round 12 and raised two P2s, both in the execution-controller lifecycle. Both are accepted; none is declined. This is the sixth round in a row to find a controller cleanup or recovery interleaving (rounds 2, 4, 5, 6, 12 and 13). So rev 14 also closes the whole class with a property test over real hostexec controllers, not only the two examples. |
| 114 | |
| 115 | The changes were dry-run on a clean 3407d114 with stage A applied in task order, building, vetting and running each task's tests after it, plus vet for darwin and windows and the cross-builds. The property test ran its 200 seeds niced, in about three minutes. Each lifecycle guard was then removed in turn (mutation check); the results are below. |
| 116 | |
| 117 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 118 | |---|---|---|---|---|---| |
| 119 | | 1 (P2) | A | One failed lease put makes hostexec latch the controller lost (controller.go, `run`: `loseLocked(putErr)`), and a lost controller never becomes ready again. Rev 13's `Ready` reported the error and kept the controller, `Step` reopens only a missing one, and `Sweep` keeps a running operation's controller. A transient storage fault in `installing` therefore stalled the relocation until the agent restarted. | accepted | `RelocationControllers.live`, used by `Ready` and `Acknowledge`, recovers within the same operation. If the open controller has lost its lease (`Lost()` is closed, newly on `RelocationController`), it is joined (`Close`, bounded by the pass) and the scoped cleanup is repeated. The object is then forgotten, but its journal entry is kept. `Ready` reports `ErrNoController`, and the participant's `Acknowledge` opens a fresh controller with fresh authority and its own lease. The claim keeps the first acknowledgement, as after a restart (rev 2, finding 10). The record on disk stays `lost` until the fresh controller replaces it, so denial is durable throughout. A failed join or cleanup keeps the lost controller, and the next pass repeats both. | Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`: lost lease, cleanup failing, then recovered and reopened, no restart); the property test | |
| 120 | | 2 (P2) | A | `ClearStale` returns `ErrState` for an absent or invalid claim (controller.go:619) and for an unreadable or mismatched local record, as well as for a record that names another binding. Rev 13's `finish` treated every `ErrState` as replacement and forgot the participation while this binding's record stayed. | accepted | `finish` no longer reads anything into the error. It asks the record itself (`recordIsMine`: `<DataDir>/.hostexec/<host>.json`, compared on box, enrollment, incarnation and operation, as `ClearStale` compares them). The participation is forgotten only when the record is absent or names another binding. An unreadable record, or this binding's record, keeps the participation and reports the error. `RelocationControllers.DataDir` is new; production sets it to the assembly's `Guard.DataDir`, and with it empty nothing is ever forgotten on a failed clear. | Task 7; the property test (claim read failure, absent claim, invalid local record, then restart and recovery) | |
| 121 | | C1 | A | The class. | added | `TestRelocationControllerLifecycleProperties` (`internal/host/relocationlifecycle_linux_test.go`) runs over real hostexec controllers, a store that injects faults, and a clock the test drives. Each seed runs 30 random events: lease-renewal failure, a close that times out (a cleanup that blocks), cleanup failure, `ClearStale` failing for each of its causes (claim unreadable, claim absent, invalid local record, a successor that lists this box), agent restart, a successor starting before the sweep, completion, and recovery of the storage. Participant passes are the real `Participant.Step` and `Sweep`. After every step it asserts: (a) no clear without a successful scoped cleanup earlier in the same call; (b) no participation forgotten while its record exists; (c) at most one controller per host; (d) no pass past its deadline. Then (e), liveness: once faults stop, within eight passes the running relocation is ready again; after completion every record is cleared, no controller remains, and a new relocation opens. 200 seeds by default, 20 with `-short`. | Task 7 | |
| 122 | | C2 | A | Found while building C1. `hostexec.Open` waits for its Cleanup hook without a context (`err = <-cleaned`), so rev 13's `openBounded`, which waited for `Open` to return, did not bound a pass whose cleanup blocks. | fixed | `openBounded` runs the open apart and returns when the pass's context ends. A controller that opens too late is closed, never kept. | Task 7; invariant (d) | |
| 123 | | C3 | A | Found while building C1. After a handover, sweeping the predecessor called `ClearStale` while the successor's live controller held the host's lock, which waited out the whole pass. | fixed | With no controller object, `finish` reads the record first. If it is verifiably not this binding's (absent, or the successor's), the participation is forgotten without a clear: nothing of its own is left to clear. The handover no longer forgets superseded entries itself; the next sweep does, through that check. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController` now sweeps A after the handover) | |
| 124 | | C4 | A | Found by the in-order dry run. `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` (rev 7) failed about once in five full-package runs. Main's `TestFDHoldersUnsharedThread` can run its unshare on the leader thread, which then keeps a private table for the rest of the process, so no thread shares the leader's. | fixed | The real-kernel test re-executes the test binary and runs in a fresh process. Its guards still fail it (the `KCMP_FILES` check was rechecked). | Task 4 | |
| 125 | |
| 126 | Mutation check: each lifecycle guard was removed in turn, and the property test was run with `-failfast`. |
| 127 | |
| 128 | | Guard removed | Result | |
| 129 | |---|---| |
| 130 | | `finish` repeating the scoped cleanup before a clear | killed, seed 1: invariant (a) | |
| 131 | | `finish` forgetting only a verified other binding | killed, seed 3: invariant (b) | |
| 132 | | `live` recovering a lost controller | killed, seed 13: invariant (e) (the relocation never recovers) | |
| 133 | | `openBounded` | killed: invariant (d) (a pass never returns) | |
| 134 | | the journal written before the open | killed, seed 3: invariant (b) | |
| 135 | | the handover before a successor opens | killed, seed 53: invariant (e) | |
| 136 | | `finish` closing before the clear | killed, seed 2: invariant (e) (never clean) | |
| 137 | | `recordIsMine` comparing the binding | killed, seed 3: invariant (b) | |
| 138 | | a handover treating hostexec's `ErrInactive` as superseded | killed, seed 7: invariant (e) | |
| 139 | | the restart path verifying the record before forgetting | killed, seed 3: invariant (b) | |
| 140 | | `live` repeating the scoped cleanup before the reopen | survived 200 seeds | |
| 141 | |
| 142 | The survivor is defence in depth, not a load-bearing guard. `hostexec.Open` runs the controller's Cleanup hook, which is the same scoped `relocationCleanup` (Task 8a), and fails if the cleanup fails. So no fresh controller, and no fresh authority, exists without a successful cleanup, and invariant (a) is about clears, which recovery never makes. The explicit cleanup is kept so that a failure surfaces at the pass that notices the loss. |
| 143 | |
| 144 | Astra's readings of rulings 1 to 24 stand. Rev 13's statement that the predecessor's entry is removed "once the successor's open succeeds" is replaced by C3. |
| 145 | |
| 146 | ## Rev 13: round 12 dispositions |
| 147 | |
| 148 | Astra's round 12 (`CODEX-PLAN-REGION-HA-1BN-R12.md`) resolved the file-origin finding and raised two P2s. Both are accepted; none is declined. Finding 1 is stage A code. It was dry-run on a clean 3407d114 with stage A applied in task order, building and running each task's tests after it, plus vet for darwin and windows and the cross-builds. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). Finding 2 is a stage B contract and has no code. |
| 149 | |
| 150 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 151 | |---|---|---|---|---|---| |
| 152 | | 1 (P2) | A | Rev 12's `Acknowledge` looked controllers up by host and operation, and opened a missing one under `e.Lifetime`. `hostexec.Open` loops on `ErrControllerBusy` until its context ends (controller.go, `Open`), and a predecessor's controller keeps the host's lifetime lock until `Close`. `ClearStale` refuses with `ErrInactive` whenever the running operation lists this box, whatever its id (controller.go:643). The regression reproduces it: without the handover, B's acknowledgement waits out the whole pass. | accepted | Before opening a successor's controller, `Acknowledge` calls `RelocationControllers.handover`. It retires every other participation of the same host under the pass's context: close, then the repeated scoped cleanup, then the clear. A clear refused with `ErrInactive` because the successor lists this box counts as superseded. The lock is free by then, and the successor's `Open` replaces the record under it; the predecessor's journal entry is removed only once that open succeeds. A clear that finds another binding's record (`ErrState`) means this participation has no record left. The open itself is bounded by `openBounded`: the controller still lives under the agent's lifetime, but waits for the lock no longer than the pass. A controller that opens in the same instant the pass ends is closed, never kept with a cancelled life. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController`, on real hostexec controllers: A acknowledged; A completes and B starts before A's sweep. B's acknowledgement succeeds, only B is retained, and B's record blocks updates. A lock held by a controller this box does not retain is waited for no longer than the pass, and nothing is kept.) | |
| 153 | | 2 (P2) | B | 6b rev 5: `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable`. `runtime.Prerequisites` calls `Roles.Preflight` before dispatch. The "External completion gates" section requires "the reservation-bearing extension of main's isolated root launch protocol … before any tenant shape can run with claims enabled", which "is not supplied by the external owner adapters". | accepted (contract) | Request **R10** asks the claim session and the activator's owner for that protocol, with a relocation probe's `"probe"` work admitted through it. Task 8c is now gated on R10. Task 10 gains an activation gate: the passes stay dormant and refuse while the launcher lacks the protocol. The stage B re-plan names the production probe tests: success through the protocol, refusal without it before any attempt is counted, and the pass staying off. | Requests (R10), Task 8c, Task 10, stage B re-plan | |
| 154 | |
| 155 | Astra's readings of rulings 1 to 24 stand. |
| 156 | |
| 157 | ## Rev 12: round 11 dispositions |
| 158 | |
| 159 | Astra's round 11 (`CODEX-PLAN-REGION-HA-1BN-R11.md`) resolved the chooser-test finding and raised one P2. It is accepted; none is declined. |
| 160 | |
| 161 | The coordinator also asked for a check of every stage A use of `holder.admissions` as authority. That check found one more case, S1. Both changes were dry-run on a clean 3407d114 with stage A applied in task order (1, 2, 3, 4, 5, 6, 7, 9a), building and running each task's tests after it. Vet for darwin and windows and the cross-builds are clean. Four new guards were each removed in turn, and each removal failed its test (listed in the Self-review). |
| 162 | |
| 163 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 164 | |---|---|---|---|---|---| |
| 165 | | 1 (P2) | A | `AdmitWriter` drops a covered, loss-archived head once more than eight admissions are kept (generation.go:154). Rev 11's `filesPin` built its origins only from `holder.generation` and the retained admissions. A file published at g1 and never changed is still `files/g1/objects/...` at g10, because `CopyManifest` carries the entry and `Stage` writes nothing for unchanged content. The pin accepted that manifest, but `CopyPinnedManifest` refused it with `ErrOrigin`. Claim §6.2a ("Files in the live set (L13)") makes the live set every object the live manifests reference, computed independently of the database baseline. | accepted | `filesPin` adds `referencedOrigins` to the retained chain: the files prefix of every generation whose objects the pinned manifest's entries and platform items reference. Each referenced object must lie under this holder's location, which binds it to the incarnation, in this region's data bucket (the region binding), at `files/<generation>/objects/`. Any other reference refuses the pin (`ErrRefused`, `hostfiles.ErrOrigin`), so the pin never admits what the copy would refuse. `CopyPinnedManifest`'s check is unchanged. | Task 5 (`TestPinKeepsAFileOriginTheAdmissionListTrimmed`: ten real `AdmitWriter` and `PublishGeneration` cycles with real `CopyManifest` and publication. The admission list is trimmed to eight and no longer names g1, yet g10's manifest still references g1's object for the unchanged file. The pin and the copy succeed, and a reference to another incarnation refuses.) | |
| 166 | | S1 | A | From the check. `preparedOpened`, the boundary that stops P being opened twice or its local install remade (B1-1, B2-5), scanned only the retained admissions. The same goes for `coverageFor`'s "P not opened" test. A covered P admission that is trimmed would let a later probe open P again. | fixed | The admission that opens P journals `prepared_opened` in the same CAS (`relocationAdmission`). `preparedOpened`, now exported as `PreparedOpened` for the participant, reads that key before the admission list. | Task 1 (`TestRelocationPOpenedOutlivesTheAdmissionTrim`, with P's admission removed as the trim would remove it: P's reopening is refused), Task 6 (`coverageFor`) | |
| 167 | |
| 168 | The check found no other authority taken from the trimmed list. `relocationAdmission` and `coverageFor` compare against the last admission, which the trim never removes: it drops only the head, and only past eight entries. `retainAutomaticLoss` records evidence, not authority. An admission the trim dropped was covered and loss-archived (`admissionLossArchived`), so nothing it could vouch for is lost; it falls back to the holder's live writer session. |
| 169 | |
| 170 | Astra's readings of rulings 1 to 24 stand. |
| 171 | |
| 172 | ## Rev 11: round 10 dispositions |
| 173 | |
| 174 | Astra's round 10 (`CODEX-PLAN-REGION-HA-1BN-R10.md`) resolved both findings from round 9 and raised one P2: a test that would still pass with its production wiring disconnected. It is accepted; none is declined. |
| 175 | |
| 176 | I then checked every stage A test for the same weakness. For each test of production wiring, the question was whether its assertions still hold with the wiring replaced by a stub. Two more tests failed that check. Both are fixed here, and the table lists them as S1 and S2. |
| 177 | |
| 178 | The changes were dry-run on a clean 3407d114, applying stage A in task order: build, vet for darwin and windows, cross-builds, and the targeted `hostops`, `console` and `cmd/console` tests. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). |
| 179 | |
| 180 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 181 | |---|---|---|---|---|---| |
| 182 | | 1 (P2) | A | `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims` read an empty bucket. Both readers returning nothing satisfied every assertion. | accepted | The bucket now holds evidence for two boxes in `ie`: their edges files (box-b advertising `region-ha-1`) and fresh status snapshots. It also holds three serving claims, two on box-a for bab/jam and one on box-b for bab/tea. The test reads through the constructed chooser's `Edges` and `Loads`. It asserts each box's identity, freshness, placement and capabilities, and each box's host and per-app counts. Then the evidence changes: box-a starts draining and goes stale, and a fourth host serves on box-b. The next read must show both changes. | Task 2 (`TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) | |
| 183 | | S1 | A | Found in the scan. `TestProductionAuthoritySelectsOnlyWithAChooser` used a one-box fixture, so a `SelectExecutor` that ignored the chooser and returned a fixed box-a still passed. | fixed | Two boxes, with box-a draining. The selection must be box-b with its roster enrollment. With both draining it must be `ErrNoEligibleBox` and no selection. | Task 2 | |
| 184 | | S2 | A | Found in the scan. `regionHAEnabled` sets `Config.RegionHA` from `CARLOS_REGION_HA`, and no test covered it. | fixed | `cmd/console/regionha_test.go`: on only for `strconv.ParseBool` true values, with whitespace trimmed. Unset, `0`, `false`, `no` and garbage are off. | Task 2 | |
| 185 | |
| 186 | The scan found nothing else in stage A. These tests already run against real wiring: |
| 187 | - hostclaim's transitions, on a real store. |
| 188 | - `ClaimLoads`, on real claims. |
| 189 | - `Guard.AdmissionFenced` and Task 9a's fence observer, against a real `BeginFence`. |
| 190 | - `RelocationPolicy`, through `Policy.Check`. |
| 191 | - The writers-stopped scan, against the real kernel and a fixture `/proc`. |
| 192 | - The worker, over a signed roster, `hostactors.Directory` and `hostfence.ClaimFenceAuthority`. |
| 193 | - `StopReceiptReleases`, against stored receipts. |
| 194 | - `ReleaseFlushFiles`, against a real store and registry. |
| 195 | - `ReleasePositionsFile` and `RelocationControllers`, on disk. |
| 196 | - `OpenRelocationController`, with a real hostexec controller. |
| 197 | |
| 198 | The remaining stubs are providers that Task 10 (stage B) supplies: the worker's `Choose`, `Eligible`, `Leases` and `Prepared`; `StorePinner.SelectDB`, whose production value is `genselect.Select` over the versioned bucket; and `RelocationEffects`' providers. The stage B re-plan has a row that requires each one to be tested with its production value. |
| 199 | |
| 200 | Astra's readings of rulings 1 to 24 stand. |
| 201 | |
| 202 | ## Rev 10: round 9 dispositions |
| 203 | |
| 204 | Astra's round 9 (`CODEX-PLAN-REGION-HA-1BN-R9.md`) resolved both round 8 findings and raised two new ones. Both are accepted; none is declined. |
| 205 | |
| 206 | This revision also adds a dry run that applies stage A in task order. The earlier dry runs applied every task at once, which is how finding 1 got through. This time a clean 3407d114 took Tasks 1, 2, 3, 4, 5, 6, 7 and 9a one at a time. After each task: `go build ./...`, vet of that task's packages, and that task's tests. With finding 1 fixed, every step built and passed and turned up no other ordering fault. The tree as a whole then passed vet for darwin and windows and the cross-builds. |
| 207 | |
| 208 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 209 | |---|---|---|---|---|---| |
| 210 | | 1 (P1) | A | Applied in order, Task 5 failed to build: `internal/hostrelocate/pin.go:259:23: undefined: hostfiles.ValidateManifest`. The helper was first defined in Task 7's `internal/hostfiles/platform.go`. | accepted | `ValidateManifest` now lives in its own file, `internal/hostfiles/manifestcheck.go`, which Task 5 creates (Files list, Step 4, the Step 5 run and the Step 6 `git add`). Task 7's `platform.go` keeps only `EnvItem` and `StagePlatformDelete` and uses the helper unchanged. | Task 5, Task 7; the in-order dry run | |
| 211 | | 2 (P2) | A | Selection and completion in rev 9's test both named box-a, and the intent check called `BeginCreateOn` itself. The test still passed with `Attempt`'s selection removed. | accepted | The fake authority now has a selected box, the box completion offers, and a call counter. With selection on box-b and completion offering box-a, `Attempt` must refuse with `ErrWrongState`. The persisted claim must still be `creating`, with `selected_executor` box-b and `executor_enrollment` enrollment-b. `Complete` with box-b then serves on box-b, and selection must have run exactly once. Removing `Attempt`'s use of the selection (`if selected` made false) fails the test: the completion substitutes box-a. | Task 2 (`TestCreateIsTodaysWithoutASelection`) | |
| 212 | |
| 213 | Astra's readings of rulings 1 to 24 stand. |
| 214 | |
| 215 | ## Rev 9: round 8 dispositions |
| 216 | |
| 217 | Astra's round 8 (`CODEX-PLAN-REGION-HA-1BN-R8.md`) resolved the release-retry finding and added two P2s. I checked both against `internal/host/claimstop.go` and `internal/hostguard/fence.go` and accepted both; none is declined. Both fixes are stage A code. They were dry-run on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `hostclaim`, `hostrelocate` and `host` tests. Two new guards were each removed in turn and their tests failed (listed in the Self-review). Ruling 23 is amended. |
| 218 | |
| 219 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 220 | |---|---|---|---|---|---| |
| 221 | | 1 (P2) | A | Rev 8's `FenceExhaustedTarget` always set `op.exec` to "none" and the phase to `pinning`. In `starting` and `releasing`, `op.exec` is the live source, and the verified fence is the target's. Root's stop refuses a requester that is not `op.exec` (claimstop.go:184, `observed.Claim.Op.Exec != req.Box`). | accepted | The fix depends on who holds `op.exec`. If another box holds it (the live source in `starting` or `releasing`), the CAS consumes the target's fence, retains its loss, puts it on `lost`, drops it from the participants and records `stuck`. It leaves `op.exec`, the phase, the stamp and every release record unchanged. The old move to a stuck `pinning` with `op.exec` "none" now happens only when `op.exec` is the dead target itself (its own fence revokes it) or is already "none". | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning/dead_while_the_source_executes_in_starting` and `…_in_releasing`, the latter with the source's release journaled first); ruling 23 | |
| 222 | | 2 (P2) | A | Rev 8's `Release` restored `recorded.ExpectedDirectoryRevision` only when the boot or controller had changed. Root's stop compares that revision with its stop inventory (claimstop.go:191). The fence target also carries it (`hostguard.FenceTarget.DirectoryRevision`, fence.go). | accepted | When matching positions are recorded, `Release` always asks under the recorded `ExpectedDirectoryRevision`, recovery or not. Only the recovery branch takes the recorded boot and controller. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the retry runs under the same boot and controller, with the roster at revision 5. Root is asked at revision 4, and only the producers' drain and the receipt run.) | |
| 223 | |
| 224 | Astra's readings of rulings 1 to 22 and 24 stand. Ruling 23 is amended as finding 1 requires. |
| 225 | |
| 226 | ## Rev 8: round 7 dispositions |
| 227 | |
| 228 | Astra's round 7 (`CODEX-PLAN-REGION-HA-1BN-R7.md`) resolved the kcmp finding, confirmed that stage B's contracts are complete, and added one P2. We checked it against `internal/host/claimstop.go` and accepted it; none is declined. The fix is stage A code. It was dry-run again on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `internal/host` tests. Two new guards were removed in turn and each made the new test fail (listed in the Self-review). |
| 229 | |
| 230 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 231 | |---|---|---|---|---|---| |
| 232 | | 1 (P2) | A | In `ClaimStopCoordinator`, `DrainClaim` runs before `Kernel.VerifyStopped` and `Guard.CompleteFence` (claimstop.go:219 onward). If either fails, root has no durable receipt. On retry, `BeginFence` returns the same fence, `FenceReceiptFor` returns `ErrNoFenceReceipt`, and root asks for the drain again. Rev 7's `Release` read the recorded positions but still built a fresh `releaseDrain`, which repeated `StopWriters`, the observation and both flushes against a replicator its own producer drain had already closed. | accepted | When this controller has already recorded positions for the exact stop (same boot and controller, root without a receipt), `Release` marks its drain `resumed` and carries the recorded TXID and sequence. A resumed drain goes straight to the producers' own drain, which is idempotent, and leaves verification and the receipt to root. A restarted agent keeps the recovery path; a replayed receipt keeps pairing with the record. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the drain closes the producers, root fails before persisting its receipt, and the retry without a restart runs only `producers` and `receipt`, with no flush, record or writers stop, and returns 41 and 8) | |
| 233 | |
| 234 | Astra's readings of rulings 1 to 24 stand. |
| 235 | |
| 236 | ## Rev 7: round 6 dispositions |
| 237 | |
| 238 | Astra's round 6 (`CODEX-PLAN-REGION-HA-1BN-R6.md`, against `222ff1c6`) resolved all six of round 5's findings and added one. It was checked against the plan and the kernel before anything changed, and is accepted; none is declined. The fix is stage A code, dry-run again on a clean 3407d114 (build; vet for darwin, windows and linux/arm64; cross-builds; the stopevidence tests run five times against the real kernel), with 3 new guards removed in turn (listed in the Self-review). Stage B is unchanged: R3's root endpoint calls the same `WritersStopped`. |
| 239 | |
| 240 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 241 | |---|---|---|---|---|---| |
| 242 | | 1 (P1) | A | `scanOne` exempted a thread's descriptor when the leader showed the same number and path. Confirmed on this kernel: a thread that runs `unshare(CLONE_FILES)` keeps a copied table with the database at the same number, and it can `dup3` an independent open onto that number; `/proc/<pid>/task/<tid>/fd/<n>` reads the same path either way. kcmp(2) is the positive check: `KCMP_FILES` compares two tasks' descriptor tables and `KCMP_FILE` compares two descriptors' open file descriptions. golang.org/x/sys v0.46.0 has `unix.SYS_KCMP` on all 15 linux architectures but no `unix.Kcmp` and no constants, so the plan makes a raw `Syscall6` with `KCMP_FILE = 0` and `KCMP_FILES = 2` from linux/kcmp.h. kcmp needs ptrace read access to both tasks. The agent always has it within its own thread group, and root's complete scan has it through `CAP_SYS_PTRACE`. This box runs `CONFIG_KCMP=y` and Yama scope 0. | accepted | A thread's descriptor is the leader's only if three things hold: the leader shows the same number and path, `KCMP_FILES(pid, tid)` is 0 (the thread shares the leader's table), and `KCMP_FILE(pid, tid, n, n)` is 0 (the same open file description). Otherwise it is reported with FD -1, which no allow-list exempts. A copied table counts as foreign even when the description is shared, because the thread could still use that description after the replicator closes. If a task exits mid-scan (ESRCH), it is skipped as before. EBADF is treated as "not the leader's". Any other kcmp error, including ENOSYS and EPERM, is `ErrIncomplete`. kcmp takes pids in the caller's pid namespace, so it is used only for the caller's own `/proc`. Any other root fails closed unless a test replaces `kcmpFor`. The `//go:build linux` file and the `!linux` stub stay as they were. | Task 4 (`TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` on the real kernel: a shared locked thread adds nothing; an unshared thread with the copied descriptor, and one that reopened the database and `dup3`'d it onto the leader's number, are each reported FD -1. `TestWritersStoppedRefusesAnyOtherHolder`'s new fixture cases: an unshared thread at the leader's number and path, a shared table whose descriptor is another description, and kcmp unavailable (ENOSYS, EPERM) as `ErrIncomplete` for both scans) | |
| 243 | |
| 244 | Astra's readings of rulings 1 to 24 stand. |
| 245 | |
| 246 | ## Rev 6: round 5 dispositions |
| 247 | |
| 248 | Astra's round 5 (`CODEX-PLAN-REGION-HA-1BN-R5.md`, against `996b24c3`) left R4-5 partly resolved and added five findings. Each was checked against the tree at 3407d114 and the plan before anything changed; all six are accepted and none is declined. Every one lands in stage A code, dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds for darwin, windows and linux/arm64, targeted tests), with 11 new guards removed in turn (listed in the Self-review); R4-5's restart obligations are carried into the stage B re-plan as well. |
| 249 | |
| 250 | | # | Stage | Evidence checked | Disposition | What changed | Where | |
| 251 | |---|---|---|---|---|---| |
| 252 | | R4-5 | A, B | `hostexec.Controller.Close` (controller.go:480) sets `closed` before returning `errors.Join(err, c.CleanupError())` and returns nil on every later call; rev 5's `Finish` cleared after any nil `Close`, and the restart path cleared from the journaled binding with no cleanup at all. Rev 5 also journaled only inside `Finish`, so a record left by a crash (or a failed `Open`, which writes `lost` once it holds the live lock) before the first `Finish` was never found. | accepted | Repeat, do not trust: `RelocationControllers.Cleanup(ctx, binding)` is the scoped cleanup (Task 8a's `relocationCleanup`, authenticated to the binding's box and enrollment) and `Finish` runs it, and needs it to succeed, in the same call as every clear, after `Close` and on the restart path alike; a failure keeps the record, the entry and the journal. The participation is journaled before the controller opens (`Acknowledge` refuses without a journal), so a restart finds every record it may have left; the journal write syncs its directory. | Task 7 (`TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`: cleanup failure, a sweep after it, a restart while it still fails, then success; `TestARestartFindsAParticipationThatNeverReachedFinish`; `TestFinishClearsTheDurableRecordSoUpdatesAreEligible` with a real hostexec controller whose cleanup fails); stage B re-plan | |
| 253 | | 1 | A | `hostops` `eligible` excluded only `Lost`; `Worker.next` excluded the source, `Lost` and replaced targets; `ReplaceTarget` reaches `install` directly in `installing` and `installed`, never through `ToInstalling`'s `park-local` check. | accepted | `hostclaim.OwesLocal` and `LocalChoreBoxes`. Selection excludes every box owing a `park-local` (the chooser's eligible set, so create, `Permits` and the worker's spare all agree; `Worker.next` too), and `replaceTargetSlot` refuses one with `ErrPendingChore` inside the CAS, for `ReplaceTarget` at every phase and `NameTarget`. | Task 1 (`TestRelocationReplacementRefusesABoxOwingALocalRetirement`, in `starting`, `installing` and `installed`), Task 2 (`TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/owing_a_local_retirement`), Task 5 (`TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement`) | |
| 254 | | 2 | A | `fenceBox` journaled the fence, added `lost`, discharged `park-local` and retained the loss, and touched no regional obligation; `lost.go:302` reassigns `box:<b>`'s `park-generation` and `park-replica` to `region:<r>` for a declared loss. | accepted | `fenceBox` passes every `park-generation` and `park-replica` owned by `box:<b>` to `region:<obligation's region>` in the same CAS, changing nothing but `owner` (an obligation with no region refuses, as in `DeclareLost`); the retained `AutomaticLoss` names them (`reassigned`, the D4 record's `reassign_regional`). Every automatic fence CAS shares it: start, `FenceSource`, `ReplaceTarget`, `FenceExhaustedTarget`. | Task 1 (`TestRelocationFenceReassignsTheDeadBoxsRegionalWork`, unqualified and qualified discharge); R5 | |
| 255 | | 3 | A | Task 9a's `check` replaced `Serving` with `RelocationRow` unconditionally; with both callbacks nil a `moving` executor participant is `RowHeld` and passed, and `Install` went on to `Activate` and `RecordInventory` before `WorkSource.SelectAt` (work.go:94) refused. | accepted | `Reconciler.relocationAware()` is `FenceClosed != nil \|\| HoldRow != nil` (Task 10 sets both only with `CARLOS_REGION_HA`). Off, `check` is today's `Serving` and `Install` skips the row decision and the hold-only path entirely. | Task 9a (`TestReconcileWithoutAFenceObserverIsTodays`: a relocating source in `starting` and `releasing` and a `moving` executor, dry run and real, refused with the claim's ETag and every file under the guard's data directory unchanged) | |
| 256 | | 4 | A | `journaled` decoded every directory entry, so `writeFileSynced`'s truncated `.json.tmp` failed the whole listing, and `Retained` dropped the error. | accepted | Only names ending `.json` are committed records; a `.tmp` is ignored (its participation opened nothing: the record precedes the open). A committed record that cannot be read is reported, joined into the error, beside every readable one. `Effects.Retained` returns `([]Participation, error)`; `Sweep` reports the error and still finishes every readable participation. | Task 7 (`TestARestartFindsAParticipationThatNeverReachedFinish`: one valid journal, one truncated `.tmp`, then one damaged committed entry), Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`) | |
| 257 | | 5 | A, B | `Flush` loads the existing manifest through `hostfiles.Open` and did nothing on `ENOENT`, so `Platform[EnvItem]` kept the old bytes. | accepted | `(*hostfiles.Publisher).StagePlatformDelete(name)` (in `platform.go`) drops the item at a new sequence; `Flush` stages it on `ENOENT`, so `ReadPlatform` on the final revision returns `os.ErrNotExist`. Stage B's local install treats that as "no env file" (re-plan). | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`: published, removed, flushed again, then a no-op repeat) | |
| 258 | |
| 259 | Astra's readings of rulings 1 to 24 stand; ruling 20's cleanup is completed by R4-5 above. |
| 260 | |
| 261 | ## Rev 5: round 4 dispositions |
| 262 | |
| 263 | Astra's round 4 (`CODEX-PLAN-REGION-HA-1BN-R4.md`, against `418daf58`) left three earlier findings partly resolved and added six. All nine are accepted; none is declined. Under ruling 24, stage A findings are fixed in code (dry-run on a clean 3407d114, 7 new guards removed in turn, listed in the Self-review); stage B findings are fixed at the contract level: what must happen, which request covers it, which test proves it. |
| 264 | |
| 265 | | # | Stage | Disposition | What changed | Where | |
| 266 | |---|---|---|---|---| |
| 267 | | R1-9 | B | accepted | `Region` is 6b's field: Task 8a's `Collect` provider sets it on every request from the verified actor (the box's region never changes), initially and on recovery alike, beside `Recover` for the recorded owner's stop. | Task 8a | |
| 268 | | N1 | B | accepted | The fixture's `withExistingRow` takes the hold everywhere it is called; the census literal carries 6b's actual fields (`Inventory`, `Registry`, `Launches`, `Box`, `Enrollment`, `DirectoryRevision`, `Roots`, `SharedPaths`), no `Guard`. | Task 9 | |
| 269 | | N3 | B | accepted | R8's insertion point is after actor verification: 6b's `Build` calls an `Extend(a *ClaimAssembly, actor hostbox.Actor)` hook before it captures `Policy.Check`; `applyRegionHA` is that hook's body and binds the policy to `actor.Box`. | Task 10, R8 | |
| 270 | | 1 | A | accepted | `Guard.AdmissionFenced(host, claim, captured)` takes no lock (one atomic record read; the claim is the caller's fresh read); `Reconciler.FenceClosed` takes the context and the observation and returns an error, which keeps rows and mints none; `holdOnly` re-decides on a fresh read under `WithLocal`. Tested with the real observer, a real `BeginFence`, and a bounded context on both the activation and the hold-only path. | Task 3, Task 9a (`TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver`) | |
| 271 | | 2 | A, B | accepted | Policy half (A): a relocation target's probe is its own selected action, `probe`, which `Policy.Check` admits for a relocating claim and `RelocationPolicy` sends to `WakePhase(probe=true)`; an ordinary `spawn` by the target stays refused. Dispatch half (B): R1 extended, 6b's runtime selects `"probe"` work for a relocation probe. | Task 3 (`TestRelocationPolicyGivesEachActionItsRole`, `TestPolicyLetsAHeldParticipantReconcileItsRow`), R1 | |
| 272 | | 3 | B | accepted | The restore destination (the host's files directory and env path) is created and inventoried under the installation's authority before `Restore`, through 6b's typed inventory (a `files` entry at `install_stamp`), never an untracked `MkdirAll`; tested with a fresh spare that has no host directory. | Task 8c | |
| 273 | | 4 | B | accepted | Held release staging needs root: request R9 asks moves for a relocation-aware relay authorizer (bound to the operation, executor, install stamp and the pinned release); the install verifies staging before it records its receipt; tested under the edge unit's filesystem restrictions. | Task 8c, R9 | |
| 274 | | 5 | A | accepted | `RelocationController.Clear` (hostexec's `ClearStale` with the controller's config); `Finish` journals the participation before closing, keeps the entry and the record on a failed close or clear, clears the exact record after a successful cleanup, and a restart retries through the journaled binding. A real hostexec controller on disk shows root's §5.10 check (`hostexecParticipation`) refusing before and admitting after. | Task 7 (`TestFinishClearsTheDurableRecordSoUpdatesAreEligible`, `TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`) | |
| 275 | | 6 | A | accepted | The obsolete-hold clearance in `Policy.Check` needs `Relocation` set (region HA on); off, today's refusal. Tested both ways on the same held serving row. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`) | |
| 276 | |
| 277 | ## Rev 4: round 3 dispositions |
| 278 | |
| 279 | Astra's round 3 (`CODEX-PLAN-REGION-HA-1BN-R3.md`, against `d93c94d0`) left three earlier findings partly resolved, one unresolved, and added six. All ten are accepted; none is declined. The un-gated changes were dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (8 removals, each failing its test; listed in the Self-review). |
| 280 | |
| 281 | | # | Disposition | What changed | Where | |
| 282 | |---|---|---|---| |
| 283 | | R1-5 | accepted | `PreparedFromInstallManifest.Verify` compares P's entries against the pinner's retained copy (`pin.Files.Retained`), never the source manifest; a late source PUT changes nothing it reads. | Task 8b | |
| 284 | | R1-9 | accepted | After a restart `Release` names the recorded owner's boot, controller and directory revision and offers `RecoveryReleaseDrain`; Task 8a's `Collect` provider sets 6b's `Recover` on that drain, so root's authenticated recovery path accepts the tuple and replays the receipt. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`), Task 8a | |
| 285 | | R2-N4 | accepted | `Policy.Check`'s serving branch lets exactly the serving executor's `reconcile` through an obsolete hold (one naming no operation the claim runs), so the adapter's clear is reachable; every other write under a hold still waits. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`), Task 9 | |
| 286 | | R2-N5 | accepted | See N2. | Task 7 | |
| 287 | | N1 | accepted | Task 9's adapter insertion re-reads the claim under the captured authority (`observed` is defined there, validated against the captured incarnation and stamp); the test fixture (`adapterFixture` and its methods) is this task's code, over 6b's adapter and registry. | Task 9 | |
| 288 | | N2 | accepted | `Finish` keeps the map entry until `Close` succeeds and forgets only that same controller, so a close the pass's context cut short is finished by a later sweep. | Task 7 (`…RetainsTheControllerUnderTheAgentLifetime`: a cancelled close, then a successful one) | |
| 289 | | N3 | accepted | `hostbox.RelocationPolicy(box)` (un-gated, tested) and `hostguard.Guard.AdmissionFenced` (un-gated, tested) are the two providers; Task 10 wires them into the assembly between 6b's `OpenProductionClaimAssembly` and `Build` (which captures `Policy.Check`), gated on `CARLOS_REGION_HA`; with it off neither is set and behaviour is 6b's. The insertion point is request R8. | Task 3, Task 10, R8 | |
| 290 | | N4 | accepted | `Reconciler.Install` takes the hold-only path for a held row whenever the box is not the executor or its admission fence is closed, so a fenced source still named `op.exec` in `releasing` writes its hold under `WithLocal`, never through the guarded path `With` refuses. | Task 9 | |
| 291 | | N5 | accepted | `FenceExhaustedTarget`: a target terminated after the replacements are used is fenced into a stuck `pinning` in one CAS (loss retained, box on `lost`, `op.exec` "none", receipt dropped, P's remote build kept when unopened, nested when opened); the worker uses it instead of `MarkStuck`, which stays the live target's rule. | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning`), Task 5 (`TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall`); ruling 23 | |
| 292 | | N6 | accepted | `PinAfterRelease` from `releasing` requires the acknowledgement barrier, so a target replaced after the source's release acknowledges before the operation leaves the phase. | Task 1 (`TestRelocationReplacementInReleasingAcknowledgesBeforePinning`) | |
| 293 | |
| 294 | ## Rev 3: round 2 dispositions |
| 295 | |
| 296 | Astra's round 2 (`CODEX-PLAN-REGION-HA-1BN-R2.md`, against `ed992c46`) left seven round 1 findings partly resolved and added seven. All fourteen are accepted; none is declined. Every un-gated change was dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (11 removals, each failing its test; listed in the Self-review). |
| 297 | |
| 298 | | # | Disposition | What changed | Where | |
| 299 | |---|---|---|---| |
| 300 | | R1-2 | accepted | The build's authority is checked at every write, not once: `authorityStore` wraps the versioned store so each `PutIfAbsent` and `CopyVersion` re-reads the claim and the host-guard record (`hostguard.Capture` at `install_stamp`) first; the lease's loss cancels the rest. | Task 8b | |
| 301 | | R1-4 | accepted | `AutomaticLoss` retains the holder's incarnation and stamp, the box's last writer session whether or not it is still the writer (its last admission record on the holder, `LastAdmission`), so a source fenced after the target's admission keeps its session and consequences. | Task 1 (`TestRelocationLossRetainsAFormerWritersSession`); R5 | |
| 302 | | R1-5 | accepted | The pinner retains the exact manifest bytes in the control bucket (`RetainedManifestKey`, create-only) before the pin; `CopyPinnedManifest` writes P from that copy, validated by digest, sequence and origins (`hostfiles.ValidateManifest`), and never reads the source again. A late source PUT changes nothing. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`), ruling 21 | |
| 303 | | R1-9 | accepted | `ReleasePositions.Recorded` matches the installation and session, not the controller, and returns the request the flush was made under; after a restart `Release` asks root under that identity (so root finds the stop's durable receipt), offers no drain of its own, and pairs the replayed receipt with the recorded positions. A stop root never completed cannot be finished by a new process: the release deadline makes it stuck. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`, `TestReleasePositionsFileBindsTheExactStop`) | |
| 304 | | R1-13 | accepted | A later probe attempt whose predecessor generation holds no LTX file (`genselect.ErrEmpty`) is an ordinary empty cold admission: `Empty` set, no baseline, 6b's own empty authority. | Task 8c | |
| 305 | | R1-14 | accepted | See new finding 2. | Task 9 | |
| 306 | | R1-16 | accepted | Exhaustion is checked before a replacement is looked for, in both paths: with the replacements used the operation is stuck now, never retried on "no eligible box". | Task 5 (`TestWorkerMarksExhaustionStuckBeforeLookingForASpare`) | |
| 307 | | N1 | accepted | `Prepared` is set only by the two branches that open P; the retry branch resets `Prepared` and `Empty` and selects its baseline as an ordinary cold wake, so its marker follows the first durable root. | Task 8c | |
| 308 | | N2 | accepted | The adapter test separates "no row to mint" (a box with no role and no row: `ErrChanged`, nothing written) from "an existing unexplained row to keep" (held, never served, never deleted), and asserts `hold_op`, not a status the implementation does not write. | Task 9 | |
| 309 | | N3 | accepted | `ProcessDescriptors` reads only `/proc/<pid>` (its own process; another process's unreadable table does not matter); the complete scan is root's: `RelocationEffects.Observe` is a provider, production asks root over the instance-control socket (`CollectWritersStopped`, R3's server half), and an agent-side scan fails closed on the first unreadable table. | Task 4 (`TestProcessDescriptorsNamesOneProcessesHolders`), Task 7, Task 8a, R3 | |
| 310 | | N4 | accepted | A `RowReady` row whose `hold_op` names an operation the claim no longer runs has its hold cleared under the activation's verified ownership (`SetClaimHold(owner, old, "")`); a hold of the running operation is never touched. | Task 9 (`TestReconcileAdapterClearsTheHoldAfterCompletion`), ruling 20 | |
| 311 | | N5 | accepted | `Effects.Retained` and `Finish`; `Participant.Sweep` finishes every retained controller whose operation the claim no longer runs with this box as a participant, once per pass; the relocation controller's `Cleanup` hook is scoped to the relocation's own partial work and is a no-op once this box has admitted under the operation or serves the host, so finishing never stops the serving writer. | Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`), Task 7 (`Finish` in `TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`), Task 8a, Task 10; ruling 20 | |
| 312 | | N6 | accepted | `MaxProbeAttempts = 3`; `BeginProbeAttempt` counts the attempt in the claim before it is made (per box and operation, so a replacement starts at zero); the participant enters `probe-failed` on the fourth. Durable across restarts and lost responses. | Task 1 (`TestRelocationProbeAttemptsAreBounded`), Task 6 (`TestAParticipantGivesUpAfterThreeProbeAttempts`); ruling 19 | |
| 313 | | N7 | accepted | A qualified discharge completes each `park-local` with `CompleteObligation`'s own semantics, so a discovery operation's final chore marks discovery complete exactly as a retirement would. | Task 1 (`TestRelocationDischargeCompletesDiscoveryLikeARetirement`) | |
| 314 | |
| 315 | Astra's readings of rulings 1 to 4, 6 to 8 and 10 to 17 stand; rulings 5, 9 and 18 are completed by R1-5, N6 and R1-4 above. |
| 316 | |
| 317 | ## Rev 2: round 1 dispositions |
| 318 | |
| 319 | Astra's round 1 (`CODEX-PLAN-REGION-HA-1BN-R1.md`, against `eb5e9051`) returned sixteen findings, "not ready". Each was checked against the tree at 3407d114 and the cited plans before anything changed. Fifteen are accepted in full and one in part (8: the un-gated half is built, the production body is a request to the claim session). None is declined. Every un-gated change below was dry-run again: applied to a clean 3407d114, built, vetted for darwin and windows, cross-built, tested, and each new guard removed in turn (25 removals, each failing its test; listed in the Self-review). |
| 320 | |
| 321 | | # | Disposition | What changed | Where | |
| 322 | |---|---|---|---| |
| 323 | | 1 | accepted | Task 8 is three tasks against the declared types: `moves.SealedObject{Key, VersionID, Size}`, `moves.InstallManifest` (on main), `moves.BuildInputs` with `Now` from this plan's config, 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`. What no plan supplies is now this plan's code (`RelocationWakeInputs`, the install journal, `PreparedFromInstallManifest` reading `moves.InstallManifestKey`) or a named request (R1 the probe dispatch 6b refuses today, R2, R3). | Tasks 8a, 8b, 8c; Requests | |
| 324 | | 2 | accepted | The worker holds the replica lease on `holder.location` from pin selection to the journal CAS (`Worker.Leases`, `pinUnderLease`); a lease lost before the CAS refuses. The build runs under the same lease, re-reads the claim before each durable step and is cancelled when the lease is lost. | Task 5 (`TestWorkerDrivesADrain`'s order, `TestWorkerRefusesToJournalAPinAfterTheLeaseIsLost`); Task 8b (`TestRelocationBuildStopsWhenAuthorityChanges`) | |
| 325 | | 3 | accepted | `ReleaseFlushFiles`: `Open`, `StageWake(ExportWake)`, `StagePlatform(EnvItem)`, `SyncDir`, `Commit` as one revision; `hostfiles.EnvItem` added. | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`) | |
| 326 | | 4 | accepted | D4 retention outside the operation: `Claim.EvidenceRefs []AutomaticLoss` (fence reference, observation time, box, role, last writer session, discharge verdict, timers `unknown`, reporting pending) and the loss id in `LossRefs`, written by `fenceBox` in the consuming CAS with a deterministic id; both survive `CompleteRelocation`, `free` and holder replacement. `Claim.Validate` checks them. | Task 1 (the failover test's completion checks, `TestRelocationRetainsEveryFencedBoxAcrossCompletion`); ruling 18 | |
| 327 | | 5 | accepted | `FilesPin.Origins` (the admitted chain as pinned); `hostrelocate.CopyPinnedManifest` verifies the source bytes against the pin's digest and sequence, copies with the pinned origins, and refuses a copy whose entries differ from the pinned ones; the build and the verifier use it. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`: a late revision, different bytes at the same sequence, inherited objects); Tasks 8b | |
| 328 | | 6 | accepted | `PinObject.N` is gone. `ClaimGen` names the observation; the build takes `pin.DB.MaxTXID`; the empty branch is `DB == nil`. | Task 5, Task 8b | |
| 329 | | 7 | accepted | `ToInstalled` requires `FilesSeq == pin.FilesSeq` and a `FilesManifestSHA256` always (an empty pin included); the build writes P's files manifest before the database; the verifier reads both back. | Task 1 (`TestRelocationDrainEndToEnd`, `TestRelocationEmptyPinNeedsItsEvidence`); Task 8b (`TestPreparedRefusesAPartialBuild`) | |
| 330 | | 8 | partly | `RelocationEffects.StopWriters` runs first in the drain, before the observation; its production body is request R2, because 6b has no writers-only stop and its `ClaimProducers.DrainClaim` cancels every producer. The live-process test is Task 8a's, gated on R2. | Task 7 (`TestRelocationReleaseRunsStepsOneToThreeInsideTheDrain`); Task 8a; Requests | |
| 331 | | 9 | accepted | `ReleasePositions`: the flushed positions are recorded durably, bound to the exact stop request, before the producers close; a replayed receipt is paired with that record or refused. `ReleasePositionsFile` on disk. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions`, `TestReleasePositionsFileBindsTheExactStop`) | |
| 332 | | 10 | accepted | `RelocationControllers` retains controllers under `Lifetime` (the agent's context); `Effects.Ready` runs before every effect; a restarted agent reopens a controller and never re-journals the acknowledgement. | Task 6 (`TestAParticipantChecksItsControllerBeforeEveryEffectAndReopensAfterARestart`); Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`); Task 10 | |
| 333 | | 11 | accepted | The pin's key names the claim generation it was selected against (`hostclaim.RecoveryPinKey`, `pin-<gen>.json`); `JournalPin` binds it, so a pin whose CAS lost is abandoned and the retry selects afresh; every bound is checked before the object is written; the counter is gone. | Task 1, Task 5 (`TestStorePinnerPinsTheNewestFilesManifestInTheChain`, `TestStorePinnerChecksTheBoundsBeforeItWrites`); ruling 16 | |
| 334 | | 12 | accepted | `Worker.Eligible` (`ExecutorChooser.Permits`: the complete eligible set, so stale edges, a missing capability or key, a draining box refuse) and `Worker.Relocatable` (the record's shape) at `Start`; a sidecar family is refused outright. | Task 2 (`TestExecutorChooserPermitsOnlyTheEligibleSet`), Task 5 (`TestWorkerStartRefusesAnIneligibleTargetAndANonRelocatableHost`), Task 10 | |
| 335 | | 13 | accepted, narrowed | An empty pin is evidenced only by a journaled release at `final_txid` 0 or a holder that never admitted a writer; any other `ErrEmpty` refuses (ruling 15). A drain of a session-0 host is refused at `Start` (ruling 11 amended, request R4). `FlushDB` reports 0 on `ErrEmpty` after the sync; the verifier requires only the files manifest for an empty pin; the first probe is a cold admission into an empty generation. | Task 1 (`TestRelocationEmptyPinNeedsItsEvidence`), Task 5 (`TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerRefusesToDrainAHostThatNeverHadAWriter`), Tasks 8a to 8c | |
| 336 | | 14 | accepted | `Policy.Check` gains the `reconcile` action for a relocating participant (un-gated); Task 9 is rewritten around three authorities: activation only for the executor, `WithLocal` plus a conditional `SetClaimHold` for a kept row, and no mint for a target before `installing` (ruling 17). Its test drives 6b's `ReconcileAdapter.Install` for the source and the target in every phase. | Task 3 (`TestPolicyLetsAHeldParticipantReconcileItsRow`), Task 9 | |
| 337 | | 15 | accepted | `Authority.SelectExecutor` returns `selected`; without one `Attempt` runs today's `BeginCreate`. `ProductionAuthority.Executors` nil means no selection; the console's chooser is nil unless `Config.RegionHA` (parsed by `strconv.ParseBool`) is on. | Task 2 (`TestCreateIsTodaysWithoutASelection`, `TestProductionAuthoritySelectsOnlyWithAChooser`, `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) | |
| 338 | | 16 | accepted | A live target's exhausted replacement goes through `stuckOn` from `probe-failed`; `ReleaseDeadline` marks a release never journaled stuck in `releasing` and `probe-failed` (ruling 14); `relocationAdmission` checks `stuck` before any phase. | Task 1 (`a stuck source does not wake`), Task 5 (`TestWorkerMarksALiveTargetsThirdReplacementStuck`, `TestWorkerMarksAnOverdueReleaseStuck`) | |
| 339 | |
| 340 | Astra's readings of rulings 1 to 13 stand; ruling 5's exact-files enforcement is finding 5's fix, and ruling 9's promised behaviour is finding 16's. |
| 341 | |
| 342 | ## Requests to the claim and moves sessions |
| 343 | |
| 344 | Interfaces this plan needs that no merged or declared plan supplies. Each gated task names the request it waits for; nothing here is invented silently. |
| 345 | |
| 346 | | # | To | Request | Waited for by | |
| 347 | |---|---|---|---| |
| 348 | | R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. *Rev 15 (round 14 finding 3):* claim §4 and Task 1's phase table also let the **source** admit, cold, in `starting` and `releasing`, but 6b's `runtime.Prerequisites` refuses every non-serving claim (`p.Probe || o.Claim.Op.Kind != KindServing`, 6b's `ClaimDependencyError` naming moves C1). Ask: dispatch a relocating claim's ordinary wake to the existing wake inputs when this box is `op.exec` and the source, in `starting` or `releasing`, with its admission fence open (Task 3's `WakePhase` and `RelocationPolicy`); refuse it once the fence closes or the claim no longer names this box. A relocation target's ordinary wake stays refused; its only wake is the probe. *Rev 15 (round 14 finding 1):* the probe's first admission consumes P's prepared manifest (sequence 0) as the session publisher's baseline, by the digest `prepared_authority.files_manifest_sha256` names (the store has no version-id read, ruling 5). The publisher starts at sequence 1. | Task 8c | |
| 349 | | R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) | |
| 350 | | R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a | |
| 351 | | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | |
| 352 | | R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences), and the regional obligations the fence passed to the region (rev 6: `reassigned`, D4's `reassign_regional`). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | |
| 353 | | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | |
| 354 | | R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | |
| 355 | | R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | |
| 356 | | R10 | claim 6b (and the activator's owner) | The reservation-bearing extension of main's isolated root launch protocol, which 6b's own external completion gate names as required "before any tenant shape can run with claims enabled" and as "not supplied by the external owner adapters" (6b rev 5, "External completion gates"). Until it exists, `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable` for every launch (6b, `Preflight`), and the runtime's `Prerequisites` calls `Roles.Preflight` before any dispatch (6b, `runtime.Prerequisites`). Even with R1 every production probe would therefore be refused. The ask: the protocol binds each supported role launch to the durable claim reservation and preserves main's namespace isolation; `Preflight` admits a relocation probe's `"probe"` work through it, bound to `op.exec`, the operation and `install_stamp`; without the protocol, `Preflight` refuses before any probe attempt is counted. | Task 8c, Task 10 | |
| 357 | | L21 | region HA spec (Paul) | **Open: the durability of the always-continuous items between an app minting them and their first publish.** §5.6 has every cold admission copy its manifest before the app starts, and H6 makes `instance.key`, `report.token` and the env files continuous. Continuous is not synchronous: an app can mint `instance.key`, write database rows sealed with it, and replicate them before the publisher's first manifest with the key lands. A box terminated in that window leaves a nonempty database whose key exists nowhere. Rev 18's pinner refuses that case and the operation goes `stuck` for an operator (ruling 25), but the database is not recoverable by any relocation. Closing the window needs an upstream durability contract, and there are two candidates, neither chosen here: the platform pre-provisions `instance.key` (and `report.token`) into the files directory and publishes them before the app's first start, or the replicator holds its first upload until the first manifest carrying the key has landed. **Decided 2026-10-09 (Paul): the platform pre-provisions the key.** At creation the platform generates it on the box (CSPRNG), writes it create-only (never overwriting) into the instance's overlay env file as `CARLOS_INSTANCE_KEY`, and publishes it before the app's first start. That file is platform-managed and always continuous. Existing instances get one forced publish before region HA is enabled. The app side is a framework contract, requested as rastrillo/rastrillo #66: detect the platform, use the app's own existing key first and the platform's only when it has none, never mint on the platform, and keep today's behaviour off it. The platform side is built only once #66 agrees the contract. The pinner's refusal (ruling 25) stays as the backstop. Residual: an app that later replaces its key reopens the window for the new key. | ruling 25 | |
| 358 | | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | |
| 359 | |
| 360 | ## Rulings made while planning |
| 361 | |
| 362 | Each was a conflict between the authorities or a gap in them. Each says what it costs if it is wrong. |
| 363 | |
| 364 | 1. **Relocation lives in `internal/hostclaim`.** Main already has `KindRelocating`, its phase validator and `ConvertMovingToRelocating` there, and Plan 6b's boundary table gives "`relocating`, released coverage, once-selected executor and two-box witnesses" to region HA. One implementation per concept: no second package grows claim transitions. *Cost if wrong:* the claim session moves the functions; the names stay. |
| 365 | 2. **The relocation release's stop request names the obligation `relocation-release`.** Root's `FenceAndStop` requires an obligation component, and 6b's `collectCapturedStop` builds only retirement stops (it looks for this box's `park-local` under the operation). A release retires nothing. `host.StopReceiptReleases` accepts only receipts carrying `relocation-release`, so a retirement stop can never be offered as a release, nor the reverse. *Cost if wrong:* a one-line constant, and the 6b session may prefer another literal. |
| 366 | 3. **Discharge stays off until the roster records an enrollment time.** `boxvolumes.Discharge` needs the enrollment time (§24 item 4) and `hostactors.Box` carries none. The production `Discharge` provider returns "the roster records no enrollment time", so a fenced box's `park-local` chores stay open for an operator. H19: this costs cleanup, never recovery. *Ask of the claim session:* add `enrolled_at` to box entries (D7); then the provider reads `boxvolumes.List` and calls `boxvolumes.Discharge`. |
| 367 | 4. **A box-started relocation needs claim §5.1's app-admission registration**, with a `{box, enrollment, controller}` owner. Main has no `apps/<acct>/<app>/admission.json` registry yet. `hostrelocate.Worker` takes an `Admission` interface and refuses to start without one (`ErrUnsupported`). Task 10 is gated on claim Plan 5's app gate as well as 6b. *Cost if wrong:* none until that lands; nothing starts. |
| 368 | 5. **The pinned recovery set is an immutable object** at `control/relocations/<host>/<op>/pin-<claim gen>.json` (rev 2, ruling 16: the generation it was selected against), written create-only and named in the journal by key and SHA-256 (spec §18's first correction: the store has no version-id read, so content is identified by digest). The files pin names its manifest by prefix, sequence and digest. *Cost if wrong:* a deny statement for that prefix in `deployment-buckets` is a follow-up; a rewritten pin is already caught, because every reader checks the journaled digest. |
| 369 | 6. **`installed` is the worker's CAS**, made once a `Prepared` verifier reads P's install manifest complete (§5.8's phase table). The target enters `probing` itself. *Cost if wrong:* who makes one CAS; the claim rules are unchanged. |
| 370 | 7. **A converted move names its target in `pinning`** (`hostclaim.NameTarget`), before `installing`, so the target acknowledges (D6) before it is handed `op.exec` (moves §7.10: "the replacement box `choose()` names when the relocation leaves `pinning`"). |
| 371 | 8. **The final TXID N is read from the remote after `SyncAndWait` returns with every writer stopped**, as the newest level-0 `MaxTXID` (§5.8 release step 2). With writers stopped and the sync acknowledged, the stopped database's position and that listing agree; a sync error is returned, never swallowed (claim P6-D4). |
| 372 | 9. **`stuck` is a journal entry, not a phase.** The claim says exhaustion "records `stuck` without resetting the count"; `validateRelocating` lists no `stuck` phase. `MarkStuck` writes `journal.stuck` and every relocation transition refuses afterwards; it refuses between `installed` and a probe. |
| 373 | 10. **The probe deadline starts at the `probing` CAS** (`journal.probing_since`), and the worker moves a live target to `probe-failed` 10 minutes later. |
| 374 | 11. **A host whose source never had a writer** (session 0, `writer_box` empty) is relocated with coverage `""`, which declares nothing (claim §6.4a: "a created incarnation's first admission … declares nothing"), never `released`. *Rev 2:* only by failover. A drain of such a host is refused at `Worker.Start`: root refuses stop evidence for writer session 0 (`validateRequest`), and 6b's `collectCapturedStop` waits for claim Plan 5's no-process witness (R4). Its failover pin is an evidenced empty one (ruling 15). *Cost if wrong:* a box with never-woken hosts cannot be drained until R4 lands; nothing is lost. |
| 375 | 12. **L18's owner.** The coordination post on #236 says moves owns `internal/sim/hostlife`; claim rev 6.30 §1.1 calls the two-box model "region HA's own … as part of region HA's plan"; the moves conflicts note defers it to "a joint round with the region HA session". This plan writes no model and ends with a merge gate that names the cases; whichever session builds the model, relocation does not merge until that round passes for a one-box and a two-box region. |
| 376 | 13. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1). *Corrected in rev 15 (round 14 finding 1):* until rev 14 this plan did not follow that binding, because it copied the source manifest's sequence into P. Under claim rev 6.30 (R18-1) and moves rev 7.22 §7.6, P's prepared manifest is create-only at sequence 0 and bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. The source pin's sequence is provenance only (`PreparedAuthority.SourceFilesSeq`). `CopyPinnedManifest` writes sequence 0, and `ToInstalled` requires it and binds the stamp. Nothing in L14 to L17 changed against it. |
| 377 | 14. **The release deadline is 10 minutes** (`hostrelocate.ReleaseDeadline`, rev 2). §5.8 says a flush that never reaches N, and a `probe-failed` target that cannot finish its release, go to `stuck`, and names no bound. `ToReleasing` and `ToProbeFailed` journal when the phase began; a worker marks the operation stuck once that long has passed with no release journaled. The host stays where it is: nothing moves away from a live box without its stop evidence. *Cost if wrong:* a number; the stuck state and the operator path are the claim's. |
| 378 | 15. **An empty pin needs evidence a writer can give** (rev 2; claim §6.4a "Empty bootstrap": no LTX file exists until the first commit, so a listing that finds nothing is a definite observation but says nothing about who wrote). A pin is `Empty` only when the journaled release reports `final_txid` 0 (the stopped writer flushed and found nothing), or the holder never admitted a writer (session 0, failover). A failover of a host whose session is above 0 with an empty listing refuses and waits for an operator; `JournalPin` enforces it, the pinner never writes it. *Cost if wrong:* that one case recovers by hand until R6 names its evidence. |
| 379 | 16. **A pin is bound to the claim generation it was selected against** (rev 2; B2-3). Its key is `control/relocations/<host>/<op>/pin-<gen>.json` and `JournalPin` requires `gen` to be the observed claim's. A pin whose journal CAS lost is abandoned where it lies (never referenced, cheap, reaped by the storage follow-up of ruling 5); the next pass selects afresh at the new generation. Every bound `JournalPin` checks (the release's positions, the failover window) is checked before the object is written, so a doomed pin is never published. *Cost if wrong:* abandoned objects under a prefix nothing reads. |
| 380 | 17. **Reconcile has three authorities for a relocation row** (rev 2; §5.3, 6b's `ClaimOwnership.Valid` needs a stamp). The executor (the source until its fence closes, the target from `installing` on) activates its own installation stamp and mints or keeps its row through the guarded path. A kept row of a box that is no longer the executor (the source after its release, a replaced target) gets its hold through `registry.SetClaimHold` under the flock, conditional on the row's recorded ownership: a hold grants nothing, so it needs no installation authority. A target before `installing` has no installation and mints nothing; the edges file already withholds the host from a box that is not `op.exec` (§24). *Cost if wrong:* a request that reaches a pre-installation target meets the box's unknown-host answer instead of a hold, for the seconds between `starting` and `installing`. |
| 381 | 19. **Probe attempts are counted before they are made** (rev 3; §5.8 "limits": at most three, then `probe-failed`). `BeginProbeAttempt` is one CAS per attempt, keyed by box, so a crash, a lost response or a restart never buys a fourth and a replacement target starts at zero; the participant enters `probe-failed` when refused. *Cost if wrong:* an attempt lost to a crash before its admission still counts; the bound is the spec's. |
| 382 | 20. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. *Rev 6:* a durable record is cleared only straight after that scoped cleanup succeeds in the same call (`Finish` repeats it after `Close`, and on a restart), because hostexec reports a cleanup failure from `Close` once and nil afterwards; the participation is journaled before its controller opens, so a restart finds every record. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's. |
| 383 | 21. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations/<host>/<op>/manifest-<sha256>.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers. |
| 384 | 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| 385 | 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). *Rev 9:* that holds only when the dead target is `op.exec`, or `op.exec` is already "none". While the live source is `op.exec` (`starting`, `releasing`), the CAS consumes the target's fence and records `stuck`. It keeps the source's execution identity, phase and release evidence, because a target's fence is no evidence against the source (claim §4), and root's stop needs the source to remain `op.exec`. Otherwise `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| 386 | 24. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive. |
| 387 | 25. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `<host>.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. *Rev 18 (round 17 finding 1):* an existing manifest is not evidence either. When a writer was admitted and the pinned database is nonempty, the pinned manifest must carry a live `instance.key`, the minimum of H6's always-continuous items. A manifest copied before the app started, with the key minted after and never published, refuses the pin, and the operation is `stuck`. The durability window itself is not closed by this plan. It is the open request L21, for Paul to choose between the platform pre-provisioning the key and the replicator holding until it lands. *Cost if wrong (rev 18):* an app that never writes `instance.key` and has a nonempty database is stuck on every relocation until an operator resolves it; H6 names the key as platform-managed on every plan, so such an app should not exist. |
| 388 | 26. **A drain's probe-failed with no eligible spare is stuck after `ReleaseDeadline`, never sent back to the source** (rev 17, finding F1). §5.8 excludes `to` from the choice after a failed probe, and a relocation's target is never its source (the start CAS, `replaceTargetSlot`). Letting the live source be the "replacement" would put a fresh install of P' into the same host paths that the source's own `park-local` chore retires once the operation completes (§5.8, "What a relocation leaves behind"). The same box would then hold the retiring installation and the new one under one name. So the worker waits for a spare and, `ReleaseDeadline` after `probe_failed_since`, marks the operation `stuck`. This follows the spec's own bound ("It alerts, and the op waits for `carlos ops host resolve`"). The operator's resolve is the claim's: it may return the host to the source, as the model's ruling 13 does. *Cost if wrong:* in a two-box region a failed drain waits ten minutes plus an operator instead of an automatic return. |
| 389 | 27. **Every abandoned prepared generation is parked by the operation that built it** (rev 17, finding F3). §5.8 says "an abandoned `P` is parked by the at-rest sweep". This plan adds an earlier owner: `install` journals every unopened P it replaces (`abandoned_prepared`), and `CompleteRelocation` queues a `park-generation` chore for each in the region, beside its `park-local` chores. The at-rest sweep stays the backstop for an operation that never completes. A P that any admission opened is part of the writer lineage and is never on the list. *Cost if wrong:* one chore per abandoned build. Its objects are outside every live set (a prepared manifest names the source's objects, not P's), so parking them retires nothing live. |
| 390 | 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. |
| 391 | |
| 392 | ## Why this is 1b-next, and what it leaves out |
| 393 | |
| 394 | Plan 1b built every part of rev 5.4 that needed none of L1 to L20. Since then claim Plans 1 to 8b and 5 landed on main (3407d114): the roster (`hostactors.Directory`), the fence writer (`hostfence.TerminationWorker`), the fence reader for claims (`hostclaim.VerifyFence`, `hostfence.ClaimFenceAuthority`), the execution controller (`hostexec`), stop receipts (`host.ClaimStopCoordinator`), the box-started census (`Directory.CensusBoxRelocation`) and H14's start check (`Directory.CheckBoxStart`). This plan uses all of them and duplicates none. |
| 395 | |
| 396 | Left out, with where it goes: |
| 397 | |
| 398 | - **The placer loop that decides what to relocate** (spec 3): reading the termination queue, relocating every host off a dead box in bounded parallel, rebalancing. This plan's worker starts a relocation when asked and replays any it finds; Task 10's pass only replays. |
| 399 | - **Automatic enrollment and the cross-region confirmer** (spec 4, H16, H17). |
| 400 | - **The console's copy of a fence into a D4 loss record and the roster's retirement of the box** (claim D4 "Automatic loss persistence"). The fence CAS keeps the reference and the discharge verdict in the operation journal and puts the box on `lost`; materializing the loss record is the console's, and is listed under "After this plan". |
| 401 | - **Storage policy for `control/relocations/`** (ruling 5). |
| 402 | |
| 403 | ## Global Constraints |
| 404 | |
| 405 | - **Dormant.** Nothing changes on a box or console where `CARLOS_REGION_HA` is unset or not true (`strconv.ParseBool`), and it is read from `/etc/carlos/host.env` by `regionHAEnabled` (`internal/host/boxrecord.go`). With one box per region every rule is a no-op (spec §4, §8 step 2). |
| 406 | - **Activation gate (claim §1.1, L18).** "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) … validate[s] them." Relocation may not merge until Task 11's joint round passes. |
| 407 | - **One implementation per concept; main's wins (#236).** Do not write a termination-fence writer (`hostfence.TerminationWorker` writes `control/box-fences/<enrollment>.json` using `termfence.Verify`). Do not write a roster reader (`hostactors.Directory`). Read fences only through `hostclaim.VerifyFence` (in claim CASes) and `hostactors.ValidateTerminationFence` (before them). |
| 408 | - **H19, verbatim:** "The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores." No volume record gates a fence, a relocation step or an admission. |
| 409 | - **Absence is never evidence (§5.9).** A fence that cannot be read or does not validate is an error, never "not fenced"; a box missing from the roster refuses. |
| 410 | - **Every phase change is one claim CAS conditional on `op.id`, the phase and the ETag (claim C2).** Two workers racing make the same CAS and one gets `hostclaim.ErrConflict`. |
| 411 | - **Exact values.** Failover pin delay **30 seconds** after the fence's `observed_at` (`hostclaim.FailoverPinDelay`); probe deadline **10 minutes** (`hostrelocate.ProbeDeadline`); at most **2** target replacements per operation (`hostclaim.MaxTargetReplacements`); reconcile's kept-row sweep **10 minutes** (H9); release deadline **10 minutes** (`hostrelocate.ReleaseDeadline`, ruling 14); probe attempts **3** per target installation (`hostclaim.MaxProbeAttempts`, ruling 19); create intent keys **`selected_executor`** and **`executor_enrollment`** (claim P5-D6); journal fence key **`fence.<box>`** holding `{"enrollment","key","sha256"}` (claim §7.3); release obligation **`relocation-release`**; pin objects **`control/relocations/<host>/<op>/pin-<claim gen>.json`** (`hostclaim.RecoveryPinKey`, ruling 16); loss ids **`auto-<32 hex>`** over (op, box, fence); the retained manifest **`control/relocations/<host>/<op>/manifest-<sha256>.json`** (ruling 21); the env platform item **`host.env`** (`hostfiles.EnvItem`). |
| 412 | - **Gated tasks.** Tasks 8a, 8b, 8c, 9 and 10 start with a "Gated on" line. Their code is written against the interfaces claim Plan 6b rev 5 declares (`ClaimAssembly`, `ClaimWakeAssembly`, `ClaimWakeInputs`, `CollectClaimStop`, `localStopIdentity`, `hostbox.ReconcileAdapter`, `registry.SetClaimHold`), moves Phase 1 Plan 2 Task 8 declares (`moves.BuildGeneration`, `BuildInputs`, `SealedObject`, the key helpers) and moves Plan 1 Task 7b declares (`RenderHeld`, `MaterializeHeld`), plus the requests R1 to R3 and R7. They were not compiled. When 6b and moves land, re-read their merged code and bind the same contracts to the names they actually have; the un-gated tasks do not change. |
| 413 | - **No new module dependencies. Do not add a method to `release.Store`.** Linux-only syscalls in `*_linux.go` with a `!linux` stub; CI cross-compiles darwin, windows and linux/arm64. |
| 414 | - **Copy.** No user-facing copy. Errors and log lines are operator-facing and follow the surrounding style. If a task finds it needs console or CLI text, stop: it waits for Paul's copy review. |
| 415 | - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| 416 | - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| 417 | - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| 418 | - **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's, rev 17's eight, rev 18's five and rev 19's two (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| 419 | |
| 420 | ## Review Focus |
| 421 | |
| 422 | 1. **`released` coverage naming the wrong predecessor.** A drain's first probe admission declares nothing; if the session it vouches for is not exactly the one whose release was journaled (another box, another session, another generation), a real loss goes unrecorded. Expect refusal. Task 1, `TestRelocationAdmissionRules/released_names_another_predecessor`, `…/released_names_another_predecessor_box`, `…/released_after_P_was_opened`. |
| 423 | 2. **P opened twice.** A restarted probe that restores P again loses everything attempt two replicated (B1-1). Expect every attempt after the first to take a fresh generation, and P refused. Task 1, `TestRelocationAdmissionRules/a_third_attempt_reopens_P`. |
| 424 | 3. **A failover pinned too early.** Requests S3 accepted from the terminated box can still land; a pin taken before `observed_at` + 30 s widens H3's window. Expect the worker to wait and the claim to refuse. Task 1 `TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss`, Task 5 `TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence`. |
| 425 | 4. **A fence that cannot be read taken as no fence.** A garbled `control/box-fences/` object must stop the drain and the create choice, not let them proceed as if the box were alive. Task 2 `TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/an_unreadable_fence`, Task 5 `TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence`. |
| 426 | 5. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. A matching number and path is not enough: the kernel must say the table and the description are shared (rev 7). Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`, `…/an_unshared_thread_at_the_leader's_number_and_path`, `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity`. |
| 427 | |
| 428 | ## File Structure |
| 429 | |
| 430 | | Path | Responsibility | Task | |
| 431 | |---|---|---| |
| 432 | | `internal/hostclaim/relocation.go` | The relocation transitions: start (drain, failover), source loss, barrier, release journal, pin, install, local install receipt, probe, serving, target replacement, stuck; fences consumed in the same CAS | 1 | |
| 433 | | `internal/hostclaim/generation.go`, `claim.go` | `AdmitWriter`'s relocating phase table and `released` coverage (L1, L9); `evidence_refs`, the retained automatic loss (D4) | 1 | |
| 434 | | `internal/hostclaim/create.go` | `ExecutorSelection`, `BeginCreateOn`; `CompleteCreate` never substitutes | 2 | |
| 435 | | `internal/console/server.go`, `cmd/console/main.go` | `Config.RegionHA` from `CARLOS_REGION_HA` | 2 | |
| 436 | | `internal/hostops/executor.go`, `loads.go`, `operations.go`, `backfill.go`, `backfill_controller.go` | `ExecutorChooser` over the roster, the edges snapshot and claim loads, and `Permits`; `Authority.SelectExecutor` before the create CAS, or today's create without one | 2 | |
| 437 | | `internal/hostbox/relocationrows.go`, `wakepolicy.go`, `internal/hostguard/admissionfence.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner, its `reconcile` action and `RelocationPolicy`; the admission-fence reader | 3 | |
| 438 | | `internal/stopevidence/writers*.go` | L10's writers-stopped observation; one process's descriptors on the database | 4 | |
| 439 | | `internal/hostrelocate/worker.go`, `pin.go`, `internal/hostfiles/manifestcheck.go` | The regional worker: start (eligibility, shape), one replayed CAS per step, the pin under the location lease, the release deadline; the pinned recovery set by claim generation, `ReadPin`, `CopyPinnedManifest` | 5 | |
| 440 | | `internal/hostrelocate/participant.go` | A participant box's transitions, ordered over its local effects | 6 | |
| 441 | | `internal/host/relocationreleases.go`, `relocationeffects.go`, `relocationflush.go`, `internal/hostfiles/platform.go` | Stop receipts as release evidence; the release sequence inside root's stop collection (stop writers, observe, flush, record, drain); retained controllers; durable positions; the full files flush; `EnvItem` | 7 | |
| 442 | | `internal/host/relocationassembly.go` | Gated: the release providers bound to 6b's assembly | 8a | |
| 443 | | `internal/host/relocationbuild.go` | Gated: P's build under the lease; the `Prepared` verifier | 8b | |
| 444 | | `internal/host/relocationinstall.go`, `relocationwake.go` | Gated: the local install under its stamp; the probe's inputs | 8c | |
| 445 | | `internal/hostbox/reconcile.go`, 6b's `reconcile_adapter.go` | Gated: held rows and the 10-minute sweep | 9 | |
| 446 | | `internal/console/hostclaimexecutor.go` | The console's create chooser (constructed, not yet used by a door) | 2 | |
| 447 | | `internal/host/relocatepass.go` | Gated: the passes, behind `CARLOS_REGION_HA`, and the create door's chooser | 10 | |
| 448 | |
| 449 | --- |
| 450 | |
| 451 | ### Task 1: The `relocating` operation in the claim library (L1, L2 to L4, L9, L11) |
| 452 | |
| 453 | Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.<box>`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies; the same CAS passes the box's regional work (`park-generation`, `park-replica` owned by `box:<b>`) to `region:<r>`, as `DeclareLost` does (rev 6). A box owing a `park-local` is never named a target (`replaceTargetSlot`, rev 6). |
| 454 | |
| 455 | **Files:** |
| 456 | - Create: `internal/hostclaim/relocation.go` |
| 457 | - Modify: `internal/hostclaim/generation.go` (the `AdmitWriter` kind switch, the coverage switch, a new `relocationAdmission`), `internal/hostclaim/claim.go` (`EvidenceRefs`, validated) |
| 458 | - Test: `internal/hostclaim/relocation_test.go` |
| 459 | |
| 460 | **Interfaces:** |
| 461 | - Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`. |
| 462 | - Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`, `Reassigned`) and `Claim.EvidenceRefs`; `OwesLocal(c, box) bool`, `LocalChoreBoxes(c) []string`, `PreparedOpened(c) bool`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. |
| 463 | - Journal keys later tasks read: `exec_ack.<box>`, `fence.<box>`, `fence.<box>.observed_at`, `discharge.<box>`, `reason`, `releasing_since`, `release`, `release.phase`, `released_by_target`, `pin`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe_attempts.<box>`, `probe`, `probe_failed_since`, `replacements`, `excluded_targets`, `stuck`, `prepared_opened` (rev 12). Claim fields: `evidence_refs`, `loss_refs`. |
| 464 | |
| 465 | - [ ] **Step 1: Write the failing tests** |
| 466 | |
| 467 | `internal/hostclaim/relocation_test.go`: |
| 468 | |
| 469 | ```go |
| 470 | package hostclaim |
| 471 | |
| 472 | import ( |
| 473 | "context" |
| 474 | "crypto/sha256" |
| 475 | "encoding/hex" |
| 476 | "encoding/json" |
| 477 | "errors" |
| 478 | "reflect" |
| 479 | "slices" |
| 480 | "strings" |
| 481 | "testing" |
| 482 | "time" |
| 483 | |
| 484 | "github.com/carlosframework/platform/internal/boxid" |
| 485 | "github.com/carlosframework/platform/internal/release" |
| 486 | "github.com/carlosframework/platform/internal/termfence" |
| 487 | ) |
| 488 | |
| 489 | const ( |
| 490 | relSource = "adi-knitting" |
| 491 | relTarget = "bri-knitting" |
| 492 | relSpare = "cai-knitting" |
| 493 | relSpare2 = "dov-knitting" |
| 494 | ) |
| 495 | |
| 496 | var relObserved = time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) |
| 497 | |
| 498 | func relEnrollment(box string) string { return strings.Repeat(string(box[0]), 32) } |
| 499 | |
| 500 | // relAuthority accepts any fence record for a box in this region. |
| 501 | type relAuthority struct{} |
| 502 | |
| 503 | func (relAuthority) ValidateFence(_ context.Context, f termfence.Record) (FenceBinding, error) { |
| 504 | if f.Enrollment != relEnrollment(f.Box) { |
| 505 | return FenceBinding{}, ErrFence |
| 506 | } |
| 507 | return FenceBinding{Region: "ie", RosterRevision: 9, RosterETag: "roster"}, nil |
| 508 | } |
| 509 | |
| 510 | type relReleases struct{ err error } |
| 511 | |
| 512 | func (r relReleases) VerifyRelease(context.Context, Claim, Release) error { return r.err } |
| 513 | |
| 514 | func putFence(t *testing.T, s release.Store, box string) FenceRef { |
| 515 | t.Helper() |
| 516 | f := termfence.Record{V: 1, Box: box, Enrollment: relEnrollment(box), State: "terminated", |
| 517 | Cloud: boxid.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box}, |
| 518 | ObservedAt: relObserved.Format(time.RFC3339Nano), Via: "describe", RequestID: "req", |
| 519 | WrittenBy: termfence.WrittenBy{Box: "worker", Enrollment: strings.Repeat("e", 32)}} |
| 520 | raw, err := json.Marshal(f) |
| 521 | if err != nil { |
| 522 | t.Fatal(err) |
| 523 | } |
| 524 | sum := sha256.Sum256(raw) |
| 525 | ref := FenceRef{Enrollment: f.Enrollment, Key: "control/box-fences/" + f.Enrollment + ".json", SHA256: hex.EncodeToString(sum[:])} |
| 526 | if _, err := s.Put(context.Background(), ref.Key, raw, ""); err != nil && !errors.Is(err, release.ErrCASConflict) { |
| 527 | t.Fatal(err) |
| 528 | } |
| 529 | return ref |
| 530 | } |
| 531 | |
| 532 | // relServing is a serving claim on relSource whose source has admitted one |
| 533 | // cold writer session (generation g1). |
| 534 | func relServing(t *testing.T) (release.Store, Observed) { |
| 535 | t.Helper() |
| 536 | ctx := context.Background() |
| 537 | s := release.NewDirStore(t.TempDir()) |
| 538 | o := servingClaim(t, ctx, s) |
| 539 | o, err := Update(ctx, s, o, AdmitWriter(Admission{Box: relSource, Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| 540 | if err != nil { |
| 541 | t.Fatal(err) |
| 542 | } |
| 543 | return s, o |
| 544 | } |
| 545 | |
| 546 | func relStart(reason string, fence *FenceRef) RelocationStart { |
| 547 | return RelocationStart{OpID: "rel-1", By: "worker", Now: relObserved, Region: "ie", Reason: reason, |
| 548 | SourceBox: relSource, SourceEnrollment: relEnrollment(relSource), |
| 549 | TargetBox: relTarget, TargetEnrollment: relEnrollment(relTarget), |
| 550 | DirectoryRevision: 9, AdmissionID: "adm-1", Fence: fence} |
| 551 | } |
| 552 | |
| 553 | func step(t *testing.T, s release.Store, o Observed, tr func(*Claim) error) Observed { |
| 554 | t.Helper() |
| 555 | next, err := Update(context.Background(), s, o, tr) |
| 556 | if err != nil { |
| 557 | t.Fatal(err) |
| 558 | } |
| 559 | return next |
| 560 | } |
| 561 | |
| 562 | func ack(box string) func(*Claim) error { |
| 563 | return AcknowledgeExecution(box, ExecutionAcknowledgement{Op: "rel-1", Enrollment: relEnrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: relObserved.Add(time.Hour)}) |
| 564 | } |
| 565 | |
| 566 | // relPin is a pin selected against o: its key names o's claim generation. |
| 567 | func relPin(o Observed, maxTXID uint64, seq int64, at time.Time) RecoveryPin { |
| 568 | return RecoveryPin{Key: RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen), SHA256: strings.Repeat("a", 64), Generation: "g1", MaxTXID: maxTXID, FilesManifest: "instances/x/files/g1/manifest.json", FilesSeq: seq, PinnedAt: at, Empty: maxTXID == 0} |
| 569 | } |
| 570 | |
| 571 | func relPrepared(gen string, maxTXID uint64, filesSeq int64) PreparedAuthority { |
| 572 | a := PreparedAuthority{Generation: gen, SnapshotTXID: maxTXID, FilesSeq: 0, SourceFilesSeq: filesSeq, FilesManifestSHA256: strings.Repeat("f", 64)} |
| 573 | if maxTXID > 0 { |
| 574 | a.InstallManifest, a.SHA256 = "m", strings.Repeat("c", 64) |
| 575 | } |
| 576 | return a |
| 577 | } |
| 578 | |
| 579 | func localInstall(c *Claim) LocalInstall { |
| 580 | return LocalInstall{Box: c.Op.Exec, Enrollment: relEnrollment(c.Op.Exec), InstallStamp: journalUint(c.Op, "install_stamp"), FilesManifestVersion: 3, JournalSHA256: strings.Repeat("b", 64)} |
| 581 | } |
| 582 | |
| 583 | // toProbing drives an operation at pinning (with its pin) through |
| 584 | // installing, installed and the local install receipt into probing. |
| 585 | func toProbing(t *testing.T, s release.Store, o Observed, prepared string, maxTXID uint64) Observed { |
| 586 | t.Helper() |
| 587 | target := Target{Box: o.Claim.Op.Intent["target_box"], Enrollment: o.Claim.Op.Intent["target_enrollment"]} |
| 588 | o = step(t, s, o, ToInstalling("rel-1", target, prepared)) |
| 589 | var pin RecoveryPin |
| 590 | readJournal(o.Claim.Op, "pin", &pin) |
| 591 | o = step(t, s, o, ToInstalled("rel-1", relPrepared(prepared, maxTXID, pin.FilesSeq))) |
| 592 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 593 | return step(t, s, o, ToProbing("rel-1", relObserved)) |
| 594 | } |
| 595 | |
| 596 | func TestRelocationDrainEndToEnd(t *testing.T) { |
| 597 | ctx := context.Background() |
| 598 | s, o := relServing(t) |
| 599 | o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) |
| 600 | if err != nil { |
| 601 | t.Fatal(err) |
| 602 | } |
| 603 | if c := o.Claim; c.Op.Exec != relSource || c.Op.Phase != PhaseStarting || !slices.Equal(c.Op.Participants, []string{relSource, relTarget}) || c.Op.Intent["stamp_from"] == "" { |
| 604 | t.Fatalf("start: %+v", c.Op) |
| 605 | } |
| 606 | if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) { |
| 607 | t.Fatalf("barrier without acknowledgements: %v", err) |
| 608 | } |
| 609 | o = step(t, s, o, ack(relSource)) |
| 610 | o = step(t, s, o, ack(relTarget)) |
| 611 | o = step(t, s, o, ToReleasing("rel-1", relObserved)) |
| 612 | if o.Claim.Op.Journal["releasing_since"] == "" { |
| 613 | t.Fatal("releasing did not start its deadline") |
| 614 | } |
| 615 | h := o.Claim.Holder |
| 616 | rel := Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "control/stop/r.json", ReceiptSHA256: strings.Repeat("d", 64)} |
| 617 | o = step(t, s, o, JournalRelease("rel-1", rel)) |
| 618 | if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{err: errors.New("bad receipt")}, nil); err == nil { |
| 619 | t.Fatal("pinned on an unverified receipt") |
| 620 | } |
| 621 | if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil { |
| 622 | t.Fatal(err) |
| 623 | } |
| 624 | stamp := o.Claim.Holder.Stamp |
| 625 | if o.Claim.Op.Exec != "none" || o.Claim.Op.Phase != PhasePinning { |
| 626 | t.Fatalf("pinning: %+v", o.Claim.Op) |
| 627 | } |
| 628 | if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 39, 7, relObserved))); !errors.Is(err, ErrRelocation) { |
| 629 | t.Fatalf("pin short of the release: %v", err) |
| 630 | } |
| 631 | stale := relPin(o, 40, 7, relObserved) |
| 632 | stale.Key = RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen-1) |
| 633 | if _, err := Update(ctx, s, o, JournalPin("rel-1", stale)); !errors.Is(err, ErrRelocation) { |
| 634 | t.Fatalf("a pin selected against an earlier observation was journaled: %v", err) |
| 635 | } |
| 636 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved))) |
| 637 | if o.Claim.Holder.Stamp != stamp { |
| 638 | t.Fatal("pinning changed the stamp") |
| 639 | } |
| 640 | o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) |
| 641 | for name, a := range map[string]PreparedAuthority{ |
| 642 | "no files manifest": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40}, |
| 643 | "another files seq": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40, FilesSeq: 6, FilesManifestSHA256: strings.Repeat("f", 64)}, |
| 644 | "no install manifest": {Generation: "P1", SnapshotTXID: 40, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)}, |
| 645 | "another snapshot txid": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 41, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)}, |
| 646 | } { |
| 647 | if _, err := Update(ctx, s, o, ToInstalled("rel-1", a)); !errors.Is(err, ErrRelocation) { |
| 648 | t.Fatalf("%s: installed on an incomplete build: %v", name, err) |
| 649 | } |
| 650 | } |
| 651 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| 652 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 653 | o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| 654 | install := journalUint(o.Claim.Op, "install_stamp") |
| 655 | if o.Claim.Op.Exec != relTarget || o.Claim.Holder.Stamp != install || !slices.Contains(o.Claim.Touched.Boxes, relTarget) { |
| 656 | t.Fatalf("installation: exec %s stamp %d install %d", o.Claim.Op.Exec, o.Claim.Holder.Stamp, install) |
| 657 | } |
| 658 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 659 | Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| 660 | last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] |
| 661 | if last.Coverage != "released" || last.WritesLost != nil || o.Claim.Holder.Stamp != install { |
| 662 | t.Fatalf("released admission: %+v stamp %d", last, o.Claim.Holder.Stamp) |
| 663 | } |
| 664 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session})) |
| 665 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 666 | c := o.Claim |
| 667 | if c.Op.Kind != KindServing || c.Op.ID != "" || c.Op.Exec != relTarget || !slices.Equal(c.Op.Participants, []string{relTarget}) { |
| 668 | t.Fatalf("serving: %+v", c.Op) |
| 669 | } |
| 670 | if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-local" || c.Obligations[0].Box != relSource || c.Obligations[0].Op != "rel-1" { |
| 671 | t.Fatalf("the source's park-local: %+v", c.Obligations) |
| 672 | } |
| 673 | } |
| 674 | |
| 675 | func TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss(t *testing.T) { |
| 676 | ctx := context.Background() |
| 677 | for _, qualified := range []bool{false, true} { |
| 678 | s, o := relServing(t) |
| 679 | o = step(t, s, o, func(c *Claim) error { // a chore the dead source already owed |
| 680 | c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource}) |
| 681 | return nil |
| 682 | }) |
| 683 | ref := putFence(t, s, relSource) |
| 684 | sourceSession := o.Claim.Holder.Session |
| 685 | in := relStart(ReasonFailover, &ref) |
| 686 | in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}} |
| 687 | o, err := StartRelocation(ctx, s, o, in, relAuthority{}) |
| 688 | if err != nil { |
| 689 | t.Fatal(err) |
| 690 | } |
| 691 | c := o.Claim |
| 692 | if c.Op.Exec != "none" || !slices.Equal(c.Op.Participants, []string{relTarget}) || !slices.Contains(c.Lost, relSource) || c.Op.Journal["fence."+relSource] == "" { |
| 693 | t.Fatalf("failover start: %+v lost %v", c.Op, c.Lost) |
| 694 | } |
| 695 | if owes := slices.ContainsFunc(c.Obligations, func(o Obligation) bool { return o.Box == relSource }); owes == qualified { |
| 696 | t.Fatalf("discharge qualified=%v left chores %+v", qualified, c.Obligations) |
| 697 | } |
| 698 | if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); err == nil { |
| 699 | t.Fatal("a failover entered releasing") |
| 700 | } |
| 701 | o = step(t, s, o, ack(relTarget)) |
| 702 | o = step(t, s, o, PinFailover("rel-1")) |
| 703 | if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(29*time.Second)))); !errors.Is(err, ErrRelocation) { |
| 704 | t.Fatalf("pinned inside the 30-second window: %v", err) |
| 705 | } |
| 706 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay)))) |
| 707 | o = toProbing(t, s, o, "P1", 40) |
| 708 | probe := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 709 | Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"} |
| 710 | if _, err := Update(ctx, s, o, AdmitWriter(probe)); !errors.Is(err, ErrWrongState) { |
| 711 | t.Fatalf("a failover admission claimed released coverage: %v", err) |
| 712 | } |
| 713 | probe.Coverage = "" |
| 714 | o = step(t, s, o, AdmitWriter(probe)) |
| 715 | last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] |
| 716 | if last.WritesLost == nil || last.WritesLost.Box != relSource || last.WritesLost.Generation != "g1" { |
| 717 | t.Fatalf("failover must declare the source's session: %+v", last) |
| 718 | } |
| 719 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session})) |
| 720 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 721 | owes := slices.ContainsFunc(o.Claim.Obligations, func(ob Obligation) bool { return ob.Kind == "park-local" && ob.Box == relSource }) |
| 722 | if owes == qualified { |
| 723 | t.Fatalf("qualified=%v: source chores at serving %+v", qualified, o.Claim.Obligations) |
| 724 | } |
| 725 | // D4: the loss outlives the operation, with the console still to |
| 726 | // materialize it (nothing here stands in for the console). |
| 727 | c = o.Claim |
| 728 | if len(c.EvidenceRefs) != 1 || len(c.LossRefs) != 1 || c.EvidenceRefs[0].ID != c.LossRefs[0] { |
| 729 | t.Fatalf("retained losses after serving: %+v refs %v", c.EvidenceRefs, c.LossRefs) |
| 730 | } |
| 731 | l := c.EvidenceRefs[0] |
| 732 | if l.Box != relSource || l.Fence != ref || l.ObservedAt != relObserved.Format(time.RFC3339Nano) || l.Op != "rel-1" || l.Role != "source" || |
| 733 | l.Session != sourceSession || l.Generation != "g1" || l.Discharge.Qualified != qualified || l.Timers != TimersUnknown || l.Reported { |
| 734 | t.Fatalf("retained loss: %+v", l) |
| 735 | } |
| 736 | if c.Op.Journal != nil || !slices.Contains(c.Lost, relSource) { |
| 737 | t.Fatalf("serving kept the journal, or lost the box: %+v %v", c.Op, c.Lost) |
| 738 | } |
| 739 | } |
| 740 | } |
| 741 | |
| 742 | func TestRelocationSourceLossBecomesFailover(t *testing.T) { |
| 743 | ctx := context.Background() |
| 744 | for _, tc := range []struct { |
| 745 | name string |
| 746 | prepare func(*testing.T, release.Store, Observed) Observed |
| 747 | wantPhase string |
| 748 | wantExec string |
| 749 | wantParts []string |
| 750 | }{ |
| 751 | {"starting before acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return o }, PhaseStarting, "none", []string{relTarget}}, |
| 752 | {"starting after acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return step(t, s, o, ack(relSource)) }, PhaseStarting, "none", []string{relTarget}}, |
| 753 | {"releasing before its receipt", func(t *testing.T, s release.Store, o Observed) Observed { |
| 754 | o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) |
| 755 | return step(t, s, o, ToReleasing("rel-1", relObserved)) |
| 756 | }, PhasePinning, "none", []string{relTarget}}, |
| 757 | {"releasing after its receipt", func(t *testing.T, s release.Store, o Observed) Observed { |
| 758 | o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) |
| 759 | o = step(t, s, o, ToReleasing("rel-1", relObserved)) |
| 760 | h := o.Claim.Holder |
| 761 | return step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 1, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 762 | }, PhaseReleasing, relSource, []string{relSource, relTarget}}, |
| 763 | } { |
| 764 | t.Run(tc.name, func(t *testing.T) { |
| 765 | s, o := relServing(t) |
| 766 | o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) |
| 767 | if err != nil { |
| 768 | t.Fatal(err) |
| 769 | } |
| 770 | o = tc.prepare(t, s, o) |
| 771 | stamp := o.Claim.Holder.Stamp |
| 772 | o, err = FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{}) |
| 773 | if err != nil { |
| 774 | t.Fatal(err) |
| 775 | } |
| 776 | c := o.Claim |
| 777 | if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || !slices.Equal(c.Op.Participants, tc.wantParts) || c.Holder.Stamp != stamp || !slices.Contains(c.Lost, relSource) { |
| 778 | t.Fatalf("phase %s exec %s parts %v stamp %d lost %v", c.Op.Phase, c.Op.Exec, c.Op.Participants, c.Holder.Stamp, c.Lost) |
| 779 | } |
| 780 | if c.Op.Intent["reason"] != ReasonDrain { |
| 781 | t.Fatal("the intent was rewritten") |
| 782 | } |
| 783 | }) |
| 784 | } |
| 785 | } |
| 786 | |
| 787 | // relAtPhase starts a drain and drives it to phase. "probing+P" is probing |
| 788 | // after the first admission opened P. |
| 789 | func relAtPhase(t *testing.T, phase string) (release.Store, Observed) { |
| 790 | t.Helper() |
| 791 | ctx := context.Background() |
| 792 | s, o := relServing(t) |
| 793 | o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) |
| 794 | if err != nil { |
| 795 | t.Fatal(err) |
| 796 | } |
| 797 | o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) |
| 798 | if phase == PhaseStarting { |
| 799 | return s, o |
| 800 | } |
| 801 | o = step(t, s, o, ToReleasing("rel-1", relObserved)) |
| 802 | if phase == PhaseReleasing { |
| 803 | return s, o |
| 804 | } |
| 805 | h := o.Claim.Holder |
| 806 | o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 807 | if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil { |
| 808 | t.Fatal(err) |
| 809 | } |
| 810 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved))) |
| 811 | if phase == PhasePinning { |
| 812 | return s, o |
| 813 | } |
| 814 | o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) |
| 815 | if phase == PhaseInstalling { |
| 816 | return s, o |
| 817 | } |
| 818 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| 819 | if phase == PhaseInstalled { |
| 820 | return s, o |
| 821 | } |
| 822 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 823 | o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| 824 | if phase == PhaseProbing { |
| 825 | return s, o |
| 826 | } |
| 827 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 828 | Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| 829 | return s, o |
| 830 | } |
| 831 | |
| 832 | func TestRelocationReplacementTable(t *testing.T) { |
| 833 | ctx := context.Background() |
| 834 | for _, tc := range []struct { |
| 835 | at, wantPhase, wantExec string |
| 836 | newStamp, keepP bool |
| 837 | }{ |
| 838 | {PhaseStarting, PhaseStarting, relSource, false, false}, |
| 839 | {PhaseReleasing, PhaseReleasing, relSource, false, false}, |
| 840 | {PhasePinning, PhasePinning, "none", false, false}, |
| 841 | {PhaseInstalling, PhaseInstalling, relSpare, true, false}, |
| 842 | {PhaseInstalled, PhaseInstalled, relSpare, true, true}, |
| 843 | {PhaseProbing, PhaseInstalled, relSpare, true, true}, |
| 844 | {"probing+P", PhasePinning, "none", false, false}, |
| 845 | } { |
| 846 | t.Run(tc.at, func(t *testing.T) { |
| 847 | s, o := relAtPhase(t, tc.at) |
| 848 | before := o.Claim.Holder.Stamp |
| 849 | o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) |
| 850 | if err != nil { |
| 851 | t.Fatal(err) |
| 852 | } |
| 853 | c := o.Claim |
| 854 | if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || c.Op.Intent["target_box"] != relSpare || !slices.Contains(c.Lost, relTarget) || slices.Contains(c.Op.Participants, relTarget) { |
| 855 | t.Fatalf("phase %s exec %s intent %v lost %v parts %v", c.Op.Phase, c.Op.Exec, c.Op.Intent, c.Lost, c.Op.Participants) |
| 856 | } |
| 857 | if (c.Holder.Stamp != before) != tc.newStamp { |
| 858 | t.Fatalf("stamp %d -> %d", before, c.Holder.Stamp) |
| 859 | } |
| 860 | if tc.keepP && c.Op.Journal["prepared"] != "P1" { |
| 861 | t.Fatalf("the unopened P must be reused: %q", c.Op.Journal["prepared"]) |
| 862 | } |
| 863 | if c.Op.Journal["local_install"] != "" { |
| 864 | t.Fatal("a replacement inherited its predecessor's receipt") |
| 865 | } |
| 866 | if tc.wantPhase == PhaseInstalled { |
| 867 | if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); err == nil { |
| 868 | t.Fatal("probing without the replacement's own local install") |
| 869 | } |
| 870 | } |
| 871 | }) |
| 872 | } |
| 873 | } |
| 874 | |
| 875 | func TestRelocationTwoReplacementsThenStuck(t *testing.T) { |
| 876 | ctx := context.Background() |
| 877 | s, o := relAtPhase(t, PhasePinning) |
| 878 | var err error |
| 879 | for _, next := range []string{relSpare, relSpare2} { |
| 880 | old := o.Claim.Op.Intent["target_box"] |
| 881 | if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil { |
| 882 | t.Fatal(err) |
| 883 | } |
| 884 | } |
| 885 | if _, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, Target{"eve-knitting", relEnrollment("eve-knitting")}, "", relAuthority{}); !errors.Is(err, ErrReplacementsExceeded) { |
| 886 | t.Fatalf("third replacement: %v", err) |
| 887 | } |
| 888 | o = step(t, s, o, MarkStuck("rel-1", "two target replacements used")) |
| 889 | if journalUint(o.Claim.Op, "replacements") != 2 { |
| 890 | t.Fatal("stuck reset the count") |
| 891 | } |
| 892 | if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P9")); !errors.Is(err, ErrRelocation) { |
| 893 | t.Fatalf("a stuck operation moved: %v", err) |
| 894 | } |
| 895 | } |
| 896 | |
| 897 | func TestRelocationReplacementInReleasingAcknowledgesBeforePinning(t *testing.T) { |
| 898 | ctx := context.Background() |
| 899 | s, o := relAtPhase(t, PhaseReleasing) |
| 900 | h := o.Claim.Holder |
| 901 | o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 902 | o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) |
| 903 | if err != nil || o.Claim.Op.Phase != PhaseReleasing { |
| 904 | t.Fatalf("replacement in releasing: %v %+v", err, o.Claim.Op) |
| 905 | } |
| 906 | if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); !errors.Is(err, ErrRelocation) { |
| 907 | t.Fatalf("left releasing before the replacement acknowledged: %v", err) |
| 908 | } |
| 909 | o = step(t, s, o, ack(relSpare)) |
| 910 | if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil || o.Claim.Op.Phase != PhasePinning { |
| 911 | t.Fatalf("after the acknowledgement: %v %+v", err, o.Claim.Op) |
| 912 | } |
| 913 | } |
| 914 | |
| 915 | func TestRelocationExhaustedTargetIsFencedIntoAStuckPinning(t *testing.T) { |
| 916 | ctx := context.Background() |
| 917 | exhaust := func(t *testing.T, s release.Store, o Observed) Observed { |
| 918 | var err error |
| 919 | for _, next := range []string{relSpare, relSpare2} { |
| 920 | old := o.Claim.Op.Intent["target_box"] |
| 921 | if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil { |
| 922 | t.Fatal(err) |
| 923 | } |
| 924 | } |
| 925 | return o |
| 926 | } |
| 927 | t.Run("before exhaustion it is a replacement", func(t *testing.T) { |
| 928 | s, o := relAtPhase(t, PhasePinning) |
| 929 | if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, relAuthority{}); !errors.Is(err, ErrRelocation) { |
| 930 | t.Fatalf("fenced as exhausted with replacements left: %v", err) |
| 931 | } |
| 932 | }) |
| 933 | // Round 8 finding 1: while the live source is still op.exec, the |
| 934 | // target's fence consumes the target and records stuck; the source's |
| 935 | // execution identity, phase and release evidence stay. |
| 936 | for _, phase := range []string{PhaseStarting, PhaseReleasing} { |
| 937 | t.Run("dead while the source executes in "+phase, func(t *testing.T) { |
| 938 | s, o := relAtPhase(t, phase) |
| 939 | if phase == PhaseReleasing { |
| 940 | h := o.Claim.Holder |
| 941 | o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 942 | } |
| 943 | o = exhaust(t, s, o) |
| 944 | before := o.Claim |
| 945 | ref := putFence(t, s, relSpare2) |
| 946 | o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}) |
| 947 | if err != nil { |
| 948 | t.Fatal(err) |
| 949 | } |
| 950 | c := o.Claim |
| 951 | if c.Op.Phase != phase || c.Op.Exec != relSource || c.Holder.Stamp != before.Holder.Stamp || c.Op.Journal["stuck"] == "" || |
| 952 | !slices.Contains(c.Op.Participants, relSource) || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) { |
| 953 | t.Fatalf("the live source was revoked: %+v lost %v", c.Op, c.Lost) |
| 954 | } |
| 955 | for _, k := range []string{"release", "release.phase", "releasing_since", "exec_ack." + relSource} { |
| 956 | if c.Op.Journal[k] != before.Op.Journal[k] { |
| 957 | t.Fatalf("%s changed: %q -> %q", k, before.Op.Journal[k], c.Op.Journal[k]) |
| 958 | } |
| 959 | } |
| 960 | if phase == PhaseReleasing && c.Op.Journal["release"] == "" { |
| 961 | t.Fatal("the source's release evidence was dropped") |
| 962 | } |
| 963 | if l := c.EvidenceRefs[len(c.EvidenceRefs)-1]; l.Box != relSpare2 || l.Fence != ref { |
| 964 | t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs) |
| 965 | } |
| 966 | if err := c.Validate(); err != nil { |
| 967 | t.Fatal(err) |
| 968 | } |
| 969 | }) |
| 970 | } |
| 971 | t.Run("dead after its local install, P unopened", func(t *testing.T) { |
| 972 | s, o := relAtPhase(t, PhasePinning) |
| 973 | o = exhaust(t, s, o) |
| 974 | o = step(t, s, o, ack(relSpare2)) |
| 975 | o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P1")) |
| 976 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| 977 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 978 | if _, err := Update(ctx, s, o, MarkStuck("rel-1", "x")); !errors.Is(err, ErrRelocation) { |
| 979 | t.Fatalf("MarkStuck between installed and a probe: %v", err) |
| 980 | } |
| 981 | ref := putFence(t, s, relSpare2) |
| 982 | o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}) |
| 983 | if err != nil { |
| 984 | t.Fatal(err) |
| 985 | } |
| 986 | c := o.Claim |
| 987 | if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) || |
| 988 | c.Op.Journal["local_install"] != "" || c.Op.Journal["pin"] == "" || c.Op.Journal["prepared"] != "P1" || c.Op.Journal["prepared_authority"] == "" { |
| 989 | t.Fatalf("exhausted and dead: %+v lost %v", c.Op, c.Lost) |
| 990 | } |
| 991 | if len(c.EvidenceRefs) != 3 || c.EvidenceRefs[2].Box != relSpare2 || c.EvidenceRefs[2].Fence != ref { |
| 992 | t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs) |
| 993 | } |
| 994 | if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P2")); !errors.Is(err, ErrRelocation) { |
| 995 | t.Fatalf("a stuck operation moved: %v", err) |
| 996 | } |
| 997 | if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}); err == nil { |
| 998 | t.Fatal("the fence was consumed twice") |
| 999 | } |
| 1000 | }) |
| 1001 | t.Run("dead after P was opened", func(t *testing.T) { |
| 1002 | s, o := relAtPhase(t, "probing+P") |
| 1003 | o = exhaust(t, s, o) // after P's admission each replacement is a nested pinning |
| 1004 | o = step(t, s, o, ack(relSpare2)) |
| 1005 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 60, 9, relObserved.Add(FailoverPinDelay)))) |
| 1006 | o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P3")) |
| 1007 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P3", 60, 9))) |
| 1008 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 1009 | o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| 1010 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1011 | Box: relSpare2, Mode: "cold", NewGeneration: "P3", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) |
| 1012 | o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, relAuthority{}) |
| 1013 | if err != nil { |
| 1014 | t.Fatal(err) |
| 1015 | } |
| 1016 | c := o.Claim |
| 1017 | if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || c.Op.Journal["pin"] != "" || c.Op.Journal["prepared"] != "" { |
| 1018 | t.Fatalf("nested stuck pinning: %+v", c.Op) |
| 1019 | } |
| 1020 | }) |
| 1021 | } |
| 1022 | |
| 1023 | func TestRelocationProbeFailureIsANestedRelocation(t *testing.T) { |
| 1024 | ctx := context.Background() |
| 1025 | s, o := relAtPhase(t, "probing+P") |
| 1026 | o = step(t, s, o, ToProbeFailed("rel-1", relObserved)) |
| 1027 | if o.Claim.Op.Journal["probe_failed_since"] == "" { |
| 1028 | t.Fatal("probe-failed did not start its release deadline") |
| 1029 | } |
| 1030 | h := o.Claim.Holder |
| 1031 | rel := Release{ReleasedBy: relTarget, Enrollment: relEnrollment(relTarget), Generation: h.Generation, Session: h.Session, FinalTXID: 55, ManifestSeq: 9, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)} |
| 1032 | o = step(t, s, o, JournalRelease("rel-1", rel)) |
| 1033 | if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relTarget, relEnrollment(relTarget)}); err == nil { |
| 1034 | t.Fatal("the failed target was named again") |
| 1035 | } |
| 1036 | o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relSpare, relEnrollment(relSpare)}) |
| 1037 | if err != nil { |
| 1038 | t.Fatal(err) |
| 1039 | } |
| 1040 | if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Exec != "none" { |
| 1041 | t.Fatalf("nested relocation: %+v", o.Claim.Op) |
| 1042 | } |
| 1043 | if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 54, 9, relObserved))); !errors.Is(err, ErrRelocation) { |
| 1044 | t.Fatalf("pin lost the probe's writes: %v", err) |
| 1045 | } |
| 1046 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 55, 9, relObserved))) |
| 1047 | o = step(t, s, o, ack(relSpare)) |
| 1048 | o = toProbing(t, s, o, "P2", 55) |
| 1049 | // released coverage on P2's opening: the predecessor is the failed |
| 1050 | // target's probe session, whose release this operation journaled. |
| 1051 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1052 | Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| 1053 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| 1054 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 1055 | var owed []string |
| 1056 | for _, ob := range o.Claim.Obligations { |
| 1057 | owed = append(owed, ob.Box) |
| 1058 | } |
| 1059 | slices.Sort(owed) |
| 1060 | if !slices.Equal(owed, []string{relSource, relTarget}) { |
| 1061 | t.Fatalf("park-local owed by the source and the abandoned target: %v", owed) |
| 1062 | } |
| 1063 | } |
| 1064 | |
| 1065 | func TestRelocationAdmissionRules(t *testing.T) { |
| 1066 | ctx := context.Background() |
| 1067 | probe := func(o Observed, gen, coverage string) Admission { |
| 1068 | return Admission{ObservedOpID: "rel-1", ObservedPhase: o.Claim.Op.Phase, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1069 | Box: o.Claim.Op.Exec, Mode: "cold", NewGeneration: gen, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"} |
| 1070 | } |
| 1071 | for _, tc := range []struct { |
| 1072 | name, at string |
| 1073 | admit func(Observed) Admission |
| 1074 | }{ |
| 1075 | {"the target admits in installing", PhaseInstalling, func(o Observed) Admission { return probe(o, "P1", "") }}, |
| 1076 | {"the target admits in installed", PhaseInstalled, func(o Observed) Admission { return probe(o, "P1", "") }}, |
| 1077 | {"the first probe skips P", PhaseProbing, func(o Observed) Admission { return probe(o, "other", "") }}, |
| 1078 | {"P is opened twice", "probing+P", func(o Observed) Admission { return probe(o, "P1", "") }}, |
| 1079 | {"released after P was opened", "probing+P", func(o Observed) Admission { return probe(o, "P3", "released") }}, |
| 1080 | {"warm probe", PhaseProbing, func(o Observed) Admission { |
| 1081 | a := probe(o, "", "") |
| 1082 | a.Mode, a.PredecessorCertified = "warm", true |
| 1083 | return a |
| 1084 | }}, |
| 1085 | {"the source claims released", PhaseStarting, func(o Observed) Admission { return probe(o, "g2", "released") }}, |
| 1086 | } { |
| 1087 | t.Run(tc.name, func(t *testing.T) { |
| 1088 | s, o := relAtPhase(t, tc.at) |
| 1089 | if _, err := Update(ctx, s, o, AdmitWriter(tc.admit(o))); !errors.Is(err, ErrWrongState) { |
| 1090 | t.Fatalf("admitted: %v", err) |
| 1091 | } |
| 1092 | }) |
| 1093 | } |
| 1094 | t.Run("released names another predecessor", func(t *testing.T) { |
| 1095 | s, o := relAtPhase(t, PhaseProbing) |
| 1096 | o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Session++ |
| 1097 | if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) { |
| 1098 | t.Fatalf("admitted: %v", err) |
| 1099 | } |
| 1100 | }) |
| 1101 | t.Run("released names another predecessor box", func(t *testing.T) { |
| 1102 | s, o := relAtPhase(t, PhaseProbing) |
| 1103 | o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Box = relSpare |
| 1104 | if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) { |
| 1105 | t.Fatalf("admitted: %v", err) |
| 1106 | } |
| 1107 | }) |
| 1108 | t.Run("a third attempt reopens P", func(t *testing.T) { |
| 1109 | s, o := relAtPhase(t, "probing+P") |
| 1110 | o = step(t, s, o, AdmitWriter(probe(o, "P3", ""))) // attempt two: a fresh generation |
| 1111 | if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", ""))); !errors.Is(err, ErrWrongState) { |
| 1112 | t.Fatalf("P was opened again: %v", err) |
| 1113 | } |
| 1114 | }) |
| 1115 | t.Run("released outside relocating", func(t *testing.T) { |
| 1116 | s, o := relServing(t) |
| 1117 | if _, err := Update(ctx, s, o, AdmitWriter(Admission{ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSource, Mode: "cold", NewGeneration: "g2", Coverage: "released", Region: "ie"})); !errors.Is(err, ErrWrongState) { |
| 1118 | t.Fatalf("admitted: %v", err) |
| 1119 | } |
| 1120 | }) |
| 1121 | t.Run("the source wakes in starting", func(t *testing.T) { |
| 1122 | s, o := relAtPhase(t, PhaseStarting) |
| 1123 | if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); err != nil { |
| 1124 | t.Fatal(err) |
| 1125 | } |
| 1126 | }) |
| 1127 | t.Run("a stuck source does not wake", func(t *testing.T) { |
| 1128 | s, o := relAtPhase(t, PhaseStarting) |
| 1129 | o = step(t, s, o, MarkStuck("rel-1", "operator")) |
| 1130 | if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); !errors.Is(err, ErrWrongState) { |
| 1131 | t.Fatalf("admitted: %v", err) |
| 1132 | } |
| 1133 | }) |
| 1134 | } |
| 1135 | |
| 1136 | func TestRelocationEmptyPinNeedsItsEvidence(t *testing.T) { |
| 1137 | ctx := context.Background() |
| 1138 | t.Run("a release at txid 0 vouches", func(t *testing.T) { |
| 1139 | s, o := relAtPhase(t, PhaseReleasing) |
| 1140 | h := o.Claim.Holder |
| 1141 | o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 0, ManifestSeq: 0, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 1142 | o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil) |
| 1143 | if err != nil { |
| 1144 | t.Fatal(err) |
| 1145 | } |
| 1146 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved))) |
| 1147 | o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) |
| 1148 | if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 0))); !errors.Is(err, ErrRelocation) { |
| 1149 | t.Fatalf("an empty pin accepted a database build: %v", err) |
| 1150 | } |
| 1151 | if _, err := Update(ctx, s, o, ToInstalled("rel-1", PreparedAuthority{Generation: "P1", FilesSeq: 0})); !errors.Is(err, ErrRelocation) { |
| 1152 | t.Fatalf("an empty P without its files manifest: %v", err) |
| 1153 | } |
| 1154 | step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 0, 0))) |
| 1155 | }) |
| 1156 | t.Run("a release at txid 40 does not", func(t *testing.T) { |
| 1157 | s, o := relAtPhase(t, PhaseReleasing) |
| 1158 | h := o.Claim.Holder |
| 1159 | o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| 1160 | o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil) |
| 1161 | if err != nil { |
| 1162 | t.Fatal(err) |
| 1163 | } |
| 1164 | if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved))); !errors.Is(err, ErrRelocation) { |
| 1165 | t.Fatalf("an empty pin against a release at 40: %v", err) |
| 1166 | } |
| 1167 | }) |
| 1168 | t.Run("a failover of a host that had a writer does not", func(t *testing.T) { |
| 1169 | s, o := relServing(t) |
| 1170 | ref := putFence(t, s, relSource) |
| 1171 | o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{}) |
| 1172 | if err != nil { |
| 1173 | t.Fatal(err) |
| 1174 | } |
| 1175 | o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1")) |
| 1176 | if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay)))); !errors.Is(err, ErrRelocation) { |
| 1177 | t.Fatalf("a listing that found nothing was taken as empty: %v", err) |
| 1178 | } |
| 1179 | }) |
| 1180 | t.Run("a failover of a never-woken host does", func(t *testing.T) { |
| 1181 | s := release.NewDirStore(t.TempDir()) |
| 1182 | o := servingClaim(t, ctx, s) // session 0: no writer was ever admitted |
| 1183 | ref := putFence(t, s, relSource) |
| 1184 | o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{}) |
| 1185 | if err != nil { |
| 1186 | t.Fatal(err) |
| 1187 | } |
| 1188 | o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1")) |
| 1189 | step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay)))) |
| 1190 | }) |
| 1191 | } |
| 1192 | |
| 1193 | func TestRelocationProbeAttemptsAreBounded(t *testing.T) { |
| 1194 | ctx := context.Background() |
| 1195 | s, o := relAtPhase(t, PhaseProbing) |
| 1196 | for i := 0; i < MaxProbeAttempts; i++ { |
| 1197 | o = step(t, s, o, BeginProbeAttempt("rel-1", relTarget)) |
| 1198 | } |
| 1199 | if journalUint(o.Claim.Op, "probe_attempts."+relTarget) != MaxProbeAttempts { |
| 1200 | t.Fatalf("count: %v", o.Claim.Op.Journal) |
| 1201 | } |
| 1202 | if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relTarget)); !errors.Is(err, ErrProbeAttemptsExhausted) { |
| 1203 | t.Fatalf("a fourth attempt: %v", err) |
| 1204 | } |
| 1205 | if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relSpare)); !errors.Is(err, ErrRelocation) { |
| 1206 | t.Fatalf("a non-executor counted an attempt: %v", err) |
| 1207 | } |
| 1208 | // A replacement target has its own count. |
| 1209 | o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) |
| 1210 | if err != nil { |
| 1211 | t.Fatal(err) |
| 1212 | } |
| 1213 | if journalUint(o.Claim.Op, "probe_attempts."+relSpare) != 0 { |
| 1214 | t.Fatal("the replacement inherited attempts") |
| 1215 | } |
| 1216 | } |
| 1217 | |
| 1218 | func TestRelocationDischargeCompletesDiscoveryLikeARetirement(t *testing.T) { |
| 1219 | ctx := context.Background() |
| 1220 | s, o := relServing(t) |
| 1221 | o = step(t, s, o, func(c *Claim) error { |
| 1222 | c.Touched.DiscoveryOp = "disc" |
| 1223 | c.Obligations = append(c.Obligations, Obligation{ID: "d1", Op: "disc", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource}) |
| 1224 | return nil |
| 1225 | }) |
| 1226 | ref := putFence(t, s, relSource) |
| 1227 | in := relStart(ReasonFailover, &ref) |
| 1228 | in.Discharge = &LocalDischarge{Qualified: true} |
| 1229 | o, err := StartRelocation(ctx, s, o, in, relAuthority{}) |
| 1230 | if err != nil { |
| 1231 | t.Fatal(err) |
| 1232 | } |
| 1233 | if len(o.Claim.Obligations) != 0 || !o.Claim.Touched.Complete || o.Claim.Touched.DiscoveryOp != "" { |
| 1234 | t.Fatalf("discharging the final discovery chore: %+v touched %+v", o.Claim.Obligations, o.Claim.Touched) |
| 1235 | } |
| 1236 | } |
| 1237 | |
| 1238 | func TestRelocationLossRetainsAFormerWritersSession(t *testing.T) { |
| 1239 | // The source is fenced after the target's probe admission made the |
| 1240 | // target the writer: the loss still names the source's last session. |
| 1241 | ctx := context.Background() |
| 1242 | s, o := relAtPhase(t, "probing+P") |
| 1243 | before := o.Claim.Holder |
| 1244 | var sourceAdmission AdmissionRecord |
| 1245 | for _, a := range before.Admissions { |
| 1246 | if a.Box == relSource { |
| 1247 | sourceAdmission = a |
| 1248 | } |
| 1249 | } |
| 1250 | o, err := FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{}) |
| 1251 | if err != nil { |
| 1252 | t.Fatal(err) |
| 1253 | } |
| 1254 | l := o.Claim.EvidenceRefs[0] |
| 1255 | if l.Box != relSource || l.Session != sourceAdmission.Session || l.Generation != sourceAdmission.Generation || l.LastAdmission == nil || !reflect.DeepEqual(*l.LastAdmission, sourceAdmission) || |
| 1256 | l.Incarnation != before.Incarnation || l.Stamp != before.Stamp { |
| 1257 | t.Fatalf("retained: %+v, source admission %+v", l, sourceAdmission) |
| 1258 | } |
| 1259 | if err := o.Claim.Validate(); err != nil { |
| 1260 | t.Fatal(err) |
| 1261 | } |
| 1262 | } |
| 1263 | |
| 1264 | func TestRelocationRetainsEveryFencedBoxAcrossCompletion(t *testing.T) { |
| 1265 | // A drain whose target is terminated in installing: the replacement |
| 1266 | // completes, and the dead target's loss is still on the claim. |
| 1267 | ctx := context.Background() |
| 1268 | s, o := relAtPhase(t, PhaseInstalling) |
| 1269 | ref := putFence(t, s, relTarget) |
| 1270 | o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) |
| 1271 | if err != nil { |
| 1272 | t.Fatal(err) |
| 1273 | } |
| 1274 | o = step(t, s, o, ack(relSpare)) |
| 1275 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7))) |
| 1276 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 1277 | o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| 1278 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1279 | Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| 1280 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| 1281 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 1282 | c := o.Claim |
| 1283 | if len(c.EvidenceRefs) != 1 || c.EvidenceRefs[0].Box != relTarget || c.EvidenceRefs[0].Role != "target" || c.EvidenceRefs[0].Session != 0 || !slices.Contains(c.LossRefs, c.EvidenceRefs[0].ID) { |
| 1284 | t.Fatalf("the abandoned target's loss: %+v refs %v", c.EvidenceRefs, c.LossRefs) |
| 1285 | } |
| 1286 | if err := c.Validate(); err != nil { |
| 1287 | t.Fatal(err) |
| 1288 | } |
| 1289 | } |
| 1290 | |
| 1291 | func TestRelocationStartGuard(t *testing.T) { |
| 1292 | serving := func(obs ...Obligation) *Claim { |
| 1293 | return &Claim{Holder: &Holder{Incarnation: "i"}, Op: Op{Kind: KindServing, Exec: relSource}, Obligations: obs} |
| 1294 | } |
| 1295 | for _, tc := range []struct { |
| 1296 | name string |
| 1297 | ob Obligation |
| 1298 | fenced bool |
| 1299 | want error |
| 1300 | }{ |
| 1301 | {"park-generation in the region (L11)", Obligation{Kind: "park-generation", Region: "ie"}, false, nil}, |
| 1302 | {"park-replica in the region", Obligation{Kind: "park-replica", Region: "ie"}, false, ErrPendingChore}, |
| 1303 | {"park-replica elsewhere", Obligation{Kind: "park-replica", Region: "uk"}, false, nil}, |
| 1304 | {"park-local on the target", Obligation{Kind: "park-local", Box: relTarget}, true, ErrPendingChore}, |
| 1305 | {"park-local on a live source", Obligation{Kind: "park-local", Box: relSource}, false, ErrPendingChore}, |
| 1306 | {"park-local on a fenced source", Obligation{Kind: "park-local", Box: relSource}, true, nil}, |
| 1307 | } { |
| 1308 | t.Run(tc.name, func(t *testing.T) { |
| 1309 | if err := RelocationStartGuard(serving(tc.ob), "ie", relSource, relTarget, tc.fenced); !errors.Is(err, tc.want) { |
| 1310 | t.Fatalf("got %v, want %v", err, tc.want) |
| 1311 | } |
| 1312 | }) |
| 1313 | } |
| 1314 | } |
| 1315 | |
| 1316 | func TestRelocationLocalInstallReceiptIsBound(t *testing.T) { |
| 1317 | ctx := context.Background() |
| 1318 | for _, tc := range []struct { |
| 1319 | name string |
| 1320 | mutate func(*LocalInstall) |
| 1321 | }{ |
| 1322 | {"another box", func(l *LocalInstall) { l.Box = relSpare }}, |
| 1323 | {"another enrollment", func(l *LocalInstall) { l.Enrollment = relEnrollment(relSpare) }}, |
| 1324 | {"a later writer stamp", func(l *LocalInstall) { l.InstallStamp++ }}, |
| 1325 | } { |
| 1326 | t.Run(tc.name, func(t *testing.T) { |
| 1327 | s, o := relAtPhase(t, PhaseInstalled) |
| 1328 | l := localInstall(o.Claim) |
| 1329 | tc.mutate(&l) |
| 1330 | if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrRelocation) { |
| 1331 | t.Fatalf("recorded: %v", err) |
| 1332 | } |
| 1333 | }) |
| 1334 | } |
| 1335 | s, o := relAtPhase(t, PhaseInstalled) |
| 1336 | l := localInstall(o.Claim) |
| 1337 | o = step(t, s, o, RecordLocalInstall("rel-1", l)) |
| 1338 | if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrAlreadyRecorded) { |
| 1339 | t.Fatalf("replay: %v", err) |
| 1340 | } |
| 1341 | if _, err := Update(ctx, s, o, MarkStuck("rel-1", "operator")); !errors.Is(err, ErrRelocation) { |
| 1342 | t.Fatalf("stuck between installed and a probe: %v", err) |
| 1343 | } |
| 1344 | // B1-2: probing is refused on another box's receipt, even one in the journal. |
| 1345 | foreign := l |
| 1346 | foreign.Box = relSpare |
| 1347 | raw, _ := json.Marshal(foreign) |
| 1348 | o.Claim.Op.Journal["local_install"] = string(raw) |
| 1349 | if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) { |
| 1350 | t.Fatalf("probing on another box's receipt: %v", err) |
| 1351 | } |
| 1352 | } |
| 1353 | |
| 1354 | // Round 5 finding 1: a spare that still owes a local retirement for the |
| 1355 | // host is never named a replacement target, whatever phase the |
| 1356 | // replacement resumes at (installing and installed hand it op.exec in the |
| 1357 | // same CAS); a clean spare is. |
| 1358 | func TestRelocationReplacementRefusesABoxOwingALocalRetirement(t *testing.T) { |
| 1359 | ctx := context.Background() |
| 1360 | for _, phase := range []string{PhaseStarting, PhaseInstalling, PhaseInstalled} { |
| 1361 | t.Run(phase, func(t *testing.T) { |
| 1362 | s, o := relAtPhase(t, phase) |
| 1363 | o = step(t, s, o, func(c *Claim) error { |
| 1364 | c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSpare, Retirement: "r", Owner: "box:" + relSpare}) |
| 1365 | return nil |
| 1366 | }) |
| 1367 | ref := putFence(t, s, relTarget) |
| 1368 | if _, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}); !errors.Is(err, ErrPendingChore) { |
| 1369 | t.Fatalf("a box owing park-local was named: %v", err) |
| 1370 | } |
| 1371 | if _, err := Update(ctx, s, o, NameTarget("rel-1", Target{relSpare, relEnrollment(relSpare)})); err == nil { |
| 1372 | t.Fatal("NameTarget named a box owing park-local") |
| 1373 | } |
| 1374 | o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare2, relEnrollment(relSpare2)}, "P2", relAuthority{}) |
| 1375 | if err != nil { |
| 1376 | t.Fatal(err) |
| 1377 | } |
| 1378 | c := o.Claim |
| 1379 | if c.Op.Intent["target_box"] != relSpare2 || slices.Contains(c.Op.Participants, relSpare) || slices.Contains(c.Touched.Boxes, relSpare) { |
| 1380 | t.Fatalf("the clean spare was not chosen: %+v touched %v", c.Op, c.Touched.Boxes) |
| 1381 | } |
| 1382 | if phase != PhaseStarting && c.Op.Exec != relSpare2 { |
| 1383 | t.Fatalf("exec %s", c.Op.Exec) |
| 1384 | } |
| 1385 | }) |
| 1386 | } |
| 1387 | } |
| 1388 | |
| 1389 | // Round 5 finding 2: the automatic fence passes the dead box's regional |
| 1390 | // work (park-generation, park-replica owned by box:<b>) to its region in |
| 1391 | // the same CAS, as DeclareLost does for a declared loss, keeping every |
| 1392 | // other field; another box's work is untouched; local discharge, qualified |
| 1393 | // or not, changes nothing about it. |
| 1394 | func TestRelocationFenceReassignsTheDeadBoxsRegionalWork(t *testing.T) { |
| 1395 | ctx := context.Background() |
| 1396 | gen := Obligation{ID: "pg", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0", Retirement: "rg", Owner: "box:" + relSource} |
| 1397 | rep := Obligation{ID: "pr", Op: "old-op", Kind: "park-replica", Region: "uk", Retirement: "rr", Owner: "box:" + relSource} |
| 1398 | other := Obligation{ID: "po", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0b", Retirement: "ro", Owner: "box:" + relSpare} |
| 1399 | local := Obligation{ID: "pl", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "rl", Owner: "box:" + relSource} |
| 1400 | for _, qualified := range []bool{false, true} { |
| 1401 | s, o := relServing(t) |
| 1402 | o = step(t, s, o, func(c *Claim) error { |
| 1403 | c.Obligations = append(c.Obligations, gen, rep, other, local) |
| 1404 | return nil |
| 1405 | }) |
| 1406 | ref := putFence(t, s, relSource) |
| 1407 | in := relStart(ReasonFailover, &ref) |
| 1408 | in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}} |
| 1409 | o, err := StartRelocation(ctx, s, o, in, relAuthority{}) |
| 1410 | if err != nil { |
| 1411 | t.Fatal(err) |
| 1412 | } |
| 1413 | byID := map[string]Obligation{} |
| 1414 | for _, ob := range o.Claim.Obligations { |
| 1415 | byID[ob.ID] = ob |
| 1416 | } |
| 1417 | wantGen, wantRep := gen, rep |
| 1418 | wantGen.Owner, wantRep.Owner = "region:ie", "region:uk" |
| 1419 | if !reflect.DeepEqual(byID["pg"], wantGen) || !reflect.DeepEqual(byID["pr"], wantRep) || !reflect.DeepEqual(byID["po"], other) { |
| 1420 | t.Fatalf("qualified=%v: regional work %+v", qualified, o.Claim.Obligations) |
| 1421 | } |
| 1422 | if _, owes := byID["pl"]; owes == qualified { |
| 1423 | t.Fatalf("qualified=%v: the local chore %+v", qualified, o.Claim.Obligations) |
| 1424 | } |
| 1425 | want := []ObligationRef{{ID: "pg", Op: "old-op", Retirement: "rg"}, {ID: "pr", Op: "old-op", Retirement: "rr"}} |
| 1426 | if len(o.Claim.EvidenceRefs) != 1 || !reflect.DeepEqual(o.Claim.EvidenceRefs[0].Reassigned, want) { |
| 1427 | t.Fatalf("qualified=%v: the retained loss does not name the reassignment: %+v", qualified, o.Claim.EvidenceRefs) |
| 1428 | } |
| 1429 | if err := o.Claim.Validate(); err != nil { |
| 1430 | t.Fatal(err) |
| 1431 | } |
| 1432 | } |
| 1433 | } |
| 1434 | |
| 1435 | // Round 11's sweep: P's opening is journaled with the admission that made |
| 1436 | // it, so it survives AdmitWriter's trim of holder.admissions to eight. The |
| 1437 | // removal below stands in for that trim (generation.go drops a covered, |
| 1438 | // loss-archived head once more than eight are kept); with P's admission |
| 1439 | // gone from the list, P is still opened: never reopened, and the local |
| 1440 | // install is never remade. |
| 1441 | func TestRelocationPOpenedOutlivesTheAdmissionTrim(t *testing.T) { |
| 1442 | ctx := context.Background() |
| 1443 | s, o := relAtPhase(t, "probing+P") |
| 1444 | if o.Claim.Op.Journal["prepared_opened"] != "P1" { |
| 1445 | t.Fatalf("the opening was not journaled: %+v", o.Claim.Op.Journal) |
| 1446 | } |
| 1447 | o = step(t, s, o, func(c *Claim) error { |
| 1448 | c.Holder.Admissions = slices.DeleteFunc(c.Holder.Admissions, func(a AdmissionRecord) bool { return a.Generation == "P1" }) |
| 1449 | return nil |
| 1450 | }) |
| 1451 | if !PreparedOpened(o.Claim) { |
| 1452 | t.Fatal("P forgotten with its admission") |
| 1453 | } |
| 1454 | again := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1455 | Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"} |
| 1456 | if _, err := Update(ctx, s, o, AdmitWriter(again)); !errors.Is(err, ErrWrongState) { |
| 1457 | t.Fatalf("P reopened after the trim: %v", err) |
| 1458 | } |
| 1459 | if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))); err == nil { |
| 1460 | t.Fatal("the local install was remade after P was opened") |
| 1461 | } |
| 1462 | } |
| 1463 | |
| 1464 | // Round 14 finding 1: P's prepared manifest is at sequence 0 whatever the |
| 1465 | // source's; the pin's sequence (7 here) is its provenance. ToInstalled |
| 1466 | // refuses a prepared manifest claimed at the source's sequence, and one |
| 1467 | // built from another source revision, and binds the one it accepts to |
| 1468 | // the installation stamp. |
| 1469 | func TestRelocationPreparedManifestIsSequenceZero(t *testing.T) { |
| 1470 | ctx := context.Background() |
| 1471 | s, o := relAtPhase(t, PhaseInstalling) |
| 1472 | var pin RecoveryPin |
| 1473 | readJournal(o.Claim.Op, "pin", &pin) |
| 1474 | if pin.FilesSeq != 7 { |
| 1475 | t.Fatalf("pin at %d", pin.FilesSeq) |
| 1476 | } |
| 1477 | atSource := relPrepared("P1", 40, 7) |
| 1478 | atSource.FilesSeq = 7 |
| 1479 | if _, err := Update(ctx, s, o, ToInstalled("rel-1", atSource)); !errors.Is(err, ErrRelocation) { |
| 1480 | t.Fatalf("a prepared manifest at the source's sequence: %v", err) |
| 1481 | } |
| 1482 | if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 6))); !errors.Is(err, ErrRelocation) { |
| 1483 | t.Fatalf("a prepared manifest from another source revision: %v", err) |
| 1484 | } |
| 1485 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| 1486 | var a PreparedAuthority |
| 1487 | readJournal(o.Claim.Op, "prepared_authority", &a) |
| 1488 | if a.FilesSeq != 0 || a.SourceFilesSeq != 7 || a.InstallStamp == 0 || a.InstallStamp != journalUint(o.Claim.Op, "install_stamp") { |
| 1489 | t.Fatalf("journaled authority: %+v", a) |
| 1490 | } |
| 1491 | } |
| 1492 | |
| 1493 | // Two-box model finding F3: a target replaced in installing leaves its P |
| 1494 | // unopened; the replacement builds P'. The serving CAS queues a |
| 1495 | // park-generation for P in the region, and none for the opened P'. |
| 1496 | func TestRelocationParksEveryAbandonedPreparedGeneration(t *testing.T) { |
| 1497 | ctx := context.Background() |
| 1498 | s, o := relAtPhase(t, PhaseInstalling) |
| 1499 | o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) |
| 1500 | if err != nil { |
| 1501 | t.Fatal(err) |
| 1502 | } |
| 1503 | var abandoned []string |
| 1504 | if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { |
| 1505 | t.Fatalf("abandoned: %v", abandoned) |
| 1506 | } |
| 1507 | o = step(t, s, o, ack(relSpare)) |
| 1508 | o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7))) |
| 1509 | o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| 1510 | o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| 1511 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1512 | Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| 1513 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| 1514 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 1515 | var parks []string |
| 1516 | for _, ob := range o.Claim.Obligations { |
| 1517 | if ob.Kind == "park-generation" && ob.Region == "ie" && ob.Owner == "region:ie" { |
| 1518 | parks = append(parks, ob.Generation) |
| 1519 | } |
| 1520 | } |
| 1521 | if !slices.Contains(parks, "P1") || slices.Contains(parks, "P2") { |
| 1522 | t.Fatalf("park-generation chores: %v (%+v)", parks, o.Claim.Obligations) |
| 1523 | } |
| 1524 | if err := o.Claim.Validate(); err != nil { |
| 1525 | t.Fatal(err) |
| 1526 | } |
| 1527 | } |
| 1528 | |
| 1529 | // Round 17 finding 3: the target's probe deadline passes before any |
| 1530 | // admission (P1 unopened), it releases in probe-failed, then it is |
| 1531 | // terminated; the replacement's default branch drops P1. It is recorded |
| 1532 | // first, so the completion of the nested relocation parks it. |
| 1533 | func TestRelocationParksAPDroppedFromProbeFailed(t *testing.T) { |
| 1534 | ctx := context.Background() |
| 1535 | s, o := relAtPhase(t, PhaseProbing) |
| 1536 | if PreparedOpened(o.Claim) { |
| 1537 | t.Fatal("P1 was opened") |
| 1538 | } |
| 1539 | o = step(t, s, o, ToProbeFailed("rel-1", relObserved)) |
| 1540 | o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) |
| 1541 | if err != nil { |
| 1542 | t.Fatal(err) |
| 1543 | } |
| 1544 | if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Journal["prepared"] != "" { |
| 1545 | t.Fatalf("nested: %+v", o.Claim.Op) |
| 1546 | } |
| 1547 | var abandoned []string |
| 1548 | if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { |
| 1549 | t.Fatalf("abandoned: %v", abandoned) |
| 1550 | } |
| 1551 | o = step(t, s, o, ack(relSpare)) |
| 1552 | o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay)))) |
| 1553 | o = toProbing(t, s, o, "P2", 40) |
| 1554 | o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| 1555 | Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) |
| 1556 | o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| 1557 | o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| 1558 | parked := false |
| 1559 | for _, ob := range o.Claim.Obligations { |
| 1560 | if ob.Kind == "park-generation" && ob.Generation == "P1" && ob.Region == "ie" { |
| 1561 | parked = true |
| 1562 | } |
| 1563 | } |
| 1564 | if !parked { |
| 1565 | t.Fatalf("P1 was not parked: %+v", o.Claim.Obligations) |
| 1566 | } |
| 1567 | } |
| 1568 | ``` |
| 1569 | |
| 1570 | - [ ] **Step 2: Run them to see them fail** |
| 1571 | |
| 1572 | Run: `nice -n 19 go test ./internal/hostclaim/ -run 'Relocation' -count=1` |
| 1573 | Expected: FAIL to compile: `undefined: RelocationStart`, `undefined: StartRelocation`, and the rest. |
| 1574 | |
| 1575 | - [ ] **Step 3: Write the transitions** |
| 1576 | |
| 1577 | `internal/hostclaim/relocation.go`: |
| 1578 | |
| 1579 | ```go |
| 1580 | package hostclaim |
| 1581 | |
| 1582 | import ( |
| 1583 | "context" |
| 1584 | "crypto/sha256" |
| 1585 | "encoding/hex" |
| 1586 | "encoding/json" |
| 1587 | "errors" |
| 1588 | "fmt" |
| 1589 | "slices" |
| 1590 | "strconv" |
| 1591 | "time" |
| 1592 | |
| 1593 | "github.com/carlosframework/platform/internal/release" |
| 1594 | ) |
| 1595 | |
| 1596 | // Relocation (claim §4 "Same-region relocation", region HA spec 1 §5.8): |
| 1597 | // the reasons, the phases and the bound on target replacements. |
| 1598 | const ( |
| 1599 | ReasonDrain = "drain" |
| 1600 | ReasonRebalance = "rebalance" |
| 1601 | ReasonFailover = "failover" |
| 1602 | |
| 1603 | PhaseStarting = "starting" |
| 1604 | PhaseReleasing = "releasing" |
| 1605 | PhasePinning = "pinning" |
| 1606 | PhaseInstalling = "installing" |
| 1607 | PhaseInstalled = "installed" |
| 1608 | PhaseProbing = "probing" |
| 1609 | PhaseProbeFailed = "probe-failed" |
| 1610 | |
| 1611 | // MaxTargetReplacements is spec 1 §5.8's bound: after two |
| 1612 | // replacements of the target in one operation it goes to stuck. |
| 1613 | MaxTargetReplacements = 2 |
| 1614 | |
| 1615 | // MaxProbeAttempts is §5.8's probe bound: at most three admissions |
| 1616 | // per target installation within the probe deadline; running out is |
| 1617 | // probe-failed. Attempts are counted in the claim before they are |
| 1618 | // made, so a crash or a lost response never buys a fourth. |
| 1619 | MaxProbeAttempts = 3 |
| 1620 | |
| 1621 | // FailoverPinDelay is §5.8's failover pin rule: pin no earlier than |
| 1622 | // the fence's observed_at plus 30 seconds, so requests S3 had already |
| 1623 | // accepted from the terminated box have completed (H3's window). |
| 1624 | FailoverPinDelay = 30 * time.Second |
| 1625 | ) |
| 1626 | |
| 1627 | var ( |
| 1628 | ErrRelocation = errors.New("hostclaim: relocation transition refused") |
| 1629 | ErrReplacementsExceeded = errors.New("hostclaim: relocation has used its target replacements") |
| 1630 | ErrAlreadyRecorded = errors.New("hostclaim: relocation journal entry already recorded") |
| 1631 | ErrProbeAttemptsExhausted = errors.New("hostclaim: the target has used its probe attempts") |
| 1632 | ) |
| 1633 | |
| 1634 | // RelocationStart is the start CAS's intent (spec 1 §5.8, claim §4). Fence |
| 1635 | // is set for a failover only and names the source's termination fence. |
| 1636 | type RelocationStart struct { |
| 1637 | OpID, By string |
| 1638 | Now time.Time |
| 1639 | Region, Reason string |
| 1640 | SourceBox, SourceEnrollment string |
| 1641 | TargetBox, TargetEnrollment string |
| 1642 | DirectoryRevision uint64 |
| 1643 | AdmissionID string |
| 1644 | Fence *FenceRef |
| 1645 | Discharge *LocalDischarge |
| 1646 | } |
| 1647 | |
| 1648 | // LocalDischarge is a fenced box's volume evidence (claim D5 as amended by |
| 1649 | // L3, spec 1 §5.9, H19): the volume records the caller read and judged with |
| 1650 | // boxvolumes.Discharge. When it is nil, or Qualified is false, the box's |
| 1651 | // park-local chores stay open for the operator path. It never gates the |
| 1652 | // fence or the relocation. |
| 1653 | type LocalDischarge struct { |
| 1654 | Qualified bool `json:"qualified"` |
| 1655 | Records []string `json:"records,omitempty"` |
| 1656 | Reason string `json:"reason,omitempty"` |
| 1657 | } |
| 1658 | |
| 1659 | // Release is the journal entry a live box writes after its release |
| 1660 | // sequence (spec 1 §5.8 steps 1 to 4): its stopped database's final TXID, |
| 1661 | // its final files manifest sequence and its full stop receipt. |
| 1662 | type Release struct { |
| 1663 | ReleasedBy string `json:"released_by"` |
| 1664 | Enrollment string `json:"enrollment"` |
| 1665 | Generation string `json:"generation"` |
| 1666 | Session uint64 `json:"session"` |
| 1667 | FinalTXID uint64 `json:"final_txid"` |
| 1668 | ManifestSeq int64 `json:"manifest_seq"` |
| 1669 | Receipt string `json:"receipt"` |
| 1670 | ReceiptSHA256 string `json:"receipt_sha256"` |
| 1671 | } |
| 1672 | |
| 1673 | // ReleaseAuthority verifies a journaled release's full stop receipt: the |
| 1674 | // immutable object at Receipt hashes to ReceiptSHA256 and is complete stop |
| 1675 | // evidence (claim §6.1, P6-D1) for this host, operation, box, incarnation |
| 1676 | // and session. Plan 6b's receipt format is its only implementation. |
| 1677 | type ReleaseAuthority interface { |
| 1678 | VerifyRelease(context.Context, Claim, Release) error |
| 1679 | } |
| 1680 | |
| 1681 | // RecoveryPin is the journal's reference to one pinned recovery set |
| 1682 | // (§5.8 "The pinned recovery set"): the immutable pin object holding the |
| 1683 | // exact versions, its digest, the database position it reaches, the files |
| 1684 | // manifest it pins, and when it was taken. |
| 1685 | type RecoveryPin struct { |
| 1686 | Key string `json:"key"` |
| 1687 | SHA256 string `json:"sha256"` |
| 1688 | Generation string `json:"generation"` |
| 1689 | MaxTXID uint64 `json:"max_txid"` |
| 1690 | FilesManifest string `json:"files_manifest"` |
| 1691 | FilesSeq int64 `json:"files_seq"` |
| 1692 | PinnedAt time.Time `json:"pinned_at"` |
| 1693 | // Empty is the evidenced empty bootstrap (claim §6.4a "Empty |
| 1694 | // bootstrap", spec 1 §5.8 "A host with nothing replicated"): no LTX |
| 1695 | // file exists because no transaction was ever committed. It is legal |
| 1696 | // only when a journaled release reports final_txid 0, or the holder |
| 1697 | // never admitted a writer (session 0). A listing that merely finds |
| 1698 | // nothing is never evidence (JournalPin refuses it). |
| 1699 | Empty bool `json:"empty,omitempty"` |
| 1700 | } |
| 1701 | |
| 1702 | // RecoveryPinKey is where the pin selected against claim generation gen |
| 1703 | // lives: control/relocations/<host>/<op>/pin-<gen>.json. The key binds the |
| 1704 | // object to the exact observation it was selected against, so a pin whose |
| 1705 | // journal CAS lost (the claim moved) is abandoned, never reused, and a |
| 1706 | // retry selects afresh at the new generation (B2-3's round binding). |
| 1707 | func RecoveryPinKey(host, op string, gen uint64) string { |
| 1708 | return "control/relocations/" + host + "/" + op + "/pin-" + strconv.FormatUint(gen, 10) + ".json" |
| 1709 | } |
| 1710 | |
| 1711 | // PreparedAuthority is the `installed` CAS's record of the remote build |
| 1712 | // (§5.8 "Building the prepared generation"): the install manifest of |
| 1713 | // generation P, verified complete by a fresh read. |
| 1714 | type PreparedAuthority struct { |
| 1715 | Generation string `json:"generation"` |
| 1716 | InstallManifest string `json:"install_manifest"` |
| 1717 | SHA256 string `json:"sha256"` |
| 1718 | SnapshotTXID uint64 `json:"snapshot_txid"` |
| 1719 | // FilesSeq and FilesManifestSHA256 identify P's prepared files |
| 1720 | // manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): create-only, |
| 1721 | // at sequence 0, never replaced; FilesManifestSHA256 is the digest of |
| 1722 | // the exact bytes a fresh read verified, the exact version the first |
| 1723 | // admission consumes as its baseline (the session publisher then |
| 1724 | // starts at sequence 1). SourceFilesSeq is provenance, not identity: |
| 1725 | // the pinned source manifest's sequence it was built from, which must |
| 1726 | // be the pin's (rev 15, round 14 finding 1). Required always, an empty |
| 1727 | // bootstrap included: a P without its files manifest is not built. |
| 1728 | FilesSeq int64 `json:"files_seq"` |
| 1729 | SourceFilesSeq int64 `json:"source_files_seq"` |
| 1730 | FilesManifestSHA256 string `json:"files_manifest_sha256"` |
| 1731 | // InstallStamp binds the prepared manifest to the installation that |
| 1732 | // built it ({location, P, op id, installation stamp}): ToInstalled |
| 1733 | // sets it from install_stamp in the CAS that journals the authority. |
| 1734 | InstallStamp uint64 `json:"install_stamp,omitempty"` |
| 1735 | } |
| 1736 | |
| 1737 | // AutomaticLoss is claim §7.3 D4's "Automatic loss persistence": what the |
| 1738 | // transition that consumed a termination fence retains on the claim itself, |
| 1739 | // outside the replaceable operation, until the console materializes the |
| 1740 | // D4 host loss record. It survives CompleteRelocation, free and holder |
| 1741 | // replacement because it lives beside loss_refs, not in op.journal. |
| 1742 | type AutomaticLoss struct { |
| 1743 | ID string `json:"id"` // the loss identity, also in loss_refs |
| 1744 | Box string `json:"box"` |
| 1745 | Enrollment string `json:"enrollment"` |
| 1746 | Fence FenceRef `json:"fence"` |
| 1747 | ObservedAt string `json:"observed_at"` |
| 1748 | Op string `json:"op"` |
| 1749 | Kind string `json:"kind"` |
| 1750 | Phase string `json:"phase"` |
| 1751 | Role string `json:"role"` // "source" | "target" |
| 1752 | // Incarnation and Stamp bind the loss to the holder it was taken |
| 1753 | // against; Generation and Session are the box's last writer session, |
| 1754 | // whether it is still the writer or was superseded, and LastAdmission |
| 1755 | // is that session's admission record when the claim holds one (the |
| 1756 | // consequences the console reports: its coverage and declarations). |
| 1757 | Incarnation string `json:"incarnation"` |
| 1758 | Stamp uint64 `json:"stamp"` |
| 1759 | Generation string `json:"generation,omitempty"` |
| 1760 | Session uint64 `json:"session,omitempty"` |
| 1761 | LastAdmission *AdmissionRecord `json:"last_admission,omitempty"` |
| 1762 | Discharge LocalDischarge `json:"discharge"` |
| 1763 | // Reassigned is the regional work (park-generation, park-replica) |
| 1764 | // the box owned that the same CAS passed to its region (L3, as |
| 1765 | // DeclareLost's reassign_regional): the console's D4 record names it. |
| 1766 | Reassigned []ObligationRef `json:"reassigned,omitempty"` |
| 1767 | Timers TimerKnowledge `json:"timers"` |
| 1768 | // Reported is set by the console once the D4 record holds this loss |
| 1769 | // and its alert work exists. Until then reporting is pending. |
| 1770 | Reported bool `json:"reported,omitempty"` |
| 1771 | } |
| 1772 | |
| 1773 | // automaticLossID is deterministic for one fence consumed by one |
| 1774 | // operation for one box, so a replayed transition names the same loss and |
| 1775 | // the console's materialization deduplicates on it (D4: "Retries |
| 1776 | // deduplicate by the retained loss identity"). |
| 1777 | func automaticLossID(opID, box string, ref FenceRef) string { |
| 1778 | sum := sha256.Sum256([]byte(opID + "\x00" + box + "\x00" + ref.Enrollment + "\x00" + ref.SHA256)) |
| 1779 | return "auto-" + hex.EncodeToString(sum[:16]) |
| 1780 | } |
| 1781 | |
| 1782 | // LocalInstall is one box's local install receipt (§5.8, B1-2, B2-5). |
| 1783 | type LocalInstall struct { |
| 1784 | Box string `json:"box"` |
| 1785 | Enrollment string `json:"enrollment"` |
| 1786 | InstallStamp uint64 `json:"install_stamp"` |
| 1787 | FilesManifestVersion int64 `json:"files_manifest_version"` |
| 1788 | JournalSHA256 string `json:"journal_sha256"` |
| 1789 | } |
| 1790 | |
| 1791 | // Probe is the target's journaled probe result for its admitted session. |
| 1792 | type Probe struct { |
| 1793 | Box string `json:"box"` |
| 1794 | Session uint64 `json:"session"` |
| 1795 | } |
| 1796 | |
| 1797 | // Target names a box for the relocation's target slot. |
| 1798 | type Target struct { |
| 1799 | Box, Enrollment string |
| 1800 | } |
| 1801 | |
| 1802 | func journalJSON(c *Claim, key string, v any) error { |
| 1803 | raw, err := json.Marshal(v) |
| 1804 | if err != nil { |
| 1805 | return err |
| 1806 | } |
| 1807 | if c.Op.Journal == nil { |
| 1808 | c.Op.Journal = map[string]string{} |
| 1809 | } |
| 1810 | c.Op.Journal[key] = string(raw) |
| 1811 | return nil |
| 1812 | } |
| 1813 | |
| 1814 | func readJournal(op Op, key string, v any) bool { |
| 1815 | raw, ok := op.Journal[key] |
| 1816 | return ok && json.Unmarshal([]byte(raw), v) == nil |
| 1817 | } |
| 1818 | |
| 1819 | func journalUint(op Op, key string) uint64 { |
| 1820 | n, _ := strconv.ParseUint(op.Journal[key], 10, 64) |
| 1821 | return n |
| 1822 | } |
| 1823 | |
| 1824 | func requireRelocation(c *Claim, opID string, phases ...string) error { |
| 1825 | if err := RequireOp(opID)(c); err != nil { |
| 1826 | return err |
| 1827 | } |
| 1828 | if c.Holder == nil || c.Op.Kind != KindRelocating || !slices.Contains(phases, c.Op.Phase) { |
| 1829 | return fmt.Errorf("%w: %s/%s", ErrWrongState, c.Op.Kind, c.Op.Phase) |
| 1830 | } |
| 1831 | if c.Op.Journal["stuck"] != "" { |
| 1832 | return fmt.Errorf("%w: operation is stuck", ErrRelocation) |
| 1833 | } |
| 1834 | ensureJournal(c) |
| 1835 | return nil |
| 1836 | } |
| 1837 | |
| 1838 | // ensureJournal makes the journal writable: an empty one round-trips as nil. |
| 1839 | func ensureJournal(c *Claim) { |
| 1840 | if c.Op.Journal == nil { |
| 1841 | c.Op.Journal = map[string]string{} |
| 1842 | } |
| 1843 | } |
| 1844 | |
| 1845 | // requireAcks is the D6 barrier: every current participant has recorded |
| 1846 | // its execution acknowledgement before the operation leaves its phase. |
| 1847 | func requireAcks(c *Claim) error { |
| 1848 | for _, box := range c.Op.Participants { |
| 1849 | if c.Op.Journal["exec_ack."+box] == "" { |
| 1850 | return fmt.Errorf("%w: %s has not acknowledged", ErrRelocation, box) |
| 1851 | } |
| 1852 | } |
| 1853 | return nil |
| 1854 | } |
| 1855 | |
| 1856 | func failover(op Op) bool { |
| 1857 | return op.Intent["reason"] == ReasonFailover || op.Journal["reason"] == ReasonFailover |
| 1858 | } |
| 1859 | |
| 1860 | // RelocationStartGuard is §4's start guard with the relocation exception |
| 1861 | // (L11): a park-generation chore in the region does not block, a |
| 1862 | // park-replica in the region does, a park-local on the target does, and a |
| 1863 | // park-local on a fenced source does not block a failover. |
| 1864 | func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced bool) error { |
| 1865 | if c.Holder == nil || c.Op.Kind != KindServing || c.Op.ID != "" { |
| 1866 | return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind) |
| 1867 | } |
| 1868 | for _, o := range c.Obligations { |
| 1869 | switch { |
| 1870 | case o.Kind == "park-replica" && o.Region == region: |
| 1871 | return fmt.Errorf("%w: park-replica in %s", ErrPendingChore, region) |
| 1872 | case o.Kind == "park-local" && o.Box == target: |
| 1873 | return fmt.Errorf("%w: park-local on %s", ErrPendingChore, target) |
| 1874 | case o.Kind == "park-local" && o.Box == source && !sourceFenced: |
| 1875 | return fmt.Errorf("%w: park-local on %s", ErrPendingChore, source) |
| 1876 | } |
| 1877 | } |
| 1878 | return nil |
| 1879 | } |
| 1880 | |
| 1881 | // OwesLocal reports whether box owes a park-local chore on this claim: a |
| 1882 | // local retirement not yet done (spec 1 §2: such a box is never chosen to |
| 1883 | // install the host, and §4: a park-local on the target refuses). |
| 1884 | func OwesLocal(c *Claim, box string) bool { |
| 1885 | for _, o := range c.Obligations { |
| 1886 | if o.Kind == "park-local" && o.Box == box { |
| 1887 | return true |
| 1888 | } |
| 1889 | } |
| 1890 | return false |
| 1891 | } |
| 1892 | |
| 1893 | // LocalChoreBoxes lists every box that owes a park-local chore on the |
| 1894 | // claim, for a chooser's exclusions. |
| 1895 | func LocalChoreBoxes(c *Claim) []string { |
| 1896 | var out []string |
| 1897 | for _, o := range c.Obligations { |
| 1898 | if o.Kind == "park-local" && o.Box != "" && !slices.Contains(out, o.Box) { |
| 1899 | out = append(out, o.Box) |
| 1900 | } |
| 1901 | } |
| 1902 | return out |
| 1903 | } |
| 1904 | |
| 1905 | // fenceBox puts box on lost with its fence reference (L2, L4), passes the |
| 1906 | // regional work the box owned (park-generation, park-replica owned by |
| 1907 | // box:<box>) to its region, region:<r>, keeping each obligation's id, |
| 1908 | // operation, retirement, region, generation and park scope (L3, exactly |
| 1909 | // as DeclareLost does for a declared loss, lost.go), and, when the volume |
| 1910 | // evidence qualifies (L3), discharges that box's park-local chores for |
| 1911 | // this host, all in the same CAS. It never removes anything else. D4: the |
| 1912 | // same CAS retains the loss outside the operation (evidence_refs, |
| 1913 | // loss_refs) with the box's last writer session, the discharge verdict and |
| 1914 | // timer knowledge "unknown", so the console can materialize the loss record |
| 1915 | // after the operation is long gone. |
| 1916 | func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDischarge) error { |
| 1917 | key := "fence." + box |
| 1918 | raw, err := json.Marshal(ref) |
| 1919 | if err != nil { |
| 1920 | return err |
| 1921 | } |
| 1922 | if old, ok := c.Op.Journal[key]; ok && old != string(raw) { |
| 1923 | return fmt.Errorf("%w: fence reference already bound", ErrFence) |
| 1924 | } |
| 1925 | if c.Op.Journal == nil { |
| 1926 | c.Op.Journal = map[string]string{} |
| 1927 | } |
| 1928 | c.Op.Journal[key] = string(raw) |
| 1929 | c.Op.Journal[key+".observed_at"] = observedAt |
| 1930 | if !slices.Contains(c.Lost, box) { |
| 1931 | c.Lost = append(c.Lost, box) |
| 1932 | } |
| 1933 | var reassigned []ObligationRef |
| 1934 | for i := range c.Obligations { |
| 1935 | o := &c.Obligations[i] |
| 1936 | if (o.Kind == "park-generation" || o.Kind == "park-replica") && o.Owner == "box:"+box { |
| 1937 | if o.Region == "" { |
| 1938 | return fmt.Errorf("%w: regional obligation %q names no region", ErrWrongState, o.ID) |
| 1939 | } |
| 1940 | o.Owner = "region:" + o.Region |
| 1941 | reassigned = append(reassigned, ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}) |
| 1942 | } |
| 1943 | } |
| 1944 | if d == nil { |
| 1945 | d = &LocalDischarge{Reason: "no volume evidence supplied"} |
| 1946 | } |
| 1947 | if err := journalJSON(c, "discharge."+box, d); err != nil { |
| 1948 | return err |
| 1949 | } |
| 1950 | if d.Qualified { |
| 1951 | // Each chore is completed with CompleteObligation's own semantics, |
| 1952 | // so a discovery operation whose final chore this discharges is |
| 1953 | // marked complete exactly as a retirement would (delete.go). |
| 1954 | for { |
| 1955 | var chore *Obligation |
| 1956 | for i := range c.Obligations { |
| 1957 | if c.Obligations[i].Kind == "park-local" && c.Obligations[i].Box == box { |
| 1958 | chore = &c.Obligations[i] |
| 1959 | break |
| 1960 | } |
| 1961 | } |
| 1962 | if chore == nil { |
| 1963 | break |
| 1964 | } |
| 1965 | if err := CompleteObligation(chore.Op, chore.ID)(c); err != nil { |
| 1966 | return err |
| 1967 | } |
| 1968 | } |
| 1969 | } |
| 1970 | return retainAutomaticLoss(c, box, ref, observedAt, *d, reassigned) |
| 1971 | } |
| 1972 | |
| 1973 | // retainAutomaticLoss is D4's retention, written once per (operation, box, |
| 1974 | // fence): a replay finds its record and changes nothing. |
| 1975 | func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, d LocalDischarge, reassigned []ObligationRef) error { |
| 1976 | id := automaticLossID(c.Op.ID, box, ref) |
| 1977 | for _, l := range c.EvidenceRefs { |
| 1978 | if l.ID == id { |
| 1979 | return nil |
| 1980 | } |
| 1981 | } |
| 1982 | role := "target" |
| 1983 | if box == c.Op.Intent["source_box"] { |
| 1984 | role = "source" |
| 1985 | } |
| 1986 | loss := AutomaticLoss{ID: id, Box: box, Enrollment: ref.Enrollment, Fence: ref, ObservedAt: observedAt, |
| 1987 | Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: role, Discharge: d, Reassigned: reassigned, Timers: TimersUnknown} |
| 1988 | if h := c.Holder; h != nil { |
| 1989 | loss.Incarnation, loss.Stamp = h.Incarnation, h.Stamp |
| 1990 | // The box's last writer session: the live one when it is still |
| 1991 | // the writer, otherwise its last admission on this holder. |
| 1992 | if h.WriterBox == box { |
| 1993 | loss.Generation, loss.Session = h.Generation, h.Session |
| 1994 | } |
| 1995 | for i := len(h.Admissions) - 1; i >= 0; i-- { |
| 1996 | if h.Admissions[i].Box == box { |
| 1997 | last := h.Admissions[i] |
| 1998 | loss.LastAdmission = &last |
| 1999 | if loss.Session == 0 { |
| 2000 | loss.Generation, loss.Session = last.Generation, last.Session |
| 2001 | } |
| 2002 | break |
| 2003 | } |
| 2004 | } |
| 2005 | } |
| 2006 | c.EvidenceRefs = append(c.EvidenceRefs, loss) |
| 2007 | if !slices.Contains(c.LossRefs, id) { |
| 2008 | c.LossRefs = append(c.LossRefs, id) |
| 2009 | } |
| 2010 | return nil |
| 2011 | } |
| 2012 | |
| 2013 | func removeParticipant(c *Claim, box string) { |
| 2014 | c.Op.Participants = slices.DeleteFunc(slices.Clone(c.Op.Participants), func(b string) bool { return b == box }) |
| 2015 | } |
| 2016 | |
| 2017 | func addParticipant(c *Claim, box string) { |
| 2018 | if !slices.Contains(c.Op.Participants, box) { |
| 2019 | c.Op.Participants = append(c.Op.Participants, box) |
| 2020 | } |
| 2021 | } |
| 2022 | |
| 2023 | // installedBoxes is every box this operation handed op.exec for an |
| 2024 | // installation; each gets a park-local unless discharged (claim §4: |
| 2025 | // "Every abandoned local installation receives an exact park-local"). |
| 2026 | func noteInstalledBox(c *Claim, box string) error { |
| 2027 | var boxes []string |
| 2028 | readJournal(c.Op, "installed_boxes", &boxes) |
| 2029 | if !slices.Contains(boxes, box) { |
| 2030 | boxes = append(boxes, box) |
| 2031 | } |
| 2032 | return journalJSON(c, "installed_boxes", boxes) |
| 2033 | } |
| 2034 | |
| 2035 | // StartRelocation is the start CAS (spec 1 §5.8, claim §4). A drain or |
| 2036 | // rebalance starts with op.exec on the source; a failover cites the |
| 2037 | // source's termination fence, starts with op.exec "none", keeps the stamp, |
| 2038 | // and puts the source on lost. The caller has already checked the roster |
| 2039 | // (hostactors.Directory.CheckBoxStart), the target's eligibility and the |
| 2040 | // app-admission registration; this CAS records what it checked. |
| 2041 | func StartRelocation(ctx context.Context, s release.Store, observed Observed, in RelocationStart, authority FenceAuthority) (Observed, error) { |
| 2042 | if observed.Absent || observed.Claim == nil || observed.ETag == "" { |
| 2043 | return Observed{}, ErrWrongState |
| 2044 | } |
| 2045 | switch in.Reason { |
| 2046 | case ReasonDrain, ReasonRebalance: |
| 2047 | if in.Fence != nil { |
| 2048 | return Observed{}, fmt.Errorf("%w: only a failover cites a fence", ErrRelocation) |
| 2049 | } |
| 2050 | case ReasonFailover: |
| 2051 | if in.Fence == nil { |
| 2052 | return Observed{}, fmt.Errorf("%w: a failover cites the source's fence", ErrRelocation) |
| 2053 | } |
| 2054 | default: |
| 2055 | return Observed{}, fmt.Errorf("%w: reason %q", ErrRelocation, in.Reason) |
| 2056 | } |
| 2057 | if in.OpID == "" || in.Region == "" || in.AdmissionID == "" || in.DirectoryRevision == 0 || in.Now.IsZero() || |
| 2058 | !validBoxReference(in.SourceBox) || !validBoxReference(in.TargetBox) || in.SourceBox == "none" || in.TargetBox == "none" || |
| 2059 | in.SourceBox == in.TargetBox || in.SourceEnrollment == "" || in.TargetEnrollment == "" { |
| 2060 | return Observed{}, fmt.Errorf("%w: incomplete start", ErrRelocation) |
| 2061 | } |
| 2062 | var observedAt string |
| 2063 | if in.Fence != nil { |
| 2064 | v, err := VerifyFence(ctx, s, authority, *in.Fence) |
| 2065 | if err != nil { |
| 2066 | return Observed{}, err |
| 2067 | } |
| 2068 | if v.record.Box != in.SourceBox || v.record.Enrollment != in.SourceEnrollment || v.binding.Region != in.Region { |
| 2069 | return Observed{}, ErrFence |
| 2070 | } |
| 2071 | observedAt = v.record.ObservedAt |
| 2072 | } |
| 2073 | return Update(ctx, s, observed, func(c *Claim) error { |
| 2074 | if c.Op.Exec != in.SourceBox || slices.Contains(c.Lost, in.TargetBox) { |
| 2075 | return fmt.Errorf("%w: the source is not the executor, or the target is lost", ErrWrongState) |
| 2076 | } |
| 2077 | if err := RelocationStartGuard(c, in.Region, in.SourceBox, in.TargetBox, in.Fence != nil); err != nil { |
| 2078 | return err |
| 2079 | } |
| 2080 | exec, participants := in.SourceBox, []string{in.SourceBox, in.TargetBox} |
| 2081 | if in.Fence != nil { |
| 2082 | exec, participants = "none", []string{in.TargetBox} |
| 2083 | } |
| 2084 | c.Op = Op{Kind: KindRelocating, ID: in.OpID, Phase: PhaseStarting, By: in.By, StartedAt: in.Now.UTC(), |
| 2085 | Exec: exec, Participants: participants, |
| 2086 | Intent: map[string]string{ |
| 2087 | "region": in.Region, "reason": in.Reason, |
| 2088 | "source_box": in.SourceBox, "source_enrollment": in.SourceEnrollment, |
| 2089 | "target_box": in.TargetBox, "target_enrollment": in.TargetEnrollment, |
| 2090 | "stamp_from": strconv.FormatUint(c.Holder.Stamp, 10), |
| 2091 | "generation_from": c.Holder.Generation, |
| 2092 | "directory_revision": strconv.FormatUint(in.DirectoryRevision, 10), |
| 2093 | "admission_id": in.AdmissionID, |
| 2094 | }, |
| 2095 | Journal: map[string]string{}} |
| 2096 | if in.Fence != nil { |
| 2097 | return fenceBox(c, in.SourceBox, *in.Fence, observedAt, in.Discharge) |
| 2098 | } |
| 2099 | return nil |
| 2100 | }) |
| 2101 | } |
| 2102 | |
| 2103 | // FenceSource is §5.8's source-loss rule (B1-6): a drain or rebalance |
| 2104 | // whose source is terminated becomes a failover in one CAS citing the |
| 2105 | // source's fence. In starting it stays in starting with the source off the |
| 2106 | // barrier; in releasing before a release is journaled it goes to pinning |
| 2107 | // with the fence as evidence; after a release (pinning on) the source only |
| 2108 | // goes on lost. Every case keeps the stamp and the intent. |
| 2109 | func FenceSource(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { |
| 2110 | if observed.Absent || observed.Claim == nil { |
| 2111 | return Observed{}, ErrWrongState |
| 2112 | } |
| 2113 | before := observed.Claim.Op |
| 2114 | source := before.Intent["source_box"] |
| 2115 | v, err := VerifyFence(ctx, s, authority, ref) |
| 2116 | if err != nil { |
| 2117 | return Observed{}, err |
| 2118 | } |
| 2119 | if v.record.Box != source || v.record.Enrollment != before.Intent["source_enrollment"] || v.binding.Region != before.Intent["region"] { |
| 2120 | return Observed{}, ErrFence |
| 2121 | } |
| 2122 | return Update(ctx, s, observed, func(c *Claim) error { |
| 2123 | if err := RequireOp(opID)(c); err != nil { |
| 2124 | return err |
| 2125 | } |
| 2126 | if c.Op.Kind != KindRelocating || c.Op.Intent["source_box"] != source { |
| 2127 | return ErrWrongState |
| 2128 | } |
| 2129 | ensureJournal(c) |
| 2130 | // Before its release is journaled the source's writes are not |
| 2131 | // kept by a release, so the op becomes a failover. After it, only |
| 2132 | // the source's liveness changes. |
| 2133 | if c.Op.Phase == PhaseStarting || (c.Op.Phase == PhaseReleasing && c.Op.Journal["release"] == "") { |
| 2134 | c.Op.Journal["reason"] = ReasonFailover |
| 2135 | removeParticipant(c, source) |
| 2136 | c.Op.Exec = "none" |
| 2137 | if c.Op.Phase == PhaseReleasing { |
| 2138 | c.Op.Phase = PhasePinning |
| 2139 | } |
| 2140 | } |
| 2141 | return fenceBox(c, source, ref, v.record.ObservedAt, d) |
| 2142 | }) |
| 2143 | } |
| 2144 | |
| 2145 | // ToReleasing is a drain's barrier CAS: from starting, once every |
| 2146 | // participant has acknowledged. A failover never enters releasing. now |
| 2147 | // starts the release deadline (hostrelocate.ReleaseDeadline): a source |
| 2148 | // that never journals its release goes to stuck. |
| 2149 | func ToReleasing(opID string, now time.Time) func(*Claim) error { |
| 2150 | return func(c *Claim) error { |
| 2151 | if err := requireRelocation(c, opID, PhaseStarting); err != nil { |
| 2152 | return err |
| 2153 | } |
| 2154 | if failover(c.Op) { |
| 2155 | return fmt.Errorf("%w: a failover skips releasing", ErrRelocation) |
| 2156 | } |
| 2157 | if err := requireAcks(c); err != nil { |
| 2158 | return err |
| 2159 | } |
| 2160 | if now.IsZero() { |
| 2161 | return fmt.Errorf("%w: releasing needs its start time", ErrRelocation) |
| 2162 | } |
| 2163 | c.Op.Phase = PhaseReleasing |
| 2164 | c.Op.Journal["releasing_since"] = now.UTC().Format(time.RFC3339Nano) |
| 2165 | return nil |
| 2166 | } |
| 2167 | } |
| 2168 | |
| 2169 | // JournalRelease records a live box's completed release sequence. The |
| 2170 | // source records it in releasing; a failed target in probe-failed. The |
| 2171 | // release must be of the box's own current writer session. |
| 2172 | func JournalRelease(opID string, r Release) func(*Claim) error { |
| 2173 | return func(c *Claim) error { |
| 2174 | if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil { |
| 2175 | return err |
| 2176 | } |
| 2177 | h := c.Holder |
| 2178 | writerMismatch := h.Session != 0 && h.WriterBox != r.ReleasedBy // a host never woken has no writer |
| 2179 | if r.ReleasedBy == "" || r.ReleasedBy != c.Op.Exec || r.Enrollment == "" || r.Receipt == "" || !isHex64(r.ReceiptSHA256) || |
| 2180 | r.Generation != h.Generation || r.Session != h.Session || writerMismatch { |
| 2181 | return fmt.Errorf("%w: release is not the executor's current session", ErrRelocation) |
| 2182 | } |
| 2183 | want := c.Op.Intent["source_enrollment"] |
| 2184 | if c.Op.Phase == PhaseProbeFailed { |
| 2185 | want = c.Op.Intent["target_enrollment"] |
| 2186 | } |
| 2187 | if r.Enrollment != want { |
| 2188 | return fmt.Errorf("%w: release enrollment", ErrRelocation) |
| 2189 | } |
| 2190 | var old Release |
| 2191 | if readJournal(c.Op, "release", &old) && c.Op.Journal["release.phase"] == c.Op.Phase { |
| 2192 | if old == r { |
| 2193 | return ErrAlreadyRecorded |
| 2194 | } |
| 2195 | return fmt.Errorf("%w: a different release is journaled", ErrRelocation) |
| 2196 | } |
| 2197 | c.Op.Journal["release.phase"] = c.Op.Phase |
| 2198 | return journalJSON(c, "release", r) |
| 2199 | } |
| 2200 | } |
| 2201 | |
| 2202 | // PinAfterRelease leaves releasing (a drain) or probe-failed (a failed |
| 2203 | // target, §5.8's nested relocation) for pinning, citing the journaled |
| 2204 | // release's verified receipt. op.exec becomes "none" and the stamp is kept |
| 2205 | // (claim R6-4, D5's non-installing form). From probe-failed, next names |
| 2206 | // the replacement target, which must not be the failed one. |
| 2207 | func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, opID string, authority ReleaseAuthority, next *Target) (Observed, error) { |
| 2208 | if observed.Absent || observed.Claim == nil || authority == nil { |
| 2209 | return Observed{}, ErrWrongState |
| 2210 | } |
| 2211 | var r Release |
| 2212 | if !readJournal(observed.Claim.Op, "release", &r) { |
| 2213 | return Observed{}, fmt.Errorf("%w: no journaled release", ErrRelocation) |
| 2214 | } |
| 2215 | if err := authority.VerifyRelease(ctx, *observed.Claim, r); err != nil { |
| 2216 | return Observed{}, err |
| 2217 | } |
| 2218 | return Update(ctx, s, observed, func(c *Claim) error { |
| 2219 | if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil { |
| 2220 | return err |
| 2221 | } |
| 2222 | var now Release |
| 2223 | if !readJournal(c.Op, "release", &now) || now != r || c.Op.Journal["release.phase"] != c.Op.Phase || r.ReleasedBy != c.Op.Exec { |
| 2224 | return fmt.Errorf("%w: release changed", ErrRelocation) |
| 2225 | } |
| 2226 | if c.Op.Phase == PhaseReleasing { |
| 2227 | if next != nil { |
| 2228 | return fmt.Errorf("%w: a drain keeps its target", ErrRelocation) |
| 2229 | } |
| 2230 | // §5.8: a target replaced in releasing acknowledges before the |
| 2231 | // operation leaves the phase. |
| 2232 | if err := requireAcks(c); err != nil { |
| 2233 | return err |
| 2234 | } |
| 2235 | } else { |
| 2236 | if next == nil || next.Box == r.ReleasedBy { |
| 2237 | return fmt.Errorf("%w: a nested relocation names another target", ErrRelocation) |
| 2238 | } |
| 2239 | if err := replaceTargetSlot(c, *next); err != nil { |
| 2240 | return err |
| 2241 | } |
| 2242 | c.Op.Journal["released_by_target"] = r.ReleasedBy |
| 2243 | } |
| 2244 | c.Op.Exec, c.Op.Phase = "none", PhasePinning |
| 2245 | delete(c.Op.Journal, "pin") |
| 2246 | return nil |
| 2247 | }) |
| 2248 | } |
| 2249 | |
| 2250 | // PinFailover leaves starting for pinning on a failover: the source's |
| 2251 | // fence is journaled and every remaining participant has acknowledged. |
| 2252 | func PinFailover(opID string) func(*Claim) error { |
| 2253 | return func(c *Claim) error { |
| 2254 | if err := requireRelocation(c, opID, PhaseStarting); err != nil { |
| 2255 | return err |
| 2256 | } |
| 2257 | if !failover(c.Op) || c.Op.Journal["fence."+c.Op.Intent["source_box"]] == "" || c.Op.Exec != "none" { |
| 2258 | return fmt.Errorf("%w: pinning without a release needs the source's fence", ErrRelocation) |
| 2259 | } |
| 2260 | if err := requireAcks(c); err != nil { |
| 2261 | return err |
| 2262 | } |
| 2263 | c.Op.Phase = PhasePinning |
| 2264 | return nil |
| 2265 | } |
| 2266 | } |
| 2267 | |
| 2268 | // JournalPin records the pinned recovery set in pinning. The pin's key |
| 2269 | // must be RecoveryPinKey for this claim generation: the object was selected |
| 2270 | // against exactly this observation (B2-3). After a release it must reach |
| 2271 | // the journaled final TXID and files sequence; on a failover it must be |
| 2272 | // taken no earlier than the fence's observed_at plus FailoverPinDelay. An |
| 2273 | // Empty pin needs its evidence: a release at final_txid 0, or a holder |
| 2274 | // that never admitted a writer. |
| 2275 | func JournalPin(opID string, p RecoveryPin) func(*Claim) error { |
| 2276 | return func(c *Claim) error { |
| 2277 | if err := requireRelocation(c, opID, PhasePinning); err != nil { |
| 2278 | return err |
| 2279 | } |
| 2280 | if p.Key != RecoveryPinKey(c.Host, opID, c.Gen) { |
| 2281 | return fmt.Errorf("%w: pin %s was not selected against this observation (gen %d)", ErrRelocation, p.Key, c.Gen) |
| 2282 | } |
| 2283 | if !isHex64(p.SHA256) || p.FilesManifest == "" || p.PinnedAt.IsZero() || (p.Empty != (p.MaxTXID == 0)) { |
| 2284 | return fmt.Errorf("%w: incomplete pin", ErrRelocation) |
| 2285 | } |
| 2286 | var r Release |
| 2287 | released := readJournal(c.Op, "release", &r) |
| 2288 | if released { |
| 2289 | if p.MaxTXID < r.FinalTXID || p.FilesSeq < r.ManifestSeq { |
| 2290 | return fmt.Errorf("%w: pin %d/%d is short of the release's %d/%d", ErrRelocation, p.MaxTXID, p.FilesSeq, r.FinalTXID, r.ManifestSeq) |
| 2291 | } |
| 2292 | } |
| 2293 | if p.Empty && !((released && r.FinalTXID == 0) || (!released && c.Holder.Session == 0)) { |
| 2294 | return fmt.Errorf("%w: an empty pin needs a release at txid 0 or a holder that never had a writer", ErrRelocation) |
| 2295 | } |
| 2296 | for key, at := range c.Op.Journal { |
| 2297 | if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" { |
| 2298 | t, err := time.Parse(time.RFC3339Nano, at) |
| 2299 | if err != nil || p.PinnedAt.Before(t.Add(FailoverPinDelay)) { |
| 2300 | return fmt.Errorf("%w: pinned before the fence's observation plus %s", ErrRelocation, FailoverPinDelay) |
| 2301 | } |
| 2302 | } |
| 2303 | } |
| 2304 | if readJournal(c.Op, "pin", new(RecoveryPin)) { |
| 2305 | return ErrAlreadyRecorded |
| 2306 | } |
| 2307 | return journalJSON(c, "pin", p) |
| 2308 | } |
| 2309 | } |
| 2310 | |
| 2311 | // NameTarget names the target of an operation that has none: a move |
| 2312 | // converted to relocating before its flip (claim §4, moves §7.10). The |
| 2313 | // named box acknowledges (D6) before ToInstalling hands it op.exec. |
| 2314 | func NameTarget(opID string, t Target) func(*Claim) error { |
| 2315 | return func(c *Claim) error { |
| 2316 | if err := requireRelocation(c, opID, PhasePinning); err != nil { |
| 2317 | return err |
| 2318 | } |
| 2319 | if c.Op.Intent["target_box"] != "" { |
| 2320 | return fmt.Errorf("%w: the operation already has a target", ErrRelocation) |
| 2321 | } |
| 2322 | return replaceTargetSlot(c, t) |
| 2323 | } |
| 2324 | } |
| 2325 | |
| 2326 | // ToInstalling hands op.exec to the target for a new installation: a new |
| 2327 | // stamp (this CAS's gen, R6-4), journaled as install_stamp, a fresh |
| 2328 | // prepared generation id, and the target added to touched.boxes before its |
| 2329 | // first effect. A converted operation with no target names it here. |
| 2330 | func ToInstalling(opID string, t Target, prepared string) func(*Claim) error { |
| 2331 | return func(c *Claim) error { |
| 2332 | if err := requireRelocation(c, opID, PhasePinning); err != nil { |
| 2333 | return err |
| 2334 | } |
| 2335 | if !readJournal(c.Op, "pin", new(RecoveryPin)) || prepared == "" || prepared == c.Holder.Generation { |
| 2336 | return fmt.Errorf("%w: installing needs a pin and a fresh prepared generation", ErrRelocation) |
| 2337 | } |
| 2338 | if t.Box != c.Op.Intent["target_box"] || t.Enrollment != c.Op.Intent["target_enrollment"] || slices.Contains(c.Lost, t.Box) { |
| 2339 | return fmt.Errorf("%w: not the operation's target", ErrRelocation) |
| 2340 | } |
| 2341 | if OwesLocal(c, t.Box) { |
| 2342 | return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) |
| 2343 | } |
| 2344 | if err := requireAcks(c); err != nil { |
| 2345 | return err |
| 2346 | } |
| 2347 | return install(c, t.Box, prepared, PhaseInstalling) |
| 2348 | } |
| 2349 | } |
| 2350 | |
| 2351 | // abandonPrepared journals the operation's current P as abandoned when |
| 2352 | // no writer opened it, before a transition replaces or drops it: nothing |
| 2353 | // will open it now, and the serving CAS parks it (rev 17, finding F3; |
| 2354 | // rev 18: every transition that discards an unopened P calls it). |
| 2355 | func abandonPrepared(c *Claim) error { |
| 2356 | old := c.Op.Journal["prepared"] |
| 2357 | if old == "" || preparedOpened(c) { |
| 2358 | return nil |
| 2359 | } |
| 2360 | var abandoned []string |
| 2361 | readJournal(c.Op, "abandoned_prepared", &abandoned) |
| 2362 | if slices.Contains(abandoned, old) { |
| 2363 | return nil |
| 2364 | } |
| 2365 | return journalJSON(c, "abandoned_prepared", append(abandoned, old)) |
| 2366 | } |
| 2367 | |
| 2368 | func install(c *Claim, box, prepared, phase string) error { |
| 2369 | if c.Op.Journal["prepared"] != prepared { |
| 2370 | if err := abandonPrepared(c); err != nil { |
| 2371 | return err |
| 2372 | } |
| 2373 | } |
| 2374 | stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1 |
| 2375 | c.Op.Exec, c.Op.Phase = box, phase |
| 2376 | c.Holder.Stamp = stamp |
| 2377 | c.Op.Journal["install_stamp"] = strconv.FormatUint(stamp, 10) |
| 2378 | c.Op.Journal["prepared"] = prepared |
| 2379 | delete(c.Op.Journal, "local_install") |
| 2380 | if !slices.Contains(c.Touched.Boxes, box) { |
| 2381 | c.Touched.Boxes = append(c.Touched.Boxes, box) |
| 2382 | } |
| 2383 | return noteInstalledBox(c, box) |
| 2384 | } |
| 2385 | |
| 2386 | // ToInstalled records the verified remote build of P (§5.8, the |
| 2387 | // no-reinstall boundary): the install manifest at the pin's N (none for an |
| 2388 | // empty pin) and, always, P's files manifest at the pinned sequence, each |
| 2389 | // verified by a fresh read. Nobody builds P again after this CAS. |
| 2390 | func ToInstalled(opID string, a PreparedAuthority) func(*Claim) error { |
| 2391 | return func(c *Claim) error { |
| 2392 | if err := requireRelocation(c, opID, PhaseInstalling); err != nil { |
| 2393 | return err |
| 2394 | } |
| 2395 | var pin RecoveryPin |
| 2396 | readJournal(c.Op, "pin", &pin) |
| 2397 | if a.Generation == "" || a.Generation != c.Op.Journal["prepared"] || a.SnapshotTXID != pin.MaxTXID || |
| 2398 | (pin.MaxTXID > 0 && (a.InstallManifest == "" || !isHex64(a.SHA256))) || |
| 2399 | (pin.MaxTXID == 0 && (a.InstallManifest != "" || a.SHA256 != "")) { |
| 2400 | return fmt.Errorf("%w: prepared authority is not this pin's build of P", ErrRelocation) |
| 2401 | } |
| 2402 | if a.FilesSeq != 0 || a.SourceFilesSeq != pin.FilesSeq || !isHex64(a.FilesManifestSHA256) { |
| 2403 | return fmt.Errorf("%w: prepared authority lacks P's prepared files manifest (sequence 0, built from the pinned sequence)", ErrRelocation) |
| 2404 | } |
| 2405 | if err := requireAcks(c); err != nil { |
| 2406 | return err |
| 2407 | } |
| 2408 | c.Op.Phase = PhaseInstalled |
| 2409 | a.InstallStamp = journalUint(c.Op, "install_stamp") |
| 2410 | return journalJSON(c, "prepared_authority", a) |
| 2411 | } |
| 2412 | } |
| 2413 | |
| 2414 | // PreparedOpened reports whether P has been opened: journaled as |
| 2415 | // prepared_opened by the admission that opened it (rev 12: AdmitWriter |
| 2416 | // trims holder.admissions to eight, so the admission list alone could |
| 2417 | // forget it), or named by a retained admission. |
| 2418 | func PreparedOpened(c *Claim) bool { return preparedOpened(c) } |
| 2419 | |
| 2420 | func preparedOpened(c *Claim) bool { |
| 2421 | p := c.Op.Journal["prepared"] |
| 2422 | if p == "" || c.Holder == nil { |
| 2423 | return false |
| 2424 | } |
| 2425 | if c.Op.Journal["prepared_opened"] == p { |
| 2426 | return true |
| 2427 | } |
| 2428 | for _, a := range c.Holder.Admissions { |
| 2429 | if a.Generation == p { |
| 2430 | return true |
| 2431 | } |
| 2432 | } |
| 2433 | return false |
| 2434 | } |
| 2435 | |
| 2436 | // RecordLocalInstall journals the executor's local install receipt in |
| 2437 | // installed (B1-2). It is bound to op.exec, the target's enrollment and the |
| 2438 | // journaled install_stamp, and refused once any admission has named P |
| 2439 | // (B2-5): after that the local install is never remade. |
| 2440 | func RecordLocalInstall(opID string, l LocalInstall) func(*Claim) error { |
| 2441 | return func(c *Claim) error { |
| 2442 | if err := requireRelocation(c, opID, PhaseInstalled); err != nil { |
| 2443 | return err |
| 2444 | } |
| 2445 | if l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] || |
| 2446 | l.InstallStamp == 0 || l.InstallStamp != journalUint(c.Op, "install_stamp") || !isHex64(l.JournalSHA256) { |
| 2447 | return fmt.Errorf("%w: receipt is not this installation's", ErrRelocation) |
| 2448 | } |
| 2449 | if preparedOpened(c) { |
| 2450 | return fmt.Errorf("%w: P was admitted; the local install is never remade", ErrRelocation) |
| 2451 | } |
| 2452 | var old LocalInstall |
| 2453 | if readJournal(c.Op, "local_install", &old) { |
| 2454 | if old == l { |
| 2455 | return ErrAlreadyRecorded |
| 2456 | } |
| 2457 | return fmt.Errorf("%w: a different receipt is journaled", ErrRelocation) |
| 2458 | } |
| 2459 | return journalJSON(c, "local_install", l) |
| 2460 | } |
| 2461 | } |
| 2462 | |
| 2463 | // ToProbing enters probing, which requires a local install receipt for |
| 2464 | // exactly op.exec, its enrollment and the journaled install_stamp. now |
| 2465 | // starts the probe deadline (§5.8: 10 minutes). |
| 2466 | func ToProbing(opID string, now time.Time) func(*Claim) error { |
| 2467 | return func(c *Claim) error { |
| 2468 | if err := requireRelocation(c, opID, PhaseInstalled); err != nil { |
| 2469 | return err |
| 2470 | } |
| 2471 | var l LocalInstall |
| 2472 | if !readJournal(c.Op, "local_install", &l) || l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] || |
| 2473 | l.InstallStamp != journalUint(c.Op, "install_stamp") { |
| 2474 | return fmt.Errorf("%w: no local install receipt for the executor", ErrRelocation) |
| 2475 | } |
| 2476 | if err := requireAcks(c); err != nil { |
| 2477 | return err |
| 2478 | } |
| 2479 | if now.IsZero() { |
| 2480 | return fmt.Errorf("%w: probing needs its start time", ErrRelocation) |
| 2481 | } |
| 2482 | c.Op.Phase = PhaseProbing |
| 2483 | c.Op.Journal["probing_since"] = now.UTC().Format(time.RFC3339Nano) |
| 2484 | return nil |
| 2485 | } |
| 2486 | } |
| 2487 | |
| 2488 | // BeginProbeAttempt counts one probe attempt for the executor before it |
| 2489 | // is made (§5.8 "limits"): the fourth is refused with |
| 2490 | // ErrProbeAttemptsExhausted, and the participant then enters |
| 2491 | // probe-failed. The count is per box and per operation, so a replacement |
| 2492 | // target starts at zero and a fenced one is never counted again. |
| 2493 | func BeginProbeAttempt(opID, box string) func(*Claim) error { |
| 2494 | return func(c *Claim) error { |
| 2495 | if err := requireRelocation(c, opID, PhaseProbing); err != nil { |
| 2496 | return err |
| 2497 | } |
| 2498 | if box == "" || box != c.Op.Exec { |
| 2499 | return fmt.Errorf("%w: only the executor probes", ErrRelocation) |
| 2500 | } |
| 2501 | key := "probe_attempts." + box |
| 2502 | n := journalUint(c.Op, key) |
| 2503 | if n >= MaxProbeAttempts { |
| 2504 | return ErrProbeAttemptsExhausted |
| 2505 | } |
| 2506 | c.Op.Journal[key] = strconv.FormatUint(n+1, 10) |
| 2507 | return nil |
| 2508 | } |
| 2509 | } |
| 2510 | |
| 2511 | // RecordProbe journals the executor's passing probe of its admitted session. |
| 2512 | func RecordProbe(opID string, p Probe) func(*Claim) error { |
| 2513 | return func(c *Claim) error { |
| 2514 | if err := requireRelocation(c, opID, PhaseProbing); err != nil { |
| 2515 | return err |
| 2516 | } |
| 2517 | if p.Box != c.Op.Exec || p.Box != c.Holder.WriterBox || p.Session == 0 || p.Session != c.Holder.Session || |
| 2518 | p.Session < journalUint(c.Op, "install_stamp") { |
| 2519 | return fmt.Errorf("%w: probe is not of the executor's admitted session", ErrRelocation) |
| 2520 | } |
| 2521 | return journalJSON(c, "probe", p) |
| 2522 | } |
| 2523 | } |
| 2524 | |
| 2525 | // ToProbeFailed records that the live target's probe failed or missed its |
| 2526 | // deadline. The target then runs its release sequence (JournalRelease); |
| 2527 | // now starts its release deadline. |
| 2528 | func ToProbeFailed(opID string, now time.Time) func(*Claim) error { |
| 2529 | return func(c *Claim) error { |
| 2530 | if err := requireRelocation(c, opID, PhaseProbing); err != nil { |
| 2531 | return err |
| 2532 | } |
| 2533 | if now.IsZero() { |
| 2534 | return fmt.Errorf("%w: probe-failed needs its start time", ErrRelocation) |
| 2535 | } |
| 2536 | c.Op.Phase = PhaseProbeFailed |
| 2537 | c.Op.Journal["probe_failed_since"] = now.UTC().Format(time.RFC3339Nano) |
| 2538 | delete(c.Op.Journal, "release") |
| 2539 | delete(c.Op.Journal, "release.phase") |
| 2540 | return nil |
| 2541 | } |
| 2542 | } |
| 2543 | |
| 2544 | // CompleteRelocation is the serving CAS: the target's probe passed for its |
| 2545 | // admitted session, so the host serves on it alone. Every other box this |
| 2546 | // operation installed on, and a live source, owes a park-local unless a |
| 2547 | // fence discharged it. |
| 2548 | func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| 2549 | return func(c *Claim) error { |
| 2550 | if err := requireRelocation(c, opID, PhaseProbing); err != nil { |
| 2551 | return err |
| 2552 | } |
| 2553 | var p Probe |
| 2554 | if !readJournal(c.Op, "probe", &p) || p.Box != c.Op.Exec || p.Session != c.Holder.Session || c.Holder.WriterBox != p.Box { |
| 2555 | return fmt.Errorf("%w: no passing probe of the current session", ErrRelocation) |
| 2556 | } |
| 2557 | target := c.Op.Exec |
| 2558 | var owed []string |
| 2559 | readJournal(c.Op, "installed_boxes", &owed) |
| 2560 | owed = append([]string{c.Op.Intent["source_box"]}, owed...) |
| 2561 | for _, box := range owed { |
| 2562 | if box == target || box == "" { |
| 2563 | continue |
| 2564 | } |
| 2565 | var d LocalDischarge |
| 2566 | if readJournal(c.Op, "discharge."+box, &d) && d.Qualified { |
| 2567 | continue |
| 2568 | } |
| 2569 | exists := false |
| 2570 | for _, o := range c.Obligations { |
| 2571 | if o.Kind == "park-local" && o.Box == box { |
| 2572 | exists = true |
| 2573 | } |
| 2574 | } |
| 2575 | if !exists { |
| 2576 | c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: box, Retirement: newID(), Owner: "box:" + box}) |
| 2577 | } |
| 2578 | } |
| 2579 | // Every prepared generation this operation built and no writer |
| 2580 | // opened is parked here, in the region, by the claim's own |
| 2581 | // park-generation chore (rev 17, finding F3); the at-rest sweep |
| 2582 | // stays the backstop the spec names. |
| 2583 | var abandoned []string |
| 2584 | readJournal(c.Op, "abandoned_prepared", &abandoned) |
| 2585 | region := c.Op.Intent["region"] |
| 2586 | for _, g := range abandoned { |
| 2587 | if g == "" || g == c.Holder.Generation || region == "" { |
| 2588 | continue |
| 2589 | } |
| 2590 | queued := false |
| 2591 | for _, o := range c.Obligations { |
| 2592 | if o.Kind == "park-generation" && o.Region == region && o.Generation == g { |
| 2593 | queued = true |
| 2594 | } |
| 2595 | } |
| 2596 | if !queued { |
| 2597 | c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-generation", Region: region, Generation: g, Retirement: newID(), Owner: "region:" + region}) |
| 2598 | } |
| 2599 | } |
| 2600 | c.Op = Op{Kind: KindServing, Exec: target, Participants: []string{target}} |
| 2601 | return nil |
| 2602 | } |
| 2603 | } |
| 2604 | |
| 2605 | // replaceTargetSlot names t as the target, counting a replacement when it |
| 2606 | // replaces another box. A replaced box is never named again for this host. |
| 2607 | func replaceTargetSlot(c *Claim, t Target) error { |
| 2608 | if !validBoxReference(t.Box) || t.Box == "none" || t.Enrollment == "" || t.Box == c.Op.Intent["source_box"] || slices.Contains(c.Lost, t.Box) { |
| 2609 | return fmt.Errorf("%w: invalid target", ErrRelocation) |
| 2610 | } |
| 2611 | var excluded []string |
| 2612 | readJournal(c.Op, "excluded_targets", &excluded) |
| 2613 | if slices.Contains(excluded, t.Box) { |
| 2614 | return fmt.Errorf("%w: %s was replaced in this operation", ErrRelocation, t.Box) |
| 2615 | } |
| 2616 | if OwesLocal(c, t.Box) { |
| 2617 | // Spec 1 §2: a box owing a local retirement for this host is never |
| 2618 | // handed an installation, in any phase a replacement resumes at |
| 2619 | // (install is called below without ToInstalling's check). |
| 2620 | return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) |
| 2621 | } |
| 2622 | old := c.Op.Intent["target_box"] |
| 2623 | if old != "" && old != t.Box { |
| 2624 | n := journalUint(c.Op, "replacements") |
| 2625 | if n >= MaxTargetReplacements { |
| 2626 | return ErrReplacementsExceeded |
| 2627 | } |
| 2628 | c.Op.Journal["replacements"] = strconv.FormatUint(n+1, 10) |
| 2629 | excluded = append(excluded, old) |
| 2630 | if err := journalJSON(c, "excluded_targets", excluded); err != nil { |
| 2631 | return err |
| 2632 | } |
| 2633 | removeParticipant(c, old) |
| 2634 | } |
| 2635 | c.Op.Intent["target_box"], c.Op.Intent["target_enrollment"] = t.Box, t.Enrollment |
| 2636 | addParticipant(c, t.Box) |
| 2637 | return nil |
| 2638 | } |
| 2639 | |
| 2640 | // ReplaceTarget is §5.8's replacement table for a target terminated by its |
| 2641 | // fence. It records the fence (the old target goes on lost), names next, |
| 2642 | // and resumes at the phase the table gives: |
| 2643 | // |
| 2644 | // starting, releasing, pinning -> the same phase |
| 2645 | // installing -> installing: next is op.exec, new stamp, fresh P' |
| 2646 | // installed, or probing before P -> installed: next is op.exec, new stamp, same P, no receipt |
| 2647 | // probing after P, or probe-failed -> pinning: a nested failover from holder.generation |
| 2648 | // |
| 2649 | // prepared is P' for installing and ignored otherwise. A third |
| 2650 | // replacement is refused with ErrReplacementsExceeded; the worker then |
| 2651 | // marks the operation stuck. |
| 2652 | func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, next Target, prepared string, authority FenceAuthority) (Observed, error) { |
| 2653 | if observed.Absent || observed.Claim == nil { |
| 2654 | return Observed{}, ErrWrongState |
| 2655 | } |
| 2656 | old := observed.Claim.Op.Intent["target_box"] |
| 2657 | v, err := VerifyFence(ctx, s, authority, ref) |
| 2658 | if err != nil { |
| 2659 | return Observed{}, err |
| 2660 | } |
| 2661 | if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] { |
| 2662 | return Observed{}, ErrFence |
| 2663 | } |
| 2664 | return Update(ctx, s, observed, func(c *Claim) error { |
| 2665 | if err := requireRelocation(c, opID, PhaseStarting, PhaseReleasing, PhasePinning, PhaseInstalling, PhaseInstalled, PhaseProbing, PhaseProbeFailed); err != nil { |
| 2666 | return err |
| 2667 | } |
| 2668 | if c.Op.Intent["target_box"] != old { |
| 2669 | return ErrStaleOp |
| 2670 | } |
| 2671 | phase := c.Op.Phase |
| 2672 | if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { |
| 2673 | return err |
| 2674 | } |
| 2675 | if err := replaceTargetSlot(c, next); err != nil { |
| 2676 | return err |
| 2677 | } |
| 2678 | switch { |
| 2679 | case phase == PhaseStarting || phase == PhaseReleasing || phase == PhasePinning: |
| 2680 | return nil |
| 2681 | case phase == PhaseInstalling: |
| 2682 | if prepared == "" || prepared == c.Op.Journal["prepared"] || prepared == c.Holder.Generation { |
| 2683 | return fmt.Errorf("%w: a replacement in installing builds a fresh P'", ErrRelocation) |
| 2684 | } |
| 2685 | return install(c, next.Box, prepared, PhaseInstalling) |
| 2686 | case (phase == PhaseInstalled || phase == PhaseProbing) && !preparedOpened(c): |
| 2687 | return install(c, next.Box, c.Op.Journal["prepared"], PhaseInstalled) |
| 2688 | default: |
| 2689 | // After P's first admission, or from probe-failed: nested |
| 2690 | // failover from pinning, pinning from holder.generation as |
| 2691 | // observed. A P no writer opened (a probe deadline before any |
| 2692 | // admission) is recorded before it is dropped. |
| 2693 | if err := abandonPrepared(c); err != nil { |
| 2694 | return err |
| 2695 | } |
| 2696 | c.Op.Exec, c.Op.Phase = "none", PhasePinning |
| 2697 | for _, k := range []string{"pin", "prepared", "prepared_authority", "local_install", "probe", "release", "release.phase"} { |
| 2698 | delete(c.Op.Journal, k) |
| 2699 | } |
| 2700 | return nil |
| 2701 | } |
| 2702 | }) |
| 2703 | } |
| 2704 | |
| 2705 | // FenceExhaustedTarget is the terminal transition for a target terminated |
| 2706 | // after the operation has used its replacements (§5.8 "The bound"; round 3 |
| 2707 | // finding 5): it consumes the fence (the loss is retained, the box goes on |
| 2708 | // lost) and records stuck. While another box is op.exec (the live source |
| 2709 | // in starting or releasing; round 8 finding 1) nothing else changes: the |
| 2710 | // target's fence revokes nothing of the source's. Otherwise (op.exec is |
| 2711 | // the dead target, or none) it revokes the dead installation's authority (op.exec "none", the |
| 2712 | // target off the barrier, its receipt dropped), leaves the operation at |
| 2713 | // pinning with its pin and, when P was never opened, its remote build kept |
| 2714 | // for the operator's replacement, or a nested pinning when P was opened, |
| 2715 | // and records stuck in the same CAS. No third target is chosen. MarkStuck's |
| 2716 | // "never between installed and a probe" is for a live target; a dead one |
| 2717 | // has no probe to make. |
| 2718 | func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { |
| 2719 | return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, "") |
| 2720 | } |
| 2721 | |
| 2722 | // FenceTargetWithoutSpare is FenceExhaustedTarget's transition for a |
| 2723 | // terminated target with replacements left but no eligible box to name |
| 2724 | // (rev 18, round 17 finding 2): the worker calls it once its bound has |
| 2725 | // passed. The fence is consumed in the same CAS (loss retained, the box |
| 2726 | // on lost, a dead executor's authority revoked), and the operation is |
| 2727 | // stuck for an operator with reason, never left waiting unbounded. |
| 2728 | func FenceTargetWithoutSpare(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, reason string) (Observed, error) { |
| 2729 | if reason == "" { |
| 2730 | return Observed{}, fmt.Errorf("%w: a stuck operation names its reason", ErrRelocation) |
| 2731 | } |
| 2732 | return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, reason) |
| 2733 | } |
| 2734 | |
| 2735 | // fenceTargetStuck is both: noSpare "" is the exhausted bound. |
| 2736 | func fenceTargetStuck(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, noSpare string) (Observed, error) { |
| 2737 | if observed.Absent || observed.Claim == nil { |
| 2738 | return Observed{}, ErrWrongState |
| 2739 | } |
| 2740 | old := observed.Claim.Op.Intent["target_box"] |
| 2741 | v, err := VerifyFence(ctx, s, authority, ref) |
| 2742 | if err != nil { |
| 2743 | return Observed{}, err |
| 2744 | } |
| 2745 | if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] { |
| 2746 | return Observed{}, ErrFence |
| 2747 | } |
| 2748 | return Update(ctx, s, observed, func(c *Claim) error { |
| 2749 | if err := RequireOp(opID)(c); err != nil { |
| 2750 | return err |
| 2751 | } |
| 2752 | if c.Op.Kind != KindRelocating || c.Op.Intent["target_box"] != old { |
| 2753 | return ErrStaleOp |
| 2754 | } |
| 2755 | ensureJournal(c) |
| 2756 | if c.Op.Journal["stuck"] != "" { |
| 2757 | return ErrAlreadyRecorded |
| 2758 | } |
| 2759 | reason := noSpare |
| 2760 | if reason == "" { |
| 2761 | if journalUint(c.Op, "replacements") < MaxTargetReplacements { |
| 2762 | return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation) |
| 2763 | } |
| 2764 | reason = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) |
| 2765 | } |
| 2766 | if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { |
| 2767 | return err |
| 2768 | } |
| 2769 | removeParticipant(c, old) |
| 2770 | if c.Op.Exec != old && c.Op.Exec != "none" { |
| 2771 | // The executor is another box (the live source in starting or |
| 2772 | // releasing): the target's fence is no evidence against it. |
| 2773 | // Its execution identity, phase and release records stay, so |
| 2774 | // root still accepts its own stop (claimstop.go requires the |
| 2775 | // requester to be op.exec); only stuck is new. |
| 2776 | c.Op.Journal["stuck"] = reason |
| 2777 | return nil |
| 2778 | } |
| 2779 | opened := preparedOpened(c) |
| 2780 | c.Op.Exec, c.Op.Phase = "none", PhasePinning |
| 2781 | keys := []string{"local_install", "probe", "release", "release.phase", "probing_since", "probe_failed_since"} |
| 2782 | if opened { |
| 2783 | // Nested: pinning again from holder.generation, as a replacement |
| 2784 | // after P's first admission would. |
| 2785 | keys = append(keys, "pin", "prepared", "prepared_authority") |
| 2786 | } |
| 2787 | // An unopened P is kept for the operator's replacement; it is on |
| 2788 | // the abandoned list only if a later transition drops it. |
| 2789 | for _, k := range keys { |
| 2790 | delete(c.Op.Journal, k) |
| 2791 | } |
| 2792 | if c.Op.Journal["stuck"] == "" { |
| 2793 | c.Op.Journal["stuck"] = reason |
| 2794 | } |
| 2795 | return nil |
| 2796 | }) |
| 2797 | } |
| 2798 | |
| 2799 | // MarkStuck stops the operation at its current phase for an operator |
| 2800 | // (`carlos ops host resolve`). It never resets the replacement count and |
| 2801 | // never invents evidence; every later relocation transition refuses. |
| 2802 | func MarkStuck(opID, reason string) func(*Claim) error { |
| 2803 | return func(c *Claim) error { |
| 2804 | if err := RequireOp(opID)(c); err != nil { |
| 2805 | return err |
| 2806 | } |
| 2807 | if c.Op.Kind != KindRelocating || reason == "" { |
| 2808 | return ErrWrongState |
| 2809 | } |
| 2810 | ensureJournal(c) |
| 2811 | if c.Op.Phase == PhaseInstalled && c.Op.Journal["local_install"] != "" && !preparedOpened(c) { |
| 2812 | return fmt.Errorf("%w: never stop between installed and a probe", ErrRelocation) |
| 2813 | } |
| 2814 | if c.Op.Journal["stuck"] != "" { |
| 2815 | return ErrAlreadyRecorded |
| 2816 | } |
| 2817 | c.Op.Journal["stuck"] = reason |
| 2818 | return nil |
| 2819 | } |
| 2820 | } |
| 2821 | ``` |
| 2822 | |
| 2823 | - [ ] **Step 4: Teach `AdmitWriter` the phase table and `released`, and the claim its retained losses** |
| 2824 | |
| 2825 | Apply to `internal/hostclaim/claim.go` (`git apply`): `EvidenceRefs`, D4's retained automatic losses, validated beside `loss_refs`: |
| 2826 | |
| 2827 | ```diff |
| 2828 | diff --git a/internal/hostclaim/claim.go b/internal/hostclaim/claim.go |
| 2829 | index 23d17858..50e55006 100644 |
| 2830 | --- a/internal/hostclaim/claim.go |
| 2831 | +++ b/internal/hostclaim/claim.go |
| 2832 | @@ -162,6 +162,7 @@ type Claim struct { |
| 2833 | Lost []string `json:"lost,omitempty"` |
| 2834 | LostRevisions map[string]uint64 `json:"lost_revisions,omitempty"` |
| 2835 | LossRefs []string `json:"loss_refs,omitempty"` |
| 2836 | + EvidenceRefs []AutomaticLoss `json:"evidence_refs,omitempty"` |
| 2837 | RetiredIncarnations []string `json:"retired_incarnations,omitempty"` |
| 2838 | QuietAfter map[string]time.Time `json:"quiet_after,omitempty"` |
| 2839 | CancelledCreates []CancelledCreate `json:"cancelled_creates,omitempty"` |
| 2840 | @@ -289,6 +290,16 @@ func (c *Claim) Validate() error { |
| 2841 | } |
| 2842 | seenLossRefs[id] = true |
| 2843 | } |
| 2844 | + seenEvidence := map[string]bool{} |
| 2845 | + for _, l := range c.EvidenceRefs { |
| 2846 | + // D4: a retained automatic loss names a loss identity the claim |
| 2847 | + // also lists, a box, and a well-formed fence reference. |
| 2848 | + if !seenLossRefs[l.ID] || seenEvidence[l.ID] || l.Box == "" || !validFenceRef(l.Fence) || l.Enrollment != l.Fence.Enrollment || l.Op == "" || |
| 2849 | + (l.Role != "source" && l.Role != "target") || (l.Timers != TimersComplete && l.Timers != TimersPartial && l.Timers != TimersUnknown) { |
| 2850 | + return fmt.Errorf("%w: invalid automatic loss evidence", ErrMalformed) |
| 2851 | + } |
| 2852 | + seenEvidence[l.ID] = true |
| 2853 | + } |
| 2854 | if len(c.RetiredIncarnations) > retiredCap { |
| 2855 | return fmt.Errorf("%w: retired_incarnations over cap", ErrMalformed) |
| 2856 | } |
| 2857 | ``` |
| 2858 | |
| 2859 | Apply to `internal/hostclaim/generation.go` (`git apply`): |
| 2860 | |
| 2861 | ```diff |
| 2862 | diff --git a/internal/hostclaim/generation.go b/internal/hostclaim/generation.go |
| 2863 | index ffd7e058..74947b56 100644 |
| 2864 | --- a/internal/hostclaim/generation.go |
| 2865 | +++ b/internal/hostclaim/generation.go |
| 2866 | @@ -43,6 +43,10 @@ func AdmitWriter(in Admission) func(*Claim) error { |
| 2867 | switch c.Op.Kind { |
| 2868 | case KindCreating, KindDeleting, KindParking, KindPurging, KindPlacing: |
| 2869 | return fmt.Errorf("%w: op kind %q cannot admit a writer", ErrWrongState, c.Op.Kind) |
| 2870 | + case KindRelocating: |
| 2871 | + if err := relocationAdmission(c, in); err != nil { |
| 2872 | + return err |
| 2873 | + } |
| 2874 | } |
| 2875 | h := c.Holder |
| 2876 | if in.LossID != "" && !in.lossVerified { |
| 2877 | @@ -87,6 +91,10 @@ func AdmitWriter(in Admission) func(*Claim) error { |
| 2878 | |
| 2879 | switch in.Coverage { |
| 2880 | case "", "sealed", "clean-return": |
| 2881 | + case "released": // checked by relocationAdmission (L9) |
| 2882 | + if c.Op.Kind != KindRelocating { |
| 2883 | + return ErrWrongState |
| 2884 | + } |
| 2885 | default: |
| 2886 | return ErrWrongState |
| 2887 | } |
| 2888 | @@ -261,3 +269,60 @@ func CoverParkedPredecessors(observedOpID, observedPhase string, region string, |
| 2889 | return nil |
| 2890 | } |
| 2891 | } |
| 2892 | + |
| 2893 | +// relocationAdmission is §4's phase table for relocating (L1) and §6.4a's |
| 2894 | +// `released` coverage (L9). The source may admit in starting and releasing |
| 2895 | +// (its local admission fence closes releasing); only op.exec admits in |
| 2896 | +// probing, cold; nobody admits in any other phase. The first probe |
| 2897 | +// admission opens the prepared generation P, which is opened at most once. |
| 2898 | +// `released` is legal only on that opening, when the predecessor is exactly |
| 2899 | +// the session whose release this operation journaled and its pin reaches |
| 2900 | +// that release's final position. |
| 2901 | +func relocationAdmission(c *Claim, in Admission) error { |
| 2902 | + if c.Op.Journal["stuck"] != "" { |
| 2903 | + // Ruling 9: a stuck operation runs nothing, the source's wake |
| 2904 | + // included; the operator resolves it (carlos ops host resolve). |
| 2905 | + return fmt.Errorf("%w: relocation is stuck", ErrWrongState) |
| 2906 | + } |
| 2907 | + switch c.Op.Phase { |
| 2908 | + case PhaseStarting, PhaseReleasing: |
| 2909 | + if in.Box != c.Op.Intent["source_box"] || in.Coverage == "released" { |
| 2910 | + return fmt.Errorf("%w: only the source admits before pinning", ErrWrongState) |
| 2911 | + } |
| 2912 | + return nil |
| 2913 | + case PhaseProbing: |
| 2914 | + default: |
| 2915 | + return fmt.Errorf("%w: relocating/%s admits no writer", ErrWrongState, c.Op.Phase) |
| 2916 | + } |
| 2917 | + if in.Mode != "cold" || in.Box != c.Op.Exec { |
| 2918 | + return fmt.Errorf("%w: a probe admission is the executor's, cold", ErrWrongState) |
| 2919 | + } |
| 2920 | + p, opened := c.Op.Journal["prepared"], preparedOpened(c) |
| 2921 | + if p == "" || (!opened && in.NewGeneration != p) || (opened && in.NewGeneration == p) { |
| 2922 | + return fmt.Errorf("%w: the first probe opens P, and only it", ErrWrongState) |
| 2923 | + } |
| 2924 | + if !opened { |
| 2925 | + // The opening is journaled with the admission, so it outlives the |
| 2926 | + // admission list's trim (rev 12): P is never opened twice. |
| 2927 | + ensureJournal(c) |
| 2928 | + c.Op.Journal["prepared_opened"] = p |
| 2929 | + } |
| 2930 | + if in.Coverage != "released" { |
| 2931 | + return nil |
| 2932 | + } |
| 2933 | + var r Release |
| 2934 | + var pin RecoveryPin |
| 2935 | + if opened || in.PredecessorCertified || !readJournal(c.Op, "release", &r) || !readJournal(c.Op, "pin", &pin) || |
| 2936 | + pin.MaxTXID < r.FinalTXID || pin.FilesSeq < r.ManifestSeq { |
| 2937 | + return fmt.Errorf("%w: released coverage needs this operation's release and a pin reaching it", ErrWrongState) |
| 2938 | + } |
| 2939 | + h := c.Holder |
| 2940 | + predecessor := AdmissionRecord{Session: h.Session, Box: h.WriterBox, Generation: h.Generation} |
| 2941 | + if len(h.Admissions) > 0 { |
| 2942 | + predecessor = h.Admissions[len(h.Admissions)-1] |
| 2943 | + } |
| 2944 | + if predecessor.Box != r.ReleasedBy || predecessor.Session != r.Session || predecessor.Generation != r.Generation { |
| 2945 | + return fmt.Errorf("%w: released coverage names another predecessor", ErrWrongState) |
| 2946 | + } |
| 2947 | + return nil |
| 2948 | +} |
| 2949 | ``` |
| 2950 | |
| 2951 | - [ ] **Step 5: Run the tests** |
| 2952 | |
| 2953 | Run: `nice -n 19 go test ./internal/hostclaim/ -count=1` |
| 2954 | Expected: PASS, every existing test included (`TestC7…` covers the conversion from `moving`, unchanged). |
| 2955 | |
| 2956 | - [ ] **Step 6: Commit** |
| 2957 | |
| 2958 | ```bash |
| 2959 | gofmt -l internal/hostclaim |
| 2960 | git add internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostclaim/generation.go internal/hostclaim/claim.go |
| 2961 | git commit -m "hostclaim: relocating within one region, released by stop evidence or a termination fence, with released coverage on P's one opening and the fenced box's loss retained |
| 2962 | |
| 2963 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 2964 | ``` |
| 2965 | |
| 2966 | --- |
| 2967 | |
| 2968 | ### Task 2: Create names its executor once (H12, L8, P5-D2, P5-D6) |
| 2969 | |
| 2970 | Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the roster's active, unfenced boxes in the record's placement, chosen before the create CAS and recorded in its intent as `selected_executor` and `executor_enrollment`. `CompleteCreate` refuses any other box, so recovery never substitutes. Identity, placement and AWS region come from the roster (`Roster.Regions[placement].AWSRegion`); the edges snapshot gives liveness and configuration only; claim loads only balance. A fence that cannot be read refuses the choice. `Permits` is the same read without the choice: whether one named box is in the eligible set (Task 5's start check). **Dormant:** `SelectExecutor` reports whether it selected at all; with no chooser (`ProductionAuthority.Executors` nil, which the console leaves nil unless `Config.RegionHA` is on) the create is byte-for-byte today's, `BeginCreate` and `Executor` at completion, claim-enabled one-box regions included. |
| 2971 | |
| 2972 | **Files:** |
| 2973 | - Modify: `internal/hostclaim/create.go`, `internal/hostclaim/create_test.go` |
| 2974 | - Create: `internal/hostops/executor.go`, `internal/hostops/loads.go`, `internal/hostops/executor_test.go` |
| 2975 | - Modify: `internal/hostops/operations.go` (`Authority.SelectExecutor`, `Attempt`), `internal/hostops/backfill_controller.go`, `internal/hostops/backfill.go` (`ProductionAuthority.Executors`, `SelectExecutor`), `internal/hostops/operations_test.go` (the fake authority) |
| 2976 | - Create: `internal/console/hostclaimexecutor.go`, `internal/console/hostclaimexecutor_test.go` (the console's chooser; the create door uses it once claim Plan 5's door lands, Task 10) |
| 2977 | - Modify: `internal/console/server.go` (`Config.RegionHA`), `cmd/console/main.go` (sets it from `CARLOS_REGION_HA` with `strconv.ParseBool`) |
| 2978 | - Test: `cmd/console/regionha_test.go` (rev 11: the switch itself) |
| 2979 | |
| 2980 | **Interfaces:** |
| 2981 | - Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`; Task 1's `hostclaim.LocalChoreBoxes` (the eligible set excludes a box owing a `park-local`, rev 6). |
| 2982 | - Produces: `hostclaim.ExecutorSelection{Box, Enrollment}`, `hostclaim.BeginCreateOn(nh, body, opID, by, now, sel)`; `hostops.ErrNoEligibleBox`; `hostops.ExecutorChooser{Directory, Edges func(ctx) (boxchoice.Snapshot, error), Loads func(ctx) (map[string]boxchoice.Load, error), SecretsKeyFP, HomeName, Now}` with `Choose(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)` and `Permits(ctx, hostclaim.Observed, Record, box string) error`; `hostops.ClaimLoads(ctx, release.Store) (map[string]boxchoice.Load, error)`; `Authority.SelectExecutor(ctx, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error)`; `ProductionAuthority.Executors *ExecutorChooser`; `console.Config.RegionHA`; `(*console.Server).executorChooser(hostactors.Directory) *hostops.ExecutorChooser` (nil with HA off). |
| 2983 | |
| 2984 | - [ ] **Step 1: Write the failing tests** |
| 2985 | |
| 2986 | Append to `internal/hostclaim/create_test.go`: |
| 2987 | |
| 2988 | ```diff |
| 2989 | diff --git a/internal/hostclaim/create_test.go b/internal/hostclaim/create_test.go |
| 2990 | index 53968923..c9a100e6 100644 |
| 2991 | --- a/internal/hostclaim/create_test.go |
| 2992 | +++ b/internal/hostclaim/create_test.go |
| 2993 | @@ -276,3 +276,27 @@ func TestCompleteCreateRefusesExecutionAuthority(t *testing.T) { |
| 2994 | }) |
| 2995 | } |
| 2996 | } |
| 2997 | + |
| 2998 | +func TestCreateCompletesOnlyOnTheSelectedExecutor(t *testing.T) { |
| 2999 | + ctx := context.Background() |
| 3000 | + s := release.NewDirStore(t.TempDir()) |
| 3001 | + now := time.Date(2026, 10, 4, 1, 0, 1, 0, time.UTC) |
| 3002 | + reg, _ := Read(ctx, s, testHost) |
| 3003 | + sel := ExecutorSelection{Box: "bri-knitting", Enrollment: "enrollment-b"} |
| 3004 | + if _, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, ExecutorSelection{})); !errors.Is(err, ErrMalformed) { |
| 3005 | + t.Fatalf("create with no selection: %v", err) |
| 3006 | + } |
| 3007 | + o, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, sel)) |
| 3008 | + if err != nil { |
| 3009 | + t.Fatal(err) |
| 3010 | + } |
| 3011 | + if o.Claim.Op.Intent["selected_executor"] != sel.Box || o.Claim.Op.Intent["executor_enrollment"] != sel.Enrollment || o.Claim.Op.Exec != "" { |
| 3012 | + t.Fatalf("intent: %+v", o.Claim.Op) |
| 3013 | + } |
| 3014 | + if _, err := Update(ctx, s, o, CompleteCreate("op_c1", "adi-knitting")); !errors.Is(err, ErrWrongState) { |
| 3015 | + t.Fatalf("completion substituted a box: %v", err) |
| 3016 | + } |
| 3017 | + if o, err = Update(ctx, s, o, CompleteCreate("op_c1", sel.Box)); err != nil || o.Claim.Op.Exec != sel.Box { |
| 3018 | + t.Fatalf("complete: %v %+v", err, o.Claim.Op) |
| 3019 | + } |
| 3020 | +} |
| 3021 | ``` |
| 3022 | |
| 3023 | `internal/hostops/executor_test.go`: |
| 3024 | |
| 3025 | ```go |
| 3026 | package hostops |
| 3027 | |
| 3028 | import ( |
| 3029 | "context" |
| 3030 | "crypto/ed25519" |
| 3031 | "crypto/rand" |
| 3032 | "crypto/sha256" |
| 3033 | "encoding/hex" |
| 3034 | "encoding/json" |
| 3035 | "errors" |
| 3036 | "strings" |
| 3037 | "testing" |
| 3038 | "time" |
| 3039 | |
| 3040 | "github.com/carlosframework/platform/internal/boxchoice" |
| 3041 | "github.com/carlosframework/platform/internal/hostactors" |
| 3042 | "github.com/carlosframework/platform/internal/hostclaim" |
| 3043 | "github.com/carlosframework/platform/internal/release" |
| 3044 | "github.com/carlosframework/platform/internal/termfence" |
| 3045 | ) |
| 3046 | |
| 3047 | func chooserFixture(t *testing.T, boxes ...string) (ExecutorChooser, release.Store, *boxchoice.Snapshot, map[string]boxchoice.Load) { |
| 3048 | t.Helper() |
| 3049 | s := release.NewDirStore(t.TempDir()) |
| 3050 | pub, key, err := ed25519.GenerateKey(rand.Reader) |
| 3051 | if err != nil { |
| 3052 | t.Fatal(err) |
| 3053 | } |
| 3054 | roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, |
| 3055 | Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}}, |
| 3056 | Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} |
| 3057 | snap := &boxchoice.Snapshot{Complete: true} |
| 3058 | yes := true |
| 3059 | for i, ref := range boxes { |
| 3060 | roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", |
| 3061 | Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} |
| 3062 | snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true, |
| 3063 | HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) |
| 3064 | } |
| 3065 | wire, err := release.SignRootEnvelope(release.RootSigner(key), roster) |
| 3066 | if err != nil { |
| 3067 | t.Fatal(err) |
| 3068 | } |
| 3069 | if _, err := s.Put(context.Background(), hostactors.Key, wire, ""); err != nil { |
| 3070 | t.Fatal(err) |
| 3071 | } |
| 3072 | loads := map[string]boxchoice.Load{} |
| 3073 | x := ExecutorChooser{Directory: hostactors.Directory{Store: s, PublicKey: pub, Bucket: "bucket"}, SecretsKeyFP: "fp", |
| 3074 | Edges: func(context.Context) (boxchoice.Snapshot, error) { return *snap, nil }, |
| 3075 | Loads: func(context.Context) (map[string]boxchoice.Load, error) { return loads, nil }, |
| 3076 | Now: func() time.Time { return time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) }} |
| 3077 | return x, s, snap, loads |
| 3078 | } |
| 3079 | |
| 3080 | func chooserRecord() Record { |
| 3081 | return Record{Holder: hostclaim.NewHolder{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.oncarlos.com", Region: "ie"}} |
| 3082 | } |
| 3083 | |
| 3084 | func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) { |
| 3085 | ctx := context.Background() |
| 3086 | x, _, _, loads := chooserFixture(t, "box-a", "box-b") |
| 3087 | absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} |
| 3088 | first, err := x.Choose(ctx, absent, chooserRecord()) |
| 3089 | if err != nil { |
| 3090 | t.Fatal(err) |
| 3091 | } |
| 3092 | other := map[string]string{"box-a": "box-b", "box-b": "box-a"}[first.Box] |
| 3093 | if first.Enrollment != map[string]string{"box-a": strings.Repeat("a", 32), "box-b": strings.Repeat("b", 32)}[first.Box] { |
| 3094 | t.Fatalf("enrollment not the roster's: %+v", first) |
| 3095 | } |
| 3096 | // The hash winner already runs this app: spread prefers the other box. |
| 3097 | loads[first.Box] = boxchoice.Load{Hosts: 1, Apps: map[string]int{"bab/jam": 1}} |
| 3098 | if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other { |
| 3099 | t.Fatalf("soft spread: got %s, want %s", got.Box, other) |
| 3100 | } |
| 3101 | // Both run the app: the fewest hosts wins. |
| 3102 | loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}} |
| 3103 | if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box { |
| 3104 | t.Fatalf("headroom: got %s, want %s", got.Box, first.Box) |
| 3105 | } |
| 3106 | } |
| 3107 | |
| 3108 | // Round 18 finding 2: with equal loads the winner is the greatest |
| 3109 | // SHA-256 of host + "\x00" + box id (claim P5-D2), computed here |
| 3110 | // independently of boxchoice. The two hostnames are chosen so that their |
| 3111 | // winners differ: a chooser that hashed anything but the record's host |
| 3112 | // would get at least one of them wrong. |
| 3113 | func TestExecutorChooserHashesTheRecordsHostOnEqualLoads(t *testing.T) { |
| 3114 | ctx := context.Background() |
| 3115 | winner := func(host string) string { |
| 3116 | a := sha256.Sum256([]byte(host + "\x00box-a")) |
| 3117 | b := sha256.Sum256([]byte(host + "\x00box-b")) |
| 3118 | if hex.EncodeToString(a[:]) > hex.EncodeToString(b[:]) { |
| 3119 | return "box-a" |
| 3120 | } |
| 3121 | return "box-b" |
| 3122 | } |
| 3123 | hosts := []string{"shop.bab.oncarlos.com", "cafe.bab.oncarlos.com"} |
| 3124 | if winner(hosts[0]) == winner(hosts[1]) { |
| 3125 | t.Fatal("the fixture's hostnames must choose opposite boxes") |
| 3126 | } |
| 3127 | for _, host := range hosts { |
| 3128 | x, _, _, _ := chooserFixture(t, "box-a", "box-b") |
| 3129 | r := chooserRecord() |
| 3130 | r.Holder.Host = host |
| 3131 | got, err := x.Choose(ctx, hostclaim.Observed{Host: host, Absent: true}, r) |
| 3132 | if err != nil || got.Box != winner(host) { |
| 3133 | t.Fatalf("%s: chose %+v %v, want %s", host, got, err, winner(host)) |
| 3134 | } |
| 3135 | } |
| 3136 | } |
| 3137 | |
| 3138 | func TestExecutorChooserNeverNamesAFencedLostOrUnknownBox(t *testing.T) { |
| 3139 | ctx := context.Background() |
| 3140 | absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} |
| 3141 | t.Run("lost", func(t *testing.T) { |
| 3142 | x, _, _, _ := chooserFixture(t, "box-a") |
| 3143 | obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Lost: []string{"box-a"}}} |
| 3144 | if _, err := x.Choose(ctx, obs, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { |
| 3145 | t.Fatalf("chose a lost box: %v", err) |
| 3146 | } |
| 3147 | }) |
| 3148 | t.Run("owing a local retirement", func(t *testing.T) { |
| 3149 | // Spec 1 §2: a box whose old installation of the host is not yet |
| 3150 | // retired (park-local) is never chosen; the clean one is. |
| 3151 | x, _, _, _ := chooserFixture(t, "box-a", "box-b") |
| 3152 | obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Obligations: []hostclaim.Obligation{{ID: "old", Op: "old-op", Kind: "park-local", Box: "box-a", Retirement: "r", Owner: "box:box-a"}}}} |
| 3153 | for range 4 { |
| 3154 | if got, err := x.Choose(ctx, obs, chooserRecord()); err != nil || got.Box != "box-b" { |
| 3155 | t.Fatalf("got %+v %v, want the box with no chore", got, err) |
| 3156 | } |
| 3157 | } |
| 3158 | if err := x.Permits(ctx, obs, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { |
| 3159 | t.Fatalf("permitted a box owing park-local: %v", err) |
| 3160 | } |
| 3161 | }) |
| 3162 | t.Run("fenced", func(t *testing.T) { |
| 3163 | x, s, _, _ := chooserFixture(t, "box-a", "box-w") |
| 3164 | f := termfence.Record{V: 1, Box: "box-a", Enrollment: strings.Repeat("a", 32), State: "terminated", |
| 3165 | Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-box-a"}, |
| 3166 | ObservedAt: "2026-10-07T08:00:00Z", Via: termfence.ViaDescribe, RequestID: "req", |
| 3167 | WrittenBy: termfence.WrittenBy{Box: "box-w", Enrollment: strings.Repeat("b", 32)}} |
| 3168 | raw, _ := json.Marshal(f) |
| 3169 | if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", raw, ""); err != nil { |
| 3170 | t.Fatal(err) |
| 3171 | } |
| 3172 | got, err := x.Choose(ctx, absent, chooserRecord()) |
| 3173 | if err != nil || got.Box != "box-w" { |
| 3174 | t.Fatalf("got %+v %v, want the unfenced box", got, err) |
| 3175 | } |
| 3176 | }) |
| 3177 | t.Run("an edges file for a box the roster does not hold", func(t *testing.T) { |
| 3178 | x, _, snap, _ := chooserFixture(t) |
| 3179 | yes := true |
| 3180 | snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: "stray", EIP: "203.0.113.9", Region: "eu-west-1", Placement: "ie", Fresh: true, |
| 3181 | HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) |
| 3182 | if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { |
| 3183 | t.Fatalf("chose an unenrolled box: %v", err) |
| 3184 | } |
| 3185 | }) |
| 3186 | t.Run("an incomplete snapshot", func(t *testing.T) { |
| 3187 | x, _, snap, _ := chooserFixture(t, "box-a") |
| 3188 | snap.Complete = false |
| 3189 | if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { |
| 3190 | t.Fatalf("chose from a partial read: %v", err) |
| 3191 | } |
| 3192 | }) |
| 3193 | t.Run("an unreadable fence", func(t *testing.T) { |
| 3194 | x, s, _, _ := chooserFixture(t, "box-a") |
| 3195 | if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", []byte("{"), ""); err != nil { |
| 3196 | t.Fatal(err) |
| 3197 | } |
| 3198 | if _, err := x.Choose(ctx, absent, chooserRecord()); err == nil || errors.Is(err, ErrNoEligibleBox) { |
| 3199 | t.Fatalf("an unreadable fence must refuse, not skip: %v", err) |
| 3200 | } |
| 3201 | }) |
| 3202 | } |
| 3203 | |
| 3204 | func TestExecutorChooserPermitsOnlyTheEligibleSet(t *testing.T) { |
| 3205 | ctx := context.Background() |
| 3206 | absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} |
| 3207 | for _, tc := range []struct { |
| 3208 | name string |
| 3209 | change func(*boxchoice.Box) |
| 3210 | }{ |
| 3211 | {"a stale edges file", func(b *boxchoice.Box) { b.Fresh = false }}, |
| 3212 | {"no region-ha-1 capability", func(b *boxchoice.Box) { b.Capabilities = nil }}, |
| 3213 | {"another secrets key", func(b *boxchoice.Box) { b.SecretsKeyFP = "other" }}, |
| 3214 | {"a draining box", func(b *boxchoice.Box) { b.Draining = true }}, |
| 3215 | } { |
| 3216 | t.Run(tc.name, func(t *testing.T) { |
| 3217 | x, _, snap, _ := chooserFixture(t, "box-a", "box-b") |
| 3218 | if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); err != nil { |
| 3219 | t.Fatal(err) |
| 3220 | } |
| 3221 | tc.change(&snap.Boxes[0]) |
| 3222 | if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { |
| 3223 | t.Fatalf("permitted: %v", err) |
| 3224 | } |
| 3225 | if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { |
| 3226 | t.Fatalf("the other box: %v", err) |
| 3227 | } |
| 3228 | }) |
| 3229 | } |
| 3230 | t.Run("an unknown box", func(t *testing.T) { |
| 3231 | x, _, _, _ := chooserFixture(t, "box-a") |
| 3232 | if err := x.Permits(ctx, absent, chooserRecord(), "box-z"); !errors.Is(err, ErrNoEligibleBox) { |
| 3233 | t.Fatalf("permitted: %v", err) |
| 3234 | } |
| 3235 | }) |
| 3236 | } |
| 3237 | |
| 3238 | func TestClaimLoadsCountsExecutorsPerBoxAndApp(t *testing.T) { |
| 3239 | ctx := context.Background() |
| 3240 | s := release.NewDirStore(t.TempDir()) |
| 3241 | created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| 3242 | for i, tc := range []struct{ host, app, box string }{ |
| 3243 | {"a.bab.oncarlos.com", "jam", "box-a"}, {"b.bab.oncarlos.com", "jam", "box-a"}, {"c.bab.oncarlos.com", "tea", "box-b"}, |
| 3244 | } { |
| 3245 | nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + tc.app + "/instances/" + tc.host + ".json", Account: "bab", App: tc.app, Host: tc.host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| 3246 | body := []byte(`{"host":"` + tc.host + `","account":"bab","app":"` + tc.app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) |
| 3247 | o, _ := hostclaim.Read(ctx, s, tc.host) |
| 3248 | o, err := hostclaim.Update(ctx, s, o, hostclaim.BeginCreateOn(nh, body, "c", "console", created, hostclaim.ExecutorSelection{Box: tc.box, Enrollment: "e"})) |
| 3249 | if err != nil { |
| 3250 | t.Fatal(err) |
| 3251 | } |
| 3252 | if i < 2 { // the third is still creating: it has no executor yet |
| 3253 | if _, err := hostclaim.Update(ctx, s, o, hostclaim.CompleteCreate("c", tc.box)); err != nil { |
| 3254 | t.Fatal(err) |
| 3255 | } |
| 3256 | } |
| 3257 | } |
| 3258 | if _, err := s.Put(ctx, "control/hosts/broken.bab.oncarlos.com.json", []byte("{"), ""); err != nil { |
| 3259 | t.Fatal(err) |
| 3260 | } |
| 3261 | loads, err := ClaimLoads(ctx, s) |
| 3262 | if err != nil { |
| 3263 | t.Fatal(err) |
| 3264 | } |
| 3265 | if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 0 { |
| 3266 | t.Fatalf("loads: %+v", loads) |
| 3267 | } |
| 3268 | } |
| 3269 | |
| 3270 | func TestProductionAuthoritySelectsOnlyWithAChooser(t *testing.T) { |
| 3271 | // No chooser (CARLOS_REGION_HA off): no selection and no error, so the |
| 3272 | // create is today's. |
| 3273 | sel, selected, err := (ProductionAuthority{}).SelectExecutor(context.Background(), hostclaim.Observed{}, Record{}) |
| 3274 | if err != nil || selected || sel != (hostclaim.ExecutorSelection{}) { |
| 3275 | t.Fatalf("no chooser: %+v %v %v", sel, selected, err) |
| 3276 | } |
| 3277 | // With a chooser the selection is the chooser's, read from its |
| 3278 | // evidence: box-a is draining, so box-b with its roster enrollment; |
| 3279 | // with both draining, no box and an error, never a selection. |
| 3280 | x, _, snap, _ := chooserFixture(t, "box-a", "box-b") |
| 3281 | snap.Boxes[0].Draining = true |
| 3282 | absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} |
| 3283 | sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()) |
| 3284 | if err != nil || !selected || sel.Box != "box-b" || sel.Enrollment != strings.Repeat("b", 32) { |
| 3285 | t.Fatalf("with a chooser: %+v %v %v", sel, selected, err) |
| 3286 | } |
| 3287 | snap.Boxes[1].Draining = true |
| 3288 | if sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) || selected { |
| 3289 | t.Fatalf("no eligible box: %+v %v %v", sel, selected, err) |
| 3290 | } |
| 3291 | } |
| 3292 | ``` |
| 3293 | |
| 3294 | `internal/console/hostclaimexecutor_test.go`: |
| 3295 | |
| 3296 | ```go |
| 3297 | package console |
| 3298 | |
| 3299 | import ( |
| 3300 | "context" |
| 3301 | "slices" |
| 3302 | "testing" |
| 3303 | "time" |
| 3304 | |
| 3305 | "github.com/carlosframework/platform/internal/boxchoice" |
| 3306 | "github.com/carlosframework/platform/internal/hostactors" |
| 3307 | "github.com/carlosframework/platform/internal/hostclaim" |
| 3308 | "github.com/carlosframework/platform/internal/release" |
| 3309 | ) |
| 3310 | |
| 3311 | // The console's chooser reads the bucket through its two production |
| 3312 | // readers: the live edges (each box's edges file joined with its status |
| 3313 | // snapshot) and the serving claims' loads. The bucket holds real evidence, |
| 3314 | // the assertions name boxes and counts, and a change to the stored |
| 3315 | // evidence shows in the next read, so neither reader can be replaced by |
| 3316 | // one returning nothing (round 10). |
| 3317 | func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) { |
| 3318 | ctx := context.Background() |
| 3319 | store := release.NewDirStore(t.TempDir()) |
| 3320 | off := &Server{cfg: Config{Releases: store, HomeRegion: "ie"}} |
| 3321 | if off.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) != nil { |
| 3322 | t.Fatal("a chooser with CARLOS_REGION_HA off: create would change") |
| 3323 | } |
| 3324 | s := &Server{cfg: Config{Releases: store, HomeRegion: "ie", RegionHA: true}} |
| 3325 | x := s.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) |
| 3326 | if x == nil || x.Edges == nil || x.Loads == nil || x.Now == nil || x.HomeName != "ie" || x.SecretsKeyFP != s.expectedSecretsFP() { |
| 3327 | t.Fatalf("chooser: %+v", x) |
| 3328 | } |
| 3329 | |
| 3330 | // Two live boxes in ie, each with its edges file and a fresh status |
| 3331 | // snapshot; box-b advertises the region-ha-1 capability. |
| 3332 | edges := func(box, extra string) string { |
| 3333 | return `{"v":1,"box":"` + box + `","box_id":"` + box + `","eip":"203.0.113.1","region":"eu-west-1","placement":"ie","hibernation_configured":true` + extra + `,"hosts":{}}` |
| 3334 | } |
| 3335 | for box, body := range map[string]string{"box-a": edges("box-a", ""), "box-b": edges("box-b", `,"capabilities":["`+boxchoice.Capability+`"]`)} { |
| 3336 | setBoxSnapshot(t, store, box, time.Now()) |
| 3337 | if _, err := store.Put(ctx, "control/edges/"+box+".json", []byte(body), ""); err != nil { |
| 3338 | t.Fatal(err) |
| 3339 | } |
| 3340 | } |
| 3341 | // Two serving claims on box-a for bab/jam, one on box-b for bab/tea. |
| 3342 | created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| 3343 | serve := func(host, app, box string) { |
| 3344 | t.Helper() |
| 3345 | nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + app + "/instances/" + host + ".json", Account: "bab", App: app, Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| 3346 | body := []byte(`{"host":"` + host + `","account":"bab","app":"` + app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) |
| 3347 | o, err := hostclaim.Read(ctx, store, host) |
| 3348 | if err != nil { |
| 3349 | t.Fatal(err) |
| 3350 | } |
| 3351 | if o, err = hostclaim.Update(ctx, store, o, hostclaim.BeginCreate(nh, body, "c", "console", created)); err != nil { |
| 3352 | t.Fatal(err) |
| 3353 | } |
| 3354 | if _, err = hostclaim.Update(ctx, store, o, hostclaim.CompleteCreate("c", box)); err != nil { |
| 3355 | t.Fatal(err) |
| 3356 | } |
| 3357 | } |
| 3358 | serve("a.bab.oncarlos.com", "jam", "box-a") |
| 3359 | serve("b.bab.oncarlos.com", "jam", "box-a") |
| 3360 | serve("c.bab.oncarlos.com", "tea", "box-b") |
| 3361 | |
| 3362 | read := func() (map[string]boxchoice.Box, map[string]boxchoice.Load) { |
| 3363 | t.Helper() |
| 3364 | snap, err := x.Edges(ctx) |
| 3365 | if err != nil || !snap.Complete { |
| 3366 | t.Fatalf("edges: %+v %v", snap, err) |
| 3367 | } |
| 3368 | boxes := map[string]boxchoice.Box{} |
| 3369 | for _, b := range snap.Boxes { |
| 3370 | boxes[b.BoxID] = b |
| 3371 | } |
| 3372 | loads, err := x.Loads(ctx) |
| 3373 | if err != nil { |
| 3374 | t.Fatal(err) |
| 3375 | } |
| 3376 | return boxes, loads |
| 3377 | } |
| 3378 | boxes, loads := read() |
| 3379 | if len(boxes) != 2 || !boxes["box-a"].Fresh || !boxes["box-b"].Fresh || boxes["box-a"].Placement != "ie" || |
| 3380 | slices.Contains(boxes["box-a"].Capabilities, boxchoice.Capability) || !slices.Contains(boxes["box-b"].Capabilities, boxchoice.Capability) { |
| 3381 | t.Fatalf("live edges: %+v", boxes) |
| 3382 | } |
| 3383 | if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 1 || loads["box-b"].Apps["bab/tea"] != 1 { |
| 3384 | t.Fatalf("loads: %+v", loads) |
| 3385 | } |
| 3386 | |
| 3387 | // The evidence changes: box-a starts draining and stops reporting |
| 3388 | // (stale), and a fourth host serves on box-b. The next read says so. |
| 3389 | _, etag, err := store.Get(ctx, "control/edges/box-a.json") |
| 3390 | if err != nil { |
| 3391 | t.Fatal(err) |
| 3392 | } |
| 3393 | if _, err := store.Put(ctx, "control/edges/box-a.json", []byte(edges("box-a", `,"draining":true`)), etag); err != nil { |
| 3394 | t.Fatal(err) |
| 3395 | } |
| 3396 | setBoxSnapshot(t, store, "box-a", time.Now().Add(-time.Hour)) |
| 3397 | serve("d.bab.oncarlos.com", "tea", "box-b") |
| 3398 | boxes, loads = read() |
| 3399 | if !boxes["box-a"].Draining || boxes["box-a"].Fresh || !boxes["box-b"].Fresh { |
| 3400 | t.Fatalf("live edges after the change: %+v", boxes) |
| 3401 | } |
| 3402 | if loads["box-b"].Hosts != 2 || loads["box-b"].Apps["bab/tea"] != 2 || loads["box-a"].Hosts != 2 { |
| 3403 | t.Fatalf("loads after the change: %+v", loads) |
| 3404 | } |
| 3405 | } |
| 3406 | ``` |
| 3407 | |
| 3408 | `cmd/console/regionha_test.go` (the switch that sets `Config.RegionHA`): |
| 3409 | |
| 3410 | ```go |
| 3411 | package main |
| 3412 | |
| 3413 | import "testing" |
| 3414 | |
| 3415 | // CARLOS_REGION_HA reaches Config.RegionHA through regionHAEnabled |
| 3416 | // (strconv.ParseBool): only a true value turns the console's create-time |
| 3417 | // chooser on; unset, false and garbage leave create as today. |
| 3418 | func TestRegionHAEnabledParsesTheSwitch(t *testing.T) { |
| 3419 | for v, want := range map[string]bool{"": false, "0": false, "false": false, "no": false, "garbage": false, |
| 3420 | "1": true, "t": true, "true": true, "TRUE": true, " true ": true} { |
| 3421 | got := regionHAEnabled(func(k string) string { |
| 3422 | if k != "CARLOS_REGION_HA" { |
| 3423 | t.Fatalf("read %s", k) |
| 3424 | } |
| 3425 | return v |
| 3426 | }) |
| 3427 | if got != want { |
| 3428 | t.Errorf("CARLOS_REGION_HA=%q: %v, want %v", v, got, want) |
| 3429 | } |
| 3430 | } |
| 3431 | } |
| 3432 | ``` |
| 3433 | |
| 3434 | - [ ] **Step 2: Run them to see them fail** |
| 3435 | |
| 3436 | Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ ./internal/console/ -run 'SelectedExecutor|ExecutorChooser|ClaimLoads|ProductionAuthoritySelects|CreateIsTodays' -count=1` |
| 3437 | Expected: FAIL to compile: `undefined: ExecutorSelection`, `undefined: BeginCreateOn`, `undefined: ExecutorChooser`, `undefined: ClaimLoads`, `s.executorChooser undefined`. |
| 3438 | |
| 3439 | - [ ] **Step 3: Record the selection in the create intent** |
| 3440 | |
| 3441 | Apply to `internal/hostclaim/create.go`: |
| 3442 | |
| 3443 | ```diff |
| 3444 | diff --git a/internal/hostclaim/create.go b/internal/hostclaim/create.go |
| 3445 | index c8fe97cc..39badc04 100644 |
| 3446 | --- a/internal/hostclaim/create.go |
| 3447 | +++ b/internal/hostclaim/create.go |
| 3448 | @@ -80,6 +80,32 @@ func BeginCreate(nh NewHolder, recordBody []byte, opID, by string, now time.Time |
| 3449 | } |
| 3450 | } |
| 3451 | |
| 3452 | +// ExecutorSelection is the box create selects once, with |
| 3453 | +// boxchoice.Choose over the eligible set (claim P5-D2, region HA §5.2, |
| 3454 | +// H12), and its roster enrollment. |
| 3455 | +type ExecutorSelection struct { |
| 3456 | + Box, Enrollment string |
| 3457 | +} |
| 3458 | + |
| 3459 | +// BeginCreateOn is BeginCreate recording the selected executor in the |
| 3460 | +// create intent, in the same step 2 CAS, before any effect (claim §5.1, |
| 3461 | +// P5-D6 `selected_executor`, `executor_enrollment`). CompleteCreate then |
| 3462 | +// sets op.exec to exactly that box; recovery never substitutes another. |
| 3463 | +func BeginCreateOn(nh NewHolder, recordBody []byte, opID, by string, now time.Time, sel ExecutorSelection) func(*Claim) error { |
| 3464 | + begin := BeginCreate(nh, recordBody, opID, by, now) |
| 3465 | + return func(c *Claim) error { |
| 3466 | + if !validBoxReference(sel.Box) || sel.Box == "none" || sel.Enrollment == "" { |
| 3467 | + return fmt.Errorf("%w: create needs a selected executor", ErrMalformed) |
| 3468 | + } |
| 3469 | + if err := begin(c); err != nil { |
| 3470 | + return err |
| 3471 | + } |
| 3472 | + c.Op.Intent["selected_executor"] = sel.Box |
| 3473 | + c.Op.Intent["executor_enrollment"] = sel.Enrollment |
| 3474 | + return nil |
| 3475 | + } |
| 3476 | +} |
| 3477 | + |
| 3478 | func CompleteCreate(opID, execBox string) func(*Claim) error { |
| 3479 | return func(c *Claim) error { |
| 3480 | if err := RequireOp(opID)(c); err != nil { |
| 3481 | @@ -88,6 +114,9 @@ func CompleteCreate(opID, execBox string) func(*Claim) error { |
| 3482 | if c.Op.Kind != KindCreating || execBox == "" { |
| 3483 | return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind) |
| 3484 | } |
| 3485 | + if selected := c.Op.Intent["selected_executor"]; selected != "" && execBox != selected { |
| 3486 | + return fmt.Errorf("%w: create selected %s; completion never substitutes %s", ErrWrongState, selected, execBox) |
| 3487 | + } |
| 3488 | if err := requireNoCreateAuthority(c); err != nil { |
| 3489 | return err |
| 3490 | } |
| 3491 | ``` |
| 3492 | |
| 3493 | - [ ] **Step 4: The chooser and the loads** |
| 3494 | |
| 3495 | `internal/hostops/executor.go`: |
| 3496 | |
| 3497 | ```go |
| 3498 | package hostops |
| 3499 | |
| 3500 | import ( |
| 3501 | "context" |
| 3502 | "errors" |
| 3503 | "fmt" |
| 3504 | "time" |
| 3505 | |
| 3506 | "github.com/carlosframework/platform/internal/boxchoice" |
| 3507 | "github.com/carlosframework/platform/internal/hostactors" |
| 3508 | "github.com/carlosframework/platform/internal/hostclaim" |
| 3509 | ) |
| 3510 | |
| 3511 | // ErrNoEligibleBox is a create refused because no box in the placement is |
| 3512 | // eligible (region HA §5.2). The console maps it to today's refusal. |
| 3513 | var ErrNoEligibleBox = errors.New("hostops: region has no box that can take this") |
| 3514 | |
| 3515 | // ExecutorChooser is create's executor selection: claim P5-D2's choose() |
| 3516 | // over the complete eligible set (region HA spec 1 §5.2, H12, L8). It is |
| 3517 | // made once, before the create CAS, and recorded in the create intent. |
| 3518 | type ExecutorChooser struct { |
| 3519 | Directory hostactors.Directory |
| 3520 | // Edges is one read of every box's edges file with the caller's |
| 3521 | // liveness verdict per box. An incomplete snapshot selects nothing. |
| 3522 | Edges func(context.Context) (boxchoice.Snapshot, error) |
| 3523 | // Loads counts claimed hosts per box and per "acct/app", from the |
| 3524 | // claim-recovery listing (claim §5.1 step 5). A stale count only |
| 3525 | // affects balance, never safety. |
| 3526 | Loads func(context.Context) (map[string]boxchoice.Load, error) |
| 3527 | SecretsKeyFP string |
| 3528 | HomeName string |
| 3529 | Now func() time.Time |
| 3530 | } |
| 3531 | |
| 3532 | // Choose returns the box, and its roster enrollment, that a new host in |
| 3533 | // r's placement is created on. Identity, placement and AWS region come from |
| 3534 | // the roster; a box with a valid termination fence, or not active, is never |
| 3535 | // chosen; a box on the claim's lost list, or owing a park-local chore on |
| 3536 | // it, is never chosen. |
| 3537 | func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) { |
| 3538 | o, cands, err := x.eligible(ctx, obs, r) |
| 3539 | if err != nil { |
| 3540 | return hostclaim.ExecutorSelection{}, err |
| 3541 | } |
| 3542 | loads, err := x.Loads(ctx) |
| 3543 | if err != nil { |
| 3544 | return hostclaim.ExecutorSelection{}, err |
| 3545 | } |
| 3546 | best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) |
| 3547 | if !ok { |
| 3548 | return hostclaim.ExecutorSelection{}, ErrNoEligibleBox |
| 3549 | } |
| 3550 | return hostclaim.ExecutorSelection{Box: best.BoxID, Enrollment: o.Roster.Boxes[best.BoxID].Enrollment}, nil |
| 3551 | } |
| 3552 | |
| 3553 | // Permits reports whether box is in the complete eligible set for a host |
| 3554 | // in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ... |
| 3555 | // when `to` is not eligible for the host"): the same read Choose makes, |
| 3556 | // without choosing. A stale edges file, a missing capability or secrets |
| 3557 | // key, a draining box, a fence, or an incomplete snapshot refuses. |
| 3558 | func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error { |
| 3559 | _, cands, err := x.eligible(ctx, obs, r) |
| 3560 | if err != nil { |
| 3561 | return err |
| 3562 | } |
| 3563 | for _, c := range cands { |
| 3564 | if c.BoxID == box { |
| 3565 | return nil |
| 3566 | } |
| 3567 | } |
| 3568 | return fmt.Errorf("%w: %s", ErrNoEligibleBox, box) |
| 3569 | } |
| 3570 | |
| 3571 | // eligible is one read of the roster, the fences and the edges files, |
| 3572 | // reduced to boxchoice.Eligible's candidates. |
| 3573 | func (x ExecutorChooser) eligible(ctx context.Context, obs hostclaim.Observed, r Record) (hostactors.Observed, []boxchoice.Candidate, error) { |
| 3574 | placement := r.Holder.Region |
| 3575 | if x.Edges == nil || x.Loads == nil || x.Now == nil || placement == "" { |
| 3576 | return hostactors.Observed{}, nil, ErrContract |
| 3577 | } |
| 3578 | o, err := x.Directory.Read(ctx) |
| 3579 | if err != nil { |
| 3580 | return hostactors.Observed{}, nil, err |
| 3581 | } |
| 3582 | region, ok := o.Roster.Regions[placement] |
| 3583 | if !ok || region.AWSRegion == "" { |
| 3584 | return hostactors.Observed{}, nil, ErrContract |
| 3585 | } |
| 3586 | roster := map[string]boxchoice.RosterBox{} |
| 3587 | for ref, b := range o.Roster.Boxes { |
| 3588 | if b.State != "active" || b.Region != placement { |
| 3589 | continue |
| 3590 | } |
| 3591 | fenced, err := x.Directory.TerminationFence(ctx, o, ref, x.Now()) |
| 3592 | if err != nil { |
| 3593 | return hostactors.Observed{}, nil, err // C6: a read error never means "not fenced" |
| 3594 | } |
| 3595 | if !fenced { |
| 3596 | roster[ref] = boxchoice.RosterBox{Placement: b.Region, Cloud: b.Cloud} |
| 3597 | } |
| 3598 | } |
| 3599 | snap, err := x.Edges(ctx) |
| 3600 | if err != nil { |
| 3601 | return hostactors.Observed{}, nil, err |
| 3602 | } |
| 3603 | // Never a box on the claim's lost list, nor one that still owes a |
| 3604 | // park-local chore for this host (spec 1 §2: its old installation is |
| 3605 | // not yet retired). |
| 3606 | excluded := map[string]bool{} |
| 3607 | if !obs.Absent && obs.Claim != nil { |
| 3608 | for _, box := range obs.Claim.Lost { |
| 3609 | excluded[box] = true |
| 3610 | } |
| 3611 | for _, box := range hostclaim.LocalChoreBoxes(obs.Claim) { |
| 3612 | excluded[box] = true |
| 3613 | } |
| 3614 | } |
| 3615 | return o, boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, excluded), nil |
| 3616 | } |
| 3617 | ``` |
| 3618 | |
| 3619 | `internal/hostops/loads.go`: |
| 3620 | |
| 3621 | ```go |
| 3622 | package hostops |
| 3623 | |
| 3624 | import ( |
| 3625 | "context" |
| 3626 | "log" |
| 3627 | "strings" |
| 3628 | |
| 3629 | "github.com/carlosframework/platform/internal/boxchoice" |
| 3630 | "github.com/carlosframework/platform/internal/hostclaim" |
| 3631 | "github.com/carlosframework/platform/internal/release" |
| 3632 | ) |
| 3633 | |
| 3634 | // ClaimLoads counts, from one listing of control/hosts/, the hosts whose |
| 3635 | // claim names each box as op.exec, in total and per "acct/app": choose()'s |
| 3636 | // second rule until spec 2 (region HA §5.2, claim P5-D2). The count only |
| 3637 | // balances, so a claim that cannot be read is logged and left out; a |
| 3638 | // listing that fails is an error, because a count of nothing would send |
| 3639 | // every create to one box. |
| 3640 | func ClaimLoads(ctx context.Context, s release.Store) (map[string]boxchoice.Load, error) { |
| 3641 | keys, err := s.List(ctx, "control/hosts/") |
| 3642 | if err != nil { |
| 3643 | return nil, err |
| 3644 | } |
| 3645 | out := map[string]boxchoice.Load{} |
| 3646 | for _, key := range keys { |
| 3647 | name, ok := strings.CutPrefix(key, "control/hosts/") |
| 3648 | host, isJSON := strings.CutSuffix(name, ".json") |
| 3649 | if !ok || !isJSON || strings.Contains(host, "/") { |
| 3650 | continue |
| 3651 | } |
| 3652 | obs, err := hostclaim.Read(ctx, s, host) |
| 3653 | if err != nil { |
| 3654 | log.Printf("hostops: claim loads: %s: %v", host, err) |
| 3655 | continue |
| 3656 | } |
| 3657 | if obs.Absent || obs.Claim.Holder == nil || obs.Claim.Op.Exec == "" || obs.Claim.Op.Exec == "none" { |
| 3658 | continue |
| 3659 | } |
| 3660 | box, h := obs.Claim.Op.Exec, obs.Claim.Holder |
| 3661 | l := out[box] |
| 3662 | if l.Apps == nil { |
| 3663 | l.Apps = map[string]int{} |
| 3664 | } |
| 3665 | l.Hosts++ |
| 3666 | l.Apps[h.Account+"/"+h.App]++ |
| 3667 | out[box] = l |
| 3668 | } |
| 3669 | return out, nil |
| 3670 | } |
| 3671 | ``` |
| 3672 | |
| 3673 | - [ ] **Step 5: Select before the create CAS, or create as today** |
| 3674 | |
| 3675 | Apply to `internal/hostops/operations.go`: |
| 3676 | |
| 3677 | ```diff |
| 3678 | diff --git a/internal/hostops/operations.go b/internal/hostops/operations.go |
| 3679 | index 8313b80e..d3f1665a 100644 |
| 3680 | --- a/internal/hostops/operations.go |
| 3681 | +++ b/internal/hostops/operations.go |
| 3682 | @@ -14,6 +14,12 @@ import ( |
| 3683 | // These callbacks are authority providers, not optional best-effort hints. |
| 3684 | // Production construction waits for the decisions named in this plan. |
| 3685 | type Authority interface { |
| 3686 | + // SelectExecutor chooses the new host's executor once, before the |
| 3687 | + // create CAS (claim P5-D2, region HA §5.2). selected false means this |
| 3688 | + // deployment does not choose at create (CARLOS_REGION_HA off): the |
| 3689 | + // create proceeds exactly as today, and Executor names the box at |
| 3690 | + // completion. Production is ExecutorChooser.Choose behind that switch. |
| 3691 | + SelectExecutor(context.Context, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error) |
| 3692 | Create(context.Context, hostclaim.Observed, Record, string) error |
| 3693 | Executor(context.Context, hostclaim.Observed, Record) (string, error) |
| 3694 | StartDelete(context.Context, hostclaim.Observed, Record) error |
| 3695 | @@ -46,11 +52,21 @@ func (c Controller) Attempt(ctx context.Context, before hostclaim.Observed, r Re |
| 3696 | if before.Host != r.Holder.Host || r.Holder.CreatedAt.IsZero() || id == "" { |
| 3697 | return hostclaim.Observed{}, ErrIdentity |
| 3698 | } |
| 3699 | + // With region HA on, the executor is selected once, before the CAS, |
| 3700 | + // and recorded in it; off, the create is today's. |
| 3701 | + sel, selected, err := c.Authority.SelectExecutor(ctx, before, r) |
| 3702 | + if err != nil { |
| 3703 | + return hostclaim.Observed{}, err |
| 3704 | + } |
| 3705 | + begin := hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now()) |
| 3706 | + if selected { |
| 3707 | + begin = hostclaim.BeginCreateOn(r.Holder, r.Body, id, by, c.Now(), sel) |
| 3708 | + } |
| 3709 | // Last pre-CAS read of registration, open admission and purge reservation. |
| 3710 | if err := c.Authority.Create(ctx, before, r, id); err != nil { |
| 3711 | return hostclaim.Observed{}, err |
| 3712 | } |
| 3713 | - obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now())) |
| 3714 | + obs, err := hostclaim.Update(ctx, c.Store, before, begin) |
| 3715 | if err != nil { |
| 3716 | return hostclaim.Observed{}, err |
| 3717 | } |
| 3718 | ``` |
| 3719 | |
| 3720 | Apply to `internal/hostops/backfill_controller.go` (the backfill grants no create): |
| 3721 | |
| 3722 | ```diff |
| 3723 | diff --git a/internal/hostops/backfill_controller.go b/internal/hostops/backfill_controller.go |
| 3724 | index 08ebd856..dcbf750e 100644 |
| 3725 | --- a/internal/hostops/backfill_controller.go |
| 3726 | +++ b/internal/hostops/backfill_controller.go |
| 3727 | @@ -23,6 +23,9 @@ func (a BackfillAuthority) Adoption(ctx context.Context, r Record) (string, stri |
| 3728 | } |
| 3729 | func (a BackfillAuthority) ClaimBackfill() *Backfill { return a.p.Backfill } |
| 3730 | |
| 3731 | +func (a BackfillAuthority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, bool, error) { |
| 3732 | + return hostclaim.ExecutorSelection{}, false, nil // the backfill grants no create (Create refuses) |
| 3733 | +} |
| 3734 | func (a BackfillAuthority) Create(context.Context, hostclaim.Observed, Record, string) error { |
| 3735 | return ErrContract |
| 3736 | } |
| 3737 | ``` |
| 3738 | |
| 3739 | Apply to `internal/hostops/backfill.go`: |
| 3740 | |
| 3741 | ```diff |
| 3742 | diff --git a/internal/hostops/backfill.go b/internal/hostops/backfill.go |
| 3743 | index e20b1982..7c727f9d 100644 |
| 3744 | --- a/internal/hostops/backfill.go |
| 3745 | +++ b/internal/hostops/backfill.go |
| 3746 | @@ -89,6 +89,22 @@ type ProductionAuthority struct { |
| 3747 | Signer release.Signer |
| 3748 | Evidence hostactors.Evidence |
| 3749 | Backfill *Backfill |
| 3750 | + // Executors is create's executor selection (region HA §5.2, L8). The |
| 3751 | + // create door sets it only with CARLOS_REGION_HA on; nil keeps today's |
| 3752 | + // create path, so a c |