carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

File too large to show in full — download it above (showing a truncated preview below).

1# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 19)
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18).
6
7**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 15 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader.
8
9**Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`).
10
11**Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds.
12
13## Rev 19: round 18 dispositions
14
15Astra's round 18 (`CODEX-PLAN-REGION-HA-1BN-R18.md`) resolved all three round 17 findings, accepted rulings 1 to 27, and raised two P2 test gaps. Both were checked against the plan's tests and accepted; none is declined. Neither exposed a production defect: the new events and oracle found no failing seed in the unchanged code. Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, and then each lifecycle guard was removed in turn.
16
17| # | Stage | Evidence checked | Disposition | What changed | Where |
18|---|---|---|---|---|---|
19| 1 (P2) | A | The property's events only started and completed operations. Its recovery oracle wanted every relocation to be ready again on this box. So it could not express a box excluded from an operation that goes on, and it would have rejected correct recovery there. | accepted | New events: `replaced` (this box, the target, is replaced within the same operation by `sp2`, and the relocation continues) and `probeFailedAndReleased`. In every fourth seed, a script follows round 18's trace: target installed, probe fails, release journaled, replaced, cleanup blocked, a pass, an agent crash, then the random events and recovery. The oracle now asks a participating box to be ready, and an excluded box to have finished its cleanup: no controller, no hostexec record, no journaled participation, with the operation still running. New invariant (i): nothing for an operation opens again on a box excluded from it. Invariants (a), cleanup before clear, and (h), persistence across a crash, stay. | Task 7 (`TestRelocationControllerLifecycleProperties`) |
20| 2 (P2) | A | `TestExecutorChooserSpreadsThenBalancesThenHashes` took the first choice as given and derived its later expectations from it. A chooser that hashed a constant instead of the record's host passed. | accepted | `TestExecutorChooserHashesTheRecordsHostOnEqualLoads` uses equal loads. It computes the winner independently (the greatest SHA-256 of host + "\x00" + box id, claim P5-D2) for `shop.bab.oncarlos.com` (box-b) and `cafe.bab.oncarlos.com` (box-a). The test itself asserts that the two winners differ, and requires `Choose` to return each one. Substituting a constant hostname in `ExecutorChooser.Choose` fails it (cafe chose box-b). | Task 2 |
21
22Mutation check (property at 200 seeds, `-failfast`): `Sweep` finishing a participation whose box is excluded, killed at seed 4 (the excluded box never finishes its cleanup). Rev 14's guards, rechecked under the new events, were all killed again: the cleanup before a clear (seed 1, (a)), the verified forget (seed 3, (b)), lost-controller recovery (seed 10), the journal before the open (seed 1, (b)), the close before the clear (seed 6), and the restart path's verification (seed 1, (b)).
23
24## Rev 18: round 17 dispositions
25
26Astra's round 17 (`CODEX-PLAN-REGION-HA-1BN-R17.md`) rated F1 to F3 as only partly closed, with one P1 and two P2s. It accepted ruling 26 and its cost. All three findings were checked against the rev 17 code, reproduced, and accepted; none is declined.
27
28Stage A was dry-run on a clean 3407d114 in task order, and the lifecycle property passed 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test.
29
30| # | Stage | Evidence checked | Disposition | What changed | Where |
31|---|---|---|---|---|---|
32| 1 (P1, F2) | A, spec | Rev 17's `filesPin` returned the first manifest it found before checking any evidence. §5.6's pre-start copy can lack the `instance.key` the app mints after starting, so a nonempty database was pinned with no key. | accepted | `Pin` computes `needKey`: a writer was admitted and the pinned database is nonempty. `filesPin` then refuses with `ErrFilesUnevidenced` unless the pinned manifest has a live `instance.key` entry (`InstanceKeyPath`, `liveEntry`), and the worker marks the operation `stuck` (rev 17's path). Ruling 25 is amended. The durability window is recorded as the open request L21, with both candidate contracts, for Paul. | Task 5 (`TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase`: keyless with a nonempty database refuses; keyed pins; keyless with an empty database at a release of 0 pins); Requests (L21) |
33| 2 (P2, F1) | A | `Worker.Step` sends a fenced target to `replace` before any phase check. `replace` returned the chooser's error for ever, so a target terminated after its release, with no spare, bypassed rev 17's bound. | accepted | New fence-consuming `hostclaim.FenceTargetWithoutSpare`, sharing `FenceExhaustedTarget`'s body (`fenceTargetStuck`): it retains the loss, puts the box on `lost`, revokes a dead executor and records `stuck` with its reason. `replace` calls it `ReleaseDeadline` after the fence's `observed_at` when no eligible box exists. | Task 1; Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare/target_terminated_after_its_release_true`: waiting before the deadline; after it, `stuck` with the fence consumed, `dst` lost and `op.exec` "none") |
34| 3 (P2, F3) | A | `ReplaceTarget`'s default branch (`probe-failed`, or after P's admission) deleted `prepared` without recording an unopened P. That happens when the probe deadline passes before any admission and the target is then terminated. | accepted | `abandonPrepared(c)` records the current P when no writer opened it. `install` and `ReplaceTarget`'s default branch call it before they replace or drop P. `FenceExhaustedTarget` and `FenceTargetWithoutSpare` keep an unopened P for the operator, so they drop nothing. | Task 1 (`TestRelocationParksAPDroppedFromProbeFailed`: probe deadline before admission, target terminated, replaced, nested relocation completed; P1 is parked) |
35
36Guards removed, each failing its test: the keyless refusal; `needKey` only for a nonempty database; the dead-target bound; the bound consuming the fence; `ReplaceTarget` recording the P it drops. That is five.
37
38## Rev 17: two-box model findings
39
40The joint model (`origin/region-ha-hostlife-two-box`, model plan rev 2 at 6d25c54e) reported three gaps in rev 16 that review rounds 1 to 16 missed. Each comes with a direct test (`TestFailoverBeforeTheFirstManifest`, `TestProbeFailedInATwoBoxRegion`, `TestAbandonedPreparedGenerationParked`). I checked each against this plan's code first, and all three reproduce. All are accepted; none is declined.
41
42Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed its 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test.
43
44| # | Stage | Evidence checked | Disposition | What changed | Where |
45|---|---|---|---|---|---|
46| F2 (data loss) | A, B | `StorePinner.filesPin` pinned an empty files set whenever no manifest existed in the admitted chain, whatever the holder's writer history. An install from it starts with no `instance.key`, and the app mints a new one over data sealed with the old (model INV-4). | accepted (ruling 25) | `filesPin` refuses with `ErrFilesUnevidenced` unless no writer was ever admitted (session 0) or the journaled release flushed nothing (`manifest_seq` 0). `Worker.pinUnderLease` marks the operation `stuck` on that error. Stage B: 6b's cold admission writes its manifest copy before the app starts (§5.6), with a named test. | Task 5 (`TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter`, `TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerMarksAnUnevidencedFilesPinStuck`); stage B re-plan |
47| F1 (liveness) | A | In `probe-failed` with the release journaled, `Worker.Step` returned the chooser's error forever ("a later pass may find one"). With two boxes no spare exists, and `replaceTargetSlot` refuses the source. | accepted (ruling 26: stuck, not the source) | `ReleaseDeadline` after `probe_failed_since` with still no eligible box, the worker marks the operation `stuck`. Before that it waits as before. | Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare`: waiting before the deadline, `stuck` after it) |
48| F3 (leak) | A | `CompleteRelocation` queued `park-local` chores only. A P replaced while unopened (at `installing`, or a deadline `probe-failed` before any admission) stayed in `holder.location` for the at-rest sweep alone. | accepted (ruling 27) | `install` journals every unopened P it replaces in `abandoned_prepared`. The serving CAS queues a `park-generation` for each in the region (`Owner` `region:<r>`), deduplicated, never for an opened P. | Task 1 (`TestRelocationParksEveryAbandonedPreparedGeneration`: replaced at `installing`, completed on P2. P1 is parked and P2 is not.) |
49
50Guards removed, each failing its test: `install` journaling the abandoned P; the serving CAS queuing its park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; and waiting before the bound. That is eight.
51
52Checked and unchanged: a target terminated with no spare in `starting` or `releasing` also waits for a spare. The live source still serves there (it is `op.exec` until its admission fence closes), so that is not an outage. The model's finding is the `probe-failed` case only.
53
54## Rev 16: round 15 dispositions
55
56Astra's round 15 (`CODEX-PLAN-REGION-HA-1BN-R15.md`) resolved all four round 14 items and raised two P2s. I checked both against spec §5.8 (line 847, "a box that finds its receipt missing after that point … the op goes to `stuck`") and against the rev 15 property code. Both are accepted; none is declined.
57
58Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, in about four and a half minutes. Each new guard was then removed in turn.
59
60| # | Stage | Evidence checked | Disposition | What changed | Where |
61|---|---|---|---|---|---|
62| 1 (P2) | A, B | Rev 15's `Step` called `EnsureLocal` only while P was unopened. After P opened it went straight to the next attempt, so a lost receipt never made the operation `stuck` as §5.8 requires. | accepted | Receipt verification is now separate from restoration. A new effect, `VerifyReceipt`, runs before every attempt once P is opened. It compares this box's durable receipt (the new `Receipt` provider: the install journal under its `install_stamp`) with the journaled `local_install`, and reads nothing else. If the receipt is missing or different, `Step` marks the operation `stuck` (`MarkStuck`, allowed in `probing` once P is opened) and makes no admission and no reinstall. Files, env and timers changed by the admitted writers are never compared or reverted. Stage B: the production `Receipt`, with named tests. | Task 6 (`TestAParticipantIsStuckWhenItsReceiptIsLostAfterPOpens`: P is opened and the agent restarts. With the receipt lost, the claim goes `stuck` and no admission follows. With it intact, the next attempt goes ahead and nothing is reinstalled or verified against content.) Task 7 (`TestVerifyReceiptChecksTheReceiptOnly`). Stage B re-plan |
63| 2 (P2) | A | Rev 15's `restart` repaired the faults it was meant to survive: it unblocked and un-failed the cleanup, and closed the old controllers in the background, so their hooks ran. | accepted | `restart` becomes `crash`, an abrupt death. No controller is closed and no hook runs or completes. A dead generation's cleanups, including ones waiting when it died, never return. Its clock never fires again, and the next generation uses fresh lock files: the kernel releases a dead process's locks. Durable records and every external fault persist; recovery stays its own later event. New invariant (h): a record goes only through a successful clear or a successor's open over it, never through a crash or any other step, and while a record exists root's §5.10 check refuses an executable update. | Task 7 (the property, 200 seeds) |
64| C1 | A | Found by the new crash event. In-process, a dead generation's hostexec goroutines still run: `publish` retries its "lost" write through a directory fd it already holds open. After a repair it recreated a record a live generation had just cleared (seed 75). | fixed in the test | Each generation reaches the data directory through its own symlink, and a crash removes the dead one, so newly opened paths fail. The repair waits 300 ms so a write already in flight through an open fd lands as part of the repair. A real dead process writes nothing, so production code is unchanged. | Task 7 (the property's `dataLink`, `recoverAll`) |
65
66Mutation check, with the property at 200 seeds and `-failfast`, plus targeted tests:
67
68| Guard removed | Result |
69|---|---|
70| `Step` marking `stuck` on a lost receipt after P opens | killed: `TestAParticipantIsStuck…` |
71| `VerifyReceipt` comparing the receipt exactly | killed: `TestVerifyReceiptChecksTheReceiptOnly` |
72| both `EnsureLocal` calls | killed, seed 182: (e) |
73| `EnsureLocal` before an attempt alone / before entering probing alone | each survived: the other covers it |
74| rev 14's ten lifecycle guards | all killed again; the crash event now kills "journal before the open" at (b) and the handover at (h) |
75
76While building C1 I also tried an in-flight-open guard: no forgetting while an abandoned open might still write. It survived 200 seeds, so I removed it. `ClearStale` already refuses while such an open holds the host's lock, and `recordIsMine` refuses on anything unclear.
77
78Astra's readings of rulings 1 to 24 stand.
79
80## Rev 15: round 14 dispositions
81
82Astra's round 14 (`CODEX-PLAN-REGION-HA-1BN-R14.md`) resolved the lease-recovery finding. It left round 13's finding 2 partly resolved and raised three new P2s. All four are accepted; none is declined.
83
84Stage A was dry-run again on a clean 3407d114, applied in task order. The lifecycle property passed its 200 seeds (about three minutes, niced) and was re-run for every mutation. Each new guard was removed in turn; the results are below.
85
86| # | Stage | Evidence checked | Disposition | What changed | Where |
87|---|---|---|---|---|---|
88| R13-2 (partly) | A | Rev 14's `recordIsMine` decoded the JSON itself. A parseable record without its enrollment or incarnation, which hostexec's own reader refuses (`State.validate`, `Binding.valid`), was taken for another binding, and the participation was forgotten. | accepted | New `hostexec.ReadLocal(dataDir, host)` (`internal/hostexec/readlocal.go`) is hostexec's validated reader: the same checks `Open`, `CheckLocal` and `ClearStale` apply. `recordIsMine` uses it. Only an absent record, or a valid record of another binding, is "not mine"; anything hostexec refuses is an error, and the participation is kept. | Task 7 (`TestARecordHostexecRefusesKeepsItsParticipation`, for a missing enrollment and a missing incarnation: refused and kept, still kept after a restart, then the record is repaired and a cleanup and clear succeed) |
89| 1 (P2) | A, B | Claim rev 6.30 R18-1 and moves rev 7.22 §7.6 require the same thing: a create-only prepared manifest at sequence 0, bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. Rev 14's `ToInstalled` required P's sequence to equal the pin's, and `CopyPinnedManifest` kept the source's sequence. | accepted | `CopyPinnedManifest` writes P's prepared manifest at sequence 0, keeping the pinned entries by their content-addressed objects. `PreparedAuthority` separates the manifest's identity from its provenance: `FilesSeq` must be 0, `SourceFilesSeq` must be the pin's sequence, and `InstallStamp` is bound by `ToInstalled` from `install_stamp`. Ruling 13 is corrected. Stage B: R1 has the first admission consume that manifest by its journaled digest as the publisher's baseline, and the re-plan names the tests. | Task 1 (`TestRelocationPreparedManifestIsSequenceZero`: a prepared manifest claimed at the source's sequence 7 is refused, one built from another source revision is refused, and the accepted one is bound to the stamp); Task 5 (`TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne`: source at 7, prepared at 0, first session publication at 1, inherited entries intact); stage B re-plan |
90| 2 (P2) | A, B | `Step` in `installed` with `local_install` journaled went straight to probing. Nothing checked that the install still existed after a lost lease's scoped cleanup retired the unopened P's local install. The property never reached installation. | accepted | `hostrelocate.Effects.EnsureLocal` runs before probing and before every attempt until P is opened. `host.RelocationEffects.EnsureLocal` verifies the receipt's install (`VerifyLocal`). If the install is retired, it restores it with the same `Install` under the same `install_stamp`, refuses a restore that is not the receipt's, and verifies again. It never touches an opened P. The property now also covers the target's role: half the seeds start with the local install journaled, and a new event hands the installation to this box. Its effects are stateful: `Install` makes the install, the scoped cleanup retires it unless P is opened, and `Admit` opens P. Two invariants are added: (f) P's first admission never meets a missing local install, and (g) an opened P is never reinstalled. Stage B: the production `VerifyLocal` and restore, with named tests. | Task 6 (`TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission`), Task 7 (`TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens`, the property), stage B re-plan |
91| 3 (P2) | B | 6b's `runtime.Prerequisites` returns `ClaimDependencyError` for any probe or non-serving claim (6b plan, `if p.Probe \|\| o.Claim.Op.Kind != hostclaim.KindServing`). So a source that claim §4 and Task 1 let wake in `starting` and `releasing` could not, even after R1. | accepted (contract) | R1 now also asks for the source's cold-admission dispatch in `starting` and `releasing` while its admission fence is open, refused once the fence closes or the claim no longer names the box. A target's ordinary wake stays refused. The stage B checklist names the tests. | Requests (R1), stage B re-plan |
92
93Mutation check, with the lifecycle property at 200 seeds and `-failfast`, plus targeted tests:
94
95| Guard removed | Result |
96|---|---|
97| both `EnsureLocal` calls | killed, seed 32: (f) |
98| `EnsureLocal` before an attempt while P is unopened | killed, seed 142: (f) |
99| `EnsureLocal` before entering probing | survived: the attempt-time check covers it (defence in depth) |
100| `EnsureLocal` restoring a retired install / refusing a restore under another stamp | killed: `TestEnsureLocal…` |
101| `recordIsMine` through `hostexec.ReadLocal` | killed: `TestARecordHostexecRefusesKeepsItsParticipation` |
102| `recordIsMine` comparing the binding | killed, seed 3: (b) |
103| `ToInstalled` requiring sequence 0 / the pinned source sequence / binding the stamp | killed: `TestRelocationPreparedManifestIsSequenceZero` (each) |
104| `CopyPinnedManifest` writing sequence 0 | killed: `TestCopyPinnedManifestEnforcesThePin` |
105| rev 14's lifecycle guards (`finish`'s cleanup before a clear, verified forget, lost recovery, bounded open, journal before open, handover, close before clear, `ErrInactive` superseded, verified restart forget) | all killed again (seeds 2 to 46) |
106
107The `EnsureLocal` short-circuit for an opened P also survives, because the participant never calls `EnsureLocal` once P is opened. It is kept so the effect is safe on its own.
108
109Astra's readings of rulings 1 to 24 stand. Ruling 13 is corrected as finding 1 requires.
110
111## Rev 14: round 13 dispositions
112
113Astra's round 13 (`CODEX-PLAN-REGION-HA-1BN-R13.md`) resolved both findings from round 12 and raised two P2s, both in the execution-controller lifecycle. Both are accepted; none is declined. This is the sixth round in a row to find a controller cleanup or recovery interleaving (rounds 2, 4, 5, 6, 12 and 13). So rev 14 also closes the whole class with a property test over real hostexec controllers, not only the two examples.
114
115The changes were dry-run on a clean 3407d114 with stage A applied in task order, building, vetting and running each task's tests after it, plus vet for darwin and windows and the cross-builds. The property test ran its 200 seeds niced, in about three minutes. Each lifecycle guard was then removed in turn (mutation check); the results are below.
116
117| # | Stage | Evidence checked | Disposition | What changed | Where |
118|---|---|---|---|---|---|
119| 1 (P2) | A | One failed lease put makes hostexec latch the controller lost (controller.go, `run`: `loseLocked(putErr)`), and a lost controller never becomes ready again. Rev 13's `Ready` reported the error and kept the controller, `Step` reopens only a missing one, and `Sweep` keeps a running operation's controller. A transient storage fault in `installing` therefore stalled the relocation until the agent restarted. | accepted | `RelocationControllers.live`, used by `Ready` and `Acknowledge`, recovers within the same operation. If the open controller has lost its lease (`Lost()` is closed, newly on `RelocationController`), it is joined (`Close`, bounded by the pass) and the scoped cleanup is repeated. The object is then forgotten, but its journal entry is kept. `Ready` reports `ErrNoController`, and the participant's `Acknowledge` opens a fresh controller with fresh authority and its own lease. The claim keeps the first acknowledgement, as after a restart (rev 2, finding 10). The record on disk stays `lost` until the fresh controller replaces it, so denial is durable throughout. A failed join or cleanup keeps the lost controller, and the next pass repeats both. | Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`: lost lease, cleanup failing, then recovered and reopened, no restart); the property test |
120| 2 (P2) | A | `ClearStale` returns `ErrState` for an absent or invalid claim (controller.go:619) and for an unreadable or mismatched local record, as well as for a record that names another binding. Rev 13's `finish` treated every `ErrState` as replacement and forgot the participation while this binding's record stayed. | accepted | `finish` no longer reads anything into the error. It asks the record itself (`recordIsMine`: `<DataDir>/.hostexec/<host>.json`, compared on box, enrollment, incarnation and operation, as `ClearStale` compares them). The participation is forgotten only when the record is absent or names another binding. An unreadable record, or this binding's record, keeps the participation and reports the error. `RelocationControllers.DataDir` is new; production sets it to the assembly's `Guard.DataDir`, and with it empty nothing is ever forgotten on a failed clear. | Task 7; the property test (claim read failure, absent claim, invalid local record, then restart and recovery) |
121| C1 | A | The class. | added | `TestRelocationControllerLifecycleProperties` (`internal/host/relocationlifecycle_linux_test.go`) runs over real hostexec controllers, a store that injects faults, and a clock the test drives. Each seed runs 30 random events: lease-renewal failure, a close that times out (a cleanup that blocks), cleanup failure, `ClearStale` failing for each of its causes (claim unreadable, claim absent, invalid local record, a successor that lists this box), agent restart, a successor starting before the sweep, completion, and recovery of the storage. Participant passes are the real `Participant.Step` and `Sweep`. After every step it asserts: (a) no clear without a successful scoped cleanup earlier in the same call; (b) no participation forgotten while its record exists; (c) at most one controller per host; (d) no pass past its deadline. Then (e), liveness: once faults stop, within eight passes the running relocation is ready again; after completion every record is cleared, no controller remains, and a new relocation opens. 200 seeds by default, 20 with `-short`. | Task 7 |
122| C2 | A | Found while building C1. `hostexec.Open` waits for its Cleanup hook without a context (`err = <-cleaned`), so rev 13's `openBounded`, which waited for `Open` to return, did not bound a pass whose cleanup blocks. | fixed | `openBounded` runs the open apart and returns when the pass's context ends. A controller that opens too late is closed, never kept. | Task 7; invariant (d) |
123| C3 | A | Found while building C1. After a handover, sweeping the predecessor called `ClearStale` while the successor's live controller held the host's lock, which waited out the whole pass. | fixed | With no controller object, `finish` reads the record first. If it is verifiably not this binding's (absent, or the successor's), the participation is forgotten without a clear: nothing of its own is left to clear. The handover no longer forgets superseded entries itself; the next sweep does, through that check. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController` now sweeps A after the handover) |
124| C4 | A | Found by the in-order dry run. `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` (rev 7) failed about once in five full-package runs. Main's `TestFDHoldersUnsharedThread` can run its unshare on the leader thread, which then keeps a private table for the rest of the process, so no thread shares the leader's. | fixed | The real-kernel test re-executes the test binary and runs in a fresh process. Its guards still fail it (the `KCMP_FILES` check was rechecked). | Task 4 |
125
126Mutation check: each lifecycle guard was removed in turn, and the property test was run with `-failfast`.
127
128| Guard removed | Result |
129|---|---|
130| `finish` repeating the scoped cleanup before a clear | killed, seed 1: invariant (a) |
131| `finish` forgetting only a verified other binding | killed, seed 3: invariant (b) |
132| `live` recovering a lost controller | killed, seed 13: invariant (e) (the relocation never recovers) |
133| `openBounded` | killed: invariant (d) (a pass never returns) |
134| the journal written before the open | killed, seed 3: invariant (b) |
135| the handover before a successor opens | killed, seed 53: invariant (e) |
136| `finish` closing before the clear | killed, seed 2: invariant (e) (never clean) |
137| `recordIsMine` comparing the binding | killed, seed 3: invariant (b) |
138| a handover treating hostexec's `ErrInactive` as superseded | killed, seed 7: invariant (e) |
139| the restart path verifying the record before forgetting | killed, seed 3: invariant (b) |
140| `live` repeating the scoped cleanup before the reopen | survived 200 seeds |
141
142The survivor is defence in depth, not a load-bearing guard. `hostexec.Open` runs the controller's Cleanup hook, which is the same scoped `relocationCleanup` (Task 8a), and fails if the cleanup fails. So no fresh controller, and no fresh authority, exists without a successful cleanup, and invariant (a) is about clears, which recovery never makes. The explicit cleanup is kept so that a failure surfaces at the pass that notices the loss.
143
144Astra's readings of rulings 1 to 24 stand. Rev 13's statement that the predecessor's entry is removed "once the successor's open succeeds" is replaced by C3.
145
146## Rev 13: round 12 dispositions
147
148Astra's round 12 (`CODEX-PLAN-REGION-HA-1BN-R12.md`) resolved the file-origin finding and raised two P2s. Both are accepted; none is declined. Finding 1 is stage A code. It was dry-run on a clean 3407d114 with stage A applied in task order, building and running each task's tests after it, plus vet for darwin and windows and the cross-builds. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). Finding 2 is a stage B contract and has no code.
149
150| # | Stage | Evidence checked | Disposition | What changed | Where |
151|---|---|---|---|---|---|
152| 1 (P2) | A | Rev 12's `Acknowledge` looked controllers up by host and operation, and opened a missing one under `e.Lifetime`. `hostexec.Open` loops on `ErrControllerBusy` until its context ends (controller.go, `Open`), and a predecessor's controller keeps the host's lifetime lock until `Close`. `ClearStale` refuses with `ErrInactive` whenever the running operation lists this box, whatever its id (controller.go:643). The regression reproduces it: without the handover, B's acknowledgement waits out the whole pass. | accepted | Before opening a successor's controller, `Acknowledge` calls `RelocationControllers.handover`. It retires every other participation of the same host under the pass's context: close, then the repeated scoped cleanup, then the clear. A clear refused with `ErrInactive` because the successor lists this box counts as superseded. The lock is free by then, and the successor's `Open` replaces the record under it; the predecessor's journal entry is removed only once that open succeeds. A clear that finds another binding's record (`ErrState`) means this participation has no record left. The open itself is bounded by `openBounded`: the controller still lives under the agent's lifetime, but waits for the lock no longer than the pass. A controller that opens in the same instant the pass ends is closed, never kept with a cancelled life. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController`, on real hostexec controllers: A acknowledged; A completes and B starts before A's sweep. B's acknowledgement succeeds, only B is retained, and B's record blocks updates. A lock held by a controller this box does not retain is waited for no longer than the pass, and nothing is kept.) |
153| 2 (P2) | B | 6b rev 5: `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable`. `runtime.Prerequisites` calls `Roles.Preflight` before dispatch. The "External completion gates" section requires "the reservation-bearing extension of main's isolated root launch protocol … before any tenant shape can run with claims enabled", which "is not supplied by the external owner adapters". | accepted (contract) | Request **R10** asks the claim session and the activator's owner for that protocol, with a relocation probe's `"probe"` work admitted through it. Task 8c is now gated on R10. Task 10 gains an activation gate: the passes stay dormant and refuse while the launcher lacks the protocol. The stage B re-plan names the production probe tests: success through the protocol, refusal without it before any attempt is counted, and the pass staying off. | Requests (R10), Task 8c, Task 10, stage B re-plan |
154
155Astra's readings of rulings 1 to 24 stand.
156
157## Rev 12: round 11 dispositions
158
159Astra's round 11 (`CODEX-PLAN-REGION-HA-1BN-R11.md`) resolved the chooser-test finding and raised one P2. It is accepted; none is declined.
160
161The coordinator also asked for a check of every stage A use of `holder.admissions` as authority. That check found one more case, S1. Both changes were dry-run on a clean 3407d114 with stage A applied in task order (1, 2, 3, 4, 5, 6, 7, 9a), building and running each task's tests after it. Vet for darwin and windows and the cross-builds are clean. Four new guards were each removed in turn, and each removal failed its test (listed in the Self-review).
162
163| # | Stage | Evidence checked | Disposition | What changed | Where |
164|---|---|---|---|---|---|
165| 1 (P2) | A | `AdmitWriter` drops a covered, loss-archived head once more than eight admissions are kept (generation.go:154). Rev 11's `filesPin` built its origins only from `holder.generation` and the retained admissions. A file published at g1 and never changed is still `files/g1/objects/...` at g10, because `CopyManifest` carries the entry and `Stage` writes nothing for unchanged content. The pin accepted that manifest, but `CopyPinnedManifest` refused it with `ErrOrigin`. Claim §6.2a ("Files in the live set (L13)") makes the live set every object the live manifests reference, computed independently of the database baseline. | accepted | `filesPin` adds `referencedOrigins` to the retained chain: the files prefix of every generation whose objects the pinned manifest's entries and platform items reference. Each referenced object must lie under this holder's location, which binds it to the incarnation, in this region's data bucket (the region binding), at `files/<generation>/objects/`. Any other reference refuses the pin (`ErrRefused`, `hostfiles.ErrOrigin`), so the pin never admits what the copy would refuse. `CopyPinnedManifest`'s check is unchanged. | Task 5 (`TestPinKeepsAFileOriginTheAdmissionListTrimmed`: ten real `AdmitWriter` and `PublishGeneration` cycles with real `CopyManifest` and publication. The admission list is trimmed to eight and no longer names g1, yet g10's manifest still references g1's object for the unchanged file. The pin and the copy succeed, and a reference to another incarnation refuses.) |
166| S1 | A | From the check. `preparedOpened`, the boundary that stops P being opened twice or its local install remade (B1-1, B2-5), scanned only the retained admissions. The same goes for `coverageFor`'s "P not opened" test. A covered P admission that is trimmed would let a later probe open P again. | fixed | The admission that opens P journals `prepared_opened` in the same CAS (`relocationAdmission`). `preparedOpened`, now exported as `PreparedOpened` for the participant, reads that key before the admission list. | Task 1 (`TestRelocationPOpenedOutlivesTheAdmissionTrim`, with P's admission removed as the trim would remove it: P's reopening is refused), Task 6 (`coverageFor`) |
167
168The check found no other authority taken from the trimmed list. `relocationAdmission` and `coverageFor` compare against the last admission, which the trim never removes: it drops only the head, and only past eight entries. `retainAutomaticLoss` records evidence, not authority. An admission the trim dropped was covered and loss-archived (`admissionLossArchived`), so nothing it could vouch for is lost; it falls back to the holder's live writer session.
169
170Astra's readings of rulings 1 to 24 stand.
171
172## Rev 11: round 10 dispositions
173
174Astra's round 10 (`CODEX-PLAN-REGION-HA-1BN-R10.md`) resolved both findings from round 9 and raised one P2: a test that would still pass with its production wiring disconnected. It is accepted; none is declined.
175
176I then checked every stage A test for the same weakness. For each test of production wiring, the question was whether its assertions still hold with the wiring replaced by a stub. Two more tests failed that check. Both are fixed here, and the table lists them as S1 and S2.
177
178The changes were dry-run on a clean 3407d114, applying stage A in task order: build, vet for darwin and windows, cross-builds, and the targeted `hostops`, `console` and `cmd/console` tests. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review).
179
180| # | Stage | Evidence checked | Disposition | What changed | Where |
181|---|---|---|---|---|---|
182| 1 (P2) | A | `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims` read an empty bucket. Both readers returning nothing satisfied every assertion. | accepted | The bucket now holds evidence for two boxes in `ie`: their edges files (box-b advertising `region-ha-1`) and fresh status snapshots. It also holds three serving claims, two on box-a for bab/jam and one on box-b for bab/tea. The test reads through the constructed chooser's `Edges` and `Loads`. It asserts each box's identity, freshness, placement and capabilities, and each box's host and per-app counts. Then the evidence changes: box-a starts draining and goes stale, and a fourth host serves on box-b. The next read must show both changes. | Task 2 (`TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) |
183| S1 | A | Found in the scan. `TestProductionAuthoritySelectsOnlyWithAChooser` used a one-box fixture, so a `SelectExecutor` that ignored the chooser and returned a fixed box-a still passed. | fixed | Two boxes, with box-a draining. The selection must be box-b with its roster enrollment. With both draining it must be `ErrNoEligibleBox` and no selection. | Task 2 |
184| S2 | A | Found in the scan. `regionHAEnabled` sets `Config.RegionHA` from `CARLOS_REGION_HA`, and no test covered it. | fixed | `cmd/console/regionha_test.go`: on only for `strconv.ParseBool` true values, with whitespace trimmed. Unset, `0`, `false`, `no` and garbage are off. | Task 2 |
185
186The scan found nothing else in stage A. These tests already run against real wiring:
187- hostclaim's transitions, on a real store.
188- `ClaimLoads`, on real claims.
189- `Guard.AdmissionFenced` and Task 9a's fence observer, against a real `BeginFence`.
190- `RelocationPolicy`, through `Policy.Check`.
191- The writers-stopped scan, against the real kernel and a fixture `/proc`.
192- The worker, over a signed roster, `hostactors.Directory` and `hostfence.ClaimFenceAuthority`.
193- `StopReceiptReleases`, against stored receipts.
194- `ReleaseFlushFiles`, against a real store and registry.
195- `ReleasePositionsFile` and `RelocationControllers`, on disk.
196- `OpenRelocationController`, with a real hostexec controller.
197
198The remaining stubs are providers that Task 10 (stage B) supplies: the worker's `Choose`, `Eligible`, `Leases` and `Prepared`; `StorePinner.SelectDB`, whose production value is `genselect.Select` over the versioned bucket; and `RelocationEffects`' providers. The stage B re-plan has a row that requires each one to be tested with its production value.
199
200Astra's readings of rulings 1 to 24 stand.
201
202## Rev 10: round 9 dispositions
203
204Astra's round 9 (`CODEX-PLAN-REGION-HA-1BN-R9.md`) resolved both round 8 findings and raised two new ones. Both are accepted; none is declined.
205
206This revision also adds a dry run that applies stage A in task order. The earlier dry runs applied every task at once, which is how finding 1 got through. This time a clean 3407d114 took Tasks 1, 2, 3, 4, 5, 6, 7 and 9a one at a time. After each task: `go build ./...`, vet of that task's packages, and that task's tests. With finding 1 fixed, every step built and passed and turned up no other ordering fault. The tree as a whole then passed vet for darwin and windows and the cross-builds.
207
208| # | Stage | Evidence checked | Disposition | What changed | Where |
209|---|---|---|---|---|---|
210| 1 (P1) | A | Applied in order, Task 5 failed to build: `internal/hostrelocate/pin.go:259:23: undefined: hostfiles.ValidateManifest`. The helper was first defined in Task 7's `internal/hostfiles/platform.go`. | accepted | `ValidateManifest` now lives in its own file, `internal/hostfiles/manifestcheck.go`, which Task 5 creates (Files list, Step 4, the Step 5 run and the Step 6 `git add`). Task 7's `platform.go` keeps only `EnvItem` and `StagePlatformDelete` and uses the helper unchanged. | Task 5, Task 7; the in-order dry run |
211| 2 (P2) | A | Selection and completion in rev 9's test both named box-a, and the intent check called `BeginCreateOn` itself. The test still passed with `Attempt`'s selection removed. | accepted | The fake authority now has a selected box, the box completion offers, and a call counter. With selection on box-b and completion offering box-a, `Attempt` must refuse with `ErrWrongState`. The persisted claim must still be `creating`, with `selected_executor` box-b and `executor_enrollment` enrollment-b. `Complete` with box-b then serves on box-b, and selection must have run exactly once. Removing `Attempt`'s use of the selection (`if selected` made false) fails the test: the completion substitutes box-a. | Task 2 (`TestCreateIsTodaysWithoutASelection`) |
212
213Astra's readings of rulings 1 to 24 stand.
214
215## Rev 9: round 8 dispositions
216
217Astra's round 8 (`CODEX-PLAN-REGION-HA-1BN-R8.md`) resolved the release-retry finding and added two P2s. I checked both against `internal/host/claimstop.go` and `internal/hostguard/fence.go` and accepted both; none is declined. Both fixes are stage A code. They were dry-run on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `hostclaim`, `hostrelocate` and `host` tests. Two new guards were each removed in turn and their tests failed (listed in the Self-review). Ruling 23 is amended.
218
219| # | Stage | Evidence checked | Disposition | What changed | Where |
220|---|---|---|---|---|---|
221| 1 (P2) | A | Rev 8's `FenceExhaustedTarget` always set `op.exec` to "none" and the phase to `pinning`. In `starting` and `releasing`, `op.exec` is the live source, and the verified fence is the target's. Root's stop refuses a requester that is not `op.exec` (claimstop.go:184, `observed.Claim.Op.Exec != req.Box`). | accepted | The fix depends on who holds `op.exec`. If another box holds it (the live source in `starting` or `releasing`), the CAS consumes the target's fence, retains its loss, puts it on `lost`, drops it from the participants and records `stuck`. It leaves `op.exec`, the phase, the stamp and every release record unchanged. The old move to a stuck `pinning` with `op.exec` "none" now happens only when `op.exec` is the dead target itself (its own fence revokes it) or is already "none". | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning/dead_while_the_source_executes_in_starting` and `…_in_releasing`, the latter with the source's release journaled first); ruling 23 |
222| 2 (P2) | A | Rev 8's `Release` restored `recorded.ExpectedDirectoryRevision` only when the boot or controller had changed. Root's stop compares that revision with its stop inventory (claimstop.go:191). The fence target also carries it (`hostguard.FenceTarget.DirectoryRevision`, fence.go). | accepted | When matching positions are recorded, `Release` always asks under the recorded `ExpectedDirectoryRevision`, recovery or not. Only the recovery branch takes the recorded boot and controller. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the retry runs under the same boot and controller, with the roster at revision 5. Root is asked at revision 4, and only the producers' drain and the receipt run.) |
223
224Astra's readings of rulings 1 to 22 and 24 stand. Ruling 23 is amended as finding 1 requires.
225
226## Rev 8: round 7 dispositions
227
228Astra's round 7 (`CODEX-PLAN-REGION-HA-1BN-R7.md`) resolved the kcmp finding, confirmed that stage B's contracts are complete, and added one P2. We checked it against `internal/host/claimstop.go` and accepted it; none is declined. The fix is stage A code. It was dry-run again on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `internal/host` tests. Two new guards were removed in turn and each made the new test fail (listed in the Self-review).
229
230| # | Stage | Evidence checked | Disposition | What changed | Where |
231|---|---|---|---|---|---|
232| 1 (P2) | A | In `ClaimStopCoordinator`, `DrainClaim` runs before `Kernel.VerifyStopped` and `Guard.CompleteFence` (claimstop.go:219 onward). If either fails, root has no durable receipt. On retry, `BeginFence` returns the same fence, `FenceReceiptFor` returns `ErrNoFenceReceipt`, and root asks for the drain again. Rev 7's `Release` read the recorded positions but still built a fresh `releaseDrain`, which repeated `StopWriters`, the observation and both flushes against a replicator its own producer drain had already closed. | accepted | When this controller has already recorded positions for the exact stop (same boot and controller, root without a receipt), `Release` marks its drain `resumed` and carries the recorded TXID and sequence. A resumed drain goes straight to the producers' own drain, which is idempotent, and leaves verification and the receipt to root. A restarted agent keeps the recovery path; a replayed receipt keeps pairing with the record. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the drain closes the producers, root fails before persisting its receipt, and the retry without a restart runs only `producers` and `receipt`, with no flush, record or writers stop, and returns 41 and 8) |
233
234Astra's readings of rulings 1 to 24 stand.
235
236## Rev 7: round 6 dispositions
237
238Astra's round 6 (`CODEX-PLAN-REGION-HA-1BN-R6.md`, against `222ff1c6`) resolved all six of round 5's findings and added one. It was checked against the plan and the kernel before anything changed, and is accepted; none is declined. The fix is stage A code, dry-run again on a clean 3407d114 (build; vet for darwin, windows and linux/arm64; cross-builds; the stopevidence tests run five times against the real kernel), with 3 new guards removed in turn (listed in the Self-review). Stage B is unchanged: R3's root endpoint calls the same `WritersStopped`.
239
240| # | Stage | Evidence checked | Disposition | What changed | Where |
241|---|---|---|---|---|---|
242| 1 (P1) | A | `scanOne` exempted a thread's descriptor when the leader showed the same number and path. Confirmed on this kernel: a thread that runs `unshare(CLONE_FILES)` keeps a copied table with the database at the same number, and it can `dup3` an independent open onto that number; `/proc/<pid>/task/<tid>/fd/<n>` reads the same path either way. kcmp(2) is the positive check: `KCMP_FILES` compares two tasks' descriptor tables and `KCMP_FILE` compares two descriptors' open file descriptions. golang.org/x/sys v0.46.0 has `unix.SYS_KCMP` on all 15 linux architectures but no `unix.Kcmp` and no constants, so the plan makes a raw `Syscall6` with `KCMP_FILE = 0` and `KCMP_FILES = 2` from linux/kcmp.h. kcmp needs ptrace read access to both tasks. The agent always has it within its own thread group, and root's complete scan has it through `CAP_SYS_PTRACE`. This box runs `CONFIG_KCMP=y` and Yama scope 0. | accepted | A thread's descriptor is the leader's only if three things hold: the leader shows the same number and path, `KCMP_FILES(pid, tid)` is 0 (the thread shares the leader's table), and `KCMP_FILE(pid, tid, n, n)` is 0 (the same open file description). Otherwise it is reported with FD -1, which no allow-list exempts. A copied table counts as foreign even when the description is shared, because the thread could still use that description after the replicator closes. If a task exits mid-scan (ESRCH), it is skipped as before. EBADF is treated as "not the leader's". Any other kcmp error, including ENOSYS and EPERM, is `ErrIncomplete`. kcmp takes pids in the caller's pid namespace, so it is used only for the caller's own `/proc`. Any other root fails closed unless a test replaces `kcmpFor`. The `//go:build linux` file and the `!linux` stub stay as they were. | Task 4 (`TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` on the real kernel: a shared locked thread adds nothing; an unshared thread with the copied descriptor, and one that reopened the database and `dup3`'d it onto the leader's number, are each reported FD -1. `TestWritersStoppedRefusesAnyOtherHolder`'s new fixture cases: an unshared thread at the leader's number and path, a shared table whose descriptor is another description, and kcmp unavailable (ENOSYS, EPERM) as `ErrIncomplete` for both scans) |
243
244Astra's readings of rulings 1 to 24 stand.
245
246## Rev 6: round 5 dispositions
247
248Astra's round 5 (`CODEX-PLAN-REGION-HA-1BN-R5.md`, against `996b24c3`) left R4-5 partly resolved and added five findings. Each was checked against the tree at 3407d114 and the plan before anything changed; all six are accepted and none is declined. Every one lands in stage A code, dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds for darwin, windows and linux/arm64, targeted tests), with 11 new guards removed in turn (listed in the Self-review); R4-5's restart obligations are carried into the stage B re-plan as well.
249
250| # | Stage | Evidence checked | Disposition | What changed | Where |
251|---|---|---|---|---|---|
252| R4-5 | A, B | `hostexec.Controller.Close` (controller.go:480) sets `closed` before returning `errors.Join(err, c.CleanupError())` and returns nil on every later call; rev 5's `Finish` cleared after any nil `Close`, and the restart path cleared from the journaled binding with no cleanup at all. Rev 5 also journaled only inside `Finish`, so a record left by a crash (or a failed `Open`, which writes `lost` once it holds the live lock) before the first `Finish` was never found. | accepted | Repeat, do not trust: `RelocationControllers.Cleanup(ctx, binding)` is the scoped cleanup (Task 8a's `relocationCleanup`, authenticated to the binding's box and enrollment) and `Finish` runs it, and needs it to succeed, in the same call as every clear, after `Close` and on the restart path alike; a failure keeps the record, the entry and the journal. The participation is journaled before the controller opens (`Acknowledge` refuses without a journal), so a restart finds every record it may have left; the journal write syncs its directory. | Task 7 (`TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`: cleanup failure, a sweep after it, a restart while it still fails, then success; `TestARestartFindsAParticipationThatNeverReachedFinish`; `TestFinishClearsTheDurableRecordSoUpdatesAreEligible` with a real hostexec controller whose cleanup fails); stage B re-plan |
253| 1 | A | `hostops` `eligible` excluded only `Lost`; `Worker.next` excluded the source, `Lost` and replaced targets; `ReplaceTarget` reaches `install` directly in `installing` and `installed`, never through `ToInstalling`'s `park-local` check. | accepted | `hostclaim.OwesLocal` and `LocalChoreBoxes`. Selection excludes every box owing a `park-local` (the chooser's eligible set, so create, `Permits` and the worker's spare all agree; `Worker.next` too), and `replaceTargetSlot` refuses one with `ErrPendingChore` inside the CAS, for `ReplaceTarget` at every phase and `NameTarget`. | Task 1 (`TestRelocationReplacementRefusesABoxOwingALocalRetirement`, in `starting`, `installing` and `installed`), Task 2 (`TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/owing_a_local_retirement`), Task 5 (`TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement`) |
254| 2 | A | `fenceBox` journaled the fence, added `lost`, discharged `park-local` and retained the loss, and touched no regional obligation; `lost.go:302` reassigns `box:<b>`'s `park-generation` and `park-replica` to `region:<r>` for a declared loss. | accepted | `fenceBox` passes every `park-generation` and `park-replica` owned by `box:<b>` to `region:<obligation's region>` in the same CAS, changing nothing but `owner` (an obligation with no region refuses, as in `DeclareLost`); the retained `AutomaticLoss` names them (`reassigned`, the D4 record's `reassign_regional`). Every automatic fence CAS shares it: start, `FenceSource`, `ReplaceTarget`, `FenceExhaustedTarget`. | Task 1 (`TestRelocationFenceReassignsTheDeadBoxsRegionalWork`, unqualified and qualified discharge); R5 |
255| 3 | A | Task 9a's `check` replaced `Serving` with `RelocationRow` unconditionally; with both callbacks nil a `moving` executor participant is `RowHeld` and passed, and `Install` went on to `Activate` and `RecordInventory` before `WorkSource.SelectAt` (work.go:94) refused. | accepted | `Reconciler.relocationAware()` is `FenceClosed != nil \|\| HoldRow != nil` (Task 10 sets both only with `CARLOS_REGION_HA`). Off, `check` is today's `Serving` and `Install` skips the row decision and the hold-only path entirely. | Task 9a (`TestReconcileWithoutAFenceObserverIsTodays`: a relocating source in `starting` and `releasing` and a `moving` executor, dry run and real, refused with the claim's ETag and every file under the guard's data directory unchanged) |
256| 4 | A | `journaled` decoded every directory entry, so `writeFileSynced`'s truncated `.json.tmp` failed the whole listing, and `Retained` dropped the error. | accepted | Only names ending `.json` are committed records; a `.tmp` is ignored (its participation opened nothing: the record precedes the open). A committed record that cannot be read is reported, joined into the error, beside every readable one. `Effects.Retained` returns `([]Participation, error)`; `Sweep` reports the error and still finishes every readable participation. | Task 7 (`TestARestartFindsAParticipationThatNeverReachedFinish`: one valid journal, one truncated `.tmp`, then one damaged committed entry), Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`) |
257| 5 | A, B | `Flush` loads the existing manifest through `hostfiles.Open` and did nothing on `ENOENT`, so `Platform[EnvItem]` kept the old bytes. | accepted | `(*hostfiles.Publisher).StagePlatformDelete(name)` (in `platform.go`) drops the item at a new sequence; `Flush` stages it on `ENOENT`, so `ReadPlatform` on the final revision returns `os.ErrNotExist`. Stage B's local install treats that as "no env file" (re-plan). | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`: published, removed, flushed again, then a no-op repeat) |
258
259Astra's readings of rulings 1 to 24 stand; ruling 20's cleanup is completed by R4-5 above.
260
261## Rev 5: round 4 dispositions
262
263Astra's round 4 (`CODEX-PLAN-REGION-HA-1BN-R4.md`, against `418daf58`) left three earlier findings partly resolved and added six. All nine are accepted; none is declined. Under ruling 24, stage A findings are fixed in code (dry-run on a clean 3407d114, 7 new guards removed in turn, listed in the Self-review); stage B findings are fixed at the contract level: what must happen, which request covers it, which test proves it.
264
265| # | Stage | Disposition | What changed | Where |
266|---|---|---|---|---|
267| R1-9 | B | accepted | `Region` is 6b's field: Task 8a's `Collect` provider sets it on every request from the verified actor (the box's region never changes), initially and on recovery alike, beside `Recover` for the recorded owner's stop. | Task 8a |
268| N1 | B | accepted | The fixture's `withExistingRow` takes the hold everywhere it is called; the census literal carries 6b's actual fields (`Inventory`, `Registry`, `Launches`, `Box`, `Enrollment`, `DirectoryRevision`, `Roots`, `SharedPaths`), no `Guard`. | Task 9 |
269| N3 | B | accepted | R8's insertion point is after actor verification: 6b's `Build` calls an `Extend(a *ClaimAssembly, actor hostbox.Actor)` hook before it captures `Policy.Check`; `applyRegionHA` is that hook's body and binds the policy to `actor.Box`. | Task 10, R8 |
270| 1 | A | accepted | `Guard.AdmissionFenced(host, claim, captured)` takes no lock (one atomic record read; the claim is the caller's fresh read); `Reconciler.FenceClosed` takes the context and the observation and returns an error, which keeps rows and mints none; `holdOnly` re-decides on a fresh read under `WithLocal`. Tested with the real observer, a real `BeginFence`, and a bounded context on both the activation and the hold-only path. | Task 3, Task 9a (`TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver`) |
271| 2 | A, B | accepted | Policy half (A): a relocation target's probe is its own selected action, `probe`, which `Policy.Check` admits for a relocating claim and `RelocationPolicy` sends to `WakePhase(probe=true)`; an ordinary `spawn` by the target stays refused. Dispatch half (B): R1 extended, 6b's runtime selects `"probe"` work for a relocation probe. | Task 3 (`TestRelocationPolicyGivesEachActionItsRole`, `TestPolicyLetsAHeldParticipantReconcileItsRow`), R1 |
272| 3 | B | accepted | The restore destination (the host's files directory and env path) is created and inventoried under the installation's authority before `Restore`, through 6b's typed inventory (a `files` entry at `install_stamp`), never an untracked `MkdirAll`; tested with a fresh spare that has no host directory. | Task 8c |
273| 4 | B | accepted | Held release staging needs root: request R9 asks moves for a relocation-aware relay authorizer (bound to the operation, executor, install stamp and the pinned release); the install verifies staging before it records its receipt; tested under the edge unit's filesystem restrictions. | Task 8c, R9 |
274| 5 | A | accepted | `RelocationController.Clear` (hostexec's `ClearStale` with the controller's config); `Finish` journals the participation before closing, keeps the entry and the record on a failed close or clear, clears the exact record after a successful cleanup, and a restart retries through the journaled binding. A real hostexec controller on disk shows root's §5.10 check (`hostexecParticipation`) refusing before and admitting after. | Task 7 (`TestFinishClearsTheDurableRecordSoUpdatesAreEligible`, `TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`) |
275| 6 | A | accepted | The obsolete-hold clearance in `Policy.Check` needs `Relocation` set (region HA on); off, today's refusal. Tested both ways on the same held serving row. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`) |
276
277## Rev 4: round 3 dispositions
278
279Astra's round 3 (`CODEX-PLAN-REGION-HA-1BN-R3.md`, against `d93c94d0`) left three earlier findings partly resolved, one unresolved, and added six. All ten are accepted; none is declined. The un-gated changes were dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (8 removals, each failing its test; listed in the Self-review).
280
281| # | Disposition | What changed | Where |
282|---|---|---|---|
283| R1-5 | accepted | `PreparedFromInstallManifest.Verify` compares P's entries against the pinner's retained copy (`pin.Files.Retained`), never the source manifest; a late source PUT changes nothing it reads. | Task 8b |
284| R1-9 | accepted | After a restart `Release` names the recorded owner's boot, controller and directory revision and offers `RecoveryReleaseDrain`; Task 8a's `Collect` provider sets 6b's `Recover` on that drain, so root's authenticated recovery path accepts the tuple and replays the receipt. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`), Task 8a |
285| R2-N4 | accepted | `Policy.Check`'s serving branch lets exactly the serving executor's `reconcile` through an obsolete hold (one naming no operation the claim runs), so the adapter's clear is reachable; every other write under a hold still waits. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`), Task 9 |
286| R2-N5 | accepted | See N2. | Task 7 |
287| N1 | accepted | Task 9's adapter insertion re-reads the claim under the captured authority (`observed` is defined there, validated against the captured incarnation and stamp); the test fixture (`adapterFixture` and its methods) is this task's code, over 6b's adapter and registry. | Task 9 |
288| N2 | accepted | `Finish` keeps the map entry until `Close` succeeds and forgets only that same controller, so a close the pass's context cut short is finished by a later sweep. | Task 7 (`…RetainsTheControllerUnderTheAgentLifetime`: a cancelled close, then a successful one) |
289| N3 | accepted | `hostbox.RelocationPolicy(box)` (un-gated, tested) and `hostguard.Guard.AdmissionFenced` (un-gated, tested) are the two providers; Task 10 wires them into the assembly between 6b's `OpenProductionClaimAssembly` and `Build` (which captures `Policy.Check`), gated on `CARLOS_REGION_HA`; with it off neither is set and behaviour is 6b's. The insertion point is request R8. | Task 3, Task 10, R8 |
290| N4 | accepted | `Reconciler.Install` takes the hold-only path for a held row whenever the box is not the executor or its admission fence is closed, so a fenced source still named `op.exec` in `releasing` writes its hold under `WithLocal`, never through the guarded path `With` refuses. | Task 9 |
291| N5 | accepted | `FenceExhaustedTarget`: a target terminated after the replacements are used is fenced into a stuck `pinning` in one CAS (loss retained, box on `lost`, `op.exec` "none", receipt dropped, P's remote build kept when unopened, nested when opened); the worker uses it instead of `MarkStuck`, which stays the live target's rule. | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning`), Task 5 (`TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall`); ruling 23 |
292| N6 | accepted | `PinAfterRelease` from `releasing` requires the acknowledgement barrier, so a target replaced after the source's release acknowledges before the operation leaves the phase. | Task 1 (`TestRelocationReplacementInReleasingAcknowledgesBeforePinning`) |
293
294## Rev 3: round 2 dispositions
295
296Astra's round 2 (`CODEX-PLAN-REGION-HA-1BN-R2.md`, against `ed992c46`) left seven round 1 findings partly resolved and added seven. All fourteen are accepted; none is declined. Every un-gated change was dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (11 removals, each failing its test; listed in the Self-review).
297
298| # | Disposition | What changed | Where |
299|---|---|---|---|
300| R1-2 | accepted | The build's authority is checked at every write, not once: `authorityStore` wraps the versioned store so each `PutIfAbsent` and `CopyVersion` re-reads the claim and the host-guard record (`hostguard.Capture` at `install_stamp`) first; the lease's loss cancels the rest. | Task 8b |
301| R1-4 | accepted | `AutomaticLoss` retains the holder's incarnation and stamp, the box's last writer session whether or not it is still the writer (its last admission record on the holder, `LastAdmission`), so a source fenced after the target's admission keeps its session and consequences. | Task 1 (`TestRelocationLossRetainsAFormerWritersSession`); R5 |
302| R1-5 | accepted | The pinner retains the exact manifest bytes in the control bucket (`RetainedManifestKey`, create-only) before the pin; `CopyPinnedManifest` writes P from that copy, validated by digest, sequence and origins (`hostfiles.ValidateManifest`), and never reads the source again. A late source PUT changes nothing. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`), ruling 21 |
303| R1-9 | accepted | `ReleasePositions.Recorded` matches the installation and session, not the controller, and returns the request the flush was made under; after a restart `Release` asks root under that identity (so root finds the stop's durable receipt), offers no drain of its own, and pairs the replayed receipt with the recorded positions. A stop root never completed cannot be finished by a new process: the release deadline makes it stuck. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`, `TestReleasePositionsFileBindsTheExactStop`) |
304| R1-13 | accepted | A later probe attempt whose predecessor generation holds no LTX file (`genselect.ErrEmpty`) is an ordinary empty cold admission: `Empty` set, no baseline, 6b's own empty authority. | Task 8c |
305| R1-14 | accepted | See new finding 2. | Task 9 |
306| R1-16 | accepted | Exhaustion is checked before a replacement is looked for, in both paths: with the replacements used the operation is stuck now, never retried on "no eligible box". | Task 5 (`TestWorkerMarksExhaustionStuckBeforeLookingForASpare`) |
307| N1 | accepted | `Prepared` is set only by the two branches that open P; the retry branch resets `Prepared` and `Empty` and selects its baseline as an ordinary cold wake, so its marker follows the first durable root. | Task 8c |
308| N2 | accepted | The adapter test separates "no row to mint" (a box with no role and no row: `ErrChanged`, nothing written) from "an existing unexplained row to keep" (held, never served, never deleted), and asserts `hold_op`, not a status the implementation does not write. | Task 9 |
309| N3 | accepted | `ProcessDescriptors` reads only `/proc/<pid>` (its own process; another process's unreadable table does not matter); the complete scan is root's: `RelocationEffects.Observe` is a provider, production asks root over the instance-control socket (`CollectWritersStopped`, R3's server half), and an agent-side scan fails closed on the first unreadable table. | Task 4 (`TestProcessDescriptorsNamesOneProcessesHolders`), Task 7, Task 8a, R3 |
310| N4 | accepted | A `RowReady` row whose `hold_op` names an operation the claim no longer runs has its hold cleared under the activation's verified ownership (`SetClaimHold(owner, old, "")`); a hold of the running operation is never touched. | Task 9 (`TestReconcileAdapterClearsTheHoldAfterCompletion`), ruling 20 |
311| N5 | accepted | `Effects.Retained` and `Finish`; `Participant.Sweep` finishes every retained controller whose operation the claim no longer runs with this box as a participant, once per pass; the relocation controller's `Cleanup` hook is scoped to the relocation's own partial work and is a no-op once this box has admitted under the operation or serves the host, so finishing never stops the serving writer. | Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`), Task 7 (`Finish` in `TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`), Task 8a, Task 10; ruling 20 |
312| N6 | accepted | `MaxProbeAttempts = 3`; `BeginProbeAttempt` counts the attempt in the claim before it is made (per box and operation, so a replacement starts at zero); the participant enters `probe-failed` on the fourth. Durable across restarts and lost responses. | Task 1 (`TestRelocationProbeAttemptsAreBounded`), Task 6 (`TestAParticipantGivesUpAfterThreeProbeAttempts`); ruling 19 |
313| N7 | accepted | A qualified discharge completes each `park-local` with `CompleteObligation`'s own semantics, so a discovery operation's final chore marks discovery complete exactly as a retirement would. | Task 1 (`TestRelocationDischargeCompletesDiscoveryLikeARetirement`) |
314
315Astra's readings of rulings 1 to 4, 6 to 8 and 10 to 17 stand; rulings 5, 9 and 18 are completed by R1-5, N6 and R1-4 above.
316
317## Rev 2: round 1 dispositions
318
319Astra's round 1 (`CODEX-PLAN-REGION-HA-1BN-R1.md`, against `eb5e9051`) returned sixteen findings, "not ready". Each was checked against the tree at 3407d114 and the cited plans before anything changed. Fifteen are accepted in full and one in part (8: the un-gated half is built, the production body is a request to the claim session). None is declined. Every un-gated change below was dry-run again: applied to a clean 3407d114, built, vetted for darwin and windows, cross-built, tested, and each new guard removed in turn (25 removals, each failing its test; listed in the Self-review).
320
321| # | Disposition | What changed | Where |
322|---|---|---|---|
323| 1 | accepted | Task 8 is three tasks against the declared types: `moves.SealedObject{Key, VersionID, Size}`, `moves.InstallManifest` (on main), `moves.BuildInputs` with `Now` from this plan's config, 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`. What no plan supplies is now this plan's code (`RelocationWakeInputs`, the install journal, `PreparedFromInstallManifest` reading `moves.InstallManifestKey`) or a named request (R1 the probe dispatch 6b refuses today, R2, R3). | Tasks 8a, 8b, 8c; Requests |
324| 2 | accepted | The worker holds the replica lease on `holder.location` from pin selection to the journal CAS (`Worker.Leases`, `pinUnderLease`); a lease lost before the CAS refuses. The build runs under the same lease, re-reads the claim before each durable step and is cancelled when the lease is lost. | Task 5 (`TestWorkerDrivesADrain`'s order, `TestWorkerRefusesToJournalAPinAfterTheLeaseIsLost`); Task 8b (`TestRelocationBuildStopsWhenAuthorityChanges`) |
325| 3 | accepted | `ReleaseFlushFiles`: `Open`, `StageWake(ExportWake)`, `StagePlatform(EnvItem)`, `SyncDir`, `Commit` as one revision; `hostfiles.EnvItem` added. | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`) |
326| 4 | accepted | D4 retention outside the operation: `Claim.EvidenceRefs []AutomaticLoss` (fence reference, observation time, box, role, last writer session, discharge verdict, timers `unknown`, reporting pending) and the loss id in `LossRefs`, written by `fenceBox` in the consuming CAS with a deterministic id; both survive `CompleteRelocation`, `free` and holder replacement. `Claim.Validate` checks them. | Task 1 (the failover test's completion checks, `TestRelocationRetainsEveryFencedBoxAcrossCompletion`); ruling 18 |
327| 5 | accepted | `FilesPin.Origins` (the admitted chain as pinned); `hostrelocate.CopyPinnedManifest` verifies the source bytes against the pin's digest and sequence, copies with the pinned origins, and refuses a copy whose entries differ from the pinned ones; the build and the verifier use it. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`: a late revision, different bytes at the same sequence, inherited objects); Tasks 8b |
328| 6 | accepted | `PinObject.N` is gone. `ClaimGen` names the observation; the build takes `pin.DB.MaxTXID`; the empty branch is `DB == nil`. | Task 5, Task 8b |
329| 7 | accepted | `ToInstalled` requires `FilesSeq == pin.FilesSeq` and a `FilesManifestSHA256` always (an empty pin included); the build writes P's files manifest before the database; the verifier reads both back. | Task 1 (`TestRelocationDrainEndToEnd`, `TestRelocationEmptyPinNeedsItsEvidence`); Task 8b (`TestPreparedRefusesAPartialBuild`) |
330| 8 | partly | `RelocationEffects.StopWriters` runs first in the drain, before the observation; its production body is request R2, because 6b has no writers-only stop and its `ClaimProducers.DrainClaim` cancels every producer. The live-process test is Task 8a's, gated on R2. | Task 7 (`TestRelocationReleaseRunsStepsOneToThreeInsideTheDrain`); Task 8a; Requests |
331| 9 | accepted | `ReleasePositions`: the flushed positions are recorded durably, bound to the exact stop request, before the producers close; a replayed receipt is paired with that record or refused. `ReleasePositionsFile` on disk. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions`, `TestReleasePositionsFileBindsTheExactStop`) |
332| 10 | accepted | `RelocationControllers` retains controllers under `Lifetime` (the agent's context); `Effects.Ready` runs before every effect; a restarted agent reopens a controller and never re-journals the acknowledgement. | Task 6 (`TestAParticipantChecksItsControllerBeforeEveryEffectAndReopensAfterARestart`); Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`); Task 10 |
333| 11 | accepted | The pin's key names the claim generation it was selected against (`hostclaim.RecoveryPinKey`, `pin-<gen>.json`); `JournalPin` binds it, so a pin whose CAS lost is abandoned and the retry selects afresh; every bound is checked before the object is written; the counter is gone. | Task 1, Task 5 (`TestStorePinnerPinsTheNewestFilesManifestInTheChain`, `TestStorePinnerChecksTheBoundsBeforeItWrites`); ruling 16 |
334| 12 | accepted | `Worker.Eligible` (`ExecutorChooser.Permits`: the complete eligible set, so stale edges, a missing capability or key, a draining box refuse) and `Worker.Relocatable` (the record's shape) at `Start`; a sidecar family is refused outright. | Task 2 (`TestExecutorChooserPermitsOnlyTheEligibleSet`), Task 5 (`TestWorkerStartRefusesAnIneligibleTargetAndANonRelocatableHost`), Task 10 |
335| 13 | accepted, narrowed | An empty pin is evidenced only by a journaled release at `final_txid` 0 or a holder that never admitted a writer; any other `ErrEmpty` refuses (ruling 15). A drain of a session-0 host is refused at `Start` (ruling 11 amended, request R4). `FlushDB` reports 0 on `ErrEmpty` after the sync; the verifier requires only the files manifest for an empty pin; the first probe is a cold admission into an empty generation. | Task 1 (`TestRelocationEmptyPinNeedsItsEvidence`), Task 5 (`TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerRefusesToDrainAHostThatNeverHadAWriter`), Tasks 8a to 8c |
336| 14 | accepted | `Policy.Check` gains the `reconcile` action for a relocating participant (un-gated); Task 9 is rewritten around three authorities: activation only for the executor, `WithLocal` plus a conditional `SetClaimHold` for a kept row, and no mint for a target before `installing` (ruling 17). Its test drives 6b's `ReconcileAdapter.Install` for the source and the target in every phase. | Task 3 (`TestPolicyLetsAHeldParticipantReconcileItsRow`), Task 9 |
337| 15 | accepted | `Authority.SelectExecutor` returns `selected`; without one `Attempt` runs today's `BeginCreate`. `ProductionAuthority.Executors` nil means no selection; the console's chooser is nil unless `Config.RegionHA` (parsed by `strconv.ParseBool`) is on. | Task 2 (`TestCreateIsTodaysWithoutASelection`, `TestProductionAuthoritySelectsOnlyWithAChooser`, `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) |
338| 16 | accepted | A live target's exhausted replacement goes through `stuckOn` from `probe-failed`; `ReleaseDeadline` marks a release never journaled stuck in `releasing` and `probe-failed` (ruling 14); `relocationAdmission` checks `stuck` before any phase. | Task 1 (`a stuck source does not wake`), Task 5 (`TestWorkerMarksALiveTargetsThirdReplacementStuck`, `TestWorkerMarksAnOverdueReleaseStuck`) |
339
340Astra's readings of rulings 1 to 13 stand; ruling 5's exact-files enforcement is finding 5's fix, and ruling 9's promised behaviour is finding 16's.
341
342## Requests to the claim and moves sessions
343
344Interfaces this plan needs that no merged or declared plan supplies. Each gated task names the request it waits for; nothing here is invented silently.
345
346| # | To | Request | Waited for by |
347|---|---|---|---|
348| R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. *Rev 15 (round 14 finding 3):* claim §4 and Task 1's phase table also let the **source** admit, cold, in `starting` and `releasing`, but 6b's `runtime.Prerequisites` refuses every non-serving claim (`p.Probe || o.Claim.Op.Kind != KindServing`, 6b's `ClaimDependencyError` naming moves C1). Ask: dispatch a relocating claim's ordinary wake to the existing wake inputs when this box is `op.exec` and the source, in `starting` or `releasing`, with its admission fence open (Task 3's `WakePhase` and `RelocationPolicy`); refuse it once the fence closes or the claim no longer names this box. A relocation target's ordinary wake stays refused; its only wake is the probe. *Rev 15 (round 14 finding 1):* the probe's first admission consumes P's prepared manifest (sequence 0) as the session publisher's baseline, by the digest `prepared_authority.files_manifest_sha256` names (the store has no version-id read, ruling 5). The publisher starts at sequence 1. | Task 8c |
349| R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) |
350| R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a |
351| R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 |
352| R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences), and the regional obligations the fence passed to the region (rev 6: `reassigned`, D4's `reassign_regional`). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan |
353| R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 |
354| R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 |
355| R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c |
356| R10 | claim 6b (and the activator's owner) | The reservation-bearing extension of main's isolated root launch protocol, which 6b's own external completion gate names as required "before any tenant shape can run with claims enabled" and as "not supplied by the external owner adapters" (6b rev 5, "External completion gates"). Until it exists, `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable` for every launch (6b, `Preflight`), and the runtime's `Prerequisites` calls `Roles.Preflight` before any dispatch (6b, `runtime.Prerequisites`). Even with R1 every production probe would therefore be refused. The ask: the protocol binds each supported role launch to the durable claim reservation and preserves main's namespace isolation; `Preflight` admits a relocation probe's `"probe"` work through it, bound to `op.exec`, the operation and `install_stamp`; without the protocol, `Preflight` refuses before any probe attempt is counted. | Task 8c, Task 10 |
357| L21 | region HA spec (Paul) | **Open: the durability of the always-continuous items between an app minting them and their first publish.** §5.6 has every cold admission copy its manifest before the app starts, and H6 makes `instance.key`, `report.token` and the env files continuous. Continuous is not synchronous: an app can mint `instance.key`, write database rows sealed with it, and replicate them before the publisher's first manifest with the key lands. A box terminated in that window leaves a nonempty database whose key exists nowhere. Rev 18's pinner refuses that case and the operation goes `stuck` for an operator (ruling 25), but the database is not recoverable by any relocation. Closing the window needs an upstream durability contract, and there are two candidates, neither chosen here: the platform pre-provisions `instance.key` (and `report.token`) into the files directory and publishes them before the app's first start, or the replicator holds its first upload until the first manifest carrying the key has landed. **Decided 2026-10-09 (Paul): the platform pre-provisions the key.** At creation the platform generates it on the box (CSPRNG), writes it create-only (never overwriting) into the instance's overlay env file as `CARLOS_INSTANCE_KEY`, and publishes it before the app's first start. That file is platform-managed and always continuous. Existing instances get one forced publish before region HA is enabled. The app side is a framework contract, requested as rastrillo/rastrillo #66: detect the platform, use the app's own existing key first and the platform's only when it has none, never mint on the platform, and keep today's behaviour off it. The platform side is built only once #66 agrees the contract. The pinner's refusal (ruling 25) stays as the backstop. Residual: an app that later replaces its key reopens the window for the new key. | ruling 25 |
358| R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b |
359
360## Rulings made while planning
361
362Each was a conflict between the authorities or a gap in them. Each says what it costs if it is wrong.
363
3641. **Relocation lives in `internal/hostclaim`.** Main already has `KindRelocating`, its phase validator and `ConvertMovingToRelocating` there, and Plan 6b's boundary table gives "`relocating`, released coverage, once-selected executor and two-box witnesses" to region HA. One implementation per concept: no second package grows claim transitions. *Cost if wrong:* the claim session moves the functions; the names stay.
3652. **The relocation release's stop request names the obligation `relocation-release`.** Root's `FenceAndStop` requires an obligation component, and 6b's `collectCapturedStop` builds only retirement stops (it looks for this box's `park-local` under the operation). A release retires nothing. `host.StopReceiptReleases` accepts only receipts carrying `relocation-release`, so a retirement stop can never be offered as a release, nor the reverse. *Cost if wrong:* a one-line constant, and the 6b session may prefer another literal.
3663. **Discharge stays off until the roster records an enrollment time.** `boxvolumes.Discharge` needs the enrollment time (§24 item 4) and `hostactors.Box` carries none. The production `Discharge` provider returns "the roster records no enrollment time", so a fenced box's `park-local` chores stay open for an operator. H19: this costs cleanup, never recovery. *Ask of the claim session:* add `enrolled_at` to box entries (D7); then the provider reads `boxvolumes.List` and calls `boxvolumes.Discharge`.
3674. **A box-started relocation needs claim §5.1's app-admission registration**, with a `{box, enrollment, controller}` owner. Main has no `apps/<acct>/<app>/admission.json` registry yet. `hostrelocate.Worker` takes an `Admission` interface and refuses to start without one (`ErrUnsupported`). Task 10 is gated on claim Plan 5's app gate as well as 6b. *Cost if wrong:* none until that lands; nothing starts.
3685. **The pinned recovery set is an immutable object** at `control/relocations/<host>/<op>/pin-<claim gen>.json` (rev 2, ruling 16: the generation it was selected against), written create-only and named in the journal by key and SHA-256 (spec §18's first correction: the store has no version-id read, so content is identified by digest). The files pin names its manifest by prefix, sequence and digest. *Cost if wrong:* a deny statement for that prefix in `deployment-buckets` is a follow-up; a rewritten pin is already caught, because every reader checks the journaled digest.
3696. **`installed` is the worker's CAS**, made once a `Prepared` verifier reads P's install manifest complete (§5.8's phase table). The target enters `probing` itself. *Cost if wrong:* who makes one CAS; the claim rules are unchanged.
3707. **A converted move names its target in `pinning`** (`hostclaim.NameTarget`), before `installing`, so the target acknowledges (D6) before it is handed `op.exec` (moves §7.10: "the replacement box `choose()` names when the relocation leaves `pinning`").
3718. **The final TXID N is read from the remote after `SyncAndWait` returns with every writer stopped**, as the newest level-0 `MaxTXID` (§5.8 release step 2). With writers stopped and the sync acknowledged, the stopped database's position and that listing agree; a sync error is returned, never swallowed (claim P6-D4).
3729. **`stuck` is a journal entry, not a phase.** The claim says exhaustion "records `stuck` without resetting the count"; `validateRelocating` lists no `stuck` phase. `MarkStuck` writes `journal.stuck` and every relocation transition refuses afterwards; it refuses between `installed` and a probe.
37310. **The probe deadline starts at the `probing` CAS** (`journal.probing_since`), and the worker moves a live target to `probe-failed` 10 minutes later.
37411. **A host whose source never had a writer** (session 0, `writer_box` empty) is relocated with coverage `""`, which declares nothing (claim §6.4a: "a created incarnation's first admission … declares nothing"), never `released`. *Rev 2:* only by failover. A drain of such a host is refused at `Worker.Start`: root refuses stop evidence for writer session 0 (`validateRequest`), and 6b's `collectCapturedStop` waits for claim Plan 5's no-process witness (R4). Its failover pin is an evidenced empty one (ruling 15). *Cost if wrong:* a box with never-woken hosts cannot be drained until R4 lands; nothing is lost.
37512. **L18's owner.** The coordination post on #236 says moves owns `internal/sim/hostlife`; claim rev 6.30 §1.1 calls the two-box model "region HA's own … as part of region HA's plan"; the moves conflicts note defers it to "a joint round with the region HA session". This plan writes no model and ends with a merge gate that names the cases; whichever session builds the model, relocation does not merge until that round passes for a one-box and a two-box region.
37613. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1). *Corrected in rev 15 (round 14 finding 1):* until rev 14 this plan did not follow that binding, because it copied the source manifest's sequence into P. Under claim rev 6.30 (R18-1) and moves rev 7.22 §7.6, P's prepared manifest is create-only at sequence 0 and bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. The source pin's sequence is provenance only (`PreparedAuthority.SourceFilesSeq`). `CopyPinnedManifest` writes sequence 0, and `ToInstalled` requires it and binds the stamp. Nothing in L14 to L17 changed against it.
37714. **The release deadline is 10 minutes** (`hostrelocate.ReleaseDeadline`, rev 2). §5.8 says a flush that never reaches N, and a `probe-failed` target that cannot finish its release, go to `stuck`, and names no bound. `ToReleasing` and `ToProbeFailed` journal when the phase began; a worker marks the operation stuck once that long has passed with no release journaled. The host stays where it is: nothing moves away from a live box without its stop evidence. *Cost if wrong:* a number; the stuck state and the operator path are the claim's.
37815. **An empty pin needs evidence a writer can give** (rev 2; claim §6.4a "Empty bootstrap": no LTX file exists until the first commit, so a listing that finds nothing is a definite observation but says nothing about who wrote). A pin is `Empty` only when the journaled release reports `final_txid` 0 (the stopped writer flushed and found nothing), or the holder never admitted a writer (session 0, failover). A failover of a host whose session is above 0 with an empty listing refuses and waits for an operator; `JournalPin` enforces it, the pinner never writes it. *Cost if wrong:* that one case recovers by hand until R6 names its evidence.
37916. **A pin is bound to the claim generation it was selected against** (rev 2; B2-3). Its key is `control/relocations/<host>/<op>/pin-<gen>.json` and `JournalPin` requires `gen` to be the observed claim's. A pin whose journal CAS lost is abandoned where it lies (never referenced, cheap, reaped by the storage follow-up of ruling 5); the next pass selects afresh at the new generation. Every bound `JournalPin` checks (the release's positions, the failover window) is checked before the object is written, so a doomed pin is never published. *Cost if wrong:* abandoned objects under a prefix nothing reads.
38017. **Reconcile has three authorities for a relocation row** (rev 2; §5.3, 6b's `ClaimOwnership.Valid` needs a stamp). The executor (the source until its fence closes, the target from `installing` on) activates its own installation stamp and mints or keeps its row through the guarded path. A kept row of a box that is no longer the executor (the source after its release, a replaced target) gets its hold through `registry.SetClaimHold` under the flock, conditional on the row's recorded ownership: a hold grants nothing, so it needs no installation authority. A target before `installing` has no installation and mints nothing; the edges file already withholds the host from a box that is not `op.exec` (§24). *Cost if wrong:* a request that reaches a pre-installation target meets the box's unknown-host answer instead of a hold, for the seconds between `starting` and `installing`.
38119. **Probe attempts are counted before they are made** (rev 3; §5.8 "limits": at most three, then `probe-failed`). `BeginProbeAttempt` is one CAS per attempt, keyed by box, so a crash, a lost response or a restart never buys a fourth and a replacement target starts at zero; the participant enters `probe-failed` when refused. *Cost if wrong:* an attempt lost to a crash before its admission still counts; the bound is the spec's.
38220. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. *Rev 6:* a durable record is cleared only straight after that scoped cleanup succeeds in the same call (`Finish` repeats it after `Close`, and on a restart), because hostexec reports a cleanup failure from `Close` once and nil afterwards; the participation is journaled before its controller opens, so a restart finds every record. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's.
38321. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations/<host>/<op>/manifest-<sha256>.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers.
38422. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's.
38523. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). *Rev 9:* that holds only when the dead target is `op.exec`, or `op.exec` is already "none". While the live source is `op.exec` (`starting`, `releasing`), the CAS consumes the target's fence and records `stuck`. It keeps the source's execution identity, phase and release evidence, because a target's fence is no evidence against the source (claim §4), and root's stop needs the source to remain `op.exec`. Otherwise `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's.
38624. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive.
38725. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `<host>.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. *Rev 18 (round 17 finding 1):* an existing manifest is not evidence either. When a writer was admitted and the pinned database is nonempty, the pinned manifest must carry a live `instance.key`, the minimum of H6's always-continuous items. A manifest copied before the app started, with the key minted after and never published, refuses the pin, and the operation is `stuck`. The durability window itself is not closed by this plan. It is the open request L21, for Paul to choose between the platform pre-provisioning the key and the replicator holding until it lands. *Cost if wrong (rev 18):* an app that never writes `instance.key` and has a nonempty database is stuck on every relocation until an operator resolves it; H6 names the key as platform-managed on every plan, so such an app should not exist.
38826. **A drain's probe-failed with no eligible spare is stuck after `ReleaseDeadline`, never sent back to the source** (rev 17, finding F1). §5.8 excludes `to` from the choice after a failed probe, and a relocation's target is never its source (the start CAS, `replaceTargetSlot`). Letting the live source be the "replacement" would put a fresh install of P' into the same host paths that the source's own `park-local` chore retires once the operation completes (§5.8, "What a relocation leaves behind"). The same box would then hold the retiring installation and the new one under one name. So the worker waits for a spare and, `ReleaseDeadline` after `probe_failed_since`, marks the operation `stuck`. This follows the spec's own bound ("It alerts, and the op waits for `carlos ops host resolve`"). The operator's resolve is the claim's: it may return the host to the source, as the model's ruling 13 does. *Cost if wrong:* in a two-box region a failed drain waits ten minutes plus an operator instead of an automatic return.
38927. **Every abandoned prepared generation is parked by the operation that built it** (rev 17, finding F3). §5.8 says "an abandoned `P` is parked by the at-rest sweep". This plan adds an earlier owner: `install` journals every unopened P it replaces (`abandoned_prepared`), and `CompleteRelocation` queues a `park-generation` chore for each in the region, beside its `park-local` chores. The at-rest sweep stays the backstop for an operation that never completes. A P that any admission opened is part of the writer lineage and is never on the list. *Cost if wrong:* one chore per abandoned build. Its objects are outside every live set (a prepared manifest names the source's objects, not P's), so parking them retires nothing live.
39018. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires.
391
392## Why this is 1b-next, and what it leaves out
393
394Plan 1b built every part of rev 5.4 that needed none of L1 to L20. Since then claim Plans 1 to 8b and 5 landed on main (3407d114): the roster (`hostactors.Directory`), the fence writer (`hostfence.TerminationWorker`), the fence reader for claims (`hostclaim.VerifyFence`, `hostfence.ClaimFenceAuthority`), the execution controller (`hostexec`), stop receipts (`host.ClaimStopCoordinator`), the box-started census (`Directory.CensusBoxRelocation`) and H14's start check (`Directory.CheckBoxStart`). This plan uses all of them and duplicates none.
395
396Left out, with where it goes:
397
398- **The placer loop that decides what to relocate** (spec 3): reading the termination queue, relocating every host off a dead box in bounded parallel, rebalancing. This plan's worker starts a relocation when asked and replays any it finds; Task 10's pass only replays.
399- **Automatic enrollment and the cross-region confirmer** (spec 4, H16, H17).
400- **The console's copy of a fence into a D4 loss record and the roster's retirement of the box** (claim D4 "Automatic loss persistence"). The fence CAS keeps the reference and the discharge verdict in the operation journal and puts the box on `lost`; materializing the loss record is the console's, and is listed under "After this plan".
401- **Storage policy for `control/relocations/`** (ruling 5).
402
403## Global Constraints
404
405- **Dormant.** Nothing changes on a box or console where `CARLOS_REGION_HA` is unset or not true (`strconv.ParseBool`), and it is read from `/etc/carlos/host.env` by `regionHAEnabled` (`internal/host/boxrecord.go`). With one box per region every rule is a no-op (spec §4, §8 step 2).
406- **Activation gate (claim §1.1, L18).** "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) … validate[s] them." Relocation may not merge until Task 11's joint round passes.
407- **One implementation per concept; main's wins (#236).** Do not write a termination-fence writer (`hostfence.TerminationWorker` writes `control/box-fences/<enrollment>.json` using `termfence.Verify`). Do not write a roster reader (`hostactors.Directory`). Read fences only through `hostclaim.VerifyFence` (in claim CASes) and `hostactors.ValidateTerminationFence` (before them).
408- **H19, verbatim:** "The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores." No volume record gates a fence, a relocation step or an admission.
409- **Absence is never evidence (§5.9).** A fence that cannot be read or does not validate is an error, never "not fenced"; a box missing from the roster refuses.
410- **Every phase change is one claim CAS conditional on `op.id`, the phase and the ETag (claim C2).** Two workers racing make the same CAS and one gets `hostclaim.ErrConflict`.
411- **Exact values.** Failover pin delay **30 seconds** after the fence's `observed_at` (`hostclaim.FailoverPinDelay`); probe deadline **10 minutes** (`hostrelocate.ProbeDeadline`); at most **2** target replacements per operation (`hostclaim.MaxTargetReplacements`); reconcile's kept-row sweep **10 minutes** (H9); release deadline **10 minutes** (`hostrelocate.ReleaseDeadline`, ruling 14); probe attempts **3** per target installation (`hostclaim.MaxProbeAttempts`, ruling 19); create intent keys **`selected_executor`** and **`executor_enrollment`** (claim P5-D6); journal fence key **`fence.<box>`** holding `{"enrollment","key","sha256"}` (claim §7.3); release obligation **`relocation-release`**; pin objects **`control/relocations/<host>/<op>/pin-<claim gen>.json`** (`hostclaim.RecoveryPinKey`, ruling 16); loss ids **`auto-<32 hex>`** over (op, box, fence); the retained manifest **`control/relocations/<host>/<op>/manifest-<sha256>.json`** (ruling 21); the env platform item **`host.env`** (`hostfiles.EnvItem`).
412- **Gated tasks.** Tasks 8a, 8b, 8c, 9 and 10 start with a "Gated on" line. Their code is written against the interfaces claim Plan 6b rev 5 declares (`ClaimAssembly`, `ClaimWakeAssembly`, `ClaimWakeInputs`, `CollectClaimStop`, `localStopIdentity`, `hostbox.ReconcileAdapter`, `registry.SetClaimHold`), moves Phase 1 Plan 2 Task 8 declares (`moves.BuildGeneration`, `BuildInputs`, `SealedObject`, the key helpers) and moves Plan 1 Task 7b declares (`RenderHeld`, `MaterializeHeld`), plus the requests R1 to R3 and R7. They were not compiled. When 6b and moves land, re-read their merged code and bind the same contracts to the names they actually have; the un-gated tasks do not change.
413- **No new module dependencies. Do not add a method to `release.Store`.** Linux-only syscalls in `*_linux.go` with a `!linux` stub; CI cross-compiles darwin, windows and linux/arm64.
414- **Copy.** No user-facing copy. Errors and log lines are operator-facing and follow the surrounding style. If a task finds it needs console or CLI text, stop: it waits for Paul's copy review.
415- **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`.
416- **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.)
417- Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11.
418- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's, rev 17's eight, rev 18's five and rev 19's two (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed.
419
420## Review Focus
421
4221. **`released` coverage naming the wrong predecessor.** A drain's first probe admission declares nothing; if the session it vouches for is not exactly the one whose release was journaled (another box, another session, another generation), a real loss goes unrecorded. Expect refusal. Task 1, `TestRelocationAdmissionRules/released_names_another_predecessor`, `…/released_names_another_predecessor_box`, `…/released_after_P_was_opened`.
4232. **P opened twice.** A restarted probe that restores P again loses everything attempt two replicated (B1-1). Expect every attempt after the first to take a fresh generation, and P refused. Task 1, `TestRelocationAdmissionRules/a_third_attempt_reopens_P`.
4243. **A failover pinned too early.** Requests S3 accepted from the terminated box can still land; a pin taken before `observed_at` + 30 s widens H3's window. Expect the worker to wait and the claim to refuse. Task 1 `TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss`, Task 5 `TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence`.
4254. **A fence that cannot be read taken as no fence.** A garbled `control/box-fences/` object must stop the drain and the create choice, not let them proceed as if the box were alive. Task 2 `TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/an_unreadable_fence`, Task 5 `TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence`.
4265. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. A matching number and path is not enough: the kernel must say the table and the description are shared (rev 7). Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`, `…/an_unshared_thread_at_the_leader's_number_and_path`, `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity`.
427
428## File Structure
429
430| Path | Responsibility | Task |
431|---|---|---|
432| `internal/hostclaim/relocation.go` | The relocation transitions: start (drain, failover), source loss, barrier, release journal, pin, install, local install receipt, probe, serving, target replacement, stuck; fences consumed in the same CAS | 1 |
433| `internal/hostclaim/generation.go`, `claim.go` | `AdmitWriter`'s relocating phase table and `released` coverage (L1, L9); `evidence_refs`, the retained automatic loss (D4) | 1 |
434| `internal/hostclaim/create.go` | `ExecutorSelection`, `BeginCreateOn`; `CompleteCreate` never substitutes | 2 |
435| `internal/console/server.go`, `cmd/console/main.go` | `Config.RegionHA` from `CARLOS_REGION_HA` | 2 |
436| `internal/hostops/executor.go`, `loads.go`, `operations.go`, `backfill.go`, `backfill_controller.go` | `ExecutorChooser` over the roster, the edges snapshot and claim loads, and `Permits`; `Authority.SelectExecutor` before the create CAS, or today's create without one | 2 |
437| `internal/hostbox/relocationrows.go`, `wakepolicy.go`, `internal/hostguard/admissionfence.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner, its `reconcile` action and `RelocationPolicy`; the admission-fence reader | 3 |
438| `internal/stopevidence/writers*.go` | L10's writers-stopped observation; one process's descriptors on the database | 4 |
439| `internal/hostrelocate/worker.go`, `pin.go`, `internal/hostfiles/manifestcheck.go` | The regional worker: start (eligibility, shape), one replayed CAS per step, the pin under the location lease, the release deadline; the pinned recovery set by claim generation, `ReadPin`, `CopyPinnedManifest` | 5 |
440| `internal/hostrelocate/participant.go` | A participant box's transitions, ordered over its local effects | 6 |
441| `internal/host/relocationreleases.go`, `relocationeffects.go`, `relocationflush.go`, `internal/hostfiles/platform.go` | Stop receipts as release evidence; the release sequence inside root's stop collection (stop writers, observe, flush, record, drain); retained controllers; durable positions; the full files flush; `EnvItem` | 7 |
442| `internal/host/relocationassembly.go` | Gated: the release providers bound to 6b's assembly | 8a |
443| `internal/host/relocationbuild.go` | Gated: P's build under the lease; the `Prepared` verifier | 8b |
444| `internal/host/relocationinstall.go`, `relocationwake.go` | Gated: the local install under its stamp; the probe's inputs | 8c |
445| `internal/hostbox/reconcile.go`, 6b's `reconcile_adapter.go` | Gated: held rows and the 10-minute sweep | 9 |
446| `internal/console/hostclaimexecutor.go` | The console's create chooser (constructed, not yet used by a door) | 2 |
447| `internal/host/relocatepass.go` | Gated: the passes, behind `CARLOS_REGION_HA`, and the create door's chooser | 10 |
448
449---
450
451### Task 1: The `relocating` operation in the claim library (L1, L2 to L4, L9, L11)
452
453Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.<box>`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies; the same CAS passes the box's regional work (`park-generation`, `park-replica` owned by `box:<b>`) to `region:<r>`, as `DeclareLost` does (rev 6). A box owing a `park-local` is never named a target (`replaceTargetSlot`, rev 6).
454
455**Files:**
456- Create: `internal/hostclaim/relocation.go`
457- Modify: `internal/hostclaim/generation.go` (the `AdmitWriter` kind switch, the coverage switch, a new `relocationAdmission`), `internal/hostclaim/claim.go` (`EvidenceRefs`, validated)
458- Test: `internal/hostclaim/relocation_test.go`
459
460**Interfaces:**
461- Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`.
462- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`, `Reassigned`) and `Claim.EvidenceRefs`; `OwesLocal(c, box) bool`, `LocalChoreBoxes(c) []string`, `PreparedOpened(c) bool`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`.
463- Journal keys later tasks read: `exec_ack.<box>`, `fence.<box>`, `fence.<box>.observed_at`, `discharge.<box>`, `reason`, `releasing_since`, `release`, `release.phase`, `released_by_target`, `pin`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe_attempts.<box>`, `probe`, `probe_failed_since`, `replacements`, `excluded_targets`, `stuck`, `prepared_opened` (rev 12). Claim fields: `evidence_refs`, `loss_refs`.
464
465- [ ] **Step 1: Write the failing tests**
466
467`internal/hostclaim/relocation_test.go`:
468
469```go
470package hostclaim
471
472import (
473 "context"
474 "crypto/sha256"
475 "encoding/hex"
476 "encoding/json"
477 "errors"
478 "reflect"
479 "slices"
480 "strings"
481 "testing"
482 "time"
483
484 "github.com/carlosframework/platform/internal/boxid"
485 "github.com/carlosframework/platform/internal/release"
486 "github.com/carlosframework/platform/internal/termfence"
487)
488
489const (
490 relSource = "adi-knitting"
491 relTarget = "bri-knitting"
492 relSpare = "cai-knitting"
493 relSpare2 = "dov-knitting"
494)
495
496var relObserved = time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)
497
498func relEnrollment(box string) string { return strings.Repeat(string(box[0]), 32) }
499
500// relAuthority accepts any fence record for a box in this region.
501type relAuthority struct{}
502
503func (relAuthority) ValidateFence(_ context.Context, f termfence.Record) (FenceBinding, error) {
504 if f.Enrollment != relEnrollment(f.Box) {
505 return FenceBinding{}, ErrFence
506 }
507 return FenceBinding{Region: "ie", RosterRevision: 9, RosterETag: "roster"}, nil
508}
509
510type relReleases struct{ err error }
511
512func (r relReleases) VerifyRelease(context.Context, Claim, Release) error { return r.err }
513
514func putFence(t *testing.T, s release.Store, box string) FenceRef {
515 t.Helper()
516 f := termfence.Record{V: 1, Box: box, Enrollment: relEnrollment(box), State: "terminated",
517 Cloud: boxid.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box},
518 ObservedAt: relObserved.Format(time.RFC3339Nano), Via: "describe", RequestID: "req",
519 WrittenBy: termfence.WrittenBy{Box: "worker", Enrollment: strings.Repeat("e", 32)}}
520 raw, err := json.Marshal(f)
521 if err != nil {
522 t.Fatal(err)
523 }
524 sum := sha256.Sum256(raw)
525 ref := FenceRef{Enrollment: f.Enrollment, Key: "control/box-fences/" + f.Enrollment + ".json", SHA256: hex.EncodeToString(sum[:])}
526 if _, err := s.Put(context.Background(), ref.Key, raw, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
527 t.Fatal(err)
528 }
529 return ref
530}
531
532// relServing is a serving claim on relSource whose source has admitted one
533// cold writer session (generation g1).
534func relServing(t *testing.T) (release.Store, Observed) {
535 t.Helper()
536 ctx := context.Background()
537 s := release.NewDirStore(t.TempDir())
538 o := servingClaim(t, ctx, s)
539 o, err := Update(ctx, s, o, AdmitWriter(Admission{Box: relSource, Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true}))
540 if err != nil {
541 t.Fatal(err)
542 }
543 return s, o
544}
545
546func relStart(reason string, fence *FenceRef) RelocationStart {
547 return RelocationStart{OpID: "rel-1", By: "worker", Now: relObserved, Region: "ie", Reason: reason,
548 SourceBox: relSource, SourceEnrollment: relEnrollment(relSource),
549 TargetBox: relTarget, TargetEnrollment: relEnrollment(relTarget),
550 DirectoryRevision: 9, AdmissionID: "adm-1", Fence: fence}
551}
552
553func step(t *testing.T, s release.Store, o Observed, tr func(*Claim) error) Observed {
554 t.Helper()
555 next, err := Update(context.Background(), s, o, tr)
556 if err != nil {
557 t.Fatal(err)
558 }
559 return next
560}
561
562func ack(box string) func(*Claim) error {
563 return AcknowledgeExecution(box, ExecutionAcknowledgement{Op: "rel-1", Enrollment: relEnrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: relObserved.Add(time.Hour)})
564}
565
566// relPin is a pin selected against o: its key names o's claim generation.
567func relPin(o Observed, maxTXID uint64, seq int64, at time.Time) RecoveryPin {
568 return RecoveryPin{Key: RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen), SHA256: strings.Repeat("a", 64), Generation: "g1", MaxTXID: maxTXID, FilesManifest: "instances/x/files/g1/manifest.json", FilesSeq: seq, PinnedAt: at, Empty: maxTXID == 0}
569}
570
571func relPrepared(gen string, maxTXID uint64, filesSeq int64) PreparedAuthority {
572 a := PreparedAuthority{Generation: gen, SnapshotTXID: maxTXID, FilesSeq: 0, SourceFilesSeq: filesSeq, FilesManifestSHA256: strings.Repeat("f", 64)}
573 if maxTXID > 0 {
574 a.InstallManifest, a.SHA256 = "m", strings.Repeat("c", 64)
575 }
576 return a
577}
578
579func localInstall(c *Claim) LocalInstall {
580 return LocalInstall{Box: c.Op.Exec, Enrollment: relEnrollment(c.Op.Exec), InstallStamp: journalUint(c.Op, "install_stamp"), FilesManifestVersion: 3, JournalSHA256: strings.Repeat("b", 64)}
581}
582
583// toProbing drives an operation at pinning (with its pin) through
584// installing, installed and the local install receipt into probing.
585func toProbing(t *testing.T, s release.Store, o Observed, prepared string, maxTXID uint64) Observed {
586 t.Helper()
587 target := Target{Box: o.Claim.Op.Intent["target_box"], Enrollment: o.Claim.Op.Intent["target_enrollment"]}
588 o = step(t, s, o, ToInstalling("rel-1", target, prepared))
589 var pin RecoveryPin
590 readJournal(o.Claim.Op, "pin", &pin)
591 o = step(t, s, o, ToInstalled("rel-1", relPrepared(prepared, maxTXID, pin.FilesSeq)))
592 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
593 return step(t, s, o, ToProbing("rel-1", relObserved))
594}
595
596func TestRelocationDrainEndToEnd(t *testing.T) {
597 ctx := context.Background()
598 s, o := relServing(t)
599 o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
600 if err != nil {
601 t.Fatal(err)
602 }
603 if c := o.Claim; c.Op.Exec != relSource || c.Op.Phase != PhaseStarting || !slices.Equal(c.Op.Participants, []string{relSource, relTarget}) || c.Op.Intent["stamp_from"] == "" {
604 t.Fatalf("start: %+v", c.Op)
605 }
606 if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) {
607 t.Fatalf("barrier without acknowledgements: %v", err)
608 }
609 o = step(t, s, o, ack(relSource))
610 o = step(t, s, o, ack(relTarget))
611 o = step(t, s, o, ToReleasing("rel-1", relObserved))
612 if o.Claim.Op.Journal["releasing_since"] == "" {
613 t.Fatal("releasing did not start its deadline")
614 }
615 h := o.Claim.Holder
616 rel := Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "control/stop/r.json", ReceiptSHA256: strings.Repeat("d", 64)}
617 o = step(t, s, o, JournalRelease("rel-1", rel))
618 if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{err: errors.New("bad receipt")}, nil); err == nil {
619 t.Fatal("pinned on an unverified receipt")
620 }
621 if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil {
622 t.Fatal(err)
623 }
624 stamp := o.Claim.Holder.Stamp
625 if o.Claim.Op.Exec != "none" || o.Claim.Op.Phase != PhasePinning {
626 t.Fatalf("pinning: %+v", o.Claim.Op)
627 }
628 if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 39, 7, relObserved))); !errors.Is(err, ErrRelocation) {
629 t.Fatalf("pin short of the release: %v", err)
630 }
631 stale := relPin(o, 40, 7, relObserved)
632 stale.Key = RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen-1)
633 if _, err := Update(ctx, s, o, JournalPin("rel-1", stale)); !errors.Is(err, ErrRelocation) {
634 t.Fatalf("a pin selected against an earlier observation was journaled: %v", err)
635 }
636 o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved)))
637 if o.Claim.Holder.Stamp != stamp {
638 t.Fatal("pinning changed the stamp")
639 }
640 o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1"))
641 for name, a := range map[string]PreparedAuthority{
642 "no files manifest": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40},
643 "another files seq": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40, FilesSeq: 6, FilesManifestSHA256: strings.Repeat("f", 64)},
644 "no install manifest": {Generation: "P1", SnapshotTXID: 40, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)},
645 "another snapshot txid": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 41, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)},
646 } {
647 if _, err := Update(ctx, s, o, ToInstalled("rel-1", a)); !errors.Is(err, ErrRelocation) {
648 t.Fatalf("%s: installed on an incomplete build: %v", name, err)
649 }
650 }
651 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7)))
652 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
653 o = step(t, s, o, ToProbing("rel-1", relObserved))
654 install := journalUint(o.Claim.Op, "install_stamp")
655 if o.Claim.Op.Exec != relTarget || o.Claim.Holder.Stamp != install || !slices.Contains(o.Claim.Touched.Boxes, relTarget) {
656 t.Fatalf("installation: exec %s stamp %d install %d", o.Claim.Op.Exec, o.Claim.Holder.Stamp, install)
657 }
658 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
659 Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
660 last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
661 if last.Coverage != "released" || last.WritesLost != nil || o.Claim.Holder.Stamp != install {
662 t.Fatalf("released admission: %+v stamp %d", last, o.Claim.Holder.Stamp)
663 }
664 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session}))
665 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
666 c := o.Claim
667 if c.Op.Kind != KindServing || c.Op.ID != "" || c.Op.Exec != relTarget || !slices.Equal(c.Op.Participants, []string{relTarget}) {
668 t.Fatalf("serving: %+v", c.Op)
669 }
670 if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-local" || c.Obligations[0].Box != relSource || c.Obligations[0].Op != "rel-1" {
671 t.Fatalf("the source's park-local: %+v", c.Obligations)
672 }
673}
674
675func TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss(t *testing.T) {
676 ctx := context.Background()
677 for _, qualified := range []bool{false, true} {
678 s, o := relServing(t)
679 o = step(t, s, o, func(c *Claim) error { // a chore the dead source already owed
680 c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource})
681 return nil
682 })
683 ref := putFence(t, s, relSource)
684 sourceSession := o.Claim.Holder.Session
685 in := relStart(ReasonFailover, &ref)
686 in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}}
687 o, err := StartRelocation(ctx, s, o, in, relAuthority{})
688 if err != nil {
689 t.Fatal(err)
690 }
691 c := o.Claim
692 if c.Op.Exec != "none" || !slices.Equal(c.Op.Participants, []string{relTarget}) || !slices.Contains(c.Lost, relSource) || c.Op.Journal["fence."+relSource] == "" {
693 t.Fatalf("failover start: %+v lost %v", c.Op, c.Lost)
694 }
695 if owes := slices.ContainsFunc(c.Obligations, func(o Obligation) bool { return o.Box == relSource }); owes == qualified {
696 t.Fatalf("discharge qualified=%v left chores %+v", qualified, c.Obligations)
697 }
698 if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); err == nil {
699 t.Fatal("a failover entered releasing")
700 }
701 o = step(t, s, o, ack(relTarget))
702 o = step(t, s, o, PinFailover("rel-1"))
703 if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(29*time.Second)))); !errors.Is(err, ErrRelocation) {
704 t.Fatalf("pinned inside the 30-second window: %v", err)
705 }
706 o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay))))
707 o = toProbing(t, s, o, "P1", 40)
708 probe := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
709 Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}
710 if _, err := Update(ctx, s, o, AdmitWriter(probe)); !errors.Is(err, ErrWrongState) {
711 t.Fatalf("a failover admission claimed released coverage: %v", err)
712 }
713 probe.Coverage = ""
714 o = step(t, s, o, AdmitWriter(probe))
715 last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
716 if last.WritesLost == nil || last.WritesLost.Box != relSource || last.WritesLost.Generation != "g1" {
717 t.Fatalf("failover must declare the source's session: %+v", last)
718 }
719 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session}))
720 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
721 owes := slices.ContainsFunc(o.Claim.Obligations, func(ob Obligation) bool { return ob.Kind == "park-local" && ob.Box == relSource })
722 if owes == qualified {
723 t.Fatalf("qualified=%v: source chores at serving %+v", qualified, o.Claim.Obligations)
724 }
725 // D4: the loss outlives the operation, with the console still to
726 // materialize it (nothing here stands in for the console).
727 c = o.Claim
728 if len(c.EvidenceRefs) != 1 || len(c.LossRefs) != 1 || c.EvidenceRefs[0].ID != c.LossRefs[0] {
729 t.Fatalf("retained losses after serving: %+v refs %v", c.EvidenceRefs, c.LossRefs)
730 }
731 l := c.EvidenceRefs[0]
732 if l.Box != relSource || l.Fence != ref || l.ObservedAt != relObserved.Format(time.RFC3339Nano) || l.Op != "rel-1" || l.Role != "source" ||
733 l.Session != sourceSession || l.Generation != "g1" || l.Discharge.Qualified != qualified || l.Timers != TimersUnknown || l.Reported {
734 t.Fatalf("retained loss: %+v", l)
735 }
736 if c.Op.Journal != nil || !slices.Contains(c.Lost, relSource) {
737 t.Fatalf("serving kept the journal, or lost the box: %+v %v", c.Op, c.Lost)
738 }
739 }
740}
741
742func TestRelocationSourceLossBecomesFailover(t *testing.T) {
743 ctx := context.Background()
744 for _, tc := range []struct {
745 name string
746 prepare func(*testing.T, release.Store, Observed) Observed
747 wantPhase string
748 wantExec string
749 wantParts []string
750 }{
751 {"starting before acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return o }, PhaseStarting, "none", []string{relTarget}},
752 {"starting after acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return step(t, s, o, ack(relSource)) }, PhaseStarting, "none", []string{relTarget}},
753 {"releasing before its receipt", func(t *testing.T, s release.Store, o Observed) Observed {
754 o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
755 return step(t, s, o, ToReleasing("rel-1", relObserved))
756 }, PhasePinning, "none", []string{relTarget}},
757 {"releasing after its receipt", func(t *testing.T, s release.Store, o Observed) Observed {
758 o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
759 o = step(t, s, o, ToReleasing("rel-1", relObserved))
760 h := o.Claim.Holder
761 return step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 1, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
762 }, PhaseReleasing, relSource, []string{relSource, relTarget}},
763 } {
764 t.Run(tc.name, func(t *testing.T) {
765 s, o := relServing(t)
766 o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
767 if err != nil {
768 t.Fatal(err)
769 }
770 o = tc.prepare(t, s, o)
771 stamp := o.Claim.Holder.Stamp
772 o, err = FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{})
773 if err != nil {
774 t.Fatal(err)
775 }
776 c := o.Claim
777 if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || !slices.Equal(c.Op.Participants, tc.wantParts) || c.Holder.Stamp != stamp || !slices.Contains(c.Lost, relSource) {
778 t.Fatalf("phase %s exec %s parts %v stamp %d lost %v", c.Op.Phase, c.Op.Exec, c.Op.Participants, c.Holder.Stamp, c.Lost)
779 }
780 if c.Op.Intent["reason"] != ReasonDrain {
781 t.Fatal("the intent was rewritten")
782 }
783 })
784 }
785}
786
787// relAtPhase starts a drain and drives it to phase. "probing+P" is probing
788// after the first admission opened P.
789func relAtPhase(t *testing.T, phase string) (release.Store, Observed) {
790 t.Helper()
791 ctx := context.Background()
792 s, o := relServing(t)
793 o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
794 if err != nil {
795 t.Fatal(err)
796 }
797 o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
798 if phase == PhaseStarting {
799 return s, o
800 }
801 o = step(t, s, o, ToReleasing("rel-1", relObserved))
802 if phase == PhaseReleasing {
803 return s, o
804 }
805 h := o.Claim.Holder
806 o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
807 if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil {
808 t.Fatal(err)
809 }
810 o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved)))
811 if phase == PhasePinning {
812 return s, o
813 }
814 o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1"))
815 if phase == PhaseInstalling {
816 return s, o
817 }
818 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7)))
819 if phase == PhaseInstalled {
820 return s, o
821 }
822 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
823 o = step(t, s, o, ToProbing("rel-1", relObserved))
824 if phase == PhaseProbing {
825 return s, o
826 }
827 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
828 Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
829 return s, o
830}
831
832func TestRelocationReplacementTable(t *testing.T) {
833 ctx := context.Background()
834 for _, tc := range []struct {
835 at, wantPhase, wantExec string
836 newStamp, keepP bool
837 }{
838 {PhaseStarting, PhaseStarting, relSource, false, false},
839 {PhaseReleasing, PhaseReleasing, relSource, false, false},
840 {PhasePinning, PhasePinning, "none", false, false},
841 {PhaseInstalling, PhaseInstalling, relSpare, true, false},
842 {PhaseInstalled, PhaseInstalled, relSpare, true, true},
843 {PhaseProbing, PhaseInstalled, relSpare, true, true},
844 {"probing+P", PhasePinning, "none", false, false},
845 } {
846 t.Run(tc.at, func(t *testing.T) {
847 s, o := relAtPhase(t, tc.at)
848 before := o.Claim.Holder.Stamp
849 o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{})
850 if err != nil {
851 t.Fatal(err)
852 }
853 c := o.Claim
854 if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || c.Op.Intent["target_box"] != relSpare || !slices.Contains(c.Lost, relTarget) || slices.Contains(c.Op.Participants, relTarget) {
855 t.Fatalf("phase %s exec %s intent %v lost %v parts %v", c.Op.Phase, c.Op.Exec, c.Op.Intent, c.Lost, c.Op.Participants)
856 }
857 if (c.Holder.Stamp != before) != tc.newStamp {
858 t.Fatalf("stamp %d -> %d", before, c.Holder.Stamp)
859 }
860 if tc.keepP && c.Op.Journal["prepared"] != "P1" {
861 t.Fatalf("the unopened P must be reused: %q", c.Op.Journal["prepared"])
862 }
863 if c.Op.Journal["local_install"] != "" {
864 t.Fatal("a replacement inherited its predecessor's receipt")
865 }
866 if tc.wantPhase == PhaseInstalled {
867 if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); err == nil {
868 t.Fatal("probing without the replacement's own local install")
869 }
870 }
871 })
872 }
873}
874
875func TestRelocationTwoReplacementsThenStuck(t *testing.T) {
876 ctx := context.Background()
877 s, o := relAtPhase(t, PhasePinning)
878 var err error
879 for _, next := range []string{relSpare, relSpare2} {
880 old := o.Claim.Op.Intent["target_box"]
881 if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil {
882 t.Fatal(err)
883 }
884 }
885 if _, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, Target{"eve-knitting", relEnrollment("eve-knitting")}, "", relAuthority{}); !errors.Is(err, ErrReplacementsExceeded) {
886 t.Fatalf("third replacement: %v", err)
887 }
888 o = step(t, s, o, MarkStuck("rel-1", "two target replacements used"))
889 if journalUint(o.Claim.Op, "replacements") != 2 {
890 t.Fatal("stuck reset the count")
891 }
892 if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P9")); !errors.Is(err, ErrRelocation) {
893 t.Fatalf("a stuck operation moved: %v", err)
894 }
895}
896
897func TestRelocationReplacementInReleasingAcknowledgesBeforePinning(t *testing.T) {
898 ctx := context.Background()
899 s, o := relAtPhase(t, PhaseReleasing)
900 h := o.Claim.Holder
901 o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
902 o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{})
903 if err != nil || o.Claim.Op.Phase != PhaseReleasing {
904 t.Fatalf("replacement in releasing: %v %+v", err, o.Claim.Op)
905 }
906 if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); !errors.Is(err, ErrRelocation) {
907 t.Fatalf("left releasing before the replacement acknowledged: %v", err)
908 }
909 o = step(t, s, o, ack(relSpare))
910 if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil || o.Claim.Op.Phase != PhasePinning {
911 t.Fatalf("after the acknowledgement: %v %+v", err, o.Claim.Op)
912 }
913}
914
915func TestRelocationExhaustedTargetIsFencedIntoAStuckPinning(t *testing.T) {
916 ctx := context.Background()
917 exhaust := func(t *testing.T, s release.Store, o Observed) Observed {
918 var err error
919 for _, next := range []string{relSpare, relSpare2} {
920 old := o.Claim.Op.Intent["target_box"]
921 if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil {
922 t.Fatal(err)
923 }
924 }
925 return o
926 }
927 t.Run("before exhaustion it is a replacement", func(t *testing.T) {
928 s, o := relAtPhase(t, PhasePinning)
929 if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, relAuthority{}); !errors.Is(err, ErrRelocation) {
930 t.Fatalf("fenced as exhausted with replacements left: %v", err)
931 }
932 })
933 // Round 8 finding 1: while the live source is still op.exec, the
934 // target's fence consumes the target and records stuck; the source's
935 // execution identity, phase and release evidence stay.
936 for _, phase := range []string{PhaseStarting, PhaseReleasing} {
937 t.Run("dead while the source executes in "+phase, func(t *testing.T) {
938 s, o := relAtPhase(t, phase)
939 if phase == PhaseReleasing {
940 h := o.Claim.Holder
941 o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
942 }
943 o = exhaust(t, s, o)
944 before := o.Claim
945 ref := putFence(t, s, relSpare2)
946 o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{})
947 if err != nil {
948 t.Fatal(err)
949 }
950 c := o.Claim
951 if c.Op.Phase != phase || c.Op.Exec != relSource || c.Holder.Stamp != before.Holder.Stamp || c.Op.Journal["stuck"] == "" ||
952 !slices.Contains(c.Op.Participants, relSource) || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) {
953 t.Fatalf("the live source was revoked: %+v lost %v", c.Op, c.Lost)
954 }
955 for _, k := range []string{"release", "release.phase", "releasing_since", "exec_ack." + relSource} {
956 if c.Op.Journal[k] != before.Op.Journal[k] {
957 t.Fatalf("%s changed: %q -> %q", k, before.Op.Journal[k], c.Op.Journal[k])
958 }
959 }
960 if phase == PhaseReleasing && c.Op.Journal["release"] == "" {
961 t.Fatal("the source's release evidence was dropped")
962 }
963 if l := c.EvidenceRefs[len(c.EvidenceRefs)-1]; l.Box != relSpare2 || l.Fence != ref {
964 t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs)
965 }
966 if err := c.Validate(); err != nil {
967 t.Fatal(err)
968 }
969 })
970 }
971 t.Run("dead after its local install, P unopened", func(t *testing.T) {
972 s, o := relAtPhase(t, PhasePinning)
973 o = exhaust(t, s, o)
974 o = step(t, s, o, ack(relSpare2))
975 o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P1"))
976 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7)))
977 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
978 if _, err := Update(ctx, s, o, MarkStuck("rel-1", "x")); !errors.Is(err, ErrRelocation) {
979 t.Fatalf("MarkStuck between installed and a probe: %v", err)
980 }
981 ref := putFence(t, s, relSpare2)
982 o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{})
983 if err != nil {
984 t.Fatal(err)
985 }
986 c := o.Claim
987 if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) ||
988 c.Op.Journal["local_install"] != "" || c.Op.Journal["pin"] == "" || c.Op.Journal["prepared"] != "P1" || c.Op.Journal["prepared_authority"] == "" {
989 t.Fatalf("exhausted and dead: %+v lost %v", c.Op, c.Lost)
990 }
991 if len(c.EvidenceRefs) != 3 || c.EvidenceRefs[2].Box != relSpare2 || c.EvidenceRefs[2].Fence != ref {
992 t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs)
993 }
994 if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P2")); !errors.Is(err, ErrRelocation) {
995 t.Fatalf("a stuck operation moved: %v", err)
996 }
997 if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}); err == nil {
998 t.Fatal("the fence was consumed twice")
999 }
1000 })
1001 t.Run("dead after P was opened", func(t *testing.T) {
1002 s, o := relAtPhase(t, "probing+P")
1003 o = exhaust(t, s, o) // after P's admission each replacement is a nested pinning
1004 o = step(t, s, o, ack(relSpare2))
1005 o = step(t, s, o, JournalPin("rel-1", relPin(o, 60, 9, relObserved.Add(FailoverPinDelay))))
1006 o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P3"))
1007 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P3", 60, 9)))
1008 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
1009 o = step(t, s, o, ToProbing("rel-1", relObserved))
1010 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1011 Box: relSpare2, Mode: "cold", NewGeneration: "P3", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"}))
1012 o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, relAuthority{})
1013 if err != nil {
1014 t.Fatal(err)
1015 }
1016 c := o.Claim
1017 if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || c.Op.Journal["pin"] != "" || c.Op.Journal["prepared"] != "" {
1018 t.Fatalf("nested stuck pinning: %+v", c.Op)
1019 }
1020 })
1021}
1022
1023func TestRelocationProbeFailureIsANestedRelocation(t *testing.T) {
1024 ctx := context.Background()
1025 s, o := relAtPhase(t, "probing+P")
1026 o = step(t, s, o, ToProbeFailed("rel-1", relObserved))
1027 if o.Claim.Op.Journal["probe_failed_since"] == "" {
1028 t.Fatal("probe-failed did not start its release deadline")
1029 }
1030 h := o.Claim.Holder
1031 rel := Release{ReleasedBy: relTarget, Enrollment: relEnrollment(relTarget), Generation: h.Generation, Session: h.Session, FinalTXID: 55, ManifestSeq: 9, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)}
1032 o = step(t, s, o, JournalRelease("rel-1", rel))
1033 if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relTarget, relEnrollment(relTarget)}); err == nil {
1034 t.Fatal("the failed target was named again")
1035 }
1036 o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relSpare, relEnrollment(relSpare)})
1037 if err != nil {
1038 t.Fatal(err)
1039 }
1040 if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Exec != "none" {
1041 t.Fatalf("nested relocation: %+v", o.Claim.Op)
1042 }
1043 if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 54, 9, relObserved))); !errors.Is(err, ErrRelocation) {
1044 t.Fatalf("pin lost the probe's writes: %v", err)
1045 }
1046 o = step(t, s, o, JournalPin("rel-1", relPin(o, 55, 9, relObserved)))
1047 o = step(t, s, o, ack(relSpare))
1048 o = toProbing(t, s, o, "P2", 55)
1049 // released coverage on P2's opening: the predecessor is the failed
1050 // target's probe session, whose release this operation journaled.
1051 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1052 Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
1053 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session}))
1054 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
1055 var owed []string
1056 for _, ob := range o.Claim.Obligations {
1057 owed = append(owed, ob.Box)
1058 }
1059 slices.Sort(owed)
1060 if !slices.Equal(owed, []string{relSource, relTarget}) {
1061 t.Fatalf("park-local owed by the source and the abandoned target: %v", owed)
1062 }
1063}
1064
1065func TestRelocationAdmissionRules(t *testing.T) {
1066 ctx := context.Background()
1067 probe := func(o Observed, gen, coverage string) Admission {
1068 return Admission{ObservedOpID: "rel-1", ObservedPhase: o.Claim.Op.Phase, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1069 Box: o.Claim.Op.Exec, Mode: "cold", NewGeneration: gen, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"}
1070 }
1071 for _, tc := range []struct {
1072 name, at string
1073 admit func(Observed) Admission
1074 }{
1075 {"the target admits in installing", PhaseInstalling, func(o Observed) Admission { return probe(o, "P1", "") }},
1076 {"the target admits in installed", PhaseInstalled, func(o Observed) Admission { return probe(o, "P1", "") }},
1077 {"the first probe skips P", PhaseProbing, func(o Observed) Admission { return probe(o, "other", "") }},
1078 {"P is opened twice", "probing+P", func(o Observed) Admission { return probe(o, "P1", "") }},
1079 {"released after P was opened", "probing+P", func(o Observed) Admission { return probe(o, "P3", "released") }},
1080 {"warm probe", PhaseProbing, func(o Observed) Admission {
1081 a := probe(o, "", "")
1082 a.Mode, a.PredecessorCertified = "warm", true
1083 return a
1084 }},
1085 {"the source claims released", PhaseStarting, func(o Observed) Admission { return probe(o, "g2", "released") }},
1086 } {
1087 t.Run(tc.name, func(t *testing.T) {
1088 s, o := relAtPhase(t, tc.at)
1089 if _, err := Update(ctx, s, o, AdmitWriter(tc.admit(o))); !errors.Is(err, ErrWrongState) {
1090 t.Fatalf("admitted: %v", err)
1091 }
1092 })
1093 }
1094 t.Run("released names another predecessor", func(t *testing.T) {
1095 s, o := relAtPhase(t, PhaseProbing)
1096 o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Session++
1097 if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) {
1098 t.Fatalf("admitted: %v", err)
1099 }
1100 })
1101 t.Run("released names another predecessor box", func(t *testing.T) {
1102 s, o := relAtPhase(t, PhaseProbing)
1103 o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Box = relSpare
1104 if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) {
1105 t.Fatalf("admitted: %v", err)
1106 }
1107 })
1108 t.Run("a third attempt reopens P", func(t *testing.T) {
1109 s, o := relAtPhase(t, "probing+P")
1110 o = step(t, s, o, AdmitWriter(probe(o, "P3", ""))) // attempt two: a fresh generation
1111 if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", ""))); !errors.Is(err, ErrWrongState) {
1112 t.Fatalf("P was opened again: %v", err)
1113 }
1114 })
1115 t.Run("released outside relocating", func(t *testing.T) {
1116 s, o := relServing(t)
1117 if _, err := Update(ctx, s, o, AdmitWriter(Admission{ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSource, Mode: "cold", NewGeneration: "g2", Coverage: "released", Region: "ie"})); !errors.Is(err, ErrWrongState) {
1118 t.Fatalf("admitted: %v", err)
1119 }
1120 })
1121 t.Run("the source wakes in starting", func(t *testing.T) {
1122 s, o := relAtPhase(t, PhaseStarting)
1123 if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); err != nil {
1124 t.Fatal(err)
1125 }
1126 })
1127 t.Run("a stuck source does not wake", func(t *testing.T) {
1128 s, o := relAtPhase(t, PhaseStarting)
1129 o = step(t, s, o, MarkStuck("rel-1", "operator"))
1130 if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); !errors.Is(err, ErrWrongState) {
1131 t.Fatalf("admitted: %v", err)
1132 }
1133 })
1134}
1135
1136func TestRelocationEmptyPinNeedsItsEvidence(t *testing.T) {
1137 ctx := context.Background()
1138 t.Run("a release at txid 0 vouches", func(t *testing.T) {
1139 s, o := relAtPhase(t, PhaseReleasing)
1140 h := o.Claim.Holder
1141 o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 0, ManifestSeq: 0, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
1142 o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil)
1143 if err != nil {
1144 t.Fatal(err)
1145 }
1146 o = step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved)))
1147 o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1"))
1148 if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 0))); !errors.Is(err, ErrRelocation) {
1149 t.Fatalf("an empty pin accepted a database build: %v", err)
1150 }
1151 if _, err := Update(ctx, s, o, ToInstalled("rel-1", PreparedAuthority{Generation: "P1", FilesSeq: 0})); !errors.Is(err, ErrRelocation) {
1152 t.Fatalf("an empty P without its files manifest: %v", err)
1153 }
1154 step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 0, 0)))
1155 })
1156 t.Run("a release at txid 40 does not", func(t *testing.T) {
1157 s, o := relAtPhase(t, PhaseReleasing)
1158 h := o.Claim.Holder
1159 o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
1160 o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil)
1161 if err != nil {
1162 t.Fatal(err)
1163 }
1164 if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved))); !errors.Is(err, ErrRelocation) {
1165 t.Fatalf("an empty pin against a release at 40: %v", err)
1166 }
1167 })
1168 t.Run("a failover of a host that had a writer does not", func(t *testing.T) {
1169 s, o := relServing(t)
1170 ref := putFence(t, s, relSource)
1171 o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{})
1172 if err != nil {
1173 t.Fatal(err)
1174 }
1175 o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1"))
1176 if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay)))); !errors.Is(err, ErrRelocation) {
1177 t.Fatalf("a listing that found nothing was taken as empty: %v", err)
1178 }
1179 })
1180 t.Run("a failover of a never-woken host does", func(t *testing.T) {
1181 s := release.NewDirStore(t.TempDir())
1182 o := servingClaim(t, ctx, s) // session 0: no writer was ever admitted
1183 ref := putFence(t, s, relSource)
1184 o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{})
1185 if err != nil {
1186 t.Fatal(err)
1187 }
1188 o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1"))
1189 step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay))))
1190 })
1191}
1192
1193func TestRelocationProbeAttemptsAreBounded(t *testing.T) {
1194 ctx := context.Background()
1195 s, o := relAtPhase(t, PhaseProbing)
1196 for i := 0; i < MaxProbeAttempts; i++ {
1197 o = step(t, s, o, BeginProbeAttempt("rel-1", relTarget))
1198 }
1199 if journalUint(o.Claim.Op, "probe_attempts."+relTarget) != MaxProbeAttempts {
1200 t.Fatalf("count: %v", o.Claim.Op.Journal)
1201 }
1202 if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relTarget)); !errors.Is(err, ErrProbeAttemptsExhausted) {
1203 t.Fatalf("a fourth attempt: %v", err)
1204 }
1205 if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relSpare)); !errors.Is(err, ErrRelocation) {
1206 t.Fatalf("a non-executor counted an attempt: %v", err)
1207 }
1208 // A replacement target has its own count.
1209 o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{})
1210 if err != nil {
1211 t.Fatal(err)
1212 }
1213 if journalUint(o.Claim.Op, "probe_attempts."+relSpare) != 0 {
1214 t.Fatal("the replacement inherited attempts")
1215 }
1216}
1217
1218func TestRelocationDischargeCompletesDiscoveryLikeARetirement(t *testing.T) {
1219 ctx := context.Background()
1220 s, o := relServing(t)
1221 o = step(t, s, o, func(c *Claim) error {
1222 c.Touched.DiscoveryOp = "disc"
1223 c.Obligations = append(c.Obligations, Obligation{ID: "d1", Op: "disc", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource})
1224 return nil
1225 })
1226 ref := putFence(t, s, relSource)
1227 in := relStart(ReasonFailover, &ref)
1228 in.Discharge = &LocalDischarge{Qualified: true}
1229 o, err := StartRelocation(ctx, s, o, in, relAuthority{})
1230 if err != nil {
1231 t.Fatal(err)
1232 }
1233 if len(o.Claim.Obligations) != 0 || !o.Claim.Touched.Complete || o.Claim.Touched.DiscoveryOp != "" {
1234 t.Fatalf("discharging the final discovery chore: %+v touched %+v", o.Claim.Obligations, o.Claim.Touched)
1235 }
1236}
1237
1238func TestRelocationLossRetainsAFormerWritersSession(t *testing.T) {
1239 // The source is fenced after the target's probe admission made the
1240 // target the writer: the loss still names the source's last session.
1241 ctx := context.Background()
1242 s, o := relAtPhase(t, "probing+P")
1243 before := o.Claim.Holder
1244 var sourceAdmission AdmissionRecord
1245 for _, a := range before.Admissions {
1246 if a.Box == relSource {
1247 sourceAdmission = a
1248 }
1249 }
1250 o, err := FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{})
1251 if err != nil {
1252 t.Fatal(err)
1253 }
1254 l := o.Claim.EvidenceRefs[0]
1255 if l.Box != relSource || l.Session != sourceAdmission.Session || l.Generation != sourceAdmission.Generation || l.LastAdmission == nil || !reflect.DeepEqual(*l.LastAdmission, sourceAdmission) ||
1256 l.Incarnation != before.Incarnation || l.Stamp != before.Stamp {
1257 t.Fatalf("retained: %+v, source admission %+v", l, sourceAdmission)
1258 }
1259 if err := o.Claim.Validate(); err != nil {
1260 t.Fatal(err)
1261 }
1262}
1263
1264func TestRelocationRetainsEveryFencedBoxAcrossCompletion(t *testing.T) {
1265 // A drain whose target is terminated in installing: the replacement
1266 // completes, and the dead target's loss is still on the claim.
1267 ctx := context.Background()
1268 s, o := relAtPhase(t, PhaseInstalling)
1269 ref := putFence(t, s, relTarget)
1270 o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{})
1271 if err != nil {
1272 t.Fatal(err)
1273 }
1274 o = step(t, s, o, ack(relSpare))
1275 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7)))
1276 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
1277 o = step(t, s, o, ToProbing("rel-1", relObserved))
1278 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1279 Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
1280 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session}))
1281 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
1282 c := o.Claim
1283 if len(c.EvidenceRefs) != 1 || c.EvidenceRefs[0].Box != relTarget || c.EvidenceRefs[0].Role != "target" || c.EvidenceRefs[0].Session != 0 || !slices.Contains(c.LossRefs, c.EvidenceRefs[0].ID) {
1284 t.Fatalf("the abandoned target's loss: %+v refs %v", c.EvidenceRefs, c.LossRefs)
1285 }
1286 if err := c.Validate(); err != nil {
1287 t.Fatal(err)
1288 }
1289}
1290
1291func TestRelocationStartGuard(t *testing.T) {
1292 serving := func(obs ...Obligation) *Claim {
1293 return &Claim{Holder: &Holder{Incarnation: "i"}, Op: Op{Kind: KindServing, Exec: relSource}, Obligations: obs}
1294 }
1295 for _, tc := range []struct {
1296 name string
1297 ob Obligation
1298 fenced bool
1299 want error
1300 }{
1301 {"park-generation in the region (L11)", Obligation{Kind: "park-generation", Region: "ie"}, false, nil},
1302 {"park-replica in the region", Obligation{Kind: "park-replica", Region: "ie"}, false, ErrPendingChore},
1303 {"park-replica elsewhere", Obligation{Kind: "park-replica", Region: "uk"}, false, nil},
1304 {"park-local on the target", Obligation{Kind: "park-local", Box: relTarget}, true, ErrPendingChore},
1305 {"park-local on a live source", Obligation{Kind: "park-local", Box: relSource}, false, ErrPendingChore},
1306 {"park-local on a fenced source", Obligation{Kind: "park-local", Box: relSource}, true, nil},
1307 } {
1308 t.Run(tc.name, func(t *testing.T) {
1309 if err := RelocationStartGuard(serving(tc.ob), "ie", relSource, relTarget, tc.fenced); !errors.Is(err, tc.want) {
1310 t.Fatalf("got %v, want %v", err, tc.want)
1311 }
1312 })
1313 }
1314}
1315
1316func TestRelocationLocalInstallReceiptIsBound(t *testing.T) {
1317 ctx := context.Background()
1318 for _, tc := range []struct {
1319 name string
1320 mutate func(*LocalInstall)
1321 }{
1322 {"another box", func(l *LocalInstall) { l.Box = relSpare }},
1323 {"another enrollment", func(l *LocalInstall) { l.Enrollment = relEnrollment(relSpare) }},
1324 {"a later writer stamp", func(l *LocalInstall) { l.InstallStamp++ }},
1325 } {
1326 t.Run(tc.name, func(t *testing.T) {
1327 s, o := relAtPhase(t, PhaseInstalled)
1328 l := localInstall(o.Claim)
1329 tc.mutate(&l)
1330 if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrRelocation) {
1331 t.Fatalf("recorded: %v", err)
1332 }
1333 })
1334 }
1335 s, o := relAtPhase(t, PhaseInstalled)
1336 l := localInstall(o.Claim)
1337 o = step(t, s, o, RecordLocalInstall("rel-1", l))
1338 if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrAlreadyRecorded) {
1339 t.Fatalf("replay: %v", err)
1340 }
1341 if _, err := Update(ctx, s, o, MarkStuck("rel-1", "operator")); !errors.Is(err, ErrRelocation) {
1342 t.Fatalf("stuck between installed and a probe: %v", err)
1343 }
1344 // B1-2: probing is refused on another box's receipt, even one in the journal.
1345 foreign := l
1346 foreign.Box = relSpare
1347 raw, _ := json.Marshal(foreign)
1348 o.Claim.Op.Journal["local_install"] = string(raw)
1349 if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) {
1350 t.Fatalf("probing on another box's receipt: %v", err)
1351 }
1352}
1353
1354// Round 5 finding 1: a spare that still owes a local retirement for the
1355// host is never named a replacement target, whatever phase the
1356// replacement resumes at (installing and installed hand it op.exec in the
1357// same CAS); a clean spare is.
1358func TestRelocationReplacementRefusesABoxOwingALocalRetirement(t *testing.T) {
1359 ctx := context.Background()
1360 for _, phase := range []string{PhaseStarting, PhaseInstalling, PhaseInstalled} {
1361 t.Run(phase, func(t *testing.T) {
1362 s, o := relAtPhase(t, phase)
1363 o = step(t, s, o, func(c *Claim) error {
1364 c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSpare, Retirement: "r", Owner: "box:" + relSpare})
1365 return nil
1366 })
1367 ref := putFence(t, s, relTarget)
1368 if _, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}); !errors.Is(err, ErrPendingChore) {
1369 t.Fatalf("a box owing park-local was named: %v", err)
1370 }
1371 if _, err := Update(ctx, s, o, NameTarget("rel-1", Target{relSpare, relEnrollment(relSpare)})); err == nil {
1372 t.Fatal("NameTarget named a box owing park-local")
1373 }
1374 o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare2, relEnrollment(relSpare2)}, "P2", relAuthority{})
1375 if err != nil {
1376 t.Fatal(err)
1377 }
1378 c := o.Claim
1379 if c.Op.Intent["target_box"] != relSpare2 || slices.Contains(c.Op.Participants, relSpare) || slices.Contains(c.Touched.Boxes, relSpare) {
1380 t.Fatalf("the clean spare was not chosen: %+v touched %v", c.Op, c.Touched.Boxes)
1381 }
1382 if phase != PhaseStarting && c.Op.Exec != relSpare2 {
1383 t.Fatalf("exec %s", c.Op.Exec)
1384 }
1385 })
1386 }
1387}
1388
1389// Round 5 finding 2: the automatic fence passes the dead box's regional
1390// work (park-generation, park-replica owned by box:<b>) to its region in
1391// the same CAS, as DeclareLost does for a declared loss, keeping every
1392// other field; another box's work is untouched; local discharge, qualified
1393// or not, changes nothing about it.
1394func TestRelocationFenceReassignsTheDeadBoxsRegionalWork(t *testing.T) {
1395 ctx := context.Background()
1396 gen := Obligation{ID: "pg", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0", Retirement: "rg", Owner: "box:" + relSource}
1397 rep := Obligation{ID: "pr", Op: "old-op", Kind: "park-replica", Region: "uk", Retirement: "rr", Owner: "box:" + relSource}
1398 other := Obligation{ID: "po", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0b", Retirement: "ro", Owner: "box:" + relSpare}
1399 local := Obligation{ID: "pl", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "rl", Owner: "box:" + relSource}
1400 for _, qualified := range []bool{false, true} {
1401 s, o := relServing(t)
1402 o = step(t, s, o, func(c *Claim) error {
1403 c.Obligations = append(c.Obligations, gen, rep, other, local)
1404 return nil
1405 })
1406 ref := putFence(t, s, relSource)
1407 in := relStart(ReasonFailover, &ref)
1408 in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}}
1409 o, err := StartRelocation(ctx, s, o, in, relAuthority{})
1410 if err != nil {
1411 t.Fatal(err)
1412 }
1413 byID := map[string]Obligation{}
1414 for _, ob := range o.Claim.Obligations {
1415 byID[ob.ID] = ob
1416 }
1417 wantGen, wantRep := gen, rep
1418 wantGen.Owner, wantRep.Owner = "region:ie", "region:uk"
1419 if !reflect.DeepEqual(byID["pg"], wantGen) || !reflect.DeepEqual(byID["pr"], wantRep) || !reflect.DeepEqual(byID["po"], other) {
1420 t.Fatalf("qualified=%v: regional work %+v", qualified, o.Claim.Obligations)
1421 }
1422 if _, owes := byID["pl"]; owes == qualified {
1423 t.Fatalf("qualified=%v: the local chore %+v", qualified, o.Claim.Obligations)
1424 }
1425 want := []ObligationRef{{ID: "pg", Op: "old-op", Retirement: "rg"}, {ID: "pr", Op: "old-op", Retirement: "rr"}}
1426 if len(o.Claim.EvidenceRefs) != 1 || !reflect.DeepEqual(o.Claim.EvidenceRefs[0].Reassigned, want) {
1427 t.Fatalf("qualified=%v: the retained loss does not name the reassignment: %+v", qualified, o.Claim.EvidenceRefs)
1428 }
1429 if err := o.Claim.Validate(); err != nil {
1430 t.Fatal(err)
1431 }
1432 }
1433}
1434
1435// Round 11's sweep: P's opening is journaled with the admission that made
1436// it, so it survives AdmitWriter's trim of holder.admissions to eight. The
1437// removal below stands in for that trim (generation.go drops a covered,
1438// loss-archived head once more than eight are kept); with P's admission
1439// gone from the list, P is still opened: never reopened, and the local
1440// install is never remade.
1441func TestRelocationPOpenedOutlivesTheAdmissionTrim(t *testing.T) {
1442 ctx := context.Background()
1443 s, o := relAtPhase(t, "probing+P")
1444 if o.Claim.Op.Journal["prepared_opened"] != "P1" {
1445 t.Fatalf("the opening was not journaled: %+v", o.Claim.Op.Journal)
1446 }
1447 o = step(t, s, o, func(c *Claim) error {
1448 c.Holder.Admissions = slices.DeleteFunc(c.Holder.Admissions, func(a AdmissionRecord) bool { return a.Generation == "P1" })
1449 return nil
1450 })
1451 if !PreparedOpened(o.Claim) {
1452 t.Fatal("P forgotten with its admission")
1453 }
1454 again := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1455 Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"}
1456 if _, err := Update(ctx, s, o, AdmitWriter(again)); !errors.Is(err, ErrWrongState) {
1457 t.Fatalf("P reopened after the trim: %v", err)
1458 }
1459 if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))); err == nil {
1460 t.Fatal("the local install was remade after P was opened")
1461 }
1462}
1463
1464// Round 14 finding 1: P's prepared manifest is at sequence 0 whatever the
1465// source's; the pin's sequence (7 here) is its provenance. ToInstalled
1466// refuses a prepared manifest claimed at the source's sequence, and one
1467// built from another source revision, and binds the one it accepts to
1468// the installation stamp.
1469func TestRelocationPreparedManifestIsSequenceZero(t *testing.T) {
1470 ctx := context.Background()
1471 s, o := relAtPhase(t, PhaseInstalling)
1472 var pin RecoveryPin
1473 readJournal(o.Claim.Op, "pin", &pin)
1474 if pin.FilesSeq != 7 {
1475 t.Fatalf("pin at %d", pin.FilesSeq)
1476 }
1477 atSource := relPrepared("P1", 40, 7)
1478 atSource.FilesSeq = 7
1479 if _, err := Update(ctx, s, o, ToInstalled("rel-1", atSource)); !errors.Is(err, ErrRelocation) {
1480 t.Fatalf("a prepared manifest at the source's sequence: %v", err)
1481 }
1482 if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 6))); !errors.Is(err, ErrRelocation) {
1483 t.Fatalf("a prepared manifest from another source revision: %v", err)
1484 }
1485 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7)))
1486 var a PreparedAuthority
1487 readJournal(o.Claim.Op, "prepared_authority", &a)
1488 if a.FilesSeq != 0 || a.SourceFilesSeq != 7 || a.InstallStamp == 0 || a.InstallStamp != journalUint(o.Claim.Op, "install_stamp") {
1489 t.Fatalf("journaled authority: %+v", a)
1490 }
1491}
1492
1493// Two-box model finding F3: a target replaced in installing leaves its P
1494// unopened; the replacement builds P'. The serving CAS queues a
1495// park-generation for P in the region, and none for the opened P'.
1496func TestRelocationParksEveryAbandonedPreparedGeneration(t *testing.T) {
1497 ctx := context.Background()
1498 s, o := relAtPhase(t, PhaseInstalling)
1499 o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{})
1500 if err != nil {
1501 t.Fatal(err)
1502 }
1503 var abandoned []string
1504 if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) {
1505 t.Fatalf("abandoned: %v", abandoned)
1506 }
1507 o = step(t, s, o, ack(relSpare))
1508 o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7)))
1509 o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
1510 o = step(t, s, o, ToProbing("rel-1", relObserved))
1511 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1512 Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
1513 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session}))
1514 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
1515 var parks []string
1516 for _, ob := range o.Claim.Obligations {
1517 if ob.Kind == "park-generation" && ob.Region == "ie" && ob.Owner == "region:ie" {
1518 parks = append(parks, ob.Generation)
1519 }
1520 }
1521 if !slices.Contains(parks, "P1") || slices.Contains(parks, "P2") {
1522 t.Fatalf("park-generation chores: %v (%+v)", parks, o.Claim.Obligations)
1523 }
1524 if err := o.Claim.Validate(); err != nil {
1525 t.Fatal(err)
1526 }
1527}
1528
1529// Round 17 finding 3: the target's probe deadline passes before any
1530// admission (P1 unopened), it releases in probe-failed, then it is
1531// terminated; the replacement's default branch drops P1. It is recorded
1532// first, so the completion of the nested relocation parks it.
1533func TestRelocationParksAPDroppedFromProbeFailed(t *testing.T) {
1534 ctx := context.Background()
1535 s, o := relAtPhase(t, PhaseProbing)
1536 if PreparedOpened(o.Claim) {
1537 t.Fatal("P1 was opened")
1538 }
1539 o = step(t, s, o, ToProbeFailed("rel-1", relObserved))
1540 o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{})
1541 if err != nil {
1542 t.Fatal(err)
1543 }
1544 if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Journal["prepared"] != "" {
1545 t.Fatalf("nested: %+v", o.Claim.Op)
1546 }
1547 var abandoned []string
1548 if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) {
1549 t.Fatalf("abandoned: %v", abandoned)
1550 }
1551 o = step(t, s, o, ack(relSpare))
1552 o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay))))
1553 o = toProbing(t, s, o, "P2", 40)
1554 o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
1555 Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"}))
1556 o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session}))
1557 o = step(t, s, o, CompleteRelocation("rel-1", counter()))
1558 parked := false
1559 for _, ob := range o.Claim.Obligations {
1560 if ob.Kind == "park-generation" && ob.Generation == "P1" && ob.Region == "ie" {
1561 parked = true
1562 }
1563 }
1564 if !parked {
1565 t.Fatalf("P1 was not parked: %+v", o.Claim.Obligations)
1566 }
1567}
1568```
1569
1570- [ ] **Step 2: Run them to see them fail**
1571
1572Run: `nice -n 19 go test ./internal/hostclaim/ -run 'Relocation' -count=1`
1573Expected: FAIL to compile: `undefined: RelocationStart`, `undefined: StartRelocation`, and the rest.
1574
1575- [ ] **Step 3: Write the transitions**
1576
1577`internal/hostclaim/relocation.go`:
1578
1579```go
1580package hostclaim
1581
1582import (
1583 "context"
1584 "crypto/sha256"
1585 "encoding/hex"
1586 "encoding/json"
1587 "errors"
1588 "fmt"
1589 "slices"
1590 "strconv"
1591 "time"
1592
1593 "github.com/carlosframework/platform/internal/release"
1594)
1595
1596// Relocation (claim §4 "Same-region relocation", region HA spec 1 §5.8):
1597// the reasons, the phases and the bound on target replacements.
1598const (
1599 ReasonDrain = "drain"
1600 ReasonRebalance = "rebalance"
1601 ReasonFailover = "failover"
1602
1603 PhaseStarting = "starting"
1604 PhaseReleasing = "releasing"
1605 PhasePinning = "pinning"
1606 PhaseInstalling = "installing"
1607 PhaseInstalled = "installed"
1608 PhaseProbing = "probing"
1609 PhaseProbeFailed = "probe-failed"
1610
1611 // MaxTargetReplacements is spec 1 §5.8's bound: after two
1612 // replacements of the target in one operation it goes to stuck.
1613 MaxTargetReplacements = 2
1614
1615 // MaxProbeAttempts is §5.8's probe bound: at most three admissions
1616 // per target installation within the probe deadline; running out is
1617 // probe-failed. Attempts are counted in the claim before they are
1618 // made, so a crash or a lost response never buys a fourth.
1619 MaxProbeAttempts = 3
1620
1621 // FailoverPinDelay is §5.8's failover pin rule: pin no earlier than
1622 // the fence's observed_at plus 30 seconds, so requests S3 had already
1623 // accepted from the terminated box have completed (H3's window).
1624 FailoverPinDelay = 30 * time.Second
1625)
1626
1627var (
1628 ErrRelocation = errors.New("hostclaim: relocation transition refused")
1629 ErrReplacementsExceeded = errors.New("hostclaim: relocation has used its target replacements")
1630 ErrAlreadyRecorded = errors.New("hostclaim: relocation journal entry already recorded")
1631 ErrProbeAttemptsExhausted = errors.New("hostclaim: the target has used its probe attempts")
1632)
1633
1634// RelocationStart is the start CAS's intent (spec 1 §5.8, claim §4). Fence
1635// is set for a failover only and names the source's termination fence.
1636type RelocationStart struct {
1637 OpID, By string
1638 Now time.Time
1639 Region, Reason string
1640 SourceBox, SourceEnrollment string
1641 TargetBox, TargetEnrollment string
1642 DirectoryRevision uint64
1643 AdmissionID string
1644 Fence *FenceRef
1645 Discharge *LocalDischarge
1646}
1647
1648// LocalDischarge is a fenced box's volume evidence (claim D5 as amended by
1649// L3, spec 1 §5.9, H19): the volume records the caller read and judged with
1650// boxvolumes.Discharge. When it is nil, or Qualified is false, the box's
1651// park-local chores stay open for the operator path. It never gates the
1652// fence or the relocation.
1653type LocalDischarge struct {
1654 Qualified bool `json:"qualified"`
1655 Records []string `json:"records,omitempty"`
1656 Reason string `json:"reason,omitempty"`
1657}
1658
1659// Release is the journal entry a live box writes after its release
1660// sequence (spec 1 §5.8 steps 1 to 4): its stopped database's final TXID,
1661// its final files manifest sequence and its full stop receipt.
1662type Release struct {
1663 ReleasedBy string `json:"released_by"`
1664 Enrollment string `json:"enrollment"`
1665 Generation string `json:"generation"`
1666 Session uint64 `json:"session"`
1667 FinalTXID uint64 `json:"final_txid"`
1668 ManifestSeq int64 `json:"manifest_seq"`
1669 Receipt string `json:"receipt"`
1670 ReceiptSHA256 string `json:"receipt_sha256"`
1671}
1672
1673// ReleaseAuthority verifies a journaled release's full stop receipt: the
1674// immutable object at Receipt hashes to ReceiptSHA256 and is complete stop
1675// evidence (claim §6.1, P6-D1) for this host, operation, box, incarnation
1676// and session. Plan 6b's receipt format is its only implementation.
1677type ReleaseAuthority interface {
1678 VerifyRelease(context.Context, Claim, Release) error
1679}
1680
1681// RecoveryPin is the journal's reference to one pinned recovery set
1682// (§5.8 "The pinned recovery set"): the immutable pin object holding the
1683// exact versions, its digest, the database position it reaches, the files
1684// manifest it pins, and when it was taken.
1685type RecoveryPin struct {
1686 Key string `json:"key"`
1687 SHA256 string `json:"sha256"`
1688 Generation string `json:"generation"`
1689 MaxTXID uint64 `json:"max_txid"`
1690 FilesManifest string `json:"files_manifest"`
1691 FilesSeq int64 `json:"files_seq"`
1692 PinnedAt time.Time `json:"pinned_at"`
1693 // Empty is the evidenced empty bootstrap (claim §6.4a "Empty
1694 // bootstrap", spec 1 §5.8 "A host with nothing replicated"): no LTX
1695 // file exists because no transaction was ever committed. It is legal
1696 // only when a journaled release reports final_txid 0, or the holder
1697 // never admitted a writer (session 0). A listing that merely finds
1698 // nothing is never evidence (JournalPin refuses it).
1699 Empty bool `json:"empty,omitempty"`
1700}
1701
1702// RecoveryPinKey is where the pin selected against claim generation gen
1703// lives: control/relocations/<host>/<op>/pin-<gen>.json. The key binds the
1704// object to the exact observation it was selected against, so a pin whose
1705// journal CAS lost (the claim moved) is abandoned, never reused, and a
1706// retry selects afresh at the new generation (B2-3's round binding).
1707func RecoveryPinKey(host, op string, gen uint64) string {
1708 return "control/relocations/" + host + "/" + op + "/pin-" + strconv.FormatUint(gen, 10) + ".json"
1709}
1710
1711// PreparedAuthority is the `installed` CAS's record of the remote build
1712// (§5.8 "Building the prepared generation"): the install manifest of
1713// generation P, verified complete by a fresh read.
1714type PreparedAuthority struct {
1715 Generation string `json:"generation"`
1716 InstallManifest string `json:"install_manifest"`
1717 SHA256 string `json:"sha256"`
1718 SnapshotTXID uint64 `json:"snapshot_txid"`
1719 // FilesSeq and FilesManifestSHA256 identify P's prepared files
1720 // manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): create-only,
1721 // at sequence 0, never replaced; FilesManifestSHA256 is the digest of
1722 // the exact bytes a fresh read verified, the exact version the first
1723 // admission consumes as its baseline (the session publisher then
1724 // starts at sequence 1). SourceFilesSeq is provenance, not identity:
1725 // the pinned source manifest's sequence it was built from, which must
1726 // be the pin's (rev 15, round 14 finding 1). Required always, an empty
1727 // bootstrap included: a P without its files manifest is not built.
1728 FilesSeq int64 `json:"files_seq"`
1729 SourceFilesSeq int64 `json:"source_files_seq"`
1730 FilesManifestSHA256 string `json:"files_manifest_sha256"`
1731 // InstallStamp binds the prepared manifest to the installation that
1732 // built it ({location, P, op id, installation stamp}): ToInstalled
1733 // sets it from install_stamp in the CAS that journals the authority.
1734 InstallStamp uint64 `json:"install_stamp,omitempty"`
1735}
1736
1737// AutomaticLoss is claim §7.3 D4's "Automatic loss persistence": what the
1738// transition that consumed a termination fence retains on the claim itself,
1739// outside the replaceable operation, until the console materializes the
1740// D4 host loss record. It survives CompleteRelocation, free and holder
1741// replacement because it lives beside loss_refs, not in op.journal.
1742type AutomaticLoss struct {
1743 ID string `json:"id"` // the loss identity, also in loss_refs
1744 Box string `json:"box"`
1745 Enrollment string `json:"enrollment"`
1746 Fence FenceRef `json:"fence"`
1747 ObservedAt string `json:"observed_at"`
1748 Op string `json:"op"`
1749 Kind string `json:"kind"`
1750 Phase string `json:"phase"`
1751 Role string `json:"role"` // "source" | "target"
1752 // Incarnation and Stamp bind the loss to the holder it was taken
1753 // against; Generation and Session are the box's last writer session,
1754 // whether it is still the writer or was superseded, and LastAdmission
1755 // is that session's admission record when the claim holds one (the
1756 // consequences the console reports: its coverage and declarations).
1757 Incarnation string `json:"incarnation"`
1758 Stamp uint64 `json:"stamp"`
1759 Generation string `json:"generation,omitempty"`
1760 Session uint64 `json:"session,omitempty"`
1761 LastAdmission *AdmissionRecord `json:"last_admission,omitempty"`
1762 Discharge LocalDischarge `json:"discharge"`
1763 // Reassigned is the regional work (park-generation, park-replica)
1764 // the box owned that the same CAS passed to its region (L3, as
1765 // DeclareLost's reassign_regional): the console's D4 record names it.
1766 Reassigned []ObligationRef `json:"reassigned,omitempty"`
1767 Timers TimerKnowledge `json:"timers"`
1768 // Reported is set by the console once the D4 record holds this loss
1769 // and its alert work exists. Until then reporting is pending.
1770 Reported bool `json:"reported,omitempty"`
1771}
1772
1773// automaticLossID is deterministic for one fence consumed by one
1774// operation for one box, so a replayed transition names the same loss and
1775// the console's materialization deduplicates on it (D4: "Retries
1776// deduplicate by the retained loss identity").
1777func automaticLossID(opID, box string, ref FenceRef) string {
1778 sum := sha256.Sum256([]byte(opID + "\x00" + box + "\x00" + ref.Enrollment + "\x00" + ref.SHA256))
1779 return "auto-" + hex.EncodeToString(sum[:16])
1780}
1781
1782// LocalInstall is one box's local install receipt (§5.8, B1-2, B2-5).
1783type LocalInstall struct {
1784 Box string `json:"box"`
1785 Enrollment string `json:"enrollment"`
1786 InstallStamp uint64 `json:"install_stamp"`
1787 FilesManifestVersion int64 `json:"files_manifest_version"`
1788 JournalSHA256 string `json:"journal_sha256"`
1789}
1790
1791// Probe is the target's journaled probe result for its admitted session.
1792type Probe struct {
1793 Box string `json:"box"`
1794 Session uint64 `json:"session"`
1795}
1796
1797// Target names a box for the relocation's target slot.
1798type Target struct {
1799 Box, Enrollment string
1800}
1801
1802func journalJSON(c *Claim, key string, v any) error {
1803 raw, err := json.Marshal(v)
1804 if err != nil {
1805 return err
1806 }
1807 if c.Op.Journal == nil {
1808 c.Op.Journal = map[string]string{}
1809 }
1810 c.Op.Journal[key] = string(raw)
1811 return nil
1812}
1813
1814func readJournal(op Op, key string, v any) bool {
1815 raw, ok := op.Journal[key]
1816 return ok && json.Unmarshal([]byte(raw), v) == nil
1817}
1818
1819func journalUint(op Op, key string) uint64 {
1820 n, _ := strconv.ParseUint(op.Journal[key], 10, 64)
1821 return n
1822}
1823
1824func requireRelocation(c *Claim, opID string, phases ...string) error {
1825 if err := RequireOp(opID)(c); err != nil {
1826 return err
1827 }
1828 if c.Holder == nil || c.Op.Kind != KindRelocating || !slices.Contains(phases, c.Op.Phase) {
1829 return fmt.Errorf("%w: %s/%s", ErrWrongState, c.Op.Kind, c.Op.Phase)
1830 }
1831 if c.Op.Journal["stuck"] != "" {
1832 return fmt.Errorf("%w: operation is stuck", ErrRelocation)
1833 }
1834 ensureJournal(c)
1835 return nil
1836}
1837
1838// ensureJournal makes the journal writable: an empty one round-trips as nil.
1839func ensureJournal(c *Claim) {
1840 if c.Op.Journal == nil {
1841 c.Op.Journal = map[string]string{}
1842 }
1843}
1844
1845// requireAcks is the D6 barrier: every current participant has recorded
1846// its execution acknowledgement before the operation leaves its phase.
1847func requireAcks(c *Claim) error {
1848 for _, box := range c.Op.Participants {
1849 if c.Op.Journal["exec_ack."+box] == "" {
1850 return fmt.Errorf("%w: %s has not acknowledged", ErrRelocation, box)
1851 }
1852 }
1853 return nil
1854}
1855
1856func failover(op Op) bool {
1857 return op.Intent["reason"] == ReasonFailover || op.Journal["reason"] == ReasonFailover
1858}
1859
1860// RelocationStartGuard is §4's start guard with the relocation exception
1861// (L11): a park-generation chore in the region does not block, a
1862// park-replica in the region does, a park-local on the target does, and a
1863// park-local on a fenced source does not block a failover.
1864func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced bool) error {
1865 if c.Holder == nil || c.Op.Kind != KindServing || c.Op.ID != "" {
1866 return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind)
1867 }
1868 for _, o := range c.Obligations {
1869 switch {
1870 case o.Kind == "park-replica" && o.Region == region:
1871 return fmt.Errorf("%w: park-replica in %s", ErrPendingChore, region)
1872 case o.Kind == "park-local" && o.Box == target:
1873 return fmt.Errorf("%w: park-local on %s", ErrPendingChore, target)
1874 case o.Kind == "park-local" && o.Box == source && !sourceFenced:
1875 return fmt.Errorf("%w: park-local on %s", ErrPendingChore, source)
1876 }
1877 }
1878 return nil
1879}
1880
1881// OwesLocal reports whether box owes a park-local chore on this claim: a
1882// local retirement not yet done (spec 1 §2: such a box is never chosen to
1883// install the host, and §4: a park-local on the target refuses).
1884func OwesLocal(c *Claim, box string) bool {
1885 for _, o := range c.Obligations {
1886 if o.Kind == "park-local" && o.Box == box {
1887 return true
1888 }
1889 }
1890 return false
1891}
1892
1893// LocalChoreBoxes lists every box that owes a park-local chore on the
1894// claim, for a chooser's exclusions.
1895func LocalChoreBoxes(c *Claim) []string {
1896 var out []string
1897 for _, o := range c.Obligations {
1898 if o.Kind == "park-local" && o.Box != "" && !slices.Contains(out, o.Box) {
1899 out = append(out, o.Box)
1900 }
1901 }
1902 return out
1903}
1904
1905// fenceBox puts box on lost with its fence reference (L2, L4), passes the
1906// regional work the box owned (park-generation, park-replica owned by
1907// box:<box>) to its region, region:<r>, keeping each obligation's id,
1908// operation, retirement, region, generation and park scope (L3, exactly
1909// as DeclareLost does for a declared loss, lost.go), and, when the volume
1910// evidence qualifies (L3), discharges that box's park-local chores for
1911// this host, all in the same CAS. It never removes anything else. D4: the
1912// same CAS retains the loss outside the operation (evidence_refs,
1913// loss_refs) with the box's last writer session, the discharge verdict and
1914// timer knowledge "unknown", so the console can materialize the loss record
1915// after the operation is long gone.
1916func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDischarge) error {
1917 key := "fence." + box
1918 raw, err := json.Marshal(ref)
1919 if err != nil {
1920 return err
1921 }
1922 if old, ok := c.Op.Journal[key]; ok && old != string(raw) {
1923 return fmt.Errorf("%w: fence reference already bound", ErrFence)
1924 }
1925 if c.Op.Journal == nil {
1926 c.Op.Journal = map[string]string{}
1927 }
1928 c.Op.Journal[key] = string(raw)
1929 c.Op.Journal[key+".observed_at"] = observedAt
1930 if !slices.Contains(c.Lost, box) {
1931 c.Lost = append(c.Lost, box)
1932 }
1933 var reassigned []ObligationRef
1934 for i := range c.Obligations {
1935 o := &c.Obligations[i]
1936 if (o.Kind == "park-generation" || o.Kind == "park-replica") && o.Owner == "box:"+box {
1937 if o.Region == "" {
1938 return fmt.Errorf("%w: regional obligation %q names no region", ErrWrongState, o.ID)
1939 }
1940 o.Owner = "region:" + o.Region
1941 reassigned = append(reassigned, ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement})
1942 }
1943 }
1944 if d == nil {
1945 d = &LocalDischarge{Reason: "no volume evidence supplied"}
1946 }
1947 if err := journalJSON(c, "discharge."+box, d); err != nil {
1948 return err
1949 }
1950 if d.Qualified {
1951 // Each chore is completed with CompleteObligation's own semantics,
1952 // so a discovery operation whose final chore this discharges is
1953 // marked complete exactly as a retirement would (delete.go).
1954 for {
1955 var chore *Obligation
1956 for i := range c.Obligations {
1957 if c.Obligations[i].Kind == "park-local" && c.Obligations[i].Box == box {
1958 chore = &c.Obligations[i]
1959 break
1960 }
1961 }
1962 if chore == nil {
1963 break
1964 }
1965 if err := CompleteObligation(chore.Op, chore.ID)(c); err != nil {
1966 return err
1967 }
1968 }
1969 }
1970 return retainAutomaticLoss(c, box, ref, observedAt, *d, reassigned)
1971}
1972
1973// retainAutomaticLoss is D4's retention, written once per (operation, box,
1974// fence): a replay finds its record and changes nothing.
1975func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, d LocalDischarge, reassigned []ObligationRef) error {
1976 id := automaticLossID(c.Op.ID, box, ref)
1977 for _, l := range c.EvidenceRefs {
1978 if l.ID == id {
1979 return nil
1980 }
1981 }
1982 role := "target"
1983 if box == c.Op.Intent["source_box"] {
1984 role = "source"
1985 }
1986 loss := AutomaticLoss{ID: id, Box: box, Enrollment: ref.Enrollment, Fence: ref, ObservedAt: observedAt,
1987 Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: role, Discharge: d, Reassigned: reassigned, Timers: TimersUnknown}
1988 if h := c.Holder; h != nil {
1989 loss.Incarnation, loss.Stamp = h.Incarnation, h.Stamp
1990 // The box's last writer session: the live one when it is still
1991 // the writer, otherwise its last admission on this holder.
1992 if h.WriterBox == box {
1993 loss.Generation, loss.Session = h.Generation, h.Session
1994 }
1995 for i := len(h.Admissions) - 1; i >= 0; i-- {
1996 if h.Admissions[i].Box == box {
1997 last := h.Admissions[i]
1998 loss.LastAdmission = &last
1999 if loss.Session == 0 {
2000 loss.Generation, loss.Session = last.Generation, last.Session
2001 }
2002 break
2003 }
2004 }
2005 }
2006 c.EvidenceRefs = append(c.EvidenceRefs, loss)
2007 if !slices.Contains(c.LossRefs, id) {
2008 c.LossRefs = append(c.LossRefs, id)
2009 }
2010 return nil
2011}
2012
2013func removeParticipant(c *Claim, box string) {
2014 c.Op.Participants = slices.DeleteFunc(slices.Clone(c.Op.Participants), func(b string) bool { return b == box })
2015}
2016
2017func addParticipant(c *Claim, box string) {
2018 if !slices.Contains(c.Op.Participants, box) {
2019 c.Op.Participants = append(c.Op.Participants, box)
2020 }
2021}
2022
2023// installedBoxes is every box this operation handed op.exec for an
2024// installation; each gets a park-local unless discharged (claim §4:
2025// "Every abandoned local installation receives an exact park-local").
2026func noteInstalledBox(c *Claim, box string) error {
2027 var boxes []string
2028 readJournal(c.Op, "installed_boxes", &boxes)
2029 if !slices.Contains(boxes, box) {
2030 boxes = append(boxes, box)
2031 }
2032 return journalJSON(c, "installed_boxes", boxes)
2033}
2034
2035// StartRelocation is the start CAS (spec 1 §5.8, claim §4). A drain or
2036// rebalance starts with op.exec on the source; a failover cites the
2037// source's termination fence, starts with op.exec "none", keeps the stamp,
2038// and puts the source on lost. The caller has already checked the roster
2039// (hostactors.Directory.CheckBoxStart), the target's eligibility and the
2040// app-admission registration; this CAS records what it checked.
2041func StartRelocation(ctx context.Context, s release.Store, observed Observed, in RelocationStart, authority FenceAuthority) (Observed, error) {
2042 if observed.Absent || observed.Claim == nil || observed.ETag == "" {
2043 return Observed{}, ErrWrongState
2044 }
2045 switch in.Reason {
2046 case ReasonDrain, ReasonRebalance:
2047 if in.Fence != nil {
2048 return Observed{}, fmt.Errorf("%w: only a failover cites a fence", ErrRelocation)
2049 }
2050 case ReasonFailover:
2051 if in.Fence == nil {
2052 return Observed{}, fmt.Errorf("%w: a failover cites the source's fence", ErrRelocation)
2053 }
2054 default:
2055 return Observed{}, fmt.Errorf("%w: reason %q", ErrRelocation, in.Reason)
2056 }
2057 if in.OpID == "" || in.Region == "" || in.AdmissionID == "" || in.DirectoryRevision == 0 || in.Now.IsZero() ||
2058 !validBoxReference(in.SourceBox) || !validBoxReference(in.TargetBox) || in.SourceBox == "none" || in.TargetBox == "none" ||
2059 in.SourceBox == in.TargetBox || in.SourceEnrollment == "" || in.TargetEnrollment == "" {
2060 return Observed{}, fmt.Errorf("%w: incomplete start", ErrRelocation)
2061 }
2062 var observedAt string
2063 if in.Fence != nil {
2064 v, err := VerifyFence(ctx, s, authority, *in.Fence)
2065 if err != nil {
2066 return Observed{}, err
2067 }
2068 if v.record.Box != in.SourceBox || v.record.Enrollment != in.SourceEnrollment || v.binding.Region != in.Region {
2069 return Observed{}, ErrFence
2070 }
2071 observedAt = v.record.ObservedAt
2072 }
2073 return Update(ctx, s, observed, func(c *Claim) error {
2074 if c.Op.Exec != in.SourceBox || slices.Contains(c.Lost, in.TargetBox) {
2075 return fmt.Errorf("%w: the source is not the executor, or the target is lost", ErrWrongState)
2076 }
2077 if err := RelocationStartGuard(c, in.Region, in.SourceBox, in.TargetBox, in.Fence != nil); err != nil {
2078 return err
2079 }
2080 exec, participants := in.SourceBox, []string{in.SourceBox, in.TargetBox}
2081 if in.Fence != nil {
2082 exec, participants = "none", []string{in.TargetBox}
2083 }
2084 c.Op = Op{Kind: KindRelocating, ID: in.OpID, Phase: PhaseStarting, By: in.By, StartedAt: in.Now.UTC(),
2085 Exec: exec, Participants: participants,
2086 Intent: map[string]string{
2087 "region": in.Region, "reason": in.Reason,
2088 "source_box": in.SourceBox, "source_enrollment": in.SourceEnrollment,
2089 "target_box": in.TargetBox, "target_enrollment": in.TargetEnrollment,
2090 "stamp_from": strconv.FormatUint(c.Holder.Stamp, 10),
2091 "generation_from": c.Holder.Generation,
2092 "directory_revision": strconv.FormatUint(in.DirectoryRevision, 10),
2093 "admission_id": in.AdmissionID,
2094 },
2095 Journal: map[string]string{}}
2096 if in.Fence != nil {
2097 return fenceBox(c, in.SourceBox, *in.Fence, observedAt, in.Discharge)
2098 }
2099 return nil
2100 })
2101}
2102
2103// FenceSource is §5.8's source-loss rule (B1-6): a drain or rebalance
2104// whose source is terminated becomes a failover in one CAS citing the
2105// source's fence. In starting it stays in starting with the source off the
2106// barrier; in releasing before a release is journaled it goes to pinning
2107// with the fence as evidence; after a release (pinning on) the source only
2108// goes on lost. Every case keeps the stamp and the intent.
2109func FenceSource(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) {
2110 if observed.Absent || observed.Claim == nil {
2111 return Observed{}, ErrWrongState
2112 }
2113 before := observed.Claim.Op
2114 source := before.Intent["source_box"]
2115 v, err := VerifyFence(ctx, s, authority, ref)
2116 if err != nil {
2117 return Observed{}, err
2118 }
2119 if v.record.Box != source || v.record.Enrollment != before.Intent["source_enrollment"] || v.binding.Region != before.Intent["region"] {
2120 return Observed{}, ErrFence
2121 }
2122 return Update(ctx, s, observed, func(c *Claim) error {
2123 if err := RequireOp(opID)(c); err != nil {
2124 return err
2125 }
2126 if c.Op.Kind != KindRelocating || c.Op.Intent["source_box"] != source {
2127 return ErrWrongState
2128 }
2129 ensureJournal(c)
2130 // Before its release is journaled the source's writes are not
2131 // kept by a release, so the op becomes a failover. After it, only
2132 // the source's liveness changes.
2133 if c.Op.Phase == PhaseStarting || (c.Op.Phase == PhaseReleasing && c.Op.Journal["release"] == "") {
2134 c.Op.Journal["reason"] = ReasonFailover
2135 removeParticipant(c, source)
2136 c.Op.Exec = "none"
2137 if c.Op.Phase == PhaseReleasing {
2138 c.Op.Phase = PhasePinning
2139 }
2140 }
2141 return fenceBox(c, source, ref, v.record.ObservedAt, d)
2142 })
2143}
2144
2145// ToReleasing is a drain's barrier CAS: from starting, once every
2146// participant has acknowledged. A failover never enters releasing. now
2147// starts the release deadline (hostrelocate.ReleaseDeadline): a source
2148// that never journals its release goes to stuck.
2149func ToReleasing(opID string, now time.Time) func(*Claim) error {
2150 return func(c *Claim) error {
2151 if err := requireRelocation(c, opID, PhaseStarting); err != nil {
2152 return err
2153 }
2154 if failover(c.Op) {
2155 return fmt.Errorf("%w: a failover skips releasing", ErrRelocation)
2156 }
2157 if err := requireAcks(c); err != nil {
2158 return err
2159 }
2160 if now.IsZero() {
2161 return fmt.Errorf("%w: releasing needs its start time", ErrRelocation)
2162 }
2163 c.Op.Phase = PhaseReleasing
2164 c.Op.Journal["releasing_since"] = now.UTC().Format(time.RFC3339Nano)
2165 return nil
2166 }
2167}
2168
2169// JournalRelease records a live box's completed release sequence. The
2170// source records it in releasing; a failed target in probe-failed. The
2171// release must be of the box's own current writer session.
2172func JournalRelease(opID string, r Release) func(*Claim) error {
2173 return func(c *Claim) error {
2174 if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil {
2175 return err
2176 }
2177 h := c.Holder
2178 writerMismatch := h.Session != 0 && h.WriterBox != r.ReleasedBy // a host never woken has no writer
2179 if r.ReleasedBy == "" || r.ReleasedBy != c.Op.Exec || r.Enrollment == "" || r.Receipt == "" || !isHex64(r.ReceiptSHA256) ||
2180 r.Generation != h.Generation || r.Session != h.Session || writerMismatch {
2181 return fmt.Errorf("%w: release is not the executor's current session", ErrRelocation)
2182 }
2183 want := c.Op.Intent["source_enrollment"]
2184 if c.Op.Phase == PhaseProbeFailed {
2185 want = c.Op.Intent["target_enrollment"]
2186 }
2187 if r.Enrollment != want {
2188 return fmt.Errorf("%w: release enrollment", ErrRelocation)
2189 }
2190 var old Release
2191 if readJournal(c.Op, "release", &old) && c.Op.Journal["release.phase"] == c.Op.Phase {
2192 if old == r {
2193 return ErrAlreadyRecorded
2194 }
2195 return fmt.Errorf("%w: a different release is journaled", ErrRelocation)
2196 }
2197 c.Op.Journal["release.phase"] = c.Op.Phase
2198 return journalJSON(c, "release", r)
2199 }
2200}
2201
2202// PinAfterRelease leaves releasing (a drain) or probe-failed (a failed
2203// target, §5.8's nested relocation) for pinning, citing the journaled
2204// release's verified receipt. op.exec becomes "none" and the stamp is kept
2205// (claim R6-4, D5's non-installing form). From probe-failed, next names
2206// the replacement target, which must not be the failed one.
2207func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, opID string, authority ReleaseAuthority, next *Target) (Observed, error) {
2208 if observed.Absent || observed.Claim == nil || authority == nil {
2209 return Observed{}, ErrWrongState
2210 }
2211 var r Release
2212 if !readJournal(observed.Claim.Op, "release", &r) {
2213 return Observed{}, fmt.Errorf("%w: no journaled release", ErrRelocation)
2214 }
2215 if err := authority.VerifyRelease(ctx, *observed.Claim, r); err != nil {
2216 return Observed{}, err
2217 }
2218 return Update(ctx, s, observed, func(c *Claim) error {
2219 if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil {
2220 return err
2221 }
2222 var now Release
2223 if !readJournal(c.Op, "release", &now) || now != r || c.Op.Journal["release.phase"] != c.Op.Phase || r.ReleasedBy != c.Op.Exec {
2224 return fmt.Errorf("%w: release changed", ErrRelocation)
2225 }
2226 if c.Op.Phase == PhaseReleasing {
2227 if next != nil {
2228 return fmt.Errorf("%w: a drain keeps its target", ErrRelocation)
2229 }
2230 // §5.8: a target replaced in releasing acknowledges before the
2231 // operation leaves the phase.
2232 if err := requireAcks(c); err != nil {
2233 return err
2234 }
2235 } else {
2236 if next == nil || next.Box == r.ReleasedBy {
2237 return fmt.Errorf("%w: a nested relocation names another target", ErrRelocation)
2238 }
2239 if err := replaceTargetSlot(c, *next); err != nil {
2240 return err
2241 }
2242 c.Op.Journal["released_by_target"] = r.ReleasedBy
2243 }
2244 c.Op.Exec, c.Op.Phase = "none", PhasePinning
2245 delete(c.Op.Journal, "pin")
2246 return nil
2247 })
2248}
2249
2250// PinFailover leaves starting for pinning on a failover: the source's
2251// fence is journaled and every remaining participant has acknowledged.
2252func PinFailover(opID string) func(*Claim) error {
2253 return func(c *Claim) error {
2254 if err := requireRelocation(c, opID, PhaseStarting); err != nil {
2255 return err
2256 }
2257 if !failover(c.Op) || c.Op.Journal["fence."+c.Op.Intent["source_box"]] == "" || c.Op.Exec != "none" {
2258 return fmt.Errorf("%w: pinning without a release needs the source's fence", ErrRelocation)
2259 }
2260 if err := requireAcks(c); err != nil {
2261 return err
2262 }
2263 c.Op.Phase = PhasePinning
2264 return nil
2265 }
2266}
2267
2268// JournalPin records the pinned recovery set in pinning. The pin's key
2269// must be RecoveryPinKey for this claim generation: the object was selected
2270// against exactly this observation (B2-3). After a release it must reach
2271// the journaled final TXID and files sequence; on a failover it must be
2272// taken no earlier than the fence's observed_at plus FailoverPinDelay. An
2273// Empty pin needs its evidence: a release at final_txid 0, or a holder
2274// that never admitted a writer.
2275func JournalPin(opID string, p RecoveryPin) func(*Claim) error {
2276 return func(c *Claim) error {
2277 if err := requireRelocation(c, opID, PhasePinning); err != nil {
2278 return err
2279 }
2280 if p.Key != RecoveryPinKey(c.Host, opID, c.Gen) {
2281 return fmt.Errorf("%w: pin %s was not selected against this observation (gen %d)", ErrRelocation, p.Key, c.Gen)
2282 }
2283 if !isHex64(p.SHA256) || p.FilesManifest == "" || p.PinnedAt.IsZero() || (p.Empty != (p.MaxTXID == 0)) {
2284 return fmt.Errorf("%w: incomplete pin", ErrRelocation)
2285 }
2286 var r Release
2287 released := readJournal(c.Op, "release", &r)
2288 if released {
2289 if p.MaxTXID < r.FinalTXID || p.FilesSeq < r.ManifestSeq {
2290 return fmt.Errorf("%w: pin %d/%d is short of the release's %d/%d", ErrRelocation, p.MaxTXID, p.FilesSeq, r.FinalTXID, r.ManifestSeq)
2291 }
2292 }
2293 if p.Empty && !((released && r.FinalTXID == 0) || (!released && c.Holder.Session == 0)) {
2294 return fmt.Errorf("%w: an empty pin needs a release at txid 0 or a holder that never had a writer", ErrRelocation)
2295 }
2296 for key, at := range c.Op.Journal {
2297 if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" {
2298 t, err := time.Parse(time.RFC3339Nano, at)
2299 if err != nil || p.PinnedAt.Before(t.Add(FailoverPinDelay)) {
2300 return fmt.Errorf("%w: pinned before the fence's observation plus %s", ErrRelocation, FailoverPinDelay)
2301 }
2302 }
2303 }
2304 if readJournal(c.Op, "pin", new(RecoveryPin)) {
2305 return ErrAlreadyRecorded
2306 }
2307 return journalJSON(c, "pin", p)
2308 }
2309}
2310
2311// NameTarget names the target of an operation that has none: a move
2312// converted to relocating before its flip (claim §4, moves §7.10). The
2313// named box acknowledges (D6) before ToInstalling hands it op.exec.
2314func NameTarget(opID string, t Target) func(*Claim) error {
2315 return func(c *Claim) error {
2316 if err := requireRelocation(c, opID, PhasePinning); err != nil {
2317 return err
2318 }
2319 if c.Op.Intent["target_box"] != "" {
2320 return fmt.Errorf("%w: the operation already has a target", ErrRelocation)
2321 }
2322 return replaceTargetSlot(c, t)
2323 }
2324}
2325
2326// ToInstalling hands op.exec to the target for a new installation: a new
2327// stamp (this CAS's gen, R6-4), journaled as install_stamp, a fresh
2328// prepared generation id, and the target added to touched.boxes before its
2329// first effect. A converted operation with no target names it here.
2330func ToInstalling(opID string, t Target, prepared string) func(*Claim) error {
2331 return func(c *Claim) error {
2332 if err := requireRelocation(c, opID, PhasePinning); err != nil {
2333 return err
2334 }
2335 if !readJournal(c.Op, "pin", new(RecoveryPin)) || prepared == "" || prepared == c.Holder.Generation {
2336 return fmt.Errorf("%w: installing needs a pin and a fresh prepared generation", ErrRelocation)
2337 }
2338 if t.Box != c.Op.Intent["target_box"] || t.Enrollment != c.Op.Intent["target_enrollment"] || slices.Contains(c.Lost, t.Box) {
2339 return fmt.Errorf("%w: not the operation's target", ErrRelocation)
2340 }
2341 if OwesLocal(c, t.Box) {
2342 return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box)
2343 }
2344 if err := requireAcks(c); err != nil {
2345 return err
2346 }
2347 return install(c, t.Box, prepared, PhaseInstalling)
2348 }
2349}
2350
2351// abandonPrepared journals the operation's current P as abandoned when
2352// no writer opened it, before a transition replaces or drops it: nothing
2353// will open it now, and the serving CAS parks it (rev 17, finding F3;
2354// rev 18: every transition that discards an unopened P calls it).
2355func abandonPrepared(c *Claim) error {
2356 old := c.Op.Journal["prepared"]
2357 if old == "" || preparedOpened(c) {
2358 return nil
2359 }
2360 var abandoned []string
2361 readJournal(c.Op, "abandoned_prepared", &abandoned)
2362 if slices.Contains(abandoned, old) {
2363 return nil
2364 }
2365 return journalJSON(c, "abandoned_prepared", append(abandoned, old))
2366}
2367
2368func install(c *Claim, box, prepared, phase string) error {
2369 if c.Op.Journal["prepared"] != prepared {
2370 if err := abandonPrepared(c); err != nil {
2371 return err
2372 }
2373 }
2374 stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1
2375 c.Op.Exec, c.Op.Phase = box, phase
2376 c.Holder.Stamp = stamp
2377 c.Op.Journal["install_stamp"] = strconv.FormatUint(stamp, 10)
2378 c.Op.Journal["prepared"] = prepared
2379 delete(c.Op.Journal, "local_install")
2380 if !slices.Contains(c.Touched.Boxes, box) {
2381 c.Touched.Boxes = append(c.Touched.Boxes, box)
2382 }
2383 return noteInstalledBox(c, box)
2384}
2385
2386// ToInstalled records the verified remote build of P (§5.8, the
2387// no-reinstall boundary): the install manifest at the pin's N (none for an
2388// empty pin) and, always, P's files manifest at the pinned sequence, each
2389// verified by a fresh read. Nobody builds P again after this CAS.
2390func ToInstalled(opID string, a PreparedAuthority) func(*Claim) error {
2391 return func(c *Claim) error {
2392 if err := requireRelocation(c, opID, PhaseInstalling); err != nil {
2393 return err
2394 }
2395 var pin RecoveryPin
2396 readJournal(c.Op, "pin", &pin)
2397 if a.Generation == "" || a.Generation != c.Op.Journal["prepared"] || a.SnapshotTXID != pin.MaxTXID ||
2398 (pin.MaxTXID > 0 && (a.InstallManifest == "" || !isHex64(a.SHA256))) ||
2399 (pin.MaxTXID == 0 && (a.InstallManifest != "" || a.SHA256 != "")) {
2400 return fmt.Errorf("%w: prepared authority is not this pin's build of P", ErrRelocation)
2401 }
2402 if a.FilesSeq != 0 || a.SourceFilesSeq != pin.FilesSeq || !isHex64(a.FilesManifestSHA256) {
2403 return fmt.Errorf("%w: prepared authority lacks P's prepared files manifest (sequence 0, built from the pinned sequence)", ErrRelocation)
2404 }
2405 if err := requireAcks(c); err != nil {
2406 return err
2407 }
2408 c.Op.Phase = PhaseInstalled
2409 a.InstallStamp = journalUint(c.Op, "install_stamp")
2410 return journalJSON(c, "prepared_authority", a)
2411 }
2412}
2413
2414// PreparedOpened reports whether P has been opened: journaled as
2415// prepared_opened by the admission that opened it (rev 12: AdmitWriter
2416// trims holder.admissions to eight, so the admission list alone could
2417// forget it), or named by a retained admission.
2418func PreparedOpened(c *Claim) bool { return preparedOpened(c) }
2419
2420func preparedOpened(c *Claim) bool {
2421 p := c.Op.Journal["prepared"]
2422 if p == "" || c.Holder == nil {
2423 return false
2424 }
2425 if c.Op.Journal["prepared_opened"] == p {
2426 return true
2427 }
2428 for _, a := range c.Holder.Admissions {
2429 if a.Generation == p {
2430 return true
2431 }
2432 }
2433 return false
2434}
2435
2436// RecordLocalInstall journals the executor's local install receipt in
2437// installed (B1-2). It is bound to op.exec, the target's enrollment and the
2438// journaled install_stamp, and refused once any admission has named P
2439// (B2-5): after that the local install is never remade.
2440func RecordLocalInstall(opID string, l LocalInstall) func(*Claim) error {
2441 return func(c *Claim) error {
2442 if err := requireRelocation(c, opID, PhaseInstalled); err != nil {
2443 return err
2444 }
2445 if l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] ||
2446 l.InstallStamp == 0 || l.InstallStamp != journalUint(c.Op, "install_stamp") || !isHex64(l.JournalSHA256) {
2447 return fmt.Errorf("%w: receipt is not this installation's", ErrRelocation)
2448 }
2449 if preparedOpened(c) {
2450 return fmt.Errorf("%w: P was admitted; the local install is never remade", ErrRelocation)
2451 }
2452 var old LocalInstall
2453 if readJournal(c.Op, "local_install", &old) {
2454 if old == l {
2455 return ErrAlreadyRecorded
2456 }
2457 return fmt.Errorf("%w: a different receipt is journaled", ErrRelocation)
2458 }
2459 return journalJSON(c, "local_install", l)
2460 }
2461}
2462
2463// ToProbing enters probing, which requires a local install receipt for
2464// exactly op.exec, its enrollment and the journaled install_stamp. now
2465// starts the probe deadline (§5.8: 10 minutes).
2466func ToProbing(opID string, now time.Time) func(*Claim) error {
2467 return func(c *Claim) error {
2468 if err := requireRelocation(c, opID, PhaseInstalled); err != nil {
2469 return err
2470 }
2471 var l LocalInstall
2472 if !readJournal(c.Op, "local_install", &l) || l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] ||
2473 l.InstallStamp != journalUint(c.Op, "install_stamp") {
2474 return fmt.Errorf("%w: no local install receipt for the executor", ErrRelocation)
2475 }
2476 if err := requireAcks(c); err != nil {
2477 return err
2478 }
2479 if now.IsZero() {
2480 return fmt.Errorf("%w: probing needs its start time", ErrRelocation)
2481 }
2482 c.Op.Phase = PhaseProbing
2483 c.Op.Journal["probing_since"] = now.UTC().Format(time.RFC3339Nano)
2484 return nil
2485 }
2486}
2487
2488// BeginProbeAttempt counts one probe attempt for the executor before it
2489// is made (§5.8 "limits"): the fourth is refused with
2490// ErrProbeAttemptsExhausted, and the participant then enters
2491// probe-failed. The count is per box and per operation, so a replacement
2492// target starts at zero and a fenced one is never counted again.
2493func BeginProbeAttempt(opID, box string) func(*Claim) error {
2494 return func(c *Claim) error {
2495 if err := requireRelocation(c, opID, PhaseProbing); err != nil {
2496 return err
2497 }
2498 if box == "" || box != c.Op.Exec {
2499 return fmt.Errorf("%w: only the executor probes", ErrRelocation)
2500 }
2501 key := "probe_attempts." + box
2502 n := journalUint(c.Op, key)
2503 if n >= MaxProbeAttempts {
2504 return ErrProbeAttemptsExhausted
2505 }
2506 c.Op.Journal[key] = strconv.FormatUint(n+1, 10)
2507 return nil
2508 }
2509}
2510
2511// RecordProbe journals the executor's passing probe of its admitted session.
2512func RecordProbe(opID string, p Probe) func(*Claim) error {
2513 return func(c *Claim) error {
2514 if err := requireRelocation(c, opID, PhaseProbing); err != nil {
2515 return err
2516 }
2517 if p.Box != c.Op.Exec || p.Box != c.Holder.WriterBox || p.Session == 0 || p.Session != c.Holder.Session ||
2518 p.Session < journalUint(c.Op, "install_stamp") {
2519 return fmt.Errorf("%w: probe is not of the executor's admitted session", ErrRelocation)
2520 }
2521 return journalJSON(c, "probe", p)
2522 }
2523}
2524
2525// ToProbeFailed records that the live target's probe failed or missed its
2526// deadline. The target then runs its release sequence (JournalRelease);
2527// now starts its release deadline.
2528func ToProbeFailed(opID string, now time.Time) func(*Claim) error {
2529 return func(c *Claim) error {
2530 if err := requireRelocation(c, opID, PhaseProbing); err != nil {
2531 return err
2532 }
2533 if now.IsZero() {
2534 return fmt.Errorf("%w: probe-failed needs its start time", ErrRelocation)
2535 }
2536 c.Op.Phase = PhaseProbeFailed
2537 c.Op.Journal["probe_failed_since"] = now.UTC().Format(time.RFC3339Nano)
2538 delete(c.Op.Journal, "release")
2539 delete(c.Op.Journal, "release.phase")
2540 return nil
2541 }
2542}
2543
2544// CompleteRelocation is the serving CAS: the target's probe passed for its
2545// admitted session, so the host serves on it alone. Every other box this
2546// operation installed on, and a live source, owes a park-local unless a
2547// fence discharged it.
2548func CompleteRelocation(opID string, newID func() string) func(*Claim) error {
2549 return func(c *Claim) error {
2550 if err := requireRelocation(c, opID, PhaseProbing); err != nil {
2551 return err
2552 }
2553 var p Probe
2554 if !readJournal(c.Op, "probe", &p) || p.Box != c.Op.Exec || p.Session != c.Holder.Session || c.Holder.WriterBox != p.Box {
2555 return fmt.Errorf("%w: no passing probe of the current session", ErrRelocation)
2556 }
2557 target := c.Op.Exec
2558 var owed []string
2559 readJournal(c.Op, "installed_boxes", &owed)
2560 owed = append([]string{c.Op.Intent["source_box"]}, owed...)
2561 for _, box := range owed {
2562 if box == target || box == "" {
2563 continue
2564 }
2565 var d LocalDischarge
2566 if readJournal(c.Op, "discharge."+box, &d) && d.Qualified {
2567 continue
2568 }
2569 exists := false
2570 for _, o := range c.Obligations {
2571 if o.Kind == "park-local" && o.Box == box {
2572 exists = true
2573 }
2574 }
2575 if !exists {
2576 c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: box, Retirement: newID(), Owner: "box:" + box})
2577 }
2578 }
2579 // Every prepared generation this operation built and no writer
2580 // opened is parked here, in the region, by the claim's own
2581 // park-generation chore (rev 17, finding F3); the at-rest sweep
2582 // stays the backstop the spec names.
2583 var abandoned []string
2584 readJournal(c.Op, "abandoned_prepared", &abandoned)
2585 region := c.Op.Intent["region"]
2586 for _, g := range abandoned {
2587 if g == "" || g == c.Holder.Generation || region == "" {
2588 continue
2589 }
2590 queued := false
2591 for _, o := range c.Obligations {
2592 if o.Kind == "park-generation" && o.Region == region && o.Generation == g {
2593 queued = true
2594 }
2595 }
2596 if !queued {
2597 c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-generation", Region: region, Generation: g, Retirement: newID(), Owner: "region:" + region})
2598 }
2599 }
2600 c.Op = Op{Kind: KindServing, Exec: target, Participants: []string{target}}
2601 return nil
2602 }
2603}
2604
2605// replaceTargetSlot names t as the target, counting a replacement when it
2606// replaces another box. A replaced box is never named again for this host.
2607func replaceTargetSlot(c *Claim, t Target) error {
2608 if !validBoxReference(t.Box) || t.Box == "none" || t.Enrollment == "" || t.Box == c.Op.Intent["source_box"] || slices.Contains(c.Lost, t.Box) {
2609 return fmt.Errorf("%w: invalid target", ErrRelocation)
2610 }
2611 var excluded []string
2612 readJournal(c.Op, "excluded_targets", &excluded)
2613 if slices.Contains(excluded, t.Box) {
2614 return fmt.Errorf("%w: %s was replaced in this operation", ErrRelocation, t.Box)
2615 }
2616 if OwesLocal(c, t.Box) {
2617 // Spec 1 §2: a box owing a local retirement for this host is never
2618 // handed an installation, in any phase a replacement resumes at
2619 // (install is called below without ToInstalling's check).
2620 return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box)
2621 }
2622 old := c.Op.Intent["target_box"]
2623 if old != "" && old != t.Box {
2624 n := journalUint(c.Op, "replacements")
2625 if n >= MaxTargetReplacements {
2626 return ErrReplacementsExceeded
2627 }
2628 c.Op.Journal["replacements"] = strconv.FormatUint(n+1, 10)
2629 excluded = append(excluded, old)
2630 if err := journalJSON(c, "excluded_targets", excluded); err != nil {
2631 return err
2632 }
2633 removeParticipant(c, old)
2634 }
2635 c.Op.Intent["target_box"], c.Op.Intent["target_enrollment"] = t.Box, t.Enrollment
2636 addParticipant(c, t.Box)
2637 return nil
2638}
2639
2640// ReplaceTarget is §5.8's replacement table for a target terminated by its
2641// fence. It records the fence (the old target goes on lost), names next,
2642// and resumes at the phase the table gives:
2643//
2644// starting, releasing, pinning -> the same phase
2645// installing -> installing: next is op.exec, new stamp, fresh P'
2646// installed, or probing before P -> installed: next is op.exec, new stamp, same P, no receipt
2647// probing after P, or probe-failed -> pinning: a nested failover from holder.generation
2648//
2649// prepared is P' for installing and ignored otherwise. A third
2650// replacement is refused with ErrReplacementsExceeded; the worker then
2651// marks the operation stuck.
2652func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, next Target, prepared string, authority FenceAuthority) (Observed, error) {
2653 if observed.Absent || observed.Claim == nil {
2654 return Observed{}, ErrWrongState
2655 }
2656 old := observed.Claim.Op.Intent["target_box"]
2657 v, err := VerifyFence(ctx, s, authority, ref)
2658 if err != nil {
2659 return Observed{}, err
2660 }
2661 if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] {
2662 return Observed{}, ErrFence
2663 }
2664 return Update(ctx, s, observed, func(c *Claim) error {
2665 if err := requireRelocation(c, opID, PhaseStarting, PhaseReleasing, PhasePinning, PhaseInstalling, PhaseInstalled, PhaseProbing, PhaseProbeFailed); err != nil {
2666 return err
2667 }
2668 if c.Op.Intent["target_box"] != old {
2669 return ErrStaleOp
2670 }
2671 phase := c.Op.Phase
2672 if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil {
2673 return err
2674 }
2675 if err := replaceTargetSlot(c, next); err != nil {
2676 return err
2677 }
2678 switch {
2679 case phase == PhaseStarting || phase == PhaseReleasing || phase == PhasePinning:
2680 return nil
2681 case phase == PhaseInstalling:
2682 if prepared == "" || prepared == c.Op.Journal["prepared"] || prepared == c.Holder.Generation {
2683 return fmt.Errorf("%w: a replacement in installing builds a fresh P'", ErrRelocation)
2684 }
2685 return install(c, next.Box, prepared, PhaseInstalling)
2686 case (phase == PhaseInstalled || phase == PhaseProbing) && !preparedOpened(c):
2687 return install(c, next.Box, c.Op.Journal["prepared"], PhaseInstalled)
2688 default:
2689 // After P's first admission, or from probe-failed: nested
2690 // failover from pinning, pinning from holder.generation as
2691 // observed. A P no writer opened (a probe deadline before any
2692 // admission) is recorded before it is dropped.
2693 if err := abandonPrepared(c); err != nil {
2694 return err
2695 }
2696 c.Op.Exec, c.Op.Phase = "none", PhasePinning
2697 for _, k := range []string{"pin", "prepared", "prepared_authority", "local_install", "probe", "release", "release.phase"} {
2698 delete(c.Op.Journal, k)
2699 }
2700 return nil
2701 }
2702 })
2703}
2704
2705// FenceExhaustedTarget is the terminal transition for a target terminated
2706// after the operation has used its replacements (§5.8 "The bound"; round 3
2707// finding 5): it consumes the fence (the loss is retained, the box goes on
2708// lost) and records stuck. While another box is op.exec (the live source
2709// in starting or releasing; round 8 finding 1) nothing else changes: the
2710// target's fence revokes nothing of the source's. Otherwise (op.exec is
2711// the dead target, or none) it revokes the dead installation's authority (op.exec "none", the
2712// target off the barrier, its receipt dropped), leaves the operation at
2713// pinning with its pin and, when P was never opened, its remote build kept
2714// for the operator's replacement, or a nested pinning when P was opened,
2715// and records stuck in the same CAS. No third target is chosen. MarkStuck's
2716// "never between installed and a probe" is for a live target; a dead one
2717// has no probe to make.
2718func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) {
2719 return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, "")
2720}
2721
2722// FenceTargetWithoutSpare is FenceExhaustedTarget's transition for a
2723// terminated target with replacements left but no eligible box to name
2724// (rev 18, round 17 finding 2): the worker calls it once its bound has
2725// passed. The fence is consumed in the same CAS (loss retained, the box
2726// on lost, a dead executor's authority revoked), and the operation is
2727// stuck for an operator with reason, never left waiting unbounded.
2728func FenceTargetWithoutSpare(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, reason string) (Observed, error) {
2729 if reason == "" {
2730 return Observed{}, fmt.Errorf("%w: a stuck operation names its reason", ErrRelocation)
2731 }
2732 return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, reason)
2733}
2734
2735// fenceTargetStuck is both: noSpare "" is the exhausted bound.
2736func fenceTargetStuck(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, noSpare string) (Observed, error) {
2737 if observed.Absent || observed.Claim == nil {
2738 return Observed{}, ErrWrongState
2739 }
2740 old := observed.Claim.Op.Intent["target_box"]
2741 v, err := VerifyFence(ctx, s, authority, ref)
2742 if err != nil {
2743 return Observed{}, err
2744 }
2745 if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] {
2746 return Observed{}, ErrFence
2747 }
2748 return Update(ctx, s, observed, func(c *Claim) error {
2749 if err := RequireOp(opID)(c); err != nil {
2750 return err
2751 }
2752 if c.Op.Kind != KindRelocating || c.Op.Intent["target_box"] != old {
2753 return ErrStaleOp
2754 }
2755 ensureJournal(c)
2756 if c.Op.Journal["stuck"] != "" {
2757 return ErrAlreadyRecorded
2758 }
2759 reason := noSpare
2760 if reason == "" {
2761 if journalUint(c.Op, "replacements") < MaxTargetReplacements {
2762 return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation)
2763 }
2764 reason = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements)
2765 }
2766 if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil {
2767 return err
2768 }
2769 removeParticipant(c, old)
2770 if c.Op.Exec != old && c.Op.Exec != "none" {
2771 // The executor is another box (the live source in starting or
2772 // releasing): the target's fence is no evidence against it.
2773 // Its execution identity, phase and release records stay, so
2774 // root still accepts its own stop (claimstop.go requires the
2775 // requester to be op.exec); only stuck is new.
2776 c.Op.Journal["stuck"] = reason
2777 return nil
2778 }
2779 opened := preparedOpened(c)
2780 c.Op.Exec, c.Op.Phase = "none", PhasePinning
2781 keys := []string{"local_install", "probe", "release", "release.phase", "probing_since", "probe_failed_since"}
2782 if opened {
2783 // Nested: pinning again from holder.generation, as a replacement
2784 // after P's first admission would.
2785 keys = append(keys, "pin", "prepared", "prepared_authority")
2786 }
2787 // An unopened P is kept for the operator's replacement; it is on
2788 // the abandoned list only if a later transition drops it.
2789 for _, k := range keys {
2790 delete(c.Op.Journal, k)
2791 }
2792 if c.Op.Journal["stuck"] == "" {
2793 c.Op.Journal["stuck"] = reason
2794 }
2795 return nil
2796 })
2797}
2798
2799// MarkStuck stops the operation at its current phase for an operator
2800// (`carlos ops host resolve`). It never resets the replacement count and
2801// never invents evidence; every later relocation transition refuses.
2802func MarkStuck(opID, reason string) func(*Claim) error {
2803 return func(c *Claim) error {
2804 if err := RequireOp(opID)(c); err != nil {
2805 return err
2806 }
2807 if c.Op.Kind != KindRelocating || reason == "" {
2808 return ErrWrongState
2809 }
2810 ensureJournal(c)
2811 if c.Op.Phase == PhaseInstalled && c.Op.Journal["local_install"] != "" && !preparedOpened(c) {
2812 return fmt.Errorf("%w: never stop between installed and a probe", ErrRelocation)
2813 }
2814 if c.Op.Journal["stuck"] != "" {
2815 return ErrAlreadyRecorded
2816 }
2817 c.Op.Journal["stuck"] = reason
2818 return nil
2819 }
2820}
2821```
2822
2823- [ ] **Step 4: Teach `AdmitWriter` the phase table and `released`, and the claim its retained losses**
2824
2825Apply to `internal/hostclaim/claim.go` (`git apply`): `EvidenceRefs`, D4's retained automatic losses, validated beside `loss_refs`:
2826
2827```diff
2828diff --git a/internal/hostclaim/claim.go b/internal/hostclaim/claim.go
2829index 23d17858..50e55006 100644
2830--- a/internal/hostclaim/claim.go
2831+++ b/internal/hostclaim/claim.go
2832@@ -162,6 +162,7 @@ type Claim struct {
2833 Lost []string `json:"lost,omitempty"`
2834 LostRevisions map[string]uint64 `json:"lost_revisions,omitempty"`
2835 LossRefs []string `json:"loss_refs,omitempty"`
2836+ EvidenceRefs []AutomaticLoss `json:"evidence_refs,omitempty"`
2837 RetiredIncarnations []string `json:"retired_incarnations,omitempty"`
2838 QuietAfter map[string]time.Time `json:"quiet_after,omitempty"`
2839 CancelledCreates []CancelledCreate `json:"cancelled_creates,omitempty"`
2840@@ -289,6 +290,16 @@ func (c *Claim) Validate() error {
2841 }
2842 seenLossRefs[id] = true
2843 }
2844+ seenEvidence := map[string]bool{}
2845+ for _, l := range c.EvidenceRefs {
2846+ // D4: a retained automatic loss names a loss identity the claim
2847+ // also lists, a box, and a well-formed fence reference.
2848+ if !seenLossRefs[l.ID] || seenEvidence[l.ID] || l.Box == "" || !validFenceRef(l.Fence) || l.Enrollment != l.Fence.Enrollment || l.Op == "" ||
2849+ (l.Role != "source" && l.Role != "target") || (l.Timers != TimersComplete && l.Timers != TimersPartial && l.Timers != TimersUnknown) {
2850+ return fmt.Errorf("%w: invalid automatic loss evidence", ErrMalformed)
2851+ }
2852+ seenEvidence[l.ID] = true
2853+ }
2854 if len(c.RetiredIncarnations) > retiredCap {
2855 return fmt.Errorf("%w: retired_incarnations over cap", ErrMalformed)
2856 }
2857```
2858
2859Apply to `internal/hostclaim/generation.go` (`git apply`):
2860
2861```diff
2862diff --git a/internal/hostclaim/generation.go b/internal/hostclaim/generation.go
2863index ffd7e058..74947b56 100644
2864--- a/internal/hostclaim/generation.go
2865+++ b/internal/hostclaim/generation.go
2866@@ -43,6 +43,10 @@ func AdmitWriter(in Admission) func(*Claim) error {
2867 switch c.Op.Kind {
2868 case KindCreating, KindDeleting, KindParking, KindPurging, KindPlacing:
2869 return fmt.Errorf("%w: op kind %q cannot admit a writer", ErrWrongState, c.Op.Kind)
2870+ case KindRelocating:
2871+ if err := relocationAdmission(c, in); err != nil {
2872+ return err
2873+ }
2874 }
2875 h := c.Holder
2876 if in.LossID != "" && !in.lossVerified {
2877@@ -87,6 +91,10 @@ func AdmitWriter(in Admission) func(*Claim) error {
2878
2879 switch in.Coverage {
2880 case "", "sealed", "clean-return":
2881+ case "released": // checked by relocationAdmission (L9)
2882+ if c.Op.Kind != KindRelocating {
2883+ return ErrWrongState
2884+ }
2885 default:
2886 return ErrWrongState
2887 }
2888@@ -261,3 +269,60 @@ func CoverParkedPredecessors(observedOpID, observedPhase string, region string,
2889 return nil
2890 }
2891 }
2892+
2893+// relocationAdmission is §4's phase table for relocating (L1) and §6.4a's
2894+// `released` coverage (L9). The source may admit in starting and releasing
2895+// (its local admission fence closes releasing); only op.exec admits in
2896+// probing, cold; nobody admits in any other phase. The first probe
2897+// admission opens the prepared generation P, which is opened at most once.
2898+// `released` is legal only on that opening, when the predecessor is exactly
2899+// the session whose release this operation journaled and its pin reaches
2900+// that release's final position.
2901+func relocationAdmission(c *Claim, in Admission) error {
2902+ if c.Op.Journal["stuck"] != "" {
2903+ // Ruling 9: a stuck operation runs nothing, the source's wake
2904+ // included; the operator resolves it (carlos ops host resolve).
2905+ return fmt.Errorf("%w: relocation is stuck", ErrWrongState)
2906+ }
2907+ switch c.Op.Phase {
2908+ case PhaseStarting, PhaseReleasing:
2909+ if in.Box != c.Op.Intent["source_box"] || in.Coverage == "released" {
2910+ return fmt.Errorf("%w: only the source admits before pinning", ErrWrongState)
2911+ }
2912+ return nil
2913+ case PhaseProbing:
2914+ default:
2915+ return fmt.Errorf("%w: relocating/%s admits no writer", ErrWrongState, c.Op.Phase)
2916+ }
2917+ if in.Mode != "cold" || in.Box != c.Op.Exec {
2918+ return fmt.Errorf("%w: a probe admission is the executor's, cold", ErrWrongState)
2919+ }
2920+ p, opened := c.Op.Journal["prepared"], preparedOpened(c)
2921+ if p == "" || (!opened && in.NewGeneration != p) || (opened && in.NewGeneration == p) {
2922+ return fmt.Errorf("%w: the first probe opens P, and only it", ErrWrongState)
2923+ }
2924+ if !opened {
2925+ // The opening is journaled with the admission, so it outlives the
2926+ // admission list's trim (rev 12): P is never opened twice.
2927+ ensureJournal(c)
2928+ c.Op.Journal["prepared_opened"] = p
2929+ }
2930+ if in.Coverage != "released" {
2931+ return nil
2932+ }
2933+ var r Release
2934+ var pin RecoveryPin
2935+ if opened || in.PredecessorCertified || !readJournal(c.Op, "release", &r) || !readJournal(c.Op, "pin", &pin) ||
2936+ pin.MaxTXID < r.FinalTXID || pin.FilesSeq < r.ManifestSeq {
2937+ return fmt.Errorf("%w: released coverage needs this operation's release and a pin reaching it", ErrWrongState)
2938+ }
2939+ h := c.Holder
2940+ predecessor := AdmissionRecord{Session: h.Session, Box: h.WriterBox, Generation: h.Generation}
2941+ if len(h.Admissions) > 0 {
2942+ predecessor = h.Admissions[len(h.Admissions)-1]
2943+ }
2944+ if predecessor.Box != r.ReleasedBy || predecessor.Session != r.Session || predecessor.Generation != r.Generation {
2945+ return fmt.Errorf("%w: released coverage names another predecessor", ErrWrongState)
2946+ }
2947+ return nil
2948+}
2949```
2950
2951- [ ] **Step 5: Run the tests**
2952
2953Run: `nice -n 19 go test ./internal/hostclaim/ -count=1`
2954Expected: PASS, every existing test included (`TestC7…` covers the conversion from `moving`, unchanged).
2955
2956- [ ] **Step 6: Commit**
2957
2958```bash
2959gofmt -l internal/hostclaim
2960git add internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostclaim/generation.go internal/hostclaim/claim.go
2961git commit -m "hostclaim: relocating within one region, released by stop evidence or a termination fence, with released coverage on P's one opening and the fenced box's loss retained
2962
2963Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
2964```
2965
2966---
2967
2968### Task 2: Create names its executor once (H12, L8, P5-D2, P5-D6)
2969
2970Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the roster's active, unfenced boxes in the record's placement, chosen before the create CAS and recorded in its intent as `selected_executor` and `executor_enrollment`. `CompleteCreate` refuses any other box, so recovery never substitutes. Identity, placement and AWS region come from the roster (`Roster.Regions[placement].AWSRegion`); the edges snapshot gives liveness and configuration only; claim loads only balance. A fence that cannot be read refuses the choice. `Permits` is the same read without the choice: whether one named box is in the eligible set (Task 5's start check). **Dormant:** `SelectExecutor` reports whether it selected at all; with no chooser (`ProductionAuthority.Executors` nil, which the console leaves nil unless `Config.RegionHA` is on) the create is byte-for-byte today's, `BeginCreate` and `Executor` at completion, claim-enabled one-box regions included.
2971
2972**Files:**
2973- Modify: `internal/hostclaim/create.go`, `internal/hostclaim/create_test.go`
2974- Create: `internal/hostops/executor.go`, `internal/hostops/loads.go`, `internal/hostops/executor_test.go`
2975- Modify: `internal/hostops/operations.go` (`Authority.SelectExecutor`, `Attempt`), `internal/hostops/backfill_controller.go`, `internal/hostops/backfill.go` (`ProductionAuthority.Executors`, `SelectExecutor`), `internal/hostops/operations_test.go` (the fake authority)
2976- Create: `internal/console/hostclaimexecutor.go`, `internal/console/hostclaimexecutor_test.go` (the console's chooser; the create door uses it once claim Plan 5's door lands, Task 10)
2977- Modify: `internal/console/server.go` (`Config.RegionHA`), `cmd/console/main.go` (sets it from `CARLOS_REGION_HA` with `strconv.ParseBool`)
2978- Test: `cmd/console/regionha_test.go` (rev 11: the switch itself)
2979
2980**Interfaces:**
2981- Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`; Task 1's `hostclaim.LocalChoreBoxes` (the eligible set excludes a box owing a `park-local`, rev 6).
2982- Produces: `hostclaim.ExecutorSelection{Box, Enrollment}`, `hostclaim.BeginCreateOn(nh, body, opID, by, now, sel)`; `hostops.ErrNoEligibleBox`; `hostops.ExecutorChooser{Directory, Edges func(ctx) (boxchoice.Snapshot, error), Loads func(ctx) (map[string]boxchoice.Load, error), SecretsKeyFP, HomeName, Now}` with `Choose(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)` and `Permits(ctx, hostclaim.Observed, Record, box string) error`; `hostops.ClaimLoads(ctx, release.Store) (map[string]boxchoice.Load, error)`; `Authority.SelectExecutor(ctx, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error)`; `ProductionAuthority.Executors *ExecutorChooser`; `console.Config.RegionHA`; `(*console.Server).executorChooser(hostactors.Directory) *hostops.ExecutorChooser` (nil with HA off).
2983
2984- [ ] **Step 1: Write the failing tests**
2985
2986Append to `internal/hostclaim/create_test.go`:
2987
2988```diff
2989diff --git a/internal/hostclaim/create_test.go b/internal/hostclaim/create_test.go
2990index 53968923..c9a100e6 100644
2991--- a/internal/hostclaim/create_test.go
2992+++ b/internal/hostclaim/create_test.go
2993@@ -276,3 +276,27 @@ func TestCompleteCreateRefusesExecutionAuthority(t *testing.T) {
2994 })
2995 }
2996 }
2997+
2998+func TestCreateCompletesOnlyOnTheSelectedExecutor(t *testing.T) {
2999+ ctx := context.Background()
3000+ s := release.NewDirStore(t.TempDir())
3001+ now := time.Date(2026, 10, 4, 1, 0, 1, 0, time.UTC)
3002+ reg, _ := Read(ctx, s, testHost)
3003+ sel := ExecutorSelection{Box: "bri-knitting", Enrollment: "enrollment-b"}
3004+ if _, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, ExecutorSelection{})); !errors.Is(err, ErrMalformed) {
3005+ t.Fatalf("create with no selection: %v", err)
3006+ }
3007+ o, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, sel))
3008+ if err != nil {
3009+ t.Fatal(err)
3010+ }
3011+ if o.Claim.Op.Intent["selected_executor"] != sel.Box || o.Claim.Op.Intent["executor_enrollment"] != sel.Enrollment || o.Claim.Op.Exec != "" {
3012+ t.Fatalf("intent: %+v", o.Claim.Op)
3013+ }
3014+ if _, err := Update(ctx, s, o, CompleteCreate("op_c1", "adi-knitting")); !errors.Is(err, ErrWrongState) {
3015+ t.Fatalf("completion substituted a box: %v", err)
3016+ }
3017+ if o, err = Update(ctx, s, o, CompleteCreate("op_c1", sel.Box)); err != nil || o.Claim.Op.Exec != sel.Box {
3018+ t.Fatalf("complete: %v %+v", err, o.Claim.Op)
3019+ }
3020+}
3021```
3022
3023`internal/hostops/executor_test.go`:
3024
3025```go
3026package hostops
3027
3028import (
3029 "context"
3030 "crypto/ed25519"
3031 "crypto/rand"
3032 "crypto/sha256"
3033 "encoding/hex"
3034 "encoding/json"
3035 "errors"
3036 "strings"
3037 "testing"
3038 "time"
3039
3040 "github.com/carlosframework/platform/internal/boxchoice"
3041 "github.com/carlosframework/platform/internal/hostactors"
3042 "github.com/carlosframework/platform/internal/hostclaim"
3043 "github.com/carlosframework/platform/internal/release"
3044 "github.com/carlosframework/platform/internal/termfence"
3045)
3046
3047func chooserFixture(t *testing.T, boxes ...string) (ExecutorChooser, release.Store, *boxchoice.Snapshot, map[string]boxchoice.Load) {
3048 t.Helper()
3049 s := release.NewDirStore(t.TempDir())
3050 pub, key, err := ed25519.GenerateKey(rand.Reader)
3051 if err != nil {
3052 t.Fatal(err)
3053 }
3054 roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1,
3055 Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}},
3056 Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}}
3057 snap := &boxchoice.Snapshot{Complete: true}
3058 yes := true
3059 for i, ref := range boxes {
3060 roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default",
3061 Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"}
3062 snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,
3063 HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
3064 }
3065 wire, err := release.SignRootEnvelope(release.RootSigner(key), roster)
3066 if err != nil {
3067 t.Fatal(err)
3068 }
3069 if _, err := s.Put(context.Background(), hostactors.Key, wire, ""); err != nil {
3070 t.Fatal(err)
3071 }
3072 loads := map[string]boxchoice.Load{}
3073 x := ExecutorChooser{Directory: hostactors.Directory{Store: s, PublicKey: pub, Bucket: "bucket"}, SecretsKeyFP: "fp",
3074 Edges: func(context.Context) (boxchoice.Snapshot, error) { return *snap, nil },
3075 Loads: func(context.Context) (map[string]boxchoice.Load, error) { return loads, nil },
3076 Now: func() time.Time { return time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) }}
3077 return x, s, snap, loads
3078}
3079
3080func chooserRecord() Record {
3081 return Record{Holder: hostclaim.NewHolder{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.oncarlos.com", Region: "ie"}}
3082}
3083
3084func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) {
3085 ctx := context.Background()
3086 x, _, _, loads := chooserFixture(t, "box-a", "box-b")
3087 absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
3088 first, err := x.Choose(ctx, absent, chooserRecord())
3089 if err != nil {
3090 t.Fatal(err)
3091 }
3092 other := map[string]string{"box-a": "box-b", "box-b": "box-a"}[first.Box]
3093 if first.Enrollment != map[string]string{"box-a": strings.Repeat("a", 32), "box-b": strings.Repeat("b", 32)}[first.Box] {
3094 t.Fatalf("enrollment not the roster's: %+v", first)
3095 }
3096 // The hash winner already runs this app: spread prefers the other box.
3097 loads[first.Box] = boxchoice.Load{Hosts: 1, Apps: map[string]int{"bab/jam": 1}}
3098 if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other {
3099 t.Fatalf("soft spread: got %s, want %s", got.Box, other)
3100 }
3101 // Both run the app: the fewest hosts wins.
3102 loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}}
3103 if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box {
3104 t.Fatalf("headroom: got %s, want %s", got.Box, first.Box)
3105 }
3106}
3107
3108// Round 18 finding 2: with equal loads the winner is the greatest
3109// SHA-256 of host + "\x00" + box id (claim P5-D2), computed here
3110// independently of boxchoice. The two hostnames are chosen so that their
3111// winners differ: a chooser that hashed anything but the record's host
3112// would get at least one of them wrong.
3113func TestExecutorChooserHashesTheRecordsHostOnEqualLoads(t *testing.T) {
3114 ctx := context.Background()
3115 winner := func(host string) string {
3116 a := sha256.Sum256([]byte(host + "\x00box-a"))
3117 b := sha256.Sum256([]byte(host + "\x00box-b"))
3118 if hex.EncodeToString(a[:]) > hex.EncodeToString(b[:]) {
3119 return "box-a"
3120 }
3121 return "box-b"
3122 }
3123 hosts := []string{"shop.bab.oncarlos.com", "cafe.bab.oncarlos.com"}
3124 if winner(hosts[0]) == winner(hosts[1]) {
3125 t.Fatal("the fixture's hostnames must choose opposite boxes")
3126 }
3127 for _, host := range hosts {
3128 x, _, _, _ := chooserFixture(t, "box-a", "box-b")
3129 r := chooserRecord()
3130 r.Holder.Host = host
3131 got, err := x.Choose(ctx, hostclaim.Observed{Host: host, Absent: true}, r)
3132 if err != nil || got.Box != winner(host) {
3133 t.Fatalf("%s: chose %+v %v, want %s", host, got, err, winner(host))
3134 }
3135 }
3136}
3137
3138func TestExecutorChooserNeverNamesAFencedLostOrUnknownBox(t *testing.T) {
3139 ctx := context.Background()
3140 absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
3141 t.Run("lost", func(t *testing.T) {
3142 x, _, _, _ := chooserFixture(t, "box-a")
3143 obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Lost: []string{"box-a"}}}
3144 if _, err := x.Choose(ctx, obs, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
3145 t.Fatalf("chose a lost box: %v", err)
3146 }
3147 })
3148 t.Run("owing a local retirement", func(t *testing.T) {
3149 // Spec 1 §2: a box whose old installation of the host is not yet
3150 // retired (park-local) is never chosen; the clean one is.
3151 x, _, _, _ := chooserFixture(t, "box-a", "box-b")
3152 obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Obligations: []hostclaim.Obligation{{ID: "old", Op: "old-op", Kind: "park-local", Box: "box-a", Retirement: "r", Owner: "box:box-a"}}}}
3153 for range 4 {
3154 if got, err := x.Choose(ctx, obs, chooserRecord()); err != nil || got.Box != "box-b" {
3155 t.Fatalf("got %+v %v, want the box with no chore", got, err)
3156 }
3157 }
3158 if err := x.Permits(ctx, obs, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) {
3159 t.Fatalf("permitted a box owing park-local: %v", err)
3160 }
3161 })
3162 t.Run("fenced", func(t *testing.T) {
3163 x, s, _, _ := chooserFixture(t, "box-a", "box-w")
3164 f := termfence.Record{V: 1, Box: "box-a", Enrollment: strings.Repeat("a", 32), State: "terminated",
3165 Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-box-a"},
3166 ObservedAt: "2026-10-07T08:00:00Z", Via: termfence.ViaDescribe, RequestID: "req",
3167 WrittenBy: termfence.WrittenBy{Box: "box-w", Enrollment: strings.Repeat("b", 32)}}
3168 raw, _ := json.Marshal(f)
3169 if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", raw, ""); err != nil {
3170 t.Fatal(err)
3171 }
3172 got, err := x.Choose(ctx, absent, chooserRecord())
3173 if err != nil || got.Box != "box-w" {
3174 t.Fatalf("got %+v %v, want the unfenced box", got, err)
3175 }
3176 })
3177 t.Run("an edges file for a box the roster does not hold", func(t *testing.T) {
3178 x, _, snap, _ := chooserFixture(t)
3179 yes := true
3180 snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: "stray", EIP: "203.0.113.9", Region: "eu-west-1", Placement: "ie", Fresh: true,
3181 HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
3182 if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
3183 t.Fatalf("chose an unenrolled box: %v", err)
3184 }
3185 })
3186 t.Run("an incomplete snapshot", func(t *testing.T) {
3187 x, _, snap, _ := chooserFixture(t, "box-a")
3188 snap.Complete = false
3189 if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
3190 t.Fatalf("chose from a partial read: %v", err)
3191 }
3192 })
3193 t.Run("an unreadable fence", func(t *testing.T) {
3194 x, s, _, _ := chooserFixture(t, "box-a")
3195 if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", []byte("{"), ""); err != nil {
3196 t.Fatal(err)
3197 }
3198 if _, err := x.Choose(ctx, absent, chooserRecord()); err == nil || errors.Is(err, ErrNoEligibleBox) {
3199 t.Fatalf("an unreadable fence must refuse, not skip: %v", err)
3200 }
3201 })
3202}
3203
3204func TestExecutorChooserPermitsOnlyTheEligibleSet(t *testing.T) {
3205 ctx := context.Background()
3206 absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
3207 for _, tc := range []struct {
3208 name string
3209 change func(*boxchoice.Box)
3210 }{
3211 {"a stale edges file", func(b *boxchoice.Box) { b.Fresh = false }},
3212 {"no region-ha-1 capability", func(b *boxchoice.Box) { b.Capabilities = nil }},
3213 {"another secrets key", func(b *boxchoice.Box) { b.SecretsKeyFP = "other" }},
3214 {"a draining box", func(b *boxchoice.Box) { b.Draining = true }},
3215 } {
3216 t.Run(tc.name, func(t *testing.T) {
3217 x, _, snap, _ := chooserFixture(t, "box-a", "box-b")
3218 if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); err != nil {
3219 t.Fatal(err)
3220 }
3221 tc.change(&snap.Boxes[0])
3222 if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) {
3223 t.Fatalf("permitted: %v", err)
3224 }
3225 if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil {
3226 t.Fatalf("the other box: %v", err)
3227 }
3228 })
3229 }
3230 t.Run("an unknown box", func(t *testing.T) {
3231 x, _, _, _ := chooserFixture(t, "box-a")
3232 if err := x.Permits(ctx, absent, chooserRecord(), "box-z"); !errors.Is(err, ErrNoEligibleBox) {
3233 t.Fatalf("permitted: %v", err)
3234 }
3235 })
3236}
3237
3238func TestClaimLoadsCountsExecutorsPerBoxAndApp(t *testing.T) {
3239 ctx := context.Background()
3240 s := release.NewDirStore(t.TempDir())
3241 created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC)
3242 for i, tc := range []struct{ host, app, box string }{
3243 {"a.bab.oncarlos.com", "jam", "box-a"}, {"b.bab.oncarlos.com", "jam", "box-a"}, {"c.bab.oncarlos.com", "tea", "box-b"},
3244 } {
3245 nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + tc.app + "/instances/" + tc.host + ".json", Account: "bab", App: tc.app, Host: tc.host, Region: "ie", RecordRegion: "ie", CreatedAt: created}
3246 body := []byte(`{"host":"` + tc.host + `","account":"bab","app":"` + tc.app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`)
3247 o, _ := hostclaim.Read(ctx, s, tc.host)
3248 o, err := hostclaim.Update(ctx, s, o, hostclaim.BeginCreateOn(nh, body, "c", "console", created, hostclaim.ExecutorSelection{Box: tc.box, Enrollment: "e"}))
3249 if err != nil {
3250 t.Fatal(err)
3251 }
3252 if i < 2 { // the third is still creating: it has no executor yet
3253 if _, err := hostclaim.Update(ctx, s, o, hostclaim.CompleteCreate("c", tc.box)); err != nil {
3254 t.Fatal(err)
3255 }
3256 }
3257 }
3258 if _, err := s.Put(ctx, "control/hosts/broken.bab.oncarlos.com.json", []byte("{"), ""); err != nil {
3259 t.Fatal(err)
3260 }
3261 loads, err := ClaimLoads(ctx, s)
3262 if err != nil {
3263 t.Fatal(err)
3264 }
3265 if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 0 {
3266 t.Fatalf("loads: %+v", loads)
3267 }
3268}
3269
3270func TestProductionAuthoritySelectsOnlyWithAChooser(t *testing.T) {
3271 // No chooser (CARLOS_REGION_HA off): no selection and no error, so the
3272 // create is today's.
3273 sel, selected, err := (ProductionAuthority{}).SelectExecutor(context.Background(), hostclaim.Observed{}, Record{})
3274 if err != nil || selected || sel != (hostclaim.ExecutorSelection{}) {
3275 t.Fatalf("no chooser: %+v %v %v", sel, selected, err)
3276 }
3277 // With a chooser the selection is the chooser's, read from its
3278 // evidence: box-a is draining, so box-b with its roster enrollment;
3279 // with both draining, no box and an error, never a selection.
3280 x, _, snap, _ := chooserFixture(t, "box-a", "box-b")
3281 snap.Boxes[0].Draining = true
3282 absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
3283 sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord())
3284 if err != nil || !selected || sel.Box != "box-b" || sel.Enrollment != strings.Repeat("b", 32) {
3285 t.Fatalf("with a chooser: %+v %v %v", sel, selected, err)
3286 }
3287 snap.Boxes[1].Draining = true
3288 if sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) || selected {
3289 t.Fatalf("no eligible box: %+v %v %v", sel, selected, err)
3290 }
3291}
3292```
3293
3294`internal/console/hostclaimexecutor_test.go`:
3295
3296```go
3297package console
3298
3299import (
3300 "context"
3301 "slices"
3302 "testing"
3303 "time"
3304
3305 "github.com/carlosframework/platform/internal/boxchoice"
3306 "github.com/carlosframework/platform/internal/hostactors"
3307 "github.com/carlosframework/platform/internal/hostclaim"
3308 "github.com/carlosframework/platform/internal/release"
3309)
3310
3311// The console's chooser reads the bucket through its two production
3312// readers: the live edges (each box's edges file joined with its status
3313// snapshot) and the serving claims' loads. The bucket holds real evidence,
3314// the assertions name boxes and counts, and a change to the stored
3315// evidence shows in the next read, so neither reader can be replaced by
3316// one returning nothing (round 10).
3317func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) {
3318 ctx := context.Background()
3319 store := release.NewDirStore(t.TempDir())
3320 off := &Server{cfg: Config{Releases: store, HomeRegion: "ie"}}
3321 if off.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) != nil {
3322 t.Fatal("a chooser with CARLOS_REGION_HA off: create would change")
3323 }
3324 s := &Server{cfg: Config{Releases: store, HomeRegion: "ie", RegionHA: true}}
3325 x := s.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"})
3326 if x == nil || x.Edges == nil || x.Loads == nil || x.Now == nil || x.HomeName != "ie" || x.SecretsKeyFP != s.expectedSecretsFP() {
3327 t.Fatalf("chooser: %+v", x)
3328 }
3329
3330 // Two live boxes in ie, each with its edges file and a fresh status
3331 // snapshot; box-b advertises the region-ha-1 capability.
3332 edges := func(box, extra string) string {
3333 return `{"v":1,"box":"` + box + `","box_id":"` + box + `","eip":"203.0.113.1","region":"eu-west-1","placement":"ie","hibernation_configured":true` + extra + `,"hosts":{}}`
3334 }
3335 for box, body := range map[string]string{"box-a": edges("box-a", ""), "box-b": edges("box-b", `,"capabilities":["`+boxchoice.Capability+`"]`)} {
3336 setBoxSnapshot(t, store, box, time.Now())
3337 if _, err := store.Put(ctx, "control/edges/"+box+".json", []byte(body), ""); err != nil {
3338 t.Fatal(err)
3339 }
3340 }
3341 // Two serving claims on box-a for bab/jam, one on box-b for bab/tea.
3342 created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC)
3343 serve := func(host, app, box string) {
3344 t.Helper()
3345 nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + app + "/instances/" + host + ".json", Account: "bab", App: app, Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created}
3346 body := []byte(`{"host":"` + host + `","account":"bab","app":"` + app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`)
3347 o, err := hostclaim.Read(ctx, store, host)
3348 if err != nil {
3349 t.Fatal(err)
3350 }
3351 if o, err = hostclaim.Update(ctx, store, o, hostclaim.BeginCreate(nh, body, "c", "console", created)); err != nil {
3352 t.Fatal(err)
3353 }
3354 if _, err = hostclaim.Update(ctx, store, o, hostclaim.CompleteCreate("c", box)); err != nil {
3355 t.Fatal(err)
3356 }
3357 }
3358 serve("a.bab.oncarlos.com", "jam", "box-a")
3359 serve("b.bab.oncarlos.com", "jam", "box-a")
3360 serve("c.bab.oncarlos.com", "tea", "box-b")
3361
3362 read := func() (map[string]boxchoice.Box, map[string]boxchoice.Load) {
3363 t.Helper()
3364 snap, err := x.Edges(ctx)
3365 if err != nil || !snap.Complete {
3366 t.Fatalf("edges: %+v %v", snap, err)
3367 }
3368 boxes := map[string]boxchoice.Box{}
3369 for _, b := range snap.Boxes {
3370 boxes[b.BoxID] = b
3371 }
3372 loads, err := x.Loads(ctx)
3373 if err != nil {
3374 t.Fatal(err)
3375 }
3376 return boxes, loads
3377 }
3378 boxes, loads := read()
3379 if len(boxes) != 2 || !boxes["box-a"].Fresh || !boxes["box-b"].Fresh || boxes["box-a"].Placement != "ie" ||
3380 slices.Contains(boxes["box-a"].Capabilities, boxchoice.Capability) || !slices.Contains(boxes["box-b"].Capabilities, boxchoice.Capability) {
3381 t.Fatalf("live edges: %+v", boxes)
3382 }
3383 if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 1 || loads["box-b"].Apps["bab/tea"] != 1 {
3384 t.Fatalf("loads: %+v", loads)
3385 }
3386
3387 // The evidence changes: box-a starts draining and stops reporting
3388 // (stale), and a fourth host serves on box-b. The next read says so.
3389 _, etag, err := store.Get(ctx, "control/edges/box-a.json")
3390 if err != nil {
3391 t.Fatal(err)
3392 }
3393 if _, err := store.Put(ctx, "control/edges/box-a.json", []byte(edges("box-a", `,"draining":true`)), etag); err != nil {
3394 t.Fatal(err)
3395 }
3396 setBoxSnapshot(t, store, "box-a", time.Now().Add(-time.Hour))
3397 serve("d.bab.oncarlos.com", "tea", "box-b")
3398 boxes, loads = read()
3399 if !boxes["box-a"].Draining || boxes["box-a"].Fresh || !boxes["box-b"].Fresh {
3400 t.Fatalf("live edges after the change: %+v", boxes)
3401 }
3402 if loads["box-b"].Hosts != 2 || loads["box-b"].Apps["bab/tea"] != 2 || loads["box-a"].Hosts != 2 {
3403 t.Fatalf("loads after the change: %+v", loads)
3404 }
3405}
3406```
3407
3408`cmd/console/regionha_test.go` (the switch that sets `Config.RegionHA`):
3409
3410```go
3411package main
3412
3413import "testing"
3414
3415// CARLOS_REGION_HA reaches Config.RegionHA through regionHAEnabled
3416// (strconv.ParseBool): only a true value turns the console's create-time
3417// chooser on; unset, false and garbage leave create as today.
3418func TestRegionHAEnabledParsesTheSwitch(t *testing.T) {
3419 for v, want := range map[string]bool{"": false, "0": false, "false": false, "no": false, "garbage": false,
3420 "1": true, "t": true, "true": true, "TRUE": true, " true ": true} {
3421 got := regionHAEnabled(func(k string) string {
3422 if k != "CARLOS_REGION_HA" {
3423 t.Fatalf("read %s", k)
3424 }
3425 return v
3426 })
3427 if got != want {
3428 t.Errorf("CARLOS_REGION_HA=%q: %v, want %v", v, got, want)
3429 }
3430 }
3431}
3432```
3433
3434- [ ] **Step 2: Run them to see them fail**
3435
3436Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ ./internal/console/ -run 'SelectedExecutor|ExecutorChooser|ClaimLoads|ProductionAuthoritySelects|CreateIsTodays' -count=1`
3437Expected: FAIL to compile: `undefined: ExecutorSelection`, `undefined: BeginCreateOn`, `undefined: ExecutorChooser`, `undefined: ClaimLoads`, `s.executorChooser undefined`.
3438
3439- [ ] **Step 3: Record the selection in the create intent**
3440
3441Apply to `internal/hostclaim/create.go`:
3442
3443```diff
3444diff --git a/internal/hostclaim/create.go b/internal/hostclaim/create.go
3445index c8fe97cc..39badc04 100644
3446--- a/internal/hostclaim/create.go
3447+++ b/internal/hostclaim/create.go
3448@@ -80,6 +80,32 @@ func BeginCreate(nh NewHolder, recordBody []byte, opID, by string, now time.Time
3449 }
3450 }
3451
3452+// ExecutorSelection is the box create selects once, with
3453+// boxchoice.Choose over the eligible set (claim P5-D2, region HA §5.2,
3454+// H12), and its roster enrollment.
3455+type ExecutorSelection struct {
3456+ Box, Enrollment string
3457+}
3458+
3459+// BeginCreateOn is BeginCreate recording the selected executor in the
3460+// create intent, in the same step 2 CAS, before any effect (claim §5.1,
3461+// P5-D6 `selected_executor`, `executor_enrollment`). CompleteCreate then
3462+// sets op.exec to exactly that box; recovery never substitutes another.
3463+func BeginCreateOn(nh NewHolder, recordBody []byte, opID, by string, now time.Time, sel ExecutorSelection) func(*Claim) error {
3464+ begin := BeginCreate(nh, recordBody, opID, by, now)
3465+ return func(c *Claim) error {
3466+ if !validBoxReference(sel.Box) || sel.Box == "none" || sel.Enrollment == "" {
3467+ return fmt.Errorf("%w: create needs a selected executor", ErrMalformed)
3468+ }
3469+ if err := begin(c); err != nil {
3470+ return err
3471+ }
3472+ c.Op.Intent["selected_executor"] = sel.Box
3473+ c.Op.Intent["executor_enrollment"] = sel.Enrollment
3474+ return nil
3475+ }
3476+}
3477+
3478 func CompleteCreate(opID, execBox string) func(*Claim) error {
3479 return func(c *Claim) error {
3480 if err := RequireOp(opID)(c); err != nil {
3481@@ -88,6 +114,9 @@ func CompleteCreate(opID, execBox string) func(*Claim) error {
3482 if c.Op.Kind != KindCreating || execBox == "" {
3483 return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind)
3484 }
3485+ if selected := c.Op.Intent["selected_executor"]; selected != "" && execBox != selected {
3486+ return fmt.Errorf("%w: create selected %s; completion never substitutes %s", ErrWrongState, selected, execBox)
3487+ }
3488 if err := requireNoCreateAuthority(c); err != nil {
3489 return err
3490 }
3491```
3492
3493- [ ] **Step 4: The chooser and the loads**
3494
3495`internal/hostops/executor.go`:
3496
3497```go
3498package hostops
3499
3500import (
3501 "context"
3502 "errors"
3503 "fmt"
3504 "time"
3505
3506 "github.com/carlosframework/platform/internal/boxchoice"
3507 "github.com/carlosframework/platform/internal/hostactors"
3508 "github.com/carlosframework/platform/internal/hostclaim"
3509)
3510
3511// ErrNoEligibleBox is a create refused because no box in the placement is
3512// eligible (region HA §5.2). The console maps it to today's refusal.
3513var ErrNoEligibleBox = errors.New("hostops: region has no box that can take this")
3514
3515// ExecutorChooser is create's executor selection: claim P5-D2's choose()
3516// over the complete eligible set (region HA spec 1 §5.2, H12, L8). It is
3517// made once, before the create CAS, and recorded in the create intent.
3518type ExecutorChooser struct {
3519 Directory hostactors.Directory
3520 // Edges is one read of every box's edges file with the caller's
3521 // liveness verdict per box. An incomplete snapshot selects nothing.
3522 Edges func(context.Context) (boxchoice.Snapshot, error)
3523 // Loads counts claimed hosts per box and per "acct/app", from the
3524 // claim-recovery listing (claim §5.1 step 5). A stale count only
3525 // affects balance, never safety.
3526 Loads func(context.Context) (map[string]boxchoice.Load, error)
3527 SecretsKeyFP string
3528 HomeName string
3529 Now func() time.Time
3530}
3531
3532// Choose returns the box, and its roster enrollment, that a new host in
3533// r's placement is created on. Identity, placement and AWS region come from
3534// the roster; a box with a valid termination fence, or not active, is never
3535// chosen; a box on the claim's lost list, or owing a park-local chore on
3536// it, is never chosen.
3537func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) {
3538 o, cands, err := x.eligible(ctx, obs, r)
3539 if err != nil {
3540 return hostclaim.ExecutorSelection{}, err
3541 }
3542 loads, err := x.Loads(ctx)
3543 if err != nil {
3544 return hostclaim.ExecutorSelection{}, err
3545 }
3546 best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App)
3547 if !ok {
3548 return hostclaim.ExecutorSelection{}, ErrNoEligibleBox
3549 }
3550 return hostclaim.ExecutorSelection{Box: best.BoxID, Enrollment: o.Roster.Boxes[best.BoxID].Enrollment}, nil
3551}
3552
3553// Permits reports whether box is in the complete eligible set for a host
3554// in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ...
3555// when `to` is not eligible for the host"): the same read Choose makes,
3556// without choosing. A stale edges file, a missing capability or secrets
3557// key, a draining box, a fence, or an incomplete snapshot refuses.
3558func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error {
3559 _, cands, err := x.eligible(ctx, obs, r)
3560 if err != nil {
3561 return err
3562 }
3563 for _, c := range cands {
3564 if c.BoxID == box {
3565 return nil
3566 }
3567 }
3568 return fmt.Errorf("%w: %s", ErrNoEligibleBox, box)
3569}
3570
3571// eligible is one read of the roster, the fences and the edges files,
3572// reduced to boxchoice.Eligible's candidates.
3573func (x ExecutorChooser) eligible(ctx context.Context, obs hostclaim.Observed, r Record) (hostactors.Observed, []boxchoice.Candidate, error) {
3574 placement := r.Holder.Region
3575 if x.Edges == nil || x.Loads == nil || x.Now == nil || placement == "" {
3576 return hostactors.Observed{}, nil, ErrContract
3577 }
3578 o, err := x.Directory.Read(ctx)
3579 if err != nil {
3580 return hostactors.Observed{}, nil, err
3581 }
3582 region, ok := o.Roster.Regions[placement]
3583 if !ok || region.AWSRegion == "" {
3584 return hostactors.Observed{}, nil, ErrContract
3585 }
3586 roster := map[string]boxchoice.RosterBox{}
3587 for ref, b := range o.Roster.Boxes {
3588 if b.State != "active" || b.Region != placement {
3589 continue
3590 }
3591 fenced, err := x.Directory.TerminationFence(ctx, o, ref, x.Now())
3592 if err != nil {
3593 return hostactors.Observed{}, nil, err // C6: a read error never means "not fenced"
3594 }
3595 if !fenced {
3596 roster[ref] = boxchoice.RosterBox{Placement: b.Region, Cloud: b.Cloud}
3597 }
3598 }
3599 snap, err := x.Edges(ctx)
3600 if err != nil {
3601 return hostactors.Observed{}, nil, err
3602 }
3603 // Never a box on the claim's lost list, nor one that still owes a
3604 // park-local chore for this host (spec 1 §2: its old installation is
3605 // not yet retired).
3606 excluded := map[string]bool{}
3607 if !obs.Absent && obs.Claim != nil {
3608 for _, box := range obs.Claim.Lost {
3609 excluded[box] = true
3610 }
3611 for _, box := range hostclaim.LocalChoreBoxes(obs.Claim) {
3612 excluded[box] = true
3613 }
3614 }
3615 return o, boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, excluded), nil
3616}
3617```
3618
3619`internal/hostops/loads.go`:
3620
3621```go
3622package hostops
3623
3624import (
3625 "context"
3626 "log"
3627 "strings"
3628
3629 "github.com/carlosframework/platform/internal/boxchoice"
3630 "github.com/carlosframework/platform/internal/hostclaim"
3631 "github.com/carlosframework/platform/internal/release"
3632)
3633
3634// ClaimLoads counts, from one listing of control/hosts/, the hosts whose
3635// claim names each box as op.exec, in total and per "acct/app": choose()'s
3636// second rule until spec 2 (region HA §5.2, claim P5-D2). The count only
3637// balances, so a claim that cannot be read is logged and left out; a
3638// listing that fails is an error, because a count of nothing would send
3639// every create to one box.
3640func ClaimLoads(ctx context.Context, s release.Store) (map[string]boxchoice.Load, error) {
3641 keys, err := s.List(ctx, "control/hosts/")
3642 if err != nil {
3643 return nil, err
3644 }
3645 out := map[string]boxchoice.Load{}
3646 for _, key := range keys {
3647 name, ok := strings.CutPrefix(key, "control/hosts/")
3648 host, isJSON := strings.CutSuffix(name, ".json")
3649 if !ok || !isJSON || strings.Contains(host, "/") {
3650 continue
3651 }
3652 obs, err := hostclaim.Read(ctx, s, host)
3653 if err != nil {
3654 log.Printf("hostops: claim loads: %s: %v", host, err)
3655 continue
3656 }
3657 if obs.Absent || obs.Claim.Holder == nil || obs.Claim.Op.Exec == "" || obs.Claim.Op.Exec == "none" {
3658 continue
3659 }
3660 box, h := obs.Claim.Op.Exec, obs.Claim.Holder
3661 l := out[box]
3662 if l.Apps == nil {
3663 l.Apps = map[string]int{}
3664 }
3665 l.Hosts++
3666 l.Apps[h.Account+"/"+h.App]++
3667 out[box] = l
3668 }
3669 return out, nil
3670}
3671```
3672
3673- [ ] **Step 5: Select before the create CAS, or create as today**
3674
3675Apply to `internal/hostops/operations.go`:
3676
3677```diff
3678diff --git a/internal/hostops/operations.go b/internal/hostops/operations.go
3679index 8313b80e..d3f1665a 100644
3680--- a/internal/hostops/operations.go
3681+++ b/internal/hostops/operations.go
3682@@ -14,6 +14,12 @@ import (
3683 // These callbacks are authority providers, not optional best-effort hints.
3684 // Production construction waits for the decisions named in this plan.
3685 type Authority interface {
3686+ // SelectExecutor chooses the new host's executor once, before the
3687+ // create CAS (claim P5-D2, region HA §5.2). selected false means this
3688+ // deployment does not choose at create (CARLOS_REGION_HA off): the
3689+ // create proceeds exactly as today, and Executor names the box at
3690+ // completion. Production is ExecutorChooser.Choose behind that switch.
3691+ SelectExecutor(context.Context, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error)
3692 Create(context.Context, hostclaim.Observed, Record, string) error
3693 Executor(context.Context, hostclaim.Observed, Record) (string, error)
3694 StartDelete(context.Context, hostclaim.Observed, Record) error
3695@@ -46,11 +52,21 @@ func (c Controller) Attempt(ctx context.Context, before hostclaim.Observed, r Re
3696 if before.Host != r.Holder.Host || r.Holder.CreatedAt.IsZero() || id == "" {
3697 return hostclaim.Observed{}, ErrIdentity
3698 }
3699+ // With region HA on, the executor is selected once, before the CAS,
3700+ // and recorded in it; off, the create is today's.
3701+ sel, selected, err := c.Authority.SelectExecutor(ctx, before, r)
3702+ if err != nil {
3703+ return hostclaim.Observed{}, err
3704+ }
3705+ begin := hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now())
3706+ if selected {
3707+ begin = hostclaim.BeginCreateOn(r.Holder, r.Body, id, by, c.Now(), sel)
3708+ }
3709 // Last pre-CAS read of registration, open admission and purge reservation.
3710 if err := c.Authority.Create(ctx, before, r, id); err != nil {
3711 return hostclaim.Observed{}, err
3712 }
3713- obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now()))
3714+ obs, err := hostclaim.Update(ctx, c.Store, before, begin)
3715 if err != nil {
3716 return hostclaim.Observed{}, err
3717 }
3718```
3719
3720Apply to `internal/hostops/backfill_controller.go` (the backfill grants no create):
3721
3722```diff
3723diff --git a/internal/hostops/backfill_controller.go b/internal/hostops/backfill_controller.go
3724index 08ebd856..dcbf750e 100644
3725--- a/internal/hostops/backfill_controller.go
3726+++ b/internal/hostops/backfill_controller.go
3727@@ -23,6 +23,9 @@ func (a BackfillAuthority) Adoption(ctx context.Context, r Record) (string, stri
3728 }
3729 func (a BackfillAuthority) ClaimBackfill() *Backfill { return a.p.Backfill }
3730
3731+func (a BackfillAuthority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, bool, error) {
3732+ return hostclaim.ExecutorSelection{}, false, nil // the backfill grants no create (Create refuses)
3733+}
3734 func (a BackfillAuthority) Create(context.Context, hostclaim.Observed, Record, string) error {
3735 return ErrContract
3736 }
3737```
3738
3739Apply to `internal/hostops/backfill.go`:
3740
3741```diff
3742diff --git a/internal/hostops/backfill.go b/internal/hostops/backfill.go
3743index e20b1982..7c727f9d 100644
3744--- a/internal/hostops/backfill.go
3745+++ b/internal/hostops/backfill.go
3746@@ -89,6 +89,22 @@ type ProductionAuthority struct {
3747 Signer release.Signer
3748 Evidence hostactors.Evidence
3749 Backfill *Backfill
3750+ // Executors is create's executor selection (region HA §5.2, L8). The
3751+ // create door sets it only with CARLOS_REGION_HA on; nil keeps today's
3752+ // create path, so a c