amadan
docs: region HA stage B plan rev 5.1, Astra round 6
Fixes the four P2s the way the review directs. The probe stop's activator fence is owned (activator FenceAndStopAs/LiftFenceAs): the owner is persisted before the fence goes in, so a failed marker write installs nothing and a retry proceeds, and a marker that survives a restart can never take over another operation's fence (rulings 75, 76). A successor relocation cannot start from a box that has not yet discharged its completed relocation: the target's hold release ends by marking it, and RelocationStartGuard waits for the mark (ruling 77). The landing conversion runs as Worker.ConvertLanding under Step's own worker checks (ruling 78). Task 10b is named and mandatory before activation, and checklist item 18 says so (ruling 79). Each task was recompiled and retested in order in a throwaway copy, and the plan's code was extracted back out and matched the checked commits exactly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+1230
−311
docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md+1230 −311
diff --git a/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md b/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md| index 2b98444eb..7bba0ab83 100644 |
| --- a/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md |
| +++ b/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md |
| @@ -1,4 +1,4 @@ |
| -# Region HA 1b-next Stage B: Relocation Wired Into the Claim Assembly Implementation Plan (rev 5) |
| +# Region HA 1b-next Stage B: Relocation Wired Into the Claim Assembly Implementation Plan (rev 5.1) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| @@ -8,12 +8,25 @@ |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store`; `hostclaim`, `hostactors`, `hostfence`, `hostguard`, `hostexec`, `hostbox`, `hostrelocate`, `hostfiles`, `genselect`, `moves`, `lease`, `activator`, `registry`, `appconfig`, `edgesfile`, `instancekey`, `release`. |
| -**Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md` (rev 5.4 with §24): §5.3, §5.8, §6, §8. The 1b-next plan's rulings 1 to 27 stand; this plan's rulings continue the numbering at 28. Claim Plan 11 (`docs/superpowers/plans/2026-10-09-host-lifecycle-claim-11-region-ha-integration.md` on `origin/design-replica-ownership`, "Region HA integration and rollout", items 1 to 7) and the #236 answers (R1 to R10, C1 to C5, M1 to M3, J2, J3) are the contracts. Rev 5's code was compiled and run on this branch at b9a753f0 with `internal/instancekey` from `instance-key-build` Task 1 as built (e8487c3a; its API is unchanged since); file:line cites are to this branch. |
| +**Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md` (rev 5.4 with §24): §5.3, §5.8, §6, §8. The 1b-next plan's rulings 1 to 27 stand; this plan's rulings continue the numbering at 28. Claim Plan 11 (`docs/superpowers/plans/2026-10-09-host-lifecycle-claim-11-region-ha-integration.md` on `origin/design-replica-ownership`, "Region HA integration and rollout", items 1 to 7) and the #236 answers (R1 to R10, C1 to C5, M1 to M3, J2, J3) are the contracts. Rev 5.1's code was compiled and run on this branch at b9a753f0 with `internal/instancekey` from `instance-key-build` Task 1 as built (e8487c3a; its API is unchanged since); file:line cites are to this branch. |
| **Rev 5 in one paragraph.** Rev 5 is the build-ready consolidation of rev 4.2 against the pre-flight scan (`.superpowers/sdd/2026-10-10-region-ha-1b-next-stage-b/preflight-scan.md`, K1 to K25 and D1 to D9). Every task's code was written out, compiled and run task by task in a throwaway copy, and each task's text below is generated from its own commit there: its Files, its tests, its `-run` filter and its `git add` match its code. No task depends on a later task. Nothing in a task refers to a revision section; the revision layers (rev 2 to 4.2) are folded into the tasks, and their disposition tables are kept below as history. Every test has a body that asserts behaviour, and every function has its body. C10 is committed in Task 9 as region HA code behind the activation gate. Four promised tests are withdrawn with reasons (R5-2, R5-4), and one rev 1 change is withdrawn (R5-1); they are listed under "Unresolved in rev 5". |
| +**Rev 5.1 in one paragraph.** Astra's round 6 on rev 5 found four P2s and no P1, accepted R5-1, R5-3 (with Task 10b mandatory before activation) and R5-4, and judged R5-2's replacement coverage insufficient for a successor relocation. Rev 5.1 fixes each finding the way the review directs: the probe stop's activator fence is owned and checked under the activator's lock (findings 1 and 2, rulings 75 and 76), a successor cannot start from a box that has not discharged its completed relocation (finding 3, ruling 77), and the landing conversion runs under the worker's own checks (finding 4, ruling 78). Task 10b is named and mandatory before activation (ruling 79). Every task was rebuilt and rerun in order (the record below). |
| + |
| --- |
| +## Rev 5.1: Astra round 6 |
| + |
| +| # | Finding | Fix in rev 5.1 | Where | |
| +|---|---|---|---| |
| +| 1 (P2) | The probe stop's marker recorded only the session. After a restart it survived while the activator's fence did not, so it could authorize `FenceAndStop` over another operation's lower or equal fence, which completion then lifted. | The fence is owned (ruling 75). The activator gains `FenceAndStopAs(ctx, host, generation, owner)` and `LiftFenceAs(host, generation, owner)`: under the lock that installs, a held fence is the caller's only when `FenceAndStopAs` installed it with that owner at that generation, and anything else is `ErrFenceHeld` with nothing stopped. Fences live only in the activator's process, so a marker that survives a restart matches no fence. `RelocationActivator` is `ProbeWake`, `FenceAndStopAs` and `LiftFenceAs`; plain `FenceAndStop`, `FenceAndStopIfUnfenced` and `LiftFence` are no longer reachable from relocation. Tests: `TestFenceAndStopAsMeetsOnlyItsOwnFence`, `TestAPlainFenceTakesOwnershipAway`, `TestLiftFenceAsLiftsOnlyTheOwnersFence` (activator); `TestRelocationStopProbeAfterARestartNeverTakesAForeignFence` (a restart, then a foreign fence at 5 and at 12: `ErrFenceHeld`, the fence as it was, root not asked, then the stop completes once the other fence goes); `TestRelocationReleaseHoldWaitsForDischarge`'s restart case (the foreign fence is never lifted). | 8a (activator, interface), 9 | |
| +| 2 (P2) | The fence went in before its marker was written; a failed write left every retry at `ErrFenceHeld` with no marker, stuck. | The owner is persisted first (`probe-fence.json`: host, session, owner), then `FenceAndStopAs` is asked. A failed write installs nothing, and the next pass writes it again and proceeds. A retry after the fence went in meets its own fence under the persisted owner. The journal sweep keeps the op's directory until the completion is marked discharged (ruling 76), so the lift always finds the owner. Tests: `TestRelocationStopProbeRetriesAFailedMarkerWrite` (the first write fails: nothing asked of the activator, no fence; the retry persists, fences under that owner and stops), `TestRelocationJournalsOfACompletedOpStayUntilItsDischarge`. | 8a (sweep), 9 | |
| +| 3 (P2) | A to B completes, then B to C starts before B's C10 discharge; B's slot is still the old op's, so main's `Prepare` answers `ErrLost` for the successor for good. R5-2's replacements had no successor test. | Prevented (ruling 77). `CompletedRelocation` gains `Discharged`, set by `hostclaim.MarkCompletedDischarged(op)`. `RelocationStartGuard` refuses (`ErrPendingChore`) while the serving op's completed relocation is undischarged, unless the start is a failover citing the completion box's fence. The target's hold release ends by marking it (ruling 76: discharge, release the hold, lift the owned fence, mark). Tests through the real start CAS: `TestRelocationSuccessorWaitsForTheCompletedDischarge` (hostclaim: A to B through every real transition, then B to C refused for drain and rebalance with the claim unchanged, allowed after the mark, and a failover from a fenced B allowed at once), `TestAParticipantFinishesItsControllerOnceTheOperationIsOver` (hostrelocate: `Worker.Start` refuses the third relocation until the second target's Sweep finished it), `TestRelocationSuccessorStartsOnlyAfterTheTargetsDischarge` (host: the real hold release, refused then succeeding, gating `StartRelocation`). | 8a (claim, sweep), 9 | |
| +| 4 (P2) | `convertFencedLanding` never checked the converting worker's enrollment, status or region. | `hostrelocate.Worker.ConvertLanding` (ruling 78) runs `Worker.self` first, exactly as `Step` does (the roster entry is this box, active, at the enrollment it runs as, in its region, unfenced; else `ErrNotWorker`), then leaves alone a landing whose `to` region is not the worker's. It replaces `Worker.Fence`; `convertFencedLanding` only calls it. Tests: `TestRelocatePassConversionRefusesAnUnauthorizedWorker` (a revoked enrollment, a retired worker, a worker enrolled in another region, a fenced worker: `ErrNotWorker`; a worker of another region: left alone; the claim's ETag unchanged in every case). | 10 | |
| +| R5-3 | Accepted while dormant, on condition that Task 10b is mandatory before activation and checklist item 18 drops "if wired". | Task 10b is named after Task 10 and is a gate in the activation line, the checklist (item 18) and "After this plan" (ruling 79). | 10b | |
| +| R5-2 | Replacement coverage insufficient (finding 3). | The successor case is prevented and tested through the real start transitions (finding 3). The ported lifecycle property stays withdrawn. | 8a, 9 | |
| + |
| ## Rev 5: pre-flight scan |
| Each row is the scan's finding and rev 5's fix, verified in the task-by-task build. |
| @@ -38,7 +51,7 @@ Each row is the scan's finding and rev 5's fix, verified in the task-by-task bui |
| | K16 **B** | Task 10 called `relocationAssemblyConfig` with six arguments; it took seven. | One signature, called once: `relocationAssemblyConfig(a, providers, stores, reg, act, directory, optDir, discharge)`. | 10 step 5 | |
| | K17 | N5's site was the call at claimproduction.go:150, outside the gate. | N5 is in `openClaimRelocationOwners` (claimrelocation.go), inside its region HA gate, before `source.Open`. | 10 step 6 | |
| | K18 **B** | 8d's "complete" `relocationowners.go` held Task 9's `relocationStopProbe` and `relocationReleaseHold`. | Both are Task 9's. | 9 step 5 | |
| -| K19 | The spec expected `ErrFenceHeld` from `FenceAndStop`, which never returns it. | `relocationStopProbe` takes the fence with `FenceAndStopIfUnfenced`: another operation's fence is `ErrFenceHeld`, with nothing asked of root. A repeat of this probe's own stop finds its fence recorded under `relocate/ops/<op>/probe-fence.json` and re-collects with `FenceAndStop` at the same generation. `RelocationActivator` gains `FenceAndStopIfUnfenced`. Ruling 71. | 9 step 5 | |
| +| K19 | The spec expected `ErrFenceHeld` from `FenceAndStop`, which never returns it. | `relocationStopProbe` takes the fence with `FenceAndStopIfUnfenced`: another operation's fence is `ErrFenceHeld`, with nothing asked of root. A repeat of this probe's own stop finds its fence recorded under `relocate/ops/<op>/probe-fence.json` and re-collects with `FenceAndStop` at the same generation. `RelocationActivator` gains `FenceAndStopIfUnfenced`. Ruling 71. Superseded in rev 5.1 by the owned fence (ruling 75). | 9 step 5 | |
| | K20 | Task 10 step 9's heading was duplicated. | One step. | 10 | |
| | K21 | The self-review said `MintHeld` is called by 8c and 9. | 8c only. | Self-review | |
| | K22 | The obligations table cited `releaseDrain.resumed`, which no longer exists. | The row says what holds: main's producers' drain, called at most once per stop by the release (the retry after root's failure resumes at the producers, `TestReleaseNeverReturnsBeforeRootsReceipt`). | Obligations | |
| @@ -52,7 +65,7 @@ Each row is the scan's finding and rev 5's fix, verified in the task-by-task bui |
| | D5 | Duplicated logic: (a) C10 twice in the document; (b) the install Selection twice; (c) the op-id check twice; (d) the acknowledgement binding check twice; (e) the writer-session receipt scan twice; (f) K20. | (a) C10 once, in Task 9's code; (b) `relocationInstallLocal`; (c) `validOpID`; (d) `RelocationEffects.Acknowledge` only; (e) `writerSessionReceipt`; (f) as K20. | 8a, 8b, 9 | |
| | D6 | `relocationFence` only called `writerFence`. | Removed; the inputs call `writerFence`. | 8d | |
| | D7 | No placeholder markers existed. | None to replace. | none | |
| -| D8 | Bodies cited, not given. | Written: `relocationBuildP`, `authorityStore`, `PreparedFromInstallManifest.Verify` (8b); `NewRelocationRuntime` with the worker literal, `Run`, `convertFencedLanding`, `Worker.Fence`, the no-target bound, `edgesfile.LiveSnapshot` (10). Not needed: `relocationAppAdmission`, `localBoxReport` and `hostfloor.Read` served only `Worker.Start`, which the pass never calls; its `Admission` and `Census` refuse (ruling 72). `expectedSecretsFP` is `edgesFileSecretsFP`; `MovesRecordsFrom` exists (movespolicy.go). Every rev 1 test name is written or withdrawn. | 8b, 10 | |
| +| D8 | Bodies cited, not given. | Written: `relocationBuildP`, `authorityStore`, `PreparedFromInstallManifest.Verify` (8b); `NewRelocationRuntime` with the worker literal, `Run`, `convertFencedLanding`, `Worker.Fence` (rev 5.1: `Worker.ConvertLanding`), the no-target bound, `edgesfile.LiveSnapshot` (10). Not needed: `relocationAppAdmission`, `localBoxReport` and `hostfloor.Read` served only `Worker.Start`, which the pass never calls; its `Admission` and `Census` refuse (ruling 72). `expectedSecretsFP` is `edgesFileSecretsFP`; `MovesRecordsFrom` exists (movespolicy.go). Every rev 1 test name is written or withdrawn. | 8b, 10 | |
| | D9 | Typed-nil returns through interfaces. | `relocationLeases` and the files-store wrappers return a nil interface on error. | 10 | |
| | R5-1 | Rev 1's Task 9 made every `RowHeld` hold-only in `hostbox/reconcile.go`. Stage A's `check` already refuses the executor's reconcile under a relocating claim before any write, and the change broke stage A's `TestReconcileRelocationTargetWithNoLocalRecordIsRefusedBeforeActivation`. | Withdrawn. Reconcile leaves the target's row to its install (`TestReconcileLeavesARelocatingTargetsRowToItsInstall`); after completion and the hold release, reconcile keeps the row ready under the same ownership (`TestReconcileAdapterClearsTheHoldAfterCompletion`). Ruling 31 is corrected. | 9 | |
| | R5-2 | Rev 3's ported 200-seed lifecycle property (ruling 63) needs main's participation owner to open controllers under a test clock and to replay root's stops, beyond any seam main has. | Withdrawn and replaced by deterministic tests of region HA's parts: C10 (`TestRelocationCompletionDischargesParticipationWithoutRestart`, `TestRelocationRestartAfterServingCASRecoversParticipation`, `TestRelocationRestartDischargeSurvivesAFailedClaimRead`, each failing with C10's call removed), `sameInstallation` (`TestParticipantRecordsItsReceiptAfterAnEffectWroteTheClaim`, `TestRelocationInstallRecordsItsReceiptThroughParticipantStep`), retired stamps (`TestRelocationEnsureLocalRefusesARetiredStamp`, `TestRelocationBuildActivatesTheGuardFromAnEmptyTarget`), and C6's first outcome (`TestRelocationAcknowledgesThroughMainsParticipation`, the op ending without the box). **Unresolved: needs acceptance.** | 8a to 9 | |
| @@ -88,23 +101,21 @@ The labels and their order are rev 4.2's (8a, 8b, 8c, 8d, 9, 10, 11). What moved |
| | The lease-region test | 8d (`relocation_rev2_test.go`) | 10 | |
| | The hold-only reconcile change | 9 | withdrawn (R5-1) | |
| -## Rev 5: per-task compile and test record |
| +## Per-task compile and test record (rev 5.1) |
| -Each task was committed in a throwaway copy of this branch at b9a753f0, on top of `internal/instancekey` from `instance-key-build` Task 1 as built (e8487c3a; its API is unchanged since), and checked at its own commit with `GOFLAGS=-buildvcs=false`, niced, and the socket tests run outside the sandbox with `TMPDIR` in a private `/tmp/k.XXXX`: |
| +Each task was committed in a throwaway copy of this branch (code as at b9a753f0; the branch has only plan commits since), on top of `internal/instancekey` from `instance-key-build` Task 1 as built (e8487c3a), in the order 8a, 8b, 8c, 8d, 9, 10, and checked at its own commit with `GOFLAGS=-buildvcs=false`, niced, and the socket tests run outside the sandbox with `TMPDIR` in a private `/tmp/k.XXXX`. Each task's code in this plan was then extracted from the text into a second copy and found identical, file for file, to the commit that was checked. |
| | Task | `go build ./...` | `go vet` (touched packages) | `gofmt -l` | Tests at the task's commit | |
| |---|---|---|---|---| |
| | IK-1 (precondition) | clean | clean: instancekey | empty | `instancekey` passes in full | |
| -| 8a | clean | clean: host, hostclaim, hostrelocate | empty | `hostclaim` and `hostrelocate` pass in full; host's release, census, sweep and participation tests and the Plan 9 and Plan 11 regressions pass | |
| +| 8a | clean | clean: host, hostclaim, hostrelocate, activator | empty | `hostclaim` and `hostrelocate` pass in full; the activator's fence and maintenance tests (the owned fence's three among them) pass; host's release, census, sweep and participation tests and the Plan 9 and Plan 11 regressions pass | |
| | 8b | clean | clean: host, hostrelocate | empty | `hostrelocate` passes in full; the build and verifier tests pass | |
| | 8c | clean | clean: host, hostbox, hostfiles, registry, provision | empty | `registry` and `hostfiles` pass in full; hostbox's adapter and census tests, provision's reconcile and route tests and host's install tests pass | |
| | 8d | clean | clean: host | empty | the owner, wake, attempt and root tests pass, with the moves probe-owner, Plan 9 D5, Plan 11 and claim-launch regressions | |
| -| 9 | clean | clean: host, hostbox | empty | hostbox's reconcile tests and host's probe-stop, hold-release, discharge, handover, participation, Plan 9, Plan 11 and handover regressions pass | |
| -| 10 | clean | clean: host, hostrelocate, edgesfile, console | empty | `hostrelocate` and `edgesfile` pass in full; the console's edges, eligibility, placement, region, executor and claim-runtime tests pass; 74 region HA host tests pass with the Plan 9, Plan 11, claim-launch, moves and handover regressions | |
| - |
| -At Task 10 the whole `internal/host`, `internal/hostbox` and `internal/console` packages also pass, once, outside the sandbox. At Task 10, `GOOS=darwin`, `GOOS=windows` and `GOARCH=arm64` builds of `./...` are clean, and so is `go vet` of host, hostbox, hostrelocate, hostclaim, edgesfile and console for darwin and arm64, and of all of them but hostbox for windows. `GOOS=windows go vet ./internal/hostbox/` fails at b9a753f0 already (`plan9_fix2_minors_test.go:100`, `undefined: plan9D1Probe`); this plan does not touch it. |
| - |
| +| 9 | clean | clean: host, hostbox | empty | hostbox's reconcile tests and host's probe-stop (restart, failed marker write), hold-release, successor, discharge, handover, sweep, Plan 9, Plan 11 and handover regressions pass | |
| +| 10 | clean | clean: host, hostrelocate, edgesfile, console | empty | `hostrelocate` and `edgesfile` pass in full; the console's edges, eligibility, placement, region, executor and claim-runtime tests pass; 79 region HA host tests pass with the Plan 9, Plan 11, claim-launch, moves and handover regressions | |
| +At Task 10 the whole `internal/activator` (142s), `internal/hostclaim`, `internal/hostrelocate`, `internal/hostbox` and `internal/host` (45s) packages also pass, once, outside the sandbox; the console's whole package passed at rev 5 and rev 5.1 does not touch it. At Task 10, `GOOS=darwin`, `GOOS=windows` and `GOARCH=arm64` builds of `./...` are clean, and so is `go vet` of host, hostrelocate, hostclaim and edgesfile for all three and of activator for darwin. Two windows vets fail at b9a753f0 already and this plan does not touch either cause: `./internal/hostbox/` (`plan9_fix2_minors_test.go:100`, `undefined: plan9D1Probe`) and `./internal/activator/` (`activator_test.go:4528`, `undefined: unix.Mkfifo`). |
| ## Disposition history (rev 2 to rev 4.2) |
| @@ -329,7 +340,7 @@ Rulings 28 to 41 are rev 1's; 42 to 54 rev 2's; 55 to 64 rev 3's; 65 to 69 rev 4 |
| 46. **The install's authority is main's WorkSource, with only its policy replaced** (findings 5 and 10; supersedes 34). `relocationInstallSource` copies the bound assembly's `hostbox.WorkSource` (`ClaimProviders.Selector`, claimassembly.go:156 and :177-183: `Guard`, `Box`, `Region`, `Now`, `Prepare` = `productionClaimLocal.Prepare`, `Local` = `Identity.Local(productionClaimLocal)`), sets `Continue` nil, and sets `Policy` to `relocationInstallPolicy`. That policy admits only `install`, `held-config` and `held-release` under `relocationInstallAuthority` (installed, or probing before P), at the holder's capture, with the hold absent or the op's, then runs region HA's `Policy.Relocation` owner. `WorkSource.With` runs `Local` before `Policy` on every call (work.go:80-101, `Local` at :93), so a lost lease, a passed deadline or a quarantine stops the next write. The held config goes through `RenderHeldUnder` with a `held-config` selection, and the release through `MaterializeHeldUnder` with a `held-release` selection (finding 5; never `MaterializeHeld`, which refuses under the claim). *Cost if wrong:* the three actions' policy is region HA's rather than the claim's until C9. Every other check is main's. |
| 47. **P's opening publishes through main's prepared-publication contract** (finding 6), as moves' landing probe does. `Materialize` records the admitted spawn selection as the session's fence and, for a prepared admission, publishes P's marker under it (`preparedMarker`: install.json must be the prepared authority's bytes; the marker names P with the build's pin, or is already P's). `Publish`, which the assembly calls before replication only for a prepared admission, finds or writes it under that recorded fence and records the receipt. `PublishFenced` is main's `publishFresh` (N3) for a later attempt's fresh generation and for an empty P, which `Select` marks `Prepared` false so the assembly's root-pending wait applies (claimwakeassembly.go:612-617). `PublicationEvidence` checks the prepared receipt, or delegates to main's `freshEvidence`. *Cost if wrong:* none beyond moves'; no shared assembly change. |
| 48. **Root reads relocation's attempts, and nothing is counted until it can** (finding 7). `relocationAttemptReader.ReadAttempt` returns the reserved start of attempt k = `probe_attempts.<op.exec>` only when the claim is the relocating op at `probing`, unstuck, holder stamp equal to `install_stamp`, and the journal binds the holder's session to k with the same generation and stamp. Root's `authorize` then binds the reservation id, invocation, launch-input digest and peer (claimlaunch_authority.go:67-112). Root construction installs it with `CARLOS_REGION_HA` on (N2). The participant calls the new `Effects.Preflight` (root's protocol, then `PreflightKind(relocating)`) before `BeginProbeAttempt`; a refusal returns `ErrNotYet` with nothing counted. *Cost if wrong:* while root refuses, the probe deadline passes with no attempt counted and the op goes `probe-failed`, then a replacement or `stuck`. It is visible and spends no attempt. |
| -49. **The cold restart is an explicit stop, receipt and completion** (finding 8; amends 32). After `RecordProbe`, the target, with the op still `relocating/probing` so the captured session still matches `sameStopWriter`: (1) `activator.Service.FenceAndStopIfUnfenced(host, S)` closes local admission for the host and stops the probe entry through the activator's own stop path (maintenance.go:53-155), never over another operation's fence (ruling 71); `ErrStopUnproven` still means the fence is installed and is accepted, any other error stops here. (2) `ClaimWakeAssembly.Stop` (main's `stopCapturedSession`) asks root for a writer-session stop of S, which `RelocationWriterStop` authorizes (hostguard/writerstop.go:72-88); its obligation id is deterministic, so a repeat meets the same fence and receipt, and main then releases the attempt's lease. (3) Root's receipt for S is read from this box's guard journal (`CompletedWriterStops`), and `RecordProbeStop` journals `probe_stop` {box, session, fence id, sha256}. (4) The worker's `CompleteRelocation` requires `probe_stop` naming the holder's session. (5) The serving box releases its row's hold (`ReleaseHold`) and lifts the fence at S. (6) The next request wakes the host cold as serving; rev 3 (ruling 56): its handover runs root's `ColdHandover.Run` with S's receipt as prior (N6), and the participation is discharged by the same evidence (C10). The pass runs participant, worker and completed hold in that order per host, so steps 1 to 5 take one pass. *Cost if wrong:* without C10 the completed target's participation denies serving; that is the activation gate (rev 3), proved by the end-to-end test. |
| +49. **The cold restart is an explicit stop, receipt and completion** (finding 8; amends 32). After `RecordProbe`, the target, with the op still `relocating/probing` so the captured session still matches `sameStopWriter`: (1) `activator.Service.FenceAndStopAs(host, S, owner)`, under the owner persisted first, closes local admission for the host and stops the probe entry through the activator's own stop path (maintenance.go), never over another operation's fence (rulings 71, 75); `ErrStopUnproven` still means the fence is installed and is accepted, any other error stops here. (2) `ClaimWakeAssembly.Stop` (main's `stopCapturedSession`) asks root for a writer-session stop of S, which `RelocationWriterStop` authorizes (hostguard/writerstop.go:72-88); its obligation id is deterministic, so a repeat meets the same fence and receipt, and main then releases the attempt's lease. (3) Root's receipt for S is read from this box's guard journal (`CompletedWriterStops`), and `RecordProbeStop` journals `probe_stop` {box, session, fence id, sha256}. (4) The worker's `CompleteRelocation` requires `probe_stop` naming the holder's session. (5) The serving box releases its row's hold (`ReleaseHold`), lifts the fence at S under its persisted owner (`LiftFenceAs`), and marks the completion discharged (rulings 66, 76, 77). (6) The next request wakes the host cold as serving; rev 3 (ruling 56): its handover runs root's `ColdHandover.Run` with S's receipt as prior (N6), and the participation is discharged by the same evidence (C10). The pass runs participant, worker and completed hold in that order per host, so steps 1 to 5 take one pass. *Cost if wrong:* without C10 the completed target's participation denies serving; that is the activation gate (rev 3), proved by the end-to-end test. |
| 50. **The data bucket is opened as a `release.Store` of its own** (finding 9): `release.NewS3Store(ctx, dataBucket)` (store.go:680) beside the versioned client, in `openRelocationStores` (the runtime) and through `RelocationSource.FilesStore` (the owners). No type assertion. *Cost if wrong:* `NewS3Store` signs with the default chain's region, the box's own, which is the hibernate bucket's region for a box in its region; a cross-region data bucket would need a region-pinned client, which no relocation uses (a relocation never leaves its region). |
| 51. **The lease signs for the AWS region; everything else uses the logical one** (finding 12). `LeaseRegion` = roster `Cloud.Region`, as `Wake.LeaseRegion` (claimassembly.go:186), for `lease.Acquire` and `NewVersionedS3`. `Region` = roster `Region` for genselect, pins, the worker and the chooser. *Cost if wrong:* none; two fields. |
| 52. **An install stamp is never reactivated** (finding 13). `Guard.Activate` refuses retired stamps (guard.go:207-210), so stage A's "restore a retired install" was unreachable and is retired. `EnsureLocal` repairs damage only while the install stamp is the active one (the install checks `Capture` equals `install_stamp` and accepts `installed` and `probing` before P). A retired stamp, or a lost lease, which is main's retained denial (ruling 42), refuses. The probe deadline then makes the op `probe-failed` and the worker replaces the target, whose `install()` issues a new stamp. *Cost if wrong:* a damaged install after a lease loss costs a target replacement rather than a repair. |
| @@ -355,10 +366,15 @@ Rulings 28 to 41 are rev 1's; 42 to 54 rev 2's; 55 to 64 rev 3's; 65 to 69 rev 4 |
| 70. **C10 never falls through on an error** (Astra round 4). A failed claim read in `completedRelocationDischarge` is `(true, err)`, and `productionClaimLocal.cleanup` returns on any error from it, so no ordinary cleanup, serving-shaped stop or `BeginFence` runs while the claim read fails. The participation stays denied, the row stays held (ruling 66), and the next pass retries. *Cost if wrong:* none. A pass is lost to a transient read error. |
| -71. **The probe stop never takes over another operation's fence** (K19). `relocationStopProbe` installs its activator fence with `FenceAndStopIfUnfenced`. Another operation's fence is `ErrFenceHeld`: nothing is stopped and nothing is asked of root, and the participant retries on the next pass (the passed probe is bounded by ruling 67). Once installed, the fence's session is recorded under `<DataDir>/relocate/ops/<op>/probe-fence.json`, so a repeat of the same stop recognises its own fence and re-collects the stop evidence with `FenceAndStop` at the same generation. `ReleaseHold`'s `LiftFence` therefore only ever lifts this probe's fence. *Cost if wrong:* a probe whose stop meets a foreign fence waits for it, then goes `stuck` at `ReleaseDeadline` after the probe passed, for an operator. |
| +71. **The probe stop never takes over another operation's fence** (K19; amended by 75, which replaces the session-only record and `FenceAndStopIfUnfenced` with an owned fence). `relocationStopProbe` installs its activator fence with `FenceAndStopIfUnfenced`. Another operation's fence is `ErrFenceHeld`: nothing is stopped and nothing is asked of root, and the participant retries on the next pass (the passed probe is bounded by ruling 67). Once installed, the fence's session is recorded under `<DataDir>/relocate/ops/<op>/probe-fence.json`, so a repeat of the same stop recognises its own fence and re-collects the stop evidence with `FenceAndStop` at the same generation. `ReleaseHold`'s `LiftFence` therefore only ever lifts this probe's fence. *Cost if wrong:* a probe whose stop meets a foreign fence waits for it, then goes `stuck` at `ReleaseDeadline` after the probe passed, for an operator. |
| 72. **The relocation pass starts no relocation** (D8). `Worker.Start` is the operator's (`carlos ops host relocate`) and spec 3's placer's; the pass only replays. The pass's worker therefore has an `Admission` and a `Census` that refuse (`errRelocationNotStarted`), and rev 1's `relocationAppAdmission`, `localBoxReport` and `hostfloor.Read` belong to that starter's plan. *Cost if wrong:* none here; a starter binds its own. |
| 73. **Moves' held-stage relay is census-opaque until moves classifies it** (K25, M-K25, N11). A relocation target whose stage is blocked leaves a request keyed by its host, and root's pass leaves an ack; both live under `moves/`, outside every installation. Region HA's `claimDataCensus` entry classifies `moves/held-stage` and `moves/held-staged` opaque, as N9 does `relocate`, so another host's empty installation still initialises on the box. *Cost if wrong:* the census does not look inside two relay directories whose contents root validates before any effect. |
| 74. **Every task compiles and passes its tests at its own commit** (rev 5). Code lives in the earliest task that uses it; a later task may extend an earlier task's file, never the reverse. *Cost if wrong:* none. |
| +75. **The probe stop's fence is owned** (Astra round 6 findings 1 and 2; amends 71). `relocationStopProbe` persists `{host, session, owner}` in `relocate/ops/<op>/probe-fence.json` before asking the activator, with a fresh random owner unless one is persisted for this host and session. `activator.FenceAndStopAs` installs only on an unfenced host, or meets again exactly the fence it installed under that owner at that generation, deciding under the lock that installs; any other fence is `ErrFenceHeld`, never replaced. `LiftFenceAs` lifts only that fence. A failed marker write installs nothing; a restart empties the activator, so a surviving owner matches no fence. *Cost if wrong:* a probe stop waits behind another operation's fence until ruling 67's bound. |
| +76. **The target ends a completed relocation by marking it discharged** (Astra round 6 findings 2 and 3; extends 66). `relocationReleaseHold`: while the row is held for the op, the holder must still be the probe session, then main's discharge (C10), then `registry.ReleaseHold`; then `LiftFenceAs` under the persisted owner; then `hostclaim.MarkCompletedDischarged`. Once the row is no longer held for the op, a replay only lifts and marks; a missing row is refused. The journal sweep keeps a completed op's directory until the mark. *Cost if wrong:* a replay after a crash between the release and the mark repeats one lift. |
| +77. **No successor relocation starts from an undischarged completion** (Astra round 6 finding 3). `RelocationStartGuard` refuses with `ErrPendingChore` while the serving op's `CompletedRelocation` is not `Discharged`, unless the start is a failover citing the completion box's fence (a dead box's slot binds nothing on the new target). *Cost if wrong:* a drain from a just-completed target waits one pass. |
| +78. **The landing conversion is the worker's, under Step's checks** (Astra round 6 finding 4). `hostrelocate.Worker.ConvertLanding` checks this box's current roster entry (enrollment, `active`, region, no termination fence) as `Step` does, and converts only a landing into its own region. *Cost if wrong:* none. |
| +79. **Task 10b is mandatory before activation** (Astra round 6 on R5-3). The source's row is held at its fence's close by Task 10b; relocation is not switched on anywhere until it has landed and its tests pass. *Cost if wrong:* none while dormant. |
| ## L21: where the platform-provisioned `CARLOS_INSTANCE_KEY` belongs |
| @@ -373,7 +389,7 @@ Stage B's cost without the L21 plan is unchanged from rev 1: a host whose app mi |
| ## Global Constraints |
| - **Dormant.** Nothing changes on a box or console where `CARLOS_REGION_HA` is unset or not true (`regionHAEnabled`, boxrecord.go:25). The relocation owners open only with it on (claimrelocation.go:53) and with moves' C8 provider bound (ruling 53); root installs its relocating attempt reader only with it on (N2); the passes start only with it and `CARLOS_HOST_CLAIM` on; the console's chooser only with `Config.RegionHA`. |
| -- **Activation gate (claim §1.1, L18).** Relocation does not merge until Task 11's joint round passes on the two-box model, and is not switched on until the claim confirms C10, C8 with O1 is registered, R5-3 is decided, and the two-box run passes on a dev box (checklist item 12). |
| +- **Activation gate (claim §1.1, L18).** Relocation does not merge until Task 11's joint round passes on the two-box model, and is not switched on until the claim confirms C10, C8 with O1 is registered, Task 10b has landed (R5-3, ruling 79), and the two-box run passes on a dev box (checklist item 12). |
| - **One implementation per concept; main's wins.** No second scanner, receipt verifier, roster reader, fence writer, inventory, retirement, marker publisher, participation controller or WorkSource. The probe is main's attempt path; the attempt journal is region HA's only because `OperationAttempts` is per kind. The install policy is region HA's only until C9. |
| - **Absence is never evidence.** A fence, receipt, pin, manifest, env item or inventory that cannot be read is an error. |
| - **No region HA file under a managed root names a host** (ruling 44): journals under `<DataDir>/relocate/ops/<op>/`, positions under `<DataDir>/relocate/release/<op>.json`, the build's scratch removed before the install. |
| @@ -403,7 +419,7 @@ Non-test files only; each task lists its tests. |
| | `internal/hostclaim/relocation.go` | `Probe.At`; `CompleteRelocation` requires the probe stop, journals `completed_relocation` and, for a converted landing, `released_move` | 8a | |
| | `internal/hostclaim/relocation_probestop.go` (new) | `ProbeStop`, `RecordProbeStop`, `CompletedRelocation`, `ReadCompletedRelocation`, `ToInstalledAt` | 8a | |
| | `internal/hostrelocate/participant.go` | `Effects.Preflight`, `Effects.EndProbe`; the gate; the probe-stop branch; `sameInstallation`; `ErrLaterAttemptUnlaunchable`; `CertifiedFor` | 8a | |
| -| `internal/hostrelocate/worker.go` | Completion waits for `probe_stop`; the `installed` and passed-probe bounds (8a); `Fence` and the no-target bound (10) | 8a, 10 | |
| +| `internal/hostrelocate/worker.go` | Completion waits for `probe_stop`; the `installed` and passed-probe bounds (8a); `ConvertLanding` and the no-target bound (10) | 8a, 10 | |
| | `internal/host/claimparticipation_relocation.go` (new, N1) | `Acknowledged` (8a), `DischargeCompleted` (9), `boundAssembly` (10) | 8a, 9, 10 | |
| | `internal/host/claimparticipation.go` | N12 (8a); C10's binding argument to `cleanup` (9) | 8a, 9 | |
| | `internal/host/relocationeffects.go` | Rewritten: main's participation; `Release` in ruling 28's order; `Preflight`, `EndProbe`; `Finish` is the hold release | 8a | |
| @@ -422,7 +438,7 @@ Non-test files only; each task lists its tests. |
| | `internal/host/claimlaunch_production.go` (N2) | The relocating attempt reader with region HA on | 8d | |
| | `internal/host/relocationwake.go` (new) | `RelocationWakeInputs` | 8d | |
| | `internal/host/relocationattempts.go` (new) | `relocationProbeAttempts`, `relocationAttemptReader` | 8d | |
| -| `internal/host/relocationowners.go` (new) | `RelocationSource`, refusals, hooks, `relocationAdmit` (8d); `relocationStopProbe`, the probe-fence record, `relocationReleaseHold` (9) | 8d, 9 | |
| +| `internal/host/relocationowners.go` (new) | `RelocationSource`, refusals, hooks, `relocationAdmit` (8d); `relocationStopProbe`, the probe stop's persisted fence owner, `relocationReleaseHold` with its discharge mark (9) | 8d, 9 | |
| | `internal/host/claimparticipation_completed.go` (new, C10) | `completedRelocationDischarge` | 9 | |
| | `internal/host/claimparticipation_stale.go` | C10's binding at `clearUntracked` | 9 | |
| | `internal/host/relocationhandover.go` (new, N6) | `relocationCompletedHandover`, `writerSessionReceipt` | 9 | |
| @@ -437,7 +453,7 @@ Non-test files only; each task lists its tests. |
| ### Task 8a: Main's participation owner, the release providers, and the probe-stop primitives |
| -Region HA reads main's one participation controller per box instead of opening its own (ruling 42), and the release runs in claim Plan 11's order (ruling 28). The claim's probe-stop primitives, the participant's preflight gate and probe-stop branch, and the worker's bounds land here too, because `RelocationEffects` implements `hostrelocate.Effects` in this task and every later task builds on those types (K1, K2, K5: rev 4.2 had them in Tasks 8d and 9). Stage A's controller set and its two tests are deleted. The journals are census-opaque (N9) and swept by op (ruling 61), release positions are keyed by op with one op-id check (`validOpID`, D5c), and the flush publishes the releasing writer's adoption item alone (O5). N12 is a test seam in main's participation owner. |
| +Region HA reads main's one participation controller per box instead of opening its own (ruling 42), and the release runs in claim Plan 11's order (ruling 28). The claim's probe-stop primitives, the participant's preflight gate and probe-stop branch, and the worker's bounds land here too, because `RelocationEffects` implements `hostrelocate.Effects` in this task and every later task builds on those types (K1, K2, K5: rev 4.2 had them in Tasks 8d and 9). Stage A's controller set and its two tests are deleted. The journals are census-opaque (N9) and swept by op (ruling 61), release positions are keyed by op with one op-id check (`validOpID`, D5c), and the flush publishes the releasing writer's adoption item alone (O5). N12 is a test seam in main's participation owner. Rev 5.1 adds the activator's owned fence (ruling 75), the completed relocation's discharge mark and the start guard that waits for it (ruling 77), and keeps a completed op's journals until that mark (ruling 76). |
| Order notes: |
| @@ -446,19 +462,19 @@ Order notes: |
| - The binding check on an acknowledgement (op, incarnation, host) lives once, in `RelocationEffects.Acknowledge` (D5d). |
| **Files:** |
| -- Create: `internal/host/claimparticipation_relocation.go`, `internal/host/relocationassembly.go`, `internal/host/relocationcensus_test.go`, `internal/host/relocationflush_adoption_test.go`, `internal/host/relocationparticipation_linux_test.go`, `internal/host/relocationsweep.go`, `internal/host/relocationsweep_test.go`, `internal/hostclaim/relocation_probestop.go`, `internal/hostclaim/relocation_probestop_test.go`, `internal/hostrelocate/participant_gate_test.go`, `internal/hostrelocate/worker_bounds_test.go` |
| -- Modify: `internal/host/claimcensus.go` (N9), `internal/host/claimparticipation.go` (N12 only), `internal/host/relocationeffects.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationflush.go`, `internal/hostclaim/relocation.go`, `internal/hostclaim/relocation_test.go` (each `CompleteRelocation` preceded by `RecordProbeStop`), `internal/hostrelocate/participant.go`, `internal/hostrelocate/participant_test.go`, `internal/hostrelocate/worker.go`, `internal/hostrelocate/worker_test.go` (`TestWorkerDrivesADrain` waits for the probe stop) |
| +- Create: `internal/activator/fenceowner_test.go`, `internal/host/claimparticipation_relocation.go`, `internal/host/relocationassembly.go`, `internal/host/relocationcensus_test.go`, `internal/host/relocationflush_adoption_test.go`, `internal/host/relocationparticipation_linux_test.go`, `internal/host/relocationsweep.go`, `internal/host/relocationsweep_test.go`, `internal/hostclaim/relocation_probestop.go`, `internal/hostclaim/relocation_probestop_test.go`, `internal/hostrelocate/participant_gate_test.go`, `internal/hostrelocate/worker_bounds_test.go` |
| +- Modify: `internal/activator/activator.go` (`fenceOwners`), `internal/activator/maintenance.go` (`FenceAndStopAs`, `LiftFenceAs`), `internal/host/claimcensus.go` (N9), `internal/host/claimparticipation.go` (N12 only), `internal/host/relocationeffects.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationflush.go`, `internal/hostclaim/relocation.go`, `internal/hostclaim/relocation_test.go` (each `CompleteRelocation` preceded by `RecordProbeStop`), `internal/hostrelocate/participant.go`, `internal/hostrelocate/participant_test.go` (the fake's `Finish` marks the completion discharged, as production's hold release does), `internal/hostrelocate/worker.go`, `internal/hostrelocate/worker_test.go` (`TestWorkerDrivesADrain` waits for the probe stop) |
| - Delete: `internal/host/relocationcontrollers_linux_test.go`, `internal/host/relocationlifecycle_linux_test.go` (they test stage A's controller set, which this task deletes, ruling 42) |
| **Interfaces:** |
| -- Consumes: main's `productionClaimLocal` (`Prepare`, `prepareWith`, `slots`, `claimParticipation`: claimparticipation.go); `execlease.State`, `ErrLost`, `ErrInactive`, `InspectLocal`; `ClaimAssembly` as `Build` binds it, `localStopIdentity`, `ClaimWakeAssembly.Producers`, `get`, `Roles` (`Preflight`, `PreflightKind`); `ClaimAssembly.WritersStopped`; `CollectClaimStop`; `StopRequest.Recover`; `ReleaseObligation`; `ReleaseFlushFiles`; `genselect.Select`, `Plan.MaxLevel0TXID`; `activator.Classify`; `provision.InstanceDir`; hostclaim's `requireRelocation`, `readJournal`, `journalJSON`, `isHex64`, `ToInstalled`, `MarkStuck`; `instancekey` (IK-1: `Binding`, `ParseAdoption`, `Binds`, `AdoptionItem`, `AdoptionSuffix`). |
| -- Produces: `hostclaim.ProbeStop`, `RecordProbeStop`, `Probe.At`, `CompletedRelocation`, `JournalCompleted`, `ReadCompletedRelocation`, `ToInstalledAt`, `CompleteRelocation`'s probe-stop gate and `released_move`; `hostrelocate.Effects.Preflight` and `EndProbe`, `ErrLaterAttemptUnlaunchable`, `Participant.sameInstallation`, `CertifiedFor`, the worker's probe-stop wait and its `installed` and passed-probe bounds; `(*productionClaimLocal).Acknowledged` (N1), the N12 seam (`lockRoot`, `lockDir`, `updateFloor`); `RelocationActivator`, `RelocationAssemblyConfig`, `(ClaimAssembly) RelocationEffects`, `relocationPreflight`, `relocationPathsFor`, `relocationHostPaths`, `relocationFlushDB`; the rewritten `RelocationEffects`; `validOpID`; `sweepRelocationJournals`. |
| +- Consumes: main's `productionClaimLocal` (`Prepare`, `prepareWith`, `slots`, `claimParticipation`: claimparticipation.go); `execlease.State`, `ErrLost`, `ErrInactive`, `InspectLocal`; `ClaimAssembly` as `Build` binds it, `localStopIdentity`, `ClaimWakeAssembly.Producers`, `get`, `Roles` (`Preflight`, `PreflightKind`); `ClaimAssembly.WritersStopped`; `CollectClaimStop`; `StopRequest.Recover`; `ReleaseObligation`; `ReleaseFlushFiles`; `genselect.Select`, `Plan.MaxLevel0TXID`; `activator.Classify`; `provision.InstanceDir`; hostclaim's `requireRelocation`, `readJournal`, `journalJSON`, `isHex64`, `ToInstalled`, `MarkStuck`, `RelocationStartGuard`; the activator's fence map and `fenceAndStop` (maintenance.go); `instancekey` (IK-1: `Binding`, `ParseAdoption`, `Binds`, `AdoptionItem`, `AdoptionSuffix`). |
| +- Produces: `hostclaim.ProbeStop`, `RecordProbeStop`, `Probe.At`, `CompletedRelocation`, `JournalCompleted`, `ReadCompletedRelocation`, `ToInstalledAt`, `CompleteRelocation`'s probe-stop gate and `released_move`; `CompletedRelocation.Discharged`, `MarkCompletedDischarged`, `RelocationStartGuard`'s refusal of a start from an undischarged completion (ruling 77); `activator.(*Service).FenceAndStopAs`, `LiftFenceAs` (ruling 75); `hostrelocate.Effects.Preflight` and `EndProbe`, `ErrLaterAttemptUnlaunchable`, `Participant.sameInstallation`, `CertifiedFor`, the worker's probe-stop wait and its `installed` and passed-probe bounds; `(*productionClaimLocal).Acknowledged` (N1), the N12 seam (`lockRoot`, `lockDir`, `updateFloor`); `RelocationActivator`, `RelocationAssemblyConfig`, `(ClaimAssembly) RelocationEffects`, `relocationPreflight`, `relocationPathsFor`, `relocationHostPaths`, `relocationFlushDB`; the rewritten `RelocationEffects`; `validOpID`; `sweepRelocationJournals`, which keeps a completed op's journals until its discharge mark (ruling 76). |
| - [ ] **Step 1: Write the failing tests** |
| The release tests drive `RelocationEffects.Release` over recording providers (`effectsFixture`). `TestRelocationAcknowledgesThroughMainsParticipation` runs main's real `productionClaimLocal` in agent mode over real hostexec controllers, a signed roster and the real census, with its locks under the N12 seam. It is Linux-only and runs inside the sandbox. |
| -New tests: `TestCensusInitializesWithOldRelocationJournals`, `TestReleaseFlushPublishesOnlyTheReleasingWritersAdoption`, `TestRelocationAcknowledgesThroughMainsParticipation`, `TestRelocationJournalsAreSweptWhenTheirOpEnds`, `TestCompleteRelocationRequiresTheProbeStop`, `TestCompleteRelocationJournalsReleasedMoveForAConvertedLanding`, `TestToInstalledAtJournalsItsStart`, `TestParticipantCountsNoAttemptWhileRootRefusesTheKind`, `TestParticipantFailsAProbeRootCannotRelaunch`, `TestParticipantRecordsItsReceiptAfterAnEffectWroteTheClaim`, `TestParticipantEndsNoProbeWithoutItsLease`, `TestParticipantRecordsNoProbeStopWhileRootRefuses`, `TestWorkerBoundsAPassedProbeAwaitingItsStop`, `TestWorkerBoundsAnUntimedPassedProbeFromProbingSince`, `TestWorkerBoundsAnInstallThatNeverProbes`, `TestWorkerBoundsARefusedProbe`, `TestReleaseStopsWritersAndFlushesBeforeItQueues`, `TestReleaseNeverReturnsBeforeRootsReceipt`, `TestReleaseAfterARestartRecoversTheRecordedOwner`, `TestEnsureLocalRepairsOnlyUnderTheActiveStampBeforePOpens`. |
| +New tests: `TestFenceAndStopAsMeetsOnlyItsOwnFence`, `TestAPlainFenceTakesOwnershipAway`, `TestLiftFenceAsLiftsOnlyTheOwnersFence`, `TestRelocationJournalsOfACompletedOpStayUntilItsDischarge`, `TestRelocationSuccessorWaitsForTheCompletedDischarge`, `TestCensusInitializesWithOldRelocationJournals`, `TestReleaseFlushPublishesOnlyTheReleasingWritersAdoption`, `TestRelocationAcknowledgesThroughMainsParticipation`, `TestRelocationJournalsAreSweptWhenTheirOpEnds`, `TestCompleteRelocationRequiresTheProbeStop`, `TestCompleteRelocationJournalsReleasedMoveForAConvertedLanding`, `TestToInstalledAtJournalsItsStart`, `TestParticipantCountsNoAttemptWhileRootRefusesTheKind`, `TestParticipantFailsAProbeRootCannotRelaunch`, `TestParticipantRecordsItsReceiptAfterAnEffectWroteTheClaim`, `TestParticipantEndsNoProbeWithoutItsLease`, `TestParticipantRecordsNoProbeStopWhileRootRefuses`, `TestWorkerBoundsAPassedProbeAwaitingItsStop`, `TestWorkerBoundsAnUntimedPassedProbeFromProbingSince`, `TestWorkerBoundsAnInstallThatNeverProbes`, `TestWorkerBoundsARefusedProbe`, `TestReleaseStopsWritersAndFlushesBeforeItQueues`, `TestReleaseNeverReturnsBeforeRootsReceipt`, `TestReleaseAfterARestartRecoversTheRecordedOwner`, `TestEnsureLocalRepairsOnlyUnderTheActiveStampBeforePOpens`. |
| `internal/host/relocationcensus_test.go` (new): |
| @@ -896,9 +912,11 @@ func TestRelocationAcknowledgesThroughMainsParticipation(t *testing.T) { |
| package host |
| import ( |
| + "encoding/json" |
| "errors" |
| "os" |
| "path/filepath" |
| + "strings" |
| "testing" |
| "time" |
| @@ -984,6 +1002,68 @@ func TestRelocationJournalsAreSweptWhenTheirOpEnds(t *testing.T) { |
| t.Fatalf("the second pass swept the live op: %v", err) |
| } |
| } |
| + |
| +// Ruling 76: the op a serving claim records as completed keeps its |
| +// journals, the probe stop's fence owner among them, until the completed |
| +// relocation is marked discharged, so the hold release always finds the |
| +// owner it lifts under. Then the next sweep removes them. |
| +func TestRelocationJournalsOfACompletedOpStayUntilItsDischarge(t *testing.T) { |
| + ctx := t.Context() |
| + store := release.NewDirStore(t.TempDir()) |
| + created := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC) |
| + record := provision.Record{Schema: 1, Host: "shop.example", Slug: "shop", App: "app", Account: "acct", Region: "ie", Channel: "stable", CreatedAt: created} |
| + if err := provision.CreateRecord(ctx, store, record); err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err := hostclaim.Read(ctx, store, record.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Update(ctx, store, obs, hostclaim.Adopt(hostclaim.NewHolder{Family: "instance", Key: provision.RecordKey(record.Account, record.App, record.Host), |
| + Account: record.Account, App: record.App, Host: record.Host, Region: record.Region, CreatedAt: created}, "dst", nil, nil, "adopt", func() string { return "id" })) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + done, _ := json.Marshal(hostclaim.CompletedRelocation{Op: "rel-done", Stop: hostclaim.ProbeStop{Box: "dst", Session: 12, Receipt: "fence-1", ReceiptSHA256: strings.Repeat("a", 64)}, Phase: hostclaim.PhaseRelocatingProbing}) |
| + if obs, err = hostclaim.Update(ctx, store, obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}, Journal: map[string]string{hostclaim.JournalCompleted: string(done)}} |
| + return nil |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + data := t.TempDir() |
| + files := []string{"relocate/ops/rel-done/install.json", "relocate/ops/rel-done/probe-fence.json", "relocate/ops/rel-fence-only/probe-fence.json"} |
| + bodies := []string{`{"v":1,"host":"shop.example","op":"rel-done"}`, `{"host":"shop.example","session":12,"owner":"0123"}`, `{"host":"shop.example","session":3,"owner":"4567"}`} |
| + for i, path := range files { |
| + full := filepath.Join(data, filepath.FromSlash(path)) |
| + if err := os.MkdirAll(filepath.Dir(full), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(full, []byte(bodies[i]), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := sweepRelocationJournals(ctx, data, store); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, path := range files[:2] { |
| + if _, err := os.Stat(filepath.Join(data, filepath.FromSlash(path))); err != nil { |
| + t.Fatalf("%s was swept before the discharge: %v", path, err) |
| + } |
| + } |
| + if _, err := os.Stat(filepath.Join(data, "relocate/ops/rel-fence-only")); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatalf("an ended op known only by its fence owner survived: %v", err) |
| + } |
| + if _, err := hostclaim.Update(ctx, store, obs, hostclaim.MarkCompletedDischarged("rel-done")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := sweepRelocationJournals(ctx, data, store); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := os.Stat(filepath.Join(data, "relocate/ops/rel-done")); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatalf("the discharged op's journals survived: %v", err) |
| + } |
| +} |
| ``` |
| `internal/hostclaim/relocation_probestop_test.go` (new): |
| @@ -995,15 +1075,35 @@ import ( |
| "context" |
| "encoding/json" |
| "errors" |
| + "slices" |
| "strings" |
| "testing" |
| "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| ) |
| // probedDrain drives a drain of the serving fixture to a passed probe on |
| // relTarget: released, pinned, installed, probing, P admitted, the probe |
| // journaled with its time. |
| func probedDrain(t *testing.T) (func(func(*Claim) error) Observed, func() Observed) { |
| + t.Helper() |
| + ctx := context.Background() |
| + s, cur := probedDrainIn(t) |
| + apply := func(tr func(*Claim) error) Observed { |
| + t.Helper() |
| + next, err := Update(ctx, s, cur, tr) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + cur = next |
| + return next |
| + } |
| + return apply, func() Observed { return cur } |
| +} |
| + |
| +// probedDrainIn is probedDrain's claim and its store. |
| +func probedDrainIn(t *testing.T) (release.Store, Observed) { |
| t.Helper() |
| ctx := context.Background() |
| s, o := relServing(t) |
| @@ -1024,17 +1124,7 @@ func probedDrain(t *testing.T) (func(func(*Claim) error) Observed, func() Observ |
| o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseRelocatingProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, PredecessorCertified: true, Region: "ie", PredecessorRegion: "ie"})) |
| o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session, At: relObserved.Add(time.Minute)})) |
| - cur := o |
| - apply := func(tr func(*Claim) error) Observed { |
| - t.Helper() |
| - next, err := Update(ctx, s, cur, tr) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - cur = next |
| - return next |
| - } |
| - return apply, func() Observed { return cur } |
| + return s, o |
| } |
| // copyOf is a deep copy, for trying a transition without writing it. |
| @@ -1139,6 +1229,193 @@ func TestToInstalledAtJournalsItsStart(t *testing.T) { |
| t.Fatalf("installed: %s %v", o.Claim.Op.Phase, o.Claim.Op.Journal) |
| } |
| } |
| + |
| +// Astra round 6, finding 3: A to B completes, then a successor B to C |
| +// asks to start before B has discharged its probe's participation (C10). |
| +// Through the real start CAS: a drain or rebalance from B is refused, with |
| +// the claim unchanged, until MarkCompletedDischarged; a failover that |
| +// cites B's termination fence is not, since a dead B's slot binds nothing |
| +// on C. |
| +func TestRelocationSuccessorWaitsForTheCompletedDischarge(t *testing.T) { |
| + ctx := context.Background() |
| + successor := func(reason string, fence *FenceRef) RelocationStart { |
| + return RelocationStart{OpID: "rel-2", By: "worker", Now: relObserved.Add(time.Hour), Region: "ie", Reason: reason, |
| + SourceBox: relTarget, SourceEnrollment: relEnrollment(relTarget), TargetBox: relSpare, TargetEnrollment: relEnrollment(relSpare), |
| + DirectoryRevision: 9, AdmissionID: "adm-2", Fence: fence} |
| + } |
| + completed := func(t *testing.T) (release.Store, Observed) { |
| + t.Helper() |
| + s, o := probedDrainIn(t) |
| + o = step(t, s, o, RecordProbeStop("rel-1", stopOf(o))) |
| + o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| + if done, ok := ReadCompletedRelocation(o.Claim); !ok || done.Discharged || o.Claim.Op.Exec != relTarget { |
| + t.Fatalf("A to B did not complete undischarged: %+v %v", done, ok) |
| + } |
| + return s, o |
| + } |
| + t.Run("a drain from B waits for B's discharge", func(t *testing.T) { |
| + s, o := completed(t) |
| + for _, reason := range []string{ReasonDrain, ReasonRebalance} { |
| + if _, err := StartRelocation(ctx, s, o, successor(reason, nil), relAuthority{}); !errors.Is(err, ErrPendingChore) { |
| + t.Fatalf("%s started before the discharge: %v", reason, err) |
| + } |
| + } |
| + if now, err := Read(ctx, s, o.Claim.Host); err != nil || now.ETag != o.ETag { |
| + t.Fatalf("a refused start wrote the claim: %v", err) |
| + } |
| + if _, err := Update(ctx, s, o, MarkCompletedDischarged("rel-9")); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("another op's discharge: %v", err) |
| + } |
| + o = step(t, s, o, MarkCompletedDischarged("rel-1")) |
| + if _, err := Update(ctx, s, o, MarkCompletedDischarged("rel-1")); !errors.Is(err, ErrAlreadyRecorded) { |
| + t.Fatalf("a repeat: %v", err) |
| + } |
| + if done, ok := ReadCompletedRelocation(o.Claim); !ok || !done.Discharged || done.Stop != stopOf(o) { |
| + t.Fatalf("the mark changed the record: %+v", done) |
| + } |
| + next, err := StartRelocation(ctx, s, o, successor(ReasonDrain, nil), relAuthority{}) |
| + if err != nil { |
| + t.Fatalf("the successor after the discharge: %v", err) |
| + } |
| + if next.Claim.Op.ID != "rel-2" || next.Claim.Op.Intent["source_box"] != relTarget || next.Claim.Op.Intent["target_box"] != relSpare { |
| + t.Fatalf("successor op %+v", next.Claim.Op) |
| + } |
| + }) |
| + t.Run("a failover from a fenced B does not wait", func(t *testing.T) { |
| + s, o := completed(t) |
| + ref := putFence(t, s, relTarget) |
| + next, err := StartRelocation(ctx, s, o, successor(ReasonFailover, &ref), relAuthority{}) |
| + if err != nil { |
| + t.Fatalf("failover from the fenced source: %v", err) |
| + } |
| + if next.Claim.Op.Exec != "none" || !slices.Contains(next.Claim.Lost, relTarget) { |
| + t.Fatalf("failover op %+v lost %v", next.Claim.Op, next.Claim.Lost) |
| + } |
| + }) |
| + t.Run("no completed relocation, nothing to mark", func(t *testing.T) { |
| + _, o := relServing(t) |
| + if err := MarkCompletedDischarged("rel-1")(copyOf(t, o.Claim)); !errors.Is(err, ErrWrongState) { |
| + t.Fatalf("%v", err) |
| + } |
| + }) |
| +} |
| +``` |
| + |
| +`internal/activator/fenceowner_test.go` (new): |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "errors" |
| + "fmt" |
| + "testing" |
| + "time" |
| +) |
| + |
| +// TestFenceAndStopAsMeetsOnlyItsOwnFence: an owned fence is the caller's |
| +// again only when it holds at the same generation with the same owner. |
| +// Any other fence (a plain one at a lower, equal or higher generation, or |
| +// another owner's at the same generation) refuses with ErrFenceHeld and |
| +// stays exactly as it was, with nothing stopped. |
| +func TestFenceAndStopAsMeetsOnlyItsOwnFence(t *testing.T) { |
| + t.Run("own fence, re-collected", func(t *testing.T) { |
| + a, e, _, _, _ := maintenanceAwake(t) |
| + proof, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1") |
| + if err != nil || !proof.Matches(e.host, 12) { |
| + t.Fatalf("first install: %v", err) |
| + } |
| + again, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1") |
| + if err != nil || !again.Matches(e.host, 12) { |
| + t.Fatalf("the owner's repeat: %v", err) |
| + } |
| + if g := a.FenceGeneration(e.host); g != 12 { |
| + t.Fatalf("generation %d, want 12", g) |
| + } |
| + }) |
| + type held struct { |
| + gen uint64 |
| + owner string |
| + } |
| + for _, h := range []held{{5, ""}, {12, ""}, {20, ""}, {12, "owner-2"}, {5, "owner-1"}} { |
| + t.Run(fmt.Sprintf("held %d by %q", h.gen, h.owner), func(t *testing.T) { |
| + a, e, main, _, _ := maintenanceAwake(t) |
| + a.mu.Lock() |
| + a.fences = map[string]uint64{e.host: h.gen} |
| + if h.owner != "" { |
| + a.fenceOwners = map[string]string{e.host: h.owner} |
| + } |
| + a.mu.Unlock() |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1"); !errors.Is(err, ErrFenceHeld) { |
| + t.Fatalf("err = %v, want ErrFenceHeld", err) |
| + } |
| + a.mu.Lock() |
| + gen, owner := a.fences[e.host], a.fenceOwners[e.host] |
| + a.mu.Unlock() |
| + if gen != h.gen || owner != h.owner { |
| + t.Fatalf("held fence is %d by %q, want %d by %q", gen, owner, h.gen, h.owner) |
| + } |
| + if main.calls.Load() != 0 || !a.Owns(e.host) { |
| + t.Fatal("a refused owned fence stopped the host") |
| + } |
| + }) |
| + } |
| + t.Run("an empty owner", func(t *testing.T) { |
| + a, e, _, _, _ := maintenanceAwake(t) |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, ""); !errors.Is(err, ErrFenceGeneration) || a.Fenced(e.host) { |
| + t.Fatalf("err = %v, fenced %v", err, a.Fenced(e.host)) |
| + } |
| + }) |
| +} |
| + |
| +// TestAPlainFenceTakesOwnershipAway: FenceAndStop at the same generation |
| +// over an owned fence leaves a fence that is no longer the owner's, so |
| +// the owner can neither re-collect under it nor lift it. |
| +func TestAPlainFenceTakesOwnershipAway(t *testing.T) { |
| + a, e, _, _, _ := maintenanceAwake(t) |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := a.FenceAndStop(maintenanceContext(t, time.Second), e.host, 12); !errors.Is(err, ErrStopUnproven) && err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1"); !errors.Is(err, ErrFenceHeld) { |
| + t.Fatalf("err = %v, want ErrFenceHeld", err) |
| + } |
| + if a.LiftFenceAs(e.host, 12, "owner-1") { |
| + t.Fatal("the owner lifted a fence it no longer owns") |
| + } |
| + if !a.Fenced(e.host) { |
| + t.Fatal("the plain fence was lifted") |
| + } |
| +} |
| + |
| +// TestLiftFenceAsLiftsOnlyTheOwnersFence: the lift names the generation |
| +// and the owner, and a plain LiftFence also clears the owner. |
| +func TestLiftFenceAsLiftsOnlyTheOwnersFence(t *testing.T) { |
| + a, e, _, _, _ := maintenanceAwake(t) |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 12, "owner-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if a.LiftFenceAs(e.host, 12, "owner-2") || a.LiftFenceAs(e.host, 11, "owner-1") || a.LiftFenceAs(e.host, 12, "") { |
| + t.Fatal("lifted with the wrong owner or generation") |
| + } |
| + if !a.LiftFenceAs(e.host, 12, "owner-1") || a.Fenced(e.host) { |
| + t.Fatal("the owner's lift did not lift its fence") |
| + } |
| + if _, err := a.FenceAndStopAs(maintenanceContext(t, time.Second), e.host, 13, "owner-1"); err != nil && !errors.Is(err, ErrStopUnproven) { |
| + t.Fatal(err) |
| + } |
| + if !a.LiftFence(e.host, 13) { |
| + t.Fatal("plain lift") |
| + } |
| + a.mu.Lock() |
| + _, left := a.fenceOwners[e.host] |
| + a.mu.Unlock() |
| + if left { |
| + t.Fatal("a plain lift left the owner behind") |
| + } |
| +} |
| ``` |
| `internal/hostrelocate/participant_gate_test.go` (new): |
| @@ -2130,7 +2407,29 @@ func TestVerifyReceiptChecksTheReceiptOnly(t *testing.T) { |
| } |
| func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) { |
| -@@ -71,6 +79,9 @@ func (b *fakeBox) Build(context.Context, hostclaim.Observed) error { |
| +@@ -48,10 +56,19 @@ func (b *fakeBox) Retained() ([]Participation, error) { |
| + } |
| + return []Participation{{Host: host, Op: b.op}}, b.retainedErr |
| + } |
| +-func (b *fakeBox) Finish(_ context.Context, h, op string) error { |
| ++func (b *fakeBox) Finish(ctx context.Context, h, op string) error { |
| + b.finished = true |
| + b.effects = append(b.effects, "finish "+op) |
| +- return nil |
| ++ // Production's Finish is the target's hold release, which ends by |
| ++ // marking the completed relocation discharged (region HA ruling 76). |
| ++ obs, err := hostclaim.Read(ctx, b.f.s, h) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if done, ok := hostclaim.ReadCompletedRelocation(obs.Claim); ok && done.Op == op && done.Stop.Box == b.box && !done.Discharged { |
| ++ _, err = hostclaim.Update(ctx, b.f.s, obs, hostclaim.MarkCompletedDischarged(op)) |
| ++ } |
| ++ return err |
| + } |
| + func (b *fakeBox) Ready(context.Context, hostclaim.Observed) error { |
| + b.effects = append(b.effects, "ready") |
| +@@ -71,6 +88,9 @@ func (b *fakeBox) Build(context.Context, hostclaim.Observed) error { |
| } |
| func (b *fakeBox) InstallLocal(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) { |
| b.effects = append(b.effects, "install") |
| @@ -2140,7 +2439,29 @@ func TestVerifyReceiptChecksTheReceiptOnly(t *testing.T) { |
| n, _ := strconv.ParseUint(o.Claim.Op.Journal["install_stamp"], 10, 64) |
| return hostclaim.LocalInstall{Box: b.box, Enrollment: enrollment(b.box), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil |
| } |
| -@@ -390,3 +401,16 @@ func TestAParticipantRestoresALocalInstallRetiredAfterItEntersProbing(t *testing |
| +@@ -273,11 +293,19 @@ func TestAParticipantFinishesItsControllerOnceTheOperationIsOver(t *testing.T) { |
| + if dst.opened != before+1 { |
| + t.Fatalf("the second relocation did not open its own controller: %d -> %d", before, dst.opened) |
| + } |
| +- // Mid-operation, Sweep keeps the controller. |
| ++ // No third relocation starts from spa until spa has finished the |
| ++ // second one (region HA ruling 77); then, mid-operation, Sweep keeps |
| ++ // the controller. |
| ++ pspa := Participant{Claims: f.s, Box: "spa", Enrollment: enrollment("spa"), Effects: spa, Now: f.clock} |
| ++ if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "spa", Target: "sp2"}); !errors.Is(err, hostclaim.ErrPendingChore) { |
| ++ t.Fatalf("a successor started before spa finished: %v", err) |
| ++ } |
| ++ if err := pspa.Sweep(ctx); err != nil || !spa.finished { |
| ++ t.Fatalf("spa kept the second relocation's controller: %v", err) |
| ++ } |
| + if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "spa", Target: "sp2"}); err != nil { |
| + t.Fatal(err) |
| + } |
| +- pspa := Participant{Claims: f.s, Box: "spa", Enrollment: enrollment("spa"), Effects: spa, Now: f.clock} |
| + spa.finished = false |
| + if _, err := pspa.Step(ctx, host); err != nil { |
| + t.Fatal(err) |
| +@@ -390,3 +418,16 @@ func TestAParticipantRestoresALocalInstallRetiredAfterItEntersProbing(t *testing |
| t.Fatalf("restores %d, probes over a missing install %d, reinstalls after P %d", dst.restores, dst.probedMissing, dst.reinstalls) |
| } |
| } |
| @@ -2184,15 +2505,16 @@ Delete `internal/host/relocationlifecycle_linux_test.go`: they test stage A's co |
| Run: |
| ```bash |
| -nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelease|TestRelocationRelease|TestRelocationJournalsAreSwept|TestRelocationAcknowledges|TestCensusInitializesWithOld)' |
| +nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelease|TestRelocationRelease|TestRelocationJournals|TestRelocationAcknowledges|TestCensusInitializesWithOld)' |
| nice -n 19 go test ./internal/hostclaim/ ./internal/hostrelocate/ -count=1 |
| +nice -n 19 go test ./internal/activator/ -count=1 -run 'TestFenceAndStopAs|TestAPlainFence|TestLiftFenceAs' |
| ``` |
| -Expected: FAIL to compile: `undefined: hostclaim.ProbeStop`. |
| +Expected: FAIL to compile: `undefined: hostclaim.ProbeStop`, `a.FenceAndStopAs undefined`. |
| - [ ] **Step 3: The claim's probe-stop primitives** |
| -Ruling 49 step 4, ruling 56's record, ruling 60's `installed_at`, and ruling 67's `Probe.At`. |
| +Ruling 49 step 4, ruling 56's record, ruling 60's `installed_at`, ruling 67's `Probe.At`, and ruling 77's discharge mark and start guard. |
| `internal/hostclaim/relocation.go`: |
| @@ -2210,7 +2532,21 @@ Ruling 49 step 4, ruling 56's record, ruling 60's `installed_at`, and ruling 67' |
| } |
| // Target names a box for the relocation's target slot. |
| -@@ -1009,6 +1013,11 @@ func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| +@@ -258,6 +262,13 @@ func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced |
| + if c.Holder == nil || c.Op.Kind != KindServing || c.Op.ID != "" { |
| + return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind) |
| + } |
| ++ // A completed relocation whose box has not yet discharged its probe's |
| ++ // participation (C10, MarkCompletedDischarged) keeps that box's slot |
| ++ // for the old op, so a successor from it would never be prepared. A |
| ++ // failover from a fenced source does not need the source's slot. |
| ++ if done, ok := ReadCompletedRelocation(c); ok && !done.Discharged && !(sourceFenced && source == done.Stop.Box) { |
| ++ return fmt.Errorf("%w: relocation %s is not yet discharged on %s", ErrPendingChore, done.Op, done.Stop.Box) |
| ++ } |
| + for _, o := range c.Obligations { |
| + switch { |
| + case o.Kind == "park-replica" && o.Region == region: |
| +@@ -1009,6 +1020,11 @@ func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| if !readJournal(c.Op, "probe", &p) || p.Box != c.Op.Exec || p.Session != c.Holder.Session || c.Holder.WriterBox != p.Box { |
| return fmt.Errorf("%w: no passing probe of the current session", ErrRelocation) |
| } |
| @@ -2222,7 +2558,7 @@ Ruling 49 step 4, ruling 56's record, ruling 60's `installed_at`, and ruling 67' |
| target := c.Op.Exec |
| var owed []string |
| readJournal(c.Op, "installed_boxes", &owed) |
| -@@ -1052,7 +1061,21 @@ func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| +@@ -1052,7 +1068,21 @@ func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-generation", Region: region, Generation: g, Retirement: newID(), Owner: "region:" + region}) |
| } |
| } |
| @@ -2314,6 +2650,12 @@ type CompletedRelocation struct { |
| Stamp uint64 `json:"stamp"` // the stopped session's installation stamp |
| Phase string `json:"phase"` // the stop's phase: probing |
| Generation string `json:"generation"` // the stopped session's generation |
| + // Discharged is set by MarkCompletedDischarged once the box the |
| + // relocation completed on has discharged its probe's participation |
| + // (claim ask C10). Until then no successor relocation starts from that |
| + // box (RelocationStartGuard): its participation slot is still the old |
| + // op's, and main's Prepare for a successor would answer ErrLost. |
| + Discharged bool `json:"discharged,omitempty"` |
| } |
| // JournalCompleted is the serving op's journal key for CompletedRelocation. |
| @@ -2328,6 +2670,28 @@ func ReadCompletedRelocation(c *Claim) (CompletedRelocation, bool) { |
| return r, true |
| } |
| +// MarkCompletedDischarged records, on the serving op, that the box the |
| +// relocation opID completed on has discharged its participation. It |
| +// changes nothing else; a repeat is ErrAlreadyRecorded. A serving op with |
| +// no completed relocation is ErrWrongState, and one that records another |
| +// op is ErrRelocation. |
| +func MarkCompletedDischarged(opID string) func(*Claim) error { |
| + return func(c *Claim) error { |
| + r, ok := ReadCompletedRelocation(c) |
| + if !ok { |
| + return fmt.Errorf("%w: no completed relocation on the serving op", ErrWrongState) |
| + } |
| + if r.Op != opID { |
| + return fmt.Errorf("%w: the serving op records %s, not %s", ErrRelocation, r.Op, opID) |
| + } |
| + if r.Discharged { |
| + return ErrAlreadyRecorded |
| + } |
| + r.Discharged = true |
| + return journalJSON(c, JournalCompleted, r) |
| + } |
| +} |
| + |
| // ToInstalledAt is ToInstalled that also journals installed_at, the start |
| // of the installed bound (ruling 60). |
| func ToInstalledAt(opID string, a PreparedAuthority, now time.Time) func(*Claim) error { |
| @@ -2345,6 +2709,134 @@ func ToInstalledAt(opID string, a PreparedAuthority, now time.Time) func(*Claim) |
| } |
| ``` |
| +The activator's owned fence (ruling 75). `FenceAndStopAs` is `FenceAndStopIfUnfenced` that also meets again exactly the fence its owner installed at that generation; a plain install or lift clears the owner. |
| + |
| +`internal/activator/maintenance.go`: |
| + |
| +```diff |
| +--- a/internal/activator/maintenance.go |
| ++++ b/internal/activator/maintenance.go |
| +@@ -51,7 +51,7 @@ func (o *stopObservation) complete() bool { |
| + // ErrStopUnproven guarantees this call installed a fence, but proves no stop. |
| + // A missing entry is not evidence that a prior process or replica stopped. |
| + func (a *Service) FenceAndStop(ctx context.Context, host string, generation uint64) (*StopEvidence, error) { |
| +- return a.fenceAndStop(ctx, host, generation, false) |
| ++ return a.fenceAndStop(ctx, host, generation, false, "") |
| + } |
| + |
| + // FenceAndStopIfUnfenced is FenceAndStop for a caller that must never take |
| +@@ -62,10 +62,28 @@ func (a *Service) FenceAndStop(ctx context.Context, host string, generation uint |
| + // generation, it answers ErrFenceHeld, installs nothing and stops nothing. |
| + // Otherwise it is FenceAndStop, answers included. |
| + func (a *Service) FenceAndStopIfUnfenced(ctx context.Context, host string, generation uint64) (*StopEvidence, error) { |
| +- return a.fenceAndStop(ctx, host, generation, true) |
| ++ return a.fenceAndStop(ctx, host, generation, true, "") |
| + } |
| + |
| +-func (a *Service) fenceAndStop(ctx context.Context, host string, generation uint64, exclusive bool) (*StopEvidence, error) { |
| ++// FenceAndStopAs is FenceAndStopIfUnfenced for a caller that must be able |
| ++// to meet its own fence again: a retry after a lost answer, or after the |
| ++// caller's own later step failed. owner is the caller's persisted name for |
| ++// this one fence. Under the lock that installs, a held fence is the |
| ++// caller's only when it was installed by FenceAndStopAs with exactly this |
| ++// owner at exactly this generation; then the call is FenceAndStop under it |
| ++// (it re-collects the stop evidence). Any other held fence, of any |
| ++// generation and however installed, answers ErrFenceHeld, installs nothing |
| ++// and stops nothing. Fences live only in this process, so after a restart |
| ++// no fence is the caller's, whatever it persisted. An empty owner is |
| ++// ErrFenceGeneration. |
| ++func (a *Service) FenceAndStopAs(ctx context.Context, host string, generation uint64, owner string) (*StopEvidence, error) { |
| ++ if owner == "" { |
| ++ return nil, ErrFenceGeneration |
| ++ } |
| ++ return a.fenceAndStop(ctx, host, generation, true, owner) |
| ++} |
| ++ |
| ++func (a *Service) fenceAndStop(ctx context.Context, host string, generation uint64, exclusive bool, owner string) (*StopEvidence, error) { |
| + if a == nil || !controlname.ValidHostShape(host) || generation == 0 { |
| + return nil, ErrFenceGeneration |
| + } |
| +@@ -100,17 +118,32 @@ func (a *Service) fenceAndStop(ctx context.Context, host string, generation uint |
| + a.fences = make(map[string]uint64) |
| + } |
| + prior := a.fences[host] |
| ++ own := owner != "" && prior == generation && a.fenceOwners[host] == owner |
| + // Only before this call installs: on a later round of the loop the |
| + // fence held is this call's own. |
| +- if exclusive && !installed && prior != 0 { |
| ++ if exclusive && !installed && prior != 0 && !own { |
| + a.mu.Unlock() |
| + return nil, ErrFenceHeld |
| + } |
| ++ if owner != "" && installed && !own { |
| ++ // An owned fence replaced while this call waited is no longer |
| ++ // this call's: it installed one, and proves nothing more. |
| ++ a.mu.Unlock() |
| ++ return nil, ErrStopUnproven |
| ++ } |
| + if prior > generation { |
| + a.mu.Unlock() |
| + return nil, ErrFenceGeneration |
| + } |
| + a.fences[host] = generation |
| ++ if owner != "" { |
| ++ if a.fenceOwners == nil { |
| ++ a.fenceOwners = make(map[string]string) |
| ++ } |
| ++ a.fenceOwners[host] = owner |
| ++ } else { |
| ++ delete(a.fenceOwners, host) |
| ++ } |
| + installed = true |
| + e := a.entries[host] |
| + if e == nil { |
| +@@ -165,6 +198,25 @@ func (a *Service) LiftFence(host string, generation uint64) bool { |
| + return false |
| + } |
| + delete(a.fences, host) |
| ++ delete(a.fenceOwners, host) |
| ++ return true |
| ++} |
| ++ |
| ++// LiftFenceAs is LiftFence for an owned fence: it lifts host's fence only |
| ++// when it is exactly the one FenceAndStopAs installed with owner at |
| ++// generation, and reports whether it lifted anything. A fence another |
| ++// operation installed at the same generation stays. |
| ++func (a *Service) LiftFenceAs(host string, generation uint64, owner string) bool { |
| ++ if a == nil || generation == 0 || owner == "" { |
| ++ return false |
| ++ } |
| ++ a.mu.Lock() |
| ++ defer a.mu.Unlock() |
| ++ if a.fences[host] != generation || a.fenceOwners[host] != owner { |
| ++ return false |
| ++ } |
| ++ delete(a.fences, host) |
| ++ delete(a.fenceOwners, host) |
| + return true |
| + } |
| + |
| +``` |
| + |
| +`internal/activator/activator.go`: |
| + |
| +```diff |
| +--- a/internal/activator/activator.go |
| ++++ b/internal/activator/activator.go |
| +@@ -943,7 +943,10 @@ type Service struct { |
| + // converges (recycleWorkerLocked), so the two share one pace. Under a.mu. |
| + convergeBudget int |
| + |
| +- fences map[string]uint64 |
| ++ fences map[string]uint64 |
| ++ // fenceOwners names the owner of an owned fence (FenceAndStopAs), by |
| ++ // host. A fence installed any other way has no entry. Under a.mu. |
| ++ fenceOwners map[string]string |
| + tokenPreparations map[string]chan struct{} |
| + stopHistory map[string]*stopObservation |
| + // straySidecars are legacy sidecars whose termination is unproven, per |
| +``` |
| + |
| - [ ] **Step 4: The participant's gate and probe stop, and the worker's bounds** |
| Ruling 48 (nothing counted while root refuses the kind), ruling 58 (`ErrLaterAttemptUnlaunchable` until C11), ruling 62 (`sameInstallation`), ruling 49 (the probe-stop branch), and rulings 60 and 67 (the bounds). |
| @@ -3109,11 +3601,13 @@ import ( |
| // RelocationActivator is the part of activator.Service relocation uses: |
| // the probe attempt (Task 8d) and the local fence that ends a passed |
| -// probe's session before the serving CAS (ruling 49). |
| +// probe's session before the serving CAS (ruling 49). The fence is an |
| +// owned one (ruling 75): taken and met again only under the owner the |
| +// probe stop persisted, and lifted only under it. |
| type RelocationActivator interface { |
| ProbeWake(context.Context, activator.ProbeConfig) (activator.ProbeResult, error) |
| - FenceAndStop(context.Context, string, uint64) (*activator.StopEvidence, error) |
| - LiftFence(string, uint64) bool |
| + FenceAndStopAs(context.Context, string, uint64, string) (*activator.StopEvidence, error) |
| + LiftFenceAs(string, uint64, string) bool |
| } |
| var _ RelocationActivator = (*activator.Service)(nil) |
| @@ -3415,7 +3909,8 @@ import ( |
| // sweepRelocationJournals is the lifecycle half of N9 (ruling 61): every |
| // region HA journal under <DataDir>/relocate names its host inside and its |
| // op in its path, and is removed once the host's claim no longer runs that |
| -// op (completed, replaced by a successor, or the claim gone). Positions, |
| +// op (completed and marked discharged, replaced by a successor, or the |
| +// claim gone). Positions, |
| // attempts and install journals are needed only while their op runs; |
| // the build scratch only while its op is installing. A journal that |
| // cannot be read is left in place and reported: it is census-opaque, so |
| @@ -3428,7 +3923,16 @@ func sweepRelocationJournals(ctx context.Context, dataDir string, claims release |
| if err != nil { |
| return false, err |
| } |
| - return obs.Absent || obs.Claim == nil || obs.Claim.Op.ID != op, nil |
| + if obs.Absent || obs.Claim == nil { |
| + return true, nil |
| + } |
| + // Ruling 76: a completed relocation's journals, the probe stop's |
| + // fence owner among them, stay until the box it completed on has |
| + // discharged, released its hold, lifted its fence and marked it. |
| + if done, ok := hostclaim.ReadCompletedRelocation(obs.Claim); ok && done.Op == op && !done.Discharged { |
| + return false, nil |
| + } |
| + return obs.Claim.Op.ID != op, nil |
| } |
| hostOf := func(path string) (string, error) { |
| raw, err := os.ReadFile(path) |
| @@ -3455,7 +3959,7 @@ func sweepRelocationJournals(ctx context.Context, dataDir string, claims release |
| for _, e := range ops { |
| dir := filepath.Join(root, "ops", e.Name()) |
| host, readErr := "", error(nil) |
| - for _, name := range []string{"install.json", "attempts.json"} { |
| + for _, name := range []string{"install.json", "attempts.json", "probe-fence.json"} { |
| h, err := hostOf(filepath.Join(dir, name)) |
| if err == nil && h != "" { |
| host = h |
| @@ -3532,7 +4036,8 @@ Run: |
| ```bash |
| nice -n 19 go test ./internal/hostclaim/ ./internal/hostrelocate/ -count=1 |
| -TMPDIR=$(mktemp -d /tmp/k.XXXX) nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelease|TestRelocationRelease|TestEnsureLocal|TestVerifyReceipt|TestCensusInitializesWithOld|TestRelocationJournalsAreSwept|TestRelocationAcknowledges|TestPlan9|TestPlan11|TestClaimCensus)' |
| +nice -n 19 go test ./internal/activator/ -count=1 -run 'TestFenceAndStopAs|TestAPlainFence|TestLiftFenceAs|TestFenceAndStopIfUnfenced|TestFenced|TestFenceGeneration|TestMaintenance' |
| +TMPDIR=$(mktemp -d /tmp/k.XXXX) nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelease|TestRelocationRelease|TestEnsureLocal|TestVerifyReceipt|TestCensusInitializesWithOld|TestRelocationJournals|TestRelocationAcknowledges|TestPlan9|TestPlan11|TestClaimCensus)' |
| ``` |
| Expected: PASS. Run outside the sandbox (the `TestPlan9`/`TestPlan11` regressions listen on unix sockets), with `TMPDIR=$(mktemp -d /tmp/k.XXXX)`, removed afterwards. |
| @@ -3541,8 +4046,8 @@ Expected: PASS. Run outside the sandbox (the `TestPlan9`/`TestPlan11` regression |
| ```bash |
| git rm internal/host/relocationcontrollers_linux_test.go internal/host/relocationlifecycle_linux_test.go |
| -git add internal/host/claimparticipation_relocation.go internal/host/relocationassembly.go internal/host/relocationcensus_test.go internal/host/relocationflush_adoption_test.go internal/host/relocationparticipation_linux_test.go internal/host/relocationsweep.go internal/host/relocationsweep_test.go internal/hostclaim/relocation_probestop.go internal/hostclaim/relocation_probestop_test.go internal/hostrelocate/participant_gate_test.go internal/hostrelocate/worker_bounds_test.go internal/host/claimcensus.go internal/host/claimparticipation.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationflush.go internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostrelocate/participant.go internal/hostrelocate/participant_test.go internal/hostrelocate/worker.go internal/hostrelocate/worker_test.go |
| -git commit -m "host: relocation reads main's participation owner and releases in Plan 11's order; hostclaim: a relocation completes only after its passed probe is stopped with root's receipt |
| +git add internal/activator/fenceowner_test.go internal/activator/activator.go internal/activator/maintenance.go internal/host/claimparticipation_relocation.go internal/host/relocationassembly.go internal/host/relocationcensus_test.go internal/host/relocationflush_adoption_test.go internal/host/relocationparticipation_linux_test.go internal/host/relocationsweep.go internal/host/relocationsweep_test.go internal/hostclaim/relocation_probestop.go internal/hostclaim/relocation_probestop_test.go internal/hostrelocate/participant_gate_test.go internal/hostrelocate/worker_bounds_test.go internal/host/claimcensus.go internal/host/claimparticipation.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationflush.go internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostrelocate/participant.go internal/hostrelocate/participant_test.go internal/hostrelocate/worker.go internal/hostrelocate/worker_test.go |
| +git commit -m "host: relocation reads main's participation owner and releases in Plan 11's order; hostclaim: a relocation completes only after its passed probe is stopped with root's receipt, and no successor starts before its target discharges it; activator: owned fences |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ``` |
| @@ -4971,7 +5476,7 @@ func sameEntries(a, b hostfiles.Manifest) bool { |
| ```diff |
| --- a/internal/host/relocationassembly.go |
| +++ b/internal/host/relocationassembly.go |
| -@@ -102,6 +102,7 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| +@@ -104,6 +104,7 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| FilesDir: func(host string) string { return paths(host).FilesDir }, EnvPath: func(host string) string { return paths(host).EnvPath }}.Flush, |
| Positions: ReleasePositionsFile{Dir: cfg.ReleaseDir}, |
| PreflightKind: a.relocationPreflight(cfg), |
| @@ -7219,7 +7724,7 @@ func (a ClaimAssembly) relocationReceipt(cfg RelocationAssemblyConfig) func(cont |
| "github.com/carlosframework/platform/internal/lease" |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/registry" |
| -@@ -49,6 +51,7 @@ type RelocationAssemblyConfig struct { |
| +@@ -51,6 +53,7 @@ type RelocationAssemblyConfig struct { |
| Region, LeaseRegion string |
| Registry *registry.Registry |
| DataDir string // hostguard.Guard.DataDir, the managed data root |
| @@ -7227,7 +7732,7 @@ func (a ClaimAssembly) relocationReceipt(cfg RelocationAssemblyConfig) func(cont |
| WorkDir string // the build's scratch: <DataDir>/relocate/build |
| ReleaseDir string // <DataDir>/relocate/release |
| SecretsKey *ecdh.PrivateKey |
| -@@ -61,6 +64,11 @@ type RelocationAssemblyConfig struct { |
| +@@ -63,6 +66,11 @@ type RelocationAssemblyConfig struct { |
| // writer's stamp, not only at install_stamp (ruling 58). |
| LaterProbeAttempts bool |
| Now func() time.Time |
| @@ -7239,7 +7744,7 @@ func (a ClaimAssembly) relocationReceipt(cfg RelocationAssemblyConfig) func(cont |
| } |
| // RelocationEffects binds every provider to this bound assembly. ack is |
| -@@ -103,6 +111,9 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| +@@ -105,6 +113,9 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| Positions: ReleasePositionsFile{Dir: cfg.ReleaseDir}, |
| PreflightKind: a.relocationPreflight(cfg), |
| BuildP: a.relocationBuild(cfg), |
| @@ -8044,15 +8549,13 @@ func (a *relocationProbeActivator) ProbeWake(_ context.Context, pc activator.Pro |
| return a.result, a.err |
| } |
| -func (a *relocationProbeActivator) FenceAndStop(context.Context, string, uint64) (*activator.StopEvidence, error) { |
| - return nil, nil |
| -} |
| - |
| -func (a *relocationProbeActivator) FenceAndStopIfUnfenced(context.Context, string, uint64) (*activator.StopEvidence, error) { |
| - return nil, nil |
| +// The probe attempt never fences: a fence asked of this double fails the |
| +// test's call with an error naming it, and lifts nothing. |
| +func (a *relocationProbeActivator) FenceAndStopAs(context.Context, string, uint64, string) (*activator.StopEvidence, error) { |
| + return nil, errors.New("the probe attempt asked for a fence") |
| } |
| -func (a *relocationProbeActivator) LiftFence(string, uint64) bool { return true } |
| +func (a *relocationProbeActivator) LiftFenceAs(string, uint64, string) bool { return false } |
| // One probe attempt is one ProbeWake, bounded by the probe deadline, |
| // carrying P's install manifest on P's opening; a failed or overdue |
| @@ -9477,7 +9980,7 @@ func (a ClaimAssembly) relocationAdmit(cfg RelocationAssemblyConfig) func(contex |
| ```diff |
| --- a/internal/host/relocationassembly.go |
| +++ b/internal/host/relocationassembly.go |
| -@@ -114,6 +114,7 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| +@@ -116,6 +116,7 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| Install: a.relocationInstall(cfg), |
| VerifyLocal: a.relocationVerifyLocal(cfg), |
| Receipt: a.relocationReceipt(cfg), |
| @@ -9514,21 +10017,21 @@ Ruling 49's sequence on the target: the activator's fence and stop, main's write |
| C10 is committed here as region HA code behind the activation gate (rule 5 of rev 5): `completedRelocationDischarge` (`claimparticipation_completed.go`) and the binding argument to main's `productionClaimLocal.cleanup`, first in every cleanup path. The claim session still confirms it on #236; the flags stay off until it does. Its tests compile and run unconditionally. |
| -K19: `relocationStopProbe` takes the fence with `FenceAndStopIfUnfenced`, which answers `ErrFenceHeld` for another operation's fence (`FenceAndStop` would replace a lower generation, and `ReleaseHold`'s `LiftFence` would then reopen admission that operation closed). A repeat of the same probe stop finds its own fence, recorded under `relocate/ops/<op>/probe-fence.json` once installed, and re-collects with `FenceAndStop` at the same generation. The writer-session receipt is read by one function for the stop and for N6 (`writerSessionReceipt`, D5e). `DischargeCompleted` and `RelocationAssemblyConfig.Discharge` are here (K15). Rev 1's hold-only change to `hostbox/reconcile.go` is withdrawn (R5-1): stage A's `check` already refuses the target's reconcile before any write. |
| +Rulings 75 and 76 (Astra round 6 findings 1 to 3): `relocationStopProbe` persists its fence's owner (`relocate/ops/<op>/probe-fence.json`: host, session, owner) before it asks the activator, then takes the fence with `FenceAndStopAs`, which answers `ErrFenceHeld` for any fence that is not exactly the one it installed under that owner at that session, including one another operation installed after an agent restart. A failed write installs nothing and is retried; a repeat meets its own fence. `relocationReleaseHold` discharges, releases the hold, lifts the fence with `LiftFenceAs` under the persisted owner, and marks the completed relocation discharged, which is what lets a successor start from this box (ruling 77). The writer-session receipt is read by one function for the stop and for N6 (`writerSessionReceipt`, D5e). `DischargeCompleted` and `RelocationAssemblyConfig.Discharge` are here (K15). Rev 1's hold-only change to `hostbox/reconcile.go` is withdrawn (R5-1): stage A's `check` already refuses the target's reconcile before any write. |
| **Files:** |
| - Create: `internal/host/claimparticipation_completed.go`, `internal/host/relocationdischarge_linux_test.go`, `internal/host/relocationdischarge_restart_unix_test.go`, `internal/host/relocationhandover.go`, `internal/host/relocationhandover_unix_test.go`, `internal/host/relocationreleasehold_test.go`, `internal/host/relocationstopprobe_unix_test.go`, `internal/hostbox/reconcile_relocation_target_test.go` |
| - Modify: `internal/host/claimconstruction.go` (N6), `internal/host/claimhandover_crossbox.go` (N6), `internal/host/claimhandover_transport.go` (N6), `internal/host/claimparticipation.go` (C10: `cleanup` takes the binding), `internal/host/claimparticipation_relocation.go`, `internal/host/claimparticipation_stale.go` (C10), `internal/host/plan11_fix1_relocation_test.go` (K11: `wireClaimAdmission`'s sixth argument), `internal/host/plan9_production_test.go` (C10: `cleanup`'s binding), `internal/host/relocationassembly.go`, `internal/host/relocationowners.go` |
| **Interfaces:** |
| -- Consumes: `activator.Service.FenceAndStopIfUnfenced`, `FenceAndStop`, `ErrFenceHeld`, `ErrStopUnproven`, `LiftFence`; `ClaimWakeAssembly.Stop`; `hostguard.CompletedWriterStops`, `WriterStopObligation`, `FenceReceipt`, `BeginColdHandover`; `verifyRetirementReceipt`; `coldAdmissionHook`, `claimPredecessorStop`, `claimAdmissionHandover`, `wireClaimAdmission`; `productionClaimLocal.cleanup`, `prepareWith`, `clearUntracked`; `execlease.Binding`, `ClearStale`; `registry.ReleaseHold`; `hostclaim.ReadCompletedRelocation`. |
| -- Produces: `completedRelocationDischarge` and the three-argument `cleanup` (C10); `relocationCompletedHandover`, `writerSessionReceipt`, `coldHandoverWith`, `relocationBefore`, `relocationAfter`, `wireClaimAdmission`'s completed branch (N6); `(*productionClaimLocal).DischargeCompleted`; `RelocationAssemblyConfig.Discharge`; `RelocationActivator.FenceAndStopIfUnfenced`; `relocationStopProbe`, `relocationProbeFencePath`, `relocationRecordProbeFence`, `relocationOwnsProbeFence`, `relocationReleaseHold`; `RelocationEffects.StopProbe`, `ReleaseHold` set. |
| +- Consumes: `RelocationActivator.FenceAndStopAs`, `LiftFenceAs` (8a), `activator.ErrFenceHeld`, `ErrStopUnproven`; `hostclaim.MarkCompletedDischarged`, `ErrAlreadyRecorded` (8a); `ClaimWakeAssembly.Stop`; `hostguard.CompletedWriterStops`, `WriterStopObligation`, `FenceReceipt`, `BeginColdHandover`; `verifyRetirementReceipt`; `coldAdmissionHook`, `claimPredecessorStop`, `claimAdmissionHandover`, `wireClaimAdmission`; `productionClaimLocal.cleanup`, `prepareWith`, `clearUntracked`; `execlease.Binding`, `ClearStale`; `registry.ReleaseHold`; `hostclaim.ReadCompletedRelocation`. |
| +- Produces: `completedRelocationDischarge` and the three-argument `cleanup` (C10); `relocationCompletedHandover`, `writerSessionReceipt`, `coldHandoverWith`, `relocationBefore`, `relocationAfter`, `wireClaimAdmission`'s completed branch (N6); `(*productionClaimLocal).DischargeCompleted`; `RelocationAssemblyConfig.Discharge`; `relocationStopProbe`, `relocationProbeFencePath`, `relocationProbeFence`, `relocationProbeFenceOwner`, `readRelocationProbeFence`, `relocationReleaseHold` with its discharge mark; `RelocationEffects.StopProbe`, `ReleaseHold` set. |
| - [ ] **Step 1: Write the failing tests** |
| The discharge tests (`relocationdischarge_linux_test.go`) use Task 8a's participation box: main's real `productionClaimLocal`, its real controller opened by `Prepare` at installing, the probe's launch in the box's launch journal (so main's ordinary cleanup would ask root to stop that session), and root's writer-session receipt completed in the guard. With C10's call removed from `cleanup`, both fail; with it, both pass. `TestRelocationCompletedHandoverRoutesOnlyTheCompletedPrior` posts to a unix socket and runs outside the sandbox. The two hostbox reconcile tests pass before this task's code: they pin stage A's behaviour, which rev 5 keeps (R5-1). |
| -New tests: `TestRelocationCompletionDischargesParticipationWithoutRestart`, `TestRelocationRestartAfterServingCASRecoversParticipation`, `TestRelocationRestartDischargeSurvivesAFailedClaimRead`, `TestRelocationCompletedHandoverRoutesOnlyTheCompletedPrior`, `TestRelocationReleaseHoldWaitsForDischarge`, `TestRelocationStopProbeFencesStopsAndReadsRootsReceipt`, `TestReconcileLeavesARelocatingTargetsRowToItsInstall`, `TestReconcileAdapterClearsTheHoldAfterCompletion`. |
| +New tests: `TestRelocationCompletionDischargesParticipationWithoutRestart`, `TestRelocationRestartAfterServingCASRecoversParticipation`, `TestRelocationRestartDischargeSurvivesAFailedClaimRead`, `TestRelocationCompletedHandoverRoutesOnlyTheCompletedPrior`, `TestRelocationReleaseHoldWaitsForDischarge`, `TestRelocationSuccessorStartsOnlyAfterTheTargetsDischarge`, `TestRelocationStopProbeFencesStopsAndReadsRootsReceipt`, `TestRelocationStopProbeAfterARestartNeverTakesAForeignFence`, `TestRelocationStopProbeRetriesAFailedMarkerWrite`, `TestReconcileLeavesARelocatingTargetsRowToItsInstall`, `TestReconcileAdapterClearsTheHoldAfterCompletion`. |
| `internal/host/relocationdischarge_linux_test.go` (new): |
| @@ -10002,6 +10505,7 @@ import ( |
| "testing" |
| "time" |
| + "github.com/carlosframework/platform/internal/activator" |
| "github.com/carlosframework/platform/internal/hostbox" |
| "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/hostrelocate" |
| @@ -10009,12 +10513,16 @@ import ( |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| -// Ruling 66 (Astra round 3 finding 2): after the serving CAS, the target |
| -// releases its row's relocating hold and lifts the probe's activator fence |
| -// only once main's participation is discharged. A refused discharge |
| -// leaves both, so the edge keeps the row held and the activator wakes |
| -// nothing; a holder already moved off the probe session is refused; a |
| -// claim that does not serve on this box is left alone. |
| +// Ruling 66 (Astra round 3 finding 2) and ruling 76 (Astra round 6 |
| +// findings 1 and 3): after the serving CAS, the target discharges main's |
| +// participation, releases its row's relocating hold, lifts the fence its |
| +// probe stop installed under the persisted owner, and marks the completed |
| +// relocation discharged, in that order. A refused discharge leaves all of |
| +// it, so the edge keeps the row held, the activator wakes nothing and no |
| +// successor starts; a holder moved off the probe session while the row is |
| +// held is refused; another operation's fence is never lifted; a replay |
| +// after the hold went only lifts and marks; a claim that does not serve on |
| +// this box is left alone. |
| func TestRelocationReleaseHoldWaitsForDischarge(t *testing.T) { |
| const host, box = "shop.example", "dst" |
| ctx := t.Context() |
| @@ -10035,18 +10543,20 @@ func TestRelocationReleaseHoldWaitsForDischarge(t *testing.T) { |
| return c |
| } |
| type world struct { |
| - order []string |
| - reg *registry.Registry |
| - act *stopProbeActivator |
| - hold func(context.Context, string, string) error |
| - } |
| - setup := func(t *testing.T, c *hostclaim.Claim, discharge error) *world { |
| + order []string |
| + store release.Store |
| + reg *registry.Registry |
| + act *stopProbeActivator |
| + owner string |
| + discharge error |
| + hold func(context.Context, string, string) error |
| + } |
| + setup := func(t *testing.T, c *hostclaim.Claim) *world { |
| t.Helper() |
| - w := &world{} |
| - store := release.NewDirStore(t.TempDir()) |
| + w := &world{store: release.NewDirStore(t.TempDir())} |
| raw, _ := json.Marshal(c) |
| key, _ := hostclaim.Key(host) |
| - if _, err := store.Put(ctx, key, raw, ""); err != nil { |
| + if _, err := w.store.Put(ctx, key, raw, ""); err != nil { |
| t.Fatal(err) |
| } |
| if w.reg, err = registry.Open(filepath.Join(t.TempDir(), "registry.db")); err != nil { |
| @@ -10056,18 +10566,30 @@ func TestRelocationReleaseHoldWaitsForDischarge(t *testing.T) { |
| if err := w.reg.Upsert(registry.Route{Host: host, Status: "relocating", HoldOp: "rel-1", Socket: "/run/shop.sock"}); err != nil { |
| t.Fatal(err) |
| } |
| - w.act = &stopProbeActivator{order: &w.order} |
| - a := ClaimAssembly{Claims: store, stopLocal: func(context.Context) (claimStopLocal, error) { |
| + // The probe stop's fence, under the owner it persisted. |
| + dataDir := t.TempDir() |
| + marker, err := relocationProbeFencePath(dataDir, "rel-1") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if w.owner, err = relocationProbeFenceOwner(marker, host, 12); err != nil { |
| + t.Fatal(err) |
| + } |
| + w.act = newStopProbeActivator(&w.order) |
| + fctx, cancel := context.WithTimeout(ctx, time.Second) |
| + defer cancel() |
| + if _, err := w.act.svc.FenceAndStopAs(fctx, host, 12, w.owner); err != nil && !errors.Is(err, activator.ErrStopUnproven) { |
| + t.Fatal(err) |
| + } |
| + a := ClaimAssembly{Claims: w.store, stopLocal: func(context.Context) (claimStopLocal, error) { |
| return claimStopLocal{Actor: hostbox.Actor{Box: box}}, nil |
| }} |
| - cfg := RelocationAssemblyConfig{Registry: w.reg, Activator: w.act, Discharge: func(_ context.Context, o hostclaim.Observed) error { |
| - if !slices.Contains(w.order, "discharge") { |
| - if row, _ := w.reg.Get(host); row == nil || row.HoldOp != "rel-1" { |
| - t.Error("the hold was released before the discharge") |
| - } |
| + cfg := RelocationAssemblyConfig{Registry: w.reg, Activator: w.act, DataDir: dataDir, Discharge: func(_ context.Context, o hostclaim.Observed) error { |
| + if row, _ := w.reg.Get(host); row == nil || row.HoldOp != "rel-1" { |
| + t.Error("the hold was released before the discharge") |
| } |
| w.order = append(w.order, "discharge") |
| - return discharge |
| + return w.discharge |
| }} |
| w.hold = a.relocationReleaseHold(cfg) |
| return w |
| @@ -10076,36 +10598,78 @@ func TestRelocationReleaseHoldWaitsForDischarge(t *testing.T) { |
| row, err := w.reg.Get(host) |
| return err == nil && row != nil && row.Status == "relocating" && row.HoldOp == "rel-1" |
| } |
| - t.Run("the discharge first, then the hold and the fence", func(t *testing.T) { |
| - w := setup(t, claim(nil), nil) |
| + marked := func(w *world) bool { |
| + obs, err := hostclaim.Read(ctx, w.store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + r, ok := hostclaim.ReadCompletedRelocation(obs.Claim) |
| + return ok && r.Discharged |
| + } |
| + t.Run("the discharge first, then the hold, the fence and the mark", func(t *testing.T) { |
| + w := setup(t, claim(nil)) |
| if err := w.hold(ctx, host, "rel-1"); err != nil { |
| t.Fatal(err) |
| } |
| - if want := []string{"discharge", "lift"}; !slices.Equal(w.order, want) || held(w) || !slices.Equal(w.act.lifted, []uint64{12}) { |
| - t.Fatalf("order %v held=%v lifted=%v", w.order, held(w), w.act.lifted) |
| + if want := []string{"discharge", "lift"}; !slices.Equal(w.order, want) || held(w) || w.act.svc.Fenced(host) || !marked(w) { |
| + t.Fatalf("order %v held=%v fenced=%v marked=%v", w.order, held(w), w.act.svc.Fenced(host), marked(w)) |
| } |
| - // A replay: discharged again, the row already ready, the fence |
| - // lift idempotent. |
| - if err := w.hold(ctx, host, "rel-1"); err != nil || held(w) { |
| - t.Fatalf("replay: %v", err) |
| + // A replay of a marked relocation does nothing at all. |
| + w.order = nil |
| + if err := w.hold(ctx, host, "rel-1"); err != nil || len(w.order) != 0 { |
| + t.Fatalf("replay: %v %v", err, w.order) |
| } |
| }) |
| - t.Run("a refused discharge keeps both", func(t *testing.T) { |
| - refused := errors.New("the participation slot is still open") |
| - w := setup(t, claim(nil), refused) |
| - if err := w.hold(ctx, host, "rel-1"); !errors.Is(err, refused) { |
| + t.Run("a refused discharge keeps everything", func(t *testing.T) { |
| + w := setup(t, claim(nil)) |
| + w.discharge = errors.New("the participation slot is still open") |
| + if err := w.hold(ctx, host, "rel-1"); !errors.Is(err, w.discharge) { |
| t.Fatalf("got %v", err) |
| } |
| - if want := []string{"discharge"}; !slices.Equal(w.order, want) || !held(w) { |
| - t.Fatalf("order %v held=%v", w.order, held(w)) |
| + if want := []string{"discharge"}; !slices.Equal(w.order, want) || !held(w) || w.act.svc.FenceGeneration(host) != 12 || marked(w) { |
| + t.Fatalf("order %v held=%v fence=%d marked=%v", w.order, held(w), w.act.svc.FenceGeneration(host), marked(w)) |
| } |
| }) |
| - t.Run("a holder moved off the probe session", func(t *testing.T) { |
| - w := setup(t, claim(func(c *hostclaim.Claim) { c.Holder.Session, c.Holder.Stamp = 40, 40 }), nil) |
| - if err := w.hold(ctx, host, "rel-1"); !errors.Is(err, hostrelocate.ErrRefused) || len(w.order) != 0 || !held(w) { |
| + t.Run("another operation's fence after a restart stays", func(t *testing.T) { |
| + w := setup(t, claim(nil)) |
| + w.act.svc = &activator.Service{} // the restart: this box's fence is gone |
| + w.act.foreignFence(t, host, 12) |
| + if err := w.hold(ctx, host, "rel-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if held(w) || w.act.svc.FenceGeneration(host) != 12 || !marked(w) { |
| + t.Fatalf("held=%v fence=%d marked=%v", held(w), w.act.svc.FenceGeneration(host), marked(w)) |
| + } |
| + }) |
| + t.Run("a replay after the hold went only lifts and marks", func(t *testing.T) { |
| + // The hold was released, then the agent stopped before the lift; |
| + // a serving wake has since moved the holder on. |
| + w := setup(t, claim(func(c *hostclaim.Claim) { c.Holder.Session = 13 })) |
| + if _, err := w.reg.ReleaseHold(host, "rel-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := w.hold(ctx, host, "rel-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if want := []string{"lift"}; !slices.Equal(w.order, want) || w.act.svc.Fenced(host) || !marked(w) { |
| + t.Fatalf("order %v fenced=%v marked=%v", w.order, w.act.svc.Fenced(host), marked(w)) |
| + } |
| + }) |
| + t.Run("a holder moved off the probe session while held", func(t *testing.T) { |
| + w := setup(t, claim(func(c *hostclaim.Claim) { c.Holder.Session, c.Holder.Stamp = 40, 40 })) |
| + if err := w.hold(ctx, host, "rel-1"); !errors.Is(err, hostrelocate.ErrRefused) || len(w.order) != 0 || !held(w) || marked(w) { |
| t.Fatalf("%v order %v held=%v", err, w.order, held(w)) |
| } |
| }) |
| + t.Run("no row is refused", func(t *testing.T) { |
| + w := setup(t, claim(nil)) |
| + if err := w.reg.Delete(host); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := w.hold(ctx, host, "rel-1"); !errors.Is(err, hostrelocate.ErrRefused) || len(w.order) != 0 || marked(w) { |
| + t.Fatalf("%v order %v", err, w.order) |
| + } |
| + }) |
| for name, change := range map[string]func(*hostclaim.Claim){ |
| "still relocating": func(c *hostclaim.Claim) { |
| c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseRelocatingProbing, Exec: box, Participants: []string{"src", box}, |
| @@ -10116,13 +10680,76 @@ func TestRelocationReleaseHoldWaitsForDischarge(t *testing.T) { |
| }, |
| } { |
| t.Run(name, func(t *testing.T) { |
| - w := setup(t, claim(change), nil) |
| - if err := w.hold(ctx, host, "rel-1"); err != nil || len(w.order) != 0 || !held(w) { |
| + w := setup(t, claim(change)) |
| + if err := w.hold(ctx, host, "rel-1"); err != nil || len(w.order) != 0 || !held(w) || w.act.svc.FenceGeneration(host) != 12 { |
| t.Fatalf("%v order %v held=%v", err, w.order, held(w)) |
| } |
| }) |
| } |
| } |
| + |
| +// Astra round 6, finding 3, through the target's real hold release and |
| +// the real start CAS: a successor relocation from the box a relocation |
| +// completed on starts only once that box has discharged its probe's |
| +// participation and marked it. |
| +func TestRelocationSuccessorStartsOnlyAfterTheTargetsDischarge(t *testing.T) { |
| + const host, box = "shop.example", "dst" |
| + ctx := t.Context() |
| + created := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC) |
| + inc, err := hostclaim.Incarnation("acct", "app", host, created) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + done, _ := json.Marshal(hostclaim.CompletedRelocation{Op: "rel-1", Stop: hostclaim.ProbeStop{Box: box, Session: 12, Receipt: "fence-1", ReceiptSHA256: strings.Repeat("a", 64)}, Stamp: 9, Phase: hostclaim.PhaseRelocatingProbing, Generation: "P1"}) |
| + c := &hostclaim.Claim{V: 2, Gen: 31, Host: host, |
| + Holder: &hostclaim.Holder{Family: "instance", Key: "apps/acct/app/instances/" + host + ".json", Account: "acct", App: "app", Incarnation: inc, CreatedAt: created, |
| + Location: "instances/" + host, Stamp: 9, Session: 12, Generation: "P1", WriterBox: box, Admissions: []hostclaim.AdmissionRecord{{Session: 12, Box: box, Generation: "P1", Mode: "cold", Region: "ie"}}}, |
| + Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: box, Participants: []string{box}, Journal: map[string]string{hostclaim.JournalCompleted: string(done)}}} |
| + store := release.NewDirStore(t.TempDir()) |
| + raw, _ := json.Marshal(c) |
| + key, _ := hostclaim.Key(host) |
| + if _, err := store.Put(ctx, key, raw, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer reg.Close() |
| + if err := reg.Upsert(registry.Route{Host: host, Status: "relocating", HoldOp: "rel-1", Socket: "/run/shop.sock"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + var order []string |
| + refused := errors.New("the participation slot is still open") |
| + discharge := refused |
| + a := ClaimAssembly{Claims: store, stopLocal: func(context.Context) (claimStopLocal, error) { |
| + return claimStopLocal{Actor: hostbox.Actor{Box: box}}, nil |
| + }} |
| + hold := a.relocationReleaseHold(RelocationAssemblyConfig{Registry: reg, Activator: newStopProbeActivator(&order), DataDir: t.TempDir(), |
| + Discharge: func(context.Context, hostclaim.Observed) error { return discharge }}) |
| + start := func() error { |
| + obs, err := hostclaim.Read(ctx, store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + _, err = hostclaim.StartRelocation(ctx, store, obs, hostclaim.RelocationStart{OpID: "rel-2", By: "box:wrk", Now: created.Add(3 * time.Hour), Region: "ie", Reason: hostclaim.ReasonDrain, |
| + SourceBox: box, SourceEnrollment: strings.Repeat("c", 32), TargetBox: "spa", TargetEnrollment: strings.Repeat("d", 32), DirectoryRevision: 2, AdmissionID: "adm-2"}, nil) |
| + return err |
| + } |
| + if err := hold(ctx, host, "rel-1"); !errors.Is(err, refused) { |
| + t.Fatalf("hold: %v", err) |
| + } |
| + if err := start(); !errors.Is(err, hostclaim.ErrPendingChore) { |
| + t.Fatalf("the successor started before the target's discharge: %v", err) |
| + } |
| + discharge = nil |
| + if err := hold(ctx, host, "rel-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := start(); err != nil { |
| + t.Fatalf("the successor after the discharge: %v", err) |
| + } |
| +} |
| ``` |
| `internal/host/relocationstopprobe_unix_test.go` (new): |
| @@ -10138,6 +10765,7 @@ import ( |
| "encoding/hex" |
| "encoding/json" |
| "errors" |
| + "fmt" |
| "os" |
| "path/filepath" |
| "slices" |
| @@ -10150,32 +10778,46 @@ import ( |
| "github.com/carlosframework/platform/internal/hostguard" |
| ) |
| -// stopProbeActivator is the activator's fence and stop, recorded in order. |
| +// stopProbeActivator is a real activator.Service's fences, the calls |
| +// recorded in order. Replacing svc is an agent restart: the activator's |
| +// fences live only in its process. failNext, when set, is the next |
| +// FenceAndStopAs's answer instead of the service's. |
| type stopProbeActivator struct { |
| - order *[]string |
| - ifErr error // FenceAndStopIfUnfenced's answer |
| - stopErr error // FenceAndStop's answer |
| - lifted []uint64 |
| + order *[]string |
| + svc *activator.Service |
| + failNext error |
| +} |
| + |
| +func newStopProbeActivator(order *[]string) *stopProbeActivator { |
| + return &stopProbeActivator{order: order, svc: &activator.Service{}} |
| } |
| func (a *stopProbeActivator) ProbeWake(context.Context, activator.ProbeConfig) (activator.ProbeResult, error) { |
| return activator.ProbeResult{}, errors.New("not a probe") |
| } |
| -func (a *stopProbeActivator) FenceAndStop(_ context.Context, host string, generation uint64) (*activator.StopEvidence, error) { |
| - *a.order = append(*a.order, "fence-and-stop") |
| - return nil, a.stopErr |
| +func (a *stopProbeActivator) FenceAndStopAs(ctx context.Context, host string, generation uint64, owner string) (*activator.StopEvidence, error) { |
| + *a.order = append(*a.order, "fence-as") |
| + if err := a.failNext; err != nil { |
| + a.failNext = nil |
| + return nil, err |
| + } |
| + return a.svc.FenceAndStopAs(ctx, host, generation, owner) |
| } |
| -func (a *stopProbeActivator) FenceAndStopIfUnfenced(_ context.Context, host string, generation uint64) (*activator.StopEvidence, error) { |
| - *a.order = append(*a.order, "fence-if-unfenced") |
| - return nil, a.ifErr |
| +func (a *stopProbeActivator) LiftFenceAs(host string, generation uint64, owner string) bool { |
| + *a.order = append(*a.order, "lift") |
| + return a.svc.LiftFenceAs(host, generation, owner) |
| } |
| -func (a *stopProbeActivator) LiftFence(_ string, generation uint64) bool { |
| - *a.order = append(*a.order, "lift") |
| - a.lifted = append(a.lifted, generation) |
| - return true |
| +// foreignFence installs another operation's plain fence on a.svc. |
| +func (a *stopProbeActivator) foreignFence(t *testing.T, host string, generation uint64) { |
| + t.Helper() |
| + ctx, cancel := context.WithTimeout(t.Context(), time.Second) |
| + defer cancel() |
| + if _, err := a.svc.FenceAndStop(ctx, host, generation); err != nil && !errors.Is(err, activator.ErrStopUnproven) { |
| + t.Fatal(err) |
| + } |
| } |
| // stopProbeFixture is a target at relocating/probing whose probe passed |
| @@ -10234,7 +10876,7 @@ func newStopProbeFixture(t *testing.T) *stopProbeFixture { |
| } |
| return guard.CompleteFence(ctx, f.target, f.receipt, at, func(context.Context, hostguard.FenceTarget, []byte, time.Time) error { return nil }) |
| }} |
| - f.act = &stopProbeActivator{order: &f.order} |
| + f.act = newStopProbeActivator(&f.order) |
| f.a = ClaimAssembly{Guard: guard, Wake: wake, stopLocal: func(context.Context) (claimStopLocal, error) { |
| return claimStopLocal{Actor: hostbox.Actor{Box: box, Enrollment: "enr", Region: "ie", Revision: 1}, BootID: "boot", ControllerID: "ctl", PID: 100, ProcessStart: "5000"}, nil |
| }} |
| @@ -10246,15 +10888,30 @@ func (f *stopProbeFixture) stop() (hostclaim.ProbeStop, error) { |
| return f.a.relocationStopProbe(f.cfg)(f.t.Context(), f.obs) |
| } |
| -// Ruling 49 steps 1 to 3, and K19 against the activator's real API: the |
| -// activator's fence goes in before root is asked (FenceAndStopIfUnfenced, |
| -// never over another operation's fence); an unproven stop still asks |
| -// root; root's receipt is the guard journal's writer-session stop of the |
| -// probe's session; a repeat re-collects under its own fence and returns |
| -// the same ProbeStop; another operation's fence (ErrFenceHeld) and an |
| -// uninstalled fence stop everything with nothing asked of root. |
| +// marker is the probe stop's persisted owner for rel-1. |
| +func (f *stopProbeFixture) marker() relocationProbeFence { |
| + f.t.Helper() |
| + path, err := relocationProbeFencePath(f.cfg.DataDir, "rel-1") |
| + if err != nil { |
| + f.t.Fatal(err) |
| + } |
| + rec, ok, err := readRelocationProbeFence(path) |
| + if err != nil || !ok { |
| + f.t.Fatalf("no probe-fence marker: %v", err) |
| + } |
| + return rec |
| +} |
| + |
| +// Ruling 49 steps 1 to 3 with ruling 75's owned fence, against the real |
| +// activator: the owner is persisted, then the fence goes in before root is |
| +// asked; an unproven stop (the probe's entry is gone) still asks root; |
| +// root's receipt is the guard journal's writer-session stop of the probe's |
| +// session; a repeat meets its own fence and returns the same ProbeStop; |
| +// another operation's fence and an uninstalled fence stop everything with |
| +// nothing asked of root and the other fence as it was. |
| func TestRelocationStopProbeFencesStopsAndReadsRootsReceipt(t *testing.T) { |
| sum := func(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) } |
| + const host = "shop.example" |
| t.Run("the probe's session", func(t *testing.T) { |
| f := newStopProbeFixture(t) |
| s, err := f.stop() |
| @@ -10264,39 +10921,46 @@ func TestRelocationStopProbeFencesStopsAndReadsRootsReceipt(t *testing.T) { |
| if want := (hostclaim.ProbeStop{Box: "dst", Session: 12, Receipt: "fence-1", ReceiptSHA256: sum(f.receipt)}); s != want { |
| t.Fatalf("stop %+v, want %+v", s, want) |
| } |
| - if want := []string{"fence-if-unfenced", "wake-stop"}; !slices.Equal(f.order, want) { |
| + if want := []string{"fence-as", "wake-stop"}; !slices.Equal(f.order, want) { |
| t.Fatalf("order %v, want %v", f.order, want) |
| } |
| - // A repeat (the claim CAS lost, or a later pass): the activator now |
| - // holds this probe's own fence at the same session. |
| - f.order, f.act.ifErr = nil, activator.ErrFenceHeld |
| + m := f.marker() |
| + if m.Host != host || m.Session != 12 || len(m.Owner) != 32 || f.act.svc.FenceGeneration(host) != 12 { |
| + t.Fatalf("marker %+v, fence %d", m, f.act.svc.FenceGeneration(host)) |
| + } |
| + // A repeat (the claim CAS lost, or a later pass) meets its own |
| + // fence under the same owner. |
| + f.order = nil |
| again, err := f.stop() |
| if err != nil || again != s { |
| t.Fatalf("repeat: %+v %v", again, err) |
| } |
| - if want := []string{"fence-if-unfenced", "fence-and-stop", "wake-stop"}; !slices.Equal(f.order, want) { |
| - t.Fatalf("repeat order %v, want %v", f.order, want) |
| - } |
| - }) |
| - t.Run("an unproven stop still asks root", func(t *testing.T) { |
| - f := newStopProbeFixture(t) |
| - f.act.ifErr = activator.ErrStopUnproven |
| - if _, err := f.stop(); err != nil || !slices.Contains(f.order, "wake-stop") { |
| - t.Fatalf("%v %v", err, f.order) |
| + if want := []string{"fence-as", "wake-stop"}; !slices.Equal(f.order, want) || f.marker() != m { |
| + t.Fatalf("repeat order %v, marker %+v", f.order, f.marker()) |
| } |
| }) |
| - for name, err := range map[string]error{"another operation's fence": activator.ErrFenceHeld, "no fence installed": activator.ErrFenceNotInstalled} { |
| - t.Run(name, func(t *testing.T) { |
| + for _, gen := range []uint64{5, 12, 20} { |
| + t.Run(fmt.Sprintf("another operation's fence at %d", gen), func(t *testing.T) { |
| f := newStopProbeFixture(t) |
| - f.act.ifErr = err |
| - if _, got := f.stop(); !errors.Is(got, err) { |
| - t.Fatalf("got %v, want %v", got, err) |
| + f.act.foreignFence(t, host, gen) |
| + if _, err := f.stop(); !errors.Is(err, activator.ErrFenceHeld) { |
| + t.Fatalf("got %v, want ErrFenceHeld", err) |
| } |
| - if want := []string{"fence-if-unfenced"}; !slices.Equal(f.order, want) { |
| - t.Fatalf("root was asked: %v", f.order) |
| + if want := []string{"fence-as"}; !slices.Equal(f.order, want) || f.act.svc.FenceGeneration(host) != gen { |
| + t.Fatalf("root was asked or the fence changed: %v, fence %d", f.order, f.act.svc.FenceGeneration(host)) |
| } |
| }) |
| } |
| + t.Run("no fence installed", func(t *testing.T) { |
| + f := newStopProbeFixture(t) |
| + f.act.failNext = activator.ErrFenceNotInstalled |
| + if _, err := f.stop(); !errors.Is(err, activator.ErrFenceNotInstalled) { |
| + t.Fatalf("got %v", err) |
| + } |
| + if want := []string{"fence-as"}; !slices.Equal(f.order, want) { |
| + t.Fatalf("root was asked: %v", f.order) |
| + } |
| + }) |
| t.Run("not the probe's session", func(t *testing.T) { |
| f := newStopProbeFixture(t) |
| f.obs.Claim.Holder.Session = 13 |
| @@ -10305,6 +10969,85 @@ func TestRelocationStopProbeFencesStopsAndReadsRootsReceipt(t *testing.T) { |
| } |
| }) |
| } |
| + |
| +// Astra round 6, finding 1: the agent restarts after the probe stop |
| +// installed its fence and persisted its owner. The restart empties the |
| +// activator's fences; the marker survives. Another operation then fences |
| +// the host at a lower or equal generation. The surviving marker must not |
| +// let the probe stop take that fence over: ErrFenceHeld, the other fence |
| +// exactly as it was, root not asked. Once the other operation lifts its |
| +// fence, the next pass installs this probe's own fence and completes. |
| +func TestRelocationStopProbeAfterARestartNeverTakesAForeignFence(t *testing.T) { |
| + const host = "shop.example" |
| + for _, gen := range []uint64{5, 12} { |
| + t.Run(fmt.Sprint(gen), func(t *testing.T) { |
| + f := newStopProbeFixture(t) |
| + first, err := f.stop() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + before := f.marker() |
| + f.act.svc = &activator.Service{} // the restart |
| + f.act.foreignFence(t, host, gen) |
| + f.order = nil |
| + if _, err := f.stop(); !errors.Is(err, activator.ErrFenceHeld) { |
| + t.Fatalf("got %v, want ErrFenceHeld", err) |
| + } |
| + if want := []string{"fence-as"}; !slices.Equal(f.order, want) || f.act.svc.FenceGeneration(host) != gen || f.marker() != before { |
| + t.Fatalf("order %v, fence %d, marker %+v", f.order, f.act.svc.FenceGeneration(host), f.marker()) |
| + } |
| + if f.act.svc.LiftFenceAs(host, gen, before.Owner) { |
| + t.Fatal("the surviving owner lifted the other operation's fence") |
| + } |
| + if !f.act.svc.LiftFence(host, gen) { |
| + t.Fatal("the other operation's fence was not as it installed it") |
| + } |
| + f.order = nil |
| + again, err := f.stop() |
| + if err != nil || again != first || f.act.svc.FenceGeneration(host) != 12 { |
| + t.Fatalf("after the other fence went: %+v %v, fence %d", again, err, f.act.svc.FenceGeneration(host)) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// Astra round 6, finding 2: the owner is persisted before the activator is |
| +// asked, so a failed write installs nothing and blocks nothing. The |
| +// retry persists it and proceeds, under the owner it persisted. |
| +func TestRelocationStopProbeRetriesAFailedMarkerWrite(t *testing.T) { |
| + const host = "shop.example" |
| + f := newStopProbeFixture(t) |
| + path, err := relocationProbeFencePath(f.cfg.DataDir, "rel-1") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + // The op's journal directory cannot be made: a file stands in its way. |
| + if err := os.MkdirAll(filepath.Dir(filepath.Dir(path)), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(filepath.Dir(path), []byte("in the way"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := f.stop(); err == nil { |
| + t.Fatal("the probe stop went on without its owner persisted") |
| + } |
| + if len(f.order) != 0 || f.act.svc.Fenced(host) { |
| + t.Fatalf("the activator was asked before the owner was persisted: %v", f.order) |
| + } |
| + if err := os.Remove(filepath.Dir(path)); err != nil { |
| + t.Fatal(err) |
| + } |
| + s, err := f.stop() |
| + if err != nil || s.Session != 12 { |
| + t.Fatalf("retry: %+v %v", s, err) |
| + } |
| + if want := []string{"fence-as", "wake-stop"}; !slices.Equal(f.order, want) { |
| + t.Fatalf("retry order %v", f.order) |
| + } |
| + if m := f.marker(); !f.act.svc.LiftFenceAs(host, 12, m.Owner) { |
| + t.Fatal("the fence installed is not the persisted owner's") |
| + } |
| +} |
| ``` |
| `internal/hostbox/reconcile_relocation_target_test.go` (new): |
| @@ -10567,7 +11310,7 @@ func TestReconcileAdapterClearsTheHoldAfterCompletion(t *testing.T) { |
| Run: |
| ```bash |
| -nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelocationStopProbe|TestRelocationReleaseHold|TestRelocationCompletionDischarges|TestRelocationRestart|TestRelocationCompletedHandover)' |
| +nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelocationStopProbe|TestRelocationReleaseHold|TestRelocationSuccessor|TestRelocationCompletionDischarges|TestRelocationRestart|TestRelocationCompletedHandover)' |
| ``` |
| Expected: FAIL to compile: `undefined: relocationCompletedHandover`. |
| @@ -10936,7 +11679,7 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| }, Guard: guard, Claims: store, DataDir: r.DataDir, Box: actor.Box} |
| ``` |
| -- [ ] **Step 5: The probe stop and the hold release (rulings 49, 66; K19)** |
| +- [ ] **Step 5: The probe stop and the hold release (rulings 49, 66, 75, 76)** |
| `internal/host/relocationowners.go`: |
| @@ -10959,7 +11702,7 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| "github.com/carlosframework/platform/internal/hostrelocate" |
| "github.com/carlosframework/platform/internal/hostwrite" |
| "github.com/carlosframework/platform/internal/lease" |
| -@@ -211,3 +213,143 @@ func (a ClaimAssembly) relocationAdmit(cfg RelocationAssemblyConfig) func(contex |
| +@@ -211,3 +213,201 @@ func (a ClaimAssembly) relocationAdmit(cfg RelocationAssemblyConfig) func(contex |
| } |
| } |
| } |
| @@ -10971,13 +11714,18 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| +// meets the same fence and receipt), which also releases the attempt's |
| +// lease; then root's receipt for exactly that session. |
| +// |
| -+// The fence is taken with FenceAndStopIfUnfenced, never over another |
| -+// operation's fence (K19): FenceAndStop would replace a lower generation |
| -+// held by someone else, and ReleaseHold's LiftFence would then reopen |
| -+// admission they closed. Another operation's fence is ErrFenceHeld, with |
| -+// nothing asked of root. A repeat of this probe stop finds its own fence |
| -+// at the same session, recorded in the op's journal directory once it was |
| -+// installed, and re-collects the stop evidence under it. |
| ++// The fence is an owned one (ruling 75, Astra round 6 findings 1 and 2). |
| ++// Before anything is asked of the activator, the owner this probe stop |
| ++// will install under is persisted with the session in the op's journal |
| ++// directory (probe-fence.json); a failed write installs nothing, and the |
| ++// next pass writes it again. FenceAndStopAs then installs only on an |
| ++// unfenced host, or meets again exactly the fence it installed under |
| ++// that owner at that session, deciding under the activator's own lock. |
| ++// Any other fence (another operation's, of any generation, including one |
| ++// installed after an agent restart emptied the activator's fences while |
| ++// the marker survived) is ErrFenceHeld, with nothing stopped and nothing |
| ++// asked of root, and is never replaced, so ReleaseHold's lift can never |
| ++// reopen admission another operation closed. |
| +func (a ClaimAssembly) relocationStopProbe(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (hostclaim.ProbeStop, error) { |
| + return func(ctx context.Context, o hostclaim.Observed) (hostclaim.ProbeStop, error) { |
| + if cfg.Activator == nil || a.Wake == nil || a.Wake.Stop == nil || o.Claim == nil || o.Claim.Holder == nil { |
| @@ -10996,21 +11744,15 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| + if err != nil { |
| + return hostclaim.ProbeStop{}, err |
| + } |
| -+ fenced, cancel := context.WithTimeout(ctx, hostbox.RequestLimit) |
| -+ _, err = cfg.Activator.FenceAndStopIfUnfenced(fenced, o.Host, h.Session) |
| -+ if errors.Is(err, activator.ErrFenceHeld) { |
| -+ if own, rerr := relocationOwnsProbeFence(marker, h.Session); rerr != nil || !own { |
| -+ cancel() |
| -+ return hostclaim.ProbeStop{}, errors.Join(err, rerr) // another operation's fence: nothing stopped, nothing asked |
| -+ } |
| -+ _, err = cfg.Activator.FenceAndStop(fenced, o.Host, h.Session) // this probe's own fence, the same generation |
| ++ owner, err := relocationProbeFenceOwner(marker, o.Host, h.Session) |
| ++ if err != nil { |
| ++ return hostclaim.ProbeStop{}, err // nothing installed: the next pass persists it again |
| + } |
| ++ fenced, cancel := context.WithTimeout(ctx, hostbox.RequestLimit) |
| ++ _, err = cfg.Activator.FenceAndStopAs(fenced, o.Host, h.Session, owner) |
| + cancel() |
| + if err != nil && !errors.Is(err, activator.ErrStopUnproven) { |
| -+ return hostclaim.ProbeStop{}, err // no fence installed: nothing stopped, nothing journaled |
| -+ } |
| -+ if err := relocationRecordProbeFence(marker, h.Session); err != nil { |
| -+ return hostclaim.ProbeStop{}, err |
| ++ return hostclaim.ProbeStop{}, err // not this probe's fence: nothing stopped, nothing asked |
| + } |
| + if err := a.Wake.Stop(ctx, o); err != nil { |
| + return hostclaim.ProbeStop{}, err |
| @@ -11024,8 +11766,11 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| + } |
| +} |
| + |
| -+// relocationProbeFencePath records the session whose activator fence this |
| -+// box's probe stop installed: <DataDir>/relocate/ops/<op>/probe-fence.json. |
| ++// relocationProbeFencePath is where this box's probe stop persists the |
| ++// owner of the activator fence it installs, with the session and host: |
| ++// <DataDir>/relocate/ops/<op>/probe-fence.json. The journal sweep keeps |
| ++// the op's directory until the completed relocation is marked discharged |
| ++// (ruling 76), so the lift always finds it. |
| +func relocationProbeFencePath(dataDir, op string) (string, error) { |
| + dir, err := relocationOpDir(dataDir, op) |
| + if err != nil { |
| @@ -11034,41 +11779,72 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| + return filepath.Join(dir, "probe-fence.json"), nil |
| +} |
| + |
| -+func relocationRecordProbeFence(path string, session uint64) error { |
| -+ if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { |
| -+ return err |
| ++// relocationProbeFence is the persisted owner of one probe stop's fence. |
| ++type relocationProbeFence struct { |
| ++ Host string `json:"host"` |
| ++ Session uint64 `json:"session"` |
| ++ Owner string `json:"owner"` |
| ++} |
| ++ |
| ++// relocationProbeFenceOwner returns the owner persisted for host's probe |
| ++// session, or persists a new, random one first. A marker for another |
| ++// session, or one that does not read, is replaced: a fresh owner can only |
| ++// ever match a fence installed under it, so replacing one takes nothing |
| ++// over. |
| ++func relocationProbeFenceOwner(path, host string, session uint64) (string, error) { |
| ++ if rec, ok, err := readRelocationProbeFence(path); err == nil && ok && rec.Host == host && rec.Session == session && rec.Owner != "" { |
| ++ return rec.Owner, nil |
| + } |
| -+ raw, err := json.Marshal(struct { |
| -+ Session uint64 `json:"session"` |
| -+ }{session}) |
| ++ var b [16]byte |
| ++ if _, err := rand.Read(b[:]); err != nil { |
| ++ return "", err |
| ++ } |
| ++ rec := relocationProbeFence{Host: host, Session: session, Owner: hex.EncodeToString(b[:])} |
| ++ raw, err := json.Marshal(rec) |
| + if err != nil { |
| -+ return err |
| ++ return "", err |
| ++ } |
| ++ if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { |
| ++ return "", err |
| + } |
| -+ return writeFileSynced(path, raw, 0o600) |
| ++ if err := writeFileSynced(path, raw, 0o600); err != nil { |
| ++ return "", err |
| ++ } |
| ++ return rec.Owner, nil |
| +} |
| + |
| -+func relocationOwnsProbeFence(path string, session uint64) (bool, error) { |
| ++// readRelocationProbeFence reads the persisted owner; false when there is |
| ++// none. |
| ++func readRelocationProbeFence(path string) (relocationProbeFence, bool, error) { |
| + raw, err := os.ReadFile(path) |
| + if errors.Is(err, os.ErrNotExist) { |
| -+ return false, nil |
| ++ return relocationProbeFence{}, false, nil |
| + } |
| + if err != nil { |
| -+ return false, err |
| -+ } |
| -+ var rec struct { |
| -+ Session uint64 `json:"session"` |
| ++ return relocationProbeFence{}, false, err |
| + } |
| ++ var rec relocationProbeFence |
| + if err := json.Unmarshal(raw, &rec); err != nil { |
| -+ return false, err |
| ++ return relocationProbeFence{}, false, err |
| + } |
| -+ return rec.Session == session, nil |
| ++ return rec, true, nil |
| +} |
| + |
| -+// relocationReleaseHold is ruling 49 step 5: once the claim serves on |
| -+// this box under kind serving, the row's relocating hold is released |
| -+// (registry.ReleaseHold; false is the replay) and the activator fence the |
| -+// probe stop installed at the session is lifted. Anything else leaves the |
| -+// hold: an excluded box's row is its park-local's. |
| ++// relocationReleaseHold is ruling 49 step 5 and ruling 76: once the claim |
| ++// serves on this box under kind serving, main's participation is |
| ++// discharged (C10), the row's relocating hold is released, the activator |
| ++// fence the probe stop installed under its persisted owner is lifted, and |
| ++// the serving op's completed relocation is marked discharged, which is |
| ++// what lets a successor relocation start from this box |
| ++// (hostclaim.RelocationStartGuard). Ruling 66: while the row is held for |
| ++// the op, the holder must still be the stopped probe session and the |
| ++// discharge succeeds before anything is released; a refusal keeps the |
| ++// hold and the fence and is retried next pass. Once the row is no longer |
| ++// held for the op, the discharge has happened (nothing else releases a |
| ++// relocating hold for a completed op), and a replay only lifts and marks. |
| ++// A missing row is refused: absence is never evidence of the discharge. |
| ++// Anything else leaves everything: an excluded box's row is its |
| ++// park-local's. |
| +func (a ClaimAssembly) relocationReleaseHold(cfg RelocationAssemblyConfig) func(context.Context, string, string) error { |
| + return func(ctx context.Context, host, op string) error { |
| + if cfg.Registry == nil || cfg.Activator == nil || cfg.Discharge == nil { |
| @@ -11085,22 +11861,47 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| + if obs.Absent || obs.Claim.Holder == nil || obs.Claim.Op.Kind != hostclaim.KindServing || obs.Claim.Op.Exec != local.Actor.Box || obs.Claim.Holder.WriterBox != local.Actor.Box { |
| + return nil |
| + } |
| -+ // Ruling 66: no serving admission may become possible before the |
| -+ // participation is discharged, or the admission moves the holder |
| -+ // off the probe session and C10's evidence no longer matches. The |
| -+ // hold and the fence go only after the discharge succeeded; a |
| -+ // refusal keeps both and is retried next pass. |
| -+ if done, ok := hostclaim.ReadCompletedRelocation(obs.Claim); !ok || done.Op != op || done.Stop.Session != obs.Claim.Holder.Session { |
| -+ return fmt.Errorf("%w: %s's serving holder is not %s's stopped probe session", hostrelocate.ErrRefused, host, op) |
| ++ done, ok := hostclaim.ReadCompletedRelocation(obs.Claim) |
| ++ if !ok || done.Op != op || done.Stop.Box != local.Actor.Box { |
| ++ return fmt.Errorf("%w: %s's serving op does not record %s as completed on this box", hostrelocate.ErrRefused, host, op) |
| + } |
| -+ if err := cfg.Discharge(ctx, obs); err != nil { |
| ++ if done.Discharged { |
| ++ return nil |
| ++ } |
| ++ row, err := cfg.Registry.Get(host) |
| ++ if err != nil { |
| + return err |
| + } |
| -+ if _, err := cfg.Registry.ReleaseHold(host, op); err != nil { |
| ++ if row == nil { |
| ++ return fmt.Errorf("%w: %s has no row on this box", hostrelocate.ErrRefused, host) |
| ++ } |
| ++ if row.HoldOp == op { |
| ++ if done.Stop.Session != obs.Claim.Holder.Session { |
| ++ return fmt.Errorf("%w: %s's serving holder is not %s's stopped probe session", hostrelocate.ErrRefused, host, op) |
| ++ } |
| ++ if err := cfg.Discharge(ctx, obs); err != nil { |
| ++ return err |
| ++ } |
| ++ if _, err := cfg.Registry.ReleaseHold(host, op); err != nil { |
| ++ return err |
| ++ } |
| ++ } |
| ++ marker, err := relocationProbeFencePath(cfg.DataDir, op) |
| ++ if err != nil { |
| + return err |
| + } |
| -+ cfg.Activator.LiftFence(host, obs.Claim.Holder.Session) |
| -+ return nil |
| ++ fence, found, err := readRelocationProbeFence(marker) |
| ++ if err != nil { |
| ++ return err // the owner is unreadable: the fence stays, the claim is not marked |
| ++ } |
| ++ if found && fence.Host == host && fence.Session == done.Stop.Session { |
| ++ cfg.Activator.LiftFenceAs(host, fence.Session, fence.Owner) |
| ++ } |
| ++ _, err = hostclaim.Update(ctx, a.Claims, obs, hostclaim.MarkCompletedDischarged(op)) |
| ++ if errors.Is(err, hostclaim.ErrAlreadyRecorded) { |
| ++ return nil |
| ++ } |
| ++ return err |
| + } |
| +} |
| ``` |
| @@ -11166,15 +11967,7 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| ```diff |
| --- a/internal/host/relocationassembly.go |
| +++ b/internal/host/relocationassembly.go |
| -@@ -30,6 +30,7 @@ import ( |
| - type RelocationActivator interface { |
| - ProbeWake(context.Context, activator.ProbeConfig) (activator.ProbeResult, error) |
| - FenceAndStop(context.Context, string, uint64) (*activator.StopEvidence, error) |
| -+ FenceAndStopIfUnfenced(context.Context, string, uint64) (*activator.StopEvidence, error) |
| - LiftFence(string, uint64) bool |
| - } |
| - |
| -@@ -63,7 +64,11 @@ type RelocationAssemblyConfig struct { |
| +@@ -65,7 +65,11 @@ type RelocationAssemblyConfig struct { |
| // main: root then authorizes a relocating launch at the admitted |
| // writer's stamp, not only at install_stamp (ruling 58). |
| LaterProbeAttempts bool |
| @@ -11187,7 +11980,7 @@ func (h relocationCompletedHandover) After() func(context.Context, hostclaim.Obs |
| // render and materialize are appconfig.RenderHeldUnder and |
| // release.MaterializeHeldUnder; nil is those. Tests replace them. |
| -@@ -115,6 +120,8 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| +@@ -117,6 +121,8 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig, ack func( |
| VerifyLocal: a.relocationVerifyLocal(cfg), |
| Receipt: a.relocationReceipt(cfg), |
| Admit: a.relocationAdmit(cfg), |
| @@ -11204,7 +11997,7 @@ Run: |
| ```bash |
| nice -n 19 go test ./internal/hostbox/ -run TestReconcile -count=1 |
| -TMPDIR=$(mktemp -d /tmp/k.XXXX) nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelocationStopProbe|TestRelocationReleaseHold|TestRelocationCompletionDischarges|TestRelocationRestart|TestRelocationCompletedHandover|TestRelocationAcknowledges|TestPlan9|TestPlan11|TestClaimHandover|TestCold)' |
| +TMPDIR=$(mktemp -d /tmp/k.XXXX) nice -n 19 go test ./internal/host/ -count=1 -run '^(TestRelocationStopProbe|TestRelocationReleaseHold|TestRelocationSuccessor|TestRelocationCompletionDischarges|TestRelocationRestart|TestRelocationCompletedHandover|TestRelocationAcknowledges|TestRelocationJournals|TestPlan9|TestPlan11|TestClaimHandover|TestCold)' |
| ``` |
| Expected: PASS. Run outside the sandbox (the `TestPlan9`/`TestPlan11` regressions listen on unix sockets), with `TMPDIR=$(mktemp -d /tmp/k.XXXX)`, removed afterwards. |
| @@ -11213,7 +12006,7 @@ Expected: PASS. Run outside the sandbox (the `TestPlan9`/`TestPlan11` regression |
| ```bash |
| git add internal/host/claimparticipation_completed.go internal/host/relocationdischarge_linux_test.go internal/host/relocationdischarge_restart_unix_test.go internal/host/relocationhandover.go internal/host/relocationhandover_unix_test.go internal/host/relocationreleasehold_test.go internal/host/relocationstopprobe_unix_test.go internal/hostbox/reconcile_relocation_target_test.go internal/host/claimconstruction.go internal/host/claimhandover_crossbox.go internal/host/claimhandover_transport.go internal/host/claimparticipation.go internal/host/claimparticipation_relocation.go internal/host/claimparticipation_stale.go internal/host/plan11_fix1_relocation_test.go internal/host/plan9_production_test.go internal/host/relocationassembly.go internal/host/relocationowners.go |
| -git commit -m "host: the passed probe is stopped with root's receipt, its participation is discharged by that receipt before the hold is released (C10), and the first serving wake hands over from it (N6) |
| +git commit -m "host: the passed probe is stopped with root's receipt under an owned fence, its participation is discharged by that receipt before the hold is released and the completion is marked (C10), and the first serving wake hands over from it (N6) |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ``` |
| @@ -11232,13 +12025,13 @@ The agent registers the source with the roster directory the claim's owners read |
| **Interfaces:** |
| - Consumes: `hostrelocate.Worker`, `Participant`, `RegionChooser`, `StorePinner`; `hostfence.ClaimFenceAuthority`; `StopReceiptReleases`, `RelocationReleaseReceipts`; `hostops.ExecutorChooser`, `ClaimLoads`; `release.Channel`, `NewS3Store`, `BumpSerial`, `ParsePublicKey`; `lease.Acquire`, `NewVersionedS3`; `genselect.Select`; `provision.GetRecord`, `MoveState`, `MoveCommitted`; `moves.CommitScopes`; `hostclaim.ConvertCommittedLanding`, `CommittedLandingGuard`; `boxvolumes.List`, `Discharge`; `hostfloor.ReadEnrollment`; `appconfig.ParsePrivateKey`; `edgesFileSecretsFP`, `MovesRecordsFrom`; `ensureProductionClaimAssembly`, `claimEntry`; `status.LoadBoxesComplete`, `BoxStale`. |
| -- Produces: `relocatePassEnabled`, `RelocationRuntime` (`Run`, `once`, `releaseCompletedHold`, `alert`), `relocationStepper`, `relocationStores`, `openRelocationStores`, `relocationLeases`, `relocationAssemblyConfig`, `channelRelease`, `volumeDischarge`, `relocationRelocatable`, `relocationStartRefused`, `relocationRuntimeInputs`, `productionRelocationRuntimeInputs`, `NewRelocationRuntime`, `wakeDispatch`, `convertFencedLanding`, `registerRelocationSource`, `startRelocationPass`; `(*productionClaimLocal).boundAssembly` (N1); `CommittedLandingRecords`; `hostrelocate.Worker.Fence`, `newestFenceObservation` and the no-target bound; `edgesfile.PlacementView`, `Snapshot`, `ReadSnapshot`, `ReadLiveSnapshot`, `LiveSnapshot`; `ProductionAuthority.Executors` from the console's chooser. |
| +- Produces: `relocatePassEnabled`, `RelocationRuntime` (`Run`, `once`, `releaseCompletedHold`, `alert`), `relocationStepper`, `relocationStores`, `openRelocationStores`, `relocationLeases`, `relocationAssemblyConfig`, `channelRelease`, `volumeDischarge`, `relocationRelocatable`, `relocationStartRefused`, `relocationRuntimeInputs`, `productionRelocationRuntimeInputs`, `NewRelocationRuntime`, `wakeDispatch`, `convertFencedLanding`, `registerRelocationSource`, `startRelocationPass`; `(*productionClaimLocal).boundAssembly` (N1); `CommittedLandingRecords`; `hostrelocate.Worker.ConvertLanding` (ruling 78), `newestFenceObservation` and the no-target bound; `edgesfile.PlacementView`, `Snapshot`, `ReadSnapshot`, `ReadLiveSnapshot`, `LiveSnapshot`; `ProductionAuthority.Executors` from the console's chooser. |
| - [ ] **Step 1: Write the failing tests** |
| -The pass tests drive `once` over a DirStore control bucket with recording steppers; the steps test holds a landing claim beside the relocating ones and asserts only it is offered to `Convert` and nothing steps it. `TestRelocatePassConvertsAFencedCommittedLanding` runs the production conversion over a real landing claim, a signed roster, a termination fence and the authoritative record, committed and not. `TestRelocationReleaseIsTheChannelPointer` ships and promotes real releases with a generated root key. `TestRelocationPinnerSelectsOverTheVersionedBucket` (hostrelocate) runs the pinner over `genselect.Select` and a versioned bucket holding a marker, a snapshot and a level-0 file. Rev 1's `TestRelocatePassStaysOffWhileThePreflightRefuses` is not a pass test in rev 5: the preflight is the participant's (Task 8a, `TestParticipantCountsNoAttemptWhileRootRefusesTheKind`). Finding 12's test (`TestRelocationLeasesSignForTheCloudRegion`) is folded into `TestRelocationRuntimeOpensADataBucketStore` (K6). |
| +The pass tests drive `once` over a DirStore control bucket with recording steppers; the steps test holds a landing claim beside the relocating ones and asserts only it is offered to `Convert` and nothing steps it. `TestRelocatePassConvertsAFencedCommittedLanding` runs the production conversion over a real landing claim, a signed roster, a termination fence and the authoritative record, committed and not; `TestRelocatePassConversionRefusesAnUnauthorizedWorker` runs it as a revoked, retired, fenced, wrongly enrolled or other-region worker and finds the claim unwritten each time (ruling 78). `TestRelocationReleaseIsTheChannelPointer` ships and promotes real releases with a generated root key. `TestRelocationPinnerSelectsOverTheVersionedBucket` (hostrelocate) runs the pinner over `genselect.Select` and a versioned bucket holding a marker, a snapshot and a level-0 file. Rev 1's `TestRelocatePassStaysOffWhileThePreflightRefuses` is not a pass test in rev 5: the preflight is the participant's (Task 8a, `TestParticipantCountsNoAttemptWhileRootRefusesTheKind`). Finding 12's test (`TestRelocationLeasesSignForTheCloudRegion`) is folded into `TestRelocationRuntimeOpensADataBucketStore` (K6). |
| -New tests: `TestCreateDoorUsesTheChooserOnlyWithRegionHAOn`, `TestLiveSnapshotJoinsEdgesFilesWithLiveness`, `TestRelocatePassIsOffUnlessBothFlagsAreOn`, `TestRelocatePassStepsEveryRelocationAndSweeps`, `TestRelocatePassAlertsOnceWhileAConversionWaitsForATarget`, `TestRelocatePassStopsCompletesAndReleasesInOnePass`, `TestCommittedLandingRecordsAdmitOnlyACommittedMove`, `TestRelocationReleaseIsTheChannelPointer`, `TestDischargeIsUnqualifiedWithoutEnrolledAt`, `TestRelocationRuntimeOpensADataBucketStore`, `TestRelocationBindsWithARegisteredMovesSource`, `TestRelocatePassConvertsAFencedCommittedLanding`, `TestRelocationPinnerSelectsOverTheVersionedBucket`, `TestWorkerBoundsAConversionWaitingForATarget`. |
| +New tests: `TestCreateDoorUsesTheChooserOnlyWithRegionHAOn`, `TestLiveSnapshotJoinsEdgesFilesWithLiveness`, `TestRelocatePassIsOffUnlessBothFlagsAreOn`, `TestRelocatePassStepsEveryRelocationAndSweeps`, `TestRelocatePassAlertsOnceWhileAConversionWaitsForATarget`, `TestRelocatePassStopsCompletesAndReleasesInOnePass`, `TestCommittedLandingRecordsAdmitOnlyACommittedMove`, `TestRelocationReleaseIsTheChannelPointer`, `TestDischargeIsUnqualifiedWithoutEnrolledAt`, `TestRelocationRuntimeOpensADataBucketStore`, `TestRelocationBindsWithARegisteredMovesSource`, `TestRelocatePassConvertsAFencedCommittedLanding`, `TestRelocatePassConversionRefusesAnUnauthorizedWorker`, `TestRelocationPinnerSelectsOverTheVersionedBucket`, `TestWorkerBoundsAConversionWaitingForATarget`. |
| `internal/console/claim_runtime_chooser_test.go` (new): |
| @@ -11738,6 +12531,7 @@ import ( |
| "crypto/ed25519" |
| "crypto/rand" |
| "encoding/json" |
| + "errors" |
| "strings" |
| "testing" |
| "time" |
| @@ -11753,10 +12547,79 @@ import ( |
| "github.com/superfly/ltx" |
| ) |
| -// J3: the pass converts a landing whose target box is terminated (its |
| -// fence in the roster's bucket) into a failover relocation when the move |
| -// committed, through CommittedLandingRecords over the authoritative record; |
| -// a move that has not committed is moves' case, and nothing is written. |
| +// landingWorld is a landing claim at probed whose target dst is |
| +// terminated (its fence in the roster's bucket), the authoritative record |
| +// of its move (committed or still landing) and a signed roster in which |
| +// wrk, the converting worker, is as change leaves it. |
| +type landingWorld struct { |
| + store release.Store |
| + directory hostactors.Directory |
| + obs hostclaim.Observed |
| + now time.Time |
| + enrollment map[string]string |
| +} |
| + |
| +func newLandingWorld(t *testing.T, committed bool, change func(*hostactors.Roster)) landingWorld { |
| + t.Helper() |
| + ctx := t.Context() |
| + const host, id = "shop.example", "mv_abcdefghijklmnopqrstuvwxyz" |
| + w := landingWorld{store: release.NewDirStore(t.TempDir()), now: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), |
| + enrollment: map[string]string{"src": strings.Repeat("b", 32), "dst": strings.Repeat("c", 32), "spa": strings.Repeat("d", 32), "wrk": strings.Repeat("e", 32)}} |
| + pub, key, err := ed25519.GenerateKey(rand.Reader) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, |
| + Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}, "uk": {Account: "123", AWSRegion: "eu-west-2"}}, |
| + Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} |
| + for box, enr := range w.enrollment { |
| + roster.Boxes[box] = hostactors.Box{Enrollment: enr, Namespace: "platform", BoxID: box, Label: box, Region: "ie", Pool: "default", |
| + Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} |
| + } |
| + if change != nil { |
| + change(&roster) |
| + } |
| + raw, err := release.SignRootEnvelope(release.RootSigner(key), roster) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := w.store.Put(ctx, hostactors.Key, raw, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + w.directory = hostactors.Directory{Store: w.store, PublicKey: pub, Bucket: "bucket"} |
| + w.obs = landingAtProbed(t, w.store, host, id, w.now, w.enrollment) |
| + dst := roster.Boxes["dst"] |
| + fence, _ := json.Marshal(termfence.Record{V: 1, Box: "dst", Enrollment: dst.Enrollment, Cloud: dst.Cloud, State: "terminated", ObservedAt: w.now.Add(-time.Minute).Format(time.RFC3339Nano), |
| + Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "spa", Enrollment: w.enrollment["spa"]}}) |
| + if _, err := w.store.Put(ctx, hostfence.FenceKey(dst.Enrollment), fence, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| + state, committedAt, flipped := provision.MoveLanding, (*time.Time)(nil), w.now.Add(-90*time.Minute) |
| + if committed { |
| + at := w.now.Add(-30 * time.Minute) |
| + state, committedAt = provision.MoveCommitted, &at |
| + } |
| + record := provision.Record{Schema: 1, Host: host, Slug: "shop", App: "jam", Account: "bab", Region: "ie", Channel: "stable", CreatedAt: created, |
| + Move: &provision.MoveState{ID: id, Incarnation: w.obs.Claim.Holder.Incarnation, From: "uk", To: "ie", State: state, CommittedAt: committedAt, FlippedAt: &flipped}} |
| + if err := provision.CreateRecord(ctx, w.store, record); err != nil { |
| + t.Fatal(err) |
| + } |
| + return w |
| +} |
| + |
| +// worker is the converting worker: wrk, running as enrollment, in region. |
| +func (w landingWorld) worker(enrollment, region string) hostrelocate.Worker { |
| + now := func() time.Time { return w.now } |
| + return hostrelocate.Worker{Claims: w.store, Directory: w.directory, Fences: hostfence.ClaimFenceAuthority{Directory: w.directory, Now: now}, |
| + Discharge: func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge { return nil }, |
| + Self: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment}, Region: region, NewID: newClaimID, Now: now} |
| +} |
| + |
| +// J3: the pass converts a landing whose target box is terminated into a |
| +// failover relocation when the move committed, through |
| +// CommittedLandingRecords over the authoritative record; a move that has |
| +// not committed is moves' case, and nothing is written. |
| func TestRelocatePassConvertsAFencedCommittedLanding(t *testing.T) { |
| for _, committed := range []bool{true, false} { |
| name := "uncommitted" |
| @@ -11765,62 +12628,16 @@ func TestRelocatePassConvertsAFencedCommittedLanding(t *testing.T) { |
| } |
| t.Run(name, func(t *testing.T) { |
| ctx := t.Context() |
| - const host, id = "shop.example", "mv_abcdefghijklmnopqrstuvwxyz" |
| - now := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) |
| - store := release.NewDirStore(t.TempDir()) |
| - pub, key, err := ed25519.GenerateKey(rand.Reader) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - enrollment := map[string]string{"src": strings.Repeat("b", 32), "dst": strings.Repeat("c", 32), "spa": strings.Repeat("d", 32), "wrk": strings.Repeat("e", 32)} |
| - roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}}, |
| - Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} |
| - for box, enr := range enrollment { |
| - roster.Boxes[box] = hostactors.Box{Enrollment: enr, Namespace: "platform", BoxID: box, Label: box, Region: "ie", Pool: "default", |
| - Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} |
| - } |
| - raw, err := release.SignRootEnvelope(release.RootSigner(key), roster) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - if _, err := store.Put(ctx, hostactors.Key, raw, ""); err != nil { |
| - t.Fatal(err) |
| - } |
| - d := hostactors.Directory{Store: store, PublicKey: pub, Bucket: "bucket"} |
| - |
| - o := landingAtProbed(t, store, host, id, now, enrollment) |
| - dst := roster.Boxes["dst"] |
| - fence, _ := json.Marshal(termfence.Record{V: 1, Box: "dst", Enrollment: dst.Enrollment, Cloud: dst.Cloud, State: "terminated", ObservedAt: now.Add(-time.Minute).Format(time.RFC3339Nano), |
| - Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment["wrk"]}}) |
| - if _, err := store.Put(ctx, hostfence.FenceKey(dst.Enrollment), fence, ""); err != nil { |
| - t.Fatal(err) |
| - } |
| - |
| - // The authoritative record: the move committed, or still landing. |
| - created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| - state, committedAt, flipped := provision.MoveLanding, (*time.Time)(nil), now.Add(-90*time.Minute) |
| - if committed { |
| - at := now.Add(-30 * time.Minute) |
| - state, committedAt = provision.MoveCommitted, &at |
| - } |
| - record := provision.Record{Schema: 1, Host: host, Slug: "shop", App: "jam", Account: "bab", Region: "ie", Channel: "stable", CreatedAt: created, |
| - Move: &provision.MoveState{ID: id, Incarnation: o.Claim.Holder.Incarnation, From: "uk", To: "ie", State: state, CommittedAt: committedAt, FlippedAt: &flipped}} |
| - if err := provision.CreateRecord(ctx, store, record); err != nil { |
| - t.Fatal(err) |
| - } |
| - |
| - w := hostrelocate.Worker{Claims: store, Directory: d, Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: func() time.Time { return now }}, |
| - Discharge: func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge { return nil }, Now: func() time.Time { return now }} |
| - before := o |
| - if err := convertFencedLanding(w, store)(ctx, store, before); err != nil { |
| + w := newLandingWorld(t, committed, nil) |
| + if err := convertFencedLanding(w.worker(w.enrollment["wrk"], "ie"), w.store)(ctx, w.store, w.obs); err != nil { |
| t.Fatalf("convert: %v", err) |
| } |
| - after, err := hostclaim.Read(ctx, store, host) |
| + after, err := hostclaim.Read(ctx, w.store, w.obs.Host) |
| if err != nil { |
| t.Fatal(err) |
| } |
| if !committed { |
| - if after.ETag != before.ETag || after.Claim.Op.Kind != hostclaim.KindLanding { |
| + if after.ETag != w.obs.ETag || after.Claim.Op.Kind != hostclaim.KindLanding { |
| t.Fatalf("an uncommitted landing was written: %+v", after.Claim.Op) |
| } |
| return |
| @@ -11833,6 +12650,79 @@ func TestRelocatePassConvertsAFencedCommittedLanding(t *testing.T) { |
| } |
| } |
| +// Astra round 6, finding 4: the conversion is the worker's, under Step's |
| +// own checks of the caller. A worker whose roster entry is another |
| +// enrollment (revoked and re-enrolled), one that is not active, one that |
| +// is fenced, or one enrolled outside the region it claims, is refused |
| +// with ErrNotWorker; a worker of another region leaves the landing alone. |
| +// None of them writes the claim, though the move committed and its target |
| +// is fenced. |
| +func TestRelocatePassConversionRefusesAnUnauthorizedWorker(t *testing.T) { |
| + type tc struct { |
| + name string |
| + change func(*hostactors.Roster) |
| + enrollment, region string |
| + want error |
| + } |
| + moved := func(state, region string) func(*hostactors.Roster) { |
| + return func(r *hostactors.Roster) { |
| + b := r.Boxes["wrk"] |
| + if state != "" { |
| + b.State = state |
| + } |
| + if region != "" { |
| + b.Region = region |
| + } |
| + r.Boxes["wrk"] = b |
| + } |
| + } |
| + for _, c := range []tc{ |
| + {name: "a revoked enrollment", enrollment: strings.Repeat("f", 32), region: "ie", want: hostrelocate.ErrNotWorker}, |
| + {name: "a retired worker", change: moved("retired", ""), region: "ie", want: hostrelocate.ErrNotWorker}, |
| + {name: "enrolled in another region", change: moved("", "uk"), region: "ie", want: hostrelocate.ErrNotWorker}, |
| + {name: "a worker of another region", change: moved("", "uk"), region: "uk"}, |
| + } { |
| + t.Run(c.name, func(t *testing.T) { |
| + ctx := t.Context() |
| + w := newLandingWorld(t, true, c.change) |
| + if c.enrollment == "" { |
| + c.enrollment = w.enrollment["wrk"] |
| + } |
| + err := convertFencedLanding(w.worker(c.enrollment, c.region), w.store)(ctx, w.store, w.obs) |
| + if c.want == nil && err != nil || c.want != nil && !errors.Is(err, c.want) { |
| + t.Fatalf("got %v, want %v", err, c.want) |
| + } |
| + after, err := hostclaim.Read(ctx, w.store, w.obs.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if after.ETag != w.obs.ETag || after.Claim.Op.Kind != hostclaim.KindLanding { |
| + t.Fatalf("an unauthorized worker wrote the claim: %+v", after.Claim.Op) |
| + } |
| + }) |
| + } |
| + t.Run("a fenced worker", func(t *testing.T) { |
| + ctx := t.Context() |
| + w := newLandingWorld(t, true, nil) |
| + o, err := w.directory.Read(ctx) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + wrk := o.Roster.Boxes["wrk"] |
| + fence, _ := json.Marshal(termfence.Record{V: 1, Box: "wrk", Enrollment: wrk.Enrollment, Cloud: wrk.Cloud, State: "terminated", ObservedAt: w.now.Add(-time.Minute).Format(time.RFC3339Nano), |
| + Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "spa", Enrollment: w.enrollment["spa"]}}) |
| + if _, err := w.store.Put(ctx, hostfence.FenceKey(wrk.Enrollment), fence, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := convertFencedLanding(w.worker(w.enrollment["wrk"], "ie"), w.store)(ctx, w.store, w.obs); !errors.Is(err, hostrelocate.ErrNotWorker) { |
| + t.Fatalf("got %v", err) |
| + } |
| + if after, err := hostclaim.Read(ctx, w.store, w.obs.Host); err != nil || after.ETag != w.obs.ETag { |
| + t.Fatalf("a fenced worker wrote the claim: %v", err) |
| + } |
| + }) |
| +} |
| + |
| // landingAtProbed is a landing claim as moves leaves it at probed: the |
| // move from src to dst flipped, the target's probe admitted. Its seal is a |
| // real one, so moves.CommitScopes reads it. |
| @@ -12617,7 +13507,7 @@ func chooseBox(host string, cands []boxCandidate, running map[string]boxLoad, ap |
| log.Printf("console: an incomplete read of the edges files or box snapshots (edges complete %v, snapshots complete %v), so no region outside the home is automatically eligible this request; said once", snap.Complete, snap.BoxesComplete) |
| ``` |
| -- [ ] **Step 4: The worker's fence and the no-target bound (J3-3, ruling 35)** |
| +- [ ] **Step 4: The worker's landing conversion and the no-target bound (J3-3, rulings 35, 78)** |
| `internal/hostrelocate/worker.go`: |
| @@ -12632,19 +13522,42 @@ func chooseBox(host string, cands []boxCandidate, running map[string]boxLoad, ap |
| "time" |
| "github.com/carlosframework/platform/internal/hostactors" |
| -@@ -195,6 +196,32 @@ func (w Worker) fence(ctx context.Context, o hostactors.Observed, box string) (* |
| +@@ -195,6 +196,55 @@ func (w Worker) fence(ctx context.Context, o hostactors.Observed, box string) (* |
| return &hostclaim.FenceRef{Enrollment: b.Enrollment, Key: key, SHA256: hex.EncodeToString(sum[:])}, f, nil |
| } |
| -+// Fence is box's termination fence as the worker reads it: the reference |
| -+// the claim's fence authority verifies, and the record; nil when the box |
| -+// has none. |
| -+func (w Worker) Fence(ctx context.Context, box string) (*hostclaim.FenceRef, termfence.Record, error) { |
| -+ o, err := w.Directory.Read(ctx) |
| ++// ConvertLanding is J3 for one landing claim, under the same worker |
| ++// authorization as Step (Astra round 6 finding 4): this box must be an |
| ++// active, unfenced worker whose current roster entry is the enrollment it |
| ++// runs as, in the region the landing lands in. A landing whose op.exec |
| ++// box (the move's target) has a termination fence becomes a failover |
| ++// relocation when its move committed (guard); an uncommitted one is |
| ++// moves' case (ErrWrongState). A landing into another region, one with no |
| ++// executor, or one whose target has no fence is left alone with nothing |
| ++// written; a caller that is not such a worker is ErrNotWorker with nothing |
| ++// written. |
| ++func (w Worker) ConvertLanding(ctx context.Context, obs hostclaim.Observed, guard hostclaim.CommittedLandingGuard) (hostclaim.Observed, error) { |
| ++ if w.Claims == nil || w.Directory.Store == nil || w.Fences == nil || w.Discharge == nil || w.NewID == nil || w.Now == nil || w.Self.Box == "" || w.Self.Enrollment == "" || w.Region == "" || guard == nil { |
| ++ return obs, ErrUnsupported |
| ++ } |
| ++ o, err := w.self(ctx) |
| + if err != nil { |
| -+ return nil, termfence.Record{}, err |
| ++ return obs, err |
| ++ } |
| ++ if obs.Absent || obs.Claim == nil { |
| ++ return obs, nil |
| ++ } |
| ++ c := obs.Claim |
| ++ box := c.Op.Exec |
| ++ if c.Op.Kind != hostclaim.KindLanding || c.Op.Intent["to"] != w.Region || box == "" || box == "none" { |
| ++ return obs, nil |
| ++ } |
| ++ ref, record, err := w.fence(ctx, o, box) |
| ++ if err != nil || ref == nil { |
| ++ return obs, err |
| + } |
| -+ return w.fence(ctx, o, box) |
| ++ d := w.Discharge(ctx, o.Roster.Boxes[box], record) |
| ++ return hostclaim.ConvertCommittedLanding(ctx, w.Claims, obs, c.Op.ID, box, *ref, w.Fences, guard, d, w.NewID) |
| +} |
| + |
| +// newestFenceObservation is the latest fence.<box>.observed_at the op |
| @@ -12665,7 +13578,7 @@ func chooseBox(host string, cands []boxCandidate, running map[string]boxLoad, ap |
| // Start writes the start CAS for req (§5.8 "The start CAS", §5.10). It |
| // refuses unless the claim serves on req.Source; the roster shows the |
| // starter, the target and (for a drain) the source active, unfenced and |
| -@@ -344,6 +371,13 @@ func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, erro |
| +@@ -344,6 +394,13 @@ func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, erro |
| // as an initial failover's target does. |
| next, err := w.next(ctx, obs) |
| if err != nil { |
| @@ -13062,28 +13975,16 @@ func (r *RelocationRuntime) releaseCompletedHold(ctx context.Context, host strin |
| } |
| } |
| -// convertFencedLanding is J3: a landing whose op.exec box has a |
| +// convertFencedLanding is J3 through the worker's own authorization |
| +// (hostrelocate.Worker.ConvertLanding): a landing whose op.exec box has a |
| // termination fence becomes a failover relocation when its move committed |
| // (CommittedLandingRecords). An uncommitted one is moves' case and answers |
| -// nil (ErrWrongState); any other refusal is returned for the pass to |
| +// nil (ErrWrongState); any other refusal, a caller that is not an active |
| +// worker of the landing's region included, is returned for the pass to |
| // alert once. |
| func convertFencedLanding(w hostrelocate.Worker, store release.Store) func(context.Context, release.Store, hostclaim.Observed) error { |
| return func(ctx context.Context, _ release.Store, obs hostclaim.Observed) error { |
| - c := obs.Claim |
| - box := c.Op.Exec |
| - if box == "" || box == "none" { |
| - return nil |
| - } |
| - ref, record, err := w.Fence(ctx, box) |
| - if err != nil || ref == nil { |
| - return err |
| - } |
| - o, err := w.Directory.Read(ctx) |
| - if err != nil { |
| - return err |
| - } |
| - d := w.Discharge(ctx, o.Roster.Boxes[box], record) |
| - _, err = hostclaim.ConvertCommittedLanding(ctx, store, obs, c.Op.ID, box, *ref, w.Fences, CommittedLandingRecords{Records: MovesRecordsFrom(store)}, d, newClaimID) |
| + _, err := w.ConvertLanding(ctx, obs, CommittedLandingRecords{Records: MovesRecordsFrom(store)}) |
| if errors.Is(err, hostclaim.ErrWrongState) { |
| return nil |
| } |
| @@ -13333,10 +14234,28 @@ Before `CARLOS_REGION_HA` is set on any box with a second box in its region, on |
| 15. **Census.** After a relocation completes on dev, `find /data -path '*<host>*'` on the target lists only the host's instance directory, the census-opaque `relocate/` (N9) and moves' relay directories (N11, until M-K25), and on the source nothing outside them after its `park-local`; within a pass of completion `relocate/` holds nothing of that op. |
| 16. **C11.** Until it lands each target gets one launchable probe attempt; set `LaterProbeAttempts` only after it does. |
| 17. **O2′ on dev.** A relocation of an app on its own `instance.key` pins with no `InstanceKey`. After O6, a relocation of a platform-key app pins `InstanceKey`, and changing a configured key between pin and install makes the install refuse (`VerifyDelivered`). |
| -18. **R5-3 (gate).** `Reconciler.FenceClosed` and `HoldRow` are decided and, if wired, the source's row goes held at its fence's close with a status the edge does not serve. Until then the source's row stays as reconcile left it during `releasing`. |
| +18. **Task 10b (gate; R5-3).** Task 10b has landed and its tests pass: `Reconciler.FenceClosed` and `HoldRow` are wired, the source's row goes held at its fence's close with a status the edge does not serve, and the completion or the source's park-local ends the hold. Relocation is not switched on anywhere before this. |
| --- |
| +### Task 10b: The source's row held at its fence's close (R5-3), mandatory before activation |
| + |
| +Astra round 6 accepted R5-3 deferred while relocation is dormant, on one condition: this task lands, with its tests passing, before relocation is switched on anywhere (ruling 79; checklist item 18; the activation line in Global Constraints; "After this plan"). It is not built in rev 5.1 and nothing in Tasks 8a to 11 depends on it. Its code and test bodies are written when it is scheduled, against the code as it then is, and compiled task by task as this plan's tasks were. |
| + |
| +What it does. Stage A's `hostbox.Reconciler` has two hooks no production assembly sets: `FenceClosed(ctx, obs) (bool, error)` and `HoldRow(ctx, host, holdOp) error` (hostbox/reconcile.go:33-61, 250-260). Until they are set, reconcile under a relocating claim is main's `Serving` path, which refuses and leaves every row unchanged, so the source's row is not held during `releasing`. |
| + |
| +**Files (expected):** |
| +- Modify: `internal/registry/registry.go` (a hold write that sets the row's status to one the edge does not serve together with `hold_op`, under the existing `hold_op` guard; `SetClaimHold` sets only `hold_op`), `internal/host/claimassembly*.go` or wherever the box's `Reconciler` is built (set `FenceClosed` from this box's activator fence for the host and `HoldRow` from the new registry write, only with `CARLOS_REGION_HA` on), `internal/host/relocationowners.go` (`relocationReleaseHold` and the source's park-local release the source's hold). |
| +- Test: `internal/hostbox/reconcile_relocation_source_test.go`, `internal/registry/relocating_test.go`, `internal/host/relocationsourcehold_test.go`. |
| + |
| +**Tests it must have, each asserting behaviour:** |
| +1. During `releasing`, once this box's admission fence for the host is closed, reconcile holds the source's row for the op with a status the edge does not serve; before the fence closes, the row is unchanged. |
| +2. The hold write never takes a row another op holds, and a replay is a no-op. |
| +3. The completion (or the source's park-local) ends the source's hold; a failover from a fenced source leaves the row to the park-local. |
| +4. With `CARLOS_REGION_HA` off, reconcile is exactly main's. |
| + |
| +**Gate:** relocation is not switched on anywhere until this task has landed and its tests pass. |
| + |
| ### Task 11: The merge gate (L18) |
| Unchanged from the 1b-next plan's Task 11 (L13558-13586), with the model at `origin/region-ha-two-box-build` (e814ece3 keeps the landing's op id). Add to the joint round's list: the committed-landing conversion on both sides of the commit; the one-cold-restart completion by ruling 49's sequence, through the serving wake's N6 handover with C10; a drain under ruling 28's order; a probe-failed target with C1 open going `stuck` at the deadline and never queuing an obligation; a target replaced mid-install keeping main's denial until its `park-local` (C6); and no attempt counted while root refuses the kind (ruling 48). |
| @@ -13352,7 +14271,7 @@ Unchanged from the 1b-next plan's Task 11 (L13558-13586), with the model at `ori |
| | What | Waits for | |
| |---|---| |
| | Drains (`reason: drain`, `rebalance`) | C1 | |
| -| Switching relocation on anywhere | C10 confirmed on #236, C8 with O1 registered, O2′ wired, R5-3 decided, the two-box run on dev (checklist item 12), Task 11 | |
| +| Switching relocation on anywhere | C10 confirmed on #236, C8 with O1 registered, O2′ wired, Task 10b landed (R5-3), the two-box run on dev (checklist item 12), Task 11 | |
| | More than one probe attempt per target | C11 | |
| | A shorter denial for an excluded target | C6 | |
| | The install's three actions under the claim's own Policy | C9 | |
| @@ -13369,18 +14288,18 @@ Unchanged from the 1b-next plan's Task 11 (L13558-13586), with the model at `ori |
| - **The two-box end-to-end run** (R5-4) needs a two-box harness and an activator test service; until one exists it is checklist item 12 and Task 11's joint round. |
| - **W1, W2 and W4** are observable only on a box. |
| -## Unresolved in rev 5 |
| +## Unresolved in rev 5.1 |
| -- **R5-2.** Rev 3's ported lifecycle property (ruling 63) is withdrawn and replaced by deterministic tests of region HA's parts; Astra and Paul to accept or ask for the seams it needs (main's hostexec clock and root's stop replay in `productionClaimLocal`). |
| -- **R5-3.** `Reconciler.FenceClosed` and `HoldRow` are not wired in production; a decision is needed, including a hold write that sets the status (`registry.SetClaimHold` sets only `hold_op`). |
| -- **R5-4.** `TestRelocationFromAnEmptyTargetToTheServingWake` and rev 4's harness tests 1 and 2 are withdrawn; the per-hop integration tests stand in for them until a two-box harness exists. |
| +- **R5-2.** Rev 3's ported lifecycle property (ruling 63) stays withdrawn. The successor case Astra named is now prevented (ruling 77) and tested through the real start transitions; other interleavings the property explored are covered only by the per-hop tests. |
| +- **R5-3.** Accepted while dormant; Task 10b is written and landed before activation (ruling 79). |
| +- **R5-4.** Accepted as an explicit activation gate (checklist item 12), with no claim that the per-hop tests are equivalent integration coverage. |
| - **M-K25** is a new moves ask; N11 is its committed fallback. |
| - **C10** is committed as region HA code; the claim session's confirmation is outstanding. |
| ## Self-review |
| -- **Spec coverage.** §5.8 release (8a), pin and build (8b), local install and receipt (8c), probe admission with `KeepStamp`, `PredecessorCertified`, P opened once, P's marker (8d), the end of the probe and the serving restart (9), who acts and the passes (10); §5.3 under ruling 31 (reconcile leaves relocating rows, R5-1, R5-3); §6 and §8 as the checklist (10); L18 (11). |
| -- **Order.** No task uses a symbol a later task introduces: each task's code compiled and its tests passed at its own commit (Rev 5 record). |
| +- **Spec coverage.** §5.8 release (8a), pin and build (8b), local install and receipt (8c), probe admission with `KeepStamp`, `PredecessorCertified`, P opened once, P's marker (8d), the end of the probe and the serving restart (9), who acts and the passes (10); §5.3 under ruling 31 (reconcile leaves relocating rows, R5-1; the source's hold is Task 10b); §6 and §8 as the checklist (10); L18 (11). |
| +- **Order.** No task uses a symbol a later task introduces: each task's code compiled and its tests passed at its own commit (the rev 5.1 record). Task 10b is not built and nothing depends on it. |
| - **Placeholders.** None. Every function has its body and every test asserts behaviour; the withdrawn tests are named in R5-2 and R5-4. |
| - **Type consistency.** `RelocationEffects`' fields are 8a's; `RelocationAssemblyConfig` is 8a's, extended by 8c (`OptDir`, the two seams) and 9 (`Discharge`); `relocationOpDir` names every op journal; `CertifiedFor` is shared by the participant and 8d; `MintHeld(rec, captured, status, op)` is called by 8c only; `ProbeStop` by the participant, 9 and N6; `writerSessionReceipt` by 9's stop and N6. |
| - **Dependencies.** Every open ask is in the dependency table with a fallback or an explicit activation gate. C7 is withdrawn. |