amadan
docs: region HA plan 1b-next rev 6 resolves Astra round 5 (6 accepted, none declined)
R4-5: Finish clears a controller record only straight after the scoped cleanup succeeds in the same call (hostexec's Close reports a cleanup failure once), and participations are journaled before the controller opens, so a restart finds every record. New 1: boxes owing park-local are excluded from selection and refused in the replacement CAS. New 2: the automatic fence passes park-generation and park-replica to region:<r>. New 3: Task 9a keeps the serving-only check with region HA off. New 4: the journal listing ignores .tmp and reports damaged entries beside the readable ones. New 5: the release flush stages the env file's deletion. Dry-run on 3407d114; 11 guards removed in turn, each failing its test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+702
−174
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+702 −174
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index 084a16226..24ef7beb0 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,30 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 5) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 6) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 4 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 5 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 6: round 5 dispositions |
| + |
| +Astra's round 5 (`CODEX-PLAN-REGION-HA-1BN-R5.md`, against `996b24c3`) left R4-5 partly resolved and added five findings. Each was checked against the tree at 3407d114 and the plan before anything changed; all six are accepted and none is declined. Every one lands in stage A code, dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds for darwin, windows and linux/arm64, targeted tests), with 11 new guards removed in turn (listed in the Self-review); R4-5's restart obligations are carried into the stage B re-plan as well. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| R4-5 | A, B | `hostexec.Controller.Close` (controller.go:480) sets `closed` before returning `errors.Join(err, c.CleanupError())` and returns nil on every later call; rev 5's `Finish` cleared after any nil `Close`, and the restart path cleared from the journaled binding with no cleanup at all. Rev 5 also journaled only inside `Finish`, so a record left by a crash (or a failed `Open`, which writes `lost` once it holds the live lock) before the first `Finish` was never found. | accepted | Repeat, do not trust: `RelocationControllers.Cleanup(ctx, binding)` is the scoped cleanup (Task 8a's `relocationCleanup`, authenticated to the binding's box and enrollment) and `Finish` runs it, and needs it to succeed, in the same call as every clear, after `Close` and on the restart path alike; a failure keeps the record, the entry and the journal. The participation is journaled before the controller opens (`Acknowledge` refuses without a journal), so a restart finds every record it may have left; the journal write syncs its directory. | Task 7 (`TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`: cleanup failure, a sweep after it, a restart while it still fails, then success; `TestARestartFindsAParticipationThatNeverReachedFinish`; `TestFinishClearsTheDurableRecordSoUpdatesAreEligible` with a real hostexec controller whose cleanup fails); stage B re-plan | |
| +| 1 | A | `hostops` `eligible` excluded only `Lost`; `Worker.next` excluded the source, `Lost` and replaced targets; `ReplaceTarget` reaches `install` directly in `installing` and `installed`, never through `ToInstalling`'s `park-local` check. | accepted | `hostclaim.OwesLocal` and `LocalChoreBoxes`. Selection excludes every box owing a `park-local` (the chooser's eligible set, so create, `Permits` and the worker's spare all agree; `Worker.next` too), and `replaceTargetSlot` refuses one with `ErrPendingChore` inside the CAS, for `ReplaceTarget` at every phase and `NameTarget`. | Task 1 (`TestRelocationReplacementRefusesABoxOwingALocalRetirement`, in `starting`, `installing` and `installed`), Task 2 (`TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/owing_a_local_retirement`), Task 5 (`TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement`) | |
| +| 2 | A | `fenceBox` journaled the fence, added `lost`, discharged `park-local` and retained the loss, and touched no regional obligation; `lost.go:302` reassigns `box:<b>`'s `park-generation` and `park-replica` to `region:<r>` for a declared loss. | accepted | `fenceBox` passes every `park-generation` and `park-replica` owned by `box:<b>` to `region:<obligation's region>` in the same CAS, changing nothing but `owner` (an obligation with no region refuses, as in `DeclareLost`); the retained `AutomaticLoss` names them (`reassigned`, the D4 record's `reassign_regional`). Every automatic fence CAS shares it: start, `FenceSource`, `ReplaceTarget`, `FenceExhaustedTarget`. | Task 1 (`TestRelocationFenceReassignsTheDeadBoxsRegionalWork`, unqualified and qualified discharge); R5 | |
| +| 3 | A | Task 9a's `check` replaced `Serving` with `RelocationRow` unconditionally; with both callbacks nil a `moving` executor participant is `RowHeld` and passed, and `Install` went on to `Activate` and `RecordInventory` before `WorkSource.SelectAt` (work.go:94) refused. | accepted | `Reconciler.relocationAware()` is `FenceClosed != nil \|\| HoldRow != nil` (Task 10 sets both only with `CARLOS_REGION_HA`). Off, `check` is today's `Serving` and `Install` skips the row decision and the hold-only path entirely. | Task 9a (`TestReconcileWithoutAFenceObserverIsTodays`: a relocating source in `starting` and `releasing` and a `moving` executor, dry run and real, refused with the claim's ETag and every file under the guard's data directory unchanged) | |
| +| 4 | A | `journaled` decoded every directory entry, so `writeFileSynced`'s truncated `.json.tmp` failed the whole listing, and `Retained` dropped the error. | accepted | Only names ending `.json` are committed records; a `.tmp` is ignored (its participation opened nothing: the record precedes the open). A committed record that cannot be read is reported, joined into the error, beside every readable one. `Effects.Retained` returns `([]Participation, error)`; `Sweep` reports the error and still finishes every readable participation. | Task 7 (`TestARestartFindsAParticipationThatNeverReachedFinish`: one valid journal, one truncated `.tmp`, then one damaged committed entry), Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`) | |
| +| 5 | A, B | `Flush` loads the existing manifest through `hostfiles.Open` and did nothing on `ENOENT`, so `Platform[EnvItem]` kept the old bytes. | accepted | `(*hostfiles.Publisher).StagePlatformDelete(name)` (in `platform.go`) drops the item at a new sequence; `Flush` stages it on `ENOENT`, so `ReadPlatform` on the final revision returns `os.ErrNotExist`. Stage B's local install treats that as "no env file" (re-plan). | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`: published, removed, flushed again, then a no-op repeat) | |
| + |
| +Astra's readings of rulings 1 to 24 stand; ruling 20's cleanup is completed by R4-5 above. |
| + |
| ## Rev 5: round 4 dispositions |
| Astra's round 4 (`CODEX-PLAN-REGION-HA-1BN-R4.md`, against `418daf58`) left three earlier findings partly resolved and added six. All nine are accepted; none is declined. Under ruling 24, stage A findings are fixed in code (dry-run on a clean 3407d114, 7 new guards removed in turn, listed in the Self-review); stage B findings are fixed at the contract level: what must happen, which request covers it, which test proves it. |
| @@ -101,7 +116,7 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) | |
| | R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a | |
| | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | |
| -| R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | |
| +| R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences), and the regional obligations the fence passed to the region (rev 6: `reassigned`, D4's `reassign_regional`). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | |
| | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | |
| | R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | |
| | R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | |
| @@ -129,7 +144,7 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 16. **A pin is bound to the claim generation it was selected against** (rev 2; B2-3). Its key is `control/relocations/<host>/<op>/pin-<gen>.json` and `JournalPin` requires `gen` to be the observed claim's. A pin whose journal CAS lost is abandoned where it lies (never referenced, cheap, reaped by the storage follow-up of ruling 5); the next pass selects afresh at the new generation. Every bound `JournalPin` checks (the release's positions, the failover window) is checked before the object is written, so a doomed pin is never published. *Cost if wrong:* abandoned objects under a prefix nothing reads. |
| 17. **Reconcile has three authorities for a relocation row** (rev 2; §5.3, 6b's `ClaimOwnership.Valid` needs a stamp). The executor (the source until its fence closes, the target from `installing` on) activates its own installation stamp and mints or keeps its row through the guarded path. A kept row of a box that is no longer the executor (the source after its release, a replaced target) gets its hold through `registry.SetClaimHold` under the flock, conditional on the row's recorded ownership: a hold grants nothing, so it needs no installation authority. A target before `installing` has no installation and mints nothing; the edges file already withholds the host from a box that is not `op.exec` (§24). *Cost if wrong:* a request that reaches a pre-installation target meets the box's unknown-host answer instead of a hold, for the seconds between `starting` and `installing`. |
| 19. **Probe attempts are counted before they are made** (rev 3; §5.8 "limits": at most three, then `probe-failed`). `BeginProbeAttempt` is one CAS per attempt, keyed by box, so a crash, a lost response or a restart never buys a fourth and a replacement target starts at zero; the participant enters `probe-failed` when refused. *Cost if wrong:* an attempt lost to a crash before its admission still counts; the bound is the spec's. |
| -20. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's. |
| +20. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. *Rev 6:* a durable record is cleared only straight after that scoped cleanup succeeds in the same call (`Finish` repeats it after `Close`, and on a restart), because hostexec reports a cleanup failure from `Close` once and nil afterwards; the participation is journaled before its controller opens, so a restart finds every record. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's. |
| 21. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations/<host>/<op>/manifest-<sha256>.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers. |
| 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| @@ -162,7 +177,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight and rev 5's seven (listed in the Self-review). Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven and rev 6's eleven (listed in the Self-review). Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -197,7 +212,7 @@ Left out, with where it goes: |
| ### Task 1: The `relocating` operation in the claim library (L1, L2 to L4, L9, L11) |
| -Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.<box>`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies. |
| +Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.<box>`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies; the same CAS passes the box's regional work (`park-generation`, `park-replica` owned by `box:<b>`) to `region:<r>`, as `DeclareLost` does (rev 6). A box owing a `park-local` is never named a target (`replaceTargetSlot`, rev 6). |
| **Files:** |
| - Create: `internal/hostclaim/relocation.go` |
| @@ -206,7 +221,7 @@ Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostcl |
| **Interfaces:** |
| - Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`. |
| -- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`) and `Claim.EvidenceRefs`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. |
| +- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`, `Reassigned`) and `Claim.EvidenceRefs`; `OwesLocal(c, box) bool`, `LocalChoreBoxes(c) []string`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. |
| - Journal keys later tasks read: `exec_ack.<box>`, `fence.<box>`, `fence.<box>.observed_at`, `discharge.<box>`, `reason`, `releasing_since`, `release`, `release.phase`, `released_by_target`, `pin`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe_attempts.<box>`, `probe`, `probe_failed_since`, `replacements`, `excluded_targets`, `stuck`. Claim fields: `evidence_refs`, `loss_refs`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -1059,6 +1074,87 @@ func TestRelocationLocalInstallReceiptIsBound(t *testing.T) { |
| t.Fatalf("probing on another box's receipt: %v", err) |
| } |
| } |
| + |
| +// Round 5 finding 1: a spare that still owes a local retirement for the |
| +// host is never named a replacement target, whatever phase the |
| +// replacement resumes at (installing and installed hand it op.exec in the |
| +// same CAS); a clean spare is. |
| +func TestRelocationReplacementRefusesABoxOwingALocalRetirement(t *testing.T) { |
| + ctx := context.Background() |
| + for _, phase := range []string{PhaseStarting, PhaseInstalling, PhaseInstalled} { |
| + t.Run(phase, func(t *testing.T) { |
| + s, o := relAtPhase(t, phase) |
| + o = step(t, s, o, func(c *Claim) error { |
| + c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSpare, Retirement: "r", Owner: "box:" + relSpare}) |
| + return nil |
| + }) |
| + ref := putFence(t, s, relTarget) |
| + if _, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}); !errors.Is(err, ErrPendingChore) { |
| + t.Fatalf("a box owing park-local was named: %v", err) |
| + } |
| + if _, err := Update(ctx, s, o, NameTarget("rel-1", Target{relSpare, relEnrollment(relSpare)})); err == nil { |
| + t.Fatal("NameTarget named a box owing park-local") |
| + } |
| + o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare2, relEnrollment(relSpare2)}, "P2", relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + c := o.Claim |
| + if c.Op.Intent["target_box"] != relSpare2 || slices.Contains(c.Op.Participants, relSpare) || slices.Contains(c.Touched.Boxes, relSpare) { |
| + t.Fatalf("the clean spare was not chosen: %+v touched %v", c.Op, c.Touched.Boxes) |
| + } |
| + if phase != PhaseStarting && c.Op.Exec != relSpare2 { |
| + t.Fatalf("exec %s", c.Op.Exec) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// Round 5 finding 2: the automatic fence passes the dead box's regional |
| +// work (park-generation, park-replica owned by box:<b>) to its region in |
| +// the same CAS, as DeclareLost does for a declared loss, keeping every |
| +// other field; another box's work is untouched; local discharge, qualified |
| +// or not, changes nothing about it. |
| +func TestRelocationFenceReassignsTheDeadBoxsRegionalWork(t *testing.T) { |
| + ctx := context.Background() |
| + gen := Obligation{ID: "pg", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0", Retirement: "rg", Owner: "box:" + relSource} |
| + rep := Obligation{ID: "pr", Op: "old-op", Kind: "park-replica", Region: "uk", Retirement: "rr", Owner: "box:" + relSource} |
| + other := Obligation{ID: "po", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0b", Retirement: "ro", Owner: "box:" + relSpare} |
| + local := Obligation{ID: "pl", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "rl", Owner: "box:" + relSource} |
| + for _, qualified := range []bool{false, true} { |
| + s, o := relServing(t) |
| + o = step(t, s, o, func(c *Claim) error { |
| + c.Obligations = append(c.Obligations, gen, rep, other, local) |
| + return nil |
| + }) |
| + ref := putFence(t, s, relSource) |
| + in := relStart(ReasonFailover, &ref) |
| + in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}} |
| + o, err := StartRelocation(ctx, s, o, in, relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + byID := map[string]Obligation{} |
| + for _, ob := range o.Claim.Obligations { |
| + byID[ob.ID] = ob |
| + } |
| + wantGen, wantRep := gen, rep |
| + wantGen.Owner, wantRep.Owner = "region:ie", "region:uk" |
| + if !reflect.DeepEqual(byID["pg"], wantGen) || !reflect.DeepEqual(byID["pr"], wantRep) || !reflect.DeepEqual(byID["po"], other) { |
| + t.Fatalf("qualified=%v: regional work %+v", qualified, o.Claim.Obligations) |
| + } |
| + if _, owes := byID["pl"]; owes == qualified { |
| + t.Fatalf("qualified=%v: the local chore %+v", qualified, o.Claim.Obligations) |
| + } |
| + want := []ObligationRef{{ID: "pg", Op: "old-op", Retirement: "rg"}, {ID: "pr", Op: "old-op", Retirement: "rr"}} |
| + if len(o.Claim.EvidenceRefs) != 1 || !reflect.DeepEqual(o.Claim.EvidenceRefs[0].Reassigned, want) { |
| + t.Fatalf("qualified=%v: the retained loss does not name the reassignment: %+v", qualified, o.Claim.EvidenceRefs) |
| + } |
| + if err := o.Claim.Validate(); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -1245,7 +1341,11 @@ type AutomaticLoss struct { |
| Session uint64 `json:"session,omitempty"` |
| LastAdmission *AdmissionRecord `json:"last_admission,omitempty"` |
| Discharge LocalDischarge `json:"discharge"` |
| - Timers TimerKnowledge `json:"timers"` |
| + // Reassigned is the regional work (park-generation, park-replica) |
| + // the box owned that the same CAS passed to its region (L3, as |
| + // DeclareLost's reassign_regional): the console's D4 record names it. |
| + Reassigned []ObligationRef `json:"reassigned,omitempty"` |
| + Timers TimerKnowledge `json:"timers"` |
| // Reported is set by the console once the D4 record holds this loss |
| // and its alert work exists. Until then reporting is pending. |
| Reported bool `json:"reported,omitempty"` |
| @@ -1359,9 +1459,37 @@ func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced |
| return nil |
| } |
| -// fenceBox puts box on lost with its fence reference (L2, L4) and, when the |
| -// volume evidence qualifies (L3), discharges that box's park-local chores |
| -// for this host in the same CAS. It never removes anything else. D4: the |
| +// OwesLocal reports whether box owes a park-local chore on this claim: a |
| +// local retirement not yet done (spec 1 §2: such a box is never chosen to |
| +// install the host, and §4: a park-local on the target refuses). |
| +func OwesLocal(c *Claim, box string) bool { |
| + for _, o := range c.Obligations { |
| + if o.Kind == "park-local" && o.Box == box { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| +// LocalChoreBoxes lists every box that owes a park-local chore on the |
| +// claim, for a chooser's exclusions. |
| +func LocalChoreBoxes(c *Claim) []string { |
| + var out []string |
| + for _, o := range c.Obligations { |
| + if o.Kind == "park-local" && o.Box != "" && !slices.Contains(out, o.Box) { |
| + out = append(out, o.Box) |
| + } |
| + } |
| + return out |
| +} |
| + |
| +// fenceBox puts box on lost with its fence reference (L2, L4), passes the |
| +// regional work the box owned (park-generation, park-replica owned by |
| +// box:<box>) to its region, region:<r>, keeping each obligation's id, |
| +// operation, retirement, region, generation and park scope (L3, exactly |
| +// as DeclareLost does for a declared loss, lost.go), and, when the volume |
| +// evidence qualifies (L3), discharges that box's park-local chores for |
| +// this host, all in the same CAS. It never removes anything else. D4: the |
| // same CAS retains the loss outside the operation (evidence_refs, |
| // loss_refs) with the box's last writer session, the discharge verdict and |
| // timer knowledge "unknown", so the console can materialize the loss record |
| @@ -1383,6 +1511,17 @@ func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDis |
| if !slices.Contains(c.Lost, box) { |
| c.Lost = append(c.Lost, box) |
| } |
| + var reassigned []ObligationRef |
| + for i := range c.Obligations { |
| + o := &c.Obligations[i] |
| + if (o.Kind == "park-generation" || o.Kind == "park-replica") && o.Owner == "box:"+box { |
| + if o.Region == "" { |
| + return fmt.Errorf("%w: regional obligation %q names no region", ErrWrongState, o.ID) |
| + } |
| + o.Owner = "region:" + o.Region |
| + reassigned = append(reassigned, ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}) |
| + } |
| + } |
| if d == nil { |
| d = &LocalDischarge{Reason: "no volume evidence supplied"} |
| } |
| @@ -1409,12 +1548,12 @@ func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDis |
| } |
| } |
| } |
| - return retainAutomaticLoss(c, box, ref, observedAt, *d) |
| + return retainAutomaticLoss(c, box, ref, observedAt, *d, reassigned) |
| } |
| // retainAutomaticLoss is D4's retention, written once per (operation, box, |
| // fence): a replay finds its record and changes nothing. |
| -func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, d LocalDischarge) error { |
| +func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, d LocalDischarge, reassigned []ObligationRef) error { |
| id := automaticLossID(c.Op.ID, box, ref) |
| for _, l := range c.EvidenceRefs { |
| if l.ID == id { |
| @@ -1426,7 +1565,7 @@ func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, |
| role = "source" |
| } |
| loss := AutomaticLoss{ID: id, Box: box, Enrollment: ref.Enrollment, Fence: ref, ObservedAt: observedAt, |
| - Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: role, Discharge: d, Timers: TimersUnknown} |
| + Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: role, Discharge: d, Reassigned: reassigned, Timers: TimersUnknown} |
| if h := c.Holder; h != nil { |
| loss.Incarnation, loss.Stamp = h.Incarnation, h.Stamp |
| // The box's last writer session: the live one when it is still |
| @@ -1780,10 +1919,8 @@ func ToInstalling(opID string, t Target, prepared string) func(*Claim) error { |
| if t.Box != c.Op.Intent["target_box"] || t.Enrollment != c.Op.Intent["target_enrollment"] || slices.Contains(c.Lost, t.Box) { |
| return fmt.Errorf("%w: not the operation's target", ErrRelocation) |
| } |
| - for _, o := range c.Obligations { |
| - if o.Kind == "park-local" && o.Box == t.Box { |
| - return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) |
| - } |
| + if OwesLocal(c, t.Box) { |
| + return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) |
| } |
| if err := requireAcks(c); err != nil { |
| return err |
| @@ -2005,6 +2142,12 @@ func replaceTargetSlot(c *Claim, t Target) error { |
| if slices.Contains(excluded, t.Box) { |
| return fmt.Errorf("%w: %s was replaced in this operation", ErrRelocation, t.Box) |
| } |
| + if OwesLocal(c, t.Box) { |
| + // Spec 1 §2: a box owing a local retirement for this host is never |
| + // handed an installation, in any phase a replacement resumes at |
| + // (install is called below without ToInstalling's check). |
| + return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) |
| + } |
| old := c.Op.Intent["target_box"] |
| if old != "" && old != t.Box { |
| n := journalUint(c.Op, "replacements") |
| @@ -2316,7 +2459,7 @@ Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the |
| - Modify: `internal/console/server.go` (`Config.RegionHA`), `cmd/console/main.go` (sets it from `CARLOS_REGION_HA` with `strconv.ParseBool`) |
| **Interfaces:** |
| -- Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`. |
| +- Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`; Task 1's `hostclaim.LocalChoreBoxes` (the eligible set excludes a box owing a `park-local`, rev 6). |
| - Produces: `hostclaim.ExecutorSelection{Box, Enrollment}`, `hostclaim.BeginCreateOn(nh, body, opID, by, now, sel)`; `hostops.ErrNoEligibleBox`; `hostops.ExecutorChooser{Directory, Edges func(ctx) (boxchoice.Snapshot, error), Loads func(ctx) (map[string]boxchoice.Load, error), SecretsKeyFP, HomeName, Now}` with `Choose(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)` and `Permits(ctx, hostclaim.Observed, Record, box string) error`; `hostops.ClaimLoads(ctx, release.Store) (map[string]boxchoice.Load, error)`; `Authority.SelectExecutor(ctx, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error)`; `ProductionAuthority.Executors *ExecutorChooser`; `console.Config.RegionHA`; `(*console.Server).executorChooser(hostactors.Directory) *hostops.ExecutorChooser` (nil with HA off). |
| - [ ] **Step 1: Write the failing tests** |
| @@ -2451,6 +2594,20 @@ func TestExecutorChooserNeverNamesAFencedLostOrUnknownBox(t *testing.T) { |
| t.Fatalf("chose a lost box: %v", err) |
| } |
| }) |
| + t.Run("owing a local retirement", func(t *testing.T) { |
| + // Spec 1 §2: a box whose old installation of the host is not yet |
| + // retired (park-local) is never chosen; the clean one is. |
| + x, _, _, _ := chooserFixture(t, "box-a", "box-b") |
| + obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Obligations: []hostclaim.Obligation{{ID: "old", Op: "old-op", Kind: "park-local", Box: "box-a", Retirement: "r", Owner: "box:box-a"}}}} |
| + for range 4 { |
| + if got, err := x.Choose(ctx, obs, chooserRecord()); err != nil || got.Box != "box-b" { |
| + t.Fatalf("got %+v %v, want the box with no chore", got, err) |
| + } |
| + } |
| + if err := x.Permits(ctx, obs, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { |
| + t.Fatalf("permitted a box owing park-local: %v", err) |
| + } |
| + }) |
| t.Run("fenced", func(t *testing.T) { |
| x, s, _, _ := chooserFixture(t, "box-a", "box-w") |
| f := termfence.Record{V: 1, Box: "box-a", Enrollment: strings.Repeat("a", 32), State: "terminated", |
| @@ -2710,7 +2867,8 @@ type ExecutorChooser struct { |
| // Choose returns the box, and its roster enrollment, that a new host in |
| // r's placement is created on. Identity, placement and AWS region come from |
| // the roster; a box with a valid termination fence, or not active, is never |
| -// chosen; a box on the claim's lost list is never chosen. |
| +// chosen; a box on the claim's lost list, or owing a park-local chore on |
| +// it, is never chosen. |
| func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) { |
| o, cands, err := x.eligible(ctx, obs, r) |
| if err != nil { |
| @@ -2777,13 +2935,19 @@ func (x ExecutorChooser) eligible(ctx context.Context, obs hostclaim.Observed, r |
| if err != nil { |
| return hostactors.Observed{}, nil, err |
| } |
| - lost := map[string]bool{} |
| + // Never a box on the claim's lost list, nor one that still owes a |
| + // park-local chore for this host (spec 1 §2: its old installation is |
| + // not yet retired). |
| + excluded := map[string]bool{} |
| if !obs.Absent && obs.Claim != nil { |
| for _, box := range obs.Claim.Lost { |
| - lost[box] = true |
| + excluded[box] = true |
| + } |
| + for _, box := range hostclaim.LocalChoreBoxes(obs.Claim) { |
| + excluded[box] = true |
| } |
| } |
| - return o, boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, lost), nil |
| + return o, boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, excluded), nil |
| } |
| ``` |
| @@ -4811,6 +4975,25 @@ func TestWorkerMarksAnOverdueReleaseStuck(t *testing.T) { |
| t.Fatalf("a source that never released must leave the operation stuck on the source: %+v", o.Claim.Op) |
| } |
| } |
| + |
| +// Round 5 finding 1: a spare that owes an older local retirement for the |
| +// host is never offered to the chooser; the worker replaces the dead |
| +// target with a clean spare (the claim refuses the other in its CAS). |
| +func TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement(t *testing.T) { |
| + f := newFixture(t) |
| + if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.participant(func(c *hostclaim.Claim) error { |
| + c.Obligations = append(c.Obligations, hostclaim.Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: "spa", Retirement: "r", Owner: "box:spa"}) |
| + return nil |
| + }) |
| + f.terminate("dst") |
| + o := f.worker(nil) |
| + if o.Claim.Op.Intent["target_box"] != "sp2" || slices.Contains(o.Claim.Op.Participants, "spa") { |
| + t.Fatalf("replacement: %+v participants %v", o.Claim.Op.Intent, o.Claim.Op.Participants) |
| + } |
| +} |
| ``` |
| `internal/hostrelocate/pin_test.go`: |
| @@ -5589,10 +5772,12 @@ func (w Worker) releaseOverdue(ctx context.Context, obs hostclaim.Observed, sinc |
| } |
| // next names the operation's next target: never the source, a lost box, |
| -// or a box this operation already replaced. |
| +// a box this operation already replaced, or a box that owes a park-local |
| +// chore for the host (spec 1 §2; ReplaceTarget refuses one in its CAS). |
| func (w Worker) next(ctx context.Context, obs hostclaim.Observed) (hostclaim.Target, error) { |
| c := obs.Claim |
| exclude := append([]string{c.Op.Intent["source_box"]}, c.Lost...) |
| + exclude = append(exclude, hostclaim.LocalChoreBoxes(c)...) |
| if t := c.Op.Intent["target_box"]; t != "" { |
| exclude = append(exclude, t) |
| } |
| @@ -5945,7 +6130,7 @@ func CopyPinnedManifest(ctx context.Context, claims, data release.Store, pin Fil |
| - [ ] **Step 5: Run the tests** |
| Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1 -v 2>&1 | grep -E '^(---|ok|FAIL)'` |
| -Expected: PASS for all twenty-one tests. |
| +Expected: PASS for all twenty-two tests. |
| - [ ] **Step 6: Commit** |
| @@ -5968,7 +6153,7 @@ A participant box acknowledges first (D6, after §5.10's pending-state and updat |
| **Interfaces:** |
| - Consumes: Tasks 1 and 5 (the worker fixture in `worker_test.go`). |
| -- Produces: `hostrelocate.Effects` (`Acknowledge`, `Ready`, `Release`, `Build`, `InstallLocal`, `Probe`, `Retained`, `Finish`), `hostrelocate.Participation{Host, Op}`, `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)` and `Sweep(ctx) error`. |
| +- Produces: `hostrelocate.Effects` (`Acknowledge`, `Ready`, `Release`, `Build`, `InstallLocal`, `Probe`, `Retained() ([]Participation, error)`, `Finish`), `hostrelocate.Participation{Host, Op}`, `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)` and `Sweep(ctx) error`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -6003,6 +6188,8 @@ type fakeBox struct { |
| finished bool |
| // failProbes makes every probe admission fail readiness. |
| failProbes bool |
| + // retainedErr is a damaged journal entry Retained reports. |
| + retainedErr error |
| } |
| func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) { |
| @@ -6010,11 +6197,11 @@ func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclai |
| b.restarted, b.finished, b.op = false, false, o.Claim.Op.ID |
| return hostclaim.ExecutionAcknowledgement{Op: o.Claim.Op.ID, Enrollment: enrollment(b.box), BootID: "boot", ControllerID: "ctl-" + b.box, LeaseETag: "lease", Expires: b.f.now.Add(15 * 60e9)}, nil |
| } |
| -func (b *fakeBox) Retained() []Participation { |
| +func (b *fakeBox) Retained() ([]Participation, error) { |
| if b.opened == 0 || b.finished { |
| - return nil |
| + return nil, b.retainedErr |
| } |
| - return []Participation{{Host: host, Op: b.op}} |
| + return []Participation{{Host: host, Op: b.op}}, b.retainedErr |
| } |
| func (b *fakeBox) Finish(_ context.Context, h, op string) error { |
| b.finished = true |
| @@ -6206,9 +6393,13 @@ func TestAParticipantFinishesItsControllerOnceTheOperationIsOver(t *testing.T) { |
| if err := pd.Sweep(ctx); err != nil || !dst.finished { |
| t.Fatalf("the serving target kept its relocation controller: %v finished=%v", err, dst.finished) |
| } |
| - if err := ps.Sweep(ctx); err != nil || !src.finished { |
| - t.Fatalf("the former source kept its controller: %v", err) |
| + // A damaged journal entry beside it is reported, and the readable |
| + // participation is still finished. |
| + src.retainedErr = errors.New("a damaged journal entry") |
| + if err := ps.Sweep(ctx); !errors.Is(err, src.retainedErr) || !src.finished { |
| + t.Fatalf("the former source kept its controller, or the damaged entry was hidden: %v finished=%v", err, src.finished) |
| } |
| + src.retainedErr = nil |
| // A second relocation opens a fresh controller. |
| spa := &fakeBox{f: f, box: "spa"} |
| if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "dst", Target: "spa"}); err != nil { |
| @@ -6295,8 +6486,10 @@ type Effects interface { |
| // A controller is finished once the claim no longer runs that |
| // operation with this box as a participant (Sweep), so the next |
| // relocation of the host can open its own (hostexec keeps the lock |
| - // until Close). |
| - Retained() []Participation |
| + // until Close). Retained also lists every participation journaled on |
| + // disk whose durable record a restart still has to clear; an entry it |
| + // cannot read is the error, returned beside every one it can. |
| + Retained() ([]Participation, error) |
| Finish(ctx context.Context, host, op string) error |
| } |
| @@ -6403,8 +6596,9 @@ func (p Participant) Sweep(ctx context.Context) error { |
| if p.Claims == nil || p.Effects == nil { |
| return ErrUnsupported |
| } |
| - var errs []error |
| - for _, r := range p.Effects.Retained() { |
| + retained, err := p.Effects.Retained() |
| + errs := []error{err} // reported, while every readable participation is still finished |
| + for _, r := range retained { |
| obs, err := hostclaim.Read(ctx, p.Claims, r.Host) |
| if err != nil { |
| errs = append(errs, err) |
| @@ -6473,15 +6667,15 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 7: Release evidence on the box (`internal/host`) |
| -`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. |
| +`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). |
| **Files:** |
| - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` |
| - Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationcontrollers_linux_test.go` |
| **Interfaces:** |
| -- Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `registry.Registry`. |
| -- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `host.RelocationController` (`Ready`, `Close`, `Clear`), `host.OpenRelocationController(ctx, execctl.Config)` (hostexec's controller with `ClearStale` as its `Clear`), `RelocationControllers{Journal, Clear}` with the finishing journal; `hostfiles.EnvItem`. `StopRequest.Region` is 6b's field: Task 8a's `Collect` sets it on every request. |
| +- Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `ReadPlatform`, `registry.Registry`, `hostexec.ClearStale`. |
| +- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `host.RelocationController` (`Ready`, `Close`, `Clear`), `host.OpenRelocationController(ctx, execctl.Config)` (hostexec's controller with `ClearStale` as its `Clear`), `RelocationControllers{Journal, Cleanup, Clear}` with the participation journal (written before each open) and `Retained() ([]hostrelocate.Participation, error)`; `hostfiles.EnvItem`, `(*hostfiles.Publisher).StagePlatformDelete(name)`. `StopRequest.Region` is 6b's field: Task 8a's `Collect` sets it on every request. |
| Note: `internal/host`'s tests declare a function named `hostexec`, so this file imports the package as `execctl`. |
| @@ -6601,6 +6795,7 @@ type readyController struct { |
| cleared bool |
| lost error |
| clearErr error |
| + closeErr error |
| } |
| func (c *readyController) Clear(context.Context) error { |
| @@ -6621,8 +6816,25 @@ func (c *readyController) Close(ctx context.Context) error { |
| if err := ctx.Err(); err != nil { |
| return err // hostexec retains the lifetime lock on a cut-short join |
| } |
| + if c.closed { |
| + return nil // hostexec: every Close after the first returns nil |
| + } |
| c.closed = true |
| - return nil |
| + return c.closeErr // hostexec: a cleanup failure is returned once |
| +} |
| + |
| +// testControllers is a RelocationControllers with a journal on disk and a |
| +// scoped cleanup that succeeds. |
| +func testControllers(t *testing.T) *RelocationControllers { |
| + return &RelocationControllers{Journal: filepath.Join(t.TempDir(), "participations"), Cleanup: func(context.Context, execctl.Binding) error { return nil }} |
| +} |
| + |
| +// kept is Retained's list, for tests whose journal holds nothing damaged. |
| +func kept(r interface { |
| + Retained() ([]hostrelocate.Participation, error) |
| +}) []hostrelocate.Participation { |
| + got, _ := r.Retained() |
| + return got |
| } |
| type memPositions struct { |
| @@ -6653,7 +6865,7 @@ func effectsFixture(t *testing.T) (*RelocationEffects, hostclaim.Observed, *[]st |
| exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) |
| e := &RelocationEffects{ |
| Lifetime: context.Background(), |
| - Controllers: &RelocationControllers{}, |
| + Controllers: testControllers(t), |
| Identity: func(context.Context) (RelocationIdentity, error) { |
| return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil |
| }, |
| @@ -6906,7 +7118,7 @@ func TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime(t *testi |
| // An agent restart has no retained controller: Ready says so, and |
| // Acknowledge opens a fresh one. |
| restarted := *e |
| - restarted.Controllers = &RelocationControllers{} |
| + restarted.Controllers = testControllers(t) |
| if err := restarted.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { |
| t.Fatalf("after a restart: %v", err) |
| } |
| @@ -6915,18 +7127,18 @@ func TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime(t *testi |
| } |
| // Finishing a participation closes its controller and forgets it; a |
| // later Acknowledge for the same host opens a new one. |
| - if got := restarted.Retained(); len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { |
| + if got := kept(&restarted); len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { |
| t.Fatalf("retained: %+v", got) |
| } |
| // A close cut short by the pass's context keeps the entry, so a later |
| // sweep can finish it; then it is forgotten. |
| cut, cancel := context.WithCancel(context.Background()) |
| cancel() |
| - if err := restarted.Finish(cut, "shop.example", "rel-1"); err == nil || opened[1].closed || len(restarted.Retained()) != 1 { |
| - t.Fatalf("a cut-short close forgot the controller: %v closed=%v retained=%v", err, opened[1].closed, restarted.Retained()) |
| + if err := restarted.Finish(cut, "shop.example", "rel-1"); err == nil || opened[1].closed || len(kept(&restarted)) != 1 { |
| + t.Fatalf("a cut-short close forgot the controller: %v closed=%v retained=%v", err, opened[1].closed, kept(&restarted)) |
| } |
| - if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[1].closed || !opened[1].cleared || len(restarted.Retained()) != 0 { |
| - t.Fatalf("finish: %v closed=%v cleared=%v retained=%v", err, opened[1].closed, opened[1].cleared, restarted.Retained()) |
| + if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[1].closed || !opened[1].cleared || len(kept(&restarted)) != 0 { |
| + t.Fatalf("finish: %v closed=%v cleared=%v retained=%v", err, opened[1].closed, opened[1].cleared, kept(&restarted)) |
| } |
| if err := restarted.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { |
| t.Fatalf("after finish: %v", err) |
| @@ -7000,6 +7212,25 @@ func TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv(t *testing.T) { |
| if err != nil || again != seq { |
| t.Fatalf("repeat: %d %v", again, err) |
| } |
| + // The env file is removed before a later release: the flush stages its |
| + // deletion, so the final state carries no env, not the old KEY=v. |
| + if err := os.Remove(filepath.Join(envDir, host+".env")); err != nil { |
| + t.Fatal(err) |
| + } |
| + gone, err := f.Flush(ctx, obs) |
| + if err != nil || gone <= seq { |
| + t.Fatalf("flush after the env file was removed: %d %v", gone, err) |
| + } |
| + m, ok, err = hostfiles.Load(ctx, data, prefix) |
| + if err != nil || !ok || m.Seq != gone { |
| + t.Fatalf("manifest after removal: %+v %v %v", m, ok, err) |
| + } |
| + if env, err := hostfiles.ReadPlatform(ctx, data, prefix, m, hostfiles.EnvItem, nil); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatalf("a removed env file was still published: %q %v", env, err) |
| + } |
| + if again, err := f.Flush(ctx, obs); err != nil || again != gone { |
| + t.Fatalf("repeat after removal: %d %v", again, err) |
| + } |
| var raw map[string]any |
| body, _, _ := data.Get(ctx, hostfiles.ManifestKey(prefix)) |
| _ = json.Unmarshal(body, &raw) |
| @@ -7009,43 +7240,117 @@ func TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv(t *testing.T) { |
| } |
| func TestFinishClearsTheDurableRecordAndRetriesAcrossARestart(t *testing.T) { |
| - journal := filepath.Join(t.TempDir(), "finishing") |
| + journal := filepath.Join(t.TempDir(), "participations") |
| e, obs, _, _ := effectsFixture(t) |
| - e.Controllers = &RelocationControllers{Journal: journal} |
| + cleanups := 0 |
| + var cleanupErr error |
| + cleanup := func(_ context.Context, b execctl.Binding) error { |
| + if b.Op != "rel-1" || b.Box != "src" || b.Host != "shop.example" || b.Incarnation != "inc" { |
| + t.Fatalf("cleanup for another binding: %+v", b) |
| + } |
| + cleanups++ |
| + return cleanupErr |
| + } |
| + e.Controllers = &RelocationControllers{Journal: journal, Cleanup: cleanup} |
| var ctl *readyController |
| exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) |
| e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { |
| ctl = &readyController{opened: ctx, st: execctl.State{Binding: b, LeaseETag: "lease-1", Expires: exp}} |
| return ctl, nil |
| } |
| + // Without a journal nothing opens: a record no restart could find. |
| + unjournaled := *e |
| + unjournaled.Controllers = &RelocationControllers{Cleanup: cleanup} |
| + if _, err := unjournaled.Acknowledge(context.Background(), obs); !errors.Is(err, ErrClaimStopUnavailable) || ctl != nil { |
| + t.Fatalf("a controller opened without a journal: %v", err) |
| + } |
| if _, err := e.Acknowledge(context.Background(), obs); err != nil { |
| t.Fatal(err) |
| } |
| - // The clear fails: the controller is closed, the entry and the |
| - // journal stay, so it is retried. |
| + // The controller's cleanup fails. hostexec's Close reports it once. |
| + ctl.closeErr = errors.New("cleanup: a process would not stop") |
| + cleanupErr = ctl.closeErr |
| + if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || !ctl.closed || ctl.cleared || len(kept(e)) != 1 { |
| + t.Fatalf("a failed cleanup: %v closed=%v cleared=%v retained=%v", err, ctl.closed, ctl.cleared, kept(e)) |
| + } |
| + // The next sweep: Close now returns nil, which proves nothing. The |
| + // repeated cleanup still fails, so the record stays. |
| + if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || ctl.cleared || len(kept(e)) != 1 { |
| + t.Fatalf("a nil Close after a failed cleanup cleared the record: %v cleared=%v retained=%v", err, ctl.cleared, kept(e)) |
| + } |
| + // The clear itself fails: the entry and the journal stay. |
| + cleanupErr = nil |
| ctl.clearErr = errors.New("claim read failed") |
| - if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || !ctl.closed || len(e.Retained()) != 1 { |
| - t.Fatalf("a failed clear: %v closed=%v retained=%v", err, ctl.closed, e.Retained()) |
| + if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || ctl.cleared || len(kept(e)) != 1 { |
| + t.Fatalf("a failed clear: %v cleared=%v retained=%v", err, ctl.cleared, kept(e)) |
| } |
| // The agent restarts: no controller object, the journal remains. The |
| - // clear runs through the recorded binding, then the journal is gone. |
| + // cleanup is repeated first; while it fails nothing is cleared. |
| cleared := []execctl.Binding{} |
| - again := &RelocationControllers{Journal: journal, Clear: func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil }} |
| - if got := again.Retained(); len(got) != 1 || got[0].Op != "rel-1" { |
| + clear := func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil } |
| + again := &RelocationControllers{Journal: journal, Cleanup: cleanup, Clear: clear} |
| + if got := kept(again); len(got) != 1 || got[0].Op != "rel-1" { |
| t.Fatalf("after a restart the journaled participation is forgotten: %+v", got) |
| } |
| - if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || len(cleared) != 1 || cleared[0].Op != "rel-1" || cleared[0].Box != "src" { |
| - t.Fatalf("retry after restart: %v cleared=%+v", err, cleared) |
| + cleanupErr = errors.New("cleanup: still failing") |
| + if err := again.Finish(context.Background(), "shop.example", "rel-1"); err == nil || len(cleared) != 0 || len(kept(again)) != 1 { |
| + t.Fatalf("a restart cleared without a successful cleanup: %v cleared=%+v", err, cleared) |
| } |
| - if got := again.Retained(); len(got) != 0 { |
| + cleanupErr = nil |
| + before := cleanups |
| + if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || cleanups != before+1 || len(cleared) != 1 || cleared[0].Op != "rel-1" || cleared[0].Box != "src" { |
| + t.Fatalf("retry after restart: %v cleanups=%d cleared=%+v", err, cleanups-before, cleared) |
| + } |
| + if got := kept(again); len(got) != 0 { |
| t.Fatalf("the journal entry outlived the clear: %+v", got) |
| } |
| } |
| + |
| +// A crash before the first Finish (or a failed Open after hostexec wrote |
| +// its record) leaves no controller and no finishing state, only the |
| +// participation journaled before the open: a restart finds it, and an |
| +// uncommitted, truncated journal write beside it hides nothing. |
| +func TestARestartFindsAParticipationThatNeverReachedFinish(t *testing.T) { |
| + journal := filepath.Join(t.TempDir(), "participations") |
| + e, obs, _, _ := effectsFixture(t) |
| + e.Controllers = &RelocationControllers{Journal: journal, Cleanup: func(context.Context, execctl.Binding) error { return nil }} |
| + if _, err := e.Acknowledge(context.Background(), obs); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The crash: the controller object is gone. Another participation's |
| + // journal write was cut short, leaving a truncated temporary file. |
| + if err := os.WriteFile(filepath.Join(journal, "spa.example~rel-9.json.tmp"), []byte(`{"v":1,"ho`), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + cleaned, cleared := []execctl.Binding{}, []execctl.Binding{} |
| + again := &RelocationControllers{Journal: journal, |
| + Cleanup: func(_ context.Context, b execctl.Binding) error { cleaned = append(cleaned, b); return nil }, |
| + Clear: func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil }} |
| + got, err := again.Retained() |
| + if err != nil || len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { |
| + t.Fatalf("restart with a truncated temporary file: %+v %v", got, err) |
| + } |
| + // A committed entry that cannot be read is reported, never hiding the |
| + // readable one. |
| + if err := os.WriteFile(filepath.Join(journal, "bad.example~rel-3.json"), []byte("{"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err = again.Retained() |
| + if !errors.Is(err, ErrClaimStopUnavailable) || len(got) != 1 || got[0].Host != "shop.example" { |
| + t.Fatalf("a damaged committed entry: %+v %v", got, err) |
| + } |
| + if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || len(cleaned) != 1 || len(cleared) != 1 || cleared[0].Host != "shop.example" || cleared[0].Incarnation != "inc" { |
| + t.Fatalf("finish after the crash: %v cleaned=%+v cleared=%+v", err, cleaned, cleared) |
| + } |
| + if got, _ := again.Retained(); len(got) != 0 { |
| + t.Fatalf("the participation outlived its clear: %+v", got) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| -Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|RelocationRelease|RelocationAcknowledge|ReleasePositions|ReleaseFlushFiles' -count=1` |
| +Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|RelocationRelease|RelocationAcknowledge|ReleasePositions|ReleaseFlushFiles|Finish|ARestartFinds' -count=1` |
| Expected: FAIL to compile: `undefined: StopReceiptReleases`, `undefined: RelocationEffects`. |
| - [ ] **Step 3: The receipt verifier** |
| @@ -7116,7 +7421,10 @@ func (a StopReceiptReleases) VerifyRelease(ctx context.Context, c hostclaim.Clai |
| ```go |
| package hostfiles |
| -import "fmt" |
| +import ( |
| + "fmt" |
| + "strings" |
| +) |
| // EnvItem is the host's env file (<host>.env) among the manifest's |
| // platform items. A release flush stages it beside the wake state so a |
| @@ -7125,6 +7433,26 @@ import "fmt" |
| // with ReadPlatform. |
| const EnvItem = "host.env" |
| +// StagePlatformDelete records that a platform item is gone: the next |
| +// Commit publishes a revision without it, at a new sequence, so a reader |
| +// of that revision gets os.ErrNotExist from ReadPlatform instead of the |
| +// last bytes it published. Deleting an item the manifest does not hold |
| +// is a no-op, like StageDelete. |
| +func (p *Publisher) StagePlatformDelete(name string) error { |
| + if name == "" || strings.ContainsAny(name, "/\x00") { |
| + return fmt.Errorf("hostfiles: platform item %q is not a plain name", name) |
| + } |
| + p.mu.Lock() |
| + defer p.mu.Unlock() |
| + if _, ok := p.man.Platform[name]; !ok { |
| + return nil |
| + } |
| + p.next++ |
| + delete(p.man.Platform, name) |
| + p.dirty = true |
| + return nil |
| +} |
| + |
| // ValidateManifest checks every entry and platform item of m against the |
| // admitted origins, as Restore and CopyManifest do before they touch |
| // anything (contracts note §1). A caller holding a manifest's exact bytes |
| @@ -7184,7 +7512,9 @@ type ReleaseFlushFiles struct { |
| } |
| // Flush publishes and returns the committed manifest sequence. The env |
| -// file is staged when it exists; a host with none publishes none. A |
| +// file is staged when it exists; when it does not, its deletion is staged |
| +// (StagePlatformDelete), so a host whose env file was removed after an |
| +// earlier flush publishes none rather than the old bytes. A |
| // manifest revision is committed even when nothing changed (Commit |
| // returns the current sequence), so the release's manifest_seq is always |
| // the sequence that holds this box's final state. |
| @@ -7213,7 +7543,13 @@ func (f ReleaseFlushFiles) Flush(ctx context.Context, obs hostclaim.Observed) (i |
| if err := p.StagePlatform(ctx, hostfiles.EnvItem, env); err != nil { |
| return 0, err |
| } |
| - case !errors.Is(err, os.ErrNotExist): |
| + case errors.Is(err, os.ErrNotExist): |
| + // The env file is gone: the release's final state has none, so a |
| + // previously published one is deleted, never carried forward. |
| + if err := p.StagePlatformDelete(hostfiles.EnvItem); err != nil { |
| + return 0, err |
| + } |
| + default: |
| return 0, err |
| } |
| st, err := hostfiles.SyncDir(ctx, p, f.FilesDir(obs.Host), f.Skip) |
| @@ -7355,7 +7691,10 @@ func (activeDirectory) Lookup(context.Context, string) (hostquarantine.Actor, er |
| // A real hostexec controller on disk: after the relocation completes, |
| // Finish closes it and clears its durable record, so root's §5.10 check |
| // (hostexecParticipation) lets an executable update through. While the |
| -// operation still runs, the clear refuses and the record stays. |
| +// operation still runs, the clear refuses and the record stays. A cleanup |
| +// that failed keeps the record however often Close is called afterwards |
| +// (hostexec returns the failure once), and across a restart, until a |
| +// repeated cleanup succeeds. |
| func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { |
| ctx := context.Background() |
| dataDir := t.TempDir() |
| @@ -7389,11 +7728,28 @@ func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { |
| Intent: map[string]string{"region": "ie", "source_box": "src", "target_box": "dst"}} |
| return nil |
| }) |
| + failing := false |
| + scoped := func(context.Context) error { |
| + if failing { |
| + return errors.New("cleanup: the partial install would not go") |
| + } |
| + return nil |
| + } |
| cfg := execctl.Config{Store: store, Guard: hostguard.Guard{DataDir: dataDir, LockDir: filepath.Join(dataDir, "locks"), Claims: hostguard.StoreReader{Store: store}}, |
| Directory: activeDirectory{}, Clock: execctl.SystemClock{}, LockDir: filepath.Join(dataDir, "exec-locks"), |
| - CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: func(context.Context) error { return nil }} |
| + CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: scoped} |
| + journal := filepath.Join(dataDir, "relocate", "participations") |
| + controllers := func() *RelocationControllers { |
| + return &RelocationControllers{Journal: journal, |
| + Cleanup: func(ctx context.Context, _ execctl.Binding) error { return scoped(ctx) }, |
| + Clear: func(ctx context.Context, b execctl.Binding) error { |
| + c := cfg |
| + c.Binding = b |
| + return execctl.ClearStale(ctx, c) |
| + }} |
| + } |
| e, _, _, _ := effectsFixture(t) |
| - e.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "finishing")} |
| + e.Controllers = controllers() |
| e.Identity = func(context.Context) (RelocationIdentity, error) { |
| return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil |
| } |
| @@ -7408,26 +7764,53 @@ func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { |
| if err := hostexecParticipation(dataDir); err == nil { |
| t.Fatal("a participant's record did not block an update") |
| } |
| + // The controller's cleanup fails: Close reports it, the record stays. |
| + failing = true |
| + if err := e.Finish(ctx, host, "rel-1"); err == nil { |
| + t.Fatal("a failed cleanup was not reported") |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { |
| + t.Fatalf("a failed cleanup cleared the record: %v retained=%v", err, kept(e)) |
| + } |
| // Still relocating: the clear refuses, the record and the entry stay. |
| + failing = false |
| if err := e.Finish(ctx, host, "rel-1"); !errors.Is(err, execctl.ErrInactive) { |
| t.Fatalf("finish during the operation: %v", err) |
| } |
| - if err := hostexecParticipation(dataDir); err == nil || len(e.Retained()) != 1 { |
| - t.Fatalf("the record was cleared while the operation ran: %v retained=%v", err, e.Retained()) |
| + if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { |
| + t.Fatalf("the record was cleared while the operation ran: %v retained=%v", err, kept(e)) |
| } |
| - // Served on the target: the retry clears it. |
| + // Served on the target. The cleanup fails again: hostexec's Close now |
| + // returns nil, and the next sweep must still not clear. |
| step(obs, func(c *hostclaim.Claim) error { |
| c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}} |
| return nil |
| }) |
| - if err := e.Finish(ctx, host, "rel-1"); err != nil { |
| + failing = true |
| + if err := e.Finish(ctx, host, "rel-1"); err == nil { |
| + t.Fatal("a sweep after a failed cleanup reported success") |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { |
| + t.Fatalf("a nil Close cleared the record: %v retained=%v", err, kept(e)) |
| + } |
| + // The agent restarts with the cleanup still failing: nothing clears. |
| + restarted := controllers() |
| + if err := restarted.Finish(ctx, host, "rel-1"); err == nil { |
| + t.Fatal("a restart cleared without a successful cleanup") |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil || len(kept(restarted)) != 1 { |
| + t.Fatalf("a restart cleared the record: %v retained=%v", err, kept(restarted)) |
| + } |
| + // The cleanup succeeds: the retry clears the record and the journal. |
| + failing = false |
| + if err := restarted.Finish(ctx, host, "rel-1"); err != nil { |
| t.Fatal(err) |
| } |
| if err := hostexecParticipation(dataDir); err != nil { |
| t.Fatalf("after the relocation an update is still refused: %v", err) |
| } |
| - if len(e.Retained()) != 0 { |
| - t.Fatalf("retained: %v", e.Retained()) |
| + if len(kept(restarted)) != 0 { |
| + t.Fatalf("retained: %v", kept(restarted)) |
| } |
| } |
| ``` |
| @@ -7474,10 +7857,13 @@ type RelocationController interface { |
| Clear(context.Context) error |
| } |
| -// finishing is one participation whose controller is closed or closing |
| -// and whose durable record is not yet cleared: the retry state Finish |
| -// keeps on disk (Journal) so a restart retries the clear. |
| -type finishing struct { |
| +// participation is one journaled participation whose controller may have |
| +// left a durable record: written before the controller is opened |
| +// (Acknowledge), removed only after the record is cleared (Finish). A |
| +// crash anywhere in between leaves the entry, so a restarted agent finds |
| +// every record it may have left, including one from a controller that |
| +// never reached its first Finish or whose Open failed after writing. |
| +type participation struct { |
| V int `json:"v"` |
| Host string `json:"host"` |
| Op string `json:"op"` |
| @@ -7492,13 +7878,27 @@ type finishing struct { |
| type RelocationControllers struct { |
| mu sync.Mutex |
| open map[string]RelocationController |
| - // Journal is where finishing participations are recorded before |
| - // their close (<DataDir>/relocate/finishing); Clear retries a journaled |
| - // one after a restart, when no controller object exists, through the |
| - // binding it recorded (hostexec.ClearStale needs only the config). |
| + // Journal is where participations are recorded before their |
| + // controller opens (<DataDir>/relocate/participations); Finish |
| + // retries a journaled one after a restart, when no controller object |
| + // exists, through the binding it recorded. Required: Acknowledge |
| + // opens nothing without it. |
| Journal string |
| - Clear func(context.Context, execctl.Binding) error |
| - // bindings remembers each open controller's binding for the journal. |
| + // Cleanup is the participation's scoped cleanup for one binding |
| + // (Task 8a's relocationCleanup: it retires only that operation's own |
| + // partial installation of that host and incarnation, after checking |
| + // the binding's box and enrollment are this box's verified actor, and |
| + // does nothing once this box has admitted a writer under the |
| + // operation or serves the host). Finish repeats it, and needs it to |
| + // succeed, in the same call as every clear: hostexec.Controller.Close |
| + // returns a cleanup failure once and nil on every later call, and a |
| + // restarted agent has no controller at all, so neither a nil Close |
| + // nor a journal entry proves the cleanup ran. Nil clears nothing. |
| + Cleanup func(context.Context, execctl.Binding) error |
| + // Clear removes a journaled binding's durable record after a restart |
| + // (hostexec.ClearStale with the assembly's config for the binding). |
| + Clear func(context.Context, execctl.Binding) error |
| + // bindings remembers each open controller's binding. |
| bindings map[string]execctl.Binding |
| } |
| @@ -7524,12 +7924,35 @@ func (r *RelocationControllers) put(host, op string, b execctl.Binding, c Reloca |
| return c, true |
| } |
| +const journalSuffix = ".json" |
| + |
| func (r *RelocationControllers) journalPath(host, op string) string { |
| - return filepath.Join(r.Journal, host+"~"+op+".json") |
| + return filepath.Join(r.Journal, host+"~"+op+journalSuffix) |
| +} |
| + |
| +// record journals a participation before its controller opens. |
| +func (r *RelocationControllers) record(host, op string, b execctl.Binding) error { |
| + if r.Journal == "" { |
| + return ErrClaimStopUnavailable |
| + } |
| + raw, err := json.Marshal(participation{V: 1, Host: host, Op: op, Binding: b}) |
| + if err != nil { |
| + return err |
| + } |
| + if err := os.MkdirAll(r.Journal, 0o700); err != nil { |
| + return err |
| + } |
| + return writeFileSynced(r.journalPath(host, op), raw, 0o600) |
| } |
| -// journaled lists the finishing participations on disk. |
| -func (r *RelocationControllers) journaled() ([]finishing, error) { |
| +// journaled lists the committed participations on disk. Only names |
| +// ending in .json are committed records: writeFileSynced's .json.tmp is |
| +// an uncommitted write a crash may have truncated, and is ignored (its |
| +// participation opened nothing, because the record is written before the |
| +// controller opens). A committed record that cannot be read is reported, |
| +// joined into the error, and every readable one is still returned, so |
| +// one damaged entry never hides the others' cleanup. |
| +func (r *RelocationControllers) journaled() ([]participation, error) { |
| if r.Journal == "" { |
| return nil, nil |
| } |
| @@ -7540,24 +7963,31 @@ func (r *RelocationControllers) journaled() ([]finishing, error) { |
| if err != nil { |
| return nil, err |
| } |
| - var out []finishing |
| + var out []participation |
| + var errs []error |
| for _, e := range entries { |
| + if e.IsDir() || !strings.HasSuffix(e.Name(), journalSuffix) { |
| + continue |
| + } |
| raw, err := os.ReadFile(filepath.Join(r.Journal, e.Name())) |
| if err != nil { |
| - return nil, err |
| + errs = append(errs, err) |
| + continue |
| } |
| - var f finishing |
| - if err := json.Unmarshal(raw, &f); err != nil || f.V != 1 { |
| - return nil, fmt.Errorf("%w: finishing journal %s", ErrClaimStopUnavailable, e.Name()) |
| + var f participation |
| + if err := json.Unmarshal(raw, &f); err != nil || f.V != 1 || f.Host == "" || f.Op == "" { |
| + errs = append(errs, fmt.Errorf("%w: participation journal %s", ErrClaimStopUnavailable, e.Name())) |
| + continue |
| } |
| out = append(out, f) |
| } |
| - return out, nil |
| + return out, errors.Join(errs...) |
| } |
| // Retained lists the participations whose controllers are open, and the |
| -// journaled ones whose durable record a restart still has to clear. |
| -func (r *RelocationControllers) Retained() []hostrelocate.Participation { |
| +// journaled ones whose durable record may still need clearing. A damaged |
| +// journal entry is returned as the error beside every readable one. |
| +func (r *RelocationControllers) Retained() ([]hostrelocate.Participation, error) { |
| r.mu.Lock() |
| seen := map[string]bool{} |
| var out []hostrelocate.Participation |
| @@ -7567,64 +7997,44 @@ func (r *RelocationControllers) Retained() []hostrelocate.Participation { |
| out = append(out, hostrelocate.Participation{Host: host, Op: op}) |
| } |
| r.mu.Unlock() |
| - journaled, _ := r.journaled() |
| + journaled, err := r.journaled() |
| for _, f := range journaled { |
| if !seen[controllerKey(f.Host, f.Op)] { |
| + seen[controllerKey(f.Host, f.Op)] = true |
| out = append(out, hostrelocate.Participation{Host: f.Host, Op: f.Op}) |
| } |
| } |
| sort.Slice(out, func(i, j int) bool { |
| return out[i].Host < out[j].Host || (out[i].Host == out[j].Host && out[i].Op < out[j].Op) |
| }) |
| - return out |
| + return out, err |
| } |
| // Finish closes and forgets one participation's controller, releasing the |
| -// host's lifetime lock for the next operation. Closing runs hostexec's |
| -// Cleanup hook: the relocation's controller is opened with a cleanup |
| -// scoped to the relocation's own partial work (Task 8a), never the serving |
| -// writer's session, so finishing after a successful relocation stops |
| -// nothing that serves. |
| +// host's lifetime lock for the next operation, and clears its durable |
| +// record. Closing runs hostexec's Cleanup hook: the relocation's |
| +// controller is opened with a cleanup scoped to the relocation's own |
| +// partial work (Task 8a), never the serving writer's session, so |
| +// finishing after a successful relocation stops nothing that serves. |
| // |
| -// The participation is journaled before its close, so a restart retries |
| -// what is left. The entry stays until Close succeeds: hostexec retains the |
| -// lifetime lock on a timed-out join and needs another Close, so a |
| -// controller whose close the pass's context cut short is found by the |
| -// next sweep. After the close, Clear removes the durable record (its |
| -// cleanup result, Close's error, is kept by refusing to clear a failed |
| -// one); the journal entry goes only once the record is gone, so a |
| -// restart in between retries the clear through the recorded binding. |
| +// The entry stays until Close succeeds: hostexec retains the lifetime |
| +// lock on a timed-out join and needs another Close, so a controller whose |
| +// close the pass's context cut short is found by the next sweep. Then the |
| +// scoped cleanup is repeated (Cleanup) and the record is cleared only if |
| +// that cleanup succeeded in this call: a nil Close after an earlier |
| +// failed one, or a restart with no controller, proves nothing. The |
| +// journal entry goes only once the record is gone, so a restart in |
| +// between retries the cleanup and the clear through the recorded binding. |
| func (r *RelocationControllers) Finish(ctx context.Context, host, op string) error { |
| key := controllerKey(host, op) |
| r.mu.Lock() |
| c, ok := r.open[key] |
| b := r.bindings[key] |
| r.mu.Unlock() |
| - if ok { |
| - if r.Journal != "" { |
| - raw, err := json.Marshal(finishing{V: 1, Host: host, Op: op, Binding: b}) |
| - if err != nil { |
| - return err |
| - } |
| - if err := os.MkdirAll(r.Journal, 0o700); err != nil { |
| - return err |
| - } |
| - if err := writeFileSynced(r.journalPath(host, op), raw, 0o600); err != nil { |
| - return err |
| - } |
| - } |
| - if err := c.Close(ctx); err != nil { |
| - return err // cleanup failed or was cut short: the record stays, and so does the entry |
| - } |
| - if err := c.Clear(ctx); err != nil { |
| - return err |
| - } |
| - } else { |
| - // No controller object: a restart. Clear through the journaled binding. |
| + if !ok { |
| + // No controller object: a restart, or a crash before the first |
| + // Finish. Retry through the journaled binding. |
| journaled, err := r.journaled() |
| - if err != nil { |
| - return err |
| - } |
| found := false |
| for _, f := range journaled { |
| if f.Host == host && f.Op == op { |
| @@ -7632,17 +8042,29 @@ func (r *RelocationControllers) Finish(ctx context.Context, host, op string) err |
| } |
| } |
| if !found { |
| - return nil |
| + return err // nil: nothing journaled; else the damaged entry may be this one |
| } |
| - if r.Clear == nil { |
| - return ErrClaimStopUnavailable |
| - } |
| - if err := r.Clear(ctx, b); err != nil { |
| - return err |
| + } |
| + if r.Cleanup == nil || (!ok && r.Clear == nil) { |
| + return ErrClaimStopUnavailable |
| + } |
| + if ok { |
| + if err := c.Close(ctx); err != nil { |
| + return err // cleanup failed or was cut short: the record stays, and so does the entry |
| } |
| } |
| + if err := r.Cleanup(ctx, b); err != nil { |
| + return err // the record stays: no clear without a cleanup that succeeded now |
| + } |
| + clear := func() error { return r.Clear(ctx, b) } |
| + if ok { |
| + clear = func() error { return c.Clear(ctx) } |
| + } |
| + if err := clear(); err != nil { |
| + return err |
| + } |
| r.mu.Lock() |
| - if r.open[key] == c { |
| + if ok && r.open[key] == c { |
| delete(r.open, key) |
| delete(r.bindings, key) |
| } |
| @@ -7655,6 +8077,8 @@ func (r *RelocationControllers) Finish(ctx context.Context, host, op string) err |
| return nil |
| } |
| +// writeFileSynced writes path whole: a synced temporary file renamed into |
| +// place, then the directory synced, so a committed entry survives a crash. |
| func writeFileSynced(path string, body []byte, mode os.FileMode) error { |
| tmp := path + ".tmp" |
| if err := os.WriteFile(tmp, body, mode); err != nil { |
| @@ -7669,7 +8093,15 @@ func writeFileSynced(path string, body []byte, mode os.FileMode) error { |
| return err |
| } |
| _ = f.Close() |
| - return os.Rename(tmp, path) |
| + if err := os.Rename(tmp, path); err != nil { |
| + return err |
| + } |
| + dir, err := os.Open(filepath.Dir(path)) |
| + if err != nil { |
| + return err |
| + } |
| + defer dir.Close() |
| + return dir.Sync() |
| } |
| // Close closes every retained controller (agent shutdown). |
| @@ -7772,6 +8204,12 @@ func (e *RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Obser |
| return hostclaim.ExecutionAcknowledgement{}, err |
| } |
| b := execctl.Binding{Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, Host: c.Host, Incarnation: c.Holder.Incarnation, Op: c.Op.ID, BootID: id.BootID} |
| + // Journaled before the open: hostexec writes its durable record |
| + // inside Open, so a crash or a failed Open leaves a record only |
| + // the journal lets a restarted agent find and clear. |
| + if err := e.Controllers.record(c.Host, c.Op.ID, b); err != nil { |
| + return hostclaim.ExecutionAcknowledgement{}, err |
| + } |
| opened, err := e.Controller(e.Lifetime, b) |
| if err != nil { |
| return hostclaim.ExecutionAcknowledgement{}, err |
| @@ -7906,9 +8344,9 @@ func (d *releaseDrain) DrainClaim(ctx context.Context, r StopRequest, fence stri |
| return ack, nil |
| } |
| -func (e *RelocationEffects) Retained() []hostrelocate.Participation { |
| +func (e *RelocationEffects) Retained() ([]hostrelocate.Participation, error) { |
| if e.Controllers == nil { |
| - return nil |
| + return nil, nil |
| } |
| return e.Controllers.Retained() |
| } |
| @@ -7982,7 +8420,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 9a: Reconcile's row decision on main (stage A) |
| -The half of Task 9 that compiles on main: `Reconciler.check` decides a relocatable host's row with `RelocationRow` on a fresh read, the fence observer is consulted inside the already-locked operation (never reacquiring the host lock: round 4 finding 1), its errors keep rows and mint none (C6), and a held row whose box is not the executor, or whose admission fence is closed, takes the hold-only path under the flock (`WithLocal`, `HoldRow`) where the guarded path would refuse it (`ErrFenced`). 6b's adapter supplies `HoldRow` and the registry's conditional hold write (Task 9, stage B); with `FenceClosed` and `HoldRow` nil the reconciler is today's. |
| +The half of Task 9 that compiles on main: `Reconciler.check` decides a relocatable host's row with `RelocationRow` on a fresh read, the fence observer is consulted inside the already-locked operation (never reacquiring the host lock: round 4 finding 1), its errors keep rows and mint none (C6), and a held row whose box is not the executor, or whose admission fence is closed, takes the hold-only path under the flock (`WithLocal`, `HoldRow`) where the guarded path would refuse it (`ErrFenced`). 6b's adapter supplies `HoldRow` and the registry's conditional hold write (Task 9, stage B). With `FenceClosed` and `HoldRow` both nil (region HA off) the reconciler is today's to the letter: the first check is main's `Serving`, so an executor of any other operation is refused before the guard is activated or inventory recorded, and `Install` never takes the row decision or the hold-only path (rev 6, round 5 finding 3). |
| **Files:** |
| - Modify: `internal/hostbox/reconcile.go` |
| @@ -7990,7 +8428,7 @@ The half of Task 9 that compiles on main: `Reconciler.check` decides a relocatab |
| **Interfaces:** |
| - Consumes: Task 3's `RelocationRow`, `RelocationPolicy`, `hostguard.Guard.AdmissionFenced`, `Guard.WithLocal`. |
| -- Produces: `Reconciler.FenceClosed func(ctx, hostclaim.Observed) (bool, error)`, `Reconciler.HoldRow func(ctx, host, holdOp string) error`, `(Reconciler) rowDecision(ctx, obs) (RowDecision, error)`, `(Reconciler) holdOnly(ctx, selected, RowDecision) error`. |
| +- Produces: `Reconciler.FenceClosed func(ctx, hostclaim.Observed) (bool, error)`, `Reconciler.HoldRow func(ctx, host, holdOp string) error`, `(Reconciler) relocationAware() bool`, `(Reconciler) rowDecision(ctx, obs) (RowDecision, error)`, `(Reconciler) holdOnly(ctx, selected, RowDecision) error`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -8004,6 +8442,10 @@ package hostbox |
| import ( |
| "context" |
| "errors" |
| + "io/fs" |
| + "maps" |
| + "os" |
| + "path/filepath" |
| "testing" |
| "time" |
| @@ -8112,14 +8554,78 @@ func TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver(t *testi |
| } |
| // With no observer (region HA off) the reconciler is today's: the serving |
| -// executor is ready, a non-executor is refused. |
| +// executor is ready; an executor of any other operation is refused by the |
| +// first check, before the guard is activated or inventory recorded, dry |
| +// run or not (round 5 finding 3). |
| func TestReconcileWithoutAFenceObserverIsTodays(t *testing.T) { |
| - r, selected, _, _ := relocationReconciler(t) |
| + r, selected, _, guard := relocationReconciler(t) |
| r.FenceClosed, r.HoldRow = nil, nil |
| + ctx := context.Background() |
| writes := 0 |
| - if err := r.Install(context.Background(), selected, func() error { writes++; return nil }); err != nil || writes != 1 { |
| + if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil || writes != 1 { |
| t.Fatalf("serving: %v writes %d", err, writes) |
| } |
| + moving := func() hostclaim.Observed { |
| + obs, err := hostclaim.Read(ctx, r.Claims, selected.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Update(ctx, r.Claims, obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindMoving, ID: "mv-1", Phase: "copying", Exec: "box", Participants: []string{"box", "dst"}, Intent: map[string]string{"admission_id": "adm"}} |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return obs |
| + } |
| + for _, phase := range []string{hostclaim.PhaseStarting, hostclaim.PhaseReleasing, "moving"} { |
| + var obs hostclaim.Observed |
| + if phase == "moving" { |
| + obs = moving() // a held executor participant with HA on (RowHeld) |
| + } else { |
| + obs = relocate(t, r, selected.Host, phase) // ready with HA on (RowReady) |
| + } |
| + before := snapshotDir(t, guard.DataDir) |
| + for _, dry := range []bool{true, false} { |
| + sel := selected |
| + sel.DryRun = dry |
| + if err := r.Install(ctx, sel, func() error { writes++; return nil }); !errors.Is(err, ErrChanged) || writes != 1 { |
| + t.Fatalf("%s (dry run %v) with HA off: %v writes %d", phase, dry, err, writes) |
| + } |
| + } |
| + after, err := hostclaim.Read(ctx, r.Claims, selected.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if after.ETag != obs.ETag { |
| + t.Fatalf("%s: the claim (inventory) was written with HA off", phase) |
| + } |
| + if got := snapshotDir(t, guard.DataDir); !maps.Equal(got, before) { |
| + t.Fatalf("%s: the guard's local state changed with HA off", phase) |
| + } |
| + } |
| +} |
| + |
| +// snapshotDir is every regular file under dir, by relative path. |
| +func snapshotDir(t *testing.T, dir string) map[string]string { |
| + t.Helper() |
| + out := map[string]string{} |
| + err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error { |
| + if err != nil || !d.Type().IsRegular() { |
| + return err |
| + } |
| + raw, err := os.ReadFile(path) |
| + if err != nil { |
| + return err |
| + } |
| + out[path] = string(raw) |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return out |
| } |
| ``` |
| @@ -8134,10 +8640,10 @@ Apply to `internal/hostbox/reconcile.go` (`git apply`): |
| ```diff |
| diff --git a/internal/hostbox/reconcile.go b/internal/hostbox/reconcile.go |
| -index 7cf5d4a5..7fc95731 100644 |
| +index 7cf5d4a5..cacefbc0 100644 |
| --- a/internal/hostbox/reconcile.go |
| +++ b/internal/hostbox/reconcile.go |
| -@@ -30,7 +30,33 @@ type Reconciler struct { |
| +@@ -30,7 +30,38 @@ type Reconciler struct { |
| CheckLocal func(context.Context, hostclaim.Observed, RecordSelection) error |
| // Adopt is Plan 5's #232 backfill coordinator, never a second adoption rule. |
| Adopt func(context.Context, RecordSelection) (hostclaim.Observed, error) |
| @@ -8152,6 +8658,11 @@ index 7cf5d4a5..7fc95731 100644 |
| + HoldRow func(ctx context.Context, host, holdOp string) error |
| } |
| + |
| ++// relocationAware is region HA on: Task 10 sets FenceClosed and HoldRow |
| ++// only with CARLOS_REGION_HA. With neither, reconcile is today's: the |
| ++// serving executor alone (Serving), and no hold-only path. |
| ++func (r Reconciler) relocationAware() bool { return r.FenceClosed != nil || r.HoldRow != nil } |
| ++ |
| +func (r Reconciler) fenceClosed(ctx context.Context, obs hostclaim.Observed) (bool, error) { |
| + if r.FenceClosed == nil { |
| + return false, nil |
| @@ -8171,43 +8682,52 @@ index 7cf5d4a5..7fc95731 100644 |
| type activationReader struct { |
| r Reconciler |
| selected RecordSelection |
| -@@ -57,9 +83,15 @@ func (r Reconciler) check(ctx context.Context, obs hostclaim.Observed, selected |
| +@@ -57,8 +88,20 @@ func (r Reconciler) check(ctx context.Context, obs hostclaim.Observed, selected |
| if obs.Host != selected.Host || h.Key != selected.Key || h.Incarnation != selected.Incarnation || h.Family != selected.Family || h.Stamp == 0 { |
| return ErrChanged |
| } |
| - if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { |
| -+ d, err := r.rowDecision(ctx, obs) |
| -+ if err != nil { |
| - return err |
| +- return err |
| ++ if !r.relocationAware() { |
| ++ if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { |
| ++ return err |
| ++ } |
| ++ } else { |
| ++ d, err := r.rowDecision(ctx, obs) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ switch d.Case { |
| ++ case RowReady, RowHeld: |
| ++ default: |
| ++ return ErrChanged |
| ++ } |
| } |
| -+ switch d.Case { |
| -+ case RowReady, RowHeld: |
| -+ default: |
| -+ return ErrChanged |
| -+ } |
| if until, ok := obs.Claim.QuietAfter[r.Region]; ok && !r.Now().After(until) { |
| return ErrChanged |
| - } |
| -@@ -100,6 +132,18 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| +@@ -100,6 +143,21 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| if selected.DryRun { |
| return nil |
| } |
| -+ d, err := r.rowDecision(ctx, obs) |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ closed, _ := r.fenceClosed(ctx, obs) |
| -+ if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || closed) { |
| -+ // No installation of ours to activate, or one whose admission |
| -+ // fence is closed (the source in releasing, still op.exec, which |
| -+ // With refuses): a kept row's hold only, under the flock. A target |
| -+ // before installing has no row and mints none (ruling 17). |
| -+ return r.holdOnly(ctx, selected, d) |
| ++ if r.relocationAware() { |
| ++ d, err := r.rowDecision(ctx, obs) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ closed, _ := r.fenceClosed(ctx, obs) |
| ++ if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || closed) { |
| ++ // No installation of ours to activate, or one whose admission |
| ++ // fence is closed (the source in releasing, still op.exec, |
| ++ // which With refuses): a kept row's hold only, under the |
| ++ // flock. A target before installing has no row and mints none |
| ++ // (ruling 17). |
| ++ return r.holdOnly(ctx, selected, d) |
| ++ } |
| + } |
| // Activate's participant callback is not an admission validator for executors. |
| // The reader validates EVERY fresh read while Activate holds hostguard. |
| guard := r.Guard |
| -@@ -131,3 +175,32 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| +@@ -131,3 +189,32 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| } |
| return work.With(ctx, hostwrite.Identity{}, write) |
| } |
| @@ -8260,7 +8780,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ## Stage B re-plan (ruling 24) |
| -When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c, 9 and 10 against their merged code, then re-review. Every obligation rounds 1 to 4 placed on stage B, with the request that covers it and the test that proves it: |
| +When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c, 9 and 10 against their merged code, then re-review. Every obligation rounds 1 to 5 placed on stage B, with the request that covers it and the test that proves it: |
| | Obligation | Round | Covered by | Proved by | |
| |---|---|---|---| |
| @@ -8268,6 +8788,8 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| | The relocation controller's cleanup scoped to its own partial work; `Clear` is `ClearStale` with its config | 3 (N5), 4 (5) | `relocationCleanup`, `OpenRelocationController` | `TestRelocationCleanupNeverStopsAnAdmittedWriter`, the on-disk eligibility test over the assembly | |
| +| Clearing rests on a cleanup that succeeded now, never on `Close`'s nil (hostexec returns a cleanup failure once): the assembly sets `RelocationControllers.Cleanup` to `relocationCleanup` bound to the journaled binding (it checks the binding's box and enrollment against the verified actor, touches only that operation's partial install of that host and incarnation, and is a no-op once this box admitted a writer under the operation or serves the host) and `Clear` to `ClearStale` with the assembly's config for the binding; `Journal` is `<DataDir>/relocate/participations`, written before every open, so a restart (or a crash before the first `Finish`, or a failed `Open`) finds every record | 5 (R4-5, 4) | Task 7's `Finish`, `record`, `journaled`; R8's `applyRegionHA` | over the assembly: cleanup fails, a sweep, a restart with it still failing (the record stays and `hostexecParticipation` refuses), then success (cleared); a crash between `record` and `Open`, and between `Open` and `Finish`, each cleared after the restart; a truncated `.tmp` beside a valid entry | |
| +| The local install restores the env file from P's manifest exactly: `ReadPlatform(EnvItem)` returning `os.ErrNotExist` means the host has no env file, and any env file at the destination is removed under the install's authority, never kept | 5 (5) | Task 7's `StagePlatformDelete` on `ENOENT`; 6b's typed inventory | `…RestoresNoEnvFileWhenTheReleaseRemovedIt` | |
| | P built under the location lease with authority re-checked at every write; files manifest first; the verifier reads both manifests, the pinned one from the retained copy | 1 (2, 5, 7), 3 (R1-2), 4 (R1-5) | `authorityStore`, `CopyPinnedManifest`, `PreparedFromInstallManifest`; R7 | `TestRelocationBuildStopsWhenAuthorityChanges`, `TestPreparedRefusesAPartialBuild` | |
| | The local install under `install_stamp`; destination directories created and inventoried first; held staging through root's relay; never remade after P's admission | 1 (1), 4 (3, 4) | 6b's typed inventory; R9; moves Plan 1 Task 7b | `TestRelocationInstallIsBoundToTheInstallStamp`, `…CreatesAndInventoriesTheDestination`, `…WaitsForRootsHeldStaging` | |
| | The probe: inputs open P once with `Prepared`, later attempts ordinary (empty included), selected as `"probe"` work, one attempt per `BeginProbeAttempt` | 1 (1), 2 (N1, N6), 3 (R1-13), 4 (2) | R1 (dispatch, `Probe`, `"probe"` action) | `TestRelocationWakeInputsOpenPOnceThenRestoreThePredecessor`, `TestASecondProbeAttemptPublishesNoMarkerBeforeItsRoot` | |
| @@ -8392,8 +8914,13 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *Relocati |
| } |
| return &RelocationEffects{ |
| Lifetime: cfg.Lifetime, |
| - Controllers: &RelocationControllers{Journal: filepath.Join(cfg.DataDir, "relocate", "finishing"), |
| - Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg.Exec; c.Binding = b; return execctl.ClearStale(ctx, c) }}, |
| + // Rev 6 (round 5, R4-5): the journal is written before every open, |
| + // and Finish clears only straight after Cleanup succeeds. The |
| + // re-plan authenticates the binding against the verified actor |
| + // before relocationCleanup runs (stage B re-plan table). |
| + Controllers: &RelocationControllers{Journal: filepath.Join(cfg.DataDir, "relocate", "participations"), |
| + Cleanup: func(ctx context.Context, b execctl.Binding) error { return a.relocationCleanup(ctx, cfg, b.Host, b.Op) }, |
| + Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg.Exec; c.Binding = b; return execctl.ClearStale(ctx, c) }}, |
| Identity: func(ctx context.Context) (RelocationIdentity, error) { |
| l, err := a.localStopIdentity(ctx) // 6b: the verified actor, boot id and controller id |
| if err != nil { |
| @@ -10068,3 +10595,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 3 guards**, each removed in turn and its test watched to fail: the probe attempt bound; discharge through `CompleteObligation`; the former writer's last admission; the per-process descriptor scan; exhaustion before a spare is chosen; the retained manifest's digest; the attempt counted before the probe; the sweep's running-operation check; the recorded identity after a restart; the positions' box binding; `Finish` forgetting the controller. Eleven in all. |
| - **Rev 4 guards**, each removed in turn and its test watched to fail: the acknowledgement barrier on `releasing → pinning`; `FenceExhaustedTarget`'s exhaustion check; the worker's use of it over `MarkStuck`; the serving branch's obsolete-hold clearance; `RelocationPolicy`'s executor check; `AdmissionFenced`'s active fence; `Finish` keeping its entry until `Close` succeeds; the recorded directory revision after a restart. Eight in all. |
| - **Rev 5 guards**, each removed in turn and its test watched to fail: the lock-free fence observer (re-locking deadlocks the bounded test); the `probe` action in `Policy.Check`; the HA gate on the obsolete-hold clearance; `Finish`'s durable clear; the journal's retry after a restart; the hold-only path for a fenced executor; the observer error keeping rows. Seven in all. |
| +- **Rev 6 guards**, each removed in turn and its test watched to fail: `Finish`'s repeated scoped cleanup before every clear (the real hostexec test's sweep after a failed cleanup clears); the participation journaled before the open; the journal listing's `.tmp` exclusion; a damaged committed entry keeping the readable ones; `Sweep` reporting `Retained`'s error; the env file's staged deletion; `replaceTargetSlot`'s `park-local` refusal; the worker's `park-local` exclusion; the chooser's `park-local` exclusion; `fenceBox`'s regional reassignment; Task 9a's serving-only check with HA off. Eleven in all. |