amadan
docs: region HA plan 1b-next rev 15 prepared manifest at sequence 0, local install restored before P opens
Astra round 14: one partial and three P2s, all accepted. recordIsMine now reads the record through hostexec's own validated reader (hostexec.ReadLocal), so it keeps a participation whose record hostexec refuses. P's prepared files manifest is now create-only at sequence 0. PreparedAuthority keeps the source sequence as provenance and binds the install stamp, and ruling 13 is corrected. Before P's first admission, EnsureLocal verifies the journaled local install and restores it under install_stamp; an opened P is never reinstalled. The lifecycle property now covers installation and probing, with invariants (f) and (g). R1 gains the prepared-baseline consumption and source cold-admission dispatch, with named stage B tests. Dry-run in task order on 3407d114; 200 seeds pass; mutation results are in the dispositions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+501
−43
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+501 −43
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index 1542777a0..f7a526a9c 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,46 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 14) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 15) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 13 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 14 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 15: round 14 dispositions |
| + |
| +Astra's round 14 (`CODEX-PLAN-REGION-HA-1BN-R14.md`) resolved the lease-recovery finding. It left round 13's finding 2 partly resolved and raised three new P2s. All four are accepted; none is declined. |
| + |
| +Stage A was dry-run again on a clean 3407d114, applied in task order. The lifecycle property passed its 200 seeds (about three minutes, niced) and was re-run for every mutation. Each new guard was removed in turn; the results are below. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| R13-2 (partly) | A | Rev 14's `recordIsMine` decoded the JSON itself. A parseable record without its enrollment or incarnation, which hostexec's own reader refuses (`State.validate`, `Binding.valid`), was taken for another binding, and the participation was forgotten. | accepted | New `hostexec.ReadLocal(dataDir, host)` (`internal/hostexec/readlocal.go`) is hostexec's validated reader: the same checks `Open`, `CheckLocal` and `ClearStale` apply. `recordIsMine` uses it. Only an absent record, or a valid record of another binding, is "not mine"; anything hostexec refuses is an error, and the participation is kept. | Task 7 (`TestARecordHostexecRefusesKeepsItsParticipation`, for a missing enrollment and a missing incarnation: refused and kept, still kept after a restart, then the record is repaired and a cleanup and clear succeed) | |
| +| 1 (P2) | A, B | Claim rev 6.30 R18-1 and moves rev 7.22 §7.6 require the same thing: a create-only prepared manifest at sequence 0, bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. Rev 14's `ToInstalled` required P's sequence to equal the pin's, and `CopyPinnedManifest` kept the source's sequence. | accepted | `CopyPinnedManifest` writes P's prepared manifest at sequence 0, keeping the pinned entries by their content-addressed objects. `PreparedAuthority` separates the manifest's identity from its provenance: `FilesSeq` must be 0, `SourceFilesSeq` must be the pin's sequence, and `InstallStamp` is bound by `ToInstalled` from `install_stamp`. Ruling 13 is corrected. Stage B: R1 has the first admission consume that manifest by its journaled digest as the publisher's baseline, and the re-plan names the tests. | Task 1 (`TestRelocationPreparedManifestIsSequenceZero`: a prepared manifest claimed at the source's sequence 7 is refused, one built from another source revision is refused, and the accepted one is bound to the stamp); Task 5 (`TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne`: source at 7, prepared at 0, first session publication at 1, inherited entries intact); stage B re-plan | |
| +| 2 (P2) | A, B | `Step` in `installed` with `local_install` journaled went straight to probing. Nothing checked that the install still existed after a lost lease's scoped cleanup retired the unopened P's local install. The property never reached installation. | accepted | `hostrelocate.Effects.EnsureLocal` runs before probing and before every attempt until P is opened. `host.RelocationEffects.EnsureLocal` verifies the receipt's install (`VerifyLocal`). If the install is retired, it restores it with the same `Install` under the same `install_stamp`, refuses a restore that is not the receipt's, and verifies again. It never touches an opened P. The property now also covers the target's role: half the seeds start with the local install journaled, and a new event hands the installation to this box. Its effects are stateful: `Install` makes the install, the scoped cleanup retires it unless P is opened, and `Admit` opens P. Two invariants are added: (f) P's first admission never meets a missing local install, and (g) an opened P is never reinstalled. Stage B: the production `VerifyLocal` and restore, with named tests. | Task 6 (`TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission`), Task 7 (`TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens`, the property), stage B re-plan | |
| +| 3 (P2) | B | 6b's `runtime.Prerequisites` returns `ClaimDependencyError` for any probe or non-serving claim (6b plan, `if p.Probe \|\| o.Claim.Op.Kind != hostclaim.KindServing`). So a source that claim §4 and Task 1 let wake in `starting` and `releasing` could not, even after R1. | accepted (contract) | R1 now also asks for the source's cold-admission dispatch in `starting` and `releasing` while its admission fence is open, refused once the fence closes or the claim no longer names the box. A target's ordinary wake stays refused. The stage B checklist names the tests. | Requests (R1), stage B re-plan | |
| + |
| +Mutation check, with the lifecycle property at 200 seeds and `-failfast`, plus targeted tests: |
| + |
| +| Guard removed | Result | |
| +|---|---| |
| +| both `EnsureLocal` calls | killed, seed 32: (f) | |
| +| `EnsureLocal` before an attempt while P is unopened | killed, seed 142: (f) | |
| +| `EnsureLocal` before entering probing | survived: the attempt-time check covers it (defence in depth) | |
| +| `EnsureLocal` restoring a retired install / refusing a restore under another stamp | killed: `TestEnsureLocal…` | |
| +| `recordIsMine` through `hostexec.ReadLocal` | killed: `TestARecordHostexecRefusesKeepsItsParticipation` | |
| +| `recordIsMine` comparing the binding | killed, seed 3: (b) | |
| +| `ToInstalled` requiring sequence 0 / the pinned source sequence / binding the stamp | killed: `TestRelocationPreparedManifestIsSequenceZero` (each) | |
| +| `CopyPinnedManifest` writing sequence 0 | killed: `TestCopyPinnedManifestEnforcesThePin` | |
| +| rev 14's lifecycle guards (`finish`'s cleanup before a clear, verified forget, lost recovery, bounded open, journal before open, handover, close before clear, `ErrInactive` superseded, verified restart forget) | all killed again (seeds 2 to 46) | |
| + |
| +The `EnsureLocal` short-circuit for an opened P also survives, because the participant never calls `EnsureLocal` once P is opened. It is kept so the effect is safe on its own. |
| + |
| +Astra's readings of rulings 1 to 24 stand. Ruling 13 is corrected as finding 1 requires. |
| + |
| ## Rev 14: round 13 dispositions |
| Astra's round 13 (`CODEX-PLAN-REGION-HA-1BN-R13.md`) resolved both findings from round 12 and raised two P2s, both in the execution-controller lifecycle. Both are accepted; none is declined. This is the sixth round in a row to find a controller cleanup or recovery interleaving (rounds 2, 4, 5, 6, 12 and 13). So rev 14 also closes the whole class with a property test over real hostexec controllers, not only the two examples. |
| @@ -247,7 +278,7 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | # | To | Request | Waited for by | |
| |---|---|---|---| |
| -| R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. | Task 8c | |
| +| R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. *Rev 15 (round 14 finding 3):* claim §4 and Task 1's phase table also let the **source** admit, cold, in `starting` and `releasing`, but 6b's `runtime.Prerequisites` refuses every non-serving claim (`p.Probe || o.Claim.Op.Kind != KindServing`, 6b's `ClaimDependencyError` naming moves C1). Ask: dispatch a relocating claim's ordinary wake to the existing wake inputs when this box is `op.exec` and the source, in `starting` or `releasing`, with its admission fence open (Task 3's `WakePhase` and `RelocationPolicy`); refuse it once the fence closes or the claim no longer names this box. A relocation target's ordinary wake stays refused; its only wake is the probe. *Rev 15 (round 14 finding 1):* the probe's first admission consumes P's prepared manifest (sequence 0) as the session publisher's baseline, by the digest `prepared_authority.files_manifest_sha256` names (the store has no version-id read, ruling 5). The publisher starts at sequence 1. | Task 8c | |
| | R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) | |
| | R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a | |
| | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | |
| @@ -274,7 +305,7 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 10. **The probe deadline starts at the `probing` CAS** (`journal.probing_since`), and the worker moves a live target to `probe-failed` 10 minutes later. |
| 11. **A host whose source never had a writer** (session 0, `writer_box` empty) is relocated with coverage `""`, which declares nothing (claim §6.4a: "a created incarnation's first admission … declares nothing"), never `released`. *Rev 2:* only by failover. A drain of such a host is refused at `Worker.Start`: root refuses stop evidence for writer session 0 (`validateRequest`), and 6b's `collectCapturedStop` waits for claim Plan 5's no-process witness (R4). Its failover pin is an evidenced empty one (ruling 15). *Cost if wrong:* a box with never-woken hosts cannot be drained until R4 lands; nothing is lost. |
| 12. **L18's owner.** The coordination post on #236 says moves owns `internal/sim/hostlife`; claim rev 6.30 §1.1 calls the two-box model "region HA's own … as part of region HA's plan"; the moves conflicts note defers it to "a joint round with the region HA session". This plan writes no model and ends with a merge gate that names the cases; whichever session builds the model, relocation does not merge until that round passes for a one-box and a two-box region. |
| -13. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1), which is what this plan's install already assumes. Nothing in L14 to L17 changed against it. |
| +13. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1). *Corrected in rev 15 (round 14 finding 1):* until rev 14 this plan did not follow that binding, because it copied the source manifest's sequence into P. Under claim rev 6.30 (R18-1) and moves rev 7.22 §7.6, P's prepared manifest is create-only at sequence 0 and bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. The source pin's sequence is provenance only (`PreparedAuthority.SourceFilesSeq`). `CopyPinnedManifest` writes sequence 0, and `ToInstalled` requires it and binds the stamp. Nothing in L14 to L17 changed against it. |
| 14. **The release deadline is 10 minutes** (`hostrelocate.ReleaseDeadline`, rev 2). §5.8 says a flush that never reaches N, and a `probe-failed` target that cannot finish its release, go to `stuck`, and names no bound. `ToReleasing` and `ToProbeFailed` journal when the phase began; a worker marks the operation stuck once that long has passed with no release journaled. The host stays where it is: nothing moves away from a live box without its stop evidence. *Cost if wrong:* a number; the stuck state and the operator path are the claim's. |
| 15. **An empty pin needs evidence a writer can give** (rev 2; claim §6.4a "Empty bootstrap": no LTX file exists until the first commit, so a listing that finds nothing is a definite observation but says nothing about who wrote). A pin is `Empty` only when the journaled release reports `final_txid` 0 (the stopped writer flushed and found nothing), or the holder never admitted a writer (session 0, failover). A failover of a host whose session is above 0 with an empty listing refuses and waits for an operator; `JournalPin` enforces it, the pinner never writes it. *Cost if wrong:* that one case recovers by hand until R6 names its evidence. |
| 16. **A pin is bound to the claim generation it was selected against** (rev 2; B2-3). Its key is `control/relocations/<host>/<op>/pin-<gen>.json` and `JournalPin` requires `gen` to be the observed claim's. A pin whose journal CAS lost is abandoned where it lies (never referenced, cheap, reaped by the storage follow-up of ruling 5); the next pass selects afresh at the new generation. Every bound `JournalPin` checks (the release's positions, the failover window) is checked before the object is written, so a doomed pin is never published. *Cost if wrong:* abandoned objects under a prefix nothing reads. |
| @@ -313,7 +344,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four and rev 14's ten killed mutations of the lifecycle property (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property and rev 15's twelve (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -467,7 +498,7 @@ func relPin(o Observed, maxTXID uint64, seq int64, at time.Time) RecoveryPin { |
| } |
| func relPrepared(gen string, maxTXID uint64, filesSeq int64) PreparedAuthority { |
| - a := PreparedAuthority{Generation: gen, SnapshotTXID: maxTXID, FilesSeq: filesSeq, FilesManifestSHA256: strings.Repeat("f", 64)} |
| + a := PreparedAuthority{Generation: gen, SnapshotTXID: maxTXID, FilesSeq: 0, SourceFilesSeq: filesSeq, FilesManifestSHA256: strings.Repeat("f", 64)} |
| if maxTXID > 0 { |
| a.InstallManifest, a.SHA256 = "m", strings.Repeat("c", 64) |
| } |
| @@ -1358,6 +1389,35 @@ func TestRelocationPOpenedOutlivesTheAdmissionTrim(t *testing.T) { |
| t.Fatal("the local install was remade after P was opened") |
| } |
| } |
| + |
| +// Round 14 finding 1: P's prepared manifest is at sequence 0 whatever the |
| +// source's; the pin's sequence (7 here) is its provenance. ToInstalled |
| +// refuses a prepared manifest claimed at the source's sequence, and one |
| +// built from another source revision, and binds the one it accepts to |
| +// the installation stamp. |
| +func TestRelocationPreparedManifestIsSequenceZero(t *testing.T) { |
| + ctx := context.Background() |
| + s, o := relAtPhase(t, PhaseInstalling) |
| + var pin RecoveryPin |
| + readJournal(o.Claim.Op, "pin", &pin) |
| + if pin.FilesSeq != 7 { |
| + t.Fatalf("pin at %d", pin.FilesSeq) |
| + } |
| + atSource := relPrepared("P1", 40, 7) |
| + atSource.FilesSeq = 7 |
| + if _, err := Update(ctx, s, o, ToInstalled("rel-1", atSource)); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("a prepared manifest at the source's sequence: %v", err) |
| + } |
| + if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 6))); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("a prepared manifest from another source revision: %v", err) |
| + } |
| + o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| + var a PreparedAuthority |
| + readJournal(o.Claim.Op, "prepared_authority", &a) |
| + if a.FilesSeq != 0 || a.SourceFilesSeq != 7 || a.InstallStamp == 0 || a.InstallStamp != journalUint(o.Claim.Op, "install_stamp") { |
| + t.Fatalf("journaled authority: %+v", a) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -1509,13 +1569,22 @@ type PreparedAuthority struct { |
| InstallManifest string `json:"install_manifest"` |
| SHA256 string `json:"sha256"` |
| SnapshotTXID uint64 `json:"snapshot_txid"` |
| - // FilesSeq and FilesManifestSHA256 are P's prepared files manifest |
| - // (moves §7.6 L17): its sequence, which must be the pin's, and the |
| - // digest of the exact bytes a fresh read verified. Required always, |
| - // an empty bootstrap included: a P without its files manifest is not |
| - // built (A1-2's "all three objects"). |
| + // FilesSeq and FilesManifestSHA256 identify P's prepared files |
| + // manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): create-only, |
| + // at sequence 0, never replaced; FilesManifestSHA256 is the digest of |
| + // the exact bytes a fresh read verified, the exact version the first |
| + // admission consumes as its baseline (the session publisher then |
| + // starts at sequence 1). SourceFilesSeq is provenance, not identity: |
| + // the pinned source manifest's sequence it was built from, which must |
| + // be the pin's (rev 15, round 14 finding 1). Required always, an empty |
| + // bootstrap included: a P without its files manifest is not built. |
| FilesSeq int64 `json:"files_seq"` |
| + SourceFilesSeq int64 `json:"source_files_seq"` |
| FilesManifestSHA256 string `json:"files_manifest_sha256"` |
| + // InstallStamp binds the prepared manifest to the installation that |
| + // built it ({location, P, op id, installation stamp}): ToInstalled |
| + // sets it from install_stamp in the CAS that journals the authority. |
| + InstallStamp uint64 `json:"install_stamp,omitempty"` |
| } |
| // AutomaticLoss is claim §7.3 D4's "Automatic loss persistence": what the |
| @@ -2161,13 +2230,14 @@ func ToInstalled(opID string, a PreparedAuthority) func(*Claim) error { |
| (pin.MaxTXID == 0 && (a.InstallManifest != "" || a.SHA256 != "")) { |
| return fmt.Errorf("%w: prepared authority is not this pin's build of P", ErrRelocation) |
| } |
| - if a.FilesSeq != pin.FilesSeq || !isHex64(a.FilesManifestSHA256) { |
| - return fmt.Errorf("%w: prepared authority lacks P's files manifest at the pinned sequence", ErrRelocation) |
| + if a.FilesSeq != 0 || a.SourceFilesSeq != pin.FilesSeq || !isHex64(a.FilesManifestSHA256) { |
| + return fmt.Errorf("%w: prepared authority lacks P's prepared files manifest (sequence 0, built from the pinned sequence)", ErrRelocation) |
| } |
| if err := requireAcks(c); err != nil { |
| return err |
| } |
| c.Op.Phase = PhaseInstalled |
| + a.InstallStamp = journalUint(c.Op, "install_stamp") |
| return journalJSON(c, "prepared_authority", a) |
| } |
| } |
| @@ -5127,7 +5197,7 @@ func (built) Verify(_ context.Context, o hostclaim.Observed) (hostclaim.Prepared |
| var pin hostclaim.RecoveryPin |
| _ = json.Unmarshal([]byte(c.Op.Journal["pin"]), &pin) |
| return hostclaim.PreparedAuthority{Generation: c.Op.Journal["prepared"], InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: pin.MaxTXID, |
| - FilesSeq: pin.FilesSeq, FilesManifestSHA256: strings.Repeat("f", 64)}, true, nil |
| + SourceFilesSeq: pin.FilesSeq, FilesManifestSHA256: strings.Repeat("f", 64)}, true, nil |
| } |
| type pins struct{ f *fixture } |
| @@ -5906,7 +5976,7 @@ func TestCopyPinnedManifestEnforcesThePin(t *testing.T) { |
| claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) |
| pin, want := pinnedManifest(t, claims, data, "P1", pinLocation+"/files/g1/", 3) |
| got, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P2/") |
| - if err != nil || got.Seq != want.Seq || got.Entries["a.txt"] != want.Entries["a.txt"] { |
| + if err != nil || got.Seq != 0 || want.Seq != 3 || got.Entries["a.txt"] != want.Entries["a.txt"] { |
| t.Fatalf("copy: %+v %v", got, err) |
| } |
| // A replay is a success; the copy's entries still name g1's object. |
| @@ -5926,13 +5996,14 @@ func TestCopyPinnedManifestEnforcesThePin(t *testing.T) { |
| claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) |
| pin, m := pinnedManifest(t, claims, data, "P1", pinLocation+"/files/g1/", 3) |
| m.Seq = 4 |
| + m.Entries["late.txt"] = m.Entries["a.txt"] |
| raw, _ := json.Marshal(m) |
| _, etag, _ := data.Get(ctx, hostfiles.ManifestKey(pin.Prefix)) |
| if _, err := data.Put(ctx, hostfiles.ManifestKey(pin.Prefix), raw, etag); err != nil { |
| t.Fatal(err) |
| } |
| got, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P2/") |
| - if err != nil || got.Seq != 3 { |
| + if _, late := got.Entries["late.txt"]; err != nil || got.Seq != 0 || late || len(got.Entries) != 1 { |
| t.Fatalf("the pinned revision: %+v %v", got, err) |
| } |
| }) |
| @@ -6067,6 +6138,62 @@ func TestPinKeepsAFileOriginTheAdmissionListTrimmed(t *testing.T) { |
| t.Fatalf("a foreign reference was pinned: %v", err) |
| } |
| } |
| + |
| +// Round 14 finding 1: the source's files manifest is at sequence 7; P's |
| +// prepared manifest is a create-only copy of its entries at sequence 0 |
| +// (claim rev 6.30 R18-1, moves rev 7.22 §7.6); the session publisher that |
| +// starts after the first admission takes it as its baseline and publishes |
| +// sequence 1, the inherited entries intact. |
| +func TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne(t *testing.T) { |
| + ctx := context.Background() |
| + claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) |
| + src := pinLocation + "/files/g3/" |
| + p, err := hostfiles.Open(ctx, data, src) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for i := 1; i <= 7; i++ { |
| + if err := p.Stage(ctx, "a.txt", []byte(fmt.Sprintf("revision %d", i)), 0o644, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := p.Commit(ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + pinner := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} |
| + ref, err := pinner.Pin(ctx, pinClaim(), time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)) |
| + if err != nil || ref.FilesSeq != 7 { |
| + t.Fatalf("pin: %+v %v", ref, err) |
| + } |
| + obj, err := ReadPin(ctx, claims, ref) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + prepared := pinLocation + "/files/P1/" |
| + m, err := CopyPinnedManifest(ctx, claims, data, obj.Files, prepared) |
| + if err != nil || m.Seq != 0 { |
| + t.Fatalf("prepared manifest: %+v %v", m, err) |
| + } |
| + stored, _, err := hostfiles.Load(ctx, data, prepared) |
| + if err != nil || stored.Seq != 0 || stored.Entries["a.txt"].Object == "" || !strings.HasPrefix(stored.Entries["a.txt"].Object, src+"objects/") { |
| + t.Fatalf("stored prepared manifest: %+v %v", stored, err) |
| + } |
| + session, err := hostfiles.Open(ctx, data, prepared) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := session.Stage(ctx, "b.txt", []byte("first session write"), 0o644, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + seq, err := session.Commit(ctx) |
| + if err != nil || seq != 1 { |
| + t.Fatalf("the first session publication: %d %v", seq, err) |
| + } |
| + after, _, err := hostfiles.Load(ctx, data, prepared) |
| + if err != nil || after.Seq != 1 || after.Entries["a.txt"] != stored.Entries["a.txt"] || after.Entries["b.txt"].Object == "" { |
| + t.Fatalf("after the first publication: %+v %v", after, err) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -6932,6 +7059,14 @@ func CopyPinnedManifest(ctx context.Context, claims, data release.Store, pin Fil |
| return hostfiles.Manifest{}, err |
| } |
| } |
| + // P's prepared manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): |
| + // the pinned entries, by their exact content-addressed objects, at |
| + // sequence 0. The source's sequence is provenance (the pin keeps it), |
| + // never P's: the session publisher starts at 1 after the first |
| + // admission. The binding {location, P, op, install stamp} is the |
| + // prefix (location, P) and the claim's prepared_authority (op, stamp, |
| + // this manifest's digest). |
| + pinned.Seq = 0 |
| body, err := json.Marshal(pinned) |
| if err != nil { |
| return hostfiles.Manifest{}, err |
| @@ -7000,7 +7135,7 @@ A participant box acknowledges first (D6, after §5.10's pending-state and updat |
| **Interfaces:** |
| - Consumes: Tasks 1 and 5 (the worker fixture in `worker_test.go`). |
| -- Produces: `hostrelocate.Effects` (`Acknowledge`, `Ready`, `Release`, `Build`, `InstallLocal`, `Probe`, `Retained() ([]Participation, error)`, `Finish`), `hostrelocate.Participation{Host, Op}`, `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)` and `Sweep(ctx) error`. |
| +- Produces: `hostrelocate.Effects` (`Acknowledge`, `Ready`, `Release`, `Build`, `InstallLocal`, `EnsureLocal` (rev 15), `Probe`, `Retained() ([]Participation, error)`, `Finish`), `hostrelocate.Participation{Host, Op}`, `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)` and `Sweep(ctx) error`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -7037,6 +7172,10 @@ type fakeBox struct { |
| failProbes bool |
| // retainedErr is a damaged journal entry Retained reports. |
| retainedErr error |
| + // retired: a scoped cleanup removed the local install; restores, |
| + // reinstalls and probedMissing count what EnsureLocal and Probe saw. |
| + retired bool |
| + restores, reinstalls, probedMissing int |
| } |
| func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) { |
| @@ -7075,7 +7214,20 @@ func (b *fakeBox) InstallLocal(_ context.Context, o hostclaim.Observed) (hostcla |
| n, _ := strconv.ParseUint(o.Claim.Op.Journal["install_stamp"], 10, 64) |
| return hostclaim.LocalInstall{Box: b.box, Enrollment: enrollment(b.box), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil |
| } |
| +func (b *fakeBox) EnsureLocal(_ context.Context, o hostclaim.Observed) error { |
| + b.effects = append(b.effects, "ensure-local") |
| + if hostclaim.PreparedOpened(o.Claim) { |
| + b.reinstalls++ // never: an opened P is not reinstalled |
| + } |
| + if b.retired { |
| + b.retired, b.restores = false, b.restores+1 |
| + } |
| + return nil |
| +} |
| func (b *fakeBox) Probe(ctx context.Context, o hostclaim.Observed, coverage string) (uint64, bool, error) { |
| + if b.retired { |
| + b.probedMissing++ // a probe admission over a missing local install |
| + } |
| b.coverages = append(b.coverages, coverage) |
| c := o.Claim |
| next, err := hostclaim.Update(ctx, b.f.s, o, hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: c.Op.ID, ObservedPhase: c.Op.Phase, ObservedSession: c.Holder.Session, ObservedGeneration: c.Holder.Generation, |
| @@ -7270,6 +7422,34 @@ func TestAParticipantFinishesItsControllerOnceTheOperationIsOver(t *testing.T) { |
| t.Fatalf("a running operation's controller was finished: %v", err) |
| } |
| } |
| + |
| +// Round 14 finding 2: the local install is journaled, the lease is lost, |
| +// and the scoped cleanup retires the unopened P's local install. Before |
| +// probing, and before P's first admission, the participant has the |
| +// install verified and restored; once P is opened it is never touched. |
| +func TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission(t *testing.T) { |
| + f := newFixture(t) |
| + ctx := context.Background() |
| + src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"} |
| + if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} |
| + pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} |
| + for i := 0; i < 40 && f.claim().Claim.Op.Journal["local_install"] == ""; i++ { |
| + _, _ = f.w.Step(ctx, host) |
| + ps.Step(ctx, host) |
| + pd.Step(ctx, host) |
| + } |
| + if f.claim().Claim.Op.Journal["local_install"] == "" { |
| + t.Fatalf("never installed locally: %+v", f.claim().Claim.Op) |
| + } |
| + dst.retired = true // the lease was lost; the scoped cleanup retired it |
| + run(t, f, ps, pd) |
| + if dst.restores != 1 || dst.probedMissing != 0 || dst.reinstalls != 0 { |
| + t.Fatalf("restores %d, probes over a missing install %d, reinstalls after P %d", dst.restores, dst.probedMissing, dst.reinstalls) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -7325,6 +7505,13 @@ type Effects interface { |
| // InstallLocal makes this box's local install from P's files manifest |
| // and returns the receipt (§5.8 "The local install"). |
| InstallLocal(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) |
| + // EnsureLocal is called before P's first admission, each time: it |
| + // verifies that the local install the journaled receipt names is |
| + // complete on this box and, when it is not (a lost lease's scoped |
| + // cleanup retires an unopened P's local install), restores it under |
| + // install_stamp from P's files manifest. It is never called once P is |
| + // opened: an opened P is never reinstalled (rev 15, round 14). |
| + EnsureLocal(context.Context, hostclaim.Observed) error |
| // Probe makes one cold probe admission (AdmitWriter, with coverage the |
| // caller names) and runs the probe. passed is false while it runs. |
| Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (session uint64, passed bool, err error) |
| @@ -7398,6 +7585,11 @@ func (p Participant) Step(ctx context.Context, host string) (hostclaim.Observed, |
| return obs, errors.Join(ErrNotYet, p.Effects.Build(ctx, obs)) |
| case executor && c.Op.Phase == hostclaim.PhaseInstalled: |
| if c.Op.Journal["local_install"] != "" { |
| + // The receipt is a past fact; the files, env and timers it |
| + // names are checked (and restored) before probing. |
| + if err := p.Effects.EnsureLocal(ctx, obs); err != nil { |
| + return obs, err |
| + } |
| // The target enters probing itself (§5.8 phase table). |
| return p.update(ctx, obs, hostclaim.ToProbing(id, p.Now())) |
| } |
| @@ -7410,6 +7602,12 @@ func (p Participant) Step(ctx context.Context, host string) (hostclaim.Observed, |
| } |
| return p.update(ctx, obs, hostclaim.RecordLocalInstall(id, l)) |
| case executor && c.Op.Phase == hostclaim.PhaseProbing && c.Op.Journal["probe"] == "": |
| + if !hostclaim.PreparedOpened(c) { |
| + // Before P's first admission the local install must be whole. |
| + if err := p.Effects.EnsureLocal(ctx, obs); err != nil { |
| + return obs, err |
| + } |
| + } |
| // The attempt is counted in the claim before it is made (§5.8: |
| // at most three); the fourth is probe-failed, by this CAS. |
| counted, err := hostclaim.Update(ctx, p.Claims, obs, hostclaim.BeginProbeAttempt(id, p.Box)) |
| @@ -7516,6 +7714,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Files:** |
| - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` |
| - Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationcontrollers_linux_test.go`, `internal/host/relocationlifecycle_linux_test.go` (rev 14) |
| +- Create: `internal/hostexec/readlocal.go` (rev 15: `hostexec.ReadLocal`) |
| **Interfaces:** |
| - Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `ReadPlatform`, `registry.Registry`, `hostexec.ClearStale`. |
| @@ -8249,6 +8448,42 @@ func TestARestartFindsAParticipationThatNeverReachedFinish(t *testing.T) { |
| t.Fatalf("the participation outlived its clear: %+v", got) |
| } |
| } |
| + |
| +// Round 14 finding 2: EnsureLocal verifies the receipt's install, restores |
| +// a retired one under the same stamp and verifies it again, refuses a |
| +// restore that is not the receipt's, and never touches an opened P. |
| +func TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens(t *testing.T) { |
| + ctx := context.Background() |
| + e, obs, _, _ := effectsFixture(t) |
| + receipt := hostclaim.LocalInstall{Box: "dst", Enrollment: "enr-d", InstallStamp: 9, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)} |
| + raw, _ := json.Marshal(receipt) |
| + obs.Claim.Op.Phase = hostclaim.PhaseProbing |
| + obs.Claim.Op.Journal = map[string]string{"local_install": string(raw), "prepared": "P1", "install_stamp": "9"} |
| + present, installs := true, 0 |
| + e.VerifyLocal = func(context.Context, hostclaim.Observed, hostclaim.LocalInstall) (bool, error) { return present, nil } |
| + restored := receipt |
| + e.Install = func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) { |
| + installs++ |
| + present = true |
| + return restored, nil |
| + } |
| + if err := e.EnsureLocal(ctx, obs); err != nil || installs != 0 { |
| + t.Fatalf("a complete install: %v installs=%d", err, installs) |
| + } |
| + present = false |
| + if err := e.EnsureLocal(ctx, obs); err != nil || installs != 1 || !present { |
| + t.Fatalf("a retired install: %v installs=%d", err, installs) |
| + } |
| + present, restored.InstallStamp = false, 10 |
| + if err := e.EnsureLocal(ctx, obs); !errors.Is(err, ErrClaimStopReceipt) { |
| + t.Fatalf("a restore under another stamp: %v", err) |
| + } |
| + present, installs = false, 0 |
| + obs.Claim.Op.Journal["prepared_opened"] = "P1" |
| + if err := e.EnsureLocal(ctx, obs); err != nil || installs != 0 { |
| + t.Fatalf("an opened P was reinstalled: %v installs=%d", err, installs) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -8550,6 +8785,7 @@ package host |
| import ( |
| "context" |
| + "encoding/json" |
| "errors" |
| "os" |
| "path/filepath" |
| @@ -8828,6 +9064,78 @@ func TestASuccessorRelocationRetiresItsPredecessorsController(t *testing.T) { |
| t.Fatal("a controller was kept from a cancelled open") |
| } |
| } |
| + |
| +// Round 14: a failed clear forgets a participation only on hostexec's own |
| +// validated reading of the record. A parseable record missing its |
| +// enrollment or incarnation is one hostexec refuses, so it is never |
| +// "another binding": the participation is kept, across a restart, until |
| +// the record is repaired and a cleanup and clear succeed. |
| +func TestARecordHostexecRefusesKeepsItsParticipation(t *testing.T) { |
| + ctx := context.Background() |
| + const host = "shop.bab.oncarlos.com" |
| + b := execctl.Binding{Box: "src", Enrollment: strings.Repeat("a", 32), Host: host, Incarnation: "inc", Op: "rel-1", BootID: "boot", ControllerID: "ctl"} |
| + valid := execctl.State{V: 1, Binding: b, Expires: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), SafeUntil: 1, State: "lost"} |
| + for _, missing := range []string{"enrollment", "incarnation"} { |
| + t.Run(missing, func(t *testing.T) { |
| + dataDir := t.TempDir() |
| + if err := os.Mkdir(filepath.Join(dataDir, ".hostexec"), 0o770); err != nil { |
| + t.Fatal(err) |
| + } |
| + path := filepath.Join(dataDir, ".hostexec", host+".json") |
| + write := func(st execctl.State, drop string) { |
| + t.Helper() |
| + raw, _ := json.Marshal(st) |
| + var m map[string]any |
| + _ = json.Unmarshal(raw, &m) |
| + delete(m, drop) |
| + raw, _ = json.Marshal(m) |
| + if err := os.WriteFile(path, raw, 0o660); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + write(valid, missing) |
| + cleared, cleanups := 0, 0 |
| + clearErr := execctl.ErrState // ClearStale refuses the invalid record |
| + controllers := func() *RelocationControllers { |
| + return &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "participations"), DataDir: dataDir, |
| + Cleanup: func(context.Context, execctl.Binding) error { cleanups++; return nil }, |
| + Clear: func(context.Context, execctl.Binding) error { |
| + if clearErr != nil { |
| + return clearErr |
| + } |
| + cleared++ |
| + return os.Remove(path) |
| + }} |
| + } |
| + r := controllers() |
| + if err := r.record(host, "rel-1", b); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := r.Finish(ctx, host, "rel-1"); err == nil { |
| + t.Fatal("a record hostexec refuses was taken as another binding") |
| + } |
| + if got := kept(&RelocationEffects{Controllers: r}); len(got) != 1 { |
| + t.Fatalf("the participation was forgotten: %+v", got) |
| + } |
| + // A restart: the journal still names it, and still nothing clears. |
| + r = controllers() |
| + if err := r.Finish(ctx, host, "rel-1"); err == nil || len(kept(&RelocationEffects{Controllers: r})) != 1 { |
| + t.Fatalf("after a restart: %v", err) |
| + } |
| + // Repaired: the record is valid and this binding's; the cleanup |
| + // and the clear run, and only then is the participation gone. |
| + write(valid, "") |
| + clearErr = nil |
| + before := cleanups |
| + if err := r.Finish(ctx, host, "rel-1"); err != nil || cleared != 1 || cleanups != before+1 { |
| + t.Fatalf("after the repair: %v cleared=%d cleanups=%d", err, cleared, cleanups-before) |
| + } |
| + if got := kept(&RelocationEffects{Controllers: r}); len(got) != 0 { |
| + t.Fatalf("retained after the clear: %+v", got) |
| + } |
| + }) |
| + } |
| +} |
| ``` |
| `internal/host/relocationlifecycle_linux_test.go` (rev 14: the controller lifecycle as a property over real hostexec controllers; 200 seeds, `-short` 20): |
| @@ -8846,6 +9154,7 @@ import ( |
| "os" |
| "path/filepath" |
| "slices" |
| + "strconv" |
| "strings" |
| "sync" |
| "testing" |
| @@ -8950,8 +9259,12 @@ type lifeWorld struct { |
| cleanErr bool // both cleanups fail |
| block chan struct{} // non-nil: both cleanups wait for it (a close that times out) |
| cleanOK map[string]bool |
| - saved []byte // a record the seed corrupted, to restore on recovery |
| - violated []string |
| + // installed is this box's local install per operation (rev 15): the |
| + // Install provider makes it, a scoped cleanup of an operation whose P |
| + // is unopened retires it, VerifyLocal reads it. |
| + installed map[string]bool |
| + saved []byte // a record the seed corrupted, to restore on recovery |
| + violated []string |
| } |
| const lifeDeadline = 400 * time.Millisecond |
| @@ -8959,7 +9272,7 @@ const lifeDeadline = 400 * time.Millisecond |
| func newLifeWorld(t *testing.T) *lifeWorld { |
| t.Helper() |
| ctx := context.Background() |
| - w := &lifeWorld{t: t, ctx: ctx, host: "shop.bab.oncarlos.com", clock: &lifeClock{wall: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), boot: int64(time.Hour)}, cleanOK: map[string]bool{}} |
| + w := &lifeWorld{t: t, ctx: ctx, host: "shop.bab.oncarlos.com", clock: &lifeClock{wall: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), boot: int64(time.Hour)}, cleanOK: map[string]bool{}, installed: map[string]bool{}} |
| w.dataDir = t.TempDir() |
| for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { |
| if err := os.Mkdir(filepath.Join(w.dataDir, name), 0o770); err != nil { |
| @@ -8977,7 +9290,7 @@ func newLifeWorld(t *testing.T) *lifeWorld { |
| w.claim(hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| w.cfg = execctl.Config{Store: w.store, Guard: hostguard.Guard{DataDir: w.dataDir, LockDir: filepath.Join(w.dataDir, "locks"), Claims: hostguard.StoreReader{Store: w.store}}, |
| Directory: activeDirectory{}, Clock: w.clock, LockDir: filepath.Join(w.dataDir, "exec-locks"), |
| - CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: func(context.Context) error { return w.cleanup(context.Background(), "") }} |
| + CloseAdmission: func() {}, CancelWork: func() {}} |
| lifetime, stop := context.WithCancel(ctx) |
| t.Cleanup(stop) |
| w.e = &RelocationEffects{Lifetime: lifetime, |
| @@ -8987,11 +9300,47 @@ func newLifeWorld(t *testing.T) *lifeWorld { |
| Controller: func(ctx context.Context, b execctl.Binding) (RelocationController, error) { |
| c := w.cfg |
| c.Binding = b |
| + // hostexec's own Cleanup hook is the same scoped cleanup, |
| + // bound to the binding (Task 8a); it does not count for (a). |
| + c.Cleanup = func(context.Context) error { return w.cleanupScoped(context.Background(), b.Op, false) } |
| opened, err := OpenRelocationController(ctx, c) |
| if err != nil { |
| return nil, err |
| } |
| return &lifeController{RelocationController: opened, w: w, op: b.Op}, nil |
| + }, |
| + Install: func(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) { |
| + c := w.current() |
| + if hostclaim.PreparedOpened(c) { |
| + w.violate("(g) the local install was remade after P was opened") |
| + } |
| + w.mu.Lock() |
| + w.installed[c.Op.ID] = true |
| + w.mu.Unlock() |
| + n, _ := strconv.ParseUint(c.Op.Journal["install_stamp"], 10, 64) |
| + return hostclaim.LocalInstall{Box: "src", Enrollment: strings.Repeat("a", 32), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil |
| + }, |
| + VerifyLocal: func(_ context.Context, o hostclaim.Observed, _ hostclaim.LocalInstall) (bool, error) { |
| + w.mu.Lock() |
| + defer w.mu.Unlock() |
| + return w.installed[o.Claim.Op.ID], nil |
| + }, |
| + Admit: func(_ context.Context, o hostclaim.Observed, _ string) (uint64, bool, error) { |
| + c := w.current() |
| + w.mu.Lock() |
| + present := w.installed[c.Op.ID] |
| + w.mu.Unlock() |
| + if !hostclaim.PreparedOpened(c) && !present { |
| + w.violate("(f) P's first admission over a missing local install") |
| + } |
| + // The admission opens P (the claim journals it). |
| + if _, err := hostclaim.Update(context.Background(), w.inner, o, func(c *hostclaim.Claim) error { |
| + c.Op.Journal["prepared_opened"] = c.Op.Journal["prepared"] |
| + return nil |
| + }); err != nil { |
| + return 0, false, err |
| + } |
| + return 0, false, nil // the probe is still running |
| }} |
| w.e.Controllers = w.controllers() |
| w.p = hostrelocate.Participant{Claims: w.store, Box: "src", Enrollment: strings.Repeat("a", 32), Effects: w.e, Now: func() time.Time { return w.clock.wall }} |
| @@ -9013,7 +9362,7 @@ func (c *lifeController) Clear(ctx context.Context) error { |
| func (w *lifeWorld) controllers() *RelocationControllers { |
| return &RelocationControllers{Journal: filepath.Join(w.dataDir, "relocate", "participations"), DataDir: w.dataDir, |
| - Cleanup: func(ctx context.Context, b execctl.Binding) error { return w.cleanup(ctx, b.Op) }, |
| + Cleanup: func(ctx context.Context, b execctl.Binding) error { return w.cleanupScoped(ctx, b.Op, true) }, |
| Clear: func(ctx context.Context, b execctl.Binding) error { |
| w.clearing(b.Op) |
| c := w.cfg |
| @@ -9022,11 +9371,14 @@ func (w *lifeWorld) controllers() *RelocationControllers { |
| }} |
| } |
| -// cleanup is both cleanups: hostexec's hook (op "") and the scoped one. |
| -func (w *lifeWorld) cleanup(ctx context.Context, op string) error { |
| +// cleanupScoped is the scoped cleanup (Task 8a's relocationCleanup): |
| +// it retires the operation's local install unless the claim has opened |
| +// that operation's P (it never stops what serves). counts says whether it |
| +// is RelocationControllers' own call, the one invariant (a) asks for. |
| +func (w *lifeWorld) cleanupScoped(ctx context.Context, op string, counts bool) error { |
| w.mu.Lock() |
| block, fail := w.block, w.cleanErr |
| - if op != "" { |
| + if counts { |
| w.cleanOK[op] = false |
| } |
| w.mu.Unlock() |
| @@ -9040,11 +9392,16 @@ func (w *lifeWorld) cleanup(ctx context.Context, op string) error { |
| if fail { |
| return errors.New("cleanup: a partial install would not go") |
| } |
| - if op != "" { |
| - w.mu.Lock() |
| + c, err := hostclaim.Read(context.Background(), w.inner, w.host) |
| + opened := err == nil && !c.Absent && c.Claim.Op.ID == op && hostclaim.PreparedOpened(c.Claim) |
| + w.mu.Lock() |
| + if !opened { |
| + w.installed[op] = false |
| + } |
| + if counts { |
| w.cleanOK[op] = true |
| - w.mu.Unlock() |
| } |
| + w.mu.Unlock() |
| return nil |
| } |
| @@ -9090,7 +9447,28 @@ func (w *lifeWorld) startRelocation() { |
| } |
| w.claim(func(c *hostclaim.Claim) error { |
| c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: id, Phase: hostclaim.PhaseStarting, Exec: from, Participants: []string{from, to}, |
| - Intent: map[string]string{"region": "ie", "source_box": from, "target_box": to}} |
| + Intent: map[string]string{"region": "ie", "source_box": from, "target_box": to, "target_enrollment": map[string]string{"src": strings.Repeat("a", 32), "dst": strings.Repeat("d", 32)}[to]}} |
| + return nil |
| + }) |
| +} |
| + |
| +// install is the worker's half while this box is the target: P built and |
| +// the operation handed to it (installed), its local install still to make. |
| +func (w *lifeWorld) install() { |
| + c := w.current() |
| + if c.Op.Kind != hostclaim.KindRelocating || c.Op.Intent["target_box"] != "src" || (c.Op.Phase != hostclaim.PhaseStarting && c.Op.Phase != hostclaim.PhaseReleasing) { |
| + return |
| + } |
| + w.claim(func(c *hostclaim.Claim) error { |
| + c.Op.Phase, c.Op.Exec = hostclaim.PhaseInstalled, "src" |
| + if c.Op.Journal == nil { |
| + c.Op.Journal = map[string]string{} |
| + } |
| + c.Op.Journal["install_stamp"] = strconv.FormatUint(c.Gen+1, 10) |
| + c.Op.Journal["prepared"] = "P-" + c.Op.ID |
| + if c.Op.Journal["exec_ack.dst"] == "" { |
| + c.Op.Journal["exec_ack.dst"] = "{}" |
| + } |
| return nil |
| }) |
| } |
| @@ -9216,7 +9594,10 @@ func (w *lifeWorld) recoverAll() { |
| } |
| func (w *lifeWorld) event(r *rand.Rand) string { |
| - switch n := r.Intn(14); n { |
| + switch n := r.Intn(15); n { |
| + case 14: |
| + w.install() |
| + return "the target's installation (P built)" |
| case 0, 1, 2, 3: |
| w.pass() |
| return "pass" |
| @@ -9297,6 +9678,21 @@ func TestRelocationControllerLifecycleProperties(t *testing.T) { |
| defer w.recoverAll() |
| w.startRelocation() |
| var trace []string |
| + if seed%2 == 0 { |
| + // Every other seed starts as the target with its local |
| + // install journaled (rev 15), so the random events fall on |
| + // an installation that a lost lease's cleanup can retire |
| + // before P's first admission. |
| + w.complete() |
| + w.startRelocation() // dst back to src: this box is the target |
| + w.pass() |
| + w.install() |
| + w.pass() |
| + trace = append(trace, "target, installed, local install journaled") |
| + if w.current().Op.Journal["local_install"] == "" { |
| + t.Fatalf("seed %d: the local install was not journaled: %+v", seed, w.current().Op) |
| + } |
| + } |
| for i := 0; i < 30; i++ { |
| ev := w.event(r) |
| trace = append(trace, ev) |
| @@ -9357,6 +9753,28 @@ func TestRelocationControllerLifecycleProperties(t *testing.T) { |
| } |
| ``` |
| +`internal/hostexec/readlocal.go` (rev 15: hostexec's validated reader of its own local record, for `recordIsMine`): |
| + |
| +```go |
| +package hostexec |
| + |
| +// ReadLocal reads host's durable local record (<dataDir>/.hostexec/<host>.json) |
| +// with the validation Open, CheckLocal and ClearStale apply to it: a |
| +// regular file, a complete binding (box, enrollment, host, incarnation, |
| +// op, boot and controller ids), a known state. An absent record is |
| +// os.ErrNotExist; anything it would refuse is an error, never a State. |
| +// It takes no lock and grants nothing: a caller deciding whether a record |
| +// is still its own uses it, and keeps its own state on any error. |
| +func ReadLocal(dataDir, host string) (State, error) { |
| + d, err := openLocal(dataDir) |
| + if err != nil { |
| + return State{}, err |
| + } |
| + defer d.close() |
| + return d.read(host) |
| +} |
| +``` |
| + |
| `internal/host/relocationeffects.go`: |
| ```go |
| @@ -9664,25 +10082,23 @@ func (r *RelocationControllers) finish(ctx context.Context, host, op string, suc |
| return r.forget(host, op) |
| } |
| -// recordIsMine reads hostexec's durable record for host and reports |
| -// whether it is b's binding (box, enrollment, host, incarnation, |
| -// operation: what ClearStale matches). An absent record is not b's. A |
| -// record that cannot be read or decoded is an error: never a reason to |
| -// forget. |
| +// recordIsMine reads hostexec's durable record for host through |
| +// hostexec's own validated reader (ReadLocal) and reports whether it is |
| +// b's binding (box, enrollment, host, incarnation, operation: what |
| +// ClearStale matches). Only an absent record, or a valid record of another |
| +// binding, is "not mine"; a record hostexec would refuse (one missing its |
| +// enrollment or incarnation, say) is an error, and the participation is |
| +// kept (round 14). |
| func (r *RelocationControllers) recordIsMine(host string, b execctl.Binding) (bool, error) { |
| if r.DataDir == "" { |
| return false, fmt.Errorf("%w: no hostexec data directory to verify the record", ErrClaimStopUnavailable) |
| } |
| - raw, err := os.ReadFile(filepath.Join(r.DataDir, ".hostexec", host+".json")) |
| + st, err := execctl.ReadLocal(r.DataDir, host) |
| if errors.Is(err, os.ErrNotExist) { |
| return false, nil |
| } |
| if err != nil { |
| - return false, err |
| - } |
| - var st execctl.State |
| - if err := json.Unmarshal(raw, &st); err != nil || st.V != 1 || st.Host != host || st.Box == "" || st.Op == "" { |
| - return false, fmt.Errorf("%w: hostexec record for %s is not readable", execctl.ErrState, host) |
| + return false, fmt.Errorf("hostexec record for %s: %w", host, err) |
| } |
| return st.Box == b.Box && st.Enrollment == b.Enrollment && st.Incarnation == b.Incarnation && st.Op == b.Op, nil |
| } |
| @@ -9866,7 +10282,12 @@ type RelocationEffects struct { |
| // and one cold probe admission followed by the probe. |
| BuildP func(context.Context, hostclaim.Observed) error |
| Install func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) |
| - Admit func(context.Context, hostclaim.Observed, string) (uint64, bool, error) |
| + // VerifyLocal reports whether the local install the receipt names is |
| + // complete on this box now: the files directory as P's prepared |
| + // manifest lists it, the env file and the wake state, under the |
| + // receipt's install_stamp (Task 8c's production body). |
| + VerifyLocal func(context.Context, hostclaim.Observed, hostclaim.LocalInstall) (bool, error) |
| + Admit func(context.Context, hostclaim.Observed, string) (uint64, bool, error) |
| } |
| var _ hostrelocate.Effects = (*RelocationEffects)(nil) |
| @@ -10130,6 +10551,39 @@ func (e *RelocationEffects) InstallLocal(ctx context.Context, obs hostclaim.Obse |
| return e.Install(ctx, obs) |
| } |
| +// EnsureLocal is hostrelocate.Effects' check before P's first admission: |
| +// the journaled receipt's install is verified and, if a lost lease's scoped |
| +// cleanup retired it, restored by the same Install provider under the same |
| +// install_stamp, then verified again. An opened P is never reinstalled. |
| +func (e *RelocationEffects) EnsureLocal(ctx context.Context, obs hostclaim.Observed) error { |
| + if e.VerifyLocal == nil || e.Install == nil || obs.Claim == nil { |
| + return ErrClaimStopUnavailable |
| + } |
| + c := obs.Claim |
| + if hostclaim.PreparedOpened(c) { |
| + return nil |
| + } |
| + var l hostclaim.LocalInstall |
| + if err := json.Unmarshal([]byte(c.Op.Journal["local_install"]), &l); err != nil || l.InstallStamp == 0 { |
| + return fmt.Errorf("%w: no local install receipt to verify", ErrClaimStopReceipt) |
| + } |
| + if ok, err := e.VerifyLocal(ctx, obs, l); err != nil || ok { |
| + return err |
| + } |
| + got, err := e.Install(ctx, obs) |
| + if err != nil { |
| + return err |
| + } |
| + if got.Box != l.Box || got.Enrollment != l.Enrollment || got.InstallStamp != l.InstallStamp { |
| + return fmt.Errorf("%w: the restored install is not the receipt's", ErrClaimStopReceipt) |
| + } |
| + ok, err := e.VerifyLocal(ctx, obs, l) |
| + if err == nil && !ok { |
| + err = fmt.Errorf("%w: the restored install did not verify", ErrClaimStopReceipt) |
| + } |
| + return err |
| +} |
| + |
| func (e *RelocationEffects) Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (uint64, bool, error) { |
| if e.Admit == nil { |
| return 0, false, ErrClaimStopUnavailable |
| @@ -10170,7 +10624,7 @@ Expected: PASS. |
| - [ ] **Step 6: Commit** |
| ```bash |
| -git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationcontrollers_linux_test.go internal/host/relocationlifecycle_linux_test.go internal/host/relocationflush.go internal/hostfiles/platform.go |
| +git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationcontrollers_linux_test.go internal/host/relocationlifecycle_linux_test.go internal/hostexec/readlocal.go internal/host/relocationflush.go internal/hostfiles/platform.go |
| git commit -m "host: a release is proved by its full stop receipt; it stops the writers, flushes files, wake state and env, records its positions and only then drains |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| @@ -10547,6 +11001,9 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| | Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | |
| | Every stage A provider a test stubs is tested with its production value once wired: the worker's `Choose` and `Eligible` (`ExecutorChooser` adapters), `Leases` (`*lease.Held`), `Prepared` (`PreparedFromInstallManifest`), `StorePinner.SelectDB` (`genselect.Select` over the versioned data bucket), and `RelocationEffects`' providers; replacing any with a stub that returns nothing must fail a test | 10 (scan) | Task 10's adapters; Tasks 8a to 8c | a test per provider over real evidence (genselect's versioned-bucket fixture for the pinner) | |
| | A production probe launches only through the reservation-bearing isolated root launch protocol. It succeeds through it, and without it is refused at `Preflight` before any attempt is counted and before any admission. The passes stay off until the protocol exists. | 12 (2) | R10; Task 8c's gate; Task 10's activation gate | `TestRelocationProbeLaunchesThroughTheIsolatedRootProtocol` (the probe's `"probe"` work launched under the reservation, admitted and healthy, under the edge unit's restrictions); `TestRelocationProbeRefusesWithoutTheIsolatedRootProtocol` (`ErrClaimIsolationUnavailable`, no `BeginProbeAttempt`, no admission, no write); `TestRelocatePassStaysOffWithoutTheIsolatedRootProtocol` | |
| +| P's prepared files manifest is create-only at sequence 0, bound to `{location, P, op, install_stamp}`; the verifier (`PreparedFromInstallManifest`) reports `FilesSeq` 0, `SourceFilesSeq` the pin's, and the digest it read; the first admission takes that exact manifest, by digest, as the publisher's baseline, and the session's first publication is sequence 1 | 14 (1) | Task 5's `CopyPinnedManifest`, Task 1's `ToInstalled`; R1 (rev 15) | `TestPreparedFromInstallManifestReportsSequenceZero`, `TestRelocationProbeAdmissionConsumesThePreparedManifestByDigest` (a digest mismatch refuses the admission; the first commit is sequence 1) | |
| +| Before P's first admission, `EnsureLocal`'s production body: `VerifyLocal` checks the receipt's files (against P's prepared manifest), env file and wake state under its `install_stamp`; a retired or partial install is restored by the same `Install` under the same stamp, then verified; an opened P is never reinstalled | 14 (2) | Task 7's `EnsureLocal`; Task 8c's `VerifyLocal` and `Install` | `TestRelocationLocalInstallIsRestoredAfterALostLeaseBeforeTheProbe` (journal the receipt, lose the lease, scoped cleanup retires the install, recover, the probe sees a complete install); `TestRelocationOpenedPIsNeverReinstalled` | |
| +| A source's ordinary cold wake in `starting` and `releasing` dispatches through the wake inputs while its admission fence is open; refused once the fence closes or the claim no longer names it; a target's ordinary wake stays refused | 14 (3) | R1 (rev 15) | `TestRelocationSourceWakesInStarting`, `TestRelocationSourceWakesInReleasing`, `TestRelocationSourceWakeRefusedAfterItsFenceCloses`, `TestRelocationSourceWakeRefusedWithoutAuthority`, `TestRelocationTargetOrdinaryWakeStaysRefused` | |
| | The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| @@ -12367,3 +12824,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 12 guards**, each removed in turn and its test watched to fail: the pin's referenced origins (the copy refuses g1's object with `ErrOrigin`); the refusal of a reference outside the incarnation's files (a foreign reference is pinned); `preparedOpened` reading `prepared_opened` (P is forgotten with its admission); the opening admission journaling it. Four in all. |
| - **Rev 13 guards**, each removed in turn and its test watched to fail: `Acknowledge`'s handover (B's acknowledgement waits out the pass); `openBounded` (the open under the agent's lifetime never returns: the test's own timeout); the superseded case for hostexec's `ErrInactive` (the handover fails); removing the superseded journal entry once B opens (A is still retained). Four in all. |
| - **Rev 14 guards**, removed one at a time under `TestRelocationControllerLifecycleProperties` with 200 seeds (the table in the Rev 14 dispositions): ten killed, each with its seed and invariant. One, the extra scoped cleanup before a recovery reopen, survives as defence in depth, because `hostexec.Open` runs the same cleanup and refuses on its failure. Rev 13's guard "removing the superseded journal entry once B opens" is gone with that code (C3). |
| +- **Rev 15 guards**, removed one at a time (the table in the Rev 15 dispositions): twelve killed. The survivors are defence in depth: `EnsureLocal` before entering probing is covered by the attempt-time check, and `EnsureLocal`'s own short-circuit for an opened P is covered because the participant never calls it after P opens. |