amadan
docs: region HA plan 1b-next rev 13 hands a host's controller to its successor and gates probes on root's isolated launch
Astra round 12, two P2s, both accepted. Acknowledge now retires this box's earlier participations of the host before it opens a successor's controller. ClearStale's ErrInactive, raised while the successor lists this box, counts as superseded; the successor's open replaces that record. The wait for the host lock is bounded by the pass. A real-controller test starts B before A's sweep. Request R10 asks the claim session for the reservation-bearing isolated root launch protocol that 6b's Preflight requires; Task 8c is gated on it, Task 10 gains an activation gate, and the stage B re-plan names the probe tests. Dry-run in task order on 3407d114; 4 guards removed in turn, each failing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+258
−8
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+258 −8
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index 1ed12dd19..cfccb5e44 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,26 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 12) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 13) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 11 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 12 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 13: round 12 dispositions |
| + |
| +Astra's round 12 (`CODEX-PLAN-REGION-HA-1BN-R12.md`) resolved the file-origin finding and raised two P2s. Both are accepted; none is declined. Finding 1 is stage A code. It was dry-run on a clean 3407d114 with stage A applied in task order, building and running each task's tests after it, plus vet for darwin and windows and the cross-builds. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). Finding 2 is a stage B contract and has no code. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| 1 (P2) | A | Rev 12's `Acknowledge` looked controllers up by host and operation, and opened a missing one under `e.Lifetime`. `hostexec.Open` loops on `ErrControllerBusy` until its context ends (controller.go, `Open`), and a predecessor's controller keeps the host's lifetime lock until `Close`. `ClearStale` refuses with `ErrInactive` whenever the running operation lists this box, whatever its id (controller.go:643). The regression reproduces it: without the handover, B's acknowledgement waits out the whole pass. | accepted | Before opening a successor's controller, `Acknowledge` calls `RelocationControllers.handover`. It retires every other participation of the same host under the pass's context: close, then the repeated scoped cleanup, then the clear. A clear refused with `ErrInactive` because the successor lists this box counts as superseded. The lock is free by then, and the successor's `Open` replaces the record under it; the predecessor's journal entry is removed only once that open succeeds. A clear that finds another binding's record (`ErrState`) means this participation has no record left. The open itself is bounded by `openBounded`: the controller still lives under the agent's lifetime, but waits for the lock no longer than the pass. A controller that opens in the same instant the pass ends is closed, never kept with a cancelled life. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController`, on real hostexec controllers: A acknowledged; A completes and B starts before A's sweep. B's acknowledgement succeeds, only B is retained, and B's record blocks updates. A lock held by a controller this box does not retain is waited for no longer than the pass, and nothing is kept.) | |
| +| 2 (P2) | B | 6b rev 5: `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable`. `runtime.Prerequisites` calls `Roles.Preflight` before dispatch. The "External completion gates" section requires "the reservation-bearing extension of main's isolated root launch protocol … before any tenant shape can run with claims enabled", which "is not supplied by the external owner adapters". | accepted (contract) | Request **R10** asks the claim session and the activator's owner for that protocol, with a relocation probe's `"probe"` work admitted through it. Task 8c is now gated on R10. Task 10 gains an activation gate: the passes stay dormant and refuse while the launcher lacks the protocol. The stage B re-plan names the production probe tests: success through the protocol, refusal without it before any attempt is counted, and the pass staying off. | Requests (R10), Task 8c, Task 10, stage B re-plan | |
| + |
| +Astra's readings of rulings 1 to 24 stand. |
| + |
| ## Rev 12: round 11 dispositions |
| Astra's round 11 (`CODEX-PLAN-REGION-HA-1BN-R11.md`) resolved the chooser-test finding and raised one P2. It is accepted; none is declined. |
| @@ -209,6 +220,7 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | |
| | R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | |
| | R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | |
| +| R10 | claim 6b (and the activator's owner) | The reservation-bearing extension of main's isolated root launch protocol, which 6b's own external completion gate names as required "before any tenant shape can run with claims enabled" and as "not supplied by the external owner adapters" (6b rev 5, "External completion gates"). Until it exists, `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable` for every launch (6b, `Preflight`), and the runtime's `Prerequisites` calls `Roles.Preflight` before any dispatch (6b, `runtime.Prerequisites`). Even with R1 every production probe would therefore be refused. The ask: the protocol binds each supported role launch to the durable claim reservation and preserves main's namespace isolation; `Preflight` admits a relocation probe's `"probe"` work through it, bound to `op.exec`, the operation and `install_stamp`; without the protocol, `Preflight` refuses before any probe attempt is counted. | Task 8c, Task 10 | |
| | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | |
| ## Rulings made while planning |
| @@ -266,7 +278,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four and rev 12's four (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four and rev 13's four (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -7449,7 +7461,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 7: Release evidence on the box (`internal/host`) |
| -`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. *Rev 8:* a retry by the same controller, after root failed between the drain and its durable receipt, resumes at the producers' idempotent drain with the recorded positions and never flushes again. *Rev 9:* whenever positions are recorded for the stop, the request keeps the recorded directory revision, even if the roster has moved on. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). |
| +`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. *Rev 8:* a retry by the same controller, after root failed between the drain and its durable receipt, resumes at the producers' idempotent drain with the recorded positions and never flushes again. *Rev 9:* whenever positions are recorded for the stop, the request keeps the recorded directory revision, even if the roster has moved on. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). *Rev 13:* a successor's controller opens only after this box's earlier participations of the host are retired (`handover`), and its wait for the host's lifetime lock is bounded by the pass (`openBounded`). |
| **Files:** |
| - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` |
| @@ -8611,6 +8623,133 @@ func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { |
| t.Fatalf("retained: %v", kept(restarted)) |
| } |
| } |
| + |
| +// Round 12 finding 1: relocation A completes and B starts before this box |
| +// sweeps A. A's real controller still holds the host's lifetime lock; |
| +// Acknowledge for B retires it first (hostexec refuses to clear A's record |
| +// while B lists this box, so B's Open replaces it), under the pass's |
| +// bounded context, and B's controller opens. A lock held by a controller |
| +// this box does not retain is waited for no longer than the pass. |
| +func TestASuccessorRelocationRetiresItsPredecessorsController(t *testing.T) { |
| + ctx := context.Background() |
| + dataDir := t.TempDir() |
| + for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { |
| + if err := os.Mkdir(filepath.Join(dataDir, name), 0o770); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + store := release.NewDirStore(t.TempDir()) |
| + const host = "shop.bab.oncarlos.com" |
| + created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| + nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| + body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) |
| + obs, err := hostclaim.Read(ctx, store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + step := func(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { |
| + t.Helper() |
| + next, err := hostclaim.Update(ctx, store, o, tr) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return next |
| + } |
| + relocating := func(id, from, to string) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: id, Phase: hostclaim.PhaseStarting, Exec: from, Participants: []string{from, to}, |
| + Intent: map[string]string{"region": "ie", "source_box": from, "target_box": to}} |
| + return nil |
| + } |
| + } |
| + obs = step(obs, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("a", 32)})) |
| + obs = step(obs, hostclaim.CompleteCreate("c1", "src")) |
| + obs = step(obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| + obs = step(obs, relocating("rel-1", "src", "dst")) |
| + cfg := execctl.Config{Store: store, Guard: hostguard.Guard{DataDir: dataDir, LockDir: filepath.Join(dataDir, "locks"), Claims: hostguard.StoreReader{Store: store}}, |
| + Directory: activeDirectory{}, Clock: execctl.SystemClock{}, LockDir: filepath.Join(dataDir, "exec-locks"), |
| + CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: func(context.Context) error { return nil }} |
| + lifetime, stop := context.WithCancel(ctx) |
| + t.Cleanup(stop) |
| + e, _, _, _ := effectsFixture(t) |
| + e.Lifetime = lifetime |
| + e.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "participations"), |
| + Cleanup: func(context.Context, execctl.Binding) error { return nil }, |
| + Clear: func(ctx context.Context, b execctl.Binding) error { |
| + c := cfg |
| + c.Binding = b |
| + return execctl.ClearStale(ctx, c) |
| + }} |
| + e.Identity = func(context.Context) (RelocationIdentity, error) { |
| + return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil |
| + } |
| + e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { |
| + c := cfg |
| + c.Binding = b |
| + return OpenRelocationController(ctx, c) |
| + } |
| + if _, err := e.Acknowledge(ctx, obs); err != nil { |
| + t.Fatal(err) |
| + } |
| + // A completes on dst, and B (dst back to src) starts before this box's |
| + // sweep has finished A. |
| + obs = step(obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}} |
| + return nil |
| + }) |
| + obs = step(obs, relocating("rel-2", "dst", "src")) |
| + pass, cancel := context.WithTimeout(ctx, 10*time.Second) |
| + defer cancel() |
| + ack, err := e.Acknowledge(pass, obs) |
| + if err != nil { |
| + t.Fatalf("the successor's acknowledgement: %v", err) |
| + } |
| + if ack.Op != "rel-2" { |
| + t.Fatalf("ack: %+v", ack) |
| + } |
| + got, err := e.Retained() |
| + if err != nil || len(got) != 1 || got[0].Op != "rel-2" { |
| + t.Fatalf("retained after the handover: %+v %v", got, err) |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil { |
| + t.Fatal("B's record did not block an update") |
| + } |
| + |
| + // A lock held by a controller this box does not retain (here, another |
| + // operation's opened directly): the next open waits no longer than the |
| + // pass, and opens nothing. |
| + if err := e.Controllers.Close(ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + obs = step(obs, relocating("rel-3", "dst", "src")) |
| + foreign := cfg |
| + foreign.Binding = execctl.Binding{Box: "src", Enrollment: strings.Repeat("a", 32), Host: host, Incarnation: obs.Claim.Holder.Incarnation, Op: "rel-3", BootID: "boot"} |
| + held, err := execctl.Open(lifetime, foreign) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer held.Close(ctx) |
| + other := *e |
| + other.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "other"), Cleanup: func(context.Context, execctl.Binding) error { return nil }} |
| + short, cancelShort := context.WithTimeout(ctx, 300*time.Millisecond) |
| + defer cancelShort() |
| + began := time.Now() |
| + if _, err := other.Acknowledge(short, obs); err == nil { |
| + t.Fatal("opened while another controller held the host") |
| + } |
| + if waited := time.Since(began); waited > 5*time.Second { |
| + t.Fatalf("the open waited %v, past the pass's bound", waited) |
| + } |
| + if got, _ := other.Controllers.Retained(); len(got) != 1 || got[0].Op != "rel-3" { |
| + t.Fatalf("the bounded open retained a controller: %+v", got) // only the journal entry |
| + } |
| + other.Controllers.mu.Lock() |
| + n := len(other.Controllers.open) |
| + other.Controllers.mu.Unlock() |
| + if n != 0 { |
| + t.Fatal("a controller was kept from a cancelled open") |
| + } |
| +} |
| ``` |
| `internal/host/relocationeffects.go`: |
| @@ -8824,6 +8963,18 @@ func (r *RelocationControllers) Retained() ([]hostrelocate.Participation, error) |
| // journal entry goes only once the record is gone, so a restart in |
| // between retries the cleanup and the clear through the recorded binding. |
| func (r *RelocationControllers) Finish(ctx context.Context, host, op string) error { |
| + return r.finish(ctx, host, op, false) |
| +} |
| + |
| +// errSuperseded is finish's answer, for a successor's handover, when the |
| +// predecessor's controller is closed and cleaned up but hostexec refuses |
| +// to clear its record because the claim's running operation lists this |
| +// box (hostexec.ClearStale, controller.go: "a fresh successor operation |
| +// may exclude this box"; here it includes it). The lifetime lock is free, |
| +// and the successor's Open replaces the record under it. |
| +var errSuperseded = errors.New("relocation: the predecessor's record passes to its successor") |
| + |
| +func (r *RelocationControllers) finish(ctx context.Context, host, op string, successor bool) error { |
| key := controllerKey(host, op) |
| r.mu.Lock() |
| c, ok := r.open[key] |
| @@ -8858,7 +9009,16 @@ func (r *RelocationControllers) Finish(ctx context.Context, host, op string) err |
| if ok { |
| clear = func() error { return c.Clear(ctx) } |
| } |
| - if err := clear(); err != nil { |
| + superseded := false |
| + switch err := clear(); { |
| + case err == nil: |
| + case errors.Is(err, execctl.ErrState): |
| + // The host's record names another binding (a successor opened |
| + // over it, or another operation's): this participation has no |
| + // record left to clear. |
| + case successor && errors.Is(err, execctl.ErrInactive): |
| + superseded = true // the lock is free; the journal stays until the successor opens |
| + default: |
| return err |
| } |
| r.mu.Lock() |
| @@ -8867,6 +9027,14 @@ func (r *RelocationControllers) Finish(ctx context.Context, host, op string) err |
| delete(r.bindings, key) |
| } |
| r.mu.Unlock() |
| + if superseded { |
| + return errSuperseded |
| + } |
| + return r.forget(host, op) |
| +} |
| + |
| +// forget removes one participation's journal entry. |
| +func (r *RelocationControllers) forget(host, op string) error { |
| if r.Journal != "" { |
| if err := os.Remove(r.journalPath(host, op)); err != nil && !errors.Is(err, os.ErrNotExist) { |
| return err |
| @@ -8875,6 +9043,33 @@ func (r *RelocationControllers) Finish(ctx context.Context, host, op string) err |
| return nil |
| } |
| +// handover retires every other participation of host this box still holds |
| +// before op's controller opens (round 12 finding 1): hostexec keeps one |
| +// controller per host, and a predecessor whose sweep has not yet run still |
| +// holds the host's lifetime lock, so the successor's Open would wait on it |
| +// for as long as the agent lives. Each predecessor is closed and its |
| +// scoped cleanup repeated under the pass's context (bounded); one whose |
| +// record hostexec will not clear because the successor lists this box is |
| +// returned as superseded, its journal entry removed only once the |
| +// successor's controller has opened over its record. |
| +func (r *RelocationControllers) handover(ctx context.Context, host, op string) ([]string, error) { |
| + retained, _ := r.Retained() // a damaged entry is Sweep's to report |
| + var superseded []string |
| + for _, p := range retained { |
| + if p.Host != host || p.Op == op { |
| + continue |
| + } |
| + switch err := r.finish(ctx, host, p.Op, true); { |
| + case err == nil: |
| + case errors.Is(err, errSuperseded): |
| + superseded = append(superseded, p.Op) |
| + default: |
| + return nil, fmt.Errorf("relocation: retiring %s's controller for %s before %s: %w", p.Op, host, op, err) |
| + } |
| + } |
| + return superseded, nil |
| +} |
| + |
| // writeFileSynced writes path whole: a synced temporary file renamed into |
| // place, then the directory synced, so a committed entry survives a crash. |
| func writeFileSynced(path string, body []byte, mode os.FileMode) error { |
| @@ -9008,13 +9203,25 @@ func (e *RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Obser |
| if err := e.Controllers.record(c.Host, c.Op.ID, b); err != nil { |
| return hostclaim.ExecutionAcknowledgement{}, err |
| } |
| - opened, err := e.Controller(e.Lifetime, b) |
| + // A predecessor this box has not swept yet still holds the host's |
| + // lifetime lock: retire it first, under this pass's context. |
| + superseded, err := e.Controllers.handover(ctx, c.Host, c.Op.ID) |
| + if err != nil { |
| + return hostclaim.ExecutionAcknowledgement{}, err |
| + } |
| + opened, err := openBounded(ctx, e.Lifetime, func(octx context.Context) (RelocationController, error) { return e.Controller(octx, b) }) |
| if err != nil { |
| return hostclaim.ExecutionAcknowledgement{}, err |
| } |
| if ctl, ok = e.Controllers.put(c.Host, c.Op.ID, b, opened); !ok { |
| _ = opened.Close(ctx) // another pass opened one first |
| } |
| + for _, old := range superseded { |
| + // The successor's Open replaced the predecessor's record. |
| + if err := e.Controllers.forget(c.Host, old); err != nil { |
| + return hostclaim.ExecutionAcknowledgement{}, err |
| + } |
| + } |
| } |
| st, err := ctl.Ready(ctx) |
| if err != nil { |
| @@ -9023,6 +9230,47 @@ func (e *RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Obser |
| return hostclaim.ExecutionAcknowledgement{Op: c.Op.ID, Enrollment: st.Enrollment, BootID: st.BootID, ControllerID: st.ControllerID, LeaseETag: st.LeaseETag, Expires: st.Expires}, nil |
| } |
| +// openBounded opens a controller that lives under lifetime (claim D6) but |
| +// waits for the host's lifetime lock no longer than the pass's ctx: |
| +// hostexec.Open takes one context for both. If ctx ends first, the open is |
| +// cancelled (Open returns, releasing what it took); a controller that |
| +// opened in the same instant is closed, never kept with a cancelled life. |
| +func openBounded(ctx, lifetime context.Context, open func(context.Context) (RelocationController, error)) (RelocationController, error) { |
| + octx, cancel := context.WithCancel(lifetime) |
| + var mu sync.Mutex |
| + opened, cancelled := false, false |
| + done := make(chan struct{}) |
| + go func() { |
| + select { |
| + case <-ctx.Done(): |
| + mu.Lock() |
| + if !opened { |
| + cancelled = true |
| + cancel() |
| + } |
| + mu.Unlock() |
| + case <-done: |
| + } |
| + }() |
| + c, err := open(octx) |
| + mu.Lock() |
| + opened = true |
| + wasCancelled := cancelled |
| + mu.Unlock() |
| + close(done) |
| + if wasCancelled { |
| + if c != nil { |
| + _ = c.Close(context.Background()) |
| + } |
| + return nil, errors.Join(ctx.Err(), err) |
| + } |
| + if err != nil { |
| + cancel() |
| + return nil, err |
| + } |
| + return c, nil // octx is cancelled only with lifetime |
| +} |
| + |
| // Ready checks the retained controller for the operation is active now. |
| func (e *RelocationEffects) Ready(ctx context.Context, obs hostclaim.Observed) error { |
| if e.Controllers == nil || obs.Claim == nil { |
| @@ -9612,6 +9860,7 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| |---|---|---|---| |
| | Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | |
| | Every stage A provider a test stubs is tested with its production value once wired: the worker's `Choose` and `Eligible` (`ExecutorChooser` adapters), `Leases` (`*lease.Held`), `Prepared` (`PreparedFromInstallManifest`), `StorePinner.SelectDB` (`genselect.Select` over the versioned data bucket), and `RelocationEffects`' providers; replacing any with a stub that returns nothing must fail a test | 10 (scan) | Task 10's adapters; Tasks 8a to 8c | a test per provider over real evidence (genselect's versioned-bucket fixture for the pinner) | |
| +| A production probe launches only through the reservation-bearing isolated root launch protocol. It succeeds through it, and without it is refused at `Preflight` before any attempt is counted and before any admission. The passes stay off until the protocol exists. | 12 (2) | R10; Task 8c's gate; Task 10's activation gate | `TestRelocationProbeLaunchesThroughTheIsolatedRootProtocol` (the probe's `"probe"` work launched under the reservation, admitted and healthy, under the edge unit's restrictions); `TestRelocationProbeRefusesWithoutTheIsolatedRootProtocol` (`ErrClaimIsolationUnavailable`, no `BeginProbeAttempt`, no admission, no write); `TestRelocatePassStaysOffWithoutTheIsolatedRootProtocol` | |
| | The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| @@ -10345,7 +10594,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 8c: The local install and the probe |
| -**Gated on:** claim Plan 6b (`ClaimWakeInputs`, `ClaimLaunchInputs`, `activator.StartPolicy.Probe`, `ClaimWakeAssembly.Build`'s runtime) and request **R1** (6b's runtime dispatches `relocating/probing` to this provider instead of refusing every probe); moves Phase 1 Plan 1 Task 7b (`appconfig.RenderHeld(ctx, s, priv, dataDir, rt, op, box) (appconfig.HeldRender, error)`, `release.MaterializeHeld(ctx, s, optDir, rt, version) (*release.Manifest, error)`); Tasks 8a and 8b. Not compiled in the dry run. |
| +**Gated on:** claim Plan 6b (`ClaimWakeInputs`, `ClaimLaunchInputs`, `activator.StartPolicy.Probe`, `ClaimWakeAssembly.Build`'s runtime) and request **R1** (6b's runtime dispatches `relocating/probing` to this provider instead of refusing every probe); request **R10** (the reservation-bearing isolated root launch protocol, without which 6b's `Preflight` refuses every launch: rev 13); moves Phase 1 Plan 1 Task 7b (`appconfig.RenderHeld(ctx, s, priv, dataDir, rt, op, box) (appconfig.HeldRender, error)`, `release.MaterializeHeld(ctx, s, optDir, rt, version) (*release.Manifest, error)`); Tasks 8a and 8b. Not compiled in the dry run. |
| The local install is §5.8's "The local install (B1-2)", under the host guard with the stamp the `installing` CAS issued: files and env from P's manifest (origins: the pin's chain plus P), the wake state imported, `RenderHeld` and `MaterializeHeld` with current inputs, every step journaled locally, then the receipt. The probe is one cold admission bound to `op.id` and `probing`, through 6b's wake runtime with `StartPolicy{Probe: true}`, whose inputs this task supplies. |
| @@ -11184,7 +11433,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 10: Switch the passes on, behind `CARLOS_REGION_HA` |
| -**Gated on:** claim Plan 6b merged (Tasks 8a to 8c's effects, Task 9's rows); claim Plan 5's app gate merged (the `Admission` provider, ruling 4); and claim Plan 5's create door merged (`ProductionAuthority` as the console's create `Authority`). Not compiled in the dry run. |
| +**Gated on:** claim Plan 6b merged (Tasks 8a to 8c's effects, Task 9's rows); claim Plan 5's app gate merged (the `Admission` provider, ruling 4); and claim Plan 5's create door merged (`ProductionAuthority` as the console's create `Authority`); and request **R10**. *Activation gate (rev 13):* `applyRegionHA` turns nothing on, and `runRelocatePass` refuses with `activator.ErrClaimIsolationUnavailable`, while the assembly's launcher lacks R10's protocol. A box with `CARLOS_REGION_HA` on but no protocol logs the refusal once and stays dormant; it never starts a relocation whose probes cannot run. Not compiled in the dry run. |
| On a box with `CARLOS_HOST_CLAIM` and `CARLOS_REGION_HA` both on, one pass every 30 seconds, and at once on a control serial change: for every claim under `control/hosts/` whose operation is `relocating` in this box's region, run `Worker.Step` (any active worker may), and for every claim naming this box as a participant, `Participant.Step`. The pass never starts a relocation by itself (spec 3's placer does); `carlos ops host relocate` (later, with Paul's copy) and the spec 3 placer call `Worker.Start`. The participant's effects are one `*RelocationEffects` per agent, opened with the agent's lifetime, so a pass's context bounds the pass and nothing else; each pass ends with `Participant.Sweep`, which finishes the controllers of participations the claim no longer runs (ruling 20). On the console, create's `Authority.SelectExecutor` is the `ExecutorChooser` over console's live edges read, and only when `Config.RegionHA` is on (Task 2); off, `Executors` is nil and the create is today's. |
| @@ -11430,3 +11679,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 10 guard**, removed and its test watched to fail: `Attempt` creating with the selection it made (`TestCreateIsTodaysWithoutASelection`: the completion substitutes box-a). The in-order dry run is the check for finding 1. One in all. |
| - **Rev 11 guards**, each removed in turn and its test watched to fail: the console chooser's live-edges reader (an empty snapshot fails on the box identities); its claim-loads reader (an empty map fails on the counts); `ProductionAuthority.SelectExecutor` consulting its chooser (a fixed box-a fails); `regionHAEnabled` honouring the parsed value (any parseable value as on fails on `0`). Four in all. |
| - **Rev 12 guards**, each removed in turn and its test watched to fail: the pin's referenced origins (the copy refuses g1's object with `ErrOrigin`); the refusal of a reference outside the incarnation's files (a foreign reference is pinned); `preparedOpened` reading `prepared_opened` (P is forgotten with its admission); the opening admission journaling it. Four in all. |
| +- **Rev 13 guards**, each removed in turn and its test watched to fail: `Acknowledge`'s handover (B's acknowledgement waits out the pass); `openBounded` (the open under the agent's lifetime never returns: the test's own timeout); the superseded case for hostexec's `ErrInactive` (the handover fails); removing the superseded journal entry once B opens (A is still retained). Four in all. |