amadan
docs: region HA plan 1b-next rev 8 resumes an interrupted release at the producers' drain
Astra round 7 finding 1 (P2), accepted. Root can fail after the drain acknowledges and before its receipt is durable (claimstop.go). When that happens, a retry by the same controller with recorded positions now goes straight to the producers' idempotent drain and reuses those positions. It never flushes again. The new test drains, fails root before the receipt, retries without a restart, and asserts only the producers' drain and the receipt run. Dry-run on 3407d114; 2 guards removed in turn, each failing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+64
−4
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+64 −4
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index e07aec0ac..626e6ff85 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,25 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 7) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 8) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 6 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 7 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 8: round 7 dispositions |
| + |
| +Astra's round 7 (`CODEX-PLAN-REGION-HA-1BN-R7.md`) resolved the kcmp finding, confirmed that stage B's contracts are complete, and added one P2. We checked it against `internal/host/claimstop.go` and accepted it; none is declined. The fix is stage A code. It was dry-run again on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `internal/host` tests. Two new guards were removed in turn and each made the new test fail (listed in the Self-review). |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| 1 (P2) | A | In `ClaimStopCoordinator`, `DrainClaim` runs before `Kernel.VerifyStopped` and `Guard.CompleteFence` (claimstop.go:219 onward). If either fails, root has no durable receipt. On retry, `BeginFence` returns the same fence, `FenceReceiptFor` returns `ErrNoFenceReceipt`, and root asks for the drain again. Rev 7's `Release` read the recorded positions but still built a fresh `releaseDrain`, which repeated `StopWriters`, the observation and both flushes against a replicator its own producer drain had already closed. | accepted | When this controller has already recorded positions for the exact stop (same boot and controller, root without a receipt), `Release` marks its drain `resumed` and carries the recorded TXID and sequence. A resumed drain goes straight to the producers' own drain, which is idempotent, and leaves verification and the receipt to root. A restarted agent keeps the recovery path; a replayed receipt keeps pairing with the record. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the drain closes the producers, root fails before persisting its receipt, and the retry without a restart runs only `producers` and `receipt`, with no flush, record or writers stop, and returns 41 and 8) | |
| + |
| +Astra's readings of rulings 1 to 24 stand. |
| + |
| ## Rev 7: round 6 dispositions |
| Astra's round 6 (`CODEX-PLAN-REGION-HA-1BN-R6.md`, against `222ff1c6`) resolved all six of round 5's findings and added one. It was checked against the plan and the kernel before anything changed, and is accepted; none is declined. The fix is stage A code, dry-run again on a clean 3407d114 (build; vet for darwin, windows and linux/arm64; cross-builds; the stopevidence tests run five times against the real kernel), with 3 new guards removed in turn (listed in the Self-review). Stage B is unchanged: R3's root endpoint calls the same `WritersStopped`. |
| @@ -187,7 +197,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven and rev 7's three (listed in the Self-review). Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three and rev 8's two (listed in the Self-review). Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -6961,7 +6971,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 7: Release evidence on the box (`internal/host`) |
| -`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). |
| +`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. *Rev 8:* a retry by the same controller, after root failed between the drain and its durable receipt, resumes at the producers' idempotent drain with the recorded positions and never flushes again. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). |
| **Files:** |
| - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` |
| @@ -7286,6 +7296,33 @@ func TestRelocationReleaseReplaysOnlyRecordedPositions(t *testing.T) { |
| t.Fatalf("replay: %+v %v", r, err) |
| } |
| }) |
| + t.Run("root fails after the producers closed", func(t *testing.T) { |
| + // Round 7: the drain flushed, recorded and closed the producers; |
| + // root then failed before its receipt was durable (kernel |
| + // verification or CompleteFence). The retry, same controller, no |
| + // restart, resumes at the producers' drain with the recorded |
| + // positions: no step of the release runs again. |
| + e, obs, calls, _ := effectsFixture(t) |
| + collect := e.Collect |
| + e.Collect = func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { |
| + if _, err := d.DrainClaim(ctx, r, "fence"); err != nil { |
| + return StopResult{}, err |
| + } |
| + return StopResult{}, errors.New("verify stopped: cgroup not yet empty") // no receipt persisted |
| + } |
| + if _, err := e.Release(context.Background(), obs); err == nil { |
| + t.Fatal("a release without a receipt succeeded") |
| + } |
| + *calls = nil |
| + e.Collect = collect |
| + r, err := e.Release(context.Background(), obs) |
| + if err != nil || r.FinalTXID != 41 || r.ManifestSeq != 8 { |
| + t.Fatalf("retry: %+v %v", r, err) |
| + } |
| + if want := []string{"producers", "receipt"}; !slices.Equal(*calls, want) { |
| + t.Fatalf("the retry ran %v, want %v (no flush, no record, no writers stop again)", *calls, want) |
| + } |
| + }) |
| t.Run("after a restart the recorded identity asks root", func(t *testing.T) { |
| // The agent flushed and recorded under controller "ctl", root |
| // completed its receipt, the agent restarted. The new process asks |
| @@ -8572,6 +8609,15 @@ func (e *RelocationEffects) Release(ctx context.Context, obs hostclaim.Observed) |
| if drain.inner, err = e.Producers(ctx, obs); err != nil { |
| return hostclaim.Release{}, err |
| } |
| + if flushed { |
| + // This process already flushed and recorded this stop, and |
| + // root has no durable receipt yet (it failed after the drain |
| + // acknowledged, before CompleteFence: claimstop.go). Steps 1 |
| + // to 3 are done and the replicator may be closed: resume at |
| + // the producers' idempotent drain with the recorded positions, |
| + // and let root verify and persist the receipt. |
| + drain.resumed, drain.txid, drain.seq = true, txid, seq |
| + } |
| d = drain |
| } |
| result, err := e.Collect(ctx, req, d) |
| @@ -8599,6 +8645,10 @@ type releaseDrain struct { |
| txid uint64 |
| seq int64 |
| done bool |
| + // resumed: this stop's positions are already durable (an earlier |
| + // attempt by this controller flushed and recorded them), so the |
| + // drain goes straight to the producers' own drain. |
| + resumed bool |
| } |
| // RecoveryReleaseDrain refuses: a restarted agent's producers are not the |
| @@ -8611,6 +8661,14 @@ func (RecoveryReleaseDrain) DrainClaim(context.Context, StopRequest, string) (Dr |
| } |
| func (d *releaseDrain) DrainClaim(ctx context.Context, r StopRequest, fence string) (DrainAcknowledgement, error) { |
| + if d.resumed { |
| + ack, err := d.inner.DrainClaim(ctx, r, fence) |
| + if err != nil { |
| + return DrainAcknowledgement{}, err |
| + } |
| + d.done = true |
| + return ack, nil |
| + } |
| if err := d.e.StopWriters(ctx, d.obs); err != nil { |
| return DrainAcknowledgement{}, err |
| } |
| @@ -9079,6 +9137,7 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| | Obligation | Round | Covered by | Proved by | |
| |---|---|---|---| |
| | Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | |
| +| The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| | The relocation controller's cleanup scoped to its own partial work; `Clear` is `ClearStale` with its config | 3 (N5), 4 (5) | `relocationCleanup`, `OpenRelocationController` | `TestRelocationCleanupNeverStopsAnAdmittedWriter`, the on-disk eligibility test over the assembly | |
| @@ -10891,3 +10950,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 5 guards**, each removed in turn and its test watched to fail: the lock-free fence observer (re-locking deadlocks the bounded test); the `probe` action in `Policy.Check`; the HA gate on the obsolete-hold clearance; `Finish`'s durable clear; the journal's retry after a restart; the hold-only path for a fenced executor; the observer error keeping rows. Seven in all. |
| - **Rev 6 guards**, each removed in turn and its test watched to fail: `Finish`'s repeated scoped cleanup before every clear (the real hostexec test's sweep after a failed cleanup clears); the participation journaled before the open; the journal listing's `.tmp` exclusion; a damaged committed entry keeping the readable ones; `Sweep` reporting `Retained`'s error; the env file's staged deletion; `replaceTargetSlot`'s `park-local` refusal; the worker's `park-local` exclusion; the chooser's `park-local` exclusion; `fenceBox`'s regional reassignment; Task 9a's serving-only check with HA off. Eleven in all. |
| - **Rev 7 guards**, each removed in turn and its test watched to fail: the `KCMP_FILES` shared-table check (the real kernel's unshared thread with the copied descriptor passes as the leader's); the `KCMP_FILE` description check (the fixture's shared table with another description passes); an unavailable kcmp failing closed (ENOSYS and EPERM pass instead of `ErrIncomplete`). Three in all. |
| +- **Rev 8 guards**, each removed in turn and its test watched to fail: `Release` marking a same-controller retry with recorded positions as resumed; the resumed drain skipping straight to the producers' drain. Two in all. |