amadan
docs: region HA stage B plan rev 4.2: C10 never falls through to ordinary cleanup on a claim read error, with the restart test that fails only C10's re-read once
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+236
−13
docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md+236 −13
diff --git a/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md b/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md| index aa55528a6..43f9cc5ac 100644 |
| --- a/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md |
| +++ b/docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md |
| @@ -1,4 +1,4 @@ |
| -# Region HA 1b-next Stage B: Relocation Wired Into the Claim Assembly Implementation Plan (rev 4.1) |
| +# Region HA 1b-next Stage B: Relocation Wired Into the Claim Assembly Implementation Plan (rev 4.2) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| @@ -12,6 +12,207 @@ |
| --- |
| +## Rev 4.2: round 4 disposition |
| + |
| +Astra's round 4 (`CODEX-PLAN-REGION-HA-STAGE-B-R4.md`) closed every round 3 finding but one, held ruling 66, and accepted the post-render key check. It found one P1, verified against the code: `completedRelocationDischarge` returned `(false, nil)` on a failed claim read, so `cleanup` fell through to ordinary cleanup. **Accepted 1, changed 0, declined 0.** |
| + |
| +| # | Task | Finding | Disposition | Change | Test | |
| +|---|---|---|---|---|---| |
| +| 1 (P1) | 9 (C10) | After a restart between the serving CAS and the discharge, a failure of only C10's re-read fell through to ordinary cleanup. Its serving-shaped stop's `BeginFence` replaces the completed probe's active fence, and `BeginColdHandover` (hostguard/coldhandover.go:42) then refuses N6's receipt for good. | accepted | Ruling 70. A claim read error is `(true, err)`, and `cleanup` also stops on any error (`done \|\| err != nil`). This holds at all three call sites, because they share `cleanup`. The cleanup fails, nothing is stopped, and the next pass retries. | `TestRelocationRestartDischargeSurvivesAFailedClaimRead`, written and passing (below). It sets up a real guard with the probe's completed writer-session fence and receipt, and a serving claim journaling `completed_relocation`. It calls `cleanup` with the restart's current observation and persisted binding, with only C10's claim read failing once. It asserts the error is the read error, no stop was asked, and the probe receipt is unchanged. On retry it asserts discharge, N6's `Before` verifying, N6's prior being the probe receipt, and `BeginColdHandover` accepting the serving barrier, which a replacement fence refuses with `ErrFenceActive`. Mutation: reverting the fix (fall through on a read error) fails it. | |
| + |
| +70. **C10 never falls through on an error** (Astra round 4). A failed claim read in `completedRelocationDischarge` is `(true, err)`, and `productionClaimLocal.cleanup` returns on any error from it, so no ordinary cleanup, serving-shaped stop or `BeginFence` runs while the claim read fails. The participation stays denied, the row stays held (ruling 66), and the next pass retries. *Cost if wrong:* none. A pass is lost to a transient read error. |
| + |
| +**C10 on #236, added sentence:** |
| + |
| +> Any error while making this check, a failed claim read included, ends the cleanup with that error: it never falls through to main's ordinary cleanup, so no serving-shaped stop or `BeginFence` runs, and the next pass retries. |
| + |
| +**Compile record (rev 4.2).** The check ran in a throwaway copy of ba748dcc, with rev 4.1's code restored from this plan and `internal/instancekey` taken from the instance-key plan: |
| +- `go build ./...` is clean. |
| +- `go vet` is clean on host, hostrelocate and hostclaim. |
| +- darwin, windows and linux/arm64 builds of `./internal/...` are clean. |
| +- The new test passes, and so do the host participation, Plan 9, handover, census and claim-retire tests, run with the sandbox off for their unix sockets. |
| + |
| +`internal/host/relocationdischarge_restart_unix_test.go` (Task 9): |
| + |
| +```go |
| +//go:build unix |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "sync" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + execlease "github.com/carlosframework/platform/internal/hostexec" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// guardClaims is the guard's claim reader, set by the test. |
| +type guardClaims struct { |
| + mu sync.Mutex |
| + obs hostclaim.Observed |
| +} |
| + |
| +func (g *guardClaims) set(o hostclaim.Observed) { g.mu.Lock(); g.obs = o; g.mu.Unlock() } |
| +func (g *guardClaims) Read(context.Context, string) (hostclaim.Observed, error) { |
| + g.mu.Lock() |
| + defer g.mu.Unlock() |
| + return g.obs, nil |
| +} |
| + |
| +// failFirstClaimRead fails the first Get of the claim key only: C10's |
| +// re-read (Astra round 4). |
| +type failFirstClaimRead struct { |
| + release.Store |
| + key string |
| + failed bool |
| + mu sync.Mutex |
| +} |
| + |
| +var errClaimReadDown = errors.New("control store unreachable") |
| + |
| +func (s *failFirstClaimRead) Get(ctx context.Context, key string) ([]byte, string, error) { |
| + s.mu.Lock() |
| + first := key == s.key && !s.failed |
| + if first { |
| + s.failed = true |
| + } |
| + s.mu.Unlock() |
| + if first { |
| + return nil, "", errClaimReadDown |
| + } |
| + return s.Store.Get(ctx, key) |
| +} |
| + |
| +// Astra round 4 (rev 4.2, ruling 70): after a restart between the serving |
| +// CAS and the participation discharge, a failure of only C10's claim |
| +// re-read never falls through to ordinary cleanup. Cleanup fails with that |
| +// error, nothing is stopped, the completed probe's fence is not replaced; |
| +// on retry the participation is discharged by the binding and the probe's |
| +// receipt, and the first serving wake's cold handover (N6's prior, the |
| +// probe's receipt) is accepted by the guard. |
| +func TestRelocationRestartDischargeSurvivesAFailedClaimRead(t *testing.T) { |
| + ctx := t.Context() |
| + const host, box = "shop.example", "dst" |
| + created := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC) |
| + inc, err := hostclaim.Incarnation("acct", "app", host, created) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + holder := func(session, stamp uint64, gen string) *hostclaim.Holder { |
| + return &hostclaim.Holder{Family: "instance", Key: "apps/acct/app/instances/" + host + ".json", Account: "acct", App: "app", Incarnation: inc, CreatedAt: created, |
| + Location: "instances/" + host, Stamp: stamp, Session: session, Generation: gen, WriterBox: box, |
| + Admissions: []hostclaim.AdmissionRecord{{Session: 12, Box: box, Generation: "P1", Mode: "cold", Region: "ie"}}} |
| + } |
| + probing := hostclaim.Observed{Host: host, ETag: "e1", Claim: &hostclaim.Claim{V: 2, Gen: 30, Host: host, Holder: holder(12, 9, "P1"), |
| + Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseRelocatingProbing, Exec: box, Participants: []string{box}}}} |
| + reader := &guardClaims{obs: probing} |
| + dataDir := t.TempDir() |
| + if err := os.Mkdir(filepath.Join(dataDir, "hostguard"), 0o770); err != nil { |
| + t.Fatal(err) |
| + } |
| + guard := hostguard.Guard{DataDir: dataDir, LockDir: t.TempDir(), Claims: reader} |
| + if err := guard.Activate(ctx, host, box, nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The probe's writer-session stop, completed with root's receipt |
| + // (ruling 49), while the op was still relocating/probing. |
| + prior := hostguard.FenceTarget{FenceID: "fence-1", Region: "ie", Host: host, Incarnation: inc, Stamp: 9, Session: 12, Op: "rel-1", Phase: hostclaim.PhaseRelocatingProbing, |
| + ObligationID: hostguard.WriterStopObligation(12), Box: box, Enrollment: "enr", BootID: "boot", ControllerID: "ctl", ClaimETag: "e1", DirectoryRevision: "1"} |
| + if _, err := guard.BeginFence(ctx, prior); err != nil { |
| + t.Fatal(err) |
| + } |
| + at := time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC) |
| + receipt := mustMarshalStopReceipt(StopReceipt{V: 1, ID: "r1", Kind: StopReceiptStopped, Region: "ie", ClaimETag: "e1", ObligationID: prior.ObligationID, FenceID: "fence-1", Host: host, Incarnation: inc, |
| + InstallationStamp: 9, WriterSession: 12, OperationID: "rel-1", Phase: hostclaim.PhaseRelocatingProbing, Box: box, Enrollment: "enr", BootID: "boot", ControllerID: "ctl", DirectoryRevision: "1", EvidenceAt: at}) |
| + if err := guard.CompleteFence(ctx, prior, receipt, at, func(context.Context, hostguard.FenceTarget, []byte, time.Time) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + sum := sha256.Sum256(receipt) |
| + done, _ := json.Marshal(hostclaim.CompletedRelocation{Op: "rel-1", Stop: hostclaim.ProbeStop{Box: box, Session: 12, Receipt: "fence-1", ReceiptSHA256: hex.EncodeToString(sum[:])}, Stamp: 9, Phase: hostclaim.PhaseRelocatingProbing, Generation: "P1"}) |
| + serving := &hostclaim.Claim{V: 2, Gen: 31, Host: host, Holder: holder(12, 9, "P1"), |
| + Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: box, Participants: []string{box}, Journal: map[string]string{hostclaim.JournalCompleted: string(done)}}} |
| + raw, err := json.Marshal(serving) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + key, _ := hostclaim.Key(host) |
| + dir := release.NewDirStore(t.TempDir()) |
| + if _, err := dir.Put(ctx, key, raw, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if obs, err := hostclaim.Read(ctx, dir, host); err != nil || obs.Claim.Op.Kind != hostclaim.KindServing { |
| + t.Fatalf("the serving claim does not read back: %v", err) |
| + } |
| + reader.set(hostclaim.Observed{Host: host, ETag: "e2", Claim: serving}) |
| + store := &failFirstClaimRead{Store: dir, key: key} |
| + stops := 0 |
| + a := &ClaimAssembly{Claims: store, Guard: guard, Wake: &ClaimWakeAssembly{}, Launches: &hostbox.LaunchJournal{}, |
| + Reconcile: &hostbox.ReconcileAdapter{Census: &hostbox.EmptyInstallationCensus{}}, |
| + collectStop: func(context.Context, StopRequest, StopDrainer) (StopResult, error) { |
| + stops++ |
| + return StopResult{}, errors.New("a serving-shaped stop") |
| + }} |
| + p := &productionClaimLocal{assembly: a, slots: map[string]*claimParticipation{}} |
| + // The restart: clearUntracked hands cleanup the current serving |
| + // observation and the persisted .hostexec binding. |
| + binding := execlease.Binding{Box: box, Enrollment: "enr", Host: host, Incarnation: inc, Op: "rel-1", BootID: "boot-0"} |
| + current := hostclaim.Observed{Host: host, ETag: "e2", Claim: serving} |
| + |
| + if err := p.cleanup(ctx, current, binding); !errors.Is(err, errClaimReadDown) { |
| + t.Fatalf("a failed C10 read fell through: %v", err) |
| + } |
| + if stops != 0 { |
| + t.Fatal("ordinary cleanup asked a stop while the claim read was failing") |
| + } |
| + if got, err := guard.CompletedWriterStops(ctx, host, hostguard.Captured{Incarnation: inc, Stamp: 9}, "rel-1", box); err != nil || len(got) != 1 || got[0].Target != prior { |
| + t.Fatalf("the probe's receipt changed: %+v %v", got, err) |
| + } |
| + |
| + // The retry: storage recovered. |
| + if err := p.cleanup(ctx, current, binding); err != nil { |
| + t.Fatalf("the retry did not discharge: %v", err) |
| + } |
| + if stops != 0 { |
| + t.Fatal("the discharge asked a stop") |
| + } |
| + // The first serving wake: N6's Before verifies the receipt, stopping |
| + // nothing; after its cold admission (session and stamp 40), the |
| + // guard's handover barrier over the probe's receipt is accepted, |
| + // which a replacement fence would have refused (ErrFenceActive). |
| + h := relocationCompletedHandover{Guard: guard, Box: box} |
| + if err := h.Before(ctx, current); err != nil { |
| + t.Fatalf("N6 before admission: %v", err) |
| + } |
| + r, err := h.receipt(ctx, current) |
| + if err != nil || r.Target != prior { |
| + t.Fatalf("N6's prior is not the probe's receipt: %+v %v", r.Target, err) |
| + } |
| + admitted := *serving |
| + admitted.Holder = holder(40, 40, "g40") |
| + admitted.Holder.Admissions = append(admitted.Holder.Admissions, hostclaim.AdmissionRecord{Session: 40, Box: box, Generation: "g40", Mode: "cold", Region: "ie"}) |
| + reader.set(hostclaim.Observed{Host: host, ETag: "e3", Claim: &admitted}) |
| + barrier := prior |
| + barrier.FenceID, barrier.Stamp, barrier.Session, barrier.Op, barrier.Phase, barrier.ObligationID, barrier.ClaimETag = "handover-1", 40, 40, "", "", "", "e3" |
| + if err := guard.BeginColdHandover(ctx, prior, barrier, func(*hostclaim.Claim) error { return nil }); err != nil { |
| + t.Fatalf("the serving handover from the probe's receipt: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +--- |
| + |
| ## Rev 4.1: O2′ aligned to the instance-key plan rev 3 (3947e1d9) |
| The instance-key plan rev 3 is committed at 3947e1d9 (`docs/superpowers/plans/2026-10-10-instance-key-provision.md`, "O2' for stage B"). Its planner compiled and ran its O2′ against stage B at 919662d2: a 17-case evidence test plus every existing pinner test. Rev 4.1 adopts that O2′ exactly, and deletes everything in stage B that differed. |
| @@ -204,16 +405,27 @@ import ( |
| // receipt for it is in this box's guard journal and verifies, the session |
| // is stopped with root's evidence and the participation is discharged. |
| // Nothing is stopped again. It runs first in every cleanup path, the |
| -// slot's close and clearUntracked alike. done false falls through to |
| -// main's cleanup unchanged. Region HA never releases the row's hold or |
| -// the activator fence before this has succeeded (ruling 66), so no |
| -// serving admission can move the holder off the probe session first. |
| +// slot's close and clearUntracked alike. done false with no error falls |
| +// through to main's cleanup unchanged. Any error, a failed claim read |
| +// included, is (true, err), and the caller also stops on any error: the |
| +// cleanup fails, nothing is stopped, and the next pass retries (rev 4.2). |
| +// Region HA never releases the row's hold or the activator fence before |
| +// this has succeeded (ruling 66), so no serving admission can move the |
| +// holder off the probe session first. |
| func (a ClaimAssembly) completedRelocationDischarge(ctx context.Context, binding execlease.Binding) (done bool, err error) { |
| if binding.Host == "" || binding.Op == "" || binding.Box == "" { |
| return false, nil |
| } |
| now, err := hostclaim.Read(ctx, a.Claims, binding.Host) |
| - if err != nil || now.Absent || now.Claim == nil || now.Claim.Holder == nil { |
| + if err != nil { |
| + // Rev 4.2 (Astra round 4): a failed read never falls through to |
| + // ordinary cleanup, whose serving-shaped stop would replace the |
| + // completed probe's active fence (BeginFence) and make the first |
| + // serving wake's handover refuse for good. The cleanup fails and |
| + // is retried on the next pass, with nothing stopped. |
| + return true, err |
| + } |
| + if now.Absent || now.Claim == nil || now.Claim.Holder == nil { |
| return false, nil |
| } |
| h := relocationCompletedHandover{Guard: a.Guard, Box: binding.Box} |
| @@ -268,7 +480,7 @@ func (a ClaimAssembly) completedRelocationDischarge(ctx context.Context, binding |
| + // its exact completed probe evidence, before any ordinary cleanup, so |
| + // no serving-shaped stop ever replaces the probe's fence. The captured |
| + // observation is not consulted: after a restart it is the current one. |
| -+ if done, err := a.completedRelocationDischarge(ctx, binding); done { |
| ++ if done, err := a.completedRelocationDischarge(ctx, binding); done || err != nil { |
| + return err |
| + } |
| // Cleanup uses the journal and producer set directly. It must never call |
| @@ -758,16 +970,27 @@ import ( |
| // receipt for it is in this box's guard journal and verifies, the session |
| // is stopped with root's evidence and the participation is discharged. |
| // Nothing is stopped again. It runs first in every cleanup path, the |
| -// slot's close and clearUntracked alike. done false falls through to |
| -// main's cleanup unchanged. Region HA never releases the row's hold or |
| -// the activator fence before this has succeeded (ruling 66), so no |
| -// serving admission can move the holder off the probe session first. |
| +// slot's close and clearUntracked alike. done false with no error falls |
| +// through to main's cleanup unchanged. Any error, a failed claim read |
| +// included, is (true, err), and the caller also stops on any error: the |
| +// cleanup fails, nothing is stopped, and the next pass retries (rev 4.2). |
| +// Region HA never releases the row's hold or the activator fence before |
| +// this has succeeded (ruling 66), so no serving admission can move the |
| +// holder off the probe session first. |
| func (a ClaimAssembly) completedRelocationDischarge(ctx context.Context, binding execlease.Binding) (done bool, err error) { |
| if binding.Host == "" || binding.Op == "" || binding.Box == "" { |
| return false, nil |
| } |
| now, err := hostclaim.Read(ctx, a.Claims, binding.Host) |
| - if err != nil || now.Absent || now.Claim == nil || now.Claim.Holder == nil { |
| + if err != nil { |
| + // Rev 4.2 (Astra round 4): a failed read never falls through to |
| + // ordinary cleanup, whose serving-shaped stop would replace the |
| + // completed probe's active fence (BeginFence) and make the first |
| + // serving wake's handover refuse for good. The cleanup fails and |
| + // is retried on the next pass, with nothing stopped. |
| + return true, err |
| + } |
| + if now.Absent || now.Claim == nil || now.Claim.Holder == nil { |
| return false, nil |
| } |
| h := relocationCompletedHandover{Guard: a.Guard, Box: binding.Box} |
| @@ -822,7 +1045,7 @@ func (a ClaimAssembly) completedRelocationDischarge(ctx context.Context, binding |
| + // its exact completed probe evidence, before any ordinary cleanup, so |
| + // no serving-shaped stop ever replaces the probe's fence. The captured |
| + // observation is not consulted: after a restart it is the current one. |
| -+ if done, err := a.completedRelocationDischarge(ctx, binding); done { |
| ++ if done, err := a.completedRelocationDischarge(ctx, binding); done || err != nil { |
| + return err |
| + } |
| // Cleanup uses the journal and producer set directly. It must never call |