amadan
docs: region HA plan 1b-next rev 17 closes the two-box model's three gaps
F2: an empty files pin now needs positive evidence. Either no writer was ever admitted, or the release flush reported nothing. Otherwise the pin refuses with ErrFilesUnevidenced and the worker marks the operation stuck, so an install never mints a fresh instance.key over sealed data (ruling 25). F1: probe-failed with no eligible spare is stuck ReleaseDeadline after the target failed; it is never sent back to the source, whose park-local chore would retire the same host paths (ruling 26). F3: install journals every unopened P it replaces, and CompleteRelocation queues a park-generation for each (ruling 27). Dry-run in task order on 3407d114; 8 guards removed in turn, each failing; lifecycle property at 200 seeds passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+215
−6
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+215 −6
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index d1e392e37..eb2140b62 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,4 +1,4 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 16) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 17) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| @@ -10,6 +10,22 @@ |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 17: two-box model findings |
| + |
| +The joint model (`origin/region-ha-hostlife-two-box`, model plan rev 2 at 6d25c54e) reported three gaps in rev 16 that review rounds 1 to 16 missed. Each comes with a direct test (`TestFailoverBeforeTheFirstManifest`, `TestProbeFailedInATwoBoxRegion`, `TestAbandonedPreparedGenerationParked`). I checked each against this plan's code first, and all three reproduce. All are accepted; none is declined. |
| + |
| +Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed its 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| F2 (data loss) | A, B | `StorePinner.filesPin` pinned an empty files set whenever no manifest existed in the admitted chain, whatever the holder's writer history. An install from it starts with no `instance.key`, and the app mints a new one over data sealed with the old (model INV-4). | accepted (ruling 25) | `filesPin` refuses with `ErrFilesUnevidenced` unless no writer was ever admitted (session 0) or the journaled release flushed nothing (`manifest_seq` 0). `Worker.pinUnderLease` marks the operation `stuck` on that error. Stage B: 6b's cold admission writes its manifest copy before the app starts (§5.6), with a named test. | Task 5 (`TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter`, `TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerMarksAnUnevidencedFilesPinStuck`); stage B re-plan | |
| +| F1 (liveness) | A | In `probe-failed` with the release journaled, `Worker.Step` returned the chooser's error forever ("a later pass may find one"). With two boxes no spare exists, and `replaceTargetSlot` refuses the source. | accepted (ruling 26: stuck, not the source) | `ReleaseDeadline` after `probe_failed_since` with still no eligible box, the worker marks the operation `stuck`. Before that it waits as before. | Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare`: waiting before the deadline, `stuck` after it) | |
| +| F3 (leak) | A | `CompleteRelocation` queued `park-local` chores only. A P replaced while unopened (at `installing`, or a deadline `probe-failed` before any admission) stayed in `holder.location` for the at-rest sweep alone. | accepted (ruling 27) | `install` journals every unopened P it replaces in `abandoned_prepared`. The serving CAS queues a `park-generation` for each in the region (`Owner` `region:<r>`), deduplicated, never for an opened P. | Task 1 (`TestRelocationParksEveryAbandonedPreparedGeneration`: replaced at `installing`, completed on P2. P1 is parked and P2 is not.) | |
| + |
| +Guards removed, each failing its test: `install` journaling the abandoned P; the serving CAS queuing its park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; and waiting before the bound. That is eight. |
| + |
| +Checked and unchanged: a target terminated with no spare in `starting` or `releasing` also waits for a spare. The live source still serves there (it is `op.exec` until its admission fence closes), so that is not an outage. The model's finding is the `probe-failed` case only. |
| + |
| ## Rev 16: round 15 dispositions |
| Astra's round 15 (`CODEX-PLAN-REGION-HA-1BN-R15.md`) resolved all four round 14 items and raised two P2s. I checked both against spec §5.8 (line 847, "a box that finds its receipt missing after that point … the op goes to `stuck`") and against the rev 15 property code. Both are accepted; none is declined. |
| @@ -342,6 +358,9 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). *Rev 9:* that holds only when the dead target is `op.exec`, or `op.exec` is already "none". While the live source is `op.exec` (`starting`, `releasing`), the CAS consumes the target's fence and records `stuck`. It keeps the source's execution identity, phase and release evidence, because a target's fence is no evidence against the source (claim §4), and root's stop needs the source to remain `op.exec`. Otherwise `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| 24. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive. |
| +25. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `<host>.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. |
| +26. **A drain's probe-failed with no eligible spare is stuck after `ReleaseDeadline`, never sent back to the source** (rev 17, finding F1). §5.8 excludes `to` from the choice after a failed probe, and a relocation's target is never its source (the start CAS, `replaceTargetSlot`). Letting the live source be the "replacement" would put a fresh install of P' into the same host paths that the source's own `park-local` chore retires once the operation completes (§5.8, "What a relocation leaves behind"). The same box would then hold the retiring installation and the new one under one name. So the worker waits for a spare and, `ReleaseDeadline` after `probe_failed_since`, marks the operation `stuck`. This follows the spec's own bound ("It alerts, and the op waits for `carlos ops host resolve`"). The operator's resolve is the claim's: it may return the host to the source, as the model's ruling 13 does. *Cost if wrong:* in a two-box region a failed drain waits ten minutes plus an operator instead of an automatic return. |
| +27. **Every abandoned prepared generation is parked by the operation that built it** (rev 17, finding F3). §5.8 says "an abandoned `P` is parked by the at-rest sweep". This plan adds an earlier owner: `install` journals every unopened P it replaces (`abandoned_prepared`), and `CompleteRelocation` queues a `park-generation` chore for each in the region, beside its `park-local` chores. The at-rest sweep stays the backstop for an operation that never completes. A P that any admission opened is part of the writer lineage and is never on the list. *Cost if wrong:* one chore per abandoned build. Its objects are outside every live set (a prepared manifest names the source's objects, not P's), so parking them retires nothing live. |
| 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. |
| ## Why this is 1b-next, and what it leaves out |
| @@ -370,7 +389,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve and rev 16's (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's and rev 17's eight (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -1444,6 +1463,42 @@ func TestRelocationPreparedManifestIsSequenceZero(t *testing.T) { |
| t.Fatalf("journaled authority: %+v", a) |
| } |
| } |
| + |
| +// Two-box model finding F3: a target replaced in installing leaves its P |
| +// unopened; the replacement builds P'. The serving CAS queues a |
| +// park-generation for P in the region, and none for the opened P'. |
| +func TestRelocationParksEveryAbandonedPreparedGeneration(t *testing.T) { |
| + ctx := context.Background() |
| + s, o := relAtPhase(t, PhaseInstalling) |
| + o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var abandoned []string |
| + if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { |
| + t.Fatalf("abandoned: %v", abandoned) |
| + } |
| + o = step(t, s, o, ack(relSpare)) |
| + o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7))) |
| + o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| + o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| + o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| + Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) |
| + o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| + o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| + var parks []string |
| + for _, ob := range o.Claim.Obligations { |
| + if ob.Kind == "park-generation" && ob.Region == "ie" && ob.Owner == "region:ie" { |
| + parks = append(parks, ob.Generation) |
| + } |
| + } |
| + if !slices.Contains(parks, "P1") || slices.Contains(parks, "P2") { |
| + t.Fatalf("park-generation chores: %v (%+v)", parks, o.Claim.Obligations) |
| + } |
| + if err := o.Claim.Validate(); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -2228,6 +2283,18 @@ func ToInstalling(opID string, t Target, prepared string) func(*Claim) error { |
| } |
| func install(c *Claim, box, prepared, phase string) error { |
| + if old := c.Op.Journal["prepared"]; old != "" && old != prepared && !preparedOpened(c) { |
| + // P was built (or partly built) and never opened: nothing will |
| + // open it now. Journaled so the serving CAS parks it (rev 17, |
| + // two-box model finding F3). |
| + var abandoned []string |
| + readJournal(c.Op, "abandoned_prepared", &abandoned) |
| + if !slices.Contains(abandoned, old) { |
| + if err := journalJSON(c, "abandoned_prepared", append(abandoned, old)); err != nil { |
| + return err |
| + } |
| + } |
| + } |
| stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1 |
| c.Op.Exec, c.Op.Phase = box, phase |
| c.Holder.Stamp = stamp |
| @@ -2433,6 +2500,27 @@ func CompleteRelocation(opID string, newID func() string) func(*Claim) error { |
| c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: box, Retirement: newID(), Owner: "box:" + box}) |
| } |
| } |
| + // Every prepared generation this operation built and no writer |
| + // opened is parked here, in the region, by the claim's own |
| + // park-generation chore (rev 17, finding F3); the at-rest sweep |
| + // stays the backstop the spec names. |
| + var abandoned []string |
| + readJournal(c.Op, "abandoned_prepared", &abandoned) |
| + region := c.Op.Intent["region"] |
| + for _, g := range abandoned { |
| + if g == "" || g == c.Holder.Generation || region == "" { |
| + continue |
| + } |
| + queued := false |
| + for _, o := range c.Obligations { |
| + if o.Kind == "park-generation" && o.Region == region && o.Generation == g { |
| + queued = true |
| + } |
| + } |
| + if !queued { |
| + c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-generation", Region: region, Generation: g, Retirement: newID(), Owner: "region:" + region}) |
| + } |
| + } |
| c.Op = Op{Kind: KindServing, Exec: target, Participants: []string{target}} |
| return nil |
| } |
| @@ -5149,6 +5237,7 @@ import ( |
| "crypto/rand" |
| "encoding/json" |
| "errors" |
| + "fmt" |
| "slices" |
| "strings" |
| "testing" |
| @@ -5168,6 +5257,7 @@ var boxes = map[string]string{"wrk": "a", "src": "b", "dst": "c", "spa": "d", "s |
| func enrollment(box string) string { return strings.Repeat(boxes[box], 32) } |
| type fixture struct { |
| + pinErr error |
| t *testing.T |
| s release.Store |
| w Worker |
| @@ -5230,6 +5320,9 @@ type pins struct{ f *fixture } |
| func (p pins) Pin(_ context.Context, o hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) { |
| p.f.leases = append(p.f.leases, "pin") |
| + if p.f.pinErr != nil { |
| + return hostclaim.RecoveryPin{}, p.f.pinErr |
| + } |
| var r hostclaim.Release |
| _ = json.Unmarshal([]byte(o.Claim.Op.Journal["release"]), &r) |
| return hostclaim.RecoveryPin{Key: PinKey(o.Claim.Host, o.Claim.Op.ID, o.Claim.Gen), SHA256: strings.Repeat("a", 64), Generation: o.Claim.Holder.Generation, |
| @@ -5758,6 +5851,65 @@ func TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement(t *testing.T) { |
| t.Fatalf("replacement: %+v participants %v", o.Claim.Op.Intent, o.Claim.Op.Participants) |
| } |
| } |
| + |
| +// Two-box model finding F2: a pin that cannot show the platform files |
| +// never existed (StorePinner's ErrFilesUnevidenced) stops the operation |
| +// for an operator instead of retrying or pinning empty. |
| +func TestWorkerMarksAnUnevidencedFilesPinStuck(t *testing.T) { |
| + f := newFixture(t) |
| + if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.ack("src") |
| + f.ack("dst") |
| + f.worker(nil) // releasing |
| + h := f.claim().Claim.Holder |
| + f.participant(hostclaim.JournalRelease(f.claim().Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| + f.worker(nil) // pinning |
| + f.pinErr = fmt.Errorf("%w: test", ErrFilesUnevidenced) |
| + if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Journal["pin"] != "" { |
| + t.Fatalf("an unevidenced files pin: %+v", o.Claim.Op) |
| + } |
| +} |
| + |
| +// Two-box model finding F1: a drain's target fails its probe and |
| +// releases; with no spare (two boxes: the source is excluded) the worker |
| +// waits, then, ReleaseDeadline after the target failed, marks the |
| +// operation stuck for an operator. |
| +func TestWorkerBoundsAProbeFailedWithNoSpare(t *testing.T) { |
| + f := newFixture(t) |
| + ctx := context.Background() |
| + if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.ack("src") |
| + f.ack("dst") |
| + f.worker(nil) // releasing |
| + h := f.claim().Claim.Holder |
| + id := f.claim().Claim.Op.ID |
| + f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| + f.worker(nil) // pinning |
| + f.worker(nil) // pin |
| + f.worker(nil) // installing on dst |
| + f.installAndProbe("dst") |
| + f.now = f.now.Add(ProbeDeadline + time.Second) |
| + if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { |
| + t.Fatalf("phase %s", o.Claim.Op.Phase) |
| + } |
| + h = f.claim().Claim.Holder |
| + f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "dst", Enrollment: enrollment("dst"), Generation: h.Generation, Session: h.Session, FinalTXID: 41, ManifestSeq: 7, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)})) |
| + noSpare := errors.New("no eligible box") |
| + f.w.Choose = func(context.Context, hostclaim.Observed, []string) (hostclaim.Target, error) { |
| + return hostclaim.Target{}, noSpare |
| + } |
| + if _, err := f.w.Step(ctx, host); !errors.Is(err, noSpare) || f.claim().Claim.Op.Journal["stuck"] != "" { |
| + t.Fatalf("before the deadline: %v %+v", err, f.claim().Claim.Op.Journal) |
| + } |
| + f.now = f.now.Add(ReleaseDeadline + time.Second) |
| + if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { |
| + t.Fatalf("no spare after the deadline: %+v", o.Claim.Op) |
| + } |
| +} |
| ``` |
| `internal/hostrelocate/pin_test.go`: |
| @@ -6220,6 +6372,28 @@ func TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne(t *testing.T) { |
| t.Fatalf("after the first publication: %+v %v", after, err) |
| } |
| } |
| + |
| +// Two-box model finding F2: no files manifest anywhere in the chain, and |
| +// a writer was admitted (session 3): the pin refuses rather than pin an |
| +// empty set an install would mint a fresh instance.key over. A holder |
| +// that never admitted a writer pins empty. |
| +func TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter(t *testing.T) { |
| + ctx := context.Background() |
| + at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) |
| + for _, tc := range []struct { |
| + session uint64 |
| + want error |
| + }{{3, ErrFilesUnevidenced}, {0, nil}} { |
| + claims, data := release_NewDirStore(t), release_NewDirStore(t) |
| + o := pinClaim() |
| + o.Claim.Holder.Session = tc.session |
| + p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} |
| + pin, err := p.Pin(ctx, o, at) |
| + if !errors.Is(err, tc.want) { |
| + t.Fatalf("session %d: %+v %v", tc.session, pin, err) |
| + } |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -6586,7 +6760,15 @@ func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, erro |
| } |
| next, err := w.next(ctx, obs) |
| if err != nil { |
| - return obs, err // no eligible box yet: a later pass may find one |
| + // No eligible box yet; a later pass may find one, but not for |
| + // ever: in a two-box region the source is excluded and there |
| + // may be none (rev 17, finding F1). ReleaseDeadline after the |
| + // target failed, the operation is stuck for an operator. |
| + since, perr := time.Parse(time.RFC3339Nano, c.Op.Journal["probe_failed_since"]) |
| + if perr == nil && w.Now().Sub(since) > ReleaseDeadline { |
| + return w.update(ctx, obs, hostclaim.MarkStuck(id, fmt.Sprintf("no eligible box for the nested relocation after %s: %v", ReleaseDeadline, err))) |
| + } |
| + return obs, err |
| } |
| out, err := hostclaim.PinAfterRelease(ctx, w.Claims, obs, id, w.Releases, &next) |
| if err != nil { |
| @@ -6669,6 +6851,11 @@ func (w Worker) pinUnderLease(ctx context.Context, obs hostclaim.Observed) (host |
| } |
| defer func() { _ = held.Release(ctx) }() |
| pin, err := w.Pins.Pin(ctx, obs, w.Now()) |
| + if errors.Is(err, ErrFilesUnevidenced) { |
| + // H6: instance.key, report.token and the env are continuous; a pin |
| + // that cannot show them never existed waits for an operator. |
| + return w.update(ctx, obs, hostclaim.MarkStuck(obs.Claim.Op.ID, err.Error())) |
| + } |
| if err != nil { |
| return obs, err |
| } |
| @@ -6894,7 +7081,7 @@ func (p StorePinner) Pin(ctx context.Context, obs hostclaim.Observed, now time.T |
| if released && obj.DB != nil && obj.DB.MaxTXID < rel.FinalTXID { |
| return hostclaim.RecoveryPin{}, fmt.Errorf("%w: the listing is at %d, the release at %d", ErrNotYet, obj.DB.MaxTXID, rel.FinalTXID) |
| } |
| - files, retained, err := p.filesPin(ctx, c.Holder) |
| + files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0) |
| if err != nil { |
| return hostclaim.RecoveryPin{}, err |
| } |
| @@ -6944,6 +7131,12 @@ func pinRef(key string, raw []byte, obj PinObject) hostclaim.RecoveryPin { |
| return ref |
| } |
| +// ErrFilesUnevidenced is a pin refused because nothing shows the host's |
| +// platform files were never minted: a writer was admitted and no files |
| +// manifest exists in the admitted chain. The worker marks the operation |
| +// stuck (rev 17). |
| +var ErrFilesUnevidenced = errors.New("hostrelocate: no files manifest, and a writer was admitted") |
| + |
| // filesPin is §5.6's files pin: the manifest of holder.generation as |
| // observed or, if that generation has none, of the nearest predecessor in |
| // holder.admissions that has one. It is chosen independently of the |
| @@ -6957,7 +7150,7 @@ func pinRef(key string, raw []byte, obj PinObject) hostclaim.RecoveryPin { |
| // stays restorable at g10. Every referenced object must lie under this |
| // holder's location (its incarnation) in this region's data bucket, at |
| // files/<generation>/objects/; any other reference refuses the pin. |
| -func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder) (FilesPin, []byte, error) { |
| +func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty bool) (FilesPin, []byte, error) { |
| chain := []string{h.Generation} |
| for i := len(h.Admissions) - 1; i >= 0; i-- { |
| chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor) |
| @@ -6989,6 +7182,20 @@ func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder) (FilesPi |
| sum := sha256.Sum256(raw) |
| return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:]), Origins: origins}, raw, nil |
| } |
| + // No manifest anywhere in the chain. An empty files pin needs the |
| + // same kind of evidence an empty database pin does (ruling 15): no |
| + // writer was ever admitted on this holder (session 0), so no app ever |
| + // ran to mint instance.key, report.token or an env (H6); or the |
| + // stopped writer's release flush, which publishes every file whatever |
| + // the setting, reported nothing (manifest_seq 0). Any other admitted |
| + // writer without a manifest (an adopted host whose files never |
| + // published, a first manifest that never landed, a trimmed chain) |
| + // refuses: the worker marks the operation stuck for an operator (rev |
| + // 17, finding F2), never an install that mints a fresh key over data |
| + // sealed with the old one. |
| + if h.Session != 0 && !releasedEmpty { |
| + return FilesPin{}, nil, fmt.Errorf("%w: %s has admitted a writer but no files manifest exists in its chain", ErrFilesUnevidenced, h.Location) |
| + } |
| gen := h.Generation |
| if gen == "" { |
| gen = hostclaim.RootGeneration |
| @@ -7138,7 +7345,7 @@ func ValidateManifest(m Manifest, origins []string) error { |
| - [ ] **Step 5: Run the tests** |
| Run: `nice -n 19 go test ./internal/hostrelocate/ ./internal/hostfiles/ -count=1 -v 2>&1 | grep -E '^(---|ok|FAIL)'` |
| -Expected: PASS for all twenty-two `hostrelocate` tests, and `hostfiles` unchanged. |
| +Expected: PASS for all twenty-seven `hostrelocate` tests (worker and pin), and `hostfiles` unchanged. |
| - [ ] **Step 6: Commit** |
| @@ -11263,6 +11470,7 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| | Before P's first admission, `EnsureLocal`'s production body: `VerifyLocal` checks the receipt's files (against P's prepared manifest), env file and wake state under its `install_stamp`; a retired or partial install is restored by the same `Install` under the same stamp, then verified; an opened P is never reinstalled | 14 (2) | Task 7's `EnsureLocal`; Task 8c's `VerifyLocal` and `Install` | `TestRelocationLocalInstallIsRestoredAfterALostLeaseBeforeTheProbe` (journal the receipt, lose the lease, scoped cleanup retires the install, recover, the probe sees a complete install); `TestRelocationOpenedPIsNeverReinstalled` | |
| | A source's ordinary cold wake in `starting` and `releasing` dispatches through the wake inputs while its admission fence is open; refused once the fence closes or the claim no longer names it; a target's ordinary wake stays refused | 14 (3) | R1 (rev 15) | `TestRelocationSourceWakesInStarting`, `TestRelocationSourceWakesInReleasing`, `TestRelocationSourceWakeRefusedAfterItsFenceCloses`, `TestRelocationSourceWakeRefusedWithoutAuthority`, `TestRelocationTargetOrdinaryWakeStaysRefused` | |
| | After P's first admission, `Receipt`'s production body reads this box's durable local install receipt (the install journal written under its `install_stamp`) and `VerifyReceipt` compares it with the journaled one; a missing or different receipt marks the operation `stuck` before any later admission, never a reinstall; legitimate post-admission changes to files, env and timers are never compared or reverted | 15 (1) | Task 6's participant, Task 7's `VerifyReceipt`; Task 8c's `Receipt` | `TestRelocationLostReceiptAfterPOpensIsStuckAcrossARestart` (open P, delete the install journal, restart the agent, the next probe attempt is refused and the claim is stuck); `TestRelocationPostAdmissionChangesSurviveTheReceiptCheck` | |
| +| Every cold admission writes its files manifest (a copy of the files pin) create-only before the app starts (§5.6), so a session that ran its app always leaves a manifest; the first manifest carrying a freshly minted `instance.key` is published by the continuous publisher (H6) | 17 (F2) | 6b's cold admission; R1 | `TestColdAdmissionWritesItsManifestBeforeTheAppStarts` (a store that refuses the manifest PUT: the app never starts) | |
| | The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| @@ -13085,3 +13293,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 14 guards**, removed one at a time under `TestRelocationControllerLifecycleProperties` with 200 seeds (the table in the Rev 14 dispositions): ten killed, each with its seed and invariant. One, the extra scoped cleanup before a recovery reopen, survives as defence in depth, because `hostexec.Open` runs the same cleanup and refuses on its failure. Rev 13's guard "removing the superseded journal entry once B opens" is gone with that code (C3). |
| - **Rev 15 guards**, removed one at a time (the table in the Rev 15 dispositions): twelve killed. The survivors are defence in depth: `EnsureLocal` before entering probing is covered by the attempt-time check, and `EnsureLocal`'s own short-circuit for an opened P is covered because the participant never calls it after P opens. |
| - **Rev 16 guards** (the table in the Rev 16 dispositions): the stuck-on-lost-receipt step and the exact receipt comparison are killed by their tests; both `EnsureLocal` calls removed together are killed by the property, seed 182. Rev 14's lifecycle guards are killed again under the crash event. |
| +- **Rev 17 guards**, each removed in turn and its test watched to fail: the abandoned-P journal in `install`; the serving CAS's park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; the wait before it. Eight in all. |