amadan
docs: region HA plan 1b-next rev 18 requires instance.key for a nonempty pin, bounds a dead target with no spare, records every dropped P
Astra round 17: one P1 and two P2s, all accepted. When a writer was admitted and the pinned database is nonempty, the pinned manifest must carry instance.key; otherwise the operation goes stuck. The durability window is recorded as open request L21 for Paul to decide, with two candidates: the platform pre-provisions the key, or the replicator holds until the key lands. A target terminated with no spare is now bounded by FenceTargetWithoutSpare, which consumes the fence. ReplaceTarget's default branch records an unopened P before dropping it, so completion parks it. Dry-run in task order on 3407d114; 5 guards removed in turn, each failing; lifecycle property at 200 seeds passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+246
−36
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+246 −36
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index eb2140b62..954891827 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,4 +1,4 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 17) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 18) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| @@ -10,6 +10,20 @@ |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 18: round 17 dispositions |
| + |
| +Astra's round 17 (`CODEX-PLAN-REGION-HA-1BN-R17.md`) rated F1 to F3 as only partly closed, with one P1 and two P2s. It accepted ruling 26 and its cost. All three findings were checked against the rev 17 code, reproduced, and accepted; none is declined. |
| + |
| +Stage A was dry-run on a clean 3407d114 in task order, and the lifecycle property passed 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| 1 (P1, F2) | A, spec | Rev 17's `filesPin` returned the first manifest it found before checking any evidence. §5.6's pre-start copy can lack the `instance.key` the app mints after starting, so a nonempty database was pinned with no key. | accepted | `Pin` computes `needKey`: a writer was admitted and the pinned database is nonempty. `filesPin` then refuses with `ErrFilesUnevidenced` unless the pinned manifest has a live `instance.key` entry (`InstanceKeyPath`, `liveEntry`), and the worker marks the operation `stuck` (rev 17's path). Ruling 25 is amended. The durability window is recorded as the open request L21, with both candidate contracts, for Paul. | Task 5 (`TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase`: keyless with a nonempty database refuses; keyed pins; keyless with an empty database at a release of 0 pins); Requests (L21) | |
| +| 2 (P2, F1) | A | `Worker.Step` sends a fenced target to `replace` before any phase check. `replace` returned the chooser's error for ever, so a target terminated after its release, with no spare, bypassed rev 17's bound. | accepted | New fence-consuming `hostclaim.FenceTargetWithoutSpare`, sharing `FenceExhaustedTarget`'s body (`fenceTargetStuck`): it retains the loss, puts the box on `lost`, revokes a dead executor and records `stuck` with its reason. `replace` calls it `ReleaseDeadline` after the fence's `observed_at` when no eligible box exists. | Task 1; Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare/target_terminated_after_its_release_true`: waiting before the deadline; after it, `stuck` with the fence consumed, `dst` lost and `op.exec` "none") | |
| +| 3 (P2, F3) | A | `ReplaceTarget`'s default branch (`probe-failed`, or after P's admission) deleted `prepared` without recording an unopened P. That happens when the probe deadline passes before any admission and the target is then terminated. | accepted | `abandonPrepared(c)` records the current P when no writer opened it. `install` and `ReplaceTarget`'s default branch call it before they replace or drop P. `FenceExhaustedTarget` and `FenceTargetWithoutSpare` keep an unopened P for the operator, so they drop nothing. | Task 1 (`TestRelocationParksAPDroppedFromProbeFailed`: probe deadline before admission, target terminated, replaced, nested relocation completed; P1 is parked) | |
| + |
| +Guards removed, each failing its test: the keyless refusal; `needKey` only for a nonempty database; the dead-target bound; the bound consuming the fence; `ReplaceTarget` recording the P it drops. That is five. |
| + |
| ## Rev 17: two-box model findings |
| The joint model (`origin/region-ha-hostlife-two-box`, model plan rev 2 at 6d25c54e) reported three gaps in rev 16 that review rounds 1 to 16 missed. Each comes with a direct test (`TestFailoverBeforeTheFirstManifest`, `TestProbeFailedInATwoBoxRegion`, `TestAbandonedPreparedGenerationParked`). I checked each against this plan's code first, and all three reproduce. All are accepted; none is declined. |
| @@ -329,6 +343,7 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | |
| | R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | |
| | R10 | claim 6b (and the activator's owner) | The reservation-bearing extension of main's isolated root launch protocol, which 6b's own external completion gate names as required "before any tenant shape can run with claims enabled" and as "not supplied by the external owner adapters" (6b rev 5, "External completion gates"). Until it exists, `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable` for every launch (6b, `Preflight`), and the runtime's `Prerequisites` calls `Roles.Preflight` before any dispatch (6b, `runtime.Prerequisites`). Even with R1 every production probe would therefore be refused. The ask: the protocol binds each supported role launch to the durable claim reservation and preserves main's namespace isolation; `Preflight` admits a relocation probe's `"probe"` work through it, bound to `op.exec`, the operation and `install_stamp`; without the protocol, `Preflight` refuses before any probe attempt is counted. | Task 8c, Task 10 | |
| +| L21 | region HA spec (Paul) | **Open: the durability of the always-continuous items between an app minting them and their first publish.** §5.6 has every cold admission copy its manifest before the app starts, and H6 makes `instance.key`, `report.token` and the env files continuous. Continuous is not synchronous: an app can mint `instance.key`, write database rows sealed with it, and replicate them before the publisher's first manifest with the key lands. A box terminated in that window leaves a nonempty database whose key exists nowhere. Rev 18's pinner refuses that case and the operation goes `stuck` for an operator (ruling 25), but the database is not recoverable by any relocation. Closing the window needs an upstream durability contract, and there are two candidates, neither chosen here: the platform pre-provisions `instance.key` (and `report.token`) into the files directory and publishes them before the app's first start, or the replicator holds its first upload until the first manifest carrying the key has landed. The choice is Paul's. | ruling 25 | |
| | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | |
| ## Rulings made while planning |
| @@ -358,7 +373,7 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). *Rev 9:* that holds only when the dead target is `op.exec`, or `op.exec` is already "none". While the live source is `op.exec` (`starting`, `releasing`), the CAS consumes the target's fence and records `stuck`. It keeps the source's execution identity, phase and release evidence, because a target's fence is no evidence against the source (claim §4), and root's stop needs the source to remain `op.exec`. Otherwise `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| 24. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive. |
| -25. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `<host>.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. |
| +25. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `<host>.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. *Rev 18 (round 17 finding 1):* an existing manifest is not evidence either. When a writer was admitted and the pinned database is nonempty, the pinned manifest must carry a live `instance.key`, the minimum of H6's always-continuous items. A manifest copied before the app started, with the key minted after and never published, refuses the pin, and the operation is `stuck`. The durability window itself is not closed by this plan. It is the open request L21, for Paul to choose between the platform pre-provisioning the key and the replicator holding until it lands. *Cost if wrong (rev 18):* an app that never writes `instance.key` and has a nonempty database is stuck on every relocation until an operator resolves it; H6 names the key as platform-managed on every plan, so such an app should not exist. |
| 26. **A drain's probe-failed with no eligible spare is stuck after `ReleaseDeadline`, never sent back to the source** (rev 17, finding F1). §5.8 excludes `to` from the choice after a failed probe, and a relocation's target is never its source (the start CAS, `replaceTargetSlot`). Letting the live source be the "replacement" would put a fresh install of P' into the same host paths that the source's own `park-local` chore retires once the operation completes (§5.8, "What a relocation leaves behind"). The same box would then hold the retiring installation and the new one under one name. So the worker waits for a spare and, `ReleaseDeadline` after `probe_failed_since`, marks the operation `stuck`. This follows the spec's own bound ("It alerts, and the op waits for `carlos ops host resolve`"). The operator's resolve is the claim's: it may return the host to the source, as the model's ruling 13 does. *Cost if wrong:* in a two-box region a failed drain waits ten minutes plus an operator instead of an automatic return. |
| 27. **Every abandoned prepared generation is parked by the operation that built it** (rev 17, finding F3). §5.8 says "an abandoned `P` is parked by the at-rest sweep". This plan adds an earlier owner: `install` journals every unopened P it replaces (`abandoned_prepared`), and `CompleteRelocation` queues a `park-generation` chore for each in the region, beside its `park-local` chores. The at-rest sweep stays the backstop for an operation that never completes. A P that any admission opened is part of the writer lineage and is never on the list. *Cost if wrong:* one chore per abandoned build. Its objects are outside every live set (a prepared manifest names the source's objects, not P's), so parking them retires nothing live. |
| 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. |
| @@ -389,7 +404,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's and rev 17's eight (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's, rev 17's eight and rev 18's five (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -1499,6 +1514,46 @@ func TestRelocationParksEveryAbandonedPreparedGeneration(t *testing.T) { |
| t.Fatal(err) |
| } |
| } |
| + |
| +// Round 17 finding 3: the target's probe deadline passes before any |
| +// admission (P1 unopened), it releases in probe-failed, then it is |
| +// terminated; the replacement's default branch drops P1. It is recorded |
| +// first, so the completion of the nested relocation parks it. |
| +func TestRelocationParksAPDroppedFromProbeFailed(t *testing.T) { |
| + ctx := context.Background() |
| + s, o := relAtPhase(t, PhaseProbing) |
| + if PreparedOpened(o.Claim) { |
| + t.Fatal("P1 was opened") |
| + } |
| + o = step(t, s, o, ToProbeFailed("rel-1", relObserved)) |
| + o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Journal["prepared"] != "" { |
| + t.Fatalf("nested: %+v", o.Claim.Op) |
| + } |
| + var abandoned []string |
| + if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { |
| + t.Fatalf("abandoned: %v", abandoned) |
| + } |
| + o = step(t, s, o, ack(relSpare)) |
| + o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay)))) |
| + o = toProbing(t, s, o, "P2", 40) |
| + o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| + Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) |
| + o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) |
| + o = step(t, s, o, CompleteRelocation("rel-1", counter())) |
| + parked := false |
| + for _, ob := range o.Claim.Obligations { |
| + if ob.Kind == "park-generation" && ob.Generation == "P1" && ob.Region == "ie" { |
| + parked = true |
| + } |
| + } |
| + if !parked { |
| + t.Fatalf("P1 was not parked: %+v", o.Claim.Obligations) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -2282,17 +2337,27 @@ func ToInstalling(opID string, t Target, prepared string) func(*Claim) error { |
| } |
| } |
| +// abandonPrepared journals the operation's current P as abandoned when |
| +// no writer opened it, before a transition replaces or drops it: nothing |
| +// will open it now, and the serving CAS parks it (rev 17, finding F3; |
| +// rev 18: every transition that discards an unopened P calls it). |
| +func abandonPrepared(c *Claim) error { |
| + old := c.Op.Journal["prepared"] |
| + if old == "" || preparedOpened(c) { |
| + return nil |
| + } |
| + var abandoned []string |
| + readJournal(c.Op, "abandoned_prepared", &abandoned) |
| + if slices.Contains(abandoned, old) { |
| + return nil |
| + } |
| + return journalJSON(c, "abandoned_prepared", append(abandoned, old)) |
| +} |
| + |
| func install(c *Claim, box, prepared, phase string) error { |
| - if old := c.Op.Journal["prepared"]; old != "" && old != prepared && !preparedOpened(c) { |
| - // P was built (or partly built) and never opened: nothing will |
| - // open it now. Journaled so the serving CAS parks it (rev 17, |
| - // two-box model finding F3). |
| - var abandoned []string |
| - readJournal(c.Op, "abandoned_prepared", &abandoned) |
| - if !slices.Contains(abandoned, old) { |
| - if err := journalJSON(c, "abandoned_prepared", append(abandoned, old)); err != nil { |
| - return err |
| - } |
| + if c.Op.Journal["prepared"] != prepared { |
| + if err := abandonPrepared(c); err != nil { |
| + return err |
| } |
| } |
| stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1 |
| @@ -2610,8 +2675,13 @@ func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID |
| case (phase == PhaseInstalled || phase == PhaseProbing) && !preparedOpened(c): |
| return install(c, next.Box, c.Op.Journal["prepared"], PhaseInstalled) |
| default: |
| - // After P's first admission: nested failover from pinning, |
| - // pinning from holder.generation as observed. |
| + // After P's first admission, or from probe-failed: nested |
| + // failover from pinning, pinning from holder.generation as |
| + // observed. A P no writer opened (a probe deadline before any |
| + // admission) is recorded before it is dropped. |
| + if err := abandonPrepared(c); err != nil { |
| + return err |
| + } |
| c.Op.Exec, c.Op.Phase = "none", PhasePinning |
| for _, k := range []string{"pin", "prepared", "prepared_authority", "local_install", "probe", "release", "release.phase"} { |
| delete(c.Op.Journal, k) |
| @@ -2635,6 +2705,24 @@ func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID |
| // "never between installed and a probe" is for a live target; a dead one |
| // has no probe to make. |
| func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { |
| + return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, "") |
| +} |
| + |
| +// FenceTargetWithoutSpare is FenceExhaustedTarget's transition for a |
| +// terminated target with replacements left but no eligible box to name |
| +// (rev 18, round 17 finding 2): the worker calls it once its bound has |
| +// passed. The fence is consumed in the same CAS (loss retained, the box |
| +// on lost, a dead executor's authority revoked), and the operation is |
| +// stuck for an operator with reason, never left waiting unbounded. |
| +func FenceTargetWithoutSpare(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, reason string) (Observed, error) { |
| + if reason == "" { |
| + return Observed{}, fmt.Errorf("%w: a stuck operation names its reason", ErrRelocation) |
| + } |
| + return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, reason) |
| +} |
| + |
| +// fenceTargetStuck is both: noSpare "" is the exhausted bound. |
| +func fenceTargetStuck(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, noSpare string) (Observed, error) { |
| if observed.Absent || observed.Claim == nil { |
| return Observed{}, ErrWrongState |
| } |
| @@ -2657,8 +2745,12 @@ func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observe |
| if c.Op.Journal["stuck"] != "" { |
| return ErrAlreadyRecorded |
| } |
| - if journalUint(c.Op, "replacements") < MaxTargetReplacements { |
| - return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation) |
| + reason := noSpare |
| + if reason == "" { |
| + if journalUint(c.Op, "replacements") < MaxTargetReplacements { |
| + return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation) |
| + } |
| + reason = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) |
| } |
| if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { |
| return err |
| @@ -2670,7 +2762,7 @@ func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observe |
| // Its execution identity, phase and release records stay, so |
| // root still accepts its own stop (claimstop.go requires the |
| // requester to be op.exec); only stuck is new. |
| - c.Op.Journal["stuck"] = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) |
| + c.Op.Journal["stuck"] = reason |
| return nil |
| } |
| opened := preparedOpened(c) |
| @@ -2681,11 +2773,13 @@ func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observe |
| // after P's first admission would. |
| keys = append(keys, "pin", "prepared", "prepared_authority") |
| } |
| + // An unopened P is kept for the operator's replacement; it is on |
| + // the abandoned list only if a later transition drops it. |
| for _, k := range keys { |
| delete(c.Op.Journal, k) |
| } |
| if c.Op.Journal["stuck"] == "" { |
| - c.Op.Journal["stuck"] = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) |
| + c.Op.Journal["stuck"] = reason |
| } |
| return nil |
| }) |
| @@ -5877,6 +5971,12 @@ func TestWorkerMarksAnUnevidencedFilesPinStuck(t *testing.T) { |
| // waits, then, ReleaseDeadline after the target failed, marks the |
| // operation stuck for an operator. |
| func TestWorkerBoundsAProbeFailedWithNoSpare(t *testing.T) { |
| + for _, terminated := range []bool{false, true} { |
| + t.Run(fmt.Sprintf("target terminated after its release %v", terminated), func(t *testing.T) { boundsAProbeFailed(t, terminated) }) |
| + } |
| +} |
| + |
| +func boundsAProbeFailed(t *testing.T, terminated bool) { |
| f := newFixture(t) |
| ctx := context.Background() |
| if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| @@ -5905,10 +6005,26 @@ func TestWorkerBoundsAProbeFailedWithNoSpare(t *testing.T) { |
| if _, err := f.w.Step(ctx, host); !errors.Is(err, noSpare) || f.claim().Claim.Op.Journal["stuck"] != "" { |
| t.Fatalf("before the deadline: %v %+v", err, f.claim().Claim.Op.Journal) |
| } |
| + if terminated { |
| + // Round 17 finding 2: the released target is terminated. Its |
| + // fence goes to the replacement path first; with no spare that |
| + // path is bounded too, and consumes the fence as it stops. |
| + f.terminate("dst") |
| + if _, err := f.w.Step(ctx, host); !errors.Is(err, noSpare) || f.claim().Claim.Op.Journal["stuck"] != "" { |
| + t.Fatalf("a dead target, before the deadline: %v %+v", err, f.claim().Claim.Op.Journal) |
| + } |
| + } |
| f.now = f.now.Add(ReleaseDeadline + time.Second) |
| - if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { |
| + o := f.worker(nil) |
| + if o.Claim.Op.Journal["stuck"] == "" { |
| t.Fatalf("no spare after the deadline: %+v", o.Claim.Op) |
| } |
| + if terminated && (!slices.Contains(o.Claim.Lost, "dst") || o.Claim.Op.Journal["fence.dst"] == "" || o.Claim.Op.Exec != "none") { |
| + t.Fatalf("the dead target's fence was not consumed: %+v lost %v", o.Claim.Op, o.Claim.Lost) |
| + } |
| + if !terminated && o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { |
| + t.Fatalf("a live failed target: %+v", o.Claim.Op) |
| + } |
| } |
| ``` |
| @@ -5942,6 +6058,14 @@ func pinClaim() hostclaim.Observed { |
| Admissions: []hostclaim.AdmissionRecord{{Generation: "g2", Predecessor: "g1"}, {Generation: "g3", Predecessor: "g2"}}}}} |
| } |
| +// keyed adds the host's instance.key (H6), as every manifest of a host |
| +// whose database is nonempty must carry (rev 18). |
| +func keyed(m hostfiles.Manifest) hostfiles.Manifest { |
| + sha := strings.Repeat("b", 64) |
| + m.Entries["instance.key"] = hostfiles.Entry{Object: pinLocation + "/files/g1/objects/0000000000000001-" + sha, SHA256: sha, Size: 32, Mode: 0o600, Seq: 1} |
| + return m |
| +} |
| + |
| func putManifest(t *testing.T, data release.Store, gen string, m hostfiles.Manifest) (string, []byte) { |
| t.Helper() |
| raw, _ := json.Marshal(m) |
| @@ -5963,7 +6087,7 @@ func TestStorePinnerPinsTheNewestFilesManifestInTheChain(t *testing.T) { |
| claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) |
| // g3 (holder.generation) never published files; g2 did, then g1 earlier. |
| for gen, seq := range map[string]int64{"g2": 5, "g1": 9} { |
| - putManifest(t, data, gen, hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}}) |
| + putManifest(t, data, gen, keyed(hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}})) |
| } |
| selects := 0 |
| p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", |
| @@ -6056,7 +6180,7 @@ func TestStorePinnerChecksTheBoundsBeforeItWrites(t *testing.T) { |
| } { |
| t.Run(tc.name, func(t *testing.T) { |
| claims, data := release_NewDirStore(t), release_NewDirStore(t) |
| - putManifest(t, data, "g3", hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: tc.seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}}) |
| + putManifest(t, data, "g3", keyed(hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: tc.seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}})) |
| o := pinClaim() |
| o.Claim.Op.Journal = tc.journal |
| p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(tc.txid)} |
| @@ -6262,10 +6386,14 @@ func TestPinKeepsAFileOriginTheAdmissionListTrimmed(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - // kept.txt never changes after g1; changed.txt changes every time. |
| + // kept.txt and instance.key never change after g1; changed.txt |
| + // changes every time. |
| if err := p.Stage(ctx, "kept.txt", []byte("written once"), 0o644, false); err != nil { |
| t.Fatal(err) |
| } |
| + if err := p.Stage(ctx, "instance.key", []byte("the host's key"), 0o600, false); err != nil { |
| + t.Fatal(err) |
| + } |
| if err := p.Stage(ctx, "changed.txt", []byte(gen), 0o644, false); err != nil { |
| t.Fatal(err) |
| } |
| @@ -6305,7 +6433,7 @@ func TestPinKeepsAFileOriginTheAdmissionListTrimmed(t *testing.T) { |
| } |
| // A reference outside this incarnation's files is never admitted. |
| foreign := m |
| - foreign.Entries = map[string]hostfiles.Entry{"x.txt": {Object: "instances/" + host + "/other-inc/files/g1/objects/" + strings.Repeat("a", 64), Seq: 1, SHA256: strings.Repeat("a", 64)}} |
| + foreign.Entries = map[string]hostfiles.Entry{"instance.key": m.Entries["instance.key"], "x.txt": {Object: "instances/" + host + "/other-inc/files/g1/objects/" + strings.Repeat("a", 64), Seq: 1, SHA256: strings.Repeat("a", 64)}} |
| raw, _ := json.Marshal(foreign) |
| _, etag, _ := data.Get(ctx, hostfiles.ManifestKey(loc+"/files/g10/")) |
| if _, err := data.Put(ctx, hostfiles.ManifestKey(loc+"/files/g10/"), raw, etag); err != nil { |
| @@ -6317,7 +6445,7 @@ func TestPinKeepsAFileOriginTheAdmissionListTrimmed(t *testing.T) { |
| } |
| } |
| -// Round 14 finding 1: the source's files manifest is at sequence 7; P's |
| +// Round 14 finding 1: the source's files manifest is at sequence 8; P's |
| // prepared manifest is a create-only copy of its entries at sequence 0 |
| // (claim rev 6.30 R18-1, moves rev 7.22 §7.6); the session publisher that |
| // starts after the first admission takes it as its baseline and publishes |
| @@ -6331,6 +6459,12 @@ func TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne(t *testing.T) { |
| t.Fatal(err) |
| } |
| for i := 1; i <= 7; i++ { |
| + if i == 7 { |
| + // The seventh revision also publishes the host's key (H6). |
| + if err := p.Stage(ctx, "instance.key", []byte("the host's key"), 0o600, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| if err := p.Stage(ctx, "a.txt", []byte(fmt.Sprintf("revision %d", i)), 0o644, false); err != nil { |
| t.Fatal(err) |
| } |
| @@ -6340,7 +6474,7 @@ func TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne(t *testing.T) { |
| } |
| pinner := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} |
| ref, err := pinner.Pin(ctx, pinClaim(), time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)) |
| - if err != nil || ref.FilesSeq != 7 { |
| + if err != nil || ref.FilesSeq != 8 { // seven revisions of a.txt and the key: eight objects |
| t.Fatalf("pin: %+v %v", ref, err) |
| } |
| obj, err := ReadPin(ctx, claims, ref) |
| @@ -6394,6 +6528,47 @@ func TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// Round 17 finding 1 (P1): a manifest exists, copied before the app |
| +// started, without the instance.key the app minted afterwards; the writer |
| +// was admitted and the database is nonempty. The pin refuses (the worker |
| +// marks the operation stuck); with the key in the manifest, or an empty |
| +// database, it pins. |
| +func TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase(t *testing.T) { |
| + ctx := context.Background() |
| + at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) |
| + empty := func(context.Context, genselect.Request) (*genselect.Plan, error) { return nil, genselect.ErrEmpty } |
| + released0 := func() string { |
| + raw, _ := json.Marshal(hostclaim.Release{ReleasedBy: "src", Enrollment: "e", Generation: "g3", Session: 3, FinalTXID: 0, ManifestSeq: 4, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}) |
| + return string(raw) |
| + } |
| + for _, tc := range []struct { |
| + name string |
| + key bool |
| + db func(context.Context, genselect.Request) (*genselect.Plan, error) |
| + journal map[string]string |
| + want error |
| + }{ |
| + {"keyless, nonempty database", false, selectAt(41), map[string]string{}, ErrFilesUnevidenced}, |
| + {"keyed, nonempty database", true, selectAt(41), map[string]string{}, nil}, |
| + {"keyless, empty database at a release of 0", false, empty, map[string]string{"release": released0()}, nil}, |
| + } { |
| + t.Run(tc.name, func(t *testing.T) { |
| + claims, data := release_NewDirStore(t), release_NewDirStore(t) |
| + m := hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: 4, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}} |
| + if tc.key { |
| + m = keyed(m) |
| + } |
| + putManifest(t, data, "g3", m) |
| + o := pinClaim() |
| + o.Claim.Op.Journal = tc.journal |
| + p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: tc.db} |
| + if _, err := p.Pin(ctx, o, at); !errors.Is(err, tc.want) { |
| + t.Fatalf("got %v, want %v", err, tc.want) |
| + } |
| + }) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -6734,7 +6909,7 @@ func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, erro |
| return obs, err |
| } |
| if ref != nil { |
| - return w.replace(ctx, obs, *ref, w.Discharge(ctx, o.Roster.Boxes[dst], record)) |
| + return w.replace(ctx, obs, *ref, w.Discharge(ctx, o.Roster.Boxes[dst], record), record.ObservedAt) |
| } |
| } |
| switch c.Op.Phase { |
| @@ -6900,7 +7075,7 @@ func (w Worker) next(ctx context.Context, obs hostclaim.Observed) (hostclaim.Tar |
| return w.Choose(ctx, obs, append(exclude, replaced...)) |
| } |
| -func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostclaim.FenceRef, d *hostclaim.LocalDischarge) (hostclaim.Observed, error) { |
| +func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostclaim.FenceRef, d *hostclaim.LocalDischarge, observedAt string) (hostclaim.Observed, error) { |
| // Exhaustion is checked before a box is chosen (§5.8 "The bound"): |
| // with the replacements used, no spare is looked for; the dead |
| // target's fence is consumed into a stuck pinning (the loss retained, |
| @@ -6914,6 +7089,19 @@ func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostcla |
| } |
| next, err := w.next(ctx, obs) |
| if err != nil { |
| + // No eligible box to replace the dead target. A later pass may |
| + // find one, but not for ever (rev 18, round 17 finding 2): |
| + // ReleaseDeadline after the fence's observation the fence is |
| + // consumed and the operation is stuck for an operator. |
| + at, perr := time.Parse(time.RFC3339Nano, observedAt) |
| + if perr == nil && w.Now().Sub(at) > ReleaseDeadline { |
| + out, ferr := hostclaim.FenceTargetWithoutSpare(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, w.Fences, |
| + fmt.Sprintf("%s terminated and no eligible box replaced it after %s: %v", obs.Claim.Op.Intent["target_box"], ReleaseDeadline, err)) |
| + if ferr == nil { |
| + w.Notify(ctx) |
| + } |
| + return out, ferr |
| + } |
| return obs, err |
| } |
| out, err := hostclaim.ReplaceTarget(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, next, w.NewID(), w.Fences) |
| @@ -7081,7 +7269,11 @@ func (p StorePinner) Pin(ctx context.Context, obs hostclaim.Observed, now time.T |
| if released && obj.DB != nil && obj.DB.MaxTXID < rel.FinalTXID { |
| return hostclaim.RecoveryPin{}, fmt.Errorf("%w: the listing is at %d, the release at %d", ErrNotYet, obj.DB.MaxTXID, rel.FinalTXID) |
| } |
| - files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0) |
| + // A nonempty database written by an admitted writer is sealed with |
| + // that writer's instance.key: the pinned manifest must carry it (rev |
| + // 18, round 17 finding 1). |
| + needKey := c.Holder.Session != 0 && obj.DB != nil && obj.DB.MaxTXID > 0 |
| + files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0, needKey) |
| if err != nil { |
| return hostclaim.RecoveryPin{}, err |
| } |
| @@ -7131,11 +7323,21 @@ func pinRef(key string, raw []byte, obj PinObject) hostclaim.RecoveryPin { |
| return ref |
| } |
| -// ErrFilesUnevidenced is a pin refused because nothing shows the host's |
| -// platform files were never minted: a writer was admitted and no files |
| -// manifest exists in the admitted chain. The worker marks the operation |
| -// stuck (rev 17). |
| -var ErrFilesUnevidenced = errors.New("hostrelocate: no files manifest, and a writer was admitted") |
| +// InstanceKeyPath is H6's always-continuous instance.key, at the root of |
| +// the host's files directory. |
| +const InstanceKeyPath = "instance.key" |
| + |
| +func liveEntry(m hostfiles.Manifest, path string) bool { |
| + e, ok := m.Entries[path] |
| + return ok && !e.Deleted && e.Skipped == "" && e.Object != "" |
| +} |
| + |
| +// ErrFilesUnevidenced is a pin refused because its files set cannot be |
| +// shown to hold the host's platform files: a writer was admitted and no |
| +// files manifest exists in the admitted chain (rev 17), or the database |
| +// is nonempty and the pinned manifest has no instance.key (rev 18). The |
| +// worker marks the operation stuck. |
| +var ErrFilesUnevidenced = errors.New("hostrelocate: the files evidence does not cover the host's platform files") |
| // filesPin is §5.6's files pin: the manifest of holder.generation as |
| // observed or, if that generation has none, of the nearest predecessor in |
| @@ -7150,7 +7352,7 @@ var ErrFilesUnevidenced = errors.New("hostrelocate: no files manifest, and a wri |
| // stays restorable at g10. Every referenced object must lie under this |
| // holder's location (its incarnation) in this region's data bucket, at |
| // files/<generation>/objects/; any other reference refuses the pin. |
| -func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty bool) (FilesPin, []byte, error) { |
| +func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, []byte, error) { |
| chain := []string{h.Generation} |
| for i := len(h.Admissions) - 1; i >= 0; i-- { |
| chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor) |
| @@ -7176,6 +7378,13 @@ func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, released |
| if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion { |
| return FilesPin{}, nil, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err) |
| } |
| + if needKey && !liveEntry(m, InstanceKeyPath) { |
| + // The manifest was copied before the app started and the |
| + // key it minted afterwards never published (§5.6's |
| + // continuous-sync window, L21): installing this set would mint |
| + // a new key over data sealed with the old one. |
| + return FilesPin{}, nil, fmt.Errorf("%w: %s's pinned manifest has no %s for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath) |
| + } |
| if origins, err = referencedOrigins(h.Location, origins, m); err != nil { |
| return FilesPin{}, nil, err |
| } |
| @@ -7345,7 +7554,7 @@ func ValidateManifest(m Manifest, origins []string) error { |
| - [ ] **Step 5: Run the tests** |
| Run: `nice -n 19 go test ./internal/hostrelocate/ ./internal/hostfiles/ -count=1 -v 2>&1 | grep -E '^(---|ok|FAIL)'` |
| -Expected: PASS for all twenty-seven `hostrelocate` tests (worker and pin), and `hostfiles` unchanged. |
| +Expected: PASS for all twenty-eight `hostrelocate` tests (worker and pin), and `hostfiles` unchanged. |
| - [ ] **Step 6: Commit** |
| @@ -13294,3 +13503,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 15 guards**, removed one at a time (the table in the Rev 15 dispositions): twelve killed. The survivors are defence in depth: `EnsureLocal` before entering probing is covered by the attempt-time check, and `EnsureLocal`'s own short-circuit for an opened P is covered because the participant never calls it after P opens. |
| - **Rev 16 guards** (the table in the Rev 16 dispositions): the stuck-on-lost-receipt step and the exact receipt comparison are killed by their tests; both `EnsureLocal` calls removed together are killed by the property, seed 182. Rev 14's lifecycle guards are killed again under the crash event. |
| - **Rev 17 guards**, each removed in turn and its test watched to fail: the abandoned-P journal in `install`; the serving CAS's park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; the wait before it. Eight in all. |
| +- **Rev 18 guards**, each removed in turn and its test watched to fail: the keyless-manifest refusal; the key required only for a nonempty database; the dead-target-without-spare bound; that bound consuming the fence; `ReplaceTarget` recording the unopened P it drops. Five in all. |