amadan
Region HA plan 1b-next rev 4: Astra round 3 resolved (4 open closed, 6 new fixed, none declined)
The ack barrier on releasing -> pinning; FenceExhaustedTarget for a target that dies after the replacements are used; the serving executor's reconcile through an obsolete hold and RelocationPolicy; the admission fence read through Guard.AdmissionFenced, with a fenced source taking the hold-only path; Finish keeping its controller until Close succeeds; the restart release as 6b's authenticated Recover with the recorded owner's directory revision; Verify over the retained manifest copy; Task 9's fixture and claim read under captured authority; production wiring of both providers between Open and Build, gated on the flag (R8). Eight new guards removed in turn; un-gated tasks re-extracted and dry-run on 3407d114.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+661
−45
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+661 −45
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index bc3c662d9..2deedcfc1 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,4 +1,4 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 3) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 4) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| @@ -10,6 +10,23 @@ |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 4: round 3 dispositions |
| + |
| +Astra's round 3 (`CODEX-PLAN-REGION-HA-1BN-R3.md`, against `d93c94d0`) left three earlier findings partly resolved, one unresolved, and added six. All ten are accepted; none is declined. The un-gated changes were dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (8 removals, each failing its test; listed in the Self-review). |
| + |
| +| # | Disposition | What changed | Where | |
| +|---|---|---|---| |
| +| R1-5 | accepted | `PreparedFromInstallManifest.Verify` compares P's entries against the pinner's retained copy (`pin.Files.Retained`), never the source manifest; a late source PUT changes nothing it reads. | Task 8b | |
| +| R1-9 | accepted | After a restart `Release` names the recorded owner's boot, controller and directory revision and offers `RecoveryReleaseDrain`; Task 8a's `Collect` provider sets 6b's `Recover` on that drain, so root's authenticated recovery path accepts the tuple and replays the receipt. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`), Task 8a | |
| +| R2-N4 | accepted | `Policy.Check`'s serving branch lets exactly the serving executor's `reconcile` through an obsolete hold (one naming no operation the claim runs), so the adapter's clear is reachable; every other write under a hold still waits. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`), Task 9 | |
| +| R2-N5 | accepted | See N2. | Task 7 | |
| +| N1 | accepted | Task 9's adapter insertion re-reads the claim under the captured authority (`observed` is defined there, validated against the captured incarnation and stamp); the test fixture (`adapterFixture` and its methods) is this task's code, over 6b's adapter and registry. | Task 9 | |
| +| N2 | accepted | `Finish` keeps the map entry until `Close` succeeds and forgets only that same controller, so a close the pass's context cut short is finished by a later sweep. | Task 7 (`…RetainsTheControllerUnderTheAgentLifetime`: a cancelled close, then a successful one) | |
| +| N3 | accepted | `hostbox.RelocationPolicy(box)` (un-gated, tested) and `hostguard.Guard.AdmissionFenced` (un-gated, tested) are the two providers; Task 10 wires them into the assembly between 6b's `OpenProductionClaimAssembly` and `Build` (which captures `Policy.Check`), gated on `CARLOS_REGION_HA`; with it off neither is set and behaviour is 6b's. The insertion point is request R8. | Task 3, Task 10, R8 | |
| +| N4 | accepted | `Reconciler.Install` takes the hold-only path for a held row whenever the box is not the executor or its admission fence is closed, so a fenced source still named `op.exec` in `releasing` writes its hold under `WithLocal`, never through the guarded path `With` refuses. | Task 9 | |
| +| N5 | accepted | `FenceExhaustedTarget`: a target terminated after the replacements are used is fenced into a stuck `pinning` in one CAS (loss retained, box on `lost`, `op.exec` "none", receipt dropped, P's remote build kept when unopened, nested when opened); the worker uses it instead of `MarkStuck`, which stays the live target's rule. | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning`), Task 5 (`TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall`); ruling 23 | |
| +| N6 | accepted | `PinAfterRelease` from `releasing` requires the acknowledgement barrier, so a target replaced after the source's release acknowledges before the operation leaves the phase. | Task 1 (`TestRelocationReplacementInReleasingAcknowledgesBeforePinning`) | |
| + |
| ## Rev 3: round 2 dispositions |
| Astra's round 2 (`CODEX-PLAN-REGION-HA-1BN-R2.md`, against `ed992c46`) left seven round 1 findings partly resolved and added seven. All fourteen are accepted; none is declined. Every un-gated change was dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (11 removals, each failing its test; listed in the Self-review). |
| @@ -70,6 +87,7 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | |
| | R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | |
| | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | |
| +| R8 | claim 6b | One insertion point: after `OpenProductionClaimAssembly` returns and before `Build` captures `Policy.Check`, call region HA's `applyRegionHA(assembly, os.Getenv)` (Task 10), which with `CARLOS_REGION_HA` on sets `assembly.Policy.Relocation = hostbox.RelocationPolicy(box)` and `assembly.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`), and with it off touches nothing. 6b's `ensureProductionClaimAssembly` is where `entry.Assembly` is stored; the call goes there. | Task 10 | |
| | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | |
| ## Rulings made while planning |
| @@ -97,6 +115,7 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 20. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's. |
| 21. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations/<host>/<op>/manifest-<sha256>.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers. |
| 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| +23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. |
| ## Why this is 1b-next, and what it leaves out |
| @@ -125,7 +144,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five and rev 3's eleven (listed in the Self-review). Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven and rev 4's eight (listed in the Self-review). Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -144,7 +163,7 @@ Left out, with where it goes: |
| | `internal/hostclaim/create.go` | `ExecutorSelection`, `BeginCreateOn`; `CompleteCreate` never substitutes | 2 | |
| | `internal/console/server.go`, `cmd/console/main.go` | `Config.RegionHA` from `CARLOS_REGION_HA` | 2 | |
| | `internal/hostops/executor.go`, `loads.go`, `operations.go`, `backfill.go`, `backfill_controller.go` | `ExecutorChooser` over the roster, the edges snapshot and claim loads, and `Permits`; `Authority.SelectExecutor` before the create CAS, or today's create without one | 2 | |
| -| `internal/hostbox/relocationrows.go`, `wakepolicy.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner and its `reconcile` action | 3 | |
| +| `internal/hostbox/relocationrows.go`, `wakepolicy.go`, `internal/hostguard/admissionfence.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner, its `reconcile` action and `RelocationPolicy`; the admission-fence reader | 3 | |
| | `internal/stopevidence/writers*.go` | L10's writers-stopped observation; one process's descriptors on the database | 4 | |
| | `internal/hostrelocate/worker.go`, `pin.go` | The regional worker: start (eligibility, shape), one replayed CAS per step, the pin under the location lease, the release deadline; the pinned recovery set by claim generation, `ReadPin`, `CopyPinnedManifest` | 5 | |
| | `internal/hostrelocate/participant.go` | A participant box's transitions, ordered over its local effects | 6 | |
| @@ -169,7 +188,7 @@ Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostcl |
| **Interfaces:** |
| - Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`. |
| -- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`) and `Claim.EvidenceRefs`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. |
| +- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`) and `Claim.EvidenceRefs`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. |
| - Journal keys later tasks read: `exec_ack.<box>`, `fence.<box>`, `fence.<box>.observed_at`, `discharge.<box>`, `reason`, `releasing_since`, `release`, `release.phase`, `released_by_target`, `pin`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe_attempts.<box>`, `probe`, `probe_failed_since`, `replacements`, `excluded_targets`, `stuck`. Claim fields: `evidence_refs`, `loss_refs`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -604,6 +623,94 @@ func TestRelocationTwoReplacementsThenStuck(t *testing.T) { |
| } |
| } |
| +func TestRelocationReplacementInReleasingAcknowledgesBeforePinning(t *testing.T) { |
| + ctx := context.Background() |
| + s, o := relAtPhase(t, PhaseReleasing) |
| + h := o.Claim.Holder |
| + o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| + o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) |
| + if err != nil || o.Claim.Op.Phase != PhaseReleasing { |
| + t.Fatalf("replacement in releasing: %v %+v", err, o.Claim.Op) |
| + } |
| + if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("left releasing before the replacement acknowledged: %v", err) |
| + } |
| + o = step(t, s, o, ack(relSpare)) |
| + if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil || o.Claim.Op.Phase != PhasePinning { |
| + t.Fatalf("after the acknowledgement: %v %+v", err, o.Claim.Op) |
| + } |
| +} |
| + |
| +func TestRelocationExhaustedTargetIsFencedIntoAStuckPinning(t *testing.T) { |
| + ctx := context.Background() |
| + exhaust := func(t *testing.T, s release.Store, o Observed) Observed { |
| + var err error |
| + for _, next := range []string{relSpare, relSpare2} { |
| + old := o.Claim.Op.Intent["target_box"] |
| + if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + return o |
| + } |
| + t.Run("before exhaustion it is a replacement", func(t *testing.T) { |
| + s, o := relAtPhase(t, PhasePinning) |
| + if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, relAuthority{}); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("fenced as exhausted with replacements left: %v", err) |
| + } |
| + }) |
| + t.Run("dead after its local install, P unopened", func(t *testing.T) { |
| + s, o := relAtPhase(t, PhasePinning) |
| + o = exhaust(t, s, o) |
| + o = step(t, s, o, ack(relSpare2)) |
| + o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P1")) |
| + o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) |
| + o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| + if _, err := Update(ctx, s, o, MarkStuck("rel-1", "x")); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("MarkStuck between installed and a probe: %v", err) |
| + } |
| + ref := putFence(t, s, relSpare2) |
| + o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + c := o.Claim |
| + if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) || |
| + c.Op.Journal["local_install"] != "" || c.Op.Journal["pin"] == "" || c.Op.Journal["prepared"] != "P1" || c.Op.Journal["prepared_authority"] == "" { |
| + t.Fatalf("exhausted and dead: %+v lost %v", c.Op, c.Lost) |
| + } |
| + if len(c.EvidenceRefs) != 3 || c.EvidenceRefs[2].Box != relSpare2 || c.EvidenceRefs[2].Fence != ref { |
| + t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs) |
| + } |
| + if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P2")); !errors.Is(err, ErrRelocation) { |
| + t.Fatalf("a stuck operation moved: %v", err) |
| + } |
| + if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}); err == nil { |
| + t.Fatal("the fence was consumed twice") |
| + } |
| + }) |
| + t.Run("dead after P was opened", func(t *testing.T) { |
| + s, o := relAtPhase(t, "probing+P") |
| + o = exhaust(t, s, o) // after P's admission each replacement is a nested pinning |
| + o = step(t, s, o, ack(relSpare2)) |
| + o = step(t, s, o, JournalPin("rel-1", relPin(o, 60, 9, relObserved.Add(FailoverPinDelay)))) |
| + o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P3")) |
| + o = step(t, s, o, ToInstalled("rel-1", relPrepared("P3", 60, 9))) |
| + o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) |
| + o = step(t, s, o, ToProbing("rel-1", relObserved)) |
| + o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, |
| + Box: relSpare2, Mode: "cold", NewGeneration: "P3", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) |
| + o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, relAuthority{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + c := o.Claim |
| + if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || c.Op.Journal["pin"] != "" || c.Op.Journal["prepared"] != "" { |
| + t.Fatalf("nested stuck pinning: %+v", c.Op) |
| + } |
| + }) |
| +} |
| + |
| func TestRelocationProbeFailureIsANestedRelocation(t *testing.T) { |
| ctx := context.Background() |
| s, o := relAtPhase(t, "probing+P") |
| @@ -1544,6 +1651,11 @@ func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, op |
| if next != nil { |
| return fmt.Errorf("%w: a drain keeps its target", ErrRelocation) |
| } |
| + // §5.8: a target replaced in releasing acknowledges before the |
| + // operation leaves the phase. |
| + if err := requireAcks(c); err != nil { |
| + return err |
| + } |
| } else { |
| if next == nil || next.Box == r.ReleasedBy { |
| return fmt.Errorf("%w: a nested relocation names another target", ErrRelocation) |
| @@ -1953,6 +2065,64 @@ func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID |
| }) |
| } |
| +// FenceExhaustedTarget is the terminal transition for a target terminated |
| +// after the operation has used its replacements (§5.8 "The bound"; round 3 |
| +// finding 5): it consumes the fence (the loss is retained, the box goes on |
| +// lost), revokes the dead installation's authority (op.exec "none", the |
| +// target off the barrier, its receipt dropped), leaves the operation at |
| +// pinning with its pin and, when P was never opened, its remote build kept |
| +// for the operator's replacement, or a nested pinning when P was opened, |
| +// and records stuck in the same CAS. No third target is chosen. MarkStuck's |
| +// "never between installed and a probe" is for a live target; a dead one |
| +// has no probe to make. |
| +func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { |
| + if observed.Absent || observed.Claim == nil { |
| + return Observed{}, ErrWrongState |
| + } |
| + old := observed.Claim.Op.Intent["target_box"] |
| + v, err := VerifyFence(ctx, s, authority, ref) |
| + if err != nil { |
| + return Observed{}, err |
| + } |
| + if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] { |
| + return Observed{}, ErrFence |
| + } |
| + return Update(ctx, s, observed, func(c *Claim) error { |
| + if err := RequireOp(opID)(c); err != nil { |
| + return err |
| + } |
| + if c.Op.Kind != KindRelocating || c.Op.Intent["target_box"] != old { |
| + return ErrStaleOp |
| + } |
| + ensureJournal(c) |
| + if c.Op.Journal["stuck"] != "" { |
| + return ErrAlreadyRecorded |
| + } |
| + if journalUint(c.Op, "replacements") < MaxTargetReplacements { |
| + return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation) |
| + } |
| + if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { |
| + return err |
| + } |
| + removeParticipant(c, old) |
| + opened := preparedOpened(c) |
| + c.Op.Exec, c.Op.Phase = "none", PhasePinning |
| + keys := []string{"local_install", "probe", "release", "release.phase", "probing_since", "probe_failed_since"} |
| + if opened { |
| + // Nested: pinning again from holder.generation, as a replacement |
| + // after P's first admission would. |
| + keys = append(keys, "pin", "prepared", "prepared_authority") |
| + } |
| + for _, k := range keys { |
| + delete(c.Op.Journal, k) |
| + } |
| + if c.Op.Journal["stuck"] == "" { |
| + c.Op.Journal["stuck"] = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) |
| + } |
| + return nil |
| + }) |
| +} |
| + |
| // MarkStuck stops the operation at its current phase for an operator |
| // (`carlos ops host resolve`). It never resets the replacement count and |
| // never invents evidence; every later relocation transition refuses. |
| @@ -2931,7 +3101,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Consumes: Task 1's phase constants. |
| -- Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`; `Policy.Check`'s `reconcile` action. |
| +- Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`, `hostbox.RelocationPolicy(box string)` (its production value); `Policy.Check`'s `reconcile` action, and the serving executor's `reconcile` through an obsolete hold; `hostguard.Guard.AdmissionFenced(ctx, host, Captured) (bool, error)`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -3035,6 +3205,48 @@ func TestPolicyRoutesRelocationToItsOwner(t *testing.T) { |
| } |
| } |
| +func TestPolicyLetsTheServingExecutorClearAnObsoleteHold(t *testing.T) { |
| + serving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}}} |
| + obsolete := func(context.Context, string) (string, error) { return "rel-1", nil } |
| + policy := Policy{Held: obsolete} |
| + if err := policy.Check(context.Background(), serving, Selection{Host: "x.example", Box: "dst", Action: "reconcile"}, hostwrite.Identity{}); err != nil { |
| + t.Fatalf("the serving executor could not reconcile an obsolete hold away: %v", err) |
| + } |
| + for name, s := range map[string]Selection{ |
| + "a spawn under the hold": {Host: "x.example", Box: "dst", Action: "spawn"}, |
| + "another box's reconcile": {Host: "x.example", Box: "src", Action: "reconcile"}, |
| + } { |
| + if err := policy.Check(context.Background(), serving, s, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { |
| + t.Fatalf("%s: %v", name, err) |
| + } |
| + } |
| +} |
| + |
| +func TestRelocationPolicyGivesEachActionItsRole(t *testing.T) { |
| + policy := RelocationPolicy("dst") |
| + at := func(phase, exec string) *hostclaim.Claim { return relocating(phase, exec) } |
| + for name, tc := range map[string]struct { |
| + c *hostclaim.Claim |
| + action string |
| + want error |
| + }{ |
| + "a participant reconciles": {at(hostclaim.PhaseStarting, "src"), "reconcile", nil}, |
| + "a non-executor's held config": {at(hostclaim.PhaseStarting, "src"), "held-config", ErrChanged}, |
| + "the executor's held config": {at(hostclaim.PhaseInstalling, "dst"), "held-config", nil}, |
| + "the executor installs": {at(hostclaim.PhaseInstalled, "dst"), "install", nil}, |
| + "the target spawns in installing": {at(hostclaim.PhaseInstalling, "dst"), "spawn", ErrChanged}, |
| + "the target spawns in probing": {at(hostclaim.PhaseProbing, "dst"), "spawn", ErrChanged}, // a probe spawn carries probe=true elsewhere |
| + "an unknown action": {at(hostclaim.PhaseProbing, "dst"), "purge", ErrChanged}, |
| + } { |
| + if err := policy(context.Background(), tc.c, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: tc.action}, hostwrite.Identity{}); !errors.Is(err, tc.want) { |
| + t.Fatalf("%s: %v, want %v", name, err, tc.want) |
| + } |
| + } |
| + if err := policy(context.Background(), at(hostclaim.PhaseStarting, "src"), Selection{Host: "x.example", Box: "src", Op: "rel-1", Action: "reconcile"}, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { |
| + t.Fatalf("another box's selection: %v", err) |
| + } |
| +} |
| + |
| func TestPolicyLetsAHeldParticipantReconcileItsRow(t *testing.T) { |
| held := func(context.Context, string) (string, error) { return "rel-1", nil } |
| reloc := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { return nil } |
| @@ -3167,7 +3379,7 @@ Apply to `internal/hostbox/wakepolicy.go`: |
| ```diff |
| diff --git a/internal/hostbox/wakepolicy.go b/internal/hostbox/wakepolicy.go |
| -index 9c3c00d6..c26aa474 100644 |
| +index 9c3c00d6..8bc3bb5f 100644 |
| --- a/internal/hostbox/wakepolicy.go |
| +++ b/internal/hostbox/wakepolicy.go |
| @@ -2,6 +2,7 @@ package hostbox |
| @@ -3213,7 +3425,17 @@ index 9c3c00d6..c26aa474 100644 |
| } |
| func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id hostwrite.Identity) error { |
| -@@ -65,7 +80,23 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| +@@ -61,11 +76,32 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| + } |
| + } |
| + if hold != "" { |
| +- return ErrChanged |
| ++ // A hold of an operation the claim no longer runs is obsolete. |
| ++ // Only the serving executor's reconcile may pass it, and only |
| ++ // to clear it (round 3 finding N4); every other write waits. |
| ++ if s.Action != "reconcile" || hold == c.Op.ID || c.Op.ID != "" { |
| ++ return ErrChanged |
| ++ } |
| } |
| return Serving(c, s) |
| } |
| @@ -3238,26 +3460,122 @@ index 9c3c00d6..c26aa474 100644 |
| return ErrChanged |
| } |
| switch s.Action { |
| -@@ -84,7 +115,7 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| +@@ -84,7 +120,33 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| default: |
| return ErrChanged |
| } |
| - return p.Moves(ctx, c, s, id) |
| + return owner(ctx, c, s, id) |
| ++} |
| ++ |
| ++// RelocationPolicy is the Policy.Relocation owner for one box: the role |
| ++// and phase rights of each action under a relocating claim, after |
| ++// Policy.Check has matched the operation and the participant. A |
| ++// reconcile is the hold write any participant makes; held inputs are the |
| ++// executor's; execution actions are what WakePhase allows the executor. |
| ++// Production sets it with CARLOS_REGION_HA on (Task 10); nil refuses. |
| ++func RelocationPolicy(box string) func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { |
| ++ return func(_ context.Context, c *hostclaim.Claim, s Selection, _ hostwrite.Identity) error { |
| ++ if box == "" || s.Box != box || c.Op.Kind != hostclaim.KindRelocating { |
| ++ return ErrChanged |
| ++ } |
| ++ switch s.Action { |
| ++ case "reconcile": |
| ++ return nil |
| ++ case "held-config", "held-release", "held-timers", "install", "restore": |
| ++ if c.Op.Exec != box { |
| ++ return ErrChanged |
| ++ } |
| ++ return nil |
| ++ case "spawn", "seed", "writer": |
| ++ return WakePhase(c, box, false) |
| ++ } |
| ++ return ErrChanged |
| ++ } |
| } |
| const RequestLimit = 120 * time.Second |
| ``` |
| +- [ ] **Step 4b: The admission-fence reader** |
| + |
| +`internal/hostguard/admissionfence.go` (the reading `Reconciler.FenceClosed` is wired to in Task 10; it grants nothing): |
| + |
| +```go |
| +package hostguard |
| + |
| +import "context" |
| + |
| +// AdmissionFenced reports whether the host's durable admission fence is |
| +// closed for the installation and session captured names: a stop root |
| +// began (ActiveFence) or completed (a FenceJournal receipt) for that |
| +// session. It is the same reading With refuses on, exposed for region HA |
| +// §5.3's row decision ("the source in releasing until its admission fence |
| +// closes"). It grants nothing. |
| +func (g Guard) AdmissionFenced(ctx context.Context, host string, captured Captured) (bool, error) { |
| + fenced := false |
| + err := g.locked(ctx, host, func(d *recordDir) error { |
| + c, err := g.fresh(ctx, host) |
| + if err != nil { |
| + return err |
| + } |
| + r, err := d.read(host) |
| + if err != nil { |
| + return err |
| + } |
| + if r.ActiveFence != nil && r.ActiveFence.matchesSession(c, captured) { |
| + fenced = true |
| + return nil |
| + } |
| + for _, receipt := range r.FenceJournal { |
| + if receipt.Target.matchesSession(c, captured) { |
| + fenced = true |
| + return nil |
| + } |
| + } |
| + return nil |
| + }) |
| + return fenced, err |
| +} |
| +``` |
| + |
| +`internal/hostguard/admissionfence_test.go`: |
| + |
| +```go |
| +//go:build unix |
| + |
| +package hostguard |
| + |
| +import "testing" |
| + |
| +func TestAdmissionFencedFollowsTheDurableFence(t *testing.T) { |
| + f, target, captured := fenceFixture(t) |
| + if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, captured); err != nil || fenced { |
| + t.Fatalf("before any fence: %v %v", fenced, err) |
| + } |
| + if _, err := f.g.BeginFence(t.Context(), target); err != nil { |
| + t.Fatal(err) |
| + } |
| + if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, captured); err != nil || !fenced { |
| + t.Fatalf("after BeginFence: %v %v", fenced, err) |
| + } |
| + other := captured |
| + other.Stamp++ |
| + if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, other); err != nil || fenced { |
| + t.Fatalf("another installation: %v %v", fenced, err) |
| + } |
| +} |
| +``` |
| + |
| - [ ] **Step 5: Run the tests** |
| -Run: `nice -n 19 go test ./internal/hostbox/ ./internal/activator/ -run 'Relocation|WakePhase|Policy|ClaimWake|Wake' -count=1` |
| +Run: `nice -n 19 go test ./internal/hostbox/ ./internal/hostguard/ ./internal/activator/ -run 'Relocation|WakePhase|Policy|ClaimWake|Wake|AdmissionFenced' -count=1` |
| Expected: PASS (`activator`'s claim wake calls `WakePhase`; serving and moves cases are unchanged). |
| - [ ] **Step 6: Commit** |
| ```bash |
| -git add internal/hostbox/relocationrows.go internal/hostbox/relocationrows_test.go internal/hostbox/wakepolicy.go |
| +git add internal/hostbox/relocationrows.go internal/hostbox/relocationrows_test.go internal/hostbox/wakepolicy.go internal/hostguard/admissionfence.go internal/hostguard/admissionfence_test.go |
| git commit -m "hostbox: reconcile's relocation row cases, and only the source or a probing target wakes under relocating |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| @@ -4380,6 +4698,42 @@ func TestWorkerMarksExhaustionStuckBeforeLookingForASpare(t *testing.T) { |
| } |
| } |
| +func TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall(t *testing.T) { |
| + f := newFixture(t) |
| + ctx := context.Background() |
| + if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, dead := range []string{"dst", "spa"} { |
| + f.terminate(dead) |
| + f.worker(nil) |
| + } |
| + f.ack("src") |
| + f.ack("sp2") |
| + f.worker(nil) // releasing |
| + h := f.claim().Claim.Holder |
| + id := f.claim().Claim.Op.ID |
| + f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) |
| + f.worker(nil) // pinning |
| + f.worker(nil) // pin |
| + f.worker(nil) // installing on sp2 |
| + c := f.claim().Claim |
| + f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = c.Op.Journal["prepared"]; return nil }) |
| + f.worker(nil) // installed |
| + var n uint64 |
| + _ = json.Unmarshal([]byte(f.claim().Claim.Op.Journal["install_stamp"]), &n) |
| + f.participant(hostclaim.RecordLocalInstall(c.Op.ID, hostclaim.LocalInstall{Box: "sp2", Enrollment: enrollment("sp2"), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("d", 64)})) |
| + f.terminate("sp2") // dead between installed and its probe |
| + o := f.worker(nil) |
| + if o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhasePinning || o.Claim.Op.Exec != "none" || !slices.Contains(o.Claim.Lost, "sp2") { |
| + t.Fatalf("an exhausted dead target was not fenced into a stuck pinning: %+v", o.Claim.Op) |
| + } |
| + before := o.Claim.Gen |
| + if o = f.worker(nil); o.Claim.Gen != before { |
| + t.Fatal("a worker moved a stuck operation") |
| + } |
| +} |
| + |
| func TestWorkerMarksAnOverdueReleaseStuck(t *testing.T) { |
| f := newFixture(t) |
| if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { |
| @@ -5192,10 +5546,15 @@ func (w Worker) next(ctx context.Context, obs hostclaim.Observed) (hostclaim.Tar |
| func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostclaim.FenceRef, d *hostclaim.LocalDischarge) (hostclaim.Observed, error) { |
| // Exhaustion is checked before a box is chosen (§5.8 "The bound"): |
| - // with the replacements used, no spare is looked for and the |
| - // operation is stuck now, not after an eligible box never appears. |
| + // with the replacements used, no spare is looked for; the dead |
| + // target's fence is consumed into a stuck pinning (the loss retained, |
| + // its installation's authority revoked, no probe to wait for). |
| if exhausted(obs.Claim) { |
| - return w.stuckOn(ctx, obs, hostclaim.ErrReplacementsExceeded) |
| + out, err := hostclaim.FenceExhaustedTarget(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, w.Fences) |
| + if err == nil { |
| + w.Notify(ctx) |
| + } |
| + return out, err |
| } |
| next, err := w.next(ctx, obs) |
| if err != nil { |
| @@ -6061,7 +6420,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `registry.Registry`. |
| -- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `hostfiles.EnvItem`. |
| +- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `hostfiles.EnvItem`. |
| Note: `internal/host`'s tests declare a function named `hostexec`, so this file imports the package as `execctl`. |
| @@ -6187,7 +6546,13 @@ func (c *readyController) Ready(context.Context) (execctl.State, error) { |
| } |
| return c.st, nil |
| } |
| -func (c *readyController) Close(context.Context) error { c.closed = true; return nil } |
| +func (c *readyController) Close(ctx context.Context) error { |
| + if err := ctx.Err(); err != nil { |
| + return err // hostexec retains the lifetime lock on a cut-short join |
| + } |
| + c.closed = true |
| + return nil |
| +} |
| type memPositions struct { |
| calls *[]string |
| @@ -6355,11 +6720,13 @@ func TestRelocationReleaseReplaysOnlyRecordedPositions(t *testing.T) { |
| t.Fatal(err) |
| } |
| e.Identity = func(context.Context) (RelocationIdentity, error) { |
| - return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 4}, BootID: "boot-2", ControllerID: "ctl-2"}, nil |
| + return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 5}, BootID: "boot-2", ControllerID: "ctl-2"}, nil |
| } |
| var asked StopRequest |
| + recovery := false |
| e.Collect = func(_ context.Context, r StopRequest, d StopDrainer) (StopResult, error) { |
| asked = r |
| + _, recovery = d.(RecoveryReleaseDrain) // Task 8a sets req.Recover from this |
| if _, err := d.DrainClaim(context.Background(), r, "fence"); !errors.Is(err, ErrClaimStopIdentity) { |
| return StopResult{}, errors.New("a restarted agent offered a drain for another process's stop") |
| } |
| @@ -6369,8 +6736,8 @@ func TestRelocationReleaseReplaysOnlyRecordedPositions(t *testing.T) { |
| if err != nil || r.FinalTXID != 41 || r.ManifestSeq != 8 { |
| t.Fatalf("after restart: %+v %v", r, err) |
| } |
| - if asked.BootID != "boot" || asked.ControllerID != "ctl" { |
| - t.Fatalf("root was asked under the new identity: %+v", asked) |
| + if asked.BootID != "boot" || asked.ControllerID != "ctl" || asked.ExpectedDirectoryRevision != "4" || !recovery { |
| + t.Fatalf("root was not asked as the recorded owner's recovery: %+v recovery=%v", asked, recovery) |
| } |
| }) |
| t.Run("a record of another session", func(t *testing.T) { |
| @@ -6480,6 +6847,13 @@ func TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime(t *testi |
| if got := restarted.Retained(); len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { |
| t.Fatalf("retained: %+v", got) |
| } |
| + // A close cut short by the pass's context keeps the entry, so a later |
| + // sweep can finish it; then it is forgotten. |
| + cut, cancel := context.WithCancel(context.Background()) |
| + cancel() |
| + if err := restarted.Finish(cut, "shop.example", "rel-1"); err == nil || opened[1].closed || len(restarted.Retained()) != 1 { |
| + t.Fatalf("a cut-short close forgot the controller: %v closed=%v retained=%v", err, opened[1].closed, restarted.Retained()) |
| + } |
| if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[1].closed || len(restarted.Retained()) != 0 { |
| t.Fatalf("finish: %v closed=%v retained=%v", err, opened[1].closed, restarted.Retained()) |
| } |
| @@ -6929,17 +7303,26 @@ func (r *RelocationControllers) Retained() []hostrelocate.Participation { |
| // scoped to the relocation's own partial work (Task 8a), never the serving |
| // writer's session, so finishing after a successful relocation stops |
| // nothing that serves. |
| +// |
| +// The entry stays until Close succeeds: hostexec retains the lifetime |
| +// lock on a timed-out join and needs another Close, so a controller |
| +// whose close the pass's context cut short is found by the next sweep. |
| func (r *RelocationControllers) Finish(ctx context.Context, host, op string) error { |
| r.mu.Lock() |
| c, ok := r.open[controllerKey(host, op)] |
| - if ok { |
| - delete(r.open, controllerKey(host, op)) |
| - } |
| r.mu.Unlock() |
| if !ok { |
| return nil |
| } |
| - return c.Close(ctx) |
| + if err := c.Close(ctx); err != nil { |
| + return err |
| + } |
| + r.mu.Lock() |
| + if r.open[controllerKey(host, op)] == c { |
| + delete(r.open, controllerKey(host, op)) |
| + } |
| + r.mu.Unlock() |
| + return nil |
| } |
| // Close closes every retained controller (agent shutdown). |
| @@ -7094,13 +7477,17 @@ func (e *RelocationEffects) Release(ctx context.Context, obs hostclaim.Observed) |
| if err != nil { |
| return hostclaim.Release{}, err |
| } |
| - var d StopDrainer = recoveryReleaseDrain{} |
| + var d StopDrainer = RecoveryReleaseDrain{} |
| drain := &releaseDrain{e: e, obs: obs} |
| if flushed && (recorded.BootID != req.BootID || recorded.ControllerID != req.ControllerID) { |
| // The flushing process is gone with its producers: only root's |
| // durable receipt for the recorded identity can finish this |
| - // release. A drain is never offered for another process's stop. |
| - req.BootID, req.ControllerID = recorded.BootID, recorded.ControllerID |
| + // release. The request names the recorded owner (boot, controller, |
| + // directory revision) and the drain refuses; the Collect provider |
| + // marks it a recovery request (6b's authenticated Recover path, |
| + // Task 8a), under which root accepts another tuple for this box's |
| + // session and replays the receipt. |
| + req.BootID, req.ControllerID, req.ExpectedDirectoryRevision = recorded.BootID, recorded.ControllerID, recorded.ExpectedDirectoryRevision |
| } else { |
| if drain.inner, err = e.Producers(ctx, obs); err != nil { |
| return hostclaim.Release{}, err |
| @@ -7134,11 +7521,12 @@ type releaseDrain struct { |
| done bool |
| } |
| -// recoveryReleaseDrain refuses: a restarted agent's producers are not the |
| -// stop's, and root must replay the durable receipt or nothing. |
| -type recoveryReleaseDrain struct{} |
| +// RecoveryReleaseDrain refuses: a restarted agent's producers are not the |
| +// stop's, and root must replay the durable receipt or nothing. The Collect |
| +// provider recognises it and submits the request as a recovery (6b). |
| +type RecoveryReleaseDrain struct{} |
| -func (recoveryReleaseDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) { |
| +func (RecoveryReleaseDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) { |
| return DrainAcknowledgement{}, fmt.Errorf("%w: the process that flushed this stop is gone; only its durable receipt can finish the release", ErrClaimStopIdentity) |
| } |
| @@ -7353,6 +7741,16 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *Relocati |
| }, |
| Producers: func(ctx context.Context, o hostclaim.Observed) (StopDrainer, error) { return a.Wake.Producers(ctx, o) }, |
| Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { |
| + if _, ok := d.(RecoveryReleaseDrain); ok { |
| + // A restarted agent finishing the recorded owner's stop: 6b's |
| + // authenticated recovery path accepts another tuple for this |
| + // box's session and replays the durable receipt (R1-9). |
| + local, err := a.localStopIdentity(ctx) |
| + if err != nil { |
| + return StopResult{}, err |
| + } |
| + r.Recover, r.Region = true, local.Actor.Region |
| + } |
| return CollectClaimStop(ctx, controlsock.Path, r, d) |
| }, |
| StopWriters: a.relocationStopWriters(), |
| @@ -7820,9 +8218,10 @@ func (v PreparedFromInstallManifest) Verify(ctx context.Context, o hostclaim.Obs |
| if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion || m.Seq != pin.Files.Seq { |
| return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: P's files manifest is not the pinned revision", hostrelocate.ErrRefused) |
| } |
| - if pin.Files.SHA256 != "" { |
| - // The pinned bytes, read again: P's entries must be exactly theirs. |
| - pinnedRaw, _, err := v.Files.Get(ctx, hostfiles.ManifestKey(pin.Files.Prefix)) |
| + if pin.Files.Retained != "" { |
| + // The pinner's retained copy of the pinned bytes, never the source |
| + // prefix (ruling 21): P's entries must be exactly theirs. |
| + pinnedRaw, _, err := v.Claims.Get(ctx, pin.Files.Retained) |
| if err != nil { |
| return hostclaim.PreparedAuthority{}, false, err |
| } |
| @@ -8315,7 +8714,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| |---|---|---|---| |
| | `RowReady` | the serving executor, or the source in `starting`/`releasing` before its fence closes | today's: `Activate` as `op.exec`, captured stamp, `SelectAt(…, "reconcile", captured)`, 6b's inventory and census | mint or keep `ready`, `hold_op` empty | |
| | `RowHeld`, `op.exec == box` | the target from `installing` on | the same activation path: the stamp is `install_stamp`, this box's own installation (§5.8) | mint or keep the row held, `hold_op = op.id`, through `SetClaimHold` under the captured work | |
| -| `RowHeld`, `op.exec != box`, a row exists | the source after its fence closed, or in `pinning` and later; a replaced target with its old installation | `Guard.WithLocal` (the flock, no installation authority) around `registry.SetClaimHold` conditional on the row's recorded ownership and observed hold | keep the row, set `hold_op = op.id`; never `ready` | |
| +| `RowHeld`, `op.exec != box` or this box's admission fence closed, a row exists | the source after its fence closed (still `op.exec` in `releasing`, or in `pinning` and later); a replaced target with its old installation | `Guard.WithLocal` (the flock, no installation authority) around `registry.SetClaimHold` conditional on the row's recorded ownership and observed hold; the guarded path would refuse a fenced session (`With`: `ErrFenced`) | keep the row, set `hold_op = op.id`; never `ready` | |
| | `RowHeld`, `op.exec != box`, no row | the target before `installing` | none | mint nothing (log once); the edges file withholds the host from a box that is not `op.exec` (§24), so no request routes here | |
| | `RowRetiring` | the owner of an open `park-local` | none here: the chore retires it under the host guard (P6-D2) | keep, held | |
| | `RowUnexplained` | nobody | none | keep, held, alert once per pass; never delete by hostname | |
| @@ -8335,11 +8734,23 @@ package hostbox |
| import ( |
| "context" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| "errors" |
| + "fmt" |
| + "path/filepath" |
| + "strconv" |
| + "strings" |
| "testing" |
| + "time" |
| "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| ) |
| // Every phase, for the source and the target, through the real adapter: |
| @@ -8360,7 +8771,7 @@ func TestReconcileAdapterInstallFollowsTheRowCases(t *testing.T) { |
| {"source starting", hostclaim.PhaseStarting, "src", "src", false, true, true, "", nil}, |
| {"target before installing, no row", hostclaim.PhaseStarting, "src", "dst", false, false, false, "", ErrChanged}, |
| {"source releasing, fence open", hostclaim.PhaseReleasing, "src", "src", false, true, true, "", nil}, |
| - {"source releasing, fence closed", hostclaim.PhaseReleasing, "src", "src", true, true, true, "rel-1", nil}, // kept, held, under WithLocal |
| + {"source releasing, fence closed", hostclaim.PhaseReleasing, "src", "src", true, true, true, "rel-1", nil}, // still op.exec; kept, held, under WithLocal (a real durable fence: f.beginFence) |
| {"source pinning", hostclaim.PhasePinning, "none", "src", true, true, true, "rel-1", nil}, |
| {"target installing, first pass", hostclaim.PhaseInstalling, "dst", "dst", false, false, true, "rel-1", nil}, // minted held at install_stamp |
| {"target installed", hostclaim.PhaseInstalled, "dst", "dst", false, true, true, "rel-1", nil}, |
| @@ -8375,7 +8786,17 @@ func TestReconcileAdapterInstallFollowsTheRowCases(t *testing.T) { |
| if tc.existing { |
| f.withExistingRow("rel-0") // minted by an earlier pass, with an earlier hold where the case says |
| } |
| - f.adapter.Reconciler.FenceClosed = func(string) bool { return tc.fenceClosed } |
| + if tc.fenceClosed { |
| + f.beginFence() // root's durable BeginFence for the captured session; FenceClosed reads it through Guard.AdmissionFenced |
| + } |
| + f.adapter.Reconciler.FenceClosed = func(host string) bool { |
| + captured, err := hostguard.Capture(f.guard.DataDir, host) |
| + if err != nil { |
| + return false |
| + } |
| + fenced, _ := f.guard.AdmissionFenced(context.Background(), host, captured) |
| + return fenced |
| + } |
| err := f.adapter.Install(context.Background(), f.record(), f.write) |
| if !errors.Is(err, tc.wantErr) { |
| t.Fatalf("err %v, want %v", err, tc.wantErr) |
| @@ -8428,6 +8849,161 @@ func TestReconcileAdapterClearsTheHoldAfterCompletion(t *testing.T) { |
| } |
| } |
| +// adapterFixture is this task's: 6b's ReconcileAdapter over a real |
| +// registry, inventory, guard directory and a dir-store claim, with one |
| +// host whose record is rec. Its methods drive the claim and the registry |
| +// so each case starts where the table says. |
| +type adapterTestFixture struct { |
| + t *testing.T |
| + host string |
| + box string |
| + claims release.Store |
| + registry *registry.Registry |
| + adapter ReconcileAdapter |
| + rec provision.ClaimRecord |
| + guard hostguard.Guard |
| +} |
| + |
| +func adapterFixture(t *testing.T, box string) *adapterTestFixture { |
| + t.Helper() |
| + dataDir := t.TempDir() |
| + reg, err := registry.Open(filepath.Join(dataDir, "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { reg.Close() }) |
| + claims := release.NewDirStore(t.TempDir()) |
| + inv, err := OpenStopInventoryDB(filepath.Join(dataDir, "inventory.db")) // 6b's |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + guard := hostguard.Guard{DataDir: dataDir, Claims: hostguard.StoreReader{Store: claims}} |
| + f := &adapterTestFixture{t: t, host: "shop.bab.oncarlos.com", box: box, claims: claims, registry: reg, guard: guard} |
| + f.adapter = ReconcileAdapter{Registry: reg, Inventory: inv, Box: box, Enrollment: strings.Repeat(box[0:1], 32), DirectoryRevision: "4", |
| + Reconciler: Reconciler{Claims: claims, Guard: guard, Box: box, Region: "ie", Now: time.Now, |
| + Writes: hostwrite.Config{Enabled: true, Selector: WorkSource{Guard: guard, Box: box, Region: "ie", Now: time.Now, Policy: Policy{Held: RegistryHoldPolicy(reg), Relocation: RelocationPolicy(box)}.Check, Local: func(context.Context, *hostclaim.Claim, Selection) error { return nil }}}, |
| + CheckRecord: func(context.Context, RecordSelection) error { return nil }, |
| + CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil }}, |
| + Census: &EmptyInstallationCensus{Inventory: inv, Guard: guard, Box: box}} |
| + f.adapter.Reconciler.HoldRow = holdRowFor(f.adapter) |
| + created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| + record := provision.Record{Schema: 1, Account: "bab", App: "jam", Host: f.host, Slug: "shop", Channel: "edge", Region: "ie", CreatedAt: created} |
| + body, _ := json.Marshal(record) |
| + key := provision.RecordKey("bab", "jam", f.host) |
| + if _, err := claims.Put(context.Background(), key, body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.rec = provision.ClaimRecord{Host: f.host, Key: key, Family: "instance", Account: "bab", App: "jam", CreatedAt: created, Value: record, Route: registry.Route{Host: f.host, Kind: "instance", Account: "bab", App: "jam", Provisioned: true}} |
| + // The claim: created on src, one cold session, as Task 5's fixture. |
| + nh := hostclaim.NewHolder{Family: "instance", Key: key, Account: "bab", App: "jam", Host: f.host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| + o, _ := hostclaim.Read(context.Background(), claims, f.host) |
| + o = f.step(o, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("s", 32)})) |
| + o = f.step(o, hostclaim.CompleteCreate("c1", "src")) |
| + f.step(o, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| + return f |
| +} |
| + |
| +func (f *adapterTestFixture) step(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { |
| + f.t.Helper() |
| + next, err := hostclaim.Update(context.Background(), f.claims, o, tr) |
| + if err != nil { |
| + f.t.Fatal(err) |
| + } |
| + return next |
| +} |
| + |
| +func (f *adapterTestFixture) claim() hostclaim.Observed { |
| + o, err := hostclaim.Read(context.Background(), f.claims, f.host) |
| + if err != nil { |
| + f.t.Fatal(err) |
| + } |
| + return o |
| +} |
| + |
| +// claimAt rewrites the operation to phase with exec, keeping the holder, |
| +// as the relocation transitions would leave it (install_stamp journaled |
| +// for the target's phases, the stamp activated on this box when it is |
| +// the executor). |
| +func (f *adapterTestFixture) claimAt(phase, exec string) { |
| + f.t.Helper() |
| + f.step(f.claim(), func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: exec, Participants: []string{"src", "dst"}, |
| + Intent: map[string]string{"region": "ie", "source_box": "src", "target_box": "dst", "source_enrollment": strings.Repeat("s", 32), "target_enrollment": strings.Repeat("d", 32)}, |
| + Journal: map[string]string{"exec_ack.src": "{}", "exec_ack.dst": "{}"}} |
| + if exec == "dst" { |
| + c.Holder.Stamp = c.Gen + 1 |
| + c.Op.Journal["install_stamp"] = strconv.FormatUint(c.Holder.Stamp, 10) |
| + } |
| + return nil |
| + }) |
| + if exec == f.box { |
| + if err := f.guard.Activate(context.Background(), f.host, f.box, nil); err != nil { |
| + f.t.Fatal(err) |
| + } |
| + } |
| +} |
| + |
| +// serving is the completed relocation: serving on box. |
| +func (f *adapterTestFixture) serving(box string) { |
| + f.t.Helper() |
| + f.step(f.claim(), func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: box, Participants: []string{box}} |
| + return nil |
| + }) |
| +} |
| + |
| +func (f *adapterTestFixture) setOp(id string) { |
| + f.step(f.claim(), func(c *hostclaim.Claim) error { c.Op.ID = id; return nil }) |
| +} |
| + |
| +// withExistingRow mints the row an earlier pass would have, owned by this |
| +// box's current installation, with hold. |
| +func (f *adapterTestFixture) withExistingRow(hold string) { |
| + f.t.Helper() |
| + captured, err := hostguard.Capture(f.guard.DataDir, f.host) |
| + if err != nil { |
| + if err := f.guard.Activate(context.Background(), f.host, f.box, func(*hostclaim.Claim) bool { return true }); err != nil { |
| + f.t.Fatal(err) |
| + } |
| + captured, err = hostguard.Capture(f.guard.DataDir, f.host) |
| + if err != nil { |
| + f.t.Fatal(err) |
| + } |
| + } |
| + digest := sha256.Sum256([]byte(fmt.Sprintf("%s\x00%s\x00%d\x00%s", f.host, captured.Incarnation, captured.Stamp, f.box))) |
| + rt := f.rec.Route |
| + rt.ClaimOwnership = registry.ClaimOwnership{ClaimIncarnation: captured.Incarnation, ClaimStamp: captured.Stamp, ClaimOwner: hex.EncodeToString(digest[:]), ClaimRowID: "row-1"} |
| + rt.HoldOp = hold |
| + if err := f.registry.Upsert(rt); err != nil { |
| + f.t.Fatal(err) |
| + } |
| +} |
| + |
| +func (f *adapterTestFixture) record() provision.ClaimRecord { return f.rec } |
| + |
| +// beginFence closes this box's admission fence for the captured session, |
| +// as root's stop collector does (hostguard.BeginFence). |
| +func (f *adapterTestFixture) beginFence() { |
| + f.t.Helper() |
| + o := f.claim() |
| + captured, err := hostguard.Capture(f.guard.DataDir, f.host) |
| + if err != nil { |
| + f.t.Fatal(err) |
| + } |
| + target := hostguard.FenceTarget{FenceID: "fence-1", Host: f.host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Session: o.Claim.Holder.Session, |
| + Op: o.Claim.Op.ID, Phase: o.Claim.Op.Phase, Box: f.box, Enrollment: f.adapter.Enrollment, BootID: "boot", ControllerID: "ctl", ClaimETag: o.ETag, DirectoryRevision: "4"} |
| + if _, err := f.guard.BeginFence(context.Background(), target); err != nil { |
| + f.t.Fatal(err) |
| + } |
| +} |
| + |
| +// write is provision's claim-branch write: the row with its ownership. |
| +func (f *adapterTestFixture) write(owner registry.ClaimOwnership) error { |
| + rt := f.rec.Route |
| + rt.ClaimOwnership = owner |
| + return f.registry.Upsert(rt) |
| +} |
| + |
| // A held row's hold is never cleared or moved by another operation, and |
| // a kept row's hold write needs the row's recorded ownership. |
| func TestReconcileAdapterHoldIsConditional(t *testing.T) { |
| @@ -8482,10 +9058,11 @@ with |
| ```go |
| d := RelocationRow(obs.Claim, r.Box, true, r.fenceClosed(selected.Host)) |
| - if d.Case == RowHeld && obs.Claim.Op.Exec != r.Box { |
| - // No installation of ours to activate: a kept row's hold only, |
| - // under the flock. A target before installing has no row and |
| - // mints none (ruling 17). |
| + if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || r.fenceClosed(selected.Host)) { |
| + // No installation of ours to activate, or one whose admission |
| + // fence is closed (the source in releasing, still op.exec, which |
| + // With refuses): a kept row's hold only, under the flock. A target |
| + // before installing has no row and mints none (ruling 17). |
| return r.holdOnly(ctx, selected, d) |
| } |
| if err := guard.Activate(ctx, selected.Host, r.Box, nil); err != nil { |
| @@ -8526,7 +9103,16 @@ and the field `HoldRow func(ctx context.Context, host, holdOp string) error` on |
| In 6b's `ReconcileAdapter.Install`, after the row is written through `write(owner)` (the `RowReady` and executor `RowHeld` paths), set its hold from the same decision, conditional on the captured ownership and the observed hold, never clearing another operation's: |
| ```go |
| - d := RelocationRow(observed.Claim, a.Box, true, a.Reconciler.fenceClosed(rec.Host)) |
| + // The claim under the captured authority, read here: the earlier |
| + // `observed` is scoped to the census block above (6b). |
| + current, err := hostclaim.Read(ctx, a.Reconciler.Claims, rec.Host) |
| + if err != nil { |
| + return err |
| + } |
| + if current.Absent || current.Claim.Holder == nil || current.Claim.Holder.Incarnation != captured.Incarnation || current.Claim.Holder.Stamp != captured.Stamp { |
| + return ErrChanged |
| + } |
| + d := RelocationRow(current.Claim, a.Box, true, a.Reconciler.fenceClosed(rec.Host)) |
| switch { |
| case d.Case == RowHeld && row.HoldOp != d.HoldOp: |
| // Held for the running operation; an earlier operation's hold is |
| @@ -8534,7 +9120,7 @@ In 6b's `ReconcileAdapter.Install`, after the row is written through `write(owne |
| if err := a.Registry.SetClaimHold(ctx, rec.Host, owner, row.HoldOp, d.HoldOp); err != nil { |
| return err |
| } |
| - case d.Case == RowReady && row.HoldOp != "" && row.HoldOp != observed.Claim.Op.ID: |
| + case d.Case == RowReady && row.HoldOp != "" && row.HoldOp != current.Claim.Op.ID: |
| // The operation that held this row is over and this box is the |
| // serving executor with the captured stamp (the activation above |
| // verified both): the row routes again (N4). |
| @@ -8548,7 +9134,15 @@ and give the reconciler its `HoldRow`: |
| ```go |
| func (a ReconcileAdapter) Configure(cfg *provision.ReconcileConfig) { |
| - a.Reconciler.HoldRow = func(ctx context.Context, host, holdOp string) error { |
| + a.Reconciler.HoldRow = holdRowFor(a) |
| + cfg.ClaimEnabled = a.Reconciler.Writes.Enabled |
| + cfg.ClaimInstall = a.Install |
| + cfg.ClaimRetire = a.Retire |
| +} |
| + |
| +// holdRowFor is the adapter's hold-only write for a kept row. |
| +func holdRowFor(a ReconcileAdapter) func(ctx context.Context, host, holdOp string) error { |
| + return func(ctx context.Context, host, holdOp string) error { |
| row, err := a.Registry.Get(host) |
| if err != nil { |
| return err |
| @@ -8561,9 +9155,6 @@ func (a ReconcileAdapter) Configure(cfg *provision.ReconcileConfig) { |
| } |
| return a.Registry.SetClaimHold(ctx, host, row.ClaimOwnership, row.HoldOp, holdOp) |
| } |
| - cfg.ClaimEnabled = a.Reconciler.Writes.Enabled |
| - cfg.ClaimInstall = a.Install |
| - cfg.ClaimRetire = a.Retire |
| } |
| ``` |
| @@ -8698,6 +9289,30 @@ func runRelocatePass(ctx context.Context, store release.Store, w hostrelocate.Wo |
| } |
| ``` |
| +`applyRegionHA` is Task 10's and runs at request R8's insertion point (after `OpenProductionClaimAssembly`, before `Build` captures `Policy.Check`), gated on the flag: |
| + |
| +```go |
| +// applyRegionHA gives the assembly region HA's two providers (round 3 |
| +// finding N3), only with CARLOS_REGION_HA on; off, the assembly is 6b's. |
| +func applyRegionHA(a *ClaimAssembly, getenv func(string) string) { |
| + if !regionHAEnabled(getenv) || a == nil { |
| + return |
| + } |
| + a.Policy.Relocation = hostbox.RelocationPolicy(a.Reconcile.Box) |
| + guard := a.Guard |
| + a.Reconcile.Reconciler.FenceClosed = func(host string) bool { |
| + captured, err := hostguard.Capture(guard.DataDir, host) |
| + if err != nil { |
| + return false // no installation of ours: nothing to be fenced |
| + } |
| + fenced, err := guard.AdmissionFenced(context.Background(), host, captured) |
| + return err == nil && fenced |
| + } |
| +} |
| +``` |
| + |
| +Its test, `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, builds a `ClaimAssembly` with 6b's test wiring, applies with the flag off (both nil, `Build` behaves as 6b's) and on (both set, `Policy.Check` admits a relocating participant's `reconcile`). |
| + |
| In `internal/host/agent.go`, next to `go runHostFencePass(agentCtx)`, build the worker and participant from the assembly once it exists and start `go runRelocatePass(agentCtx, store, worker, participant, serialChanges)`. The worker's providers: |
| - `Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: time.Now}`; `Releases: StopReceiptReleases{Evidence: store}`; |
| @@ -8795,3 +9410,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Review Focus.** Each of the five has its test in the owning task. The dry run removed these guards one at a time and watched a test fail: `released`'s predecessor box; P's single opening; the failover pin delay (claim and worker); the pin reaching the release; the replacement bound; the nested failover after P; the receipt bound to `op.exec`; create's no-substitution; the start guard's live-source chore and region `park-replica`; the row case's closed fence and `lost`; the chooser's fence exclusion; discharge only when qualified; the writers-stopped thread-table check; the release receipt's obligation; the refusal of a replayed receipt; and a failed files flush stopping the release. Nineteen in all. |
| - **Rev 2 guards**, each removed in turn and its test watched to fail: the pin key bound to the claim generation; `ToInstalled`'s files manifest; the empty pin's evidence (claim); `stuck` before the phase table; the retained automatic loss; the lease lost before the pin CAS; the release bounds before the write (two); the empty pin's evidence (pinner); the pinned manifest's digest; the copied entries' equality; the target's eligibility at start; the session-0 drain refusal; `stuckOn` from `probe-failed`; the release deadline; the participant's readiness gate; `StopWriters` first in the drain; the replayed receipt without a record; the positions' session binding; the controller under the agent lifetime; the wake state in the flush; `Permits`; today's create without a selection; the `reconcile` action's participant and lost checks; the console's chooser off with HA off. Twenty-five in all. |
| - **Rev 3 guards**, each removed in turn and its test watched to fail: the probe attempt bound; discharge through `CompleteObligation`; the former writer's last admission; the per-process descriptor scan; exhaustion before a spare is chosen; the retained manifest's digest; the attempt counted before the probe; the sweep's running-operation check; the recorded identity after a restart; the positions' box binding; `Finish` forgetting the controller. Eleven in all. |
| +- **Rev 4 guards**, each removed in turn and its test watched to fail: the acknowledgement barrier on `releasing → pinning`; `FenceExhaustedTarget`'s exhaustion check; the worker's use of it over `MarkStuck`; the serving branch's obsolete-hold clearance; `RelocationPolicy`'s executor check; `AdmissionFenced`'s active fence; `Finish` keeping its entry until `Close` succeeds; the recorded directory revision after a restart. Eight in all. |