amadan
Region HA plan 1b-next rev 5: Astra round 4 resolved; two stages (ruling 24)
Stage A (Tasks 1-7, 9a), build-ready: the fence observer reads without the host lock inside the locked operation, with its errors keeping rows (new 1); a relocation target's probe is the "probe" action through the policy (new 2, dispatch half in R1); Finish journals the participation, clears the exact durable hostexec record after a successful cleanup and retries across a restart, proved on disk against root's §5.10 check (new 5); the obsolete-hold clearance is HA-gated (new 6); Task 9a carries the reconciler patch that compiles on main. Stage B (8a-8c, 9, 10) answered at the contract level: Region on every stop request, the Extend hook after actor verification (R8), the fixture and census fields, the inventoried restore destination, and root's relocation-aware held staging (R9). A "Stage B re-plan" step lists every stage B obligation from rounds 1-4. Seven new guards removed in turn; stage A re-extracted and dry-run on 3407d114.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+843
−186
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+843 −186
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index 2deedcfc1..084a16226 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,31 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 4) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 5) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Thirteen tasks. Seven are buildable on main today and were dry-run (twice: rev 1, then rev 2's changes): the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). Five are gated on claim Plan 6b's box assembly and moves Plan 2, neither merged: the release providers, the build and its verifier, the local install and the probe, reconcile's held rows, and the passes. What those need and no plan declares is listed as requests, not assumed. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 4 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 5: round 4 dispositions |
| + |
| +Astra's round 4 (`CODEX-PLAN-REGION-HA-1BN-R4.md`, against `418daf58`) left three earlier findings partly resolved and added six. All nine are accepted; none is declined. Under ruling 24, stage A findings are fixed in code (dry-run on a clean 3407d114, 7 new guards removed in turn, listed in the Self-review); stage B findings are fixed at the contract level: what must happen, which request covers it, which test proves it. |
| + |
| +| # | Stage | Disposition | What changed | Where | |
| +|---|---|---|---|---| |
| +| R1-9 | B | accepted | `Region` is 6b's field: Task 8a's `Collect` provider sets it on every request from the verified actor (the box's region never changes), initially and on recovery alike, beside `Recover` for the recorded owner's stop. | Task 8a | |
| +| N1 | B | accepted | The fixture's `withExistingRow` takes the hold everywhere it is called; the census literal carries 6b's actual fields (`Inventory`, `Registry`, `Launches`, `Box`, `Enrollment`, `DirectoryRevision`, `Roots`, `SharedPaths`), no `Guard`. | Task 9 | |
| +| N3 | B | accepted | R8's insertion point is after actor verification: 6b's `Build` calls an `Extend(a *ClaimAssembly, actor hostbox.Actor)` hook before it captures `Policy.Check`; `applyRegionHA` is that hook's body and binds the policy to `actor.Box`. | Task 10, R8 | |
| +| 1 | A | accepted | `Guard.AdmissionFenced(host, claim, captured)` takes no lock (one atomic record read; the claim is the caller's fresh read); `Reconciler.FenceClosed` takes the context and the observation and returns an error, which keeps rows and mints none; `holdOnly` re-decides on a fresh read under `WithLocal`. Tested with the real observer, a real `BeginFence`, and a bounded context on both the activation and the hold-only path. | Task 3, Task 9a (`TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver`) | |
| +| 2 | A, B | accepted | Policy half (A): a relocation target's probe is its own selected action, `probe`, which `Policy.Check` admits for a relocating claim and `RelocationPolicy` sends to `WakePhase(probe=true)`; an ordinary `spawn` by the target stays refused. Dispatch half (B): R1 extended, 6b's runtime selects `"probe"` work for a relocation probe. | Task 3 (`TestRelocationPolicyGivesEachActionItsRole`, `TestPolicyLetsAHeldParticipantReconcileItsRow`), R1 | |
| +| 3 | B | accepted | The restore destination (the host's files directory and env path) is created and inventoried under the installation's authority before `Restore`, through 6b's typed inventory (a `files` entry at `install_stamp`), never an untracked `MkdirAll`; tested with a fresh spare that has no host directory. | Task 8c | |
| +| 4 | B | accepted | Held release staging needs root: request R9 asks moves for a relocation-aware relay authorizer (bound to the operation, executor, install stamp and the pinned release); the install verifies staging before it records its receipt; tested under the edge unit's filesystem restrictions. | Task 8c, R9 | |
| +| 5 | A | accepted | `RelocationController.Clear` (hostexec's `ClearStale` with the controller's config); `Finish` journals the participation before closing, keeps the entry and the record on a failed close or clear, clears the exact record after a successful cleanup, and a restart retries through the journaled binding. A real hostexec controller on disk shows root's §5.10 check (`hostexecParticipation`) refusing before and admitting after. | Task 7 (`TestFinishClearsTheDurableRecordSoUpdatesAreEligible`, `TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`) | |
| +| 6 | A | accepted | The obsolete-hold clearance in `Policy.Check` needs `Relocation` set (region HA on); off, today's refusal. Tested both ways on the same held serving row. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`) | |
| + |
| ## Rev 4: round 3 dispositions |
| Astra's round 3 (`CODEX-PLAN-REGION-HA-1BN-R3.md`, against `d93c94d0`) left three earlier findings partly resolved, one unresolved, and added six. All ten are accepted; none is declined. The un-gated changes were dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (8 removals, each failing its test; listed in the Self-review). |
| @@ -81,13 +97,14 @@ Interfaces this plan needs that no merged or declared plan supplies. Each gated |
| | # | To | Request | Waited for by | |
| |---|---|---|---| |
| -| R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. | Task 8c | |
| +| R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. | Task 8c | |
| | R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) | |
| | R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a | |
| | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | |
| | R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences). The console's claim-recovery pass materializes each into `control/host-loss/<id>.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | |
| | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | |
| -| R8 | claim 6b | One insertion point: after `OpenProductionClaimAssembly` returns and before `Build` captures `Policy.Check`, call region HA's `applyRegionHA(assembly, os.Getenv)` (Task 10), which with `CARLOS_REGION_HA` on sets `assembly.Policy.Relocation = hostbox.RelocationPolicy(box)` and `assembly.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`), and with it off touches nothing. 6b's `ensureProductionClaimAssembly` is where `entry.Assembly` is stored; the call goes there. | Task 10 | |
| +| R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | |
| +| R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | |
| | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | |
| ## Rulings made while planning |
| @@ -116,6 +133,7 @@ Each was a conflict between the authorities or a gap in them. Each says what it |
| 21. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations/<host>/<op>/manifest-<sha256>.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers. |
| 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. |
| 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. |
| +24. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive. |
| 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. |
| ## Why this is 1b-next, and what it leaves out |
| @@ -144,7 +162,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven and rev 4's eight (listed in the Self-review). Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight and rev 5's seven (listed in the Self-review). Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -3101,7 +3119,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Consumes: Task 1's phase constants. |
| -- Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`, `hostbox.RelocationPolicy(box string)` (its production value); `Policy.Check`'s `reconcile` action, and the serving executor's `reconcile` through an obsolete hold; `hostguard.Guard.AdmissionFenced(ctx, host, Captured) (bool, error)`. |
| +- Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`, `hostbox.RelocationPolicy(box string)` (its production value); `Policy.Check`'s `reconcile` and `probe` actions, and the serving executor's `reconcile` through an obsolete hold (only with `Relocation` set); `hostguard.Guard.AdmissionFenced(host string, c *hostclaim.Claim, Captured) (bool, error)`, lock-free. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -3208,8 +3226,13 @@ func TestPolicyRoutesRelocationToItsOwner(t *testing.T) { |
| func TestPolicyLetsTheServingExecutorClearAnObsoleteHold(t *testing.T) { |
| serving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}}} |
| obsolete := func(context.Context, string) (string, error) { return "rel-1", nil } |
| - policy := Policy{Held: obsolete} |
| - if err := policy.Check(context.Background(), serving, Selection{Host: "x.example", Box: "dst", Action: "reconcile"}, hostwrite.Identity{}); err != nil { |
| + reconcile := Selection{Host: "x.example", Box: "dst", Action: "reconcile"} |
| + // Region HA off (no Relocation policy): today's refusal, unchanged. |
| + if err := (Policy{Held: obsolete}).Check(context.Background(), serving, reconcile, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { |
| + t.Fatalf("with HA off a held serving row was reconciled: %v", err) |
| + } |
| + policy := Policy{Held: obsolete, Relocation: RelocationPolicy("dst")} |
| + if err := policy.Check(context.Background(), serving, reconcile, hostwrite.Identity{}); err != nil { |
| t.Fatalf("the serving executor could not reconcile an obsolete hold away: %v", err) |
| } |
| for name, s := range map[string]Selection{ |
| @@ -3235,7 +3258,10 @@ func TestRelocationPolicyGivesEachActionItsRole(t *testing.T) { |
| "the executor's held config": {at(hostclaim.PhaseInstalling, "dst"), "held-config", nil}, |
| "the executor installs": {at(hostclaim.PhaseInstalled, "dst"), "install", nil}, |
| "the target spawns in installing": {at(hostclaim.PhaseInstalling, "dst"), "spawn", ErrChanged}, |
| - "the target spawns in probing": {at(hostclaim.PhaseProbing, "dst"), "spawn", ErrChanged}, // a probe spawn carries probe=true elsewhere |
| + "the target spawns in probing": {at(hostclaim.PhaseProbing, "dst"), "spawn", ErrChanged}, // only a probe |
| + "the target probes in probing": {at(hostclaim.PhaseProbing, "dst"), "probe", nil}, |
| + "the target probes in installed": {at(hostclaim.PhaseInstalled, "dst"), "probe", ErrChanged}, |
| + "the source probes": {at(hostclaim.PhaseProbing, "src"), "probe", ErrChanged}, |
| "an unknown action": {at(hostclaim.PhaseProbing, "dst"), "purge", ErrChanged}, |
| } { |
| if err := policy(context.Background(), tc.c, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: tc.action}, hostwrite.Identity{}); !errors.Is(err, tc.want) { |
| @@ -3275,6 +3301,15 @@ func TestPolicyLetsAHeldParticipantReconcileItsRow(t *testing.T) { |
| t.Fatalf("%s: %v", name, err) |
| } |
| } |
| + // A target probe passes the whole policy; its ordinary spawn does not. |
| + probing := relocating(hostclaim.PhaseProbing, "dst") |
| + full := Policy{Held: held, Relocation: RelocationPolicy("dst")} |
| + if err := full.Check(context.Background(), probing, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: "probe"}, hostwrite.Identity{}); err != nil { |
| + t.Fatalf("the target's probe: %v", err) |
| + } |
| + if err := full.Check(context.Background(), probing, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: "spawn"}, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { |
| + t.Fatalf("an ordinary spawn by the probing target: %v", err) |
| + } |
| moving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindMoving, ID: "mv-1", Phase: "sealed", Exec: "src", Participants: []string{"src", "dst"}}} |
| if err := (Policy{Held: held, Moves: reloc, Relocation: reloc}).Check(context.Background(), moving, row("dst", "mv-1"), hostwrite.Identity{}); !errors.Is(err, ErrChanged) { |
| t.Fatalf("a move's participant reconciled through the relocation action: %v", err) |
| @@ -3379,7 +3414,7 @@ Apply to `internal/hostbox/wakepolicy.go`: |
| ```diff |
| diff --git a/internal/hostbox/wakepolicy.go b/internal/hostbox/wakepolicy.go |
| -index 9c3c00d6..8bc3bb5f 100644 |
| +index 9c3c00d6..5dec10df 100644 |
| --- a/internal/hostbox/wakepolicy.go |
| +++ b/internal/hostbox/wakepolicy.go |
| @@ -2,6 +2,7 @@ package hostbox |
| @@ -3425,15 +3460,16 @@ index 9c3c00d6..8bc3bb5f 100644 |
| } |
| func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id hostwrite.Identity) error { |
| -@@ -61,11 +76,32 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| +@@ -61,14 +76,43 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| } |
| } |
| if hold != "" { |
| - return ErrChanged |
| + // A hold of an operation the claim no longer runs is obsolete. |
| + // Only the serving executor's reconcile may pass it, and only |
| -+ // to clear it (round 3 finding N4); every other write waits. |
| -+ if s.Action != "reconcile" || hold == c.Op.ID || c.Op.ID != "" { |
| ++ // to clear it (round 3 finding N4), and only where region HA |
| ++ // is on (Relocation set): with it off, today's refusal. |
| ++ if p.Relocation == nil || s.Action != "reconcile" || hold == c.Op.ID || c.Op.ID != "" { |
| + return ErrChanged |
| + } |
| } |
| @@ -3460,7 +3496,17 @@ index 9c3c00d6..8bc3bb5f 100644 |
| return ErrChanged |
| } |
| switch s.Action { |
| -@@ -84,7 +120,33 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| ++ case "probe": |
| ++ // A relocation target's probe spawn (§5.8, R1): selected as its own |
| ++ // action, so the policy can give it probe authority where an |
| ++ // ordinary spawn has none. |
| ++ if c.Op.Kind != hostclaim.KindRelocating || (hold != "" && hold != s.Op) { |
| ++ return ErrChanged |
| ++ } |
| + case "held-config", "held-release", "held-timers": |
| + if hold != s.Op { |
| + return ErrChanged |
| +@@ -84,7 +128,37 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h |
| default: |
| return ErrChanged |
| } |
| @@ -3472,8 +3518,10 @@ index 9c3c00d6..8bc3bb5f 100644 |
| +// and phase rights of each action under a relocating claim, after |
| +// Policy.Check has matched the operation and the participant. A |
| +// reconcile is the hold write any participant makes; held inputs are the |
| -+// executor's; execution actions are what WakePhase allows the executor. |
| -+// Production sets it with CARLOS_REGION_HA on (Task 10); nil refuses. |
| ++// executor's; execution actions are what WakePhase allows the executor, |
| ++// and "probe" is the target's probe spawn in probing (WakePhase with |
| ++// probe set), which an ordinary spawn never gets. Production sets it with |
| ++// CARLOS_REGION_HA on (Task 10); nil refuses. |
| +func RelocationPolicy(box string) func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { |
| + return func(_ context.Context, c *hostclaim.Claim, s Selection, _ hostwrite.Identity) error { |
| + if box == "" || s.Box != box || c.Op.Kind != hostclaim.KindRelocating { |
| @@ -3489,6 +3537,8 @@ index 9c3c00d6..8bc3bb5f 100644 |
| + return nil |
| + case "spawn", "seed", "writer": |
| + return WakePhase(c, box, false) |
| ++ case "probe": |
| ++ return WakePhase(c, box, true) |
| + } |
| + return ErrChanged |
| + } |
| @@ -3504,38 +3554,34 @@ index 9c3c00d6..8bc3bb5f 100644 |
| ```go |
| package hostguard |
| -import "context" |
| +import "github.com/carlosframework/platform/internal/hostclaim" |
| // AdmissionFenced reports whether the host's durable admission fence is |
| -// closed for the installation and session captured names: a stop root |
| -// began (ActiveFence) or completed (a FenceJournal receipt) for that |
| -// session. It is the same reading With refuses on, exposed for region HA |
| -// §5.3's row decision ("the source in releasing until its admission fence |
| -// closes"). It grants nothing. |
| -func (g Guard) AdmissionFenced(ctx context.Context, host string, captured Captured) (bool, error) { |
| - fenced := false |
| - err := g.locked(ctx, host, func(d *recordDir) error { |
| - c, err := g.fresh(ctx, host) |
| - if err != nil { |
| - return err |
| - } |
| - r, err := d.read(host) |
| - if err != nil { |
| - return err |
| - } |
| - if r.ActiveFence != nil && r.ActiveFence.matchesSession(c, captured) { |
| - fenced = true |
| - return nil |
| - } |
| - for _, receipt := range r.FenceJournal { |
| - if receipt.Target.matchesSession(c, captured) { |
| - fenced = true |
| - return nil |
| - } |
| +// closed for the installation and session captured names on claim c: a |
| +// stop root began (ActiveFence) or completed (a FenceJournal receipt) for |
| +// that session. It is the same reading With refuses on, exposed for |
| +// region HA §5.3's row decision ("the source in releasing until its |
| +// admission fence closes"). It takes no lock: the record is replaced |
| +// atomically, and callers already hold the host lock (Activate's reader, |
| +// WithLocal) or need no lock for one consistent read. The claim is the |
| +// caller's fresh read. It grants nothing. |
| +func (g Guard) AdmissionFenced(host string, c *hostclaim.Claim, captured Captured) (bool, error) { |
| + if c == nil || c.Holder == nil { |
| + return false, ErrNoHolder |
| + } |
| + r, err := readRecord(g.DataDir, host) |
| + if err != nil { |
| + return false, err |
| + } |
| + if r.ActiveFence != nil && r.ActiveFence.matchesSession(c, captured) { |
| + return true, nil |
| + } |
| + for _, receipt := range r.FenceJournal { |
| + if receipt.Target.matchesSession(c, captured) { |
| + return true, nil |
| } |
| - return nil |
| - }) |
| - return fenced, err |
| + } |
| + return false, nil |
| } |
| ``` |
| @@ -3546,24 +3592,39 @@ func (g Guard) AdmissionFenced(ctx context.Context, host string, captured Captur |
| package hostguard |
| -import "testing" |
| +import ( |
| + "errors" |
| + "testing" |
| +) |
| func TestAdmissionFencedFollowsTheDurableFence(t *testing.T) { |
| f, target, captured := fenceFixture(t) |
| - if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, captured); err != nil || fenced { |
| + if fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, captured); err != nil || fenced { |
| t.Fatalf("before any fence: %v %v", fenced, err) |
| } |
| if _, err := f.g.BeginFence(t.Context(), target); err != nil { |
| t.Fatal(err) |
| } |
| - if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, captured); err != nil || !fenced { |
| - t.Fatalf("after BeginFence: %v %v", fenced, err) |
| + // Read inside the host lock, as Activate's reader and WithLocal do: |
| + // no second acquisition, no deadlock. |
| + err := f.g.WithLocal(t.Context(), target.Host, func() error { |
| + fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, captured) |
| + if err != nil || !fenced { |
| + return errors.New("after BeginFence, under the lock: not fenced") |
| + } |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| } |
| other := captured |
| other.Stamp++ |
| - if fenced, err := f.g.AdmissionFenced(t.Context(), target.Host, other); err != nil || fenced { |
| + if fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, other); err != nil || fenced { |
| t.Fatalf("another installation: %v %v", fenced, err) |
| } |
| + if _, err := f.g.AdmissionFenced("no.such.host", f.obs.Claim, captured); !errors.Is(err, ErrNoRecord) { |
| + t.Fatalf("a host with no record: %v", err) |
| + } |
| } |
| ``` |
| @@ -6416,11 +6477,11 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Files:** |
| - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` |
| -- Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go` |
| +- Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationcontrollers_linux_test.go` |
| **Interfaces:** |
| - Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `registry.Registry`. |
| -- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `hostfiles.EnvItem`. |
| +- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `host.RelocationController` (`Ready`, `Close`, `Clear`), `host.OpenRelocationController(ctx, execctl.Config)` (hostexec's controller with `ClearStale` as its `Clear`), `RelocationControllers{Journal, Clear}` with the finishing journal; `hostfiles.EnvItem`. `StopRequest.Region` is 6b's field: Task 8a's `Collect` sets it on every request. |
| Note: `internal/host`'s tests declare a function named `hostexec`, so this file imports the package as `execctl`. |
| @@ -6534,10 +6595,20 @@ func (d recordingDrain) DrainClaim(context.Context, StopRequest, string) (DrainA |
| // readyController is one opened controller: it reports the context it was |
| // opened under, so a test can see it was the agent's, not a pass's. |
| type readyController struct { |
| - st execctl.State |
| - opened context.Context |
| - closed bool |
| - lost error |
| + st execctl.State |
| + opened context.Context |
| + closed bool |
| + cleared bool |
| + lost error |
| + clearErr error |
| +} |
| + |
| +func (c *readyController) Clear(context.Context) error { |
| + if c.clearErr != nil { |
| + return c.clearErr |
| + } |
| + c.cleared = true |
| + return nil |
| } |
| func (c *readyController) Ready(context.Context) (execctl.State, error) { |
| @@ -6854,8 +6925,8 @@ func TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime(t *testi |
| if err := restarted.Finish(cut, "shop.example", "rel-1"); err == nil || opened[1].closed || len(restarted.Retained()) != 1 { |
| t.Fatalf("a cut-short close forgot the controller: %v closed=%v retained=%v", err, opened[1].closed, restarted.Retained()) |
| } |
| - if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[1].closed || len(restarted.Retained()) != 0 { |
| - t.Fatalf("finish: %v closed=%v retained=%v", err, opened[1].closed, restarted.Retained()) |
| + if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[1].closed || !opened[1].cleared || len(restarted.Retained()) != 0 { |
| + t.Fatalf("finish: %v closed=%v cleared=%v retained=%v", err, opened[1].closed, opened[1].cleared, restarted.Retained()) |
| } |
| if err := restarted.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { |
| t.Fatalf("after finish: %v", err) |
| @@ -6936,6 +7007,40 @@ func TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv(t *testing.T) { |
| t.Fatalf("manifest version: %v", raw["v"]) |
| } |
| } |
| + |
| +func TestFinishClearsTheDurableRecordAndRetriesAcrossARestart(t *testing.T) { |
| + journal := filepath.Join(t.TempDir(), "finishing") |
| + e, obs, _, _ := effectsFixture(t) |
| + e.Controllers = &RelocationControllers{Journal: journal} |
| + var ctl *readyController |
| + exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) |
| + e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { |
| + ctl = &readyController{opened: ctx, st: execctl.State{Binding: b, LeaseETag: "lease-1", Expires: exp}} |
| + return ctl, nil |
| + } |
| + if _, err := e.Acknowledge(context.Background(), obs); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The clear fails: the controller is closed, the entry and the |
| + // journal stay, so it is retried. |
| + ctl.clearErr = errors.New("claim read failed") |
| + if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || !ctl.closed || len(e.Retained()) != 1 { |
| + t.Fatalf("a failed clear: %v closed=%v retained=%v", err, ctl.closed, e.Retained()) |
| + } |
| + // The agent restarts: no controller object, the journal remains. The |
| + // clear runs through the recorded binding, then the journal is gone. |
| + cleared := []execctl.Binding{} |
| + again := &RelocationControllers{Journal: journal, Clear: func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil }} |
| + if got := again.Retained(); len(got) != 1 || got[0].Op != "rel-1" { |
| + t.Fatalf("after a restart the journaled participation is forgotten: %+v", got) |
| + } |
| + if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || len(cleared) != 1 || cleared[0].Op != "rel-1" || cleared[0].Box != "src" { |
| + t.Fatalf("retry after restart: %v cleared=%+v", err, cleared) |
| + } |
| + if got := again.Retained(); len(got) != 0 { |
| + t.Fatalf("the journal entry outlived the clear: %+v", got) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| @@ -7217,6 +7322,116 @@ func (f ReleasePositionsFile) Recorded(_ context.Context, r StopRequest) (StopRe |
| } |
| ``` |
| +`internal/host/relocationcontrollers_linux_test.go` (a real hostexec controller on disk; root's §5.10 check is `hostexecParticipation`): |
| + |
| +```go |
| +//go:build linux |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + execctl "github.com/carlosframework/platform/internal/hostexec" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostquarantine" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type activeDirectory struct{} |
| + |
| +func (activeDirectory) Lookup(context.Context, string) (hostquarantine.Actor, error) { |
| + return hostquarantine.Actor{Enrollment: strings.Repeat("a", 32), State: "active"}, nil |
| +} |
| + |
| +// A real hostexec controller on disk: after the relocation completes, |
| +// Finish closes it and clears its durable record, so root's §5.10 check |
| +// (hostexecParticipation) lets an executable update through. While the |
| +// operation still runs, the clear refuses and the record stays. |
| +func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { |
| + ctx := context.Background() |
| + dataDir := t.TempDir() |
| + for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { |
| + if err := os.Mkdir(filepath.Join(dataDir, name), 0o770); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + store := release.NewDirStore(t.TempDir()) |
| + const host = "shop.bab.oncarlos.com" |
| + created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| + nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| + body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) |
| + obs, err := hostclaim.Read(ctx, store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + step := func(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { |
| + t.Helper() |
| + next, err := hostclaim.Update(ctx, store, o, tr) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return next |
| + } |
| + obs = step(obs, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("a", 32)})) |
| + obs = step(obs, hostclaim.CompleteCreate("c1", "src")) |
| + obs = step(obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| + obs = step(obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseStarting, Exec: "src", Participants: []string{"src", "dst"}, |
| + Intent: map[string]string{"region": "ie", "source_box": "src", "target_box": "dst"}} |
| + return nil |
| + }) |
| + cfg := execctl.Config{Store: store, Guard: hostguard.Guard{DataDir: dataDir, LockDir: filepath.Join(dataDir, "locks"), Claims: hostguard.StoreReader{Store: store}}, |
| + Directory: activeDirectory{}, Clock: execctl.SystemClock{}, LockDir: filepath.Join(dataDir, "exec-locks"), |
| + CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: func(context.Context) error { return nil }} |
| + e, _, _, _ := effectsFixture(t) |
| + e.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "finishing")} |
| + e.Identity = func(context.Context) (RelocationIdentity, error) { |
| + return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil |
| + } |
| + e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { |
| + c := cfg |
| + c.Binding = b |
| + return OpenRelocationController(ctx, c) |
| + } |
| + if _, err := e.Acknowledge(ctx, obs); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil { |
| + t.Fatal("a participant's record did not block an update") |
| + } |
| + // Still relocating: the clear refuses, the record and the entry stay. |
| + if err := e.Finish(ctx, host, "rel-1"); !errors.Is(err, execctl.ErrInactive) { |
| + t.Fatalf("finish during the operation: %v", err) |
| + } |
| + if err := hostexecParticipation(dataDir); err == nil || len(e.Retained()) != 1 { |
| + t.Fatalf("the record was cleared while the operation ran: %v retained=%v", err, e.Retained()) |
| + } |
| + // Served on the target: the retry clears it. |
| + step(obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}} |
| + return nil |
| + }) |
| + if err := e.Finish(ctx, host, "rel-1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := hostexecParticipation(dataDir); err != nil { |
| + t.Fatalf("after the relocation an update is still refused: %v", err) |
| + } |
| + if len(e.Retained()) != 0 { |
| + t.Fatalf("retained: %v", e.Retained()) |
| + } |
| +} |
| +``` |
| + |
| `internal/host/relocationeffects.go`: |
| ```go |
| @@ -7224,7 +7439,11 @@ package host |
| import ( |
| "context" |
| + "encoding/json" |
| + "errors" |
| "fmt" |
| + "os" |
| + "path/filepath" |
| "sort" |
| "strings" |
| "sync" |
| @@ -7244,10 +7463,25 @@ type RelocationIdentity struct { |
| } |
| // RelocationController is the execution controller as this package uses |
| -// it: hostexec.Controller's Ready and Close. |
| +// it: hostexec.Controller's Ready and Close, and Clear, which removes the |
| +// controller's durable local record once its cleanup is done |
| +// (hostexec.ClearStale with the controller's config): Close leaves the |
| +// record in "lost", and under §5.10 any record blocks an executable |
| +// update until it is cleared. |
| type RelocationController interface { |
| Ready(context.Context) (execctl.State, error) |
| Close(context.Context) error |
| + Clear(context.Context) error |
| +} |
| + |
| +// finishing is one participation whose controller is closed or closing |
| +// and whose durable record is not yet cleared: the retry state Finish |
| +// keeps on disk (Journal) so a restart retries the clear. |
| +type finishing struct { |
| + V int `json:"v"` |
| + Host string `json:"host"` |
| + Op string `json:"op"` |
| + Binding execctl.Binding `json:"binding"` |
| } |
| // RelocationControllers retains the execution controllers this agent has |
| @@ -7258,6 +7492,14 @@ type RelocationController interface { |
| type RelocationControllers struct { |
| mu sync.Mutex |
| open map[string]RelocationController |
| + // Journal is where finishing participations are recorded before |
| + // their close (<DataDir>/relocate/finishing); Clear retries a journaled |
| + // one after a restart, when no controller object exists, through the |
| + // binding it recorded (hostexec.ClearStale needs only the config). |
| + Journal string |
| + Clear func(context.Context, execctl.Binding) error |
| + // bindings remembers each open controller's binding for the journal. |
| + bindings map[string]execctl.Binding |
| } |
| func controllerKey(host, op string) string { return host + "\x00" + op } |
| @@ -7269,28 +7511,68 @@ func (r *RelocationControllers) get(host, op string) (RelocationController, bool |
| return c, ok |
| } |
| -func (r *RelocationControllers) put(host, op string, c RelocationController) (RelocationController, bool) { |
| +func (r *RelocationControllers) put(host, op string, b execctl.Binding, c RelocationController) (RelocationController, bool) { |
| r.mu.Lock() |
| defer r.mu.Unlock() |
| if r.open == nil { |
| - r.open = map[string]RelocationController{} |
| + r.open, r.bindings = map[string]RelocationController{}, map[string]execctl.Binding{} |
| } |
| if prior, ok := r.open[controllerKey(host, op)]; ok { |
| return prior, false |
| } |
| - r.open[controllerKey(host, op)] = c |
| + r.open[controllerKey(host, op)], r.bindings[controllerKey(host, op)] = c, b |
| return c, true |
| } |
| -// Retained lists the participations whose controllers are open. |
| +func (r *RelocationControllers) journalPath(host, op string) string { |
| + return filepath.Join(r.Journal, host+"~"+op+".json") |
| +} |
| + |
| +// journaled lists the finishing participations on disk. |
| +func (r *RelocationControllers) journaled() ([]finishing, error) { |
| + if r.Journal == "" { |
| + return nil, nil |
| + } |
| + entries, err := os.ReadDir(r.Journal) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + var out []finishing |
| + for _, e := range entries { |
| + raw, err := os.ReadFile(filepath.Join(r.Journal, e.Name())) |
| + if err != nil { |
| + return nil, err |
| + } |
| + var f finishing |
| + if err := json.Unmarshal(raw, &f); err != nil || f.V != 1 { |
| + return nil, fmt.Errorf("%w: finishing journal %s", ErrClaimStopUnavailable, e.Name()) |
| + } |
| + out = append(out, f) |
| + } |
| + return out, nil |
| +} |
| + |
| +// Retained lists the participations whose controllers are open, and the |
| +// journaled ones whose durable record a restart still has to clear. |
| func (r *RelocationControllers) Retained() []hostrelocate.Participation { |
| r.mu.Lock() |
| - defer r.mu.Unlock() |
| + seen := map[string]bool{} |
| var out []hostrelocate.Participation |
| for key := range r.open { |
| host, op, _ := strings.Cut(key, "\x00") |
| + seen[key] = true |
| out = append(out, hostrelocate.Participation{Host: host, Op: op}) |
| } |
| + r.mu.Unlock() |
| + journaled, _ := r.journaled() |
| + for _, f := range journaled { |
| + if !seen[controllerKey(f.Host, f.Op)] { |
| + out = append(out, hostrelocate.Participation{Host: f.Host, Op: f.Op}) |
| + } |
| + } |
| sort.Slice(out, func(i, j int) bool { |
| return out[i].Host < out[j].Host || (out[i].Host == out[j].Host && out[i].Op < out[j].Op) |
| }) |
| @@ -7304,27 +7586,92 @@ func (r *RelocationControllers) Retained() []hostrelocate.Participation { |
| // writer's session, so finishing after a successful relocation stops |
| // nothing that serves. |
| // |
| -// The entry stays until Close succeeds: hostexec retains the lifetime |
| -// lock on a timed-out join and needs another Close, so a controller |
| -// whose close the pass's context cut short is found by the next sweep. |
| +// The participation is journaled before its close, so a restart retries |
| +// what is left. The entry stays until Close succeeds: hostexec retains the |
| +// lifetime lock on a timed-out join and needs another Close, so a |
| +// controller whose close the pass's context cut short is found by the |
| +// next sweep. After the close, Clear removes the durable record (its |
| +// cleanup result, Close's error, is kept by refusing to clear a failed |
| +// one); the journal entry goes only once the record is gone, so a |
| +// restart in between retries the clear through the recorded binding. |
| func (r *RelocationControllers) Finish(ctx context.Context, host, op string) error { |
| + key := controllerKey(host, op) |
| r.mu.Lock() |
| - c, ok := r.open[controllerKey(host, op)] |
| + c, ok := r.open[key] |
| + b := r.bindings[key] |
| r.mu.Unlock() |
| - if !ok { |
| - return nil |
| - } |
| - if err := c.Close(ctx); err != nil { |
| - return err |
| + if ok { |
| + if r.Journal != "" { |
| + raw, err := json.Marshal(finishing{V: 1, Host: host, Op: op, Binding: b}) |
| + if err != nil { |
| + return err |
| + } |
| + if err := os.MkdirAll(r.Journal, 0o700); err != nil { |
| + return err |
| + } |
| + if err := writeFileSynced(r.journalPath(host, op), raw, 0o600); err != nil { |
| + return err |
| + } |
| + } |
| + if err := c.Close(ctx); err != nil { |
| + return err // cleanup failed or was cut short: the record stays, and so does the entry |
| + } |
| + if err := c.Clear(ctx); err != nil { |
| + return err |
| + } |
| + } else { |
| + // No controller object: a restart. Clear through the journaled binding. |
| + journaled, err := r.journaled() |
| + if err != nil { |
| + return err |
| + } |
| + found := false |
| + for _, f := range journaled { |
| + if f.Host == host && f.Op == op { |
| + found, b = true, f.Binding |
| + } |
| + } |
| + if !found { |
| + return nil |
| + } |
| + if r.Clear == nil { |
| + return ErrClaimStopUnavailable |
| + } |
| + if err := r.Clear(ctx, b); err != nil { |
| + return err |
| + } |
| } |
| r.mu.Lock() |
| - if r.open[controllerKey(host, op)] == c { |
| - delete(r.open, controllerKey(host, op)) |
| + if r.open[key] == c { |
| + delete(r.open, key) |
| + delete(r.bindings, key) |
| } |
| r.mu.Unlock() |
| + if r.Journal != "" { |
| + if err := os.Remove(r.journalPath(host, op)); err != nil && !errors.Is(err, os.ErrNotExist) { |
| + return err |
| + } |
| + } |
| return nil |
| } |
| +func writeFileSynced(path string, body []byte, mode os.FileMode) error { |
| + tmp := path + ".tmp" |
| + if err := os.WriteFile(tmp, body, mode); err != nil { |
| + return err |
| + } |
| + f, err := os.Open(tmp) |
| + if err != nil { |
| + return err |
| + } |
| + if err := f.Sync(); err != nil { |
| + _ = f.Close() |
| + return err |
| + } |
| + _ = f.Close() |
| + return os.Rename(tmp, path) |
| +} |
| + |
| // Close closes every retained controller (agent shutdown). |
| func (r *RelocationControllers) Close(ctx context.Context) error { |
| r.mu.Lock() |
| @@ -7424,11 +7771,12 @@ func (e *RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Obser |
| if err != nil { |
| return hostclaim.ExecutionAcknowledgement{}, err |
| } |
| - opened, err := e.Controller(e.Lifetime, execctl.Binding{Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, Host: c.Host, Incarnation: c.Holder.Incarnation, Op: c.Op.ID, BootID: id.BootID}) |
| + b := execctl.Binding{Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, Host: c.Host, Incarnation: c.Holder.Incarnation, Op: c.Op.ID, BootID: id.BootID} |
| + opened, err := e.Controller(e.Lifetime, b) |
| if err != nil { |
| return hostclaim.ExecutionAcknowledgement{}, err |
| } |
| - if ctl, ok = e.Controllers.put(c.Host, c.Op.ID, opened); !ok { |
| + if ctl, ok = e.Controllers.put(c.Host, c.Op.ID, b, opened); !ok { |
| _ = opened.Close(ctx) // another pass opened one first |
| } |
| } |
| @@ -7592,17 +7940,39 @@ func (e *RelocationEffects) Probe(ctx context.Context, obs hostclaim.Observed, c |
| } |
| return e.Admit(ctx, obs, coverage) |
| } |
| + |
| +// execController is hostexec.Controller as a RelocationController: Clear |
| +// is hostexec.ClearStale with the controller's own config, the durable |
| +// record's removal once the claim no longer runs the operation with this |
| +// box (CC9). It is the Controller provider's production value (Task 8a |
| +// fills the assembly's hooks into cfg). |
| +type execController struct { |
| + *execctl.Controller |
| + cfg execctl.Config |
| +} |
| + |
| +func (c *execController) Clear(ctx context.Context) error { return execctl.ClearStale(ctx, c.cfg) } |
| + |
| +// OpenRelocationController opens hostexec's controller for cfg and wraps |
| +// it with its durable clear. |
| +func OpenRelocationController(ctx context.Context, cfg execctl.Config) (RelocationController, error) { |
| + c, err := execctl.Open(ctx, cfg) |
| + if err != nil { |
| + return nil, err |
| + } |
| + return &execController{Controller: c, cfg: cfg}, nil // a pointer: the retained set compares controllers by identity |
| +} |
| ``` |
| - [ ] **Step 5: Run the tests** |
| -Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|Relocation|ReleasePositions|ReleaseFlushFiles|ClaimStop' -count=1 && nice -n 19 go test ./internal/hostfiles/ -count=1` |
| +Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|Relocation|ReleasePositions|ReleaseFlushFiles|ClaimStop|Finish' -count=1 && nice -n 19 go test ./internal/hostfiles/ -count=1` |
| Expected: PASS. |
| - [ ] **Step 6: Commit** |
| ```bash |
| -git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationflush.go internal/hostfiles/platform.go |
| +git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationcontrollers_linux_test.go internal/host/relocationflush.go internal/hostfiles/platform.go |
| git commit -m "host: a release is proved by its full stop receipt; it stops the writers, flushes files, wake state and env, records its positions and only then drains |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| @@ -7610,6 +7980,304 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| --- |
| +### Task 9a: Reconcile's row decision on main (stage A) |
| + |
| +The half of Task 9 that compiles on main: `Reconciler.check` decides a relocatable host's row with `RelocationRow` on a fresh read, the fence observer is consulted inside the already-locked operation (never reacquiring the host lock: round 4 finding 1), its errors keep rows and mint none (C6), and a held row whose box is not the executor, or whose admission fence is closed, takes the hold-only path under the flock (`WithLocal`, `HoldRow`) where the guarded path would refuse it (`ErrFenced`). 6b's adapter supplies `HoldRow` and the registry's conditional hold write (Task 9, stage B); with `FenceClosed` and `HoldRow` nil the reconciler is today's. |
| + |
| +**Files:** |
| +- Modify: `internal/hostbox/reconcile.go` |
| +- Test: `internal/hostbox/reconcile_relocation_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 3's `RelocationRow`, `RelocationPolicy`, `hostguard.Guard.AdmissionFenced`, `Guard.WithLocal`. |
| +- Produces: `Reconciler.FenceClosed func(ctx, hostclaim.Observed) (bool, error)`, `Reconciler.HoldRow func(ctx, host, holdOp string) error`, `(Reconciler) rowDecision(ctx, obs) (RowDecision, error)`, `(Reconciler) holdOnly(ctx, selected, RowDecision) error`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`internal/hostbox/reconcile_relocation_test.go`: |
| + |
| +```go |
| +//go:build unix |
| + |
| +package hostbox |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/park" |
| +) |
| + |
| +// relocationReconciler is the reconciler over workFixture's guard and |
| +// store, with the real fence observer (Guard.AdmissionFenced) and a |
| +// recording hold writer. The claim relocates the host off "box". |
| +func relocationReconciler(t *testing.T) (Reconciler, RecordSelection, *[]string, hostguard.Guard) { |
| + t.Helper() |
| + source, store, obs := workFixture(t) |
| + source.Policy = Policy{Held: func(context.Context, string) (string, error) { return "", nil }, Relocation: RelocationPolicy("box")}.Check |
| + // One cold writer session, so the source has a session a stop can fence. |
| + obs, err := hostclaim.Update(context.Background(), store, obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "box", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + h := obs.Claim.Holder |
| + selected := RecordSelection{Host: obs.Host, Key: h.Key, ETag: "record-etag", Incarnation: h.Incarnation, Family: h.Family, Layout: "flat", Paths: []string{"shop.example.db"}} |
| + holds := &[]string{} |
| + r := Reconciler{Claims: store, Guard: source.Guard, Local: park.Local{Guard: source.Guard, Claims: store, DataDir: source.Guard.DataDir, Box: "box"}, Writes: hostwrite.Config{Enabled: true, Selector: source}, Box: "box", Region: "ie", Now: func() time.Time { return time.Unix(100, 0) }, |
| + CheckRecord: func(context.Context, RecordSelection) error { return nil }, |
| + CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil }, |
| + HoldRow: func(_ context.Context, _ string, op string) error { *holds = append(*holds, op); return nil }} |
| + r.FenceClosed = func(_ context.Context, o hostclaim.Observed) (bool, error) { |
| + captured, err := hostguard.Capture(source.Guard.DataDir, o.Host) |
| + if err != nil { |
| + return false, err |
| + } |
| + return source.Guard.AdmissionFenced(o.Host, o.Claim, captured) |
| + } |
| + return r, selected, holds, source.Guard |
| +} |
| + |
| +func relocate(t *testing.T, r Reconciler, host, phase string) hostclaim.Observed { |
| + t.Helper() |
| + ctx := context.Background() |
| + obs, err := hostclaim.Read(ctx, r.Claims, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Update(ctx, r.Claims, obs, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: "box", Participants: []string{"box", "dst"}, |
| + Intent: map[string]string{"region": "ie", "source_box": "box", "target_box": "dst"}, Journal: map[string]string{"exec_ack.box": "{}", "exec_ack.dst": "{}"}} |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return obs |
| +} |
| + |
| +// The source in releasing takes the activation path (ready) until root |
| +// begins its stop; then the hold-only path, under the flock, with the |
| +// observer read inside the existing locked operation: a bounded context |
| +// finishes both, so neither reacquires the host lock. |
| +func TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver(t *testing.T) { |
| + r, selected, holds, guard := relocationReconciler(t) |
| + obs := relocate(t, r, selected.Host, hostclaim.PhaseReleasing) |
| + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) |
| + defer cancel() |
| + writes := 0 |
| + if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil { |
| + t.Fatalf("the source before its fence closed: %v", err) |
| + } |
| + if writes != 1 || len(*holds) != 0 { |
| + t.Fatalf("writes %d holds %v", writes, *holds) |
| + } |
| + // Root begins the source's stop: the durable admission fence closes. |
| + // (The inventory write above moved the claim: read it fresh.) |
| + captured, err := hostguard.Capture(guard.DataDir, selected.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Read(ctx, r.Claims, selected.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + target := hostguard.FenceTarget{FenceID: "fence-1", Host: selected.Host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Session: obs.Claim.Holder.Session, |
| + Op: "rel-1", Phase: hostclaim.PhaseReleasing, Box: "box", Enrollment: "enrollment-1", BootID: "boot", ControllerID: "ctl", ClaimETag: obs.ETag, DirectoryRevision: "4"} |
| + if _, err := guard.BeginFence(ctx, target); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil { |
| + t.Fatalf("the fenced source's hold: %v", err) |
| + } |
| + if writes != 1 || len(*holds) != 1 || (*holds)[0] != "rel-1" { |
| + t.Fatalf("after the fence: writes %d holds %v (the guarded path must not run)", writes, *holds) |
| + } |
| + // A box with no role mints nothing; an observer error keeps rows and |
| + // mints none. |
| + r.Box = "zzz" |
| + if err := r.Install(ctx, selected, func() error { writes++; return nil }); !errors.Is(err, ErrChanged) || writes != 1 { |
| + t.Fatalf("no role: %v writes %d", err, writes) |
| + } |
| + r.Box = "box" |
| + broken := errors.New("record unreadable") |
| + r.FenceClosed = func(context.Context, hostclaim.Observed) (bool, error) { return false, broken } |
| + if err := r.Install(ctx, selected, func() error { writes++; return nil }); !errors.Is(err, broken) || writes != 1 { |
| + t.Fatalf("an observer error: %v writes %d", err, writes) |
| + } |
| +} |
| + |
| +// With no observer (region HA off) the reconciler is today's: the serving |
| +// executor is ready, a non-executor is refused. |
| +func TestReconcileWithoutAFenceObserverIsTodays(t *testing.T) { |
| + r, selected, _, _ := relocationReconciler(t) |
| + r.FenceClosed, r.HoldRow = nil, nil |
| + writes := 0 |
| + if err := r.Install(context.Background(), selected, func() error { writes++; return nil }); err != nil || writes != 1 { |
| + t.Fatalf("serving: %v writes %d", err, writes) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to see them fail** |
| + |
| +Run: `nice -n 19 go test ./internal/hostbox/ -run 'ReconcileRelocation|ReconcileWithoutAFenceObserver' -count=1` |
| +Expected: FAIL to compile: `unknown field FenceClosed in struct literal`. |
| + |
| +- [ ] **Step 3: The decision, the observer and the hold-only path** |
| + |
| +Apply to `internal/hostbox/reconcile.go` (`git apply`): |
| + |
| +```diff |
| +diff --git a/internal/hostbox/reconcile.go b/internal/hostbox/reconcile.go |
| +index 7cf5d4a5..7fc95731 100644 |
| +--- a/internal/hostbox/reconcile.go |
| ++++ b/internal/hostbox/reconcile.go |
| +@@ -30,7 +30,33 @@ type Reconciler struct { |
| + CheckLocal func(context.Context, hostclaim.Observed, RecordSelection) error |
| + // Adopt is Plan 5's #232 backfill coordinator, never a second adoption rule. |
| + Adopt func(context.Context, RecordSelection) (hostclaim.Observed, error) |
| ++ // FenceClosed reports whether this box's admission fence for the host |
| ++ // is closed on the claim given (region HA §5.3: the source in |
| ++ // releasing is ready until then). It is called under the host lock |
| ++ // and must not take it (hostguard.Guard.AdmissionFenced). nil: never |
| ++ // closed, today's behaviour. An error keeps rows and mints none (C6). |
| ++ FenceClosed func(context.Context, hostclaim.Observed) (bool, error) |
| ++ // HoldRow keeps an existing row held for an operation without |
| ++ // installation authority (ruling 17); 6b's adapter supplies it. |
| ++ HoldRow func(ctx context.Context, host, holdOp string) error |
| + } |
| ++ |
| ++func (r Reconciler) fenceClosed(ctx context.Context, obs hostclaim.Observed) (bool, error) { |
| ++ if r.FenceClosed == nil { |
| ++ return false, nil |
| ++ } |
| ++ return r.FenceClosed(ctx, obs) |
| ++} |
| ++ |
| ++// rowDecision is §5.3's decision for this box on a fresh read. |
| ++func (r Reconciler) rowDecision(ctx context.Context, obs hostclaim.Observed) (RowDecision, error) { |
| ++ closed, err := r.fenceClosed(ctx, obs) |
| ++ if err != nil { |
| ++ return RowDecision{}, err |
| ++ } |
| ++ return RelocationRow(obs.Claim, r.Box, true, closed), nil |
| ++} |
| ++ |
| + type activationReader struct { |
| + r Reconciler |
| + selected RecordSelection |
| +@@ -57,9 +83,15 @@ func (r Reconciler) check(ctx context.Context, obs hostclaim.Observed, selected |
| + if obs.Host != selected.Host || h.Key != selected.Key || h.Incarnation != selected.Incarnation || h.Family != selected.Family || h.Stamp == 0 { |
| + return ErrChanged |
| + } |
| +- if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { |
| ++ d, err := r.rowDecision(ctx, obs) |
| ++ if err != nil { |
| + return err |
| + } |
| ++ switch d.Case { |
| ++ case RowReady, RowHeld: |
| ++ default: |
| ++ return ErrChanged |
| ++ } |
| + if until, ok := obs.Claim.QuietAfter[r.Region]; ok && !r.Now().After(until) { |
| + return ErrChanged |
| + } |
| +@@ -100,6 +132,18 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| + if selected.DryRun { |
| + return nil |
| + } |
| ++ d, err := r.rowDecision(ctx, obs) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ closed, _ := r.fenceClosed(ctx, obs) |
| ++ if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || closed) { |
| ++ // No installation of ours to activate, or one whose admission |
| ++ // fence is closed (the source in releasing, still op.exec, which |
| ++ // With refuses): a kept row's hold only, under the flock. A target |
| ++ // before installing has no row and mints none (ruling 17). |
| ++ return r.holdOnly(ctx, selected, d) |
| ++ } |
| + // Activate's participant callback is not an admission validator for executors. |
| + // The reader validates EVERY fresh read while Activate holds hostguard. |
| + guard := r.Guard |
| +@@ -131,3 +175,32 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| + } |
| + return work.With(ctx, hostwrite.Identity{}, write) |
| + } |
| ++ |
| ++// holdOnly keeps an existing row held for the operation, with no |
| ++// installation authority: the write is conditional on the row's recorded |
| ++// ownership and its observed hold (6b's registry.SetClaimHold, through |
| ++// HoldRow), under the host's flock. The decision is remade on a fresh |
| ++// read under that lock. ErrChanged when there is no row: nothing is |
| ++// minted. |
| ++func (r Reconciler) holdOnly(ctx context.Context, selected RecordSelection, d RowDecision) error { |
| ++ if r.HoldRow == nil { |
| ++ return ErrUnsupported |
| ++ } |
| ++ return r.Guard.WithLocal(ctx, selected.Host, func() error { |
| ++ fresh, err := hostclaim.Read(ctx, r.Claims, selected.Host) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if fresh.Absent || fresh.Claim.Holder == nil || fresh.Claim.Holder.Incarnation != selected.Incarnation { |
| ++ return ErrChanged |
| ++ } |
| ++ again, err := r.rowDecision(ctx, fresh) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if again != d { |
| ++ return ErrChanged |
| ++ } |
| ++ return r.HoldRow(ctx, selected.Host, d.HoldOp) |
| ++ }) |
| ++} |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests** |
| + |
| +Run: `nice -n 19 go test ./internal/hostbox/ -count=1` |
| +Expected: PASS, main's reconcile tests included (no observer: today's behaviour). |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/hostbox/reconcile.go internal/hostbox/reconcile_relocation_test.go |
| +git commit -m "hostbox: reconcile decides a relocation row on a fresh read, observes the fence inside its lock, and keeps a fenced source's row held without installation authority |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +## Stage B re-plan (ruling 24) |
| + |
| +When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c, 9 and 10 against their merged code, then re-review. Every obligation rounds 1 to 4 placed on stage B, with the request that covers it and the test that proves it: |
| + |
| +| Obligation | Round | Covered by | Proved by | |
| +|---|---|---|---| |
| +| Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | |
| +| The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| +| `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| +| The relocation controller's cleanup scoped to its own partial work; `Clear` is `ClearStale` with its config | 3 (N5), 4 (5) | `relocationCleanup`, `OpenRelocationController` | `TestRelocationCleanupNeverStopsAnAdmittedWriter`, the on-disk eligibility test over the assembly | |
| +| P built under the location lease with authority re-checked at every write; files manifest first; the verifier reads both manifests, the pinned one from the retained copy | 1 (2, 5, 7), 3 (R1-2), 4 (R1-5) | `authorityStore`, `CopyPinnedManifest`, `PreparedFromInstallManifest`; R7 | `TestRelocationBuildStopsWhenAuthorityChanges`, `TestPreparedRefusesAPartialBuild` | |
| +| The local install under `install_stamp`; destination directories created and inventoried first; held staging through root's relay; never remade after P's admission | 1 (1), 4 (3, 4) | 6b's typed inventory; R9; moves Plan 1 Task 7b | `TestRelocationInstallIsBoundToTheInstallStamp`, `…CreatesAndInventoriesTheDestination`, `…WaitsForRootsHeldStaging` | |
| +| The probe: inputs open P once with `Prepared`, later attempts ordinary (empty included), selected as `"probe"` work, one attempt per `BeginProbeAttempt` | 1 (1), 2 (N1, N6), 3 (R1-13), 4 (2) | R1 (dispatch, `Probe`, `"probe"` action) | `TestRelocationWakeInputsOpenPOnceThenRestoreThePredecessor`, `TestASecondProbeAttemptPublishesNoMarkerBeforeItsRoot` | |
| +| Reconcile through 6b's adapter: hold writes conditional on ownership, the completed hold cleared, nothing minted for a box with no role, an unexplained row kept | 1 (14), 2 (N2, N4), 3 (N1, N4), 4 (N1) | `holdRowFor`, the adapter's hold switch; Task 9a's decision | `TestReconcileAdapterInstallFollowsTheRowCases`, `…HoldIsConditional`, `…ClearsTheHoldAfterCompletion` | |
| +| Production wiring: the policy and the fence observer bound to the verified actor, flag-gated; the passes with the sweep; create's chooser only with HA on | 2 (15), 3 (N3), 4 (N3) | R8 (`Extend`), `applyRegionHA`, `runRelocatePass` | `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, `TestRelocatePassIsOffUnlessBothFlagsAreOn` | |
| + |
| +Nothing in stage A changes for stage B's rewrite: the contracts stage B consumes (`hostrelocate.Effects`, `Prepared`, `Pinner`, `Lease`, `hostclaim`'s transitions, `RelocationRow`, `RelocationPolicy`, `AdmissionFenced`, `ReleasePositions`, `RelocationControllers`) are stage A's and tested there. |
| + |
| +--- |
| + |
| ### Task 8a: The release providers on the claim assembly |
| **Gated on:** claim Plan 6b (`docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md` on `origin/design-replica-ownership`, rev 5) merged to main, for `ClaimAssembly` (its `Wake *ClaimWakeAssembly`, `Guard`, `Claims`, `Reconcile`), `(*ClaimWakeAssembly).Producers`, `(*ClaimWakeAssembly).get(host) *claimWakeAttempt` and its `rep *lease.Replicator`, `(ClaimAssembly).localStopIdentity` returning `claimStopLocal{Actor, BootID, ControllerID, PID, ProcessStart}`, and `CollectClaimStop(ctx, path, StopRequest, StopDrainer)`; and on requests **R2** (a writers-only stop on the attempt) and **R3** (the replicator's descriptors recorded at open), which 6b does not declare. Not compiled in the dry run. |
| @@ -7723,8 +8391,9 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *Relocati |
| cfg.Now = time.Now |
| } |
| return &RelocationEffects{ |
| - Lifetime: cfg.Lifetime, |
| - Controllers: &RelocationControllers{}, |
| + Lifetime: cfg.Lifetime, |
| + Controllers: &RelocationControllers{Journal: filepath.Join(cfg.DataDir, "relocate", "finishing"), |
| + Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg.Exec; c.Binding = b; return execctl.ClearStale(ctx, c) }}, |
| Identity: func(ctx context.Context) (RelocationIdentity, error) { |
| l, err := a.localStopIdentity(ctx) // 6b: the verified actor, boot id and controller id |
| if err != nil { |
| @@ -7737,19 +8406,23 @@ func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *Relocati |
| c.Binding = b |
| // Ruling 20: this controller's cleanup is the relocation's own. |
| c.Cleanup = func(ctx context.Context) error { return a.relocationCleanup(ctx, cfg, b.Host, b.Op) } |
| - return cfg.Open(ctx, c) |
| + return OpenRelocationController(ctx, c) // Clear is ClearStale with this config (round 4 finding 5) |
| }, |
| Producers: func(ctx context.Context, o hostclaim.Observed) (StopDrainer, error) { return a.Wake.Producers(ctx, o) }, |
| Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { |
| + // Region is 6b's field and part of its fence identity: set on every |
| + // request from the verified actor (a box never changes region), so |
| + // the initial request and a recovery name the same stop. |
| + local, err := a.localStopIdentity(ctx) |
| + if err != nil { |
| + return StopResult{}, err |
| + } |
| + r.Region = local.Actor.Region |
| if _, ok := d.(RecoveryReleaseDrain); ok { |
| // A restarted agent finishing the recorded owner's stop: 6b's |
| // authenticated recovery path accepts another tuple for this |
| // box's session and replays the durable receipt (R1-9). |
| - local, err := a.localStopIdentity(ctx) |
| - if err != nil { |
| - return StopResult{}, err |
| - } |
| - r.Recover, r.Region = true, local.Actor.Region |
| + r.Recover = true |
| } |
| return CollectClaimStop(ctx, controlsock.Path, r, d) |
| }, |
| @@ -8338,6 +9011,16 @@ package host |
| // and never once any admission has named P. |
| func TestRelocationInstallIsBoundToTheInstallStamp(t *testing.T) { /* Guard record at another stamp: ErrStampMismatch; P opened: hostrelocate.ErrRefused; happy path: the receipt's InstallStamp is the journaled one and JournalSHA256 is the journal file's */ } |
| +// A fresh spare with no host directory and a non-empty files manifest: |
| +// the directories are created and inventoried at install_stamp before |
| +// Restore, and retirement finds them (round 4 finding 3). |
| +func TestRelocationInstallCreatesAndInventoriesTheDestination(t *testing.T) { /* no files dir; after install the dir exists, the inventory holds a files entry at the stamp, and 6b's census refuses an untracked sibling */ } |
| + |
| +// Under the edge unit's filesystem restrictions MaterializeHeld is |
| +// blocked: the receipt waits (ErrNotYet) until root's relay (R9) has |
| +// pointed live/<host> at the pinned release, then is recorded. |
| +func TestRelocationInstallWaitsForRootsHeldStaging(t *testing.T) { /* a read-only optDir; the relay served; the receipt only after LiveVersion == rt.Version */ } |
| + |
| // The probe's inputs: the first opening restores P from its prepared pin |
| // with KeepStamp, Prepared and the operation's coverage; a later attempt |
| // restores the predecessor with a fresh generation, no KeepStamp, coverage |
| @@ -8368,6 +9051,7 @@ import ( |
| "crypto/sha256" |
| "encoding/hex" |
| "encoding/json" |
| + "errors" |
| "fmt" |
| "os" |
| "path/filepath" |
| @@ -8381,6 +9065,9 @@ import ( |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| +// 6b's StopInventoryDB.EnsureDirectories is the typed inventory's directory |
| +// creation; if 6b's merged inventory names it otherwise, bind that name. |
| + |
| // relocationInstallJournal is the box's local record of its install steps |
| // for one operation (<DataDir>/relocate/<host>/<op>/install.json). Its |
| // digest is the receipt's journal_sha256. |
| @@ -8451,6 +9138,14 @@ func (a ClaimAssembly) relocationInstall(cfg RelocationAssemblyConfig) func(cont |
| if err := ensureDir(dir); err != nil { |
| return err |
| } |
| + // A fresh spare has no host directory yet (round 4 finding 3): |
| + // the files directory and the env path's parent are created |
| + // and inventoried under this installation's authority through |
| + // 6b's typed inventory (a "files" entry at install_stamp), so |
| + // retirement knows them; never an untracked MkdirAll. |
| + if err := a.Reconcile.Inventory.EnsureDirectories(ctx, o.Host, captured, stamp, cfg.FilesDir(o.Host), filepath.Dir(cfg.EnvPath(o.Host))); err != nil { |
| + return err |
| + } |
| m, _, err := hostfiles.Restore(ctx, cfg.Data, prefix, cfg.FilesDir(o.Host), origins) |
| if err != nil { |
| return err |
| @@ -8487,9 +9182,18 @@ func (a ClaimAssembly) relocationInstall(cfg RelocationAssemblyConfig) func(cont |
| return err |
| } |
| journal.Steps["render"] = rendered.SHA256 |
| - if _, err := release.MaterializeHeld(ctx, a.Claims, release.OptDir(), *rt, rt.Version); err != nil { |
| + // The edge unit cannot write optDir (ProtectSystem=strict): |
| + // MaterializeHeld returns ErrStageBlocked and the held stage |
| + // request is relayed to root, whose authorizer accepts a |
| + // relocation bound to this operation, executor, install stamp |
| + // and the pinned release (R9). The receipt is recorded only |
| + // once live/<host> names that version (release.LiveVersion). |
| + if _, err := release.MaterializeHeld(ctx, a.Claims, release.OptDir(), *rt, rt.Version); err != nil && !errors.Is(err, release.ErrStageBlocked) { |
| return err |
| } |
| + if live, err := release.LiveVersion(release.OptDir(), o.Host); err != nil || live != rt.Version { |
| + return fmt.Errorf("%w: the held release %s is not staged yet (live %q)", hostrelocate.ErrNotYet, rt.Version, live) |
| + } |
| journal.Steps["materialize"] = rt.Version |
| raw, err := json.Marshal(journal) |
| if err != nil { |
| @@ -8704,7 +9408,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| --- |
| -### Task 9: Reconcile keeps relocation rows held |
| +### Task 9: Reconcile keeps relocation rows held (stage B) |
| **Gated on:** claim Plan 6b merged, for `hostbox.ReconcileAdapter`, `registry.Route.HoldOp`, `registry.SetClaimHold`, `registry.ClaimOwnership.Valid` (which requires a non-zero stamp), `hostbox.RegistryHoldPolicy` and `provision.ReconcileConfig.ClaimInstall`/`ClaimRetire`. Task 3's `Policy.Check` action `reconcile` is un-gated and already lets a relocating participant through the write policy. Not compiled in the dry run. |
| @@ -8725,7 +9429,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Consumes: Task 3's `RelocationRow`, `Policy.Check`'s `reconcile` action; 6b's `ReconcileAdapter{Registry, Box, Reconciler}`, `registry.SetClaimHold(ctx, host, owner, expected, next string)`, `registry.ClaimOwnership`; main's `hostguard.Guard.WithLocal`. |
| -- Produces: `hostbox.Reconciler.FenceClosed func(host string) bool` (the source's local admission fence, from the wake assembly); `(Reconciler) rowDecision(ctx, obs) (RowDecision, error)`; `(ReconcileAdapter) holdRow(ctx, rec, d RowDecision) error`. |
| +- Produces: `(ReconcileAdapter) Configure` wiring `HoldRow` (`holdRowFor`), the adapter's hold writes after `write(owner)`. Task 9a (stage A) owns `Reconciler.FenceClosed`, `HoldRow`, `rowDecision` and `holdOnly`. |
| - [ ] **Step 1: Write the failing test** |
| @@ -8789,14 +9493,7 @@ func TestReconcileAdapterInstallFollowsTheRowCases(t *testing.T) { |
| if tc.fenceClosed { |
| f.beginFence() // root's durable BeginFence for the captured session; FenceClosed reads it through Guard.AdmissionFenced |
| } |
| - f.adapter.Reconciler.FenceClosed = func(host string) bool { |
| - captured, err := hostguard.Capture(f.guard.DataDir, host) |
| - if err != nil { |
| - return false |
| - } |
| - fenced, _ := f.guard.AdmissionFenced(context.Background(), host, captured) |
| - return fenced |
| - } |
| + f.adapter.Reconciler.FenceClosed = f.fenceClosed |
| err := f.adapter.Install(context.Background(), f.record(), f.write) |
| if !errors.Is(err, tc.wantErr) { |
| t.Fatalf("err %v, want %v", err, tc.wantErr) |
| @@ -8877,6 +9574,11 @@ func adapterFixture(t *testing.T, box string) *adapterTestFixture { |
| if err != nil { |
| t.Fatal(err) |
| } |
| + launches, err := OpenLaunchJournal(filepath.Join(dataDir, "launches")) // 6b's |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + optDir, unitsDir := t.TempDir(), t.TempDir() |
| guard := hostguard.Guard{DataDir: dataDir, Claims: hostguard.StoreReader{Store: claims}} |
| f := &adapterTestFixture{t: t, host: "shop.bab.oncarlos.com", box: box, claims: claims, registry: reg, guard: guard} |
| f.adapter = ReconcileAdapter{Registry: reg, Inventory: inv, Box: box, Enrollment: strings.Repeat(box[0:1], 32), DirectoryRevision: "4", |
| @@ -8884,8 +9586,10 @@ func adapterFixture(t *testing.T, box string) *adapterTestFixture { |
| Writes: hostwrite.Config{Enabled: true, Selector: WorkSource{Guard: guard, Box: box, Region: "ie", Now: time.Now, Policy: Policy{Held: RegistryHoldPolicy(reg), Relocation: RelocationPolicy(box)}.Check, Local: func(context.Context, *hostclaim.Claim, Selection) error { return nil }}}, |
| CheckRecord: func(context.Context, RecordSelection) error { return nil }, |
| CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil }}, |
| - Census: &EmptyInstallationCensus{Inventory: inv, Guard: guard, Box: box}} |
| + Census: &EmptyInstallationCensus{Inventory: inv, Registry: reg, Launches: launches, Box: box, Enrollment: strings.Repeat(box[0:1], 32), DirectoryRevision: "4", |
| + Roots: map[string]string{"data": dataDir, "opt": optDir, "units": unitsDir}, SharedPaths: map[string][]string{}}} // 6b's fields, nothing else |
| f.adapter.Reconciler.HoldRow = holdRowFor(f.adapter) |
| + f.adapter.Reconciler.FenceClosed = f.fenceClosed |
| created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| record := provision.Record{Schema: 1, Account: "bab", App: "jam", Host: f.host, Slug: "shop", Channel: "edge", Region: "ie", CreatedAt: created} |
| body, _ := json.Marshal(record) |
| @@ -8981,6 +9685,16 @@ func (f *adapterTestFixture) withExistingRow(hold string) { |
| func (f *adapterTestFixture) record() provision.ClaimRecord { return f.rec } |
| +// fenceClosed is Task 10's observer: the captured installation's fence, |
| +// read without the lock, on the observation given. |
| +func (f *adapterTestFixture) fenceClosed(_ context.Context, o hostclaim.Observed) (bool, error) { |
| + captured, err := hostguard.Capture(f.guard.DataDir, o.Host) |
| + if err != nil { |
| + return false, err |
| + } |
| + return f.guard.AdmissionFenced(o.Host, o.Claim, captured) |
| +} |
| + |
| // beginFence closes this box's admission fence for the captured session, |
| // as root's stop collector does (hostguard.BeginFence). |
| func (f *adapterTestFixture) beginFence() { |
| @@ -9009,8 +9723,7 @@ func (f *adapterTestFixture) write(owner registry.ClaimOwnership) error { |
| func TestReconcileAdapterHoldIsConditional(t *testing.T) { |
| f := adapterFixture(t, "src") |
| f.claimAt(hostclaim.PhasePinning, "none") |
| - f.withExistingRow() |
| - f.adapter.Reconciler.FenceClosed = func(string) bool { return true } |
| + f.withExistingRow("") |
| if err := f.adapter.Install(context.Background(), f.record(), f.write); err != nil { |
| t.Fatal(err) |
| } |
| @@ -9028,75 +9741,7 @@ Expected: FAIL to compile: `f.adapter.Reconciler.FenceClosed undefined`. |
| - [ ] **Step 3: The decision in `Reconciler`** |
| -In `internal/hostbox/reconcile.go`, add the field `FenceClosed func(host string) bool` to `Reconciler`, and replace in `Reconciler.check` |
| - |
| -```go |
| - if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { |
| - return err |
| - } |
| -``` |
| - |
| -with |
| - |
| -```go |
| - switch RelocationRow(obs.Claim, r.Box, true, r.fenceClosed(obs.Host)).Case { |
| - case RowReady, RowHeld: |
| - default: |
| - return ErrChanged |
| - } |
| -``` |
| - |
| -and in `Reconciler.Install`, replace the activation |
| - |
| -```go |
| - if err := guard.Activate(ctx, selected.Host, r.Box, nil); err != nil { |
| - return err |
| - } |
| -``` |
| - |
| -with |
| - |
| -```go |
| - d := RelocationRow(obs.Claim, r.Box, true, r.fenceClosed(selected.Host)) |
| - if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || r.fenceClosed(selected.Host)) { |
| - // No installation of ours to activate, or one whose admission |
| - // fence is closed (the source in releasing, still op.exec, which |
| - // With refuses): a kept row's hold only, under the flock. A target |
| - // before installing has no row and mints none (ruling 17). |
| - return r.holdOnly(ctx, selected, d) |
| - } |
| - if err := guard.Activate(ctx, selected.Host, r.Box, nil); err != nil { |
| - return err |
| - } |
| -``` |
| - |
| -with |
| - |
| -```go |
| -func (r Reconciler) fenceClosed(host string) bool { return r.FenceClosed != nil && r.FenceClosed(host) } |
| - |
| -// holdOnly keeps an existing row held for the operation, with no |
| -// installation authority: the write is conditional on the row's recorded |
| -// ownership and its observed hold (registry.SetClaimHold), under the |
| -// host's flock. ErrChanged when there is no row: nothing is minted. |
| -func (r Reconciler) holdOnly(ctx context.Context, selected RecordSelection, d RowDecision) error { |
| - if r.HoldRow == nil { |
| - return ErrUnsupported |
| - } |
| - return r.Guard.WithLocal(ctx, selected.Host, func() error { |
| - fresh, err := hostclaim.Read(ctx, r.Claims, selected.Host) |
| - if err != nil { |
| - return err |
| - } |
| - if again := RelocationRow(fresh.Claim, r.Box, true, r.fenceClosed(selected.Host)); again != d { |
| - return ErrChanged |
| - } |
| - return r.HoldRow(ctx, selected.Host, d.HoldOp) |
| - }) |
| -} |
| -``` |
| - |
| -and the field `HoldRow func(ctx context.Context, host, holdOp string) error` on `Reconciler`, which 6b's adapter supplies (below). The retiring and unexplained cases never reach `Install`: `check` refuses them, and `provision`'s claim branch keeps the row. |
| +Task 9a (stage A) built it: `Reconciler.check` decides with `RelocationRow` on a fresh read, `FenceClosed(ctx, obs) (bool, error)` is consulted inside the locked operation and an error keeps rows, and a held row whose box is not the executor or whose admission fence is closed goes through `holdOnly` under `WithLocal` with the adapter's `HoldRow`. Nothing here changes `reconcile.go` again. |
| - [ ] **Step 4: The adapter's hold writes** |
| @@ -9112,7 +9757,11 @@ In 6b's `ReconcileAdapter.Install`, after the row is written through `write(owne |
| if current.Absent || current.Claim.Holder == nil || current.Claim.Holder.Incarnation != captured.Incarnation || current.Claim.Holder.Stamp != captured.Stamp { |
| return ErrChanged |
| } |
| - d := RelocationRow(current.Claim, a.Box, true, a.Reconciler.fenceClosed(rec.Host)) |
| + closed, err := a.Reconciler.fenceClosed(ctx, current) |
| + if err != nil { |
| + return err |
| + } |
| + d := RelocationRow(current.Claim, a.Box, true, closed) |
| switch { |
| case d.Case == RowHeld && row.HoldOp != d.HoldOp: |
| // Held for the running operation; an earlier operation's hold is |
| @@ -9289,29 +9938,36 @@ func runRelocatePass(ctx context.Context, store release.Store, w hostrelocate.Wo |
| } |
| ``` |
| -`applyRegionHA` is Task 10's and runs at request R8's insertion point (after `OpenProductionClaimAssembly`, before `Build` captures `Policy.Check`), gated on the flag: |
| +`applyRegionHA` is Task 10's and is the body of request R8's `Extend` hook, which 6b's `Build` calls after actor verification and before it captures `Policy.Check` (round 4 N3: the adapter's box is set only there), gated on the flag: |
| ```go |
| // applyRegionHA gives the assembly region HA's two providers (round 3 |
| // finding N3), only with CARLOS_REGION_HA on; off, the assembly is 6b's. |
| -func applyRegionHA(a *ClaimAssembly, getenv func(string) string) { |
| - if !regionHAEnabled(getenv) || a == nil { |
| - return |
| - } |
| - a.Policy.Relocation = hostbox.RelocationPolicy(a.Reconcile.Box) |
| - guard := a.Guard |
| - a.Reconcile.Reconciler.FenceClosed = func(host string) bool { |
| - captured, err := hostguard.Capture(guard.DataDir, host) |
| - if err != nil { |
| - return false // no installation of ours: nothing to be fenced |
| +// It is ClaimAssembly.Extend (R8), called with the verified actor. |
| +func applyRegionHA(getenv func(string) string) func(*ClaimAssembly, hostbox.Actor) { |
| + return func(a *ClaimAssembly, actor hostbox.Actor) { |
| + if !regionHAEnabled(getenv) || a == nil || actor.Box == "" { |
| + return |
| + } |
| + a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box) |
| + guard := a.Guard |
| + // Read inside the caller's locked operation, never reacquiring the |
| + // host lock (round 4 finding 1); an error keeps rows and mints none. |
| + a.Reconcile.Reconciler.FenceClosed = func(_ context.Context, o hostclaim.Observed) (bool, error) { |
| + captured, err := hostguard.Capture(guard.DataDir, o.Host) |
| + if errors.Is(err, hostguard.ErrNoRecord) { |
| + return false, nil // no installation of ours: nothing to be fenced |
| + } |
| + if err != nil { |
| + return false, err |
| + } |
| + return guard.AdmissionFenced(o.Host, o.Claim, captured) |
| } |
| - fenced, err := guard.AdmissionFenced(context.Background(), host, captured) |
| - return err == nil && fenced |
| } |
| } |
| ``` |
| -Its test, `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, builds a `ClaimAssembly` with 6b's test wiring, applies with the flag off (both nil, `Build` behaves as 6b's) and on (both set, `Policy.Check` admits a relocating participant's `reconcile`). |
| +Its test, `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, builds a `ClaimAssembly` with 6b's test wiring and `Extend: applyRegionHA(getenv)`, runs `Build` with the flag off (both nil, `Build` behaves as 6b's) and on (both set, bound to the verified actor's box; `Policy.Check` admits a relocating participant's `reconcile` and the target's `probe`), and reconciles one host under a deadline with the real observer. |
| In `internal/host/agent.go`, next to `go runHostFencePass(agentCtx)`, build the worker and participant from the assembly once it exists and start `go runRelocatePass(agentCtx, store, worker, participant, serialChanges)`. The worker's providers: |
| @@ -9396,7 +10052,7 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| ## What the dry run could not check |
| -- Tasks 8a, 8b, 8c, 9 and 10 were not compiled: they bind to Plan 6b rev 5's declared names, moves Plan 2 Task 8's `BuildGeneration` and key helpers, moves Plan 1 Task 7b's `RenderHeld` and `MaterializeHeld`, claim Plan 5's app gate and create door, and requests R1 to R3 and R7, none on main at 3407d114. |
| +- Stage B (Tasks 8a, 8b, 8c, 9 and 10) was not compiled: it binds to Plan 6b rev 5's declared names, moves Plan 2 Task 8's `BuildGeneration` and key helpers, moves Plan 1 Task 7b's `RenderHeld` and `MaterializeHeld`, claim Plan 5's app gate and create door, and requests R1 to R3 and R7, none on main at 3407d114. |
| - `StopWriters` against a live attempt (R2), root's writers-stopped endpoint (R3) and the probe through 6b's runtime (R1): their un-gated halves are tested with recorders and the fixture `/proc`; the live tests, under the production uid, are Tasks 8a and 8c's. |
| - `WritersStopped` against a live agent with an embedded Litestream replicator. Its tests use a fixture `/proc`; the descriptor identification at replicator open belongs to Task 8. |
| - `genselect.Select` inside `StorePinner` against a real versioned bucket. The pin tests stub `SelectDB`. |
| @@ -9411,3 +10067,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 2 guards**, each removed in turn and its test watched to fail: the pin key bound to the claim generation; `ToInstalled`'s files manifest; the empty pin's evidence (claim); `stuck` before the phase table; the retained automatic loss; the lease lost before the pin CAS; the release bounds before the write (two); the empty pin's evidence (pinner); the pinned manifest's digest; the copied entries' equality; the target's eligibility at start; the session-0 drain refusal; `stuckOn` from `probe-failed`; the release deadline; the participant's readiness gate; `StopWriters` first in the drain; the replayed receipt without a record; the positions' session binding; the controller under the agent lifetime; the wake state in the flush; `Permits`; today's create without a selection; the `reconcile` action's participant and lost checks; the console's chooser off with HA off. Twenty-five in all. |
| - **Rev 3 guards**, each removed in turn and its test watched to fail: the probe attempt bound; discharge through `CompleteObligation`; the former writer's last admission; the per-process descriptor scan; exhaustion before a spare is chosen; the retained manifest's digest; the attempt counted before the probe; the sweep's running-operation check; the recorded identity after a restart; the positions' box binding; `Finish` forgetting the controller. Eleven in all. |
| - **Rev 4 guards**, each removed in turn and its test watched to fail: the acknowledgement barrier on `releasing → pinning`; `FenceExhaustedTarget`'s exhaustion check; the worker's use of it over `MarkStuck`; the serving branch's obsolete-hold clearance; `RelocationPolicy`'s executor check; `AdmissionFenced`'s active fence; `Finish` keeping its entry until `Close` succeeds; the recorded directory revision after a restart. Eight in all. |
| +- **Rev 5 guards**, each removed in turn and its test watched to fail: the lock-free fence observer (re-locking deadlocks the bounded test); the `probe` action in `Policy.Check`; the HA gate on the obsolete-hold clearance; `Finish`'s durable clear; the journal's retry after a restart; the hold-only path for a fenced executor; the observer error keeping rows. Seven in all. |