amadan

carlos / platform Public

plan: region HA 1b-next, relocation released by stop evidence or a termination fence, reconcile row cases, create's once-only executor choice, and fences consumed in claim CASes

Tasks 1 to 7 dry-run on 3407d114 from the plan's own text; Tasks 8 to 10 are gated on claim Plan 6b (and moves Plan 2, claim Plan 5's app gate and create door); Task 11 is the L18 merge gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude d2b913c9b1e0bf48d9d04e0f9154731973c8a71e parent 3407d11
1 file changed, +5449 −0
  • docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md +5449 −0

CI cancelled — superseded by a newer push — run details

diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md
new file mode 100644
index 000000000..aa3f8a365
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md
@@ -0,0 +1,5449 @@
+# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18).
+
+**Architecture:** Eleven tasks. Seven are buildable on main today and were dry-run: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). Three are gated on claim Plan 6b's box assembly, which is not merged: binding the box half to that assembly, wiring reconcile's held rows, and switching the passes on. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader.
+
+**Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`).
+
+**Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds.
+
+## Rulings made while planning
+
+Each was a conflict between the authorities or a gap in them. Each says what it costs if it is wrong.
+
+1. **Relocation lives in `internal/hostclaim`.** Main already has `KindRelocating`, its phase validator and `ConvertMovingToRelocating` there, and Plan 6b's boundary table gives "`relocating`, released coverage, once-selected executor and two-box witnesses" to region HA. One implementation per concept: no second package grows claim transitions. *Cost if wrong:* the claim session moves the functions; the names stay.
+2. **The relocation release's stop request names the obligation `relocation-release`.** Root's `FenceAndStop` requires an obligation component, and 6b's `collectCapturedStop` builds only retirement stops (it looks for this box's `park-local` under the operation). A release retires nothing. `host.StopReceiptReleases` accepts only receipts carrying `relocation-release`, so a retirement stop can never be offered as a release, nor the reverse. *Cost if wrong:* a one-line constant, and the 6b session may prefer another literal.
+3. **Discharge stays off until the roster records an enrollment time.** `boxvolumes.Discharge` needs the enrollment time (§24 item 4) and `hostactors.Box` carries none. The production `Discharge` provider returns "the roster records no enrollment time", so a fenced box's `park-local` chores stay open for an operator. H19: this costs cleanup, never recovery. *Ask of the claim session:* add `enrolled_at` to box entries (D7); then the provider reads `boxvolumes.List` and calls `boxvolumes.Discharge`.
+4. **A box-started relocation needs claim §5.1's app-admission registration**, with a `{box, enrollment, controller}` owner. Main has no `apps/<acct>/<app>/admission.json` registry yet. `hostrelocate.Worker` takes an `Admission` interface and refuses to start without one (`ErrUnsupported`). Task 10 is gated on claim Plan 5's app gate as well as 6b. *Cost if wrong:* none until that lands; nothing starts.
+5. **The pinned recovery set is an immutable object** at `control/relocations/<host>/<op>/pin-<n>.json`, written create-only and named in the journal by key and SHA-256 (spec §18's first correction: the store has no version-id read, so content is identified by digest). The files pin names its manifest by prefix, sequence and digest. *Cost if wrong:* a deny statement for that prefix in `deployment-buckets` is a follow-up; a rewritten pin is already caught, because every reader checks the journaled digest.
+6. **`installed` is the worker's CAS**, made once a `Prepared` verifier reads P's install manifest complete (§5.8's phase table). The target enters `probing` itself. *Cost if wrong:* who makes one CAS; the claim rules are unchanged.
+7. **A converted move names its target in `pinning`** (`hostclaim.NameTarget`), before `installing`, so the target acknowledges (D6) before it is handed `op.exec` (moves §7.10: "the replacement box `choose()` names when the relocation leaves `pinning`").
+8. **The final TXID N is read from the remote after `SyncAndWait` returns with every writer stopped**, as the newest level-0 `MaxTXID` (§5.8 release step 2). With writers stopped and the sync acknowledged, the stopped database's position and that listing agree; a sync error is returned, never swallowed (claim P6-D4).
+9. **`stuck` is a journal entry, not a phase.** The claim says exhaustion "records `stuck` without resetting the count"; `validateRelocating` lists no `stuck` phase. `MarkStuck` writes `journal.stuck` and every relocation transition refuses afterwards; it refuses between `installed` and a probe.
+10. **The probe deadline starts at the `probing` CAS** (`journal.probing_since`), and the worker moves a live target to `probe-failed` 10 minutes later.
+11. **A host whose source never had a writer** (session 0, `writer_box` empty) is relocated with coverage `""`, which declares nothing (claim §6.4a: "a created incarnation's first admission … declares nothing"), never `released`.
+12. **L18's owner.** The coordination post on #236 says moves owns `internal/sim/hostlife`; claim rev 6.30 §1.1 calls the two-box model "region HA's own … as part of region HA's plan"; the moves conflicts note defers it to "a joint round with the region HA session". This plan writes no model and ends with a merge gate that names the cases; whichever session builds the model, relocation does not merge until that round passes for a one-box and a two-box region.
+13. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1), which is what this plan's install already assumes. Nothing in L14 to L17 changed against it.
+
+## Why this is 1b-next, and what it leaves out
+
+Plan 1b built every part of rev 5.4 that needed none of L1 to L20. Since then claim Plans 1 to 8b and 5 landed on main (3407d114): the roster (`hostactors.Directory`), the fence writer (`hostfence.TerminationWorker`), the fence reader for claims (`hostclaim.VerifyFence`, `hostfence.ClaimFenceAuthority`), the execution controller (`hostexec`), stop receipts (`host.ClaimStopCoordinator`), the box-started census (`Directory.CensusBoxRelocation`) and H14's start check (`Directory.CheckBoxStart`). This plan uses all of them and duplicates none.
+
+Left out, with where it goes:
+
+- **The placer loop that decides what to relocate** (spec 3): reading the termination queue, relocating every host off a dead box in bounded parallel, rebalancing. This plan's worker starts a relocation when asked and replays any it finds; Task 10's pass only replays.
+- **Automatic enrollment and the cross-region confirmer** (spec 4, H16, H17).
+- **The console's copy of a fence into a D4 loss record and the roster's retirement of the box** (claim D4 "Automatic loss persistence"). The fence CAS keeps the reference and the discharge verdict in the operation journal and puts the box on `lost`; materializing the loss record is the console's, and is listed under "After this plan".
+- **Storage policy for `control/relocations/`** (ruling 5).
+
+## Global Constraints
+
+- **Dormant.** Nothing changes on a box or console where `CARLOS_REGION_HA` is unset or not true (`strconv.ParseBool`), and it is read from `/etc/carlos/host.env` by `regionHAEnabled` (`internal/host/boxrecord.go`). With one box per region every rule is a no-op (spec §4, §8 step 2).
+- **Activation gate (claim §1.1, L18).** "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) … validate[s] them." Relocation may not merge until Task 11's joint round passes.
+- **One implementation per concept; main's wins (#236).** Do not write a termination-fence writer (`hostfence.TerminationWorker` writes `control/box-fences/<enrollment>.json` using `termfence.Verify`). Do not write a roster reader (`hostactors.Directory`). Read fences only through `hostclaim.VerifyFence` (in claim CASes) and `hostactors.ValidateTerminationFence` (before them).
+- **H19, verbatim:** "The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores." No volume record gates a fence, a relocation step or an admission.
+- **Absence is never evidence (§5.9).** A fence that cannot be read or does not validate is an error, never "not fenced"; a box missing from the roster refuses.
+- **Every phase change is one claim CAS conditional on `op.id`, the phase and the ETag (claim C2).** Two workers racing make the same CAS and one gets `hostclaim.ErrConflict`.
+- **Exact values.** Failover pin delay **30 seconds** after the fence's `observed_at` (`hostclaim.FailoverPinDelay`); probe deadline **10 minutes** (`hostrelocate.ProbeDeadline`); at most **2** target replacements per operation (`hostclaim.MaxTargetReplacements`); reconcile's kept-row sweep **10 minutes** (H9); create intent keys **`selected_executor`** and **`executor_enrollment`** (claim P5-D6); journal fence key **`fence.<box>`** holding `{"enrollment","key","sha256"}` (claim §7.3); release obligation **`relocation-release`**; pin objects **`control/relocations/<host>/<op>/pin-<n>.json`**.
+- **Gated tasks.** Tasks 8, 9 and 10 start with a "Gated on" line. Their code is written against the interfaces claim Plan 6b rev 4 declares (`ClaimAssembly`, `ClaimWakeAssembly`, `CollectClaimStop`, `hostbox.ReconcileAdapter`, `registry.SetClaimHold`) and moves Phase 1 Plan 2 Task 8 declares (`moves.BuildGeneration`). They were not compiled. When 6b lands, re-read its merged code and bind the same contracts to the names it actually has; the un-gated tasks do not change.
+- **No new module dependencies. Do not add a method to `release.Store`.** Linux-only syscalls in `*_linux.go` with a `!linux` stub; CI cross-compiles darwin, windows and linux/arm64.
+- **Copy.** No user-facing copy. Errors and log lines are operator-facing and follow the surrounding style. If a task finds it needs console or CLI text, stop: it waits for Paul's copy review.
+- **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`.
+- **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.)
+- Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11.
+- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (listed in the Self-review). Then the code was removed; only this document is committed.
+
+## Review Focus
+
+1. **`released` coverage naming the wrong predecessor.** A drain's first probe admission declares nothing; if the session it vouches for is not exactly the one whose release was journaled (another box, another session, another generation), a real loss goes unrecorded. Expect refusal. Task 1, `TestRelocationAdmissionRules/released_names_another_predecessor`, `…/released_names_another_predecessor_box`, `…/released_after_P_was_opened`.
+2. **P opened twice.** A restarted probe that restores P again loses everything attempt two replicated (B1-1). Expect every attempt after the first to take a fresh generation, and P refused. Task 1, `TestRelocationAdmissionRules/a_third_attempt_reopens_P`.
+3. **A failover pinned too early.** Requests S3 accepted from the terminated box can still land; a pin taken before `observed_at` + 30 s widens H3's window. Expect the worker to wait and the claim to refuse. Task 1 `TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss`, Task 5 `TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence`.
+4. **A fence that cannot be read taken as no fence.** A garbled `control/box-fences/` object must stop the drain and the create choice, not let them proceed as if the box were alive. Task 2 `TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/an_unreadable_fence`, Task 5 `TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence`.
+5. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`.
+
+## File Structure
+
+| Path | Responsibility | Task |
+|---|---|---|
+| `internal/hostclaim/relocation.go` | The relocation transitions: start (drain, failover), source loss, barrier, release journal, pin, install, local install receipt, probe, serving, target replacement, stuck; fences consumed in the same CAS | 1 |
+| `internal/hostclaim/generation.go` | `AdmitWriter`'s relocating phase table and `released` coverage (L1, L9) | 1 |
+| `internal/hostclaim/create.go` | `ExecutorSelection`, `BeginCreateOn`; `CompleteCreate` never substitutes | 2 |
+| `internal/hostops/executor.go`, `loads.go`, `operations.go`, `backfill.go`, `backfill_controller.go` | `ExecutorChooser` over the roster, the edges snapshot and claim loads; `Authority.SelectExecutor` before the create CAS | 2 |
+| `internal/hostbox/relocationrows.go`, `wakepolicy.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner | 3 |
+| `internal/stopevidence/writers*.go` | L10's writers-stopped observation | 4 |
+| `internal/hostrelocate/worker.go`, `pin.go` | The regional worker: start, and one replayed CAS per step; the pinned recovery set | 5 |
+| `internal/hostrelocate/participant.go` | A participant box's transitions, ordered over its local effects | 6 |
+| `internal/host/relocationreleases.go`, `relocationeffects.go` | Stop receipts as release evidence; the release sequence inside root's stop collection | 7 |
+| `internal/host/relocationassembly.go` | Gated: the effects bound to 6b's assembly; the `Prepared` verifier | 8 |
+| `internal/hostbox/reconcile.go`, 6b's `reconcile_adapter.go` | Gated: held rows and the 10-minute sweep | 9 |
+| `internal/console/hostclaimexecutor.go` | The console's create chooser (constructed, not yet used by a door) | 2 |
+| `internal/host/relocatepass.go` | Gated: the passes, behind `CARLOS_REGION_HA`, and the create door's chooser | 10 |
+
+---
+
+### Task 1: The `relocating` operation in the claim library (L1, L2 to L4, L9, L11)
+
+Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.<box>`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies.
+
+**Files:**
+- Create: `internal/hostclaim/relocation.go`
+- Modify: `internal/hostclaim/generation.go` (the `AdmitWriter` kind switch, the coverage switch, a new `relocationAdmission`)
+- Test: `internal/hostclaim/relocation_test.go`
+
+**Interfaces:**
+- Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`.
+- Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin`, `PreparedAuthority`, `LocalInstall`, `Probe`, `Target{Box, Enrollment}`; functions `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`.
+- Journal keys later tasks read: `exec_ack.<box>`, `fence.<box>`, `fence.<box>.observed_at`, `discharge.<box>`, `reason`, `release`, `release.phase`, `released_by_target`, `pin`, `pins`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe`, `replacements`, `excluded_targets`, `stuck`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostclaim/relocation_test.go`:
+
+```go
+package hostclaim
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxid"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/termfence"
+)
+
+const (
+ relSource = "adi-knitting"
+ relTarget = "bri-knitting"
+ relSpare = "cai-knitting"
+ relSpare2 = "dov-knitting"
+)
+
+var relObserved = time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)
+
+func relEnrollment(box string) string { return strings.Repeat(string(box[0]), 32) }
+
+// relAuthority accepts any fence record for a box in this region.
+type relAuthority struct{}
+
+func (relAuthority) ValidateFence(_ context.Context, f termfence.Record) (FenceBinding, error) {
+ if f.Enrollment != relEnrollment(f.Box) {
+ return FenceBinding{}, ErrFence
+ }
+ return FenceBinding{Region: "ie", RosterRevision: 9, RosterETag: "roster"}, nil
+}
+
+type relReleases struct{ err error }
+
+func (r relReleases) VerifyRelease(context.Context, Claim, Release) error { return r.err }
+
+func putFence(t *testing.T, s release.Store, box string) FenceRef {
+ t.Helper()
+ f := termfence.Record{V: 1, Box: box, Enrollment: relEnrollment(box), State: "terminated",
+ Cloud: boxid.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box},
+ ObservedAt: relObserved.Format(time.RFC3339Nano), Via: "describe", RequestID: "req",
+ WrittenBy: termfence.WrittenBy{Box: "worker", Enrollment: strings.Repeat("e", 32)}}
+ raw, err := json.Marshal(f)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(raw)
+ ref := FenceRef{Enrollment: f.Enrollment, Key: "control/box-fences/" + f.Enrollment + ".json", SHA256: hex.EncodeToString(sum[:])}
+ if _, err := s.Put(context.Background(), ref.Key, raw, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
+ t.Fatal(err)
+ }
+ return ref
+}
+
+// relServing is a serving claim on relSource whose source has admitted one
+// cold writer session (generation g1).
+func relServing(t *testing.T) (release.Store, Observed) {
+ t.Helper()
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ o := servingClaim(t, ctx, s)
+ o, err := Update(ctx, s, o, AdmitWriter(Admission{Box: relSource, Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true}))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return s, o
+}
+
+func relStart(reason string, fence *FenceRef) RelocationStart {
+ return RelocationStart{OpID: "rel-1", By: "worker", Now: relObserved, Region: "ie", Reason: reason,
+ SourceBox: relSource, SourceEnrollment: relEnrollment(relSource),
+ TargetBox: relTarget, TargetEnrollment: relEnrollment(relTarget),
+ DirectoryRevision: 9, AdmissionID: "adm-1", Fence: fence}
+}
+
+func step(t *testing.T, s release.Store, o Observed, tr func(*Claim) error) Observed {
+ t.Helper()
+ next, err := Update(context.Background(), s, o, tr)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return next
+}
+
+func ack(box string) func(*Claim) error {
+ return AcknowledgeExecution(box, ExecutionAcknowledgement{Op: "rel-1", Enrollment: relEnrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: relObserved.Add(time.Hour)})
+}
+
+func relPin(maxTXID uint64, seq int64, at time.Time) RecoveryPin {
+ return RecoveryPin{Key: "control/relocations/pin.json", SHA256: strings.Repeat("a", 64), Generation: "g1", MaxTXID: maxTXID, FilesManifest: "instances/x/files/g1/manifest.json", FilesSeq: seq, PinnedAt: at}
+}
+
+func localInstall(c *Claim) LocalInstall {
+ return LocalInstall{Box: c.Op.Exec, Enrollment: relEnrollment(c.Op.Exec), InstallStamp: journalUint(c.Op, "install_stamp"), FilesManifestVersion: 3, JournalSHA256: strings.Repeat("b", 64)}
+}
+
+// toProbing drives an operation at pinning (with its pin) through
+// installing, installed and the local install receipt into probing.
+func toProbing(t *testing.T, s release.Store, o Observed, prepared string, maxTXID uint64) Observed {
+ t.Helper()
+ target := Target{Box: o.Claim.Op.Intent["target_box"], Enrollment: o.Claim.Op.Intent["target_enrollment"]}
+ o = step(t, s, o, ToInstalling("rel-1", target, prepared))
+ o = step(t, s, o, ToInstalled("rel-1", PreparedAuthority{Generation: prepared, InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: maxTXID}))
+ o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
+ return step(t, s, o, ToProbing("rel-1", relObserved))
+}
+
+func TestRelocationDrainEndToEnd(t *testing.T) {
+ ctx := context.Background()
+ s, o := relServing(t)
+ o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if c := o.Claim; c.Op.Exec != relSource || c.Op.Phase != PhaseStarting || !slices.Equal(c.Op.Participants, []string{relSource, relTarget}) || c.Op.Intent["stamp_from"] == "" {
+ t.Fatalf("start: %+v", c.Op)
+ }
+ if _, err := Update(ctx, s, o, ToReleasing("rel-1")); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("barrier without acknowledgements: %v", err)
+ }
+ o = step(t, s, o, ack(relSource))
+ o = step(t, s, o, ack(relTarget))
+ o = step(t, s, o, ToReleasing("rel-1"))
+ h := o.Claim.Holder
+ rel := Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "control/stop/r.json", ReceiptSHA256: strings.Repeat("d", 64)}
+ o = step(t, s, o, JournalRelease("rel-1", rel))
+ if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{err: errors.New("bad receipt")}, nil); err == nil {
+ t.Fatal("pinned on an unverified receipt")
+ }
+ if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil {
+ t.Fatal(err)
+ }
+ stamp := o.Claim.Holder.Stamp
+ if o.Claim.Op.Exec != "none" || o.Claim.Op.Phase != PhasePinning {
+ t.Fatalf("pinning: %+v", o.Claim.Op)
+ }
+ if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(39, 7, relObserved))); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("pin short of the release: %v", err)
+ }
+ o = step(t, s, o, JournalPin("rel-1", relPin(40, 7, relObserved)))
+ if o.Claim.Holder.Stamp != stamp {
+ t.Fatal("pinning changed the stamp")
+ }
+ o = toProbing(t, s, o, "P1", 40)
+ install := journalUint(o.Claim.Op, "install_stamp")
+ if o.Claim.Op.Exec != relTarget || o.Claim.Holder.Stamp != install || !slices.Contains(o.Claim.Touched.Boxes, relTarget) {
+ t.Fatalf("installation: exec %s stamp %d install %d", o.Claim.Op.Exec, o.Claim.Holder.Stamp, install)
+ }
+ o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
+ last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
+ if last.Coverage != "released" || last.WritesLost != nil || o.Claim.Holder.Stamp != install {
+ t.Fatalf("released admission: %+v stamp %d", last, o.Claim.Holder.Stamp)
+ }
+ o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session}))
+ o = step(t, s, o, CompleteRelocation("rel-1", counter()))
+ c := o.Claim
+ if c.Op.Kind != KindServing || c.Op.ID != "" || c.Op.Exec != relTarget || !slices.Equal(c.Op.Participants, []string{relTarget}) {
+ t.Fatalf("serving: %+v", c.Op)
+ }
+ if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-local" || c.Obligations[0].Box != relSource || c.Obligations[0].Op != "rel-1" {
+ t.Fatalf("the source's park-local: %+v", c.Obligations)
+ }
+}
+
+func TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss(t *testing.T) {
+ ctx := context.Background()
+ for _, qualified := range []bool{false, true} {
+ s, o := relServing(t)
+ o = step(t, s, o, func(c *Claim) error { // a chore the dead source already owed
+ c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource})
+ return nil
+ })
+ ref := putFence(t, s, relSource)
+ in := relStart(ReasonFailover, &ref)
+ in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}}
+ o, err := StartRelocation(ctx, s, o, in, relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ c := o.Claim
+ if c.Op.Exec != "none" || !slices.Equal(c.Op.Participants, []string{relTarget}) || !slices.Contains(c.Lost, relSource) || c.Op.Journal["fence."+relSource] == "" {
+ t.Fatalf("failover start: %+v lost %v", c.Op, c.Lost)
+ }
+ if owes := slices.ContainsFunc(c.Obligations, func(o Obligation) bool { return o.Box == relSource }); owes == qualified {
+ t.Fatalf("discharge qualified=%v left chores %+v", qualified, c.Obligations)
+ }
+ if _, err := Update(ctx, s, o, ToReleasing("rel-1")); err == nil {
+ t.Fatal("a failover entered releasing")
+ }
+ o = step(t, s, o, ack(relTarget))
+ o = step(t, s, o, PinFailover("rel-1"))
+ if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(40, 7, relObserved.Add(29*time.Second)))); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("pinned inside the 30-second window: %v", err)
+ }
+ o = step(t, s, o, JournalPin("rel-1", relPin(40, 7, relObserved.Add(FailoverPinDelay))))
+ o = toProbing(t, s, o, "P1", 40)
+ probe := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}
+ if _, err := Update(ctx, s, o, AdmitWriter(probe)); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("a failover admission claimed released coverage: %v", err)
+ }
+ probe.Coverage = ""
+ o = step(t, s, o, AdmitWriter(probe))
+ last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
+ if last.WritesLost == nil || last.WritesLost.Box != relSource || last.WritesLost.Generation != "g1" {
+ t.Fatalf("failover must declare the source's session: %+v", last)
+ }
+ o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session}))
+ o = step(t, s, o, CompleteRelocation("rel-1", counter()))
+ owes := slices.ContainsFunc(o.Claim.Obligations, func(ob Obligation) bool { return ob.Kind == "park-local" && ob.Box == relSource })
+ if owes == qualified {
+ t.Fatalf("qualified=%v: source chores at serving %+v", qualified, o.Claim.Obligations)
+ }
+ }
+}
+
+func TestRelocationSourceLossBecomesFailover(t *testing.T) {
+ ctx := context.Background()
+ for _, tc := range []struct {
+ name string
+ prepare func(*testing.T, release.Store, Observed) Observed
+ wantPhase string
+ wantExec string
+ wantParts []string
+ }{
+ {"starting before acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return o }, PhaseStarting, "none", []string{relTarget}},
+ {"starting after acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return step(t, s, o, ack(relSource)) }, PhaseStarting, "none", []string{relTarget}},
+ {"releasing before its receipt", func(t *testing.T, s release.Store, o Observed) Observed {
+ o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
+ return step(t, s, o, ToReleasing("rel-1"))
+ }, PhasePinning, "none", []string{relTarget}},
+ {"releasing after its receipt", func(t *testing.T, s release.Store, o Observed) Observed {
+ o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
+ o = step(t, s, o, ToReleasing("rel-1"))
+ h := o.Claim.Holder
+ return step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 1, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
+ }, PhaseReleasing, relSource, []string{relSource, relTarget}},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s, o := relServing(t)
+ o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ o = tc.prepare(t, s, o)
+ stamp := o.Claim.Holder.Stamp
+ o, err = FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ c := o.Claim
+ if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || !slices.Equal(c.Op.Participants, tc.wantParts) || c.Holder.Stamp != stamp || !slices.Contains(c.Lost, relSource) {
+ t.Fatalf("phase %s exec %s parts %v stamp %d lost %v", c.Op.Phase, c.Op.Exec, c.Op.Participants, c.Holder.Stamp, c.Lost)
+ }
+ if c.Op.Intent["reason"] != ReasonDrain {
+ t.Fatal("the intent was rewritten")
+ }
+ })
+ }
+}
+
+// relAtPhase starts a drain and drives it to phase. "probing+P" is probing
+// after the first admission opened P.
+func relAtPhase(t *testing.T, phase string) (release.Store, Observed) {
+ t.Helper()
+ ctx := context.Background()
+ s, o := relServing(t)
+ o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget))
+ if phase == PhaseStarting {
+ return s, o
+ }
+ o = step(t, s, o, ToReleasing("rel-1"))
+ if phase == PhaseReleasing {
+ return s, o
+ }
+ h := o.Claim.Holder
+ o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
+ if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil {
+ t.Fatal(err)
+ }
+ o = step(t, s, o, JournalPin("rel-1", relPin(40, 7, relObserved)))
+ if phase == PhasePinning {
+ return s, o
+ }
+ o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1"))
+ if phase == PhaseInstalling {
+ return s, o
+ }
+ o = step(t, s, o, ToInstalled("rel-1", PreparedAuthority{Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40}))
+ if phase == PhaseInstalled {
+ return s, o
+ }
+ o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim)))
+ o = step(t, s, o, ToProbing("rel-1", relObserved))
+ if phase == PhaseProbing {
+ return s, o
+ }
+ o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
+ return s, o
+}
+
+func TestRelocationReplacementTable(t *testing.T) {
+ ctx := context.Background()
+ for _, tc := range []struct {
+ at, wantPhase, wantExec string
+ newStamp, keepP bool
+ }{
+ {PhaseStarting, PhaseStarting, relSource, false, false},
+ {PhaseReleasing, PhaseReleasing, relSource, false, false},
+ {PhasePinning, PhasePinning, "none", false, false},
+ {PhaseInstalling, PhaseInstalling, relSpare, true, false},
+ {PhaseInstalled, PhaseInstalled, relSpare, true, true},
+ {PhaseProbing, PhaseInstalled, relSpare, true, true},
+ {"probing+P", PhasePinning, "none", false, false},
+ } {
+ t.Run(tc.at, func(t *testing.T) {
+ s, o := relAtPhase(t, tc.at)
+ before := o.Claim.Holder.Stamp
+ o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ c := o.Claim
+ if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || c.Op.Intent["target_box"] != relSpare || !slices.Contains(c.Lost, relTarget) || slices.Contains(c.Op.Participants, relTarget) {
+ t.Fatalf("phase %s exec %s intent %v lost %v parts %v", c.Op.Phase, c.Op.Exec, c.Op.Intent, c.Lost, c.Op.Participants)
+ }
+ if (c.Holder.Stamp != before) != tc.newStamp {
+ t.Fatalf("stamp %d -> %d", before, c.Holder.Stamp)
+ }
+ if tc.keepP && c.Op.Journal["prepared"] != "P1" {
+ t.Fatalf("the unopened P must be reused: %q", c.Op.Journal["prepared"])
+ }
+ if c.Op.Journal["local_install"] != "" {
+ t.Fatal("a replacement inherited its predecessor's receipt")
+ }
+ if tc.wantPhase == PhaseInstalled {
+ if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); err == nil {
+ t.Fatal("probing without the replacement's own local install")
+ }
+ }
+ })
+ }
+}
+
+func TestRelocationTwoReplacementsThenStuck(t *testing.T) {
+ ctx := context.Background()
+ s, o := relAtPhase(t, PhasePinning)
+ var err error
+ for _, next := range []string{relSpare, relSpare2} {
+ old := o.Claim.Op.Intent["target_box"]
+ if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if _, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, Target{"eve-knitting", relEnrollment("eve-knitting")}, "", relAuthority{}); !errors.Is(err, ErrReplacementsExceeded) {
+ t.Fatalf("third replacement: %v", err)
+ }
+ o = step(t, s, o, MarkStuck("rel-1", "two target replacements used"))
+ if journalUint(o.Claim.Op, "replacements") != 2 {
+ t.Fatal("stuck reset the count")
+ }
+ if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P9")); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("a stuck operation moved: %v", err)
+ }
+}
+
+func TestRelocationProbeFailureIsANestedRelocation(t *testing.T) {
+ ctx := context.Background()
+ s, o := relAtPhase(t, "probing+P")
+ o = step(t, s, o, ToProbeFailed("rel-1"))
+ h := o.Claim.Holder
+ rel := Release{ReleasedBy: relTarget, Enrollment: relEnrollment(relTarget), Generation: h.Generation, Session: h.Session, FinalTXID: 55, ManifestSeq: 9, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)}
+ o = step(t, s, o, JournalRelease("rel-1", rel))
+ if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relTarget, relEnrollment(relTarget)}); err == nil {
+ t.Fatal("the failed target was named again")
+ }
+ o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relSpare, relEnrollment(relSpare)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Exec != "none" {
+ t.Fatalf("nested relocation: %+v", o.Claim.Op)
+ }
+ if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(54, 9, relObserved))); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("pin lost the probe's writes: %v", err)
+ }
+ o = step(t, s, o, JournalPin("rel-1", relPin(55, 9, relObserved)))
+ o = step(t, s, o, ack(relSpare))
+ o = toProbing(t, s, o, "P2", 55)
+ // released coverage on P2's opening: the predecessor is the failed
+ // target's probe session, whose release this operation journaled.
+ o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"}))
+ o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session}))
+ o = step(t, s, o, CompleteRelocation("rel-1", counter()))
+ var owed []string
+ for _, ob := range o.Claim.Obligations {
+ owed = append(owed, ob.Box)
+ }
+ slices.Sort(owed)
+ if !slices.Equal(owed, []string{relSource, relTarget}) {
+ t.Fatalf("park-local owed by the source and the abandoned target: %v", owed)
+ }
+}
+
+func TestRelocationAdmissionRules(t *testing.T) {
+ ctx := context.Background()
+ probe := func(o Observed, gen, coverage string) Admission {
+ return Admission{ObservedOpID: "rel-1", ObservedPhase: o.Claim.Op.Phase, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: o.Claim.Op.Exec, Mode: "cold", NewGeneration: gen, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"}
+ }
+ for _, tc := range []struct {
+ name, at string
+ admit func(Observed) Admission
+ }{
+ {"the target admits in installing", PhaseInstalling, func(o Observed) Admission { return probe(o, "P1", "") }},
+ {"the target admits in installed", PhaseInstalled, func(o Observed) Admission { return probe(o, "P1", "") }},
+ {"the first probe skips P", PhaseProbing, func(o Observed) Admission { return probe(o, "other", "") }},
+ {"P is opened twice", "probing+P", func(o Observed) Admission { return probe(o, "P1", "") }},
+ {"released after P was opened", "probing+P", func(o Observed) Admission { return probe(o, "P3", "released") }},
+ {"warm probe", PhaseProbing, func(o Observed) Admission {
+ a := probe(o, "", "")
+ a.Mode, a.PredecessorCertified = "warm", true
+ return a
+ }},
+ {"the source claims released", PhaseStarting, func(o Observed) Admission { return probe(o, "g2", "released") }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s, o := relAtPhase(t, tc.at)
+ if _, err := Update(ctx, s, o, AdmitWriter(tc.admit(o))); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("admitted: %v", err)
+ }
+ })
+ }
+ t.Run("released names another predecessor", func(t *testing.T) {
+ s, o := relAtPhase(t, PhaseProbing)
+ o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Session++
+ if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("admitted: %v", err)
+ }
+ })
+ t.Run("released names another predecessor box", func(t *testing.T) {
+ s, o := relAtPhase(t, PhaseProbing)
+ o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Box = relSpare
+ if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("admitted: %v", err)
+ }
+ })
+ t.Run("a third attempt reopens P", func(t *testing.T) {
+ s, o := relAtPhase(t, "probing+P")
+ o = step(t, s, o, AdmitWriter(probe(o, "P3", ""))) // attempt two: a fresh generation
+ if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", ""))); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("P was opened again: %v", err)
+ }
+ })
+ t.Run("released outside relocating", func(t *testing.T) {
+ s, o := relServing(t)
+ if _, err := Update(ctx, s, o, AdmitWriter(Admission{ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSource, Mode: "cold", NewGeneration: "g2", Coverage: "released", Region: "ie"})); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("admitted: %v", err)
+ }
+ })
+ t.Run("the source wakes in starting", func(t *testing.T) {
+ s, o := relAtPhase(t, PhaseStarting)
+ if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); err != nil {
+ t.Fatal(err)
+ }
+ })
+}
+
+func TestRelocationStartGuard(t *testing.T) {
+ serving := func(obs ...Obligation) *Claim {
+ return &Claim{Holder: &Holder{Incarnation: "i"}, Op: Op{Kind: KindServing, Exec: relSource}, Obligations: obs}
+ }
+ for _, tc := range []struct {
+ name string
+ ob Obligation
+ fenced bool
+ want error
+ }{
+ {"park-generation in the region (L11)", Obligation{Kind: "park-generation", Region: "ie"}, false, nil},
+ {"park-replica in the region", Obligation{Kind: "park-replica", Region: "ie"}, false, ErrPendingChore},
+ {"park-replica elsewhere", Obligation{Kind: "park-replica", Region: "uk"}, false, nil},
+ {"park-local on the target", Obligation{Kind: "park-local", Box: relTarget}, true, ErrPendingChore},
+ {"park-local on a live source", Obligation{Kind: "park-local", Box: relSource}, false, ErrPendingChore},
+ {"park-local on a fenced source", Obligation{Kind: "park-local", Box: relSource}, true, nil},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ if err := RelocationStartGuard(serving(tc.ob), "ie", relSource, relTarget, tc.fenced); !errors.Is(err, tc.want) {
+ t.Fatalf("got %v, want %v", err, tc.want)
+ }
+ })
+ }
+}
+
+func TestRelocationLocalInstallReceiptIsBound(t *testing.T) {
+ ctx := context.Background()
+ for _, tc := range []struct {
+ name string
+ mutate func(*LocalInstall)
+ }{
+ {"another box", func(l *LocalInstall) { l.Box = relSpare }},
+ {"another enrollment", func(l *LocalInstall) { l.Enrollment = relEnrollment(relSpare) }},
+ {"a later writer stamp", func(l *LocalInstall) { l.InstallStamp++ }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s, o := relAtPhase(t, PhaseInstalled)
+ l := localInstall(o.Claim)
+ tc.mutate(&l)
+ if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("recorded: %v", err)
+ }
+ })
+ }
+ s, o := relAtPhase(t, PhaseInstalled)
+ l := localInstall(o.Claim)
+ o = step(t, s, o, RecordLocalInstall("rel-1", l))
+ if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrAlreadyRecorded) {
+ t.Fatalf("replay: %v", err)
+ }
+ if _, err := Update(ctx, s, o, MarkStuck("rel-1", "operator")); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("stuck between installed and a probe: %v", err)
+ }
+ // B1-2: probing is refused on another box's receipt, even one in the journal.
+ foreign := l
+ foreign.Box = relSpare
+ raw, _ := json.Marshal(foreign)
+ o.Claim.Op.Journal["local_install"] = string(raw)
+ if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("probing on another box's receipt: %v", err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostclaim/ -run 'Relocation' -count=1`
+Expected: FAIL to compile: `undefined: RelocationStart`, `undefined: StartRelocation`, and the rest.
+
+- [ ] **Step 3: Write the transitions**
+
+`internal/hostclaim/relocation.go`:
+
+```go
+package hostclaim
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "slices"
+ "strconv"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// Relocation (claim §4 "Same-region relocation", region HA spec 1 §5.8):
+// the reasons, the phases and the bound on target replacements.
+const (
+ ReasonDrain = "drain"
+ ReasonRebalance = "rebalance"
+ ReasonFailover = "failover"
+
+ PhaseStarting = "starting"
+ PhaseReleasing = "releasing"
+ PhasePinning = "pinning"
+ PhaseInstalling = "installing"
+ PhaseInstalled = "installed"
+ PhaseProbing = "probing"
+ PhaseProbeFailed = "probe-failed"
+
+ // MaxTargetReplacements is spec 1 §5.8's bound: after two
+ // replacements of the target in one operation it goes to stuck.
+ MaxTargetReplacements = 2
+
+ // FailoverPinDelay is §5.8's failover pin rule: pin no earlier than
+ // the fence's observed_at plus 30 seconds, so requests S3 had already
+ // accepted from the terminated box have completed (H3's window).
+ FailoverPinDelay = 30 * time.Second
+)
+
+var (
+ ErrRelocation = errors.New("hostclaim: relocation transition refused")
+ ErrReplacementsExceeded = errors.New("hostclaim: relocation has used its target replacements")
+ ErrAlreadyRecorded = errors.New("hostclaim: relocation journal entry already recorded")
+)
+
+// RelocationStart is the start CAS's intent (spec 1 §5.8, claim §4). Fence
+// is set for a failover only and names the source's termination fence.
+type RelocationStart struct {
+ OpID, By string
+ Now time.Time
+ Region, Reason string
+ SourceBox, SourceEnrollment string
+ TargetBox, TargetEnrollment string
+ DirectoryRevision uint64
+ AdmissionID string
+ Fence *FenceRef
+ Discharge *LocalDischarge
+}
+
+// LocalDischarge is a fenced box's volume evidence (claim D5 as amended by
+// L3, spec 1 §5.9, H19): the volume records the caller read and judged with
+// boxvolumes.Discharge. When it is nil, or Qualified is false, the box's
+// park-local chores stay open for the operator path. It never gates the
+// fence or the relocation.
+type LocalDischarge struct {
+ Qualified bool `json:"qualified"`
+ Records []string `json:"records,omitempty"`
+ Reason string `json:"reason,omitempty"`
+}
+
+// Release is the journal entry a live box writes after its release
+// sequence (spec 1 §5.8 steps 1 to 4): its stopped database's final TXID,
+// its final files manifest sequence and its full stop receipt.
+type Release struct {
+ ReleasedBy string `json:"released_by"`
+ Enrollment string `json:"enrollment"`
+ Generation string `json:"generation"`
+ Session uint64 `json:"session"`
+ FinalTXID uint64 `json:"final_txid"`
+ ManifestSeq int64 `json:"manifest_seq"`
+ Receipt string `json:"receipt"`
+ ReceiptSHA256 string `json:"receipt_sha256"`
+}
+
+// ReleaseAuthority verifies a journaled release's full stop receipt: the
+// immutable object at Receipt hashes to ReceiptSHA256 and is complete stop
+// evidence (claim §6.1, P6-D1) for this host, operation, box, incarnation
+// and session. Plan 6b's receipt format is its only implementation.
+type ReleaseAuthority interface {
+ VerifyRelease(context.Context, Claim, Release) error
+}
+
+// RecoveryPin is the journal's reference to one pinned recovery set
+// (§5.8 "The pinned recovery set"): the immutable pin object holding the
+// exact versions, its digest, the database position it reaches, the files
+// manifest it pins, and when it was taken.
+type RecoveryPin struct {
+ Key string `json:"key"`
+ SHA256 string `json:"sha256"`
+ Generation string `json:"generation"`
+ MaxTXID uint64 `json:"max_txid"`
+ FilesManifest string `json:"files_manifest"`
+ FilesSeq int64 `json:"files_seq"`
+ PinnedAt time.Time `json:"pinned_at"`
+}
+
+// PreparedAuthority is the `installed` CAS's record of the remote build
+// (§5.8 "Building the prepared generation"): the install manifest of
+// generation P, verified complete by a fresh read.
+type PreparedAuthority struct {
+ Generation string `json:"generation"`
+ InstallManifest string `json:"install_manifest"`
+ SHA256 string `json:"sha256"`
+ SnapshotTXID uint64 `json:"snapshot_txid"`
+ FilesSeq int64 `json:"files_seq"`
+}
+
+// LocalInstall is one box's local install receipt (§5.8, B1-2, B2-5).
+type LocalInstall struct {
+ Box string `json:"box"`
+ Enrollment string `json:"enrollment"`
+ InstallStamp uint64 `json:"install_stamp"`
+ FilesManifestVersion int64 `json:"files_manifest_version"`
+ JournalSHA256 string `json:"journal_sha256"`
+}
+
+// Probe is the target's journaled probe result for its admitted session.
+type Probe struct {
+ Box string `json:"box"`
+ Session uint64 `json:"session"`
+}
+
+// Target names a box for the relocation's target slot.
+type Target struct {
+ Box, Enrollment string
+}
+
+func journalJSON(c *Claim, key string, v any) error {
+ raw, err := json.Marshal(v)
+ if err != nil {
+ return err
+ }
+ if c.Op.Journal == nil {
+ c.Op.Journal = map[string]string{}
+ }
+ c.Op.Journal[key] = string(raw)
+ return nil
+}
+
+func readJournal(op Op, key string, v any) bool {
+ raw, ok := op.Journal[key]
+ return ok && json.Unmarshal([]byte(raw), v) == nil
+}
+
+func journalUint(op Op, key string) uint64 {
+ n, _ := strconv.ParseUint(op.Journal[key], 10, 64)
+ return n
+}
+
+func requireRelocation(c *Claim, opID string, phases ...string) error {
+ if err := RequireOp(opID)(c); err != nil {
+ return err
+ }
+ if c.Holder == nil || c.Op.Kind != KindRelocating || !slices.Contains(phases, c.Op.Phase) {
+ return fmt.Errorf("%w: %s/%s", ErrWrongState, c.Op.Kind, c.Op.Phase)
+ }
+ if c.Op.Journal["stuck"] != "" {
+ return fmt.Errorf("%w: operation is stuck", ErrRelocation)
+ }
+ ensureJournal(c)
+ return nil
+}
+
+// ensureJournal makes the journal writable: an empty one round-trips as nil.
+func ensureJournal(c *Claim) {
+ if c.Op.Journal == nil {
+ c.Op.Journal = map[string]string{}
+ }
+}
+
+// requireAcks is the D6 barrier: every current participant has recorded
+// its execution acknowledgement before the operation leaves its phase.
+func requireAcks(c *Claim) error {
+ for _, box := range c.Op.Participants {
+ if c.Op.Journal["exec_ack."+box] == "" {
+ return fmt.Errorf("%w: %s has not acknowledged", ErrRelocation, box)
+ }
+ }
+ return nil
+}
+
+func failover(op Op) bool {
+ return op.Intent["reason"] == ReasonFailover || op.Journal["reason"] == ReasonFailover
+}
+
+// RelocationStartGuard is §4's start guard with the relocation exception
+// (L11): a park-generation chore in the region does not block, a
+// park-replica in the region does, a park-local on the target does, and a
+// park-local on a fenced source does not block a failover.
+func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced bool) error {
+ if c.Holder == nil || c.Op.Kind != KindServing || c.Op.ID != "" {
+ return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind)
+ }
+ for _, o := range c.Obligations {
+ switch {
+ case o.Kind == "park-replica" && o.Region == region:
+ return fmt.Errorf("%w: park-replica in %s", ErrPendingChore, region)
+ case o.Kind == "park-local" && o.Box == target:
+ return fmt.Errorf("%w: park-local on %s", ErrPendingChore, target)
+ case o.Kind == "park-local" && o.Box == source && !sourceFenced:
+ return fmt.Errorf("%w: park-local on %s", ErrPendingChore, source)
+ }
+ }
+ return nil
+}
+
+// fenceBox puts box on lost with its fence reference (L2, L4) and, when the
+// volume evidence qualifies (L3), discharges that box's park-local chores
+// for this host in the same CAS. It never removes anything else.
+func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDischarge) error {
+ key := "fence." + box
+ raw, err := json.Marshal(ref)
+ if err != nil {
+ return err
+ }
+ if old, ok := c.Op.Journal[key]; ok && old != string(raw) {
+ return fmt.Errorf("%w: fence reference already bound", ErrFence)
+ }
+ if c.Op.Journal == nil {
+ c.Op.Journal = map[string]string{}
+ }
+ c.Op.Journal[key] = string(raw)
+ c.Op.Journal[key+".observed_at"] = observedAt
+ if !slices.Contains(c.Lost, box) {
+ c.Lost = append(c.Lost, box)
+ }
+ if d == nil {
+ d = &LocalDischarge{Reason: "no volume evidence supplied"}
+ }
+ if err := journalJSON(c, "discharge."+box, d); err != nil {
+ return err
+ }
+ if d.Qualified {
+ kept := c.Obligations[:0]
+ for _, o := range c.Obligations {
+ if o.Kind != "park-local" || o.Box != box {
+ kept = append(kept, o)
+ }
+ }
+ c.Obligations = kept
+ }
+ return nil
+}
+
+func removeParticipant(c *Claim, box string) {
+ c.Op.Participants = slices.DeleteFunc(slices.Clone(c.Op.Participants), func(b string) bool { return b == box })
+}
+
+func addParticipant(c *Claim, box string) {
+ if !slices.Contains(c.Op.Participants, box) {
+ c.Op.Participants = append(c.Op.Participants, box)
+ }
+}
+
+// installedBoxes is every box this operation handed op.exec for an
+// installation; each gets a park-local unless discharged (claim §4:
+// "Every abandoned local installation receives an exact park-local").
+func noteInstalledBox(c *Claim, box string) error {
+ var boxes []string
+ readJournal(c.Op, "installed_boxes", &boxes)
+ if !slices.Contains(boxes, box) {
+ boxes = append(boxes, box)
+ }
+ return journalJSON(c, "installed_boxes", boxes)
+}
+
+// StartRelocation is the start CAS (spec 1 §5.8, claim §4). A drain or
+// rebalance starts with op.exec on the source; a failover cites the
+// source's termination fence, starts with op.exec "none", keeps the stamp,
+// and puts the source on lost. The caller has already checked the roster
+// (hostactors.Directory.CheckBoxStart), the target's eligibility and the
+// app-admission registration; this CAS records what it checked.
+func StartRelocation(ctx context.Context, s release.Store, observed Observed, in RelocationStart, authority FenceAuthority) (Observed, error) {
+ if observed.Absent || observed.Claim == nil || observed.ETag == "" {
+ return Observed{}, ErrWrongState
+ }
+ switch in.Reason {
+ case ReasonDrain, ReasonRebalance:
+ if in.Fence != nil {
+ return Observed{}, fmt.Errorf("%w: only a failover cites a fence", ErrRelocation)
+ }
+ case ReasonFailover:
+ if in.Fence == nil {
+ return Observed{}, fmt.Errorf("%w: a failover cites the source's fence", ErrRelocation)
+ }
+ default:
+ return Observed{}, fmt.Errorf("%w: reason %q", ErrRelocation, in.Reason)
+ }
+ if in.OpID == "" || in.Region == "" || in.AdmissionID == "" || in.DirectoryRevision == 0 || in.Now.IsZero() ||
+ !validBoxReference(in.SourceBox) || !validBoxReference(in.TargetBox) || in.SourceBox == "none" || in.TargetBox == "none" ||
+ in.SourceBox == in.TargetBox || in.SourceEnrollment == "" || in.TargetEnrollment == "" {
+ return Observed{}, fmt.Errorf("%w: incomplete start", ErrRelocation)
+ }
+ var observedAt string
+ if in.Fence != nil {
+ v, err := VerifyFence(ctx, s, authority, *in.Fence)
+ if err != nil {
+ return Observed{}, err
+ }
+ if v.record.Box != in.SourceBox || v.record.Enrollment != in.SourceEnrollment || v.binding.Region != in.Region {
+ return Observed{}, ErrFence
+ }
+ observedAt = v.record.ObservedAt
+ }
+ return Update(ctx, s, observed, func(c *Claim) error {
+ if c.Op.Exec != in.SourceBox || slices.Contains(c.Lost, in.TargetBox) {
+ return fmt.Errorf("%w: the source is not the executor, or the target is lost", ErrWrongState)
+ }
+ if err := RelocationStartGuard(c, in.Region, in.SourceBox, in.TargetBox, in.Fence != nil); err != nil {
+ return err
+ }
+ exec, participants := in.SourceBox, []string{in.SourceBox, in.TargetBox}
+ if in.Fence != nil {
+ exec, participants = "none", []string{in.TargetBox}
+ }
+ c.Op = Op{Kind: KindRelocating, ID: in.OpID, Phase: PhaseStarting, By: in.By, StartedAt: in.Now.UTC(),
+ Exec: exec, Participants: participants,
+ Intent: map[string]string{
+ "region": in.Region, "reason": in.Reason,
+ "source_box": in.SourceBox, "source_enrollment": in.SourceEnrollment,
+ "target_box": in.TargetBox, "target_enrollment": in.TargetEnrollment,
+ "stamp_from": strconv.FormatUint(c.Holder.Stamp, 10),
+ "generation_from": c.Holder.Generation,
+ "directory_revision": strconv.FormatUint(in.DirectoryRevision, 10),
+ "admission_id": in.AdmissionID,
+ },
+ Journal: map[string]string{}}
+ if in.Fence != nil {
+ return fenceBox(c, in.SourceBox, *in.Fence, observedAt, in.Discharge)
+ }
+ return nil
+ })
+}
+
+// FenceSource is §5.8's source-loss rule (B1-6): a drain or rebalance
+// whose source is terminated becomes a failover in one CAS citing the
+// source's fence. In starting it stays in starting with the source off the
+// barrier; in releasing before a release is journaled it goes to pinning
+// with the fence as evidence; after a release (pinning on) the source only
+// goes on lost. Every case keeps the stamp and the intent.
+func FenceSource(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) {
+ if observed.Absent || observed.Claim == nil {
+ return Observed{}, ErrWrongState
+ }
+ before := observed.Claim.Op
+ source := before.Intent["source_box"]
+ v, err := VerifyFence(ctx, s, authority, ref)
+ if err != nil {
+ return Observed{}, err
+ }
+ if v.record.Box != source || v.record.Enrollment != before.Intent["source_enrollment"] || v.binding.Region != before.Intent["region"] {
+ return Observed{}, ErrFence
+ }
+ return Update(ctx, s, observed, func(c *Claim) error {
+ if err := RequireOp(opID)(c); err != nil {
+ return err
+ }
+ if c.Op.Kind != KindRelocating || c.Op.Intent["source_box"] != source {
+ return ErrWrongState
+ }
+ ensureJournal(c)
+ // Before its release is journaled the source's writes are not
+ // kept by a release, so the op becomes a failover. After it, only
+ // the source's liveness changes.
+ if c.Op.Phase == PhaseStarting || (c.Op.Phase == PhaseReleasing && c.Op.Journal["release"] == "") {
+ c.Op.Journal["reason"] = ReasonFailover
+ removeParticipant(c, source)
+ c.Op.Exec = "none"
+ if c.Op.Phase == PhaseReleasing {
+ c.Op.Phase = PhasePinning
+ }
+ }
+ return fenceBox(c, source, ref, v.record.ObservedAt, d)
+ })
+}
+
+// ToReleasing is a drain's barrier CAS: from starting, once every
+// participant has acknowledged. A failover never enters releasing.
+func ToReleasing(opID string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseStarting); err != nil {
+ return err
+ }
+ if failover(c.Op) {
+ return fmt.Errorf("%w: a failover skips releasing", ErrRelocation)
+ }
+ if err := requireAcks(c); err != nil {
+ return err
+ }
+ c.Op.Phase = PhaseReleasing
+ return nil
+ }
+}
+
+// JournalRelease records a live box's completed release sequence. The
+// source records it in releasing; a failed target in probe-failed. The
+// release must be of the box's own current writer session.
+func JournalRelease(opID string, r Release) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil {
+ return err
+ }
+ h := c.Holder
+ writerMismatch := h.Session != 0 && h.WriterBox != r.ReleasedBy // a host never woken has no writer
+ if r.ReleasedBy == "" || r.ReleasedBy != c.Op.Exec || r.Enrollment == "" || r.Receipt == "" || !isHex64(r.ReceiptSHA256) ||
+ r.Generation != h.Generation || r.Session != h.Session || writerMismatch {
+ return fmt.Errorf("%w: release is not the executor's current session", ErrRelocation)
+ }
+ want := c.Op.Intent["source_enrollment"]
+ if c.Op.Phase == PhaseProbeFailed {
+ want = c.Op.Intent["target_enrollment"]
+ }
+ if r.Enrollment != want {
+ return fmt.Errorf("%w: release enrollment", ErrRelocation)
+ }
+ var old Release
+ if readJournal(c.Op, "release", &old) && c.Op.Journal["release.phase"] == c.Op.Phase {
+ if old == r {
+ return ErrAlreadyRecorded
+ }
+ return fmt.Errorf("%w: a different release is journaled", ErrRelocation)
+ }
+ c.Op.Journal["release.phase"] = c.Op.Phase
+ return journalJSON(c, "release", r)
+ }
+}
+
+// PinAfterRelease leaves releasing (a drain) or probe-failed (a failed
+// target, §5.8's nested relocation) for pinning, citing the journaled
+// release's verified receipt. op.exec becomes "none" and the stamp is kept
+// (claim R6-4, D5's non-installing form). From probe-failed, next names
+// the replacement target, which must not be the failed one.
+func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, opID string, authority ReleaseAuthority, next *Target) (Observed, error) {
+ if observed.Absent || observed.Claim == nil || authority == nil {
+ return Observed{}, ErrWrongState
+ }
+ var r Release
+ if !readJournal(observed.Claim.Op, "release", &r) {
+ return Observed{}, fmt.Errorf("%w: no journaled release", ErrRelocation)
+ }
+ if err := authority.VerifyRelease(ctx, *observed.Claim, r); err != nil {
+ return Observed{}, err
+ }
+ return Update(ctx, s, observed, func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil {
+ return err
+ }
+ var now Release
+ if !readJournal(c.Op, "release", &now) || now != r || c.Op.Journal["release.phase"] != c.Op.Phase || r.ReleasedBy != c.Op.Exec {
+ return fmt.Errorf("%w: release changed", ErrRelocation)
+ }
+ if c.Op.Phase == PhaseReleasing {
+ if next != nil {
+ return fmt.Errorf("%w: a drain keeps its target", ErrRelocation)
+ }
+ } else {
+ if next == nil || next.Box == r.ReleasedBy {
+ return fmt.Errorf("%w: a nested relocation names another target", ErrRelocation)
+ }
+ if err := replaceTargetSlot(c, *next); err != nil {
+ return err
+ }
+ c.Op.Journal["released_by_target"] = r.ReleasedBy
+ }
+ c.Op.Exec, c.Op.Phase = "none", PhasePinning
+ delete(c.Op.Journal, "pin")
+ return nil
+ })
+}
+
+// PinFailover leaves starting for pinning on a failover: the source's
+// fence is journaled and every remaining participant has acknowledged.
+func PinFailover(opID string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseStarting); err != nil {
+ return err
+ }
+ if !failover(c.Op) || c.Op.Journal["fence."+c.Op.Intent["source_box"]] == "" || c.Op.Exec != "none" {
+ return fmt.Errorf("%w: pinning without a release needs the source's fence", ErrRelocation)
+ }
+ if err := requireAcks(c); err != nil {
+ return err
+ }
+ c.Op.Phase = PhasePinning
+ return nil
+ }
+}
+
+// JournalPin records the pinned recovery set in pinning. After a release
+// it must reach the journaled final TXID and files sequence; on a failover
+// it must be taken no earlier than the fence's observed_at plus
+// FailoverPinDelay.
+func JournalPin(opID string, p RecoveryPin) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhasePinning); err != nil {
+ return err
+ }
+ if p.Key == "" || !isHex64(p.SHA256) || p.FilesManifest == "" || p.PinnedAt.IsZero() {
+ return fmt.Errorf("%w: incomplete pin", ErrRelocation)
+ }
+ var r Release
+ if readJournal(c.Op, "release", &r) {
+ if p.MaxTXID < r.FinalTXID || p.FilesSeq < r.ManifestSeq {
+ return fmt.Errorf("%w: pin %d/%d is short of the release's %d/%d", ErrRelocation, p.MaxTXID, p.FilesSeq, r.FinalTXID, r.ManifestSeq)
+ }
+ }
+ for key, at := range c.Op.Journal {
+ if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" {
+ t, err := time.Parse(time.RFC3339Nano, at)
+ if err != nil || p.PinnedAt.Before(t.Add(FailoverPinDelay)) {
+ return fmt.Errorf("%w: pinned before the fence's observation plus %s", ErrRelocation, FailoverPinDelay)
+ }
+ }
+ }
+ if readJournal(c.Op, "pin", new(RecoveryPin)) {
+ return ErrAlreadyRecorded
+ }
+ c.Op.Journal["pins"] = strconv.FormatUint(journalUint(c.Op, "pins")+1, 10)
+ return journalJSON(c, "pin", p)
+ }
+}
+
+// NameTarget names the target of an operation that has none: a move
+// converted to relocating before its flip (claim §4, moves §7.10). The
+// named box acknowledges (D6) before ToInstalling hands it op.exec.
+func NameTarget(opID string, t Target) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhasePinning); err != nil {
+ return err
+ }
+ if c.Op.Intent["target_box"] != "" {
+ return fmt.Errorf("%w: the operation already has a target", ErrRelocation)
+ }
+ return replaceTargetSlot(c, t)
+ }
+}
+
+// ToInstalling hands op.exec to the target for a new installation: a new
+// stamp (this CAS's gen, R6-4), journaled as install_stamp, a fresh
+// prepared generation id, and the target added to touched.boxes before its
+// first effect. A converted operation with no target names it here.
+func ToInstalling(opID string, t Target, prepared string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhasePinning); err != nil {
+ return err
+ }
+ if !readJournal(c.Op, "pin", new(RecoveryPin)) || prepared == "" || prepared == c.Holder.Generation {
+ return fmt.Errorf("%w: installing needs a pin and a fresh prepared generation", ErrRelocation)
+ }
+ if t.Box != c.Op.Intent["target_box"] || t.Enrollment != c.Op.Intent["target_enrollment"] || slices.Contains(c.Lost, t.Box) {
+ return fmt.Errorf("%w: not the operation's target", ErrRelocation)
+ }
+ for _, o := range c.Obligations {
+ if o.Kind == "park-local" && o.Box == t.Box {
+ return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box)
+ }
+ }
+ if err := requireAcks(c); err != nil {
+ return err
+ }
+ return install(c, t.Box, prepared, PhaseInstalling)
+ }
+}
+
+func install(c *Claim, box, prepared, phase string) error {
+ stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1
+ c.Op.Exec, c.Op.Phase = box, phase
+ c.Holder.Stamp = stamp
+ c.Op.Journal["install_stamp"] = strconv.FormatUint(stamp, 10)
+ c.Op.Journal["prepared"] = prepared
+ delete(c.Op.Journal, "local_install")
+ if !slices.Contains(c.Touched.Boxes, box) {
+ c.Touched.Boxes = append(c.Touched.Boxes, box)
+ }
+ return noteInstalledBox(c, box)
+}
+
+// ToInstalled records the verified remote build of P (§5.8, the
+// no-reinstall boundary). Nobody builds P again after this CAS.
+func ToInstalled(opID string, a PreparedAuthority) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseInstalling); err != nil {
+ return err
+ }
+ var pin RecoveryPin
+ readJournal(c.Op, "pin", &pin)
+ if a.Generation == "" || a.Generation != c.Op.Journal["prepared"] || a.SnapshotTXID != pin.MaxTXID ||
+ (pin.MaxTXID > 0 && (a.InstallManifest == "" || !isHex64(a.SHA256))) {
+ return fmt.Errorf("%w: prepared authority is not this pin's build of P", ErrRelocation)
+ }
+ if err := requireAcks(c); err != nil {
+ return err
+ }
+ c.Op.Phase = PhaseInstalled
+ return journalJSON(c, "prepared_authority", a)
+ }
+}
+
+// preparedOpened reports whether any admission has named P.
+func preparedOpened(c *Claim) bool {
+ p := c.Op.Journal["prepared"]
+ if p == "" || c.Holder == nil {
+ return false
+ }
+ for _, a := range c.Holder.Admissions {
+ if a.Generation == p {
+ return true
+ }
+ }
+ return false
+}
+
+// RecordLocalInstall journals the executor's local install receipt in
+// installed (B1-2). It is bound to op.exec, the target's enrollment and the
+// journaled install_stamp, and refused once any admission has named P
+// (B2-5): after that the local install is never remade.
+func RecordLocalInstall(opID string, l LocalInstall) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseInstalled); err != nil {
+ return err
+ }
+ if l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] ||
+ l.InstallStamp == 0 || l.InstallStamp != journalUint(c.Op, "install_stamp") || !isHex64(l.JournalSHA256) {
+ return fmt.Errorf("%w: receipt is not this installation's", ErrRelocation)
+ }
+ if preparedOpened(c) {
+ return fmt.Errorf("%w: P was admitted; the local install is never remade", ErrRelocation)
+ }
+ var old LocalInstall
+ if readJournal(c.Op, "local_install", &old) {
+ if old == l {
+ return ErrAlreadyRecorded
+ }
+ return fmt.Errorf("%w: a different receipt is journaled", ErrRelocation)
+ }
+ return journalJSON(c, "local_install", l)
+ }
+}
+
+// ToProbing enters probing, which requires a local install receipt for
+// exactly op.exec, its enrollment and the journaled install_stamp. now
+// starts the probe deadline (§5.8: 10 minutes).
+func ToProbing(opID string, now time.Time) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseInstalled); err != nil {
+ return err
+ }
+ var l LocalInstall
+ if !readJournal(c.Op, "local_install", &l) || l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] ||
+ l.InstallStamp != journalUint(c.Op, "install_stamp") {
+ return fmt.Errorf("%w: no local install receipt for the executor", ErrRelocation)
+ }
+ if err := requireAcks(c); err != nil {
+ return err
+ }
+ if now.IsZero() {
+ return fmt.Errorf("%w: probing needs its start time", ErrRelocation)
+ }
+ c.Op.Phase = PhaseProbing
+ c.Op.Journal["probing_since"] = now.UTC().Format(time.RFC3339Nano)
+ return nil
+ }
+}
+
+// RecordProbe journals the executor's passing probe of its admitted session.
+func RecordProbe(opID string, p Probe) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseProbing); err != nil {
+ return err
+ }
+ if p.Box != c.Op.Exec || p.Box != c.Holder.WriterBox || p.Session == 0 || p.Session != c.Holder.Session ||
+ p.Session < journalUint(c.Op, "install_stamp") {
+ return fmt.Errorf("%w: probe is not of the executor's admitted session", ErrRelocation)
+ }
+ return journalJSON(c, "probe", p)
+ }
+}
+
+// ToProbeFailed records that the live target's probe failed or missed its
+// deadline. The target then runs its release sequence (JournalRelease).
+func ToProbeFailed(opID string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseProbing); err != nil {
+ return err
+ }
+ c.Op.Phase = PhaseProbeFailed
+ delete(c.Op.Journal, "release")
+ delete(c.Op.Journal, "release.phase")
+ return nil
+ }
+}
+
+// CompleteRelocation is the serving CAS: the target's probe passed for its
+// admitted session, so the host serves on it alone. Every other box this
+// operation installed on, and a live source, owes a park-local unless a
+// fence discharged it.
+func CompleteRelocation(opID string, newID func() string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseProbing); err != nil {
+ return err
+ }
+ var p Probe
+ if !readJournal(c.Op, "probe", &p) || p.Box != c.Op.Exec || p.Session != c.Holder.Session || c.Holder.WriterBox != p.Box {
+ return fmt.Errorf("%w: no passing probe of the current session", ErrRelocation)
+ }
+ target := c.Op.Exec
+ var owed []string
+ readJournal(c.Op, "installed_boxes", &owed)
+ owed = append([]string{c.Op.Intent["source_box"]}, owed...)
+ for _, box := range owed {
+ if box == target || box == "" {
+ continue
+ }
+ var d LocalDischarge
+ if readJournal(c.Op, "discharge."+box, &d) && d.Qualified {
+ continue
+ }
+ exists := false
+ for _, o := range c.Obligations {
+ if o.Kind == "park-local" && o.Box == box {
+ exists = true
+ }
+ }
+ if !exists {
+ c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: box, Retirement: newID(), Owner: "box:" + box})
+ }
+ }
+ c.Op = Op{Kind: KindServing, Exec: target, Participants: []string{target}}
+ return nil
+ }
+}
+
+// replaceTargetSlot names t as the target, counting a replacement when it
+// replaces another box. A replaced box is never named again for this host.
+func replaceTargetSlot(c *Claim, t Target) error {
+ if !validBoxReference(t.Box) || t.Box == "none" || t.Enrollment == "" || t.Box == c.Op.Intent["source_box"] || slices.Contains(c.Lost, t.Box) {
+ return fmt.Errorf("%w: invalid target", ErrRelocation)
+ }
+ var excluded []string
+ readJournal(c.Op, "excluded_targets", &excluded)
+ if slices.Contains(excluded, t.Box) {
+ return fmt.Errorf("%w: %s was replaced in this operation", ErrRelocation, t.Box)
+ }
+ old := c.Op.Intent["target_box"]
+ if old != "" && old != t.Box {
+ n := journalUint(c.Op, "replacements")
+ if n >= MaxTargetReplacements {
+ return ErrReplacementsExceeded
+ }
+ c.Op.Journal["replacements"] = strconv.FormatUint(n+1, 10)
+ excluded = append(excluded, old)
+ if err := journalJSON(c, "excluded_targets", excluded); err != nil {
+ return err
+ }
+ removeParticipant(c, old)
+ }
+ c.Op.Intent["target_box"], c.Op.Intent["target_enrollment"] = t.Box, t.Enrollment
+ addParticipant(c, t.Box)
+ return nil
+}
+
+// ReplaceTarget is §5.8's replacement table for a target terminated by its
+// fence. It records the fence (the old target goes on lost), names next,
+// and resumes at the phase the table gives:
+//
+// starting, releasing, pinning -> the same phase
+// installing -> installing: next is op.exec, new stamp, fresh P'
+// installed, or probing before P -> installed: next is op.exec, new stamp, same P, no receipt
+// probing after P, or probe-failed -> pinning: a nested failover from holder.generation
+//
+// prepared is P' for installing and ignored otherwise. A third
+// replacement is refused with ErrReplacementsExceeded; the worker then
+// marks the operation stuck.
+func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, next Target, prepared string, authority FenceAuthority) (Observed, error) {
+ if observed.Absent || observed.Claim == nil {
+ return Observed{}, ErrWrongState
+ }
+ old := observed.Claim.Op.Intent["target_box"]
+ v, err := VerifyFence(ctx, s, authority, ref)
+ if err != nil {
+ return Observed{}, err
+ }
+ if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] {
+ return Observed{}, ErrFence
+ }
+ return Update(ctx, s, observed, func(c *Claim) error {
+ if err := requireRelocation(c, opID, PhaseStarting, PhaseReleasing, PhasePinning, PhaseInstalling, PhaseInstalled, PhaseProbing, PhaseProbeFailed); err != nil {
+ return err
+ }
+ if c.Op.Intent["target_box"] != old {
+ return ErrStaleOp
+ }
+ phase := c.Op.Phase
+ if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil {
+ return err
+ }
+ if err := replaceTargetSlot(c, next); err != nil {
+ return err
+ }
+ switch {
+ case phase == PhaseStarting || phase == PhaseReleasing || phase == PhasePinning:
+ return nil
+ case phase == PhaseInstalling:
+ if prepared == "" || prepared == c.Op.Journal["prepared"] || prepared == c.Holder.Generation {
+ return fmt.Errorf("%w: a replacement in installing builds a fresh P'", ErrRelocation)
+ }
+ return install(c, next.Box, prepared, PhaseInstalling)
+ case (phase == PhaseInstalled || phase == PhaseProbing) && !preparedOpened(c):
+ return install(c, next.Box, c.Op.Journal["prepared"], PhaseInstalled)
+ default:
+ // After P's first admission: nested failover from pinning,
+ // pinning from holder.generation as observed.
+ c.Op.Exec, c.Op.Phase = "none", PhasePinning
+ for _, k := range []string{"pin", "prepared", "prepared_authority", "local_install", "probe", "release", "release.phase"} {
+ delete(c.Op.Journal, k)
+ }
+ return nil
+ }
+ })
+}
+
+// MarkStuck stops the operation at its current phase for an operator
+// (`carlos ops host resolve`). It never resets the replacement count and
+// never invents evidence; every later relocation transition refuses.
+func MarkStuck(opID, reason string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := RequireOp(opID)(c); err != nil {
+ return err
+ }
+ if c.Op.Kind != KindRelocating || reason == "" {
+ return ErrWrongState
+ }
+ ensureJournal(c)
+ if c.Op.Phase == PhaseInstalled && c.Op.Journal["local_install"] != "" && !preparedOpened(c) {
+ return fmt.Errorf("%w: never stop between installed and a probe", ErrRelocation)
+ }
+ if c.Op.Journal["stuck"] != "" {
+ return ErrAlreadyRecorded
+ }
+ c.Op.Journal["stuck"] = reason
+ return nil
+ }
+}
+```
+
+- [ ] **Step 4: Teach `AdmitWriter` the phase table and `released`**
+
+Apply to `internal/hostclaim/generation.go` (`git apply`):
+
+```diff
+diff --git a/internal/hostclaim/generation.go b/internal/hostclaim/generation.go
+index ffd7e058..c4261fa1 100644
+--- a/internal/hostclaim/generation.go
++++ b/internal/hostclaim/generation.go
+@@ -43,6 +43,10 @@ func AdmitWriter(in Admission) func(*Claim) error {
+ switch c.Op.Kind {
+ case KindCreating, KindDeleting, KindParking, KindPurging, KindPlacing:
+ return fmt.Errorf("%w: op kind %q cannot admit a writer", ErrWrongState, c.Op.Kind)
++ case KindRelocating:
++ if err := relocationAdmission(c, in); err != nil {
++ return err
++ }
+ }
+ h := c.Holder
+ if in.LossID != "" && !in.lossVerified {
+@@ -87,6 +91,10 @@ func AdmitWriter(in Admission) func(*Claim) error {
+
+ switch in.Coverage {
+ case "", "sealed", "clean-return":
++ case "released": // checked by relocationAdmission (L9)
++ if c.Op.Kind != KindRelocating {
++ return ErrWrongState
++ }
+ default:
+ return ErrWrongState
+ }
+@@ -261,3 +269,49 @@ func CoverParkedPredecessors(observedOpID, observedPhase string, region string,
+ return nil
+ }
+ }
++
++// relocationAdmission is §4's phase table for relocating (L1) and §6.4a's
++// `released` coverage (L9). The source may admit in starting and releasing
++// (its local admission fence closes releasing); only op.exec admits in
++// probing, cold; nobody admits in any other phase. The first probe
++// admission opens the prepared generation P, which is opened at most once.
++// `released` is legal only on that opening, when the predecessor is exactly
++// the session whose release this operation journaled and its pin reaches
++// that release's final position.
++func relocationAdmission(c *Claim, in Admission) error {
++ switch c.Op.Phase {
++ case PhaseStarting, PhaseReleasing:
++ if in.Box != c.Op.Intent["source_box"] || in.Coverage == "released" {
++ return fmt.Errorf("%w: only the source admits before pinning", ErrWrongState)
++ }
++ return nil
++ case PhaseProbing:
++ default:
++ return fmt.Errorf("%w: relocating/%s admits no writer", ErrWrongState, c.Op.Phase)
++ }
++ if in.Mode != "cold" || in.Box != c.Op.Exec || c.Op.Journal["stuck"] != "" {
++ return fmt.Errorf("%w: a probe admission is the executor's, cold", ErrWrongState)
++ }
++ p, opened := c.Op.Journal["prepared"], preparedOpened(c)
++ if p == "" || (!opened && in.NewGeneration != p) || (opened && in.NewGeneration == p) {
++ return fmt.Errorf("%w: the first probe opens P, and only it", ErrWrongState)
++ }
++ if in.Coverage != "released" {
++ return nil
++ }
++ var r Release
++ var pin RecoveryPin
++ if opened || in.PredecessorCertified || !readJournal(c.Op, "release", &r) || !readJournal(c.Op, "pin", &pin) ||
++ pin.MaxTXID < r.FinalTXID || pin.FilesSeq < r.ManifestSeq {
++ return fmt.Errorf("%w: released coverage needs this operation's release and a pin reaching it", ErrWrongState)
++ }
++ h := c.Holder
++ predecessor := AdmissionRecord{Session: h.Session, Box: h.WriterBox, Generation: h.Generation}
++ if len(h.Admissions) > 0 {
++ predecessor = h.Admissions[len(h.Admissions)-1]
++ }
++ if predecessor.Box != r.ReleasedBy || predecessor.Session != r.Session || predecessor.Generation != r.Generation {
++ return fmt.Errorf("%w: released coverage names another predecessor", ErrWrongState)
++ }
++ return nil
++}
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostclaim/ -count=1`
+Expected: PASS, every existing test included (`TestC7…` covers the conversion from `moving`, unchanged).
+
+- [ ] **Step 6: Commit**
+
+```bash
+gofmt -l internal/hostclaim
+git add internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostclaim/generation.go
+git commit -m "hostclaim: relocating within one region, released by stop evidence or a termination fence, with released coverage on P's one opening
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 2: Create names its executor once (H12, L8, P5-D2, P5-D6)
+
+Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the roster's active, unfenced boxes in the record's placement, chosen before the create CAS and recorded in its intent as `selected_executor` and `executor_enrollment`. `CompleteCreate` refuses any other box, so recovery never substitutes. Identity, placement and AWS region come from the roster (`Roster.Regions[placement].AWSRegion`); the edges snapshot gives liveness and configuration only; claim loads only balance. A fence that cannot be read refuses the choice.
+
+**Files:**
+- Modify: `internal/hostclaim/create.go`, `internal/hostclaim/create_test.go`
+- Create: `internal/hostops/executor.go`, `internal/hostops/loads.go`, `internal/hostops/executor_test.go`
+- Modify: `internal/hostops/operations.go` (`Authority.SelectExecutor`, `Attempt`), `internal/hostops/backfill_controller.go`, `internal/hostops/backfill.go` (`ProductionAuthority.Executors`, `SelectExecutor`), `internal/hostops/operations_test.go` (the fake authority)
+- Create: `internal/console/hostclaimexecutor.go`, `internal/console/hostclaimexecutor_test.go` (the console's chooser; the create door uses it once claim Plan 5's door lands, Task 10)
+
+**Interfaces:**
+- Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`.
+- Produces: `hostclaim.ExecutorSelection{Box, Enrollment}`, `hostclaim.BeginCreateOn(nh, body, opID, by, now, sel)`; `hostops.ErrNoEligibleBox`; `hostops.ExecutorChooser{Directory, Edges func(ctx) (boxchoice.Snapshot, error), Loads func(ctx) (map[string]boxchoice.Load, error), SecretsKeyFP, HomeName, Now}` with `Choose(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)`; `hostops.ClaimLoads(ctx, release.Store) (map[string]boxchoice.Load, error)`; `Authority.SelectExecutor(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)`; `ProductionAuthority.Executors *ExecutorChooser`; `(*console.Server).executorChooser(hostactors.Directory) hostops.ExecutorChooser`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `internal/hostclaim/create_test.go`:
+
+```diff
+diff --git a/internal/hostclaim/create_test.go b/internal/hostclaim/create_test.go
+index 53968923..c9a100e6 100644
+--- a/internal/hostclaim/create_test.go
++++ b/internal/hostclaim/create_test.go
+@@ -276,3 +276,27 @@ func TestCompleteCreateRefusesExecutionAuthority(t *testing.T) {
+ })
+ }
+ }
++
++func TestCreateCompletesOnlyOnTheSelectedExecutor(t *testing.T) {
++ ctx := context.Background()
++ s := release.NewDirStore(t.TempDir())
++ now := time.Date(2026, 10, 4, 1, 0, 1, 0, time.UTC)
++ reg, _ := Read(ctx, s, testHost)
++ sel := ExecutorSelection{Box: "bri-knitting", Enrollment: "enrollment-b"}
++ if _, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, ExecutorSelection{})); !errors.Is(err, ErrMalformed) {
++ t.Fatalf("create with no selection: %v", err)
++ }
++ o, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, sel))
++ if err != nil {
++ t.Fatal(err)
++ }
++ if o.Claim.Op.Intent["selected_executor"] != sel.Box || o.Claim.Op.Intent["executor_enrollment"] != sel.Enrollment || o.Claim.Op.Exec != "" {
++ t.Fatalf("intent: %+v", o.Claim.Op)
++ }
++ if _, err := Update(ctx, s, o, CompleteCreate("op_c1", "adi-knitting")); !errors.Is(err, ErrWrongState) {
++ t.Fatalf("completion substituted a box: %v", err)
++ }
++ if o, err = Update(ctx, s, o, CompleteCreate("op_c1", sel.Box)); err != nil || o.Claim.Op.Exec != sel.Box {
++ t.Fatalf("complete: %v %+v", err, o.Claim.Op)
++ }
++}
+```
+
+`internal/hostops/executor_test.go`:
+
+```go
+package hostops
+
+import (
+ "context"
+ "crypto/ed25519"
+ "crypto/rand"
+ "encoding/json"
+ "errors"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/termfence"
+)
+
+func chooserFixture(t *testing.T, boxes ...string) (ExecutorChooser, release.Store, *boxchoice.Snapshot, map[string]boxchoice.Load) {
+ t.Helper()
+ s := release.NewDirStore(t.TempDir())
+ pub, key, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1,
+ Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}},
+ Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}}
+ snap := &boxchoice.Snapshot{Complete: true}
+ yes := true
+ for i, ref := range boxes {
+ roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default",
+ Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"}
+ snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,
+ HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
+ }
+ wire, err := release.SignRootEnvelope(release.RootSigner(key), roster)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Put(context.Background(), hostactors.Key, wire, ""); err != nil {
+ t.Fatal(err)
+ }
+ loads := map[string]boxchoice.Load{}
+ x := ExecutorChooser{Directory: hostactors.Directory{Store: s, PublicKey: pub, Bucket: "bucket"}, SecretsKeyFP: "fp",
+ Edges: func(context.Context) (boxchoice.Snapshot, error) { return *snap, nil },
+ Loads: func(context.Context) (map[string]boxchoice.Load, error) { return loads, nil },
+ Now: func() time.Time { return time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) }}
+ return x, s, snap, loads
+}
+
+func chooserRecord() Record {
+ return Record{Holder: hostclaim.NewHolder{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.oncarlos.com", Region: "ie"}}
+}
+
+func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) {
+ ctx := context.Background()
+ x, _, _, loads := chooserFixture(t, "box-a", "box-b")
+ absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
+ first, err := x.Choose(ctx, absent, chooserRecord())
+ if err != nil {
+ t.Fatal(err)
+ }
+ other := map[string]string{"box-a": "box-b", "box-b": "box-a"}[first.Box]
+ if first.Enrollment != map[string]string{"box-a": strings.Repeat("a", 32), "box-b": strings.Repeat("b", 32)}[first.Box] {
+ t.Fatalf("enrollment not the roster's: %+v", first)
+ }
+ // The hash winner already runs this app: spread prefers the other box.
+ loads[first.Box] = boxchoice.Load{Hosts: 1, Apps: map[string]int{"bab/jam": 1}}
+ if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other {
+ t.Fatalf("soft spread: got %s, want %s", got.Box, other)
+ }
+ // Both run the app: the fewest hosts wins.
+ loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}}
+ if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box {
+ t.Fatalf("headroom: got %s, want %s", got.Box, first.Box)
+ }
+}
+
+func TestExecutorChooserNeverNamesAFencedLostOrUnknownBox(t *testing.T) {
+ ctx := context.Background()
+ absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
+ t.Run("lost", func(t *testing.T) {
+ x, _, _, _ := chooserFixture(t, "box-a")
+ obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Lost: []string{"box-a"}}}
+ if _, err := x.Choose(ctx, obs, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
+ t.Fatalf("chose a lost box: %v", err)
+ }
+ })
+ t.Run("fenced", func(t *testing.T) {
+ x, s, _, _ := chooserFixture(t, "box-a", "box-w")
+ f := termfence.Record{V: 1, Box: "box-a", Enrollment: strings.Repeat("a", 32), State: "terminated",
+ Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-box-a"},
+ ObservedAt: "2026-10-07T08:00:00Z", Via: termfence.ViaDescribe, RequestID: "req",
+ WrittenBy: termfence.WrittenBy{Box: "box-w", Enrollment: strings.Repeat("b", 32)}}
+ raw, _ := json.Marshal(f)
+ if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ got, err := x.Choose(ctx, absent, chooserRecord())
+ if err != nil || got.Box != "box-w" {
+ t.Fatalf("got %+v %v, want the unfenced box", got, err)
+ }
+ })
+ t.Run("an edges file for a box the roster does not hold", func(t *testing.T) {
+ x, _, snap, _ := chooserFixture(t)
+ yes := true
+ snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: "stray", EIP: "203.0.113.9", Region: "eu-west-1", Placement: "ie", Fresh: true,
+ HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
+ if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
+ t.Fatalf("chose an unenrolled box: %v", err)
+ }
+ })
+ t.Run("an incomplete snapshot", func(t *testing.T) {
+ x, _, snap, _ := chooserFixture(t, "box-a")
+ snap.Complete = false
+ if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) {
+ t.Fatalf("chose from a partial read: %v", err)
+ }
+ })
+ t.Run("an unreadable fence", func(t *testing.T) {
+ x, s, _, _ := chooserFixture(t, "box-a")
+ if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", []byte("{"), ""); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := x.Choose(ctx, absent, chooserRecord()); err == nil || errors.Is(err, ErrNoEligibleBox) {
+ t.Fatalf("an unreadable fence must refuse, not skip: %v", err)
+ }
+ })
+}
+
+func TestClaimLoadsCountsExecutorsPerBoxAndApp(t *testing.T) {
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC)
+ for i, tc := range []struct{ host, app, box string }{
+ {"a.bab.oncarlos.com", "jam", "box-a"}, {"b.bab.oncarlos.com", "jam", "box-a"}, {"c.bab.oncarlos.com", "tea", "box-b"},
+ } {
+ nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + tc.app + "/instances/" + tc.host + ".json", Account: "bab", App: tc.app, Host: tc.host, Region: "ie", RecordRegion: "ie", CreatedAt: created}
+ body := []byte(`{"host":"` + tc.host + `","account":"bab","app":"` + tc.app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`)
+ o, _ := hostclaim.Read(ctx, s, tc.host)
+ o, err := hostclaim.Update(ctx, s, o, hostclaim.BeginCreateOn(nh, body, "c", "console", created, hostclaim.ExecutorSelection{Box: tc.box, Enrollment: "e"}))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if i < 2 { // the third is still creating: it has no executor yet
+ if _, err := hostclaim.Update(ctx, s, o, hostclaim.CompleteCreate("c", tc.box)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ if _, err := s.Put(ctx, "control/hosts/broken.bab.oncarlos.com.json", []byte("{"), ""); err != nil {
+ t.Fatal(err)
+ }
+ loads, err := ClaimLoads(ctx, s)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 0 {
+ t.Fatalf("loads: %+v", loads)
+ }
+}
+
+func TestProductionAuthoritySelectsOnlyWithAChooser(t *testing.T) {
+ if _, err := (ProductionAuthority{}).SelectExecutor(context.Background(), hostclaim.Observed{}, Record{}); !errors.Is(err, ErrContract) {
+ t.Fatalf("selected with no chooser: %v", err)
+ }
+}
+```
+
+`internal/console/hostclaimexecutor_test.go`:
+
+```go
+package console
+
+import (
+ "context"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) {
+ store := release.NewDirStore(t.TempDir())
+ s := &Server{cfg: Config{Releases: store, HomeRegion: "ie"}}
+ x := s.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"})
+ if x.Edges == nil || x.Loads == nil || x.Now == nil || x.HomeName != "ie" || x.SecretsKeyFP != s.expectedSecretsFP() {
+ t.Fatalf("chooser: %+v", x)
+ }
+ snap, err := x.Edges(context.Background())
+ if err != nil || len(snap.Boxes) != 0 {
+ t.Fatalf("an empty bucket: %+v %v", snap, err)
+ }
+ loads, err := x.Loads(context.Background())
+ if err != nil || len(loads) != 0 {
+ t.Fatalf("no claims: %+v %v", loads, err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ ./internal/console/ -run 'SelectedExecutor|ExecutorChooser|ClaimLoads|ProductionAuthoritySelects' -count=1`
+Expected: FAIL to compile: `undefined: ExecutorSelection`, `undefined: BeginCreateOn`, `undefined: ExecutorChooser`, `undefined: ClaimLoads`, `s.executorChooser undefined`.
+
+- [ ] **Step 3: Record the selection in the create intent**
+
+Apply to `internal/hostclaim/create.go`:
+
+```diff
+diff --git a/internal/hostclaim/create.go b/internal/hostclaim/create.go
+index c8fe97cc..39badc04 100644
+--- a/internal/hostclaim/create.go
++++ b/internal/hostclaim/create.go
+@@ -80,6 +80,32 @@ func BeginCreate(nh NewHolder, recordBody []byte, opID, by string, now time.Time
+ }
+ }
+
++// ExecutorSelection is the box create selects once, with
++// boxchoice.Choose over the eligible set (claim P5-D2, region HA §5.2,
++// H12), and its roster enrollment.
++type ExecutorSelection struct {
++ Box, Enrollment string
++}
++
++// BeginCreateOn is BeginCreate recording the selected executor in the
++// create intent, in the same step 2 CAS, before any effect (claim §5.1,
++// P5-D6 `selected_executor`, `executor_enrollment`). CompleteCreate then
++// sets op.exec to exactly that box; recovery never substitutes another.
++func BeginCreateOn(nh NewHolder, recordBody []byte, opID, by string, now time.Time, sel ExecutorSelection) func(*Claim) error {
++ begin := BeginCreate(nh, recordBody, opID, by, now)
++ return func(c *Claim) error {
++ if !validBoxReference(sel.Box) || sel.Box == "none" || sel.Enrollment == "" {
++ return fmt.Errorf("%w: create needs a selected executor", ErrMalformed)
++ }
++ if err := begin(c); err != nil {
++ return err
++ }
++ c.Op.Intent["selected_executor"] = sel.Box
++ c.Op.Intent["executor_enrollment"] = sel.Enrollment
++ return nil
++ }
++}
++
+ func CompleteCreate(opID, execBox string) func(*Claim) error {
+ return func(c *Claim) error {
+ if err := RequireOp(opID)(c); err != nil {
+@@ -88,6 +114,9 @@ func CompleteCreate(opID, execBox string) func(*Claim) error {
+ if c.Op.Kind != KindCreating || execBox == "" {
+ return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind)
+ }
++ if selected := c.Op.Intent["selected_executor"]; selected != "" && execBox != selected {
++ return fmt.Errorf("%w: create selected %s; completion never substitutes %s", ErrWrongState, selected, execBox)
++ }
+ if err := requireNoCreateAuthority(c); err != nil {
+ return err
+ }
+```
+
+- [ ] **Step 4: The chooser and the loads**
+
+`internal/hostops/executor.go`:
+
+```go
+package hostops
+
+import (
+ "context"
+ "errors"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostclaim"
+)
+
+// ErrNoEligibleBox is a create refused because no box in the placement is
+// eligible (region HA §5.2). The console maps it to today's refusal.
+var ErrNoEligibleBox = errors.New("hostops: region has no box that can take this")
+
+// ExecutorChooser is create's executor selection: claim P5-D2's choose()
+// over the complete eligible set (region HA spec 1 §5.2, H12, L8). It is
+// made once, before the create CAS, and recorded in the create intent.
+type ExecutorChooser struct {
+ Directory hostactors.Directory
+ // Edges is one read of every box's edges file with the caller's
+ // liveness verdict per box. An incomplete snapshot selects nothing.
+ Edges func(context.Context) (boxchoice.Snapshot, error)
+ // Loads counts claimed hosts per box and per "acct/app", from the
+ // claim-recovery listing (claim §5.1 step 5). A stale count only
+ // affects balance, never safety.
+ Loads func(context.Context) (map[string]boxchoice.Load, error)
+ SecretsKeyFP string
+ HomeName string
+ Now func() time.Time
+}
+
+// Choose returns the box, and its roster enrollment, that a new host in
+// r's placement is created on. Identity, placement and AWS region come from
+// the roster; a box with a valid termination fence, or not active, is never
+// chosen; a box on the claim's lost list is never chosen.
+func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) {
+ placement := r.Holder.Region
+ if x.Edges == nil || x.Loads == nil || x.Now == nil || placement == "" {
+ return hostclaim.ExecutorSelection{}, ErrContract
+ }
+ o, err := x.Directory.Read(ctx)
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err
+ }
+ region, ok := o.Roster.Regions[placement]
+ if !ok || region.AWSRegion == "" {
+ return hostclaim.ExecutorSelection{}, ErrContract
+ }
+ roster := map[string]boxchoice.RosterBox{}
+ for ref, b := range o.Roster.Boxes {
+ if b.State != "active" || b.Region != placement {
+ continue
+ }
+ fenced, err := x.Directory.TerminationFence(ctx, o, ref, x.Now())
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err // C6: a read error never means "not fenced"
+ }
+ if !fenced {
+ roster[ref] = boxchoice.RosterBox{Placement: b.Region, Cloud: b.Cloud}
+ }
+ }
+ snap, err := x.Edges(ctx)
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err
+ }
+ loads, err := x.Loads(ctx)
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err
+ }
+ lost := map[string]bool{}
+ if !obs.Absent && obs.Claim != nil {
+ for _, box := range obs.Claim.Lost {
+ lost[box] = true
+ }
+ }
+ cands := boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, lost)
+ best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App)
+ if !ok {
+ return hostclaim.ExecutorSelection{}, ErrNoEligibleBox
+ }
+ return hostclaim.ExecutorSelection{Box: best.BoxID, Enrollment: o.Roster.Boxes[best.BoxID].Enrollment}, nil
+}
+```
+
+`internal/hostops/loads.go`:
+
+```go
+package hostops
+
+import (
+ "context"
+ "log"
+ "strings"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// ClaimLoads counts, from one listing of control/hosts/, the hosts whose
+// claim names each box as op.exec, in total and per "acct/app": choose()'s
+// second rule until spec 2 (region HA §5.2, claim P5-D2). The count only
+// balances, so a claim that cannot be read is logged and left out; a
+// listing that fails is an error, because a count of nothing would send
+// every create to one box.
+func ClaimLoads(ctx context.Context, s release.Store) (map[string]boxchoice.Load, error) {
+ keys, err := s.List(ctx, "control/hosts/")
+ if err != nil {
+ return nil, err
+ }
+ out := map[string]boxchoice.Load{}
+ for _, key := range keys {
+ name, ok := strings.CutPrefix(key, "control/hosts/")
+ host, isJSON := strings.CutSuffix(name, ".json")
+ if !ok || !isJSON || strings.Contains(host, "/") {
+ continue
+ }
+ obs, err := hostclaim.Read(ctx, s, host)
+ if err != nil {
+ log.Printf("hostops: claim loads: %s: %v", host, err)
+ continue
+ }
+ if obs.Absent || obs.Claim.Holder == nil || obs.Claim.Op.Exec == "" || obs.Claim.Op.Exec == "none" {
+ continue
+ }
+ box, h := obs.Claim.Op.Exec, obs.Claim.Holder
+ l := out[box]
+ if l.Apps == nil {
+ l.Apps = map[string]int{}
+ }
+ l.Hosts++
+ l.Apps[h.Account+"/"+h.App]++
+ out[box] = l
+ }
+ return out, nil
+}
+```
+
+- [ ] **Step 5: Select before the create CAS**
+
+Apply to `internal/hostops/operations.go`:
+
+```diff
+diff --git a/internal/hostops/operations.go b/internal/hostops/operations.go
+index 8313b80e..13865c27 100644
+--- a/internal/hostops/operations.go
++++ b/internal/hostops/operations.go
+@@ -14,6 +14,10 @@ import (
+ // These callbacks are authority providers, not optional best-effort hints.
+ // Production construction waits for the decisions named in this plan.
+ type Authority interface {
++ // SelectExecutor chooses the new host's executor once, before the
++ // create CAS (claim P5-D2, region HA §5.2). Production is
++ // ExecutorChooser.Choose.
++ SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)
+ Create(context.Context, hostclaim.Observed, Record, string) error
+ Executor(context.Context, hostclaim.Observed, Record) (string, error)
+ StartDelete(context.Context, hostclaim.Observed, Record) error
+@@ -46,11 +50,16 @@ func (c Controller) Attempt(ctx context.Context, before hostclaim.Observed, r Re
+ if before.Host != r.Holder.Host || r.Holder.CreatedAt.IsZero() || id == "" {
+ return hostclaim.Observed{}, ErrIdentity
+ }
++ // The executor is selected once, before the CAS, and recorded in it.
++ sel, err := c.Authority.SelectExecutor(ctx, before, r)
++ if err != nil {
++ return hostclaim.Observed{}, err
++ }
+ // Last pre-CAS read of registration, open admission and purge reservation.
+ if err := c.Authority.Create(ctx, before, r, id); err != nil {
+ return hostclaim.Observed{}, err
+ }
+- obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now()))
++ obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreateOn(r.Holder, r.Body, id, by, c.Now(), sel))
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+```
+
+Apply to `internal/hostops/backfill_controller.go` (the backfill grants no create):
+
+```diff
+diff --git a/internal/hostops/backfill_controller.go b/internal/hostops/backfill_controller.go
+index 08ebd856..e0ca8c49 100644
+--- a/internal/hostops/backfill_controller.go
++++ b/internal/hostops/backfill_controller.go
+@@ -23,6 +23,9 @@ func (a BackfillAuthority) Adoption(ctx context.Context, r Record) (string, stri
+ }
+ func (a BackfillAuthority) ClaimBackfill() *Backfill { return a.p.Backfill }
+
++func (a BackfillAuthority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error) {
++ return hostclaim.ExecutorSelection{}, ErrContract
++}
+ func (a BackfillAuthority) Create(context.Context, hostclaim.Observed, Record, string) error {
+ return ErrContract
+ }
+```
+
+Apply to `internal/hostops/backfill.go`:
+
+```diff
+diff --git a/internal/hostops/backfill.go b/internal/hostops/backfill.go
+index e20b1982..806f59e2 100644
+--- a/internal/hostops/backfill.go
++++ b/internal/hostops/backfill.go
+@@ -89,6 +89,18 @@ type ProductionAuthority struct {
+ Signer release.Signer
+ Evidence hostactors.Evidence
+ Backfill *Backfill
++ // Executors is create's executor selection (region HA §5.2, L8). The
++ // create door's Authority uses it through SelectExecutor.
++ Executors *ExecutorChooser
++}
++
++// SelectExecutor is Authority's create-time selection: choose() once,
++// before the create CAS. Without a chooser it refuses.
++func (a ProductionAuthority) SelectExecutor(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) {
++ if a.Executors == nil {
++ return hostclaim.ExecutorSelection{}, ErrContract
++ }
++ return a.Executors.Choose(ctx, obs, r)
+ }
+
+ func reportKey(reservation, box string) (string, error) {
+```
+
+Apply to `internal/hostops/operations_test.go` (the fake authority implements the new method):
+
+```diff
+diff --git a/internal/hostops/operations_test.go b/internal/hostops/operations_test.go
+index 42ebf9d1..e2ba4df6 100644
+--- a/internal/hostops/operations_test.go
++++ b/internal/hostops/operations_test.go
+@@ -17,6 +17,9 @@ import (
+
+ type authority struct{ err error }
+
++func (a *authority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error) {
++ return hostclaim.ExecutorSelection{Box: "box-a", Enrollment: "enrollment-a"}, a.err
++}
+ func (a *authority) Create(context.Context, hostclaim.Observed, Record, string) error { return a.err }
+ func (a *authority) Executor(context.Context, hostclaim.Observed, Record) (string, error) {
+ return "box-a", a.err
+```
+
+`internal/console/hostclaimexecutor.go`:
+
+```go
+package console
+
+import (
+ "context"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostops"
+)
+
+// executorChooser is create's executor choice (region HA §5.2, L8) over
+// console's live edges read and the claim listing's loads. The create
+// door's hostops.ProductionAuthority uses it as Executors once that door
+// lands; nothing calls it before then.
+func (s *Server) executorChooser(d hostactors.Directory) hostops.ExecutorChooser {
+ return hostops.ExecutorChooser{Directory: d,
+ Edges: func(ctx context.Context) (boxchoice.Snapshot, error) {
+ snap, err := s.liveEdges(ctx)
+ if err != nil {
+ return boxchoice.Snapshot{}, err
+ }
+ return boxSnapshot(snap), nil
+ },
+ Loads: func(ctx context.Context) (map[string]boxchoice.Load, error) {
+ return hostops.ClaimLoads(ctx, s.cfg.Releases)
+ },
+ SecretsKeyFP: s.expectedSecretsFP(), HomeName: s.cfg.HomeRegion, Now: time.Now}
+}
+```
+
+- [ ] **Step 6: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ -count=1 && nice -n 19 go test ./internal/console/ -run 'CreateExecutorChooser|Claim|BoxChoice' -count=1 && go vet ./internal/console/ ./internal/host/`
+Expected: PASS; vet clean (no other `hostops.Authority` implementation exists on main).
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add internal/hostclaim/create.go internal/hostclaim/create_test.go internal/hostops/ internal/console/hostclaimexecutor.go internal/console/hostclaimexecutor_test.go
+git commit -m "hostops: create chooses its executor once with boxchoice over the roster and records it in the create intent
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 3: Reconcile's row cases and who may wake (§5.3, L12)
+
+`RelocationRow` is §5.3's table as a pure decision over a fresh claim read: ready on the serving executor and on the source until its admission fence closes; held with `hold_op` = the operation for any other participant; retiring (held, never minted) for a box owing a `park-local`; nothing, or an unexplained held row, otherwise. A box on `lost` is never ready. `WakePhase` gains `relocating`: the source wakes in `starting` and `releasing`, the target only probes, in `probing`, and a stuck operation runs nothing. `Policy` routes relocating actions to its own owner, never to moves'.
+
+**Files:**
+- Create: `internal/hostbox/relocationrows.go`, `internal/hostbox/relocationrows_test.go`
+- Modify: `internal/hostbox/wakepolicy.go`
+
+**Interfaces:**
+- Consumes: Task 1's phase constants.
+- Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostbox/relocationrows_test.go`:
+
+```go
+package hostbox
+
+import (
+ "context"
+ "errors"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostwrite"
+)
+
+func relocating(phase, exec string) *hostclaim.Claim {
+ return &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: exec,
+ Participants: []string{"src", "dst"}, Intent: map[string]string{"source_box": "src", "target_box": "dst", "region": "ie"}}}
+}
+
+func TestRelocationRowCases(t *testing.T) {
+ withChore := func(c *hostclaim.Claim, box string) *hostclaim.Claim {
+ c.Obligations = append(c.Obligations, hostclaim.Obligation{ID: "o", Op: "rel-0", Kind: "park-local", Box: box, Owner: "box:" + box, Retirement: "r"})
+ return c
+ }
+ serving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "src", Participants: []string{"src"}}}
+ lostSource := relocating(hostclaim.PhaseStarting, "src")
+ lostSource.Lost = []string{"src"}
+ for _, tc := range []struct {
+ name string
+ c *hostclaim.Claim
+ box string
+ exists, fenceClosed bool
+ want RowDecision
+ }{
+ {"the serving executor", serving, "src", false, false, RowDecision{Case: RowReady}},
+ {"a serving host elsewhere, no row", serving, "dst", false, false, RowDecision{Case: RowNone}},
+ {"a serving host elsewhere, a row", serving, "dst", true, false, RowDecision{Case: RowUnexplained}},
+ {"the source in starting", relocating(hostclaim.PhaseStarting, "src"), "src", true, false, RowDecision{Case: RowReady}},
+ {"the source releasing, fence open", relocating(hostclaim.PhaseReleasing, "src"), "src", true, false, RowDecision{Case: RowReady}},
+ {"the source releasing, fence closed", relocating(hostclaim.PhaseReleasing, "src"), "src", true, true, RowDecision{Case: RowHeld, HoldOp: "rel-1"}},
+ {"the source in pinning", relocating(hostclaim.PhasePinning, "none"), "src", true, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}},
+ {"the target in starting", relocating(hostclaim.PhaseStarting, "src"), "dst", false, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}},
+ {"the target executing in probing", relocating(hostclaim.PhaseProbing, "dst"), "dst", true, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}},
+ {"a lost source", lostSource, "src", true, false, RowDecision{Case: RowUnexplained}},
+ {"a former executor owing park-local", withChore(serving, "old"), "old", true, false, RowDecision{Case: RowRetiring, HoldOp: "rel-0"}},
+ {"a former executor with no row", withChore(serving, "old"), "old", false, false, RowDecision{Case: RowNone}},
+ {"no claim, a row", nil, "src", true, false, RowDecision{Case: RowUnexplained}},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := RelocationRow(tc.c, tc.box, tc.exists, tc.fenceClosed); got != tc.want {
+ t.Fatalf("got %+v, want %+v", got, tc.want)
+ }
+ })
+ }
+}
+
+func TestWakePhaseRelocating(t *testing.T) {
+ for _, tc := range []struct {
+ phase, exec, box string
+ probe, want bool
+ }{
+ {hostclaim.PhaseStarting, "src", "src", false, true},
+ {hostclaim.PhaseReleasing, "src", "src", false, true},
+ {hostclaim.PhaseStarting, "src", "src", true, false},
+ {hostclaim.PhaseInstalling, "dst", "dst", false, false},
+ {hostclaim.PhaseInstalled, "dst", "dst", true, false},
+ {hostclaim.PhaseProbing, "dst", "dst", true, true},
+ {hostclaim.PhaseProbing, "dst", "dst", false, false},
+ {hostclaim.PhaseProbeFailed, "dst", "dst", true, false},
+ {hostclaim.PhasePinning, "none", "src", false, false},
+ } {
+ if got := WakePhase(relocating(tc.phase, tc.exec), tc.box, tc.probe) == nil; got != tc.want {
+ t.Fatalf("%+v: %v", tc, got)
+ }
+ }
+ stuck := relocating(hostclaim.PhaseProbing, "dst")
+ stuck.Op.Journal = map[string]string{"stuck": "operator"}
+ if WakePhase(stuck, "dst", true) == nil {
+ t.Fatal("a stuck operation probed")
+ }
+}
+
+func TestPolicyRoutesRelocationToItsOwner(t *testing.T) {
+ c := relocating(hostclaim.PhaseStarting, "src")
+ held := func(context.Context, string) (string, error) { return "", nil }
+ s := Selection{Host: "x.example", Box: "src", Op: "rel-1", Action: "spawn"}
+ moves := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { return nil }
+ if err := (Policy{Held: held, Moves: moves}).Check(context.Background(), c, s, hostwrite.Identity{}); !errors.Is(err, ErrChanged) {
+ t.Fatalf("moves' policy authorized a relocation: %v", err)
+ }
+ called := false
+ reloc := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error {
+ called = true
+ return nil
+ }
+ if err := (Policy{Held: held, Relocation: reloc}).Check(context.Background(), c, s, hostwrite.Identity{}); err != nil || !called {
+ t.Fatalf("relocation policy: %v called=%v", err, called)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostbox/ -run 'RelocationRow|WakePhaseRelocating|PolicyRoutesRelocation' -count=1`
+Expected: FAIL to compile: `undefined: RelocationRow`, `unknown field Relocation in struct literal`.
+
+- [ ] **Step 3: The row decision**
+
+`internal/hostbox/relocationrows.go`:
+
+```go
+package hostbox
+
+import (
+ "slices"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+)
+
+// RowCase is what reconcile does with one relocatable host's registry row
+// on this box (region HA spec 1 §5.3, claim C4 as amended by L12).
+type RowCase int
+
+const (
+ // RowNone: mint nothing, and there is no row to keep.
+ RowNone RowCase = iota
+ // RowReady: mint the row, or keep it, ready.
+ RowReady
+ // RowHeld: mint the row, or keep it, held with HoldOp.
+ RowHeld
+ // RowRetiring: keep an existing row held; mint none. A park-local chore
+ // this box owes retires it under the host guard (claim P6-D2).
+ RowRetiring
+ // RowUnexplained: the claim gives this box no role, yet a row exists.
+ // Keep it held, never serve it, and alert. Never delete it by hostname.
+ RowUnexplained
+)
+
+// RowDecision is RelocationRow's answer.
+type RowDecision struct {
+ Case RowCase
+ HoldOp string
+}
+
+// RelocationRow decides one row from a fresh claim read. box is this box's
+// reference; exists says whether the registry holds a row for the host;
+// fenceClosed says whether this box's local admission fence for the host is
+// closed (the source's release has begun, §5.8's release sequence step 1).
+//
+// op.exec, serving ready
+// relocating source in starting, or releasing
+// until its admission fence closes ready
+// any other participant of a running operation held, hold_op = op id
+// owner of an open park-local chore for the host retiring (held, no mint)
+// none of these none; an existing row is unexplained
+//
+// A box on the claim's lost list is never ready. A nil or holderless claim
+// (an error is the caller's: claim C6, keep rows and mint none) gives none.
+func RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision {
+ if c == nil || c.Holder == nil || box == "" {
+ return unexplained(exists)
+ }
+ lost := slices.Contains(c.Lost, box)
+ switch {
+ case lost:
+ case c.Op.Kind == hostclaim.KindServing && c.Op.ID == "" && c.Op.Exec == box:
+ return RowDecision{Case: RowReady}
+ case c.Op.Kind == hostclaim.KindRelocating && c.Op.Exec == box && c.Op.Intent["source_box"] == box &&
+ (c.Op.Phase == hostclaim.PhaseStarting || (c.Op.Phase == hostclaim.PhaseReleasing && !fenceClosed)):
+ return RowDecision{Case: RowReady}
+ case c.Op.Kind != hostclaim.KindServing && c.Op.ID != "" && slices.Contains(c.Op.Participants, box):
+ return RowDecision{Case: RowHeld, HoldOp: c.Op.ID}
+ }
+ for _, o := range c.Obligations {
+ if o.Kind == "park-local" && o.Box == box {
+ if !exists {
+ return RowDecision{Case: RowNone}
+ }
+ return RowDecision{Case: RowRetiring, HoldOp: o.Op}
+ }
+ }
+ return unexplained(exists)
+}
+
+func unexplained(exists bool) RowDecision {
+ if exists {
+ return RowDecision{Case: RowUnexplained}
+ }
+ return RowDecision{Case: RowNone}
+}
+```
+
+- [ ] **Step 4: Wake phase and policy**
+
+Apply to `internal/hostbox/wakepolicy.go`:
+
+```diff
+diff --git a/internal/hostbox/wakepolicy.go b/internal/hostbox/wakepolicy.go
+index 9c3c00d6..5ab7b820 100644
+--- a/internal/hostbox/wakepolicy.go
++++ b/internal/hostbox/wakepolicy.go
+@@ -27,6 +27,17 @@ func WakePhase(c *hostclaim.Claim, box string, probe bool) error {
+ if c.Op.ID != "" && ((!probe && c.Op.Phase == "committed") || (probe && c.Op.Phase == "probing")) {
+ return nil
+ }
++ case hostclaim.KindRelocating:
++ // Region HA §5.8: the source keeps serving in starting and in
++ // releasing (its local admission fence closes releasing); the
++ // target runs only probes, in probing. Nobody else, ever.
++ if c.Op.ID == "" || c.Op.Journal["stuck"] != "" {
++ break
++ }
++ source := c.Op.Intent["source_box"] == box && (c.Op.Phase == hostclaim.PhaseStarting || c.Op.Phase == hostclaim.PhaseReleasing)
++ if (!probe && source) || (probe && c.Op.Phase == hostclaim.PhaseProbing) {
++ return nil
++ }
+ case hostclaim.KindReverting, hostclaim.KindRestoring:
+ if c.Op.ID != "" && ((!probe && (c.Op.Phase == "probed" || c.Op.Phase == "opened")) || (probe && c.Op.Phase == "probing")) {
+ return nil
+@@ -37,10 +48,13 @@ func WakePhase(c *hostclaim.Claim, box string, probe bool) error {
+
+ // Policy separates ordinary writes from the moves-owned held actions.
+ // Moves validates roles, held inputs and phase-specific rights with the same capture.
++// Relocation does the same for a relocating claim (region HA §5.8); with it
++// nil, every relocating action is refused.
+ type Policy struct {
+- Inputs func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error
+- Moves func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error
+- Held func(context.Context, string) (string, error)
++ Inputs func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error
++ Moves func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error
++ Relocation func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error
++ Held func(context.Context, string) (string, error)
+ }
+
+ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id hostwrite.Identity) error {
+@@ -65,7 +79,11 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h
+ }
+ return Serving(c, s)
+ }
+- if p.Moves == nil || s.Op == "" || c.Op.ID != s.Op || c.Op.Exec != s.Box {
++ owner := p.Moves
++ if c.Op.Kind == hostclaim.KindRelocating {
++ owner = p.Relocation
++ }
++ if owner == nil || s.Op == "" || c.Op.ID != s.Op || c.Op.Exec != s.Box {
+ return ErrChanged
+ }
+ switch s.Action {
+@@ -84,7 +102,7 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h
+ default:
+ return ErrChanged
+ }
+- return p.Moves(ctx, c, s, id)
++ return owner(ctx, c, s, id)
+ }
+
+ const RequestLimit = 120 * time.Second
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostbox/ ./internal/activator/ -run 'Relocation|WakePhase|Policy|ClaimWake|Wake' -count=1`
+Expected: PASS (`activator`'s claim wake calls `WakePhase`; serving and moves cases are unchanged).
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/hostbox/relocationrows.go internal/hostbox/relocationrows_test.go internal/hostbox/wakepolicy.go
+git commit -m "hostbox: reconcile's relocation row cases, and only the source or a probing target wakes under relocating
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 4: Writers stopped, with the replicator still open (L10)
+
+Claim §6.1, "Release before full stop": release step 1 needs the admission fence, settled spawns and drained work (the drain's own facts), an empty cgroup with no members, and a complete descriptor scan in which the only holders of the database, WAL and SHM files are descriptors positively identified as the replicator's: its process (PID and start time, field 22 of `/proc/<pid>/stat`) and each descriptor number and target. A descriptor seen in a thread whose table the leader's does not match is never exempt. `Collect` is unchanged: full stop evidence follows once the replicator closes.
+
+**Files:**
+- Create: `internal/stopevidence/writers.go`, `internal/stopevidence/writers_linux.go`, `internal/stopevidence/writers_other.go`
+- Test: `internal/stopevidence/writers_linux_test.go`
+
+**Interfaces:**
+- Consumes: `realFDOps`, `readProcDir`, `procGone`, `ErrIncomplete`, `ErrInvalidInput`, `ErrUnsupported`, `ErrNoCgroupEvidence`, `ErrPopulated`, `ErrMembers` (all on main).
+- Produces: `stopevidence.Descriptor{PID, FD int; Target string}`, `stopevidence.WritersInputs`, `stopevidence.WritersEvidence`, `stopevidence.WritersStopped(ctx, WritersInputs) (WritersEvidence, error)`, `ErrWorkOpen`, `ErrForeignHolder`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/stopevidence/writers_linux_test.go` (it reuses `evidenceGroup` from `evidence_linux_test.go`):
+
+```go
+//go:build linux
+
+package stopevidence
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strconv"
+ "testing"
+)
+
+// writersProc builds a /proc fixture: the agent (pid 20, threads 20 and
+// 7, sharing the table) holds the database on fds 5 and 6.
+func writersProc(t *testing.T) (string, string) {
+ t.Helper()
+ root, db := t.TempDir(), filepath.Join(t.TempDir(), "data.db")
+ for _, tid := range []string{"20", "7"} {
+ dir := filepath.Join(root, "20", "task", tid, "fd")
+ if err := os.MkdirAll(dir, 0700); err != nil {
+ t.Fatal(err)
+ }
+ for fd, target := range map[int]string{5: db, 6: db + "-wal", 9: "/dev/null"} {
+ if err := os.Symlink(target, filepath.Join(dir, strconv.Itoa(fd))); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ if err := os.WriteFile(filepath.Join(root, "20", "stat"), []byte("20 (carlos edge) S 1 20 20 0 -1 4194560 1 0 0 0 0 0 0 0 20 0 9 0 777 0 0"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ return root, db
+}
+
+func writersInputs(root, db string) WritersInputs {
+ return WritersInputs{Host: "shop.example", FenceClosed: true, SpawnsSettled: true, WorkDrained: true, Cgroup: &evidenceGroup{},
+ DBPaths: []string{db}, ReplicatorPID: 20, ReplicatorStart: "777", ProcRoot: root,
+ Replicator: []Descriptor{{PID: 20, FD: 5, Target: db}, {PID: 20, FD: 6, Target: db + "-wal"}}}
+}
+
+func TestWritersStoppedAllowsOnlyTheReplicatorsDescriptors(t *testing.T) {
+ root, db := writersProc(t)
+ ev, err := WritersStopped(t.Context(), writersInputs(root, db))
+ if err != nil || len(ev.Held) != 2 {
+ t.Fatalf("the replicator alone: %+v %v", ev, err)
+ }
+ for _, tc := range []struct {
+ name string
+ change func(*WritersInputs)
+ want error
+ }{
+ {"admission open", func(in *WritersInputs) { in.FenceClosed = false }, ErrWorkOpen},
+ {"timer work running", func(in *WritersInputs) { in.WorkDrained = false }, ErrWorkOpen},
+ {"cgroup populated", func(in *WritersInputs) { in.Cgroup = &evidenceGroup{populated: true} }, ErrPopulated},
+ {"cgroup members", func(in *WritersInputs) { in.Cgroup = &evidenceGroup{members: []int{8}} }, ErrMembers},
+ {"one replicator descriptor not identified", func(in *WritersInputs) { in.Replicator = in.Replicator[:1] }, ErrForeignHolder},
+ {"another process start", func(in *WritersInputs) { in.ReplicatorStart = "778" }, ErrForeignHolder},
+ {"descriptor of another process", func(in *WritersInputs) { in.Replicator[0].PID = 21 }, ErrInvalidInput},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ in := writersInputs(root, db)
+ tc.change(&in)
+ if _, err := WritersStopped(t.Context(), in); !errors.Is(err, tc.want) {
+ t.Fatalf("got %v, want %v", err, tc.want)
+ }
+ })
+ }
+}
+
+func TestWritersStoppedRefusesAnyOtherHolder(t *testing.T) {
+ for _, tc := range []struct{ name, pid, tid string }{
+ {"another process", "30", "30"},
+ {"an unshared thread of the agent", "20", "8"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ root, db := writersProc(t)
+ dir := filepath.Join(root, tc.pid, "task", tc.tid, "fd")
+ if err := os.MkdirAll(dir, 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(db+"-shm", filepath.Join(dir, "11")); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) {
+ t.Fatalf("got %v", err)
+ }
+ })
+ }
+ t.Run("a thread's descriptor the leader's table does not show", func(t *testing.T) {
+ root, db := writersProc(t)
+ if err := os.Remove(filepath.Join(root, "20", "task", "20", "fd", "5")); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) {
+ t.Fatalf("got %v", err)
+ }
+ })
+ t.Run("an unreadable descriptor directory", func(t *testing.T) {
+ root, db := writersProc(t)
+ if _, err := WritersStopped(t.Context(), writersInputs(filepath.Join(root, "missing"), db)); !errors.Is(err, ErrIncomplete) {
+ t.Fatalf("got %v", err)
+ }
+ })
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/stopevidence/ -run 'Writers' -count=1`
+Expected: FAIL to compile: `undefined: WritersInputs`.
+
+- [ ] **Step 3: The observation**
+
+`internal/stopevidence/writers.go`:
+
+```go
+package stopevidence
+
+import (
+ "context"
+ "errors"
+ "path/filepath"
+ "sort"
+ "time"
+
+ "github.com/carlosframework/platform/internal/controlname"
+)
+
+var (
+ ErrWorkOpen = errors.New("stopevidence: admission, spawns or work not drained")
+ ErrForeignHolder = errors.New("stopevidence: a descriptor other than the replicator's holds the database")
+)
+
+// Descriptor is one open file descriptor of one process, as /proc shows it.
+type Descriptor struct {
+ PID int
+ FD int
+ Target string
+}
+
+// WritersInputs is claim §6.1's writers-stopped observation (L10, region
+// HA spec 1 §5.8 release step 1). The caller's drain acknowledgement
+// supplies the three drained facts; this package checks the kernel.
+type WritersInputs struct {
+ Host string
+ // FenceClosed, SpawnsSettled and WorkDrained are the release
+ // controller's durable admission fence, settled spawn reservations, and
+ // drained application and timer work.
+ FenceClosed, SpawnsSettled, WorkDrained bool
+ Cgroup interface {
+ Populated() (bool, error)
+ Members() ([]int, error)
+ }
+ DBPaths []string
+ // Replicator names the only descriptors that may hold the database:
+ // the agent's own replicator for this host, by PID, its process start
+ // time (field 22 of /proc/<pid>/stat), and each descriptor.
+ ReplicatorPID int
+ ReplicatorStart string
+ Replicator []Descriptor
+ ProcRoot string
+}
+
+// WritersEvidence records what WritersStopped observed.
+type WritersEvidence struct {
+ Host string
+ CheckedAt time.Time
+ CheckedPaths []string
+ Held []Descriptor // the replicator descriptors still open
+ ProcRoot string
+}
+
+// WritersStopped passes only when admission is fenced, spawns settled,
+// work drained, the host's cgroup empty with no members, and every holder
+// of the database, WAL or SHM files is one of the replicator's identified
+// descriptors in the replicator's own process, started at
+// ReplicatorStart. An incomplete scan, an unreadable start time, or any
+// other holder fails. It grants no handover: full stop evidence (Collect)
+// still follows, after the replicator closes.
+func WritersStopped(ctx context.Context, in WritersInputs) (WritersEvidence, error) {
+ if ctx == nil || !controlname.ValidHostShape(in.Host) || len(in.DBPaths) == 0 || in.ReplicatorPID <= 0 || in.ReplicatorStart == "" {
+ return WritersEvidence{}, ErrInvalidInput
+ }
+ if !in.FenceClosed || !in.SpawnsSettled || !in.WorkDrained {
+ return WritersEvidence{}, ErrWorkOpen
+ }
+ if in.Cgroup == nil {
+ return WritersEvidence{}, ErrNoCgroupEvidence
+ }
+ paths, err := dbPaths(in.DBPaths)
+ if err != nil {
+ return WritersEvidence{}, err
+ }
+ if populated, err := in.Cgroup.Populated(); err != nil {
+ return WritersEvidence{}, err
+ } else if populated {
+ return WritersEvidence{}, ErrPopulated
+ }
+ if members, err := in.Cgroup.Members(); err != nil {
+ return WritersEvidence{}, err
+ } else if len(members) != 0 {
+ return WritersEvidence{}, ErrMembers
+ }
+ root := in.ProcRoot
+ if root == "" {
+ root = "/proc"
+ }
+ held, err := scanDescriptors(ctx, root, paths)
+ if err != nil {
+ return WritersEvidence{}, err
+ }
+ allowed := map[Descriptor]bool{}
+ for _, d := range in.Replicator {
+ if d.PID != in.ReplicatorPID {
+ return WritersEvidence{}, ErrInvalidInput
+ }
+ allowed[Descriptor{PID: d.PID, FD: d.FD, Target: filepath.Clean(d.Target)}] = true
+ }
+ for _, d := range held {
+ if !allowed[d] {
+ return WritersEvidence{}, ErrForeignHolder
+ }
+ }
+ if len(held) > 0 {
+ start, err := processStart(root, in.ReplicatorPID)
+ if err != nil {
+ return WritersEvidence{}, errors.Join(ErrIncomplete, err)
+ }
+ if start != in.ReplicatorStart {
+ return WritersEvidence{}, ErrForeignHolder
+ }
+ }
+ if err := ctx.Err(); err != nil {
+ return WritersEvidence{}, err
+ }
+ return WritersEvidence{Host: in.Host, CheckedAt: time.Now().UTC(), CheckedPaths: paths, Held: held, ProcRoot: root}, nil
+}
+
+func dbPaths(dbs []string) ([]string, error) {
+ seen := map[string]bool{}
+ for _, db := range dbs {
+ if !filepath.IsAbs(db) {
+ return nil, ErrInvalidInput
+ }
+ db = filepath.Clean(db)
+ seen[db], seen[db+"-wal"], seen[db+"-shm"] = true, true, true
+ }
+ paths := make([]string, 0, len(seen))
+ for path := range seen {
+ paths = append(paths, path)
+ }
+ sort.Strings(paths)
+ return paths, nil
+}
+```
+
+`internal/stopevidence/writers_linux.go`:
+
+```go
+//go:build linux
+
+package stopevidence
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "path/filepath"
+ "sort"
+ "strconv"
+ "strings"
+)
+
+// scanDescriptors lists every descriptor under root that resolves to one
+// of paths, deduplicated by process and descriptor number. A descriptor
+// seen only in a task that does not share the leader's table (an unshared
+// thread) is reported against the leader's PID with FD -1, so no
+// allow-list can exempt it.
+func scanDescriptors(ctx context.Context, root string, paths []string) ([]Descriptor, error) {
+ ops := realFDOps()
+ if !filepath.IsAbs(root) {
+ return nil, ErrInvalidInput
+ }
+ wanted := make(map[string]bool, len(paths))
+ for _, p := range paths {
+ wanted[filepath.Clean(p)] = true
+ }
+ entries, err := ops.readDir(root)
+ if err != nil {
+ return nil, fmt.Errorf("%w: %w", ErrIncomplete, err)
+ }
+ seen := map[Descriptor]bool{}
+ var out []Descriptor
+ for _, entry := range entries {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ pid, err := strconv.Atoi(entry.Name())
+ if err != nil || pid <= 0 {
+ continue
+ }
+ pidDir := filepath.Join(root, entry.Name())
+ tasks, err := readProcDir(filepath.Join(pidDir, "task"), pidDir, ops)
+ if err != nil {
+ return nil, fmt.Errorf("%w: pid %d: %w", ErrIncomplete, pid, err)
+ }
+ // The leader's table first: a thread's descriptor is the leader's
+ // only when the leader shows the same number for the same file.
+ sort.SliceStable(tasks, func(i, j int) bool { return tasks[i].Name() == entry.Name() && tasks[j].Name() != entry.Name() })
+ leader := map[int]string{}
+ for _, task := range tasks {
+ tid, err := strconv.Atoi(task.Name())
+ if err != nil || tid <= 0 {
+ continue
+ }
+ taskDir := filepath.Join(pidDir, "task", task.Name())
+ fdDir := filepath.Join(taskDir, "fd")
+ fds, err := readProcDir(fdDir, taskDir, ops)
+ if err != nil {
+ return nil, fmt.Errorf("%w: pid %d tid %d: %w", ErrIncomplete, pid, tid, err)
+ }
+ for _, fd := range fds {
+ n, err := strconv.Atoi(fd.Name())
+ if err != nil {
+ continue
+ }
+ target, err := ops.readlink(filepath.Join(fdDir, fd.Name()))
+ if procGone(err) {
+ continue
+ }
+ if err != nil {
+ return nil, fmt.Errorf("%w: pid %d tid %d fd %d: %w", ErrIncomplete, pid, tid, n, err)
+ }
+ clean := filepath.Clean(strings.TrimSuffix(target, " (deleted)"))
+ if !filepath.IsAbs(target) || !wanted[clean] {
+ continue
+ }
+ d := Descriptor{PID: pid, FD: n, Target: clean}
+ if tid == pid {
+ leader[n] = clean
+ } else if leader[n] != clean {
+ d.FD = -1 // not provably the leader's shared table
+ }
+ if !seen[d] {
+ seen[d] = true
+ out = append(out, d)
+ }
+ }
+ }
+ }
+ sort.Slice(out, func(i, j int) bool {
+ if out[i].PID != out[j].PID {
+ return out[i].PID < out[j].PID
+ }
+ return out[i].FD < out[j].FD
+ })
+ return out, nil
+}
+
+// processStart is field 22 of /proc/<pid>/stat: the process start time in
+// clock ticks since boot, which with the boot ID names one process.
+func processStart(root string, pid int) (string, error) {
+ raw, err := os.ReadFile(filepath.Join(root, strconv.Itoa(pid), "stat"))
+ if err != nil {
+ return "", err
+ }
+ s := string(raw)
+ end := strings.LastIndexByte(s, ')')
+ if end < 0 {
+ return "", ErrIncomplete
+ }
+ fields := strings.Fields(s[end+1:])
+ if len(fields) < 20 { // fields 3.. start after the comm
+ return "", ErrIncomplete
+ }
+ return fields[19], nil
+}
+```
+
+`internal/stopevidence/writers_other.go`:
+
+```go
+//go:build !linux
+
+package stopevidence
+
+import "context"
+
+func scanDescriptors(context.Context, string, []string) ([]Descriptor, error) {
+ return nil, ErrUnsupported
+}
+
+func processStart(string, int) (string, error) { return "", ErrUnsupported }
+```
+
+- [ ] **Step 4: Run the tests and the cross-builds**
+
+Run: `nice -n 19 go test ./internal/stopevidence/ -count=1 && GOOS=darwin go vet ./internal/stopevidence/ && GOOS=windows go vet ./internal/stopevidence/`
+Expected: PASS; both vets clean.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/stopevidence/writers.go internal/stopevidence/writers_linux.go internal/stopevidence/writers_other.go internal/stopevidence/writers_linux_test.go
+git commit -m "stopevidence: writers-stopped allows only the replicator's identified descriptors, so a release can flush before full stop
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 5: The regional worker (`internal/hostrelocate`)
+
+Spec §5.8 "Who acts". `Worker.Start` checks that this box is an active, unfenced worker of the region; that the claim serves on the source; the source's fence (a failover needs one, a drain refuses one); `Directory.CheckBoxStart` for the starter, the target and a live source (active, unfenced, no executable update reserved: H14); the box-started census (L5, L20); and the app admission registration; then `StartRelocation`. `Worker.Step` makes at most one worker CAS from the claim alone: a fenced source converts the operation (`FenceSource`), a fenced target is replaced (`ReplaceTarget`, or `MarkStuck` once two are used), then the phase's own step: the barrier, the pin after a release or after the failover window, naming and installing the target, recording `installed` once `Prepared` verifies P, serving after a passing probe, `probe-failed` past the deadline. `StorePinner` selects the database plan with `genselect.Select` and the files pin (the newest manifest in the admitted chain, independently of the database baseline, B2-4) and writes the pin object create-only; a retry reuses its own object.
+
+**Files:**
+- Create: `internal/hostrelocate/worker.go`, `internal/hostrelocate/pin.go`
+- Test: `internal/hostrelocate/worker_test.go`, `internal/hostrelocate/pin_test.go`
+
+**Interfaces:**
+- Consumes: Task 1; `hostactors.Directory` (`Read`, `TerminationFence`, `CheckBoxStart`), `hostactors.ValidateTerminationFence`, `hostfence.FenceKey`, `hostfence.ClaimFenceAuthority` (tests), `genselect.Request`, `genselect.Plan`, `hostfiles.ManifestKey`, `hostfiles.Manifest`, `release.CreateExact`.
+- Produces: `hostrelocate.Worker{Claims, Directory, Fences, Releases, Admission, Census, Choose, Pins, Prepared, Discharge, Notify, Self, Region, Controller, NewID, Now}` with `Start(ctx, host, Request) (hostclaim.Observed, error)` and `Step(ctx, host) (hostclaim.Observed, error)`; `Request{Reason, Source, Target}`; interfaces `Admission`, `Pinner`, `Prepared`; `Owner`; `ProbeDeadline`; errors `ErrUnsupported`, `ErrNotWorker`, `ErrNotYet`, `ErrRefused`; `StorePinner{Claims, Data, SelectDB, Bucket, Region}`, `PinObject`, `FilesPin`, `PinKey(host, op, n)`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostrelocate/worker_test.go`:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "crypto/ed25519"
+ "crypto/rand"
+ "encoding/json"
+ "errors"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfence"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/termfence"
+)
+
+const host = "shop.bab.oncarlos.com"
+
+var boxes = map[string]string{"wrk": "a", "src": "b", "dst": "c", "spa": "d", "sp2": "e", "sp3": "f"}
+
+func enrollment(box string) string { return strings.Repeat(boxes[box], 32) }
+
+type fixture struct {
+ t *testing.T
+ s release.Store
+ w Worker
+ now time.Time
+ notified int
+ sign release.Signer
+ roster hostactors.Roster
+}
+
+type okReleases struct{}
+
+func (okReleases) VerifyRelease(context.Context, hostclaim.Claim, hostclaim.Release) error {
+ return nil
+}
+
+type admission struct{ registered []string }
+
+func (a *admission) Register(_ context.Context, _ hostclaim.Observed, o Owner, id string) (string, error) {
+ if o.Box != "wrk" || o.Enrollment != enrollment("wrk") {
+ return "", errors.New("wrong owner")
+ }
+ a.registered = append(a.registered, id)
+ return "adm-" + id, nil
+}
+func (a *admission) Recheck(context.Context, string) error { return nil }
+
+// built reports P built once the target has said so (the participant's
+// half), as moves' install routine would leave it.
+type built struct{}
+
+func (built) Verify(_ context.Context, o hostclaim.Observed) (hostclaim.PreparedAuthority, bool, error) {
+ c := o.Claim
+ if c.Op.Journal["test.built"] != c.Op.Journal["prepared"] {
+ return hostclaim.PreparedAuthority{}, false, nil
+ }
+ var pin hostclaim.RecoveryPin
+ _ = json.Unmarshal([]byte(c.Op.Journal["pin"]), &pin)
+ return hostclaim.PreparedAuthority{Generation: c.Op.Journal["prepared"], InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: pin.MaxTXID}, true, nil
+}
+
+type pins struct{}
+
+func (pins) Pin(_ context.Context, o hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) {
+ var r hostclaim.Release
+ _ = json.Unmarshal([]byte(o.Claim.Op.Journal["release"]), &r)
+ return hostclaim.RecoveryPin{Key: "control/relocations/pin.json", SHA256: strings.Repeat("a", 64), Generation: o.Claim.Holder.Generation,
+ MaxTXID: max(r.FinalTXID, 40), FilesManifest: "m", FilesSeq: max(r.ManifestSeq, 7), PinnedAt: now}, nil
+}
+
+func newFixture(t *testing.T) *fixture {
+ t.Helper()
+ ctx := context.Background()
+ f := &fixture{t: t, s: release.NewDirStore(t.TempDir()), now: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)}
+ pub, key, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.sign = release.RootSigner(key)
+ f.roster = hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}},
+ Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}}
+ for ref := range boxes {
+ f.roster.Boxes[ref] = hostactors.Box{Enrollment: enrollment(ref), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default",
+ Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"}
+ }
+ f.writeRoster()
+ d := hostactors.Directory{Store: f.s, PublicKey: pub, Bucket: "bucket"}
+ clock := func() time.Time { return f.now }
+ n := 0
+ f.w = Worker{Claims: f.s, Directory: d, Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: clock}, Releases: okReleases{}, Admission: &admission{},
+ Census: func(context.Context, string, []string) error { return nil },
+ Choose: func(_ context.Context, _ hostclaim.Observed, exclude []string) (hostclaim.Target, error) {
+ for _, b := range []string{"dst", "spa", "sp2", "sp3"} {
+ if !slices.Contains(exclude, b) {
+ return hostclaim.Target{Box: b, Enrollment: enrollment(b)}, nil
+ }
+ }
+ return hostclaim.Target{}, errors.New("no eligible box")
+ },
+ Pins: pins{},
+ Prepared: built{},
+ Discharge: func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge { return nil },
+ Notify: func(context.Context) { f.notified++ },
+ Self: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment("wrk")}, Region: "ie", Controller: "ctl",
+ NewID: func() string { n++; return "id" + string(rune('0'+n)) }, Now: clock}
+ // A host serving on src, whose source admitted one cold session.
+ created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC)
+ nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created}
+ body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`)
+ o, err := hostclaim.Read(ctx, f.s, host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o = f.step(o, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: enrollment("src")}))
+ o = f.step(o, hostclaim.CompleteCreate("c1", "src"))
+ f.step(o, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true}))
+ return f
+}
+
+func (f *fixture) writeRoster() {
+ f.t.Helper()
+ raw, err := release.SignRootEnvelope(f.sign, f.roster)
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ _, etag, err := f.s.Get(context.Background(), hostactors.Key)
+ if err != nil {
+ etag = ""
+ }
+ if _, err := f.s.Put(context.Background(), hostactors.Key, raw, etag); err != nil {
+ f.t.Fatal(err)
+ }
+}
+
+func (f *fixture) step(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed {
+ f.t.Helper()
+ next, err := hostclaim.Update(context.Background(), f.s, o, tr)
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ return next
+}
+
+func (f *fixture) claim() hostclaim.Observed {
+ f.t.Helper()
+ o, err := hostclaim.Read(context.Background(), f.s, host)
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ return o
+}
+
+// participant applies a box-side transition (Plan 6b's half) to the claim.
+func (f *fixture) participant(tr func(*hostclaim.Claim) error) {
+ f.t.Helper()
+ f.step(f.claim(), tr)
+}
+
+func (f *fixture) ack(box string) {
+ f.participant(hostclaim.AcknowledgeExecution(box, hostclaim.ExecutionAcknowledgement{Op: f.claim().Claim.Op.ID, Enrollment: enrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: f.now.Add(time.Hour)}))
+}
+
+// terminate writes box's termination fence as hostfence.TerminationWorker
+// would, observed a minute ago by wrk.
+func (f *fixture) terminate(box string) {
+ f.t.Helper()
+ b := f.roster.Boxes[box]
+ rec := termfence.Record{V: 1, Box: box, Enrollment: b.Enrollment, Cloud: b.Cloud, State: "terminated", ObservedAt: f.now.Add(-time.Minute).Format(time.RFC3339Nano),
+ Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment("wrk")}}
+ raw, _ := json.Marshal(rec)
+ if _, err := f.s.Put(context.Background(), hostfence.FenceKey(b.Enrollment), raw, ""); err != nil {
+ f.t.Fatal(err)
+ }
+}
+
+func (f *fixture) worker(want error) hostclaim.Observed {
+ f.t.Helper()
+ o, err := f.w.Step(context.Background(), host)
+ if !errors.Is(err, want) {
+ f.t.Fatalf("step: %v, want %v", err, want)
+ }
+ return o
+}
+
+func (f *fixture) installAndProbe(target string) {
+ f.t.Helper()
+ c := f.claim().Claim
+ id, p := c.Op.ID, c.Op.Journal["prepared"]
+ f.worker(ErrNotYet) // P is not built yet
+ f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = p; return nil })
+ if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseInstalled {
+ f.t.Fatalf("the worker records installed once P verifies: %s", o.Claim.Op.Phase)
+ }
+ stamp := f.claim().Claim.Op.Journal["install_stamp"]
+ var n uint64
+ _ = json.Unmarshal([]byte(stamp), &n)
+ f.participant(hostclaim.RecordLocalInstall(id, hostclaim.LocalInstall{Box: target, Enrollment: enrollment(target), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("d", 64)}))
+ f.participant(hostclaim.ToProbing(id, f.now))
+ o := f.claim()
+ f.participant(hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: id, ObservedPhase: hostclaim.PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation,
+ Box: target, Mode: "cold", NewGeneration: p, KeepStamp: true, Region: "ie", PredecessorRegion: "ie"}))
+}
+
+func TestWorkerDrivesADrain(t *testing.T) {
+ f := newFixture(t)
+ o, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if o.Claim.Op.Intent["admission_id"] == "" || o.Claim.Op.Intent["directory_revision"] != "1" {
+ t.Fatalf("intent: %+v", o.Claim.Op.Intent)
+ }
+ f.worker(ErrNotYet) // no acknowledgements yet
+ f.ack("src")
+ f.ack("dst")
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseReleasing {
+ t.Fatalf("phase %s", o.Claim.Op.Phase)
+ }
+ f.worker(ErrNotYet) // the source has not released
+ h := f.claim().Claim.Holder
+ f.participant(hostclaim.JournalRelease(o.Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning || o.Claim.Op.Exec != "none" {
+ t.Fatalf("pinning: %+v", o.Claim.Op)
+ }
+ if o = f.worker(nil); o.Claim.Op.Journal["pin"] == "" {
+ t.Fatal("no pin journaled")
+ }
+ before := f.notified
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseInstalling || o.Claim.Op.Exec != "dst" || f.notified != before+1 {
+ t.Fatalf("installing: %+v notified %d", o.Claim.Op, f.notified-before)
+ }
+ f.installAndProbe("dst")
+ f.worker(ErrNotYet) // probing, no probe result, within the deadline
+ o = f.claim()
+ f.participant(hostclaim.RecordProbe(o.Claim.Op.ID, hostclaim.Probe{Box: "dst", Session: o.Claim.Holder.Session}))
+ if o = f.worker(nil); o.Claim.Op.Kind != hostclaim.KindServing || o.Claim.Op.Exec != "dst" {
+ t.Fatalf("serving: %+v", o.Claim.Op)
+ }
+}
+
+func TestWorkerFailoverWaitsForThePinWindow(t *testing.T) {
+ f := newFixture(t)
+ ctx := context.Background()
+ if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); !errors.Is(err, ErrRefused) {
+ t.Fatalf("failover with no fence: %v", err)
+ }
+ f.terminate("src")
+ if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, ErrRefused) {
+ t.Fatalf("drain of a fenced source: %v", err)
+ }
+ o, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if o.Claim.Op.Exec != "none" || !slices.Contains(o.Claim.Lost, "src") {
+ t.Fatalf("failover start: %+v", o.Claim.Op)
+ }
+ f.ack("dst")
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning {
+ t.Fatalf("phase %s", o.Claim.Op.Phase)
+ }
+ // The fence was observed a minute ago, so the 30-second window is open.
+ if o = f.worker(nil); o.Claim.Op.Journal["pin"] == "" {
+ t.Fatal("not pinned")
+ }
+}
+
+func TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence(t *testing.T) {
+ f := newFixture(t)
+ ctx := context.Background()
+ f.terminate("src")
+ f.now = f.now.Add(-time.Minute + 10*time.Second) // ten seconds after the observation
+ if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ f.ack("dst")
+ f.worker(nil) // pinning
+ f.worker(ErrNotYet)
+ f.now = f.now.Add(20 * time.Second)
+ if o := f.worker(nil); o.Claim.Op.Journal["pin"] == "" {
+ t.Fatal("not pinned at observation plus thirty seconds")
+ }
+}
+
+func TestWorkerConvertsADrainWhoseSourceIsTerminated(t *testing.T) {
+ f := newFixture(t)
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ f.ack("dst")
+ f.terminate("src")
+ o := f.worker(nil)
+ if o.Claim.Op.Phase != hostclaim.PhaseStarting || o.Claim.Op.Exec != "none" || slices.Contains(o.Claim.Op.Participants, "src") || !slices.Contains(o.Claim.Lost, "src") {
+ t.Fatalf("conversion: %+v lost %v", o.Claim.Op, o.Claim.Lost)
+ }
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning {
+ t.Fatalf("the barrier now needs only the target: %s", o.Claim.Op.Phase)
+ }
+}
+
+func TestWorkerReplacesATerminatedTargetTwiceThenStops(t *testing.T) {
+ f := newFixture(t)
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ for _, tc := range []struct{ dead, next string }{{"dst", "spa"}, {"spa", "sp2"}} {
+ f.terminate(tc.dead)
+ o := f.worker(nil)
+ if o.Claim.Op.Intent["target_box"] != tc.next || !slices.Contains(o.Claim.Lost, tc.dead) {
+ t.Fatalf("replacement of %s: %+v", tc.dead, o.Claim.Op.Intent)
+ }
+ }
+ f.terminate("sp2")
+ o := f.worker(nil)
+ if o.Claim.Op.Journal["stuck"] == "" {
+ t.Fatalf("a third replacement must stop the operation: %+v", o.Claim.Op.Journal)
+ }
+ before := o.Claim.Gen
+ if o = f.worker(nil); o.Claim.Gen != before {
+ t.Fatal("a worker moved a stuck operation")
+ }
+}
+
+func TestWorkerFailsAProbePastItsDeadline(t *testing.T) {
+ f := newFixture(t)
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ f.ack("src")
+ f.ack("dst")
+ f.worker(nil)
+ h := f.claim().Claim.Holder
+ f.participant(hostclaim.JournalRelease(f.claim().Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}))
+ f.worker(nil)
+ f.worker(nil)
+ f.worker(nil)
+ f.installAndProbe("dst")
+ f.now = f.now.Add(ProbeDeadline + time.Second)
+ if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseProbeFailed {
+ t.Fatalf("phase %s", o.Claim.Op.Phase)
+ }
+ f.worker(ErrNotYet) // the failed target has not released
+}
+
+func TestWorkerRefusesWhenItIsNotAnActiveWorker(t *testing.T) {
+ f := newFixture(t)
+ b := f.roster.Boxes["wrk"]
+ b.Region = "uk"
+ f.roster.Boxes["wrk"] = b
+ f.roster.Revision++
+ f.writeRoster()
+ before := f.claim().ETag
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, ErrNotWorker) {
+ t.Fatalf("start: %v", err)
+ }
+ if f.claim().ETag != before {
+ t.Fatal("a refused start wrote the claim")
+ }
+}
+
+func TestWorkerStartRefusesAnUpdateReservedForAParticipant(t *testing.T) {
+ f := newFixture(t)
+ f.roster.Activations["upd-1"] = hostactors.Reservation{ID: "upd-1", FloorSHA256: strings.Repeat("f", 64),
+ Actors: map[string]string{"box:dst": enrollment("dst")}, Releases: map[string]hostactors.Report{"box:dst": {Enrollment: enrollment("dst")}}}
+ f.roster.Revision++
+ f.writeRoster()
+ before := f.claim().ETag
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err == nil {
+ t.Fatal("started while an executable update is reserved for the target (H14)")
+ }
+ if f.claim().ETag != before {
+ t.Fatal("a refused start wrote the claim")
+ }
+}
+
+func TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence(t *testing.T) {
+ f := newFixture(t)
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ f.ack("src")
+ f.ack("dst")
+ if _, err := f.s.Put(context.Background(), hostfence.FenceKey(enrollment("src")), []byte("{"), ""); err != nil {
+ t.Fatal(err)
+ }
+ before := f.claim().ETag
+ if _, err := f.w.Step(context.Background(), host); err == nil || errors.Is(err, ErrNotYet) {
+ t.Fatalf("an unreadable fence let the drain proceed: %v", err)
+ }
+ if f.claim().ETag != before {
+ t.Fatal("the claim moved")
+ }
+}
+```
+
+`internal/hostrelocate/pin_test.go`:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func pinClaim() hostclaim.Observed {
+ return hostclaim.Observed{Host: host, Claim: &hostclaim.Claim{Host: host, Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhasePinning},
+ Holder: &hostclaim.Holder{Location: "instances/" + host + "/inc", Generation: "g3",
+ Admissions: []hostclaim.AdmissionRecord{{Generation: "g2", Predecessor: "g1"}, {Generation: "g3", Predecessor: "g2"}}}}}
+}
+
+func TestStorePinnerPinsTheNewestFilesManifestInTheChain(t *testing.T) {
+ ctx := context.Background()
+ claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
+ // g3 (holder.generation) never published files; g2 did, then g1 earlier.
+ for gen, seq := range map[string]int64{"g2": 5, "g1": 9} {
+ raw, _ := json.Marshal(hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}})
+ if _, err := data.Put(ctx, hostfiles.ManifestKey("instances/"+host+"/inc/files/"+gen+"/"), raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ selects := 0
+ p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie",
+ SelectDB: func(_ context.Context, req genselect.Request) (*genselect.Plan, error) {
+ selects++
+ return &genselect.Plan{Generation: req.Holder.Generation, MaxTXID: 41}, nil
+ }}
+ at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)
+ pin, err := p.Pin(ctx, pinClaim(), at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if pin.FilesManifest != hostfiles.ManifestKey("instances/"+host+"/inc/files/g2/") || pin.FilesSeq != 5 || pin.MaxTXID != 41 || pin.Key != PinKey(host, "rel-1", 1) {
+ t.Fatalf("pin: %+v", pin)
+ }
+ // A retry (the journal CAS was lost) reuses the object unchanged.
+ again, err := p.Pin(ctx, pinClaim(), at.Add(time.Minute))
+ if err != nil || again != pin || selects != 1 {
+ t.Fatalf("replay: %+v %v selects=%d", again, err, selects)
+ }
+ // An object at the key that is not this operation's is refused.
+ if _, err := claims.Put(ctx, PinKey(host, "rel-2", 1), []byte(`{"v":1,"host":"other","op":"rel-2","db":{}}`), ""); err != nil {
+ t.Fatal(err)
+ }
+ other := pinClaim()
+ other.Claim.Op.ID = "rel-2"
+ if _, err := p.Pin(ctx, other, at); !errors.Is(err, ErrRefused) {
+ t.Fatalf("foreign pin object: %v", err)
+ }
+}
+
+func TestStorePinnerRefusesAnUnreadableManifest(t *testing.T) {
+ ctx := context.Background()
+ claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
+ if _, err := data.Put(ctx, hostfiles.ManifestKey("instances/"+host+"/inc/files/g3/"), []byte(`{"v":1}`), ""); err != nil {
+ t.Fatal(err)
+ }
+ p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie",
+ SelectDB: func(context.Context, genselect.Request) (*genselect.Plan, error) { return &genselect.Plan{}, nil }}
+ if _, err := p.Pin(ctx, pinClaim(), time.Now()); !errors.Is(err, ErrRefused) {
+ t.Fatalf("a version 1 manifest was pinned: %v", err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1`
+Expected: FAIL: `no non-test Go files` / `undefined: Worker`.
+
+- [ ] **Step 3: The worker**
+
+`internal/hostrelocate/worker.go`:
+
+```go
+// Package hostrelocate is the regional worker's half of relocation (region
+// HA spec 1 §5.8, claim §4 L1): starting a relocation, and replaying every
+// worker CAS from the claim alone. Any active enrolled box of the region
+// may act (claim P6-D5); correctness never depends on which, because every
+// step is a claim CAS conditional on the operation, its phase and the
+// claim's ETag. The participants' halves (acknowledgement, release,
+// install, probe) are not here: they run on the boxes, under the host
+// guard, through Plan 6b's box assembly.
+//
+// This package only READS termination fences. hostfence.TerminationWorker
+// writes them (control/box-fences/<enrollment>.json).
+package hostrelocate
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "slices"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfence"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/termfence"
+)
+
+// ProbeDeadline is §5.8's probe deadline: a live target whose probe has not
+// passed this long after entering probing goes to probe-failed.
+const ProbeDeadline = 10 * time.Minute
+
+var (
+ ErrUnsupported = errors.New("hostrelocate: a required provider is missing")
+ ErrNotWorker = errors.New("hostrelocate: this box is not an active, unfenced worker of the region")
+ ErrNotYet = errors.New("hostrelocate: waiting for a participant or a time bound")
+ ErrRefused = errors.New("hostrelocate: relocation refused")
+)
+
+// Request asks for one relocation. Target is required; Source must be the
+// claim's executor. Reason is drain, rebalance or failover.
+type Request struct {
+ Reason, Source, Target string
+}
+
+// Owner is the app-admission registration owner of a box-started
+// operation (claim §5.1, "A regional worker starting relocating").
+type Owner struct {
+ Box, Enrollment, Controller string
+}
+
+// Admission is claim §5.1's app-admission registration for a box-started
+// relocation: Register binds the saved claim observation and holder
+// incarnation while admission is open; Recheck is the final check before
+// the single start CAS. Claim Plan 5's app gate implements it.
+type Admission interface {
+ Register(context.Context, hostclaim.Observed, Owner, string) (string, error)
+ Recheck(context.Context, string) error
+}
+
+// Pinner selects, writes and returns one pinned recovery set (§5.8).
+type Pinner interface {
+ Pin(context.Context, hostclaim.Observed, time.Time) (hostclaim.RecoveryPin, error)
+}
+
+// Prepared verifies the remote build of the operation's prepared
+// generation P (§5.8 "Building the prepared generation"): done is true only
+// when a fresh read verifies P's snapshot, level-0 anchor and completed
+// install manifest (or, for a host with nothing replicated, P's files
+// manifest alone). Moves' shared install routine (L16) writes them.
+type Prepared interface {
+ Verify(context.Context, hostclaim.Observed) (a hostclaim.PreparedAuthority, done bool, err error)
+}
+
+// Worker drives relocation for one region. Every field is required.
+type Worker struct {
+ Claims release.Store // the deployment (control) bucket
+ Directory hostactors.Directory
+ Fences hostclaim.FenceAuthority // hostfence.ClaimFenceAuthority
+ Releases hostclaim.ReleaseAuthority // Plan 6b's stop receipt reader
+ Admission Admission
+ // Census is the box-started start census (claim D7 with L5 and L20):
+ // hostactors.Directory.CensusBoxRelocation with the deployment's
+ // evidence and floor.
+ Census func(ctx context.Context, starter string, participants []string) error
+ // Choose names a target in the operation's region, never one in
+ // exclude (hostops.ExecutorChooser behind an adapter).
+ Choose func(ctx context.Context, obs hostclaim.Observed, exclude []string) (hostclaim.Target, error)
+ Pins Pinner
+ Prepared Prepared
+ // Discharge judges a fenced box's volume records (boxvolumes.Discharge).
+ // It never gates anything: nil, or an unqualified verdict, leaves the
+ // box's park-local chores open for an operator (H19).
+ Discharge func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge
+ // Notify bumps the control serial after an executor CAS (§5.3,
+ // "Prompt changes"). A failed bump costs cadence, never correctness.
+ Notify func(context.Context)
+ Self termfence.WrittenBy
+ Region string
+ Controller string
+ NewID func() string
+ Now func() time.Time
+}
+
+func (w Worker) ready() error {
+ if w.Claims == nil || w.Directory.Store == nil || w.Fences == nil || w.Releases == nil || w.Admission == nil || w.Census == nil ||
+ w.Choose == nil || w.Pins == nil || w.Prepared == nil || w.Discharge == nil || w.Notify == nil || w.Self.Box == "" || w.Self.Enrollment == "" ||
+ w.Region == "" || w.Controller == "" || w.NewID == nil || w.Now == nil {
+ return ErrUnsupported
+ }
+ return nil
+}
+
+// self checks, on every call, that this box is an active, unfenced worker
+// enrolled in the region (§5.8 "Each worker first checks its own roster
+// entry is active in this region").
+func (w Worker) self(ctx context.Context) (hostactors.Observed, error) {
+ o, err := w.Directory.Read(ctx)
+ if err != nil {
+ return hostactors.Observed{}, err
+ }
+ b, ok := o.Roster.Boxes[w.Self.Box]
+ if !ok || b.State != "active" || b.Enrollment != w.Self.Enrollment || b.Region != w.Region {
+ return hostactors.Observed{}, ErrNotWorker
+ }
+ fenced, err := w.Directory.TerminationFence(ctx, o, w.Self.Box, w.Now())
+ if err != nil || fenced {
+ return hostactors.Observed{}, errors.Join(ErrNotWorker, err)
+ }
+ return o, nil
+}
+
+// fence reads box's termination fence, if any, as the exact bytes a claim
+// CAS will cite. Absence is never evidence: no fence is (nil, nil); an
+// unreadable or invalid one is an error, never "not fenced".
+func (w Worker) fence(ctx context.Context, o hostactors.Observed, box string) (*hostclaim.FenceRef, termfence.Record, error) {
+ b, ok := o.Roster.Boxes[box]
+ if !ok {
+ return nil, termfence.Record{}, fmt.Errorf("%w: %s is not in the roster", ErrRefused, box)
+ }
+ key := hostfence.FenceKey(b.Enrollment)
+ raw, _, err := w.Directory.Store.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ return nil, termfence.Record{}, nil
+ }
+ if err != nil {
+ return nil, termfence.Record{}, err
+ }
+ var f termfence.Record
+ if err := json.Unmarshal(raw, &f); err != nil {
+ return nil, termfence.Record{}, err
+ }
+ if err := hostactors.ValidateTerminationFence(o, box, f, w.Now()); err != nil {
+ return nil, termfence.Record{}, err
+ }
+ sum := sha256.Sum256(raw)
+ return &hostclaim.FenceRef{Enrollment: b.Enrollment, Key: key, SHA256: hex.EncodeToString(sum[:])}, f, nil
+}
+
+// Start writes the start CAS for req (§5.8 "The start CAS", §5.10). It
+// refuses unless the claim serves on req.Source; the roster shows the
+// starter, the target and (for a drain) the source active, unfenced and
+// with no executable update reserved (H14); the census passes; and the
+// app admits the registration. A failover requires the source's fence.
+func (w Worker) Start(ctx context.Context, host string, req Request) (hostclaim.Observed, error) {
+ if err := w.ready(); err != nil {
+ return hostclaim.Observed{}, err
+ }
+ o, err := w.self(ctx)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ obs, err := hostclaim.Read(ctx, w.Claims, host)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ if obs.Absent || obs.Claim.Op.Kind != hostclaim.KindServing || obs.Claim.Op.Exec != req.Source || req.Target == req.Source {
+ return hostclaim.Observed{}, fmt.Errorf("%w: %s does not serve on %s", ErrRefused, host, req.Source)
+ }
+ source, sok := o.Roster.Boxes[req.Source]
+ target, tok := o.Roster.Boxes[req.Target]
+ if !sok || !tok || source.Region != w.Region || target.Region != w.Region {
+ return hostclaim.Observed{}, fmt.Errorf("%w: source and target must be enrolled in %s", ErrRefused, w.Region)
+ }
+ ref, record, err := w.fence(ctx, o, req.Source)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ refs := []string{w.Self.Box, req.Target}
+ switch req.Reason {
+ case hostclaim.ReasonFailover:
+ if ref == nil {
+ return hostclaim.Observed{}, fmt.Errorf("%w: a failover needs the source's termination fence", ErrRefused)
+ }
+ case hostclaim.ReasonDrain, hostclaim.ReasonRebalance:
+ if ref != nil {
+ return hostclaim.Observed{}, fmt.Errorf("%w: the source is fenced; start a failover", ErrRefused)
+ }
+ refs = append(refs, req.Source)
+ default:
+ return hostclaim.Observed{}, fmt.Errorf("%w: reason %q", ErrRefused, req.Reason)
+ }
+ refs = slices.Compact(slices.Sorted(slices.Values(refs)))
+ revision, err := w.Directory.CheckBoxStart(ctx, refs...)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ if err := w.Census(ctx, w.Self.Box, refs); err != nil {
+ return hostclaim.Observed{}, err
+ }
+ id := w.NewID()
+ admission, err := w.Admission.Register(ctx, obs, Owner{Box: w.Self.Box, Enrollment: w.Self.Enrollment, Controller: w.Controller}, id)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ if err := w.Admission.Recheck(ctx, admission); err != nil {
+ return hostclaim.Observed{}, err
+ }
+ in := hostclaim.RelocationStart{OpID: id, By: "box:" + w.Self.Box, Now: w.Now(), Region: w.Region, Reason: req.Reason,
+ SourceBox: req.Source, SourceEnrollment: source.Enrollment, TargetBox: req.Target, TargetEnrollment: target.Enrollment,
+ DirectoryRevision: revision, AdmissionID: admission, Fence: ref}
+ if ref != nil {
+ in.Discharge = w.Discharge(ctx, source, record)
+ }
+ next, err := hostclaim.StartRelocation(ctx, w.Claims, obs, in, w.Fences)
+ if err == nil && ref != nil {
+ w.Notify(ctx)
+ }
+ return next, err
+}
+
+// Step makes at most one worker CAS for host's relocation, the one its
+// phase and evidence call for, and returns the claim it left. ErrNotYet
+// means a participant or a time bound is awaited; nothing was written.
+// A host that is not relocating, or whose operation is stuck, is left
+// alone.
+func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, error) {
+ if err := w.ready(); err != nil {
+ return hostclaim.Observed{}, err
+ }
+ o, err := w.self(ctx)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ obs, err := hostclaim.Read(ctx, w.Claims, host)
+ if err != nil || obs.Absent {
+ return obs, err
+ }
+ c := obs.Claim
+ if c.Op.Kind != hostclaim.KindRelocating || c.Op.Intent["region"] != w.Region || c.Op.Journal["stuck"] != "" {
+ return obs, nil
+ }
+ id := c.Op.ID
+ // Evidence first: a fenced participant changes what every phase may do.
+ if src := c.Op.Intent["source_box"]; !slices.Contains(c.Lost, src) {
+ ref, record, err := w.fence(ctx, o, src)
+ if err != nil {
+ return obs, err
+ }
+ if ref != nil {
+ next, err := hostclaim.FenceSource(ctx, w.Claims, obs, id, *ref, w.Discharge(ctx, o.Roster.Boxes[src], record), w.Fences)
+ if err == nil {
+ w.Notify(ctx)
+ }
+ return next, err
+ }
+ }
+ if dst := c.Op.Intent["target_box"]; dst != "" && !slices.Contains(c.Lost, dst) {
+ ref, record, err := w.fence(ctx, o, dst)
+ if err != nil {
+ return obs, err
+ }
+ if ref != nil {
+ return w.replace(ctx, obs, *ref, w.Discharge(ctx, o.Roster.Boxes[dst], record))
+ }
+ }
+ switch c.Op.Phase {
+ case hostclaim.PhaseStarting:
+ if !acked(c) {
+ return obs, ErrNotYet
+ }
+ if c.Op.Exec == "none" {
+ return w.update(ctx, obs, hostclaim.PinFailover(id))
+ }
+ return w.update(ctx, obs, hostclaim.ToReleasing(id))
+ case hostclaim.PhaseReleasing:
+ if c.Op.Journal["release"] == "" {
+ return obs, ErrNotYet
+ }
+ return hostclaim.PinAfterRelease(ctx, w.Claims, obs, id, w.Releases, nil)
+ case hostclaim.PhaseProbeFailed:
+ if c.Op.Journal["release"] == "" {
+ return obs, ErrNotYet
+ }
+ next, err := w.next(ctx, obs)
+ if err != nil {
+ return w.stuckOn(ctx, obs, err)
+ }
+ return hostclaim.PinAfterRelease(ctx, w.Claims, obs, id, w.Releases, &next)
+ case hostclaim.PhasePinning:
+ return w.pinning(ctx, obs)
+ case hostclaim.PhaseProbing:
+ since, err := time.Parse(time.RFC3339Nano, c.Op.Journal["probing_since"])
+ if err != nil {
+ return obs, err
+ }
+ if c.Op.Journal["probe"] != "" {
+ next, err := w.update(ctx, obs, hostclaim.CompleteRelocation(id, w.NewID))
+ if err == nil {
+ w.Notify(ctx)
+ }
+ return next, err
+ }
+ if w.Now().Sub(since) > ProbeDeadline {
+ return w.update(ctx, obs, hostclaim.ToProbeFailed(id))
+ }
+ return obs, ErrNotYet
+ case hostclaim.PhaseInstalling:
+ a, done, err := w.Prepared.Verify(ctx, obs)
+ if err != nil || !done || !acked(c) {
+ return obs, errors.Join(ErrNotYet, err)
+ }
+ return w.update(ctx, obs, hostclaim.ToInstalled(id, a))
+ }
+ // installed is the target's: its local install and its probe (Plan 6b).
+ return obs, ErrNotYet
+}
+
+func (w Worker) pinning(ctx context.Context, obs hostclaim.Observed) (hostclaim.Observed, error) {
+ c := obs.Claim
+ id := c.Op.ID
+ if c.Op.Intent["target_box"] == "" {
+ next, err := w.next(ctx, obs)
+ if err != nil {
+ return obs, err
+ }
+ return w.update(ctx, obs, hostclaim.NameTarget(id, next))
+ }
+ if c.Op.Journal["pin"] == "" {
+ for key, at := range c.Op.Journal {
+ if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" {
+ t, err := time.Parse(time.RFC3339Nano, at)
+ if err != nil {
+ return obs, err
+ }
+ if w.Now().Before(t.Add(hostclaim.FailoverPinDelay)) {
+ return obs, ErrNotYet
+ }
+ }
+ }
+ pin, err := w.Pins.Pin(ctx, obs, w.Now())
+ if err != nil {
+ return obs, err
+ }
+ return w.update(ctx, obs, hostclaim.JournalPin(id, pin))
+ }
+ t := hostclaim.Target{Box: c.Op.Intent["target_box"], Enrollment: c.Op.Intent["target_enrollment"]}
+ if !acked(c) {
+ return obs, ErrNotYet
+ }
+ next, err := w.update(ctx, obs, hostclaim.ToInstalling(id, t, w.NewID()))
+ if err == nil {
+ w.Notify(ctx)
+ }
+ return next, err
+}
+
+// next names the operation's next target: never the source, a lost box,
+// or a box this operation already replaced.
+func (w Worker) next(ctx context.Context, obs hostclaim.Observed) (hostclaim.Target, error) {
+ c := obs.Claim
+ exclude := append([]string{c.Op.Intent["source_box"]}, c.Lost...)
+ if t := c.Op.Intent["target_box"]; t != "" {
+ exclude = append(exclude, t)
+ }
+ var replaced []string
+ if raw := c.Op.Journal["excluded_targets"]; raw != "" {
+ if err := json.Unmarshal([]byte(raw), &replaced); err != nil {
+ return hostclaim.Target{}, err
+ }
+ }
+ return w.Choose(ctx, obs, append(exclude, replaced...))
+}
+
+func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostclaim.FenceRef, d *hostclaim.LocalDischarge) (hostclaim.Observed, error) {
+ next, err := w.next(ctx, obs)
+ if err != nil {
+ return w.stuckOn(ctx, obs, err)
+ }
+ out, err := hostclaim.ReplaceTarget(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, next, w.NewID(), w.Fences)
+ if errors.Is(err, hostclaim.ErrReplacementsExceeded) {
+ return w.stuckOn(ctx, obs, err)
+ }
+ if err == nil {
+ w.Notify(ctx)
+ }
+ return out, err
+}
+
+// stuckOn marks the operation stuck when it has used its replacements;
+// any other error is returned for a later pass to retry.
+func (w Worker) stuckOn(ctx context.Context, obs hostclaim.Observed, cause error) (hostclaim.Observed, error) {
+ if !errors.Is(cause, hostclaim.ErrReplacementsExceeded) {
+ return obs, cause
+ }
+ return w.update(ctx, obs, hostclaim.MarkStuck(obs.Claim.Op.ID, cause.Error()))
+}
+
+func (w Worker) update(ctx context.Context, obs hostclaim.Observed, tr func(*hostclaim.Claim) error) (hostclaim.Observed, error) {
+ return hostclaim.Update(ctx, w.Claims, obs, tr)
+}
+
+// acked is the D6 barrier as the worker reads it before a CAS that needs it.
+func acked(c *hostclaim.Claim) bool {
+ for _, box := range c.Op.Participants {
+ if c.Op.Journal["exec_ack."+box] == "" {
+ return false
+ }
+ }
+ return true
+}
+```
+
+- [ ] **Step 4: The pin**
+
+`internal/hostrelocate/pin.go`:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "strconv"
+ "time"
+
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// PinObject is the immutable pinned recovery set (§5.8): the database
+// plan genselect selected, with every file's exact version, and the files
+// pin, the newest manifest in the admitted chain, by digest (spec §18: the
+// store has no version-id read, so content is identified by SHA-256).
+type PinObject struct {
+ V int `json:"v"`
+ Host string `json:"host"`
+ Op string `json:"op"`
+ N uint64 `json:"n"`
+ DB *genselect.Plan `json:"db"`
+ Files FilesPin `json:"files"`
+ PinnedAt time.Time `json:"pinned_at"`
+}
+
+// FilesPin names one generation's files manifest by prefix, sequence and
+// the digest of its exact bytes.
+type FilesPin struct {
+ Prefix string `json:"prefix"`
+ Seq int64 `json:"seq"`
+ SHA256 string `json:"sha256"`
+}
+
+// PinKey is where the operation's nth pin lives in the control bucket.
+func PinKey(host, op string, n uint64) string {
+ return "control/relocations/" + host + "/" + op + "/pin-" + strconv.FormatUint(n, 10) + ".json"
+}
+
+// StorePinner selects the pinned recovery set against the claim
+// observation it is given (§5.8 "The pinned recovery set") and writes it
+// create-only. A retry finds its own object and reuses it unchanged.
+type StorePinner struct {
+ Claims release.Store // control bucket: pin objects
+ Data release.Store // the region's data bucket: files manifests
+ SelectDB func(context.Context, genselect.Request) (*genselect.Plan, error)
+ Bucket string // the region's data bucket name, for genselect
+ Region string
+}
+
+func (p StorePinner) Pin(ctx context.Context, obs hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) {
+ if p.Claims == nil || p.Data == nil || p.SelectDB == nil || p.Bucket == "" || p.Region == "" || obs.Claim == nil || obs.Claim.Holder == nil {
+ return hostclaim.RecoveryPin{}, ErrUnsupported
+ }
+ c := obs.Claim
+ n := uint64(1)
+ if raw := c.Op.Journal["pins"]; raw != "" {
+ prev, err := strconv.ParseUint(raw, 10, 64)
+ if err != nil {
+ return hostclaim.RecoveryPin{}, err
+ }
+ n = prev + 1
+ }
+ key := PinKey(c.Host, c.Op.ID, n)
+ if raw, _, err := p.Claims.Get(ctx, key); err == nil {
+ return p.reuse(c, key, raw)
+ } else if !errors.Is(err, os.ErrNotExist) {
+ return hostclaim.RecoveryPin{}, err
+ }
+ plan, err := p.SelectDB(ctx, genselect.Request{Bucket: p.Bucket, Region: p.Region, Host: c.Host, Holder: *c.Holder})
+ if err != nil {
+ return hostclaim.RecoveryPin{}, err
+ }
+ files, err := p.filesPin(ctx, c.Holder)
+ if err != nil {
+ return hostclaim.RecoveryPin{}, err
+ }
+ obj := PinObject{V: 1, Host: c.Host, Op: c.Op.ID, N: n, DB: plan, Files: files, PinnedAt: now.UTC()}
+ raw, err := json.Marshal(obj)
+ if err != nil {
+ return hostclaim.RecoveryPin{}, err
+ }
+ if err := release.CreateExact(ctx, p.Claims, key, raw); err != nil {
+ if !errors.Is(err, release.ErrCASConflict) {
+ return hostclaim.RecoveryPin{}, err
+ }
+ got, _, gerr := p.Claims.Get(ctx, key) // another worker pinned first
+ if gerr != nil {
+ return hostclaim.RecoveryPin{}, gerr
+ }
+ return p.reuse(c, key, got)
+ }
+ return pinRef(key, raw, obj), nil
+}
+
+func (p StorePinner) reuse(c *hostclaim.Claim, key string, raw []byte) (hostclaim.RecoveryPin, error) {
+ var obj PinObject
+ if err := json.Unmarshal(raw, &obj); err != nil || obj.V != 1 || obj.Host != c.Host || obj.Op != c.Op.ID || obj.DB == nil {
+ return hostclaim.RecoveryPin{}, fmt.Errorf("%w: pin object %s is not this operation's", ErrRefused, key)
+ }
+ return pinRef(key, raw, obj), nil
+}
+
+func pinRef(key string, raw []byte, obj PinObject) hostclaim.RecoveryPin {
+ sum := sha256.Sum256(raw)
+ return hostclaim.RecoveryPin{Key: key, SHA256: hex.EncodeToString(sum[:]), Generation: obj.DB.Generation, MaxTXID: obj.DB.MaxTXID,
+ FilesManifest: hostfiles.ManifestKey(obj.Files.Prefix), FilesSeq: obj.Files.Seq, PinnedAt: obj.PinnedAt}
+}
+
+// filesPin is §5.6's files pin: the manifest of holder.generation as
+// observed or, if that generation has none, of the nearest predecessor in
+// holder.admissions that has one. It is chosen independently of the
+// database baseline (B2-4). A read error refuses; no manifest anywhere in
+// the chain is a host that never published files, pinned at sequence 0.
+func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder) (FilesPin, error) {
+ chain := []string{h.Generation}
+ for i := len(h.Admissions) - 1; i >= 0; i-- {
+ chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor)
+ }
+ seen := map[string]bool{}
+ for _, gen := range chain {
+ if gen == "" || gen == hostclaim.RootGeneration || seen[gen] {
+ continue
+ }
+ seen[gen] = true
+ prefix := h.Location + "/files/" + gen + "/"
+ raw, _, err := p.Data.Get(ctx, hostfiles.ManifestKey(prefix))
+ if errors.Is(err, os.ErrNotExist) {
+ continue
+ }
+ if err != nil {
+ return FilesPin{}, err
+ }
+ var m hostfiles.Manifest // decoded from the bytes the digest names
+ if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion {
+ return FilesPin{}, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err)
+ }
+ sum := sha256.Sum256(raw)
+ return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:])}, nil
+ }
+ gen := h.Generation
+ if gen == "" {
+ gen = hostclaim.RootGeneration
+ }
+ return FilesPin{Prefix: h.Location + "/files/" + gen + "/"}, nil
+}
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1 -v 2>&1 | grep -E '^(---|ok|FAIL)'`
+Expected: PASS for all eleven tests.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/hostrelocate/worker.go internal/hostrelocate/pin.go internal/hostrelocate/worker_test.go internal/hostrelocate/pin_test.go
+git commit -m "hostrelocate: a regional worker starts a relocation and replays it one claim CAS at a time, reading fences only
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 6: The participant's half, ordered (`internal/hostrelocate`)
+
+A participant box acknowledges first (D6, after §5.10's pending-state and update-begun check, inside `Effects.Acknowledge`); then, as `op.exec`, it releases in `releasing` or `probe-failed`, builds P in `installing`, makes its local install and enters `probing` in `installed`, and probes in `probing`. Coverage for the probe admission is `released` only when this operation journaled a release whose session is the predecessor writer and P has not been opened; `AdmitWriter` checks the rest. A box on `lost`, or not a participant, does nothing. The two end-to-end tests drive the worker and both participants against one claim store: a drain completes with `released` and no declared loss; a failover completes and declares the source's session.
+
+**Files:**
+- Create: `internal/hostrelocate/participant.go`
+- Test: `internal/hostrelocate/participant_test.go`
+
+**Interfaces:**
+- Consumes: Tasks 1 and 5 (the worker fixture in `worker_test.go`).
+- Produces: `hostrelocate.Effects` (`Acknowledge`, `Release`, `Build`, `InstallLocal`, `Probe`), `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostrelocate/participant_test.go`:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "errors"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+)
+
+// fakeBox is one box's local effects. Its writes to the claim stand in for
+// what the real effects leave in storage (P built, a probe admitted).
+type fakeBox struct {
+ f *fixture
+ box string
+ coverages []string
+}
+
+func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) {
+ return hostclaim.ExecutionAcknowledgement{Op: o.Claim.Op.ID, Enrollment: enrollment(b.box), BootID: "boot", ControllerID: "ctl-" + b.box, LeaseETag: "lease", Expires: b.f.now.Add(15 * 60e9)}, nil
+}
+func (b *fakeBox) Release(_ context.Context, o hostclaim.Observed) (hostclaim.Release, error) {
+ h := o.Claim.Holder
+ return hostclaim.Release{ReleasedBy: b.box, Enrollment: enrollment(b.box), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r-" + b.box, ReceiptSHA256: strings.Repeat("d", 64)}, nil
+}
+func (b *fakeBox) Build(context.Context, hostclaim.Observed) error {
+ b.f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = c.Op.Journal["prepared"]; return nil })
+ return nil
+}
+func (b *fakeBox) InstallLocal(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) {
+ n, _ := strconv.ParseUint(o.Claim.Op.Journal["install_stamp"], 10, 64)
+ return hostclaim.LocalInstall{Box: b.box, Enrollment: enrollment(b.box), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil
+}
+func (b *fakeBox) Probe(ctx context.Context, o hostclaim.Observed, coverage string) (uint64, bool, error) {
+ b.coverages = append(b.coverages, coverage)
+ c := o.Claim
+ next, err := hostclaim.Update(ctx, b.f.s, o, hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: c.Op.ID, ObservedPhase: c.Op.Phase, ObservedSession: c.Holder.Session, ObservedGeneration: c.Holder.Generation,
+ Box: b.box, Mode: "cold", NewGeneration: c.Op.Journal["prepared"], KeepStamp: true, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"}))
+ if err != nil {
+ return 0, false, err
+ }
+ return next.Claim.Holder.Session, true, nil
+}
+
+func (f *fixture) clock() time.Time { return f.now }
+
+// run steps the worker and both participants until the host serves again.
+func run(t *testing.T, f *fixture, parts ...Participant) hostclaim.Observed {
+ t.Helper()
+ ctx := context.Background()
+ for i := 0; i < 60; i++ {
+ if o := f.claim(); o.Claim.Op.Kind == hostclaim.KindServing {
+ return o
+ }
+ if _, err := f.w.Step(ctx, host); err != nil && !errors.Is(err, ErrNotYet) {
+ t.Fatalf("worker: %v", err)
+ }
+ for _, p := range parts {
+ if _, err := p.Step(ctx, host); err != nil && !errors.Is(err, ErrNotYet) {
+ t.Fatalf("%s: %v", p.Box, err)
+ }
+ }
+ }
+ t.Fatalf("no progress: %+v", f.claim().Claim.Op)
+ return hostclaim.Observed{}
+}
+
+func TestADrainCompletesWithBothParticipants(t *testing.T) {
+ f := newFixture(t)
+ src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"}
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ o := run(t, f, Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock}, Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock})
+ if o.Claim.Op.Exec != "dst" || len(dst.coverages) != 1 || dst.coverages[0] != "released" {
+ t.Fatalf("exec %s coverages %v", o.Claim.Op.Exec, dst.coverages)
+ }
+ last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
+ if last.WritesLost != nil {
+ t.Fatalf("a drain declared a loss: %+v", last)
+ }
+}
+
+func TestAFailoverCompletesAndDeclaresTheSource(t *testing.T) {
+ f := newFixture(t)
+ f.terminate("src")
+ dst := &fakeBox{f: f, box: "dst"}
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ o := run(t, f, Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock})
+ last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1]
+ if dst.coverages[0] != "" || last.WritesLost == nil || last.WritesLost.Box != "src" {
+ t.Fatalf("coverages %v last %+v", dst.coverages, last)
+ }
+}
+
+func TestALostParticipantDoesNothing(t *testing.T) {
+ f := newFixture(t)
+ f.terminate("src")
+ if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ src := &fakeBox{f: f, box: "src"}
+ before := f.claim().ETag
+ if _, err := (Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock}).Step(context.Background(), host); !errors.Is(err, ErrNotYet) || f.claim().ETag != before {
+ t.Fatalf("a fenced source acted: %v", err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostrelocate/ -run 'Completes|LostParticipant' -count=1`
+Expected: FAIL to compile: `undefined: Participant`.
+
+- [ ] **Step 3: The participant**
+
+`internal/hostrelocate/participant.go`:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "slices"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// Effects are a participant box's local effects, each under the host
+// guard and bound to the claim observation it is given. Plan 6b's box
+// assembly implements them (ClaimAssembly); this package only orders them
+// and records each result in the claim.
+type Effects interface {
+ // Acknowledge writes this box's pending controller state, refuses while
+ // root has an update begun (§5.10), takes the execution lease
+ // (hostexec.Open, Ready) and returns the acknowledgement to journal.
+ Acknowledge(context.Context, hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error)
+ // Release runs §5.8's release sequence for this box's current writer
+ // session: stop every writer (stopevidence.WritersStopped), flush the
+ // database to its final TXID, publish every pending file and wake-state
+ // mutation, close the replicator and publisher, and complete full stop
+ // evidence (ClaimStopCoordinator.FenceAndStop). It returns the journal
+ // entry; it is idempotent for one session.
+ Release(context.Context, hostclaim.Observed) (hostclaim.Release, error)
+ // Build builds the prepared generation P from the journaled pin with
+ // moves' shared install routine (L16), under the replica lease.
+ Build(context.Context, hostclaim.Observed) error
+ // InstallLocal makes this box's local install from P's files manifest
+ // and returns the receipt (§5.8 "The local install").
+ InstallLocal(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error)
+ // Probe makes one cold probe admission (AdmitWriter, with coverage the
+ // caller names) and runs the probe. passed is false while it runs.
+ Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (session uint64, passed bool, err error)
+}
+
+// Participant is one box's half of relocation. Step makes at most one
+// participant transition for one host, chosen from the claim alone.
+type Participant struct {
+ Claims release.Store
+ Box string
+ Enrollment string
+ Effects Effects
+ Now func() time.Time
+}
+
+// Step acts for host if this box has work in its relocation, and reports
+// ErrNotYet when it has none.
+func (p Participant) Step(ctx context.Context, host string) (hostclaim.Observed, error) {
+ if p.Claims == nil || p.Box == "" || p.Enrollment == "" || p.Effects == nil || p.Now == nil {
+ return hostclaim.Observed{}, ErrUnsupported
+ }
+ obs, err := hostclaim.Read(ctx, p.Claims, host)
+ if err != nil || obs.Absent {
+ return obs, err
+ }
+ c := obs.Claim
+ if c.Op.Kind != hostclaim.KindRelocating || c.Op.Journal["stuck"] != "" || !slices.Contains(c.Op.Participants, p.Box) || slices.Contains(c.Lost, p.Box) {
+ return obs, ErrNotYet
+ }
+ id := c.Op.ID
+ if c.Op.Journal["exec_ack."+p.Box] == "" {
+ ack, err := p.Effects.Acknowledge(ctx, obs)
+ if err != nil {
+ return obs, err
+ }
+ return p.update(ctx, obs, hostclaim.AcknowledgeExecution(p.Box, ack))
+ }
+ executor := c.Op.Exec == p.Box
+ switch {
+ case executor && (c.Op.Phase == hostclaim.PhaseReleasing || c.Op.Phase == hostclaim.PhaseProbeFailed) && c.Op.Journal["release"] == "":
+ r, err := p.Effects.Release(ctx, obs)
+ if err != nil {
+ return obs, err
+ }
+ return p.update(ctx, obs, hostclaim.JournalRelease(id, r))
+ case executor && c.Op.Phase == hostclaim.PhaseInstalling:
+ return obs, errors.Join(ErrNotYet, p.Effects.Build(ctx, obs))
+ case executor && c.Op.Phase == hostclaim.PhaseInstalled:
+ if c.Op.Journal["local_install"] != "" {
+ // The target enters probing itself (§5.8 phase table).
+ return p.update(ctx, obs, hostclaim.ToProbing(id, p.Now()))
+ }
+ l, err := p.Effects.InstallLocal(ctx, obs)
+ if err != nil {
+ return obs, err
+ }
+ if l.Box != p.Box || l.Enrollment != p.Enrollment {
+ return obs, ErrRefused
+ }
+ return p.update(ctx, obs, hostclaim.RecordLocalInstall(id, l))
+ case executor && c.Op.Phase == hostclaim.PhaseProbing && c.Op.Journal["probe"] == "":
+ session, passed, err := p.Effects.Probe(ctx, obs, coverageFor(c))
+ if err != nil || !passed {
+ return obs, errors.Join(ErrNotYet, err)
+ }
+ // The probe's admission wrote the claim: record the result on a
+ // fresh read, which RecordProbe binds to the admitted session.
+ fresh, err := hostclaim.Read(ctx, p.Claims, host)
+ if err != nil || fresh.Absent {
+ return obs, errors.Join(ErrNotYet, err)
+ }
+ return p.update(ctx, fresh, hostclaim.RecordProbe(id, hostclaim.Probe{Box: p.Box, Session: session}))
+ }
+ return obs, ErrNotYet
+}
+
+// coverageFor is the probe admission's coverage (§5.8, L9): `released`
+// only for the admission that opens P, when this operation journaled a
+// release whose box is the predecessor writer; "" otherwise (a failover,
+// a later attempt, or a host that never had a writer). AdmitWriter checks
+// the rest.
+func coverageFor(c *hostclaim.Claim) string {
+ var r hostclaim.Release
+ if json.Unmarshal([]byte(c.Op.Journal["release"]), &r) != nil || r.ReleasedBy == "" {
+ return ""
+ }
+ p := c.Op.Journal["prepared"]
+ for _, a := range c.Holder.Admissions {
+ if a.Generation == p {
+ return ""
+ }
+ }
+ pred := hostclaim.AdmissionRecord{Session: c.Holder.Session, Box: c.Holder.WriterBox, Generation: c.Holder.Generation}
+ if n := len(c.Holder.Admissions); n > 0 {
+ pred = c.Holder.Admissions[n-1]
+ }
+ if pred.Box != r.ReleasedBy || pred.Session != r.Session || pred.Generation != r.Generation {
+ return ""
+ }
+ return "released"
+}
+
+func (p Participant) update(ctx context.Context, obs hostclaim.Observed, tr func(*hostclaim.Claim) error) (hostclaim.Observed, error) {
+ next, err := hostclaim.Update(ctx, p.Claims, obs, tr)
+ if errors.Is(err, hostclaim.ErrAlreadyRecorded) || errors.Is(err, hostclaim.ErrAckAlreadyApplied) {
+ return obs, nil
+ }
+ return next, err
+}
+```
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/hostrelocate/participant.go internal/hostrelocate/participant_test.go
+git commit -m "hostrelocate: a participant acknowledges, releases, installs and probes in the order its phase allows
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 7: Release evidence on the box (`internal/host`)
+
+`StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (writers stopped, database flush, files flush), and only then lets the producers drain; a failed step stops everything after it; a receipt root replays without this process having flushed is refused rather than guessed at.
+
+**Files:**
+- Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`
+- Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go`
+
+**Interfaces:**
+- Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`.
+- Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationEffects{Identity, Controller, Producers, Collect, Writers, FlushDB, FlushFiles, BuildP, Install, Admit}`.
+
+Note: `internal/host`'s tests declare a function named `hostexec`, so this file imports the package as `execctl`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/host/relocationreleases_test.go`:
+
+```go
+package host
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "errors"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func TestStopReceiptReleasesBindsTheReceiptToTheRelease(t *testing.T) {
+ ctx := context.Background()
+ inc := strings.Repeat("a", 64)
+ c := hostclaim.Claim{Host: "shop.example", Holder: &hostclaim.Holder{Incarnation: inc}, Op: hostclaim.Op{ID: "rel-1", Kind: hostclaim.KindRelocating, Phase: hostclaim.PhaseReleasing}}
+ receipt := StopReceipt{V: 1, ID: "r1", Host: "shop.example", Incarnation: inc, WriterSession: 12, OperationID: "rel-1", Phase: hostclaim.PhaseReleasing, ObligationID: RelocationReleaseObligation,
+ Box: "src", Enrollment: "enr", Kind: "stopped", EvidenceAt: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)}
+ put := func(s release.Store, r StopReceipt) hostclaim.Release {
+ body := mustMarshalStopReceipt(r)
+ key := "control/hosts/shop.example/evidence/" + r.OperationID + "/" + r.Box + "/" + r.ID + ".json"
+ if _, err := s.Put(ctx, key, body, ""); err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(body)
+ return hostclaim.Release{ReleasedBy: "src", Enrollment: "enr", Session: 12, Receipt: key, ReceiptSHA256: hex.EncodeToString(sum[:])}
+ }
+ s := release.NewDirStore(t.TempDir())
+ a := StopReceiptReleases{Evidence: s}
+ if err := a.VerifyRelease(ctx, c, put(s, receipt)); err != nil {
+ t.Fatal(err)
+ }
+ for _, tc := range []struct {
+ name string
+ mutate func(*StopReceipt, *hostclaim.Release)
+ }{
+ {"another session", func(r *StopReceipt, _ *hostclaim.Release) { r.WriterSession = 11 }},
+ {"another box", func(r *StopReceipt, _ *hostclaim.Release) { r.Box = "dst" }},
+ {"another operation", func(r *StopReceipt, _ *hostclaim.Release) { r.OperationID = "rel-0" }},
+ {"a retirement's stop", func(r *StopReceipt, _ *hostclaim.Release) { r.ObligationID = "park-1" }},
+ {"another phase", func(r *StopReceipt, _ *hostclaim.Release) { r.Phase = hostclaim.PhaseProbeFailed }},
+ {"another incarnation", func(r *StopReceipt, _ *hostclaim.Release) { r.Incarnation = strings.Repeat("b", 64) }},
+ {"a digest that is not the bytes", func(_ *StopReceipt, rel *hostclaim.Release) { rel.ReceiptSHA256 = strings.Repeat("0", 64) }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ r := receipt
+ var mutated hostclaim.Release
+ tc.mutate(&r, &mutated)
+ rel := put(s, r)
+ if mutated.ReceiptSHA256 != "" {
+ rel.ReceiptSHA256 = mutated.ReceiptSHA256
+ }
+ if err := (StopReceiptReleases{Evidence: s}).VerifyRelease(ctx, c, rel); !errors.Is(err, ErrClaimStopReceipt) {
+ t.Fatalf("accepted: %v", err)
+ }
+ })
+ }
+}
+```
+
+`internal/host/relocationeffects_test.go`:
+
+```go
+package host
+
+import (
+ "context"
+ "errors"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ execctl "github.com/carlosframework/platform/internal/hostexec"
+ "github.com/carlosframework/platform/internal/stopevidence"
+)
+
+type emptyGroup struct{}
+
+func (emptyGroup) Populated() (bool, error) { return false, nil }
+func (emptyGroup) Members() ([]int, error) { return nil, nil }
+
+type recordingDrain struct{ calls *[]string }
+
+func (d recordingDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) {
+ *d.calls = append(*d.calls, "producers")
+ return DrainAcknowledgement{FenceID: "fence"}, nil
+}
+
+type readyController struct{ st execctl.State }
+
+func (c readyController) Ready(context.Context) (execctl.State, error) { return c.st, nil }
+
+func effectsFixture(t *testing.T) (RelocationEffects, hostclaim.Observed, *[]string, *StopRequest) {
+ t.Helper()
+ calls := &[]string{}
+ seen := &StopRequest{}
+ proc := t.TempDir() // no process holds anything
+ e := RelocationEffects{
+ Identity: func(context.Context) (RelocationIdentity, error) {
+ return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil
+ },
+ Producers: func(context.Context, hostclaim.Observed) (StopDrainer, error) { return recordingDrain{calls}, nil },
+ Writers: func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error) {
+ *calls = append(*calls, "writers")
+ return stopevidence.WritersInputs{Host: "shop.example", FenceClosed: true, SpawnsSettled: true, WorkDrained: true, Cgroup: emptyGroup{},
+ DBPaths: []string{"/data/shop.db"}, ReplicatorPID: 9, ReplicatorStart: "1", ProcRoot: proc}, nil
+ },
+ FlushDB: func(context.Context, hostclaim.Observed) (uint64, error) {
+ *calls = append(*calls, "db")
+ return 41, nil
+ },
+ FlushFiles: func(context.Context, hostclaim.Observed) (int64, error) {
+ *calls = append(*calls, "files")
+ return 8, nil
+ },
+ Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) {
+ *seen = r
+ if _, err := d.DrainClaim(ctx, r, "fence"); err != nil {
+ return StopResult{}, err
+ }
+ *calls = append(*calls, "receipt")
+ return StopResult{Key: "control/hosts/shop.example/evidence/rel-1/src/r.json", Digest: strings.Repeat("a", 64)}, nil
+ },
+ }
+ obs := hostclaim.Observed{Host: "shop.example", ETag: "etag-7", Claim: &hostclaim.Claim{Host: "shop.example",
+ Holder: &hostclaim.Holder{Incarnation: "inc", Stamp: 5, Session: 6, Generation: "g6", WriterBox: "src"},
+ Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseReleasing, Exec: "src"}}}
+ return e, obs, calls, seen
+}
+
+func TestRelocationReleaseRunsStepsOneToThreeInsideTheDrain(t *testing.T) {
+ e, obs, calls, seen := effectsFixture(t)
+ r, err := e.Release(context.Background(), obs)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if want := []string{"writers", "db", "files", "producers", "receipt"}; !slices.Equal(*calls, want) {
+ t.Fatalf("order %v, want %v", *calls, want)
+ }
+ if r.FinalTXID != 41 || r.ManifestSeq != 8 || r.Session != 6 || r.Generation != "g6" || r.ReleasedBy != "src" || r.Receipt == "" {
+ t.Fatalf("release: %+v", r)
+ }
+ if seen.ObligationID != RelocationReleaseObligation || seen.Phase != hostclaim.PhaseReleasing || seen.ExpectedClaimETag != "etag-7" || seen.ExpectedDirectoryRevision != "4" {
+ t.Fatalf("stop request: %+v", *seen)
+ }
+}
+
+func TestRelocationReleaseStopsAtTheFirstFailedStep(t *testing.T) {
+ failure := errors.New("sync failed")
+ for _, tc := range []struct {
+ name string
+ change func(*RelocationEffects)
+ want []string
+ }{
+ {"writers still running", func(e *RelocationEffects) {
+ w := e.Writers
+ e.Writers = func(ctx context.Context, o hostclaim.Observed) (stopevidence.WritersInputs, error) {
+ in, err := w(ctx, o)
+ in.WorkDrained = false
+ return in, err
+ }
+ }, []string{"writers"}},
+ {"the database flush", func(e *RelocationEffects) {
+ e.FlushDB = func(context.Context, hostclaim.Observed) (uint64, error) { return 0, failure }
+ }, []string{"writers"}},
+ {"the files flush", func(e *RelocationEffects) {
+ e.FlushFiles = func(context.Context, hostclaim.Observed) (int64, error) { return 0, failure }
+ }, []string{"writers", "db"}},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ e, obs, calls, _ := effectsFixture(t)
+ tc.change(&e)
+ if _, err := e.Release(context.Background(), obs); err == nil {
+ t.Fatal("released")
+ }
+ if !slices.Equal(*calls, tc.want) {
+ t.Fatalf("ran %v, want %v: nothing after a failed step", *calls, tc.want)
+ }
+ })
+ }
+}
+
+func TestRelocationReleaseNeverGuessesPositionsForAReplayedReceipt(t *testing.T) {
+ e, obs, _, _ := effectsFixture(t)
+ e.Collect = func(context.Context, StopRequest, StopDrainer) (StopResult, error) {
+ return StopResult{Key: "k", Digest: strings.Repeat("a", 64)}, nil // root replayed; no drain ran here
+ }
+ if _, err := e.Release(context.Background(), obs); !errors.Is(err, ErrClaimStopReceipt) {
+ t.Fatalf("got %v", err)
+ }
+}
+
+func TestRelocationAcknowledgeJournalsTheControllersLease(t *testing.T) {
+ e, obs, _, _ := effectsFixture(t)
+ var bound execctl.Binding
+ exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC)
+ e.Controller = func(_ context.Context, b execctl.Binding) (interface {
+ Ready(context.Context) (execctl.State, error)
+ }, error) {
+ bound = b
+ return readyController{execctl.State{Binding: execctl.Binding{Enrollment: "enr", BootID: "boot", ControllerID: "c9"}, LeaseETag: "lease-1", Expires: exp}}, nil
+ }
+ ack, err := e.Acknowledge(context.Background(), obs)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if bound.Op != "rel-1" || bound.Host != "shop.example" || bound.Incarnation != "inc" || ack.LeaseETag != "lease-1" || ack.ControllerID != "c9" || !ack.Expires.Equal(exp) {
+ t.Fatalf("binding %+v ack %+v", bound, ack)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|RelocationRelease|RelocationAcknowledge' -count=1`
+Expected: FAIL to compile: `undefined: StopReceiptReleases`, `undefined: RelocationEffects`.
+
+- [ ] **Step 3: The receipt verifier**
+
+`internal/host/relocationreleases.go`:
+
+```go
+package host
+
+import (
+ "bytes"
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// RelocationReleaseObligation is the stop request's obligation component
+// for a relocation's release (§5.8 release step 4). A release retires
+// nothing, so it names no park-local chore; root's FenceAndStop requires
+// the component, and the receipt carries it so a retirement stop can never
+// be offered as a release, nor a release as a retirement.
+const RelocationReleaseObligation = "relocation-release"
+
+// StopReceiptReleases is hostclaim.ReleaseAuthority over this package's
+// stop receipts (ClaimStopCoordinator.Publish): a relocation's journaled
+// release counts only when the immutable receipt it names hashes to the
+// journaled digest and is the full stop of exactly that box, enrollment,
+// writer session, host, incarnation, operation and phase (region HA spec 1
+// §5.8 release step 4, claim §6.1).
+type StopReceiptReleases struct {
+ Evidence release.Store
+}
+
+var _ hostclaim.ReleaseAuthority = StopReceiptReleases{}
+
+func (a StopReceiptReleases) VerifyRelease(ctx context.Context, c hostclaim.Claim, r hostclaim.Release) error {
+ if a.Evidence == nil || c.Holder == nil {
+ return ErrClaimStopUnavailable
+ }
+ raw, _, err := a.Evidence.Get(ctx, r.Receipt)
+ if err != nil {
+ return err
+ }
+ sum := sha256.Sum256(raw)
+ var receipt StopReceipt
+ if hex.EncodeToString(sum[:]) != r.ReceiptSHA256 || json.Unmarshal(raw, &receipt) != nil || !bytes.Equal(raw, mustMarshalStopReceipt(receipt)) {
+ return ErrClaimStopReceipt
+ }
+ key := fmt.Sprintf("control/hosts/%s/evidence/%s/%s/%s.json", receipt.Host, receipt.OperationID, receipt.Box, receipt.ID)
+ if receipt.V != 1 || receipt.Kind != "stopped" || r.Receipt != key || receipt.Host != c.Host || receipt.Incarnation != c.Holder.Incarnation ||
+ receipt.OperationID != c.Op.ID || receipt.Phase != c.Op.Phase || receipt.ObligationID != RelocationReleaseObligation || receipt.Box != r.ReleasedBy || receipt.Enrollment != r.Enrollment ||
+ receipt.WriterSession != r.Session || receipt.EvidenceAt.IsZero() {
+ return ErrClaimStopReceipt
+ }
+ return nil
+}
+```
+
+- [ ] **Step 4: The release sequence**
+
+`internal/host/relocationeffects.go`:
+
+```go
+package host
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ execctl "github.com/carlosframework/platform/internal/hostexec"
+ "github.com/carlosframework/platform/internal/hostrelocate"
+ "github.com/carlosframework/platform/internal/stopevidence"
+)
+
+// RelocationIdentity is this box's verified actor and stop identity, as
+// Plan 6b's assembly reports it (claimStopLocal).
+type RelocationIdentity struct {
+ Actor hostbox.Actor
+ BootID, ControllerID string
+}
+
+// RelocationEffects is hostrelocate.Effects over providers Plan 6b's box
+// assembly supplies. It owns only the order of §5.8's release sequence and
+// the shape of each record; every provider acts under the host guard.
+type RelocationEffects struct {
+ Identity func(context.Context) (RelocationIdentity, error)
+ // Controller is Plan 8's execution controller for one binding
+ // (hostexec.Open with the assembly's CloseAdmission, CancelWork and
+ // Cleanup). It writes pending state, checks root's update-begun marker,
+ // and takes the execution lease (§5.10, claim D6).
+ Controller func(context.Context, execctl.Binding) (interface {
+ Ready(context.Context) (execctl.State, error)
+ }, error)
+ // Producers is the admitted session's drain (ClaimWakeAssembly.Producers).
+ Producers func(context.Context, hostclaim.Observed) (StopDrainer, error)
+ // Collect asks root for full stop evidence (Plan 6b's CollectClaimStop
+ // over the instance-control socket).
+ Collect func(context.Context, StopRequest, StopDrainer) (StopResult, error)
+ // Writers reports the drained facts and the replicator's descriptors
+ // for release step 1.
+ Writers func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error)
+ // FlushDB is release step 2: final checkpoint and SyncAndWait, then the
+ // stopped database's final TXID N, returned only once the newest
+ // level-0 file's MaxTXID is N. A sync error is returned, never swallowed.
+ FlushDB func(context.Context, hostclaim.Observed) (uint64, error)
+ // FlushFiles is release step 3: publish every pending file and
+ // wake-state mutation whatever files_durability says (hostfiles), and
+ // return the manifest's final sequence.
+ FlushFiles func(context.Context, hostclaim.Observed) (int64, error)
+ // BuildP, Install and Admit are the target's (§5.8): moves' shared
+ // install routine for P, the local install from P's files manifest,
+ // and one cold probe admission followed by the probe.
+ BuildP func(context.Context, hostclaim.Observed) error
+ Install func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error)
+ Admit func(context.Context, hostclaim.Observed, string) (uint64, bool, error)
+}
+
+var _ hostrelocate.Effects = RelocationEffects{}
+
+func (e RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) {
+ if e.Identity == nil || e.Controller == nil || obs.Claim == nil || obs.Claim.Holder == nil {
+ return hostclaim.ExecutionAcknowledgement{}, ErrClaimStopUnavailable
+ }
+ id, err := e.Identity(ctx)
+ if err != nil {
+ return hostclaim.ExecutionAcknowledgement{}, err
+ }
+ c := obs.Claim
+ ctl, err := e.Controller(ctx, execctl.Binding{Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, Host: c.Host, Incarnation: c.Holder.Incarnation, Op: c.Op.ID, BootID: id.BootID})
+ if err != nil {
+ return hostclaim.ExecutionAcknowledgement{}, err
+ }
+ st, err := ctl.Ready(ctx)
+ if err != nil {
+ return hostclaim.ExecutionAcknowledgement{}, err
+ }
+ return hostclaim.ExecutionAcknowledgement{Op: c.Op.ID, Enrollment: st.Enrollment, BootID: st.BootID, ControllerID: st.ControllerID, LeaseETag: st.LeaseETag, Expires: st.Expires}, nil
+}
+
+// Release runs the release sequence inside root's stop collection: root
+// records its durable fence and asks the agent to drain; the drain does
+// steps 1 to 3 before it closes the producers and acknowledges (§5.8,
+// claim P6-D1). The receipt is step 4.
+func (e RelocationEffects) Release(ctx context.Context, obs hostclaim.Observed) (hostclaim.Release, error) {
+ if e.Identity == nil || e.Producers == nil || e.Collect == nil || e.Writers == nil || e.FlushDB == nil || e.FlushFiles == nil || obs.Claim == nil || obs.Claim.Holder == nil {
+ return hostclaim.Release{}, ErrClaimStopUnavailable
+ }
+ id, err := e.Identity(ctx)
+ if err != nil {
+ return hostclaim.Release{}, err
+ }
+ c, h := obs.Claim, obs.Claim.Holder
+ inner, err := e.Producers(ctx, obs)
+ if err != nil {
+ return hostclaim.Release{}, err
+ }
+ d := &releaseDrain{e: e, obs: obs, inner: inner}
+ req := StopRequest{Host: c.Host, Incarnation: h.Incarnation, InstallationStamp: h.Stamp, WriterSession: h.Session,
+ OperationID: c.Op.ID, Phase: c.Op.Phase, ObligationID: RelocationReleaseObligation, Box: id.Actor.Box, Enrollment: id.Actor.Enrollment,
+ BootID: id.BootID, ControllerID: id.ControllerID, ExpectedClaimETag: obs.ETag, ExpectedDirectoryRevision: fmt.Sprint(id.Actor.Revision)}
+ result, err := e.Collect(ctx, req, d)
+ if err != nil {
+ return hostclaim.Release{}, err
+ }
+ if !d.done {
+ // Root replayed a durable receipt: this process did not flush, so
+ // it cannot vouch for N or the manifest sequence. Never guess.
+ return hostclaim.Release{}, fmt.Errorf("%w: a replayed receipt carries no flushed positions", ErrClaimStopReceipt)
+ }
+ return hostclaim.Release{ReleasedBy: id.Actor.Box, Enrollment: id.Actor.Enrollment, Generation: h.Generation, Session: h.Session,
+ FinalTXID: d.txid, ManifestSeq: d.seq, Receipt: result.Key, ReceiptSHA256: result.Digest}, nil
+}
+
+// releaseDrain is the agent's drain for a release: steps 1 to 3, in order,
+// and only then the producers' own drain and acknowledgement.
+type releaseDrain struct {
+ e RelocationEffects
+ obs hostclaim.Observed
+ inner StopDrainer
+ txid uint64
+ seq int64
+ done bool
+}
+
+func (d *releaseDrain) DrainClaim(ctx context.Context, r StopRequest, fence string) (DrainAcknowledgement, error) {
+ in, err := d.e.Writers(ctx, d.obs)
+ if err != nil {
+ return DrainAcknowledgement{}, err
+ }
+ if _, err := stopevidence.WritersStopped(ctx, in); err != nil {
+ return DrainAcknowledgement{}, err
+ }
+ if d.txid, err = d.e.FlushDB(ctx, d.obs); err != nil {
+ return DrainAcknowledgement{}, err
+ }
+ if d.seq, err = d.e.FlushFiles(ctx, d.obs); err != nil {
+ return DrainAcknowledgement{}, err
+ }
+ ack, err := d.inner.DrainClaim(ctx, r, fence)
+ if err != nil {
+ return DrainAcknowledgement{}, err
+ }
+ d.done = true
+ return ack, nil
+}
+
+func (e RelocationEffects) Build(ctx context.Context, obs hostclaim.Observed) error {
+ if e.BuildP == nil {
+ return ErrClaimStopUnavailable
+ }
+ return e.BuildP(ctx, obs)
+}
+
+func (e RelocationEffects) InstallLocal(ctx context.Context, obs hostclaim.Observed) (hostclaim.LocalInstall, error) {
+ if e.Install == nil {
+ return hostclaim.LocalInstall{}, ErrClaimStopUnavailable
+ }
+ return e.Install(ctx, obs)
+}
+
+func (e RelocationEffects) Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (uint64, bool, error) {
+ if e.Admit == nil {
+ return 0, false, ErrClaimStopUnavailable
+ }
+ return e.Admit(ctx, obs, coverage)
+}
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|Relocation|ClaimStop' -count=1`
+Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go
+git commit -m "host: a release is proved by its full stop receipt, and flushes inside root's stop collection before the producers drain
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 8: Bind the box half to the claim assembly
+
+**Gated on:** claim Plan 6b (`docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md` on `origin/design-replica-ownership`, rev 4) merged to main, for `ClaimAssembly`, `ClaimWakeAssembly.Producers`, `CollectClaimStop`, the `stopLocal` identity and the admitted attempt's replicator; and moves Phase 1 Plan 2 Task 8 (`moves.BuildGeneration`) merged, for building P. Not compiled in the dry run.
+
+This task gives every `RelocationEffects` provider its production body, and the worker its `Prepared` verifier. Each provider acts only through the assembly's existing guarded paths: no new way to admit a writer, spawn, or touch the registry.
+
+**Files:**
+- Create: `internal/host/relocationassembly.go`
+- Test: `internal/host/relocationassembly_test.go`
+
+**Interfaces:**
+- Consumes (6b rev 4, by name): `ClaimAssembly{Claims, Guard, Wake, Stop, Data, Reconcile}`, `(*ClaimWakeAssembly).Producers(ctx, hostclaim.Observed) (*ClaimProducers, error)`, `(*ClaimWakeAssembly).get(host) *claimWakeAttempt` and its `rep *lease.Replicator`, `CollectClaimStop(ctx, path, StopRequest, StopDrainer) (StopResult, error)`, `controlsock.Path`, the assembly's `stopLocal` production path returning `claimStopLocal{Actor, BootID, ControllerID, PID, ProcessStart}`; moves: `moves.BuildGeneration(ctx, moves.BuildInputs) (moves.InstallManifest, error)`; main: `hostexec.Open`, `hostfiles.Publish`, `hostfiles.Restore`, `hostfiles.LoadWake`, `hostfiles.ImportWake`, `genselect.Select`, `hostrelocate.PinObject`.
+- Produces: `(ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) RelocationEffects`; `host.PreparedFromInstallManifest{Data lease.VersionedStore, Bucket string}` implementing `hostrelocate.Prepared`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Each provider is tested with 6b's own test seams (`collectStop`, `stopLocal`) and a `release.NewDirStore`:
+
+```go
+package host
+
+import (
+ "context"
+ "errors"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+)
+
+// The release provider refuses a host whose admitted attempt has no
+// replicator: there is nothing to flush, and N must not be guessed.
+func TestRelocationFlushRefusesWithoutTheAdmittedReplicator(t *testing.T) {
+ a := ClaimAssembly{Wake: &ClaimWakeAssembly{}}
+ cfg := RelocationAssemblyConfig{}
+ _, err := a.relocationFlushDB(cfg)(context.Background(), hostclaim.Observed{Host: "shop.example", Claim: &hostclaim.Claim{Host: "shop.example", Holder: &hostclaim.Holder{Session: 3}}})
+ if !errors.Is(err, ErrClaimStopUnavailable) {
+ t.Fatalf("got %v", err)
+ }
+}
+
+// Prepared is done only when the install manifest P names verifies.
+func TestPreparedFromInstallManifestNeedsTheCompletedManifest(t *testing.T) {
+ p := PreparedFromInstallManifest{}
+ if _, done, err := p.Verify(context.Background(), hostclaim.Observed{Claim: &hostclaim.Claim{Op: hostclaim.Op{Journal: map[string]string{"prepared": "P1"}}, Holder: &hostclaim.Holder{Location: "instances/x/i"}}}); done || err == nil {
+ t.Fatalf("an unconfigured verifier reported P built: %v", err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'RelocationFlush|PreparedFromInstallManifest' -count=1`
+Expected: FAIL to compile: `undefined: RelocationAssemblyConfig`.
+
+- [ ] **Step 3: The bindings**
+
+`internal/host/relocationassembly.go`:
+
+```go
+package host
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "path/filepath"
+
+ "github.com/carlosframework/platform/internal/controlsock"
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ execctl "github.com/carlosframework/platform/internal/hostexec"
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/hostrelocate"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/moves"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/stopevidence"
+)
+
+// RelocationAssemblyConfig is what the relocation providers need beyond
+// the assembly itself.
+type RelocationAssemblyConfig struct {
+ Open func(context.Context, execctl.Config) (*execctl.Controller, error) // execctl.Open
+ Exec execctl.Config // the assembly's hooks; Binding is filled per host
+ Data release.Store // the region's data bucket (files)
+ Versions lease.VersionedStore // the same bucket, versioned
+ Bucket string
+ Registry *registry.Registry
+ FilesDir func(host string) string // the host's files directory on this box
+ WorkDir string
+}
+
+// RelocationEffects binds every provider to this assembly.
+func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) RelocationEffects {
+ return RelocationEffects{
+ Identity: func(ctx context.Context) (RelocationIdentity, error) {
+ l, err := a.localStopIdentity(ctx) // 6b: the production path behind the stopLocal seam
+ if err != nil {
+ return RelocationIdentity{}, err
+ }
+ return RelocationIdentity{Actor: l.Actor, BootID: l.BootID, ControllerID: l.ControllerID}, nil
+ },
+ Controller: func(ctx context.Context, b execctl.Binding) (interface {
+ Ready(context.Context) (execctl.State, error)
+ }, error) {
+ c := cfg.Exec
+ c.Binding = b
+ return cfg.Open(ctx, c)
+ },
+ Producers: func(ctx context.Context, o hostclaim.Observed) (StopDrainer, error) { return a.Wake.Producers(ctx, o) },
+ Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) {
+ return CollectClaimStop(ctx, controlsock.Path, r, d)
+ },
+ Writers: a.relocationWriters(),
+ FlushDB: a.relocationFlushDB(cfg),
+ FlushFiles: a.relocationFlushFiles(cfg),
+ BuildP: a.relocationBuild(cfg),
+ Install: a.relocationInstall(cfg),
+ Admit: a.relocationAdmit(),
+ }
+}
+
+// relocationWriters reports the admitted attempt's drained facts and its
+// replicator's descriptors, as recorded when the replicator opened (6b
+// records them on the attempt before any app process starts).
+func (a ClaimAssembly) relocationWriters() func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error) {
+ return func(ctx context.Context, o hostclaim.Observed) (stopevidence.WritersInputs, error) {
+ att := a.Wake.get(o.Host)
+ if att == nil {
+ return stopevidence.WritersInputs{}, ErrClaimStopUnavailable
+ }
+ return att.writersInputs(ctx) // 6b: fence closed, spawns settled, work drained, cgroup, DB paths, replicator descriptors
+ }
+}
+
+// relocationFlushDB is release step 2 (ruling 8): SyncAndWait with every
+// writer stopped, then N is the newest level-0 MaxTXID, which must equal
+// the plan's MaxTXID. A sync error is returned, never swallowed.
+func (a ClaimAssembly) relocationFlushDB(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (uint64, error) {
+ return func(ctx context.Context, o hostclaim.Observed) (uint64, error) {
+ att := a.Wake.get(o.Host)
+ if att == nil || att.rep == nil || o.Claim == nil || o.Claim.Holder == nil {
+ return 0, ErrClaimStopUnavailable
+ }
+ if err := att.rep.SyncAndWait(ctx); err != nil {
+ return 0, err
+ }
+ plan, err := genselect.Select(ctx, cfg.Versions, genselect.Request{Bucket: cfg.Bucket, Region: o.Claim.Op.Intent["region"], Host: o.Host, Holder: *o.Claim.Holder})
+ if err != nil {
+ return 0, err
+ }
+ if plan.MaxLevel0TXID != plan.MaxTXID {
+ return 0, fmt.Errorf("%w: level 0 at %d, the database at %d", ErrClaimStopUnavailable, plan.MaxLevel0TXID, plan.MaxTXID)
+ }
+ return plan.MaxTXID, nil
+ }
+}
+
+// relocationFlushFiles is release step 3: one complete publish of the
+// files directory into the session's generation, whatever its durability.
+func (a ClaimAssembly) relocationFlushFiles(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (int64, error) {
+ return func(ctx context.Context, o hostclaim.Observed) (int64, error) {
+ h := o.Claim.Holder
+ prefix := h.Location + "/files/" + h.Generation + "/"
+ seq, _, err := hostfiles.Publish(ctx, cfg.Data, prefix, cfg.FilesDir(o.Host), nil)
+ return seq, err
+ }
+}
+
+// relocationBuild builds P from the journaled pin with moves' shared
+// install routine (L16), then copies the pinned files manifest into P
+// (contracts note §1, CopyManifest; L17).
+func (a ClaimAssembly) relocationBuild(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) error {
+ return func(ctx context.Context, o hostclaim.Observed) error {
+ c := o.Claim
+ var ref hostclaim.RecoveryPin
+ if err := json.Unmarshal([]byte(c.Op.Journal["pin"]), &ref); err != nil {
+ return err
+ }
+ raw, _, err := a.Claims.Get(ctx, ref.Key)
+ if err != nil {
+ return err
+ }
+ if sum := sha256.Sum256(raw); hex.EncodeToString(sum[:]) != ref.SHA256 {
+ return ErrClaimStopReceipt
+ }
+ var pin hostrelocate.PinObject
+ if err := json.Unmarshal(raw, &pin); err != nil {
+ return err
+ }
+ p := c.Op.Journal["prepared"]
+ prefix := c.Holder.Location + "/" + p + "/"
+ if pin.N > 0 {
+ src := make([]moves.SealedObject, 0, len(pin.DB.Files))
+ for _, f := range pin.DB.Files {
+ src = append(src, moves.SealedObject{Key: f.Key, VersionID: f.VersionID, Level: f.Level, MinTXID: f.MinTXID, MaxTXID: f.MaxTXID})
+ }
+ if _, err := moves.BuildGeneration(ctx, moves.BuildInputs{Store: cfg.Versions, Bucket: cfg.Bucket, Source: src, N: pin.N,
+ GenerationPrefix: prefix, Op: c.Op.ID, Host: o.Host, WorkDir: filepath.Join(cfg.WorkDir, "relocate"), Now: a.now, Verify: lease.CheckDB}); err != nil {
+ return err
+ }
+ }
+ _, err = hostfiles.CopyManifest(ctx, cfg.Data, pin.Files.Prefix, c.Holder.Location+"/files/"+p+"/", pin.Files.Seq, nil)
+ return err
+ }
+}
+```
+
+The two remaining providers follow the same pattern and 6b's existing paths:
+
+```go
+// relocationInstall is §5.8's local install, under the host guard with the
+// stamp this box activated: restore files and env from P's files manifest
+// (origins: P's chain), import the wake state, run RenderHeld and
+// MaterializeHeld through the assembly's held-input policy, journal each
+// step locally, and return the receipt.
+func (a ClaimAssembly) relocationInstall(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) {
+ return func(ctx context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) {
+ c := o.Claim
+ prefix := c.Holder.Location + "/files/" + c.Op.Journal["prepared"] + "/"
+ var receipt hostclaim.LocalInstall
+ err := a.withRelocationInstall(ctx, o, func(local relocationInstallJournal) error { // 6b: hostguard.With on the activated stamp
+ m, _, err := hostfiles.Restore(ctx, cfg.Data, prefix, cfg.FilesDir(o.Host), local.Origins)
+ if err != nil {
+ return err
+ }
+ if ws, ok, err := hostfiles.LoadWake(ctx, cfg.Data, prefix, m, local.Origins); err != nil {
+ return err
+ } else if ok {
+ if err := hostfiles.ImportWake(cfg.Registry, o.Host, ws); err != nil {
+ return err
+ }
+ }
+ if err := a.materializeHeld(ctx, o); err != nil { // 6b: RenderHeld and MaterializeHeld via Policy.Inputs
+ return err
+ }
+ sum, err := local.Seal()
+ receipt = hostclaim.LocalInstall{Box: local.Box, Enrollment: local.Enrollment, InstallStamp: local.Stamp, FilesManifestVersion: m.Seq, JournalSHA256: sum}
+ return err
+ })
+ return receipt, err
+ }
+}
+
+// relocationAdmit is one probe attempt: the wake assembly's cold admission
+// bound to the observed op id and phase probing, with coverage, then the
+// probe (moves §7.7, same region). It reports passed only for a healthy
+// probe of the admitted session.
+func (a ClaimAssembly) relocationAdmit() func(context.Context, hostclaim.Observed, string) (uint64, bool, error) {
+ return func(ctx context.Context, o hostclaim.Observed, coverage string) (uint64, bool, error) {
+ return a.Wake.ProbeAdmission(ctx, o, coverage) // 6b: the probe-bound cold admission (Policy.Probe=true)
+ }
+}
+
+// PreparedFromInstallManifest is hostrelocate.Prepared: P is built when its
+// install manifest exists, verifies (snapshot, anchor and versions as
+// moves.InstallManifest records them), and its N is the pin's.
+type PreparedFromInstallManifest struct {
+ Data lease.VersionedStore
+ Bucket string
+}
+
+func (p PreparedFromInstallManifest) Verify(ctx context.Context, o hostclaim.Observed) (hostclaim.PreparedAuthority, bool, error) {
+ if p.Data == nil || p.Bucket == "" {
+ return hostclaim.PreparedAuthority{}, false, ErrClaimStopUnavailable
+ }
+ m, raw, err := moves.ReadInstallManifest(ctx, p.Data, p.Bucket, o.Claim.Holder.Location+"/"+o.Claim.Op.Journal["prepared"]+"/")
+ if err != nil {
+ return hostclaim.PreparedAuthority{}, false, err
+ }
+ sum := sha256.Sum256(raw)
+ return hostclaim.PreparedAuthority{Generation: o.Claim.Op.Journal["prepared"], InstallManifest: m.SnapshotKey, SHA256: hex.EncodeToString(sum[:]), SnapshotTXID: m.N}, true, nil
+}
+```
+
+Bind these names when 6b and moves Plan 2 land: `localStopIdentity` is 6b's production `stopLocal`; `(*claimWakeAttempt).writersInputs` records the replicator's descriptors when the replicator opens (scan `/proc/self/fd` for the database paths before any app process exists) and the drained facts from the producers' drain state; `withRelocationInstall` is `hostguard.Guard.With` on the stamp `op.journal.install_stamp` names; `materializeHeld` is the policy's `RenderHeld`/`MaterializeHeld` (CC13); `ProbeAdmission` is the wake assembly's admission with `startPolicy.Probe` set; `moves.SealedObject`'s fields and `moves.ReadInstallManifest` are moves Plan 2 Task 8's. An empty pin (`N == 0`) builds no snapshot or anchor: claim P6-D3 forbids `[1, 0]`; the first probe is an empty bootstrap.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'Relocation|PreparedFromInstallManifest' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/host/relocationassembly.go internal/host/relocationassembly_test.go
+git commit -m "host: the relocation effects run on the claim assembly's guarded paths, and P is built by moves' install routine
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 9: Reconcile keeps relocation rows held
+
+**Gated on:** claim Plan 6b merged, for `hostbox.ReconcileAdapter`, `registry.Route.HoldOp`, `registry.SetClaimHold`, `hostbox.RegistryHoldPolicy` and `provision.ReconcileConfig.ClaimInstall`/`ClaimRetire`. Not compiled in the dry run.
+
+`Reconciler.check` today admits only `Serving`. Under `relocating` it must mint or keep exactly what `RelocationRow` says, with the hold column carrying the operation; reconcile never deletes a relocatable host's row (a `park-local` retires it), and re-reads kept rows on the 10-minute sweep (H9, a stated exception to C4 for retention only).
+
+**Files:**
+- Modify: `internal/hostbox/reconcile.go` (`Reconciler.check`), 6b's `internal/hostbox/reconcile_adapter.go` (`Install`), 6b's `internal/provision/reconcile.go` (the sweep)
+- Test: `internal/hostbox/reconcile_relocation_test.go`
+
+**Interfaces:**
+- Consumes: Task 3's `RelocationRow`; 6b's `ReconcileAdapter{Registry, Box}`, `registry.SetClaimHold(ctx, host, owner, expected, next string)`.
+- Produces: `hostbox.Reconciler.FenceClosed func(host string) bool` (the source's local admission fence, from the wake assembly).
+
+- [ ] **Step 1: Write the failing test**
+
+```go
+package hostbox
+
+import (
+ "errors"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+)
+
+func TestReconcileAdmitsOnlyWhatTheRowCaseAllows(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ c *hostclaim.Claim
+ box string
+ want error
+ }{
+ {"the serving executor", &hostclaim.Claim{Holder: &hostclaim.Holder{Stamp: 1}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "a"}}, "a", nil},
+ {"a held target", relocating(hostclaim.PhaseInstalling, "dst"), "dst", nil},
+ {"a box with no role", relocating(hostclaim.PhaseInstalling, "dst"), "zzz", ErrChanged},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ r := Reconciler{Box: tc.box}
+ if err := r.rowAllowed(tc.c, false); !errors.Is(err, tc.want) {
+ t.Fatalf("got %v, want %v", err, tc.want)
+ }
+ })
+ }
+}
+```
+
+- [ ] **Step 2: Run it to see it fail**
+
+Run: `nice -n 19 go test ./internal/hostbox/ -run 'ReconcileAdmitsOnly' -count=1`
+Expected: FAIL to compile: `r.rowAllowed undefined`.
+
+- [ ] **Step 3: Replace the serving-only check**
+
+In `internal/hostbox/reconcile.go`, replace in `Reconciler.check`
+
+```go
+ if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil {
+ return err
+ }
+```
+
+with
+
+```go
+ if err := r.rowAllowed(obs.Claim, r.FenceClosed != nil && r.FenceClosed(obs.Host)); err != nil {
+ return err
+ }
+```
+
+and add, with the field `FenceClosed func(host string) bool` on `Reconciler`:
+
+```go
+// rowAllowed is §5.3: a row may be minted or kept for the serving executor,
+// a relocating source until its fence closes, and any other participant
+// held. Retiring and unexplained rows are never installed.
+func (r Reconciler) rowAllowed(c *hostclaim.Claim, fenceClosed bool) error {
+ switch RelocationRow(c, r.Box, true, fenceClosed).Case {
+ case RowReady, RowHeld:
+ return nil
+ }
+ return ErrChanged
+}
+```
+
+In 6b's `ReconcileAdapter.Install`, after the row is written, set its hold from the same decision, conditional on the captured ownership and the observed hold (never clearing another operation's):
+
+```go
+ d := RelocationRow(observed.Claim, a.Box, true, a.Reconciler.FenceClosed != nil && a.Reconciler.FenceClosed(rec.Host))
+ if d.Case == RowHeld && prior.HoldOp != d.HoldOp {
+ if err := a.Registry.SetClaimHold(ctx, rec.Host, owner, prior.HoldOp, d.HoldOp); err != nil {
+ return err
+ }
+ }
+```
+
+In 6b's `provision/reconcile.go` claim branch, a row whose decision is `RowRetiring` or `RowUnexplained` is kept, held, and left for typed retirement (`ClaimRetire`); log an unexplained row once per pass (`provision: %s: claim gives this box no role — held, not served`). Kept rows are re-read on a 10-minute sweep (`claimSweepEvery = 10 * time.Minute`), and immediately when the control serial changes (§5.3 "Prompt changes"); a read error keeps the row and mints none (C6).
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostbox/ ./internal/provision/ -run 'Reconcile|Relocation|Claim' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/hostbox/ internal/provision/reconcile.go
+git commit -m "hostbox: reconcile mints and keeps relocation rows held by the operation and never deletes a former executor's row
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 10: Switch the passes on, behind `CARLOS_REGION_HA`
+
+**Gated on:** claim Plan 6b merged (Task 8's effects, Task 9's rows); claim Plan 5's app gate merged (the `Admission` provider, ruling 4); and claim Plan 5's create door merged (`ProductionAuthority` as the console's create `Authority`). Not compiled in the dry run.
+
+On a box with `CARLOS_HOST_CLAIM` and `CARLOS_REGION_HA` both on, one pass every 30 seconds, and at once on a control serial change: for every claim under `control/hosts/` whose operation is `relocating` in this box's region, run `Worker.Step` (any active worker may), and for every claim naming this box as a participant, `Participant.Step`. The pass never starts a relocation by itself (spec 3's placer does); `carlos ops host relocate` (later, with Paul's copy) and the spec 3 placer call `Worker.Start`. On the console, create's `Authority.SelectExecutor` is the `ExecutorChooser` over console's live edges read.
+
+**Files:**
+- Create: `internal/host/relocatepass.go`
+- Modify: `internal/host/agent.go` (start the pass next to `runHostFencePass`); the create door's construction of `hostops.ProductionAuthority` (claim Plan 5)
+- Test: `internal/host/relocatepass_test.go`
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+package host
+
+import "testing"
+
+func TestRelocatePassIsOffUnlessBothFlagsAreOn(t *testing.T) {
+ for _, tc := range []struct {
+ claim, ha string
+ want bool
+ }{{"", "1", false}, {"1", "", false}, {"1", "0", false}, {"1", "true", true}} {
+ env := map[string]string{"CARLOS_HOST_CLAIM": tc.claim, "CARLOS_REGION_HA": tc.ha}
+ if got := relocatePassEnabled(func(k string) string { return env[k] }); got != tc.want {
+ t.Fatalf("%+v: %v", tc, got)
+ }
+ }
+}
+```
+
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'RelocatePass' -count=1`
+Expected: FAIL to compile: `undefined: relocatePassEnabled`.
+
+- [ ] **Step 3: The pass**
+
+`internal/host/relocatepass.go`:
+
+```go
+package host
+
+import (
+ "context"
+ "errors"
+ "log"
+ "os"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostrelocate"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// relocatePassEnabled: the claim and region HA both on (spec §8 step 2).
+func relocatePassEnabled(getenv func(string) string) bool {
+ return strings.TrimSpace(getenv("CARLOS_HOST_CLAIM")) != "" && regionHAEnabled(getenv)
+}
+
+// runRelocatePass replays every relocation in this box's region and this
+// box's participations. It starts none.
+func runRelocatePass(ctx context.Context, store release.Store, w hostrelocate.Worker, p hostrelocate.Participant) {
+ if !relocatePassEnabled(os.Getenv) {
+ return
+ }
+ tick := time.NewTicker(30 * time.Second)
+ defer tick.Stop()
+ for {
+ pass, cancel := context.WithTimeout(ctx, 2*time.Minute)
+ keys, err := store.List(pass, "control/hosts/")
+ if err != nil {
+ log.Printf("relocate: list claims: %v", err)
+ }
+ for _, key := range keys {
+ host, ok := strings.CutSuffix(strings.TrimPrefix(key, "control/hosts/"), ".json")
+ if !ok || strings.Contains(host, "/") {
+ continue
+ }
+ obs, err := hostclaim.Read(pass, store, host)
+ if err != nil || obs.Absent || obs.Claim.Op.Kind != hostclaim.KindRelocating {
+ continue
+ }
+ if _, err := w.Step(pass, host); err != nil && !errors.Is(err, hostrelocate.ErrNotYet) {
+ log.Printf("relocate %s: worker: %v", host, err)
+ }
+ if _, err := p.Step(pass, host); err != nil && !errors.Is(err, hostrelocate.ErrNotYet) {
+ log.Printf("relocate %s: %s: %v", host, p.Box, err)
+ }
+ }
+ cancel()
+ select {
+ case <-ctx.Done():
+ return
+ case <-tick.C:
+ }
+ }
+}
+```
+
+In `internal/host/agent.go`, next to `go runHostFencePass(agentCtx)`, build the worker and participant from the assembly once it exists and start `go runRelocatePass(agentCtx, store, worker, participant)`. The worker's providers: `Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: time.Now}`; `Releases: StopReceiptReleases{Evidence: store}`; `Admission`: claim Plan 5's app gate; `Census: func(ctx, starter, parts) error { floor, err := hostfloor.Read(ctx, store, pub, d.Bucket); …; _, err = d.CensusBoxRelocation(ctx, hostactors.ExactEvidence{…}, floor, hostactors.Scope{StarterBox: starter, Participants: parts, Namespace: …, Regions: []string{region}, Pools: …}, time.Now()); return err }`; `Choose`: an adapter that hides the excluded boxes behind `lost` and calls `hostops.ExecutorChooser.Choose`; `Pins: hostrelocate.StorePinner{Claims: store, Data: dataStore, SelectDB: func(ctx, r) { return genselect.Select(ctx, versioned, r) }, Bucket: dataBucket, Region: region}`; `Prepared: PreparedFromInstallManifest{…}`; `Discharge`: ruling 3's provider (unqualified, "the roster records no enrollment time"); `Notify: func(ctx) { _ = release.BumpSerial(ctx, store, time.Now()) }`; `Self` from `/etc/carlos/host-actor.json` (`hostfloor.ReadEnrollment`).
+
+- [ ] **Step 4: The console's create door uses the chooser**
+
+Where claim Plan 5's create door constructs `hostops.ProductionAuthority` for the console, set `Executors` to the chooser Task 2 built:
+
+```go
+ x := s.executorChooser(hostactors.Directory{Store: s.cfg.Releases, PublicKey: s.cfg.ReleasePub, Bucket: s.cfg.Bucket})
+ authority := hostops.ProductionAuthority{ /* the door's fields */ Executors: &x}
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/host/ ./internal/console/ -run 'RelocatePass|CreateExecutorChooser|Claim' -count=1`
+Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/host/relocatepass.go internal/host/relocatepass_test.go internal/host/agent.go internal/console/
+git commit -m "host: with CARLOS_REGION_HA on, every box replays its region's relocations and its own participations
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 11: The merge gate (L18)
+
+Claim rev 6.30 §1.1: "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) and the analysis's extra witnesses (§11) validate them." The #236 coordination rules: relocation may not merge until a joint simulation round passes on `internal/sim/hostlife` covering a one-box and a two-box region. Moves owns that model. This plan writes none of it.
+
+- [ ] **Step 1: Confirm the joint round has passed**
+
+The round's report must name `internal/sim/hostlife` at a commit containing this branch's claim transitions (Task 1) or a faithful model of them, and pass, with no open finding against relocation, for both a one-box region and a two-box region, covering spec §9's L18 list: create spread; relocation by drain; termination mid-serve and in every relocation phase (`starting`, `releasing`, `pinning`, `installing`, `installed`, `probing`, `probe-failed`); nested failover; a stale edges file; an unenrolled reference; a volume without `DeleteOnTermination`; a crash mid-install; late file uploads; and late PUTs after termination. Also claim §11's extra witnesses: a relocation start racing purge or app-placement admission closure; generation parking while relocation pins referenced files; the first `released` admission followed by an uncertified failed probe; and the one-box conversion of a pre-flip move holding at `pinning`.
+
+If any is missing or failing, stop. The branch stays unmerged; record the gap on amadan #236.
+
+- [ ] **Step 2: Run the branch gate**
+
+```bash
+mkdir -p /tmp/k
+go build ./... && TMPDIR=/tmp/k nice -n 19 make test
+GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...
+```
+
+Expected: PASS, apart from the socket tests the sandbox refuses (run those with the sandbox off, or read CI).
+
+- [ ] **Step 3: Push and read CI**
+
+```bash
+git push origin region-ha-1b-next
+amadan ci status carlos/platform -branch region-ha-1b-next
+```
+
+Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log and fix. Merging is Paul's call, through `amadan branch merge`, after re-merging main and waiting for CI on the merged tip.
+
+---
+
+## After this plan
+
+| What | Waits for |
+|---|---|
+| The console materializing a D4 loss record from a fence CAS, with timer knowledge `partial`, and retiring the roster entry | the console's claim-recovery pass (claim D4 "Automatic loss persistence") |
+| Automatic discharge of a fenced box's `park-local` chores | `enrolled_at` in the roster (ruling 3) |
+| `control/relocations/` create-only and never deleted in `deployment-buckets` | a tofu change with its policy test (ruling 5) |
+| The placer that decides what to relocate, `awaiting-evidence`, bounded parallel failover | spec 3 |
+| `carlos ops host relocate` and the console's view of a relocation | Paul's copy review |
+| A move into a multi-box region choosing its target with `ExecutorChooser` (L15) and a terminated move target taken off by the fence (L14) | moves Phase 1 Plans 2 and 4 |
+
+## What the dry run could not check
+
+- Tasks 8, 9 and 10 were not compiled: they bind to Plan 6b rev 4's declared names, moves Plan 2 Task 8's `BuildGeneration`, claim Plan 5's app gate and create door, none on main at 3407d114.
+- `WritersStopped` against a live agent with an embedded Litestream replicator. Its tests use a fixture `/proc`; the descriptor identification at replicator open belongs to Task 8.
+- `genselect.Select` inside `StorePinner` against a real versioned bucket. The pin tests stub `SelectDB`.
+- Socket-backed tests in `internal/host` inside the sandbox (the existing list in plan 1b's Global Constraints).
+
+## Self-review
+
+- **Spec coverage.** §5.8: the start CAS and its refusals (Tasks 1, 5), the phase table and who acts (1, 3, 5, 6), the release sequence (4, 6, 7, 8), the pinned recovery set (1, 5), building P and the local install receipt (1, 6, 8), the probe admission with `KeepStamp`, per-admission coverage and the three-attempt bound's first-opening rule (1, 6), the no-reinstall boundary (1), the replacement table and the two-replacement bound (1, 5), source loss (1, 5), what a relocation leaves behind (1: `park-local` for the source and every abandoned installation). §5.9 consumption: fences read only through `VerifyFence`/`ValidateTerminationFence`, `lost`, journal reference, discharge verdict (1, 5), the census exclusion already on main (`CheckBoxStart`, `CensusBoxRelocation`). §5.10: the worker's roster read (5, `CheckBoxStart`); the participant half and root's half are main's (`hostexec`, `systemparticipation.go`). §5.2: Task 2. §5.3: Tasks 3 and 9. §10's relocation unit tests: every replacement-table row, the bound, the probe opening P with `KeepStamp` and a restarted probe taking a fresh generation, a failover declaring `writes_lost` and a drain declaring nothing, the start refused when the roster reserves an update, a `probing` CAS refused with another box's receipt, `from` terminated in `starting` before and after acknowledging. The two-agent integration test is Task 6's worker-plus-participants run against one store; the joint model is Task 11.
+- **Placeholders.** Tasks 1 to 7: whole files or exact `git apply` patches, all dry-run. Tasks 8 to 10: complete code against named external contracts, each name listed in its "Gated on" line and "Bind these names" note.
+- **Type consistency.** `hostclaim.Release`, `RecoveryPin`, `PreparedAuthority`, `LocalInstall`, `Probe`, `Target` (Task 1) are used unchanged by Tasks 5 to 8; `hostrelocate.Effects` (Task 6) is implemented by `host.RelocationEffects` (Task 7, `var _ hostrelocate.Effects`); `hostclaim.ReleaseAuthority` by `host.StopReceiptReleases` (`var _`); `hostops.ExecutorChooser` (Task 2) by Task 10's console constructor; `hostbox.RelocationRow` (Task 3) by Task 9.
+- **Review Focus.** Each of the five has its test in the owning task. The dry run removed these guards one at a time and watched a test fail: `released`'s predecessor box; P's single opening; the failover pin delay (claim and worker); the pin reaching the release; the replacement bound; the nested failover after P; the receipt bound to `op.exec`; create's no-substitution; the start guard's live-source chore and region `park-replica`; the row case's closed fence and `lost`; the chooser's fence exclusion; discharge only when qualified; the writers-stopped thread-table check; the release receipt's obligation; the refusal of a replayed receipt; and a failed files flush stopping the release. Nineteen in all.