amadan
plan: claim Plan 12 rev 3 (Astra r2 and moves re-check: restore baseline separate from frozen loss maxima, ReleaseFencedTarget records Lost/LossRefs/AutomaticLoss atomically, StopInventoryDB before Prelaunch, authenticated never-admitted stop path, consumed fence independent of the live roster, grant-bound lost-target stamp; re-anchored on main e79d71b5)
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+2420
−1071
docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md+2420 −1071
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md b/docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md| index 24ec05838..12651f8b9 100644 |
| --- a/docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md |
| +++ b/docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md |
| @@ -1,23 +1,25 @@ |
| # Host lifecycle claim, Plan 12: moves integration completion |
| -Implementation plan, 2026-10-10, rev 2. This supplies the claim side of the controller's accepted moves delta answers M2, M3, M5, M6, M7, M9, M12 and M21. Apply Tasks A–H in order. The amended claim spec is rev 6.32. This plan authorizes no production changes. |
| +Implementation plan, 2026-10-11, rev 3. This supplies the claim side of the controller's accepted moves delta answers M2, M3, M5, M6, M7, M9, M12 and M21. Apply Tasks A–H in order. The amended claim spec is rev 6.32. This plan authorizes no production changes. |
| Requirements: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-brief.md`, and sections 2 (M1–M21) and 5 of `/home/paulca/amadan.net/carlos/platform-moves-p5/.superpowers/sdd/2026-10-06-moves-phase1-5-cleanup-and-parks/delta-after-plan10.md`. The controller's accepted answers override the delta's suggested rulings. In particular, the claim owns the instance-residue census; moves cannot be asked to invent that history. |
| ## Revision log |
| +- Rev 3, 2026-10-11: applies `plan12-rev3.md`, all four Astra round-two findings, the moves recheck P1-A/P1-B/P2-A–C and record-first notes (a)–(c). Separates restore and frozen-loss maxima, installs recovery inventory before real prelaunch, authenticates the initial source cut through the stop wiring, retains automatic fence-loss consequences and reports them after operation replacement. Removes live-roster dependence after consumption, preserves the first rollback reason, repairs record-first target death and moves the lost-target stamp into the grant. Re-anchors every patch on `e79d71b59aff4815f04973fd92b3f4a95cb18b52`. Verification and per-finding dispositions: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev3-report.md`. |
| + |
| - Rev 2, 2026-10-10: applies every controller ruling in `plan12-rev2.md`, the Astra round-one findings, and the moves review's seven ordered asks. Replaces the moves stop adapter with the target's own release journal and an optional enrolled root receipt. Adds fenced-target release under the recovery grant, first-consumption loss routing, claim-owned return retirement, independent census refusals and a retirement-history write gate. Replays against `44c1b6d7` and runs the Go checks in the shared-cache scratch export. The per-finding disposition and actual verification results are in `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev2-report.md`. |
| - Rev 1, `7facdb61`: initial A–H patches; static review identified the issues corrected here. |
| -Binding inputs: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev2.md`, `/home/paulca/.cache/tmp/claude-1001/own/findings-plan12-r1.md`, and `/home/paulca/amadan.net/carlos/platform-moves-p5/.superpowers/sdd/2026-10-06-moves-phase1-5-cleanup-and-parks/plan12-moves-review.md`. Rev 2's controller rulings supersede rev 1's signed-stop proposal and responsibility split. |
| +Binding inputs for rev 3: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev3.md`, `/home/paulca/.cache/tmp/claude-1001/own/findings-plan12-r2.md`, and `/home/paulca/amadan.net/carlos/platform-moves-p5/.superpowers/sdd/2026-10-06-moves-phase1-5-cleanup-and-parks/plan12-rev2-recheck.md`. Earlier binding inputs: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev2.md`, `/home/paulca/.cache/tmp/claude-1001/own/findings-plan12-r1.md`, and `/home/paulca/amadan.net/carlos/platform-moves-p5/.superpowers/sdd/2026-10-06-moves-phase1-5-cleanup-and-parks/plan12-moves-review.md`. Rev 2's controller rulings supersede rev 1's signed-stop proposal and responsibility split. |
| ## Global Constraints |
| -- Every code hunk is anchored to `44c1b6d77ed16f92e46e892b8254835edbd567d8`, the required `origin/main` base carrying claim Plans 1–11, then the earlier tasks in this document. Never replace that input with the moving remote-tracking ref. It advanced during authoring; none of its subsequent changes are included. |
| +- Every code hunk is anchored to `e79d71b59aff4815f04973fd92b3f4a95cb18b52`, the required main carrying moves P5 Tasks 5 and 7, F1 `0d338c14` and #268, then the earlier tasks in this document. Never substitute a moving remote-tracking ref. |
| - Everything remains behind `CARLOS_HOST_CLAIM`. Group D's existing moves gate remains in place. No new initializer, service or production activation is added. The retired-history write gate only restricts claim-on writes. With the flag off, existing bytes and behavior stay identical to the pinned base. |
| - Keep #267: no by-name root file effect, no root SQLite open, and all AWS configuration loads through `rootaws`. This plan adds no AWS client and no privileged file effect. `Guard.Activate` and `Guard.With` retain their descriptor-based implementation. A source never manufactures a root receipt for a remote target session. |
| - Read and preserve the underlying admission, stop, publication, park and loss checks. A cross-box predecessor proof is not a coverage certificate. A lease does not replace authorization. The new local authorizer does not weaken the regional one. |
| -- Existing wire data still parses. The new `retired_locations` map is optional. Because old claim readers are strict, leave `CARLOS_HOST_CLAIM_RETIRED_LOCATIONS` unset until every claim consumer has the new reader. Set it to `1` only after that rollout and its release-floor check; `retire` otherwise writes no new field. Missing historical mappings remain a typed refusal. |
| +- Existing wire data still parses. The new `retired_locations` map and signed loss record `automatic` payload and `alerts_pending` field are optional. Upgrade strict loss readers before enabling fenced-target release. Because old claim readers are strict, leave `CARLOS_HOST_CLAIM_RETIRED_LOCATIONS` unset until every claim consumer has the new reader. Set it to `1` only after that rollout and its release-floor check; `retire` otherwise writes no new field. Missing historical mappings remain a typed refusal. |
| - Task G classifies the recovery-grant record writer, operator reason and changed identity/transport units. No console route is added. |
| - Author without git writes, commits, subagents or production activity. Export the pinned commit under `$TMPDIR`. Never create a private Go cache, never put one under `/tmp`, and remove only scratch paths this session created. |
| - The inventoried diagnostic and rollback strings are inventory drafts, not approved product copy. The implementation owner must obtain copy approval before shipping them. This document and the report are implementation instructions for the controller. |
| @@ -25,10 +27,10 @@ Binding inputs: `/home/paulca/.cache/tmp/claude-1001/own/after236/plan12-rev2.md |
| ## Review Focus |
| 1. First loss consumption accepts frozen target, source-cut and empty source sets. The marker is written with admission and survives trimming. A restarted probe and a later source session do not consume the same declaration again; all still refuse clean-return or sealed coverage. |
| -2. Remote recovery selects only restoring with a nonempty target writer. Reverting, never-admitted holders and subsequent source writers retain their local path. The existing string-valued `claimRecoveryKind` and new observation helper have distinct names. |
| +2. Remote recovery selects only restoring with a nonempty target writer. The first revert or never-admitted source authenticates its source cut through the real stop wiring; subsequent source writers retain their own local receipts. The existing string-valued `claimRecoveryKind` and new observation helper have distinct names. |
| 3. `TargetReleased` captures the target's session and generation from the holder in its own exec-to-none CAS. Recovery checks that journal plus the ordinary complete report, or an enrolled root receipt, or the operation's signed loss, or a verified target fence. No moves adapter or release-key signature fixture is introduced. |
| 4. `ReleaseFencedTarget` revokes execution, removes participation, reaches `released` and queues every owed park atomically. The console grant verifies record ETag and sequence again after fence verification. The source waits for the claim phase and accepts the consumed-fence marker. |
| -5. The claim's restore hook retires the source database and marker for restoring and reverting under the exact source stamp, generation, inventory and stop evidence. The Linux acceptance test continues through `Prepare` and `Restore`. |
| +5. The claim's restore hook retires the source database and marker for restoring and reverting under the exact source stamp, generation, inventory and stop evidence. After activation, the source rebinds the exact registry rows and StopInventoryDB installation under the guard before prelaunch. The Linux acceptance test uses real Prelaunch and PrepareState through restore. |
| 6. `RunHeld` keeps the same authorizer and census and borrows the exact caller-held location lease. The claim owns `ReplicaParkLeaseSet`; moves no longer registers a resolver. The caller keeps its lease through `published`. |
| 7. The holder's bare location gets its incarnation label outside the live region. Qualified paths use their own identity regardless of the 32-entry cap. Ambiguous bare history gets a typed refusal for that location. The claim proves free-host lanes from its retained identity and quiet bound; one namespace's refusal does not stop the other's sweep. |
| 8. Preserve #267, the unwrapped local park authorizer, flag-off behavior and the C8 files-provider gate. Spec §6.6 P6-D3 already requires destination-local objects after cross-region materialization. |
| @@ -66,19 +68,32 @@ func OperationLossID(*Claim) (string, error) |
| func OperationLossConsumed(*Claim, string) bool |
| func VerifyMoveTargetFence(context.Context, release.Store, Observed, FenceRef, FenceAuthority) (ReturnFence, error) |
| func ReleaseFencedTarget(FencedTargetRelease) func(*Claim) error |
| +func ConsumedMoveTargetFence(*Claim) (FenceRef, bool) |
| +func LostMoveTargetParked(*Claim, string) bool |
| // internal/hostops |
| func MoveLossRecoveryOwner() hostclaim.LossRecoveryOwner |
| func ParseMoveLossBaseline(string) (kind, region, identity string, err error) |
| func (*MoveRecoveryGrant) ReleaseFencedTarget(context.Context, hostclaim.FenceRef) error |
| +func (*MoveRecoveryGrant) StampLostMoveTargetParked(context.Context) error |
| +func (*ControlWriter) ReportAutomaticLosses(context.Context, release.Signer, string) error |
| ``` |
| -The grant performs one mutation per invocation: record the fence rollback |
| -using `RollbackTargetFenced`, `LossRef` and a fresh `LostAt` bound; release the |
| -claim with its parks; then complete the record with the fence as evidence. |
| -Moves calls the same method on successive recovery passes. A dirty fenced |
| -target follows declared loss. Sources validates the frozen record once; |
| -the writer derives admission maxima before calling `AdmissionEvidence`. |
| +The grant performs one mutation per invocation: record a fence rollback when |
| +needed, revoke the target and retain its loss evidence, then complete the record |
| +from that consumed evidence. Existing rollback reasons stay unchanged. A clean |
| +released record with the canonical complete target report needs only the missing |
| +claim revocation. Parks use the fence's original observation time. Moves calls |
| +the same method on successive passes. Dirty targets retain J4 declared loss. |
| +The grant-bound lost-target stamp replaces the console's separate closure. |
| +Automatic reporting uses the recovery-scoped ControlWriter and root signer; |
| +it can materialize retained losses after the move has closed. The signed record |
| +retains pending alert work; materialization is not notification delivery. |
| + |
| +The writer validates Select's restore baseline first. Frozen predecessor maxima |
| +then replace only the matching loss tuple, before AdmissionEvidence. No frozen |
| +predecessor means Select's validated maximum remains. The restore plan is never |
| +compared with an unrelated frozen target maximum. |
| The existing J4 owner and `AdmitWriterWithLoss` signatures remain compatible. |
| `MoveParks` gains the local authorizer before its error. Move-park registration |
| @@ -90,7 +105,7 @@ The Go tree comes from the pinned main commit. The design spec is intentionally |
| ```bash |
| CLAIM12_REPO=/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership |
| -CLAIM12_BASE=44c1b6d77ed16f92e46e892b8254835edbd567d8 |
| +CLAIM12_BASE=e79d71b59aff4815f04973fd92b3f4a95cb18b52 |
| CLAIM12_PLAN="$CLAIM12_REPO/docs/superpowers/plans/2026-10-10-host-lifecycle-claim-12-moves-integration.md" |
| CLAIM12_SCRATCH=$(mktemp -d "$TMPDIR/claim12.XXXXXX") |
| export CLAIM12_REPO CLAIM12_BASE CLAIM12_PLAN CLAIM12_SCRATCH |
| @@ -140,11 +155,11 @@ go test ./internal/hostbox ./internal/hostclaim -run 'TestPlan12|TestAdmitWriter |
| Route the first current-operation loss through the signed declaration primitive, including a source-cut predecessor or no frozen sources. Record `loss_admitted.<id>` in the admission CAS. Later source admissions keep the original losses and use normal admission; a repeated matching input LossID is cleared only after durable consumption. Coverage remains uncovered throughout the loss-bearing operation. |
| -Implement `ValidateLossRecovery` on the existing hostops loss owner and export its pure policy adapter and baseline decoder. `LossAdmissionSources` is the one source of both primary and additional maxima; the writer fills them before invoking admission evidence. Every frozen tuple still passes the primitive's exact validation. Missing adapters refuse first consumption. Stale observations return one CAS conflict. |
| +Implement `ValidateLossRecovery` on the existing hostops loss owner and export its pure policy adapter and baseline decoder. `LossAdmissionSources` supplies the frozen loss maxima. First compare the non-nil restore plan with Select's maximum. Then override only a frozen predecessor's maximum and fill additional losses. Empty frozen sources preserve the restore maximum. AdmissionEvidence receives these loss maxima alongside the original restore plan. Every frozen tuple still passes the primitive's exact validation. Missing adapters refuse first consumption. Stale observations return one CAS conflict. |
| Focused command: `go test ./internal/hostbox ./internal/hostclaim ./internal/hostops -run 'TestPlan12Loss|TestLossBeforeTargetAdmissionRecovers|TestAdmitWriterWithLoss' -count=1`. |
| -Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops policy API is undefined. Tests include a second cold admission using a reconstructed writer, loss before target admission, direct bypass, invalid coverage, bad source identity, wrong operation, bad signature and stale CAS. |
| +Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops policy API is undefined. Tests use nonzero restore baseline N=3 and frozen maximum M=7 for target and source-cut losses; covered-source and no-source cases also use N=3. No valid case passes a nil plan. They include a second cold admission using a reconstructed writer, loss before target admission, direct bypass, invalid coverage, bad source identity, wrong operation, bad signature and stale CAS. |
| ### Failing tests |
| @@ -152,7 +167,7 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| ```diff |
| --- /dev/null |
| +++ b/internal/hostbox/plan12_loss_test.go |
| -@@ -0,0 +1,154 @@ |
| +@@ -0,0 +1,175 @@ |
| +//go:build unix |
| + |
| +package hostbox |
| @@ -179,10 +194,10 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + return nil |
| +} |
| +func TestPlan12LossAdmission(t *testing.T) { |
| -+ plan12LossAdmissionCases(t, []string{"ok", "source-cut", "no-source", "clean-return", "sealed", "missing-owner", "missing-source", "wrong-source", "wrong-op", "wrong-signature", "stale", "bypass"}) |
| ++ plan12LossAdmissionCases(t, []string{"ok", "source-cut", "covered-source", "no-source", "clean-return", "sealed", "missing-owner", "missing-source", "wrong-source", "wrong-op", "wrong-signature", "stale", "restore-mismatch", "bypass"}) |
| +} |
| +func TestLossBeforeTargetAdmissionRecovers(t *testing.T) { |
| -+ plan12LossAdmissionCases(t, []string{"source-cut", "no-source"}) |
| ++ plan12LossAdmissionCases(t, []string{"source-cut", "covered-source", "no-source"}) |
| +} |
| +func plan12LossAdmissionCases(t *testing.T, modes []string) { |
| + for _, mode := range modes { |
| @@ -192,7 +207,7 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + o, err = hostclaim.Update(t.Context(), store, o, func(c *hostclaim.Claim) error { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindRestoring, ID: "op", Phase: hostclaim.PhaseProbing, Exec: "box", Participants: []string{"box"}, Journal: map[string]string{"loss.loss_1": "{}"}} |
| + c.Holder.Session, c.Holder.Generation, c.Holder.WriterBox = 2, "target-gen", "target" |
| -+ if mode == "source-cut" || mode == "no-source" { |
| ++ if mode == "source-cut" || mode == "covered-source" || mode == "no-source" { |
| + c.Holder.WriterBox = "box" |
| + } |
| + if mode == "no-source" { |
| @@ -215,7 +230,7 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + if mode == "source-cut" { |
| + r.WriteSources[0].Reason, r.WriteSources[0].Box = "source-cut", "box" |
| + } |
| -+ if mode == "no-source" { |
| ++ if mode == "no-source" || mode == "covered-source" { |
| + r.WriteSources = nil |
| + r.HolderSession = 0 |
| + } |
| @@ -231,9 +246,21 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + t.Fatal(err) |
| + } |
| + owner := &plan12LossOwner{} |
| -+ w := Writer{Claims: store, Box: "box", Region: "ie", AdmissionEvidence: func(context.Context, hostclaim.Observed, hostclaim.Admission, *genselect.Plan) error { return nil }, Loss: &LossAdmission{PublicKey: pub, Owner: owner}} |
| ++ w := Writer{Claims: store, Box: "box", Region: "ie", AdmissionEvidence: func(_ context.Context, obs hostclaim.Observed, admission hostclaim.Admission, baseline *genselect.Plan) error { |
| ++ if baseline == nil || baseline.MaxTXID != 3 { |
| ++ t.Fatal("restore baseline changed", baseline) |
| ++ } |
| ++ want := uint64(7) |
| ++ if mode == "no-source" || mode == "covered-source" || mode == "wrong-source" || hostclaim.OperationLossConsumed(obs.Claim, "loss_1") { |
| ++ want = 3 |
| ++ } |
| ++ if admission.RemoteMaxTXID != want { |
| ++ t.Fatalf("consumer maximum=%d want=%d", admission.RemoteMaxTXID, want) |
| ++ } |
| ++ return nil |
| ++ }, Loss: &LossAdmission{PublicKey: pub, Owner: owner}} |
| + w.Loss.Sources = func(_ context.Context, _ hostclaim.Observed, r hostclaim.LossRecord, _ hostclaim.Admission, _ *genselect.Plan) ([]hostclaim.ValidatedWriteSource, error) { |
| -+ if mode == "no-source" { |
| ++ if mode == "no-source" || mode == "covered-source" { |
| + return nil, nil |
| + } |
| + f := r.WriteSources[0] |
| @@ -252,7 +279,7 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + if mode == "wrong-signature" { |
| + w.Loss.PublicKey, _, _ = ed25519.GenerateKey(nil) |
| + } |
| -+ in := hostclaim.Admission{ObservedOpID: "op", ObservedPhase: hostclaim.PhaseProbing, ObservedSession: 2, ObservedGeneration: "target-gen", Box: "box", Mode: "cold", NewGeneration: "returned", Region: "ie", PredecessorRegion: "uk", RemoteMaxTXID: 999} |
| ++ in := hostclaim.Admission{ObservedOpID: "op", ObservedPhase: hostclaim.PhaseProbing, ObservedSession: 2, ObservedGeneration: "target-gen", Box: "box", Mode: "cold", NewGeneration: "returned", Region: "ie", PredecessorRegion: "uk", RemoteMaxTXID: 3} |
| + if mode == "no-source" { |
| + in.ObservedSession, in.ObservedGeneration = 0, "" |
| + } |
| @@ -265,13 +292,16 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + t.Fatal(err) |
| + } |
| + } |
| ++ if mode == "restore-mismatch" { |
| ++ in.RemoteMaxTXID = 4 |
| ++ } |
| + var admitted hostclaim.Observed |
| + if mode == "bypass" { |
| + admitted, err = hostclaim.Update(t.Context(), store, o, hostclaim.AdmitWriter(in)) |
| + } else { |
| -+ admitted, err = w.Admit(t.Context(), o, in, nil) |
| ++ admitted, err = w.Admit(t.Context(), o, in, &genselect.Plan{MaxTXID: 3}) |
| + } |
| -+ if mode != "ok" && mode != "source-cut" && mode != "no-source" { |
| ++ if mode != "ok" && mode != "source-cut" && mode != "covered-source" && mode != "no-source" { |
| + if err == nil { |
| + t.Fatal("accepted", mode) |
| + } |
| @@ -284,9 +314,15 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + t.Fatal(err) |
| + } |
| + a := admitted.Claim.Holder.Admissions[0] |
| -+ if a.LossID != "loss_1" || a.Coverage != "" || (mode != "no-source" && (a.WritesLost == nil || a.WritesLost.Session != 2 || a.WritesLost.RemoteMaxTXID != 7)) || owner.calls != 1 { |
| ++ if a.LossID != "loss_1" || a.Coverage != "" || (mode != "no-source" && mode != "covered-source" && (a.WritesLost == nil || a.WritesLost.Session != 2 || a.WritesLost.RemoteMaxTXID != 7)) || owner.calls != 1 { |
| + t.Fatalf("%+v owner=%d", a, owner.calls) |
| + } |
| ++ if mode == "covered-source" && (a.WritesLost == nil || a.WritesLost.RemoteMaxTXID != 3) { |
| ++ t.Fatalf("restore maximum lost: %+v", a) |
| ++ } |
| ++ if mode == "no-source" && a.WritesLost != nil { |
| ++ t.Fatalf("invented predecessor: %+v", a) |
| ++ } |
| + // Restart loses the writer object, but not the claim's consumption marker. |
| + restarted := Writer{Claims: store, Box: "box", Region: "ie", AdmissionEvidence: w.AdmissionEvidence} |
| + fresh, err := hostclaim.Read(t.Context(), store, o.Host) |
| @@ -296,7 +332,7 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + in.ObservedSession, in.ObservedGeneration, in.NewGeneration = fresh.Claim.Holder.Session, fresh.Claim.Holder.Generation, "second-cold" |
| + in.PredecessorRegion = "ie" |
| + in.LossID = "loss_1" |
| -+ again, err := restarted.Admit(t.Context(), fresh, in, nil) |
| ++ again, err := restarted.Admit(t.Context(), fresh, in, &genselect.Plan{MaxTXID: 3}) |
| + if err != nil { |
| + t.Fatal("second cold admission after restart", err) |
| + } |
| @@ -358,9 +394,89 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| <!-- apply: A implementation --> |
| ```diff |
| +--- a/internal/host/claimassembly.go |
| ++++ b/internal/host/claimassembly.go |
| +@@ -91,6 +91,7 @@ |
| + // ClaimAssembly owns the concrete wiring. Dependencies retain their contract |
| + // names so a missing implementation cannot silently become permissive policy. |
| + type ClaimAssembly struct { |
| ++ LossAdmission *hostbox.LossAdmission |
| + moveRead *claimMoveReadAccess |
| + wakeUnavailable error |
| + Journals *hostbox.JournalService |
| +@@ -176,7 +177,7 @@ |
| + a.Provision = a.Reconcile.Configure |
| + check := claimBoundIdentity(a.Identity.Verify, actor) |
| + data := hostbox.RequestStore{Store: a.Data, Check: check} |
| +- writer := hostbox.Writer{Claims: a.Claims, Data: data, Bucket: actor.Bucket, Region: actor.Region, Box: actor.Box, AdmissionEvidence: a.AdmissionEvidence, PublicationEvidence: a.PublicationEvidence, NewID: a.NewID} |
| ++ writer := hostbox.Writer{Loss: a.LossAdmission, Claims: a.Claims, Data: data, Bucket: actor.Bucket, Region: actor.Region, Box: actor.Box, AdmissionEvidence: a.AdmissionEvidence, PublicationEvidence: a.PublicationEvidence, NewID: a.NewID} |
| + a.bindClaimWakeStop() |
| + if local, ok := a.Local.(*productionClaimLocal); ok { |
| + if err := local.bind(ctx, &a); err != nil { |
| +--- a/internal/host/claimconstruction.go |
| ++++ b/internal/host/claimconstruction.go |
| +@@ -331,6 +331,9 @@ |
| + stop.Inventory = inventory |
| + stop.NewID = newID |
| + assembly := &ClaimAssembly{moveRead: owners.moveRead, wakeUnavailable: owners.wakeUnavailable, Journals: journals, cleanupEnrollment: owners.cleanupEnrollment, RetirementAuthority: owners.RetirementAuthority, SharedPaths: claimGlobalSharedPaths(r), Mode: r.Mode, Producers: producers, Launches: launches, ManagedRoots: roots, Reconcile: reconcile, RootJournal: launches, Controllers: owners.Controllers, Claims: store, Guard: guard, Identity: owners.Identity, Local: owners.Local, Policy: hostbox.Policy{Inputs: owners.InputPolicy, Moves: owners.MovesPolicy, Relocation: groupD.Relocation, Held: hostbox.RegistryHoldPolicy(r.Registry)}.WithRefusals(owners.PolicyRefusals...), Data: owners.Data, Wake: wake, Stop: stop, Evidence: owners.Evidence, LeaseSet: owners.LeaseSet, View: owners.View, Authorize: owners.Authorize, AdmissionEvidence: admissionEvidence, PublicationEvidence: publicationEvidence, NewID: newID} |
| ++ if movesC1 && owners.RetirementAuthority != nil { |
| ++ assembly.LossAdmission = &hostbox.LossAdmission{PublicKey: owners.RetirementAuthority.Directory.PublicKey, Owner: owners.Moves.LossRecovery, Sources: owners.Moves.LossSources} |
| ++ } |
| + var closeMu sync.Mutex |
| + closed := false |
| + assembly.Close = func(ctx context.Context) error { |
| +--- a/internal/host/claimowners.go |
| ++++ b/internal/host/claimowners.go |
| +@@ -39,13 +39,15 @@ |
| + // with CARLOS_MOVES_PHASE1 on. ProbeReady replaces the claim's probe |
| + // readiness check when set. |
| + type ClaimMovesOwners struct { |
| ++ LossRecovery hostclaim.LossRecoveryOwner |
| ++ LossSources hostbox.LossAdmissionSources |
| + ClaimKindOwners |
| + Operations OperationAttempts |
| + ProbeReady ClaimProbeReadiness |
| + } |
| + |
| + func (m ClaimMovesOwners) zero() bool { |
| +- return m.Inputs == nil && m.AdmissionEvidence == nil && m.PublicationEvidence == nil && m.Operations == nil && m.ProbeReady == nil |
| ++ return m.LossRecovery == nil && m.LossSources == nil && m.Inputs == nil && m.AdmissionEvidence == nil && m.PublicationEvidence == nil && m.Operations == nil && m.ProbeReady == nil |
| + } |
| + |
| + // ClaimRelocationOwners is region HA's owner set for the relocating kind |
| +--- a/internal/hostbox/writer.go |
| ++++ b/internal/hostbox/writer.go |
| +@@ -12,6 +12,7 @@ |
| + |
| + // Writer supplies transition adapters, not stop, certificate or marker evidence. |
| + type Writer struct { |
| ++ Loss *LossAdmission |
| + Claims release.Store |
| + Data genselect.Reader |
| + Bucket, Region, Box string |
| +@@ -33,6 +34,19 @@ |
| + } |
| + if w.Claims == nil || w.AdmissionEvidence == nil || w.Box == "" || w.Region == "" || observed.Absent || observed.Claim == nil || observed.Claim.Holder == nil || in.Box != w.Box || in.Region != w.Region { |
| + return hostclaim.Observed{}, ErrUnsupported |
| ++ } |
| ++ lossID, err := hostclaim.OperationLossID(observed.Claim) |
| ++ if err != nil { |
| ++ return hostclaim.Observed{}, err |
| ++ } |
| ++ if lossID != "" && !hostclaim.OperationLossConsumed(observed.Claim, lossID) { |
| ++ return w.admitLoss(ctx, observed, in, baseline, lossID) |
| ++ } |
| ++ if in.LossID != "" { |
| ++ if lossID == "" || in.LossID != lossID || !hostclaim.OperationLossConsumed(observed.Claim, lossID) { |
| ++ return hostclaim.Observed{}, hostclaim.ErrLossRecord |
| ++ } |
| ++ in.LossID = "" |
| + } |
| + if baseline != nil && baseline.MaxTXID != in.RemoteMaxTXID { |
| + return hostclaim.Observed{}, ErrChanged |
| --- /dev/null |
| +++ b/internal/hostbox/writer_loss.go |
| -@@ -0,0 +1,58 @@ |
| +@@ -0,0 +1,59 @@ |
| +package hostbox |
| + |
| +import ( |
| @@ -403,7 +519,11 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + if a := o.Claim.Holder.Admissions; len(a) > 0 { |
| + p = a[len(a)-1] |
| + } |
| -+ in.RemoteMaxTXID, in.AdditionalWritesLost = 0, nil |
| ++ // Select validates the restore point; loss sources describe independent baselines. |
| ++ if baseline == nil || baseline.MaxTXID != in.RemoteMaxTXID { |
| ++ return hostclaim.Observed{}, ErrChanged |
| ++ } |
| ++ in.AdditionalWritesLost = nil |
| + for _, s := range sources { |
| + if s.Box == p.Box && s.Session == p.Session && s.Generation == p.Generation { |
| + in.RemoteMaxTXID = s.RemoteMaxTXID |
| @@ -411,83 +531,11 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| + in.AdditionalWritesLost = append(in.AdditionalWritesLost, hostclaim.WritesLost{Box: s.Box, Session: s.Session, Generation: s.Generation, RemoteMaxTXID: s.RemoteMaxTXID}) |
| + } |
| + } |
| -+ if baseline != nil && baseline.MaxTXID != in.RemoteMaxTXID { |
| -+ return hostclaim.Observed{}, ErrChanged |
| -+ } |
| + if err := w.AdmissionEvidence(ctx, o, in, baseline); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + return hostclaim.AdmitWriterWithLoss(ctx, w.Claims, w.Loss.PublicKey, o, in, id, sources, w.Loss.Owner) |
| +} |
| ---- /dev/null |
| -+++ b/internal/hostclaim/operation_loss.go |
| -@@ -0,0 +1,36 @@ |
| -+package hostclaim |
| -+ |
| -+import ( |
| -+ "strconv" |
| -+ "strings" |
| -+) |
| -+ |
| -+// OperationLossID selects only a declaration journalled under this operation. |
| -+// Old claim-level LossRefs alone do not turn a later move into a loss recovery. |
| -+func OperationLossID(c *Claim) (string, error) { |
| -+ if c == nil { |
| -+ return "", ErrLossRecord |
| -+ } |
| -+ id := "" |
| -+ for key := range c.Op.Journal { |
| -+ if !strings.HasPrefix(key, "loss.") { |
| -+ continue |
| -+ } |
| -+ next := strings.TrimPrefix(key, "loss.") |
| -+ if c.Op.ID == "" || !safeLossID(next) || !lossRefPresent(c, next) || id != "" { |
| -+ return "", ErrLossRecord |
| -+ } |
| -+ id = next |
| -+ } |
| -+ return id, nil |
| -+} |
| -+ |
| -+// OperationLossConsumed survives admission-history trimming and process restart. |
| -+// Only the verified admission CAS records this operation-local marker. |
| -+func OperationLossConsumed(c *Claim, id string) bool { |
| -+ if c == nil || c.Holder == nil || id == "" { |
| -+ return false |
| -+ } |
| -+ n, err := strconv.ParseUint(c.Op.Journal["loss_admitted."+id], 10, 64) |
| -+ return err == nil && n > 0 && n <= c.Holder.Session |
| -+} |
| ---- a/internal/hostbox/writer.go |
| -+++ b/internal/hostbox/writer.go |
| -@@ -12,6 +12,7 @@ |
| - |
| - // Writer supplies transition adapters, not stop, certificate or marker evidence. |
| - type Writer struct { |
| -+ Loss *LossAdmission |
| - Claims release.Store |
| - Data genselect.Reader |
| - Bucket, Region, Box string |
| -@@ -33,6 +34,19 @@ |
| - } |
| - if w.Claims == nil || w.AdmissionEvidence == nil || w.Box == "" || w.Region == "" || observed.Absent || observed.Claim == nil || observed.Claim.Holder == nil || in.Box != w.Box || in.Region != w.Region { |
| - return hostclaim.Observed{}, ErrUnsupported |
| -+ } |
| -+ lossID, err := hostclaim.OperationLossID(observed.Claim) |
| -+ if err != nil { |
| -+ return hostclaim.Observed{}, err |
| -+ } |
| -+ if lossID != "" && !hostclaim.OperationLossConsumed(observed.Claim, lossID) { |
| -+ return w.admitLoss(ctx, observed, in, baseline, lossID) |
| -+ } |
| -+ if in.LossID != "" { |
| -+ if lossID == "" || in.LossID != lossID || !hostclaim.OperationLossConsumed(observed.Claim, lossID) { |
| -+ return hostclaim.Observed{}, hostclaim.ErrLossRecord |
| -+ } |
| -+ in.LossID = "" |
| - } |
| - if baseline != nil && baseline.MaxTXID != in.RemoteMaxTXID { |
| - return hostclaim.Observed{}, ErrChanged |
| --- a/internal/hostclaim/generation.go |
| +++ b/internal/hostclaim/generation.go |
| @@ -3,6 +3,7 @@ |
| @@ -534,56 +582,6 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| h := c.Holder |
| admitted := false |
| for _, record := range h.Admissions { |
| ---- a/internal/host/claimowners.go |
| -+++ b/internal/host/claimowners.go |
| -@@ -39,13 +39,15 @@ |
| - // with CARLOS_MOVES_PHASE1 on. ProbeReady replaces the claim's probe |
| - // readiness check when set. |
| - type ClaimMovesOwners struct { |
| -+ LossRecovery hostclaim.LossRecoveryOwner |
| -+ LossSources hostbox.LossAdmissionSources |
| - ClaimKindOwners |
| - Operations OperationAttempts |
| - ProbeReady ClaimProbeReadiness |
| - } |
| - |
| - func (m ClaimMovesOwners) zero() bool { |
| -- return m.Inputs == nil && m.AdmissionEvidence == nil && m.PublicationEvidence == nil && m.Operations == nil && m.ProbeReady == nil |
| -+ return m.LossRecovery == nil && m.LossSources == nil && m.Inputs == nil && m.AdmissionEvidence == nil && m.PublicationEvidence == nil && m.Operations == nil && m.ProbeReady == nil |
| - } |
| - |
| - // ClaimRelocationOwners is region HA's owner set for the relocating kind |
| ---- a/internal/host/claimassembly.go |
| -+++ b/internal/host/claimassembly.go |
| -@@ -91,6 +91,7 @@ |
| - // ClaimAssembly owns the concrete wiring. Dependencies retain their contract |
| - // names so a missing implementation cannot silently become permissive policy. |
| - type ClaimAssembly struct { |
| -+ LossAdmission *hostbox.LossAdmission |
| - moveRead *claimMoveReadAccess |
| - wakeUnavailable error |
| - Journals *hostbox.JournalService |
| -@@ -176,7 +177,7 @@ |
| - a.Provision = a.Reconcile.Configure |
| - check := claimBoundIdentity(a.Identity.Verify, actor) |
| - data := hostbox.RequestStore{Store: a.Data, Check: check} |
| -- writer := hostbox.Writer{Claims: a.Claims, Data: data, Bucket: actor.Bucket, Region: actor.Region, Box: actor.Box, AdmissionEvidence: a.AdmissionEvidence, PublicationEvidence: a.PublicationEvidence, NewID: a.NewID} |
| -+ writer := hostbox.Writer{Loss: a.LossAdmission, Claims: a.Claims, Data: data, Bucket: actor.Bucket, Region: actor.Region, Box: actor.Box, AdmissionEvidence: a.AdmissionEvidence, PublicationEvidence: a.PublicationEvidence, NewID: a.NewID} |
| - a.bindClaimWakeStop() |
| - if local, ok := a.Local.(*productionClaimLocal); ok { |
| - if err := local.bind(ctx, &a); err != nil { |
| ---- a/internal/host/claimconstruction.go |
| -+++ b/internal/host/claimconstruction.go |
| -@@ -331,6 +331,9 @@ |
| - stop.Inventory = inventory |
| - stop.NewID = newID |
| - assembly := &ClaimAssembly{moveRead: owners.moveRead, wakeUnavailable: owners.wakeUnavailable, Journals: journals, cleanupEnrollment: owners.cleanupEnrollment, RetirementAuthority: owners.RetirementAuthority, SharedPaths: claimGlobalSharedPaths(r), Mode: r.Mode, Producers: producers, Launches: launches, ManagedRoots: roots, Reconcile: reconcile, RootJournal: launches, Controllers: owners.Controllers, Claims: store, Guard: guard, Identity: owners.Identity, Local: owners.Local, Policy: hostbox.Policy{Inputs: owners.InputPolicy, Moves: owners.MovesPolicy, Relocation: groupD.Relocation, Held: hostbox.RegistryHoldPolicy(r.Registry)}.WithRefusals(owners.PolicyRefusals...), Data: owners.Data, Wake: wake, Stop: stop, Evidence: owners.Evidence, LeaseSet: owners.LeaseSet, View: owners.View, Authorize: owners.Authorize, AdmissionEvidence: admissionEvidence, PublicationEvidence: publicationEvidence, NewID: newID} |
| -+ if movesC1 && owners.RetirementAuthority != nil { |
| -+ assembly.LossAdmission = &hostbox.LossAdmission{PublicKey: owners.RetirementAuthority.Directory.PublicKey, Owner: owners.Moves.LossRecovery, Sources: owners.Moves.LossSources} |
| -+ } |
| - var closeMu sync.Mutex |
| - closed := false |
| - assembly.Close = func(ctx context.Context) error { |
| --- a/internal/hostclaim/lost.go |
| +++ b/internal/hostclaim/lost.go |
| @@ -152,14 +152,17 @@ |
| @@ -608,6 +606,45 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| key := fmt.Sprintf("%s\x00%s\x00%d", additional.Box, additional.Generation, additional.Session) |
| if _, ok := want[key]; !ok { |
| --- /dev/null |
| ++++ b/internal/hostclaim/operation_loss.go |
| +@@ -0,0 +1,36 @@ |
| ++package hostclaim |
| ++ |
| ++import ( |
| ++ "strconv" |
| ++ "strings" |
| ++) |
| ++ |
| ++// OperationLossID selects only a declaration journalled under this operation. |
| ++// Old claim-level LossRefs alone do not turn a later move into a loss recovery. |
| ++func OperationLossID(c *Claim) (string, error) { |
| ++ if c == nil { |
| ++ return "", ErrLossRecord |
| ++ } |
| ++ id := "" |
| ++ for key := range c.Op.Journal { |
| ++ if !strings.HasPrefix(key, "loss.") { |
| ++ continue |
| ++ } |
| ++ next := strings.TrimPrefix(key, "loss.") |
| ++ if c.Op.ID == "" || !safeLossID(next) || !lossRefPresent(c, next) || id != "" { |
| ++ return "", ErrLossRecord |
| ++ } |
| ++ id = next |
| ++ } |
| ++ return id, nil |
| ++} |
| ++ |
| ++// OperationLossConsumed survives admission-history trimming and process restart. |
| ++// Only the verified admission CAS records this operation-local marker. |
| ++func OperationLossConsumed(c *Claim, id string) bool { |
| ++ if c == nil || c.Holder == nil || id == "" { |
| ++ return false |
| ++ } |
| ++ n, err := strconv.ParseUint(c.Op.Journal["loss_admitted."+id], 10, 64) |
| ++ return err == nil && n > 0 && n <= c.Holder.Session |
| ++} |
| +--- /dev/null |
| +++ b/internal/hostops/move_loss_admission.go |
| @@ -0,0 +1,49 @@ |
| +package hostops |
| @@ -686,20 +723,56 @@ Expected first failure: `LossAdmission`, `OperationLossConsumed` or the hostops |
| ## Task B: Declare and release a fenced target atomically (M12, J5) |
| -`VerifyMoveTargetFence` verifies the immutable fence against the roster, target enrollment, region and current operation. `ReleaseFencedTarget` consumes its typed result and queues the target location, staging and optional partial return lane in one CAS. It shares the park-queue implementation with `ReleaseLostTarget`, removes the target from participants and journals the consumed fence. `ReturnToSource` accepts that marker. |
| +`VerifyMoveTargetFence` verifies the immutable fence against the roster, target enrollment, region and current operation. `ReleaseFencedTarget` consumes its typed result and queues the target location, staging and optional partial return lane in one CAS. It shares the park-queue implementation with `ReleaseLostTarget`, removes the target from participants and journals the consumed fence. The same CAS retains Lost, LossRefs and AutomaticLoss independently of the operation, including last admission, original observation time, regional obligations, unknown timers and pending reporting. It does not discharge local storage. `ReturnToSource` accepts the consumed evidence. |
| + |
| +The console action lives on `MoveRecoveryGrant`: only a live `RunRecovery` grant bound to the current record can invoke it. Its successive passes write the fence rollback decision, release the claim, and finish the target release record, with one mutation per callback. It checks ETag and `record_seq` again after fence reads and immediately before the claim CAS. The verified fence's original observation time bounds new parks. An existing rollback keeps its first reason; LossRef identifies a fence decision. Completion and handover use the consumed claim evidence without live-roster checks. A clean released record with the complete canonical target report permits the missing claim release directly. Moves' `Recovery.Moves` hook calls this method on each pass. The source gates on the claim's released phase. Dirty target records refuse this action and require J4 loss. The box-side capability verifies a journalled fence with the same roster authority and an enrollment-bound identity check. |
| -The console action lives on `MoveRecoveryGrant`: only a live `RunRecovery` grant bound to the current record can invoke it. Its successive passes write the fence rollback decision, release the claim, and finish the target release record, with one mutation per callback. It checks ETag and `record_seq` again after fence reads and immediately before the claim CAS. The persisted `LostAt` from the fence decision bounds the parks, even when an earlier rollback already existed. Moves' `Recovery.Moves` hook calls this method on each pass. The source gates on the claim's released phase. Dirty target records refuse this action and require J4 loss. The box-side capability verifies a journalled fence with the same roster authority and an enrollment-bound identity check. |
| +The grant also supplies `StampLostMoveTargetParked`; the console calls it directly after all target chores finish. The unused abort-complete action and obsolete closure are removed. The recovery pass materializes retained automatic evidence as a signed loss record before marking it reported, even after operation replacement. |
| Focused command: `go test ./internal/hostclaim ./internal/hostops -run 'TestPlan12.*Fence|TestReleaseLostTarget|TestMoveRecovery' -count=1`. |
| -Expected first failure: the verifier, release transition and grant method are undefined. Regressions cover both kinds, wrong tuple, absent proof, committed landing, stale CAS, a sequence bump during fence verification, an escaped grant, participant removal, parks, partial-return scope, record-first landing and restoring recovery, and replay. No release marker acknowledges a park. |
| +Expected first failure: the verifier, release transition and grant method are undefined. Regressions cover both kinds, wrong tuple, absent proof, committed landing, stale CAS, a sequence bump during fence verification, an escaped grant, participant removal, parks, partial-return scope, record-first landing and restoring recovery, and replay. Tests continue through lost-target stamping, record closure, claim release and signed loss reporting. They reject stamping while target chores remain and preserve installed targets' local obligations. The consumed-fence replay runs with a rejecting roster authority. No release marker acknowledges a park. |
| ### Failing tests |
| <!-- apply: B tests --> |
| ```diff |
| ---- /dev/null |
| -+++ b/internal/hostclaim/plan12_fence_test.go |
| +--- a/internal/console/claim_invariant_moves_test.go |
| ++++ b/internal/console/claim_invariant_moves_test.go |
| +@@ -140,11 +140,11 @@ |
| + // recovery pass skips move claims as main does. |
| + func TestClaimMovesConsoleBindsTheMovePass(t *testing.T) { |
| + off := newClaimCase(t).srv |
| +- if off.cfg.ClaimMoveRecovery != nil || off.cfg.HostClaimRecovery == nil || off.cfg.HostClaimRecovery.Moves != nil || off.stampLostMoveTarget != nil { |
| ++ if off.cfg.ClaimMoveRecovery != nil || off.cfg.HostClaimRecovery == nil || off.cfg.HostClaimRecovery.Moves != nil { |
| + t.Fatal("a moves-off console bound the move pass") |
| + } |
| + on := newClaimCaseWith(t, claimMovesOn).srv |
| +- if on.cfg.ClaimMoveRecovery == nil || on.cfg.HostClaimRecovery == nil || on.cfg.HostClaimRecovery.Moves == nil || on.stampLostMoveTarget == nil { |
| ++ if on.cfg.ClaimMoveRecovery == nil || on.cfg.HostClaimRecovery == nil || on.cfg.HostClaimRecovery.Moves == nil { |
| + t.Fatal("a moves-on console did not bind the move pass") |
| + } |
| + if on.cfg.HostClaimRecovery.MoveFenceAuthority == nil { |
| +--- a/internal/console/movesrecovery_test.go |
| ++++ b/internal/console/movesrecovery_test.go |
| +@@ -192,15 +192,6 @@ |
| + t.Helper() |
| + w := &movePassWorld{fx: fx, id: id, clock: time.Now().UTC()} |
| + fx.srv.movesClock = func() time.Time { w.mu.Lock(); defer w.mu.Unlock(); return w.clock } |
| +- // As ConfigureClaimRuntime binds it: the lost target's stamp over the |
| +- // conditional record store under the fixture's data boundary. |
| +- records, err := hostops.NewConditionalRecordStore(fx.rel, fx.pub) |
| +- if err != nil { |
| +- t.Fatal(err) |
| +- } |
| +- fx.srv.stampLostMoveTarget = func(ctx context.Context, host string, now time.Time) error { |
| +- return hostops.StampLostMoveTargetParked(ctx, records, host, now) |
| +- } |
| + alerting.Arm(&alerting.Sinks{WebhookURL: "http://pages.test/hook", Post: func(_ string, body []byte) error { |
| + var p pagePayload |
| + if err := json.Unmarshal(body, &p); err != nil { |
| +--- /dev/null |
| ++++ b/internal/hostclaim/plan12_fence_test.go |
| @@ -0,0 +1,113 @@ |
| +package hostclaim |
| + |
| @@ -752,7 +825,7 @@ Expected first failure: the verifier, release transition and grant method are un |
| + f, err := VerifyMoveTargetFence(t.Context(), s, o, ref, a) |
| + n := 0 |
| + scope := func(loc string) ParkScope { |
| -+ return ParkScope{Location: loc, Incarnation: o.Claim.Holder.Incarnation, EvidenceAt: time.Now().UTC(), RequestLimit: 2 * time.Minute} |
| ++ return ParkScope{Location: loc, Incarnation: o.Claim.Holder.Incarnation, EvidenceAt: f.ObservedAt(), RequestLimit: 2 * time.Minute} |
| + } |
| + r := FencedTargetRelease{OpID: o.Claim.Op.ID, RollbackID: lostRB, Target: "adi-knitting", TargetRegion: "ie", SourceRegion: "uk", Location: scope(o.Claim.Holder.Location), Lane: scope("moves/" + o.Host + "/" + o.Claim.Op.ID), Fence: f, NewID: func() string { n++; return "fence-" + strconv.Itoa(n) }} |
| + if mode == "wrong-op" { |
| @@ -814,16 +887,76 @@ Expected first failure: the verifier, release transition and grant method are un |
| + t.Fatal(err) |
| + } |
| +} |
| +--- a/internal/hostops/move_recovery_grant_test.go |
| ++++ b/internal/hostops/move_recovery_grant_test.go |
| +@@ -411,55 +411,6 @@ |
| + } |
| + if err := r.Pass(saved); !errors.Is(err, ErrContract) || calls != 1 { |
| + t.Fatalf("retained pass context: %v calls=%d", err, calls) |
| +- } |
| +-} |
| +- |
| +-func TestMoveRecoveryAbortCloseFreshGrantThenRelease(t *testing.T) { |
| +- w := newC7World(t, func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) { |
| +- c.Op.Kind, c.Op.Phase, c.Op.Exec = hostclaim.KindMoving, hostclaim.PhaseAborting, "source" |
| +- c.Obligations = nil |
| +- r.Move.State = provision.MoveAborting |
| +- }) |
| +- ctx := context.Background() |
| +- run := func(action func(context.Context, *MoveRecoveryGrant) error) error { |
| +- return w.control.RunRecovery(ctx, func(ctx context.Context) error { return runMoveRecovery(ctx, w.claim(t), action) }) |
| +- } |
| +- before := *w.claim(t).Claim.Holder |
| +- if err := run(func(ctx context.Context, g *MoveRecoveryGrant) error { |
| +- if err := g.Record(ctx, RecoveryMoveAbortComplete); err != nil { |
| +- return err |
| +- } |
| +- // Closing spends the grant and advances record_seq. A new grant is required. |
| +- if err := g.Claim(ctx, RecoveryMoveRelease); !errors.Is(err, ErrPending) { |
| +- t.Fatalf("spent close grant = %v", err) |
| +- } |
| +- return nil |
| +- }); err != nil { |
| +- t.Fatal(err) |
| +- } |
| +- rec, err := provision.GetRecord(ctx, w.raw, "bab", "app", w.host) |
| +- if err != nil { |
| +- t.Fatal(err) |
| +- } |
| +- if rec.Move != nil || rec.LastMove == nil || rec.LastMove.ID != "move" || rec.LastMove.Outcome != "aborted" { |
| +- t.Fatalf("abort not closed: %+v", rec) |
| +- } |
| +- if c := w.claim(t).Claim; c.Op.Kind != hostclaim.KindMoving || c.Op.Phase != hostclaim.PhaseAborting { |
| +- t.Fatal("close released claim") |
| +- } |
| +- // A later recovery pass mints a new grant over the closed record. |
| +- if err := run(func(ctx context.Context, g *MoveRecoveryGrant) error { return g.Claim(ctx, RecoveryMoveRelease) }); err != nil { |
| +- t.Fatal(err) |
| +- } |
| +- c := w.claim(t).Claim |
| +- if c.Op.Kind != hostclaim.KindServing || c.Op.Exec != "source" { |
| +- t.Fatalf("abort not released: %+v", c.Op) |
| +- } |
| +- if c.Holder.Incarnation != before.Incarnation || c.Holder.Stamp != before.Stamp || c.Holder.Session != before.Session || c.Holder.Generation != before.Generation { |
| +- t.Fatal("release changed writer identity") |
| +- } |
| +- if err := hostclaim.RequireReleasedMove("move")(c); err != nil { |
| +- t.Fatal(err) |
| + } |
| + } |
| + |
| --- /dev/null |
| +++ b/internal/hostops/plan12_fence_test.go |
| -@@ -0,0 +1,129 @@ |
| +@@ -0,0 +1,250 @@ |
| +package hostops |
| + |
| +import ( |
| + "context" |
| ++ "crypto/ed25519" |
| + "encoding/json" |
| + "errors" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/moves" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/termfence" |
| @@ -833,10 +966,14 @@ Expected first failure: the verifier, release transition and grant method are un |
| + |
| +type plan12BumpFence struct { |
| + c7ReplayAuthority |
| -+ bump func() |
| ++ bump func() |
| ++ reject bool |
| +} |
| + |
| +func (a *plan12BumpFence) ValidateFence(ctx context.Context, f termfence.Record) (hostclaim.FenceBinding, error) { |
| ++ if a.reject { |
| ++ return hostclaim.FenceBinding{}, hostclaim.ErrFence |
| ++ } |
| + if a.bump != nil { |
| + fn := a.bump |
| + a.bump = nil |
| @@ -845,7 +982,7 @@ Expected first failure: the verifier, release transition and grant method are un |
| + return a.c7ReplayAuthority.ValidateFence(ctx, f) |
| +} |
| +func TestPlan12RecoveryFenceGrant(t *testing.T) { |
| -+ for _, mode := range []string{"ok", "from-landing", "from-restoring", "changed-sequence", "dirty", "escaped", "wrong-ref"} { |
| ++ for _, mode := range []string{"ok", "admitted", "installed", "from-landing", "from-restoring", "record-released", "changed-sequence", "dirty", "escaped", "wrong-ref"} { |
| + t.Run(mode, func(t *testing.T) { |
| + w := newC7World(t, func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) { |
| + c.Op.Kind, c.Op.Phase, c.Op.Exec = hostclaim.KindLanding, hostclaim.PhaseProbing, "source" |
| @@ -857,7 +994,17 @@ Expected first failure: the verifier, release transition and grant method are un |
| + r.Move.From, r.Move.To, r.Move.State = "uk", "ie", provision.MoveRollingBack |
| + at := r.CreatedAt |
| + r.Move.Rollback = &provision.MoveRollback{ID: "rb_abcdefghijklmnopqrstuvwxyz", Reason: provision.RollbackTargetFenced, Evidence: "control/box-fences/" + strings.Repeat("a", 32) + ".json", RequestedAt: &at, TargetPhase: provision.RbTargetLost, LossRef: "control/box-fences/" + strings.Repeat("a", 32) + ".json", LostAt: &at, SourcePhase: provision.RbSourceWaiting, Dirty: mode == "dirty"} |
| -+ if mode == "from-restoring" { |
| ++ if mode == "admitted" || mode == "installed" { |
| ++ c.Holder.Session = 7 |
| ++ c.Holder.Stamp = 7 |
| ++ c.Holder.Generation = "target-gen" |
| ++ c.Holder.WriterBox = "source" |
| ++ c.Holder.Admissions = []hostclaim.AdmissionRecord{{Session: 7, Box: "source", Generation: "target-gen", Mode: "cold", Region: "ie"}} |
| ++ } |
| ++ if mode == "installed" { |
| ++ c.Inventory = map[string]hostclaim.InventoryEntry{"source": {Layout: "flat", Paths: map[string]uint64{"app.db": 7}}} |
| ++ } |
| ++ if mode == "from-restoring" || mode == "record-released" { |
| + c.Op.Kind, c.Op.Phase = hostclaim.KindRestoring, hostclaim.PhaseReleasing |
| + c.Op.Intent["rollback"] = r.Move.Rollback.ID |
| + c.Op.Journal[hostclaim.JournalTargetStamp] = "7" |
| @@ -866,12 +1013,24 @@ Expected first failure: the verifier, release transition and grant method are un |
| + r.Move.Rollback.LostAt = nil |
| + r.Move.Rollback.Reason = provision.RollbackLandingDeadline |
| + } |
| ++ if mode == "record-released" { |
| ++ r.Move.Rollback.TargetPhase = provision.RbTargetReleased |
| ++ r.Move.Rollback.Evidence = moves.StoppedKey(moves.ReportPrefix("bab", "app", c.Host, c.Op.ID), "target") |
| ++ } |
| + if mode == "from-landing" { |
| + r.Region = "ie" |
| + r.Move.State = provision.MoveLanding |
| + r.Move.Rollback = nil |
| + } |
| + }) |
| ++ if mode == "record-released" { |
| ++ c := w.claim(t).Claim |
| ++ report := moves.StoppedReport{V: 1, Role: "target", Box: "source", Host: w.host, OpID: c.Op.ID, Incarnation: c.Holder.Incarnation, CheckedAt: w.now, Steps: map[string]moves.Check{moves.StepTree: {OK: true}, moves.StepStray: {OK: true}, moves.StepFDs: {OK: true}, moves.StepLease: {OK: true}, moves.StepReplica: {OK: true}}} |
| ++ body, _ := json.Marshal(report) |
| ++ if _, err := w.raw.Put(t.Context(), moves.StoppedKey(moves.ReportPrefix("bab", "app", w.host, c.Op.ID), "target"), body, ""); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ } |
| + authority := &plan12BumpFence{c7ReplayAuthority: w.authority} |
| + if mode == "changed-sequence" { |
| + authority.bump = func() { |
| @@ -904,7 +1063,7 @@ Expected first failure: the verifier, release transition and grant method are un |
| + if mode == "escaped" { |
| + err = escaped.ReleaseFencedTarget(t.Context(), ref) |
| + } |
| -+ if mode != "ok" && mode != "from-landing" && mode != "from-restoring" { |
| ++ if mode != "ok" && mode != "admitted" && mode != "installed" && mode != "from-landing" && mode != "from-restoring" && mode != "record-released" { |
| + if err == nil { |
| + t.Fatal("accepted", mode) |
| + } |
| @@ -925,9 +1084,25 @@ Expected first failure: the verifier, release transition and grant method are un |
| + } |
| + } |
| + o := w.claim(t) |
| -+ if o.Claim.Op.Phase != hostclaim.PhaseReleased || len(o.Claim.Op.Participants) != 1 || len(o.Claim.Obligations) != 2 { |
| ++ if o.Claim.Op.Phase != hostclaim.PhaseReleased || len(o.Claim.Op.Participants) != 1 || len(o.Claim.Obligations) != func() int { |
| ++ if mode == "installed" { |
| ++ return 3 |
| ++ } |
| ++ return 2 |
| ++ }() { |
| + t.Fatal(o.Claim) |
| + } |
| ++ if len(o.Claim.Lost) != 1 || len(o.Claim.LossRefs) != 1 || len(o.Claim.EvidenceRefs) != 1 || o.Claim.EvidenceRefs[0].Reported { |
| ++ t.Fatal("missing pending automatic loss", o.Claim) |
| ++ } |
| ++ if mode == "admitted" || mode == "installed" { |
| ++ l := o.Claim.EvidenceRefs[0] |
| ++ if l.Session != 7 || l.Generation != "target-gen" || l.LastAdmission == nil || l.LastAdmission.Covered { |
| ++ t.Fatal("lost writer provenance", l) |
| ++ } |
| ++ } |
| ++ // A retired target or witness no longer validates against today's roster. |
| ++ authority.reject = true |
| + before, _ := json.Marshal(o.Claim.Obligations) |
| + if err := run(); err != nil { |
| + t.Fatal("replay", err) |
| @@ -937,12 +1112,89 @@ Expected first failure: the verifier, release transition and grant method are un |
| + t.Fatal("requeued parks") |
| + } |
| + rec, err := provision.GetRecord(t.Context(), w.raw, "bab", "app", w.host) |
| -+ if err != nil || rec.Move.Rollback.TargetPhase != provision.RbTargetReleased || rec.Move.Rollback.Evidence != ref.Key { |
| ++ if err != nil || rec.Move.Rollback.TargetPhase != provision.RbTargetReleased || (mode != "record-released" && rec.Move.Rollback.Evidence != ref.Key) { |
| + t.Fatal(rec, err) |
| + } |
| + if err := run(); err != nil { |
| + t.Fatal("completed replay", err) |
| + } |
| ++ if mode == "from-restoring" && rec.Move.Rollback.Reason != provision.RollbackLandingDeadline { |
| ++ t.Fatal("rewrote first reason") |
| ++ } |
| ++ act := func(fn func(context.Context, *MoveRecoveryGrant) error) error { |
| ++ return w.control.RunRecovery(t.Context(), func(ctx context.Context) error { return runMoveRecovery(ctx, w.claim(t), fn, authority) }) |
| ++ } |
| ++ if err := act(func(ctx context.Context, g *MoveRecoveryGrant) error { return g.StampLostMoveTargetParked(ctx) }); err == nil { |
| ++ t.Fatal("stamped with parks owed") |
| ++ } |
| ++ current := w.claim(t) |
| ++ for _, ob := range current.Claim.Obligations { |
| ++ var e error |
| ++ current, e = hostclaim.Update(t.Context(), w.raw, current, hostclaim.CompleteObligation(ob.Op, ob.ID)) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ } |
| ++ if err := act(func(ctx context.Context, g *MoveRecoveryGrant) error { return g.StampLostMoveTargetParked(ctx) }); err != nil { |
| ++ t.Fatal("stamp", err) |
| ++ } |
| ++ // Source opening is owned by the source; seed its completed evidence. |
| ++ current = w.claim(t) |
| ++ if _, err := hostclaim.Update(t.Context(), w.raw, current, func(c *hostclaim.Claim) error { |
| ++ c.Op.Phase = hostclaim.PhaseOpened |
| ++ c.Op.Exec = c.Op.Intent["source"] |
| ++ return nil |
| ++ }); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ rr, e := ReadRecord(t.Context(), w.raw, w.key) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ decoded, e := provision.DecodeRecord(rr.Body) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ rec = &decoded |
| ++ rec.Move.Rollback.OpenedAt, rec.Move.Rollback.RecoveredAt = &w.now, &w.now |
| ++ body, _ := json.Marshal(rec) |
| ++ if _, _, e = provision.WriteRecord(provision.WithRecordSequence(t.Context()), w.raw, w.key, rr.Body, body, rr.ETag); e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ if err := act(func(ctx context.Context, g *MoveRecoveryGrant) error { |
| ++ return g.Record(ctx, RecoveryMoveRolledBackClose) |
| ++ }); err != nil { |
| ++ t.Fatal("close", err) |
| ++ } |
| ++ if err := act(func(ctx context.Context, g *MoveRecoveryGrant) error { return g.Claim(ctx, RecoveryMoveRelease) }); err != nil { |
| ++ t.Fatal("release", err) |
| ++ } |
| ++ afterClose := w.claim(t).Claim |
| ++ if afterClose.Op.Kind != hostclaim.KindServing || len(afterClose.EvidenceRefs) != 1 || len(afterClose.LossRefs) != 1 { |
| ++ t.Fatal("loss evidence erased", afterClose) |
| ++ } |
| ++ // Reporting works after operation replacement and needs no live roster. |
| ++ pub, priv, e := ed25519.GenerateKey(nil) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ w.control.pub = pub |
| ++ if err := w.control.RunRecovery(t.Context(), func(ctx context.Context) error { |
| ++ return w.control.ReportAutomaticLosses(ctx, release.RootSigner(priv), w.host) |
| ++ }); err != nil { |
| ++ t.Fatal("report", err) |
| ++ } |
| ++ l := w.claim(t).Claim.EvidenceRefs[0] |
| ++ reported, e := hostclaim.ReadLossRecord(t.Context(), w.raw, pub, l.ID) |
| ++ if e != nil || reported.Record.State != "committed" || !reported.Record.AlertsPending || reported.Record.Automatic == nil || reported.Record.Automatic.Fence != ref || !l.Reported { |
| ++ t.Fatal("durable reporting", reported, e) |
| ++ } |
| ++ if err := w.control.RunRecovery(t.Context(), func(ctx context.Context) error { |
| ++ return w.control.ReportAutomaticLosses(ctx, release.RootSigner(priv), w.host) |
| ++ }); err != nil { |
| ++ t.Fatal("report replay", err) |
| ++ } |
| ++ |
| + }) |
| + } |
| +} |
| @@ -952,312 +1204,236 @@ Expected first failure: the verifier, release transition and grant method are un |
| <!-- apply: B implementation --> |
| ```diff |
| +--- a/internal/console/claim_runtime.go |
| ++++ b/internal/console/claim_runtime.go |
| +@@ -115,16 +115,12 @@ |
| + // switch on, and only now that every box close is record first |
| + // (ruling M4, P3-6; TestClaimMoveRecoveryWiringWaitsForRecordFirstOwners). |
| + s.cfg.ClaimMoveRecovery = s.movesRecovery |
| +- // The lost target's target_parked_at (p10 Task G) is written |
| +- // outside the grant by the lifecycle record writer, which the |
| +- // conditional record store carries and the data boundary (an app |
| +- // permit's) does not. The pass gets this one transition, never |
| +- // the store. |
| +- s.stampLostMoveTarget = func(ctx context.Context, host string, now time.Time) error { |
| +- return hostops.StampLostMoveTargetParked(ctx, records, host, now) |
| +- } |
| ++ |
| + } |
| + s.cfg.HostClaimRecovery = &hostops.Recovery{ |
| ++ ReportAutomaticLosses: func(ctx context.Context, host string) error { |
| ++ return a.Control.ReportAutomaticLosses(ctx, a.Signer, host) |
| ++ }, |
| + Moves: s.cfg.ClaimMoveRecovery, |
| + MoveFenceAuthority: hostfence.ClaimFenceAuthority{Directory: a.Directory, Now: c.Now}, |
| + Controller: c, |
| +--- a/internal/console/movesrecovery.go |
| ++++ b/internal/console/movesrecovery.go |
| +@@ -8,7 +8,6 @@ |
| + "fmt" |
| + "log" |
| + "os" |
| +- "slices" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/alerting" |
| +@@ -51,7 +50,7 @@ |
| + // deadlines is rolled back by the record (J11, J9's recorded reason). |
| + // 6. A lost target released with nothing of its box or region owed: |
| + // target_parked_at, through hostops.StampLostMoveTargetParked over the |
| +-// conditional record store (stampLostMoveTarget), outside the grant |
| ++// record-sequenced recovery grant (StampLostMoveTargetParked) |
| + // (p10 Task G). |
| + // 7. A close whose stamps are in and whose op owes nothing: the record's |
| + // close (close, rolled-back-close or revert-close). The boxes close on |
| +@@ -196,11 +195,7 @@ |
| + case m.State == provision.MoveLanding && op.Kind == hostclaim.KindLanding && op.Phase != hostclaim.PhaseCommitted && moveDeadlinePassed(m, in.Now): |
| + d.Action, d.Record = moveActRecord, hostops.RecoveryMoveDeadlineRollback |
| + case m.State == provision.MoveRollingBack && rb != nil && rb.TargetPhase == provision.RbTargetReleased && !rb.Dirty && rb.TargetParkedAt == nil && |
| +- (op.Kind == hostclaim.KindRestoring || op.Kind == hostclaim.KindLanding) && m.To != "" && op.Intent["target"] != "" && slices.Contains(c.Lost, op.Intent["target"]) && |
| +- owedUnderOp(c, op.ID, func(o hostclaim.Obligation) bool { |
| +- t := op.Intent["target"] |
| +- return o.Box == t || o.Owner == "box:"+t || o.Region == m.To || o.Owner == "region:"+m.To |
| +- }) == 0: |
| ++ hostclaim.LostMoveTargetParked(c, m.To): |
| + d.Action = moveActStampLost |
| + case owed > 0: |
| + case m.State == provision.MoveCommitted && m.OpenedAt != nil && m.SourceParkedAt != nil && rec.Region == m.To && op.Kind == hostclaim.KindLanding && op.Phase == hostclaim.PhaseCommitted: |
| +@@ -333,10 +328,7 @@ |
| + case moveActRecord: |
| + return g.Record(ctx, d.Record) |
| + case moveActStampLost: |
| +- if s.stampLostMoveTarget == nil { |
| +- return hostops.ErrContract |
| +- } |
| +- return s.stampLostMoveTarget(ctx, o.Host, now) |
| ++ return g.StampLostMoveTargetParked(ctx) |
| + case moveActClose: |
| + if !s.moveCloseOverdue(o, d.Record, now) { |
| + return nil |
| +--- a/internal/console/server.go |
| ++++ b/internal/console/server.go |
| +@@ -833,13 +833,10 @@ |
| + movesClock func() time.Time |
| + moveCloseMu sync.Mutex |
| + moveCloseSeen map[string]time.Time |
| +- // stampLostMoveTarget is hostops.StampLostMoveTargetParked over the |
| +- // conditional record store, bound by ConfigureClaimRuntime with the |
| +- // moves switch on; nil otherwise. |
| +- stampLostMoveTarget func(ctx context.Context, host string, now time.Time) error |
| +- dbHandleKey []byte |
| +- cfg Config |
| +- store *Store |
| ++ |
| ++ dbHandleKey []byte |
| ++ cfg Config |
| ++ store *Store |
| + // appRegionOf and beforeClaimInsert are TEST SEAMS (nil in |
| + // production): a store failure on the app-region read (C20), and a |
| + // competing insert landing after the region was resolved. |
| --- /dev/null |
| -+++ b/internal/hostclaim/move_target_fence.go |
| -@@ -0,0 +1,81 @@ |
| -+package hostclaim |
| ++++ b/internal/host/claimmovefence.go |
| +@@ -0,0 +1,41 @@ |
| ++package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| ++ "github.com/carlosframework/platform/internal/hostbox" |
| ++ "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/hostfence" |
| ++ "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/release" |
| ++ "time" |
| +) |
| + |
| -+const journalFencedTargetReleased = "fenced_target_released" |
| -+ |
| -+// VerifyMoveTargetFence supplies a target-bound capability without writing the claim. |
| -+func VerifyMoveTargetFence(ctx context.Context, s release.Store, o Observed, ref FenceRef, authority FenceAuthority) (ReturnFence, error) { |
| -+ if o.Absent || o.Claim == nil || o.Claim.Holder == nil || o.ETag == "" { |
| -+ return ReturnFence{}, ErrStaleOp |
| -+ } |
| -+ c := o.Claim |
| -+ if (c.Op.Kind != KindLanding && c.Op.Kind != KindRestoring) || c.Op.Phase == PhaseCommitted || c.Op.ID == "" || c.Op.Intent["target"] == c.Op.Intent["source"] || c.Op.Intent["target_enrollment"] != ref.Enrollment { |
| -+ return ReturnFence{}, ErrWrongState |
| -+ } |
| -+ f, err := VerifyFence(ctx, s, authority, ref) |
| -+ if err != nil { |
| -+ return ReturnFence{}, err |
| -+ } |
| -+ if f.Box() != c.Op.Intent["target"] || f.Region() != c.Op.Intent["to"] { |
| -+ return ReturnFence{}, ErrFence |
| -+ } |
| -+ raw, err := json.Marshal(ref) |
| -+ if err != nil { |
| -+ return ReturnFence{}, err |
| ++func (p ClaimProviders) VerifyMoveTargetFence(ctx context.Context, o hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| ++ if p.moveTargetFence == nil { |
| ++ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| + } |
| -+ return ReturnFence{op: c.Op.ID, box: f.Box(), ref: string(raw)}, nil |
| -+} |
| -+ |
| -+// FencedTargetRelease has the same scopes as loss release, with verified fence authority. |
| -+type FencedTargetRelease struct { |
| -+ OpID, RollbackID, Target, TargetRegion, SourceRegion string |
| -+ Location, Lane ParkScope |
| -+ ReturnLane *ParkScope |
| -+ NewID func() string |
| -+ Fence ReturnFence |
| ++ return p.moveTargetFence(ctx, o) |
| +} |
| -+ |
| -+// ReleaseFencedTarget declares and consumes the fence in one CAS, including all parks. |
| -+func ReleaseFencedTarget(r FencedTargetRelease) func(*Claim) error { |
| -+ return func(c *Claim) error { |
| -+ if err := RequireOp(r.OpID)(c); err != nil { |
| -+ return err |
| -+ } |
| -+ if c.Holder == nil || !rollbackIDShape.MatchString(r.RollbackID) || r.Target == "" || r.NewID == nil || r.SourceRegion == "" || r.SourceRegion == r.TargetRegion { |
| -+ return ErrMalformed |
| -+ } |
| -+ f := r.Fence |
| -+ if f.op != r.OpID || f.box != r.Target || f.ref == "" || c.Op.Intent["target"] != r.Target || c.Op.Intent["source"] == r.Target || c.Op.Intent["to"] != r.TargetRegion { |
| -+ return ErrFence |
| ++func claimBoxFenceVerifier(check func(context.Context) error, store release.Store, authority *hostops.ProductionAuthority) func(context.Context, hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| ++ return func(ctx context.Context, o hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| ++ if check == nil || store == nil || authority == nil || o.Claim == nil { |
| ++ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| + } |
| -+ var ref FenceRef |
| -+ if json.Unmarshal([]byte(f.ref), &ref) != nil || c.Op.Intent["target_enrollment"] != ref.Enrollment { |
| -+ return ErrFence |
| ++ if err := check(ctx); err != nil { |
| ++ return hostclaim.ReturnFence{}, err |
| + } |
| -+ if old := c.Op.Journal["fence."+r.Target]; old != "" && old != f.ref { |
| -+ return ErrFence |
| ++ fresh, err := hostclaim.Read(ctx, store, o.Host) |
| ++ if err != nil { |
| ++ return hostclaim.ReturnFence{}, err |
| + } |
| -+ if c.Op.Kind == KindRestoring && c.Op.Intent["rollback"] == r.RollbackID && c.Op.Journal[journalFencedTargetReleased] == f.ref { |
| -+ return nil |
| ++ if fresh.ETag != o.ETag { |
| ++ return hostclaim.ReturnFence{}, hostbox.ErrChanged |
| + } |
| -+ if c.Op.Exec != r.Target && c.Op.Exec != "" && c.Op.Exec != "none" { |
| -+ return ErrWrongState |
| ++ var ref hostclaim.FenceRef |
| ++ if json.Unmarshal([]byte(o.Claim.Op.Journal["fence."+o.Claim.Op.Intent["target"]]), &ref) != nil { |
| ++ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| + } |
| -+ return inOneWrite(c, func(c *Claim) error { |
| -+ setJournal(c, "fence."+r.Target, f.ref) |
| -+ c.Op.Exec = "none" |
| -+ c.Op.Participants = removeString(c.Op.Participants, r.Target) |
| -+ return releaseTargetParks(c, LostTargetRelease{OpID: r.OpID, RollbackID: r.RollbackID, Target: r.Target, TargetRegion: r.TargetRegion, SourceRegion: r.SourceRegion, Location: r.Location, Lane: r.Lane, ReturnLane: r.ReturnLane, NewID: r.NewID}) |
| -+ }, func(c *Claim) error { setJournal(c, journalFencedTargetReleased, f.ref); return nil }) |
| ++ return hostclaim.VerifyMoveTargetFence(ctx, store, o, ref, hostfence.ClaimFenceAuthority{Directory: authority.Directory, Now: time.Now}) |
| + } |
| +} |
| -+ |
| -+// FencedTargetReleased is the durable marker, checked without creating another CAS. |
| -+func FencedTargetReleased(c *Claim, fence ReturnFence) bool { |
| -+ return c != nil && c.Op.Kind == KindRestoring && c.Op.ID == fence.op && c.Op.Intent["target"] == fence.box && fence.ref != "" && c.Op.Journal["fence."+fence.box] == fence.ref && c.Op.Journal[journalFencedTargetReleased] == fence.ref |
| -+} |
| ---- /dev/null |
| -+++ b/internal/hostops/move_fence_release.go |
| -@@ -0,0 +1,112 @@ |
| -+package hostops |
| -+ |
| -+import ( |
| -+ "context" |
| -+ "crypto/rand" |
| +--- a/internal/host/claimwiring.go |
| ++++ b/internal/host/claimwiring.go |
| +@@ -7,6 +7,7 @@ |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| -+ "github.com/carlosframework/platform/internal/moves" |
| -+ "github.com/carlosframework/platform/internal/provision" |
| -+ "github.com/carlosframework/platform/internal/release" |
| -+) |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/schedule" |
| +@@ -19,14 +20,15 @@ |
| + } |
| + |
| + type ClaimProviders struct { |
| +- moveParks *claimMoveParks |
| +- Delivery schedule.DeliveryAdmission |
| +- Selector hostwrite.Selector |
| +- Wake *activator.ClaimRuntime |
| +- Regional RegionalLoop |
| +- Stop *ClaimStopCoordinator |
| +- Provision func(*provision.ReconcileConfig) |
| +- Close func(context.Context) error |
| ++ moveTargetFence func(context.Context, hostclaim.Observed) (hostclaim.ReturnFence, error) |
| ++ moveParks *claimMoveParks |
| ++ Delivery schedule.DeliveryAdmission |
| ++ Selector hostwrite.Selector |
| ++ Wake *activator.ClaimRuntime |
| ++ Regional RegionalLoop |
| ++ Stop *ClaimStopCoordinator |
| ++ Provision func(*provision.ReconcileConfig) |
| ++ Close func(context.Context) error |
| + } |
| + type ClaimFactory func(context.Context) (ClaimProviders, error) |
| + |
| +--- a/internal/hostclaim/lossrecord.go |
| ++++ b/internal/hostclaim/lossrecord.go |
| +@@ -95,6 +95,8 @@ |
| + } |
| + |
| + type LossRecord struct { |
| ++ AlertsPending bool `json:"alerts_pending,omitempty"` |
| ++ Automatic *AutomaticLoss `json:"automatic,omitempty"` |
| + V int `json:"v"` |
| + ID string `json:"id"` |
| + State string `json:"state"` // prepared | committed | conflicted |
| +@@ -147,6 +149,9 @@ |
| + } |
| + |
| + func (r LossRecord) Validate() error { |
| ++ if r.Automatic != nil { |
| ++ return r.validateAutomatic() |
| ++ } |
| + if r.V != lossRecordVersion || !safeLossID(r.ID) || (r.State != "prepared" && r.State != "committed" && r.State != "conflicted") || |
| + r.Actor == "" || r.AcceptedAt.IsZero() || r.Host == "" || !isHex64(r.HolderIncarnation) || r.HolderStamp == 0 || r.ExpectedClaimGen == 0 || |
| + r.ExpectedClaimETag == "" || !knownKinds[r.OpKind] || r.Box == "" || r.Enrollment == "" || |
| +@@ -393,3 +398,22 @@ |
| + } |
| + return nil |
| + } |
| + |
| -+// ReleaseFencedTarget advances one record-first recovery step per grant: |
| -+// record decision, claim release with parks, then record release. Each pass |
| -+// verifies the fence again; the source continues to gate on the claim phase. |
| -+func (g *MoveRecoveryGrant) ReleaseFencedTarget(ctx context.Context, ref hostclaim.FenceRef) error { |
| -+ if g == nil || g.state == nil { |
| -+ return ErrPending |
| -+ } |
| -+ g.state.mu.Lock() |
| -+ defer g.state.mu.Unlock() |
| -+ o, record, rec, err := g.current(ctx) |
| -+ if err != nil { |
| -+ return err |
| ++// Automatic records retain the exact claim evidence, without fabricating operator authority. |
| ++func (r LossRecord) validateAutomatic() error { |
| ++ l := r.Automatic |
| ++ if l == nil || l.Reported || !r.AlertsPending || r.V != lossRecordVersion || r.Actor != "automatic" || r.ID != l.ID || r.Host == "" || r.HolderIncarnation != l.Incarnation || r.HolderStamp != l.Stamp || r.HolderSession != l.Session || r.OpID != l.Op || r.OpKind != l.Kind || r.OpPhase != l.Phase || r.Box != l.Box || r.Enrollment != l.Enrollment || r.QuarantineRevision != 0 || r.ExpectedClaimGen == 0 || r.ExpectedClaimETag == "" || (r.State != "prepared" && r.State != "committed") || (r.State == "prepared" && r.ResultClaimGen != 0) || (r.State == "committed" && r.ResultClaimGen == 0) { |
| ++ return ErrLossRecord |
| + } |
| -+ |
| -+ m := rec.Move |
| -+ if m == nil || m.ID != g.o.OpID || m.Incarnation != g.o.Incarnation { |
| -+ return ErrIdentity |
| ++ at, err := time.Parse(time.RFC3339Nano, l.ObservedAt) |
| ++ if err != nil || !r.AcceptedAt.Equal(at) || !r.FencedAt.Equal(at) || r.OperatorAttestation.MaintainComputeFence || r.OperatorAttestation.ColdQuarantineStartOnly || len(r.OperatorAttestation.DestroyedStorage) != 0 || r.CloudObservation != (LossCloudObservation{}) || r.Timers.Knowledge != l.Timers || r.Timers.Source != nil || len(r.Timers.Entries) != 0 || len(r.WriteSources) != 0 { |
| ++ return ErrLossRecord |
| + } |
| -+ fence, err := hostclaim.VerifyMoveTargetFence(ctx, g.w.raw, o, ref, g.authority) |
| -+ if err != nil { |
| -+ return err |
| ++ if _, err = Key(r.Host); err != nil { |
| ++ return ErrLossRecord |
| + } |
| -+ guard := provision.MoveGuard{ID: m.ID, Incarnation: m.Incarnation} |
| -+ writeRecord := func() error { |
| -+ body, err := g.prepareRecord(ctx, rec) |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ _, _, err = provision.WriteRecord(provision.WithRecordSequence(ctx), g.w.raw, record.Holder.Key, record.Body, body, record.ETag) |
| -+ return err |
| ++ if err = validateAutomaticLosses(&Claim{EvidenceRefs: []AutomaticLoss{*l}}, map[string]bool{l.ID: true}); err != nil { |
| ++ return ErrLossRecord |
| + } |
| -+ if m.State == provision.MoveLanding { |
| -+ if rec.Region != m.To { |
| -+ return ErrIdentity |
| -+ } |
| -+ id, err := moves.NewRollbackID(rand.Reader) |
| -+ if err != nil { |
| -+ return err |
| ++ return nil |
| ++} |
| +--- a/internal/hostclaim/move.go |
| ++++ b/internal/hostclaim/move.go |
| +@@ -933,7 +933,7 @@ |
| + // verified before Update (VerifyReturnFence, J5): the op, the box and the |
| + // exact fence.<box> journal entry it verified. The zero value vouches for |
| + // nothing. |
| +-type ReturnFence struct{ op, box, ref string } |
| ++type ReturnFence struct{ op, box, ref, observedAt string } |
| + |
| + // VerifyReturnFence re-reads and verifies the fence the op journals for box |
| + // (VerifyOperationFence) and returns what ReturnToSource checks inside its |
| +@@ -959,6 +959,9 @@ |
| + return true |
| + } |
| + if !stringPresent(c.Op.Participants, target) { |
| ++ if _, consumed := ConsumedMoveTargetFence(c); consumed { |
| ++ return true |
| + } |
| -+ if err := provision.MoveRollBack(guard, id, provision.RollbackTargetFenced, g.w.now())(&rec); err != nil { |
| -+ return err |
| -+ } |
| -+ } else if m.State != provision.MoveRollingBack || m.Rollback == nil || rec.Region != m.From { |
| -+ return ErrIdentity |
| -+ } |
| -+ rb := m.Rollback |
| -+ if rb.Dirty { |
| -+ return ErrPending |
| -+ } // J4: a dead dirty target cannot build a return lane. |
| -+ if hostclaim.FencedTargetReleased(o.Claim, fence) { |
| -+ if rb.LossRef != ref.Key { |
| -+ return ErrIdentity |
| -+ } |
| -+ if rb.TargetPhase == provision.RbTargetReleased && rb.Evidence == ref.Key { |
| -+ return nil |
| -+ } |
| -+ if err := provision.MoveRollbackLossCommitted(guard)(&rec); err != nil { |
| -+ return err |
| -+ } |
| -+ return writeRecord() |
| -+ } |
| -+ if rb.TargetPhase != provision.RbTargetLost || rb.LossRef != ref.Key { |
| -+ if err := provision.MoveRollbackTargetLost(guard, ref.Key, g.w.now())(&rec); err != nil { |
| -+ return err |
| -+ } |
| -+ rb.Reason = provision.RollbackTargetFenced |
| -+ return writeRecord() |
| -+ } |
| -+ if rb.Reason != provision.RollbackTargetFenced || rb.LostAt == nil || rb.LostAt.IsZero() { |
| -+ return ErrPending |
| -+ } |
| -+ h := o.Claim.Holder |
| -+ r := hostclaim.FencedTargetRelease{OpID: m.ID, RollbackID: rb.ID, Target: o.Claim.Op.Intent["target"], TargetRegion: m.To, SourceRegion: m.From, |
| -+ Location: moves.TargetLocationScope(*h, *rb.LostAt), Lane: moves.StagingScope(o.Host, m.ID, h.Incarnation, *rb.LostAt), NewID: moveLossNewID(ref.SHA256), Fence: fence} |
| -+ if err := CheckCreateFreeze(ctx, g.w.raw, g.w.pub); err != nil { |
| -+ return err |
| -+ } |
| -+ // Fence and roster reads may have raced a new record. Bind both ETag and sequence again. |
| -+ if _, _, _, err := g.current(ctx); err != nil { |
| -+ return err |
| -+ } |
| -+ g.state.active = false |
| -+ _, err = hostclaim.Update(ctx, g.w.raw, o, func(c *hostclaim.Claim) error { |
| -+ record, err := ReadRecord(ctx, g.w.raw, h.Key) |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ _, seq, err := recordSequence(record.Body) |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ if record.ETag != g.o.RecordETag || seq != g.o.RecordSeq { |
| -+ return release.ErrCASConflict |
| -+ } |
| -+ if err := moveAdmission(ctx, g.w.raw, o.Host, c); err != nil { |
| -+ return err |
| -+ } |
| -+ return hostclaim.ReleaseFencedTarget(r)(c) |
| -+ }) |
| -+ return err |
| -+} |
| ---- a/internal/provision/provision.go |
| -+++ b/internal/provision/provision.go |
| -@@ -1817,6 +1817,8 @@ |
| - // records (§7.10): the target box was declared lost before the move |
| - // committed. A recorded sentence, so it names no box; the box is in the |
| - // loss record LossRef names. |
| -+const RollbackTargetFenced = "target fenced" |
| -+ |
| - const RollbackDeclaredLost = "the target box was declared lost" |
| - |
| - // The reasons a move may record when it stops for an operator (§7.8 stuck, |
| -@@ -1940,6 +1942,8 @@ |
| - rb.Reason = RollbackCommitDeadline |
| - case RollbackProbeUnpriced: |
| - rb.Reason = RollbackProbeUnpriced |
| -+ case RollbackTargetFenced: |
| -+ rb.Reason = RollbackTargetFenced |
| - case RollbackDeclaredLost: |
| - rb.Reason = RollbackDeclaredLost |
| - default: |
| -@@ -1959,7 +1963,7 @@ |
| - func recordedRollbackReason(r string) bool { |
| - switch r { |
| - case RollbackConfigChanged, RollbackLandingDeadline, RollbackClaimWired, RollbackLandingLayout, RollbackInstallMismatch, |
| -- RollbackFilesOccupied, RollbackProbeAttempts, RollbackCommitDeadline, RollbackDeclaredLost, RollbackProbeUnpriced: |
| -+ RollbackFilesOccupied, RollbackProbeAttempts, RollbackCommitDeadline, RollbackDeclaredLost, RollbackTargetFenced, RollbackProbeUnpriced: |
| - return true |
| - } |
| - return false |
| ---- /dev/null |
| -+++ b/internal/host/claimmovefence.go |
| -@@ -0,0 +1,41 @@ |
| -+package host |
| -+ |
| -+import ( |
| -+ "context" |
| -+ "encoding/json" |
| -+ "github.com/carlosframework/platform/internal/hostbox" |
| -+ "github.com/carlosframework/platform/internal/hostclaim" |
| -+ "github.com/carlosframework/platform/internal/hostfence" |
| -+ "github.com/carlosframework/platform/internal/hostops" |
| -+ "github.com/carlosframework/platform/internal/release" |
| -+ "time" |
| -+) |
| -+ |
| -+func (p ClaimProviders) VerifyMoveTargetFence(ctx context.Context, o hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| -+ if p.moveTargetFence == nil { |
| -+ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| -+ } |
| -+ return p.moveTargetFence(ctx, o) |
| -+} |
| -+func claimBoxFenceVerifier(check func(context.Context) error, store release.Store, authority *hostops.ProductionAuthority) func(context.Context, hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| -+ return func(ctx context.Context, o hostclaim.Observed) (hostclaim.ReturnFence, error) { |
| -+ if check == nil || store == nil || authority == nil || o.Claim == nil { |
| -+ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| -+ } |
| -+ if err := check(ctx); err != nil { |
| -+ return hostclaim.ReturnFence{}, err |
| -+ } |
| -+ fresh, err := hostclaim.Read(ctx, store, o.Host) |
| -+ if err != nil { |
| -+ return hostclaim.ReturnFence{}, err |
| -+ } |
| -+ if fresh.ETag != o.ETag { |
| -+ return hostclaim.ReturnFence{}, hostbox.ErrChanged |
| -+ } |
| -+ var ref hostclaim.FenceRef |
| -+ if json.Unmarshal([]byte(o.Claim.Op.Journal["fence."+o.Claim.Op.Intent["target"]]), &ref) != nil { |
| -+ return hostclaim.ReturnFence{}, hostclaim.ErrFence |
| -+ } |
| -+ return hostclaim.VerifyMoveTargetFence(ctx, store, o, ref, hostfence.ClaimFenceAuthority{Directory: authority.Directory, Now: time.Now}) |
| -+ } |
| -+} |
| ---- a/internal/host/claimwiring.go |
| -+++ b/internal/host/claimwiring.go |
| -@@ -7,6 +7,7 @@ |
| - |
| - "github.com/carlosframework/platform/internal/activator" |
| - "github.com/carlosframework/platform/internal/hostbox" |
| -+ "github.com/carlosframework/platform/internal/hostclaim" |
| - "github.com/carlosframework/platform/internal/hostwrite" |
| - "github.com/carlosframework/platform/internal/provision" |
| - "github.com/carlosframework/platform/internal/schedule" |
| -@@ -19,14 +20,15 @@ |
| - } |
| - |
| - type ClaimProviders struct { |
| -- moveParks *claimMoveParks |
| -- Delivery schedule.DeliveryAdmission |
| -- Selector hostwrite.Selector |
| -- Wake *activator.ClaimRuntime |
| -- Regional RegionalLoop |
| -- Stop *ClaimStopCoordinator |
| -- Provision func(*provision.ReconcileConfig) |
| -- Close func(context.Context) error |
| -+ moveTargetFence func(context.Context, hostclaim.Observed) (hostclaim.ReturnFence, error) |
| -+ moveParks *claimMoveParks |
| -+ Delivery schedule.DeliveryAdmission |
| -+ Selector hostwrite.Selector |
| -+ Wake *activator.ClaimRuntime |
| -+ Regional RegionalLoop |
| -+ Stop *ClaimStopCoordinator |
| -+ Provision func(*provision.ReconcileConfig) |
| -+ Close func(context.Context) error |
| - } |
| - type ClaimFactory func(context.Context) (ClaimProviders, error) |
| - |
| + for _, id := range c.LossRefs { |
| + if _, ok := c.Op.Journal["loss."+id]; ok { |
| + return true |
| --- a/internal/hostclaim/move_loss.go |
| +++ b/internal/hostclaim/move_loss.go |
| @@ -68,88 +68,7 @@ |
| @@ -1355,112 +1531,628 @@ Expected first failure: the verifier, release transition and grant method are un |
| return c != nil && lossID != "" && c.Op.Kind == KindRestoring && c.Op.Journal[journalLostTargetReleased] == lossID |
| } |
| + |
| -+func releaseTargetParks(c *Claim, r LostTargetRelease) error { |
| -+ location := r.Location |
| -+ if err := location.Validate(c.Host); err != nil { |
| -+ return err |
| ++func releaseTargetParks(c *Claim, r LostTargetRelease) error { |
| ++ location := r.Location |
| ++ if err := location.Validate(c.Host); err != nil { |
| ++ return err |
| ++ } |
| ++ lane, err := parkQueueGuard(c, r.OpID, r.TargetRegion, r.Lane, r.NewID) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ // The location park's scope is checked against holder.location by |
| ++ // QueueLocationPark; the lane is this move's own. |
| ++ if err := requireHolderLocation(c, r.Location); err != nil { |
| ++ return err |
| ++ } |
| ++ if lane.Location != "moves/"+c.Host+"/"+r.OpID { |
| ++ return fmt.Errorf("%w: a lost target's lane park is the move's own lane", ErrMalformed) |
| ++ } |
| ++ var returnLane *ParkScope |
| ++ if r.ReturnLane != nil { |
| ++ s, err := parkQueueGuard(c, r.OpID, r.SourceRegion, *r.ReturnLane, r.NewID) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ returnLane = &s |
| ++ if s.Location != "moves/"+c.Host+"/"+r.RollbackID { |
| ++ return fmt.Errorf("%w: the return lane is this rollback's own lane", ErrMalformed) |
| ++ } |
| ++ } |
| ++ steps := []func(*Claim) error{} |
| ++ switch { |
| ++ case c.Op.Kind == KindLanding && c.Op.Phase != PhaseCommitted: |
| ++ steps = append(steps, RollBack(r.OpID, r.RollbackID)) |
| ++ case c.Op.Kind == KindRestoring && c.Op.Intent["rollback"] == r.RollbackID: |
| ++ default: |
| ++ return fmt.Errorf("%w: op is %s/%q, a lost target is released from a landing before its commit or from its own rollback", ErrWrongState, c.Op.Kind, c.Op.Phase) |
| ++ } |
| ++ steps = append(steps, func(c *Claim) error { |
| ++ switch c.Op.Phase { |
| ++ case PhaseReleasing: |
| ++ if c.Op.Exec != "" && c.Op.Exec != "none" { |
| ++ return fmt.Errorf("%w: exec is %q", ErrWrongState, c.Op.Exec) |
| ++ } |
| ++ c.Op.Phase = PhaseReleased |
| ++ parkBox := "" |
| ++ if stamp, err := strconv.ParseUint(c.Op.Journal[JournalTargetStamp], 10, 64); err == nil && InventoryHasStamp(c, r.Target, stamp) { |
| ++ parkBox = r.Target |
| ++ } |
| ++ if err := QueueLocationPark(r.OpID, r.TargetRegion, parkBox, r.Location, r.NewID)(c); err != nil { |
| ++ return err |
| ++ } |
| ++ return CoverRegionPredecessors(r.OpID, PhaseReleased, r.TargetRegion)(c) |
| ++ case PhaseReleased: |
| ++ // The target released itself before it was lost: its release |
| ++ // queued the location park and covered the region. |
| ++ if c.Op.Journal[JournalEvidenceTarget] == "" { |
| ++ return fmt.Errorf("%w: released without the target's evidence", ErrWrongState) |
| ++ } |
| ++ if returnLane == nil { |
| ++ return fmt.Errorf("%w: a dirty release that died before its return owes the return lane's park", ErrMalformed) |
| ++ } |
| ++ return nil |
| ++ } |
| ++ return fmt.Errorf("%w: op is restoring/%q, the lost target's release precedes the source's claim", ErrWrongState, c.Op.Phase) |
| ++ }, func(c *Claim) error { |
| ++ if owesStagingPark(c, r.OpID, r.TargetRegion) { |
| ++ return nil |
| ++ } |
| ++ _, err := queueReplicaPark(c, r.OpID, r.TargetRegion, lane, r.NewID) |
| ++ return err |
| ++ }, func(c *Claim) error { |
| ++ if returnLane == nil { |
| ++ return nil |
| ++ } |
| ++ if _, err := queueReplicaPark(c, r.OpID, r.SourceRegion, *returnLane, r.NewID); err != nil { |
| ++ return err |
| ++ } |
| ++ setJournal(c, journalReturnLane, returnLane.Location) |
| ++ return nil |
| ++ }) |
| ++ return inOneWrite(c, steps...) |
| ++} |
| +--- /dev/null |
| ++++ b/internal/hostclaim/move_target_fence.go |
| +@@ -0,0 +1,161 @@ |
| ++package hostclaim |
| ++ |
| ++import ( |
| ++ "context" |
| ++ "encoding/json" |
| ++ "github.com/carlosframework/platform/internal/release" |
| ++ "time" |
| ++) |
| ++ |
| ++const journalFencedTargetReleased = "fenced_target_released" |
| ++ |
| ++// VerifyMoveTargetFence supplies a target-bound capability without writing the claim. |
| ++func VerifyMoveTargetFence(ctx context.Context, s release.Store, o Observed, ref FenceRef, authority FenceAuthority) (ReturnFence, error) { |
| ++ if o.Absent || o.Claim == nil || o.Claim.Holder == nil || o.ETag == "" { |
| ++ return ReturnFence{}, ErrStaleOp |
| ++ } |
| ++ c := o.Claim |
| ++ if (c.Op.Kind != KindLanding && c.Op.Kind != KindRestoring) || c.Op.Phase == PhaseCommitted || c.Op.ID == "" || c.Op.Intent["target"] == c.Op.Intent["source"] || c.Op.Intent["target_enrollment"] != ref.Enrollment { |
| ++ return ReturnFence{}, ErrWrongState |
| ++ } |
| ++ f, err := VerifyFence(ctx, s, authority, ref) |
| ++ if err != nil { |
| ++ return ReturnFence{}, err |
| ++ } |
| ++ if f.Box() != c.Op.Intent["target"] || f.Region() != c.Op.Intent["to"] { |
| ++ return ReturnFence{}, ErrFence |
| ++ } |
| ++ raw, err := json.Marshal(ref) |
| ++ if err != nil { |
| ++ return ReturnFence{}, err |
| ++ } |
| ++ at, err := time.Parse(time.RFC3339Nano, f.record.ObservedAt) |
| ++ if err != nil || at.IsZero() { |
| ++ return ReturnFence{}, ErrFence |
| ++ } |
| ++ return ReturnFence{op: c.Op.ID, box: f.Box(), ref: string(raw), observedAt: f.record.ObservedAt}, nil |
| ++} |
| ++ |
| ++// FencedTargetRelease has the same scopes as loss release, with verified fence authority. |
| ++type FencedTargetRelease struct { |
| ++ OpID, RollbackID, Target, TargetRegion, SourceRegion string |
| ++ Location, Lane ParkScope |
| ++ ReturnLane *ParkScope |
| ++ NewID func() string |
| ++ Fence ReturnFence |
| ++} |
| ++ |
| ++// ReleaseFencedTarget declares and consumes the fence in one CAS, including all parks. |
| ++func ReleaseFencedTarget(r FencedTargetRelease) func(*Claim) error { |
| ++ return func(c *Claim) error { |
| ++ if err := RequireOp(r.OpID)(c); err != nil { |
| ++ return err |
| ++ } |
| ++ if c.Holder == nil || !rollbackIDShape.MatchString(r.RollbackID) || r.Target == "" || r.NewID == nil || r.SourceRegion == "" || r.SourceRegion == r.TargetRegion { |
| ++ return ErrMalformed |
| ++ } |
| ++ f := r.Fence |
| ++ if f.op != r.OpID || f.box != r.Target || f.ref == "" || c.Op.Intent["target"] != r.Target || c.Op.Intent["source"] == r.Target || c.Op.Intent["to"] != r.TargetRegion { |
| ++ return ErrFence |
| ++ } |
| ++ var ref FenceRef |
| ++ if json.Unmarshal([]byte(f.ref), &ref) != nil || c.Op.Intent["target_enrollment"] != ref.Enrollment { |
| ++ return ErrFence |
| ++ } |
| ++ if old := c.Op.Journal["fence."+r.Target]; old != "" && old != f.ref { |
| ++ return ErrFence |
| ++ } |
| ++ if c.Op.Kind == KindRestoring && c.Op.Intent["rollback"] == r.RollbackID && c.Op.Journal[journalFencedTargetReleased] == f.ref { |
| ++ return nil |
| ++ } |
| ++ if c.Op.Exec != r.Target && c.Op.Exec != "" && c.Op.Exec != "none" { |
| ++ return ErrWrongState |
| ++ } |
| ++ at, err := time.Parse(time.RFC3339Nano, f.observedAt) |
| ++ if err != nil || at.IsZero() || !r.Location.EvidenceAt.Equal(at) || !r.Lane.EvidenceAt.Equal(at) || (r.ReturnLane != nil && !r.ReturnLane.EvidenceAt.Equal(at)) { |
| ++ return ErrFence |
| ++ } |
| ++ loss := AutomaticLoss{ID: automaticLossID(c.Op.ID, r.Target, ref), Box: r.Target, Enrollment: ref.Enrollment, Fence: ref, ObservedAt: f.observedAt, Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: "target", Incarnation: c.Holder.Incarnation, Stamp: c.Holder.Stamp, Timers: TimersUnknown} |
| ++ if c.Holder.WriterBox == r.Target { |
| ++ loss.Session, loss.Generation = c.Holder.Session, c.Holder.Generation |
| ++ } |
| ++ for i := len(c.Holder.Admissions) - 1; i >= 0; i-- { |
| ++ a := c.Holder.Admissions[i] |
| ++ if a.Box == r.Target { |
| ++ loss.Session, loss.Generation, loss.LastAdmission = a.Session, a.Generation, &a |
| ++ break |
| ++ } |
| ++ } |
| ++ return inOneWrite(c, func(c *Claim) error { |
| ++ setJournal(c, "fence."+r.Target, f.ref) |
| ++ c.Op.Exec = "none" |
| ++ c.Op.Participants = removeString(c.Op.Participants, r.Target) |
| ++ return releaseTargetParks(c, LostTargetRelease{OpID: r.OpID, RollbackID: r.RollbackID, Target: r.Target, TargetRegion: r.TargetRegion, SourceRegion: r.SourceRegion, Location: r.Location, Lane: r.Lane, ReturnLane: r.ReturnLane, NewID: r.NewID}) |
| ++ }, func(c *Claim) error { |
| ++ for i := range c.Obligations { |
| ++ o := &c.Obligations[i] |
| ++ if o.Op == r.OpID && (o.Kind == "park-generation" || o.Kind == "park-replica") && (o.Region == r.TargetRegion || o.Owner == "box:"+r.Target) { |
| ++ o.Owner = "region:" + o.Region |
| ++ loss.Reassigned = append(loss.Reassigned, ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}) |
| ++ } |
| ++ } |
| ++ if !stringPresent(c.Lost, r.Target) { |
| ++ c.Lost = append(c.Lost, r.Target) |
| ++ } |
| ++ if !lossRefPresent(c, loss.ID) { |
| ++ c.LossRefs = append(c.LossRefs, loss.ID) |
| ++ c.EvidenceRefs = append(c.EvidenceRefs, loss) |
| ++ } |
| ++ setJournal(c, journalFencedTargetReleased, f.ref) |
| ++ return nil |
| ++ }) |
| ++ } |
| ++} |
| ++ |
| ++// FencedTargetReleased is the durable marker, checked without creating another CAS. |
| ++func FencedTargetReleased(c *Claim, fence ReturnFence) bool { |
| ++ ref, ok := ConsumedMoveTargetFence(c) |
| ++ var expected FenceRef |
| ++ return ok && c.Op.ID == fence.op && c.Op.Intent["target"] == fence.box && json.Unmarshal([]byte(fence.ref), &expected) == nil && ref == expected |
| ++} |
| ++ |
| ++// ConsumedMoveTargetFence reads the claim's completed revocation, not live roster authority. |
| ++func ConsumedMoveTargetFence(c *Claim) (FenceRef, bool) { |
| ++ if c == nil || c.Holder == nil || c.Op.Kind != KindRestoring { |
| ++ return FenceRef{}, false |
| ++ } |
| ++ target := c.Op.Intent["target"] |
| ++ raw := c.Op.Journal["fence."+target] |
| ++ var ref FenceRef |
| ++ if raw == "" || raw != c.Op.Journal[journalFencedTargetReleased] || json.Unmarshal([]byte(raw), &ref) != nil || !validFenceRef(ref) || ref.Enrollment != c.Op.Intent["target_enrollment"] || c.Op.Exec == target || stringPresent(c.Op.Participants, target) || !stringPresent(c.Lost, target) { |
| ++ return FenceRef{}, false |
| ++ } |
| ++ for _, l := range c.EvidenceRefs { |
| ++ if l.ID == automaticLossID(c.Op.ID, target, ref) && l.Op == c.Op.ID && l.Box == target && l.Fence == ref && l.Incarnation == c.Holder.Incarnation && lossRefPresent(c, l.ID) { |
| ++ return ref, true |
| ++ } |
| ++ } |
| ++ return FenceRef{}, false |
| ++} |
| ++ |
| ++// LostMoveTargetParked requires revocation and completion of every target chore. |
| ++func LostMoveTargetParked(c *Claim, region string) bool { |
| ++ if c == nil || c.Holder == nil || c.Op.Kind != KindRestoring || region == "" || region != c.Op.Intent["to"] { |
| ++ return false |
| ++ } |
| ++ target := c.Op.Intent["target"] |
| ++ if target == "" || !stringPresent(c.Lost, target) || c.Op.Exec == target || stringPresent(c.Op.Participants, target) { |
| ++ return false |
| ++ } |
| ++ for _, o := range c.Obligations { |
| ++ if o.Op == c.Op.ID && (o.Box == target || o.Owner == "box:"+target || o.Region == region || o.Owner == "region:"+region) { |
| ++ return false |
| ++ } |
| ++ } |
| ++ return true |
| ++} |
| ++ |
| ++func (f ReturnFence) ObservedAt() time.Time { |
| ++ at, _ := time.Parse(time.RFC3339Nano, f.observedAt) |
| ++ return at |
| ++} |
| +--- /dev/null |
| ++++ b/internal/hostops/automatic_loss.go |
| +@@ -0,0 +1,76 @@ |
| ++package hostops |
| ++ |
| ++import ( |
| ++ "context" |
| ++ "errors" |
| ++ "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/release" |
| ++ "reflect" |
| ++ "time" |
| ++) |
| ++ |
| ++// ReportAutomaticLosses materializes retained consequences even after operation |
| ++// or holder replacement. The signed record is durable before reporting is marked. |
| ++func (w *ControlWriter) ReportAutomaticLosses(ctx context.Context, signer release.Signer, host string) error { |
| ++ if recoveryWriter(ctx) != w { |
| ++ return ErrContract |
| ++ } |
| ++ if err := w.lossSigner(signer); err != nil { |
| ++ return err |
| ++ } |
| ++ o, err := hostclaim.Read(ctx, w.raw, host) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if o.Absent { |
| ++ return nil |
| ++ } |
| ++ for _, l := range o.Claim.EvidenceRefs { |
| ++ if l.Reported { |
| ++ continue |
| ++ } |
| ++ record, err := hostclaim.ReadLossRecord(ctx, w.raw, w.pub, l.ID) |
| ++ if errors.Is(err, hostclaim.ErrLossRecordAbsent) { |
| ++ at, e := time.Parse(time.RFC3339Nano, l.ObservedAt) |
| ++ if e != nil { |
| ++ return e |
| ++ } |
| ++ r := hostclaim.LossRecord{AlertsPending: true, V: 1, ID: l.ID, State: "prepared", Actor: "automatic", AcceptedAt: at, Host: host, HolderIncarnation: l.Incarnation, HolderStamp: l.Stamp, HolderSession: l.Session, ExpectedClaimGen: o.Claim.Gen, ExpectedClaimETag: o.ETag, OpID: l.Op, OpKind: l.Kind, OpPhase: l.Phase, Box: l.Box, Enrollment: l.Enrollment, FencedAt: at, Timers: hostclaim.TimerLoss{Knowledge: l.Timers}, Automatic: &l} |
| ++ record, err = hostclaim.CreateLossRecord(ctx, w.raw, signer, r) |
| ++ } |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if !reflect.DeepEqual(record.Record.Automatic, &l) || record.Record.Host != host { |
| ++ return hostclaim.ErrLossRecord |
| ++ } |
| ++ if record.Record.State == "prepared" { |
| ++ next := record.Record |
| ++ next.State = "committed" |
| ++ next.ResultClaimGen = next.ExpectedClaimGen |
| ++ record, err = hostclaim.ReplaceLossRecord(ctx, w.raw, signer, record, next) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ } |
| ++ if record.Record.State != "committed" { |
| ++ return hostclaim.ErrLossRecord |
| ++ } |
| ++ o, err = hostclaim.Update(ctx, w.raw, o, func(c *hostclaim.Claim) error { |
| ++ for i := range c.EvidenceRefs { |
| ++ if c.EvidenceRefs[i].ID == l.ID { |
| ++ if !reflect.DeepEqual(c.EvidenceRefs[i], l) { |
| ++ return hostclaim.ErrLossRecord |
| ++ } |
| ++ c.EvidenceRefs[i].Reported = true |
| ++ return nil |
| ++ } |
| ++ } |
| ++ return hostclaim.ErrLossRecord |
| ++ }) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ } |
| ++ return nil |
| ++} |
| +--- /dev/null |
| ++++ b/internal/hostops/move_fence_release.go |
| +@@ -0,0 +1,174 @@ |
| ++package hostops |
| ++ |
| ++import ( |
| ++ "context" |
| ++ "crypto/rand" |
| ++ "encoding/json" |
| ++ "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/moves" |
| ++ "github.com/carlosframework/platform/internal/provision" |
| ++ "github.com/carlosframework/platform/internal/release" |
| ++) |
| ++ |
| ++// ReleaseFencedTarget advances one record-first recovery step per grant: |
| ++// record decision, claim release with parks, then record release. Each pass |
| ++// verifies unconsumed authority; completion uses the durable claim evidence. |
| ++func (g *MoveRecoveryGrant) ReleaseFencedTarget(ctx context.Context, ref hostclaim.FenceRef) error { |
| ++ if g == nil || g.state == nil { |
| ++ return ErrPending |
| ++ } |
| ++ g.state.mu.Lock() |
| ++ defer g.state.mu.Unlock() |
| ++ o, record, rec, err := g.current(ctx) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ |
| ++ m := rec.Move |
| ++ if m == nil || m.ID != g.o.OpID || m.Incarnation != g.o.Incarnation { |
| ++ return ErrIdentity |
| ++ } |
| ++ guard := provision.MoveGuard{ID: m.ID, Incarnation: m.Incarnation} |
| ++ writeRecord := func() error { |
| ++ body, err := g.prepareRecord(ctx, rec) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ _, _, err = provision.WriteRecord(provision.WithRecordSequence(ctx), g.w.raw, record.Holder.Key, record.Body, body, record.ETag) |
| ++ return err |
| ++ } |
| ++ if consumed, ok := hostclaim.ConsumedMoveTargetFence(o.Claim); ok { |
| ++ if consumed != ref || m.Rollback == nil || m.State != provision.MoveRollingBack || rec.Region != m.From { |
| ++ return ErrIdentity |
| ++ } |
| ++ rb := m.Rollback |
| ++ if rb.TargetPhase == provision.RbTargetReleased && !rb.Dirty { |
| ++ if rb.LossRef == ref.Key && rb.Evidence == ref.Key { |
| ++ return nil |
| ++ } |
| ++ return g.verifyReleasedTarget(ctx, o, rec) |
| ++ } |
| ++ if rb.LossRef != ref.Key { |
| ++ return ErrIdentity |
| ++ } |
| ++ if err := provision.MoveRollbackLossCommitted(guard)(&rec); err != nil { |
| ++ return err |
| ++ } |
| ++ return writeRecord() |
| ++ } |
| ++ fence, err := hostclaim.VerifyMoveTargetFence(ctx, g.w.raw, o, ref, g.authority) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if m.State == provision.MoveLanding { |
| ++ if rec.Region != m.To { |
| ++ return ErrIdentity |
| ++ } |
| ++ id, err := moves.NewRollbackID(rand.Reader) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if err := provision.MoveRollBack(guard, id, provision.RollbackTargetFenced, g.w.now())(&rec); err != nil { |
| ++ return err |
| ++ } |
| ++ } else if m.State != provision.MoveRollingBack || m.Rollback == nil || rec.Region != m.From { |
| ++ return ErrIdentity |
| ++ } |
| ++ rb := m.Rollback |
| ++ if rb.Dirty { |
| ++ return ErrPending |
| ++ } // J4: a dead dirty target cannot build a return lane. |
| ++ released := rb.TargetPhase == provision.RbTargetReleased && !rb.Dirty |
| ++ if released { |
| ++ if err := g.verifyReleasedTarget(ctx, o, rec); err != nil { |
| ++ return err |
| ++ } |
| ++ if o.Claim.Op.Kind != hostclaim.KindRestoring || o.Claim.Op.Phase != hostclaim.PhaseReleasing || o.Claim.Op.Exec != o.Claim.Op.Intent["target"] { |
| ++ return ErrPending |
| ++ } |
| ++ } else if rb.TargetPhase != provision.RbTargetLost || rb.LossRef != ref.Key { |
| ++ if err := provision.MoveRollbackTargetLost(guard, ref.Key, g.w.now())(&rec); err != nil { |
| ++ return err |
| ++ } |
| ++ return writeRecord() |
| ++ } |
| ++ h := o.Claim.Holder |
| ++ r := hostclaim.FencedTargetRelease{OpID: m.ID, RollbackID: rb.ID, Target: o.Claim.Op.Intent["target"], TargetRegion: m.To, SourceRegion: m.From, |
| ++ Location: moves.TargetLocationScope(*h, fence.ObservedAt()), Lane: moves.StagingScope(o.Host, m.ID, h.Incarnation, fence.ObservedAt()), NewID: moveLossNewID(ref.SHA256), Fence: fence} |
| ++ if err := CheckCreateFreeze(ctx, g.w.raw, g.w.pub); err != nil { |
| ++ return err |
| ++ } |
| ++ // Fence and roster reads may have raced a new record. Bind both ETag and sequence again. |
| ++ if _, _, _, err := g.current(ctx); err != nil { |
| ++ return err |
| ++ } |
| ++ g.state.active = false |
| ++ _, err = hostclaim.Update(ctx, g.w.raw, o, func(c *hostclaim.Claim) error { |
| ++ record, err := ReadRecord(ctx, g.w.raw, h.Key) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ _, seq, err := recordSequence(record.Body) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if record.ETag != g.o.RecordETag || seq != g.o.RecordSeq { |
| ++ return release.ErrCASConflict |
| ++ } |
| ++ if err := moveAdmission(ctx, g.w.raw, o.Host, c); err != nil { |
| ++ return err |
| ++ } |
| ++ return hostclaim.ReleaseFencedTarget(r)(c) |
| ++ }) |
| ++ return err |
| ++} |
| ++ |
| ++// The record-first release can outlive its target. Require the canonical complete report. |
| ++func (g *MoveRecoveryGrant) verifyReleasedTarget(ctx context.Context, o hostclaim.Observed, rec provision.Record) error { |
| ++ m, h := rec.Move, o.Claim.Holder |
| ++ if m == nil || m.Rollback == nil || m.Rollback.Dirty || m.Rollback.TargetPhase != provision.RbTargetReleased { |
| ++ return ErrIdentity |
| ++ } |
| ++ key := moves.StoppedKey(moves.ReportPrefix(h.Account, h.App, o.Host, m.ID), "target") |
| ++ if m.Rollback.Evidence != key { |
| ++ return ErrIdentity |
| ++ } |
| ++ raw, _, err := g.w.raw.Get(ctx, key) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ var report moves.StoppedReport |
| ++ if json.Unmarshal(raw, &report) != nil || !report.Complete() || report.Role != "target" || report.Box != o.Claim.Op.Intent["target"] || report.Host != o.Host || report.OpID != m.ID || report.Incarnation != h.Incarnation { |
| ++ return ErrIdentity |
| ++ } |
| ++ return nil |
| ++} |
| ++ |
| ++// StampLostMoveTargetParked uses the same expiring, record-sequenced grant as closure. |
| ++func (g *MoveRecoveryGrant) StampLostMoveTargetParked(ctx context.Context) error { |
| ++ if g == nil || g.state == nil { |
| ++ return ErrPending |
| + } |
| -+ lane, err := parkQueueGuard(c, r.OpID, r.TargetRegion, r.Lane, r.NewID) |
| ++ g.state.mu.Lock() |
| ++ defer g.state.mu.Unlock() |
| ++ o, r, rec, err := g.current(ctx) |
| + if err != nil { |
| + return err |
| + } |
| -+ // The location park's scope is checked against holder.location by |
| -+ // QueueLocationPark; the lane is this move's own. |
| -+ if err := requireHolderLocation(c, r.Location); err != nil { |
| -+ return err |
| ++ m := rec.Move |
| ++ if m == nil || m.ID != g.o.OpID || m.Incarnation != g.o.Incarnation || m.Rollback == nil || !hostclaim.LostMoveTargetParked(o.Claim, m.To) { |
| ++ return ErrPending |
| + } |
| -+ if lane.Location != "moves/"+c.Host+"/"+r.OpID { |
| -+ return fmt.Errorf("%w: a lost target's lane park is the move's own lane", ErrMalformed) |
| ++ if m.Rollback.TargetParkedAt != nil { |
| ++ return nil |
| + } |
| -+ var returnLane *ParkScope |
| -+ if r.ReturnLane != nil { |
| -+ s, err := parkQueueGuard(c, r.OpID, r.SourceRegion, *r.ReturnLane, r.NewID) |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ returnLane = &s |
| -+ if s.Location != "moves/"+c.Host+"/"+r.RollbackID { |
| -+ return fmt.Errorf("%w: the return lane is this rollback's own lane", ErrMalformed) |
| -+ } |
| ++ if err := provision.MoveLostTargetParked(provision.MoveGuard{ID: m.ID, Incarnation: m.Incarnation}, m.Rollback.ID, g.w.now())(&rec); err != nil { |
| ++ return err |
| + } |
| -+ steps := []func(*Claim) error{} |
| -+ switch { |
| -+ case c.Op.Kind == KindLanding && c.Op.Phase != PhaseCommitted: |
| -+ steps = append(steps, RollBack(r.OpID, r.RollbackID)) |
| -+ case c.Op.Kind == KindRestoring && c.Op.Intent["rollback"] == r.RollbackID: |
| -+ default: |
| -+ return fmt.Errorf("%w: op is %s/%q, a lost target is released from a landing before its commit or from its own rollback", ErrWrongState, c.Op.Kind, c.Op.Phase) |
| ++ body, err := g.prepareRecord(ctx, rec) |
| ++ if err != nil { |
| ++ return err |
| + } |
| -+ steps = append(steps, func(c *Claim) error { |
| -+ switch c.Op.Phase { |
| -+ case PhaseReleasing: |
| -+ if c.Op.Exec != "" && c.Op.Exec != "none" { |
| -+ return fmt.Errorf("%w: exec is %q", ErrWrongState, c.Op.Exec) |
| -+ } |
| -+ c.Op.Phase = PhaseReleased |
| -+ parkBox := "" |
| -+ if stamp, err := strconv.ParseUint(c.Op.Journal[JournalTargetStamp], 10, 64); err == nil && InventoryHasStamp(c, r.Target, stamp) { |
| -+ parkBox = r.Target |
| -+ } |
| -+ if err := QueueLocationPark(r.OpID, r.TargetRegion, parkBox, r.Location, r.NewID)(c); err != nil { |
| -+ return err |
| -+ } |
| -+ return CoverRegionPredecessors(r.OpID, PhaseReleased, r.TargetRegion)(c) |
| -+ case PhaseReleased: |
| -+ // The target released itself before it was lost: its release |
| -+ // queued the location park and covered the region. |
| -+ if c.Op.Journal[JournalEvidenceTarget] == "" { |
| -+ return fmt.Errorf("%w: released without the target's evidence", ErrWrongState) |
| ++ _, _, err = provision.WriteRecord(provision.WithRecordSequence(ctx), g.w.raw, r.Holder.Key, r.Body, body, r.ETag) |
| ++ return err |
| ++} |
| +--- a/internal/hostops/move_recovery_grant.go |
| ++++ b/internal/hostops/move_recovery_grant.go |
| +@@ -18,9 +18,8 @@ |
| + // MoveRecoveryGrant is one callback's authority over an exact move and record. |
| + // It exposes no store or arbitrary transition. Mutations consume the observation; |
| + // another action requires another pass. The grant expires when the hook returns. |
| +-// Plan 5 must align its owners to close the record before releasing the claim |
| +-// before wiring this hook. Existing release-first owners must finish their own |
| +-// closes: a serving claim with a live move record cannot acquire this grant. |
| ++// Record-first owners close before releasing the claim; LastMove repairs a |
| ++// crash between those writes on the next pass. |
| + type MoveRecoveryGrant struct { |
| + state *moveRecoveryGrantState |
| + w *ControlWriter |
| +@@ -131,7 +130,6 @@ |
| + RecoveryMoveClose MoveRecoveryRecordAction = "close" |
| + RecoveryMoveRolledBackClose MoveRecoveryRecordAction = "rolled-back-close" |
| + RecoveryMoveRevertClose MoveRecoveryRecordAction = "revert-close" |
| +- RecoveryMoveAbortComplete MoveRecoveryRecordAction = "abort-complete" |
| + RecoveryMoveDeadlineRollback MoveRecoveryRecordAction = "deadline-rollback" |
| + ) |
| + |
| +@@ -150,7 +148,7 @@ |
| + return ErrIdentity |
| + } |
| + switch action { |
| +- case RecoveryMoveClose, RecoveryMoveRolledBackClose, RecoveryMoveRevertClose, RecoveryMoveAbortComplete: |
| ++ case RecoveryMoveClose, RecoveryMoveRolledBackClose, RecoveryMoveRevertClose: |
| + for _, chore := range o.Claim.Obligations { |
| + if chore.Op == g.o.OpID { |
| + return ErrPending |
| +@@ -175,11 +173,6 @@ |
| + return ErrPending |
| + } |
| + mutate = provision.MoveRevertClose(guard, now) |
| +- case RecoveryMoveAbortComplete: |
| +- if o.Claim.Op.Kind != hostclaim.KindMoving || o.Claim.Op.Phase != hostclaim.PhaseAborting || len(o.Claim.Obligations) != 0 { |
| +- return ErrPending |
| +- } |
| +- mutate = provision.MoveGuardedAbortComplete(guard, now) |
| + case RecoveryMoveDeadlineRollback: |
| + if o.Claim.Op.Kind != hostclaim.KindLanding && !(o.Claim.Op.Kind == hostclaim.KindMoving && o.Claim.Op.Phase == hostclaim.PhaseSealed) { |
| + return ErrPending |
| +--- a/internal/hostops/recovery.go |
| ++++ b/internal/hostops/recovery.go |
| +@@ -36,10 +36,11 @@ |
| + } |
| + |
| + type Recovery struct { |
| +- MoveFenceAuthority hostclaim.FenceAuthority |
| +- Moves func(context.Context, *MoveRecoveryGrant) error |
| +- SweepAdmissions func(context.Context) error |
| +- Controller Controller |
| ++ ReportAutomaticLosses func(context.Context, string) error |
| ++ MoveFenceAuthority hostclaim.FenceAuthority |
| ++ Moves func(context.Context, *MoveRecoveryGrant) error |
| ++ SweepAdmissions func(context.Context) error |
| ++ Controller Controller |
| + // These are required integrations, not no-op defaults. |
| + Purge func(context.Context, hostclaim.Observed) error |
| + Placement func(context.Context, hostclaim.Observed) error |
| +@@ -71,6 +72,17 @@ |
| + } |
| + if obs.Absent { |
| + continue |
| ++ } |
| ++ if r.ReportAutomaticLosses != nil && len(obs.Claim.EvidenceRefs) > 0 { |
| ++ if err := r.ReportAutomaticLosses(ctx, host); err != nil { |
| ++ all = append(all, err) |
| ++ continue |
| + } |
| -+ if returnLane == nil { |
| -+ return fmt.Errorf("%w: a dirty release that died before its return owes the return lane's park", ErrMalformed) |
| ++ obs, err = hostclaim.Read(ctx, c.Store, host) |
| ++ if err != nil { |
| ++ all = append(all, err) |
| ++ continue |
| + } |
| -+ return nil |
| -+ } |
| -+ return fmt.Errorf("%w: op is restoring/%q, the lost target's release precedes the source's claim", ErrWrongState, c.Op.Phase) |
| -+ }, func(c *Claim) error { |
| -+ if owesStagingPark(c, r.OpID, r.TargetRegion) { |
| -+ return nil |
| -+ } |
| -+ _, err := queueReplicaPark(c, r.OpID, r.TargetRegion, lane, r.NewID) |
| -+ return err |
| -+ }, func(c *Claim) error { |
| -+ if returnLane == nil { |
| -+ return nil |
| -+ } |
| -+ if _, err := queueReplicaPark(c, r.OpID, r.SourceRegion, *returnLane, r.NewID); err != nil { |
| -+ return err |
| -+ } |
| -+ setJournal(c, journalReturnLane, returnLane.Location) |
| -+ return nil |
| -+ }) |
| -+ return inOneWrite(c, steps...) |
| -+} |
| ---- a/internal/hostclaim/move.go |
| -+++ b/internal/hostclaim/move.go |
| -@@ -959,6 +959,9 @@ |
| + } |
| + op := obs.Claim.Op |
| + // Zero time is unknown age, never old enough. Freeing empty parking is safe |
| +--- a/internal/provision/provision.go |
| ++++ b/internal/provision/provision.go |
| +@@ -1817,6 +1817,8 @@ |
| + // records (§7.10): the target box was declared lost before the move |
| + // committed. A recorded sentence, so it names no box; the box is in the |
| + // loss record LossRef names. |
| ++const RollbackTargetFenced = "target fenced" |
| ++ |
| + const RollbackDeclaredLost = "the target box was declared lost" |
| + |
| + // The reasons a move may record when it stops for an operator (§7.8 stuck, |
| +@@ -1940,6 +1942,8 @@ |
| + rb.Reason = RollbackCommitDeadline |
| + case RollbackProbeUnpriced: |
| + rb.Reason = RollbackProbeUnpriced |
| ++ case RollbackTargetFenced: |
| ++ rb.Reason = RollbackTargetFenced |
| + case RollbackDeclaredLost: |
| + rb.Reason = RollbackDeclaredLost |
| + default: |
| +@@ -1959,7 +1963,7 @@ |
| + func recordedRollbackReason(r string) bool { |
| + switch r { |
| + case RollbackConfigChanged, RollbackLandingDeadline, RollbackClaimWired, RollbackLandingLayout, RollbackInstallMismatch, |
| +- RollbackFilesOccupied, RollbackProbeAttempts, RollbackCommitDeadline, RollbackDeclaredLost, RollbackProbeUnpriced: |
| ++ RollbackFilesOccupied, RollbackProbeAttempts, RollbackCommitDeadline, RollbackDeclaredLost, RollbackTargetFenced, RollbackProbeUnpriced: |
| return true |
| } |
| - if !stringPresent(c.Op.Participants, target) { |
| -+ if ref := c.Op.Journal["fence."+target]; ref != "" && c.Op.Journal[journalFencedTargetReleased] == ref { |
| -+ return true |
| -+ } |
| - for _, id := range c.LossRefs { |
| - if _, ok := c.Op.Journal["loss."+id]; ok { |
| - return true |
| + return false |
| ``` |
| ## Task C: Hand recovery over and restore the source database (M3, M21) |
| -Use `claimRecoveryObservation` alongside the existing string-valued `claimRecoveryKind`. Remote dispatch accepts only restoring whose predecessor is the nonempty target writer. Every revert and never-admitted holder takes the local path. |
| +Use `claimRecoveryObservation` alongside the existing string-valued `claimRecoveryKind`. Remote dispatch accepts only restoring whose predecessor is the nonempty target writer. Initial source returns take the real stop wiring, which verifies the journalled source stamp, cut session, generation, canonical complete report and current guarded claim. This includes session zero and invents no park-local obligation. Later source sessions retain the local writer-stop path. |
| -`TargetReleased` journals `target_session` and `target_generation` from the holder in its own authority-releasing CAS; it invents neither for a never-admitted target. Recovery checks those fields and the complete ordinary report using `moves.StoppedKey`. Signed root receipts remain an alternative with exact tuple, digest and enrolled signing-key checks. The operation's signed loss and freshly verified fence remain alternatives. |
| +`TargetReleased` journals `target_session` and `target_generation` from the holder in its own authority-releasing CAS; it invents neither for a never-admitted target. Recovery checks those fields and the complete ordinary report using `moves.StoppedKey`. Signed root receipts remain an alternative with exact tuple, digest and enrolled signing-key checks. The operation's signed loss and consumed fence remain alternatives. Unconsumed fences still require fresh verification. |
| -Extend the claim's guarded artifact retirement to restoring and reverting. The first source return uses `JournalSourceStamp`, `src_gen`, cut session and complete source report. Revert journals its source stamp and generation before returning authority. Later source sessions use completed root writer-stop receipts; add their stop authority alongside landing and relocation. Inventory and descriptor guards still select the exact database and marker. |
| +Extend the claim's guarded artifact retirement to restoring and reverting. The first source return uses `JournalSourceStamp`, `src_gen`, cut session and complete source report. Revert journals its source stamp and generation before returning authority. Later source sessions use completed root writer-stop receipts; add their stop authority alongside landing and relocation. Before Prelaunch, require the exact StopInventoryDB source installation and transfer its physical identities under the activated guard. Rebind the stopped registry route and timers in one compare-and-swap transaction, then write the successor inventory through the existing transport. Preserve the old inventory for retirement and replay; refuse missing or changed ownership. Registry replay preserves all non-ownership columns and timer registrations. Inventory and descriptor guards still select the exact database and marker. |
| -Focused command: `go test ./internal/host ./internal/hostclaim ./internal/hostguard -run 'TestPlan12Recovery|TestPlan12TargetReleased|TestPlan9D2CrossBox|TestPlan9D4|TestPlan10J2|TestPlan11.*Relocation' -count=1`. |
| +Focused command: `go test ./internal/host ./internal/hostclaim ./internal/hostguard ./internal/registry -run 'TestPlan12Recovery|TestPlan12ConsumedFence|TestPlan12Rebind|TestPlan12TargetReleased|TestPlan9D2CrossBox|TestPlan9D4|TestPlan10J2|TestPlan11.*Relocation' -count=1`. |
| -Expected first failure: the target journal constants and recovery APIs are absent. Tests accept target journal, enrolled root receipt, signed loss and verified fence and refuse mismatched sessions, stamps, operations, enrollments, signatures and unverified fences. The Linux acceptance test runs the real wake assembly through admission, Prepare and Restore over source-stamped database files, for both recovery kinds, including clean-return coverage with the holder generation rewound to src_gen and a target predecessor admission. Its transport, lease and process kernel are fixtures; the revert case starts after an already completed source cut and stubs that local stop. It is not a live moves or privileged-kernel test. |
| +Expected first failure: the target journal constants and recovery APIs are absent. Tests accept target journal, enrolled root receipt, signed loss and verified fence and refuse mismatched sessions, stamps, operations, enrollments, signatures and unverified fences. The Linux acceptance test runs the real wake assembly through admission, Prepare and Restore over source-stamped database files, for both recovery kinds, including clean-return coverage with the holder generation rewound to src_gen and a target predecessor admission. Prelaunch and PrepareState are the real ClaimRoleAssembly methods. Only the cgroup preparation kernel seam is substituted; inventory, guard, registry, launch journal and restore execute normally. Reverts and never-admitted returns use the real stop wiring without a stop stub. Session-zero variants reach admission and Prepare; restoring, reverting and clean-return fixtures continue through Restore. Missing inventory, wrong enrollment and missing registry refuse before prelaunch. This is not a live fleet or privileged-kernel test. |
| ### Failing tests |
| @@ -1519,7 +2211,7 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| } |
| --- /dev/null |
| +++ b/internal/host/plan12_handover_test.go |
| -@@ -0,0 +1,173 @@ |
| +@@ -0,0 +1,244 @@ |
| +package host |
| + |
| +import ( |
| @@ -1528,13 +2220,17 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| ++ "fmt" |
| + "github.com/carlosframework/platform/internal/activator" |
| ++ "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/moves" |
| ++ "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/termfence" |
| + "os/exec" |
| ++ "path/filepath" |
| + "strings" |
| + "testing" |
| + "time" |
| @@ -1560,7 +2256,7 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + pub := f.signer.Public().(ed25519.PublicKey) |
| + h := claimCrossBoxHandover{Guard: f.guard, Evidence: f.claims, Box: "box-1", PublicKey: pub, FenceAuthority: plan12FenceAuthority{reject: mode == "unverified-fence"}} |
| + if mode == "fence" || mode == "unverified-fence" { |
| -+ record := termfence.Record{V: 1, Box: "box-2", Enrollment: strings.Repeat("a", 32), State: "terminated"} |
| ++ record := termfence.Record{V: 1, Box: "box-2", Enrollment: strings.Repeat("a", 32), State: "terminated", ObservedAt: time.Now().UTC().Format(time.RFC3339Nano)} |
| + raw, err := json.Marshal(record) |
| + if err != nil { |
| + t.Fatal(err) |
| @@ -1622,6 +2318,31 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + t.Fatal(err) |
| + } |
| + } |
| ++ h.Identity = handoverRecoveryIdentity(t, hostbox.Actor{Box: "box-1", Enrollment: "enrollment-1", Revision: 7, Region: "ie"}) |
| ++ h.Inventory, err = hostbox.OpenStopInventoryDB(filepath.Join(t.TempDir(), "inventory.db")) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ t.Cleanup(func() { h.Inventory.Close() }) |
| ++ h.Registry, err = registry.Open(filepath.Join(t.TempDir(), "registry.db")) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ t.Cleanup(func() { h.Registry.Close() }) |
| ++ row := registry.Route{Host: o.Host, Kind: "instance", DBPath: provision.NestedDBPath(f.guard.DataDir, o.Host), ClaimOwnership: registry.ClaimOwnership{ClaimIncarnation: o.Claim.Holder.Incarnation, ClaimStamp: o.Claim.Holder.Stamp, ClaimOwner: "old-owner", ClaimRowID: "old-row"}} |
| ++ if err = h.Registry.Upsert(row); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ if err = h.Inventory.WriteStopInventory(t.Context(), hostbox.StopInventory{Complete: true, Host: o.Host, Incarnation: row.ClaimIncarnation, InstallationStamp: row.ClaimStamp, Box: "box-1", Enrollment: "enrollment-1", DirectoryRevision: "7", Entries: []hostbox.StopInventoryEntry{{Kind: "route", ArtifactID: "route", ManagedRoot: "registry", RelativePath: o.Host, RowIdentity: "old-row", Ownership: "old-owner", Materialization: "materialized"}}}); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ o, err = hostclaim.Update(t.Context(), f.claims, o, func(c *hostclaim.Claim) error { |
| ++ c.Op.Journal[hostclaim.JournalSourceStamp] = fmt.Sprint(row.ClaimStamp) |
| ++ return nil |
| ++ }) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| + return f, o, h |
| +} |
| +func TestPlan12RecoveryPredecessorEvidence(t *testing.T) { |
| @@ -1693,42 +2414,121 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + } |
| + } |
| +} |
| ++ |
| ++func TestPlan12ConsumedFenceSurvivesRosterChange(t *testing.T) { |
| ++ f, o, h := plan12Handover(t, hostclaim.KindRestoring, "fence") |
| ++ rb := "rb_abcdefghijklmnopqrstuvwxyz" |
| ++ var ref hostclaim.FenceRef |
| ++ if err := json.Unmarshal([]byte(o.Claim.Op.Journal["fence.box-2"]), &ref); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ o, err := hostclaim.Update(t.Context(), f.claims, o, func(c *hostclaim.Claim) error { |
| ++ c.Op.Exec = "box-2" |
| ++ c.Op.Phase = hostclaim.PhaseReleasing |
| ++ c.Op.Intent["rollback"] = rb |
| ++ return nil |
| ++ }) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ fence, err := hostclaim.VerifyMoveTargetFence(t.Context(), f.claims, o, ref, h.FenceAuthority) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ n := 0 |
| ++ scope := func(loc string) hostclaim.ParkScope { |
| ++ return hostclaim.ParkScope{Location: loc, Incarnation: o.Claim.Holder.Incarnation, EvidenceAt: fence.ObservedAt(), RequestLimit: 2 * time.Minute} |
| ++ } |
| ++ o, err = hostclaim.Update(t.Context(), f.claims, o, hostclaim.ReleaseFencedTarget(hostclaim.FencedTargetRelease{OpID: o.Claim.Op.ID, RollbackID: rb, Target: "box-2", TargetRegion: "uk", SourceRegion: "ie", Fence: fence, Location: scope(o.Claim.Holder.Location), Lane: scope("moves/" + o.Host + "/" + o.Claim.Op.ID), NewID: func() string { n++; return fmt.Sprint("park-", n) }})) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ o, err = hostclaim.Update(t.Context(), f.claims, o, func(c *hostclaim.Claim) error { c.Op.Exec = "box-1"; c.Op.Phase = hostclaim.PhaseProbing; return nil }) |
| ++ if err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ h.FenceAuthority = plan12FenceAuthority{reject: true} |
| ++ if err := h.Before(t.Context(), o); err != nil { |
| ++ t.Fatal("consumed evidence depended on roster", err) |
| ++ } |
| ++ o.Claim.EvidenceRefs = nil |
| ++ if err := h.Before(t.Context(), o); err == nil { |
| ++ t.Fatal("journal alone accepted as durable loss") |
| ++ } |
| ++} |
| --- /dev/null |
| -+++ b/internal/hostclaim/plan12_target_release_test.go |
| -@@ -0,0 +1,30 @@ |
| -+package hostclaim |
| ++++ b/internal/host/plan12_receipt_test.go |
| +@@ -0,0 +1,67 @@ |
| ++package host |
| + |
| +import ( |
| -+ "strconv" |
| ++ "crypto/ed25519" |
| ++ "crypto/sha256" |
| ++ "encoding/hex" |
| ++ "encoding/json" |
| ++ "github.com/carlosframework/platform/internal/hostactors" |
| ++ "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/hostevidence" |
| ++ "github.com/carlosframework/platform/internal/release" |
| ++ "strings" |
| + "testing" |
| ++ "time" |
| +) |
| + |
| -+func TestPlan12TargetReleasedCapturesWriter(t *testing.T) { |
| -+ for _, admitted := range []bool{false, true} { |
| -+ c := lostTargetClaim(t, KindRestoring, PhaseReleasing) |
| -+ c.Lost, c.LossRefs, c.LostRevisions = nil, nil, nil |
| -+ delete(c.Op.Journal, "loss."+lostID) |
| -+ c.Op.Participants = []string{"ie-box", "uk-box"} |
| -+ c.Op.Exec = "uk-box" |
| -+ if !admitted { |
| -+ c.Holder.WriterBox = "ie-box" |
| -+ } |
| -+ r := lostRelease(c, false) |
| -+ if err := TargetReleased(lostOp, "uk-box", "uk", "evidence", r.Location, r.NewID)(c); err != nil { |
| -+ t.Fatal(err) |
| -+ } |
| -+ if admitted { |
| -+ if c.Op.Journal[JournalTargetSession] != strconv.FormatUint(c.Holder.Session, 10) || c.Op.Journal[JournalTargetGeneration] != c.Holder.Generation { |
| -+ t.Fatal(c.Op.Journal) |
| ++func TestPlan12RecoveryRootReceiptAlternative(t *testing.T) { |
| ++ for _, bad := range []string{"", "session", "stamp", "enrollment", "signature", "operation", "region", "digest"} { |
| ++ t.Run(bad, func(t *testing.T) { |
| ++ f, o, h := plan12Handover(t, hostclaim.KindRestoring, "stop") |
| ++ key := ed25519.NewKeyFromSeed([]byte(strings.Repeat("b", 32))) |
| ++ en := strings.Repeat("a", 32) |
| ++ d := hostactors.Directory{Store: f.claims, PublicKey: f.signer.Public().(ed25519.PublicKey), Bucket: "bucket"} |
| ++ roster := hostactors.Roster{V: 1, Revision: 1, DeploymentBucket: "bucket", Boxes: map[string]hostactors.Box{"box-2": {Enrollment: en, BoxID: "box-2", Label: "box-2", Namespace: "platform", Region: "uk", Pool: "default", State: "active", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-2", InstanceID: "i-target"}, EvidenceSHA256: strings.Repeat("d", 64), CensusPublicKey: key.Public().(ed25519.PublicKey)}}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} |
| ++ wire, _ := release.SignRootEnvelope(release.RootSigner(f.signer), roster) |
| ++ if _, err := f.claims.Put(t.Context(), hostactors.Key, wire, ""); err != nil { |
| ++ t.Fatal(err) |
| + } |
| -+ } else if _, ok := c.Op.Journal[JournalTargetSession]; ok { |
| -+ t.Fatal("invented target session") |
| -+ } |
| ++ r := StopReceipt{V: 1, ID: "stop", Kind: "stopped", Host: o.Host, Incarnation: o.Claim.Holder.Incarnation, InstallationStamp: 2, WriterSession: 2, OperationID: "m1", Phase: "releasing", ObligationID: "writer-session-2", Box: "box-2", Region: "uk", Enrollment: en, EvidenceAt: time.Now(), RequestLimitNS: int64(2 * time.Minute), ClaimGen: 1, FenceID: "fence", InventorySHA256: strings.Repeat("e", 64), LocalJournalSHA256: strings.Repeat("f", 64), ProducerDrainDigest: strings.Repeat("a", 64), CgroupIdentities: []string{"root-role"}} |
| ++ switch bad { |
| ++ case "session": |
| ++ r.WriterSession++ |
| ++ case "stamp": |
| ++ r.InstallationStamp++ |
| ++ case "enrollment": |
| ++ r.Enrollment = strings.Repeat("e", 32) |
| ++ case "operation": |
| ++ r.OperationID = "other" |
| ++ case "region": |
| ++ r.Region = "ie" |
| ++ } |
| ++ body, _ := json.Marshal(r) |
| ++ receiptKey := stopEvidenceKey(o.Host, "m1", "box-2", "stop") |
| ++ sum := sha256.Sum256(body) |
| ++ signer := key |
| ++ if bad == "signature" { |
| ++ signer = f.signer |
| ++ } |
| ++ if err := hostevidence.Publish(t.Context(), f.claims, signer, receiptKey, body); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ h.Directory = d |
| ++ o.Claim.Op.Journal["target_stop_receipt"] = receiptKey |
| ++ o.Claim.Op.Journal["target_stop_sha256"] = hex.EncodeToString(sum[:]) |
| ++ o.Claim.Op.Journal[hostclaim.JournalTargetStamp] = "2" |
| ++ // Remove the unsigned report path: the receipt alone must suffice. |
| ++ delete(o.Claim.Op.Journal, hostclaim.JournalEvidenceTarget) |
| ++ if bad == "digest" { |
| ++ o.Claim.Op.Journal["target_stop_sha256"] = strings.Repeat("0", 64) |
| ++ } |
| ++ err := h.recoveryEvidence(t.Context(), o) |
| ++ if (err == nil) != (bad == "") { |
| ++ t.Fatal(bad, err) |
| ++ } |
| ++ }) |
| + } |
| +} |
| --- /dev/null |
| +++ b/internal/host/plan12_restore_linux_test.go |
| -@@ -0,0 +1,133 @@ |
| +@@ -0,0 +1,208 @@ |
| +//go:build linux |
| + |
| +package host |
| @@ -1737,12 +2537,15 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + "context" |
| + "encoding/json" |
| + "github.com/carlosframework/platform/internal/activator" |
| ++ "github.com/carlosframework/platform/internal/cgroup" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/moves" |
| + "github.com/carlosframework/platform/internal/park" |
| ++ "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| ++ "golang.org/x/sys/unix" |
| + "os" |
| + "os/exec" |
| + "path/filepath" |
| @@ -1751,15 +2554,28 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| +) |
| + |
| +func TestPlan12RecoveryRestoreSourceDatabase(t *testing.T) { |
| -+ for _, name := range []string{hostclaim.KindRestoring, hostclaim.KindReverting, "clean-return"} { |
| ++ for _, name := range []string{hostclaim.KindRestoring, hostclaim.KindReverting, "clean-return", "never-restoring", "never-reverting", "never-clean-return", "missing-inventory", "wrong-inventory", "missing-registry"} { |
| + t.Run(name, func(t *testing.T) { |
| -+ kind, clean := name, name == "clean-return" |
| ++ kind, clean := name, name == "clean-return" || name == "never-clean-return" |
| ++ bad := name == "missing-inventory" || name == "wrong-inventory" || name == "missing-registry" |
| ++ if bad { |
| ++ kind = hostclaim.KindRestoring |
| ++ } |
| ++ if name == "never-restoring" { |
| ++ kind = hostclaim.KindRestoring |
| ++ } |
| ++ if name == "never-reverting" { |
| ++ kind = hostclaim.KindReverting |
| ++ } |
| ++ never := name == "never-restoring" || name == "never-reverting" || name == "never-clean-return" |
| + if clean { |
| + kind = hostclaim.KindRestoring |
| + } |
| + f, o, h := plan12Handover(t, hostclaim.KindRestoring, "stop") |
| + stamp := o.Claim.Holder.Stamp |
| -+ db := filepath.Join(f.guard.DataDir, "state", "data.db") |
| ++ db := provision.NestedDBPath(f.guard.DataDir, o.Host) |
| ++ dbRel, _ := filepath.Rel(f.guard.DataDir, db) |
| ++ dirRel, _ := filepath.Rel(f.guard.DataDir, filepath.Dir(db)) |
| + if err := os.MkdirAll(filepath.Dir(db), 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| @@ -1783,7 +2599,7 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + c.Op.Kind = kind |
| + c.Op.Journal[hostclaim.JournalSourceStamp] = strconv.FormatUint(stamp, 10) |
| + c.Op.Journal[hostclaim.JournalSrcGen] = "src" |
| -+ c.Op.Journal[hostclaim.JournalCutSession] = "1" |
| ++ c.Op.Journal[hostclaim.JournalCutSession] = "2" |
| + c.Op.Journal[hostclaim.JournalEvidenceSource] = key |
| + if kind == hostclaim.KindRestoring { |
| + c.Holder.Admissions = []hostclaim.AdmissionRecord{{Session: 2, Box: "box-2", Generation: "target-gen", Mode: "cold", Region: "uk", Predecessor: "src", PredecessorRegion: "ie"}} |
| @@ -1797,13 +2613,29 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + c.Holder.Generation = "src" |
| + c.Holder.Admissions = []hostclaim.AdmissionRecord{{Session: 2, Box: "box-1", Generation: "src", Mode: "cold", Region: "ie"}} |
| + } |
| ++ if never { |
| ++ c.Holder.WriterBox = "" |
| ++ c.Holder.Session = 0 |
| ++ c.Holder.Generation = "" |
| ++ c.Holder.Admissions = nil |
| ++ c.Op.Journal[hostclaim.JournalCutSession] = "0" |
| ++ c.Op.Journal[hostclaim.JournalSrcGen] = "" |
| ++ } |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| ++ if never { |
| ++ if err = os.Remove(db); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ if err = os.Remove(db + ".gen"); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ } |
| + local := park.Local{Guard: f.guard, Claims: f.claims, DataDir: f.guard.DataDir, Box: "box-1"} |
| -+ if err = local.RecordInventory(t.Context(), o.Host, hostguard.Captured{Incarnation: o.Claim.Holder.Incarnation, Stamp: stamp}, "flat", []string{"state/data.db", "state/data.db.gen"}, func(*hostclaim.Claim) error { return nil }); err != nil { |
| ++ if err = local.RecordInventory(t.Context(), o.Host, hostguard.Captured{Incarnation: o.Claim.Holder.Incarnation, Stamp: stamp}, "flat", []string{dbRel, dbRel + ".gen"}, func(*hostclaim.Claim) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + o = f.read() |
| @@ -1814,9 +2646,41 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + c.wake.ProbeReady = func(context.Context, *exec.Cmd, registry.Route, ClaimLaunchInputs, string) error { return nil } |
| + retirement := claimArtifactRetirement{Local: local, Continue: hostclaim.SessionContinues} |
| + c.wake.RetireArtifacts, c.wake.RecoverArtifacts = retirement.Retire, retirement.Recover |
| -+ // The cut's complete report is already durable. This fixture has no source launch journal. |
| -+ if kind == hostclaim.KindReverting { |
| -+ c.wake.Stop = func(context.Context, hostclaim.Observed) error { return nil } |
| ++ c.assembly.Guard = f.guard |
| ++ c.assembly.bindClaimWakeStop() |
| ++ inv, e := h.Inventory.ReadInstallation(t.Context(), o.Host, o.Claim.Holder.Incarnation, stamp) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ var stat unix.Stat_t |
| ++ if e = unix.Lstat(filepath.Dir(db), &stat); e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ inv.Entries = append(inv.Entries, hostbox.StopInventoryEntry{Kind: "path", ArtifactID: "state", ManagedRoot: "data", RelativePath: dirRel, Ownership: "old-owner", Device: uint64(stat.Dev), Inode: stat.Ino, Type: unix.S_IFDIR, Materialization: "materialized"}) |
| ++ if e = h.Inventory.WriteStopInventory(t.Context(), inv); e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ roles := &ClaimRoleAssembly{Manager: c.wake.Manager, Journal: f.journal, Inventory: h.Inventory, ManagedRoots: map[string]string{"data": f.guard.DataDir}, Identity: h.Identity, BootID: "boot-1", ControllerID: "controller-1"} |
| ++ roles.prepareGroup = func(name string, _ cgroup.Limit) (*cgroup.Prepared, hostbox.LaunchGroup, error) { |
| ++ return nil, hostbox.LaunchGroup{RelativePath: "carlos/" + name, Device: 1, Inode: 1}, nil |
| ++ } |
| ++ c.wake.Prelaunch, c.wake.PrepareState = roles.Prelaunch, roles.PrepareState |
| ++ if name == "missing-inventory" { |
| ++ replacement, e := hostbox.OpenStopInventoryDB(filepath.Join(t.TempDir(), "empty.db")) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ defer replacement.Close() |
| ++ h.Inventory = replacement |
| ++ } |
| ++ if name == "wrong-inventory" { |
| ++ inv.Enrollment = "other" |
| ++ if e = h.Inventory.WriteStopInventory(t.Context(), inv); e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ } |
| ++ if name == "missing-registry" { |
| ++ h.Registry = nil |
| + } |
| + wireClaimAdmission(c.wake, "box-1", c.wake.AfterAdmission, h, nil) |
| + if err := c.wake.stopPredecessor(t.Context(), o); err != nil { |
| @@ -1830,6 +2694,12 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + c.wake.pending[o.Host] = &claimWakeAttempt{observed: o, held: recoveryHeld{}, ctx: live, cancel: cancel, done: done} |
| + c.wake.mu.Unlock() |
| + admitted, err := c.runtime.Admit(t.Context(), registry.Route{Host: o.Host}, o, activator.StartPolicy{Probe: true}) |
| ++ if bad { |
| ++ if err == nil { |
| ++ t.Fatal("unproved inventory admitted") |
| ++ } |
| ++ return |
| ++ } |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| @@ -1837,13 +2707,16 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| ++ if never { |
| ++ return |
| ++ } |
| + if err := launch.Restore(t.Context()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := os.ReadFile(db + ".gen"); err != nil || string(got) == "src" { |
| + t.Fatal(string(got), err) |
| + } |
| -+ if tag := f.read().Claim.Inventory["box-1"].Paths["state/data.db"]; tag != admitted.Claim.Holder.Stamp { |
| ++ if tag := f.read().Claim.Inventory["box-1"].Paths[dbRel]; tag != admitted.Claim.Holder.Stamp { |
| + t.Fatal("old ownership", tag) |
| + } |
| + }) |
| @@ -1857,77 +2730,104 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + |
| +func (in plan12ReturnInputs) Select(ctx context.Context, rt registry.Route, o hostclaim.Observed, p activator.StartPolicy) (ClaimLaunchInputs, error) { |
| + selected, err := in.plan9D2Inputs.Select(ctx, rt, o, p) |
| ++ selected.Admission.KeepStamp = false |
| ++ selected.DBPath = provision.NestedDBPath(in.f.guard.DataDir, o.Host) |
| + if in.clean { |
| + selected.Admission.Coverage = "clean-return" |
| + } |
| + return selected, err |
| +} |
| --- /dev/null |
| -+++ b/internal/host/plan12_receipt_test.go |
| -@@ -0,0 +1,67 @@ |
| -+package host |
| ++++ b/internal/hostclaim/plan12_target_release_test.go |
| +@@ -0,0 +1,30 @@ |
| ++package hostclaim |
| + |
| +import ( |
| -+ "crypto/ed25519" |
| -+ "crypto/sha256" |
| -+ "encoding/hex" |
| -+ "encoding/json" |
| -+ "github.com/carlosframework/platform/internal/hostactors" |
| -+ "github.com/carlosframework/platform/internal/hostclaim" |
| -+ "github.com/carlosframework/platform/internal/hostevidence" |
| -+ "github.com/carlosframework/platform/internal/release" |
| -+ "strings" |
| ++ "strconv" |
| + "testing" |
| -+ "time" |
| +) |
| + |
| -+func TestPlan12RecoveryRootReceiptAlternative(t *testing.T) { |
| -+ for _, bad := range []string{"", "session", "stamp", "enrollment", "signature", "operation", "region", "digest"} { |
| -+ t.Run(bad, func(t *testing.T) { |
| -+ f, o, h := plan12Handover(t, hostclaim.KindRestoring, "stop") |
| -+ key := ed25519.NewKeyFromSeed([]byte(strings.Repeat("b", 32))) |
| -+ en := strings.Repeat("a", 32) |
| -+ d := hostactors.Directory{Store: f.claims, PublicKey: f.signer.Public().(ed25519.PublicKey), Bucket: "bucket"} |
| -+ roster := hostactors.Roster{V: 1, Revision: 1, DeploymentBucket: "bucket", Boxes: map[string]hostactors.Box{"box-2": {Enrollment: en, BoxID: "box-2", Label: "box-2", Namespace: "platform", Region: "uk", Pool: "default", State: "active", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-2", InstanceID: "i-target"}, EvidenceSHA256: strings.Repeat("d", 64), CensusPublicKey: key.Public().(ed25519.PublicKey)}}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} |
| -+ wire, _ := release.SignRootEnvelope(release.RootSigner(f.signer), roster) |
| -+ if _, err := f.claims.Put(t.Context(), hostactors.Key, wire, ""); err != nil { |
| -+ t.Fatal(err) |
| ++func TestPlan12TargetReleasedCapturesWriter(t *testing.T) { |
| ++ for _, admitted := range []bool{false, true} { |
| ++ c := lostTargetClaim(t, KindRestoring, PhaseReleasing) |
| ++ c.Lost, c.LossRefs, c.LostRevisions = nil, nil, nil |
| ++ delete(c.Op.Journal, "loss."+lostID) |
| ++ c.Op.Participants = []string{"ie-box", "uk-box"} |
| ++ c.Op.Exec = "uk-box" |
| ++ if !admitted { |
| ++ c.Holder.WriterBox = "ie-box" |
| ++ } |
| ++ r := lostRelease(c, false) |
| ++ if err := TargetReleased(lostOp, "uk-box", "uk", "evidence", r.Location, r.NewID)(c); err != nil { |
| ++ t.Fatal(err) |
| ++ } |
| ++ if admitted { |
| ++ if c.Op.Journal[JournalTargetSession] != strconv.FormatUint(c.Holder.Session, 10) || c.Op.Journal[JournalTargetGeneration] != c.Holder.Generation { |
| ++ t.Fatal(c.Op.Journal) |
| + } |
| -+ r := StopReceipt{V: 1, ID: "stop", Kind: "stopped", Host: o.Host, Incarnation: o.Claim.Holder.Incarnation, InstallationStamp: 2, WriterSession: 2, OperationID: "m1", Phase: "releasing", ObligationID: "writer-session-2", Box: "box-2", Region: "uk", Enrollment: en, EvidenceAt: time.Now(), RequestLimitNS: int64(2 * time.Minute), ClaimGen: 1, FenceID: "fence", InventorySHA256: strings.Repeat("e", 64), LocalJournalSHA256: strings.Repeat("f", 64), ProducerDrainDigest: strings.Repeat("a", 64), CgroupIdentities: []string{"root-role"}} |
| -+ switch bad { |
| -+ case "session": |
| -+ r.WriterSession++ |
| -+ case "stamp": |
| -+ r.InstallationStamp++ |
| -+ case "enrollment": |
| -+ r.Enrollment = strings.Repeat("e", 32) |
| -+ case "operation": |
| -+ r.OperationID = "other" |
| -+ case "region": |
| -+ r.Region = "ie" |
| ++ } else if _, ok := c.Op.Journal[JournalTargetSession]; ok { |
| ++ t.Fatal("invented target session") |
| ++ } |
| ++ } |
| ++} |
| +--- /dev/null |
| ++++ b/internal/registry/plan12_recovery_test.go |
| +@@ -0,0 +1,59 @@ |
| ++package registry |
| ++ |
| ++import ( |
| ++ "errors" |
| ++ "path/filepath" |
| ++ "testing" |
| ++) |
| ++ |
| ++func TestPlan12RebindClaimRows(t *testing.T) { |
| ++ for _, mode := range []string{"ok", "changed-route", "changed-owner", "changed-payload", "extra-timer"} { |
| ++ t.Run(mode, func(t *testing.T) { |
| ++ r, e := Open(filepath.Join(t.TempDir(), "registry.db")) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| + } |
| -+ body, _ := json.Marshal(r) |
| -+ receiptKey := stopEvidenceKey(o.Host, "m1", "box-2", "stop") |
| -+ sum := sha256.Sum256(body) |
| -+ signer := key |
| -+ if bad == "signature" { |
| -+ signer = f.signer |
| ++ defer r.Close() |
| ++ row := Route{Host: "app.example", Kind: "instance", ClaimOwnership: ClaimOwnership{"inc", 1, "old", "old-row"}} |
| ++ if e = r.Upsert(row); e != nil { |
| ++ t.Fatal(e) |
| + } |
| -+ if err := hostevidence.Publish(t.Context(), f.claims, signer, receiptKey, body); err != nil { |
| -+ t.Fatal(err) |
| ++ saved, e := r.Get(row.Host) |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| + } |
| -+ h.Directory = d |
| -+ o.Claim.Op.Journal["target_stop_receipt"] = receiptKey |
| -+ o.Claim.Op.Journal["target_stop_sha256"] = hex.EncodeToString(sum[:]) |
| -+ o.Claim.Op.Journal[hostclaim.JournalTargetStamp] = "2" |
| -+ // Remove the unsigned report path: the receipt alone must suffice. |
| -+ delete(o.Claim.Op.Journal, hostclaim.JournalEvidenceTarget) |
| -+ if bad == "digest" { |
| -+ o.Claim.Op.Journal["target_stop_sha256"] = strings.Repeat("0", 64) |
| ++ old := ColdRows{Route: *saved} |
| ++ next := old |
| ++ next.Route.ClaimOwnership = ClaimOwnership{"inc", 2, "new", "new-row"} |
| ++ switch mode { |
| ++ case "changed-route": |
| ++ r.sql.Exec("UPDATE routes SET version='other' WHERE host=?", row.Host) |
| ++ case "changed-owner": |
| ++ r.sql.Exec("UPDATE routes SET claim_owner='other' WHERE host=?", row.Host) |
| ++ case "changed-payload": |
| ++ next.Route.Version = "other" |
| ++ case "extra-timer": |
| ++ if e = r.UpsertTimer(Timer{Host: row.Host, Name: "extra", Kind: "cron", Spec: "* * * * *"}); e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| + } |
| -+ err := h.recoveryEvidence(t.Context(), o) |
| -+ if (err == nil) != (bad == "") { |
| -+ t.Fatal(bad, err) |
| ++ e = r.RebindClaimRows(t.Context(), old, next) |
| ++ if mode != "ok" { |
| ++ if !errors.Is(e, ErrClaimOwnership) { |
| ++ t.Fatal("accepted changed installation", e) |
| ++ } |
| ++ return |
| ++ } |
| ++ if e != nil { |
| ++ t.Fatal(e) |
| ++ } |
| ++ if e = r.RebindClaimRows(t.Context(), old, next); e != nil { |
| ++ t.Fatal("replay", e) |
| ++ } |
| ++ got, e := r.Get(row.Host) |
| ++ if e != nil || *got != next.Route { |
| ++ t.Fatal(got, e) |
| + } |
| + }) |
| + } |
| @@ -1938,9 +2838,33 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| <!-- apply: C implementation --> |
| ```diff |
| +--- a/internal/host/claimconstruction.go |
| ++++ b/internal/host/claimconstruction.go |
| +@@ -22,6 +22,7 @@ |
| + "github.com/carlosframework/platform/internal/controlsock" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| ++ "github.com/carlosframework/platform/internal/hostfence" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| +@@ -282,7 +283,12 @@ |
| + wake.AfterAdmission = coldAdmissionHook(guard, owners.Identity, controlsock.Path) |
| + wake.RetireArtifacts = func(context.Context, hostclaim.Observed, ClaimLaunchInputs, string) error { return hostbox.ErrChanged } |
| + if movesC1 { |
| +- handover := claimCrossBoxHandover{Guard: guard, Evidence: store, Box: actor.Box} |
| ++ handover := claimCrossBoxHandover{Guard: guard, Evidence: store, Box: actor.Box, Inventory: inventory, Identity: owners.Identity, Registry: r.Registry} |
| ++ if owners.RetirementAuthority != nil { |
| ++ handover.PublicKey = owners.RetirementAuthority.Directory.PublicKey |
| ++ handover.Directory = owners.RetirementAuthority.Directory |
| ++ handover.FenceAuthority = hostfence.ClaimFenceAuthority{Directory: owners.RetirementAuthority.Directory, Now: time.Now} |
| ++ } |
| + var relocation *ClaimRelocationOwners |
| + if groupD.Dispatch.Relocation != nil { |
| + relocation = &owners.Relocation |
| --- a/internal/host/claimhandover_crossbox.go |
| +++ b/internal/host/claimhandover_crossbox.go |
| -@@ -2,8 +2,10 @@ |
| +@@ -2,12 +2,15 @@ |
| import ( |
| "context" |
| @@ -1951,7 +2875,12 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| "github.com/carlosframework/platform/internal/hostbox" |
| "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/hostguard" |
| -@@ -39,12 +41,18 @@ |
| + "github.com/carlosframework/platform/internal/moves" |
| ++ "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + ) |
| + |
| +@@ -39,12 +42,21 @@ |
| // installation stamp (item 4), so the local record this box initialized for |
| // the landing installation is the one the session runs under. |
| type claimCrossBoxHandover struct { |
| @@ -1963,6 +2892,9 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + Directory hostactors.Directory |
| + Guard hostguard.Guard |
| + Evidence release.Store |
| ++ Inventory *hostbox.StopInventoryDB |
| ++ Registry *registry.Registry |
| ++ Identity hostbox.IdentityProvider |
| + Box string |
| } |
| @@ -1973,7 +2905,7 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| if h.Evidence == nil || h.Box == "" || prior.Claim == nil || prior.Claim.Holder == nil || admitted.Claim == nil || admitted.Claim.Holder == nil || prior.Host != admitted.Host || prior.Absent || admitted.Absent { |
| return hostbox.ErrChanged |
| } |
| -@@ -115,6 +123,9 @@ |
| +@@ -115,6 +127,9 @@ |
| } |
| func (h claimCrossBoxHandover) Before(ctx context.Context, prior hostclaim.Observed) error { |
| @@ -1983,47 +2915,44 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| if _, err := h.sourceEvidence(ctx, prior); err != nil { |
| return err |
| } |
| -@@ -171,5 +182,5 @@ |
| +@@ -164,12 +179,17 @@ |
| + } |
| + } |
| + w.BeforeAdmission = claimPredecessorStop(w, box, before) |
| +- w.AfterAdmission = claimAdmissionHandover(box, local, after) |
| ++ w.AfterAdmission = claimAdmissionHandover(box, func(ctx context.Context, prior, admitted hostclaim.Observed) error { |
| ++ if claimInitialSourceRecovery(prior, box) { |
| ++ return landing.afterRecovery(ctx, prior, admitted) |
| ++ } |
| ++ return local(ctx, prior, admitted) |
| ++ }, after) |
| + } |
| + |
| + func claimRemotePredecessor(prior hostclaim.Observed, box string) bool { |
| if prior.Claim == nil || prior.Claim.Holder == nil || prior.Claim.Holder.WriterBox == box { |
| return false |
| } |
| - return prior.Claim.Op.Kind == hostclaim.KindRelocating || (prior.Claim.Op.Kind == hostclaim.KindLanding && prior.Claim.Holder.WriterBox != "") |
| + return (prior.Claim.Op.Kind == hostclaim.KindRestoring && prior.Claim.Holder.WriterBox != "" && prior.Claim.Holder.WriterBox == prior.Claim.Op.Intent["target"]) || prior.Claim.Op.Kind == hostclaim.KindRelocating || (prior.Claim.Op.Kind == hostclaim.KindLanding && prior.Claim.Holder.WriterBox != "") |
| } |
| ---- a/internal/host/claimconstruction.go |
| -+++ b/internal/host/claimconstruction.go |
| -@@ -22,6 +22,7 @@ |
| - "github.com/carlosframework/platform/internal/controlsock" |
| - "github.com/carlosframework/platform/internal/hostbox" |
| - "github.com/carlosframework/platform/internal/hostclaim" |
| -+ "github.com/carlosframework/platform/internal/hostfence" |
| - "github.com/carlosframework/platform/internal/hostguard" |
| - "github.com/carlosframework/platform/internal/hostops" |
| - "github.com/carlosframework/platform/internal/hostwrite" |
| -@@ -283,6 +284,11 @@ |
| - wake.RetireArtifacts = func(context.Context, hostclaim.Observed, ClaimLaunchInputs, string) error { return hostbox.ErrChanged } |
| - if movesC1 { |
| - handover := claimCrossBoxHandover{Guard: guard, Evidence: store, Box: actor.Box} |
| -+ if owners.RetirementAuthority != nil { |
| -+ handover.PublicKey = owners.RetirementAuthority.Directory.PublicKey |
| -+ handover.Directory = owners.RetirementAuthority.Directory |
| -+ handover.FenceAuthority = hostfence.ClaimFenceAuthority{Directory: owners.RetirementAuthority.Directory, Now: time.Now} |
| -+ } |
| - var relocation *ClaimRelocationOwners |
| - if groupD.Dispatch.Relocation != nil { |
| - relocation = &owners.Relocation |
| --- /dev/null |
| +++ b/internal/host/claimhandover_recovery.go |
| -@@ -0,0 +1,143 @@ |
| +@@ -0,0 +1,316 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| ++ "crypto/sha256" |
| ++ "database/sql" |
| ++ "encoding/hex" |
| + "encoding/json" |
| ++ "errors" |
| ++ "fmt" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/moves" |
| ++ "github.com/carlosframework/platform/internal/registry" |
| + "strconv" |
| +) |
| + |
| @@ -2045,6 +2974,9 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + c := o.Claim |
| + p := recoveryPredecessor(o) |
| + op := c.Op |
| ++ if claimInitialSourceRecovery(o, h.Box) { |
| ++ return h.sourceRecoveryEvidence(ctx, o) |
| ++ } |
| + if op.ID == "" || op.Phase != hostclaim.PhaseProbing || op.Exec != h.Box || op.Intent["source"] != h.Box || p.Box == "" || p.Box == h.Box || p.Box != op.Intent["target"] || p.Box != c.Holder.WriterBox || p.Session != c.Holder.Session { |
| + return hostbox.ErrChanged |
| + } |
| @@ -2068,6 +3000,9 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + } |
| + return hostclaim.ErrLossRecord |
| + } |
| ++ if _, ok := hostclaim.ConsumedMoveTargetFence(c); ok { |
| ++ return nil |
| ++ } |
| + if op.Journal["fence."+p.Box] != "" { |
| + var ref hostclaim.FenceRef |
| + if json.Unmarshal([]byte(op.Journal["fence."+p.Box]), &ref) != nil || op.Intent["target_enrollment"] == "" || ref.Enrollment != op.Intent["target_enrollment"] { |
| @@ -2133,7 +3068,7 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + if err != nil { |
| + return err |
| + } |
| -+ if lossID != "" && (a.LossID != lossID || a.Coverage != "" || a.WritesLost == nil || a.WritesLost.Box != p.Box || a.WritesLost.Session != p.Session) { |
| ++ if lossID != "" && (a.LossID != lossID || a.Coverage != "" || (p.Session != 0 && (a.WritesLost == nil || a.WritesLost.Box != p.Box || a.WritesLost.Session != p.Session))) { |
| + return hostclaim.ErrLossRecord |
| + } |
| + if lossID == "" && a.LossID != "" { |
| @@ -2156,7 +3091,168 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| -+ }, func() error { return nil }) |
| ++ }, func() error { return h.installRecoveryInventory(ctx, prior, admitted) }) |
| ++} |
| ++ |
| ++// The first return to the cut installation has not launched under this operation. |
| ++// Later source sessions cannot reuse the cut's stop evidence. |
| ++func claimInitialSourceRecovery(o hostclaim.Observed, box string) bool { |
| ++ if !claimRecoveryObservation(o) || o.Claim.Holder == nil { |
| ++ return false |
| ++ } |
| ++ c, h := o.Claim, o.Claim.Holder |
| ++ cut, e := strconv.ParseUint(c.Op.Journal[hostclaim.JournalCutSession], 10, 64) |
| ++ stamp, se := strconv.ParseUint(c.Op.Journal[hostclaim.JournalSourceStamp], 10, 64) |
| ++ return e == nil && se == nil && stamp > 0 && h.Stamp == stamp && c.Op.ID != "" && c.Op.Phase == hostclaim.PhaseProbing && c.Op.Exec == box && c.Op.Intent["source"] == box && h.Session == cut && (h.WriterBox == box || h.Session == 0 && h.WriterBox == "" && len(h.Admissions) == 0) && h.Generation == c.Op.Journal[hostclaim.JournalSrcGen] |
| ++} |
| ++func (h claimCrossBoxHandover) sourceRecoveryEvidence(ctx context.Context, o hostclaim.Observed) error { |
| ++ if !claimInitialSourceRecovery(o, h.Box) || h.Evidence == nil { |
| ++ return hostbox.ErrChanged |
| ++ } |
| ++ c := o.Claim |
| ++ key := moves.StoppedKey(moves.ReportPrefix(c.Holder.Account, c.Holder.App, o.Host, c.Op.ID), "source") |
| ++ if c.Op.Journal[hostclaim.JournalEvidenceSource] != key { |
| ++ return hostbox.ErrChanged |
| ++ } |
| ++ report, err := readStoppedReport(ctx, h.Evidence, key) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if report.Role != "source" || report.Box != h.Box || report.Host != o.Host || report.OpID != c.Op.ID || report.Incarnation != c.Holder.Incarnation { |
| ++ return hostbox.ErrChanged |
| ++ } |
| ++ return nil |
| ++} |
| ++ |
| ++// The guard is already held. Preserve exact inventory entries and their physical |
| ++// identities; only ownership moves to the admitted installation. The old inventory |
| ++// remains available for retirement and replay. Never infer an empty installation. |
| ++func (h claimCrossBoxHandover) installRecoveryInventory(ctx context.Context, prior, admitted hostclaim.Observed) error { |
| ++ if h.Inventory == nil { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ actor, err := h.Identity.Verify(ctx) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if actor.Box != h.Box { |
| ++ return hostbox.ErrChanged |
| ++ } |
| ++ old, next := prior.Claim.Holder, admitted.Claim.Holder |
| ++ stamp, err := strconv.ParseUint(prior.Claim.Op.Journal[hostclaim.JournalSourceStamp], 10, 64) |
| ++ if err != nil || stamp == 0 || stamp >= next.Stamp { |
| ++ return hostbox.ErrChanged |
| ++ } |
| ++ inv, err := h.Inventory.ReadInstallation(ctx, prior.Host, old.Incarnation, stamp) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if !inv.Complete || inv.Host != prior.Host || inv.Incarnation != old.Incarnation || inv.InstallationStamp != stamp || inv.Box != actor.Box || inv.Enrollment != actor.Enrollment { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ current, err := h.Inventory.ReadInstallation(ctx, prior.Host, next.Incarnation, next.Stamp) |
| ++ if err == nil { |
| ++ if !current.Complete || current.Box != actor.Box || current.Enrollment != actor.Enrollment || current.DirectoryRevision != fmt.Sprint(actor.Revision) { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ if h.Registry == nil { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ row, e := h.Registry.Get(prior.Host) |
| ++ if e != nil { |
| ++ return e |
| ++ } |
| ++ return hostbox.ValidateAgentInventory(current, row, h.Guard.DataDir) |
| ++ } |
| ++ if !errors.Is(err, sql.ErrNoRows) { |
| ++ return err |
| ++ } |
| ++ inv.InstallationStamp, inv.DirectoryRevision = next.Stamp, fmt.Sprint(actor.Revision) |
| ++ sum := sha256.Sum256([]byte(fmt.Sprintf("%s\x00%s\x00%d\x00%s", prior.Host, next.Incarnation, next.Stamp, actor.Box))) |
| ++ owner := hex.EncodeToString(sum[:]) |
| ++ if err := h.rebindRecoveryRows(ctx, &inv, stamp, owner); err != nil { |
| ++ return err |
| ++ } |
| ++ for i := range inv.Entries { |
| ++ inv.Entries[i].Ownership = owner |
| ++ } |
| ++ return h.Inventory.WriteStopInventory(ctx, inv) |
| ++} |
| ++ |
| ++func (h claimCrossBoxHandover) rebindRecoveryRows(ctx context.Context, inv *hostbox.StopInventory, oldStamp uint64, owner string) error { |
| ++ if h.Registry == nil { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ row, err := h.Registry.Get(inv.Host) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if row == nil || row.ClaimIncarnation != inv.Incarnation || (row.ClaimStamp != oldStamp && row.ClaimStamp != inv.InstallationStamp) || !row.ClaimOwnership.Valid() { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ timers, err := h.Registry.TimersFor(inv.Host) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ old := registry.ColdRows{Route: *row, Timers: append([]registry.Timer(nil), timers...)} |
| ++ next := registry.ColdRows{Route: *row, Timers: append([]registry.Timer(nil), timers...)} |
| ++ ownership := func(kind, name string) registry.ClaimOwnership { |
| ++ sum := sha256.Sum256([]byte(owner + "\x00" + kind + "\x00" + name)) |
| ++ return registry.ClaimOwnership{ClaimIncarnation: inv.Incarnation, ClaimStamp: inv.InstallationStamp, ClaimOwner: owner, ClaimRowID: hex.EncodeToString(sum[:])} |
| ++ } |
| ++ next.Route.ClaimOwnership = ownership("route", inv.Host) |
| ++ for i := range next.Timers { |
| ++ next.Timers[i].ClaimOwnership = ownership("timer", next.Timers[i].Name) |
| ++ } |
| ++ // Retained source inventory recovers the old row identities after a lost reply. |
| ++ routeFound := false |
| ++ for i := range inv.Entries { |
| ++ e := &inv.Entries[i] |
| ++ if e.Kind == "route" { |
| ++ if routeFound || e.RelativePath != inv.Host { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ routeFound = true |
| ++ old.Route.ClaimOwnership = registry.ClaimOwnership{ClaimIncarnation: inv.Incarnation, ClaimStamp: oldStamp, ClaimOwner: e.Ownership, ClaimRowID: e.RowIdentity} |
| ++ e.RowIdentity = next.Route.ClaimRowID |
| ++ } |
| ++ if e.Kind == "timer" { |
| ++ found := false |
| ++ for j := range old.Timers { |
| ++ if e.RelativePath == inv.Host+"/"+old.Timers[j].Name { |
| ++ old.Timers[j].ClaimOwnership = registry.ClaimOwnership{ClaimIncarnation: inv.Incarnation, ClaimStamp: oldStamp, ClaimOwner: e.Ownership, ClaimRowID: e.RowIdentity} |
| ++ e.RowIdentity = next.Timers[j].ClaimRowID |
| ++ found = true |
| ++ } |
| ++ } |
| ++ if !found { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ } |
| ++ } |
| ++ if !routeFound { |
| ++ return hostbox.ErrTypedInventory |
| ++ } |
| ++ source := *inv |
| ++ source.InstallationStamp = oldStamp |
| ++ source.Entries = append([]hostbox.StopInventoryEntry(nil), inv.Entries...) |
| ++ for i := range source.Entries { |
| ++ e := &source.Entries[i] |
| ++ if e.Kind == "route" { |
| ++ e.RowIdentity = old.Route.ClaimRowID |
| ++ } |
| ++ if e.Kind == "timer" { |
| ++ for _, t := range old.Timers { |
| ++ if e.RelativePath == inv.Host+"/"+t.Name { |
| ++ e.RowIdentity = t.ClaimRowID |
| ++ } |
| ++ } |
| ++ } |
| ++ } |
| ++ if err := hostbox.ValidateAgentInventory(source, &old.Route, h.Guard.DataDir); err != nil { |
| ++ return err |
| ++ } |
| ++ return h.Registry.RebindClaimRows(ctx, old, next) |
| +} |
| --- /dev/null |
| +++ b/internal/host/claimrecoveryreceipt.go |
| @@ -2373,40 +3469,90 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| + } |
| + return nil |
| +} |
| ---- a/internal/hostguard/writerstop.go |
| -+++ b/internal/hostguard/writerstop.go |
| -@@ -110,3 +110,16 @@ |
| +--- a/internal/host/claimroleassembly.go |
| ++++ b/internal/host/claimroleassembly.go |
| +@@ -21,6 +21,7 @@ |
| + type ClaimRoleAssembly struct { |
| + // Root is the reservation-bearing isolated root launch protocol (claim |
| + // Plan 11, R10). Nil keeps every claimed launch refused at Preflight. |
| ++ prepareGroup func(string, cgroup.Limit) (*cgroup.Prepared, hostbox.LaunchGroup, error) |
| + Root activator.RootRoleLauncher |
| + Manager *cgroup.Manager |
| + Journal *hostbox.LaunchJournal |
| +--- a/internal/host/claimroleassembly_linux.go |
| ++++ b/internal/host/claimroleassembly_linux.go |
| +@@ -51,21 +51,19 @@ |
| + return nil, err |
| + } |
| + parentName := claimContainmentName(target) |
| +- parentPath, err := filepath.Rel("/sys/fs/cgroup", filepath.Join(a.Manager.Root(), cgroup.ExecName(parentName))) |
| +- if err != nil || !filepath.IsLocal(parentPath) { |
| +- return nil, hostwrite.ErrUnavailable |
| +- } |
| + var parent *cgroup.Prepared |
| ++ var group hostbox.LaunchGroup |
| + err = work.With(ctx, in.Identity, func() error { |
| + var e error |
| +- if parent, e = a.Manager.PrepareStrict(parentName, in.Limit); e != nil { |
| +- return e |
| +- } |
| +- id, e := parent.Identity() |
| ++ prepare := a.prepareGroup |
| ++ if prepare == nil { |
| ++ prepare = a.preparePrelaunchGroup |
| ++ } |
| ++ parent, group, e = prepare(parentName, in.Limit) |
| + if e != nil { |
| + return e |
| + } |
| +- return a.Journal.Prelaunch(ctx, hostbox.LaunchInventory{ControllerPID: os.Getpid(), ProcessStart: start, Target: target, Groups: []hostbox.LaunchGroup{{RelativePath: parentPath, Device: id.Device, Inode: id.Inode}}, NoState: true}) |
| ++ return a.Journal.Prelaunch(ctx, hostbox.LaunchInventory{ControllerPID: os.Getpid(), ProcessStart: start, Target: target, Groups: []hostbox.LaunchGroup{group}, NoState: true}) |
| + }) |
| + if err != nil { |
| + if parent != nil { |
| +@@ -73,7 +71,7 @@ |
| + } |
| + return nil, err |
| + } |
| +- return &activator.RoleLaunch{Parent: parent, ParentPath: parentPath, CgroupRoot: "/sys/fs/cgroup", Target: target, Work: work, Identity: in.Identity, Journal: a.Journal}, nil |
| ++ return &activator.RoleLaunch{Parent: parent, ParentPath: group.RelativePath, CgroupRoot: "/sys/fs/cgroup", Target: target, Work: work, Identity: in.Identity, Journal: a.Journal}, nil |
| + } |
| + |
| + // Launch completes the pre-launch record after restore and materialization. |
| +@@ -377,3 +375,20 @@ |
| + return a.Inventory.WriteStopInventory(ctx, inventory) |
| }) |
| - return out, err |
| } |
| + |
| -+func RecoveryWriter(c *hostclaim.Claim, box string) (hostclaim.AdmissionRecord, bool) { |
| -+ if c == nil || (c.Op.Kind != hostclaim.KindRestoring && c.Op.Kind != hostclaim.KindReverting) || c.Op.Intent["source"] != box { |
| -+ return hostclaim.AdmissionRecord{}, false |
| ++func (a *ClaimRoleAssembly) preparePrelaunchGroup(name string, limit cgroup.Limit) (*cgroup.Prepared, hostbox.LaunchGroup, error) { |
| ++ path, err := filepath.Rel("/sys/fs/cgroup", filepath.Join(a.Manager.Root(), cgroup.ExecName(name))) |
| ++ if err != nil || !filepath.IsLocal(path) { |
| ++ return nil, hostbox.LaunchGroup{}, hostwrite.ErrUnavailable |
| + } |
| -+ return operationWriter(c, box, c.Op.Kind) |
| -+} |
| -+func RecoveryWriterStop(c *hostclaim.Claim, box, region string) (string, bool) { |
| -+ if _, ok := RecoveryWriter(c, box); !ok { |
| -+ return "", false |
| ++ parent, err := a.Manager.PrepareStrict(name, limit) |
| ++ if err != nil { |
| ++ return nil, hostbox.LaunchGroup{}, err |
| + } |
| -+ return operationWriterStop(c, box, region, c.Op.Kind) |
| ++ id, err := parent.Identity() |
| ++ if err != nil { |
| ++ parent.Close() |
| ++ return nil, hostbox.LaunchGroup{}, err |
| ++ } |
| ++ return parent, hostbox.LaunchGroup{RelativePath: path, Device: id.Device, Inode: id.Inode}, nil |
| +} |
| ---- a/internal/hostguard/fence.go |
| -+++ b/internal/hostguard/fence.go |
| -@@ -268,6 +268,9 @@ |
| - // landing's own current writer on this box (claim Plan 9 D4). |
| - if writerStop(t.ObligationID) { |
| - id, ok := LandingWriterStop(c, t.Box, t.Region) |
| -+ if c.Op.Kind == hostclaim.KindRestoring || c.Op.Kind == hostclaim.KindReverting { |
| -+ id, ok = RecoveryWriterStop(c, t.Box, t.Region) |
| -+ } |
| - if c.Op.Kind == hostclaim.KindRelocating { |
| - id, ok = RelocationWriterStop(c, t.Box, t.Region) |
| - } |
| --- a/internal/host/claimwakestop.go |
| +++ b/internal/host/claimwakestop.go |
| -@@ -365,5 +365,8 @@ |
| +@@ -174,6 +174,9 @@ |
| + local, err := a.localStopIdentity(ctx) |
| + if err != nil { |
| + return err |
| ++ } |
| ++ if a.stopReaches() != nil && claimInitialSourceRecovery(captured, local.Actor.Box) { |
| ++ return (claimCrossBoxHandover{Guard: a.Guard, Evidence: a.Claims, Box: local.Actor.Box}).beforeRecovery(ctx, captured) |
| + } |
| + if h.Session == 0 { |
| + // No writer was ever admitted, so nothing was launched or journaled. |
| +@@ -365,5 +368,8 @@ |
| if c != nil && c.Op.Kind == hostclaim.KindRelocating { |
| return hostguard.RelocationWriterStop(c, actor.Box, actor.Region) |
| } |
| @@ -2450,6 +3596,146 @@ Expected first failure: the target journal constants and recovery APIs are absen |
| ExecToNone(opID, targetBox, evidenceKey, JournalEvidenceTarget), |
| func(c *Claim) error { c.Op.Phase = PhaseReleased; return nil }, |
| QueueLocationPark(opID, targetRegion, parkBox, scope, newID), |
| +--- a/internal/hostguard/fence.go |
| ++++ b/internal/hostguard/fence.go |
| +@@ -268,6 +268,9 @@ |
| + // landing's own current writer on this box (claim Plan 9 D4). |
| + if writerStop(t.ObligationID) { |
| + id, ok := LandingWriterStop(c, t.Box, t.Region) |
| ++ if c.Op.Kind == hostclaim.KindRestoring || c.Op.Kind == hostclaim.KindReverting { |
| ++ id, ok = RecoveryWriterStop(c, t.Box, t.Region) |
| ++ } |
| + if c.Op.Kind == hostclaim.KindRelocating { |
| + id, ok = RelocationWriterStop(c, t.Box, t.Region) |
| + } |
| +--- a/internal/hostguard/writerstop.go |
| ++++ b/internal/hostguard/writerstop.go |
| +@@ -110,3 +110,16 @@ |
| + }) |
| + return out, err |
| + } |
| ++ |
| ++func RecoveryWriter(c *hostclaim.Claim, box string) (hostclaim.AdmissionRecord, bool) { |
| ++ if c == nil || (c.Op.Kind != hostclaim.KindRestoring && c.Op.Kind != hostclaim.KindReverting) || c.Op.Intent["source"] != box { |
| ++ return hostclaim.AdmissionRecord{}, false |
| ++ } |
| ++ return operationWriter(c, box, c.Op.Kind) |
| ++} |
| ++func RecoveryWriterStop(c *hostclaim.Claim, box, region string) (string, bool) { |
| ++ if _, ok := RecoveryWriter(c, box); !ok { |
| ++ return "", false |
| ++ } |
| ++ return operationWriterStop(c, box, region, c.Op.Kind) |
| ++} |
| +--- /dev/null |
| ++++ b/internal/registry/recoveryhandover.go |
| +@@ -0,0 +1,95 @@ |
| ++package registry |
| ++ |
| ++import ( |
| ++ "context" |
| ++ "reflect" |
| ++) |
| ++ |
| ++// RebindClaimRows transfers an exact stopped installation's registry rows. |
| ++// The caller holds hostguard and proves the source cut and successor admission. |
| ++// A transaction preserves row contents and refuses any concurrent replacement. |
| ++func (r *Registry) RebindClaimRows(ctx context.Context, old, next ColdRows) error { |
| ++ if r.remote != nil { |
| ++ _, err := r.remote.call(ctx, remoteRequest{Method: "RebindClaimRows", Old: old, NextValue: next}, nil) |
| ++ return err |
| ++ } |
| ++ if !old.Route.ClaimOwnership.Valid() || !next.Route.ClaimOwnership.Valid() || old.Route.Host != next.Route.Host || old.Route.ClaimIncarnation != next.Route.ClaimIncarnation || next.Route.ClaimStamp <= old.Route.ClaimStamp || len(old.Timers) != len(next.Timers) { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ a, b := old.Route, next.Route |
| ++ a.ClaimOwnership, b.ClaimOwnership = ClaimOwnership{}, ClaimOwnership{} |
| ++ if a != b { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ for i, t := range next.Timers { |
| ++ a, b := old.Timers[i], t |
| ++ a.ClaimOwnership, b.ClaimOwnership = ClaimOwnership{}, ClaimOwnership{} |
| ++ if a != b || !t.ClaimOwnership.Valid() || t.Host != next.Route.Host || t.ClaimIncarnation != next.Route.ClaimIncarnation || t.ClaimStamp != next.Route.ClaimStamp || t.ClaimOwner != next.Route.ClaimOwner { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ } |
| ++ tx, err := r.beginClaimTransaction(ctx, true) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ defer tx.Rollback() |
| ++ row, err := scanRoute(tx.QueryRowContext(ctx, `SELECT `+routeCols+` FROM routes WHERE host=?`, old.Route.Host).Scan) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ replay := row == next.Route |
| ++ if !replay && row != old.Route { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ rows, err := tx.conn.QueryContext(ctx, `SELECT `+timerCols+` FROM timers WHERE host=? ORDER BY name`, old.Route.Host) |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ var timers []Timer |
| ++ for rows.Next() { |
| ++ t, e := scanTimer(rows) |
| ++ if e != nil { |
| ++ rows.Close() |
| ++ return e |
| ++ } |
| ++ timers = append(timers, t) |
| ++ } |
| ++ err = rows.Err() |
| ++ rows.Close() |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ want := old.Timers |
| ++ if replay { |
| ++ want = next.Timers |
| ++ } |
| ++ if len(timers) != len(want) { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ for _, t := range timers { |
| ++ found := false |
| ++ for _, w := range want { |
| ++ if reflect.DeepEqual(t, w) { |
| ++ found = true |
| ++ break |
| ++ } |
| ++ } |
| ++ if !found { |
| ++ return ErrClaimOwnership |
| ++ } |
| ++ } |
| ++ if replay { |
| ++ return nil |
| ++ } |
| ++ o := next.Route.ClaimOwnership |
| ++ if _, err = tx.ExecContext(ctx, `UPDATE routes SET claim_incarnation=?,claim_stamp=?,claim_owner=?,claim_row_id=? WHERE host=?`, o.ClaimIncarnation, o.ClaimStamp, o.ClaimOwner, o.ClaimRowID, next.Route.Host); err != nil { |
| ++ return err |
| ++ } |
| ++ for _, t := range next.Timers { |
| ++ o := t.ClaimOwnership |
| ++ if _, err = tx.ExecContext(ctx, `UPDATE timers SET claim_incarnation=?,claim_stamp=?,claim_owner=?,claim_row_id=? WHERE host=? AND name=?`, o.ClaimIncarnation, o.ClaimStamp, o.ClaimOwner, o.ClaimRowID, t.Host, t.Name); err != nil { |
| ++ return err |
| ++ } |
| ++ } |
| ++ return tx.Commit() |
| ++} |
| +--- a/internal/registry/remote_dispatch.go |
| ++++ b/internal/registry/remote_dispatch.go |
| +@@ -133,6 +133,8 @@ |
| + err = r.SetTimerStatusRegistration(q.Host, q.Name, q.Registration, q.Status) |
| + case "FireTimerNowRegistration": |
| + err = r.FireTimerNowRegistration(q.Host, q.Name, q.Registration, q.Now) |
| ++ case "RebindClaimRows": |
| ++ err = r.RebindClaimRows(ctx, q.Old, q.NextValue) |
| + case "RestoreClaimRows": |
| + err = r.RestoreClaimRows(ctx, q.Id, q.Retirement, q.Old, q.NextValue) |
| + case "UpsertFleet": |
| ``` |
| ## Task D: Run parks under the caller’s location lease (M5, M6) |
| @@ -2593,92 +3879,6 @@ Expected first failure: `RunHeld` and `ReplicaParkLeaseSet` are undefined. The t |
| <!-- apply: D implementation --> |
| ```diff |
| ---- a/internal/park/remote.go |
| -+++ b/internal/park/remote.go |
| -@@ -385,7 +385,18 @@ |
| - |
| - // Run performs one resumable event. ErrPending is a scheduler retry, never an |
| - // acknowledgement. A newly rotated event is persisted before returning it. |
| --func (e Engine) Run(ctx context.Context, w Work) error { |
| -+func (e Engine) Run(ctx context.Context, w Work) error { return e.run(ctx, w, nil) } |
| -+ |
| -+// RunHeld borrows an exact location lease. The caller retains its renewal and |
| -+// release duties through publication. Only the configured lease resolver may |
| -+// borrow it; the same authorizer, census and engine effects still run. |
| -+func (e Engine) RunHeld(ctx context.Context, w Work, held HeldLease) error { |
| -+ if held == nil { |
| -+ return ErrUnsafe |
| -+ } |
| -+ return e.run(ctx, w, held) |
| -+} |
| -+func (e Engine) run(ctx context.Context, w Work, own HeldLease) error { |
| - if e.Store == nil || e.Claims == nil || e.Leases == nil || e.Now == nil || e.NewID == nil || e.View == nil || e.Authorize == nil || e.Bucket == "" || e.Region == "" { |
| - return ErrUnsafe |
| - } |
| -@@ -406,9 +417,23 @@ |
| - if o.Region != e.Region { |
| - return ErrUnsafe |
| - } |
| -- held, err := e.Leases.Acquire(ctx, e.Bucket, o.Park.Location) |
| -- if err != nil { |
| -- return err |
| -+ var held HeldLease |
| -+ if own == nil { |
| -+ held, err = e.Leases.Acquire(ctx, e.Bucket, o.Park.Location) |
| -+ } else { |
| -+ resolver, ok := e.Leases.(interface { |
| -+ AcquireHeld(context.Context, string, string, HeldLease) (HeldLease, error) |
| -+ }) |
| -+ if !ok { |
| -+ return ErrUnsafe |
| -+ } |
| -+ held, err = resolver.AcquireHeld(ctx, e.Bucket, o.Park.Location, own) |
| -+ } |
| -+ if err != nil { |
| -+ return err |
| -+ } |
| -+ if held == nil { |
| -+ return ErrUnsafe |
| - } |
| - defer held.Close() |
| - if bound, ok := held.(interface { |
| ---- a/internal/hostbox/renewing.go |
| -+++ b/internal/hostbox/renewing.go |
| -@@ -37,7 +37,7 @@ |
| - return nil, ErrUnsupported |
| - } |
| - cctx, stop := context.WithCancel(ctx) |
| -- held := &renewingHeld{h: h, ctx: cctx, cancel: stop, done: make(chan struct{})} |
| -+ held := &renewingHeld{bucket: bucket, scope: scope, h: h, ctx: cctx, cancel: stop, done: make(chan struct{})} |
| - go func() { |
| - defer close(held.done) |
| - select { |
| -@@ -54,12 +54,13 @@ |
| - } |
| - |
| - type renewingHeld struct { |
| -- h Renewable |
| -- ctx context.Context |
| -- cancel context.CancelFunc |
| -- done chan struct{} |
| -- once sync.Once |
| -- releaseErr error |
| -+ bucket, scope string |
| -+ h Renewable |
| -+ ctx context.Context |
| -+ cancel context.CancelFunc |
| -+ done chan struct{} |
| -+ once sync.Once |
| -+ releaseErr error |
| - } |
| - |
| - func (h *renewingHeld) Check(ctx context.Context) error { |
| -@@ -108,3 +109,5 @@ |
| - } |
| - return errors.Join(ctx.Err(), health(), authority(ctx)) |
| - } |
| -+ |
| -+func (h *renewingHeld) LeaseScope() (string, string) { return h.bucket, h.scope } |
| --- a/internal/hostbox/aggregateleases.go |
| +++ b/internal/hostbox/aggregateleases.go |
| @@ -8,9 +8,11 @@ |
| @@ -2803,25 +4003,111 @@ Expected first failure: `RunHeld` and `ReplicaParkLeaseSet` are undefined. The t |
| + if len(p) < 2 || len(p) > 3 || !controlname.ValidHostShape(p[1]) { |
| + return nil, ErrUnsupported |
| + } |
| -+ switch p[0] { |
| -+ case "instances": |
| -+ if len(p) == 3 { |
| -+ if len(p[2]) != 64 || strings.ToLower(p[2]) != p[2] { |
| -+ return nil, ErrUnsupported |
| -+ } |
| -+ if _, err := hex.DecodeString(p[2]); err != nil { |
| -+ return nil, ErrUnsupported |
| -+ } |
| -+ } |
| -+ case "moves": |
| -+ if len(p) == 3 && (p[2] == "" || p[2] == "." || p[2] == ".." || strings.ContainsAny(p[2], "\\\x00:")) { |
| -+ return nil, ErrUnsupported |
| ++ switch p[0] { |
| ++ case "instances": |
| ++ if len(p) == 3 { |
| ++ if len(p[2]) != 64 || strings.ToLower(p[2]) != p[2] { |
| ++ return nil, ErrUnsupported |
| ++ } |
| ++ if _, err := hex.DecodeString(p[2]); err != nil { |
| ++ return nil, ErrUnsupported |
| ++ } |
| ++ } |
| ++ case "moves": |
| ++ if len(p) == 3 && (p[2] == "" || p[2] == "." || p[2] == ".." || strings.ContainsAny(p[2], "\\\x00:")) { |
| ++ return nil, ErrUnsupported |
| ++ } |
| ++ default: |
| ++ return nil, ErrUnsupported |
| ++ } |
| ++ return []string{scope}, nil |
| ++} |
| +--- a/internal/hostbox/renewing.go |
| ++++ b/internal/hostbox/renewing.go |
| +@@ -37,7 +37,7 @@ |
| + return nil, ErrUnsupported |
| + } |
| + cctx, stop := context.WithCancel(ctx) |
| +- held := &renewingHeld{h: h, ctx: cctx, cancel: stop, done: make(chan struct{})} |
| ++ held := &renewingHeld{bucket: bucket, scope: scope, h: h, ctx: cctx, cancel: stop, done: make(chan struct{})} |
| + go func() { |
| + defer close(held.done) |
| + select { |
| +@@ -54,12 +54,13 @@ |
| + } |
| + |
| + type renewingHeld struct { |
| +- h Renewable |
| +- ctx context.Context |
| +- cancel context.CancelFunc |
| +- done chan struct{} |
| +- once sync.Once |
| +- releaseErr error |
| ++ bucket, scope string |
| ++ h Renewable |
| ++ ctx context.Context |
| ++ cancel context.CancelFunc |
| ++ done chan struct{} |
| ++ once sync.Once |
| ++ releaseErr error |
| + } |
| + |
| + func (h *renewingHeld) Check(ctx context.Context) error { |
| +@@ -108,3 +109,5 @@ |
| + } |
| + return errors.Join(ctx.Err(), health(), authority(ctx)) |
| + } |
| ++ |
| ++func (h *renewingHeld) LeaseScope() (string, string) { return h.bucket, h.scope } |
| +--- a/internal/park/remote.go |
| ++++ b/internal/park/remote.go |
| +@@ -385,7 +385,18 @@ |
| + |
| + // Run performs one resumable event. ErrPending is a scheduler retry, never an |
| + // acknowledgement. A newly rotated event is persisted before returning it. |
| +-func (e Engine) Run(ctx context.Context, w Work) error { |
| ++func (e Engine) Run(ctx context.Context, w Work) error { return e.run(ctx, w, nil) } |
| ++ |
| ++// RunHeld borrows an exact location lease. The caller retains its renewal and |
| ++// release duties through publication. Only the configured lease resolver may |
| ++// borrow it; the same authorizer, census and engine effects still run. |
| ++func (e Engine) RunHeld(ctx context.Context, w Work, held HeldLease) error { |
| ++ if held == nil { |
| ++ return ErrUnsafe |
| ++ } |
| ++ return e.run(ctx, w, held) |
| ++} |
| ++func (e Engine) run(ctx context.Context, w Work, own HeldLease) error { |
| + if e.Store == nil || e.Claims == nil || e.Leases == nil || e.Now == nil || e.NewID == nil || e.View == nil || e.Authorize == nil || e.Bucket == "" || e.Region == "" { |
| + return ErrUnsafe |
| + } |
| +@@ -406,9 +417,23 @@ |
| + if o.Region != e.Region { |
| + return ErrUnsafe |
| + } |
| +- held, err := e.Leases.Acquire(ctx, e.Bucket, o.Park.Location) |
| +- if err != nil { |
| +- return err |
| ++ var held HeldLease |
| ++ if own == nil { |
| ++ held, err = e.Leases.Acquire(ctx, e.Bucket, o.Park.Location) |
| ++ } else { |
| ++ resolver, ok := e.Leases.(interface { |
| ++ AcquireHeld(context.Context, string, string, HeldLease) (HeldLease, error) |
| ++ }) |
| ++ if !ok { |
| ++ return ErrUnsafe |
| + } |
| -+ default: |
| -+ return nil, ErrUnsupported |
| ++ held, err = resolver.AcquireHeld(ctx, e.Bucket, o.Park.Location, own) |
| + } |
| -+ return []string{scope}, nil |
| -+} |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ if held == nil { |
| ++ return ErrUnsafe |
| + } |
| + defer held.Close() |
| + if bound, ok := held.(interface { |
| ``` |
| ## Task E: Label residue independently and prove free-host lanes (M7) |
| @@ -3034,115 +4320,78 @@ Expected first failure: the census, view labels and write-gated field are absent |
| <!-- apply: E implementation --> |
| ```diff |
| ---- a/internal/hostclaim/claim.go |
| -+++ b/internal/hostclaim/claim.go |
| -@@ -155,6 +155,7 @@ |
| +--- a/internal/host/claimproduction.go |
| ++++ b/internal/host/claimproduction.go |
| +@@ -36,7 +36,6 @@ |
| + Open(context.Context, release.Store, lease.VersionedStore, hostbox.Actor, ClaimMoveReadAccess) (ClaimMovesOwners, ClaimFilesManifest, error) |
| + View(context.Context, hostclaim.Observed) (park.View, error) |
| + VerifyRetirement(context.Context, hostclaim.Observed, []lease.ObjectVersion) (string, time.Time, error) |
| +- LeaseSet(context.Context, string, string) ([]string, error) |
| + Close(context.Context) error |
| } |
| - type Claim struct { |
| -+ RetiredLocations map[string]string `json:"retired_locations,omitempty"` |
| - EvidenceRefs []AutomaticLoss `json:"evidence_refs,omitempty"` |
| - V int `json:"v"` |
| - Host string `json:"host"` |
| -@@ -297,6 +298,11 @@ |
| - if err := validateAutomaticLosses(c, seenLossRefs); err != nil { |
| - return err |
| +@@ -53,7 +52,7 @@ |
| + return hostwrite.ErrUnavailable |
| } |
| -+ for inc, location := range c.RetiredLocations { |
| -+ if !isHex64(inc) || !stringPresent(c.RetiredIncarnations, inc) || (location != "instances/"+c.Host && location != "instances/"+c.Host+"/"+inc) { |
| -+ return ErrMalformed |
| -+ } |
| -+ } |
| - if len(c.RetiredIncarnations) > retiredCap { |
| - return fmt.Errorf("%w: retired_incarnations over cap", ErrMalformed) |
| + if claimMoveSources.Parks == nil { |
| +- claimMoveSources.Parks = &claimMoveParkSources{View: s.View, Evidence: s, Leases: s.LeaseSet} |
| ++ claimMoveSources.Parks = &claimMoveParkSources{View: s.View, Evidence: s} |
| } |
| ---- a/internal/hostclaim/create.go |
| -+++ b/internal/hostclaim/create.go |
| -@@ -4,6 +4,7 @@ |
| - "encoding/json" |
| - "errors" |
| - "fmt" |
| -+ "os" |
| - "time" |
| - ) |
| - |
| -@@ -42,9 +43,20 @@ |
| + claimMoveSources.Source = s |
| + return nil |
| +@@ -297,6 +296,7 @@ |
| } |
| - func retire(c *Claim, inc string) { |
| -+ if os.Getenv("CARLOS_HOST_CLAIM_RETIRED_LOCATIONS") == "1" && c.Holder != nil && c.Holder.Incarnation == inc && isHex64(inc) { |
| -+ if c.RetiredLocations == nil { |
| -+ c.RetiredLocations = map[string]string{} |
| -+ } |
| -+ c.RetiredLocations[inc] = c.Holder.Location |
| + type productionClaimRegional struct { |
| ++ Now func() time.Time |
| + Identity hostbox.IdentityProvider |
| + Parks *claimMoveParkSources |
| + Local *productionClaimLocal |
| +@@ -309,12 +309,21 @@ |
| + if _, err := p.Identity.Verify(ctx); err != nil { |
| + return park.View{}, err |
| + } |
| +- return p.Parks.View(ctx, o) |
| ++ v, err := p.Parks.View(ctx, o) |
| ++ if err != nil { |
| ++ return park.View{}, err |
| + } |
| - c.RetiredIncarnations = append([]string{inc}, c.RetiredIncarnations...) |
| - if len(c.RetiredIncarnations) > retiredCap { |
| - c.RetiredIncarnations = c.RetiredIncarnations[:retiredCap] |
| -+ for retired := range c.RetiredLocations { |
| -+ if !stringPresent(c.RetiredIncarnations, retired) { |
| -+ delete(c.RetiredLocations, retired) |
| -+ } |
| -+ } |
| ++ v.ArchivedIncarnations = claimArchivedLocations(o.Claim) |
| ++ v.RefusedLocations = map[string]error{} |
| ++ if _, err := claimLegacyIncarnation(o.Claim); err != nil { |
| ++ v.RefusedLocations["instances/"+o.Host] = err |
| ++ } |
| ++ if o.Claim.Holder == nil && len(o.Claim.RetiredIncarnations) == 0 { |
| ++ v.RefusedLocations["moves/"+o.Host] = claimRetirementDependency() |
| ++ } |
| ++ return v, nil |
| + } |
| + func (p productionClaimRegional) LeaseSet(ctx context.Context, bucket, scope string) ([]string, error) { |
| +- if p.Parks == nil || p.Parks.Leases == nil { |
| +- return nil, &ClaimDependencyError{Contracts: []string{"moves Plan 5 complete replica and staging lease inventory"}} |
| +- } |
| + a, err := p.Identity.Verify(ctx) |
| + if err != nil { |
| + return nil, err |
| +@@ -322,16 +331,13 @@ |
| + if a.Bucket != bucket { |
| + return nil, hostbox.ErrChanged |
| } |
| +- return p.Parks.Leases(ctx, bucket, scope) |
| ++ return hostbox.ReplicaParkLeaseSet(ctx, bucket, scope) |
| } |
| - |
| ---- a/internal/park/sweep.go |
| -+++ b/internal/park/sweep.go |
| -@@ -18,6 +18,10 @@ |
| - // Pinned includes the exact noncurrent versions needed by the installed root. |
| - // MarkerKnown means its read succeeded; an absent marker may denote @root. |
| - type View struct { |
| -+ // ArchivedIncarnations labels proven retired locations independently of |
| -+ // the moves-sweep proof's incarnation. Missing entries never invent labels. |
| -+ ArchivedIncarnations map[string]string |
| -+ RefusedLocations map[string]error |
| - ClaimETag, Region, AuthoritativeRegion string |
| - MarkerKnown bool |
| - MarkerGeneration string |
| -@@ -142,11 +146,18 @@ |
| - if isLock(c.Host, loc, x.Key) { |
| - continue |
| - } |
| -+ if refusal := v.RefusedLocations[loc]; refusal != nil && !(c.Holder != nil && loc == c.Holder.Location && v.Region == v.AuthoritativeRegion) { |
| -+ d.Alerts = append(d.Alerts, Alert{loc, refusal.Error()}) |
| -+ continue |
| -+ } |
| - candidate := Candidate{Location: loc} |
| - if tail := strings.TrimPrefix(loc, "instances/"+c.Host+"/"); tail != loc { |
| - candidate.Incarnation = tail |
| - } else { |
| - candidate.Incarnation = legacyIncarnation |
| -+ if loc == "instances/"+c.Host && v.ArchivedIncarnations[loc] != "" { |
| -+ candidate.Incarnation = v.ArchivedIncarnations[loc] |
| -+ } |
| - } |
| - if c.Holder != nil && loc == c.Holder.Location && v.Region == v.AuthoritativeRegion { |
| - if liveErr != nil { |
| -@@ -403,3 +414,23 @@ |
| + func (p productionClaimRegional) VerifyRetirement(ctx context.Context, o hostclaim.Observed, v []lease.ObjectVersion) (string, time.Time, error) { |
| +- if p.Parks == nil || p.Parks.Evidence == nil { |
| +- return "", time.Time{}, &ClaimDependencyError{Contracts: []string{"moves Plan 5 archived-incarnation regional evidence"}} |
| +- } |
| + if _, err := p.Identity.Verify(ctx); err != nil { |
| + return "", time.Time{}, err |
| } |
| - return d, nil |
| +- return p.Parks.Evidence.VerifyRetirement(ctx, o, v) |
| ++ return p.retirement(ctx, o, v) |
| } |
| -+ |
| -+// ResidueLocations uses the same namespace and lock classifier as Decide. |
| -+func ResidueLocations(host string, versions []lease.ObjectVersion) ([]string, error) { |
| -+ seen := map[string]bool{} |
| -+ for _, v := range versions { |
| -+ loc, err := locationOf(host, v.Key) |
| -+ if err != nil || v.VersionID == "" { |
| -+ return nil, ErrUnsafe |
| -+ } |
| -+ if !isLock(host, loc, v.Key) { |
| -+ seen[loc] = true |
| -+ } |
| -+ } |
| -+ out := make([]string, 0, len(seen)) |
| -+ for loc := range seen { |
| -+ out = append(out, loc) |
| -+ } |
| -+ sort.Strings(out) |
| -+ return out, nil |
| -+} |
| + func (p productionClaimRegional) Authorize(ctx context.Context, o hostclaim.Observed, ob hostclaim.Obligation) error { |
| + a, err := p.Identity.Verify(ctx) |
| --- /dev/null |
| +++ b/internal/host/claimretirementcensus.go |
| @@ -0,0 +1,142 @@ |
| @@ -3288,78 +4537,115 @@ Expected first failure: the census, view labels and write-gated field are absent |
| + // A refused lane is dropped by VerifyMovesSweep; it cannot block proven instance residue. |
| + return label, now, nil |
| +} |
| ---- a/internal/host/claimproduction.go |
| -+++ b/internal/host/claimproduction.go |
| -@@ -36,7 +36,6 @@ |
| - Open(context.Context, release.Store, lease.VersionedStore, hostbox.Actor, ClaimMoveReadAccess) (ClaimMovesOwners, ClaimFilesManifest, error) |
| - View(context.Context, hostclaim.Observed) (park.View, error) |
| - VerifyRetirement(context.Context, hostclaim.Observed, []lease.ObjectVersion) (string, time.Time, error) |
| -- LeaseSet(context.Context, string, string) ([]string, error) |
| - Close(context.Context) error |
| +--- a/internal/hostclaim/claim.go |
| ++++ b/internal/hostclaim/claim.go |
| +@@ -155,6 +155,7 @@ |
| } |
| -@@ -53,7 +52,7 @@ |
| - return hostwrite.ErrUnavailable |
| + type Claim struct { |
| ++ RetiredLocations map[string]string `json:"retired_locations,omitempty"` |
| + EvidenceRefs []AutomaticLoss `json:"evidence_refs,omitempty"` |
| + V int `json:"v"` |
| + Host string `json:"host"` |
| +@@ -297,6 +298,11 @@ |
| + if err := validateAutomaticLosses(c, seenLossRefs); err != nil { |
| + return err |
| } |
| - if claimMoveSources.Parks == nil { |
| -- claimMoveSources.Parks = &claimMoveParkSources{View: s.View, Evidence: s, Leases: s.LeaseSet} |
| -+ claimMoveSources.Parks = &claimMoveParkSources{View: s.View, Evidence: s} |
| ++ for inc, location := range c.RetiredLocations { |
| ++ if !isHex64(inc) || !stringPresent(c.RetiredIncarnations, inc) || (location != "instances/"+c.Host && location != "instances/"+c.Host+"/"+inc) { |
| ++ return ErrMalformed |
| ++ } |
| ++ } |
| + if len(c.RetiredIncarnations) > retiredCap { |
| + return fmt.Errorf("%w: retired_incarnations over cap", ErrMalformed) |
| } |
| - claimMoveSources.Source = s |
| - return nil |
| -@@ -297,6 +296,7 @@ |
| +--- a/internal/hostclaim/create.go |
| ++++ b/internal/hostclaim/create.go |
| +@@ -4,6 +4,7 @@ |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| ++ "os" |
| + "time" |
| + ) |
| + |
| +@@ -42,9 +43,20 @@ |
| } |
| - type productionClaimRegional struct { |
| -+ Now func() time.Time |
| - Identity hostbox.IdentityProvider |
| - Parks *claimMoveParkSources |
| - Local *productionClaimLocal |
| -@@ -309,12 +309,21 @@ |
| - if _, err := p.Identity.Verify(ctx); err != nil { |
| - return park.View{}, err |
| - } |
| -- return p.Parks.View(ctx, o) |
| -+ v, err := p.Parks.View(ctx, o) |
| -+ if err != nil { |
| -+ return park.View{}, err |
| -+ } |
| -+ v.ArchivedIncarnations = claimArchivedLocations(o.Claim) |
| -+ v.RefusedLocations = map[string]error{} |
| -+ if _, err := claimLegacyIncarnation(o.Claim); err != nil { |
| -+ v.RefusedLocations["instances/"+o.Host] = err |
| -+ } |
| -+ if o.Claim.Holder == nil && len(o.Claim.RetiredIncarnations) == 0 { |
| -+ v.RefusedLocations["moves/"+o.Host] = claimRetirementDependency() |
| + func retire(c *Claim, inc string) { |
| ++ if os.Getenv("CARLOS_HOST_CLAIM_RETIRED_LOCATIONS") == "1" && c.Holder != nil && c.Holder.Incarnation == inc && isHex64(inc) { |
| ++ if c.RetiredLocations == nil { |
| ++ c.RetiredLocations = map[string]string{} |
| ++ } |
| ++ c.RetiredLocations[inc] = c.Holder.Location |
| + } |
| -+ return v, nil |
| - } |
| - func (p productionClaimRegional) LeaseSet(ctx context.Context, bucket, scope string) ([]string, error) { |
| -- if p.Parks == nil || p.Parks.Leases == nil { |
| -- return nil, &ClaimDependencyError{Contracts: []string{"moves Plan 5 complete replica and staging lease inventory"}} |
| -- } |
| - a, err := p.Identity.Verify(ctx) |
| - if err != nil { |
| - return nil, err |
| -@@ -322,16 +331,13 @@ |
| - if a.Bucket != bucket { |
| - return nil, hostbox.ErrChanged |
| + c.RetiredIncarnations = append([]string{inc}, c.RetiredIncarnations...) |
| + if len(c.RetiredIncarnations) > retiredCap { |
| + c.RetiredIncarnations = c.RetiredIncarnations[:retiredCap] |
| ++ for retired := range c.RetiredLocations { |
| ++ if !stringPresent(c.RetiredIncarnations, retired) { |
| ++ delete(c.RetiredLocations, retired) |
| ++ } |
| ++ } |
| } |
| -- return p.Parks.Leases(ctx, bucket, scope) |
| -+ return hostbox.ReplicaParkLeaseSet(ctx, bucket, scope) |
| } |
| - func (p productionClaimRegional) VerifyRetirement(ctx context.Context, o hostclaim.Observed, v []lease.ObjectVersion) (string, time.Time, error) { |
| -- if p.Parks == nil || p.Parks.Evidence == nil { |
| -- return "", time.Time{}, &ClaimDependencyError{Contracts: []string{"moves Plan 5 archived-incarnation regional evidence"}} |
| -- } |
| - if _, err := p.Identity.Verify(ctx); err != nil { |
| - return "", time.Time{}, err |
| + |
| +--- a/internal/park/sweep.go |
| ++++ b/internal/park/sweep.go |
| +@@ -18,6 +18,10 @@ |
| + // Pinned includes the exact noncurrent versions needed by the installed root. |
| + // MarkerKnown means its read succeeded; an absent marker may denote @root. |
| + type View struct { |
| ++ // ArchivedIncarnations labels proven retired locations independently of |
| ++ // the moves-sweep proof's incarnation. Missing entries never invent labels. |
| ++ ArchivedIncarnations map[string]string |
| ++ RefusedLocations map[string]error |
| + ClaimETag, Region, AuthoritativeRegion string |
| + MarkerKnown bool |
| + MarkerGeneration string |
| +@@ -142,11 +146,18 @@ |
| + if isLock(c.Host, loc, x.Key) { |
| + continue |
| + } |
| ++ if refusal := v.RefusedLocations[loc]; refusal != nil && !(c.Holder != nil && loc == c.Holder.Location && v.Region == v.AuthoritativeRegion) { |
| ++ d.Alerts = append(d.Alerts, Alert{loc, refusal.Error()}) |
| ++ continue |
| ++ } |
| + candidate := Candidate{Location: loc} |
| + if tail := strings.TrimPrefix(loc, "instances/"+c.Host+"/"); tail != loc { |
| + candidate.Incarnation = tail |
| + } else { |
| + candidate.Incarnation = legacyIncarnation |
| ++ if loc == "instances/"+c.Host && v.ArchivedIncarnations[loc] != "" { |
| ++ candidate.Incarnation = v.ArchivedIncarnations[loc] |
| ++ } |
| + } |
| + if c.Holder != nil && loc == c.Holder.Location && v.Region == v.AuthoritativeRegion { |
| + if liveErr != nil { |
| +@@ -403,3 +414,23 @@ |
| } |
| -- return p.Parks.Evidence.VerifyRetirement(ctx, o, v) |
| -+ return p.retirement(ctx, o, v) |
| + return d, nil |
| } |
| - func (p productionClaimRegional) Authorize(ctx context.Context, o hostclaim.Observed, ob hostclaim.Obligation) error { |
| - a, err := p.Identity.Verify(ctx) |
| ++ |
| ++// ResidueLocations uses the same namespace and lock classifier as Decide. |
| ++func ResidueLocations(host string, versions []lease.ObjectVersion) ([]string, error) { |
| ++ seen := map[string]bool{} |
| ++ for _, v := range versions { |
| ++ loc, err := locationOf(host, v.Key) |
| ++ if err != nil || v.VersionID == "" { |
| ++ return nil, ErrUnsafe |
| ++ } |
| ++ if !isLock(host, loc, v.Key) { |
| ++ seen[loc] = true |
| ++ } |
| ++ } |
| ++ out := make([]string, 0, len(seen)) |
| ++ for loc := range seen { |
| ++ out = append(out, loc) |
| ++ } |
| ++ sort.Strings(out) |
| ++ return out, nil |
| ++} |
| ``` |
| ## Task F: Expose the local park authorizer in MoveParks (M9) |
| @@ -3485,6 +4771,34 @@ Expected first failure: the new four-value accessor does not exist. Also verify |
| <!-- apply: F implementation --> |
| ```diff |
| +--- a/internal/host/claimassembly.go |
| ++++ b/internal/host/claimassembly.go |
| +@@ -251,7 +251,7 @@ |
| + if err := a.startWakeResetWorker(ctx); err != nil { |
| + return ClaimProviders{}, err |
| + } |
| +- return ClaimProviders{moveParks: &claimMoveParks{engine: regional.Regional.Engine, leases: leases, check: check}, Delivery: ClaimTimerDelivery(a.Producers, a.NewID), Selector: source, Wake: wake, Regional: regional, Stop: a.Stop, Provision: a.Provision, Close: a.Close}, nil |
| ++ return ClaimProviders{moveTargetFence: claimBoxFenceVerifier(check, a.Claims, a.RetirementAuthority), moveParks: &claimMoveParks{local: claimMoveLocalAuthorizer(check, a.Authorize), engine: regional.Regional.Engine, leases: leases, check: check}, Delivery: ClaimTimerDelivery(a.Producers, a.NewID), Selector: source, Wake: wake, Regional: regional, Stop: a.Stop, Provision: a.Provision, Close: a.Close}, nil |
| + } |
| + |
| + type claimRegionalEnrollment struct { |
| +@@ -348,3 +348,16 @@ |
| + reconcile.Box, reconcile.Enrollment = actor.Box, actor.Enrollment |
| + return ClaimRootHTTP{Journals: assembly.Journals, RequireNewWork: func(ctx context.Context) error { _, err := assembly.Identity.Verify(ctx); return err }, Launches: assembly.Launches, Stop: stop, Reconcile: reconcile, Roots: assembly.ManagedRoots}, nil |
| + } |
| ++ |
| ++func claimMoveLocalAuthorizer(check func(context.Context) error, owner func(context.Context, hostclaim.Observed, hostclaim.Obligation) error) ClaimLocalParkAuthorizer { |
| ++ return func(ctx context.Context, c *hostclaim.Claim, ob hostclaim.Obligation) error { |
| ++ if check == nil || owner == nil || c == nil || ob.Kind != "park-local" { |
| ++ return hostwrite.ErrUnavailable |
| ++ } |
| ++ if err := check(ctx); err != nil { |
| ++ return err |
| ++ } |
| ++ // Preserve the unwrapped owner used by the assembly's own park.Local. |
| ++ return owner(ctx, hostclaim.Observed{Host: c.Host, Claim: c}, ob) |
| ++ } |
| ++} |
| --- a/internal/host/claimmoveparks.go |
| +++ b/internal/host/claimmoveparks.go |
| @@ -16,21 +16,17 @@ |
| @@ -3564,39 +4878,11 @@ Expected first failure: the new four-value accessor does not exist. Also verify |
| if p.Parks == nil { |
| return park.MoveSweepProof{}, &ClaimDependencyError{Contracts: []string{"moves Plan 5 newest retired move fence or release evidence"}} |
| } |
| ---- a/internal/host/claimassembly.go |
| -+++ b/internal/host/claimassembly.go |
| -@@ -251,7 +251,7 @@ |
| - if err := a.startWakeResetWorker(ctx); err != nil { |
| - return ClaimProviders{}, err |
| - } |
| -- return ClaimProviders{moveParks: &claimMoveParks{engine: regional.Regional.Engine, leases: leases, check: check}, Delivery: ClaimTimerDelivery(a.Producers, a.NewID), Selector: source, Wake: wake, Regional: regional, Stop: a.Stop, Provision: a.Provision, Close: a.Close}, nil |
| -+ return ClaimProviders{moveTargetFence: claimBoxFenceVerifier(check, a.Claims, a.RetirementAuthority), moveParks: &claimMoveParks{local: claimMoveLocalAuthorizer(check, a.Authorize), engine: regional.Regional.Engine, leases: leases, check: check}, Delivery: ClaimTimerDelivery(a.Producers, a.NewID), Selector: source, Wake: wake, Regional: regional, Stop: a.Stop, Provision: a.Provision, Close: a.Close}, nil |
| - } |
| - |
| - type claimRegionalEnrollment struct { |
| -@@ -348,3 +348,16 @@ |
| - reconcile.Box, reconcile.Enrollment = actor.Box, actor.Enrollment |
| - return ClaimRootHTTP{Journals: assembly.Journals, RequireNewWork: func(ctx context.Context) error { _, err := assembly.Identity.Verify(ctx); return err }, Launches: assembly.Launches, Stop: stop, Reconcile: reconcile, Roots: assembly.ManagedRoots}, nil |
| - } |
| -+ |
| -+func claimMoveLocalAuthorizer(check func(context.Context) error, owner func(context.Context, hostclaim.Observed, hostclaim.Obligation) error) ClaimLocalParkAuthorizer { |
| -+ return func(ctx context.Context, c *hostclaim.Claim, ob hostclaim.Obligation) error { |
| -+ if check == nil || owner == nil || c == nil || ob.Kind != "park-local" { |
| -+ return hostwrite.ErrUnavailable |
| -+ } |
| -+ if err := check(ctx); err != nil { |
| -+ return err |
| -+ } |
| -+ // Preserve the unwrapped owner used by the assembly's own park.Local. |
| -+ return owner(ctx, hostclaim.Observed{Host: c.Host, Claim: c}, ob) |
| -+ } |
| -+} |
| ``` |
| ## Task G: Classify every changed census against the pinned base |
| -Run `go test ./internal/provision -run TestPlan9B8 -count=1` before and after this patch. The initial failure must identify the changed units below. Every hunk is generated against `44c1b6d7` plus Tasks A–F, including the then-current B8 inventory. Do not apply this census to `70f8119f` or a concurrent moves branch without reconciling its own changes. |
| +Run `go test ./internal/provision -run TestPlan9B8 -count=1` before and after this patch. The initial failure must identify the changed units below. Every hunk is generated against `e79d71b5` plus Tasks A–F, including the merged moves P5 census changes. |
| | Unit | Classification and reason | |
| |---|---| |
| @@ -3609,7 +4895,7 @@ Run `go test ./internal/provision -run TestPlan9B8 -count=1` before and after th |
| | `hostops.moveLossOwner.ValidateLossRecovery` | `key-site`, `key:RecordKey`; compares the signed intent's record identity. | |
| | `hostops.MoveRecoveryGrant.ReleaseFencedTarget` | `record-writer`, `key:WriteRecord,key:writeRecord`; one sequenced record effect or claim effect per expiring grant. | |
| -The console authority scanner also gains the names `VerifyMoveTargetFence` and `ReleaseFencedTarget`. Its fixture adds exactly the two hostclaim calls in `hostops/move_fence_release.go`; both are under the expiring, record-bound recovery grant. Preserve repeated fixture entries: they represent separate source calls. The route inventory is unchanged. The moves scan adds the new grant file and pending row MV-N319 for the controller's exact `target fenced` reason, plus its pinned copy ID. No sink exemption is added; serial integration must reconcile any competing use of that ID. |
| +The authority scanner includes VerifyMoveTargetFence, ReleaseFencedTarget and ReportAutomaticLosses. Its fixture records the recovery-scoped reporter and grant-bound stamp, and removes the obsolete raw closure sites. The B8 inventory includes the stamp's sequenced writer and the changed grant record action. Existing repeated fixture entries remain. The route inventory is unchanged. The moves scan uses pending MV-N335 for `target fenced`; MV-N319–324 are reserved and main already uses MV-N325–334. No sink exemption is added. |
| After applying, run `go test ./internal/provision -run TestPlan9B8 -count=1`, `go test ./internal/console -run 'TestClaimAuthoritySourceCensus|TestClaim.*Routes' -count=1`, and `go test . -run 'TestMoves(StringsAreRecorded|SinksAreSentences)' -count=1`. Socket-dependent route tests require an environment that permits their test listeners. |
| @@ -3617,13 +4903,103 @@ After applying, run `go test ./internal/provision -run TestPlan9B8 -count=1`, `g |
| <!-- apply: G implementation --> |
| ```diff |
| +--- a/approved_moves_copy_test.go |
| ++++ b/approved_moves_copy_test.go |
| +@@ -75,7 +75,7 @@ |
| + "MV-N307", "MV-N308", "MV-N309", "MV-N310", "MV-N311", "MV-N312", |
| + "MV-N313", "MV-N314", |
| + "MV-N315", "MV-N316", "MV-N317", "MV-N318", |
| +- "MV-N325", "MV-N326", "MV-N327", "MV-N328", "MV-N329", "MV-N330", "MV-N331", "MV-N332", "MV-N333", "MV-N334", |
| ++ "MV-N325", "MV-N326", "MV-N327", "MV-N328", "MV-N329", "MV-N330", "MV-N331", "MV-N332", "MV-N333", "MV-N334", "MV-N335", |
| + } |
| + |
| + // copyRow is one row of the record. The File cell is the file the sentence is |
| +@@ -544,6 +544,7 @@ |
| + // copy" task appends the files it touched; TestMovesScanFilesCoverTheDiff |
| + // fails when the list falls behind. |
| + var movesScanFiles = []string{ |
| ++ "internal/hostops/move_fence_release.go", |
| + "internal/console/movesrecovery.go", |
| + "internal/console/claim_runtime.go", |
| + "internal/alerting/alerting.go", |
| +--- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| ++++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| +@@ -323,3 +323,9 @@ |
| + - MV-N146: the row's text carries a note, `(MV-N68 with a new <which>)`. The code says `no fresh rate measurement for the disk's per-file sync` through MV-N68's `NoFreshRate`, which is correct; the note is not a part of any sentence the code could say. |
| + |
| + The 16 rows added after that review (MV-N184 to MV-N186, MV-N300 to MV-N312) were approved by Paul on 2026-10-09 with no edits. |
| ++ |
| ++## Claim Plan 12 rev 3 |
| ++ |
| ++| id | file | text | status | |
| ++|---|---|---|---| |
| ++| MV-N335 | internal/provision/provision.go | target fenced | pending | |
| +--- a/internal/console/claim_authority_census_test.go |
| ++++ b/internal/console/claim_authority_census_test.go |
| +@@ -25,10 +25,10 @@ |
| + "NewClamAVScanner": true, "NewRoute53Writer": true, "BeginExclusive": true, |
| + "ResumeExclusive": true, "RegisterStart": true, "FinishWakeRetryReset": true, "ReservePurge": true, |
| + "ConsoleMove": true, "InspectMoveLoss": true, "DeclareMoveLoss": true, |
| +- "RunRecovery": true, "runMoveRecovery": true, "captureMoveStartBase": true, "resolveUncertain": true, "ConvertMovingToRelocating": true, |
| ++ "RunRecovery": true, "runMoveRecovery": true, "captureMoveStartBase": true, "resolveUncertain": true, "ConvertMovingToRelocating": true, "ReleaseFencedTarget": true, "VerifyMoveTargetFence": true, "ReportAutomaticLosses": true, |
| + // The console's move pass (moves Plan 5 Task 7): every write of the |
| + // move-recovery grant, and the lost target's stamp. |
| +- "ResolveStart": true, "Convert": true, "Claim": true, "Record": true, "StampLostMoveTargetParked": true, "stampLostMoveTarget": true, |
| ++ "ResolveStart": true, "Convert": true, "Claim": true, "Record": true, "StampLostMoveTargetParked": true, |
| + "BeginPurge": true, "SavePurgeJournal": true, "RemovePurgeJournal": true, |
| + "AcquireMailGate": true, "ReleaseMailGate": true, "EnrollNewApp": true, |
| + "EnrollBackfillApp": true, "OpenGated": true, "WithAppMutation": true, |
| +--- a/internal/console/testdata/claim-authority-sites.txt |
| ++++ b/internal/console/testdata/claim-authority-sites.txt |
| +@@ -12,6 +12,8 @@ |
| + ../hostops/lifecycle_grant.go|RecoveryPass|w.RunRecovery(ctx, r.Pass) |
| + ../hostops/move_enrollment.go|RegisterMove|a.Control.RegisterStart(ctx, h.Account, h.App, appID, id, reg, a.Now()) |
| + ../hostops/move_enrollment.go|StartClaimMoveRecord|h.captureMoveStartBase(ctx, seq) |
| ++../hostops/move_fence_release.go|ReleaseFencedTarget|hostclaim.ReleaseFencedTarget(r) |
| ++../hostops/move_fence_release.go|ReleaseFencedTarget|hostclaim.VerifyMoveTargetFence(ctx, g.w.raw, o, ref, g.authority) |
| + ../hostops/move_loss_authority.go|DeclareMoveLoss|a.Control.DeclareMoveLoss(ctx, a.Signer, a.Directory.Bucket, actor, account, app, host, box, in) |
| + ../hostops/move_loss_authority.go|InspectMoveLoss|a.Control.InspectMoveLoss(ctx, a.Signer, a.Directory.Bucket, actor, account, app, host, box) |
| + ../hostops/move_recovery_bridge.go|RecoverTerminatedMovingHost|hostclaim.ConvertMovingToRelocating(mh.Lifecycle(ctx), s, fresh, c.Op.ID, ref, authority, guard) |
| +@@ -35,6 +37,7 @@ |
| + ../hostops/purge_journal.go|DeletePurgeSweep|w.SavePurgeJournal(ctx, fresh, tag) |
| + ../hostops/purge_journal.go|PreparePurgeSweep|w.SavePurgeJournal(ctx, j, tag) |
| + ../hostops/purge_journal.go|VerifyPurgeSweep|w.SavePurgeJournal(ctx, j, tag) |
| ++../hostops/recovery.go|Pass|r.ReportAutomaticLosses(ctx, host) |
| + ../hostops/recovery.go|Pass|runMoveRecovery(ctx, obs, r.Moves, r.MoveFenceAuthority) |
| + ../hostops/sidecarwake.go|TouchSidecarWake|a.Control.RegisterStart(ctx, account, app, appID, id, reg, a.Now()) |
| + ../mail/aws.go|NewAWSProvisioner|iam.NewFromConfig(cfg) |
| +@@ -86,13 +89,13 @@ |
| + claim_retire.go|retireClaimHostRecord|c.Control.BeginExclusive(ctx, ref.Account, ref.App, appID, begin, s.claimOwner) |
| + claim_retire.go|retireClaimHostRecord|c.Control.ResumeExclusive(ctx, ref.Account, ref.App, appID, op.ID, hostops.OpDeleting, s.claimOwner) |
| + claim_route_writes.go|appMutation|release.WithAppMutation(ctx, s.cfg.Releases, acct.Sqid, app, action, fn) |
| ++claim_runtime.go|ConfigureClaimRuntime|a.Control.ReportAutomaticLosses(ctx, a.Signer, host) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.BindMutationRunner(data, control, owner, s.mutationAppID) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.NewConditionalRecordStore(raw, s.cfg.ReleasePub) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.NewControlWriter(raw, s.cfg.ReleasePub, time.Now) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.NewObservedMutationBoundary(true, records, check, s.claimMutationObserver) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.NewProductionMutationCheck(raw, control) |
| + claim_runtime.go|ConfigureClaimRuntime|hostops.NewVersionedBoundary(true, s.cfg.DBVersions, raw, control, s.cfg.Bucket, dataBuckets, s.claimMutationObserver) |
| +-claim_runtime.go|ConfigureClaimRuntime|hostops.StampLostMoveTargetParked(ctx, records, host, now) |
| + claim_runtime.go|sweepOneClaimAdmission|c.Control.RecoverSQLTransaction(ctx, app.Account, app.Name, appID, id, s.store) |
| + claim_sql_auto.go|withAutoAppSQL|s.WithAppMutation(pctx, []sqlGrant{permitGrant(pctx, s.claimBucket, acct.Sqid, scope)}, call) |
| + claim_sql_grants.go|WithAppMutation|p.BindTransaction(database) |
| +@@ -112,7 +115,7 @@ |
| + movesrecovery.go|movesRecovery|g.Record(ctx, d.Record) |
| + movesrecovery.go|movesRecovery|g.Record(ctx, d.Record) |
| + movesrecovery.go|movesRecovery|g.ResolveStart(ctx) |
| +-movesrecovery.go|movesRecovery|s.stampLostMoveTarget(ctx, o.Host, now) |
| ++movesrecovery.go|movesRecovery|g.StampLostMoveTargetParked(ctx) |
| + scanpass.go|storeLister|objstore.S3ClientFor(ctx, p.Region, p.Endpoint) |
| + store.go|Open|sql.Open("sqlite", path) |
| + store_gate.go|Connect|k.g.d.Open(k.g.dsn) |
| --- a/internal/provision/plan9_b8_census_test.go |
| +++ b/internal/provision/plan9_b8_census_test.go |
| -@@ -472,6 +472,14 @@ |
| +@@ -472,8 +472,17 @@ |
| // BEGIN REVIEWED INVENTORY |
| var plan9B8CensusInventory = []censusEntry{ |
| -+ {"internal/hostops/move_fence_release.go", "MoveRecoveryGrant.ReleaseFencedTarget", "record-writer", "key:WriteRecord,key:writeRecord", "69e582fe41487e31ba16b2cb56bd7d8d197152209fb6ca3cb2fd9862afe4c12d"}, |
| ++ {"internal/hostops/move_fence_release.go", "MoveRecoveryGrant.StampLostMoveTargetParked", "record-writer", "key:WriteRecord", "b082db5634a2f4d9a5b45cc4a9cb725964e9bb24b7aeb53a6346167d7737227c"}, |
| ++ {"internal/hostops/move_fence_release.go", "MoveRecoveryGrant.ReleaseFencedTarget", "record-writer", "key:WriteRecord,key:writeRecord", "9d643789e5f2acba708ca9551fb9ce516b9b534c2108a95a39a0e9eb5d43a8ac"}, |
| + {"internal/host/claimretirementcensus.go", "claimLegacyIncarnation", "key-site", "lit:instances", "31a14ea105c67e00a24f06b9500a221ee5bfc47402b9c33cecfde807e51db530"}, |
| + {"internal/host/claimretirementcensus.go", "claimArchivedLocations", "key-site", "lit:instances", "9ba8ea4ecb5ad63621c1322467b9a80f8f5703896ba1fc3f7cb1addd56418950"}, |
| + {"internal/hostops/move_loss_admission.go", "moveLossOwner.ValidateLossRecovery", "key-site", "key:RecordKey", "ea2aa35a069ffdc846ecb208c33940245fc865795188818e86ea7a93135dd6a9"}, |
| @@ -3632,9 +5008,12 @@ After applying, run `go test ./internal/provision -run TestPlan9B8 -count=1`, `g |
| + {"internal/host/claimretirementcensus.go", "ClaimRetirementCensus", "key-site", "lit:instances", "abc0bd8c5f5d328abfac8821f42f8f23954723ed6506747f33b0197824e2611f"}, |
| + {"internal/hostbox/parkleases.go", "ReplicaParkLeaseSet", "key-site", "lit:instances", "0ca8d9b743402fda9d15cc215464423b35e21c1c2b47f91294d76188b7d43fb6"}, |
| {"internal/hostops/move_recovery_grant.go", "MoveRecoveryGrant.Convert", "record-writer", "key:WriteRecord", "794963a24a0a50d980505f83ac912973a10267b21f74bd41c56584ab9b0b25a1"}, |
| - {"internal/hostops/move_recovery_grant.go", "MoveRecoveryGrant.Record", "record-writer", "key:WriteRecord", "6b8bd4131842d8f22a6ab55aaae736329df40e34abb348d3d68cbd0a713503ad"}, |
| +- {"internal/hostops/move_recovery_grant.go", "MoveRecoveryGrant.Record", "record-writer", "key:WriteRecord", "6b8bd4131842d8f22a6ab55aaae736329df40e34abb348d3d68cbd0a713503ad"}, |
| ++ {"internal/hostops/move_recovery_grant.go", "MoveRecoveryGrant.Record", "record-writer", "key:WriteRecord", "69e37a694018cd5ab7d63efd660f5d85d157e7b7bc6e1dce85b9f1343eb5dd57"}, |
| // Plan 10: reviewed typed console move and loss writers. |
| -@@ -686,7 +694,7 @@ |
| + {"internal/hostops/console_move_grant.go", "ControlWriter.ConsoleMove", "record-writer", "key:WriteRecord", "21e7a6ddc4dbe5b3bf06e65b39a228e2eb588af1edc43beb09d91dd72570dd44"}, |
| + {"internal/hostops/console_move_grant.go", "ControlWriter.consoleMoveClaim", "key-site", "key:RecordKey", "265291bd356de061663d89e5ebe33b74ea32c086f8238736d6f971a5afcab722"}, |
| +@@ -696,7 +705,7 @@ |
| {"internal/hostclaim/adopt.go", "Adopt", "key-site", "lit:instances", "24c06b99cce6c638acb19826985cfadd3c736fd0ace7289bcf7ac3dfe784f05a"}, |
| {"internal/hostclaim/claim.go", "ParkScope.Validate", "key-site", "lit:instances", "80b798b0cb679d8c533ff562c0c4804c06f53e7dba126250a3c8182f7ff0c08c"}, |
| {"internal/hostclaim/claim.go", "holderKeyMatches", "key-site", "lit:instances,lit:sidecars", "d15a6841ea6f8cbfa9c8407fd3af020ddeacdfa0743b9e17de412f113faf7979"}, |
| @@ -3643,7 +5022,7 @@ After applying, run `go test ./internal/provision -run TestPlan9B8 -count=1`, `g |
| {"internal/hostclaim/create.go", "BeginCreate", "key-site", "lit:instances", "45ee1e2b1fc9fa8ffdef4f27cf983dccfa3acf7218dccd06f0ee6d12572f4a44"}, |
| {"internal/hostclaim/lossrecord.go", "LossRecordKey", "key-site", "key:LossRecordKey", "1b109b49a8fcdb09f01dbf908a173394072f01afb0c9e324023fb9b3f71f98b3"}, |
| {"internal/hostclaim/lossrecord.go", "ReadLossRecord", "key-site", "key:LossRecordKey", "1237dd3fa3e79a81e6c35f371bf545c9a04841a6297c078d96852daf25b7b483"}, |
| -@@ -808,8 +816,8 @@ |
| +@@ -818,8 +827,8 @@ |
| {"internal/objstore/sidecar.go", "claimRawPutSidecar", "nonrecord", "sel:Put", "233e3d808dd462ff83aecdac5176b180556b948c552aa26f6e0b778df91a24c7"}, |
| {"internal/ops/litestream.go", "FleetDBsWithSkipped", "key-site", "lit:instances", "f62291df0685db15529c4946dca358a29aba66fc31e81aab299de579e37f60f1"}, |
| {"internal/park/remote.go", "locationOf", "key-site", "lit:instances", "83b3efca31b3faadcd5997dc181d81b7bb8fc866cc4a448683bf8423b1bea866"}, |
| @@ -3654,59 +5033,6 @@ After applying, run `go test ./internal/provision -run TestPlan9B8 -count=1`, `g |
| {"internal/park/sweep.go", "Engine.Discover", "key-site", "lit:instances", "69b83e6e859c1c9809ba76d46cb0192888df1ecdcd161b4f2b6435733eecdfa3"}, |
| {"internal/provision/claim_app_writes.go", "UpdateRecord", "record-writer", "key:UpdateRecord,key:claimRawUpdateRecord", "caf714cb78e1280d489c7ad5b3cefb1420e812d541108b1fe13d733abed97229"}, |
| {"internal/provision/provision.go", "ManifestKey", "key-site", "lit:instances", "b8ee0e77bdfade19af79a4012c411c1eb0b6ddca16037e5536fa8597f7cde270"}, |
| ---- a/internal/console/claim_authority_census_test.go |
| -+++ b/internal/console/claim_authority_census_test.go |
| -@@ -25,7 +25,7 @@ |
| - "NewClamAVScanner": true, "NewRoute53Writer": true, "BeginExclusive": true, |
| - "ResumeExclusive": true, "RegisterStart": true, "FinishWakeRetryReset": true, "ReservePurge": true, |
| - "ConsoleMove": true, "InspectMoveLoss": true, "DeclareMoveLoss": true, |
| -- "RunRecovery": true, "runMoveRecovery": true, "captureMoveStartBase": true, "resolveUncertain": true, "ConvertMovingToRelocating": true, |
| -+ "RunRecovery": true, "runMoveRecovery": true, "captureMoveStartBase": true, "resolveUncertain": true, "ConvertMovingToRelocating": true, "ReleaseFencedTarget": true, "VerifyMoveTargetFence": true, |
| - "BeginPurge": true, "SavePurgeJournal": true, "RemovePurgeJournal": true, |
| - "AcquireMailGate": true, "ReleaseMailGate": true, "EnrollNewApp": true, |
| - "EnrollBackfillApp": true, "OpenGated": true, "WithAppMutation": true, |
| ---- a/internal/console/testdata/claim-authority-sites.txt |
| -+++ b/internal/console/testdata/claim-authority-sites.txt |
| -@@ -12,6 +12,8 @@ |
| - ../hostops/lifecycle_grant.go|RecoveryPass|w.RunRecovery(ctx, r.Pass) |
| - ../hostops/move_enrollment.go|RegisterMove|a.Control.RegisterStart(ctx, h.Account, h.App, appID, id, reg, a.Now()) |
| - ../hostops/move_enrollment.go|StartClaimMoveRecord|h.captureMoveStartBase(ctx, seq) |
| -+../hostops/move_fence_release.go|ReleaseFencedTarget|hostclaim.ReleaseFencedTarget(r) |
| -+../hostops/move_fence_release.go|ReleaseFencedTarget|hostclaim.VerifyMoveTargetFence(ctx, g.w.raw, o, ref, g.authority) |
| - ../hostops/move_loss_authority.go|DeclareMoveLoss|a.Control.DeclareMoveLoss(ctx, a.Signer, a.Directory.Bucket, actor, account, app, host, box, in) |
| - ../hostops/move_loss_authority.go|InspectMoveLoss|a.Control.InspectMoveLoss(ctx, a.Signer, a.Directory.Bucket, actor, account, app, host, box) |
| - ../hostops/move_recovery_bridge.go|RecoverTerminatedMovingHost|hostclaim.ConvertMovingToRelocating(mh.Lifecycle(ctx), s, fresh, c.Op.ID, ref, authority, guard) |
| ---- a/approved_moves_copy_test.go |
| -+++ b/approved_moves_copy_test.go |
| -@@ -74,7 +74,7 @@ |
| - "MV-N300", "MV-N301", "MV-N302", "MV-N303", "MV-N304", "MV-N305", "MV-N306", |
| - "MV-N307", "MV-N308", "MV-N309", "MV-N310", "MV-N311", "MV-N312", |
| - "MV-N313", "MV-N314", |
| -- "MV-N315", "MV-N316", "MV-N317", "MV-N318", |
| -+ "MV-N315", "MV-N316", "MV-N317", "MV-N318", "MV-N319", |
| - } |
| - |
| - // copyRow is one row of the record. The File cell is the file the sentence is |
| -@@ -543,6 +543,7 @@ |
| - // copy" task appends the files it touched; TestMovesScanFilesCoverTheDiff |
| - // fails when the list falls behind. |
| - var movesScanFiles = []string{ |
| -+ "internal/hostops/move_fence_release.go", |
| - "internal/hostops/move_recovery_grant.go", |
| - "internal/hostops/move_loss_owner.go", |
| - "internal/host/movesrecoveryprobe.go", |
| ---- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| -+++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| -@@ -308,3 +308,9 @@ |
| - - MV-N146: the row's text carries a note, `(MV-N68 with a new <which>)`. The code says `no fresh rate measurement for the disk's per-file sync` through MV-N68's `NoFreshRate`, which is correct; the note is not a part of any sentence the code could say. |
| - |
| - The 16 rows added after that review (MV-N184 to MV-N186, MV-N300 to MV-N312) were approved by Paul on 2026-10-09 with no edits. |
| -+ |
| -+## Claim Plan 12 rev 2 |
| -+ |
| -+| id | file | text | status | |
| -+|---|---|---|---| |
| -+| MV-N319 | internal/provision/provision.go | target fenced | pending | |
| ``` |
| ## Task H: Amend the claim spec to rev 6.32 |
| @@ -3810,7 +5136,7 @@ The code archive has no copy of this spec, so copying the checked document input |
| ### 6.3 Local retirement (J-I-7, 199-13, 199-14) |
| Retirement covers both layouts. |
| -@@ -2295,6 +2351,81 @@ |
| +@@ -2295,6 +2351,113 @@ |
| the old session certified that every one of its requests had completed. |
| Dead generations are parked by the at-rest sweep (§6.2a). |
| @@ -3823,6 +5149,11 @@ The code archive has no copy of this spec, so copying the checked document input |
| +policy. `LossAdmissionSources` validates each frozen baseline; the writer |
| +fills both primary and additional remote maxima from that one result. |
| +`ParseMoveLossBaseline` is the shared decoder of replica and seal baselines. |
| ++The writer first compares Select's restore maximum with its non-nil restore |
| ++plan. Frozen loss maxima are independent: target M can exceed restored source N. |
| ++Only a frozen ordinary predecessor replaces `RemoteMaxTXID`; an empty frozen |
| ++set preserves Select's validated maximum. AdmissionEvidence receives that loss |
| ++maximum alongside the unchanged restore plan. Never infer zero from no sources. |
| + |
| +The admission CAS records `loss_admitted.<id>` with the admitted session. |
| +This operation-local marker survives admission-history trimming. A restarted |
| @@ -3834,8 +5165,10 @@ The code archive has no copy of this spec, so copying the checked document input |
| +remain refused pending a combination protocol. |
| + |
| +Only a `restoring` source whose nonempty predecessor writer is the move's |
| -+target uses cross-box recovery handover. Reverts and never-admitted holders |
| -+retain the local path. Recovery accepts any of these evidence forms: |
| ++target uses cross-box recovery handover. Initial source returns, including session zero, use the exact source cut, |
| ++source stamp and canonical complete source report through the normal stop |
| ++wiring. A missing report, mismatched cut or changed claim refuses. Later source |
| ++sessions retain their own local writer-stop receipts. Recovery accepts any of these evidence forms: |
| + |
| +- The target's own `TargetReleased` exec-to-none CAS records |
| + `target_session` and `target_generation` from the holder when the target |
| @@ -3851,8 +5184,11 @@ The code archive has no copy of this spec, so copying the checked document input |
| + region, enrollment and completed stop evidence. |
| +- This operation's committed, signed `loss.<id>` naming the exact target |
| + predecessor in its frozen sources. |
| -+- A freshly verified `fence.<target>` naming the operation's target |
| -+ enrollment and region. |
| ++- A consumed fence whose journal marker, claim-level automatic loss identity, |
| ++ enrollment, target and holder incarnation agree. Consumption removes target |
| ++ execution and participation. Completion and handover do not revalidate the |
| ++ dead target or witness against a later roster. An unconsumed fence still |
| ++ requires fresh verification against the target enrollment and region. |
| + |
| +Before admission the claim checks the source executor and predecessor; |
| +afterward it checks the cold session and fresh stamp. The source never |
| @@ -3860,7 +5196,13 @@ The code archive has no copy of this spec, so copying the checked document input |
| +for both restoring and reverting: the guarded local hook retires the source's |
| +old database and marker at `JournalSourceStamp`, using `src_gen` and the |
| +source cut evidence. Later source sessions retain the root writer-stop receipt |
| -+requirement. The descriptor-based guard and exact inventory checks remain. |
| ++requirement. After activation and before prelaunch, the source reads its exact original |
| ++StopInventoryDB installation. Under the descriptor guard it rebinds the exact |
| ++registry route and timers transactionally, then transfers the inventoried paths |
| ++and ownership to the admitted stamp. Physical identities and old inventory are |
| ++retained. Missing inventory refuses; a replay verifies the existing successor. |
| ++Root's inventory-write transport still validates registry ownership. The |
| ++registry call uses the existing isolated transport; root opens no SQLite DB. |
| + |
| +`ReleaseFencedTarget` both declares and consumes a verified `ReturnFence` in |
| +one CAS. It rolls back an uncommitted landing if needed, removes the target |
| @@ -3869,18 +5211,34 @@ The code archive has no copy of this spec, so copying the checked document input |
| +return lane as obligations. It records a release marker that `ReturnToSource` |
| +accepts without giving the source console authority. The reference never |
| +acknowledges those parks. Replays preserve scopes and do not requeue completed |
| -+work. A fenced recovery cannot claim clean-return coverage. |
| ++work. A fenced recovery cannot claim clean-return coverage. The same CAS appends |
| ++Lost, LossRefs and AutomaticLoss outside the operation journal, retaining the |
| ++original fence observation time, last target admission, regional chores, |
| ++unknown timer knowledge, no local-discharge assertion and pending reporting. |
| ++No quarantine revision is invented. These values survive operation and holder |
| ++replacement. The console recovery pass materializes a signed loss record with |
| ++an optional `automatic` authority payload before marking the retained evidence |
| ++reported. The signed record also retains `alerts_pending:true`; materialization |
| ++does not count as notification, and the delivery owner must consume this work. |
| ++Retries preserve and compare the exact consequences. The optional |
| ++payload is part of the reader rollout before this writer is enabled. |
| + |
| +Moves consumes `MoveRecoveryGrant.ReleaseFencedTarget` from the console's |
| -+`Recovery.Moves` hook. The grant first records the rollback decision with reason `target fenced`, |
| -+the immutable fence key in `LossRef` and a new `LostAt` bound, then releases |
| -+the claim, then records the target release with the fence as evidence. Each |
| -+callback performs at most one mutation and verifies the fence again; moves |
| -+calls the same method on later passes. The grant rechecks both record ETag |
| -+and `record_seq` before its claim CAS. The persisted fence-decision time, |
| -+not an older rollback request time, bounds the parks. The source continues to wait |
| -+for the claim's `released` phase, so record-first ordering remains safe. |
| -+A dirty fenced target requires J4 declared loss; it cannot create a return lane. |
| ++`Recovery.Moves` hook. Its first pass records a fence rollback decision when |
| ++needed, preserving the first rollback reason. `LossRef` identifies that fence |
| ++form. The next pass revokes the claim; the final pass completes the record from |
| ++the consumed claim evidence. Each grant performs at most one mutation and |
| ++rechecks record ETag and sequence before its claim CAS. New parks use the |
| ++verified fence's original observation time, never a retry or rollback clock. |
| ++If a target wrote a clean released record and died before releasing the claim, |
| ++the grant verifies its canonical complete stopped-target report and revokes the |
| ++claim directly, preserving the record and its original evidence. |
| ++`MoveRecoveryGrant.StampLostMoveTargetParked` replaces the console's separate |
| ++raw-record closure. It requires revoked target authority and no owed target or |
| ++target-region chore. A target fenced before install needs no invented local |
| ++park. Existing local parks remain owed until their own authorized completion. |
| ++A dirty target still requires J4 declared loss. The unused abort-complete action |
| ++is removed; abort owners retain their record-first close. |
| +`ClaimProviders.VerifyMoveTargetFence` exposes the box-side roster-backed |
| +verifier for callers that need a typed fence, without granting a write. |
| + |
| @@ -3892,7 +5250,7 @@ The code archive has no copy of this spec, so copying the checked document input |
| ### 6.5 Door reads pinned to versions (J-N10) |
| The pull door's listing: |
| -@@ -2595,7 +2726,7 @@ |
| +@@ -2595,7 +2758,7 @@ |
| 1. Validate its enrollment, effective region, bucket and retirement authority. |
| 2. Resolve immutable evidence references covering every producer that could write the scope. |
| @@ -3901,7 +5259,7 @@ The code archive has no copy of this spec, so copying the checked document input |
| 4. Re-read the claim and validate the exact obligation, original operation, phase and immutable scope. |
| 5. Obtain P6-D3’s view under that observation. |
| -@@ -2618,8 +2749,8 @@ |
| +@@ -2618,8 +2781,8 @@ |
| Rejected alternatives: |
| @@ -3918,19 +5276,19 @@ The code archive has no copy of this spec, so copying the checked document input |
| | Delta / moves task | Consumer work and gate | |
| |---|---| |
| -| M2 / Plan 4 Task 10 | Register `hostops.MoveLossRecoveryOwner()` and a baseline validator using `ParseMoveLossBaseline`. Select the declared return kind and refuse clean-return whenever loss evidence exists. The writer fills maxima from the validated frozen sources. Run moves' real-declaration and every-session-once acceptance tests, including a restarted probe and later cold source admission. | |
| +| M2 / Plan 4 Task 10 | Register `hostops.MoveLossRecoveryOwner()` and a baseline validator using `ParseMoveLossBaseline`. Select the declared return kind and refuse clean-return whenever loss evidence exists. The writer validates the restore baseline independently. For loss admissions, AdmissionEvidence receives the frozen predecessor maximum, not the restore point; with no frozen predecessor it receives Select's unchanged maximum. Run moves' real-declaration and every-session-once acceptance tests, including a restarted probe and later cold source admission. | |
| | M3 / Task 9b | Use the existing unsigned target stop report and `TargetReleased`; no `TargetStop` adapter. Preserve recovery input policy and fresh stamp. Run `TestRecoveryAdmissionAfterTargetSession` with real registered owners for target journal, loss and fence, then continue through restore. | |
| | M5/M6 / Task 9a | Retain the own-location `RenewingLeases` handle through `published` and call `RunHeld`. Use the claim-returned canonical resolver. Bind staging parks to persisted evidence time and pin the prepared generation while parking. Test with real `AggregateLeases`. | |
| | M7/M8 / Task 5 | Provide `VerifyRetirement` and `VerifyMovesSweep` for occupied hosts' moves namespace only, using no live move, matching retained release/LastMove and complete lane evidence. Drop the obsolete instance-residue and bare-live-location refusals. The claim supplies free-host proof and instance labels. | |
| | M9 / Task 9a | Consume all four `MoveParks` results and use its unwrapped local authorizer. Keep each move arm's exact-scope checks and the regional engine's stronger wrapper. Check bucket/region before arming. | |
| -| M12 / Task 7 recovery hook | In `Recovery.Moves`, call the record-bound `MoveRecoveryGrant.ReleaseFencedTarget` on successive passes. It records the decision, releases the claim, then records completion; no raw record or claim handle is exposed. The source waits for claim `released`. An admitted fenced target returns cold and uncovered. A dirty fenced target requires J4 loss. The box-side verifier is available through `ClaimProviders`. | |
| +| M12 / Task 7 recovery hook | In `Recovery.Moves`, call the record-bound `MoveRecoveryGrant.ReleaseFencedTarget` on successive passes. It records the decision, releases the claim, then records completion; no raw record or claim handle is exposed. The source waits for claim `released`. Any fenced target returns cold and uncovered. Use the grant-bound lost-target stamp; consumed-fence completion must not read the live roster. Preserve existing rollback reasons and handle clean record release before claim release. A dirty fenced target requires J4 loss. The box-side verifier is available through `ClaimProviders`. | |
| | M21 / Task 9b | Claim-owned retirement now handles restoring and reverting. Moves supplies the selected restore inputs and runs `TestCleanReturnAdmissionRetiresTheSTaggedDatabase` against the integrated provider through restore. | |
| | M1/M10 | Implement C8's files provider. Spec §6.6 P6-D3 already requires cross-region landing to materialize destination-local objects and reference those. There is no open decision allowing retained source-region object references after commit. | |
| | M4/M11/M13–M20 | Moves retains record-first closes, recorded revert inputs, its single recovery/alerting hook, durable rate limits, lost-box gate, carries/install fixes, stuck reason, provider arming and its own censuses. J4's declare-lost sequence stays. | |
| This authoring session makes no commit, merge, activation or production change. |
| Land the implementation through a reviewed amadan branch and verify CI on its |
| -exact tip, then main. Integrate moves' Task 5 and Task 7 census changes serially. |
| +exact tip, then main. The pinned base already contains moves' Task 5 and Task 7 changes. |
| Keep the retired-location write gate off through reader rollout. Missing legacy |
| history remains a typed refusal; any evidence backfill needs its own design. |
| Production moves stays unarmed until C8 and the integrated recovery/restore tests |
| @@ -3945,7 +5303,7 @@ errors are reused. No CLI, HTTP or page text changes. |
| |---|---|---| |
| | HC12-D1 | `host claims: missing provider: claim archived incarnation and location evidence` | `claimRetirementDependency`; typed census refusal. | |
| | HC12-D2 | `host claims: missing provider: moves Plan 5 retired-move regional evidence` | Move-park registration dependency. | |
| -| HC12-D3 / MV-N319 | `target fenced` | `provision.RollbackTargetFenced`; controller-specified recorded reason, with a pending moves-copy row. | |
| +| HC12-D3 / MV-N335 | `target fenced` | `provision.RollbackTargetFenced`; controller-specified recorded reason, with a pending moves-copy row. | |
| The implementation owner must complete applicable copy approval before shipment. |
| Test assertion strings are not shipped. The report is the controller's per-finding |
| @@ -3953,32 +5311,23 @@ implementation handoff. |
| ## Verification and authoring results |
| -Verified in a fresh export of the pinned base with the controller-designated |
| -shared cache. All 14 patch stages applied with `--fuzz=0`, without offsets; |
| -the 54 resulting files matched the authored implementation byte for byte. |
| -Tasks A–G each demonstrated a failing pre-implementation check and a passing |
| -post-implementation check. The final B, C and G revisions repeated that |
| -sequence; unchanged A, D, E and F retained their earlier results. |
| - |
| -On the complete final replay, the selected recovery/loss/retirement/park tests |
| -passed in all six affected packages. The B8 record-write census, console |
| -authority census and moves copy-inventory tests passed. `go build ./...`, |
| -`go vet ./...`, Darwin amd64 and Windows amd64 builds passed; `gofmt -l .` |
| -returned no files. The restore acceptance fixture covers restoring, reverting |
| -and clean-return through restore, with its fixture limits recorded in the report. |
| - |
| -Broader checks encountered sandbox listener refusals: the persisted-launch |
| -retirement test could not open its Unix socket, and console tests could not |
| -open loopback TCP listeners (`operation not permitted`). Those are not passes; |
| -full listener-dependent CI remains required before landing. Markdown whitespace |
| -checking found only the required space prefixes on unified-diff context lines; |
| -no prose whitespace defect was found. |
| +The final replay on `e79d71b5` applied all 14 patch stages without offsets or |
| +fuzz, and its 69 changed files matched the authored tree byte for byte. Every |
| +tests-first check failed for the absent API or behavior, then passed after its |
| +implementation. The final focused tests, censuses, build, vet, Darwin/amd64 and |
| +Windows/amd64 builds passed; `gofmt -l .` was empty. The broader affected-package |
| +run passed seven packages but stopped on sandbox TCP listener failures in |
| +`TestOpenDeployStoreSelection/fleet_box_with_no_bucket_speaks_the_door` and |
| +`TestAccountFleetTabMemberReadable`. These are not passes; full CI still needs |
| +an environment permitting listeners. Commands and evidence are in the rev 3 report. |
| +The source-return acceptance uses actual Prelaunch and PrepareState; only kernel |
| +cgroup preparation is a fixture. |
| Replay and validate the final plan as follows, using the shared-cache exports |
| from the bootstrap section: |
| ```bash |
| -go test ./internal/host ./internal/hostclaim ./internal/hostbox ./internal/hostops ./internal/hostguard ./internal/park -run 'TestPlan12|TestLossBeforeTargetAdmissionRecovers|TestAdmitWriterWithLoss|TestAggregateLeases|TestRegionalWorker|Test.*Sweep|TestReleaseLostTarget|TestPlan9D2CrossBox|TestPlan9D4|TestPlan10J2' -count=1 |
| +go test ./internal/host ./internal/hostclaim ./internal/hostbox ./internal/hostops ./internal/hostguard ./internal/park ./internal/registry -run 'TestPlan12|TestLossBeforeTargetAdmissionRecovers|TestAdmitWriterWithLoss|TestAggregateLeases|TestRegionalWorker|Test.*Sweep|TestReleaseLostTarget|TestPlan9D2CrossBox|TestPlan9D4|TestPlan10J2' -count=1 |
| go test ./internal/provision -run TestPlan9B8 -count=1 |
| go test ./internal/console -run TestClaimAuthoritySourceCensus -count=1 |
| go test . -run 'TestMoves(StringsAreRecorded|SinksAreSentences)' -count=1 |