plan: claim Plan 10 rev 3 (review r2: supervised held recovery probes under the cut fence, legacy cut identity in declarations)
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+570
−44
docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md+570 −44
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md b/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md| index f9a0f8d4c..efcd22cea 100644 |
| --- a/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md |
| +++ b/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md |
| @@ -1,9 +1,11 @@ |
| # Host lifecycle claim, Plan 10: moves recovery and console grants |
| -Implementation plan, 2026-10-09, rev 2. This plan extends the claim's recovery-probe contract, gives move loss declarations a registered operation owner, and admits console move writes through typed authority. Scope: J2, J4 and J17, as ruled with Paul on 2026-10-08. It authorizes no production change. |
| +Implementation plan, 2026-10-09, rev 3. This plan extends the claim's recovery-probe contract, gives move loss declarations a registered operation owner, and admits console move writes through typed authority. Scope: J2, J4 and J17, as ruled with Paul on 2026-10-08. It authorizes no production change. |
| **Rev 2:** answers review round 1's three P2 findings and the moves Plan 4 addendum. Console grants retain the gate-read move ID, incarnation and `record_seq`; lost-target release retains inventory-bound local retirement; D5 discharges only separately attested local chores, including after commit. Task A records clean-return inputs at `probing` and connects the recovery hook to Group D through the activator. The tests below reproduce the rev 1 failures before applying these changes. |
| +**Rev 3:** answers both review round 2 findings. Task A supervises held recovery probes before the sweep skips fenced hosts, retires failed probes on sweep or join, and releases only the adapter-bound operation fence when the exact admitted session opens. Task D accepts the paired legacy cut identity `{session: 0, generation: ""}` while refusing missing journal evidence and mismatched pairs. Failing-first tests cover the real sweep with its cut fence present and complete loss declarations before and after the target's first admission. The J4 signature stays unchanged. |
| + |
| ## J4 owner interface |
| **The J4 `OperationOwner` signature is unchanged from rev 1.** Moves Plan 4 callers may keep building against it. Local-only declarations use the same inspection, signed consequences and owner hooks; no extra caller-supplied authority or interface method is introduced. |
| @@ -153,7 +155,7 @@ cd "$CLAIM10_SCRATCH/overlay" |
| Apply A, B, C, D and E in order. In each task, apply `tests`, run its command and |
| record the expected failure, then apply `implementation` and repeat the same |
| -command to obtain green. Do not accept unrelated failures as the expected red. |
| +command to obtain green, then build every package before the next task. Do not accept unrelated failures as the expected red. |
| The bootstrap separately repairs the combined-base alias, so the task tests |
| reach their behavioral assertions. |
| @@ -167,7 +169,13 @@ Reuse `OperationStart`, `OperationAttempts`, `ClaimProbeReadiness` and `ClaimOwn |
| Task A supplies `recoveryProbeAdapter` and the production binding `armRecoveryProbe(cfg, w, svc.RecoveryProbe, svc.RecoveryProbeReady)`, reached only with both moves and host claims enabled. `activator.Service.RecoveryProbe(context.Context, registry.Route, activator.StartPolicy) error` runs the admitted Group D start path and retains the healthy process for the later opening step. The hook keeps the existing `recoveryProbeRequest` and outcome signature, so moves Plan 4 Task 9 can consume it directly. Missing claim runtime, operation owners or restoration evidence still refuses before execution. With host claims off, `armPhase1` keeps the legacy nil recovery hook. |
| -The adapter passes the recorded inputs in `StartPolicy.Inputs`, sets `Probe`, and uses a 30-second readiness window with at most three starts in one wake. The moves `OperationAttempts` provider owns the durable nine-start recovery budget across wakes and the original first-start time; its exhausted result maps to `recoveryProbeExhausted`. A failed wake window or changed authority leaves `probing` for another pass. The readiness predicate checks runtime wiring before input recording. An in-flight or held probe with superseded inputs is cancelled or retired, including a late healthy result after cancellation; it cannot strand the newly recorded inputs behind the old process. Edge reloads preserve held operation-owned probes, while explicit fences and claim/session checks still stop them. The restore/install, park leases, recovery admission evidence and source/target cleanup remain the moves owner's work. This task adds no generic recovery takeover or artifact-retirement authority. |
| +The adapter passes the recorded inputs in `StartPolicy.Inputs`, sets `Probe`, and uses a 30-second readiness window with at most three starts in one wake. The moves `OperationAttempts` provider owns the durable nine-start recovery budget across wakes and the original first-start time; its exhausted result maps to `recoveryProbeExhausted`. A failed wake window or changed authority leaves `probing` for another pass. The readiness predicate checks runtime wiring before input recording. An in-flight or held probe with superseded inputs is cancelled or retired, including a late healthy result after cancellation; it cannot strand the newly recorded inputs behind the old process. |
| + |
| +Edge reloads preserve held operation-owned probes. The actual sweep supervises these entries before its ordinary fenced-host skip: a dead process, lost replication lease or replaced fence retires the probe and retains the installed fence for a retry. Joining a failed held probe also starts retirement. |
| + |
| +The adapter supplies `StartPolicy.RecoveryFence` from `moveFenceGeneration(rec.Move)`; an unrelated installed fence refuses admission. The entry retains the fence observed at start. After the off-lock claim read, it rechecks health and fence identity under `a.mu`. Only the exact admitted session at `opened` (or retained through release to `serving`) ends the hold and removes the owning generation under that same lock. A replacement fence is preserved even if it arrives during the claim read. If the moves worker already lifted its own fence, keep a healthy exact session held until the claim confirms opening; a stale `probing` answer cannot retire an already recovered process or enable routing. Removing a fence never substitutes for an opened claim. No fence is required after an agent restart, and an absent fence grants no authority to lift a later one. |
| + |
| +The restore/install, park leases, recovery admission evidence and source/target cleanup remain the moves owner's work. This task adds no generic recovery takeover or artifact-retirement authority. |
| A clean return records inputs **at `Rollback.SourcePhase == probing`, before rendering or materializing anything**. The adapter calls `recoveryProbeRequest.Record`, which uses `provision.MoveRecordRecoveryInputs` under the move and rollback IDs. That transition is already on the pinned moves base and accepts `probing` for both clean and dirty returns. Its `Release` must be nonempty. A failed record write stops the attempt before render. The clean return's path has no `configured` phase and needs none. Each restart records the newly selected inputs before its own render. A revert analogously uses the already-built `MoveRecordRevertInputs`. None of these values is compared to the original `Move.Inputs` pin. |
| @@ -177,6 +185,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| # After recording the expected red: |
| python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" A implementation |
| flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostclaim ./internal/host ./internal/activator -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10J2|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe' |
| +flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./... |
| ``` |
| #### New file: `internal/host/plan10_j2_test.go` |
| @@ -933,7 +942,7 @@ func TestPlan10J2KeepWriterSessionGuardsARecoveryStep(t *testing.T) { |
| +++ b/internal/activator/activator.go |
| @@ -71,0 +72 @@ |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| -@@ -347,0 +349,11 @@ |
| +@@ -347,0 +349,12 @@ |
| + // Recovery probe entries (RecoveryProbe, recoveryprobe.go). probe is the |
| + // caller's probe policy, set at construction and never written again; |
| + // nil for every other entry, whose wake is main's. probeAdmitted is the |
| @@ -945,34 +954,38 @@ func TestPlan10J2KeepWriterSessionGuardsARecoveryStep(t *testing.T) { |
| + probeHeld bool |
| + probeReviewing bool |
| + probeObsolete bool |
| -@@ -751,0 +764,5 @@ |
| ++ probeFence uint64 |
| +@@ -751,0 +765,5 @@ |
| + // A held recovery probe serves nothing until the source opens. |
| + if e.probeHeld { |
| + a.mu.Unlock() |
| + return nil, ErrFenced |
| + } |
| -@@ -836,0 +854,4 @@ |
| +@@ -836,0 +855,4 @@ |
| + } |
| + if e.probeHeld { |
| + a.mu.Unlock() |
| + return ErrFenced |
| -@@ -1428,0 +1450,5 @@ |
| +@@ -1428,0 +1451,5 @@ |
| + if e != nil && e.probeHeld { |
| + // A held recovery probe runs the moves-held inputs, not delivered |
| + // config: the recovery owns its process until the source opens. |
| + return false |
| + } |
| -@@ -2507,0 +2534,9 @@ |
| -+ continue |
| -+ } |
| -+ // A held recovery probe has no idle clock and no convergence: the |
| -+ // claim decides when it serves or stops (reviewRecoveryHold). |
| -+ if e.probeHeld { |
| +@@ -2450,0 +2478,12 @@ |
| ++ // Recovery retains its cut fence. Supervise its process and lease |
| ++ // before the ordinary fenced-host skip; only the claim can open it. |
| ++ if e.probeHeld && e.state == actAwake { |
| ++ if a.retireRecoveryProbeLocked(e) { |
| ++ continue |
| ++ } |
| + if !e.probeReviewing { |
| + e.probeReviewing = true |
| + go a.reviewRecoveryHold(e) |
| + } |
| -@@ -2835 +2870,3 @@ |
| ++ continue |
| ++ } |
| +@@ -2835 +2874,3 @@ |
| - if e.worker { |
| + // A held recovery belongs to its claim operation, not the edge's |
| + // serving set. Explicit fences and claim/session review still stop it. |
| @@ -984,7 +997,22 @@ func TestPlan10J2KeepWriterSessionGuardsARecoveryStep(t *testing.T) { |
| ```diff |
| --- a/internal/activator/claimwake.go |
| +++ b/internal/activator/claimwake.go |
| -@@ -105 +105,7 @@ |
| +@@ -22,6 +22,8 @@ |
| +- MaxStarts int |
| +- Window time.Duration |
| +- Deadline time.Time |
| +- Probe bool |
| +- Prepared *moves.InstallManifest |
| +- Inputs *provision.MoveInputs |
| ++ // RecoveryFence is the owning move's cut generation, supplied by its adapter. |
| ++ RecoveryFence uint64 |
| ++ MaxStarts int |
| ++ Window time.Duration |
| ++ Deadline time.Time |
| ++ Probe bool |
| ++ Prepared *moves.InstallManifest |
| ++ Inputs *provision.MoveInputs |
| +@@ -105 +107,7 @@ |
| - if err := hostbox.WakePhase(observed.Claim, r.Box, false); err != nil { |
| + // A recovery probe entry carries its caller's probe policy; every other |
| + // wake runs main's serving policy. |
| @@ -993,13 +1021,13 @@ func TestPlan10J2KeepWriterSessionGuardsARecoveryStep(t *testing.T) { |
| + policy = *e.probe |
| + } |
| + if err := hostbox.WakePhase(observed.Claim, r.Box, policy.Probe); err != nil { |
| -@@ -106,0 +113,3 @@ |
| +@@ -106,0 +115,3 @@ |
| + } |
| + if e.probe != nil && (!policy.Probe || !recoveryProbeKind(observed.Claim.Op.Kind)) { |
| + return nil, hostbox.ErrChanged |
| -@@ -122 +130,0 @@ |
| +@@ -122 +132,0 @@ |
| - policy := startPolicy{MaxStarts: 5, Window: 30 * time.Second} |
| -@@ -133,0 +142,3 @@ |
| +@@ -133,0 +144,3 @@ |
| + if e.probe != nil { |
| + e.probeAdmitted = admitted |
| + } |
| @@ -1046,10 +1074,12 @@ import ( |
| // |
| // Fences. The rollback holds its cut fence on this host through the whole |
| // recovery, and the claim's exact probing executor, not the fence, is the |
| -// authority this entry starts under: the probe runs with the fence installed |
| -// and the fence keeps everything else out. A fence installed, replaced or |
| -// lifted while the probe runs ends it: FenceAndStop cancels it like any |
| -// wake, and a healthy start that finds the fence changed is stopped. |
| +// authority this entry starts under. The adapter supplies its own cut fence |
| +// generation in RecoveryFence; an unrelated installed fence refuses the wake. |
| +// The probe runs with its own fence installed |
| +// and the fence keeps everything else out. Installing or replacing a fence |
| +// stops the probe. If the moves worker already lifted its own fence, the |
| +// claim must still confirm the exact opened session before the entry serves. |
| // |
| // A healthy start (OperationAttempts.Healthy accepted it) is retained: the |
| // entry is awake with the started process and its replication, held. While |
| @@ -1057,7 +1087,8 @@ import ( |
| // Bounce; a lost lease or an exit still stands it down. The sweep reads the |
| // claim for it (reviewRecoveryHold): once the source opened on the same |
| // session (or the op released to serving keeping it) the hold ends and the |
| -// entry serves as any awake entry; once the claim names another session or |
| +// owning cut fence is released and the entry serves as any awake entry; |
| +// once the claim names another session or |
| // executor, the entry is stopped. |
| func (a *Service) RecoveryProbe(ctx context.Context, rt registry.Route, p StartPolicy) error { |
| if a == nil || ctx == nil { |
| @@ -1076,6 +1107,10 @@ func (a *Service) RecoveryProbe(ctx context.Context, rt registry.Route, p StartP |
| a.mu.Unlock() |
| return ErrFenced |
| } |
| + if fence := a.fences[rt.Host]; fence != 0 && fence != p.RecoveryFence { |
| + a.mu.Unlock() |
| + return ErrFenced |
| + } |
| if e := a.entries[rt.Host]; e != nil { |
| a.mu.Unlock() |
| return a.joinRecoveryProbe(ctx, e, p) |
| @@ -1130,14 +1165,15 @@ func recoveryProbeKind(kind string) bool { |
| } |
| func sameRecoveryProbe(x, y StartPolicy) bool { |
| - return x.MaxStarts == y.MaxStarts && x.Window == y.Window && x.Deadline.Equal(y.Deadline) && x.Probe == y.Probe && |
| + return x.RecoveryFence == y.RecoveryFence && x.MaxStarts == y.MaxStarts && x.Window == y.Window && x.Deadline.Equal(y.Deadline) && x.Probe == y.Probe && |
| x.Prepared == nil && y.Prepared == nil && x.Inputs != nil && y.Inputs != nil && reflect.DeepEqual(*x.Inputs, *y.Inputs) |
| } |
| // joinRecoveryProbe answers a call that found host's entry already present. |
| // Only a probe entry under the same policy is joined: a probe in flight |
| // shares its outcome, and a retained healthy probe answers once its process, |
| -// its replication and the claim's session still hold. It changes nothing. |
| +// its replication and the claim's session still hold. A failed or superseded |
| +// held probe is retired so the next pass can spend its remaining attempts. |
| func (a *Service) joinRecoveryProbe(ctx context.Context, e *actEntry, p StartPolicy) error { |
| for { |
| a.mu.Lock() |
| @@ -1175,17 +1211,26 @@ func (a *Service) joinRecoveryProbe(ctx context.Context, e *actEntry, p StartPol |
| return ctx.Err() |
| } |
| case actAwake: |
| - admitted, proc, rep := e.probeAdmitted, e.proc, e.rep |
| - a.mu.Unlock() |
| - if proc == nil || proc.Exited() || rep == nil || rep.Err() != nil { |
| + if a.retireRecoveryProbeLocked(e) { |
| + a.mu.Unlock() |
| return hostbox.ErrChanged |
| } |
| + admitted := e.probeAdmitted |
| + a.mu.Unlock() |
| r := a.cfg.Claims |
| fresh, err := r.Read(ctx, e.host) |
| if err != nil { |
| return err |
| } |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + if a.entries[e.host] != e || e.state != actAwake || a.retireRecoveryProbeLocked(e) { |
| + return hostbox.ErrChanged |
| + } |
| _, err = recoveryHoldVerdict(admitted, fresh, r.Box) |
| + if err != nil && e.probeHeld { |
| + a.beginStopLocked(e) |
| + } |
| return err |
| default: |
| a.mu.Unlock() |
| @@ -1196,7 +1241,8 @@ func (a *Service) joinRecoveryProbe(ctx context.Context, e *actEntry, p StartPol |
| // finishRecoveryProbe resolves the probe entry: a failed probe is forgotten |
| // as a failed wake is; a healthy one is retained, held, unless the fence it |
| -// started under changed, when it is stopped and the call answers ErrFenced. |
| +// started under was replaced, when it stops and answers ErrFenced. An owner |
| +// lift is checked against the exact claim session by the hold review. |
| func (a *Service) finishRecoveryProbe(e *actEntry, fence uint64, w *wokenInstance, err error) error { |
| a.mu.Lock() |
| if err == nil && (w == nil || w.proc == nil || w.rep == nil) { |
| @@ -1218,6 +1264,7 @@ func (a *Service) finishRecoveryProbe(e *actEntry, fence uint64, w *wokenInstanc |
| } |
| now := a.now() |
| e.state = actAwake |
| + e.probeFence = fence |
| e.rep, e.proc = w.rep, w.proc |
| e.spawnVersion = w.spawnVersion |
| e.lastActive, e.spawnedAt = now, now |
| @@ -1225,7 +1272,7 @@ func (a *Service) finishRecoveryProbe(e *actEntry, fence uint64, w *wokenInstanc |
| if e.probeObsolete { |
| e.err = hostbox.ErrChanged |
| a.beginStopLocked(e) |
| - } else if a.shuttingDown || a.fences[e.host] != fence { |
| + } else if a.shuttingDown || a.recoveryFenceChangedLocked(e) { |
| log.Printf("activator: %s recovery probe healthy, but its fence changed; stopping it", e.host) |
| e.err = ErrFenced |
| a.beginStopLocked(e) |
| @@ -1289,7 +1336,12 @@ func (a *Service) reviewRecoveryHold(e *actEntry) { |
| a.mu.Lock() |
| defer a.mu.Unlock() |
| e.probeReviewing = false |
| - if err != nil || a.entries[e.host] != e || e.state != actAwake || !e.probeHeld { |
| + if a.entries[e.host] != e || e.state != actAwake || !e.probeHeld { |
| + return |
| + } |
| + // Recheck after the off-lock read: a failed child, lost lease or a |
| + // replacement fence must not be opened by a late claim answer. |
| + if a.retireRecoveryProbeLocked(e) || err != nil { |
| return |
| } |
| open, err := recoveryHoldVerdict(admitted, fresh, r.Box) |
| @@ -1298,11 +1350,40 @@ func (a *Service) reviewRecoveryHold(e *actEntry) { |
| log.Printf("activator: %s recovery probe's session is over (%v); stopping it", e.host, err) |
| a.beginStopLocked(e) |
| case open: |
| + // The entry retains both the observed fence and the adapter's |
| + // owning generation. Check and remove under the same lock as the |
| + // hold transition; never lift a replacement operation's fence. |
| + if e.probe.RecoveryFence != 0 && a.fences[e.host] == e.probe.RecoveryFence { |
| + delete(a.fences, e.host) |
| + } |
| log.Printf("activator: %s source opened on the recovery probe's session; serving", e.host) |
| e.probeHeld = false |
| e.lastActive = a.now() |
| } |
| } |
| + |
| +// retireRecoveryProbeLocked is shared by the real sweep and callers joining |
| +// a retained probe. Retirement leaves the cut fence in place for the retry. |
| +// The caller holds a.mu and has verified that e is the current awake entry. |
| +func (a *Service) retireRecoveryProbeLocked(e *actEntry) bool { |
| + fenceChanged := e.probeHeld && a.recoveryFenceChangedLocked(e) |
| + if e.probeObsolete || fenceChanged || e.proc == nil || e.proc.Exited() || e.rep == nil || e.rep.Err() != nil { |
| + a.beginStopLocked(e) |
| + return true |
| + } |
| + return false |
| +} |
| + |
| +// A moves worker may have closed the rollback and lifted its own fence |
| +// before the activator reads opened/serving. Defer that case to the claim |
| +// verdict, keeping probeHeld in force while the read is outstanding. |
| +func (a *Service) recoveryFenceChangedLocked(e *actEntry) bool { |
| + fence := a.fences[e.host] |
| + if fence == 0 && e.probe != nil && e.probeFence == e.probe.RecoveryFence { |
| + return false |
| + } |
| + return fence != e.probeFence || (fence != 0 && (e.probe == nil || fence != e.probe.RecoveryFence)) |
| +} |
| ``` |
| #### New file: `internal/activator/recoveryprobe_test.go` |
| @@ -1696,6 +1777,7 @@ func TestRecoveryProbeIsCancelledByAFenceInstalledMidProbe(t *testing.T) { |
| func heldProbe(t *testing.T, a *Service, fence uint64, admitted hostclaim.Observed, in *provision.MoveInputs) (*actEntry, *fakeProc, *fakeRep, error) { |
| t.Helper() |
| p := probePolicy(in) |
| + p.RecoveryFence = fence |
| e := &actEntry{host: probeHost, route: registry.Route{Host: probeHost}, state: actWaking, ready: make(chan struct{}), probe: &p, probeHeld: true} |
| a.mu.Lock() |
| a.entries[probeHost] = e |
| @@ -1723,7 +1805,6 @@ func TestRecoveryProbeRetainsTheHealthyProcessHeldUntilOpened(t *testing.T) { |
| if state != actAwake || !held || !kept || proc.isStopped() { |
| t.Fatalf("state=%v held=%v kept=%v stopped=%v", state, held, kept, proc.isStopped()) |
| } |
| - delete(a.fences, probeHost) |
| if _, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) { |
| t.Fatalf("a request reached the held probe: %v", err) |
| } |
| @@ -1733,24 +1814,24 @@ func TestRecoveryProbeRetainsTheHealthyProcessHeldUntilOpened(t *testing.T) { |
| // A replayed call for the same session answers from the retained process. |
| r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil } |
| - if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); err != nil { |
| + if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, *e.probe); err != nil { |
| t.Fatalf("replay: %v", err) |
| } |
| moved := probeAdmitted(admitted) |
| r.Read = func(context.Context, string) (hostclaim.Observed, error) { return moved, nil } |
| - if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, hostbox.ErrChanged) { |
| + if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, *e.probe); !errors.Is(err, hostbox.ErrChanged) { |
| t.Fatalf("replay on another session: %v", err) |
| } |
| proc.exit() |
| r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil } |
| - if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, hostbox.ErrChanged) { |
| + if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, *e.probe); !errors.Is(err, hostbox.ErrChanged) { |
| t.Fatalf("replay on an exited process: %v", err) |
| } |
| } |
| func TestRecoveryProbeStopsAHealthyStartWhenTheFenceMoved(t *testing.T) { |
| obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box")) |
| - for name, now := range map[string]map[string]uint64{"installed": {probeHost: 3}, "lifted": {}, "replaced": {probeHost: 12}} { |
| + for name, now := range map[string]map[string]uint64{"installed": {probeHost: 3}, "replaced": {probeHost: 12}} { |
| t.Run(name, func(t *testing.T) { |
| a, _, _ := probeService(t, obs) |
| a.fences = now |
| @@ -2208,7 +2289,7 @@ func (a recoveryProbeAdapter) probe(ctx context.Context, s release.Store, rec pr |
| for k, v := range in.BundleETags { |
| recorded.BundleETags[k] = v |
| } |
| - err = a.Wake(ctx, *rt, activator.StartPolicy{MaxStarts: recoveryProbeStartsPerWake, Window: recoveryProbeWindow, Probe: true, Inputs: &recorded}) |
| + err = a.Wake(ctx, *rt, activator.StartPolicy{RecoveryFence: moveFenceGeneration(rec.Move), MaxStarts: recoveryProbeStartsPerWake, Window: recoveryProbeWindow, Probe: true, Inputs: &recorded}) |
| switch { |
| case err == nil: |
| return recoveryProbeAnswered, nil |
| @@ -2401,7 +2482,7 @@ func TestPlan10J2AdapterRecordsInputsBeforeTheGroupDProbe(t *testing.T) { |
| if got := strings.Join(events, ","); got != "record,wake,admit,render" { |
| t.Fatalf("order: %s", got) |
| } |
| - if len(policies) != 1 || !policies[0].Probe || policies[0].MaxStarts != recoveryProbeStartsPerWake || policies[0].Window != 30*time.Second { |
| + if len(policies) != 1 || !policies[0].Probe || policies[0].MaxStarts != recoveryProbeStartsPerWake || policies[0].Window != 30*time.Second || policies[0].RecoveryFence != moveFenceGeneration(rec.Move) { |
| t.Fatalf("policy: %+v", policies) |
| } |
| if got := strings.Join(ops.calls, ","); got != "open,reserve-0" { |
| @@ -2655,6 +2736,298 @@ func TestPlan10J2ArmPhase1ConnectsRecoveryOnlyWithClaimFlag(t *testing.T) { |
| } |
| ``` |
| +#### New file: `internal/activator/recoveryprobe_sweep_test.go` |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// These tests enter sweepOnce with the cut fence still installed. Calling |
| +// reviewRecoveryHold directly would miss the fenced-host early continue. |
| +func sweptProbe(t *testing.T) (*Service, *ClaimRuntime, *actEntry, *fakeProc, *fakeRep, hostclaim.Observed) { |
| + t.Helper() |
| + obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box")) |
| + admitted := probeAdmitted(obs) |
| + a, r, _ := probeService(t, obs) |
| + a.fences = map[string]uint64{probeHost: 9} |
| + e, proc, rep, err := heldProbe(t, a, 9, admitted, probeInputs()) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + e.probe.RecoveryFence = 9 |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil } |
| + return a, r, e, proc, rep, admitted |
| +} |
| + |
| +func waitProbe(t *testing.T, a *Service, predicate func() bool) { |
| + t.Helper() |
| + deadline := time.Now().Add(time.Second) |
| + for time.Now().Before(deadline) { |
| + a.mu.Lock() |
| + ok := predicate() |
| + a.mu.Unlock() |
| + if ok { |
| + return |
| + } |
| + time.Sleep(time.Millisecond) |
| + } |
| + t.Fatal("probe supervision did not finish") |
| +} |
| + |
| +func TestRecoveryProbeSweepRetiresFailedBehindCutFence(t *testing.T) { |
| + for _, mode := range []string{"exit", "lease", "join-exit", "join-lease"} { |
| + t.Run(mode, func(t *testing.T) { |
| + a, r, e, proc, rep, admitted := sweptProbe(t) |
| + if mode == "exit" || mode == "join-exit" { |
| + proc.exit() |
| + } else { |
| + rep.setErr(errors.New("lease lost")) |
| + } |
| + if mode == "join-exit" || mode == "join-lease" { |
| + if err := a.joinRecoveryProbe(t.Context(), e, *e.probe); !errors.Is(err, hostbox.ErrChanged) { |
| + t.Fatalf("join: %v", err) |
| + } |
| + } else { |
| + a.sweepOnce() |
| + } |
| + waitProbe(t, a, func() bool { return a.entries[probeHost] == nil }) |
| + if !proc.isStopped() || !rep.isClosed() || a.FenceGeneration(probeHost) != 9 { |
| + t.Fatal("failed probe was not retired with its cut fence retained") |
| + } |
| + // A retry reaches the admission/start path again, rather than joining |
| + // the dead entry forever. The fixture launcher deliberately refuses. |
| + next := probeAdmitted(admitted) |
| + next.Claim.Holder.Generation = "three" |
| + reads := 0 |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { |
| + reads++ |
| + if reads == 1 { |
| + return admitted, nil |
| + } |
| + return next, nil |
| + } |
| + r.Admit = func(context.Context, registry.Route, hostclaim.Observed, StartPolicy) (hostclaim.Observed, error) { |
| + return next, nil |
| + } |
| + if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, *e.probe); !errors.Is(err, ErrClaimIsolationUnavailable) { |
| + t.Fatalf("retry did not reach the launcher: %v", err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestRecoveryProbeSweepOpensOnlyExactSessionAndFence(t *testing.T) { |
| + for _, phase := range []string{hostclaim.PhaseProbing, hostclaim.PhaseProbed, hostclaim.PhaseOpened, "serving", "new-session", "foreign-executor", "unreadable"} { |
| + t.Run(phase, func(t *testing.T) { |
| + a, r, e, proc, _, admitted := sweptProbe(t) |
| + fresh := admitted |
| + c, h := *admitted.Claim, *admitted.Claim.Holder |
| + c.Holder = &h |
| + fresh.Claim = &c |
| + c.Op.Phase = phase |
| + switch phase { |
| + case "serving": |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"} |
| + case "new-session": |
| + c.Op.Phase = hostclaim.PhaseOpened |
| + h.Session++ |
| + case "foreign-executor": |
| + c.Op.Phase = hostclaim.PhaseOpened |
| + c.Op.Exec = "other" |
| + } |
| + read := make(chan struct{}, 1) |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { |
| + read <- struct{}{} |
| + if phase == "unreadable" { |
| + return hostclaim.Observed{}, errors.New("unreadable") |
| + } |
| + return fresh, nil |
| + } |
| + a.sweepOnce() |
| + select { |
| + case <-read: |
| + case <-time.After(time.Second): |
| + t.Fatal("sweep skipped held probe behind its cut fence") |
| + } |
| + waitProbe(t, a, func() bool { return !e.probeReviewing }) |
| + opens := phase == hostclaim.PhaseOpened || phase == "serving" |
| + stops := phase == "new-session" || phase == "foreign-executor" |
| + if stops { |
| + waitProbe(t, a, func() bool { return a.entries[probeHost] == nil }) |
| + } |
| + if opens { |
| + if a.Fenced(probeHost) || proc.isStopped() { |
| + t.Fatal("exact opening did not release the cut fence and retain its process") |
| + } |
| + release, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}) |
| + if err != nil { |
| + t.Fatalf("opened session cannot serve: %v", err) |
| + } |
| + release() |
| + } else { |
| + if a.FenceGeneration(probeHost) != 9 { |
| + t.Fatal("non-opening changed fence") |
| + } |
| + if _, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) { |
| + t.Fatalf("request: %v", err) |
| + } |
| + if !stops && proc.isStopped() { |
| + t.Fatal("a held or unreadable session was stopped") |
| + } |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestRecoveryProbeSweepPreservesReplacementFence(t *testing.T) { |
| + for _, afterRead := range []bool{false, true} { |
| + t.Run(map[bool]string{false: "before-read", true: "during-read"}[afterRead], func(t *testing.T) { |
| + a, r, e, proc, rep, admitted := sweptProbe(t) |
| + read, resume := make(chan struct{}), make(chan struct{}) |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { |
| + close(read) |
| + <-resume |
| + c := *admitted.Claim |
| + c.Op.Phase = hostclaim.PhaseOpened |
| + admitted.Claim = &c |
| + return admitted, nil |
| + } |
| + replace := func() { a.mu.Lock(); a.fences[probeHost] = 12; a.mu.Unlock() } |
| + if !afterRead { |
| + replace() |
| + } |
| + a.sweepOnce() |
| + if afterRead { |
| + select { |
| + case <-read: |
| + case <-time.After(time.Second): |
| + close(resume) |
| + t.Fatal("sweep skipped cut fence") |
| + } |
| + replace() |
| + } |
| + close(resume) |
| + waitProbe(t, a, func() bool { return a.entries[probeHost] == nil && !e.probeReviewing }) |
| + if a.FenceGeneration(probeHost) != 12 || !proc.isStopped() || !rep.isClosed() { |
| + t.Fatal("replacement fence or retirement lost") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestRecoveryProbeRefusesAnotherOperationsFence(t *testing.T) { |
| + obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box")) |
| + a, _, run := probeService(t, obs) |
| + a.fences = map[string]uint64{probeHost: 12} |
| + p := probePolicy(probeInputs()) |
| + p.RecoveryFence = 9 |
| + if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, p); !errors.Is(err, ErrFenced) { |
| + t.Fatalf("foreign fence: %v", err) |
| + } |
| + if len(run.seen()) != 0 || a.Owns(probeHost) || a.FenceGeneration(probeHost) != 12 { |
| + t.Fatal("foreign fence allowed recovery admission") |
| + } |
| +} |
| + |
| +// The moves worker may finish the rollback and lift its own fence before |
| +// the slower activator sweep observes opened/serving. The exact session |
| +// still owns its process; a missing fence alone is not an opening proof. |
| +func TestRecoveryProbeSweepAcceptsOwnerLiftOnlyWhenOpened(t *testing.T) { |
| + for _, phase := range []string{hostclaim.PhaseOpened, "serving", hostclaim.PhaseProbing} { |
| + for _, duringRead := range []bool{false, true} { |
| + t.Run(phase+map[bool]string{false: "/before", true: "/during"}[duringRead], func(t *testing.T) { |
| + a, r, e, proc, _, admitted := sweptProbe(t) |
| + read, resume := make(chan struct{}), make(chan struct{}) |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { |
| + close(read) |
| + <-resume |
| + c := *admitted.Claim |
| + c.Op.Phase = phase |
| + if phase == "serving" { |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"} |
| + } |
| + admitted.Claim = &c |
| + return admitted, nil |
| + } |
| + if !duringRead && !a.LiftFence(probeHost, 9) { |
| + t.Fatal("owner lift refused") |
| + } |
| + a.sweepOnce() |
| + select { |
| + case <-read: |
| + case <-time.After(time.Second): |
| + close(resume) |
| + t.Fatal("owner lift retired probe before reading claim") |
| + } |
| + if duringRead && !a.LiftFence(probeHost, 9) { |
| + close(resume) |
| + t.Fatal("owner lift refused") |
| + } |
| + close(resume) |
| + waitProbe(t, a, func() bool { return !e.probeReviewing }) |
| + if phase == hostclaim.PhaseProbing { |
| + a.mu.Lock() |
| + held := a.entries[probeHost] == e && e.state == actAwake && e.probeHeld |
| + a.mu.Unlock() |
| + if !held || proc.isStopped() { |
| + t.Fatal("an owner lift with a probing read retired the session") |
| + } |
| + if _, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) { |
| + t.Fatalf("a missing fence opened a still-held probe: %v", err) |
| + } |
| + // The earlier read may precede opening. A later sweep |
| + // must see the same session opened before it can serve. |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { |
| + c := *admitted.Claim |
| + c.Op.Phase = hostclaim.PhaseOpened |
| + return hostclaim.Observed{Host: admitted.Host, Claim: &c, ETag: admitted.ETag}, nil |
| + } |
| + a.sweepOnce() |
| + waitProbe(t, a, func() bool { return !e.probeHeld }) |
| + } |
| + if proc.isStopped() || a.Fenced(probeHost) { |
| + t.Fatal("opened session lost its process") |
| + } |
| + release, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + release() |
| + }) |
| + } |
| + } |
| +} |
| + |
| +func TestRecoveryProbeHealthyFinishAfterOwnerLift(t *testing.T) { |
| + obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box")) |
| + a, r, _ := probeService(t, obs) |
| + admitted := probeAdmitted(obs) |
| + e, proc, _, err := heldProbe(t, a, 9, admitted, probeInputs()) |
| + if err != nil { |
| + t.Fatalf("owner lift prematurely retired healthy finish: %v", err) |
| + } |
| + c := *admitted.Claim |
| + c.Op.Phase = hostclaim.PhaseOpened |
| + admitted.Claim = &c |
| + r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil } |
| + a.sweepOnce() |
| + waitProbe(t, a, func() bool { return !e.probeHeld }) |
| + if proc.isStopped() { |
| + t.Fatal("opened process stopped") |
| + } |
| +} |
| +``` |
| + |
| ### Task B: let hostresolve dispatch declaration policy to its registered owner |
| Invariant: operation policy cannot bypass the signed challenge, exact claim/record observation, global quarantine or durable loss sequence. An unregistered move kind acquires no declaration authority. Accepted evidence and write-loss sources never change on retry. |
| @@ -2671,6 +3044,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| # After recording the expected red: |
| python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" B implementation |
| flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostresolve -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10|TestLostWiring|TestFailedClaim|TestDeclareCrash|TestPreparedLoss' |
| +flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./... |
| ``` |
| #### New file: `internal/hostresolve/operation_owner_test.go` |
| @@ -3136,6 +3510,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| # After recording the expected red: |
| python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" C implementation |
| flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/console -run 'TestJ17|TestPlan9B8|TestMoveDoorAdmitsTheMoveUnderTheHostClaim|TestResolveRetryRefusedUnderHostClaim|TestMoveDoorPhase1RefusedUnderHostClaim' |
| +flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./... |
| ``` |
| #### Patch: `internal/console/apiinstancemove_claim_test.go` |
| @@ -4489,6 +4864,8 @@ The concrete owner registers `moving`, `landing`, `reverting` and `restoring` fo |
| The move door first returns the shared signed inspection, then accepts its unchanged challenge and explicit confirmation with compute-fence evidence. It keeps the existing operator gate. An ambiguous retry/declaration request refuses. The declaration owner checks the authoritative record's commit state and exact target, not just the claim's possibly lagging phase. The data-authority path refuses the source and unrelated participants. Declared target loss cannot be described as certified `clean-return` coverage. The next admission must preserve the loss record's target sessions and any uncovered source-cut session. |
| +Claim §6.4a permits the adopted writer's paired legacy cut identity, session `0` and generation `""` (the `@root` generation). Parse the cut journal with an existence check. For an uncovered source, require the source-generation key to exist and require `(cut == 0) == (generation == "")`; an absent key is not a legacy value. `LossWriteSources` uses the same paired check and still requires a box and region. Before the target's first admission, only an uncovered source cut is frozen. After admission, freeze the positive target session once, plus the legacy source cut if uncovered. A source-covered cut contributes no source write loss. The full inspection/declaration tests exercise all four cases and retain refusal-before-acceptance checks for missing, malformed and mismatched evidence. |
| + |
| Use the existing `MoveRollbackTargetLost` and `MoveRollbackLossCommitted` transitions for the record decision and completion. A recorded `lost` phase is a hold, not permission for the source to restart. Only committed claim loss permits the release. Preserve target-region retirement and any partial return-lane retirement; a fenced box still owes local cleanup unless the operator separately asserted the exact destroyed storage. `MoveRolledBackClose` continues to require `RecoveredAt`, `OpenedAt` and `TargetParkedAt`. |
| `ReleaseLostTarget` reads the journalled target stamp after `RollBack` and supplies the target box to `QueueLocationPark` only when `InventoryHasStamp` finds that installation. The local chore keeps its original operation and retirement identity across retries. Recovery may proceed on the source while it remains pending, but `ReleaseMove` cannot close over it. A normal retirement acknowledgement or a new, separately verified D5 declaration may complete it. The first compute-loss declaration never asserts destruction of a chore it has not yet queued. |
| @@ -4504,6 +4881,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| # After recording the expected red: |
| python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" D implementation |
| flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/hostresolve ./internal/console ./internal/hostclaim ./internal/provision ./internal/hostfence -run 'TestJ4|TestPlan10|TestJ17|TestReleaseLostTarget|TestMoveLoss|TestMoveRollbackLossRebind' |
| +flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./... |
| ``` |
| #### New file: `internal/console/apiinstancemove_j4_test.go` |
| @@ -7578,15 +7956,24 @@ func claimLossIntent(c *hostclaim.Claim, box string) (moveLossIntent, error) { |
| if err != nil { |
| return moveLossIntent{}, hostresolve.ErrResolve |
| } |
| - cut, err := strconv.ParseUint(op.Journal[hostclaim.JournalCutSession], 10, 64) |
| - if err != nil || cut == 0 { |
| + // The cut is a paired identity: a real session with its generation, or |
| + // the legacy {session 0, generation ""} of a source that never had one |
| + // (claim spec §6.4a). A missing key or a mismatched pair is refused. |
| + rawCut, ok := op.Journal[hostclaim.JournalCutSession] |
| + if !ok { |
| + return moveLossIntent{}, hostresolve.ErrResolve |
| + } |
| + cut, err := strconv.ParseUint(rawCut, 10, 64) |
| + if err != nil { |
| return moveLossIntent{}, hostresolve.ErrResolve |
| } |
| in.SrcCovered, in.CutSession = seal.SrcCovered, cut |
| if !seal.SrcCovered { |
| - if in.SrcGen = op.Journal[hostclaim.JournalSrcGen]; in.SrcGen == "" { |
| + gen, ok := op.Journal[hostclaim.JournalSrcGen] |
| + if !ok || (cut == 0) != (gen == "") { |
| return moveLossIntent{}, hostresolve.ErrResolve |
| } |
| + in.SrcGen = gen |
| } |
| return in, nil |
| } |
| @@ -7895,7 +8282,7 @@ func (m *moveLossOwner) LossWriteSources(_ context.Context, o hostclaim.Observed |
| var out []hostclaim.WriteSource |
| seen := map[string]bool{} |
| add := func(s hostclaim.WriteSource) error { |
| - if s.Generation == "" || s.Session == 0 || s.Region == "" { |
| + if s.Box == "" || (s.Session == 0) != (s.Generation == "") || s.Region == "" { |
| return hostresolve.ErrResolve |
| } |
| key := s.Box + "\x00" + strconv.FormatUint(s.Session, 10) |
| @@ -8686,6 +9073,136 @@ func TestJ4DestroyedLocalOnlyRejectsWidenedEffects(t *testing.T) { |
| } |
| ``` |
| +#### New file: `internal/hostops/move_loss_legacy_test.go` |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "reflect" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/moves" |
| + "github.com/carlosframework/platform/internal/provision" |
| +) |
| + |
| +// A source that never had a session (claim spec §6.4a) is the paired legacy |
| +// identity {session 0, generation ""}. The move's cut then names session 0 |
| +// and an empty source generation; a mismatched pair or a missing journal |
| +// key is still refused. |
| + |
| +// legacySource makes the holder the legacy source before the target's first |
| +// admission, with the move's cut at the legacy identity. |
| +func legacySource(covered bool, afterAdmission bool) func(*hostclaim.Claim, *provision.Record, *AppAdmission) { |
| + return func(c *hostclaim.Claim, _ *provision.Record, _ *AppAdmission) { |
| + h := c.Holder |
| + h.WriterBox, h.Session, h.Generation = "ie-box", 0, "" |
| + h.Admissions = nil |
| + if afterAdmission { |
| + h.WriterBox, h.Session, h.Generation = "uk-box", 1, "mv" |
| + h.Admissions = []hostclaim.AdmissionRecord{{Session: 1, Box: "uk-box", Generation: "mv", Predecessor: "", Region: "uk", PredecessorRegion: "ie", Mode: "cold", Coverage: "sealed"}} |
| + } |
| + c.Op.Journal[hostclaim.JournalCutSession] = "0" |
| + c.Op.Journal[hostclaim.JournalSrcGen] = "" |
| + if covered { |
| + seal, err := moves.DecodeSeal(c.Op.Journal[hostclaim.JournalSeal]) |
| + if err != nil { |
| + panic(err) |
| + } |
| + seal.SrcCovered = true |
| + c.Op.Journal[hostclaim.JournalSeal] = seal.Encode() |
| + } |
| + } |
| +} |
| + |
| +func declareLegacy(t *testing.T, w *lossWorld) []hostclaim.WriteSource { |
| + t.Helper() |
| + in := w.inspect(t, "uk-label") |
| + if _, err := w.declare(w.submission(in)); err != nil { |
| + t.Fatal(err) |
| + } |
| + loss, err := hostclaim.ReadLossRecord(context.Background(), w.raw, w.pub, in.Details.ID) |
| + if err != nil || loss.Record.State != "committed" { |
| + t.Fatalf("loss record = %+v %v", loss.Record, err) |
| + } |
| + return loss.Record.WriteSources |
| +} |
| + |
| +func TestJ4LegacySourceCutDeclares(t *testing.T) { |
| + sourceCut := hostclaim.WriteSource{Reason: "source-cut", Box: "ie-box", Generation: "", Session: 0, Region: "ie", Baseline: "seal:" + lossOp} |
| + cases := map[string]struct { |
| + covered, admitted bool |
| + want []hostclaim.WriteSource |
| + }{ |
| + "before first admission": {false, false, []hostclaim.WriteSource{sourceCut}}, |
| + "after first admission": {false, true, []hostclaim.WriteSource{ |
| + {Reason: "target", Box: "uk-box", Generation: "mv", Session: 1, Region: "uk", Baseline: "replica:uk/mv"}, sourceCut}}, |
| + "covered after first admission": {true, true, []hostclaim.WriteSource{ |
| + {Reason: "target", Box: "uk-box", Generation: "mv", Session: 1, Region: "uk", Baseline: "replica:uk/mv"}}}, |
| + } |
| + for name, tc := range cases { |
| + t.Run(name, func(t *testing.T) { |
| + w := newLossWorld(t, "landing", legacySource(tc.covered, tc.admitted)) |
| + if got := declareLegacy(t, w); !reflect.DeepEqual(got, tc.want) { |
| + t.Fatalf("write sources = %+v, want %+v", got, tc.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestJ4LegacyCoveredBeforeAdmissionDeclares(t *testing.T) { |
| + w := newLossWorld(t, "landing", legacySource(true, false)) |
| + if got := declareLegacy(t, w); len(got) != 0 { |
| + t.Fatalf("write sources = %+v", got) |
| + } |
| +} |
| + |
| +func TestJ4LegacyJournalRefusedBeforeAcceptance(t *testing.T) { |
| + set := func(key, v string) func(*hostclaim.Claim, *provision.Record, *AppAdmission) { |
| + return func(c *hostclaim.Claim, _ *provision.Record, _ *AppAdmission) { c.Op.Journal[key] = v } |
| + } |
| + del := func(key string) func(*hostclaim.Claim, *provision.Record, *AppAdmission) { |
| + return func(c *hostclaim.Claim, _ *provision.Record, _ *AppAdmission) { delete(c.Op.Journal, key) } |
| + } |
| + both := func(fs ...func(*hostclaim.Claim, *provision.Record, *AppAdmission)) func(*hostclaim.Claim, *provision.Record, *AppAdmission) { |
| + return func(c *hostclaim.Claim, r *provision.Record, a *AppAdmission) { |
| + for _, f := range fs { |
| + f(c, r, a) |
| + } |
| + } |
| + } |
| + legacy := legacySource(false, false) |
| + cases := map[string]func(*hostclaim.Claim, *provision.Record, *AppAdmission){ |
| + "missing cut session": both(legacy, del(hostclaim.JournalCutSession)), |
| + "malformed cut session": both(legacy, set(hostclaim.JournalCutSession, "x")), |
| + "negative cut session": both(legacy, set(hostclaim.JournalCutSession, "-1")), |
| + "empty cut session": both(legacy, set(hostclaim.JournalCutSession, "")), |
| + "missing source generation": both(legacy, del(hostclaim.JournalSrcGen)), |
| + "session 0 with generation": both(legacy, set(hostclaim.JournalSrcGen, "src")), |
| + "session with empty generation": both(legacy, set(hostclaim.JournalCutSession, "15")), |
| + "covered missing cut session": both(legacySource(true, false), del(hostclaim.JournalCutSession)), |
| + "covered malformed cut": both(legacySource(true, false), set(hostclaim.JournalCutSession, "x")), |
| + } |
| + for name, mutate := range cases { |
| + t.Run(name, func(t *testing.T) { |
| + w := newLossWorld(t, "landing", mutate) |
| + snap := lossSnapshot(t, w) |
| + if _, err := w.control.InspectMoveLoss(context.Background(), w.signer, lossBucket, "operator", "bab", "app", lossHost, "uk-box"); err == nil { |
| + t.Fatal("inspection accepted") |
| + } |
| + if lossSnapshot(t, w) != snap { |
| + t.Fatal("a refused inspection wrote") |
| + } |
| + if len(w.acceptances(t)) != 0 || w.roster(t).Roster.Boxes["uk-box"].State == "quarantined" { |
| + t.Fatal("accepted or quarantined") |
| + } |
| + }) |
| + } |
| +} |
| +``` |
| + |
| ### Task E: reconcile the reviewed writer inventories |
| Invariant: every production declaration that can name a record write remains explicitly inventoried, and every record writer still passes through the sequenced provision primitive. Do not weaken or regenerate the census policy. Run its existing tests first: the combined base and Tasks C/D introduce unlisted sites. After reviewing the sources and their callers, apply only the inventory patch below and rerun the same tests. The provision inventory is a maintenance patch to an existing test, so the extractor applies that file in the implementation stage. The console source census first adds the three new typed method names to its selection, then refuses their unlisted call sites. Its reviewed fixture is applied in the implementation stage. |
| @@ -8700,6 +9217,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| # After recording the expected red: |
| python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" E implementation |
| flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/provision ./internal/console -run 'TestPlan9B8RecordWriteCensus|TestPlan9B8CensusRejectsNewSites|TestClaimAuthoritySourceCensus' |
| +flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./... |
| ``` |
| #### Patch: `internal/console/claim_authority_census_test.go` |
| @@ -8744,7 +9262,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou |
| + {"internal/hostops/move_loss_grant.go", "claimSimulation.Put", "nonrecord", "def:Put", "08f341b2c797aa30cda0e4e56249f2479f1ad032ee086442ba543f84e601c118"}, |
| + {"internal/hostops/move_loss_grant.go", "lossScopedStore.Put", "record-transport", "def:Put,key:IsRecordKey,key:LossRecordKey,key:putSequencedRecord,sel:Put", "88d231d1f91bf0ff0bd4ea5992e5099b0cfcd942fbd1b79cf81e828133fbd64f"}, |
| + {"internal/hostops/move_loss_grant.go", "readOnlyStore.Put", "nonrecord", "def:Put", "e5f2d05226614d6a23afe06dea4fda7d45958aeac6ee5f42474872b2cbbc8b8e"}, |
| -+ {"internal/hostops/move_loss_owner.go", "claimLossIntent", "key-site", "key:RecordKey", "1b72b043ec7c1b20b95af9c312bae449cf19d29f5759a906fd14af6f845cb938"}, |
| ++ {"internal/hostops/move_loss_owner.go", "claimLossIntent", "key-site", "key:RecordKey", "c3eeb176db52e6642bd026bbf97b2b5f6520d08e7fccbdc8c8a9106851725802"}, |
| + {"internal/hostops/move_loss_owner.go", "decodeMoveLossIntent", "key-site", "key:RecordKey", "e868448e696de4614a362f27301e6fa25506394fd7582d3076d126a5f72e93db"}, |
| + {"internal/hostops/move_loss_owner.go", "lossIntentOf", "key-site", "key:LossRecordKey", "bd0d6d6d2c53ed4daddae47791f4de5fda5935bd59be0a04e41ad22a0b3a2954"}, |
| + {"internal/hostops/move_loss_owner.go", "moveLossOwner.FinishLoss", "record-writer", "key:RecordKey,key:WriteRecord", "fb936dc901ec18ac2aeb1cd638eed0600df0b5dfe3bf2caf6efa6b4f000af1d2"}, |
| @@ -8906,6 +9424,14 @@ The displayed transition token `move-target-lost` is new. It identifies the prop |
| ## Verification and self-review |
| +Rev 3 was re-extracted into a fresh export of the pinned inputs and replayed in task order. Every task A–E produced its expected red, then passed its listed tests and `go build -p 1 ./...` before the next task. The final eight-package integration command, the complete `hostclaim`, `hostresolve` and `hostfence` packages, and the activator recovery/hold tests under `-race` passed. Whole-tree `gofmt -l .` was empty. All extracted source and test targets match the validated overlay byte-for-byte. The default Go cache was read-only; validation used the permitted shared fallback without deleting it. |
| + |
| +Rev 3 reproduces both round 2 failures before the fixes. Against rev 2, `TestRecoveryProbeSweepRetiresFailedBehindCutFence` leaves exited and lease-failed entries resident on sweep and join; `TestRecoveryProbeSweepOpensOnlyExactSessionAndFence` never reaches its claim read with the cut fence present. `TestRecoveryProbeRefusesAnotherOperationsFence` reaches admission under the foreign fence. This behavioral red run adds only the new `StartPolicy.RecoveryFence` field to compile the new fixtures; it leaves rev 2's wake, join and sweep behavior intact. The revised tests retain the actual fence, exercise `sweepOnce`, prove retirement permits another admission attempt, and prove exact opening serves without waiting for move closure. |
| + |
| +The four legacy declaration cases fail against the rev 2 owner: before and after the first target admission, with the source cut uncovered or covered. The revised owner accepts the paired legacy values; the tests read back the committed signed loss record and check the exact write sources. Missing cut or required source-generation keys, malformed cuts and mismatched uncovered identities still refuse without acceptance or quarantine. The J4 interface and its implementation declaration are byte-identical to rev 2. |
| + |
| +Opus reviewed both fixes. The recovery review also identified the moves worker lifting its own fence before the sweep observes opening. Failing-first tests reproduce that race, including a stale `probing` answer during the lift. A healthy exact session stays held, with requests refused, until a later claim read confirms opening. Nonzero foreign or replacement fences still stop it. The final read-only reviews report no remaining blocker. |
| + |
| Rev 2 was checked in an external export of the two pinned commits. No git state or implementation file in the authoring repository was written. The rev 1 authority regressions failed first: all nine HTTP races returned 202, the inventory-backed lost target had no local chore, and committed-source local inspection was refused. The revised embedded code passes those cases, plus exact-attestation and stale-evidence refusals, already-declared target discharge, unchanged writer state and unrelated obligations, discovery completion and local-retirement release blocking. |
| Task A adds adapter and production-entry tests. A separate Opus review found the changed-input replay that could strand an older held probe; its regression failed first and passes with cancellation/retirement and late-success refusal. Caller input maps are copied, and the mutation test failed before that copy was added. Clean and dirty returns record recovery inputs before rendering and launching; a failed record write prevents both. The Group D tests retain operation/session bindings, attempt accounting and role-cgroup readiness. The production recovery hook is connected only with the claim flag; missing dependencies refuse instead of reporting recovery. Failing-first replay installs every task's tests before its code. Initial missing-symbol failures establish new API dependencies; the rev 1 HTTP and discharge regressions above establish behavioral failures. |
| @@ -8945,7 +9471,7 @@ for name, parent, prefix in ( |
| raise SystemExit("unexpected cleanup path: " + str(path)) |
| shutil.rmtree(path) |
| CLEANUP |
| -git status --short |
| +GIT_OPTIONAL_LOCKS=0 git status --short |
| ``` |
| The authoring worktree must show only `docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md`. Do not update a handoff file, format another repository file or write git state as part of this plan-only task. |