carlos / platform Public

plan: claim Plan 10 rev 2 (review r1: move-bound console grant, lost-target park-local, §7.3 D5 local-only discharge; clean-return recovery inputs; Group D recovery wiring)

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex agent, operated by paul@keymail.dev pushed by claude agent, operated by paul@keymail.dev 47cc648f1b725350b0c755326c108b6cfba3f419 parent c4881b8
1 file changed, +2712 −217
  • docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md +2712 −217

CI passed — run details

diff --git a/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md b/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md
index ca530f818..f9a0f8d4c 100644
--- a/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md
+++ b/docs/superpowers/plans/2026-10-09-host-lifecycle-claim-10-moves-recovery-grants.md
@@ -1,9 +1,13 @@
# Host lifecycle claim, Plan 10: moves recovery and console grants
-Implementation plan, 2026-10-09. This plan extends the claim's recovery-probe contract, gives move loss declarations a registered operation owner, and admits console move writes through typed authority. Scope: J2, J4 and J17, as ruled with Paul on 2026-10-08. It authorizes no production change.
+Implementation plan, 2026-10-09, rev 2. This plan extends the claim's recovery-probe contract, gives move loss declarations a registered operation owner, and admits console move writes through typed authority. Scope: J2, J4 and J17, as ruled with Paul on 2026-10-08. It authorizes no production change.
+
+**Rev 2:** answers review round 1's three P2 findings and the moves Plan 4 addendum. Console grants retain the gate-read move ID, incarnation and `record_seq`; lost-target release retains inventory-bound local retirement; D5 discharges only separately attested local chores, including after commit. Task A records clean-return inputs at `probing` and connects the recovery hook to Group D through the activator. The tests below reproduce the rev 1 failures before applying these changes.
## J4 owner interface
+**The J4 `OperationOwner` signature is unchanged from rev 1.** Moves Plan 4 callers may keep building against it. Local-only declarations use the same inspection, signed consequences and owner hooks; no extra caller-supplied authority or interface method is introduced.
+
Publish this contract to the moves session before implementing the callers. It belongs to `internal/hostresolve`; imports are `context`, `internal/hostclaim` and `internal/hostfence`.
```go
@@ -18,7 +22,7 @@ type OperationOwner interface {
`Service.Owners` is a `map[string]OperationOwner`, keyed by the exact claim kind. Built-in serving, parking and deleting behavior remains owned by hostresolve. Moving, landing, reverting and restoring have no fallback owner. Register only the kinds the move implementation supports. A missing owner refuses inspection and declaration before new acceptance or quarantine; an accepted retry also requires its original owner.
-The service calls `InspectLoss` on a detached claim and the authoritative record observation. It signs the displayed consequences with the claim and record identities. At declaration it repeats the read-only decision against that signed observation. `LossWriteSources` freezes the target sessions and any uncovered source-cut session into the immutable, signed loss record before acceptance. It never invents a remote maximum; later validated recovery evidence supplies each maximum.
+The service calls `InspectLoss` on a detached claim and the authoritative record observation. It signs the displayed consequences with the claim and record identities. At declaration it repeats the read-only decision against that signed observation. `LossWriteSources` freezes the target sessions and any uncovered source-cut session into the immutable, signed loss record before acceptance. It never invents a remote maximum; later validated recovery evidence supplies each maximum. A local-only declaration freezes no write sources; its hooks validate the durable evidence without writing the move record.
The write order is signed acceptance, global roster quarantine, prepared loss record, `PrepareLoss`, claim declaration, committed loss record, `FinishLoss`. Moves records its own target-loss decision in `PrepareLoss` and releases only the declared authority in `FinishLoss`. Both hooks are idempotent. A retained claim loss reference skips preparation on replay. Completion runs again even after the loss record is committed, so a lost answer cannot strand the move. The service dispatches retries by the accepted loss's `OpKind`, not the current claim kind. Callback arguments and returned consequences are copied; callbacks cannot mutate the service's evidence through shared maps, slices or pointers.
@@ -161,16 +165,18 @@ Apply the tests first. The bootstrap has already repaired the combined export's
Reuse `OperationStart`, `OperationAttempts`, `ClaimProbeReadiness` and `ClaimOwnerDispatch`. No interface reconstruction is needed: dispatch already routes both recovery kinds to the move owners. Every operation-attempt binding and admitted role path must use the expanded kind predicate. A recovery probe uses the journalled role's cgroup for readiness and the bound readiness callback for the settle check. A serving wake and the committed landing wake keep their existing readiness behavior. Missing owners continue to refuse.
-The restore and install remain the moves owner's work. This task grants no generic cross-box recovery handover and no new artifact-retirement authority. Plan 4 Task 9 must supply the actual recovery adapter with the correct stop, admission and restoration evidence. The production `recoveryProbe` hook remains unarmed until those dependencies exist. Do not wire a fake success callback to make a pass advance.
+Task A supplies `recoveryProbeAdapter` and the production binding `armRecoveryProbe(cfg, w, svc.RecoveryProbe, svc.RecoveryProbeReady)`, reached only with both moves and host claims enabled. `activator.Service.RecoveryProbe(context.Context, registry.Route, activator.StartPolicy) error` runs the admitted Group D start path and retains the healthy process for the later opening step. The hook keeps the existing `recoveryProbeRequest` and outcome signature, so moves Plan 4 Task 9 can consume it directly. Missing claim runtime, operation owners or restoration evidence still refuses before execution. With host claims off, `armPhase1` keeps the legacy nil recovery hook.
+
+The adapter passes the recorded inputs in `StartPolicy.Inputs`, sets `Probe`, and uses a 30-second readiness window with at most three starts in one wake. The moves `OperationAttempts` provider owns the durable nine-start recovery budget across wakes and the original first-start time; its exhausted result maps to `recoveryProbeExhausted`. A failed wake window or changed authority leaves `probing` for another pass. The readiness predicate checks runtime wiring before input recording. An in-flight or held probe with superseded inputs is cancelled or retired, including a late healthy result after cancellation; it cannot strand the newly recorded inputs behind the old process. Edge reloads preserve held operation-owned probes, while explicit fences and claim/session checks still stop them. The restore/install, park leases, recovery admission evidence and source/target cleanup remain the moves owner's work. This task adds no generic recovery takeover or artifact-retirement authority.
A clean return records inputs **at `Rollback.SourcePhase == probing`, before rendering or materializing anything**. The adapter calls `recoveryProbeRequest.Record`, which uses `provision.MoveRecordRecoveryInputs` under the move and rollback IDs. That transition is already on the pinned moves base and accepts `probing` for both clean and dirty returns. Its `Release` must be nonempty. A failed record write stops the attempt before render. The clean return's path has no `configured` phase and needs none. Each restart records the newly selected inputs before its own render. A revert analogously uses the already-built `MoveRecordRevertInputs`. None of these values is compared to the original `Move.Inputs` pin.
```bash
python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" A tests
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostclaim ./internal/host -run 'TestPlan10J2|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe'
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostclaim ./internal/host ./internal/activator -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10J2|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe'
# After recording the expected red:
python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" A implementation
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostclaim ./internal/host -run 'TestPlan10J2|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe'
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostclaim ./internal/host ./internal/activator -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10J2|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe'
```
#### New file: `internal/host/plan10_j2_test.go`
@@ -530,7 +536,7 @@ func TestPlan10J2CrossBoxCodeStaysLandingOnly(t *testing.T) {
// A clean return records its current inputs at probing before the held
// render and launch. No configured phase is needed. The adapter must call
-// recoveryProbeRequest.Record first; Task 9 supplies that production adapter.
+// recoveryProbeRequest.Record first; movesrecoveryprobe.go supplies the adapter.
// This test exercises the existing recording transition and the policy checks
// for the recorded configuration, release and launch.
func TestPlan10J2CleanReturnRecordsRecoveryInputsBeforeRenderAndLaunch(t *testing.T) {
@@ -920,155 +926,1884 @@ func TestPlan10J2KeepWriterSessionGuardsARecoveryStep(t *testing.T) {
// SessionContinues reports whether a writer session last checked while the op
```
-### Task B: let hostresolve dispatch declaration policy to its registered owner
+#### Patch: `internal/activator/activator.go`
-Invariant: operation policy cannot bypass the signed challenge, exact claim/record observation, global quarantine or durable loss sequence. An unregistered move kind acquires no declaration authority. Accepted evidence and write-loss sources never change on retry.
+```diff
+--- a/internal/activator/activator.go
++++ b/internal/activator/activator.go
+@@ -71,0 +72 @@
++ "github.com/carlosframework/platform/internal/hostclaim"
+@@ -347,0 +349,11 @@
++ // Recovery probe entries (RecoveryProbe, recoveryprobe.go). probe is the
++ // caller's probe policy, set at construction and never written again;
++ // nil for every other entry, whose wake is main's. probeAdmitted is the
++ // admission the probe started under, written by the wake before the
++ // awake transition. probeHeld keeps requests, bounces and the idle clock
++ // off the entry until the claim says the source opened on that session.
++ probe *StartPolicy
++ probeAdmitted hostclaim.Observed
++ probeHeld bool
++ probeReviewing bool
++ probeObsolete bool
+@@ -751,0 +764,5 @@
++ // A held recovery probe serves nothing until the source opens.
++ if e.probeHeld {
++ a.mu.Unlock()
++ return nil, ErrFenced
++ }
+@@ -836,0 +854,4 @@
++ }
++ if e.probeHeld {
++ a.mu.Unlock()
++ return ErrFenced
+@@ -1428,0 +1450,5 @@
++ if e != nil && e.probeHeld {
++ // A held recovery probe runs the moves-held inputs, not delivered
++ // config: the recovery owns its process until the source opens.
++ return false
++ }
+@@ -2507,0 +2534,9 @@
++ continue
++ }
++ // A held recovery probe has no idle clock and no convergence: the
++ // claim decides when it serves or stops (reviewRecoveryHold).
++ if e.probeHeld {
++ if !e.probeReviewing {
++ e.probeReviewing = true
++ go a.reviewRecoveryHold(e)
++ }
+@@ -2835 +2870,3 @@
+- if e.worker {
++ // A held recovery belongs to its claim operation, not the edge's
++ // serving set. Explicit fences and claim/session review still stop it.
++ if e.worker || e.probeHeld {
+```
-Apply the tests before the implementation. The expected initial failure names the missing `OperationOwner` and `Service.Owners`. Tests then force failures before preparation, at the claim write, during committed-loss bookkeeping and during move completion. They retry through the public service and check the retained declaration rather than accepting a successful callback as proof. A separate test changes the current claim kind after success and proves that the original registered owner still receives completion.
+#### Patch: `internal/activator/claimwake.go`
-The interface at the top of this plan is the contract. Preserve the existing owner for serving, parking and deleting; the registration map does not override it. The signed loss record freezes `WriteSources` before the roster is changed. Malformed source tuples refuse before acceptance. Unknown or incomplete timer evidence remains `TimersUnknown`; compute fencing is never treated as disk destruction or a timer inventory.
+```diff
+--- a/internal/activator/claimwake.go
++++ b/internal/activator/claimwake.go
+@@ -105 +105,7 @@
+- if err := hostbox.WakePhase(observed.Claim, r.Box, false); err != nil {
++ // A recovery probe entry carries its caller's probe policy; every other
++ // wake runs main's serving policy.
++ policy := startPolicy{MaxStarts: 5, Window: 30 * time.Second}
++ if e.probe != nil {
++ policy = *e.probe
++ }
++ if err := hostbox.WakePhase(observed.Claim, r.Box, policy.Probe); err != nil {
+@@ -106,0 +113,3 @@
++ }
++ if e.probe != nil && (!policy.Probe || !recoveryProbeKind(observed.Claim.Op.Kind)) {
++ return nil, hostbox.ErrChanged
+@@ -122 +130,0 @@
+- policy := startPolicy{MaxStarts: 5, Window: 30 * time.Second}
+@@ -133,0 +142,3 @@
++ if e.probe != nil {
++ e.probeAdmitted = admitted
++ }
+```
-The service's returned observation identifies its declaration CAS. `FinishLoss` may advance the move further; callers needing that final state read it through the ordinary record API. Neither hook is allowed to invent a new holder or reuse the request as a blanket lifecycle context.
+#### New file: `internal/activator/recoveryprobe.go`
-```bash
-python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" B tests
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostresolve -run 'TestPlan10|TestLostWiring|TestFailedClaim|TestDeclareCrash|TestPreparedLoss'
-# After recording the expected red:
-python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" B implementation
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostresolve -run 'TestPlan10|TestLostWiring|TestFailedClaim|TestDeclareCrash|TestPreparedLoss'
+```go
+package activator
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "maps"
+ "reflect"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// The source's recovery probe entry (claim Plan 10 J2; moves §7.8 source
+// probing). The host's recovery probe adapter calls RecoveryProbe once the
+// record and the claim say probing with this box executing.
+//
+// One call is one claim wake of rt under the caller's probe policy, through
+// the same path as every claim wake: K6 (a foreign executor is stopped and
+// refused before anything else), the queued incarnation, WakePhase with
+// probe set, then ClaimRuntime's Prerequisites, Admit and Prepare and
+// startAdmitted's start loop. The policy, its Probe and its exact Inputs,
+// reaches Prerequisites, Admit and Prepare unchanged: the host's
+// admitsOperation, the moves Inputs owner and the attempt journal decide
+// what is admitted, rendered and counted. Only a reverting or restoring op
+// at probing is probed here; a landing probe is not this entry's.
+//
+// Lifecycle. The probe owns the host's entry exactly as a wake does: one per
+// host. An identical call joins the probe in flight (or the healthy one it
+// left) and shares its outcome; any other entry, or a different policy, is
+// refused with hostbox.ErrChanged. Requests and sidecar requests never join
+// a probe and are refused with ErrFenced, as under a fence.
+//
+// Fences. The rollback holds its cut fence on this host through the whole
+// recovery, and the claim's exact probing executor, not the fence, is the
+// authority this entry starts under: the probe runs with the fence installed
+// and the fence keeps everything else out. A fence installed, replaced or
+// lifted while the probe runs ends it: FenceAndStop cancels it like any
+// wake, and a healthy start that finds the fence changed is stopped.
+//
+// A healthy start (OperationAttempts.Healthy accepted it) is retained: the
+// entry is awake with the started process and its replication, held. While
+// held it serves nothing, has no idle clock or convergence and ignores
+// Bounce; a lost lease or an exit still stands it down. The sweep reads the
+// claim for it (reviewRecoveryHold): once the source opened on the same
+// session (or the op released to serving keeping it) the hold ends and the
+// entry serves as any awake entry; once the claim names another session or
+// executor, the entry is stopped.
+func (a *Service) RecoveryProbe(ctx context.Context, rt registry.Route, p StartPolicy) error {
+ if a == nil || ctx == nil {
+ return hostwrite.ErrUnavailable
+ }
+ // Every dependency of the run is checked before the first read, stop or
+ // write.
+ if !a.RecoveryProbeReady() {
+ return hostwrite.ErrUnavailable
+ }
+ if rt.Host == "" || !recoveryProbePolicy(p) {
+ return errRecoveryProbePolicy
+ }
+ a.mu.Lock()
+ if a.shuttingDown {
+ a.mu.Unlock()
+ return ErrFenced
+ }
+ if e := a.entries[rt.Host]; e != nil {
+ a.mu.Unlock()
+ return a.joinRecoveryProbe(ctx, e, p)
+ }
+ ctx, cancel := context.WithTimeout(ctx, recoveryProbeLimit)
+ defer cancel()
+ policy := p
+ inputs := *p.Inputs
+ inputs.BundleETags = maps.Clone(p.Inputs.BundleETags)
+ policy.Inputs = &inputs
+ e := &actEntry{host: rt.Host, route: rt, state: actWaking, ready: make(chan struct{}), probe: &policy, probeHeld: true, wakeCancel: cancel}
+ if a.entries == nil {
+ a.entries = make(map[string]*actEntry)
+ }
+ a.entries[rt.Host] = e
+ fence := a.fences[rt.Host]
+ a.mu.Unlock()
+ if a.lifeCtx != nil {
+ stop := context.AfterFunc(a.lifeCtx, cancel)
+ defer stop()
+ }
+ w, err := a.doClaimWake(ctx, e)
+ return a.finishRecoveryProbe(e, fence, w, err)
+}
+
+// RecoveryProbeReady checks wiring only. The host calls it before recording
+// inputs; admission still validates the exact claim and session on every run.
+func (a *Service) RecoveryProbeReady() bool {
+ if a == nil || !a.cfg.HostWrites.Enabled {
+ return false
+ }
+ r := a.cfg.Claims
+ return r != nil && r.Box != "" && r.Read != nil && r.StopForeign != nil && r.Prerequisites != nil && r.Admit != nil && r.Prepare != nil && r.FailedAdmission != nil
+}
+
+// recoveryProbeLimit bounds one probe call, as wake bounds a wake.
+const recoveryProbeLimit = 5 * time.Minute
+
+// errRecoveryProbePolicy is a call that is not a recovery probe: no host, an
+// ordinary or prepared policy, no inputs, or starts and window outside the
+// start loop's bounds. A caller bug, refused before anything runs.
+var errRecoveryProbePolicy = fmt.Errorf("%w: not a recovery probe policy", hostbox.ErrUnsupported)
+
+func recoveryProbePolicy(p StartPolicy) bool {
+ return p.Probe && p.Inputs != nil && p.Prepared == nil && p.MaxStarts >= 1 && p.MaxStarts <= 5 && p.Window > 0 && p.Window <= 30*time.Second
+}
+
+// recoveryProbeKind reports whether kind is the source's recovery, the only
+// op this entry probes.
+func recoveryProbeKind(kind string) bool {
+ return kind == hostclaim.KindRestoring || kind == hostclaim.KindReverting
+}
+
+func sameRecoveryProbe(x, y StartPolicy) bool {
+ return x.MaxStarts == y.MaxStarts && x.Window == y.Window && x.Deadline.Equal(y.Deadline) && x.Probe == y.Probe &&
+ x.Prepared == nil && y.Prepared == nil && x.Inputs != nil && y.Inputs != nil && reflect.DeepEqual(*x.Inputs, *y.Inputs)
+}
+
+// joinRecoveryProbe answers a call that found host's entry already present.
+// Only a probe entry under the same policy is joined: a probe in flight
+// shares its outcome, and a retained healthy probe answers once its process,
+// its replication and the claim's session still hold. It changes nothing.
+func (a *Service) joinRecoveryProbe(ctx context.Context, e *actEntry, p StartPolicy) error {
+ for {
+ a.mu.Lock()
+ if a.entries[e.host] != e || e.probe == nil {
+ a.mu.Unlock()
+ return hostbox.ErrChanged
+ }
+ if !sameRecoveryProbe(*e.probe, p) {
+ // A new input selection must not strand an older held probe.
+ // Cancel a pending start, including a late healthy result, or
+ // retire the held process before the next pass can start anew.
+ if e.probeHeld {
+ e.probeObsolete = true
+ if e.state == actWaking && e.wakeCancel != nil {
+ e.wakeCancel()
+ }
+ if e.state == actAwake {
+ a.beginStopLocked(e)
+ }
+ }
+ a.mu.Unlock()
+ return hostbox.ErrChanged
+ }
+ switch e.state {
+ case actWaking:
+ ready := e.ready
+ a.mu.Unlock()
+ select {
+ case <-ready:
+ if e.err != nil {
+ return e.err
+ }
+ continue
+ case <-ctx.Done():
+ return ctx.Err()
+ }
+ case actAwake:
+ admitted, proc, rep := e.probeAdmitted, e.proc, e.rep
+ a.mu.Unlock()
+ if proc == nil || proc.Exited() || rep == nil || rep.Err() != nil {
+ return hostbox.ErrChanged
+ }
+ r := a.cfg.Claims
+ fresh, err := r.Read(ctx, e.host)
+ if err != nil {
+ return err
+ }
+ _, err = recoveryHoldVerdict(admitted, fresh, r.Box)
+ return err
+ default:
+ a.mu.Unlock()
+ return hostbox.ErrChanged
+ }
+ }
+}
+
+// finishRecoveryProbe resolves the probe entry: a failed probe is forgotten
+// as a failed wake is; a healthy one is retained, held, unless the fence it
+// started under changed, when it is stopped and the call answers ErrFenced.
+func (a *Service) finishRecoveryProbe(e *actEntry, fence uint64, w *wokenInstance, err error) error {
+ a.mu.Lock()
+ if err == nil && (w == nil || w.proc == nil || w.rep == nil) {
+ err = hostwrite.ErrUnavailable
+ }
+ if err != nil {
+ e.err = err
+ outcome := &stopObservation{uncertain: true}
+ if w != nil && w.stopOutcome != nil {
+ outcome = w.stopOutcome
+ }
+ a.rememberStopLocked(e.host, outcome)
+ if a.entries[e.host] == e {
+ delete(a.entries, e.host)
+ }
+ a.mu.Unlock()
+ close(e.ready)
+ return err
+ }
+ now := a.now()
+ e.state = actAwake
+ e.rep, e.proc = w.rep, w.proc
+ e.spawnVersion = w.spawnVersion
+ e.lastActive, e.spawnedAt = now, now
+ e.bounceAfterWake = false
+ if e.probeObsolete {
+ e.err = hostbox.ErrChanged
+ a.beginStopLocked(e)
+ } else if a.shuttingDown || a.fences[e.host] != fence {
+ log.Printf("activator: %s recovery probe healthy, but its fence changed; stopping it", e.host)
+ e.err = ErrFenced
+ a.beginStopLocked(e)
+ } else {
+ log.Printf("activator: %s recovery probe healthy; held until the source opens", e.host)
+ }
+ err = e.err
+ a.mu.Unlock()
+ close(e.ready)
+ return err
+}
+
+// recoveryHoldVerdict compares the claim read now with the admission a held
+// probe started under. The same writer (incarnation, stamp, session,
+// generation) on the same op and executor at probing or probed keeps the
+// hold; at opened, or released to serving with this box still the writer,
+// the hold ends (open). Anything else is hostbox.ErrChanged: the probe's
+// session is over.
+func recoveryHoldVerdict(admitted, fresh hostclaim.Observed, box string) (open bool, err error) {
+ x, y := admitted.Claim, fresh.Claim
+ if fresh.Absent || fresh.Host != admitted.Host || box == "" || x == nil || y == nil || x.Holder == nil || y.Holder == nil {
+ return false, hostbox.ErrChanged
+ }
+ for _, lost := range y.Lost {
+ if lost == box {
+ return false, hostbox.ErrChanged
+ }
+ }
+ h, g := x.Holder, y.Holder
+ if h.Incarnation != g.Incarnation || h.Stamp != g.Stamp || h.Session != g.Session || h.Generation != g.Generation {
+ return false, hostbox.ErrChanged
+ }
+ if y.Op.Kind == hostclaim.KindServing && y.Op.ID == "" && y.Op.Phase == "" && g.WriterBox == box && (y.Op.Exec == "" || y.Op.Exec == box) {
+ return true, nil
+ }
+ if y.Op.ID != x.Op.ID || y.Op.Kind != x.Op.Kind || x.Op.Exec != box || y.Op.Exec != box || !hostclaim.SessionReaches(y.Op.Kind, x.Op.Phase, y.Op.Phase) {
+ return false, hostbox.ErrChanged
+ }
+ return y.Op.Phase == hostclaim.PhaseOpened, nil
+}
+
+// reviewRecoveryHold reads the claim for a held probe entry (off a.mu) and
+// opens or stops it per recoveryHoldVerdict. An unreadable claim changes
+// nothing; the next sweep reads again.
+func (a *Service) reviewRecoveryHold(e *actEntry) {
+ r := a.cfg.Claims
+ a.mu.Lock()
+ admitted := e.probeAdmitted
+ a.mu.Unlock()
+ var fresh hostclaim.Observed
+ err := hostwrite.ErrUnavailable
+ if r != nil && r.Read != nil {
+ parent := a.lifeCtx
+ if parent == nil {
+ parent = context.Background()
+ }
+ ctx, cancel := context.WithTimeout(parent, 10*time.Second)
+ fresh, err = r.Read(ctx, e.host)
+ cancel()
+ }
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ e.probeReviewing = false
+ if err != nil || a.entries[e.host] != e || e.state != actAwake || !e.probeHeld {
+ return
+ }
+ open, err := recoveryHoldVerdict(admitted, fresh, r.Box)
+ switch {
+ case err != nil:
+ log.Printf("activator: %s recovery probe's session is over (%v); stopping it", e.host, err)
+ a.beginStopLocked(e)
+ case open:
+ log.Printf("activator: %s source opened on the recovery probe's session; serving", e.host)
+ e.probeHeld = false
+ e.lastActive = a.now()
+ }
+}
```
-#### New file: `internal/hostresolve/operation_owner_test.go`
+#### New file: `internal/activator/recoveryprobe_test.go`
```go
-package hostresolve
+package activator
import (
"context"
- "encoding/json"
"errors"
+ "os/exec"
"reflect"
- "slices"
+ "sync"
"testing"
+ "time"
+ "github.com/carlosframework/platform/internal/hostbox"
"github.com/carlosframework/platform/internal/hostclaim"
- "github.com/carlosframework/platform/internal/hostfence"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/moves"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
)
-type plan10LossOwner struct {
- service *Service
- store *failClaimStore
- calls []string
- failPrepare, failFinish, mutate bool
- sources []hostclaim.WriteSource
+// The recovery probe entry (claim Plan 10 J2, moves Plan 4 Task 9), driven
+// through the claim runtime the host assembles: K6, Prerequisites, Admit,
+// Prepare and the start loop. RoleLaunch.Start still refuses every launch
+// (claim Plan 9 Group D's isolated root launcher), so a full run ends in that
+// refusal as an ordinary failed start; retention of a healthy start is driven
+// through finishRecoveryProbe with the instance a healthy startAdmitted yields.
+
+const probeHost = "x.example"
+
+func probeClaim(kind, phase, exec string) *hostclaim.Claim {
+ return &hostclaim.Claim{
+ Holder: &hostclaim.Holder{Incarnation: "inc", Stamp: 1, Session: 1, Generation: "one", Location: "instances/x.example/inc", WriterBox: "box"},
+ Op: hostclaim.Op{Kind: kind, ID: "move", Phase: phase, Exec: exec},
+ }
}
-func (p *plan10LossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, record hostfence.RecordObservation, box string) (hostfence.Consequences, hostclaim.LossConsequences, error) {
- if o.Claim.Op.Kind != hostclaim.KindMoving || box != "dead" || o.Claim.Op.Intent["target"] != box || record.Key != o.Claim.Holder.Key {
- return hostfence.Consequences{}, hostclaim.LossConsequences{}, ErrResolve
+func probeObserved(c *hostclaim.Claim) hostclaim.Observed {
+ return hostclaim.Observed{Host: probeHost, ETag: "one", Claim: c}
+}
+
+func probeAdmitted(o hostclaim.Observed) hostclaim.Observed {
+ c := *o.Claim
+ h := *c.Holder
+ h.Session, h.Generation = h.Session+1, "two"
+ c.Holder = &h
+ return hostclaim.Observed{Host: o.Host, ETag: "two", Claim: &c}
+}
+
+func probeInputs() *provision.MoveInputs {
+ return &provision.MoveInputs{Release: "release", OverlayETag: "overlay", BundleETags: map[string]string{"main": "etag"}}
+}
+
+func probePolicy(in *provision.MoveInputs) StartPolicy {
+ return StartPolicy{MaxStarts: 1, Window: 30 * time.Second, Probe: true, Inputs: in}
+}
+
+// probeRun records what the claim runtime saw, in order.
+type probeRun struct {
+ mu sync.Mutex
+ steps []string
+ policies []StartPolicy
+ failedAdmission *hostclaim.Observed
+ reserved, failed, stopped int
+}
+
+func (r *probeRun) step(name string, p StartPolicy) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ r.steps = append(r.steps, name)
+ r.policies = append(r.policies, p)
+}
+
+func (r *probeRun) seen() []string {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ return append([]string(nil), r.steps...)
+}
+
+// probeService wires a Service whose claim runtime reads obs, admits it to
+// the next session and prepares a start that reaches RoleLaunch.Start.
+func probeService(t *testing.T, obs hostclaim.Observed) (*Service, *ClaimRuntime, *probeRun) {
+ t.Helper()
+ a, _, _, _ := testActivator(t, time.Hour)
+ run := &probeRun{}
+ admitted := probeAdmitted(obs)
+ work, _ := (hostwrite.Config{}).Select(t.Context(), probeHost, "spawn")
+ reads := 0
+ r := &ClaimRuntime{
+ Box: "box",
+ // The wake's first read sees obs; every later one (startAdmitted's
+ // fresh read) sees the admission.
+ Read: func(_ context.Context, host string) (hostclaim.Observed, error) {
+ run.step("read", StartPolicy{})
+ run.mu.Lock()
+ defer run.mu.Unlock()
+ if reads++; reads > 1 {
+ return admitted, nil
+ }
+ return obs, nil
+ },
+ StopForeign: func(context.Context, string) error { run.step("stop-foreign", StartPolicy{}); return nil },
+ Prerequisites: func(_ context.Context, rt registry.Route, o hostclaim.Observed, p StartPolicy) error {
+ run.step("prerequisites", p)
+ return nil
+ },
+ Admit: func(_ context.Context, _ registry.Route, o hostclaim.Observed, p StartPolicy) (hostclaim.Observed, error) {
+ run.step("admit", p)
+ return admitted, nil
+ },
+ Prepare: func(_ context.Context, _ registry.Route, o hostclaim.Observed, p StartPolicy) (ClaimStart, error) {
+ run.step("prepare", p)
+ return ClaimStart{RoleLauncher: &RoleLaunch{}, CurrentAttempt: func() hostclaim.WakeRetryAttempt { return hostclaim.WakeRetryAttempt{} }, Host: probeHost, Work: work, Identity: hostwrite.Identity{Config: "c", Release: "r"},
+ OpenProbe: func(context.Context, time.Time) (time.Time, error) { return time.Now(), nil },
+ Restore: func(context.Context) error { return nil },
+ Materialize: func(context.Context) error { return nil },
+ Build: func() *exec.Cmd { return nil },
+ Ready: func(context.Context, *exec.Cmd) error { return nil },
+ Reserve: func(context.Context, int) error { run.mu.Lock(); run.reserved++; run.mu.Unlock(); return nil },
+ Failed: func(context.Context, int) error { run.mu.Lock(); run.failed++; run.mu.Unlock(); return nil },
+ Healthy: func(context.Context, int, *exec.Cmd) error { return nil },
+ Replicate: func(context.Context, string) (Rep, error) { return &fakeRep{}, nil },
+ Publish: func(_ context.Context, o hostclaim.Observed) (hostclaim.Observed, error) { return o, nil },
+ Retain: func(Proc, Rep, error) { t.Error("a refused start is not retained") },
+ }, nil
+ },
+ FailedAdmission: func(_ context.Context, o hostclaim.Observed, _ error) error {
+ run.mu.Lock()
+ run.failedAdmission = &o
+ run.mu.Unlock()
+ return nil
+ },
}
- actual := hostclaim.LossConsequences{RemoveParticipation: true, WritesPotentiallyLost: true, TimersPotentiallyLost: true, RecoveryIntent: json.RawMessage(`{"kind":"test-move-loss"}`)}
- if p.mutate {
- o.Claim.Op.Intent["target"] = "wrong"
- o.Claim.Holder.Stamp++
+ a.cfg.HostWrites = hostwrite.Config{Enabled: true}
+ a.cfg.Claims = r
+ return a, r, run
+}
+
+func TestRecoveryProbeRunsTheClaimRuntimeWithTheProbePolicyAndExactInputs(t *testing.T) {
+ for _, kind := range []string{hostclaim.KindRestoring, hostclaim.KindReverting} {
+ t.Run(kind, func(t *testing.T) {
+ obs := probeObserved(probeClaim(kind, hostclaim.PhaseProbing, "box"))
+ a, _, run := probeService(t, obs)
+ in := probeInputs()
+ want := *in
+ err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(in))
+ // The start reached RoleLaunch.Start, which refuses until Group D:
+ // an honest failed start, never a success.
+ if !errors.Is(err, ErrClaimIsolationUnavailable) {
+ t.Fatalf("err = %v, want the launcher's refusal", err)
+ }
+ steps := run.seen()
+ if len(steps) < 4 || steps[0] != "read" || steps[1] != "prerequisites" || steps[2] != "admit" || steps[len(steps)-1] != "prepare" {
+ t.Fatalf("steps = %v", steps)
+ }
+ run.mu.Lock()
+ defer run.mu.Unlock()
+ for i, p := range run.policies {
+ if run.steps[i] == "read" || run.steps[i] == "stop-foreign" {
+ continue
+ }
+ if !p.Probe || !reflect.DeepEqual(p.Inputs, in) || p.MaxStarts != 1 || p.Window != 30*time.Second || p.Prepared != nil {
+ t.Fatalf("%s saw %+v, want the caller's probe policy", run.steps[i], p)
+ }
+ }
+ if in.Release != want.Release || in.OverlayETag != want.OverlayETag || in.BundleETags["main"] != "etag" {
+ t.Fatalf("inputs changed: %+v", in)
+ }
+ if run.reserved != 1 || run.failed != 1 {
+ t.Fatalf("reserved=%d failed=%d, want one refused start", run.reserved, run.failed)
+ }
+ if run.failedAdmission == nil || run.failedAdmission.Claim.Holder.Session != 2 {
+ t.Fatalf("the failed admission was not handed back: %+v", run.failedAdmission)
+ }
+ if a.Owns(probeHost) {
+ t.Fatal("a failed probe left an entry")
+ }
+ })
}
- return hostfence.Consequences{Transition: "test-move-loss", PossibleWriteLoss: true, PossibleTimerLoss: true}, actual, nil
}
-func (p *plan10LossOwner) LossWriteSources(context.Context, hostclaim.Observed, hostclaim.LossConsequences) ([]hostclaim.WriteSource, error) {
- return p.sources, nil
+
+func TestRecoveryProbeK6StopsForeignBeforeAnyAdmission(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "elsewhere"))
+ a, _, run := probeService(t, obs)
+ err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs()))
+ if !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("err = %v", err)
+ }
+ if got := run.seen(); len(got) != 2 || got[0] != "read" || got[1] != "stop-foreign" {
+ t.Fatalf("steps = %v, want read then stop-foreign only", got)
+ }
+ if a.Owns(probeHost) {
+ t.Fatal("entry left")
+ }
}
-func (p *plan10LossOwner) ValidateDeclareLost(c hostclaim.Claim, d hostclaim.LostDeclaration) error {
- if c.Op.Kind != hostclaim.KindMoving || c.Op.Intent["target"] != d.Box || d.Box != "dead" || !d.Consequences.RemoveParticipation {
- return ErrResolve
+
+func TestRecoveryProbeRefusesWrongKindOrPhaseBeforePrerequisites(t *testing.T) {
+ for _, c := range []*hostclaim.Claim{
+ probeClaim(hostclaim.KindRestoring, hostclaim.PhaseClaimed, "box"),
+ probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbed, "box"),
+ probeClaim(hostclaim.KindReverting, hostclaim.PhaseOpened, "box"),
+ probeClaim(hostclaim.KindLanding, "probing", "box"),
+ {Holder: &hostclaim.Holder{Incarnation: "inc", Session: 1, Generation: "one", Location: "l", WriterBox: "box"}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"}},
+ } {
+ t.Run(c.Op.Kind+"/"+c.Op.Phase, func(t *testing.T) {
+ a, _, run := probeService(t, probeObserved(c))
+ err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs()))
+ if !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("err = %v", err)
+ }
+ if got := run.seen(); len(got) != 1 || got[0] != "read" {
+ t.Fatalf("steps = %v, want the read alone", got)
+ }
+ if a.Owns(probeHost) {
+ t.Fatal("entry left")
+ }
+ })
}
- return nil
}
-func (p *plan10LossOwner) PrepareLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
- p.calls = append(p.calls, "prepare")
- roster, err := p.service.Directory.Read(ctx)
- if err != nil || roster.Roster.Boxes[loss.Box].State != "quarantined" {
- return ErrResolve
+
+func TestRecoveryProbeRefusesMissingRuntimeOrPolicyBeforeAnyRead(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ in := probeInputs()
+ runtimes := map[string]func(*Service, *ClaimRuntime){
+ "host writes off": func(a *Service, _ *ClaimRuntime) { a.cfg.HostWrites.Enabled = false },
+ "no claims": func(a *Service, _ *ClaimRuntime) { a.cfg.Claims = nil },
+ "no box": func(_ *Service, r *ClaimRuntime) { r.Box = "" },
+ "no stop foreign": func(_ *Service, r *ClaimRuntime) { r.StopForeign = nil },
+ "no prerequisites": func(_ *Service, r *ClaimRuntime) { r.Prerequisites = nil },
+ "no admit": func(_ *Service, r *ClaimRuntime) { r.Admit = nil },
+ "no prepare": func(_ *Service, r *ClaimRuntime) { r.Prepare = nil },
+ "no failed admission": func(_ *Service, r *ClaimRuntime) { r.FailedAdmission = nil },
}
- durable, err := hostclaim.ReadLossRecord(ctx, p.store, p.service.Directory.PublicKey, loss.ID)
- if err != nil || durable.Record.State != "prepared" || slices.Contains(o.Claim.LossRefs, loss.ID) {
- return ErrResolve
+ for name, breakIt := range runtimes {
+ t.Run(name, func(t *testing.T) {
+ a, r, run := probeService(t, obs)
+ breakIt(a, r)
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(in)); !errors.Is(err, hostwrite.ErrUnavailable) {
+ t.Fatalf("err = %v", err)
+ }
+ if got := run.seen(); len(got) != 0 || a.Owns(probeHost) {
+ t.Fatalf("steps = %v owns=%v", got, a.Owns(probeHost))
+ }
+ })
}
- if p.failPrepare {
- return ErrResolve
+ policies := map[string]StartPolicy{
+ "ordinary": {MaxStarts: 1, Window: 30 * time.Second, Inputs: in},
+ "no inputs": {MaxStarts: 1, Window: 30 * time.Second, Probe: true},
+ "prepared": {MaxStarts: 1, Window: 30 * time.Second, Probe: true, Inputs: in, Prepared: &moves.InstallManifest{Op: "move"}},
+ "no starts": {Window: 30 * time.Second, Probe: true, Inputs: in},
+ "too many": {MaxStarts: 6, Window: 30 * time.Second, Probe: true, Inputs: in},
+ "no window": {MaxStarts: 1, Probe: true, Inputs: in},
+ "window too big": {MaxStarts: 1, Window: 31 * time.Second, Probe: true, Inputs: in},
}
- if p.mutate {
- o.Claim.Op.Intent["target"] = "wrong"
- loss.ConfirmedConsequences.RecoveryIntent[0] = 'x'
+ for name, p := range policies {
+ t.Run(name, func(t *testing.T) {
+ a, _, run := probeService(t, obs)
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, p); !errors.Is(err, hostbox.ErrUnsupported) {
+ t.Fatalf("err = %v", err)
+ }
+ if got := run.seen(); len(got) != 0 || a.Owns(probeHost) {
+ t.Fatalf("steps = %v", got)
+ }
+ })
+ }
+ a, _, run := probeService(t, obs)
+ if err := a.RecoveryProbe(t.Context(), registry.Route{}, probePolicy(in)); !errors.Is(err, hostbox.ErrUnsupported) || len(run.seen()) != 0 {
+ t.Fatalf("no host: %v", err)
}
- return nil
}
-func (p *plan10LossOwner) FinishLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
- p.calls = append(p.calls, "finish")
- durable, err := hostclaim.ReadLossRecord(ctx, p.store, p.service.Directory.PublicKey, loss.ID)
- if err != nil || durable.Record.State != "committed" || !slices.Contains(o.Claim.LossRefs, loss.ID) {
- return ErrResolve
+
+func TestRecoveryProbeRefusesAnOrdinaryEntryAndShutdown(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ for _, state := range []actState{actWaking, actAwake, actStopping} {
+ a, _, run := probeService(t, obs)
+ a.entries[probeHost] = &actEntry{host: probeHost, state: state, ready: make(chan struct{}), stopped: make(chan struct{})}
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("state %v: %v", state, err)
+ }
+ if got := run.seen(); len(got) != 0 {
+ t.Fatalf("state %v: steps %v", state, got)
+ }
}
- if p.failFinish {
- return ErrResolve
+ a, _, run := probeService(t, obs)
+ a.shuttingDown = true
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, ErrFenced) || len(run.seen()) != 0 {
+ t.Fatalf("shutdown: %v", err)
+ }
+}
+
+// One probe per host: an identical second call joins the one in flight and
+// shares its outcome; a different policy is refused; a request never joins.
+func TestRecoveryProbeCoalescesTheSamePolicyAndRefusesOthers(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, r, run := probeService(t, obs)
+ entered, release := make(chan struct{}), make(chan struct{})
+ var once sync.Once
+ refuse := errors.New("prerequisites refused")
+ r.Prerequisites = func(_ context.Context, _ registry.Route, _ hostclaim.Observed, p StartPolicy) error {
+ run.step("prerequisites", p)
+ once.Do(func() { close(entered) })
+ <-release
+ return refuse
+ }
+ in := probeInputs()
+ first := make(chan error, 1)
+ go func() {
+ first <- a.RecoveryProbe(context.Background(), registry.Route{Host: probeHost}, probePolicy(in))
+ }()
+ <-entered
+
+ other := probeInputs()
+ other.Release = "other"
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(other)); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("different inputs: %v", err)
+ }
+ longer := probePolicy(in)
+ longer.MaxStarts = 2
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, longer); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("different starts: %v", err)
+ }
+ if _, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) {
+ t.Fatalf("a request joined the probe: %v", err)
+ }
+ if err := a.BeginSidecarRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) {
+ t.Fatalf("a sidecar request joined the probe: %v", err)
+ }
+ if a.Bounce(probeHost) {
+ t.Fatal("bounce stopped the probe")
+ }
+
+ same := probeInputs()
+ second := make(chan error, 1)
+ go func() {
+ second <- a.RecoveryProbe(context.Background(), registry.Route{Host: probeHost}, probePolicy(same))
+ }()
+ time.Sleep(100 * time.Millisecond) // let the second call reach the entry
+ close(release)
+ if err := <-first; !errors.Is(err, refuse) {
+ t.Fatalf("first: %v", err)
+ }
+ if err := <-second; !errors.Is(err, refuse) {
+ t.Fatalf("joined: %v", err)
}
- if p.mutate {
- o.Claim.Op.Intent["target"] = "wrong"
- loss.ConfirmedConsequences.RecoveryIntent[0] = 'x'
+ n := 0
+ for _, s := range run.seen() {
+ if s == "prerequisites" {
+ n++
+ }
+ }
+ if n != 1 || a.Owns(probeHost) {
+ t.Fatalf("prerequisites ran %d times, owns=%v", n, a.Owns(probeHost))
+ }
+}
+
+// A maintenance fence installed while the probe runs cancels it, exactly
+// as it cancels an ordinary wake.
+func TestRecoveryProbeIsCancelledByAFenceInstalledMidProbe(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, r, run := probeService(t, obs)
+ entered := make(chan struct{})
+ r.Prerequisites = func(ctx context.Context, _ registry.Route, _ hostclaim.Observed, p StartPolicy) error {
+ run.step("prerequisites", p)
+ close(entered)
+ <-ctx.Done()
+ return ctx.Err()
+ }
+ done := make(chan error, 1)
+ go func() {
+ done <- a.RecoveryProbe(context.Background(), registry.Route{Host: probeHost}, probePolicy(probeInputs()))
+ }()
+ <-entered
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ if _, err := a.FenceAndStop(ctx, probeHost, 7); err != nil && !errors.Is(err, ErrStopUnproven) {
+ t.Fatal(err)
+ }
+ if err := <-done; !errors.Is(err, context.Canceled) {
+ t.Fatalf("err = %v", err)
+ }
+ if a.Owns(probeHost) {
+ t.Fatal("entry left")
}
- return nil
}
-func plan10ResolveMove(t *testing.T) (Service, *plan10LossOwner, hostfence.Submission, string) {
+
+// heldProbe installs a probe entry as RecoveryProbe does and finishes it with
+// the instance a healthy startAdmitted yields.
+func heldProbe(t *testing.T, a *Service, fence uint64, admitted hostclaim.Observed, in *provision.MoveInputs) (*actEntry, *fakeProc, *fakeRep, error) {
t.Helper()
- s, store, now, host := resolveFixture(t)
- observed, err := hostclaim.Read(t.Context(), store, host)
+ p := probePolicy(in)
+ e := &actEntry{host: probeHost, route: registry.Route{Host: probeHost}, state: actWaking, ready: make(chan struct{}), probe: &p, probeHeld: true}
+ a.mu.Lock()
+ a.entries[probeHost] = e
+ a.mu.Unlock()
+ e.probeAdmitted = admitted
+ proc, rep := &fakeProc{}, &fakeRep{}
+ err := a.finishRecoveryProbe(e, fence, &wokenInstance{proc: proc, rep: rep, spawnVersion: "v", stopOutcome: &stopObservation{}}, nil)
+ return e, proc, rep, err
+}
+
+func TestRecoveryProbeRetainsTheHealthyProcessHeldUntilOpened(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ admitted := probeAdmitted(obs)
+ a, r, _ := probeService(t, obs)
+ // The rollback's fence stays installed through the recovery.
+ a.fences = map[string]uint64{probeHost: 9}
+ in := probeInputs()
+ e, proc, _, err := heldProbe(t, a, 9, admitted, in)
if err != nil {
t.Fatal(err)
}
- _, err = hostclaim.Update(t.Context(), store, observed, func(c *hostclaim.Claim) error {
- c.Op = hostclaim.Op{ID: "move-test", Kind: hostclaim.KindMoving, Phase: hostclaim.PhaseMoving, Exec: "live", Participants: []string{"live", "dead"}, Intent: map[string]string{"source": "live", "target": "dead", "to": "de"}}
- return nil
- })
- if err != nil {
- t.Fatal(err)
+ a.mu.Lock()
+ state, held, kept := e.state, e.probeHeld, e.proc == Proc(proc)
+ a.mu.Unlock()
+ if state != actAwake || !held || !kept || proc.isStopped() {
+ t.Fatalf("state=%v held=%v kept=%v stopped=%v", state, held, kept, proc.isStopped())
}
- owner := &plan10LossOwner{store: store}
- s.Owners = map[string]OperationOwner{hostclaim.KindMoving: owner}
- owner.service = &s
- return s, owner, submitted(t, s, now, host), host
-}
-func TestPlan10MoveOwnerRequiredBeforeAcceptance(t *testing.T) {
- s, p, in, host := plan10ResolveMove(t)
- s.Owners = nil
- before, _ := s.Directory.Read(t.Context())
- if _, err := s.Inspect(t.Context(), "operator", host, "dead"); !errors.Is(err, ErrResolve) {
- t.Fatal(err)
+ delete(a.fences, probeHost)
+ if _, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost}); !errors.Is(err, ErrFenced) {
+ t.Fatalf("a request reached the held probe: %v", err)
}
- if _, err := s.Declare(t.Context(), "operator", host, in); !errors.Is(err, ErrResolve) {
- t.Fatal(err)
+ if a.Bounce(probeHost) || proc.isStopped() {
+ t.Fatal("bounce stopped the held probe")
}
- after, _ := s.Directory.Read(t.Context())
- if after.ETag != before.ETag {
- t.Fatal("missing owner quarantined")
+
+ // A replayed call for the same session answers from the retained process.
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil }
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); err != nil {
+ t.Fatalf("replay: %v", err)
}
- keys, err := p.store.List(t.Context(), "control/host-loss-acceptances/")
- if err != nil || len(keys) != 0 {
- t.Fatal(keys, err)
+ moved := probeAdmitted(admitted)
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) { return moved, nil }
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("replay on another session: %v", err)
+ }
+ proc.exit()
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil }
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(probeInputs())); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("replay on an exited process: %v", err)
}
}
-func TestPlan10OwnerHookOrderAndReplay(t *testing.T) {
- for _, fault := range []string{"prepare", "claim", "bookkeeping", "finish"} {
- t.Run(fault, func(t *testing.T) {
- s, p, in, host := plan10ResolveMove(t)
- p.failPrepare = fault == "prepare"
- p.store.fail = fault == "claim"
- p.failFinish = fault == "finish"
- bookkeeping := &lossBookkeepingStore{Store: p.store, fail: fault == "bookkeeping"}
- s.Directory.Store = bookkeeping
- if _, err := s.Declare(t.Context(), "operator", host, in); err == nil {
- t.Fatal("fault not reached")
+
+func TestRecoveryProbeStopsAHealthyStartWhenTheFenceMoved(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ for name, now := range map[string]map[string]uint64{"installed": {probeHost: 3}, "lifted": {}, "replaced": {probeHost: 12}} {
+ t.Run(name, func(t *testing.T) {
+ a, _, _ := probeService(t, obs)
+ a.fences = now
+ start := uint64(0)
+ if name != "installed" {
+ start = 9
}
- p.failPrepare = false
- p.store.fail = false
+ e, proc, rep, err := heldProbe(t, a, start, probeAdmitted(obs), probeInputs())
+ if !errors.Is(err, ErrFenced) {
+ t.Fatalf("err = %v", err)
+ }
+ select {
+ case <-e.stopped:
+ case <-time.After(5 * time.Second):
+ t.Fatal("not stopped")
+ }
+ if !proc.isStopped() || !rep.isClosed() || a.Owns(probeHost) {
+ t.Fatalf("stopped=%v closed=%v owns=%v", proc.isStopped(), rep.isClosed(), a.Owns(probeHost))
+ }
+ })
+ }
+}
+
+func TestRecoveryHoldVerdict(t *testing.T) {
+ admitted := probeAdmitted(probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box")))
+ at := func(mutate func(*hostclaim.Claim)) hostclaim.Observed {
+ c := *admitted.Claim
+ h := *c.Holder
+ c.Holder = &h
+ mutate(&c)
+ return hostclaim.Observed{Host: probeHost, ETag: "later", Claim: &c}
+ }
+ cases := []struct {
+ name string
+ obs hostclaim.Observed
+ open bool
+ err bool
+ }{
+ {"probing", at(func(*hostclaim.Claim) {}), false, false},
+ {"probed", at(func(c *hostclaim.Claim) { c.Op.Phase = hostclaim.PhaseProbed }), false, false},
+ {"opened", at(func(c *hostclaim.Claim) { c.Op.Phase = hostclaim.PhaseOpened }), true, false},
+ {"released to serving", at(func(c *hostclaim.Claim) { c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"} }), true, false},
+ {"claimed", at(func(c *hostclaim.Claim) { c.Op.Phase = hostclaim.PhaseClaimed }), false, true},
+ {"new session", at(func(c *hostclaim.Claim) { c.Holder.Session++ }), false, true},
+ {"new generation", at(func(c *hostclaim.Claim) { c.Holder.Generation = "three" }), false, true},
+ {"other op", at(func(c *hostclaim.Claim) { c.Op.ID = "other" }), false, true},
+ {"other exec", at(func(c *hostclaim.Claim) { c.Op.Exec = "elsewhere" }), false, true},
+ {"lost", at(func(c *hostclaim.Claim) { c.Lost = []string{"box"} }), false, true},
+ {"serving elsewhere", at(func(c *hostclaim.Claim) {
+ c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "elsewhere"}
+ c.Holder.WriterBox = "elsewhere"
+ }), false, true},
+ {"absent", hostclaim.Observed{Host: probeHost, Absent: true}, false, true},
+ }
+ for _, c := range cases {
+ open, err := recoveryHoldVerdict(admitted, c.obs, "box")
+ if open != c.open || (err != nil) != c.err || (err != nil && !errors.Is(err, hostbox.ErrChanged)) {
+ t.Errorf("%s: open=%v err=%v", c.name, open, err)
+ }
+ }
+}
+
+func TestRecoveryHoldReviewOpensOrStops(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ admitted := probeAdmitted(obs)
+ opened := probeAdmitted(obs)
+ oc := *opened.Claim
+ oc.Op.Phase = hostclaim.PhaseOpened
+ opened.Claim = &oc
+
+ a, r, _ := probeService(t, obs)
+ e, proc, _, err := heldProbe(t, a, 0, admitted, probeInputs())
+ if err != nil {
+ t.Fatal(err)
+ }
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) { return admitted, nil }
+ a.reviewRecoveryHold(e)
+ a.mu.Lock()
+ held := e.probeHeld
+ a.mu.Unlock()
+ if !held {
+ t.Fatal("released at probing")
+ }
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) {
+ return hostclaim.Observed{}, errors.New("offline")
+ }
+ a.reviewRecoveryHold(e)
+ a.mu.Lock()
+ held, state := e.probeHeld, e.state
+ a.mu.Unlock()
+ if !held || state != actAwake {
+ t.Fatal("an unreadable claim changed the hold")
+ }
+ r.Read = func(context.Context, string) (hostclaim.Observed, error) { return opened, nil }
+ a.reviewRecoveryHold(e)
+ a.mu.Lock()
+ held, state = e.probeHeld, e.state
+ a.mu.Unlock()
+ if held || state != actAwake || proc.isStopped() {
+ t.Fatalf("opened: held=%v state=%v", held, state)
+ }
+ release, err := a.beginRequest(t.Context(), registry.Route{Host: probeHost})
+ if err != nil {
+ t.Fatalf("an opened probe does not serve: %v", err)
+ }
+ release()
+
+ b, rb, _ := probeService(t, obs)
+ e, proc, _, err = heldProbe(t, b, 0, admitted, probeInputs())
+ if err != nil {
+ t.Fatal(err)
+ }
+ moved := probeAdmitted(admitted)
+ rb.Read = func(context.Context, string) (hostclaim.Observed, error) { return moved, nil }
+ b.reviewRecoveryHold(e)
+ select {
+ case <-e.stopped:
+ case <-time.After(5 * time.Second):
+ t.Fatal("a changed session kept its process")
+ }
+ if !proc.isStopped() || b.Owns(probeHost) {
+ t.Fatal("not stopped")
+ }
+}
+
+// With the probe fields unset, the claim wake is main's: the ordinary
+// policy, never Probe, never Inputs.
+func TestOrdinaryClaimWakeKeepsTheServingPolicy(t *testing.T) {
+ c := &hostclaim.Claim{Holder: &hostclaim.Holder{Incarnation: "inc", Session: 1, Generation: "one", Location: "l", WriterBox: "box"}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"}}
+ a, r, run := probeService(t, probeObserved(c))
+ stop := errors.New("stop")
+ r.Prerequisites = func(_ context.Context, _ registry.Route, _ hostclaim.Observed, p StartPolicy) error {
+ run.step("prerequisites", p)
+ return stop
+ }
+ if _, err := a.doClaimWake(t.Context(), &actEntry{host: probeHost, route: registry.Route{Host: probeHost}}); !errors.Is(err, stop) {
+ t.Fatal(err)
+ }
+ run.mu.Lock()
+ defer run.mu.Unlock()
+ p := run.policies[len(run.policies)-1]
+ if p != (StartPolicy{MaxStarts: 5, Window: 30 * time.Second}) {
+ t.Fatalf("ordinary policy %+v", p)
+ }
+}
+
+func TestRecoveryProbeOwnsItsInputs(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, r, _ := probeService(t, obs)
+ in := probeInputs()
+ called := false
+ r.Prerequisites = func(_ context.Context, _ registry.Route, _ hostclaim.Observed, p StartPolicy) error {
+ called = true
+ in.Release = "changed"
+ in.BundleETags["main"] = "changed"
+ if p.Inputs.Release != "release" || p.Inputs.BundleETags["main"] != "etag" {
+ t.Error("caller mutated captured probe inputs")
+ }
+ return hostbox.ErrChanged
+ }
+ _ = a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(in))
+ if !called {
+ t.Fatal("did not reach prerequisites")
+ }
+}
+
+func TestRecoveryProbeChangedInputsRetireHeldEntry(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, _, _ := probeService(t, obs)
+ e, proc, rep, err := heldProbe(t, a, 0, probeAdmitted(obs), probeInputs())
+ if err != nil {
+ t.Fatal(err)
+ }
+ in := probeInputs()
+ in.Release = "new-release"
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(in)); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatal(err)
+ }
+ a.mu.Lock()
+ stopped := e.stopped
+ state := e.state
+ a.mu.Unlock()
+ if state != actStopping && a.Owns(probeHost) {
+ t.Fatal("changed inputs stranded old held probe")
+ }
+ select {
+ case <-stopped:
+ case <-time.After(5 * time.Second):
+ t.Fatal("old probe did not stop")
+ }
+ if !proc.isStopped() || !rep.isClosed() || a.Owns(probeHost) {
+ t.Fatal("old probe retained resources")
+ }
+}
+
+func TestRecoveryProbeChangedInputsCancelWakingEntry(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, _, _ := probeService(t, obs)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ p := probePolicy(probeInputs())
+ e := &actEntry{host: probeHost, probe: &p, probeHeld: true, state: actWaking, wakeCancel: cancel, ready: make(chan struct{})}
+ a.entries[probeHost] = e
+ in := probeInputs()
+ in.Release = "new"
+ if err := a.RecoveryProbe(t.Context(), registry.Route{Host: probeHost}, probePolicy(in)); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatal(err)
+ }
+ if ctx.Err() == nil {
+ t.Fatal("stale in-flight probe was not cancelled")
+ }
+ delete(a.entries, probeHost)
+}
+
+func TestRecoveryHoldSurvivesEdgeRetireExcept(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, _, _ := probeService(t, obs)
+ e, _, _, err := heldProbe(t, a, 0, probeAdmitted(obs), probeInputs())
+ if err != nil {
+ t.Fatal(err)
+ }
+ a.RetireExcept(map[string]bool{})
+ a.mu.Lock()
+ state := e.state
+ a.mu.Unlock()
+ if state != actAwake {
+ t.Fatal("edge reload retired operation-owned recovery probe")
+ }
+}
+
+func TestRecoveryProbeLateHealthyAfterInputChangeStops(t *testing.T) {
+ obs := probeObserved(probeClaim(hostclaim.KindRestoring, hostclaim.PhaseProbing, "box"))
+ a, _, _ := probeService(t, obs)
+ p := probePolicy(probeInputs())
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ e := &actEntry{host: probeHost, probe: &p, probeHeld: true, state: actWaking, wakeCancel: cancel, ready: make(chan struct{}), probeAdmitted: probeAdmitted(obs)}
+ a.entries[probeHost] = e
+ in := probeInputs()
+ in.Release = "new"
+ _ = a.joinRecoveryProbe(t.Context(), e, probePolicy(in))
+ if ctx.Err() == nil {
+ t.Fatal("not cancelled")
+ }
+ proc, rep := &fakeProc{}, &fakeRep{}
+ err := a.finishRecoveryProbe(e, 0, &wokenInstance{proc: proc, rep: rep, stopOutcome: &stopObservation{}}, nil)
+ if !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatal("late success kept obsolete inputs", err)
+ }
+ select {
+ case <-e.stopped:
+ case <-time.After(5 * time.Second):
+ t.Fatal("not stopped")
+ }
+ if !proc.isStopped() || !rep.isClosed() {
+ t.Fatal("obsolete process remains")
+ }
+}
+```
+
+#### Patch: `internal/host/movespass.go`
+
+```diff
+--- a/internal/host/movespass.go
++++ b/internal/host/movespass.go
+@@ -388,0 +389,5 @@
++ }
++ // Recovery uses the same admitted Group D start path as other claim
++ // writers. Preserve the legacy nil hook with the claim switch off.
++ if cfg.claimRecords && svc != nil {
++ armRecoveryProbe(cfg, w, svc.RecoveryProbe, svc.RecoveryProbeReady)
+@@ -590,2 +595,3 @@
+- // probe (Task 9, BLOCKED-ON-CLAIM C1, rulings J2 and J18): nil in
+- // production, where probing logs and waits, and faked in the tests.
++ // probe (Task 9, rulings J2 and J18; claim Plan 10 J2): in production
++ // armRecoveryProbe binds the activator's Group D entry when host claims
++ // are enabled. Missing runtime wiring waits before recording inputs.
+```
+
+#### Patch: `internal/host/movesrecover.go`
+
+```diff
+--- a/internal/host/movesrecover.go
++++ b/internal/host/movesrecover.go
+@@ -67 +67,3 @@
+-// renders before it renders them (MoveRecordRecoveryInputs). Once the
++// renders before it renders them (MoveRecordRecoveryInputs, at
++// probing: recoveryProbeAdapter, movesrecoveryprobe.go, claim Plan 10
++// J2; a clean return records only here). Once the
+@@ -96 +98,3 @@
+-// nil in production too, until Task 9; probing then logs and waits. LogEvent
++// the recovery probe adapter (armRecoveryProbe) with CARLOS_HOST_CLAIM on,
++// bound to the activator's Group D entry. Without the claim switch the
++// legacy nil hook still waits, recording nothing. LogEvent
+@@ -625,0 +630,7 @@
++ var missing *ClaimDependencyError
++ if errors.As(err, &missing) {
++ // The adapter refused before recording or waking (Plan 10 J2):
++ // probing waits, as it did with no hook.
++ r.w.notice(host+"#rc-probe", "moves: %s: the recovery probe of rollback %s waits: %v", host, r.rb, err)
++ return false, nil
++ }
+```
+
+#### New file: `internal/host/movesrecoveryprobe.go`
+
+```go
+package host
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/moves"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The source's recovery probe adapter (moves §7.8 source probing; claim
+// Plan 10 J2). It is the production recoveryProbe hook: movesrecover.go's
+// probing calls it once the record says probing and the claim is this
+// rollback's restoring op at probing with this box executing.
+//
+// One call is one probe-shaped claim wake, in this order:
+//
+// 1. the request is checked against the record (this rollback at probing,
+// nine attempts, the coverage and generation of its return) and the
+// claim is read fresh (restoring/probing, this op, this rollback, this
+// box executing, the record's incarnation, a clean return's src_gen);
+// 2. the inputs the probe will render are selected (the box's current
+// inputs, as the cut reads them) and recorded through req.Record, which
+// is provision.MoveRecordRecoveryInputs under the move and rollback ids,
+// legal only at probing. A clean return records here: it has no
+// configured phase. A dirty return records again, replacing what its
+// install recorded at configured, because this render is the probe's.
+// A failed record write stops the attempt: nothing is rendered or run;
+// 3. the activator is asked for a probe-shaped claim wake (StartPolicy
+// Probe, carrying exactly the recorded inputs). That wake is the claim's
+// Group D path unchanged: Prerequisites and admitsOperation admit a
+// recovery probe only at probing with the moves C1 owners bound; Admit,
+// the pre-launch record and Prepare bind OperationAttempts and the
+// role-cgroup probe readiness to the admitted session; the moves Inputs
+// owner selects and materializes (the held render) the recorded inputs,
+// and moves' policy admits only those (MovesClaimPolicy).
+//
+// The adapter writes nothing besides req.Record. The restore, install and
+// evidence remain the moves owners'; the nine-attempt budget is journalled
+// by moves' OperationAttempts, which refuses a start past it with
+// ErrRecoveryProbeExhausted.
+
+// ClaimRecoveryProbeWake is the activator's recovery probe entry: one
+// claim wake of rt under p (p.Probe set), through ClaimRuntime's
+// Prerequisites, Admit and Prepare and the activator's own start loop. It
+// returns nil once a start was healthy (OperationAttempts.Healthy accepted
+// it). Production binds Service.RecoveryProbe, which retains the process
+// in the activator and refuses missing claim-runtime dependencies.
+type ClaimRecoveryProbeWake func(ctx context.Context, rt registry.Route, p activator.StartPolicy) error
+
+// ErrRecoveryProbeExhausted is the moves attempt journal's refusal once the
+// recovery's nine starts are spent. The recovery stops for an operator at
+// probing (StuckRecoveryProbeAttempts).
+var ErrRecoveryProbeExhausted = errors.New("moves: the recovery probe's attempts are spent")
+
+// errRecoveryProbeRequest is a request that does not match its record: a
+// caller bug, never retried into a wake.
+var errRecoveryProbeRequest = errors.New("moves: the recovery probe request does not match its record")
+
+// errRecoveryInputsNoRelease: the selected inputs name no release, which the
+// record transition would refuse; the adapter refuses first.
+var errRecoveryInputsNoRelease = errors.New("moves: the recovery probe's inputs name no release")
+
+// recoveryProbeStartsPerWake is the starts one probe wake may run. The
+// activator allows at most five per wake; the recovery's nine are a budget
+// across wakes, journalled by moves.
+const recoveryProbeStartsPerWake = 3
+
+// recoveryProbeWindow is the probe's readiness window from its first
+// app start (§7.7's 30 s).
+const recoveryProbeWindow = 30 * time.Second
+
+// recoveryProbeAdapter is the source's recovery probe. Claims is the
+// captured CARLOS_HOST_CLAIM; Route reads this box's registry row; Current
+// selects the inputs the probe will render; Wake is the activator's entry.
+type recoveryProbeAdapter struct {
+ Box string
+ Claims bool
+ Route func(host string) (*registry.Route, error)
+ Current func(ctx context.Context, s release.Store, rec provision.Record) (provision.MoveInputs, error)
+ Wake ClaimRecoveryProbeWake
+ Ready func() bool
+}
+
+// armRecoveryProbe binds the production recoveryProbe hook: the adapter
+// over this worker's configuration, read when the probe runs. wake is the
+// activator's recovery probe entry; tests may leave it nil to prove refusal.
+func armRecoveryProbe(cfg *moveCfg, w *movesWorker, wake ClaimRecoveryProbeWake, ready func() bool) {
+ cfg.recoveryProbe = func(ctx context.Context, s release.Store, rec provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ c := w.cfg
+ a := recoveryProbeAdapter{Box: c.Box, Claims: c.claimRecords, Route: c.Route, Current: w.currentInputs, Wake: wake, Ready: ready}
+ return a.probe(ctx, s, rec, req)
+ }
+}
+
+// probe runs one recovery probe wake for rec (see the file comment).
+func (a recoveryProbeAdapter) probe(ctx context.Context, s release.Store, rec provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ var missing []string
+ if !a.Claims {
+ missing = append(missing, "CARLOS_HOST_CLAIM Group D recovery admission")
+ }
+ if a.Wake == nil || (a.Ready != nil && !a.Ready()) {
+ missing = append(missing, "activator recovery probe entry")
+ }
+ if a.Box == "" || a.Route == nil || a.Current == nil {
+ missing = append(missing, "moves recovery probe route and inputs")
+ }
+ if len(missing) > 0 {
+ return recoveryProbeWaiting, &ClaimDependencyError{Contracts: missing}
+ }
+ if err := recoveryProbeMatches(rec, req); err != nil {
+ return recoveryProbeWaiting, err
+ }
+ obs, err := hostclaim.Read(ctx, s, rec.Host)
+ if err != nil {
+ return recoveryProbeWaiting, fmt.Errorf("read the host claim: %w", err)
+ }
+ if err := a.admits(obs, rec, req); err != nil {
+ return recoveryProbeWaiting, err
+ }
+ rt, err := a.Route(rec.Host)
+ if err != nil {
+ return recoveryProbeWaiting, fmt.Errorf("read the registry row: %w", err)
+ }
+ if rt == nil || rt.Host != rec.Host {
+ return recoveryProbeWaiting, fmt.Errorf("%w: %s has no registry row on this box", hostbox.ErrChanged, rec.Host)
+ }
+ in, err := a.Current(ctx, s, rec)
+ if err != nil {
+ return recoveryProbeWaiting, fmt.Errorf("select the recovery inputs: %w", err)
+ }
+ if in.Release == "" {
+ return recoveryProbeWaiting, errRecoveryInputsNoRelease
+ }
+ // The record precedes every render and launch of this wake.
+ if err := req.Record(ctx, in); err != nil {
+ return recoveryProbeWaiting, fmt.Errorf("record the recovery inputs: %w", err)
+ }
+ recorded := in
+ recorded.BundleETags = make(map[string]string, len(in.BundleETags))
+ for k, v := range in.BundleETags {
+ recorded.BundleETags[k] = v
+ }
+ err = a.Wake(ctx, *rt, activator.StartPolicy{MaxStarts: recoveryProbeStartsPerWake, Window: recoveryProbeWindow, Probe: true, Inputs: &recorded})
+ switch {
+ case err == nil:
+ return recoveryProbeAnswered, nil
+ case errors.Is(err, ErrRecoveryProbeExhausted):
+ return recoveryProbeExhausted, nil
+ case errors.Is(err, moves.ErrAttemptOver), errors.Is(err, hostbox.ErrChanged):
+ // This wake's window lapsed or its admission no longer holds: the
+ // next pass probes again, and only the probe.
+ return recoveryProbeWaiting, nil
+ }
+ return recoveryProbeWaiting, fmt.Errorf("the recovery probe wake: %w", err)
+}
+
+// recoveryProbeMatches checks the request against its record: this
+// rollback, rolling back with the source at probing, the nine-attempt
+// budget and the coverage and generation of its return.
+func recoveryProbeMatches(rec provision.Record, req recoveryProbeRequest) error {
+ m := rec.Move
+ if m == nil || m.ID == "" || m.State != provision.MoveRollingBack || m.Rollback == nil || m.Rollback.ID != req.RollbackID || m.Rollback.SourcePhase != provision.RbSourceProbing {
+ return fmt.Errorf("%w: not rollback %s at probing", errRecoveryProbeRequest, req.RollbackID)
+ }
+ if req.Record == nil || req.Attempts != recoveryProbeAttempts || req.Clean == m.Rollback.Dirty {
+ return fmt.Errorf("%w: budget %d, clean %v", errRecoveryProbeRequest, req.Attempts, req.Clean)
+ }
+ if req.Clean {
+ if req.Coverage != recoveryCoverageClean || req.Generation != "" || req.SourceGeneration == "" {
+ return fmt.Errorf("%w: a clean return admits a fresh generation restored from src_gen", errRecoveryProbeRequest)
+ }
+ } else if req.Coverage != recoveryCoverageSealed || req.Generation != req.RollbackID {
+ return fmt.Errorf("%w: a dirty return admits its sealed return generation", errRecoveryProbeRequest)
+ }
+ return nil
+}
+
+// admits checks the fresh claim: this move's restoring op for this
+// rollback, at probing, executed by this box under the record's
+// incarnation, naming the request's src_gen. Anything else waits.
+func (a recoveryProbeAdapter) admits(obs hostclaim.Observed, rec provision.Record, req recoveryProbeRequest) error {
+ c := obs.Claim
+ if obs.Absent || c == nil || c.Holder == nil || c.Op.Kind != hostclaim.KindRestoring || c.Op.ID != rec.Move.ID || c.Op.Intent["rollback"] != req.RollbackID ||
+ c.Op.Phase != hostclaim.PhaseProbing || c.Op.Exec != a.Box || c.Holder.Incarnation != rec.Move.Incarnation {
+ return fmt.Errorf("%w: the claim is not rollback %s's probe on %s", hostbox.ErrChanged, req.RollbackID, a.Box)
+ }
+ if req.SourceGeneration != "" && c.Op.Journal[hostclaim.JournalSrcGen] != req.SourceGeneration {
+ return fmt.Errorf("%w: the claim's src_gen is not the request's", hostbox.ErrChanged)
+ }
+ return nil
+}
+```
+
+#### New file: `internal/host/plan10_j2_adapter_test.go`
+
+```go
+package host
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os/exec"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/moves"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// plan10J2AdapterInputs are what the source would render now.
+func plan10J2AdapterInputs() provision.MoveInputs {
+ return provision.MoveInputs{BundleETags: map[string]string{"default": "b1"}, OverlayETag: "o1", Release: "v7", EnvReport: "env"}
+}
+
+// plan10J2AdapterRecord is the rolling_back record of the recovery fixture's
+// claim: rollback plan10J2Rollback with the source at probing.
+func plan10J2AdapterRecord(incarnation string, dirty bool) provision.Record {
+ return provision.Record{Account: "bab", App: "jam", Host: recoveryHost, Move: &provision.MoveState{ID: "m1", Incarnation: incarnation, From: "ie", To: "uk", State: provision.MoveRollingBack,
+ Rollback: &provision.MoveRollback{ID: plan10J2Rollback, Dirty: dirty, SourcePhase: provision.RbSourceProbing}}}
+}
+
+// plan10J2AdapterRequest is the request movesrecover.go's probing builds,
+// recording into rec through the real provision transition.
+func plan10J2AdapterRequest(rec *provision.Record, srcGen string, events *[]string) recoveryProbeRequest {
+ g := provision.MoveGuard{ID: rec.Move.ID, Incarnation: rec.Move.Incarnation}
+ req := recoveryProbeRequest{RollbackID: rec.Move.Rollback.ID, Clean: !rec.Move.Rollback.Dirty, Attempts: recoveryProbeAttempts, SourceGeneration: srcGen,
+ Record: func(_ context.Context, in provision.MoveInputs) error {
+ *events = append(*events, "record")
+ return provision.MoveRecordRecoveryInputs(g, rec.Move.Rollback.ID, in)(rec)
+ }}
+ if rec.Move.Rollback.Dirty {
+ req.Coverage, req.Generation = recoveryCoverageSealed, rec.Move.Rollback.ID
+ } else {
+ req.Coverage = recoveryCoverageClean
+ }
+ return req
+}
+
+// plan10J2AdapterClaim puts the fixture's claim at restoring/probing with
+// box-1 executing and returns the incarnation.
+func plan10J2AdapterClaim(t *testing.T, f *recoveryFixture, phase, srcGen string) string {
+ t.Helper()
+ o, err := hostclaim.Update(t.Context(), f.claims, f.read(), func(c *hostclaim.Claim) error {
+ c.Op = plan10J2Op(hostclaim.KindRestoring, phase)
+ c.Op.Journal = map[string]string{hostclaim.JournalSrcGen: srcGen}
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return o.Claim.Holder.Incarnation
+}
+
+// The adapter records the recovery inputs at probing before anything is
+// rendered, then asks the activator for a probe-shaped wake carrying exactly
+// those inputs. That wake runs the real Group D path: it is admitted at
+// probing as a recovery probe, and Prepare binds the operation attempts and
+// probe readiness to the admitted session. The held render (Materialize)
+// follows the record.
+func TestPlan10J2AdapterRecordsInputsBeforeTheGroupDProbe(t *testing.T) {
+ for _, dirty := range []bool{false, true} {
+ t.Run(fmt.Sprintf("dirty=%v", dirty), func(t *testing.T) {
+ f := newRecoveryFixture(t)
+ inc := plan10J2AdapterClaim(t, f, hostclaim.PhaseProbing, "src-gen")
+ c := f.controller("boot-1", "controller-1", 4101, "start-1")
+ ops := &plan9D2Attempts{}
+ c.wake.Inputs, c.wake.Continue, c.wake.Operations = plan9D2Inputs{recoveryInputs{f}}, hostclaim.SessionContinues, ops
+ c.wake.ProbeReady = func(context.Context, *exec.Cmd, registry.Route, ClaimLaunchInputs, string) error { return nil }
+ rec := plan10J2AdapterRecord(inc, dirty)
+ var events []string
+ f.materialize = func(context.Context, hostclaim.Observed) error {
+ events = append(events, "render")
+ if rec.Move.Rollback.RecoveryInputs == nil {
+ t.Error("rendered before the inputs were recorded")
+ }
+ return nil
+ }
+ var policies []activator.StartPolicy
+ wake := func(ctx context.Context, rt registry.Route, p activator.StartPolicy) error {
+ events = append(events, "wake")
+ policies = append(policies, p)
+ if rec.Move.Rollback.RecoveryInputs == nil || p.Inputs == nil || !reflect.DeepEqual(*p.Inputs, *rec.Move.Rollback.RecoveryInputs) {
+ t.Errorf("the wake does not carry the recorded inputs: %+v / %+v", p.Inputs, rec.Move.Rollback.RecoveryInputs)
+ }
+ if err := c.wake.admitsOperation(f.read(), p, "box-1"); err != nil {
+ return err
+ }
+ // Prerequisites' S3 lease stands in, as plan10J2Recovery does.
+ live, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ done := make(chan struct{})
+ go func() { <-live.Done(); close(done) }()
+ o := f.read()
+ c.wake.mu.Lock()
+ c.wake.pending[rt.Host] = &claimWakeAttempt{observed: o, held: recoveryHeld{}, ctx: live, cancel: cancel, done: done}
+ c.wake.mu.Unlock()
+ admitted, err := c.runtime.Admit(ctx, rt, o, p)
+ if err != nil {
+ return err
+ }
+ events = append(events, "admit")
+ s, err := c.runtime.Prepare(ctx, rt, admitted, p)
+ if err != nil {
+ return err
+ }
+ if _, err := s.OpenProbe(ctx, time.Now()); err != nil {
+ return err
+ }
+ if err := s.Reserve(ctx, 0); err != nil {
+ return err
+ }
+ return s.Materialize(ctx)
+ }
+ a := recoveryProbeAdapter{Box: "box-1", Claims: true, Wake: wake,
+ Route: func(host string) (*registry.Route, error) { return &registry.Route{Host: host}, nil },
+ Current: func(context.Context, release.Store, provision.Record) (provision.MoveInputs, error) {
+ return plan10J2AdapterInputs(), nil
+ }}
+ out, err := a.probe(t.Context(), f.claims, rec, plan10J2AdapterRequest(&rec, "src-gen", &events))
+ if err != nil || out != recoveryProbeAnswered {
+ t.Fatalf("probe: %v %v", out, err)
+ }
+ if got := strings.Join(events, ","); got != "record,wake,admit,render" {
+ t.Fatalf("order: %s", got)
+ }
+ if len(policies) != 1 || !policies[0].Probe || policies[0].MaxStarts != recoveryProbeStartsPerWake || policies[0].Window != 30*time.Second {
+ t.Fatalf("policy: %+v", policies)
+ }
+ if got := strings.Join(ops.calls, ","); got != "open,reserve-0" {
+ t.Fatalf("attempt journal calls: %s", got)
+ }
+ for _, s := range ops.starts {
+ if !s.Policy.Probe || s.Admitted.Claim.Op.Kind != hostclaim.KindRestoring || s.Admitted.Claim.Op.Phase != hostclaim.PhaseProbing || !reflect.DeepEqual(*s.Policy.Inputs, plan10J2AdapterInputs()) {
+ t.Fatalf("attempt bound to %+v", s)
+ }
+ }
+ })
+ }
+}
+
+// A failed record write, inputs naming no release, or a claim that is not
+// this rollback's probe at probing by this box stops the attempt before any
+// render or wake.
+func TestPlan10J2AdapterStopsBeforeTheWake(t *testing.T) {
+ type tc struct {
+ claimPhase, exec, srcGen string
+ in provision.MoveInputs
+ recordErr error
+ mutateReq func(*recoveryProbeRequest)
+ mutateRec func(*provision.Record)
+ want error
+ }
+ recordFailed := errors.New("the record write failed")
+ for name, c := range map[string]tc{
+ "record write fails": {recordErr: recordFailed, want: recordFailed},
+ "no release": {in: provision.MoveInputs{OverlayETag: "o1"}, want: errRecoveryInputsNoRelease},
+ "claim at claimed": {claimPhase: hostclaim.PhaseClaimed, want: hostbox.ErrChanged},
+ "claim at probed": {claimPhase: hostclaim.PhaseProbed, want: hostbox.ErrChanged},
+ "other executor": {exec: "box-2", want: hostbox.ErrChanged},
+ "other rollback": {mutateReq: func(r *recoveryProbeRequest) { r.RollbackID = "rb_zzzzzzzzzzzzzzzzzzzzzzzzzz" }, want: errRecoveryProbeRequest},
+ "record not probing": {mutateRec: func(r *provision.Record) { r.Move.Rollback.SourcePhase = provision.RbSourceProbed }, want: errRecoveryProbeRequest},
+ "budget not nine": {mutateReq: func(r *recoveryProbeRequest) { r.Attempts = 3 }, want: errRecoveryProbeRequest},
+ "clean as sealed": {mutateReq: func(r *recoveryProbeRequest) { r.Coverage = recoveryCoverageSealed }, want: errRecoveryProbeRequest},
+ "clean with a gen": {mutateReq: func(r *recoveryProbeRequest) { r.Generation = plan10J2Rollback }, want: errRecoveryProbeRequest},
+ "src_gen differs": {srcGen: "other-gen", want: hostbox.ErrChanged},
+ "no record hook": {mutateReq: func(r *recoveryProbeRequest) { r.Record = nil }, want: errRecoveryProbeRequest},
+ } {
+ t.Run(name, func(t *testing.T) {
+ f := newRecoveryFixture(t)
+ phase := c.claimPhase
+ if phase == "" {
+ phase = hostclaim.PhaseProbing
+ }
+ srcGen := c.srcGen
+ if srcGen == "" {
+ srcGen = "src-gen"
+ }
+ inc := plan10J2AdapterClaim(t, f, phase, srcGen)
+ if c.exec != "" {
+ if _, err := hostclaim.Update(t.Context(), f.claims, f.read(), func(cl *hostclaim.Claim) error { cl.Op.Exec = c.exec; return nil }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ rec := plan10J2AdapterRecord(inc, false)
+ var events []string
+ req := plan10J2AdapterRequest(&rec, "src-gen", &events)
+ if c.recordErr != nil {
+ req.Record = func(context.Context, provision.MoveInputs) error {
+ events = append(events, "record")
+ return c.recordErr
+ }
+ }
+ if c.mutateReq != nil {
+ c.mutateReq(&req)
+ }
+ if c.mutateRec != nil {
+ c.mutateRec(&rec)
+ }
+ in := c.in
+ if in.Release == "" && in.OverlayETag == "" {
+ in = plan10J2AdapterInputs()
+ }
+ a := recoveryProbeAdapter{Box: "box-1", Claims: true,
+ Wake: func(context.Context, registry.Route, activator.StartPolicy) error {
+ events = append(events, "wake")
+ return nil
+ },
+ Route: func(host string) (*registry.Route, error) { return &registry.Route{Host: host}, nil },
+ Current: func(context.Context, release.Store, provision.Record) (provision.MoveInputs, error) { return in, nil }}
+ out, err := a.probe(t.Context(), f.claims, rec, req)
+ if !errors.Is(err, c.want) || out != recoveryProbeWaiting {
+ t.Fatalf("got %v %v, want %v", out, err, c.want)
+ }
+ for _, e := range events {
+ if e == "wake" {
+ t.Fatalf("woke: %v", events)
+ }
+ }
+ if c.recordErr == nil && len(events) != 0 {
+ t.Fatalf("recorded: %v", events)
+ }
+ })
+ }
+}
+
+// Without the host claim or the activator's recovery probe entry the
+// adapter is a typed dependency refusal: nothing is read, recorded or woken.
+func TestPlan10J2AdapterRefusesWithoutItsDependencies(t *testing.T) {
+ route := func(host string) (*registry.Route, error) { return &registry.Route{Host: host}, nil }
+ current := func(context.Context, release.Store, provision.Record) (provision.MoveInputs, error) {
+ return plan10J2AdapterInputs(), nil
+ }
+ wake := func(context.Context, registry.Route, activator.StartPolicy) error { return nil }
+ for name, a := range map[string]recoveryProbeAdapter{
+ "runtime unavailable": {Box: "box-1", Claims: true, Wake: wake, Route: route, Current: current, Ready: func() bool { return false }},
+ "claim off": {Box: "box-1", Claims: false, Wake: wake, Route: route, Current: current},
+ "no wake entry": {Box: "box-1", Claims: true, Route: route, Current: current},
+ "no route": {Box: "box-1", Claims: true, Wake: wake, Current: current},
+ "no inputs": {Box: "box-1", Claims: true, Wake: wake, Route: route},
+ } {
+ t.Run(name, func(t *testing.T) {
+ rec := plan10J2AdapterRecord("inc", false)
+ var events []string
+ out, err := a.probe(t.Context(), release.NewDirStore(t.TempDir()), rec, plan10J2AdapterRequest(&rec, "src-gen", &events))
+ var refusal *ClaimDependencyError
+ if !errors.As(err, &refusal) || out != recoveryProbeWaiting || len(events) != 0 || rec.Move.Rollback.RecoveryInputs != nil {
+ t.Fatalf("%v %v %v", out, err, events)
+ }
+ })
+ }
+}
+
+// The wake's outcome maps onto the recovery's: answered goes to probed, a
+// spent nine-attempt budget stops for an operator, a lapsed window or a
+// changed claim waits for the next pass, anything else is an error.
+func TestPlan10J2AdapterOutcomes(t *testing.T) {
+ other := errors.New("boom")
+ for name, tc := range map[string]struct {
+ wakeErr error
+ out recoveryProbeOutcome
+ err error
+ }{
+ "answered": {nil, recoveryProbeAnswered, nil},
+ "exhausted": {fmt.Errorf("reserve: %w", ErrRecoveryProbeExhausted), recoveryProbeExhausted, nil},
+ "window": {moves.ErrAttemptOver, recoveryProbeWaiting, nil},
+ "changed": {hostbox.ErrChanged, recoveryProbeWaiting, nil},
+ "other": {other, recoveryProbeWaiting, other},
+ } {
+ t.Run(name, func(t *testing.T) {
+ f := newRecoveryFixture(t)
+ rec := plan10J2AdapterRecord(plan10J2AdapterClaim(t, f, hostclaim.PhaseProbing, "src-gen"), false)
+ var events []string
+ a := recoveryProbeAdapter{Box: "box-1", Claims: true,
+ Wake: func(context.Context, registry.Route, activator.StartPolicy) error { return tc.wakeErr },
+ Route: func(host string) (*registry.Route, error) { return &registry.Route{Host: host}, nil },
+ Current: func(context.Context, release.Store, provision.Record) (provision.MoveInputs, error) {
+ return plan10J2AdapterInputs(), nil
+ }}
+ out, err := a.probe(t.Context(), f.claims, rec, plan10J2AdapterRequest(&rec, "src-gen", &events))
+ if out != tc.out || (tc.err == nil) != (err == nil) || (tc.err != nil && !errors.Is(err, tc.err)) {
+ t.Fatalf("got %v %v", out, err)
+ }
+ })
+ }
+}
+
+// An unavailable wake entry keeps the recovery at probing without recording
+// inputs. The production binding itself is checked below with both flags.
+func TestPlan10J2MissingWakeKeepsRecoveryAtProbing(t *testing.T) {
+ cfg := moveCfg{Box: "ie-box", claimRecords: true, Route: func(string) (*registry.Route, error) { return nil, nil }}
+ w := &movesWorker{mirrors: map[string]lease.MirrorStore{}}
+ armRecoveryProbe(&cfg, w, nil, nil)
+ if cfg.recoveryProbe == nil {
+ t.Fatal("the production recovery probe hook is nil")
+ }
+ h := newRecoverHarness(t, false)
+ h.released(t)
+ h.sw = &movesWorker{mirrors: map[string]lease.MirrorStore{}}
+ for i := 0; i < 2; i++ {
+ c := h.sourceCfg(t)
+ c.claimRecords = true
+ armRecoveryProbe(&c, h.sw, nil, nil)
+ h.sw.cfg = c
+ if err := movesPass(context.Background(), h.s, h.sw); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if rb := h.sourceAt(t, provision.RbSourceProbing); rb.RecoveryInputs != nil {
+ t.Fatalf("recorded without a wake entry: %+v", rb.RecoveryInputs)
+ }
+}
+
+// With a wake entry the same production adapter drives the real recovery
+// pass: each return records the inputs it renders at probing, before the
+// wake, and the recovery goes on to opened.
+func TestPlan10J2AdapterDrivesTheRecoveryPass(t *testing.T) {
+ for _, dirty := range []bool{false, true} {
+ t.Run(fmt.Sprintf("dirty=%v", dirty), func(t *testing.T) {
+ h := newRecoverHarness(t, dirty)
+ h.released(t)
+ h.sw = &movesWorker{mirrors: map[string]lease.MirrorStore{}}
+ var wakes []activator.StartPolicy
+ wake := func(ctx context.Context, rt registry.Route, p activator.StartPolicy) error {
+ rb := h.read(t, p1Host).Move.Rollback
+ if rb.SourcePhase != provision.RbSourceProbing || rb.RecoveryInputs == nil || p.Inputs == nil || !reflect.DeepEqual(*rb.RecoveryInputs, *p.Inputs) {
+ t.Errorf("woke before the record: %+v / %+v", rb, p.Inputs)
+ }
+ c := h.claim(t)
+ if rt.Host != p1Host || c.Op.Kind != hostclaim.KindRestoring || c.Op.Phase != hostclaim.PhaseProbing || c.Op.Exec != "ie-box" {
+ t.Errorf("woke %s with the claim at %+v", rt.Host, c.Op)
+ }
+ wakes = append(wakes, p)
+ return nil
+ }
+ for i := 0; i < 3; i++ {
+ c := h.sourceCfg(t)
+ c.claimRecords = true
+ armRecoveryProbe(&c, h.sw, wake, nil)
+ a := c.recoveryProbe
+ c.recoveryProbe = func(ctx context.Context, s release.Store, rec provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ h.probes = append(h.probes, req)
+ return a(ctx, s, rec, req)
+ }
+ h.sw.cfg = c
+ h.sw.cfg.LaunchEnv = func(string) (map[string]string, []string, error) { return map[string]string{"A": "1"}, nil, nil }
+ h.sw.cfg.InputVersions = func(context.Context, release.Store, provision.Record) (map[string]string, string, error) {
+ in := h.renderedInputs()
+ return in.BundleETags, in.OverlayETag, nil
+ }
+ if err := movesPass(context.Background(), h.s, h.sw); err != nil {
+ t.Fatal(err)
+ }
+ }
+ h.opened(t)
+ if len(wakes) != 1 || !wakes[0].Probe || len(h.probes) != 1 {
+ t.Fatalf("wakes %+v probes %d", wakes, len(h.probes))
+ }
+ req := h.probes[0]
+ if req.Clean == dirty || req.Attempts != recoveryProbeAttempts || (dirty && (req.Coverage != recoveryCoverageSealed || req.Generation == "")) || (!dirty && (req.Coverage != recoveryCoverageClean || req.SourceGeneration == "")) {
+ t.Fatalf("request %+v", req)
+ }
+ })
+ }
+}
+
+func TestPlan10J2ArmPhase1ConnectsRecoveryOnlyWithClaimFlag(t *testing.T) {
+ for _, enabled := range []bool{false, true} {
+ t.Run(fmt.Sprint(enabled), func(t *testing.T) {
+ cfg := moveCfg{claimRecords: enabled}
+ w := &movesWorker{mirrors: map[string]lease.MirrorStore{}}
+ armPhase1(&cfg, w, activator.Config{}, t.TempDir(), nil, &activator.Service{}, nil, hostwrite.Config{Enabled: enabled}, func(string) string { return "" })
+ if (cfg.recoveryProbe != nil) != enabled {
+ t.Fatal("production recovery hook does not follow claim flag")
+ }
+ })
+ }
+}
+```
+
+### Task B: let hostresolve dispatch declaration policy to its registered owner
+
+Invariant: operation policy cannot bypass the signed challenge, exact claim/record observation, global quarantine or durable loss sequence. An unregistered move kind acquires no declaration authority. Accepted evidence and write-loss sources never change on retry.
+
+Apply the tests before the implementation. The expected initial failure names the missing `OperationOwner` and `Service.Owners`. Tests then force failures before preparation, at the claim write, during committed-loss bookkeeping and during move completion. They retry through the public service and check the retained declaration rather than accepting a successful callback as proof. A separate test changes the current claim kind after success and proves that the original registered owner still receives completion.
+
+The interface at the top of this plan is the contract. Preserve the existing owner for serving, parking and deleting; the registration map does not override it. The signed loss record freezes `WriteSources` before the roster is changed. Malformed source tuples refuse before acceptance. Unknown or incomplete timer evidence remains `TimersUnknown`; compute fencing is never treated as disk destruction or a timer inventory.
+
+The service's returned observation identifies its declaration CAS. `FinishLoss` may advance the move further; callers needing that final state read it through the ordinary record API. Neither hook is allowed to invent a new holder or reuse the request as a blanket lifecycle context.
+
+```bash
+python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" B tests
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostresolve -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10|TestLostWiring|TestFailedClaim|TestDeclareCrash|TestPreparedLoss'
+# After recording the expected red:
+python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" B implementation
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostresolve -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10|TestLostWiring|TestFailedClaim|TestDeclareCrash|TestPreparedLoss'
+```
+
+#### New file: `internal/hostresolve/operation_owner_test.go`
+
+```go
+package hostresolve
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "reflect"
+ "slices"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfence"
+)
+
+type plan10LossOwner struct {
+ service *Service
+ store *failClaimStore
+ calls []string
+ failPrepare, failFinish, mutate bool
+ sources []hostclaim.WriteSource
+}
+
+func (p *plan10LossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, record hostfence.RecordObservation, box string) (hostfence.Consequences, hostclaim.LossConsequences, error) {
+ if o.Claim.Op.Kind != hostclaim.KindMoving || box != "dead" || o.Claim.Op.Intent["target"] != box || record.Key != o.Claim.Holder.Key {
+ return hostfence.Consequences{}, hostclaim.LossConsequences{}, ErrResolve
+ }
+ actual := hostclaim.LossConsequences{RemoveParticipation: true, WritesPotentiallyLost: true, TimersPotentiallyLost: true, RecoveryIntent: json.RawMessage(`{"kind":"test-move-loss"}`)}
+ if p.mutate {
+ o.Claim.Op.Intent["target"] = "wrong"
+ o.Claim.Holder.Stamp++
+ }
+ return hostfence.Consequences{Transition: "test-move-loss", PossibleWriteLoss: true, PossibleTimerLoss: true}, actual, nil
+}
+func (p *plan10LossOwner) LossWriteSources(context.Context, hostclaim.Observed, hostclaim.LossConsequences) ([]hostclaim.WriteSource, error) {
+ return p.sources, nil
+}
+func (p *plan10LossOwner) ValidateDeclareLost(c hostclaim.Claim, d hostclaim.LostDeclaration) error {
+ if c.Op.Kind != hostclaim.KindMoving || c.Op.Intent["target"] != d.Box || d.Box != "dead" || !d.Consequences.RemoveParticipation {
+ return ErrResolve
+ }
+ return nil
+}
+func (p *plan10LossOwner) PrepareLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
+ p.calls = append(p.calls, "prepare")
+ roster, err := p.service.Directory.Read(ctx)
+ if err != nil || roster.Roster.Boxes[loss.Box].State != "quarantined" {
+ return ErrResolve
+ }
+ durable, err := hostclaim.ReadLossRecord(ctx, p.store, p.service.Directory.PublicKey, loss.ID)
+ if err != nil || durable.Record.State != "prepared" || slices.Contains(o.Claim.LossRefs, loss.ID) {
+ return ErrResolve
+ }
+ if p.failPrepare {
+ return ErrResolve
+ }
+ if p.mutate {
+ o.Claim.Op.Intent["target"] = "wrong"
+ loss.ConfirmedConsequences.RecoveryIntent[0] = 'x'
+ }
+ return nil
+}
+func (p *plan10LossOwner) FinishLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
+ p.calls = append(p.calls, "finish")
+ durable, err := hostclaim.ReadLossRecord(ctx, p.store, p.service.Directory.PublicKey, loss.ID)
+ if err != nil || durable.Record.State != "committed" || !slices.Contains(o.Claim.LossRefs, loss.ID) {
+ return ErrResolve
+ }
+ if p.failFinish {
+ return ErrResolve
+ }
+ if p.mutate {
+ o.Claim.Op.Intent["target"] = "wrong"
+ loss.ConfirmedConsequences.RecoveryIntent[0] = 'x'
+ }
+ return nil
+}
+func plan10ResolveMove(t *testing.T) (Service, *plan10LossOwner, hostfence.Submission, string) {
+ t.Helper()
+ s, store, now, host := resolveFixture(t)
+ observed, err := hostclaim.Read(t.Context(), store, host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, err = hostclaim.Update(t.Context(), store, observed, func(c *hostclaim.Claim) error {
+ c.Op = hostclaim.Op{ID: "move-test", Kind: hostclaim.KindMoving, Phase: hostclaim.PhaseMoving, Exec: "live", Participants: []string{"live", "dead"}, Intent: map[string]string{"source": "live", "target": "dead", "to": "de"}}
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ owner := &plan10LossOwner{store: store}
+ s.Owners = map[string]OperationOwner{hostclaim.KindMoving: owner}
+ owner.service = &s
+ return s, owner, submitted(t, s, now, host), host
+}
+func TestPlan10MoveOwnerRequiredBeforeAcceptance(t *testing.T) {
+ s, p, in, host := plan10ResolveMove(t)
+ s.Owners = nil
+ before, _ := s.Directory.Read(t.Context())
+ if _, err := s.Inspect(t.Context(), "operator", host, "dead"); !errors.Is(err, ErrResolve) {
+ t.Fatal(err)
+ }
+ if _, err := s.Declare(t.Context(), "operator", host, in); !errors.Is(err, ErrResolve) {
+ t.Fatal(err)
+ }
+ after, _ := s.Directory.Read(t.Context())
+ if after.ETag != before.ETag {
+ t.Fatal("missing owner quarantined")
+ }
+ keys, err := p.store.List(t.Context(), "control/host-loss-acceptances/")
+ if err != nil || len(keys) != 0 {
+ t.Fatal(keys, err)
+ }
+}
+func TestPlan10OwnerHookOrderAndReplay(t *testing.T) {
+ for _, fault := range []string{"prepare", "claim", "bookkeeping", "finish"} {
+ t.Run(fault, func(t *testing.T) {
+ s, p, in, host := plan10ResolveMove(t)
+ p.failPrepare = fault == "prepare"
+ p.store.fail = fault == "claim"
+ p.failFinish = fault == "finish"
+ bookkeeping := &lossBookkeepingStore{Store: p.store, fail: fault == "bookkeeping"}
+ s.Directory.Store = bookkeeping
+ if _, err := s.Declare(t.Context(), "operator", host, in); err == nil {
+ t.Fatal("fault not reached")
+ }
+ p.failPrepare = false
+ p.store.fail = false
p.failFinish = false
bookkeeping.fail = false
if _, err := s.Declare(t.Context(), "operator", host, in); err != nil {
@@ -1393,7 +3128,7 @@ func ownerHook(ctx context.Context, owner OperationOwner, o hostclaim.Observed,
Invariant: a console can change only the observed move's named transition, with its exact admission, holder and record identities checked at the write. An action token grants no unrelated record, app, SQL, claim or storage mutation. Accepted work can drain after admission closes; a successor registration or incarnation cannot borrow its authority.
-Apply the tests first. The expected failures identify the missing typed console move methods; the door tests then observe HC5-N1 until their typed writer is connected. Keep HC5-N1 until their implementation and wiring are present. `ObserveConsoleMove` captures account, app, app ID, host, incarnation, move ID, claim ETag, record ETag and `record_seq`. The grant is one `ControlWriter.ConsoleMove` call with a closed `ConsoleMoveAction`; it returns no reusable context permit or raw store. Cut, abort and retry each select a typed provision transition. Retry resumes the recorded recovery state and clears the stuck reason; it never bypasses verification, resets a session by fiat or completes a probe.
+Apply the tests first. Against rev 1, the new HTTP race tests return 202 and mutate the successor; after the fix they return the existing move-changed 409 without changing the claim, record or serial. The tests independently replace the move ID, incarnation and record sequence between `moveOperatorGate` and `ObserveConsoleMove` for cut, abort and retry. On the pinned base the expected failures identify the missing typed console move methods; the door tests then observe HC5-N1 until their typed writer is connected. Keep HC5-N1 until their implementation and wiring are present. `ObserveConsoleMove` captures account, app, app ID, host, incarnation, move ID, claim ETag, record ETag and `record_seq`. The grant is one `ControlWriter.ConsoleMove` call with a closed `ConsoleMoveAction`; it returns no reusable context permit or raw store. Cut, abort and retry each select a typed provision transition. Retry resumes the recorded recovery state and clears the stuck reason; it never bypasses verification, resets a session by fiat or completes a probe. Before calling `ControlWriter.ConsoleMove`, `claimConsoleMove` compares the fresh observation to the gate-read record's `Move.ID`, `Move.Incarnation` and `RecordSeq`. Reading a coherent successor is a conflict with this request, not permission to retarget it.
```bash
python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" C tests
@@ -2205,40 +3940,19 @@ func TestJ17ReviewClaimCheckedAtTheFinalWrite(t *testing.T) {
```diff
--- a/internal/console/apiinstancemove.go
+++ b/internal/console/apiinstancemove.go
-@@ -22,6 +22,7 @@
- "github.com/carlosframework/platform/internal/objstore"
- "github.com/carlosframework/platform/internal/provision"
- "github.com/carlosframework/platform/internal/registry"
+@@ -24,0 +25 @@
+ "github.com/carlosframework/platform/internal/release"
- "github.com/carlosframework/platform/internal/svcrole"
- )
-
-@@ -818,6 +819,10 @@
- jsonError(w, http.StatusConflict, "this move was started by an older console and has no id. abort it and request it again")
- return
- }
+@@ -820,0 +822,4 @@
+ if s.cfg.HostClaim {
+ s.claimConsoleMove(w, r, acct, app, host, rec, hostops.ConsoleMoveCut)
+ return
+ }
- s.moveTransitionP1(w, r, acct, app, host, provision.MoveGuardedCut(moveGuardOf(rec)), http.StatusAccepted)
- }
-
-@@ -838,6 +843,10 @@
- }
- if rec.Move.Legacy() {
- s.moveTransitionP1(w, r, acct, app, host, legacyAbortOf(*rec.Move), http.StatusAccepted)
+@@ -840,0 +846,4 @@
+ return
+ }
+ if s.cfg.HostClaim {
+ s.claimConsoleMove(w, r, acct, app, host, rec, hostops.ConsoleMoveAbort)
- return
- }
- s.moveTransitionP1(w, r, acct, app, host, provision.MoveGuardedAbort(moveGuardOf(rec)), http.StatusAccepted)
-@@ -873,11 +882,61 @@
- jsonError(w, http.StatusConflict, "nothing to resolve yet")
- return
- }
+@@ -876,4 +885,7 @@
- // The record write is a console move write, so on a host-claim console
- // moveTransitionSaying refuses it with HC5-N1 before any write (ruling
- // J17). Claim Plan 10's console move grant (platform-1e accepted J17)
@@ -2250,7 +3964,7 @@ func TestJ17ReviewClaimCheckedAtTheFinalWrite(t *testing.T) {
+ s.claimConsoleMove(w, r, acct, app, host, rec, hostops.ConsoleMoveRetry)
+ return
+ }
- s.moveTransitionP1(w, r, acct, app, host, provision.MoveRetry(moveGuardOf(rec)), http.StatusAccepted)
+@@ -880,0 +893,52 @@
+}
+
+// claimConsoleMove is the cut, abort and retry doors on a host-claim console
@@ -2272,6 +3986,11 @@ func TestJ17ReviewClaimCheckedAtTheFinalWrite(t *testing.T) {
+ ctx := r.Context()
+ o, err := hostops.ObserveConsoleMove(ctx, s.cfg.Releases, acct.Sqid, app, host)
+ var got *provision.Record
++ if err == nil && (o.OpID != rec.Move.ID || o.Incarnation != rec.Move.Incarnation || o.RecordSeq != rec.RecordSeq) {
++ // The gate read a different move, incarnation or record than the
++ // claim and record observed now: something replaced it in between.
++ err = hostops.ErrIdentity
++ }
+ if err == nil {
+ got, err = a.Control.ConsoleMove(ctx, o, action)
+ }
@@ -2298,9 +4017,6 @@ func TestJ17ReviewClaimCheckedAtTheFinalWrite(t *testing.T) {
+ log.Printf("console: api instance move %s/%s/%s: typed move write: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not write the move. try again")
+ }
- }
-
- // registerMovesPhase1Routes adds the routes that exist only with the switch
```
#### New file: `internal/hostops/console_move_grant.go`
@@ -2516,6 +4232,249 @@ func (w *ControlWriter) ConsoleMove(ctx context.Context, o ConsoleMoveObservatio
}
```
+#### New file: `internal/console/apiinstancemove_j17_rev2_test.go`
+
+```go
+package console
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostops"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// J17 rev2: the typed door binds the fresh observation to the move and record
+// its operator gate read. A store wrapper runs a replacement between the two
+// reads, on the real door.
+
+// j17RaceStore runs hook once, just before the door's first read of the
+// host claim (the fresh ObserveConsoleMove, after the gate's record read).
+type j17RaceStore struct {
+ release.Store
+ key string
+ hook func()
+}
+
+func (s *j17RaceStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ if key == s.key && s.hook != nil {
+ h := s.hook
+ s.hook = nil
+ h()
+ }
+ return s.Store.Get(ctx, key)
+}
+
+func j17Arm(fx *consoleFixture, hook func()) {
+ fx.srv.cfg.Releases = &j17RaceStore{Store: fx.srv.cfg.Releases, key: "control/hosts/acme.go.tito.io.json", hook: hook}
+}
+
+// j17Replace swaps the claim's op, the record's move and the registration to
+// a successor move id, as a finished move and a new request would.
+func j17Replace(t *testing.T, fx *consoleFixture, oldID string) {
+ t.Helper()
+ ctx := context.Background()
+ id := "mv_abcdefghijklmnopqrstuvwxyz"
+ if id == oldID {
+ id = "mv_zyxwvutsrqponmlkjihgfedcba"
+ }
+ obs, err := hostclaim.Read(ctx, fx.rel, "acme.go.tito.io")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := hostclaim.Update(ctx, fx.rel, obs, func(c *hostclaim.Claim) error {
+ c.Op.ID, c.Op.Intent["admission_id"] = id, id
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ j17SetRecord(t, fx, func(r *provision.Record) { r.Move.ID = id })
+ a, err := hostops.ReadAdmission(ctx, fx.rel, "bab", "titogo", claimMoveAppID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a.Value.Registrations[id] = a.Value.Registrations[oldID]
+ b, _ := json.Marshal(a.Value)
+ if _, err := fx.rel.Put(ctx, a.Key, b, a.ETag); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func j17RecordBytes(t *testing.T, fx *consoleFixture) string {
+ t.Helper()
+ b, _, err := fx.rel.Get(context.Background(), provision.RecordKey("bab", "titogo", "acme.go.tito.io"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return string(b)
+}
+
+func j17Serial(t *testing.T, fx *consoleFixture) int64 {
+ t.Helper()
+ d, _, err := release.ReadSerial(context.Background(), fx.rel)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return d.Serial
+}
+
+// j17Raced fires the request with the hook between the gate and the
+// observation and requires the identity 409 with nothing written.
+func j17Raced(t *testing.T, fx *consoleFixture, method, url, token, body string, hook func()) {
+ t.Helper()
+ var claim, rec string
+ j17Arm(fx, func() {
+ hook()
+ claim, rec = claimBytes(t, fx, "acme.go.tito.io"), j17RecordBytes(t, fx)
+ })
+ serial := j17Serial(t, fx)
+ code, _, out := rawReq(t, method, url, token, body)
+ if claim == "" {
+ t.Fatal("the race hook never ran")
+ }
+ if code != http.StatusConflict || errorText(out) != moveChangedSentence {
+ t.Fatalf("raced door = %d %s", code, out)
+ }
+ if claimBytes(t, fx, "acme.go.tito.io") != claim {
+ t.Fatal("the raced door wrote the claim")
+ }
+ if j17RecordBytes(t, fx) != rec {
+ t.Fatal("the raced door mutated the successor record")
+ }
+ if j17Serial(t, fx) != serial {
+ t.Fatal("the raced door bumped the serial")
+ }
+}
+
+func j17RetryReady(t *testing.T, fx *consoleFixture) {
+ t.Helper()
+ ctx := context.Background()
+ obs, err := hostclaim.Read(ctx, fx.rel, "acme.go.tito.io")
+ if err != nil {
+ t.Fatal(err)
+ }
+ rb := "rb_abcdefghijklmnopqrstuvwxyz"
+ if _, err := hostclaim.Update(ctx, fx.rel, obs, func(c *hostclaim.Claim) error {
+ c.Op.Kind, c.Op.Phase = hostclaim.KindRestoring, hostclaim.PhaseProbing
+ c.Op.Intent["rollback"] = rb
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ j17SetRecord(t, fx, func(r *provision.Record) {
+ r.Move.State, r.Move.StuckReason = provision.MoveStuck, provision.StuckRecoveryProbeAttempts
+ r.Move.Rollback = &provision.MoveRollback{ID: rb, TargetPhase: provision.RbTargetReleased, SourcePhase: provision.RbSourceProbing, StuckAt: provision.RbSourceProbing}
+ })
+}
+
+func TestJ17DoorCutRefusesReplacedMove(t *testing.T) {
+ fx, token, moveURL, id := j17StartedMove(t, "code-j17-r2-cut")
+ j17SetRecord(t, fx, j17Ready)
+ j17Raced(t, fx, http.MethodPost, moveURL+"/cut", token, "", func() { j17Replace(t, fx, id) })
+}
+
+func TestJ17DoorAbortRefusesReplacedMove(t *testing.T) {
+ fx, token, moveURL, id := j17StartedMove(t, "code-j17-r2-abort")
+ j17Raced(t, fx, http.MethodDelete, moveURL, token, "", func() { j17Replace(t, fx, id) })
+}
+
+func TestJ17DoorRetryRefusesReplacedMove(t *testing.T) {
+ fx, token, moveURL, id := j17StartedMove(t, "code-j17-r2-retry")
+ j17RetryReady(t, fx)
+ j17Raced(t, fx, http.MethodPost, moveURL+"/resolve", token, `{"retry":true}`, func() { j17Replace(t, fx, id) })
+}
+
+// The same move and incarnation, the record advanced: the fresh observation
+// carries a newer record_seq than the one the gate read.
+func TestJ17DoorRefusesAdvancedRecordSeq(t *testing.T) {
+ advance := func(t *testing.T, fx *consoleFixture) func() {
+ return func() {
+ before := j17RecordBytes(t, fx)
+ j17SetRecord(t, fx, func(r *provision.Record) {})
+ if j17RecordBytes(t, fx) == before {
+ t.Fatal("the record did not advance")
+ }
+ }
+ }
+ t.Run("cut", func(t *testing.T) {
+ fx, token, moveURL, _ := j17StartedMove(t, "code-j17-r2-seq-cut")
+ j17SetRecord(t, fx, j17Ready)
+ j17Raced(t, fx, http.MethodPost, moveURL+"/cut", token, "", advance(t, fx))
+ })
+ t.Run("abort", func(t *testing.T) {
+ fx, token, moveURL, _ := j17StartedMove(t, "code-j17-r2-seq-abort")
+ j17Raced(t, fx, http.MethodDelete, moveURL, token, "", advance(t, fx))
+ })
+ t.Run("retry", func(t *testing.T) {
+ fx, token, moveURL, _ := j17StartedMove(t, "code-j17-r2-seq-retry")
+ j17RetryReady(t, fx)
+ j17Raced(t, fx, http.MethodPost, moveURL+"/resolve", token, `{"retry":true}`, advance(t, fx))
+ })
+}
+
+func TestJ17DoorRefusesReplacedIncarnation(t *testing.T) {
+ for _, action := range []string{"cut", "abort", "retry"} {
+ t.Run(action, func(t *testing.T) {
+ fx, token, url, id := j17StartedMove(t, "code-j17-r2-inc-"+action)
+ method, body := http.MethodDelete, ""
+ switch action {
+ case "cut":
+ j17SetRecord(t, fx, j17Ready)
+ method, url = http.MethodPost, url+"/cut"
+ case "retry":
+ j17RetryReady(t, fx)
+ method, url, body = http.MethodPost, url+"/resolve", `{"retry":true}`
+ }
+ j17Raced(t, fx, method, url, token, body, func() {
+ ctx := context.Background()
+ r, tag, err := provision.GetRecordETag(ctx, fx.rel, "bab", "titogo", "acme.go.tito.io")
+ if err != nil {
+ t.Fatal(err)
+ }
+ r.CreatedAt = r.CreatedAt.Add(time.Second)
+ inc, err := hostclaim.Incarnation(r.Account, r.App, r.Host, r.CreatedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ r.Move.Incarnation = inc
+ b, _ := json.Marshal(r)
+ if _, err := fx.rel.Put(ctx, provision.RecordKey(r.Account, r.App, r.Host), b, tag); err != nil {
+ t.Fatal(err)
+ }
+ o, err := hostclaim.Read(ctx, fx.rel, r.Host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := hostclaim.Update(ctx, fx.rel, o, func(c *hostclaim.Claim) error {
+ c.Holder.Incarnation = inc
+ c.Holder.CreatedAt = r.CreatedAt
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ a, err := hostops.ReadAdmission(ctx, fx.rel, "bab", "titogo", claimMoveAppID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ reg := a.Value.Registrations[id]
+ reg.Incarnation = inc
+ a.Value.Registrations[id] = reg
+ b, _ = json.Marshal(a.Value)
+ if _, err := fx.rel.Put(ctx, a.Key, b, a.ETag); err != nil {
+ t.Fatal(err)
+ }
+ })
+ })
+ }
+}
+```
+
### Task D: connect the move owner to inspect and declare
Invariant: declaring the exact target lost never races a committed record into rollback, never revokes the source, and never reuses an old declaration against a successor. A prepared record keeps the source waiting until both the signed loss and the claim's loss reference establish commitment.
@@ -2526,18 +4485,25 @@ Preparation can arrive after a concurrent retry has finished, or after the loss
If preparation wrote `lost/L1` but L1's claim CAS later conflicts, the source must not be stranded behind a dead reference. A fresh inspection may propose L2 only after verifying that L1 is signed, conflicted, belongs to this exact host/move/incarnation/target/rollback, and never appears in the claim's retained loss references. The new signed intent carries `supersedes` and preserves L1's original target phase, dirty flag and return-lane obligation. `MoveRollbackLossRebind` changes only the exact old reference and loss time at `lost`; the private writer rereads the signed proof at its final conditional write. Neither a prepared nor a committed loss can be replaced. Repeated conflicts preserve the same rollback and original return-lane duty.
-The concrete owner registers only `landing` and `restoring`. A lagging `moving/sealed` claim must catch up through the moves owner's existing `Flip` or `RollBackFlipped` before a fresh inspection; declaration never removes its second participant first. Otherwise `Flip` could no longer journal the source stamp and generation. `moving` and `reverting` remain refused by this adapter; Task B allows a future reviewed owner to register them without weakening the generic fence.
+The concrete owner registers `moving`, `landing`, `reverting` and `restoring` for D5 local-only cleanup. Data-authority target loss remains restricted to `landing` before commit and `restoring` before source takeover. A lagging `moving/sealed` claim must catch up through `Flip` or `RollBackFlipped` before requesting data-authority loss; removing its second participant first would prevent `Flip` from journalling the source stamp and generation. Moving and reverting gain no target-loss or takeover authority.
-The move door first returns the shared signed inspection, then accepts its unchanged challenge and explicit confirmation with compute-fence evidence. It keeps the existing operator gate. An ambiguous retry/declaration request refuses. The declaration owner checks the authoritative record's commit state and exact target, not just the claim's possibly lagging phase. It refuses the source and unrelated participants. Declared target loss cannot be described as certified `clean-return` coverage. The next admission must preserve the loss record's target sessions and any uncovered source-cut session.
+The move door first returns the shared signed inspection, then accepts its unchanged challenge and explicit confirmation with compute-fence evidence. It keeps the existing operator gate. An ambiguous retry/declaration request refuses. The declaration owner checks the authoritative record's commit state and exact target, not just the claim's possibly lagging phase. The data-authority path refuses the source and unrelated participants. Declared target loss cannot be described as certified `clean-return` coverage. The next admission must preserve the loss record's target sessions and any uncovered source-cut session.
Use the existing `MoveRollbackTargetLost` and `MoveRollbackLossCommitted` transitions for the record decision and completion. A recorded `lost` phase is a hold, not permission for the source to restart. Only committed claim loss permits the release. Preserve target-region retirement and any partial return-lane retirement; a fenced box still owes local cleanup unless the operator separately asserted the exact destroyed storage. `MoveRolledBackClose` continues to require `RecoveredAt`, `OpenedAt` and `TargetParkedAt`.
+`ReleaseLostTarget` reads the journalled target stamp after `RollBack` and supplies the target box to `QueueLocationPark` only when `InventoryHasStamp` finds that installation. The local chore keeps its original operation and retirement identity across retries. Recovery may proceed on the source while it remains pending, but `ReleaseMove` cannot close over it. A normal retirement acknowledgement or a new, separately verified D5 declaration may complete it. The first compute-loss declaration never asserts destruction of a chore it has not yet queued.
+
+D5 uses the existing `destroyed-local-only` consequence shape. A move box must owe exact `park-local` chores, must not be the current executor, and must already be declared lost, belong to a committed landing, or no longer participate. Inspection binds the current claim revision and the authoritative move record. Each displayed storage assertion names the chore's original operation, ID and retirement. Declaration requires that exact attestation; `terminated` alone, a missing disk list, another retirement identity or a stale claim/record refuses. An already declared target may take this path in a fresh declaration, with a new quarantine revision.
+
+The pure claim transition validates the owner and signed identities, applies `CompleteObligation` to the exact local chores (including discovery completion), and retains loss evidence. It preserves the entire operation, holder stamp, writer session, admission history, inventory and every other obligation. It neither reassigns regional work nor declares new write/timer losses. Local-only hooks verify the durable declaration and perform no record or claim follow-up write: post-commit cleanup never invokes `MoveRollbackTargetLost`, `ReleaseLostTarget` or `MoveRollbackLossCommitted`. The scoped store continues to reject record writes and a second release CAS for this consequence shape.
+
+
```bash
python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" D tests
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/hostresolve ./internal/console ./internal/hostclaim ./internal/provision -run 'TestJ4|TestPlan10|TestJ17|TestReleaseLostTarget|TestMoveLoss|TestMoveRollbackLossRebind'
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/hostresolve ./internal/console ./internal/hostclaim ./internal/provision ./internal/hostfence -run 'TestJ4|TestPlan10|TestJ17|TestReleaseLostTarget|TestMoveLoss|TestMoveRollbackLossRebind'
# After recording the expected red:
python3 "$CLAIM10_SCRATCH/apply.py" "$CLAIM10_SCRATCH/plan.md" D implementation
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/hostresolve ./internal/console ./internal/hostclaim ./internal/provision -run 'TestJ4|TestPlan10|TestJ17|TestReleaseLostTarget|TestMoveLoss|TestMoveRollbackLossRebind'
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -count=1 ./internal/hostops ./internal/hostresolve ./internal/console ./internal/hostclaim ./internal/provision ./internal/hostfence -run 'TestJ4|TestPlan10|TestJ17|TestReleaseLostTarget|TestMoveLoss|TestMoveRollbackLossRebind'
```
#### New file: `internal/console/apiinstancemove_j4_test.go`
@@ -2913,7 +4879,7 @@ func TestReleaseLostTargetFromLanding(t *testing.T) {
}
for _, o := range c.Obligations {
if o.Kind == "park-local" || o.Box != "" {
- t.Fatalf("a lost box's park-local asserts nothing about its disk: %+v", o)
+ t.Fatalf("a target with no installation in inventory acquired local cleanup: %+v", o)
}
}
// The target's own predecessor is covered by the park; the source's
@@ -3134,6 +5100,52 @@ func TestReleaseLostTargetReplayAfterReadmit(t *testing.T) {
t.Fatal("a replay changed the claim")
}
}
+
+func TestReleaseLostTargetRetainsInventoryBoundLocal(t *testing.T) {
+ for _, kind := range []string{KindLanding, KindRestoring} {
+ t.Run(kind, func(t *testing.T) {
+ phase := PhaseReleasing
+ if kind == KindLanding {
+ phase = PhaseProbing
+ }
+ c := lostTargetClaim(t, kind, phase)
+ c.Inventory = map[string]InventoryEntry{"uk-box": {Paths: map[string]uint64{"root": 21}}}
+ if err := ReleaseLostTarget(lostRelease(c, false))(c); err != nil {
+ t.Fatal(err)
+ }
+ var local Obligation
+ for _, o := range c.Obligations {
+ if o.Kind == "park-local" && o.Box == "uk-box" {
+ local = o
+ }
+ }
+ if local.ID == "" || local.Op != lostOp || local.Retirement == "" {
+ t.Fatal("target inventory's retirement obligation missing")
+ }
+ before := claimJSON(t, c)
+ if err := ReleaseLostTarget(lostRelease(c, false))(c); err != nil {
+ t.Fatal(err)
+ }
+ if claimJSON(t, c) != before {
+ t.Fatal("retry lost or replaced local chore")
+ }
+ if err := ReturnToSource(lostOp, "ie-box", true, ReturnFence{})(c); err != nil {
+ t.Fatal("pending target cleanup blocked source recovery", err)
+ }
+ c.Op.Phase, c.Op.Exec = PhaseOpened, "ie-box"
+ c.Obligations = []Obligation{local}
+ if err := ReleaseMove(lostOp, "ie-box")(c); err == nil {
+ t.Fatal("move released with target-local retirement pending")
+ }
+ if err := CompleteObligation(local.Op, local.ID)(c); err != nil {
+ t.Fatal(err)
+ }
+ if err := ReleaseMove(lostOp, "ie-box")(c); err != nil {
+ t.Fatal("retired local still blocks release", err)
+ }
+ })
+ }
+}
```
#### New file: `internal/hostops/move_loss_test.go`
@@ -4541,34 +6553,19 @@ func TestMoveRollbackLossRebind(t *testing.T) {
```diff
--- a/internal/console/apiinstancemove.go
+++ b/internal/console/apiinstancemove.go
-@@ -17,7 +17,9 @@
- "github.com/carlosframework/platform/internal/appconfig"
- "github.com/carlosframework/platform/internal/hostactors"
- "github.com/carlosframework/platform/internal/hostclaim"
+@@ -19,0 +20 @@
+ "github.com/carlosframework/platform/internal/hostfence"
- "github.com/carlosframework/platform/internal/hostops"
+@@ -20,0 +22 @@
+ "github.com/carlosframework/platform/internal/hostresolve"
- "github.com/carlosframework/platform/internal/moves"
- "github.com/carlosframework/platform/internal/objstore"
- "github.com/carlosframework/platform/internal/provision"
-@@ -868,8 +870,12 @@
- http.NotFound(w, r)
- return
- }
+@@ -871 +873 @@
- acct, app, host, rec, _, ok := s.moveOperatorGate(w, r)
+ acct, app, host, rec, addr, ok := s.moveOperatorGate(w, r)
- if !ok {
+@@ -872,0 +875,4 @@
+ return
+ }
+ if s.cfg.HostClaim {
+ s.claimMoveResolve(w, r, acct, app, host, rec, addr)
- return
- }
- var req struct {
-@@ -882,14 +888,88 @@
- jsonError(w, http.StatusConflict, "nothing to resolve yet")
- return
- }
+@@ -885,4 +891,26 @@
- // The record write is a console move write. On a host-claim console it is
- // the typed console move grant's (claim Plan 10, ruling J17); without the
- // claim runtime moveTransitionSaying still refuses it with HC5-N1.
@@ -4599,7 +6596,7 @@ func TestMoveRollbackLossRebind(t *testing.T) {
+ case req.Retry && (req.DeclareLost != "" || req.Submission != nil), req.DeclareLost == "" && req.Submission != nil:
+ jsonError(w, http.StatusBadRequest, "invalid resolution request")
+ case req.Retry:
- s.claimConsoleMove(w, r, acct, app, host, rec, hostops.ConsoleMoveRetry)
+@@ -890,3 +918,55 @@
- return
- }
- s.moveTransitionP1(w, r, acct, app, host, provision.MoveRetry(moveGuardOf(rec)), http.StatusAccepted)
@@ -4658,9 +6655,6 @@ func TestMoveRollbackLossRebind(t *testing.T) {
+ default:
+ claimConsoleMoveFailed(w, acct, app, host, err)
+ }
- }
-
- // claimConsoleMove is the cut, abort and retry doors on a host-claim console
```
#### New file: `internal/hostclaim/move_loss.go`
@@ -4670,6 +6664,7 @@ package hostclaim
import (
"fmt"
+ "strconv"
)
// journalLostTargetReleased marks the console's release of a declared-lost
@@ -4702,8 +6697,8 @@ type LostTargetRelease struct {
// rollback;
// - restoring/releasing steps to released, queues the target region's
// whole location park and covers that region's predecessors, as
-// TargetReleased does for a target that released itself. A lost box is
-// never given a park-local: the park asserts nothing about its disk;
+// TargetReleased does for a target that released itself. Inventory of
+// the target installation also requires its exact local retirement;
// - restoring/released (the target released dirty and died before its
// return) keeps the location park its own release queued;
// - the move's staging lane in the target region is queued, and a
@@ -4777,7 +6772,11 @@ func ReleaseLostTarget(r LostTargetRelease) func(*Claim) error {
return fmt.Errorf("%w: exec is %q", ErrWrongState, c.Op.Exec)
}
c.Op.Phase = PhaseReleased
- if err := QueueLocationPark(r.OpID, r.TargetRegion, "", r.Location, r.NewID)(c); err != nil {
+ parkBox := ""
+ if stamp, err := strconv.ParseUint(c.Op.Journal[JournalTargetStamp], 10, 64); err == nil && InventoryHasStamp(c, r.Target, stamp) {
+ parkBox = r.Target
+ }
+ if err := QueueLocationPark(r.OpID, r.TargetRegion, parkBox, r.Location, r.NewID)(c); err != nil {
return err
}
return CoverRegionPredecessors(r.OpID, PhaseReleased, r.TargetRegion)(c)
@@ -5427,11 +7426,10 @@ const (
moveLossRecordRollingBack = "rolling_back"
)
-// moveLossKinds are the claim op kinds this owner serves: a landing before
-// its commit, and its rollback before the target released or while it owes
-// its return. A moving op (before the flip) and a revert are not a target
-// loss this owner resolves; they stay refused.
-var moveLossKinds = []string{hostclaim.KindLanding, hostclaim.KindRestoring}
+// Move kinds share the exact local-only cleanup owner. Data-authority
+// target loss remains restricted to landing before commit and restoring
+// before source takeover; moving and reverting gain no takeover authority.
+var moveLossKinds = []string{hostclaim.KindMoving, hostclaim.KindLanding, hostclaim.KindReverting, hostclaim.KindRestoring}
// moveLossIntent is the signed recovery intent: everything the hooks and a
// crash recovery need, bound to the inspected claim and record.
@@ -5775,10 +7773,9 @@ func (m *moveLossOwner) clock() time.Time {
// InspectLoss is read-only. The record must be exactly the observation the
// service bound into the challenge.
-func (m *moveLossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, record hostfence.RecordObservation, box string) (hostfence.Consequences, hostclaim.LossConsequences, error) {
- in, err := claimLossIntent(o.Claim, box)
- if err != nil {
- return hostfence.Consequences{}, hostclaim.LossConsequences{}, err
+func (m *moveLossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, record hostfence.RecordObservation, box string) (hostfence.Consequences, hostclaim.LossConsequences, error) {
+ if o.Claim == nil || o.Claim.Holder == nil {
+ return hostfence.Consequences{}, hostclaim.LossConsequences{}, hostresolve.ErrResolve
}
if o.Absent || o.ETag == "" || record.Key != o.Claim.Holder.Key || record.CommitState != "present" || record.ETag == "" {
return hostfence.Consequences{}, hostclaim.LossConsequences{}, hostresolve.ErrResolve
@@ -5790,7 +7787,18 @@ func (m *moveLossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, r
if r.ETag != record.ETag {
return hostfence.Consequences{}, hostclaim.LossConsequences{}, hostclaim.ErrConflict
}
- if in, err = lossIntentAt(ctx, m.store, m.pub, o, r, box); err != nil {
+ if moveLocalLossEligible(o.Claim, box) {
+ if !matches(r, o.Claim.Holder) {
+ return hostfence.Consequences{}, hostclaim.LossConsequences{}, ErrIdentity
+ }
+ rec, e := provision.DecodeRecord(r.Body)
+ if e != nil || rec.Move == nil || rec.Move.ID != o.Claim.Op.ID || rec.Move.Incarnation != o.Claim.Holder.Incarnation {
+ return hostfence.Consequences{}, hostclaim.LossConsequences{}, ErrIdentity
+ }
+ return moveLocalLossConsequences(o.Claim, box)
+ }
+ in, err := lossIntentAt(ctx, m.store, m.pub, o, r, box)
+ if err != nil {
return hostfence.Consequences{}, hostclaim.LossConsequences{}, err
}
return moveLossConsequences(o.Claim, box, in)
@@ -5800,6 +7808,13 @@ func (m *moveLossOwner) InspectLoss(ctx context.Context, o hostclaim.Observed, r
// must be exactly what this claim implies for the signed intent, whose
// claim-derived values must be this claim's.
func (m *moveLossOwner) ValidateDeclareLost(c hostclaim.Claim, d hostclaim.LostDeclaration) error {
+ if isMoveLocalLoss(d.Consequences) {
+ _, actual, err := moveLocalLossConsequences(&c, d.Box)
+ if err != nil || !reflect.DeepEqual(actual, d.Consequences) {
+ return hostresolve.ErrResolve
+ }
+ return nil
+ }
in, err := decodeMoveLossIntent(d.Consequences.RecoveryIntent)
if err != nil {
return err
@@ -5865,6 +7880,9 @@ func (m *moveLossOwner) ValidateDeclareLost(c hostclaim.Claim, d hostclaim.LostD
// source's own replica (the seal in the lost target's region is the only
// holder of those writes, §7.10). Pure.
func (m *moveLossOwner) LossWriteSources(_ context.Context, o hostclaim.Observed, actual hostclaim.LossConsequences) ([]hostclaim.WriteSource, error) {
+ if isMoveLocalLoss(actual) {
+ return nil, nil
+ }
in, err := decodeMoveLossIntent(actual.RecoveryIntent)
if err != nil {
return nil, err
@@ -6001,6 +8019,9 @@ func prepareMoveLoss(rec *provision.Record, in moveLossIntent, ref string, at ti
// signed loss is prepared, the claim is the declaration's exact observation
// and the record is the inspected one (ETag and record_seq).
func (m *moveLossOwner) PrepareLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
+ if isMoveLocalLoss(loss.ConfirmedConsequences) {
+ return m.localLossHook(ctx, o, loss, false)
+ }
in, ref, err := lossIntentOf(loss)
if err != nil {
return err
@@ -6086,6 +8107,9 @@ func moveLossRelease(in moveLossIntent, loss hostclaim.LossRecord, h *hostclaim.
// successor holder or operation is never touched: only a record that
// already shows this declaration complete is success then.
func (m *moveLossOwner) FinishLoss(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord) error {
+ if isMoveLocalLoss(loss.ConfirmedConsequences) {
+ return m.localLossHook(ctx, o, loss, true)
+ }
in, ref, err := lossIntentOf(loss)
if err != nil {
return err
@@ -6202,6 +8226,466 @@ func MoveRollbackLossRebind(g MoveGuard, oldRef, newRef string, now time.Time) f
}
```
+#### Patch: `internal/hostclaim/lossrecord.go`
+
+```diff
+--- a/internal/hostclaim/lossrecord.go
++++ b/internal/hostclaim/lossrecord.go
+@@ -163,0 +164,7 @@
++ intent, intentErr := canonicalRaw(r.ConfirmedConsequences.RecoveryIntent)
++ if intentErr == nil && intent == `{"kind":"destroyed-local-only"}` && !destroyedLocalOnly(r.ConfirmedConsequences) {
++ return ErrLossRecord
++ }
++ if destroyedLocalOnly(r.ConfirmedConsequences) && (len(r.WriteSources) != 0 || len(r.ArchivedAdmissionLosses) != 0) {
++ return ErrLossRecord
++ }
+@@ -375,0 +383,4 @@
++
++// IsDestroyedLocalOnly reports the complete attested consequence shape, not
++// merely its intent marker. Owners and durable records use the same predicate.
++func IsDestroyedLocalOnly(c LossConsequences) bool { return destroyedLocalOnly(c) }
+```
+
+#### Patch: `internal/hostclaim/lost.go`
+
+```diff
+--- a/internal/hostclaim/lost.go
++++ b/internal/hostclaim/lost.go
+@@ -258,2 +258,5 @@
+- if c.Op.Kind == KindServing {
+- if in.OpID != "" || in.OpPhase != "" || !destroyedLocalOnly(in.Consequences) {
++ if c.Op.Kind == KindServing || destroyedLocalOnly(in.Consequences) {
++ if c.Op.Kind != KindServing && c.Op.Exec == in.Box {
++ return ErrWrongState
++ }
++ if (c.Op.Kind == KindServing && (in.OpID != "" || in.OpPhase != "")) || !destroyedLocalOnly(in.Consequences) {
+```
+
+#### New file: `internal/hostclaim/move_local_loss_test.go`
+
+```go
+package hostclaim
+
+import (
+ "bytes"
+ "crypto/ed25519"
+ "encoding/json"
+ "fmt"
+ "github.com/carlosframework/platform/internal/release"
+ "testing"
+)
+
+func TestPlan10MoveLocalDischargeCompletesDiscovery(t *testing.T) {
+ for _, otherChores := range []bool{false, true} {
+ t.Run(fmt.Sprint(otherChores), func(t *testing.T) {
+ ctx := t.Context()
+ s := release.NewDirStore(t.TempDir())
+ pub, priv, err := ed25519.GenerateKey(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ obs, err := Read(ctx, s, testHost)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var regions []string
+ boxes := []string{"box-a"}
+ if otherChores {
+ regions = []string{"uk"}
+ boxes = append(boxes, "other-box")
+ }
+ obs, err = Update(ctx, s, obs, Adopt(newHolder(), "serving-box", regions, boxes, "discovery", counter()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if obs.Claim.Op.Kind != KindServing || obs.Claim.Op.ID != "" {
+ t.Fatal("fixture is not an adopted serving claim")
+ }
+ obs, err = Update(ctx, s, obs, func(c *Claim) error {
+ c.Op = Op{ID: lostOp, Kind: KindLanding, Phase: PhaseCommitted, Exec: "serving-box", Participants: []string{"box-a", "serving-box"}, Journal: map[string]string{"keep": "unchanged"}}
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ var ref ObligationRef
+ for _, o := range obs.Claim.Obligations {
+ if o.Kind == "park-local" && o.Box == "box-a" {
+ ref = ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}
+ }
+ }
+ effects := LossConsequences{DischargeLocal: []ObligationRef{ref}, DestroyedStorage: []string{ref.Op + "/" + ref.ID + "/" + ref.Retirement}, RecoveryIntent: []byte(`{"kind":"destroyed-local-only"}`)}
+ r := lossForObservation(t, obs, "local_only", 11, "box-a", effects)
+ if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), r); err != nil {
+ t.Fatal(err)
+ }
+ got, err := DeclareLost(ctx, s, pub, obs, r.ID, testLossOwner{mutate: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ want, err := cloneClaim(obs.Claim)
+ if err != nil {
+ t.Fatal(err)
+ }
+ remaining := make([]Obligation, 0, len(want.Obligations)-1)
+ for _, o := range want.Obligations {
+ if o.ID != ref.ID {
+ remaining = append(remaining, o)
+ }
+ }
+ want.Obligations = remaining
+ if !otherChores {
+ want.Touched.Complete, want.Touched.DiscoveryOp = true, ""
+ }
+ want.Gen = got.Claim.Gen
+ want.Lost, want.LostRevisions, want.LossRefs = []string{"box-a"}, map[string]uint64{"box-a": 11}, []string{r.ID}
+ // Compare serialized claims so nil/empty omitted arrays are equivalent.
+ wantBytes, _ := json.Marshal(want)
+ gotBytes, _ := json.Marshal(got.Claim)
+ if !bytes.Equal(wantBytes, gotBytes) {
+ t.Fatalf("local-only changed unrelated state:\nwant %s\ngot %s", wantBytes, gotBytes)
+ }
+ if retry, err := DeclareLost(ctx, s, pub, got, r.ID, testLossOwner{}); err != nil || retry.ETag != got.ETag {
+ t.Fatal("local-only replay", err)
+ }
+ })
+ }
+}
+
+func TestPlan10MoveLocalDischargeRefusesExecutingBoxAndWidening(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ pub, priv, err := ed25519.GenerateKey(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err := Read(t.Context(), s, testHost)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err = Update(t.Context(), s, o, Adopt(newHolder(), "box-a", nil, []string{"box-a"}, "discovery", counter()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err = Update(t.Context(), s, o, func(c *Claim) error {
+ c.Op = Op{ID: lostOp, Kind: KindLanding, Phase: PhaseCommitted, Exec: "box-a"}
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ chore := o.Claim.Obligations[0]
+ e := LossConsequences{DischargeLocal: []ObligationRef{{ID: chore.ID, Op: chore.Op, Retirement: chore.Retirement}}, DestroyedStorage: []string{chore.Op + "/" + chore.ID + "/" + chore.Retirement}, RecoveryIntent: []byte(`{"kind":"destroyed-local-only"}`)}
+ r := lossForObservation(t, o, "local_exec", 11, "box-a", e)
+ if _, err := CreateLossRecord(t.Context(), s, release.RootSigner(priv), r); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := DeclareLost(t.Context(), s, pub, o, r.ID, testLossOwner{}); err == nil {
+ t.Fatal("local-only discharged executing box")
+ }
+ r.ID = "widened"
+ r.ConfirmedConsequences.RevokeExecution = true
+ if err := r.Validate(); err == nil {
+ t.Fatal("local-only intent accepted widened effects")
+ }
+}
+```
+
+#### Patch: `internal/hostfence/challenge.go`
+
+```diff
+--- a/internal/hostfence/challenge.go
++++ b/internal/hostfence/challenge.go
+@@ -126 +126,3 @@
+- if c.Op.Kind != hostclaim.KindServing || c.Op.ID != "" || c.Op.Phase != "" ||
++ serving := c.Op.Kind == hostclaim.KindServing
++ move := c.Op.Kind == hostclaim.KindMoving || c.Op.Kind == hostclaim.KindLanding || c.Op.Kind == hostclaim.KindReverting || c.Op.Kind == hostclaim.KindRestoring
++ if (!serving && !move) || (serving && (c.Op.ID != "" || c.Op.Phase != "")) || (move && c.Op.ID == "") ||
+```
+
+#### New file: `internal/hostops/move_local_loss.go`
+
+```go
+package hostops
+
+import (
+ "context"
+ "encoding/json"
+ "reflect"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfence"
+ "github.com/carlosframework/platform/internal/hostresolve"
+)
+
+// Local destruction is a separate declaration. An active target still takes
+// the data-loss path; its newly queued local chore needs a fresh inspection.
+// A committed move can discharge its retired source's local installation.
+func moveLocalLossEligible(c *hostclaim.Claim, box string) bool {
+ if c == nil || c.Holder == nil || box == "" || c.Op.Exec == box {
+ return false
+ }
+ inactive := slicesContain(c.Lost, box) || (c.Op.Kind == hostclaim.KindLanding && c.Op.Phase == hostclaim.PhaseCommitted) || !slicesContain(c.Op.Participants, box)
+ if !inactive {
+ return false
+ }
+ for _, o := range c.Obligations {
+ if o.Kind == "park-local" && o.Box == box {
+ return true
+ }
+ }
+ return false
+}
+
+func isMoveLocalLoss(actual hostclaim.LossConsequences) bool {
+ return hostclaim.IsDestroyedLocalOnly(actual)
+}
+
+// The signed claim ETag binds each original operation, chore and retirement
+// identity to its inventory. The operator attests exactly this displayed set.
+func moveLocalLossConsequences(c *hostclaim.Claim, box string) (hostfence.Consequences, hostclaim.LossConsequences, error) {
+ if !moveLocalLossEligible(c, box) {
+ return hostfence.Consequences{}, hostclaim.LossConsequences{}, hostresolve.ErrResolve
+ }
+ shown := hostfence.Consequences{Transition: "destroyed-local-only"}
+ actual := hostclaim.LossConsequences{RecoveryIntent: json.RawMessage(`{"kind":"destroyed-local-only"}`)}
+ for _, o := range c.Obligations {
+ if o.Kind != "park-local" || o.Box != box {
+ continue
+ }
+ ref := hostclaim.ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}
+ key := o.Op + "/" + o.ID + "/" + o.Retirement
+ shown.DischargeLocal = append(shown.DischargeLocal, ref)
+ shown.DestroyedStorageAssertion = append(shown.DestroyedStorageAssertion, key)
+ actual.DischargeLocal = append(actual.DischargeLocal, ref)
+ actual.DestroyedStorage = append(actual.DestroyedStorage, key)
+ }
+ return shown, actual, nil
+}
+
+// No record rollback, release, execution handover or post-commit effect is
+// authorized by this declaration. The generic claim CAS is its entire effect.
+func (m *moveLossOwner) localLossHook(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord, finish bool) error {
+ durable, err := m.durableLoss(ctx, loss)
+ if err != nil {
+ return err
+ }
+ if durable.Validate() != nil || !reflect.DeepEqual(durable.ConfirmedConsequences, loss.ConfirmedConsequences) || durable.Host != o.Host {
+ return hostresolve.ErrResolve
+ }
+ if finish {
+ if durable.State != "committed" || o.Claim == nil || !slicesContain(o.Claim.LossRefs, loss.ID) {
+ return hostresolve.ErrResolve
+ }
+ return nil
+ }
+ if durable.State == "committed" {
+ return nil
+ }
+ if durable.State != "prepared" || o.Absent || o.ETag != durable.ExpectedClaimETag {
+ return hostresolve.ErrResolve
+ }
+ _, want, err := moveLocalLossConsequences(o.Claim, durable.Box)
+ if err != nil || !reflect.DeepEqual(want, durable.ConfirmedConsequences) {
+ return hostresolve.ErrResolve
+ }
+ return nil
+}
+```
+
+#### New file: `internal/hostops/move_local_loss_test.go`
+
+```go
+package hostops
+
+import (
+ "context"
+ "encoding/json"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+func TestJ4DestroyedLocalOnlyAfterCommit(t *testing.T) {
+ w := newLossWorld(t, "landing", func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) {
+ c.Op.Phase = hostclaim.PhaseCommitted
+ r.Move.State = provision.MoveCommitted
+ c.Obligations = []hostclaim.Obligation{
+ {ID: "local", Op: lossOp, Kind: "park-local", Box: "ie-box", Retirement: "ret-local", Owner: "box:ie-box"},
+ {ID: "remote", Op: lossOp, Kind: "park-replica", Region: "ie", Retirement: "ret-remote", Owner: "box:ie-box", Park: &hostclaim.ParkScope{Location: c.Holder.Location, Incarnation: c.Holder.Incarnation, EvidenceAt: r.CreatedAt, RequestLimit: 1}},
+ {ID: "other", Op: "older-op", Kind: "park-local", Box: "uk-box", Retirement: "ret-other", Owner: "box:uk-box"},
+ }
+ })
+ before := w.claim(t)
+ record, tag := w.record(t)
+ in := w.inspect(t, "ie-label")
+ if in.Details.Consequences.Transition != "destroyed-local-only" || len(in.Details.Consequences.DischargeLocal) != 1 || in.Details.Consequences.PossibleWriteLoss || in.Details.Consequences.PossibleTimerLoss {
+ t.Fatalf("effects %+v", in.Details.Consequences)
+ }
+ sub := w.submission(in)
+ sub.Observation.Region = "eu-west-1"
+ sub.Observation.InstanceID = "i-ie"
+ got, err := w.control.DeclareMoveLoss(context.Background(), w.signer, lossBucket, "operator", "bab", "app", lossHost, "ie-box", sub)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(before.Claim.Op, got.Claim.Op) || !reflect.DeepEqual(before.Claim.Holder, got.Claim.Holder) || !reflect.DeepEqual(before.Claim.Inventory, got.Claim.Inventory) {
+ t.Fatal("local discharge changed execution, sessions, holder or inventory")
+ }
+ if !reflect.DeepEqual(got.Claim.Obligations, before.Claim.Obligations[1:]) {
+ t.Fatalf("other obligations changed: %+v", got.Claim.Obligations)
+ }
+ after, afterTag := w.record(t)
+ if tag != afterTag || !reflect.DeepEqual(record, after) {
+ t.Fatal("local discharge wrote committed record")
+ }
+ if _, err := w.control.DeclareMoveLoss(context.Background(), w.signer, lossBucket, "operator", "bab", "app", lossHost, "ie-box", sub); err != nil {
+ t.Fatal("replay", err)
+ }
+ // A local-only signed record grants no second claim or record write.
+ scoped := &lossScopedStore{raw: w.raw, pub: w.pub, now: func() time.Time { return w.clock }, b: lossBinding{lossID: in.Details.ID, host: lossHost, opID: lossOp, kind: hostclaim.KindLanding, box: "ie-box", claimETag: before.ETag, incarnation: w.inc, account: "bab", app: "app"}}
+ for _, key := range []string{w.key, w.claimK} {
+ b, tag, err := w.raw.Get(context.Background(), key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := scoped.Put(context.Background(), key, b, tag); err == nil {
+ t.Fatal("local-only grant authorized follow-up write", key)
+ }
+ }
+}
+
+func TestJ4DestroyedLocalOnlyAlreadyDeclaredTarget(t *testing.T) {
+ w := newLossWorld(t, "landing", func(c *hostclaim.Claim, _ *provision.Record, _ *AppAdmission) {
+ c.Inventory = map[string]hostclaim.InventoryEntry{"uk-box": {Paths: map[string]uint64{"root": 21}}}
+ })
+ if _, err := w.declare(w.submission(w.inspect(t, "uk-label"))); err != nil {
+ t.Fatal(err)
+ }
+ before := w.claim(t)
+ _, tag := w.record(t)
+ var local hostclaim.Obligation
+ for _, o := range before.Claim.Obligations {
+ if o.Kind == "park-local" && o.Box == "uk-box" {
+ local = o
+ }
+ }
+ if local.ID == "" {
+ t.Fatal("lost target installation has no park-local obligation")
+ }
+ in := w.inspect(t, "uk-label")
+ sub := w.submission(in)
+ got, err := w.declare(sub)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(before.Claim.Op, got.Claim.Op) || !reflect.DeepEqual(before.Claim.Holder, got.Claim.Holder) {
+ t.Fatal("second declaration changed data authority")
+ }
+ want := *before.Claim
+ want.Obligations = append([]hostclaim.Obligation(nil), before.Claim.Obligations...)
+ if err := hostclaim.CompleteObligation(local.Op, local.ID)(&want); err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(want.Obligations, got.Claim.Obligations) {
+ t.Fatal("regional or staging obligations changed")
+ }
+ _, afterTag := w.record(t)
+ if tag != afterTag {
+ t.Fatal("local-only changed rollback record")
+ }
+ loss, err := hostclaim.ReadLossRecord(context.Background(), w.raw, w.pub, in.Details.ID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(loss.Record.WriteSources) != 0 || loss.Record.ConfirmedConsequences.RevokeExecution || loss.Record.ConfirmedConsequences.RemoveParticipation {
+ t.Fatal("local-only reopened data authority")
+ }
+}
+
+func TestJ4DestroyedLocalOnlyRequiresExactAttestation(t *testing.T) {
+ for _, which := range []string{"missing", "wrong-retirement", "terminated-only", "stale-claim", "stale-record"} {
+ t.Run(which, func(t *testing.T) {
+ w := newLossWorld(t, "landing", func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) {
+ c.Op.Phase = hostclaim.PhaseCommitted
+ r.Move.State = provision.MoveCommitted
+ c.Obligations = []hostclaim.Obligation{{ID: "local", Op: lossOp, Kind: "park-local", Box: "ie-box", Owner: "box:ie-box", Retirement: "ret"}}
+ })
+ in := w.inspect(t, "ie-label")
+ sub := w.submission(in)
+ sub.Observation.Region = "eu-west-1"
+ sub.Observation.InstanceID = "i-ie"
+ switch which {
+ case "missing":
+ sub.Attestation.Disks = nil
+ case "wrong-retirement":
+ sub.Attestation.Disks = []string{lossOp + "/local/other"}
+ case "terminated-only":
+ sub.Observation.State = "terminated"
+ sub.Attestation.Disks = nil
+ case "stale-claim":
+ o := w.claim(t)
+ o.Claim.Obligations[0].Retirement = "other"
+ b, _ := json.Marshal(o.Claim)
+ if _, err := w.raw.Store.Put(context.Background(), w.claimK, b, o.ETag); err != nil {
+ t.Fatal(err)
+ }
+ case "stale-record":
+ r, tag := w.record(t)
+ r.RecordSeq++
+ b, _ := json.Marshal(r)
+ if _, err := w.raw.Store.Put(context.Background(), w.key, b, tag); err != nil {
+ t.Fatal(err)
+ }
+ }
+ snap := lossSnapshot(t, w)
+ if _, err := w.control.DeclareMoveLoss(context.Background(), w.signer, lossBucket, "operator", "bab", "app", lossHost, "ie-box", sub); err == nil {
+ t.Fatal("unattested or stale discharge accepted")
+ }
+ if lossSnapshot(t, w) != snap {
+ t.Fatal("refusal wrote state")
+ }
+ })
+ }
+}
+
+func TestJ4DestroyedLocalOnlyRejectsWidenedEffects(t *testing.T) {
+ w := newLossWorld(t, "landing", func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) {
+ c.Op.Phase = hostclaim.PhaseCommitted
+ r.Move.State = provision.MoveCommitted
+ c.Obligations = []hostclaim.Obligation{{ID: "local", Op: "discovery", Kind: "park-local", Box: "ie-box", Retirement: "ret-local", Owner: "box:ie-box"}}
+ })
+ c := w.claim(t).Claim
+ _, base, err := moveLocalLossConsequences(c, "ie-box")
+ if err != nil {
+ t.Fatal(err)
+ }
+ owner := &moveLossOwner{}
+ for name, change := range map[string]func(*hostclaim.LossConsequences){
+ "execution": func(e *hostclaim.LossConsequences) { e.RevokeExecution = true },
+ "participation": func(e *hostclaim.LossConsequences) { e.RemoveParticipation = true },
+ "writes": func(e *hostclaim.LossConsequences) { e.WritesPotentiallyLost = true },
+ "timers": func(e *hostclaim.LossConsequences) { e.TimersPotentiallyLost = true },
+ "regional": func(e *hostclaim.LossConsequences) { e.ReassignRegional = e.DischargeLocal },
+ "original-op": func(e *hostclaim.LossConsequences) { e.DischargeLocal[0].Op = "other" },
+ "retirement": func(e *hostclaim.LossConsequences) { e.DischargeLocal[0].Retirement = "other" },
+ "no-attestation": func(e *hostclaim.LossConsequences) { e.DestroyedStorage = nil },
+ } {
+ t.Run(name, func(t *testing.T) {
+ b, _ := json.Marshal(base)
+ var effects hostclaim.LossConsequences
+ json.Unmarshal(b, &effects)
+ change(&effects)
+ if err := owner.ValidateDeclareLost(*c, hostclaim.LostDeclaration{Box: "ie-box", Consequences: effects}); err == nil {
+ t.Fatal("widened local-only consequences accepted")
+ }
+ })
+ }
+}
+```
+
### Task E: reconcile the reviewed writer inventories
Invariant: every production declaration that can name a record write remains explicitly inventoried, and every record writer still passes through the sequenced provision primitive. Do not weaken or regenerate the census policy. Run its existing tests first: the combined base and Tasks C/D introduce unlisted sites. After reviewing the sources and their callers, apply only the inventory patch below and rerun the same tests. The provision inventory is a maintenance patch to an existing test, so the extractor applies that file in the implementation stage. The console source census first adds the three new typed method names to its selection, then refuses their unlisted call sites. Its reviewed fixture is applied in the implementation stage.
@@ -6238,10 +8722,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou
```diff
--- a/internal/provision/plan9_b8_census_test.go
+++ b/internal/provision/plan9_b8_census_test.go
-@@ -472,6 +472,37 @@
-
- // BEGIN REVIEWED INVENTORY
- var plan9B8CensusInventory = []censusEntry{
+@@ -474,0 +475,31 @@
+ // Plan 10: reviewed combined moves base and typed console loss writers.
+ {"internal/host/movesfinal.go", "sourceFinal.configAck", "nonrecord", "sel:Put", "cca53109e6945a73b7f04705d8965b3b0997f2b7d2a82fa6d86cffa4b4174e93"},
+ {"internal/host/movesfinal.go", "sourceFinal.stampParked", "record-writer", "key:SetRecordMove", "8af033d29c8e2275d996df9a4e53227da30a15e8fd4c198a4ab40865db7fabb9"},
@@ -6249,7 +8730,7 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou
+ {"internal/host/movesinstall.go", "recoveryInstallSource", "record-writer", "key:SetRecordMove", "b4bb0798a032c1599b9063a165656ab76c00b6ea349cb17b5f6a95769432bd5f"},
+ {"internal/host/movesrecover.go", "recoverRun.finalize", "record-writer", "key:SetRecordMove", "fff2e28c78db68bccb16da6ab70ca877b51243b0aa56b9cb2c513c6f5660f7f7"},
+ {"internal/host/movesrecover.go", "recoverRun.phase", "record-writer", "key:SetRecordMove", "d67132b6e797a88a9677d962c5c7aa30fc7f85fdbdff06ce8318562397a9ee78"},
-+ {"internal/host/movesrecover.go", "recoverRun.probing", "record-writer", "key:SetRecordMove", "7184b393fc356deb85181e25e4cafd61079598bf7ea4dafc28f90cbe26d50a31"},
++ {"internal/host/movesrecover.go", "recoverRun.probing", "record-writer", "key:SetRecordMove", "a0144975bf52b29bbc6c2354636d3665487365d24838d3ef4baef1c665ce52ab"},
+ {"internal/host/movesrecover.go", "recoverRun.stuck", "record-writer", "key:SetRecordMove", "e2942947ddb1559e44b494e9900fa82ebbe3782a86333167797f4a94375a4a6c"},
+ {"internal/host/movesrollback.go", "movesWorker.rollBackLanding", "record-writer", "key:SetRecordMove", "40ff87900fbfacb5a4dff481db37ae7dbed41e5727f13185ba8623daf3e1681e"},
+ {"internal/host/movesrollback.go", "rollbackRun.laneMoveJSON", "nonrecord", "sel:PutIfAbsentVersioned", "5fd541f4a1daf2600ff266583fc9627cc0cab35e276f1d9c8f097f128c2c4e53"},
@@ -6266,16 +8747,13 @@ flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 10m -cou
+ {"internal/hostops/move_loss_owner.go", "claimLossIntent", "key-site", "key:RecordKey", "1b72b043ec7c1b20b95af9c312bae449cf19d29f5759a906fd14af6f845cb938"},
+ {"internal/hostops/move_loss_owner.go", "decodeMoveLossIntent", "key-site", "key:RecordKey", "e868448e696de4614a362f27301e6fa25506394fd7582d3076d126a5f72e93db"},
+ {"internal/hostops/move_loss_owner.go", "lossIntentOf", "key-site", "key:LossRecordKey", "bd0d6d6d2c53ed4daddae47791f4de5fda5935bd59be0a04e41ad22a0b3a2954"},
-+ {"internal/hostops/move_loss_owner.go", "moveLossOwner.FinishLoss", "record-writer", "key:RecordKey,key:WriteRecord", "b247221614a81675ba6ca1017c361af7290e7cf39a38fa3d39b204399a0c9a5f"},
-+ {"internal/hostops/move_loss_owner.go", "moveLossOwner.PrepareLoss", "record-writer", "key:RecordKey,key:WriteRecord", "ce938a605daab823e1be11a47736774950648b2a7b73468f43073a4b94348f35"},
-+ {"internal/hostops/move_loss_owner.go", "moveLossOwner.ValidateDeclareLost", "key-site", "key:LossRecordKey,key:RecordKey", "770de4254f482d78080587ca293a3176b06ae67e9e1af296a4db8a9b00ca4eb7"},
++ {"internal/hostops/move_loss_owner.go", "moveLossOwner.FinishLoss", "record-writer", "key:RecordKey,key:WriteRecord", "fb936dc901ec18ac2aeb1cd638eed0600df0b5dfe3bf2caf6efa6b4f000af1d2"},
++ {"internal/hostops/move_loss_owner.go", "moveLossOwner.PrepareLoss", "record-writer", "key:RecordKey,key:WriteRecord", "bf616b8ef1e26e3799d67872a380e1818b9d1bba168b3ee951ba1c1ad0c27a9b"},
++ {"internal/hostops/move_loss_owner.go", "moveLossOwner.ValidateDeclareLost", "key-site", "key:LossRecordKey,key:RecordKey", "c4375ea8e4311dcd6ec38be7bc1edbb04aaa2648d7d37fd177a326a8486362a8"},
+ {"internal/hostops/move_loss_owner.go", "readSuperseded", "key-site", "key:LossRecordKey", "c6e947a3d2eb726bbc7688fd5ea1320efd4092475c45b7e6a0af3f0cb956b2c4"},
+ {"internal/hostops/move_loss_owner.go", "sameMoveTarget", "key-site", "key:RecordKey", "ede06bcfca80df25af7c438629863e711f95eeb17c7c47f6f9d73459ce7ae8eb"},
+ {"internal/hostops/move_loss_owner.go", "type moveLossIntent", "key-site", "key:RecordKey", "ff9200092a5e3b55e6b422501f121c51613b8922c201ec29a6012b7bcb0e01ff"},
+ {"internal/layout/extras.go", "var platformNames", "reference", "lit:instances", "0e98837e22835749384ee57a1c1e3ccd687f5099057af7cb1ad56ffc415512c4"},
- // Hibernate generation markers use a different bucket and key grammar from deployment records.
- {"internal/host/claimproduction_marker.go", "productionClaimMarkers.PutMarker", "foreign-recordkey", "lit:instances", "69301cdddce2bc61a24f0dd8297534814521adc1ddc76a7d3054ed007444a0f5"},
- {"accounts.go", "putOverwrite", "nonrecord", "sel:Put", "a85f380434a2d297f14c8068083f41ca7f9c9b735efd7d3abf8d91520fe6453c"},
```
#### Patch: `internal/console/testdata/claim-authority-sites.txt`
@@ -6337,7 +8815,7 @@ Keep `CARLOS_HOST_CLAIM` and `CARLOS_MOVES_PHASE1` unset while landing these cha
Land the pinned moves preflight and the complete Plan 9 base first. Replay this plan against their reviewed merge result if it differs from the export above. Keep B8's uncertain record-start settlement and `record_seq` regressions green. Cut/abort/retry authority is independent of SQL permits and wake-reset receipts; B3 and B4 must retain their existing refusal tests.
-Moves Plan 4 Task 9 consumes Task A's APIs. Its adapter must record clean and dirty recovery inputs at `probing` before the held render, use nine recovery attempts, preserve the original first-start time where the move contract requires it, bind each reservation to the admitted session and launch digest, and use the same role-cgroup readiness check for the settle request. `KeepWriterSession` wraps each adjacent recovery phase CAS. The moves policy must continue admitting that exact session at the later row phases while refusing changed identities. Do not claim production recovery from a table test: the adapter, its park leases, prepared-generation/open evidence, retirement and production owner registration remain separate rollout prerequisites.
+Moves Plan 4 Task 9 consumes Task A's concrete adapter and activator entry. The adapter records clean and dirty recovery inputs at `probing` before the held render. Task 9 still supplies the durable nine-attempt `OperationAttempts` provider, original first-start time, restoration/admission evidence and production owner registration. Each reservation binds the admitted session and launch digest, and the settle request uses the same role-cgroup readiness check. `KeepWriterSession` wraps each adjacent recovery phase CAS. The moves policy must continue admitting that exact session at the later row phases while refusing changed identities. Do not claim production recovery from a table test: the concrete entry is wired, while its park leases, prepared-generation/open evidence, retirement and production owner registration remain rollout prerequisites.
Exercise a clean return, a dirty return, a probe restart and budget exhaustion/retry before rollout. For a declared target loss, exercise the target still executing, a dirty release without a completed return, a partially published return lane, a record commit racing the declaration, and retries at each durable boundary. Verify loss accounting through the later admission and retained signed record. A successful declaration by itself does not prove that an instance recovered. Target-local chores still require local retirement or exact destroyed-storage evidence, and the final close still waits for the target park stamp.
@@ -6391,47 +8869,64 @@ The inventory is deliberately conservative: it includes new diagnostic templates
"%w: another loss is recorded for this target"
```
+`internal/host/movesrecoveryprobe.go`:
+
+```text
+"moves: the recovery probe's attempts are spent"
+"moves: the recovery probe request does not match its record"
+"moves: the recovery probe's inputs name no release"
+"CARLOS_HOST_CLAIM Group D recovery admission"
+"activator recovery probe entry"
+"moves recovery probe route and inputs"
+"read the host claim: %w"
+"read the registry row: %w"
+"%w: %s has no registry row on this box"
+"select the recovery inputs: %w"
+"record the recovery inputs: %w"
+"the recovery probe wake: %w"
+"%w: not rollback %s at probing"
+"%w: budget %d, clean %v"
+"%w: a clean return admits a fresh generation restored from src_gen"
+"%w: a dirty return admits its sealed return generation"
+"%w: the claim is not rollback %s's probe on %s"
+"%w: the claim's src_gen is not the request's"
+```
+
+`internal/activator/recoveryprobe.go`:
+
+```text
+"%w: not a recovery probe policy"
+"activator: %s recovery probe healthy, but its fence changed; stopping it"
+"activator: %s recovery probe healthy; held until the source opens"
+"activator: %s recovery probe's session is over (%v); stopping it"
+"activator: %s source opened on the recovery probe's session; serving"
+```
+
The displayed transition token `move-target-lost` is new. It identifies the proposed target-loss consequence in the shared inspection. The HTTP paths reuse the existing `nothing to resolve yet`, `could not read the instance`, move-changed, missing-runtime HC5-N1 and fixed write-failure sentences. Their spelling and legacy-path behavior are unchanged.
## Verification and self-review
-The author used exported, pinned objects, with no git writes. Each task's tests were installed before its implementation in a separate replay. Initial failures were the missing recovery continuation/admission, missing registered-owner API, missing typed console methods and HC5-N1, then missing loss-release/owner methods and the HTTP door still answering `nothing to resolve yet`. The census first rejected the combined input's unlisted writers, then specifically rejected direct record forwarding in the new private store. No census policy was weakened.
+Rev 2 was checked in an external export of the two pinned commits. No git state or implementation file in the authoring repository was written. The rev 1 authority regressions failed first: all nine HTTP races returned 202, the inventory-backed lost target had no local chore, and committed-source local inspection was refused. The revised embedded code passes those cases, plus exact-attestation and stale-evidence refusals, already-declared target discharge, unchanged writer state and unrelated obligations, discovery completion and local-retirement release blocking.
-Focused checks, serial under `nice`, included:
+Task A adds adapter and production-entry tests. A separate Opus review found the changed-input replay that could strand an older held probe; its regression failed first and passes with cancellation/retirement and late-success refusal. Caller input maps are copied, and the mutation test failed before that copy was added. Clean and dirty returns record recovery inputs before rendering and launching; a failed record write prevents both. The Group D tests retain operation/session bindings, attempt accounting and role-cgroup readiness. The production recovery hook is connected only with the claim flag; missing dependencies refuse instead of reporting recovery. Failing-first replay installs every task's tests before its code. Initial missing-symbol failures establish new API dependencies; the rev 1 HTTP and discharge regressions above establish behavioral failures.
-- Recovery session binding, admission, attempt accounting and readiness: `TestPlan10J2`, Plan 9 D1-D3 and the adjacent session/observation regressions. The three socket-based D3 witnesses ran with the short TMPDIR and passed, with no skips.
-- The complete `internal/hostresolve` package, including accepted-request replay and failures at the durable boundaries.
-- Typed cut/abort/retry plus the real HTTP doors, stale identities, record-sequence protection, closed admission, expired registration and the final registration/claim reread.
-- Target-loss release, per-kind owner, scoped-store refusal, loss-source freezing, stale challenge, late/conflicted hooks and successor identity tests.
-- Both record-census policy and inventory checks against the combined tree.
+The final focused run passed all eight packages below, including the writer census and flag-off cases. `go build -p 1 ./...` passed and whole-tree `gofmt -l .` was empty. The complete `hostclaim`, `hostresolve` and `hostfence` test packages also passed, as did the activator recovery/hold tests under `-race`. Extraction into a fresh pinned export reproduced the implementation and test files byte-for-byte.
-The final extraction reproduced every implementation and test source byte-for-byte. Its focused package results were: hostclaim 0.015s, host 4.286s, hostresolve 0.030s, hostops 0.442s, console 4.979s and provision 0.309s. The complete build passed and whole-tree `gofmt -l .` returned no paths. The final replay command was:
+Run the focused integration check serially, including the census and flag-off regressions, then build and check formatting:
```bash
-flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 3m -count=1 ./internal/hostclaim ./internal/host ./internal/hostresolve ./internal/hostops ./internal/console ./internal/provision -run 'TestPlan10|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe|TestPlan9B8|TestPlan9C2|TestJ17|TestJ4|TestReleaseLostTarget|TestMoveRollbackLossRebind|TestClaimAuthoritySourceCensus|TestMoveDoorSwitchOffIsLegacy|TestMoveDoorLegacy|TestMoveDoorPhase1RefusedUnderHostClaim|TestResolveRetryRefusedUnderHostClaim|TestClaimPublishMovesConsoleIsInertWithTheSwitchOff|TestIngest|TestPreparedSQL'
+flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go test -p 1 -timeout 3m -count=1 ./internal/hostclaim ./internal/host ./internal/activator ./internal/hostresolve ./internal/hostops ./internal/console ./internal/provision ./internal/hostfence -run 'TestRecoveryProbe|TestRecoveryHold|TestOrdinaryClaimWake|TestPlan10|TestPlan9D[123]|TestPlan9Fix1Session|TestPlan9Fix2Observe|TestPlan9B8|TestPlan9C2|TestJ17|TestJ4|TestReleaseLostTarget|TestMoveRollbackLossRebind|TestServingDestroyedLocal|TestClaimAuthoritySourceCensus|TestMoveDoorSwitchOffIsLegacy|TestMoveDoorLegacy|TestMoveDoorPhase1RefusedUnderHostClaim|TestResolveRetryRefusedUnderHostClaim|TestClaimPublishMovesConsoleIsInertWithTheSwitchOff|TestIngest|TestPreparedSQL'
flock -o "$CLAIM10_SCRATCH/tests.lock" nice -n 10 go build -p 1 ./...
gofmt -l .
```
-This includes the existing B8 record-start/sequence cases, C2 wake-reset restrictions, SQL/ingest grant refusals and explicit legacy switch-off cases. The complete resolver package was also green independently before the final extraction. A broader console mutation-matrix run was stopped; no result from that stopped run is claimed here.
-
-Self-review found and corrected these issues before writing the plan:
-
-1. The combined bases renamed the committed phase constant; the bootstrap makes the repair explicit before tests run.
-2. A shallow record copy could alias nested move state. The typed grant now decodes its proposed record independently, protects maintenance bytes, and explicitly refuses sidecar records. A crafted sidecar observation failed before the fix and passed after it.
-3. Owner callbacks must tolerate late prepare and repeated completion without regressing a record or touching a successor. The contract states this, and the concrete owner uses durable signed identities and conditional writes.
-4. The pure target-release helper previously accepted lane parks from another incarnation, could omit a dirty return lane, and refused replay after readmission. Regression tests failed before the fixes and pass with the shared park guard, required return-lane scope and early durable replay check.
-5. A moving/sealed claim whose record has already flipped cannot lose its target participant before claim catch-up. The concrete owner refuses that lagging kind before acceptance; moves must catch it up before requesting a new inspection.
-6. The new record transport must still use the sequenced record primitive. The existing census caught direct forwarding, and it remains an enforced test.
-7. A record prepared as target-lost could be stranded after its declaration became conflicted. A fresh declaration now rebinds only a verified conflicted reference under the exact signed identities, preserving the original rollback and dirty-return cleanup. The regression fails at fresh inspection before the fix and proves declaration plus source takeover afterward.
-
-Opus reviewed the generic owner boundary, typed console grant, claim release and concrete move owner; the review findings above were resolved in the embedded code. Source digests record the exact reviewed writers rather than treating their addition to the inventory as authorization.
+Opus reviewed the console gate binding, inventory-bound release, local-only owner and scoped store. The review found no hard blocker in those paths and prompted library guards against discharging the current executor and against a widened consequence shape under the local-only marker; the added executing-box test failed before that guard. The source can claim recovery while target cleanup remains pending, but `ReleaseMove` stays blocked until the move's obligations complete. Local discharge deliberately preserves inventory and unrelated regional obligations; it does not claim that a destroyed box performed final artifact cleanup or write a `TargetParkedAt` stamp.
-The final bounded Opus review of the conflicted-reference fix found no blockers. It checked signed old-loss identity, terminal conflicted state, retained-reference refusal, preservation of the original rollback and dirty-return cleanup across a chain, the final scoped-store recheck and late-hook monotonicity. Mutation checks removed the old-state guard and the final-write recheck separately; their regression tests failed, then passed after the guards were restored.
+The healthy-process lifecycle tests supply the instance a successful start would return; the runtime-entry tests also verify the base launcher's honest refusal. This is a concrete adapter and entry contract for Plan 4 Task 9, not evidence of a successful production spawn.
-Coverage limits: the concrete owner has no dedicated test for a successor operation under the same holder or for late hooks across a multi-declaration rebind chain; the generic service tests original-owner retry after a kind change, the concrete owner tests successor-holder refusal, and the final review checked those additional paths in code. The scoped completion writer relies on `FinishLoss` to check the loss reference before calling the typed completion transition; the reviewed call order makes a mismatched reference unreachable. These are review notes, not authority granted to bypass those checks.
+Self-review checks each authority separately: the console gate identity survives the second read; a compute fence cannot erase local retirement; local-only discharge cannot invoke move rollback or completion; the current executor cannot use it; the signed claim revision binds original chore identities; retry retains evidence without reopening data authority. The J4 interface above is byte-identical to rev 1. Reviewed source digests must be updated only after inspecting the changed writers; no census policy is weakened.
-Remaining dependencies are explicit: the pinned local moves additions must land, proposed copy still needs Paul, and the production recovery adapter and its evidence/leases must land through moves Plan 4 Task 9 before rollout. Neither a full repository gate nor a production recovery was claimed by this plan-authoring task.
+The pinned moves additions, the isolated root launcher (`RoleLaunch.Start` still refuses in this base), production recovery owners and evidence, and Paul's review of proposed product copy remain rollout dependencies. These authoring checks do not claim the full repository landing gate or a production recovery witness.
Remove the replay only after all Go tests and delegates have exited. Cleanup deletes the created source exports, generated scripts and logs, and the short socket directory. It retains the shared Go cache. From a directory outside the scratch tree: