amadan

carlos / platform Public

moves: a declared or fenced return asks no coverage, and its loss admission is vouched for from the recorded maxima (Task 4: Q5, Q6, Q7's loss half)

The probe's declared kind: the record's Rollback.LossRef and the claim's
declaration or fence of the target must agree, by identity, or the probe
waits; agreed, every session asks no coverage and carries no loss id.

The loss baseline validator (LossSources) and the loss owner
(hostops.MoveLossRecoveryOwner) are on the moves owner set. No
cross-region read: a target replica's maximum is the admission's frozen
maximum (the source baseline, read in this box's own bucket by
genselect.GenerationMax), a seal's the journalled cut seal's N. An
unreadable or moved maximum waits. AdmissionEvidence accepts exactly the
writer's substitution from the same LossSources call, re-validated, and
still journals the lineage link and the source baseline before the CAS.
Task 3's interim refusal and its test are replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 04b5aa7b494ba4926facc17d0268a57fb43da159 parent bb93758
9 files changed, +1146 −61
  • approved_moves_copy_test.go +1 −0
  • internal/host/movesloss.go +308 −0
  • internal/host/movesloss_test.go +758 −0
  • internal/host/movesprobeowners.go +13 −5
  • internal/host/movesrecover.go +18 −3
  • internal/host/movesrecoveryowners.go +16 −22
  • internal/host/movesrecoveryowners_test.go +0 −23
  • internal/host/movesrecoveryprobe.go +26 −6
  • internal/provision/plan9_b8_census_test.go +6 −2
diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go
index 65eb303f6..6e858ff38 100644
--- a/approved_moves_copy_test.go
+++ b/approved_moves_copy_test.go
@@ -1166,6 +1166,7 @@ var movesScanFiles = []string{
"internal/hostops/move_loss_owner.go",
"internal/host/movesrecoveryprobe.go",
"internal/host/movesrecoveryowners.go",
+ "internal/host/movesloss.go",
"internal/host/claimproduction.go",
"internal/host/claimowners.go",
"internal/host/claimmoveaccess.go",
diff --git a/internal/host/movesloss.go b/internal/host/movesloss.go
new file mode 100644
index 000000000..92c23bcf4
--- /dev/null
+++ b/internal/host/movesloss.go
@@ -0,0 +1,308 @@
+package host
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "reflect"
+ "strconv"
+
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostops"
+ "github.com/carlosframework/platform/internal/park"
+)
+
+// The loss and fence rules of the source's recovery (moves-after-Plan-12
+// Task 4: owed (a) and (b), Plan 4 Task 10's box half; rulings Q6, Q7's
+// loss half and Q5 as platform-1e answered it on #236).
+//
+// A declared or fenced return (Q6). The claim journals a declaration of the
+// target's loss under the op (hostclaim.OperationLossID) or a termination
+// fence of the target (fence.<target>); the record's rollback cites it
+// (Rollback.LossRef: the declaration's record key, or the fence record's
+// key). Such a return is clean for want of a target and covered by nothing:
+// its probe asks no coverage, for its first admission and every later one,
+// since the declaration and the fence stay the op's after the first
+// admission consumed them. A record and a claim that disagree, by presence
+// or by identity, wait.
+//
+// The loss baseline validator (Q5, owed (b)). The claim's writer admits the
+// first recovery admission under a pending loss through admitLoss or
+// admitFencedLoss (hostbox/writer_loss.go), which call LossSources with the
+// frozen record. No cross-region read exists (the box role reads only its
+// own data bucket; platform-1e): each frozen source is vouched for from the
+// maxima already recorded.
+//
+// - replica:<region>/<gen>, a target session since the cut: the target's
+// replica is in the target's region, which this box never reads. Its
+// maximum is the admission's frozen maximum: the restore point the
+// recovery admits from (the source baseline, src_gen, frozen in the
+// recovery journal once evidenced), whose maximum this box reads in its
+// own bucket by genselect.GenerationMax, the rule the claim's writer
+// reads too. Writes past it are the ones this admission declares lost.
+// - seal:<move>, the source's cut session when the seal did not cover the
+// source's replica: the cut seal the claim journals, its maximum N.
+//
+// An unreadable or moved maximum is a wait (hostbox.ErrChanged), within
+// the probe's nine attempts; a frozen source this recovery cannot be the
+// owner of is a refusal (hostclaim.ErrLossRecord).
+//
+// AdmissionEvidence's loss contract (Q7). The writer substitutes the
+// validated predecessor's maximum for RemoteMaxTXID and the other validated
+// sources for AdditionalWritesLost before it calls AdmissionEvidence. The
+// recovery owners accept exactly that substitution, from the same
+// LossSources call for the same observation, re-validated; every other
+// check of the recovery's evidence, the lineage link and the source
+// baseline's record before the CAS included, is unchanged.
+
+// claimLossRefs are the keys of the target's loss the claim journals under
+// the op: the declaration's record key, and the fence record's key.
+func claimLossRefs(c *hostclaim.Claim) (map[string]bool, error) {
+ refs := map[string]bool{}
+ id, err := hostclaim.OperationLossID(c)
+ if err != nil {
+ return nil, err
+ }
+ if id != "" {
+ key, err := hostclaim.LossRecordKey(id)
+ if err != nil {
+ return nil, err
+ }
+ refs[key] = true
+ }
+ if raw := c.Op.Journal["fence."+c.Op.Intent["target"]]; raw != "" {
+ var ref hostclaim.FenceRef
+ if err := json.Unmarshal([]byte(raw), &ref); err != nil || ref.Key == "" {
+ return nil, fmt.Errorf("%w: the target's fence does not read", hostclaim.ErrFence)
+ }
+ refs[ref.Key] = true
+ }
+ return refs, nil
+}
+
+// recoveryDeclared reports whether a return is a declared or fenced one
+// (Q6): the claim journals the target's loss and the record's rollback
+// cites it (lossRef). A disagreement, by presence or by identity, is
+// hostbox.ErrChanged: the probe waits for the two to agree.
+func recoveryDeclared(c *hostclaim.Claim, lossRef string) (bool, error) {
+ refs, err := claimLossRefs(c)
+ if err != nil {
+ return false, fmt.Errorf("%w: the claim's loss of the target: %v", hostbox.ErrChanged, err)
+ }
+ if len(refs) == 0 && lossRef == "" {
+ return false, nil
+ }
+ if !refs[lossRef] {
+ return false, fmt.Errorf("%w: the record and the claim disagree on the target's loss", hostbox.ErrChanged)
+ }
+ return true, nil
+}
+
+// lossValidation is one LossSources call's answer, kept for the
+// AdmissionEvidence call of the same admission.
+type lossValidation struct {
+ etag string
+ record hostclaim.LossRecord
+ valid []hostclaim.ValidatedWriteSource
+}
+
+// LossSources is the moves loss baseline validator
+// (hostbox.LossAdmissionSources): the recovery owners' movesLossSources
+// for a recovery, refused for every other kind.
+func (m *movesProbeOwners) LossSources(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord, in hostclaim.Admission, baseline *genselect.Plan) ([]hostclaim.ValidatedWriteSource, error) {
+ if !recoveryKind(o) {
+ return nil, hostbox.ErrUnsupported
+ }
+ r := m.recovery()
+ valid, err := r.movesLossSources(ctx, o, loss, baseline)
+ if err != nil {
+ return nil, err
+ }
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ if m.lossValidated == nil {
+ m.lossValidated = map[string]lossValidation{}
+ }
+ // One per host: a later call for the host replaces an earlier one.
+ m.lossValidated[o.Host] = lossValidation{etag: o.ETag, record: loss, valid: append([]hostclaim.ValidatedWriteSource(nil), valid...)}
+ return valid, nil
+}
+
+// movesLossSources vouches for each of loss's frozen sources (see the file
+// comment), for the recovery's first admission while loss is the pending
+// declaration or the pending fence of this op's target.
+func (r movesRecoveryOwners) movesLossSources(ctx context.Context, o hostclaim.Observed, loss hostclaim.LossRecord, baseline *genselect.Plan) ([]hostclaim.ValidatedWriteSource, error) {
+ v, err := r.recovery(ctx, o)
+ if err != nil {
+ return nil, err
+ }
+ c := o.Claim
+ if v.kind != hostclaim.KindRestoring || v.dirty || !v.uncovered || !v.first {
+ return nil, fmt.Errorf("%w: only a declared or fenced return's first admission declares a loss", hostclaim.ErrLossRecord)
+ }
+ if err := r.pendingLoss(c, v, loss); err != nil {
+ return nil, err
+ }
+ frozen, err := r.frozenMaximum(ctx, v, baseline)
+ if err != nil {
+ return nil, err
+ }
+ cut, err := strconv.ParseUint(v.cut, 10, 64)
+ if err != nil {
+ return nil, fmt.Errorf("%w: the cut's session is not journalled", hostbox.ErrChanged)
+ }
+ out := make([]hostclaim.ValidatedWriteSource, 0, len(loss.WriteSources))
+ for _, s := range loss.WriteSources {
+ max, err := r.sourceMaximum(c, v, cut, frozen, s)
+ if err != nil {
+ return nil, err
+ }
+ out = append(out, hostclaim.ValidatedWriteSource{Reason: s.Reason, Box: s.Box, Generation: s.Generation, Session: s.Session, Region: s.Region, Baseline: s.Baseline, RemoteMaxTXID: max})
+ }
+ return out, nil
+}
+
+// pendingLoss requires loss to be what the claim's writer admits next: the
+// fence's derived record while the consumed fence's target writer awaits
+// its first admission, or else the op's unconsumed declaration; and either
+// to be this op's, this holder's, of this op's target, and the one the
+// record's rollback cites.
+func (r movesRecoveryOwners) pendingLoss(c *hostclaim.Claim, v recoveryProbe, loss hostclaim.LossRecord) error {
+ refused := func(why string) error { return fmt.Errorf("%w: %s", hostclaim.ErrLossRecord, why) }
+ if loss.OpID != v.op || loss.Box != c.Op.Intent["target"] || loss.HolderIncarnation != v.holder.Incarnation {
+ return refused("the loss is not this op's target's under this holder")
+ }
+ if fenced, pending := hostclaim.PendingMoveFenceLoss(c); pending {
+ ref, _ := hostclaim.ConsumedMoveTargetFence(c)
+ if loss.ID != fenced.ID || loss.Automatic == nil || ref.Key != v.rec.Move.Rollback.LossRef {
+ return refused("the loss is not the pending fence the record cites")
+ }
+ return nil
+ }
+ id, err := hostclaim.OperationLossID(c)
+ if err != nil {
+ return err
+ }
+ if id == "" || hostclaim.OperationLossConsumed(c, id) || loss.ID != id {
+ return refused("the loss is not the op's pending declaration")
+ }
+ if key, err := hostclaim.LossRecordKey(id); err != nil || key != v.rec.Move.Rollback.LossRef {
+ return refused("the record does not cite the pending declaration")
+ }
+ return nil
+}
+
+// frozenMaximum is the admission's frozen maximum: the restore point's
+// maximum, the restore point being the recovery's source baseline (src_gen,
+// as the first admission selects it, and the recorded one once recorded),
+// read again in this box's own bucket by genselect.GenerationMax. A read
+// that fails or differs from the selection waits.
+func (r movesRecoveryOwners) frozenMaximum(ctx context.Context, v recoveryProbe, baseline *genselect.Plan) (uint64, error) {
+ m := r.m
+ if baseline == nil || baseline.Generation != selectedGeneration(v.srcGen) {
+ return 0, fmt.Errorf("%w: the restore point is not the source baseline", hostbox.ErrChanged)
+ }
+ recorded, ok, err := r.recordedSourceBaseline(v.rec.Host, v)
+ if err != nil {
+ return 0, err
+ }
+ if ok && (recorded.Generation != baseline.Generation || recorded.MaxTXID != baseline.MaxTXID) {
+ return 0, fmt.Errorf("%w: the restore point is not the recorded source baseline", hostbox.ErrChanged)
+ }
+ max, err := genselect.GenerationMax(ctx, m.Data, m.Bucket, v.holder.Location, baseline.Generation)
+ if err != nil {
+ return 0, fmt.Errorf("%w: the source baseline's maximum: %v", hostbox.ErrChanged, err)
+ }
+ if max != baseline.MaxTXID {
+ return 0, fmt.Errorf("%w: the source baseline's maximum is %d, not the selection's %d", hostbox.ErrChanged, max, baseline.MaxTXID)
+ }
+ return max, nil
+}
+
+// sourceMaximum vouches for one frozen source (see the file comment) and
+// returns its maximum.
+func (r movesRecoveryOwners) sourceMaximum(c *hostclaim.Claim, v recoveryProbe, cut, frozen uint64, s hostclaim.WriteSource) (uint64, error) {
+ m := r.m
+ refused := func(why string) (uint64, error) {
+ return 0, fmt.Errorf("%w: frozen source %s/%s/%d: %s", hostclaim.ErrLossRecord, s.Box, s.Generation, s.Session, why)
+ }
+ kind, region, identity, err := hostops.ParseMoveLossBaseline(s.Baseline)
+ if err != nil {
+ return refused("the baseline does not parse")
+ }
+ switch kind {
+ case "replica":
+ if s.Reason != "target" || s.Box != c.Op.Intent["target"] || s.Generation == "" || identity != s.Generation || s.Session <= cut {
+ return refused("not a target session since the cut")
+ }
+ if !park.SameRegion(v.homeRegion, region, s.Region) || !park.SameRegion(v.homeRegion, s.Region, c.Op.Intent["to"]) {
+ return refused("not the target region's replica")
+ }
+ return frozen, nil
+ case "seal":
+ if s.Reason != "source-cut" || identity != v.op || s.Box != m.Box || selectedGeneration(s.Generation) != selectedGeneration(v.srcGen) || s.Session != cut ||
+ !park.SameRegion(v.homeRegion, s.Region, m.Region) {
+ return refused("not the source's cut session")
+ }
+ seal, err := journalledSeal(c)
+ if err != nil {
+ return 0, fmt.Errorf("%w: the cut seal: %v", hostbox.ErrChanged, err)
+ }
+ if seal.SrcCovered {
+ return refused("the cut seal covered the source's replica")
+ }
+ return seal.N, nil
+ }
+ return refused("an unknown baseline")
+}
+
+// lossEvidence is AdmissionEvidence's loss contract: the admission is the
+// writer's substitution of this observation's validated sources, which are
+// validated again. It consumes the validation.
+func (r movesRecoveryOwners) lossEvidence(ctx context.Context, o hostclaim.Observed, in hostclaim.Admission, baseline *genselect.Plan) error {
+ m := r.m
+ m.mu.Lock()
+ kept, ok := m.lossValidated[o.Host]
+ delete(m.lossValidated, o.Host)
+ m.mu.Unlock()
+ if !ok || kept.etag != o.ETag {
+ return fmt.Errorf("%w: no loss sources were validated for this observation", hostbox.ErrChanged)
+ }
+ valid, err := r.movesLossSources(ctx, o, kept.record, baseline)
+ if err != nil {
+ return err
+ }
+ if len(valid) != len(kept.valid) || (len(valid) != 0 && !reflect.DeepEqual(valid, kept.valid)) {
+ return fmt.Errorf("%w: the loss sources changed", hostbox.ErrChanged)
+ }
+ // The writer's substitution (writer_loss.go): the predecessor's
+ // validated maximum, the rest as additional declarations; a fence
+ // admits its one frozen target writer and nothing more.
+ h := o.Claim.Holder
+ p := hostclaim.AdmissionRecord{Box: h.WriterBox, Session: h.Session, Generation: h.Generation}
+ if a := h.Admissions; len(a) > 0 {
+ p = a[len(a)-1]
+ }
+ max := baseline.MaxTXID
+ var more []hostclaim.WritesLost
+ if _, fenced := hostclaim.PendingMoveFenceLoss(o.Claim); fenced {
+ if len(valid) != 1 {
+ return fmt.Errorf("%w: a fence freezes one target writer", hostclaim.ErrFence)
+ }
+ max = valid[0].RemoteMaxTXID
+ } else {
+ for _, s := range valid {
+ if s.Box == p.Box && s.Session == p.Session && s.Generation == p.Generation {
+ max = s.RemoteMaxTXID
+ } else {
+ more = append(more, hostclaim.WritesLost{Box: s.Box, Generation: s.Generation, Session: s.Session, RemoteMaxTXID: s.RemoteMaxTXID})
+ }
+ }
+ }
+ if in.RemoteMaxTXID != max || len(in.AdditionalWritesLost) != len(more) || (len(more) != 0 && !reflect.DeepEqual(in.AdditionalWritesLost, more)) {
+ return fmt.Errorf("%w: the admission is not the validated loss sources' substitution", hostbox.ErrChanged)
+ }
+ return nil
+}
diff --git a/internal/host/movesloss_test.go b/internal/host/movesloss_test.go
new file mode 100644
index 000000000..61ab154b4
--- /dev/null
+++ b/internal/host/movesloss_test.go
@@ -0,0 +1,758 @@
+package host
+
+import (
+ "context"
+ "crypto/ed25519"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "reflect"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostops"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/moves"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The loss and fence rules on the box (moves-after-Plan-12 Task 4: owed (a)
+// and (b), Plan 4 Task 10's box half; rulings Q5 as platform-1e answered
+// it, Q6, Q7's loss half, Q9's gate).
+
+const (
+ rcLossID = "loss_1"
+ // rcSealN is the cut seal's maximum: the stopped database's position,
+ // beyond the source replica's 30 when src_covered is false.
+ rcSealN = 35
+)
+
+// rcLossIntent is the moves recovery intent a declaration signs
+// (hostops.moveLossIntent's wire form).
+func (f *rcFixture) rcLossIntent(srcCovered bool) json.RawMessage {
+ raw, err := json.Marshal(map[string]any{"kind": "move-target-lost", "move": poOp, "incarnation": f.inc, "rollback": rpRB, "target": "uk-box", "source": "ie-box",
+ "target_region": "uk", "source_region": "ie", "record_key": "apps/bab/jam/instances/" + poHost + ".json", "record_etag": "etag-1", "record_seq": 1,
+ "record_state": "rolling_back", "dirty": false, "return_lane": false, "src_covered": srcCovered, "cut_session": rcS, "src_gen": "src"})
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ return raw
+}
+
+// rcCutSeal is the cut's seal the claim journals.
+func rcCutSeal(srcCovered bool) moves.Seal {
+ at := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
+ return moves.Seal{V: moves.SealVersion, N: rcSealN, FromDB: true, SrcCovered: srcCovered, SealedAt: at, LeaseUntil: at.Add(time.Minute),
+ Replica: []moves.SealedObject{{Key: "moves/" + poHost + "/" + poOp + "/replica/0009/0000000000000001-0000000000000023.ltx", VersionID: "s1", Size: 100}},
+ Files: []moves.SealedObject{{Key: "moves/" + poHost + "/" + poOp + "/files/archive.sealed", VersionID: "s2", Size: 10}, {Key: "moves/" + poHost + "/" + poOp + "/files/archive.json", VersionID: "s3", Size: 10}}}
+}
+
+// rcTargetSource is the frozen source of one target session since the cut.
+func rcTargetSource(gen string, session uint64) hostclaim.WriteSource {
+ return hostclaim.WriteSource{Reason: "target", Box: "uk-box", Generation: gen, Session: session, Region: "uk", Baseline: hostops.MoveLossReplicaBaseline + "uk/" + gen}
+}
+
+// rcCutSource is the frozen source of the source's cut session.
+func rcCutSource() hostclaim.WriteSource {
+ return hostclaim.WriteSource{Reason: "source-cut", Box: "ie-box", Generation: "src", Session: rcS, Region: "ie", Baseline: hostops.MoveLossSealBaseline + poOp}
+}
+
+// declare turns f's clean return into the return after a declared-lost
+// target (Plan 4 Task 10): the claim as DeclareLost and the lost target's
+// release leave it, edit applied, the record's rollback citing the
+// declaration, and the root-signed declaration committed with sources. It
+// returns the root's public key.
+func (f *rcFixture) declare(srcCovered bool, sources []hostclaim.WriteSource, edit func(*hostclaim.Claim)) ed25519.PublicKey {
+ t := f.t
+ t.Helper()
+ c := f.recoveryClaim()
+ intent := f.rcLossIntent(srcCovered)
+ c.Lost, c.LossRefs, c.Op.Participants = []string{"uk-box"}, []string{rcLossID}, []string{"ie-box"}
+ c.Op.Journal["loss."+rcLossID] = string(intent)
+ c.Op.Journal["loss_gate."+rcLossID] = `{"box":"uk-box","revoke_execution":true}`
+ c.Op.Journal["lost_target_released"] = rcLossID
+ c.Op.Journal[hostclaim.JournalSeal] = rcCutSeal(srcCovered).Encode()
+ if edit != nil {
+ edit(c)
+ }
+ f.putClaim(c)
+ key, err := hostclaim.LossRecordKey(rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.rec.Move.Rollback.LossRef, f.rec.Move.Rollback.Evidence = key, key
+ pub, priv, err := ed25519.GenerateKey(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ now := time.Now().UTC()
+ r := hostclaim.LossRecord{V: 1, ID: rcLossID, State: "prepared", Actor: "operator", AcceptedAt: now, Host: poHost, HolderIncarnation: f.inc, HolderStamp: rcS, HolderSession: c.Holder.Session,
+ ExpectedClaimGen: c.Gen, ExpectedClaimETag: "claim-etag", OpID: poOp, OpKind: hostclaim.KindRestoring, OpPhase: hostclaim.PhaseReleasing, Box: "uk-box", Enrollment: "enroll", QuarantineRevision: 1,
+ CloudObservation: hostclaim.LossCloudObservation{ObservedAt: now, Account: "123", Region: "eu-west-2", InstanceID: "i-1", State: "terminated", RequestID: "request"},
+ OperatorAttestation: hostclaim.LossOperatorAttestation{MaintainComputeFence: true, ColdQuarantineStartOnly: true}, FencedAt: now,
+ ConfirmedConsequences: hostclaim.LossConsequences{RevokeExecution: true, RemoveParticipation: true, RecoveryIntent: intent},
+ Timers: hostclaim.TimerLoss{Knowledge: hostclaim.TimersUnknown}, WriteSources: sources}
+ created, err := hostclaim.CreateLossRecord(t.Context(), f.claims, release.RootSigner(priv), r)
+ if err != nil {
+ t.Fatal(err)
+ }
+ r.State, r.ResultClaimGen = "committed", c.Gen
+ if _, err := hostclaim.ReplaceLossRecord(t.Context(), f.claims, release.RootSigner(priv), created, r); err != nil {
+ t.Fatal(err)
+ }
+ return pub
+}
+
+// rcFenceEnrollment is the fenced target's enrollment.
+const rcFenceEnrollment = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+
+// fence turns f's clean return into the return after the target's
+// termination fence (spec rev 7.20 §7.10): the consumed fence, the
+// automatic loss of the target's session 20 in poOp, the record's rollback
+// citing the fence. It returns the derived record the writer passes.
+func (f *rcFixture) fence() hostclaim.LossRecord {
+ t := f.t
+ t.Helper()
+ c := f.recoveryClaim()
+ ref := hostclaim.FenceRef{Enrollment: rcFenceEnrollment, Key: "control/box-fences/" + rcFenceEnrollment + ".json", SHA256: strings.Repeat("b", 64)}
+ raw, err := json.Marshal(ref)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256([]byte(poOp + "\x00uk-box\x00" + ref.Enrollment + "\x00" + ref.SHA256))
+ id := "auto-" + hex.EncodeToString(sum[:16])
+ c.Op.Intent["target_enrollment"] = ref.Enrollment
+ c.Op.Journal["fence.uk-box"], c.Op.Journal["fenced_target_released"] = string(raw), string(raw)
+ c.Op.Journal[hostclaim.JournalSeal] = rcCutSeal(true).Encode()
+ c.Op.Participants, c.Lost, c.LossRefs = []string{"ie-box"}, []string{"uk-box"}, []string{id}
+ c.EvidenceRefs = []hostclaim.AutomaticLoss{{ID: id, Box: "uk-box", Enrollment: ref.Enrollment, Fence: ref, ObservedAt: time.Date(2026, 10, 1, 1, 0, 0, 0, time.UTC).Format(time.RFC3339Nano),
+ Op: poOp, Kind: hostclaim.KindLanding, Phase: hostclaim.PhaseLandingProbing, Role: "target", Incarnation: f.inc, Stamp: 20, Session: 20, Generation: poOp, Timers: hostclaim.TimersUnknown}}
+ f.putClaim(c)
+ f.rec.Move.Rollback.LossRef, f.rec.Move.Rollback.Evidence = ref.Key, ref.Key
+ record, ok := hostclaim.PendingMoveFenceLoss(c)
+ if !ok {
+ t.Fatal("no pending fence loss")
+ }
+ return record
+}
+
+// lossWriter is the claim's writer with the moves loss owner and baseline
+// validator, over the source's own bucket.
+func (f *rcFixture) lossWriter(pub ed25519.PublicKey) hostbox.Writer {
+ return hostbox.Writer{Claims: f.claims, Data: f.bucket, Bucket: "data", Region: "ie", Box: "ie-box", AdmissionEvidence: f.owners.AdmissionEvidence,
+ Loss: &hostbox.LossAdmission{PublicKey: pub, Owner: hostops.MoveLossRecoveryOwner(), Sources: f.owners.LossSources}}
+}
+
+// Q6: a return after a declared loss is never covered: the probe's request
+// asks no coverage, the request matcher refuses clean-return for it, and the
+// recovery owners select and admit it uncovered.
+func TestDeclaredLossNeverAdmitsCleanReturn(t *testing.T) {
+ t.Run("the probing pass asks no coverage", func(t *testing.T) {
+ h := newRecoverHarness(t, false)
+ h.released(t)
+ h.probe = func(recoveryProbeRequest) (recoveryProbeOutcome, error) { return recoveryProbeWaiting, nil }
+ h.srcPass(t)
+ h.sourceAt(t, provision.RbSourceProbing)
+ if len(h.probes) != 1 || h.probes[0].Coverage != recoveryCoverageClean || h.probes[0].Declared {
+ t.Fatalf("before the loss: %+v", h.probes)
+ }
+ key := rcDeclareOnHarness(t, h)
+ h.srcPass(t)
+ if len(h.probes) != 2 {
+ t.Fatalf("probes %+v", h.probes)
+ }
+ req := h.probes[1]
+ if !req.Declared || !req.Clean || req.Coverage != "" || req.Generation != "" || req.SourceGeneration == "" {
+ t.Fatalf("a declared return's request: %+v", req)
+ }
+ rec := h.read(t, p1Host)
+ if rec.Move.Rollback.LossRef != key || recoveryProbeMatches(rec, req) != nil {
+ t.Fatalf("the request does not match its record: %v", recoveryProbeMatches(rec, req))
+ }
+ for name, mutate := range map[string]func(*recoveryProbeRequest){
+ "clean-return": func(r *recoveryProbeRequest) { r.Coverage = recoveryCoverageClean },
+ "sealed": func(r *recoveryProbeRequest) { r.Coverage = recoveryCoverageSealed },
+ "not declared": func(r *recoveryProbeRequest) { r.Declared = false },
+ "undeclared kind": func(r *recoveryProbeRequest) { r.Declared, r.Coverage = false, recoveryCoverageClean },
+ "dirty": func(r *recoveryProbeRequest) { r.Clean, r.Generation = false, r.RollbackID },
+ } {
+ bad := req
+ mutate(&bad)
+ if err := recoveryProbeMatches(rec, bad); !errors.Is(err, errRecoveryProbeRequest) {
+ t.Fatalf("%s: %v", name, err)
+ }
+ }
+ // A request that says declared for a record that does not is refused.
+ undeclared := rec
+ rb := *rec.Move.Rollback
+ rb.LossRef = ""
+ mv := *rec.Move
+ mv.Rollback = &rb
+ undeclared.Move = &mv
+ if err := recoveryProbeMatches(undeclared, req); !errors.Is(err, errRecoveryProbeRequest) {
+ t.Fatalf("declared over an undeclared record: %v", err)
+ }
+ })
+ t.Run("the owners admit it uncovered, through the loss owner", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ if in.Admission.Coverage != "" || in.Admission.LossID != "" {
+ t.Fatalf("admission %+v", in.Admission)
+ }
+ w := f.lossWriter(pub)
+ covered := in.Admission
+ covered.Coverage = recoveryCoverageClean
+ if _, err := w.Admit(t.Context(), o, covered, in.Baseline); err == nil {
+ t.Fatal("a declared return admitted clean-return")
+ }
+ admitted, err := w.Admit(t.Context(), f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ if a.Coverage != "" || a.LossID != rcLossID || a.Covered || !hostclaim.OperationLossConsumed(admitted.Claim, rcLossID) {
+ t.Fatalf("the loss admission: %+v", a)
+ }
+ })
+}
+
+// rcDeclareOnHarness declares the recover harness's target lost: the claim
+// journals the declaration and the record's rollback cites its key. It
+// returns the key.
+func rcDeclareOnHarness(t *testing.T, h *recoverHarness) string {
+ t.Helper()
+ key, err := hostclaim.LossRecordKey(rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rewriteP1Claim(t, h.s, func(c *hostclaim.Claim) {
+ c.LossRefs = append(c.LossRefs, rcLossID)
+ c.Op.Journal["loss."+rcLossID] = `{"kind":"move-target-lost","target":"uk-box"}`
+ })
+ rcSetLossRef(t, h, key)
+ return key
+}
+
+// rcSetLossRef writes the record's rollback LossRef through provision's
+// record writer.
+func rcSetLossRef(t *testing.T, h *recoverHarness, ref string) {
+ t.Helper()
+ if _, err := provision.SetRecordMove(t.Context(), h.s, "bab", "app", p1Host, func(r *provision.Record) error {
+ r.Move.Rollback.LossRef = ref
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Q6: a return after the target's termination fence asks no coverage, and
+// its first admission takes the fenced target's frozen maximum.
+func TestFencedTargetReturnAsksUncovered(t *testing.T) {
+ t.Run("the probing pass asks no coverage", func(t *testing.T) {
+ h := newRecoverHarness(t, false)
+ h.released(t)
+ h.probe = func(recoveryProbeRequest) (recoveryProbeOutcome, error) { return recoveryProbeWaiting, nil }
+ h.srcPass(t)
+ ref := hostclaim.FenceRef{Enrollment: rcFenceEnrollment, Key: "control/box-fences/" + rcFenceEnrollment + ".json", SHA256: strings.Repeat("b", 64)}
+ raw, _ := json.Marshal(ref)
+ rewriteP1Claim(t, h.s, func(c *hostclaim.Claim) { c.Op.Journal["fence.uk-box"] = string(raw) })
+ rcSetLossRef(t, h, ref.Key)
+ h.srcPass(t)
+ if len(h.probes) != 2 || !h.probes[1].Declared || h.probes[1].Coverage != "" {
+ t.Fatalf("probes %+v", h.probes)
+ }
+ })
+ t.Run("the owners admit the frozen target", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ f.fence()
+ o := f.read()
+ in := f.selectAdmission(o)
+ if in.Admission.Coverage != "" {
+ t.Fatalf("a fenced return asked %q", in.Admission.Coverage)
+ }
+ covered := in.Admission
+ covered.Coverage = recoveryCoverageClean
+ w := f.lossWriter(nil)
+ if _, err := w.Admit(t.Context(), o, covered, in.Baseline); err == nil {
+ t.Fatal("a fenced return admitted clean-return")
+ }
+ admitted, err := w.Admit(t.Context(), f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ // The target's replica is in its own region, which this box never
+ // reads (platform-1e): the declaration names the admission's frozen
+ // maximum, src_gen's 30 in this box's own bucket.
+ if a.Coverage != "" || a.WritesLost == nil || *a.WritesLost != (hostclaim.WritesLost{Box: "uk-box", Generation: poOp, Session: 20, RemoteMaxTXID: 30}) || len(a.AdditionalWritesLost) != 0 {
+ t.Fatalf("the fenced admission: %+v", a)
+ }
+ })
+}
+
+// Q6: every later session of the recovered source stays uncovered: the
+// loss is consumed by the first admission, but the declaration (or the
+// fence) is still the op's, so a restart asks, selects and admits no
+// coverage, and the claim refuses any.
+func TestLaterSourceSessionStaysUncovered(t *testing.T) {
+ t.Run("the probing pass", func(t *testing.T) {
+ h := newRecoverHarness(t, false)
+ h.released(t)
+ h.probe = func(recoveryProbeRequest) (recoveryProbeOutcome, error) { return recoveryProbeWaiting, nil }
+ h.srcPass(t)
+ rcDeclareOnHarness(t, h)
+ rewriteP1Claim(t, h.s, func(c *hostclaim.Claim) {
+ c.Op.Journal["loss_admitted."+rcLossID] = strconv.FormatUint(c.Holder.Session, 10)
+ })
+ h.srcPass(t)
+ if len(h.probes) != 2 || !h.probes[1].Declared || h.probes[1].Coverage != "" {
+ t.Fatalf("probes %+v", h.probes)
+ }
+ })
+ for _, path := range []string{"declared", "fenced"} {
+ t.Run("the owners: "+path, func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ var w hostbox.Writer
+ if path == "declared" {
+ w = f.lossWriter(f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil))
+ } else {
+ f.fence()
+ w = f.lossWriter(nil)
+ }
+ in := f.selectAdmission(f.read())
+ first, err := w.Admit(t.Context(), f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if pending, err := recoveryLossPending(first.Claim); err != nil || pending {
+ t.Fatalf("the loss is still pending: %v %v", pending, err)
+ }
+ restart := f.selectAdmission(first)
+ if restart.Admission.Coverage != "" || restart.Admission.ObservedGeneration != in.Admission.NewGeneration {
+ t.Fatalf("the restart: %+v", restart.Admission)
+ }
+ covered := restart.Admission
+ covered.Coverage = recoveryCoverageClean
+ if err := f.owners.AdmissionEvidence(t.Context(), first, covered, restart.Baseline); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("a later session admitted clean-return: %v", err)
+ }
+ again, err := w.Admit(t.Context(), first, restart.Admission, restart.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := again.Claim.Holder.Admissions[len(again.Claim.Holder.Admissions)-1]
+ if a.Coverage != "" || a.LossID != "" || a.Generation != restart.Admission.NewGeneration {
+ t.Fatalf("the restart's admission: %+v", a)
+ }
+ })
+ }
+}
+
+// Q6: the record and the claim must agree on the target's loss, by
+// identity; until they do the probe waits: no request is made, the owners
+// select nothing, and the recovery stays at probing.
+func TestRecordAndClaimDisagreeWaits(t *testing.T) {
+ t.Run("the probing pass", func(t *testing.T) {
+ for name, disagree := range map[string]func(t *testing.T, h *recoverHarness){
+ "the record alone": func(t *testing.T, h *recoverHarness) { rcSetLossRef(t, h, "control/host-loss/"+rcLossID+".json") },
+ "the claim alone": func(t *testing.T, h *recoverHarness) { rcDeclareOnHarness(t, h); rcSetLossRef(t, h, "") },
+ "another loss": func(t *testing.T, h *recoverHarness) {
+ rcDeclareOnHarness(t, h)
+ rcSetLossRef(t, h, "control/host-loss/loss_2.json")
+ },
+ "a fence's for loss": func(t *testing.T, h *recoverHarness) {
+ rcDeclareOnHarness(t, h)
+ rcSetLossRef(t, h, "control/box-fences/x.json")
+ },
+ } {
+ t.Run(name, func(t *testing.T) {
+ h := newRecoverHarness(t, false)
+ h.released(t)
+ h.probe = func(recoveryProbeRequest) (recoveryProbeOutcome, error) { return recoveryProbeWaiting, nil }
+ h.srcPass(t)
+ disagree(t, h)
+ h.srcPass(t)
+ if len(h.probes) != 1 {
+ t.Fatalf("probed while the record and the claim disagree: %+v", h.probes)
+ }
+ h.sourceAt(t, provision.RbSourceProbing)
+ })
+ }
+ })
+ t.Run("the owners", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil)
+ for name, ref := range map[string]string{"no loss": "", "another loss": "control/host-loss/loss_2.json"} {
+ f.rec.Move.Rollback.LossRef = ref
+ if _, err := f.owners.Select(t.Context(), f.route, f.read(), f.policy()); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("%s: %v", name, err)
+ }
+ }
+ })
+ t.Run("the adapter", func(t *testing.T) {
+ f := newRecoveryFixture(t)
+ inc := plan10J2AdapterClaim(t, f, hostclaim.PhaseProbing, "src-gen")
+ rec := plan10J2AdapterRecord(inc, false)
+ var events []string
+ req := plan10J2AdapterRequest(&rec, "src-gen", &events)
+ a := recoveryProbeAdapter{Box: "box-1", Claims: true,
+ Wake: func(context.Context, registry.Route, activator.StartPolicy) error {
+ events = append(events, "wake")
+ return nil
+ },
+ Route: func(host string) (*registry.Route, error) { return &registry.Route{Host: host}, nil },
+ Current: func(context.Context, release.Store, provision.Record) (provision.MoveInputs, error) {
+ return plan10J2AdapterInputs(), nil
+ }}
+ // The claim journals a declaration the record does not cite.
+ if _, err := hostclaim.Update(t.Context(), f.claims, f.read(), func(c *hostclaim.Claim) error {
+ c.LossRefs = append(c.LossRefs, rcLossID)
+ c.Op.Journal["loss."+rcLossID] = `{"kind":"move-target-lost","target":"` + c.Op.Intent["target"] + `"}`
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if out, err := a.probe(t.Context(), f.claims, rec, req); !errors.Is(err, hostbox.ErrChanged) || out != recoveryProbeWaiting || len(events) != 0 {
+ t.Fatalf("%v %v %v", out, err, events)
+ }
+ })
+}
+
+// Q5 as platform-1e answered it: the loss baseline validator vouches for
+// each frozen source from the maxima already recorded, never reading
+// another region's bucket: a target replica's is the admission's frozen
+// maximum (the source baseline's, read in this box's own bucket by
+// genselect.GenerationMax), a seal's the journalled cut seal's.
+func TestLossSourcesValidatesEachFrozenBaseline(t *testing.T) {
+ ctx := context.Background()
+ t.Run("a declared loss's replica and seal", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ sources := []hostclaim.WriteSource{rcTargetSource(poOp, 20), rcCutSource()}
+ pub := f.declare(false, sources, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ loss, err := hostclaim.ReadLossRecord(ctx, f.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ got, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []hostclaim.ValidatedWriteSource{
+ {Reason: "target", Box: "uk-box", Generation: poOp, Session: 20, Region: "uk", Baseline: "replica:uk/" + poOp, RemoteMaxTXID: 30},
+ {Reason: "source-cut", Box: "ie-box", Generation: "src", Session: rcS, Region: "ie", Baseline: "seal:" + poOp, RemoteMaxTXID: rcSealN},
+ }
+ if !reflect.DeepEqual(got, want) {
+ t.Fatalf("validated %+v\nwant %+v", got, want)
+ }
+ for name, mutate := range map[string]func(*hostclaim.WriteSource){
+ "another target box": func(s *hostclaim.WriteSource) { s.Box = "de-box" },
+ "a replica of another gen": func(s *hostclaim.WriteSource) { s.Baseline = "replica:uk/other" },
+ "a replica of another region": func(s *hostclaim.WriteSource) { s.Baseline = "replica:de/" + poOp },
+ "a target outside its region": func(s *hostclaim.WriteSource) { s.Region, s.Baseline = "de", "replica:de/"+poOp },
+ "a session before the cut": func(s *hostclaim.WriteSource) { s.Session = rcS },
+ "a target named source-cut": func(s *hostclaim.WriteSource) { s.Reason = "source-cut" },
+ "a malformed baseline": func(s *hostclaim.WriteSource) { s.Baseline = "replica:uk" },
+ "an unknown baseline": func(s *hostclaim.WriteSource) { s.Baseline = "snapshot:x" },
+ } {
+ bad := loss.Record
+ bad.WriteSources = append([]hostclaim.WriteSource(nil), sources...)
+ mutate(&bad.WriteSources[0])
+ if _, err := f.owners.LossSources(ctx, o, bad, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("target, %s: %v", name, err)
+ }
+ }
+ for name, mutate := range map[string]func(*hostclaim.WriteSource){
+ "another move's seal": func(s *hostclaim.WriteSource) { s.Baseline = "seal:mv_other" },
+ "another box": func(s *hostclaim.WriteSource) { s.Box = "uk-box" },
+ "another generation": func(s *hostclaim.WriteSource) { s.Generation = "other" },
+ "another session": func(s *hostclaim.WriteSource) { s.Session = rcS + 1 },
+ "another region": func(s *hostclaim.WriteSource) { s.Region = "uk" },
+ "a seal named target": func(s *hostclaim.WriteSource) { s.Reason = "target" },
+ "an unknown reason": func(s *hostclaim.WriteSource) { s.Reason = "other" },
+ "a replica for the cut seal": func(s *hostclaim.WriteSource) { s.Baseline = "replica:ie/src" },
+ } {
+ bad := loss.Record
+ bad.WriteSources = append([]hostclaim.WriteSource(nil), sources...)
+ mutate(&bad.WriteSources[1])
+ if _, err := f.owners.LossSources(ctx, o, bad, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("seal, %s: %v", name, err)
+ }
+ }
+ // A seal that covered the source's replica holds no source-cut
+ // writes: a source-cut source under it is not the cut's.
+ covered := newRCFixture(t, rcClean)
+ pub = covered.declare(true, sources, nil)
+ o = covered.read()
+ in = covered.selectAdmission(o)
+ loss, err = hostclaim.ReadLossRecord(ctx, covered.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := covered.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("a source-cut under a covering seal: %v", err)
+ }
+ })
+ t.Run("the record is the pending declaration", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ loss, err := hostclaim.ReadLossRecord(ctx, f.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for name, mutate := range map[string]func(*hostclaim.LossRecord){
+ "another id": func(r *hostclaim.LossRecord) { r.ID = "loss_2" },
+ "another op": func(r *hostclaim.LossRecord) { r.OpID = "mv_other" },
+ "another box": func(r *hostclaim.LossRecord) { r.Box = "de-box" },
+ "another incarnation": func(r *hostclaim.LossRecord) { r.HolderIncarnation = "other" },
+ } {
+ bad := loss.Record
+ mutate(&bad)
+ if _, err := f.owners.LossSources(ctx, o, bad, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("%s: %v", name, err)
+ }
+ }
+ // The baseline must be the source baseline the recovery restores.
+ other := *in.Baseline
+ other.Generation = poOp
+ if _, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, &other); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("another restore point: %v", err)
+ }
+ if _, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, nil); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("no restore point: %v", err)
+ }
+ })
+ t.Run("the automatic fence record", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ record := f.fence()
+ o := f.read()
+ in := f.selectAdmission(o)
+ got, err := f.owners.LossSources(ctx, o, record, in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []hostclaim.ValidatedWriteSource{{Reason: "target", Box: "uk-box", Generation: poOp, Session: 20, Region: "uk", Baseline: "replica:uk/" + poOp, RemoteMaxTXID: 30}}
+ if !reflect.DeepEqual(got, want) {
+ t.Fatalf("validated %+v", got)
+ }
+ bad := record
+ bad.ID = "auto-other"
+ if _, err := f.owners.LossSources(ctx, o, bad, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("another automatic loss: %v", err)
+ }
+ })
+ t.Run("only a first recovery admission's loss", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ loss, err := hostclaim.ReadLossRecord(ctx, f.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := f.owners.LossSources(ctx, hostclaim.Observed{Host: poHost, Absent: true}, loss.Record, in.Admission, in.Baseline); err == nil {
+ t.Fatal("no claim")
+ }
+ // No loss pending: the validator vouches for nothing.
+ undeclared := newRCFixture(t, rcClean)
+ uo := undeclared.read()
+ if _, err := undeclared.owners.LossSources(ctx, uo, loss.Record, in.Admission, in.Baseline); !errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("no pending loss: %v", err)
+ }
+ })
+}
+
+// failingList is the source's own bucket, unreadable for listings.
+type failingList struct{ genselect.Reader }
+
+func (failingList) ListVersions(context.Context, string, string) ([]lease.ObjectVersion, error) {
+ return nil, errors.New("SlowDown")
+}
+
+// An unreadable source maximum is a wait, never a refusal (Q5): with no
+// cross-region read the one read is the admission's frozen maximum in this
+// box's own bucket, and while it cannot be read, or reads otherwise than
+// the selection, the validator answers hostbox.ErrChanged, which the probe
+// adapter turns into another pass.
+func TestLossSourcesWaitsOnAnUnreadableTarget(t *testing.T) {
+ ctx := context.Background()
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(true, []hostclaim.WriteSource{rcTargetSource(poOp, 20)}, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ loss, err := hostclaim.ReadLossRecord(ctx, f.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ data := f.owners.Data
+ f.owners.Data = failingList{data}
+ _, err = f.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline)
+ if !errors.Is(err, hostbox.ErrChanged) || errors.Is(err, hostclaim.ErrLossRecord) {
+ t.Fatalf("an unreadable bucket: %v", err)
+ }
+ f.owners.Data = data
+ // The remote moved past the selection: wait for the next selection.
+ f.bucket.Put("data", "instances/"+poHost+"/g-src/0000/0000000000000031-0000000000000031.ltx", f.ltxHeader(31, 31))
+ if _, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("a moved remote: %v", err)
+ }
+}
+
+// Q7's loss half: the recovery's AdmissionEvidence accepts the writer's
+// loss substitution exactly: RemoteMaxTXID is the validated predecessor's
+// maximum and AdditionalWritesLost the other validated sources, from the
+// same LossSources call; anything else is refused. It journals the
+// lineage link and the source baseline before the CAS, as every recovery
+// admission does.
+func TestRecoveryAdmissionEvidenceAcceptsTheFrozenMaximum(t *testing.T) {
+ ctx := context.Background()
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(false, []hostclaim.WriteSource{rcTargetSource(poOp, 20), rcCutSource()}, nil)
+ o := f.read()
+ in := f.selectAdmission(o)
+ loss, err := hostclaim.ReadLossRecord(ctx, f.claims, pub, rcLossID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Without the validator's call for this observation, nothing vouches.
+ if err := f.owners.AdmissionEvidence(ctx, o, in.Admission, in.Baseline); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("no validated sources: %v", err)
+ }
+ substituted := in.Admission
+ substituted.RemoteMaxTXID = 30
+ substituted.AdditionalWritesLost = []hostclaim.WritesLost{{Box: "ie-box", Generation: "src", Session: rcS, RemoteMaxTXID: rcSealN}}
+ for name, mutate := range map[string]func(*hostclaim.Admission){
+ "Select's maximum": func(a *hostclaim.Admission) { a.RemoteMaxTXID = in.Baseline.MaxTXID + 1 },
+ "no additional": func(a *hostclaim.Admission) { a.AdditionalWritesLost = nil },
+ "another additional maximum": func(a *hostclaim.Admission) { a.AdditionalWritesLost[0].RemoteMaxTXID = 30 },
+ "an extra additional": func(a *hostclaim.Admission) {
+ a.AdditionalWritesLost = append(a.AdditionalWritesLost, hostclaim.WritesLost{Box: "uk-box", Generation: poOp, Session: 20, RemoteMaxTXID: 30})
+ },
+ "coverage": func(a *hostclaim.Admission) { a.Coverage = recoveryCoverageClean },
+ "a loss id": func(a *hostclaim.Admission) { a.LossID = rcLossID },
+ "kept": func(a *hostclaim.Admission) { a.KeepStamp = true },
+ } {
+ if _, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline); err != nil {
+ t.Fatal(err)
+ }
+ bad := substituted
+ bad.AdditionalWritesLost = append([]hostclaim.WritesLost(nil), substituted.AdditionalWritesLost...)
+ mutate(&bad)
+ if err := f.owners.AdmissionEvidence(ctx, o, bad, in.Baseline); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("%s: %v", name, err)
+ }
+ }
+ // A validation for an earlier observation does not vouch for this one.
+ if _, err := f.owners.LossSources(ctx, o, loss.Record, in.Admission, in.Baseline); err != nil {
+ t.Fatal(err)
+ }
+ stale := o
+ stale.ETag = "another"
+ if err := f.owners.AdmissionEvidence(ctx, stale, substituted, in.Baseline); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("another observation: %v", err)
+ }
+ // Through the claim's writer, with the moves loss owner: the
+ // substitution passes, and the lineage and the source baseline are
+ // journalled before the CAS.
+ admitted, err := f.lossWriter(pub).Admit(ctx, f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ if a.WritesLost == nil || *a.WritesLost != (hostclaim.WritesLost{Box: "uk-box", Generation: poOp, Session: 20, RemoteMaxTXID: 30}) ||
+ !reflect.DeepEqual(a.AdditionalWritesLost, substituted.AdditionalWritesLost) || a.LossID != rcLossID {
+ t.Fatalf("the loss admission: %+v", a)
+ }
+ j, err := moves.ReadJournal(f.dataDir, poHost, rpRB)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, ok := j.Steps[rcStepLineage+a.Generation]; !ok {
+ t.Fatalf("no lineage link for %s: %v", a.Generation, j.Steps)
+ }
+ if _, ok := j.Steps[rcStepSourceBaseline]; !ok {
+ t.Fatalf("no source baseline: %v", j.Steps)
+ }
+ // The admitted session is the recovery's own.
+ v, err := movesRecoveryOwners{f.owners}.recovery(ctx, admitted)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := (movesRecoveryOwners{f.owners}).lineage(poHost, v); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Plan 4 Task 10 (witnesses wTwoSessionDeclaration,
+// wNoDuplicateDeclarationEarly/Late): behind a lost target the source's
+// recovery declares every target session since the cut and, when the seal
+// did not cover the source's replica, the cut session, each once: the
+// predecessor's as writes_lost, the rest as additional_writes_lost.
+func TestLostTargetReturnDeclaresEverySessionOnce(t *testing.T) {
+ ctx := context.Background()
+ t.Run("two target sessions and the cut", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ sources := []hostclaim.WriteSource{rcTargetSource("t2", 25), rcTargetSource(poOp, 20), rcCutSource()}
+ pub := f.declare(false, sources, func(c *hostclaim.Claim) {
+ // The target's probe session closed and a restart died.
+ c.Holder.Session = 25
+ c.Holder.Admissions = append(c.Holder.Admissions, hostclaim.AdmissionRecord{Session: 25, Box: "uk-box", Generation: "t2", Predecessor: poOp, Region: "uk", PredecessorRegion: "uk", Mode: "cold"})
+ })
+ in := f.selectAdmission(f.read())
+ admitted, err := f.lossWriter(pub).Admit(ctx, f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ want := []hostclaim.WritesLost{{Box: "uk-box", Generation: poOp, Session: 20, RemoteMaxTXID: 30}, {Box: "ie-box", Generation: "src", Session: rcS, RemoteMaxTXID: rcSealN}}
+ if a.WritesLost == nil || *a.WritesLost != (hostclaim.WritesLost{Box: "uk-box", Generation: "t2", Session: 25, RemoteMaxTXID: 30}) || !reflect.DeepEqual(a.AdditionalWritesLost, want) {
+ t.Fatalf("declared %+v / %+v", a.WritesLost, a.AdditionalWritesLost)
+ }
+ })
+ t.Run("a target lost before its first admission", func(t *testing.T) {
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(false, []hostclaim.WriteSource{rcCutSource()}, func(c *hostclaim.Claim) {
+ c.Holder.Session, c.Holder.WriterBox, c.Holder.Admissions = rcS, "ie-box", c.Holder.Admissions[:1]
+ })
+ in := f.selectAdmission(f.read())
+ admitted, err := f.lossWriter(pub).Admit(ctx, f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ if a.WritesLost == nil || *a.WritesLost != (hostclaim.WritesLost{Box: "ie-box", Generation: "src", Session: rcS, RemoteMaxTXID: rcSealN}) || len(a.AdditionalWritesLost) != 0 {
+ t.Fatalf("the cut session, once: %+v / %+v", a.WritesLost, a.AdditionalWritesLost)
+ }
+ })
+ t.Run("a target lost before its first admission, the replica covered", func(t *testing.T) {
+ // Nothing is frozen: the claim records its own predecessor at the
+ // restore point, and moves vouches for an empty set.
+ f := newRCFixture(t, rcClean)
+ pub := f.declare(true, nil, func(c *hostclaim.Claim) {
+ c.Holder.Session, c.Holder.WriterBox, c.Holder.Admissions = rcS, "ie-box", c.Holder.Admissions[:1]
+ })
+ in := f.selectAdmission(f.read())
+ admitted, err := f.lossWriter(pub).Admit(ctx, f.read(), in.Admission, in.Baseline)
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ if a.LossID != rcLossID || len(a.AdditionalWritesLost) != 0 || (a.WritesLost != nil && a.WritesLost.RemoteMaxTXID != 30) {
+ t.Fatalf("no frozen source: %+v", a)
+ }
+ })
+}
diff --git a/internal/host/movesprobeowners.go b/internal/host/movesprobeowners.go
index d3f257f4b..7fda90e53 100644
--- a/internal/host/movesprobeowners.go
+++ b/internal/host/movesprobeowners.go
@@ -23,6 +23,7 @@ import (
"github.com/carlosframework/platform/internal/hostbox"
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostops"
"github.com/carlosframework/platform/internal/hostwrite"
"github.com/carlosframework/platform/internal/moves"
"github.com/carlosframework/platform/internal/moves/archive"
@@ -151,6 +152,10 @@ type movesProbeOwners struct {
// fences are the admitted spawn selections Materialize ran under, by
// host and session: the live fence for the session's marker writes.
fences map[string]hostwrite.Work
+ // lossValidated is each host's last LossSources answer, which the
+ // recovery's AdmissionEvidence of the same admission consumes
+ // (movesloss.go).
+ lossValidated map[string]lossValidation
}
var (
@@ -176,12 +181,15 @@ func (m *movesProbeOwners) owners() ClaimKindOwners {
return ClaimKindOwners{Inputs: m, AdmissionEvidence: m.AdmissionEvidence, PublicationEvidence: m.PublicationEvidence}
}
-// MovesOwners is ClaimMovesOwners for the probe: this owner set and ops.
-// ProbeReady stays nil, so the claim binds its own probe readiness check
-// (claimProbeSocketReady: the declared path, 2xx only, a peer in the
-// started role's cgroup; H12).
+// MovesOwners is ClaimMovesOwners for the probe: this owner set and ops,
+// with the moves loss owner (the resolve owner's signed-intent policy,
+// hostops.MoveLossRecoveryOwner) and the loss baseline validator
+// (LossSources, movesloss.go) the claim's writer admits a declared or fenced
+// return through. ProbeReady stays nil, so the claim binds its own probe
+// readiness check (claimProbeSocketReady: the declared path, 2xx only, a
+// peer in the started role's cgroup; H12).
func (m *movesProbeOwners) MovesOwners(ops OperationAttempts) ClaimMovesOwners {
- return ClaimMovesOwners{ClaimKindOwners: m.owners(), Operations: ops}
+ return ClaimMovesOwners{LossRecovery: hostops.MoveLossRecoveryOwner(), LossSources: m.LossSources, ClaimKindOwners: m.owners(), Operations: ops}
}
func (m *movesProbeOwners) valid() bool {
diff --git a/internal/host/movesrecover.go b/internal/host/movesrecover.go
index 3f98ce8f3..3912124d4 100644
--- a/internal/host/movesrecover.go
+++ b/internal/host/movesrecover.go
@@ -140,9 +140,14 @@ type recoveryProbeRequest struct {
// generation and journal.
RollbackID string
// Clean is the return's kind; Coverage is the admission's coverage,
- // "sealed" (dirty) or "clean-return".
+ // "sealed" (dirty), "clean-return", or "" for a declared return.
Clean bool
Coverage string
+ // Declared is a clean return after a declared-lost or fenced target
+ // (Q6): the record's rollback cites the loss the claim journals. It asks
+ // no coverage, for every session, and carries no loss id: the claim's
+ // writer derives it.
+ Declared bool
// Generation is the generation the dirty return installed (the
// rollback id); empty for a clean return, whose probe admits a fresh
// one restored from SourceGeneration (src_gen, J14).
@@ -631,9 +636,19 @@ func (r *recoverRun) probing(ctx context.Context) (bool, error) {
_, err := provision.SetRecordMove(ctx, r.s, r.rec.Account, r.rec.App, host, provision.MoveRecordRecoveryInputs(r.g, r.rb, in))
return err
}}
- if rb.Dirty {
+ // A declared or fenced return asks no coverage (Q6); while the record
+ // and the claim disagree on the target's loss the probe waits.
+ declared, err := recoveryDeclared(obs.Claim, rb.LossRef)
+ if err != nil {
+ r.w.notice(host+"#rc-probe", "moves: %s: the recovery probe of rollback %s waits: %v", host, r.rb, err)
+ return false, nil
+ }
+ switch {
+ case rb.Dirty:
req.Coverage, req.Generation = recoveryCoverageSealed, r.rb
- } else {
+ case declared:
+ req.Declared = true
+ default:
req.Coverage = recoveryCoverageClean
}
out, err := cfg.recoveryProbe(ctx, r.s, r.rec, req)
diff --git a/internal/host/movesrecoveryowners.go b/internal/host/movesrecoveryowners.go
index 2c80f6c7b..20c59d112 100644
--- a/internal/host/movesrecoveryowners.go
+++ b/internal/host/movesrecoveryowners.go
@@ -64,9 +64,9 @@ import (
// - AdmissionEvidence: the same decisions re-derived and compared against
// a baseline selected again. While a declared loss or a target fence is
// pending, the writer replaces RemoteMaxTXID with the frozen
-// predecessor's maximum from its LossSources call (writer_loss.go), which
-// this owner cannot vouch for until that contract is bound (Task 4): it
-// refuses, typed.
+// predecessor's maximum from its LossSources call (writer_loss.go), and
+// AdditionalWritesLost with the other frozen sources: the loss contract
+// (movesloss.go, Task 4) accepts exactly that substitution.
// - Materialize, before every start: the held render and the held release
// of the recorded inputs, each under its own claim selection at the
// admitted capture (Works: held-config, held-release; MovesClaimPolicy's
@@ -121,12 +121,6 @@ func (v recoveryProbe) prepared() probeLanding {
return probeLanding{homeRegion: v.homeRegion, op: v.rb, holder: v.holder, rec: v.rec, seal: v.seal, manifest: v.manifest}
}
-// recoveryLossContract is the typed refusal while the recovery's loss and
-// fence admission evidence is unbound (Task 4 binds LossSources).
-func recoveryLossContract() error {
- return &ClaimDependencyError{Contracts: []string{"moves recovery loss and fence admission evidence (LossSources)"}}
-}
-
// recoveryInputsChanged is a launch input that is not the recorded one: the
// next pass records the inputs again and probes again.
func recoveryInputsChanged(what string) error {
@@ -191,16 +185,11 @@ func (r movesRecoveryOwners) recovery(ctx context.Context, o hostclaim.Observed)
return v, hostbox.ErrChanged
}
v.rb, v.holdOp, v.dirty, v.inputs = rb.ID, rb.ID, rb.Dirty, rb.RecoveryInputs
- lossID, err := hostclaim.OperationLossID(c)
- if err != nil {
+ // A declared or fenced return (Q6); the record and the claim
+ // disagreeing on the target's loss wait (movesloss.go).
+ if v.uncovered, err = recoveryDeclared(c, rb.LossRef); err != nil {
return v, err
}
- claimed := lossID != "" || c.Op.Journal["fence."+c.Op.Intent["target"]] != ""
- if claimed != (rb.LossRef != "") {
- // The record and the claim disagree on the target's loss: wait.
- return v, fmt.Errorf("%w: the record and the claim disagree on the target's loss", hostbox.ErrChanged)
- }
- v.uncovered = claimed
if v.dirty && v.uncovered {
return v, fmt.Errorf("%w: a declared or fenced return is never dirty", hostbox.ErrChanged)
}
@@ -409,16 +398,21 @@ func (r movesRecoveryOwners) AdmissionEvidence(ctx context.Context, o hostclaim.
if err != nil {
return err
}
- // The writer substitutes the frozen maximum under a pending loss or
- // fence; only the loss sources vouch for it (Task 4).
+ if baseline == nil || in.Mode != "cold" || in.PredecessorCertified || in.LossID != "" || in.KeepStamp {
+ return hostbox.ErrChanged
+ }
+ // Under a pending loss or fence the writer substitutes the validated
+ // sources' maxima; only the loss contract vouches for them
+ // (movesloss.go). Otherwise the admission is Select's.
pending, err := recoveryLossPending(o.Claim)
if err != nil {
return err
}
if pending {
- return recoveryLossContract()
- }
- if baseline == nil || in.Mode != "cold" || in.PredecessorCertified || in.LossID != "" || len(in.AdditionalWritesLost) != 0 || in.KeepStamp || in.RemoteMaxTXID != baseline.MaxTXID {
+ if err := r.lossEvidence(ctx, o, in, baseline); err != nil {
+ return err
+ }
+ } else if len(in.AdditionalWritesLost) != 0 || in.RemoteMaxTXID != baseline.MaxTXID {
return hostbox.ErrChanged
}
if !v.preparedAdmission() && !v.freshGeneration(in.NewGeneration) {
diff --git a/internal/host/movesrecoveryowners_test.go b/internal/host/movesrecoveryowners_test.go
index 0226f3e8c..5140430ec 100644
--- a/internal/host/movesrecoveryowners_test.go
+++ b/internal/host/movesrecoveryowners_test.go
@@ -761,29 +761,6 @@ func TestRecoveryMaterializeRendersTheRecordedInputsUnderHeldWork(t *testing.T)
})
}
-// Until Task 4 binds the loss contract (LossSources), an admission while a
-// declared loss is pending is refused, typed, by the recovery's
-// AdmissionEvidence: the writer substitutes the frozen maximum, which only
-// the loss sources can vouch for.
-func TestRecoveryAdmissionEvidenceRefusesAPendingLossUntilTheLossContract(t *testing.T) {
- f := newRCFixture(t, rcClean)
- c := f.recoveryClaim()
- c.LossRefs = []string{"loss-1"}
- c.Op.Journal["loss.loss-1"] = `{"kind":"move-target-lost","target":"uk-box"}`
- f.putClaim(c)
- f.rec.Move.Rollback.LossRef = "losses/loss-1"
- o := f.read()
- in := f.selectAdmission(o)
- if in.Admission.Coverage != "" {
- t.Fatalf("a declared return asked coverage %q", in.Admission.Coverage)
- }
- err := f.owners.AdmissionEvidence(t.Context(), o, in.Admission, in.Baseline)
- var missing *ClaimDependencyError
- if !errors.As(err, &missing) {
- t.Fatalf("not the typed refusal: %v", err)
- }
-}
-
// rcInOrder requires each of want in order, as a subsequence of got.
func rcInOrder(t *testing.T, got []string, want ...string) {
t.Helper()
diff --git a/internal/host/movesrecoveryprobe.go b/internal/host/movesrecoveryprobe.go
index 13dccacb7..1c69bce44 100644
--- a/internal/host/movesrecoveryprobe.go
+++ b/internal/host/movesrecoveryprobe.go
@@ -23,9 +23,11 @@ import (
// One call is one probe-shaped claim wake, in this order:
//
// 1. the request is checked against the record (this rollback at probing,
-// nine attempts, the coverage and generation of its return) and the
-// claim is read fresh (restoring/probing, this op, this rollback, this
-// box executing, the record's incarnation, a clean return's src_gen);
+// nine attempts, the coverage and generation of its return; a declared
+// return, whose record cites the target's declared loss or fence, asks
+// no coverage, Q6) and the claim is read fresh (restoring/probing, this
+// op, this rollback, this box executing, the record's incarnation, a
+// clean return's src_gen, the target's loss the record cites);
// 2. the inputs the probe will render are selected (the box's current
// inputs, as the cut reads them) and recorded through req.Record, which
// is provision.MoveRecordRecoveryInputs under the move and rollback ids,
@@ -168,7 +170,7 @@ func (a recoveryProbeAdapter) probe(ctx context.Context, s release.Store, rec pr
func recoveryProbeMatches(rec provision.Record, req recoveryProbeRequest) error {
m := rec.Move
if m != nil && m.ID != "" && m.State == provision.MoveReverting {
- if m.ReopenedAt != nil || req.RollbackID != "" || req.Coverage != "" || req.Generation != "" || !req.Clean || req.Attempts != recoveryProbeAttempts || req.Record == nil {
+ if m.ReopenedAt != nil || req.RollbackID != "" || req.Coverage != "" || req.Generation != "" || !req.Clean || req.Declared || req.Attempts != recoveryProbeAttempts || req.Record == nil {
return errRecoveryProbeRequest
}
return nil
@@ -179,11 +181,20 @@ func recoveryProbeMatches(rec provision.Record, req recoveryProbeRequest) error
if req.Record == nil || req.Attempts != recoveryProbeAttempts || req.Clean == m.Rollback.Dirty {
return fmt.Errorf("%w: budget %d, clean %v", errRecoveryProbeRequest, req.Attempts, req.Clean)
}
- if req.Clean {
+ // The declared kind (Q6): a return whose record cites a declared loss
+ // or a fence of the target is clean and asks no coverage.
+ declared := m.Rollback.LossRef != ""
+ switch {
+ case req.Declared != declared:
+ return fmt.Errorf("%w: the request's declared return is not the record's", errRecoveryProbeRequest)
+ case declared && (!req.Clean || req.Coverage != "" || req.Generation != "" || req.SourceGeneration == ""):
+ return fmt.Errorf("%w: a declared return admits a fresh generation restored from src_gen, uncovered", errRecoveryProbeRequest)
+ case declared:
+ case req.Clean:
if req.Coverage != recoveryCoverageClean || req.Generation != "" || req.SourceGeneration == "" {
return fmt.Errorf("%w: a clean return admits a fresh generation restored from src_gen", errRecoveryProbeRequest)
}
- } else if req.Coverage != recoveryCoverageSealed || req.Generation != req.RollbackID {
+ case req.Coverage != recoveryCoverageSealed || req.Generation != req.RollbackID:
return fmt.Errorf("%w: a dirty return admits its sealed return generation", errRecoveryProbeRequest)
}
return nil
@@ -206,6 +217,15 @@ func (a recoveryProbeAdapter) admits(obs hostclaim.Observed, rec provision.Recor
if c.Op.Kind != hostclaim.KindRestoring || c.Op.Intent["rollback"] != req.RollbackID {
return hostbox.ErrChanged
}
+ // The fresh claim must still agree with the record on the target's
+ // loss, and with the request's kind (Q6).
+ declared, err := recoveryDeclared(c, rec.Move.Rollback.LossRef)
+ if err != nil {
+ return err
+ }
+ if declared != req.Declared {
+ return fmt.Errorf("%w: the claim's loss of the target is not the request's", hostbox.ErrChanged)
+ }
if req.SourceGeneration != "" && c.Op.Journal[hostclaim.JournalSrcGen] != req.SourceGeneration {
return fmt.Errorf("%w: the claim's src_gen is not the request's", hostbox.ErrChanged)
}
diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go
index 017db2449..1edf558b3 100644
--- a/internal/provision/plan9_b8_census_test.go
+++ b/internal/provision/plan9_b8_census_test.go
@@ -552,8 +552,12 @@ var plan9B8CensusInventory = []censusEntry{
{"internal/host/movesrecover.go", "recoverRun.finalize", "record-writer", "key:SetRecordMove", "f2bab5b8b4deb1e1698370f1d7ecd52c226978579e3682746759cab9698f39c0"},
// Advances the source recovery phase through SetRecordMove and its sequenced writer.
{"internal/host/movesrecover.go", "recoverRun.phase", "record-writer", "key:SetRecordMove", "be641eaba77b940ce471696720b637b9dfb9454de0893aca1fcc40750135719d"},
- // Records probe inputs through SetRecordMove; phase and stuck helpers use the same writer.
- {"internal/host/movesrecover.go", "recoverRun.probing", "record-writer", "key:SetRecordMove", "a0144975bf52b29bbc6c2354636d3665487365d24838d3ef4baef1c665ce52ab"},
+ // Records probe inputs through SetRecordMove; phase and stuck helpers use the same writer. Moves Task 4: it also reads the claim's loss of the target (recoveryDeclared) to ask a declared return's coverage; no new write.
+ {"internal/host/movesrecover.go", "recoverRun.probing", "record-writer", "key:SetRecordMove", "6d5d83343b5091faed0da4e3c1026965b2f9c2c593e2633185b766532189bfcc"},
+ // Moves Task 4: derives the declaration's loss record key to compare with the record's Rollback.LossRef; it reads the claim only and writes nothing.
+ {"internal/host/movesloss.go", "claimLossRefs", "key-site", "key:LossRecordKey", "288effcd9c701a2ee15619b0c40a2f474ca00cbd95d32fdef0f922e08285080a"},
+ // Moves Task 4: derives the pending declaration's loss record key to compare with the record's Rollback.LossRef before vouching for its frozen sources; it writes nothing.
+ {"internal/host/movesloss.go", "movesRecoveryOwners.pendingLoss", "key-site", "key:LossRecordKey", "fc0855fc74578814a25c3dd10e1efbba202885a6bd1215c923c98d53638d5262"},
// Marks recovery stuck through SetRecordMove; event and serial writes are not record objects.
{"internal/host/movesrecover.go", "recoverRun.stuck", "record-writer", "key:SetRecordMove", "d634bbd18794aedf081c51f6e3c16b62a9b6b82b21b24f636b949c840c8e5776"},
// Records revert inputs through SetRecordMove; stuck uses the reviewed record helper and journals are local. Plan 6 fix round 2: its journal write is a Commit through moves.UpdateJournal, and its two step removals are Unmarks.