amadan

carlos / platform Public

region HA 1b-next J3: ConvertCommittedLanding

Paul's ruling J3/F5 (amadan #236), the model's
fence-converts-committed-landing: a landing whose move record has
committed and whose target is then terminated becomes a same-region
relocating failover in the target's region, in one claim CAS.

- The record's commit is the evidence, read through an injected
  CommittedLandingGuard (moves' record, terminal once committed), and
  rechecked inside the CAS. The claim may lag at probing or probed (the
  record-commit race): the same CAS first follows the commit as moves'
  commitClaim does (probed, then Commit under KeepWriterSession, with the
  guard's moves.CommitScopes), so the source's location park, its
  park-local and the staging lane's park exist before the op stops being
  a landing. A record not committed refuses with ErrWrongState and the
  fence stays moves' declared-loss rollback.
- The fence must name the landing's target, with its roster enrollment,
  bound to the record's destination region.
- Result: relocating in starting, op.exec none, no target and no
  participant, the dead target as source_box, consumed by fenceBox as
  FenceSource does (lost, fence and observed_at journaled, the loss in
  evidence_refs). Op id, holder, journal and every obligation are kept;
  the move's intent is kept under move_* keys.
- As the model's initial failover, the target acknowledges before the
  pin: NameTarget also names a failover's target in starting, PinFailover
  refuses without a named target, and Worker.Step names one in starting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 2ced0c5d9232d81c98a80392b2c070d8a6d50f8e parent ccce367
6 files changed, +526 −8
  • internal/hostclaim/relocating.go +152 −0
  • internal/hostclaim/relocating_test.go +294 −0
  • internal/hostclaim/relocation.go +21 −8
  • internal/hostclaim/relocation_test.go +11 −0
  • internal/hostrelocate/worker.go +11 −0
  • internal/hostrelocate/worker_test.go +37 −0
diff --git a/internal/hostclaim/relocating.go b/internal/hostclaim/relocating.go
index fbce8d0e5..ef0ae2d36 100644
--- a/internal/hostclaim/relocating.go
+++ b/internal/hostclaim/relocating.go
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
+ "slices"
"strconv"
"github.com/carlosframework/platform/internal/release"
@@ -165,6 +166,157 @@ func ConvertMovingToRelocating(ctx context.Context, s release.Store, observed Ob
})
}
+// CommittedLanding is the authoritative instance record's committed move
+// as a CommittedLandingGuard read it: the regions it moved between and the
+// two parks the commit queues, built by moves.CommitScopes from the
+// record's flip time and the claim's seal (the one builder moves' own
+// commit uses). The scopes are used only when the claim has not yet
+// followed the commit.
+type CommittedLanding struct {
+ FromRegion, ToRegion string
+ SourceScope, StagingScope ParkScope
+}
+
+// CommittedLandingGuard verifies in the authoritative instance record that
+// move opID has committed (moves §7.5: the record's commit CAS, record
+// first, terminal) for this claim's incarnation. A record still landing,
+// rolling back, or naming another move or incarnation refuses with
+// ErrWrongState: before the commit the target's fence takes moves'
+// declared-loss rollback (moves §7.10), never this conversion. A snapshot
+// of an undecided record is not an implementation of this interface.
+type CommittedLandingGuard interface {
+ CheckLandingCommitted(context.Context, Observed, string) (CommittedLanding, error)
+}
+
+// landingConvertible reports the landing phases at which a committed
+// record can stand: the record commits only after its probed, so the
+// claim is at probing or probed (lagging, the record-commit race) or has
+// followed it to committed.
+func landingConvertible(phase string) bool {
+ return phase == PhaseLandingProbing || phase == PhaseLandingProbed || phase == PhaseLandingCommitted
+}
+
+// ConvertCommittedLanding is J3 (Paul's ruling J3/F5 on amadan #236; the
+// model's fence-converts-committed-landing): a landing whose move record
+// has committed and whose target is then terminated becomes a same-region
+// relocating failover in the target's region, in exactly one claim CAS.
+// The record's commit is the evidence (guard), never the claim's phase
+// alone; the claim may lag the record at probing or probed, and then the
+// same CAS first follows the commit exactly as moves does (probed, then
+// Commit under KeepWriterSession, queueing the source region's location
+// park, the source's park-local and the staging lane's park), so every
+// cleanup obligation the commit owes exists before the op stops being a
+// landing. Before the commit the guard refuses and the fence is moves'
+// (declared-loss rollback).
+//
+// The result is the failover StartRelocation starts for a fenced
+// executor, without a target: relocating in starting, op.exec none, the
+// dead target as source_box (on lost, its fence journaled with its
+// observation time and the loss retained in evidence_refs by fenceBox, as
+// FenceSource does), the op id, holder and every obligation kept. The
+// target is named in starting (NameTarget) and acknowledges before
+// PinFailover, as an initial failover does. The move's own intent is kept
+// as history under move_* keys and its journal is kept whole. It
+// deliberately exposes no transition capable of granting execution.
+func ConvertCommittedLanding(ctx context.Context, s release.Store, observed Observed, opID, targetBox string, ref FenceRef, authority FenceAuthority, guard CommittedLandingGuard, d *LocalDischarge, newID func() string) (Observed, error) {
+ if guard == nil || newID == nil || observed.Absent || observed.Claim == nil || observed.ETag == "" {
+ return Observed{}, ErrWrongState
+ }
+ before := observed.Claim
+ if opID == "" || before.Op.ID != opID {
+ return Observed{}, ErrStaleOp
+ }
+ // The landing's own intent, as BeginAdmittedMove wrote it: "source",
+ // "target" and "to", the target's enrollment from the signed roster,
+ // and the app admission the move ran under.
+ in := before.Op.Intent
+ if before.Holder == nil || before.Holder.Family != "instance" || before.Op.Kind != KindLanding || !landingConvertible(before.Op.Phase) ||
+ !validBoxReference(targetBox) || targetBox == "none" || in["target"] != targetBox || before.Op.Exec != targetBox || slices.Contains(before.Lost, targetBox) ||
+ !validBoxReference(in["source"]) || in["source"] == "none" || in["source"] == targetBox || in["to"] == "" ||
+ !validEnrollment(in["target_enrollment"]) || in["app_id"] == "" || in["admission_id"] != opID {
+ return Observed{}, ErrWrongState
+ }
+ committed, err := guard.CheckLandingCommitted(ctx, observed, opID)
+ if err != nil {
+ return Observed{}, err
+ }
+ if committed.FromRegion == "" || committed.ToRegion != in["to"] || committed.FromRegion == committed.ToRegion {
+ return Observed{}, ErrWrongState
+ }
+ fence, err := VerifyFence(ctx, s, authority, ref)
+ if err != nil {
+ return Observed{}, err
+ }
+ // The fence is the target's, bound to the target's region in the
+ // roster, which is the record's destination.
+ if fence.record.Box != targetBox || fence.record.Enrollment != in["target_enrollment"] || fence.binding.Region != committed.ToRegion {
+ return Observed{}, ErrFence
+ }
+ return Update(ctx, s, observed, func(c *Claim) error {
+ if c.Op.ID != opID || c.Op.Kind != KindLanding || c.Op.Phase != before.Op.Phase || c.Op.Exec != targetBox || slices.Contains(c.Lost, targetBox) {
+ return ErrStaleOp
+ }
+ // Recheck the record immediately before the claim CAS.
+ again, err := guard.CheckLandingCommitted(ctx, observed, opID)
+ if err != nil {
+ return err
+ }
+ if again.FromRegion != committed.FromRegion || again.ToRegion != committed.ToRegion {
+ return ErrWrongState
+ }
+ phase := c.Op.Phase
+ // The claim half of the record's commit, as moves writes it
+ // (moveslandcommit.go commitClaim): a claim lagging at probing is
+ // stepped to probed first, then Commit, on the same session.
+ if c.Op.Phase == PhaseLandingProbing {
+ if err := KeepWriterSession(opID, SetMovePhase(opID, KindLanding, PhaseLandingProbing, PhaseLandingProbed))(c); err != nil {
+ return err
+ }
+ }
+ if c.Op.Phase == PhaseLandingProbed {
+ if err := KeepWriterSession(opID, Commit(opID, committed.FromRegion, in["source"], committed.ToRegion, committed.SourceScope, committed.StagingScope, newID))(c); err != nil {
+ return err
+ }
+ }
+ if c.Op.Phase != PhaseLandingCommitted {
+ return ErrWrongState
+ }
+ ensureJournal(c)
+ c.Op.Journal["reason"] = ReasonFailover
+ c.Op.Journal["conversion"] = "committed-landing"
+ c.Op.Journal["conversion_landing_phase"] = phase
+ c.Op.Journal["conversion_claim_gen"] = strconv.FormatUint(before.Gen, 10)
+ c.Op.Journal["conversion_claim_etag"] = observed.ETag
+ c.Op.Journal["move_from_region"] = committed.FromRegion
+ c.Op.Journal["roster_revision"] = strconv.FormatUint(fence.binding.RosterRevision, 10)
+ c.Op.Journal["roster_etag"] = fence.binding.RosterETag
+ c.Op.Kind, c.Op.Phase, c.Op.Exec = KindRelocating, PhaseStarting, "none"
+ intent := make(map[string]string, len(c.Op.Intent)+4)
+ for k, v := range c.Op.Intent {
+ intent[k] = v
+ }
+ // The move's other side is in another region and is no participant:
+ // its finalization chores are obligations, kept as they are. Keep
+ // the move's history without authorizing it.
+ for from, to := range map[string]string{"source": "move_source_box", "source_enrollment": "move_source_enrollment", "to": "move_to_region"} {
+ if v := intent[from]; v != "" {
+ intent[to] = v
+ }
+ delete(intent, from)
+ }
+ delete(intent, "target")
+ delete(intent, "target_enrollment")
+ intent["region"] = committed.ToRegion
+ intent["source_box"] = targetBox
+ intent["source_enrollment"] = in["target_enrollment"]
+ intent["stamp_from"] = strconv.FormatUint(c.Holder.Stamp, 10)
+ intent["generation_from"] = c.Holder.Generation
+ c.Op.Intent = intent
+ c.Op.Participants = nil
+ return fenceBox(c, targetBox, ref, fence.record.ObservedAt, d)
+ })
+}
+
// Box and Region expose a verified fence's terminated box and its roster
// region binding, so a caller binds them to the authoritative record rather
// than to a decoded reference.
diff --git a/internal/hostclaim/relocating_test.go b/internal/hostclaim/relocating_test.go
index e38853b05..b2cdb4825 100644
--- a/internal/hostclaim/relocating_test.go
+++ b/internal/hostclaim/relocating_test.go
@@ -7,6 +7,8 @@ import (
"encoding/json"
"errors"
"reflect"
+ "slices"
+ "strconv"
"strings"
"testing"
"time"
@@ -343,3 +345,295 @@ func TestBeginAdmittedMoveIsOneClosure(t *testing.T) {
}
}
}
+
+// J3 (Paul's ruling J3/F5 on amadan #236; the model's
+// fence-converts-committed-landing): a landing whose record committed and
+// whose target is then terminated becomes a failover in the target's
+// region.
+
+const (
+ j3Source = "ie-box"
+ j3Target = "uk-box"
+ j3Spare = "uk-box2"
+ j3SourceEnr = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+ j3TargetEnr = "cccccccccccccccccccccccccccccccc"
+ j3SpareEnr = "dddddddddddddddddddddddddddddddd"
+)
+
+// j3Authority binds every fence to the target's region, uk.
+type j3Authority struct{ region string }
+
+func (a j3Authority) ValidateFence(_ context.Context, f termfence.Record) (FenceBinding, error) {
+ if f.Box != j3Target || f.Enrollment != j3TargetEnr {
+ return FenceBinding{}, ErrFence
+ }
+ region := a.region
+ if region == "" {
+ region = "uk"
+ }
+ return FenceBinding{Region: region, RosterRevision: 11, RosterETag: "roster-uk"}, nil
+}
+
+// j3Guard is the record: committed (moves §7.5) unless refuse says
+// otherwise, read on the calls counted.
+type j3Guard struct {
+ calls int
+ rejectAt int
+ to string
+}
+
+func (g *j3Guard) CheckLandingCommitted(_ context.Context, o Observed, id string) (CommittedLanding, error) {
+ g.calls++
+ if id != o.Claim.Op.ID {
+ return CommittedLanding{}, ErrWrongState
+ }
+ if g.rejectAt > 0 && g.calls >= g.rejectAt {
+ return CommittedLanding{}, ErrWrongState // the record is landing or rolling back
+ }
+ to := g.to
+ if to == "" {
+ to = "uk"
+ }
+ src, stg := commitScopes(o.Claim)
+ return CommittedLanding{FromRegion: "ie", ToRegion: to, SourceScope: src, StagingScope: stg}, nil
+}
+
+// j3Landing is a stored admitted landing at phase (probing, probed, or
+// committed: the claim followed the record's commit), its source box
+// holding paths of the stay the move leaves so the commit owes its
+// park-local, plus an unrelated chore of another operation.
+func j3Landing(t *testing.T, phase string) (*c7Store, Observed) {
+ t.Helper()
+ ctx := context.Background()
+ c := probedLanding(t)
+ c.Op.Intent["app_id"], c.Op.Intent["admission_id"] = "app-canonical", moveA
+ c.Op.Intent["source_enrollment"], c.Op.Intent["target_enrollment"] = j3SourceEnr, j3TargetEnr
+ // The probe's cold admission made its generation holder.generation and
+ // the target the writer (claim rev 6.9).
+ c.Holder.WriterBox, c.Holder.Generation = j3Target, moveA
+ c.Inventory = map[string]InventoryEntry{j3Source: {Layout: "flat", Paths: map[string]uint64{"/data/a": 3}}}
+ c.Obligations = []Obligation{{ID: "older", Op: "op_old", Kind: "park-generation", Region: "uk", Generation: "gen-1", Retirement: "r-old", Owner: "region:uk"}}
+ switch phase {
+ case PhaseLandingProbing:
+ c.Op.Phase = PhaseLandingProbing
+ case PhaseLandingCommitted:
+ src, stg := commitScopes(c)
+ if err := KeepWriterSession(moveA, Commit(moveA, "ie", j3Source, "uk", src, stg, counter()))(c); err != nil {
+ t.Fatal(err)
+ }
+ }
+ s := &c7Store{Store: release.NewDirStore(t.TempDir())}
+ o, err := Update(ctx, s, Observed{Host: c.Host, Absent: true}, func(n *Claim) error {
+ *n = *c
+ n.Gen = 0
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ s.puts = 0
+ return s, o
+}
+
+func j3Fence(t *testing.T, s release.Store) FenceRef {
+ t.Helper()
+ f := termfence.Record{V: 1, Box: j3Target, Enrollment: j3TargetEnr, Cloud: boxid.Cloud{Provider: "aws", Account: "123", Region: "eu-west-2", InstanceID: "i-target"},
+ State: "terminated", ObservedAt: "2026-10-10T08:00:00Z", Via: "describe", RequestID: "request", WrittenBy: termfence.WrittenBy{Box: "uk-worker", Enrollment: strings.Repeat("b", 32)}}
+ raw, err := json.Marshal(f)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(raw)
+ ref := FenceRef{Enrollment: f.Enrollment, Key: "control/box-fences/" + f.Enrollment + ".json", SHA256: hex.EncodeToString(sum[:])}
+ if _, err := s.Put(context.Background(), ref.Key, raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ return ref
+}
+
+func TestJ3CommittedLandingBecomesAFailoverInTheTargetsRegion(t *testing.T) {
+ ctx := context.Background()
+ for _, phase := range []string{PhaseLandingCommitted, PhaseLandingProbed, PhaseLandingProbing} {
+ t.Run("claim at "+phase, func(t *testing.T) {
+ s, before := j3Landing(t, phase)
+ ref := j3Fence(t, s)
+ s.puts = 0
+ // The obligations the claim owes once it follows the commit, as
+ // moves' own commit writes them with the same ids.
+ want := cloneForTest(t, before.Claim)
+ if want.Op.Phase == PhaseLandingProbing {
+ want.Op.Phase = PhaseLandingProbed
+ }
+ if want.Op.Phase == PhaseLandingProbed {
+ src, stg := commitScopes(want)
+ if err := KeepWriterSession(moveA, Commit(moveA, "ie", j3Source, "uk", src, stg, counter()))(want); err != nil {
+ t.Fatal(err)
+ }
+ }
+ guard := &j3Guard{}
+ after, err := ConvertCommittedLanding(ctx, s, before, moveA, j3Target, ref, j3Authority{}, guard, nil, counter())
+ if err != nil {
+ t.Fatal(err)
+ }
+ c := after.Claim
+ if s.puts != 1 || c.Gen != before.Claim.Gen+1 || guard.calls != 2 {
+ t.Fatalf("writes=%d gen=%d record reads=%d", s.puts, c.Gen, guard.calls)
+ }
+ // Every cleanup obligation is kept byte for byte: the source's
+ // location park, its park-local, the staging lane's park, and
+ // the unrelated chore.
+ if string(mustJSONAny(t, c.Obligations)) != string(mustJSONAny(t, want.Obligations)) {
+ t.Fatalf("obligations:\n got %s\nwant %s", mustJSONAny(t, c.Obligations), mustJSONAny(t, want.Obligations))
+ }
+ if k := obligationKinds(c); k["park-replica/ie"] != 1 || k["park-replica/uk"] != 1 || k["park-local/"+j3Source] != 1 || k["park-generation/uk"] != 1 {
+ t.Fatalf("the commit's chores: %v", k)
+ }
+ if !reflect.DeepEqual(c.Holder, want.Holder) || !reflect.DeepEqual(c.Touched, before.Claim.Touched) || !reflect.DeepEqual(c.Inventory, before.Claim.Inventory) {
+ t.Fatal("holder, touched or inventory changed")
+ }
+ // A failover in uk with no executor and no target, its source the
+ // dead target, the op id kept.
+ if c.Op.ID != moveA || c.Op.Kind != KindRelocating || c.Op.Phase != PhaseStarting || c.Op.Exec != "none" || len(c.Op.Participants) != 0 || !failover(c.Op) {
+ t.Fatalf("op: %+v", c.Op)
+ }
+ wantIntent := map[string]string{"region": "uk", "source_box": j3Target, "source_enrollment": j3TargetEnr,
+ "move_source_box": j3Source, "move_source_enrollment": j3SourceEnr, "move_to_region": "uk",
+ "app_id": "app-canonical", "admission_id": moveA,
+ "stamp_from": strconv.FormatUint(before.Claim.Holder.Stamp, 10), "generation_from": before.Claim.Holder.Generation}
+ if !reflect.DeepEqual(c.Op.Intent, wantIntent) {
+ t.Fatalf("intent %v", c.Op.Intent)
+ }
+ // The move's journal is kept whole.
+ for k, v := range want.Op.Journal {
+ if c.Op.Journal[k] != v {
+ t.Fatalf("journal %s: %q, was %q", k, c.Op.Journal[k], v)
+ }
+ }
+ if c.Op.Journal["conversion"] != "committed-landing" || c.Op.Journal["conversion_landing_phase"] != phase || c.Op.Journal["move_from_region"] != "ie" ||
+ c.Op.Journal["conversion_claim_etag"] != before.ETag || c.Op.Journal["roster_revision"] != "11" || c.Op.Journal["roster_etag"] != "roster-uk" {
+ t.Fatalf("conversion record: %v", c.Op.Journal)
+ }
+ // The fence is consumed as FenceSource consumes it: lost, the
+ // reference and its observation time, the loss retained.
+ var saved FenceRef
+ if err := json.Unmarshal([]byte(c.Op.Journal["fence."+j3Target]), &saved); err != nil || saved != ref || c.Op.Journal["fence."+j3Target+".observed_at"] != "2026-10-10T08:00:00Z" {
+ t.Fatalf("fence journal: %+v %v", saved, err)
+ }
+ if !slices.Contains(c.Lost, j3Target) || len(c.EvidenceRefs) != 1 || c.EvidenceRefs[0].Box != j3Target || c.EvidenceRefs[0].Role != "source" ||
+ c.EvidenceRefs[0].Kind != KindRelocating || c.EvidenceRefs[0].Fence != ref || !slices.Contains(c.LossRefs, c.EvidenceRefs[0].ID) {
+ t.Fatalf("lost %v evidence %+v loss_refs %v", c.Lost, c.EvidenceRefs, c.LossRefs)
+ }
+ if owner, err := RelocationRecoveryOwner(c); err != nil || owner != "region:uk" {
+ t.Fatalf("owner=%q error=%v", owner, err)
+ }
+ // A stale replay conflicts; a replay of the fresh claim finds no
+ // landing (the sibling conversions' behaviour).
+ if _, err := ConvertCommittedLanding(ctx, s, before, moveA, j3Target, ref, j3Authority{}, &j3Guard{}, nil, counter()); !errors.Is(err, ErrConflict) {
+ t.Fatalf("stale replay: %v", err)
+ }
+ if _, err := ConvertCommittedLanding(ctx, s, after, moveA, j3Target, ref, j3Authority{}, &j3Guard{}, nil, counter()); !errors.Is(err, ErrWrongState) {
+ t.Fatalf("replay over the failover: %v", err)
+ }
+ // It proceeds as an initial failover: the target is named in
+ // starting and acknowledges before any pin.
+ if _, err := Update(ctx, s, after, PinFailover(moveA)); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("pinned before a target was named: %v", err)
+ }
+ o := step(t, s, after, NameTarget(moveA, Target{j3Spare, j3SpareEnr}))
+ if _, err := Update(ctx, s, o, PinFailover(moveA)); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("pinned before the target acknowledged: %v", err)
+ }
+ o = step(t, s, o, AcknowledgeExecution(j3Spare, ExecutionAcknowledgement{Op: moveA, Enrollment: j3SpareEnr, BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: time.Now().Add(time.Hour)}))
+ o = step(t, s, o, PinFailover(moveA))
+ if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Exec != "none" || !reflect.DeepEqual(o.Claim.Op.Participants, []string{j3Spare}) {
+ t.Fatalf("after PinFailover: %+v", o.Claim.Op)
+ }
+ if string(mustJSONAny(t, o.Claim.Obligations)) != string(mustJSONAny(t, want.Obligations)) {
+ t.Fatal("the failover's own steps changed an obligation")
+ }
+ })
+ }
+}
+
+func TestJ3ConversionRefusesWithoutWriting(t *testing.T) {
+ for _, tc := range []struct{ name string }{
+ {"record-not-committed"}, {"record-rolled-back-during-verify"}, {"record-elsewhere"}, {"claim-before-probing"}, {"not-landing"},
+ {"wrong-op"}, {"wrong-box"}, {"source-box"}, {"fence-region"}, {"stale-fence"}, {"fence-gone"}, {"target-lost"}, {"journal-rebind"},
+ {"exec-elsewhere"}, {"no-guard"}, {"no-ids"}, {"missing-admission"}, {"missing-enrollment"}, {"authority"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ ctx := context.Background()
+ s, o := j3Landing(t, PhaseLandingProbed)
+ ref := j3Fence(t, s)
+ s.puts = 0
+ id, box := moveA, j3Target
+ var guard CommittedLandingGuard = &j3Guard{}
+ var authority FenceAuthority = j3Authority{}
+ newID := counter()
+ switch tc.name {
+ case "record-not-committed":
+ guard = &j3Guard{rejectAt: 1} // moves' declared-loss rollback owns the fence
+ case "record-rolled-back-during-verify":
+ guard = &j3Guard{rejectAt: 2}
+ case "record-elsewhere":
+ guard = &j3Guard{to: "us"}
+ case "claim-before-probing":
+ o.Claim.Op.Phase = "restored"
+ case "not-landing":
+ o.Claim.Op.Kind = KindRestoring
+ case "wrong-op":
+ id = moveB
+ case "wrong-box":
+ box = j3Spare
+ case "source-box":
+ box = j3Source
+ case "fence-region":
+ authority = j3Authority{region: "ie"}
+ case "stale-fence":
+ ref.SHA256 = strings.Repeat("0", 64)
+ case "fence-gone":
+ ref.Key = "control/box-fences/" + strings.Repeat("e", 32) + ".json"
+ case "target-lost":
+ o.Claim.Lost = []string{j3Target}
+ case "journal-rebind":
+ o.Claim.Op.Journal["fence."+j3Target] = `{"enrollment":"other"}`
+ case "exec-elsewhere":
+ o.Claim.Op.Exec = j3Source
+ case "no-guard":
+ guard = nil
+ case "no-ids":
+ newID = nil
+ case "missing-admission":
+ delete(o.Claim.Op.Intent, "admission_id")
+ case "missing-enrollment":
+ delete(o.Claim.Op.Intent, "target_enrollment")
+ case "authority":
+ authority = c7Authority{err: ErrFence}
+ }
+ raw := mustJSON(t, o.Claim)
+ _, err := ConvertCommittedLanding(ctx, s, o, id, box, ref, authority, guard, nil, newID)
+ if err == nil {
+ t.Fatal("accepted")
+ }
+ t.Logf("refused: %v", err)
+ if strings.HasPrefix(tc.name, "record-") && tc.name != "record-elsewhere" && !errors.Is(err, ErrWrongState) {
+ t.Fatalf("an uncommitted record: %v", err)
+ }
+ if s.puts != 0 || string(raw) != string(mustJSON(t, o.Claim)) {
+ t.Fatal("refusal wrote or mutated the claim")
+ }
+ if stored, rerr := Read(ctx, s, o.Host); rerr != nil || stored.ETag != o.ETag {
+ t.Fatalf("stored claim moved: %v", rerr)
+ }
+ })
+ }
+}
+
+func mustJSONAny(t *testing.T, v any) []byte {
+ t.Helper()
+ b, err := json.Marshal(v)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return b
+}
diff --git a/internal/hostclaim/relocation.go b/internal/hostclaim/relocation.go
index 07de31589..5c756f29b 100644
--- a/internal/hostclaim/relocation.go
+++ b/internal/hostclaim/relocation.go
@@ -641,7 +641,10 @@ func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, op
}
// PinFailover leaves starting for pinning on a failover: the source's
-// fence is journaled and every remaining participant has acknowledged.
+// fence is journaled, the target is named (a converted committed landing
+// names it in starting, NameTarget) and every remaining participant has
+// acknowledged, so the pin is selected only after the target's
+// acknowledgement.
func PinFailover(opID string) func(*Claim) error {
return func(c *Claim) error {
if err := requireRelocation(c, opID, PhaseStarting); err != nil {
@@ -650,6 +653,9 @@ func PinFailover(opID string) func(*Claim) error {
if !failover(c.Op) || c.Op.Journal["fence."+c.Op.Intent["source_box"]] == "" || c.Op.Exec != "none" {
return fmt.Errorf("%w: pinning without a release needs the source's fence", ErrRelocation)
}
+ if c.Op.Intent["target_box"] == "" {
+ return fmt.Errorf("%w: a failover names its target before it pins", ErrRelocation)
+ }
if err := requireAcks(c); err != nil {
return err
}
@@ -712,17 +718,24 @@ func JournalPin(opID string, p RecoveryPin) func(*Claim) error {
}
// NameTarget names the target of an operation that has none: a move
-// converted to relocating before its flip (claim §4, moves §7.10). The
-// named box acknowledges (D6) before ToInstalling hands it op.exec. The
-// move's own target (move_target_box, in the region the move was going
-// to) leaves the barrier in the same CAS: readiness binds to the box named
-// here (moves §7.10), and a failover's participant is its target alone
-// (§5.8). It never held op.exec on the converted operation.
+// converted to relocating before its flip (claim §4, moves §7.10), at
+// pinning; or a committed landing converted to a failover
+// (ConvertCommittedLanding), in starting, where the named box then
+// acknowledges before PinFailover selects any pin, as an initial
+// failover's target does. The named box acknowledges (D6) before
+// ToInstalling hands it op.exec. The move's own target (move_target_box,
+// in the region the move was going to) leaves the barrier in the same CAS:
+// readiness binds to the box named here (moves §7.10), and a failover's
+// participant is its target alone (§5.8). It never held op.exec on the
+// converted operation.
func NameTarget(opID string, t Target) func(*Claim) error {
return func(c *Claim) error {
- if err := requireRelocation(c, opID, PhasePinning); err != nil {
+ if err := requireRelocation(c, opID, PhaseStarting, PhasePinning); err != nil {
return err
}
+ if c.Op.Phase == PhaseStarting && (!failover(c.Op) || c.Op.Exec != "none") {
+ return fmt.Errorf("%w: only a failover with no executor names its target in starting", ErrRelocation)
+ }
if c.Op.Intent["target_box"] != "" {
return fmt.Errorf("%w: the operation already has a target", ErrRelocation)
}
diff --git a/internal/hostclaim/relocation_test.go b/internal/hostclaim/relocation_test.go
index 022b1ddbd..c4ea1849b 100644
--- a/internal/hostclaim/relocation_test.go
+++ b/internal/hostclaim/relocation_test.go
@@ -1314,3 +1314,14 @@ func cloneForTest(t *testing.T, c *Claim) *Claim {
}
return out
}
+
+// NameTarget in starting is a converted committed landing's alone: a
+// failover with no executor. A drain in starting never names one.
+func TestRelocationNameTargetInStartingIsAFailoversAlone(t *testing.T) {
+ _, o := relAtPhase(t, PhaseStarting)
+ c := cloneForTest(t, o.Claim)
+ delete(c.Op.Intent, "target_box")
+ if err := NameTarget("rel-1", Target{relSpare, relEnrollment(relSpare)})(c); !errors.Is(err, ErrRelocation) {
+ t.Fatalf("a drain named a target in starting: %v", err)
+ }
+}
diff --git a/internal/hostrelocate/worker.go b/internal/hostrelocate/worker.go
index 414869432..4255f604c 100644
--- a/internal/hostrelocate/worker.go
+++ b/internal/hostrelocate/worker.go
@@ -337,6 +337,17 @@ func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, erro
}
switch c.Op.Phase {
case hostclaim.PhaseStarting:
+ if c.Op.Intent["target_box"] == "" && c.Op.Exec == "none" {
+ // A committed landing converted to a failover
+ // (hostclaim.ConvertCommittedLanding) starts with no target: name
+ // one here, so it acknowledges before PinFailover selects a pin,
+ // as an initial failover's target does.
+ next, err := w.next(ctx, obs)
+ if err != nil {
+ return obs, err
+ }
+ return w.update(ctx, obs, hostclaim.NameTarget(id, next))
+ }
if !acked(c) {
return obs, ErrNotYet
}
diff --git a/internal/hostrelocate/worker_test.go b/internal/hostrelocate/worker_test.go
index 846ac24ac..4305a7ecc 100644
--- a/internal/hostrelocate/worker_test.go
+++ b/internal/hostrelocate/worker_test.go
@@ -882,3 +882,40 @@ func TestWorkerRefusesAConvertedMoveWhoseBoundFenceIsMissing(t *testing.T) {
t.Fatalf("the worker acted on a missing fence: %+v leases %v", o.Claim.Op, f.leases)
}
}
+
+// J3: a committed landing converted to a failover (ConvertCommittedLanding)
+// starts with no target. The worker names one in starting; it acknowledges
+// before PinFailover, and the pin is selected after that, as an initial
+// failover's is; installing then copies the pin's release (R9).
+func TestWorkerNamesAConvertedLandingsTargetBeforeItsPin(t *testing.T) {
+ f := newFixture(t)
+ ctx := context.Background()
+ f.terminate("src")
+ if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil {
+ t.Fatal(err)
+ }
+ // The shape the conversion leaves: starting, op.exec none, the dead box
+ // the source (on lost, its fence consumed), no target, no participant.
+ f.participant(func(c *hostclaim.Claim) error {
+ delete(c.Op.Intent, "target_box")
+ delete(c.Op.Intent, "target_enrollment")
+ c.Op.Participants = nil
+ c.Op.Journal["conversion"] = "committed-landing"
+ return nil
+ })
+ o := f.worker(nil)
+ if o.Claim.Op.Phase != hostclaim.PhaseStarting || o.Claim.Op.Intent["target_box"] != "dst" || !slices.Equal(o.Claim.Op.Participants, []string{"dst"}) || o.Claim.Op.Journal["pin"] != "" {
+ t.Fatalf("named in starting: %+v", o.Claim.Op)
+ }
+ f.worker(ErrNotYet) // the target has not acknowledged: no pin yet
+ f.ack("dst")
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning {
+ t.Fatalf("phase %s", o.Claim.Op.Phase)
+ }
+ if o = f.worker(nil); o.Claim.Op.Journal["pin"] == "" {
+ t.Fatal("not pinned")
+ }
+ if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseInstalling || o.Claim.Op.Exec != "dst" || o.Claim.Op.Journal["install_release"] != "v1.4.2" {
+ t.Fatalf("installing: %+v", o.Claim.Op)
+ }
+}