| index 7f91d1587..dd493cf1a 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| @@ -25,7 +25,7 @@ |
| ## Review Focus |
| |
| 1. A record whose `move` was written by a v0.22 console (`MoveState` without `id`) must be refused by the new worker on every pass, logged once a minute, and never mirrored or cut. Test in Task 2 (`TestMoveStateLegacyRecordIsRefused`). |
| -2. A `move.json` that already exists in the target bucket when the source has no record of writing it must stop the round with a refusal, not overwrite it; a `move.json` the source itself wrote but whose ledger entry was lost in a crash must be accepted without reading it back (M17 forbids the read). Test in Task 14 (`TestStagingMoveJSONIsCreateOnly`, `TestStagingMoveJSONReplayAfterLostLedger`). |
| +2. A `move.json` the source itself wrote but whose ledger entry was lost in a crash must be accepted without reading it back (M17 forbids the read), and a lane holding another move's `move.json` must be caught by the target, which can read it, so the move never becomes ready. Tests in Task 14 (`TestStagingMoveJSONReplayAfterLostLedger`) and the target-side `TestTargetReportPassFlagsEachCheck` (T9's lane case). |
| 3. A files directory containing a symlink must refuse the archive with the path, and the round must keep mirroring the replica (the refusal is reported in `target.json`'s T5 input, not fatal to the move until the cut). Test in Task 7 (`TestArchiveRefusesSpecialFiles`). |
| 4. An operator who names the instance's own current region, or a region outside the manifest's `regions`, gets the refusal sentence and no claim write. Test in Task 4 (`TestMoveDoorRefusesWithoutClaimWrite`). |
| 5. A target report older than 10 minutes, even one that passed, must not promote `ready`; a stale report after `ready` must demote nothing (the cut re-checks freshness at `admitted`, Plan 2). Test in Task 11 (`TestReadyNeedsFreshReports`). |
| @@ -60,6 +60,82 @@ |
| |
| --- |
| |
| +### Task 0: The switch boundary |
| + |
| +**Files:** |
| +- Create: `internal/moves/switch.go`, `internal/moves/switch_test.go` |
| +- Modify: `internal/console/server.go` (`Config.MovesEnabled bool`, set where the console reads its environment: the same place `HomeRegion` and `SecretsPub` are read), `internal/host/movespass.go` (`moveCfg.Enabled`, set in `armMovesPass` from `moves.Enabled(os.Getenv)`) |
| + |
| +**Interfaces:** |
| +- Produces: `moves.SwitchEnv = "CARLOS_MOVES_PHASE1"`; `moves.DefaultEnabled = false` (a compiled constant the copy gate reads, Task 15); `moves.Enabled(getenv func(string) string) bool` (`"on"`, `"1"`, `"true"` → true; `""` → `DefaultEnabled`; anything else → false with no panic); `console.Config.MovesEnabled`; `moveCfg.Enabled`. |
| +- **The rule for every task of this series (index D13):** with the switch off, `main` behaves exactly as it does today, legacy move protocol included: the existing `MoveRequest` door under the Instance role, the existing worker (preparing, ready, the staging-wins cut, aborting), and `promoteStagedReplica` all stay reachable and unchanged while the switch is off. New code is reached only when the switch is on. Tasks therefore add beside the legacy path and never delete it; the legacy path is retired in one final task after Paul approves the copy record and the default flips (Plan 6 Task 8 names it). A legacy record (`Move.Legacy()`) is the legacy path's; an id'd record is the new path's; when the switch is off, an id'd record is skipped with one log line a minute (none exists in production while the switch has never been on). |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +package moves |
| + |
| +import "testing" |
| + |
| +func TestSwitchIsOffByDefault(t *testing.T) { |
| + if DefaultEnabled { |
| + t.Fatal("the Phase 1 switch must default to off until the copy record is approved") |
| + } |
| + env := map[string]string{} |
| + get := func(k string) string { return env[k] } |
| + if Enabled(get) { |
| + t.Fatal("unset is off") |
| + } |
| + for _, v := range []string{"on", "1", "true"} { |
| + env[SwitchEnv] = v |
| + if !Enabled(get) { |
| + t.Fatalf("%q should enable", v) |
| + } |
| + } |
| + for _, v := range []string{"off", "0", "false", "yes", " on"} { |
| + env[SwitchEnv] = v |
| + if Enabled(get) { |
| + t.Fatalf("%q should not enable", v) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify it fails**: `nice -n 10 go test ./internal/moves -run TestSwitchIsOffByDefault -count=1` → FAIL, undefined. |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +package moves |
| + |
| +// SwitchEnv is the operator-off switch of Phase 1 (index D13): every move |
| +// entry point is behind it, and its default is off until the copy record |
| +// is approved. |
| +const SwitchEnv = "CARLOS_MOVES_PHASE1" |
| + |
| +// DefaultEnabled is the compiled default. The copy gate |
| +// (approved_copy_test.go) requires it false while any copy row is pending. |
| +const DefaultEnabled = false |
| + |
| +// Enabled reads the switch. Unset means the default; "on", "1" and "true" |
| +// enable; anything else is off. |
| +func Enabled(getenv func(string) string) bool { |
| + switch getenv(SwitchEnv) { |
| + case "": |
| + return DefaultEnabled |
| + case "on", "1", "true": |
| + return true |
| + } |
| + return false |
| +} |
| +``` |
| + |
| +In `server.go`, `Config.MovesEnabled bool` with the doc "the Phase 1 moves switch (moves.Enabled)"; where the console builds its `Config` from the environment, `MovesEnabled: moves.Enabled(os.Getenv)`. In `armMovesPass`, `cfg.Enabled = moves.Enabled(os.Getenv)`; `movesPass` reads it (Task 14 uses it). Console tests set `fx.srv.cfg.MovesEnabled = true` where they exercise the new door (Task 4). |
| + |
| +- [ ] **Step 4: Run** → PASS; `go build ./...`. |
| +- [ ] **Step 5: Commit**: `git commit -m "moves: the Phase 1 switch, off by default (index D13)" ...` |
| + |
| +--- |
| + |
| ### Task 1: Operation ids |
| |
| **Files:** |
| @@ -189,7 +265,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/provision/move_test.go` |
| |
| **Interfaces:** |
| -- Produces: the extended `provision.MoveState` below; `Record.LastMove *MoveOutcome \`json:"last_move,omitempty"\`` beside `Record.Move` (`provision.go:58`); state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`. |
| +- Produces: the extended `provision.MoveState` below; `Record.LastMove *MoveOutcome \`json:"last_move,omitempty"\`` beside `Record.Move` (`provision.go:58`); `MoveGuard` and the guarded transitions `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbort`, `MoveGuardedAbortComplete` (below); the legacy `MoveRequest`, `MovePromoteReady`, `MoveMarkCutting`, `MoveAbort`, `MoveAbortComplete`, `MoveComplete` unchanged; state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`. |
| - Later plans add the phase transitions (Plan 2 Task 1 the cut phases, Plan 2 Task 8 the landing phases, Plan 4 Task 1 the rollback phases). |
| |
| - [ ] **Step 1: Write the failing test** |
| @@ -263,31 +339,34 @@ func TestMoveStartWritesTheJournalHead(t *testing.T) { |
| } |
| |
| // Review Focus 1: a move a v0.22 console wrote has no id. The journal |
| -// calls it legacy and no new transition accepts it. |
| -func TestMoveStateLegacyRecordIsRefused(t *testing.T) { |
| - m := &MoveState{To: "uk", State: MovePreparing, RequestedAt: time.Now()} |
| - if !m.Legacy() { |
| - t.Fatal("an id-less move is legacy") |
| - } |
| +// calls it legacy; the legacy transitions still drive it (the legacy path |
| +// stays while the switch is off, Task 0), and no guarded transition |
| +// accepts it. |
| +func TestMoveStateLegacyRecordIsRefusedByTheGuard(t *testing.T) { |
| s := release.NewDirStore(t.TempDir()) |
| rec := moveTestRecord(t, s) |
| - rec.Move = m |
| - if err := UpdateRecord(context.Background(), s, rec, ""); err != nil { |
| + if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MoveRequest("uk", time.Now())); err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MovePromoteReady(time.Now())); !errors.Is(err, ErrMoveConflict) { |
| - t.Fatalf("promote on a legacy move: %v", err) |
| + got, _ := GetRecord(context.Background(), s, rec.Account, rec.App, rec.Host) |
| + if !got.Move.Legacy() { |
| + t.Fatal("an id-less move is legacy") |
| } |
| - if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MoveMarkCutting()); !errors.Is(err, ErrMoveConflict) { |
| - t.Fatalf("cut on a legacy move: %v", err) |
| + g := MoveGuard{ID: "mv_abcdefghijklmnopqrstuvwxyz", Incarnation: strings.Repeat("ab", 32)} |
| + if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MoveGuardedPromoteReady(g, time.Now())); !errors.Is(err, ErrMoveConflict) { |
| + t.Fatalf("guarded promote on a legacy move: %v", err) |
| + } |
| + // The legacy transitions still work, unchanged. |
| + if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MovePromoteReady(time.Now())); err != nil { |
| + t.Fatal(err) |
| } |
| - // Abort still clears it: that is the one way out for a legacy record. |
| if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, MoveAbort()); err != nil { |
| t.Fatal(err) |
| } |
| } |
| |
| -func TestMoveAbortLegalStates(t *testing.T) { |
| +func TestGuardedAbortLegalStates(t *testing.T) { |
| + g := MoveGuard{ID: "mv_abcdefghijklmnopqrstuvwxyz", Incarnation: strings.Repeat("ab", 32)} |
| cases := []struct { |
| state, cutPhase string |
| ok bool |
| @@ -298,13 +377,33 @@ func TestMoveAbortLegalStates(t *testing.T) { |
| {MoveAborting, "", false}, {"future", "", false}, |
| } |
| for _, c := range cases { |
| - m := &MoveState{ID: "mv_abcdefghijklmnopqrstuvwxyz", State: c.state, CutPhase: c.cutPhase} |
| + m := &MoveState{ID: g.ID, Incarnation: g.Incarnation, State: c.state, CutPhase: c.cutPhase} |
| rec := &Record{Move: m} |
| - err := MoveAbort()(rec) |
| + err := MoveGuardedAbort(g)(rec) |
| if (err == nil) != c.ok { |
| t.Fatalf("abort from %s/%s: err %v", c.state, c.cutPhase, err) |
| } |
| } |
| + // The legacy MoveAbort keeps its allowlist (P0's rule): cutting refused. |
| + rec := &Record{Move: &MoveState{State: MoveCutting}} |
| + if err := MoveAbort()(rec); !errors.Is(err, ErrMoveConflict) { |
| + t.Fatal("legacy abort from cutting must stay refused") |
| + } |
| +} |
| + |
| +func TestGuardedTransitionsRefuseAReplacementMove(t *testing.T) { |
| + now := time.Now() |
| + gA := MoveGuard{ID: "mv_aaaaaaaaaaaaaaaaaaaaaaaaaa", Incarnation: strings.Repeat("ab", 32)} |
| + // A's worker observed preparing; the operator aborted A and requested |
| + // B, which is preparing too. |
| + rec := &Record{Region: "ie", Move: &MoveState{ID: "mv_bbbbbbbbbbbbbbbbbbbbbbbbbb", Incarnation: gA.Incarnation, From: "ie", To: "uk", State: MovePreparing}} |
| + if err := MoveGuardedPromoteReady(gA, now)(rec); !errors.Is(err, ErrMoveConflict) || rec.Move.State != MovePreparing { |
| + t.Fatalf("A's round promoted B: %v %+v", err, rec.Move) |
| + } |
| + rec.Move.State = MoveAborting |
| + if err := MoveGuardedAbortComplete(gA, now)(rec); !errors.Is(err, ErrMoveConflict) || rec.Move == nil { |
| + t.Fatalf("A's abort cleared B: %v", err) |
| + } |
| } |
| |
| func TestManifestReadyPathDefault(t *testing.T) { |
| @@ -501,7 +600,7 @@ const ( |
| ) |
| ``` |
| |
| -Replace `MoveRequest` with: |
| +Add beside `MoveRequest` (which stays for the legacy path): |
| |
| ```go |
| // MoveStart is what the request door decided (§7.1, §7.2): the id it |
| @@ -537,21 +636,61 @@ func MoveStartTransition(p MoveStart) func(*Record) error { |
| var moveIDShape = regexp.MustCompile(`^mv_[a-z2-7]{26}$`) |
| ``` |
| |
| -Add the legacy guard to `MovePromoteReady` and `MoveMarkCutting` (first line of each closure): |
| +**The legacy transitions stay exactly as they are** (Task 0's rule): `MoveRequest`, `MovePromoteReady`, `MoveMarkCutting`, `MoveAbort`, `MoveAbortComplete` and `MoveComplete` keep their names, bodies and `provision_test.go` tests (`provision_test.go:916-1005` call them); the legacy worker and door use them while the switch is off. **The new path is guarded by op identity from its first transition** (round-3 finding 7): every new transition takes a `MoveGuard`: |
| |
| ```go |
| - if rec.Move.Legacy() { |
| - return fmt.Errorf("%w: this move was started by an older console and has no id. abort it and request it again", ErrMoveConflict) |
| +// MoveGuard is what every Phase 1 transition carries (§7.2): the move id |
| +// and the holder incarnation. A write whose guard does not match the |
| +// record's move is a late write for another move or another incarnation |
| +// and is refused, so a delayed round for move A can never advance a |
| +// replacement move B that happens to be in the same state. |
| +type MoveGuard struct{ ID, Incarnation string } |
| + |
| +func (g MoveGuard) check(rec *Record) error { |
| + if rec.Move == nil || rec.Move.Legacy() { |
| + return fmt.Errorf("%w: no journaled move in flight", ErrMoveConflict) |
| + } |
| + if rec.Move.ID != g.ID || rec.Move.Incarnation != g.Incarnation { |
| + return fmt.Errorf("%w: the record's move is %s for incarnation %s", ErrMoveConflict, rec.Move.ID, rec.Move.Incarnation) |
| + } |
| + return nil |
| +} |
| + |
| +// MoveGuardedPromoteReady is the new worker's preparing → ready. |
| +func MoveGuardedPromoteReady(g MoveGuard, now time.Time) func(*Record) error { |
| + return func(rec *Record) error { |
| + if err := g.check(rec); err != nil { |
| + return err |
| } |
| -``` |
| + return MovePromoteReady(now)(rec) |
| + } |
| +} |
| |
| -(after the existing nil check in each; `MoveMarkCutting` also sets `rec.Move.CutPhase = CutRequested`). Rewrite `MoveAbort`'s switch: |
| +// MoveGuardedCut is the cut's entry (the console's cut door, or the source |
| +// for cut == "auto"): ready → cutting/requested. |
| +func MoveGuardedCut(g MoveGuard) func(*Record) error { |
| + return func(rec *Record) error { |
| + if err := g.check(rec); err != nil { |
| + return err |
| + } |
| + if rec.Move.State != MoveReady { |
| + return fmt.Errorf("%w: cut is only legal from ready", ErrMoveConflict) |
| + } |
| + rec.Move.State, rec.Move.CutPhase = MoveCutting, CutRequested |
| + return nil |
| + } |
| +} |
| |
| -```go |
| +// MoveGuardedAbort: legal in preparing, ready and cutting/requested (§7.11). |
| +func MoveGuardedAbort(g MoveGuard) func(*Record) error { |
| + return func(rec *Record) error { |
| + if err := g.check(rec); err != nil { |
| + return err |
| + } |
| switch rec.Move.State { |
| case MovePreparing, MoveReady: |
| case MoveCutting: |
| - if rec.Move.CutPhase != "" && rec.Move.CutPhase != CutRequested { |
| + if rec.Move.CutPhase != CutRequested { |
| return fmt.Errorf("%w: past the point of no return. the cut completes forward or moves back on its own", ErrMoveConflict) |
| } |
| case MoveAborting: |
| @@ -559,9 +698,28 @@ Add the legacy guard to `MovePromoteReady` and `MoveMarkCutting` (first line of |
| default: |
| return fmt.Errorf("%w: a move in state %q cannot be aborted", ErrMoveConflict, rec.Move.State) |
| } |
| + rec.Move.State = MoveAborting |
| + return nil |
| + } |
| +} |
| + |
| +// MoveGuardedAbortComplete clears an aborting move and records the outcome. |
| +func MoveGuardedAbortComplete(g MoveGuard, now time.Time) func(*Record) error { |
| + return func(rec *Record) error { |
| + if err := g.check(rec); err != nil { |
| + return err |
| + } |
| + if rec.Move.State != MoveAborting { |
| + return fmt.Errorf("%w: abort-complete is only legal from aborting", ErrMoveConflict) |
| + } |
| + rec.LastMove = &MoveOutcome{ID: g.ID, Outcome: "aborted", At: now.UTC()} |
| + rec.Move = nil |
| + return nil |
| + } |
| +} |
| ``` |
| |
| -(a legacy `cutting` record has `CutPhase == ""` and stays refused by the old rule: add `if rec.Move.Legacy() { return ...past the point... }` inside `case MoveCutting` before the phase check.) Delete `MoveComplete` and `MoveMarkCutting`'s reference to it in comments; Plan 2 adds `MoveFlip`. Add to `Manifest`: |
| +Plan 2's cut-phase transitions build on this `MoveGuard` (Plan 2 Task 1 no longer defines it). Test `TestGuardedTransitionsRefuseAReplacementMove`: move A is observed; the record's move is replaced by B (same state `preparing`, another id); `MoveGuardedPromoteReady(gA, now)` is `ErrMoveConflict` and B is untouched; `MoveGuardedAbortComplete` likewise. Add to `Manifest`: |
| |
| ```go |
| // Moves is the app's move declaration (move-any-app design §7.3 S1): |
| @@ -582,7 +740,7 @@ func (m Manifest) ReadyPathOrDefault() string { |
| } |
| ``` |
| |
| -Add `"regexp"` to the imports. In `internal/console/apiinstancemove.go:233` replace `provision.MoveRequest(to, time.Now())` with a `provision.MoveStartTransition(provision.MoveStart{ID: id, Incarnation: inc, From: rec.Region, To: to, Cut: "auto", By: addr, Now: time.Now()})` where, for this task only, `id, _ := moves.NewMoveID(rand.Reader)` and `inc, _ := hostclaim.Incarnation(acct.Sqid, app, host, rec.CreatedAt)` and `addr` is the first return of `s.cliUser`-style lookup (Task 4 replaces this door wholesale; here the aim is a compiling, passing tree). In `internal/host/movespass_test.go` and `internal/console/apiinstancemove_test.go`, replace each `provision.MoveRequest("uk", now)` with `provision.MoveStartTransition(provision.MoveStart{ID: "mv_abcdefghijklmnopqrstuvwxyz", Incarnation: strings.Repeat("ab", 32), From: "ie", To: "uk", Cut: "auto", Now: now})` via a test helper `startMove(to string) func(*provision.Record) error` in each test package. `TestMovesPassCutSequence` and the other cut tests that expect `MoveComplete`'s flip (`got.Region != "uk"`) are deleted in this task and rewritten in Plan 2 Task 3; list them in the commit message (`TestMovesPassCutSequence`, `TestMovesPassCutUsesRouteDBPath`, `TestMovesPassCutFencesWakesUntilTheRouteRetires`, `TestMovesPassCutStopsWhenTheFenceFails`, `TestMovesPassCutAfterBusyReadyLandsByteExact`, `TestMovesPassCutWithMultipartSourcesLandsExact`, `TestMovesPassCutStopsOnADeferredCopy`, `TestMovesPassCutFencedByLayoutJournal`, `TestMovesPassCutFencedWhenTheJournalCannotBeRead`). In `movespass.go` the `MoveCutting` arm becomes `return nil // Plan 2 rewrites the cut` after the `SetStatus`/fence steps are removed, so no test depends on `MoveComplete`. |
| +Add `"regexp"` to the imports. Nothing else changes in this task: the console's door, the worker and every existing test (`provision_test.go`, `movespass_test.go`, `apiinstancemove_test.go`) keep calling `MoveRequest`, `MoveComplete` and the rest, because the legacy path stays reachable while the switch is off (Task 0). The new transitions get their own tests here; the new door is Task 4's and the new worker Task 14's, each beside the legacy code under the switch. |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -623,7 +781,9 @@ import ( |
| "time" |
| ) |
| |
| -func servingClaim() *Claim { |
| +// movingTestClaim is this file's fixture (delete_test.go already defines |
| +// servingClaim(t, ctx, s) with another shape). |
| +func movingTestClaim() *Claim { |
| return &Claim{V: 1, Host: "a.go.example.com", Gen: 3, |
| Holder: &Holder{Family: "instance", Key: "k", Account: "bab", App: "app", Incarnation: "ab", Location: "instances/a.go.example.com/ab/", Stamp: 3}, |
| Op: Op{Kind: KindServing, Exec: "ie-box"}, |
| @@ -632,7 +792,7 @@ func servingClaim() *Claim { |
| |
| func TestBeginMoveFromServingOnly(t *testing.T) { |
| now := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| - c := servingClaim() |
| + c := movingTestClaim() |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", now)(c); err != nil { |
| t.Fatal(err) |
| } |
| @@ -649,12 +809,12 @@ func TestBeginMoveFromServingOnly(t *testing.T) { |
| t.Fatalf("second move: %v", err) |
| } |
| // exec must be the source. |
| - d := servingClaim() |
| + d := movingTestClaim() |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "us-box", "uk-box", "uk", now)(d); !errors.Is(err, ErrWrongState) { |
| t.Fatalf("source not exec: %v", err) |
| } |
| // A pending chore in the target region refuses (CC11). |
| - e := servingClaim() |
| + e := movingTestClaim() |
| e.Obligations = []Obligation{{ID: "o1", Op: "x", Kind: "park-replica", Region: "uk", Retirement: "r", Owner: "region:uk"}} |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", now)(e); !errors.Is(err, ErrPendingChore) { |
| t.Fatalf("pending chore: %v", err) |
| @@ -662,7 +822,7 @@ func TestBeginMoveFromServingOnly(t *testing.T) { |
| } |
| |
| func TestTouchAppendsOnceUnderTheOp(t *testing.T) { |
| - c := servingClaim() |
| + c := movingTestClaim() |
| _ = BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", time.Now())(c) |
| if err := Touch("mv_abcdefghijklmnopqrstuvwxyz", []string{"uk"}, nil)(c); err != nil { |
| t.Fatal(err) |
| @@ -781,7 +941,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/console/apiinstancemove_test.go` (extend `TestAPIInstanceMoveVerbs`; add `TestMoveDoorRefusesWithoutClaimWrite`, `TestMoveMutationsAreOperatorOnly`) |
| |
| **Interfaces:** |
| -- Consumes: `hostclaim.Read`, `hostclaim.Update`, `hostclaim.BeginMove`, `hostclaim.Incarnation`, `moves.NewMoveID`, `s.store.IsOperator`, `s.store.CLITokenAddress`, `provision.GetManifest` (regions and `PrivateStorage`). This task creates `internal/moves/floor.go` with the one constant `const ProtocolVersion = 1` and adds `MovesProtocol int \`json:"moves_protocol"\`` to the console's edges struct in `instancesplacement.go`; Task 12 fills the file (the floor object, the box's writer, the activation refusal). |
| +- Consumes: `hostclaim.Read`, `hostclaim.Update`, `hostclaim.BeginMove`, `hostclaim.Incarnation`, `moves.NewMoveID`, `s.store.IsOperator`, `s.store.CLITokenAddress`, `provision.GetManifest` (regions and `PrivateStorage`), `s.cfg.MovesEnabled` (Task 0), `eligibleBoxes` and `chooseBox` (`boxchoice.go`). This task creates `internal/moves/floor.go` with the one constant `const ProtocolVersion = 1` and adds `MovesProtocol int \`json:"moves_protocol"\`` to the console's edges struct in `instancesplacement.go`; Task 13 fills the file (the floor object, the box's writer, the activation refusal). |
| - Produces: `(s *Server) moveOperatorGate(w, r) (acct *Account, app, host string, rec *provision.Record, addr string, ok bool)`; the door's refusal sentences (copy ids below). |
| |
| The door's order (§7.1, K2): `moveGate` (class rails), operator (`IsOperator`, 404 otherwise), body, region validity and `placements`, the manifest's `regions` when non-empty, a home with replicas (`len(rec.Replicas) > 0`), private storage (the instances manifest's `PrivateStorage`; the refusal is M-C17), then the claim: `hostclaim.Read`; absent or `Holder == nil` → 409 "the instance has no host claim yet. it is adopted on its next reconcile pass"; holder incarnation ≠ `hostclaim.Incarnation(acct, app, host, rec.CreatedAt)` → 409; `Op.Kind != KindServing` → 409 naming the kind; the target box: the lexically-first box publishing the target placement in `readEdgesFiles` (T8 is the target's own check); `hostclaim.Update(ctx, s.cfg.Releases, obs, hostclaim.BeginMove(id, addr, obs.Claim.Op.Exec, targetBox, to, now))`; then `provision.SetRecordMove(... MoveStartTransition ...)`. If the record write fails after the claim CAS, the door answers 502 and logs; the console pass of Plan 5 Task 7 finds a `moving` claim with no record move and releases it (noted there). |
| @@ -889,7 +1049,7 @@ func TestMoveDoorRefusesWithoutClaimWrite(t *testing.T) { |
| |
| `operatorInstanceFixture` and `writeServingClaim` are test helpers added to `apiinstancemove_test.go`; `operatorInstanceFixture` calls `instanceFixture` then marks the token's address an operator with `fx.st.SetOperator(addr, true)` (`store.go:1952`, the call `alertwatch_test.go:52` and `account_test.go:250` already make), where `addr` is the address `instanceFixture` signed the token for. |
| |
| -Update `TestAPIInstanceMoveVerbs` to use `operatorInstanceFixture`, the `moves_protocol` field on its edges file, and a serving claim written before the first request. |
| +`TestAPIInstanceMoveVerbs` stays as it is (the legacy door, switch off). Add `TestAPIInstanceMoveVerbsPhase1` with `fx.srv.cfg.MovesEnabled = true`, `operatorInstanceFixture`, the `moves_protocol` field on its edges file and a serving claim written before the first request; it mirrors the legacy test's matrix with the new rules: a second request while in flight is 409; abort from `cutting/requested` is **202** (the guarded abort allows it, Task 2); after an abort completes the test releases the claim (`hostclaim.Update(ReleaseMove(id, "ie-box"))`, Task 3) before the next request, because a request against a `moving` claim is 409 by design. |
| |
| - [ ] **Step 2: Run tests to verify they fail** |
| |
| @@ -933,12 +1093,12 @@ func (s *Server) moveOperatorGate(w http.ResponseWriter, r *http.Request) (acct |
| |
| ```go |
| func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Request) { |
| - acct, app, host, rec, addr, ok := s.moveOperatorGate(w, r) |
| - if !ok { |
| + if !s.cfg.MovesEnabled { // Task 0: off = today's door, byte for byte |
| + s.legacyHandleAPIInstanceMoveRequest(w, r) |
| return |
| } |
| - if !s.cfg.MovesEnabled { // D13: CARLOS_MOVES_PHASE1=on, off by default |
| - jsonError(w, http.StatusConflict, "moves are not enabled on this deployment") |
| + acct, app, host, rec, addr, ok := s.moveOperatorGate(w, r) |
| + if !ok { |
| return |
| } |
| ctx := r.Context() |
| @@ -1100,7 +1260,11 @@ func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box s |
| } |
| ``` |
| |
| -`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 12 adds the writer's side). Private storage is the instances manifest's `PrivateStorage` (`provision.go:253`, the `private_storage` field), read through the `GetManifest` call the regions check already makes: one read, two refusals. `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` switch to `moveOperatorGate`. In `authcli.go`, the three mutation routes keep `svcrole.Allow(svcrole.Instance)` only so a service token meets the operator 404 rather than the role 403 leak; add `s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveResolve)` with a handler that, for this plan, runs `moveOperatorGate` then answers 409 `"nothing to resolve yet"`; Plan 4 Task 10 and Plan 6 Task 2 give it its body. Imports: `crypto/rand`, `slices`, `github.com/carlosframework/platform/internal/hostclaim`, `.../internal/moves`, `.../internal/release`. |
| +`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 13 adds the writer's side). Private storage is the instances manifest's `PrivateStorage` (`provision.go:253`, the `private_storage` field), read through the `GetManifest` call the regions check already makes: one read, two refusals. |
| + |
| +**The legacy door stays** (Task 0): today's bodies of `handleAPIInstanceMoveRequest`, `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` are renamed `legacyHandleAPIInstanceMoveRequest` (and siblings) unchanged, and each new handler's first line is `if !s.cfg.MovesEnabled { s.legacyHandleAPIInstanceMove…(w, r); return }`; `handleAPIInstanceMoveResolve` answers 404 when the switch is off (the route did not exist before). The existing `TestAPIInstanceMoveVerbs` and `TestAPIInstanceMoveClassRails` run as they are with the switch off (no change in behaviour, which is the point); the new tests below set `fx.srv.cfg.MovesEnabled = true`. |
| + |
| +**The target box** (spec rev 7.20 §7.1, D15): `moveTargetBox` uses main's same-package `eligibleBoxes(snap edgesSnapshot, verified map[string]boxid.Identity, placement, awsRegion, homeName, expectedFP string, lost map[string]bool) []boxCandidate` and `chooseBox(cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool)` (`internal/console/boxchoice.go:28`, `:76`), the create door's own helpers, with the same `verified`, `expectedFP` (`s.expectedSecretsFP()`) and `running` inputs the create door passes; more than one candidate on either side → M-C21; `chooseBox`'s last tie-break is changed in this task from "the lowest box id" to the approved SHA-256 of `host + "\x00" + box reference` (region HA §5.2, P5-D2), with `TestChooseBoxTieBreakIsTheApprovedHash` beside boxchoice's existing tests, since the create door and the move door must agree. The chosen box goes into `op.intent.target` (`BeginMove`'s `targetBox`), and T8 and T9 (Task 11) bind to it: the reporting box's own id must equal `op.intent.target`, not merely publish the placement. `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` switch to `moveOperatorGate`. In `authcli.go`, the three mutation routes keep `svcrole.Allow(svcrole.Instance)` only so a service token meets the operator 404 rather than the role 403 leak; add `s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveResolve)` with a handler that, for this plan, runs `moveOperatorGate` then answers 409 `"nothing to resolve yet"`; Plan 4 Task 10 and Plan 6 Task 2 give it its body. Imports: `crypto/rand`, `slices`, `github.com/carlosframework/platform/internal/hostclaim`, `.../internal/moves`, `.../internal/release`. |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -1213,19 +1377,26 @@ func TestMirrorPrefixRecordsDestinationVersions(t *testing.T) { |
| // the destination's ETag equal to the source's but no receipt (or a |
| // receipt for an older version of the key). It recopies rather than |
| // sealing a version it cannot vouch for. |
| - f.put("src", "instances/h/0000/a.ltx", "e1-new") |
| + f.putSized("src", "instances/h/0000/a.ltx", "e1-new", 64) |
| f.n++ // a copy whose receipt was never persisted |
| - f.store("dst", "moves/h/mv/replica/0000/a.ltx", fakeObject{MirrorObject{ETag: "e1-new"}, "e1-new"}) |
| + f.store("dst", "moves/h/mv/replica/0000/a.ltx", fakeObject{MirrorObject{ETag: "e1-new", Size: 64}, "e1-new"}) |
| r4, err := MirrorPrefix(context.Background(), f, "src", "instances/h/", "dst", "moves/h/mv/replica/", rc) |
| - if err != nil || r4.Versions["0000/a.ltx"] == r1.Versions["0000/a.ltx"] || r4.BytesCopied == 0 { |
| + if err != nil || r4.Versions["0000/a.ltx"] == r1.Versions["0000/a.ltx"] || r4.Changed == 0 || r4.BytesCopied != 64 { |
| t.Fatalf("stale receipt sealed: %+v %v", r4, err) |
| } |
| - // A plain MirrorStore records no versions and still works. |
| + // The plain store's second identical round copies nothing (the |
| + // existing TestMirrorPrefix contract, kept): see the next block. |
| + // A plain MirrorStore records no versions and keeps today's rule: an |
| + // equal ETag skips the copy with no receipt needed. |
| g := newFakeMirror() |
| - g.put("src", "instances/h/0000/a.ltx", "e1") |
| + g.putSized("src", "instances/h/0000/a.ltx", "e1", 64) |
| r3, err := MirrorPrefix(context.Background(), g, "src", "instances/h/", "dst", "moves/h/mv/replica/", rc) |
| - if err != nil || len(r3.Versions) != 0 { |
| - t.Fatalf("plain store: %+v %v", r3, err) |
| + if err != nil || r3.Versions != nil || r3.BytesCopied != 64 { |
| + t.Fatalf("plain store round 1: %+v %v", r3, err) |
| + } |
| + r3b, err := MirrorPrefix(context.Background(), g, "src", "instances/h/", "dst", "moves/h/mv/replica/", rc) |
| + if err != nil || r3b.Changed != 0 || r3b.BytesCopied != 0 { |
| + t.Fatalf("plain store recopied an unchanged object: %+v %v", r3b, err) |
| } |
| } |
| ``` |
| @@ -1306,7 +1477,7 @@ type VersionedPutter interface { |
| } |
| ``` |
| |
| -`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: initialise `round.Versions = map[string]string{}`. A receipt vouches for a version only when it matches the **current** source and destination completely: `rcpt.SrcETag == s.ETag && rcpt.SrcSize == s.Size && rcpt.DstETag == d.ETag && d.Size == s.Size && rcpt.DstVersionID != ""`. The `staged && d.ETag == s.ETag` early-continue therefore becomes: read the receipt; if it matches completely, record `round.Versions[r] = rcpt.DstVersionID` and continue; otherwise fall through to the copy below (a copy whose receipt was lost, or a receipt from an earlier version of the key, is recopied, which yields a fresh version id; the stale version stays unreferenced). The existing receipt-match branch records the version the same way. Replace the copy call with: |
| +`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: `versioned := m.(VersionedCopier)` decides the mode once; `round.Versions` is non-nil only when `versioned`. **The plain `MirrorStore` path is unchanged** (round-3 finding 4: `TestMirrorPrefix` requires a second identical round to copy nothing, and the legacy worker depends on it): the `staged && d.ETag == s.ETag` early-continue and the receipt-match branch keep their exact behaviour. **Only when `versioned`** does a receipt have to vouch for the version: in the equal-ETag branch, read the receipt; if `rcpt.SrcETag == s.ETag && rcpt.SrcSize == s.Size && rcpt.DstETag == d.ETag && d.Size == s.Size && rcpt.DstVersionID != ""`, record `round.Versions[r] = rcpt.DstVersionID` and continue; otherwise fall through to the copy (a copy whose receipt was lost, or a receipt from an earlier version of the key, is recopied, which yields a fresh version id; the stale version stays unreferenced). Replace the copy call with: |
| |
| ```go |
| var etag, ver string |
| @@ -1342,7 +1513,13 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/moves/lease.go`, `internal/moves/lease_test.go` |
| |
| **Interfaces:** |
| -- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; `lease.NewS3Mirror` (`mirror.go:388`), extended in this task with `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (StagingMirror, error)`: a `MirrorStore` plus `VersionedCopier`/`VersionedPutter` whose every mutation is **presigned at the call** and sent once. `CopyVersioned` builds the request with `s3.NewPresignClient(client).PresignCopyObject(ctx, in, s3.WithPresignClientFromClientOptions(...), func(o *s3.PresignOptions){ o.Expires = 15 * time.Minute })` using `o.ClientOptions` with `aws.Config.Retryer = aws.NopRetryer{}`, so `X-Amz-Date` is the call's clock and the signature exists before the send; the presigned URL is sent with one `http.Client{Timeout: 120 * time.Second}` `Do`, no retry; the copy's `x-amz-version-id` response header is the version id. `PutVersioned` presigns `PutObject` the same way with the body's SHA-256 in the signed headers. `Delete` presigns `DeleteObject`. A caller that wants a retry calls again, through a fresh `Guarded.Before`, which signs again at that check. This is §7.3's "the signature is computed at that check, with the check's time as its `X-Amz-Date`, and never recomputed" made literal; the only pause the rule cannot cover is between `Before`'s clock read and the presigner's clock read, two statements apart in one goroutine, which the skew allowance covers. |
| +- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; the SDK's SigV4 signer, `github.com/aws/aws-sdk-go-v2/aws/signer/v4` (in the pinned `aws-sdk-go-v2 v1.44.0`): `v4.NewSigner()` and `(Signer).SignHTTP(ctx, credentials aws.Credentials, r *http.Request, payloadHash, service, region string, signingTime time.Time, ...) error`; `aws.Config.Credentials.Retrieve(ctx)` for the credentials the mirror's clients already load. **There is no copy presigner in S3 v1.105.0** (round-3 finding 3), so the staging mirror signs by hand: `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (*StagingMirror, error)` is a `MirrorStore` plus `VersionedCopier`/`VersionedPutter` whose every mutation builds the plain S3 REST request itself, signs it with `SignHTTP` at **`signingTime = Clock()`** (the call's clock, two statements after `Guarded.Before`'s check), and sends it once with `http.Client{Timeout: 120 * time.Second}` and no retry: |
| + - `CopyVersioned(ctx, srcBucket, srcKey, srcETag, dstBucket, dstKey)`: `PUT https://<dstBucket>.s3.<dstRegion>.amazonaws.com/<dstKey>` with headers `x-amz-copy-source: /<srcBucket>/<url-escaped srcKey>`, `x-amz-copy-source-if-match: "<srcETag>"` when `srcETag != ""`, payload hash `UNSIGNED-PAYLOAD`; a 200 whose body is `<CopyObjectResult><ETag>…</ETag>…` gives the ETag (unquoted), and the `x-amz-version-id` response header the version id; a 412 `PreconditionFailed` (the error XML's `<Code>`) is `ErrCopySourceChanged`; any other non-2xx is an error naming the code; a 200 with an `<Error>` body (S3's late-failure form for copies) is an error too. |
| + - `PutVersioned(ctx, bucket, key, body)`: `PUT` with the body's SHA-256 as the payload hash (`x-amz-content-sha256`) and `Content-Type: application/json`; `x-amz-version-id` is the version id. `PutIfAbsentVersioned(ctx, bucket, key, body)` adds `If-None-Match: *` and maps 412 to `lease.ErrExists` (the guarded create-only put `move.json` needs, Task 14). |
| + - `Delete(ctx, bucket, key)`: `DELETE`, 204. |
| + - `List` and `Get` delegate to the ordinary `s3Mirror` (reads are not mutations; the guard is about writes landing after the lease). |
| + - `Clock func() time.Time` on the struct (production `time.Now`; a test injects); `Credentials aws.CredentialsProvider` from the dst client's config; `Endpoint` override for tests (`NewStagingMirrorAt(t, url, bucketRegion)`). |
| + A caller that wants a retry calls again, through a fresh `Guarded.Before`, which signs again at that check. This is §7.3's "the signature is computed at that check, with the check's time as its `X-Amz-Date`, and never recomputed" made literal; the only pause the rule cannot cover is between `Before`'s clock read and `SignHTTP`'s `signingTime`, two statements apart in one goroutine, which the skew allowance covers. |
| - Produces: `moves.StagingLeaseTTL = 45 * time.Minute` (D3: the ordinary 15-minute `LeaseTTL` cannot leave the signature's life in front of `SafeUntil`); `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func() error) error`: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life, so `Before` refuses with `moves.ErrLeaseUnsafe` when `Now().Add(SignatureLife + Skew).After(safe)` (`SignatureLife = 15 * time.Minute`, `Skew` default 1 min) and otherwise calls `fn` at once in the same goroutine. The lane's retry loop (Task 14) calls `Before` again for each retry; nothing retries inside `fn`. |
| |
| - [ ] **Step 1: Write the failing test** |
| @@ -1384,51 +1561,41 @@ func TestGuardedRefusesWithinTheSignatureLife(t *testing.T) { |
| } |
| } |
| |
| -// Against a local S3 (internal/lease/leasetest: NewVersionedBucket and |
| -// NewS3Server, the package's own fixture), a lease acquired with |
| -// StagingLeaseTTL passes the guard right after acquisition and a lease |
| -// acquired with lease.LeaseTTL does not. The client is pointed at the |
| -// server the way lease's own tests do (read lease_test.go for the |
| -// endpoint override the package uses; use that seam, not a new one). |
| +// The real lease settings, by arithmetic: lease.Held's SafeUntil is |
| +// Until minus the renewal margin, so a fresh lease's SafeUntil is at most |
| +// TTL ahead. A fresh StagingLeaseTTL lease passes the guard; a fresh |
| +// lease.LeaseTTL lease never can, which is why StagingLeaseTTL exists. |
| +// lease.AcquireTTL itself is covered where lease's own Acquire tests are |
| +// (lease_test.go, S3-gated): this package adds no second S3 fixture. |
| func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| - ctx := context.Background() |
| - bucket := leasetest.NewVersionedBucket() |
| - handler, port := leasetest.NewS3Server(bucket, "data-ie") |
| - srv := httptest.NewServer(handler) |
| - defer srv.Close() |
| - _ = port |
| - region := lease.TestEndpoint(t, srv.URL) // the package's endpoint seam for tests; name per lease_test.go |
| - held, err := lease.AcquireTTL(ctx, "data-ie", "control/staging-leases/uk/h", region, StagingLeaseTTL) |
| - if err != nil { |
| - t.Fatal(err) |
| + t0 := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| + fresh := func(ttl time.Duration) SafeUntilGuard { |
| + return fixedLease{t0.Add(ttl - lease.RenewMargin)} // what Held.SafeUntil answers right after Acquire; RenewMargin exported in this task |
| } |
| - defer held.Release(ctx) |
| - if err := (Guarded{Lease: held, Now: time.Now}).Before(func() error { return nil }); err != nil { |
| - t.Fatalf("staging lease refused at acquisition: %v", err) |
| + if err := (Guarded{Lease: fresh(StagingLeaseTTL), Now: func() time.Time { return t0 }}).Before(func() error { return nil }); err != nil { |
| + t.Fatalf("a fresh staging lease is refused: %v", err) |
| } |
| - short, err := lease.AcquireTTL(ctx, "data-ie", "control/staging-leases/uk/h2", region, lease.LeaseTTL) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - defer short.Release(ctx) |
| - if err := (Guarded{Lease: short, Now: time.Now}).Before(func() error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| - t.Fatalf("a 15 minute lease passed the guard: %v", err) |
| + if err := (Guarded{Lease: fresh(lease.LeaseTTL), Now: func() time.Time { return t0 }}).Before(func() error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| + t.Fatalf("a fresh 15 minute lease passed the guard: %v", err) |
| } |
| } |
| |
| -// The staging mirror signs at the check and never again: a presigned |
| -// request carries the X-Amz-Date of the moment it was built, and a second |
| -// attempt is a second Before. |
| -func TestStagingMirrorPresignsAtTheCheck(t *testing.T) { |
| +// The staging mirror signs at the check and never again: a request carries |
| +// the X-Amz-Date of the moment it was signed, and a second attempt is a |
| +// second Before. The fixture is internal/lease/leasetest's local S3 |
| +// (NewVersionedBucket + NewS3Server), which serves plain PUT, create-only |
| +// PUT and version-specific DELETE; the test asserts the signed header, |
| +// not AWS's acceptance of it. |
| +func TestStagingMirrorSignsAtTheCheck(t *testing.T) { |
| bucket := leasetest.NewVersionedBucket() |
| handler, _ := leasetest.NewS3Server(bucket, "data-uk") |
| var dates []string |
| srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| - dates = append(dates, r.URL.Query().Get("X-Amz-Date")) |
| + dates = append(dates, r.Header.Get("X-Amz-Date")) |
| handler.ServeHTTP(w, r) |
| })) |
| defer srv.Close() |
| - m := lease.NewStagingMirrorAt(t, srv.URL) // the test constructor the task adds beside NewStagingMirror |
| + m := lease.NewStagingMirrorAt(t, srv.URL, "eu-west-2") // the test constructor beside NewStagingMirror: static test credentials, the endpoint override, path-style addressing |
| t0 := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| clock := t0 |
| m.Clock = func() time.Time { return clock } |
| @@ -1442,6 +1609,13 @@ func TestStagingMirrorPresignsAtTheCheck(t *testing.T) { |
| if len(dates) != 2 || dates[0] != "20261006T090000Z" || dates[1] != "20261006T090300Z" { |
| t.Fatalf("X-Amz-Date per request: %v", dates) |
| } |
| + // create-only through the same signed path |
| + if _, err := m.PutIfAbsentVersioned(context.Background(), "data-uk", "moves/h/mv/move.json", []byte("{}")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := m.PutIfAbsentVersioned(context.Background(), "data-uk", "moves/h/mv/move.json", []byte("{}")); !errors.Is(err, lease.ErrExists) { |
| + t.Fatalf("second create-only put: %v", err) |
| + } |
| } |
| ``` |
| |
| @@ -1962,9 +2136,12 @@ func Extract(r io.Reader, filesDir, envPath string) error { |
| seen[dst] = true |
| switch hdr.Typeflag { |
| case tar.TypeDir: |
| - if err := os.MkdirAll(dst, fs.FileMode(hdr.Mode).Perm()|0o700); err != nil { |
| + // created writable, so children can land; the exact mode is |
| + // restored after every entry is written (modes below) |
| + if err := os.MkdirAll(dst, 0o700); err != nil { |
| return err |
| } |
| + modes = append(modes, modeFix{dst, fs.FileMode(hdr.Mode).Perm()}) |
| case tar.TypeReg: |
| if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { |
| return err |
| @@ -1984,13 +2161,99 @@ func Extract(r io.Reader, filesDir, envPath string) error { |
| if err := f.Close(); err != nil { |
| return err |
| } |
| + // the umask applied at OpenFile; the archived mode is exact (§8.1) |
| + if err := os.Chmod(dst, fs.FileMode(hdr.Mode).Perm()); err != nil { |
| + return err |
| + } |
| + default: |
| + return fmt.Errorf("%w: %s", ErrSpecialFile, hdr.Name) |
| + } |
| + } |
| +} |
| + |
| +type modeFix struct { |
| + path string |
| + mode fs.FileMode |
| +} |
| + |
| +// restoreModes sets directory modes deepest first, after their contents |
| +// exist, so a read-only directory (0555) is reconstructed exactly without |
| +// blocking its own extraction. |
| +func restoreModes(fixes []modeFix) error { |
| + sort.Slice(fixes, func(i, j int) bool { return len(fixes[i].path) > len(fixes[j].path) }) |
| + for _, m := range fixes { |
| + if err := os.Chmod(m.path, m.mode); err != nil { |
| + return err |
| + } |
| + } |
| + return nil |
| +} |
| + |
| +// Verify is T5 (§7.3): it opens the sealed object with priv, streams the |
| +// tar out of zstd and checks everything the manifest promised: the |
| +// plaintext digest, every entry's path, size and sha256, the entry types |
| +// (regular files and directories only), and the uncompressed cap. It |
| +// writes nothing and returns the first mismatch. |
| +func Verify(sealed io.Reader, m Manifest, priv *ecdh.PrivateKey, b appconfig.Binding) error { |
| + pr, pw := io.Pipe() |
| + go func() { pw.CloseWithError(Open(sealed, m.SealedKey, priv, b, pw)) }() |
| + h := sha256.New() |
| + zr, err := zstd.NewReader(io.TeeReader(pr, h)) |
| + if err != nil { |
| + return err |
| + } |
| + defer zr.Close() |
| + tr := tar.NewReader(zr) |
| + want := map[string]Entry{} |
| + for _, e := range m.Entries { |
| + want[e.Path] = e |
| + } |
| + var bytes int64 |
| + for { |
| + hdr, err := tr.Next() |
| + if err == io.EOF { |
| + break |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + switch hdr.Typeflag { |
| + case tar.TypeDir: |
| + continue |
| + case tar.TypeReg: |
| default: |
| return fmt.Errorf("%w: %s", ErrSpecialFile, hdr.Name) |
| } |
| + e, ok := want[hdr.Name] |
| + if !ok || e.Size != hdr.Size { |
| + return fmt.Errorf("archive: entry %q not in the manifest or its size differs", hdr.Name) |
| + } |
| + fh := sha256.New() |
| + n, err := io.Copy(fh, tr) |
| + if err != nil { |
| + return err |
| + } |
| + bytes += n |
| + if bytes > Cap { |
| + return ErrOverCap |
| + } |
| + if digestString(fh) != e.SHA256 { |
| + return fmt.Errorf("archive: entry %q digest differs", hdr.Name) |
| + } |
| + delete(want, hdr.Name) |
| + } |
| + if len(want) != 0 { |
| + return fmt.Errorf("archive: %d manifest entries missing from the archive", len(want)) |
| } |
| + if got := digestString(h); got != m.Plain { |
| + return fmt.Errorf("archive: plaintext digest %s, manifest says %s", got, m.Plain) |
| + } |
| + return nil |
| } |
| ``` |
| |
| +(`Extract` declares `var modes []modeFix` before the loop and calls `restoreModes(modes)` on `io.EOF` before returning nil; `Verify`'s `h` hashes the zstd stream, which is what `Build`'s `capWriter` hashed, so `Plain` is the same bytes.) Tests added: `TestExtractPreservesModes` (a `0555` directory holding a `0400` file round-trips to exactly those modes; `PlainDigest` of the extracted tree equals the original) and `TestVerifyRefusesWhatTheManifestDoesNotPromise` (a correctly sealed archive with a manifest whose `Plain` is wrong, whose entry list lacks a file, whose entry sha256 differs, plus a hand-built sealed archive containing a symlink entry, and one over the cap: each returns the named error; the good archive verifies). |
| + |
| `seal.go`: |
| |
| ```go |
| @@ -2143,6 +2406,25 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| --- |
| |
| +### Task 7b: Held-row render, materialization and the strict next spawn |
| + |
| +Moved here from Plan 2 (its Tasks 4 and 7) because T3, T7 and S2 need the real thing, not scaffolding (round-3 review). |
| + |
| +**Files:** |
| +- Create: `internal/appconfig/renderheld.go`, `internal/appconfig/renderheld_test.go`, `internal/release/materializeheld.go`, `internal/release/materializeheld_test.go` |
| +- Modify: `internal/activator/backend_exec.go` (`Strict`, `NextSpawnStrict`, the strict spawn path at `spawnInCgroup`), `internal/activator/backend_exec_test.go` |
| + |
| +**Interfaces:** |
| +- `appconfig.RenderHeld(ctx, s release.Store, priv *ecdh.PrivateKey, dataDir string, rt registry.Route, op string, box string) (HeldRender, error)` with `HeldRender{Path, SHA256 string; BundleETags map[string]string; OverlayETag string}`: `LoadMergedHost(ctx, s, acct, app, rt.Environment, rt.Host)`, rendered through the same file writer `syncExecHost` uses (the file-writing half extracted into `renderExecConfig(rt, m, key, priv, dataDir) (path string, changed bool, err error)` and called from both), written whether or not it changed, and a move-bound ack through `WriteConfigAck` with three additive `ConfigAck` fields: `MoveOp string \`json:"move_op,omitempty"\``, `FileSHA256 string \`json:"file_sha256,omitempty"\``, `BundleETags map[string]string \`json:"bundle_etags,omitempty"\``. `SyncExec` keeps skipping non-ready rows (`execRouteBase`), so the two never race on one file. |
| +- `release.MaterializeHeld(ctx, s Store, optDir string, rt registry.Route, version string) (*Manifest, error)`: `ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, version, rt.Host)` then the live symlink exactly as `adoptHost` points it (that symlink step extracted into `pointLive(optDir, host, target string) error` if it is inline), no restart, no bake-hold check: a held row runs exactly the pinned release. |
| +- `ExecBackend.Strict func(host string) bool` (when true the spawn uses `Cgroups.PrepareStrict(host, limit)` then `Prepared.Start(cmd)`, keeping the `*cgroup.Prepared` for the process's life and running `KillAndEmpty` then `Close` on exit; a failure refuses the spawn with `ErrStrictRequired`); `(b *ExecBackend) NextSpawnStrict(host string) moves.Check` (S2: OK when `Strict(host)` and cgroup v2 delegation is present, `cgroup.Manager.Root() != ""`, and `PrepareStrict` is supported on this build). The host package sets `eb.Strict` from the hold view (a host whose record has a move in flight and the switch on); off → never strict, today's behaviour. |
| + |
| +Tests: `RenderHeld` writes the file for a `landing` row `SyncExec` would skip and the ack carries `move_op`, the sha256 and the bundle etags; a second call with no change still writes an ack. `MaterializeHeld` stages the pinned version into a temp `optDir` and points `live/<host>` at it, never at the channel head. `TestSpawnUnderAMoveIsStrictOrRefused` (a fake `Cgroups` whose `PrepareStrict` fails → `Start` returns `ErrStrictRequired` and spawns nothing; with `Strict` false the plain path runs as today). |
| + |
| +- [ ] **Steps 1 to 5**. Commit: `git commit -m "appconfig.RenderHeld, release.MaterializeHeld, strict spawns under a move (§9.1, §9.2, §5.3)" ...` |
| + |
| +--- |
| + |
| ### Task 8: Timers snapshot and the canonical digest |
| |
| **Files:** |
| @@ -2354,6 +2636,15 @@ func TestEnvReportRevealsNamesAndEqualityOnly(t *testing.T) { |
| if len(diffs) != 1 || diffs[0].Name != "CARLOS_REGION" || diffs[0].Kind != "region-same" { |
| t.Fatalf("same region: %v", diffs) |
| } |
| + noRegion := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_CONTROL_SOCKET": "/run/y", "SECRET": "s3"}) |
| + if diffs := CompareEnv(src, noRegion, true); len(diffs) != 1 || diffs[0].Kind != "region-missing" { |
| + t.Fatalf("missing region: %v", diffs) |
| + } |
| + // a layout variable absent from both: still required of a flat landing |
| + srcNoLayout := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "ie"}) |
| + if diffs := CompareEnv(srcNoLayout, ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "uk"}), true); len(diffs) != 2 { |
| + t.Fatalf("layout names absent from both: %v", diffs) |
| + } |
| drift := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:b", "CARLOS_REGION": "uk", "EXTRA": "x"}) |
| kinds := map[string]string{} |
| for _, d := range CompareEnv(src, drift, false) { |
| @@ -2493,7 +2784,9 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| tv, ok := target.Vars[name] |
| switch Class(name) { |
| case "region": |
| - if ok && tv == s { |
| + if !ok { |
| + diffs = append(diffs, EnvDiff{name, "region-missing"}) // the target must set its own region |
| + } else if tv == s { |
| diffs = append(diffs, EnvDiff{name, "region-same"}) |
| } |
| case "layout": |
| @@ -2517,6 +2810,16 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| diffs = append(diffs, EnvDiff{name, "extra"}) |
| } |
| } |
| + // layout names absent from BOTH reports are still checked: a flat |
| + // landing must provide every layout variable the flat layout provides |
| + for name := range LayoutVars { |
| + if _, inSrc := source.Vars[name]; inSrc { |
| + continue // handled above |
| + } |
| + if _, inTgt := target.Vars[name]; flatLayout && !inTgt { |
| + diffs = append(diffs, EnvDiff{name, "layout-missing"}) |
| + } |
| + } |
| sort.Slice(diffs, func(i, j int) bool { return diffs[i].Name < diffs[j].Name }) |
| return diffs |
| } |
| @@ -2526,17 +2829,24 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| |
| ```go |
| // LaunchEnvironment is the composed environment the next Start of rt |
| -// would spawn with: childEnv over instanceEnv plus the agent-owned |
| -// variables Start layers on top, decoded back to a map. The move's launch |
| -// report (moves spec §7.3, codex R2-8) is computed over exactly this, so |
| -// a target compares what would run, not a reconstruction of it. |
| +// would spawn with: childEnv over agentOwnedVars over instanceEnv, the |
| +// exact composition Start uses (backend_exec.go:1006, agentOwnedVars(rt, |
| +// fileEnv, haveControl)), decoded back to a map. agentOwnedVars REMOVES |
| +// tenant-provided CARLOS_CONTROL_SOCKET, CARLOS_PURGE and CARLOS_REGION |
| +// when the agent does not provide them, so a report over mergeEnv alone |
| +// would misstate the environment; haveControl is whether this box's |
| +// spawn binds the control listener (b.Control != nil && the layout's |
| +// control socket path can be bound), computed the way Start computes it |
| +// before the spawn. The move's launch report (moves spec §7.3, codex |
| +// R2-8) is computed over exactly this, so a target compares what would |
| +// run, not a reconstruction of it. |
| func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, error) { |
| fileEnv, err := b.instanceEnv(rt) |
| if err != nil { |
| return nil, err |
| } |
| out := map[string]string{} |
| - for _, kv := range childEnv(mergeEnv(fileEnv, b.agentOwnedEnv(rt))) { |
| + for _, kv := range childEnv(b.agentOwnedVars(rt, fileEnv, b.controlAvailable(rt))) { |
| if i := strings.IndexByte(kv, '='); i > 0 { |
| out[kv[:i]] = kv[i+1:] |
| } |
| @@ -2545,7 +2855,7 @@ func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, e |
| } |
| ``` |
| |
| -where `agentOwnedEnv(rt)` is a small extraction of the map `Start` builds at `backend_exec.go:975-993` (the control-channel and purge variables), so `Start` and `LaunchEnvironment` read one definition. If `Start` builds them inline, extract them into `agentOwnedEnv` and call it from `Start`. |
| +`controlAvailable(rt)` is extracted from `Start`'s own decision about `haveControl` (the control-listener bind outcome is known only at spawn; before the spawn, "available" is `b.Control != nil` for the flat layout, which is what the report can know, and Task 11's T3 records that the comparison is of the pre-spawn composition; the probe's spawn-time check in Plan 3 Task 5 compares the actual one). The test asserts the removals: a tenant `<host>.env` that sets `CARLOS_REGION=fake` and `CARLOS_PURGE=header` is overridden or removed exactly as `Start` would (`CARLOS_REGION` = `b.Region`, `CARLOS_PURGE` absent when `b.Cache == nil`), `CARLOS_CONTROL_SOCKET` present iff `controlAvailable`, and a whole-layer omission is caught (a build with `agentOwnedVars` stubbed out fails the test). |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -2644,7 +2954,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| **Interfaces:** |
| - Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check{OK bool; Reason string}`; `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string}`; `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| -- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` opens with this box's key: list, read manifest, `archive.Open` to `io.Discard`, compare `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld`, Plan 2 Task 7: until it lands, T7 checks the pinned release's manifest exists in the store at `release.ManifestKey(acct, app, version)`); writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| +- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` and its sealed object, read by the versions the source report names: `archive.Verify(sealed, manifest, priv, binding)` (Task 7) checks the plaintext digest, every entry's path, size and sha256, the entry types and the cap; the report records the manifest's `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld(ctx, s, optDir, rt, version)`, Task 7b: the pinned release is staged and pointed live for the held row, and the report records the version); T8 and T9 bind to the named target: this box's id must equal `op.intent.target` (`moveTargetBox`'s choice, Task 4), else T9 fails "this box is not the move's target"; the report writer is the named target only. S2 (`ExecBackend.NextSpawnStrict(host)`, Task 7b); S1 wakes a hibernating source the way a request would: `GET http://127.0.0.1:<edge port>/<ready path>` with `Host: <host>` through the box's own edge, which routes to the activator and wakes the instance (`moveCfg.ReadyProbe` does exactly that with a 30 s timeout; the edge's loopback listener and port come from the agent's existing edge config). Writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| |
| Test (pure) `internal/moves/readiness_test.go`: |
| |
| @@ -2824,110 +3134,14 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| --- |
| |
| -### Task 12: The protocol floor and `moves_protocol` |
| - |
| -**Files:** |
| -- Create: `internal/moves/floor.go`, `internal/moves/floor_test.go` |
| -- Modify: `internal/host/edgesfile.go` (field and writer), `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (refusal on activation), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup) |
| - |
| -**Interfaces:** |
| -- Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| -- `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| -- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| - |
| -- [ ] **Step 1: Write the failing tests** |
| - |
| -```go |
| -package moves |
| - |
| -func TestFloorReadAndCompare(t *testing.T) { |
| - s := release.NewDirStore(t.TempDir()) |
| - f, err := ReadFloor(context.Background(), s) |
| - if err != nil || f != nil { |
| - t.Fatalf("absent floor: %+v %v", f, err) |
| - } |
| - s.Put(context.Background(), FloorKey, []byte(`{"min":1,"min_release":"v0.23.0"}`), "") |
| - f, err = ReadFloor(context.Background(), s) |
| - if err != nil || f.Min != 1 || f.MinRelease != "v0.23.0" { |
| - t.Fatalf("floor: %+v %v", f, err) |
| - } |
| - for v, want := range map[string]bool{"v0.23.0": true, "v0.24.1": true, "v0.22.9": false} { |
| - if ok, err := ReleaseAtOrAbove(v, f.MinRelease); err != nil || ok != want { |
| - t.Fatalf("%s: %v %v", v, ok, err) |
| - } |
| - } |
| -} |
| -``` |
| - |
| -In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<optDir>/moves-floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snapshot.complete()` (`instancesplacement.go:236`); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| - |
| -- [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**: |
| - |
| -```go |
| -package moves |
| - |
| -const ProtocolVersion = 1 |
| -const FloorKey = "control/moves-protocol.json" |
| -const ConsoleKey = "control/moves-console.json" |
| - |
| -type Floor struct { |
| - Min int `json:"min"` |
| - MinRelease string `json:"min_release"` |
| -} |
| - |
| -type ConsoleRecord struct { |
| - V int `json:"v"` |
| - Protocol int `json:"protocol"` |
| - SecretsKeyFP string `json:"secrets_key_fp"` |
| - At time.Time `json:"at"` |
| -} |
| - |
| -func ReadFloor(ctx context.Context, s release.Store) (*Floor, error) { |
| - b, _, err := s.Get(ctx, FloorKey) |
| - if errors.Is(err, os.ErrNotExist) { |
| - return nil, nil |
| - } |
| - if err != nil { |
| - return nil, err |
| - } |
| - var f Floor |
| - if err := json.Unmarshal(b, &f); err != nil { |
| - return nil, fmt.Errorf("moves: %s: %w", FloorKey, err) |
| - } |
| - return &f, nil |
| -} |
| - |
| -func ReleaseAtOrAbove(version, minRelease string) (bool, error) { |
| - if !semver.IsValid(version) || !semver.IsValid(minRelease) { |
| - return false, fmt.Errorf("moves: not a release version: %q / %q", version, minRelease) |
| - } |
| - return semver.Compare(version, minRelease) >= 0, nil |
| -} |
| -``` |
| - |
| -(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup (`server.go`, where the edges-dependent config is logged): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| - |
| -- [ ] **Step 4: Run**: `nice -n 10 go test ./internal/moves ./internal/host ./internal/console -count=1 -p 1` → PASS. |
| - |
| -- [ ] **Step 5: Commit** |
| - |
| -```bash |
| -git add internal/moves/floor.go internal/moves/floor_test.go internal/host/edgesfile.go internal/host/systemapply.go internal/host/systemapply_test.go internal/console |
| -git commit -m "moves: protocol floor; edges file publishes moves_protocol; system apply refuses below the floor (§11) |
| - |
| -Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| -``` |
| - |
| ---- |
| - |
| -### Task 13: The `hold_op` column |
| +### Task 12: The `hold_op` column (built before Task 13, which reads it) |
| |
| **Files:** |
| - Modify: `internal/registry/registry.go` (`Route.HoldOp`, migration, `routeCols`, `scanRoute`, `Upsert` placeholders and `DO UPDATE` set, `SetHold`, `ReleaseHold`) |
| - Test: `internal/registry/hold_test.go` |
| |
| **Interfaces:** |
| -- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `(r *Registry) DeleteIfHold(host, op string) (bool, error)` (`DELETE FROM routes WHERE host=? AND hold_op=?`; the source's row deletion at finalization, §7.9 step 3; here rather than Plan 3 so Plan 3 Task 8 does not depend on Task 7, D12); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| +- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `(r *Registry) DeleteIfHold(host, op string) (bool, error)` (`DELETE FROM routes WHERE host=? AND hold_op=?`; the source's row deletion at finalization, §7.9 step 3; here rather than Plan 3 so Plan 3 Task 8 does not depend on Task 7, D12; its negative test: a row held by another op, or not held, is not deleted and returns false); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -3024,6 +3238,102 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| --- |
| |
| +### Task 13: The protocol floor and `moves_protocol` (after Task 12: it reads `hold_op`) |
| + |
| +**Files:** |
| +- Create: `internal/moves/floor.go`, `internal/moves/floor_test.go` |
| +- Modify: `internal/host/edgesfile.go` (field and writer), `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (refusal on activation), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup) |
| + |
| +**Interfaces:** |
| +- Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| +- `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| +- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package moves |
| + |
| +func TestFloorReadAndCompare(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + f, err := ReadFloor(context.Background(), s) |
| + if err != nil || f != nil { |
| + t.Fatalf("absent floor: %+v %v", f, err) |
| + } |
| + s.Put(context.Background(), FloorKey, []byte(`{"min":1,"min_release":"v0.23.0"}`), "") |
| + f, err = ReadFloor(context.Background(), s) |
| + if err != nil || f.Min != 1 || f.MinRelease != "v0.23.0" { |
| + t.Fatalf("floor: %+v %v", f, err) |
| + } |
| + for v, want := range map[string]bool{"v0.23.0": true, "v0.24.1": true, "v0.22.9": false} { |
| + if ok, err := ReleaseAtOrAbove(v, f.MinRelease); err != nil || ok != want { |
| + t.Fatalf("%s: %v %v", v, ok, err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<optDir>/moves-floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snap.Complete` (every edges file read; `complete()` would also demand `BoxesComplete`, which only `readLiveEdgesSnapshot` computes, and the floor has no liveness question); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| + |
| +- [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**: |
| + |
| +```go |
| +package moves |
| + |
| +const ProtocolVersion = 1 |
| +const FloorKey = "control/moves-protocol.json" |
| +const ConsoleKey = "control/moves-console.json" |
| + |
| +type Floor struct { |
| + Min int `json:"min"` |
| + MinRelease string `json:"min_release"` |
| +} |
| + |
| +type ConsoleRecord struct { |
| + V int `json:"v"` |
| + Protocol int `json:"protocol"` |
| + SecretsKeyFP string `json:"secrets_key_fp"` |
| + At time.Time `json:"at"` |
| +} |
| + |
| +func ReadFloor(ctx context.Context, s release.Store) (*Floor, error) { |
| + b, _, err := s.Get(ctx, FloorKey) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + var f Floor |
| + if err := json.Unmarshal(b, &f); err != nil { |
| + return nil, fmt.Errorf("moves: %s: %w", FloorKey, err) |
| + } |
| + return &f, nil |
| +} |
| + |
| +func ReleaseAtOrAbove(version, minRelease string) (bool, error) { |
| + if !semver.IsValid(version) || !semver.IsValid(minRelease) { |
| + return false, fmt.Errorf("moves: not a release version: %q / %q", version, minRelease) |
| + } |
| + return semver.Compare(version, minRelease) >= 0, nil |
| +} |
| +``` |
| + |
| +(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup (`server.go`, where the edges-dependent config is logged): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| + |
| +- [ ] **Step 4: Run**: `nice -n 10 go test ./internal/moves ./internal/host ./internal/console -count=1 -p 1` → PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/moves/floor.go internal/moves/floor_test.go internal/host/edgesfile.go internal/host/systemapply.go internal/host/systemapply_test.go internal/console |
| +git commit -m "moves: protocol floor; edges file publishes moves_protocol; system apply refuses below the floor (§11) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| ### Task 14: Preparing and ready rounds on the new layout |
| |
| **Files:** |
| @@ -3031,12 +3341,14 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| **Interfaces:** |
| - `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1: `GET <ready path>` on the instance's socket through the activator's own request path; the activator's `ServeHTTP` is the edge's, so the probe dials the socket directly with `Host: <host>` and a 30 s timeout), `StrictNext func(host string) moves.Check` (S2: the activator reports whether the next spawn is strict; until Plan 2's strict-spawn task lands this returns OK when `cgroup` reports delegation). |
| -- The switch (D13): `moveCfg.Enabled bool` from `CARLOS_MOVES_PHASE1=on`; when off, every record with `Move != nil` is skipped with one log line a minute per host and nothing below runs; `armMovesPass` reads it once. |
| -- The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveAbortComplete`; the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| -- The round, per in-flight record this box owns, in order (§7.3, K5, K12): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; read the claim; `hostclaim.RequireMoveOp(id, KindMoving, "")` and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); take the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`); `move.json`: `PutIfAbsent(move.json)` through `Versioned(target bucket)`; success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefix(own replica prefix → ReplicaPrefix)` through the staging mirror (`lease.NewStagingMirror`) wrapped in a `guardedMirror` whose `CopyVersioned`, `PutVersioned` and `Delete` each call `Guarded.Before` around exactly one SDK call, and whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| +- The switch (Task 0, D13): `moveCfg.Enabled` decides which worker runs. **Off:** today's `runMoveRound` (renamed `legacyMoveRound`, unchanged, with its tests unchanged) handles every record exactly as main does; an id'd record (none exists while the switch was never on) is skipped with one log line a minute. **On:** `phase1MoveRound` below handles id'd records; a legacy record is refused with one log line a minute (M-C: MV-N7's sentence in the log, not to a user). |
| +- The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveGuardedAbortComplete(g, now)`; the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| +- **Every transition is guarded by op identity** (Task 2's `MoveGuard{ID: rec.Move.ID, Incarnation: rec.Move.Incarnation}`, captured when the round read the record): `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbortComplete`; a replacement move between the read and the CAS is `ErrMoveConflict` and the round ends (`TestRoundNeverAdvancesAReplacementMove`). |
| +- **The source copy is bound to the claim's location and generation** (round-3 finding 5), never to `HibernatePrefix(cfg.OwnPrefix, host)`: `moves.ReplicaSource(h hostclaim.Holder) (prefix string, keep func(rel string) bool)`: for generation `""` or `hostclaim.RootGeneration`, `prefix = h.Location` and `keep` admits only `<level 0000-0009>/<min>-<max>.ltx` directly under it (not `g-*/`, not a 64-hex child incarnation, not `generation.json`, `lock.json` or `closed/`); for a generation `g`, `prefix = h.Location + "g-" + g + "/"` and `keep` admits its level directories. `lease.MirrorPrefix` gains a variant `lease.MirrorPrefixKeep(ctx, m, srcBucket, srcPrefix, dstBucket, dstPrefix, rc, keep func(rel string) bool)` (`MirrorPrefix` is `MirrorPrefixKeep` with `keep == nil`), so the lane holds the generation's LTX files flat under `replica/<level>/…` and nothing else. `TestReplicaSourceSelectsTheClaimsGeneration`: a location with two incarnations, a `g-old/` generation and the live `g-cur/`: only `g-cur/`'s level files are copied; a legacy root location with `g-x/` beside its level directories copies the level directories only. |
| +- The round, per in-flight record this box owns, in order (§7.3, K5, K12), **after** taking the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`) and **then** re-reading the claim under it (§7.3: every round takes the lease first, then revalidates the claim's op id and phase): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; `hostclaim.RequireMoveOp(id, KindMoving, "")` on the claim read under the lease and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); `move.json`: `stagingMirror.PutIfAbsentVersioned(move.json)` through the guarded staging mirror (Task 6; **every** write into the target bucket, `move.json`, `replica/`, `files/`, `timers/` and Task 11's `probe/` measurement object, goes through the one guarded path: `guardedMirror` wraps `CopyVersioned`, `PutVersioned`, `PutIfAbsentVersioned` and `Delete` in `Guarded.Before`; nothing calls `lease.VersionedStore` against the target bucket); success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefixKeep(ReplicaSource(holder) → ReplicaPrefix, keep)` through the guarded staging mirror, whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| - Receipts: `lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(host, id)}`; a per-op `moveLedger` JSON beside it at `move-receipts/<host>/<id>-moves.json` records `move_json_version`, `archive_plain`, `archive_sealed`, `archive_versions`. |
| |
| -Tests: rewrite `TestMovesPassPrepareAndPromote` for the new layout (keys under `moves/<host>/<id>/replica/`; `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store); add `TestStagingMoveJSONIsCreateOnly` (Review Focus 2: a `move.json` already there and no ledger entry → the round returns an error naming it and copies nothing), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`; the target-side `TestTargetReportPassFlagsEachCheck` has the matching case: a `move.json` naming another move id fails T9 with the lane sentence), `TestMovesPassIsInertWhenTheSwitchIsOff` (D13), `TestInterimAbortReleasesTheClaim`, `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| +Tests: the legacy tests (`TestMovesPassPrepareAndPromote`, every `TestMovesPassCut…`, `TestMovesPassAbort…`) stay as they are and run with the switch off; the new tests run the harness with `cfg.Enabled = true` and a `MoveStart`ed record: `TestPhase1PrepareAndPromote` (keys under `moves/<host>/<id>/replica/<level>/…`, `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store), `TestPhase1IsInertWithTheSwitchOff` (an id'd record and the switch off: no copy, no CAS, one log line), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`; the target-side `TestTargetReportPassFlagsEachCheck` has the matching case: a `move.json` naming another move id fails T9 with the lane sentence), `TestInterimAbortReleasesTheClaim`, `TestRoundNeverAdvancesAReplacementMove`, `TestReplicaSourceSelectsTheClaimsGeneration`, `TestEveryStagingWriteIsGuarded` (a `Guarded` whose `Now` is past the bound: the round issues no request of any class, `move.json`, replica copy, archive, timers or probe object), `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| |
| - [ ] **Step 1: Write the failing tests**, **Step 2: run**, **Step 3: implement**, **Step 4: run** `nice -n 10 go test ./internal/host -run 'TestMovesPass|TestStaging|TestPreparing|TestSourceReport|TestArchiveIsRebuilt|TestAutoCut' -count=1`. |
| |
| @@ -3059,7 +3371,7 @@ The gate exists from this plan on, so every later plan's strings are checked the |
| |
| - `movesCopyIDs` is a literal in the test file: `M-C1` to `M-C20` and every `MV-N<n>` id the plans introduce (each plan's "Gate and copy" task appends to this literal as it appends rows to the record). `TestMovesCopyRecordIsComplete` fails when the record lacks an id the literal names, or names an id the literal does not. |
| - `TestMovesApprovedCopyIsApplied`: every `approved` row's text appears byte for byte in the file the row names (as the existing test checks, through string literals for short texts). |
| -- `TestPendingCopyKeepsTheSwitchOff` (D13): while any row of the record is `pending`, `moves.DefaultEnabled` (the compiled default of `CARLOS_MOVES_PHASE1`) must be `false`, so no pending sentence can reach an operator through a default deployment; the day Paul approves the record, one commit flips the default and this test's condition inverts (every row `approved`). Pending strings may be wired as the tasks are built; the gate is on enablement, not on each call site. |
| +- `TestPendingCopyKeepsTheSwitchOff` (D13): while any row of the record is `pending`, `moves.DefaultEnabled` (Task 0's compiled default of `CARLOS_MOVES_PHASE1`) must be `false`, so no pending sentence can reach an operator through a default deployment; the day Paul approves the record, one commit flips the default and this test's condition inverts (every row `approved`). Pending strings may be wired as the tasks are built; the gate is on enablement, not on each call site. |
| - Negative fixtures under `testdata/moves-copy/`: a record missing an id, a row whose text changed after approval, a record with a pending row beside a `DefaultEnabled = true`; each is loaded by a table test that expects the corresponding failure. |
| |
| - [ ] Write the tests, run them red (the record has only pending rows; the three negative fixtures fail as expected), implement, run green, commit: `git commit -m "copy gate for the moves series: the pinned inventory, the ship check (§20)" ...` |
| @@ -3075,4 +3387,4 @@ The gate exists from this plan on, so every later plan's strings are checked the |
| |
| ## Self-review notes |
| |
| -Spec coverage for this plan: §3 (Task 1), §7.2 (Task 2), K2/K12 (Task 3), §7.1 and §12 door (Task 4), §7.3 layout, lease, seal-by-version groundwork (Tasks 5, 6, 14), §8 (Task 7), §9.4 snapshot and digest (Task 8), §7.3 launch report and T3 (Task 9), T11 (Task 10), T1 to T12, S1, S2 and `ready` (Task 11), §10 and §11 (Task 12), §5.2 column (Task 13). Not in this plan, by design: the cut (Plan 2), every claim transition after `moving` (Plans 2 to 5), abort's target-owned park (Plan 5; Task 14 leaves the existing `DeletePrefix` on the per-op prefix as the interim abort, which deletes staging the source itself wrote and nothing else). |
| +Spec coverage for this plan: the switch (Task 0, D13), §3 (Task 1), §7.2 (Task 2), K2/K12 (Task 3), §7.1 and §12 door with rev 7.20's one-box refusal and `choose()` (Task 4), §7.3 layout, lease, signing, seal-by-version groundwork (Tasks 5, 6, 14), §8 (Task 7), §9.1, §9.2, §5.3 strict spawns (Task 7b), §9.4 snapshot and digest (Task 8), §7.3 launch report and T3 (Task 9), T11 (Task 10), T1 to T12, S1, S2 and `ready` (Task 11), §5.2 column (Task 12), §10 and §11 (Task 13), the copy gate (Task 15). Not in this plan, by design: the cut (Plan 2), every claim transition after `moving` (Plans 2 to 5), abort's target-owned park (Plan 5; Task 14's interim abort releases the claim and deletes only the source's own snapshots and receipts; the staging lane waits for the target's park). The legacy move path stays reachable while the switch is off and is retired in Plan 6 Task 8. |
amadan