| index dd493cf1a..3eb1d4e89 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| @@ -4,11 +4,13 @@ |
| |
| **Goal:** Give a move an identity, a journal, a claim, a per-operation staging layout whose copies can be sealed by version, the files archive, the timers snapshot, and the readiness checks that decide `ready`, with nothing yet cutting over. |
| |
| -**Architecture:** A new pure package `internal/moves` holds the protocol's data shapes and arithmetic (ids, seal, archive, timers digest, env report, budget, readiness, floor). `provision.MoveState` grows the record journal of spec §7.2 and `provision.MoveStart` replaces `MoveRequest`. `internal/hostclaim` gains `BeginMove` and `Touch` (K2, K12). `internal/lease` learns to record copy version ids. `internal/host/movespass.go` is extended: preparing rounds mirror into `moves/<host>/<id>/replica/` under the staging-work lease, publish the archive and the launch report, and promote to `ready` only on a fresh passing target report and source readiness. The console's request door becomes operator-only and CASes the claim before it writes the record. |
| +**Architecture:** A new pure package `internal/moves` holds the protocol's data shapes and arithmetic (ids, seal, archive, timers digest, env report, budget, readiness, floor). `provision.MoveState` grows the record journal of spec §7.2 and `provision.MoveStart` joins `MoveRequest` (the legacy transitions stay while the switch is off, index D13). `internal/hostclaim` gains `BeginMove` and `Touch` (K2, K12). `internal/lease` learns to record copy version ids. `internal/host/movespass.go` is extended: preparing rounds mirror into `moves/<host>/<id>/replica/` under the staging-work lease, publish the archive and the launch report, and promote to `ready` only on a fresh passing target report and source readiness. The console's request door becomes operator-only and CASes the claim before it writes the record. |
| |
| **Tech Stack:** Go 1.25, modernc sqlite, litestream v0.5.14 and superfly/ltx v0.5.1 (pinned), AWS SDK v2 S3, `github.com/klauspost/compress/zstd` (new, pure Go), the repository's `release.Store`, `lease`, `hostclaim`, `genselect`, `registry`, `provision`, `appconfig` packages. |
| |
| -**Spec:** `docs/superpowers/specs/2026-10-02-move-any-app-design.md` rev 7.18: §3, §4, §6.1, §6.2, §7.1, §7.2, §7.3, §8, §10, §11, §12 (door), §19.2 K2, K5, K12. Index: `2026-10-06-moves-phase1-0-index.md`. |
| +**Spec:** `docs/superpowers/specs/2026-10-02-move-any-app-design.md` rev 7.21: §3, §4, §6.1, §6.2, §7.1, §7.2, §7.3, §8, §10, §11, §12 (door), §19.2 K2, K5, K12. Index: `2026-10-06-moves-phase1-0-index.md`. |
| + |
| +**Plan revision:** rev 5 (after the round-4 Plan 1 review, `CODEX-PLAN1-REVIEW-R4.md`; dispositions in the index). Every API this plan names was checked against `origin/main` at 1c43c2ce: `boxchoice.go`'s signatures and the absence of a production caller, `Claim.Validate`'s location grammar, the two system units' direct `carlos.prev` exec, the edge unit's `ProtectSystem=strict`, `edge.go`'s cache-before-activator order, `agentOwnedVars` and `startChild`'s unconditional `ListenControl`, `LoadMergedHost`'s single etag, `objstore.GetIn`, `leasetest`'s create-only `PUT`, `cgroup.Prepared`'s method set and `s3Mirror`'s `Copy`. |
| |
| ## Global Constraints |
| |
| @@ -64,7 +66,7 @@ |
| |
| **Files:** |
| - Create: `internal/moves/switch.go`, `internal/moves/switch_test.go` |
| -- Modify: `internal/console/server.go` (`Config.MovesEnabled bool`, set where the console reads its environment: the same place `HomeRegion` and `SecretsPub` are read), `internal/host/movespass.go` (`moveCfg.Enabled`, set in `armMovesPass` from `moves.Enabled(os.Getenv)`) |
| +- Modify: `internal/console/server.go` (`Config.MovesEnabled bool`), `cmd/console/main.go` (the `console.NewServer(console.Config{...})` literal at `main.go:283`, where `CONSOLE_ORIGIN` and `CARLOS_RELEASE_PUBKEY` are read: that is where the console reads its environment, not `server.go`), `internal/host/movespass.go` (`moveCfg.Enabled`, set in `armMovesPass` from `moves.Enabled(os.Getenv)`) |
| |
| **Interfaces:** |
| - Produces: `moves.SwitchEnv = "CARLOS_MOVES_PHASE1"`; `moves.DefaultEnabled = false` (a compiled constant the copy gate reads, Task 15); `moves.Enabled(getenv func(string) string) bool` (`"on"`, `"1"`, `"true"` → true; `""` → `DefaultEnabled`; anything else → false with no panic); `console.Config.MovesEnabled`; `moveCfg.Enabled`. |
| @@ -129,7 +131,7 @@ func Enabled(getenv func(string) string) bool { |
| } |
| ``` |
| |
| -In `server.go`, `Config.MovesEnabled bool` with the doc "the Phase 1 moves switch (moves.Enabled)"; where the console builds its `Config` from the environment, `MovesEnabled: moves.Enabled(os.Getenv)`. In `armMovesPass`, `cfg.Enabled = moves.Enabled(os.Getenv)`; `movesPass` reads it (Task 14 uses it). Console tests set `fx.srv.cfg.MovesEnabled = true` where they exercise the new door (Task 4). |
| +In `server.go`, `Config.MovesEnabled bool` with the doc "the Phase 1 moves switch (moves.Enabled)"; in `cmd/console/main.go:283`'s `console.Config{...}` literal, `MovesEnabled: moves.Enabled(os.Getenv)`. The unit test covers the parser; the wiring is one field in a literal `cmd/console` has no test seam for, so it is checked by `go vet ./cmd/console` here and by Plan 7's rehearsal P-1, which turns the switch on per deployment and sees the new door answer (an unwired field would leave the legacy door answering, which P-1's first request detects). In `armMovesPass`, `cfg.Enabled = moves.Enabled(os.Getenv)`; `movesPass` reads it (Task 14 uses it). Console tests set `fx.srv.cfg.MovesEnabled = true` where they exercise the new door (Task 4). |
| |
| - [ ] **Step 4: Run** → PASS; `go build ./...`. |
| - [ ] **Step 5: Commit**: `git commit -m "moves: the Phase 1 switch, off by default (index D13)" ...` |
| @@ -139,11 +141,11 @@ In `server.go`, `Config.MovesEnabled bool` with the doc "the Phase 1 moves switc |
| ### Task 1: Operation ids |
| |
| **Files:** |
| -- Create: `internal/moves/ids.go` |
| +- Create: `internal/moves/ids.go`, `internal/moves/check.go` |
| - Test: `internal/moves/ids_test.go` |
| |
| **Interfaces:** |
| -- Produces: `moves.NewMoveID(r io.Reader) (string, error)`, `moves.NewRollbackID(r io.Reader) (string, error)`, `moves.ValidOpID(id string) bool`, `moves.OpIDKind(id string) string` ("move" | "rollback" | ""). |
| +- Produces: `moves.NewMoveID(r io.Reader) (string, error)`, `moves.NewRollbackID(r io.Reader) (string, error)`, `moves.ValidOpID(id string) bool`, `moves.OpIDKind(id string) string` ("move" | "rollback" | ""); and, in `check.go`, `moves.Check{OK bool; Reason string}`, the shared result type Tasks 7b, 11 and 14 consume. It lives here, in the first task of the package, so Task 7b's `NextSpawnStrict(host) moves.Check` compiles before Task 11 exists (round-4 finding 2); `TestCheckZeroValueFails` pins that a zero `Check` is not OK. |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -261,7 +263,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| **Files:** |
| - Modify: `internal/provision/provision.go:900-1070` (`MoveState`, states, transitions), `internal/provision/provision.go:226-260` (`Manifest`) |
| -- Modify: `internal/console/apiinstancemove.go:233` (the one `MoveRequest` caller), `internal/console/apiinstancemove_test.go`, `internal/host/movespass_test.go` (every `MoveRequest` caller becomes `MoveStart`) |
| +- Modify: `internal/console/apiinstancemove_test.go`, `internal/host/movespass_test.go` (new tests only, written with `MoveStart`; every existing `MoveRequest` caller, `apiinstancemove.go:233` included, stays exactly as it is: index D13) |
| - Test: `internal/provision/move_test.go` |
| |
| **Interfaces:** |
| @@ -751,9 +753,9 @@ Expected: PASS |
| |
| ```bash |
| git add internal/provision internal/console internal/host |
| -git commit -m "provision: the move journal of spec §7.2; MoveStart replaces MoveRequest |
| +git commit -m "provision: the move journal of spec §7.2; MoveStart beside MoveRequest, guarded transitions beside the legacy ones |
| |
| -Deletes the staging-wins cut tests; Plan 2 rewrites the cut. |
| +The legacy transitions, their tests and the staging-wins cut tests are untouched (index D13). |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ``` |
| @@ -782,17 +784,34 @@ import ( |
| ) |
| |
| // movingTestClaim is this file's fixture (delete_test.go already defines |
| -// servingClaim(t, ctx, s) with another shape). |
| -func movingTestClaim() *Claim { |
| - return &Claim{V: 1, Host: "a.go.example.com", Gen: 3, |
| - Holder: &Holder{Family: "instance", Key: "k", Account: "bab", App: "app", Incarnation: "ab", Location: "instances/a.go.example.com/ab/", Stamp: 3}, |
| +// servingClaim(t, ctx, s) with another shape). It is a VALID claim: the |
| +// holder's key, incarnation and location follow the grammar Validate |
| +// enforces (claim.go:236-247: the key is this host's, the incarnation is |
| +// Incarnation(account, app, host, createdAt), the location is exactly |
| +// "instances/<host>/<inc>" with no trailing slash), built with the same |
| +// helpers delete_test.go's servingClaim uses, and the fixture asserts |
| +// Validate() so it can never drift from the real grammar (round-4 |
| +// findings 4 and 9). |
| +func movingTestClaim(t *testing.T) *Claim { |
| + t.Helper() |
| + created := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC) |
| + inc, err := Incarnation("bab", "app", "a.go.example.com", created) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + c := &Claim{V: 1, Host: "a.go.example.com", Gen: 3, |
| + Holder: &Holder{Family: "instance", Key: testHolderKey("bab", "app", "a.go.example.com"), Account: "bab", App: "app", Incarnation: inc, CreatedAt: created, Location: "instances/a.go.example.com/" + inc, Stamp: 3}, |
| Op: Op{Kind: KindServing, Exec: "ie-box"}, |
| Touched: Touched{Complete: true, Regions: []string{"ie"}, Boxes: []string{"ie-box"}}} |
| + if err := c.Validate(); err != nil { |
| + t.Fatalf("fixture must validate: %v", err) |
| + } |
| + return c |
| } |
| |
| func TestBeginMoveFromServingOnly(t *testing.T) { |
| now := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| - c := movingTestClaim() |
| + c := movingTestClaim(t) |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", now)(c); err != nil { |
| t.Fatal(err) |
| } |
| @@ -809,12 +828,12 @@ func TestBeginMoveFromServingOnly(t *testing.T) { |
| t.Fatalf("second move: %v", err) |
| } |
| // exec must be the source. |
| - d := movingTestClaim() |
| + d := movingTestClaim(t) |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "us-box", "uk-box", "uk", now)(d); !errors.Is(err, ErrWrongState) { |
| t.Fatalf("source not exec: %v", err) |
| } |
| // A pending chore in the target region refuses (CC11). |
| - e := movingTestClaim() |
| + e := movingTestClaim(t) |
| e.Obligations = []Obligation{{ID: "o1", Op: "x", Kind: "park-replica", Region: "uk", Retirement: "r", Owner: "region:uk"}} |
| if err := BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", now)(e); !errors.Is(err, ErrPendingChore) { |
| t.Fatalf("pending chore: %v", err) |
| @@ -822,7 +841,7 @@ func TestBeginMoveFromServingOnly(t *testing.T) { |
| } |
| |
| func TestTouchAppendsOnceUnderTheOp(t *testing.T) { |
| - c := movingTestClaim() |
| + c := movingTestClaim(t) |
| _ = BeginMove("mv_abcdefghijklmnopqrstuvwxyz", "ops", "ie-box", "uk-box", "uk", time.Now())(c) |
| if err := Touch("mv_abcdefghijklmnopqrstuvwxyz", []string{"uk"}, nil)(c); err != nil { |
| t.Fatal(err) |
| @@ -1017,7 +1036,7 @@ func TestMoveDoorRefusesWithoutClaimWrite(t *testing.T) { |
| t.Fatal(err) |
| } |
| inc, _ := hostclaim.Incarnation("bab", "titogo", "acme.go.tito.io", rec.CreatedAt) |
| - writeServingClaim(t, fx.rel, "acme.go.tito.io", inc, rec.CreatedAt, "ie-box", "ie") // test helper: hostclaim.Update on an absent Observed writing a serving claim with Holder{Family:"instance", Key: provision.RecordKey(...), Account, App, Incarnation, CreatedAt, Location: "instances/<host>/<inc>/", Stamp: 1}, Op{Kind: KindServing, Exec: box}, Touched{Complete: true, Regions: []string{region}, Boxes: []string{box}} |
| + writeServingClaim(t, fx.rel, "acme.go.tito.io", inc, rec.CreatedAt, "ie-box", "ie") // test helper: hostclaim.Update on an absent Observed writing a serving claim with Holder{Family:"instance", Key: provision.RecordKey(...), Account, App, Incarnation, CreatedAt, Location: "instances/<host>/<inc>", Stamp: 1}, Op{Kind: KindServing, Exec: box}, Touched{Complete: true, Regions: []string{region}, Boxes: []string{box}}; no trailing slash (Claim.Validate, claim.go:245), and the helper fails the test on any error from Update, which validates what it writes (store.go:98), so an invalid fixture dies here and not in the door |
| before, _, _ := fx.rel.Get(context.Background(), claimKey) |
| // Current region → refused; outside regions → refused; neither writes. |
| for _, body := range []string{`{"region":"ie"}`, `{"region":"se"}`} { |
| @@ -1139,11 +1158,25 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| jsonError(w, http.StatusConflict, "the instance is already in "+s.homeRegionOut(to)) |
| return |
| } |
| - if m, err := provision.GetManifest(ctx, s.cfg.Releases, acct.Sqid, app); err == nil && len(m.Regions) > 0 && !slices.Contains(m.Regions, to) { |
| + // Both policy reads fail CLOSED (round-4 finding 13): an unreadable |
| + // manifest or store record is a 503, never a pass. |
| + m, err := provision.GetManifest(ctx, s.cfg.Releases, acct.Sqid, app) |
| + if err != nil && !errors.Is(err, os.ErrNotExist) { |
| + jsonError(w, http.StatusServiceUnavailable, "could not read the app's instances manifest. try again") |
| + return |
| + } |
| + if m != nil && len(m.Regions) > 0 && !slices.Contains(m.Regions, to) { |
| jsonError(w, http.StatusBadRequest, "region "+s.homeRegionOut(to)+" is not one of this app's regions: "+strings.Join(m.Regions, ", ")) |
| return |
| } |
| - if m, err := provision.GetManifest(ctx, s.cfg.Releases, acct.Sqid, app); err == nil && m.PrivateStorage { |
| + // Private storage is the instance's BINDING, not the app's create-time |
| + // policy: the create door stamps a store per instance under its |
| + // environment name, or its slug when it has none (apiinstances.go: |
| + // 1074-1079), and turning the policy off later removes no store. |
| + if store, _, err := objstore.GetIn(ctx, s.cfg.Releases, acct.Sqid, app, instanceStoreEnv(rec)); err != nil { |
| + jsonError(w, http.StatusServiceUnavailable, "could not read this instance's storage binding. try again") |
| + return |
| + } else if store != nil { |
| jsonError(w, http.StatusBadRequest, host+" uses private storage. Instances with private storage cannot move yet.") |
| return |
| } |
| @@ -1156,26 +1189,23 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| jsonError(w, http.StatusBadRequest, "region "+to+" cannot take a move in this deployment. it publishes no staging bucket. Regions that can: "+stagingCodes(caps)) |
| return |
| } |
| - targetBox, floorOK, complete := moveTargetBox(ctx, s.cfg.Releases, to, rec.Region) |
| - if !complete { |
| + sel, err := s.selectMoveBoxes(ctx, to, rec.Region, app) |
| + if err != nil { |
| jsonError(w, http.StatusConflict, "the fleet's edges files could not all be read. try again") |
| return |
| } |
| // Phase 1 is for one-box regions (spec rev 7.20 §7.1, index D15): a |
| // region with more than one eligible box, on either side, is refused. |
| - if n := len(eligibleBoxes(snapshotFor(ctx, s.cfg.Releases), to, rec.Pool)); n > 1 { |
| - jsonError(w, http.StatusBadRequest, s.homeRegionOut(to)+" has more than one box. Moves between regions with several boxes are not supported yet.") |
| - return |
| - } |
| - if n := len(eligibleBoxes(snapshotFor(ctx, s.cfg.Releases), rec.Region, rec.Pool)); n > 1 { |
| - jsonError(w, http.StatusBadRequest, s.homeRegionOut(rec.Region)+" has more than one box. Moves between regions with several boxes are not supported yet.") |
| + if sel.Many != "" { |
| + jsonError(w, http.StatusBadRequest, s.homeRegionOut(sel.Many)+" has more than one box. Moves between regions with several boxes are not supported yet.") |
| return |
| } |
| - if targetBox == "" { |
| + if sel.Target == "" { |
| jsonError(w, http.StatusBadRequest, "no box in "+to+" publishes an edges file yet") |
| return |
| } |
| - if !floorOK { |
| + targetBox := sel.Target |
| + if !sel.FloorOK { |
| jsonError(w, http.StatusConflict, "a box in "+rec.Region+" or "+to+" is below the moves protocol floor. roll the platform first") |
| return |
| } |
| @@ -1220,51 +1250,93 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| s.moveTransition(w, r, acct, app, host, provision.MoveStartTransition(provision.MoveStart{ID: id, Incarnation: inc, From: rec.Region, To: to, Cut: cut, By: addr, Now: now}), http.StatusAccepted) |
| } |
| |
| -// moveTargetBox is the lexically-first box publishing placement to (the |
| -// same rule as moveTargets in internal/host), and whether every box in |
| -// either region is at or above the protocol floor (§11, K10). An |
| -// incomplete snapshot answers neither: a box whose edges file could not |
| -// be read may be below the floor, so the door refuses (complete == false). |
| -func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box string, floorOK, complete bool) { |
| - snap, err := readEdgesSnapshot(ctx, s) |
| - // snap.Complete: every edges file listed was read and decoded. That is |
| - // the evidence the floor needs. snap.complete() also wants |
| - // BoxesComplete, which only readLiveEdgesSnapshot computes (liveness, |
| - // instancesplacement.go:103-130), and a floor decision has no liveness |
| - // question, so it is not used here. |
| - if err != nil || !snap.Complete { |
| - return "", false, false |
| - } |
| - floorOK, complete = true, true |
| - // The target box is region HA's choose() over the eligible boxes (spec |
| - // rev 7.20 §7.1, L15): main's same-package eligibleBoxes and chooseBox |
| - // (internal/console/boxchoice.go) are the one implementation; the |
| - // create door uses them too. Phase 1 refuses a multi-box region at the |
| - // door (D15), so choose() names the region's one box here, and this |
| - // loop only finds it; TestMoveTargetBoxUsesChoose pins that the call |
| - // is chooseBox's and that chooseBox's tie-break is the approved |
| - // SHA-256 of host + "\x00" + box reference (region HA §5.2), fixing |
| - // chooseBox where it still differs. |
| +// moveSelection is the door's answer about the two regions of a move. |
| +type moveSelection struct { |
| + Target string // the chosen box in `to`; "" when none is eligible |
| + Many string // the region (to or from) with more than one eligible box; "" when neither |
| + FloorOK bool // every edges file of either region publishes moves_protocol >= moves.ProtocolVersion |
| +} |
| + |
| +var errEdgesIncomplete = errors.New("console: the fleet's edges files could not all be read") |
| + |
| +// selectMoveBoxes is the FIRST production caller of boxchoice.go's |
| +// eligibleBoxes and chooseBox (main's tests exercise them; no door on |
| +// main calls them yet, round-4 finding 1), so every input is composed |
| +// here and none is borrowed from an integration that has not landed: |
| +// |
| +// - the snapshot is s.liveEdges(ctx) (readLiveEdgesSnapshot, liveness |
| +// joined, regioncandidates.go:204): eligibleBoxes refuses a snapshot |
| +// whose complete() is false, and complete() wants BoxesComplete, which |
| +// only the live read computes; a plain readEdgesSnapshot would make |
| +// every box ineligible; |
| +// - verified is s.verifiedBoxes(ctx, snap, to, from): for every file of |
| +// the two regions, boxid.Load(ctx, s.cfg.Releases, f.BoxID) then |
| +// boxid.Verifier{Account: s.cfg.AWSAccount, Certs: s.boxCerts}.Verify |
| +// (verify.go:32; certs from boxid.EmbeddedCerts() once at server |
| +// start); a record that fails to load or verify leaves its box out of |
| +// the map, which eligibleBoxes reads as not eligible. s.cfg.AWSAccount |
| +// is new in this task: console.Config.AWSAccount, CARLOS_AWS_ACCOUNT in |
| +// cmd/console/main.go, the deployment's AWS account id (the Verifier's |
| +// trust anchor, boxid_test.go:62 shows the shape); unset means no box |
| +// verifies and the door answers "no box in <to> publishes an edges |
| +// file yet" (Plan 7's roll notes name the variable). If region HA's |
| +// plan 1a lands a field for the same value first, this task reuses |
| +// that name instead of adding a second; |
| +// - awsRegion per code is s.regionCands(snap)'s AWSRegion for that code |
| +// (the majority of the code's signed regions, regioncandidates.go: |
| +// 178); a code with no candidate has no AWS region and no eligible box; |
| +// - homeName is s.cfg.HomeRegion and expectedFP is s.expectedSecretsFP(), |
| +// as regionCands passes them; lost is nil (Phase 1 consumes no loss |
| +// records: contract C6); |
| +// - running is nil, and chooseBox is UNCHANGED: its spread and |
| +// host-count preferences and its tie-break are never reached, because |
| +// a region with two eligible boxes is refused before chooseBox runs |
| +// (D15). Region HA owns the hash tie-break (L15, P5-D2); the one-box |
| +// refusal is what makes that safe here. |
| +// |
| +// The floor (§11, K10) is read from the same snapshot: every file of |
| +// either region must publish moves_protocol >= moves.ProtocolVersion |
| +// (Task 13 adds the field; absent reads as 0, below any floor). An |
| +// incomplete snapshot answers nothing: a box whose file could not be read |
| +// may be below the floor or may be the second box of a region. |
| +func (s *Server) selectMoveBoxes(ctx context.Context, to, from, app string) (moveSelection, error) { |
| + snap, err := s.liveEdges(ctx) |
| + if err != nil || !snap.complete() { |
| + return moveSelection{}, errEdgesIncomplete |
| + } |
| + aws := map[string]string{} |
| + for _, c := range s.regionCands(snap) { |
| + aws[c.Code] = c.AWSRegion |
| + } |
| + verified := s.verifiedBoxes(ctx, snap, to, from) |
| + fp := s.expectedSecretsFP() |
| + sel := moveSelection{FloorOK: true} |
| for _, f := range snap.Files { |
| - if f.Placement != to && f.Placement != from { |
| - continue |
| + if (f.Placement == to || f.Placement == from) && f.MovesProtocol < moves.ProtocolVersion { |
| + sel.FloorOK = false |
| } |
| - if f.MovesProtocol < moves.ProtocolVersion { |
| - floorOK = false |
| + } |
| + for _, code := range []string{to, from} { |
| + cands := eligibleBoxes(snap, verified, code, aws[code], s.cfg.HomeRegion, fp, nil) |
| + if len(cands) > 1 { |
| + sel.Many = code |
| + return sel, nil |
| } |
| - if f.Placement == to && (box == "" || f.Box < box) { |
| - box = f.Box |
| + if code == to { |
| + if c, ok := chooseBox(cands, nil, app); ok { |
| + sel.Target = c.BoxID |
| + } |
| } |
| } |
| - return box, floorOK, complete |
| + return sel, nil |
| } |
| ``` |
| |
| -`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 13 adds the writer's side). Private storage is the instances manifest's `PrivateStorage` (`provision.go:253`, the `private_storage` field), read through the `GetManifest` call the regions check already makes: one read, two refusals. |
| +`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 13 adds the writer's side). Private storage is the instance's store binding (round-4 finding 13): `instanceStoreEnv(rec)` is `rec.Environment`, or `rec.Slug` when it has none, exactly the name the create door stamps (`apiinstances.go:1074-1079`), and `objstore.GetIn` (`objstore.go:236`) answers `nil` for no store; a read error is a 503, never a pass. The manifest's `PrivateStorage` policy is not consulted: it is mutable and create-time (`apiinstances.go:1960-1966`), so it says nothing about an existing instance. Task 14's round makes the same `GetIn` read before its first staging write and fails S1 with M-C17's sentence when a store has appeared since the request (`carlos store create --environment`). Tests: `TestMoveDoorRefusesABoundStoreNotAPolicy` (a record with a store under its slug and the policy off → M-C17 and no claim written; the policy on and no store → admitted; the store read failing → 503 and no claim written; the manifest read failing → 503). `TestSelectMoveBoxesComposesTheCreateDoorsInputs` builds a live snapshot with two verified boxes in `uk` (M-C21 names `uk`), one verified box in `uk` and one unverifiable (its `boxid` record missing: the one box is chosen), a box whose edges file lacks `moves_protocol` (`FloorOK` false), and an incomplete snapshot (`errEdgesIncomplete`); the fixtures are `boxid.Publish`ed records signed with the test certificate `boxid_test.go` uses, so `Verifier.Verify` runs for real. |
| |
| **The legacy door stays** (Task 0): today's bodies of `handleAPIInstanceMoveRequest`, `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` are renamed `legacyHandleAPIInstanceMoveRequest` (and siblings) unchanged, and each new handler's first line is `if !s.cfg.MovesEnabled { s.legacyHandleAPIInstanceMove…(w, r); return }`; `handleAPIInstanceMoveResolve` answers 404 when the switch is off (the route did not exist before). The existing `TestAPIInstanceMoveVerbs` and `TestAPIInstanceMoveClassRails` run as they are with the switch off (no change in behaviour, which is the point); the new tests below set `fx.srv.cfg.MovesEnabled = true`. |
| |
| -**The target box** (spec rev 7.20 §7.1, D15): `moveTargetBox` uses main's same-package `eligibleBoxes(snap edgesSnapshot, verified map[string]boxid.Identity, placement, awsRegion, homeName, expectedFP string, lost map[string]bool) []boxCandidate` and `chooseBox(cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool)` (`internal/console/boxchoice.go:28`, `:76`), the create door's own helpers, with the same `verified`, `expectedFP` (`s.expectedSecretsFP()`) and `running` inputs the create door passes; more than one candidate on either side → M-C21; `chooseBox`'s last tie-break is changed in this task from "the lowest box id" to the approved SHA-256 of `host + "\x00" + box reference` (region HA §5.2, P5-D2), with `TestChooseBoxTieBreakIsTheApprovedHash` beside boxchoice's existing tests, since the create door and the move door must agree. The chosen box goes into `op.intent.target` (`BeginMove`'s `targetBox`), and T8 and T9 (Task 11) bind to it: the reporting box's own id must equal `op.intent.target`, not merely publish the placement. `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` switch to `moveOperatorGate`. In `authcli.go`, the three mutation routes keep `svcrole.Allow(svcrole.Instance)` only so a service token meets the operator 404 rather than the role 403 leak; add `s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveResolve)` with a handler that, for this plan, runs `moveOperatorGate` then answers 409 `"nothing to resolve yet"`; Plan 4 Task 10 and Plan 6 Task 2 give it its body. Imports: `crypto/rand`, `slices`, `github.com/carlosframework/platform/internal/hostclaim`, `.../internal/moves`, `.../internal/release`. |
| +**The target box** (spec rev 7.20 §7.1, D15): `selectMoveBoxes` above calls main's `eligibleBoxes(snap edgesSnapshot, verified map[string]boxid.Identity, placement, awsRegion, homeName, expectedFP string, lost map[string]bool) []boxCandidate` and `chooseBox(cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool)` (`internal/console/boxchoice.go:28`, `:76`) with every input composed in this task (the doc comment names each source); **`chooseBox` is not changed**: Phase 1 refuses two candidates before it runs, so its tie-break is unreachable here and stays region HA's to redefine (L15). More than one candidate on either side → M-C21. The chosen box goes into `op.intent.target` (`BeginMove`'s `targetBox`), and T8 and T9 (Task 11) bind to it: the reporting box's own id must equal `op.intent.target`, not merely publish the placement. `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` switch to `moveOperatorGate`. In `authcli.go`, the three mutation routes keep `svcrole.Allow(svcrole.Instance)` only so a service token meets the operator 404 rather than the role 403 leak; add `s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveResolve)` with a handler that, for this plan, runs `moveOperatorGate` then answers 409 `"nothing to resolve yet"`; Plan 4 Task 10 and Plan 6 Task 2 give it its body. Imports: `crypto/rand`, `errors`, `os`, `slices`, `github.com/carlosframework/platform/internal/boxid`, `.../internal/hostclaim`, `.../internal/moves`, `.../internal/objstore`, `.../internal/release`. |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -1464,8 +1536,11 @@ func LogKey(prefix string, seq int, event string) string { |
| // supports: the copy's response carries the destination's version id, |
| // which a move's seal records (moves spec §7.3, "seal by version"). The |
| // source never lists the target's staging, so the response is the only |
| -// place the id can come from. s3Mirror implements it; a plain MirrorStore |
| -// records no versions and MirrorRound.Versions stays empty. |
| +// place the id can come from. Only the staging mirror (Task 6, |
| +// lease.NewStagingMirror, through Task 14's guardedMirror) implements it; |
| +// the ordinary s3Mirror and every plain MirrorStore record no versions and |
| +// MirrorRound.Versions stays empty, so production mirroring (NewS3Mirror, |
| +// the legacy worker) is unchanged (round-4 finding 7). |
| type VersionedCopier interface { |
| CopyVersioned(ctx context.Context, srcBucket, srcKey, srcETag, dstBucket, dstKey string) (etag, versionID string, err error) |
| } |
| @@ -1489,7 +1564,7 @@ type VersionedPutter interface { |
| } |
| ``` |
| |
| -and after a successful copy always write a receipt when `ver != ""` or the ETag differs: `book.set(ctx, r, copyReceipt{SrcETag: s.ETag, SrcSize: s.Size, DstETag: etagOrSrc, DstVersionID: ver})` where `etagOrSrc` is `etag` when non-empty else `s.ETag`; record `round.Versions[r] = ver` when `ver != ""`. `s3Mirror.CopyVersioned` is `Copy` returning also `aws.ToString(out.VersionId)`; `s3Mirror.PutVersioned` is `Put` returning `aws.ToString(out.VersionId)`; `Copy` and `Put` delegate to them. |
| +and after a successful copy always write a receipt when `ver != ""` or the ETag differs: `book.set(ctx, r, copyReceipt{SrcETag: s.ETag, SrcSize: s.Size, DstETag: etagOrSrc, DstVersionID: ver})` where `etagOrSrc` is `etag` when non-empty else `s.ETag`; record `round.Versions[r] = ver` when `ver != ""`. **`s3Mirror` does not implement either interface** (round-4 finding 7): `NewS3Mirror`'s production path, the legacy worker's, would otherwise enter the receipt-required mode with the switch off, and an unversioned destination can never supply a version receipt, so every unchanged object would be recopied every round. Version collection is opt-in: the only producer is Task 6's `*lease.StagingMirror`, reached only through Task 14's `guardedMirror` under the switch. `TestPlainS3MirrorRecordsNoVersions` asserts `_, ok := interface{}(&s3Mirror{}).(VersionedCopier); !ok` and the same for `VersionedPutter`, so a later change cannot flip the production mode by accident. |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -1510,17 +1585,19 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ### Task 6: The staging-work lease and the `SafeUntil` guard |
| |
| **Files:** |
| -- Create: `internal/moves/lease.go`, `internal/moves/lease_test.go` |
| +- Create: `internal/moves/lease.go`, `internal/moves/lease_test.go`, `internal/lease/staging.go`, `internal/lease/staging_test.go` |
| +- Modify: `internal/lease/lease.go` (`AcquireTTL`, `RenewMargin` exported), `internal/lease/leasetest/bucket.go` and `s3server.go` (a plain `PUT` stores a new version beside the create-only `PUT`, as S3 with versioning on does; `putLocked`), `internal/lease/leasetest/s3server_test.go` (`TestS3ServerPlainPutCreatesAVersion`; the create-only path keeps its existing test) |
| |
| **Interfaces:** |
| -- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; the SDK's SigV4 signer, `github.com/aws/aws-sdk-go-v2/aws/signer/v4` (in the pinned `aws-sdk-go-v2 v1.44.0`): `v4.NewSigner()` and `(Signer).SignHTTP(ctx, credentials aws.Credentials, r *http.Request, payloadHash, service, region string, signingTime time.Time, ...) error`; `aws.Config.Credentials.Retrieve(ctx)` for the credentials the mirror's clients already load. **There is no copy presigner in S3 v1.105.0** (round-3 finding 3), so the staging mirror signs by hand: `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (*StagingMirror, error)` is a `MirrorStore` plus `VersionedCopier`/`VersionedPutter` whose every mutation builds the plain S3 REST request itself, signs it with `SignHTTP` at **`signingTime = Clock()`** (the call's clock, two statements after `Guarded.Before`'s check), and sends it once with `http.Client{Timeout: 120 * time.Second}` and no retry: |
| - - `CopyVersioned(ctx, srcBucket, srcKey, srcETag, dstBucket, dstKey)`: `PUT https://<dstBucket>.s3.<dstRegion>.amazonaws.com/<dstKey>` with headers `x-amz-copy-source: /<srcBucket>/<url-escaped srcKey>`, `x-amz-copy-source-if-match: "<srcETag>"` when `srcETag != ""`, payload hash `UNSIGNED-PAYLOAD`; a 200 whose body is `<CopyObjectResult><ETag>…</ETag>…` gives the ETag (unquoted), and the `x-amz-version-id` response header the version id; a 412 `PreconditionFailed` (the error XML's `<Code>`) is `ErrCopySourceChanged`; any other non-2xx is an error naming the code; a 200 with an `<Error>` body (S3's late-failure form for copies) is an error too. |
| - - `PutVersioned(ctx, bucket, key, body)`: `PUT` with the body's SHA-256 as the payload hash (`x-amz-content-sha256`) and `Content-Type: application/json`; `x-amz-version-id` is the version id. `PutIfAbsentVersioned(ctx, bucket, key, body)` adds `If-None-Match: *` and maps 412 to `lease.ErrExists` (the guarded create-only put `move.json` needs, Task 14). |
| - - `Delete(ctx, bucket, key)`: `DELETE`, 204. |
| +- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; the SDK's SigV4 signer, `github.com/aws/aws-sdk-go-v2/aws/signer/v4` (in the pinned `aws-sdk-go-v2 v1.44.0`): `v4.NewSigner()` and `(Signer).SignHTTP(ctx, credentials aws.Credentials, r *http.Request, payloadHash, service, region string, signingTime time.Time, ...) error`; `aws.Config.Credentials.Retrieve(ctx)` for the credentials the mirror's clients already load. **There is no copy presigner in S3 v1.105.0** (round-3 finding 3), so the staging mirror signs by hand: `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (*StagingMirror, error)` is a `MirrorStore` plus `VersionedCopier`/`VersionedPutter` whose every mutation builds the plain S3 REST request itself, signs it with `SignHTTP` at **the timestamp `Guarded.Before` captured for its check** (`signAt`, a parameter of every mutation; the mirror reads no clock of its own for signing, round-4 finding 3), and sends it once with `http.Client{Timeout: 120 * time.Second}` and no retry: |
| + - `CopyVersionedAt(ctx, signAt time.Time, srcBucket, srcKey, srcETag, dstBucket, dstKey)`: `PUT https://<dstBucket>.s3.<dstRegion>.amazonaws.com/<dstKey>` with headers `x-amz-copy-source: /<srcBucket>/<url-escaped srcKey>`, `x-amz-copy-source-if-match: "<srcETag>"` when `srcETag != ""`, payload hash `UNSIGNED-PAYLOAD`; a 200 whose body is `<CopyObjectResult><ETag>…</ETag>…` gives the ETag (unquoted), and the `x-amz-version-id` response header the version id; a 412 `PreconditionFailed` (the error XML's `<Code>`) is `ErrCopySourceChanged`; any other non-2xx is an error naming the code; a 200 with an `<Error>` body (S3's late-failure form for copies) is an error too. |
| + - `PutVersionedAt(ctx, signAt, bucket, key, body)`: `PUT` with the body's SHA-256 as the payload hash (`x-amz-content-sha256`) and `Content-Type: application/json`; `x-amz-version-id` is the version id. `PutIfAbsentVersionedAt(ctx, signAt, bucket, key, body)` adds `If-None-Match: *` and maps 412 to `lease.ErrExists` (the guarded create-only put `move.json` needs, Task 14). |
| + - `DeleteAt(ctx, signAt, bucket, key)`: `DELETE`, 204. |
| + - `*StagingMirror` deliberately satisfies **none** of `VersionedCopier`, `VersionedPutter` or `MirrorStore`'s mutations: every signed mutation needs a `signAt`, and only `Guarded.Before` hands one out, so no code path can sign outside the check. Task 14's `guardedMirror` is the adapter that implements those interfaces by calling `Before` and the `At` methods. |
| - `List` and `Get` delegate to the ordinary `s3Mirror` (reads are not mutations; the guard is about writes landing after the lease). |
| - - `Clock func() time.Time` on the struct (production `time.Now`; a test injects); `Credentials aws.CredentialsProvider` from the dst client's config; `Endpoint` override for tests (`NewStagingMirrorAt(t, url, bucketRegion)`). |
| - A caller that wants a retry calls again, through a fresh `Guarded.Before`, which signs again at that check. This is §7.3's "the signature is computed at that check, with the check's time as its `X-Amz-Date`, and never recomputed" made literal; the only pause the rule cannot cover is between `Before`'s clock read and `SignHTTP`'s `signingTime`, two statements apart in one goroutine, which the skew allowance covers. |
| -- Produces: `moves.StagingLeaseTTL = 45 * time.Minute` (D3: the ordinary 15-minute `LeaseTTL` cannot leave the signature's life in front of `SafeUntil`); `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func() error) error`: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life, so `Before` refuses with `moves.ErrLeaseUnsafe` when `Now().Add(SignatureLife + Skew).After(safe)` (`SignatureLife = 15 * time.Minute`, `Skew` default 1 min) and otherwise calls `fn` at once in the same goroutine. The lane's retry loop (Task 14) calls `Before` again for each retry; nothing retries inside `fn`. |
| + - No clock on the struct: the signing time is the guard's. `Credentials aws.CredentialsProvider` from the dst client's config; `Endpoint` override for tests (`NewStagingMirrorAt(t, url, bucketRegion)`). |
| + A caller that wants a retry calls again, through a fresh `Guarded.Before`, which captures a new timestamp. This is §7.3's "the signature is computed at that check, with the check's time as its `X-Amz-Date`, and never recomputed" made literal: the check's timestamp IS the signing time, so a pause of any length between the check and the signing (a VM freeze included) yields a signature dated at the check, which S3 refuses once that date is 15 minutes old, exactly the life the guard required of the lease. The residual gap is the one the spec names (§7.3, model 14): between the signing and the socket write, covered by the skew allowance and the bounded request timeout. |
| +- Produces: `moves.StagingLeaseTTL = 45 * time.Minute` (D3: the ordinary 15-minute `LeaseTTL` cannot leave the signature's life in front of `SafeUntil`); `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func(signAt time.Time) error) error`: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life, so `Before` reads `now := g.Now()` once, refuses with `moves.ErrLeaseUnsafe` when `now.Add(SignatureLife + Skew).After(safe)` (`SignatureLife = 15 * time.Minute`, `Skew` default 1 min) and otherwise calls `fn(now)` at once in the same goroutine, handing it the very timestamp it checked. The lane's retry loop (Task 14) calls `Before` again for each retry; nothing retries inside `fn`. |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -1543,11 +1620,12 @@ func TestGuardedRefusesWithinTheSignatureLife(t *testing.T) { |
| t0 := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| g := Guarded{Lease: fixedLease{t0.Add(30 * time.Minute)}, Now: func() time.Time { return t0 }, Skew: time.Minute} |
| ran := 0 |
| - if err := g.Before(func() error { ran++; return nil }); err != nil || ran != 1 { |
| - t.Fatalf("29 minutes of room: err %v ran %d", err, ran) |
| + var got time.Time |
| + if err := g.Before(func(at time.Time) error { ran++; got = at; return nil }); err != nil || ran != 1 || !got.Equal(t0) { |
| + t.Fatalf("29 minutes of room: err %v ran %d signAt %v", err, ran, got) |
| } |
| g.Now = func() time.Time { return t0.Add(14*time.Minute + 1*time.Second) } |
| - if err := g.Before(func() error { ran++; return nil }); !errors.Is(err, ErrLeaseUnsafe) || ran != 1 { |
| + if err := g.Before(func(time.Time) error { ran++; return nil }); !errors.Is(err, ErrLeaseUnsafe) || ran != 1 { |
| t.Fatalf("inside the 15 min + skew window: err %v ran %d", err, ran) |
| } |
| if StagingLeasePrefix("uk", "a.go.example.com") != "control/staging-leases/uk/a.go.example.com" { |
| @@ -1572,22 +1650,25 @@ func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| fresh := func(ttl time.Duration) SafeUntilGuard { |
| return fixedLease{t0.Add(ttl - lease.RenewMargin)} // what Held.SafeUntil answers right after Acquire; RenewMargin exported in this task |
| } |
| - if err := (Guarded{Lease: fresh(StagingLeaseTTL), Now: func() time.Time { return t0 }}).Before(func() error { return nil }); err != nil { |
| + if err := (Guarded{Lease: fresh(StagingLeaseTTL), Now: func() time.Time { return t0 }}).Before(func(time.Time) error { return nil }); err != nil { |
| t.Fatalf("a fresh staging lease is refused: %v", err) |
| } |
| - if err := (Guarded{Lease: fresh(lease.LeaseTTL), Now: func() time.Time { return t0 }}).Before(func() error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| + if err := (Guarded{Lease: fresh(lease.LeaseTTL), Now: func() time.Time { return t0 }}).Before(func(time.Time) error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| t.Fatalf("a fresh 15 minute lease passed the guard: %v", err) |
| } |
| } |
| |
| -// The staging mirror signs at the check and never again: a request carries |
| -// the X-Amz-Date of the moment it was signed, and a second attempt is a |
| -// second Before. The fixture is internal/lease/leasetest's local S3 |
| -// (NewVersionedBucket + NewS3Server), which serves plain PUT, create-only |
| -// PUT and version-specific DELETE; the test asserts the signed header, |
| -// not AWS's acceptance of it. |
| +// The staging mirror signs at the check and never again: a request |
| +// carries the X-Amz-Date of the timestamp Guarded.Before captured, and a |
| +// second attempt is a second Before. The fixture is a recording handler |
| +// (the test asserts the signed header, not AWS's acceptance of it) in |
| +// front of internal/lease/leasetest's local S3 (NewVersionedBucket with |
| +// the bucket added, NewS3Server), which this task extends to store a plain |
| +// PUT as a new version beside its create-only PUT (round-4 finding 9: on |
| +// main a plain PUT answers 501; S3 with versioning on stores a version). |
| func TestStagingMirrorSignsAtTheCheck(t *testing.T) { |
| bucket := leasetest.NewVersionedBucket() |
| + bucket.AddBucket("data-uk") |
| handler, _ := leasetest.NewS3Server(bucket, "data-uk") |
| var dates []string |
| srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| @@ -1597,23 +1678,46 @@ func TestStagingMirrorSignsAtTheCheck(t *testing.T) { |
| defer srv.Close() |
| m := lease.NewStagingMirrorAt(t, srv.URL, "eu-west-2") // the test constructor beside NewStagingMirror: static test credentials, the endpoint override, path-style addressing |
| t0 := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| - clock := t0 |
| - m.Clock = func() time.Time { return clock } |
| - if _, err := m.PutVersioned(context.Background(), "data-uk", "moves/h/mv/timers/t.json", []byte("{}")); err != nil { |
| + now := t0 |
| + g := Guarded{Lease: fixedLease{t0.Add(30 * time.Minute)}, Now: func() time.Time { return now }} |
| + // First write: checked and signed at t0. |
| + if err := g.Before(func(at time.Time) error { |
| + _, err := m.PutVersionedAt(context.Background(), at, "data-uk", "moves/h/mv/timers/t.json", []byte("{}")) |
| + return err |
| + }); err != nil { |
| t.Fatal(err) |
| } |
| - clock = t0.Add(3 * time.Minute) |
| - if _, err := m.PutVersioned(context.Background(), "data-uk", "moves/h/mv/timers/t.json", []byte("{}")); err != nil { |
| + // Second write: the check happens at t0+3m; then the goroutine pauses |
| + // past every bound before the request is signed and sent. The |
| + // signature must carry the CHECK's time, so no pause can mint a fresh |
| + // signature after the lease is gone (round-4 finding 3). |
| + now = t0.Add(3 * time.Minute) |
| + if err := g.Before(func(at time.Time) error { |
| + now = t0.Add(40 * time.Minute) // the pause: the wall clock is past the lease |
| + _, err := m.PutVersionedAt(context.Background(), at, "data-uk", "moves/h/mv/timers/t.json", []byte("{}")) |
| + return err |
| + }); err != nil { |
| t.Fatal(err) |
| } |
| if len(dates) != 2 || dates[0] != "20261006T090000Z" || dates[1] != "20261006T090300Z" { |
| - t.Fatalf("X-Amz-Date per request: %v", dates) |
| + t.Fatalf("X-Amz-Date per request must be the check's time: %v", dates) |
| + } |
| + // The lease is now unsafe: a third Before refuses and sends nothing. |
| + if err := g.Before(func(time.Time) error { t.Fatal("signed after the bound"); return nil }); !errors.Is(err, ErrLeaseUnsafe) || len(dates) != 2 { |
| + t.Fatalf("after the bound: %v, %d requests", err, len(dates)) |
| } |
| // create-only through the same signed path |
| - if _, err := m.PutIfAbsentVersioned(context.Background(), "data-uk", "moves/h/mv/move.json", []byte("{}")); err != nil { |
| + now = t0.Add(4 * time.Minute) |
| + put := func() error { |
| + return g.Before(func(at time.Time) error { |
| + _, err := m.PutIfAbsentVersionedAt(context.Background(), at, "data-uk", "moves/h/mv/move.json", []byte("{}")) |
| + return err |
| + }) |
| + } |
| + if err := put(); err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := m.PutIfAbsentVersioned(context.Background(), "data-uk", "moves/h/mv/move.json", []byte("{}")); !errors.Is(err, lease.ErrExists) { |
| + if err := put(); !errors.Is(err, lease.ErrExists) { |
| t.Fatalf("second create-only put: %v", err) |
| } |
| } |
| @@ -1663,17 +1767,20 @@ func StagingLeasePrefix(targetRegion, host string) string { |
| type SafeUntilGuard interface{ SafeUntil() (time.Time, error) } |
| |
| // Guarded wraps every staging request: the check runs immediately before |
| -// fn signs, after any pause, and a request that cannot complete within the |
| -// signature's life before the lease ends is dropped. fn must issue exactly |
| -// one request through a client that signs once and never retries |
| -// (lease.NewStagingMirror); a caller that wants a retry calls Before again. |
| +// fn signs, and fn signs AT THE CHECK'S TIMESTAMP, which Before hands it. |
| +// A request that cannot complete within the signature's life before the |
| +// lease ends is dropped; a request signed at the check and sent after any |
| +// pause carries a date S3 stops accepting exactly when the guard's bound |
| +// said it must. fn must issue exactly one request through a client that |
| +// signs once, at signAt, and never retries (lease.NewStagingMirror's At |
| +// methods); a caller that wants a retry calls Before again. |
| type Guarded struct { |
| Lease SafeUntilGuard |
| Now func() time.Time |
| Skew time.Duration |
| } |
| |
| -func (g Guarded) Before(fn func() error) error { |
| +func (g Guarded) Before(fn func(signAt time.Time) error) error { |
| safe, err := g.Lease.SafeUntil() |
| if err != nil { |
| return fmt.Errorf("%w: %v", ErrLeaseUnsafe, err) |
| @@ -1682,10 +1789,11 @@ func (g Guarded) Before(fn func() error) error { |
| if skew == 0 { |
| skew = DefaultSkew |
| } |
| - if g.Now().Add(SignatureLife + skew).After(safe) { |
| + now := g.Now() |
| + if now.Add(SignatureLife + skew).After(safe) { |
| return ErrLeaseUnsafe |
| } |
| - return fn() |
| + return fn(now) |
| } |
| ``` |
| |
| @@ -2195,15 +2303,23 @@ func restoreModes(fixes []modeFix) error { |
| // (regular files and directories only), and the uncompressed cap. It |
| // writes nothing and returns the first mismatch. |
| func Verify(sealed io.Reader, m Manifest, priv *ecdh.PrivateKey, b appconfig.Binding) error { |
| + if m.Sealed == "" || m.SealedKey == "" { |
| + return errors.New("archive: manifest carries no sealed digest or key") |
| + } |
| + sealedHash := sha256.New() |
| pr, pw := io.Pipe() |
| - go func() { pw.CloseWithError(Open(sealed, m.SealedKey, priv, b, pw)) }() |
| - h := sha256.New() |
| - zr, err := zstd.NewReader(io.TeeReader(pr, h)) |
| + go func() { pw.CloseWithError(Open(io.TeeReader(sealed, sealedHash), m.SealedKey, priv, b, pw)) }() |
| + zr, err := zstd.NewReader(pr) |
| if err != nil { |
| return err |
| } |
| defer zr.Close() |
| - tr := tar.NewReader(zr) |
| + // h hashes the DECODED tar stream: that is what Build's capWriter |
| + // hashed (it sits between tar.Writer and the zstd writer, trailer |
| + // included), so Plain is the same bytes on both sides (round-4 |
| + // finding 8). Hashing the compressed stream would never match. |
| + h := sha256.New() |
| + tr := tar.NewReader(io.TeeReader(zr, h)) |
| want := map[string]Entry{} |
| for _, e := range m.Entries { |
| want[e.Path] = e |
| @@ -2242,17 +2358,32 @@ func Verify(sealed io.Reader, m Manifest, priv *ecdh.PrivateKey, b appconfig.Bin |
| } |
| delete(want, hdr.Name) |
| } |
| + // Drain: tar.Reader stops at the end-of-archive marker, so the trailer |
| + // blocks still have to reach h; then the pipe is read to EOF so Open's |
| + // TERMINAL result (the last chunk's authentication, the final-chunk |
| + // flag) becomes this read's error through CloseWithError. A Verify |
| + // that returned at tar EOF would trust an archive whose tail was never |
| + // authenticated. |
| + if _, err := io.Copy(h, zr); err != nil { |
| + return err |
| + } |
| + if _, err := io.Copy(io.Discard, pr); err != nil { |
| + return err |
| + } |
| if len(want) != 0 { |
| return fmt.Errorf("archive: %d manifest entries missing from the archive", len(want)) |
| } |
| if got := digestString(h); got != m.Plain { |
| return fmt.Errorf("archive: plaintext digest %s, manifest says %s", got, m.Plain) |
| } |
| + if got := digestString(sealedHash); got != m.Sealed { |
| + return fmt.Errorf("archive: sealed digest %s, manifest says %s", got, m.Sealed) |
| + } |
| return nil |
| } |
| ``` |
| |
| -(`Extract` declares `var modes []modeFix` before the loop and calls `restoreModes(modes)` on `io.EOF` before returning nil; `Verify`'s `h` hashes the zstd stream, which is what `Build`'s `capWriter` hashed, so `Plain` is the same bytes.) Tests added: `TestExtractPreservesModes` (a `0555` directory holding a `0400` file round-trips to exactly those modes; `PlainDigest` of the extracted tree equals the original) and `TestVerifyRefusesWhatTheManifestDoesNotPromise` (a correctly sealed archive with a manifest whose `Plain` is wrong, whose entry list lacks a file, whose entry sha256 differs, plus a hand-built sealed archive containing a symlink entry, and one over the cap: each returns the named error; the good archive verifies). |
| +(`Extract` declares `var modes []modeFix` before the loop and calls `restoreModes(modes)` on `io.EOF` before returning nil. `Verify`'s `h` hashes the decoded tar, trailer included, which is exactly what `Build`'s `capWriter` hashed, since `capWriter` sits between the `tar.Writer` and the zstd writer; `Verify` also checks `m.Sealed` over the sealed bytes it read and drains both streams to EOF so `Open`'s terminal authentication result is the verdict.) Tests added: `TestExtractPreservesModes` (a `0555` directory holding a `0400` file round-trips to exactly those modes; `PlainDigest` of the extracted tree equals the original) and `TestVerifyRefusesWhatTheManifestDoesNotPromise` (a correctly sealed archive with a manifest whose `Plain` is wrong, whose entry list lacks a file, whose entry sha256 differs, plus a hand-built sealed archive containing a symlink entry, one over the cap, one whose last sealed byte is flipped and one whose final chunk is cut off (the last two return `Open`'s authentication error, never nil): each returns the named error; the good archive verifies, and a manifest whose `sealed` differs from the bytes is refused). |
| |
| `seal.go`: |
| |
| @@ -2412,14 +2543,15 @@ Moved here from Plan 2 (its Tasks 4 and 7) because T3, T7 and S2 need the real t |
| |
| **Files:** |
| - Create: `internal/appconfig/renderheld.go`, `internal/appconfig/renderheld_test.go`, `internal/release/materializeheld.go`, `internal/release/materializeheld_test.go` |
| -- Modify: `internal/activator/backend_exec.go` (`Strict`, `NextSpawnStrict`, the strict spawn path at `spawnInCgroup`), `internal/activator/backend_exec_test.go` |
| +- Modify: `internal/activator/backend_exec.go` (`Strict`, `prepareStrict`, `NextSpawnStrict`, the strict spawn path at `spawnInCgroup`), `internal/activator/backend_exec_test.go`, `internal/appconfig/instances.go` (`LoadMergedHostVersions`), `internal/host/agent.go` (`carlos adopt --stage-only` calls `release.ServeHeldStageRequests`), `internal/host/agent_test.go` |
| |
| **Interfaces:** |
| -- `appconfig.RenderHeld(ctx, s release.Store, priv *ecdh.PrivateKey, dataDir string, rt registry.Route, op string, box string) (HeldRender, error)` with `HeldRender{Path, SHA256 string; BundleETags map[string]string; OverlayETag string}`: `LoadMergedHost(ctx, s, acct, app, rt.Environment, rt.Host)`, rendered through the same file writer `syncExecHost` uses (the file-writing half extracted into `renderExecConfig(rt, m, key, priv, dataDir) (path string, changed bool, err error)` and called from both), written whether or not it changed, and a move-bound ack through `WriteConfigAck` with three additive `ConfigAck` fields: `MoveOp string \`json:"move_op,omitempty"\``, `FileSHA256 string \`json:"file_sha256,omitempty"\``, `BundleETags map[string]string \`json:"bundle_etags,omitempty"\``. `SyncExec` keeps skipping non-ready rows (`execRouteBase`), so the two never race on one file. |
| -- `release.MaterializeHeld(ctx, s Store, optDir string, rt registry.Route, version string) (*Manifest, error)`: `ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, version, rt.Host)` then the live symlink exactly as `adoptHost` points it (that symlink step extracted into `pointLive(optDir, host, target string) error` if it is inline), no restart, no bake-hold check: a held row runs exactly the pinned release. |
| -- `ExecBackend.Strict func(host string) bool` (when true the spawn uses `Cgroups.PrepareStrict(host, limit)` then `Prepared.Start(cmd)`, keeping the `*cgroup.Prepared` for the process's life and running `KillAndEmpty` then `Close` on exit; a failure refuses the spawn with `ErrStrictRequired`); `(b *ExecBackend) NextSpawnStrict(host string) moves.Check` (S2: OK when `Strict(host)` and cgroup v2 delegation is present, `cgroup.Manager.Root() != ""`, and `PrepareStrict` is supported on this build). The host package sets `eb.Strict` from the hold view (a host whose record has a move in flight and the switch on); off → never strict, today's behaviour. |
| +- `appconfig.LoadMergedHostVersions(ctx, s release.Store, acct, app, envName, host string) (*Manifest, MergedVersions, error)` with `MergedVersions{Default, Environment, Overlay string}`: the same three reads `LoadMergedHost` and `LoadMerged` make (`instances.go:202`, `environments.go:169`), through `getIfExistsAtETag` (`store.go:65`) so the etag of each object actually merged is returned with the bytes; `LoadMergedHost` becomes a wrapper that discards all but the overlay etag (behaviour unchanged, its tests untouched). A later independent read could not establish which versions were rendered (round-4 finding 12), so this is the one loader `RenderHeld` and T3 use. |
| +- `appconfig.RenderHeld(ctx, s release.Store, priv *ecdh.PrivateKey, dataDir string, rt registry.Route, op string, box string) (HeldRender, error)` with `HeldRender{Path, SHA256 string; BundleETags map[string]string; OverlayETag string}` (`BundleETags` = `{"default": v.Default, "environment": v.Environment}`, `OverlayETag` = `v.Overlay`): `LoadMergedHostVersions(ctx, s, acct, app, rt.Environment, rt.Host)`, rendered through the same file writer `syncExecHost` uses (the file-writing half extracted into `renderExecConfig(rt, m, key, priv, dataDir) (path string, changed bool, err error)` and called from both), written whether or not it changed, and a move-bound ack through `WriteConfigAck` with three additive `ConfigAck` fields: `MoveOp string \`json:"move_op,omitempty"\``, `FileSHA256 string \`json:"file_sha256,omitempty"\``, `BundleETags map[string]string \`json:"bundle_etags,omitempty"\``. `SyncExec` keeps skipping non-ready rows (`execRouteBase`), so the two never race on one file. |
| +- `release.MaterializeHeld(ctx, s Store, optDir string, rt registry.Route, version string) (*Manifest, error)`: `ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, version, rt.Host)` (`adopt.go:1317`) then `PointLive(optDir, host, acct, version, app)` (`versions.go:267`, already exported), no restart, no bake-hold check: a held row runs exactly the pinned release. When this process cannot write `optDir` (`errors.Is(err, fs.ErrPermission)` from the stage or the symlink: the edge unit has `ProtectSystem=strict` and writes only `/data` and `/run/carlos`, `carlos-edge.service:35-37`), it returns `release.ErrStageBlocked` wrapping the cause, so the caller relays instead of failing (Task 11's T7, round-4 finding 6). `release.ServeHeldStageRequests(ctx, s Store, pub ed25519.PublicKey, optDir, dataDir string) error` is the root side of that relay: for every `<dataDir>/moves/held-stage/<host>.json` (`{acct, app, host, version, op}`), it runs `MaterializeHeld` and writes `<dataDir>/moves/held-staged/<host>.json` (`{version, op, at}`), then removes the request; `carlos adopt --stage-only` (the root pass `carlos-stage.service` runs when `/data/stage-requested` changes, `agent.go:3525`) calls it after its release adoption. `release.LiveVersion(optDir, host string) (string, error)` reads the `live/<host>` symlink's target version (readable from the sandboxed agent), which is the proof T7 trusts; the ack file is only a hint. |
| +- `ExecBackend.Strict func(host string) bool` (when true the spawn uses `b.prepareStrict(host, limit)` then `Start(cmd)` on what it returns, keeping it for the process's life and running `KillAndEmpty(ctx)` then `Close()` on exit; a failure refuses the spawn with `ErrStrictRequired`). `prepareStrict func(host string, limit cgroup.Limit) (strictProc, error)` is an unexported seam with `strictProc interface{ Start(*exec.Cmd) error; KillAndEmpty(context.Context) error; Close() error }`, which `*cgroup.Prepared` satisfies on every build (`strict_linux.go:233-348`, `strict_other.go:15-18`); nil → `b.Cgroups.PrepareStrict`, and a nil `Cgroups` under `Strict` is `ErrStrictRequired`. `Cgroups` stays the concrete `*cgroup.Manager` (`backend_exec.go:100`): tests replace the seam, never the field (round-4 finding 2). `(b *ExecBackend) NextSpawnStrict(host string) moves.Check` (`moves.Check` is Task 1's; S2: OK when `Strict(host)` and cgroup v2 delegation is present, `cgroup.Manager.Root() != ""`, and `PrepareStrict` is supported on this build). The host package sets `eb.Strict` from the hold view (a host whose record has a move in flight and the switch on); off → never strict, today's behaviour. |
| |
| -Tests: `RenderHeld` writes the file for a `landing` row `SyncExec` would skip and the ack carries `move_op`, the sha256 and the bundle etags; a second call with no change still writes an ack. `MaterializeHeld` stages the pinned version into a temp `optDir` and points `live/<host>` at it, never at the channel head. `TestSpawnUnderAMoveIsStrictOrRefused` (a fake `Cgroups` whose `PrepareStrict` fails → `Start` returns `ErrStrictRequired` and spawns nothing; with `Strict` false the plain path runs as today). |
| +Tests: `RenderHeld` writes the file for a `landing` row `SyncExec` would skip and the ack carries `move_op`, the sha256 and the bundle etags, which equal the etags the dir store reports for the fixture's default and environment objects; a second call with no change still writes an ack. `MaterializeHeld` stages the pinned version into a temp `optDir` and points `live/<host>` at it, never at the channel head; with `optDir` made read-only (`os.Chmod(optDir, 0o555)`, skipped as root) it returns `ErrStageBlocked` and writes nothing. `ServeHeldStageRequests` materializes a request file into `optDir`, writes the ack and removes the request; `LiveVersion` reads the result. `TestSpawnUnderAMoveIsStrictOrRefused` (the `prepareStrict` seam returning an error → `Start` returns `ErrStrictRequired` and spawns nothing; the seam returning a recording `strictProc` → `Start` is called on it and `KillAndEmpty` then `Close` on exit; with `Strict` false the plain path runs as today). |
| |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "appconfig.RenderHeld, release.MaterializeHeld, strict spawns under a move (§9.1, §9.2, §5.3)" ...` |
| |
| @@ -2640,10 +2772,18 @@ func TestEnvReportRevealsNamesAndEqualityOnly(t *testing.T) { |
| if diffs := CompareEnv(src, noRegion, true); len(diffs) != 1 || diffs[0].Kind != "region-missing" { |
| t.Fatalf("missing region: %v", diffs) |
| } |
| - // a layout variable absent from both: still required of a flat landing |
| + // the layout variable absent from both: still required of a flat landing |
| srcNoLayout := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "ie"}) |
| - if diffs := CompareEnv(srcNoLayout, ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "uk"}), true); len(diffs) != 2 { |
| - t.Fatalf("layout names absent from both: %v", diffs) |
| + if diffs := CompareEnv(srcNoLayout, ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "uk"}), true); len(diffs) != 1 || diffs[0].Name != "CARLOS_CONTROL_SOCKET" || diffs[0].Kind != "layout-missing" { |
| + t.Fatalf("layout name absent from both: %v", diffs) |
| + } |
| + // CARLOS_PURGE is advisory (set iff the box has a cache): a presence |
| + // difference is a note, never a failure; main omits the variable on a |
| + // box without a cache (backend_exec.go:1016), so neither example above |
| + // carries it and they compare clean. |
| + purge := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:a", "CARLOS_REGION": "uk", "CARLOS_CONTROL_SOCKET": "/run/y", "SECRET": "s3", "CARLOS_PURGE": "header"}) |
| + if diffs := CompareEnv(src, purge, true); len(diffs) != 1 || diffs[0].Name != "CARLOS_PURGE" || diffs[0].Kind != "advisory-differs" || !diffs[0].Note() { |
| + t.Fatalf("purge presence: %v", diffs) |
| } |
| drift := ReportEnv(key, map[string]string{"DATABASE_URL": "sqlite:b", "CARLOS_REGION": "uk", "EXTRA": "x"}) |
| kinds := map[string]string{} |
| @@ -2666,8 +2806,11 @@ And in `internal/activator/backend_exec_test.go`: |
| |
| ```go |
| func TestLaunchEnvironmentComposesTheSpawnEnvironment(t *testing.T) { |
| - b, rt := execBackendFixture(t) // the package's existing fixture for an ExecBackend with a DataDir; name per the file |
| - os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("MINE=1\nCARLOS_ADMIN_TOKEN=t\n"), 0o600) |
| + // Built the way this file's spawn tests build theirs (grep ExecBackend{ |
| + // in backend_exec_test.go; there is no shared fixture helper on main). |
| + b := &ExecBackend{DataDir: t.TempDir(), Region: "ie"} // Cache nil: no purge channel on this box |
| + rt := registry.Route{Host: "a.go.example.com", Socket: filepath.Join(b.DataDir, "a.sock")} |
| + os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("MINE=1\nCARLOS_ADMIN_TOKEN=t\nCARLOS_PURGE=header\nCARLOS_REGION=fake\n"), 0o600) |
| t.Setenv("CARLOS_SECRETS_KEY", "never") |
| t.Setenv("UNRELATED", "yes") |
| env, err := b.LaunchEnvironment(rt) |
| @@ -2677,11 +2820,21 @@ func TestLaunchEnvironmentComposesTheSpawnEnvironment(t *testing.T) { |
| if env["MINE"] != "1" || env["UNRELATED"] != "yes" || env["CARLOS_SECRETS_KEY"] != "" || env["CARLOS_ADMIN_TOKEN"] != "t" { |
| t.Fatalf("launch env: %v", env) |
| } |
| + // The agent-owned layer, exactly as agentOwnedVars composes it: the |
| + // tenant's region and purge values are overridden or removed. |
| + if env["CARLOS_REGION"] != "ie" || env["CARLOS_CONTROL_SOCKET"] != rt.Socket+".control" { |
| + t.Fatalf("agent-owned vars: %v", env) |
| + } |
| + if _, ok := env["CARLOS_PURGE"]; ok { |
| + t.Fatal("CARLOS_PURGE survived on a box without a cache") |
| + } |
| + b.Cache = edge.NewCache(16, 1<<20) |
| + if env, _ = b.LaunchEnvironment(rt); env["CARLOS_PURGE"] != "header" { |
| + t.Fatalf("CARLOS_PURGE on a box with a cache: %v", env) |
| + } |
| } |
| ``` |
| |
| -(`execBackendFixture` is whatever helper the file already uses to build an `ExecBackend` for a flat route; read the file and use that name.) |
| - |
| - [ ] **Step 2: Run tests to verify they fail** |
| |
| Run: `nice -n 10 go test ./internal/moves ./internal/activator -run 'TestEnvReport|TestLaunchEnvironment' -count=1` |
| @@ -2709,7 +2862,14 @@ import ( |
| const RegionVar = "CARLOS_REGION" |
| |
| var ( |
| - LayoutVars = map[string]bool{"CARLOS_CONTROL_SOCKET": true, "CARLOS_PURGE": true} |
| + // LayoutVars name a channel the flat layout always provides: present |
| + // on both sides of a flat move, values not compared (they are paths |
| + // on each box). |
| + LayoutVars = map[string]bool{"CARLOS_CONTROL_SOCKET": true} |
| + // AdvisoryVars follow a box fact (CARLOS_PURGE is set iff the box has |
| + // a response cache, backend_exec.go:1013-1016): a presence difference |
| + // is reported as a note, never a failure (round-4 finding 10). |
| + AdvisoryVars = map[string]bool{"CARLOS_PURGE": true} |
| BoxVars = map[string]bool{"INVOCATION_ID": true, "JOURNAL_STREAM": true, "HOSTNAME": true, "PWD": true, "OLDPWD": true, "SHLVL": true, "_": true} |
| ) |
| |
| @@ -2762,6 +2922,8 @@ func Class(name string) string { |
| return "region" |
| case LayoutVars[name]: |
| return "layout" |
| + case AdvisoryVars[name]: |
| + return "advisory" |
| case BoxVars[name]: |
| return "box" |
| default: |
| @@ -2769,13 +2931,18 @@ func Class(name string) string { |
| } |
| } |
| |
| +// Note reports a diff T3 records without failing. |
| +func (d EnvDiff) Note() bool { return d.Kind == "advisory-differs" } |
| + |
| type EnvDiff struct { |
| Name, Kind string |
| } |
| |
| // CompareEnv is T3: region must differ, layout variables must be present |
| -// iff the flat layout provides them (value not compared), box variables |
| -// are not compared, everything else must be equal by HMAC. |
| +// iff the flat layout provides them (value not compared), advisory |
| +// variables report a presence difference as a note (EnvDiff.Note), box |
| +// variables are not compared, everything else must be equal by HMAC. T3 |
| +// fails on any diff that is not a note. |
| func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| var diffs []EnvDiff |
| seen := map[string]bool{} |
| @@ -2796,6 +2963,10 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| if !flatLayout && ok { |
| diffs = append(diffs, EnvDiff{name, "layout-present"}) |
| } |
| + case "advisory": |
| + if !ok { |
| + diffs = append(diffs, EnvDiff{name, "advisory-differs"}) |
| + } |
| case "box": |
| default: |
| if !ok { |
| @@ -2806,8 +2977,14 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| } |
| } |
| for name := range target.Vars { |
| - if !seen[name] && Class(name) == "equal" { |
| + if seen[name] { |
| + continue |
| + } |
| + switch Class(name) { |
| + case "equal": |
| diffs = append(diffs, EnvDiff{name, "extra"}) |
| + case "advisory": |
| + diffs = append(diffs, EnvDiff{name, "advisory-differs"}) |
| } |
| } |
| // layout names absent from BOTH reports are still checked: a flat |
| @@ -2834,19 +3011,23 @@ func CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff { |
| // fileEnv, haveControl)), decoded back to a map. agentOwnedVars REMOVES |
| // tenant-provided CARLOS_CONTROL_SOCKET, CARLOS_PURGE and CARLOS_REGION |
| // when the agent does not provide them, so a report over mergeEnv alone |
| -// would misstate the environment; haveControl is whether this box's |
| -// spawn binds the control listener (b.Control != nil && the layout's |
| -// control socket path can be bound), computed the way Start computes it |
| -// before the spawn. The move's launch report (moves spec §7.3, codex |
| -// R2-8) is computed over exactly this, so a target compares what would |
| -// run, not a reconstruction of it. |
| +// would misstate the environment. haveControl is true: startChild calls |
| +// edge.ListenControl for EVERY exec child (backend_exec.go:890; the strict |
| +// path at prepare.go:121 too), and b.Control is the nested root-control |
| +// client, which says nothing about the flat channel (round-4 finding 10), |
| +// so before the spawn the honest composition is "the agent will bind it"; |
| +// a bind that fails at spawn is logged there and the probe's spawn-time |
| +// check (Plan 3 Task 5) compares the actual outcome. CARLOS_PURGE follows |
| +// b.Cache != nil exactly as agentOwnedVars does. The move's launch report |
| +// (moves spec §7.3, codex R2-8) is computed over exactly this, so a |
| +// target compares what would run, not a reconstruction of it. |
| func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, error) { |
| fileEnv, err := b.instanceEnv(rt) |
| if err != nil { |
| return nil, err |
| } |
| out := map[string]string{} |
| - for _, kv := range childEnv(b.agentOwnedVars(rt, fileEnv, b.controlAvailable(rt))) { |
| + for _, kv := range childEnv(b.agentOwnedVars(rt, fileEnv, true)) { |
| if i := strings.IndexByte(kv, '='); i > 0 { |
| out[kv[:i]] = kv[i+1:] |
| } |
| @@ -2855,7 +3036,7 @@ func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, e |
| } |
| ``` |
| |
| -`controlAvailable(rt)` is extracted from `Start`'s own decision about `haveControl` (the control-listener bind outcome is known only at spawn; before the spawn, "available" is `b.Control != nil` for the flat layout, which is what the report can know, and Task 11's T3 records that the comparison is of the pre-spawn composition; the probe's spawn-time check in Plan 3 Task 5 compares the actual one). The test asserts the removals: a tenant `<host>.env` that sets `CARLOS_REGION=fake` and `CARLOS_PURGE=header` is overridden or removed exactly as `Start` would (`CARLOS_REGION` = `b.Region`, `CARLOS_PURGE` absent when `b.Cache == nil`), `CARLOS_CONTROL_SOCKET` present iff `controlAvailable`, and a whole-layer omission is caught (a build with `agentOwnedVars` stubbed out fails the test). |
| +There is no `controlAvailable`: `agentOwnedVars(rt, fileEnv, true)` is the pre-spawn composition, because every exec child gets a control-listener bind attempt (`startChild`, `backend_exec.go:890`; `prepare.go:121` for the strict path) and `b.Control` (`backend_exec.go:129`) is the nested root-control client, not the flat channel; Task 11's T3 records that the comparison is of the pre-spawn composition, and the probe's spawn-time check in Plan 3 Task 5 compares the actual one. The test asserts the removals: a tenant `<host>.env` that sets `CARLOS_REGION=fake` and `CARLOS_PURGE=header` is overridden or removed exactly as `Start` would (`CARLOS_REGION` = `b.Region`; `CARLOS_PURGE` absent when `b.Cache == nil` and `header` when it is set), `CARLOS_CONTROL_SOCKET` always present at `rt.Socket+".control"`, and a whole-layer omission is caught (a build with `agentOwnedVars` stubbed out fails the test). The test builds its `ExecBackend` the way `backend_exec_test.go`'s existing spawn tests build theirs (there is no `execBackendFixture` helper on main; grep `ExecBackend{` in that file and reuse its construction), once with `Cache` nil and once with a cache. |
| |
| - [ ] **Step 4: Run tests to verify they pass** |
| |
| @@ -2953,8 +3134,8 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/host/movesreport.go`, `internal/host/movesreport_test.go` |
| |
| **Interfaces:** |
| -- Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check{OK bool; Reason string}`; `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string}`; `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| -- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` and its sealed object, read by the versions the source report names: `archive.Verify(sealed, manifest, priv, binding)` (Task 7) checks the plaintext digest, every entry's path, size and sha256, the entry types and the cap; the report records the manifest's `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld(ctx, s, optDir, rt, version)`, Task 7b: the pinned release is staged and pointed live for the held row, and the report records the version); T8 and T9 bind to the named target: this box's id must equal `op.intent.target` (`moveTargetBox`'s choice, Task 4), else T9 fails "this box is not the move's target"; the report writer is the named target only. S2 (`ExecBackend.NextSpawnStrict(host)`, Task 7b); S1 wakes a hibernating source the way a request would: `GET http://127.0.0.1:<edge port>/<ready path>` with `Host: <host>` through the box's own edge, which routes to the activator and wakes the instance (`moveCfg.ReadyProbe` does exactly that with a 30 s timeout; the edge's loopback listener and port come from the agent's existing edge config). Writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| +- Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check` (Task 1's); `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string; Archive *ArchiveRef}` with `ArchiveRef{ManifestKey, ManifestVersion, SealedKey, SealedVersion, Plain string}` (the versions the source's `PutVersionedAt` responses returned, Task 14's ledger `archive_versions`, published so T5 reads by exact version: round-4 finding 12); `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| +- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string, archive *moves.ArchiveRef) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the archive manifest and sealed object the source report's `Archive` names, read by exact version through `lease.VersionedStore`'s `GetVersion`/`OpenVersion` in the target's own bucket; a report without `Archive` fails T5 "the source has published no archive yet": `archive.Verify(sealed, manifest, priv, binding)` (Task 7) checks the plaintext digest, every entry's path, size and sha256, the entry types and the cap; the report records the manifest's `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld(ctx, s, optDir, rt, version)`, Task 7b; on a hardened box it returns `ErrStageBlocked`, and then the pass writes `<DataDir>/moves/held-stage/<host>.json` and touches the `stage-requested` marker through `stageRequester(dd)` (`agent.go:1262`, the existing relay to the root `carlos-stage.service`, whose `carlos adopt --stage-only` now calls `release.ServeHeldStageRequests`); T7 passes only once `release.LiveVersion(optDir, rt.Host)` equals the pinned version, and until then fails "waiting for the root stage pass"; the report records the version; round-4 finding 6); T8 and T9 bind to the named target: this box's id must equal `op.intent.target` (`selectMoveBoxes`'s choice, Task 4), else T9 fails "this box is not the move's target"; the report writer is the named target only. S2 (`ExecBackend.NextSpawnStrict(host)`, Task 7b); S1 wakes a hibernating source and reads the answer from its own process, never from the response cache (round-4 finding 11: `edge.go:1651-1663` serves a cached GET before the activator runs, so a cached 200 could certify a source that cannot wake): `(*edge.Edge).ServeUncached(w, r)` (Task 14 adds it: `ServeHTTP` with the request marked `uncachedProbe` in its context, the same marker pattern as `doorCallerFrom`, which makes `cacheable` false at `edge.go:1651` so neither `GetFor` nor `maybeStore` runs, while the hibernating branch still wakes through `e.act.ServeHTTP`) with `GET /<ready path>`, `Host: <host>`, a recording `ResponseWriter` and a 30 s timeout; `moveCfg.ReadyProbe` is set by `setMovesReadyProbe(e)` from `edge.RunWithActivatorReady`'s callback (`agent.go:576`, where the agent receives its `*edge.Edge`) and fails "edge not ready" before that. Direct socket dialing is not used: it cannot wake a sleeping instance. Writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| |
| Test (pure) `internal/moves/readiness_test.go`: |
| |
| @@ -3060,6 +3241,16 @@ type SourceReport struct { |
| S1 Check `json:"s1"` |
| S2 Check `json:"s2"` |
| FilesPlain string `json:"files_plain"` |
| + Archive *ArchiveRef `json:"archive,omitempty"` |
| +} |
| + |
| +// ArchiveRef names the archive the source published, by exact version. |
| +type ArchiveRef struct { |
| + ManifestKey string `json:"manifest_key"` |
| + ManifestVersion string `json:"manifest_version"` |
| + SealedKey string `json:"sealed_key"` |
| + SealedVersion string `json:"sealed_version"` |
| + Plain string `json:"plain"` |
| } |
| ``` |
| |
| @@ -3242,12 +3433,16 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| **Files:** |
| - Create: `internal/moves/floor.go`, `internal/moves/floor_test.go` |
| -- Modify: `internal/host/edgesfile.go` (field and writer), `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (refusal on activation), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup) |
| +- Modify: `internal/host/edgesfile.go` (field and writer), `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (refusal on activation), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup, switch on only), `internal/host/units/carlos-system-recover.service` and `internal/host/units/carlos-system-update.service` (the `.prev` selection honours the hold-prev marker), `internal/host/host_test.go` (the unit assertions), `internal/host/movesfloor.go` and `internal/host/movesfloor_test.go` (the agent's floor arming, `participants.json`, the hold-prev marker) |
| |
| **Interfaces:** |
| - Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| - `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| -- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| +- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<DataDir>/.moves/floor.json` durably (temp file, fsync, rename; under `/data`, which the sandboxed agent may write (`carlos-edge.service:37`, `ReadWritePaths=/data /run/carlos`) and root's `system apply` may read; `<optDir>` is not writable by the agent, round-4 finding 6); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| + |
| + **The recovery entry points are the units, not only the applier** (round-4 finding 5): `carlos-system-recover.service:17` and `carlos-system-update.service:25` exec **`carlos.prev system apply [--recover]`** directly when `/opt/carlos/system/router/journal.json` exists, and an older `.prev` runs its own recovery without these checks. So the agent maintains a **hold-prev marker**, `/data/.moves/hold-prev` (a one-line reason): written whenever this box is a participant (the registry's `hold_op` rows or `participants.json`, as above) and either no floor is armed or `.prev`'s version (the previous build `decidePrev` recorded, `systemapply.go:473-489`, read through `a.prevBuild(ctx)`) is below `Floor.MinRelease`; removed when the condition clears. Both units' `ExecStart` lines gain `&& ! [ -e /data/.moves/hold-prev ]` in the `.prev` condition, so a held box falls through to the current `carlos system apply`, whose `floorAllows` checks run; `host_test.go:1314-1330`'s unit assertions gain the marker string for both units (red before the unit edit). The hold is honest about its cost: a held box whose current `carlos` cannot start recovery loops on `OnFailure` until an operator acts, which is what §11 asks of a participant below the floor. |
| + |
| + **Dormancy (D13):** arming, `participants.json` and the marker run only with `moveCfg.Enabled`. With the switch off none of the three files exists, `floorAllows` finds no floor and no participant and proceeds with its log line, and the unit lines select `.prev` exactly as today: the dormant boundary is the absence of those files, and `TestFloorIsDormantWithTheSwitchOff` (an agent pass with the switch off writes none of them over a registry with a held row) pins it. |
| |
| - [ ] **Step 1: Write the failing tests** |
| |
| @@ -3273,7 +3468,7 @@ func TestFloorReadAndCompare(t *testing.T) { |
| } |
| ``` |
| |
| -In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<optDir>/moves-floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snap.Complete` (every edges file read; `complete()` would also demand `BoxesComplete`, which only `readLiveEdgesSnapshot` computes, and the floor has no liveness question); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| +In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<DataDir>/.moves/floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestHoldPrevMarkerFollowsTheFloor` (participant and no floor → marker with its reason; participant and `.prev` below the floor → marker; `.prev` at the floor → no marker; not a participant → no marker; switch off → never written); `TestSystemUnitsSkipPrevUnderHold` (the `ExecStart` line's `sh -c` script is run with a temp root substituted for `/opt/carlos` and `/data` and stub `carlos`/`carlos.prev` scripts that record which ran: with the marker the current `carlos` runs, without it `carlos.prev`); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snap.Complete` (every edges file read; `complete()` would also demand `BoxesComplete`, which only `readLiveEdgesSnapshot` computes, and the floor has no liveness question); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| |
| - [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**: |
| |
| @@ -3319,7 +3514,7 @@ func ReleaseAtOrAbove(version, minRelease string) (bool, error) { |
| } |
| ``` |
| |
| -(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup (`server.go`, where the edges-dependent config is logged): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| +(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup, **only when `s.cfg.MovesEnabled`** (D13: off publishes nothing; `TestConsoleRecordNeedsTheSwitch`), where the edges-dependent config is logged (`server.go`): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| |
| - [ ] **Step 4: Run**: `nice -n 10 go test ./internal/moves ./internal/host ./internal/console -count=1 -p 1` → PASS. |
| |
| @@ -3337,15 +3532,15 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ### Task 14: Preparing and ready rounds on the new layout |
| |
| **Files:** |
| -- Modify: `internal/host/movespass.go` (`moveCfg`, `armMovesPass`, `runMoveRound`'s `MovePreparing`/`MoveReady` arm, `MoveAborting` arm's prefix), `internal/host/movespass_test.go` (`fakeMoveMirror` gains `CopyVersioned`/`PutVersioned`; `moveHarness` gains a `lease.VersionedStore` fake and a staging-lease fake) |
| +- Modify: `internal/host/movespass.go` (`moveCfg`, `armMovesPass`, `runMoveRound`'s `MovePreparing`/`MoveReady` arm, `MoveAborting` arm's prefix), `internal/host/movespass_test.go` (`fakeMoveMirror` gains `CopyVersioned`/`PutVersioned`; `moveHarness` gains a `lease.VersionedStore` fake and a staging-lease fake), `internal/host/agent.go` (`setMovesReadyProbe` in the `RunWithActivatorReady` callback), `internal/edge/edge.go` (`ServeUncached` and the `uncachedProbe` context marker beside `doorCallerFrom`), `internal/edge/edge_test.go` (`TestServeUncachedBypassesTheCacheAndWakes`: the cache primed with a 200 for the path and a fake activator whose wake fails: `ServeHTTP` answers the cached 200, `ServeUncached` answers the wake failure and stores nothing) |
| |
| **Interfaces:** |
| -- `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1: `GET <ready path>` on the instance's socket through the activator's own request path; the activator's `ServeHTTP` is the edge's, so the probe dials the socket directly with `Host: <host>` and a 30 s timeout), `StrictNext func(host string) moves.Check` (S2: the activator reports whether the next spawn is strict; until Plan 2's strict-spawn task lands this returns OK when `cgroup` reports delegation). |
| +- `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1 through `(*edge.Edge).ServeUncached`, Task 11's definition; set by `setMovesReadyProbe(e *edge.Edge)` from `RunWithActivatorReady`'s callback, `agent.go:576`; "edge not ready" before that), `StrictNext func(host string) moves.Check` (S2: `ExecBackend.NextSpawnStrict`, Task 7b). |
| - The switch (Task 0, D13): `moveCfg.Enabled` decides which worker runs. **Off:** today's `runMoveRound` (renamed `legacyMoveRound`, unchanged, with its tests unchanged) handles every record exactly as main does; an id'd record (none exists while the switch was never on) is skipped with one log line a minute. **On:** `phase1MoveRound` below handles id'd records; a legacy record is refused with one log line a minute (M-C: MV-N7's sentence in the log, not to a user). |
| - The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveGuardedAbortComplete(g, now)`; the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| - **Every transition is guarded by op identity** (Task 2's `MoveGuard{ID: rec.Move.ID, Incarnation: rec.Move.Incarnation}`, captured when the round read the record): `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbortComplete`; a replacement move between the read and the CAS is `ErrMoveConflict` and the round ends (`TestRoundNeverAdvancesAReplacementMove`). |
| -- **The source copy is bound to the claim's location and generation** (round-3 finding 5), never to `HibernatePrefix(cfg.OwnPrefix, host)`: `moves.ReplicaSource(h hostclaim.Holder) (prefix string, keep func(rel string) bool)`: for generation `""` or `hostclaim.RootGeneration`, `prefix = h.Location` and `keep` admits only `<level 0000-0009>/<min>-<max>.ltx` directly under it (not `g-*/`, not a 64-hex child incarnation, not `generation.json`, `lock.json` or `closed/`); for a generation `g`, `prefix = h.Location + "g-" + g + "/"` and `keep` admits its level directories. `lease.MirrorPrefix` gains a variant `lease.MirrorPrefixKeep(ctx, m, srcBucket, srcPrefix, dstBucket, dstPrefix, rc, keep func(rel string) bool)` (`MirrorPrefix` is `MirrorPrefixKeep` with `keep == nil`), so the lane holds the generation's LTX files flat under `replica/<level>/…` and nothing else. `TestReplicaSourceSelectsTheClaimsGeneration`: a location with two incarnations, a `g-old/` generation and the live `g-cur/`: only `g-cur/`'s level files are copied; a legacy root location with `g-x/` beside its level directories copies the level directories only. |
| -- The round, per in-flight record this box owns, in order (§7.3, K5, K12), **after** taking the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`) and **then** re-reading the claim under it (§7.3: every round takes the lease first, then revalidates the claim's op id and phase): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; `hostclaim.RequireMoveOp(id, KindMoving, "")` on the claim read under the lease and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); `move.json`: `stagingMirror.PutIfAbsentVersioned(move.json)` through the guarded staging mirror (Task 6; **every** write into the target bucket, `move.json`, `replica/`, `files/`, `timers/` and Task 11's `probe/` measurement object, goes through the one guarded path: `guardedMirror` wraps `CopyVersioned`, `PutVersioned`, `PutIfAbsentVersioned` and `Delete` in `Guarded.Before`; nothing calls `lease.VersionedStore` against the target bucket); success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefixKeep(ReplicaSource(holder) → ReplicaPrefix, keep)` through the guarded staging mirror, whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| +- **The source copy is bound to the claim's location and generation** (round-3 finding 5), never to `HibernatePrefix(cfg.OwnPrefix, host)`: `moves.ReplicaSource(h hostclaim.Holder) (prefix string, keep func(rel string) bool)`: `base := strings.TrimSuffix(h.Location, "/") + "/"` first (`Claim.Validate` accepts only a location WITHOUT a trailing slash, `claim.go:245`, and `genselect.requestLocation` adds the separator the same way, `select.go:160-166`; a naive join would yield `instances/h/<inc>g-cur/` and an empty listing, round-4 finding 4); for generation `""` or `hostclaim.RootGeneration`, `prefix = base` and `keep` admits only `<level 0000-0009>/<min>-<max>.ltx` directly under it (not `g-*/`, not a 64-hex child incarnation, not `generation.json`, `lock.json` or `closed/`); for a generation `g`, `prefix = base + "g-" + g + "/"` and `keep` admits its level directories. `lease.MirrorPrefix` gains a variant `lease.MirrorPrefixKeep(ctx, m, srcBucket, srcPrefix, dstBucket, dstPrefix, rc, keep func(rel string) bool)` (`MirrorPrefix` is `MirrorPrefixKeep` with `keep == nil`), so the lane holds the generation's LTX files flat under `replica/<level>/…` and nothing else. `TestReplicaSourceSelectsTheClaimsGeneration`: the holder comes from a claim written through `hostclaim.Update` (so its location is the validated `instances/<host>/<inc>` form, no trailing slash) and the asserted prefix is `instances/<host>/<inc>/g-cur/`; a location with two incarnations, a `g-old/` generation and the live `g-cur/`: only `g-cur/`'s level files are copied; a legacy root location (`instances/<host>`) with `g-x/` beside its level directories copies the level directories only. |
| +- The round, per in-flight record this box owns, in order (§7.3, K5, K12), **after** taking the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`) and **then** re-reading the claim under it (§7.3: every round takes the lease first, then revalidates the claim's op id and phase): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; refuse likewise (S1, M-C17's sentence) a source whose store binding exists: `objstore.GetIn(ctx, s, acct, app, instanceStoreEnv(rec))` non-nil, the door's check (Task 4) repeated every round before the first staging write, a read error ending the round; `hostclaim.RequireMoveOp(id, KindMoving, "")` on the claim read under the lease and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); `move.json`: `stagingMirror.PutIfAbsentVersioned(move.json)` through the guarded staging mirror (Task 6; **every** write into the target bucket, `move.json`, `replica/`, `files/`, `timers/` and Task 11's `probe/` measurement object, goes through the one guarded path: `guardedMirror` implements `lease.MirrorStore`, `VersionedCopier` and `VersionedPutter` over `*lease.StagingMirror` by calling `Guarded.Before` and passing its `signAt` to `CopyVersionedAt`, `PutVersionedAt`, `PutIfAbsentVersionedAt` and `DeleteAt`, Task 6; nothing calls `lease.VersionedStore` against the target bucket); success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefixKeep(ReplicaSource(holder) → ReplicaPrefix, keep)` through the guarded staging mirror, whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record both keys and both returned versions in the ledger's `archive_versions` and publish them as the source report's `Archive`; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MoveGuardedPromoteReady(g, now)` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveGuardedCut(g, now)` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| - Receipts: `lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(host, id)}`; a per-op `moveLedger` JSON beside it at `move-receipts/<host>/<id>-moves.json` records `move_json_version`, `archive_plain`, `archive_sealed`, `archive_versions`. |
| |
| Tests: the legacy tests (`TestMovesPassPrepareAndPromote`, every `TestMovesPassCut…`, `TestMovesPassAbort…`) stay as they are and run with the switch off; the new tests run the harness with `cfg.Enabled = true` and a `MoveStart`ed record: `TestPhase1PrepareAndPromote` (keys under `moves/<host>/<id>/replica/<level>/…`, `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store), `TestPhase1IsInertWithTheSwitchOff` (an id'd record and the switch off: no copy, no CAS, one log line), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`; the target-side `TestTargetReportPassFlagsEachCheck` has the matching case: a `move.json` naming another move id fails T9 with the lane sentence), `TestInterimAbortReleasesTheClaim`, `TestRoundNeverAdvancesAReplacementMove`, `TestReplicaSourceSelectsTheClaimsGeneration`, `TestEveryStagingWriteIsGuarded` (a `Guarded` whose `Now` is past the bound: the round issues no request of any class, `move.json`, replica copy, archive, timers or probe object), `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| @@ -3365,11 +3560,11 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| ### Task 15: The copy gate |
| |
| -**Files:** modify `approved_copy_test.go`; the record `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (already on the branch; this task makes the test read it). |
| +**Files:** modify `approved_copy_test.go`; the record `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`. The record lives on the design branch `design-moves-any-region`, not on `main` (round-4 finding 14): this task's first step copies it into the implementation branch (`git show <design sha>:docs/superpowers/plans/2026-10-06-moves-phase1-copy.md > docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`, the design sha named in the commit message) and the gate reads that committed copy; a record missing from the tree fails `TestMovesCopyRecordIsComplete`. |
| |
| The gate exists from this plan on, so every later plan's strings are checked the day they are written (index D12). Two checks, modelled on the existing `TestApprovedCopyIsApplied` (`approved_copy_test.go:87`), which compares the record against a **separately pinned inventory** so a deleted row is caught: |
| |
| -- `movesCopyIDs` is a literal in the test file: `M-C1` to `M-C20` and every `MV-N<n>` id the plans introduce (each plan's "Gate and copy" task appends to this literal as it appends rows to the record). `TestMovesCopyRecordIsComplete` fails when the record lacks an id the literal names, or names an id the literal does not. |
| +- `movesCopyIDs` is a literal in the test file: `M-C1` to `M-C22` (the record's spec rows as of spec rev 7.21; M-C21 and M-C22 arrived with revs 7.20 and 7.21) and every `MV-N<n>` id the plans introduce (each plan's "Gate and copy" task appends to this literal as it appends rows to the record). `TestMovesCopyRecordIsComplete` fails when the record lacks an id the literal names, or names an id the literal does not. |
| - `TestMovesApprovedCopyIsApplied`: every `approved` row's text appears byte for byte in the file the row names (as the existing test checks, through string literals for short texts). |
| - `TestPendingCopyKeepsTheSwitchOff` (D13): while any row of the record is `pending`, `moves.DefaultEnabled` (Task 0's compiled default of `CARLOS_MOVES_PHASE1`) must be `false`, so no pending sentence can reach an operator through a default deployment; the day Paul approves the record, one commit flips the default and this test's condition inverts (every row `approved`). Pending strings may be wired as the tasks are built; the gate is on enablement, not on each call site. |
| - Negative fixtures under `testdata/moves-copy/`: a record missing an id, a row whose text changed after approval, a record with a pending row beside a `DefaultEnabled = true`; each is loaded by a table test that expects the corresponding failure. |
amadan