| index 4b1bcb60a..2d81397ae 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| @@ -8,9 +8,9 @@ |
| |
| **Tech Stack:** Go 1.25, modernc sqlite, litestream v0.5.14 and superfly/ltx v0.5.1 (pinned), AWS SDK v2 S3, `github.com/klauspost/compress/zstd` (new, pure Go), the repository's `release.Store`, `lease`, `hostclaim`, `genselect`, `registry`, `provision`, `appconfig` packages. |
| |
| -**Spec:** `docs/superpowers/specs/2026-10-02-move-any-app-design.md` rev 7.21: §3, §4, §6.1, §6.2, §7.1, §7.2, §7.3, §8, §10, §11, §12 (door), §19.2 K2, K5, K12. Index: `2026-10-06-moves-phase1-0-index.md`. |
| +**Spec:** `docs/superpowers/specs/2026-10-02-move-any-app-design.md` rev 7.22: §3, §4, §6.1, §6.2, §7.1, §7.2, §7.3, §8, §10, §11, §12 (door), §19.2 K2, K5, K12. Index: `2026-10-06-moves-phase1-0-index.md`. |
| |
| -**Plan revision:** rev 5 (after the round-4 Plan 1 review, `CODEX-PLAN1-REVIEW-R4.md`; dispositions in the index). Every API this plan names was checked against `origin/main` at 1c43c2ce: `boxchoice.go`'s signatures and the absence of a production caller, `Claim.Validate`'s location grammar, the two system units' direct `carlos.prev` exec, the edge unit's `ProtectSystem=strict`, `edge.go`'s cache-before-activator order, `agentOwnedVars` and `startChild`'s unconditional `ListenControl`, `LoadMergedHost`'s single etag, `objstore.GetIn`, `leasetest`'s create-only `PUT`, `cgroup.Prepared`'s method set and `s3Mirror`'s `Copy`. |
| +**Plan revision:** rev 6 (after the round-5 Plan 1 review, `CODEX-PLAN1-REVIEW-R5.md`; rev 5 answered round 4; both dispositions in the index). Every API this plan names was checked against `origin/main` at 1c43c2ce: `boxchoice.go`'s signatures and the absence of a production caller, `Claim.Validate`'s location grammar, the two system units' direct `carlos.prev` exec, the edge unit's `ProtectSystem=strict`, `edge.go`'s cache-before-activator order, `agentOwnedVars` and `startChild`'s unconditional `ListenControl`, `LoadMergedHost`'s single etag, `objstore.GetIn`, `leasetest`'s create-only `PUT`, `cgroup.Prepared`'s method set, `s3Mirror`'s `Copy`, `stageBlocked`'s error classes (`adopt.go:250`), the existing `LiveVersion` (`versions.go:290`), `decidePrev`'s `.prev version` call, `MirrorPrefix`'s two receipt branches and `GetRecord`'s `*Record`. |
| |
| ## Global Constraints |
| |
| @@ -141,11 +141,11 @@ In `server.go`, `Config.MovesEnabled bool` with the doc "the Phase 1 moves switc |
| ### Task 1: Operation ids |
| |
| **Files:** |
| -- Create: `internal/moves/ids.go`, `internal/moves/check.go` |
| -- Test: `internal/moves/ids_test.go` |
| +- Create: `internal/moves/ids.go`, `internal/moves/check.go`, `internal/moves/floor.go` |
| +- Test: `internal/moves/ids_test.go`, `internal/moves/check_test.go`, `internal/moves/floor_test.go` |
| |
| **Interfaces:** |
| -- Produces: `moves.NewMoveID(r io.Reader) (string, error)`, `moves.NewRollbackID(r io.Reader) (string, error)`, `moves.ValidOpID(id string) bool`, `moves.OpIDKind(id string) string` ("move" | "rollback" | ""); and, in `check.go`, `moves.Check{OK bool; Reason string}`, the shared result type Tasks 7b, 11 and 14 consume. It lives here, in the first task of the package, so Task 7b's `NextSpawnStrict(host) moves.Check` compiles before Task 11 exists (round-4 finding 2); `TestCheckZeroValueFails` pins that a zero `Check` is not OK. |
| +- Produces: `moves.NewMoveID(r io.Reader) (string, error)`, `moves.NewRollbackID(r io.Reader) (string, error)`, `moves.ValidOpID(id string) bool`, `moves.OpIDKind(id string) string` ("move" | "rollback" | ""); and, in `check.go`, `moves.Check{OK bool; Reason string}`, the shared result type Tasks 7b, 11 and 14 consume. It lives here, in the first task of the package, so Task 7b's `NextSpawnStrict(host) moves.Check` compiles before Task 11 exists (round-4 finding 2); `TestCheckZeroValueFails` pins that a zero `Check` is not OK. Also here, in `floor.go` (moved from Task 13 in rev 6: Task 4's door reads them, which was a forward dependency): `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int \`json:"min"\`; MinRelease string \`json:"min_release"\`}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → `nil, nil`); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` (`golang.org/x/mod/semver` on `v`-prefixed strings; confirm `x/mod` is in `go.mod`, else `release`'s own comparison, grep `func Compare` there); `TestFloorReadAndCompare` (absent floor is nil; a written floor reads back; `v0.23.0`, `v0.24.1` at or above `v0.23.0`, `v0.22.9` below). |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -251,8 +251,8 @@ Expected: PASS |
| - [ ] **Step 5: Commit** |
| |
| ```bash |
| -git add internal/moves/ids.go internal/moves/ids_test.go |
| -git commit -m "moves: operation ids (spec §3) |
| +git add internal/moves/ids.go internal/moves/ids_test.go internal/moves/check.go internal/moves/check_test.go internal/moves/floor.go internal/moves/floor_test.go go.mod go.sum |
| +git commit -m "moves: operation ids, the Check result type and the protocol floor types (spec §3, §11) |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ``` |
| @@ -267,7 +267,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/provision/move_test.go` |
| |
| **Interfaces:** |
| -- Produces: the extended `provision.MoveState` below; `Record.LastMove *MoveOutcome \`json:"last_move,omitempty"\`` beside `Record.Move` (`provision.go:58`); `MoveGuard` and the guarded transitions `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbort`, `MoveGuardedAbortComplete` (below); the legacy `MoveRequest`, `MovePromoteReady`, `MoveMarkCutting`, `MoveAbort`, `MoveAbortComplete`, `MoveComplete` unchanged; state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`. |
| +- Produces: the extended `provision.MoveState` below; `Record.LastMove *MoveOutcome \`json:"last_move,omitempty"\`` beside `Record.Move` (`provision.go:58`); `MoveGuard` and the guarded transitions `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbort`, `MoveGuardedAbortComplete` (below); the legacy `MoveRequest`, `MovePromoteReady`, `MoveMarkCutting`, `MoveAbort`, `MoveAbortComplete`, `MoveComplete` unchanged; state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`; `provision.InstanceStoreEnv(rec *Record) string` (`rec.Environment`, or `rec.Slug` when it has none: the store name the create door stamps, `apiinstances.go:1074-1079`; `GetRecord` returns `*Record`, `provision.go:1128`; shared by the console door (Task 4) and the host round (Task 14), which cannot see a console-local helper, round-5 finding 1). |
| - **Shared contract type (coordination with claim Plan 6 Task 6, `feat/hostclaim-box` bf63e250):** `provision.MoveInputs` is defined by this task below AND by claim Plan 6 in `internal/provision/moveinputs_contract.go`, with exactly the same shape (`BundleETags map[string]string \`json:"bundle_etags,omitempty"\`; OverlayETag string \`json:"overlay_etag,omitempty"\`; Release string \`json:"release,omitempty"\`; FilesPlain string \`json:"files_plain,omitempty"\`; EnvReport string \`json:"env_report,omitempty"\``, verified field for field and tag for tag against bf63e250). Whichever lands on `main` first owns the type and the other imports it (the coordinator's rule). **If `internal/provision/moveinputs_contract.go` is on `main` when this task is built, do not redefine `MoveInputs`: delete it from the code block below and use the existing type.** If this task lands first, it owns the type and claim Plan 6 imports it. |
| - Later plans add the phase transitions (Plan 2 Task 1 the cut phases, Plan 2 Task 8 the landing phases, Plan 4 Task 1 the rollback phases). |
| |
| @@ -771,7 +771,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| |
| **Interfaces:** |
| - Consumes: `hostclaim.StartGuard`, `RequireOp`, `KindServing`, `KindMoving`, `Op`, `Touched`, `ErrWrongState`, `ErrMalformed`. |
| -- Produces: `hostclaim.BeginMove(opID, by, sourceBox, targetBox, targetRegion string, now time.Time) func(*Claim) error`; `hostclaim.Touch(opID string, regions, boxes []string) func(*Claim) error`; `hostclaim.RequireMoveOp(opID, kind, phase string) func(*Claim) error`; phase constants `PhaseMoving = ""`, `PhaseSealed = "sealed"`; `hostclaim.ReleaseMove(opID, execBox string) func(*Claim) error` (K8, in this plan so the interim abort can release: legal from `moving` with phase `""` or `"aborting"` when no obligation under the op remains, sets `Op = Op{Kind: KindServing, Exec: execBox}`; Plan 4 Task 2 widens it to `landing/committed`, `restoring/opened`, `reverting/opened`). Test `TestReleaseMoveFromMovingNeedsNoChores` beside the two below. |
| +- Produces: `hostclaim.BeginMove(opID, by, sourceBox, targetBox, targetRegion string, now time.Time) func(*Claim) error`; `hostclaim.Touch(opID string, regions, boxes []string) func(*Claim) error`; `hostclaim.RequireMoveOp(opID, kind, phase string) func(*Claim) error`; phase constants `PhaseMoving = ""`, `PhaseSealed = "sealed"`; `hostclaim.ReleaseMove(opID, execBox string) func(*Claim) error` (K8, in this plan so the interim abort can release: legal from `moving` with phase `""` or `"aborting"` when no obligation under the op remains, sets `Op = Op{Kind: KindServing, Exec: execBox, Journal: map[string]string{"released_move": opID}}`: the serving op remembers which move released it, **operation-bound completion evidence** for the interim abort's crash boundary (round-5 finding 10; `Claim.Validate` accepts an op journal on any kind, `claim.go:89`; if it does not for `serving`, widen it in this task); `hostclaim.RequireReleasedMove(opID string) func(*Claim) error` (nil when the claim is `serving` and `Op.Journal["released_move"] == opID`, else `ErrOpMismatch`), the replay rule Task 14 uses; `TestReleaseMoveLeavesEvidence` (after `ReleaseMove`, `RequireReleasedMove(id)` holds and `RequireReleasedMove(other)` does not; a plain serving claim satisfies neither); Plan 4 Task 2 widens it to `landing/committed`, `restoring/opened`, `reverting/opened`). Test `TestReleaseMoveFromMovingNeedsNoChores` beside the two below. |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -789,10 +789,11 @@ import ( |
| // holder's key, incarnation and location follow the grammar Validate |
| // enforces (claim.go:236-247: the key is this host's, the incarnation is |
| // Incarnation(account, app, host, createdAt), the location is exactly |
| -// "instances/<host>/<inc>" with no trailing slash), built with the same |
| -// helpers delete_test.go's servingClaim uses, and the fixture asserts |
| -// Validate() so it can never drift from the real grammar (round-4 |
| -// findings 4 and 9). |
| +// "instances/<host>/<inc>" with no trailing slash; the key is the record |
| +// key grammar create_test.go:14 uses, which holderKeyMatches checks), and |
| +// the fixture asserts Validate() so it can never drift from the real |
| +// grammar (round-4 findings 4 and 9; round-5 finding 1: no undefined |
| +// helper). |
| func movingTestClaim(t *testing.T) *Claim { |
| t.Helper() |
| created := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC) |
| @@ -801,7 +802,7 @@ func movingTestClaim(t *testing.T) *Claim { |
| t.Fatal(err) |
| } |
| c := &Claim{V: 1, Host: "a.go.example.com", Gen: 3, |
| - Holder: &Holder{Family: "instance", Key: testHolderKey("bab", "app", "a.go.example.com"), Account: "bab", App: "app", Incarnation: inc, CreatedAt: created, Location: "instances/a.go.example.com/" + inc, Stamp: 3}, |
| + Holder: &Holder{Family: "instance", Key: "apps/bab/app/instances/a.go.example.com.json", Account: "bab", App: "app", Incarnation: inc, CreatedAt: created, Location: "instances/a.go.example.com/" + inc, Stamp: 3}, |
| Op: Op{Kind: KindServing, Exec: "ie-box"}, |
| Touched: Touched{Complete: true, Regions: []string{"ie"}, Boxes: []string{"ie-box"}}} |
| if err := c.Validate(); err != nil { |
| @@ -957,6 +958,8 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ### Task 4: The request door: operator-only, refusals, the claim CAS |
| |
| **Files:** |
| +- Create: `internal/boxid/boxidtest/boxidtest.go` (the exported form of `boxid_test.go`'s `newSigner`/`record`/`verifier`: `boxidtest.NewSigner(t) *Signer`, `(*Signer).Record(t, box, account, region, instance) (boxid.Record, []byte)`, `(*Signer).Certs() map[string]*x509.Certificate`; `boxid_test.go` is rewritten to use it, behaviour unchanged), `internal/console/boxselect.go` (`selectMoveBoxes`, `verifiedBoxes`), `internal/console/boxselect_test.go` |
| +- Modify: `internal/console/server.go` (`Config.AWSAccount`, `Server.boxCerts` loaded from `boxid.EmbeddedCerts()` at construction), `cmd/console/main.go` (`AWSAccount: os.Getenv("CARLOS_AWS_ACCOUNT")`) |
| - Modify: `internal/console/apiinstancemove.go` (whole `handleAPIInstanceMoveRequest`, `moveGate`, new `moveOperatorGate`), `internal/console/authcli.go:633-636` (routes) |
| - Test: `internal/console/apiinstancemove_test.go` (extend `TestAPIInstanceMoveVerbs`; add `TestMoveDoorRefusesWithoutClaimWrite`, `TestMoveMutationsAreOperatorOnly`) |
| |
| @@ -964,7 +967,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Consumes: `hostclaim.Read`, `hostclaim.Update`, `hostclaim.BeginMove`, `hostclaim.Incarnation`, `moves.NewMoveID`, `s.store.IsOperator`, `s.store.CLITokenAddress`, `provision.GetManifest` (regions and `PrivateStorage`), `s.cfg.MovesEnabled` (Task 0), `eligibleBoxes` and `chooseBox` (`boxchoice.go`). This task creates `internal/moves/floor.go` with the one constant `const ProtocolVersion = 1` and adds `MovesProtocol int \`json:"moves_protocol"\`` to the console's edges struct in `instancesplacement.go`; Task 13 fills the file (the floor object, the box's writer, the activation refusal). |
| - Produces: `(s *Server) moveOperatorGate(w, r) (acct *Account, app, host string, rec *provision.Record, addr string, ok bool)`; the door's refusal sentences (copy ids below). |
| |
| -The door's order (§7.1, K2): `moveGate` (class rails), operator (`IsOperator`, 404 otherwise), body, region validity and `placements`, the manifest's `regions` when non-empty, a home with replicas (`len(rec.Replicas) > 0`), private storage (the instances manifest's `PrivateStorage`; the refusal is M-C17), then the claim: `hostclaim.Read`; absent or `Holder == nil` → 409 "the instance has no host claim yet. it is adopted on its next reconcile pass"; holder incarnation ≠ `hostclaim.Incarnation(acct, app, host, rec.CreatedAt)` → 409; `Op.Kind != KindServing` → 409 naming the kind; the target box: the lexically-first box publishing the target placement in `readEdgesFiles` (T8 is the target's own check); `hostclaim.Update(ctx, s.cfg.Releases, obs, hostclaim.BeginMove(id, addr, obs.Claim.Op.Exec, targetBox, to, now))`; then `provision.SetRecordMove(... MoveStartTransition ...)`. If the record write fails after the claim CAS, the door answers 502 and logs; the console pass of Plan 5 Task 7 finds a `moving` claim with no record move and releases it (noted there). |
| +The door's order (§7.1, K2): `moveGate` (class rails), operator (`IsOperator`, 404 otherwise), body, region validity and `placements`, the manifest's `regions` when non-empty (the manifest read fails closed, 503), a home with replicas (`len(rec.Replicas) > 0`), private storage (the instance's store binding, `objstore.GetIn` under `provision.InstanceStoreEnv(rec)`; the refusal is M-C17; a read error is 503), the fleet: `selectMoveBoxes` (the live snapshot, verified identities, exactly one eligible box in each of the two regions, the floor; M-C21 for a second box; 400 for a region with none), then the claim: `hostclaim.Read`; absent or `Holder == nil` → 409 "the instance has no host claim yet. it is adopted on its next reconcile pass"; holder incarnation ≠ `hostclaim.Incarnation(acct, app, host, rec.CreatedAt)` → 409; `Op.Kind != KindServing` → 409 naming the kind; `Op.Exec != sel.Source` → 409 "the source box <exec> is not the eligible box of <from>" (§7.1 binds the source to the claim's executor, round-5 finding 7); `hostclaim.Update(ctx, s.cfg.Releases, obs, hostclaim.BeginMove(id, addr, obs.Claim.Op.Exec, targetBox, to, now))`; then `provision.SetRecordMove(... MoveStartTransition ...)`. If the record write fails after the claim CAS, the door answers 502 and logs; the console pass of Plan 5 Task 7 finds a `moving` claim with no record move and releases it (noted there). |
| |
| Copy for the gate (all pending review, Plan 6 Task 7): M-C17 "<host> uses private storage. Instances with private storage cannot move yet."; new strings "a home with replicas does not move. remove its replicas first", "region <to> is not one of this app's regions: <list>", "the instance has no host claim yet. it is adopted on its next reconcile pass", "the instance is under a <kind> operation. wait for it to finish". |
| |
| @@ -1020,8 +1023,16 @@ func TestMoveDoorRefusesWithoutClaimWrite(t *testing.T) { |
| t.Fatal(err) |
| } |
| } |
| - put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"data-uk","data_bucket_region":"eu-west-2","hosts":{},"moves_protocol":1}`) |
| - put("control/edges/se-box.json", `{"v":1,"box":"se-box","eip":"1.2.3.5","placement":"se","data_bucket":"data-se","data_bucket_region":"eu-north-1","hosts":{},"moves_protocol":1}`) |
| + // A production-shaped fleet (round-5 finding 6): selectMoveBoxes needs |
| + // a COMPLETE live snapshot and verified identities, so every box gets |
| + // an edges file with every eligibility field, a fresh status snapshot |
| + // and a signed boxid record; the claimless case below then reaches its |
| + // claim refusal instead of an incomplete-fleet 409. |
| + put("control/moves-protocol.json", `{"min":1,"min_release":"v0.23.0"}`) |
| + fleetFixture(t, fx, |
| + fleetBox{Box: "ie-box", Placement: "ie", AWS: "eu-west-1", EIP: "1.2.3.3", Bucket: "data-ie"}, |
| + fleetBox{Box: "uk-box", Placement: "uk", AWS: "eu-west-2", EIP: "1.2.3.4", Bucket: "data-uk"}, |
| + fleetBox{Box: "se-box", Placement: "se", AWS: "eu-north-1", EIP: "1.2.3.5", Bucket: "data-se"}) |
| moveURL := u + "/acme.go.tito.io/move" |
| claimKey := "control/hosts/acme.go.tito.io.json" |
| // No claim yet → 409, no claim written. |
| @@ -1067,7 +1078,7 @@ func TestMoveDoorRefusesWithoutClaimWrite(t *testing.T) { |
| } |
| ``` |
| |
| -`operatorInstanceFixture` and `writeServingClaim` are test helpers added to `apiinstancemove_test.go`; `operatorInstanceFixture` calls `instanceFixture` then marks the token's address an operator with `fx.st.SetOperator(addr, true)` (`store.go:1952`, the call `alertwatch_test.go:52` and `account_test.go:250` already make), where `addr` is the address `instanceFixture` signed the token for. |
| +`operatorInstanceFixture` and `writeServingClaim` are test helpers added to `apiinstancemove_test.go`; `operatorInstanceFixture` calls `instanceFixture` then marks the token's address an operator with `fx.st.SetOperator(addr, true)` (`store.go:1952`, the call `alertwatch_test.go:52` and `account_test.go:250` already make), where `addr` is the address `instanceFixture` signed the token for. `fleetFixture(t, fx, boxes ...fleetBox)` (in `boxselect_test.go`, shared by the door tests and the selector test) makes the fleet production-shaped: it generates a secrets key pair and sets `fx.srv.cfg.SecretsPub`, sets `fx.srv.cfg.AWSAccount = "111122223333"` and `fx.srv.boxCerts = signer.Certs()` from `boxidtest.NewSigner(t)`, and for each box writes (1) the edges file `control/edges/<box>.json` with `v`, `box`, `box_id` (= box), `eip`, `region` (the AWS region), `placement`, `data_bucket`, `data_bucket_region`, `hibernation_configured: true`, `serves_shared_pool: true`, `accepts_inbound: true`, `secrets_key_fp` (the pair's fingerprint), `capabilities: ["region-ha-1"]`, `hosts: {}`, `moves_protocol: 1`, `moves_floor_min_release` (the published floor's, when `fleetBox.FloorArmed` is not set to false) and `prev_release: "none"` (overridable per box); (2) a fresh status snapshot through `status.WriteBox(ctx, fx.rel, status.Box{Label: box, GeneratedAt: now})` (`accountfleet_test.go:34` makes the same call), which is what `readLiveEdgesSnapshot` joins on; (3) the signed identity record `signer.Record(t, box, "111122223333", aws, box)` written with `boxid.Publish`. `fleetBox{Box, Placement, AWS, EIP, Bucket, PrevRelease string; Unverified, Stale bool}`: `Unverified` skips the identity record, `Stale` writes an old snapshot. |
| |
| `TestAPIInstanceMoveVerbs` stays as it is (the legacy door, switch off). Add `TestAPIInstanceMoveVerbsPhase1` with `fx.srv.cfg.MovesEnabled = true`, `operatorInstanceFixture`, the `moves_protocol` field on its edges file and a serving claim written before the first request; it mirrors the legacy test's matrix with the new rules: a second request while in flight is 409; abort from `cutting/requested` is **202** (the guarded abort allows it, Task 2); after an abort completes the test releases the claim (`hostclaim.Update(ReleaseMove(id, "ie-box"))`, Task 3) before the next request, because a request against a `moving` claim is 409 by design. |
| |
| @@ -1174,7 +1185,7 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| // policy: the create door stamps a store per instance under its |
| // environment name, or its slug when it has none (apiinstances.go: |
| // 1074-1079), and turning the policy off later removes no store. |
| - if store, _, err := objstore.GetIn(ctx, s.cfg.Releases, acct.Sqid, app, instanceStoreEnv(rec)); err != nil { |
| + if store, _, err := objstore.GetIn(ctx, s.cfg.Releases, acct.Sqid, app, provision.InstanceStoreEnv(rec)); err != nil { |
| jsonError(w, http.StatusServiceUnavailable, "could not read this instance's storage binding. try again") |
| return |
| } else if store != nil { |
| @@ -1205,6 +1216,10 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| jsonError(w, http.StatusBadRequest, "no box in "+to+" publishes an edges file yet") |
| return |
| } |
| + if sel.Source == "" { |
| + jsonError(w, http.StatusBadRequest, "no eligible box in "+s.homeRegionOut(rec.Region)+" publishes an edges file") |
| + return |
| + } |
| targetBox := sel.Target |
| if !sel.FloorOK { |
| jsonError(w, http.StatusConflict, "a box in "+rec.Region+" or "+to+" is below the moves protocol floor. roll the platform first") |
| @@ -1253,9 +1268,10 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| |
| // moveSelection is the door's answer about the two regions of a move. |
| type moveSelection struct { |
| - Target string // the chosen box in `to`; "" when none is eligible |
| + Target string // the one eligible box in `to`; "" when none |
| + Source string // the one eligible box in `from`; "" when none (the door requires it to equal the claim's Op.Exec) |
| Many string // the region (to or from) with more than one eligible box; "" when neither |
| - FloorOK bool // every edges file of either region publishes moves_protocol >= moves.ProtocolVersion |
| + FloorOK bool // both regions have files; every file publishes moves_protocol >= ProtocolVersion and, when a floor is published, moves_floor_min_release == floor.MinRelease and prev_release "none" or at or above it |
| } |
| |
| var errEdgesIncomplete = errors.New("console: the fleet's edges files could not all be read") |
| @@ -1295,16 +1311,30 @@ var errEdgesIncomplete = errors.New("console: the fleet's edges files could not |
| // (D15). Region HA owns the hash tie-break (L15, P5-D2); the one-box |
| // refusal is what makes that safe here. |
| // |
| -// The floor (§11, K10) is read from the same snapshot: every file of |
| -// either region must publish moves_protocol >= moves.ProtocolVersion |
| -// (Task 13 adds the field; absent reads as 0, below any floor). An |
| -// incomplete snapshot answers nothing: a box whose file could not be read |
| -// may be below the floor or may be the second box of a region. |
| +// Both regions are selected (round-5 finding 7): §7.1 requires the source |
| +// to be the one eligible box of its region, and the door then requires |
| +// the claim's Op.Exec to be that box. |
| +// |
| +// The floor (§11, K10, Task 13; round-5 finding 3): both regions must |
| +// have at least one file; every file of either region must publish |
| +// moves_protocol >= moves.ProtocolVersion (Task 1; absent reads as 0) and, |
| +// when control/moves-protocol.json is published, moves_floor_min_release |
| +// equal to its min_release (the box has armed its local floor files, so |
| +// its units already select the recovery executable by the floor) and |
| +// prev_release either "none" or at or above min_release (the box's |
| +// carlos.prev is not below the floor). That is what makes the admission |
| +// safe without any later asynchronous step on the boxes. An incomplete |
| +// snapshot answers nothing: a box whose file could not be read may be |
| +// below the floor or may be the second box of a region. |
| func (s *Server) selectMoveBoxes(ctx context.Context, to, from, app string) (moveSelection, error) { |
| snap, err := s.liveEdges(ctx) |
| if err != nil || !snap.complete() { |
| return moveSelection{}, errEdgesIncomplete |
| } |
| + floor, err := moves.ReadFloor(ctx, s.cfg.Releases) |
| + if err != nil { |
| + return moveSelection{}, err |
| + } |
| aws := map[string]string{} |
| for _, c := range s.regionCands(snap) { |
| aws[c.Code] = c.AWSRegion |
| @@ -1312,10 +1342,27 @@ func (s *Server) selectMoveBoxes(ctx context.Context, to, from, app string) (mov |
| verified := s.verifiedBoxes(ctx, snap, to, from) |
| fp := s.expectedSecretsFP() |
| sel := moveSelection{FloorOK: true} |
| + seen := map[string]bool{} |
| for _, f := range snap.Files { |
| - if (f.Placement == to || f.Placement == from) && f.MovesProtocol < moves.ProtocolVersion { |
| + if f.Placement != to && f.Placement != from { |
| + continue |
| + } |
| + seen[f.Placement] = true |
| + if f.MovesProtocol < moves.ProtocolVersion { |
| sel.FloorOK = false |
| } |
| + if floor != nil { |
| + prevOK := f.PrevRelease == "none" |
| + if !prevOK && f.PrevRelease != "" { |
| + prevOK, _ = moves.ReleaseAtOrAbove(f.PrevRelease, floor.MinRelease) |
| + } |
| + if f.MovesFloorMinRelease != floor.MinRelease || !prevOK { |
| + sel.FloorOK = false |
| + } |
| + } |
| + } |
| + if !seen[to] || !seen[from] { |
| + sel.FloorOK = false |
| } |
| for _, code := range []string{to, from} { |
| cands := eligibleBoxes(snap, verified, code, aws[code], s.cfg.HomeRegion, fp, nil) |
| @@ -1323,9 +1370,11 @@ func (s *Server) selectMoveBoxes(ctx context.Context, to, from, app string) (mov |
| sel.Many = code |
| return sel, nil |
| } |
| - if code == to { |
| - if c, ok := chooseBox(cands, nil, app); ok { |
| + if c, ok := chooseBox(cands, nil, app); ok { |
| + if code == to { |
| sel.Target = c.BoxID |
| + } else { |
| + sel.Source = c.BoxID |
| } |
| } |
| } |
| @@ -1333,7 +1382,7 @@ func (s *Server) selectMoveBoxes(ctx context.Context, to, from, app string) (mov |
| } |
| ``` |
| |
| -`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 13 adds the writer's side). Private storage is the instance's store binding (round-4 finding 13): `instanceStoreEnv(rec)` is `rec.Environment`, or `rec.Slug` when it has none, exactly the name the create door stamps (`apiinstances.go:1074-1079`), and `objstore.GetIn` (`objstore.go:236`) answers `nil` for no store; a read error is a 503, never a pass. The manifest's `PrivateStorage` policy is not consulted: it is mutable and create-time (`apiinstances.go:1960-1966`), so it says nothing about an existing instance. Task 14's round makes the same `GetIn` read before its first staging write and fails S1 with M-C17's sentence when a store has appeared since the request (`carlos store create --environment`). Tests: `TestMoveDoorRefusesABoundStoreNotAPolicy` (a record with a store under its slug and the policy off → M-C17 and no claim written; the policy on and no store → admitted; the store read failing → 503 and no claim written; the manifest read failing → 503). `TestSelectMoveBoxesComposesTheCreateDoorsInputs` builds a live snapshot with two verified boxes in `uk` (M-C21 names `uk`), one verified box in `uk` and one unverifiable (its `boxid` record missing: the one box is chosen), a box whose edges file lacks `moves_protocol` (`FloorOK` false), and an incomplete snapshot (`errEdgesIncomplete`); the fixtures are `boxid.Publish`ed records signed with the test certificate `boxid_test.go` uses, so `Verifier.Verify` runs for real. |
| +`edgesFilePlacementView` (`instancesplacement.go:38`, what `readEdgesFiles` decodes) gains `MovesProtocol int \`json:"moves_protocol"\`` (Task 13 adds the writer's side). Private storage is the instance's store binding (round-4 finding 13): `instanceStoreEnv(rec)` is `rec.Environment`, or `rec.Slug` when it has none, exactly the name the create door stamps (`apiinstances.go:1074-1079`), and `objstore.GetIn` (`objstore.go:236`) answers `nil` for no store; a read error is a 503, never a pass. The manifest's `PrivateStorage` policy is not consulted: it is mutable and create-time (`apiinstances.go:1960-1966`), so it says nothing about an existing instance. Task 14's round makes the same `GetIn` read before its first staging write and fails S1 with M-C17's sentence when a store has appeared since the request (`carlos store create --environment`). Tests: `TestMoveDoorRefusesABoundStoreNotAPolicy` (a record with a store under its slug and the policy off → M-C17 and no claim written; the policy on and no store → admitted; the store read failing → 503 and no claim written; the manifest read failing → 503). `TestSelectMoveBoxesComposesTheCreateDoorsInputs` (`boxselect_test.go`) builds a live fleet with `fleetFixture` (below) and checks: two verified boxes in `uk` → `Many == "uk"`; one verified box in `uk` and one unverifiable (its `boxid` record missing) → that one box is the target; no file in the source region → `Source == ""` and `FloorOK` false; a box whose file lacks `moves_protocol` → `FloorOK` false; a published floor and a box whose `prev_release` is below it, or whose `moves_floor_min_release` is absent → `FloorOK` false, `"none"` → true; an incomplete snapshot → `errEdgesIncomplete`. The door test adds: a claim whose `Op.Exec` is not the source region's eligible box → 409 and no claim written (round-5 finding 7). The fixtures are `boxid.Publish`ed records signed by `boxidtest.NewSigner(t)`, so `Verifier.Verify` runs for real. |
| |
| **The legacy door stays** (Task 0): today's bodies of `handleAPIInstanceMoveRequest`, `handleAPIInstanceMoveCut` and `handleAPIInstanceMoveAbort` are renamed `legacyHandleAPIInstanceMoveRequest` (and siblings) unchanged, and each new handler's first line is `if !s.cfg.MovesEnabled { s.legacyHandleAPIInstanceMove…(w, r); return }`; `handleAPIInstanceMoveResolve` answers 404 when the switch is off (the route did not exist before). The existing `TestAPIInstanceMoveVerbs` and `TestAPIInstanceMoveClassRails` run as they are with the switch off (no change in behaviour, which is the point); the new tests below set `fx.srv.cfg.MovesEnabled = true`. |
| |
| @@ -1347,7 +1396,7 @@ Expected: PASS |
| - [ ] **Step 5: Commit** |
| |
| ```bash |
| -git add internal/console |
| +git add internal/console cmd/console internal/boxid |
| git commit -m "console: the move request door is operator-only and CASes the claim to moving (§7.1, §12, K2) |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| @@ -1553,7 +1602,7 @@ type VersionedPutter interface { |
| } |
| ``` |
| |
| -`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: `versioned := m.(VersionedCopier)` decides the mode once; `round.Versions` is non-nil only when `versioned`. **The plain `MirrorStore` path is unchanged** (round-3 finding 4: `TestMirrorPrefix` requires a second identical round to copy nothing, and the legacy worker depends on it): the `staged && d.ETag == s.ETag` early-continue and the receipt-match branch keep their exact behaviour. **Only when `versioned`** does a receipt have to vouch for the version: in the equal-ETag branch, read the receipt; if `rcpt.SrcETag == s.ETag && rcpt.SrcSize == s.Size && rcpt.DstETag == d.ETag && d.Size == s.Size && rcpt.DstVersionID != ""`, record `round.Versions[r] = rcpt.DstVersionID` and continue; otherwise fall through to the copy (a copy whose receipt was lost, or a receipt from an earlier version of the key, is recopied, which yields a fresh version id; the stale version stays unreferenced). Replace the copy call with: |
| +`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: `versioned := m.(VersionedCopier)` decides the mode once; `round.Versions` is non-nil only when `versioned`. **The plain `MirrorStore` path is unchanged** (round-3 finding 4: `TestMirrorPrefix` requires a second identical round to copy nothing, and the legacy worker depends on it): the `staged && d.ETag == s.ETag` early-continue and the receipt-match branch keep their exact behaviour. **Only when `versioned`** does a receipt have to vouch for the version: in the equal-ETag branch, read the receipt; if `rcpt.SrcETag == s.ETag && rcpt.SrcSize == s.Size && rcpt.DstETag == d.ETag && d.Size == s.Size && rcpt.DstVersionID != ""`, record `round.Versions[r] = rcpt.DstVersionID` and continue; otherwise fall through to the copy (a copy whose receipt was lost, or a receipt from an earlier version of the key, is recopied, which yields a fresh version id; the stale version stays unreferenced). The **receipt-match branch** (`mirror.go:195-203`: `staged` with differing ETags, the multipart-source case, where the destination's ETag never equals the source's) gets the same rule (round-5 finding 8): when `versioned`, the match additionally requires `rcpt.DstVersionID != ""`, which is recorded in `round.Versions[r]`; a matching receipt without a version (written by the plain path, or before this task) falls through to the copy. `TestVersionedMirrorRecordsMultipartVersions`: a fake whose `CopyVersioned` answers a destination ETag different from the source's (as a multipart copy does) and a version id; round one copies and records every version; round two copies nothing and `Versions` still names every key; a receipt hand-written without `dst_version_id` is recopied exactly once and then carries one. Replace the copy call with: |
| |
| ```go |
| var etag, ver string |
| @@ -1806,7 +1855,7 @@ Expected: PASS |
| - [ ] **Step 5: Commit** |
| |
| ```bash |
| -git add internal/moves/lease.go internal/moves/lease_test.go |
| +git add internal/moves/lease.go internal/moves/lease_test.go internal/lease |
| git commit -m "moves: the staging-work lease (45 min TTL) and the SafeUntil guard; lease: AcquireTTL and the single-signature staging mirror (§7.3) |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| @@ -2309,7 +2358,17 @@ func Verify(sealed io.Reader, m Manifest, priv *ecdh.PrivateKey, b appconfig.Bin |
| } |
| sealedHash := sha256.New() |
| pr, pw := io.Pipe() |
| - go func() { pw.CloseWithError(Open(io.TeeReader(sealed, sealedHash), m.SealedKey, priv, b, pw)) }() |
| + done := make(chan error, 1) |
| + go func() { |
| + err := Open(io.TeeReader(sealed, sealedHash), m.SealedKey, priv, b, pw) |
| + pw.CloseWithError(err) |
| + done <- err |
| + }() |
| + // Every return, early or late, breaks the pipe so the producer's next |
| + // write fails and the goroutine exits, then waits for it: a Verify |
| + // that returned on an early mismatch would otherwise leave Open |
| + // blocked on the pipe for the rest of the archive (round-5 finding 12). |
| + defer func() { pr.CloseWithError(errVerifyAborted); <-done }() |
| zr, err := zstd.NewReader(pr) |
| if err != nil { |
| return err |
| @@ -2384,7 +2443,7 @@ func Verify(sealed io.Reader, m Manifest, priv *ecdh.PrivateKey, b appconfig.Bin |
| } |
| ``` |
| |
| -(`Extract` declares `var modes []modeFix` before the loop and calls `restoreModes(modes)` on `io.EOF` before returning nil. `Verify`'s `h` hashes the decoded tar, trailer included, which is exactly what `Build`'s `capWriter` hashed, since `capWriter` sits between the `tar.Writer` and the zstd writer; `Verify` also checks `m.Sealed` over the sealed bytes it read and drains both streams to EOF so `Open`'s terminal authentication result is the verdict.) Tests added: `TestExtractPreservesModes` (a `0555` directory holding a `0400` file round-trips to exactly those modes; `PlainDigest` of the extracted tree equals the original) and `TestVerifyRefusesWhatTheManifestDoesNotPromise` (a correctly sealed archive with a manifest whose `Plain` is wrong, whose entry list lacks a file, whose entry sha256 differs, plus a hand-built sealed archive containing a symlink entry, one over the cap, one whose last sealed byte is flipped and one whose final chunk is cut off (the last two return `Open`'s authentication error, never nil): each returns the named error; the good archive verifies, and a manifest whose `sealed` differs from the bytes is refused). |
| +(`Extract` declares `var modes []modeFix` before the loop and calls `restoreModes(modes)` on `io.EOF` before returning nil. `Verify`'s `h` hashes the decoded tar, trailer included, which is exactly what `Build`'s `capWriter` hashed, since `capWriter` sits between the `tar.Writer` and the zstd writer; `Verify` also checks `m.Sealed` over the sealed bytes it read and drains both streams to EOF so `Open`'s terminal authentication result is the verdict.) Tests added: `TestExtractPreservesModes` (a `0555` directory holding a `0400` file round-trips to exactly those modes; `PlainDigest` of the extracted tree equals the original) and `TestVerifyRefusesWhatTheManifestDoesNotPromise` (a correctly sealed archive with a manifest whose `Plain` is wrong, whose entry list lacks a file, whose entry sha256 differs, plus a hand-built sealed archive containing a symlink entry, one over the cap, one whose last sealed byte is flipped and one whose final chunk is cut off (the last two return `Open`'s authentication error, never nil): each returns the named error; the good archive verifies, and a manifest whose `sealed` differs from the bytes is refused) and `TestVerifyReleasesTheProducerOnEarlyMismatch` (a three-chunk archive and a manifest whose first entry's size is wrong: `Verify` returns the entry error and `runtime.NumGoroutine()` returns to its baseline within a second; `errVerifyAborted` is the package-level sentinel the deferred close uses). |
| |
| `seal.go`: |
| |
| @@ -2544,15 +2603,15 @@ Moved here from Plan 2 (its Tasks 4 and 7) because T3, T7 and S2 need the real t |
| |
| **Files:** |
| - Create: `internal/appconfig/renderheld.go`, `internal/appconfig/renderheld_test.go`, `internal/release/materializeheld.go`, `internal/release/materializeheld_test.go` |
| -- Modify: `internal/activator/backend_exec.go` (`Strict`, `prepareStrict`, `NextSpawnStrict`, the strict spawn path at `spawnInCgroup`), `internal/activator/backend_exec_test.go`, `internal/appconfig/instances.go` (`LoadMergedHostVersions`), `internal/host/agent.go` (`carlos adopt --stage-only` calls `release.ServeHeldStageRequests`), `internal/host/agent_test.go` |
| +- Modify: `internal/activator/backend_exec.go` (`Strict`, `prepareStrict`, `NextSpawnStrict`, the strict spawn path at `spawnInCgroup`), `internal/activator/backend_exec_test.go`, `internal/appconfig/instances.go` (`LoadMergedHostVersions`), `internal/release/adopt.go` (`StageBlocked` exported), `internal/host/agent.go` (`carlos adopt --stage-only` calls `release.ServeHeldStageRequests` with `heldStageAuthority`), `internal/host/heldstage.go`, `internal/host/heldstage_test.go` |
| |
| **Interfaces:** |
| -- `appconfig.LoadMergedHostVersions(ctx, s release.Store, acct, app, envName, host string) (*Manifest, MergedVersions, error)` with `MergedVersions{Default, Environment, Overlay string}`: the same three reads `LoadMergedHost` and `LoadMerged` make (`instances.go:202`, `environments.go:169`), through `getIfExistsAtETag` (`store.go:65`) so the etag of each object actually merged is returned with the bytes; `LoadMergedHost` becomes a wrapper that discards all but the overlay etag (behaviour unchanged, its tests untouched). A later independent read could not establish which versions were rendered (round-4 finding 12), so this is the one loader `RenderHeld` and T3 use. |
| +- `appconfig.LoadMergedHostVersions(ctx, s release.Store, acct, app, envName, host string) (*Manifest, MergedVersions, error)` with `MergedVersions{Default, Environment, Overlay string; Shadowed []string}` (`Shadowed`: the bundle variables the instance overlay overrides, collected in `applyInstanceOverlay`'s loop; T3 reports them, round-5 finding 9): the same three reads `LoadMergedHost` and `LoadMerged` make (`instances.go:202`, `environments.go:169`), through `getIfExistsAtETag` (`store.go:65`) so the etag of each object actually merged is returned with the bytes; `LoadMergedHost` becomes a wrapper that discards all but the overlay etag (behaviour unchanged, its tests untouched). A later independent read could not establish which versions were rendered (round-4 finding 12), so this is the one loader `RenderHeld` and T3 use. |
| - `appconfig.RenderHeld(ctx, s release.Store, priv *ecdh.PrivateKey, dataDir string, rt registry.Route, op string, box string) (HeldRender, error)` with `HeldRender{Path, SHA256 string; BundleETags map[string]string; OverlayETag string}` (`BundleETags` = `{"default": v.Default, "environment": v.Environment}`, `OverlayETag` = `v.Overlay`): `LoadMergedHostVersions(ctx, s, acct, app, rt.Environment, rt.Host)`, rendered through the same file writer `syncExecHost` uses (the file-writing half extracted into `renderExecConfig(rt, m, key, priv, dataDir) (path string, changed bool, err error)` and called from both), written whether or not it changed, and a move-bound ack through `WriteConfigAck` with three additive `ConfigAck` fields: `MoveOp string \`json:"move_op,omitempty"\``, `FileSHA256 string \`json:"file_sha256,omitempty"\``, `BundleETags map[string]string \`json:"bundle_etags,omitempty"\``. `SyncExec` keeps skipping non-ready rows (`execRouteBase`), so the two never race on one file. |
| -- `release.MaterializeHeld(ctx, s Store, optDir string, rt registry.Route, version string) (*Manifest, error)`: `ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, version, rt.Host)` (`adopt.go:1317`) then `PointLive(optDir, host, acct, version, app)` (`versions.go:267`, already exported), no restart, no bake-hold check: a held row runs exactly the pinned release. When this process cannot write `optDir` (`errors.Is(err, fs.ErrPermission)` from the stage or the symlink: the edge unit has `ProtectSystem=strict` and writes only `/data` and `/run/carlos`, `carlos-edge.service:35-37`), it returns `release.ErrStageBlocked` wrapping the cause, so the caller relays instead of failing (Task 11's T7, round-4 finding 6). `release.ServeHeldStageRequests(ctx, s Store, pub ed25519.PublicKey, optDir, dataDir string) error` is the root side of that relay: for every `<dataDir>/moves/held-stage/<host>.json` (`{acct, app, host, version, op}`), it runs `MaterializeHeld` and writes `<dataDir>/moves/held-staged/<host>.json` (`{version, op, at}`), then removes the request; `carlos adopt --stage-only` (the root pass `carlos-stage.service` runs when `/data/stage-requested` changes, `agent.go:3525`) calls it after its release adoption. `release.LiveVersion(optDir, host string) (string, error)` reads the `live/<host>` symlink's target version (readable from the sandboxed agent), which is the proof T7 trusts; the ack file is only a hint. |
| +- `release.MaterializeHeld(ctx, s Store, optDir string, rt registry.Route, version string) (*Manifest, error)`: `ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, version, rt.Host)` (`adopt.go:1317`) then `PointLive(optDir, host, acct, version, app)` (`versions.go:267`, already exported), no restart, no bake-hold check: a held row runs exactly the pinned release. When this process cannot write `optDir` (classified by main's own `stageBlocked`, `adopt.go:250`: `EROFS`, `EPERM`, `EACCES`; `ProtectSystem=strict` answers a write with `EROFS`, which is **not** `fs.ErrPermission`, round-5 finding 5; exported in this task as `release.StageBlocked(err error) bool`; the edge unit writes only `/data` and `/run/carlos`, `carlos-edge.service:35-37`), it returns `release.ErrStageBlocked` wrapping the cause, so the caller relays instead of failing (Task 11's T7, round-4 finding 6). `release.HeldStageRequest{Account, App, Host, Version, Op, Box string}` and `release.ServeHeldStageRequests(ctx, s Store, optDir, dataDir string, authorize func(HeldStageRequest) error) error` is the root side of that relay (round-5 finding 2). For every `<dataDir>/moves/held-stage/<name>.json` it (1) decodes the request and **validates every component before any filesystem effect**: `Host == <name>` and `ValidHostName(Host)` (letters, digits, `.` and `-` only: no `/`, no `..`, no NUL), `ValidAccount(Account)`, `ValidApp(App)`, `ValidVersion(Version)` (`versions.go:69`), `Op != ""`, `Box != ""`; a request failing any of these is removed with a log line and never joined into a path; (2) calls `authorize(req)` and on error removes the request with a log line; (3) runs `MaterializeHeld` and writes `<dataDir>/moves/held-staged/<host>.json` (`{version, op, at}`), then removes the request. `authorize` is the host package's closure `heldStageAuthority(reg *registry.Registry, store release.Store, box string)` (`internal/host/heldstage.go`; `release` never imports `hostclaim`): it reads the registry row for `req.Host` and requires `hold_op == req.Op` (Task 12) and the route's account and app equal to the request's; reads the claim (`hostclaim.Read`) and requires `Op.ID == req.Op`, `Op.Kind` in `{KindMoving, KindLanding}`, `Op.Intent["target"] == req.Box` and `req.Box == box` (this box); reads the record and requires its pinned release (`rec.Move.Inputs.Release`, the version T7 is checking) equal to `req.Version`; anything else is `ErrStaleHeldStage`. So a request queued for operation A cannot run after A is aborted or replaced: the claim no longer names A, and `live/<host>` is never repointed to A's build. `carlos adopt --stage-only` (the root pass `carlos-stage.service` runs when `/data/stage-requested` changes, `agent.go:3525`) calls it after its release adoption, with `heldStageAuthority`. There is no `pub` parameter: `ensureStaged` (`adopt.go:1317`) is main's own staging step and verifies what the adopt pass verifies at that layer. `release.LiveVersion(optDir, host string) (string, error)` **already exists on main** (`versions.go:290`) and is reused: it reads the `live/<host>` symlink's target version (readable from the sandboxed agent), which is the proof T7 trusts; the ack file is only a hint. |
| - `ExecBackend.Strict func(host string) bool` (when true the spawn uses `b.prepareStrict(host, limit)` then `Start(cmd)` on what it returns, keeping it for the process's life and running `KillAndEmpty(ctx)` then `Close()` on exit; a failure refuses the spawn with `ErrStrictRequired`). `prepareStrict func(host string, limit cgroup.Limit) (strictProc, error)` is an unexported seam with `strictProc interface{ Start(*exec.Cmd) error; KillAndEmpty(context.Context) error; Close() error }`, which `*cgroup.Prepared` satisfies on every build (`strict_linux.go:233-348`, `strict_other.go:15-18`); nil → `b.Cgroups.PrepareStrict`, and a nil `Cgroups` under `Strict` is `ErrStrictRequired`. `Cgroups` stays the concrete `*cgroup.Manager` (`backend_exec.go:100`): tests replace the seam, never the field (round-4 finding 2). `(b *ExecBackend) NextSpawnStrict(host string) moves.Check` (`moves.Check` is Task 1's; S2: OK when `Strict(host)` and cgroup v2 delegation is present, `cgroup.Manager.Root() != ""`, and `PrepareStrict` is supported on this build). The host package sets `eb.Strict` from the hold view (a host whose record has a move in flight and the switch on); off → never strict, today's behaviour. |
| |
| -Tests: `RenderHeld` writes the file for a `landing` row `SyncExec` would skip and the ack carries `move_op`, the sha256 and the bundle etags, which equal the etags the dir store reports for the fixture's default and environment objects; a second call with no change still writes an ack. `MaterializeHeld` stages the pinned version into a temp `optDir` and points `live/<host>` at it, never at the channel head; with `optDir` made read-only (`os.Chmod(optDir, 0o555)`, skipped as root) it returns `ErrStageBlocked` and writes nothing. `ServeHeldStageRequests` materializes a request file into `optDir`, writes the ack and removes the request; `LiveVersion` reads the result. `TestSpawnUnderAMoveIsStrictOrRefused` (the `prepareStrict` seam returning an error → `Start` returns `ErrStrictRequired` and spawns nothing; the seam returning a recording `strictProc` → `Start` is called on it and `KillAndEmpty` then `Close` on exit; with `Strict` false the plain path runs as today). |
| +Tests: `RenderHeld` writes the file for a `landing` row `SyncExec` would skip and the ack carries `move_op`, the sha256 and the bundle etags, which equal the etags the dir store reports for the fixture's default and environment objects; a second call with no change still writes an ack. `MaterializeHeld` stages the pinned version into a temp `optDir` and points `live/<host>` at it, never at the channel head; with the `materializeHeldStage` seam (a package variable `= ensureStaged`, replaced by the test) failing with `&os.PathError{Err: syscall.EROFS}`, and separately `EACCES`, it returns `ErrStageBlocked` and writes nothing (the `chmod` fixture is gone: it tested discretionary permissions, not a read-only mount). `TestServeHeldStageRequestsRefusesStaleAndTraversal`: a request whose op the claim no longer names → removed, `live/<host>` untouched; a request whose `Host` is `../x`, or does not match its file name → removed before any path join (a `release.Store` fake records no reads); the happy path materializes, acks and removes; `LiveVersion` reads the result. `heldStageAuthority`'s own test covers each refused field. `TestSpawnUnderAMoveIsStrictOrRefused` (the `prepareStrict` seam returning an error → `Start` returns `ErrStrictRequired` and spawns nothing; the seam returning a recording `strictProc` → `Start` is called on it and `KillAndEmpty` then `Close` on exit; with `Strict` false the plain path runs as today). |
| |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "appconfig.RenderHeld, release.MaterializeHeld, strict spawns under a move (§9.1, §9.2, §5.3)" ...` |
| |
| @@ -2742,7 +2801,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Modify: `internal/activator/backend_exec.go` (export `LaunchEnvironment`), `internal/activator/backend_exec_test.go` (one test) |
| |
| **Interfaces:** |
| -- Produces: `moves.ReportKeyFromSecrets(secretsKeyB64 string) ([]byte, error)` (HKDF-SHA256 over the raw private key bytes, info `"carlos-move-env-report-v1"`); `moves.EnvReport{V int; Vars map[string]string; Digest string}`; `moves.ReportEnv(key []byte, env map[string]string) EnvReport` (each value HMAC-SHA256 under key, hex; `Digest` the sha256 of the sorted `name=mac` lines); `moves.Class(name string) string` returning `"region" | "layout" | "box" | "equal"`; `moves.CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff` with `EnvDiff{Name, Kind string}` (kinds: `missing`, `extra`, `differs`, `region-same`, `layout-missing`, `layout-present`); `moves.RegionVar = "CARLOS_REGION"`, `LayoutVars = {"CARLOS_CONTROL_SOCKET", "CARLOS_PURGE"}`, `BoxVars = {"INVOCATION_ID", "JOURNAL_STREAM", "HOSTNAME", "PWD", "OLDPWD", "SHLVL", "_"}`. |
| +- Produces: `moves.ReportKeyFromSecrets(secretsKeyB64 string) ([]byte, error)` (HKDF-SHA256 over the raw private key bytes, info `"carlos-move-env-report-v1"`); `moves.EnvReport{V int; Vars map[string]string; Digest string}`; `moves.ReportEnv(key []byte, env map[string]string) EnvReport` (each value HMAC-SHA256 under key, hex; `Digest` the sha256 of the sorted `name=mac` lines); `moves.Class(name string) string` returning `"region" | "layout" | "advisory" | "box" | "equal"` (`LayoutVars` is `CARLOS_CONTROL_SOCKET` alone; `AdvisoryVars` is `CARLOS_PURGE`); `moves.CompareEnv(source, target EnvReport, flatLayout bool) []EnvDiff` with `EnvDiff{Name, Kind string}` (kinds: `missing`, `extra`, `differs`, `region-missing`, `region-same`, `layout-missing`, `layout-present`, `advisory-differs`; `(EnvDiff).Note() bool` is true for `advisory-differs` alone, the one kind T3 records without failing); `moves.RegionVar = "CARLOS_REGION"`, `LayoutVars = {"CARLOS_CONTROL_SOCKET", "CARLOS_PURGE"}`, `BoxVars = {"INVOCATION_ID", "JOURNAL_STREAM", "HOSTNAME", "PWD", "OLDPWD", "SHLVL", "_"}`. |
| - `(b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, error)`: `childEnv(instanceEnv(rt))` plus the agent-owned variables `Start` adds, decoded back into a map (the composed launch environment, codex R2-8). |
| |
| - [ ] **Step 1: Write the failing tests** |
| @@ -3135,10 +3194,10 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/host/movesreport.go`, `internal/host/movesreport_test.go` |
| |
| **Interfaces:** |
| -- Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check` (Task 1's); `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string; Archive *ArchiveRef}` with `ArchiveRef{ManifestKey, ManifestVersion, SealedKey, SealedVersion, Plain string}` (the versions the source's `PutVersionedAt` responses returned, Task 14's ledger `archive_versions`, published so T5 reads by exact version: round-4 finding 12); `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| -- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string, archive *moves.ArchiveRef) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the archive manifest and sealed object the source report's `Archive` names, read by exact version through `lease.VersionedStore`'s `GetVersion`/`OpenVersion` in the target's own bucket; a report without `Archive` fails T5 "the source has published no archive yet": `archive.Verify(sealed, manifest, priv, binding)` (Task 7) checks the plaintext digest, every entry's path, size and sha256, the entry types and the cap; the report records the manifest's `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld(ctx, s, optDir, rt, version)`, Task 7b; on a hardened box it returns `ErrStageBlocked`, and then the pass writes `<DataDir>/moves/held-stage/<host>.json` and touches the `stage-requested` marker through `stageRequester(dd)` (`agent.go:1262`, the existing relay to the root `carlos-stage.service`, whose `carlos adopt --stage-only` now calls `release.ServeHeldStageRequests`); T7 passes only once `release.LiveVersion(optDir, rt.Host)` equals the pinned version, and until then fails "waiting for the root stage pass"; the report records the version; round-4 finding 6); T8 and T9 bind to the named target: this box's id must equal `op.intent.target` (`selectMoveBoxes`'s choice, Task 4), else T9 fails "this box is not the move's target"; the report writer is the named target only. S2 (`ExecBackend.NextSpawnStrict(host)`, Task 7b); S1 wakes a hibernating source and reads the answer from its own process, never from the response cache (round-4 finding 11: `edge.go:1651-1663` serves a cached GET before the activator runs, so a cached 200 could certify a source that cannot wake): `(*edge.Edge).ServeUncached(w, r)` (Task 14 adds it: `ServeHTTP` with the request marked `uncachedProbe` in its context, the same marker pattern as `doorCallerFrom`, which makes `cacheable` false at `edge.go:1651` so neither `GetFor` nor `maybeStore` runs, while the hibernating branch still wakes through `e.act.ServeHTTP`) with `GET /<ready path>`, `Host: <host>`, a recording `ResponseWriter` and a 30 s timeout; `moveCfg.ReadyProbe` is set by `setMovesReadyProbe(e)` from `edge.RunWithActivatorReady`'s callback (`agent.go:576`, where the agent receives its `*edge.Edge`) and fails "edge not ready" before that. Direct socket dialing is not used: it cannot wake a sleeping instance. Writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| +- Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check` (Task 1's); `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int; Rates Rates; Notes []string}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string; Archive *ArchiveRef}` with `ArchiveRef{ManifestKey, ManifestVersion, SealedKey, SealedVersion, Plain string}` (the versions the source's `PutVersionedAt` responses returned, Task 14's ledger `archive_versions`, published so T5 reads by exact version: round-4 finding 12); `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| +- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string, archive *moves.ArchiveRef) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the archive manifest and sealed object the source report's `Archive` names, read by exact version through `lease.VersionedStore`'s `GetVersion`/`OpenVersion` in the target's own bucket; a report without `Archive` fails T5 "the source has published no archive yet": `archive.Verify(sealed, manifest, priv, binding)` (Task 7) checks the plaintext digest, every entry's path, size and sha256, the entry types and the cap; the report records the manifest's `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHostVersions` (Task 7b) for the record's environment rendered to a temp config file, its `MergedVersions` recorded as `Inputs.BundleETags` and `Inputs.OverlayETag` and its `Shadowed` names (bundle variables the instance overlay overrides) recorded in `TargetReport.Notes []string`, which `TestTargetReportPassFlagsEachCheck` asserts for an overlay that shadows one bundle variable (round-5 finding 9), the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld(ctx, s, optDir, rt, version)`, Task 7b; on a hardened box it returns `ErrStageBlocked`, and then the pass writes `<DataDir>/moves/held-stage/<host>.json` and touches the `stage-requested` marker through `stageRequester(dd)` (`agent.go:1262`, the existing relay to the root `carlos-stage.service`, whose `carlos adopt --stage-only` now calls `release.ServeHeldStageRequests`); T7 passes only once `release.LiveVersion(optDir, rt.Host)` equals the pinned version, and until then fails "waiting for the root stage pass"; the report records the version; round-4 finding 6); T8 and T9 bind to the named target: this box's id must equal `op.intent.target` (`selectMoveBoxes`'s choice, Task 4), else T9 fails "this box is not the move's target"; the report writer is the named target only. S2 (`ExecBackend.NextSpawnStrict(host)`, Task 7b); S1 wakes a hibernating source and reads the answer from its own process, never from the response cache (round-4 finding 11: `edge.go:1651-1663` serves a cached GET before the activator runs, so a cached 200 could certify a source that cannot wake): `(*edge.Edge).ServeUncached(w, r)` (Task 14 adds it: `ServeHTTP` with the request marked `uncachedProbe` in its context, the same marker pattern as `doorCallerFrom`, which makes `cacheable` false at `edge.go:1651` so neither `GetFor` nor `maybeStore` runs, while the hibernating branch still wakes through `e.act.ServeHTTP`) with `GET /<ready path>`, `Host: <host>`, a recording `ResponseWriter` and a 30 s timeout; `moveCfg.ReadyProbe` is set by `setMovesReadyProbe(e)` from `edge.RunWithActivatorReady`'s callback (`agent.go:576`, where the agent receives its `*edge.Edge`) and fails "edge not ready" before that. Direct socket dialing is not used: it cannot wake a sleeping instance. Writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| |
| -Test (pure) `internal/moves/readiness_test.go`: |
| +Tests (pure) `internal/moves/readiness_test.go`: `TestReadyNeedsFreshReports` below, and `TestReadyRejectsMismatchedInputs` (two fresh passing reports whose `target.Inputs.FilesPlain` and `source.FilesPlain` differ → not ready, the reason naming both digests; the same for `Inputs.EnvReport` against `source.Env.Digest`; equal inputs → ready): |
| |
| ```go |
| func TestReadyNeedsFreshReports(t *testing.T) { |
| @@ -3215,10 +3274,7 @@ var AllChecks = []string{CheckProtocol, CheckSecrets, CheckEnv, CheckStorage, Ch |
| // (§7.3); the cut re-checks it at `admitted` (§7.4). |
| const ReportFreshness = 10 * time.Minute |
| |
| -type Check struct { |
| - OK bool `json:"ok"` |
| - Reason string `json:"reason,omitempty"` |
| -} |
| +// Check is Task 1's (check.go). |
| |
| type TargetReport struct { |
| V int `json:"v"` |
| @@ -3302,6 +3358,15 @@ func Ready(roundDelta, bar int64, target *TargetReport, source *SourceReport, no |
| if !source.S2.OK { |
| return false, "S2: " + source.S2.Reason |
| } |
| + // The target's checks must have been made against what the source |
| + // publishes NOW (round-5 finding 9): a fresh passing report for |
| + // archive A must not promote after the source has published archive B. |
| + if target.Inputs.FilesPlain != source.FilesPlain { |
| + return false, fmt.Sprintf("target checked files %s, the source now publishes %s", target.Inputs.FilesPlain, source.FilesPlain) |
| + } |
| + if target.Inputs.EnvReport != source.Env.Digest { |
| + return false, "target checked another launch environment than the source now reports" |
| + } |
| return true, "" |
| } |
| ``` |
| @@ -3333,7 +3398,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/registry/hold_test.go` |
| |
| **Interfaces:** |
| -- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `(r *Registry) DeleteIfHold(host, op string) (bool, error)` (`DELETE FROM routes WHERE host=? AND hold_op=?`; the source's row deletion at finalization, §7.9 step 3; here rather than Plan 3 so Plan 3 Task 8 does not depend on Task 7, D12; its negative test: a row held by another op, or not held, is not deleted and returns false); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| +- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `(r *Registry) HoldRows() ([]Route, error)` (every row with `hold_op != ''`; Task 13's rollback hold rule and Task 7b's `heldStageAuthority` read it); `(r *Registry) DeleteIfHold(host, op string) (bool, error)` (`DELETE FROM routes WHERE host=? AND hold_op=?`; the source's row deletion at finalization, §7.9 step 3; here rather than Plan 3 so Plan 3 Task 8 does not depend on Task 7, D12; its negative test: a row held by another op, or not held, is not deleted and returns false); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| @@ -3433,97 +3498,33 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ### Task 13: The protocol floor and `moves_protocol` (after Task 12: it reads `hold_op`) |
| |
| **Files:** |
| -- Create: `internal/moves/floor.go`, `internal/moves/floor_test.go` |
| -- Modify: `internal/host/edgesfile.go` (field and writer), `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (refusal on activation), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup, switch on only), `internal/host/units/carlos-system-recover.service` and `internal/host/units/carlos-system-update.service` (the `.prev` selection honours the hold-prev marker), `internal/host/host_test.go` (the unit assertions), `internal/host/movesfloor.go` and `internal/host/movesfloor_test.go` (the agent's floor arming, `participants.json`, the hold-prev marker) |
| +- Create: `internal/host/movesfloor.go`, `internal/host/movesfloor_test.go` |
| +- Modify: `internal/host/edgesfile.go` (fields and writer), `internal/host/edgesfile_test.go`, `internal/console/instancesplacement.go` (reader struct), `internal/host/systemapply.go` (`armFloor`, `floorAllows`, the `decidePrev` sidecar, the rollback hold rule), `internal/host/systemapply_test.go`, `internal/console/server.go` (publish the console's fingerprint and protocol at startup, switch on only), `internal/host/units/carlos-system-recover.service` and `internal/host/units/carlos-system-update.service` (the `.prev` selection by the floor, `prev_ok`), `internal/host/host_test.go` (the unit assertions and the shell-level selection test), `internal/host/movesfloor.go` and `internal/host/movesfloor_test.go` (the agent's hourly refresh of the local floor files and the edges-file fields' readers) |
| |
| **Interfaces:** |
| -- Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| -- `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| -- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<DataDir>/.moves/floor.json` durably (temp file, fsync, rename; under `/data`, which the sandboxed agent may write (`carlos-edge.service:37`, `ReadWritePaths=/data /run/carlos`) and root's `system apply` may read; `<optDir>` is not writable by the agent, round-4 finding 6); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| +- Produces: `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`; `moves.FloorFile = ".moves/floor.json"` and `moves.FloorMinReleaseFile = ".moves/floor.min-release"` (relative to `DataDir`); `moves.PrevVersionFile = "carlos.prev.version"` (beside `.prev` under `optDir`). `moves.ProtocolVersion`, `FloorKey`, `Floor`, `ReadFloor` and `ReleaseAtOrAbove` **are Task 1's** (`floor.go`; Task 4 reads them). |
| +- `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor); `edgesFile.MovesFloorMinRelease string \`json:"moves_floor_min_release,omitempty"\`` (the local floor file's `min_release`; absent means the box has not armed a floor); `edgesFile.PrevRelease string \`json:"prev_release,omitempty"\`` (`carlos.prev.version`'s content; `"none"` when there is no `.prev`; absent means unknown, which the door treats as below the floor). |
| +- **The floor is enforced at executable selection, by the units themselves, and by the applier** (§11, "every path that activates an executable"; round-5 findings 3 and 4). Three durable files, all written synchronously by root, never deleted once written: (1) `<DataDir>/.moves/floor.json` and the one-line `<DataDir>/.moves/floor.min-release`, written by `carlos system apply` from `control/moves-protocol.json` **before it stages anything** (`install`'s first step, after `openSystemStore`; temp file, fsync, rename) and refreshed by the agent's hourly edges-file loop (the agent may write `/data`, `carlos-edge.service:37`; `<optDir>` it may not, round-4 finding 6); (2) `<optDir>/carlos.prev.version`, written by `decidePrev` (`systemapply.go:473-489`, which already runs `.prev version` to learn it) with that version, durably, **before the update journal is written**, so a `.prev` and its sidecar are never out of step at a recovery. The units then select the executable with the floor in hand. Both `ExecStart` lines become `/bin/sh -c 'prev_ok() { [ -s /data/.moves/floor.min-release ] || return 0; [ -s /opt/carlos/carlos.prev.version ] || return 1; f=$(cat /data/.moves/floor.min-release); p=$(cat /opt/carlos/carlos.prev.version); [ "$(printf "%s\n%s\n" "$f" "$p" | sort -V | head -n1)" = "$f" ]; }; if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ] && prev_ok; then exec /opt/carlos/carlos.prev system apply --recover; fi; exec /opt/carlos/carlos system apply --recover'` (`system apply` without `--recover` in the update unit; `sort -V` is GNU coreutils, on every Debian box; the `v` prefix sorts naturally; equal is at-or-above). No floor file → `prev_ok` is true and the line behaves exactly as today. A floor file with a `.prev` below it, or without a sidecar → the current `carlos system apply` runs, and its `floorAllows` refuses a below-floor activation itself: `install` for `req.Version` (after arming), `rollback` and `restoreFiles` for the sidecar's version, **reading only the local floor file** in `rollback` and `restoreFiles` (boot recovery runs before networking, `systemapply.go:109`). `host_test.go:1314-1330`'s unit assertions gain `prev_ok`, `floor.min-release` and `carlos.prev.version` for both units (red before the unit edit). |
| |
| - **The recovery entry points are the units, not only the applier** (round-4 finding 5): `carlos-system-recover.service:17` and `carlos-system-update.service:25` exec **`carlos.prev system apply [--recover]`** directly when `/opt/carlos/system/router/journal.json` exists, and an older `.prev` runs its own recovery without these checks. So the agent maintains a **hold-prev marker**, `/data/.moves/hold-prev` (a one-line reason): written whenever this box is a participant (the registry's `hold_op` rows or `participants.json`, as above) and either no floor is armed or `.prev`'s version (the previous build `decidePrev` recorded, `systemapply.go:473-489`, read through `a.prevBuild(ctx)`) is below `Floor.MinRelease`; removed when the condition clears. Both units' `ExecStart` lines gain `&& ! [ -e /data/.moves/hold-prev ]` in the `.prev` condition, so a held box falls through to the current `carlos system apply`, whose `floorAllows` checks run; `host_test.go:1314-1330`'s unit assertions gain the marker string for both units (red before the unit edit). The hold is honest about its cost: a held box whose current `carlos` cannot start recovery loops on `OnFailure` until an operator acts, which is what §11 asks of a participant below the floor. |
| + **The admission-to-enforcement boundary** (round-5 finding 3): the console admits a move only when every edges file of both regions publishes `moves_floor_min_release` equal to the published floor's `min_release` (so the box has written its local floor files and its units already select by them) and `prev_release` either `"none"` or at or above it (Task 4's `FloorOK`). So at the moment a move is committed, every participant's recovery already selects by the floor and its `.prev` is already at or above it; nothing asynchronous has to happen on a box after admission, and a crash right after the commit recovers correctly. A floor raised later is enforced at the next recovery by the same files once the hourly refresh has written them, and the door refuses new moves until every box's file shows the new value. |
| |
| - **Dormancy (D13):** arming, `participants.json` and the marker run only with `moveCfg.Enabled`. With the switch off none of the three files exists, `floorAllows` finds no floor and no participant and proceeds with its log line, and the unit lines select `.prev` exactly as today: the dormant boundary is the absence of those files, and `TestFloorIsDormantWithTheSwitchOff` (an agent pass with the switch off writes none of them over a registry with a held row) pins it. |
| + **Dormancy (D13; round-5 finding 4):** the floor is the one Phase 1 element not behind the switch, by design: it is a safety property armed by an operator action (publishing `control/moves-protocol.json`, create-only, before the first production move, §11), not by the switch. A deployment that never published the object has no floor files anywhere, `prev_ok` is true, `floorAllows` finds no local file and proceeds with one log line, and the registry is not consulted: `main` behaves exactly as today. Once armed, the floor stays armed if the switch is later turned off. `rollback` additionally refuses while the registry holds any `hold_op` row (§11, "any downgrade of a participant in a held operation"); no such row exists in a deployment where the switch was never on. The agent writes no marker and no `participants.json`: rev 5's `hold-prev` marker is withdrawn (its asynchronous publication left exactly the window the review named). |
| |
| - [ ] **Step 1: Write the failing tests** |
| |
| -```go |
| -package moves |
| - |
| -func TestFloorReadAndCompare(t *testing.T) { |
| - s := release.NewDirStore(t.TempDir()) |
| - f, err := ReadFloor(context.Background(), s) |
| - if err != nil || f != nil { |
| - t.Fatalf("absent floor: %+v %v", f, err) |
| - } |
| - s.Put(context.Background(), FloorKey, []byte(`{"min":1,"min_release":"v0.23.0"}`), "") |
| - f, err = ReadFloor(context.Background(), s) |
| - if err != nil || f.Min != 1 || f.MinRelease != "v0.23.0" { |
| - t.Fatalf("floor: %+v %v", f, err) |
| - } |
| - for v, want := range map[string]bool{"v0.23.0": true, "v0.24.1": true, "v0.22.9": false} { |
| - if ok, err := ReleaseAtOrAbove(v, f.MinRelease); err != nil || ok != want { |
| - t.Fatalf("%s: %v %v", v, ok, err) |
| - } |
| - } |
| -} |
| -``` |
| - |
| -In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<DataDir>/.moves/floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestHoldPrevMarkerFollowsTheFloor` (participant and no floor → marker with its reason; participant and `.prev` below the floor → marker; `.prev` at the floor → no marker; not a participant → no marker; switch off → never written); `TestSystemUnitsSkipPrevUnderHold` (the `ExecStart` line's `sh -c` script is run with a temp root substituted for `/opt/carlos` and `/data` and stub `carlos`/`carlos.prev` scripts that record which ran: with the marker the current `carlos` runs, without it `carlos.prev`); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snap.Complete` (every edges file read; `complete()` would also demand `BoxesComplete`, which only `readLiveEdgesSnapshot` computes, and the floor has no liveness question); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| +(`TestFloorReadAndCompare` and `floor.go` are Task 1's.) In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyArmsTheFloorBeforeStaging` (the store holds `min_release: "v0.30.0"`; `install` of a pointer at `v0.29.0` writes both local floor files first, then refuses with "moves protocol floor" and leaves the binary untouched); `TestBootRecoveryUsesTheArmedFloorOffline` (the store errors on every read; the local floor names `v0.30.0`; `carlos.prev.version` says `v0.29.0` → `restoreFiles` refuses; `v0.30.0` → proceeds; no sidecar → refuses); `TestDecidePrevWritesTheVersionSidecarBeforeTheJournal` (the fake `ops.run` answers `version`; the sidecar exists with `.prev`'s version before `journal.json` appears); `TestFloorIsDormantWithoutTheObject` (no object and no local files, a registry with a held row: `install`, `rollback` and `restoreFiles` all proceed; then an armed local file and the switch off: a below-floor `restoreFiles` still refuses); `TestRollbackRefusesUnderAHold` (a `hold_op` row refuses `rollback`; none does not). In `host_test.go`: `TestSystemUnitsSelectPrevByTheFloor` (both `ExecStart` scripts are extracted from the unit text and run under `sh -c` with `/opt/carlos` and `/data` rewritten to a temp root and stub `carlos`/`carlos.prev` scripts that record which ran: no floor file → `.prev`; floor `v0.30.0` and sidecar `v0.29.0` → current; sidecar `v0.30.0` → `.prev`; floor present and no sidecar → current). In `edgesfile_test.go`: `TestEdgesFilePublishesMovesProtocolFloorAndPrev` (`"moves_protocol":1`; `moves_floor_min_release` from the local file, absent without it; `prev_release` from the sidecar, `"none"` without a `.prev`). The door (Task 4) reads the live snapshot and `complete()` (every edges file and every status snapshot read; the floor question needs every file, and liveness is already required for eligibility). |
| |
| - [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**: |
| |
| -```go |
| -package moves |
| - |
| -const ProtocolVersion = 1 |
| -const FloorKey = "control/moves-protocol.json" |
| -const ConsoleKey = "control/moves-console.json" |
| - |
| -type Floor struct { |
| - Min int `json:"min"` |
| - MinRelease string `json:"min_release"` |
| -} |
| - |
| -type ConsoleRecord struct { |
| - V int `json:"v"` |
| - Protocol int `json:"protocol"` |
| - SecretsKeyFP string `json:"secrets_key_fp"` |
| - At time.Time `json:"at"` |
| -} |
| - |
| -func ReadFloor(ctx context.Context, s release.Store) (*Floor, error) { |
| - b, _, err := s.Get(ctx, FloorKey) |
| - if errors.Is(err, os.ErrNotExist) { |
| - return nil, nil |
| - } |
| - if err != nil { |
| - return nil, err |
| - } |
| - var f Floor |
| - if err := json.Unmarshal(b, &f); err != nil { |
| - return nil, fmt.Errorf("moves: %s: %w", FloorKey, err) |
| - } |
| - return &f, nil |
| -} |
| - |
| -func ReleaseAtOrAbove(version, minRelease string) (bool, error) { |
| - if !semver.IsValid(version) || !semver.IsValid(minRelease) { |
| - return false, fmt.Errorf("moves: not a release version: %q / %q", version, minRelease) |
| - } |
| - return semver.Compare(version, minRelease) >= 0, nil |
| -} |
| -``` |
| - |
| -(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup, **only when `s.cfg.MovesEnabled`** (D13: off publishes nothing; `TestConsoleRecordNeedsTheSwitch`), where the edges-dependent config is logged (`server.go`): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| +`internal/moves`: `ConsoleKey`, `ConsoleRecord` and the three file-name constants only (`floor.go` is Task 1's). Edges file: add the three fields in the writer at `edgesfile.go:722-736` (`MovesProtocol: moves.ProtocolVersion`; `MovesFloorMinRelease` read from `<DataDir>/.moves/floor.min-release` and `PrevRelease` from `<optDir>/carlos.prev.version`, or `"none"` when `.prev` is absent, both read at each write); `internal/console/instancesplacement.go`'s `edgesFilePlacementView` gains the same fields. Units: the two `ExecStart` lines above. Console startup, **only when `s.cfg.MovesEnabled`** (D13: off publishes nothing; `TestConsoleRecordNeedsTheSwitch`), where the edges-dependent config is logged (`server.go`): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: `(a *systemApplier) armFloor(ctx) error` (first in `install`: `moves.ReadFloor` from `a.store`, the store `openSystemStore` returned, stored on the applier (add the field if absent); when present, both local files written durably; a store error with a local file present keeps the local file; absent everywhere → no floor) and `(a *systemApplier) floorAllows(version string) error` reading **only** `<DataDir>/.moves/floor.json` and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below (a `.prev` without a sidecar is below any armed floor); called in `install` for `req.Version` after `armFloor`, and in `rollback` and `restoreFiles` for the version `carlos.prev.version` records; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. `decidePrev` writes the sidecar durably with the version it already obtains from `.prev version`, before the journal. `rollback` refuses while `reg.HoldRows()` (Task 12) is non-empty. |
| |
| - [ ] **Step 4: Run**: `nice -n 10 go test ./internal/moves ./internal/host ./internal/console -count=1 -p 1` → PASS. |
| |
| - [ ] **Step 5: Commit** |
| |
| ```bash |
| -git add internal/moves/floor.go internal/moves/floor_test.go internal/host/edgesfile.go internal/host/systemapply.go internal/host/systemapply_test.go internal/console |
| -git commit -m "moves: protocol floor; edges file publishes moves_protocol; system apply refuses below the floor (§11) |
| +git add internal/moves internal/host internal/console |
| +git commit -m "moves: the protocol floor armed on disk and enforced at executable selection; edges file publishes moves_protocol, the armed floor and prev_release (§11) |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ``` |
| @@ -3536,12 +3537,12 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Modify: `internal/host/movespass.go` (`moveCfg`, `armMovesPass`, `runMoveRound`'s `MovePreparing`/`MoveReady` arm, `MoveAborting` arm's prefix), `internal/host/movespass_test.go` (`fakeMoveMirror` gains `CopyVersioned`/`PutVersioned`; `moveHarness` gains a `lease.VersionedStore` fake and a staging-lease fake), `internal/host/agent.go` (`setMovesReadyProbe` in the `RunWithActivatorReady` callback), `internal/edge/edge.go` (`ServeUncached` and the `uncachedProbe` context marker beside `doorCallerFrom`), `internal/edge/edge_test.go` (`TestServeUncachedBypassesTheCacheAndWakes`: the cache primed with a 200 for the path and a fake activator whose wake fails: `ServeHTTP` answers the cached 200, `ServeUncached` answers the wake failure and stores nothing) |
| |
| **Interfaces:** |
| -- `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1 through `(*edge.Edge).ServeUncached`, Task 11's definition; set by `setMovesReadyProbe(e *edge.Edge)` from `RunWithActivatorReady`'s callback, `agent.go:576`; "edge not ready" before that), `StrictNext func(host string) moves.Check` (S2: `ExecBackend.NextSpawnStrict`, Task 7b). |
| +- `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; **empty means Phase 1 rounds refuse** (round-5 finding 11: a 15-minute `lease.Acquire` can never satisfy the signing guard, and a per-participant bucket is not one shared coordination location): the round writes S1 "this deployment has no deployment bucket for the staging lease" and does nothing else; the harness supplies a staging-lease fake, so tests never need the bucket), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1 through `(*edge.Edge).ServeUncached`, Task 11's definition; set by `setMovesReadyProbe(e *edge.Edge)` from `RunWithActivatorReady`'s callback, `agent.go:576`; "edge not ready" before that), `StrictNext func(host string) moves.Check` (S2: `ExecBackend.NextSpawnStrict`, Task 7b). |
| - The switch (Task 0, D13): `moveCfg.Enabled` decides which worker runs. **Off:** today's `runMoveRound` (renamed `legacyMoveRound`, unchanged, with its tests unchanged) handles every record exactly as main does; an id'd record (none exists while the switch was never on) is skipped with one log line a minute. **On:** `phase1MoveRound` below handles id'd records; a legacy record is refused with one log line a minute (M-C: MV-N7's sentence in the log, not to a user). |
| -- The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveGuardedAbortComplete(g, now)`; the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| +- The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveGuardedAbortComplete(g, now)`; **the crash boundary between those two writes** (round-5 finding 10): a round that finds the record `aborting` and the claim already `serving` completes the record only when `hostclaim.RequireReleasedMove(id)` holds (the serving op's journal names this move as the one it released, Task 3); a serving claim without that evidence was released by someone else (the console pass, Plan 5) and the round leaves the record to it with one log line; `TestInterimAbortReleasesTheClaim` adds the crash case (stop the round after the claim release; run it again; the record completes; and a serving claim with no evidence does not complete it); the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| - **Every transition is guarded by op identity** (Task 2's `MoveGuard{ID: rec.Move.ID, Incarnation: rec.Move.Incarnation}`, captured when the round read the record): `MoveGuardedPromoteReady`, `MoveGuardedCut`, `MoveGuardedAbortComplete`; a replacement move between the read and the CAS is `ErrMoveConflict` and the round ends (`TestRoundNeverAdvancesAReplacementMove`). |
| - **The source copy is bound to the claim's location and generation** (round-3 finding 5), never to `HibernatePrefix(cfg.OwnPrefix, host)`: `moves.ReplicaSource(h hostclaim.Holder) (prefix string, keep func(rel string) bool)`: `base := strings.TrimSuffix(h.Location, "/") + "/"` first (`Claim.Validate` accepts only a location WITHOUT a trailing slash, `claim.go:245`, and `genselect.requestLocation` adds the separator the same way, `select.go:160-166`; a naive join would yield `instances/h/<inc>g-cur/` and an empty listing, round-4 finding 4); for generation `""` or `hostclaim.RootGeneration`, `prefix = base` and `keep` admits only `<level 0000-0009>/<min>-<max>.ltx` directly under it (not `g-*/`, not a 64-hex child incarnation, not `generation.json`, `lock.json` or `closed/`); for a generation `g`, `prefix = base + "g-" + g + "/"` and `keep` admits its level directories. `lease.MirrorPrefix` gains a variant `lease.MirrorPrefixKeep(ctx, m, srcBucket, srcPrefix, dstBucket, dstPrefix, rc, keep func(rel string) bool)` (`MirrorPrefix` is `MirrorPrefixKeep` with `keep == nil`), so the lane holds the generation's LTX files flat under `replica/<level>/…` and nothing else. `TestReplicaSourceSelectsTheClaimsGeneration`: the holder comes from a claim written through `hostclaim.Update` (so its location is the validated `instances/<host>/<inc>` form, no trailing slash) and the asserted prefix is `instances/<host>/<inc>/g-cur/`; a location with two incarnations, a `g-old/` generation and the live `g-cur/`: only `g-cur/`'s level files are copied; a legacy root location (`instances/<host>`) with `g-x/` beside its level directories copies the level directories only. |
| -- The round, per in-flight record this box owns, in order (§7.3, K5, K12), **after** taking the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`) and **then** re-reading the claim under it (§7.3: every round takes the lease first, then revalidates the claim's op id and phase): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; refuse likewise (S1, M-C17's sentence) a source whose store binding exists: `objstore.GetIn(ctx, s, acct, app, instanceStoreEnv(rec))` non-nil, the door's check (Task 4) repeated every round before the first staging write, a read error ending the round; `hostclaim.RequireMoveOp(id, KindMoving, "")` on the claim read under the lease and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); `move.json`: `stagingMirror.PutIfAbsentVersioned(move.json)` through the guarded staging mirror (Task 6; **every** write into the target bucket, `move.json`, `replica/`, `files/`, `timers/` and Task 11's `probe/` measurement object, goes through the one guarded path: `guardedMirror` implements `lease.MirrorStore`, `VersionedCopier` and `VersionedPutter` over `*lease.StagingMirror` by calling `Guarded.Before` and passing its `signAt` to `CopyVersionedAt`, `PutVersionedAt`, `PutIfAbsentVersionedAt` and `DeleteAt`, Task 6; nothing calls `lease.VersionedStore` against the target bucket); success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefixKeep(ReplicaSource(holder) → ReplicaPrefix, keep)` through the guarded staging mirror, whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record both keys and both returned versions in the ledger's `archive_versions` and publish them as the source report's `Archive`; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MoveGuardedPromoteReady(g, now)` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveGuardedCut(g, now)` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| +- The round, per in-flight record this box owns, in order (§7.3, K5, K12), **after** taking the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`) and **then** re-reading the claim under it (§7.3: every round takes the lease first, then revalidates the claim's op id and phase): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; refuse likewise (S1, M-C17's sentence) a source whose store binding exists: `objstore.GetIn(ctx, s, acct, app, provision.InstanceStoreEnv(rec))` non-nil, the door's check (Task 4) repeated every round before the first staging write, a read error ending the round; `hostclaim.RequireMoveOp(id, KindMoving, "")` on the claim read under the lease and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); `move.json`: `stagingMirror.PutIfAbsentVersioned(move.json)` through the guarded staging mirror (Task 6; **every** write into the target bucket, `move.json`, `replica/`, `files/`, `timers/` and Task 11's `probe/` measurement object, goes through the one guarded path: `guardedMirror` implements `lease.MirrorStore`, `VersionedCopier` and `VersionedPutter` over `*lease.StagingMirror` by calling `Guarded.Before` and passing its `signAt` to `CopyVersionedAt`, `PutVersionedAt`, `PutIfAbsentVersionedAt` and `DeleteAt`, Task 6; nothing calls `lease.VersionedStore` against the target bucket); success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefixKeep(ReplicaSource(holder) → ReplicaPrefix, keep)` through the guarded staging mirror, whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record both keys and both returned versions in the ledger's `archive_versions` and publish them as the source report's `Archive`; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MoveGuardedPromoteReady(g, now)` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveGuardedCut(g)` (Task 2's signature: the cut stamps nothing at `requested`) (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| - Receipts: `lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(host, id)}`; a per-op `moveLedger` JSON beside it at `move-receipts/<host>/<id>-moves.json` records `move_json_version`, `archive_plain`, `archive_sealed`, `archive_versions`. |
| |
| Tests: the legacy tests (`TestMovesPassPrepareAndPromote`, every `TestMovesPassCut…`, `TestMovesPassAbort…`) stay as they are and run with the switch off; the new tests run the harness with `cfg.Enabled = true` and a `MoveStart`ed record: `TestPhase1PrepareAndPromote` (keys under `moves/<host>/<id>/replica/<level>/…`, `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store), `TestPhase1IsInertWithTheSwitchOff` (an id'd record and the switch off: no copy, no CAS, one log line), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`; the target-side `TestTargetReportPassFlagsEachCheck` has the matching case: a `move.json` naming another move id fails T9 with the lane sentence), `TestInterimAbortReleasesTheClaim`, `TestRoundNeverAdvancesAReplacementMove`, `TestReplicaSourceSelectsTheClaimsGeneration`, `TestEveryStagingWriteIsGuarded` (a `Guarded` whose `Now` is past the bound: the round issues no request of any class, `move.json`, replica copy, archive, timers or probe object), `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| @@ -3551,7 +3552,7 @@ Tests: the legacy tests (`TestMovesPassPrepareAndPromote`, every `TestMovesPassC |
| - [ ] **Step 5: Commit** |
| |
| ```bash |
| -git add internal/host/movespass.go internal/host/movespass_test.go |
| +git add internal/host internal/edge internal/moves |
| git commit -m "host: preparing and ready rounds on the per-operation staging layout, under the staging lease (§7.3) |
| |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
amadan