amadan
plan: self-serve moves 2, rev 3 (Astra plan 2 review 2: 21 findings)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+163
−165
docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md+163 −165
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md| index 9e0f2c460..6a20b80f5 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| @@ -2,277 +2,275 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 2, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan2-review.md`, NOT READY, 20 findings). Rev 2 answers them: authorisation before any evidence read; one open move per host enforced by the store and every phase change a predicated update; the admitted lifetime pinned on the row and checked inside the queue mutation; one shared start validation with every existing check; explicit box dispatch for the pre-stop reports and a cut barrier that waits for them; initial emptiness and staged ownership as two checks; a pass/refuse/unknown rule for every fact; participants pinned by box id and enforced at every transition; the catch-up rule from copied plus deferred bytes; a journal that can express every wait, a recovery table per write boundary and a console lease; a serial bump on every transition; the no-return boundary at the committed record state; outcome before release with a pending-release state; a mixed-version protocol for `active.json`; paging armed in this mode with durable attempts and an orphan scan; the three abort facts in storage and the view; rendering and recovery independent of admission with a self-serve view type that does not collide with Phase 1; status facts with identity, age and error; the queued state's exact semantics; and every interface the review found missing. |
| +Rev 3, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21 findings). Rev 3 answers rev 2's twenty-one: the doors' dispatcher order is stated (authenticate, host shape, capability, then the row, then the record) and the route tests count SQL reads too; every phase advance is predicated on no abort request and cut/cancel are tested with barriers in both orders; a legacy record is migrated **bound** to the observed ETag and the record checks re-run on the migrated record; one `recordEligibility` carries `moveGate`'s class checks and `offlineStartValidation` is called immediately before the queue write and before the start; the lane is move-bound and the two prechecks request-bound, with a validator matrix; the stager publishes a complete destination inventory bound to its request and coordinates and the console compares whole inventories; every fact has known/ok/detail and the console enumerates the required sets itself; one participant resolver is used by every box path and every console gate, with the operator's region path kept only without participants; the journal gains issued/consumed identities, per-wait deadlines, a recovery table per boundary, whole-record predicates, deadline evaluation before evidence, and a lease checked before every write; the serial bump is journaled; every record-clearing path finalises through one finaliser with outcome before release and terminal effects before the release clears; the `active.json` rollout and rollback are stated with the matching-projection hazard; the pass and paging start on reservations too; the sealer publishes a durable `tombstone_lifted` fact before its cleanup; the view's precedence among record move, open row and terminal row is defined with a retention window and admission is checked inside prepare; status facts carry identity, freshness, over-cap and truncation; the queued transitions have exact signatures; the remaining interfaces are typed; and the operator return's leftover acceptance is in scope. |
| Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`): records carry a lifetime and `move_seq`, rows carry the pair, readers and writers are judged. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| -**Goal:** With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, Tito's existing residency calls (prepare, status, cut, cancel) run the offline move as a console job: an owner-delegated move-capable token asks, the door refuses what cannot be offered before anything is written, per-box reservations serialise moves per box, the boxes prove the lane, the target's cleanliness and the source's eligibility before any copy and again before the stop, staging runs until caught up, the cut runs start, seal, flip, open, health, finish with a deadline on every wait, every failure before the committed `open` initiates a lossless abort, and the view tells Tito `preparing`, `ready`, `cutting`, `none` with `last` and the abort's three facts. |
| +**Goal:** With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, Tito's existing residency calls (prepare, status, cut, cancel) run the offline move as a console job: an owner-delegated move-capable token asks, the door refuses what cannot be offered before anything is written, per-box reservations serialise moves per box, the boxes prove the lane, the target's cleanliness and the source's eligibility before any copy and again before the stop, staging runs until caught up, the cut runs start, seal, flip, open, health, finish with a deadline on every wait, every failure before the committed `open` initiates a lossless abort, and the view tells Tito `preparing`, `ready`, `cutting`, `none` with `last` and the abort's three facts. The operator's return accepts exactly the forward move's leftovers. |
| -**Architecture:** One `self_serve_moves` SQLite row per move drives a console pass (`StartSelfServeMovesPass`, every 5 s, under a console lease in the deployment bucket) that advances each move by at most one step per tick from the evidence the operator doors already read (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the door bodies refactored into functions the doors and the job share. A new record state `offline-queued` lets the console reserve boxes and collect the lane, source-precheck and target-ready reports before a byte is copied; a precheck request stamp on the record makes both participants report again before the cut. `control/offline-moves/active.json` becomes a per-box reservation table with a legacy projection. The move-capable token is a `moves` column on `cli_tokens`, owner-minted, checked at the three self-serve mutations before any record is read. |
| +**Architecture:** One `self_serve_moves` SQLite row per move drives a console pass (`StartSelfServeMovesPass`, every 5 s, under a console lease in the deployment bucket) that advances each move by at most one step per tick from the evidence the operator doors already read, through the door bodies refactored into functions the doors and the job share. A new record state `offline-queued` lets the console reserve boxes and collect the lane, source-precheck and target-ready reports before a byte is copied; a precheck request stamp makes both participants report again before the cut. `control/offline-moves/active.json` becomes a per-box reservation table with a legacy projection. The move-capable token is a `moves` column on `cli_tokens`, owner-minted, checked at the three self-serve mutations before any evidence is read. |
| -**Tech Stack:** Go. `internal/console` (doors, store, passes, alerting), `internal/provision` (record transitions), `internal/moves` (reports, reservations), `internal/host` (the box's offline round, status report), `internal/moves/archive` (the eligibility dry run), `internal/alerting`, `internal/status`. Tests: the console in-process with `release.NewDirStore`, and the two-box harness of `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`) driven through the real serial bump. |
| +**Tech Stack:** Go. `internal/console`, `internal/provision`, `internal/moves`, `internal/host`, `internal/moves/archive`, `internal/alerting`, `internal/status`, `internal/lease`. Tests: the console in-process with `release.NewDirStore`; the integrated witnesses in a new external test package `internal/e2e/selfserve` that imports both `internal/host`'s exported harness (`hosttest`, Task 10) and the console, driven by the real serial bump. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §8, §11. Two stated deviations: the record's `self_serve` is an object, not a boolean (plan 3 reads its fields); the estimate's pre-request input is the replica size alone (the status report publishes no "last-day churn"; the rolling medians carry the rates). |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §7.5's operator return, §8, §11. Stated deviations: the record's `self_serve` is an object, not a boolean; the pre-request estimate input is the replica size alone; the claims-off "move log" is the console's `self_serve_events` table (shown by `carlos move offline status --log`), not the claims-on `hostops.MoveLogWriter`, which requires claim evidence. |
| ## Global Constraints |
| -- Behind `CARLOS_SELF_SERVE_MOVES` (`cfg.SelfServeMoves string`; only `offline` enables **admission**), and only when `!cfg.HostClaim && !cfg.MovesEnabled`. **Rendering and recovery** of a self-serve move are enabled by the existence of its row, not by the switch: with the switch off, an open move runs to completion, its view keeps the self-serve shape until Tito has read its terminal `last`, and no new move is admitted. With no row for the host the four doors behave byte for byte as today (`TestSwitchOffIsByteIdentical`). |
| -- The operator doors (`.../move/offline*`) keep their answers byte for byte (fixtures snapshot every refusal of the start, stage, seal, flip, open, finish and abort doors before and after); their bodies move into functions the job calls. A job tick never calls an HTTP door. |
| -- Every wait has a deadline (§7.2's table, Task 8); expiry before the committed `open` initiates abort; after it pages and retries. The job never calls `finish` without the gate's evidence. |
| -- Every record transition the job writes goes through one helper (`selfServeApply`, Task 7) that re-reads the record by the move id on a lost response and bumps `control/serial` after every landed write, including one it discovers landed. |
| -- One console runs the pass: the pass holds `control/offline-moves/console.json` (a 60 s lease, conditional writes, the console's identity) and a second console refuses to run it (Task 4). |
| -- Refusals come before writes: the door writes the `self_serve_moves` row only after every check in §6.1's refusal list has passed, and the record only after the reservation. |
| -- Boxes are named by **box id** (`edgesfile.PlacementView.BoxID`) in reservations and participants, with the label beside it for reports and logs; a box whose `BoxID` is empty cannot participate ("<label> runs an older release", the existing sentence). Every box compares its own id; region membership alone never makes a box act for a self-serve move. |
| -- Copy rows pending Paul's review (listed in Tasks 5 and 8); existing sentences reused where named. No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| -- Census: the box-side units take rows in `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus` (`plan14_root_reads_test.go`), `rev3=PROPOSED-self-serve-moves-2`; `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's inventory with the new transitions' digests and the box's new report writers; both green. |
| +- `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) enables **admission**, and only when `!cfg.HostClaim && !cfg.MovesEnabled`. **Rendering and recovery** follow the rows and the reservations, not the switch: open moves run to completion, terminal rows render `last` for their retention window, the pass and paging start whenever open rows or reservations exist. With no row for the host the four doors answer byte for byte as today. |
| +- The operator doors keep their answers byte for byte (snapshots of every refusal of every operator door, including combinations of a malformed body with a class refusal, taken from main and replayed). A job tick never calls an HTTP door. |
| +- Every wait has a deadline (Task 8's table); a deadline is evaluated from the row **before** any evidence is read, so an unreadable report never postpones expiry or paging. Expiry before the committed `open` initiates abort; after it pages and retries. The job never calls `finish` without the gate's evidence. |
| +- Every record transition the job writes goes through `selfServeApply` (Task 7): a `want func(provision.Record) bool` on the whole record decides whether a lost response landed; `control/serial` is bumped after every landed write, and the row records `notified_state` so a crash between commit and bump is repaired on the next tick. |
| +- One console runs the pass: `control/offline-moves/console.json` is a 60 s lease held by the console's identity, renewed every 20 s during a tick and checked immediately before every write; a write after the lease lapsed is refused and the tick ends. The SQLite journal is the console's own; a replacement console starts from the rows and the records (there is one console per deployment today; the lease is the guard for the day there are two). |
| +- Refusals come before writes: the row only after every check of §6.1's list; the record only after the reservation; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| +- Participants are box ids with labels, pinned at admission from the edges files; **one resolver** (`moves.ParticipantRole`, Task 6) decides a box's role for a move that carries `Participants`, in every box path (`offlineRound`, `offlineReassert`, the offline view's `hold` and `keep`, `offlineTarget`) and every console gate (`offlineSealCheck`, `offlineFlipCheck`, `offlineOpenCheck`, `offlineFinishCheck`, the job's report selection); a report whose `BoxID` is not a participant is ignored; a box publishing a participant's label with another id is "box <label> was replaced". A move without `Participants` (an operator move begun before this code) keeps today's region rule everywhere. |
| +- Timestamps in SQLite are Unix **nanoseconds** (`INTEGER`), zero meaning unset; stage and precheck requests compare by equality with the record's `time.Time` at nanosecond precision. |
| +- Copy rows pending Paul's review (Tasks 5 and 8). No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| +- Census: the box-side units take rows in `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus`, `rev3=PROPOSED-self-serve-moves-2`, with `@fields` for the new record fields and reviewed dispositions for changed frozen units and envelopes (`internal/host`, `internal/status`, `internal/moves`, `internal/console`); `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's with every added or changed writer (the new transitions, the box's report writers, the console's `self_serve_events`); both green. |
| - Local tests: touched packages with `-run`; CI runs the full suite. |
| ## Review Focus |
| -1. A console crash between the row insert and the reservation, between the reservation and the queued record (with the record write's response lost), and between the queued record and its reports: each resumes from the row and the evidence with exactly one reservation and one record move (Task 7, `TestAdmissionResumesFromEveryBoundary`). |
| -2. A reservation whose row says `requested` after the grace period: reclaimable only when the re-read record carries no move with its id **and** the row is absent or terminal; unreadable evidence, a malformed body or an uncertain cleanup status never reclaim (Task 3, `TestReclaimOnlyOnEvidence`). |
| -3. A deadline expiring while a box is mid-copy or mid-seal: `abort.requested_at` is set, the box's revert reports arrive later, `abort.source_reopened_at` needs the sealer's `lifted` and a source `reverted` report, and `abort.finished_at` needs every report (Task 8, `TestAbortIsInitiatedNotCompleted`). |
| -4. `cut` naming a stale or missing id → 409; a second `cut` during `cutting` → 202 with the view; `cancel` during `cutting` → 409; concurrent `cut` and `cancel` on a `ready` move: exactly one wins and the loser answers from the committed row (Task 5, `TestMutationsNameTheMoveAndRace`). |
| -5. At the three self-serve mutations: a plain instance token, an admin token, another app's move-capable token and a member's token get the move family's byte-identical 404 **before any record read**, against absent, eligible, ineligible and unreadable records; an invalid or revoked credential keeps today's authentication answer (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| +1. A console crash after a landed write but before its SQL update or its serial bump, at every transition boundary (queue, staging, restage, precheck, start, seal, flip, open, finishing, finish, abort), and a second crash during the recovery of each: the row is reconciled to the record, the bump is made once, and no second request is issued while the first is unanswered (Task 7, `TestEveryBoundaryRecovers`). |
| +2. A reservation older than the grace period: reclaimable only on positive evidence (the record carries no move with its id **and** the row is absent or terminal-and-released); a SQL read failure, a store read failure, a malformed body, an absent or incomplete cleanup status, or an outstanding delayed queue write never reclaim; a retry preserves the original `ReservedAt`; switch-off recovery re-acquires its own reservation (Task 3, `TestReclaimOnlyOnEvidence`). |
| +3. A deadline expiring while a box is mid-copy or mid-seal: `abort.requested_at` is set; `abort.source_reopened_at` needs the sealer's durable `tombstone_lifted` (when a tombstone existed) and every source's `reverted`; `abort.finished_at` needs every report; a source `reverted` before the sealer's lift sets nothing (Task 8, `TestAbortIsInitiatedNotCompleted`, with the real sealer). |
| +4. `cut` naming a stale or missing id → 409; a second `cut` during `cutting` → 202; `cancel` during `cutting` → 409; cancel committed first then cut → cut 409 from the row; cut committed first then cancel → cancel 409; a worker holding an older row snapshot cannot advance a cancelled row (Task 5, `TestMutationsNameTheMoveAndRace`, barrier-ordered). |
| +5. At the three self-serve mutations: a plain instance token, an admin token, another app's move-capable token and a member's token get the move family's byte-identical 404 **before any store or SQL evidence read**, against absent, eligible, ineligible and unreadable records, with admission on, off with an open row, and off with a terminal row; an invalid or revoked credential keeps today's authentication answer (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| --- |
| -### Task 1: The move-capable token |
| +### Task 1: The move-capable token and the self-serve gate |
| **Files:** |
| -- Modify: `internal/console/authstore.go` (`ALTER TABLE cli_tokens ADD COLUMN moves INTEGER NOT NULL DEFAULT 0` in the ALTER list; `claimRawNewAppToken` gains `moves bool`; `NewAppToken` and `NewAppTokenContext` gain it; `CLITokens` and `scanCLITokenRows` and every other `SELECT ... FROM cli_tokens` that feeds the scanner (the account-scoped listing too) select `moves` into `CLIToken.Moves`), `internal/console/svcprincipal.go` (`Principal.Moves bool`; `CLIPrincipal` selects and sets it for `cliKindApp` rows), `internal/console/apiinstances.go` (`handleAPIInstanceToken`: body `{"label","moves"}`; `moves` requires `s.store.Role(acct.ID, addr) == RoleOwner` else 403 "only an account owner can mint a move-capable instance token"; the audit detail "minted a move-capable instance token for <app> (cli), token <id>"), the tokens page template (a "move-capable" tag) |
| +- Modify: `internal/console/authstore.go` (`ALTER TABLE cli_tokens ADD COLUMN moves INTEGER NOT NULL DEFAULT 0`; `claimRawNewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `CLITokens` and every `SELECT ... FROM cli_tokens` feeding `scanCLITokenRows` select `moves`), `internal/console/svcprincipal.go` (`Principal.Moves bool`, `Principal.TokenID string` (the stored `token_hash`, the listing id); `CLIPrincipal` sets both), `internal/console/apiinstances.go` (`handleAPIInstanceToken`: body `{"label","moves"}`; `moves` requires `RoleOwner` else 403 "only an account owner can mint a move-capable instance token"; audit "minted a move-capable instance token for <app> (cli), token <id>"), the tokens page template |
| - Create: `internal/console/selfserve_gate.go` |
| - Test: `internal/console/instancetoken_moves_test.go`, `internal/console/selfserve_gate_test.go` |
| **Interfaces:** |
| -- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)` (the non-context wrapper, as today's); every existing caller passes `false`. |
| -- `CLIToken.Moves bool`; `Principal.Moves bool`. The token id for audit and `requested_by` is `CLIToken.ID` as the listing shows it (the stored hash), the value the revoke door takes. |
| -- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (acct *Account, app, host string, ok bool)`: `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` (principal, account, app: today's authentication answers for an invalid or revoked credential), then `host := r.PathValue("host")` and `ValidHostShape` (404 as `moveGate`), then **`moveCapable(principal, acct, app)` else `http.NotFound`** (the family's byte-identical 404), and returns **without reading the record**. `moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`. |
| -- The CLI (`carlos instance-token --moves`) is a carlos-repo change sending `{"moves": true}`. Existing tokens have `moves = 0`. When #155's instance-bound tokens merge, the mint door refuses `moves` with a host binding (a comment names it; no code until the column exists). |
| +- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; existing callers pass `false`. |
| +- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`: `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` then `p, _ = principalFrom(r.Context())` (today's authentication answers for an invalid or revoked credential), `host := r.PathValue("host")` with `ValidHostShape` (404 as `moveGate`), then `moveCapable(p, acct, app)` else `http.NotFound(w, r)`; **no store and no SQL read before this returns**. `moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`. |
| +- **The dispatcher** at the four doors (Task 5): `handleAPIInstanceMoveStatus`: `moveGate` as today, then `s.store.SelfServeMoveForHost` (a SQL failure → 502 "could not read the move"); a row → the self-serve view (Task 5's precedence); none → today's view. `Request`, `Cut`, `Abort`: when `s.cfg.MovesEnabled` → Phase 1 as today; else `selfServeGate` → the row read → a row, or admission on (`Request` only) → the self-serve body; else → the legacy body **after** re-running `moveGate` so its answers are byte-identical (the capability check is only reached when a self-serve body may run: with no row and admission off, the legacy body is served without a capability check, as today). |
| +- The CLI (`carlos instance-token --moves`) is a carlos-repo change. Existing tokens have `moves = 0`. #155's host-bound tokens: a comment names the future refusal. |
| -- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (201; `moves = 1`; `CLIPrincipal.Moves`; both listings show it; the audit line names the id); `TestMemberCannotMintAMoveCapableToken` (403; no row); `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (through the real routes with a counting store: a plain instance token, an admin service token, another app's move-capable token and a member's token → the 404 and zero record reads, against an absent record, an eligible one, a replica and a store whose record `Get` fails; the owner's move-capable token → past the gate; a revoked move-capable token → today's 401). |
| +- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (both listings, `Principal.Moves`, `Principal.TokenID` equals the listing id, the audit line); `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (a counting store **and** a counting SQL wrapper: for each of a plain instance token, an admin token, another app's move-capable token and a member's token, against an absent, an eligible, an ineligible and an unreadable record, with admission on, off with an open row, and off with a terminal row → the 404 with zero store reads and zero `self_serve_moves` reads; the owner's move-capable token passes; a revoked one → today's 401; with no row and admission off a plain instance token reaches the legacy body exactly as today). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'MoveCapable|Mint' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the move-capable instance token, owner-minted, checked before any move evidence is read`. |
| +- [ ] **Step 5: Commit** `console: the move-capable instance token, owner-minted, and the self-serve gate that authorises before any evidence is read`. |
| ### Task 2: The queued state, the precheck stamp and the self-serve fields on the record |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove \`json:"self_serve,omitempty"\``, `Participants *Participants \`json:"participants,omitempty"\``, `QueuedAt *time.Time \`json:"queued_at,omitempty"\``, `PrecheckAt *time.Time \`json:"precheck_at,omitempty"\``), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: `offline-queued` → ""; `offlineViewOf`: `State` passes through, `CanAbort` true, `NeverOffline` from `StartedOffline`), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`'s allowlist) |
| +- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove \`json:"self_serve,omitempty"\``, `Participants *Participants \`json:"participants,omitempty"\``, `QueuedAt *time.Time \`json:"queued_at,omitempty"\``, `PrecheckAt *time.Time \`json:"precheck_at,omitempty"\``, `ReturnOf string \`json:"return_of,omitempty"\``), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: `offline-queued` → ""; `offlineViewOf`: `CanAbort` true in queued, `NeverOffline` from `StartedOffline`), `internal/provision/offline_move_test.go` (the allowlist) |
| - Test: `internal/provision/offline_queue_test.go` |
| **Interfaces:** |
| - `const OfflineQueued = "offline-queued"`. |
| -- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\` }` (`ID` is the row id, equal to the move id; `Incarnation` the lifetime admitted; `RequestedBy` is `"token:" + CLIToken.ID + " " + request's requested_by`). |
| +- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\` }`. |
| - `type Participant struct { BoxID string \`json:"box_id"\`; Label string \`json:"label"\` }`; `type Participants struct { Source []Participant \`json:"source"\`; Target Participant \`json:"target"\` }`. |
| -- `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, now time.Time) func(*Record) error`: the refusals of `OfflineStage` (a move in flight, a seed block, an open run, the same region) plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager` (the source participant's label), `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: now`, `QueuedAt: &now`, `StagedAt: &now` (the pre-copy intent exists, so `StartedOffline()` is false until the start), `PrecheckAt: &now`. `move_seq` unchanged. |
| -- `func OfflineQueuedToStaging(id string, now time.Time) func(*Record) error`: `offline-queued` → `offline-staging`, `StagedAt = &now` (the first stage request). `move_seq` unchanged. |
| -- `func OfflinePrecheck(id string, now time.Time) func(*Record) error`: allowed in `offline-queued` and `offline-staging`; sets `PrecheckAt = &now`; nothing else; `move_seq` unchanged. |
| -- `func OfflineSetEstimate(id string, seconds int) func(*Record) error`: allowed in `offline-queued`, `offline-staging`; sets `SelfServe.EstimateSeconds`; `errRecordUnchanged` when equal. |
| -- `OfflineStartStaged` sets `SelfServe.OfflineAt = &now` when `SelfServe != nil` (plan 1's `+1` applies). |
| -- `OfflineAbort` from `offline-queued`: the move clears at once with `LastMove = {ID, aborted, At: now, OfflineS: 0}` and `NeverOffline` semantics (as an abort-from-staging finish); `move_seq` unchanged. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing offline-state sentence (`offlineStateRefusal`'s words). |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (queue → the fields above, `StartedOffline() == false`; to staging → `StagedAt` moves; start-staged → `OfflineAt`); `TestOfflineQueueRefusesARecreatedLifetime` (`Incarnation` mismatch → `ErrMoveConflict` with the sentence; nothing written); `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestAbortFromQueuedClearsAtOnce` (after an earlier completed move at `move_seq 3`: `LastMove` aborted, `move_seq` still 3, `NeverOffline`); `TestOfflineSetEstimateChangesNothingElse`; the allowlist test. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|AbortFromQueued|SetEstimate|DiffersOnlyWhere' -count=1`. |
| +- `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, now time.Time) func(*Record) error`: `OfflineStage`'s refusals plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager: p.Source[0].Label`, `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: now`, `QueuedAt: &now`, `StagedAt: &now`, `PrecheckAt: &now`, `Seq: rec.MoveSeq`; `move_seq` unchanged. |
| +- `func OfflineQueuedToStaging(id string, now time.Time) func(*Record) error` (`StagedAt = &now`); `func OfflinePrecheck(id string, now time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &now`); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, now time.Time) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: now, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour, so the console calls `OfflineAbortAt` for a queued move and `OfflineAbort` otherwise; a repeated abort on a reverting record stays a success without a write). `OfflineStartStaged` sets `SelfServe.OfflineAt = &now` when `SelfServe != nil`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove` (after an earlier move at `move_seq 3`: `LastMove` aborted at `now`, `move_seq` 3; from any other state it behaves as `OfflineAbort`; a repeated abort is a no-write success); `TestOfflineSetEstimateRefusesWithoutSelfServe`; the allowlist test. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|DiffersOnlyWhere' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass**, then the record-write census. |
| -- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp and the self-serve move fields`. |
| +- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort and the self-serve move fields`. |
| ### Task 3: Per-box reservations in active.json, with a legacy projection |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineActive` gains `V int \`json:"v,omitempty"\`` and `Reservations map[string]OfflineReservation \`json:"reservations,omitempty"\``; the legacy fields stay as the **projection**: `Host, Op, Account, App, At` of the oldest `move` reservation, so an older console reads a busy lock while anything is reserved), `internal/console/apiinstancemove_offline.go` (the start and stage doors call `reserveBoxes` with the selection Task 5 returns; `offlineReleaseLock` → `releaseBoxes`) |
| +- Modify: `internal/moves/offline.go` (`OfflineActive` gains `V int \`json:"v,omitempty"\`` and `Reservations map[string]OfflineReservation \`json:"reservations,omitempty"\`` keyed by box id; the legacy fields are the **projection**), `internal/console/apiinstancemove_offline.go` (the start and stage doors reserve from the selection; `offlineReleaseLock` → `releaseBoxes`) |
| - Create: `internal/console/offlinereserve.go` |
| - Test: `internal/console/offlinereserve_test.go` |
| **Interfaces:** |
| -- `type moves.OfflineReservation struct { Kind string \`json:"kind"\`; ID string \`json:"id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; ReservedAt time.Time \`json:"reserved_at"\` }` (`Kind` is `move` or `cleanup`). |
| -- `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`: reads `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}`; refuses `busy` ("<host> is moving offline. move one instance at a time", naming the holder's host) when any participant (`sel.Source[*].BoxID`, `sel.Target.BoxID`) is held by another `ID`, unless that reservation is reclaimable; otherwise writes every participant's entry **and the projection** in one conditional `Put` on the object's ETag (create-only when absent); a CAS conflict re-reads once and retries, then `busy`. **Concurrency gate:** while `cfg.SelfServeMoves != "offline"` the helper refuses when any reservation exists at all (today's deployment-wide semantics) and still writes the projection, so an older console in the same deployment never sees a table with entries it would delete as its own; per-box concurrency begins with the switch, which is turned on only once every console of the deployment runs this code (§9). |
| -- `func (s *Server) releaseBoxes(ctx context.Context, id string) error`: removes every entry whose `ID == id`, rewrites the projection from the remaining oldest `move` entry, and when the table is empty **deletes the object** (`DeleteIfMatch`), so an old console's `offlineTakeLock` finds absence rather than a `Host == ""` body it reads as unreadable; one conditional write; absence is success; a failure is logged and the row keeps `release_pending` (Task 8). |
| -- `func (s *Server) reservationsOf(ctx context.Context) (map[string]moves.OfflineReservation, string, error)`; `func (s *Server) reservationsFor(ctx context.Context, id string) ([]string, error)`. |
| -- Reclaim: an entry is reclaimable when `time.Since(ReservedAt) >= offlineLockGrace` (2 minutes; a retry re-reserves with the **original** `ReservedAt`) and, for `Kind == move`, the record of `(Account, App, Host)` (judged read) carries no offline move with `ID` **and** the console's row with that id is absent or terminal; for `Kind == cleanup` (plan 4), the owning box's status report, no older than two status intervals, lists no `RetiredSummary` with that id, or lists it `cleaned`/`superseded` with no root request outstanding. A read that fails, a malformed body, or a cleanup status that is absent or stale → not reclaimable (`unreadable`, the existing 502 sentence). |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestDisjointBoxesReserveAtOnceWithTheSwitch` and `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyReservesEveryBox`; `TestReclaimOnlyOnEvidence` (young → busy; old with the record carrying the move → busy; old with a non-terminal row → busy; both gone → reclaimed with the original `ReservedAt` preserved on the retry; a failing record `Get`, a malformed body, a cleanup kind with a stale status → unreadable, nothing written); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody` (replay today's `offlineTakeLock` and `offlineReleaseLock` logic from a fixture copy against a v2 body: take → busy; release of its own op → leaves other reservations because the projection's `Op` is not its own; an empty table is never left behind); `TestOperatorDoorsUseReservations`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|OperatorDoorsUse' -count=1`. |
| +- `type moves.OfflineReservation struct { Kind string \`json:"kind"\`; ID string \`json:"id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; ReservedAt time.Time \`json:"reserved_at"\` }` (`Kind` is `move` or `cleanup`; an entry with an unknown kind, an empty id or host, or a zero time is malformed: the whole object is `unreadable` until an operator repairs it). |
| +- `func (s *Server) reserveBoxes(ctx context.Context, boxes []provision.Participant, r moves.OfflineReservation) *offlineRefusal`: reads `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}` and is normalised into the table on the first new write; refuses `busy` ("<host> is moving offline. move one instance at a time") when any participant is held by an entry with another `(Kind, ID)`, unless reclaimable; an entry with the same `(Kind, ID)` is this operation's own (a retry) and keeps its `ReservedAt`; writes every participant's entry and the projection (`Host, Op, Account, App, At` of the oldest `move` entry; of the oldest entry of any kind when no move holds) in one conditional `Put`; a CAS conflict re-reads once and retries, then `busy`. **Concurrency gate:** while `cfg.SelfServeMoves != "offline"` the helper refuses when any entry of another operation exists (today's deployment-wide semantics). |
| +- `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error`: removes every entry with that `(Kind, ID)`, rewrites the projection, deletes the object when empty (`DeleteIfMatch`); one conditional write; absence is success. |
| +- Reclaim: an entry is reclaimable when `time.Since(ReservedAt) >= offlineLockGrace` and, for `move`: the judged record of `(Account, App, Host)` carries no offline move with `ID` **and** `SelfServeMove(id)` is absent or terminal with `release_pending = 0` **and** no `request_issued` admission write is outstanding for that row (a row in `reserving` whose `request_issued` is set is not reclaimable until its own recovery resolves it); for `cleanup` (plan 4): the owning box's status report (`status/boxes/<label>.json` of the box id's label, `GeneratedAt` within 2 status intervals, `Metrics.BoxID` equal to the key) lists the id in a terminal state with `RootRequest == ""` and `ReleasePending == false`, or its `Retirements` block is **complete** (`RetirementsComplete`) and lists no such id. Any read failure, a malformed body, or an absent or incomplete status → not reclaimable. |
| +- **Rollout and rollback, stated:** per-box concurrency starts when the switch is turned on, after every console runs this code; an older console cannot be reintroduced while concurrent reservations exist, because its release deletes the whole object when the projection names its own move (the hazard `TestOldConsoleReleaseErasesATable` documents) and its take refuses everything else; the supported sequence is roll consoles → turn the switch on; rollback is switch off → wait for the table to drain to one entry → the old console. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnReservationIsIdempotentAndKeepsReservedAt`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive and each refusing case above, including a SQL failure and an outstanding admission write); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody` (the old take is busy on any table; the old release leaves a table whose projection names another move; `TestOldConsoleReleaseErasesATable` documents the hazard the rollout sequence avoids); `TestSwitchOffRecoveryReacquiresItsOwnReservation`; `TestOperatorDoorsUseReservations`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|OperatorDoorsUse|Malformed' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Offline' -count=1`. |
| -- [ ] **Step 5: Commit** `console: active.json holds per-box reservations with a legacy projection; the operator doors reserve their participants`. |
| +- [ ] **Step 5: Commit** `console: active.json holds per-box reservations with a legacy projection and a stated rollout`. |
| -### Task 4: The self_serve_moves table, the console lease and the pass skeleton |
| +### Task 4: The self_serve_moves journal, the console lease and the pass skeleton |
| **Files:** |
| -- Modify: `internal/console/store.go` (`migrate`), `internal/console/store_gate.go` (`claimSQLTables`: `"self_serve_moves": {Sqid: "account", App: "app"}`, `"move_timings": {NonApp: "..."}`), `internal/console/claim_workers.go` (register `"StartSelfServeMovesPass"`), `cmd/console/main.go` (start it; `cfg.SelfServeMoves = os.Getenv("CARLOS_SELF_SERVE_MOVES")`; `armMovesPaging`: arm when `cfg.HostClaim || cfg.SelfServeMoves == "offline" || the store has open self-serve rows at start`), `internal/console/server.go` (`Config.SelfServeMoves string`; `func (s *Server) selfServeAdmits() bool`) |
| +- Modify: `internal/console/store.go`, `internal/console/store_gate.go` (`"self_serve_moves"`, `"self_serve_events"`: `{Sqid: "account", App: "app"}`; `"move_timings"`: `{NonApp: ...}`), `internal/console/claim_workers.go`, `cmd/console/main.go` (`cfg.SelfServeMoves`; `armMovesPaging(ctx, cfg, store, getenv)`: arm when `cfg.HostClaim`, or `cfg.SelfServeMoves == "offline"`, or the store has open rows or rows with `release_pending`, or `active.json` has any entry (a store read failure → arm, logged)), `internal/console/server.go` (`Config.SelfServeMoves string`; `func (s *Server) selfServeAdmits() bool`) |
| - Create: `internal/console/selfserve_store.go`, `internal/console/selfserve_pass.go`, `internal/console/selfserve_lease.go` |
| - Test: `internal/console/selfserve_store_test.go`, `internal/console/selfserve_pass_test.go` |
| **Interfaces:** |
| -- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, source_boxes TEXT, target_box TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, release_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| -- Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. Outcomes: `done`, `aborted`. |
| -- `type SelfServeMove struct { ID, Account, App, Host string; Incarnation uint64; From, To string; SourceBoxes []provision.Participant; TargetBox provision.Participant; DisplayName, RequestedBy, Phase string; PhaseStartedAt, StepDeadline, WindowDeadline time.Time; StageRequest, PrecheckRequest time.Time; Rounds int; EstimateSeconds int; EstimateTerms estimateTerms; Reason string; HealthAttempts int; HealthFirstOKAt, HealthLastProbeAt time.Time; HealthLastError string; FinishGate string; Outcome, OutcomeReason, Warning string; AbortRequestedAt, SourceReopenedAt, AbortFinishedAt time.Time; ReleasePending bool; CreatedAt, UpdatedAt time.Time }`. |
| -- On `*Store`: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)` (one transaction: insert, or return the open row the unique index protects); `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (*SelfServeMove, error)` (the open one, else the newest terminal one, nil when none); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1`, oldest first); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, now time.Time) (bool, error)` (`UPDATE ... WHERE id = ? AND phase = ?`); `SetSelfServeStageRequest(id string, at time.Time) error`, `SetSelfServeRounds(id string, n int) error`, `SetSelfServePrecheckRequest(id string, at time.Time) error`, `SetSelfServeEstimate(id string, seconds int, terms estimateTerms) error`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error`, `SetSelfServeFinishGate(id, gate string) error`, `SetSelfServeWarning(id, warning string) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE id = ? AND phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened(id string, at time.Time) error`, `SetSelfServeAbortFinished(id string, at time.Time) error`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (sets `release_pending = 1`), `ClearSelfServeRelease(id string) error`. |
| -- Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `func (s *Server) holdSelfServeLease(ctx context.Context, now time.Time) (held bool, err error)`: the body `{console, until}`; create-only when absent; `If-Match` when ours or expired; another console's live lease → not held (logged once per minute). The pass runs only while held, renewing each tick; losing the lease stops the pass's writes until it is held again. |
| -- `func (s *Server) StartSelfServeMovesPass(ctx context.Context)` (runs when `selfServeAdmits()` or `OpenSelfServeMoves()` is non-empty at start; every 5 s: the lease, then `s.selfServeTick(ctx, time.Now())`); `func (s *Server) selfServeTick(ctx context.Context, now time.Time)`: the orphan reservation scan (Task 8), then for each open row `s.selfServeStep(ctx, row, now)`, at most one step, errors logged per row. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic` (two concurrent admits for one host → one row, the loser gets the open row); `TestAdvanceIsPredicated` (a stale `from` → false, row unchanged); `TestRequestAbortOnce`; `TestOpenRowsIncludePendingRelease`; `TestPassIsOffWithoutAdmissionAndNoOpenRows`; `TestPassRunsOpenRowsWithoutAdmission`; `TestTickAdvancesEachRowAtMostOneStep`; `TestConsoleLeaseExcludesASecondConsole` (two in-process consoles over one `DirStore`: one holds, the other's tick writes nothing; after expiry the other takes it); `TestPagingArmedForSelfServe` (the real `armMovesPaging` with `HostClaim` off and the switch on → armed; off with open rows → armed; off with none → as today); `TestClaimStoreOpensWithSelfServeTables`. |
| +- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, stage_request INTEGER, consumed_round INTEGER, rounds INTEGER, precheck_request INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_state TEXT, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| +- Table `self_serve_events (id TEXT, phase TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, phase))`: the claims-off move log and the member-stream outbox (Task 9). |
| +- Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. |
| +- `type SelfServeMove struct { ID, Account, App, Host string; Incarnation uint64; From, To string; Participants provision.Participants; DisplayName, RequestedBy, Phase string; PhaseStartedAt, StepDeadline, WindowDeadline, StagingStartedAt time.Time; RequestIssued time.Time; RequestKind string; StageRequest, ConsumedRound time.Time; Rounds int; PrecheckRequest time.Time; EstimateSeconds int; EstimateTerms estimateTerms; Reason string; HealthConsecutiveOK int; HealthFirstOKAt, HealthLastProbeAt time.Time; HealthLastError string; FinishGate, NotifiedState string; Outcome, OutcomeReason, Warning string; AbortRequestedAt, SourceReopenedAt, AbortFinishedAt time.Time; EffectsDone, ReleasePending bool; CreatedAt, UpdatedAt time.Time }`. |
| +- On `*Store`: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open *SelfServeMove, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0`); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, now time.Time) (bool, error)` (`UPDATE ... WHERE id = ? AND phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done'))`); `IssueSelfServeRequest(id, kind string, at time.Time) (bool, error)` (`WHERE request_issued = 0`); `ConfirmSelfServeRequest(id string, at time.Time) error` (`stage_request`/`precheck_request = at`, `request_issued = 0`); `ConsumeSelfServeRound(id string, request time.Time) (bool, error)` (`WHERE consumed_round <> ?` then `rounds = rounds + 1, consumed_round = ?`); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` set on the first ok and kept), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotified(id, state string) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE id = ? AND phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`; `InsertSelfServeEvent(e SelfServeEvent) (inserted bool, err error)`, `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`, `MarkSelfServeEventEmitted(id, phase string) error`. |
| +- Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `type consoleLease struct { Console string; Until time.Time }`; `func (s *Server) acquireSelfServeLease(ctx context.Context, now time.Time) (*leaseToken, bool, error)`; `func (t *leaseToken) renew(ctx context.Context, now time.Time) error` (every 20 s during a tick, from the tick's loop between steps and inside the health probe wait); `func (t *leaseToken) valid(now time.Time) bool` (checked by `selfServeApply` and every SQL mutation of the pass before writing; false → the tick ends with "lease lost"). Another console's live lease → not acquired (logged once per minute). |
| +- `func (s *Server) StartSelfServeMovesPass(ctx context.Context)` (runs when `selfServeAdmits()`, or `OpenSelfServeMoves()` is non-empty, or `active.json` has any entry at start; every 5 s: acquire or renew the lease, `s.selfServeTick`); `func (s *Server) selfServeTick(ctx context.Context, now time.Time)`: the reservation scan (Task 8), then per open row `s.selfServeStep` (at most one step), errors logged per row. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestAdvanceIsPredicatedOnPhaseAndNoAbort` (a cancelled row cannot advance to `restage`; it can to `reverting`); `TestIssueConfirmConsume` (issue refused while one is outstanding; consume counts a request once); `TestRequestAbortOnce`; `TestHealthCounters`; `TestOpenRowsIncludePendingReleaseAndEffects`; `TestPassStartsOnAdmissionOpenRowsOrReservations` (each alone; none → off); `TestPassRunsOpenRowsWithoutAdmission`; `TestTickAdvancesEachRowAtMostOneStep`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseLapseStopsWrites` (the holder is paused past 60 s with a barrier; another console acquires; the paused holder's next write is refused); `TestPagingArmedForSelfServe` (the real `armMovesPaging` with its store argument: each condition); `TestClaimStoreOpensWithSelfServeTables`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'SelfServe|ConsoleLease|PagingArmed|ClaimStoreOpensWithSelfServe' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self_serve_moves table with atomic admission and predicated phases, the console lease and the pass skeleton`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseLapse|PagingArmed|IssueConfirm|ClaimStoreOpensWithSelfServe' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self_serve_moves journal with issued and consumed identities, the events table, the console lease and the pass skeleton`. |
| -### Task 5: The shared start validation, the door bodies and the view |
| +### Task 5: Record eligibility, the shared start validation, the door bodies and the view |
| **Files:** |
| -- Modify: `internal/console/apiinstancemove_offline.go` (`offlineBeginChecks` becomes a thin wrapper over `offlineStartValidation`; the start and stage doors use the returned selection; `offlineStepFor`'s `offline-staging` branch calls `offlineStartValidation` too, so the start-from-staged path carries every check), `internal/console/apiinstancemove.go` (`handleAPIInstanceMoveRequest`, `Cut`, `Abort`, `Status`: when `!s.cfg.MovesEnabled` and (`s.selfServeAdmits()` or a self-serve row exists for the host) dispatch to the self-serve bodies; otherwise unchanged) |
| +- Modify: `internal/console/apiinstancemove.go` (`moveGate`'s record checks move into `recordEligibility`; `moveGate` calls it so its bytes are unchanged; the dispatcher of Task 1), `internal/console/apiinstancemove_offline.go` (`offlineBeginChecks` becomes a wrapper over `offlineStartValidation`; the start and stage doors use the selection; `offlineStepFor`'s staging branch calls `offlineStartValidation` so the start-from-staged path carries every check; the stage door accepts `return_of` (Task 8's operator return)) |
| - Create: `internal/console/selfserve_door.go`, `internal/console/selfserve_view.go` |
| - Test: `internal/console/selfserve_door_test.go`, `internal/console/apiinstancemove_offline_snapshot_test.go` |
| **Interfaces:** |
| -- `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget }`. |
| -- `func (s *Server) offlineStartValidation(ctx context.Context, acct *Account, app, host string, rec *provision.Record, region string, stage bool) (offlineSelection, *offlineRefusal)`: **every** check `offlineBeginChecks` and the stage door make today, in their order and with their sentences: `rec.Seed.MoveBlock()` (the seed block; a seeded record whose seed does not block is movable), `rec.Role != ""`, `len(rec.Replicas) > 0`, `rec.Steering != "" || rec.Unit != ""`, `offlineClaimEvidence`, the region (invalid, same region), the manifest's regions, `readOfflineFleet`, the home-region placement rule, `offlineSelect`; with `stage`: every box of both regions carries `offline-stage-1`, the distinct-bucket rule, the stager pinned; plus, for every participant, a non-empty `BoxID` (else the existing older-release sentence). The doors' bytes are pinned by `TestOperatorDoorAnswersAreUnchanged` (snapshots of every refusal of every operator door, taken from main before the change, replayed after). |
| -- `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}`: `selfServeGate` (Task 1; no record read yet); `s.readRecord` (plan 1's judged read; absent or stale → 404; unreadable → 502 "could not read the instance"); then in order: `rec.ActiveRun.Open()` → 409 the existing run sentence; `rec.Move != nil` → 409 `offlineConflict`'s existing "a move to X is already Y"; an open row for the host (`SelfServeMoveForHost` open) → the same 409 (the row is the admission); the body; `start_at` non-empty and not within 60 s of now → 400 "scheduled starts are not available yet"; `display_name` over 80 runes or with a control character → 400 "display_name must be at most 80 characters of plain text"; `offlineStartValidation(..., region, stage: true)`; the source region has exactly one box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the status facts (Task 6) for the source box fresh and known → else 409 "the source box has not reported recently. try again in a minute"; `files_bytes > archive.Cap` → 409 "this account's files are too large to move this way"; the lane (`Lanes` of the source box's report names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `s.migrateHostLifetimeLocked` under `withHostname` (plan 1; a conflict or settling → 409 "another change to this instance is still settling. try again in a minute"). Then `AdmitSelfServeMove({ID: moves.NewMoveID, Incarnation: rec.Incarnation, ..., Phase: requested, RequestedBy, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the move-in-flight sentence from the open row; `bumpSerial`; 202 with the view. |
| -- `GET .../move`: `moveGate` (today's; any instance token); a self-serve row for the host (open or terminal) → the self-serve view; else today's view. |
| -- `GET .../move?estimate=1®ion=uk`: `moveGate`; the refusal list above through the lane, without writing or migrating; 200 `{"estimate_seconds": n, "files_bytes": b, "replica_bytes": r}` or the refusal. |
| -- `POST .../move/cut {"id"}`: `selfServeGate`; `SelfServeMoveForHost` open; none, or `row.ID != body.id` → 409 "this is not the move you confirmed"; phase `ready` → `AdvanceSelfServe(id, ready, restage, ...)`; `true` → 202; `false` → re-read: a cut phase → 202 (idempotent), `aborted`/abort requested → 409 "the move was cancelled"; any other phase → 409 "the move is not ready to cut yet. it is <phase>". |
| -- `DELETE .../move {"id"}`: `selfServeGate`; the row and id as above; `RequestSelfServeAbort(id, [requested, reserving, queued, reports, staging, prechecks, ready], now)`; `true` → 202 (the job runs the abort); `false` → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| -- The view: `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`, `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`, `type selfServeLast struct { ID, Outcome, Reason, Warning, From, To string; At int64 }` (unix seconds; `from`/`to` through `homeRegionOut`). `State`: the row terminal → `none`; `requested`..`prechecks` → `preparing`; `ready` → `ready`; the rest → `cutting`. `Kind` is `offline`. `UpdatedAt` is the row's. Phase 1's `moveView` is untouched (a Phase-1 console never has a self-serve row). |
| -- Sentences new here are copy rows: "scheduled starts are not available yet"; "display_name must be at most 80 characters of plain text"; "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; "the source box has not reported recently. try again in a minute"; "this account's files are too large to move this way"; "no staging lane is configured from <from> to <to>"; "this is not the move you confirmed"; "the move was cancelled"; "the move is not ready to cut yet. it is <phase>"; "the move was confirmed and cannot be cancelled. it completes or returns on its own"; "no move is in flight"; "waiting for another move on this box". |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged`; `TestSwitchOffIsByteIdentical` (no row: the four doors answer as main; Phase 1 fixtures unchanged); `TestPrepareRefusesBeforeWriting` (one subtest per refusal: no row, no record write, no reservation, no migration); `TestPrepareMigratesALegacyRecordThenAdmits`; `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (wrong id, no id, an operator move's host; a second cut during cutting → 202; cancel during cutting → 409; concurrent cut and cancel on `ready` → one 202 and one 409 from the committed row); `TestViewMapsPhases` (one subtest per phase; `abort` facts; `last` after done and aborted; the view with the switch off and a terminal row); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|Prepare|MutationsName|ViewMaps|EstimateDoor' -count=1`. |
| +- `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal`: `moveGate`'s checks in order: `Kind` not instance (`c.kind`), `Addr`/`Addrs` (`c.addr`), fleet placement (`c.fleet`), `Pool` (`c.pool`), `!HibernateOrDefault()` (`c.hibernate`); 400s. |
| +- `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string }`. |
| +- `func (s *Server) offlineStartValidation(ctx context.Context, acct *Account, app, host string, rec *provision.Record, region string, stage bool, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)`: `recordEligibility(rec, mainMoveCopy())`, then every check of `offlineBeginChecks` and the stage door in order with their sentences (`rec.Seed.MoveBlock()`, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`, the region, the manifest's regions, `readOfflineFleet`, the home-region rule, `offlineSelect`; with `stage`: `offline-stage-1` on every box, the distinct-bucket rule, the stager pinned); with `requireIDs` (the self-serve door) every participant needs a non-empty `BoxID` (else the older-release sentence); the operator doors pass `requireIDs: false` and keep accepting publications without ids (compatibility, stated). `returnOf` is Task 8's. `TestOperatorDoorAnswersAreUnchanged` snapshots every operator refusal, including a malformed body combined with each class refusal, from main and replays them. |
| +- `POST .../move` (self-serve body): `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` → 400 "display_name must be at most 80 characters of plain text"; `offlineStartValidation(..., region, stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source box's status facts (Task 6: the report of the pinned source's label, `Metrics.BoxID == Source[0].BoxID`, `GeneratedAt` within 10 minutes, `Hosts[host].Known`, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` becomes the returned record and `recordEligibility`, the run and move checks run again on it. Then `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. |
| +- `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating (a legacy record is estimated as is); 200 `{"estimate_seconds": n, "files_bytes": b, "replica_bytes": r}` or the refusal. |
| +- `POST .../move/cut {"id"}`: the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `AdvanceSelfServe(id, ready, restage, ...)` → 202; false → re-read: a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". |
| +- `DELETE .../move {"id"}`: the row and id; `RequestSelfServeAbort(id, [requested..ready], now)` → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| +- **The view's precedence** (`selfServeViewOf(rec, open, terminal)`): (1) the record carries an offline move with `SelfServe != nil` → the open row's view (`cutting`/`ready`/`preparing` by phase); (2) no such record move but an open row whose `Incarnation == rec.Incarnation` → `preparing` (pre-record phases); (3) the record carries another move (an operator's) → today's legacy view, no `last`; (4) otherwise a terminal row whose `Incarnation == rec.Incarnation` (or the record is absent and the row is the newest) and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| +- Copy rows: as rev 2 plus "self-serve moves are not enabled on this console"; "could not measure the account's files. try again in a minute"; "could not read the move". |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged`; `TestSwitchOffIsByteIdentical`; `TestRecordEligibilityIsMoveGatesChecks`; `TestPrepareRefusesBeforeWriting` (one subtest per refusal including admission off, over-cap and truncated facts, a stale status report, a status report of another box id); `TestPrepareMigratesALegacyRecordBoundThenAdmits` (a recreate between the observation and the migration → 409 settling, nothing admitted; the admitted row carries the migrated record's incarnation and its eligibility was re-checked); `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (barriers: cancel commits before cut → cut 409; cut commits before cancel → cancel 409; a worker with an older row snapshot cannot advance a cancelled row; a second cut during cutting → 202); `TestViewPrecedence` (each of the five cases, a terminal row older than 7 days, a recreation after a terminal row, an operator move after a terminal row); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|RecordEligibility|Prepare|MutationsName|ViewPrecedence|EstimateDoor' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Move' -count=1`. |
| -- [ ] **Step 5: Commit** `console: one start validation for the operator doors and the job; the self-serve door, its refusals and its view`. |
| +- [ ] **Step 5: Commit** `console: record eligibility and one start validation shared by the operator doors and the job; the self-serve door, its refusals and its view`. |
| -### Task 6: The boxes' reports: the lane, the source precheck, target-ready, and the per-host status block |
| +### Task 6: The boxes' reports: the participant resolver, the lane, the source precheck, target-ready, the staged inventory, and the per-host status block |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineReport` gains `BoxID string`, `Request time.Time` (the `PrecheckAt` answered, for `precheck` and `ready`), `Lane *LaneReport`, `Ready *TargetReadyReport`, `Precheck *SourcePrecheck`, `RoundCopiedBytes int64`, `RoundDeferredBytes int64`, `RoundDeferredObjects int`, `Rounds int` (successful rounds so far); roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`; `ValidateOfflineReport` admits the new role and phases and requires `BoxID` on them; `offlineCarries` unchanged), `internal/host/movesoffline.go` (`offlineRound`: participation is `cfg.BoxID` ∈ `Participants` when the record carries `Participants`, else today's region rule (the operator compatibility path); `offline-queued` → the source participant runs `offlineLaneProbe` once per move and `offlineSourcePrecheck` for the current `PrecheckAt`, the target participant runs `offlineTargetReady` for the current `PrecheckAt`; `offline-staging` → the stager round as today **and** both participants re-report when `PrecheckAt` is newer than their last report's `Request`; `sourceReport` (the stop report) gains the precheck facts too), `internal/host/movesoffline_stage.go` (the stager report publishes `RoundCopiedBytes`, `RoundDeferredBytes`, `RoundDeferredObjects` from `MirrorRound`, and `Rounds`), `internal/host/movesoffline_land.go` (`offlineTargetReady`), `internal/host/agent.go` (the status tick: `p.Hosts`, `p.Lanes`), `internal/status/snapshot.go` (`Probes.Hosts map[string]HostFacts`, `Probes.Lanes []string`; `CollectBox`'s attachment condition includes `len(p.Hosts) > 0 || len(p.Lanes) > 0`; `BoxMetrics.Hosts`, `BoxMetrics.Lanes` private) |
| +- Modify: `internal/moves/offline.go` (`OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `Staged map[string]StagedObject` (exists; now the stager's complete destination inventory), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`; **the validator matrix** in `ValidateOfflineReport`: role `lane` → phase `lane`, `Lane != nil`, `BoxID != ""`; role `source` + phase `precheck` → `Precheck != nil`, `BoxID`, `Request` non-zero; role `target` + phase `ready` → `Ready != nil`, `BoxID`, `Request`; role `stager` + phase `staged` → `StagedBucket`, `StagedRegion` non-empty and `Staged` present (nil is "not reported"); role `sealer` + phase `reverting` → `TombstoneLifted` set; any report of a move with `Participants` → `BoxID` non-empty; `func ParticipantRole(m *provision.MoveState, boxID, label, placement, home string) (source, target, sealer, stager bool)` (with `Participants`: by `BoxID` only; without: today's region and label rules)), `internal/host/movesoffline.go` (`moveCfg.BoxID` from the agent's box id, the value the edges writer publishes; `offlineRound` and `offlineReassert` use `ParticipantRole`; `offline-queued` → the source participant: `laneProbe` (the existing `movesstore.go:90` helper over `cfg.Versioned`, key `moves.LaneKey(id)` in the target bucket) once per move then the `lane` report, and `offlineSourcePrecheck` for the current `PrecheckAt`; the target participant: `offlineTargetReady` for `PrecheckAt`; `offline-staging` → the stager round and both participants re-reporting when `PrecheckAt` is newer than their last `Request`; `sourceReport` (the stop report) gains the precheck facts), `internal/host/movesoffline.go` (`offlineView.hold`/`keep` consult `ParticipantRole` when the move carries `Participants`), `internal/host/movesoffline_seal.go` (`offlineTarget`: with `Participants`, the target participant's edges file by `BoxID`, else by region; `offlineSealRevert`: after `LiftTombstone` succeeds, **before** the prefix cleanup, write a sealer report `{Phase: reverting, TombstoneLifted: true, TombstoneLiftedAt}`; the final `reverted` report keeps both fields; the normal seal's final-copy bytes go in `FinalCopiedBytes`), `internal/host/movesoffline_stage.go` (the stager report: `RoundCopiedBytes/RoundDeferredBytes/RoundDeferredObjects` from `MirrorRound`, `Rounds`, and `Staged` = `MirrorStore.List(StageBucket, moves/<host>/)` taken after the round, `StagedBucket/StagedRegion` from the record), `internal/host/movesoffline_land.go` (`offlineTargetReady`), `internal/host/agent.go` (the status tick: `p.Hosts`, `p.Lanes`, `p.BoxID`), `internal/status/snapshot.go` (`Probes.Hosts map[string]HostFacts`, `Probes.Lanes []string`, `Probes.BoxID string`; `CollectBox`'s attachment condition includes them; `BoxMetrics.Hosts`, `BoxMetrics.Lanes`, `BoxMetrics.BoxID` private), `internal/host/movespass.go` (`armOffline`: `cfg.Versioned` (today assigned only in `armPhase1`) and `cfg.LaneProbe` are armed for offline-capable boxes too) |
| - Test: `internal/moves/offline_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| **Interfaces:** |
| -- `type moves.LaneReport struct { Bucket, Region, Key string; Put, Listed, Deleted bool; Error string }` (key `moves/@lanes/<move id>/<box id>.probe`). |
| -- `type moves.Fact struct { Known bool; OK bool; Detail string }` (`Known == false` is a probe that failed: `Detail` says why; the console refuses "could not determine <fact> on <label>"). |
| -- `type moves.TargetReadyReport struct { RouteAbsent, TimersAbsent Fact; LiveLink Fact (OK: absent, or naming ChannelBuild; Detail: the target); ChannelBuild string; ProgramsFile Fact (OK: absent or naming ChannelBuild); DataPaths map[string]Fact (one entry per path of §7.3, OK = absent); ParkDirClean Fact; DiskFreeBytes int64; DiskKnown bool; MovesPrefix PrefixFacts; InstancesPrefix PrefixFacts; LockAbsent Fact; Capabilities []string; SecretsKeyFP string; StagePathArmed Fact }`; `type PrefixFacts struct { Known bool; Objects map[string]StagedObject }` (every current object under the prefix with its ETag and size; empty means clean). |
| -- `type moves.SourcePrecheck struct { FilesBytes int64; FilesEntries int; ArchiveEligible Fact; ExtraDirs []ExtraDir; Build string; StagePathArmed Fact; ExtraDirsRemoved []string; ExtraDirsLeft []ExtraDir }`; `type ExtraDir struct { Path string; Bytes int64; Entries int }`. The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s `files` and `envPath` (the layout-aware selection the sealer uses); its error is `ArchiveEligible.Detail`. The empty extra directories are removed by the source at `start`, after quiescence: `os.ReadDir` returns no entries → `os.Remove` (never `RemoveAll`); a directory found non-empty at that moment is left and reported in `ExtraDirsLeft`; the seal then refuses stray entries as today. |
| -- `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Err string \`json:"err,omitempty"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; FilesTruncated bool \`json:"files_truncated,omitempty"\`; ExtraDirs []string \`json:"extra_dirs,omitempty"\` }`: the replica from the last mirror listing of `instances/<host>/` (cached per tick), the files from `archive.PlainDigest(files, envPath)` over `offlineFilesOf`'s selection under a 60 s budget and a 2 GiB cap (over either → `FilesTruncated`), `Err` when any part failed (the console refuses on `Err != ""`). |
| -- `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES` (comma-separated target bucket names, as the box publishes them). |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestLaneProbePutsListsDeletes`; `TestTargetReadyReportsEveryFactWithKnown` (each fact OK, refused and unknown; the full data-path set; a stale live link's target; a `lock.json`; the prefix inventories); `TestSourcePrecheckFacts` (an unsupported file type → `ArchiveEligible` not OK with the archive's reason; the layout-aware files dir on a nested instance; extra directories with sizes); `TestEmptyExtraDirsRemovedAtStartOnly` (empty → removed after quiescence and listed in `ExtraDirsRemoved`; one that gained a file → left and reported); `TestQueuedRoundRunsOnlyLanePrecheckAndReady` (a queued record: the source writes `lane` and `precheck`, the target `ready`, no stage round, no hold; a non-participant box in the region does nothing); `TestParticipantsReReportOnANewPrecheckStamp`; `TestStagerReportCarriesCopiedAndDeferred`; `TestStatusHostsBlockIsPrivateAndBounded` (through the real `PublishFleet`: `Hosts` in `status/boxes/<label>.json`, absent from `fleet.json`; the budget sets `FilesTruncated`; a failed listing sets `Err`); `TestLanesPublished`. |
| +- `type moves.Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. |
| +- `type moves.LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the probe is `laneProbe`'s put-if-absent, list and delete-version under `moves/_lane/<move id>`; this is the existing Phase 1 lane probe reused, so the lane evidence is move-bound by its key). |
| +- `type moves.PrefixFacts struct { Known bool \`json:"known"\`; Objects map[string]StagedObject \`json:"objects,omitempty"\` }`. |
| +- `type moves.TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\` (Detail: bytes); MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\` (OK: absent; Detail: the owner and expiry when present); Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }` (`Leftovers` for the operator return, Task 8). |
| +- `type moves.ExtraDirsFact struct { Known bool \`json:"known"\`; Dirs []ExtraDir \`json:"dirs,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type ExtraDir struct { Path string \`json:"path"\`; Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\` }`; `type moves.SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; ArchiveEligible Fact \`json:"archive_eligible"\`; ExtraDirs ExtraDirsFact \`json:"extra_dirs"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; ExtraDirsRemoved []string \`json:"extra_dirs_removed,omitempty"\`; ExtraDirsLeft []ExtraDir \`json:"extra_dirs_left,omitempty"\` }`. The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. Empty extra directories are removed at `start` after quiescence: `os.ReadDir` → zero entries → `os.Remove`; a `ReadDir` or `Remove` error, or a non-empty directory, → `ExtraDirsLeft` with `Detail`; a non-directory stray entry is left for the sealer's refusal as today. |
| +- `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Known bool \`json:"known"\`; Err string \`json:"err,omitempty"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; ReplicaCollectedAt int64 \`json:"replica_collected_at"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; OverCap bool \`json:"over_cap,omitempty"\`; Truncated bool \`json:"truncated,omitempty"\`; ExtraDirs []string \`json:"extra_dirs,omitempty"\` }`: `measureFiles(ctx, files, envPath string, budget time.Duration, capBytes int64) (bytes int64, entries int, overCap, truncated bool, err error)` (a new bounded walker in `internal/host` applying the archive's counting rules (the env file counted, `archive.Cap`) without stopping at the cap: it records `OverCap` and continues until the budget (60 s) or 2 × the cap, where it records `Truncated`); the replica listing cached per tick with its own `ReplicaCollectedAt`; `Known` only when every part succeeded. |
| +- `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES`; `BoxMetrics.BoxID string` (the edges writer's value) so the console joins a participant to its report. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestParticipantRoleWithAndWithoutParticipants`; `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown` (each fact OK, refused and unknown; the full data-path set; the prefix inventories; a present lock's owner); `TestSourcePrecheckFacts` (unsupported file type; nested layout; extra directories; an inspection failure → `ExtraDirs.Known false`); `TestEmptyExtraDirsRemovedAtStartOnly` (empty → removed; one with a zero-byte file → left with `Detail`; a `ReadDir` error → left); `TestStagerPublishesACompleteInventory` (after a round, `Staged` equals a listing of the destination prefix; an overwrite between the copy and the listing shows the listed ETag; `StagedBucket/Region` from the record); `TestSealerPublishesTombstoneLiftedBeforeCleanup` (the `reverting` report exists after the lift and before the prefix deletion; a cleanup failure after the lift leaves it readable; the final `reverted` keeps the fact); `TestQueuedRoundRunsOnlyLanePrecheckAndReady` (non-participant boxes in the region do nothing; `offlineReassert` and the keep view likewise); `TestParticipantsReReportOnANewPrecheckStamp`; `TestValidatorMatrix` (every role/phase/payload combination through `ValidateOfflineReport` and `readOfflineEvidence`); `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID` in the private report, absent from `fleet.json`; `OverCap`, `Truncated`, `Known false` on a failed listing); `TestVersionedArmedForOffline`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/moves -run 'Lane|TargetReady|Precheck|Report' -count=1`, `go test ./internal/host -run 'LaneProbe|TargetReady|SourcePrecheck|ExtraDirs|QueuedRound|ReReport|StagerReport|StatusHosts|Lanes' -count=1`, `go test ./internal/status -run 'Hosts|Lanes' -count=1`, then the root-read census. |
| -- [ ] **Step 5: Commit** `boxes: the lane probe, the source precheck, the target-ready facts and the per-host status block, dispatched by participant id`. |
| +- [ ] **Step 4: Run** `go test ./internal/moves -run 'Participant|Lane|TargetReady|Precheck|Validator|Report' -count=1`, `go test ./internal/host -run 'ParticipantRole|LaneReport|TargetReady|SourcePrecheck|ExtraDirs|StagerPublishes|SealerPublishes|QueuedRound|ReReport|StatusHosts|VersionedArmed' -count=1`, `go test ./internal/status -run 'Hosts|Lanes|BoxID' -count=1`, then the root-read census. |
| +- [ ] **Step 5: Commit** `boxes: one participant resolver, the lane, the source precheck, the target-ready facts, the staged inventory, the lifted-tombstone fact and the per-host status block`. |
| -### Task 7: The job, from requested to ready |
| +### Task 7: The job, from requested to ready, and its recovery table |
| **Files:** |
| - Create: `internal/console/selfserve_job.go`, `internal/console/selfserve_checks.go`, `internal/console/selfserve_apply.go` |
| - Test: `internal/console/selfserve_job_test.go` |
| **Interfaces:** |
| -- `func (s *Server) selfServeApply(ctx context.Context, row *SelfServeMove, mut func(*provision.Record) error, want func(*provision.MoveState) bool) (*provision.Record, error)`: `provision.SetRecordMove` with the judged context; on any error other than `ErrMoveConflict`, re-read the record by `(Account, App, Host)` and, when `rec.Move != nil && rec.Move.ID == row.ID && want(rec.Move)`, treat the write as landed; after every landed write (including a discovered one) `bumpSerial`. `ErrMoveConflict` is returned as is (the next tick re-reads the record and reconciles the row to it: an operator's transition found already made moves the row to the matching phase; a cleared record with `LastMove.ID == row.ID` finalises the row with that outcome). |
| -- `func (s *Server) selfServeStep(ctx context.Context, row SelfServeMove, now time.Time) error`: re-reads the record and the reports first; reconciles the row's phase to the record's state when they disagree (the table below); then, if `row.AbortRequestedAt != 0` and the record's state is at or before `offline-landed` (or there is no record move), the abort step (Task 8) first; else the phase function. Every phase has `StepDeadline` (the wait) and the move has `WindowDeadline` (24 h from `created_at` until ready; then ready's 24 h). |
| +- `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, mut func(*provision.Record) error, want func(provision.Record) bool) (*provision.Record, error)`: `t.valid(now)` else `errLeaseLost`; `provision.SetRecordMove` under the judged context with a 30 s bound (`offlineStartWrite`); on an error other than `ErrMoveConflict`, re-read the record (judged) and, when `want(*rec)`, treat the write as landed; after a landed write `bumpSerial` and `SetSelfServeNotified(id, state)`; a failed bump leaves `notified_state` behind, and the next tick bumps again when `notified_state != the record's state`. `ErrMoveConflict` is returned for the reconciler. |
| +- `func (s *Server) selfServeStep(ctx context.Context, t *leaseToken, row SelfServeMove, now time.Time) error`: (1) deadlines from the row: `StepDeadline` or `WindowDeadline` passed → the phase's expiry action (abort, page, or blocked) **before** any evidence is read; (2) the record (judged) and the reports; (3) **reconcile** the row to the record by the table below; (4) `AbortRequestedAt != 0` and the record's state at or before `offline-landed` (or no record move) → the abort step (Task 8); (5) the phase function. |
| + |
| +**Reconciliation and recovery table** (the record state is the truth; the row's `request_issued` says a write was issued whose answer was lost): |
| + |
| +| row phase | record | action | |
| +|---|---|---| |
| +| `requested`, `reserving` | no move | continue the phase | |
| +| `reserving` with `request_issued` set (`request_kind = queue`) | no move | re-read once more after the 30 s bound has passed since `request_issued`; still none → the write did not land: clear `request_issued`, retry the queue write; a move with `ID == row.ID` → confirm (`queued`) | |
| +| `reserving`, `queued` | `offline-queued` with this id | `queued` → `reports` | |
| +| `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier` | `offline-staging` with this id | `staging` unless the row is already past it; a row in `restage` or `barrier` stays | |
| +| `staging` (any sub-phase) with `request_issued` (`restage` or `precheck`) | the record's `StagedAt`/`PrecheckAt` equals the issued value | `ConfirmSelfServeRequest`; otherwise, after the bound, clear and reissue | |
| +| any pre-cut phase | `offline-stopping` | `stopping` (an operator started it) | |
| +| any | `offline-sealing`/`sealed` | `sealing`; `landing`/`landed` → `landing`/`landed`; `offline-open` → `opened`; `offline-finishing` → `finishing` (gate `lifted`); `offline-reverting` → `reverting` | |
| +| any | no move and `LastMove.ID == row.ID` | the finaliser with `LastMove.Outcome` | |
| +| any | another move (an operator's, or a later self-serve) | finalise `aborted` "another move is in flight" without touching the record | |
| +| any | the record absent or stale (judged) | finalise `aborted` "the instance was recreated" without touching the record | |
| + |
| - Phases: |
| - - `requested` → `reserving` (`AdvanceSelfServe`). |
| - - `reserving`: `reserveBoxes(sel, {move, row.ID, host, acct, app, now})` where `sel` is rebuilt from the row's participants and the fleet (a participant whose `BoxID` no longer publishes → abort "box <label> was replaced"); busy → stay, `reason` "waiting for another move on this box"; `WindowDeadline` passed → abort "the move was not started in time"; reserved → `selfServeApply(OfflineQueue(...))` with `want: state == offline-queued`; landed → `queued`; `ErrMoveConflict` → reconcile (another move appeared → abort "another move is in flight", release). |
| - - `queued` → `reports` when the record carries the queued move (else re-read; 5 min → abort). |
| - - `reports`: the `lane` report (`Put && Listed && Deleted`), the source `precheck` report and the target `ready` report, each with `BoxID` equal to the pinned participant and `Request == PrecheckAt`; `targetReadyDecision` and `sourcePrecheckDecision` (every fact: `!Known` → "could not determine <fact> on <label>"; not OK → the fact's refusal: "the target box already has a route for <host>", "the target box has timers for <host>", "the target box has a stale live link for <host> naming <build>", "the target box's programs file names <build>", "the target box already holds data for <host>: <path>", "the target box's park directory is not clean", "the target box has too little free disk for <host>" (`!DiskKnown` or `< 2 × replica`), "the target bucket already holds objects for <host>" (`MovesPrefix` or `InstancesPrefix` non-empty before staging), "the target bucket holds a lock for <host>", the existing secrets and capability sentences, "root's stage path is not armed on <label>"; the source: `ArchiveEligible` → the archive's reason, a non-empty `ExtraDirs` entry → "<host> has data outside its files directory: <path>", `Build` must equal the channel's current build → "the channel moved to <build> during the move"); both pass → `selfServeApply(OfflineQueuedToStaging)`, `SetSelfServeStageRequest(StagedAt)`, `staging`; 5 min → abort naming the missing report or the failed fact. |
| - - `staging`: the stager report with `StageRequest == row.StageRequest` and phase `staged` → `Rounds++` persisted; ready rule: `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `selfServeApply(OfflineRestage)` only when the current request has been answered, and persist the new `StageRequest` (a lost response: the re-read record's `StagedAt` is the truth; the row follows it); a `failed` report → abort with `StageError`; a round past `max(10 min, 3 × est.Copy)` → counts failed → abort; the bound (30 min since `staging` began, or 10 rounds) → `prechecks` only from a completed successful round (the last one), else wait for it; the estimate refined from the last round's remaining bytes. |
| - - `prechecks`: `selfServeApply(OfflinePrecheck(now))`, `SetSelfServePrecheckRequest`; wait for the source `precheck` and target `ready` reports answering it (the target's `MovesPrefix` is now compared with the stager report's `Staged` receipts: every object under `moves/<host>/` must appear in `Staged` with an equal ETag → else "foreign objects under the staging prefix for <host>"; `InstancesPrefix` must still be empty); both pass → `ready` with `WindowDeadline = now + 24 h`, the estimate refined and `selfServeApply(OfflineSetEstimate)` in the same tick (a disagreement found later is resolved towards the row); 5 min → abort. |
| - - `ready`: every 10 min `OfflineRestage` as in staging (answered requests only); at `WindowDeadline` → abort "the move was not confirmed in time"; the cut moves it to `restage` (Task 5). |
| -- The pre-offline abort (`AbortRequestedAt` set, or any abort reason above, while the record is in `offline-queued` or `offline-staging`, or there is no record move yet): no record move → finalise directly; queued → `selfServeApply(OfflineAbort)` clears the record; staging → `OfflineAbort` → `offline-reverting`, wait for the stager's `reverted` report, then `OfflineFinish`; then the finaliser (Task 8). Failed admission: a reservation for this row older than the grace period with no record move → abort "the move could not be started", finalise. |
| - |
| -- [ ] **Step 1: Write the failing tests** with an in-process console, a `DirStore`, fixture edges files and status reports for two boxes with ids, the real `offlineRound` of two fake-box workers driven by the serial bump (`twoBoxes` helper from the e2e harness; no hand-inserted reports): |
| - - `TestAdmissionResumesFromEveryBoundary` (crash after the insert; after the reservation; after a queued record write whose response was lost; after the reports; after the staging transition: each resumes with exactly one reservation and one record move; a reopened SQLite store each time). |
| - - `TestReservingWaitsThenAborts`; `TestReplacedParticipantAborts`. |
| - - `TestReportsDecideFromFacts` (one subtest per refusal above including every unknown fact; a lane failure; a report from a non-participant box id is ignored). |
| - - `TestStagingReadyRule` (two rounds under the delta → prechecks; a round that copied 1 GiB with zero remaining and zero deferred → not ready by the delta rule, ready after the next quiet round; a never-settling instance → ready at the bound from the last successful round; a failed round → abort; a lost restage response → the row follows the record's `StagedAt`, no second request while the current one is unanswered). |
| - - `TestPrechecksRefuse` (ineligible archive; a non-empty extra dir; foreign staged objects; a promote during staging; a stale report with the old `Request`). |
| - - `TestReadyRestagesAndExpires`; `TestEstimateOnRowAndRecordAgree`. |
| - - `TestAbortBeforeOfflineIsLossless` (requested, reserving, queued, staging: each releases and records `aborted`; staging waits for the stager's reverted report). |
| - - `TestLostResponsesAreReconciledByTheRecord` (for queue, restage and the staging transition). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Admission|Reserving|Replaced|ReportsDecide|StagingReady|Prechecks|ReadyRestages|EstimateOnRow|AbortBeforeOffline|LostResponses' -count=1`. |
| + - `requested` → `reserving`. |
| + - `reserving`: the participants' boxes must still publish their ids (else abort "box <label> was replaced"); `reserveBoxes(participants, {move, row.ID, ...})`; busy → `reason` "waiting for another move on this box"; `WindowDeadline` → abort "the move was not started in time"; reserved → the fresh judged record → `recordEligibility`, the run and move checks, `offlineStartValidation(..., stage: true, requireIDs: true, "")` again (a refusal → abort with its sentence, release) → `IssueSelfServeRequest(id, queue, now)` → `selfServeApply(OfflineQueue(...), want: Move != nil && Move.ID == row.ID)` → confirm → `queued`. |
| + - `queued` → `reports` (the record carries the queued move; 5 min → abort). |
| + - `reports`: the `lane` report (`BoxID == Source[0].BoxID`, `OK`), the source `precheck` and the target `ready` (`BoxID` the pinned ids, `Request == PrecheckAt`); `targetReadyDecision` and `sourcePrecheckDecision` (the console enumerates the required path set and fact set itself: a missing entry or `!Known` → "could not determine <fact> on <label>"; not OK → the fact's sentence as rev 2, plus "the target box has timers for <host>", "the target bucket holds a lock for <host> (<owner>)"; before staging both prefixes must be `Known` and empty); both pass → `IssueSelfServeRequest(id, staging, now)`, `selfServeApply(OfflineQueuedToStaging)`, confirm (`stage_request`), `StagingStartedAt = now`, `staging`; 5 min → abort. |
| + - `staging`: the stager report with `StageRequest == row.StageRequest`, phase `staged`, `StagedBucket/Region` equal to the record's: `ConsumeSelfServeRound(id, StageRequest)` (counted once); ready rule `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `IssueSelfServeRequest(id, restage)` → `selfServeApply(OfflineRestage)` → confirm; a `failed` report → abort; a round past `max(10 min, 3 × est.Copy)` → abort; `now - StagingStartedAt > 30 min` or `Rounds >= 10` → `prechecks` from the last completed successful round. |
| + - `prechecks`: `IssueSelfServeRequest(id, precheck)` → `selfServeApply(OfflinePrecheck)` → confirm; the source `precheck` and target `ready` answering it; the staged-ownership comparison: `Ready.MovesPrefix.Known`, the stager report's `Staged` present with `StagedBucket/Region` equal to the record's and `StageRequest == row.StageRequest`, and the two inventories **equal** (same key set, equal ETags and sizes) → else "the staging prefix for <host> does not match what was staged"; `InstancesPrefix` known and empty; both pass → `ready`, `WindowDeadline = now + 24 h`, the estimate refined and `selfServeApply(OfflineSetEstimate)` (a later disagreement resolves towards the row); 5 min → abort. |
| + - `ready`: every 10 min a restage (issue/confirm); `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| +- The pre-offline abort (requested, reserving, queued, staging): no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`, never offline); queued → `selfServeApply(OfflineAbortAt(now))`; staging → `OfflineAbort` → wait for the stager's `reverted` → `OfflineFinish`; then the finaliser. Failed admission: a reservation older than the grace period with no record move and no outstanding `request_issued` → abort "the move could not be started". |
| + |
| +- [ ] **Step 1: Write the failing tests** on the harness of Task 10 (real boxes, the serial bump): `TestEveryBoundaryRecovers` (for queue, staging, restage, precheck: a landed write whose response was lost; a crash before `ConfirmSelfServeRequest`; a crash before the bump; a second crash during recovery; SQLite reopened each time: one request per wait, one bump per landed write); `TestReconcileTableEveryRow`; `TestReservingWaitsThenAborts`; `TestReplacedParticipantAborts`; `TestEligibilityRechecksBeforeQueue` (a record made ineligible after admission → abort, no queue); `TestReportsDecideFromFacts` (every fact's refusal and unknown; a missing path entry; a report from a non-participant id ignored); `TestStagingReadyRule` (the rev 2 cases plus a counted round counted once across a crash); `TestPrechecksCompareInventories` (missing, extra, changed ETag, changed size, wrong round, wrong destination, unknown prefix; a valid match); `TestReadyRestagesAndExpires`; `TestDeadlinesEvaluateBeforeEvidence` (an unreadable report store never postpones an expiry); `TestAbortBeforeOfflineIsLossless`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileTable|Reserving|Replaced|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, deciding from the boxes' reports through one apply helper`. |
| +- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, with issued requests, a reconciliation table and deadlines evaluated before evidence`. |
| -### Task 8: The job's cut, abort, finalisation and paging |
| +### Task 8: The job's cut, abort, finalisation, paging, and the operator return |
| **Files:** |
| -- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (the operator finish door: when a self-serve row exists for the move id, it calls `s.selfServeFinalise` rather than `offlineReleaseLock`) |
| -- Create: `internal/console/selfserve_health.go`, `internal/console/selfserve_pages.go`, `internal/console/selfserve_finalise.go` |
| -- Test: `internal/console/selfserve_cut_test.go` |
| +- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (every operator door that clears a self-serve move's record (`abort` from queued, `finish`) calls `s.selfServeFinalise`; the stage door's body gains `return_of`) |
| +- Create: `internal/console/selfserve_health.go`, `internal/console/selfserve_pages.go`, `internal/console/selfserve_finalise.go`, `internal/console/offlinereturn.go` |
| +- Test: `internal/console/selfserve_cut_test.go`, `internal/console/offlinereturn_test.go` |
| **Interfaces:** |
| -- Phases and deadlines (`est` is the row's `EstimateTerms`): |
| - - `restage`: `selfServeApply(OfflineRestage)`, wait for the matching `staged` report; `max(5 min, 2 × est.Copy)`; expiry → abort. Then `selfServeApply(OfflinePrecheck(now))` and wait for both participants' fresh reports passing the precheck decisions (the **cut barrier**); `max(5 min, ...)`; expiry or a failed fact → abort. |
| - - `stopping`: `offlineStartValidation(..., stage: false)` once more (every existing start check), then `selfServeApply(OfflineStartStaged(...))` with `want: state == offline-stopping`; then the seal check (`offlineSealCheck`: every source report held, fenced, quiesced, no process, with `BoxID` the pinned participants) → `selfServeApply(OfflineSeal)`; `max(5 min, 2 × est.Stop)`; expiry → abort. |
| - - `sealing`: the sealer's `sealed` report; `max(10 min, 3 × est.Copy)`; expiry → abort. |
| - - `landing`: `offlineFlipCheck` → `selfServeApply(OfflineFlip)`; the target's `landed` report (`BoxID` the pinned target) with every item OK; `max(10 min, 3 × est.Land)`; expiry → abort. |
| - - `landed`: `offlineOpenCheck` → `selfServeApply(OfflineOpenTransition)` with `want: state == offline-open`; a refusal of the open check or `max(3 min, 2 × est.Open)` **while the record is still `offline-landed`** → abort (the record never left the abortable state); a write error without a confirmed commit → `open-uncertain`. |
| - - `open-uncertain`: re-read the record each tick: `offline-open` → `opened`; still `offline-landed` → back to `landed` (the write did not land; the deadline continues); 5 min unreadable → page, stay. |
| - - `opened`: the target's `opened` report; `max(3 min, 2 × est.Open)`; expiry → **blocked**: page, stay, retry each tick (the open is committed; no abort). |
| - - `health`: a probe when `now - HealthLastProbeAt >= 30 s` (never sleeping in the tick), `RecordSelfServeHealth`; two consecutive OK probes → `finishing`; `max(3 min, 2 × est.Promote) + 30 s` without two OK → `SetSelfServeWarning("the account did not answer on <region> within <n> minutes")`, page, `finishing`; a failed probe between two OK ones resets the count. |
| - - `finishing`: `FinishGate == ""` → `offlineFinishCheck` from `offline-open` (opened + every pinned source parked) → `selfServeApply(OfflineFinishingTransition)`, `SetSelfServeFinishGate("lifted")`; then from `offline-finishing` (lifted + every pinned source gone) → `selfServeApply(OfflineFinish)` with `want: Move == nil && LastMove.ID == row.ID`; 10 min each gate → blocked: page, retry, never bypass. |
| - - `done`: the finaliser. |
| - - `reverting`: `selfServeApply(OfflineAbort)` once (`AbortRequestedAt` set by the door or by the job's reason); then the evidence: `SourceReopenedAt` when (the sealer's `lifted` report when `mv.Sealer != ""`, i.e. a tombstone existed) **and** a source `reverted` report from every pinned source; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes (every report in, the stager's `reverted` when staged) → `selfServeApply(OfflineFinish)`; 10 min → blocked: page, retry; a worker returning from an interrupted copy after the abort writes its reports late and the facts land when they arrive. |
| -- The finaliser (`selfServeFinalise(ctx, row, outcome, reason, warning)`): `FinishSelfServeMove` (outcome, `release_pending = 1`) **then** `releaseBoxes(row.ID)` then `ClearSelfServeRelease`; a crash or failure between leaves `release_pending`, and `OpenSelfServeMoves` returns the row until the release lands; the timings (Task 9); `bumpSerial`; the member stream event. The operator finish door routes a self-serve move's finish through it. A reservation with no row (or a terminal row with `release_pending = 0`) older than the grace period → the orphan scan reclaims it per Task 3's rule. |
| -- The probe (`probeTarget`): the target box's `EIP` from its edges file, TLS with the host as SNI and `Host`, `GET <ReadyPathOrDefault>?carlos-probe=<random>`, `Cache-Control: no-cache`, no redirects, 30 s; OK when 2xx and `x-carlos-version` equals the sealer report's `Release`. Plan 3 adds `X-Carlos-Probe`/`X-Carlos-Served-By`; the comment names the limitation until then. |
| -- Paging (`selfserve_pages.go`): `alerting.Page(ctx, alerting.LevelIncident, subject, body)` through the store's durable `MovePageAttempt`/`SetMovePageAttempt` with rule keys `selfserve:<id>:<condition>` and a 10-minute cadence, for: `opened` past its deadline; a health failure; `finishing` or `reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; an orphan reservation (the scan each tick over `active.json`: a `move` entry whose row is absent or terminal-and-released, older than the grace period, that the reclaim rule cannot reclaim because the record still carries the move); a lane probe failing on a configured pair. |
| -- Sentences (copy rows): "the account did not answer on <region> within <n> minutes"; "the move was not started in time"; "the move was not confirmed in time"; "another move is in flight"; "box <label> was replaced"; "the move could not be started"; the fact refusals of Task 7. |
| - |
| -- [ ] **Step 1: Write the failing tests** on the serial-driven two-box harness with the console pass in-process: |
| - - `TestCutRunsEveryPhaseToDone` (Tito's sequence: prepare, poll to ready, cut, poll to none, `last.outcome == done`; the record cleared; the row terminal with `release_pending = 0`; reservations released; timings recorded). |
| - - `TestEveryDeadlineBeforeCommittedOpenAborts` (restage, the cut barrier, stopping, sealing, landing, and the open check refused or expired while `offline-landed`: each sets `abort.requested_at`, the revert reports arrive, `source_reopened_at` only after the sealer lifted and the sources reverted, then `finished_at`, `aborted` with the reason). |
| - - `TestOpenUncertainIsReconciledByTheRecord` (a lost response with the write landed → `opened`; without → back to `landed` and abortable). |
| - - `TestAbortIsInitiatedNotCompleted` (a sealing box still writing after the deadline: `requested_at` set, no `source_reopened_at`; a source `reverted` report before the sealer's `lifted` does not set it; then both facts, then `finished_at`). |
| - - `TestOpenedBlockedPagesAndRetries` (durable attempts: a restart does not page twice within 10 minutes). |
| - - `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthProbesNeverSleepInTheTick`. |
| - - `TestFinishingNeverBypassesTheGate`. |
| - - `TestFinaliseOrderAndPendingRelease` (crash between outcome and release: the row stays in `OpenSelfServeMoves` until the release lands; a release failure likewise; the operator finish door finalises a self-serve move the same way). |
| - - `TestJobYieldsToAnOperatorTransition` (an operator's open, abort and finish between ticks each reconcile the row). |
| - - `TestSwitchOffMidFlightCompletesAndRenders`. |
| - - `TestTwoMovesOnDisjointBoxesRunAtOnce`, `TestTwoMovesOnOneBoxQueueInOrder`. |
| - - `TestOrphanReservationIsPaged`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Cut|Deadline|OpenUncertain|AbortIsInitiated|OpenedBlocked|HealthFailure|HealthProbes|FinishingNever|Finalise|JobYields|SwitchOffMidFlight|TwoMoves|OrphanReservation' -count=1`. |
| +- Phases and deadlines (`est` the row's terms): |
| + - `restage`: issue/confirm `OfflineRestage`; the matching `staged` report; `max(5 min, 2 × est.Copy)`; expiry → abort. → `barrier`. |
| + - `barrier`: issue/confirm `OfflinePrecheck`; both participants' fresh reports passing the decisions and the inventory comparison; `max(5 min, 2 × est.Copy)`; a failed fact or expiry → abort. → `stopping`. |
| + - `stopping`: `offlineStartValidation(..., stage: false, requireIDs: true, "")` on the fresh record, then `selfServeApply(OfflineStartStaged, want: State == offline-stopping)`; then the seal check with participants → `selfServeApply(OfflineSeal)`; `max(5 min, 2 × est.Stop)`; expiry → abort. |
| + - `sealing`: the sealer's `sealed`; `max(10 min, 3 × est.Copy)`; expiry → abort. |
| + - `landing`: `offlineFlipCheck` → `OfflineFlip`; the pinned target's `landed` with every item OK; `max(10 min, 3 × est.Land)`; expiry → abort. |
| + - `landed`: `offlineOpenCheck` → `selfServeApply(OfflineOpenTransition, want: State == offline-open)`; a refusal or `max(3 min, 2 × est.Open)` **while the record is `offline-landed`** → abort; a write error with no confirmed commit → `open-uncertain`. |
| + - `open-uncertain`: re-read each tick: `offline-open` → `opened`; `offline-landed` → `landed` (deadline continues); 5 min unreadable → page, stay. |
| + - `opened`: the target's `opened`; `max(3 min, 2 × est.Open)`; expiry → blocked: page, retry. |
| + - `health`: a probe when `now - HealthLastProbeAt >= 30 s`; `RecordSelfServeHealth`; `HealthConsecutiveOK >= 2` → `finishing`; `max(3 min, 2 × est.Promote) + 30 s` → `SetSelfServeWarning(...)`, page, `finishing`. |
| + - `finishing`: gate 1 (`offlineFinishCheck` opened + every pinned source parked) → `OfflineFinishingTransition`, `FinishGate = lifted`; gate 2 (lifted + every pinned source gone) → `selfServeApply(OfflineFinish, want: Move == nil && LastMove != nil && LastMove.ID == row.ID)`; 10 min each → blocked: page, retry. |
| + - `reverting`: `OfflineAbort` once (or `OfflineAbortAt` from queued); `SourceReopenedAt` when (no sealer, or the sealer's `reverting`/`reverted` report with `TombstoneLifted`) **and** every pinned source's `reverted`; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes → `OfflineFinish`; 10 min → blocked: page, retry. |
| +- **The finaliser** `selfServeFinalise(ctx, t, row, outcome, reason, warning)`: (1) `FinishSelfServeMove` (outcome; `release_pending = 1`, `effects_done = 0`); (2) the terminal effects: the timings (Task 9, from the stamps persisted at each phase end), the terminal `self_serve_events` row, `bumpSerial`; `SetSelfServeEffectsDone`; (3) `releaseBoxes(move, row.ID)`; (4) `ClearSelfServeRelease`. A crash between any two leaves the row in `OpenSelfServeMoves` and the next tick resumes at the first unfinished step (effects are idempotent by `(id, phase)`). The operator doors that clear a self-serve move's record route through it. An outstanding admission write (`request_issued` set in `reserving`) is never declared failed before the 30 s bound has passed and a re-read shows no move. |
| +- **The operator return** (§7.5): `POST .../move/offline/stage {"region", "return_of": "<forward move id>"}` (and `.../offline` start): `offlineStartValidation(..., returnOf)` reads the forward move's evidence: `control/offline-moves/<host>/<fwd>/` reports (the sealer's `lifted` or `reverted`, the target's `opened`), and requires the target region of the return to be the forward's `From`; `targetReadyDecision` with `Leftovers` accepts exactly: `Lock` present with owner `moves.TombstoneOwner(fwd)` and lifted (`lease.IsLiftedTombstone`); `InstancesPrefix.Objects` whose every `LastModified` is earlier than the forward's `OpenedAt` (the report gains `LastModified` per object through `ListVersions`); `DataPaths` entries for `/data/.instances/<host>` and `/data/instances/<host>` present and **empty** (a `Fact` with `Detail: "empty"`); `/data/.offline-moves/<host>/<fwd>/source/` and `/sealer/` present; `LiveLink` and `ProgramsFile` naming the channel's build (already accepted); anything else present → the ordinary refusal. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\` }`. The box's `offlineTargetReady` fills `Leftovers` when the record's `ReturnOf` is set (the stage door writes `ReturnOf` on the record through `OfflineStage`'s new parameter, additive). |
| +- The probe (`probeTarget`): as rev 2; plan 3 adds the header contract. |
| +- Paging: `alerting.Page` through `MovePageAttempt`/`SetMovePageAttempt` with keys `selfserve:<id>:<condition>` every 10 minutes for: `opened` past its deadline; a health failure; `finishing`/`reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; a reservation of an unknown or terminal-released move older than the grace period that the reclaim rule cannot reclaim (the per-tick scan, which also runs at start when there are no open rows); a lane probe failing on a configured pair. |
| +- Copy rows: as rev 2. |
| + |
| +- [ ] **Step 1: Write the failing tests** on the harness: `TestCutRunsEveryPhaseToDone` (Tito's sequence; the record cleared; the row terminal with `effects_done = 1`, `release_pending = 0`; one bump per landed transition); `TestEveryDeadlineBeforeCommittedOpenAborts` (restage, barrier, stopping, sealing, landing, the open check refused and expired while landed); `TestOpenUncertainIsReconciledByTheRecord`; `TestAbortIsInitiatedNotCompleted` (the real sealer: a source `reverted` before the sealer's lift sets nothing; the sealer's `reverting` report with `TombstoneLifted` plus every source's `reverted` sets `source_reopened_at`; a cleanup failure after the lift still shows reopened; then `finished_at`); `TestOpenedBlockedPagesAndRetries` (durable attempts across a restart); `TestHealthFailureFinishesWithAWarningAndPages`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep` (crashes between (1)/(2), (2)/(3), (3)/(4); the row stays open until (4); effects once); `TestOperatorDoorsFinaliseASelfServeMove` (queued abort and finish through the operator doors); `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRenders`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap` (only an old orphan reservation, admission off, no rows); `TestOperatorReturnAcceptsExactlyTheForwardLeftovers` (a completed forward; the return's stage door accepts the lifted lock, the old objects, the park dirs, the empty nested dirs, the matching link and programs file; a newer object, a live lock, a foreign park dir, a non-empty nested dir each refuse; a `return_of` naming a move that did not open refuses). |
| +- [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'SelfServe|Offline' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self-serve cut from restage to finish, the no-return boundary at the committed open, abort facts, finalisation and durable paging`. |
| +- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve cut from restage to finish, the no-return boundary at the committed open, the abort facts, one finaliser, durable paging and the operator return's leftover acceptance`. |
| -### Task 9: The estimate, its storage, the move log and the status view fields |
| +### Task 9: The estimate, timings, the events table and the status view |
| **Files:** |
| -- Create: `internal/console/selfserve_estimate.go` |
| -- Modify: `internal/console/store.go` (`move_timings (id TEXT, phase TEXT, from_region TEXT, to_region TEXT, bytes INTEGER, seconds REAL, at INTEGER, PRIMARY KEY (id, phase))`), `internal/console/selfserve_finalise.go` (record the five phases from the row's and reports' timestamps: `stop` = `OfflineAt` → every source `stopped` report's `At`, bytes 0; `copy` = the seal transition → the sealer's `sealed` report, bytes = the sum of the sealer report's `Staged` sizes; `land` = flip → the target's `landed` report, bytes = `FilesBytes` + replica bytes; `open` = open → the `opened` report, bytes 0; `promote` = open → `HealthFirstOKAt`, bytes = replica bytes), `internal/console/apiinstancemove_offline.go` (`offlineView` gains `SelfServe *selfServeMoveView` so `carlos move offline status` shows the job's facts), the member stream (`s.auditActivity(acct, app, "self-serve move", Activity{Action: "instance-move", TargetType: "instance", Target: host, Message: <phase>, Detail: <reason>})` on every phase change, keyed on `(id, phase)` so a retried tick does not repeat it), the move log (the existing `movelog` writer gains the job's lines: each phase, each refusal and its reason, the facts, the lane probes, the health probes, the timings) |
| -- Test: `internal/console/selfserve_estimate_test.go` |
| +- Create: `internal/console/selfserve_estimate.go`, `internal/console/selfserve_events.go` |
| +- Modify: `internal/console/store.go` (`move_timings (id TEXT, phase TEXT, from_region TEXT, to_region TEXT, bytes INTEGER, seconds REAL, at INTEGER, PRIMARY KEY (id, phase))`; `self_serve_timings (id TEXT, mark TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))` the per-phase stamps persisted **when consumed**: `offline_at`, `stopped_at` (every source `stopped` seen), `seal_at`, `sealed_at` (+ `FinalCopiedBytes`), `flip_at`, `landed_at` (+ files and replica bytes), `open_at`, `opened_at`, `first_ok_at`), `internal/console/selfserve_finalise.go` (the five `move_timings` rows from the stamps: `stop` = `offline_at → stopped_at`; `copy` = `seal_at → sealed_at`, bytes `FinalCopiedBytes`; `land` = `flip_at → landed_at`; `open` = `open_at → opened_at`; `promote` = `open_at → first_ok_at`, bytes replica), `internal/console/apiinstancemove_offline.go` (`offlineView.SelfServe *selfServeMoveView \`json:"self_serve,omitempty"\``; `GET .../move/offline?log=1` adds `Events []selfServeEventView`), the member stream (a `StartSelfServeEventsPass`-free design: the pass's tick drains `UnemittedSelfServeEvents` through `auditActivity(acct, app, "self-serve move", Activity{Action: "instance-move", TargetType: "instance", Target: host, Message: phase, Detail: reason})` and marks each emitted; an insert that finds `(id, phase)` present is a no-op, so a retried tick never duplicates) |
| +- Test: `internal/console/selfserve_estimate_test.go`, `internal/console/selfserve_events_test.go` |
| **Interfaces:** |
| -- `type estimateTerms struct { Stop, Copy, Land, Open, Promote time.Duration }` (stored as JSON in `estimate_terms`); `type term struct { Fixed time.Duration; BytesPerSecond float64 }`; defaults from the flagship's measurements (2026-10-21): `Stop {20 s, 0}`, `Copy {10 s, 25 MiB/s}`, `Land {30 s, 40 MiB/s}`, `Open {30 s, 0}`, `Promote {10 s, 70 MiB/s}`. |
| -- `func (s *Server) estimate(from, to string, remaining, files, replica int64) (terms estimateTerms, seconds int)`: each term `Fixed + bytes / rate` with the bytes it moves (`copy`: `remaining`; `land`: `files + replica`; `promote`: `replica`; `stop`, `open`: none); a rate term's rate is replaced by the median `bytes/seconds` of the last 20 `move_timings` rows for `(from, to, phase)` with `bytes > 0`, a fixed term's `Fixed` by the median `seconds`, each when at least 5 rows exist; the total rounded up to whole minutes, minimum 2. Before a request: `remaining = replica` from the source box's `Hosts[host]`; after staging: `RoundCopiedBytes + RoundDeferredBytes` of the last successful round. |
| -- On `*Store`: `InsertMoveTiming(t MoveTiming) error` (`type MoveTiming struct { ID, Phase, From, To string; Bytes int64; Seconds float64; At time.Time }`), `MoveTimings(from, to, phase string, n int) ([]MoveTiming, error)`. |
| +- `type estimateTerms struct { Stop, Copy, Land, Open, Promote time.Duration }`; `type term struct { Fixed time.Duration; BytesPerSecond float64 }`; the defaults of rev 2. |
| +- `func (s *Server) estimate(from, to string, remaining, files, replica int64) (terms estimateTerms, seconds int)`: as rev 2 (rate terms fitted from `bytes/seconds` medians of rows with `bytes > 0`; fixed terms from `seconds` medians; 5 rows minimum; whole minutes; minimum 2). |
| +- `type SelfServeEvent struct { ID, Phase string; At time.Time; Reason, Detail string; Emitted bool }`; `type selfServeEventView struct { Phase string \`json:"phase"\`; At int64 \`json:"at"\`; Reason string \`json:"reason,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`. Every phase change, refusal, fact decision, lane probe, health probe and timing inserts an event (`Phase` carries a suffix for non-phase events: `reports:lane`, `health:probe:<n>`). |
| +- On `*Store`: `InsertMoveTiming`, `MoveTimings`, `MarkSelfServeTiming(id, mark string, at time.Time, bytes int64) error`, `SelfServeTimings(id string) (map[string]timingMark, error)`. |
| -- [ ] **Step 1: Write the failing tests**: `TestEstimateUsesDefaultsThenMedians` (no rows → the defaults' sum, rounded up, minimum 2; 5 copy rows → the median rate; 5 open rows → the median fixed term; 4 rows → not yet); `TestEstimateMinimumIsTwoMinutes`; `TestEstimateBeforeAndAfterStaging`; `TestPhaseTimingsRecordedAtDone` (five rows keyed `(id, phase)`, the bytes as defined); `TestOfflineStatusShowsSelfServeFields`; `TestMemberStreamEventsOncePerPhase`; `TestMoveLogCarriesTheJobsLines`. |
| +- [ ] **Step 1: Write the failing tests**: `TestEstimateUsesDefaultsThenMedians`; `TestEstimateMinimumIsTwoMinutes`; `TestEstimateBeforeAndAfterStaging`; `TestTimingMarksPersistWhenConsumed` (a crash after the stamp is consumed and before the phase advances keeps the mark; the finaliser's five rows from the marks; `copy` bytes are the sealer's `FinalCopiedBytes`); `TestEventsOncePerPhaseAcrossRetries`; `TestEventsDrainToTheMemberStream`; `TestOfflineStatusShowsSelfServeAndLog`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Estimate|Timing|OfflineStatusShows|MemberStream|MoveLogCarries' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the move estimate from measured terms and rolling medians; the view, the member stream and the log carry the job`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Estimate|Timing|Events|OfflineStatusShows' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the move estimate from measured terms and rolling medians; timings persisted when consumed; the events table as the claims-off move log and member stream`. |
| -### Task 10: End to end on the two-box harness |
| +### Task 10: The shared harness and the end-to-end witnesses |
| **Files:** |
| -- Test: `internal/host/selfserve_e2e_test.go` (the two-box fixtures of `TestOfflineMoveE2E` with a console running the pass, boxes driven by the real serial bump) |
| +- Create: `internal/host/hosttest/harness.go` (an exported wrapper over `TestOfflineMoveE2E`'s two-box fixtures: `type Boxes struct{...}`, `func New(t testing.TB, store release.Store) *Boxes`, `func (b *Boxes) Pass(ctx) ` (one moves pass on every box), `func (b *Boxes) Serial(ctx)` (run passes until the serial is consumed), with the box ids, labels, registries and data buckets exposed; a clock hook), `internal/e2e/selfserve/selfserve_test.go` (an external test package importing `internal/console` and `hosttest`) |
| +- Test: the witnesses of Tasks 7 and 8 live here; plus: `TestSelfServeForwardMove`; `TestSelfServeCancelInReady`; `TestSelfServeAbortByFailedPrecheck`; `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal`; `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord`; `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchangedOnTheHarness`; `TestSelfServeThenOperatorReturn`. |
| -- [ ] **Step 1: Write the tests**: `TestSelfServeForwardMove` (Tito's sequence against the real doors and real boxes with a move-capable token; the instance answers on the target box; reservations released; the source row gone); `TestSelfServeCancelInReady` (never offline); `TestSelfServeAbortByFailedPrecheck` (a stale live link on the target); `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal` (the open check never passes while `offline-landed` → abort, not a page); `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord` (delete and recreate between the admission and the queue → the row ends `aborted` "the instance was recreated", the successor untouched); `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchanged` (the operator door snapshots again, on the harness). |
| +- [ ] **Step 1: Write the harness and the tests.** |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-2.md`. |
| -- [ ] **Step 4: Commit** `self-serve moves: end-to-end witnesses on the two-box harness`; push. |
| +- [ ] **Step 4: Commit** `self-serve moves: a shared two-box harness and the end-to-end witnesses`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage: §5 (T1), §6.1 door, view, refusals, duplicates (T5), §6.2 (T7), §6.3 (T9, T5), §7 admission order and recovery boundaries (T7), §7.1 reservations with the legacy projection (T3), §7.2 deadlines and the three facts (T8), §7.3 lane, clean-then-staged, target-ready, source precheck, build, empty-directory removal (T6, T7), §7.4 without the probe mode (T8), §7.5 the no-return boundary at the committed open (T8), §7.7 (T5), §8 pages, log, member stream (T8, T9), §9 (behind the switch; per-box concurrency gated by it). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup and the `cleanup` reservation's producer (plan 4; T3 states what it needs: identity, freshness and absence semantics of the status's `RetiredSummary`). |
| -- Dependencies stated for plan 3: the record's `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}` and `Participants.Target.BoxID` for the probe's identity check; for plan 4: `RetiredSummary{ID, Host, State}` with the report's `GeneratedAt` as its freshness. |
| -- Names used across tasks: `selfServeGate`, `moveCapable`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `provision.SelfServeMove`, `Participant`, `Participants`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `offlineSelection`, `offlineStartValidation`, `selfServeAdmits`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `selfServeFinalise`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `moves.LaneReport`, `moves.TargetReadyReport`, `moves.SourcePrecheck`, `status.HostFacts`, `BoxMetrics.Lanes`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `estimate`, `MoveTiming`, `holdSelfServeLease`. |
| +- Spec coverage: §5 (T1), §6.1 (T5), §6.2 (T7), §6.3 (T9, T5, T6), §7 admission and recovery (T7's table), §7.1 (T3), §7.2 and the three facts with the sealer's durable lift (T6, T8), §7.3 (T6, T7, T8's barrier), §7.4 without the probe mode (T8), §7.5 the no-return boundary and the operator return (T8), §7.7 (T5), §8 pages, log and member stream (T8, T9), §9 (T3's rollout). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup (plan 4), whose status contract this plan's reclaim rule names exactly (`RetirementsComplete`, a terminal state, `RootRequest == ""`, `ReleasePending == false`, `Metrics.BoxID`). |
| +- Dependencies stated for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}` on the record, `Participants.Target.BoxID` for the probe's identity, `moveCfg.BoxID`; for plan 4: the status `Retirements` contract above. |
| +- Names used across tasks: `selfServeGate`, `moveCapable`, `Principal.TokenID`, `recordEligibility`, `offlineStartValidation`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant(s)`, `moves.ParticipantRole`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeAdmits`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `selfServeFinalise`, `leaseToken`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `ConsumeSelfServeRound`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck`, `ExtraDirsFact`, `ReturnLeftovers`, `status.HostFacts`, `BoxMetrics.Lanes/BoxID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MoveTiming`, `SelfServeEvent`, `hosttest`. |
| - Review Focus 1 to 5 are pinned by T7, T3, T8, T5, T1. |