amadan
plan: self-serve moves 2, rev 4 (Astra plan 2 review 3)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+112
−184
docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md+112 −184
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md| index 6a20b80f5..ca9e5f4d2 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| @@ -2,275 +2,203 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 3, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21 findings). Rev 3 answers rev 2's twenty-one: the doors' dispatcher order is stated (authenticate, host shape, capability, then the row, then the record) and the route tests count SQL reads too; every phase advance is predicated on no abort request and cut/cancel are tested with barriers in both orders; a legacy record is migrated **bound** to the observed ETag and the record checks re-run on the migrated record; one `recordEligibility` carries `moveGate`'s class checks and `offlineStartValidation` is called immediately before the queue write and before the start; the lane is move-bound and the two prechecks request-bound, with a validator matrix; the stager publishes a complete destination inventory bound to its request and coordinates and the console compares whole inventories; every fact has known/ok/detail and the console enumerates the required sets itself; one participant resolver is used by every box path and every console gate, with the operator's region path kept only without participants; the journal gains issued/consumed identities, per-wait deadlines, a recovery table per boundary, whole-record predicates, deadline evaluation before evidence, and a lease checked before every write; the serial bump is journaled; every record-clearing path finalises through one finaliser with outcome before release and terminal effects before the release clears; the `active.json` rollout and rollback are stated with the matching-projection hazard; the pass and paging start on reservations too; the sealer publishes a durable `tombstone_lifted` fact before its cleanup; the view's precedence among record move, open row and terminal row is defined with a retention window and admission is checked inside prepare; status facts carry identity, freshness, over-cap and truncation; the queued transitions have exact signatures; the remaining interfaces are typed; and the operator return's leftover acceptance is in scope. |
| +Rev 4, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items, five closed). Rev 4 answers rev 3's open items: the doors dispatch on a **process-level mode**, decided without host evidence, so the capability check always precedes every read; the self-serve gate keeps the instance gate's rate limiter and no-bucket rail, and the start validation is split around the body decode; legacy reports keep their structural validation and the new evidence is validated contextually against the move, with an explicit empty-inventory encoding; the staged inventory comes from the stager's **copy receipts**, not from a listing, and the console compares the target's listing against those receipts in both directions; stray entries carry their kind and non-directory strays refuse before the stop; participants carry the box's instance and boot ids from its status report and are verified against the signed box record when one exists, so a replacement is detected at every gate; the journal keeps each request's issued, confirmed-from-the-record and consumed identities, SQL phases map to record states as compatible sets that never regress, the lease bounds each write attempt, and notification is at-least-once keyed on the record's `record_seq`; an expired `landed` deadline defers to the record before any abort; an abandoned admission write is fenced with plan 1's `fenceRecord` before the reservation is released; a deployment-wide reservation sentinel covers id-less selections and exact retries are accepted before the exclusive gate; startup read failures fail closed; the view gives an unrelated active record move precedence over an open row and joins the self-serve row by id and lifetime; host facts carry the lifetime and both collection ages; every new table is classified; events and their audit delivery are one transaction; the harness is an exported package; the journal's continuity is stated; and the operator return has a request-and-report handshake with a durable open record. |
| -Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`): records carry a lifetime and `move_seq`, rows carry the pair, readers and writers are judged. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| +Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 8): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord` exists. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| **Goal:** With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, Tito's existing residency calls (prepare, status, cut, cancel) run the offline move as a console job: an owner-delegated move-capable token asks, the door refuses what cannot be offered before anything is written, per-box reservations serialise moves per box, the boxes prove the lane, the target's cleanliness and the source's eligibility before any copy and again before the stop, staging runs until caught up, the cut runs start, seal, flip, open, health, finish with a deadline on every wait, every failure before the committed `open` initiates a lossless abort, and the view tells Tito `preparing`, `ready`, `cutting`, `none` with `last` and the abort's three facts. The operator's return accepts exactly the forward move's leftovers. |
| -**Architecture:** One `self_serve_moves` SQLite row per move drives a console pass (`StartSelfServeMovesPass`, every 5 s, under a console lease in the deployment bucket) that advances each move by at most one step per tick from the evidence the operator doors already read, through the door bodies refactored into functions the doors and the job share. A new record state `offline-queued` lets the console reserve boxes and collect the lane, source-precheck and target-ready reports before a byte is copied; a precheck request stamp makes both participants report again before the cut. `control/offline-moves/active.json` becomes a per-box reservation table with a legacy projection. The move-capable token is a `moves` column on `cli_tokens`, owner-minted, checked at the three self-serve mutations before any evidence is read. |
| +**Architecture:** One `self_serve_moves` SQLite row per move drives a console pass (every 5 s, under a console lease) that advances each move by at most one step per tick from the evidence the operator doors already read, through the door bodies refactored into functions the doors and the job share. A new record state `offline-queued` lets the console reserve boxes and collect the lane, source-precheck and target-ready reports before a byte is copied; a precheck stamp makes both participants report again before the cut. `active.json` is a per-box reservation table with a legacy projection and a deployment-wide sentinel. The move-capable token is a `moves` column on `cli_tokens`, owner-minted, checked at the three self-serve mutations before any evidence is read. |
| -**Tech Stack:** Go. `internal/console`, `internal/provision`, `internal/moves`, `internal/host`, `internal/moves/archive`, `internal/alerting`, `internal/status`, `internal/lease`. Tests: the console in-process with `release.NewDirStore`; the integrated witnesses in a new external test package `internal/e2e/selfserve` that imports both `internal/host`'s exported harness (`hosttest`, Task 10) and the console, driven by the real serial bump. |
| +**Tech Stack:** Go. `internal/console`, `internal/provision`, `internal/moves`, `internal/host`, `internal/moves/archive`, `internal/alerting`, `internal/status`, `internal/lease`, `internal/boxid`. Tests: the console in-process with `release.NewDirStore`; integrated witnesses in `internal/e2e/selfserve` over the exported harness `internal/host/hosttest` and the console's exported test hooks. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §7.5's operator return, §8, §11. Stated deviations: the record's `self_serve` is an object, not a boolean; the pre-request estimate input is the replica size alone; the claims-off "move log" is the console's `self_serve_events` table (shown by `carlos move offline status --log`), not the claims-on `hostops.MoveLogWriter`, which requires claim evidence. |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §7.5's operator return, §8, §11. Stated deviations: the record's `self_serve` is an object; the pre-request estimate input is the replica size alone; the claims-off "move log" is the console's `self_serve_events` table; **in self-serve mode the legacy move mutations are not served** (a plain instance token gets the family's 404 at prepare, cut and cancel; the status read stays open), because the dispatch must be decided before any evidence is read. |
| ## Global Constraints |
| -- `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) enables **admission**, and only when `!cfg.HostClaim && !cfg.MovesEnabled`. **Rendering and recovery** follow the rows and the reservations, not the switch: open moves run to completion, terminal rows render `last` for their retention window, the pass and paging start whenever open rows or reservations exist. With no row for the host the four doors answer byte for byte as today. |
| -- The operator doors keep their answers byte for byte (snapshots of every refusal of every operator door, including combinations of a malformed body with a class refusal, taken from main and replayed). A job tick never calls an HTTP door. |
| -- Every wait has a deadline (Task 8's table); a deadline is evaluated from the row **before** any evidence is read, so an unreadable report never postpones expiry or paging. Expiry before the committed `open` initiates abort; after it pages and retries. The job never calls `finish` without the gate's evidence. |
| -- Every record transition the job writes goes through `selfServeApply` (Task 7): a `want func(provision.Record) bool` on the whole record decides whether a lost response landed; `control/serial` is bumped after every landed write, and the row records `notified_state` so a crash between commit and bump is repaired on the next tick. |
| -- One console runs the pass: `control/offline-moves/console.json` is a 60 s lease held by the console's identity, renewed every 20 s during a tick and checked immediately before every write; a write after the lease lapsed is refused and the tick ends. The SQLite journal is the console's own; a replacement console starts from the rows and the records (there is one console per deployment today; the lease is the guard for the day there are two). |
| -- Refusals come before writes: the row only after every check of §6.1's list; the record only after the reservation; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| -- Participants are box ids with labels, pinned at admission from the edges files; **one resolver** (`moves.ParticipantRole`, Task 6) decides a box's role for a move that carries `Participants`, in every box path (`offlineRound`, `offlineReassert`, the offline view's `hold` and `keep`, `offlineTarget`) and every console gate (`offlineSealCheck`, `offlineFlipCheck`, `offlineOpenCheck`, `offlineFinishCheck`, the job's report selection); a report whose `BoxID` is not a participant is ignored; a box publishing a participant's label with another id is "box <label> was replaced". A move without `Participants` (an operator move begun before this code) keeps today's region rule everywhere. |
| -- Timestamps in SQLite are Unix **nanoseconds** (`INTEGER`), zero meaning unset; stage and precheck requests compare by equality with the record's `time.Time` at nanosecond precision. |
| -- Copy rows pending Paul's review (Tasks 5 and 8). No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| -- Census: the box-side units take rows in `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus`, `rev3=PROPOSED-self-serve-moves-2`, with `@fields` for the new record fields and reviewed dispositions for changed frozen units and envelopes (`internal/host`, `internal/status`, `internal/moves`, `internal/console`); `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's with every added or changed writer (the new transitions, the box's report writers, the console's `self_serve_events`); both green. |
| +- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. Rendering follows the rows (Task 5). |
| +- The operator doors keep their answers byte for byte (snapshots of every refusal, including a malformed body combined with each class refusal). A job tick never calls an HTTP door. |
| +- Every wait has a deadline evaluated from the row before any evidence is read; **the action an expiry takes is decided by the record** (Task 8): an expiry while the record is `offline-landed` aborts, while it is committed open pages, while it is unreadable waits as `open-uncertain`. |
| +- Every record transition the job writes goes through `selfServeApply` (Task 7): the write is bounded to the lease's remaining validity; a lost response is reconciled by re-reading the record; after every landed write the serial is bumped **at least once**, tracked by the record's `record_seq` in `notified_seq`. |
| +- One console runs the pass under `control/offline-moves/console.json` (a 60 s lease, renewed every 20 s during a tick, checked before every write attempt with a fresh clock, and bounding every store write's context). The SQLite journal is the console's; a replacement console takes over by opening the **same** database file (one console host per deployment; the lease guards the day there are two) and recovers from the rows and the records. |
| +- Refusals before writes; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| +- Participants are pinned at admission as `{BoxID, Label, InstanceID, BootID, Verified}` from the edges file and the box's status report; **one resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate; **one identity check** (`participantStillPresent`, Task 6) is applied at every gate: the label's current status report must carry the pinned `InstanceID` (and, when the box record `control/boxes/<label>.json` exists, its signed identity must name that instance), else "box <label> was replaced". A move without `Participants` keeps today's region rule everywhere. |
| +- Timestamps in SQLite are Unix nanoseconds; request identities are the **committed** record stamps read back after the write. |
| +- Copy rows pending Paul's review (Tasks 5 and 8). No tenant names. |
| +- Census: as rev 3 (`plan14-root-reads-self-serve-moves-2.tsv` loaded last with `@fields` and reviewed envelopes; `plan14SelfServeMoves2WriteCensusInventory()` with every added or changed writer including the console's `open.json`, events and timings; both green). |
| - Local tests: touched packages with `-run`; CI runs the full suite. |
| ## Review Focus |
| -1. A console crash after a landed write but before its SQL update or its serial bump, at every transition boundary (queue, staging, restage, precheck, start, seal, flip, open, finishing, finish, abort), and a second crash during the recovery of each: the row is reconciled to the record, the bump is made once, and no second request is issued while the first is unanswered (Task 7, `TestEveryBoundaryRecovers`). |
| -2. A reservation older than the grace period: reclaimable only on positive evidence (the record carries no move with its id **and** the row is absent or terminal-and-released); a SQL read failure, a store read failure, a malformed body, an absent or incomplete cleanup status, or an outstanding delayed queue write never reclaim; a retry preserves the original `ReservedAt`; switch-off recovery re-acquires its own reservation (Task 3, `TestReclaimOnlyOnEvidence`). |
| -3. A deadline expiring while a box is mid-copy or mid-seal: `abort.requested_at` is set; `abort.source_reopened_at` needs the sealer's durable `tombstone_lifted` (when a tombstone existed) and every source's `reverted`; `abort.finished_at` needs every report; a source `reverted` before the sealer's lift sets nothing (Task 8, `TestAbortIsInitiatedNotCompleted`, with the real sealer). |
| -4. `cut` naming a stale or missing id → 409; a second `cut` during `cutting` → 202; `cancel` during `cutting` → 409; cancel committed first then cut → cut 409 from the row; cut committed first then cancel → cancel 409; a worker holding an older row snapshot cannot advance a cancelled row (Task 5, `TestMutationsNameTheMoveAndRace`, barrier-ordered). |
| -5. At the three self-serve mutations: a plain instance token, an admin token, another app's move-capable token and a member's token get the move family's byte-identical 404 **before any store or SQL evidence read**, against absent, eligible, ineligible and unreadable records, with admission on, off with an open row, and off with a terminal row; an invalid or revoked credential keeps today's authentication answer (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| +1. A crash after a landed write at every transition boundary (queue, staging, restage, precheck, start, seal, flip, open, both finish gates, finish, abort), before the SQL update and before the bump, with a second crash during each recovery: the row reconciles forward without regressing, requests keep their committed identities and deadlines, and the serial is bumped at least once (Task 7, `TestEveryBoundaryRecovers`). |
| +2. A reservation older than the grace period: reclaimable only on positive evidence; a SQL or store read failure, a malformed body, an incomplete or non-terminal cleanup status, an outstanding admission write (even one the server commits after the client timed out), an id-less sentinel reservation, or switch-off with two disjoint existing reservations never reclaim or obstruct recovery (Task 3, `TestReclaimOnlyOnEvidence`). |
| +3. A deadline expiring mid-copy or mid-seal: the three abort facts in order from the real sealer's durable lift (Task 8, `TestAbortIsInitiatedNotCompleted`). |
| +4. Cut and cancel in both barrier orders and against a stale worker snapshot; cut repeated during cutting → 202 (Task 5, `TestMutationsNameTheMoveAndRace`). |
| +5. At the three mutations in self-serve mode: a plain instance token, an admin token, another app's move-capable token and a member's token get the family's 404 with zero store and zero SQL reads, against every record state and with admission on or off; outside self-serve mode the legacy doors answer as main (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| --- |
| ### Task 1: The move-capable token and the self-serve gate |
| **Files:** |
| -- Modify: `internal/console/authstore.go` (`ALTER TABLE cli_tokens ADD COLUMN moves INTEGER NOT NULL DEFAULT 0`; `claimRawNewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `CLITokens` and every `SELECT ... FROM cli_tokens` feeding `scanCLITokenRows` select `moves`), `internal/console/svcprincipal.go` (`Principal.Moves bool`, `Principal.TokenID string` (the stored `token_hash`, the listing id); `CLIPrincipal` sets both), `internal/console/apiinstances.go` (`handleAPIInstanceToken`: body `{"label","moves"}`; `moves` requires `RoleOwner` else 403 "only an account owner can mint a move-capable instance token"; audit "minted a move-capable instance token for <app> (cli), token <id>"), the tokens page template |
| +- Modify: `internal/console/authstore.go` (`moves` column; `claimRawNewAppToken(..., moves bool, ...)`; every `cli_tokens` SELECT feeding `scanCLITokenRows` selects it), `internal/console/svcprincipal.go` (`Principal.Moves bool`, `Principal.TokenID string` = the stored `token_hash`), `internal/console/apiinstances.go` (`handleAPIInstanceToken` with `moves`; owner-only; the audit sentence), the tokens page template |
| - Create: `internal/console/selfserve_gate.go` |
| - Test: `internal/console/instancetoken_moves_test.go`, `internal/console/selfserve_gate_test.go` |
| **Interfaces:** |
| -- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; existing callers pass `false`. |
| -- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`: `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` then `p, _ = principalFrom(r.Context())` (today's authentication answers for an invalid or revoked credential), `host := r.PathValue("host")` with `ValidHostShape` (404 as `moveGate`), then `moveCapable(p, acct, app)` else `http.NotFound(w, r)`; **no store and no SQL read before this returns**. `moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`. |
| -- **The dispatcher** at the four doors (Task 5): `handleAPIInstanceMoveStatus`: `moveGate` as today, then `s.store.SelfServeMoveForHost` (a SQL failure → 502 "could not read the move"); a row → the self-serve view (Task 5's precedence); none → today's view. `Request`, `Cut`, `Abort`: when `s.cfg.MovesEnabled` → Phase 1 as today; else `selfServeGate` → the row read → a row, or admission on (`Request` only) → the self-serve body; else → the legacy body **after** re-running `moveGate` so its answers are byte-identical (the capability check is only reached when a self-serve body may run: with no row and admission off, the legacy body is served without a capability check, as today). |
| -- The CLI (`carlos instance-token --moves`) is a carlos-repo change. Existing tokens have `moves = 0`. #155's host-bound tokens: a comment names the future refusal. |
| +- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`. |
| +- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`: `instanceGateSaying(w, r, "instance", mainMoveCopy().gate)` (the principal, the account and app, **the rate limiter and the no-deployment-bucket 503**, as every instance door), `principalFrom(r.Context())`, `host` with `ValidHostShape` (404), then `moveCapable(p, acct, app)` else `http.NotFound`; no store and no SQL read before it returns. |
| +- **The dispatcher**: `handleAPIInstanceMoveRequest`, `Cut`, `Abort`: `s.cfg.MovesEnabled` → Phase 1 as today; `s.selfServeMode()` → the self-serve body (Task 5); else the legacy body. `handleAPIInstanceMoveStatus`: `s.cfg.MovesEnabled` → Phase 1 as today; else `moveGate` (today's; an absent record is today's 404), then in self-serve mode the row join (Task 5's precedence; a SQL failure → 502 "could not read the move"), else today's view. |
| -- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (both listings, `Principal.Moves`, `Principal.TokenID` equals the listing id, the audit line); `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (a counting store **and** a counting SQL wrapper: for each of a plain instance token, an admin token, another app's move-capable token and a member's token, against an absent, an eligible, an ineligible and an unreadable record, with admission on, off with an open row, and off with a terminal row → the 404 with zero store reads and zero `self_serve_moves` reads; the owner's move-capable token passes; a revoked one → today's 401; with no row and admission off a plain instance token reaches the legacy body exactly as today). |
| +- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken`; `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (counting store and SQL: in self-serve mode, each unauthorised token against absent, eligible, ineligible and unreadable records, with admission on and off → 404 with zero reads; the owner's token passes; a revoked token → 401; the rate limiter and the no-bucket 503 answer as the instance gate does; outside self-serve mode a plain instance token reaches the legacy body exactly as main). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'MoveCapable|Mint' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the move-capable instance token, owner-minted, and the self-serve gate that authorises before any evidence is read`. |
| +- [ ] **Step 5: Commit** `console: the move-capable instance token and the self-serve gate, dispatched by a process-level mode`. |
| -### Task 2: The queued state, the precheck stamp and the self-serve fields on the record |
| +### Task 2: The queued state, the precheck stamp, the open record and the self-serve fields |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove \`json:"self_serve,omitempty"\``, `Participants *Participants \`json:"participants,omitempty"\``, `QueuedAt *time.Time \`json:"queued_at,omitempty"\``, `PrecheckAt *time.Time \`json:"precheck_at,omitempty"\``, `ReturnOf string \`json:"return_of,omitempty"\``), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: `offline-queued` → ""; `offlineViewOf`: `CanAbort` true in queued, `NeverOffline` from `StartedOffline`), `internal/provision/offline_move_test.go` (the allowlist) |
| -- Test: `internal/provision/offline_queue_test.go` |
| +- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove`, `Participants *Participants`, `QueuedAt`, `PrecheckAt *time.Time`, `ReturnOf string`, all `omitempty`), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: queued → ""; `offlineViewOf`: `CanAbort` in queued, `NeverOffline` from `StartedOffline`; **the open door and the job write the open record** `control/offline-moves/<host>/<id>/open.json` create-only after a committed `OfflineOpenTransition`: `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; OpenedAt time.Time \`json:"opened_at"\` }` with `OpenedAt = Move.OpenedAt`; a failed write is retried on the next tick or the next operator door call (idempotent, create-only)), `internal/moves/offline.go` (`OpenRecordKey(host, id)`), the allowlist test |
| +- Test: `internal/provision/offline_queue_test.go`, `internal/console/openrecord_test.go` |
| **Interfaces:** |
| -- `const OfflineQueued = "offline-queued"`. |
| -- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\` }`. |
| -- `type Participant struct { BoxID string \`json:"box_id"\`; Label string \`json:"label"\` }`; `type Participants struct { Source []Participant \`json:"source"\`; Target Participant \`json:"target"\` }`. |
| -- `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, now time.Time) func(*Record) error`: `OfflineStage`'s refusals plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager: p.Source[0].Label`, `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: now`, `QueuedAt: &now`, `StagedAt: &now`, `PrecheckAt: &now`, `Seq: rec.MoveSeq`; `move_seq` unchanged. |
| -- `func OfflineQueuedToStaging(id string, now time.Time) func(*Record) error` (`StagedAt = &now`); `func OfflinePrecheck(id string, now time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &now`); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, now time.Time) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: now, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour, so the console calls `OfflineAbortAt` for a queued move and `OfflineAbort` otherwise; a repeated abort on a reverting record stays a success without a write). `OfflineStartStaged` sets `SelfServe.OfflineAt = &now` when `SelfServe != nil`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove` (after an earlier move at `move_seq 3`: `LastMove` aborted at `now`, `move_seq` 3; from any other state it behaves as `OfflineAbort`; a repeated abort is a no-write success); `TestOfflineSetEstimateRefusesWithoutSelfServe`; the allowlist test. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|DiffersOnlyWhere' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, then the record-write census. |
| -- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort and the self-serve move fields`. |
| +- `const OfflineQueued = "offline-queued"`; `SelfServeMove`, `Participant{BoxID, Label, InstanceID, BootID string; Verified bool}` (all tagged snake_case), `Participants{Source []Participant; Target Participant}`; `OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, now time.Time)` (as rev 3: refusals, `Incarnation` match, `StagedAt = PrecheckAt = &now`, `Seq = MoveSeq`); `OfflineQueuedToStaging(id, now)`; `OfflinePrecheck(id, now)`; `OfflineSetEstimate(id, seconds)`; `OfflineAbortAt(id, now)`; `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged` sets `SelfServe.OfflineAt`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: as rev 3 plus `TestOpenRecordWrittenOnceAfterCommittedOpen` (the operator open door and the job both write it; a second write is a no-op; a failed write is retried); `TestReturnOfCarriedByStageAndStart`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|ReturnOf|DiffersOnlyWhere' -count=1`, `go test ./internal/console -run 'OpenRecord' -count=1`, then the record-write census. |
| +- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort, the open record and the self-serve fields`. |
| -### Task 3: Per-box reservations in active.json, with a legacy projection |
| +### Task 3: Per-box reservations with a legacy projection and a sentinel |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineActive` gains `V int \`json:"v,omitempty"\`` and `Reservations map[string]OfflineReservation \`json:"reservations,omitempty"\`` keyed by box id; the legacy fields are the **projection**), `internal/console/apiinstancemove_offline.go` (the start and stage doors reserve from the selection; `offlineReleaseLock` → `releaseBoxes`) |
| +- Modify: `internal/moves/offline.go` (`OfflineActive.V`, `Reservations map[string]OfflineReservation` keyed by box id, the legacy projection), `internal/console/apiinstancemove_offline.go` (the doors reserve from the selection) |
| - Create: `internal/console/offlinereserve.go` |
| - Test: `internal/console/offlinereserve_test.go` |
| **Interfaces:** |
| -- `type moves.OfflineReservation struct { Kind string \`json:"kind"\`; ID string \`json:"id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; ReservedAt time.Time \`json:"reserved_at"\` }` (`Kind` is `move` or `cleanup`; an entry with an unknown kind, an empty id or host, or a zero time is malformed: the whole object is `unreadable` until an operator repairs it). |
| -- `func (s *Server) reserveBoxes(ctx context.Context, boxes []provision.Participant, r moves.OfflineReservation) *offlineRefusal`: reads `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}` and is normalised into the table on the first new write; refuses `busy` ("<host> is moving offline. move one instance at a time") when any participant is held by an entry with another `(Kind, ID)`, unless reclaimable; an entry with the same `(Kind, ID)` is this operation's own (a retry) and keeps its `ReservedAt`; writes every participant's entry and the projection (`Host, Op, Account, App, At` of the oldest `move` entry; of the oldest entry of any kind when no move holds) in one conditional `Put`; a CAS conflict re-reads once and retries, then `busy`. **Concurrency gate:** while `cfg.SelfServeMoves != "offline"` the helper refuses when any entry of another operation exists (today's deployment-wide semantics). |
| -- `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error`: removes every entry with that `(Kind, ID)`, rewrites the projection, deletes the object when empty (`DeleteIfMatch`); one conditional write; absence is success. |
| -- Reclaim: an entry is reclaimable when `time.Since(ReservedAt) >= offlineLockGrace` and, for `move`: the judged record of `(Account, App, Host)` carries no offline move with `ID` **and** `SelfServeMove(id)` is absent or terminal with `release_pending = 0` **and** no `request_issued` admission write is outstanding for that row (a row in `reserving` whose `request_issued` is set is not reclaimable until its own recovery resolves it); for `cleanup` (plan 4): the owning box's status report (`status/boxes/<label>.json` of the box id's label, `GeneratedAt` within 2 status intervals, `Metrics.BoxID` equal to the key) lists the id in a terminal state with `RootRequest == ""` and `ReleasePending == false`, or its `Retirements` block is **complete** (`RetirementsComplete`) and lists no such id. Any read failure, a malformed body, or an absent or incomplete status → not reclaimable. |
| -- **Rollout and rollback, stated:** per-box concurrency starts when the switch is turned on, after every console runs this code; an older console cannot be reintroduced while concurrent reservations exist, because its release deletes the whole object when the projection names its own move (the hazard `TestOldConsoleReleaseErasesATable` documents) and its take refuses everything else; the supported sequence is roll consoles → turn the switch on; rollback is switch off → wait for the table to drain to one entry → the old console. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnReservationIsIdempotentAndKeepsReservedAt`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive and each refusing case above, including a SQL failure and an outstanding admission write); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody` (the old take is busy on any table; the old release leaves a table whose projection names another move; `TestOldConsoleReleaseErasesATable` documents the hazard the rollout sequence avoids); `TestSwitchOffRecoveryReacquiresItsOwnReservation`; `TestOperatorDoorsUseReservations`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|OperatorDoorsUse|Malformed' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Offline' -count=1`. |
| -- [ ] **Step 5: Commit** `console: active.json holds per-box reservations with a legacy projection and a stated rollout`. |
| +- `moves.OfflineReservation{Kind, ID, Host, Account, App string; ReservedAt time.Time}` (tagged); **the sentinel key `"*"`**: a reservation of a selection whose participants lack a verified id (the operator's `requireIDs: false` path) is written under `"*"` and conflicts with every other acquisition, and every acquisition conflicts with it (so an id-less operator move and a self-serve move on the same physical box never coexist). |
| +- `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`: an own `(Kind, ID)` entry (any key) is a retry: accepted first, keeping `ReservedAt`, **before** the exclusive gate; then: the legacy body normalised; a malformed entry → unreadable; `cfg.SelfServeMoves != "offline"` → refuse when any other operation holds anything; any participant (or `"*"`) held by another `(Kind, ID)` → busy unless reclaimable; one conditional write with the projection; one CAS retry. |
| +- `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error` (as rev 3). |
| +- Reclaim (`move`): the judged record carries no move with `ID`, **and** `SelfServeMove(id)` is absent or terminal with `release_pending = 0`, **and** no row for the host has `request_issued` set; a SQL or store failure → not reclaimable. (`cleanup`): plan 4's contract (`RetirementsComplete`, `State == ended`, `Metrics.BoxID`). |
| +- Rollout: as rev 3, stated in terms of operations: old consoles cannot be reintroduced while any reservation exists that the old console did not create; rollback = switch off, wait for every self-serve operation to end, then the old console. |
| + |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestSentinelReservationConflictsBothWays`; `TestOwnRetryAcceptedBeforeTheExclusiveGate` (switch off, two disjoint reservations, each retries its own → accepted); `TestServerCommittedQueueWriteIsNotReclaimable` (a queue write that timed out client-side and commits later: the reservation is not reclaimed while `request_issued` is set, and Task 7's fence makes the late commit impossible before release). |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|Sentinel|OwnRetry' -count=1`. |
| +- [ ] **Step 5: Commit** `console: per-box reservations with a legacy projection, a deployment-wide sentinel and own retries`. |
| -### Task 4: The self_serve_moves journal, the console lease and the pass skeleton |
| +### Task 4: The journal, the events table, the console lease and the pass skeleton |
| **Files:** |
| -- Modify: `internal/console/store.go`, `internal/console/store_gate.go` (`"self_serve_moves"`, `"self_serve_events"`: `{Sqid: "account", App: "app"}`; `"move_timings"`: `{NonApp: ...}`), `internal/console/claim_workers.go`, `cmd/console/main.go` (`cfg.SelfServeMoves`; `armMovesPaging(ctx, cfg, store, getenv)`: arm when `cfg.HostClaim`, or `cfg.SelfServeMoves == "offline"`, or the store has open rows or rows with `release_pending`, or `active.json` has any entry (a store read failure → arm, logged)), `internal/console/server.go` (`Config.SelfServeMoves string`; `func (s *Server) selfServeAdmits() bool`) |
| +- Modify: `internal/console/store.go`, `internal/console/store_gate.go` (`"self_serve_moves"`, `"self_serve_events"`, `"self_serve_timings"`: `{Sqid: "account", App: "app"}`; `"move_timings"`: `{NonApp: ...}`), `internal/console/claim_workers.go`, `cmd/console/main.go` (`armMovesPaging(ctx, cfg, store, getenv)`), `internal/console/server.go` (`selfServeMode`, `selfServeAdmits`), `internal/console/serialbump.go` (`func (s *Server) bumpSerialErr(ctx) error`, the existing `bumpSerial` calling it and logging) |
| - Create: `internal/console/selfserve_store.go`, `internal/console/selfserve_pass.go`, `internal/console/selfserve_lease.go` |
| - Test: `internal/console/selfserve_store_test.go`, `internal/console/selfserve_pass_test.go` |
| **Interfaces:** |
| -- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, stage_request INTEGER, consumed_round INTEGER, rounds INTEGER, precheck_request INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_state TEXT, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| -- Table `self_serve_events (id TEXT, phase TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, phase))`: the claims-off move log and the member-stream outbox (Task 9). |
| -- Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. |
| -- `type SelfServeMove struct { ID, Account, App, Host string; Incarnation uint64; From, To string; Participants provision.Participants; DisplayName, RequestedBy, Phase string; PhaseStartedAt, StepDeadline, WindowDeadline, StagingStartedAt time.Time; RequestIssued time.Time; RequestKind string; StageRequest, ConsumedRound time.Time; Rounds int; PrecheckRequest time.Time; EstimateSeconds int; EstimateTerms estimateTerms; Reason string; HealthConsecutiveOK int; HealthFirstOKAt, HealthLastProbeAt time.Time; HealthLastError string; FinishGate, NotifiedState string; Outcome, OutcomeReason, Warning string; AbortRequestedAt, SourceReopenedAt, AbortFinishedAt time.Time; EffectsDone, ReleasePending bool; CreatedAt, UpdatedAt time.Time }`. |
| -- On `*Store`: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open *SelfServeMove, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0`); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, now time.Time) (bool, error)` (`UPDATE ... WHERE id = ? AND phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done'))`); `IssueSelfServeRequest(id, kind string, at time.Time) (bool, error)` (`WHERE request_issued = 0`); `ConfirmSelfServeRequest(id string, at time.Time) error` (`stage_request`/`precheck_request = at`, `request_issued = 0`); `ConsumeSelfServeRound(id string, request time.Time) (bool, error)` (`WHERE consumed_round <> ?` then `rounds = rounds + 1, consumed_round = ?`); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` set on the first ok and kept), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotified(id, state string) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE id = ? AND phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`; `InsertSelfServeEvent(e SelfServeEvent) (inserted bool, err error)`, `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`, `MarkSelfServeEventEmitted(id, phase string) error`. |
| -- Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `type consoleLease struct { Console string; Until time.Time }`; `func (s *Server) acquireSelfServeLease(ctx context.Context, now time.Time) (*leaseToken, bool, error)`; `func (t *leaseToken) renew(ctx context.Context, now time.Time) error` (every 20 s during a tick, from the tick's loop between steps and inside the health probe wait); `func (t *leaseToken) valid(now time.Time) bool` (checked by `selfServeApply` and every SQL mutation of the pass before writing; false → the tick ends with "lease lost"). Another console's live lease → not acquired (logged once per minute). |
| -- `func (s *Server) StartSelfServeMovesPass(ctx context.Context)` (runs when `selfServeAdmits()`, or `OpenSelfServeMoves()` is non-empty, or `active.json` has any entry at start; every 5 s: acquire or renew the lease, `s.selfServeTick`); `func (s *Server) selfServeTick(ctx context.Context, now time.Time)`: the reservation scan (Task 8), then per open row `s.selfServeStep` (at most one step), errors logged per row. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestAdvanceIsPredicatedOnPhaseAndNoAbort` (a cancelled row cannot advance to `restage`; it can to `reverting`); `TestIssueConfirmConsume` (issue refused while one is outstanding; consume counts a request once); `TestRequestAbortOnce`; `TestHealthCounters`; `TestOpenRowsIncludePendingReleaseAndEffects`; `TestPassStartsOnAdmissionOpenRowsOrReservations` (each alone; none → off); `TestPassRunsOpenRowsWithoutAdmission`; `TestTickAdvancesEachRowAtMostOneStep`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseLapseStopsWrites` (the holder is paused past 60 s with a barrier; another console acquires; the paused holder's next write is refused); `TestPagingArmedForSelfServe` (the real `armMovesPaging` with its store argument: each condition); `TestClaimStoreOpensWithSelfServeTables`. |
| -- [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseLapse|PagingArmed|IssueConfirm|ClaimStoreOpensWithSelfServe' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self_serve_moves journal with issued and consumed identities, the events table, the console lease and the pass skeleton`. |
| +- `self_serve_moves`: rev 3's columns with these changes: `request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_confirmed INTEGER, request_consumed INTEGER` (the lifecycle of one request: issued at `request_issued` by `request_phase`; confirmed with the **committed** stamp read from the record; consumed when its answer was used; a new request may be issued only when the previous one is consumed or voided), `stage_request`, `precheck_request` (the committed stamps the job currently waits on), `notified_seq INTEGER` (the record's `record_seq` last announced), `expiry_pending INTEGER`, `health_attempts INTEGER`, `health_consecutive_ok INTEGER`, and the rest as rev 3. `self_serve_events (id TEXT, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, phase))`; `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| +- Phases: rev 3's. **Compatible sets** for reconciliation (Task 7): `offline-queued` ↔ {queued, reports}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier}; `offline-stopping` ↔ {stopping}; `offline-sealing`, `offline-sealed` ↔ {sealing}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain}; `offline-open` ↔ {opened, health, finishing(gate 1)}; `offline-finishing` ↔ {finishing(gate 2)}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not already in that set. |
| +- Store methods: rev 3's, with `IssueSelfServeRequest(id, kind, phase string, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`), `ConfirmSelfServeRequest(id string, committed time.Time) error`, `ConsumeSelfServeRequest(id string) error`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiryPending(id string, pending bool) error`, `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, now time.Time) (bool, error)` with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing'))`, `InsertSelfServeEventTx(tx, e)` used inside the same transaction as `AdvanceSelfServe` (one transaction advances the phase and inserts its event), `DeliverSelfServeEvent(id, phase string, deliver func() error) error` (marks emitted in the same transaction as the audit row's insert, through `auditActivityTx`; a failure leaves it undelivered). |
| +- Lease: `acquireSelfServeLease`, `leaseToken.renew`, `leaseToken.valid(now)`, `leaseToken.deadline() time.Time`; `selfServeApply` derives `ctx, cancel := context.WithDeadline(ctx, t.deadline())` for the record write so no PUT or retry outlives the lease, and re-checks `t.valid(now)` before each retry inside `SetRecordMove` through a `provision.WithWriteGuard(ctx, func() error)` hook that `WriteRecord` calls immediately before every `Put`. |
| +- `StartSelfServeMovesPass`: starts when `selfServeMode()` (which includes the recovery conditions); a failing startup read of rows, events or `active.json` → the pass runs (fail closed) and retries the reads each tick, logging; `selfServeTick`: renew the lease, the reservation scan, `drainSelfServeEvents`, then one step per open row. |
| + |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestRequestLifecycleIssueConfirmConsume` (no second issue while unconsumed; confirmation records the committed stamp; consume once); `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestEventAndAdvanceAreOneTransaction` (a crash after the advance cannot lose its event); `TestEventDeliveryIsTransactional` (an audit insert failure leaves the event undelivered; a crash after the audit insert does not duplicate it); `TestPassStartsAndRetriesOnReadFailure`; `TestLeaseBoundsEveryWriteAttempt` (a `Put` paused past the lease deadline is cancelled by the context; a retry after lapse is refused by the guard); `TestClaimStoreOpensWithEveryNewTable`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseBounds|RequestLifecycle|EventAnd|EventDelivery|PagingArmed|ClaimStoreOpens' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve journal with request lifecycles and compatible phase sets, transactional events, a lease that bounds every write, and the pass skeleton`. |
| -### Task 5: Record eligibility, the shared start validation, the door bodies and the view |
| +### Task 5: Record eligibility, the split start validation, the door bodies and the view |
| **Files:** |
| -- Modify: `internal/console/apiinstancemove.go` (`moveGate`'s record checks move into `recordEligibility`; `moveGate` calls it so its bytes are unchanged; the dispatcher of Task 1), `internal/console/apiinstancemove_offline.go` (`offlineBeginChecks` becomes a wrapper over `offlineStartValidation`; the start and stage doors use the selection; `offlineStepFor`'s staging branch calls `offlineStartValidation` so the start-from-staged path carries every check; the stage door accepts `return_of` (Task 8's operator return)) |
| +- Modify: `internal/console/apiinstancemove.go` (`recordEligibility`; `moveGate` calls it), `internal/console/apiinstancemove_offline.go` (`offlineBeginChecks` = `offlineRecordChecks` → `readJSON` → `offlineRegionChecks`, byte-identical; the staging branch of `offlineStepFor` calls both) |
| - Create: `internal/console/selfserve_door.go`, `internal/console/selfserve_view.go` |
| - Test: `internal/console/selfserve_door_test.go`, `internal/console/apiinstancemove_offline_snapshot_test.go` |
| **Interfaces:** |
| -- `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal`: `moveGate`'s checks in order: `Kind` not instance (`c.kind`), `Addr`/`Addrs` (`c.addr`), fleet placement (`c.fleet`), `Pool` (`c.pool`), `!HibernateOrDefault()` (`c.hibernate`); 400s. |
| -- `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string }`. |
| -- `func (s *Server) offlineStartValidation(ctx context.Context, acct *Account, app, host string, rec *provision.Record, region string, stage bool, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)`: `recordEligibility(rec, mainMoveCopy())`, then every check of `offlineBeginChecks` and the stage door in order with their sentences (`rec.Seed.MoveBlock()`, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`, the region, the manifest's regions, `readOfflineFleet`, the home-region rule, `offlineSelect`; with `stage`: `offline-stage-1` on every box, the distinct-bucket rule, the stager pinned); with `requireIDs` (the self-serve door) every participant needs a non-empty `BoxID` (else the older-release sentence); the operator doors pass `requireIDs: false` and keep accepting publications without ids (compatibility, stated). `returnOf` is Task 8's. `TestOperatorDoorAnswersAreUnchanged` snapshots every operator refusal, including a malformed body combined with each class refusal, from main and replays them. |
| -- `POST .../move` (self-serve body): `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` → 400 "display_name must be at most 80 characters of plain text"; `offlineStartValidation(..., region, stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source box's status facts (Task 6: the report of the pinned source's label, `Metrics.BoxID == Source[0].BoxID`, `GeneratedAt` within 10 minutes, `Hosts[host].Known`, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` becomes the returned record and `recordEligibility`, the run and move checks run again on it. Then `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. |
| -- `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating (a legacy record is estimated as is); 200 `{"estimate_seconds": n, "files_bytes": b, "replica_bytes": r}` or the refusal. |
| -- `POST .../move/cut {"id"}`: the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `AdvanceSelfServe(id, ready, restage, ...)` → 202; false → re-read: a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". |
| -- `DELETE .../move {"id"}`: the row and id; `RequestSelfServeAbort(id, [requested..ready], now)` → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| -- **The view's precedence** (`selfServeViewOf(rec, open, terminal)`): (1) the record carries an offline move with `SelfServe != nil` → the open row's view (`cutting`/`ready`/`preparing` by phase); (2) no such record move but an open row whose `Incarnation == rec.Incarnation` → `preparing` (pre-record phases); (3) the record carries another move (an operator's) → today's legacy view, no `last`; (4) otherwise a terminal row whose `Incarnation == rec.Incarnation` (or the record is absent and the row is the newest) and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| -- Copy rows: as rev 2 plus "self-serve moves are not enabled on this console"; "could not measure the account's files. try again in a minute"; "could not read the move". |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged`; `TestSwitchOffIsByteIdentical`; `TestRecordEligibilityIsMoveGatesChecks`; `TestPrepareRefusesBeforeWriting` (one subtest per refusal including admission off, over-cap and truncated facts, a stale status report, a status report of another box id); `TestPrepareMigratesALegacyRecordBoundThenAdmits` (a recreate between the observation and the migration → 409 settling, nothing admitted; the admitted row carries the migrated record's incarnation and its eligibility was re-checked); `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (barriers: cancel commits before cut → cut 409; cut commits before cancel → cancel 409; a worker with an older row snapshot cannot advance a cancelled row; a second cut during cutting → 202); `TestViewPrecedence` (each of the five cases, a terminal row older than 7 days, a recreation after a terminal row, an operator move after a terminal row); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|RecordEligibility|Prepare|MutationsName|ViewPrecedence|EstimateDoor' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Move' -count=1`. |
| -- [ ] **Step 5: Commit** `console: record eligibility and one start validation shared by the operator doors and the job; the self-serve door, its refusals and its view`. |
| +- `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal` (`moveGate`'s class checks in order). |
| +- `func (s *Server) offlineRecordChecks(ctx context.Context, acct *Account, app, host string, rec *provision.Record) *offlineRefusal` (the pre-body half: the seed block, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`); `func (s *Server) offlineRegionChecks(ctx context.Context, acct *Account, app, host string, rec *provision.Record, region string, stage, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)` (the post-body half: the region, the manifest, the fleet, the home rule, `offlineSelect`, the stage rules, participant pinning with `requireIDs`, the return's evidence); `offlineSelection{From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string; Sentinel bool}` (`Sentinel` when any participant lacks a verified id). Participant pinning: `BoxID` and `Label` from the edges file; `InstanceID` and `BootID` from `status/boxes/<label>.json`'s metrics (fresh within 10 minutes; else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, label)` finds a record and `boxid.Verifier.Enroll` yields evidence naming that `InstanceID` (a record that exists but fails → refuse "box <label> has an identity record that does not verify"); with `requireIDs`, `BoxID`, `InstanceID` and `BootID` must all be present (else the older-release sentence). |
| +- The self-serve bodies: as rev 3, with: `selfServeGate` (Task 1); `s.selfServeAdmits()` checked first in prepare; the record observation; `recordEligibility`; the run and move checks; the open row; the body; `offlineRecordChecks` and `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box; the status facts (Task 6: the pinned source's report, `Metrics.BoxID`, `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future; `OverCap` → the files sentence; `Truncated` → the measurement sentence); the lane; the bound migration; `AdmitSelfServeMove`. |
| +- **The view's precedence** (`selfServeViewOf(rec, open, terminal)`): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.SelfServe.ID` → the open row's view; (2) the record carries any other move (an operator's, or a self-serve move whose row this console does not hold) → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. An absent record is today's 404 from `moveGate`. The types as rev 3. |
| +- Copy rows: rev 3's plus "box <label> has an identity record that does not verify"; "the box <label> has not reported recently". |
| + |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestStartValidationSplitPreservesOperatorOrdering` (the pre-body refusals answer before a malformed body, as main); `TestParticipantPinning` (verified with a box record; pinned unverified without one; a record that fails to verify refuses; `requireIDs` refusals); `TestViewPrecedenceOperatorMoveOverOpenRow` (an open row in `reserving` and an operator move on the record → the legacy view). |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|RecordEligibility|StartValidationSplit|ParticipantPinning|Prepare|MutationsName|ViewPrecedence|EstimateDoor' -count=1`. |
| +- [ ] **Step 5: Commit** `console: record eligibility and the split start validation shared by the operator doors and the job; participants pinned with identity; the self-serve door and view`. |
| -### Task 6: The boxes' reports: the participant resolver, the lane, the source precheck, target-ready, the staged inventory, and the per-host status block |
| +### Task 6: The boxes' reports: the participant resolver and identity, the lane, the source precheck, target-ready, the staged receipts, and the per-host status block |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `Staged map[string]StagedObject` (exists; now the stager's complete destination inventory), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`; **the validator matrix** in `ValidateOfflineReport`: role `lane` → phase `lane`, `Lane != nil`, `BoxID != ""`; role `source` + phase `precheck` → `Precheck != nil`, `BoxID`, `Request` non-zero; role `target` + phase `ready` → `Ready != nil`, `BoxID`, `Request`; role `stager` + phase `staged` → `StagedBucket`, `StagedRegion` non-empty and `Staged` present (nil is "not reported"); role `sealer` + phase `reverting` → `TombstoneLifted` set; any report of a move with `Participants` → `BoxID` non-empty; `func ParticipantRole(m *provision.MoveState, boxID, label, placement, home string) (source, target, sealer, stager bool)` (with `Participants`: by `BoxID` only; without: today's region and label rules)), `internal/host/movesoffline.go` (`moveCfg.BoxID` from the agent's box id, the value the edges writer publishes; `offlineRound` and `offlineReassert` use `ParticipantRole`; `offline-queued` → the source participant: `laneProbe` (the existing `movesstore.go:90` helper over `cfg.Versioned`, key `moves.LaneKey(id)` in the target bucket) once per move then the `lane` report, and `offlineSourcePrecheck` for the current `PrecheckAt`; the target participant: `offlineTargetReady` for `PrecheckAt`; `offline-staging` → the stager round and both participants re-reporting when `PrecheckAt` is newer than their last `Request`; `sourceReport` (the stop report) gains the precheck facts), `internal/host/movesoffline.go` (`offlineView.hold`/`keep` consult `ParticipantRole` when the move carries `Participants`), `internal/host/movesoffline_seal.go` (`offlineTarget`: with `Participants`, the target participant's edges file by `BoxID`, else by region; `offlineSealRevert`: after `LiftTombstone` succeeds, **before** the prefix cleanup, write a sealer report `{Phase: reverting, TombstoneLifted: true, TombstoneLiftedAt}`; the final `reverted` report keeps both fields; the normal seal's final-copy bytes go in `FinalCopiedBytes`), `internal/host/movesoffline_stage.go` (the stager report: `RoundCopiedBytes/RoundDeferredBytes/RoundDeferredObjects` from `MirrorRound`, `Rounds`, and `Staged` = `MirrorStore.List(StageBucket, moves/<host>/)` taken after the round, `StagedBucket/StagedRegion` from the record), `internal/host/movesoffline_land.go` (`offlineTargetReady`), `internal/host/agent.go` (the status tick: `p.Hosts`, `p.Lanes`, `p.BoxID`), `internal/status/snapshot.go` (`Probes.Hosts map[string]HostFacts`, `Probes.Lanes []string`, `Probes.BoxID string`; `CollectBox`'s attachment condition includes them; `BoxMetrics.Hosts`, `BoxMetrics.Lanes`, `BoxMetrics.BoxID` private), `internal/host/movespass.go` (`armOffline`: `cfg.Versioned` (today assigned only in `armPhase1`) and `cfg.LaneProbe` are armed for offline-capable boxes too) |
| -- Test: `internal/moves/offline_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineReport` gains `BoxID`, `InstanceID`, `Request`, `Lane`, `Ready`, `Precheck`, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject` (the receipts' objects: `StagedObject{Size, ETag, VersionID string \`json:"version_id,omitempty"\`}`), `StagedBucket`, `StagedRegion`, `StagedRequest time.Time`, `RoundCopiedBytes`, `RoundDeferredBytes`, `RoundDeferredObjects`, `Rounds`, `TombstoneLifted`, `TombstoneLiftedAt`, `FinalCopiedBytes`; the new roles and phases; **`ValidateOfflineReport(r)` is unchanged for legacy structure**, and `ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState) error` adds the contextual rules: a move with `Participants` → `BoxID` and `InstanceID` non-empty on every report; a move with `SelfServe` → a `staged` report has `StagedKnown`, `StagedBucket`, `StagedRegion`, `StagedRequest`; role `lane` ↔ phase `lane` with `Lane`; `precheck` with `Precheck` and `Request`; `ready` with `Ready` and `Request`; `reverting` with `TombstoneLifted`; `readOfflineEvidence` applies both and drops a report that fails the contextual rule for its move, logging it), `internal/moves/participants.go` (new: `ParticipantRole`, `ParticipantStillPresent(p Participant, m *status.BoxMetrics) bool` = `m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID`), `internal/lease/mirror.go` (`func ReadReceipts(ctx context.Context, m MirrorStore, rc Receipts) (map[string]CopyReceipt, error)` exporting the ledger's objects, `type CopyReceipt struct { SrcETag string; SrcSize int64; DstETag string; DstVersionID string }`), `internal/host/movesoffline.go` (`moveCfg.BoxID`, `moveCfg.InstanceID` (from `status.InstanceID`); `ParticipantRole` in `offlineRound`, `offlineReassert`, the view's `hold`/`keep`; the queued and staging dispatch as rev 3), `internal/host/movesoffline_seal.go` (`offlineTarget` by participant; the `reverting` report with the durable lift before cleanup, the first lift time preserved on retries; `FinalCopiedBytes`), `internal/host/movesoffline_stage.go` (the stager report's `Staged` **from `lease.ReadReceipts` after the round** (dst ETag and version id of every object this move's copies produced), `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = StagedAt`), `internal/host/movesoffline_land.go` (`offlineTargetReady` with `PrefixFacts.Objects` carrying `ETag`, `VersionID`, `LastModified`), `internal/host/agent.go` (the status tick: `p.Hosts`, `p.Lanes`, `p.BoxID`, `p.InstanceID` is already there), `internal/status/snapshot.go` (`Probes.Hosts`, `Probes.Lanes`, `Probes.BoxID`; the attachment condition; `BoxMetrics.Hosts`, `BoxMetrics.Lanes`, `BoxMetrics.BoxID`), `internal/host/movespass.go` (`armOffline` arms `cfg.Versioned`, `cfg.LaneProbe`) |
| +- Test: `internal/moves/offline_test.go`, `internal/moves/participants_test.go`, `internal/lease/receipts_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| **Interfaces:** |
| -- `type moves.Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. |
| -- `type moves.LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the probe is `laneProbe`'s put-if-absent, list and delete-version under `moves/_lane/<move id>`; this is the existing Phase 1 lane probe reused, so the lane evidence is move-bound by its key). |
| -- `type moves.PrefixFacts struct { Known bool \`json:"known"\`; Objects map[string]StagedObject \`json:"objects,omitempty"\` }`. |
| -- `type moves.TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\` (Detail: bytes); MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\` (OK: absent; Detail: the owner and expiry when present); Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }` (`Leftovers` for the operator return, Task 8). |
| -- `type moves.ExtraDirsFact struct { Known bool \`json:"known"\`; Dirs []ExtraDir \`json:"dirs,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type ExtraDir struct { Path string \`json:"path"\`; Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\` }`; `type moves.SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; ArchiveEligible Fact \`json:"archive_eligible"\`; ExtraDirs ExtraDirsFact \`json:"extra_dirs"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; ExtraDirsRemoved []string \`json:"extra_dirs_removed,omitempty"\`; ExtraDirsLeft []ExtraDir \`json:"extra_dirs_left,omitempty"\` }`. The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. Empty extra directories are removed at `start` after quiescence: `os.ReadDir` → zero entries → `os.Remove`; a `ReadDir` or `Remove` error, or a non-empty directory, → `ExtraDirsLeft` with `Detail`; a non-directory stray entry is left for the sealer's refusal as today. |
| -- `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Known bool \`json:"known"\`; Err string \`json:"err,omitempty"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; ReplicaCollectedAt int64 \`json:"replica_collected_at"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; OverCap bool \`json:"over_cap,omitempty"\`; Truncated bool \`json:"truncated,omitempty"\`; ExtraDirs []string \`json:"extra_dirs,omitempty"\` }`: `measureFiles(ctx, files, envPath string, budget time.Duration, capBytes int64) (bytes int64, entries int, overCap, truncated bool, err error)` (a new bounded walker in `internal/host` applying the archive's counting rules (the env file counted, `archive.Cap`) without stopping at the cap: it records `OverCap` and continues until the budget (60 s) or 2 × the cap, where it records `Truncated`); the replica listing cached per tick with its own `ReplicaCollectedAt`; `Known` only when every part succeeded. |
| -- `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES`; `BoxMetrics.BoxID string` (the edges writer's value) so the console joins a participant to its report. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestParticipantRoleWithAndWithoutParticipants`; `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown` (each fact OK, refused and unknown; the full data-path set; the prefix inventories; a present lock's owner); `TestSourcePrecheckFacts` (unsupported file type; nested layout; extra directories; an inspection failure → `ExtraDirs.Known false`); `TestEmptyExtraDirsRemovedAtStartOnly` (empty → removed; one with a zero-byte file → left with `Detail`; a `ReadDir` error → left); `TestStagerPublishesACompleteInventory` (after a round, `Staged` equals a listing of the destination prefix; an overwrite between the copy and the listing shows the listed ETag; `StagedBucket/Region` from the record); `TestSealerPublishesTombstoneLiftedBeforeCleanup` (the `reverting` report exists after the lift and before the prefix deletion; a cleanup failure after the lift leaves it readable; the final `reverted` keeps the fact); `TestQueuedRoundRunsOnlyLanePrecheckAndReady` (non-participant boxes in the region do nothing; `offlineReassert` and the keep view likewise); `TestParticipantsReReportOnANewPrecheckStamp`; `TestValidatorMatrix` (every role/phase/payload combination through `ValidateOfflineReport` and `readOfflineEvidence`); `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID` in the private report, absent from `fleet.json`; `OverCap`, `Truncated`, `Known false` on a failed listing); `TestVersionedArmedForOffline`. |
| -- [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/moves -run 'Participant|Lane|TargetReady|Precheck|Validator|Report' -count=1`, `go test ./internal/host -run 'ParticipantRole|LaneReport|TargetReady|SourcePrecheck|ExtraDirs|StagerPublishes|SealerPublishes|QueuedRound|ReReport|StatusHosts|VersionedArmed' -count=1`, `go test ./internal/status -run 'Hosts|Lanes|BoxID' -count=1`, then the root-read census. |
| -- [ ] **Step 5: Commit** `boxes: one participant resolver, the lane, the source precheck, the target-ready facts, the staged inventory, the lifted-tombstone fact and the per-host status block`. |
| +- `moves.Fact{Known, OK bool; Detail string}` (tagged); `moves.PrefixFacts{Known bool; Objects map[string]PrefixObject}`; `type PrefixObject struct { ETag string \`json:"etag"\`; Size int64 \`json:"size"\`; VersionID string \`json:"version_id,omitempty"\`; LastModified time.Time \`json:"last_modified"\` }` (current versions only, delete markers excluded). |
| +- `moves.TargetReadyReport` as rev 3 with `PrefixObject`; `moves.StrayEntry{Path string; Kind string (dir, file, symlink, other); Bytes int64; Entries int; Known bool; Detail string}` replaces `ExtraDir`; `SourcePrecheck{FilesBytes, FilesEntries, ArchiveEligible Fact, Strays StraysFact{Known bool; Entries []StrayEntry; Detail string}, Build, StagePathArmed Fact, StraysRemoved []string, StraysLeft []StrayEntry}`: a non-directory stray (any `Kind != dir`) is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). |
| +- `status.HostFacts{CollectedAt, Known, Err, Incarnation uint64 \`json:"incarnation"\`, ReplicaBytes, ReplicaObjects, ReplicaCollectedAt, FilesBytes, FilesEntries, OverCap, Truncated, ExtraDirs}`; `measureFiles` as rev 3. |
| +- **Staged ownership** (the console, Task 7's prechecks and Task 8's barrier): the stager report must be `StagedKnown` with `StagedBucket/Region` equal to the record's and `StagedRequest == row.StageRequest`; the target's `MovesPrefix` must be `Known`; the two sets of keys must be equal; for every key the target's `ETag` equals the receipt's `DstETag` and, when both carry a version id, the version ids are equal; a mismatch → "the staging prefix for <host> does not match what this move staged" (the overwrite and extra-object cases refuse). |
| + |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestLegacyReportsStillValidate` (serialized old report bytes pass `ValidateOfflineReport` and are kept by `readOfflineEvidence` for a move without `Participants`); `TestContextualValidationForSelfServeMoves`; `TestEmptyStagedInventoryRoundTrips` (`StagedKnown` with an empty map survives serialisation); `TestReadReceiptsExportsTheLedger`; `TestStagerPublishesReceiptsNotAListing` (an overwrite between the copy and the report → the receipt's ETag, not the listed one; the console's comparison then refuses); `TestStrayFileRefusesBeforeStop`; `TestParticipantStillPresent` (same label, another instance id → replaced); `TestHostFactsCarryTheLifetimeAndAges`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the touched packages' targeted tests, then the root-read census. |
| +- [ ] **Step 5: Commit** `boxes: participant identity, contextual report validation, the lane, the source precheck with stray kinds, target-ready, staged receipts and the per-host status block`. |
| -### Task 7: The job, from requested to ready, and its recovery table |
| +### Task 7: The job, from requested to ready, and its reconciliation |
| **Files:** |
| - Create: `internal/console/selfserve_job.go`, `internal/console/selfserve_checks.go`, `internal/console/selfserve_apply.go` |
| -- Test: `internal/console/selfserve_job_test.go` |
| +- Test: `internal/e2e/selfserve/job_test.go` |
| **Interfaces:** |
| -- `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, mut func(*provision.Record) error, want func(provision.Record) bool) (*provision.Record, error)`: `t.valid(now)` else `errLeaseLost`; `provision.SetRecordMove` under the judged context with a 30 s bound (`offlineStartWrite`); on an error other than `ErrMoveConflict`, re-read the record (judged) and, when `want(*rec)`, treat the write as landed; after a landed write `bumpSerial` and `SetSelfServeNotified(id, state)`; a failed bump leaves `notified_state` behind, and the next tick bumps again when `notified_state != the record's state`. `ErrMoveConflict` is returned for the reconciler. |
| -- `func (s *Server) selfServeStep(ctx context.Context, t *leaseToken, row SelfServeMove, now time.Time) error`: (1) deadlines from the row: `StepDeadline` or `WindowDeadline` passed → the phase's expiry action (abort, page, or blocked) **before** any evidence is read; (2) the record (judged) and the reports; (3) **reconcile** the row to the record by the table below; (4) `AbortRequestedAt != 0` and the record's state at or before `offline-landed` (or no record move) → the abort step (Task 8); (5) the phase function. |
| - |
| -**Reconciliation and recovery table** (the record state is the truth; the row's `request_issued` says a write was issued whose answer was lost): |
| - |
| -| row phase | record | action | |
| -|---|---|---| |
| -| `requested`, `reserving` | no move | continue the phase | |
| -| `reserving` with `request_issued` set (`request_kind = queue`) | no move | re-read once more after the 30 s bound has passed since `request_issued`; still none → the write did not land: clear `request_issued`, retry the queue write; a move with `ID == row.ID` → confirm (`queued`) | |
| -| `reserving`, `queued` | `offline-queued` with this id | `queued` → `reports` | |
| -| `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier` | `offline-staging` with this id | `staging` unless the row is already past it; a row in `restage` or `barrier` stays | |
| -| `staging` (any sub-phase) with `request_issued` (`restage` or `precheck`) | the record's `StagedAt`/`PrecheckAt` equals the issued value | `ConfirmSelfServeRequest`; otherwise, after the bound, clear and reissue | |
| -| any pre-cut phase | `offline-stopping` | `stopping` (an operator started it) | |
| -| any | `offline-sealing`/`sealed` | `sealing`; `landing`/`landed` → `landing`/`landed`; `offline-open` → `opened`; `offline-finishing` → `finishing` (gate `lifted`); `offline-reverting` → `reverting` | |
| -| any | no move and `LastMove.ID == row.ID` | the finaliser with `LastMove.Outcome` | |
| -| any | another move (an operator's, or a later self-serve) | finalise `aborted` "another move is in flight" without touching the record | |
| -| any | the record absent or stale (judged) | finalise `aborted` "the instance was recreated" without touching the record | |
| - |
| -- Phases: |
| - - `requested` → `reserving`. |
| - - `reserving`: the participants' boxes must still publish their ids (else abort "box <label> was replaced"); `reserveBoxes(participants, {move, row.ID, ...})`; busy → `reason` "waiting for another move on this box"; `WindowDeadline` → abort "the move was not started in time"; reserved → the fresh judged record → `recordEligibility`, the run and move checks, `offlineStartValidation(..., stage: true, requireIDs: true, "")` again (a refusal → abort with its sentence, release) → `IssueSelfServeRequest(id, queue, now)` → `selfServeApply(OfflineQueue(...), want: Move != nil && Move.ID == row.ID)` → confirm → `queued`. |
| - - `queued` → `reports` (the record carries the queued move; 5 min → abort). |
| - - `reports`: the `lane` report (`BoxID == Source[0].BoxID`, `OK`), the source `precheck` and the target `ready` (`BoxID` the pinned ids, `Request == PrecheckAt`); `targetReadyDecision` and `sourcePrecheckDecision` (the console enumerates the required path set and fact set itself: a missing entry or `!Known` → "could not determine <fact> on <label>"; not OK → the fact's sentence as rev 2, plus "the target box has timers for <host>", "the target bucket holds a lock for <host> (<owner>)"; before staging both prefixes must be `Known` and empty); both pass → `IssueSelfServeRequest(id, staging, now)`, `selfServeApply(OfflineQueuedToStaging)`, confirm (`stage_request`), `StagingStartedAt = now`, `staging`; 5 min → abort. |
| - - `staging`: the stager report with `StageRequest == row.StageRequest`, phase `staged`, `StagedBucket/Region` equal to the record's: `ConsumeSelfServeRound(id, StageRequest)` (counted once); ready rule `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `IssueSelfServeRequest(id, restage)` → `selfServeApply(OfflineRestage)` → confirm; a `failed` report → abort; a round past `max(10 min, 3 × est.Copy)` → abort; `now - StagingStartedAt > 30 min` or `Rounds >= 10` → `prechecks` from the last completed successful round. |
| - - `prechecks`: `IssueSelfServeRequest(id, precheck)` → `selfServeApply(OfflinePrecheck)` → confirm; the source `precheck` and target `ready` answering it; the staged-ownership comparison: `Ready.MovesPrefix.Known`, the stager report's `Staged` present with `StagedBucket/Region` equal to the record's and `StageRequest == row.StageRequest`, and the two inventories **equal** (same key set, equal ETags and sizes) → else "the staging prefix for <host> does not match what was staged"; `InstancesPrefix` known and empty; both pass → `ready`, `WindowDeadline = now + 24 h`, the estimate refined and `selfServeApply(OfflineSetEstimate)` (a later disagreement resolves towards the row); 5 min → abort. |
| - - `ready`: every 10 min a restage (issue/confirm); `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| -- The pre-offline abort (requested, reserving, queued, staging): no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`, never offline); queued → `selfServeApply(OfflineAbortAt(now))`; staging → `OfflineAbort` → wait for the stager's `reverted` → `OfflineFinish`; then the finaliser. Failed admission: a reservation older than the grace period with no record move and no outstanding `request_issued` → abort "the move could not be started". |
| - |
| -- [ ] **Step 1: Write the failing tests** on the harness of Task 10 (real boxes, the serial bump): `TestEveryBoundaryRecovers` (for queue, staging, restage, precheck: a landed write whose response was lost; a crash before `ConfirmSelfServeRequest`; a crash before the bump; a second crash during recovery; SQLite reopened each time: one request per wait, one bump per landed write); `TestReconcileTableEveryRow`; `TestReservingWaitsThenAborts`; `TestReplacedParticipantAborts`; `TestEligibilityRechecksBeforeQueue` (a record made ineligible after admission → abort, no queue); `TestReportsDecideFromFacts` (every fact's refusal and unknown; a missing path entry; a report from a non-participant id ignored); `TestStagingReadyRule` (the rev 2 cases plus a counted round counted once across a crash); `TestPrechecksCompareInventories` (missing, extra, changed ETag, changed size, wrong round, wrong destination, unknown prefix; a valid match); `TestReadyRestagesAndExpires`; `TestDeadlinesEvaluateBeforeEvidence` (an unreadable report store never postpones an expiry); `TestAbortBeforeOfflineIsLossless`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileTable|Reserving|Replaced|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, with issued requests, a reconciliation table and deadlines evaluated before evidence`. |
| +- `selfServeApply(ctx, t, row, mut, want func(provision.Record) bool) (*provision.Record, uint64, error)`: the lease-bounded context and the write guard; on an unknown outcome re-read (judged) and `want`; after a landed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq` (at-least-once; a duplicate bump is harmless). |
| +- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` sets `expiry_pending` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets: the row's phase not in the record's set → the set's first phase (`AdvanceSelfServe` without the abort predicate for `opened`); (4) the expiry action by the record: `expiry_pending` with the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Request handling in every issuing phase (`staging` from `reports`, `restage`, `precheck`, the barrier): `IssueSelfServeRequest` → `selfServeApply` → the committed stamp read from the returned record → `ConfirmSelfServeRequest(committed)` → the wait is for a report with `Request`/`StageRequest == committed` → `ConsumeSelfServeRequest` when used. A crash between the write and the confirmation: the next tick's reconciliation sees `request_issued` set and unconfirmed, re-reads the record, confirms from its current stamp (the write's) and continues; a request is never re-issued while unconsumed. |
| +- The admission write: `request_issued` for kind `queue`; a response lost → re-read; after the 30 s bound with no move: `fenceRecord(ctx, acct, app, host)` (plan 1) **before** the row is voided and the reservation released, so the delayed PUT cannot land afterwards; a fence failure keeps the row pending. |
| +- Phases as rev 3 with the identity check `participantStillPresent` at every gate (the status report of each pinned label, fresh within 10 minutes) → "box <label> was replaced" → abort before open, page after. |
| -### Task 8: The job's cut, abort, finalisation, paging, and the operator return |
| +- [ ] **Step 1: Write the failing tests** on the harness: `TestEveryBoundaryRecovers` (every transition in Review Focus 1, with repeated crashes; stable request identities; deadlines preserved; at least one bump); `TestReconcileForwardNeverRegresses` (health and finishing survive ticks with the record at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed` (prechecks, restage, barrier, and the first staging request); `TestCommittedStampIsTheRequestIdentity` (an `OfflineRestage` that bumped by a nanosecond is confirmed with the committed value and the report matches); `TestAbandonedAdmissionIsFenced` (a queue PUT committing server-side after the client timed out: the fence runs first and the late PUT fails CAS; the reservation is released only after); `TestReplacedParticipantAtEveryGate` (after prechecks; during abort); rev 3's remaining tests. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileForward|RequestsAreNever|CommittedStamp|AbandonedAdmission|ReplacedParticipant|Reserving|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, reconciling forward, with request lifecycles, at-least-once notification and a fenced admission`. |
| + |
| +### Task 8: The cut, abort, finalisation, paging and the operator return |
| **Files:** |
| -- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (every operator door that clears a self-serve move's record (`abort` from queued, `finish`) calls `s.selfServeFinalise`; the stage door's body gains `return_of`) |
| +- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (every clearing operator door finalises a self-serve move; the open door writes the open record; a new door `POST .../move/offline/return {"region","return_of"}`) |
| - Create: `internal/console/selfserve_health.go`, `internal/console/selfserve_pages.go`, `internal/console/selfserve_finalise.go`, `internal/console/offlinereturn.go` |
| -- Test: `internal/console/selfserve_cut_test.go`, `internal/console/offlinereturn_test.go` |
| +- Test: `internal/e2e/selfserve/cut_test.go`, `internal/console/offlinereturn_test.go` |
| **Interfaces:** |
| -- Phases and deadlines (`est` the row's terms): |
| - - `restage`: issue/confirm `OfflineRestage`; the matching `staged` report; `max(5 min, 2 × est.Copy)`; expiry → abort. → `barrier`. |
| - - `barrier`: issue/confirm `OfflinePrecheck`; both participants' fresh reports passing the decisions and the inventory comparison; `max(5 min, 2 × est.Copy)`; a failed fact or expiry → abort. → `stopping`. |
| - - `stopping`: `offlineStartValidation(..., stage: false, requireIDs: true, "")` on the fresh record, then `selfServeApply(OfflineStartStaged, want: State == offline-stopping)`; then the seal check with participants → `selfServeApply(OfflineSeal)`; `max(5 min, 2 × est.Stop)`; expiry → abort. |
| - - `sealing`: the sealer's `sealed`; `max(10 min, 3 × est.Copy)`; expiry → abort. |
| - - `landing`: `offlineFlipCheck` → `OfflineFlip`; the pinned target's `landed` with every item OK; `max(10 min, 3 × est.Land)`; expiry → abort. |
| - - `landed`: `offlineOpenCheck` → `selfServeApply(OfflineOpenTransition, want: State == offline-open)`; a refusal or `max(3 min, 2 × est.Open)` **while the record is `offline-landed`** → abort; a write error with no confirmed commit → `open-uncertain`. |
| - - `open-uncertain`: re-read each tick: `offline-open` → `opened`; `offline-landed` → `landed` (deadline continues); 5 min unreadable → page, stay. |
| - - `opened`: the target's `opened`; `max(3 min, 2 × est.Open)`; expiry → blocked: page, retry. |
| - - `health`: a probe when `now - HealthLastProbeAt >= 30 s`; `RecordSelfServeHealth`; `HealthConsecutiveOK >= 2` → `finishing`; `max(3 min, 2 × est.Promote) + 30 s` → `SetSelfServeWarning(...)`, page, `finishing`. |
| - - `finishing`: gate 1 (`offlineFinishCheck` opened + every pinned source parked) → `OfflineFinishingTransition`, `FinishGate = lifted`; gate 2 (lifted + every pinned source gone) → `selfServeApply(OfflineFinish, want: Move == nil && LastMove != nil && LastMove.ID == row.ID)`; 10 min each → blocked: page, retry. |
| - - `reverting`: `OfflineAbort` once (or `OfflineAbortAt` from queued); `SourceReopenedAt` when (no sealer, or the sealer's `reverting`/`reverted` report with `TombstoneLifted`) **and** every pinned source's `reverted`; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes → `OfflineFinish`; 10 min → blocked: page, retry. |
| -- **The finaliser** `selfServeFinalise(ctx, t, row, outcome, reason, warning)`: (1) `FinishSelfServeMove` (outcome; `release_pending = 1`, `effects_done = 0`); (2) the terminal effects: the timings (Task 9, from the stamps persisted at each phase end), the terminal `self_serve_events` row, `bumpSerial`; `SetSelfServeEffectsDone`; (3) `releaseBoxes(move, row.ID)`; (4) `ClearSelfServeRelease`. A crash between any two leaves the row in `OpenSelfServeMoves` and the next tick resumes at the first unfinished step (effects are idempotent by `(id, phase)`). The operator doors that clear a self-serve move's record route through it. An outstanding admission write (`request_issued` set in `reserving`) is never declared failed before the 30 s bound has passed and a re-read shows no move. |
| -- **The operator return** (§7.5): `POST .../move/offline/stage {"region", "return_of": "<forward move id>"}` (and `.../offline` start): `offlineStartValidation(..., returnOf)` reads the forward move's evidence: `control/offline-moves/<host>/<fwd>/` reports (the sealer's `lifted` or `reverted`, the target's `opened`), and requires the target region of the return to be the forward's `From`; `targetReadyDecision` with `Leftovers` accepts exactly: `Lock` present with owner `moves.TombstoneOwner(fwd)` and lifted (`lease.IsLiftedTombstone`); `InstancesPrefix.Objects` whose every `LastModified` is earlier than the forward's `OpenedAt` (the report gains `LastModified` per object through `ListVersions`); `DataPaths` entries for `/data/.instances/<host>` and `/data/instances/<host>` present and **empty** (a `Fact` with `Detail: "empty"`); `/data/.offline-moves/<host>/<fwd>/source/` and `/sealer/` present; `LiveLink` and `ProgramsFile` naming the channel's build (already accepted); anything else present → the ordinary refusal. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\` }`. The box's `offlineTargetReady` fills `Leftovers` when the record's `ReturnOf` is set (the stage door writes `ReturnOf` on the record through `OfflineStage`'s new parameter, additive). |
| -- The probe (`probeTarget`): as rev 2; plan 3 adds the header contract. |
| -- Paging: `alerting.Page` through `MovePageAttempt`/`SetMovePageAttempt` with keys `selfserve:<id>:<condition>` every 10 minutes for: `opened` past its deadline; a health failure; `finishing`/`reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; a reservation of an unknown or terminal-released move older than the grace period that the reclaim rule cannot reclaim (the per-tick scan, which also runs at start when there are no open rows); a lane probe failing on a configured pair. |
| -- Copy rows: as rev 2. |
| - |
| -- [ ] **Step 1: Write the failing tests** on the harness: `TestCutRunsEveryPhaseToDone` (Tito's sequence; the record cleared; the row terminal with `effects_done = 1`, `release_pending = 0`; one bump per landed transition); `TestEveryDeadlineBeforeCommittedOpenAborts` (restage, barrier, stopping, sealing, landing, the open check refused and expired while landed); `TestOpenUncertainIsReconciledByTheRecord`; `TestAbortIsInitiatedNotCompleted` (the real sealer: a source `reverted` before the sealer's lift sets nothing; the sealer's `reverting` report with `TombstoneLifted` plus every source's `reverted` sets `source_reopened_at`; a cleanup failure after the lift still shows reopened; then `finished_at`); `TestOpenedBlockedPagesAndRetries` (durable attempts across a restart); `TestHealthFailureFinishesWithAWarningAndPages`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep` (crashes between (1)/(2), (2)/(3), (3)/(4); the row stays open until (4); effects once); `TestOperatorDoorsFinaliseASelfServeMove` (queued abort and finish through the operator doors); `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRenders`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap` (only an old orphan reservation, admission off, no rows); `TestOperatorReturnAcceptsExactlyTheForwardLeftovers` (a completed forward; the return's stage door accepts the lifted lock, the old objects, the park dirs, the empty nested dirs, the matching link and programs file; a newer object, a live lock, a foreign park dir, a non-empty nested dir each refuse; a `return_of` naming a move that did not open refuses). |
| -- [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self-serve cut from restage to finish, the no-return boundary at the committed open, the abort facts, one finaliser, durable paging and the operator return's leftover acceptance`. |
| +- Phases and deadlines as rev 3, with: `landed`'s expiry decided by the record (Global Constraints); the open record written after the committed open; `health`: `health_attempts` incremented on every probe, `health_consecutive_ok` reset on failure; the finaliser as rev 3 (outcome, effects, release, clear). |
| +- **The operator return handshake**: (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): validates the forward: `OpenRecordKey(host, return_of)` exists with `To == rec.Region` and `From == region` (the return goes back to the forward's source), the record carries no move; writes `control/offline-moves/<host>/<return_of>/return/<console id>.json` `{"requested_at", "region"}` create-only and answers 202 "return requested; the target box will report its leftovers"; (2) the target box of the return (the box in `region`, region-based, operator path) sees the request on its moves pass and writes a `ready` report for role `target`, phase `ready`, `Request = requested_at`, with `Leftovers` filled against the open record's `OpenedAt`, without touching the record; (3) the operator's `stage` (or `start`) door with `return_of` requires that report (fresh, `Request` equal) and runs `targetReadyDecision` in return mode: accepted exactly: `Lock` present, owner `TombstoneOwner(return_of)`, lifted; `InstancesPrefix.Objects` current versions excluding `lock.json` all with `LastModified < OpenedAt`; `DataPaths["/data/.instances/<host>"]` and `["/data/instances/<host>"]` present and empty (`Detail: "empty"`), the mount view likewise; park dirs `<return_of>/source/` and `/sealer/` present; `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\` }`. |
| +- Paging as rev 3 with persisted attempts; the orphan scan each tick and at start. |
| -### Task 9: The estimate, timings, the events table and the status view |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestLandedExpiryDefersToTheRecord` (restart after a committed open and an old `landed` deadline → `opened`, no abort); `TestHealthAttemptsPersist`; `TestOperatorReturnHandshake` (the return door, the target's leftovers report, the stage door accepting exactly the forward's leftovers; an object written after the committed open but before the report refreshed → refused; a direct start with `return_of` → the same acceptance; a `return_of` whose open record is absent → 409). |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the targeted tests, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve cut with the no-return boundary at the committed open, the abort facts, one finaliser, durable paging and the operator return handshake`. |
| + |
| +### Task 9: The estimate, timings and the events table |
| **Files:** |
| - Create: `internal/console/selfserve_estimate.go`, `internal/console/selfserve_events.go` |
| -- Modify: `internal/console/store.go` (`move_timings (id TEXT, phase TEXT, from_region TEXT, to_region TEXT, bytes INTEGER, seconds REAL, at INTEGER, PRIMARY KEY (id, phase))`; `self_serve_timings (id TEXT, mark TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))` the per-phase stamps persisted **when consumed**: `offline_at`, `stopped_at` (every source `stopped` seen), `seal_at`, `sealed_at` (+ `FinalCopiedBytes`), `flip_at`, `landed_at` (+ files and replica bytes), `open_at`, `opened_at`, `first_ok_at`), `internal/console/selfserve_finalise.go` (the five `move_timings` rows from the stamps: `stop` = `offline_at → stopped_at`; `copy` = `seal_at → sealed_at`, bytes `FinalCopiedBytes`; `land` = `flip_at → landed_at`; `open` = `open_at → opened_at`; `promote` = `open_at → first_ok_at`, bytes replica), `internal/console/apiinstancemove_offline.go` (`offlineView.SelfServe *selfServeMoveView \`json:"self_serve,omitempty"\``; `GET .../move/offline?log=1` adds `Events []selfServeEventView`), the member stream (a `StartSelfServeEventsPass`-free design: the pass's tick drains `UnemittedSelfServeEvents` through `auditActivity(acct, app, "self-serve move", Activity{Action: "instance-move", TargetType: "instance", Target: host, Message: phase, Detail: reason})` and marks each emitted; an insert that finds `(id, phase)` present is a no-op, so a retried tick never duplicates) |
| +- Modify: `internal/console/store.go` (`move_timings`; `self_serve_timings` as Task 4), `internal/console/selfserve_finalise.go`, `internal/console/apiinstancemove_offline.go` (`offlineView.SelfServe`, `?log=1`) |
| - Test: `internal/console/selfserve_estimate_test.go`, `internal/console/selfserve_events_test.go` |
| **Interfaces:** |
| -- `type estimateTerms struct { Stop, Copy, Land, Open, Promote time.Duration }`; `type term struct { Fixed time.Duration; BytesPerSecond float64 }`; the defaults of rev 2. |
| -- `func (s *Server) estimate(from, to string, remaining, files, replica int64) (terms estimateTerms, seconds int)`: as rev 2 (rate terms fitted from `bytes/seconds` medians of rows with `bytes > 0`; fixed terms from `seconds` medians; 5 rows minimum; whole minutes; minimum 2). |
| -- `type SelfServeEvent struct { ID, Phase string; At time.Time; Reason, Detail string; Emitted bool }`; `type selfServeEventView struct { Phase string \`json:"phase"\`; At int64 \`json:"at"\`; Reason string \`json:"reason,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`. Every phase change, refusal, fact decision, lane probe, health probe and timing inserts an event (`Phase` carries a suffix for non-phase events: `reports:lane`, `health:probe:<n>`). |
| -- On `*Store`: `InsertMoveTiming`, `MoveTimings`, `MarkSelfServeTiming(id, mark string, at time.Time, bytes int64) error`, `SelfServeTimings(id string) (map[string]timingMark, error)`. |
| +- `estimateTerms`, `term`, the defaults (`Stop {20 s, 0}`, `Copy {10 s, 25 MiB/s}`, `Land {30 s, 40 MiB/s}`, `Open {30 s, 0}`, `Promote {10 s, 70 MiB/s}`), `estimate(from, to string, remaining, files, replica int64) (estimateTerms, int)` (rate terms from medians of `bytes/seconds` over rows with `bytes > 0`, fixed terms from medians of `seconds`, 5 rows minimum, whole minutes, minimum 2); the timing marks are **console observation times**, persisted on first observation (`MarkSelfServeTiming` is insert-only), never overwritten; a move that aborted or lacks a mark contributes no row for that phase. |
| +- `self_serve_events` with the account/app/host columns; `drainSelfServeEvents` each tick through `DeliverSelfServeEvent`; the pass starts while undelivered events exist (Task 4's mode). |
| -- [ ] **Step 1: Write the failing tests**: `TestEstimateUsesDefaultsThenMedians`; `TestEstimateMinimumIsTwoMinutes`; `TestEstimateBeforeAndAfterStaging`; `TestTimingMarksPersistWhenConsumed` (a crash after the stamp is consumed and before the phase advances keeps the mark; the finaliser's five rows from the marks; `copy` bytes are the sealer's `FinalCopiedBytes`); `TestEventsOncePerPhaseAcrossRetries`; `TestEventsDrainToTheMemberStream`; `TestOfflineStatusShowsSelfServeAndLog`. |
| -- [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Estimate|Timing|Events|OfflineStatusShows' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the move estimate from measured terms and rolling medians; timings persisted when consumed; the events table as the claims-off move log and member stream`. |
| +- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestTimingMarksAreFirstObservationTimes`; `TestAbortedMovesContributeNoRates`; `TestUndeliveredEventsStartThePass`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Estimate|Timing|Events|OfflineStatusShows' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the move estimate from first-observation timings; transactional events as the claims-off move log and member stream`. |
| -### Task 10: The shared harness and the end-to-end witnesses |
| +### Task 10: The exported harness and the end-to-end witnesses |
| **Files:** |
| -- Create: `internal/host/hosttest/harness.go` (an exported wrapper over `TestOfflineMoveE2E`'s two-box fixtures: `type Boxes struct{...}`, `func New(t testing.TB, store release.Store) *Boxes`, `func (b *Boxes) Pass(ctx) ` (one moves pass on every box), `func (b *Boxes) Serial(ctx)` (run passes until the serial is consumed), with the box ids, labels, registries and data buckets exposed; a clock hook), `internal/e2e/selfserve/selfserve_test.go` (an external test package importing `internal/console` and `hosttest`) |
| -- Test: the witnesses of Tasks 7 and 8 live here; plus: `TestSelfServeForwardMove`; `TestSelfServeCancelInReady`; `TestSelfServeAbortByFailedPrecheck`; `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal`; `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord`; `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchangedOnTheHarness`; `TestSelfServeThenOperatorReturn`. |
| +- Create: `internal/host/hosttest/harness.go` (a real package: the two-box fixtures **moved** out of `movesoffline_e2e_test.go` into exported constructors (`New`, `Boxes.Pass`, `Boxes.Serial`, `Boxes.Clock`), with `TestOfflineMoveE2E` rewritten to use it, behind no build tag so `go vet` covers it on every platform; the fixtures that need Linux stay behind `//go:build linux` with a stub that skips), `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}` exported for the e2e package), `internal/e2e/selfserve/*_test.go` (built with `-tags e2e`; CI runs it) |
| +- Test: the witnesses of Tasks 7 and 8 and rev 3's end-to-end list, plus `TestSelfServeThenOperatorReturn`. |
| -- [ ] **Step 1: Write the harness and the tests.** |
| -- [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| -- [ ] **Step 3: Run both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-2.md`. |
| -- [ ] **Step 4: Commit** `self-serve moves: a shared two-box harness and the end-to-end witnesses`; push. |
| +- [ ] **Step 1: Extract the harness and write the tests.** **Step 2: Run them; fix what they find.** **Step 3: Both census tests and the cross-builds**; the census note. **Step 4: Commit** `self-serve moves: an exported two-box harness and the end-to-end witnesses`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage: §5 (T1), §6.1 (T5), §6.2 (T7), §6.3 (T9, T5, T6), §7 admission and recovery (T7's table), §7.1 (T3), §7.2 and the three facts with the sealer's durable lift (T6, T8), §7.3 (T6, T7, T8's barrier), §7.4 without the probe mode (T8), §7.5 the no-return boundary and the operator return (T8), §7.7 (T5), §8 pages, log and member stream (T8, T9), §9 (T3's rollout). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup (plan 4), whose status contract this plan's reclaim rule names exactly (`RetirementsComplete`, a terminal state, `RootRequest == ""`, `ReleasePending == false`, `Metrics.BoxID`). |
| -- Dependencies stated for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}` on the record, `Participants.Target.BoxID` for the probe's identity, `moveCfg.BoxID`; for plan 4: the status `Retirements` contract above. |
| -- Names used across tasks: `selfServeGate`, `moveCapable`, `Principal.TokenID`, `recordEligibility`, `offlineStartValidation`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant(s)`, `moves.ParticipantRole`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeAdmits`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `selfServeFinalise`, `leaseToken`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `ConsumeSelfServeRound`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck`, `ExtraDirsFact`, `ReturnLeftovers`, `status.HostFacts`, `BoxMetrics.Lanes/BoxID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MoveTiming`, `SelfServeEvent`, `hosttest`. |
| +- Spec coverage as rev 3, with §7.5's operator return now a handshake (T2's open record, T8's return door and report). |
| +- Dependencies for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}`, `Participants.Target.BoxID`, `moveCfg.BoxID`; for plan 4: the status `Retirements` contract (`RetirementsComplete`, `State == ended`, `Metrics.BoxID`) and the open record. |
| +- Names used across tasks: rev 3's plus `selfServeMode`, `offlineRecordChecks`, `offlineRegionChecks`, `Participant.InstanceID/BootID/Verified`, `participantStillPresent`, `moves.OpenRecord`, `OpenRecordKey`, `lease.ReadReceipts`, `CopyReceipt`, `PrefixObject`, `StrayEntry`, `StraysFact`, `ValidateOfflineReportFor`, `IssueSelfServeRequest/ConfirmSelfServeRequest/ConsumeSelfServeRequest`, `notified_seq`, `expiry_pending`, `bumpSerialErr`, `provision.WithWriteGuard`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `console.TestHooks`, `hosttest.New`. |
| - Review Focus 1 to 5 are pinned by T7, T3, T8, T5, T1. |