amadan
plan: self-serve moves 2, rev 5 (Astra plan review 4: items 1, 4, 5, 6, 8, 10, 12, 13, 14, 17, 20, 21; self-contained; persisted requests with nonce, expected seq and proposed stamp; a move-bound stage book from observed copies; identity by box and instance; a recoverable return handshake; the bridge inside package host)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+131
−92
docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md+131 −92
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (5 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (152 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 4.049s
ok github.com/carlosframework/platform/cmd/moveprobe 11.575s
ok github.com/carlosframework/platform/cmd/zddprobe 0.704s
ok github.com/carlosframework/platform/internal/activator 60.832s
ok github.com/carlosframework/platform/internal/agentsock 0.016s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.020s
ok github.com/carlosframework/platform/internal/appconfig 2.932s
ok github.com/carlosframework/platform/internal/apperr 0.078s
ok github.com/carlosframework/platform/internal/applog 0.127s
ok github.com/carlosframework/platform/internal/awsinv 0.010s
ok github.com/carlosframework/platform/internal/boxchoice 0.005s
ok github.com/carlosframework/platform/internal/boxid 0.835s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.013s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.717s
ok github.com/carlosframework/platform/internal/changelog 0.354s
ok github.com/carlosframework/platform/internal/claimactivation 0.903s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.863s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.010s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.006s
ok github.com/carlosframework/platform/internal/domains 2.883s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.006s
ok github.com/carlosframework/platform/internal/edge 45.281s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.139s
ok github.com/carlosframework/platform/internal/gate 1.715s
ok github.com/carlosframework/platform/internal/genselect 0.023s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.031s
ok github.com/carlosframework/platform/internal/historyreadback 0.155s
ok github.com/carlosframework/platform/internal/host 141.888s
ok github.com/carlosframework/platform/internal/hostactors 6.137s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.007s
ok github.com/carlosframework/platform/internal/hostbox 3.390s
ok github.com/carlosframework/platform/internal/hostclaim 0.929s
ok github.com/carlosframework/platform/internal/hostenv 0.006s
ok github.com/carlosframework/platform/internal/hostevidence 0.022s
ok github.com/carlosframework/platform/internal/hostexec 5.680s
ok github.com/carlosframework/platform/internal/hostfence 0.088s
ok github.com/carlosframework/platform/internal/hostfiles 10.312s
ok github.com/carlosframework/platform/internal/hostfloor 0.021s
ok github.com/carlosframework/platform/internal/hostguard 1.045s
ok github.com/carlosframework/platform/internal/hostops 2.246s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.009s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.734s
ok github.com/carlosframework/platform/internal/hostresolve 0.250s
ok github.com/carlosframework/platform/internal/hostwrite 0.008s
ok github.com/carlosframework/platform/internal/infraview 0.005s
ok github.com/carlosframework/platform/internal/instancectl 5.225s
ok github.com/carlosframework/platform/internal/instanceexec 2.835s
ok github.com/carlosframework/platform/internal/instancefs 0.011s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.011s
ok github.com/carlosframework/platform/internal/layout 1.945s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 10.193s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.757s
ok github.com/carlosframework/platform/internal/ledger 0.021s
ok github.com/carlosframework/platform/internal/mail 0.310s
ok github.com/carlosframework/platform/internal/maintenance 0.085s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.547s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.103s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 21.600s
ok github.com/carlosframework/platform/internal/moves/archive 1.011s
ok github.com/carlosframework/platform/internal/objstore 0.361s
ok github.com/carlosframework/platform/internal/ops 0.227s
ok github.com/carlosframework/platform/internal/park 0.325s
ok github.com/carlosframework/platform/internal/plans 0.007s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 14.516s
ok github.com/carlosframework/platform/internal/publicrelease 0.010s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.720s
ok github.com/carlosframework/platform/internal/relay 7.248s
ok github.com/carlosframework/platform/internal/release 25.164s
ok github.com/carlosframework/platform/internal/rollout 3.938s
ok github.com/carlosframework/platform/internal/rolloutwords 0.011s
ok github.com/carlosframework/platform/internal/rootaws 0.010s
ok github.com/carlosframework/platform/internal/rootparticipation 0.075s
ok github.com/carlosframework/platform/internal/rootstate 0.029s
ok github.com/carlosframework/platform/internal/runs 0.550s
ok github.com/carlosframework/platform/internal/s3compat 0.012s
ok github.com/carlosframework/platform/internal/safefile 0.007s
ok github.com/carlosframework/platform/internal/schedule 2.716s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.150s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.230s
ok github.com/carlosframework/platform/internal/status 0.071s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.218s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.005s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.005s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.622s
=== internal_console@0 (83 s)
ok github.com/carlosframework/platform/internal/console 81.034s
=== internal_console@1 (37 s)
ok github.com/carlosframework/platform/internal/console 35.262s
=== internal_console@2 (42 s)
ok github.com/carlosframework/platform/internal/console 39.922s
=== internal_console@3 (37 s)
ok github.com/carlosframework/platform/internal/console 35.011s
=== internal_console@4 (37 s)
ok github.com/carlosframework/platform/internal/console 35.370s
=== internal_console@5 (35 s)
ok github.com/carlosframework/platform/internal/console 33.522s
=== internal_console@6 (36 s)
ok github.com/carlosframework/platform/internal/console 34.140s
=== internal_console@7 (44 s)
ok github.com/carlosframework/platform/internal/console 41.604s
=== root@0 (33 s)
ok github.com/carlosframework/platform 30.884s
=== root@1 (45 s)
ok github.com/carlosframework/platform 42.510s
=== root@2 (42 s)
ok github.com/carlosframework/platform 40.044s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.024s
=== RUN TestT4StorageProbeMinio
2026/10/22 09:31:42 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0D0235FB30B66, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md| index ca9e5f4d2..ffda239f9 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| @@ -2,108 +2,117 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 4, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items, five closed). Rev 4 answers rev 3's open items: the doors dispatch on a **process-level mode**, decided without host evidence, so the capability check always precedes every read; the self-serve gate keeps the instance gate's rate limiter and no-bucket rail, and the start validation is split around the body decode; legacy reports keep their structural validation and the new evidence is validated contextually against the move, with an explicit empty-inventory encoding; the staged inventory comes from the stager's **copy receipts**, not from a listing, and the console compares the target's listing against those receipts in both directions; stray entries carry their kind and non-directory strays refuse before the stop; participants carry the box's instance and boot ids from its status report and are verified against the signed box record when one exists, so a replacement is detected at every gate; the journal keeps each request's issued, confirmed-from-the-record and consumed identities, SQL phases map to record states as compatible sets that never regress, the lease bounds each write attempt, and notification is at-least-once keyed on the record's `record_seq`; an expired `landed` deadline defers to the record before any abort; an abandoned admission write is fenced with plan 1's `fenceRecord` before the reservation is released; a deployment-wide reservation sentinel covers id-less selections and exact retries are accepted before the exclusive gate; startup read failures fail closed; the view gives an unrelated active record move precedence over an open row and joins the self-serve row by id and lifetime; host facts carry the lifetime and both collection ages; every new table is classified; events and their audit delivery are one transaction; the harness is an exported package; the journal's continuity is stated; and the operator return has a request-and-report handshake with a durable open record. |
| +Rev 5, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items), rev 4 (`selfserve-plan2-review4.md`, 21 items, 9 closed). Rev 5 is **self-contained** (nothing is "as rev N"; every schema, phase, predicate, sentence and test list is in this document) and answers rev 4's open items: the gate orders authentication, host shape and the capability check before the limiter and the no-bucket rail (item 1); operator moves keep today's admission (nil or stale metrics, no identity record) and only the self-serve path requires fresh identity; `BootID` is dropped (items 4, 8); the evidence reader takes the move and a freshness table is stated (item 5); the staged inventory is a **move-bound book** built from every copy event the mirror reports, including same-ETag and deferred copies, through a new observer hook that leaves ordinary mirroring unchanged (item 6); a participant is identified by `BoxID` **and** `InstanceID` in the resolver, in report validation and at every gate, and abort never requires the target (item 8); every request persists its **nonce, expected sequence and proposed stamp before the write**, confirmation is conditional on the record carrying that exact stamp, replay is idempotent, and consumption and the phase advance are one UPDATE predicated on the nonce (item 10); compatible sets include the predecessor state while a phase's write is pending (item 10); expiry is bound to its phase and cleared on reconciliation into another phase, with the health rule taking precedence (item 12); abandonment is conditional on the fenced observation and persisted before release, and every reclaim fences first, operator reservations included (item 13); own retries are accepted only for a complete expected selection after the whole table validates (item 14); the status view joins rows regardless of the mode (item 17); events carry a per-row sequence and are delivered inside the caller's transaction, the harness is an exported bridge **inside package host** behind the `e2e` tag with the CI step named, the Go declarations are complete, and queue ordering is stated (item 20); the open record is written **before** the open transition with the proposed open time, return requests have unique durable identities, the report path is named, acceptance is bound to the lifetime and the forward's participants, and leftovers are judged by the forward's layout (item 21). |
| -Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 8): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord` exists. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| +Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord(ctx, acct, app, host, op fenceOwner) (fenceResult, Observation, error)` exists, the console runs behind the lifetimes cutover. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| **Goal:** With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, Tito's existing residency calls (prepare, status, cut, cancel) run the offline move as a console job: an owner-delegated move-capable token asks, the door refuses what cannot be offered before anything is written, per-box reservations serialise moves per box, the boxes prove the lane, the target's cleanliness and the source's eligibility before any copy and again before the stop, staging runs until caught up, the cut runs start, seal, flip, open, health, finish with a deadline on every wait, every failure before the committed `open` initiates a lossless abort, and the view tells Tito `preparing`, `ready`, `cutting`, `none` with `last` and the abort's three facts. The operator's return accepts exactly the forward move's leftovers. |
| **Architecture:** One `self_serve_moves` SQLite row per move drives a console pass (every 5 s, under a console lease) that advances each move by at most one step per tick from the evidence the operator doors already read, through the door bodies refactored into functions the doors and the job share. A new record state `offline-queued` lets the console reserve boxes and collect the lane, source-precheck and target-ready reports before a byte is copied; a precheck stamp makes both participants report again before the cut. `active.json` is a per-box reservation table with a legacy projection and a deployment-wide sentinel. The move-capable token is a `moves` column on `cli_tokens`, owner-minted, checked at the three self-serve mutations before any evidence is read. |
| -**Tech Stack:** Go. `internal/console`, `internal/provision`, `internal/moves`, `internal/host`, `internal/moves/archive`, `internal/alerting`, `internal/status`, `internal/lease`, `internal/boxid`. Tests: the console in-process with `release.NewDirStore`; integrated witnesses in `internal/e2e/selfserve` over the exported harness `internal/host/hosttest` and the console's exported test hooks. |
| +**Tech Stack:** Go. `internal/console`, `internal/provision`, `internal/moves`, `internal/host`, `internal/moves/archive`, `internal/alerting`, `internal/status`, `internal/lease`, `internal/boxid`. Tests: the console in-process with `release.NewDirStore`; integrated witnesses in `internal/e2e/selfserve` (build tag `e2e`) over the exported bridge `host.E2E` and the console's exported test hooks. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §7.5's operator return, §8, §11. Stated deviations: the record's `self_serve` is an object; the pre-request estimate input is the replica size alone; the claims-off "move log" is the console's `self_serve_events` table; **in self-serve mode the legacy move mutations are not served** (a plain instance token gets the family's 404 at prepare, cut and cancel; the status read stays open), because the dispatch must be decided before any evidence is read. |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §4, §5, §6.1 to §6.3, §7 (except §7.4's probe mode and §7.6), §7.5's operator return, §8, §11. Stated deviations: the record's `self_serve` is an object; the pre-request estimate input is the replica size alone; the claims-off "move log" is the console's `self_serve_events` table; **in self-serve mode the legacy move mutations are not served** (a plain instance token gets the family's 404 at prepare, cut and cancel; the status read stays open), because the dispatch must be decided before any evidence is read (Paul's ruling owed, amadan #343). |
| ## Global Constraints |
| -- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. Rendering follows the rows (Task 5). |
| -- The operator doors keep their answers byte for byte (snapshots of every refusal, including a malformed body combined with each class refusal). A job tick never calls an HTTP door. |
| -- Every wait has a deadline evaluated from the row before any evidence is read; **the action an expiry takes is decided by the record** (Task 8): an expiry while the record is `offline-landed` aborts, while it is committed open pages, while it is unreadable waits as `open-uncertain`. |
| -- Every record transition the job writes goes through `selfServeApply` (Task 7): the write is bounded to the lease's remaining validity; a lost response is reconciled by re-reading the record; after every landed write the serial is bumped **at least once**, tracked by the record's `record_seq` in `notified_seq`. |
| +- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. **Rendering never depends on the mode**: the status door joins rows whenever the console is claims-off and Phase 1 is off (Task 5). |
| +- The operator doors keep their answers byte for byte (snapshots of every refusal, including a malformed body combined with each class refusal, replayed from main). A job tick never calls an HTTP door. |
| +- Every wait has a deadline evaluated from the row before any evidence is read; an expiry is **bound to the phase that set it** (`expiry_phase`) and the action is decided by the record (Task 8): while the record is `offline-landed` or earlier → abort; committed open → page; unreadable → `open-uncertain`; the health phase's expiry → warning and `finishing` (precedence over the page rule). |
| +- Every record transition the job writes goes through `selfServeApply` (Task 7) with a **persisted request**: nonce, expected `record_seq` and the exact proposed stamp are written to the row before the record write; the mutation carries the proposed stamp verbatim; confirmation is the record carrying it; replay is the same write again; after every landed write the serial is bumped at least once (`notified_seq`). |
| - One console runs the pass under `control/offline-moves/console.json` (a 60 s lease, renewed every 20 s during a tick, checked before every write attempt with a fresh clock, and bounding every store write's context). The SQLite journal is the console's; a replacement console takes over by opening the **same** database file (one console host per deployment; the lease guards the day there are two) and recovers from the rows and the records. |
| - Refusals before writes; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| -- Participants are pinned at admission as `{BoxID, Label, InstanceID, BootID, Verified}` from the edges file and the box's status report; **one resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate; **one identity check** (`participantStillPresent`, Task 6) is applied at every gate: the label's current status report must carry the pinned `InstanceID` (and, when the box record `control/boxes/<label>.json` exists, its signed identity must name that instance), else "box <label> was replaced". A move without `Participants` keeps today's region rule everywhere. |
| -- Timestamps in SQLite are Unix nanoseconds; request identities are the **committed** record stamps read back after the write. |
| -- Copy rows pending Paul's review (Tasks 5 and 8). No tenant names. |
| -- Census: as rev 3 (`plan14-root-reads-self-serve-moves-2.tsv` loaded last with `@fields` and reviewed envelopes; `plan14SelfServeMoves2WriteCensusInventory()` with every added or changed writer including the console's `open.json`, events and timings; both green). |
| -- Local tests: touched packages with `-run`; CI runs the full suite. |
| +- **Participants** are pinned at admission as `{BoxID, Label, InstanceID string; Verified bool}` (`BootID` is not pinned: it has no claims-off producer and a reboot does not change the machine's identity). **One resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate from `BoxID` **and** `InstanceID`; **one identity check** (`moves.ParticipantStillPresent`, Task 6) at every gate. A participant pinned with an empty `InstanceID` (the operator path on a box without a fresh report) keeps today's region rule for that participant and forces the sentinel reservation. A move without `Participants` keeps today's region rule everywhere. |
| +- Timestamps in SQLite are Unix nanoseconds; request identities are **proposed** stamps persisted before the write and confirmed by the record. |
| +- Copy rows pending Paul's review (every sentence in quotes in Tasks 5, 7 and 8). No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv` loaded by a new `loadRootReadInventory(..., true)` block in `internal/provision/plan14_root_reads_test.go` after every existing overlay and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-2`, `@fields` for the new record fields (`self_serve`, `participants`, `queued_at`, `precheck_at`, `return_of`), reviewed dispositions for the changed envelopes (`internal/host`, `internal/status`, `internal/moves`, `internal/console`, `internal/lease`); `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's with every added or changed writer: the new transitions, the box's report writers (lane, precheck, ready, staged, reverting, return-ready), the console's `open.json`, the return request writer, the console lease, `active.json`'s table form, the write guard; both green. |
| +- Local tests: touched packages with `-run`; CI runs the full suite **and** `go test -tags e2e ./internal/e2e/...` (Task 10 adds the step to `.amadan/test-gate.sh`). |
| ## Review Focus |
| -1. A crash after a landed write at every transition boundary (queue, staging, restage, precheck, start, seal, flip, open, both finish gates, finish, abort), before the SQL update and before the bump, with a second crash during each recovery: the row reconciles forward without regressing, requests keep their committed identities and deadlines, and the serial is bumped at least once (Task 7, `TestEveryBoundaryRecovers`). |
| -2. A reservation older than the grace period: reclaimable only on positive evidence; a SQL or store read failure, a malformed body, an incomplete or non-terminal cleanup status, an outstanding admission write (even one the server commits after the client timed out), an id-less sentinel reservation, or switch-off with two disjoint existing reservations never reclaim or obstruct recovery (Task 3, `TestReclaimOnlyOnEvidence`). |
| -3. A deadline expiring mid-copy or mid-seal: the three abort facts in order from the real sealer's durable lift (Task 8, `TestAbortIsInitiatedNotCompleted`). |
| -4. Cut and cancel in both barrier orders and against a stale worker snapshot; cut repeated during cutting → 202 (Task 5, `TestMutationsNameTheMoveAndRace`). |
| -5. At the three mutations in self-serve mode: a plain instance token, an admin token, another app's move-capable token and a member's token get the family's 404 with zero store and zero SQL reads, against every record state and with admission on or off; outside self-serve mode the legacy doors answer as main (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| +1. A crash **before** and after each record write at every transition boundary (queue, staging, restage, precheck, start, seal, flip, open, both finish gates, finish, abort), before the SQL update and before the bump, with a second crash during each recovery, and a store that commits a write **after** the client's context was cancelled: the row reconciles forward without regressing, requests keep their persisted identities, no request is issued twice, no stale evidence satisfies a new request, and the serial is bumped at least once (Task 7, `TestEveryBoundaryRecovers`). |
| +2. A reservation older than the grace period: reclaimable only after a successful fence and on positive evidence; a delayed queue commit between a negative re-read and the fence, a delayed **operator** admission, a SQL or store read failure, a malformed body, an incomplete cleanup status, a partial own reservation, an id-less sentinel, or switch-off with two disjoint existing reservations and a cleanup-only obligation never reclaim wrongly or obstruct recovery (Task 3, `TestReclaimOnlyOnEvidence`). |
| +3. A deadline expiring mid-copy or mid-seal, and a target that disappears after landing: the three abort facts in order from the real sealer's durable lift; the source reopens without the target (Task 8, `TestAbortIsInitiatedNotCompleted`). |
| +4. Cut and cancel in both barrier orders and against a stale worker snapshot; cut repeated during cutting → 202; a stale request confirmation or consumption cannot bypass its nonce (Task 5, `TestMutationsNameTheMoveAndRace`). |
| +5. At the three mutations in self-serve mode: a plain instance token, an admin token, another app's move-capable token and a member's token get the family's 404 with zero store reads and no SQL beyond authenticating the token and resolving the account and app, against every record state, with admission on or off, **under an exhausted rate limiter and with no deployment bucket**; outside self-serve mode the legacy doors answer as main (Task 1, `TestMoveCapableGateAtTheRoutes`). |
| --- |
| ### Task 1: The move-capable token and the self-serve gate |
| **Files:** |
| -- Modify: `internal/console/authstore.go` (`moves` column; `claimRawNewAppToken(..., moves bool, ...)`; every `cli_tokens` SELECT feeding `scanCLITokenRows` selects it), `internal/console/svcprincipal.go` (`Principal.Moves bool`, `Principal.TokenID string` = the stored `token_hash`), `internal/console/apiinstances.go` (`handleAPIInstanceToken` with `moves`; owner-only; the audit sentence), the tokens page template |
| +- Modify: `internal/console/authstore.go` (`ALTER TABLE cli_tokens ADD COLUMN moves INTEGER NOT NULL DEFAULT 0`; `claimRawNewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; every `SELECT ... FROM cli_tokens` feeding `scanCLITokenRows` selects `moves`), `internal/console/svcprincipal.go` (`Principal.Moves bool`, `Principal.TokenID string` = the stored `token_hash`; `CLIPrincipal` sets both), `internal/console/apiinstances.go` (`handleAPIInstanceToken`: body `{"label","moves"}`; `moves` requires `RoleOwner` else 403 "only an account owner can mint a move-capable instance token"; audit "minted a move-capable instance token for <app> (cli), token <id>"), the tokens page template |
| - Create: `internal/console/selfserve_gate.go` |
| - Test: `internal/console/instancetoken_moves_test.go`, `internal/console/selfserve_gate_test.go` |
| **Interfaces:** |
| -- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`. |
| -- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`: `instanceGateSaying(w, r, "instance", mainMoveCopy().gate)` (the principal, the account and app, **the rate limiter and the no-deployment-bucket 503**, as every instance door), `principalFrom(r.Context())`, `host` with `ValidHostShape` (404), then `moveCapable(p, acct, app)` else `http.NotFound`; no store and no SQL read before it returns. |
| -- **The dispatcher**: `handleAPIInstanceMoveRequest`, `Cut`, `Abort`: `s.cfg.MovesEnabled` → Phase 1 as today; `s.selfServeMode()` → the self-serve body (Task 5); else the legacy body. `handleAPIInstanceMoveStatus`: `s.cfg.MovesEnabled` → Phase 1 as today; else `moveGate` (today's; an absent record is today's 404), then in self-serve mode the row join (Task 5's precedence; a SQL failure → 502 "could not read the move"), else today's view. |
| +- `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; existing callers pass `false`. |
| +- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`, **in this order**: (1) `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` (authentication and the account/app binding: today's answers for an invalid or revoked credential; its SQL is **binding SQL**, the only SQL this gate performs); (2) `host := r.PathValue("host")` with `ValidHostShape` (404 as `moveGate`); (3) `moveCapable(p, acct, app)` else `http.NotFound(w, r)` (`moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`); (4) the instance gate's **rate limiter** and (5) the **no-deployment-bucket 503**, exactly as `instanceGateSaying` applies them, factored into `instanceRails(w, r, acct)` so both gates share one implementation. An unauthorised caller therefore always gets the 404 even under an exhausted limiter or with no bucket; an authorised caller keeps those rails. No store read and no evidence SQL (no `self_serve_moves`, no records) before the gate returns. |
| +- **The dispatcher**: `handleAPIInstanceMoveRequest`, `Cut`, `Abort`: `s.cfg.MovesEnabled` → Phase 1 as today; `s.selfServeMode()` → the self-serve body (Task 5); else the legacy body. `handleAPIInstanceMoveStatus`: `s.cfg.MovesEnabled` → Phase 1 as today; else `moveGate` (today's; an absent record is today's 404), then, whenever `!cfg.HostClaim && !cfg.MovesEnabled` (**not** gated on `selfServeMode`), the row join (Task 5's precedence; a SQL failure → 502 "could not read the move"); a host with no row → today's view. |
| +- The CLI (`carlos instance-token --moves`) is a carlos-repo change. Existing tokens have `moves = 0`. #155's host-bound tokens: a comment names the future refusal. |
| -- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken`; `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (counting store and SQL: in self-serve mode, each unauthorised token against absent, eligible, ineligible and unreadable records, with admission on and off → 404 with zero reads; the owner's token passes; a revoked token → 401; the rate limiter and the no-bucket 503 answer as the instance gate does; outside self-serve mode a plain instance token reaches the legacy body exactly as main). |
| +- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (both listings, `Principal.Moves`, `Principal.TokenID` equals the listing id, the audit line); `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (a counting store and a counting SQL wrapper that classifies statements as binding (`cli_tokens`, `accounts`, `apps`) or evidence (everything else): in self-serve mode, for each of a plain instance token, an admin token, another app's move-capable token and a member's token, against absent, eligible, ineligible and unreadable records, with admission on and off, **with the limiter exhausted and with no deployment bucket** → the 404 with zero store reads and zero evidence SQL; the owner's move-capable token passes and then meets the limiter's 429 and the no-bucket 503 as the instance gate gives them; a revoked token → today's 401; outside self-serve mode a plain instance token reaches the legacy body exactly as main). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'MoveCapable|Mint' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the move-capable instance token and the self-serve gate, dispatched by a process-level mode`. |
| +- [ ] **Step 5: Commit** `console: the move-capable instance token and the self-serve gate: authenticate, bind, authorise, then the rails`. |
| ### Task 2: The queued state, the precheck stamp, the open record and the self-serve fields |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove`, `Participants *Participants`, `QueuedAt`, `PrecheckAt *time.Time`, `ReturnOf string`, all `omitempty`), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: queued → ""; `offlineViewOf`: `CanAbort` in queued, `NeverOffline` from `StartedOffline`; **the open door and the job write the open record** `control/offline-moves/<host>/<id>/open.json` create-only after a committed `OfflineOpenTransition`: `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; OpenedAt time.Time \`json:"opened_at"\` }` with `OpenedAt = Move.OpenedAt`; a failed write is retried on the next tick or the next operator door call (idempotent, create-only)), `internal/moves/offline.go` (`OpenRecordKey(host, id)`), the allowlist test |
| +- Modify: `internal/provision/offline_move.go`, `internal/provision/provision.go` (`MoveState` gains `SelfServe *SelfServeMove \`json:"self_serve,omitempty"\``, `Participants *Participants \`json:"participants,omitempty"\``, `QueuedAt *time.Time \`json:"queued_at,omitempty"\``, `PrecheckAt *time.Time \`json:"precheck_at,omitempty"\``, `ReturnOf string \`json:"return_of,omitempty"\``), `internal/console/apiinstancemove_offline.go` (`offlineVerbOf`: `offline-queued` → ""; `offlineViewOf`: `CanAbort` true in queued, `NeverOffline` from `StartedOffline`; the open door writes the open record **before** `OfflineOpenTransition`, Task 8), `internal/moves/offline.go` (`OpenRecord`, `OpenRecordKey(host, id)`), `internal/provision/offline_move_test.go` (the allowlist) |
| - Test: `internal/provision/offline_queue_test.go`, `internal/console/openrecord_test.go` |
| **Interfaces:** |
| -- `const OfflineQueued = "offline-queued"`; `SelfServeMove`, `Participant{BoxID, Label, InstanceID, BootID string; Verified bool}` (all tagged snake_case), `Participants{Source []Participant; Target Participant}`; `OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, now time.Time)` (as rev 3: refusals, `Incarnation` match, `StagedAt = PrecheckAt = &now`, `Seq = MoveSeq`); `OfflineQueuedToStaging(id, now)`; `OfflinePrecheck(id, now)`; `OfflineSetEstimate(id, seconds)`; `OfflineAbortAt(id, now)`; `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged` sets `SelfServe.OfflineAt`. |
| - |
| -- [ ] **Step 1: Write the failing tests**: as rev 3 plus `TestOpenRecordWrittenOnceAfterCommittedOpen` (the operator open door and the job both write it; a second write is a no-op; a failed write is retried); `TestReturnOfCarriedByStageAndStart`. |
| +- `const OfflineQueued = "offline-queued"`. |
| +- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\` }`. |
| +- `type Participant struct { BoxID string \`json:"box_id"\`; Label string \`json:"label"\`; InstanceID string \`json:"instance_id,omitempty"\`; Verified bool \`json:"verified,omitempty"\` }`; `type Participants struct { Source []Participant \`json:"source"\`; Target Participant \`json:"target"\` }`. |
| +- `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, at time.Time) func(*Record) error`: `OfflineStage`'s refusals (a move in flight, a seed block, an open run, the same region) plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager: p.Source[0].Label`, `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: at`, `QueuedAt: &at`, `StagedAt: &at` (the pre-copy intent exists, so `StartedOffline()` is false until the start), `PrecheckAt: &at`, `Seq: rec.MoveSeq`; `move_seq` unchanged. **`at` is the caller's proposed stamp** and is stored verbatim (no clock read inside the mutation); the same holds for every transition below that takes `at`. |
| +- `func OfflineQueuedToStaging(id string, at time.Time) func(*Record) error` (`StagedAt = &at`); `func OfflinePrecheck(id string, at time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &at`); `OfflineRestage(id, at)` stores `at` verbatim (today's signature; the console passes its proposed stamp); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, at time.Time) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: at, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour; a repeated abort on a reverting record stays a no-write success); `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged(id, at)` sets `SelfServe.OfflineAt = &at` when `SelfServe != nil`; `OfflineOpenTransition(id, at)` sets `OpenedAt = at`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| +- **The open record** `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; Participants *provision.Participants \`json:"participants,omitempty"\`; Layout string \`json:"layout"\` (`flat` or `nested`, from the source precheck's `Layout`); OpenedAt time.Time \`json:"opened_at"\` }` at `OpenRecordKey(host, id) = "control/offline-moves/<host>/<id>/open.json"`, written **create-only before** the open transition with `OpenedAt` = the proposed open stamp, by the operator open door and by the job alike; the open transition then carries the same `at`; a failed open after the record was written leaves an open record for a move whose record region never changed, which the return door rejects (`To != rec.Region`). Because the record exists before the open can commit, no clearing transition can outrun it. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`; the stamps are the caller's `at` verbatim); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove`; `TestOfflineSetEstimateRefusesWithoutSelfServe`; `TestReturnOfCarriedByStageAndStart`; `TestOpenRecordPrecedesTheOpenTransition` (the operator open door and the job write it before the transition with the same `OpenedAt`; a second write is a no-op; a failed transition leaves a record the return door rejects); the allowlist test. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|ReturnOf|DiffersOnlyWhere' -count=1`, `go test ./internal/console -run 'OpenRecord' -count=1`, then the record-write census. |
| -- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort, the open record and the self-serve fields`. |
| +- [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort, caller-proposed stamps, the open record and the self-serve fields`. |
| ### Task 3: Per-box reservations with a legacy projection and a sentinel |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineActive.V`, `Reservations map[string]OfflineReservation` keyed by box id, the legacy projection), `internal/console/apiinstancemove_offline.go` (the doors reserve from the selection) |
| +- Modify: `internal/moves/offline.go` (`OfflineActive` gains `V int \`json:"v,omitempty"\`` and `Reservations map[string]OfflineReservation \`json:"reservations,omitempty"\`` keyed by box id; the legacy fields are the **projection**), `internal/console/apiinstancemove_offline.go` (the start and stage doors reserve from the selection; `offlineReleaseLock` → `releaseBoxes`) |
| - Create: `internal/console/offlinereserve.go` |
| - Test: `internal/console/offlinereserve_test.go` |
| **Interfaces:** |
| -- `moves.OfflineReservation{Kind, ID, Host, Account, App string; ReservedAt time.Time}` (tagged); **the sentinel key `"*"`**: a reservation of a selection whose participants lack a verified id (the operator's `requireIDs: false` path) is written under `"*"` and conflicts with every other acquisition, and every acquisition conflicts with it (so an id-less operator move and a self-serve move on the same physical box never coexist). |
| -- `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`: an own `(Kind, ID)` entry (any key) is a retry: accepted first, keeping `ReservedAt`, **before** the exclusive gate; then: the legacy body normalised; a malformed entry → unreadable; `cfg.SelfServeMoves != "offline"` → refuse when any other operation holds anything; any participant (or `"*"`) held by another `(Kind, ID)` → busy unless reclaimable; one conditional write with the projection; one CAS retry. |
| -- `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error` (as rev 3). |
| -- Reclaim (`move`): the judged record carries no move with `ID`, **and** `SelfServeMove(id)` is absent or terminal with `release_pending = 0`, **and** no row for the host has `request_issued` set; a SQL or store failure → not reclaimable. (`cleanup`): plan 4's contract (`RetirementsComplete`, `State == ended`, `Metrics.BoxID`). |
| -- Rollout: as rev 3, stated in terms of operations: old consoles cannot be reintroduced while any reservation exists that the old console did not create; rollback = switch off, wait for every self-serve operation to end, then the old console. |
| +- `type moves.OfflineReservation struct { Kind string \`json:"kind"\`; ID string \`json:"id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; ReservedAt time.Time \`json:"reserved_at"\` }` (`Kind` is `move` or `cleanup`; an entry with an unknown kind, an empty id or host, or a zero time is malformed: the whole object is `unreadable` until an operator repairs it). **The sentinel key `"*"`**: a selection with any participant whose `InstanceID` is empty is written under `"*"` and conflicts with every other acquisition, and every acquisition conflicts with it. Self-serve selections never use the sentinel (Task 5 requires `InstanceID` on every self-serve participant). |
| +- `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`, in order: (1) read `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}`, normalised into the table on the first new write; (2) **validate the whole object**: any malformed entry → `unreadable`; (3) **own retry**: entries with this `(Kind, ID)` exist → accepted only when they cover **exactly** the expected keys (every participant's `BoxID`, or `"*"` for a sentinel selection) with no extra; a partial own reservation → the missing keys are acquired in the same write if free, else `busy`; an own reservation whose keys differ from the expected selection → refuse "the reservation does not match this move's participants" (the job aborts); an accepted retry keeps `ReservedAt`; (4) `cfg.SelfServeMoves != "offline"` → refuse `busy` when any entry of another operation exists (today's deployment-wide semantics); (5) any expected key held by another `(Kind, ID)` → `busy` ("<host> is moving offline. move one instance at a time") unless reclaimable (below); (6) one conditional `Put` with every expected key and the projection (`Host, Op, Account, App, At` of the oldest `move` entry; of the oldest entry of any kind when no move holds); a CAS conflict re-reads once and retries from (2), then `busy`. |
| +- `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error`: removes every entry with that `(Kind, ID)`, rewrites the projection, deletes the object when empty (`DeleteIfMatch`); absence is success. |
| +- **Reclaim** (`time.Since(ReservedAt) >= offlineLockGrace`), for `move`: (a) `fenceRecord(ctx, acct, app, host, fenceOwner{})` (plan 1) must return `fenced` or `absent` (an error → not reclaimable); (b) the fenced observation carries no move with `ID`; (c) `SelfServeMove(id)` is absent or terminal with `release_pending = 0`; (d) no row for the host has `request_issued` set. An **operator** reservation (no row) is reclaimed on (a) and (b) alone: the fence makes a delayed operator start or stage PUT fail its CAS, so a negative observation after the fence is conclusive. For `cleanup` (plan 4): the owning box's status report (`status/boxes/<label>.json`, `GeneratedAt` within 2 status intervals, `Metrics.BoxID` equal to the key) lists the id in a terminal state with `RootRequest == ""` and `ReleasePending == false`, or its `Retirements` block is complete (`RetirementsComplete`) and lists no such id. Any read failure, a malformed body, or an absent or incomplete status → not reclaimable. |
| +- **Rollout and rollback:** old consoles cannot be reintroduced while any reservation exists that the old console did not create (its release deletes the whole object when the projection names its own move; `TestOldConsoleReleaseErasesATable` documents the hazard); the supported sequence is roll consoles → turn the switch on; rollback is switch off → wait until **every incompatible operation** ends (every self-serve row terminal and released, every cleanup reservation gone, every operator reservation created under the table form released) → the old console. |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestSentinelReservationConflictsBothWays`; `TestOwnRetryAcceptedBeforeTheExclusiveGate` (switch off, two disjoint reservations, each retries its own → accepted); `TestServerCommittedQueueWriteIsNotReclaimable` (a queue write that timed out client-side and commits later: the reservation is not reclaimed while `request_issued` is set, and Task 7's fence makes the late commit impossible before release). |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|Sentinel|OwnRetry' -count=1`. |
| -- [ ] **Step 5: Commit** `console: per-box reservations with a legacy projection, a deployment-wide sentinel and own retries`. |
| +- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnRetryNeedsTheCompleteSelection` (exact → accepted keeping `ReservedAt`; partial with free keys → completed in one write; partial with a held key → busy; a changed selection → the mismatch refusal; a malformed unrelated entry is detected before the retry is accepted); `TestSentinelReservationConflictsBothWays`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive case and each refusing case above; a delayed queue PUT that commits after a negative re-read and before the fence → the fence observes it and the reservation is kept; a delayed operator start PUT → the fence fails it and the reservation is reclaimed; a fence error → kept; a SQL failure → kept; a cleanup-only obligation at switch-off → kept and recovery proceeds); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody`; `TestSwitchOffRecoveryReacquiresItsOwnReservation` (two disjoint reservations, each retries its own → accepted); `TestOperatorDoorsUseReservations`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|Sentinel|OwnRetry|OperatorDoorsUse|Malformed' -count=1`. |
| +- [ ] **Step 5: Commit** `console: per-box reservations with a legacy projection, a sentinel, complete own retries and fenced reclaim`. |
| ### Task 4: The journal, the events table, the console lease and the pass skeleton |
| **Files:** |
| -- Modify: `internal/console/store.go`, `internal/console/store_gate.go` (`"self_serve_moves"`, `"self_serve_events"`, `"self_serve_timings"`: `{Sqid: "account", App: "app"}`; `"move_timings"`: `{NonApp: ...}`), `internal/console/claim_workers.go`, `cmd/console/main.go` (`armMovesPaging(ctx, cfg, store, getenv)`), `internal/console/server.go` (`selfServeMode`, `selfServeAdmits`), `internal/console/serialbump.go` (`func (s *Server) bumpSerialErr(ctx) error`, the existing `bumpSerial` calling it and logging) |
| +- Modify: `internal/console/store.go`, `internal/console/store_gate.go` (`"self_serve_moves"`, `"self_serve_events"`, `"self_serve_timings"`: `{Sqid: "account", App: "app"}`; `"move_timings"`: `{NonApp: ...}`), `internal/console/claim_workers.go`, `cmd/console/main.go` (`cfg.SelfServeMoves`; `armMovesPaging(ctx, cfg, store, getenv)`: arm when `cfg.HostClaim`, or `selfServeMode()`'s conditions hold; a store read failure → arm, logged), `internal/console/server.go` (`Config.SelfServeMoves string`; `selfServeMode`, `selfServeAdmits`), `internal/console/serialbump.go` (`func (s *Server) bumpSerialErr(ctx) error`; the existing `bumpSerial` calls it and logs, keeping its bounded detached behaviour) |
| - Create: `internal/console/selfserve_store.go`, `internal/console/selfserve_pass.go`, `internal/console/selfserve_lease.go` |
| - Test: `internal/console/selfserve_store_test.go`, `internal/console/selfserve_pass_test.go` |
| **Interfaces:** |
| -- `self_serve_moves`: rev 3's columns with these changes: `request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_confirmed INTEGER, request_consumed INTEGER` (the lifecycle of one request: issued at `request_issued` by `request_phase`; confirmed with the **committed** stamp read from the record; consumed when its answer was used; a new request may be issued only when the previous one is consumed or voided), `stage_request`, `precheck_request` (the committed stamps the job currently waits on), `notified_seq INTEGER` (the record's `record_seq` last announced), `expiry_pending INTEGER`, `health_attempts INTEGER`, `health_consecutive_ok INTEGER`, and the rest as rev 3. `self_serve_events (id TEXT, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, phase))`; `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| -- Phases: rev 3's. **Compatible sets** for reconciliation (Task 7): `offline-queued` ↔ {queued, reports}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier}; `offline-stopping` ↔ {stopping}; `offline-sealing`, `offline-sealed` ↔ {sealing}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain}; `offline-open` ↔ {opened, health, finishing(gate 1)}; `offline-finishing` ↔ {finishing(gate 2)}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not already in that set. |
| -- Store methods: rev 3's, with `IssueSelfServeRequest(id, kind, phase string, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`), `ConfirmSelfServeRequest(id string, committed time.Time) error`, `ConsumeSelfServeRequest(id string) error`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiryPending(id string, pending bool) error`, `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, now time.Time) (bool, error)` with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing'))`, `InsertSelfServeEventTx(tx, e)` used inside the same transaction as `AdvanceSelfServe` (one transaction advances the phase and inserts its event), `DeliverSelfServeEvent(id, phase string, deliver func() error) error` (marks emitted in the same transaction as the audit row's insert, through `auditActivityTx`; a failure leaves it undelivered). |
| -- Lease: `acquireSelfServeLease`, `leaseToken.renew`, `leaseToken.valid(now)`, `leaseToken.deadline() time.Time`; `selfServeApply` derives `ctx, cancel := context.WithDeadline(ctx, t.deadline())` for the record write so no PUT or retry outlives the lease, and re-checks `t.valid(now)` before each retry inside `SetRecordMove` through a `provision.WithWriteGuard(ctx, func() error)` hook that `WriteRecord` calls immediately before every `Put`. |
| -- `StartSelfServeMovesPass`: starts when `selfServeMode()` (which includes the recovery conditions); a failing startup read of rows, events or `active.json` → the pass runs (fail closed) and retries the reads each tick, logging; `selfServeTick`: renew the lease, the reservation scan, `drainSelfServeEvents`, then one step per open row. |
| - |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestRequestLifecycleIssueConfirmConsume` (no second issue while unconsumed; confirmation records the committed stamp; consume once); `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestEventAndAdvanceAreOneTransaction` (a crash after the advance cannot lose its event); `TestEventDeliveryIsTransactional` (an audit insert failure leaves the event undelivered; a crash after the audit insert does not duplicate it); `TestPassStartsAndRetriesOnReadFailure`; `TestLeaseBoundsEveryWriteAttempt` (a `Put` paused past the lease deadline is cancelled by the context; a retry after lapse is refused by the guard); `TestClaimStoreOpensWithEveryNewTable`. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseBounds|RequestLifecycle|EventAnd|EventDelivery|PagingArmed|ClaimStoreOpens' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self-serve journal with request lifecycles and compatible phase sets, transactional events, a lease that bounds every write, and the pass skeleton`. |
| +- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, expiry_pending INTEGER, expiry_phase TEXT, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_nonce TEXT, request_expect_seq INTEGER, request_proposed INTEGER, request_confirmed INTEGER, request_consumed INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_seq INTEGER, event_seq INTEGER, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, open_record_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| +- **The request lifecycle** (one request at a time per row): `IssueSelfServeRequest(id, kind, phase string, nonce string, expectSeq uint64, proposed time.Time, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`: sets `request_issued = now, request_kind, request_phase, request_nonce, request_expect_seq, request_proposed, request_confirmed = 0, request_consumed = 0`); `ConfirmSelfServeRequest(id, nonce string, now time.Time) error` (`WHERE request_nonce = ?`: `request_confirmed = now`; for kind `stage` also `stage_request = request_proposed`, for `precheck` also `precheck_request = request_proposed`); `VoidSelfServeRequest(id, nonce string) error` (`request_consumed = 1` without confirmation; the phase function decides again); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, nonce string, now time.Time) (bool, error)`: one UPDATE with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing')) AND (? = '' OR request_nonce = ?)` that sets the phase, `phase_started_at`, the deadlines, `expiry_pending = 0, expiry_phase = ''` when `to <> expiry_phase`, and `request_consumed = 1` when a nonce is given (**consumption and the advance are one statement**), and inserts the phase event in the same transaction (`event_seq = event_seq + 1`); a stale worker holding another nonce cannot consume. |
| +- Other store methods: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0 OR open_record_pending = 1`, **ordered by `created_at, id`**: the pass steps rows in that order; a row waiting on a busy box stays in `reserving` and does not block rows with disjoint participants); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`health_attempts + 1`; `consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` kept once set), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiry(id, phase string, pending bool) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`, `SetSelfServeOpenRecordPending(id string, pending bool) error`. |
| +- Table `self_serve_events (id TEXT, seq INTEGER, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, seq))`: `InsertSelfServeEventTx(tx *sql.Tx, e SelfServeEvent) error` (allocates `seq` from the row's `event_seq` in the same transaction; used by `AdvanceSelfServe` and by every refusal, fact decision, lane probe, health probe and timing); `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`; `DeliverSelfServeEvent(ctx context.Context, e SelfServeEvent, deliver func(tx *sql.Tx) error) error` (one transaction: `deliver(tx)` inserts the audit row through `auditActivityTx(tx, ...)`, then `emitted = 1`; a failure rolls both back). Table `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| +- Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. |
| +- **Compatible sets** (Task 7's reconciliation; "pending" = the row has an unconfirmed request for that phase's write): `offline-queued` ↔ {queued, reports, staging(pending)}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier, stopping(pending)}; `offline-stopping` ↔ {stopping, sealing(pending)}; `offline-sealing` ↔ {sealing}; `offline-sealed` ↔ {sealing, landing(pending)}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain, opened(pending)}; `offline-open` ↔ {opened, health, finishing}; `offline-finishing` ↔ {finishing}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done, finishing(gate-2 pending), reverting(finish pending)}; no move and no `LastMove` for the id ↔ {requested, reserving, queued(pending)}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not in that set. |
| +- Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `type consoleLease struct { Console string; Until time.Time }`; `acquireSelfServeLease(ctx, now) (*leaseToken, bool, error)`; `leaseToken.renew(ctx, now)` (every 20 s during a tick); `leaseToken.valid(now) bool`; `leaseToken.deadline() time.Time`; `selfServeApply` derives `ctx, cancel := context.WithDeadline(ctx, t.deadline())` for the record write and re-checks `t.valid(now)` before each retry through `provision.WithWriteGuard(ctx, func() error)`, a hook `WriteRecord` calls immediately before every `Put`. A cancelled context does **not** prove the write cannot commit; that is why every write is a persisted request (Task 7). |
| +- `StartSelfServeMovesPass`: runs while `selfServeMode()`; a failing startup read of rows, events or `active.json` → the pass runs (fail closed) and retries the reads each tick, logging; `selfServeTick`: renew the lease, the reservation scan, `drainSelfServeEvents`, then one step per open row in order. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestRequestLifecycle` (no second issue while unconsumed; confirm only with the nonce; void; advance with the wrong nonce → false; advance with the right nonce consumes in the same statement); `TestAdvanceIsPredicatedOnPhaseAndNoAbort`; `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestAdvanceClearsAForeignExpiry`; `TestEventAndAdvanceAreOneTransaction`; `TestEventDeliveryIsTransactional`; `TestEventsHaveAPerRowSequence` (repeated prechecks produce distinct events); `TestOpenRowsOrderedAndIncludePending`; `TestPassStartsAndRetriesOnReadFailure`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseBoundsEveryWriteAttempt`; `TestPagingArmedForSelfServe`; `TestClaimStoreOpensWithEveryNewTable`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseBounds|RequestLifecycle|EventAnd|EventDelivery|EventsHave|PagingArmed|ClaimStoreOpens' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve journal with persisted requests and nonce-predicated advances, sequenced transactional events, the lease and the pass skeleton`. |
| ### Task 5: Record eligibility, the split start validation, the door bodies and the view |
| @@ -113,31 +122,36 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 8) |
| - Test: `internal/console/selfserve_door_test.go`, `internal/console/apiinstancemove_offline_snapshot_test.go` |
| **Interfaces:** |
| -- `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal` (`moveGate`'s class checks in order). |
| -- `func (s *Server) offlineRecordChecks(ctx context.Context, acct *Account, app, host string, rec *provision.Record) *offlineRefusal` (the pre-body half: the seed block, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`); `func (s *Server) offlineRegionChecks(ctx context.Context, acct *Account, app, host string, rec *provision.Record, region string, stage, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)` (the post-body half: the region, the manifest, the fleet, the home rule, `offlineSelect`, the stage rules, participant pinning with `requireIDs`, the return's evidence); `offlineSelection{From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string; Sentinel bool}` (`Sentinel` when any participant lacks a verified id). Participant pinning: `BoxID` and `Label` from the edges file; `InstanceID` and `BootID` from `status/boxes/<label>.json`'s metrics (fresh within 10 minutes; else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, label)` finds a record and `boxid.Verifier.Enroll` yields evidence naming that `InstanceID` (a record that exists but fails → refuse "box <label> has an identity record that does not verify"); with `requireIDs`, `BoxID`, `InstanceID` and `BootID` must all be present (else the older-release sentence). |
| -- The self-serve bodies: as rev 3, with: `selfServeGate` (Task 1); `s.selfServeAdmits()` checked first in prepare; the record observation; `recordEligibility`; the run and move checks; the open row; the body; `offlineRecordChecks` and `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box; the status facts (Task 6: the pinned source's report, `Metrics.BoxID`, `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future; `OverCap` → the files sentence; `Truncated` → the measurement sentence); the lane; the bound migration; `AdmitSelfServeMove`. |
| -- **The view's precedence** (`selfServeViewOf(rec, open, terminal)`): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.SelfServe.ID` → the open row's view; (2) the record carries any other move (an operator's, or a self-serve move whose row this console does not hold) → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. An absent record is today's 404 from `moveGate`. The types as rev 3. |
| -- Copy rows: rev 3's plus "box <label> has an identity record that does not verify"; "the box <label> has not reported recently". |
| - |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestStartValidationSplitPreservesOperatorOrdering` (the pre-body refusals answer before a malformed body, as main); `TestParticipantPinning` (verified with a box record; pinned unverified without one; a record that fails to verify refuses; `requireIDs` refusals); `TestViewPrecedenceOperatorMoveOverOpenRow` (an open row in `reserving` and an operator move on the record → the legacy view). |
| +- `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal`: `moveGate`'s checks in order (`Kind`, `Addr`/`Addrs`, fleet placement, `Pool`, `!HibernateOrDefault()`; 400s). |
| +- `func (s *Server) offlineRecordChecks(ctx, acct, app, host string, rec *provision.Record) *offlineRefusal` (the pre-body half: the seed block, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`); `func (s *Server) offlineRegionChecks(ctx, acct, app, host string, rec *provision.Record, region string, stage, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)` (the post-body half: the region, the manifest, the fleet, the home rule, `offlineSelect`, the stage rules, participant pinning, the return's evidence); `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string; Sentinel bool }` (`Sentinel` when any participant's `InstanceID` is empty). |
| +- **Participant pinning.** `BoxID` and `Label` from the edges file. With `requireIDs: false` (**every operator door**): `InstanceID` from `status/boxes/<label>.json` when the report exists, is within 10 minutes and carries `Metrics.BoxID == BoxID`; otherwise `InstanceID = ""`, `Verified = false`, and the move proceeds as today (`offlineSelect` already admits a box with nil metrics); no identity record is read. With `requireIDs: true` (**the self-serve door and the job**): the status report must be present and fresh with a non-empty `InstanceID` (else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, label)` finds a record and the verifier (constructed as `ClaimRootHTTP` constructs it: the deployment account and the manifest's AWS region for the box's region) `Enroll`s it to evidence naming that `InstanceID`; a record that exists but does not verify → refuse "box <label> has an identity record that does not verify"; no record → pinned with `Verified: false` (allowed; the status report's `InstanceID` is the identity). |
| +- **The self-serve bodies.** `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}`: `selfServeGate`; `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`'s sentence; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` over 80 runes or with a control character → 400 "display_name must be at most 80 characters of plain text"; `offlineRecordChecks`; `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source's status facts (Task 6: `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` is the returned record and `recordEligibility`, the run and move checks run again on it; `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating; 200 `{"estimate_seconds", "files_bytes", "replica_bytes"}` or the refusal. `POST .../move/cut {"id"}`: `selfServeGate`; the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `AdvanceSelfServe(id, ready, restage, ..., "")` → 202; false → re-read: a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". `DELETE .../move {"id"}`: `selfServeGate`; the row and id; `RequestSelfServeAbort(id, [requested, reserving, queued, reports, staging, prechecks, ready], now)` → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| +- **The view** (`selfServeViewOf(rec, open, terminal, now)`, rendered whenever claims-off and Phase 1 off): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.SelfServe.ID && open.Incarnation == rec.Incarnation` → the open row's view (`preparing` through `ready`, `cutting` from `restage`); (2) the record carries any other move → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| +- Copy rows: "only an account owner can mint a move-capable instance token"; "self-serve moves are not enabled on this console"; "could not read the instance"; "could not read the move"; "scheduled starts are not available yet"; "display_name must be at most 80 characters of plain text"; "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; "the source box has not reported recently. try again in a minute"; "the box <label> has not reported recently"; "box <label> has an identity record that does not verify"; "this account's files are too large to move this way"; "could not measure the account's files. try again in a minute"; "no staging lane is configured from <from> to <to>"; "another change to this instance is still settling. try again in a minute"; "this is not the move you confirmed"; "the move was cancelled"; "the move is not ready to cut yet. it is <phase>"; "the move was confirmed and cannot be cancelled. it completes or returns on its own"; "no move is in flight"; "the reservation does not match this move's participants". |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged` (snapshots from main, incl. operator moves with nil metrics and no identity record **succeeding** as today); `TestSwitchOffIsByteIdentical`; `TestRecordEligibilityIsMoveGatesChecks`; `TestStartValidationSplitPreservesOperatorOrdering`; `TestParticipantPinning` (operator: nil metrics → pinned id-less, sentinel; self-serve: fresh report required; verified with a record; unverified without; a failing record refuses); `TestPrepareRefusesBeforeWriting` (one subtest per refusal); `TestPrepareMigratesALegacyRecordBoundThenAdmits`; `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (barriers both orders; a stale snapshot; a second cut → 202; a stale nonce cannot confirm or consume); `TestViewPrecedence` (the five cases; an operator move over an open row; a recreation; a terminal row after switch-off, a restart and a later poll within 7 days → `last` rendered; older → not); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|RecordEligibility|StartValidationSplit|ParticipantPinning|Prepare|MutationsName|ViewPrecedence|EstimateDoor' -count=1`. |
| -- [ ] **Step 5: Commit** `console: record eligibility and the split start validation shared by the operator doors and the job; participants pinned with identity; the self-serve door and view`. |
| +- [ ] **Step 5: Commit** `console: record eligibility and the split start validation shared by the operator doors and the job; participants pinned by instance; the self-serve door and a mode-independent view`. |
| -### Task 6: The boxes' reports: the participant resolver and identity, the lane, the source precheck, target-ready, the staged receipts, and the per-host status block |
| +### Task 6: The boxes' reports: participant identity, the lane, the source precheck, target-ready, the staged book, and the per-host status block |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`OfflineReport` gains `BoxID`, `InstanceID`, `Request`, `Lane`, `Ready`, `Precheck`, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject` (the receipts' objects: `StagedObject{Size, ETag, VersionID string \`json:"version_id,omitempty"\`}`), `StagedBucket`, `StagedRegion`, `StagedRequest time.Time`, `RoundCopiedBytes`, `RoundDeferredBytes`, `RoundDeferredObjects`, `Rounds`, `TombstoneLifted`, `TombstoneLiftedAt`, `FinalCopiedBytes`; the new roles and phases; **`ValidateOfflineReport(r)` is unchanged for legacy structure**, and `ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState) error` adds the contextual rules: a move with `Participants` → `BoxID` and `InstanceID` non-empty on every report; a move with `SelfServe` → a `staged` report has `StagedKnown`, `StagedBucket`, `StagedRegion`, `StagedRequest`; role `lane` ↔ phase `lane` with `Lane`; `precheck` with `Precheck` and `Request`; `ready` with `Ready` and `Request`; `reverting` with `TombstoneLifted`; `readOfflineEvidence` applies both and drops a report that fails the contextual rule for its move, logging it), `internal/moves/participants.go` (new: `ParticipantRole`, `ParticipantStillPresent(p Participant, m *status.BoxMetrics) bool` = `m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID`), `internal/lease/mirror.go` (`func ReadReceipts(ctx context.Context, m MirrorStore, rc Receipts) (map[string]CopyReceipt, error)` exporting the ledger's objects, `type CopyReceipt struct { SrcETag string; SrcSize int64; DstETag string; DstVersionID string }`), `internal/host/movesoffline.go` (`moveCfg.BoxID`, `moveCfg.InstanceID` (from `status.InstanceID`); `ParticipantRole` in `offlineRound`, `offlineReassert`, the view's `hold`/`keep`; the queued and staging dispatch as rev 3), `internal/host/movesoffline_seal.go` (`offlineTarget` by participant; the `reverting` report with the durable lift before cleanup, the first lift time preserved on retries; `FinalCopiedBytes`), `internal/host/movesoffline_stage.go` (the stager report's `Staged` **from `lease.ReadReceipts` after the round** (dst ETag and version id of every object this move's copies produced), `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = StagedAt`), `internal/host/movesoffline_land.go` (`offlineTargetReady` with `PrefixFacts.Objects` carrying `ETag`, `VersionID`, `LastModified`), `internal/host/agent.go` (the status tick: `p.Hosts`, `p.Lanes`, `p.BoxID`, `p.InstanceID` is already there), `internal/status/snapshot.go` (`Probes.Hosts`, `Probes.Lanes`, `Probes.BoxID`; the attachment condition; `BoxMetrics.Hosts`, `BoxMetrics.Lanes`, `BoxMetrics.BoxID`), `internal/host/movespass.go` (`armOffline` arms `cfg.Versioned`, `cfg.LaneProbe`) |
| -- Test: `internal/moves/offline_test.go`, `internal/moves/participants_test.go`, `internal/lease/receipts_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineReport` fields, roles, phases and the two validators below), `internal/moves/participants.go` (new), `internal/lease/mirror.go` (`CopyEvent`, `MirrorPrefixKeepObserved`), `internal/host/movesoffline.go` (`moveCfg.BoxID`, `moveCfg.InstanceID` from `status.InstanceID`; `ParticipantRole` in `offlineRound`, `offlineReassert`, the view's `hold`/`keep`; the queued and staging dispatch), `internal/host/movesoffline_seal.go` (`offlineTarget` by participant; the `reverting` report with the durable lift before cleanup, the first lift time preserved on retries; `FinalCopiedBytes`), `internal/host/movesoffline_stage.go` (the stage book), `internal/host/movesoffline_land.go` (`offlineTargetReady`), `internal/host/agent.go` (the status tick), `internal/status/snapshot.go` (`Probes.Hosts`, `Probes.Lanes`, `Probes.BoxID`, `Probes.InstanceID` (exists); `BoxMetrics.Hosts`, `BoxMetrics.Lanes`, `BoxMetrics.BoxID` private), `internal/host/movespass.go` (`armOffline` arms `cfg.Versioned`, `cfg.LaneProbe`) |
| +- Test: `internal/moves/offline_test.go`, `internal/moves/participants_test.go`, `internal/lease/observed_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| **Interfaces:** |
| -- `moves.Fact{Known, OK bool; Detail string}` (tagged); `moves.PrefixFacts{Known bool; Objects map[string]PrefixObject}`; `type PrefixObject struct { ETag string \`json:"etag"\`; Size int64 \`json:"size"\`; VersionID string \`json:"version_id,omitempty"\`; LastModified time.Time \`json:"last_modified"\` }` (current versions only, delete markers excluded). |
| -- `moves.TargetReadyReport` as rev 3 with `PrefixObject`; `moves.StrayEntry{Path string; Kind string (dir, file, symlink, other); Bytes int64; Entries int; Known bool; Detail string}` replaces `ExtraDir`; `SourcePrecheck{FilesBytes, FilesEntries, ArchiveEligible Fact, Strays StraysFact{Known bool; Entries []StrayEntry; Detail string}, Build, StagePathArmed Fact, StraysRemoved []string, StraysLeft []StrayEntry}`: a non-directory stray (any `Kind != dir`) is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). |
| -- `status.HostFacts{CollectedAt, Known, Err, Incarnation uint64 \`json:"incarnation"\`, ReplicaBytes, ReplicaObjects, ReplicaCollectedAt, FilesBytes, FilesEntries, OverCap, Truncated, ExtraDirs}`; `measureFiles` as rev 3. |
| -- **Staged ownership** (the console, Task 7's prechecks and Task 8's barrier): the stager report must be `StagedKnown` with `StagedBucket/Region` equal to the record's and `StagedRequest == row.StageRequest`; the target's `MovesPrefix` must be `Known`; the two sets of keys must be equal; for every key the target's `ETag` equals the receipt's `DstETag` and, when both carry a version id, the version ids are equal; a mismatch → "the staging prefix for <host> does not match what this move staged" (the overwrite and extra-object cases refuse). |
| - |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestLegacyReportsStillValidate` (serialized old report bytes pass `ValidateOfflineReport` and are kept by `readOfflineEvidence` for a move without `Participants`); `TestContextualValidationForSelfServeMoves`; `TestEmptyStagedInventoryRoundTrips` (`StagedKnown` with an empty map survives serialisation); `TestReadReceiptsExportsTheLedger`; `TestStagerPublishesReceiptsNotAListing` (an overwrite between the copy and the report → the receipt's ETag, not the listed one; the console's comparison then refuses); `TestStrayFileRefusesBeforeStop`; `TestParticipantStillPresent` (same label, another instance id → replaced); `TestHostFactsCarryTheLifetimeAndAges`. |
| +- `OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `InstanceID string \`json:"instance_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject \`json:"staged,omitempty"\`` (the book), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `StagedRequest time.Time \`json:"staged_request,omitzero"\``, `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`. |
| +- `type StagedObject struct { Size int64 \`json:"size"\`; ETag string \`json:"etag"\`; VersionID string \`json:"version_id,omitempty"\`; Deferred bool \`json:"deferred,omitempty"\` }` (`Size` keeps today's `int64` for the seal consumers). `type PrefixObject struct { ETag string \`json:"etag"\`; Size int64 \`json:"size"\`; VersionID string \`json:"version_id,omitempty"\`; LastModified time.Time \`json:"last_modified"\` }`; `type PrefixFacts struct { Known bool \`json:"known"\`; Objects map[string]PrefixObject \`json:"objects,omitempty"\` }` (keys **relative to the prefix**; current versions only; delete markers excluded; `lock.json` excluded). |
| +- `type Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. `type LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the existing Phase 1 `laneProbe`: put-if-absent, list, delete-version under `moves/_lane/<move id>`). `type TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\`; MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\`; Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }`. `type StrayEntry struct { Path string \`json:"path"\`; Kind string \`json:"kind"\` (dir, file, symlink, other); Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\`; Known bool \`json:"known"\`; Detail string \`json:"detail,omitempty"\` }`; `type StraysFact struct { Known bool \`json:"known"\`; Entries []StrayEntry \`json:"entries,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; Layout string \`json:"layout"\` (flat, nested); ArchiveEligible Fact \`json:"archive_eligible"\`; Strays StraysFact \`json:"strays"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; StraysRemoved []string \`json:"strays_removed,omitempty"\`; StraysLeft []StrayEntry \`json:"strays_left,omitempty"\` }`: a non-directory stray is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. |
| +- `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Known bool \`json:"known"\`; Err string \`json:"err,omitempty"\`; Incarnation uint64 \`json:"incarnation"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; ReplicaCollectedAt int64 \`json:"replica_collected_at"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; OverCap bool \`json:"over_cap,omitempty"\`; Truncated bool \`json:"truncated,omitempty"\`; Strays []string \`json:"strays,omitempty"\` }`; `measureFiles(ctx, files, envPath string, budget time.Duration, capBytes int64) (bytes int64, entries int, overCap, truncated bool, err error)` (a bounded walker applying the archive's counting rules without stopping at the cap: records `OverCap` and continues until the budget (60 s) or 2 × the cap, where it records `Truncated`); the replica listing cached per tick with its own `ReplicaCollectedAt`; `Known` only when every part succeeded. `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES`; `BoxMetrics.BoxID string` (the edges writer's value); `BoxMetrics.InstanceID` exists. |
| +- **Participants:** `func ParticipantRole(m *provision.MoveState, boxID, instanceID, label, placement, home string) (source, target, sealer, stager bool)`: with `Participants`: a participant matches when `p.BoxID == boxID && (p.InstanceID == "" || p.InstanceID == instanceID)`; a box whose `BoxID` matches a participant with a non-empty different `InstanceID` has **no role** (it logs "box <label> is not the pinned instance for move <id>" once per move and does nothing); without `Participants`: today's region and label rules. `func ParticipantStillPresent(p provision.Participant, m *status.BoxMetrics) bool` = `p.InstanceID == "" || (m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID)` (the console's gate check, on the label's status report within 10 minutes; a missing report with a non-empty pinned id → "box <label> was replaced" after 10 minutes, "box <label> has not reported recently" before). |
| +- **Validation:** `ValidateOfflineReport(r)` is **unchanged** (legacy structure). `func ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState, now time.Time) error` adds the contextual rules: a move with `Participants` → the report's `BoxID` and `InstanceID` equal the pinned participant for its role (`InstanceID` compared only when the pinned one is non-empty); a move with `SelfServe` → role `lane` ↔ phase `lane` with `Lane`; `source`+`precheck` with `Precheck` and `Request`; `target`+`ready` with `Ready` and `Request`; `stager`+`staged` with `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest`; `sealer`+`reverting` with `TombstoneLifted`; a `failed` report needs only the identity rule (its diagnostic is kept). **Freshness table:** request-bound reports (`lane` by move id; `precheck`, `ready` by `Request`; `staged` by `StagedRequest`) have **no age limit** and at most 60 s of future skew; the phase reports today's gates read (`stopped`, `sealed`, `landed`, `opened`, `reverted`, `reverting`, `failed`) keep `offlineFresh` (2 min, 1 min skew) where today applies it; the seal is immutable and gets only the identity rule; participant identity from status is 10 min. `func (s *Server) readOfflineEvidence(ctx context.Context, host, id string, m *provision.MoveState) (offlineEvidence, error)` applies both validators and drops a report that fails the contextual rule, logging it; every caller (`offlineStepFor`, the seal, flip, open and finish checks, `offlineView`, the job) passes the move it holds (`nil` for a move it has not read: legacy validation only). |
| +- **The stage book (item 6).** `type lease.CopyEvent struct { Rel string; SrcETag string; SrcSize int64; DstETag string; DstVersionID string; Unchanged, Deferred, Deleted bool }`; `func MirrorPrefixKeepObserved(ctx, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string, rc Receipts, keep func(string) bool, observe func(CopyEvent)) (MirrorRound, error)`: emits one event per source key the round decided (a copy with the ETag and version the copy returned; `Unchanged` with the listed destination ETag when the round skipped it as equal; `Deferred` when the round deferred it) and `Deleted` per destination key removed; `MirrorPrefixKeep` = `...Observed(..., nil)`, so ordinary mirroring and the receipt ledger are unchanged. The stager keeps `stageBook{MoveID string; Objects map[string]moves.StagedObject}` across rounds: seeded at the first round of a move from its own last `staged` report for that move id when one exists (else empty); every event upserts (`Deleted` removes; `Deferred` marks the existing entry deferred without changing its ETag, or inserts `{Deferred: true}` for a key never copied); the `staged` report carries the whole book with `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = m.StagedAt`. **Staged ownership** (the console's prechecks and barrier): `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest == row.StageRequest`; the target's `MovesPrefix.Known`; the key sets are equal (both relative to their prefixes; entries with `Deferred` and no ETag must be **absent** from the target); for every key the target's `ETag` equals the book's, sizes are equal, and version ids are equal when both are present; the barrier additionally requires `RoundDeferredObjects == 0`; a mismatch → "the staging prefix for <host> does not match what this move staged". |
| +- **Reports to write:** the source participant in `offline-queued`: the lane probe once per move → the `lane` report; `offlineSourcePrecheck` for the current `PrecheckAt` → `precheck` with `Request = PrecheckAt`; the target participant: `offlineTargetReady` → `ready` with `Request = PrecheckAt`; in `offline-staging` both re-report when `PrecheckAt` is newer than their last `Request`; the stager's `staged` after each round; the sealer's `reverting` after `LiftTombstone` succeeds and before the prefix cleanup; the final `reverted` keeps both lift fields; the normal seal's final-copy bytes in `FinalCopiedBytes`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestParticipantRoleByBoxAndInstance` (a matching pair; an id-less participant matches any instance of the box; a replaced box has no role); `TestParticipantStillPresent`; `TestLegacyReportsStillValidate`; `TestContextualValidationForSelfServeMoves` (the matrix: every role/phase/payload; identity mismatch; a failed report kept with only identity checked); `TestFreshnessTable`; `TestEmptyStagedInventoryRoundTrips`; `TestObservedMirrorEmitsEveryDecision` (copy, unchanged same-ETag, deferred, deleted; the receipt ledger unchanged; `MirrorPrefixKeep` behaviour unchanged against the production `NewS3Mirror` wrapper through `stageDeleteGate`); `TestStageBookAcrossRounds` (seeded from the last report; same-ETag copies present; a deferred then copied key; a deleted key gone; an overwrite between the copy and the report → the copy's ETag; a crash before the report → the next round re-copies and the book converges); `TestStagedOwnershipComparison` (equal; missing; extra; changed ETag; changed size; version mismatch; deferred-and-never-copied present → refuse; wrong request; wrong destination; unknown prefix); `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown`; `TestSourcePrecheckFacts` (incl. `Layout`); `TestStrayFileRefusesBeforeStop`; `TestEmptyStrayDirsRemovedAtStartOnly`; `TestSealerPublishesTombstoneLiftedBeforeCleanup`; `TestQueuedRoundRunsOnlyLanePrecheckAndReady`; `TestParticipantsReReportOnANewPrecheckStamp`; `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID`, `InstanceID` in the private report, absent from `fleet.json`); `TestHostFactsCarryTheLifetimeAndAges`; `TestVersionedArmedForOffline`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the touched packages' targeted tests, then the root-read census. |
| -- [ ] **Step 5: Commit** `boxes: participant identity, contextual report validation, the lane, the source precheck with stray kinds, target-ready, staged receipts and the per-host status block`. |
| +- [ ] **Step 5: Commit** `boxes: participant identity by box and instance, contextual report validation with a freshness table, the lane, the source precheck, target-ready, a move-bound stage book from observed copies, and the per-host status block`. |
| ### Task 7: The job, from requested to ready, and its reconciliation |
| @@ -146,59 +160,84 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 8) |
| - Test: `internal/e2e/selfserve/job_test.go` |
| **Interfaces:** |
| -- `selfServeApply(ctx, t, row, mut, want func(provision.Record) bool) (*provision.Record, uint64, error)`: the lease-bounded context and the write guard; on an unknown outcome re-read (judged) and `want`; after a landed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq` (at-least-once; a duplicate bump is harmless). |
| -- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` sets `expiry_pending` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets: the row's phase not in the record's set → the set's first phase (`AdvanceSelfServe` without the abort predicate for `opened`); (4) the expiry action by the record: `expiry_pending` with the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Request handling in every issuing phase (`staging` from `reports`, `restage`, `precheck`, the barrier): `IssueSelfServeRequest` → `selfServeApply` → the committed stamp read from the returned record → `ConfirmSelfServeRequest(committed)` → the wait is for a report with `Request`/`StageRequest == committed` → `ConsumeSelfServeRequest` when used. A crash between the write and the confirmation: the next tick's reconciliation sees `request_issued` set and unconfirmed, re-reads the record, confirms from its current stamp (the write's) and continues; a request is never re-issued while unconsumed. |
| -- The admission write: `request_issued` for kind `queue`; a response lost → re-read; after the 30 s bound with no move: `fenceRecord(ctx, acct, app, host)` (plan 1) **before** the row is voided and the reservation released, so the delayed PUT cannot land afterwards; a fence failure keeps the row pending. |
| -- Phases as rev 3 with the identity check `participantStillPresent` at every gate (the status report of each pinned label, fresh within 10 minutes) → "box <label> was replaced" → abort before open, page after. |
| - |
| -- [ ] **Step 1: Write the failing tests** on the harness: `TestEveryBoundaryRecovers` (every transition in Review Focus 1, with repeated crashes; stable request identities; deadlines preserved; at least one bump); `TestReconcileForwardNeverRegresses` (health and finishing survive ticks with the record at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed` (prechecks, restage, barrier, and the first staging request); `TestCommittedStampIsTheRequestIdentity` (an `OfflineRestage` that bumped by a nanosecond is confirmed with the committed value and the report matches); `TestAbandonedAdmissionIsFenced` (a queue PUT committing server-side after the client timed out: the fence runs first and the late PUT fails CAS; the reservation is released only after); `TestReplacedParticipantAtEveryGate` (after prechecks; during abort); rev 3's remaining tests. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileForward|RequestsAreNever|CommittedStamp|AbandonedAdmission|ReplacedParticipant|Reserving|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, reconciling forward, with request lifecycles, at-least-once notification and a fenced admission`. |
| +- `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, req request, mut func(at time.Time) func(*provision.Record) error, confirmed func(provision.Record) bool) (*provision.Record, error)` with `type request struct { Kind, Phase string }`: (1) the judged record → `expectSeq = rec.RecordSeq`; `proposed = max(now, latest stamp on the record + 1 ns)`; `nonce = random`; `IssueSelfServeRequest(id, kind, phase, nonce, expectSeq, proposed, now)` (false → a request is outstanding: go to (3)); (2) `provision.SetRecordMove(ctx', mut(proposed))` with the lease-bounded context and the write guard; (3) **confirmation** from a fresh judged read, whatever (2) returned: `confirmed(rec)` (the record carries `proposed` in the field this request sets) → `ConfirmSelfServeRequest(id, nonce, now)`; else `rec.RecordSeq == expectSeq` (unchanged) → the write did not land: repeat (2) with the same `proposed` (idempotent replay; bounded by the lease); else (changed by another writer) → `VoidSelfServeRequest(id, nonce)` and return `errRequestVoided` (the phase function decides again next tick); after a confirmed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq`. A crash anywhere resumes at (3) on the next tick from the persisted request. |
| +- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` → `SetSelfServeExpiry(id, phase, true)` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets (Task 4): the row's phase not in the record's set → `AdvanceSelfServe` to the set's first phase (without the abort predicate for `opened`), which clears a foreign expiry; an unconfirmed request → step (3) of `selfServeApply` first; (4) the expiry action for an `expiry_pending` whose `expiry_phase == phase`: the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; `health` → warning, page, `finishing`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Every wait for a report is for `Request`/`StagedRequest == request_proposed` of the **confirmed** request; the report is consumed by the `AdvanceSelfServe` that leaves the waiting phase, with the nonce. |
| +- **Reconciliation and recovery table** (the record is the truth): |
| + |
| +| row phase | record | action | |
| +|---|---|---| |
| +| `requested`, `reserving` | no move | continue the phase | |
| +| `reserving` with an unconfirmed `queue` request | no move with this id | `selfServeApply` step (3): confirmed → `queued`; unchanged seq → replay; changed → void; a request older than 30 s whose observation shows no move → **the fence**: `fenceRecord(ctx, acct, app, host, fenceOwner{})`: `fenced`/`absent` with `obs.Rec.Move.ID == row.ID` → the queue landed after all: confirm and continue; else → persist `outcome = aborted` "the move could not be started" through the finaliser's step 1 **before** `releaseBoxes`; an error → pending | |
| +| `reserving`, `queued` | `offline-queued` with this id | `queued` → `reports` | |
| +| `queued`, `reports`, `staging(pending)` | `offline-queued` | continue | |
| +| any pre-cut phase | `offline-staging` | `staging` unless the row is already in the set | |
| +| any pre-cut phase | `offline-stopping` | `stopping` (an operator started it) | |
| +| any | `offline-sealing`/`sealed`, `landing`, `landed`, `open`, `finishing`, `reverting` | the set's first phase unless already in the set | |
| +| any | no move and `LastMove.ID == row.ID` | the finaliser with `LastMove.Outcome` | |
| +| any | another move | finalise `aborted` "another move is in flight" without touching the record | |
| +| any | absent or stale (judged) | finalise `aborted` "the instance was recreated" without touching the record | |
| + |
| +- Phases: |
| + - `requested` → `reserving`. |
| + - `reserving`: `ParticipantStillPresent` for every pinned participant (else abort "box <label> was replaced"); `reserveBoxes(sel, {move, row.ID, ...})`; busy → `reason` "waiting for another move on this box"; `WindowDeadline` → abort "the move was not started in time"; reserved → the fresh judged record → `recordEligibility`, the run and move checks, `offlineRecordChecks`, `offlineRegionChecks(..., stage: true, requireIDs: true, "")` again (a refusal → abort with its sentence, release) → `selfServeApply(queue, OfflineQueue(...), confirmed: Move != nil && Move.ID == row.ID && QueuedAt == proposed)` → `queued`. |
| + - `queued` → `reports` (5 min → abort). |
| + - `reports`: the `lane` (`OK`), the source `precheck` and the target `ready` with `Request == PrecheckAt`, each from the pinned participant; `targetReadyDecision` and `sourcePrecheckDecision` (the console enumerates the required path set and fact set itself: a missing entry or `!Known` → "could not determine <fact> on <label>"; not OK → the fact's sentence: "the target box already has a route for <host>", "the target box has timers for <host>", "the target box has a stale live link for <host> naming <build>", "the target box's programs file names <build>", "the target box already holds data for <host>: <path>", "the target box's park directory is not clean", "the target box has too little free disk for <host>" (`!Known` or `< 2 × replica`), "the target bucket already holds objects for <host>", "the target bucket holds a lock for <host> (<owner>)", "<host> has a stray entry beside its files directory: <path>", "<host>'s files cannot be archived: <detail>"; before staging both prefixes must be `Known` and empty); both pass → `selfServeApply(stage, OfflineQueuedToStaging, confirmed: StagedAt == proposed)`, `StagingStartedAt = now`, → `staging`; 5 min → abort. |
| + - `staging`: the stager report with `StagedRequest == row.StageRequest`: the ready rule `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `selfServeApply(stage, OfflineRestage, confirmed: StagedAt == proposed)` → a new wait; a `failed` report → abort with its `Detail`; a round past `max(10 min, 3 × est.Copy)` → abort; `now - StagingStartedAt > 30 min` or `Rounds >= 10` → `prechecks` from the last completed round. |
| + - `prechecks`: `selfServeApply(precheck, OfflinePrecheck, confirmed: PrecheckAt == proposed)`; both participants' reports answering it; the staged-ownership comparison (Task 6); `InstancesPrefix` known and empty; both pass → `ready`, `WindowDeadline = now + 24 h`, the estimate refined and `OfflineSetEstimate` (a later disagreement resolves towards the row); 5 min → abort. |
| + - `ready`: every 10 min a restage; `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| +- The pre-offline abort (requested, reserving, queued, staging): no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`); queued → `selfServeApply(abort, OfflineAbortAt, confirmed: Move == nil && LastMove.ID == row.ID)`; staging → `OfflineAbort` → the stager's `reverted` → `OfflineFinish`; then the finaliser. |
| + |
| +- [ ] **Step 1: Write the failing tests** on the bridge of Task 10: `TestEveryBoundaryRecovers` (Review Focus 1: for every write, a crash before the write with the request persisted, a lost response, a crash before confirmation, a crash before the bump, a second crash during recovery, SQLite reopened each time; a fake store that **commits after the context was cancelled**: the replay sees the proposed stamp and confirms, no second write; one request per wait, one bump per landed write; ordinary multi-tick progression through start and flip); `TestReconcileForwardNeverRegresses` (every row of the table; `stopping(pending)` with the record still `offline-staging` stays; `landing(pending)` with `offline-sealed` stays; health and finishing survive ticks at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed`; `TestStaleEvidenceNeverSatisfiesANewRequest` (an issued-but-unwritten restage: the previous round's report does not satisfy it; after the crash the replay lands the proposed stamp and only a report carrying it does); `TestAbandonedAdmissionIsFencedAndConditional` (a queue PUT committing after a negative re-read: the fence observes the move and the job continues as `queued`; a PUT that never lands: the fence succeeds, the row is voided durably before the release); `TestReplacedParticipantAtEveryGate`; `TestReservingWaitsThenAborts`; `TestEligibilityRechecksBeforeQueue`; `TestReportsDecideFromFacts`; `TestStagingReadyRule`; `TestPrechecksCompareTheBook`; `TestReadyRestagesAndExpires`; `TestDeadlinesEvaluateBeforeEvidence`; `TestAbortBeforeOfflineIsLossless`; `TestQueueOrderAndDisjointProgress` (three rows: two on one box in order, a third on disjoint boxes advances while the second waits). |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test -tags e2e ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileForward|RequestsAreNever|StaleEvidence|AbandonedAdmission|ReplacedParticipant|Reserving|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline|QueueOrder' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, reconciling forward with persisted requests, a conditional fenced abandonment and at-least-once notification`. |
| ### Task 8: The cut, abort, finalisation, paging and the operator return |
| **Files:** |
| -- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (every clearing operator door finalises a self-serve move; the open door writes the open record; a new door `POST .../move/offline/return {"region","return_of"}`) |
| +- Modify: `internal/console/selfserve_job.go`, `internal/console/apiinstancemove_offline.go` (every clearing operator door finalises a self-serve move; the open door writes the open record before the transition; a new door `POST .../move/offline/return`) |
| - Create: `internal/console/selfserve_health.go`, `internal/console/selfserve_pages.go`, `internal/console/selfserve_finalise.go`, `internal/console/offlinereturn.go` |
| - Test: `internal/e2e/selfserve/cut_test.go`, `internal/console/offlinereturn_test.go` |
| **Interfaces:** |
| -- Phases and deadlines as rev 3, with: `landed`'s expiry decided by the record (Global Constraints); the open record written after the committed open; `health`: `health_attempts` incremented on every probe, `health_consecutive_ok` reset on failure; the finaliser as rev 3 (outcome, effects, release, clear). |
| -- **The operator return handshake**: (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): validates the forward: `OpenRecordKey(host, return_of)` exists with `To == rec.Region` and `From == region` (the return goes back to the forward's source), the record carries no move; writes `control/offline-moves/<host>/<return_of>/return/<console id>.json` `{"requested_at", "region"}` create-only and answers 202 "return requested; the target box will report its leftovers"; (2) the target box of the return (the box in `region`, region-based, operator path) sees the request on its moves pass and writes a `ready` report for role `target`, phase `ready`, `Request = requested_at`, with `Leftovers` filled against the open record's `OpenedAt`, without touching the record; (3) the operator's `stage` (or `start`) door with `return_of` requires that report (fresh, `Request` equal) and runs `targetReadyDecision` in return mode: accepted exactly: `Lock` present, owner `TombstoneOwner(return_of)`, lifted; `InstancesPrefix.Objects` current versions excluding `lock.json` all with `LastModified < OpenedAt`; `DataPaths["/data/.instances/<host>"]` and `["/data/instances/<host>"]` present and empty (`Detail: "empty"`), the mount view likewise; park dirs `<return_of>/source/` and `/sealer/` present; `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\` }`. |
| -- Paging as rev 3 with persisted attempts; the orphan scan each tick and at start. |
| +- Phases and deadlines (`est` the row's terms): `restage`: `selfServeApply(stage, OfflineRestage)`; the matching `staged`; `max(5 min, 2 × est.Copy)`; expiry → abort → `barrier`. `barrier`: `selfServeApply(precheck, OfflinePrecheck)`; both participants' reports passing the decisions and the book comparison with zero deferrals; `max(5 min, 2 × est.Copy)`; a failed fact or expiry → abort → `stopping`. `stopping`: `offlineRecordChecks` + `offlineRegionChecks(..., stage: false, requireIDs: true, "")` on the fresh record, then `selfServeApply(start, OfflineStartStaged, confirmed: State == offline-stopping && SelfServe.OfflineAt == proposed)`; then the seal check with participants → `selfServeApply(seal, OfflineSeal, confirmed: State == offline-sealing)`; `max(5 min, 2 × est.Stop)`; expiry → abort. `sealing`: the sealer's `sealed`; `max(10 min, 3 × est.Copy)`; expiry → abort. `landing`: `offlineFlipCheck` → `selfServeApply(flip, OfflineFlip, confirmed: State == offline-landing)`; the pinned target's `landed`; `max(10 min, 3 × est.Land)`; expiry → abort. `landed`: `offlineOpenCheck` → write the open record (create-only, `OpenedAt = proposed`; `open_record_pending = 1` until it exists) → `selfServeApply(open, OfflineOpenTransition, confirmed: State == offline-open && OpenedAt == proposed)`; a refusal or `max(3 min, 2 × est.Open)` **while the record is `offline-landed`** → abort; an unconfirmable write (the record unreadable) → `open-uncertain`. `open-uncertain`: re-read each tick: `offline-open` → `opened`; `offline-landed` → `landed` (the deadline continues); 5 min unreadable → page, stay. `opened`: the target's `opened`; `max(3 min, 2 × est.Open)`; expiry → blocked: page, retry. `health`: a probe when `now - HealthLastProbeAt >= 30 s` (`probeTarget`: the target box's `EIP` from its edges file, TLS with the host as SNI and `Host`, `GET <ReadyPathOrDefault>?carlos-probe=<random>`, `Cache-Control: no-cache`, no redirects, 30 s; OK when 2xx and `x-carlos-version` equals the sealer report's `Release`; plan 3 adds `X-Carlos-Probe`/`X-Carlos-Served-By`); `RecordSelfServeHealth`; `HealthConsecutiveOK >= 2` → `finishing`; expiry `max(3 min, 2 × est.Promote) + 30 s` → `SetSelfServeWarning("the account did not answer on <region> within <n> minutes")`, page, `finishing`. `finishing`: gate 1 (`offlineFinishCheck` opened + every pinned source parked) → `selfServeApply(finish1, OfflineFinishingTransition)`, `FinishGate = lifted`; gate 2 (lifted + every pinned source gone) → `selfServeApply(finish, OfflineFinish, confirmed: Move == nil && LastMove.ID == row.ID)`; 10 min each → blocked: page, retry. `reverting`: `OfflineAbort` once (or `OfflineAbortAt` from queued); `SourceReopenedAt` when (no tombstone, or the sealer's `reverting`/`reverted` with `TombstoneLifted`) **and** every pinned source's `reverted`; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes → `OfflineFinish`; **the target is never required**: a target that stopped reporting after landing does not block the source's reopening; the abort's `Detail` records "target <label> did not report; its staged copy is left for cleanup"; 10 min → blocked: page, retry. |
| +- **The finaliser** `selfServeFinalise(ctx, t, row, outcome, reason, warning)`: (1) `FinishSelfServeMove` (outcome; `release_pending = 1`, `effects_done = 0`); (2) the terminal effects: the timings (Task 9), the terminal event, `bumpSerialErr`; `SetSelfServeEffectsDone`; (3) `releaseBoxes(move, row.ID)`; (4) `ClearSelfServeRelease`. A crash between any two leaves the row in `OpenSelfServeMoves` and the next tick resumes at the first unfinished step. The operator doors that clear a self-serve move's record route through it. |
| +- **The operator return handshake (§7.5):** (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): the open record at `OpenRecordKey(host, return_of)` must exist with `To == rec.Region`, `From == region`, `Incarnation == rec.Incarnation`, and the record carries no move; else 409 "no completed move <id> from <region> to <rec.Region> for this instance"; writes `control/offline-moves/<host>/<return_of>/return/<requested_at unix nanos>-<console id>.json` `{"requested_at", "region", "console"}` create-only (a new call makes a new request; the **newest** `requested_at` is the current request; older ones are ignored) and answers 202 "return requested; the target box will report its leftovers". (2) The moves pass of every box in `region` lists `control/offline-moves/<host>/*/return/` for hosts whose open record names `From == its region` and, when the open record carries `Participants`, whose `Source` includes its own `BoxID` (an id-less forward falls to the region rule); it reads the newest request, fills `Leftovers` against the open record (`OpenedAt`, `Layout`, `MoveID`) and writes `control/offline-moves/<host>/<return_of>/reports/return-ready-<box id>.json` (`Op = return_of`, role `target`, phase `ready`, `Request = requested_at`, `BoxID/InstanceID`), without touching the record; it re-reports whenever a newer request appears. (3) The operator's `stage` (or `start`) door with `return_of`: `readReturnReport(ctx, host, returnOf)` must find a report with `Request` equal to the newest request's `requested_at`, within 10 minutes, from a box whose `BoxID` is a forward source participant (or in `From` for an id-less forward); then `targetReadyDecision` in return mode accepts **exactly**: `Lock` present with owner `moves.TombstoneOwner(return_of)` and lifted; `InstancesPrefix.Objects` (current versions, `lock.json` excluded) all with `LastModified < OpenedAt` (any newer → `NewerObjects`, refused); `DataPaths["/data/instances/<host>"]` absent or present-and-empty; for `Layout == nested` also `DataPaths["/data/.instances/<host>"]` absent or present-and-empty, and the mount view likewise; for `flat` the nested pair must be absent; park directories: each present one must be `<return_of>/source/` or `<return_of>/sealer/` (absence allowed: cleanup may have removed them; any other → refused); `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\`; Layout string \`json:"layout"\` }`. |
| +- Paging: `alerting.Page` through `MovePageAttempt`/`SetMovePageAttempt` (seconds encoding, unchanged) with keys `selfserve:<id>:<condition>` every 10 minutes for: `opened` past its deadline; a health failure; `finishing`/`reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; an open record still pending after 5 minutes; a reservation the reclaim rule cannot reclaim older than the grace period (the per-tick scan, also at start); a lane probe failing on a configured pair; the attempt is recorded **before** the page is sent, as the existing paging does. |
| +- Copy rows: "the account did not answer on <region> within <n> minutes"; "the move was not started in time"; "the move was not confirmed in time"; "another move is in flight"; "box <label> was replaced"; "the move could not be started"; "the instance was recreated"; "target <label> did not report; its staged copy is left for cleanup"; "no completed move <id> from <region> to <rec.Region> for this instance"; "return requested; the target box will report its leftovers"; the fact refusals of Task 7. |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestLandedExpiryDefersToTheRecord` (restart after a committed open and an old `landed` deadline → `opened`, no abort); `TestHealthAttemptsPersist`; `TestOperatorReturnHandshake` (the return door, the target's leftovers report, the stage door accepting exactly the forward's leftovers; an object written after the committed open but before the report refreshed → refused; a direct start with `return_of` → the same acceptance; a `return_of` whose open record is absent → 409). |
| +- [ ] **Step 1: Write the failing tests** on the bridge: `TestCutRunsEveryPhaseToDone`; `TestEveryDeadlineBeforeCommittedOpenAborts`; `TestLandedExpiryDefersToTheRecord` (restart after a committed open with an old `landed` expiry → `opened`, no abort; the expiry is cleared and the move runs to `done`); `TestHealthExpiryTakesPrecedence`; `TestOpenUncertainIsReconciledByTheRecord`; `TestOpenRecordPrecedesTheOpen` (a crash between the open record and the transition; a crash after; both recover); `TestAbortIsInitiatedNotCompleted` (the real sealer; Review Focus 3; the target disappears after landing → the source reopens, the detail names the target); `TestOpenedBlockedPagesAndRetries`; `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthAttemptsPersist`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep`; `TestOperatorDoorsFinaliseASelfServeMove`; `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRendersAfterRestart`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap`; `TestOperatorReturnHandshake` (the return door; the box's report through its real moves pass; a second request supersedes the first and the box re-reports; the stage door accepting exactly the forward's leftovers for a flat and a nested forward, fully and partially cleaned; a newer object, a live lock, a foreign park dir, a non-empty nested dir, a report from a non-participant box, an open record of another lifetime, a `return_of` whose open record is absent, each refuse; a direct start with `return_of` → the same acceptance; restart at every handshake boundary). |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the targeted tests, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the self-serve cut with the no-return boundary at the committed open, the abort facts, one finaliser, durable paging and the operator return handshake`. |
| +- [ ] **Step 5: Commit** `console: the self-serve cut with the no-return boundary at the committed open, the abort facts without the target, one finaliser, durable paging and a recoverable operator return handshake`. |
| ### Task 9: The estimate, timings and the events table |
| **Files:** |
| - Create: `internal/console/selfserve_estimate.go`, `internal/console/selfserve_events.go` |
| -- Modify: `internal/console/store.go` (`move_timings`; `self_serve_timings` as Task 4), `internal/console/selfserve_finalise.go`, `internal/console/apiinstancemove_offline.go` (`offlineView.SelfServe`, `?log=1`) |
| +- Modify: `internal/console/store.go` (`move_timings (id TEXT, phase TEXT, from_region TEXT, to_region TEXT, bytes INTEGER, seconds REAL, at INTEGER, PRIMARY KEY (id, phase))`; `self_serve_timings` as Task 4), `internal/console/selfserve_finalise.go`, `internal/console/apiinstancemove_offline.go` (`offlineView.SelfServe *selfServeMoveView \`json:"self_serve,omitempty"\``; `GET .../move/offline?log=1` adds `Events []selfServeEventView`) |
| - Test: `internal/console/selfserve_estimate_test.go`, `internal/console/selfserve_events_test.go` |
| **Interfaces:** |
| -- `estimateTerms`, `term`, the defaults (`Stop {20 s, 0}`, `Copy {10 s, 25 MiB/s}`, `Land {30 s, 40 MiB/s}`, `Open {30 s, 0}`, `Promote {10 s, 70 MiB/s}`), `estimate(from, to string, remaining, files, replica int64) (estimateTerms, int)` (rate terms from medians of `bytes/seconds` over rows with `bytes > 0`, fixed terms from medians of `seconds`, 5 rows minimum, whole minutes, minimum 2); the timing marks are **console observation times**, persisted on first observation (`MarkSelfServeTiming` is insert-only), never overwritten; a move that aborted or lacks a mark contributes no row for that phase. |
| -- `self_serve_events` with the account/app/host columns; `drainSelfServeEvents` each tick through `DeliverSelfServeEvent`; the pass starts while undelivered events exist (Task 4's mode). |
| +- `type estimateTerms struct { Stop, Copy, Land, Open, Promote time.Duration }`; `type term struct { Fixed time.Duration; BytesPerSecond float64 }`; defaults `Stop {20 s, 0}`, `Copy {10 s, 25 MiB/s}`, `Land {30 s, 40 MiB/s}`, `Open {30 s, 0}`, `Promote {10 s, 70 MiB/s}`; `func (s *Server) estimate(from, to string, remaining, files, replica int64) (estimateTerms, int)` (rate terms from medians of `bytes/seconds` over rows with `bytes > 0`, fixed terms from medians of `seconds`, 5 rows minimum, whole minutes, minimum 2). |
| +- Timing marks are **console observation times**, insert-only (`MarkSelfServeTiming(id, mark string, at time.Time, bytes int64) error`), never overwritten: `offline_at`, `stopped_at`, `seal_at`, `sealed_at` (+ `FinalCopiedBytes`), `flip_at`, `landed_at` (+ files and replica bytes), `open_at`, `opened_at`, `first_ok_at`; the finaliser derives the five `move_timings` rows (`stop` = `offline_at → stopped_at`; `copy` = `seal_at → sealed_at`, bytes `FinalCopiedBytes`; `land` = `flip_at → landed_at`; `open` = `open_at → opened_at`; `promote` = `open_at → first_ok_at`, bytes replica); a move that aborted or lacks a mark contributes no row for that phase. `SelfServeTimings(id string) (map[string]timingMark, error)`, `InsertMoveTiming`, `MoveTimings`. |
| +- `type SelfServeEvent struct { ID string; Seq int64; Phase string; Account, App, Host string; At time.Time; Reason, Detail string; Emitted bool }`; `type selfServeEventView struct { Phase string \`json:"phase"\`; At int64 \`json:"at"\`; Reason string \`json:"reason,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`. Every phase change, refusal, fact decision, lane probe, health probe and timing inserts an event in the transaction of the change that caused it (`Phase` carries a suffix for non-phase events: `reports:lane`, `health:probe`). `drainSelfServeEvents` each tick: `DeliverSelfServeEvent(ctx, e, func(tx) error { return auditActivityTx(tx, acct, app, "self-serve move", Activity{Action: "instance-move", TargetType: "instance", Target: host, Message: phase, Detail: reason}) })`; the pass starts while undelivered events exist (Task 4's mode). |
| -- [ ] **Step 1: Write the failing tests**: rev 3's plus `TestTimingMarksAreFirstObservationTimes`; `TestAbortedMovesContributeNoRates`; `TestUndeliveredEventsStartThePass`. |
| +- [ ] **Step 1: Write the failing tests**: `TestEstimateUsesDefaultsThenMedians`; `TestEstimateMinimumIsTwoMinutes`; `TestEstimateBeforeAndAfterStaging`; `TestTimingMarksAreFirstObservationTimes`; `TestAbortedMovesContributeNoRates`; `TestEventsAreSequencedAndDeliveredOnce` (repeated prechecks → distinct events; a delivery failure leaves the event undelivered; a crash after the audit insert does not duplicate); `TestUndeliveredEventsStartThePass`; `TestOfflineStatusShowsSelfServeAndLog`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Estimate|Timing|Events|OfflineStatusShows' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the move estimate from first-observation timings; transactional events as the claims-off move log and member stream`. |
| +- [ ] **Step 5: Commit** `console: the move estimate from first-observation timings; sequenced transactional events as the claims-off move log and member stream`. |
| -### Task 10: The exported harness and the end-to-end witnesses |
| +### Task 10: The exported bridge and the end-to-end witnesses |
| **Files:** |
| -- Create: `internal/host/hosttest/harness.go` (a real package: the two-box fixtures **moved** out of `movesoffline_e2e_test.go` into exported constructors (`New`, `Boxes.Pass`, `Boxes.Serial`, `Boxes.Clock`), with `TestOfflineMoveE2E` rewritten to use it, behind no build tag so `go vet` covers it on every platform; the fixtures that need Linux stay behind `//go:build linux` with a stub that skips), `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}` exported for the e2e package), `internal/e2e/selfserve/*_test.go` (built with `-tags e2e`; CI runs it) |
| -- Test: the witnesses of Tasks 7 and 8 and rev 3's end-to-end list, plus `TestSelfServeThenOperatorReturn`. |
| +- Create: `internal/host/e2e_bridge.go` (**in package `host`**, `//go:build e2e`: `type E2E struct { ... }` with `func NewE2E(t testing.TB, store release.Store) *E2E`, `(*E2E).Pass(ctx)` (one moves pass on every box), `(*E2E).Serial(ctx)` (passes until the serial is consumed), `(*E2E).Clock() *fakeClock`, `(*E2E).Fault(point string, f func())` (the fault points the tests name), `(*E2E).Box(label) BoxHandle` exposing the box id, instance id, registry, data bucket and the worker's `armOffline`/`reconcileSerializedWith` through methods; it reaches `movesWorker`, `moveCfg`, `newMovesWorker`, `armOffline`, `reconcileSerializedWith` directly because it is in the package, so nothing private moves and no import cycle arises; the Linux-only fixtures stay behind `//go:build linux && e2e` with a stub that skips), `internal/host/e2e_bridge_compile.go` (no tag: a one-line reference that keeps `go vet` honest about the exported names on every platform), `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}`), `internal/e2e/selfserve/*_test.go` (`//go:build e2e`), `.amadan/test-gate.sh` (a new step `go test -tags e2e ./internal/e2e/...` after the ordinary suite, plus `go test -tags e2e -list . ./internal/e2e/selfserve | grep -q '^Test'` so an empty discovery fails the gate) |
| +- Test: the witnesses of Tasks 7 and 8, plus `TestSelfServeForwardMove`; `TestSelfServeCancelInReady`; `TestSelfServeAbortByFailedPrecheck`; `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal`; `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord`; `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchangedOnTheBridge`; `TestSelfServeThenOperatorReturn`; `TestClaimsOffInstanceIDIsPublished` (the agent's status tick carries `InstanceID` and `BoxID` on a claims-off box). |
| -- [ ] **Step 1: Extract the harness and write the tests.** **Step 2: Run them; fix what they find.** **Step 3: Both census tests and the cross-builds**; the census note. **Step 4: Commit** `self-serve moves: an exported two-box harness and the end-to-end witnesses`; push. |
| +- [ ] **Step 1: Write the bridge and the tests.** **Step 2: Run them with `-tags e2e`; fix what they find within this plan's scope.** **Step 3: Both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-2.md` (the overlay's loader block, the changed envelopes, the new writers incl. the return request, the open record, the console lease and `active.json`'s table form). **Step 4: Commit** `self-serve moves: an exported two-box bridge inside package host and the end-to-end witnesses, run by the gate`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage as rev 3, with §7.5's operator return now a handshake (T2's open record, T8's return door and report). |
| -- Dependencies for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}`, `Participants.Target.BoxID`, `moveCfg.BoxID`; for plan 4: the status `Retirements` contract (`RetirementsComplete`, `State == ended`, `Metrics.BoxID`) and the open record. |
| -- Names used across tasks: rev 3's plus `selfServeMode`, `offlineRecordChecks`, `offlineRegionChecks`, `Participant.InstanceID/BootID/Verified`, `participantStillPresent`, `moves.OpenRecord`, `OpenRecordKey`, `lease.ReadReceipts`, `CopyReceipt`, `PrefixObject`, `StrayEntry`, `StraysFact`, `ValidateOfflineReportFor`, `IssueSelfServeRequest/ConfirmSelfServeRequest/ConsumeSelfServeRequest`, `notified_seq`, `expiry_pending`, `bumpSerialErr`, `provision.WithWriteGuard`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `console.TestHooks`, `hosttest.New`. |
| +- Spec coverage: §5 (T1), §6.1 (T5), §6.2 (T7), §6.3 (T9, T5, T6), §7 admission and recovery (T7's table), §7.1 (T3), §7.2 and the three facts with the sealer's durable lift (T6, T8), §7.3 (T6, T7, T8's barrier), §7.4 without the probe mode (T8), §7.5 the no-return boundary and the operator return (T2's open record, T8), §7.7 (T5), §8 pages, log and member stream (T8, T9), §9 (T3's rollout). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup (plan 4), whose status contract the reclaim rule names (`RetirementsComplete`, a terminal state, `RootRequest == ""`, `ReleasePending == false`, `Metrics.BoxID`). |
| +- Dependencies for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}`, `Participants.Target.BoxID`, `moveCfg.BoxID`, `probeTarget`; for plan 4: the status `Retirements` contract and the open record. |
| +- Names used across tasks: `selfServeGate`, `instanceRails`, `moveCapable`, `Principal.Moves/TokenID`, `selfServeMode`, `selfServeAdmits`, `recordEligibility`, `offlineRecordChecks`, `offlineRegionChecks`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant{BoxID, Label, InstanceID, Verified}`, `Participants`, `moves.OpenRecord{..., Layout}`, `OpenRecordKey`, `moves.ParticipantRole`, `moves.ParticipantStillPresent`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `request`, `errRequestVoided`, `selfServeFinalise`, `leaseToken`, `provision.WithWriteGuard`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `VoidSelfServeRequest`, `AdvanceSelfServe(..., nonce)`, `SetSelfServeExpiry`, `SetSelfServeNotifiedSeq`, `SetSelfServeOpenRecordPending`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `PrefixObject`, `StagedObject{Size, ETag, VersionID, Deferred}`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck{..., Layout}`, `StrayEntry`, `StraysFact`, `ReturnLeftovers`, `ValidateOfflineReportFor`, `readOfflineEvidence(ctx, host, id, m)`, `readReturnReport`, `lease.CopyEvent`, `lease.MirrorPrefixKeepObserved`, `stageBook`, `status.HostFacts`, `BoxMetrics.Hosts/Lanes/BoxID/InstanceID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MarkSelfServeTiming`, `SelfServeEvent`, `host.E2E`/`NewE2E`, `console.TestHooks`. |
| - Review Focus 1 to 5 are pinned by T7, T3, T8, T5, T1. |