amadan
plan: self-serve moves 2, rev 7 (Astra review 6: a complete request lifecycle with resume and same-phase consume, incarnation in the bound write, the cut as a handoff, classified expiry fence, once-only events, per-box FIFO, return request identity and target binding, fixtures on a linux-only file)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+28
−28
docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md+28 −28
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.017s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (151 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.908s
ok github.com/carlosframework/platform/cmd/moveprobe 12.111s
ok github.com/carlosframework/platform/cmd/zddprobe 0.707s
ok github.com/carlosframework/platform/internal/activator 61.024s
ok github.com/carlosframework/platform/internal/agentsock 0.055s
ok github.com/carlosframework/platform/internal/alerting 0.008s
ok github.com/carlosframework/platform/internal/analytics 0.024s
ok github.com/carlosframework/platform/internal/appconfig 3.074s
ok github.com/carlosframework/platform/internal/apperr 0.131s
ok github.com/carlosframework/platform/internal/applog 0.229s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 0.734s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.009s
ok github.com/carlosframework/platform/internal/caps 0.032s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.797s
ok github.com/carlosframework/platform/internal/changelog 0.501s
ok github.com/carlosframework/platform/internal/claimactivation 1.272s
ok github.com/carlosframework/platform/internal/clamd 0.058s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.460s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.012s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.009s
ok github.com/carlosframework/platform/internal/domains 3.073s
ok github.com/carlosframework/platform/internal/doorkey 0.010s
ok github.com/carlosframework/platform/internal/economics 0.006s
ok github.com/carlosframework/platform/internal/edge 46.595s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.009s
ok github.com/carlosframework/platform/internal/fleetapi 0.206s
ok github.com/carlosframework/platform/internal/gate 2.044s
ok github.com/carlosframework/platform/internal/genselect 0.019s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.042s
ok github.com/carlosframework/platform/internal/historyreadback 0.266s
ok github.com/carlosframework/platform/internal/host 147.245s
ok github.com/carlosframework/platform/internal/hostactors 7.294s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.010s
ok github.com/carlosframework/platform/internal/hostbox 3.369s
ok github.com/carlosframework/platform/internal/hostclaim 0.884s
ok github.com/carlosframework/platform/internal/hostenv 0.006s
ok github.com/carlosframework/platform/internal/hostevidence 0.007s
ok github.com/carlosframework/platform/internal/hostexec 6.620s
ok github.com/carlosframework/platform/internal/hostfence 0.084s
ok github.com/carlosframework/platform/internal/hostfiles 11.129s
ok github.com/carlosframework/platform/internal/hostfloor 0.016s
ok github.com/carlosframework/platform/internal/hostguard 1.110s
ok github.com/carlosframework/platform/internal/hostops 2.413s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.004s
ok github.com/carlosframework/platform/internal/hostquarantine 0.003s
ok github.com/carlosframework/platform/internal/hostrelocate 0.810s
ok github.com/carlosframework/platform/internal/hostresolve 0.266s
ok github.com/carlosframework/platform/internal/hostwrite 0.008s
ok github.com/carlosframework/platform/internal/infraview 0.006s
ok github.com/carlosframework/platform/internal/instancectl 5.287s
ok github.com/carlosframework/platform/internal/instanceexec 2.874s
ok github.com/carlosframework/platform/internal/instancefs 0.013s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.007s
ok github.com/carlosframework/platform/internal/layout 2.022s
ok github.com/carlosframework/platform/internal/lbhealth 0.005s
ok github.com/carlosframework/platform/internal/lease 10.514s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.816s
ok github.com/carlosframework/platform/internal/ledger 0.017s
ok github.com/carlosframework/platform/internal/mail 0.296s
ok github.com/carlosframework/platform/internal/maintenance 0.161s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.902s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.227s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 23.965s
ok github.com/carlosframework/platform/internal/moves/archive 1.320s
ok github.com/carlosframework/platform/internal/objstore 0.354s
ok github.com/carlosframework/platform/internal/ops 0.192s
ok github.com/carlosframework/platform/internal/park 0.270s
ok github.com/carlosframework/platform/internal/plans 0.005s
ok github.com/carlosframework/platform/internal/privatefile 0.003s
ok github.com/carlosframework/platform/internal/provision 19.583s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 6.252s
ok github.com/carlosframework/platform/internal/relay 7.310s
ok github.com/carlosframework/platform/internal/release 29.935s
ok github.com/carlosframework/platform/internal/rollout 4.205s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.062s
ok github.com/carlosframework/platform/internal/rootstate 0.032s
ok github.com/carlosframework/platform/internal/runs 0.569s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.005s
ok github.com/carlosframework/platform/internal/schedule 2.831s
ok github.com/carlosframework/platform/internal/seeds 0.014s
ok github.com/carlosframework/platform/internal/sim/hostlife 22.404s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.131s
ok github.com/carlosframework/platform/internal/status 0.118s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.194s
ok github.com/carlosframework/platform/internal/svcrole 0.003s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.007s
ok github.com/carlosframework/platform/internal/tunnelclient 0.007s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.009s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.003s
ok github.com/carlosframework/platform/internal/vet 1.705s
=== internal_console@0 (81 s)
ok github.com/carlosframework/platform/internal/console 78.649s
=== internal_console@1 (40 s)
ok github.com/carlosframework/platform/internal/console 38.758s
=== internal_console@2 (46 s)
ok github.com/carlosframework/platform/internal/console 44.266s
=== internal_console@3 (42 s)
ok github.com/carlosframework/platform/internal/console 39.734s
=== internal_console@4 (41 s)
ok github.com/carlosframework/platform/internal/console 39.240s
=== internal_console@5 (40 s)
ok github.com/carlosframework/platform/internal/console 37.719s
=== internal_console@6 (40 s)
ok github.com/carlosframework/platform/internal/console 38.053s
=== internal_console@7 (47 s)
ok github.com/carlosframework/platform/internal/console 44.712s
=== root@0 (41 s)
ok github.com/carlosframework/platform 38.915s
=== root@1 (49 s)
ok github.com/carlosframework/platform 46.668s
=== root@2 (48 s)
ok github.com/carlosframework/platform 45.967s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.025s
=== RUN TestT4StorageProbeMinio
2026/10/22 10:06:52 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0D20EA7FE0C8E, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md| index ffaca85b2..909023875 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| @@ -2,7 +2,7 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 6, 2026-10-22. Rev 5's review (`selfserve-plan2-review5.md`: items 2, 3, 4, 5, 7, 9, 11, 12, 14, 15, 16, 18, 19 closed) is answered: every job and operator record write is **predecessor-bound** (`provision.SetRecordMoveAt(ctx, s, acct, app, host, expectSeq, mut)`: one CAS write against the record whose `record_seq` equals the persisted expectation, no fresh-read retry), so a voided request can never land and an operator door never rebuilds a write after a reclaim fence (items 10, 13); every self-serve mutation stamps a **nonce receipt** on the record (`SelfServe.LastRequest`, or `LastMove.Request` for a clearing write), so confirmation is exact for every transition and distinct from an operator's reconciliation (item 10); confirm and advance require an unconsumed, unvoided request, so a stale worker cannot confirm a voided one (item 10); "pending" means **no confirmed request for the phase's write**, so the interval before issuance is compatible with the predecessor state (item 10); an expired uncommitted open is fenced and aborted, never replayed (N1); the stage book is reconciled against each complete successful round so deleted keys leave it (item 6); identity records are loaded by `boxid.Key(BoxID)` and `Verified` is narrowed to what `Enroll` proves (item 8); the gate's exception for an authorised malformed hostname and plan 1's outer 503 are declared (item 1); the view joins row id, move id, self-serve id and incarnation (item 17); the open record adopts an existing complete intent, return discovery and acceptance compare `InstanceID`, incarnation is re-checked at the door and in the guarded write, and `Layout` comes from the source's `stopped` report so every operator move has it (item 21); a maintenance restage in `ready` has its own kind and keeps the confirmation deadline (N2); the gate types, `StagedRequest`, failed diagnostics, the request argument to `confirmed`, and the bridge's compile rule are stated (item 20). Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items), rev 4 (`selfserve-plan2-review4.md`, 21 items, 9 closed). Rev 5 is **self-contained** (nothing is "as rev N"; every schema, phase, predicate, sentence and test list is in this document) and answers rev 4's open items: the gate orders authentication, host shape and the capability check before the limiter and the no-bucket rail (item 1); operator moves keep today's admission (nil or stale metrics, no identity record) and only the self-serve path requires fresh identity; `BootID` is dropped (items 4, 8); the evidence reader takes the move and a freshness table is stated (item 5); the staged inventory is a **move-bound book** built from every copy event the mirror reports, including same-ETag and deferred copies, through a new observer hook that leaves ordinary mirroring unchanged (item 6); a participant is identified by `BoxID` **and** `InstanceID` in the resolver, in report validation and at every gate, and abort never requires the target (item 8); every request persists its **nonce, expected sequence and proposed stamp before the write**, confirmation is conditional on the record carrying that exact stamp, replay is idempotent, and consumption and the phase advance are one UPDATE predicated on the nonce (item 10); compatible sets include the predecessor state while a phase's write is pending (item 10); expiry is bound to its phase and cleared on reconciliation into another phase, with the health rule taking precedence (item 12); abandonment is conditional on the fenced observation and persisted before release, and every reclaim fences first, operator reservations included (item 13); own retries are accepted only for a complete expected selection after the whole table validates (item 14); the status view joins rows regardless of the mode (item 17); events carry a per-row sequence and are delivered inside the caller's transaction, the harness is an exported bridge **inside package host** behind the `e2e` tag with the CI step named, the Go declarations are complete, and queue ordering is stated (item 20); the open record is written **before** the open transition with the proposed open time, return requests have unique durable identities, the report path is named, acceptance is bound to the lifetime and the forward's participants, and leftovers are judged by the forward's layout (item 21). |
| +Rev 7, 2026-10-22. Rev 6's review (`selfserve-plan2-review6.md`: items 1, 6, 17 closed) is answered: the request lifecycle is a complete state machine (`issued → confirmed → consumed`, or `voided`) with a **resume** path for a confirmed-unconsumed request, a standalone consume for same-phase sequences (start then seal, finish gate 1 then finish, round after round, estimate), issuance that returns the existing request so a caller of another kind or phase stops instead of interpreting it, and no-op-free receipts (a request-bearing mutation always writes its receipt) (item 10); the bound write checks **incarnation and sequence** (`provision.Expect{Incarnation, Seq}`) for the job and the operator doors (N5, item 13); every finalisation requires no outstanding request after a fence (item 13); the expiry branch classifies the fenced observation (N3); the cut is a **handoff**: the door marks `cut_requested_at` and the job turns the outstanding maintenance round into the first cut round (N4); event delivery marks `emitted` inside the transaction before delivering and the terminal event is inserted with the outcome (N1); per-box FIFO is stated (N2); the open intent is prepared before the request is issued so adoption and the persisted proposal agree, return requests carry a full identity ordered deterministically, reports name the request key, and acceptance binds the report to the reverse move's selected target (item 21); `Expect`'s real fields, `Load`'s absence distinction and one id-less rule (item 8); the fixtures move to an untagged `linux` file so today's tests keep compiling, `CopyEvent` carries its completion fields, the saved book is validated before seeding, `DiskFreeBytes` is numeric (item 20). Rev 5's review (`selfserve-plan2-review5.md`: items 2, 3, 4, 5, 7, 9, 11, 12, 14, 15, 16, 18, 19 closed) is answered: every job and operator record write is **predecessor-bound** (`provision.SetRecordMoveAt(ctx, s, acct, app, host, expectSeq, mut)`: one CAS write against the record whose `record_seq` equals the persisted expectation, no fresh-read retry), so a voided request can never land and an operator door never rebuilds a write after a reclaim fence (items 10, 13); every self-serve mutation stamps a **nonce receipt** on the record (`SelfServe.LastRequest`, or `LastMove.Request` for a clearing write), so confirmation is exact for every transition and distinct from an operator's reconciliation (item 10); confirm and advance require an unconsumed, unvoided request, so a stale worker cannot confirm a voided one (item 10); "pending" means **no confirmed request for the phase's write**, so the interval before issuance is compatible with the predecessor state (item 10); an expired uncommitted open is fenced and aborted, never replayed (N1); the stage book is reconciled against each complete successful round so deleted keys leave it (item 6); identity records are loaded by `boxid.Key(BoxID)` and `Verified` is narrowed to what `Enroll` proves (item 8); the gate's exception for an authorised malformed hostname and plan 1's outer 503 are declared (item 1); the view joins row id, move id, self-serve id and incarnation (item 17); the open record adopts an existing complete intent, return discovery and acceptance compare `InstanceID`, incarnation is re-checked at the door and in the guarded write, and `Layout` comes from the source's `stopped` report so every operator move has it (item 21); a maintenance restage in `ready` has its own kind and keeps the confirmation deadline (N2); the gate types, `StagedRequest`, failed diagnostics, the request argument to `confirmed`, and the bridge's compile rule are stated (item 20). Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items), rev 4 (`selfserve-plan2-review4.md`, 21 items, 9 closed). Rev 5 is **self-contained** (nothing is "as rev N"; every schema, phase, predicate, sentence and test list is in this document) and answers rev 4's open items: the gate orders authentication, host shape and the capability check before the limiter and the no-bucket rail (item 1); operator moves keep today's admission (nil or stale metrics, no identity record) and only the self-serve path requires fresh identity; `BootID` is dropped (items 4, 8); the evidence reader takes the move and a freshness table is stated (item 5); the staged inventory is a **move-bound book** built from every copy event the mirror reports, including same-ETag and deferred copies, through a new observer hook that leaves ordinary mirroring unchanged (item 6); a participant is identified by `BoxID` **and** `InstanceID` in the resolver, in report validation and at every gate, and abort never requires the target (item 8); every request persists its **nonce, expected sequence and proposed stamp before the write**, confirmation is conditional on the record carrying that exact stamp, replay is idempotent, and consumption and the phase advance are one UPDATE predicated on the nonce (item 10); compatible sets include the predecessor state while a phase's write is pending (item 10); expiry is bound to its phase and cleared on reconciliation into another phase, with the health rule taking precedence (item 12); abandonment is conditional on the fenced observation and persisted before release, and every reclaim fences first, operator reservations included (item 13); own retries are accepted only for a complete expected selection after the whole table validates (item 14); the status view joins rows regardless of the mode (item 17); events carry a per-row sequence and are delivered inside the caller's transaction, the harness is an exported bridge **inside package host** behind the `e2e` tag with the CI step named, the Go declarations are complete, and queue ordering is stated (item 20); the open record is written **before** the open transition with the proposed open time, return requests have unique durable identities, the report path is named, acceptance is bound to the lifetime and the forward's participants, and leftovers are judged by the forward's layout (item 21). |
| Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord(ctx, acct, app, host, op fenceOwner) (fenceResult, Observation, error)` exists, the console runs behind the lifetimes cutover. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| @@ -16,13 +16,13 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| ## Global Constraints |
| -- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. **Rendering never depends on the mode**: the status door joins rows whenever the console is claims-off and Phase 1 is off (Task 5). **Plan 1's lifetimes admission (`lifetimesAdmit`) wraps every record route first**: in `unknown` or `draining` its 503 answers before any gate of this plan, authorised or not; the route tests state that precedence. |
| +- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. **Rendering never depends on the mode**: the status door joins rows whenever the console is claims-off and Phase 1 is off (Task 5). **Plan 1's lifetimes admission (`lifetimesAdmit`) wraps every record route first**: in `unknown` every record route, and in `draining` every **mutating** route, answers plan 1's 503 before any gate of this plan, authorised or not (plain reads in `draining` are admitted judged, as plan 1 says); the route tests state that precedence. |
| - The operator doors keep their answers byte for byte (snapshots of every refusal, including a malformed body combined with each class refusal, replayed from main). A job tick never calls an HTTP door. |
| - Every wait has a deadline evaluated from the row before any evidence is read; an expiry is **bound to the phase that set it** (`expiry_phase`) and the action is decided by the record (Task 8): while the record is `offline-landed` or earlier → abort; committed open → page; unreadable → `open-uncertain`; the health phase's expiry → warning and `finishing` (precedence over the page rule). |
| -- Every record transition the job writes goes through `selfServeApply` (Task 7) with a **persisted request**: nonce, expected `record_seq` and the exact proposed stamp are written to the row before the record write; the write is `provision.SetRecordMoveAt` **bound to that expected sequence** (one CAS attempt, no fresh-read retry); the mutation carries the proposed stamp verbatim **and the nonce as a receipt** (`Move.SelfServe.LastRequest = nonce`, or `LastMove.Request = nonce` for a write that clears the move); confirmation is the record carrying the receipt; replay is the same bound write again; after every landed write the serial is bumped at least once (`notified_seq`). Operator doors use the same bound write with the sequence they observed before reserving (Task 3). |
| +- Every record transition the job writes goes through `selfServeApply` (Task 7) with a **persisted request**: nonce, expected incarnation, expected `record_seq` and the exact proposed stamp are written to the row before the record write; the write is `provision.SetRecordMoveAt` **bound to that expected incarnation and sequence** (one CAS attempt, no fresh-read retry); the mutation carries the proposed stamp verbatim **and the nonce as a receipt** (`Move.SelfServe.LastRequest = nonce`, or `LastMove.Request = nonce` for a write that clears the move); confirmation is the record carrying the receipt; replay is the same bound write again; after every landed write the serial is bumped at least once (`notified_seq`). Operator doors use the same bound write with the sequence they observed before reserving (Task 3). |
| - One console runs the pass under `control/offline-moves/console.json` (a 60 s lease, renewed every 20 s during a tick, checked before every write attempt with a fresh clock, and bounding every store write's context). The SQLite journal is the console's; a replacement console takes over by opening the **same** database file (one console host per deployment; the lease guards the day there are two) and recovers from the rows and the records. |
| - Refusals before writes; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| -- **Participants** are pinned at admission as `{BoxID, Label, InstanceID string; Verified bool}` (`BootID` is not pinned: it has no claims-off producer and a reboot does not change the machine's identity). **One resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate from `BoxID` **and** `InstanceID`; **one identity check** (`moves.ParticipantStillPresent`, Task 6) at every gate. A participant pinned with an empty `InstanceID` (the operator path on a box without a fresh report) keeps today's region rule for that participant and forces the sentinel reservation. A move without `Participants` keeps today's region rule everywhere. |
| +- **Participants** are pinned at admission as `{BoxID, Label, InstanceID string; Verified bool}` (`BootID` is not pinned: it has no claims-off producer and a reboot does not change the machine's identity). **One resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate from `BoxID` **and** `InstanceID`; **one identity check** (`moves.ParticipantStillPresent`, Task 6) at every gate. A participant pinned with an empty `InstanceID` (the operator path on a box without a fresh report) matches **any instance of its `BoxID`** (the label) and forces the sentinel reservation; there is no region fallback for a pinned participant (only a move without `Participants` keeps the region rule). A move without `Participants` keeps today's region rule everywhere. |
| - Timestamps in SQLite are Unix nanoseconds; request identities are the **nonce** persisted before the write and read back from the record's receipt. |
| - Copy rows pending Paul's review (every sentence in quotes in Tasks 5, 7 and 8). No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| - Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv` loaded by a new `loadRootReadInventory(..., true)` block in `internal/provision/plan14_root_reads_test.go` after every existing overlay and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-2`, `@fields` for the new record fields (`self_serve`, `participants`, `queued_at`, `precheck_at`, `return_of`), reviewed dispositions for the changed envelopes (`internal/host`, `internal/status`, `internal/moves`, `internal/console`, `internal/lease`); `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's with every added or changed writer: the new transitions, the box's report writers (lane, precheck, ready, staged, reverting, return-ready), the console's `open.json`, the return request writer, the console lease, `active.json`'s table form, the write guard; both green. |
| @@ -65,11 +65,11 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| **Interfaces:** |
| - `const OfflineQueued = "offline-queued"`. |
| -- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\`; LastRequest string \`json:"last_request,omitempty"\` }` (**the receipt**: every self-serve mutation sets it to the request's nonce); `LastMove` gains `Request string \`json:"request,omitempty"\`` (the receipt of the clearing write: `OfflineFinish`, `OfflineAbortAt`, the abort path's finish). **Every transition the job writes takes `req provision.Receipt{Nonce string}`** as an additive final parameter (`OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineRestage`, `OfflineSetEstimate`, `OfflineStartStaged`, `OfflineSeal`, `OfflineFlip`, `OfflineOpenTransition`, `OfflineFinishingTransition`, `OfflineFinish`, `OfflineAbort`, `OfflineAbortAt`): a non-empty nonce is written to `SelfServe.LastRequest` (or `LastMove.Request`); the operator doors pass an empty receipt, which leaves the field unchanged, so an operator transition never confirms a job request. |
| +- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\`; LastRequest string \`json:"last_request,omitempty"\` }` (**the receipt**: every self-serve mutation sets it to the request's nonce); `LastMove` gains `Request string \`json:"request,omitempty"\`` (the receipt of the clearing write: `OfflineFinish`, `OfflineAbortAt`, the abort path's finish). **Every transition the job writes takes `req provision.Receipt{Nonce string}`** as an additive final parameter (`OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineRestage`, `OfflineSetEstimate`, `OfflineStartStaged`, `OfflineSeal`, `OfflineFlip`, `OfflineOpenTransition`, `OfflineFinishingTransition`, `OfflineFinish`, `OfflineAbort`, `OfflineAbortAt`): a non-empty nonce is written to `SelfServe.LastRequest` (or `LastMove.Request`); the operator doors pass an empty receipt, which leaves the field unchanged, so an operator transition never confirms a job request. **With a non-empty receipt no transition is a no-op**: `OfflineSetEstimate` with an equal estimate and `OfflineAbort` on a reverting record still write the receipt (today's `errRecordUnchanged` short-cut applies only to an empty receipt), so every request-bearing write can be confirmed. |
| - `type Participant struct { BoxID string \`json:"box_id"\`; Label string \`json:"label"\`; InstanceID string \`json:"instance_id,omitempty"\`; Verified bool \`json:"verified,omitempty"\` }`; `type Participants struct { Source []Participant \`json:"source"\`; Target Participant \`json:"target"\` }`. |
| - `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, at time.Time) func(*Record) error`: `OfflineStage`'s refusals (a move in flight, a seed block, an open run, the same region) plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager: p.Source[0].Label`, `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: at`, `QueuedAt: &at`, `StagedAt: &at` (the pre-copy intent exists, so `StartedOffline()` is false until the start), `PrecheckAt: &at`, `Seq: rec.MoveSeq`; `move_seq` unchanged. **`at` is the caller's proposed stamp** and is stored verbatim (no clock read inside the mutation); the same holds for every transition below that takes `at`. |
| - `func OfflineQueuedToStaging(id string, at time.Time) func(*Record) error` (`StagedAt = &at`); `func OfflinePrecheck(id string, at time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &at`); `OfflineRestage(id, at)` stores `at` verbatim (today's signature; the console passes its proposed stamp); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, at time.Time, req Receipt) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: at, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour; a repeated abort on a reverting record stays a no-write success); `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged(id, at)` sets `SelfServe.OfflineAt = &at` when `SelfServe != nil`; `OfflineOpenTransition(id, at)` sets `OpenedAt = at`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| -- **The open record** `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; Participants *provision.Participants \`json:"participants,omitempty"\`; Layout string \`json:"layout"\` (`flat` or `nested`, from the source precheck's `Layout`); OpenedAt time.Time \`json:"opened_at"\` }` at `OpenRecordKey(host, id) = "control/offline-moves/<host>/<id>/open.json"`, written **create-only before** the open transition, by the operator open door and by the job alike: the writer first reads the key; an existing **complete** record for this `MoveID` is **adopted** (its `OpenedAt` becomes the proposed open stamp of this attempt, so a retry after a crash between the record and the transition opens at the recorded time); absent → written with this attempt's proposed stamp; present for another move id → 409 "another move's open record is in the way" (operator repair). `Layout` comes from the source's `stopped` report (`OfflineReport.Layout`, filled by every source at stop, so a direct operator move has it too). The open transition then carries the same `at`. Proof that the open **committed** is `rec.Move.OpenedAt == open.OpenedAt` while the move is on the record, or `LastMove.ID == open.MoveID && LastMove.Outcome == done` after it cleared; the return door requires the latter (a region comparison proves nothing: `OfflineFlip` changes the region before open). Because the record exists before the open can commit, no clearing transition can outrun it. |
| +- **The open record** `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; Participants *provision.Participants \`json:"participants,omitempty"\`; Layout string \`json:"layout"\` (`flat` or `nested`, from the source precheck's `Layout`); OpenedAt time.Time \`json:"opened_at"\` }` at `OpenRecordKey(host, id) = "control/offline-moves/<host>/<id>/open.json"`, written **create-only before** the open transition, by the operator open door and by the job alike: the writer first reads the key; an existing **complete** record for this `MoveID` is **adopted** (its `OpenedAt` becomes the proposed open stamp of this attempt, so a retry after a crash between the record and the transition opens at the recorded time); absent → written with a fresh proposed stamp; **this preparation runs before the open request is issued** (`selfServeApply` takes `proposed *time.Time`: the job passes the open record's `OpenedAt`, so the persisted proposal and the adopted intent are one value; a create-only conflict on the write re-reads and adopts); present for another move id → 409 "another move's open record is in the way" (operator repair). `Layout` comes from the source's `stopped` report (`OfflineReport.Layout`, filled by every source at stop, so a direct operator move has it too). The open transition then carries the same `at`. Proof that the open **committed** is `rec.Move.OpenedAt == open.OpenedAt` while the move is on the record, or `LastMove.ID == open.MoveID && LastMove.Outcome == done` after it cleared; the return door requires the latter (a region comparison proves nothing: `OfflineFlip` changes the region before open). Because the record exists before the open can commit, no clearing transition can outrun it. |
| - [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`; the stamps are the caller's `at` verbatim); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove`; `TestOfflineSetEstimateRefusesWithoutSelfServe`; `TestReturnOfCarriedByStageAndStart`; `TestOpenRecordPrecedesTheOpenTransition` (the operator open door and the job write it before the transition with the same `OpenedAt`; a crash between the record and the transition → the retry adopts the recorded `OpenedAt` and the record agrees; another move's record → 409; a failed open leaves a record the return door rejects because `LastMove` does not say `done`); `TestReceiptsOnEveryTransition` (each listed transition with a nonce writes `LastRequest` or `LastMove.Request`; an empty receipt leaves it unchanged); the allowlist test. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|ReturnOf|DiffersOnlyWhere' -count=1`, `go test ./internal/console -run 'OpenRecord' -count=1`, then the record-write census. |
| @@ -87,10 +87,10 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| - `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`, in order: (1) read `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}`, normalised into the table on the first new write; (2) **validate the whole object**: any malformed entry → `unreadable`; (3) **own retry**: entries with this `(Kind, ID)` exist → accepted only when they cover **exactly** the expected keys (every participant's `BoxID`, or `"*"` for a sentinel selection) with no extra; a partial own reservation → the missing keys are acquired in the same write if free, else `busy`; an own reservation whose keys differ from the expected selection → refuse "the reservation does not match this move's participants" (the job aborts); an accepted retry keeps `ReservedAt`; (4) `cfg.SelfServeMoves != "offline"` → refuse `busy` when any entry of another operation exists (today's deployment-wide semantics); (5) any expected key held by another `(Kind, ID)` → `busy` ("<host> is moving offline. move one instance at a time") unless reclaimable (below); (6) one conditional `Put` with every expected key and the projection (`Host, Op, Account, App, At` of the oldest `move` entry; of the oldest entry of any kind when no move holds); a CAS conflict re-reads once and retries from (2), then `busy`. |
| - `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error`: removes every entry with that `(Kind, ID)`, rewrites the projection, deletes the object when empty (`DeleteIfMatch`); absence is success. |
| - **Reclaim** (`time.Since(ReservedAt) >= offlineLockGrace`), for `move`: (a) `fenceRecord(ctx, acct, app, host, fenceOwner{})` (plan 1) must return `fenced` or `absent` (an error → not reclaimable); (b) the fenced observation carries no move with `ID`; (c) `SelfServeMove(id)` is absent or terminal with `release_pending = 0`; (d) no row for the host has `request_issued` set. An **operator** reservation (no row) is reclaimed on (a) and (b) alone: the fence makes a delayed operator start or stage PUT fail its CAS, so a negative observation after the fence is conclusive. For `cleanup` (plan 4): the owning box's status report (`status/boxes/<label>.json`, `GeneratedAt` within 2 status intervals, `Metrics.BoxID` equal to the key) lists the id in a terminal state with `RootRequest == ""` and `ReleasePending == false`, or its `Retirements` block is complete (`RetirementsComplete`) and lists no such id. Any read failure, a malformed body, or an absent or incomplete status → not reclaimable. |
| -- **Operator admission is predecessor-bound too:** the start and stage doors observe the record (judged) **before** reserving, reserve with `OfflineReservation.ExpectSeq = rec.RecordSeq` (a new tagged field), then write the record with `provision.SetRecordMoveAt(ctx, ..., ExpectSeq, mut)`; a CAS conflict → 409 "the instance changed while the move was being started. try again" and the door releases its reservation; the door never re-reads and retries. A reclaim's fence advances `record_seq`, so a paused door that resumes after the reclaim cannot write (its bound CAS fails) and cannot rebuild a write. |
| +- **Operator admission is predecessor-bound too:** the start and stage doors observe the record (judged) **before** reserving, reserve with `OfflineReservation.ExpectSeq = rec.RecordSeq` and `ExpectIncarnation = rec.Incarnation` (new tagged fields), then write the record with `provision.SetRecordMoveAt(ctx, ..., provision.Expect{Incarnation, Seq}, mut)` (a recreated host at a new incarnation refuses even when its fresh sequence happens to match); a CAS conflict → 409 "the instance changed while the move was being started. try again" and the door releases its reservation; the door never re-reads and retries. A reclaim's fence advances `record_seq`, so a paused door that resumes after the reclaim cannot write (its bound CAS fails) and cannot rebuild a write. |
| - **Rollout and rollback:** old consoles cannot be reintroduced while any reservation exists that the old console did not create (its release deletes the whole object when the projection names its own move; `TestOldConsoleReleaseErasesATable` documents the hazard); the supported sequence is roll consoles → turn the switch on; rollback is switch off → wait until **every incompatible operation** ends (every self-serve row terminal and released, every cleanup reservation gone, every operator reservation created under the table form released) → the old console. |
| -- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnRetryNeedsTheCompleteSelection` (exact → accepted keeping `ReservedAt`; partial with free keys → completed in one write; partial with a held key → busy; a changed selection → the mismatch refusal; a malformed unrelated entry is detected before the retry is accepted); `TestSentinelReservationConflictsBothWays`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive case and each refusing case above; a delayed queue PUT that commits after a negative re-read and before the fence → the fence observes it and the reservation is kept; a delayed operator start PUT → the fence fails it and the reservation is reclaimed; **an operator door paused between reserving and its record write** while the reclaim fences and another operation is admitted → the door's bound write fails with 409, nothing starts; a fence error → kept; a SQL failure → kept; a cleanup-only obligation at switch-off → kept and recovery proceeds); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody`; `TestSwitchOffRecoveryReacquiresItsOwnReservation` (two disjoint reservations, each retries its own → accepted); `TestOperatorDoorsUseReservations`. |
| +- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnRetryNeedsTheCompleteSelection` (exact → accepted keeping `ReservedAt`; partial with free keys → completed in one write; partial with a held key → busy; a changed selection → the mismatch refusal; a malformed unrelated entry is detected before the retry is accepted); `TestSentinelReservationConflictsBothWays`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive case and each refusing case above; a delayed queue PUT that commits after a negative re-read and before the fence → the fence observes it and the reservation is kept; a delayed operator start PUT → the fence fails it and the reservation is reclaimed; **an operator door paused between reserving and its record write** while the reclaim fences and another operation is admitted → the door's bound write fails with 409, nothing starts; the same door paused across a delete and recreate of the host whose new record is at the same sequence → refused on incarnation; a fence error → kept; a SQL failure → kept; a cleanup-only obligation at switch-off → kept and recovery proceeds); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody`; `TestSwitchOffRecoveryReacquiresItsOwnReservation` (two disjoint reservations, each retries its own → accepted); `TestOperatorDoorsUseReservations`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|Sentinel|OwnRetry|OperatorDoorsUse|Malformed' -count=1`. |
| - [ ] **Step 5: Commit** `console: per-box reservations with a legacy projection, a sentinel, complete own retries and fenced reclaim`. |
| @@ -102,16 +102,16 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| - Test: `internal/console/selfserve_store_test.go`, `internal/console/selfserve_pass_test.go` |
| **Interfaces:** |
| -- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, expiry_pending INTEGER, expiry_phase TEXT, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_nonce TEXT, request_expect_seq INTEGER, request_proposed INTEGER, request_confirmed INTEGER, request_consumed INTEGER, request_voided INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_seq INTEGER, event_seq INTEGER, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, open_record_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| -- **The request lifecycle** (one request at a time per row): `IssueSelfServeRequest(id, kind, phase string, nonce string, expectSeq uint64, proposed time.Time, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`: sets `request_issued = now, request_kind, request_phase, request_nonce, request_expect_seq, request_proposed, request_confirmed = 0, request_consumed = 0`); `ConfirmSelfServeRequest(id, nonce string, now time.Time) (bool, error)` (`WHERE request_nonce = ? AND request_consumed = 0 AND request_confirmed = 0`: `request_confirmed = now`; false → a voided or already consumed request: the caller stops; for kind `stage` also `stage_request = request_proposed`, for `precheck` also `precheck_request = request_proposed`); `VoidSelfServeRequest(id, nonce string) error` (`WHERE request_nonce = ? AND request_consumed = 0`: `request_consumed = 1, request_voided = 1`; the phase function decides again); `IssueSelfServeRequest` additionally requires `phase = ? AND outcome = ''` (the issuing phase owns the request); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, nonce string, now time.Time) (bool, error)`: one UPDATE with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing')) AND (? = '' OR request_nonce = ?)` that sets the phase, `phase_started_at`, the deadlines, `expiry_pending = 0, expiry_phase = ''` when `to <> expiry_phase`, and, when a nonce is given, additionally requires `request_nonce = ? AND request_confirmed <> 0 AND request_consumed = 0` and sets `request_consumed = 1` (**consumption and the advance are one statement**), and inserts the phase event in the same transaction (`event_seq = event_seq + 1`); a stale worker holding another nonce, or the same nonce after a void, cannot confirm or consume. |
| -- Other store methods: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0 OR open_record_pending = 1`, **ordered by `created_at, id`**: the pass steps rows in that order; a row waiting on a busy box stays in `reserving` and does not block rows with disjoint participants); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`health_attempts + 1`; `consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` kept once set), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiry(id, phase string, pending bool) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`, `SetSelfServeOpenRecordPending(id string, pending bool) error`. |
| -- Table `self_serve_events (id TEXT, seq INTEGER, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, seq))`: `InsertSelfServeEventTx(tx *sql.Tx, e SelfServeEvent) error` (allocates `seq` from the row's `event_seq` in the same transaction; used by `AdvanceSelfServe` and by every refusal, fact decision, lane probe, health probe and timing); `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`; `DeliverSelfServeEvent(ctx context.Context, e SelfServeEvent, deliver func(tx *sql.Tx) error) error` (one transaction: `deliver(tx)` inserts the audit row through `auditActivityTx(tx, ...)`, then `emitted = 1`; a failure rolls both back). Table `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| +- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, expiry_pending INTEGER, expiry_phase TEXT, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_nonce TEXT, request_expect_seq INTEGER, request_proposed INTEGER, request_confirmed INTEGER, request_consumed INTEGER, request_voided INTEGER, request_expect_inc INTEGER, cut_requested_at INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_seq INTEGER, event_seq INTEGER, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, open_record_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| +- **The request lifecycle** (one request at a time per row): `IssueSelfServeRequest(id, kind, phase string, pr persistedRequest, now time.Time) (issued bool, existing *persistedRequest, err error)` (`WHERE phase = ? AND outcome = '' AND (request_consumed <> 0 OR request_issued = 0)`: sets `request_issued = now, request_kind, request_phase, request_nonce, request_expect_inc, request_expect_seq, request_proposed, request_confirmed = 0, request_consumed = 0, request_voided = 0`; not issued → `existing` is the outstanding request, whatever its kind and phase); `ConsumeSelfServeRequest(id, nonce string) (bool, error)` (`WHERE request_nonce = ? AND request_confirmed <> 0 AND request_consumed = 0`: `request_consumed = 1`; the standalone consume for a same-phase sequence); `ConfirmSelfServeRequest(id, nonce string, now time.Time) (bool, error)` (`WHERE request_nonce = ? AND request_consumed = 0 AND request_voided = 0 AND request_confirmed = 0`: `request_confirmed = now`; false → the caller re-reads the row: confirmed already → **resume** (success), voided or consumed → stop; for kind `stage` also `stage_request = request_proposed`, for `precheck` also `precheck_request = request_proposed`); `VoidSelfServeRequest(id, nonce string) error` (`WHERE request_nonce = ? AND request_consumed = 0 AND request_confirmed = 0`: `request_consumed = 1, request_voided = 1`; a confirmed request is never voided; the phase function decides again); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, nonce string, now time.Time) (bool, error)`: one UPDATE with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing')) AND (? = '' OR request_nonce = ?)` that sets the phase, `phase_started_at`, the deadlines, `expiry_pending = 0, expiry_phase = ''` when `to <> expiry_phase`, and, when a nonce is given, additionally requires `request_nonce = ? AND request_confirmed <> 0 AND request_consumed = 0` and sets `request_consumed = 1` (**consumption and the advance are one statement**), and inserts the phase event in the same transaction (`event_seq = event_seq + 1`); a stale worker holding another nonce, or the same nonce after a void, cannot confirm or consume. |
| +- Other store methods: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0 OR open_record_pending = 1`, **ordered by `created_at, id`**: the pass steps rows in that order; **per-box FIFO**: a row in `reserving` may reserve only when no older open row (by `created_at, id`) that shares any participant `BoxID` is still in `requested` or `reserving`; otherwise it waits with `reason` "waiting for an earlier move on this box"; rows with disjoint participants never wait on each other); `MarkSelfServeCut(id string, now time.Time) (bool, error)` (`WHERE phase = 'ready' AND abort_requested_at = 0 AND cut_requested_at = 0`: sets `cut_requested_at`; the cut door's only write); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`health_attempts + 1`; `consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` kept once set), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiry(id, phase string, pending bool) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`, `SetSelfServeOpenRecordPending(id string, pending bool) error`. |
| +- Table `self_serve_events (id TEXT, seq INTEGER, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, seq))`: `InsertSelfServeEventTx(tx *sql.Tx, e SelfServeEvent) error` (allocates `seq` from the row's `event_seq` in the same transaction; used by `AdvanceSelfServe` and by every refusal, fact decision, lane probe, health probe and timing); `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`; `DeliverSelfServeEvent(ctx context.Context, e SelfServeEvent, deliver func(tx *sql.Tx) error) error` (one transaction: first `UPDATE self_serve_events SET emitted = 1 WHERE id = ? AND seq = ? AND emitted = 0`; zero rows → commit nothing and return (another worker delivered it); else `deliver(tx)` inserts the audit row through `auditActivityTx(tx, ...)`; a failure rolls both back). The terminal event is inserted by `FinishSelfServeMove` in the transaction that sets the outcome, so a crash between finalisation steps never allocates a second terminal event. Table `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| - Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. |
| - **Compatible sets** (Task 7's reconciliation; "pending" = the row has **no confirmed request** for that phase's write, whether none was issued yet, or one is issued and unconfirmed): `offline-queued` ↔ {queued, reports, staging(pending)}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier, stopping(pending)}; `offline-stopping` ↔ {stopping, sealing(pending)}; `offline-sealing` ↔ {sealing}; `offline-sealed` ↔ {sealing, landing(pending)}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain, opened(pending)}; `offline-open` ↔ {opened, health, finishing}; `offline-finishing` ↔ {finishing}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done, finishing(gate-2 pending), reverting(finish pending)}; no move and no `LastMove` for the id ↔ {requested, reserving, queued(pending)}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not in that set. |
| - Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `type consoleLease struct { Console string; Until time.Time }`; `acquireSelfServeLease(ctx, now) (*leaseToken, bool, error)`; `leaseToken.renew(ctx, now)` (every 20 s during a tick); `leaseToken.valid(now) bool`; `leaseToken.deadline() time.Time`; `selfServeApply` derives `ctx, cancel := context.WithDeadline(ctx, t.deadline())` for the record write and re-checks `t.valid(now)` before each retry through `provision.WithWriteGuard(ctx, func() error)`, a hook `WriteRecord` calls immediately before every `Put`. A cancelled context does **not** prove the write cannot commit; that is why every write is a persisted request (Task 7). |
| - `StartSelfServeMovesPass`: runs while `selfServeMode()`; a failing startup read of rows, events or `active.json` → the pass runs (fail closed) and retries the reads each tick, logging; `selfServeTick`: renew the lease, the reservation scan, `drainSelfServeEvents`, then one step per open row in order. |
| -- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestRequestLifecycle` (no second issue while unconsumed; issue only from the owning phase; confirm only with the nonce and only once; a voided request cannot be confirmed or consumed even with its own nonce; advance with the wrong nonce → false; advance with the right confirmed nonce consumes in the same statement); `TestAdvanceIsPredicatedOnPhaseAndNoAbort`; `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestAdvanceClearsAForeignExpiry`; `TestEventAndAdvanceAreOneTransaction`; `TestEventDeliveryIsTransactional`; `TestEventsHaveAPerRowSequence` (repeated prechecks produce distinct events); `TestOpenRowsOrderedAndIncludePending`; `TestPassStartsAndRetriesOnReadFailure`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseBoundsEveryWriteAttempt`; `TestPagingArmedForSelfServe`; `TestClaimStoreOpensWithEveryNewTable`. |
| +- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestRequestLifecycle` (no second issue while unconsumed and the existing request is returned; issue only from the owning phase; confirm only with the nonce and only once, a second confirm resumes; a voided request cannot be confirmed or consumed even with its own nonce; a confirmed request cannot be voided; the standalone consume; advance with the wrong nonce → false; advance with the right confirmed nonce consumes in the same statement); `TestPerBoxFIFO` (older A needs X,Y with Y held, newer B needs X,Z: B waits until A has reserved or ended; C on disjoint boxes proceeds); `TestEventDeliveryOnce` (two workers holding the same unemitted snapshot deliver it once; a crash between finalisation steps produces one terminal event); `TestAdvanceIsPredicatedOnPhaseAndNoAbort`; `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestAdvanceClearsAForeignExpiry`; `TestEventAndAdvanceAreOneTransaction`; `TestEventDeliveryIsTransactional`; `TestEventsHaveAPerRowSequence` (repeated prechecks produce distinct events); `TestOpenRowsOrderedAndIncludePending`; `TestPassStartsAndRetriesOnReadFailure`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseBoundsEveryWriteAttempt`; `TestPagingArmedForSelfServe`; `TestClaimStoreOpensWithEveryNewTable`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseBounds|RequestLifecycle|EventAnd|EventDelivery|EventsHave|PagingArmed|ClaimStoreOpens' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve journal with persisted requests and nonce-predicated advances, sequenced transactional events, the lease and the pass skeleton`. |
| @@ -125,8 +125,8 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| **Interfaces:** |
| - `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal`: `moveGate`'s checks in order (`Kind`, `Addr`/`Addrs`, fleet placement, `Pool`, `!HibernateOrDefault()`; 400s). |
| - `func (s *Server) offlineRecordChecks(ctx, acct, app, host string, rec *provision.Record) *offlineRefusal` (the pre-body half: the seed block, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`); `func (s *Server) offlineRegionChecks(ctx, acct, app, host string, rec *provision.Record, region string, stage, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)` (the post-body half: the region, the manifest, the fleet, the home rule, `offlineSelect`, the stage rules, participant pinning, the return's evidence); `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string; Sentinel bool }` (`Sentinel` when any participant's `InstanceID` is empty). |
| -- **Participant pinning.** `BoxID` and `Label` from the edges file. With `requireIDs: false` (**every operator door**): `InstanceID` from `status/boxes/<label>.json` when the report exists, is within 10 minutes and carries `Metrics.BoxID == BoxID`; otherwise `InstanceID = ""`, `Verified = false`, and the move proceeds as today (`offlineSelect` already admits a box with nil metrics); no identity record is read. With `requireIDs: true` (**the self-serve door and the job**): the status report must be present and fresh with a non-empty `InstanceID` (else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, BoxID)` (the record key is `boxid.Key(BoxID)`, never the label) finds a record and the verifier (constructed as `ClaimRootHTTP` constructs it) `Enroll`s it with `boxid.Expect{Box: BoxID, AWSRegion: the manifest's AWS region for the box's region, Account: the deployment account}` to evidence naming that `InstanceID`; `Verified` means exactly what `Enroll` proves (a signed document of this deployment naming the instance, consistent with the roster); it does not claim reference ownership beyond the roster rule, and the plan relies on it only to detect a replaced machine; a record that exists but does not verify → refuse "box <label> has an identity record that does not verify"; no record → pinned with `Verified: false` (allowed; the status report's `InstanceID` is the identity). |
| -- **The self-serve bodies.** `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}`: `selfServeGate`; `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`'s sentence; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` over 80 runes or with a control character → 400 "display_name must be at most 80 characters of plain text"; `offlineRecordChecks`; `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source's status facts (Task 6: `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` is the returned record and `recordEligibility`, the run and move checks run again on it; `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating; 200 `{"estimate_seconds", "files_bytes", "replica_bytes"}` or the refusal. `POST .../move/cut {"id"}`: `selfServeGate`; the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `AdvanceSelfServe(id, ready, restage, ..., "")` → 202; false → re-read: a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". `DELETE .../move {"id"}`: `selfServeGate`; the row and id; `RequestSelfServeAbort(id, [requested, reserving, queued, reports, staging, prechecks, ready], now)` → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| +- **Participant pinning.** `BoxID` and `Label` from the edges file. With `requireIDs: false` (**every operator door**): `InstanceID` from `status/boxes/<label>.json` when the report exists, is within 10 minutes and carries `Metrics.BoxID == BoxID`; otherwise `InstanceID = ""`, `Verified = false`, and the move proceeds as today (`offlineSelect` already admits a box with nil metrics); no identity record is read. With `requireIDs: true` (**the self-serve door and the job**): the status report must be present and fresh with a non-empty `InstanceID` (else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, BoxID)` (the record key is `boxid.Key(BoxID)`, never the label) finds a record and the verifier (constructed as `ClaimRootHTTP` constructs it) `Enroll`s it with `boxid.Expect{Box: BoxID, AWSRegion: the manifest's AWS region for the box's region, TofuInstanceID: the region's tofu output when the box predates region HA as the claims path supplies it, Roster: the current roster}` (the deployment account is the verifier's own field) to evidence naming that `InstanceID`; `boxid.Load` returning `os.ErrNotExist` → the unverified fallback; any other error → refuse "could not read the identity record for box <label>"; `Verified` means exactly what `Enroll` proves (a signed document of this deployment naming the instance, consistent with the roster); it does not claim reference ownership beyond the roster rule, and the plan relies on it only to detect a replaced machine; a record that exists but does not verify → refuse "box <label> has an identity record that does not verify"; no record → pinned with `Verified: false` (allowed; the status report's `InstanceID` is the identity). |
| +- **The self-serve bodies.** `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}`: `selfServeGate`; `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`'s sentence; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` over 80 runes or with a control character → 400 "display_name must be at most 80 characters of plain text"; `offlineRecordChecks`; `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source's status facts (Task 6: `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` is the returned record and `recordEligibility`, the run and move checks run again on it; `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating; 200 `{"estimate_seconds", "files_bytes", "replica_bytes"}` or the refusal. `POST .../move/cut {"id"}`: `selfServeGate`; the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `MarkSelfServeCut(id, now)` → 202 (the **handoff**: the job's `ready` phase performs the transition, Task 7); false → re-read: `cut_requested_at` set or a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". The view shows `cutting` from `cut_requested_at` on. `DELETE .../move {"id"}`: `selfServeGate`; the row and id; `RequestSelfServeAbort(id, [requested, reserving, queued, reports, staging, prechecks, ready], now)` (its predicate also requires `cut_requested_at = 0`) → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| - **The view** (`selfServeViewOf(rec, open, terminal, now)`, rendered whenever claims-off and Phase 1 off): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.ID && open.ID == rec.Move.SelfServe.ID && open.Incarnation == rec.Incarnation` → the open row's view (`preparing` through `ready`, `cutting` from `restage`); (2) the record carries any other move → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| - Copy rows: "only an account owner can mint a move-capable instance token"; "self-serve moves are not enabled on this console"; "could not read the instance"; "could not read the move"; "scheduled starts are not available yet"; "display_name must be at most 80 characters of plain text"; "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; "the source box has not reported recently. try again in a minute"; "the box <label> has not reported recently"; "box <label> has an identity record that does not verify"; "this account's files are too large to move this way"; "could not measure the account's files. try again in a minute"; "no staging lane is configured from <from> to <to>"; "another change to this instance is still settling. try again in a minute"; "this is not the move you confirmed"; "the move was cancelled"; "the move is not ready to cut yet. it is <phase>"; "the move was confirmed and cannot be cancelled. it completes or returns on its own"; "no move is in flight"; "the reservation does not match this move's participants". |
| @@ -143,11 +143,11 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| **Interfaces:** |
| - `OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `InstanceID string \`json:"instance_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject \`json:"staged,omitempty"\`` (the book), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `StagedRequest time.Time \`json:"staged_request,omitzero"\`` (**always equal to `StageRequest`** on a stager report; the stager sets both; today's duplicate-round suppression keeps reading `StageRequest`; a report with only one of them fails contextual validation), `Layout string \`json:"layout,omitempty"\`` (every source's `stopped` and `precheck` report: `flat` or `nested`), `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`. |
| - `type StagedObject struct { Size int64 \`json:"size"\`; ETag string \`json:"etag"\`; VersionID string \`json:"version_id,omitempty"\`; Deferred bool \`json:"deferred,omitempty"\` }` (`Size` keeps today's `int64` for the seal consumers). `type PrefixObject struct { ETag string \`json:"etag"\`; Size int64 \`json:"size"\`; VersionID string \`json:"version_id,omitempty"\`; LastModified time.Time \`json:"last_modified"\` }`; `type PrefixFacts struct { Known bool \`json:"known"\`; Objects map[string]PrefixObject \`json:"objects,omitempty"\` }` (keys **relative to the prefix**; current versions only; delete markers excluded; `lock.json` excluded). |
| -- `type Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. `type LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the existing Phase 1 `laneProbe`: put-if-absent, list, delete-version under `moves/_lane/<move id>`). `type TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\`; MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\`; Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }`. `type StrayEntry struct { Path string \`json:"path"\`; Kind string \`json:"kind"\` (dir, file, symlink, other); Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\`; Known bool \`json:"known"\`; Detail string \`json:"detail,omitempty"\` }`; `type StraysFact struct { Known bool \`json:"known"\`; Entries []StrayEntry \`json:"entries,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; Layout string \`json:"layout"\` (flat, nested); ArchiveEligible Fact \`json:"archive_eligible"\`; Strays StraysFact \`json:"strays"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; StraysRemoved []string \`json:"strays_removed,omitempty"\`; StraysLeft []StrayEntry \`json:"strays_left,omitempty"\` }`: a non-directory stray is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. |
| +- `type Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. `type LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the existing Phase 1 `laneProbe`: put-if-absent, list, delete-version under `moves/_lane/<move id>`). `type TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\`; DiskFreeBytes int64 \`json:"disk_free_bytes"\` (the console compares this against 2 × the replica size; the fact's own `OK` is the box's local threshold); MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\`; Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }`. `type StrayEntry struct { Path string \`json:"path"\`; Kind string \`json:"kind"\` (dir, file, symlink, other); Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\`; Known bool \`json:"known"\`; Detail string \`json:"detail,omitempty"\` }`; `type StraysFact struct { Known bool \`json:"known"\`; Entries []StrayEntry \`json:"entries,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; Layout string \`json:"layout"\` (flat, nested); ArchiveEligible Fact \`json:"archive_eligible"\`; Strays StraysFact \`json:"strays"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; StraysRemoved []string \`json:"strays_removed,omitempty"\`; StraysLeft []StrayEntry \`json:"strays_left,omitempty"\` }`: a non-directory stray is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. |
| - `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Known bool \`json:"known"\`; Err string \`json:"err,omitempty"\`; Incarnation uint64 \`json:"incarnation"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; ReplicaCollectedAt int64 \`json:"replica_collected_at"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; OverCap bool \`json:"over_cap,omitempty"\`; Truncated bool \`json:"truncated,omitempty"\`; Strays []string \`json:"strays,omitempty"\` }`; `measureFiles(ctx, files, envPath string, budget time.Duration, capBytes int64) (bytes int64, entries int, overCap, truncated bool, err error)` (a bounded walker applying the archive's counting rules without stopping at the cap: records `OverCap` and continues until the budget (60 s) or 2 × the cap, where it records `Truncated`); the replica listing cached per tick with its own `ReplicaCollectedAt`; `Known` only when every part succeeded. `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES`; `BoxMetrics.BoxID string` (the edges writer's value); `BoxMetrics.InstanceID` exists. |
| -- **Participants:** `func ParticipantRole(m *provision.MoveState, boxID, instanceID, label, placement, home string) (source, target, sealer, stager bool)`: with `Participants`: a participant matches when `p.BoxID == boxID && (p.InstanceID == "" || p.InstanceID == instanceID)`; a box whose `BoxID` matches a participant with a non-empty different `InstanceID` has **no role** (it logs "box <label> is not the pinned instance for move <id>" once per move and does nothing); without `Participants`: today's region and label rules. `func ParticipantStillPresent(p provision.Participant, m *status.BoxMetrics) bool` = `p.InstanceID == "" || (m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID)` (the console's gate check, on the label's status report within 10 minutes; a missing report with a non-empty pinned id → "box <label> was replaced" after 10 minutes, "box <label> has not reported recently" before). |
| +- **Participants:** `func ParticipantRole(m *provision.MoveState, boxID, instanceID, label, placement, home string) (source, target, sealer, stager bool)`: with `Participants`: a participant matches when `p.BoxID == boxID && (p.InstanceID == "" || p.InstanceID == instanceID)` (an id-less participant matches any instance of its box; never another box of the region); a box whose `BoxID` matches a participant with a non-empty different `InstanceID` has **no role** (it logs "box <label> is not the pinned instance for move <id>" once per move and does nothing); without `Participants`: today's region and label rules. `func ParticipantStillPresent(p provision.Participant, m *status.BoxMetrics) bool` = `p.InstanceID == "" || (m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID)` (the console's gate check, on the label's status report within 10 minutes; a missing report with a non-empty pinned id → "box <label> was replaced" after 10 minutes, "box <label> has not reported recently" before). |
| - **Validation:** `ValidateOfflineReport(r)` is **unchanged** (legacy structure). `func ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState, now time.Time) error` adds the contextual rules: a move with `Participants` → the report's `BoxID` and `InstanceID` equal the pinned participant for its role (`InstanceID` compared only when the pinned one is non-empty); a move with `SelfServe` → role `lane` ↔ phase `lane` with `Lane`; `source`+`precheck` with `Precheck` and `Request`; `target`+`ready` with `Ready` and `Request`; `stager`+`staged` with `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest`; `sealer`+`reverting` with `TombstoneLifted`; a `failed` report needs only the identity rule (its diagnostic is kept: the console reads `moves.FailedDetail(r)` = `StageError` when set, else the first failed item's text, else "the box reported a failure without detail"). **Freshness table:** request-bound reports (`lane` by move id; `precheck`, `ready` by `Request`; `staged` by `StagedRequest`) have **no age limit** and at most 60 s of future skew; the phase reports today's gates read (`stopped`, `sealed`, `landed`, `opened`, `reverted`, `reverting`, `failed`) keep `offlineFresh` (2 min, 1 min skew) where today applies it; the seal is immutable and gets only the identity rule; participant identity from status is 10 min. `func (s *Server) readOfflineEvidence(ctx context.Context, host, id string, m *provision.MoveState) (offlineEvidence, error)` applies both validators and drops a report that fails the contextual rule, logging it; every caller (`offlineStepFor`, the seal, flip, open and finish checks, `offlineView`, the job) passes the move it holds (`nil` for a move it has not read: legacy validation only). |
| -- **The stage book (item 6).** `type lease.CopyEvent struct { Rel string; SrcETag string; SrcSize int64; DstETag string; DstVersionID string; Unchanged, Deferred, Deleted bool }`; `func MirrorPrefixKeepObserved(ctx, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string, rc Receipts, keep func(string) bool, observe func(CopyEvent)) (MirrorRound, error)`: emits one event per source key the round decided (a copy with the ETag and version the copy returned; `Unchanged` with the listed destination ETag when the round skipped it because the ETags were equal **or because a receipt vouched for the multipart copy**, the receipt's `DstETag`/`DstVersionID` in the latter case; `Deferred` when the round deferred it) and `Deleted` per destination key removed, and finally `Complete: true` once with the round's full source and destination key sets when the round finished without error; `MirrorPrefixKeep` = `...Observed(..., nil)`, so ordinary mirroring and the receipt ledger are unchanged. The stager keeps `stageBook{MoveID string; Objects map[string]moves.StagedObject}` across rounds: seeded at the first round of a move from its own last `staged` report for that move id when one exists (else empty); every event upserts (`Deleted` removes; `Deferred` marks the existing entry deferred without changing its ETag, or inserts `{Deferred: true}` for a key never copied); on `Complete` the book is **reconciled**: every entry whose key is in neither the source nor the destination set is dropped (a deletion a crashed round completed but never reported), and a deferred-never-copied entry whose key left the source is dropped; a round that fails before `Complete` leaves the book as the events left it (the next complete round reconciles); the `staged` report carries the whole book with `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = m.StagedAt`. **Staged ownership** (the console's prechecks and barrier): `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest == row.StageRequest`; the target's `MovesPrefix.Known`; the key sets are equal (both relative to their prefixes; entries with `Deferred` and no ETag must be **absent** from the target); for every key the target's `ETag` equals the book's, sizes are equal, and version ids are equal when both are present; the barrier additionally requires `RoundDeferredObjects == 0`; a mismatch → "the staging prefix for <host> does not match what this move staged". |
| +- **The stage book (item 6).** `type lease.CopyEvent struct { Rel string; SrcETag string; SrcSize int64; DstETag string; DstVersionID string; Unchanged, Deferred, Deleted bool; Complete bool; SourceKeys, DestKeys []string }` (`Complete` is the one final event of a round that finished without error and carries both relative key sets); `func MirrorPrefixKeepObserved(ctx, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string, rc Receipts, keep func(string) bool, observe func(CopyEvent)) (MirrorRound, error)`: emits one event per source key the round decided (a copy with the ETag and version the copy returned; `Unchanged` with the listed destination ETag when the round skipped it because the ETags were equal **or because a receipt vouched for the multipart copy**, the receipt's `DstETag`/`DstVersionID` in the latter case; `Deferred` when the round deferred it) and `Deleted` per destination key removed, and finally `Complete: true` once with the round's full source and destination key sets when the round finished without error; `MirrorPrefixKeep` = `...Observed(..., nil)`, so ordinary mirroring and the receipt ledger are unchanged. The stager keeps `stageBook{MoveID string; Objects map[string]moves.StagedObject}` across rounds: seeded at the first round of a move from its own last `staged` report when one exists **and validates** (`Op == m.ID`, `Box == cfg.Box`, `Role == stager`, `StagedKnown`, `StagedBucket/Region` equal to the record's; anything else → an empty book, logged); every event upserts (`Deleted` removes; `Deferred` marks the existing entry deferred without changing its ETag, or inserts `{Deferred: true}` for a key never copied); on `Complete` the book is **reconciled**: every entry whose key is in neither the source nor the destination set is dropped (a deletion a crashed round completed but never reported), and a deferred-never-copied entry whose key left the source is dropped; a round that fails before `Complete` leaves the book as the events left it (the next complete round reconciles); the `staged` report carries the whole book with `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = m.StagedAt`. **Staged ownership** (the console's prechecks and barrier): `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest == row.StageRequest`; the target's `MovesPrefix.Known`; the key sets are equal (both relative to their prefixes; entries with `Deferred` and no ETag must be **absent** from the target); for every key the target's `ETag` equals the book's, sizes are equal, and version ids are equal when both are present; the barrier additionally requires `RoundDeferredObjects == 0`; a mismatch → "the staging prefix for <host> does not match what this move staged". |
| - **Reports to write:** the source participant in `offline-queued`: the lane probe once per move → the `lane` report; `offlineSourcePrecheck` for the current `PrecheckAt` → `precheck` with `Request = PrecheckAt`; the target participant: `offlineTargetReady` → `ready` with `Request = PrecheckAt`; in `offline-staging` both re-report when `PrecheckAt` is newer than their last `Request`; the stager's `staged` after each round; the sealer's `reverting` after `LiftTombstone` succeeds and before the prefix cleanup; the final `reverted` keeps both lift fields; the normal seal's final-copy bytes in `FinalCopiedBytes`. |
| - [ ] **Step 1: Write the failing tests**: `TestParticipantRoleByBoxAndInstance` (a matching pair; an id-less participant matches any instance of the box; a replaced box has no role); `TestParticipantStillPresent`; `TestLegacyReportsStillValidate`; `TestContextualValidationForSelfServeMoves` (the matrix: every role/phase/payload; identity mismatch; a failed report kept with only identity checked); `TestFreshnessTable`; `TestEmptyStagedInventoryRoundTrips`; `TestObservedMirrorEmitsEveryDecision` (copy, unchanged same-ETag, deferred, deleted; the receipt ledger unchanged; `MirrorPrefixKeep` behaviour unchanged against the production `NewS3Mirror` wrapper through `stageDeleteGate`); `TestStageBookAcrossRounds` (through `NewS3Mirror`'s wrapper and `stageDeleteGate` against a fake store: seeded from the last report; same-ETag copies present; a receipt-backed multipart skip present with the receipt's ETag; a deferred then copied key; a deleted key gone; **a key deleted by a round that crashed before its report is dropped by the next complete round**; a deferred-never-copied key whose source vanished is dropped; an overwrite between the copy and the report → the copy's ETag; a crash before the report → the next round re-copies and the book converges); `TestStagedOwnershipComparison` (equal; missing; extra; changed ETag; changed size; version mismatch; deferred-and-never-copied present → refuse; wrong request; wrong destination; unknown prefix); `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown`; `TestSourcePrecheckFacts` (incl. `Layout`); `TestStrayFileRefusesBeforeStop`; `TestEmptyStrayDirsRemovedAtStartOnly`; `TestSealerPublishesTombstoneLiftedBeforeCleanup`; `TestQueuedRoundRunsOnlyLanePrecheckAndReady`; `TestParticipantsReReportOnANewPrecheckStamp`; `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID`, `InstanceID` in the private report, absent from `fleet.json`); `TestHostFactsCarryTheLifetimeAndAges`; `TestVersionedArmedForOffline`. |
| @@ -161,9 +161,9 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| - Test: `internal/e2e/selfserve/job_test.go` |
| **Interfaces:** |
| -- `type persistedRequest struct { Kind, Phase, Nonce string; ExpectSeq uint64; Proposed time.Time }`; `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, req request, mut func(at time.Time, rc provision.Receipt) func(*provision.Record) error, confirmed func(rec provision.Record, pr persistedRequest) bool) (*provision.Record, error)` with `type request struct { Kind, Phase string }`: (1) the judged record → `pr.ExpectSeq = rec.RecordSeq`; `pr.Proposed = max(now, latest stamp on the record + 1 ns)`; `pr.Nonce = random`; `IssueSelfServeRequest(id, kind, phase, nonce, expectSeq, proposed, now)` (false → a request is outstanding: load it as `pr` and go to (3)); (2) `provision.SetRecordMoveAt(ctx', s, acct, app, host, pr.ExpectSeq, mut(pr.Proposed, Receipt{pr.Nonce}))` with the lease-bounded context and the write guard: **one CAS attempt against the record at `ExpectSeq`; no fresh-read retry inside**; (3) **confirmation** from a fresh judged read, whatever (2) returned: `rec` carries the receipt (`SelfServe.LastRequest == pr.Nonce`, or `LastMove.Request == pr.Nonce` for a clearing write; `confirmed` may add the phase's own check) → `ConfirmSelfServeRequest(id, nonce, now)` (false → voided meanwhile: stop); else `rec.RecordSeq == pr.ExpectSeq` (unchanged) → the write did not land: repeat (2) (idempotent replay of the same bound write; bounded by the lease); else (the record advanced without the receipt: another writer, an operator) → `VoidSelfServeRequest(id, nonce)` and return `errRequestVoided`; the bound write makes this safe: a PUT transmitted against `ExpectSeq` cannot commit on a record that has moved past it; after a confirmed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq`. A crash anywhere resumes at (3) on the next tick from the persisted request. |
| -- `provision.SetRecordMoveAt(ctx, s, acct, app, host string, expectSeq uint64, mut) (*provision.Record, error)`: reads the judged record; `rec.RecordSeq != expectSeq` → `ErrCASConflict` without writing; otherwise one `WriteRecord` against the observed ETag; a CAS conflict → `ErrCASConflict` (no retry). `SetRecordMove` (today's, with its retry) is no longer used by any self-serve or operator move writer. |
| -- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` → `SetSelfServeExpiry(id, phase, true)` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets (Task 4): an unconfirmed request → step (3) of `selfServeApply` first, **except** when `expiry_pending` is set for the issuing phase and the record is at or before `offline-landed`: then the record is observed once; the receipt present → confirmed (the write won before the expiry, the expiry is post-write); absent → `fenceRecord` (plan 1; so the outstanding bound PUT can never land), `VoidSelfServeRequest`, and the expiry action runs (abort), **never a replay** (`TestExpiredOpenIsNeverReplayed`); then the row's phase not in the record's set → `AdvanceSelfServe` to the set's first phase (without the abort predicate for `opened`), which clears a foreign expiry; (4) the expiry action for an `expiry_pending` whose `expiry_phase == phase`: the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; `health` → warning, page, `finishing`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Every wait for a report is for `Request`/`StagedRequest == request_proposed` of the **confirmed** request; the report is consumed by the `AdvanceSelfServe` that leaves the waiting phase, with the nonce. |
| +- `type persistedRequest struct { Kind, Phase, Nonce string; ExpectInc, ExpectSeq uint64; Proposed time.Time; Confirmed, Consumed, Voided bool }`; `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, req request, proposed *time.Time, mut func(at time.Time, rc provision.Receipt) func(*provision.Record) error, confirmed func(rec provision.Record, pr persistedRequest) bool) (*provision.Record, error)` with `type request struct { Kind, Phase string }`: (1) the judged record → `pr.ExpectInc = rec.Incarnation`, `pr.ExpectSeq = rec.RecordSeq`; `pr.Proposed = *proposed` when given (the open intent) else `max(now, latest stamp on the record + 1 ns)`; `pr.Nonce = random`; `IssueSelfServeRequest(id, kind, phase, pr, now)`: not issued → `existing.Kind != req.Kind || existing.Phase != req.Phase` → return `errRequestForeign` (this caller never interprets another request; the step ends and reconciliation decides next tick); `existing.Confirmed && !existing.Consumed` → **resume**: return the record (the caller continues its wait and the notification catch-up); otherwise `pr = *existing` and go to (3); (2) `provision.SetRecordMoveAt(ctx', s, acct, app, host, provision.Expect{Incarnation: pr.ExpectInc, Seq: pr.ExpectSeq}, mut(pr.Proposed, Receipt{pr.Nonce}))` with the lease-bounded context and the write guard: **one CAS attempt against the record at that incarnation and sequence; no fresh-read retry inside**; (3) **confirmation** from a fresh judged read, whatever (2) returned: `rec` carries the receipt (`SelfServe.LastRequest == pr.Nonce`, or `LastMove.Request == pr.Nonce` for a clearing write; `confirmed` may add the phase's own check) → `ConfirmSelfServeRequest(id, nonce, now)` (false → re-read: already confirmed → resume; voided → stop); else `rec.Incarnation == pr.ExpectInc && rec.RecordSeq == pr.ExpectSeq` (unchanged) → the write did not land: repeat (2) (idempotent replay of the same bound write; bounded by the lease); else (the record advanced without the receipt: another writer, an operator) → `VoidSelfServeRequest(id, nonce)` and return `errRequestVoided`; the bound write makes this safe: a PUT transmitted against `ExpectSeq` cannot commit on a record that has moved past it; after a confirmed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq`. A crash anywhere resumes at (3) on the next tick from the persisted request. |
| +- `type provision.Expect struct { Incarnation, Seq uint64 }`; `provision.SetRecordMoveAt(ctx, s, acct, app, host string, expect Expect, mut) (*provision.Record, error)`: reads the judged record; `rec.Incarnation != expect.Incarnation || rec.RecordSeq != expect.Seq` → `ErrCASConflict` without writing; otherwise one `WriteRecord` against the observed ETag; a CAS conflict → `ErrCASConflict` (no retry). `SetRecordMove` (today's, with its retry) is no longer used by any self-serve or operator move writer. |
| +- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` → `SetSelfServeExpiry(id, phase, true)` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets (Task 4): an unconfirmed request → step (3) of `selfServeApply` first, **except** when `expiry_pending` is set for the issuing phase and the record is at or before `offline-landed`: then the record is observed once; the receipt present → confirmed (the write won before the expiry, the expiry is post-write); absent → `fenceRecord` (plan 1; so the outstanding bound PUT can never land) and **the fenced observation is classified**: the receipt present in it → the write won during the fence: confirm and proceed forward (an open that committed is reconciled as opened, never aborted); absent → `VoidSelfServeRequest` and the expiry action runs (abort), **never a replay** (`TestExpiredOpenIsNeverReplayed`); then the row's phase not in the record's set → `AdvanceSelfServe` to the set's first phase (without the abort predicate for `opened`), which clears a foreign expiry; (4) the expiry action for an `expiry_pending` whose `expiry_phase == phase`: the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; `health` → warning, page, `finishing`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Every wait for a report is for `Request`/`StagedRequest == request_proposed` of the **confirmed** request; the report is consumed by the `AdvanceSelfServe` that leaves the waiting phase, with the nonce, **or**, inside a phase that writes more than once (`stopping`: start then seal; `finishing`: gate 1 then gate 2; `staging`: round after round; `prechecks`: the precheck then the estimate; `ready`: each maintenance round), by `ConsumeSelfServeRequest(id, nonce)` once its wait is satisfied, immediately before the next request is issued. |
| - **Reconciliation and recovery table** (the record is the truth): |
| | row phase | record | action | |
| @@ -186,10 +186,10 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| - `reports`: the `lane` (`OK`), the source `precheck` and the target `ready` with `Request == PrecheckAt`, each from the pinned participant; `targetReadyDecision` and `sourcePrecheckDecision` (the console enumerates the required path set and fact set itself: a missing entry or `!Known` → "could not determine <fact> on <label>"; not OK → the fact's sentence: "the target box already has a route for <host>", "the target box has timers for <host>", "the target box has a stale live link for <host> naming <build>", "the target box's programs file names <build>", "the target box already holds data for <host>: <path>", "the target box's park directory is not clean", "the target box has too little free disk for <host>" (`!Known` or `< 2 × replica`), "the target bucket already holds objects for <host>", "the target bucket holds a lock for <host> (<owner>)", "<host> has a stray entry beside its files directory: <path>", "<host>'s files cannot be archived: <detail>"; before staging both prefixes must be `Known` and empty); both pass → `selfServeApply(stage, OfflineQueuedToStaging, confirmed: StagedAt == proposed)`, `StagingStartedAt = now`, → `staging`; 5 min → abort. |
| - `staging`: the stager report with `StagedRequest == row.StageRequest`: the ready rule `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `selfServeApply(stage, OfflineRestage, confirmed: StagedAt == proposed)` → a new wait; a `failed` report → abort with its `Detail`; a round past `max(10 min, 3 × est.Copy)` → abort; `now - StagingStartedAt > 30 min` or `Rounds >= 10` → `prechecks` from the last completed round. |
| - `prechecks`: `selfServeApply(precheck, OfflinePrecheck, confirmed: PrecheckAt == proposed)`; both participants' reports answering it; the staged-ownership comparison (Task 6); `InstancesPrefix` known and empty; both pass → `ready`, `WindowDeadline = now + 24 h`, the estimate refined and `OfflineSetEstimate` (a later disagreement resolves towards the row); 5 min → abort. |
| - - `ready`: every 10 min a **maintenance restage** (request kind `maintain`: `selfServeApply(maintain, OfflineRestage)`, the matching `staged` report consumed by an advance `ready → ready` with the nonce; the row stays `ready`, `WindowDeadline` is **not** touched, no prechecks run, and the cut predicate is untouched); `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| -- The pre-offline abort (requested, reserving, queued, staging): no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`); queued → `selfServeApply(abort, OfflineAbortAt, confirmed: Move == nil && LastMove.ID == row.ID)`; staging → `OfflineAbort` → the stager's `reverted` → `OfflineFinish`; then the finaliser. |
| + - `ready`: every 10 min a **maintenance restage** (request kind `maintain`: `selfServeApply(maintain, OfflineRestage)`, the matching `staged` report consumed by `ConsumeSelfServeRequest`; the row stays `ready`, `WindowDeadline` is **not** touched, no prechecks run); `WindowDeadline` → abort "the move was not confirmed in time"; **the cut handoff**: when `cut_requested_at` is set: an outstanding `maintain` request → it becomes the first cut round: its report is awaited (deadline `max(5 min, 2 × est.Copy)` from `cut_requested_at`; expiry → abort) and consumed by `AdvanceSelfServe(ready, restage, ..., nonce)`, after which `restage` issues nothing new and goes straight to `barrier`; no outstanding request → `AdvanceSelfServe(ready, restage, ..., "")` and `restage` issues its own round. The view shows `cutting` from `cut_requested_at`. |
| +- The pre-offline abort (requested, reserving, queued, staging): **no row is finalised while a request is outstanding**: an outstanding request is resolved first (confirmed → the record carries the move, so this is not the no-move branch; unconfirmed → `fenceRecord`, classify the fenced observation, void if the receipt is absent); then no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`); queued → `selfServeApply(abort, OfflineAbortAt, confirmed: Move == nil && LastMove.ID == row.ID)`; staging → `OfflineAbort` → the stager's `reverted` → `OfflineFinish`; then the finaliser. |
| -- [ ] **Step 1: Write the failing tests** on the bridge of Task 10: `TestEveryBoundaryRecovers` (Review Focus 1: for every write, a crash before the write with the request persisted, a lost response, a crash before confirmation, a crash before the bump, a second crash during recovery, SQLite reopened each time; a fake store that **commits after the context was cancelled**: the replay sees the proposed stamp and confirms, no second write; one request per wait, one bump per landed write; ordinary multi-tick progression through start and flip); `TestReconcileForwardNeverRegresses` (every row of the table; `stopping` with no request issued yet and the record still `offline-staging` stays; `stopping(pending)` likewise; `landing` with no request yet and `offline-sealed` stays; health and finishing survive ticks at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed`; `TestStaleEvidenceNeverSatisfiesANewRequest` (an issued-but-unwritten restage: the previous round's report does not satisfy it; after the crash the replay lands the proposed stamp and only a report carrying it does); `TestAbandonedAdmissionIsFencedAndConditional` (a queue PUT committing after a negative re-read: the fence observes the move and the job continues as `queued`; a PUT that never lands: the fence succeeds, the row is voided durably before the release); `TestReplacedParticipantAtEveryGate`; `TestReservingWaitsThenAborts`; `TestEligibilityRechecksBeforeQueue`; `TestReportsDecideFromFacts`; `TestStagingReadyRule`; `TestPrechecksCompareTheBook`; `TestReadyRestagesAndExpires` (the maintenance restage keeps the row in `ready`, keeps `WindowDeadline`, never enters `restage`/`barrier`, and the cut still needs its door); `TestExpiredOpenIsNeverReplayed` (an open request persisted, the PUT never sent, a restart after the landed deadline: the record is fenced, the request voided, the move aborts; the same with the PUT committed before the crash → confirmed, the move proceeds as opened); `TestBoundWritesNeverLandAfterAVoid` (another writer advances the record after a request was issued: the bound write fails, the request is voided, no later commit appears); `TestDeadlinesEvaluateBeforeEvidence`; `TestAbortBeforeOfflineIsLossless`; `TestQueueOrderAndDisjointProgress` (three rows: two on one box in order, a third on disjoint boxes advances while the second waits). |
| +- [ ] **Step 1: Write the failing tests** on the bridge of Task 10: `TestEveryBoundaryRecovers` (Review Focus 1: for every write, a crash before the write with the request persisted, a lost response, a crash before confirmation, a crash before the bump, a second crash during recovery, SQLite reopened each time; a fake store that **commits after the context was cancelled**: the replay sees the proposed stamp and confirms, no second write; one request per wait, one bump per landed write; ordinary multi-tick progression through start and flip); `TestReconcileForwardNeverRegresses` (every row of the table; `stopping` with no request issued yet and the record still `offline-staging` stays; `stopping(pending)` likewise; `landing` with no request yet and `offline-sealed` stays; health and finishing survive ticks at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed`; `TestStaleEvidenceNeverSatisfiesANewRequest` (an issued-but-unwritten restage: the previous round's report does not satisfy it; after the crash the replay lands the proposed stamp and only a report carrying it does); `TestAbandonedAdmissionIsFencedAndConditional` (a queue PUT committing after a negative re-read: the fence observes the move and the job continues as `queued`; a PUT that never lands: the fence succeeds, the row is voided durably before the release); `TestReplacedParticipantAtEveryGate`; `TestReservingWaitsThenAborts`; `TestEligibilityRechecksBeforeQueue`; `TestReportsDecideFromFacts`; `TestStagingReadyRule`; `TestPrechecksCompareTheBook`; `TestReadyRestagesAndExpires` (the maintenance restage keeps the row in `ready`, keeps `WindowDeadline`, never enters `restage`/`barrier`, and the cut still needs its door); `TestExpiredOpenIsNeverReplayed` (an open request persisted, the PUT never sent, a restart after the landed deadline: the record is fenced, the request voided, the move aborts; the same with the PUT committed before the crash → confirmed, the move proceeds as opened; **the PUT committing between the observation and the fence** → the fenced observation carries the receipt, the move proceeds as opened); `TestConfirmedRequestResumesAfterACrash` (start confirmed, crash before consume → the next tick resumes the wait, no re-issue, one bump); `TestSamePhaseSequencesConsume` (start then seal, gate 1 then gate 2, two staging rounds, the estimate after the precheck); `TestForeignRequestStopsTheCaller`; `TestNoOpMutationsStillConfirm` (an equal estimate, a repeated abort on a reverting record); `TestCutHandoffWithAnOutstandingMaintenanceRound` (cut requested while a maintain request is outstanding: its report becomes the first cut round; cut requested with none outstanding: restage issues; cancel after cut → 409); `TestBoundWritesNeverLandAfterAVoid` (another writer advances the record after a request was issued: the bound write fails, the request is voided, no later commit appears; a delete and recreate whose new record has the same sequence → refused on incarnation); `TestDeadlinesEvaluateBeforeEvidence`; `TestAbortBeforeOfflineIsLossless`; `TestQueueOrderAndDisjointProgress` (three rows: two on one box in order, a third on disjoint boxes advances while the second waits). |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test -tags e2e ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileForward|RequestsAreNever|StaleEvidence|AbandonedAdmission|ReplacedParticipant|Reserving|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline|QueueOrder' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, reconciling forward with persisted requests, a conditional fenced abandonment and at-least-once notification`. |
| @@ -203,11 +203,11 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| **Interfaces:** |
| - Phases and deadlines (`est` the row's terms): `restage`: `selfServeApply(stage, OfflineRestage)`; the matching `staged`; `max(5 min, 2 × est.Copy)`; expiry → abort → `barrier`. `barrier`: `selfServeApply(precheck, OfflinePrecheck)`; both participants' reports passing the decisions and the book comparison with zero deferrals; `max(5 min, 2 × est.Copy)`; a failed fact or expiry → abort → `stopping`. `stopping`: `offlineRecordChecks` + `offlineRegionChecks(..., stage: false, requireIDs: true, "")` on the fresh record, then `selfServeApply(start, OfflineStartStaged, confirmed: State == offline-stopping && SelfServe.OfflineAt == proposed)`; then the seal check with participants → `selfServeApply(seal, OfflineSeal, confirmed: State == offline-sealing)`; `max(5 min, 2 × est.Stop)`; expiry → abort. `sealing`: the sealer's `sealed`; `max(10 min, 3 × est.Copy)`; expiry → abort. `landing`: `offlineFlipCheck` → `selfServeApply(flip, OfflineFlip, confirmed: State == offline-landing)`; the pinned target's `landed`; `max(10 min, 3 × est.Land)`; expiry → abort. `landed`: `offlineOpenCheck` → write the open record (create-only, `OpenedAt = proposed`; `open_record_pending = 1` until it exists) → `selfServeApply(open, OfflineOpenTransition, confirmed: State == offline-open && OpenedAt == proposed)`; a refusal or `max(3 min, 2 × est.Open)` **while the record is `offline-landed`** → abort; an unconfirmable write (the record unreadable) → `open-uncertain`. `open-uncertain`: re-read each tick: `offline-open` → `opened`; `offline-landed` → `landed` (the deadline continues); 5 min unreadable → page, stay. `opened`: the target's `opened`; `max(3 min, 2 × est.Open)`; expiry → blocked: page, retry. `health`: a probe when `now - HealthLastProbeAt >= 30 s` (`probeTarget`: the target box's `EIP` from its edges file, TLS with the host as SNI and `Host`, `GET <ReadyPathOrDefault>?carlos-probe=<random>`, `Cache-Control: no-cache`, no redirects, 30 s; OK when 2xx and `x-carlos-version` equals the sealer report's `Release`; plan 3 adds `X-Carlos-Probe`/`X-Carlos-Served-By`); `RecordSelfServeHealth`; `HealthConsecutiveOK >= 2` → `finishing`; expiry `max(3 min, 2 × est.Promote) + 30 s` → `SetSelfServeWarning("the account did not answer on <region> within <n> minutes")`, page, `finishing`. `finishing`: gate 1 (`offlineFinishCheck` opened + every pinned source parked) → `selfServeApply(finish1, OfflineFinishingTransition)`, `FinishGate = lifted`; gate 2 (lifted + every pinned source gone) → `selfServeApply(finish, OfflineFinish, confirmed: Move == nil && LastMove.ID == row.ID)`; 10 min each → blocked: page, retry. `reverting`: `OfflineAbort` once (or `OfflineAbortAt` from queued); `SourceReopenedAt` when (no tombstone, or the sealer's `reverting`/`reverted` with `TombstoneLifted`) **and** every pinned source's `reverted`; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes → `OfflineFinish`; **the target is never required**: a target that stopped reporting after landing does not block the source's reopening; the abort's `Detail` records "target <label> did not report; its staged copy is left for cleanup"; 10 min → blocked: page, retry. |
| - **The finaliser** `selfServeFinalise(ctx, t, row, outcome, reason, warning)`: (1) `FinishSelfServeMove` (outcome; `release_pending = 1`, `effects_done = 0`); (2) the terminal effects: the timings (Task 9), the terminal event, `bumpSerialErr`; `SetSelfServeEffectsDone`; (3) `releaseBoxes(move, row.ID)`; (4) `ClearSelfServeRelease`. A crash between any two leaves the row in `OpenSelfServeMoves` and the next tick resumes at the first unfinished step. The operator doors that clear a self-serve move's record route through it. |
| -- **The operator return handshake (§7.5):** (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): the open record at `OpenRecordKey(host, return_of)` must exist with `To == rec.Region`, `From == region`, `Incarnation == rec.Incarnation`, and the record carries no move; else 409 "no completed move <id> from <region> to <rec.Region> for this instance"; writes `control/offline-moves/<host>/<return_of>/return/<requested_at unix nanos>-<console id>.json` `{"requested_at", "region", "console"}` create-only (a new call makes a new request; the **newest** `requested_at` is the current request; older ones are ignored) and answers 202 "return requested; the target box will report its leftovers". (2) The moves pass of every box in `region` lists `control/offline-moves/<host>/*/return/` for hosts whose open record names `From == its region` and, when the open record carries `Participants`, whose `Source` includes its own `BoxID` **and** (when pinned) its own `InstanceID` (an id-less forward falls to the region rule); a box whose `BoxID` matches but whose `InstanceID` differs does nothing; it reads the newest request, fills `Leftovers` against the open record (`OpenedAt`, `Layout`, `MoveID`) and writes `control/offline-moves/<host>/<return_of>/reports/return-ready-<box id>.json` (`Op = return_of`, role `target`, phase `ready`, `Request = requested_at`, `BoxID/InstanceID`), without touching the record; it re-reports whenever a newer request appears. (3) The operator's `stage` (or `start`) door with `return_of`: `readReturnReport(ctx, host, returnOf)` must find a report with `Request` equal to the newest request's `requested_at`, within 10 minutes, from a box whose `BoxID` **and `InstanceID`** match a forward source participant (or in `From` for an id-less forward); the open record's `Incarnation` must equal the current record's **again here** and inside the guarded write (`OfflineStage`/`OfflineStart` with `returnOf` refuse with `ErrMoveConflict` "the instance was recreated since that move" when `rec.Incarnation != open.Incarnation`, the open record being passed to the mutation); then `targetReadyDecision` in return mode accepts **exactly**: `Lock` present with owner `moves.TombstoneOwner(return_of)` and lifted; `InstancesPrefix.Objects` (current versions, `lock.json` excluded) all with `LastModified < OpenedAt` (any newer → `NewerObjects`, refused); `DataPaths["/data/instances/<host>"]` absent or present-and-empty; for `Layout == nested` also `DataPaths["/data/.instances/<host>"]` absent or present-and-empty, and the mount view likewise; for `flat` the nested pair must be absent; park directories: each present one must be `<return_of>/source/` or `<return_of>/sealer/` (absence allowed: cleanup may have removed them; any other → refused); `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\`; Layout string \`json:"layout"\` }`. |
| +- **The operator return handshake (§7.5):** (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): the open record at `OpenRecordKey(host, return_of)` must exist with `To == rec.Region`, `From == region`, `Incarnation == rec.Incarnation`, and the record carries no move; else 409 "no completed move <id> from <region> to <rec.Region> for this instance"; writes `control/offline-moves/<host>/<return_of>/return/<requested_at unix nanos>-<console id>-<8 random hex>.json` `{"requested_at", "region", "console", "key"}` create-only (the **request key** is the request's identity; a new call makes a new request; the current request is the key that sorts **last** by `(requested_at, key)`; older ones are ignored) and answers 202 "return requested; the target box will report its leftovers". (2) The moves pass of every box in `region` lists `control/offline-moves/<host>/*/return/` for hosts whose open record names `From == its region` and, when the open record carries `Participants`, whose `Source` includes its own `BoxID` **and** (when pinned) its own `InstanceID` (an id-less forward falls to the region rule); a box whose `BoxID` matches but whose `InstanceID` differs does nothing; it reads the newest request, fills `Leftovers` against the open record (`OpenedAt`, `Layout`, `MoveID`) and writes `control/offline-moves/<host>/<return_of>/reports/return-ready-<box id>.json` (`Op = return_of`, role `target`, phase `ready`, `Request = requested_at`, `RequestKey = <the request key>` (a new `OfflineReport` field), `BoxID/InstanceID`), without touching the record; it writes only when the request key is later than the one its last report named (so a delayed older pass never overwrites a newer report) and re-reports whenever a newer request appears. (3) The operator's `stage` (or `start`) door with `return_of`: `readReturnReport(ctx, host, returnOf)` must find a report whose `RequestKey` equals the current request's key, within 10 minutes, from a box whose `BoxID` **and `InstanceID`** match a forward source participant (or in `From` for an id-less forward) **and** equal the target `offlineSelect` chose for the return (the report authorises returning onto that box alone; a report from another forward source does not); the open record's `Incarnation` must equal the current record's **again here** and inside the guarded write (`OfflineStage`/`OfflineStart` with `returnOf` refuse with `ErrMoveConflict` "the instance was recreated since that move" when `rec.Incarnation != open.Incarnation`, the open record being passed to the mutation); then `targetReadyDecision` in return mode accepts **exactly**: `Lock` present with owner `moves.TombstoneOwner(return_of)` and lifted; `InstancesPrefix.Objects` (current versions, `lock.json` excluded) all with `LastModified < OpenedAt` (any newer → `NewerObjects`, refused); `DataPaths["/data/instances/<host>"]` absent or present-and-empty; for `Layout == nested` also `DataPaths["/data/.instances/<host>"]` absent or present-and-empty, and the mount view likewise; for `flat` the nested pair must be absent; park directories: each present one must be `<return_of>/source/` or `<return_of>/sealer/` (absence allowed: cleanup may have removed them; any other → refused); `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\`; Layout string \`json:"layout"\` }`. |
| - Paging: `alerting.Page` through `MovePageAttempt`/`SetMovePageAttempt` (seconds encoding, unchanged) with keys `selfserve:<id>:<condition>` every 10 minutes for: `opened` past its deadline; a health failure; `finishing`/`reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; an open record still pending after 5 minutes; a reservation the reclaim rule cannot reclaim older than the grace period (the per-tick scan, also at start); a lane probe failing on a configured pair; the attempt is recorded **before** the page is sent, as the existing paging does. |
| - Copy rows: "the account did not answer on <region> within <n> minutes"; "the move was not started in time"; "the move was not confirmed in time"; "another move is in flight"; "box <label> was replaced"; "the move could not be started"; "the instance was recreated"; "target <label> did not report; its staged copy is left for cleanup"; "no completed move <id> from <region> to <rec.Region> for this instance"; "return requested; the target box will report its leftovers"; the fact refusals of Task 7. |
| -- [ ] **Step 1: Write the failing tests** on the bridge: `TestCutRunsEveryPhaseToDone`; `TestEveryDeadlineBeforeCommittedOpenAborts`; `TestLandedExpiryDefersToTheRecord` (restart after a committed open with an old `landed` expiry → `opened`, no abort; the expiry is cleared and the move runs to `done`); `TestHealthExpiryTakesPrecedence`; `TestOpenUncertainIsReconciledByTheRecord`; `TestOpenRecordPrecedesTheOpen` (a crash between the open record and the transition; a crash after; both recover); `TestAbortIsInitiatedNotCompleted` (the real sealer; Review Focus 3; the target disappears after landing → the source reopens, the detail names the target); `TestOpenedBlockedPagesAndRetries`; `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthAttemptsPersist`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep`; `TestOperatorDoorsFinaliseASelfServeMove`; `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRendersAfterRestart`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap`; `TestOperatorReturnHandshake` (the return door; the box's report through its real moves pass; a second request supersedes the first and the box re-reports; the stage door accepting exactly the forward's leftovers for a flat and a nested forward, fully and partially cleaned; a newer object, a live lock, a foreign park dir, a non-empty nested dir, a report from a non-participant box, a report from the right `BoxID` with another `InstanceID`, an open record of another lifetime at the request door and again at the stage door and inside the write, a `return_of` whose open record is absent, each refuse; a direct start with `return_of` → the same acceptance; restart at every handshake boundary). |
| +- [ ] **Step 1: Write the failing tests** on the bridge: `TestCutRunsEveryPhaseToDone`; `TestEveryDeadlineBeforeCommittedOpenAborts`; `TestLandedExpiryDefersToTheRecord` (restart after a committed open with an old `landed` expiry → `opened`, no abort; the expiry is cleared and the move runs to `done`); `TestHealthExpiryTakesPrecedence`; `TestOpenUncertainIsReconciledByTheRecord`; `TestOpenRecordPrecedesTheOpen` (a crash between the open record and the transition; a crash after; both recover); `TestAbortIsInitiatedNotCompleted` (the real sealer; Review Focus 3; the target disappears after landing → the source reopens, the detail names the target); `TestOpenedBlockedPagesAndRetries`; `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthAttemptsPersist`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep`; `TestOperatorDoorsFinaliseASelfServeMove`; `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRendersAfterRestart`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap`; `TestOperatorReturnHandshake` (the return door; the box's report through its real moves pass; a second request supersedes the first and the box re-reports; the stage door accepting exactly the forward's leftovers for a flat and a nested forward, fully and partially cleaned; a newer object, a live lock, a foreign park dir, a non-empty nested dir, a report from a non-participant box, a report from the right `BoxID` with another `InstanceID`, a report from a forward source that is not the return's selected target, two requests at one timestamp (the key order decides), a delayed older report that cannot overwrite the newer, an open record of another lifetime at the request door and again at the stage door and inside the write, a `return_of` whose open record is absent, each refuse; a direct start with `return_of` → the same acceptance; restart at every handshake boundary). |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the targeted tests, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve cut with the no-return boundary at the committed open, the abort facts without the target, one finaliser, durable paging and a recoverable operator return handshake`. |
| @@ -230,7 +230,7 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| ### Task 10: The exported bridge and the end-to-end witnesses |
| **Files:** |
| -- Create: `internal/host/e2e_bridge.go` (**in package `host`**, `//go:build e2e`: `type E2E struct { ... }` with `func NewE2E(t testing.TB, store release.Store) *E2E`, `(*E2E).Pass(ctx)` (one moves pass on every box), `(*E2E).Serial(ctx)` (passes until the serial is consumed), `(*E2E).Clock() *fakeClock`, `(*E2E).Fault(point string, f func())` (the fault points the tests name), `(*E2E).Box(label) BoxHandle` exposing the box id, instance id, registry, data bucket and the worker's `armOffline`/`reconcileSerializedWith` through methods; it reaches `movesWorker`, `moveCfg`, `newMovesWorker`, `armOffline`, `reconcileSerializedWith` directly because it is in the package, so nothing private moves and no import cycle arises; the Linux-only fixtures stay behind `//go:build linux && e2e` with a stub that skips), (no untagged companion file: symbols declared under `e2e` cannot be referenced without the tag; instead the gate runs `go vet -tags e2e ./internal/host/ ./internal/e2e/...` so the bridge compiles on every CI run), the fixtures the bridge needs that today live in `movesoffline_e2e_test.go` (the two-box store, registries, fake clock and the pass driver) are **moved** into `e2e_bridge.go` as non-test code under the tag and the test file imports them from there, `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}`), `internal/e2e/selfserve/*_test.go` (`//go:build e2e`), `.amadan/test-gate.sh` (new steps `go vet -tags e2e ./internal/host/ ./internal/e2e/...` and `go test -tags e2e ./internal/e2e/...` after the ordinary suite, plus `go test -tags e2e -list . ./internal/e2e/selfserve | grep -q '^Test'` so an empty discovery fails the gate) |
| +- Create: `internal/host/e2e_bridge.go` (**in package `host`**, `//go:build e2e`: `type E2E struct { ... }` with `func NewE2E(t testing.TB, store release.Store) *E2E`, `(*E2E).Pass(ctx)` (one moves pass on every box), `(*E2E).Serial(ctx)` (passes until the serial is consumed), `(*E2E).Clock() *fakeClock`, `(*E2E).Fault(point string, f func())` (the fault points the tests name), `(*E2E).Box(label) BoxHandle` exposing the box id, instance id, registry, data bucket and the worker's `armOffline`/`reconcileSerializedWith` through methods; it reaches `movesWorker`, `moveCfg`, `newMovesWorker`, `armOffline`, `reconcileSerializedWith` directly because it is in the package, so nothing private moves and no import cycle arises; the Linux-only fixtures stay behind `//go:build linux && e2e` with a stub that skips), The bridge and the fixtures it exports live in `internal/host/e2e_bridge_linux.go` with **`//go:build linux` only** (no `e2e` tag), so `movesoffline_e2e_test.go` (tagged `linux`) and the other `_test.go` users (`offWorld`/`offBox`, `sealWorld`, `landWorld`) keep compiling in the ordinary suite; the moved fixtures are exactly those four worlds, the two-box store and registries, the fake clock and the pass driver, renamed to exported names with the old test-local names kept as aliases in the test files; the exported API is dead code in a production binary (stated and accepted); only the `internal/e2e/selfserve` test files carry `//go:build e2e`, `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}`), `internal/e2e/selfserve/*_test.go` (`//go:build e2e`), `.amadan/test-gate.sh` (new steps `go vet -tags e2e ./internal/e2e/...` and `go test -tags e2e ./internal/e2e/...` after the ordinary suite, plus `go test -tags e2e -list . ./internal/e2e/selfserve | grep -q '^Test'` so an empty discovery fails the gate) |
| - Test: the witnesses of Tasks 7 and 8, plus `TestSelfServeForwardMove`; `TestSelfServeCancelInReady`; `TestSelfServeAbortByFailedPrecheck`; `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal`; `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord`; `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchangedOnTheBridge`; `TestSelfServeThenOperatorReturn`; `TestClaimsOffInstanceIDIsPublished` (the agent's status tick carries `InstanceID` and `BoxID` on a claims-off box). |
| - [ ] **Step 1: Write the bridge and the tests.** **Step 2: Run them with `-tags e2e`; fix what they find within this plan's scope.** **Step 3: Both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-2.md` (the overlay's loader block, the changed envelopes, the new writers incl. the return request, the open record, the console lease and `active.json`'s table form). **Step 4: Commit** `self-serve moves: an exported two-box bridge inside package host and the end-to-end witnesses, run by the gate`; push. |
| @@ -241,5 +241,5 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13 |
| - Spec coverage: §5 (T1), §6.1 (T5), §6.2 (T7), §6.3 (T9, T5, T6), §7 admission and recovery (T7's table), §7.1 (T3), §7.2 and the three facts with the sealer's durable lift (T6, T8), §7.3 (T6, T7, T8's barrier), §7.4 without the probe mode (T8), §7.5 the no-return boundary and the operator return (T2's open record, T8), §7.7 (T5), §8 pages, log and member stream (T8, T9), §9 (T3's rollout). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup (plan 4), whose status contract the reclaim rule names (`RetirementsComplete`, a terminal state, `RootRequest == ""`, `ReleasePending == false`, `Metrics.BoxID`). |
| - Dependencies for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}`, `Participants.Target.BoxID`, `moveCfg.BoxID`, `probeTarget`; for plan 4: the status `Retirements` contract and the open record. |
| -- Names used across tasks: `selfServeGate`, `instanceRails(w, r, acct, noun, c)`, `moveCapable`, `Principal.Moves/TokenID`, `selfServeMode`, `selfServeAdmits`, `recordEligibility`, `offlineRecordChecks`, `offlineRegionChecks`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant{BoxID, Label, InstanceID, Verified}`, `Participants`, `moves.OpenRecord{..., Layout}`, `OpenRecordKey`, `moves.ParticipantRole`, `moves.ParticipantStillPresent`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `request`, `persistedRequest`, `provision.Receipt`, `SelfServeMove.LastRequest`, `LastMove.Request`, `provision.SetRecordMoveAt`, `OfflineReservation.ExpectSeq`, `moves.FailedDetail`, `OfflineReport.Layout`, `errRequestVoided`, `selfServeFinalise`, `leaseToken`, `provision.WithWriteGuard`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `VoidSelfServeRequest`, `request_voided`, `AdvanceSelfServe(..., nonce)`, `SetSelfServeExpiry`, `SetSelfServeNotifiedSeq`, `SetSelfServeOpenRecordPending`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `PrefixObject`, `StagedObject{Size, ETag, VersionID, Deferred}`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck{..., Layout}`, `StrayEntry`, `StraysFact`, `ReturnLeftovers`, `ValidateOfflineReportFor`, `readOfflineEvidence(ctx, host, id, m)`, `readReturnReport`, `lease.CopyEvent`, `lease.MirrorPrefixKeepObserved`, `stageBook`, `status.HostFacts`, `BoxMetrics.Hosts/Lanes/BoxID/InstanceID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MarkSelfServeTiming`, `SelfServeEvent`, `host.E2E`/`NewE2E`, `console.TestHooks`. |
| +- Names used across tasks: `selfServeGate`, `instanceRails(w, r, acct, noun, c)`, `moveCapable`, `Principal.Moves/TokenID`, `selfServeMode`, `selfServeAdmits`, `recordEligibility`, `offlineRecordChecks`, `offlineRegionChecks`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant{BoxID, Label, InstanceID, Verified}`, `Participants`, `moves.OpenRecord{..., Layout}`, `OpenRecordKey`, `moves.ParticipantRole`, `moves.ParticipantStillPresent`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `request`, `persistedRequest`, `provision.Receipt`, `provision.Expect`, `ConsumeSelfServeRequest`, `MarkSelfServeCut`/`cut_requested_at`, `errRequestForeign`, `OfflineReport.RequestKey`, `DiskFreeBytes`, `OfflineReservation.ExpectIncarnation`, `SelfServeMove.LastRequest`, `LastMove.Request`, `provision.SetRecordMoveAt`, `OfflineReservation.ExpectSeq`, `moves.FailedDetail`, `OfflineReport.Layout`, `errRequestVoided`, `selfServeFinalise`, `leaseToken`, `provision.WithWriteGuard`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `VoidSelfServeRequest`, `request_voided`, `AdvanceSelfServe(..., nonce)`, `SetSelfServeExpiry`, `SetSelfServeNotifiedSeq`, `SetSelfServeOpenRecordPending`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `PrefixObject`, `StagedObject{Size, ETag, VersionID, Deferred}`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck{..., Layout}`, `StrayEntry`, `StraysFact`, `ReturnLeftovers`, `ValidateOfflineReportFor`, `readOfflineEvidence(ctx, host, id, m)`, `readReturnReport`, `lease.CopyEvent`, `lease.MirrorPrefixKeepObserved`, `stageBook`, `status.HostFacts`, `BoxMetrics.Hosts/Lanes/BoxID/InstanceID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MarkSelfServeTiming`, `SelfServeEvent`, `host.E2E`/`NewE2E`, `console.TestHooks`. |
| - Review Focus 1 to 5 are pinned by T7, T3, T8, T5, T1. |