amadan
design: self-serve offline moves, rev 3 (Astra round 2)
The automatic return narrows to every failure before open, where it is lossless by construction; after open a failure to answer is caught, paged and finished with a warning, and the return is an operator's run of the offline move back whose leftover acceptance is specified. The lane probe moves to moves/@lanes/ outside any consumable prefix; reservations keep active.json's grace-and-reread discipline per box; cleanup uses delete markers from a retired.json inventory under the host's reservation; holding entries carry record_seq with a precedence rule and the routes' alias and TLS eligibility; probes are uncached, redirect-free and build-checked; cut and cancel name the move id; empty extra directories are removed at start, archive eligibility is a dry run before the stop. Q3 asks Paul to accept the narrower promise.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+57
−48
docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md+57 −48
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (18 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (167 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.717s
ok github.com/carlosframework/platform/cmd/moveprobe 11.931s
ok github.com/carlosframework/platform/cmd/zddprobe 0.764s
ok github.com/carlosframework/platform/internal/activator 61.819s
ok github.com/carlosframework/platform/internal/agentsock 0.026s
ok github.com/carlosframework/platform/internal/alerting 0.010s
ok github.com/carlosframework/platform/internal/analytics 0.055s
ok github.com/carlosframework/platform/internal/appconfig 2.851s
ok github.com/carlosframework/platform/internal/apperr 0.198s
ok github.com/carlosframework/platform/internal/applog 0.359s
ok github.com/carlosframework/platform/internal/awsinv 0.012s
ok github.com/carlosframework/platform/internal/boxchoice 0.010s
ok github.com/carlosframework/platform/internal/boxid 0.899s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.018s
ok github.com/carlosframework/platform/internal/caps 0.024s
ok github.com/carlosframework/platform/internal/certwarm 0.009s
ok github.com/carlosframework/platform/internal/cgroup 0.840s
ok github.com/carlosframework/platform/internal/changelog 0.596s
ok github.com/carlosframework/platform/internal/claimactivation 1.360s
ok github.com/carlosframework/platform/internal/clamd 0.066s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.265s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.016s
ok github.com/carlosframework/platform/internal/dbverify 0.014s
ok github.com/carlosframework/platform/internal/dnsrec 0.007s
ok github.com/carlosframework/platform/internal/docsite 0.011s
ok github.com/carlosframework/platform/internal/domains 2.764s
ok github.com/carlosframework/platform/internal/doorkey 0.019s
ok github.com/carlosframework/platform/internal/economics 0.013s
ok github.com/carlosframework/platform/internal/edge 45.530s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.136s
ok github.com/carlosframework/platform/internal/gate 1.655s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.014s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/historyreadback 0.162s
ok github.com/carlosframework/platform/internal/host 138.143s
ok github.com/carlosframework/platform/internal/hostactors 5.954s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.016s
ok github.com/carlosframework/platform/internal/hostbox 3.105s
ok github.com/carlosframework/platform/internal/hostclaim 0.807s
ok github.com/carlosframework/platform/internal/hostenv 0.014s
ok github.com/carlosframework/platform/internal/hostevidence 0.016s
ok github.com/carlosframework/platform/internal/hostexec 5.384s
ok github.com/carlosframework/platform/internal/hostfence 0.044s
ok github.com/carlosframework/platform/internal/hostfiles 10.449s
ok github.com/carlosframework/platform/internal/hostfloor 0.021s
ok github.com/carlosframework/platform/internal/hostguard 1.024s
ok github.com/carlosframework/platform/internal/hostops 1.982s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.016s
ok github.com/carlosframework/platform/internal/hostquarantine 0.016s
ok github.com/carlosframework/platform/internal/hostrelocate 0.640s
ok github.com/carlosframework/platform/internal/hostresolve 0.201s
ok github.com/carlosframework/platform/internal/hostwrite 0.016s
ok github.com/carlosframework/platform/internal/infraview 0.018s
ok github.com/carlosframework/platform/internal/instancectl 5.232s
ok github.com/carlosframework/platform/internal/instanceexec 2.790s
ok github.com/carlosframework/platform/internal/instancefs 0.025s
ok github.com/carlosframework/platform/internal/instancekey 0.021s
ok github.com/carlosframework/platform/internal/instanceupdates 0.024s
ok github.com/carlosframework/platform/internal/layout 1.755s
ok github.com/carlosframework/platform/internal/lbhealth 0.020s
ok github.com/carlosframework/platform/internal/lease 10.260s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.532s
ok github.com/carlosframework/platform/internal/ledger 0.034s
ok github.com/carlosframework/platform/internal/mail 0.233s
ok github.com/carlosframework/platform/internal/maintenance 0.081s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.621s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.130s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 22.577s
ok github.com/carlosframework/platform/internal/moves/archive 0.993s
ok github.com/carlosframework/platform/internal/objstore 0.347s
ok github.com/carlosframework/platform/internal/ops 0.311s
ok github.com/carlosframework/platform/internal/park 0.296s
ok github.com/carlosframework/platform/internal/plans 0.008s
ok github.com/carlosframework/platform/internal/privatefile 0.008s
ok github.com/carlosframework/platform/internal/provision 23.192s
ok github.com/carlosframework/platform/internal/publicrelease 0.012s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.660s
ok github.com/carlosframework/platform/internal/relay 7.239s
ok github.com/carlosframework/platform/internal/release 26.143s
ok github.com/carlosframework/platform/internal/rollout 3.829s
ok github.com/carlosframework/platform/internal/rolloutwords 0.005s
ok github.com/carlosframework/platform/internal/rootaws 0.011s
ok github.com/carlosframework/platform/internal/rootparticipation 0.059s
ok github.com/carlosframework/platform/internal/rootstate 0.029s
ok github.com/carlosframework/platform/internal/runs 0.602s
ok github.com/carlosframework/platform/internal/s3compat 0.010s
ok github.com/carlosframework/platform/internal/safefile 0.013s
ok github.com/carlosframework/platform/internal/schedule 2.685s
ok github.com/carlosframework/platform/internal/seeds 0.013s
ok github.com/carlosframework/platform/internal/sim/hostlife 12.252s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.856s
ok github.com/carlosframework/platform/internal/status 0.084s
ok github.com/carlosframework/platform/internal/steering 0.012s
ok github.com/carlosframework/platform/internal/stopevidence 0.081s
ok github.com/carlosframework/platform/internal/svcrole 0.003s
ok github.com/carlosframework/platform/internal/telemetry 0.008s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.683s
=== internal_console@0 (83 s)
ok github.com/carlosframework/platform/internal/console 82.139s
=== internal_console@1 (37 s)
ok github.com/carlosframework/platform/internal/console 35.920s
=== internal_console@2 (43 s)
ok github.com/carlosframework/platform/internal/console 41.616s
=== internal_console@3 (38 s)
ok github.com/carlosframework/platform/internal/console 36.232s
=== internal_console@4 (37 s)
ok github.com/carlosframework/platform/internal/console 35.707s
=== internal_console@5 (35 s)
ok github.com/carlosframework/platform/internal/console 34.043s
=== internal_console@6 (37 s)
ok github.com/carlosframework/platform/internal/console 36.206s
=== internal_console@7 (43 s)
ok github.com/carlosframework/platform/internal/console 41.628s
=== root@0 (34 s)
ok github.com/carlosframework/platform 32.462s
=== root@1 (44 s)
ok github.com/carlosframework/platform 43.087s
=== root@2 (42 s)
ok github.com/carlosframework/platform 41.043s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.026s
=== RUN TestT4StorageProbeMinio
2026/10/21 22:23:26 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0ABABEE6CD593, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.023s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md| index 863319042..5106be866 100644 |
| --- a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| @@ -1,6 +1,6 @@ |
| # Self-serve offline moves: a tenant's customer moves their own instance |
| -Rev 2, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 answers every one. The changes are in §6 and §7 (the job, the return, cleanup, concurrency), §5 (the delegation contract) and the questions in §10. |
| +Rev 3, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. |
| Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says exactly what the customer's request may do. |
| @@ -13,7 +13,7 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| **Paul's rulings, 2026-10-21:** |
| - The customer confirms the downtime themselves; nobody at Tito approves. |
| - No size cap: the estimate is shown and the customer decides. (Qualified in §6.3: the archive's 256 MiB files limit, M5, is a separate ruling this design does not lift; §10 Q5.) |
| -- A failure after the account has gone offline puts it back in the old region automatically. |
| +- A failure after the account has gone offline puts it back in the old region automatically. (Rev 3 keeps this for every failure before `open`, which is where every data-carrying step and every check happens, and asks Paul to accept that after `open` the return is an operator's run, §10 Q3: a lossless automatic return after the new region has started cannot be proven with the claims-off machinery.) |
| - Scheduling is nice to have, not needed first. |
| - Offline moves stay available to Tito as an operator-run option regardless. |
| @@ -22,10 +22,10 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| **Goals.** |
| 1. One request from Tito's router starts a move that runs to completion with no operator step: stage while serving, stop, seal, flip, open, health check, finish, and the cleanup that follows. |
| 2. The holding page while offline, with the account's name and a minutes estimate, on every edge that answers for the host, on every domain the host has. |
| -3. Automatic return to the old region on any failure after the stop: lossless when the new region never admitted a write, and as a second offline move when it did (§7.5). When neither is possible the account is never left silently offline: the state says so and operators are paged. |
| +3. Automatic, lossless return to the old region on any failure before `open`, which is where every copy, verification and check happens (§7.5). After `open` the account is never left silently offline: the holding page stays until the app answers, the state says what happened, operators are paged, and the operator doors can run the return. |
| 4. An estimate before the customer confirms, and a better one once the data is staged. |
| 5. Tito's router keeps its calls (prepare, status, cut, abort) and state words. It needs one new credential and reads two new fields of the status view (§6.1). |
| -6. Only an owner-delegated credential can start, cut or cancel a self-serve move (§5). A member-minted or instance-bound token cannot. |
| +6. Only an owner-delegated credential can start, cut or cancel a self-serve move (§5). A member-minted or instance-bound token cannot. Operators keep their doors on every move, self-serve included: they are the recovery path. |
| 7. Moves of different instances run at once where they share no box; moves that share a box queue, and no move's stop evidence is starved by another's copy (§7.1). |
| 8. Every trap the first production moves hit (#339 and its siblings) is a refusal before the account goes offline, or is carried. |
| @@ -41,7 +41,7 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| **Anti-goals** (a failure even if it technically works). |
| - A move that completes with data the app cannot read, or without its config or key (the kpmg outage class). The manifest and the landing checks are the guard, unchanged. |
| -- A customer write lost to a return. The return is lossless by proof (§7.5) or it is a full move of the new data, never a guess. |
| +- A customer write lost to a return. The platform returns an account on its own only while the source still holds every copy and the target has never run (§7.5); after that a return is a move an operator decides on, never a guess. |
| - A customer left offline with nothing happening. Every wait has a deadline and a recovery (§7.2); the only state with no automatic action pages operators and says so in the view. |
| - A credential that can move an instance without an account owner having said so. |
| - Request text on the holding page other than the validated display name (§6.4). |
| @@ -61,11 +61,11 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| 1. Tito's residency tab lists the armed regions from our console's `/regions`, as today. The customer picks one. Tito shows the confirm page with the estimate from `GET .../instances/{host}/move?estimate=1®ion=uk` (§6.3): "about N minutes offline", or the refusal if the move cannot be offered (over the files limit, no return lane, a region with two boxes). The customer may pick a time (Tito's field) and confirms. |
| 2. At the time, Tito's router calls `POST .../instances/{host}/move {"region":"uk","display_name":"Paul's account","requested_by":"<tito user id>"}` with its move-capable token (§5). The console answers 202 with the status view, state `preparing`. |
| -3. **Preparing.** The job takes the move's box slots (§7.1), proves both staging lanes (§7.3), stages while the instance serves until caught up (§6.2), runs the pre-checks (§7.3). When all pass the state is `ready` with the refined estimate. Tito polls `GET .../move` as today. |
| +3. **Preparing.** The job reserves the move's boxes (§7.1), proves the staging lane (§7.3), stages while the instance serves until caught up (§6.2), runs the pre-checks (§7.3). When all pass the state is `ready` with the refined estimate. Tito polls `GET .../move` as today. |
| 4. Tito freezes its own account writes (its `frozen` step) and calls `POST .../move/cut`. The console answers 202, state `cutting`. |
| 5. **Cutting.** The job restages once more, then runs `start` (every source box holds, fences, reports), `seal` (tombstone, final round), `flip` (the landing's ensures and checks), `open`, the health check (§7.4), `finish`. From `start` until the target serves, every edge answers for the host with the holding page (§6.4). Expect 2.5 to 3 minutes for a 1.6 to 3.7 GB replica. |
| 6. **Done.** The state goes to `none`, `last.outcome` is `done` with the new region, and Tito's sweep marks its row done and unfreezes. The residency tab shows the new region. |
| -7. **Failure.** Before open, any refusal or deadline aborts: the record returns to the source, the tombstone lifts, the source reopens, `last.outcome` is `aborted` with `last.reason`. After open, a failed health check returns the account (§7.5): losslessly when the new region never admitted a write, else as a second offline move back; the view stays `cutting` with `reason` "returning" throughout, so Tito sees one operation. `last.outcome` is then `returned` with the reason. If no automatic return is possible (§7.5, case C) the view says `cutting` with `reason` "recovery needs an operator" and operators are paged; the account is where the evidence says, and nothing is deleted. |
| +7. **Failure.** Before `open`, any refusal or deadline aborts: the record returns to the source, the tombstone lifts, the source reopens, `last.outcome` is `aborted` with `last.reason`, and Tito shows `failed` with that reason. After `open` (§7.5) the account is in the new region by record; if it does not answer within the health window the holding page stays up ("...taking longer than expected. Please check back in a few minutes."), operators are paged, and the move completes as `done` with `last.warning` naming the failure; Tito shows the new region with the warning. An operator then fixes forward or runs the offline move back, with the runbook. |
| 8. **Cancel.** `DELETE .../move` aborts in `preparing` and `ready` (nothing was offline). In `cutting` it is refused (409): the customer confirmed, and a cut either completes or returns on its own. |
| ## 5. Who may ask: the move-capable token |
| @@ -76,7 +76,7 @@ M18 closed the move doors to instance-role tokens because a member-minted token |
| - **Checking.** The self-serve mutations (prepare, cut, cancel) resolve the principal as every door does, then require: an unrevoked instance token of exactly this account and app, with `moves=1`. `Admin` is refused here (operators have the operator doors). Anything else gets the move family's byte-identical 404 after authentication and before any move evidence is read, the family's concealment convention; an invalid credential keeps its existing answer. The status read stays open to every instance token (M18 kept it). |
| - **What it grants.** The token is app-wide, so it can move every instance of the app. The platform does not verify Tito's customer; it verifies that this account's owner delegated moving to this credential. Tito's confirm page is Tito's responsibility, and the design says so. |
| - **Audit.** The move's `requested_by` is `token:<token id>` plus the request's `requested_by` as an opaque suffix the platform does not interpret; both go in the move log. |
| -- **Boundaries.** A self-serve mutation touches only a self-serve move (`self_serve: true` in the move); it is refused (409) against an operator-created move, and the operator doors are refused against a self-serve move except `status`. A job in flight continues after the credential is revoked: recovery never needs the credential. |
| +- **Boundaries.** A self-serve mutation touches only a self-serve move (`self_serve: true` in the move) and names it: `cut` and `cancel` carry the move's `id`, and a request naming another id, or none, is refused (409 "this is not the move you confirmed") so a delayed request never acts on a later move. It is refused against an operator-created move. The operator doors act on every move, self-serve included: `status`, `abort` before open, `finish`, and after open the operator's offline move back; the job yields to an operator's transition it finds already made. A job in flight continues after the credential is revoked: recovery never needs the credential. |
| - **Scope of the M18 answer.** With the switch off, the legacy doors still admit instance tokens as today; this design does not change that path and does not claim every move mutation now needs owner delegation. It claims it for this door. |
| This is the F1 shape ("move-capable service authority explicitly owner-delegated") for one capability. My call; §10 Q1. |
| @@ -90,16 +90,16 @@ The legacy move doors keep their paths and roles. With `CARLOS_SELF_SERVE_MOVES= |
| | call | legacy today | self-serve offline | |
| |---|---|---| |
| | `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}` | legacy prepare | begin: slots, lanes, staging, pre-checks; needs `moves=1` | |
| -| `GET .../move` | `{state, target, updated_at}` | plus `kind: "offline"`, `id`, `estimate_seconds`, `offline_at`, `reason`, `last: {id, outcome, reason, from, to, at}` | |
| +| `GET .../move` | `{state, target, updated_at}` | plus `kind: "offline"`, `id`, `estimate_seconds`, `offline_at`, `reason`, `last: {id, outcome, reason, warning, from, to, at}` | |
| | `GET .../move?estimate=1®ion=uk` | n/a | the estimate, or the refusal, for a move not yet requested | |
| -| `POST .../move/cut` | legacy cut | restage, start, seal, flip, open, health, finish; needs `moves=1` | |
| -| `DELETE .../move` | legacy abort | abort in preparing or ready; 409 in cutting; needs `moves=1` | |
| +| `POST .../move/cut {"id"}` | legacy cut | restage, start, seal, flip, open, health, finish; needs `moves=1` and the move's id | |
| +| `DELETE .../move {"id"}` | legacy abort | abort in preparing or ready; 409 in cutting; needs `moves=1` and the id | |
| -**The view's words** are the legacy ones, `none`, `preparing`, `ready`, `cutting`, mapped from the record's offline states and the job's phase (§7, persisted): `offline-staging` with the job in slots, lanes, staging or pre-checks is `preparing`; staged, checked and holding a slot is `ready`; everything from `cut` to the end, the return included, is `cutting`; cleared is `none`. `reason` names the current step or the refusal in a recorded sentence. `last` carries the outcome of the last move (`done`, `aborted`, `returned`), its reason, `from`, `to` and `at`. Tito's change: read `last.outcome` and `last.reason` when `state` returns to `none` from `cutting` (today it reads only `state`; Goal 5). |
| +**The view's words** are the legacy ones, `none`, `preparing`, `ready`, `cutting`, mapped from the record's offline states and the job's phase (§7, persisted): `offline-staging` with the job in slots, lane, staging or pre-checks is `preparing`; staged, checked and holding its slots is `ready`; everything from `cut` to `finish` is `cutting`; cleared is `none`. `reason` names the current step or the refusal in a recorded sentence. `last` carries the outcome of the last move (`done`, `aborted`), its reason or warning, `from`, `to` and `at`. Tito's change: read `last.outcome`, `last.reason` and `last.warning` when `state` returns to `none` from `cutting` (today it reads only `state`; Goal 5). |
| **Refused before anything is written:** a replica or a home with replicas; a seeded record; a run in flight; a move in flight (the existing 409 sentences); the same region; a source or target region with more than one box; a region whose boxes lack `offline-move-1` or `offline-stage-1`; a target region whose boxes' `secrets_key_fp` differs from the console's (R9); files over the archive limit (§6.3); a `start_at` other than now ("scheduled starts are not available yet"); a host with no record (404); a token without `moves=1` (404). |
| -**Duplicate requests.** A second prepare for the same host while a self-serve move is in flight answers 409 with the existing "a move to X is already ..." sentence; cut and cancel are idempotent against their own state (a second cut during `cutting` answers 202 with the view). |
| +**Duplicate requests.** A second prepare for the same host while a self-serve move is in flight answers 409 with the existing "a move to X is already ..." sentence; cut and cancel name the move id (§5) and are idempotent against their own state for that id (a second cut of the same id during `cutting` answers 202 with the view). |
| ### 6.2 Catch-up and readiness |
| @@ -115,30 +115,39 @@ The legacy move doors keep their paths and roles. With `CARLOS_SELF_SERVE_MOVES= |
| From `start` until the target's row is ready, the host is held on every source box and has no ready row anywhere. Today every edge answers a platform-domain host with the "not here yet" 503 retry page (`internal/edge/pending.go`) and a customer domain with 404. This design adds a maintenance answer, carried on the existing box to edges-file to edge channel: |
| -- **Publication.** Every box that holds the host (every source-region box; and the target box from `flip` until its row is ready) lists it in a new edges-file field, `offline_moves: [{host, aliases, move_id, offline_at, until, display_name}]`, separate from `routes` so no decoder counts it as served (the seal door's "still listed" check reads `routes`; the builder audits every edges-file reader). `aliases` are the host's alias rows, so every custom domain gets the page. `until` is `offline_at + 3 × estimate_seconds`, a self-expiry. A box stops publishing the entry when the record no longer carries that move id or when its own row for the host is ready. |
| +- **Publication.** Every box that holds the host (every source-region box; and the target box from `flip` until its row is ready) lists it in a new edges-file field, `offline_moves: [{host, aliases, move_id, record_seq, offline_at, estimate_seconds, display_name}]`, separate from `routes` so no decoder counts it as served (the seal door's "still listed" check reads `routes`; the builder audits every edges-file reader). `aliases` are exactly the alias rows the edges writer would publish for the host as a served route, under the same eligibility rules (DNS pointing, ownership, wildcard handling), so every custom domain that would reach the account gets the page and no other. A box stops publishing the entry when the record no longer carries that move id or when its own row for the host is ready. |
| - **Rendering.** An edge answers a request for a host or alias that appears in a fresh `offline_moves` entry, from any publisher, and for which no fresh edges file lists a ready route, with 503, `Retry-After: 30`, `noindex`, and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from `until` and the estimate, counting down; without a display name, "This account is temporarily offline. ...". A ready route anywhere wins over a holding entry (precedence), so the window between `open` and the first edges-file refresh is the pending page's known 503-then-retry window and nothing worse. The source box's own edge answers its held rows the same way from its registry (the home map excludes a box's own publication). The page reuses the pending page's safe reload mechanics (location.replace, sessionStorage) with its give-up replaced by `until`. |
| -- **TLS.** A host in a holding entry is allowed for TLS as a served host is, so the page is reachable on HTTPS with the host's mirrored certificate. |
| +- **TLS.** A host or alias in a holding entry is allowed for TLS exactly as it would be as a served route (the same `Allowed` decision, wildcard and mirrored certificates included), so the page is reachable on HTTPS; the entry never widens what the host could already present. |
| - **The display name.** Tito supplies it per request; the console validates it (at most 80 characters, printable, no control characters, no newlines) and stores it in the move; the edge renders it through contextual template escaping. It is the one piece of request-supplied text the page shows, by design (the customer named their own account), and it is shown to unauthenticated visitors, as the account's own pages are. Nothing else from the request, no header, path, diagnostic or `requested_by`, reaches the page. |
| +- **Ordering and freshness.** Every `offline_moves` entry and every route carries the record's `record_seq` at publication. An edge prefers, for a host, the freshest publication (edges file younger than the routes' own freshness rule, 2 minutes) with the highest `record_seq`; a ready route at a `record_seq` at or above the holding entry's wins; a holding entry at a higher `record_seq` than a stale ready route wins. The home map's last-good behaviour on a read failure applies to both. The target box's own edge serves its local ready row at once (local rows win locally), so a probe sent to the target answers from the app whatever other edges still show. |
| +- **Validity.** An entry is valid while the publishing box's record still carries the move and the box republishes it (every pass); there is no fixed expiry. The displayed estimate is separate from validity: past `offline_at + estimate` the page says "...is taking longer than expected. Please check back in a few minutes." and keeps retrying. |
| +- **Reopening.** The public reopens when the target's row is ready and its publication reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. |
| - **The health probe** (§7.4) goes to the target box directly, by its address from its edges file with the host as SNI, so it reaches the app whatever the edges show. |
| ## 7. The job |
| -A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape of the other console passes), every 5 seconds, over the console's `self_serve_moves` table: one row per move, keyed by move id, with `host`, `phase`, `phase_started_at`, `deadline`, `slots`, `display_name`, `estimate`, `health` (the observations so far), `return_of` and `outcome`. The record carries the offline states as today plus `move.self_serve: true`, `display_name`, `offline_at` and `estimate_seconds` for the edges. Each tick advances each move by at most one step, judged from the evidence the operator doors read today (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the door bodies refactored into functions the doors and the job share. Two console processes never run the pass at once (the existing single-console assumption of the other passes). The job is idempotent and restart-safe: every phase's start, deadline and observations are in its row, and a tick that finds a phase's evidence already satisfied moves on. |
| +A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape of the other console passes), every 5 seconds, over the console's `self_serve_moves` table: one row per move, keyed by move id, with `host`, `account`, `app`, `phase`, `phase_started_at`, `deadline`, `display_name`, `estimate`, `health` (the observations so far) and `outcome`. The record carries the offline states as today plus `move.self_serve: true`, `display_name`, `offline_at` and `estimate_seconds` for the edges. Each tick advances each move by at most one step, judged from the evidence the operator doors read today (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the door bodies refactored into functions the doors and the job share. Two console processes never run the pass at once (the existing single-console assumption of the other passes). The job is idempotent and restart-safe: every phase's start, deadline and observations are in its row, and a tick that finds a phase's evidence already satisfied moves on. |
| ### 7.1 Slots and concurrency |
| -Today one `control/offline-moves/active.json` holds a deployment-wide lock: a staged or moving instance blocks every other one. What it protects, beyond the record's one move: the box's moves pass holds one lock for the whole pass, so a long stage round (142 s measured at 3.7 GB) on a box delays that box's stop-report refresh for another move, past the doors' 2-minute freshness allowance. |
| +Today one `control/offline-moves/active.json` holds a deployment-wide lock with an acquisition-to-record grace period and a reread of uncertain commits (`offlineBegin`): a staged or moving instance blocks every other one. Beyond the record's one move, it protects the box's moves pass, which holds one lock for the whole pass, so a long stage round (142 s measured at 3.7 GB) would delay that box's stop-report refresh for another move past the doors' 2-minute freshness allowance. |
| -This design replaces the deployment lock with **per-box slots** in the same file: `{box: move_id}` for every participating box of every move past `preparing`'s queue. A move's participants are every box of its source region and the target box (one each in this version). The console takes the slots in the same record write that begins staging and releases them when the move clears, after `finish` or an abort, and only after any return has cleared too. A move whose participant already holds a slot waits in `preparing` before its first stage round, with `reason` "waiting for another move on this box", and takes the slots in order of request. Operator moves take slots the same way (their stage door). Moves of instances on disjoint boxes run at once. Within a box, the one move in flight is the only one staging, stopping, sealing or landing there, so the stop-report refresh is never behind another move's copy. A slot held by a move that no longer exists (a cleared record, a console restart mid-write) is released by the pass when it finds no row for the id; the file is written with the record's CAS discipline (read, compare, write). |
| +This design keeps that object and its discipline and makes it **per box**. `active.json` holds `reservations: {box: {move_id, host, reserved_at}}`, one entry per participating box of every move from reservation to release. A move's participants are every box of its source region and the target box (one each in this version). The protocol, in order: |
| + |
| +1. **Reserve.** The door (self-serve or operator stage or start) reads `active.json`, refuses if any participant is reserved by another move id, writes every participant's entry in one conditional write (the object's ETag), and only then writes the record's transition. A reservation whose record transition has not appeared within the grace period (2 minutes, today's) is reclaimable by a later door or the job, but only after rereading the record and the console's `self_serve_moves` row and finding neither names the id; an unreadable record or row fails closed (no reclaim). |
| +2. **Hold.** The reservation is held through every phase, through `finish`, and through an operator's recovery after open until the operator's `finish`. |
| +3. **Release.** The job (or the operator's finish door) removes the entries in one conditional write after the record's move has cleared and the row's outcome is written; a crash between the two leaves a reservation the next tick releases on the same evidence. |
| + |
| +A move whose participant is reserved waits in `preparing` before any lane probe or stage round, with `reason` "waiting for another move on this box", in order of request. Moves of instances on disjoint boxes run at once. Within a box, the one move in flight is the only one staging, stopping, sealing or landing there, so the stop-report refresh is never behind another move's copy. Cleanup (§7.6) takes a reservation for the host's box for its duration too, so no move, recreation through the create door (which already refuses a host whose replica holds data) or operator move starts under it. |
| ### 7.2 Deadlines and the transition table |
| -Every wait has a deadline derived from the estimate's terms with a floor, and a stall rule: a phase whose evidence has not advanced (bytes, reports, items) for 5 minutes is treated as expired. Expiry before `open` aborts; after `open` it decides by §7.5. |
| +Every wait has a deadline derived from the estimate's terms with a floor; a stage round reports only when it ends, so there is no sub-round stall rule, and a round past its deadline (max(10 min, 3 × its estimate)) counts as a failed round. Expiry before `open` aborts; after `open` it is §7.5. |
| | phase (view) | waits for | deadline | on expiry | |
| |---|---|---|---| |
| -| slots (`preparing`) | the participants' slots | none; `reason` says waiting | 24 h, then abort "not started in time" | |
| -| lanes (`preparing`) | both lane probes (§7.3) | 5 min | abort, reason names the lane | |
| +| reserving (`preparing`) | the participants' reservations | none; `reason` says waiting | 24 h, then abort "not started in time" | |
| +| lane (`preparing`) | the lane probe report (§7.3) | 5 min | abort, reason names the lane | |
| | staging (`preparing`) | a matching round under the ready rule | 30 min or 10 rounds (§6.2) | go ready with the remaining bytes in the estimate | |
| | pre-checks (`preparing`) | target-ready and source reports | 5 min | abort, reason names the check | |
| | ready | the cut | 24 h | abort "not confirmed in time" | |
| @@ -146,48 +155,47 @@ Every wait has a deadline derived from the estimate's terms with a floor, and a |
| | stopping (`cutting`) | every source box's report: held, fenced, quiesced, no process | max(5 min, 2 × stop estimate) | abort | |
| | sealing (`cutting`) | the sealer's seal report | max(10 min, 3 × final-copy estimate) | abort | |
| | landing (`cutting`) | the landing report with every item OK | max(10 min, 3 × landing estimate) | abort | |
| -| opening (`cutting`) | the target's opened report and ready row | max(3 min, 2 × open estimate) | §7.5 by admission evidence | |
| -| health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | §7.5 by admission evidence | |
| +| opening (`cutting`) | the target's opened report, or its ready row in its edges file | max(3 min, 2 × open estimate) | page; stay in opening; the record names the target; the holding page stays while the target still publishes it (§7.5) | |
| +| health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | `finish` with `last.warning`; page (§7.5) | |
| | finishing (`cutting`) | every box's finish report | 10 min | the move stays `cutting` with the reason; page after 10 min; retry each tick (nothing is lost: the data is live on the target) | |
| | reverting (`cutting`) | abort's reports | 10 min | page; retry each tick; the source's copy was never removed | |
| -| returning (`cutting`) | the return move's own phases | its own table | the return's own rules; no second return (§7.5) | |
| -`abort` before `open` is the existing abort (`offline-reverting`), lossless: the source's database was never removed and the tombstone lifts. A lost response to `open` is an uncertain commit: the next tick rereads the record and the target's evidence before deciding, never assumes. |
| +`abort` before `open` is the existing abort (`offline-reverting`), lossless: the source's database was never removed, the target never took a lease, and the tombstone lifts. `open` is the point of no automatic return (§7.5). A lost response to any door write is an uncertain commit: the next tick rereads the record before deciding, never assumes. If `CARLOS_SELF_SERVE_MOVES` is turned off with moves in flight, the job keeps running them to completion and admits no new one. |
| ### 7.3 Lanes and pre-checks before anything goes offline |
| -**Lanes.** The forward move stages from the source box into the target region's data bucket under `moves/<host>/` (source role writes the target bucket's `moves/*`); the return (§7.5) stages from the target box into the source region's bucket (target role writes the source bucket's `moves/*`). Both must exist before a move starts, because a move that cannot return must not start. Each box proves its lane by a probe on its moves pass once the record is staging: it writes and deletes `moves/<host>/<id>/.lane` in the other region's bucket and reports the result. The console requires both reports before the first stage round. M17's push-only shape stands: no role gains access to a sibling's `instances/*`. |
| +**The lane.** The forward move stages from the source box into the target region's data bucket under `moves/<host>/` (source role writes the target bucket's `moves/*`). The source box proves the lane on its moves pass once the record is staging: it puts, lists and deletes one object under `moves/@lanes/<id>/` in the target bucket, a reserved namespace no consumer reads (`promoteStagedReplica` reads `moves/<host>/`, and `@`-prefixed names are the platform's own, as `moves/@offline/` is), and reports the result with the request's id; a crash between put and delete leaves nothing a wake could promote. The console requires the report before the first stage round. The reverse lane (target role to the source bucket's `moves/*`) is not a prerequisite of a self-serve move in this version, because no automatic return runs after `open`; it is what an operator's return needs, and the infrastructure provisions both (§9). M17's push-only shape stands: no role gains access to a sibling's `instances/*`. |
| -**Clean, then staged.** Before the first stage round the target bucket's `moves/<host>/` and `instances/<host>/` must hold no current objects and no `lock.json`, except the leftovers of a return onto an originating box (§7.5) which are named and accepted. After staging, `moves/<host>/` holds exactly this move's staged objects (the stager's receipts, bound to the move id, bucket and region); foreign objects refuse. |
| +**Clean, then staged.** Before the first stage round the target bucket's `moves/<host>/` and `instances/<host>/` must hold no current objects and no `lock.json` (versions and delete markers left by earlier moves are fine, as the runbook says). After staging, `moves/<host>/` holds exactly this move's staged objects (the stager's receipts, bound to the move id, bucket and region); foreign objects refuse. |
| **Target-ready report.** The target box, on its moves pass once the record is staging and again when asked at cut, writes `control/offline-moves/<host>/<id>/target/<box>.json` with facts, and the console decides. Facts: no route or timers for the host; `/opt/carlos/live/<host>` absent, or present and naming the build the record's channel resolves to; `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db` absent; `/data/programs/<host>.json` absent, or present and naming that same build; `/data/.offline-moves/<host>` absent or holding only earlier moves' directories; free disk at least twice the replica's size; the target bucket prefixes as above; capabilities and switches as required; `secrets_key_fp`. A stale live link naming another build is a refusal with that reason (#339's cause, caught before the stop). |
| -**Source report.** The source box's existing report gains: the instance's extra app directories beside its files directory (#339 point 4) with their sizes; `files_bytes`; the build the row runs. Empty extra directories are carried (the archive gains directory entries, restored at the same relative path under the instance directory); a non-empty one refuses with the reason naming it. |
| +**Source report.** The source box's existing report gains: the instance's extra app directories beside its files directory (#339 point 4) with their sizes; an **archive eligibility** result from a dry run of the archive's own `Build` rules over the files directory and the env file (the size cap with the env file counted, the entry cap, unsupported file types, unreadable paths), refreshed at cut; `files_bytes`; the build the row runs; and whether root's stage path is armed on the box (`carlos-stage.path` active), which the target box reports too. A non-empty extra directory refuses before the stop with the reason naming it. An empty one is removed by the source box at `start`, after the stop, as the operators did with Paul's approval on 2026-10-21 (the app recreates it on the target); nothing is archived outside the files directory, so the archive's layout contract is unchanged. An eligibility failure refuses before the stop with the archive's own reason. |
| **Build.** The channel's build must be the one the source runs; a promote during the move fails the landing's `release` item as today and the job aborts. |
| ### 7.4 The health check after open |
| -"Healthy" is: the app answers on the target. The probe is the manifest's ready path (`moves.ready_path`, `Manifest.ReadyPathOrDefault`, the path the legacy move used; Tito sets a path that answers 2xx without a session, as the runbook did with `/signin`), requested from the console to the target box directly (its address from its edges file, the host as SNI and `Host`), expecting a 2xx within 30 s per attempt; a 3xx is not a pass (the ready path's contract is a 2xx). Two passes 30 s apart, inside the deadline of §7.2, and the landing's manifest checks remain mandatory before it (the health check supplements them, it does not replace them). The console records the offline time from `start` to the first pass with the staged sizes, for the estimate's rates. Application checks beyond the ready path (the store reachable, custom domains, mail) are the tenant's: the runbook's broader verification is operator work this version does not automate, and Tito's ready path can cover what it cares about. |
| - |
| -### 7.5 The return |
| +"Healthy" is: the app answers on the target. The probe is the manifest's ready path (`moves.ready_path`, `Manifest.ReadyPathOrDefault`, the path the legacy move used; Tito sets a path that answers 2xx without a session, as the runbook did with `/signin`), requested from the console to the target box directly (its address from its edges file, the host as SNI and `Host`), with a unique query value and `Cache-Control: no-cache` so the edge's cache cannot answer it, no redirects followed (a 3xx is not a pass), a 30 s timeout per attempt, and the response's platform headers proving the target box and the build the seal recorded served it (no backhaul). Two passes 30 s apart, inside the deadline of §7.2. The landing's manifest checks remain mandatory before `open` (the health check supplements them). The console records the time from `start` to the first pass with the staged sizes, for the estimate's rates. Application checks beyond the ready path (the store reachable, custom domains, mail) are the tenant's; Tito's ready path can cover what it cares about. |
| -The question that decides the return is **whether the target ever admitted a write**, read from the target bucket, not from the box: `instances/<host>/lock.json` (absent, or held or released by the target box) and the newest LTX TXID under `instances/<host>/` against the seal's `FinalTXID`. Three cases. |
| +### 7.5 Before open and after open |
| -**A. No admission** (no lock ever taken by the target, or taken and released with no LTX beyond `FinalTXID`): the opened report never came, or the app did not start, or it started and answered nothing before the deadline. The return is **revert**, a new transition `OfflineRevert` from `offline-open` to `offline-reverting` that abort's machinery already drives from landed, extended by one step: the source boxes **un-park** what `open` parked (the park is a rename into `/data/.offline-moves/<host>/<id>/source/`; the reverse rename is journaled the same way), the target retires its row and parks what it extracted (as abort from landed does) and lifts any lease it took, the tombstone lifts, the record's region returns to the source, and `finish` records `last.outcome: returned`. Lossless by construction: the source's database and files were never removed, its bucket replica is untouched under the tombstone, and the target wrote nothing. |
| +**Before `open`, every failure returns the account automatically and losslessly.** That is the existing abort from any state up to `offline-landed`: the source's database and files were never removed (the park happens at `open`), its bucket replica is untouched under the tombstone, the target never took a lease and never ran, and the target parks what it extracted. Every data-carrying step (the final copy, the archive, the config render) and every check (the manifest items, the build, the target-ready facts, the lane) happens before `open`, so this is where failures that a return can fix are found. `abort` is the job's answer to every refusal and every deadline there, and the customer sees "failed, still in <region>" with the reason. |
| -**B. Admitted** (LTX beyond `FinalTXID`, or the lock held by a target writer now): the account served and may hold new data, so nothing on the source is current. The return is a **second offline move from the target to the source**, started by the job: forward `finish` completes (the move clears, outcome `done`, but the view keeps `cutting` with `reason` "returning" from the job's row), then the job begins the return with `return_of: <id>`, through every phase of the table: stage from the target (now an ordinary serving source: it has a live replica, a ready row and the build root staged for it), stop, seal, flip onto the originating box, open, health, finish. The originating box's target-ready report accepts exactly the forward move's leftovers and nothing else: the park directory `<fwd id>/source/` and `<fwd id>/sealer/`; the lifted `lock.json` (owner `moved:<fwd id>`, expired or lifted); the root-owned live link and `/data/programs/<host>.json` naming the build the return lands; and the forward's old replica objects under `instances/<host>/` in the originating bucket, all older than the forward's `open_at`, which the originating box deletes by exact key as the first step of its landing (its own bucket, its own role), keeping `lock.json`. The return's health check fails: **no third move**; the account stays on the originating box, `reason` "recovery needs an operator", paged. The return's outcome is `returned`, with the forward health failure as the reason. |
| +**At `open` the target becomes the account's home, and there is no automatic return.** Rev 2 tried to decide a post-open return from the target bucket's lease and LTX position; Astra showed that neither proves the target never accepted a write (litestream releases the lease even when its final sync failed; a file-only change leaves no LTX; an absent lock is "never taken" or "released"), and that a safe post-open revert needs an admission barrier and a stop protocol for the target that this machinery does not have (Phase 1 has them, through the claim). So after `open`: |
| -**C. Unknown** (the target bucket's evidence cannot be read, or the lock is held by the target and no LTX listing is available): no automatic action. The move stays `cutting` with `reason` "recovery needs an operator"; operators are paged at once and every 10 minutes; nothing is deleted anywhere; the tombstone stays (so the source cannot wake stale). The operator's runbook covers it: read the target's lease and position by hand, then revert or finish. |
| +- **The app answers (two passes of §7.4):** `finish`, outcome `done`. |
| +- **The opened report is missing or the app does not answer within the window:** the job calls `finish` once the opened report exists (the row is ready and the record names the target, so the account is where the record says), records `last.outcome: done` with `last.warning` naming the failure ("the account did not answer on <region> within <n> minutes"), and pages operators. While the target box still publishes the holding entry (its row not ready) the page stays up with the "taking longer than expected" sentence; once the row is ready, visitors reach the app and see what it answers. Without an opened report within its deadline the job stays in `opening`, pages, and leaves the record alone: the operator's `finish` and `status` doors and the runbook take over. |
| +- **The operator's return** is the offline move the other way, run with the operator doors and the runbook, after reading the target's state by hand. It needs the reverse lane (§9), the originating box passing the target-ready facts of §7.3 with the forward move's leftovers in place. Those leftovers are: the park directory `<fwd id>/source/` and `<fwd id>/sealer/`, the lifted `lock.json` (owner `moved:<fwd id>`), the root-owned live link and `/data/programs/<host>.json` naming the build, for a nested source the empty `/data/.instances/<host>` and `/data/instances/<host>` and its mount view (`retirePaths`, runbook §8), and the old replica objects under `instances/<host>/` older than the forward's `open_at`. The target-ready report lists each as a fact; **the operator's start door accepts exactly that set for a return of move `<fwd id>`** (the report names the move id each leftover belongs to) and refuses anything else. That acceptance is also the contract an automatic return would need, so it is built now, and used by an operator. |
| -The holding page continues through a revert and a return (the record's move id changes for a return; the entries follow the record). Cancel is refused throughout. |
| +The customer-facing promise is therefore: the account comes back to the old region on its own for any failure up to the point its data is verified and landed in the new region; after that, it is in the new region, a failure to answer is caught within minutes and paged, and a person moves it back if that is the right call. §10 Q3 asks Paul to accept this over rev 2's broader promise. |
| ### 7.6 Cleanup: who deletes what, from what evidence |
| -- **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept until now so the return has it): deleted by the **source box** on its moves pass, by exact key keeping `lock.json`, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{move_id, host, to, open_at, final_txid, tombstone_owner}`, and only when all hold at the time of deletion: the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; every object to delete has `LastModified` before `open_at`; and the box's own registry has no row for the host. A listing-to-deletion race is closed by deleting with the listed `VersionId` where the bucket is versioned (both are), and by refusing an object whose current version differs from the listed one. Each deletion is logged key by key in the move log. A return (§7.5 B) deletes the forward's old replica as its landing's first step, under the same conditions, and its own `retired.json` on the box it left governs that bucket's cleanup in turn. |
| -- **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after `finish`, never while a `retired.json` under it still has work. |
| -- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root's hourly pass removes them for a host whose record names another region, whose `retired.json` is at least 10 minutes old, and whose row is absent from the registry, never a path a live link or a running process still names. Off every critical path: a return accepts them (§7.5 B). One new census row at the version sweep's class. |
| -- **The staged objects of an aborted or reverted move** (`moves/<host>/<id>/` in the target bucket): removed by the stager on abort as today. |
| +- **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept so an operator's return has it): retired by the **source box** on its moves pass, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{move_id, host, account, app, incarnation, from, to, open_at, final_txid, tombstone_owner, inventory}` (the inventory is the exact keys and current version ids it listed at finish). Conditions, all at the time of deletion: the box holds the host's reservation in `active.json` (§7.1), so no move, return or recreation starts during the cleanup; the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; every key is in the inventory and its current version id is the inventory's; and the box's registry has no row for the host. Deletion is a plain `DeleteObject` per key, which on these versioned buckets writes a **delete marker** and keeps every version recoverable, as the runbook's cleanup did (`VersionId` deletes are never used: deleting the current version would make an older one current). A key whose current version differs from the inventory's is skipped and reported. Completion is written to `retired.json` (`cleaned_at`) and the reservation released. Each deletion is logged key by key in the move log. A key that appears again later (a later move's staging, a recreation) is that operation's, never this cleanup's: the inventory binds the obligation to the forward move's objects. |
| +- **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after `cleaned_at`. |
| +- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root's hourly pass removes exactly the paths `retired.json` names, when it is at least 10 minutes old, the reservation is the cleanup's or none, the row is absent from the registry, and no live link or running process names the build directory; never a path outside the inventory. Off every critical path: an operator's return accepts them (§7.5). One new census row at the version sweep's class. |
| +- **The staged objects of an aborted move** (`moves/<host>/<id>/` in the target bucket): removed by the stager on abort as today. |
| ### 7.7 Scheduling |
| @@ -197,7 +205,7 @@ Tito books the time and calls prepare when it arrives (its sweep's `scheduled` s |
| - The move log gains the job's lines: each phase, each refusal and its reason, the pre-check facts, the lane probes, the health probes, the timings, the admission evidence read for a return, every cleanup key. |
| - The view carries `reason`, `offline_at`, `estimate_seconds` and `last`; `carlos move offline status` shows them. |
| -- Paging through the existing alert sink: case C, a finishing or reverting phase past 10 minutes, a return's failed health, a slot held by an unknown move id, a lane probe failing on a configured pair. |
| +- Paging through the existing alert sink: a failed health check or a missing opened report after `open`, a finishing or reverting phase past 10 minutes, a reservation held by an unknown move id past the grace period, a lane probe failing on a configured pair, a cleanup that skipped a key. |
| - Member stream: each phase is an event on the app's platform log, as the legacy move's were. |
| ## 9. Rollout |
| @@ -205,7 +213,7 @@ Tito books the time and calls prepare when it arrives (its sweep's `scheduled` s |
| Platform: |
| 1. Build behind `CARLOS_SELF_SERVE_MOVES`, default off; copy rows pending until Paul's review; census overlay PROPOSED. |
| 2. Release and roll dev, then flagship, then Tito (roll order rule; boxes before consoles). |
| -3. Rehearse on dev with a Tito-shaped app (key file, store binding, config, custom domain) through Tito's own residency flow against the dev console: forward; cancel in ready; abort by a failed pre-check; a forced case A (the app refuses to start on the target) and its revert; a forced case B (the app starts, writes, then fails its ready path) and its return; the holding page on a third edge and on a custom domain; two moves on disjoint boxes at once; two queued on one box. |
| +3. Rehearse on dev with a Tito-shaped app (key file, store binding, config, custom domain) through Tito's own residency flow against the dev console: forward; cancel in ready; abort by a failed pre-check, by a stop deadline and by a landing failure; a forced post-open failure (the app refuses to start on the target) and its paging, then the operator's return with the runbook; the holding page on a third edge and on a custom domain; two moves on disjoint boxes at once; two queued on one box; the cleanup 24 hours on. |
| 4. Turn the switch on for Tito's console with Paul's go. |
| Tito's side (titogo; from titogo-f6's notes of 2026-10-21): |
| @@ -215,13 +223,13 @@ Tito's side (titogo; from titogo-f6's notes of 2026-10-21): |
| 4. Re-check the 2026-10-02 unsafe list on the region boxes: the edge door or `routes.home` for remote accounts, the secrets key on every region box (distributed 2026-10-03), the payments relay socket per region. |
| 5. `TITO_REMOTE_PLACEMENT=on`, env sync, restart the router. Publish the "Move your account" article. |
| -Infra: both staging lanes for every region pair the console arms (today IE to UK exists; UK to IE does not), as a region-host module output, with the one-box-per-region shape stated. |
| +Infra: both staging lanes for every region pair the console arms (today IE to UK exists; UK to IE does not), as a region-host module output, with the one-box-per-region shape stated. The forward lane is a prerequisite the door checks; the reverse one is what the operator's return needs. |
| ## 10. Questions for Paul (none block the plan; defaults stated) |
| - **Q1.** The move-capable token (§5) as the answer to M18 for this door, app-wide and owner-minted. Default: yes. |
| - **Q2.** Cancel during `cutting` is refused. Default: refused; the customer confirmed. |
| -- **Q3.** Case B's return moves the new data back as a second offline move with its own few minutes offline, and a failed return stops there and pages. Default: as stated. |
| +- **Q3 (changes a ruling).** Rev 3 returns the account automatically only for failures before `open`; after `open` a failure is caught and paged within minutes and the return is an operator's run of the offline move back. Rev 2's automatic post-open return could not be made lossless with claims-off machinery (§7.5). Default: as stated. The alternative is an admission barrier and a target stop protocol, which is Phase 1's claim; say if you want that built instead and the self-serve move waits for it. |
| - **Q4.** The source replica deletion 24 hours after done, by the source box, with the runbook's checks. Default: automatic. |
| - **Q5.** The archive's 256 MiB files limit (M5) stays and is a refusal before confirming. Raising it is separate work; say if you want it in this version. |
| - **Q6.** The display name is Tito-supplied per request, validated, shown to unauthenticated visitors. Default: yes. |
| @@ -235,13 +243,14 @@ Infra: both staging lanes for every region pair the console arms (today IE to UK |
| - The archive's directory entries for empty extra directories and their restore location. |
| - The estimate's storage and the rolling median; the phase timings' recording. |
| - The `moves` column migration, the mint flag and the instance-bound refusal. |
| -- The revert transition and the un-park step's journal; the return's leftover acceptance in the target-ready contract. |
| -- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; each pre-check refusal; each deadline; case A revert; case B return and its failed-health stop; case C paging; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry; two moves at once on disjoint boxes and two queued on one; cleanup refusing when the record moved again or an object is newer than `open_at`. |
| +- The return's leftover acceptance in the operator start door's target-ready contract, keyed by the forward move id. |
| +- The `record_seq` on routes and holding entries and the edges' precedence rule; the "taking longer" sentence's trigger. |
| +- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; a cut naming a stale id; each pre-check refusal (a stale live link, a non-empty extra directory, an ineligible archive, a failed lane probe); each deadline before open ending in a lossless abort; a post-open failure to answer ending in `done` with a warning and a page; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry and a holding entry winning over a stale route; two moves at once on disjoint boxes and two queued on one; a reservation reclaimed only on evidence; cleanup refusing when the record moved again, when a current version differs from the inventory, and when the reservation is held by a move. |
| ## 12. Risks |
| -- **The return (case B) has never been run.** It is the forward machinery in the other direction onto a box with named leftovers; it gets the same witness and its first real run is the dev rehearsal. |
| -- **Admission evidence.** Case A's losslessness rests on the target bucket's lease and LTX listing being truthful about writes; the sealer's `FinalTXID` is the reference. A target whose litestream never pushed an LTX after a write it accepted is case C by construction (the lock is held, no LTX): paged, not reverted. |
| +- **A post-open failure is a person's job.** The design catches it within minutes and pages; it does not fix it. If this happens often the next step is the admission barrier (Q3), not a shortcut. |
| +- **The operator's return has never been run.** It is the forward machinery the other way onto a box with named leftovers; the start door's acceptance of those leftovers gets the two-box witness, and the first real run is the dev rehearsal. |
| - **The estimate is an estimate.** Rates differ by region pair and object count; the rolling median narrows them; the page says "approximately". |
| - **Boxes that do not report.** A source box down at `start` stalls `seal`; the deadline aborts and the customer sees "failed, still in <region>" with the box named. Nothing is lost. |
| - **Tito's freeze.** Tito freezes its own writes before cut; a Tito bug there is Tito's; the platform's seal is final regardless (the tombstone, not Tito's freeze, is the stop). |