amadan
design: self-serve offline moves, rev 2 (Astra round 1 and the builder check)
The recovery half redesigned around one question, whether the target ever admitted a write: case A revert (lossless un-park), case B a second offline move back onto a box whose named leftovers are accepted, case C paged. Lanes proven both ways before a move starts; one box per region at both ends; the staging-vs-pre-check contradiction split into clean-then-staged; per-box slots replacing the deployment lock; a transition table with derived deadlines; box-side cleanup from a durable retired.json with the runbook's checks; the files limit (M5) as a refusal before confirming; the health contract on the manifest's ready path; the holding page's publication, precedence and TLS; the delegation contract spelled out; Tito's view fields.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+140
−98
docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md+140 −98
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (143 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 4.528s
ok github.com/carlosframework/platform/cmd/moveprobe 12.148s
ok github.com/carlosframework/platform/cmd/zddprobe 0.783s
ok github.com/carlosframework/platform/internal/activator 60.317s
ok github.com/carlosframework/platform/internal/agentsock 0.022s
ok github.com/carlosframework/platform/internal/alerting 0.009s
ok github.com/carlosframework/platform/internal/analytics 0.028s
ok github.com/carlosframework/platform/internal/appconfig 3.767s
ok github.com/carlosframework/platform/internal/apperr 0.162s
ok github.com/carlosframework/platform/internal/applog 0.323s
ok github.com/carlosframework/platform/internal/awsinv 0.013s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 1.207s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.015s
ok github.com/carlosframework/platform/internal/caps 0.016s
ok github.com/carlosframework/platform/internal/certwarm 0.014s
ok github.com/carlosframework/platform/internal/cgroup 0.790s
ok github.com/carlosframework/platform/internal/changelog 0.601s
ok github.com/carlosframework/platform/internal/claimactivation 1.484s
ok github.com/carlosframework/platform/internal/clamd 0.070s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.552s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.009s
ok github.com/carlosframework/platform/internal/dnsrec 0.006s
ok github.com/carlosframework/platform/internal/docsite 0.017s
ok github.com/carlosframework/platform/internal/domains 3.753s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.007s
ok github.com/carlosframework/platform/internal/edge 46.002s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.011s
ok github.com/carlosframework/platform/internal/fleetapi 0.221s
ok github.com/carlosframework/platform/internal/gate 2.171s
ok github.com/carlosframework/platform/internal/genselect 0.048s
ok github.com/carlosframework/platform/internal/geo 0.028s
ok github.com/carlosframework/platform/internal/geoip 0.044s
ok github.com/carlosframework/platform/internal/historyreadback 0.243s
ok github.com/carlosframework/platform/internal/host 139.469s
ok github.com/carlosframework/platform/internal/hostactors 7.462s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.012s
ok github.com/carlosframework/platform/internal/hostbox 3.759s
ok github.com/carlosframework/platform/internal/hostclaim 0.922s
ok github.com/carlosframework/platform/internal/hostenv 0.010s
ok github.com/carlosframework/platform/internal/hostevidence 0.016s
ok github.com/carlosframework/platform/internal/hostexec 6.463s
ok github.com/carlosframework/platform/internal/hostfence 0.059s
ok github.com/carlosframework/platform/internal/hostfiles 10.625s
ok github.com/carlosframework/platform/internal/hostfloor 0.025s
ok github.com/carlosframework/platform/internal/hostguard 1.065s
ok github.com/carlosframework/platform/internal/hostops 2.866s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.003s
ok github.com/carlosframework/platform/internal/hostquarantine 0.005s
ok github.com/carlosframework/platform/internal/hostrelocate 0.980s
ok github.com/carlosframework/platform/internal/hostresolve 0.248s
ok github.com/carlosframework/platform/internal/hostwrite 0.004s
ok github.com/carlosframework/platform/internal/infraview 0.005s
ok github.com/carlosframework/platform/internal/instancectl 5.292s
ok github.com/carlosframework/platform/internal/instanceexec 2.856s
ok github.com/carlosframework/platform/internal/instancefs 0.016s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.009s
ok github.com/carlosframework/platform/internal/layout 2.786s
ok github.com/carlosframework/platform/internal/lbhealth 0.008s
ok github.com/carlosframework/platform/internal/lease 10.164s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.783s
ok github.com/carlosframework/platform/internal/ledger 0.025s
ok github.com/carlosframework/platform/internal/mail 0.315s
ok github.com/carlosframework/platform/internal/maintenance 0.096s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.663s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.162s
ok github.com/carlosframework/platform/internal/metrics 0.014s
ok github.com/carlosframework/platform/internal/moves 22.063s
ok github.com/carlosframework/platform/internal/moves/archive 1.433s
ok github.com/carlosframework/platform/internal/objstore 0.514s
ok github.com/carlosframework/platform/internal/ops 0.347s
ok github.com/carlosframework/platform/internal/park 0.277s
ok github.com/carlosframework/platform/internal/plans 0.006s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 14.556s
ok github.com/carlosframework/platform/internal/publicrelease 0.011s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.970s
ok github.com/carlosframework/platform/internal/relay 7.216s
ok github.com/carlosframework/platform/internal/release 25.927s
ok github.com/carlosframework/platform/internal/rollout 4.118s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.013s
ok github.com/carlosframework/platform/internal/rootparticipation 0.061s
ok github.com/carlosframework/platform/internal/rootstate 0.031s
ok github.com/carlosframework/platform/internal/runs 0.512s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.008s
ok github.com/carlosframework/platform/internal/schedule 2.954s
ok github.com/carlosframework/platform/internal/seeds 0.022s
ok github.com/carlosframework/platform/internal/sim/hostlife 12.874s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.901s
ok github.com/carlosframework/platform/internal/status 0.084s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.294s
ok github.com/carlosframework/platform/internal/svcrole 0.006s
ok github.com/carlosframework/platform/internal/telemetry 0.008s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.008s
ok github.com/carlosframework/platform/internal/updatewords 0.012s
ok github.com/carlosframework/platform/internal/usage 0.008s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.823s
=== internal_console@0 (82 s)
ok github.com/carlosframework/platform/internal/console 79.445s
=== internal_console@1 (38 s)
ok github.com/carlosframework/platform/internal/console 35.573s
=== internal_console@2 (44 s)
ok github.com/carlosframework/platform/internal/console 41.389s
=== internal_console@3 (39 s)
ok github.com/carlosframework/platform/internal/console 36.219s
=== internal_console@4 (39 s)
ok github.com/carlosframework/platform/internal/console 35.742s
=== internal_console@5 (36 s)
ok github.com/carlosframework/platform/internal/console 33.543s
=== internal_console@6 (38 s)
ok github.com/carlosframework/platform/internal/console 35.436s
=== internal_console@7 (45 s)
ok github.com/carlosframework/platform/internal/console 42.191s
=== root@0 (33 s)
ok github.com/carlosframework/platform 31.245s
=== root@1 (45 s)
ok github.com/carlosframework/platform 42.813s
=== root@2 (42 s)
ok github.com/carlosframework/platform 40.479s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.025s
=== RUN TestT4StorageProbeMinio
2026/10/21 22:11:59 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0AB0C0D909335, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.023s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md| index 537f6af80..863319042 100644 |
| --- a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| @@ -1,18 +1,18 @@ |
| # Self-serve offline moves: a tenant's customer moves their own instance |
| -Rev 1, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. |
| +Rev 2, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 answers every one. The changes are in §6 and §7 (the job, the return, cleanup, concurrency), §5 (the delegation contract) and the questions in §10. |
| -Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says what the customer's request may do and what it may not. |
| +Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says exactly what the customer's request may do. |
| ## 1. Intent |
| -A Tito customer opens Account settings, Data residency, in Tito's dashboard, sees the regions our console lists as armed, picks one and clicks Move. They are told the account will be offline for a few minutes and how many, and they confirm; the customer has the final say, nobody at Tito or at ours is in the loop. The platform copies the account's data to the new region while it keeps serving, then takes it offline, finishes the copy, opens it in the new region and reopens it. From the moment it goes offline until it is back, the customer and their own visitors see a holding page: "Paul's account is temporarily offline. Please check back in approximately 3 minutes." If anything fails after the account has gone offline, the platform puts it back in the old region on its own, and the residency tab says so. Scheduling a move for later is nice to have; Tito's dashboard already offers a time, and the platform's start is "now" in this version. |
| +A Tito customer opens Account settings, Data residency, in Tito's dashboard, sees the regions our console lists as armed, picks one and clicks Move. They are told the account will be offline for a few minutes and about how many, and they confirm; the customer has the final say, nobody at Tito or at ours is in the loop. The platform copies the account's data to the new region while it keeps serving, then takes it offline, finishes the copy, lands it in the new region, checks it answers, and reopens it. From the moment it goes offline until it is back, the customer and their own visitors see a holding page: "Paul's account is temporarily offline. Please check back in approximately 3 minutes." If anything fails after the account has gone offline, the platform puts it back in the old region on its own, and the residency tab says so. Scheduling for later is nice to have; Tito's dashboard already offers a time, and the platform's start is "now" in this version. |
| -**Why now.** Tito's customer move has been off since 2026-10-02 (`TITO_REMOTE_PLACEMENT`): the pre-Phase-1 move lost the per-host key and config (#175 P1). The offline move is the one move proven on real customer data (three Tito instances, 2026-10-21, 2.5 to 3 minutes offline each) and it carries everything (§"The dependency list" of the offline plan). It needs no host claims, so it does not wait for the claim cohort, acceptance 10.1 to 10.6 or the F1 owner surface. Phase 1's live move is weeks away for Tito and refuses Tito's router outright (M18). |
| +**Why now.** Tito's customer move has been off since 2026-10-02 (`TITO_REMOTE_PLACEMENT`): the pre-Phase-1 move lost the per-host key and config (#175 P1). The offline move is the one move proven on real customer data (three Tito instances, 2026-10-21, 2.5 to 3 minutes offline each) and it carries everything (the offline plan's dependency list). It needs no host claims, so it does not wait for the claim cohort, acceptance 10.1 to 10.6 or the F1 owner surface. Phase 1's live move is weeks away for Tito and refuses Tito's router outright (M18). |
| -**Paul's rulings in the conversation of 2026-10-21:** |
| +**Paul's rulings, 2026-10-21:** |
| - The customer confirms the downtime themselves; nobody at Tito approves. |
| -- No size cap: the estimate is shown and the customer decides. |
| +- No size cap: the estimate is shown and the customer decides. (Qualified in §6.3: the archive's 256 MiB files limit, M5, is a separate ruling this design does not lift; §10 Q5.) |
| - A failure after the account has gone offline puts it back in the old region automatically. |
| - Scheduling is nice to have, not needed first. |
| - Offline moves stay available to Tito as an operator-run option regardless. |
| @@ -20,186 +20,228 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| ## 2. Goals, non-goals, anti-goals |
| **Goals.** |
| -1. One request from Tito's router starts a move that runs to completion with no operator step: stage while serving, stop, seal, flip, verify, open, finish, clean up. |
| -2. The holding page while offline, with the account's name and a minutes estimate, on every edge that answers for the host. |
| -3. Automatic return to the old region on any failure after the stop, including a failure after open within a bounded health window. |
| -4. The estimate before the customer confirms, and a better one once the data is staged. |
| -5. Tito's existing router code (prepare, status, cut, abort; states scheduled, preparing, ready, frozen, cutting, done, failed, aborted) keeps working with at most a token change on Tito's side. |
| -6. Only an owner-delegated credential can start a move (§5). A member-minted instance token cannot. |
| -7. Several customers can move at once, bounded, without one move blocking every other (today: one offline-move lock per deployment). |
| +1. One request from Tito's router starts a move that runs to completion with no operator step: stage while serving, stop, seal, flip, open, health check, finish, and the cleanup that follows. |
| +2. The holding page while offline, with the account's name and a minutes estimate, on every edge that answers for the host, on every domain the host has. |
| +3. Automatic return to the old region on any failure after the stop: lossless when the new region never admitted a write, and as a second offline move when it did (§7.5). When neither is possible the account is never left silently offline: the state says so and operators are paged. |
| +4. An estimate before the customer confirms, and a better one once the data is staged. |
| +5. Tito's router keeps its calls (prepare, status, cut, abort) and state words. It needs one new credential and reads two new fields of the status view (§6.1). |
| +6. Only an owner-delegated credential can start, cut or cancel a self-serve move (§5). A member-minted or instance-bound token cannot. |
| +7. Moves of different instances run at once where they share no box; moves that share a box queue, and no move's stop evidence is starved by another's copy (§7.1). |
| 8. Every trap the first production moves hit (#339 and its siblings) is a refusal before the account goes offline, or is carried. |
| **Non-goals.** |
| - Live (seconds-of-cut) moves: Phase 1, claims-on only. |
| - Object-store residency: the store bucket stays where it is (M3 is Phase 1's P8). |
| -- Moving a Tito controld account: it has no platform record and the door 404s; Tito's residency tab must refuse it first (Tito side, §9). |
| +- Moving a Tito controld account: it has no platform record and the door 404s; Tito's residency tab refuses it first (§9). |
| - Moving into or out of a region with host claims or Phase 1 on: refused as today (MV-O20). |
| +- Regions with more than one box, as source or target. The offline move lands on exactly one target box (MV-O22) and its return would land on the original box, so this version requires one box per region at both ends (Tito's regions are one box each). Lifting this is Phase 1's or region HA's work. |
| - Scheduling on the platform side: Tito books the time; the platform starts when called. |
| - Replica instances and seeded records: refused as the legacy door refuses them. |
| +- Raising the archive's files limit (M5, 256 MiB): a separate change; here it is a refusal before confirming (§6.3). |
| **Anti-goals** (a failure even if it technically works). |
| - A move that completes with data the app cannot read, or without its config or key (the kpmg outage class). The manifest and the landing checks are the guard, unchanged. |
| -- A customer left offline with nothing happening. Every wait in the job has a deadline, and every deadline ends in an abort before open or a return after it. |
| +- A customer write lost to a return. The return is lossless by proof (§7.5) or it is a full move of the new data, never a guess. |
| +- A customer left offline with nothing happening. Every wait has a deadline and a recovery (§7.2); the only state with no automatic action pages operators and says so in the view. |
| - A credential that can move an instance without an account owner having said so. |
| -- Text on the holding page that came from the request. |
| +- Request text on the holding page other than the validated display name (§6.4). |
| ## 3. Constraints |
| -- No change to the offline move's data path or its stop guarantee (the three layers of the offline plan). The job drives the same transitions in the same order with the same evidence. New code is the job, the door's authorization, the pre-checks, the holding page, the estimate, the return and the cleanup. |
| -- Claims-off only: the console refuses the door when `CARLOS_HOST_CLAIM` or `CARLOS_MOVES_PHASE1` is on (MV-O20), as every offline door does. Boxes publish `offline-move-1` and `offline-stage-1` only in that state. |
| -- Off by default. A console switch `CARLOS_SELF_SERVE_MOVES=offline` turns the door on; absent, the legacy move door behaves exactly as today. The moves copy gate keeps the default off while any new copy row is pending Paul's review. |
| -- Root never runs the job. Everything the job does is console-side reads and record transitions plus the boxes' existing moves pass. The one new root-side action (§7.6, the source's leftovers) runs in root's existing hourly pass under its own census row. |
| +- No change to the offline move's data path or its stop guarantee (the three layers of the offline plan). The job drives the same transitions in the same order with the same evidence, through the door bodies refactored into functions the operator doors and the job both call, byte for byte the same behaviour on the operator side. |
| +- Claims-off only: the console refuses the door when `CARLOS_HOST_CLAIM` or `CARLOS_MOVES_PHASE1` is on (MV-O20). Boxes publish `offline-move-1` and `offline-stage-1` only in that state. |
| +- Off by default. `CARLOS_SELF_SERVE_MOVES=offline` on the console turns the door on; absent, the legacy move door behaves exactly as today. The moves copy gate keeps the default off while any new copy row is pending Paul's review. |
| +- The console writes only records and the move log. Everything in a data bucket is written or deleted by the box whose role reaches it, on its moves pass, from durable evidence it holds (§7.6). The console's role cannot list or delete in a sibling region's data bucket, and this design does not ask for that grant. |
| +- Root runs nothing on the move's path. The one root action (§7.6, root-owned leftovers) is root's existing hourly pass, off every critical path, under its own census row. |
| - Plan 14 census: every new or changed unit that reads root state takes a row in a PROPOSED overlay loaded last, ruled on #280. |
| -- Copy: every user-facing sentence (the holding page, the door's refusals, the status view's reasons, the CLI) is a copy row, reviewed before the default can turn on. |
| +- Copy: every user-facing sentence (the holding page, the door's refusals, the view's reasons, the CLI) is a copy row, reviewed before the default can turn on. |
| - No tenant names in platform code, tests or copy (`shop.example.com`, "the account"). |
| ## 4. The customer's path, end to end |
| -1. Tito's residency tab lists the armed regions from our console's `/regions`, as today. The customer picks one and clicks Move. Tito shows the confirm page with the estimate it fetched from `GET .../instances/{host}/move?estimate=1` (§6.3): "about N minutes offline". The customer may pick a time (Tito's existing field) and confirms. |
| +1. Tito's residency tab lists the armed regions from our console's `/regions`, as today. The customer picks one. Tito shows the confirm page with the estimate from `GET .../instances/{host}/move?estimate=1®ion=uk` (§6.3): "about N minutes offline", or the refusal if the move cannot be offered (over the files limit, no return lane, a region with two boxes). The customer may pick a time (Tito's field) and confirms. |
| 2. At the time, Tito's router calls `POST .../instances/{host}/move {"region":"uk","display_name":"Paul's account","requested_by":"<tito user id>"}` with its move-capable token (§5). The console answers 202 with the status view, state `preparing`. |
| -3. **Preparing.** The job stages while the instance serves: the existing `stage` round, repeated until a round copies under the catch-up threshold (§6.2), then the pre-checks (§7.3). When both pass the state is `ready` and the view carries the refined estimate. Tito polls `GET .../move` as today. |
| -4. Tito freezes its own account writes (its existing `frozen` step) and calls `POST .../move/cut`. The console answers 202, state `cutting`. |
| -5. **Cutting.** The job runs `start` (every source box holds, fences, reports), `seal` (tombstone, final round), `flip` (the landing's ensures and checks), `open`, then the health check (§7.4), then `finish`. From `start` to the first 2xx on the target, every edge answers for the host with the holding page (§6.4). Expect 2.5 to 3 minutes for a 1.6 to 3.7 GB replica. |
| -6. **Done.** The state goes to `none`, `last_move` says `done` with the new region, and Tito's sweep marks its row done and unfreezes. The residency tab shows the new region. |
| -7. **Failure.** Before open, any refusal or deadline aborts: the record returns to the source, the tombstone is lifted, the source reopens, `last_move` says `aborted` with the reason, and Tito's status shows `failed` with that reason. After open, a failed health check within the window runs the return (§7.5): a second offline move back to the source region, run by the same job, with the same holding page. If the return itself cannot be started (§7.5 says when), the move is `done` in the new region with a `last_move.warning` naming the health failure, and operators are paged. |
| -8. **Cancel.** `DELETE .../move` aborts in `preparing` and `ready` (nothing was offline). In `cutting` it is refused: the customer confirmed, and a cut that has started either completes or returns on its own. |
| +3. **Preparing.** The job takes the move's box slots (§7.1), proves both staging lanes (§7.3), stages while the instance serves until caught up (§6.2), runs the pre-checks (§7.3). When all pass the state is `ready` with the refined estimate. Tito polls `GET .../move` as today. |
| +4. Tito freezes its own account writes (its `frozen` step) and calls `POST .../move/cut`. The console answers 202, state `cutting`. |
| +5. **Cutting.** The job restages once more, then runs `start` (every source box holds, fences, reports), `seal` (tombstone, final round), `flip` (the landing's ensures and checks), `open`, the health check (§7.4), `finish`. From `start` until the target serves, every edge answers for the host with the holding page (§6.4). Expect 2.5 to 3 minutes for a 1.6 to 3.7 GB replica. |
| +6. **Done.** The state goes to `none`, `last.outcome` is `done` with the new region, and Tito's sweep marks its row done and unfreezes. The residency tab shows the new region. |
| +7. **Failure.** Before open, any refusal or deadline aborts: the record returns to the source, the tombstone lifts, the source reopens, `last.outcome` is `aborted` with `last.reason`. After open, a failed health check returns the account (§7.5): losslessly when the new region never admitted a write, else as a second offline move back; the view stays `cutting` with `reason` "returning" throughout, so Tito sees one operation. `last.outcome` is then `returned` with the reason. If no automatic return is possible (§7.5, case C) the view says `cutting` with `reason` "recovery needs an operator" and operators are paged; the account is where the evidence says, and nothing is deleted. |
| +8. **Cancel.** `DELETE .../move` aborts in `preparing` and `ready` (nothing was offline). In `cutting` it is refused (409): the customer confirmed, and a cut either completes or returns on its own. |
| ## 5. Who may ask: the move-capable token |
| -M18 closed the move doors to instance-role tokens because a member-minted token could move an instance the member had no right to move. This design opens one door, the offline self-serve move, to a token that an account owner has explicitly made move-capable: |
| +M18 closed the move doors to instance-role tokens because a member-minted token could move an instance the member had no right to move. This design opens one door, the self-serve offline move, to a credential an account owner has explicitly made move-capable. The contract: |
| -- `carlos instance-token --moves` mints an instance token with `moves=1` in its `cli_tokens` row. Only a member whose role on the account is owner may mint it; the mint is audited ("minted a move-capable instance token for <app>"). Existing tokens have `moves=0` and cannot be upgraded; the owner mints a new one and Tito's router switches to it. |
| -- The self-serve door checks the token's row. A token without `moves=1` gets the move family's byte-identical 404, as a non-operator gets from the operator doors today. The status read stays open to every instance token (M18 kept it). |
| -- The request carries `requested_by`, an opaque identifier Tito supplies for the person who confirmed, and the console records it in the move's `requested_by` as `<token name>:<requested_by>` and in the move log. The platform does not verify Tito's user; Tito's confirm page is Tito's business. What the platform verifies is that an owner delegated moving to this credential. |
| -- Operators keep the operator doors (`carlos move offline ...`) unchanged. An operator-run move and a self-serve move of the same instance exclude each other through the record's one move. |
| +- **Minting.** `carlos instance-token --moves` mints an app-wide instance token with `moves=1` on its `cli_tokens` row. Only a person whose role on the account is owner may mint it, through the existing person-only mint door; the mint is audited with the token's id ("minted a move-capable instance token for <app>"). `--moves` is refused on an instance-bound token (#155): a token bound to one instance moving that instance is the member-minted shape M18 closed. Existing tokens have `moves=0` and cannot be upgraded; the owner mints a new one and Tito's router switches to it. Revoking the token is the existing revoke. |
| +- **Checking.** The self-serve mutations (prepare, cut, cancel) resolve the principal as every door does, then require: an unrevoked instance token of exactly this account and app, with `moves=1`. `Admin` is refused here (operators have the operator doors). Anything else gets the move family's byte-identical 404 after authentication and before any move evidence is read, the family's concealment convention; an invalid credential keeps its existing answer. The status read stays open to every instance token (M18 kept it). |
| +- **What it grants.** The token is app-wide, so it can move every instance of the app. The platform does not verify Tito's customer; it verifies that this account's owner delegated moving to this credential. Tito's confirm page is Tito's responsibility, and the design says so. |
| +- **Audit.** The move's `requested_by` is `token:<token id>` plus the request's `requested_by` as an opaque suffix the platform does not interpret; both go in the move log. |
| +- **Boundaries.** A self-serve mutation touches only a self-serve move (`self_serve: true` in the move); it is refused (409) against an operator-created move, and the operator doors are refused against a self-serve move except `status`. A job in flight continues after the credential is revoked: recovery never needs the credential. |
| +- **Scope of the M18 answer.** With the switch off, the legacy doors still admit instance tokens as today; this design does not change that path and does not claim every move mutation now needs owner delegation. It claims it for this door. |
| -This is the F1 shape ("move-capable service authority explicitly owner-delegated") for one capability, without the rest of F1. My call; Paul to confirm it answers M18 (§10, Q1). |
| +This is the F1 shape ("move-capable service authority explicitly owner-delegated") for one capability. My call; §10 Q1. |
| ## 6. The door, the states, the estimate and the holding page |
| -### 6.1 The door |
| +### 6.1 The door and the view |
| The legacy move doors keep their paths and roles. With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, their bodies run the self-serve offline move instead of the legacy online move: |
| -| call | today (legacy) | self-serve offline | |
| +| call | legacy today | self-serve offline | |
| |---|---|---| |
| -| `POST .../move {"region"}` | legacy prepare | offline stage + pre-checks, as a job; needs `moves=1` | |
| -| `GET .../move` | state none/preparing/ready/cutting | the same words, plus `kind: "offline"`, `estimate_seconds`, `offline_at`, `reason` | |
| -| `GET .../move?estimate=1` | n/a | the estimate for a move not yet requested (§6.3) | |
| -| `POST .../move/cut` | legacy cut | start, seal, flip, open, health, finish, as a job; needs `moves=1` | |
| +| `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}` | legacy prepare | begin: slots, lanes, staging, pre-checks; needs `moves=1` | |
| +| `GET .../move` | `{state, target, updated_at}` | plus `kind: "offline"`, `id`, `estimate_seconds`, `offline_at`, `reason`, `last: {id, outcome, reason, from, to, at}` | |
| +| `GET .../move?estimate=1®ion=uk` | n/a | the estimate, or the refusal, for a move not yet requested | |
| +| `POST .../move/cut` | legacy cut | restage, start, seal, flip, open, health, finish; needs `moves=1` | |
| | `DELETE .../move` | legacy abort | abort in preparing or ready; 409 in cutting; needs `moves=1` | |
| -Refused before anything is written: a replica or a home with replicas, a seeded record, a run in flight, a move in flight (409, the existing sentences), the same region, a region without `offline-move-1` and `offline-stage-1` on every box, a target region whose boxes' `secrets_key_fp` differs from the console's (R9), a controld-shaped host with no record (404), and a token without `moves=1` (404). |
| +**The view's words** are the legacy ones, `none`, `preparing`, `ready`, `cutting`, mapped from the record's offline states and the job's phase (§7, persisted): `offline-staging` with the job in slots, lanes, staging or pre-checks is `preparing`; staged, checked and holding a slot is `ready`; everything from `cut` to the end, the return included, is `cutting`; cleared is `none`. `reason` names the current step or the refusal in a recorded sentence. `last` carries the outcome of the last move (`done`, `aborted`, `returned`), its reason, `from`, `to` and `at`. Tito's change: read `last.outcome` and `last.reason` when `state` returns to `none` from `cutting` (today it reads only `state`; Goal 5). |
| -The states Tito reads are the legacy words: `none`, `preparing`, `ready`, `cutting`, with `last_move.outcome` `done`, `aborted` or `failed`. The record carries the offline states underneath (`offline-staging` through `offline-finishing`, plus `offline-reverting` and the return's own move), and the view maps them. `cutting` covers start through finish; `reason` says which step is running or what refused. |
| +**Refused before anything is written:** a replica or a home with replicas; a seeded record; a run in flight; a move in flight (the existing 409 sentences); the same region; a source or target region with more than one box; a region whose boxes lack `offline-move-1` or `offline-stage-1`; a target region whose boxes' `secrets_key_fp` differs from the console's (R9); files over the archive limit (§6.3); a `start_at` other than now ("scheduled starts are not available yet"); a host with no record (404); a token without `moves=1` (404). |
| + |
| +**Duplicate requests.** A second prepare for the same host while a self-serve move is in flight answers 409 with the existing "a move to X is already ..." sentence; cut and cancel are idempotent against their own state (a second cut during `cutting` answers 202 with the view). |
| ### 6.2 Catch-up and readiness |
| -`preparing` repeats the stage round until a round copies less than 16 MiB or fewer than 8 objects (the legacy move's ready rule, which the seal's final round then finishes offline in seconds), with at least two rounds and at most 30 minutes; a busy instance that never settles goes `ready` at the deadline with the estimate reflecting the last round's delta. Then the pre-checks (§7.3). `ready` is held, re-staging every 10 minutes so the delta stays small until Tito cuts; a `ready` not cut within 24 hours aborts (nothing was offline) with the reason "the move was not confirmed in time". |
| +`preparing` repeats the stage round until a round whose report matches the request (the stager's round id) shows copied plus deferred bytes under 16 MiB (the legacy move's ready rule, `movespass.go` ready comparison), with at least two rounds and a bound of 30 minutes or 10 rounds; a busy instance that never settles goes `ready` at the bound with the estimate reflecting the last round's remaining bytes and objects. Then the pre-checks (§7.3). `ready` is held, re-staging every 10 minutes so the delta stays small until Tito cuts, and `cut` always restages once more immediately before `start` (the runbook's rule). A `ready` not cut within 24 hours aborts (nothing was offline) with the reason "the move was not confirmed in time", and frees its slots. |
| + |
| +### 6.3 The estimate and the files limit |
| -### 6.3 The estimate |
| +`estimate_seconds` is the sum of per-phase terms, each `fixed + bytes / rate`, rounded up to whole minutes, minimum 2: stop and tombstone; the final copy of `remaining_bytes` (the last stage report's copied plus deferred); landing and verification of `files_bytes` and the replica; open; first-wake promotion of `replica_bytes` (the runbook measured 42 to 73 s of first wake, 52 s of it promotion at 3.7 GB, so this term is not a constant). The fixed parts and rates start at the flagship's measured values (2026-10-21: 1.6 GB staged in 76 s, 3.7 GB in 142 s; flip 45 to 60 s; open 30 to 50 s) and the console refines them per region pair from each finished move's recorded phase timings (a rolling median of the last 20, in the console's database). Before a move is requested (`?estimate=1®ion=`) the replica size and last-day churn come from the source box's status report, which gains a per-host block `{replica_bytes, replica_objects, files_bytes, extra_dirs}` for its hosts (the console cannot list the box's bucket for every host and should not); once staged, from the stage receipts. The view and the holding page use the same number; the page counts down from `offline_at`. |
| -`estimate_seconds = base + replica_bytes / rate_replica + files_bytes / rate_files + delta_bytes / rate_delta`, rounded up to whole minutes, minimum 2. `base` is the fixed part the flagship measured: stop and tombstone about 20 s, flip about 50 s, open about 40 s, first wake about 20 s. The rates start at the flagship's measured values (the 2026-10-21 moves: 1.6 GB staged in 76 s, 3.7 GB in 142 s; seal of 1.6 GB without stage 6 min) and the console refines them from each finished move's recorded timings (a rolling median per region pair, kept in the console's database). Before a move is requested (`?estimate=1`) the replica size comes from the data bucket listing and the files size from the box's status report (`files_bytes`, a new field the box publishes per host), with `delta_bytes` assumed to be the replica's last-day growth; once staged, from the stage receipts. The view and the holding page use the same number; the holding page counts down from `offline_at`. |
| +**The files limit.** The sealed archive carries at most 256 MiB of files (M5, `internal/moves/archive/limits.go`). This design does not lift it: `?estimate=1` and `preparing` refuse an instance whose `files_bytes` exceeds it with the sentence "this account's files are too large to move this way", before anything stops. Raising the limit is an archive change with its own review (§10 Q5). The database has no size cap; its cost is time, which the estimate shows and the deadlines (§7.2) are derived from. |
| ### 6.4 The holding page |
| -From `start` to the first 2xx on the target, the host is held on every source box (not served) and has no ready row on the target. Today every edge answers a platform-domain host with the "not here yet" 503 retry page (`internal/edge/pending.go`) and a customer domain with 404. This design adds an offline answer: |
| +From `start` until the target's row is ready, the host is held on every source box and has no ready row anywhere. Today every edge answers a platform-domain host with the "not here yet" 503 retry page (`internal/edge/pending.go`) and a customer domain with 404. This design adds a maintenance answer, carried on the existing box to edges-file to edge channel: |
| -- The console publishes, in the record's move, `display_name` (as given, at most 80 characters, HTML-escaped at render) and `offline_at` and `estimate_seconds`. Every box's edges file lists a held host as `offline_move: {display_name, offline_at, estimate_seconds}` (the edges file already carries the box's routes; a held row is listed in this new field instead of dropped), and every edge's home map reads it. |
| -- An edge that finds a host in an `offline_move` entry, on any domain, answers 503 with `Retry-After: 30` and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from the estimate and `offline_at`, a self-retrying page in the pending page's style, `noindex`, nothing from the request echoed. Without a `display_name` the sentence is "This account is temporarily offline. ...". The page's text is copy rows. |
| -- After open the target's row is ready, its edges file lists the host, and the holding entry goes; edges pick it up within their edges-file refresh (seconds to half a minute), the pending page's known window. The 503 at the source after open (runbook §12 point 6) is this same window and is accepted. |
| +- **Publication.** Every box that holds the host (every source-region box; and the target box from `flip` until its row is ready) lists it in a new edges-file field, `offline_moves: [{host, aliases, move_id, offline_at, until, display_name}]`, separate from `routes` so no decoder counts it as served (the seal door's "still listed" check reads `routes`; the builder audits every edges-file reader). `aliases` are the host's alias rows, so every custom domain gets the page. `until` is `offline_at + 3 × estimate_seconds`, a self-expiry. A box stops publishing the entry when the record no longer carries that move id or when its own row for the host is ready. |
| +- **Rendering.** An edge answers a request for a host or alias that appears in a fresh `offline_moves` entry, from any publisher, and for which no fresh edges file lists a ready route, with 503, `Retry-After: 30`, `noindex`, and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from `until` and the estimate, counting down; without a display name, "This account is temporarily offline. ...". A ready route anywhere wins over a holding entry (precedence), so the window between `open` and the first edges-file refresh is the pending page's known 503-then-retry window and nothing worse. The source box's own edge answers its held rows the same way from its registry (the home map excludes a box's own publication). The page reuses the pending page's safe reload mechanics (location.replace, sessionStorage) with its give-up replaced by `until`. |
| +- **TLS.** A host in a holding entry is allowed for TLS as a served host is, so the page is reachable on HTTPS with the host's mirrored certificate. |
| +- **The display name.** Tito supplies it per request; the console validates it (at most 80 characters, printable, no control characters, no newlines) and stores it in the move; the edge renders it through contextual template escaping. It is the one piece of request-supplied text the page shows, by design (the customer named their own account), and it is shown to unauthenticated visitors, as the account's own pages are. Nothing else from the request, no header, path, diagnostic or `requested_by`, reaches the page. |
| +- **The health probe** (§7.4) goes to the target box directly, by its address from its edges file with the host as SNI, so it reaches the app whatever the edges show. |
| ## 7. The job |
| -A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape used by the other console passes), every 5 seconds, over every record whose move is an offline move with `self_serve: true`. Each tick advances each move by at most one transition, judged from the evidence the operator doors read today (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the same door bodies refactored into functions the operator doors and the job both call. The job writes nothing but the record and the move log. It is idempotent: a console restart resumes from the record's state. |
| +A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape of the other console passes), every 5 seconds, over the console's `self_serve_moves` table: one row per move, keyed by move id, with `host`, `phase`, `phase_started_at`, `deadline`, `slots`, `display_name`, `estimate`, `health` (the observations so far), `return_of` and `outcome`. The record carries the offline states as today plus `move.self_serve: true`, `display_name`, `offline_at` and `estimate_seconds` for the edges. Each tick advances each move by at most one step, judged from the evidence the operator doors read today (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the door bodies refactored into functions the doors and the job share. Two console processes never run the pass at once (the existing single-console assumption of the other passes). The job is idempotent and restart-safe: every phase's start, deadline and observations are in its row, and a tick that finds a phase's evidence already satisfied moves on. |
| + |
| +### 7.1 Slots and concurrency |
| + |
| +Today one `control/offline-moves/active.json` holds a deployment-wide lock: a staged or moving instance blocks every other one. What it protects, beyond the record's one move: the box's moves pass holds one lock for the whole pass, so a long stage round (142 s measured at 3.7 GB) on a box delays that box's stop-report refresh for another move, past the doors' 2-minute freshness allowance. |
| + |
| +This design replaces the deployment lock with **per-box slots** in the same file: `{box: move_id}` for every participating box of every move past `preparing`'s queue. A move's participants are every box of its source region and the target box (one each in this version). The console takes the slots in the same record write that begins staging and releases them when the move clears, after `finish` or an abort, and only after any return has cleared too. A move whose participant already holds a slot waits in `preparing` before its first stage round, with `reason` "waiting for another move on this box", and takes the slots in order of request. Operator moves take slots the same way (their stage door). Moves of instances on disjoint boxes run at once. Within a box, the one move in flight is the only one staging, stopping, sealing or landing there, so the stop-report refresh is never behind another move's copy. A slot held by a move that no longer exists (a cleared record, a console restart mid-write) is released by the pass when it finds no row for the id; the file is written with the record's CAS discipline (read, compare, write). |
| -### 7.1 Locks and concurrency |
| +### 7.2 Deadlines and the transition table |
| -- One move per instance: the record's `move` field, as today. |
| -- The deployment-wide offline-move lock becomes a per-source-box and per-target-box limit: at most `CARLOS_SELF_SERVE_MOVES_PER_BOX` (default 2) moves whose source or target is a given box may be past `preparing` at once, and at most one may be between `start` and `open` on a given box (the stop and seal are per instance, but the box's disk and network are shared). A move that cannot take its slot waits in `ready` with the reason "waiting for another move on this box", not refused. The builder checks what today's deployment lock protects in `movesoffline_stage.go` and the stage door, and keeps whatever invariant it guards (left to the builder, §11). |
| -- An operator-run offline move counts against the same limits. |
| +Every wait has a deadline derived from the estimate's terms with a floor, and a stall rule: a phase whose evidence has not advanced (bytes, reports, items) for 5 minutes is treated as expired. Expiry before `open` aborts; after `open` it decides by §7.5. |
| -### 7.2 Deadlines |
| +| phase (view) | waits for | deadline | on expiry | |
| +|---|---|---|---| |
| +| slots (`preparing`) | the participants' slots | none; `reason` says waiting | 24 h, then abort "not started in time" | |
| +| lanes (`preparing`) | both lane probes (§7.3) | 5 min | abort, reason names the lane | |
| +| staging (`preparing`) | a matching round under the ready rule | 30 min or 10 rounds (§6.2) | go ready with the remaining bytes in the estimate | |
| +| pre-checks (`preparing`) | target-ready and source reports | 5 min | abort, reason names the check | |
| +| ready | the cut | 24 h | abort "not confirmed in time" | |
| +| restage (`cutting`) | one more round | max(5 min, 2 × its estimate) | abort | |
| +| stopping (`cutting`) | every source box's report: held, fenced, quiesced, no process | max(5 min, 2 × stop estimate) | abort | |
| +| sealing (`cutting`) | the sealer's seal report | max(10 min, 3 × final-copy estimate) | abort | |
| +| landing (`cutting`) | the landing report with every item OK | max(10 min, 3 × landing estimate) | abort | |
| +| opening (`cutting`) | the target's opened report and ready row | max(3 min, 2 × open estimate) | §7.5 by admission evidence | |
| +| health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | §7.5 by admission evidence | |
| +| finishing (`cutting`) | every box's finish report | 10 min | the move stays `cutting` with the reason; page after 10 min; retry each tick (nothing is lost: the data is live on the target) | |
| +| reverting (`cutting`) | abort's reports | 10 min | page; retry each tick; the source's copy was never removed | |
| +| returning (`cutting`) | the return move's own phases | its own table | the return's own rules; no second return (§7.5) | |
| -Every wait has one, and every deadline before open ends in abort: stop reports from every source box (5 min), seal (10 min), landing (10 min), open (3 min), the health check (§7.4, 3 min). `preparing` and `ready` have theirs in §6.2. An abort that itself stalls (a source box not confirming the return) is reported as `reverting` with the reason and pages operators after 10 minutes; nothing is lost, the source's copy was never removed. |
| +`abort` before `open` is the existing abort (`offline-reverting`), lossless: the source's database was never removed and the tombstone lifts. A lost response to `open` is an uncertain commit: the next tick rereads the record and the target's evidence before deciding, never assumes. |
| -### 7.3 Pre-checks before anything goes offline |
| +### 7.3 Lanes and pre-checks before anything goes offline |
| -Run at the end of `preparing` and again at `cut` before `start`. Each is a refusal (abort with the reason, nothing offline) unless marked carried: |
| +**Lanes.** The forward move stages from the source box into the target region's data bucket under `moves/<host>/` (source role writes the target bucket's `moves/*`); the return (§7.5) stages from the target box into the source region's bucket (target role writes the source bucket's `moves/*`). Both must exist before a move starts, because a move that cannot return must not start. Each box proves its lane by a probe on its moves pass once the record is staging: it writes and deletes `moves/<host>/<id>/.lane` in the other region's bucket and reports the result. The console requires both reports before the first stage round. M17's push-only shape stands: no role gains access to a sibling's `instances/*`. |
| -- Target region: every box publishes `offline-move-1` and `offline-stage-1`; no claim or Phase 1 switch; `secrets_key_fp` equals the console's (R9); the target region's staging grant on the source data bucket exists (the stage round having succeeded proves it). |
| -- Target box (the one that will land it, chosen as today): a **target-ready report** written by the box under `control/offline-moves/<host>/<id>/target/<box>.json` on its moves pass once the record is staging, saying: no route or timers for the host; `/opt/carlos/live/<host>` absent, or present and naming the build the record's channel resolves to (#339's stale link); none of `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/programs/<host>.json`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db` present; `/data/.offline-moves/<host>` absent or holding only an earlier move's `target/`; free disk at least twice the replica's size; 0 current objects under `instances/<host>/` and `moves/<host>/` in the target data bucket and no `lock.json`. The box reports facts; the console decides. |
| -- Source: the instance's extra app directories beside its files directory (#339 point 4): carried when empty (the archive gains empty directory entries), refused when not, with the reason naming the directory. The source box's existing report gains the list. |
| -- Build: the channel's build is the one the source runs (`release.LiveVersion` equals the row's); a promote during the move fails the landing's `release` item as today, and the job aborts. |
| +**Clean, then staged.** Before the first stage round the target bucket's `moves/<host>/` and `instances/<host>/` must hold no current objects and no `lock.json`, except the leftovers of a return onto an originating box (§7.5) which are named and accepted. After staging, `moves/<host>/` holds exactly this move's staged objects (the stager's receipts, bound to the move id, bucket and region); foreign objects refuse. |
| + |
| +**Target-ready report.** The target box, on its moves pass once the record is staging and again when asked at cut, writes `control/offline-moves/<host>/<id>/target/<box>.json` with facts, and the console decides. Facts: no route or timers for the host; `/opt/carlos/live/<host>` absent, or present and naming the build the record's channel resolves to; `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db` absent; `/data/programs/<host>.json` absent, or present and naming that same build; `/data/.offline-moves/<host>` absent or holding only earlier moves' directories; free disk at least twice the replica's size; the target bucket prefixes as above; capabilities and switches as required; `secrets_key_fp`. A stale live link naming another build is a refusal with that reason (#339's cause, caught before the stop). |
| + |
| +**Source report.** The source box's existing report gains: the instance's extra app directories beside its files directory (#339 point 4) with their sizes; `files_bytes`; the build the row runs. Empty extra directories are carried (the archive gains directory entries, restored at the same relative path under the instance directory); a non-empty one refuses with the reason naming it. |
| + |
| +**Build.** The channel's build must be the one the source runs; a promote during the move fails the landing's `release` item as today and the job aborts. |
| ### 7.4 The health check after open |
| -After `open`, the job waits for the first 2xx from the target for the host's probe path (the record's `probe`, else `/`), through the target box, within 3 minutes, and then one more 2xx 30 s later. Both pass: `finish`. Either fails: the return (§7.5). The console measures the offline time from `start` to the first 2xx and records it with the staged sizes for the estimate's rates (§6.3). |
| +"Healthy" is: the app answers on the target. The probe is the manifest's ready path (`moves.ready_path`, `Manifest.ReadyPathOrDefault`, the path the legacy move used; Tito sets a path that answers 2xx without a session, as the runbook did with `/signin`), requested from the console to the target box directly (its address from its edges file, the host as SNI and `Host`), expecting a 2xx within 30 s per attempt; a 3xx is not a pass (the ready path's contract is a 2xx). Two passes 30 s apart, inside the deadline of §7.2, and the landing's manifest checks remain mandatory before it (the health check supplements them, it does not replace them). The console records the offline time from `start` to the first pass with the staged sizes, for the estimate's rates. Application checks beyond the ready path (the store reachable, custom domains, mail) are the tenant's: the runbook's broader verification is operator work this version does not automate, and Tito's ready path can cover what it cares about. |
| ### 7.5 The return |
| -The return is an offline move in the other direction, started by the job without a customer step, with the same holding page (the estimate for the return comes from the same rates). It needs: |
| +The question that decides the return is **whether the target ever admitted a write**, read from the target bucket, not from the box: `instances/<host>/lock.json` (absent, or held or released by the target box) and the newest LTX TXID under `instances/<host>/` against the seal's `FinalTXID`. Three cases. |
| + |
| +**A. No admission** (no lock ever taken by the target, or taken and released with no LTX beyond `FinalTXID`): the opened report never came, or the app did not start, or it started and answered nothing before the deadline. The return is **revert**, a new transition `OfflineRevert` from `offline-open` to `offline-reverting` that abort's machinery already drives from landed, extended by one step: the source boxes **un-park** what `open` parked (the park is a rename into `/data/.offline-moves/<host>/<id>/source/`; the reverse rename is journaled the same way), the target retires its row and parks what it extracted (as abort from landed does) and lifts any lease it took, the tombstone lifts, the record's region returns to the source, and `finish` records `last.outcome: returned`. Lossless by construction: the source's database and files were never removed, its bucket replica is untouched under the tombstone, and the target wrote nothing. |
| -- the source region's staging grant on the target data bucket (the reverse of the lane the forward move used). The deployment's regions are configured in pairs with grants both ways (infra; §9), and `?estimate=1` and the door refuse a target whose reverse grant is absent, so a move never starts that cannot return; |
| -- the original box to pass the target-ready report (§7.3) for the return. The forward move's `finish` therefore removes the source's leftovers that block it (§7.6), and the return's own `finish` does the same on the box it left. |
| +**B. Admitted** (LTX beyond `FinalTXID`, or the lock held by a target writer now): the account served and may hold new data, so nothing on the source is current. The return is a **second offline move from the target to the source**, started by the job: forward `finish` completes (the move clears, outcome `done`, but the view keeps `cutting` with `reason` "returning" from the job's row), then the job begins the return with `return_of: <id>`, through every phase of the table: stage from the target (now an ordinary serving source: it has a live replica, a ready row and the build root staged for it), stop, seal, flip onto the originating box, open, health, finish. The originating box's target-ready report accepts exactly the forward move's leftovers and nothing else: the park directory `<fwd id>/source/` and `<fwd id>/sealer/`; the lifted `lock.json` (owner `moved:<fwd id>`, expired or lifted); the root-owned live link and `/data/programs/<host>.json` naming the build the return lands; and the forward's old replica objects under `instances/<host>/` in the originating bucket, all older than the forward's `open_at`, which the originating box deletes by exact key as the first step of its landing (its own bucket, its own role), keeping `lock.json`. The return's health check fails: **no third move**; the account stays on the originating box, `reason` "recovery needs an operator", paged. The return's outcome is `returned`, with the forward health failure as the reason. |
| -The return's `last_move` is `returned` with the health failure as the reason; Tito's status shows `failed` with that reason and the account is where it started. If the return cannot start (no reverse grant, which the door should have refused; the original box not reporting ready within 5 minutes; a second move already holding its slot), the move stays `done` in the new region with `last_move.warning` and operators are paged (the existing alert sink). Fixing forward is then an operator's job, as the runbook says today. |
| +**C. Unknown** (the target bucket's evidence cannot be read, or the lock is held by the target and no LTX listing is available): no automatic action. The move stays `cutting` with `reason` "recovery needs an operator"; operators are paged at once and every 10 minutes; nothing is deleted anywhere; the tombstone stays (so the source cannot wake stale). The operator's runbook covers it: read the target's lease and position by hand, then revert or finish. |
| -### 7.6 Cleanup: the source's leftovers |
| +The holding page continues through a revert and a return (the record's move id changes for a return; the entries follow the record). Cancel is refused throughout. |
| -Today a finished move leaves on the source box a root-owned `/opt/carlos/live/<host>` link and `/data/programs/<host>.json`, the parked database and directories under `/data/.offline-moves/<host>/`, and the source region's replica in its data bucket until an operator deletes it (runbook §10, §12). For self-serve: |
| +### 7.6 Cleanup: who deletes what, from what evidence |
| -- Root's existing hourly pass (`carlos adopt`, the root pass) removes `/opt/carlos/live/<host>` and `/data/programs/<host>.json` for a host whose record names another region and whose `last_move` is `done` or `returned` older than 10 minutes and whose row is absent from the box's registry. This is the one new root action; it takes a census row at its sibling's class (the version sweep's) and never deletes a path that any live link or running process still names. |
| -- The agent removes the parked `/data/.offline-moves/<host>/<id>/` 7 days after `finish`. |
| -- The console's job deletes the source region's replica objects by exact key, keeping `lock.json` (the lifted tombstone), 24 hours after a `done` whose target has served since (the box's status report shows the host awake at least once). The deletion is logged key by key in the move log, as the runbook's cleanup was. |
| +- **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept until now so the return has it): deleted by the **source box** on its moves pass, by exact key keeping `lock.json`, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{move_id, host, to, open_at, final_txid, tombstone_owner}`, and only when all hold at the time of deletion: the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; every object to delete has `LastModified` before `open_at`; and the box's own registry has no row for the host. A listing-to-deletion race is closed by deleting with the listed `VersionId` where the bucket is versioned (both are), and by refusing an object whose current version differs from the listed one. Each deletion is logged key by key in the move log. A return (§7.5 B) deletes the forward's old replica as its landing's first step, under the same conditions, and its own `retired.json` on the box it left governs that bucket's cleanup in turn. |
| +- **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after `finish`, never while a `retired.json` under it still has work. |
| +- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root's hourly pass removes them for a host whose record names another region, whose `retired.json` is at least 10 minutes old, and whose row is absent from the registry, never a path a live link or a running process still names. Off every critical path: a return accepts them (§7.5 B). One new census row at the version sweep's class. |
| +- **The staged objects of an aborted or reverted move** (`moves/<host>/<id>/` in the target bucket): removed by the stager on abort as today. |
| ### 7.7 Scheduling |
| -Tito books the time and calls prepare when it arrives (its sweep's `scheduled` step does this today). The platform's door takes an optional `start_at` and refuses one that is not "now" in this version ("scheduled starts are not available yet"), so the field exists for the version that honours it. |
| +Tito books the time and calls prepare when it arrives (its sweep's `scheduled` step). The platform's door takes an optional `start_at` and refuses one that is not now ("scheduled starts are not available yet"), so the field exists for the version that honours it. |
| ## 8. Observability |
| -- The move log (the existing per-move log under `moves/<acct>/<app>/<host>/<id>/`) gains the job's lines: each transition, each refusal and its reason, the pre-check facts, the health probes, the timings, the cleanup keys. |
| -- The console's status view carries `reason` and `offline_at`; `carlos move offline status` shows them. |
| -- Paging through the existing alert sink: a return that could not start, an abort that stalls past 10 minutes, a job that finds a move it cannot advance for 30 minutes. |
| -- Member stream: each transition is an event on the app's platform log, as the legacy move's were. |
| +- The move log gains the job's lines: each phase, each refusal and its reason, the pre-check facts, the lane probes, the health probes, the timings, the admission evidence read for a return, every cleanup key. |
| +- The view carries `reason`, `offline_at`, `estimate_seconds` and `last`; `carlos move offline status` shows them. |
| +- Paging through the existing alert sink: case C, a finishing or reverting phase past 10 minutes, a return's failed health, a slot held by an unknown move id, a lane probe failing on a configured pair. |
| +- Member stream: each phase is an event on the app's platform log, as the legacy move's were. |
| ## 9. Rollout |
| Platform: |
| 1. Build behind `CARLOS_SELF_SERVE_MOVES`, default off; copy rows pending until Paul's review; census overlay PROPOSED. |
| 2. Release and roll dev, then flagship, then Tito (roll order rule; boxes before consoles). |
| -3. Rehearse on dev with a Tito-shaped app (key file, store binding, config, custom domain) through Tito's own residency flow against the dev console: forward, cancel in ready, abort by a failed pre-check, a forced health failure and its return. |
| +3. Rehearse on dev with a Tito-shaped app (key file, store binding, config, custom domain) through Tito's own residency flow against the dev console: forward; cancel in ready; abort by a failed pre-check; a forced case A (the app refuses to start on the target) and its revert; a forced case B (the app starts, writes, then fails its ready path) and its return; the holding page on a third edge and on a custom domain; two moves on disjoint boxes at once; two queued on one box. |
| 4. Turn the switch on for Tito's console with Paul's go. |
| -Tito's side (titogo, from titogo-f6's notes of 2026-10-21): |
| -1. An account owner mints a move-capable token; the router switches `CARLOS_...` to it (one env var, env sync, a router restart). |
| -2. The residency tab shows the estimate from `?estimate=1` on the confirm page and refuses controld accounts (no platform record) before posting. |
| -3. Re-check the 2026-10-02 unsafe list on the region boxes: the edge door or `routes.home` for remote accounts, the secrets key on every region box (distributed 2026-10-03), the payments relay socket per region. |
| -4. `TITO_REMOTE_PLACEMENT=on`, env sync, restart the router. Publish the "Move your account" article. |
| +Tito's side (titogo; from titogo-f6's notes of 2026-10-21): |
| +1. An account owner mints a move-capable token; the router's console credential switches to it (one env var, env sync, a router restart). |
| +2. The residency tab shows the estimate from `?estimate=1®ion=` on the confirm page, refuses controld accounts before posting, sends `display_name` and `requested_by`, and reads `last.outcome` and `last.reason` when `state` returns to `none`. |
| +3. The manifest's `moves.ready_path` names a path that answers 2xx without a session. |
| +4. Re-check the 2026-10-02 unsafe list on the region boxes: the edge door or `routes.home` for remote accounts, the secrets key on every region box (distributed 2026-10-03), the payments relay socket per region. |
| +5. `TITO_REMOTE_PLACEMENT=on`, env sync, restart the router. Publish the "Move your account" article. |
| -Infra: the staging grants both ways for every region pair the console arms (today IE to UK exists; UK to IE does not), as a region-host module output. |
| +Infra: both staging lanes for every region pair the console arms (today IE to UK exists; UK to IE does not), as a region-host module output, with the one-box-per-region shape stated. |
| ## 10. Questions for Paul (none block the plan; defaults stated) |
| -- **Q1.** Does the move-capable token (§5) answer M18 for this door? Default: yes, as stated. |
| -- **Q2.** Cancel during `cutting` is refused (§4.8). Default: refused; the customer confirmed. |
| -- **Q3.** The 24-hour `ready` abort and the 30-minute staging bound (§6.2). Defaults as stated. |
| -- **Q4.** The source replica deletion 24 hours after done (§7.6) is automatic. Default: automatic, logged key by key. |
| +- **Q1.** The move-capable token (§5) as the answer to M18 for this door, app-wide and owner-minted. Default: yes. |
| +- **Q2.** Cancel during `cutting` is refused. Default: refused; the customer confirmed. |
| +- **Q3.** Case B's return moves the new data back as a second offline move with its own few minutes offline, and a failed return stops there and pages. Default: as stated. |
| +- **Q4.** The source replica deletion 24 hours after done, by the source box, with the runbook's checks. Default: automatic. |
| +- **Q5.** The archive's 256 MiB files limit (M5) stays and is a refusal before confirming. Raising it is separate work; say if you want it in this version. |
| +- **Q6.** The display name is Tito-supplied per request, validated, shown to unauthenticated visitors. Default: yes. |
| ## 11. Left to the builder |
| -- What the deployment-wide offline-move lock protects today (`movesoffline_stage.go`, the stage door) and the smallest change that gives per-box limits without weakening it. |
| -- The exact split of the operator doors into functions the job calls (the doors keep their behaviour byte for byte). |
| -- The edges-file field for held hosts and its reader in the home map, with the pending page's retry mechanics reused for the holding page. |
| -- The target-ready report's writer in the box's moves pass and its census row. |
| -- The estimate's storage and the rolling median. |
| -- The token column (`moves`) migration and the mint flag. |
| -- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward, cancel in ready, each pre-check refusal, each deadline, a health failure and its return, the holding page's answer on a third edge, two moves at once on one box. |
| +- The split of the operator doors into functions the job calls (the doors keep their behaviour byte for byte). |
| +- The `self_serve_moves` table and the slot file's CAS discipline; the view mapping's tests against Tito's actual polling sequence (prepare, poll to ready, cut, poll to none, read `last`). |
| +- The edges-file `offline_moves` field, its readers in the home map and the local edge, the TLS allow, and the audit of every edges-file decoder. |
| +- The target-ready, lane and source report shapes and their census rows; the per-host block in the status report. |
| +- The archive's directory entries for empty extra directories and their restore location. |
| +- The estimate's storage and the rolling median; the phase timings' recording. |
| +- The `moves` column migration, the mint flag and the instance-bound refusal. |
| +- The revert transition and the un-park step's journal; the return's leftover acceptance in the target-ready contract. |
| +- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; each pre-check refusal; each deadline; case A revert; case B return and its failed-health stop; case C paging; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry; two moves at once on disjoint boxes and two queued on one; cleanup refusing when the record moved again or an object is newer than `open_at`. |
| ## 12. Risks |
| -- **The return has never been run.** The runbook says a move back "has never been run"; the return is the same machinery in the other direction and gets the same witness, but its first real run is on dev. |
| -- **The estimate is an estimate.** Rates differ by region pair and object count; the rolling median narrows it; the holding page says "approximately". |
| +- **The return (case B) has never been run.** It is the forward machinery in the other direction onto a box with named leftovers; it gets the same witness and its first real run is the dev rehearsal. |
| +- **Admission evidence.** Case A's losslessness rests on the target bucket's lease and LTX listing being truthful about writes; the sealer's `FinalTXID` is the reference. A target whose litestream never pushed an LTX after a write it accepted is case C by construction (the lock is held, no LTX): paged, not reverted. |
| +- **The estimate is an estimate.** Rates differ by region pair and object count; the rolling median narrows them; the page says "approximately". |
| - **Boxes that do not report.** A source box down at `start` stalls `seal`; the deadline aborts and the customer sees "failed, still in <region>" with the box named. Nothing is lost. |
| -- **Tito's freeze.** Tito freezes its own writes before cut; a Tito bug there is Tito's, and the platform's seal is final regardless (the tombstone, not Tito's freeze, is the stop). |
| +- **Tito's freeze.** Tito freezes its own writes before cut; a Tito bug there is Tito's; the platform's seal is final regardless (the tombstone, not Tito's freeze, is the stop). |