amadan
design: self-serve offline moves, rev 5 (Astra round 4)
A per-host move_seq on the record, persisted as serving_seq on rows and published with routes and holding entries, orders consecutive moves and makes an abort's reopening its own completion evidence at the edges; the root cleanup request ends only with root's done or an acknowledged cancel, never a timeout; offline-queued means no copying with the pinned participants' lane and cleanliness reports allowed; both finish gates in the table; stale sentences aligned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+17
−16
docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md+17 −16
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.013s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (142 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.710s
ok github.com/carlosframework/platform/cmd/moveprobe 11.919s
ok github.com/carlosframework/platform/cmd/zddprobe 0.778s
ok github.com/carlosframework/platform/internal/activator 60.309s
ok github.com/carlosframework/platform/internal/agentsock 0.025s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.021s
ok github.com/carlosframework/platform/internal/appconfig 2.999s
ok github.com/carlosframework/platform/internal/apperr 0.157s
ok github.com/carlosframework/platform/internal/applog 0.280s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 1.047s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.011s
ok github.com/carlosframework/platform/internal/caps 0.014s
ok github.com/carlosframework/platform/internal/certwarm 0.004s
ok github.com/carlosframework/platform/internal/cgroup 0.841s
ok github.com/carlosframework/platform/internal/changelog 0.557s
ok github.com/carlosframework/platform/internal/claimactivation 1.467s
ok github.com/carlosframework/platform/internal/clamd 0.059s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.241s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.011s
ok github.com/carlosframework/platform/internal/dbverify 0.005s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.009s
ok github.com/carlosframework/platform/internal/domains 3.153s
ok github.com/carlosframework/platform/internal/doorkey 0.008s
ok github.com/carlosframework/platform/internal/economics 0.007s
ok github.com/carlosframework/platform/internal/edge 45.305s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.333s
ok github.com/carlosframework/platform/internal/gate 1.823s
ok github.com/carlosframework/platform/internal/genselect 0.021s
ok github.com/carlosframework/platform/internal/geo 0.027s
ok github.com/carlosframework/platform/internal/geoip 0.033s
ok github.com/carlosframework/platform/internal/historyreadback 0.301s
ok github.com/carlosframework/platform/internal/host 139.581s
ok github.com/carlosframework/platform/internal/hostactors 5.969s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.008s
ok github.com/carlosframework/platform/internal/hostbox 3.412s
ok github.com/carlosframework/platform/internal/hostclaim 0.865s
ok github.com/carlosframework/platform/internal/hostenv 0.013s
ok github.com/carlosframework/platform/internal/hostevidence 0.008s
ok github.com/carlosframework/platform/internal/hostexec 5.667s
ok github.com/carlosframework/platform/internal/hostfence 0.064s
ok github.com/carlosframework/platform/internal/hostfiles 10.357s
ok github.com/carlosframework/platform/internal/hostfloor 0.019s
ok github.com/carlosframework/platform/internal/hostguard 1.060s
ok github.com/carlosframework/platform/internal/hostops 2.257s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.004s
ok github.com/carlosframework/platform/internal/hostquarantine 0.007s
ok github.com/carlosframework/platform/internal/hostrelocate 0.602s
ok github.com/carlosframework/platform/internal/hostresolve 0.221s
ok github.com/carlosframework/platform/internal/hostwrite 0.010s
ok github.com/carlosframework/platform/internal/infraview 0.007s
ok github.com/carlosframework/platform/internal/instancectl 5.254s
ok github.com/carlosframework/platform/internal/instanceexec 2.782s
ok github.com/carlosframework/platform/internal/instancefs 0.006s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.011s
ok github.com/carlosframework/platform/internal/layout 1.922s
ok github.com/carlosframework/platform/internal/lbhealth 0.010s
ok github.com/carlosframework/platform/internal/lease 10.182s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.566s
ok github.com/carlosframework/platform/internal/ledger 0.022s
ok github.com/carlosframework/platform/internal/mail 0.242s
ok github.com/carlosframework/platform/internal/maintenance 0.085s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.601s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.118s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 21.412s
ok github.com/carlosframework/platform/internal/moves/archive 1.109s
ok github.com/carlosframework/platform/internal/objstore 0.270s
ok github.com/carlosframework/platform/internal/ops 0.188s
ok github.com/carlosframework/platform/internal/park 0.239s
ok github.com/carlosframework/platform/internal/plans 0.009s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 13.598s
ok github.com/carlosframework/platform/internal/publicrelease 0.009s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.670s
ok github.com/carlosframework/platform/internal/relay 7.207s
ok github.com/carlosframework/platform/internal/release 24.587s
ok github.com/carlosframework/platform/internal/rollout 3.853s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.013s
ok github.com/carlosframework/platform/internal/rootparticipation 0.065s
ok github.com/carlosframework/platform/internal/rootstate 0.030s
ok github.com/carlosframework/platform/internal/runs 0.523s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.005s
ok github.com/carlosframework/platform/internal/schedule 2.715s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.222s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.632s
ok github.com/carlosframework/platform/internal/status 0.069s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.220s
ok github.com/carlosframework/platform/internal/svcrole 0.003s
ok github.com/carlosframework/platform/internal/telemetry 0.009s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.006s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.008s
ok github.com/carlosframework/platform/internal/version 0.004s
ok github.com/carlosframework/platform/internal/vet 1.532s
=== internal_console@0 (77 s)
ok github.com/carlosframework/platform/internal/console 74.628s
=== internal_console@1 (37 s)
ok github.com/carlosframework/platform/internal/console 34.841s
=== internal_console@2 (42 s)
ok github.com/carlosframework/platform/internal/console 39.962s
=== internal_console@3 (37 s)
ok github.com/carlosframework/platform/internal/console 35.453s
=== internal_console@4 (37 s)
ok github.com/carlosframework/platform/internal/console 35.257s
=== internal_console@5 (35 s)
ok github.com/carlosframework/platform/internal/console 33.182s
=== internal_console@6 (37 s)
ok github.com/carlosframework/platform/internal/console 34.634s
=== internal_console@7 (43 s)
ok github.com/carlosframework/platform/internal/console 41.398s
=== root@0 (33 s)
ok github.com/carlosframework/platform 30.679s
=== root@1 (45 s)
ok github.com/carlosframework/platform 42.639s
=== root@2 (41 s)
ok github.com/carlosframework/platform 39.280s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.024s
=== RUN TestT4StorageProbeMinio
2026/10/21 22:38:50 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0AC830D50B1A3, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md| index 37b5597ad..5db176be2 100644 |
| --- a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| @@ -1,6 +1,6 @@ |
| # Self-serve offline moves: a tenant's customer moves their own instance |
| -Rev 4, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. Rev 3 (b77c01a9) was reviewed again (`selfserve-spec-review3.md`, NOT READY, 8 findings, the abort boundary accepted); rev 4 answers them: which failures stay after `open` (§7.5), the job's durable admission order (§7), cleanup's own reservation and the root handoff (§7.6), the holding entries' causal ordering (§6.4), the probe mode (§7.4), deadlines as initiate-and-retry with blocked states (§7.2), and the stale contradictions. |
| +Rev 5, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. Rev 3 (b77c01a9) was reviewed again (`selfserve-spec-review3.md`, NOT READY, 8 findings, the abort boundary accepted); rev 4 answers them: which failures stay after `open` (§7.5), the job's durable admission order (§7), cleanup's own reservation and the root handoff (§7.6), the holding entries' causal ordering (§6.4), the probe mode (§7.4), deadlines as initiate-and-retry with blocked states (§7.2), and the stale contradictions. Rev 4 (20fdcf18) was reviewed again (`selfserve-spec-review4.md`, NOT READY, 2 P1 and 2 P2: holding entries ordered by move identity cannot order consecutive moves or show an abort completed; a timed-out root cleanup request could run after its reservation was released; the queued handshake's producers; stale text). Rev 5 answers them with a per-host `move_seq` published with every route and holding entry (§6.4), a root request that ends only with root's `done` or acknowledged `cancelled` (§7.6), queued defined as no copying with the participants' reports allowed (§7), and the stale passages aligned. |
| Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says exactly what the customer's request may do. |
| @@ -13,7 +13,7 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| **Paul's rulings, 2026-10-21:** |
| - The customer confirms the downtime themselves; nobody at Tito approves. |
| - No size cap: the estimate is shown and the customer decides. (Qualified in §6.3: the archive's 256 MiB files limit, M5, is a separate ruling this design does not lift; §10 Q5.) |
| -- A failure after the account has gone offline puts it back in the old region automatically. (Rev 3 keeps this for every failure before `open`, which is where every data-carrying step and every check happens, and asks Paul to accept that after `open` the return is an operator's run, §10 Q3: a lossless automatic return after the new region has started cannot be proven with the claims-off machinery.) |
| +- A failure after the account has gone offline puts it back in the old region automatically. (Kept for every failure before `open`, where the final copy, the archive, the config render and the landing's verification happen, as an abort the job initiates and retries; Paul is asked to accept that after `open` the return is an operator's run, §10 Q3, because a lossless automatic return after the new region has started cannot be proven with the claims-off machinery, and some failures stay after `open` by the offline move's design, §7.5.) |
| - Scheduling is nice to have, not needed first. |
| - Offline moves stay available to Tito as an operator-run option regardless. |
| @@ -59,7 +59,7 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| ## 4. The customer's path, end to end |
| -1. Tito's residency tab lists the armed regions from our console's `/regions`, as today. The customer picks one. Tito shows the confirm page with the estimate from `GET .../instances/{host}/move?estimate=1®ion=uk` (§6.3): "about N minutes offline", or the refusal if the move cannot be offered (over the files limit, no return lane, a region with two boxes). The customer may pick a time (Tito's field) and confirms. |
| +1. Tito's residency tab lists the armed regions from our console's `/regions`, as today. The customer picks one. Tito shows the confirm page with the estimate from `GET .../instances/{host}/move?estimate=1®ion=uk` (§6.3): "about N minutes offline", or the refusal if the move cannot be offered (over the files limit, a region with two boxes, no forward lane). The customer may pick a time (Tito's field) and confirms. |
| 2. At the time, Tito's router calls `POST .../instances/{host}/move {"region":"uk","display_name":"Paul's account","requested_by":"<tito user id>"}` with its move-capable token (§5). The console answers 202 with the status view, state `preparing`. |
| 3. **Preparing.** The job reserves the move's boxes (§7.1), proves the staging lane (§7.3), stages while the instance serves until caught up (§6.2), runs the pre-checks (§7.3). When all pass the state is `ready` with the refined estimate. Tito polls `GET .../move` as today. |
| 4. Tito freezes its own account writes (its `frozen` step) and calls `POST .../move/cut`. The console answers 202, state `cutting`. |
| @@ -115,13 +115,13 @@ The legacy move doors keep their paths and roles. With `CARLOS_SELF_SERVE_MOVES= |
| From `start` until the target's row is ready, the host is held on every source box and has no ready row anywhere. Today every edge answers a platform-domain host with the "not here yet" 503 retry page (`internal/edge/pending.go`) and a customer domain with 404. This design adds a maintenance answer, carried on the existing box to edges-file to edge channel: |
| -- **Publication.** Every box that holds the host (every source-region box; and the target box from `flip` until its row is ready) lists it in a new edges-file field, `offline_moves: [{host, aliases, move_id, offline_at, estimate_seconds, display_name}]`, separate from `routes` so no decoder counts it as served (the seal door's "still listed" check reads `routes`; the builder audits every edges-file reader). `aliases` are exactly the alias rows the edges writer would publish for the host as a served route, under the same eligibility rules (DNS pointing, ownership, wildcard handling), so every custom domain that would reach the account gets the page and no other. A box stops publishing the entry when the record no longer carries that move id or when its own row for the host is ready. |
| -- **Rendering.** An edge answers a request for a host or alias that appears in an `offline_moves` entry from any publisher, and for which no publication carries a ready route tagged with that move (the ordering rule below), with 503, `Retry-After: 30`, `noindex`, and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from `until` and the estimate, counting down; without a display name, "This account is temporarily offline. ...". A ready route anywhere wins over a holding entry (precedence), so the window between `open` and the first edges-file refresh is the pending page's known 503-then-retry window and nothing worse. The source box's own edge answers its held rows the same way from its registry (the home map excludes a box's own publication). The page reuses the pending page's safe reload mechanics (location.replace, sessionStorage) with its give-up replaced by `until`. |
| +- **Publication.** Every box that holds the host (every source-region box; and the target box from `flip` until its row is ready) lists it in a new edges-file field, `offline_moves: [{host, aliases, move_id, serving_seq, offline_at, estimate_seconds, display_name}]`, and every route in `routes` gains `serving_seq`; separate from `routes` so no decoder counts it as served (the seal door's "still listed" check reads `routes`; the builder audits every edges-file reader). `aliases` are exactly the alias rows the edges writer would publish for the host as a served route, under the same eligibility rules (DNS pointing, ownership, wildcard handling), so every custom domain that would reach the account gets the page and no other. A box stops publishing the entry when the record no longer carries that move id or when its own row for the host is ready. |
| +- **Rendering.** An edge answers a request for a host or alias whose highest-sequence publication is a holding entry (the ordering rule below) with 503, `Retry-After: 30`, `noindex`, and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from `until` and the estimate, counting down; without a display name, "This account is temporarily offline. ...". A ready route anywhere wins over a holding entry (precedence), so the window between `open` and the first edges-file refresh is the pending page's known 503-then-retry window and nothing worse. The source box's own edge answers its held rows the same way from its registry (the home map excludes a box's own publication). The page reuses the pending page's safe reload mechanics (location.replace, sessionStorage) with its give-up replaced by `until`. |
| - **TLS.** A host or alias in a holding entry is allowed for TLS exactly as it would be as a served route (the same `Allowed` decision, wildcard and mirrored certificates included), so the page is reachable on HTTPS; the entry never widens what the host could already present. |
| - **The display name.** Tito supplies it per request; the console validates it (at most 80 characters, printable, no control characters, no newlines) and stores it in the move; the edge renders it through contextual template escaping. It is the one piece of request-supplied text the page shows, by design (the customer named their own account), and it is shown to unauthenticated visitors, as the account's own pages are. Nothing else from the request, no header, path, diagnostic or `requested_by`, reaches the page. |
| -- **Ordering.** Claims-off records may carry no `record_seq`, so the order is causal, not a counter. A holding entry names its move id. A box publishes a ready route for a host **with `landed_move: <id>`** when its row was minted by that move's landing (the target), and plain otherwise. For a host with a holding entry for move `<id>` in any fresh publication, an edge serves the holding page unless some fresh publication carries a ready route for the host tagged `landed_move: <id>`; a plain ready route (the source's own stale publication from before the stop, retained as last-good) does not win over a holding entry. A ready route tagged with a move the holding entries do not name (a later move) wins. The source box's own edge answers its held rows the same way from its registry. This replaces rev 2's "a ready route anywhere wins". |
| +- **Ordering by a per-host sequence.** The record gains `move_seq`, a per-host counter the console initialises to 1 on a host's first offline move and increments on every offline transition that changes who serves the host: `start` (the hold), `open` (the target serves), the abort's reopening of the source, and `finish`. Every box persists on its registry row the `move_seq` of the transition that produced the row's current state (`serving_seq`: the hold's for a held row, `open`'s for a landed ready row, the abort's for a reopened source row; a row never touched by a move carries 0), and publishes it with the route and with any holding entry. An edge orders a host's publications by `serving_seq` and serves the highest: a holding entry (the hold's sequence) beats the source's stale ready publication (lower); the target's ready route after `open` beats the holding entries (higher); a source reopened by an abort publishes the abort's sequence, higher than the hold's, so it beats every holding entry of the aborted move still retained as last-good elsewhere; the next move's hold is higher again. Ties cannot occur between a ready route and a holding entry (one transition produces one state per box); a tie between two ready routes falls to the existing home-map rule. Nothing is inferred from move ids. The abort's reopening is therefore its own completion evidence at the edges, independent of the target's recovery report, which the finish gate does not require. |
| - **Freshness.** The home map's existing behaviour stands: publications are refreshed on the edges' own cadence, unchanged publications are not rewritten, and a last-good publication is kept through a read failure. A holding entry therefore lives exactly as long as its publisher lists it, which is while the record carries the move and the publisher's row is not ready; there is no age rule and no fixed expiry. The displayed estimate is separate from validity: past `offline_at + estimate` the page says "...is taking longer than expected. Please check back in a few minutes." and keeps retrying. |
| -- **Reopening.** The public reopens when the target's row is ready and its tagged publication reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. A failure after that is visible to visitors as the app's own answers, as any instance failure is. |
| +- **Reopening.** The public reopens when the target's row is ready and its publication, at `open`'s sequence, reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. A failure after that is visible to visitors as the app's own answers, as any instance failure is. |
| - **The health probe** (§7.4) goes to the target box directly, by its address from its edges file with the host as SNI, so it reaches the app whatever the edges show. |
| ## 7. The job |
| @@ -131,8 +131,8 @@ A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape of |
| **Admission order** (every boundary has a recovery): |
| 1. The door writes the `self_serve_moves` row first, phase `requested`, with the host, account, app, region pair and the participant boxes pinned by box id (`control/boxes/<box>.json` identity, not label), before anything else. A door that dies here leaves a row the pass picks up next tick. |
| 2. The pass reserves the participants (§7.1). A row in `requested` with no reservation is retried each tick; a reservation naming a row's id is this row's. |
| -3. With the reservation held, the pass writes the record's move in a new state **`offline-queued`**: the move exists (so a second prepare 409s and the view says `preparing`), but no box acts on it; the box dispatcher starts staging only from `offline-staging`, so the console controls when copying begins. A record write that may or may not have landed is reread next tick (the id is in the row). |
| -4. The lane probe and the cleanliness check run against `offline-queued`; when both pass the pass transitions the record to `offline-staging` and the box begins the first round. |
| +3. With the reservation held, the pass writes the record's move in a new state **`offline-queued`**: the move exists (so a second prepare 409s and the view says `preparing`) and **no box copies any replica object**. The box dispatcher recognises `offline-queued` for exactly two things, both by the pinned participants: the source box's lane probe (§7.3) and the target box's initial cleanliness and target-ready report (§7.3); staging starts only from `offline-staging`, so the console controls when copying begins. A record write that may or may not have landed is reread next tick (the id is in the row). |
| +4. When both reports are in and pass, the pass transitions the record to `offline-staging` and the box begins the first round. The order is reservation, queued record, reports, staging; the staged-object verification of §7.3 is a separate check after staging, distinct from the initial emptiness. |
| 5. A participant that is replaced (a new box publishes the region's bucket) does not participate: the reservation names the old box id, and the pass refuses the move at the next pre-check with the reason naming the box; an operator decides. |
| A console crash at any step leaves a row whose phase and the record's state together say where it was, and the pass resumes from there; a row with no record move and a reservation older than the grace period is a failed admission the pass aborts and releases. |
| @@ -167,18 +167,18 @@ Every wait has a deadline derived from the estimate's terms with a floor; a stag |
| | landing (`cutting`) | the landing report with every item OK | max(10 min, 3 × landing estimate) | abort | |
| | opening (`cutting`) | the target's opened report (the existing finish gate needs it; a ready row alone is not enough) | max(3 min, 2 × open estimate) | blocked: page; stay in opening; the record names the target; retry each tick; an operator fixes the missing prerequisite (§7.5) | |
| | health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | go to finishing with `last.warning`; page (§7.5) | |
| -| finishing (`cutting`) | the existing finish gate: the opened report and every source box's parked report | 10 min | blocked: page; retry each tick; the data is live on the target; an operator fixes the missing report, never bypasses the gate | |
| +| finishing (`cutting`) | the existing two finish gates in turn: from `offline-open`, the target's opened report and every source box's parked report; then from `offline-finishing`, the sealer's lifted report and every source box's gone report | 10 min each | blocked: page; retry each tick; the data is live on the target; an operator fixes the missing report, never bypasses a gate | |
| | reverting (`cutting`) | abort's reports: the stager's prefix removed, every source box reopened | 10 min | blocked: page; retry each tick; the source's copy was never removed; `source reopened` is shown separately from `abort finished` | |
| `abort` before `open` is the existing abort (`offline-reverting`), lossless: the source's database was never removed, the target never took a lease, and the tombstone lifts. `open` is the point of no automatic return (§7.5). A lost response to any door write is an uncertain commit: the next tick rereads the record before deciding, never assumes. If `CARLOS_SELF_SERVE_MOVES` is turned off with moves in flight, the job keeps running them to completion and admits no new one. |
| ### 7.3 Lanes and pre-checks before anything goes offline |
| -**The lane.** The forward move stages from the source box into the target region's data bucket under `moves/<host>/` (source role writes the target bucket's `moves/*`). The source box proves the lane on its moves pass once the record is staging: it puts, lists and deletes one object under `moves/@lanes/<id>/` in the target bucket, a reserved namespace no consumer reads (`promoteStagedReplica` reads `moves/<host>/`, and `@`-prefixed names are the platform's own, as `moves/@offline/` is), and reports the result with the request's id; a crash between put and delete leaves nothing a wake could promote. The console requires the report before the first stage round. The reverse lane (target role to the source bucket's `moves/*`) is not a prerequisite of a self-serve move in this version, because no automatic return runs after `open`; it is what an operator's return needs, and the infrastructure provisions both (§9). M17's push-only shape stands: no role gains access to a sibling's `instances/*`. |
| +**The lane.** The forward move stages from the source box into the target region's data bucket under `moves/<host>/` (source role writes the target bucket's `moves/*`). The source box proves the lane on its moves pass while the record is queued: it puts, lists and deletes one object under `moves/@lanes/<id>/` in the target bucket, a reserved namespace no consumer reads (`promoteStagedReplica` reads `moves/<host>/`, and `@`-prefixed names are the platform's own, as `moves/@offline/` is), and reports the result with the request's id; a crash between put and delete leaves nothing a wake could promote. The console requires the report before the first stage round. The reverse lane (target role to the source bucket's `moves/*`) is not a prerequisite of a self-serve move in this version, because no automatic return runs after `open`; it is what an operator's return needs, and the infrastructure provisions both (§9). M17's push-only shape stands: no role gains access to a sibling's `instances/*`. |
| **Clean, then staged.** Before the first stage round the target bucket's `moves/<host>/` and `instances/<host>/` must hold no current objects and no `lock.json` (versions and delete markers left by earlier moves are fine, as the runbook says). After staging, `moves/<host>/` holds exactly this move's staged objects (the stager's receipts, bound to the move id, bucket and region); foreign objects refuse. |
| -**Target-ready report.** The target box, on its moves pass once the record is staging and again when asked at cut, writes `control/offline-moves/<host>/<id>/target/<box>.json` with facts, and the console decides. Facts: no route or timers for the host; `/opt/carlos/live/<host>` absent, or present and naming the build the record's channel resolves to; `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db` absent; `/data/programs/<host>.json` absent, or present and naming that same build; `/data/.offline-moves/<host>` absent or holding only earlier moves' directories; free disk at least twice the replica's size; the target bucket prefixes as above; capabilities and switches as required; `secrets_key_fp`. A stale live link naming another build is a refusal with that reason (#339's cause, caught before the stop). |
| +**Target-ready report.** The target box, on its moves pass while the record is queued (the initial cleanliness) and again when asked at cut, writes `control/offline-moves/<host>/<id>/target/<box>.json` with facts, and the console decides. Facts: no route or timers for the host; `/opt/carlos/live/<host>` absent, or present and naming the build the record's channel resolves to; `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db` absent; `/data/programs/<host>.json` absent, or present and naming that same build; `/data/.offline-moves/<host>` absent or holding only earlier moves' directories; free disk at least twice the replica's size; the target bucket prefixes as above; capabilities and switches as required; `secrets_key_fp`. A stale live link naming another build is a refusal with that reason (#339's cause, caught before the stop). |
| **Source report.** The source box's existing report gains: the instance's extra app directories beside its files directory (#339 point 4) with their sizes; an **archive eligibility** result from a dry run of the archive's own `Build` rules over the files directory and the env file (the size cap with the env file counted, the entry cap, unsupported file types, unreadable paths), refreshed at cut; `files_bytes`; the build the row runs; and whether root's stage path is armed on the box (`carlos-stage.path` active), which the target box reports too. A non-empty extra directory refuses before the stop with the reason naming it. An empty one is removed by the source box at `start`, after the stop, as the operators did with Paul's approval on 2026-10-21 (the app recreates it on the target); nothing is archived outside the files directory, so the archive's layout contract is unchanged. An eligibility failure refuses before the stop with the archive's own reason. |
| @@ -190,14 +190,14 @@ Every wait has a deadline derived from the estimate's terms with a floor; a stag |
| ### 7.5 Before open and after open |
| -**Before `open`, every failure returns the account automatically and losslessly.** That is the existing abort from any state up to `offline-landed`: the source's database and files were never removed (the park happens at `open`), its bucket replica is untouched under the tombstone, the target never took a lease and never ran, and the target parks what it extracted. The final copy, the archive, the config render and the landing's verification of every manifest item happen before `open`, and so do the target-ready facts and the lane; `abort` is the job's answer to every refusal and every deadline there (initiated and retried, §7.2), and the customer sees "failed, still in <region>" with the reason. |
| +**Before `open`, every failure initiates a lossless, automatic return.** That is the existing abort from any state up to `offline-landed`: the source's database and files were never removed (the park happens at `open`), its bucket replica is untouched under the tombstone, the target never took a lease and never ran, and the target parks what it extracted. The final copy, the archive, the config render and the landing's verification of every manifest item happen before `open`, and so do the target-ready facts and the lane; `abort` is the job's answer to every refusal and every deadline there (initiated and retried, §7.2), and the customer sees "failed, still in <region>" with the reason. |
| **What stays after `open`, by design of the offline move.** The target's executable is not staged before `open`: the landing's `release` item checks that the channel resolves to the build the seal recorded and allows an absent live link, and the build reaches the target through ordinary adoption and root's stage pass after the row is ready. The staged replica is promoted into `instances/<host>/` by the first wake, another mirror operation. The certificate is checked for eligibility, not for a usable mirrored copy. And the app's own start is after execution becomes possible. So a failed executable stage or install, a failed promotion or restore, a missing certificate and an app that will not start are post-open failures in this version, found by the health check and paged. Moving them before `open` means materialising and verifying the build, the promotion and the certificate on a held target before it opens, a change to the offline move's data path and landing invariants that this design does not make (§10 Q3 names it as the alternative). |
| **At `open` the target becomes the account's home, and there is no automatic return.** Rev 2 tried to decide a post-open return from the target bucket's lease and LTX position; Astra showed that neither proves the target never accepted a write (litestream releases the lease even when its final sync failed; a file-only change leaves no LTX; an absent lock is "never taken" or "released"), and that a safe post-open revert needs an admission barrier and a stop protocol for the target that this machinery does not have (Phase 1 has them, through the claim). So after `open`: |
| - **The app answers (two passes of §7.4):** `finish`, outcome `done`. |
| -- **The app does not answer within the window:** the job goes to `finishing` with `last.warning` naming the failure ("the account did not answer on <region> within <n> minutes") and pages operators; `finish` completes when its gate is met (the opened report and every source's parked report), and the move is `done` with the warning. Visitors reach the app once the target's tagged row is published and see what it answers. |
| +- **The app does not answer within the window:** the job goes to `finishing` with `last.warning` naming the failure ("the account did not answer on <region> within <n> minutes") and pages operators; `finish` completes when both its gates are met (opened and parked, then lifted and gone), and the move is `done` with the warning; "done" says the record names the target and the move's evidence is complete, not that the app is serving. Visitors reach the app once the target's tagged row is published and see what it answers. |
| - **The opened report is missing:** the job stays in `opening`, pages, and retries each tick; it never calls `finish` without the gate's evidence. The operator's recovery is to supply the missing prerequisite (the runbook: the target's archive-prefix deletion that `opened` waits on, a box that stopped reporting) and let the job or the operator's `finish` door complete it; no door bypasses the gate. |
| - **The operator's return** is the offline move the other way, run with the operator doors and the runbook, after reading the target's state by hand. It needs the reverse lane (§9), the originating box passing the target-ready facts of §7.3 with the forward move's leftovers in place. Those leftovers are: the park directory `<fwd id>/source/` and `<fwd id>/sealer/`, the lifted `lock.json` (owner `moved:<fwd id>`), the root-owned live link and `/data/programs/<host>.json` naming the build, for a nested source the empty `/data/.instances/<host>` and `/data/instances/<host>` and its mount view (`retirePaths`, runbook §8), and the old replica objects under `instances/<host>/` older than the forward's `open_at`. The target-ready report lists each as a fact; **the operator's start door accepts exactly that set for a return of move `<fwd id>`** (the report names the move id each leftover belongs to) and refuses anything else. That acceptance is also the contract an automatic return would need, so it is built now, and used by an operator. |
| @@ -207,7 +207,7 @@ The customer-facing promise is therefore: the account comes back to the old regi |
| - **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept so an operator's return has it): retired by the **source box** on its moves pass, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{id, move_id, host, account, app, incarnation, from, to, open_at, final_txid, tombstone_owner, bucket_inventory, local_inventory}`. `bucket_inventory` is the exact keys and current version ids under `instances/<host>/` at finish **excluding `lock.json`**, which is never deleted (the lifted tombstone is the cleanup's own evidence, as in the runbook). `local_inventory` is the root-owned paths with their file identities (device, inode, and the live link's target) for root's step. The cleanup is one operation with the `retired.json`'s id: the box takes a `cleanup` reservation for its own box (§7.1) and writes `cleaning_since` to `retired.json`; then, each condition at the time of deletion: the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; the box's registry has no row for the host; and the key is in the inventory with the inventory's current version id. Deletion is a plain `DeleteObject` per key, which on these versioned buckets writes a **delete marker** and keeps every version recoverable, as the runbook's cleanup did (`VersionId` deletes are never used: deleting the current version would make an older one current). Each deletion is logged key by key in the move log. **Retry and completion:** a key already carrying a delete marker is done; a key whose current version differs from the inventory's is skipped and recorded as `superseded` (another operation's object), never deleted; the obligation is `cleaned` only when every key is done or superseded, else it stays open with the skipped keys named and retries next pass; a `retired.json` for a host whose record has moved again, or whose box has a row again, is `superseded` as a whole and left for the agent's 7-day removal. The reservation is released after root's step (below) or when the obligation ends. |
| - **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after the obligation ended (`cleaned` or `superseded`). |
| -- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root does not observe reservations, it is handed exclusion. When the bucket step is `cleaned` and the box still holds the `cleanup` reservation, the agent writes a root request `/data/.offline-moves/<host>/<id>/root-cleanup.json` naming the `local_inventory` paths with their identities and touches the stage-request marker; root's pass (the stage-requested one or the hourly one) removes exactly those paths when each still has the recorded identity (device, inode, link target), the row is absent from the registry, and no live link or running process names the build directory, then writes `root-cleanup.done`. The agent releases the reservation only after `root-cleanup.done` (or after 24 hours with the request left in place and a page). Off every critical path: an operator's return accepts the leftovers (§7.5). One new census row at the version sweep's class, keyed on the request file. |
| +- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root does not observe reservations, it is handed exclusion. When the bucket step is `cleaned` and the box still holds the `cleanup` reservation, the agent writes a root request `/data/.offline-moves/<host>/<id>/root-cleanup.json` naming the `local_inventory` paths with their identities and touches the stage-request marker; root's pass (the stage-requested one or the hourly one) removes exactly those paths when each still has the recorded identity (device, inode, link target), the row is absent from the registry, and no live link or running process names the build directory, then writes `root-cleanup.done`. The reservation is released only after root writes `root-cleanup.done`, or after a cancellation root has acknowledged: the agent writes `root-cleanup.cancel`, root checks for it immediately before each deletion and, finding it, deletes nothing further and writes `root-cleanup.cancelled` naming what it had already removed; the agent releases only on `.done` or `.cancelled`. A request root never answers keeps the reservation held and pages every 10 minutes; nothing times out into a release. Supersession (§7.6, a host moved again or recreated) uses the same cancel and waits for the same acknowledgement before the reservation goes. Off every critical path: an operator's return accepts the leftovers (§7.5). One new census row at the version sweep's class, keyed on the request file. |
| - **The staged objects of an aborted move** (the target bucket's `moves/<host>/` prefix): removed by the stager on abort as today. |
| ### 7.7 Scheduling |
| @@ -256,7 +256,8 @@ Infra: both staging lanes for every region pair the console arms (today IE to UK |
| - The estimate's storage and the rolling median; the phase timings' recording. |
| - The `moves` column migration, the mint flag and the instance-bound refusal. |
| - The return's leftover acceptance in the operator start door's target-ready contract, keyed by the forward move id. |
| -- The `landed_move` tag on routes, the holding entries' shape, the edges' precedence rule and the probe mode (`X-Carlos-Probe`, `X-Carlos-Served-By`); the "taking longer" sentence's trigger. |
| +- `move_seq` on the record and `serving_seq` on rows, routes and holding entries; the edges' highest-sequence rule; the probe mode (`X-Carlos-Probe`, `X-Carlos-Served-By`); the "taking longer" sentence's trigger. |
| +- The root cleanup request, done and cancel files and root's check before each deletion. |
| - The `offline-queued` state and the `self_serve_moves` row's phases; the cleanup reservation kind in `active.json` and the create door's check of it. |
| - Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; a cut naming a stale id; each pre-check refusal (a stale live link, a non-empty extra directory, an ineligible archive, a failed lane probe); each deadline before open ending in a lossless abort; a post-open failure to answer ending in `done` with a warning and a page; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry and a holding entry winning over a stale route; two moves at once on disjoint boxes and two queued on one; a reservation reclaimed only on evidence; cleanup refusing when the record moved again, when a current version differs from the inventory, and when the reservation is held by a move. |