design: self-serve offline moves, rev 4 (Astra round 3)
Names the failures that stay after open (executable stage, promotion, certificate, the app's start) and drops the claim that everything checkable is checked before open; a durable admission order (row, reservation, offline-queued, lane and cleanliness, then staging) with participants pinned by box id; a cleanup reservation kind and a root handoff through a request file with path identities; lock.json excluded from the inventory and retry/supersession semantics; causal ordering of holding entries by a landed_move tag instead of record_seq, with the home map's freshness as is; a probe mode that refuses backhaul and stamps the serving box; deadlines as initiate-and-retry with abort requested, source reopened and abort finished shown separately; the finish gate's evidence in the table; stale text gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+40
−27
docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md+40 −27
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md| index 5106be866..452074b5b 100644 |
| --- a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| @@ -1,12 +1,12 @@ |
| # Self-serve offline moves: a tenant's customer moves their own instance |
| -Rev 3, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. |
| +Rev 4, 2026-10-21. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. Rev 3 (b77c01a9) was reviewed again (`selfserve-spec-review3.md`, NOT READY, 8 findings, the abort boundary accepted); rev 4 answers them: which failures stay after `open` (§7.5), the job's durable admission order (§7), cleanup's own reservation and the root handoff (§7.6), the holding entries' causal ordering (§6.4), the probe mode (§7.4), deadlines as initiate-and-retry with blocked states (§7.2), and the stale contradictions. |
| Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says exactly what the customer's request may do. |
| ## 1. Intent |
| -A Tito customer opens Account settings, Data residency, in Tito's dashboard, sees the regions our console lists as armed, picks one and clicks Move. They are told the account will be offline for a few minutes and about how many, and they confirm; the customer has the final say, nobody at Tito or at ours is in the loop. The platform copies the account's data to the new region while it keeps serving, then takes it offline, finishes the copy, lands it in the new region, checks it answers, and reopens it. From the moment it goes offline until it is back, the customer and their own visitors see a holding page: "Paul's account is temporarily offline. Please check back in approximately 3 minutes." If anything fails after the account has gone offline, the platform puts it back in the old region on its own, and the residency tab says so. Scheduling for later is nice to have; Tito's dashboard already offers a time, and the platform's start is "now" in this version. |
| +A Tito customer opens Account settings, Data residency, in Tito's dashboard, sees the regions our console lists as armed, picks one and clicks Move. They are told the account will be offline for a few minutes and about how many, and they confirm; the customer has the final say, nobody at Tito or at ours is in the loop. The platform copies the account's data to the new region while it keeps serving, then takes it offline, finishes the copy, lands it in the new region, checks it answers, and reopens it. From the moment it goes offline until the new region takes it, the customer and their own visitors see a holding page: "Paul's account is temporarily offline. Please check back in approximately 3 minutes." If anything fails while the old region still holds every copy, the platform puts the account back there on its own and the residency tab says so; once the new region has taken it, a failure to answer is caught within minutes and paged. Scheduling for later is nice to have; Tito's dashboard already offers a time, and the platform's start is "now" in this version. |
| **Why now.** Tito's customer move has been off since 2026-10-02 (`TITO_REMOTE_PLACEMENT`): the pre-Phase-1 move lost the per-host key and config (#175 P1). The offline move is the one move proven on real customer data (three Tito instances, 2026-10-21, 2.5 to 3 minutes offline each) and it carries everything (the offline plan's dependency list). It needs no host claims, so it does not wait for the claim cohort, acceptance 10.1 to 10.6 or the F1 owner surface. Phase 1's live move is weeks away for Tito and refuses Tito's router outright (M18). |
| @@ -22,7 +22,7 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| **Goals.** |
| 1. One request from Tito's router starts a move that runs to completion with no operator step: stage while serving, stop, seal, flip, open, health check, finish, and the cleanup that follows. |
| 2. The holding page while offline, with the account's name and a minutes estimate, on every edge that answers for the host, on every domain the host has. |
| -3. Automatic, lossless return to the old region on any failure before `open`, which is where every copy, verification and check happens (§7.5). After `open` the account is never left silently offline: the holding page stays until the app answers, the state says what happened, operators are paged, and the operator doors can run the return. |
| +3. Automatic, lossless return to the old region on any failure before `open`: the final copy, the archive, the config render and the landing's verification all happen there (§7.5). After `open` the account is never left silently offline: the state says what happened, operators are paged, and the operator doors run the return. The failures that stay after `open` are named (§7.5). |
| 4. An estimate before the customer confirms, and a better one once the data is staged. |
| 5. Tito's router keeps its calls (prepare, status, cut, abort) and state words. It needs one new credential and reads two new fields of the status view (§6.1). |
| 6. Only an owner-delegated credential can start, cut or cancel a self-serve move (§5). A member-minted or instance-bound token cannot. Operators keep their doors on every move, self-serve included: they are the recovery path. |
| @@ -63,9 +63,9 @@ A Tito customer opens Account settings, Data residency, in Tito's dashboard, see |
| 2. At the time, Tito's router calls `POST .../instances/{host}/move {"region":"uk","display_name":"Paul's account","requested_by":"<tito user id>"}` with its move-capable token (§5). The console answers 202 with the status view, state `preparing`. |
| 3. **Preparing.** The job reserves the move's boxes (§7.1), proves the staging lane (§7.3), stages while the instance serves until caught up (§6.2), runs the pre-checks (§7.3). When all pass the state is `ready` with the refined estimate. Tito polls `GET .../move` as today. |
| 4. Tito freezes its own account writes (its `frozen` step) and calls `POST .../move/cut`. The console answers 202, state `cutting`. |
| -5. **Cutting.** The job restages once more, then runs `start` (every source box holds, fences, reports), `seal` (tombstone, final round), `flip` (the landing's ensures and checks), `open`, the health check (§7.4), `finish`. From `start` until the target serves, every edge answers for the host with the holding page (§6.4). Expect 2.5 to 3 minutes for a 1.6 to 3.7 GB replica. |
| +5. **Cutting.** The job restages once more, then runs `start` (every source box holds, fences, reports), `seal` (tombstone, final round), `flip` (the landing's ensures and checks), `open`, the health check (§7.4), `finish`. From `start` until the target's row is ready and published, every edge answers for the host with the holding page (§6.4); the first requests after that wake the app as any cold wake does. Expect 2.5 to 3 minutes for a 1.6 to 3.7 GB replica. |
| 6. **Done.** The state goes to `none`, `last.outcome` is `done` with the new region, and Tito's sweep marks its row done and unfreezes. The residency tab shows the new region. |
| -7. **Failure.** Before `open`, any refusal or deadline aborts: the record returns to the source, the tombstone lifts, the source reopens, `last.outcome` is `aborted` with `last.reason`, and Tito shows `failed` with that reason. After `open` (§7.5) the account is in the new region by record; if it does not answer within the health window the holding page stays up ("...taking longer than expected. Please check back in a few minutes."), operators are paged, and the move completes as `done` with `last.warning` naming the failure; Tito shows the new region with the warning. An operator then fixes forward or runs the offline move back, with the runbook. |
| +7. **Failure.** Before `open`, any refusal or deadline aborts: the record returns to the source, the tombstone lifts, the source reopens, `last.outcome` is `aborted` with `last.reason`, and Tito shows `failed` with that reason. After `open` (§7.5) the account is in the new region by record; if it does not answer within the health window, operators are paged, and once the evidence `finish` needs exists the move completes as `done` with `last.warning` naming the failure; Tito shows the new region with the warning. An operator then fixes forward or runs the offline move back, with the runbook. |
| 8. **Cancel.** `DELETE .../move` aborts in `preparing` and `ready` (nothing was offline). In `cutting` it is refused (409): the customer confirmed, and a cut either completes or returns on its own. |
| ## 5. Who may ask: the move-capable token |
| @@ -119,46 +119,56 @@ From `start` until the target's row is ready, the host is held on every source b |
| - **Rendering.** An edge answers a request for a host or alias that appears in a fresh `offline_moves` entry, from any publisher, and for which no fresh edges file lists a ready route, with 503, `Retry-After: 30`, `noindex`, and the holding page: "`<display_name>` is temporarily offline. Please check back in approximately N minutes." with N from `until` and the estimate, counting down; without a display name, "This account is temporarily offline. ...". A ready route anywhere wins over a holding entry (precedence), so the window between `open` and the first edges-file refresh is the pending page's known 503-then-retry window and nothing worse. The source box's own edge answers its held rows the same way from its registry (the home map excludes a box's own publication). The page reuses the pending page's safe reload mechanics (location.replace, sessionStorage) with its give-up replaced by `until`. |
| - **TLS.** A host or alias in a holding entry is allowed for TLS exactly as it would be as a served route (the same `Allowed` decision, wildcard and mirrored certificates included), so the page is reachable on HTTPS; the entry never widens what the host could already present. |
| - **The display name.** Tito supplies it per request; the console validates it (at most 80 characters, printable, no control characters, no newlines) and stores it in the move; the edge renders it through contextual template escaping. It is the one piece of request-supplied text the page shows, by design (the customer named their own account), and it is shown to unauthenticated visitors, as the account's own pages are. Nothing else from the request, no header, path, diagnostic or `requested_by`, reaches the page. |
| -- **Ordering and freshness.** Every `offline_moves` entry and every route carries the record's `record_seq` at publication. An edge prefers, for a host, the freshest publication (edges file younger than the routes' own freshness rule, 2 minutes) with the highest `record_seq`; a ready route at a `record_seq` at or above the holding entry's wins; a holding entry at a higher `record_seq` than a stale ready route wins. The home map's last-good behaviour on a read failure applies to both. The target box's own edge serves its local ready row at once (local rows win locally), so a probe sent to the target answers from the app whatever other edges still show. |
| -- **Validity.** An entry is valid while the publishing box's record still carries the move and the box republishes it (every pass); there is no fixed expiry. The displayed estimate is separate from validity: past `offline_at + estimate` the page says "...is taking longer than expected. Please check back in a few minutes." and keeps retrying. |
| -- **Reopening.** The public reopens when the target's row is ready and its publication reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. |
| +- **Ordering.** Claims-off records may carry no `record_seq`, so the order is causal, not a counter. A holding entry names its move id. A box publishes a ready route for a host **with `landed_move: <id>`** when its row was minted by that move's landing (the target), and plain otherwise. For a host with a holding entry for move `<id>` in any fresh publication, an edge serves the holding page unless some fresh publication carries a ready route for the host tagged `landed_move: <id>`; a plain ready route (the source's own stale publication from before the stop, retained as last-good) does not win over a holding entry. A ready route tagged with a move the holding entries do not name (a later move) wins. The source box's own edge answers its held rows the same way from its registry. This replaces rev 2's "a ready route anywhere wins". |
| +- **Freshness.** The home map's existing behaviour stands: publications are refreshed on the edges' own cadence, unchanged publications are not rewritten, and a last-good publication is kept through a read failure. A holding entry therefore lives exactly as long as its publisher lists it, which is while the record carries the move and the publisher's row is not ready; there is no age rule and no fixed expiry. The displayed estimate is separate from validity: past `offline_at + estimate` the page says "...is taking longer than expected. Please check back in a few minutes." and keeps retrying. |
| +- **Reopening.** The public reopens when the target's row is ready and its tagged publication reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. A failure after that is visible to visitors as the app's own answers, as any instance failure is. |
| - **The health probe** (§7.4) goes to the target box directly, by its address from its edges file with the host as SNI, so it reaches the app whatever the edges show. |
| ## 7. The job |
| A console worker pass (`StartSelfServeMovesPass`, the `RunClaimWorker` shape of the other console passes), every 5 seconds, over the console's `self_serve_moves` table: one row per move, keyed by move id, with `host`, `account`, `app`, `phase`, `phase_started_at`, `deadline`, `display_name`, `estimate`, `health` (the observations so far) and `outcome`. The record carries the offline states as today plus `move.self_serve: true`, `display_name`, `offline_at` and `estimate_seconds` for the edges. Each tick advances each move by at most one step, judged from the evidence the operator doors read today (the record, the edges files, the boxes' status reports, the reports under `control/offline-moves/`), through the door bodies refactored into functions the doors and the job share. Two console processes never run the pass at once (the existing single-console assumption of the other passes). The job is idempotent and restart-safe: every phase's start, deadline and observations are in its row, and a tick that finds a phase's evidence already satisfied moves on. |
| +**Admission order** (every boundary has a recovery): |
| +1. The door writes the `self_serve_moves` row first, phase `requested`, with the host, account, app, region pair and the participant boxes pinned by box id (`control/boxes/<box>.json` identity, not label), before anything else. A door that dies here leaves a row the pass picks up next tick. |
| +2. The pass reserves the participants (§7.1). A row in `requested` with no reservation is retried each tick; a reservation naming a row's id is this row's. |
| +3. With the reservation held, the pass writes the record's move in a new state **`offline-queued`**: the move exists (so a second prepare 409s and the view says `preparing`), but no box acts on it; the box dispatcher starts staging only from `offline-staging`, so the console controls when copying begins. A record write that may or may not have landed is reread next tick (the id is in the row). |
| +4. The lane probe and the cleanliness check run against `offline-queued`; when both pass the pass transitions the record to `offline-staging` and the box begins the first round. |
| +5. A participant that is replaced (a new box publishes the region's bucket) does not participate: the reservation names the old box id, and the pass refuses the move at the next pre-check with the reason naming the box; an operator decides. |
| + |
| +A console crash at any step leaves a row whose phase and the record's state together say where it was, and the pass resumes from there; a row with no record move and a reservation older than the grace period is a failed admission the pass aborts and releases. |
| + |
| ### 7.1 Slots and concurrency |
| Today one `control/offline-moves/active.json` holds a deployment-wide lock with an acquisition-to-record grace period and a reread of uncertain commits (`offlineBegin`): a staged or moving instance blocks every other one. Beyond the record's one move, it protects the box's moves pass, which holds one lock for the whole pass, so a long stage round (142 s measured at 3.7 GB) would delay that box's stop-report refresh for another move past the doors' 2-minute freshness allowance. |
| -This design keeps that object and its discipline and makes it **per box**. `active.json` holds `reservations: {box: {move_id, host, reserved_at}}`, one entry per participating box of every move from reservation to release. A move's participants are every box of its source region and the target box (one each in this version). The protocol, in order: |
| +This design keeps that object and its discipline and makes it **per box**. `active.json` holds `reservations: {box_id: {kind, id, host, reserved_at}}`, one entry per participating box of every operation from reservation to release, where `kind` is `move` (a self-serve or operator move, `id` the move id) or `cleanup` (§7.6, `id` the `retired.json`'s id). Boxes are named by box id, never label. A move's participants are every box of its source region and the target box (one each in this version). The protocol, in order: |
| -1. **Reserve.** The door (self-serve or operator stage or start) reads `active.json`, refuses if any participant is reserved by another move id, writes every participant's entry in one conditional write (the object's ETag), and only then writes the record's transition. A reservation whose record transition has not appeared within the grace period (2 minutes, today's) is reclaimable by a later door or the job, but only after rereading the record and the console's `self_serve_moves` row and finding neither names the id; an unreadable record or row fails closed (no reclaim). |
| +1. **Reserve.** The reserver (the job for a self-serve move after its row exists; the operator stage or start door; the source box for a cleanup) reads `active.json`, refuses if any participant is reserved by another operation, writes every participant's entry in one conditional write (the object's ETag), and only then writes its own durable state (the record's `offline-queued`, the operator move's first state, the cleanup's `cleaning_since` in `retired.json`). A reservation whose durable state has not appeared within the grace period (2 minutes, today's) is reclaimable by a later reserver, but only after rereading, for a `move`, the record and the console's `self_serve_moves` row, and for a `cleanup` the box's own `retired.json` through its status report, and finding none names the id; anything unreadable fails closed (no reclaim). |
| 2. **Hold.** The reservation is held through every phase, through `finish`, and through an operator's recovery after open until the operator's `finish`. |
| 3. **Release.** The job (or the operator's finish door) removes the entries in one conditional write after the record's move has cleared and the row's outcome is written; a crash between the two leaves a reservation the next tick releases on the same evidence. |
| -A move whose participant is reserved waits in `preparing` before any lane probe or stage round, with `reason` "waiting for another move on this box", in order of request. Moves of instances on disjoint boxes run at once. Within a box, the one move in flight is the only one staging, stopping, sealing or landing there, so the stop-report refresh is never behind another move's copy. Cleanup (§7.6) takes a reservation for the host's box for its duration too, so no move, recreation through the create door (which already refuses a host whose replica holds data) or operator move starts under it. |
| +A move whose participant is reserved waits in `requested` (the view says `preparing`) before any record write, lane probe or stage round, with `reason` "waiting for another move on this box", in order of request. Moves of instances on disjoint boxes run at once. Within a box, the one move in flight is the only one staging, stopping, sealing or landing there, so the stop-report refresh is never behind another move's copy. Cleanup (§7.6) holds a `cleanup` reservation for the host's box for its whole duration, root deletion included, so no move, operator move or recreation (the create door checks the reservation as well as the replica) starts under it, and a move's reservation blocks cleanup in turn. |
| ### 7.2 Deadlines and the transition table |
| -Every wait has a deadline derived from the estimate's terms with a floor; a stage round reports only when it ends, so there is no sub-round stall rule, and a round past its deadline (max(10 min, 3 × its estimate)) counts as a failed round. Expiry before `open` aborts; after `open` it is §7.5. |
| +Every wait has a deadline derived from the estimate's terms with a floor; a stage round reports only when it ends, so there is no sub-round stall rule, and a round past its deadline (max(10 min, 3 × its estimate)) counts as a failed round. Expiry before `open` **initiates** abort; after `open` it is §7.5. Initiating is not completing: a box may still be inside a copy or a seal when the record says `offline-reverting`, and abort's own reports (the stager's prefix removed, every source reopened) come when that work ends or fails. The job records three facts separately in its row and the view: `abort requested`, `source reopened` (the sealer lifted the tombstone and a source box reports its row ready) and `abort finished` (every report in). The promise is automatic initiation and retry of a lossless abort, with a truthful blocked state when a participant or a bucket stays unavailable. |
| | phase (view) | waits for | deadline | on expiry | |
| |---|---|---|---| |
| | reserving (`preparing`) | the participants' reservations | none; `reason` says waiting | 24 h, then abort "not started in time" | |
| | lane (`preparing`) | the lane probe report (§7.3) | 5 min | abort, reason names the lane | |
| -| staging (`preparing`) | a matching round under the ready rule | 30 min or 10 rounds (§6.2) | go ready with the remaining bytes in the estimate | |
| +| queued (`preparing`) | the record write of `offline-queued` landing | one tick | reread by id | |
| +| staging (`preparing`) | a matching, successful round under the ready rule | 30 min or 10 rounds (§6.2) | ready only from a completed successful round (the last one); a failed or still-running round is not ready: wait for it, then abort if it failed | |
| | pre-checks (`preparing`) | target-ready and source reports | 5 min | abort, reason names the check | |
| | ready | the cut | 24 h | abort "not confirmed in time" | |
| | restage (`cutting`) | one more round | max(5 min, 2 × its estimate) | abort | |
| | stopping (`cutting`) | every source box's report: held, fenced, quiesced, no process | max(5 min, 2 × stop estimate) | abort | |
| | sealing (`cutting`) | the sealer's seal report | max(10 min, 3 × final-copy estimate) | abort | |
| | landing (`cutting`) | the landing report with every item OK | max(10 min, 3 × landing estimate) | abort | |
| -| opening (`cutting`) | the target's opened report, or its ready row in its edges file | max(3 min, 2 × open estimate) | page; stay in opening; the record names the target; the holding page stays while the target still publishes it (§7.5) | |
| -| health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | `finish` with `last.warning`; page (§7.5) | |
| -| finishing (`cutting`) | every box's finish report | 10 min | the move stays `cutting` with the reason; page after 10 min; retry each tick (nothing is lost: the data is live on the target) | |
| -| reverting (`cutting`) | abort's reports | 10 min | page; retry each tick; the source's copy was never removed | |
| +| opening (`cutting`) | the target's opened report (the existing finish gate needs it; a ready row alone is not enough) | max(3 min, 2 × open estimate) | blocked: page; stay in opening; the record names the target; retry each tick; an operator fixes the missing prerequisite (§7.5) | |
| +| health (`cutting`) | two passes of §7.4 | max(3 min, 2 × promotion estimate) + 30 s | go to finishing with `last.warning`; page (§7.5) | |
| +| finishing (`cutting`) | the existing finish gate: the opened report and every source box's parked report | 10 min | blocked: page; retry each tick; the data is live on the target; an operator fixes the missing report, never bypasses the gate | |
| +| reverting (`cutting`) | abort's reports: the stager's prefix removed, every source box reopened | 10 min | blocked: page; retry each tick; the source's copy was never removed; `source reopened` is shown separately from `abort finished` | |
| `abort` before `open` is the existing abort (`offline-reverting`), lossless: the source's database was never removed, the target never took a lease, and the tombstone lifts. `open` is the point of no automatic return (§7.5). A lost response to any door write is an uncertain commit: the next tick rereads the record before deciding, never assumes. If `CARLOS_SELF_SERVE_MOVES` is turned off with moves in flight, the job keeps running them to completion and admits no new one. |
| @@ -176,26 +186,29 @@ Every wait has a deadline derived from the estimate's terms with a floor; a stag |
| ### 7.4 The health check after open |
| -"Healthy" is: the app answers on the target. The probe is the manifest's ready path (`moves.ready_path`, `Manifest.ReadyPathOrDefault`, the path the legacy move used; Tito sets a path that answers 2xx without a session, as the runbook did with `/signin`), requested from the console to the target box directly (its address from its edges file, the host as SNI and `Host`), with a unique query value and `Cache-Control: no-cache` so the edge's cache cannot answer it, no redirects followed (a 3xx is not a pass), a 30 s timeout per attempt, and the response's platform headers proving the target box and the build the seal recorded served it (no backhaul). Two passes 30 s apart, inside the deadline of §7.2. The landing's manifest checks remain mandatory before `open` (the health check supplements them). The console records the time from `start` to the first pass with the staged sizes, for the estimate's rates. Application checks beyond the ready path (the store reachable, custom domains, mail) are the tenant's; Tito's ready path can cover what it cares about. |
| +"Healthy" is: the app answers on the target. The probe is the manifest's ready path (`moves.ready_path`, `Manifest.ReadyPathOrDefault`, the path the legacy move used; Tito sets a path that answers 2xx without a session, as the runbook did with `/signin`), requested from the console to the target box directly (its address from its edges file, the host as SNI and `Host`), with a unique query value and `Cache-Control: no-cache` so the edge's cache cannot answer it, no redirects followed (a 3xx is not a pass), a 30 s timeout per attempt, and in a **probe mode** this design adds to the edge: the request carries `X-Carlos-Probe: local`, and an edge that would backhaul it answers 503 "not served here" instead (so a backhauled success cannot pass), while an edge that serves it locally stamps `X-Carlos-Served-By: <box id>` on the response after stripping any header of that name the app set. The console requires `X-Carlos-Served-By` to name the target box and `x-carlos-version` to name the build the seal recorded. Two passes 30 s apart, inside the deadline of §7.2. The landing's manifest checks remain mandatory before `open` (the health check supplements them). The console records the time from `start` to the first pass with the staged sizes, for the estimate's rates. Application checks beyond the ready path (the store reachable, custom domains, mail) are the tenant's; Tito's ready path can cover what it cares about. |
| ### 7.5 Before open and after open |
| -**Before `open`, every failure returns the account automatically and losslessly.** That is the existing abort from any state up to `offline-landed`: the source's database and files were never removed (the park happens at `open`), its bucket replica is untouched under the tombstone, the target never took a lease and never ran, and the target parks what it extracted. Every data-carrying step (the final copy, the archive, the config render) and every check (the manifest items, the build, the target-ready facts, the lane) happens before `open`, so this is where failures that a return can fix are found. `abort` is the job's answer to every refusal and every deadline there, and the customer sees "failed, still in <region>" with the reason. |
| +**Before `open`, every failure returns the account automatically and losslessly.** That is the existing abort from any state up to `offline-landed`: the source's database and files were never removed (the park happens at `open`), its bucket replica is untouched under the tombstone, the target never took a lease and never ran, and the target parks what it extracted. The final copy, the archive, the config render and the landing's verification of every manifest item happen before `open`, and so do the target-ready facts and the lane; `abort` is the job's answer to every refusal and every deadline there (initiated and retried, §7.2), and the customer sees "failed, still in <region>" with the reason. |
| + |
| +**What stays after `open`, by design of the offline move.** The target's executable is not staged before `open`: the landing's `release` item checks that the channel resolves to the build the seal recorded and allows an absent live link, and the build reaches the target through ordinary adoption and root's stage pass after the row is ready. The staged replica is promoted into `instances/<host>/` by the first wake, another mirror operation. The certificate is checked for eligibility, not for a usable mirrored copy. And the app's own start is after execution becomes possible. So a failed executable stage or install, a failed promotion or restore, a missing certificate and an app that will not start are post-open failures in this version, found by the health check and paged. Moving them before `open` means materialising and verifying the build, the promotion and the certificate on a held target before it opens, a change to the offline move's data path and landing invariants that this design does not make (§10 Q3 names it as the alternative). |
| **At `open` the target becomes the account's home, and there is no automatic return.** Rev 2 tried to decide a post-open return from the target bucket's lease and LTX position; Astra showed that neither proves the target never accepted a write (litestream releases the lease even when its final sync failed; a file-only change leaves no LTX; an absent lock is "never taken" or "released"), and that a safe post-open revert needs an admission barrier and a stop protocol for the target that this machinery does not have (Phase 1 has them, through the claim). So after `open`: |
| - **The app answers (two passes of §7.4):** `finish`, outcome `done`. |
| -- **The opened report is missing or the app does not answer within the window:** the job calls `finish` once the opened report exists (the row is ready and the record names the target, so the account is where the record says), records `last.outcome: done` with `last.warning` naming the failure ("the account did not answer on <region> within <n> minutes"), and pages operators. While the target box still publishes the holding entry (its row not ready) the page stays up with the "taking longer than expected" sentence; once the row is ready, visitors reach the app and see what it answers. Without an opened report within its deadline the job stays in `opening`, pages, and leaves the record alone: the operator's `finish` and `status` doors and the runbook take over. |
| +- **The app does not answer within the window:** the job goes to `finishing` with `last.warning` naming the failure ("the account did not answer on <region> within <n> minutes") and pages operators; `finish` completes when its gate is met (the opened report and every source's parked report), and the move is `done` with the warning. Visitors reach the app once the target's tagged row is published and see what it answers. |
| +- **The opened report is missing:** the job stays in `opening`, pages, and retries each tick; it never calls `finish` without the gate's evidence. The operator's recovery is to supply the missing prerequisite (the runbook: the target's archive-prefix deletion that `opened` waits on, a box that stopped reporting) and let the job or the operator's `finish` door complete it; no door bypasses the gate. |
| - **The operator's return** is the offline move the other way, run with the operator doors and the runbook, after reading the target's state by hand. It needs the reverse lane (§9), the originating box passing the target-ready facts of §7.3 with the forward move's leftovers in place. Those leftovers are: the park directory `<fwd id>/source/` and `<fwd id>/sealer/`, the lifted `lock.json` (owner `moved:<fwd id>`), the root-owned live link and `/data/programs/<host>.json` naming the build, for a nested source the empty `/data/.instances/<host>` and `/data/instances/<host>` and its mount view (`retirePaths`, runbook §8), and the old replica objects under `instances/<host>/` older than the forward's `open_at`. The target-ready report lists each as a fact; **the operator's start door accepts exactly that set for a return of move `<fwd id>`** (the report names the move id each leftover belongs to) and refuses anything else. That acceptance is also the contract an automatic return would need, so it is built now, and used by an operator. |
| The customer-facing promise is therefore: the account comes back to the old region on its own for any failure up to the point its data is verified and landed in the new region; after that, it is in the new region, a failure to answer is caught within minutes and paged, and a person moves it back if that is the right call. §10 Q3 asks Paul to accept this over rev 2's broader promise. |
| ### 7.6 Cleanup: who deletes what, from what evidence |
| -- **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept so an operator's return has it): retired by the **source box** on its moves pass, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{move_id, host, account, app, incarnation, from, to, open_at, final_txid, tombstone_owner, inventory}` (the inventory is the exact keys and current version ids it listed at finish). Conditions, all at the time of deletion: the box holds the host's reservation in `active.json` (§7.1), so no move, return or recreation starts during the cleanup; the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; every key is in the inventory and its current version id is the inventory's; and the box's registry has no row for the host. Deletion is a plain `DeleteObject` per key, which on these versioned buckets writes a **delete marker** and keeps every version recoverable, as the runbook's cleanup did (`VersionId` deletes are never used: deleting the current version would make an older one current). A key whose current version differs from the inventory's is skipped and reported. Completion is written to `retired.json` (`cleaned_at`) and the reservation released. Each deletion is logged key by key in the move log. A key that appears again later (a later move's staging, a recreation) is that operation's, never this cleanup's: the inventory binds the obligation to the forward move's objects. |
| -- **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after `cleaned_at`. |
| -- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root's hourly pass removes exactly the paths `retired.json` names, when it is at least 10 minutes old, the reservation is the cleanup's or none, the row is absent from the registry, and no live link or running process names the build directory; never a path outside the inventory. Off every critical path: an operator's return accepts them (§7.5). One new census row at the version sweep's class. |
| -- **The staged objects of an aborted move** (`moves/<host>/<id>/` in the target bucket): removed by the stager on abort as today. |
| +- **The source region's replica** (the forward's copy under `instances/<host>/` in the source bucket, kept so an operator's return has it): retired by the **source box** on its moves pass, no sooner than 24 hours after the move's `finish`, from a `retired.json` the source box wrote at `finish` under `/data/.offline-moves/<host>/<id>/` with `{id, move_id, host, account, app, incarnation, from, to, open_at, final_txid, tombstone_owner, bucket_inventory, local_inventory}`. `bucket_inventory` is the exact keys and current version ids under `instances/<host>/` at finish **excluding `lock.json`**, which is never deleted (the lifted tombstone is the cleanup's own evidence, as in the runbook). `local_inventory` is the root-owned paths with their file identities (device, inode, and the live link's target) for root's step. The cleanup is one operation with the `retired.json`'s id: the box takes a `cleanup` reservation for its own box (§7.1) and writes `cleaning_since` to `retired.json`; then, each condition at the time of deletion: the record names `to` (or a later region) and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id; the box's registry has no row for the host; and the key is in the inventory with the inventory's current version id. Deletion is a plain `DeleteObject` per key, which on these versioned buckets writes a **delete marker** and keeps every version recoverable, as the runbook's cleanup did (`VersionId` deletes are never used: deleting the current version would make an older one current). Each deletion is logged key by key in the move log. **Retry and completion:** a key already carrying a delete marker is done; a key whose current version differs from the inventory's is skipped and recorded as `superseded` (another operation's object), never deleted; the obligation is `cleaned` only when every key is done or superseded, else it stays open with the skipped keys named and retries next pass; a `retired.json` for a host whose record has moved again, or whose box has a row again, is `superseded` as a whole and left for the agent's 7-day removal. The reservation is released after root's step (below) or when the obligation ends. |
| +- **Parked local state** (`/data/.offline-moves/<host>/<id>/`): removed by the agent 7 days after the obligation ended (`cleaned` or `superseded`). |
| +- **Root-owned leftovers** (`/opt/carlos/live/<host>`, `/data/programs/<host>.json`): root does not observe reservations, it is handed exclusion. When the bucket step is `cleaned` and the box still holds the `cleanup` reservation, the agent writes a root request `/data/.offline-moves/<host>/<id>/root-cleanup.json` naming the `local_inventory` paths with their identities and touches the stage-request marker; root's pass (the stage-requested one or the hourly one) removes exactly those paths when each still has the recorded identity (device, inode, link target), the row is absent from the registry, and no live link or running process names the build directory, then writes `root-cleanup.done`. The agent releases the reservation only after `root-cleanup.done` (or after 24 hours with the request left in place and a page). Off every critical path: an operator's return accepts the leftovers (§7.5). One new census row at the version sweep's class, keyed on the request file. |
| +- **The staged objects of an aborted move** (the target bucket's `moves/<host>/` prefix): removed by the stager on abort as today. |
| ### 7.7 Scheduling |
| @@ -229,7 +242,7 @@ Infra: both staging lanes for every region pair the console arms (today IE to UK |
| - **Q1.** The move-capable token (§5) as the answer to M18 for this door, app-wide and owner-minted. Default: yes. |
| - **Q2.** Cancel during `cutting` is refused. Default: refused; the customer confirmed. |
| -- **Q3 (changes a ruling).** Rev 3 returns the account automatically only for failures before `open`; after `open` a failure is caught and paged within minutes and the return is an operator's run of the offline move back. Rev 2's automatic post-open return could not be made lossless with claims-off machinery (§7.5). Default: as stated. The alternative is an admission barrier and a target stop protocol, which is Phase 1's claim; say if you want that built instead and the self-serve move waits for it. |
| +- **Q3 (changes a ruling).** The account returns automatically only for failures before `open`; after `open` a failure is caught and paged within minutes and the return is an operator's run of the offline move back. The failures that stay after `open` are the target's executable stage, the replica's promotion, a missing certificate and the app's own start (§7.5). Default: as stated. The alternatives are (a) materialising and verifying the build, the promotion and the certificate on the held target before it opens (a change to the offline move's data path), or (b) an admission barrier and a target stop protocol for a post-open revert (Phase 1's claim); say if you want either built first and the self-serve move waits for it. |
| - **Q4.** The source replica deletion 24 hours after done, by the source box, with the runbook's checks. Default: automatic. |
| - **Q5.** The archive's 256 MiB files limit (M5) stays and is a refusal before confirming. Raising it is separate work; say if you want it in this version. |
| - **Q6.** The display name is Tito-supplied per request, validated, shown to unauthenticated visitors. Default: yes. |
| @@ -240,11 +253,11 @@ Infra: both staging lanes for every region pair the console arms (today IE to UK |
| - The `self_serve_moves` table and the slot file's CAS discipline; the view mapping's tests against Tito's actual polling sequence (prepare, poll to ready, cut, poll to none, read `last`). |
| - The edges-file `offline_moves` field, its readers in the home map and the local edge, the TLS allow, and the audit of every edges-file decoder. |
| - The target-ready, lane and source report shapes and their census rows; the per-host block in the status report. |
| -- The archive's directory entries for empty extra directories and their restore location. |
| - The estimate's storage and the rolling median; the phase timings' recording. |
| - The `moves` column migration, the mint flag and the instance-bound refusal. |
| - The return's leftover acceptance in the operator start door's target-ready contract, keyed by the forward move id. |
| -- The `record_seq` on routes and holding entries and the edges' precedence rule; the "taking longer" sentence's trigger. |
| +- The `landed_move` tag on routes, the holding entries' shape, the edges' precedence rule and the probe mode (`X-Carlos-Probe`, `X-Carlos-Served-By`); the "taking longer" sentence's trigger. |
| +- The `offline-queued` state and the `self_serve_moves` row's phases; the cleanup reservation kind in `active.json` and the create door's check of it. |
| - Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; a cut naming a stale id; each pre-check refusal (a stale live link, a non-empty extra directory, an ineligible archive, a failed lane probe); each deadline before open ending in a lossless abort; a post-open failure to answer ending in `done` with a warning and a page; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry and a holding entry winning over a stale route; two moves at once on disjoint boxes and two queued on one; a reservation reclaimed only on evidence; cleanup refusing when the record moved again, when a current version differs from the inventory, and when the reservation is held by a move. |
| ## 12. Risks |