carlos / platform Public
Offline operator move Implementation Plan
> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Rev 2, 2026-10-20. Written against main at 96db0efe on branch moves-offline. Rev 1 (04a2f8da) was reviewed by Fable ("build with listed changes"); rev 2 applies all of it: B1 (the hold decides from the record alone), B2 (the landing is a list of idempotent ensures), S1 to S9, the simplification (no probe, no held release, no root-side change), the revised task list and the right-sized end-to-end test. It also folds in platform-1e's rulings (a), (b), (c) as Fable sharpened them (§ "Interaction with the host claim"), and the eight requirements of Fable's claim-path review (fable-plan-review.md §9, with §8's note on the legacy move), each traced to the task that meets it (§ "Requirements from the claim-path review"). No spec of its own: this plan is the design, and it borrows its inventory and failure analysis from the move-any-app spec (docs/superpowers/specs/2026-10-02-move-any-app-design.md §1, §6, §7.13) and the v0.22 move (docs/superpowers/specs/2026-08-28-instance-move-design.md), without the host claim.
Goal: An operator-run move of one hibernating instance between two regions with planned downtime, shipped by morning, for three instances of one customer moving from Ireland (the flagship's home placement, record region "", console name ie) to the UK (uk, eu-west-2). The operator stops the instance, the move copies a checked list of every dependency, verifies every item on the target with no traffic, and only then opens it. No self-serve, no background recovery, and no dependency on CARLOS_HOST_CLAIM, CARLOS_MOVES_PHASE1, CONSOLE_APP_REGIONS or CARLOS_SWAP.
Architecture: The record carries an offline move (move.kind: "offline") through eight states, each advanced by an operator door on the console or by one box's step in the existing moves pass. The stop is three layers: the record makes every source-region box hold its row (reconcile's generic Hold hook, decided from the record alone), each of those boxes fences the activator and reports, and one box (the sealer) turns "the source region's single-writer lease is free" into a 7-day tombstone lease, re-asserted every pass and lifted at finish, before it reads the final position. The data travels through the v0.22 staging lane (moves/<host>/, IAM already applied for IE to UK), the files and <host>.env travel as the Phase 1 sealed archive, and the target lands every item on a held row as idempotent ensures. Nothing starts on the target before open. After open the hold is released and the build arrives the way every new instance's does, through ordinary adoption and root's existing stage pass; the first request promotes the staged replica. Before open, abort flips the record back to a source that nothing touched, and the target never took its lease.
Tech Stack: Go. internal/provision, internal/moves (+ archive), internal/lease, internal/host, internal/console, internal/layout, the carlos CLI (move.go). No change to internal/activator and no root-side change. Tests use leasetest.NewS3Server (the S3 wire protocol in-process), release.DirStore, real registries, real litestream replicas, and the console's doors in-process.
Answers to the brief's questions
What a source-region box does when the record's region changes away from it. On its next reconcile pass (a serial bump makes that seconds; the floor is the 5-minute timer) the record fails inScope (internal/provision/reconcile.go:401), is absent from recs, and the removal loop retires the Provisioned row as "record gone" (reconcile.go:1505-1563, cfg.claimRetire then registry.Delete). The edge reload drops the route and RetireExcept stops a running child, whose final checkpoint failure is logged and swallowed (activator.go:5023; movespass.go's own comment at :650). So it does stop serving, but not as a stop you can rely on: (1) until that pass runs the row is still ready and any request or tick can wake it; (2) the local database stays on disk, and a later flip back trusts it as a warm cache before any bucket state (internal/activator/backend_exec.go:796); (3) nothing at the data layer refuses a wake, because the source bucket's lease (instances/<host>/lock.json) is free whenever the instance sleeps; (4) a box that is down at the flip keeps its ready row, and on restart wakes are admitted once the startup latch closes while the reconcile loops start only after it (internal/host/layoutleg.go:380-422, agent.go:670-809). A record flip alone is not a stop.
What a target-region box does when the record names its region. Its next reconcile pass mints a ready row (RouteFor, reconcile.go:434), subject only to the config gate (#213) and CARLOS_INBOUND. The edges file publishes it, so every edge backhauls to it. The first request wakes it: lease in the target bucket, then restorePlain (backend_exec.go:784-880): a local DB is a warm cache and wins; else a staged replica under moves/<host>/ is promoted and wins (promoteStagedReplica, :3392); else the own replica; else a fresh boot and seed. If <host>.env is absent the activator mints a new CARLOS_INSTANCE_KEY; if the files directory is absent the app makes a new key file; <host>.config.env exists only if this box holds CARLOS_SECRETS_KEY. That is the kpmg outage, mechanically.
How the v0.22 move stopped the source, and what it missed. move/cut set state: cutting; the source box that owns the row (OwnsRoute) set status moving, installed the in-memory activator fence, waited for Quiesced, re-checkpointed the local DB under the lease, ran a final mirror round with zero deferred objects, purged the local DB, and flipped region and cleared move in one CAS (internal/host/movespass.go:1280-1380). It missed: the files directory and <host>.env (key, admin token, overrides); a target without the secrets key; every other box in the source region (each has its own registry and warm cache, #286); a fence that survives an agent restart (only the moving status and ServeScheduled's gate did); any lease-level fence; any check of the target before it opened (it opened on the flip); any rollback but a hand SetRecordRegion; timers, NextDue and Replication.
What of Phase 1 is reusable without the claim. Reused as is: lease.MirrorPrefix with copy receipts and the lock.json skip (internal/lease/mirror.go:175, :227); the legacy lane moves/<host>/ and its promotion on the target's first wake (backend_exec.go:3392); internal/moves/archive (Build, Seal, Open, Verify, Extract, PlainDigest; pure); the env pin (moves.ReportLaunch, PortableDigest, CompareEnv, Pinned; internal/moves/envreport.go) over ExecBackend.LaunchLayers (backend_exec.go:2760); appconfig.RenderHeld (internal/appconfig/renderheld.go:51, claim-off only, admits a row held by the op, :138-152); reconcile's Hold/KeepParticipant hooks, mintHeld, holdExisting (reconcile.go:1201, :1580, :1692); registry.SetHold/ReleaseHold/DeleteIfHold/SetNextDue/SetReplication/ReplaceTimers/TimersFor; activator.FenceAndStopAs/LiftFenceAs/Quiesced; lease.VerifyReplica, lease.LockState; litestream's lss3.Leaser (settable Owner and TTL, conditional writes); localLTXPosition, newestLTX (movescut.go:1443, movesreport.go:599); moves.TimersSnapshot; moves.NewMoveID; cmd/moveprobe (dev rehearsal only). Not used: ProbeWake (claim admission), the C8 files provider (movesfiles.go, claim chain), stopevidence.Reconstruct, per-op versioned staging lanes, the Phase 1 door (claim CAS), and, after Fable's review, the held release (heldReleaseReady) and any root authority: the build reaches the target after open by ordinary adoption.
Why this does not build on the legacy cut
The legacy move on main already fences, quiesces, mirrors byte-final with Deferred == 0, purges and flips last: the console door (internal/console/apiinstancemove.go:405-466, taken while MovesEnabled is off), the box round (internal/host/movespass.go:1066-1095, :1219-1377) and the source row's retirement on the next reconcile pass (reconcile.go:1533-1563). It moved kpmg on 2026-10-02 and failed only on the missing halves. Built on directly, it still needs every half this plan adds: a held target row (its flip clears move, so the target mints ready and the first request starts the app before any file lands, which is the outage), the files and env archive, the config render, the operator gates before the flip and before serving, and a data-layer fence (it fences only the OwnsRoute box in memory, and the first review's wake-path list shows a second home box, a box returning from an outage, and a direct RunCmd-shaped wake after an agent restart all reach Acquire with nothing but the lease in the way). Those halves are the same code either way; what differs is the source side, where the legacy cut would save the per-box reports but would have to change a live production state machine (MoveComplete clearing the move at the flip, the preparing/ready rounds while serving, the legacy abort) for every legacy record too. So this plan keeps its own record states and the hold/report/tombstone source side, and reuses the legacy cut's code and its order step for step: the fence keyed by the move generation (moveFenceGeneration, movespass.go:1290-1296), quiesce, the layout journal absent (:1311), CarryRequest and the final checkpoint (:1316-1348), the byte-final round with Deferred == 0 (:1349-1364), and the warm-cache purge (PurgeLocalDBArtifacts, here a rename into the park directory). The one reordering is that the purge comes at open, after the flip, instead of before it: the tombstone, the held rows and the fence already make the source unable to run, and keeping the local copy until open makes every abort before open lossless without a second move.
How stop is guaranteed
Three layers, in this order, each checked by evidence before the next step. Fable re-walked every path that can start an exec instance (edge, backhaul, ticks, the due sweep, carlos run, ZDD swaps, restarts, workers, the instance-key backfill, config sync, adoption, the layout pass, claim-only paths, and the wake itself) against these three layers at 96db0efe; each is stopped by at least one of them.
- The record holds every source-region row, durably.
startwritesmove{kind: offline, state: offline-stopping}. Every box whose placement is the source's runs reconcile with an offlineHoldhook (Task 3) that decides from the record passed to it and the box's placement, never from a snapshot, and holds its rowmovingunder the move id through the existingholdExisting/mintHeld(a box with no row mints it held). A held row is in the registry on disk, so it survives an agent restart. Held means: the edge does not serve it (internal/edge/edge.go:654), the edges file does not list it (internal/host/edgesfile.go:411-424), ticks are refused (ServeScheduled,activator.go:5317-5323), the due-wake sweep skips it (activator.go:4942), the layout pass leaves it (internal/layout/pass.go:230), config sync and adoption leave it (syncexec.go:765,release/adopt.go:1186), and the instance-key backfill skips it (internal/host/instancekeys.go:104-108). - Every source-region box fences and reports. Each such box's offline step installs the activator fence (
FenceAndStopAs, owneroffline:<id>, re-installed every pass so a restart regains it), waits forQuiesced, reads the host's cgroups for any process left (the exec cgroupcarlos-exec@<host>under the activator's root,cgroup.ExecNameand, for a nested row, thecarlos-instance@<iid>unit's), and writescontrol/offline-moves/<host>/<id>/source/<box>.json: row held or absent, fence installed, no live entry, the process count, the layout journal absent (layout.JournalPresent, as the legacy cut checks atmovespass.go:1311), local state present (DB file, the activator's litestream meta dir, files directory, env file), the local LTX position, a sidecar file. Thesealdoor refuses until every box whose edges file names the source placement (with hibernation configured) has a report saying so and its edges file no longer lists the host, and refuses if more than one box holds local state. A box that was down whenstartran still lists the host and still has a ready row, sosealwaits for it to come back and hold; anything it served in that window was before the stop, and nothing has been sealed yet. - One box seals the source region's lease. The sealer (the box with local state, else the lexically first source box) checkpoints its local DB if it is ahead of the replica (
cfg.Checkpointtakes and releases the lease), then callslease.Tombstone: a conditional write ofinstances/<host>/lock.jsonwith ownermoved:<id>and a 7-day expiry, which succeeds only if no writer holds the lease. That write is the linearization point. Every wake on every box in the source region, live, restarted or returning from an outage, must acquire that lease first (ExecBackend.Acquire,backend_exec.go:692-714, the one lease take for every wake source) and now getsHeldError. The final position (newest LTX TXID) is read after the tombstone, and every source box's reported local position must be at or below it. The sealer re-asserts the tombstone (same owner,If-Match) on every pass fromoffline-sealinguntil it lifts it atfinishor on abort; a move abandoned mid-way frees the lease on its own within a week.
The operator confirms before the flip. flip is the operator's explicit confirmation, and the door also requires every source report written within the last two minutes to show zero processes in the host's cgroups and no layout journal, so the one thing the code cannot otherwise see (a process outside the agent that installed the fence) is read on the box itself. The runbook has the operator read carlos logs for "instance exited" first.
The target cannot start before that evidence exists. It has no row until flip, and flip requires offline-sealed, which only the sealer writes after the tombstone and the final mirror round. After the flip its row is minted held, even on a target box that restarts and reconciles before its first moves pass (B1: the hook needs no view). Nothing starts on the target before open: the landing installs and verifies files, not processes, and never takes the target's lease. open itself re-checks the stop: the sealer's report from this pass must show LockState held by moved:<id>, and every source report must still say held, fenced and quiesced.
An unreachable source box never reports, so seal refuses and names it. The move waits (abort is always available before open). There is no override: on the flagship the home region has one box, the console box, and if it is unreachable there is no console either. If a box is gone for good its edges file has to be removed by hand, with Paul's approval, before any move from that region (runbook P0).
The dependency list
internal/moves/offline.go holds it as OfflineManifest, one row per item. The seal report and the landing report each carry a moves.Check per item they own, computed from the state on disk and in the bucket, not from what the pass did. TestOfflineManifestIsCovered pins that every row is owned by exactly one of {door refusal, seal, landing, not carried (with a reason)}, and the doors refuse flip and open unless every owned item is OK.
| id | item | disposition | proof, and the code that makes it complete | |---|---|---|---| | db | SQLite DB at its final position | carried: instances/<host>/ mirrored to the target's moves/<host>/; promoted by the target's first wake after open | seal: listing of the staged prefix (key, size, ETag) after a final MirrorPrefix round with Deferred == 0 (mirror.go:175); landing: its own listing equals the seal's, lease.VerifyReplica on the staged prefix (check.go:88, integrity check), newest LTX TXID equals FinalTXID | | db-tail | writes the source's stop failed to checkpoint | carried by the sealer's checkpoint | every source report's local position (localLTXPosition) at or below the replica's TXID after the tombstone; the sealer checkpoints first when it is ahead (moveCfg.Checkpoint, movespass.go:306) | | build-state | instances/<host>/build-state.json | carried inside the replica prefix | release.BuildStateKey; the sealer runs CarryRequest before the final round, as the legacy cut does (movespass.go:1344) | | lease | instances/<host>/lock.json | not carried (the mirror skips it, mirror.go:227); the source's is the tombstone until finish; a stale moved:* tombstone of another move in the target bucket is lifted at landing | lease.LockState on both sides; open re-checks the source's | | files | flat <DataDir>/<host>/; nested InstanceDir minus platform artifacts | carried: sealed archive at moves/@offline/<host>/<id>/files.sealed in the target bucket | ensure: present with archive.PlainDigest equal to the seal's | | env | <host>.env: CARLOS_INSTANCE_KEY, CARLOS_ADMIN_TOKEN, overrides | carried byte for byte inside the archive (archive/instancekey_test.go) | ensure: present, sha256 of its bytes and of the effective instance key equal to the seal's; covered again by env-pin | | key-adoption | <host>.instance-key-adoption.json | not carried: written only for claim launches (internal/activator/keyadoption.go:13-16); a target copy is parked | ensure: absent | | config | <host>.config.env: the app's settings, store binding, secrets | rendered on the target from the bucket bundle (appconfig.RenderHeld) | the start door requires the target's secrets_key_fp to equal the console's (edgesfile.go:283, R9); ensure: rendered; covered by env-pin | | env-pin | the composed launch environment | app settings must be equal; box settings are not compared (moves.Pinned: box-owned variables are left out unless the instance supplied them) | PortableDigest from LaunchLayers on both rows equal; on a mismatch the landing reports CompareEnv's variable names, never values | | release | the build the source runs | not copied: after open, ordinary adoption stages the channel's build for the now-ready row through root's existing stage pass, as for any new instance | landing: the app's channel resolves to the build the seal recorded (release.LiveVersion on the sealer); a promote during the move fails the item, and the runbook says not to promote during a move | | store | object store bucket and its credentials | not moved: the bucket stays where it is and the binding travels in config | env-pin; store data residency is a non-goal | | timers | timer rows, once-timers the app armed included | carried as a moves.TimersSnapshot beside the archive; registry.ReplaceTimers on the held row (claim-off it is replaceLegacyTimers; no foreign key to routes, registry.go:667) | ensure: TimersDigest of the row's timers equals the seal's | | next-due | the row's NextDue | carried in MoveSource.NextDue | ensure: registry.SetNextDue, read back equal | | replication | the row's Replication rung | carried in MoveSource.Replication | ensure: registry.SetReplication, read back equal | | row | the registry row | reproduced by reconcile from the record, minted held at flat paths (mintHeld) | ensure: row exists, held landing by the move id | | routing | edges file, backhaul, certificates, derived aliases | produced by existing passes once the row is ready | runbook P0 confirms a certificate path for the host on the target (#310) and that the store is reachable from the target region; after open, the URL answers from the target box | | source-retire | everything the source box keeps for the host after open | parked or removed, and listed path by path in the source's parked report: <DataDir>/<host>.db with WAL, SHM and both litestream meta dirs, /data/<host>/, <host>.env, <host>.config.env, /data/programs/<host>.json, the nested /data/.instances/<iid> contents and its /data/instances/<iid> view, /opt/carlos/live/<host>; the row removed by reconcile at finishing; certificates stay (box state) | the parked and gone reports; each path's action is parked, absent or left with a reason (/opt/carlos/live/<host> and a mount are root-owned: left, listed for the census) | | sidecar | <host>-sidecar.env | refused: a source report that sees one fails seal | source report | | restore-verify | control/restore-verify.json entry | not carried: the next drill writes a fresh verdict; until then the old verdict describes the source copy | none (stated in the runbook) | | warm-cache | source local DB, meta dirs, files, env after open; target local state found at landing | parked under <DataDir>/.offline-moves/<host>/<id>/<role>/ by rename, never deleted | parked reports; target ensure: no local DB or meta dir | | receipts | move-receipts/<host>.json in the source bucket | cleared at finish (clearMoveReceipts) | none needed (self-checking) | | class | kind, addr, fleet, pool, hibernate, replica, replicas, steering, unit, seed, open run | refused at the door (moveGate plus the transition) | door tests |
Each row has two fields. Owner is the step that proves the item where it must end up, and exactly one step owns each row; the open door requires every landing-owned item OK in the target's report. Source is the step that reports the item's expected value on the source side; the flip door requires every Source: seal item OK in the sealer's report (SourceItemsOK). The carried outage items (db, lease, files, env, timers, next-due, replication) are Owner: landing, Source: seal: the sealer records what it copied and the target proves it arrived equal, so open cannot pass on a target that lacks them. db-tail, build-state and sidecar are Owner: seal (only the source can say). TestOfflineManifestIsCovered pins each id's pair.
States, failures, and what the operator runs
record.move is {kind: "offline", id, from, to, state, sealer, boxes, source{...}, requested_by, requested_at, updated_at}. All fields are additive.
| state | record region | written by | source boxes | sealer | target box | operator next | abort | |---|---|---|---|---|---|---|---| | offline-stopping | from | start door | hold, fence, report | | | seal once every box reported | to offline-reverting, region unchanged | | offline-sealing | from | seal door | held, fenced, reporting | checkpoint, tombstone, final mirror, archive, timers, env pin, seal.json, then CAS offline-sealed | | wait | same | | offline-sealed | from | sealer | same | re-assert tombstone, report | | flip (current reports with zero processes and no layout journal) | same | | offline-landing | to | flip door | rows kept held (KeepParticipant), still reporting | same | mint held; ensure every item; landing.json; CAS offline-landed once every item is OK | open, or abort if an item failed | to offline-reverting and region back to from | | offline-landed | to | target | same | same | keeps ensuring | open (re-checks tombstone and every source report) | same | | offline-open | to | open door | park local state, lift fence, report parked | re-assert tombstone, report | ReleaseHold: row ready, ordinary adoption stages the build, first request promotes the staging; report opened | finish | refused: past this point a failure is fixed forward, or a new move back | | offline-finishing | to | finish door (first call) | rows no longer kept, so reconcile removes them; report gone | once every source box reports parked and gone: lift the tombstone, report lifted | | finish again (the CLI does it) | refused | | offline-reverting | from | abort door | release hold (under WithHostHoldLock), lift fence, report reverted | lift the tombstone if it is this move's, delete both staging prefixes, report reverted | if it landed: DeleteIfHold, park what it extracted, report reverted | finish | already reverting | | cleared | to (done) or from (aborted) | finish door | | | | | |
Which way each failure goes.
- A box step fails (S3 error, a box restarts mid-step): every step is an ensure and re-runs on the next moves pass. The tombstone re-assert is a same-owner rewrite, the mirror resumes from receipts, the archive is rebuilt and both objects overwritten before
seal.json, and the landing re-checks each item from what is on disk. The report says what failed; the state does not move. - A seal item fails for a reason that will not clear (a file the archive refuses, over 256 MiB, a special file):
seal.jsonsays so;abort. - A landing item is a mismatch (listing, digest, env pin names, the channel moved):
landing.jsonsays so;abort. Items that can clear (config render, a slow S3 read) are retried each pass. - Abort before the flip: source rows released, fences lifted, tombstone lifted on owner match, staging deleted. The source's local DB was never purged and equals the replica at
FinalTXID. Lossless. - Abort after the flip (landing or landed): the target never took its lease, so data-uk
instances/<host>/is untouched;DeleteIfHold, park the extracted files and env. The source half completes without the target. Lossless. - The target box dies after the flip:
abort;finishneeds only the source half. When the box returns, reconcile removes its held row (record out of scope, move cleared), and a later landing parks what it left. - A source box is down at
open:openrefuses (its report is not current). Wait or abort. - A source box is down after
open: it never parks,finishwaits, uk serves. No loss, a wait. - After
open: out of scope,abortrefused (MV-O27). An app that cannot boot on uk shows in the operator's check within a minute; fix forward on uk, where the only live copy is. The manual way back is a new offline move (Runbook P3).
The operator surface
Where it runs: console operator doors plus the carlos move offline subcommand. It is the least new code (the move door family, moveOperatorGate, moveGate's class rails, SetRecordMove, the serial bump, carlos move's client all exist), it needs no SSM and no box access (AGENTS.md, "Operate via API and CLI"), and every door runs in-process under httptest in CI. An SSM ops tool would need AWS credentials in tests and could not be gated by CI at all.
Doors, all cliRoute with svcrole.Allow(svcrole.Instance) so a service token meets the operator gate's byte-identical 404, all behind moveOperatorGate:
POST /api/cli/apps/{account}/{app}/instances/{host}/move/offline {"region":"uk"}
POST .../move/offline/seal
POST .../move/offline/flip
POST .../move/offline/open
POST .../move/offline/finish
POST .../move/offline/abort
GET .../move/offline status: the record's move, every box report, the manifest checklist, the next verbCLI (move.go), exit codes as carlos move (0 done or accepted, 3 aborted, 4 waiting for the operator, 5 refused, 1 error):
carlos move offline start --host <host> --to <region>
carlos move offline status --host <host>
carlos move offline seal --host <host>
carlos move offline flip --host <host>
carlos move offline open --host <host>
carlos move offline finish --host <host>
carlos move offline abort --host <host>Every mutating verb waits until its step is done or fails (polling status at 2 s, the same loop carlos move has), then prints one line; finish posts again once the tombstone is lifted. status prints the headline plus one line per manifest item and per box: <id> ok|waiting|failed <detail>, where <detail> is a box diagnostic (not copy, per the copy-review scope rule).
Every user-facing string (copy review before merge)
New rows go in docs/superpowers/plans/2026-10-06-moves-phase1-copy.md under a new heading, ids MV-O1 to MV-O31 (free on 2026-10-20; check again before writing), pending until Paul approves, and into movesCopyIDs. Reused approved templates keep their ids; only their fills change. No em dashes. Door sentences are lower case, as the move door's are.
| id | where | text | |---|---|---| | M-C1 (reused) | move.go | Moving <host> from <from> to <to>. | | MV-O1 | move.go | <host> is offline from now until the move opens or is aborted. | | M-C10 (reused, fill <from>) | move.go | Waiting for <box> in <from> to confirm it has stopped <host>. <host> is offline. | | MV-O2 | move.go | Every box in <from> has stopped <host>. Run carlos move offline seal --host <host> to copy it. | | MV-O3 | move.go | Copying <host> to <to>. | | MV-O4 | move.go | Copied and checked. Run carlos move offline flip --host <host> to start it in <to>. | | M-C6 (reused) | move.go | Stopped in <from>. Starting in <to>. | | MV-O5 | move.go | <to> holds everything <host> needs. Nothing is served yet. Run carlos move offline open --host <host> to serve it from <to>. | | MV-O6 | move.go | <to> could not take <host>: <reason>. Run carlos move offline abort --host <host> to move it back. | | MV-O7 | move.go | <host> is served from <to>. Run carlos move offline finish --host <host> once <from> has cleaned up. | | MV-O10 | move.go | Waiting for <box> to clean up <host>. | | M-C7 (reused) | move.go | Moved. <host> is served from <to>. It was offline for <seconds> seconds. | | MV-O8 | move.go | Moving <host> back to <from>. | | M-C9 (reused) | move.go | <host> is back in <from> and serving. Nothing was lost. | | MV-O9 | move.go | The move could not finish a step: <reason>. Run carlos move offline status --host <host> for each item. | | MV-O20 | apiinstancemoveoffline.go | offline moves are not available while host claims or Phase 1 moves are on | | MV-O21 | apiinstancemoveoffline.go | <box> cannot take part in an offline move. it runs an older release or has host claims on | | MV-O22 | apiinstancemoveoffline.go | <region> has <n> boxes. an offline move needs exactly one box in the region it moves to | | MV-O23 | apiinstancemoveoffline.go | <box> does not hold this deployment's secrets key | | MV-O24 | apiinstancemoveoffline.go | <host> has no offline move in flight | | MV-O25 | apiinstancemoveoffline.go | this move is <state>. run carlos move offline status --host <host> to see what it waits for | | MV-O26 | apiinstancemoveoffline.go | <host> keeps local data on more than one box: <boxes>. remove the stale copy first | | MV-O27 | apiinstancemoveoffline.go | <host> is already served from <to>. move it back with a new move | | MV-O28 | apiinstancemoveoffline.go | <box> keeps a sidecar file for <host>. instances with sidecars cannot move yet | | MV-O29 | apiinstancemoveoffline.go | <host> has a host claim. offline moves are only for hosts without one | | MV-O30 | apiinstancemoveoffline.go | <other> is moving offline. move one instance at a time | | MV-O31 | apiinstancemoveoffline.go | <box> still runs a process for <host>. wait for it to stop, then flip |
The docs/site/cli.md paragraph under ### carlos move (Task 7) is copy too. The moveGate class refusals, invalid region, the edges-incomplete 409 and the staging-bucket refusal are existing approved sentences and are reused unchanged. <from> is the home's configured name (homeRegionOut); with none configured, the word home.
Global Constraints
- Ruling 2026-10-20, from platform-1e as claim owner. It supersedes the claim and guard wording below where they conflict. 1. Instance claim state. The start door refuses (MV-O29) on any of: -
control/hosts/<host>.json; - a domains claim carryingacq; - a record carryingacquisition; - anapps/<account>/<app>/admission.json.
It refuses (MV-O20) on a control/hosts-backfill.json whose phase is collecting or adopting. A plan at done is allowed, with any number of hosts. 2. A console on the claim path refuses (MV-O20). This is unchanged. 3. FINAL (platform-1e, from code at 96db0efe). The start door refuses (MV-O21) unless every claims-on box in the source or target region meets all three conditions: - (a) It publishes NO row for the host. The evidence is that box's published edges file and box report (every host it serves), never the operator's word. - (b) It reports /var/lib/carlos-claims/authority/exec/census.json present, so root's legacy restart fallback (legacyrestart.go:21-27) can't start anything. - (c) Its pool set doesn't match the record's placement.
If any of this evidence is missing from a box's published state, the door refuses. It fails closed, and the move adds no new box report field just to satisfy this. The reasons: - On a claims-on box, wakes go through doClaimWake and refuse a claimless host (activator.go:2085, claimwake.go:182-201), and reconcile mints only through claimInstall (provision/claimassembly.go:20-38). - The exceptions are an existing non-hibernating row the edge proxies without a check (edge.go:851), root's legacy fallback when census.json is missing, and the domains apex mint (domains/reconcile.go:312).
Boxes that publish the source placement, and the target box, must also carry offline-move-1. The interim wording follows for reference. 3a. (interim, superseded by 3) The region-wide capability check narrows. The start door refuses (MV-O21) when the target box lacks offline-move-1, or when any box that publishes the source placement lacks it. Those are the boxes whose pools match the record's placement, the ones that could serve or wake the host. It also refuses when the record's placement matches the pools of any box that runs claims-on. Boxes in the region that cannot hold the host are not consulted. This point may be confirmed or tightened later. 4. An armed or enforced guard is NOT a refusal. The capability depends only on hibernation being configured and CARLOS_HOST_CLAIM and CARLOS_MOVES_PHASE1 being unset. The guard mode is not read for the capability. The target's build stages through ordinary adoption, so root's existing floor check applies on a guarded target box, and the move adds no other stage, adopt or restart path.
- Switches stay off, and claims are refused from evidence. Every offline door refuses with MV-O20 when the console runs with
HostClaimorMovesEnabled. A box publishes theoffline-move-1capability, and runs any offline step, only when hibernation is configured,CARLOS_HOST_CLAIMandCARLOS_MOVES_PHASE1are unset, and the guard policy (/opt/carlos/system/guard/policy.json,internal/hostfloor/floor.go:33,:141) is absent or itsModeis neitherarmednorenforced; otherwise it logs one notice per host and does nothing. The start door refuses (MV-O21) when any box in the source or target region lacks the capability, refuses MV-O29 whencontrol/hosts/<host>.jsonexists, and refuses MV-O20 whencontrol/hosts-backfill.json(hostops.BackfillKey) exists or the console is on the claim path at all (HostClaim, which includes theclaimMoveUnsupportedMsgcase,apiinstancemove.go:575-578). Nothing here readsCONSOLE_APP_REGIONS,CARLOS_SWAPorCARLOS_REGION_HA, and nothing takes the operator's word for any of it. - No activator change, no root change. The target is never started before
open. Root's stage pass is used only as it already is for any ready row. - The hold is a function of the record.
Hold(rec)reads onlyrecand the box's placement (throughpark.PlacementMatches, asmovesNamesBoxdoes atmovespass.go:169-185). The view's snapshot is forKeepParticipantonly, refreshed at agent startup before the reconcile loops, fail-closed only when that listing failed. - Only the record's region changes, through provision's writer. Every record write is a
SetRecordMovetransition (updateRecord, which advancesrecord_seqonce present). Across a whole move the record differs only inregion,moveand, atfinish,last_move, plusrecord_seqandmove.updated_at;acquisition, the incarnation fields, the instance key and the domains claim are never read for writing. No code in this planPuts a record. - The write set is closed. The tool writes only: the record (as above);
control/offline-moves/;control/serial(the existing bump);control/config-ack/(throughRenderHeld);control/edges/(the existing writer, one capability string); in the source data bucketinstances/<host>/lock.json(the tombstone), the replica throughcfg.Checkpoint,instances/<host>/build-state.json(throughCarryRequest) andmove-receipts/<host>.json; in the target data bucketmoves/<host>/andmoves/@offline/<host>/<id>/, and a stalemoved:*lock.json. It never writes or deletescontrol/hosts/,control/host-census/,control/host-actors.json, any admission, or any domains claim, and readscontrol/hosts/<host>.jsonandcontrol/hosts-backfill.jsononly to refuse. - One host at a time.
startrefuses while any other record carries an offline move (MV-O30). - Kinds never cross. No legacy transition (
MoveAbort,MoveMarkCutting,MoveComplete,MoveAbortComplete,MovePromoteReady) and no Phase 1 guarded transition acceptskind: offline, and no offline transition accepts a move without it. The box's moves pass routes an offline record before the Phase 1 skip atmovespass.go:991and before thelostHeregate. - Every box step is an ensure. Each step checks the state on disk and in the bucket, does only what is missing, and reports what it found. A re-run after a crash or a blip converges, never churns, never fails on its own earlier work.
- Wire shapes are additive.
MoveState.Kind,Sealer,Boxes;MoveSource.Release,EnvDigest; the reports; the edges capability. A record written before this parses byte-identically (test). Reports live undercontrol/offline-moves/, never under the create-only or no-delete prefixes (control/boxes,control/box-volumes,control/box-enrollment,control/box-fences). - The tombstone lives 7 days at most. Written at seal, re-asserted every sealer pass (same owner,
If-Match), lifted atfinishand on abort. - Rename, never delete, local state. Parking is
os.Renameinto<DataDir>/.offline-moves/<host>/<id>/<role>/(a numbered suffix when the directory exists), and.offline-movesjoins the known names ininternal/layout/extras.go:329. Bucket objects are deleted only in the staging prefixes this move wrote. - No infra change is required. IE's role already has
sibling_staging_arns = [data_uk](carloku-infrastructureregion-ie.tf:42, read at 7c99ff8, a local checkout that may lag): Put and Delete onmoves/*, List with themoves/*prefix condition, no Get, which the move never needs (server-side copy, List, Put, Delete).moves/@offline/...is inside it. A move back from the UK needs UK's grant (Runbook P0). - Files are owned by one task at a time. Each task's Files block lists every file it may touch; parallel tasks share none. Where a later task takes over a file an earlier one created, the block says so.
- Production-wired witnesses. Box steps are tested through
newMovesWorkerandreconcileSerializedWith(the functions the agent calls), againstleasetest.NewS3Server, real registries and real litestream replicas. A fake is allowed only where the plan names it (the activator's fake backend, for the fence only). - Gate. While iterating, targeted runs only, niced and serial, for example
go test ./internal/host -run 'Offline' -count=1,go test ./internal/lease -run 'Tombstone' -count=1. Never the wholeinternal/hostor console suite locally (CI runsmake test). Once before handing off:go build ./...,go vet ./...,gofmt -l ., the copy gate (go test . -run 'Moves|Copy|Docsite' -count=1), andGOOS=darwin,GOOS=windows,GOOS=linux GOARCH=arm64go build ./.... Default Go build cache.GOFLAGS=-buildvcs=falseif the worktree's VCS metadata is unreadable from the sandbox. Scratch under$TMPDIR. - Copy. Every string above is a
pendingrow in the commit that writes it. Paul reviews them before merge; merge waits for that. - Merge and roll. Push to amadan, describe, squash, re-merge
main, green CI on the tip,amadan branch merge carlos/platform moves-offline -expect <tip> -message "..."with full arguments. Nothing is released or rolled without Paul's go.
Review Focus
- B1, the pure hold.
Holdmust mint the landing row held on a fresh target box with no view built; the pinning test is a fresh worker, no refresh, a record atoffline-landingnaming this region, and reconcile mints the row heldlanding.KeepParticipantalone may fail closed, and only when the startup listing failed. - The linearization point. The tombstone must be taken after the sealer's checkpoint released its lease and before the final position is read; every source report's local position must be at or below that position; nothing between the holds and the tombstone can wake the host on a reporting box (held row and fence).
Tombstonemust never replace a live holder's lease;ReassertandLiftmust act only on ownermoved:<id>with the ETag they read. - B2, ensures. Re-running the landing at any point (after any item, after the CAS, after a box restart) must neither park and re-extract a correct files directory nor fail on its own earlier extraction. Each item reports from what is there.
- The dispatcher. An offline record reaches
offlineRoundwith Phase 1 off, ahead ofmovespass.go:991andlostHere; a Phase-1-shaped record with Phase 1 off still logs and skips. - Dormant behaviour is unchanged. With no offline record anywhere: the hold hook answers false,
keepanswers false after the startup refresh, removals happen on the first pass as today, the edges file gains one capability string, and nothing else differs. openre-checks the stop. The door requires the sealer's report from this pass withLockStateheld bymoved:<id>, every source report held, fenced and quiesced, and every landing item OK.- Claims from evidence. The capability is withheld for every claim-shaped box state (env, guard mode, Phase 1), and the door refuses a region that contains such a box.
- The record and the write set. The only record changes across a move are
region,move,last_move,record_seqandmove.updated_at(T1's diff test), and T8's recording store shows no write outside the closed write set, in particular nothing undercontrol/hosts/,control/host-census/,control/host-actors.json, admissions or domains claims. - The source is retired, and the report says how. Every path in the
source-retirerow appears in theparkedreport withparked,absentorleftand a reason; nothing is deleted. - The manifest is the checklist.
TestOfflineManifestIsCoveredmust fail if a manifest row is added with no owner, or a report carries an item the manifest does not list.
---
File structure
| path | responsibility | task | |---|---|---| | internal/provision/offline_move.go (new), internal/provision/provision.go (fields only) | record fields, offline states, transitions | T1 | | internal/moves/offline.go (new) | OfflineManifest, report shapes, keys, owner and park-path helpers | T1 | | docs/superpowers/plans/2026-10-06-moves-phase1-copy.md, approved_moves_copy_test.go | copy rows and ids | T1 | | internal/lease/tombstone.go (new) | Tombstone, Reassert, LiftTombstone | T2 | | internal/host/movesoffline.go (new) | the hold hook, the keep view, the dispatcher, the source steps, parking | T3 | | internal/host/movespass.go, internal/host/agent.go, internal/host/edgesfile.go, internal/layout/extras.go | wiring: dispatch point, newMovesWorker, every moveCfg field, reconcileSerializedWith, startup refresh, capability, known name | T3 | | internal/host/movesoffline_seal.go | the sealer and its revert (T3 creates a stub; T4 owns it) | T4 | | internal/moves/archive/archive.go, internal/layout/plan.go | BuildFiltered, NestedPlatformNames | T4 | | internal/host/movesoffline_land.go | the landing, open on the target, the target's revert (T3 creates a stub; T5 owns it) | T5 | | internal/console/apiinstancemove_offline.go (new), internal/console/authcli.go | the doors and the status view | T6 | | move.go, move_test.go, docs/site/cli.md | the CLI subcommand and its docs paragraph | T7 | | internal/host/movesoffline_e2e_test.go (new) | the two-box witness | T8 |
Order, parallelism and models
T1 ──┬── T2 ──────────────┐
├── T3 ──┬── T4 ─────┤
│ └── T5 ─────┤
└── T6 ──── T7 ──────┴── T8T1 first: its interfaces are the contract. Then T2, T3 and T6 in parallel (no shared files). Then T4, T5 and T7 in parallel: T4 needs T1, T2 and T3; T5 needs T1 and T3 (and T2 for lifting a stale tombstone); T7 needs T6. T4 and T5 touch only the files their blocks list, because T3 adds every moveCfg field and both stub files up front. T8 last. Critical path T1, T3, T5, T8: about 7 h plus reviews.
| task | model | size | |---|---|---| | T1 record shape, transitions, manifest, copy rows | Sonnet | 1.5 h | | T2 tombstone | Sonnet | 1 h | | T3 hold, keep, capability, dispatcher, source steps, wiring | Opus | 2 h | | T4 sealer | Opus | 2 h | | T5 landing, open, target revert | Opus | 1.5 h | | T6 console doors and status | Sonnet | 1.5 h | | T7 CLI, docs paragraph | Sonnet | 1 h | | T8 two-box witness | Opus | 2 h |
Opus reviews T3, T4 and T5 as they land; one whole-branch review before merge by whichever reviewer Paul's budget rule picks.
---
Task 1: Record shape, transitions, the manifest and the copy rows (Sonnet)
Files: Create internal/provision/offline_move.go, internal/provision/offline_move_test.go, internal/moves/offline.go, internal/moves/offline_test.go. Modify internal/provision/provision.go (the new struct fields only), docs/superpowers/plans/2026-10-06-moves-phase1-copy.md, approved_moves_copy_test.go. No other task touches these files.
Interfaces:
// provision
const MoveKindOffline = "offline"
const (
OfflineStopping = "offline-stopping"
OfflineSealing = "offline-sealing"
OfflineSealed = "offline-sealed"
OfflineLanding = "offline-landing"
OfflineLanded = "offline-landed"
OfflineOpen = "offline-open"
OfflineFinishing = "offline-finishing"
OfflineReverting = "offline-reverting"
)
// MoveState gains (additive, omitempty):
// Kind string `json:"kind,omitempty"`; Sealer string `json:"sealer,omitempty"`; Boxes []string `json:"boxes,omitempty"`
// MoveSource gains: Release string `json:"release,omitempty"`; EnvDigest string `json:"env_digest,omitempty"`
func (m *MoveState) Offline() bool
func OfflineStart(id, from, to, by string, now time.Time) func(*Record) error
func OfflineSeal(id, sealer string, boxes []string) func(*Record) error
func OfflineSealedBy(id, sealer string, src MoveSource) func(*Record) error
func OfflineFlip(id string) func(*Record) error // Region = To
func OfflineLandedBy(id, to string) func(*Record) error // guarded by record region == to
func OfflineOpenTransition(id string, now time.Time) func(*Record) error // the state constant owns the name OfflineOpen
func OfflineFinishingTransition(id string) func(*Record) error // from open; the state constant owns OfflineFinishing
func OfflineAbort(id string) func(*Record) error // landing/landed: Region = From
func OfflineFinish(id string, now time.Time) func(*Record) error // from finishing (done) or reverting (aborted); LastMove set
// moves
type OfflineItem struct{ ID, Item, Disposition, Owner, Source string } // Owner: door|seal|landing|none; Source: seal or ""
var OfflineManifest []OfflineItem
type OfflineReport struct {
V int; Op, Host, Box, Role, Phase string // role: source|sealer|target; phase: stopped|sealed|landed|opened|parked|gone|lifted|reverted|failed
Items map[string]Check; At time.Time
Held, Fenced, Quiesced, LocalDB, Files, Env, Sidecar, RowPresent bool; LocalTXID uint64
// LocalDB: the DB file or the activator's litestream meta dir is present (Fable S8).
FinalTXID uint64; Staged map[string]StagedObject; FilesPlain, FilesSealed string; FilesBytes int64; FilesEntries int
EnvSHA256, KeySHA256, EnvDigest, Release, TimersDigest, Replication string; NextDue int64
TombstoneOwner string; TombstoneUntil time.Time
Procs int; LayoutJournal bool
Retired []RetiredPath // {Path, Action, Reason}; the parked report's inventory (§9.4)
}
func OfflineReportKey(host, op, role, box string) string // control/offline-moves/<host>/<op>/<role>/<box>.json
func OfflineArchivePrefix(host, op string) string // moves/@offline/<host>/<op>/
func TombstoneOwner(op string) string // "moved:" + op
const TombstoneTTL = 7 * 24 * time.Hour
func OfflineParkDir(dataDir, host, op, role string) stringEach transition checks rec.Move.Offline(), the id, and its one legal prior state, and returns ErrMoveConflict otherwise. OfflineStart carries MoveRequest's refusals (seed, open run, a move in flight, same region, a replica) and stamps Kind, ID, From, To, State, RequestedBy, RequestedAt. OfflineAbort is legal from stopping, sealing, sealed, landing and landed; from landing and landed it also sets Region = From; from open and finishing it refuses; from reverting it is a no-op success.
- [ ] Failing tests: the full legal path; every illegal pair refused; wrong id refused; across the full path, a record carrying
acquisition, incarnation fields, an instance key reference andrecord_seqdiffers at the end only inregion,move,last_move,record_seq(advanced) andmove.updated_at(§9.2); abort before and after the flip (region); finish from both; a record JSON written before this change round-trips byte-identically; each legacy and Phase 1 transition refuses an offline move (MoveAbort,MoveMarkCutting,MoveComplete,MoveAbortComplete,MovePromoteReady, one guarded Phase 1 transition);TestOfflineManifestIsCovered; key helpers reject a host that failsValidHostShape. - [ ] Implement;
go test ./internal/provision ./internal/moves -run 'Offline|Move' -count=1. - [ ] Copy rows MV-O1 to MV-O10 and MV-O20 to MV-O31 (and the docs paragraph's row) as
pending, ids inmovesCopyIDs. Tasks 6 and 7 wire the strings; if either has to change a row's text, it asks the coordinator, who changes the row after that task (never in parallel with it). - [ ] Commit:
offline-move: record shape, transitions, manifest and copy rows.
Task 2: The tombstone lease (Sonnet)
Files: Create internal/lease/tombstone.go, internal/lease/tombstone_test.go. Nothing else.
Interfaces: each is lss3.Leaser{Owner: owner, TTL: ttl} over the client AcquireTTL builds (IfNoneMatch: * on create, IfMatch on replace), with no renewer.
func Tombstone(ctx context.Context, bucket, location, region, owner string, ttl time.Duration) error
// absent or expired: create {owner, now+ttl}; held by owner: nil (the caller re-asserts); held by anyone else: *HeldError.
func Reassert(ctx context.Context, bucket, location, region, owner string, ttl time.Duration) error
// held by owner: rewrite with If-Match on the read ETag, expiry now+ttl; absent or expired: Tombstone; held by anyone else: *HeldError.
func LiftTombstone(ctx context.Context, bucket, location, region string, mine func(owner string) bool) (lifted bool, err error)
// absent: (false, nil); owner not mine: (false, *HeldError); mine: delete with If-Match on the read ETag.- [ ] Failing tests against
leasetest.NewS3Server: a tombstone blockslease.AcquirewithHeldError{Owner: "moved:mv_..."}; a heldlease.AcquireblocksTombstone; an expired lease is replaced; a repeat by the same owner is a no-op;Reassertmoves the expiry and refuses another owner;LiftTombstonerefuses another owner, lifts its own, and a followingAcquiresucceeds; a concurrent rewrite between read and write makesReassertandLiftTombstonefail rather than overwrite;MirrorPrefixstill skips the tombstone'slock.json. - [ ] Implement;
go test ./internal/lease -run Tombstone -count=1. - [ ] Commit:
offline-move: tombstone lease.
Task 3: Hold, keep, capability, dispatcher and the source boxes' steps (Opus)
Files: Create internal/host/movesoffline.go, internal/host/movesoffline_test.go, and two stubs that later tasks take over: internal/host/movesoffline_seal.go (owned by T4 from then on) and internal/host/movesoffline_land.go (owned by T5 from then on), each holding only the method signatures below with a body that returns nil. Create internal/cgroup/hostprocs.go, internal/cgroup/hostprocs_linux.go, internal/cgroup/hostprocs_other.go (//go:build !linux stub answering "unknown", for the cross-compile gate) and their test. Modify internal/host/movespass.go, internal/host/agent.go, internal/host/edgesfile.go, internal/layout/extras.go. T4 and T5 must not touch any of these except their own stub file.
Interfaces:
func offlineHold(box moveCfg) func(rec provision.Record) (status, op string, ok bool)
// pure: rec.Move.Offline(); source placement and state stopping|sealing|sealed -> ("moving", id);
// target placement, rec.Region == To, state landing|landed -> ("landing", id); everything else false.
// Placements compared with park.PlacementMatches(cfg.ParkHomeRegion, code, here), as movesNamesBox.
type offlineView struct{ /* box, region; atomic snapshot host -> MoveState, built flag, failed flag */ }
func (v *offlineView) refresh(recs []provision.Record)
func (v *offlineView) keep(host string) bool
// an offline move naming this box's region as from (state before finishing) or to (state before cleared): true;
// built and none: false; startup listing failed: true.
func refreshOfflineView(ctx context.Context, s release.Store, v *offlineView) // at agent start, before the reconcile loops
func withOfflineHolds(cfg provision.ReconcileConfig, hold func(provision.Record) (string, string, bool), v *offlineView) provision.ReconcileConfig
// only when cfg.Hold and cfg.KeepParticipant are nil (Phase 1's view absent)
func reconcileSerializedWith(ctx context.Context, s release.Store, reg *registry.Registry, cfg provision.ReconcileConfig, w *movesWorker) error
func newMovesWorker(cfg moveCfg) *movesWorker // armMovesPass calls it; tests build one per box
func (w *movesWorker) offlineRound(ctx context.Context, s release.Store, rec provision.Record) error
func offlineCapable(getenv func(string) string, hibernation bool, guardMode func() (string, error)) bool
const capOfflineMove1 = "offline-move-1"
// cgroup: processes left in the host's exec cgroup and, when unit is set, the unit's cgroup;
// an absent cgroup is 0; non-linux answers ErrUnknown, which a report records as not stopped.
func HostProcs(root, host, unit string) (int, error)
// stubs, filled by T4 and T5:
func (w *movesWorker) offlineSealerStep(ctx context.Context, s release.Store, rec provision.Record) error
func (w *movesWorker) offlineTargetStep(ctx context.Context, s release.Store, rec provision.Record) errormoveCfg gains, all set by armMovesPass and settable by tests: Box (the edges box label, always), FenceAs, LiftFenceAs, ReleaseHold, DeleteIfHold, Registry (for TimersFor, ReplaceTimers, SetNextDue, SetReplication, Get), LaunchLayers (the exec backend's), SecretsKey (raw CARLOS_SECRETS_KEY), ConfigPriv and ConfigPub (the deployment config keypair), RenderHeld (closure over appconfig.RenderHeld), LiveVersion, ChannelVersion (the channel's build, as adoption resolves it), Offline *offlineView, OfflineCapable bool, Now. T4 and T5 read only these.
Dispatch: movesPass refreshes the view from its record list and, as the first thing in the record loop, ahead of the Phase 1 skip at movespass.go:991 and the lostHere gate at :1003, routes rec.Move.Offline() to offlineRound and continues. offlineRound returns at once, with one notice per host, unless OfflineCapable. It runs the source step when this box's placement is move.from, offlineSealerStep when cfg.Box == move.sealer, and offlineTargetStep when this box's placement is move.to.
Source steps (this box's placement equals move.from; "" for home), each an ensure:
- stopping through landed:
FenceAs(host, moveFenceGeneration, "offline:"+id)with a 2 s deadline (ErrStopUnprovenis fine); read the row; rewrite the source report withAton every pass (theopendoor needs it current):Held(row absent or held by id),Fenced,Quiesced,Procs(cgroup.HostProcs),LayoutJournal(layout.JournalPresent),LocalDB(DB file or meta dir),LocalTXID(localLTXPositionon the row's DBPath, flat or nested),Files,Env,Sidecar,RowPresent. - open: with the fence held and the row held, rename into
OfflineParkDir(..., "source")every path of the manifest'ssource-retirerow that exists and that the agent can write: the local DB, WAL, SHM, both litestream meta dirs,/data/<host>/,<host>.env,<host>.config.env,/data/programs/<host>.json, and for a nested row the contents of/data/.instances/<iid>(the directory and its/data/instances/<iid>view stay, empty). Lift the fence. ReportparkedwithRetired, one entry per path of that row:{path, action: parked|absent|left, reason};/opt/carlos/live/<host>and any mount areleft, "root-owned". Nothing is deleted. - finishing:
keepis false for the source, so reconcile removes the row; reportgoneonceRowPresentis false. - reverting (every source box; the sealer's extra part is T4's):
ReleaseHold(host, id)underprovision.WithHostHoldLock;LiftFenceAs; reportreverted.
Agent wiring: reconcileSerialized becomes reconcileSerializedWith(..., armedMovesWorker.Load()); refreshOfflineView runs before the reconcile loops start, next to refreshMovesHolds. Edges file: edgesFileCapabilities adds offline-move-1 when offlineCapable (hibernation configured; CARLOS_HOST_CLAIM, CARLOS_MOVES_PHASE1 unset; guard policy absent or not armed/enforced, read through the existing hostfloor reader). layout/extras.go: .offline-moves is a known name.
- [ ] Failing tests through
newMovesWorkerandreconcileSerializedWith, two source boxes and one target over oneDirStore: B1: a fresh target worker with no refresh, a record atoffline-landingnaming its region, reconcile mints the row heldlanding;startholds the row on both source boxes and neither lists it in its edges file; a box whose row already exists keeps its other columns; an agent restart (a new worker over the same registry) re-fences and re-reports; a box with only a meta dir reportsLocalDB; after the flip the source rows stay (keep); inoffline-revertingthe rows are released and serving and the next reconcile does not re-hold them; inoffline-openthe local state is renamed into the park dir, nothing is deleted, and theparkedreport lists everysource-retirepath with its action (flat and nested fixtures); a process left in the host's cgroup (asleepstarted into a test cgroup where the kernel allows, elseHostProcsover a fixture cgroupfs root) shows asProcs > 0; a present layout journal shows in the report; inoffline-finishingthe rows go andgoneis reported; the dispatcher reaches an offline record with Phase 1 off and still skips a Phase-1-shaped record; a box withCARLOS_HOST_CLAIMset, or an armed guard, publishes no capability and does nothing; with no offline record,keepis false after the startup refresh and the first reconcile removes a retired row as today. - [ ] Implement;
go test ./internal/host -run 'Offline' -count=1;go test ./internal/layout -count=1. - [ ] Commit:
offline-move: hold, keep, capability, dispatcher and source steps.
Task 4: The sealer (Opus)
Files: Take over internal/host/movesoffline_seal.go. Create internal/host/movesoffline_seal_test.go. Modify internal/moves/archive/archive.go and its test, internal/layout/plan.go and its test. Nothing in T3's or T5's files.
Interfaces:
func BuildFiltered(filesDir, envPath string, skip func(rel string) bool, w io.Writer) (plain string, entries []Entry, bytes int64, err error) // Build = BuildFiltered(nil)
func NestedPlatformNames(host string) map[string]bool // from nestedFiles' names plus <db>.gen, .files-gen, the adoption record
func (w *movesWorker) offlineSealerStep(ctx context.Context, s release.Store, rec provision.Record) errorIn offline-sealing, every step an ensure:
- Refuse unless its own source report says held, fenced, quiesced, zero processes and no layout journal, and no source report shows a sidecar (the legacy cut's order: fence, quiesce, journal absent, then the checkpoint).
- If its local TXID is above the replica's newest TXID,
cfg.Checkpoint(lease, sync, release). lease.Tombstone(OwnBucket, HibernatePrefix(OwnPrefix, host), OwnBucketRegion, TombstoneOwner(id), TombstoneTTL).- List the replica, take
FinalTXID(newestLTX), and refuse if any source report'sLocalTXIDis above it. CarryRequest, then a finalMirrorPrefixround into the target'smoves/<host>/, refusingDeferred > 0; list the staged prefix as the seal'sStaged.- Build and seal the archive (
BuildFilteredover the flat files dir or the nested instance dir withNestedPlatformNames;archive.SealunderConfigPub, binding the record's account and app);Putit and its manifest underOfflineArchivePrefix. Size and file refusals become failed items, never a partial archive. - Timers:
TimersFor,ActiveTimers,TimersSnapshot.Encode,Putbeside the archive,TimersDigest. - Env: sha256 of
<host>.envand of its effective instance key; env pin:LaunchLayerson its own row,ReportLaunchkeyed byReportKeyFromSecrets(SecretsKey),PortableDigest. ReleasefromLiveVersion,ReplicationandNextDuefrom the row.- Write
seal.json(rolesealer), then CASOfflineSealedBywith aMoveSourcecarryingFinalTXID,Files,TimersSHA256,Release,EnvDigest,Replication,NextDue, andEvidence= the report key and its sha256.
From offline-sealed through offline-open and in offline-finishing until it lifts: Reassert the tombstone, read lease.LockState, rewrite the sealer report with TombstoneOwner, TombstoneUntil and At. In offline-finishing, once every source report is parked and gone: LiftTombstone(mine: owner == TombstoneOwner(id)), clearMoveReceipts, report lifted.
In offline-reverting: LiftTombstone (this move's only), DeletePrefix on the target's moves/<host>/ and OfflineArchivePrefix, clearMoveReceipts, report reverted.
- [ ] Failing tests against
leasetest.NewS3Serverwith bucketsdata-ieanddata-ukand a real litestream replica written bylease.ReplicateWithLease: the full seal produces a staged copy whose listing matches, a tombstone, an archive thatOpen,VerifyandExtractback to the samePlainDigest, andoffline-sealed; a live writer's lease makes step 3 fail and the state stays sealing; a source report above the final TXID refuses; a nested fixture produced by the real layout pass archives without the database, env or config files and extracts flat; a file over the cap and a FIFO are failed items; a replay after a crash at each step finishes with one tombstone and identical staged objects; later passes re-assert the expiry; finishing lifts only after every source box is parked and gone; the revert lifts only this move's tombstone and deletes only the two prefixes. - [ ] Implement;
go test ./internal/host -run 'OfflineSeal' -count=1;go test ./internal/moves/archive ./internal/layout -count=1. - [ ] Commit:
offline-move: the sealer.
Task 5: The landing, open and the target's revert (Opus)
Files: Take over internal/host/movesoffline_land.go. Create internal/host/movesoffline_land_test.go. Nothing else.
Interfaces:
func (w *movesWorker) offlineTargetStep(ctx context.Context, s release.Store, rec provision.Record) errorIn offline-landing and offline-landed, each item an ensure judged from what is on disk and in the bucket, each a Check in landing.json:
row: the reconciled row exists, heldlandingby the move id. Wait otherwise.- Read
seal.jsonand verify its sha256 againstmove.source.evidence. warm-cacheandkey-adoption: no local DB, meta dir or adoption record for the host; any found is renamed intoOfflineParkDir(..., "target-before")(numbered suffix if it exists).lease: the target bucket'slock.jsonfor the host is absent or not amoved:*owner; amoved:*owner other than this move is lifted withLiftTombstone.db: listmoves/<host>/; it must equal the seal'sStagedexactly (nothing consumes it beforeopen);lease.VerifyReplicaon that prefix into a scratch directory underOfflineParkDir(..., "verify"), removed after; newest TXID equalsFinalTXID. Verified once per seal evidence and remembered in the report; a re-run re-lists and re-verifies only if the listing changed.files: OK when<DataDir>/<host>/exists and itsPlainDigestequals the seal's. Absent: fetch,archive.OpenwithConfigPriv,Verify,Extract. Present and different: rename totarget-before-<n>and extract.env: OK when<DataDir>/<host>.envexists with the seal's sha256 and key sha256; else asfiles(the archive carries it).config:RenderHeld(row, id, box); OK when the file exists and the ack names this op.env-pin:LaunchLayerson the held row,PortableDigestequalsmove.source.env_digest; on a mismatch the detail isCompareEnv's names.timers,next-due,replication: OK when the row's values equal the seal's; elseReplaceTimersfrom the snapshot (digest checked),SetNextDue,SetReplication, then re-read.release:ChannelVersionequalsmove.source.release.- Rewrite
landing.jsonevery pass; when every item is OK and the state is landing, CASOfflineLandedBy(id, to).
In offline-open: ReleaseHold(host, id) under provision.WithHostHoldLock (the row becomes ready; ordinary adoption stages the build through root's existing stage pass; the first request promotes moves/<host>/), delete OfflineArchivePrefix in its own bucket, report opened. In offline-reverting when this box holds the row: DeleteIfHold(host, id), rename the extracted files, env and config into OfflineParkDir(..., "target"), report reverted.
- [ ] Failing tests over
leasetest.NewS3Serverwith a seal produced by T4's step: the full landing reachesoffline-landed; B2: a second, third and post-CAS pass changes nothing on disk and reports the same items, and a pass after a crash between extraction and the report finishes without parking; every mismatch (an extra staged object, a changed archive, a changed bundle, a promoted channel) fails its item; a stale local DB and a stalemoved:*tombstone of another move are parked and lifted; nothing ever takes the target's lease beforeopen(the bucket'slock.jsonstays absent);openmakes the row ready and listed in the edges file and a followingExecBackend.Restorepromotes and passesCheckDB; the revert parks and deletes the row and leavesdata-ukinstances/<host>/untouched. - [ ] Implement;
go test ./internal/host -run 'OfflineLand' -count=1. - [ ] Commit:
offline-move: landing, open and the target's revert.
Task 6: The console doors and the status view (Sonnet)
Files: Create internal/console/apiinstancemove_offline.go, internal/console/apiinstancemove_offline_test.go. Modify internal/console/authcli.go (routes only). Nothing else.
Each door is moveOperatorGate, then MV-O20 (s.cfg.HostClaim || s.cfg.MovesEnabled), then its own checks, then one SetRecordMove transition, then bumpSerial, answering 202 with the status view. Read the edges files through internal/edgesfile's snapshot; an incomplete read is the existing 409.
- start:
moveGate's class rails; replica, replicas, steering and unit refusals;control/hosts/<host>.jsonpresent (MV-O29);control/hosts-backfill.jsonpresent (MV-O20); any other record with an offline move in flight (MV-O30, fromprovision.ListAllRecords); body{"region"}throughhomeRegionIn,ValidRegion, not the current region; the manifest'sregionsallow-list (moveRegionAllowed); source boxes are the edges files whose placement isfrom(home: placementless, not fleet) with hibernation configured; target boxes likewise forto; exactly one target box (MV-O22); every source and target box publishesoffline-move-1(MV-O21; the capability is the box's own witness that host claims, the guard and Phase 1 are off); the target'ssecrets_key_fpequalsappconfig.PublicFingerprint(SecretsPub)(MV-O23); the target publishes a data bucket (the existing staging refusal); the target does not list the host. TransitionOfflineStartwithmoves.NewMoveID. - seal: state stopping; for every source box: a report for this id with
Held,Fenced,Quiesced, and its edges file not listing the host, else M-C10 naming the box (409);Sidecaranywhere is MV-O28; more than one box withLocalDB,FilesorEnvis MV-O26; sealer = that box, else the lexically first. TransitionOfflineSeal(id, sealer, boxes). - flip: state sealed;
seal.jsonpresent, its sha256 equal tosource.evidence, every seal-owned item OK; every source report written within the last two minutes withProcs == 0and no layout journal, else MV-O31 naming the box. TransitionOfflineFlip. - open (S4): state landed;
landing.jsonevery landing-owned item OK; the sealer report written within the last two minutes withTombstoneOwner == moved:<id>and an unexpiredTombstoneUntil; every source box's report written within the last two minutes withHeld,Fenced,Quiesced. TransitionOfflineOpen. - finish: open: the target's
openedreport and every source box'sparked, thenOfflineFinishing; finishing: the sealer'sliftedand every source box'sgone, thenOfflineFinish; reverting: the sealer's and every source box'sreverted, thenOfflineFinish. Otherwise MV-O25. - abort:
OfflineAbort; open and finishing are MV-O27. - status (GET): the move, every report, the manifest with each item's latest check, and the next legal verb.
- [ ] Failing tests with an in-process server, an operator CLI token and a
DirStore: every refusal above with its exact sentence, MV-O21 for a box without the capability in either region; a member's and a service token's 404 byte-identical to the instance GET's; each door's happy path writes the transition and bumps the serial;openrefuses a stale sealer report, a missing tombstone owner and a source report that lostFenced;fliprefuses a report with a process left (MV-O31); MV-O20 withHostClaim(with and without a claim authority, theclaimMoveUnsupportedMsgcase), withMovesEnabled, and withcontrol/hosts-backfill.jsonpresent; MV-O29 withcontrol/hosts/<host>.jsonpresent; MV-O30 with a second offline move in flight; no door writes anything on a refusal. - [ ] Implement;
go test ./internal/console -run 'Offline' -count=1. - [ ] Commit:
offline-move: console doors and status.
Task 7: The CLI and the docs paragraph (Sonnet)
Files: Modify move.go, move_test.go, docs/site/cli.md. Nothing else (the copy record and approved_moves_copy_test.go are T1's; a text change goes through the coordinator).
parseMoveArgs gains offline with the seven verbs. The offline verbs skip carlos move's Phase 1 console check (the door refuses for itself). Each mutating verb sends, then polls status until the step it started is done or failed, prints the strings above, and exits with the family's codes; finish posts again once status shows the tombstone lifted. Tests use carlosTestEnv and a scratch cmd.Dir.
The docs paragraph, under ### carlos move (copy, pending):
> carlos move offline moves one instance with planned downtime, one step at a time. It is for operators, works with host claims and Phase 1 moves off, and never acts on its own: each verb does one step and says which verb comes next. > > ``sh > carlos move offline start --host <host> --to <region> > carlos move offline seal --host <host> > carlos move offline flip --host <host> > carlos move offline open --host <host> > carlos move offline finish --host <host> > carlos move offline abort --host <host> > carlos move offline status --host <host> > ` > > The instance is offline from start until open. Until open, abort puts it back where it was with nothing lost. After open`, moving it back is a new move.
- [ ] Failing tests: argument parsing, each verb against a stub console (accepted, refused, waits then done, failed with MV-O6 and MV-O9,
finishposting twice), exit codes; the docsite gate passes with the new examples;TestMovesStringsAreRecordedfinds every new string. - [ ] Implement;
go test . -run 'Move|Copy|Docsite' -count=1. - [ ] Commit:
offline-move: carlos move offline.
Task 8: The two-box witness (Opus)
Files: Create internal/host/movesoffline_e2e_test.go. Nothing else.
One test, TestOfflineMoveTwoBoxes, with two subtests, through production wiring, with no spawned app:
- World.
leasetest.NewS3Serverwith bucketsdata-ieanddata-uk,CARLOS_S3_ENDPOINTpointed at it; oneDirStoreas the deployment store, wrapped in a recording store that logs every Put and Delete (the console and both boxes use the wrapper); the console's doors in-process (console.NewServer, operator token,SecretsPub), asrolloutdeclined_test.gobuilds one. Two boxes,ie-box(placement"", the home) anduk-box(uk), each withnewMovesWorker, its own registry andDataDir, a realactivator.Servicewith the package's fake backend (for the fence only), a realExecBackendused only forLaunchLayersand, on uk afteropen,Restore, the same config keypair, and an edges file written by the production writer. A box'stickrunsreconcileSerializedWith, the edges-file write andmovesPass, in the agent's order. - Source state. The host's database is a real SQLite file on
ie-boxwritten throughlease.ReplicateWithLease, a few rows per step; a files directory withinstance.key; a<host>.envwith an instance key; a config bundle with a store binding and a secret; two timer rows;NextDueandReplicationset on the row. - Move. Drive
start,seal,flip,open,finish(twice) through the doors, ticking both boxes between calls until each step reports. Assert afteropenand afterfinish:lease.Acquireon data-ie returnsHeldError{Owner: "moved:<id>"}untilfinishand succeeds after it;uk-box'sExecBackend.Restoreon the now-ready row promotesmoves/<host>/and the restored file passesCheckDBand holds every row; files and<host>.envbyte-equal to the source's;<host>.config.envrendered; the timer rows,NextDueandReplicationpresent; ie's edges file does not list the host and uk's does; ie's local state is in its park dir, itsparkedreport lists everysource-retirepath, and its row is gone; the record is inukwithlast_move.outcome == doneand, against the record read beforestart, differs only inregion,move,last_move,record_seqandmove.updated_at; the recording store shows no write outside the closed write set (Global Constraints), and none undercontrol/hosts/,control/host-census/,control/host-actors.jsonor a domains claim key; the bucket fake shows none outside the listed data keys. - Rollback. A second host:
start,seal, then tamper the sealed archive indata-uk;flip; the landing'sfilesitem fails andopenrefuses;abort, ticks,finish. Assert: the record is home, the ie row is ready and listed, the tombstone is gone,uk-boxhas no row and its extracted state is parked, both staging prefixes indata-ukare empty,data-ukinstances/<host>/is empty.
The spawned-app proof (the app sees its key, its store object, its secret, its timers) is the dev rehearsal's (P1), which the runbook requires before any flagship move.
- [ ] Write the test; run it alone:
go test ./internal/host -run TestOfflineMoveTwoBoxes -count=1 -v. - [ ] Watch it fail: break each of
files,env,configand the tombstone in turn (in a scratch copy of the tree under$TMPDIR, never in the worktree) and confirm the test fails for each; record the four results in the commit message. - [ ] Commit:
offline-move: two-box witness.
Gate and hand-off
- [ ] The full local gate once (Global Constraints). Push to amadan,
amadan branch describe, wait for CI. - [ ] Opus reviews of T3, T4 and T5 as they land; one whole-branch review; fixes; squash; re-merge
main; CI green on the tip. - [ ] Paul's copy review of the MV-O rows, the reused fills and the docs paragraph; apply verbatim; CI green again.
- [ ] Merge with full arguments. Read
main's CI after.
---
Requirements from the claim-path review
Fable's claim-path review (fable-plan-review.md §9) set eight requirements so this tool never fights the claim later. Each is traced here.
- Requirement §9.1 (claims-off only, by construction): met by T3 (
offlineCapable: no capability, no box step withCARLOS_HOST_CLAIMset or an armed or enforced guard or Phase 1 on) and T6 / start (MV-O21 for any source or target box without the capability; MV-O29 forcontrol/hosts/<host>.json; MV-O20 forcontrol/hosts-backfill.json, forHostClaimincluding theclaimMoveUnsupportedMsgpath atapiinstancemove.go:575-578, and forMovesEnabled); tests in T3 and T6. - Requirement §9.2 (only
regionchanges, through provision's writer withrecord_seq): met by T1 (every transition is aSetRecordMovemut; the full-path diff test) and Global Constraints (no recordPut;acquisition, incarnation, key and domains claim untouched); witnessed again end to end by T8's record diff. - Requirement §9.3 (stop in the legacy cut's order, plus the operator's confirmation and a process read): met by T3 / source step (fence keyed by the move generation, quiesce, process count, layout journal), T4 / step 1 to 5 (journal absent,
CarryRequest, checkpoint, byte-final round withDeferred == 0), T6 / flip (the operator's confirmation, with current reports showing zero processes and no journal, MV-O31). The purge is atopen(T3 / open) instead of before the flip, for the reason in "Why this does not build on the legacy cut". - Requirement §9.4 (retire the source completely and record it): met by T3 / open and finishing (park every
source-retirepath, the row removed atfinishing, theparkedreport's per-path inventory) and the manifest'ssource-retirerow; T8 asserts the inventory. Partly:/opt/carlos/live/<host>and a nested view mount are root-owned, so they are inventoried asleftwith the reason, not removed (no root-side change in this plan). - Requirement §9.5 (target complete before the first wake): met by T5 (files and
<host>.envfrom the archive,config.envwith the store binding throughRenderHeld, every item beforeoffline-landed, and no wake beforeopen) and runbook P0.3 (a certificate path for the host on the target, #310, and the store reachable from the target region). - Requirement §9.6 (one host at a time, a written downtime budget, undo only before the purge): met by T6 / start (MV-O30), runbook P0.6 (a written budget per instance) and the state table (abort until
open; the purge is atopen; after it, a second move). - Requirement §9.7 (rehearse on dev, sophie to dennis, claims off on both): met by runbook P1. It meets ruling (b) in one place: leo is a home-placement box with claims on, so the start door refuses the rehearsal with MV-O21 while leo publishes the home placement. P1 records this as a gate that needs a decision before the rehearsal.
- Requirement §9.8 (never touch the claim's keys): met by Global Constraints (the closed write set) and T8's recording store, which asserts no write under
control/hosts/,control/host-census/,control/host-actors.json, admissions or domains claims. Moved hosts join the #250 files backfill with every other pre-claim host.
Morning runbook outline
Every step waits for Paul's go. Nothing touches production before P2.
P0. Before anything moves.
- Paul froze moves on 2026-10-16 while narrowing the claim gate; lifting it for this operator-run tool on flagship is his decision (D8), taken awake.
- Release: cut a release from the merged
main(README recipe), roll dev boxes, then the dev console, check dev, then flagship boxes, then the flagship console (dev first, always). - Confirm on every box and both consoles:
CARLOS_HOST_CLAIMandCARLOS_MOVES_PHASE1unset (an armed guard is allowed, Ruling 2026-10-20 point 4);systemctl show carlos-edge -p UserprintsUser=carloson every source-region box and the target box, and any other value is a STOP (census-ruling-moves-offline R3: the move processes bucket and tenant-reachable inputs, so it never runs where the agent runs as root); delegated cgroup accounting armed on every source box (carlos-edgeDelegate=yes; read 2026-10-20 on all seven boxes, flagship on v0.28.0);carlos system statusshows every box on the release; each box's edges file carriesoffline-move-1;carlos-stage.patharmed on the UK box (the build arrives through root's ordinary stage pass afteropen). - Facts the code assumes and cannot settle: - the flagship home region has exactly one box (the console's); the sealer rule and MV-O26 assume it; - the UK box holds
CARLOS_SECRETS_KEYand publishes asecrets_key_fpequal to the console's (MV-O23 checks the fingerprint, not the key); -carlos status --layoutfor the three instances: nested means the agent reads tenant-owned files under isolated exec; a file it cannot read fails the seal safely but blocks the move. Settle this on dev (P1) before the flagship. - a certificate path for each host on the UK box (#310), and the store, which stays in eu-west-1, reachable from eu-west-2 with the instance's credentials. - nocontrol/hosts/<host>.jsonfor the three hosts and nocontrol/hosts-backfill.json(the door checks; confirm before the window). - IAM: IE's
sibling_staging_arnsincludes the UK data bucket (it does in carloku-infrastructure as read); list data-uk'smoves/prefix once from the IE box role to prove the pinned module version carries the statement. Recommend applying UK's grant to the IE data bucket now (M17 approved the return lane; the apply is a production infra change and needs Paul's scoped approval), so that a move back afteropenis possible. - Access: Paul's login is an operator and a member of the customer's account (the doors check both).
- After
open, the first wake runs flat: the landing leaves no local database, so the layout pass cannot nest the host until it has one (layout/pass.go:352), and the next pass nests it with a second short stop. Make the first request yourself straight afteropen, then let the layout pass nest it inside the window (or force a nested intent for the UK box, Paul's call), and confirm withcarlos status --layoutbefore handing back. - For each of the three instances: database and files size (under 256 MiB of files), no sidecar, no open run, not seeded, not a replica, not steered. Write a downtime budget per instance (seal time scales with the replica and files size; landing adds a minute) and tell the customer the window. One instance at a time. No promotes to the app during a move (the landing's
releaseitem fails otherwise). - Stale edges files: any file naming the home placement for a box that no longer exists blocks
seal; remove it only with Paul's approval.
P1. Dev rehearsal: sophie-knitting (home) to dennis-stenciling (uk), claims off on both (dennis holds the secrets key since 2026-10-15; leo-climbing is not available, its claims are on). This is where the spawned-app proof lives.
- Gate: leo is a home-placement box with claims on, so ruling (b) makes the start door refuse (MV-O21) while leo publishes the home placement. Before the rehearsal, Paul or the coordinator decides how: leo's edges file is out of the fleet for the window, or the ruling is narrowed for dev with a stated reason. The plan does not choose.
- Deploy moveprobe per
cmd/moveprobe/README.mdwith a store, a secret and a schedule; create an instance in home. Wake it on sophie only, through DNS: with two home boxes the stale-DB hazard (#286) is live, and a wake on leo would give leo local state, whichsealrefuses (MV-O26). Arm a once-timer. - Offline move home to
uk, each verb by hand, readingstatusbetween. Afteropen, request the instance and check/reporton dennis: rows contiguous, the key file matches the database, the instance key fingerprint unchanged, the store object, the secret, the timers. Note the offline seconds and how long the first request waited for root's stage pass. - Rollback rehearsal: a second instance;
start,seal,flip, thenabortfromoffline-landed,finish;/reporton sophie shows nothing lost and dennis holds no row. - If dev's boxes are nested, step 2 is also the nested-archive proof. If it fails on file permissions, stop: the flagship plan changes (Risk 1).
P2. The three flagship moves, one at a time, each with its own go.
carlos move offline start --host <h> --to uk;statusuntil every IE box reported;seal;statusuntil sealed;flip;statusuntil landed;open.- Check:
curl -sI --resolvethe instance's URL against the UK EIP and against the IE EIP (backhaul): 2xx and the expectedX-Carlos-Version; the customer's own sign-in works. finish. Record the offline seconds and anything odd. Then the next instance.- Abort criteria: any failed item, or any doubt before
open;abortbeforeopenis always safe. Afteropen, a failing check in step 2 is fixed forward on uk.
P3. After open, if something is wrong. The target is the only live copy and may have taken writes, so going back is a new offline move from uk to the home, which needs UK's staging grant (P0.4). Its landing on the IE box lifts any stale tombstone and finds no warm cache (the source half parked at open). If the grant is not in place: fix forward on the UK box, and file the gap.
- A return move's landing fails the
releaseitem with MV-O177 ("the target box still holds an older build of this instance") when the box the instance left still keeps its live link/opt/carlos/live/<host>naming the build it ran then, and the app has been promoted since. Retirement leaves that link because it is root-owned, and nothing in this tool or in ordinary adoption repoints it while the row is held, so there is no safe road to clear it from here: abort the move (safe beforeopen), and the instance serves again from where it is. Before starting a return move, read the link on the target box (readlink /opt/carlos/live/<host>): if it names a different version from the one the instance runs now, do not start, and take it to Paul.
Cleanup recipe for an abandoned move (documented, rarely needed): read instances/<host>/lock.json in the source data bucket with lease.LockState; if the owner is moved:<id> for a move that is no longer in flight, delete it. The 7-day expiry frees it anyway.
Interaction with the host claim
The offline move is the claim-off stopgap. It refuses to run with claims on, at every layer, and it retires once claims are on. platform-1e's rulings, as Fable sharpened them:
- (a) The tombstone. It lives 7 days, is re-asserted by the sealer on every pass, and is lifted at
finish(and on abort). The plan documents the cleanup recipe above for a move abandoned beforefinish. Claim activation will refuse while a livemoved:<id>tombstone exists. Claim-mode wakes do not honour the tombstone (a claim wake admits through the claim, not the legacy lease), which is one more reason the tool refuses every claim-shaped box. - (b) Refusal from evidence. The start door refuses if any box in the source or target region has
CARLOS_HOST_CLAIMset, has an armed or enforced guard, or reports claims enabled, if the host has a claim (control/hosts/<host>.json) or the deployment has begun the claim backfill (control/hosts-backfill.json), or if the console has claim mode configured (HostClaim) or Phase 1 on (MovesEnabled). The box side is theoffline-move-1capability, which a box publishes in its own edges file only when none of those holds; a box without it is refused (MV-O21). Nothing comes from what the operator says. - (c) Root-side authority. Moot: rev 2 has no root-side change at all (the held release and the
heldStageAuthoritybranch are gone), so no Plan 14 census row is needed for this tool.
What it leaves behind that activation will meet: last_move with no move (fine); parked state under <DataDir>/.offline-moves/ (a known name in layout/extras.go); a stale lineage in the source bucket's instances/<host>/ with no generation markers, and the live one in the target's with none either (an origin generation, as for any host that never moved).
Remaining questions for platform-1e:
- Activation's preflight refuses while any record carries
move.kind: offlineand while anymoved:*tombstone is live (platform-1e has said yes; this plan relies on it). - Does adoption read a host's lineage only from the record's region's bucket, so the stale source lineage is never selected?
- Should the claim census list
.offline-moves/as a known name, aslayout/extras.gowill? - After activation, are the offline doors kept (refusing) or removed?
Non-goals
Self-serve and member access; automatic recovery or retries beyond a step's own ensures; a no-traffic boot proof on the target before open (the dev rehearsal is the boot proof; after open a boot failure is fixed forward); moving object store data (the bucket stays in its region; residency follows the instance's database only, F4); multi-box target regions; sidecars, seeds, runs, replicas, pools, fleets, steered, unit-backed and non-hibernating instances; rollback after open (a new move); timer run history; the restore-verify verdict; DNS (the flagship's instance hosts resolve through the wildcard and backhaul); live migration; deployments other than the flagship.
Risks
- Nested sources. If the flagship's rows are nested, the agent reads the tenant's files from the instance directory to archive them. A file the agent cannot read fails the seal (safe), but would block the move.
- No boot proof before
open. An app that cannot boot on uk for a reason the manifest does not cover is found by the operator's check afteropenand fixed forward; the dev rehearsal with moveprobe is what makes this acceptable. - The first wake after
openwaits for the agent's blocked stage and root's stage pass (about 8 s per AGENTS.md); ifcarlos-stage.pathis not armed on the UK box it waits much longer (P0.2 checks it). - Tombstone side effects. For up to 7 days, anything that reads the source's
lock.jsonand assumes a 15-minute lease meets amoved:owner. Fable checked the housekeeper (its own key) and the seed finish (never a moved host); a stale box that hits it answers 503 with a longRetry-After. - Timers. Once-timers carry; run history does not, so a schedule may fire once more or once less around the window.
- Budget. Eight tasks in about seven hours of critical path plus reviews leaves little slack.
The three things I am least sure of
- Whether nested rows can be archived by the agent (file ownership under isolated exec), and whether the flagship's three instances are nested at all. The dev rehearsal settles the first;
carlos status --layoutthe second. - Whether dropping the boot proof is the right trade for the real customer. The manifest covers every cause of the 2026-10-02 outage, and the dev rehearsal boots moveprobe, but an app-specific boot failure on uk is only seen after
open, when the fix is forward. Paul may want the probe back (Fable prices it at about three hours and one Opus review). - Whether the critical path fits the night: T3 and T5 carry most of the risk (the pure hold, the ensures), and the copy review has to happen before merge.