amadan
plan: offline operator move (stopgap, claim off)
An operator-run offline move with planned downtime for the three IE to UK moves: stop proven by record holds, per-box fence reports and a tombstone lease in the source bucket; a checked dependency manifest; a probe with no traffic before open; abort before open returns to an untouched source. Eight tasks, an end-to-end test over two in-process regions, the morning runbook and the open questions for the claim session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+513
−0
docs/superpowers/plans/2026-10-20-offline-operator-move.md+513 −0
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 334.330s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.849s
ok github.com/carlosframework/platform/cmd/moveprobe 92.919s
ok github.com/carlosframework/platform/cmd/zddprobe 0.741s
ok github.com/carlosframework/platform/internal/activator 251.653s
ok github.com/carlosframework/platform/internal/agentsock 0.030s
ok github.com/carlosframework/platform/internal/alerting 0.015s
ok github.com/carlosframework/platform/internal/analytics 0.025s
ok github.com/carlosframework/platform/internal/appconfig 3.022s
ok github.com/carlosframework/platform/internal/apperr 0.211s
ok github.com/carlosframework/platform/internal/applog 0.238s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.020s
ok github.com/carlosframework/platform/internal/boxid 0.920s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.029s
ok github.com/carlosframework/platform/internal/caps 0.049s
ok github.com/carlosframework/platform/internal/certwarm 0.005s
ok github.com/carlosframework/platform/internal/cgroup 0.774s
ok github.com/carlosframework/platform/internal/changelog 0.609s
ok github.com/carlosframework/platform/internal/claimactivation 1.565s
ok github.com/carlosframework/platform/internal/clamd 0.068s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 416.359s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.901s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.010s
ok github.com/carlosframework/platform/internal/dbverify 0.005s
ok github.com/carlosframework/platform/internal/dnsrec 0.004s
ok github.com/carlosframework/platform/internal/docsite 0.006s
ok github.com/carlosframework/platform/internal/domains 2.800s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 46.469s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.009s
ok github.com/carlosframework/platform/internal/fleetapi 0.158s
ok github.com/carlosframework/platform/internal/gate 1.814s
ok github.com/carlosframework/platform/internal/genselect 0.018s
ok github.com/carlosframework/platform/internal/geo 0.043s
ok github.com/carlosframework/platform/internal/geoip 0.039s
ok github.com/carlosframework/platform/internal/historyreadback 0.257s
ok github.com/carlosframework/platform/internal/host 185.574s
ok github.com/carlosframework/platform/internal/hostactors 8.449s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.009s
ok github.com/carlosframework/platform/internal/hostbox 3.833s
ok github.com/carlosframework/platform/internal/hostclaim 1.605s
ok github.com/carlosframework/platform/internal/hostenv 0.005s
ok github.com/carlosframework/platform/internal/hostevidence 0.025s
ok github.com/carlosframework/platform/internal/hostexec 8.454s
ok github.com/carlosframework/platform/internal/hostfence 0.076s
ok github.com/carlosframework/platform/internal/hostfiles 11.504s
ok github.com/carlosframework/platform/internal/hostfloor 0.025s
ok github.com/carlosframework/platform/internal/hostguard 1.117s
ok github.com/carlosframework/platform/internal/hostops 3.146s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.003s
ok github.com/carlosframework/platform/internal/hostrelocate 1.116s
ok github.com/carlosframework/platform/internal/hostresolve 0.307s
ok github.com/carlosframework/platform/internal/hostwrite 0.003s
ok github.com/carlosframework/platform/internal/infraview 0.025s
ok github.com/carlosframework/platform/internal/instancectl 5.284s
ok github.com/carlosframework/platform/internal/instanceexec 3.033s
ok github.com/carlosframework/platform/internal/instancefs 0.024s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.025s
ok github.com/carlosframework/platform/internal/layout 2.614s
ok github.com/carlosframework/platform/internal/lbhealth 0.020s
ok github.com/carlosframework/platform/internal/lease 10.133s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.009s
ok github.com/carlosframework/platform/internal/ledger 0.015s
ok github.com/carlosframework/platform/internal/mail 0.539s
ok github.com/carlosframework/platform/internal/maintenance 0.114s
ok github.com/carlosframework/platform/internal/maintenanceexec 6.110s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.221s
ok github.com/carlosframework/platform/internal/metrics 0.015s
ok github.com/carlosframework/platform/internal/moves 25.123s
ok github.com/carlosframework/platform/internal/moves/archive 1.620s
ok github.com/carlosframework/platform/internal/objstore 0.354s
ok github.com/carlosframework/platform/internal/ops 0.282s
ok github.com/carlosframework/platform/internal/park 0.564s
ok github.com/carlosframework/platform/internal/privatefile 0.009s
ok github.com/carlosframework/platform/internal/provision 47.511s
ok github.com/carlosframework/platform/internal/publicrelease 0.011s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 6.838s
ok github.com/carlosframework/platform/internal/relay 7.307s
ok github.com/carlosframework/platform/internal/release 30.245s
ok github.com/carlosframework/platform/internal/rollout 4.976s
ok github.com/carlosframework/platform/internal/rolloutwords 0.008s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.071s
ok github.com/carlosframework/platform/internal/rootstate 0.030s
ok github.com/carlosframework/platform/internal/runs 0.584s
ok github.com/carlosframework/platform/internal/s3compat 0.013s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 3.412s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 22.994s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.124s
ok github.com/carlosframework/platform/internal/status 0.172s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.440s
ok github.com/carlosframework/platform/internal/svcrole 0.003s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.007s
ok github.com/carlosframework/platform/internal/unixdial 0.008s
ok github.com/carlosframework/platform/internal/updatewords 0.005s
ok github.com/carlosframework/platform/internal/usage 0.009s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 2.013s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 49414306 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/19 23:38:03 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E012953692EE9E, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.021s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-20-offline-operator-move.md b/docs/superpowers/plans/2026-10-20-offline-operator-move.md| new file mode 100644 |
| index 000000000..d9cdd7bfb |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-20-offline-operator-move.md |
| @@ -0,0 +1,513 @@ |
| +# Offline operator move Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +Rev 1, 2026-10-20. Written against `main` at 96db0efe on branch `moves-offline`. No spec of its own: this plan is the design, and it borrows its inventory and failure analysis from the move-any-app spec (`docs/superpowers/specs/2026-10-02-move-any-app-design.md` §1, §6, §7.13) and the v0.22 move (`docs/superpowers/specs/2026-08-28-instance-move-design.md`), without the host claim. |
| + |
| +**Goal:** An operator-run move of one hibernating instance between two regions with planned downtime, shipped by morning, for three instances of one customer moving from Ireland (the flagship's home placement, record region `""`, console name `ie`) to the UK (`uk`, eu-west-2). The operator stops the instance, the move copies a checked list of every dependency, proves the target can start it with no traffic, and only then opens it. No self-serve, no background recovery, and no dependency on `CARLOS_HOST_CLAIM`, `CARLOS_MOVES_PHASE1`, `CONSOLE_APP_REGIONS` or `CARLOS_SWAP`. |
| + |
| +**Architecture:** The record carries an offline move (`move.kind: "offline"`) through seven states, each advanced by an operator door on the console or by one box's step in the existing moves pass. The stop is three layers, each of which already exists in some form: the record makes every source-region box hold its row (reconcile's generic `Hold` hook), each of those boxes fences the activator and reports, and one box (the sealer) turns "the source region's single-writer lease is free" into a permanent tombstone lease before it reads the final position. The data travels through the v0.22 staging lane (`moves/<host>/`, IAM already applied for IE to UK), the files and `<host>.env` travel as the Phase 1 sealed archive, and the target lands the instance on a held row, installs and verifies every item, starts it once with no traffic (`StartCheck`), stops it, and waits for `open`. Before `open`, abort flips the record back to a source that nothing touched. |
| + |
| +**Tech Stack:** Go. `internal/provision`, `internal/moves` (+ `archive`), `internal/lease`, `internal/activator`, `internal/host`, `internal/console`, `internal/layout`, the `carlos` CLI (`move.go`). Tests use `leasetest.NewS3Server` (a real S3 wire protocol in-process), `release.DirStore`, real registries, a real `activator.Service` over a real `ExecBackend`, and `cmd/moveprobe` as the app. |
| + |
| +## Answers to the brief's questions |
| + |
| +**What a source-region box does when the record's region changes away from it.** On its next reconcile pass (a serial bump makes that seconds; the floor is the 5-minute timer) the record fails `inScope` (`internal/provision/reconcile.go:401`), is absent from `recs`, and the removal loop retires the Provisioned row as "record gone" (`reconcile.go:1505-1563`, `cfg.claimRetire` then `registry.Delete`). The edge reload drops the route and `RetireExcept` stops a running child, whose final checkpoint failure is logged and swallowed (`activator.go:5023`, movespass.go's own comment at `:650`). So it does stop serving, but not as a stop you can rely on: (1) until that pass runs the row is still `ready` and any request or tick can wake it; (2) the local database stays on disk, and a later flip back trusts it as a warm cache before any bucket state (`internal/activator/backend_exec.go:796`); (3) nothing at the data layer refuses a wake, because the source bucket's lease (`instances/<host>/lock.json`) is free whenever the instance sleeps; (4) a box that is down at the flip keeps its `ready` row, and on restart wakes are admitted once the startup latch closes while the reconcile loops start only after it (`internal/host/agent.go:585-592`). A record flip alone is not a stop. |
| + |
| +**What a target-region box does when the record names its region.** Its next reconcile pass mints a `ready` row (`RouteFor`, `reconcile.go:434`), subject only to the config gate (#213) and `CARLOS_INBOUND`. The edges file publishes it, so every edge backhauls to it. The first request wakes it: lease in the target bucket, then `restorePlain` (`backend_exec.go:784-880`): a local DB is a warm cache and wins; else a staged replica under `moves/<host>/` is promoted and wins (`promoteStagedReplica`, `:3392`); else the own replica; else a **fresh boot and seed**. If `<host>.env` is absent the activator mints a new `CARLOS_INSTANCE_KEY`; if the files directory is absent the app makes a new key file; `<host>.config.env` exists only if this box holds `CARLOS_SECRETS_KEY`. That is the kpmg outage, mechanically. |
| + |
| +**How the v0.22 move stopped the source, and what it missed.** `move/cut` set `state: cutting`; the source box that owns the row (`OwnsRoute`) set status `moving`, installed the in-memory activator fence, waited for `Quiesced`, re-checkpointed the local DB under the lease, ran a final mirror round with zero deferred objects, purged the local DB, and flipped `region` and cleared `move` in one CAS (`internal/host/movespass.go:1280-1380`). It missed: the files directory and `<host>.env` (key, admin token, overrides); a target without the secrets key; every other box in the source region (each has its own registry and warm cache, #286); a fence that survives an agent restart (only the `moving` status and `ServeScheduled`'s gate did); any lease-level fence; any probe (the target opened on the flip); any rollback but a hand `SetRecordRegion`; timers, `NextDue` and `Replication`. |
| + |
| +**What of Phase 1 is reusable without the claim.** Reused as is: `lease.MirrorPrefix` with copy receipts and the `lock.json` skip (`internal/lease/mirror.go:175`, `:227`); the legacy lane `moves/<host>/` and its promotion on the target (`backend_exec.go:3392`); `internal/moves/archive` (`Build`, `Seal`, `Open`, `Verify`, `Extract`, `PlainDigest`; pure); the env pin (`moves.ReportLaunch`, `PortableDigest`, `CompareEnv`, `Pinned`; `internal/moves/envreport.go`) over `ExecBackend.LaunchLayers` (`backend_exec.go:2760`); `appconfig.RenderHeld` (`internal/appconfig/renderheld.go:51`, claim-off only, which is what we want); `heldReleaseReady` and `writeHeldStageRequest` (`internal/host/movesstage.go:262`, `:321`) with `release.MaterializeHeld`/`VerifyHeldLive`; reconcile's `Hold`/`KeepParticipant` hooks, `mintHeld`, `holdExisting` (`reconcile.go:1201`, `:1580`, `:1692`); `registry.SetHold`/`ReleaseHold`/`DeleteIfHold`/`SetNextDue`/`SetReplication`/`ReplaceTimers`/`TimersFor`; `activator.FenceAndStopAs`/`LiftFenceAs`/`Quiesced`; `lease.VerifyReplica`, `lease.LockState`; `localLTXPosition`, `newestLTX` (`movescut.go:1443`, `movesreport.go:599`); `moves.TimersSnapshot`; `moves.NewMoveID`; `cmd/moveprobe`. Reused with one change: `heldStageAuthority` (`internal/host/heldstage.go:63`), whose claim check gains an offline-move branch that reads the record (Task 5). Not reusable: `ProbeWake` (claim admission), the C8 files provider (`movesfiles.go`, claim chain), `stopevidence.Reconstruct`, per-op versioned staging lanes, the Phase 1 door (claim CAS). |
| + |
| +## How stop is guaranteed |
| + |
| +Three layers, in this order, each checked by evidence before the next step: |
| + |
| +1. **The record holds every source-region row, durably.** `start` writes `move{kind: offline, state: offline-stopping}`. Every box whose placement is the source's runs reconcile with an offline `Hold` hook (Task 3), which holds its row `moving` under the move id through the existing `holdExisting`/`mintHeld` (a box with no row mints it held). A held row is in the registry on disk, so it survives an agent restart. Held means: the edge does not serve it (`internal/edge/edge.go:654`), the edges file does not list it (`internal/host/edgesfile.go:411`), ticks are refused (`ServeScheduled`, `activator.go:5317`), the due-wake sweep skips it (`activator.go:4942`), the layout pass leaves it (`internal/layout/pass.go:230`), and config sync and adoption leave it (`syncexec.go:765`, `release/adopt.go:1186`). |
| +2. **Every source-region box fences and reports.** Each such box's offline step installs the activator fence (`FenceAndStopAs`, owner `offline:<id>`, re-installed every pass so a restart regains it), waits for `Quiesced`, and writes `control/offline-moves/<host>/<id>/source/<box>.json`: row held or absent, fence installed, no live entry, the local DB's LTX position, and whether it has a files directory or env file. The `seal` door refuses until every box whose edges file names the source placement (with hibernation configured) has a report saying so and its edges file no longer lists the host, and refuses if more than one box holds local state. |
| +3. **One box seals the source region's lease.** The sealer (the box with local state, else the first source box) checkpoints its local DB if it is ahead of the replica, then calls `lease.Tombstone`: a conditional write of `instances/<host>/lock.json` with owner `moved:<id>` and a one-year expiry, which succeeds only if no writer holds the lease. That write is the linearization point. Every wake on every box in the source region, live, restarted or returning from an outage, must acquire that lease first (`ExecBackend.Acquire`, `backend_exec.go:692`) and now gets `HeldError`. The final position (newest LTX TXID) is read after the tombstone, and every source box's reported local position must be at or below it. |
| + |
| +**The target cannot start before that evidence exists.** It has no row until `flip`, and `flip` requires `offline-sealed`, which only the sealer writes after the tombstone and the final mirror round. After the flip its row is minted held. The only start before `open` is the landing's `StartCheck`, which runs only after the landing has matched the staged replica listing, the files digest, the release, the config and the env pin against the seal report. |
| + |
| +**An unreachable source box** never reports, so `seal` refuses and names it. The move waits (abort is always available before the flip). There is no override: on the flagship the home region has one box, the console box, and if it is unreachable there is no console either. If a box is gone for good its edges file has to be removed by hand, with Paul's approval, before any move from that region (runbook P0). |
| + |
| +## The dependency list |
| + |
| +`internal/moves/offline.go` holds it as `OfflineManifest`, one row per item. The seal report and the landing report each carry a `moves.Check` per item they own; `TestOfflineManifestIsCovered` pins that every row is owned by exactly one of {door refusal, seal, landing, not carried (with a reason)}, and the doors refuse `flip` and `open` unless every owned item is OK. |
| + |
| +| id | item | disposition | proof, and the code that makes it complete | |
| +|---|---|---|---| |
| +| `db` | SQLite DB at its final position | carried: `instances/<host>/` mirrored to the target's `moves/<host>/`; promoted on the target's first wake | seal: listing of the staged prefix (key, size, ETag) after a final `MirrorPrefix` round with `Deferred == 0` (`mirror.go:175`); landing: its own listing equals the seal's, `lease.VerifyReplica` on the staged prefix (`check.go:88`, integrity check), newest LTX TXID equals `FinalTXID` | |
| +| `db-tail` | writes the source's stop failed to checkpoint | carried by the sealer's checkpoint | every source report's local position (`localLTXPosition`) at or below the replica's TXID after the tombstone; the sealer checkpoints first when it is ahead (`moveCfg.Checkpoint`, `movespass.go:306`) | |
| +| `build-state` | `instances/<host>/build-state.json` | carried inside the replica prefix | `release.BuildStateKey`; the sealer runs `CarryRequest` before the final round, as the legacy cut does (`movespass.go:1344`) | |
| +| `lease` | `instances/<host>/lock.json` | not carried (the mirror skips it, `mirror.go:227`); source replaced by the tombstone; a stale `moved:*` tombstone in the target bucket is lifted at landing | `lease.LockState` on both sides | |
| +| `files` | flat `<DataDir>/<host>/`; nested `InstanceDir` minus platform artifacts | carried: sealed archive at `moves/@offline/<host>/<id>/files.sealed` in the target bucket | `archive.PlainDigest` equal on both sides; `Extract` refuses a non-empty destination | |
| +| `env` | `<host>.env`: `CARLOS_INSTANCE_KEY`, `CARLOS_ADMIN_TOKEN`, overrides | carried byte for byte inside the archive (`archive/instancekey_test.go`) | sha256 of the effective instance key equal on both sides (`instancekey`); covered again by `env-pin` | |
| +| `key-adoption` | `<host>.instance-key-adoption.json` | not carried: written only for claim launches (`internal/activator/keyadoption.go:13-16`); a target copy is parked | landing: none present after parking | |
| +| `config` | `<host>.config.env`: the app's settings, store binding, secrets | rendered on the target from the bucket bundle (`appconfig.RenderHeld`) | the start door requires the target's `secrets_key_fp` to equal the console's (`edgesfile.go:283`, R9); covered by `env-pin` | |
| +| `env-pin` | the composed launch environment | app settings must be equal; box settings are not compared (`moves.Pinned`: box-owned variables are left out unless the instance supplied them) | `PortableDigest` from `LaunchLayers` on both rows equal; on a mismatch the landing reports `CompareEnv`'s variable names, never values | |
| +| `release` | the build the source runs | staged and published for the held row on the target | `release.LiveVersion` on the sealer; `heldReleaseReady` then `VerifyHeldLive` on the target, through root's held-stage relay on a hardened box | |
| +| `store` | object store bucket and its credentials | not moved: the bucket stays where it is and the binding travels in `config` | `env-pin`; store data residency is a non-goal | |
| +| `timers` | timer rows, once-timers the app armed included | carried as a `moves.TimersSnapshot` beside the archive; `registry.ReplaceTimers` on the held row | `TimersDigest` equal; declared schedules re-converge from `schedules.json` anyway | |
| +| `next-due` | the row's `NextDue` | carried in `MoveSource.NextDue` | `registry.SetNextDue` on the target, read back | |
| +| `replication` | the row's `Replication` rung | carried in `MoveSource.Replication` | `registry.SetReplication`, read back | |
| +| `row` | the registry row | reproduced by reconcile from the record, minted held at flat paths (`mintHeld`) | landing: row exists, held by the move id | |
| +| `routing` | edges file, backhaul, certificates, derived aliases | produced by existing passes once the row is ready | runbook check after `open`: the URL answers from the target box | |
| +| `sidecar` | `<host>-sidecar.env` | refused: a source report that sees one fails `seal` | source report | |
| +| `restore-verify` | `control/restore-verify.json` entry | not carried: the next drill writes a fresh verdict; until then the old verdict describes the source copy | none (stated in the runbook) | |
| +| `warm-cache` | source local DB, files, env after `open`; target local state found at landing | parked under `<DataDir>/.offline-moves/<host>/<id>/` by rename, never deleted | parked reports | |
| +| `receipts` | `move-receipts/<host>.json` in the source bucket | cleared at `finish` (`clearMoveReceipts`) | none needed (self-checking) | |
| +| `class` | kind, addr, fleet, pool, hibernate, replica, replicas, steering, unit, seed, open run | refused at the door (`moveGate` plus the transition) | door tests | |
| + |
| +## States, failures, and what the operator runs |
| + |
| +`record.move` is `{kind: "offline", id, from, to, state, sealer, boxes, source{...}, requested_by, requested_at, updated_at}`. All fields are additive. |
| + |
| +| state | record region | written by | source boxes | sealer | target box | operator next | abort | |
| +|---|---|---|---|---|---|---|---| |
| +| `offline-stopping` | from | `start` door | hold, fence, report | | | `seal` once every box reported | to `offline-reverting`, region unchanged | |
| +| `offline-sealing` | from | `seal` door | held | checkpoint, tombstone, final mirror, archive, timers, env pin, `seal.json`, then CAS `offline-sealed` | | wait | same | |
| +| `offline-sealed` | from | sealer | held | | | `flip` | same | |
| +| `offline-landing` | to | `flip` door | rows kept held (`KeepParticipant`) | | mint held; verify and install each item; `StartCheck`; `landing.json`; CAS `offline-landed` | `open`, or `abort` if `landing.json` failed | to `offline-reverting` and region back to from | |
| +| `offline-landed` | to | target | held | | held, stopped after the probe | `open` | same | |
| +| `offline-open` | to | `open` door | park local state, lift fence, report `parked` | | release the hold (serving), report `opened` | `finish` | refused: past this point a failure is a new move back | |
| +| `offline-reverting` | from | `abort` door | release hold, lift fence, report `reverted` | lift the tombstone if it is this move's, delete both staging prefixes, report | if it landed: stop, park local state, `DeleteIfHold`, report `reverted` | `finish` | already reverting | |
| +| cleared | to (done) or from (aborted) | `finish` door | rows removed by reconcile when region is to | | | | | |
| + |
| +**Which way each failure goes.** |
| + |
| +- A box step fails (S3 error, a box restarts mid-step): every step is idempotent and re-runs on the next moves pass. The tombstone is re-acquired as already ours, the mirror resumes from receipts, the archive is rebuilt and both objects overwritten before `seal.json`, the landing re-verifies from the top. The report says what failed; the state does not move. |
| +- A seal item fails for a reason that will not clear (a file the archive refuses, over 256 MiB, a special file): `seal.json` says so; `abort`. |
| +- A landing item fails (listing mismatch, digest mismatch, env pin names, release not authorized): `landing.json` says so; the box retries every pass for items that can clear (release staging, config render), never for a mismatch. `abort`. |
| +- The probe fails (`StartCheck` did not reach readiness): `landing.json` says so and the landing does not retry it. `abort`. Nothing was served and no tick ran, so the target wrote only what its own boot wrote, into its own bucket, which the abort discards. The source is exactly as sealed: its rows were held, its lease tombstoned, and the abort lifts both. |
| +- The target box dies after the flip: `abort`. The source half (sealer and source boxes) completes without the target, so the instance serves from the source again. `finish` needs only the source half for an abort; when the target box returns, reconcile removes its held row (record out of scope, move cleared) and a later landing parks any local state it left. |
| +- The source box dies after the seal: the console is on it on the flagship, so the operator waits. The instance was already offline. |
| +- After `open`: out of scope. The manual runbook is a new offline move back (Runbook P3). |
| + |
| +## The operator surface |
| + |
| +**Where it runs:** console operator doors plus the `carlos move offline` subcommand. It is the least new code (the move door family, `moveOperatorGate`, `moveGate`'s class rails, `SetRecordMove`, the serial bump, `carlos move`'s client all exist), it needs no SSM and no box access (AGENTS.md, "Operate via API and CLI"), and every door runs in-process under `httptest` in CI. An SSM ops tool would need AWS credentials in tests and cannot be gated by CI at all. |
| + |
| +Doors, all `cliRoute` with `svcrole.Allow(svcrole.Instance)` so a service token meets the operator gate's byte-identical 404, all behind `moveOperatorGate`: |
| + |
| +``` |
| +POST /api/cli/apps/{account}/{app}/instances/{host}/move/offline {"region":"uk"} |
| +POST .../move/offline/seal |
| +POST .../move/offline/flip |
| +POST .../move/offline/open |
| +POST .../move/offline/finish |
| +POST .../move/offline/abort |
| +GET .../move/offline status: the record's move, every box report, the manifest checklist, the next verb |
| +``` |
| + |
| +CLI (`move.go`), exit codes as `carlos move` (0 done or accepted, 3 aborted, 4 waiting for the operator, 5 refused, 1 error): |
| + |
| +``` |
| +carlos move offline start --host <host> --to <region> |
| +carlos move offline status --host <host> |
| +carlos move offline seal --host <host> |
| +carlos move offline flip --host <host> |
| +carlos move offline open --host <host> |
| +carlos move offline finish --host <host> |
| +carlos move offline abort --host <host> |
| +``` |
| + |
| +Every mutating verb waits until its step is done or fails (polling `status` at 2 s, the same loop `carlos move` has), then prints one line. `status` prints the headline plus one line per manifest item and per box: `<id> ok|waiting|failed <detail>`, where `<detail>` is a box diagnostic (not copy, per the copy-review scope rule). |
| + |
| +### Every user-facing string (copy review before merge) |
| + |
| +New rows go in `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` under a new heading, ids MV-O1 to MV-O30 (check the range is free before writing), `pending` until Paul approves, and into `movesCopyIDs`. Reused approved templates keep their ids; only their fills change. No em dashes. Door sentences are lower case, as the move door's are. |
| + |
| +| id | where | text | |
| +|---|---|---| |
| +| M-C1 (reused) | move.go | Moving <host> from <from> to <to>. | |
| +| MV-O1 | move.go | <host> is offline from now until the move opens or is aborted. | |
| +| M-C10 (reused, fill `<from>`) | move.go | Waiting for <box> in <from> to confirm it has stopped <host>. <host> is offline. | |
| +| MV-O2 | move.go | Every box in <from> has stopped <host>. Run carlos move offline seal --host <host> to copy it. | |
| +| MV-O3 | move.go | Copying <host> to <to>. | |
| +| MV-O4 | move.go | Copied and checked. Run carlos move offline flip --host <host> to start it in <to>. | |
| +| M-C6 (reused) | move.go | Stopped in <from>. Starting in <to>. | |
| +| MV-O5 | move.go | <to> started <host> and stopped it again. Nothing is served yet. Run carlos move offline open --host <host> to serve it from <to>. | |
| +| MV-O6 | move.go | <to> could not start <host>: <reason>. Run carlos move offline abort --host <host> to move it back. | |
| +| MV-O7 | move.go | <host> is served from <to>. Run carlos move offline finish --host <host> once <from> has cleaned up. | |
| +| M-C7 (reused) | move.go | Moved. <host> is served from <to>. It was offline for <seconds> seconds. | |
| +| MV-O8 | move.go | Moving <host> back to <from>. | |
| +| M-C9 (reused) | move.go | <host> is back in <from> and serving. Nothing was lost. | |
| +| MV-O9 | move.go | The move could not finish a step: <reason>. Run carlos move offline status --host <host> for each item. | |
| +| MV-O20 | apiinstancemove_offline.go | offline moves are not available while host claims or Phase 1 moves are on | |
| +| MV-O21 | apiinstancemove_offline.go | <box> cannot take part in an offline move. update it first | |
| +| MV-O22 | apiinstancemove_offline.go | <region> has <n> boxes. an offline move needs exactly one box in the region it moves to | |
| +| MV-O23 | apiinstancemove_offline.go | <box> does not hold this deployment's secrets key | |
| +| MV-O24 | apiinstancemove_offline.go | <host> has no offline move in flight | |
| +| MV-O25 | apiinstancemove_offline.go | this move is <state>. run carlos move offline status --host <host> to see what it waits for | |
| +| MV-O26 | apiinstancemove_offline.go | <host> keeps local data on more than one box: <boxes>. remove the stale copy first | |
| +| MV-O27 | apiinstancemove_offline.go | <host> is already served from <to>. move it back with a new move | |
| +| MV-O28 | apiinstancemove_offline.go | <box> keeps a sidecar file for <host>. instances with sidecars cannot move yet | |
| + |
| +The `docs/site/cli.md` paragraph under `### carlos move` (Task 7) is copy too. The `moveGate` class refusals, `invalid region`, the edges-incomplete 409 and the staging-bucket refusal are existing approved sentences and are reused unchanged. `<from>` is the home's configured name (`homeRegionOut`); with none configured, the word `home`. |
| + |
| +## Global Constraints |
| + |
| +- **Switches stay off.** Every offline door refuses with MV-O20 when the console runs with `HostClaim` or `MovesEnabled`. A box publishes the `offline-move-1` capability, and runs any offline step, only with `CARLOS_HOST_CLAIM` and `CARLOS_MOVES_PHASE1` off; otherwise it logs one notice per host and does nothing. Nothing here reads `CONSOLE_APP_REGIONS`, `CARLOS_SWAP` or `CARLOS_REGION_HA`. |
| +- **Kinds never cross.** No legacy transition (`MoveAbort`, `MoveMarkCutting`, `MoveComplete`, `MoveAbortComplete`, `MovePromoteReady`) and no Phase 1 guarded transition accepts `kind: offline`, and no offline transition accepts a move without it. The box's moves pass routes an offline record before the seed check and before the Phase 1 and legacy branches. |
| +- **Wire shapes are additive.** `MoveState.Kind`, `Sealer`, `Boxes`; `MoveSource.Release`, `EnvDigest`; the reports; the edges capability. A record written before this parses byte-identically (test). |
| +- **Rename, never delete, local state.** Parking is `os.Rename` into `<DataDir>/.offline-moves/<host>/<id>/<role>/`, and `.offline-moves` joins the known names in `internal/layout/extras.go:329`. Bucket objects are deleted only in the staging prefixes this move wrote. |
| +- **No infra change is required.** IE's role already has `sibling_staging_arns = [data_uk]` (carloku-infrastructure `region-ie.tf:42`, read at 7c99ff8, a local checkout that may lag): Put and Delete on `moves/*`, List with the `moves/*` prefix condition. `moves/@offline/...` is inside it. A move back from the UK needs UK's grant (Runbook P0). |
| +- **Production-wired witnesses.** Box steps are tested through `newMovesWorker` and `reconcileSerializedWith` (the functions the agent calls), against `leasetest.NewS3Server` and real registries. A fake is allowed only where the plan names it. |
| +- **Gate.** While iterating, targeted runs only, niced and serial, for example `go test ./internal/host -run 'Offline' -count=1`, `go test ./internal/lease -run 'Tombstone' -count=1`. Never the whole `internal/host` or console suite locally (CI runs `make test`). Once before handing off: `go build ./...`, `go vet ./...`, `gofmt -l .`, the copy gate (`go test . -run 'Moves|Copy|Docsite' -count=1`), and `GOOS=darwin`, `GOOS=windows`, `GOOS=linux GOARCH=arm64` `go build ./...`. Default Go build cache. `GOFLAGS=-buildvcs=false` if the worktree's VCS metadata is unreadable from the sandbox. Scratch under `$TMPDIR`. |
| +- **Copy.** Every string above is a `pending` row in the commit that writes it. Paul reviews them before merge; merge waits for that. |
| +- **Merge and roll.** Push to amadan, describe, squash, re-merge `main`, green CI on the tip, `amadan branch merge carlos/platform moves-offline -expect <tip> -message "..."` with full arguments. Nothing is released or rolled without Paul's go. |
| + |
| +## Review Focus |
| + |
| +1. **The linearization point.** The tombstone must be taken after the sealer's checkpoint released its lease and before the final position is read; every source report's local position must be at or below that position; and nothing between the holds and the tombstone can wake the host on a reporting box (held row and fence). Check that `Tombstone` is a conditional write that never replaces a live holder's lease and never extends anyone else's. |
| +2. **Nothing reaches the target early.** The target has no row before `flip`; its row is minted held; config sync, adoption, the edge, ticks and the due sweep all skip a held row; `StartCheck` refuses a row not held by this move id. The landing compares its own listing of `moves/<host>/` with the seal's, so a stale or extra staged object fails the landing instead of being promoted. |
| +3. **Rollback loses nothing.** Between `flip` and `open` the only process the target ran is the probe, and it served nothing. Abort before `open` returns the record to the source, lifts only this move's tombstone (owner match plus ETag), and releases only this move's holds (`ReleaseHold(host, id)`). |
| +4. **Root's new authority.** The offline branch of `heldStageAuthority` must be as narrow as the claim branch: the record names this op with `kind: offline` and state `offline-landing`, the record's region and `move.to` are this box's placement, the row is held by the op, and the version is `move.source.release`. It must refuse when `CARLOS_HOST_CLAIM` is on. Opus reviews it. |
| +5. **The hold view fails closed.** Until the first build, `KeepParticipant` keeps every row (as the Phase 1 view does), which delays removals on a fresh agent by one moves pass. Check that the startup refresh runs before the reconcile loops. |
| +6. **The manifest is the checklist.** `TestOfflineManifestIsCovered` must fail if a manifest row is added with no owner, or a report carries an item the manifest does not list. |
| + |
| +--- |
| + |
| +## File structure |
| + |
| +| path | responsibility | |
| +|---|---| |
| +| `internal/provision/provision.go`, `internal/provision/offline_move.go` (new) | record fields, offline states, transitions | |
| +| `internal/moves/offline.go` (new) | `OfflineManifest`, report shapes, keys, owner and park-path helpers | |
| +| `internal/moves/archive/archive.go` | `BuildFiltered` (a skip function for nested sources) | |
| +| `internal/layout/plan.go`, `internal/layout/extras.go` | `NestedPlatformNames(host)`; `.offline-moves` known name | |
| +| `internal/lease/tombstone.go` (new) | `Tombstone`, `LiftTombstone`, `TombstoneOwner` check | |
| +| `internal/activator/startcheck.go` (new) | `(*Service).StartCheck` | |
| +| `internal/host/movesoffline.go` (new) | the offline hold view, the dispatcher, the source and parking steps | |
| +| `internal/host/movesoffline_seal.go` (new) | the sealer and its revert | |
| +| `internal/host/movesoffline_land.go` (new) | the landing, `open` on the target, the target's revert | |
| +| `internal/host/movespass.go`, `internal/host/agent.go`, `internal/host/edgesfile.go`, `internal/host/heldstage.go` | wiring: `newMovesWorker`, `reconcileSerializedWith`, the capability, root's offline authority | |
| +| `internal/console/apiinstancemove_offline.go` (new), `internal/console/authcli.go` | the doors and the status view | |
| +| `move.go`, `docs/site/cli.md` | the CLI subcommand and its docs paragraph | |
| +| `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`, `approved_moves_copy_test.go` | copy rows | |
| +| `internal/host/movesoffline_e2e_test.go` (new) | the two-region end-to-end test | |
| + |
| +## Order and parallelism |
| + |
| +Task 1 first. Then Task 2 and Task 6 in parallel with Task 3. Task 4 needs 2 and 3; Task 5 needs 2 and 4; Task 7 needs 6; Task 8 needs everything. Critical path: 1, 3, 4, 5, 8. Delegation: Sonnet for 1, 6, 7; Opus for 2, 3, 4, 5, 8 and as reviewer of 2, 4 and 5 (production and root paths); one whole-branch review before merge by whichever reviewer Paul's budget rule picks. |
| + |
| +--- |
| + |
| +### Task 1: Record shape, transitions and the manifest |
| + |
| +**Files:** Modify `internal/provision/provision.go`. Create `internal/provision/offline_move.go`, `internal/provision/offline_move_test.go`, `internal/moves/offline.go`, `internal/moves/offline_test.go`. |
| + |
| +**Interfaces:** |
| + |
| +```go |
| +// provision |
| +const MoveKindOffline = "offline" |
| +const ( |
| + OfflineStopping = "offline-stopping" |
| + OfflineSealing = "offline-sealing" |
| + OfflineSealed = "offline-sealed" |
| + OfflineLanding = "offline-landing" |
| + OfflineLanded = "offline-landed" |
| + OfflineOpen = "offline-open" |
| + OfflineReverting = "offline-reverting" |
| +) |
| +// MoveState gains (additive, omitempty): |
| +// Kind string `json:"kind,omitempty"`; Sealer string `json:"sealer,omitempty"`; Boxes []string `json:"boxes,omitempty"` |
| +// MoveSource gains: Release string `json:"release,omitempty"`; EnvDigest string `json:"env_digest,omitempty"` |
| +func (m *MoveState) Offline() bool |
| +func OfflineStart(id, from, to, by string, now time.Time) func(*Record) error |
| +func OfflineSeal(id, sealer string, boxes []string) func(*Record) error |
| +func OfflineSealedBy(id, sealer string, src MoveSource) func(*Record) error |
| +func OfflineFlip(id string) func(*Record) error // Region = To |
| +func OfflineLandedBy(id, to string) func(*Record) error // guarded by record region == to |
| +func OfflineOpen(id string) func(*Record) error |
| +func OfflineAbort(id string) func(*Record) error // landing/landed: Region = From |
| +func OfflineFinish(id string, now time.Time) func(*Record) error // LastMove = {id, done|aborted, at} |
| + |
| +// moves |
| +type OfflineItem struct{ ID, Item, Disposition, Owner string } // Owner: door|seal|landing|none |
| +var OfflineManifest []OfflineItem |
| +type OfflineReport struct { |
| + V int; Op, Host, Box, Role, Phase string // role: source|sealer|target; phase: stopped|sealed|landed|opened|parked|reverted|failed |
| + Items map[string]Check; At time.Time |
| + Held, Fenced, Quiesced, LocalDB, Files, Env, Sidecar bool; LocalTXID uint64 |
| + FinalTXID uint64; Staged map[string]StagedObject; FilesPlain, FilesSealed string; FilesBytes int64; FilesEntries int |
| + EnvDigest, Release, TimersDigest string; Replication string; NextDue int64 |
| +} |
| +func OfflineReportKey(host, op, role, box string) string // control/offline-moves/<host>/<op>/<role>/<box>.json |
| +func OfflineArchivePrefix(host, op string) string // moves/@offline/<host>/<op>/ |
| +func TombstoneOwner(op string) string // "moved:" + op |
| +func OfflineParkDir(dataDir, host, op, role string) string |
| +``` |
| + |
| +Each transition checks `rec.Move.Offline()`, the id, and the one legal prior state, and returns `ErrMoveConflict` otherwise. `OfflineStart` carries `MoveRequest`'s refusals (seed, open run, a move in flight, same region, a replica) and stamps `Kind`, `ID`, `From`, `To`, `State`, `RequestedBy`, `RequestedAt`. `OfflineAbort` is legal from stopping, sealing, sealed, landing and landed; from landing and landed it also sets `Region = From`; from open it refuses; from reverting it is a no-op success. `OfflineFinish` is legal from open (outcome `done`) and reverting (`aborted`). |
| + |
| +- [ ] Failing tests: the full legal path; every illegal pair refused; wrong id refused; abort before and after the flip (region); finish from both; a record JSON written before this change round-trips byte-identically; each legacy and Phase 1 transition refuses an offline move (`MoveAbort`, `MoveMarkCutting`, `MoveComplete`, `MoveAbortComplete`, `MovePromoteReady`, one guarded Phase 1 transition); `TestOfflineManifestIsCovered`; key helpers reject a host that fails `ValidHostShape`. |
| +- [ ] Implement; `go test ./internal/provision ./internal/moves -run 'Offline|Move' -count=1`. |
| +- [ ] Copy rows MV-O1 to MV-O28 as `pending`, ids in `movesCopyIDs` (strings are wired in Tasks 6 and 7; the gate's "recorded" check runs once they are). |
| +- [ ] Commit: `offline-move: record shape, transitions and the dependency manifest`. |
| + |
| +### Task 2: The tombstone lease and the start check |
| + |
| +**Files:** Create `internal/lease/tombstone.go`, `internal/lease/tombstone_test.go`, `internal/activator/startcheck.go`, `internal/activator/startcheck_test.go`. |
| + |
| +**Interfaces:** |
| + |
| +```go |
| +// lease: the same client and lss3.Leaser AcquireTTL builds, with no renewer. |
| +func Tombstone(ctx context.Context, bucket, location, region, owner string, until time.Time) error |
| +// absent or expired: conditional write {owner, until}; held by owner: nil; held by anyone else: *HeldError. |
| +func LiftTombstone(ctx context.Context, bucket, location, region string, mine func(owner string) bool) (lifted bool, err error) |
| +// absent: (false, nil); owner not mine: (false, *HeldError); mine: delete with If-Match on the read ETag. |
| + |
| +// activator |
| +var ErrStartCheckRefused = errors.New("activator: start check refused") |
| +func (a *Service) StartCheck(ctx context.Context, rt registry.Route, holdOp string, generation uint64, owner string) (*StopEvidence, error) |
| +``` |
| + |
| +`StartCheck` re-reads the row (`a.Routes.Get`) and refuses unless it is held with `HoldOp == holdOp` and the host has no live entry. It then runs `a.begin(ctx, row, false)` (the ordinary wake: lease, restore, start, readiness, no activity stamp), calls the returned release, and `FenceAndStopAs(ctx, host, generation, owner)`. A wake error is returned wrapped. `ErrStopUnproven` with `Quiesced(host)` true is success with nil evidence. The fence stays installed; the caller lifts it with `LiftFenceAs`. |
| + |
| +- [ ] Failing lease tests against `leasetest.NewS3Server`: a tombstone blocks `lease.Acquire` with `HeldError{Owner: "moved:mv_..."}`; a held `lease.Acquire` blocks `Tombstone`; an expired lease is replaced; a repeat by the same owner is a no-op; `LiftTombstone` refuses another owner, lifts its own, and a following `Acquire` succeeds; `MirrorPrefix` still skips the tombstone's `lock.json`. |
| +- [ ] Failing activator tests with the package's fake backend: refused for an unheld row, a row held by another op, a live entry; a passing start returns after the stop with the fence held; a backend whose start fails returns the error and leaves nothing running; `ServeScheduled` and `ServeHTTP` on a fenced host start nothing. |
| +- [ ] Implement; `go test ./internal/lease -run Tombstone -count=1`, `go test ./internal/activator -run StartCheck -count=1`. |
| +- [ ] Commit: `offline-move: tombstone lease and the activator start check`. |
| + |
| +### Task 3: Holds, the capability and the source boxes' steps |
| + |
| +**Files:** Create `internal/host/movesoffline.go`, `internal/host/movesoffline_test.go`. Modify `internal/host/movespass.go` (`newMovesWorker`, the dispatch, `moveCfg` hooks), `internal/host/agent.go` (`reconcileSerializedWith`, startup refresh), `internal/host/edgesfile.go` (capability), `internal/layout/extras.go`. |
| + |
| +**Interfaces:** |
| + |
| +```go |
| +type offlineView struct{ /* box, region; atomic snapshot host -> MoveState */ } |
| +func newOfflineView(box, region string) *offlineView |
| +func (v *offlineView) refresh(recs []provision.Record) |
| +func (v *offlineView) hold(rec provision.Record) (status, op string, ok bool) |
| +// source region and state stopping|sealing|sealed: ("moving", id); target region and landing|landed: ("landing", id); else false. |
| +func (v *offlineView) keep(host string) bool |
| +// unbuilt: true; an offline move whose from or to is this box's region and not cleared: true. |
| +func withOfflineHolds(cfg provision.ReconcileConfig, v *offlineView) provision.ReconcileConfig // only when cfg.Hold and KeepParticipant are nil |
| +func reconcileSerializedWith(ctx context.Context, s release.Store, reg *registry.Registry, cfg provision.ReconcileConfig, v *offlineView) error |
| +func newMovesWorker(cfg moveCfg) *movesWorker // armMovesPass calls it; tests build one per box |
| +func (w *movesWorker) offlineRound(ctx context.Context, s release.Store, rec provision.Record) error |
| +const capOfflineMove1 = "offline-move-1" |
| +``` |
| + |
| +`moveCfg` gains `Box` (the edges box label, set always), `FenceAs`, `LiftFenceAs`, `ReleaseHold`, `DeleteIfHold`, `Offline *offlineView`, `HostClaim bool`. `movesPass` refreshes the view from its record list and routes `rec.Move.Offline()` to `offlineRound` first. `offlineRound` returns at once, with one notice per host, when `HostClaim` or the Phase 1 switch is on. |
| + |
| +Source role (this box's placement equals `move.from`; `""` for home): |
| + |
| +- **stopping, sealing, sealed:** `FenceAs(host, moveFenceGeneration, "offline:"+id)` with a 2 s deadline (`ErrStopUnproven` is fine); read the row; write the source report: `Held` (row absent or held by id), `Fenced`, `Quiesced`, `LocalDB` and `LocalTXID` (`localLTXPosition` on the row's DBPath, flat or nested), `Files`, `Env`, `Sidecar`. Rewrite only when a field changed. |
| +- **open:** with the fence held and the row held, rename the local DB, WAL, SHM and both litestream meta dirs, the files directory (or the nested instance directory's non-platform entries) and the env file into `OfflineParkDir(..., "source")`; lift the fence; report `parked`. |
| +- **reverting (not the sealer's part):** `ReleaseHold(host, id)` under `provision.WithHostHoldLock`; `LiftFenceAs`; report `reverted`. |
| + |
| +Edges file: `edgesFileCapabilities` adds `offline-move-1` when hibernation is configured and `CARLOS_HOST_CLAIM` and `CARLOS_MOVES_PHASE1` are off. |
| + |
| +- [ ] Failing tests through `newMovesWorker` and `reconcileSerializedWith`, two source boxes and one target over one `DirStore`: `start` holds the row on both source boxes and neither lists it in its edges file; each writes a report; a box whose row already exists keeps its other columns; an agent restart (a new worker over the same registry) re-fences and re-reports; a box with local state reports its position; after the flip the source rows stay (keep); in `offline-reverting` both rows are released and serving and the hold is not re-asserted by the next reconcile; in `offline-open` the local state is renamed into the park dir and nothing is deleted; a box with `CARLOS_HOST_CLAIM` on does nothing; the edges capability is present and absent as specified; the view is fail-closed before its first refresh. |
| +- [ ] Implement; `go test ./internal/host -run 'Offline' -count=1`; `go test ./internal/layout -count=1`. |
| +- [ ] Commit: `offline-move: holds, the capability and the source boxes' steps`. |
| + |
| +### Task 4: The sealer |
| + |
| +**Files:** Create `internal/host/movesoffline_seal.go`, `internal/host/movesoffline_seal_test.go`. Modify `internal/moves/archive/archive.go` (+ test), `internal/layout/plan.go` (+ test). |
| + |
| +**Interfaces:** |
| + |
| +```go |
| +func BuildFiltered(filesDir, envPath string, skip func(rel string) bool, w io.Writer) (plain string, entries []Entry, bytes int64, err error) // Build = BuildFiltered(nil) |
| +func NestedPlatformNames(host string) map[string]bool // from nestedFiles' names plus <db>.gen, .files-gen, the adoption record |
| +func (w *movesWorker) offlineSeal(ctx context.Context, s release.Store, rec provision.Record) error // cfg.Box == move.sealer, state sealing |
| +func (w *movesWorker) offlineSealerRevert(ctx context.Context, s release.Store, rec provision.Record) error |
| +``` |
| + |
| +The sealer runs, every step idempotent, in this order: |
| + |
| +1. Refuse unless its own source report says held, fenced and quiesced, and no other box's report shows a sidecar. |
| +2. If its local TXID is above the replica's newest TXID, `cfg.Checkpoint` (lease, sync, release). |
| +3. `lease.Tombstone(OwnBucket, HibernatePrefix(OwnPrefix, host), OwnBucketRegion, TombstoneOwner(id), now + 365 days)`. |
| +4. List the replica, take `FinalTXID` (`newestLTX`), and refuse if any source report's `LocalTXID` is above it. |
| +5. `CarryRequest`, then a final `MirrorPrefix` round into the target's `moves/<host>/`, refusing `Deferred > 0`; list the staged prefix as the seal's `Staged`. |
| +6. Build and seal the archive (`BuildFiltered` over the flat files dir or the nested instance dir with `NestedPlatformNames`; `archive.Seal` under the deployment's config public key, binding the record's account and app); `Put` it and its manifest under `OfflineArchivePrefix`. Size and file refusals become failed items. |
| +7. Timers: `TimersFor`, `ActiveTimers`, `TimersSnapshot.Encode`, `Put` beside the archive, `TimersDigest`. |
| +8. Env pin: `LaunchLayers` on its own row, `ReportLaunch` keyed by `ReportKeyFromSecrets(CARLOS_SECRETS_KEY)`, `PortableDigest`. |
| +9. `Release` from `release.LiveVersion`, `Replication` and `NextDue` from the row. |
| +10. Write `seal.json` (role `sealer`), then CAS `OfflineSealedBy` with a `MoveSource` carrying `FinalTXID`, `Files`, `TimersSHA256`, `Release`, `EnvDigest`, `Replication`, `NextDue`, and `Evidence` = the report key and its sha256. |
| + |
| +Revert (state reverting, this box the sealer): `LiftTombstone` with `mine = owner == TombstoneOwner(id)`; `DeletePrefix` on the target's `moves/<host>/` and `OfflineArchivePrefix`; `clearMoveReceipts`; then the source revert of Task 3; report `reverted`. |
| + |
| +- [ ] Failing tests against `leasetest.NewS3Server` with buckets `data-ie` and `data-uk` and a real litestream replica written by `lease.ReplicateWithLease`: the full seal produces a staged copy whose listing matches, a tombstone, an archive that `Open`, `Verify` and `Extract` back to the same `PlainDigest`, and `offline-sealed`; a live writer's lease makes step 3 fail and the state stays sealing; a source report above the final TXID refuses; a nested fixture produced by the real layout pass archives without the database, env or config files and extracts flat; a file over the cap and a FIFO are failed items, never a partial archive; a replay after a crash at each step finishes with one tombstone and identical staged objects; the revert lifts only this move's tombstone and deletes only the two prefixes. |
| +- [ ] Implement; `go test ./internal/host -run 'OfflineSeal' -count=1`; `go test ./internal/moves/archive ./internal/layout -count=1`. |
| +- [ ] Commit: `offline-move: the sealer`. |
| + |
| +### Task 5: The landing, open, and the target's revert |
| + |
| +**Files:** Create `internal/host/movesoffline_land.go`, `internal/host/movesoffline_land_test.go`. Modify `internal/host/heldstage.go` (+ test). |
| + |
| +**Interfaces:** |
| + |
| +```go |
| +func (w *movesWorker) offlineLand(ctx context.Context, s release.Store, rec provision.Record) error // this box's placement == move.to, state landing |
| +func (w *movesWorker) offlineTargetOpen(ctx context.Context, s release.Store, rec provision.Record) error // state open |
| +func (w *movesWorker) offlineTargetRevert(ctx context.Context, s release.Store, rec provision.Record) error // state reverting, this box holds the row |
| +func offlineHeldStageCheck(ctx context.Context, store release.Store, reg *registry.Registry, box, region string, req release.HeldStageRequest) error |
| +``` |
| + |
| +The landing, in order, each item a `Check` in `landing.json`, retried every pass until it passes or is a mismatch: |
| + |
| +1. `row`: the reconciled row exists, held `landing` by the move id. Wait otherwise. |
| +2. Read and verify `seal.json` against `move.source.evidence`. |
| +3. `warm-cache` and `key-adoption`: rename any local DB, meta dirs, files dir, env file and adoption record for the host into `OfflineParkDir(..., "target-before")`. |
| +4. `lease`: `LiftTombstone` on the target bucket for any owner with the `moved:` prefix (a move out of this region earlier). |
| +5. `db`: list `moves/<host>/`; it must equal the seal's `Staged` exactly; `lease.VerifyReplica` on that prefix into a scratch directory under `OfflineParkDir(..., "verify")`, removed after; newest TXID equals `FinalTXID`. |
| +6. `files`, `env`: fetch, `archive.Open` with the box's secrets key, `Verify`, `Extract` to the flat `<DataDir>/<host>/` and `<DataDir>/<host>.env`; `PlainDigest` equals the seal's; the instance key's sha256 equals the seal's. |
| +7. `release`: `heldReleaseReady` with `move.source.release` (on a hardened box this writes the held-stage request and touches the stage marker; root serves it through the offline authority below). |
| +8. `config`: `appconfig.RenderHeld(ctx, s, priv, dataDir, row, id, box)`. |
| +9. `env-pin`: `LaunchLayers` on the held row, `PortableDigest` equals `move.source.env_digest`; on a mismatch the item's detail is `CompareEnv`'s names. |
| +10. `timers`, `next-due`, `replication`: `ReplaceTimers` from the snapshot (digest checked), `SetNextDue`, `SetReplication`, each read back. |
| +11. `probe`: once per move, `StartCheck(row, id, generation, "offline:"+id)`, then `LiftFenceAs`. A failure is final for this move. |
| +12. Write `landing.json`, then CAS `OfflineLandedBy(id, to)`. |
| + |
| +`open` on the target: `ReleaseHold(host, id)` under `WithHostHoldLock`, delete `OfflineArchivePrefix` in its own bucket, report `opened`. Revert on the target: `FenceAndStopAs`, rename local state into `OfflineParkDir(..., "target")`, `DeleteIfHold(host, id)`, report `reverted`. |
| + |
| +Root's authority: `heldStageAuthority` tries the claim as today; when the host has no claim and `CARLOS_HOST_CLAIM` is off, it calls `offlineHeldStageCheck`, which reads the record named by the request's account, app and host and requires `Move.Offline()`, `Move.ID == req.Op`, state `offline-landing`, `rec.Region == move.to == region` (root's own `CARLOS_REGION`), `req.Box == box`, the row absent or held by the op, and `req.Version == move.source.release`. The freeze re-check runs the same function. |
| + |
| +- [ ] Failing tests: the full landing over `leasetest.NewS3Server` and a real `ExecBackend` with the package's minimal app; every mismatch (an extra staged object, a changed archive, a changed bundle, a version the seal did not name) fails its item and the probe never runs; a stale local DB and a stale `moved:` tombstone are parked and lifted; a failing app leaves `probe` failed, nothing running and the row held; `open` makes the row ready and listed in the edges file; the revert parks and deletes the row; root's authority admits exactly the offline case and refuses each field's mismatch, a claim-on box, and a record in any other state. |
| +- [ ] Implement; `go test ./internal/host -run 'OfflineLand|HeldStage' -count=1`. |
| +- [ ] Commit: `offline-move: landing, open and the target's revert`. |
| + |
| +### Task 6: The console doors and the status view |
| + |
| +**Files:** Create `internal/console/apiinstancemove_offline.go`, `internal/console/apiinstancemove_offline_test.go`. Modify `internal/console/authcli.go` (routes). |
| + |
| +Each door is `moveOperatorGate`, then MV-O20, then its own checks, then one `SetRecordMove` transition, then `bumpSerial`, answering 202 with the status view. Read the edges files through `internal/edgesfile`'s snapshot; an incomplete read is the existing 409. |
| + |
| +- **start:** `moveGate`'s class rails; replica, replicas, steering and unit refusals; body `{"region"}` through `homeRegionIn`, `ValidRegion`, not the current region; the manifest's `regions` allow-list (`moveRegionAllowed`); source boxes are the edges files whose placement is `from` (home: placementless, not fleet) with hibernation configured; target boxes likewise for `to`; exactly one target box (MV-O22); every source and target box has `offline-move-1` (MV-O21); the target's `secrets_key_fp` equals `appconfig.PublicFingerprint(SecretsPub)` (MV-O23); the target publishes a data bucket (the existing staging refusal); the target does not list the host. Transition `OfflineStart` with `moves.NewMoveID`. |
| +- **seal:** state stopping; for every source box: a report for this id with `Held`, `Fenced`, `Quiesced`, and its edges file not listing the host, else M-C10 naming the box (409); `Sidecar` anywhere is MV-O28; more than one box with `LocalDB`, `Files` or `Env` is MV-O26; sealer = that box, else the lexically first. Transition `OfflineSeal(id, sealer, boxes)`. |
| +- **flip:** state sealed; `seal.json` present, its sha256 equal to `source.evidence`, every seal-owned item OK. Transition `OfflineFlip`. |
| +- **open:** state landed; `landing.json` every landing-owned item OK. Transition `OfflineOpen`. |
| +- **finish:** open: the target's `opened` report and every source box's `parked`; reverting: the sealer's and every source box's `reverted`. Transition `OfflineFinish`. |
| +- **abort:** `OfflineAbort`; open is MV-O27. |
| +- **status (GET):** the move, every report, the manifest with each item's latest check, and the next legal verb. |
| + |
| +- [ ] Failing tests with an in-process server, an operator CLI token and a `DirStore`: every refusal above with its exact sentence; a member's and a service token's 404 byte-identical to the instance GET's; each door's happy path writes the transition and bumps the serial; MV-O20 with `HostClaim` and with `MovesEnabled`. |
| +- [ ] Implement; `go test ./internal/console -run 'Offline' -count=1`. |
| +- [ ] Commit: `offline-move: console doors and status`. |
| + |
| +### Task 7: The CLI, the docs paragraph and the copy gate |
| + |
| +**Files:** Modify `move.go`, `move_test.go`, `docs/site/cli.md`, `approved_moves_copy_test.go`, the copy record. |
| + |
| +`parseMoveArgs` gains `offline` with the seven verbs. The offline verbs skip `carlos move`'s Phase 1 console check (the door refuses for itself). Each mutating verb sends, then polls `status` until the step it started is done or failed, prints the strings above, and exits with the family's codes. Tests use `carlosTestEnv` and a scratch `cmd.Dir`. |
| + |
| +The docs paragraph, under `### carlos move` (copy, pending): |
| + |
| +> `carlos move offline` moves one instance with planned downtime, one step at a time. It is for operators, works with host claims and Phase 1 moves off, and never acts on its own: each verb does one step and says which verb comes next. |
| +> |
| +> ```sh |
| +> carlos move offline start --host <host> --to <region> |
| +> carlos move offline seal --host <host> |
| +> carlos move offline flip --host <host> |
| +> carlos move offline open --host <host> |
| +> carlos move offline finish --host <host> |
| +> carlos move offline abort --host <host> |
| +> carlos move offline status --host <host> |
| +> ``` |
| +> |
| +> The instance is offline from `start` until `open`. Until `open`, `abort` puts it back where it was with nothing lost. After `open`, moving it back is a new move. |
| + |
| +- [ ] Failing tests: argument parsing, each verb against a stub console (accepted, refused, waits then done, failed with MV-O6 and MV-O9), exit codes; the docsite gate passes with the new examples; `TestMovesStringsAreRecorded` finds every new string. |
| +- [ ] Implement; `go test . -run 'Move|Copy|Docsite' -count=1`. |
| +- [ ] Commit: `offline-move: carlos move offline`. |
| + |
| +### Task 8: End to end, two regions in one process |
| + |
| +**Files:** Create `internal/host/movesoffline_e2e_test.go`. |
| + |
| +One test, `TestOfflineMoveEndToEnd`, with two subtests, through production wiring only: |
| + |
| +- **World.** `leasetest.NewS3Server` with buckets `data-ie`, `data-uk` and `store`, `CARLOS_S3_ENDPOINT` pointed at it; one `DirStore` as the deployment store; an in-process console (`console.NewServer`, operator token, `SecretsPub`), as `rolloutdeclined_test.go` builds one; `cmd/moveprobe` built once with `go build` into `t.TempDir()`. Two boxes, `ie-box` (placement `""`, the home) and `uk-box` (`uk`), each with its own registry, `ExecBackend` (own `DataDir` and `OptDir`), `activator.Service`, `newMovesWorker`, `offlineView`, the same config keypair, and an edges file written by the production writer. A box's `tick` runs, in the agent's order, `reconcileSerializedWith`, `appconfig.SyncExec`, the edges-file write, `movesPass` and, on a box whose `OptDir` the test owns, the held release in-process. |
| +- **The app.** A moveprobe instance created in the home through the console, a store binding (`MOVEPROBE_*` pointing at the `store` bucket), `MOVEPROBE_SECRET` in the config bundle, and the release shipped and promoted with the test signing key. Wake it on `ie-box` through `svc.ServeHTTP` a few times, arm a once-timer, let it sleep. |
| +- **Move.** Drive `start`, `seal`, `flip`, `open`, `finish` through the console's HTTP doors, ticking both boxes between calls until each step reports. Assert: on `uk-box` a wake through `ServeHTTP` reaches moveprobe's `/report` with `instance.key` matching the database, the `CARLOS_INSTANCE_KEY` fingerprint unchanged, the store object readable, the secret present, every row the source wrote present and contiguous, and the once-timer row present; `<host>.env` byte-identical to the source's; the env pin digests equal; `data-ie` holds the `moved:<id>` tombstone and an `ie-box` wake attempt starts nothing; the source's local state is in its park dir; the record is in `uk` with `last_move.outcome == done`. |
| +- **Rollback.** A second instance with `MOVEPROBE_FAIL_READY_IN=uk`: `start`, `seal`, `flip`; the landing reports `probe` failed; `abort`, ticks, `finish`. Assert: the record is home again, `ie-box` serves it and `/report` shows every row it had before the move and none written in `uk`, the tombstone is gone, `uk-box` has no row and its probe's local state is parked, both staging prefixes in `data-uk` are empty. |
| + |
| +The one permitted seam: if root's held-stage relay cannot run in-process, the test's `uk-box` stages the release through `release.MaterializeHeld` directly (its `OptDir` is writable), and Task 5's authority tests carry the root half. Say so in the test's comment. |
| + |
| +- [ ] Write the test; run it alone: `go test ./internal/host -run TestOfflineMoveEndToEnd -count=1 -v`. |
| +- [ ] Watch it fail: break each of `files`, `env`, `config` and the tombstone in turn (in a scratch copy of the tree under `$TMPDIR`, never in the worktree) and confirm the test fails for each; record the four results in the commit message. |
| +- [ ] Commit: `offline-move: end-to-end test over two in-process regions`. |
| + |
| +### Gate and hand-off |
| + |
| +- [ ] The full local gate once (Global Constraints). Push to amadan, `amadan branch describe`, wait for CI. |
| +- [ ] Opus reviews of Tasks 2, 4 and 5 as they land; one whole-branch review; fixes; squash; re-merge `main`; CI green on the tip. |
| +- [ ] Paul's copy review of MV-O1 to MV-O28, the reused fills and the docs paragraph; apply verbatim; CI green again. |
| +- [ ] Merge with full arguments. Read `main`'s CI after. |
| + |
| +--- |
| + |
| +## Morning runbook outline |
| + |
| +Every step waits for Paul's go. Nothing touches production before P2. |
| + |
| +**P0. Before anything moves.** |
| + |
| +1. Release: cut a release from the merged `main` (README recipe), roll dev boxes, then the dev console, check dev, then flagship boxes, then the flagship console (dev first, always). |
| +2. Confirm on every box and both consoles: `CARLOS_HOST_CLAIM` and `CARLOS_MOVES_PHASE1` unset; `carlos system status` shows every box on the release; each box's edges file carries `offline-move-1`; UK's `secrets_key_fp` equals the console's; `carlos-stage.path` armed on the UK box (the held release relies on root's stage pass). |
| +3. IAM: IE's `sibling_staging_arns` includes the UK data bucket (it does in carloku-infrastructure as read). Recommend applying UK's grant to the IE data bucket now (M17 approved the return lane; the apply is a production infra change and needs Paul's scoped approval), so that a move back after `open` is possible. |
| +4. Access: Paul's login is an operator and a member of the customer's account (the doors check both). |
| +5. For each of the three instances: layout (`carlos status --layout`: flat or nested), database and files size (under 256 MiB of files), no sidecar, no open run, not seeded, not a replica, not steered. Tell the customer the window. |
| +6. Stale edges files: any file naming the home placement for a box that no longer exists blocks `seal`; remove it only with Paul's approval. |
| + |
| +**P1. Dev rehearsal** (dev: sophie-knitting and leo-climbing are home, dennis-stenciling is `uk`; dennis holds the secrets key since 2026-10-15). |
| + |
| +1. Deploy moveprobe per `cmd/moveprobe/README.md` with a store, a secret and a schedule; create an instance in home; wake it on sophie only so leo holds no local state (MV-O26 otherwise); arm a once-timer. |
| +2. Offline move home to `uk`, each verb by hand, reading `status` between. Check `/report` on dennis: rows contiguous, key unchanged, store object, secret, timers. Note the offline seconds. |
| +3. Rollback rehearsal: a second instance with `MOVEPROBE_FAIL_READY_IN=uk`; `start`, `seal`, `flip`, see the probe fail, `abort`, `finish`; `/report` on sophie shows nothing lost. |
| +4. If dev's boxes are nested, step 2 is also the nested-archive proof. If it fails on file permissions, stop: the flagship plan changes (Risk 1). |
| + |
| +**P2. The three flagship moves**, one at a time, each with its own go. |
| + |
| +1. `carlos move offline start --host <h> --to uk`; `status` until every IE box reported; `seal`; `status` until sealed; `flip`; `status` until landed; `open`. |
| +2. Check: `curl -sI --resolve` the instance's URL against the UK EIP and against the IE EIP (backhaul), status 2xx and the expected `X-Carlos-Version`; the customer's own sign-in works. |
| +3. `finish`. Record the offline seconds and anything odd. Then the next instance. |
| +4. Abort criteria: any failed item, a probe failure, or a check in step 2 failing before `finish`; `abort` before `open` is always safe. |
| + |
| +**P3. After `open`, if something is wrong.** The target took writes, so going back is a new offline move from `uk` to the home, which needs UK's staging grant (P0.3). Its landing on the IE box lifts this move's tombstone and parks nothing (the source half parked at `open`). If the grant is not in place: fix forward on the UK box, and file the gap. |
| + |
| +## Interaction with the host claim |
| + |
| +The offline move refuses to run with the claim on, at every layer: the doors refuse (MV-O20), a box neither advertises `offline-move-1` nor acts with `CARLOS_HOST_CLAIM` on, and root's offline authority refuses on a claim box. It does not compose with the claim and is not meant to: once Phase 1 is live, the offline move is the claim-off fallback or is retired. What it leaves behind that a claim activation will meet: |
| + |
| +- `moved:<id>` tombstones at `instances/<host>/lock.json` in the source region's data bucket, with a one-year expiry. |
| +- A stale replica lineage for each moved host in the source bucket, and the live one in the target's own `instances/<host>/` with no generation markers (an origin generation, as for any host that never moved). |
| +- Parked local state under `<DataDir>/.offline-moves/` on source and target boxes. |
| +- Records with `last_move` and no `move`. |
| + |
| +**Questions for platform-1e (claim activation owner):** |
| + |
| +1. Should claim activation's preflight refuse while any record carries `move.kind: offline`? (This plan assumes yes, and asks for it.) |
| +2. Does the ownership backfill (#232) or the adoption of a legacy host read `lock.json` in any bucket other than the record's region's, and how should it treat a `moved:` owner with a far expiry: ignore it, or retire it as part of adoption? |
| +3. Does adoption consult only the record's region's bucket for a host's lineage, so a stale source lineage is never selected? |
| +4. Should the claim census know `.offline-moves/` (this plan adds it to `layout/extras.go`'s known names)? |
| +5. After activation, should the offline doors stay (refusing) or be removed, and should `heldStageAuthority`'s offline branch be deleted at the same time? |
| + |
| +## Non-goals |
| + |
| +Self-serve and member access; automatic recovery or automatic retries of the probe; moving object store data (the bucket stays in its region; residency follows the instance's database only, F4); multi-box target regions; sidecars, seeds, runs, replicas, pools, fleets, steered, unit-backed and non-hibernating instances; rollback after `open` (a new move); timer run history; the restore-verify verdict; DNS (the flagship's instance hosts resolve through the wildcard and backhaul); live migration; Tito. |
| + |
| +## Risks |
| + |
| +- **Nested sources.** If the flagship's rows are nested, the agent reads the tenant's files from the instance directory to archive them. A file the agent cannot read fails the seal (safe), but would block the move. |
| +- **The held release on a hardened box** depends on root's stage pass being armed and on the new authority branch; it is the one root-side change and has never run claim-off in production. |
| +- **Tombstone side effects.** Anything that reads `lock.json` and assumes a 15-minute lease (the housekeeper, the restore drill, the db door, a future claim adoption) meets a one-year `moved:` owner. A stale box that hits it answers 503 with a very long `Retry-After`. |
| +- **Timers.** Once-timers carry; run history does not, so a schedule may fire once more or once less around the window. |
| +- **The keep view's startup.** A fresh agent removes no rows until its first moves pass has built the view. |
| +- **Budget.** Eight tasks in eight hours with reviews leaves no slack; Task 8 is the long pole and has no precedent here (no existing test runs a real `ExecBackend` wake end to end). |
| + |
| +## The three things I am least sure of |
| + |
| +1. **Whether nested rows can be archived by the agent** (file ownership under isolated exec), and whether the flagship's three instances are nested at all. The dev rehearsal settles the first; `carlos status --layout` the second. |
| +2. **Whether root's held-stage relay works claim-off for a held row on the UK box** (stage-requested marker, `carlos-stage.path`, D3 proof gate, §4.1 authorization of the source's version), since every prior use of that relay was claim-mode Phase 1. |
| +3. **Whether Task 8 fits the night**: a real `ExecBackend` running moveprobe with real litestream against `leasetest`'s S3 server, plus a signed release, has not been assembled in this repo before. The seam named in Task 8 is the fallback; dropping the end-to-end test is not. |