amadan
spec: replica ownership (#199) and carlos db push, rev 1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+399
−0
docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md+399 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md b/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md| new file mode 100644 |
| index 000000000..637f34d4a |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md |
| @@ -0,0 +1,399 @@ |
| +# Replica ownership (#199) and `carlos db push` |
| + |
| +Design, 2026-10-02. Part A closes amadan #199. Part B rebuilds `carlos db |
| +push` from `feat/db-push-pull` (Tony Wood, b014d57d) on top of Part A, |
| +fixing the review in #194. Part B depends on Part A; Part A ships alone. |
| + |
| +Rulings (Paul, 2026-10-02), not to reopen: |
| + |
| +- **R1.** A recreated host whose replica belongs to a deleted earlier |
| + incarnation gets that lineage **archived** in-region under |
| + `parked/<old-incarnation>/` and starts empty. If the earlier |
| + incarnation's record still exists, the wake is **refused** (two live |
| + owners). |
| +- **R2.** A replica or local database with no stamp is **adopted** by the |
| + record that owns the host today. |
| +- **R3.** A pushed database and its pre-push backups live in the |
| + instance's **own region's data bucket**. Push is in-region only, like |
| + pull. |
| + |
| +## 1. The problem |
| + |
| +Storage identity is the host, alone: |
| + |
| +- the replica: `activator.HibernatePrefix(prefix, host)` = |
| + `instances/<host>` (activator.go:138); |
| +- the local database: `DefaultDBPath` / `NestedDBPath` (provision.go:1219, |
| + 1277); |
| +- the restore-drill verdict: `dbverify`, keyed by host. |
| + |
| +Deleting an instance (`DeleteRecord`, provision.go:1071) removes the |
| +record only. Reconcile then drops the registry row (reconcile.go:1155) but |
| +not the local file. Nothing ever deletes the replica (no production caller |
| +of `DeleteAll`/`DeletePrefix` on `instances/`). Nothing on the wake path |
| +checks who a replica or local file belongs to. |
| + |
| +So whoever next creates the same host inherits the old database: on first |
| +wake (`ExecBackend.Restore` returns the warm local file at |
| +backend_exec.go:267, or restores the replica), and through `carlos db |
| +pull` without any wake at all (apidb.go:70). |
| + |
| +Who can recreate a host: |
| + |
| +- **The same account, commonly.** Every app in an account shares the |
| + `<label>.<sqid>.<AppsDomain>` namespace (apiinstances.go:693). titogo's |
| + signup creates and deletes instances per customer slug, so a reused slug |
| + inherits the previous customer's data. This is the realistic case. |
| +- **Another account, less commonly.** Store-mode `carlos instances enable |
| + --domain` (instances_cli.go:135) and operator `carlos add` check host |
| + shape only. Account sqids are never reused (ownership.go:650), so a |
| + platform-domain host cannot cross accounts. |
| +- **Sidecars**: `<name>.<app>.<sqid>.sidecar.internal`, recreated after a |
| + sidecar delete or an app purge and a same-named new app. |
| + |
| +A related wrinkle: if a host is deleted and recreated between two |
| +reconcile passes, the old registry row survives (reconcile.go:1141) and a |
| +running child keeps serving the old data under the new record. |
| + |
| +## 2. Part A: incarnation identity and the stamp |
| + |
| +### A1. Incarnation |
| + |
| +An instance's **incarnation** is |
| + |
| + maintenanceguard.RecordIdentity(rec.Account, rec.App, rec.Host, rec.CreatedAt) |
| + |
| +Account, App, Host and CreatedAt are immutable under every record update |
| +(`protectedFieldsEqual`, maintenance.go:81), and CreatedAt is set at create |
| +(apiinstances.go:888, domainattach.go:183, replica.go:60). Sidecar records |
| +use the same function over their own fields. |
| + |
| +A record with a zero CreatedAt (written before CreatedAt existed) gets the |
| +fixed sentinel incarnation `legacy:<sha256(account\0app\0host)>`. A later |
| +recreate always has a real CreatedAt, so its incarnation differs from the |
| +sentinel and the check below still fires. |
| + |
| +### A2. Where it is recorded |
| + |
| +- **Registry row:** a new column `routes.incarnation`, added by an |
| + idempotent `ALTER TABLE` at start, like `routes.replicas`. `routeFor` |
| + renders it from the record and `convergeOwned` converges it in both |
| + directions. It is **not** in `Fingerprint()`. |
| +- **Replica stamp:** `<prefix>/owner.json`: |
| + |
| + {"v":1,"incarnation":"…","account":"bab","app":"jam", |
| + "host":"…","created_at":"…","stamped_at":"…","box":"<machine id>"} |
| + |
| + litestream lists only `<prefix>/%04x/` per level and skips non-LTX |
| + names, so the stamp is invisible to replication, restore, retention and |
| + the db door's coordinate listing. `MirrorPrefix` copies it (a move |
| + carries the stamp to the target), and `DeletePrefix` deletes it. |
| +- **Local stamp:** `<dbPath>.owner`, the same JSON, written beside the |
| + local database whenever a restore or seed completes for an incarnation. |
| + `PurgeLocalDBArtifacts(full)` removes it with the database. |
| + |
| +### A3. An incarnation change on a live row |
| + |
| +When `convergeOwned` sees a row's incarnation change (the delete-and- |
| +recreate-between-passes window), it treats this like a retire: stop the |
| +running child, fence it, then converge. The wake gate (A4) then deals |
| +with the old local file and replica. A process never keeps serving one |
| +incarnation's data under another's record. |
| + |
| +### A4. The wake gate |
| + |
| +`doWake` order today is: registry refresh, Classifier refusal (activator.go |
| +:1090, the precedent for a pre-lease gate), `Acquire` (writes `lock.json`), |
| +`Restore`, `Seed`. The gate lives inside `ExecBackend.Restore`, after the |
| +lease is held, in three steps. |
| + |
| +**Step 1, before the warm-cache return (before backend_exec.go:267).** If |
| +the local database exists, read `<dbPath>.owner`: |
| + |
| +- equal to the route's incarnation: warm path, as today; |
| +- absent: **adopt** (R2), writing the stamp, then the warm path; |
| +- different: the local file belongs to an earlier incarnation. If the |
| + replica stamp (read now) names that same earlier incarnation, replicate |
| + the local tail into it under the held lease first, so the archive in |
| + step 3 is complete. Then purge the local file with the error-returning |
| + purge (B6) and fall through to the cold path. If the replica belongs to |
| + neither incarnation, move the local file aside to |
| + `<DataDir>/.parked/<old-incarnation>/` (never deleted automatically, |
| + logged loudly) and fall through. |
| + |
| +**Step 2, move staging.** `promoteStagedReplica` runs as today. Staging |
| +carries the source's `owner.json`. If the staged stamp differs from the |
| +route's incarnation, refuse the wake (`ErrOwnerConflict`). A move never |
| +changes incarnation. |
| + |
| +**Step 3, the replica stamp.** Read `<prefix>/owner.json`: |
| + |
| +- absent and the prefix holds no LTX: a new instance. Put the stamp |
| + create-only (`If-None-Match: *`), then restore finds nothing and Seed |
| + runs. |
| +- absent and the prefix holds LTX: **adopt** (R2). Put the stamp |
| + create-only. |
| +- equal: restore. |
| +- different: an earlier incarnation's lineage. |
| + - Look up the earlier incarnation's record: |
| + `RecordKey(stamp.account, stamp.app, host)`. If it exists and its |
| + CreatedAt equals the stamp's, **refuse** the wake (`ErrOwnerConflict`, |
| + R1). |
| + - Otherwise **archive** (R1). `MirrorPrefix` the prefix to |
| + `parked/<old-incarnation>/<host>/`, skipping `lock.json`, then verify |
| + that every source key exists at the destination with the same ETag. |
| + Write `parked/<old-incarnation>/<host>/parked.json` (who, when, from |
| + which incarnation). Delete every key under the prefix except |
| + `lock.json`. Put the new stamp. Restore then finds nothing, and Seed |
| + runs. |
| + |
| +A lookup error, a failed verify or a failed put refuses the wake with a |
| +retryable error. The gate never restores or seeds on uncertainty. |
| + |
| +`ExecBackend` has no deployment store today. The gate takes an injected |
| +`OwnerLive(account, app, host string, created time.Time) (bool, error)`, |
| +which `agent.go` wires from the release store. |
| + |
| +### A5. The db door |
| + |
| +`resolveReplica` (apidb.go:70) reads `owner.json` before serving |
| +anything: |
| + |
| +- different from the record's incarnation: refuse, as "nothing replicated |
| + for this instance yet" (the same answer an empty replica gets); |
| +- absent: allow (legacy; the box adopts it on the next wake or sweep); |
| +- equal: serve. |
| + |
| +### A6. The adoption sweep |
| + |
| +A legacy instance that never wakes would stay unstamped, and readable |
| +through the door by a reused host, indefinitely. Once per agent start, the |
| +box walks its owned rows. For each prefix that holds LTX but no stamp, it |
| +puts the stamp create-only with the row's incarnation. That is one HEAD |
| +per route and a PUT where needed. A stamp that already exists is left |
| +alone. |
| + |
| +### A7. IAM and infra |
| + |
| +The box role needs `s3:PutObject`/`GetObject`/`DeleteObject`/`ListBucket` |
| +on `parked/*` in its own data bucket (infra/modules/region-host/main.tf |
| +:160-226 grants `instances/*` today). The console role needs `GetObject` |
| +on `instances/*/owner.json`, which it already has through the pull door's |
| +`instances/*` grant. The data bucket is versioned with no lifecycle rules, |
| +so deletes leave noncurrent versions (a safety net, and a storage cost). |
| + |
| +### A8. Out of scope, recorded |
| + |
| +- A completed move leaves the source region's `instances/<host>/` behind |
| + (movespass.go never deletes it; the comment at backend_exec.go:285 is |
| + wrong). The stamp makes it harmless. Cleaning it up is #412's "list what |
| + deleted instances left behind". |
| +- `dbverify` verdicts are keyed by host, so a reused host inherits the |
| + previous verdict. Key them by incarnation later. |
| +- Listing and purging `parked/` is operator tooling, not part of this |
| + change. |
| + |
| +## 3. Part B: `carlos db push` |
| + |
| +### B1. Scope |
| + |
| +The same gate as the pull door: |
| + |
| +- members only, with the shared 403 for every service principal; |
| +- in-region, non-pooled, activator-managed exec instances only; |
| +- refused for a `role: replica` record; |
| +- refused while a move is in flight, and a move is refused while a push |
| + is in flight; |
| +- typed confirmation: the `X-Carlos-Confirm-Host` header must name the |
| + host. |
| + |
| +### B2. The request record |
| + |
| +This reuses Tony's `provision.DBTransfer` and its CAS transitions |
| +(dbtransfer.go), with three changes: |
| + |
| +- the pull op is removed (pull is the read door); |
| +- a new Step, `replacing`, sits between `backed-up` and `replaced` (B4.7); |
| +- `Incarnation` is copied onto the request at request time, and every |
| + box-side transition checks it. |
| + |
| +### B3. Upload, in-region (R3) |
| + |
| +- The console streams the body straight to the data bucket, at |
| + `db-uploads/<incarnation>/<id>.db`. It never buffers the whole file. |
| +- **Cap:** 1 GiB to start. The cap is enforced by counting bytes as they |
| + stream. |
| +- **Header check:** the first 100 bytes must be a SQLite header, checked |
| + while streaming. A bad header aborts the upload. |
| +- **IAM:** the console gets a narrow grant, `PutObject` and `DeleteObject` |
| + on `db-uploads/*` in its own region's bucket. |
| +- **Request write, and an ambiguous outcome (#194 P1-4):** |
| + - After the upload completes, the console writes the request with CAS. |
| + - If the write returns an error other than `ErrDBConflict`, the console |
| + re-reads the record. |
| + - If the record carries the request id, the write landed: answer 202. |
| + - If the record definitely lacks it, delete the upload. |
| + - If the re-read itself fails, leave the upload in place. B4.13's sweep |
| + removes it. |
| + |
| +### B4. The box worker |
| + |
| +This runs from the shared pass (B5). Steps run in this order, and each |
| +resumes from the record's Step. |
| + |
| +1. **Hold the route.** Set its status to `db-push`. This word is distinct |
| + from `maintenance`, which the maintenance feature owns |
| + (provision/maintenance.go). Everything that serves or wakes requires |
| + `ready`. The status persists in the registry across restarts. |
| +2. **Fence and quiesce**, as today. |
| +3. **Take the lease, once, for the whole swap (#194 P1-1).** |
| + - Call `lease.Acquire` on the prefix and keep the `Held` until step 12. |
| + - If another holder has it, return and retry next pass with the route |
| + still held. |
| + - Every write below goes through this `Held`. Nothing calls |
| + `ReplicateWithLease`, which takes its own lease. |
| +4. **Check ownership.** The replica stamp, the route's incarnation and |
| + the request's incarnation must all agree. If any differ, fail the |
| + request and release the route. Nothing has changed yet at this point. |
| +5. **Resolve move staging (#194 P1-2).** |
| + - If `moves/<host>/` is non-empty, run the activator's promotion under |
| + the held lease (B7), then drop the staging. |
| + - Do this before the backup, so the backup is of what a wake would |
| + actually restore. |
| +6. **Checkpoint the local tail.** If the local database exists and its |
| + stamp matches, replicate its tail under the held lease and wait for |
| + sync. |
| +7. **Back up (server side).** |
| + - `MirrorPrefix` the prefix to `db-backups/<incarnation>/<backup-id>/`, |
| + skipping `lock.json`. |
| + - Verify every key and ETag at the destination. |
| + - Record the backup id with Step `backed-up`. |
| + - Nothing is restored on the box, so the size cap and the scratch-disk |
| + problem (#194 P2-6) do not apply to the backup. |
| +8. **Download the upload to scratch.** |
| + - First check free disk is at least twice the upload size. If not, |
| + the push fails before anything changes. |
| + - Run `CheckDB` on the downloaded file. A bad file fails the push |
| + before anything changes. |
| +9. **Mark the point of no return.** Write Step `replacing` to the record |
| + **before** any delete. |
| +10. **Clear the old lineage.** Delete every key under the prefix except |
| + `lock.json` and `owner.json`. |
| +11. **Replicate the upload** from scratch under the held lease, then wait |
| + for sync. |
| +12. **Purge the local database** with the error-returning purge (B6). |
| + - Verify the database and its sidecars are gone, and remove the local |
| + stamp. |
| + - Then write Step `replaced` and release the lease. |
| +13. **Finish.** |
| + - Delete the upload and prune backups to the newest five. |
| + - Set the route's status back to `ready`. **A failure here returns an |
| + error and keeps the request in flight (#194 P2-5).** Only after |
| + that succeeds: lift the fence and mark the request done. |
| + - Upload objects that no in-flight request names and that are older |
| + than an hour are deleted here too. |
| + |
| +**Resuming from Step `replacing`:** |
| + |
| +- Redo steps 8 to 12. The delete is idempotent, and the replicate starts |
| + the same fresh lineage. |
| +- **If the upload is missing at `replacing`**, never mark the push failed |
| + and never release the route: the replica may be empty. Keep the route |
| + held, log loudly, and raise an alert. An operator restores the backup |
| + (B7). |
| + |
| +**After a restart**, the worker lifts any route left held for a request |
| +that is no longer in flight (#194 P2-5). It finds those routes by their |
| +`db-push` status, which is durable in the registry, not by in-memory |
| +fence state. For each one it sets `ready` and lifts the fence. |
| + |
| +### B5. One record listing per pass (#194 P2-7) |
| + |
| +The moves pass and the db pass share one `ListAllRecords` per converge |
| +pass, folded once and handed to both. |
| + |
| +### B6. Purge with errors (#194 P1-3) |
| + |
| +Add `activator.PurgeLocalDBArtifactsErr(dbPath, full) error`, which |
| +returns the first removal error other than not-exist, and its stamp |
| +removal. The push and the A4 gate use it. The existing best-effort |
| +function stays for its current callers. |
| + |
| +### B7. Undo |
| + |
| +- `carlos db backups --instance <h>` lists the backups under |
| + `db-backups/<incarnation>/`. |
| +- `carlos db pull --instance <h> --backup <id> --out <path>`: |
| + - the pull door serves a backup's LTX coordinates by `?backup=<id>`; |
| + - the backup id is validated, and nothing outside |
| + `db-backups/<incarnation>/<id>/` is reachable; |
| + - the CLI restores it read-only, exactly as a normal pull. |
| +- To put a backup back, pull it, then push it. The push takes its own |
| + backup first. |
| + |
| +The activator's promotion logic (`promoteStagedReplica`) is factored so |
| +step B4.5 and the wake share one implementation. |
| + |
| +### B8. CLI |
| + |
| +`carlos db push --instance <h> <file>`, with these rules: |
| + |
| +- `--instance` is required for push and is never inferred. Replacing the |
| + wrong instance's database is the failure to avoid. |
| +- An interactive confirmation names the host. |
| +- `--yes` skips the prompt but still requires `--instance`. |
| + |
| +`carlos db status --instance <h>` shows the request, its step and any |
| +error. All user-facing strings go through copy review. |
| + |
| +### B9. Tests that pin the review's findings |
| + |
| +Each is a test: |
| + |
| +- two workers on one prefix: the second cannot take the lease, so it |
| + neither deletes nor replicates (P1-1); |
| +- a push with move staging present backs up and replaces what a wake |
| + would restore, and the next wake does not resurrect staging (P1-2); |
| +- a failing purge keeps the route held and the request in flight (P1-3); |
| +- a CAS write whose response is lost (the record has the id) keeps the |
| + upload, and the push completes (P1-4); |
| +- an upload missing at `replacing` keeps the route held and never seeds |
| + (P1-4); |
| +- a `SetStatus("ready")` failure keeps the request in flight, and a later |
| + pass completes it (P2-5); |
| +- restart with a cleared record and a `db-push` route: released (P2-5); |
| +- an upload over the cap is refused while streaming; insufficient disk |
| + fails before step 9 (P2-6); |
| +- one `ListAllRecords` per pass (P2-7). |
| + |
| +Part A's own tests: |
| + |
| +- a recreated host (same account, other app, other account, sidecar) |
| + never sees the old data: |
| + - not through the warm path; |
| + - not through restore; |
| + - not through the door; |
| +- archive happens only when the old record is gone, and its ETags are |
| + verified before the delete; |
| +- the two-live-owners case refuses; |
| +- legacy adoption, by the sweep and by a wake; |
| +- a move carries the stamp, and a staged stamp that disagrees refuses; |
| +- the incarnation change on a live row stops the child. |
| + |
| +## 4. Rollout |
| + |
| +1. **Part A, IAM first:** the `parked/*` grant goes into the region-host |
| + module. Dev tracks main. For flagship, the change rides the next |
| + release tag through carloku-infrastructure. Target the change; never |
| + apply the whole root (user_data restarts every box). |
| +2. **Part A, boxes:** the stamp, the gate and the sweep. The console door |
| + check (A5) can roll at any time: an absent stamp is allowed. |
| +3. **Part B:** |
| + - IAM: the console's `db-uploads/*` grant and the box's `db-uploads/*` |
| + and `db-backups/*` grants. |
| + - Roll boxes before the console. The console exposes push, and a box |
| + without the worker would leave requests queued forever. |
| + - CLI release, and copy review of every string. |
| + |
| +Credit: Part B reuses Tony Wood's request record, console door and CLI |
| +shape from `feat/db-push-pull`. |