amadan
spec: replica ownership and db push, rev 2 (storage keyed by incarnation and generation)
Rev 1 failed Astra review (16 P1s, one cause: host-keyed storage). Rev 2 keys replicas and local state by incarnation and generation; push becomes replicate-into-a-fresh-generation then a guarded pointer CAS. Folds in the moves session's notes: per-host working directory, receipts by generation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+385
−386
docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md+385 −386
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md b/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md| index 637f34d4a..bba43870c 100644 |
| --- a/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md |
| +++ b/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md |
| @@ -1,399 +1,398 @@ |
| # Replica ownership (#199) and `carlos db push` |
| -Design, 2026-10-02. Part A closes amadan #199. Part B rebuilds `carlos db |
| -push` from `feat/db-push-pull` (Tony Wood, b014d57d) on top of Part A, |
| -fixing the review in #194. Part B depends on Part A; Part A ships alone. |
| +Design, 2026-10-02, **rev 2**. Part A closes amadan #199. Part B rebuilds |
| +`carlos db push` from `feat/db-push-pull` (Tony Wood, b014d57d), fixing |
| +the review in #194. Rev 1 (stamps on host-keyed storage) failed Astra's |
| +review with 16 P1s, all from one cause: data was still keyed by host, so |
| +every path that touched the shared location needed its own ownership |
| +check. Rev 2 removes the cause instead: **data is keyed by incarnation and |
| +generation, and the instance record says which generation is current.** |
| +Rev 1's findings are answered in §6. |
| Rulings (Paul, 2026-10-02), not to reopen: |
| -- **R1.** A recreated host whose replica belongs to a deleted earlier |
| - incarnation gets that lineage **archived** in-region under |
| - `parked/<old-incarnation>/` and starts empty. If the earlier |
| - incarnation's record still exists, the wake is **refused** (two live |
| - owners). |
| -- **R2.** A replica or local database with no stamp is **adopted** by the |
| - record that owns the host today. |
| -- **R3.** A pushed database and its pre-push backups live in the |
| - instance's **own region's data bucket**. Push is in-region only, like |
| - pull. |
| +- **R1.** A recreated host never sees an earlier incarnation's data; that |
| + data is kept in-region and the new instance starts empty. |
| +- **R2.** Data that exists before this change is adopted by the record |
| + that owns the host today. |
| +- **R3.** A pushed database and its backups live in the instance's own |
| + region. Push is in-region only, like pull. |
| +- **R4.** Storage is rekeyed by incarnation and generation (rev 2's |
| + approach), rather than stamping host-keyed storage (rev 1). |
| ## 1. The problem |
| -Storage identity is the host, alone: |
| - |
| -- the replica: `activator.HibernatePrefix(prefix, host)` = |
| - `instances/<host>` (activator.go:138); |
| -- the local database: `DefaultDBPath` / `NestedDBPath` (provision.go:1219, |
| - 1277); |
| -- the restore-drill verdict: `dbverify`, keyed by host. |
| - |
| -Deleting an instance (`DeleteRecord`, provision.go:1071) removes the |
| -record only. Reconcile then drops the registry row (reconcile.go:1155) but |
| -not the local file. Nothing ever deletes the replica (no production caller |
| -of `DeleteAll`/`DeletePrefix` on `instances/`). Nothing on the wake path |
| -checks who a replica or local file belongs to. |
| - |
| -So whoever next creates the same host inherits the old database: on first |
| -wake (`ExecBackend.Restore` returns the warm local file at |
| -backend_exec.go:267, or restores the replica), and through `carlos db |
| -pull` without any wake at all (apidb.go:70). |
| - |
| -Who can recreate a host: |
| - |
| -- **The same account, commonly.** Every app in an account shares the |
| - `<label>.<sqid>.<AppsDomain>` namespace (apiinstances.go:693). titogo's |
| - signup creates and deletes instances per customer slug, so a reused slug |
| - inherits the previous customer's data. This is the realistic case. |
| -- **Another account, less commonly.** Store-mode `carlos instances enable |
| - --domain` (instances_cli.go:135) and operator `carlos add` check host |
| - shape only. Account sqids are never reused (ownership.go:650), so a |
| - platform-domain host cannot cross accounts. |
| -- **Sidecars**: `<name>.<app>.<sqid>.sidecar.internal`, recreated after a |
| - sidecar delete or an app purge and a same-named new app. |
| - |
| -A related wrinkle: if a host is deleted and recreated between two |
| -reconcile passes, the old registry row survives (reconcile.go:1141) and a |
| -running child keeps serving the old data under the new record. |
| - |
| -## 2. Part A: incarnation identity and the stamp |
| - |
| -### A1. Incarnation |
| - |
| -An instance's **incarnation** is |
| - |
| - maintenanceguard.RecordIdentity(rec.Account, rec.App, rec.Host, rec.CreatedAt) |
| - |
| -Account, App, Host and CreatedAt are immutable under every record update |
| -(`protectedFieldsEqual`, maintenance.go:81), and CreatedAt is set at create |
| -(apiinstances.go:888, domainattach.go:183, replica.go:60). Sidecar records |
| -use the same function over their own fields. |
| - |
| -A record with a zero CreatedAt (written before CreatedAt existed) gets the |
| -fixed sentinel incarnation `legacy:<sha256(account\0app\0host)>`. A later |
| -recreate always has a real CreatedAt, so its incarnation differs from the |
| -sentinel and the check below still fires. |
| - |
| -### A2. Where it is recorded |
| - |
| -- **Registry row:** a new column `routes.incarnation`, added by an |
| - idempotent `ALTER TABLE` at start, like `routes.replicas`. `routeFor` |
| - renders it from the record and `convergeOwned` converges it in both |
| - directions. It is **not** in `Fingerprint()`. |
| -- **Replica stamp:** `<prefix>/owner.json`: |
| - |
| - {"v":1,"incarnation":"…","account":"bab","app":"jam", |
| - "host":"…","created_at":"…","stamped_at":"…","box":"<machine id>"} |
| - |
| - litestream lists only `<prefix>/%04x/` per level and skips non-LTX |
| - names, so the stamp is invisible to replication, restore, retention and |
| - the db door's coordinate listing. `MirrorPrefix` copies it (a move |
| - carries the stamp to the target), and `DeletePrefix` deletes it. |
| -- **Local stamp:** `<dbPath>.owner`, the same JSON, written beside the |
| - local database whenever a restore or seed completes for an incarnation. |
| - `PurgeLocalDBArtifacts(full)` removes it with the database. |
| - |
| -### A3. An incarnation change on a live row |
| - |
| -When `convergeOwned` sees a row's incarnation change (the delete-and- |
| -recreate-between-passes window), it treats this like a retire: stop the |
| -running child, fence it, then converge. The wake gate (A4) then deals |
| -with the old local file and replica. A process never keeps serving one |
| -incarnation's data under another's record. |
| - |
| -### A4. The wake gate |
| - |
| -`doWake` order today is: registry refresh, Classifier refusal (activator.go |
| -:1090, the precedent for a pre-lease gate), `Acquire` (writes `lock.json`), |
| -`Restore`, `Seed`. The gate lives inside `ExecBackend.Restore`, after the |
| -lease is held, in three steps. |
| - |
| -**Step 1, before the warm-cache return (before backend_exec.go:267).** If |
| -the local database exists, read `<dbPath>.owner`: |
| - |
| -- equal to the route's incarnation: warm path, as today; |
| -- absent: **adopt** (R2), writing the stamp, then the warm path; |
| -- different: the local file belongs to an earlier incarnation. If the |
| - replica stamp (read now) names that same earlier incarnation, replicate |
| - the local tail into it under the held lease first, so the archive in |
| - step 3 is complete. Then purge the local file with the error-returning |
| - purge (B6) and fall through to the cold path. If the replica belongs to |
| - neither incarnation, move the local file aside to |
| - `<DataDir>/.parked/<old-incarnation>/` (never deleted automatically, |
| - logged loudly) and fall through. |
| - |
| -**Step 2, move staging.** `promoteStagedReplica` runs as today. Staging |
| -carries the source's `owner.json`. If the staged stamp differs from the |
| -route's incarnation, refuse the wake (`ErrOwnerConflict`). A move never |
| -changes incarnation. |
| - |
| -**Step 3, the replica stamp.** Read `<prefix>/owner.json`: |
| - |
| -- absent and the prefix holds no LTX: a new instance. Put the stamp |
| - create-only (`If-None-Match: *`), then restore finds nothing and Seed |
| - runs. |
| -- absent and the prefix holds LTX: **adopt** (R2). Put the stamp |
| - create-only. |
| -- equal: restore. |
| -- different: an earlier incarnation's lineage. |
| - - Look up the earlier incarnation's record: |
| - `RecordKey(stamp.account, stamp.app, host)`. If it exists and its |
| - CreatedAt equals the stamp's, **refuse** the wake (`ErrOwnerConflict`, |
| - R1). |
| - - Otherwise **archive** (R1). `MirrorPrefix` the prefix to |
| - `parked/<old-incarnation>/<host>/`, skipping `lock.json`, then verify |
| - that every source key exists at the destination with the same ETag. |
| - Write `parked/<old-incarnation>/<host>/parked.json` (who, when, from |
| - which incarnation). Delete every key under the prefix except |
| - `lock.json`. Put the new stamp. Restore then finds nothing, and Seed |
| - runs. |
| - |
| -A lookup error, a failed verify or a failed put refuses the wake with a |
| -retryable error. The gate never restores or seeds on uncertainty. |
| - |
| -`ExecBackend` has no deployment store today. The gate takes an injected |
| -`OwnerLive(account, app, host string, created time.Time) (bool, error)`, |
| -which `agent.go` wires from the release store. |
| - |
| -### A5. The db door |
| - |
| -`resolveReplica` (apidb.go:70) reads `owner.json` before serving |
| -anything: |
| - |
| -- different from the record's incarnation: refuse, as "nothing replicated |
| - for this instance yet" (the same answer an empty replica gets); |
| -- absent: allow (legacy; the box adopts it on the next wake or sweep); |
| -- equal: serve. |
| - |
| -### A6. The adoption sweep |
| - |
| -A legacy instance that never wakes would stay unstamped, and readable |
| -through the door by a reused host, indefinitely. Once per agent start, the |
| -box walks its owned rows. For each prefix that holds LTX but no stamp, it |
| -puts the stamp create-only with the row's incarnation. That is one HEAD |
| -per route and a PUT where needed. A stamp that already exists is left |
| -alone. |
| - |
| -### A7. IAM and infra |
| - |
| -The box role needs `s3:PutObject`/`GetObject`/`DeleteObject`/`ListBucket` |
| -on `parked/*` in its own data bucket (infra/modules/region-host/main.tf |
| -:160-226 grants `instances/*` today). The console role needs `GetObject` |
| -on `instances/*/owner.json`, which it already has through the pull door's |
| -`instances/*` grant. The data bucket is versioned with no lifecycle rules, |
| -so deletes leave noncurrent versions (a safety net, and a storage cost). |
| - |
| -### A8. Out of scope, recorded |
| - |
| -- A completed move leaves the source region's `instances/<host>/` behind |
| - (movespass.go never deletes it; the comment at backend_exec.go:285 is |
| - wrong). The stamp makes it harmless. Cleaning it up is #412's "list what |
| - deleted instances left behind". |
| -- `dbverify` verdicts are keyed by host, so a reused host inherits the |
| - previous verdict. Key them by incarnation later. |
| -- Listing and purging `parked/` is operator tooling, not part of this |
| - change. |
| - |
| -## 3. Part B: `carlos db push` |
| - |
| -### B1. Scope |
| - |
| -The same gate as the pull door: |
| - |
| -- members only, with the shared 403 for every service principal; |
| -- in-region, non-pooled, activator-managed exec instances only; |
| -- refused for a `role: replica` record; |
| -- refused while a move is in flight, and a move is refused while a push |
| - is in flight; |
| -- typed confirmation: the `X-Carlos-Confirm-Host` header must name the |
| - host. |
| - |
| -### B2. The request record |
| - |
| -This reuses Tony's `provision.DBTransfer` and its CAS transitions |
| -(dbtransfer.go), with three changes: |
| - |
| -- the pull op is removed (pull is the read door); |
| -- a new Step, `replacing`, sits between `backed-up` and `replaced` (B4.7); |
| -- `Incarnation` is copied onto the request at request time, and every |
| - box-side transition checks it. |
| - |
| -### B3. Upload, in-region (R3) |
| - |
| -- The console streams the body straight to the data bucket, at |
| - `db-uploads/<incarnation>/<id>.db`. It never buffers the whole file. |
| -- **Cap:** 1 GiB to start. The cap is enforced by counting bytes as they |
| - stream. |
| -- **Header check:** the first 100 bytes must be a SQLite header, checked |
| - while streaming. A bad header aborts the upload. |
| -- **IAM:** the console gets a narrow grant, `PutObject` and `DeleteObject` |
| - on `db-uploads/*` in its own region's bucket. |
| -- **Request write, and an ambiguous outcome (#194 P1-4):** |
| - - After the upload completes, the console writes the request with CAS. |
| - - If the write returns an error other than `ErrDBConflict`, the console |
| - re-reads the record. |
| - - If the record carries the request id, the write landed: answer 202. |
| - - If the record definitely lacks it, delete the upload. |
| - - If the re-read itself fails, leave the upload in place. B4.13's sweep |
| - removes it. |
| - |
| -### B4. The box worker |
| - |
| -This runs from the shared pass (B5). Steps run in this order, and each |
| -resumes from the record's Step. |
| - |
| -1. **Hold the route.** Set its status to `db-push`. This word is distinct |
| - from `maintenance`, which the maintenance feature owns |
| - (provision/maintenance.go). Everything that serves or wakes requires |
| - `ready`. The status persists in the registry across restarts. |
| -2. **Fence and quiesce**, as today. |
| -3. **Take the lease, once, for the whole swap (#194 P1-1).** |
| - - Call `lease.Acquire` on the prefix and keep the `Held` until step 12. |
| - - If another holder has it, return and retry next pass with the route |
| - still held. |
| - - Every write below goes through this `Held`. Nothing calls |
| - `ReplicateWithLease`, which takes its own lease. |
| -4. **Check ownership.** The replica stamp, the route's incarnation and |
| - the request's incarnation must all agree. If any differ, fail the |
| - request and release the route. Nothing has changed yet at this point. |
| -5. **Resolve move staging (#194 P1-2).** |
| - - If `moves/<host>/` is non-empty, run the activator's promotion under |
| - the held lease (B7), then drop the staging. |
| - - Do this before the backup, so the backup is of what a wake would |
| - actually restore. |
| -6. **Checkpoint the local tail.** If the local database exists and its |
| - stamp matches, replicate its tail under the held lease and wait for |
| - sync. |
| -7. **Back up (server side).** |
| - - `MirrorPrefix` the prefix to `db-backups/<incarnation>/<backup-id>/`, |
| - skipping `lock.json`. |
| - - Verify every key and ETag at the destination. |
| - - Record the backup id with Step `backed-up`. |
| - - Nothing is restored on the box, so the size cap and the scratch-disk |
| - problem (#194 P2-6) do not apply to the backup. |
| -8. **Download the upload to scratch.** |
| - - First check free disk is at least twice the upload size. If not, |
| - the push fails before anything changes. |
| - - Run `CheckDB` on the downloaded file. A bad file fails the push |
| - before anything changes. |
| -9. **Mark the point of no return.** Write Step `replacing` to the record |
| - **before** any delete. |
| -10. **Clear the old lineage.** Delete every key under the prefix except |
| - `lock.json` and `owner.json`. |
| -11. **Replicate the upload** from scratch under the held lease, then wait |
| - for sync. |
| -12. **Purge the local database** with the error-returning purge (B6). |
| - - Verify the database and its sidecars are gone, and remove the local |
| - stamp. |
| - - Then write Step `replaced` and release the lease. |
| -13. **Finish.** |
| - - Delete the upload and prune backups to the newest five. |
| - - Set the route's status back to `ready`. **A failure here returns an |
| - error and keeps the request in flight (#194 P2-5).** Only after |
| - that succeeds: lift the fence and mark the request done. |
| - - Upload objects that no in-flight request names and that are older |
| - than an hour are deleted here too. |
| - |
| -**Resuming from Step `replacing`:** |
| - |
| -- Redo steps 8 to 12. The delete is idempotent, and the replicate starts |
| - the same fresh lineage. |
| -- **If the upload is missing at `replacing`**, never mark the push failed |
| - and never release the route: the replica may be empty. Keep the route |
| - held, log loudly, and raise an alert. An operator restores the backup |
| - (B7). |
| - |
| -**After a restart**, the worker lifts any route left held for a request |
| -that is no longer in flight (#194 P2-5). It finds those routes by their |
| -`db-push` status, which is durable in the registry, not by in-memory |
| -fence state. For each one it sets `ready` and lifts the fence. |
| - |
| -### B5. One record listing per pass (#194 P2-7) |
| +Storage identity is the host, alone: the replica at |
| +`activator.HibernatePrefix(prefix, host)` = `instances/<host>` |
| +(activator.go:138), the local database at `DefaultDBPath` / |
| +`NestedDBPath` (provision.go:1219, 1277). Deleting an instance removes the |
| +record only (provision.go:1071); nothing deletes the replica or the local |
| +file. Nothing checks ownership on wake or in the pull door. So whoever next |
| +creates the host inherits the data. The same account does this routinely: |
| +every app in an account shares `<label>.<sqid>.<AppsDomain>`, and titogo |
| +creates and deletes instances per customer slug. Sidecars |
| +(`<name>.<app>.<sqid>.sidecar.internal`) have the same exposure. |
| + |
| +Separately, a host deleted and recreated between two reconcile passes |
| +under a different app keeps its old registry row forever: `convergeOwned` |
| +runs only for a matching account and app (reconcile.go:947), the mismatch |
| +branch leaves the row alone (:1031), and removal skips it because the host |
| +still has a record (:1141). The old app keeps serving the new owner's |
| +host. Sidecars have the same shape (sidecar_reconcile.go:131, :467). |
| + |
| +## 2. The storage key |
| + |
| +### 2.1 Incarnation |
| + |
| + incarnation = RecordIdentity(account, app, host, createdAt) (hex, 64) |
| + |
| +`maintenanceguard.RecordIdentity` (types.go:62), factored into one shared |
| +`provision.Incarnation(rec)` for instance and sidecar records. Account, |
| +App, Host and CreatedAt are immutable under every update |
| +(`protectedFieldsEqual`, maintenance.go:81). `CreateRecord` and |
| +`CreateSidecarRecord` (provision.go:820, sidecar.go:322) now **refuse** a |
| +zero CreatedAt, so every record created from here on has an incarnation. A |
| +record with zero CreatedAt can only be legacy (§2.3). |
| + |
| +### 2.2 Generation |
| + |
| +A record gains one field: |
| + |
| + DB *DBPointer `json:"db,omitempty"` |
| + type DBPointer struct { |
| + Gen string `json:"gen"` // "g<seq>-<nonce>", e.g. "g2-7f3a91c0" |
| + Root string `json:"root"` // "instances" | "moves" |
| + } |
| + |
| +The nonce is the id of the request that made the generation (a push, a |
| +move) and is never reused, so two generations can never share a location |
| +even if a sequence number is replayed. `Root` exists so a moved instance |
| +can live where the move landed it without a copy (§4). |
| + |
| +### 2.3 Where a record's data lives |
| + |
| + v2: <Root>/<host>/<incarnation>/<Gen>/ (replica) |
| + legacy: instances/<host> (replica, as today) |
| + |
| +A record is **legacy** when `DB == nil` and its CreatedAt is before the |
| +deployment's **v2 epoch**. Otherwise it is v2. A v2 record with `DB == |
| +nil` uses `{Gen: "g0", Root: "instances"}`. |
| + |
| +The v2 epoch is one object, `control/storage-v2.json` `{"since": "<RFC3339>"}`, |
| +written create-only by an operator once every box and console runs a |
| +release that understands it (§7). Before it exists, **every record is |
| +legacy** and nothing changes. CreatedAt is immutable and every old writer |
| +preserves it, so no old binary can turn a v2 record legacy or the reverse. |
| + |
| +One function, `provision.StorageKey(rec, epoch) (StorageKey, error)`, |
| +returns the location. **Every reader and writer goes through it.** There |
| +are 17 call sites that build the prefix from the host today |
| +(activator.go ×3, backend_exec.go ×9, movespass.go ×3, apidb.go, |
| +hibernate_verify.go). The registry row carries the computed key (§2.4), so |
| +the activator never derives it. |
| + |
| +### 2.4 The registry row |
| + |
| +New columns, added by idempotent `ALTER TABLE` at start (like |
| +`routes.replicas`): `incarnation`, `storage_key` (the full replica |
| +prefix), `storage_gen`. `routeFor` and `sidecarRouteFor` render them from |
| +the record and the epoch; `convergeOwned` and `convergeSidecar` converge |
| +them in both directions. A change of `incarnation` or `storage_key` on a |
| +live row is a **retire**, not a column update (§2.6). |
| + |
| +### 2.5 The local database |
| + |
| +The tenant-visible `DBPath` does not change: the app's environment, and |
| +the isolated-exec nested layout, stay as they are. Beside it, the box |
| +keeps `<dbPath>.lineage`: |
| + |
| + {"incarnation":"…","storage_key":"…"} |
| + |
| +It is written by atomic rename **after** a restore or a seed succeeds and |
| +the first replication sync has completed, and it is removed **first**, |
| +before any purge. |
| + |
| +On wake (`ExecBackend.Restore`, before the warm-cache return at |
| +backend_exec.go:267): |
| + |
| +- **v2 row:** the warm local file is used only if `.lineage` equals the |
| + row's `incarnation` and `storage_key`. Anything else (absent, different, |
| + unreadable) purges the local file with the error-returning purge |
| + (§5.6), and the wake fails if the purge fails. Then it restores cold |
| + from `storage_key`. |
| +- **legacy row:** as today, and `.lineage` is written on first sight |
| + (R2). |
| + |
| +A database a fresh v2 instance creates in `Start` (activator.go:1168), |
| +before any `.lineage` exists, is being replicated into its own |
| +generation's location. After a crash it is restored from there, never |
| +adopted, so there is no "file before stamp" window. |
| + |
| +### 2.5b The instance's working directory |
| + |
| +The database is not the only host-keyed local state. A nested row's |
| +working directory, `InstanceDir(host)` (provision.go:1270, used as the |
| +app's data dir by activator/paths.go:37), holds whatever the app writes |
| +there, for example titogo's `instance.key` and `report.token`. A recreated |
| +host inherits these today (reported by the moves session, 2026-10-02). |
| + |
| +The two pieces of local state have different lifetimes: |
| + |
| +- **The working directory belongs to the incarnation.** It carries |
| + `.incarnation`, which is written when the directory is created and |
| + checked on every wake before the app starts. If it is absent on a v2 |
| + row, or different, the box moves the whole directory aside to |
| + `<DataDir>/.retired/<old-incarnation>/` and creates it fresh. Nothing is |
| + deleted automatically; operator purge is #412's tooling. A legacy row |
| + adopts (R2). |
| +- **The database belongs to the generation** (§2.5). A push or rollback |
| + replaces the database and keeps the directory, so `instance.key` |
| + survives a push. |
| + |
| +A flat row's app writes wherever it chooses under the shared `/data`, and |
| +the platform cannot attribute those files to an incarnation. Flat rows |
| +get the database protection only; the nested layout (#126) is what |
| +protects the rest. Carrying the working directory across a move is the |
| +moves spec's concern: it doesn't move today, and that caused the |
| +2026-10-02 kpmg outage (#175). |
| + |
| +### 2.6 Retiring a row whose identity changed |
| + |
| +In both `convergeOwned` and `convergeSidecar`, **before** the ownership |
| +branches: if the record for a host has a different incarnation or storage |
| +key from the row, the row is retired. |
| + |
| +1. Fence the host. |
| +2. Stop the child, and require stop evidence (a stopped unit, or no |
| + process), retrying next pass until it holds. |
| +3. Purge the local file with the error-returning purge. |
| +4. Replace the row from the record. |
| + |
| +Until step 4 the host is not served. The fence is lifted only after step 4. |
| +This also fixes the stale-other-app row in §1. |
| + |
| +## 3. Part A: #199, by construction |
| + |
| +With §2 in place: |
| + |
| +- A recreated host has a new CreatedAt, so a new incarnation and a new |
| + location. It cannot read the old replica: a different key, never listed. |
| + Its old local file fails the `.lineage` check and is purged. The old |
| + replica stays where it was, in-region, untouched (R1). There is no |
| + archive step, no ownership lookup and no stamp to race. |
| +- The pull door (apidb.go `resolveReplica`) uses `StorageKey(rec)` and |
| + serves only coordinates under that prefix. An old owner's request |
| + resolves the old record. Once that record is gone, the gate 404s before |
| + any key is built, and the new owner's location is a different key. |
| + Between the gate and the read, a request can only reach the prefix its |
| + own record named. |
| +- Legacy records keep `instances/<host>`, adopted by the current record |
| + (R2). A legacy host deleted **after** the epoch and recreated gets a v2 |
| + record, so it is protected. A legacy host recreated before the epoch |
| + keeps today's behaviour. That is unchanged risk until the epoch is |
| + written, and it is the reason to write the epoch promptly. |
| + |
| +Out of scope, recorded: listing and purging what deleted instances left |
| +behind (#412); `dbverify` verdicts still keyed by host (key them by |
| +storage key later); moves leave the source region's lineage behind |
| +(movespass.go never deletes it; the comment at backend_exec.go:285 is |
| +wrong). Under v2 that lineage is never pointed to, so it is garbage, not |
| +a hazard. |
| + |
| +## 4. Moves under v2 (coordinated with the moves session) |
| + |
| +Today a move mirrors the source replica into the target's `moves/<host>/` |
| +in rounds, and the target's first wake promotes staging over |
| +`instances/<host>` ("staging wins", backend_exec.go:1580). Under v2: |
| + |
| +- A move to region T allocates `Gen = g<seq+1>-<moveID>` and mirrors into |
| + `moves/<host>/<incarnation>/<Gen>/` in T's bucket. The source still |
| + writes only `moves/*` cross-region, so IAM is unchanged. |
| +- `MoveComplete` CASes `Region = T` and `DB = {Gen, Root: "moves"}` **in |
| + one write**. T's box restores that location directly and replicates into |
| + it from then on. There is **no promotion** and no copy for v2 records. |
| +- A stale copy from an earlier stay in T (ie→uk→ie) has an older Gen and |
| + is never pointed to. |
| +- An abort leaves an unreferenced location, which is garbage. |
| +- The next push moves the instance back under `instances/`. |
| +- Copy receipts (`move-receipts/<host>.json` in the source bucket, added |
| + in 7edd4b33 for multipart ETags) are keyed by the source storage key |
| + instead: `move-receipts/<host>/<incarnation>/<Gen>.json`. A stale |
| + receipt cannot vouch wrongly, since it matches ETags exactly, but it |
| + would otherwise outlive its lineage. |
| +- The cut and landing paths are being redesigned on |
| + `design-moves-any-region` (move-any-app spec). That spec builds on this |
| + section, and the generation bump at the cut is agreed with its author. |
| +- Legacy records keep today's flow until their first v2 event. A move of |
| + a legacy record after the epoch is that event: it lands as v2 at |
| + `moves/<host>/<incarnation>/g1-<moveID>/`. |
| + |
| +This section needs the moves session's review before any code; it |
| +touches `movespass.go`, `promoteStagedReplica` and `MoveComplete`. |
| + |
| +## 5. Part B: `carlos db push` |
| + |
| +### 5.1 Scope |
| + |
| +The same as pull: members only, with the shared 403 for every service |
| +principal; in-region, non-pooled, activator-managed exec instances only; |
| +**v2 records only** (push refuses a legacy record with "not available for |
| +this instance yet"); refused for a `role: replica` record, while a move is |
| +in flight, and the reverse; the typed confirmation header must name the |
| +host. Enabled only when the v2 epoch exists. |
| + |
| +### 5.2 Request |
| + |
| +Tony's `provision.DBTransfer` (b014d57d dbtransfer.go), with the pull op |
| +removed. A request records `ID`, `Incarnation`, `Region`, `FromGen` (the |
| +record's current generation at request time) and `ToGen` (= |
| +`g<seq+1>-<ID>`), plus `Step`, `Error` and `Bytes`. The acceptance CAS |
| +**refuses** unless the record's incarnation, region and generation still |
| +equal what the console saw before the upload began. That binds the upload |
| +to the instance as it was. |
| + |
| +### 5.3 Upload (R3) |
| + |
| +The console streams the body to the instance's region's data bucket at |
| +`db-uploads/<incarnation>/<ID>.db`: |
| + |
| +- The upload is multipart, and published only on complete. |
| +- A byte counter aborts it, and the multipart upload with it, past the |
| + cap (1 GiB). |
| +- The first 100 bytes must be a SQLite header. |
| +- The completed object's size and checksum go onto the request. |
| + |
| +It needs a console grant for `PutObject`, `AbortMultipartUpload` and |
| +`DeleteObject` on `db-uploads/*` in its own region's bucket. |
| + |
| +The request CAS comes after the upload. On an error other than a |
| +conflict, the console re-reads: if the id is on the record, answer 202; |
| +otherwise delete the upload. An orphaned upload is storage garbage, not a |
| +safety problem: no step below destroys anything. The box deletes uploads |
| +that are older than a day and that no request names. |
| + |
| +### 5.4 The box worker |
| + |
| +All work happens on the box that owns the route. Nothing is deleted |
| +until after the commit. |
| + |
| +1. **Hold the route.** Set its status to `db-push` (distinct from |
| + `maintenance`, which the maintenance feature owns). Fence it and wait |
| + until it is quiesced. |
| +2. **Note where the live lineage ends.** Read the current generation's |
| + newest TXID (one listing) and record it on the request. |
| +3. **Download and check the upload.** Download it to scratch, but only if |
| + free disk is at least twice its size. Then verify the size and checksum |
| + against the request and run `CheckDB`. A bad file fails the request and |
| + releases the route; nothing has changed. |
| +4. **Replicate into ToGen.** `lease.Acquire` the ToGen location, then |
| + check the record under the lease. If the record already points at |
| + ToGen, stop: this request has already committed. Otherwise replicate |
| + scratch into ToGen, wait for sync, and release. Writing into ToGen is |
| + harmless at any time before the commit: nothing points at it, and its |
| + nonce makes it this request's alone. A replay writes the same bytes |
| + again. |
| +5. **Commit.** |
| + - Re-read the current generation's newest TXID. **If it moved since |
| + step 2, abort** (something wrote despite the hold): fail the request |
| + and release. |
| + - Otherwise CAS the record with `DB.Gen = ToGen, Root = "instances"`, |
| + guarded by request ID, incarnation and `FromGen`. **This CAS is the |
| + commit.** Before it, the instance is exactly as it was; after it, the |
| + new generation is live. |
| +6. **Converge.** |
| + - Update the row's `storage_key`, then purge the local file with the |
| + error-returning purge, so the next wake restores ToGen cold. |
| + - A purge error keeps the route held and retries next pass. The |
| + `.lineage` check (§2.5) would refuse the old file anyway. |
| +7. **Release.** |
| + - Set the route's status back to `ready`. A failure returns an error |
| + and keeps the request in flight. |
| + - Then lift the fence, mark the request done and delete the upload. |
| + |
| +**The previous generation is the backup.** The newest five generations |
| +are retained; older ones are deleted after a commit. They are never the |
| +current pointer, so deleting them cannot touch live data. |
| + |
| +**Restart:** |
| +- The worker re-derives everything from the record and the row. |
| +- A route left at `db-push` whose record has no in-flight request is |
| + released, but only if the record is readable and well-formed. An |
| + unreadable or malformed record keeps the hold and raises an alert. |
| +- Before step 5, releasing on any failure is always safe, because the |
| + current generation was never touched. |
| + |
| +### 5.5 Undo and backups |
| + |
| +- `carlos db backups --instance <h>` lists the retained generations. |
| +- `carlos db rollback --instance <h> --to <gen>` is a push with no upload: |
| + hold, then CAS the pointer to `<gen>` (guarded as in 5.4), then converge |
| + and release. The generation being left is retained, so a rollback can be |
| + undone. |
| +- `carlos db pull --instance <h> --generation <gen> --out <path>`: the door |
| + serves a retained generation's coordinates, validated against the |
| + retained list. |
| + |
| +### 5.6 Purge with errors |
| + |
| +Add `activator.PurgeLocalDBArtifactsErr(dbPath, full) error`. It removes |
| +`.lineage` first, then returns the first removal error other than |
| +not-exist. Used by §2.5, §2.6, push step 6, and the move cut |
| +(movespass.go:492, which ignores purge errors today). |
| + |
| +### 5.7 One record listing per pass |
| The moves pass and the db pass share one `ListAllRecords` per converge |
| -pass, folded once and handed to both. |
| - |
| -### B6. Purge with errors (#194 P1-3) |
| - |
| -Add `activator.PurgeLocalDBArtifactsErr(dbPath, full) error`, which |
| -returns the first removal error other than not-exist, and its stamp |
| -removal. The push and the A4 gate use it. The existing best-effort |
| -function stays for its current callers. |
| - |
| -### B7. Undo |
| - |
| -- `carlos db backups --instance <h>` lists the backups under |
| - `db-backups/<incarnation>/`. |
| -- `carlos db pull --instance <h> --backup <id> --out <path>`: |
| - - the pull door serves a backup's LTX coordinates by `?backup=<id>`; |
| - - the backup id is validated, and nothing outside |
| - `db-backups/<incarnation>/<id>/` is reachable; |
| - - the CLI restores it read-only, exactly as a normal pull. |
| -- To put a backup back, pull it, then push it. The push takes its own |
| - backup first. |
| - |
| -The activator's promotion logic (`promoteStagedReplica`) is factored so |
| -step B4.5 and the wake share one implementation. |
| - |
| -### B8. CLI |
| - |
| -`carlos db push --instance <h> <file>`, with these rules: |
| - |
| -- `--instance` is required for push and is never inferred. Replacing the |
| - wrong instance's database is the failure to avoid. |
| -- An interactive confirmation names the host. |
| -- `--yes` skips the prompt but still requires `--instance`. |
| - |
| -`carlos db status --instance <h>` shows the request, its step and any |
| -error. All user-facing strings go through copy review. |
| - |
| -### B9. Tests that pin the review's findings |
| - |
| -Each is a test: |
| - |
| -- two workers on one prefix: the second cannot take the lease, so it |
| - neither deletes nor replicates (P1-1); |
| -- a push with move staging present backs up and replaces what a wake |
| - would restore, and the next wake does not resurrect staging (P1-2); |
| -- a failing purge keeps the route held and the request in flight (P1-3); |
| -- a CAS write whose response is lost (the record has the id) keeps the |
| - upload, and the push completes (P1-4); |
| -- an upload missing at `replacing` keeps the route held and never seeds |
| - (P1-4); |
| -- a `SetStatus("ready")` failure keeps the request in flight, and a later |
| - pass completes it (P2-5); |
| -- restart with a cleared record and a `db-push` route: released (P2-5); |
| -- an upload over the cap is refused while streaming; insufficient disk |
| - fails before step 9 (P2-6); |
| -- one `ListAllRecords` per pass (P2-7). |
| - |
| -Part A's own tests: |
| - |
| -- a recreated host (same account, other app, other account, sidecar) |
| - never sees the old data: |
| - - not through the warm path; |
| - - not through restore; |
| - - not through the door; |
| -- archive happens only when the old record is gone, and its ETags are |
| - verified before the delete; |
| -- the two-live-owners case refuses; |
| -- legacy adoption, by the sweep and by a wake; |
| -- a move carries the stamp, and a staged stamp that disagrees refuses; |
| -- the incarnation change on a live row stops the child. |
| - |
| -## 4. Rollout |
| - |
| -1. **Part A, IAM first:** the `parked/*` grant goes into the region-host |
| - module. Dev tracks main. For flagship, the change rides the next |
| - release tag through carloku-infrastructure. Target the change; never |
| - apply the whole root (user_data restarts every box). |
| -2. **Part A, boxes:** the stamp, the gate and the sweep. The console door |
| - check (A5) can roll at any time: an absent stamp is allowed. |
| -3. **Part B:** |
| - - IAM: the console's `db-uploads/*` grant and the box's `db-uploads/*` |
| - and `db-backups/*` grants. |
| - - Roll boxes before the console. The console exposes push, and a box |
| - without the worker would leave requests queued forever. |
| - - CLI release, and copy review of every string. |
| +pass. This is a cost change only; every destructive decision re-reads its |
| +own record (5.4 steps 4 and 5). |
| + |
| +### 5.8 CLI |
| + |
| +`carlos db push --instance <h> <file>`, `status`, `backups`, `rollback`, |
| +and `pull --generation`. `--instance` is always required for push and |
| +rollback, never inferred, and an interactive confirmation names the host |
| +(`--yes` skips the prompt only). All strings go through copy review. |
| + |
| +## 6. Rev 1's findings, answered |
| + |
| +| # | Rev 1 finding | Rev 2 | |
| +|---|---|---| |
| +| 1, 2 | other-app or sidecar row survives recreate | §2.6, both families, retire before ownership branches | |
| +| 3, 7, 9 | warm file, file before stamp, sweep races | no stamps; `.lineage` is per storage key, written after sync; no sweep | |
| +| 4, 16 | archive retries; ETag verification | no archive, no copy | |
| +| 5 | OwnerLive on wrong record family | no ownership lookup | |
| +| 6 | staging overwrites before validation | v2 has no promotion (§4) | |
| +| 8 | zero CreatedAt still creatable | creates refuse it (§2.1) | |
| +| 10 | another box's warm cache, same incarnation | `.lineage` carries the storage key, which every push and move changes | |
| +| 11 | door check and read are separate | the door reads only its record's key | |
| +| 12 | lease contract incomplete | nothing destructive runs under a lease; replicating into an unreferenced ToGen is harmless | |
| +| 13 | another box seeds an empty replica | nothing is ever emptied; the current generation is untouched until the commit CAS | |
| +| 14 | stale worker replays | ToGen is unique per request; the commit is a guarded CAS; re-check under lease | |
| +| 15 | release on absent request | release only on a readable, well-formed record; hold otherwise | |
| +| 17 | upload not bound to residence | acceptance CAS binds incarnation, region and generation | |
| +| 18 | GC deletes a committed upload | an upload is never load-bearing after step 3's checksum, and GC is a day old | |
| +| 19 | streaming completion, recovery | §5.3 multipart rules; recovery is rollback | |
| +| 20 | mixed-version writers | the epoch object gates v2 and push (§7) | |
| + |
| +## 7. Rollout |
| + |
| +1. **Release N** contains all of this, dormant. With no epoch object, |
| + every record is legacy, push is refused, and only the new columns, the |
| + `.lineage` adoption on legacy rows, the §2.6 retire fix, the zero- |
| + CreatedAt refusal and the purge-with-errors change take effect. |
| +2. Roll **every** box and console to N. Confirm with `carlos system |
| + status` and each console's `x-carlos-version`. |
| +3. Write `control/storage-v2.json` create-only, with `since` = now. From |
| + then on, new records are v2 and push works. |
| +4. **Never roll a box or console back below N once the epoch exists.** An |
| + older binary rewriting a record drops `DB` (it doesn't know the field), |
| + which would point a pushed or moved instance back at its first |
| + generation. The README release note says so in bold, and N's binaries |
| + refuse to start if the epoch exists and their own release is older than |
| + the epoch's recorded `min_release` (a field written with the epoch). |
| + |
| +IAM: the region-host box role already covers the whole own data bucket |
| +(main.tf:166, :174). The console needs the §5.3 grant in its own region. |
| Credit: Part B reuses Tony Wood's request record, console door and CLI |
| shape from `feat/db-push-pull`. |