amadan
spec: replica ownership rev 3, park a deleted instance's data at delete time
Rev 2 failed Astra review (13 P1s). Paul ruled (R5): fix #199 at delete time with a host tombstone and an in-region park; push waits for the move-any-app spec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+242
−398
docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md+0 −398docs/superpowers/specs/2026-10-02-replica-ownership-design.md+242 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md b/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md| deleted file mode 100644 |
| index bba43870c..000000000 |
| --- a/docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md |
| +++ /dev/null |
| @@ -1,398 +0,0 @@ |
| -# Replica ownership (#199) and `carlos db push` |
| - |
| -Design, 2026-10-02, **rev 2**. Part A closes amadan #199. Part B rebuilds |
| -`carlos db push` from `feat/db-push-pull` (Tony Wood, b014d57d), fixing |
| -the review in #194. Rev 1 (stamps on host-keyed storage) failed Astra's |
| -review with 16 P1s, all from one cause: data was still keyed by host, so |
| -every path that touched the shared location needed its own ownership |
| -check. Rev 2 removes the cause instead: **data is keyed by incarnation and |
| -generation, and the instance record says which generation is current.** |
| -Rev 1's findings are answered in §6. |
| - |
| -Rulings (Paul, 2026-10-02), not to reopen: |
| - |
| -- **R1.** A recreated host never sees an earlier incarnation's data; that |
| - data is kept in-region and the new instance starts empty. |
| -- **R2.** Data that exists before this change is adopted by the record |
| - that owns the host today. |
| -- **R3.** A pushed database and its backups live in the instance's own |
| - region. Push is in-region only, like pull. |
| -- **R4.** Storage is rekeyed by incarnation and generation (rev 2's |
| - approach), rather than stamping host-keyed storage (rev 1). |
| - |
| -## 1. The problem |
| - |
| -Storage identity is the host, alone: the replica at |
| -`activator.HibernatePrefix(prefix, host)` = `instances/<host>` |
| -(activator.go:138), the local database at `DefaultDBPath` / |
| -`NestedDBPath` (provision.go:1219, 1277). Deleting an instance removes the |
| -record only (provision.go:1071); nothing deletes the replica or the local |
| -file. Nothing checks ownership on wake or in the pull door. So whoever next |
| -creates the host inherits the data. The same account does this routinely: |
| -every app in an account shares `<label>.<sqid>.<AppsDomain>`, and titogo |
| -creates and deletes instances per customer slug. Sidecars |
| -(`<name>.<app>.<sqid>.sidecar.internal`) have the same exposure. |
| - |
| -Separately, a host deleted and recreated between two reconcile passes |
| -under a different app keeps its old registry row forever: `convergeOwned` |
| -runs only for a matching account and app (reconcile.go:947), the mismatch |
| -branch leaves the row alone (:1031), and removal skips it because the host |
| -still has a record (:1141). The old app keeps serving the new owner's |
| -host. Sidecars have the same shape (sidecar_reconcile.go:131, :467). |
| - |
| -## 2. The storage key |
| - |
| -### 2.1 Incarnation |
| - |
| - incarnation = RecordIdentity(account, app, host, createdAt) (hex, 64) |
| - |
| -`maintenanceguard.RecordIdentity` (types.go:62), factored into one shared |
| -`provision.Incarnation(rec)` for instance and sidecar records. Account, |
| -App, Host and CreatedAt are immutable under every update |
| -(`protectedFieldsEqual`, maintenance.go:81). `CreateRecord` and |
| -`CreateSidecarRecord` (provision.go:820, sidecar.go:322) now **refuse** a |
| -zero CreatedAt, so every record created from here on has an incarnation. A |
| -record with zero CreatedAt can only be legacy (§2.3). |
| - |
| -### 2.2 Generation |
| - |
| -A record gains one field: |
| - |
| - DB *DBPointer `json:"db,omitempty"` |
| - type DBPointer struct { |
| - Gen string `json:"gen"` // "g<seq>-<nonce>", e.g. "g2-7f3a91c0" |
| - Root string `json:"root"` // "instances" | "moves" |
| - } |
| - |
| -The nonce is the id of the request that made the generation (a push, a |
| -move) and is never reused, so two generations can never share a location |
| -even if a sequence number is replayed. `Root` exists so a moved instance |
| -can live where the move landed it without a copy (§4). |
| - |
| -### 2.3 Where a record's data lives |
| - |
| - v2: <Root>/<host>/<incarnation>/<Gen>/ (replica) |
| - legacy: instances/<host> (replica, as today) |
| - |
| -A record is **legacy** when `DB == nil` and its CreatedAt is before the |
| -deployment's **v2 epoch**. Otherwise it is v2. A v2 record with `DB == |
| -nil` uses `{Gen: "g0", Root: "instances"}`. |
| - |
| -The v2 epoch is one object, `control/storage-v2.json` `{"since": "<RFC3339>"}`, |
| -written create-only by an operator once every box and console runs a |
| -release that understands it (§7). Before it exists, **every record is |
| -legacy** and nothing changes. CreatedAt is immutable and every old writer |
| -preserves it, so no old binary can turn a v2 record legacy or the reverse. |
| - |
| -One function, `provision.StorageKey(rec, epoch) (StorageKey, error)`, |
| -returns the location. **Every reader and writer goes through it.** There |
| -are 17 call sites that build the prefix from the host today |
| -(activator.go ×3, backend_exec.go ×9, movespass.go ×3, apidb.go, |
| -hibernate_verify.go). The registry row carries the computed key (§2.4), so |
| -the activator never derives it. |
| - |
| -### 2.4 The registry row |
| - |
| -New columns, added by idempotent `ALTER TABLE` at start (like |
| -`routes.replicas`): `incarnation`, `storage_key` (the full replica |
| -prefix), `storage_gen`. `routeFor` and `sidecarRouteFor` render them from |
| -the record and the epoch; `convergeOwned` and `convergeSidecar` converge |
| -them in both directions. A change of `incarnation` or `storage_key` on a |
| -live row is a **retire**, not a column update (§2.6). |
| - |
| -### 2.5 The local database |
| - |
| -The tenant-visible `DBPath` does not change: the app's environment, and |
| -the isolated-exec nested layout, stay as they are. Beside it, the box |
| -keeps `<dbPath>.lineage`: |
| - |
| - {"incarnation":"…","storage_key":"…"} |
| - |
| -It is written by atomic rename **after** a restore or a seed succeeds and |
| -the first replication sync has completed, and it is removed **first**, |
| -before any purge. |
| - |
| -On wake (`ExecBackend.Restore`, before the warm-cache return at |
| -backend_exec.go:267): |
| - |
| -- **v2 row:** the warm local file is used only if `.lineage` equals the |
| - row's `incarnation` and `storage_key`. Anything else (absent, different, |
| - unreadable) purges the local file with the error-returning purge |
| - (§5.6), and the wake fails if the purge fails. Then it restores cold |
| - from `storage_key`. |
| -- **legacy row:** as today, and `.lineage` is written on first sight |
| - (R2). |
| - |
| -A database a fresh v2 instance creates in `Start` (activator.go:1168), |
| -before any `.lineage` exists, is being replicated into its own |
| -generation's location. After a crash it is restored from there, never |
| -adopted, so there is no "file before stamp" window. |
| - |
| -### 2.5b The instance's working directory |
| - |
| -The database is not the only host-keyed local state. A nested row's |
| -working directory, `InstanceDir(host)` (provision.go:1270, used as the |
| -app's data dir by activator/paths.go:37), holds whatever the app writes |
| -there, for example titogo's `instance.key` and `report.token`. A recreated |
| -host inherits these today (reported by the moves session, 2026-10-02). |
| - |
| -The two pieces of local state have different lifetimes: |
| - |
| -- **The working directory belongs to the incarnation.** It carries |
| - `.incarnation`, which is written when the directory is created and |
| - checked on every wake before the app starts. If it is absent on a v2 |
| - row, or different, the box moves the whole directory aside to |
| - `<DataDir>/.retired/<old-incarnation>/` and creates it fresh. Nothing is |
| - deleted automatically; operator purge is #412's tooling. A legacy row |
| - adopts (R2). |
| -- **The database belongs to the generation** (§2.5). A push or rollback |
| - replaces the database and keeps the directory, so `instance.key` |
| - survives a push. |
| - |
| -A flat row's app writes wherever it chooses under the shared `/data`, and |
| -the platform cannot attribute those files to an incarnation. Flat rows |
| -get the database protection only; the nested layout (#126) is what |
| -protects the rest. Carrying the working directory across a move is the |
| -moves spec's concern: it doesn't move today, and that caused the |
| -2026-10-02 kpmg outage (#175). |
| - |
| -### 2.6 Retiring a row whose identity changed |
| - |
| -In both `convergeOwned` and `convergeSidecar`, **before** the ownership |
| -branches: if the record for a host has a different incarnation or storage |
| -key from the row, the row is retired. |
| - |
| -1. Fence the host. |
| -2. Stop the child, and require stop evidence (a stopped unit, or no |
| - process), retrying next pass until it holds. |
| -3. Purge the local file with the error-returning purge. |
| -4. Replace the row from the record. |
| - |
| -Until step 4 the host is not served. The fence is lifted only after step 4. |
| -This also fixes the stale-other-app row in §1. |
| - |
| -## 3. Part A: #199, by construction |
| - |
| -With §2 in place: |
| - |
| -- A recreated host has a new CreatedAt, so a new incarnation and a new |
| - location. It cannot read the old replica: a different key, never listed. |
| - Its old local file fails the `.lineage` check and is purged. The old |
| - replica stays where it was, in-region, untouched (R1). There is no |
| - archive step, no ownership lookup and no stamp to race. |
| -- The pull door (apidb.go `resolveReplica`) uses `StorageKey(rec)` and |
| - serves only coordinates under that prefix. An old owner's request |
| - resolves the old record. Once that record is gone, the gate 404s before |
| - any key is built, and the new owner's location is a different key. |
| - Between the gate and the read, a request can only reach the prefix its |
| - own record named. |
| -- Legacy records keep `instances/<host>`, adopted by the current record |
| - (R2). A legacy host deleted **after** the epoch and recreated gets a v2 |
| - record, so it is protected. A legacy host recreated before the epoch |
| - keeps today's behaviour. That is unchanged risk until the epoch is |
| - written, and it is the reason to write the epoch promptly. |
| - |
| -Out of scope, recorded: listing and purging what deleted instances left |
| -behind (#412); `dbverify` verdicts still keyed by host (key them by |
| -storage key later); moves leave the source region's lineage behind |
| -(movespass.go never deletes it; the comment at backend_exec.go:285 is |
| -wrong). Under v2 that lineage is never pointed to, so it is garbage, not |
| -a hazard. |
| - |
| -## 4. Moves under v2 (coordinated with the moves session) |
| - |
| -Today a move mirrors the source replica into the target's `moves/<host>/` |
| -in rounds, and the target's first wake promotes staging over |
| -`instances/<host>` ("staging wins", backend_exec.go:1580). Under v2: |
| - |
| -- A move to region T allocates `Gen = g<seq+1>-<moveID>` and mirrors into |
| - `moves/<host>/<incarnation>/<Gen>/` in T's bucket. The source still |
| - writes only `moves/*` cross-region, so IAM is unchanged. |
| -- `MoveComplete` CASes `Region = T` and `DB = {Gen, Root: "moves"}` **in |
| - one write**. T's box restores that location directly and replicates into |
| - it from then on. There is **no promotion** and no copy for v2 records. |
| -- A stale copy from an earlier stay in T (ie→uk→ie) has an older Gen and |
| - is never pointed to. |
| -- An abort leaves an unreferenced location, which is garbage. |
| -- The next push moves the instance back under `instances/`. |
| -- Copy receipts (`move-receipts/<host>.json` in the source bucket, added |
| - in 7edd4b33 for multipart ETags) are keyed by the source storage key |
| - instead: `move-receipts/<host>/<incarnation>/<Gen>.json`. A stale |
| - receipt cannot vouch wrongly, since it matches ETags exactly, but it |
| - would otherwise outlive its lineage. |
| -- The cut and landing paths are being redesigned on |
| - `design-moves-any-region` (move-any-app spec). That spec builds on this |
| - section, and the generation bump at the cut is agreed with its author. |
| -- Legacy records keep today's flow until their first v2 event. A move of |
| - a legacy record after the epoch is that event: it lands as v2 at |
| - `moves/<host>/<incarnation>/g1-<moveID>/`. |
| - |
| -This section needs the moves session's review before any code; it |
| -touches `movespass.go`, `promoteStagedReplica` and `MoveComplete`. |
| - |
| -## 5. Part B: `carlos db push` |
| - |
| -### 5.1 Scope |
| - |
| -The same as pull: members only, with the shared 403 for every service |
| -principal; in-region, non-pooled, activator-managed exec instances only; |
| -**v2 records only** (push refuses a legacy record with "not available for |
| -this instance yet"); refused for a `role: replica` record, while a move is |
| -in flight, and the reverse; the typed confirmation header must name the |
| -host. Enabled only when the v2 epoch exists. |
| - |
| -### 5.2 Request |
| - |
| -Tony's `provision.DBTransfer` (b014d57d dbtransfer.go), with the pull op |
| -removed. A request records `ID`, `Incarnation`, `Region`, `FromGen` (the |
| -record's current generation at request time) and `ToGen` (= |
| -`g<seq+1>-<ID>`), plus `Step`, `Error` and `Bytes`. The acceptance CAS |
| -**refuses** unless the record's incarnation, region and generation still |
| -equal what the console saw before the upload began. That binds the upload |
| -to the instance as it was. |
| - |
| -### 5.3 Upload (R3) |
| - |
| -The console streams the body to the instance's region's data bucket at |
| -`db-uploads/<incarnation>/<ID>.db`: |
| - |
| -- The upload is multipart, and published only on complete. |
| -- A byte counter aborts it, and the multipart upload with it, past the |
| - cap (1 GiB). |
| -- The first 100 bytes must be a SQLite header. |
| -- The completed object's size and checksum go onto the request. |
| - |
| -It needs a console grant for `PutObject`, `AbortMultipartUpload` and |
| -`DeleteObject` on `db-uploads/*` in its own region's bucket. |
| - |
| -The request CAS comes after the upload. On an error other than a |
| -conflict, the console re-reads: if the id is on the record, answer 202; |
| -otherwise delete the upload. An orphaned upload is storage garbage, not a |
| -safety problem: no step below destroys anything. The box deletes uploads |
| -that are older than a day and that no request names. |
| - |
| -### 5.4 The box worker |
| - |
| -All work happens on the box that owns the route. Nothing is deleted |
| -until after the commit. |
| - |
| -1. **Hold the route.** Set its status to `db-push` (distinct from |
| - `maintenance`, which the maintenance feature owns). Fence it and wait |
| - until it is quiesced. |
| -2. **Note where the live lineage ends.** Read the current generation's |
| - newest TXID (one listing) and record it on the request. |
| -3. **Download and check the upload.** Download it to scratch, but only if |
| - free disk is at least twice its size. Then verify the size and checksum |
| - against the request and run `CheckDB`. A bad file fails the request and |
| - releases the route; nothing has changed. |
| -4. **Replicate into ToGen.** `lease.Acquire` the ToGen location, then |
| - check the record under the lease. If the record already points at |
| - ToGen, stop: this request has already committed. Otherwise replicate |
| - scratch into ToGen, wait for sync, and release. Writing into ToGen is |
| - harmless at any time before the commit: nothing points at it, and its |
| - nonce makes it this request's alone. A replay writes the same bytes |
| - again. |
| -5. **Commit.** |
| - - Re-read the current generation's newest TXID. **If it moved since |
| - step 2, abort** (something wrote despite the hold): fail the request |
| - and release. |
| - - Otherwise CAS the record with `DB.Gen = ToGen, Root = "instances"`, |
| - guarded by request ID, incarnation and `FromGen`. **This CAS is the |
| - commit.** Before it, the instance is exactly as it was; after it, the |
| - new generation is live. |
| -6. **Converge.** |
| - - Update the row's `storage_key`, then purge the local file with the |
| - error-returning purge, so the next wake restores ToGen cold. |
| - - A purge error keeps the route held and retries next pass. The |
| - `.lineage` check (§2.5) would refuse the old file anyway. |
| -7. **Release.** |
| - - Set the route's status back to `ready`. A failure returns an error |
| - and keeps the request in flight. |
| - - Then lift the fence, mark the request done and delete the upload. |
| - |
| -**The previous generation is the backup.** The newest five generations |
| -are retained; older ones are deleted after a commit. They are never the |
| -current pointer, so deleting them cannot touch live data. |
| - |
| -**Restart:** |
| -- The worker re-derives everything from the record and the row. |
| -- A route left at `db-push` whose record has no in-flight request is |
| - released, but only if the record is readable and well-formed. An |
| - unreadable or malformed record keeps the hold and raises an alert. |
| -- Before step 5, releasing on any failure is always safe, because the |
| - current generation was never touched. |
| - |
| -### 5.5 Undo and backups |
| - |
| -- `carlos db backups --instance <h>` lists the retained generations. |
| -- `carlos db rollback --instance <h> --to <gen>` is a push with no upload: |
| - hold, then CAS the pointer to `<gen>` (guarded as in 5.4), then converge |
| - and release. The generation being left is retained, so a rollback can be |
| - undone. |
| -- `carlos db pull --instance <h> --generation <gen> --out <path>`: the door |
| - serves a retained generation's coordinates, validated against the |
| - retained list. |
| - |
| -### 5.6 Purge with errors |
| - |
| -Add `activator.PurgeLocalDBArtifactsErr(dbPath, full) error`. It removes |
| -`.lineage` first, then returns the first removal error other than |
| -not-exist. Used by §2.5, §2.6, push step 6, and the move cut |
| -(movespass.go:492, which ignores purge errors today). |
| - |
| -### 5.7 One record listing per pass |
| - |
| -The moves pass and the db pass share one `ListAllRecords` per converge |
| -pass. This is a cost change only; every destructive decision re-reads its |
| -own record (5.4 steps 4 and 5). |
| - |
| -### 5.8 CLI |
| - |
| -`carlos db push --instance <h> <file>`, `status`, `backups`, `rollback`, |
| -and `pull --generation`. `--instance` is always required for push and |
| -rollback, never inferred, and an interactive confirmation names the host |
| -(`--yes` skips the prompt only). All strings go through copy review. |
| - |
| -## 6. Rev 1's findings, answered |
| - |
| -| # | Rev 1 finding | Rev 2 | |
| -|---|---|---| |
| -| 1, 2 | other-app or sidecar row survives recreate | §2.6, both families, retire before ownership branches | |
| -| 3, 7, 9 | warm file, file before stamp, sweep races | no stamps; `.lineage` is per storage key, written after sync; no sweep | |
| -| 4, 16 | archive retries; ETag verification | no archive, no copy | |
| -| 5 | OwnerLive on wrong record family | no ownership lookup | |
| -| 6 | staging overwrites before validation | v2 has no promotion (§4) | |
| -| 8 | zero CreatedAt still creatable | creates refuse it (§2.1) | |
| -| 10 | another box's warm cache, same incarnation | `.lineage` carries the storage key, which every push and move changes | |
| -| 11 | door check and read are separate | the door reads only its record's key | |
| -| 12 | lease contract incomplete | nothing destructive runs under a lease; replicating into an unreferenced ToGen is harmless | |
| -| 13 | another box seeds an empty replica | nothing is ever emptied; the current generation is untouched until the commit CAS | |
| -| 14 | stale worker replays | ToGen is unique per request; the commit is a guarded CAS; re-check under lease | |
| -| 15 | release on absent request | release only on a readable, well-formed record; hold otherwise | |
| -| 17 | upload not bound to residence | acceptance CAS binds incarnation, region and generation | |
| -| 18 | GC deletes a committed upload | an upload is never load-bearing after step 3's checksum, and GC is a day old | |
| -| 19 | streaming completion, recovery | §5.3 multipart rules; recovery is rollback | |
| -| 20 | mixed-version writers | the epoch object gates v2 and push (§7) | |
| - |
| -## 7. Rollout |
| - |
| -1. **Release N** contains all of this, dormant. With no epoch object, |
| - every record is legacy, push is refused, and only the new columns, the |
| - `.lineage` adoption on legacy rows, the §2.6 retire fix, the zero- |
| - CreatedAt refusal and the purge-with-errors change take effect. |
| -2. Roll **every** box and console to N. Confirm with `carlos system |
| - status` and each console's `x-carlos-version`. |
| -3. Write `control/storage-v2.json` create-only, with `since` = now. From |
| - then on, new records are v2 and push works. |
| -4. **Never roll a box or console back below N once the epoch exists.** An |
| - older binary rewriting a record drops `DB` (it doesn't know the field), |
| - which would point a pushed or moved instance back at its first |
| - generation. The README release note says so in bold, and N's binaries |
| - refuse to start if the epoch exists and their own release is older than |
| - the epoch's recorded `min_release` (a field written with the epoch). |
| - |
| -IAM: the region-host box role already covers the whole own data bucket |
| -(main.tf:166, :174). The console needs the §5.3 grant in its own region. |
| - |
| -Credit: Part B reuses Tony Wood's request record, console door and CLI |
| -shape from `feat/db-push-pull`. |
diff --git a/docs/superpowers/specs/2026-10-02-replica-ownership-design.md b/docs/superpowers/specs/2026-10-02-replica-ownership-design.md| new file mode 100644 |
| index 000000000..fb8a9de04 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-02-replica-ownership-design.md |
| @@ -0,0 +1,242 @@ |
| +# Replica ownership (#199): park a deleted instance's data |
| + |
| +Design, 2026-10-02, **rev 3**. Closes amadan #199. `carlos db push` is |
| +**deferred**: it will be designed on top of the move-any-app spec's |
| +fencing and generation handoff (`design-moves-any-region`), not here. |
| + |
| +History. Rev 1 stamped host-keyed storage with an owner and inferred |
| +ownership at wake time. It failed Astra's review with 16 P1s. Rev 2 rekeyed |
| +all storage by incarnation and generation. It failed with 13 P1s, mostly |
| +from old writers erasing a storage discriminator, and from cross-box |
| +fencing that push needs and moves is now designing. Both revs tried to |
| +**infer**, at create or wake time, whose data a host's leftovers are. Rev 3 |
| +acts at **delete** time instead, the one moment the owner is known for |
| +certain. |
| + |
| +Rulings (Paul, 2026-10-02), not to reopen: |
| + |
| +- **R1.** A recreated host never sees an earlier incarnation's data. That |
| + data is kept in-region (parked), and the new instance starts empty. |
| +- **R2.** Data whose deletion was never recorded is adopted by the record |
| + that owns the host today (unchanged behaviour). |
| +- **R5.** Fix #199 at delete time. Push waits for moves. |
| + |
| +## 1. The problem |
| + |
| +Storage identity is the host alone: |
| + |
| +- the replica at `instances/<host>` (activator.go:138), plus any move |
| + staging at `moves/<host>/` (backend_exec.go:1549); |
| +- the local database at `DefaultDBPath` / `NestedDBPath` (provision.go |
| + :1219, :1277); |
| +- a nested row's working directory `InstanceDir(host)` (provision.go:1270, |
| + used as the app's data dir by activator/paths.go:37), which holds app |
| + files such as titogo's `instance.key`; |
| +- the restore-drill verdict, keyed by host (dbverify). |
| + |
| +Every record-deleting path leaves all of these behind: |
| + |
| +- `handleAPIInstanceDelete` → `DeleteRecord` (provision.go:1071); |
| +- the sidecar delete (apisidecars.go:347); |
| +- `PurgeApp` (trash.go:182), called by the console's purge door |
| + (ownership.go:442) and the boxes' auto-purge (domains/reconcile.go:232). |
| + |
| +Reconcile then drops the registry row (reconcile.go:1155) and nothing |
| +else. Whoever next creates the host inherits all of it, on wake or through |
| +`carlos db pull`. The same account does this routinely (titogo's |
| +per-customer slugs). |
| + |
| +Second defect. A host deleted under app A and recreated under app B |
| +**between two reconcile passes** keeps A's row forever: |
| + |
| +- `convergeOwned` runs only for a matching owner (reconcile.go:947); |
| +- the mismatch branch leaves the row alone (:1031); |
| +- removal skips the host because it has a record (:1141). |
| + |
| +So A's app keeps serving B's host. Sidecars have the same shape: the |
| +ownership branch is sidecar_reconcile.go:400-412. |
| + |
| +## 2. The host tombstone |
| + |
| +Every path that deletes an instance or sidecar record first writes a |
| +**host tombstone**, create-only: |
| + |
| + control/host-tombstones/<host>/<incarnation>.json |
| + {"v":1, "host":"…", "account":"…", "app":"…", "kind":"instance|sidecar", |
| + "incarnation":"…", "created_at":"<record CreatedAt>", |
| + "region":"<record Region>", "deleted_at":"…", "by":"<actor>", |
| + "state":"pending"} |
| + |
| +- **Incarnation.** `incarnation = provision.Incarnation(rec)`. That is |
| + `maintenanceguard.RecordIdentity(account, app, host, CreatedAt)` for a |
| + record with a CreatedAt. Otherwise it is a domain-separated legacy |
| + identity, `sha256("carlos-legacy-incarnation-v1\0" + account + "\0" + |
| + app + "\0" + host)`, so every deletion is recordable. |
| +- **Ordering.** The tombstone is written **before** the record is deleted. |
| + A crash in between leaves a tombstone beside a live record, and §3.1 |
| + ignores it, because the record with that incarnation still exists. |
| +- **Writers.** The instance delete door, the sidecar delete door, and |
| + `PurgeApp` (one tombstone per instance and sidecar record of the app, |
| + written before the prefix delete) in both its console and box callers. |
| +- **Lifecycle.** A tombstone is never deleted by this change. It moves |
| + `pending → parked` (§3.4). It is a permanent, cheap record that a given |
| + incarnation of a host was deleted, and where its data went. |
| + |
| +No record field changes, so no old writer can erase anything this design |
| +relies on. The cost: a deletion made by an old binary (before this |
| +release, or by an un-rolled console or box) writes no tombstone. Its |
| +leftovers are adopted by the next incarnation, as today (R2). That is the |
| +rollout window. |
| + |
| +## 3. The park |
| + |
| +### 3.1 Who acts |
| + |
| +The **box in the tombstone's region** owns parking the replica. **Every |
| +box** parks its own local leftovers for the host: the local database, |
| +`-wal`, `-shm`, litestream meta directories, and the nested working |
| +directory. A box that served the host before a move has leftovers too. |
| + |
| +A tombstone is **actionable** only when the record for |
| +`(account, app, host)` is authoritatively absent, or present with a |
| +different CreatedAt. "Authoritatively" means a successful read that |
| +returns not-found. A read error, a malformed record, or any doubt means |
| +**wait**, never act. This is the only ownership question the design asks, |
| +and it is answered by a direct read of the one key that matters. |
| + |
| +The park runs from the converge loop, on the same trigger as the moves |
| +pass. Tombstones are found by listing `control/host-tombstones/` once per |
| +pass. |
| + |
| +### 3.2 Before touching data: the row is gone, the child is stopped |
| + |
| +The park for a host does not start until this box holds **no** registry |
| +row for that host under the tombstone's account and app, and no running |
| +child for it. Reconcile removes the row (§4 fixes the case where it |
| +didn't). The park also requires that the replica's `lock.json` is absent |
| +or expired. If a holder is still renewing, the park waits for it. |
| + |
| +### 3.3 The replica park (tombstone's region only) |
| + |
| +Source: every key under `instances/<host>/` and `moves/<host>/`. |
| +Destination: `parked/<incarnation>/<host>/` in the same bucket. |
| + |
| +1. **Manifest absent:** copy every source key except `lock.json` to the |
| + destination. Then write `parked/<incarnation>/<host>/manifest.json`, |
| + listing every copied key with its source size and the destination's |
| + size. Write the manifest only after every copy reports success and |
| + every destination size equals its source size. Sizes, not ETags: |
| + CopyObject changes a multipart ETag. |
| +2. **Manifest present** (a retry): skip copying entirely. The manifest is |
| + the record of what was parked. |
| +3. **Delete the source keys named in the manifest, then `lock.json`.** |
| + - A source key that is **not** in the manifest means something wrote |
| + after the copy. Stop, alert, and leave the tombstone pending for an |
| + operator. This cannot happen while §3.2 holds, so it is a defect |
| + signal, not a branch to handle. |
| + - A retry deletes whatever is left. A delete of an absent key is |
| + success. |
| + |
| +`MirrorPrefix` is **not** used. It deletes destination keys absent from |
| +the source, which is the rev 1 archive hazard. The copy is plain per-key |
| +`Copy`, and it never deletes at the destination. |
| + |
| +### 3.4 The local park (every box) |
| + |
| +Move the database files and, for a nested row, the working directory, |
| +into `<DataDir>/.parked/<incarnation>/<host>/` by `rename` on the same |
| +filesystem. A rename is atomic, and a retry finds the source already gone. |
| +Nothing local is deleted. Then delete this host's dbverify verdict. |
| + |
| +The box records its local park in `<DataDir>/.parked/<incarnation>/ |
| +<host>/done`. When the box in the tombstone's region has finished §3.3, |
| +it CASes the tombstone to `"state":"parked"`, adding `"parked_at"` and |
| +`"box"`. The local parks on other boxes don't gate that state; each box |
| +gates only itself (§3.5). |
| + |
| +`.parked` takes disk on the box (today's bdn/ips outage was a full |
| +volume). The box reports `.parked` bytes in its status, and listing and |
| +purging parked data is #412's operator tooling. |
| + |
| +### 3.5 Creating the host again |
| + |
| +Reconcile **does not create a row** for a host while this box has an |
| +actionable tombstone (§3.1) for it whose incarnation differs from the new |
| +record's, and either: |
| + |
| +- the tombstone is still `pending` and this box is in its region, or |
| +- this box has local leftovers for the host not yet parked. |
| + |
| +The host shows as `provisioning` until both clear. This is the "create |
| +refuses while pending" rule, enforced where the data is. The console |
| +create door stays as it is: a create succeeds, and the box defers serving |
| +it. |
| + |
| +Leftovers with **no** tombstone are adopted (R2). |
| + |
| +## 4. The stale-row fix |
| + |
| +In the **outer** loops of instance reconcile (around reconcile.go:947 and |
| +the removal at :1141) and sidecar reconcile (around :400-412), the |
| +removal decision is keyed by **(account, app, host)**, not host. A |
| +provisioned row whose own `(account, app, host)` record is |
| +authoritatively absent is retired even when another account's or app's |
| +record now names the host: |
| + |
| +1. Fence the host, then stop the child. Retry next pass until a stop is |
| + proven: the unit is inactive, or no process exists. |
| +2. Delete the row. |
| + |
| +Scope guards, unchanged from today: |
| + |
| +- only rows this pass minted (`Provisioned`, carrying account and app) |
| + are retired; |
| +- unmanaged, hand-added and platform routes are never touched; |
| +- the existing unmanaged-route conflict checks stay as they are; |
| +- a read error on the row's own record means wait (the existing `protect` |
| + set). |
| + |
| +The new record's row is then created only after §3.5 allows it. |
| + |
| +## 5. What this does not do |
| + |
| +- **Leftovers from before the release, or from an old binary's delete.** |
| + With no tombstone, they are adopted (R2). Writing the release promptly |
| + shrinks this window. A one-off operator sweep can tombstone known-dead |
| + hosts by hand. |
| +- **A flat row's app files** written anywhere under the shared `/data` |
| + cannot be attributed to a host. They are not parked. The nested layout |
| + (#126) is what scopes them. |
| +- **The source region's replica a completed move leaves behind** |
| + (movespass.go never deletes it; the comment at backend_exec.go:285 is |
| + stale). It belongs to the live incarnation, so it is not parked here. |
| + The moves spec owns cleaning it up. |
| +- **`carlos db push`.** Deferred, with Tony's branch kept as reference. |
| + |
| +## 6. Tests |
| + |
| +- Delete, then recreate, for each of: the same app, another app in the |
| + same account, another account, a sidecar, and an app purge (console and |
| + auto-purge). In every case the new incarnation starts empty: no warm |
| + file, no restore, no `carlos db pull`, no old working directory. |
| +- Delete and recreate within one reconcile pass, under another app: the |
| + old row is retired and its child stopped before the new row exists. |
| +- A crash at every step of §3.3 and §3.4, then a retry: nothing is lost, |
| + and nothing is copied twice over a written manifest. |
| +- A source key written after the manifest: stop, alert, tombstone stays |
| + pending. |
| +- A record read error leaves the tombstone pending and touches nothing. |
| +- A tombstone beside a still-live record (crash between the tombstone and |
| + the record delete): ignored. |
| +- A legacy record with zero CreatedAt: the legacy incarnation, parked the |
| + same way. |
| +- No tombstone: adopted, as today. |
| + |
| +## 7. Rollout |
| + |
| +- **IAM:** the box role already covers its whole own data bucket |
| + (main.tf:166, :174), including `parked/*`. No console IAM change. |
| +- **Order:** roll consoles and boxes in any order. A tombstone written by |
| + a new console is only acted on by new boxes. An old box ignores it, and |
| + reconcile on that old box doesn't defer, so the protection starts when |
| + the boxes roll. No record field changes, so a rollback is safe. |