amadan
spec: the host lifecycle claim, rev 1 (shared by #199 and move-any-app)
Plus Astra's three replica-ownership findings files, unedited, and a superseded note on the replica-ownership spec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 files changed,
+582
−0
docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md+350 −0docs/superpowers/specs/2026-10-02-replica-ownership-design.md+2 −0docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R1.md+54 −0docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R2.md+141 −0docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R3.md+35 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md b/docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md| new file mode 100644 |
| index 000000000..86fcf0347 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md |
| @@ -0,0 +1,350 @@ |
| +# The host lifecycle claim |
| + |
| +Design, 2026-10-02, rev 1. A small primitive cited by two specs: |
| + |
| +- **#199**, deleting an instance and parking its data: |
| + `2026-10-02-replica-ownership-design.md`, which this spec supersedes |
| + from §2 onward; |
| +- **move-any-app**, `design-moves-any-region`: |
| + `2026-10-02-move-any-app-design.md`, which owns the move operation's |
| + phases. |
| + |
| +`carlos db push` will be a third operation on it, later. |
| + |
| +## 1. Why |
| + |
| +Three #199 designs and two move-any-app rounds failed review for the same |
| +reason. Every actor that changes a host's data acts on its own snapshot, |
| +and nothing makes them take turns: |
| + |
| +- create; |
| +- delete and its park; |
| +- reconcile's row admission and retirement; |
| +- the move worker's rounds, landing and rollback; |
| +- staging promotion; |
| +- wake; |
| +- the `carlos db` pull door. |
| + |
| +The findings asked, again and again, for: |
| + |
| +- exclusive ownership held through a destructive step; |
| +- revalidation under it; |
| +- a durable list of where an incarnation's data lives; |
| +- stop evidence that survives a restart; |
| +- obligations that outlive a crash. |
| + |
| +This spec gives a host exactly one place where all of that lives. |
| + |
| +Requirements come from #199 rev 3's findings (`own/findings3.md`) and the |
| +moves R1 and R2 findings (`CODEX-FINDINGS-MOVES-R1.md`, `-R2.md`). They |
| +are cited as `199-n`, `M1-n` and `M2-n`. |
| + |
| +## 2. The object |
| + |
| +One object per host, in the control (deployment) bucket: |
| + |
| + control/hosts/<host>.json |
| + |
| +```json |
| +{ |
| + "v": 1, |
| + "host": "shop.bab.oncarlos.com", |
| + "gen": 41, |
| + "holder": { |
| + "family": "instance", // instance | sidecar |
| + "key": "apps/bab/jam/instances/shop.bab.oncarlos.com.json", |
| + "account": "bab", "app": "jam", "name": "", // name: sidecars only |
| + "incarnation": "<hex>", |
| + "created_at": "<record CreatedAt>" |
| + }, |
| + "op": { |
| + "kind": "serving", // see §4 |
| + "id": "op_…", // unique per operation; "" for serving |
| + "phase": "…", // owned by the operation's spec |
| + "by": "<actor>", "started_at": "…", |
| + "journal": { } // operation-bound durable state, small |
| + }, |
| + "touched": { |
| + "regions": ["ie", "uk"], // every bucket that ever held data for this incarnation |
| + "boxes": ["adi-knitting", "iris-clockmaking"] // every box that ever held local state |
| + }, |
| + "obligations": [ |
| + {"id": "ob_…", "kind": "park-replica", "region": "uk"}, |
| + {"id": "ob_…", "kind": "park-local", "box": "iris-clockmaking"} |
| + ] |
| +} |
| +``` |
| + |
| +These rules apply to every field: |
| + |
| +- **`gen` increases by one on every write.** It is the claim's version, |
| + and every CAS is conditional on the ETag of the read. |
| +- **`holder` is the one incarnation entitled to the host.** |
| + - It is absent only when the host is **free**. |
| + - `key` names the exact record, so a sidecar is never looked up as an |
| + instance (199-11). |
| + - `incarnation` is `provision.Incarnation`. That is |
| + `maintenanceguard.RecordIdentity` over account, app, host and |
| + CreatedAt. A record with a zero CreatedAt instead gets the |
| + domain-separated legacy identity from rev 3 §2. |
| +- **`touched` only grows** for a holder, and is cleared only when the host |
| + becomes free. A component adds its region or box with a CAS **before** |
| + its first write of that incarnation's data there. That makes "where is |
| + this incarnation's data" a durable answer (199-3, 199-9, M2-5). |
| +- **`obligations` are durable chores.** Each names who must do it. A host |
| + cannot become free while any remain (M2-3, M2-4, M2-5). |
| + |
| +The claim is small, rewritten whole, and never listed on a hot path. |
| +Readers `GET` it by host. |
| + |
| +## 3. Invariants |
| + |
| +**C1. One holder.** At most one incarnation holds a host. A record whose |
| +incarnation is not the claim's holder is an **orphan**: |
| + |
| +- it is never admitted; |
| +- reconcile doesn't serve it; |
| +- the door refuses it; |
| +- wake refuses it. |
| + |
| +**C2. Claim before record, record before release.** |
| +- **Create** CASes the claim from free (or absent; see §7) to |
| + `{holder: new, op: serving}` **before** `CreateRecord`. |
| +- **Delete** CASes `op` to `deleting` **before** removing the record, and |
| + removes it with `release.DeleteIfMatch` on the ETag it read **under** |
| + the claim (199-4). |
| +- The holder is cleared only at the end of the park (§5). |
| + |
| +**C3. Mutation under the current operation.** Any change to the host's |
| +storage is made only by the claim's current `op.id`. Steady-state |
| +replication by the admitted serving process is the one exception. The |
| +storage is: |
| + |
| +- `instances/<host>/` and `moves/<host>/` in any region; |
| +- `parked/`; |
| +- the local database, `.lineage`/generation markers and the working |
| + directory. |
| + |
| +Each destructive step is preceded by re-reading the claim and confirming |
| +that `gen`'s op id and phase are unchanged. The worker must also hold the |
| +replica lease of every location it writes, acquired under the claim, and |
| +act only before that lease's `SafeUntil` (§3.1). That reread does not |
| +serialise anything on its own (M2 I-9). The lease does that, and the |
| +reread proves the lease was taken for the current operation. |
| + |
| +**C4. Admission is checked fresh.** These actors read the claim directly, |
| +never from a listing snapshot (199-8): |
| + |
| +- wake; |
| +- reconcile's decision to create or keep a provisioned row; |
| +- the db door. |
| + |
| +Each admits only when the holder's incarnation equals the record's and |
| +`op.kind` permits serving (§4). Unknown or unreadable means refuse, not |
| +admit. |
| + |
| +**C5. Revocation needs stop evidence.** An operation that needs serving |
| +stopped does not trust admission checks to have caught every running |
| +process. It obtains stop evidence (§6) from every box in |
| +`touched.boxes` that may be running the host, before its first |
| +destructive step. |
| + |
| +**C6. Read errors never act.** A claim, record or listing read that fails, |
| +or that returns something malformed, makes the actor wait or refuse. It |
| +is never treated as absence (199-8, 199-15). |
| + |
| +### 3.1 Leases, time and the assumption we make |
| + |
| +S3 cannot fence a delete by a token. Exclusion over storage therefore |
| +comes from the litestream lease (`lock.json`, conditional writes), held |
| +by the operation's worker. A worker that cannot renew must stop. |
| + |
| +Every destructive call checks `now < SafeUntil` immediately before it is |
| +issued, and runs under a context whose deadline is `SafeUntil`. |
| + |
| +**Assumption, stated plainly:** box clocks are NTP-synced, process pauses |
| +are shorter than the lease's safety margin, and an in-flight S3 request |
| +that outlives its context is cancelled client-side. This is the same |
| +assumption the activator's lease already makes for replication. A design |
| +needing more needs an S3 primitive the platform doesn't have. |
| + |
| +## 4. Operation kinds |
| + |
| +| `op.kind` | Serving admitted? | Owner spec | Ends in | |
| +|---|---|---|---| |
| +| `serving` | yes, in the record's region | this | any op below | |
| +| `deleting` | no | this (§5) | `parking` | |
| +| `parking` | no | this (§5) | free | |
| +| `moving` | source: per the moves spec's phase; target: no | move-any-app | `serving`, or `reverting` | |
| +| `landing` | target only once that spec's phase allows | move-any-app | `serving` | |
| +| `reverting` / `restoring` | per that spec | move-any-app | `serving` | |
| +| `pushing` | no | later push spec | `serving` | |
| + |
| +Starting any operation CASes from `serving`. One operation runs at a |
| +time, so a delete during a move is refused until the move resolves, and |
| +the reverse (199-6). Operator recovery has its own command: |
| + |
| + carlos ops host resolve <host> [--declare-lost <box>] |
| + |
| +`--declare-lost` removes a lost box's `park-local` obligations and its |
| +entry in `touched.boxes`. Its data is gone with it. The moves spec's own |
| +restriction applies: it is refused after a move commits (M2 §18). |
| + |
| +## 5. Delete and park (#199) |
| + |
| +Run by the console's delete doors (instance and sidecar), and by |
| +`PurgeApp`, one per record, in both its console and box callers. The four |
| +other record-deleting call sites, domain detach (UI and API), attach |
| +rollback and create rollback (199-10), go through the same shared helper, |
| +`provision.RetireRecord`. |
| + |
| +1. **Begin.** CAS `op = {deleting, id}` from `serving`, refused |
| + otherwise. This box or console drains the host's open db-door streams. |
| + The door is in-process on the one console; §8 covers more than one. |
| +2. **Remove the record.** Read it, check its incarnation is the holder, |
| + then `DeleteIfMatch` with that ETag. |
| +3. **Write the obligations** and move to `parking`, in one CAS: |
| + - one `park-replica` for each region in `touched.regions`; |
| + - one `park-local` for each box in `touched.boxes`. |
| +4. **Each obligation is done by its owner**, the box in that region or |
| + that box: |
| + 1. **Stop evidence** (§6) for the host on this box. |
| + 2. **Lease.** Acquire the lease on every location it will touch: |
| + `instances/<host>/` and, separately, `moves/<host>/`. If a holder |
| + is renewing, wait. |
| + 3. **Replica park** (199-7, 199-16, M2-7): |
| + - **Destination:** `parked/<incarnation>/<host>/instances/…` and |
| + `…/moves/…`, keeping each subtree's relative keys. |
| + - **Copy and manifest:** copy each key with a source-ETag |
| + condition, and record source key, ETag, size and destination |
| + key in `manifest.json`. Write the manifest last. |
| + - **Retry:** a manifest that is present is never rewritten. |
| + - **Delete:** for each manifest entry, `HEAD` the source and |
| + delete it only if its ETag still matches. A mismatch, or a source |
| + key absent from the manifest, stops the park. The obligation |
| + stays, an alert goes up, and the claim records the conflict. |
| + - **Last:** delete this worker's own `lock.json`, conditional on |
| + its own lease body. |
| + 4. **Local park** (199-13, 199-14): |
| + - **Inventory.** The obligation carries the inventory. It was |
| + written into the claim's journal when the box first touched the |
| + host, and refreshed on any layout change: `DBPath`, layout, |
| + working directory, `.lineage`/generation markers, `-wal`, `-shm`, |
| + litestream meta. |
| + - **Move each artifact** to `<DataDir>/.parked/<incarnation>/<host>/` |
| + with a non-overwriting rename (`renameat2` `RENAME_NOREPLACE`). |
| + - **Journal and sync.** Record per-artifact progress in a journal |
| + file beside them, and fsync the directories. |
| + - **Errors.** `EXDEV`, or any other error, leaves the obligation in |
| + place. |
| + - **Verdict.** Delete this host's dbverify verdict. |
| + 5. **Finish.** CAS the obligation out of the claim, conditional on the |
| + claim still showing `parking` with this op id. |
| +5. **Free.** When `obligations` is empty, CAS the claim to free: |
| + holder, op and touched all cleared, `gen` incremented. The tombstone of |
| + rev 3 is not needed: the claim's last state before free, and each |
| + `manifest.json`, are the record of what was parked. |
| + |
| +**A rolled-back move's target replica (M2-5).** The moves spec adds a |
| +`park-replica` obligation for the target region on rollback. It stays |
| +`touched`, so a later delete finds it either way. |
| + |
| +## 6. Stop evidence that survives a restart (M2-1, M2-2) |
| + |
| +**Evidence is computed from system state, never from activator memory.** |
| +For a host on a box, it holds when all of these are true: |
| + |
| +1. **The cgroup is empty.** The per-host cgroup that every exec child is |
| + placed in (`cgroup.ExecName(host)` under the agent's delegated subtree, |
| + backend_exec.go:96) reports `populated 0` in `cgroup.events`, or |
| + doesn't exist. A unit-backed host's `carlos-app@<host>.service` |
| + cgroup is checked the same way. This covers descendants that outlive |
| + the main process (M2-1). The fence is installed first, and the check |
| + waits for empty, with the operation's own deadline. |
| +2. **No live lease.** This box holds no lease for any location of the |
| + host. If `lock.json` exists, it names another holder and that holder |
| + has expired. |
| +3. **No replicator.** No replicator for the host's database is open in |
| + this agent process. A fresh process has none, by construction. |
| + |
| +**A box without cgroup delegation** cannot produce evidence. Lifecycle |
| +operations on its hosts are refused with a reason. This covers old unit |
| +files and hosts with `Cgroups == nil`. |
| + |
| +**After a restart,** evidence is simply recomputed. "Never spawned" is |
| +the same check: an absent cgroup and no lease. The activator's in-memory |
| +`stopHistory` is an optimisation, never a requirement. `FenceAndStop` |
| +gains a path that returns evidence from system state when its memory is |
| +empty, instead of `ErrStopUnproven` (M2-2). |
| + |
| +## 7. Adoption of existing hosts (R2) |
| + |
| +Hosts that exist before this release have no claim. The first actor that |
| +needs one creates it, create-only, as `{holder: current record, op: |
| +serving, touched: {regions: [record region], boxes: [this box, if it has |
| +local state]}}`: |
| + |
| +- the owning box's reconcile; |
| +- any door. |
| + |
| +A box that later finds local state for an adopted host adds itself to |
| +`touched` before acting. |
| + |
| +**Create on a host with no claim and no record** uses the #199 stopgap's |
| +check: a replica prefix that holds data in the console's region refuses |
| +the create. Leftovers there have no recorded owner, and R2 adopts only |
| +for a **current** record. Elsewhere, the create proceeds, adopted as |
| +today. |
| + |
| +## 8. Admission points, concretely |
| + |
| +- **`doWake`, before `Acquire`** (the Classifier refusal precedent at |
| + activator.go:1090). Read the claim, and require: |
| + - holder = the row's record incarnation; |
| + - `op.kind` allows serving here; |
| + - this box is in `touched.boxes`, CAS-adding it if not. |
| + |
| + Unreadable means the wake fails, retryably. |
| +- **Reconcile row creation and keeping.** The row gains `incarnation` |
| + and `claim_gen` columns. A provisioned row whose incarnation is not the |
| + claim's holder is retired with stop evidence, whatever its account and |
| + app. This covers same-app recreation (199-1). Rows that a running |
| + operation lists as participants are retained until its obligations |
| + finish (M2-4). |
| +- **The db door.** Read the claim on the listing request, and bind the |
| + restore to `gen`. Each later file `GET` rechecks the claim through a |
| + short-lived cache, and refuses once `gen` moves (199-2). Delete step 1 |
| + also drains the door. With more than one console, step 1 must wait out |
| + the cache lifetime instead. |
| +- **Create.** §3 C2. |
| + |
| +## 9. Rollout |
| + |
| +Old binaries don't know the claim. Two rules make that safe: |
| + |
| +- **Fail closed toward old binaries.** A delete by an old console leaves |
| + the claim holding an incarnation whose record is gone. Admission |
| + refuses the host from then on, and create refuses too. The host is |
| + stuck until `carlos ops host resolve` runs. That is visible and safe: |
| + never a silent inheritance. |
| +- **Floor.** New lifecycle operations (delete-park, and moves) run only |
| + while the deployment's protocol floor, the moves spec's floor object, |
| + lists every box and console at or above this release. Below the floor |
| + they are refused, and deletes keep today's behaviour plus the stopgap. |
| + |
| +## 10. Simulation |
| + |
| +Before planning, the claim, delete-park and the moves operations get a |
| +model-level simulation in the shape of the moves spec's: random |
| +interleavings and crashes at every side effect. It covers: |
| + |
| +- surviving descendants; |
| +- empty activator memory; |
| +- two consoles; |
| +- delete during each move phase; |
| +- recreate during park; |
| +- reconcile running before release; |
| +- stale door streams. |
| + |
| +It runs at high seed counts, with mutation checks that drop each guard in |
| +turn and confirm the simulation catches it. |
diff --git a/docs/superpowers/specs/2026-10-02-replica-ownership-design.md b/docs/superpowers/specs/2026-10-02-replica-ownership-design.md| index fb8a9de04..3d7382026 100644 |
| --- a/docs/superpowers/specs/2026-10-02-replica-ownership-design.md |
| +++ b/docs/superpowers/specs/2026-10-02-replica-ownership-design.md |
| @@ -1,3 +1,5 @@ |
| +> Superseded from §2 onward by 2026-10-02-host-lifecycle-claim-design.md (Paul, 2026-10-02: one shared per-host claim). Kept for its problem statement and history. |
| + |
| # Replica ownership (#199): park a deleted instance's data |
| Design, 2026-10-02, **rev 3**. Closes amadan #199. `carlos db push` is |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R1.md b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R1.md| new file mode 100644 |
| index 000000000..3f5046e1e |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R1.md |
| @@ -0,0 +1,54 @@ |
| +**Rev 1 is not safe to implement as written.** R1–R3 stand; the gaps are in enforcing them. |
| + |
| +Reviewed `2fef212c` against `da866a92` and `b014d57d`. HEAD changes only the spec. Findings below use main’s code unless marked `b014d57d`; #194 refers to the findings described in the spec. |
| + |
| +1. **P1 — A3: cross-app recreation bypasses the proposed fix.** Delete app A’s record and recreate the host under app B between passes. `convergeOwned` runs only for matching account/app (`internal/provision/reconcile.go:947`); the mismatch branch leaves A’s row alone (`:1031`), and removal retains it because the **host** exists (`:1141`). A’s child keeps serving. **Fix:** compare record identity before the ownership branches; fence and prove retirement of the obsolete row before replacing it. |
| + |
| +2. **P1 — A2–A3: sidecars have separate paths.** Updating `routeFor`/`convergeOwned` does not update `sidecarRouteFor` or `convergeSidecar` (`internal/provision/sidecar_reconcile.go:131`, `:467`). Delete/recreate between passes leaves the sidecar running against its previous database. **Fix:** apply incarnation materialization and retirement to both record families. Fence **before** stopping, require stop evidence, and specify durable recovery and fence release; ordinary retirement is asynchronous (`internal/activator/activator.go:2624`). |
| + |
| +3. **P1 — A4 step 1: warm-cache acceptance bypasses R1.** Local stamp matches B—or is absent and adopted as B—but the replica belongs to still-live A. The warm return skips both staging and `OwnerLive`; subsequent replication writes B into A’s lineage. The existing early return is `internal/activator/backend_exec.go:267`. **Fix:** validate remote ownership and staging before accepting any warm database. Matching local identity alone cannot authorize replication. |
| + |
| +4. **P1 — A4 step 3: archive retries can destroy the archive.** Mirror A successfully, delete some source keys, then crash. Retrying `MirrorPrefix` against the same archive deletes destination keys now absent from the partially deleted source (`internal/lease/mirror.go:222`). Alternatively, delete `owner.json` first and crash: surviving A LTX is now “legacy” and gets adopted by B. **Fix:** persist an archive transaction and immutable verified manifest; never mirror again after deletion starts. Retain an ownership/transition marker until deletion completes, then CAS to B. |
| + |
| +5. **P1 — A4 `OwnerLive`: wrong record family can authorize destruction.** A sidecar lives at `sidecars/<name>.json`, not `RecordKey(account, app, host)` (`internal/provision/sidecar.go:133`). The proposed lookup returns absent for a live sidecar. **Fix:** stamp the validated record family/key and use its exact reader. Only authoritative absence or a different validated incarnation means dead; malformed records, wrong paths, permissions, and read errors must refuse. The `release.Store` interface states no consistency guarantee (`internal/release/store.go:37`), so cached/list-derived absence cannot authorize deletion; unsupported consistency models need a durable retirement protocol. |
| + |
| +6. **P1 — A4 step 2 / B4.5: staging can overwrite before validation.** “Promotion runs as today” means `MirrorPrefix` overwrites/deletes the destination—including its owner—before returning (`internal/activator/backend_exec.go:1606`). Checking the staged owner afterward is too late. B4 also checks ownership **before** promotion can change it. **Fix:** validate staging and destination first, bind staging to incarnation **and move ID**, and check again after promotion. Staging removal must be verified: today’s `dropStagedReplica` ignores deletion failures (`:1624`), allowing a later cold wake to resurrect pre-push data. |
| + |
| +7. **P1 — A2/A4: file-before-stamp makes new data look legacy.** Restore or Seed writes A’s database; crash before `.owner`; recreate as B; R2 now adopts A’s newly created but unstamped file. A fresh instance with **no Seed** creates its database in `Start`, which A2 never mentions (`internal/activator/activator.go:1168`). **Fix:** persist an incarnation-bound “creation in progress” marker before any database-writing process or restore, then atomically finalize it. Recovery must distinguish interrupted creation from genuinely legacy data. |
| + |
| +8. **P2 — A1: zero timestamps remain creatable.** `RecordIdentity` returns `(string, error)` and rejects zero (`internal/maintenanceguard/types.go:62`). Standard instance/domain/replica/sidecar constructors set timestamps, and instance update helpers preserve them. But `CreateRecord` and `CreateSidecarRecord` accept zero unchanged (`internal/provision/provision.go:820`; `sidecar.go:322`). Repeated zero-date recreations therefore share the sentinel. **Fix:** one shared incarnation helper; enforce nonzero fresh creation identity at every create boundary while retaining zero support only for existing records. Do not change maintenance identity semantics implicitly. |
| + |
| +9. **P1 — A6: create-only is not lifecycle synchronization.** The startup sweep uses a registry snapshot without the replica lease. It can stamp during archive/promotion, or stamp a prefix using an obsolete row. A successful `If-None-Match` proves only that the key was absent. **Fix:** sweep under the same lease/transition protocol, revalidate the authoritative record, and reread the winning stamp after conflicts or ambiguous puts. Retry failures beyond the single startup pass. |
| + |
| +10. **P1 — A4 / B4.12: incarnation does not invalidate another box’s warm cache.** Box A pushes and purges its local file. Box B retains an older local file with the **same incarnation**; its next wake accepts it and can overwrite the push. Moves have the same problem when the best-effort source purge fails (`internal/host/movespass.go:492`). **Fix:** add a database-generation/cache-validity token, changed by replacement and residence handoff, checked remotely before warm acceptance. Use error-returning purge in the move path too. After staged promotion, an older same-incarnation local tail must not be checkpointed over the promoted database. |
| + |
| +11. **P1 — A5: owner check and object read are separate.** An old owner’s request passes `dbGate`; the host is recreated and its LTX keys reused; `OpenLTXFile` then reads the new owner’s object (`internal/console/apidb.go:287`, `:312`). **Fix:** bind reads to immutable incarnation/generation storage or validated object versions. A preflight stamp read alone does not bind the bytes subsequently served. |
| + |
| +12. **P1 — A4 / B4.3–12: the lease contract is incomplete.** `Held` renews from acquisition; merely exceeding 15 minutes is not the bug. Renewal loss is. Raw mirror/delete/put operations have no Held parameter or automatic loss cancellation. Moreover, `Held.Replicate` **spends** the handle (`internal/lease/lease.go:249`), while `Held.Checkpoint` retains it (`internal/lease/checkpoint.go:17`). `Restore` currently receives no lease handle either. **Fix:** explicitly use retained checkpoints, pass ownership evidence into the gate, cancel and bound operations on `Lost`/`SafeUntil`, and stop all destructive work on loss. Delayed writes must not be able to modify a successor’s generation. |
| + |
| +13. **P1 — B4.1–3 / A4: a local status does not fence another box.** Push reaches `replacing`, clears the replica, then crashes. Another box still has a ready row; after lease expiry it acquires, sees an equal owner with no LTX, and seeds. **Fix:** durable incarnation-bound push denial checked by every wake/Seed path, including newly minted rows, before touching data. Revalidate after acquiring the lease. The current wake refreshes the registry but does not enforce its status there (`internal/activator/activator.go:1071`). |
| + |
| +14. **P1 — B2/B4/B5: CAS transitions do not guard intervening side effects.** Worker B reads `replacing`, pauses; worker A completes; a wake writes new data; B later acquires and replays the stale replacement. Incarnation still matches. Likewise, a stale move pass can recreate staging after push removed it (`internal/host/movespass.go:377`). **Fix:** reread request ID, step, incarnation and residence under ownership immediately before destructive work. Use a shared lifecycle claim for move/push/archive, not just mutual refusal at request creation. Sharing one listing does not make that listing current. |
| + |
| +15. **P1 — B4 resume/restart: “no in-flight request” does not prove safety.** `ListAllRecords` skips malformed/path-disagreeing records (`internal/provision/reconcile.go:1320`); cleanup would reopen their held routes. At `replacing`, only missing upload gets special treatment: resumed disk/validation failures must also retain denial. B4.4’s “nothing changed yet” is false on a resumed swap. **Fix:** after `replacing`, require positive completion or recovery evidence before releasing anything. Missing, corrupt, replaced or unreadable control state must retain the hold. |
| + |
| +16. **P2 — A4.3 / B4.7: ETag equality rejects valid backups.** Multipart LTX copied with `CopyObject` gets a different ETag; this is why `MirrorPrefix` already has receipts (`internal/lease/mirror.go:89`). The proposed verification wedges ordinary larger databases. **Fix:** verify source-version/size → destination-version receipts or checksums against a stable manifest; require zero deferred copies. “Every current source key exists” is insufficient if a writer can change the source during verification. |
| + |
| +17. **P1 — B3: acceptance does not bind the upload to its observed residence.** Begin upload in region A; a move completes to B during streaming; `DBRequest` rereads a record with `Move=nil` and accepts it. B looks in its bucket while the upload remains in A. Delete/recreate creates an analogous identity race. Tony’s mutation checks move/role/in-flight state, not the observed incarnation or region (`b014d57d:internal/provision/dbtransfer.go:123`). **Fix:** bind the acceptance CAS to the pre-upload incarnation, residence and class; refuse changed records and safely clean up the object. |
| + |
| +18. **P1 — B4.13: orphan sweep can delete a committed upload.** A pass lists no request; the console commits one; the pass deletes the sufficiently old upload using its earlier negative result. Another worker may already have entered `replacing`. **Fix:** use a durable upload reservation with CAS handoff to the request and CAS expiry for GC. An hour’s age and even a fresh unprotected reread do not eliminate the race. |
| + |
| +19. **P2 — B3/B7: streaming and undo need explicit completion contracts.** Streaming alone does not specify aborting multipart uploads, rejecting cap overflow rather than publishing a capped prefix, or binding the completed object’s size/checksum/version to the request. Separately, missing-upload recovery says “pull then push,” but B1 refuses a push while the broken request remains in flight. **Fix:** publish only verified completed uploads; specify abort cleanup and permissions. Add a CLI/API recovery transition that replaces the stranded request while preserving its hold. |
| + |
| +20. **P1 — §4: rollout permits incompatible writers.** Old consoles can deserialize and rewrite records without `DB`, erase the denial, or start moves without checking it. Old boxes ignore stamps; Tony’s worker deletes the entire prefix, including owner and lock (`b014d57d:internal/host/dbpass.go:591`). Old stamp-less staging can also delete a destination stamp during promotion. **Fix:** capability-gate activation across **all** serving boxes, consoles and record writers, including rollback versions. Complete adoption before enabling protected lifecycle operations. “Boxes before console” alone is insufficient. |
| + |
| +The #194 fixes therefore remain incomplete: **P1-1** needs loss-safe ownership and stale-worker protection; **P1-2** needs validated, durably consumed staging; **P1-3** fixes the direct purge error but misses other-box caches and move purge; **P1-4** improves ambiguous acceptance but leaves GC/recovery races; **P2-5** fixes `SetStatus` error handling but makes unsafe absence-based release possible; **P2-6** improves memory/disk bounds but leaves streaming completion undefined; **P2-7** is addressed as a listing-cost change, not as synchronization. |
| + |
| +**Citation audit:** all explicit anchors were checked. Corrections: |
| + |
| +- **P2 — A7/§4 IAM:** `infra/modules/region-host/main.tf:166` already grants object access to the **whole** own data bucket; `:174` grants bucket listing. The claimed existing `instances/*` restriction is false. No separate narrow console role is demonstrated there. If introducing one, include backup GET/LIST and multipart-abort permissions—not just upload PUT/DELETE. |
| +- **P3:** `instances_cli.go:135` checks missing domain; shape validation is `:146`. Shared account namespace evidence is `internal/console/apiinstances.go:698` and `:1692`, rather than `:693` alone. Domain-attach timestamp is `domainattach.go:181`; `:183` calls create. Sqid non-reuse is explained at `ownership.go:636` and implemented through retained identity/tombstones. |
| +- **P3:** Litestream’s actual layout is `<prefix>/ltx/%04x/`; the conclusion that root `owner.json` is ignored by LTX readers is correct. |
| +- The remaining cited anchors support their claims, including `reconcile.go:1141`. `backend_exec.go:285` really is stale: moves leave the source replica behind. Versioning/no lifecycle is supported by the checked-in modules. `dbtransfer.go` belongs specifically to `b014d57d`. |
| + |
| +Static review only; no files changed. |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R2.md b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R2.md| new file mode 100644 |
| index 000000000..82f9809a8 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R2.md |
| @@ -0,0 +1,141 @@ |
| +**Rev 2 still fails review.** Rekeying removes the archive protocol’s hazards, but publication, retirement, rollback and rollout remain unsafe. |
| + |
| +Reviewed `ef0b7e45` against code base `da866a92` and `origin/feat/db-push-pull` at `b014d57d`. Amadan access was blocked by the sandbox; #194 below means the findings quoted in rev 1. R1–R3 are accepted. |
| + |
| +1. **P1 — §2.3, §7: the epoch cannot exclude old writers.** |
| + An old console rewrites a pushed record and drops `DB`. A post-epoch record returns to `g0`; a pre-epoch record migrated by a move returns to **legacy**, directly contradicting §2.3. N’s startup check cannot constrain a pre-N executable. Old agents also retain recursive delete authority. |
| + **Fix:** enforce the writer cutoff outside those executables—withdraw old credentials/admission, cover CLI and offline fleet writers, and make epoch/read failures fail closed. A release note and version observations are insufficient. Existing writers decode and re-marshal the known struct: [provision.go:872](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/provision.go:872). |
| + |
| +2. **P1 — §2.1–2.3: wall-clock time is not a storage-version discriminator.** |
| + After activation, a console with a slow clock recreates a host with `CreatedAt < epoch`; `DB=nil` selects its predecessor’s legacy replica. Conversely, a pre-existing record timestamped ahead of the epoch switches to empty `g0`, violating R2. Nonzero checks prevent neither case. |
| + **Fix:** explicitly stamp storage format on creation and explicitly identify/adopt the existing population. Allocate incarnation identity afresh; do not infer creation order from caller-supplied time. The console currently uses its local clock: [apiinstances.go:888](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/console/apiinstances.go:888). |
| + |
| +3. **P1 — §2.4, §2.6, §7.1: initial metadata backfill looks like retirement.** |
| + Every existing row initially has empty `incarnation`/`storage_key`. Comparing it with the rendered record triggers stop → purge before legacy adoption. A sleeping instance with an unreplicated local tail loses it during the supposedly dormant release. |
| + **Fix:** specify a separate, non-destructive initial adoption transaction. Empty metadata is not evidence of an incarnation change. |
| + |
| +4. **P1 — §2.6: the retirement placement is internally impossible, and moving it blindly grants excess authority.** |
| + `convergeOwned` and `convergeSidecar` are called **inside** matching-owner branches. Adding checks there still misses another-app recreation. Moving retirement before those branches without scope guards lets a conflicting record retire an unmanaged/platform route. |
| + **Fix:** put the check in both outer reconcile loops; require the pass’s kind, scope and existing provisioning authority, preserve unmanaged-route conflict checks, and revalidate the record and row before replacement. See [reconcile.go:947](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/reconcile.go:947) and [sidecar_reconcile.go:400](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/sidecar_reconcile.go:400). |
| + |
| +5. **P1 — §2.5, §2.6: purging unproven lineage destroys data R1 requires retaining.** |
| + Seed/Start creates a database; the box crashes before first sync and `.lineage`. Next wake purges the sole copy. Likewise, recreation purges an old incarnation’s unsynced tail; an unreadable stamp can purge a live incarnation’s good database. The assertion that the first-boot file is already replicated is false: replication starts **after** Start. |
| + **Fix:** quarantine uncertain/retired local artifacts in-region before replacement; distinguish initialization ownership from completed replication. Require verified checkpointing where deletion depends on remote durability. See [activator.go:1168](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/activator/activator.go:1168) and [activator.go:1273](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/activator/activator.go:1273). |
| + |
| +6. **P1 — §5.4 steps 1–5: no exclusion or durability proof for FromGen.** |
| + Quiescence is local. Another box can keep writing FromGen, including immediately after the second TXID listing. Separately, local stop can fail its final checkpoint, yet `Quiesced` becomes true; step 6 then deletes the only tail and the “backup” is incomplete. |
| + **Fix:** obtain current-generation exclusion, prove process stop, checkpoint the matching local tail successfully, and hold exclusion through the pointer commit. Two TXID observations cannot replace that protocol. [finishStop:2164](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/activator/activator.go:2164), [Quiesced:2675](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/activator/activator.go:2675). |
| + |
| +7. **P1 — §2.4–2.6, §5.4: a stale box’s row remains sufficient admission authority.** |
| + Box A commits a new incarnation/generation. Box B still has the old row and matching `.lineage`; it acquires the **old prefix’s independent lease** and serves old data. Changing the storage key removed contention between these writers without adding an authoritative serving-generation check. |
| + **Fix:** bind wake and continued serving to a fenced current incarnation/generation, with a defined revocation handoff. Registry convergence alone is insufficient. Wake currently refreshes only the local registry: [activator.go:1071](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/activator/activator.go:1071). |
| + |
| +8. **P1 — §5.4 step 4, §6 #14: `current != ToGen` does not mean “never committed.”** |
| + A delayed worker resumes after its push committed G1, applications wrote to G1, and another push committed G2. It sees `current != G1` and re-replicates the original upload into G1, damaging the retained backup. A concurrent rollback can make that location live again. |
| + **Fix:** require the exact active request, precommit state and full From pointer under exclusion; permanently record publication and forbid subsequent preparation writes. Serialize publication with preparation completion. |
| + |
| +9. **P1 — §5.4 steps 5–7 and restart: replay can purge a resumed live database.** |
| + Commit succeeds; worker sets `ready`, lifts the fence, then crashes before marking done. The app wakes and writes ToGen. A retry of step 6 purges that live local file. Independently, §2.6 can observe the commit, replace the row and lift its fence while the push worker still expects the host held. “Already points at ToGen, stop” also fails to specify completion of steps 6–7. |
| + **Fix:** durable commit/converge/release states, one lifecycle owner, incarnation/request-conditional row mutations, and explicit recovery for an ambiguous commit. Never replay purge after admission resumes. |
| + |
| +10. **P1 — §5.2, §5.4 step 5: residence binding expires after acceptance.** |
| + Upload acceptance succeeds in IE. `SetRecordRegion` subsequently changes residence without using the move protocol. Commit checks ID/incarnation/FromGen but not Region; the pointer becomes live in another region where ToGen was never uploaded. First wake can seed empty. |
| + **Fix:** recheck residence and the complete storage descriptor at commit; all placement/class-changing writers must exclude an active transfer. Existing [SetRecordRegion:853](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/provision.go:853) has no such guard. |
| + |
| +11. **P1 — §4: publishing staging as live retains both foreign write authority and stale mirror writers.** |
| + After `MoveComplete`, sibling regions still have `PutObject`/`DeleteObject` over the live `moves/*` location. A delayed preparation round can also resume after landing and mirror over target writes; `MirrorPrefix` deletes destination-only objects. Unique move generations do not exclude another worker for the **same move**. Legacy parent-prefix cleanup can reach nested v2 locations too. |
| + **Fix:** introduce a sealed landing handoff into a live namespace inaccessible to staging writers; bind every round/completion to an immutable move ID and source identity. Drain old-layout moves at activation and keep legacy recursive cleanup disjoint. [IAM:217](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/infra/modules/region-host/main.tf:217), [mirror.go:222](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/lease/mirror.go:222), [movespass.go:377](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/host/movespass.go:377). |
| + |
| +12. **P1 — §5.5: rollback reintroduces storage-key reuse.** |
| + Box A caches G0 at TXID 10. Box B advances G0 to 20, pushes G1, then rolls back to G0. A never observed G1; its `.lineage` matches again and it serves TXID 10. Ordinary same-generation takeover also lacks a warm-cache freshness proof. |
| + **Fix:** rollback materializes a fresh generation from the retained snapshot; never reactivate a previous writable prefix. Validate remote freshness on cross-box takeover. |
| + |
| +13. **P1 — §5.4 retention, §5.5: “never the current pointer” is asserted, not enforced.** |
| + GC selects a noncurrent generation; rollback selects it before deletion finishes. GC deletes live data. There is likewise no specified pin for a prepared ToGen or an in-flight historical pull. |
| + **Fix:** define an authoritative generation catalogue with atomic pins/tombstones. Rollback cannot select a tombstoned generation; GC cannot select current, prepared or pinned generations. Include Root and region in retained descriptors. |
| + |
| +14. **P2 — §2.1, §4: the zero-CreatedAt sentinel disappeared.** |
| + Rev 1 defined it; rev 2 calls `RecordIdentity`, which rejects zero. Legacy records still need registry identity, and a legacy move now requires a v2 incarnation. |
| + **Fix:** restore an explicit, deterministic, domain-separated legacy identity and specify its migration behavior. Both record structs contain CreatedAt, but current create helpers accept zero. [types.go:62](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/maintenanceguard/types.go:62), [sidecar.go:322](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/sidecar.go:322). |
| + |
| +15. **P2 — §5.3, §6 #18: upload deletion still mistakes uncertainty for absence.** |
| + CAS lands, response is lost, reread fails: “otherwise delete” destroys an accepted upload. The sweep can similarly miss a malformed record or race delayed acceptance. A day’s age is not exclusion. Step 3’s checksum is not durable recovery evidence after a crash loses scratch. |
| + **Fix:** preserve uploads on uncertain reads; coordinate acceptance and GC through a reservation/tombstone protocol. Fail precommit requests safely when recovery bytes are genuinely unavailable. `ListAllRecords` deliberately skips malformed records: [reconcile.go:1320](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/reconcile.go:1320). |
| + |
| +16. **P2 — §2.2, §5.2: the persistent wire shape is unresolved.** |
| + §2.2 assigns `Record.DB` to `DBPointer`; Tony’s `DBTransfer` and every transfer transition already use `Record.DB`. No second request field is defined. Main’s MoveState also has no move ID, despite §4 depending on one. |
| + **Fix:** specify separate pointer/request fields, strict decoding, immutable move IDs, and guarded transitions before implementation. Evidence: `b014d57d:internal/provision/dbtransfer.go:77,145`; [MoveState:907](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/provision/provision.go:907). |
| + |
| +17. **P2 — §2.5b: directory isolation has an ordering hole and an explicit coverage gap.** |
| + “Before the app starts” permits directory retirement after Restore or Seed has already written there. Flat rows explicitly retain arbitrary prior-incarnation files. |
| + **Fix:** check/prepare the directory before **any** restore or seed; define crash recovery and collision-safe quarantine for absent stamps. Require isolated layout, or refuse recreation where R1 cannot be enforced. |
| + |
| +**Section 6 does not discharge the findings:** |
| + |
| +| Rev 1 finding | Verdict | |
| +|---|---| |
| +| 1, 2 | Open: retirement is placed behind the ownership branches; findings 3–4 above. | |
| +| 3 | Partial: matching lineage works only with a current row and non-reused key; 5, 7, 12. | |
| +| 4 | Removed: no archive retry protocol remains. | |
| +| 5 | Removed: no OwnerLive lookup remains. | |
| +| 6 | Partial: v2 promotion disappears, but live staging remains mutable; 11. | |
| +| 7 | Partial: prior-incarnation adoption avoided, sole-copy deletion introduced; 5. | |
| +| 8 | Partial: proposed create refusal helps; legacy zero identity and old writers remain; 1, 14. | |
| +| 9 | Removed: no adoption sweep remains. | |
| +| 10 | Open: stale rows and rollback defeat the key check; 7, 12. | |
| +| 11 | Addressed for an individual v2 door request using its resolved immutable prefix. | |
| +| 12 | Open: source exclusion and publication fencing remain necessary; 6, 8. | |
| +| 13 | Old empty-prefix swap removed; broader two-box safety remains open; 7, 10, 13. | |
| +| 14 | Open: uniqueness does not prevent replay into a previously committed generation; 8. | |
| +| 15 | Partial: readable record is insufficient without matching row identity/pointer and lifecycle state; 9. | |
| +| 16 | Archive ETag requirement removed; move-copy safety still matters; 11. | |
| +| 17 | Partial: acceptance bound, commit not fully bound; 10. | |
| +| 18 | Open: age/checksum do not establish safe deletion; 15. | |
| +| 19 | Multipart completion addressed; recovery/rollback remain unsafe; 9, 12, 15. | |
| +| 20 | Open: epoch does not fence old executables; 1–2. | |
| + |
| +**Against the quoted #194 findings:** |
| + |
| +| Finding | Rev 2 | |
| +|---|---| |
| +| P1-1: swap lease | Partial. ToGen lease protects preparation, not source quiescence or the whole commit. | |
| +| P1-2: staging resurrection | Addressed for strictly v2 push with legacy promotion disabled; §4 introduces finding 11. | |
| +| P1-3: ignored purge failures | Error-returning purge addresses the immediate defect; replay safety remains open. | |
| +| P1-4: ambiguous acceptance/missing upload | Partial; uncertain reread and recovery still fail. | |
| +| P2-5: release/status recovery | Status-write error addressed; crash after release and competing convergence remain unsafe. | |
| +| P2-6: memory/disk bounds | Streaming cap and disk check address the stated problems. Specify multipart cleanup after process death too. | |
| +| P2-7: duplicate record listings | Addressed by §5.7. | |
| + |
| +**Citation audit:** All explicit numeric anchors were checked. The principal qualifications are: |
| + |
| +- `sidecar_reconcile.go:131/:467` identify renderer/converger functions; the ownership branch is **400–412**. |
| +- `activator.go:1168` explicitly says Start precedes replication; it contradicts §2.5’s crash argument. |
| +- `maintenance.go:81` protects ordinary **Record updates** through the guarded helpers. It is not a SidecarRecord guard or protection against old/raw writers. Replica records do pass through `CreateRecord`. |
| +- Create-function anchors are correct, but zero refusal is a **proposed change**, not existing behavior. |
| +- `backend_exec.go:285` is indeed stale; staging actually wins at **1581**. |
| +- The own-bucket IAM claim at `main.tf:166/:174` is correct; it omits the consequence of sibling grants at **217–226**. |
| +- The “17 call sites” mixes definitions/logging with operational calls; it is not a complete reader/writer migration checklist. |
| + |
| +`Held` **does renew from acquisition**. Ordinary TTL passage is not the defect. Renewal failure still requires explicit health/loss handling; the existing [Held.Checkpoint](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/internal/lease/checkpoint.go:20) supplies a stronger completion contract than “wait for sync.” |
| + |
| +No archive, owner.json create-only put, or OwnerLive false-negative problem remains in rev 2 itself. The destructive risks moved to local purge, staging writers, retention and stale lifecycle work. |
| + |
| +--- |
| + |
| +**FIRST PROMPT:** |
| +Adversarially review replica ownership and db push rev 2 against both branches and quoted prior findings. |
| + |
| +**WHAT:** |
| +Verify isolation, recovery and rollout safety. |
| + |
| +**STATUS:** |
| +Review complete; design not approved. No files changed or tests run. |
| + |
| +**NEEDED:** |
| +Nothing. |
| + |
| +**RECOMMENDED:** |
| +Revise publication/fencing, legacy adoption and rollback protocols before implementation. |
| + |
| +**SESSION:** |
| +Safe to close. Workspace is read-only; no handoff file written. |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R3.md| new file mode 100644 |
| index 000000000..ef59d977a |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REPLICA-OWNERSHIP-R3.md |
| @@ -0,0 +1,35 @@ |
| +**Rev 3 still fails review.** Checked `66f1dad3` against `da866a92`, including the earlier specs under their former filename. R1/R2/R5 accepted; push excluded. |
| + |
| +1. **P1 — §§3.2, 3.5, 4: same-app recreation never retires the old row.** Delete and recreate between passes: `(account, app, host)` still exists, so §4 leaves the row; §3.2 cannot park; §3.5 only prevents *creation*. The old process/warm database remains usable. Applies to ordinary, sidecar and replica-role records. Evidence: `internal/provision/reconcile.go:947`, `internal/provision/sidecar_reconcile.go:400`. **Fix:** compare incarnation before either ownership branch; retire mismatched rows and gate existing-row admission too. |
| + |
| +2. **P1 — §3.5: `db pull` bypasses parking.** Delete A; create B; before any box parks, B’s record authorizes reads of A’s replica. No registry row is required. Evidence: `internal/console/apidb.go:134`, `:148`, `:165`; prefix remains host-only at `:70`. **Fix:** enforce the tombstone barrier in both database-read handlers, with authorization remaining valid through the actual read. |
| + |
| +3. **P1 — §§3.1, 3.3, 5: the recorded region does not cover deleted data.** Move H from S to T, delete in T, recreate in S: S’s old replica survives and §3.5 permits restoring it. Delete during preparing instead: staging in T survives and wins a later restore. Evidence: `internal/host/movespass.go:377`, `:492–502`; `internal/activator/backend_exec.go:1581`, `:1606`. **Fix:** park every affected region’s replica/staging locally, and require that region’s completion before reuse. Source cleanup cannot remain excluded after deletion. |
| + |
| +4. **P1 — §2: tombstoning and unconditional deletion can describe different records.** D1 reads incarnation A; D2 deletes A; B is created; D1’s delayed `DeleteRecord` deletes B while recording only A. Similarly, a move can change Region after the tombstone read. Evidence: `internal/provision/provision.go:1072`, `internal/provision/sidecar.go:399`, `internal/release/trash.go:261`. **Fix:** conditional deletion of the exact observed ETag, including purge; retain all observed storage locations across retries. `release.DeleteIfMatch` already exists. |
| + |
| +5. **P1 — §§3.2–3.3: checking a lease is not holding exclusion.** Two same-region parkers can both pass the absent-lock check. One finishes and permits B to start; the delayed other deletes B’s reused keys or lock. A late wake can also acquire after the check. Evidence: acquisition is independent at `internal/lease/lease.go:148`; mirror deletion is unconditional at `internal/lease/mirror.go:493`. **Fix:** exclusive park ownership shared with admissions/writers, held through publication of completion; revalidate under it and conditionally release only its own lock. |
| + |
| +6. **P1 — §§3.2–3.3: move writers remain active after deletion.** A move worker snapshots the old record, deletion occurs, then its mirror round writes staging after parking. Stopping the child does not stop this worker; move mirroring does not acquire the destination instance lease. Evidence: `internal/host/movespass.go:242`, `:269`, `:377`, `:480`. **Fix:** serialize deletion/parking against move rounds and staging promotion, with incarnation checks before storage mutation. |
| + |
| +7. **P1 — §3.3: manifest membership misses overwritten keys.** After copying K, another writer replaces K with different bytes of the same size. K is still listed in the manifest; deletion destroys the replacement without preserving it. Evidence: the available Copy interface supports source-ETag conditions, but Delete does not: `internal/lease/mirror.go:41–44`, `:467`, `:493`. **Fix:** record source identity/version, use conditional copying and deletion, and fail on changed identities—not merely additional keys. |
| + |
| +8. **P1 — §§3.1, 3.5: admission can miss or disregard a tombstone.** List tombstones; delete A/create B; list records; mint B against the earlier tombstone snapshot. Separately, an unreadable old record makes its tombstone non-actionable, which falls outside §3.5’s blocking predicate. Existing reconcile already uses collected snapshots: `internal/provision/reconcile.go:709`, `:903`. **Fix:** an admission barrier coordinated with deletion; unknown tombstone status must block, not count as clear. |
| + |
| +9. **P1 — §§3.2, 3.4–3.5: “no leftovers” is weaker than local completion.** In another region, B can mint because no local files exist, before A’s local `done` is recorded. B creates files and sleeps; A’s delayed local park sees no A-owned row or running child and moves B’s files. **Fix:** require a durable local completion acknowledgement—even for an empty park—before admitting the next incarnation. |
| + |
| +10. **P1 — §2: four deleting call sites are omitted.** UI domain detach (`internal/console/domainattach.go:287`), API domain detach (`internal/console/apidomains.go:796`), attach rollback (`internal/console/domainattach.go:475`), and instance-create rollback (`internal/console/apiinstances.go:1035`). A committed record can be observed before rollback. **Fix:** put tombstone-before-conditional-delete in the shared deletion helpers; explicitly cover static records and rollback tests. No additional raw instance/sidecar-key deletions were found outside those helpers and `PurgeApp`. |
| + |
| +11. **P1 — §§2, 3.1: sidecar lookup needs a different key.** Sidecars are keyed by **name**, not synthetic host. Looking up the stated `(account, app, host)` as an instance key—or passing host as sidecar name—returns not-found beside a live sidecar tombstone. Evidence: `internal/provision/sidecar.go:89`, `:125`, `:335`. **Fix:** record the exact record family/key, or explicitly derive and validate Name; dispatch reads by family. |
| + |
| +12. **P1 — §§2, 3.1, 3.5: reused CreatedAt defeats incarnation separation.** Both create helpers accept caller-supplied timestamps, including zero. Recreating the same tuple with the same timestamp makes the tombstone non-actionable; a second deletion also reuses its create-only key and potentially its completed manifest. Evidence: `internal/provision/provision.go:820`, `internal/provision/sidecar.go:322`. **Fix:** reject creation with an already-tombstoned identity and reject zero timestamps for new records; legacy fallback remains for existing records. |
| + |
| +13. **P2 — §§3.4, 4: row deletion loses the local inventory.** Parking requires the row already gone, but neither tombstone nor local journal records its DBPath/layout. “For a nested row” can no longer be determined from that row. **Fix:** persist the local path inventory before retirement; inspect both conventional layouts and retain their separate destination paths. Definitions: `internal/provision/provision.go:1219`, `:1270`, `:1277`. |
| + |
| +14. **P2 — §3.4: rename alone does not establish crash safety.** Separate DB/WAL/meta renames are not one transaction; rename can replace an existing destination, fails across filesystems, and does not close open handles. No durable ordering between renames and `done` is specified. **Fix:** journal per-artifact progress, use non-overwriting destinations, sync directories before `done`, and keep admission closed on EXDEV/errors. Require writer/replicator closure, not just child exit; existing stop evidence distinguishes these at `internal/activator/maintenance.go:32–41`. |
| + |
| +15. **P2 — §4: fence completion and existing retirement rules are incomplete.** No step lifts the deletion fence, so a successfully recreated row remains unwakeable (`internal/activator/maintenance.go:133`, `:167`). Also, requiring global record absence would retain source rows after moves, whose records still exist elsewhere. **Fix:** specify fence generation/recovery/release; preserve out-of-scope and trash retirement, kind/region/pool guards, and distinguish missing, invalid and unreadable records. Current guards: `internal/provision/reconcile.go:1124–1147`; sidecar equivalents `:425–434`. Read failures currently set global `readFailed`, not `protect` (`reconcile.go:735–743`). |
| + |
| +16. **P2 — §3.3: archive mapping is unspecified.** Both source prefixes can contain the same relative LTX key with different bytes. Flattening both beneath the stated destination overwrites one while size checks can pass. **Fix:** preserve distinct `instances/` and `moves/` subtrees; manifest entries must name exact source and destination keys. |
| + |
| +The cited path/helper locations otherwise check out. The IAM claim matches `infra/modules/region-host/main.tf:166` and `:174`. No files changed; this was a source/spec review, not runtime validation. |
| \ No newline at end of file |