amadan
seeded spec rev 5: answer Astra round 4 (seed-ID-scoped snapshot keys, record deleted last, moves wait for placement); run spec accepted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+93
−16
docs/superpowers/specs/2026-10-08-seeded-instances-design.md+65 −16docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R4.md+28 −0
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md| index 3293a8267..5c46027a9 100644 |
| --- a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md |
| +++ b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md |
| @@ -1,6 +1,6 @@ |
| # Instances that start from a database: `--from` (#260) and the import door (#261) |
| -**Status:** rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Paul approved rev 1 (c66792a3). Rev 2 |
| +**Status:** rev 5, 2026-10-08. Rev 3 answered round 2 (§12), rev 4 round 3 (§13), and rev 5 round 4 (§14). Paul approved rev 1 (c66792a3). Rev 2 |
| answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`; findings |
| 8 to 20 are this spec's). Each change is marked **(codex N)**, and §11 lists |
| the dispositions. |
| @@ -59,8 +59,11 @@ starts as a given one. |
| ### 3.1 The object |
| -The snapshot lives at `seeds/<host>/snapshot.db` in the home region's data |
| -bucket, beside the `instances/` and `moves/` prefixes. |
| +The snapshot lives at `seeds/<host>/<id>/snapshot.db` in the home |
| +region's data bucket, beside the `instances/` and `moves/` prefixes. |
| +`<id>` is the seed's `ID` (§3.2). Every write and delete of a snapshot |
| +names its seed `ID`, so cleaning up one seed can never touch another seed |
| +of the same host (codex R4-1). |
| - The console writes it. The owning box reads it and deletes it. |
| - The console's region is the home region, the only one whose data bucket |
| @@ -69,7 +72,7 @@ bucket, beside the `instances/` and `moves/` prefixes. |
| `ErrResidency` message. |
| - **There is no lifecycle rule on `seeds/`** (codex R2-5). A `--from` |
| instance that nobody wakes for a week still needs its snapshot. Instead, |
| - the console's deadline sweep deletes `seeds/<host>/` only in three |
| + the console's deadline sweep deletes `seeds/<host>/<id>/` only in three |
| cases: |
| - no record names it, and the snapshot is more than an hour old. Every |
| door writes its record **before** the snapshot (§4 step 7, §5.1), so a |
| @@ -123,7 +126,7 @@ other per-record handling: |
| | State | Reconcile | |
| |---|---| |
| -| `aborted` | runs the idempotent rollback (§5.6), or with no journal row deletes `seeds/<host>/` and the record | |
| +| `aborted` | runs the idempotent rollback (§5.6), or with no journal row deletes `seeds/<host>/<id>/` and the record | |
| | `reserving`, `awaiting-upload`, `uploaded`, `refused` | skips the record; nothing is minted | |
| | `ready` | mints (`--from`) or swaps (import) | |
| | `swapped`, `placed`, `committed` | ordinary handling, plus the import gate (§5.4) while not `committed` | |
| @@ -177,7 +180,7 @@ applies only to a row whose record seed is `ready`, `swapped` or `placed`: |
| So a database left behind by an earlier, aborted import of the same |
| host can never stand in for a new one. |
| -2. **Fetch and check.** Read `seeds/<host>/snapshot.db` into a temp file |
| +2. **Fetch and check.** Read `seeds/<host>/<id>/snapshot.db` into a temp file |
| inside the instance's directory. Check its length and sha256 against |
| the record, then run `lease.CheckDBContext` (integrity, and at least one |
| table). |
| @@ -188,7 +191,7 @@ applies only to a row whose record seed is `ready`, `swapped` or `placed`: |
| 4. **Replicate.** Start replication. The first upload is generation one of |
| the host's own lineage. Nothing of the source's lineage is copied. |
| 5. **Advance the state.** After the first upload succeeds, set the seed to |
| - `placed` and delete `seeds/<host>/`. |
| + `placed` and delete `seeds/<host>/<id>/`. |
| **Failures.** Any failure in steps 2 to 4 fails the wake. The reason goes |
| to the box log and the `carlos logs` sink. It **never** falls through to an |
| @@ -276,7 +279,7 @@ Both doors call it. |
| step 3. Create the record with seed `reserving`, kind `replica`, its |
| seed `ID` and the source. Nothing mints a `reserving` record, and the |
| sweep leaves its snapshot alone (§3.1). |
| - 2. **Upload.** Write `seeds/<new>/snapshot.db` with `PutFrom` from the |
| + 2. **Upload.** Write `seeds/<new>/<id>/snapshot.db` with `PutFrom` from the |
| restored file (seekable, under S3's 5 GB single-PUT limit). |
| 3. **Publish.** CAS the record from `reserving` to `ready`, with the |
| restored TXID, sha256 and bytes. |
| @@ -341,7 +344,7 @@ refused, it answers 507 or 429. Then: |
| 2. A length or hash that differs from the record is refused with 422. |
| 3. It runs `CheckDBContext` on the spooled file, under a 30-minute |
| deadline. Not a database, or failing integrity, is refused with 422. |
| -4. It writes `seeds/<host>/snapshot.db` with one `PutFrom` from the |
| +4. It writes `seeds/<host>/<id>/snapshot.db` with one `PutFrom` from the |
| spooled file. A seekable file under 5 GB needs no multipart upload, so |
| there's no unfinished multipart upload to clean up and no new IAM |
| action. |
| @@ -457,12 +460,12 @@ delete` is the verb then. |
| `DELETE …/instances/{host}` on a record whose seed is an import that isn't |
| `committed` answers 409, "abort the import instead". The only exceptions |
| are `awaiting-upload`, `uploaded` and `refused`, where no box has |
| -swapped, so the console deletes the record and `seeds/<host>/` itself. |
| +swapped, so the console deletes the record and `seeds/<host>/<id>/` itself. |
| **The record is kept until the box has rolled back.** An `aborted` record |
| stays in the bucket until the owning box deletes it at the end of the |
| rollback below. A box that never swapped (there's no journal row for this |
| -seed `ID`) only deletes `seeds/<host>/` and the record. |
| +seed `ID`) only deletes `seeds/<host>/<id>/` and the record. |
| **Rollback.** Reconcile runs it for any journal row whose import is |
| abandoned, meaning any of these: |
| @@ -512,11 +515,20 @@ journal's state says where to resume after a restart: |
| (codex R3-1). |
| 5. **Lift the fence.** |
| 6. **Clean the bucket.** Delete the host's `instances/` replica, if any, |
| - and `seeds/<host>/`. |
| -7. **Delete the record**, by CAS, if it still names this seed `ID`. |
| -8. **Delete the journal row.** |
| - |
| -**After a restart in `cleanup-pending`,** the box resumes at step 6. It |
| + and `seeds/<host>/<id>/` for this seed `ID` only. |
| +7. **Delete the journal row.** |
| +8. **Delete the record**, by CAS, if it still names this seed `ID`. It |
| + goes last (codex R4-1). Until then the aborted record holds the |
| + hostname, and `CreateRecord`'s create-only write refuses a new import or |
| + `--from` for that host (`provision.go:845`). So no second seed can |
| + arrive while this one's cleanup is still running. A crash after step 7 |
| + leaves an `aborted` record with no journal row, and §3.2's no-journal |
| + path (delete this seed's snapshot, then the record) finishes the job |
| + idempotently. |
| + |
| +**After a restart in `cleanup-pending`,** the box resumes at step 6. Its |
| +deletes are scoped to this seed `ID`, and the record still holds the |
| +hostname. It |
| doesn't swap, fence or stop anything: the route is the restored legacy |
| row, and its process is the app's own. Until the journal row is gone, the |
| host's leftover replica is this box's to delete. So a retried import never |
| @@ -549,6 +561,29 @@ record are clean. So while an |
| import is in flight, the box always knows what it replaced, whatever has |
| happened to the record. |
| +### 5.8 Moves wait for placement (codex R4-2) |
| + |
| +A seed exists only as a snapshot in the home bucket until its first wake |
| +places it. A move before then would carry no replica and no snapshot: the |
| +legacy move worker treats an empty replica as ready |
| +(`movespass.go:815`, `:919`). So a move is refused: |
| + |
| +- for a `--from` instance until its seed is `placed`; |
| +- for an import until it is `committed`. |
| + |
| +The refusal is applied in two places: |
| + |
| +- **In the move door,** under the record CAS. This covers both the legacy |
| + door (`apiinstancemove.go:399`, through `provision.go:1146`) and the |
| + moves series' request door. |
| +- **In the move worker, before any effect.** A record that slips past an |
| + old console is still refused. |
| + |
| +The message is "wake the instance once before moving it" for a `--from` |
| +instance, and "commit or abort the import first" for an import. Moving a |
| +seed before it is placed would mean handing the seed, and an import's |
| +rollback, to another region. That isn't offered. |
| + |
| ## 6. Who may, and what is checked |
| ### 6.1 Callers |
| @@ -641,6 +676,7 @@ The swap accepts nothing wider: |
| - commit before placed; |
| - abort after commit; |
| - delete of an uncommitted import (abort it instead); |
| +- move before placement or commit; |
| - abort while a run is open. |
| Some come back as the swap's refusal reasons instead: |
| @@ -711,6 +747,12 @@ into the code. |
| box was swept at start and the host is quiesced; |
| - `--from`'s `reserving` record: the sweep never deletes its snapshot, |
| and abandons it by CAS only after an hour; |
| + - abort A, crash between the journal delete and the record delete, then |
| + restart: A finishes, and a second import of the host is refused until |
| + A's record is gone; |
| + - a cleanup never deletes another seed `ID`'s snapshot; |
| + - a move refused before placement or commit, in the door (both the |
| + legacy and the moves series') and in the worker; |
| - **The gate:** |
| - the edge and the door answer 503 without the token and pass with it; |
| - no ticks while gated; |
| @@ -803,3 +845,10 @@ remaining imports wait for §3.4's claim algorithm. |
| | R2-4, still open: the fence retry can't work | **Accepted.** §5.6 step 2: the layout pass's pattern; our own fence is recognised by the journal's generation, and a cold host's stop is accepted only when the box was swept at start and the host is quiesced | |
| | R3-1 (P2) rollback forgets replica cleanup | **Accepted.** §5.6 steps 4 to 8: the journal stays as `cleanup-pending` until the bucket and record are clean; resuming never swaps or stops | |
| | R3-2 (P2) sweep deletes a snapshot mid-create | **Accepted.** §4 step 7: the record is reserved before the upload; the sweep never deletes a snapshot named by a live record, and abandons a stale reservation by CAS first | |
| + |
| +## 14. Round 4 dispositions |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| R4-1 (P1) a resumed rollback deletes the next import's snapshot | **Accepted, both fixes.** §3.1: snapshot keys carry the seed `ID`, and every delete is scoped to it. §5.6: the aborted record is deleted last, so it holds the hostname until cleanup is done | |
| +| R4-2 (P2) a move strands an unplaced seed | **Accepted.** §5.8: moves are refused until a `--from` seed is placed or an import is committed, in the door and in the worker | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R4.md b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R4.md| new file mode 100644 |
| index 000000000..6442cf047 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R4.md |
| @@ -0,0 +1,28 @@ |
| +| Finding | Disposition at `fc86ce19` | |
| +|---|---| |
| +| R2-4 — Rollback fence retry and cold-host stop proof | **Fixed at spec level.** Seeded §5.6 distinguishes its own fence, defers on another operation’s fence, and defines the evidence required for a cold host. | |
| +| R3-1 — Rollback forgets unfinished replica cleanup | **Fixed for the reported crash gap.** `cleanup-pending` retains recovery until bucket cleanup finishes. A separate hostname-reuse race appears below. | |
| +| R3-2 — Sweep deletes a snapshot during creation | **Fixed.** Seeded §4 reserves the record before uploading; §3.1 requires a terminal CAS before abandoning that reservation. | |
| +| R3-3 — Disconnected caller leaves queued work open | **Fixed.** Run §5.8 assigns queued expiry, audit completion and reservation release to the console sweep. | |
| + |
| +1. **P1 — Seeded §§5.6–5.7: resumed rollback can delete a replacement import’s snapshot.** |
| + |
| + Abort import A. Complete bucket cleanup and delete A’s record, then crash before deleting its journal—between steps 7 and 8. While the box is down, begin import B at the same hostname and upload its snapshot. On restart, A’s `cleanup-pending` journal resumes at step 6 and deletes `seeds/<host>/`, including B’s accepted snapshot. The seed-ID check at step 7 protects B’s record only; its data has already been deleted. |
| + |
| + This reuse is possible against the existing creation code: the leftover check examines replica and move-staging prefixes ([createleftovers.go:69](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/createleftovers.go:69)), while `CreateRecord` conditionally creates the bucket record without consulting a box-local journal ([provision.go:845](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:845)). Once A’s record and replica are gone, neither check prevents B. |
| + |
| + **Fix:** keep the aborted record as the hostname reservation until cleanup authority has been retired. Complete bucket cleanup, retire the journal, then delete the matching record last. Recovery after journal retirement must remain safe and idempotent. Alternatively, scope every cleanup deletion to immutable objects belonging to A’s seed ID. |
| + |
| +2. **P2 — Seeded §§2.3, 3.1–3.4: a move can strand a seed before its first placement.** |
| + |
| + Create a `--from` instance, let reconcile mint its route, and move it before its first wake. Its seed is `ready`; its database exists only under `seeds/<host>/` in the home bucket. No specified seed-state guard prevents the move. |
| + |
| + With the moves switch off, the existing legacy move door remains active ([apiinstancemove.go:399](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/apiinstancemove.go:399)). Its transition admits an ordinary instance without requiring a database ([provision.go:1146](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:1146)). The worker mirrors only the replica prefix and treats a zero-byte gap as ready ([movespass.go:815](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/movespass.go:815)); it skips checkpointing an absent local database and then completes the region flip ([movespass.go:919](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/movespass.go:919)). |
| + |
| + The destination consequently receives neither replica nor seed snapshot. Under §3.4, its first wake must fail. |
| + |
| + **Fix:** refuse move admission until a replica seed is `placed`, and until an import is `committed`. Apply that predicate under the record CAS and in the move worker before effects. Supporting earlier moves would instead require explicitly transferring seed ownership and, for imports, rollback responsibility. |
| + |
| +`carlos run`: **accept**. |
| + |
| +Seeded instances: **revise**. |
| \ No newline at end of file |