carlos / platform Public

specs rev 4: answer Astra round 3 (fence resumption, cleanup-pending journal phase, reserve-before-upload, sweep expires queued runs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 2 weeks ago fc86ce19a70f655965c33a86ff1a2160ec25af14 parent e053a33

3 files changed, +131 −24

docs/superpowers/specs/2026-10-08-carlos-run-design.md +12 −1
docs/superpowers/specs/2026-10-08-seeded-instances-design.md +76 −23
docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R3.md +43 −0

CI passed — run details

diff --git a/docs/superpowers/specs/2026-10-08-carlos-run-design.md b/docs/superpowers/specs/2026-10-08-carlos-run-design.md

index 229c7d7c3..6cc22fa02 100644

--- a/docs/superpowers/specs/2026-10-08-carlos-run-design.md

+++ b/docs/superpowers/specs/2026-10-08-carlos-run-design.md

@@ -1,6 +1,6 @@

# `carlos run`: a member runs an app command against one instance (#259)

-**Status:** rev 3, 2026-10-08. Rev 3 answers round 2 (§12). Rev 1 (c66792a3) was approved by Paul.

+**Status:** rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Rev 1 (c66792a3) was approved by Paul.

Rev 2 answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`;

findings 1 to 7 are this spec's). Each change is marked **(codex N)**, and

§11 lists the dispositions.

@@ -397,6 +397,10 @@ and prints every chunk not yet acknowledged.

- **The console sweep.** It runs in the existing deadline-sweep loop and:

- deletes any run whose result is more than 24 hours old, whether or not

it was acknowledged;

+ - expires any run still `queued` 60 seconds past its request (codex

+ R3-3). It CASes `queued` to `expired` (losing to a pickup is fine),

+ writes the `run-end` audit row as `expired`, and clears it. So a run

+ whose caller disconnected never blocks the instance;

- deals with any run still `picked` or `cancel` an hour past its

deadline: it marks the run `done` with the audit end `lost`, and

clears it.

@@ -475,6 +479,7 @@ into the code.

- a run queued before the box owned the host is picked up;

- the pickup CAS with two boxes racing;

- expiry winning over a late box, and a late pickup winning over expiry;

+ - the sweep expiring a queued run whose CLI went away;

- cancel before pickup never launching;

- a request whose incarnation doesn't match the record being refused;

- run, move and import abort racing on the record's ETag;

@@ -540,3 +545,9 @@ switch. Tito's boxes need the nested layout before Doc and Vicky can use it

| R1-3, still open: no incarnation | **Accepted.** §5.1: the request carries the record's incarnation, and recreating the host drops `ActiveRun` |

| R2-2 (P1) run gets past the import gate | **Accepted.** §5.1 step 9: refused while an import is uncommitted, checked under the record CAS and again at launch |

| R2-6 (P2) "not picked up" leaves a runnable request | **Accepted.** §5.2 and §5.7: queued, picked, cancel, done, cancelled and expired, all moved by CAS on the record; expired and cancelled are terminal before launch |

+## 13. Round 3 dispositions

+| Finding | Disposition |

+|---|---|

+| R3-3 (P2) a disconnected caller leaves the run queued | **Accepted.** §5.8: the console sweep expires queued runs 60 seconds past the request, by CAS against pickup |

diff --git a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md

index 0f646a63a..3293a8267 100644

--- a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md

+++ b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md

@@ -1,6 +1,6 @@

# Instances that start from a database: `--from` (#260) and the import door (#261)

-**Status:** rev 3, 2026-10-08. Rev 3 answers round 2 (§12). Paul approved rev 1 (c66792a3). Rev 2

+**Status:** rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Paul approved rev 1 (c66792a3). Rev 2

answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`; findings

8 to 20 are this spec's). Each change is marked **(codex N)**, and §11 lists

the dispositions.

@@ -71,11 +71,15 @@ bucket, beside the `instances/` and `moves/` prefixes.

instance that nobody wakes for a week still needs its snapshot. Instead,

the console's deadline sweep deletes `seeds/<host>/` only in three

cases:

- - no record names it;

+ - no record names it, and the snapshot is more than an hour old. Every

+ door writes its record **before** the snapshot (§4 step 7, §5.1), so a

+ snapshot without a record is from a door that died between the two

+ writes (codex R3-2);

- the record's seed is `placed` or `committed` and has another ID;

- it belongs to an import stuck in `awaiting-upload` or `uploaded` for

- more than 7 days, which the sweep aborts first (§5.6), a terminal

- transition.

+ more than 7 days, or to a `--from` stuck in `reserving` for more than an

+ hour. The sweep moves the record to the terminal `aborted` by CAS

+ first, and deletes nothing unless that CAS succeeds (§5.6).

### 3.2 The record

@@ -105,7 +109,7 @@ type SeedSource struct {

| Door | Path through the states |

|---|---|

-| `--from` | `ready` → `placed` |

+| `--from` | `reserving` → `ready` → `placed` |

| import | `awaiting-upload` → `uploaded` → `ready` → `swapped` → `placed` → `committed` |

An import can also leave the path:

@@ -120,7 +124,7 @@ other per-record handling:

| State | Reconcile |

|---|---|

| `aborted` | runs the idempotent rollback (§5.6), or with no journal row deletes `seeds/<host>/` and the record |

-| `awaiting-upload`, `uploaded`, `refused` | skips the record; nothing is minted |

+| `reserving`, `awaiting-upload`, `uploaded`, `refused` | skips the record; nothing is minted |

| `ready` | mints (`--from`) or swaps (import) |

| `swapped`, `placed`, `committed` | ordinary handling, plus the import gate (§5.4) while not `committed` |

@@ -267,10 +271,19 @@ Both doors call it.

directory under the console's scratch, `<console data>/scratch/seeds/`.

Run `CheckDBContext` under the same deadline. This never reads the live

file.

-7. **Write.** Write `seeds/<new>/snapshot.db` with `PutFrom` from the

- restored file (seekable, under S3's 5 GB single-PUT limit). Then write

- the record with seed `ready`, kind `replica`, the source and the

- restored TXID.

+7. **Write** (codex R3-2), in this order:

+ 1. **Reserve the record.** This happens before the restore, right after

+ step 3. Create the record with seed `reserving`, kind `replica`, its

+ seed `ID` and the source. Nothing mints a `reserving` record, and the

+ sweep leaves its snapshot alone (§3.1).

+ 2. **Upload.** Write `seeds/<new>/snapshot.db` with `PutFrom` from the

+ restored file (seekable, under S3's 5 GB single-PUT limit).

+ 3. **Publish.** CAS the record from `reserving` to `ready`, with the

+ restored TXID, sha256 and bytes.

+ If the door fails after step 1, it CASes the record to `aborted`,

+ which the box (or the sweep) cleans up as for an import that never

+ swapped.

**Cleanup.** The scratch is deleted on every exit path: success, refusal,

error, deadline, and client disconnect. The console also empties

@@ -469,22 +482,46 @@ journal's state says where to resume after a restart:

From here on the edge refuses even verify requests, and that survives

an agent restart.

-2. **Fence and stop** (codex R2-4). Call

- `activator.FenceAndStopIfUnfenced(host, generation)`

- (`internal/activator/maintenance.go:45-60`) and require complete stop

- evidence (`stopObservation.complete`: lease released, main and sidecar

- stopped and exited, replica closed). On `ErrStopUnproven` or

- `ErrFenceHeld`, keep the fence and try again on the next pass. After a

- restart, a `rolling-back` journal row re-installs the fence before

- anything else.

+2. **Fence and stop** (codex R2-4, R3 open). This follows the layout

+ pass's handling of the same API (`internal/layout/pass.go:654-680`):

+ - **Ours already?** If `FenceGeneration(host)` equals the journal's

+ fence generation, the fence is this rollback's own, from an earlier

+ pass. Carry on with it.

+ - **Otherwise,** call `FenceAndStopIfUnfenced(host, generation)`. It

+ refuses any fence that already exists, including one from an earlier

+ attempt (`maintenance.go:103-106`), which is why the check above

+ comes first.

+ - **`ErrFenceHeld`** for a fence that isn't ours means another

+ operation holds the host. Defer to the next pass.

+ - **`ErrStopUnproven`.** A cold host has no live entry, and a host

+ never woken has no stop history (`maintenance.go:113-121`). Proceed

+ only if all of these hold, as the layout pass does:

+ - the box swept its instances at agent start;

+ - `Quiesced(host)` is true;

+ - `StopUncertain(host)` is false.

+ Otherwise lift this attempt's fence and defer.

+ Deferring is safe. The closed gate from step 1 still refuses every

+ request, ticks are skipped for a gated row, and runs are refused, so

+ nothing can wake the instance before the next attempt.

3. **Remove the local data** through `instancefs`: the database, `-wal`,

`-shm`, `<db>.seed` and litestream's local metadata (codex R2-1).

4. **Restore the route.** In one transaction, put the journal's saved row

- back exactly as it was, and delete the journal row.

+ back exactly as it was, and set the journal row to `cleanup-pending`

+ (codex R3-1).

5. **Lift the fence.**

6. **Clean the bucket.** Delete the host's `instances/` replica, if any,

and `seeds/<host>/`.

7. **Delete the record**, by CAS, if it still names this seed `ID`.

+8. **Delete the journal row.**

+**After a restart in `cleanup-pending`,** the box resumes at step 6. It

+doesn't swap, fence or stop anything: the route is the restored legacy

+row, and its process is the app's own. Until the journal row is gone, the

+host's leftover replica is this box's to delete. So a retried import never

+meets the creation gate's "replica data remains" refusal

+(`apiinstances.go:1146`).

**Why abort loses nothing.** Under the gate, nothing but verify requests

reached the instance, so it holds no writes the old process never saw.

@@ -499,15 +536,16 @@ The journal is a box-local registry table, `import_journal`, replacing rev

- seed `ID`;

- the original row as JSON;

- record key;

-- state: `swapped` or `rolling-back`;

+- state: `swapped`, `rolling-back` or `cleanup-pending`;

- fence generation;

- time.

Reconcile reads it **before** the bucket records, on every pass and at

agent start, and acts on it per §5.4 and §5.6.

-A journal row is deleted only in the same transaction that clears the gate

-at commit, or that restores the original row at rollback. So while an

+A journal row is deleted either in the same transaction that clears the

+gate at commit, or as the last step of a rollback, after the bucket and the

+record are clean. So while an

import is in flight, the box always knows what it replaced, whatever has

happened to the record.

@@ -665,7 +703,14 @@ into the code.

or deleted outright;

- the rollback fence: a verify request during rollback is refused, stop

evidence is required, and the fence is re-installed after a restart;

- - an idempotent rollback across a crash between any two of its steps.

+ - an idempotent rollback across a crash between any two of its steps,

+ including between the route restore and the replica delete

+ (`cleanup-pending` resumes, and never stops the legacy process);

+ - the fence: resumed when it is ours by generation; deferred when it is

+ another operation's; for a never-woken host, accepted only when the

+ box was swept at start and the host is quiesced;

+ - `--from`'s `reserving` record: the sweep never deletes its snapshot,

+ and abandons it by CAS only after an hour;

- **The gate:**

- the edge and the door answer 503 without the token and pass with it;

- no ticks while gated;

@@ -750,3 +795,11 @@ remaining imports wait for §3.4's claim algorithm.

| R2-4 (P2) a verify wake during rollback | **Accepted.** §5.6: the gate is closed durably first, then `FenceAndStopIfUnfenced` with complete stop evidence; the fence is re-installed after a restart |

| R2-5 (P1) the lifecycle rule deletes needed seeds | **Accepted.** §3.1: no lifecycle rule; a console sweep deletes only unreferenced or superseded snapshots, after a terminal abort for stale imports |

| R2-7 (P2) delete bypasses rollback | **Accepted.** §5.6: delete is refused for an uncommitted import; rollback is driven by the journal, including when the record is gone |

+## 13. Round 3 dispositions

+| Finding | Disposition |

+|---|---|

+| R2-4, still open: the fence retry can't work | **Accepted.** §5.6 step 2: the layout pass's pattern; our own fence is recognised by the journal's generation, and a cold host's stop is accepted only when the box was swept at start and the host is quiesced |

+| R3-1 (P2) rollback forgets replica cleanup | **Accepted.** §5.6 steps 4 to 8: the journal stays as `cleanup-pending` until the bucket and record are clean; resuming never swaps or stops |

+| R3-2 (P2) sweep deletes a snapshot mid-create | **Accepted.** §4 step 7: the record is reserved before the upload; the sweep never deletes a snapshot named by a live record, and abandons a stale reservation by CAS first |

diff --git a/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R3.md

new file mode 100644

index 000000000..28a5aa62f

--- /dev/null

+++ b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R3.md

@@ -0,0 +1,43 @@

+| Earlier finding | Disposition at `e053a333` |

+|---|---|

+| R1-1 — Run/move admission race | **Fixed.** Run §5.1 puts both admissions on the instance record’s ETag and requires guards to be reapplied after a conflict. |

+| R1-3 — Missing incarnation | **Fixed.** Run §§5.1–5.2 bind the request to the record incarnation and check it before launch. |

+| R1-10 — Protected route columns | **Fixed.** Seeded §5.4 explicitly refuses `Backing`, `Tunnel`, `Caps`, `Platform` and `Canary`. |

+| R1-15 — Abort safety outside HTTP | **Fixed.** The import gate, commit boundary and run refusal together cover the previously identified paths. |

+| R1-18 — Historical timestamps | **Declined with an acceptable reason.** Seeded §4 limits this version to TXIDs; timestamp support is explicitly excluded. |

+| R1-19 — Scratch bounds | **Fixed at spec level.** Seeded §4 names decoded-size admission, a shared reservation pool, shared concurrency limits and cancellation on overgrowth. |

+| R2-1 — Local database survives abort/re-import | **Fixed for the reported local-data defect.** Seeded §§3.4 and 5.6 require matching installation identity and removal of local database artifacts. A distinct remote-cleanup defect appears below. |

+| R2-2 — Run bypasses import gate | **Fixed.** Run §5.1 refuses uncommitted imports under the shared CAS and requires another check at launch. |

+| R2-3 — Crash between swap and record CAS | **Fixed.** Seeded §§5.4 and 5.7 consult the identity-bound journal before applying legacy-row predicates and resume the record transition. |

+| R2-4 — Verification wake during rollback | **Not fixed as an implementable rollback.** The durable gate closes admission, but §5.6’s retry instruction cannot work with the selected API: `FenceAndStopIfUnfenced` refuses **any** existing fence, including its own previous attempt ([maintenance.go:103](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/maintenance.go:103)). A never-woken import also lacks stop history and returns `ErrStopUnproven` ([maintenance.go:114](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/maintenance.go:114)); every subsequent attempt then returns `ErrFenceHeld`. **Fix:** define resumption of the journal’s own fence and cold/restarted stop proof. The existing layout caller handles these distinctions explicitly ([pass.go:654](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/layout/pass.go:654)). |

+| R2-5 — Seven-day expiration deletes needed seeds | **Fixed for the reported expiration defect.** Seeded §3.1 removes unconditional expiration. The replacement sweep introduces a separate publication race below. |

+| R2-6 — Reported pickup failure leaves executable work | **Fixed.** Run §5.2 requires successful terminal expiry before reporting “not picked up”; §5.7 makes queued cancellation terminal. |

+| R2-7 — Instance deletion bypasses rollback | **Fixed.** Seeded §5.6 refuses deletion after import admission becomes actionable, and journal recovery does not depend on the record remaining present. |

+1. **P2 — Seeded §§5.6–5.7: rollback forgets unfinished replica cleanup.**

+ Step 4 restores the original route and deletes the journal; step 6 deletes the remote replica. Crash between them. On restart, the surviving `aborted` record has no journal, so §§3.2 and 5.6 direct the box to delete only the seed snapshot and record. Nothing resumes deletion of the replica.

+ Record deletion deliberately preserves replica data ([provision.go:1840](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:1840)). The ordinary creation gate then refuses reuse of that hostname because replica data remains ([apiinstances.go:1146](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/apiinstances.go:1146)). Thus an interrupted abort leaves the conversion unable to retry through the ordinary creation checks.

+ **Fix:** retain the journal in a `route-restored/cleanup-pending` phase until remote cleanup and record retirement finish. Recovery from that phase must neither repeat the swap nor stop the restored legacy process.

+2. **P2 — Seeded §§3.1, 4 step 7: the sweep can delete a snapshot during successful creation.**

+ `--from` writes the snapshot before writing its record. The new sweep deletes snapshots when “no record names it.” A sweep between those two writes therefore deletes the snapshot, after which creation can successfully publish `ready`. Its first wake has neither a replica nor the required snapshot and must fail.

+ These are independent conditional object writes: `PutFrom` conditions only the snapshot key ([store.go:800](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/release/store.go:800)); `CreateRecord` separately creates the record key ([provision.go:845](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:845)). Neither protects the interval from deletion.

+ **Fix:** reserve the seed identity durably before uploading, using a non-minting record state or publication intent that the sweep honors. Retire abandoned reservations through an explicit terminal transition before deleting their objects.

+3. **P2 — Run §§5.2, 5.8: a disconnected caller can leave queued work open indefinitely.**

+ Let the console accept a run while its box is unavailable, then disconnect the CLI before its 60-second expiry request. Only the CLI initiates `queued → expired`. The sweep handles overdue `picked`/`cancel` runs and terminal runs, but never `queued`. Consequently `ActiveRun` remains non-terminal indefinitely and blocks subsequent runs and moves.

+ The selected persistence mechanism is a conditional object write, with no expiry behavior ([store.go:767](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/release/store.go:767)). Existing record transitions likewise occur through explicit mutations ([provision.go:1129](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:1129)); storing `Deadline` does not perform a transition.

+ **Fix:** assign overdue queued-run expiry to the console sweep, independently of the CLI. It must CAS `queued → expired` against pickup, publish the terminal result and audit outcome, and release the reservation.

+`carlos run`: **revise**.

+Seeded instances: **revise**.

\ No newline at end of file