amadan
runbook: delta v2 for Option B (backfill-done, B1/B2/B4/B5, gates, trial assumptions)
Adds platform-1e's two checks: root's exec census.json before any restart after 8.2, and gate G5 counting every registry table on leo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+19
−7
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+19 −7
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index 113639580..74a24f4c7 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -125,9 +125,9 @@ Each failure action is the whole action. Nothing is retried in place unless the |
| - **Known blocker, found 2026-10-20 and resolved by block B1: "G2 FAIL: records without acq".** Tonight's read-only run (`after236/precheck-20261020.md`, check 2b) found three instance records, `claimtrial`, `wakecheck` and `rollcheck-0261`, none with an `acq` field. `backfill-collect` refuses a record without one (`claimactivation/backfill.go:95`). platform-1e ruled a zero-host plan: block B1 deletes all three records (and the offline-move rehearsal app's), so the plan holds no hosts and no record needs an `acq`. Fable's "two records with acquisitions" is replaced by "no instance records" (B5's re-run after B1). |
| - [ ] **G3: signing session.** Pass: the signer role has the P-B2 grants (each phase's dry run passes), and `publish-identity` for v0.28.5 wrote its identity create-only. Then roll leo. On failure: tear down, run the CloudTrail check, hold. |
| - [ ] **G4: leo on v0.28.5.** Pass: `carlos system update` completes, the guard is active, `carlos.prev` is v0.28.4, and leo's box report shows v0.28.5 and verifies (`status`). Then block B1. On failure: leo is claims-off, so this is the ordinary failed-roll path: fix forward only (H11). Hold the window. |
| -- [ ] **G5: blocks B1 and B2.** Pass: no instance record in the bucket; leo's registry has zero rows; sophie's has no `wakecheck` row; nothing left under `/data/.instances`, `/data/instances`, `/data/programs` or `/opt/carlos/live` for any B2 host, on any box. Then the window. On failure: hold. Nothing claim-mode has run. |
| +- [ ] **G5: blocks B1 and B2.** Pass: no instance record in the bucket; leo's registry has zero rows of any kind, counted over every table, apex and domain rows included (B1's check prints `total 0`); sophie's has no `wakecheck` row; nothing left under `/data/.instances`, `/data/instances`, `/data/programs` or `/opt/carlos/live` for any B2 host, on any box. Then the window. On failure: hold. Nothing claim-mode has run. |
| - [ ] **G6: 8.1 and 8.1b.** Pass: 8.1's masks and `install-control --dry-run`; 8.1b's `backfill-collect`: zero hosts (`hosts` `{}`), phase `collecting`, one create-only write. Then 8.2. On failure: case B1 undo (ruling R1): restore units, enablement and barrier. The plan, if written, stays, and is harmless with the console off. |
| -- [ ] **G7: 8.2.** Pass: `host.env`, `install-control`, `journal-migrate --window`, the mode checks, instance-control active, and leo's census report lists zero installations. Then 8.2b. On failure: stop and hold from the `host.env` edit (R1). The console stays off, so dev carries on. |
| +- [ ] **G7: 8.2.** Pass: `host.env`, `install-control`, `journal-migrate --window`, the mode checks, `/var/lib/carlos-claims/authority/exec/census.json` present before any restart, instance-control active, and leo's census report lists zero installations. Then 8.2b. On failure: stop and hold from the `host.env` edit (R1). The console stays off, so dev carries on. |
| - [ ] **G8: 8.2b `backfill-done`.** Pass: phase `done`, one If-Match write, and sophie (the roster box with no census key) listed as unreported. Then 8.4. On failure: hold. The plan stays `collecting`; with the console off nothing is frozen. |
| - [ ] **G9: 8.4, the go for 9, step 9.** Pass: 8.4's restore; Paul's go for step 9; the edge active through `guard-exec`, one `REFUSING CARLOS_SWAP=overlap (host-claims-on)`, and a fresh verified box report with `host_claim_enabled`. Then block B4. On failure: stop and hold ("Stop and hold", "Steps 9 and 10"). |
| - [ ] **G10: block B4.** Pass: the console's claims on (the "Paused state" resume block), `status/consoles/dev-console.json` moved, `console-report` applied; then `apps create claimtrial2`, `ship wc4`, `promote`, `instances enable --pool cohort1`, and `instances create` succeeding within one agent tick of `console-report`, in at most three attempts. Then acceptance. On failure: a refusal is fail-closed; capture the door's log line. After a third failed attempt, switch the console off again ("Paused state", the pause recipe) and hold leo. |
| @@ -180,7 +180,7 @@ Option B (2026-10-20): the window time is `WINDOW_ID`, set at Paul's go (top of |
| | Window ID | set at go | `WINDOW_ID`, and from it `WINDOW` and `BACKFILL_ID` (top of this package). If the time moves, all three move | |
| | T−45 | | Second root session opened and the deadline watcher started (below); block B5 and block B2's final checks re-run read-only; a fresh check that `sg-054aa165db842b6ba` is still on leo's ENI alone | |
| | T−30 | | Barrier on (section 2), checked | |
| -| T−5 | | Gate G5 re-read: leo's registry has zero rows and no `carlos-instance-gen@` or `carlos-instance@` unit is listed. If not, abort before 8.1: lift the barrier, nothing else | |
| +| T−5 | | Gate G5 re-read with B1's leo check: `total 0` over every registry table, and no `carlos-instance-gen@` or `carlos-instance@` unit listed. If not, abort before 8.1: lift the barrier, nothing else | |
| | T0 | | 8.1, then 8.1b on the signer. Budget 20 minutes | |
| | | T0+20m | 8.2. Budget 20 minutes | |
| | | T0+40m | 8.2b on the signer, then 8.4. Budget 20 minutes | |
| @@ -2649,7 +2649,13 @@ Expected: `0`, then the trash marker. The console announces each delete, so the |
| leo 'python3 -c " |
| import sqlite3 |
| c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| -print(c.execute(\"select count(*) from routes\").fetchone()[0]) |
| +ts=[r[0] for r in c.execute(\"select name from sqlite_master where type=\x27table\x27 and name not like \x27sqlite_%\x27 order by name\")] |
| +n=0 |
| +for t in ts: |
| + k=c.execute(\"select count(*) from %s\" % t).fetchone()[0] |
| + n+=k |
| + print(t,k) |
| +print(\"total\",n) |
| for r in c.execute(\"select host,app,pool from routes order by host\"): print(r)" |
| systemctl list-units --all --plain --no-legend "carlos-instance-gen@*" "carlos-instance@*" |
| journalctl -u carlos-edge.service --since -30min --no-pager | grep "removed retired"' |
| @@ -2658,7 +2664,7 @@ journalctl -u carlos-edge.service --since -30min --no-pager | grep "removed reti |
| ``` |
| Expected: |
| -- **leo: zero rows** (`0`, no route listed), no instance unit, and a `provision: removed retired instance route claimtrial.dev.carlosframework.com (record gone)` line; |
| +- **leo: zero registry rows of any kind**, in every table, apex and domain rows included (`routes`, `route_aliases`, `wildcard_domains`, `route_paths`, `timers` and the rest): every table prints `0` and the last line is `total 0`, with no route listed; no instance unit; and a `provision: removed retired instance route claimtrial.dev.carlosframework.com (record gone)` line. A table with rows is a hold for platform-1e's reading, even if it is not a route table; never delete a row by hand (platform-1e, 2026-10-20); |
| - sophie: three rows, the console (`platform.dev.carlosframework.com`), `helloworld.dev.oncarlos.com` and `website.dev.oncarlos.com`; the stale `wakecheck` row is gone; |
| - dennis: `0` (the rehearsal app's row, if it was there, is gone). |
| @@ -2850,6 +2856,11 @@ if [ -e /var/lib/carlos-swap-status ]; then |
| test "$(stat -c '%a %U:%G' /var/lib/carlos-swap-status)" = '755 root:root' |
| sudo -u carlos test -x /var/lib/carlos-swap-status |
| fi |
| +# Root's exec census must exist before any restart (platform-1e, 2026-10-20): |
| +# without it, root's legacy restart fallback ignores the claim flag. |
| +# Missing is a STOP: hold, restart nothing. |
| +test -f /var/lib/carlos-claims/authority/exec/census.json |
| +test ! -L /var/lib/carlos-claims/authority/exec/census.json |
| systemctl daemon-reload |
| systemctl add-wants multi-user.target carlos-instance-control.service |
| systemctl restart carlos-instance-control.service |
| @@ -2877,6 +2888,7 @@ Expected: |
| - the unit keeps its protected environment, `StateDirectory=carlos-claims` and `MemoryMax=256M`; |
| - `journal-migrate` succeeds and publishes the census; `/var/lib/carlos-claims/authority/migration/migration.json` exists; |
| - the mode check after it: `/var/lib/carlos-claims` `700 root:root`; `/opt/carlos/system`, `guard`, `guard/moves` and `guard/seeds` each `755 root:root` and readable and searchable by `carlos`; `guard/carlos` executable by `carlos`; `/var/lib/carlos-swap-status`, if it exists, `755 root:root`. Under `set -e` a failed `test` ends the block: that is a STOP; |
| +- `/var/lib/carlos-claims/authority/exec/census.json` exists as a regular file before instance-control is restarted. If it is missing, the block stops there: hold (case B2), and restart nothing on leo, the box included, because root's legacy restart fallback ignores the claim flag without it; |
| - `active`, and no migration, recovery or floor error in the journal. |
| Keep every migration and root record. A failure leaves serving held. |
| @@ -3113,7 +3125,7 @@ The acceptance host is the disposable app made in block B4, `HOST=claimtrial2.de |
| ``` |
| Expect only the host's own directory, and two different mount namespaces. `carlos status --layout` shows the host `nested`. |
| -5. Reboot. `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| +5. Reboot. First, on leo, `test -f /var/lib/carlos-claims/authority/exec/census.json` (8.2's check, again); if it fails, do not reboot: hold. Then `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| 6. Signed repair or update. Apply the approved signed update to leo with `carlos system update`; confirm the box comes back and `claimtrial2` wakes. |
| Pass: all six. Record each response and the journal lines on #250. After acceptance, D9: Fable recommends switching the dev console off again (the pause recipe) so dev's pool-less creates work, then recreating wakecheck (see "Trial assumptions"). |
| @@ -3194,7 +3206,7 @@ What must not be touched while leo is held: |
| - `/data`, including `registry.db`, `/data/instances` and any parked database, and block B2's `/var/lib/carlos-aside-*` directories; |
| - sophie's protected console (`/opt/carlos/system/console/console` and `console.prev`), every published identity (`control/system/router/releases/v0.28.2.json`, unused; `control/system/router/releases/v0.28.4.json`; and `v0.28.3.json` if one ever appears, unused), every staged `policy*.json` and `identity*.json`, `/opt/carlos/system/router/channel-identity.json`, and the `bab` pointer-digest marker; |
| - the roster, floor and enrollments, and the claim's S3 objects: `control/hosts/`, `control/host-actors.json`, `control/host-origins/`, admitted prefixes, and `instances/` in `carlos-dev-ie-640012953448`; |
| -- no `systemctl reset-failed` before the failed results are recorded, no reboot unless the edge is persistently masked, no rerun of `journal-migrate`, `install-control` or the seed, and no console `reset-wake-retry`. |
| +- no `systemctl reset-failed` before the failed results are recorded, no reboot unless the edge is persistently masked, no reboot or instance-control restart while `/var/lib/carlos-claims/authority/exec/census.json` is missing (8.2), no rerun of `journal-migrate`, `install-control` or the seed, and no console `reset-wake-retry`. |
| Leaving hold is a separate decision: a fix and resume, or a claims-off teardown under its own reviewed procedure. This runbook does not cover either. Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`, so a later teardown has to say how it treats that (open question 2). |