amadan
runbook: B1 record deletes confirmed; B4 ordering and the status-tick window
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+28
−30
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+28 −30
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index 837080084..b6601b25f 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -130,7 +130,7 @@ Each failure action is the whole action. Nothing is retried in place unless the |
| - [ ] **G7: 8.2.** Pass: `host.env`, `install-control`, `journal-migrate --window`, the mode checks, `/var/lib/carlos-claims/authority/exec/census.json` present before any restart, instance-control active, and leo's census report lists zero installations. Then 8.2b. On failure: stop and hold from the `host.env` edit (R1). The console stays off, so dev carries on. |
| - [ ] **G8: 8.2b `backfill-done`.** Needs only 8.2's migration census report with zero installations, never the agent's runtime report or `host_claim_enabled` (those come in step 9, G9). Pass: phase `done`, one If-Match write, and sophie (the roster box with no census key) listed as unreported. Then 8.4. On failure: hold. The plan stays `collecting`; with the console off nothing is frozen. |
| - [ ] **G9: 8.4, the go for 9, step 9.** Pass: 8.4's restore; Paul's go for step 9; the edge active through `guard-exec`, one `REFUSING CARLOS_SWAP=overlap (host-claims-on)`, and a fresh verified box report with `host_claim_enabled`. Then block B4. On failure: stop and hold ("Stop and hold", "Steps 9 and 10"). |
| -- [ ] **G10: block B4.** Pass: the console's claims on (the "Paused state" resume block), `status/consoles/dev-console.json` moved, `console-report` applied; then `apps create claimtrial2`, `ship wc4`, `promote`, `instances enable --pool cohort1`, and `instances create` succeeding within one agent tick of `console-report`, in at most three attempts. Then acceptance. On failure: a refusal is fail-closed; capture the door's log line. After a third failed attempt, switch the console off again ("Paused state", the pause recipe) and hold leo. |
| +- [ ] **G10: block B4.** Pass: the console's claims on (the "Paused state" resume block), `status/consoles/dev-console.json` moved; `apps create claimtrial2`, `ship wc4`, `promote` and the store-mode `instances enable --pool cohort1` done first; then `console-report` and `instances create` back to back, the create succeeding before sophie's next status tick, in at most three attempts. Then acceptance. On failure: a refusal is fail-closed; capture the door's log line. After a third failed attempt, switch the console off again ("Paused state", the pause recipe) and hold leo. |
| - [ ] **G11: acceptance 1 to 4** on `claimtrial2` (launch, a confirmed stop, two cold generations with rising `hits=`, isolation). Then Paul's gos for 10.5 and 10.6. On failure: stop and hold; post the evidence on #250. |
| #### Trial assumptions (Option B): PROPOSED, for Paul to confirm in the morning (D5) |
| @@ -2631,19 +2631,21 @@ carlos instances delete --app claimtrial --account bab --host claimtrial.dev.car |
| carlos instances delete --app wakecheck --account bab --host wakecheck.dev.carlosframework.com --console $DEV --no-prompt |
| carlos instances delete --app rollcheck-0261 --account bab --host rollcheck-0261.dev.carlosframework.com --console $DEV --no-prompt |
| carlos instances delete --app "$REHEARSAL_APP" --account bab --host "$REHEARSAL_HOST" --console $DEV --no-prompt |
| -carlos apps delete --account bab --console $DEV claimtrial |
| ``` |
| -Each `instances delete` prints `deleted instance <host>` and says the route retires on the next reconcile pass. The last line moves the `claimtrial` app to the Trash (`apps/bab/claimtrial/deleted.json`), which also stops it minting again; `wakecheck`, `rollcheck-0261` and the rehearsal app keep their apps, so wakecheck can be recreated under its own name. If the rehearsal app is in another account, use that account's sqid in its line. |
| +Each `instances delete` prints `deleted instance <host>` and says the route retires on the next reconcile pass. If the rehearsal app is in another account, use that account's sqid in its line. |
| + |
| +**Confirmed by platform-1e (2026-10-20).** With the console claims-off, `instances delete` removes the record object itself: a `DeleteIfMatch` on `provision.RecordKey`, which in the versioned bucket leaves a delete marker that listing ignores. **Do not use `carlos apps delete` here.** It only writes the trash marker `apps/<account>/<app>/deleted.json` and leaves the instance record in place, so `backfill-collect` would still find it. The apps themselves stay: `hostops.Apps` may still list `claimtrial`, `wakecheck`, `rollcheck-0261` and the rehearsal app, which is harmless because collect takes its hosts from the records (`hostops.Records`), and there are none. wakecheck can then be recreated under its own name. |
| + |
| +**Left after B1, owed for a later clean-up.** `rollcheck-0261`'s domain claim `domains/rollcheck-0261.dev.carlosframework.com.json` (it has no `acq`) and its ledgers stay in the bucket. They are harmless for a zero-host plan and for `claimtrial2`, but the domain claim is an unacquired legacy claim. Record it on #250 as owed. |
| Then check the bucket, from DOCS-T1: |
| ```bash |
| aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -cE '/(instances|sidecars)/[^/]+\.json$' || true |
| -aws $R s3 ls "s3://$B/apps/bab/claimtrial/deleted.json" |
| ``` |
| -Expected: `0`, then the trash marker. The console announces each delete, so the boxes retire the rows within seconds; the hourly floor and the 5-minute pass are the fallback. Then the registries, read-only (P4's query): |
| +Expected: `0`. The console announces each delete, so the boxes retire the rows within seconds; the hourly floor and the 5-minute pass are the fallback. Then the registries, read-only (P4's query): |
| ```bash |
| leo 'python3 -c " |
| @@ -3029,7 +3031,7 @@ On failure: stop and hold. |
| Paul's go: one go for block B4 and acceptance 1 to 4 (D6), after step 9 has passed (gate G9) and the barrier is lifted. Fable section 3B names this the one untested dev path: claim-mode `apps create`, `ship`, `promote`, `instances enable --pool` and `instances create`. A refusal there comes before any claim write, and the console can be switched off again, so the blast radius is `claimtrial2` alone. |
| -**Why the timing matters (H10, #312).** `instances create` on a claim-mode console needs the console's compute evidence, which `console-report` puts into sophie's box report. The agent rewrites that report whole at its next status tick, about five minutes, and the entry is gone. So `instances create` must succeed within one agent tick of `console-report`: five minutes, not six. Everything from `console-report` to `instances create` runs from one shell, in one go, with `date -u` stamped between commands, and the create gets at most three attempts. |
| +**Why the timing matters (H10, #312).** `instances create` on a claim-mode console needs the console's compute evidence, which `console-report` puts into sophie's box report. The agent rewrites that report whole at its next status tick, every five minutes plus up to one minute of jitter (`internal/host/agent.go:3781`), and the entry is gone. So `instances create` must succeed between `console-report` and sophie's next status tick. Two more limits apply: the census needs every report to be at most 15 minutes old, and the create's hold re-verifies the saved reports when it completes (`internal/hostops/production_create.go:304-349`). Any claim-mode delete has the same window. So only `console-report` and `instances create` sit inside the clock, run back to back from one shell with `date -u` stamped between them, after a check of when sophie's report was last written. Everything else in B4 runs before. The create gets at most three attempts. |
| **1. Before the clock starts (DOCS-T1).** Save 5.1's second helper block (`ssm_run`, `box_get`, `box_put`, `box_inspect`, with `LEO` and `SOPHIE`) as `$TMPDIR/cohort1/box-helpers.sh`, preceded by `R=eu-west-1` and `export AWS_PROFILE=kass`. Fetch and check the build, as step 4 did: |
| @@ -3043,22 +3045,10 @@ Expected: the sha256 above, and nothing under `apps/bab/claimtrial2/` (the name |
| **2. Console claims on.** On sophie, the "Paused state" resume block, with `box_inspect "$SOPHIE"` from DOCS-T1 first, then 7.4's checks: `claim-console.env` is `5df17bac`, the console `active` with `CARLOS_HOST_CLAIM=1`, and the run sweep and seeds pass lines back in its journal. Wait for `status/consoles/dev-console.json` to move. Stop before the resume block's own `console-report`: step 3 runs it. |
| -**3. The timed run (DOCS-T1, one shell).** Wait until sophie's box report (`status/boxes/sophie-knitting.json`) has just been rewritten, for the widest window, as 7.2 does. Then run this as one script, outside the sandbox, signed in to the dev console, with `CARLOS_DEPLOYMENT_BUCKET` unset except on the `instances enable` line: |
| +**3. The app, its release and its pool, before `console-report` (DOCS-T1, no clock).** `apps create`, `ship` and `promote` go through the console, signed in, with `CARLOS_DEPLOYMENT_BUCKET` unset. Then `instances enable --pool cohort1` runs in store mode: straight to the bucket with profile `kass`, no console, because signed-in mode refuses `--pool` (`instances_cli.go:87-91` at `96db0efe`). Store mode is chosen by `CARLOS_DEPLOYMENT_BUCKET` being set; `--domain` is required there, and `--console` is not passed. This is step 4.4's invocation, which wrote `claimtrial`'s `instances.json` on 2026-10-18. Outside the sandbox: |
| ```bash |
| -cat > "$TMPDIR/cohort1/b4.sh" <<'B4' |
| -set -euo pipefail |
| -. "$TMPDIR/cohort1/box-helpers.sh" |
| -cd "$TMPDIR/cohort1" |
| DEV=https://platform.dev.carlosframework.com |
| -H2=claimtrial2.dev.carlosframework.com |
| -CR='cd /root/claim-trial && /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 --expect-revision "$(cat rev)" --console-id "$(cat console-id)" --public-key "$(cat release.pub)"' |
| -date -u +%FT%TZ |
| -box_inspect "$SOPHIE" |
| -date -u +%FT%TZ |
| -ssm_run "$SOPHIE" "$CR --dry-run" |
| -date -u +%FT%TZ |
| -ssm_run "$SOPHIE" "$CR" |
| date -u +%FT%TZ |
| carlos apps create --app claimtrial2 --account bab --console $DEV |
| date -u +%FT%TZ |
| @@ -3070,40 +3060,48 @@ date -u +%FT%TZ |
| AWS_PROFILE=kass CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 \ |
| carlos instances enable --app claimtrial2 --account bab --domain dev.carlosframework.com --pool cohort1 --channel edge |
| date -u +%FT%TZ |
| -carlos instances create --app claimtrial2 --account bab --host $H2 --console $DEV --no-prompt |
| -date -u +%FT%TZ |
| -B4 |
| -TMPDIR="$TMPDIR" bash "$TMPDIR/cohort1/b4.sh" 2>&1 | tee "$TMPDIR/cohort1/b4-attempt1.log" |
| +aws --profile kass s3 cp s3://carlos-dev-640012953448/apps/bab/claimtrial2/instances.json - |
| ``` |
| -Expected, in order: `console-report` publishes; `apps create` gives the single `edge` channel; `ship` prints `shipped claimtrial2 wc4`; `promote` prints `promoted claimtrial2 wc4 -> edge`; `instances enable` writes `instances.json` with pool `cohort1` and channel `edge`; `instances create` prints `created instance claimtrial2.dev.carlosframework.com`. The stamp after `instances create` is less than five minutes after the stamp before the applied `console-report`. That is attempt 1. |
| +Expected: `apps create` gives the single `edge` channel; `ship` prints `shipped claimtrial2 wc4`; `promote` prints `promoted claimtrial2 wc4 -> edge`; `instances enable` prints `enabled instances for claimtrial2 under dev.carlosframework.com`; `instances.json` reads `{"schema":1,"app":"claimtrial2","domain":"dev.carlosframework.com","pool":"cohort1","channel":"edge"}`. A refusal here is fail-closed and before any claim write: capture it and go to the third-failure action below. |
| -**If `instances create` is refused** (the console's evidence has gone, or a claim-mode refusal): capture the refusal and the console's journal line for it (`journalctl -u console.service --since -10min` on sophie, read-only). Then attempt 2, and if needed attempt 3, each a fresh report and an immediate create, from the same kind of script: |
| +**4. The timed pair (DOCS-T1, one shell).** One script per attempt. It stages sophie's roster and runs the dry run first, outside the clock. Then it reads when sophie's box report was last written and prints how long ago. If the next status tick could come within 90 seconds (the report is 210 seconds old or more: the tick comes 300 to 360 seconds after the last), it waits for that tick first. Then it runs `console-report` and `instances create` back to back, with `date -u` stamps: |
| ```bash |
| -cat > "$TMPDIR/cohort1/b4-retry.sh" <<'B4R' |
| +cat > "$TMPDIR/cohort1/b4-pair.sh" <<'B4' |
| set -euo pipefail |
| . "$TMPDIR/cohort1/box-helpers.sh" |
| cd "$TMPDIR/cohort1" |
| DEV=https://platform.dev.carlosframework.com |
| H2=claimtrial2.dev.carlosframework.com |
| CR='cd /root/claim-trial && /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 --expect-revision "$(cat rev)" --console-id "$(cat console-id)" --public-key "$(cat release.pub)"' |
| +lm() { date -u -d "$(aws --region eu-west-1 s3api head-object --bucket carlos-dev-640012953448 --key status/boxes/sophie-knitting.json --query LastModified --output text)" +%s; } |
| date -u +%FT%TZ |
| box_inspect "$SOPHIE" |
| +ssm_run "$SOPHIE" "$CR --dry-run" |
| +LAST=$(lm) |
| +AGE=$(( $(date -u +%s) - LAST )) |
| +echo "sophie's report written $(date -u -d "@$LAST" +%FT%TZ), ${AGE}s ago" |
| +if [ "$AGE" -ge 210 ]; then |
| + echo "next tick within 90s: waiting for it" |
| + until [ "$(lm)" != "$LAST" ]; do sleep 5; done |
| +fi |
| date -u +%FT%TZ |
| ssm_run "$SOPHIE" "$CR" |
| date -u +%FT%TZ |
| carlos instances create --app claimtrial2 --account bab --host $H2 --console $DEV --no-prompt |
| date -u +%FT%TZ |
| -B4R |
| -TMPDIR="$TMPDIR" bash "$TMPDIR/cohort1/b4-retry.sh" 2>&1 | tee "$TMPDIR/cohort1/b4-attempt2.log" |
| +B4 |
| +TMPDIR="$TMPDIR" bash "$TMPDIR/cohort1/b4-pair.sh" 2>&1 | tee "$TMPDIR/cohort1/b4-attempt1.log" |
| ``` |
| -Run it at most twice (logs `b4-attempt2.log` and `b4-attempt3.log`). Never change an identifier between attempts, and never loop it. A refusal in any step before `instances create` is not retried: capture it and go to the third-failure action. |
| +Expected: the age line, a wait line only if the report was 210 seconds old or more, then `console-report` publishes and `instances create` prints `created instance claimtrial2.dev.carlosframework.com`. The stamp after `instances create` is well inside the tick: under a minute after the stamp before `console-report`. That is attempt 1. If the wait does not end within seven minutes, sophie's agent is not ticking: stop the script (Ctrl-C), count it as a failed attempt, and read sophie's edge journal before anything else. |
| + |
| +**If `instances create` is refused** (the console's evidence has gone, or a claim-mode refusal): capture the refusal and the console's journal line for it (`journalctl -u console.service --since -10min` on sophie, read-only). Then run the same script again, at most twice more, with logs `b4-attempt2.log` and `b4-attempt3.log`. Never change an identifier between attempts, and never loop it. |
| **On a third failed attempt:** switch the console off again ("Paused state", the pause recipe), confirm `/healthz` 200 and the empty `CARLOS_HOST_CLAIM=`, and hold leo ("Stop and hold"). Post the three logs and the console's journal lines on #250. Whatever B4 wrote stays (`claimtrial2`'s app and releases); do not delete it. |
| -**4. After the create (no clock).** Read the claim and the route, and get the host a certificate as step 4.5 did. Only the housekeeper-lease holder orders certificates, and only when a TLS hello for the name reaches it. Outside the sandbox: |
| +**5. After the create (no clock).** Read the claim and the route, and get the host a certificate as step 4.5 did. Only the housekeeper-lease holder orders certificates, and only when a TLS hello for the name reaches it. Outside the sandbox: |
| ```bash |
| aws --profile kass s3 cp s3://carlos-dev-640012953448/control/hosts/claimtrial2.dev.carlosframework.com.json - \ |