carlos / platform Public

harness: Run 11 (G10 on 64a98473): disk freed, v0.28.15 build, post-b5-run8b restore (no post-8.2b existed), pre-roll check PASS, plain system update PASS

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 4 days ago be85fd2df8c4ba79d0ca098555d3097946fb3c1a parent 0b01a72

12 files changed, +372 −3

harness/RUNLOG.md +81 −0
harness/evidence/run11/preroll.log +32 −0
harness/evidence/run11/roll.log +140 −0
harness/evidence/run11/window.log +0 −0
harness/scenarios/cohort1/blocks/b4.sh +5 −1
harness/scenarios/cohort1/blocks/g10-preroll-bucket.sh +23 −0
harness/scenarios/cohort1/blocks/g10-preroll.host.sh +7 −0
harness/scenarios/cohort1/blocks/g10-preroll.leo.sh +25 −0
harness/scenarios/cohort1/blocks/g10-roll.host.sh +36 −0
harness/scenarios/cohort1/blocks/g10-window.host.sh +21 −0
harness/scenarios/cohort1/blocks/w81.box.sh +1 −1
harness/scenarios/cohort1/blocks/w82b.sh +1 −1

CI cancelled — superseded by a newer push — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m

amadan: job cancelled — a newer push superseded it

diff --git a/harness/RUNLOG.md b/harness/RUNLOG.md

index 09e935d9d..1144ffb53 100644

--- a/harness/RUNLOG.md

+++ b/harness/RUNLOG.md

@@ -2345,3 +2345,84 @@ wakecheck on sophie. New snapshots: `post-candidate`, `post-move-r10`. Host disk

G10 should check `df -h /` first. Host-only: `/root/harness/release/v0.28.8` and `/home/operator/release-v0.28.8` (this

run), Run 6's moved aside as `*-run6-8a313548`; sampler logs `/root/harness/r10-*`. Host TTL 2026-10-21T12:00Z.

DOCS-T1 scratch (`$TMPDIR/r10src` worktree, build outputs, patch, scripts) removed.

+# Run 11: G10 on 64a98473 (overnight 2026-10-20/21)

+Brief (coordinator, for the claim owner platform-1c): re-run G10 on the owner's roster-binding fix,

+`claim-empty-cohort-done` @ **64a98473** (CI green, census-ruling-roster-binding ruled, Astra approved), runbook

+`claim-dev-cohort1-runbook` @ 5860108f (corrections, the console quiesce, 8.2b's digest, B4's report wait and 202

+handling, "Pre-roll emptiness check for the roster-binding fix"). Order: restore post-8.2b, pre-roll check, the tip on

+leo by a plain `system update`, 8.1 to 8.2b only if the snapshot needs it, step 9, B4 and G10 (sophie's console left

+on the v0.28.4 stand-in, D10), acceptance and a second create. Harness only.

+## Disk first (23:06-23:08Z): 3.2 GB -> 9.6 GB free

+`df -h /` 54G/59G used, 3.2G free. Deleted, each identified as stale from this RUNLOG (superseded failure or

+experiment states of Runs 1-7 and 9; none on the keep list post-8.2b, pre-window, post-b5*, s7, post-move-r10):

+- snapshots, on every container holding them (infra, leo [Run 10's], sophie, dennis, signer, leo-btrfs,

+ leo-ext4-resume) and on `hdir/sophie-console`: `g4-fail-cc90d542`, `g4-held-8376a468`, `g4-fail-8376a468`,

+ `g4-parent-p-refused`, `rearm-released-v0.28.7`, `post-8.1` (Run 6), `g7-fail-8a313548`, `g7-f4-census`,

+ `post-8.1-465c3a08`, `g7-fail-465c3a08`, `post-8.1-465c3a08-c8`, `g7-fail-465c3a08-c8`, `g7-fail-465c3a08-ext4`,

+ `resume-complete-ext4`, `e-refused-0285`, `pre-move-r9`;

+- old release copies: `/root/harness/release/v0.28.9`, `v0.28.10`, `v0.28.14` (the unused debug build),

+ `v0.28.8-run6-8a313548`, and `/home/operator/release-v0.28.9`, `-v0.28.10`, `-v0.28.14`, `-v0.28.8-run6-8a313548`;

+- then `fstrim` on the btrfs pool (loop image `discard=async`): 20.4 GiB trimmed. Free after: 9.6 GB (84%).

+Kept: every other snapshot, including `post-8.1-run8`, `g10-stuck-run8`/`post-window` (Run 8's G10 evidence) and

+`item6-plain-update-fail`; `/root/harness/carlos-debug` (09:07Z, not identified, left).

+## The build: harness v0.28.15 = 64a98473

+`git worktree add --detach $TMPDIR/g10src 64a98473` + the harness patch (`git diff 6b905c11^ 6b905c11 -- internal`,

+sha256 `b2f1ad8f...`, the same seven files), `-tags harness`, `GOFLAGS=-buildvcs=false`, `CGO_ENABLED=0`, linux/arm64,

+`-X .../internal/version.Build=v0.28.15`, for `.` and `./cmd/console`. Tests ok: `./internal/s3compat

+./internal/publicrelease ./internal/rootaws ./internal/hostfloor`; `-tags harness ./internal/s3compat ./internal/publicrelease`.

+|---|---|---|---|---|

+Why v0.28.15: v0.28.12 to v0.28.14 were used in Run 8's lineage (v0.28.14 the unpublished debug build); v0.28.15 is

+the first tag used nowhere and newer than leo's v0.28.12. CV `64a98473`.

+## Step 1: restore, then the pre-roll emptiness check: PASS (23:11-23:12Z)

+**There is no `post-8.2b` snapshot.** Run 8 took none between 8.2b (12:29Z) and B4's stalled create; the next

+snapshot after 8.2b is `g10-stuck-run8`, which holds the stuck create and fails the check by design (the runbook says

+so). The latest pre-G10 snapshot after 8.2b does not exist, so I restored **`post-b5-run8b`**, Run 8's clean base

+(leo ext4 on v0.28.12 = e2c006d2, armed, B1/B2(list)/B5 done, roster rev 15, pre-window), and the window (8.1 to 8.2b)

+runs again on the tip. This is also dev's real order (the fix rolls at G4, before the window), and it means

+journal-migrate's census is the fixed code's. `post-8.1-run8` was not used: its units are masked mid-window, so a

+plain `system update` cannot reach leo there (the runbook's 8.2 note: a fix reaches a masked leo only by hand).

+Swap: `leo-swap.sh leo-ext4-r10 203.0.113.127 leo-ext4-run8b` (Run 10's leo kept stopped as `leo-ext4-r10`, with

+`pre-0285`, `post-0285`, `post-candidate`, `post-move-r10`), then `snapshot.sh restore post-b5-run8b` (23:11:05-18Z).

+The runbook's two blocks verbatim (`blocks/g10-preroll.leo.sh`, `g10-preroll-bucket.sh`, driver `g10-preroll.host.sh`;

+harness names; `evidence/run11/preroll.log`): every line `0`, `leo: all empty`; `host-claim-files 0`,

+`evidence-objects-for-leo 0`, `serving-evidence-objects-for-leo 0`, `roster-holds 0`, `roster-activations 0`,

+`bucket and roster: all empty`. Context: `/var/lib/carlos-claims` and `/var/lib/carlos-release/private/restarts`

+absent, leo `carlos v0.28.12`, roster rev 15, no backfill plan. (Note: the `tools present` line did not print: the

+harness leo has neither `sqlite3` nor `jq` (python3 only); harmless here because every database and file is absent, so no line needed them.)

+## Step 2: the tip on leo by a PLAIN `carlos system update`: PASS, no re-arm (23:13-23:24Z)

+`blocks/g10-roll.host.sh v0.28.15 v0.28.15-64a98473 3cdbf32b... 8b6c9b7b... 64a98473` (`evidence/run11/roll.log`).

+- **C2 first**: `release-window.sh` signed and verified the sums (`SHA256SUMS.txt` `4a20a04d...`, `Signature Verified

+ Successfully`), operator CLI `carlos v0.28.15`, console `64a98473` shipped and promoted to `canary/console`

+ 23:13:26Z (was `e2c006d2`).

+- **Signing session** `cohort1-20261020T231326Z` (signer `carlos v0.28.15`, jq-1.7, swap 0, `/run` tmpfs, key

+ `root 600 1 45`, 23:13:45Z); `publish-identity` v0.28.15 with CV `64a98473`: one create-only write of

+ `control/system/router/releases/v0.28.15.json` (`before_etag ""`, `db345e78...`; router `3cdbf32b...`, console

+ `8b6c9b7b...`); identity file = published object; REV 15.

+- **Promote stable 23:14:00Z** (`carlos system update --channel stable --version v0.28.15 ... --yes`, rc 0; "2 box(es)

+ will roll over 5m": kathleen-juggling and noor-marquetry on v0.28.12); roster 15 -> 16 with activation

+ `9efeb80ff44e8804c91f4d55c95e1157`. Canary (sophie) not promoted.

+- dennis 23:14:57-23:15:15Z, leo **23:18:32-23:18:52Z** `carlos-system-update.service` (root apply), both

+ `result.json` `"outcome": "applied"` v0.28.15 stable, `applied.json` v0.28.15.

+- Leo after: `carlos v0.28.15` / `.prev v0.28.12`; path active; guard identities `v0.28.5 .7 .8 .12 .15`; edge pid

+ 591 exe `3cdbf32b...` = the release, NRestarts 0; no `/root/carlos-*`. Report 23:24:40Z: release v0.28.15,

+ binary `3cdbf32b`, `activation_complete true`. Roster rev **17**, `activations []`, `holds []` (23:23:41Z).

+ `system status`: leo and dennis `v0.28.15 current`, sophie `canary v0.28.5 current`.

+- Sophie's console untouched: pid 353 exe `b2b431f3...` (the v0.28.4 stand-in, D10), router v0.28.5.

+So from a v0.28.5-equivalent parent (v0.28.12 carries 8376a468 and 8a313548) the fix release rolls onto an armed leo

+by the plain update; the ruled re-arm was not needed.

diff --git a/harness/evidence/run11/preroll.log b/harness/evidence/run11/preroll.log

new file mode 100644

index 000000000..9875ab0de

--- /dev/null

+++ b/harness/evidence/run11/preroll.log

@@ -0,0 +1,32 @@

+[hssm] i-06a1ec144aa5226fb fd771dbf-fcac-4583-9e5e-0e0b512b10b7 Success exit=0

+== block 1 on leo 23:11:59.164Z

+launches 0

+stop_inventory 0

+journal_inventory_versions 0

+journal_requests 0

+authority/fences 0

+authority/serving-parks 0

+retire/serving 0

+retire/cold 0

+retire/retired 0

+retire 0

+unit-bindings 0

+unit-materializations 0

+restarts 0

+migration-items 0

+leo: all empty

+(context: ls: cannot access '/var/lib/carlos-claims': No such file or directory; ls: cannot access '/var/lib/carlos-release/private/restarts': No such file or directory; carlos carlos v0.28.12)

+[block 1 exit=0]

+== block 2 bucket 23:11:59.302Z

+host-claim-files 0 # information only; the lines below decide

+evidence-objects-for-leo 0

+serving-evidence-objects-for-leo 0

+staged

+roster-holds 0

+roster-activations 0

+bucket and roster: all empty

+(context: roster rev 15; control/hosts-backfill.json absent)

+[block 2 exit=0]

diff --git a/harness/evidence/run11/roll.log b/harness/evidence/run11/roll.log

new file mode 100644

index 000000000..85a637e92

--- /dev/null

+++ b/harness/evidence/run11/roll.log

@@ -0,0 +1,140 @@

+[hssm] i-06a1ec144aa5226fb 0ccf2e18-6745-415b-ae45-ede6bd0b446f Success exit=0

+== signing session (C2 order) 23:13:22.537Z

+== release-window v0.28.15 (23:13:22.540Z)

+== release v0.28.15

+3cdbf32b3d49a670aa66666f30c73e247e23652d5cc41d7d099ce8ec7f1cc51e carlos-linux-arm64

+f9cdffad5c10a7f352646147a4937860c08f4b017e5594f3ed4e4cb37aa42d54 carlos_0.28.15_amd64.deb

+85c7dcbc65565d6825674128af05b86d830b5f5ba94439796b822ce02bde181a carlos_0.28.15_arm64.deb

+4a20a04d21b8d44a226abe3174ea4b659fae26cc28bf2c69b1723f4a50d056e6 SHA256SUMS.txt

+967cf0889002cfdc9294137cd683a3991b9fc2e5db1d326e56bc99546a108b34 SHA256SUMS.txt.sig

+8b6c9b7b4932697c1ec8309d3aa8713a29894b015a75e23c94a9ce1001bc6032 console-linux-arm64

+verify sums: Signature Verified Successfully

+operator CLI: carlos v0.28.15

+harness info: harness build: yes endpoint: http://203.0.113.10:9000 sums key: mAnSlX23... sha256:85f6f13cb78c

+== 1a/1e

+CHANNEL VERSION PROMOTED TAG DIGESTS

+canary/console e2c006d2 2026-10-20T12:16:52Z yes

+pointer digests: on for bab since 2026-10-20T01:12:47Z

+every channel's pointer carries digests: CARLOS_REQUIRE_POINTER_DIGEST can be turned on

+== 2

+8b6c9b7b4932697c1ec8309d3aa8713a29894b015a75e23c94a9ce1001bc6032 /home/operator/release-v0.28.15/console-linux-arm64

+== 4

+shipped console 64a98473 (1 artifacts)

+promoted console 64a98473 -> canary/console

+CHANNEL VERSION PROMOTED TAG DIGESTS

+canary/console 64a98473 2026-10-20T23:13:26Z yes

+== session open (23:13:26.555Z)

+run-id cohort1-20261020T231326Z

+2026-10-20 23:13:27 267 SHA256SUMS.txt

+2026-10-20 23:13:28 89 SHA256SUMS.txt.sig

+2026-10-20 23:13:29 49873058 carlos-linux-arm64

+2026-10-20 23:13:33 203 console.json

+2026-10-20 23:13:30 37 floor.json

+2026-10-20 23:13:31 244 leo.json

+2026-10-20 23:13:31 191 regions.json

+2026-10-20 23:13:27 45 release.pub

+2026-10-20 23:13:34 1795 signer-helpers.sh

+2026-10-20 23:13:32 240 sophie.json

+Success /usr/local/bin/carlos: OK

+carlos v0.28.15

+jq-1.7

+/usr/bin/jq

+SHA256SUMS.txt: OK

+SHA256SUMS.txt.sig: OK

+helpers.sh: OK

+release.pub: OK

+swap devices: 0

+/run: tmpfs

+session-start 2026-10-20T23:13:45Z

+root 600 1 45

+ 2026/10/20 23:13:37 ALERT: root AWS file, profile, endpoint and trust-store environment overrides ignored

+== publish-identity v0.28.15, console 64a98473 (23:13:48.764Z)

+Success "writes": [

+ "key": "control/system/router/releases/v0.28.15.json",

+ "before_etag": "",

+ "key": "bootstrap/console/v0.28.15/carlos-linux-arm64",

+ "key": "bootstrap/router/v0.28.15/carlos-linux-arm64",

+ "key": "bootstrap/console/v0.28.15/carlos-linux-arm64",

+ "key": "bootstrap/router/v0.28.15/carlos-linux-arm64",

+publish-identity writes: [{"key":"control/system/router/releases/v0.28.15.json","before_etag":"","sha256":"db345e78e7a6cbaa15e773ea74d730ebc042185e35367299431a899976752970","payload":{"schema":1,"component":"router","channel":"stable","current":{"console_artifacts":{"linux-arm64":{"key":"bootstrap/console/v0.28.15/carlos-linux-arm64","sha256":"8b6c9b7b4932697c1ec8309d3aa8713a29894b015a75e23c94a9ce1001bc6032"}},"release_tag":"v0.28.15","moves_protocol":1,"version":"v0.28.15","artifacts":{"linux-arm64":{"key":"bootstrap/router/v0.28.15/carlos-linux-arm64","sha256":"3cdbf32b3d49a670aa66666f30c73e247e23652d5cc41d7d099ce8ec7f1cc51e"}},"source":"release:v0.28.15","released_at":"0001-01-01T00:00:00Z"},"spread":"0s","promoted_at":"2026-10-20T23:13:50Z"}}]

+ "identity_key": "control/system/router/releases/v0.28.15.json",

+ "release": {

+ "console_artifacts": {

+ "linux-arm64": {

+ "key": "bootstrap/console/v0.28.15/carlos-linux-arm64",

+ "sha256": "8b6c9b7b4932697c1ec8309d3aa8713a29894b015a75e23c94a9ce1001bc6032"

+ }

+ },

+ "release_tag": "v0.28.15",

+ "moves_protocol": 1,

+ "version": "v0.28.15",

+ "artifacts": {

+ "linux-arm64": {

+ "key": "bootstrap/router/v0.28.15/carlos-linux-arm64",

+ "sha256": "3cdbf32b3d49a670aa66666f30c73e247e23652d5cc41d7d099ce8ec7f1cc51e"

+ }

+ },

+ "source": "release:v0.28.15",

+ "released_at": "0001-01-01T00:00:00Z"

+ }

+db345e78e7a6cbaa

+REV=15

+identity file = published object

+identity router 3cdbf32b3d49a670aa66666f30c73e247e23652d5cc41d7d099ce8ec7f1cc51e console 8b6c9b7b4932697c1ec8309d3aa8713a29894b015a75e23c94a9ce1001bc6032

+== done (23:13:57.156Z)

+REV after identity: {"rev":15,"activations":[],"holds":[]}

+box reports fresh at 23:13:59

+roster before: {"rev":15,"activations":[],"holds":[]}

+canary v0.28.5 promoted 2026-10-20T01:22:57Z spread 5m0s file

+stable v0.28.12 promoted 2026-10-20T12:22:46Z spread 5m0s file

+ada-letterpress linux-arm64 canary v0.28.5 current

+kathleen-juggling linux-arm64 stable v0.28.12 current

+noor-marquetry linux-arm64 stable v0.28.12 current

+-- leo before

+ carlos carlos v0.28.12 / prev carlos v0.28.8; path active

+ identities: v0.28.12.json v0.28.5.json v0.28.7.json v0.28.8.json

+ edge pid 232 exe sha 28923e4117924b4b NRestarts=0 ExecMainStartTimestamp=Tue 2026-10-20 23:11:14 UTC ActiveState=active

+-- dennis before

+ carlos carlos v0.28.12 / prev carlos v0.28.8; path active

+ edge pid 217 exe sha 28923e4117924b4b NRestarts=0 ExecMainStartTimestamp=Tue 2026-10-20 23:11:14 UTC ActiveState=active

+== promote stable v0.28.15 (23:14:00.260Z), plain system update, no step on any box

+promote v0.28.15 to router/stable — 2 box(es) will roll over 5m:

+ kathleen-juggling (linux-arm64, running v0.28.12)

+ noor-marquetry (linux-arm64, running v0.28.12)

+warning: v0.28.15 has not been on canary; promoting straight to stable

+ using uploaded v0.28.15 for linux-arm64

+ mirrored bootstrap/router/v0.28.15/carlos-linux-arm64

+ pointed router/stable at v0.28.15 (spread 5m0s)

+ serial bumped: boxes on stable start their spread

+promoted boxes on stable roll themselves; watch with: carlos system status --channel stable

+rc=0 roster: {"rev":16,"activations":["9efeb80ff44e8804c91f4d55c95e1157"],"holds":[]}

+== status 23:19:36.955Z

+canary v0.28.5 promoted 2026-10-20T01:22:57Z spread 5m0s file

+stable v0.28.15 promoted 2026-10-20T23:14:00Z spread 5m0s file

+ada-letterpress linux-arm64 canary v0.28.5 current

+kathleen-juggling linux-arm64 stable v0.28.15 current

+noor-marquetry linux-arm64 stable v0.28.15 current

+-- leo after

+ carlos carlos v0.28.15 / prev carlos v0.28.12; path active

+ identities: v0.28.12.json v0.28.15.json v0.28.5.json v0.28.7.json v0.28.8.json

+ edge pid 591 exe sha 3cdbf32b3d49a670 NRestarts=0 ExecMainStartTimestamp=Tue 2026-10-20 23:18:35 UTC ActiveState=active

+ journal: 2026-10-20T23:18:32.735979+00:00 leo systemd[1]: Starting carlos-system-update.service - carlos platform self-update (root apply)...

+ journal: 2026-10-20T23:18:52.278672+00:00 leo systemd[1]: Finished carlos-system-update.service - carlos platform self-update (root apply).

+-- dennis after

+ carlos carlos v0.28.15 / prev carlos v0.28.12; path active

+ edge pid 521 exe sha 3cdbf32b3d49a670 NRestarts=0 ExecMainStartTimestamp=Tue 2026-10-20 23:14:58 UTC ActiveState=active

+ journal: 2026-10-20T23:14:57.531608+00:00 dennis systemd[1]: Starting carlos-system-update.service - carlos platform self-update (root apply)...

+ journal: 2026-10-20T23:15:15.607325+00:00 dennis systemd[1]: Finished carlos-system-update.service - carlos platform self-update (root apply).

+23:23:41 roster {"rev":17,"activations":[],"holds":[]}

+roster end: {"rev":17,"activations":[],"holds":[]}

+{"generated_at":"2026-10-20T23:24:40.391637156Z","release":"v0.28.15","binary":"3cdbf32b3d49a670","activation_complete":true,"host_claim_enabled":null}

+sophie console pid 353 exe sha b2b431f3b2fb4600; router carlos v0.28.5

diff --git a/harness/evidence/run11/window.log b/harness/evidence/run11/window.log

new file mode 100644

index 000000000..e69de29bb

diff --git a/harness/scenarios/cohort1/blocks/b4.sh b/harness/scenarios/cohort1/blocks/b4.sh

index d7aa5201f..60f1eba80 100755

--- a/harness/scenarios/cohort1/blocks/b4.sh

+++ b/harness/scenarios/cohort1/blocks/b4.sh

@@ -15,7 +15,11 @@ box_inspect "$SOPHIE" >/dev/null; echo "rev $(cat box-rev)"

B=$(aws s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text); echo "dev-console.json before: $B"

echo "== 2 resume block on sophie $(date -u +%FT%T.%3NZ)"

../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/b4-resume.sophie.sh)"

-for i in $(seq 1 40); do A=$(aws s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text); [ "$A" != "$B" ] && break; sleep 3; done

+echo "== 3a the console's report must postdate the restart (runbook 5860108f, 3a) $(date -u +%FT%TZ)"

+RESTART=$(incus_restart=$(~/cohort1/ssm.sh eu-west-1 "$SOPHIE" 'systemctl show console.service -p ActiveEnterTimestamp --value' | sed -n 's/^Success[[:space:]]*//p' | head -1); date -u -d "$incus_restart" +%s)

+echo "console restart at $(date -u -d @$RESTART +%FT%TZ)"

+until [ "$(date -u -d "$(aws --region eu-west-1 s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text)" +%s)" -gt "$RESTART" ]; do sleep 20; done

+A=$(aws s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text)

echo "dev-console.json moved: $A ($(date -u +%FT%TZ))"

aws s3 cp --quiet s3://harness-deployment/status/consoles/dev-console.json - | jq -c '{generated_at, host_claim_enabled: (.metrics.host_claim_enabled // .host_claim_enabled), release: (.metrics.release // .release)}' || true

echo "== 3 $(date -u +%FT%TZ)"

diff --git a/harness/scenarios/cohort1/blocks/g10-preroll-bucket.sh b/harness/scenarios/cohort1/blocks/g10-preroll-bucket.sh

new file mode 100644

index 000000000..ebfeb9885

--- /dev/null

+++ b/harness/scenarios/cohort1/blocks/g10-preroll-bucket.sh

@@ -0,0 +1,23 @@

+#!/bin/bash

+# Run 11: the runbook's pre-roll emptiness check, block 2 (the bucket), from the operator shell. HARNESS: profile kass ->

+# harness, bucket carlos-dev-640012953448 -> harness-deployment, leo-climbing -> kathleen-juggling, SOPHIE from box-helpers.

+. ~/cohort1/box-helpers.sh

+mkdir -p ~/cohort1/out && cd ~/cohort1/out

+export AWS_PROFILE=harness

+B=harness-deployment

+R=eu-west-1

+LEOBOX=kathleen-juggling

+TOTAL=0

+cnt() { case "$2" in ''|*[!0-9]*) echo "$1 UNREADABLE"; TOTAL=$((TOTAL + 1000000)) ;; *) echo "$1 $2"; TOTAL=$((TOTAL + $2)) ;; esac; }

+set -o pipefail

+HOSTS=$(aws s3 ls "s3://$B/control/hosts/" | awk '$1 != "PRE" && $4 ~ /\.json$/ {print $4}')

+echo "host-claim-files $(printf '%s\n' "$HOSTS" | grep -c . || true) # information only; the lines below decide"

+for f in $HOSTS; do cnt "claim-names-leo $f" "$(aws s3 cp "s3://$B/control/hosts/$f" - | jq --arg b "$LEOBOX" '[.holder.writer_box?, .op.exec?, (.op.participants[]?), (.holder.admissions[]?.box?)] | map(select(. == $b)) | length')"; done

+KEYS=$(aws s3 ls --recursive "s3://$B/control/hosts/" | awk '{print $4}')

+cnt evidence-objects-for-leo "$(printf '%s\n' "$KEYS" | grep -cE "^control/hosts/[^/]+/evidence/[^/]+/$LEOBOX/" || true)"

+cnt serving-evidence-objects-for-leo "$(printf '%s\n' "$KEYS" | grep -cE "^control/hosts/[^/]+/serving-evidence/(.*/)?$LEOBOX/" || true)" # any depth (op/box/ too; platform-1c)

+box_inspect "$SOPHIE"

+cnt roster-holds "$(jq '.roster.holds // {} | length' box-inspect.json)"

+cnt roster-activations "$(jq '.roster.activations // {} | length' box-inspect.json)"

+test "$TOTAL" -eq 0 && echo "bucket and roster: all empty"

+echo "(context: roster rev $(jq .roster.revision box-inspect.json); control/hosts-backfill.json $(aws s3 ls s3://$B/control/hosts-backfill.json >/dev/null 2>&1 && echo present || echo absent))"

diff --git a/harness/scenarios/cohort1/blocks/g10-preroll.host.sh b/harness/scenarios/cohort1/blocks/g10-preroll.host.sh

new file mode 100644

index 000000000..74dc80a00

--- /dev/null

+++ b/harness/scenarios/cohort1/blocks/g10-preroll.host.sh

@@ -0,0 +1,7 @@

+#!/bin/bash

+# Run 11 step 1: the pre-roll emptiness check on harness leo (block 1) and the bucket (block 2), read-only.

+set -uo pipefail

+echo "== block 1 on leo $(date -u +%T.%3NZ)"

+incus exec leo -- bash < /home/operator/cohort1/blocks/g10-preroll.leo.sh; echo "[block 1 exit=$?]"

+echo "== block 2 bucket $(date -u +%T.%3NZ)"

+runuser -l operator -c "bash ~/cohort1/blocks/g10-preroll-bucket.sh" 2>&1 | grep -v ALERT; echo "[block 2 exit=${PIPESTATUS[0]}]"

diff --git a/harness/scenarios/cohort1/blocks/g10-preroll.leo.sh b/harness/scenarios/cohort1/blocks/g10-preroll.leo.sh

new file mode 100644

index 000000000..a487b87c5

--- /dev/null

+++ b/harness/scenarios/cohort1/blocks/g10-preroll.leo.sh

@@ -0,0 +1,25 @@

+# Run 11: the runbook's "Pre-roll emptiness check for the roster-binding fix", block 1 (leo, as root), verbatim

+# (runbook claim-dev-cohort1-runbook @ 5860108f).

+C=/var/lib/carlos-claims

+J=$C/journals.db

+TOTAL=0

+cnt() { case "$2" in ''|*[!0-9]*) echo "$1 UNREADABLE"; TOTAL=$((TOTAL + 1000000)) ;; *) echo "$1 $2"; TOTAL=$((TOTAL + $2)) ;; esac; }

+dbn() { if [ ! -e "$J" ]; then echo 0; elif [ "$(sqlite3 -readonly "$J" "select count(*) from sqlite_schema where type='table' and name='$1'")" = 0 ]; then echo 0; else sqlite3 -readonly "$J" "select count(*) from $1"; fi; }

+dirn() { if [ -d "$1" ]; then find "$1" -mindepth 1 ! -name '*.lock' | wc -l; else echo 0; fi; }

+command -v sqlite3 jq > /dev/null && echo "tools present"

+cnt launches "$(dbn launches)"

+cnt stop_inventory "$(dbn stop_inventory)"

+cnt journal_inventory_versions "$(dbn journal_inventory_versions)"

+cnt journal_requests "$(dbn journal_requests)"

+cnt authority/fences "$(dirn $C/authority/fences)"

+cnt authority/serving-parks "$(dirn $C/authority/serving-parks)"

+cnt retire/serving "$(dirn $C/retire/serving)"

+cnt retire/cold "$(dirn $C/retire/cold)"

+cnt retire/retired "$(dirn $C/retire/retired)"

+cnt retire "$(dirn $C/retire)"

+cnt unit-bindings "$(dirn $C/unit-bindings)"

+cnt unit-materializations "$(dirn $C/unit-materializations)"

+cnt restarts "$(dirn /var/lib/carlos-release/private/restarts)"

+cnt migration-items "$(if [ -e $C/authority/migration/migration.json ]; then jq '.Snapshot.Items | length' $C/authority/migration/migration.json; else echo 0; fi)"

+test "$TOTAL" -eq 0 && echo "leo: all empty"

+echo "(context: $(ls -d $C 2>&1); $(ls -d /var/lib/carlos-release/private/restarts 2>&1); carlos $(/opt/carlos/carlos version 2>/dev/null | tail -1))"

diff --git a/harness/scenarios/cohort1/blocks/g10-roll.host.sh b/harness/scenarios/cohort1/blocks/g10-roll.host.sh

new file mode 100644

index 000000000..95fd5c2a1

--- /dev/null

+++ b/harness/scenarios/cohort1/blocks/g10-roll.host.sh

@@ -0,0 +1,36 @@

+#!/bin/bash

+# Run 11 step 2: the tip onto leo by a PLAIN `carlos system update` (stable: leo and dennis), no box step.

+# C2: release-window.sh ships and promotes the release console to canary/console first, then the

+# signing session's publish-identity (r9-release.host.sh). Sophie (canary router, console = the

+# v0.28.4 stand-in per D10) is not promoted. g10-roll.host.sh <TAG> <ART> <carlos sha> <console sha> <CV>

+set -uo pipefail

+TAG=$1 ART=$2 CS=$3 COS=$4 CV=$5

+C=http://127.0.0.1:8090

+insp() { runuser -l operator -c 'carlos system claims inspect --bucket harness-deployment --aws-region eu-west-1 --public-key "$(cat ~/release.pub)" 2>/dev/null | jq -c "{rev: .roster.revision, activations: (.roster.activations // {} | keys), holds: (.roster.holds // {} | keys)}"'; }

+st() { runuser -l operator -c "carlos system status --console $C" 2>&1 | grep -v ALERT; }

+box() { incus exec $1 -- bash -c 'echo " carlos $(/opt/carlos/carlos version | tail -1) / prev $(/opt/carlos/carlos.prev version 2>&1 | tail -1); path $(systemctl is-active carlos-system-update.path)"

+ cat /opt/carlos/system/router/result.json 2>/dev/null | jq -c "{outcome, version, channel, promoted_at, reason}" 2>/dev/null | sed "s/^/ result: /"

+ cat /opt/carlos/system/router/applied.json 2>/dev/null | jq -c . 2>/dev/null | cut -c1-200 | sed "s/^/ applied: /"

+ ls /opt/carlos/system/guard/identities 2>/dev/null | tr "\n" " " | sed "s/^/ identities: /"; echo

+ P=$(systemctl show -p MainPID --value carlos-edge.service); echo " edge pid $P exe sha $(sha256sum < /proc/$P/exe | cut -c1-16) $(systemctl show carlos-edge -p NRestarts -p ActiveState -p ExecMainStartTimestamp | tr "\n" " ")"

+ ls /root/carlos-* 2>/dev/null

+ journalctl -u carlos-system-update.service --since "-25min" --no-pager -o short-iso-precise | grep -E "Starting|Finished|refus|guard|apply|error|fail" | tail -8 | sed "s/^/ journal: /"'; }

+echo "== signing session (C2 order) $(date -u +%T.%3NZ)"

+bash /home/operator/cohort1/blocks/r9-release.host.sh $TAG $ART $CS $COS $CV 2>&1 | grep -vE '^help\| '

+echo "REV after identity: $(insp)"

+T0=$(date -u +%s)

+fresh() { runuser -l operator -c 'for b in ada-letterpress kathleen-juggling noor-marquetry; do date -u -d "$(aws s3api head-object --bucket harness-deployment --key status/boxes/$b.json --query LastModified --output text)" +%s; done' | sort -n | head -1; }

+for i in $(seq 1 60); do [ "$(fresh)" -gt "$((T0-300))" ] && break; sleep 10; done; echo "box reports fresh at $(date -u +%T)"

+echo "roster before: $(insp)"; st

+for b in leo dennis; do echo "-- $b before"; box $b; done

+echo "== promote stable $TAG ($(date -u +%T.%3NZ)), plain system update, no step on any box"

+runuser -l operator -c "carlos system update --console $C --channel stable --version $TAG --file ~/release-$TAG/carlos-linux-arm64 --platform linux-arm64 --yes" 2>&1 | grep -v ALERT

+echo "rc=${PIPESTATUS[0]} roster: $(insp)"

+for i in $(seq 1 60); do s=$(st); [ "$(echo "$s" | grep -cE "$TAG *current")" -ge 2 ] && break; sleep 15; done

+echo "== status $(date -u +%T.%3NZ)"; echo "$s"

+for b in leo dennis; do echo "-- $b after"; box $b; done

+for i in $(seq 1 40); do a=$(insp); case "$a" in *'"activations":[]'*) echo "$(date -u +%T) roster $a"; break;; esac; sleep 15; done

+echo "roster end: $(insp)"

+T1=$(date -u +%s); for i in $(seq 1 50); do t=$(runuser -l operator -c 'date -u -d "$(aws s3api head-object --bucket harness-deployment --key status/boxes/kathleen-juggling.json --query LastModified --output text)" +%s'); [ "$t" -gt "$T1" ] && break; sleep 10; done

+runuser -l operator -c 'aws s3 cp --quiet s3://harness-deployment/status/boxes/kathleen-juggling.json - | jq -c "{generated_at, release: .metrics.release, binary: .metrics.binary_sha256[0:16], activation_complete: .metrics.activation_complete, host_claim_enabled: .metrics.host_claim_enabled}"'

+incus exec sophie -- bash -c 'p=$(systemctl show -p MainPID --value console.service); echo "sophie console pid $p exe sha $(sha256sum < /proc/$p/exe | cut -c1-16); router $(/opt/carlos/carlos version | tail -1)"'

diff --git a/harness/scenarios/cohort1/blocks/g10-window.host.sh b/harness/scenarios/cohort1/blocks/g10-window.host.sh

new file mode 100644

index 000000000..6331d7cba

--- /dev/null

+++ b/harness/scenarios/cohort1/blocks/g10-window.host.sh

@@ -0,0 +1,21 @@

+#!/bin/bash

+# Run 11 step 3: the window on the tip, 8.1 to 8.2b, snapshot post-8.2b, then 8.4, step 9 and G9's checks.

+# Stops at the first block that does not reach its DONE/END line. g10-window.host.sh <WINDOW_ID> [from]

+set -uo pipefail

+WID=$1; FROM=${2:-81}

+D=/home/operator/cohort1/blocks

+step() { local name=$1 marker=$2; shift 2; echo "######## $name $(date -u +%T.%3NZ)"; out=$("$@" 2>&1); echo "$out" | grep -v '^help| '; echo "$out" | grep -q "$marker" || { echo "STOP: $name did not reach '$marker'"; exit 1; }; }

+op() { runuser -l operator -c "$*"; }

+if [ "$FROM" = 81 ]; then

+step 8.1 "8.1 DONE" bash $D/run-leo.sh $WID w81.box.sh

+step 8.1b "8.1b END" op "bash ~/cohort1/blocks/w81b.sh $WID"

+step 8.2 "8.2 DONE" bash $D/run-leo.sh $WID --env w82.box.sh

+step 8.2b "8.2b END" op "bash ~/cohort1/blocks/w82b.sh $WID"

+echo "######## snapshot post-8.2b $(date -u +%T.%3NZ)"; bash /root/harness/snapshot.sh create post-8.2b 2>&1 | tail -8

+for c in leo sophie; do incus exec $c -- systemctl --failed --no-legend --plain; done

+fi

+step 8.4 "8.4 DONE" bash $D/run-leo.sh $WID --env w84.box.sh

+step 9 "9 DONE" bash $D/run-leo.sh $WID --env w9.box.sh

+echo "######## G9 checks $(date -u +%T.%3NZ)"; op "bash ~/cohort1/blocks/g9-check.sh"

+echo "WINDOW END $(date -u +%T.%3NZ)"

diff --git a/harness/scenarios/cohort1/blocks/w81.box.sh b/harness/scenarios/cohort1/blocks/w81.box.sh

index d2880b636..60b9bd639 100755

--- a/harness/scenarios/cohort1/blocks/w81.box.sh

+++ b/harness/scenarios/cohort1/blocks/w81.box.sh

@@ -53,5 +53,5 @@ for unit in "${UNITS[@]}"; do

done

systemctl daemon-reload

systemctl show "${UNITS[@]}" -p Id -p LoadState -p ActiveState

-pub /opt/carlos/carlos system claims install-control --dry-run

+# (C7, runbook 5860108f: 8.1's closing install-control --dry-run is deleted)

echo "8.1 DONE $(date -u +%T)"

diff --git a/harness/scenarios/cohort1/blocks/w82b.sh b/harness/scenarios/cohort1/blocks/w82b.sh

index 036296da1..2d045fcd1 100755

--- a/harness/scenarios/cohort1/blocks/w82b.sh

+++ b/harness/scenarios/cohort1/blocks/w82b.sh

@@ -34,7 +34,7 @@ jq -e '(.backfill.reports | keys) == [\"kathleen-juggling\"]' backfill-done.json

jq -e '(.writes | length) == 1' backfill-done.json

jq -e '.writes[0].key == \"control/hosts-backfill.json\"' backfill-done.json

jq -e '(.writes[0].before_etag | length) > 0' backfill-done.json

-# Digest over the census object's raw bytes as stored. PENDING: confirmed in harness <run-id>.

+# Digest over the census object's raw bytes as stored: CONFIRMED in harness run cohort1-20261020T121705Z (sha256.Sum256 of the Store.Get body, internal/hostops/backfill_publish.go:219, 233-234).

LEO_CENSUS_SHA=\$(aws --region \"\$R\" s3 cp \"s3://\$B/control/host-census/\$BACKFILL_ID/kathleen-juggling.json\" - | sha256sum | cut -d' ' -f1)

test -n \"\$LEO_CENSUS_SHA\"

echo \"LEO_CENSUS_SHA=\$LEO_CENSUS_SHA reports=\$(jq -c .backfill.reports backfill-done.json)\"