amadan
harness: blocks 6.3 to 7a, ext4 console volume and asset-tag shims, defect 4 reproduced, snapshot s7 (step 10)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
18 files changed,
+420
−0
harness/RUNLOG.md+108 −0harness/boxes/dmi-shim.sh+34 −0harness/console/console-ext4.sh+24 −0harness/scenarios/cohort1/blocks/61-sign.sh+0 −0harness/scenarios/cohort1/blocks/61a-install-channel.box.sh+0 −0harness/scenarios/cohort1/blocks/62-guard-arm.box.sh+0 −0harness/scenarios/cohort1/blocks/63-run.sh+14 −0harness/scenarios/cohort1/blocks/63-sophie.box.sh+28 −0harness/scenarios/cohort1/blocks/7-console-report.box.sh+11 −0harness/scenarios/cohort1/blocks/7-evidence-key.box.sh+11 −0harness/scenarios/cohort1/blocks/7-run.sh+53 −0harness/scenarios/cohort1/blocks/7-runtime.box.sh+17 −0harness/scenarios/cohort1/blocks/74-run.sh+24 −0harness/scenarios/cohort1/blocks/7a-leo.box.sh+27 −0harness/scenarios/cohort1/blocks/7a-run.sh+25 −0harness/scenarios/cohort1/blocks/p9.box.sh+17 −0harness/scenarios/cohort1/blocks/reports.sh+6 −0harness/snapshot.sh+21 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.012s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/harness/RUNLOG.md b/harness/RUNLOG.md| index d25b59146..1ea673086 100644 |
| --- a/harness/RUNLOG.md |
| +++ b/harness/RUNLOG.md |
| @@ -582,3 +582,111 @@ The rebuilt-box route of #47 (host-init and bootstrap-box.sh from `747f699`) was |
| its host-init is in `harness/infra-copies/747f699/` for a `base-opt` build. |
| **Step 9: GREEN** (defect 1: refused on main's host-init, passes after #47's remediation), 01:35Z. |
| + |
| +## Step 10: blocks 6.3, 7 and 7a; snapshot `s7` |
| + |
| +Blocks: `harness/scenarios/cohort1/blocks/63-run.sh` (+ `p9.box.sh`, `63-sophie.box.sh`, `reports.sh`), |
| +`7-run.sh` (+ `7-console-report.box.sh`, `7-evidence-key.box.sh`), `74-run.sh` (+ `7-runtime.box.sh`), |
| +`7a-run.sh` (+ `7a-leo.box.sh`). Command changes beyond step 9's table: |
| +- 6.3: `CONSOLE_SOURCE=/opt/carlos/versions/1b635e61/console`, `ada-letterpress-enrollment.json`. |
| +- P9: the box command is verbatim; its two DOCS-T1 lines (`aws iam list-role-policies` on |
| + `carlos-dev-host`, the platform-infrastructure grep) are not run: no IAM in the harness, and the |
| + dev role must not be touched. |
| +- 7.1: dennis's check runs through the stand-in with region `eu-west-2` as written. |
| +- The signer's key was fetched again (signing-host.md's command, verbatim) at 7.1, before 7.2, |
| + because the `pre62` snapshot restart cleared the signer's tmpfs. |
| +- 7.3: `--box kathleen-juggling`; `key-proof.json` goes operator -> signer through the exchange. |
| +- 7a.2 part 1 runs on the signer with its own role (the runbook's line carries `--profile kass`); |
| + `fence-credentials.json` was seeded root 0600 on the signer from the fence writer's MinIO user |
| + (H2); `--key-id alias/cohort1-fence-leo` is accepted and ignored (no KMS); part 5 (tofu, KMS) has |
| + no counterpart. |
| +- "Closing the signing session": key shredded, exchange objects removed by exact key; the signer |
| + container is **kept** (not deleted, as the spec's close says) because the window's signer phases |
| + (8.1b, 8.2b) need it; they open a new session. |
| + |
| +### Two harness defects found and fixed on the way (both now declared shims) |
| + |
| +- **H11, `FS_IOC_GETFSUUID` on btrfs.** With `CARLOS_HOST_CLAIM=1` the guarded console exited at |
| + once with `inappropriate ioctl for device`, 59 restarts. Claim-mode console SQL reads the console |
| + DB filesystem's UUID with `FS_IOC_GETFSUUID` (`internal/console/claim_sql_identity_linux.go`). |
| + Measured on the dev AMI's kernel `6.12.96+deb13-cloud-arm64`: ext4 (the host's root) answers |
| + (`len 16`); btrfs (the harness pool, sophie's `/data`) answers `[Errno 25] Inappropriate ioctl for |
| + device`. Dev's `/data` is ext4, so this is the harness's storage, not dev's. Fix |
| + (`harness/console/console-ext4.sh`): sophie's `/data/console` is now a custom volume in an incus |
| + `dir` pool (`hdir`) on the host's ext4, bind-mounted; `harness/snapshot.sh` snapshots it with the |
| + containers. **Finding for platform:** `sqlFilesystemSupported` lists btrfs (`0x9123683e`) as a |
| + filesystem whose `FS_IOC_GETFSUUID` works, but on this kernel btrfs refuses it, so a console on |
| + btrfs cannot turn claims on. The comment and the list need checking against real kernels. |
| +- **H12, the Nitro asset tag.** Next the console refused `identity mismatch: enrolled on instance |
| + "i-019177034b088459f" but running on "i-06a1ec144aa5226fb"`: the **harness host's** id. The |
| + console reads `/sys/devices/virtual/dmi/id/board_asset_tag` before IMDS |
| + (`claim_sql_instance.go`), and containers share the host's `/sys`. Fix |
| + (`harness/boxes/dmi-shim.sh`): each box bind-mounts a file with its own instance id (the same id |
| + the IMDS stand-in serves) over the asset tag, from a oneshot unit ordered before the edge and the |
| + console. Any harness reader of DMI now sees the box's id; the IMDS-only stand-in missed this. |
| + |
| +### Results |
| + |
| +**6.3** (01:37:05-01:37:19Z): |
| +``` |
| +P9: Success checked: the running console (pid 215), console.service Environment=, and 5 files: /etc/carlos/host.env /etc/carlos/release.env |
| + /etc/systemd/system/console.service /etc/systemd/system/console.service.d/release-key.conf /run/systemd/system/service.d/zzz-lxc-service.conf |
| + console mail: OFF (none of the 7 names is set) (zzz-lxc-service.conf is LXC's own drop-in on every unit in a container) |
| +box_inspect sophie: staged x2, rev 5 |
| +install-console (dry run, apply): release_tag v0.28.5, sha256 b2b431f3...17e0, path /opt/carlos/system/console/console, |
| + previous .../console.prev, mode 0755, source /opt/carlos/versions/1b635e61/console |
| +guard-arm --console, under the block's umask 077: 755 root:root / 755 root:root, test -x passed <- #317's fix holds on 96db0efe |
| +console-config (dry run, apply): /etc/carlos/claim-console.env "CARLOS_CONSOLE_ID=dev-console CONSOLE_HOME_REGION=ie CARLOS_HOST_CLAIM=", |
| + drop-in 91-claim-runtime.conf, reload true, start false; console.service active |
| +reports (01:41:48Z): kathleen-juggling 01:41:19Z v0.28.5 111a317c True 53d81bb7 0 |
| + ada-letterpress 01:41:39Z v0.28.5 111a317c True 53d81bb7 0 |
| +``` |
| +**7.1** (01:42:36Z): `system status` all three `current` v0.28.5; leo and sophie `carlos v0.28.5` twice; |
| +dennis `carlos v0.28.5` / `.prev harness-f434a425` (not enrolled, never `install-channel`ed); operator CLI |
| +v0.28.5. Dennis: `Success absent /opt/carlos/system/guard/policy.json / absent /etc/carlos/host-actor.json / |
| +carlos v0.28.5` (exact pass). Signer key re-fetch: `root 600 1 45`. |
| + |
| +**7.2**: `console-report` at 01:43:01.999Z (dry run, apply; console process `b2b431f3...`, guard floor |
| +`53d81bb7...`), then `publish-floor` and `home` from 01:43:07.938Z: writes `control/host-actors.json` x6, |
| +`control/moves-protocol.json` x4, `control/moves-protocol-auth.json` x2 (counts of `"key"` lines across |
| +dry runs and applies); roster revision 8, home `ie`; `moves-protocol.json` 33 bytes, |
| +`moves-protocol-auth.json` 367 bytes. **publish-floor straight after console-report: PASS.** |
| + |
| +**7.3** (01:43:14Z): evidence-key dry run, apply; `key-proof.json` fetched (`e24175f7`); `bind-key` -> |
| +revision 9; `jq -e '.roster.home_region == "ie" and .roster.publication == "published" and |
| +.roster.request_limit_ns == 120000000000'` -> **`true`**; `/etc/carlos/host-evidence.key` `root 600 1`. |
| + |
| +**7.4** (after H11 and H12; first attempt at 01:44Z ended in the crash loop above): |
| +``` |
| +01:49:01.922Z console-config --enable-console-runtime (dry run, apply): CARLOS_HOST_CLAIM=1; restart; active |
| +01:49:03 console-report dry run: "claim activation: input or protected state refused" <- DEFECT 4 REPRODUCED |
| + console restarted at 01:49:03 UTC, pid 2689 |
| +01:49:07.042Z status/consoles/dev-console.json moved (generated_at 01:49:03.487Z, 0.5 s after start; |
| + release v0.28.5, process ...:2689:406309, floor_guard_ready true, host_claim_enabled true) |
| +01:49:16.435Z box_inspect, console-report again: Success (dry run, apply), process_id ...:2689:406309 |
| +``` |
| +`status/consoles/dev-console.json` exists (step 8's deferred check), first at 01:42:19Z, after 6.3. |
| + |
| +**7a.2** (01:49:58-01:50:13Z): part 1 `root 600 1`, put-parameter version `1`, fingerprint `99c21d0c`, |
| +role ARN `staged`; part 2 `root 600 1 223`, `99c21d0c`, both `fence-credentials` runs print |
| +`"target": "/etc/carlos/fence-writer/credentials.json"`, `"mode": "0600"`, role |
| +`arn:aws:iam::640012953448:role/carlos-dev-ie-b-fence-writer`; `subshell exit=0`; `staging file gone`; |
| +installed `root 600 1`. Parts 3 and 4: `deleted`, describe empty. |
| + |
| +**Session closed** (01:50Z): `key shredded`, `/run/claim-signer` gone; 17 exchange objects removed by exact |
| +key, 0 left. Release-key reads in the SSM stand-in's log: 01:12:47Z and 01:12:48Z (step 7's digests-on |
| +and ship-app promote, operator, before the session), 01:32:25Z (session open), 01:42:46Z (re-fetch after |
| +the `pre62` restart). The stand-in logs no caller address, so reads are attributed by time; adding the |
| +source address is a P2 item. |
| + |
| +**Snapshot `s7`** (`harness/snapshot.sh create s7`, 01:51:12-01:51:26Z): all five containers and the |
| +`hdir/sophie-console` volume; after the restart all `running`, `--failed` empty, console 200. |
| + |
| +**Defect 2.** On `96db0efe` (#317's fix) the 6.3 block's `umask 077` gives `755`: the defect does not |
| +recur. The pre-#317 code path was not rebuilt: a pre-#317 binary has another sha256 and would need its |
| +own release, identity and floor to reach `guard-arm`. Instead, the **effect** was replayed from `s7` |
| +(01:51:43Z): `chmod 0700 /opt/carlos/system/guard` (what pre-#317 `guard-arm` left under `umask 077`), |
| +restart: `status=203/EXEC`, `Failed to execute /opt/carlos/system/guard/carlos: Permission denied`, |
| +console `/healthz` 000. Restored `s7` (01:51:47Z): `755`, console 200. |
| + |
| +**Step 10: GREEN** (01:52Z). Defects reproduced: 1 (step 9), 2 (effect replay; the fix holds), 4. |
diff --git a/harness/boxes/dmi-shim.sh b/harness/boxes/dmi-shim.sh| new file mode 100755 |
| index 000000000..b9eb37202 |
| --- /dev/null |
| +++ b/harness/boxes/dmi-shim.sh |
| @@ -0,0 +1,34 @@ |
| +#!/bin/bash |
| +# Harness identity shim (H12): each box's Nitro asset tag says its own instance id. |
| +# The console's claim-mode SQL identity reads /sys/devices/virtual/dmi/id/board_asset_tag |
| +# before IMDS (internal/console/claim_sql_instance.go). A container shares the host's |
| +# /sys, so without this every box would read the HARNESS HOST's instance id. The file |
| +# holds the same id the IMDS stand-in serves (the donor's for leo and sophie). |
| +set -euo pipefail |
| +T=$(mktemp -d); trap 'rm -rf "$T"' EXIT |
| +declare -A ID=([leo]=$(python3 -c 'import json;print(json.load(open("/etc/harness/donors/donor-1/document"))["instanceId"])') |
| + [sophie]=$(python3 -c 'import json;print(json.load(open("/etc/harness/donors/donor-2/document"))["instanceId"])') |
| + [dennis]=i-00000000000000023) |
| +cat > $T/unit <<'U' |
| +[Unit] |
| +Description=harness shim: this box's instance id as its Nitro asset tag |
| +DefaultDependencies=no |
| +Before=carlos-edge.service console.service sysinit.target |
| +ConditionPathExists=/etc/harness/board_asset_tag |
| + |
| +[Service] |
| +Type=oneshot |
| +RemainAfterExit=yes |
| +ExecStart=/bin/mount --bind -o ro /etc/harness/board_asset_tag /sys/devices/virtual/dmi/id/board_asset_tag |
| + |
| +[Install] |
| +WantedBy=sysinit.target |
| +U |
| +for b in leo sophie dennis; do |
| + printf '%s\n' "${ID[$b]}" > $T/tag |
| + incus exec $b -- install -d -m 0755 /etc/harness |
| + incus file push $T/tag $b/etc/harness/board_asset_tag --uid 0 --gid 0 --mode 0644 -q |
| + incus file push $T/unit $b/etc/systemd/system/harness-dmi-instance.service --uid 0 --gid 0 --mode 0644 -q |
| + incus exec $b -- sh -c 'systemctl daemon-reload; systemctl enable --now harness-dmi-instance.service 2>/dev/null; echo "'$b': asset tag $(cat /sys/devices/virtual/dmi/id/board_asset_tag)"' |
| +done |
| +echo "host's own: $(cat /sys/devices/virtual/dmi/id/board_asset_tag)" |
diff --git a/harness/console/console-ext4.sh b/harness/console/console-ext4.sh| new file mode 100755 |
| index 000000000..69d583848 |
| --- /dev/null |
| +++ b/harness/console/console-ext4.sh |
| @@ -0,0 +1,24 @@ |
| +#!/bin/bash |
| +# Harness storage shim (H11): sophie's /data/console on ext4. |
| +# Claim-mode console SQL needs FS_IOC_GETFSUUID on the console DB's filesystem |
| +# (internal/console/claim_sql_identity_linux.go). On the dev AMI's kernel |
| +# (6.12.96+deb13-cloud-arm64) ext4 answers it and btrfs answers ENOTTY, so the |
| +# harness's btrfs pool cannot run a claims-on console; dev's /data is ext4. |
| +# A custom volume in an incus "dir" pool on the host's ext4 root is bind-mounted |
| +# at /data/console. Snapshot it with the containers: |
| +# incus storage volume snapshot create hdir sophie-console <name> |
| +set -euo pipefail |
| +install -d -m 0711 /var/lib/harness-dir |
| +incus storage show hdir >/dev/null 2>&1 || incus storage create hdir dir source=/var/lib/harness-dir >/dev/null |
| +incus storage volume show hdir sophie-console >/dev/null 2>&1 || incus storage volume create hdir sophie-console >/dev/null |
| +incus exec sophie -- systemctl stop console.service |
| +incus storage volume attach hdir sophie-console sophie /data/console-ext4 |
| +incus exec sophie -- sh -c 'cp -a /data/console/. /data/console-ext4/ && chown carlos:carlos /data/console-ext4 && chmod 0750 /data/console-ext4' |
| +incus storage volume detach hdir sophie-console sophie |
| +incus exec sophie -- sh -c 'rm -rf /data/console /data/console-ext4 && install -d -o carlos -g carlos -m 0750 /data/console' |
| +incus storage volume attach hdir sophie-console sophie /data/console |
| +incus exec sophie -- sh -c 'stat -f -c "/data/console: %T" /data/console; stat -c "%U:%G %a %n" /data/console /data/console/console.db; systemctl start console.service' |
| +sleep 5 |
| +sleep 3 |
| +incus exec sophie -- sh -c 'systemctl show -p ActiveState,SubState,MainPID console.service | paste -sd" "; journalctl -u console.service --since -30s --no-pager -o cat | tail -3 | cut -c1-200' |
| +curl -s -o /dev/null -w 'listener /healthz -> %{http_code}\n' http://127.0.0.1:8090/healthz |
diff --git a/harness/scenarios/cohort1/blocks/61-sign.sh b/harness/scenarios/cohort1/blocks/61-sign.sh| old mode 100644 |
| new mode 100755 |
diff --git a/harness/scenarios/cohort1/blocks/61a-install-channel.box.sh b/harness/scenarios/cohort1/blocks/61a-install-channel.box.sh| old mode 100644 |
| new mode 100755 |
diff --git a/harness/scenarios/cohort1/blocks/62-guard-arm.box.sh b/harness/scenarios/cohort1/blocks/62-guard-arm.box.sh| old mode 100644 |
| new mode 100755 |
diff --git a/harness/scenarios/cohort1/blocks/63-run.sh b/harness/scenarios/cohort1/blocks/63-run.sh| new file mode 100755 |
| index 000000000..fed7a1c9a |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/63-run.sh |
| @@ -0,0 +1,14 @@ |
| +#!/bin/bash |
| +# Runbook 6.3 from the operator: P9 re-run, box_inspect sophie, the sophie block, then the report wait. |
| +set -euo pipefail |
| +. ~/cohort1/values.env; cd ~/cohort1/out |
| +echo "== P9 ($(date -u +%TZ))"; ../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/p9.box.sh)" |
| +. ../box-helpers.sh |
| +echo "== box_inspect sophie"; box_inspect "$SOPHIE"; echo "rev $(cat box-rev)" |
| +echo "== sophie block ($(date -u +%TZ))"; ../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/63-sophie.box.sh)" |
| +echo "== waiting for floor_guard_ready on both ($(date -u +%TZ))" |
| +for i in $(seq 1 50); do |
| + out=$(../blocks/reports.sh 2>/dev/null) |
| + [ "$(echo "$out" | grep -c ' True ')" -ge 2 ] && break; sleep 10 |
| +done |
| +date -u +%TZ; echo "$out" |
diff --git a/harness/scenarios/cohort1/blocks/63-sophie.box.sh b/harness/scenarios/cohort1/blocks/63-sophie.box.sh| new file mode 100755 |
| index 000000000..7af7c5411 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/63-sophie.box.sh |
| @@ -0,0 +1,28 @@ |
| +# Runbook 6.3, the sophie block (verbatim; HARNESS: CONSOLE_SOURCE versions/f3bf1214 -> versions/1b635e61, |
| +# bucket, identity/policy v0.28.5, enrollment file ada-letterpress-enrollment.json). |
| +set -euo pipefail |
| +umask 077 |
| +cd /root/claim-trial |
| +set -a |
| +. /etc/carlos/host.env |
| +set +a |
| +PUB=$(cat release.pub) |
| +CID=$(cat console-id) |
| +REV=$(cat rev) |
| +CONSOLE_SOURCE=/opt/carlos/versions/1b635e61/console |
| +CONSOLE_BINARY=/opt/carlos/system/console/console |
| +/opt/carlos/carlos system claims install-console --identity /root/claim-trial/identity.v0.28.5.json --source "$CONSOLE_SOURCE" \ |
| + --bucket harness-deployment --aws-region eu-west-1 --public-key "$PUB" --dry-run |
| +/opt/carlos/carlos system claims install-console --identity /root/claim-trial/identity.v0.28.5.json --source "$CONSOLE_SOURCE" \ |
| + --bucket harness-deployment --aws-region eu-west-1 --public-key "$PUB" |
| +/opt/carlos/carlos system guard-arm --policy /root/claim-trial/policy.v0.28.5.json \ |
| + --identity /root/claim-trial/identity.v0.28.5.json --enrollment /root/claim-trial/ada-letterpress-enrollment.json \ |
| + --console "$CONSOLE_BINARY" |
| +stat -c '%a %U:%G' /opt/carlos/system/guard /opt/carlos/system/guard/identities |
| +sudo -u carlos test -x /opt/carlos/system/guard/carlos |
| +/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --console-id "$CID" --public-key "$PUB" --dry-run |
| +/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --console-id "$CID" --public-key "$PUB" |
| +systemctl restart console.service |
| +systemctl is-active console.service |
diff --git a/harness/scenarios/cohort1/blocks/7-console-report.box.sh b/harness/scenarios/cohort1/blocks/7-console-report.box.sh| new file mode 100755 |
| index 000000000..23d998236 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7-console-report.box.sh |
| @@ -0,0 +1,11 @@ |
| +# Runbook 7.2, the sophie block (verbatim; HARNESS: bucket). |
| +set -euo pipefail |
| +umask 077 |
| +cd /root/claim-trial |
| +PUB=$(cat release.pub) |
| +CID=$(cat console-id) |
| +REV=$(cat rev) |
| +/opt/carlos/carlos system claims console-report --bucket harness-deployment --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" --dry-run |
| +/opt/carlos/carlos system claims console-report --bucket harness-deployment --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" |
diff --git a/harness/scenarios/cohort1/blocks/7-evidence-key.box.sh b/harness/scenarios/cohort1/blocks/7-evidence-key.box.sh| new file mode 100755 |
| index 000000000..11e69f2e1 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7-evidence-key.box.sh |
| @@ -0,0 +1,11 @@ |
| +# Runbook 7.3, the leo block (verbatim; HARNESS: bucket, box name kathleen-juggling). |
| +set -euo pipefail |
| +umask 077 |
| +cd /root/claim-trial |
| +PUB=$(cat release.pub) |
| +REV=$(cat rev) |
| +LE=$(cat le) |
| +/opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --box kathleen-juggling --enrollment "$LE" --public-key "$PUB" --dry-run |
| +/opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --box kathleen-juggling --enrollment "$LE" --public-key "$PUB" > key-proof.json |
diff --git a/harness/scenarios/cohort1/blocks/7-run.sh b/harness/scenarios/cohort1/blocks/7-run.sh| new file mode 100755 |
| index 000000000..e91e55187 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7-run.sh |
| @@ -0,0 +1,53 @@ |
| +#!/bin/bash |
| +# Runbook step 7 from the operator. 7.1 readers and dennis; the signer's key re-fetch (it was |
| +# lost with the pre62 restart; fetched BEFORE 7.2 so nothing sits between console-report and |
| +# publish-floor); 7.2 console-report then publish-floor and home at once; 7.3 leo's key. |
| +set -euo pipefail |
| +. ~/cohort1/values.env; cd ~/cohort1/out |
| +RUN=$(cat ../run-id); X=s3://harness-deployment/claim-signer/$RUN |
| +. ../box-helpers.sh |
| +echo "== 7.1 P3 ($(date -u +%TZ))" |
| +carlos system status --console http://127.0.0.1:8090 |
| +for b in "$LEO" "$SOPHIE" "$DENNIS"; do ../ssm.sh eu-west-1 "$b" '/opt/carlos/carlos version; [ -e /opt/carlos/carlos.prev ] && /opt/carlos/carlos.prev version' | head -2 | tr '\n\t' ' '; echo " ($b)"; done |
| +carlos version |
| +echo "== 7.1 dennis unarmed (verbatim)" |
| +../ssm.sh eu-west-2 "$DENNIS" 'p=/opt/carlos/system/guard/policy.json; if [ -e $p ]; then echo "present $p $(grep -oE "\"mode\" *: *\"[^\"]*\"" $p | head -1)"; else echo "absent $p"; fi; [ -e /etc/carlos/host-actor.json ] && echo "present /etc/carlos/host-actor.json" || echo "absent /etc/carlos/host-actor.json"; /opt/carlos/carlos version' |
| +echo "== signer key re-fetch (signing-host.md, verbatim) ($(date -u +%TZ))" |
| +../ssm.sh eu-west-1 "$SIGNER" 'install -d -m 0700 -o root -g root /run/claim-signer |
| +aws --region eu-west-1 ssm get-parameter --name /carlos/release-key --with-decryption \ |
| + --query Parameter.Value --output text | install -m 0600 -o root -g root /dev/stdin /run/claim-signer/release.key |
| +stat -c "%U %a %h %s" /run/claim-signer/release.key' |
| +echo "== 7.2 reports before"; ../blocks/reports.sh |
| +box_inspect "$SOPHIE" >/dev/null; echo "box_inspect sophie: rev $(cat box-rev)" |
| +echo "== 7.2 console-report ($(date -u +%T.%3NZ))" |
| +../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/7-console-report.box.sh)" |
| +echo "== 7.2 publish-floor, home ($(date -u +%T.%3NZ))" |
| +../ssm.sh eu-west-1 "$SIGNER" '. /root/claim-trial/helpers.sh |
| +trap "tail -25 72.out" EXIT |
| +( |
| +exec > 72.out 2>&1 |
| +jq -n --slurpfile x floor.json "{floor:\$x[0]}" > floor-phase.json |
| +phase publish-floor floor-phase.json |
| +printf "%s\n" "{\"home_region\":\"ie\"}" > home-phase.json |
| +phase home home-phase.json |
| +) |
| +trap - EXIT |
| +refresh |
| +grep -oE "\"key\": \"control/[^\"]*\"" 72.out | sort | uniq -c |
| +echo REV=$REV home=$(jq -r .roster.home_region inspect.json)' |
| +echo "== 7.2 floor objects"; aws s3 ls s3://harness-deployment/control/ | grep moves-protocol |
| +echo "== 7.3 evidence key ($(date -u +%TZ))" |
| +box_inspect "$LEO" >/dev/null |
| +jq -er '.roster.boxes["kathleen-juggling"].enrollment' box-inspect.json > le |
| +box_put "$LEO" le /root/claim-trial/le |
| +../ssm.sh eu-west-1 "$LEO" "$(cat ../blocks/7-evidence-key.box.sh)" |
| +box_get "$LEO" /root/claim-trial/key-proof.json key-proof.json && echo "key-proof.json fetched ($(sha256sum key-proof.json | cut -c1-8))" |
| +aws s3 cp --quiet key-proof.json "$X/in/key-proof.json" |
| +../ssm.sh eu-west-1 "$SIGNER" ". /root/claim-trial/helpers.sh |
| +aws s3 cp --quiet $X/in/key-proof.json key-proof.json |
| +jq '{binding:.key_proof.binding,proof:.key_proof.proof}' key-proof.json > bind-phase.json |
| +phase bind-key bind-phase.json > 73.out 2>&1 |
| +refresh |
| +echo REV=\$REV |
| +jq -e '.roster.home_region == \"ie\" and .roster.publication == \"published\" and .roster.request_limit_ns == 120000000000' inspect.json" |
| +../ssm.sh eu-west-1 "$LEO" 'stat -c "%U %a %h" /etc/carlos/host-evidence.key' |
diff --git a/harness/scenarios/cohort1/blocks/7-runtime.box.sh b/harness/scenarios/cohort1/blocks/7-runtime.box.sh| new file mode 100755 |
| index 000000000..7a6d2a671 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7-runtime.box.sh |
| @@ -0,0 +1,17 @@ |
| +# Runbook 7.4, the sophie block (verbatim; HARNESS: bucket). |
| +set -euo pipefail |
| +umask 077 |
| +cd /root/claim-trial |
| +PUB=$(cat release.pub) |
| +CID=$(cat console-id) |
| +REV=$(cat rev) |
| +/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --console-id "$CID" --public-key "$PUB" --enable-console-runtime --dry-run |
| +/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket harness-deployment --console-id "$CID" --public-key "$PUB" --enable-console-runtime |
| +systemctl restart console.service |
| +systemctl is-active console.service |
| +/opt/carlos/carlos system claims console-report --bucket harness-deployment --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" --dry-run |
| +/opt/carlos/carlos system claims console-report --bucket harness-deployment --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" |
diff --git a/harness/scenarios/cohort1/blocks/74-run.sh b/harness/scenarios/cohort1/blocks/74-run.sh| new file mode 100755 |
| index 000000000..3e3a5a124 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/74-run.sh |
| @@ -0,0 +1,24 @@ |
| +#!/bin/bash |
| +# Runbook 7.4 from the operator: box_inspect sophie, the sophie block (console-config |
| +# --enable-console-runtime, restart, console-report at once), then, if refused, wait for |
| +# status/consoles/dev-console.json to move, box_inspect, and console-report again. |
| +set -euo pipefail |
| +. ~/cohort1/values.env; cd ~/cohort1/out |
| +. ../box-helpers.sh |
| +k=s3://harness-deployment/status/consoles/dev-console.json |
| +before=$(aws s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text 2>&1 || true) |
| +echo "status/consoles/dev-console.json before: $before" |
| +box_inspect "$SOPHIE" >/dev/null; echo "box_inspect sophie: rev $(cat box-rev)" |
| +echo "== 7.4 block ($(date -u +%T.%3NZ))" |
| +../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/7-runtime.box.sh)" | grep -v "ALERT: root AWS" | sed -E 's/"enrollment": "[0-9a-f]{32}"/"enrollment": "<redacted>"/' |
| +../ssm.sh eu-west-1 "$SOPHIE" 'echo "console restarted at $(systemctl show -p ActiveEnterTimestamp --value console.service), pid $(systemctl show -p MainPID --value console.service)"; journalctl -u console.service --since -2min --no-pager -o cat | grep -iE "refus|census|host report" | tail -3' |
| +echo "== waiting for the console's own report to move ($(date -u +%T.%3NZ))" |
| +for i in $(seq 1 60); do |
| + now=$(aws s3api head-object --bucket harness-deployment --key status/consoles/dev-console.json --query LastModified --output text 2>/dev/null || true) |
| + [ -n "$now" ] && [ "$now" != "$before" ] && break; sleep 1 |
| +done |
| +echo "status/consoles/dev-console.json now: $now ($(date -u +%T.%3NZ))" |
| +aws s3 cp --quiet $k - | python3 -c 'import json,sys; d=json.load(sys.stdin); print({k: d.get(k) for k in ("component","actor","release","release_tag","process_id","floor_guard_ready","host_claim_enabled","generated_at")})' |
| +box_inspect "$SOPHIE" >/dev/null; echo "box_inspect sophie: rev $(cat box-rev)" |
| +echo "== console-report again ($(date -u +%T.%3NZ))" |
| +../ssm.sh eu-west-1 "$SOPHIE" "$(cat ../blocks/7-console-report.box.sh)" | grep -v "ALERT: root AWS" | grep -E '^(Success|Failed)|"dry_run"|process_id|refus|key"' |
diff --git a/harness/scenarios/cohort1/blocks/7a-leo.box.sh b/harness/scenarios/cohort1/blocks/7a-leo.box.sh| new file mode 100755 |
| index 000000000..e054cba24 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7a-leo.box.sh |
| @@ -0,0 +1,27 @@ |
| +# Runbook 7a.2 part 2, the leo block (verbatim; HARNESS: bucket). The parameter name is kept. |
| +( |
| + set -euo pipefail |
| + umask 077 |
| + STAGE=/run/carlos-fence-stage |
| + SRC="$STAGE/credentials.json" |
| + install -d -m 0700 -o root -g root "$STAGE" |
| + test ! -e "$SRC" |
| + trap 'rm -f -- "$SRC"' EXIT |
| + aws --region eu-west-1 ssm get-parameter --name /carlos/cohort1/fence/carlos-dev-ie-b --with-decryption \ |
| + --query Parameter.Value --output text | install -m 0600 -o root -g root /dev/stdin "$SRC" |
| + stat -c '%U %a %h %s' "$SRC" |
| + python3 -c 'import json,sys,hashlib;print(hashlib.sha256(json.dumps(json.load(open(sys.argv[1])),sort_keys=True).encode()).hexdigest()[:8])' "$SRC" |
| + FENCE_ROLE=$(cat /root/claim-trial/fence-role-arn) |
| + /opt/carlos/carlos system claims fence-credentials --source "$SRC" \ |
| + --bucket harness-deployment --aws-region eu-west-1 --role "$FENCE_ROLE" --dry-run |
| + /opt/carlos/carlos system claims fence-credentials --source "$SRC" \ |
| + --bucket harness-deployment --aws-region eu-west-1 --role "$FENCE_ROLE" |
| +) |
| +echo "subshell exit=$?" |
| +if stat -- /run/carlos-fence-stage/credentials.json >/dev/null 2>&1; then |
| + echo "STAGING FILE STILL PRESENT: rm -f -- /run/carlos-fence-stage/credentials.json, then stop" |
| +else |
| + echo "staging file gone" |
| +fi |
| +rmdir -- /run/carlos-fence-stage |
| +stat -c '%U %a %h' /etc/carlos/fence-writer/credentials.json |
diff --git a/harness/scenarios/cohort1/blocks/7a-run.sh b/harness/scenarios/cohort1/blocks/7a-run.sh| new file mode 100755 |
| index 000000000..8805ae57e |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/7a-run.sh |
| @@ -0,0 +1,25 @@ |
| +#!/bin/bash |
| +# Runbook 7a.2 parts 1 to 4 from the operator. HARNESS: part 1 runs on the signer with its |
| +# instance role (the runbook's line carries --profile kass); fence-credentials.json was seeded |
| +# root 0600 on the signer by the host (H2: the fence writer's MinIO user stands for |
| +# carlos-dev-ie-b-fence-writer-root and its role); --key-id is accepted and ignored by the |
| +# stand-in (no KMS); part 5 (tofu, KMS key deletion) has no harness counterpart. |
| +set -euo pipefail |
| +. ~/cohort1/values.env; cd ~/cohort1/out |
| +. ../box-helpers.sh |
| +echo "== part 1 on the trusted host ($(date -u +%TZ))" |
| +../ssm.sh eu-west-1 "$SIGNER" 'set -e |
| +cd /root/claim-trial |
| +stat -c "%U %a %h" fence-credentials.json |
| +aws --region eu-west-1 ssm put-parameter --name /carlos/cohort1/fence/carlos-dev-ie-b \ |
| + --type SecureString --key-id alias/cohort1-fence-leo --value file:///root/claim-trial/fence-credentials.json \ |
| + --query Version --output text |
| +python3 -c "import json,sys,hashlib;print(hashlib.sha256(json.dumps(json.load(open(sys.argv[1])),sort_keys=True).encode()).hexdigest()[:8])" fence-credentials.json' |
| +printf '%s\n' arn:aws:iam::640012953448:role/carlos-dev-ie-b-fence-writer > fence-role-arn |
| +box_put "$LEO" fence-role-arn /root/claim-trial/fence-role-arn |
| +echo "== part 2 on leo ($(date -u +%TZ))" |
| +../ssm.sh eu-west-1 "$LEO" "$(cat ../blocks/7a-leo.box.sh)" | grep -v "ALERT: root AWS" |
| +echo "== part 3 and 4 ($(date -u +%TZ))" |
| +../ssm.sh eu-west-1 "$SIGNER" 'aws --region eu-west-1 ssm delete-parameter --name /carlos/cohort1/fence/carlos-dev-ie-b && echo deleted |
| +aws --region eu-west-1 ssm describe-parameters \ |
| + --parameter-filters Key=Name,Values=/carlos/cohort1/fence/carlos-dev-ie-b --query "Parameters[].Name" --output text; echo "(describe end)"' |
diff --git a/harness/scenarios/cohort1/blocks/p9.box.sh b/harness/scenarios/cohort1/blocks/p9.box.sh| new file mode 100755 |
| index 000000000..75187ce57 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/p9.box.sh |
| @@ -0,0 +1,17 @@ |
| +# Runbook P9's box command (sophie's console mail is off), verbatim. |
| +names="CARLOS_CONSOLE_MAIL_FROM CARLOS_MAIL_REGION CARLOS_MAIL_PLATFORM_SUFFIX CARLOS_MAIL_DMARC_RUA CARLOS_MAIL_SNS_TOPIC_ARN CARLOS_MAIL_OPERATORS CARLOS_MAIL_LEGACY_SWEEP" |
| +u=console.service |
| +pid=$(systemctl show -p MainPID --value $u) |
| +files=$( { echo /etc/carlos/host.env; systemctl show -p FragmentPath --value $u; systemctl show -p DropInPaths --value $u | tr " " "\n"; systemctl show -p EnvironmentFiles --value $u | awk "{print \$1}"; } | grep . | sort -u | tr "\n" " ") |
| +hits=0 |
| +for n in $names; do |
| + if [ "$pid" -gt 0 ] && tr "\0" "\n" < /proc/$pid/environ | grep -q "^$n="; then echo "SET $n in the running console (pid $pid)"; hits=$((hits+1)); fi |
| + if systemctl show -p Environment --value $u | tr " " "\n" | grep -q "^$n="; then echo "SET $n in $u Environment="; hits=$((hits+1)); fi |
| + for f in $files; do |
| + [ -f "$f" ] || continue |
| + if grep -qE "(^|[[:space:]\"=])$n=" "$f"; then echo "SET $n in $f"; hits=$((hits+1)); fi |
| + done |
| +done |
| +echo "checked: the running console (pid $pid), $u Environment=, and $(echo $files | wc -w) files: $files" |
| +for f in $files; do [ -e "$f" ] || echo "missing: $f"; done |
| +if [ $hits -eq 0 ]; then echo "console mail: OFF (none of the 7 names is set)"; else echo "console mail: $hits setting(s) found: STOP"; fi |
diff --git a/harness/scenarios/cohort1/blocks/reports.sh b/harness/scenarios/cohort1/blocks/reports.sh| new file mode 100755 |
| index 000000000..1385bd9e2 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/reports.sh |
| @@ -0,0 +1,6 @@ |
| +#!/bin/bash |
| +# Runbook 6.3's wait check (read-only), from the operator. HARNESS: profile, bucket, box names. |
| +for b in kathleen-juggling ada-letterpress; do |
| + aws --profile harness --region eu-west-1 s3 cp "s3://harness-deployment/status/boxes/$b.json" - \ |
| + | python3 -c 'import json,sys;d=json.load(sys.stdin);m=d.get("metrics") or {};print(sys.argv[1],d["generated_at"],m.get("release"),m.get("binary_sha256","")[:8],m.get("floor_guard_ready"),m.get("floor_sha256","")[:8],len(m.get("console_processes") or []))' "$b" |
| +done |
diff --git a/harness/snapshot.sh b/harness/snapshot.sh| new file mode 100755 |
| index 000000000..e5a3a411f |
| --- /dev/null |
| +++ b/harness/snapshot.sh |
| @@ -0,0 +1,21 @@ |
| +#!/bin/bash |
| +# Consistent harness snapshot / restore: every container stopped, plus sophie's |
| +# ext4 console volume (H11), which container snapshots do not include. |
| +# snapshot.sh create <name> | restore <name> |
| +set -euo pipefail |
| +op=$1 name=$2 |
| +ALL="infra leo sophie dennis signer" |
| +incus stop $ALL |
| +case $op in |
| + create) for c in $ALL; do incus snapshot create $c "$name"; done |
| + incus storage volume snapshot create hdir sophie-console "$name" >/dev/null ;; |
| + restore) for c in $ALL; do incus snapshot restore $c "$name"; done |
| + if incus storage volume snapshot show hdir sophie-console "$name" >/dev/null 2>&1; then |
| + incus storage volume snapshot restore hdir sophie-console "$name" |
| + fi ;; |
| + *) echo "usage: snapshot.sh create|restore <name>"; exit 2 ;; |
| +esac |
| +incus start infra; sleep 3; incus start leo sophie dennis signer |
| +for c in $ALL; do for i in $(seq 1 60); do incus exec $c -- systemctl is-system-running --wait >/dev/null 2>&1 && break; sleep 2; done; done |
| +for c in $ALL; do echo "$c: $(incus exec $c -- systemctl is-system-running) failed=[$(incus exec $c -- systemctl --failed --no-legend --plain)] snapshots=$(incus snapshot list $c -f csv -c n | paste -sd,)"; done |
| +echo "sophie-console volume snapshots: $(incus storage volume snapshot list hdir sophie-console -f csv -c n 2>/dev/null | paste -sd,)" |