amadan
harness: window run 1 on cc90d542: H13-H15, pre-window seed, G2 and G3 PASS, G4 FAIL (armed-box candidate version refusal)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 files changed,
+620
−1
harness/RUNLOG.md+116 −0harness/apps/wakecheck/main.go+72 −0harness/boxes/vendor-shims.sh+54 −0harness/hrun.sh+15 −0harness/release/release-window.sh+42 −0harness/scenarios/cohort1/blocks/b1.sh+40 −0harness/scenarios/cohort1/blocks/b2.box.sh+48 −0harness/scenarios/cohort1/blocks/b5.sh+34 −0harness/scenarios/cohort1/blocks/g4-roll.sh+19 −0harness/scenarios/cohort1/blocks/pw-seed.sh+66 −0harness/scenarios/cohort1/blocks/w-identity.sh+26 −0harness/scenarios/cohort1/blocks/w-open-signer.sh+39 −0harness/scenarios/cohort1/values.env+5 −0harness/scenarios/sync.sh+8 −0harness/snapshot.sh+6 −1harness/stand-ins/certs.sh+30 −0
diff --git a/harness/RUNLOG.md b/harness/RUNLOG.md| index 9c2036bc3..bc878b0aa 100644 |
| --- a/harness/RUNLOG.md |
| +++ b/harness/RUNLOG.md |
| @@ -753,3 +753,119 @@ If-Match on DELETE (step 3), so in the harness each of these deletes uncondition |
| Snapshots on every container: `base`, `boot`, `pre62`, `s7` (`s7` also on `hdir/sophie-console`). |
| The host is left at `s7`. No `pre-window` snapshot was taken: the window (8.1 onward) is not in this |
| phase's scope; take it with `harness/snapshot.sh create pre-window` right before 8.1. |
| + |
| +# Window (Option B), 2026-10-20 morning: gates G0 to G11 |
| + |
| +Runbook: `claim-dev-cohort1-runbook` @ `a06f17ea`, "Window package" (top) and steps 8 to 10. |
| +Two runs: **run 1** on `claim-empty-cohort-done` @ `cc90d542` (stopped at G4, below) and **run 2** on |
| +the fixed tip `d940c09f` (next section). Paul's gos are pre-authorised inside the harness only. |
| +Times are the host's UTC. |
| + |
| +## Harness repairs before the window (from snapshot `pre-window`) |
| + |
| +- **`pre-window`** taken at 07:05Z from a full `s7` restore. Retaken at 07:24Z (below). |
| +- **snapshot.sh never restored the console volume.** Its test was `incus storage volume snapshot show |
| + hdir sophie-console <name>`, which incus 6.0.4 refuses ("Invalid number of arguments"), so every |
| + `restore` since step 10 restored the containers but left sophie's console DB (`hdir/sophie-console`) |
| + live. Found when a re-seed after a restore met `apps create: that name is taken` (the console DB still |
| + held the apps; the bucket did not). Fixed (`show hdir sophie-console/<name>`, and a WARNING when absent), |
| + then `s7` restored in full and `pre-window` deleted and retaken (07:24Z), volume included. |
| +- **H14, census-neutral shim placement** (`harness/boxes/vendor-shims.sh`). The window's migration |
| + census walks `/etc/systemd/system` and refuses any regular file or non-vendor link it cannot class. |
| + Dev's leo has none of the harness's entries there (`internal/host/testdata/leo-step8/leo-tree-snapshot.tsv` |
| + compared with the harness leo's tree): the zram no-op unit, the asset-tag unit (H12) and its |
| + `sysinit.target.wants` link, and the `amazon-ssm-agent` `/dev/null` mask. Moved to |
| + `/usr/local/lib/systemd/system` (a unit path outside the census roots): same effect, checked |
| + (asset tag correct after restart, agent `inactive`, zram unit from `/usr/local/lib`). |
| +- **H15, read-only cgroupfs for tenant units.** `ProtectControlGroups=yes` is a no-op in the harness's |
| + privileged LXC containers (measured: a `systemd-run -p ProtectControlGroups=yes` unit sees |
| + `/sys/fs/cgroup rw` and `W_OK`). So instance-control's isolation probe failed `cgroup-ro` and |
| + `cgroup-ro-mount`, `layout: no intent recorded; nested by default is not in force`, and wakecheck was |
| + first minted **flat** on leo (`/data/wakecheck....db`), unlike dev. A `BindReadOnlyPaths=/sys/fs/cgroup` |
| + drop-in on the probe and the five tenant units (`/usr/local/lib/systemd/system/<unit>.d/`) gives them |
| + the read-only view (`ro`, `not writable`). After it: `layout: no intent recorded -> nested by default`, |
| + and every host below is nested. The probe's three `unverified` lines (imds-v6, private-v6, io-weight) |
| + remain, as they do not block. |
| +- **H13, certificate stand-in** (`harness/stand-ins/certs.sh`): no route to Let's Encrypt, so each host |
| + gets a certificate under a harness test CA (CA key on host tmpfs), published at `control/certs/<host>` |
| + as autocert's cache entry; the edges' mirror serves it; curl uses `--cacert /root/harness-ca.pem`. |
| +- **The tenant build** (`harness/apps/wakecheck`): dev's `wc4` artefact lives only in the dev bucket, so a |
| + stand-in with the same contract (`--socket`/`--db`; one SQLite row per request; `build= host= hits=`) |
| + is shipped as version `wc4`. sha256 `4a79c656...`. |
| + |
| +## Pre-window state: SEEDED BY PLATFORM COMMANDS |
| + |
| +`harness/scenarios/cohort1/blocks/pw-seed.sh` (07:26-07:30Z), from `pre-window` + H14/H15, every write |
| +through the ordinary doors with the console claims-off: |
| +- **sophie's pause** (dev's 2026-10-19 state): the saved copies in `/root/claim-trial/paused-20261019/` |
| + (`cp -p`, `cmp`-equal: `5df17bac...` 73 bytes, `815226c3...` 56 bytes), then the runbook's pause recipe |
| + verbatim: `active`, `CARLOS_HOST_CLAIM=` (empty), healthz 200. Its last line races (Runbook corrections, C1). |
| +- **wakecheck** (pool-less): `apps create`, `ship wc4`, `promote wc4 edge`, store-mode `instances enable` |
| + (domain `dev.harness.test`, channel `edge`), signed-in `instances create`. Minted on leo and sophie; |
| + woken on leo (`hits=2,3,4`), so leo holds its nested leftovers as dev's leo does. |
| +- **step 4.2 and 4.3** on leo, box commands verbatim: `CARLOS_POOLS=cohort1`, edge restart |
| + (`instances loop armed ... pools [cohort1]`), `carlos remove ... wakecheck.dev.harness.test` -> `removed`. |
| +- **claimtrial** (step 4.4: pool `cohort1`, store-mode enable, signed-in create): minted on leo only, |
| + woken (`hits=2,3,4`). **rollcheck-0261** (`--kind static`, pool-less): sophie and dennis. |
| + **moverehearsal** (the offline-move rehearsal app; created claims-off with `--region uk`, so it sits on |
| + dennis as the rehearsal's app does after its move; no move was run). |
| +- Result: four records, `apps/bab/{claimtrial,moverehearsal,rollcheck-0261,wakecheck}/instances/*.json`, |
| + each without `acq`; leo `claimtrial` (pool cohort1) plus wakecheck's leftovers; sophie console, |
| + rollcheck-0261, wakecheck; dennis moverehearsal, rollcheck-0261. |
| + |
| +## Run 1: cc90d542 |
| + |
| +**Build (G0 here, G1's "tag and publish").** `git archive cc90d542` + the harness patch (6b905c11's |
| +code diff, seven files, sha256 `b2f1ad8f...`), `-tags harness`, `GOFLAGS=-buildvcs=false`, |
| +`version.Build=v0.28.6`. Harness tag **v0.28.6** plays dev's v0.28.5 (harness v0.28.5 is step 7's |
| +96db0efe and plays dev's v0.28.4). carlos `9afa2deb...`, console `4e686875...`. Tagged tests green |
| +(`./internal/s3compat ./internal/publicrelease`), and `-run 'Backfill|Census|Step8|EmptyLeo|Ruling6'` |
| +over claimactivation, hostops, hostbox: ok. No CI, merge or copy review in the harness (G1 n/a). |
| +`harness/release/release-window.sh` (07:30Z): signed sums (`4d9810d1...`, verified), operator CLI |
| +v0.28.6, then **the console ship and promote for v0.28.6** (`shipped console cc90d542`, |
| +`promoted console cc90d542 -> canary/console`): publish-identity needs it (Runbook corrections, C2). |
| + |
| +**G2, B5 (07:31Z): PASS** (with the known blocker). 1: `0` and no `PRE /` (vacuous: MinIO refuses the |
| +`apps/bab//` prefix with XMinioInvalidObjectName, so the `//` segment cannot exist or be listed here). |
| +2: the four records, each `MISSING` (the known blocker B1 resolves). 3: `1`, `null` (MinIO omits |
| +KeyCount for an empty listing; S3 says `0`), `0`. 4: `Success 9 ['cohort1', 'console']`. 5: `0 0` on |
| +sophie and dennis. 6: no counterpart. 7: `200`, `c1ae2900...c412`, `CARLOS_HOST_CLAIM=`, `0`. |
| + |
| +**G3, signing session 6 (07:31-07:32Z): PASS.** `w-open-signer.sh`: run-id |
| +`cohort1-20261020T073158Z`, signer CLI `carlos v0.28.6` (`OK` against the signed sums), helpers and |
| +release.pub `OK`, no swap, `/run` tmpfs, key `root 600 1 45`. `w-identity.sh`: one create-only write |
| +(`before_etag ""`) of `control/system/router/releases/v0.28.6.json`; router `9afa2deb...`, console |
| +`4e686875...`; identity file = published object. The P-B2 grants: **not exercised** (the signer's MinIO |
| +user is `readwrite`; only a hand IAM review, as step 11 did, would show a missing grant). |
| + |
| +**G4, leo on v0.28.6: FAIL** (07:32:50Z promote, read 07:42:52Z). D10's harness equivalent is runbook |
| +step 1 item 2's new road for `stable` only (`carlos system update --channel stable --version v0.28.6 |
| +--file ... --yes`), which moves leo and dennis; sophie stays on canary at v0.28.5. |
| +``` |
| +kathleen-juggling stable v0.28.5 behind -> v0.28.6 refused v0.28.6: refused: candidate reports |
| + "2026/10/20 07:33:00 ALERT: root AWS file, profile, endpoint and trust-store environment overrides ignored\ncarlos v0.28.6", not v0.28.6 |
| +noor-marquetry stable v0.28.6 current (dennis: unarmed, rolled) |
| +leo journal 07:33:00: system apply: stable v0.28.6: refused: refused: candidate reports "...ALERT...\ncarlos v0.28.6", not v0.28.6 |
| +``` |
| +Failure action applied: hold the window (leo claims-off, v0.28.5, serving). Snapshot |
| +**`g4-fail-cc90d542`** (07:44Z). |
| + |
| +**Root cause (code, present on main and on both window tips).** On an armed box the updater runs the |
| +candidate's `version` through the guard: `systemApplier.guardRun` -> `hostfloor.Disk.Run` |
| +(`internal/hostfloor/image_linux.go:72`, `cmd.CombinedOutput()`, environment inherited). The updater |
| +is root, so it has already run `rootaws.ScrubEnvironment`, which **sets** `AWS_CONFIG_FILE` and |
| +`AWS_SHARED_CREDENTIALS_FILE` to `/dev/null/carlos-root-aws` in its own environment |
| +(`internal/rootaws/environment.go:72-73`). The child inherits them, its own scrub counts any set |
| +variable in the list as an override (`environment.go:63-66`) and logs the ALERT to stderr |
| +(`environment.go:75-76`); CombinedOutput folds that line into the output, and |
| +`carlosVersionOf` (`internal/host/systemapply.go:2049`, a `carlos ` prefix on the whole trimmed |
| +output) fails, so `stageCandidate` refuses at `systemapply.go:2107-2108`. Reproduced by hand on leo: |
| +`AWS_CONFIG_FILE=/dev/null/carlos-root-aws /opt/carlos/carlos version` prints the ALERT then |
| +`carlos v0.28.5`; without the variable, only the version. The unarmed path (`ops.run`, |
| +`systemapply.go:184-197`) uses `Output()`, stdout only, which is why dennis rolled. **Every armed box |
| +refuses every router update this way**: on dev, leo and sophie (armed since 2026-10-19) cannot take |
| +v0.28.5 through `carlos system update`. Dev never met it because no armed box has been rolled yet. The |
| +refusing code runs in the **old** updater, so a fix in the new release only helps if it is in the |
| +candidate's side (the child must not print the ALERT when the two variables already hold the disabled |
| +path); a fix in `Disk.Run` (stdout only) reaches boxes only after a roll. `carlos.prev.allowed` was |
| +written at 07:34Z by the refused attempt. |
diff --git a/harness/apps/wakecheck/main.go b/harness/apps/wakecheck/main.go| new file mode 100644 |
| index 000000000..114af510b |
| --- /dev/null |
| +++ b/harness/apps/wakecheck/main.go |
| @@ -0,0 +1,72 @@ |
| +// wakecheck is the harness's stand-in for dev's wakecheck build wc4, whose |
| +// bytes live only in the dev deployment bucket (the harness never reads it). |
| +// Same contract as testdata/echoapp (-socket, -addr or LISTEN_FDS; -db), plus |
| +// what the runbook's acceptance reads: every request to anything but /healthz |
| +// writes one SQLite row in the instance database and replies |
| +// "build=<BUILD> host=<hostname> hits=<rows>". |
| +package main |
| + |
| +import ( |
| + "database/sql" |
| + "flag" |
| + "fmt" |
| + "net" |
| + "net/http" |
| + "os" |
| + "strconv" |
| + |
| + _ "modernc.org/sqlite" |
| +) |
| + |
| +func main() { |
| + socket := flag.String("socket", "", "unix socket to listen on") |
| + addr := flag.String("addr", "", "TCP host:port to listen on") |
| + dbPath := flag.String("db", "", "instance database path") |
| + flag.Parse() |
| + if *dbPath == "" { |
| + *dbPath = "wakecheck.db" |
| + } |
| + db, err := sql.Open("sqlite", *dbPath) |
| + if err == nil { |
| + _, err = db.Exec(`PRAGMA journal_mode=WAL; CREATE TABLE IF NOT EXISTS hits (id INTEGER PRIMARY KEY, at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')))`) |
| + } |
| + if err != nil { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| + host, _ := os.Hostname() |
| + mux := http.NewServeMux() |
| + mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { fmt.Fprint(w, "ok") }) |
| + mux.HandleFunc("/", func(w http.ResponseWriter, _ *http.Request) { |
| + var n int |
| + if _, err := db.Exec(`INSERT INTO hits DEFAULT VALUES`); err == nil { |
| + err = db.QueryRow(`SELECT count(*) FROM hits`).Scan(&n) |
| + } |
| + fmt.Fprintf(w, "build=%s host=%s hits=%d\n", os.Getenv("BUILD"), host, n) |
| + }) |
| + ln, err := listen(*socket, *addr) |
| + if err != nil { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| + if err := http.Serve(ln, mux); err != nil { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| +} |
| + |
| +func listen(socket, addr string) (net.Listener, error) { |
| + if n, _ := strconv.Atoi(os.Getenv("LISTEN_FDS")); n >= 1 { |
| + if pid := os.Getenv("LISTEN_PID"); pid == "" || pid == strconv.Itoa(os.Getpid()) { |
| + return net.FileListener(os.NewFile(3, "listener")) |
| + } |
| + } |
| + if socket != "" { |
| + _ = os.Remove(socket) |
| + return net.Listen("unix", socket) |
| + } |
| + if addr == "" { |
| + addr = "127.0.0.1:8080" |
| + } |
| + return net.Listen("tcp", addr) |
| +} |
diff --git a/harness/boxes/vendor-shims.sh b/harness/boxes/vendor-shims.sh| new file mode 100644 |
| index 000000000..b4c3c5a39 |
| --- /dev/null |
| +++ b/harness/boxes/vendor-shims.sh |
| @@ -0,0 +1,54 @@ |
| +#!/bin/bash |
| +# H14 and H15 (on host, root), applied from snapshot pre-window before the seed. |
| +# |
| +# H14, census-neutral shim placement. The window's migration census walks |
| +# /etc/systemd/system and refuses any regular file or non-vendor link it |
| +# cannot class (internal/hostbox/migration_census_linux.go CheckMigrationRoots). |
| +# Dev's leo has none of the harness's three /etc/systemd/system entries |
| +# (internal/host/testdata/leo-step8/leo-tree-snapshot.tsv): the zram no-op |
| +# unit (step 5), the asset-tag unit and its sysinit.target.wants link (H12), |
| +# and the amazon-ssm-agent /dev/null mask (step 5). They move to |
| +# /usr/local/lib/systemd/system, a systemd unit path outside the census roots, |
| +# with the same effect: the zram unit there overrides /usr/lib's; the |
| +# asset-tag unit is wanted through a .wants link there; the SSM agent is kept |
| +# from starting by a drop-in condition instead of a mask. |
| +# |
| +# H15, read-only cgroupfs for tenant units. ProtectControlGroups=yes is a |
| +# no-op inside the harness's privileged LXC containers (measured: a |
| +# `systemd-run -p ProtectControlGroups=yes` unit still sees /sys/fs/cgroup rw), |
| +# so instance-control's isolation probe fails `cgroup-ro` and `cgroup-ro-mount` |
| +# and the box never puts "nested by default" in force; dev's leo runs nested. |
| +# A BindReadOnlyPaths=/sys/fs/cgroup drop-in on the probe and every tenant unit |
| +# gives those units the read-only cgroupfs ProtectControlGroups gives them on |
| +# EC2. Drop-ins live in /usr/local/lib/systemd/system/<unit>.d/. |
| +set -euo pipefail |
| +L=/usr/local/lib/systemd/system |
| +for b in leo sophie dennis; do |
| + incus exec $b -- bash -s <<EOF |
| +set -euo pipefail |
| +install -d -m 0755 $L $L/sysinit.target.wants $L/amazon-ssm-agent.service.d |
| +# zram no-op: move, do not copy |
| +if [ -f /etc/systemd/system/systemd-zram-setup@.service ] && [ ! -L /etc/systemd/system/systemd-zram-setup@.service ]; then |
| + mv -T /etc/systemd/system/systemd-zram-setup@.service $L/systemd-zram-setup@.service |
| +fi |
| +# asset tag unit: disable from /etc, move, want it from /usr/local |
| +if [ -f /etc/systemd/system/harness-dmi-instance.service ]; then |
| + systemctl disable harness-dmi-instance.service 2>/dev/null || true |
| + rm -f /etc/systemd/system/sysinit.target.wants/harness-dmi-instance.service |
| + mv -T /etc/systemd/system/harness-dmi-instance.service $L/harness-dmi-instance.service |
| + ln -sfn $L/harness-dmi-instance.service $L/sysinit.target.wants/harness-dmi-instance.service |
| +fi |
| +rmdir /etc/systemd/system/sysinit.target.wants 2>/dev/null || true |
| +# SSM agent: condition drop-in instead of the /etc mask |
| +printf '[Unit]\n# harness H14: the box agent never runs (it would register with real SSM)\nConditionPathExists=/nonexistent/harness-no-ssm-agent\n' > $L/amazon-ssm-agent.service.d/zz-harness-off.conf |
| +if [ "\$(readlink /etc/systemd/system/amazon-ssm-agent.service 2>/dev/null)" = /dev/null ]; then rm /etc/systemd/system/amazon-ssm-agent.service; fi |
| +# H15 |
| +for u in carlos-instance-probe.service carlos-instance-gen@.service carlos-instance@.service carlos-instance-seed@.service carlos-instance-cmd@.service carlos-instance-sidecar@.service; do |
| + install -d -m 0755 $L/\$u.d |
| + printf '[Service]\n# harness H15: ProtectControlGroups is a no-op in LXC; a read-only bind gives the same view\nBindReadOnlyPaths=/sys/fs/cgroup\n' > $L/\$u.d/zz-harness-cgroup-ro.conf |
| +done |
| +systemctl daemon-reload |
| +echo "$b: \$(systemctl is-enabled harness-dmi-instance.service 2>&1) dmi=\$(systemctl is-active harness-dmi-instance.service) ssm=\$(systemctl show -p LoadState -p ActiveState --value amazon-ssm-agent.service | paste -sd/) zram=\$(systemctl show -p FragmentPath --value systemd-zram-setup@zram0.service)" |
| +systemd-run -q --wait -P -p BindReadOnlyPaths=/sys/fs/cgroup sh -c 'grep " /sys/fs/cgroup " /proc/self/mountinfo | cut -d" " -f5,6; test -w /sys/fs/cgroup/cgroup.procs && echo W_OK || echo "not writable"' |
| +EOF |
| +done |
diff --git a/harness/hrun.sh b/harness/hrun.sh| new file mode 100755 |
| index 000000000..2ee820250 |
| --- /dev/null |
| +++ b/harness/hrun.sh |
| @@ -0,0 +1,15 @@ |
| +#!/bin/bash |
| +# hrun.sh FILE [ARGS...]: copy a harness script to the host as /root/harness/<basename> |
| +# (root 0700) and run it there as root with ARGS, through hssm.py (sandbox off). |
| +# HRUN_TIMEOUT sets the SSM timeout (default 1800 s). |
| +set -euo pipefail |
| +f=$1; shift |
| +n=$(basename "$f") |
| +d=$(dirname "$0") |
| +{ |
| + printf 'install -d -m 0700 /root/harness\ncat > /root/harness/%s <<'"'"'__HRUN_EOF__'"'"'\n' "$n" |
| + cat "$f" |
| + printf '__HRUN_EOF__\nchmod 0700 /root/harness/%s\nbash /root/harness/%s' "$n" "$n" |
| + for a in "$@"; do printf ' %q' "$a"; done |
| + printf '\n' |
| +} | python3 "$d/hssm.py" --timeout "${HRUN_TIMEOUT:-1800}" - |
diff --git a/harness/release/release-window.sh b/harness/release/release-window.sh| new file mode 100644 |
| index 000000000..c30e6b24a |
| --- /dev/null |
| +++ b/harness/release/release-window.sh |
| @@ -0,0 +1,42 @@ |
| +#!/bin/bash |
| +# The window release (on host, root): harness tag v0.28.6 = claim-empty-cohort-done @ cc90d542 |
| +# plus the harness patch (6b905c11's code diff), built on DOCS-T1 with -tags harness and |
| +# version.Build=v0.28.6. It plays the runbook's "tag and publish v0.28.5" (G1): the harness's |
| +# v0.28.5 is already step 7's release of 96db0efe, which plays dev's v0.28.4. |
| +# Same recipe as release-setup.sh, minus what step 7 already did once: the test sums key |
| +# (reused from /run/harness/keys/sums.key), its public half on every box, and the release |
| +# key in the fake Parameter Store. |
| +# SUMS="<carlos sha> <console sha>" CV=<console ship version> release-window.sh |
| +set -euo pipefail |
| +TAG=v0.28.6; V=${TAG#v} |
| +R=/root/harness/release/$TAG; K=/run/harness/keys |
| +test -s $K/sums.key |
| +mkdir -p $R |
| +aws s3 cp --quiet s3://carlos-harness-640012953448/artefacts/$TAG/carlos-linux-arm64 $R/carlos-linux-arm64 |
| +aws s3 cp --quiet s3://carlos-harness-640012953448/artefacts/$TAG/console-linux-arm64 $R/console-linux-arm64 |
| +set -- $SUMS |
| +[ "$(sha256sum < $R/carlos-linux-arm64 | cut -d' ' -f1)" = "$1" ] && [ "$(sha256sum < $R/console-linux-arm64 | cut -d' ' -f1)" = "$2" ] || { echo "artefact sha256 mismatch"; exit 1; } |
| +chmod 755 $R/carlos-linux-arm64 $R/console-linux-arm64 |
| +openssl pkey -in $K/sums.key -pubout -outform DER | tail -c 32 | base64 -w0 > $R/sums.pub |
| +cmp $R/sums.pub /etc/carlos/harness-sums-pubkey |
| +for a in amd64 arm64; do printf 'harness placeholder: no deb is built for %s %s\n' "$TAG" "$a" > $R/carlos_${V}_$a.deb; done |
| +(cd $R && sha256sum carlos-linux-arm64 carlos_${V}_amd64.deb carlos_${V}_arm64.deb > SHA256SUMS.txt) |
| +openssl pkeyutl -sign -inkey $K/sums.key -rawin -in $R/SHA256SUMS.txt | base64 -w0 > $R/SHA256SUMS.txt.sig |
| +echo >> $R/SHA256SUMS.txt.sig |
| +install -m 0755 $R/carlos-linux-arm64 /usr/local/bin/carlos |
| +O=/home/operator/release-$TAG; install -d -o operator -g operator -m 0755 $O |
| +for f in carlos-linux-arm64 console-linux-arm64 SHA256SUMS.txt SHA256SUMS.txt.sig carlos_${V}_amd64.deb carlos_${V}_arm64.deb sums.pub; do install -o operator -g operator -m 0644 $R/$f $O/$f; done |
| +chmod 755 $O/carlos-linux-arm64 $O/console-linux-arm64 |
| +echo "$CV" > $O/CV; chown operator:operator $O/CV |
| +echo "== release $TAG"; (cd $R && cat SHA256SUMS.txt && sha256sum SHA256SUMS.txt SHA256SUMS.txt.sig console-linux-arm64) |
| +P=$(mktemp -d); openssl pkey -in $K/sums.key -pubout > $P/p.pem; base64 -d $R/SHA256SUMS.txt.sig > $P/s.bin |
| +echo "verify sums: $(openssl pkeyutl -verify -pubin -inkey $P/p.pem -rawin -in $R/SHA256SUMS.txt -sigfile $P/s.bin)"; rm -rf $P |
| +echo "operator CLI: $(/usr/local/bin/carlos version)"; echo "harness info: $(CARLOS_HARNESS_INFO=1 /usr/local/bin/carlos | tr '\n' ' ')" |
| +# 5.0a item 4 for the window release (see RUNLOG "Runbook corrections"): ship and promote the |
| +# v0.28.6-stamped console to canary/console, as step7-operator.sh did for v0.28.5. |
| +runuser -l operator -c 'set -euo pipefail; cd ~/release-v0.28.6; CARLOS=/usr/local/bin/carlos; CONSOLE=~/release-v0.28.6/console-linux-arm64; CV=$(cat CV) |
| +export AWS_PROFILE=harness AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=harness-deployment CARLOS_RELEASE_PUBKEY="$(cat ~/release.pub)" |
| +echo "== 1a/1e"; "$CARLOS" channels --app console --account-id bab; "$CARLOS" channels digests --account-id bab |
| +echo "== 2"; sha256sum "$CONSOLE"; grep -c "$(sha256sum "$CONSOLE" | cut -d" " -f1)" SHA256SUMS.txt || true |
| +echo "== 4"; CARLOS_BIN=$CARLOS ~/infra/scripts/ship-app.sh harness harness-deployment console "$CV" canary/console binary "$CONSOLE" |
| +"$CARLOS" channels --app console --account-id bab' |
diff --git a/harness/scenarios/cohort1/blocks/b1.sh b/harness/scenarios/cohort1/blocks/b1.sh| new file mode 100755 |
| index 000000000..6a4fb4f07 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/b1.sh |
| @@ -0,0 +1,40 @@ |
| +#!/bin/bash |
| +# Block B1 (gate G5), operator shell, signed in to the harness console. Runbook text with the harness |
| +# values: DEV -> http://127.0.0.1:8090, domain dev.harness.test, REHEARSAL_APP/HOST = moverehearsal |
| +# (seeded pre-window: created claims-off with --region uk, so it sits on dennis as the rehearsal's app |
| +# does after its move); R/B/S as B5; `leo` is ssm.sh to leo. |
| +set -euo pipefail |
| +. ~/cohort1/values.env |
| +R="--profile harness --region eu-west-1" |
| +B=harness-deployment |
| +S=~/cohort1/ssm.sh |
| +leo() { "$S" eu-west-1 "$LEO" "$1"; } |
| +DEV=http://127.0.0.1:8090 |
| +REHEARSAL_APP='moverehearsal' # set at the rehearsal |
| +REHEARSAL_HOST='moverehearsal.dev.harness.test' # set at the rehearsal |
| +: "${REHEARSAL_APP:?set it from the rehearsal record}" |
| +: "${REHEARSAL_HOST:?set it from the rehearsal record}" |
| +echo "== B1 deletes $(date -u +%FT%TZ)" |
| +carlos instances delete --app claimtrial --account bab --host claimtrial.dev.harness.test --console $DEV --no-prompt |
| +carlos instances delete --app wakecheck --account bab --host wakecheck.dev.harness.test --console $DEV --no-prompt |
| +carlos instances delete --app rollcheck-0261 --account bab --host rollcheck-0261.dev.harness.test --console $DEV --no-prompt |
| +carlos instances delete --app "$REHEARSAL_APP" --account bab --host "$REHEARSAL_HOST" --console $DEV --no-prompt |
| +date -u +%FT%TZ |
| +echo "-- bucket"; aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -cE '/(instances|sidecars)/[^/]+\.json$' || true |
| +sleep 45 |
| +echo "-- registries $(date -u +%FT%TZ)" |
| +leo 'python3 -c " |
| +import sqlite3 |
| +c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| +ts=[r[0] for r in c.execute(\"select name from sqlite_master where type=\x27table\x27 and name not like \x27sqlite_%\x27 order by name\")] |
| +n=0 |
| +for t in ts: |
| + k=c.execute(\"select count(*) from %s\" % t).fetchone()[0] |
| + n+=k |
| + print(t,k) |
| +print(\"total\",n) |
| +for r in c.execute(\"select host,app,pool from routes order by host\"): print(r)" |
| +systemctl list-units --all --plain --no-legend "carlos-instance-gen@*" "carlos-instance@*" |
| +journalctl -u carlos-edge.service --since -30min --no-pager | grep "removed retired"' |
| +"$S" eu-west-1 "$SOPHIE" 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); [print(r) for r in c.execute(\"select host,app,pool from routes order by host\")]"' |
| +"$S" eu-west-2 "$DENNIS" 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes\").fetchone()[0])"' |
diff --git a/harness/scenarios/cohort1/blocks/b2.box.sh b/harness/scenarios/cohort1/blocks/b2.box.sh| new file mode 100644 |
| index 000000000..3bdfbb6cc |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/b2.box.sh |
| @@ -0,0 +1,48 @@ |
| +# Block B2, the move (runbook verbatim but for HOSTS' domain and REHEARSAL_HOST), then its check block. |
| +# Run in an attended root shell on each box (here: incus exec <box> -- bash). |
| +set -eu |
| +umask 077 |
| +STAMP=$(date -u +%Y%m%dT%H%MZ) |
| +ASIDE="/var/lib/carlos-aside-$STAMP" |
| +REHEARSAL_HOST='moverehearsal.dev.harness.test' # set at the rehearsal: the same value as B1's |
| +: "${REHEARSAL_HOST:?set it from the rehearsal record}" |
| +HOSTS=(claimtrial.dev.harness.test wakecheck.dev.harness.test rollcheck-0261.dev.harness.test "$REHEARSAL_HOST") |
| +test "$(stat -c %d /data)" = "$(stat -c %d /var/lib)" |
| +test "$(stat -c %d /opt/carlos/live)" = "$(stat -c %d /var/lib)" |
| +mkdir -m 0700 -- "$ASIDE" |
| +test "$(stat -c '%a %U:%G' "$ASIDE")" = '700 root:root' |
| +aside() { |
| + if [ -L "$1" ] || [ -e "$1" ]; then |
| + mv -T -- "$1" "$ASIDE/$H/$2" |
| + echo "moved $1" |
| + else |
| + echo "absent $1" |
| + fi |
| +} |
| +for H in "${HOSTS[@]}"; do |
| + test "$(python3 -c "import sqlite3,sys; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True); print(c.execute('select count(*) from routes where host=?',(sys.argv[1],)).fetchone()[0])" "$H")" = 0 |
| + test -z "$(systemctl list-units --all --plain --no-legend "carlos-instance-gen@$H:*" "carlos-instance@$H.service")" |
| + mkdir -m 0700 -- "$ASIDE/$H" |
| + if [ -L "/opt/carlos/live/$H" ]; then readlink -- "/opt/carlos/live/$H"; fi |
| + aside "/data/.instances/$H" instance |
| + aside "/data/programs/$H.json" program.json |
| + aside "/opt/carlos/live/$H" live-link |
| + aside "/data/$H" key-dir |
| + aside "/data/$H.config.env" config.env |
| + aside "/data/$H.env" env |
| +done |
| +for H in "${HOSTS[@]}"; do |
| + test ! -e "/data/.instances/$H" |
| + test ! -L "/data/.instances/$H" |
| + test ! -e "/data/instances/$H" |
| + test ! -L "/data/instances/$H" |
| + test ! -e "/data/programs/$H.json" |
| + test ! -e "/opt/carlos/live/$H" |
| + test ! -L "/opt/carlos/live/$H" |
| + test ! -e "/data/$H" |
| + test ! -e "/data/$H.config.env" |
| + test ! -e "/data/$H.env" |
| +done |
| +ls -la /data/instances /data/.instances /data/programs /opt/carlos/live |
| +ls -laR "$ASIDE" |
| +echo "B2 OK $ASIDE" |
diff --git a/harness/scenarios/cohort1/blocks/b5.sh b/harness/scenarios/cohort1/blocks/b5.sh| new file mode 100755 |
| index 000000000..48ca1c5b8 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/b5.sh |
| @@ -0,0 +1,34 @@ |
| +#!/bin/bash |
| +# Block B5 (gate G2, read-only), operator shell. Runbook text verbatim but for the harness values: |
| +# R: --profile kass -> harness; B: carlos-dev-640012953448 -> harness-deployment; S: ~/cohort1/ssm.sh; |
| +# sophie i-05498eada6f0afd23 -> $SOPHIE, dennis i-009d593f451cfd562 -> $DENNIS; the inspect's bucket; |
| +# item 6 has no counterpart (the three stopped dev instances do not exist here); item 7's curl goes to |
| +# the console's loopback door (http://127.0.0.1:8090) instead of https://platform.dev.carlosframework.com. |
| +set -uo pipefail |
| +. ~/cohort1/values.env |
| +mkdir -p "$TMPDIR/cohort1" |
| +R="--profile harness --region eu-west-1" |
| +B=harness-deployment |
| +S=~/cohort1/ssm.sh |
| +echo "== B5 $(date -u +%FT%TZ)" |
| +# 1. The empty app segment: no current object, no PRE / |
| +echo "-- 1"; aws $R s3 ls --recursive "s3://$B/apps/bab//" | wc -l |
| +aws $R s3 ls "s3://$B/apps/bab/" |
| +# 2. Instance records, and each one's acq (backfill-collect refuses a record without one) |
| +echo "-- 2"; aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -E '/(instances|sidecars)/[^/]+\.json$' > "$TMPDIR/cohort1/records" || true |
| +cat "$TMPDIR/cohort1/records" |
| +while read -r key; do aws $R s3 cp "s3://$B/$key" - | jq -r --arg k "$key" '[$k, (.acq // "MISSING")] | @tsv'; done < "$TMPDIR/cohort1/records" |
| +# 3. No backfill plan, no claims, no admission |
| +echo "-- 3"; aws $R s3api head-object --bucket "$B" --key control/hosts-backfill.json 2>&1 | grep -c 'Not Found' |
| +aws $R s3api list-objects-v2 --bucket "$B" --prefix control/hosts/ --query 'KeyCount' |
| +aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -cE '/admission\.json$' || true |
| +# 4. The roster's pools, from a signature-verified inspect on sophie (stdout only, nothing staged) |
| +echo "-- 4"; "$S" eu-west-1 "$SOPHIE" 'cd /root/claim-trial && /opt/carlos/carlos system claims inspect --bucket harness-deployment --aws-region eu-west-1 --public-key "$(cat release.pub)" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d[\"roster\"][\"revision\"], d[\"roster\"][\"regions\"][\"ie\"][\"pools\"])"' |
| +# 5. Sophie and dennis: no cohort1 row, no CARLOS_POOLS |
| +echo "-- 5"; "$S" eu-west-1 "$SOPHIE" 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes where pool=\x27cohort1\x27\").fetchone()[0])"; grep -c "^CARLOS_POOLS=" /etc/carlos/host.env || true' |
| +"$S" eu-west-2 "$DENNIS" 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes where pool=\x27cohort1\x27\").fetchone()[0])"; grep -c "^CARLOS_POOLS=" /etc/carlos/host.env || true' |
| +# 6. The three stopped instances carry no box role |
| +echo "-- 6: no counterpart in the harness (no stopped dev instances exist here)" |
| +# 7. The dev console is claims-off |
| +echo "-- 7"; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8090/healthz |
| +"$S" eu-west-1 "$SOPHIE" 'sha256sum /etc/carlos/claim-console.env | cut -c1-64; tr "\0" "\n" < /proc/$(systemctl show -p MainPID --value console.service)/environ | grep "^CARLOS_HOST_CLAIM="; journalctl -u console.service --since "$(systemctl show -p ActiveEnterTimestamp --value console.service)" --no-pager | grep -c "off on a host-claim console" || true' |
diff --git a/harness/scenarios/cohort1/blocks/g4-roll.sh b/harness/scenarios/cohort1/blocks/g4-roll.sh| new file mode 100755 |
| index 000000000..3924e48a6 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/g4-roll.sh |
| @@ -0,0 +1,19 @@ |
| +#!/bin/bash |
| +# Gate G4, the harness equivalent of D10's roll: runbook step 1 item 2's new road for the stable |
| +# channel only (leo and dennis follow stable; sophie stays on canary at v0.28.5, as dev's sophie stays |
| +# on v0.28.4), with the harness's one change, --file (see system-update.sh). |
| +set -euo pipefail |
| +. ~/cohort1/values.env |
| +C=http://127.0.0.1:8090 |
| +F=~/release-v0.28.6/carlos-linux-arm64 |
| +carlos system status --console $C |
| +date -u +%FT%TZ |
| +carlos system update --console $C --channel stable --version v0.28.6 --file $F --platform linux-arm64 --yes |
| +date -u +%FT%TZ |
| +for i in $(seq 1 40); do |
| + s=$(carlos system status --console $C 2>/dev/null) |
| + echo "$s" | grep -q 'kathleen-juggling.*v0.28.6.*current' && echo "$s" | grep -q 'noor-marquetry.*v0.28.6.*current' && break |
| + sleep 15 |
| +done |
| +date -u +%FT%TZ; carlos system status --console $C |
| +for b in "$LEO" "$DENNIS"; do r=eu-west-1; [ "$b" = "$DENNIS" ] && r=eu-west-2; ./../ssm.sh $r "$b" '/opt/carlos/carlos version; /opt/carlos/carlos.prev version; systemctl is-active carlos-edge.service carlos-instance-control.socket; systemctl show carlos-edge.service -p ExecStart --value | grep -o "guard-exec[^;]*" | head -1; systemctl show carlos-instance-control.service -p ActiveEnterTimestamp; cat /data/system/router/applied.json 2>/dev/null | head -c 300; echo'; done |
diff --git a/harness/scenarios/cohort1/blocks/pw-seed.sh b/harness/scenarios/cohort1/blocks/pw-seed.sh| new file mode 100755 |
| index 000000000..0bb94d576 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/pw-seed.sh |
| @@ -0,0 +1,66 @@ |
| +#!/bin/bash |
| +# Pre-window state, SEEDED BY PLATFORM COMMANDS (host root shell), from snapshot pre-window + H14/H15. |
| +# Recreates what the runbook assumes on the morning of the window, through the ordinary doors, claims off: |
| +# A sophie: the 2026-10-19 pause (saved copies, then the runbook's pause recipe, verbatim) |
| +# B wakecheck (pool-less) before leo's pool; woken on leo so leo holds its leftovers (as dev's leo does) |
| +# C runbook step 4.2 (CARLOS_POOLS=cohort1) and 4.3 (carlos remove on leo), box commands verbatim |
| +# D claimtrial (step 4.4, pool cohort1, store-mode enable, signed-in create), woken on leo (step 4.5) |
| +# E rollcheck-0261 (pool-less, --kind static) and the offline-move rehearsal app moverehearsal (--region uk) |
| +# Harness substitutions: console http://127.0.0.1:8090, domain dev.harness.test, build = the harness |
| +# wakecheck stand-in (harness/apps/wakecheck) shipped as version wc4, certificates from H13. |
| +set -euo pipefail |
| +OP() { runuser -l operator -c "set -euo pipefail; C=http://127.0.0.1:8090; F=~/cohort1/pw/linux-arm64; $1"; } |
| +echo "== A1 saved copies ($(date -u +%TZ))" |
| +incus exec sophie -- bash -c 'set -euo pipefail; umask 077 |
| +D=/root/claim-trial/paused-20261019; install -d -m 0700 "$D" |
| +cp -p /etc/carlos/claim-console.env "$D/claim-console.env"; cp -p /etc/systemd/system/console.service.d/91-claim-runtime.conf "$D/91-claim-runtime.conf" |
| +cmp /etc/carlos/claim-console.env "$D/claim-console.env"; cmp /etc/systemd/system/console.service.d/91-claim-runtime.conf "$D/91-claim-runtime.conf" |
| +sha256sum "$D"/*' |
| +echo "== A2 pause recipe, verbatim but its last line (see Runbook corrections) ($(date -u +%TZ))" |
| +incus exec sophie -- bash -c 'set -euo pipefail |
| +umask 077 |
| +[ "$(sha256sum /etc/carlos/claim-console.env | cut -c1-64)" = 5df17bac42a91a02d90584edc5476c045400af57e467b2ea0815ba50b7e31d03 ] |
| +printf '"'"'CARLOS_CONSOLE_ID=dev-console\nCONSOLE_HOME_REGION=ie\nCARLOS_HOST_CLAIM=\n'"'"' > /etc/carlos/.claim-console.env.pause |
| +[ "$(sha256sum /etc/carlos/.claim-console.env.pause | cut -c1-64)" = c1ae2900db31295090d2f4f2f9b5f6cdcca07e74545a6b1a9da039c3abf5c412 ] |
| +mv -T /etc/carlos/.claim-console.env.pause /etc/carlos/claim-console.env |
| +systemctl restart console.service |
| +systemctl is-active console.service |
| +for i in $(seq 1 40); do p=$(systemctl show -p MainPID --value console.service); [ "$(readlink /proc/$p/exe)" = /opt/carlos/system/console/console ] && break; sleep 0.25; done |
| +tr '"'"'\0'"'"' '"'"'\n'"'"' < "/proc/$p/environ" | grep '"'"'^CARLOS_HOST_CLAIM='"'"' |
| +sleep 3; curl -s -o /dev/null -w "healthz %{http_code}\n" http://127.0.0.1:8090/healthz' |
| +install -d -o operator -g operator -m 0755 /home/operator/cohort1/pw |
| +aws s3 cp --quiet s3://carlos-harness-640012953448/artefacts/v0.28.6/wakecheck-linux-arm64 /home/operator/cohort1/pw/linux-arm64 |
| +chown operator:operator /home/operator/cohort1/pw/linux-arm64 |
| +echo "== B wakecheck ($(date -u +%TZ))" |
| +OP 'sha256sum $F; carlos apps create --app wakecheck --account bab --console $C |
| +carlos ship --app wakecheck --account bab --console $C --version wc4 --label "harness wakecheck stand-in, wc4" --no-prompt $F |
| +carlos promote --app wakecheck --account bab --console $C --no-prompt wc4 edge |
| +AWS_PROFILE=harness CARLOS_DEPLOYMENT_BUCKET=harness-deployment carlos instances enable --app wakecheck --account bab --domain dev.harness.test --channel edge |
| +carlos instances create --app wakecheck --account bab --host wakecheck.dev.harness.test --console $C --no-prompt' |
| +bash /root/harness/certs.sh wakecheck.dev.harness.test claimtrial.dev.harness.test rollcheck-0261.dev.harness.test moverehearsal.dev.harness.test |
| +sleep 40 |
| +for i in 1 2 3; do incus exec leo -- curl -s -m 40 --cacert /root/harness-ca.pem -w ' [%{http_code}]\n' --resolve wakecheck.dev.harness.test:443:127.0.0.1 https://wakecheck.dev.harness.test/; done |
| +incus exec leo -- sh -c 'journalctl -u carlos-edge --since -2min --no-pager | grep -E "layout|nested" | tail -3; ls /data/instances /data/.instances' |
| +echo "== C step 4.2, 4.3 ($(date -u +%TZ))" |
| +incus exec leo -- bash -c 'cp -p /etc/carlos/host.env /etc/carlos/host.env.bak-pool-$(date -u +%Y%m%d) |
| +printf "CARLOS_POOLS=cohort1\n" >> /etc/carlos/host.env |
| +systemctl restart carlos-edge.service && systemctl is-active carlos-edge.service' |
| +sleep 8 |
| +incus exec leo -- bash -c 'runuser -u carlos -- /opt/carlos/carlos remove --registry /data/registry.db wakecheck.dev.harness.test; systemctl is-active carlos-edge.service' |
| +echo "== D, E ($(date -u +%TZ))" |
| +for app in claimtrial rollcheck-0261 moverehearsal; do |
| + OP "carlos apps create --app $app --account bab --console \$C |
| +carlos ship --app $app --account bab --console \$C --version wc4 --label 'harness pre-window $app, wakecheck stand-in wc4' --no-prompt \$F |
| +carlos promote --app $app --account bab --console \$C --no-prompt wc4 edge" |
| + sleep 30 |
| +done |
| +OP 'AWS_PROFILE=harness CARLOS_DEPLOYMENT_BUCKET=harness-deployment carlos instances enable --app claimtrial --account bab --domain dev.harness.test --pool cohort1 --channel edge |
| +AWS_PROFILE=harness CARLOS_DEPLOYMENT_BUCKET=harness-deployment carlos instances enable --app rollcheck-0261 --account bab --domain dev.harness.test --channel edge |
| +AWS_PROFILE=harness CARLOS_DEPLOYMENT_BUCKET=harness-deployment carlos instances enable --app moverehearsal --account bab --domain dev.harness.test --channel edge |
| +carlos instances create --app claimtrial --account bab --host claimtrial.dev.harness.test --console $C --no-prompt |
| +carlos instances create --app rollcheck-0261 --account bab --host rollcheck-0261.dev.harness.test --kind static --console $C --no-prompt |
| +carlos instances create --app moverehearsal --account bab --host moverehearsal.dev.harness.test --region uk --console $C --no-prompt' |
| +sleep 40 |
| +for i in 1 2 3; do incus exec leo -- curl -s -m 40 --cacert /root/harness-ca.pem -w ' [%{http_code}]\n' --resolve claimtrial.dev.harness.test:443:127.0.0.1 https://claimtrial.dev.harness.test/; done |
| +for b in leo sophie dennis; do echo "== $b"; incus exec $b -- python3 -c "import sqlite3; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True); [print(r) for r in c.execute('select host,app,pool,region,channel,hibernate from routes order by host')]"; incus exec $b -- sh -c 'ls /data/instances /data/.instances /data/programs /opt/carlos/live 2>&1 | tr "\n" " "; echo'; done |
| +OP 'aws s3 ls --recursive s3://harness-deployment/apps/ | awk "{print \$4}" | grep -E "/(instances|sidecars)/[^/]+\.json$"' |
diff --git a/harness/scenarios/cohort1/blocks/w-identity.sh b/harness/scenarios/cohort1/blocks/w-identity.sh| new file mode 100755 |
| index 000000000..5c911a604 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/w-identity.sh |
| @@ -0,0 +1,26 @@ |
| +#!/bin/bash |
| +# Gate G3: publish-identity for the window release (runbook 6.1's second phase; the floor is |
| +# unchanged, so no sign-policy). HARNESS: tag v0.28.6, console ship version cc90d542 (shipped and |
| +# promoted to canary/console by release-window.sh, the 5.0a step the window package lacks), bucket. |
| +set -euo pipefail |
| +. ~/cohort1/values.env |
| +cd ~/cohort1 |
| +RUN=$(cat run-id); X=s3://harness-deployment/claim-signer/$RUN |
| +./ssm.sh eu-west-1 "$SIGNER" " |
| +. /root/claim-trial/helpers.sh |
| +CV=cc90d542 |
| +jq -n --rawfile sums SHA256SUMS.txt --rawfile sig SHA256SUMS.txt.sig --arg cv \"\$CV\" \\ |
| + '{identity:{release_tag:\"v0.28.6\",moves_protocol:1,sha256sums:\$sums,sha256sums_sig:\$sig, |
| + console:{account:\"bab\",app:\"console\",channel:\"canary/console\",version:\$cv}}}' > identity-phase.json |
| +trap 'tail -20 publish-identity-w.log' EXIT |
| +phase_out publish-identity identity-phase.json /root/claim-trial/identity.v0.28.6.json publish-identity-w-result.json > publish-identity-w.log 2>&1 |
| +trap - EXIT |
| +grep -E '\"writes\"|before_etag|\"key\"' publish-identity-w.log | head -8 |
| +echo \"publish-identity writes: \$(jq -c .writes publish-identity-w-result.json)\" |
| +jq '{identity_key, release}' publish-identity-w-result.json |
| +aws s3 cp --quiet identity.v0.28.6.json $X/out/identity.v0.28.6.json; sha256sum identity.v0.28.6.json | cut -c1-16 |
| +echo REV=\$REV |
| +" |
| +mkdir -p out |
| +aws s3 cp --quiet "$X/out/identity.v0.28.6.json" out/identity.v0.28.6.json |
| +aws s3 cp --quiet s3://harness-deployment/control/system/router/releases/v0.28.6.json - | cmp - out/identity.v0.28.6.json && echo "identity file = published object" |
diff --git a/harness/scenarios/cohort1/blocks/w-open-signer.sh b/harness/scenarios/cohort1/blocks/w-open-signer.sh| new file mode 100755 |
| index 000000000..728690c69 |
| --- /dev/null |
| +++ b/harness/scenarios/cohort1/blocks/w-open-signer.sh |
| @@ -0,0 +1,39 @@ |
| +#!/bin/bash |
| +# Signing session 6 (window) opening, as 51-open-signer.sh, with the window release v0.28.6 (runbook "Order of work" 1; signing-host.md "Getting jq and the |
| +# carlos CLI there" and "Getting the key there"), from the operator shell. |
| +# HARNESS: the signer is the signer container (no EC2 instance, role or SG is created: its |
| +# MinIO user stands for carlos-claim-signer); the exchange is s3://harness-deployment/claim-signer/<run-id>/; |
| +# jq is Debian 13's packaged jq (installed at base), not the jq-1.7.1 GitHub binary. |
| +set -euo pipefail |
| +. ~/cohort1/values.env |
| +cd ~/cohort1 |
| +RUN=cohort1-$(date -u +%Y%m%dT%H%M%SZ); echo "$RUN" > run-id; echo "run-id $RUN" |
| +X=s3://harness-deployment/claim-signer/$RUN |
| +for f in ~/release.pub ~/release-v0.28.6/SHA256SUMS.txt ~/release-v0.28.6/SHA256SUMS.txt.sig ~/release-v0.28.6/carlos-linux-arm64 \ |
| + inputs/floor.json inputs/regions.json inputs/leo.json inputs/sophie.json inputs/console.json signer-helpers.sh; do |
| + aws s3 cp --quiet "$f" "$X/$(basename "$f")" |
| +done |
| +aws s3 ls "$X/" |
| +SUMS=$(cd ~/release-v0.28.6 && sha256sum carlos-linux-arm64 SHA256SUMS.txt SHA256SUMS.txt.sig | awk '{print $1}' | paste -sd' ') |
| +H=$(sha256sum signer-helpers.sh | cut -d' ' -f1); P=$(sha256sum ~/release.pub | cut -d' ' -f1) |
| +./ssm.sh eu-west-1 "$SIGNER" " |
| +set -euo pipefail |
| +set -- $SUMS |
| +aws s3 cp --quiet $X/carlos-linux-arm64 /usr/local/bin/carlos |
| +printf '%s %s\n' \$1 /usr/local/bin/carlos | sha256sum -c |
| +chmod 0755 /usr/local/bin/carlos |
| +/usr/local/bin/carlos version |
| +jq --version; command -v jq |
| +install -d -m 0700 -o root -g root /root/claim-trial |
| +cd /root/claim-trial |
| +for f in release.pub SHA256SUMS.txt SHA256SUMS.txt.sig floor.json regions.json leo.json sophie.json console.json signer-helpers.sh; do aws s3 cp --quiet $X/\$f \$f; done |
| +mv signer-helpers.sh helpers.sh; chmod 0600 * |
| +printf '%s %s\n' \$2 SHA256SUMS.txt \$3 SHA256SUMS.txt.sig $H helpers.sh $P release.pub | sha256sum -c |
| +swapon --show | wc -l | sed 's/^/swap devices: /'; findmnt -no FSTYPE --target /run | sed 's/^/\/run: /' |
| +/usr/local/bin/carlos system claims help 2>&1 | sed 's/^/help| /' |
| +echo session-start \$(date -u +%FT%TZ) |
| +install -d -m 0700 -o root -g root /run/claim-signer |
| +aws --region eu-west-1 ssm get-parameter --name /carlos/release-key --with-decryption \ |
| + --query Parameter.Value --output text | install -m 0600 -o root -g root /dev/stdin /run/claim-signer/release.key |
| +stat -c '%U %a %h %s' /run/claim-signer/release.key |
| +" |
diff --git a/harness/scenarios/cohort1/values.env b/harness/scenarios/cohort1/values.env| index ce2c3c1c2..2c96b171d 100644 |
| --- a/harness/scenarios/cohort1/values.env |
| +++ b/harness/scenarios/cohort1/values.env |
| @@ -10,3 +10,8 @@ BUCKET=harness-deployment # dev: carlos-dev-640012953448 |
| PROFILE=harness # dev: kass |
| TAG=v0.28.5 # dev: v0.28.4 |
| CV=1b635e61 # dev: f3bf1214 (the console's ship version) |
| +# Window (Option B) values, 2026-10-20 morning harness run: |
| +WTAG=v0.28.6 # dev: v0.28.5 (the window release, claim-empty-cohort-done); harness v0.28.6 = cc90d542 + harness patch |
| +WCV=cc90d542 # dev: the v0.28.5 console's ship version |
| +DOMAIN=dev.harness.test # dev: dev.carlosframework.com |
| +CONSOLE=http://127.0.0.1:8090 # dev: https://platform.dev.carlosframework.com |
diff --git a/harness/scenarios/sync.sh b/harness/scenarios/sync.sh| new file mode 100755 |
| index 000000000..c791fe5b8 |
| --- /dev/null |
| +++ b/harness/scenarios/sync.sh |
| @@ -0,0 +1,8 @@ |
| +#!/bin/bash |
| +# sync.sh: copy harness/scenarios/cohort1 (blocks, helpers, inputs; never out/) to the |
| +# operator's ~/cohort1 on the host, through hssm.py (DOCS-T1, sandbox off). |
| +set -euo pipefail |
| +d=$(cd "$(dirname "$0")" && pwd) |
| +b64=$(tar -C "$d/cohort1" -czf - --exclude=out . | base64 -w0) |
| +printf 'echo %s | base64 -d | tar -xzf - -C /home/operator/cohort1 --no-same-owner\nchown -R operator:operator /home/operator/cohort1\nls /home/operator/cohort1/blocks | wc -l\n' "$b64" \ |
| + | python3 "$d/../hssm.py" - |
diff --git a/harness/snapshot.sh b/harness/snapshot.sh| index e5a3a411f..4542f0916 100755 |
| --- a/harness/snapshot.sh |
| +++ b/harness/snapshot.sh |
| @@ -2,6 +2,8 @@ |
| # Consistent harness snapshot / restore: every container stopped, plus sophie's |
| # ext4 console volume (H11), which container snapshots do not include. |
| # snapshot.sh create <name> | restore <name> |
| +# (Fixed 2026-10-20 07:3xZ: the volume test used "show hdir sophie-console <name>", which incus 6.0.4 |
| +# rejects, so every restore before then skipped the console volume.) |
| set -euo pipefail |
| op=$1 name=$2 |
| ALL="infra leo sophie dennis signer" |
| @@ -10,8 +12,11 @@ case $op in |
| create) for c in $ALL; do incus snapshot create $c "$name"; done |
| incus storage volume snapshot create hdir sophie-console "$name" >/dev/null ;; |
| restore) for c in $ALL; do incus snapshot restore $c "$name"; done |
| - if incus storage volume snapshot show hdir sophie-console "$name" >/dev/null 2>&1; then |
| + if incus storage volume snapshot show hdir "sophie-console/$name" >/dev/null 2>&1; then |
| incus storage volume snapshot restore hdir sophie-console "$name" |
| + echo "sophie-console volume restored to $name" |
| + else |
| + echo "WARNING: no sophie-console volume snapshot $name; the console DB was NOT restored" |
| fi ;; |
| *) echo "usage: snapshot.sh create|restore <name>"; exit 2 ;; |
| esac |
diff --git a/harness/stand-ins/certs.sh b/harness/stand-ins/certs.sh| new file mode 100755 |
| index 000000000..ebeaa205b |
| --- /dev/null |
| +++ b/harness/stand-ins/certs.sh |
| @@ -0,0 +1,30 @@ |
| +#!/bin/bash |
| +# H13, the certificate stand-in (on host, root): the harness has no route to |
| +# Let's Encrypt, so a host's certificate is minted under a harness test CA and |
| +# published where the housekeeper's issuance would publish it, |
| +# control/certs/<host> (autocert's cache entry: private key PEM, then the |
| +# certificate PEM; internal/edge/certsync.go parseCertEntry). Every edge's |
| +# mirror then serves it, as dev's edges serve sophie's Let's Encrypt copies. |
| +# The CA key lives on tmpfs (/run/harness/keys/ca.key); its certificate goes |
| +# to /home/operator/harness-ca.pem and each box's /root/harness-ca.pem for |
| +# curl --cacert. |
| +# certs.sh <host> [<host>...] |
| +set -euo pipefail |
| +K=/run/harness/keys; W=$(mktemp -d); trap 'rm -rf "$W"' EXIT |
| +if [ ! -s $K/ca.key ]; then |
| + (umask 077; openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out $K/ca.key) |
| + openssl req -x509 -new -key $K/ca.key -subj "/CN=carlos harness test CA" -days 7 -out $K/ca.pem 2>/dev/null |
| +fi |
| +install -o operator -g operator -m 0644 $K/ca.pem /home/operator/harness-ca.pem |
| +for c in leo sophie dennis; do incus file push $K/ca.pem $c/root/harness-ca.pem --uid 0 --gid 0 --mode 0644 -q; done |
| +eval "$(python3 -c 'import json; u=json.load(open("/root/harness/minio-users.json"))["operator"]; print("export AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s" % (u["access_key"], u["secret_key"]))')" |
| +export AWS_ENDPOINT_URL_S3=http://203.0.113.10:9000 AWS_REGION=eu-west-1 |
| +for h in "$@"; do |
| + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out $W/$h.key |
| + openssl req -new -key $W/$h.key -subj "/CN=$h" -out $W/$h.csr 2>/dev/null |
| + printf 'subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\n' "$h" > $W/$h.ext |
| + openssl x509 -req -in $W/$h.csr -CA $K/ca.pem -CAkey $K/ca.key -CAcreateserial -days 3 -extfile $W/$h.ext -out $W/$h.pem 2>/dev/null |
| + cat $W/$h.key $W/$h.pem > $W/$h.entry |
| + aws s3 cp --quiet $W/$h.entry "s3://harness-deployment/control/certs/$h" |
| + echo "published control/certs/$h ($(openssl x509 -in $W/$h.pem -noout -enddate))" |
| +done |