amadan
harness: SSM stand-in, operator shell, bootstrap-box.sh unchanged, snapshot boot (step 6); TEARDOWN
carlos-edge active on all three boxes with nosymfollow mounts, cgroup BPF enforced in the nested containers, status/boxes reports in MinIO 56 s after bootstrap, stdout truncated at 24,000 by the stand-in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
14 files changed,
+744
−0
harness/RESOURCES.md+1 −0harness/RUNLOG.md+100 −0harness/TEARDOWN.md+54 −0harness/checks/check2.sh+39 −0harness/checks/check5.sh+13 −0harness/checks/check6.sh+37 −0harness/checks/check6b.sh+27 −0harness/checks/fix5.sh+6 −0harness/checks/run6b.sh+9 −0harness/checks/snap-base.sh+18 −0harness/checks/snap-boot.sh+11 −0harness/checks/ssmsmoke.sh+18 −0harness/operator/operator-setup.sh+101 −0harness/stand-ins/ssm.py+310 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/harness/RESOURCES.md b/harness/RESOURCES.md| index 96cb866d9..fbfc845ce 100644 |
| --- a/harness/RESOURCES.md |
| +++ b/harness/RESOURCES.md |
| @@ -14,3 +14,4 @@ Appended as each is created. Teardown commands: harness/TEARDOWN.md. |
| | 2026-10-20T00:50:44Z | ec2 instance (donor-2) | i-019177034b088459f | t4g.nano, profile carlos-harness, terminate on shutdown, shutdown -h +60; to be terminated once captured | |
| | 2026-10-20T00:51:50Z | (terminated) donor-1 | i-0d9175215cfcf63b8 | documents in s3://carlos-harness-640012953448/donors/donor-1/ | |
| | 2026-10-20T00:51:52Z | (terminated) donor-2 | i-019177034b088459f | documents in s3://carlos-harness-640012953448/donors/donor-2/ | |
| +| 2026-10-20T01:06:29Z | ebs volume (host root) | vol-035c92e5b506cc3db | 60 GiB gp3, DeleteOnTermination=true; goes with the host | |
diff --git a/harness/RUNLOG.md b/harness/RUNLOG.md| index 212f8c99b..1f87c8910 100644 |
| --- a/harness/RUNLOG.md |
| +++ b/harness/RUNLOG.md |
| @@ -270,3 +270,103 @@ MinIO 203.0.113.10:9000 connects. To reopen egress for re-provisioning (step 9's |
| `base-opt`): `nft add element inet harness egress_ports '{ 80, 443 }'; rm /etc/harness/egress-closed`. |
| **Step 5: GREEN** (2026-10-20T00:59:15Z). |
| + |
| +## Step 6: the SSM stand-in, `bootstrap-box.sh` unchanged, snapshot `boot` |
| + |
| +Harness binaries rebuilt at `f434a425` (`version.Build=harness-f434a425`, `-tags harness`, |
| +linux/arm64), sha256 carlos `f07a0c52...`, console `f1e85706...`, at |
| +`s3://carlos-harness-640012953448/artefacts/harness-f434a425/`. |
| + |
| +`harness/operator/operator-setup.sh` (on host, 01:02 UTC): awscli on the host; the harness |
| +`carlos` at `/usr/local/bin/carlos`; `/etc/carlos/harness-endpoint` on the host; the test |
| +**root release key** from `carlos release-keygen` at `/run/harness/keys/release.env` (root |
| +0600, tmpfs; pubkey `SN3K2ONY...`, sha256 `25b94d19adb8`); user `operator` (primary group |
| +Debian's existing `operator`, gid 37) with `~/.aws` profile `harness` = MinIO |
| +`harness-operator` and `~/.harness-env` = exactly `AWS_ENDPOINT_URL_SSM`, |
| +`AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_PROFILE`; `~/infra/scripts/` = verbatim |
| +`bootstrap-box.sh` and `update-platform-binary.sh` from a0b3f5a. Then the SSM stand-in |
| +(`harness/stand-ins/ssm.py` -> `/opt/harness/ssm.py`, unit `harness-ssm.service`, bound |
| +203.0.113.1:8100; `/etc/harness/ssm.json`: leo = `i-0d9175215cfcf63b8` (donor-1), sophie = |
| +`i-019177034b088459f` (donor-2), dennis = `i-00000000000000023`, signer = |
| +`i-00000000000000030`, the last two harness-only ids). **Seeded:** the operator copied the |
| +harness `carlos` to `s3://harness-deployment/bootstrap/carlos` (the release recipe's job). |
| + |
| +Stand-in smoke test (`harness/checks/ssmsmoke.sh`), real aws CLI 2.23.6 from the operator shell: |
| +runs as `root`, `umask 0022`, `HOME` unset, cwd |
| +`/var/lib/amazon/ssm/<iid>/document/orchestration/<cid>/awsrunShellScript/0.awsrunShellScript`, |
| +env = agent PATH + the three injected variables; unknown instance -> `InvalidInstanceId`; |
| +Parameter Store put/get (SecureString)/`ParameterAlreadyExists`/delete/`ParameterNotFound`. |
| + |
| +Bootstrap (`harness/checks/run6b.sh`), as `operator`, script unchanged: |
| + |
| +``` |
| +~/infra/scripts/bootstrap-box.sh harness i-0d9175215cfcf63b8 harness-deployment harness-data-ie ~/release.pub eu-west-1 # leo 01:03:04-01:03:11Z |
| +~/infra/scripts/bootstrap-box.sh harness i-019177034b088459f harness-deployment harness-data-ie ~/release.pub eu-west-1 # sophie 01:03:11-01:03:18Z |
| +~/infra/scripts/bootstrap-box.sh harness i-00000000000000023 harness-deployment harness-data-uk ~/release.pub eu-west-2 uk # dennis 01:03:18-01:03:25Z |
| +each: status: Success / bootstrap: units installed, carlos-edge.service + ... + carlos-instance-control.socket enabled / active / carlos harness-f434a425 |
| +``` |
| + |
| +Check (`harness/checks/check6.sh`, `check6b.sh`, 01:04-01:06 UTC): |
| + |
| +``` |
| +status/boxes/ in MinIO, each 56 s after its box's bootstrap returned (01:03:11, :18, :25): |
| + 01:04:07 status/boxes/kathleen-juggling.json (leo) |
| + 01:04:14 status/boxes/ada-letterpress.json (sophie) |
| + 01:04:21 status/boxes/noor-marquetry.json (dennis) |
| +every box: carlos-edge active, Result=success NRestarts=0 |
| + data-instances.mount active success /data/instances rw,nosuid,nodev,noexec,relatime,nosymfollow (btrfs subvol) |
| + run-carlos-instances.mount active success /run/carlos/instances rw,nosuid,nodev,noexec,nosymfollow (tmpfs) |
| + run-carlos-logs.mount active success /run/carlos/logs rw,nosuid,nodev,noexec,nosymfollow (tmpfs) |
| + systemctl --failed: empty; is-system-running: running |
| + journal this boot: [0] "Failed to set up mount namespacing" [0] IPAddressDeny [0] BPF [0] "not supported" |
| + [0] AccessDenied [0] panic: [0] amazonaws.com [0] "Permission denied" |
| +cgroup BPF inside the nested container (functional): |
| + systemd-run -p IPAddressDeny=any -p IPAddressAllow=localhost -> MinIO: blocked (timed out) |
| + systemd-run (no deny) -> MinIO: CONNECTED |
| + carlos-instance@.service IPAddressDeny covers 203.0.113.0/24? no (B2 holds) |
| +host.env (no endpoint variable; the units reach MinIO through the harness file fallback): |
| + CARLOS_DATA_DIR=/data CARLOS_DEPLOYMENT_BUCKET=harness-deployment CARLOS_HIBERNATE_BUCKET=harness-data-ie |
| + CARLOS_HIBERNATE_REGION=eu-west-1 CARLOS_HIBERNATE_PREFIX=instances AWS_REGION=eu-west-1 (+ CARLOS_RELEASE_PUBKEY; |
| + dennis: harness-data-uk, eu-west-2, CARLOS_REGION=uk) |
| +layout after bootstrap: /opt/carlos carlos:carlos 755, /opt/carlos/carlos carlos:carlos 755, host.env carlos:carlos 600 |
| +truncation: 30,000-byte stdout + 9,000-byte stderr -> Status Success, stdout 24000 chars, stderr 8000 chars |
| +``` |
| + |
| +Box labels (from the fixed machine-ids): leo = `kathleen-juggling`, sophie = `ada-letterpress`, |
| +dennis = `noor-marquetry`. |
| + |
| +Snapshot `boot` (`harness/checks/snap-boot.sh`): all five stopped, `incus snapshot create <c> boot`, |
| +started; after restart every container `running`, `--failed` empty, `carlos-edge` active with |
| +NRestarts=0 on all three boxes; stop-snapshot-start plus checks took 10 s. |
| + |
| +Noted, not acted on: the boxes' tmpfs mounts size to 20% of the **host's** RAM (3.2 GB), not the |
| +container's 1 GiB (no lxcfs meminfo); dev boxes would size to their own 1 GiB. |
| + |
| +**Step 6: GREEN** (2026-10-20T01:06:11Z). |
| + |
| +## Handoff to steps 7 to 11 |
| + |
| +State: host `i-06a1ec144aa5226fb` running (TTL 2026-10-20T12:40Z, self-terminates); containers |
| +infra, leo, sophie, dennis, signer running at snapshot `boot` (also `base`); provisioning |
| +egress closed; stand-ins `harness-imds` and `harness-ssm` active; test root release key at |
| +`/run/harness/keys/release.env` (pubkey in every box's host.env). Not yet made: the test |
| +**sums** key (write its public half to `/etc/carlos/harness-sums-pubkey`, root 0644, on |
| +every box and the signer, and on the host for the operator CLI), `/carlos/release-key` in the |
| +fake Parameter Store, the console, the Keymail stand-in. |
| + |
| +Get in: |
| + |
| +```bash |
| +# Non-interactive, from the worktree on DOCS-T1 (sandbox off): any script runs as root on the host. |
| +echo 'incus exec leo -- systemctl status carlos-edge --no-pager' | python3 harness/hssm.py - |
| +echo 'runuser -l operator -c "aws s3 ls s3://harness-deployment/status/boxes/"' | python3 harness/hssm.py - |
| + |
| +# Interactive: |
| +aws ssm start-session --profile kass --region eu-west-1 --target i-06a1ec144aa5226fb |
| +sudo -i # then: |
| +incus exec leo -- bash # a box (leo, sophie, dennis, signer, infra) |
| +runuser -l operator # the operator shell: harness carlos CLI, aws -> stand-in SSM + MinIO |
| +``` |
| + |
| +Reset: `incus stop infra leo sophie dennis signer; for c in infra leo sophie dennis signer; do |
| +incus snapshot restore $c boot; done; incus start infra; incus start leo sophie dennis signer`. |
diff --git a/harness/TEARDOWN.md b/harness/TEARDOWN.md| new file mode 100644 |
| index 000000000..c64602f70 |
| --- /dev/null |
| +++ b/harness/TEARDOWN.md |
| @@ -0,0 +1,54 @@ |
| +# Harness teardown |
| + |
| +Every resource in `RESOURCES.md`, in dependency order. Run from DOCS-T1 with the |
| +sandbox off (AWS needs it). Account 640012953448, eu-west-1, profile `kass`. |
| +Each destructive step checks the `purpose=carlos-harness` tag first and prints |
| +what it is about to remove. |
| + |
| +The host also removes itself: it runs `shutdown -h 12:40` UTC (its TTL, |
| +2026-10-20T12:40Z) with `instance-initiated-shutdown-behavior=terminate`, and its |
| +root volume has `DeleteOnTermination=true`. Steps 3 to 5 are still needed after that. |
| + |
| +```bash |
| +A="aws --profile kass --region eu-west-1" |
| +HOST=i-06a1ec144aa5226fb |
| +SG=sg-0b9cf4c2a508b7cec |
| +BUCKET=carlos-harness-640012953448 |
| + |
| +# 1. The host (and with it vol-035c92e5b506cc3db). Refuses an untagged instance. |
| +P=$($A ec2 describe-instances --instance-ids $HOST --query 'Reservations[0].Instances[0].Tags[?Key==`purpose`].Value|[0]' --output text) |
| +[ "$P" = carlos-harness ] && echo "terminating $HOST (purpose=$P)" && \ |
| + $A ec2 terminate-instances --instance-ids $HOST && $A ec2 wait instance-terminated --instance-ids $HOST |
| + |
| +# 2. Leak check: nothing tagged may be left running or attached. The donors |
| +# (i-0d9175215cfcf63b8, i-019177034b088459f) were terminated at 00:52Z. |
| +$A ec2 describe-instances --filters Name=tag:purpose,Values=carlos-harness \ |
| + Name=instance-state-name,Values=pending,running,stopping,stopped,shutting-down \ |
| + --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text # expect nothing |
| +$A ec2 describe-volumes --filters Name=tag:purpose,Values=carlos-harness \ |
| + --query 'Volumes[].[VolumeId,State]' --output text # expect nothing |
| + |
| +# 3. The security group (only once the host is gone). |
| +$A ec2 describe-security-groups --group-ids $SG --query 'SecurityGroups[0].Tags[?Key==`purpose`].Value|[0]' --output text # carlos-harness |
| +$A ec2 delete-security-group --group-id $SG |
| + |
| +# 4. The instance profile and role. |
| +$A iam remove-role-from-instance-profile --instance-profile-name carlos-harness --role-name carlos-harness |
| +$A iam delete-instance-profile --instance-profile-name carlos-harness |
| +$A iam detach-role-policy --role-name carlos-harness --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore |
| +$A iam delete-role-policy --role-name carlos-harness --policy-name carlos-harness-bucket |
| +$A iam delete-role --role-name carlos-harness |
| + |
| +# 5. The bucket (unversioned; lifecycle already expires objects after 3 days). |
| +$A s3api get-bucket-tagging --bucket $BUCKET --query 'TagSet[?Key==`purpose`].Value|[0]' --output text # carlos-harness |
| +$A s3 rm --recursive --quiet s3://$BUCKET |
| +$A s3api delete-bucket --bucket $BUCKET |
| +``` |
| + |
| +To tear down only the containers and keep the host (for example to rebuild a |
| +box from scratch), on the host as root: `incus delete --force leo sophie dennis signer infra`. |
| +To return to a known state instead: stop all five, `incus snapshot restore <c> base|boot`, |
| +start `infra` first, then the rest. |
| + |
| +Nothing harness-related lives on DOCS-T1 beyond this worktree: the scratch |
| +directory `$TMPDIR/harness-build` was deleted at the end of the phase 1 build. |
diff --git a/harness/checks/check2.sh b/harness/checks/check2.sh| new file mode 100644 |
| index 000000000..e5b2e6f5d |
| --- /dev/null |
| +++ b/harness/checks/check2.sh |
| @@ -0,0 +1,39 @@ |
| +set -u |
| +echo "== incus list (before)"; incus list -f csv; echo "(end)" |
| +echo "== shutdown --show"; shutdown --show 2>&1 |
| +echo "== nft guard"; nft list table inet harness | head -40 |
| +incus launch images:debian/13/cloud scratch -p box -q |
| +for i in $(seq 1 30); do incus exec scratch -- python3 -c 1 2>/dev/null && incus exec scratch -- ip -4 -o addr show eth0 | grep -q 203.0.113 && break; sleep 2; done |
| +incus exec scratch -- ip -4 -o addr show eth0 |
| +HOSTIP=$(ip -4 -o addr show ens5 | awk '{print $4}' | cut -d/ -f1) |
| +cat > /tmp/probe.py <<PY |
| +import socket, urllib.request, sys |
| +def tcp(h, p): |
| + s = socket.socket(); s.settimeout(4) |
| + try: |
| + s.connect((h, p)); return "CONNECTED" |
| + except ConnectionRefusedError: return "refused" |
| + except socket.timeout: return "timeout (dropped)" |
| + except OSError as e: return "error %s" % e |
| + finally: s.close() |
| +try: |
| + r = urllib.request.Request("http://169.254.169.254/latest/api/token", method="PUT", headers={"X-aws-ec2-metadata-token-ttl-seconds": "60"}) |
| + tok = urllib.request.urlopen(r, timeout=5).read() |
| + print("imds token PUT: GOT A TOKEN (len %d)" % len(tok)) |
| + r = urllib.request.Request("http://169.254.169.254/latest/meta-data/iam/security-credentials/", headers={"X-aws-ec2-metadata-token": tok.decode()}) |
| + print("imds role list:", urllib.request.urlopen(r, timeout=5).read()) |
| +except Exception as e: |
| + print("imds token PUT:", type(e).__name__, e) |
| +print("tcp 169.254.169.254:80 ->", tcp("169.254.169.254", 80)) |
| +print("tcp 169.254.169.253:53 ->", tcp("169.254.169.253", 53)) |
| +print("tcp 169.254.170.2:80 ->", tcp("169.254.170.2", 80)) |
| +print("tcp host VPC ip $HOSTIP:22 ->", tcp("$HOSTIP", 22)) |
| +print("tcp 203.0.113.1:22 ->", tcp("203.0.113.1", 22)) |
| +print("tcp 172.31.0.2:53 (VPC resolver) ->", tcp("172.31.0.2", 53)) |
| +print("tcp deb.debian.org:443 (provisioning egress) ->", tcp("deb.debian.org", 443)) |
| +PY |
| +incus file push /tmp/probe.py scratch/root/probe.py -q |
| +incus exec scratch -- python3 /root/probe.py |
| +rm -f /tmp/probe.py |
| +incus delete --force scratch |
| +echo "== incus list (after)"; incus list -f csv; echo "(end)" |
diff --git a/harness/checks/check5.sh b/harness/checks/check5.sh| new file mode 100644 |
| index 000000000..44a05727b |
| --- /dev/null |
| +++ b/harness/checks/check5.sh |
| @@ -0,0 +1,13 @@ |
| +set -u |
| +for b in leo sophie dennis; do |
| + echo "================ $b" |
| + incus exec $b -- sh -c 'echo "cloud-init: $(cloud-init status 2>&1 | head -1)"; echo "motd: $(cat /etc/motd)"; echo "machine-id: $(cat /etc/machine-id)"; echo "ssm agent: $(systemctl is-enabled amazon-ssm-agent 2>&1) / $(systemctl is-active amazon-ssm-agent 2>&1)"; echo "harness-endpoint: $(stat -c "%U:%G %a" /etc/carlos/harness-endpoint) $(cat /etc/carlos/harness-endpoint)"' |
| + echo "-- shims in cloud-init-output.log:"; incus exec $b -- grep -h "harness shim" /var/log/cloud-init-output.log |
| + echo "-- cloud-init errors:"; incus exec $b -- sh -c 'grep -ci "error\|Traceback" /var/log/cloud-init.log | sed "s/^/cloud-init.log error-ish lines: /"; cloud-init status --format json 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(\"errors:\", d.get(\"errors\"), \"recoverable:\", list((d.get(\"recoverable_errors\") or {}).keys()))"' |
| + echo "-- layout probe:"; incus exec $b -- sh < /root/harness/layout-probe.sh |
| + echo "-- systemctl --failed:"; incus exec $b -- systemctl --failed --no-legend --plain; echo "(end)" |
| + echo "-- system state: $(incus exec $b -- systemctl is-system-running 2>&1)" |
| +done |
| +echo "================ signer" |
| +incus exec signer -- sh -c 'echo "cloud-init: $(cloud-init status | head -1)"; command -v jq aws openssl; echo "endpoint: $(cat /etc/carlos/harness-endpoint)"; systemctl --failed --no-legend --plain; echo "(failed end)"' |
| +incus list -f csv -c ns4 |
diff --git a/harness/checks/check6.sh b/harness/checks/check6.sh| new file mode 100644 |
| index 000000000..89e2f01c7 |
| --- /dev/null |
| +++ b/harness/checks/check6.sh |
| @@ -0,0 +1,37 @@ |
| +set -u |
| +T0=$(date -u +%s) |
| +echo "== waiting for status/boxes/ in MinIO (bootstrap finished ~01:03:25Z)" |
| +for i in $(seq 1 36); do |
| + n=$(runuser -l operator -c 'aws s3 ls s3://harness-deployment/status/boxes/ 2>/dev/null' | grep -c json) |
| + [ "$n" -ge 3 ] && break; sleep 5 |
| +done |
| +echo "after $(( $(date -u +%s) - T0 ))s at $(date -u +%TZ):" |
| +runuser -l operator -c 'aws s3 ls --recursive s3://harness-deployment/status/boxes/' |
| +for b in leo sophie dennis; do |
| + echo "================ $b" |
| + incus exec $b -- sh -c ' |
| + echo "carlos-edge: $(systemctl is-active carlos-edge)" |
| + systemctl show -p Id,ActiveState,Result,NRestarts carlos-edge data-instances.mount run-carlos-instances.mount run-carlos-logs.mount | paste -sd" " | sed "s/ Id=/\nId=/g" |
| + echo "failed: [$(systemctl --failed --no-legend --plain)] state: $(systemctl is-system-running)" |
| + echo "mounts:"; findmnt -no TARGET,FSTYPE,OPTIONS /data/instances /run/carlos/instances /run/carlos/logs 2>&1 |
| + echo "journal (this boot) hits:" |
| + for pat in "Failed to set up mount namespacing" "IPAddressDeny" "BPF" "not supported" "AccessDenied" "panic:" "amazonaws.com" "Permission denied"; do |
| + c=$(journalctl -b --no-pager -o cat | grep -c -- "$pat"); echo " [$c] $pat" |
| + done |
| + echo "IPAccounting/IPAddressDeny on carlos-edge: $(systemctl show -p IPAddressDeny carlos-edge | cut -c1-120)" |
| + stat -c "%n %U:%G %a" /opt/carlos /opt/carlos/carlos /etc/carlos/host.env |
| + grep -v PUBKEY /etc/carlos/host.env | tr "\n" " "; echo |
| + ' |
| + echo "-- box label/report:"; incus exec $b -- sh -c 'journalctl -u carlos-edge --no-pager -o cat | grep -iE "status|report|publish" | head -3 | cut -c1-200' |
| +done |
| +echo "================ journal lines mentioning IPAddressDeny / bpf on leo" |
| +incus exec leo -- sh -c 'journalctl -b --no-pager -o cat | grep -iE "IPAddress|bpf" | sort | uniq -c | head -5' |
| +echo "================ truncation: 30,000-byte stdout through the stand-in" |
| +L=$(python3 -c 'import json; print([k for k,v in json.load(open("/etc/harness/ssm.json"))["instances"].items() if v=="leo"][0])') |
| +runuser -l operator -c " |
| +CID=\$(aws ssm send-command --instance-ids $L --document-name AWS-RunShellScript --parameters '{\"commands\":[\"head -c 30000 /dev/zero | tr \\\\\\\\0 x; head -c 9000 /dev/zero | tr \\\\\\\\0 y >&2\"]}' --query Command.CommandId --output text) |
| +for i in 1 2 3 4 5 6 7 8 9 10; do S=\$(aws ssm get-command-invocation --command-id \$CID --instance-id $L --query Status --output text); [ \$S = Success ] && break; sleep 1; done |
| +aws ssm get-command-invocation --command-id \$CID --instance-id $L --output json | python3 -c 'import json,sys; d=json.load(sys.stdin); print(\"status\", d[\"Status\"], \"stdout chars\", len(d[\"StandardOutputContent\"]), \"stderr chars\", len(d[\"StandardErrorContent\"]))' |
| +ls -l /dev/null >/dev/null; echo cid=\$CID |
| +" |
| +ls -l /var/log/harness/ssm-commands/*/ | tail -4 |
diff --git a/harness/checks/check6b.sh b/harness/checks/check6b.sh| new file mode 100644 |
| index 000000000..a754019d0 |
| --- /dev/null |
| +++ b/harness/checks/check6b.sh |
| @@ -0,0 +1,27 @@ |
| +set -u |
| +for b in leo sophie dennis; do |
| + echo "================ $b machine-id $(incus exec $b -- cat /etc/machine-id)" |
| + incus exec $b -- sh -c ' |
| + for u in data-instances.mount run-carlos-instances.mount run-carlos-logs.mount; do |
| + echo "$u: $(systemctl show -p ActiveState,Result,What,Where,Options $u | paste -sd" ")" |
| + done |
| + grep -E "^(/data|/run/carlos)" /proc/self/mountinfo >/dev/null; awk "\$5 ~ /^\/(data|run\/carlos)/ {print \" mountinfo:\", \$5, \$6}" /proc/self/mountinfo |
| + # cgroup BPF inside the nested container: a transient unit with IPAddressDeny=any must not reach MinIO. |
| + printf "IPAddressDeny=any -> MinIO: " |
| + systemd-run --quiet --wait --pipe -p IPAddressDeny=any -p IPAddressAllow=localhost /usr/bin/python3 -c " |
| +import socket; s=socket.socket(); s.settimeout(3) |
| +try: s.connect((\"203.0.113.10\",9000)); print(\"CONNECTED (deny NOT enforced)\") |
| +except Exception as e: print(\"blocked:\", type(e).__name__, e)" |
| + printf "no deny -> MinIO: " |
| + systemd-run --quiet --wait --pipe /usr/bin/python3 -c " |
| +import socket; s=socket.socket(); s.settimeout(3) |
| +try: s.connect((\"203.0.113.10\",9000)); print(\"CONNECTED\") |
| +except Exception as e: print(\"blocked:\", type(e).__name__, e)" |
| + echo "instance unit deny list includes 203.0.113? $(systemctl cat carlos-instance@.service 2>/dev/null | grep -m1 IPAddressDeny | grep -c 203.0.113) (0 = bridge allowed)" |
| + echo "units: $(systemctl list-units "carlos*" --no-legend --plain | awk "{print \$1\"=\"\$3\"/\"\$4}" | paste -sd" ")" |
| + ' |
| +done |
| +echo "================ status/boxes labels -> box" |
| +for k in ada-letterpress kathleen-juggling noor-marquetry; do |
| + runuser -l operator -c "aws s3 cp --quiet s3://harness-deployment/status/boxes/$k.json -" | python3 -c "import json,sys; d=json.load(sys.stdin); print('$k', {x: d.get(x) for x in ('instance_id','instance','region','public_ip','ip','version','build','addr') if x in d}, sorted(d)[:14])" |
| +done |
diff --git a/harness/checks/fix5.sh b/harness/checks/fix5.sh| new file mode 100644 |
| index 000000000..17758f742 |
| --- /dev/null |
| +++ b/harness/checks/fix5.sh |
| @@ -0,0 +1,6 @@ |
| +incus exec dennis -- journalctl -u amazon-ssm-agent --no-pager -n 6 -o cat | cut -c1-200 |
| +for b in leo sophie dennis; do incus exec $b -- systemctl reset-failed amazon-ssm-agent.service 2>/dev/null; echo "$b: failed=[$(incus exec $b -- systemctl --failed --no-legend --plain)] state=$(incus exec $b -- systemctl is-system-running)"; done |
| +for b in leo sophie dennis; do echo "$b zram unit: $(incus exec $b -- systemctl show -p ActiveState,Result,FragmentPath systemd-zram-setup@zram0.service | tr '\n' ' ')"; done |
| +incus exec leo -- journalctl -u systemd-zram-setup@zram0 --no-pager -o cat | tail -2 |
| +incus exec signer -- sh -c 'for t in jq aws openssl curl; do printf "%s: " $t; command -v $t || echo MISSING; done; aws --version' |
| +incus exec leo -- sh -c 'aws --version' |
diff --git a/harness/checks/run6b.sh b/harness/checks/run6b.sh| new file mode 100644 |
| index 000000000..06e916d39 |
| --- /dev/null |
| +++ b/harness/checks/run6b.sh |
| @@ -0,0 +1,9 @@ |
| +set -u |
| +id_of() { python3 -c "import json,sys; print([k for k,v in json.load(open('/etc/harness/ssm.json'))['instances'].items() if v=='$1'][0])"; } |
| +L=$(id_of leo); S=$(id_of sophie); D=$(id_of dennis) |
| +for spec in "leo $L harness-data-ie eu-west-1 " "sophie $S harness-data-ie eu-west-1 " "dennis $D harness-data-uk eu-west-2 uk"; do |
| + set -- $spec |
| + echo "================ $1: bootstrap-box.sh harness $2 harness-deployment $3 ~/release.pub $4 ${5:-} ($(date -u +%TZ))" |
| + runuser -l operator -c "~/infra/scripts/bootstrap-box.sh harness $2 harness-deployment $3 ~/release.pub $4 ${5:-}" |
| + echo "exit=$? ($(date -u +%TZ))" |
| +done |
diff --git a/harness/checks/snap-base.sh b/harness/checks/snap-base.sh| new file mode 100644 |
| index 000000000..343c11d9b |
| --- /dev/null |
| +++ b/harness/checks/snap-base.sh |
| @@ -0,0 +1,18 @@ |
| +set -eu |
| +ALL="infra leo sophie dennis signer" |
| +incus stop $ALL |
| +for c in $ALL; do incus snapshot create $c base; done |
| +incus start infra; sleep 3; incus start leo sophie dennis signer |
| +# Provisioning is over: boxes reach nothing beyond 203.0.113.0/24 (spec §3.4). |
| +nft flush set inet harness egress_ports |
| +touch /etc/harness/egress-closed |
| +for c in $ALL; do for i in $(seq 1 60); do incus exec $c -- systemctl is-system-running --wait >/dev/null 2>&1 && break; sleep 2; done; done |
| +for c in $ALL; do echo "$c: $(incus snapshot list $c -f csv -c n | tr '\n' ' ') state=$(incus exec $c -- systemctl is-system-running) failed=[$(incus exec $c -- systemctl --failed --no-legend --plain)]"; done |
| +echo "egress set: $(nft list set inet harness egress_ports | grep -c elements) element lines (0 = closed)" |
| +incus exec leo -- python3 -c ' |
| +import socket |
| +for h,p in (("deb.debian.org",443),("203.0.113.10",9000)): |
| + s=socket.socket(); s.settimeout(4) |
| + try: s.connect((h,p)); print(h,p,"CONNECTED") |
| + except Exception as e: print(h,p,type(e).__name__) |
| + s.close()' |
diff --git a/harness/checks/snap-boot.sh b/harness/checks/snap-boot.sh| new file mode 100644 |
| index 000000000..85ce9c04d |
| --- /dev/null |
| +++ b/harness/checks/snap-boot.sh |
| @@ -0,0 +1,11 @@ |
| +set -eu |
| +ALL="infra leo sophie dennis signer" |
| +for b in leo sophie dennis; do echo "$b label: $(incus exec $b -- sh -c 'journalctl -u carlos-edge --no-pager -o cat | grep -oE "(ada-letterpress|kathleen-juggling|noor-marquetry)" | sort -u | head -1')"; done |
| +incus stop $ALL |
| +for c in $ALL; do incus snapshot create $c boot; done |
| +T0=$(date -u +%s) |
| +incus start infra; sleep 3; incus start leo sophie dennis signer |
| +for c in $ALL; do for i in $(seq 1 60); do incus exec $c -- systemctl is-system-running --wait >/dev/null 2>&1 && break; sleep 2; done; done |
| +for c in $ALL; do echo "$c: snapshots=[$(incus snapshot list $c -f csv -c n | paste -sd,)] state=$(incus exec $c -- systemctl is-system-running) failed=[$(incus exec $c -- systemctl --failed --no-legend --plain)]"; done |
| +for b in leo sophie dennis; do echo "$b carlos-edge: $(incus exec $b -- systemctl show -p ActiveState,Result,NRestarts carlos-edge | paste -sd' ')"; done |
| +echo "restart+checks took $(( $(date -u +%s) - T0 ))s" |
diff --git a/harness/checks/ssmsmoke.sh b/harness/checks/ssmsmoke.sh| new file mode 100644 |
| index 000000000..d8d75f009 |
| --- /dev/null |
| +++ b/harness/checks/ssmsmoke.sh |
| @@ -0,0 +1,18 @@ |
| +runuser -l operator -c ' |
| +set -e |
| +L=$(python3 -c "import json; print([k for k,v in json.load(open(\"/dev/stdin\"))[\"instances\"].items() if v==\"leo\"][0])" < /etc/harness/ssm.json 2>/dev/null || true) |
| +' 2>/dev/null |
| +L=$(python3 -c 'import json; print([k for k,v in json.load(open("/etc/harness/ssm.json"))["instances"].items() if v=="leo"][0])') |
| +runuser -l operator -c " |
| +set -e |
| +CID=\$(aws ssm send-command --instance-ids $L --document-name AWS-RunShellScript --parameters '{\"commands\":[\"id -un; umask; pwd; env | sort\", \"echo HOME=[\$HOME]\"]}' --query Command.CommandId --output text) |
| +echo cid=\$CID |
| +for i in 1 2 3 4 5 6 7 8 9 10; do S=\$(aws ssm get-command-invocation --command-id \$CID --instance-id $L --query Status --output text); [ \$S = Success ] || [ \$S = Failed ] && break; sleep 1; done |
| +aws ssm get-command-invocation --command-id \$CID --instance-id $L --query '[Status,ResponseCode,StandardOutputContent]' --output text |
| +echo '-- bad instance:'; aws ssm send-command --instance-ids i-0123456789abcdef0 --document-name AWS-RunShellScript --parameters '{\"commands\":[\"true\"]}' 2>&1 | tail -1 |
| +echo '-- parameter store:'; aws ssm put-parameter --name /harness/smoke --type SecureString --value hello --query Version --output text |
| +aws ssm get-parameter --name /harness/smoke --with-decryption --query Parameter.Value --output text |
| +aws ssm put-parameter --name /harness/smoke --type SecureString --value again 2>&1 | tail -1 |
| +aws ssm delete-parameter --name /harness/smoke && aws ssm get-parameter --name /harness/smoke 2>&1 | tail -1 |
| +" |
| +tail -4 /var/log/harness/ssm.jsonl |
diff --git a/harness/operator/operator-setup.sh b/harness/operator/operator-setup.sh| new file mode 100644 |
| index 000000000..9b5096285 |
| --- /dev/null |
| +++ b/harness/operator/operator-setup.sh |
| @@ -0,0 +1,101 @@ |
| +#!/bin/bash |
| +# Step 6 (host side): the operator shell (B3), the test release key, the SSM |
| +# stand-in, and the bootstrap object. Run as root on the harness host. |
| +# |
| +# CARLOS_URL=<presigned> CONSOLE_URL=<presigned> SUMS="<sha256 carlos> <sha256 console>" operator-setup.sh |
| +set -euo pipefail |
| +export DEBIAN_FRONTEND=noninteractive |
| +H=/root/harness |
| + |
| +command -v aws >/dev/null || apt-get install -qy awscli >/dev/null |
| + |
| +# The harness build (-tags harness), checked by sha256. |
| +curl -sf -o $H/bin/carlos.part "$CARLOS_URL" |
| +curl -sf -o $H/bin/console.part "$CONSOLE_URL" |
| +set -- $SUMS |
| +[ "$(sha256sum < $H/bin/carlos.part | cut -d' ' -f1)" = "$1" ] || { echo "carlos sha256 mismatch" >&2; exit 1; } |
| +[ "$(sha256sum < $H/bin/console.part | cut -d' ' -f1)" = "$2" ] || { echo "console sha256 mismatch" >&2; exit 1; } |
| +chmod 755 $H/bin/carlos.part $H/bin/console.part |
| +mv $H/bin/carlos.part $H/bin/carlos; mv $H/bin/console.part $H/bin/console |
| +install -m 0755 $H/bin/carlos /usr/local/bin/carlos |
| + |
| +# The harness build's endpoint fallback, for the operator's own CLI. |
| +install -d -m 0755 /etc/carlos |
| +printf 'http://203.0.113.10:9000\n' > /etc/carlos/harness-endpoint |
| +chmod 0644 /etc/carlos/harness-endpoint |
| + |
| +# The test ROOT release key (H5): generated here, on tmpfs, for the host's |
| +# lifetime. Only its public half leaves /run. |
| +install -d -m 0700 /run/harness/keys |
| +if [ ! -s /run/harness/keys/release.env ]; then |
| + (umask 077; /usr/local/bin/carlos release-keygen > /run/harness/keys/release.env 2>/dev/null) |
| +fi |
| +sed -n 's/^CARLOS_RELEASE_PUBKEY=//p' /run/harness/keys/release.env > $H/release.pub |
| +chmod 0644 $H/release.pub |
| + |
| +# The operator: unprivileged, the harness CLI, awscli, jq, python3. Its AWS |
| +# environment is exactly the two endpoints, its MinIO keys and the region. |
| +# Debian ships a system group "operator" (gid 37, no privileges); the user joins it. |
| +id operator >/dev/null 2>&1 || useradd -m -s /bin/bash -g operator operator |
| +OH=/home/operator |
| +install -d -o operator -g operator -m 0700 $OH/.aws $OH/infra $OH/infra/scripts |
| +python3 - <<'PY' |
| +import json |
| +u = json.load(open("/root/harness/minio-users.json"))["operator"] |
| +creds = "".join(f"[{p}]\naws_access_key_id = {u['access_key']}\naws_secret_access_key = {u['secret_key']}\n\n" for p in ("default", "harness")) |
| +open("/home/operator/.aws/credentials", "w").write(creds) |
| +open("/home/operator/.aws/config", "w").write("[default]\nregion = eu-west-1\n\n[profile harness]\nregion = eu-west-1\n") |
| +PY |
| +cat > $OH/.harness-env <<'EOF' |
| +# The operator's AWS environment in the harness (spec §3.9): nothing else. |
| +export AWS_ENDPOINT_URL_SSM=http://203.0.113.1:8100 |
| +export AWS_ENDPOINT_URL_S3=http://203.0.113.10:9000 |
| +export AWS_REGION=eu-west-1 |
| +export AWS_PROFILE=harness |
| +EOF |
| +grep -q harness-env $OH/.profile 2>/dev/null || echo '. "$HOME/.harness-env"' >> $OH/.profile |
| +grep -q harness-env $OH/.bashrc 2>/dev/null || echo '. "$HOME/.harness-env"' >> $OH/.bashrc |
| +cp $H/release.pub $OH/release.pub |
| +for f in bootstrap-box.sh update-platform-binary.sh; do install -m 0755 $H/infra-copies/$f $OH/infra/scripts/$f; done |
| +chown -R operator:operator $OH/.aws $OH/.harness-env $OH/release.pub $OH/infra |
| +chmod 0600 $OH/.aws/credentials |
| + |
| +# The SSM stand-in: instance ids (the donors' for leo and sophie; harness-only |
| +# ids for dennis and the signer, which have no donor) mapped to containers. |
| +install -d -m 0700 /var/lib/harness /var/log/harness/ssm-commands |
| +D1=$(python3 -c 'import json; print(json.load(open("/etc/harness/donors/donor-1/document"))["instanceId"])') |
| +D2=$(python3 -c 'import json; print(json.load(open("/etc/harness/donors/donor-2/document"))["instanceId"])') |
| +cat > /etc/harness/ssm.json <<EOF |
| +{"instances": {"$D1": "leo", "$D2": "sophie", "i-00000000000000023": "dennis", "i-00000000000000030": "signer"}} |
| +EOF |
| +cat > /etc/systemd/system/harness-ssm.service <<'EOF' |
| +[Unit] |
| +Description=harness: SSM stand-in (203.0.113.1:8100) |
| +After=incus.service harness-guard.service |
| +Wants=incus.service |
| + |
| +[Service] |
| +ExecStart=/usr/bin/python3 /opt/harness/ssm.py |
| +Restart=always |
| +RestartSec=2 |
| + |
| +[Install] |
| +WantedBy=multi-user.target |
| +EOF |
| +systemctl daemon-reload |
| +systemctl enable harness-ssm.service >/dev/null 2>&1 |
| +systemctl restart harness-ssm.service |
| +for i in $(seq 1 20); do ss -ltn | grep -q '203.0.113.1:8100' && break; sleep 0.5; done |
| + |
| +# Seeded for step 6: the release recipe puts the box binary at |
| +# <deployment bucket>/bootstrap/carlos; here the operator does, from the harness build. |
| +runuser -l operator -c "aws s3 cp --no-progress /usr/local/bin/carlos s3://harness-deployment/bootstrap/carlos" >/dev/null |
| +echo "== operator-setup" |
| +echo "instances: $(cat /etc/harness/ssm.json)" |
| +echo "release pubkey: $(cut -c1-8 $H/release.pub)... sha256:$(sha256sum < $H/release.pub | cut -c1-12)" |
| +echo "release.env: $(stat -c '%U:%G %a' /run/harness/keys/release.env) on $(findmnt -no FSTYPE --target /run/harness/keys)" |
| +echo "carlos: $(/usr/local/bin/carlos version 2>&1 | head -1)" |
| +echo "harness info (arm64): $(CARLOS_HARNESS_INFO=1 /usr/local/bin/carlos | tr '\n' ' ')" |
| +systemctl is-active harness-ssm harness-imds |
| +ss -ltnp | grep -E '8100|8169' |
| +runuser -l operator -c 'env | grep -E "^AWS_" | sort; aws s3 ls s3://harness-deployment/bootstrap/' |
diff --git a/harness/stand-ins/ssm.py b/harness/stand-ins/ssm.py| new file mode 100644 |
| index 000000000..edd436d04 |
| --- /dev/null |
| +++ b/harness/stand-ins/ssm.py |
| @@ -0,0 +1,310 @@ |
| +#!/usr/bin/env python3 |
| +"""SSM stand-in (spec §3.9). Python 3 standard library only. Runs as root on |
| +the harness host (it needs the incus socket), bound to 203.0.113.1:8100. |
| + |
| +The aws CLI and SDKs reach it through AWS_ENDPOINT_URL_SSM. It speaks the |
| +JSON 1.1 protocol (X-Amz-Target: AmazonSSM.<Op>) and ignores signatures. |
| + |
| +Run Command: SendCommand (AWS-RunShellScript), GetCommandInvocation. |
| +Parameter Store: GetParameter, GetParameters, PutParameter, DeleteParameter, |
| +DescribeParameters. |
| + |
| +A command runs as root in its container through `incus exec`, with a cleared |
| +environment: PATH as the agent's unit has it, no HOME (the real agent sets |
| +none), plus the three variables the harness declares (B4): |
| +AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL_SSM, AWS_REGION=eu-west-1. It runs |
| +under /bin/sh with umask 022, in a per-command directory shaped like the |
| +agent's (/var/lib/amazon/ssm/<iid>/document/orchestration/<cid>/...). |
| +Results keep the real limits: StandardOutputContent cut at 24,000 characters, |
| +StandardErrorContent at 8,000; Failed on a non-zero exit; TimedOut at the |
| +execution timeout. |
| + |
| +Config /etc/harness/ssm.json: {"instances": {"<instance-id>": "<container>"}}. |
| +Logs: /var/log/harness/ssm.jsonl (one line per API call: op, ids, parameter |
| +NAMES, never values) and /var/log/harness/ssm-commands/<cid>/<iid>.{sh,out,err,rc} |
| +(full transcripts, untruncated). |
| +""" |
| +import datetime, http.server, json, os, subprocess, sys, threading, time, uuid |
| + |
| +CONFIG = os.environ.get("SSM_CONFIG", "/etc/harness/ssm.json") |
| +PARAMS = os.environ.get("SSM_PARAMS", "/var/lib/harness/ssm-params.json") |
| +LOG = os.environ.get("SSM_LOG", "/var/log/harness/ssm.jsonl") |
| +CMDLOG = os.environ.get("SSM_CMDLOG", "/var/log/harness/ssm-commands") |
| +BIND = (os.environ.get("SSM_ADDR", "203.0.113.1"), int(os.environ.get("SSM_PORT", "8100"))) |
| +STDOUT_LIMIT, STDERR_LIMIT = 24000, 8000 |
| +AGENT_PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" |
| +INJECT = {"AWS_ENDPOINT_URL_S3": "http://203.0.113.10:9000", |
| + "AWS_ENDPOINT_URL_SSM": "http://203.0.113.1:8100", |
| + "AWS_REGION": "eu-west-1"} |
| +ACCOUNT, REGION = "640012953448", "eu-west-1" |
| + |
| +lock = threading.Lock() |
| +invocations = {} # (cid, iid) -> dict |
| + |
| + |
| +class APIError(Exception): |
| + def __init__(self, typ, msg, status=400): |
| + super().__init__(msg) |
| + self.typ, self.msg, self.status = typ, msg, status |
| + |
| + |
| +def now_epoch(): |
| + return round(time.time(), 3) |
| + |
| + |
| +def log(rec): |
| + rec["t"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) |
| + try: |
| + with open(LOG, "a") as f: |
| + f.write(json.dumps(rec) + "\n") |
| + except OSError: |
| + pass |
| + |
| + |
| +def instances(): |
| + with open(CONFIG) as f: |
| + return json.load(f)["instances"] |
| + |
| + |
| +# ---- Run Command ------------------------------------------------------------- |
| + |
| +def run_invocation(cid, iid, container, commands, timeout): |
| + key = (cid, iid) |
| + wd = f"/var/lib/amazon/ssm/{iid}/document/orchestration/{cid}/awsrunShellScript/0.awsrunShellScript" |
| + script = "\n".join(commands) + "\n" |
| + os.makedirs(os.path.join(CMDLOG, cid), exist_ok=True) |
| + base = os.path.join(CMDLOG, cid, iid) |
| + with open(base + ".sh", "w") as f: |
| + f.write(script) |
| + with lock: |
| + inv = invocations[key] |
| + inv["Status"] = inv["StatusDetails"] = "InProgress" |
| + inv["ExecutionStartDateTime"] = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")[:-4] + "Z" |
| + try: |
| + # Write the script into the container, then run it under a cleared env. |
| + subprocess.run(["incus", "exec", container, "--", "mkdir", "-p", wd], check=True, capture_output=True, timeout=60) |
| + subprocess.run(["incus", "file", "push", base + ".sh", f"{container}{wd}/_script.sh", "--mode", "0700", "-q"], |
| + check=True, capture_output=True, timeout=60) |
| + env = ["env", "-i", f"PATH={AGENT_PATH}"] + [f"{k}={v}" for k, v in INJECT.items()] |
| + argv = ["incus", "exec", container, "--cwd", wd, "--", *env, |
| + "/bin/sh", "-c", "umask 022; exec /bin/sh _script.sh"] |
| + try: |
| + p = subprocess.run(argv, capture_output=True, timeout=timeout) |
| + out, err, rc, status = p.stdout, p.stderr, p.returncode, ("Success" if p.returncode == 0 else "Failed") |
| + except subprocess.TimeoutExpired as e: |
| + out, err, rc, status = e.stdout or b"", e.stderr or b"", -1, "TimedOut" |
| + subprocess.run(["incus", "exec", container, "--", "pkill", "-f", f"{wd}"], capture_output=True) |
| + except Exception as e: # the stand-in's own failure, reported as the agent would: Failed |
| + out, err, rc, status = b"", f"harness ssm stand-in: {e}".encode(), 1, "Failed" |
| + for ext, data in (("out", out), ("err", err), ("rc", str(rc).encode())): |
| + with open(f"{base}.{ext}", "wb") as f: |
| + f.write(data) |
| + so, se = out.decode("utf-8", "replace"), err.decode("utf-8", "replace") |
| + with lock: |
| + inv = invocations[key] |
| + inv.update({"Status": status, "StatusDetails": status, "ResponseCode": rc, |
| + "StandardOutputContent": so[:STDOUT_LIMIT], "StandardErrorContent": se[:STDERR_LIMIT], |
| + "ExecutionEndDateTime": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")[:-4] + "Z"}) |
| + log({"op": "CommandFinished", "cid": cid, "iid": iid, "container": container, "status": status, "rc": rc, |
| + "stdout_bytes": len(out), "stderr_bytes": len(err)}) |
| + |
| + |
| +def send_command(req): |
| + doc = req.get("DocumentName") |
| + if doc != "AWS-RunShellScript": |
| + raise APIError("InvalidDocument", f"harness stand-in serves AWS-RunShellScript only, not {doc}") |
| + ids = req.get("InstanceIds") or [] |
| + if not ids or req.get("Targets"): |
| + raise APIError("InvalidInstanceId", "InstanceIds required (Targets are not served)") |
| + inst = instances() |
| + for iid in ids: |
| + if iid not in inst: |
| + raise APIError("InvalidInstanceId", f"Instances [[{iid}]] not in a valid state for account {ACCOUNT}") |
| + params = req.get("Parameters") or {} |
| + commands = params.get("commands") or [] |
| + if not commands: |
| + raise APIError("InvalidParameters", "commands is required") |
| + timeout = int((params.get("executionTimeout") or ["3600"])[0]) |
| + cid = str(uuid.uuid4()) |
| + t = now_epoch() |
| + for iid in ids: |
| + with lock: |
| + invocations[(cid, iid)] = {"CommandId": cid, "InstanceId": iid, "Comment": req.get("Comment", ""), |
| + "DocumentName": doc, "DocumentVersion": "$DEFAULT", |
| + "PluginName": "aws:runShellScript", "ResponseCode": -1, |
| + "Status": "Pending", "StatusDetails": "Pending", |
| + "StandardOutputContent": "", "StandardErrorContent": "", |
| + "StandardOutputUrl": "", "StandardErrorUrl": "", |
| + "CloudWatchOutputConfig": {"CloudWatchLogGroupName": "", "CloudWatchOutputEnabled": False}} |
| + threading.Thread(target=run_invocation, args=(cid, iid, inst[iid], commands, timeout), daemon=True).start() |
| + log({"op": "SendCommand", "cid": cid, "iids": ids, "containers": [inst[i] for i in ids], |
| + "lines": len(commands), "timeout": timeout}) |
| + return {"Command": {"CommandId": cid, "DocumentName": doc, "DocumentVersion": "$DEFAULT", |
| + "Comment": req.get("Comment", ""), "InstanceIds": ids, "Targets": [], |
| + "Parameters": params, "Status": "Pending", "StatusDetails": "Pending", |
| + "RequestedDateTime": t, "ExpiresAfter": t + timeout + 60, |
| + "TargetCount": len(ids), "CompletedCount": 0, "ErrorCount": 0, |
| + "TimeoutSeconds": req.get("TimeoutSeconds", timeout)}} |
| + |
| + |
| +def get_command_invocation(req): |
| + key = (req.get("CommandId"), req.get("InstanceId")) |
| + with lock: |
| + inv = invocations.get(key) |
| + inv = dict(inv) if inv else None |
| + if inv is None: |
| + raise APIError("InvocationDoesNotExist", "") |
| + log({"op": "GetCommandInvocation", "cid": key[0], "iid": key[1], "status": inv["Status"]}) |
| + return inv |
| + |
| + |
| +# ---- Parameter Store ----------------------------------------------------------- |
| + |
| +def load_params(): |
| + try: |
| + with open(PARAMS) as f: |
| + return json.load(f) |
| + except FileNotFoundError: |
| + return {} |
| + |
| + |
| +def save_params(p): |
| + os.makedirs(os.path.dirname(PARAMS), exist_ok=True) |
| + tmp = PARAMS + ".tmp" |
| + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) |
| + with os.fdopen(fd, "w") as f: |
| + json.dump(p, f) |
| + os.replace(tmp, PARAMS) |
| + |
| + |
| +def arn(name): |
| + return f"arn:aws:ssm:{REGION}:{ACCOUNT}:parameter{name if name.startswith('/') else '/' + name}" |
| + |
| + |
| +def shape(name, p, value=True): |
| + out = {"Name": name, "Type": p["Type"], "Version": p["Version"], "LastModifiedDate": p["LastModifiedDate"], |
| + "ARN": arn(name), "DataType": "text"} |
| + if value: |
| + out["Value"] = p["Value"] |
| + return out |
| + |
| + |
| +def get_parameter(req): |
| + name = req.get("Name", "") |
| + with lock: |
| + p = load_params().get(name) |
| + log({"op": "GetParameter", "name": name, "decrypt": bool(req.get("WithDecryption")), "found": p is not None}) |
| + if p is None: |
| + raise APIError("ParameterNotFound", "") |
| + return {"Parameter": shape(name, p)} |
| + |
| + |
| +def get_parameters(req): |
| + names = req.get("Names") or [] |
| + with lock: |
| + ps = load_params() |
| + log({"op": "GetParameters", "names": names, "decrypt": bool(req.get("WithDecryption"))}) |
| + return {"Parameters": [shape(n, ps[n]) for n in names if n in ps], "InvalidParameters": [n for n in names if n not in ps]} |
| + |
| + |
| +def put_parameter(req): |
| + name, value = req.get("Name", ""), req.get("Value") |
| + if not name or value is None: |
| + raise APIError("ValidationException", "Name and Value are required") |
| + with lock: |
| + ps = load_params() |
| + old = ps.get(name) |
| + if old and not req.get("Overwrite"): |
| + log({"op": "PutParameter", "name": name, "refused": "exists"}) |
| + raise APIError("ParameterAlreadyExists", "The parameter already exists. To overwrite this value, set the overwrite option in the request to true.") |
| + ver = (old["Version"] + 1) if old else 1 |
| + ps[name] = {"Type": req.get("Type") or (old or {}).get("Type", "String"), "Value": value, "Version": ver, |
| + "LastModifiedDate": now_epoch()} |
| + save_params(ps) |
| + log({"op": "PutParameter", "name": name, "type": ps[name]["Type"], "version": ver}) |
| + return {"Version": ver, "Tier": "Standard"} |
| + |
| + |
| +def delete_parameter(req): |
| + name = req.get("Name", "") |
| + with lock: |
| + ps = load_params() |
| + if name not in ps: |
| + log({"op": "DeleteParameter", "name": name, "found": False}) |
| + raise APIError("ParameterNotFound", "") |
| + del ps[name] |
| + save_params(ps) |
| + log({"op": "DeleteParameter", "name": name}) |
| + return {} |
| + |
| + |
| +def describe_parameters(req): |
| + with lock: |
| + ps = load_params() |
| + names = sorted(ps) |
| + for f in req.get("ParameterFilters") or []: |
| + key, opt, vals = f.get("Key"), f.get("Option", "Equals"), f.get("Values") or [] |
| + if key == "Name": |
| + if opt == "BeginsWith": |
| + names = [n for n in names if any(n.startswith(v) for v in vals)] |
| + else: |
| + names = [n for n in names if n in vals] |
| + elif key == "Path": |
| + names = [n for n in names if any(n.startswith(v.rstrip("/") + "/") for v in vals)] |
| + for f in req.get("Filters") or []: |
| + if f.get("Key") == "Name": |
| + names = [n for n in names if any(n.startswith(v) for v in f.get("Values") or [])] |
| + log({"op": "DescribeParameters", "count": len(names)}) |
| + return {"Parameters": [shape(n, ps[n], value=False) for n in names]} |
| + |
| + |
| +OPS = {"SendCommand": send_command, "GetCommandInvocation": get_command_invocation, |
| + "GetParameter": get_parameter, "GetParameters": get_parameters, "PutParameter": put_parameter, |
| + "DeleteParameter": delete_parameter, "DescribeParameters": describe_parameters} |
| + |
| + |
| +class H(http.server.BaseHTTPRequestHandler): |
| + protocol_version = "HTTP/1.1" |
| + |
| + def send_json(self, status, obj): |
| + body = json.dumps(obj).encode() |
| + self.send_response(status) |
| + self.send_header("Content-Type", "application/x-amz-json-1.1") |
| + self.send_header("Content-Length", str(len(body))) |
| + self.send_header("x-amzn-RequestId", str(uuid.uuid4())) |
| + self.end_headers() |
| + self.wfile.write(body) |
| + |
| + def do_POST(self): |
| + n = int(self.headers.get("Content-Length") or 0) |
| + raw = self.rfile.read(n) if n else b"{}" |
| + target = self.headers.get("X-Amz-Target", "") |
| + op = target.split(".", 1)[1] if target.startswith("AmazonSSM.") else "" |
| + fn = OPS.get(op) |
| + if fn is None: |
| + log({"op": op or target, "refused": "not served"}) |
| + return self.send_json(400, {"__type": "UnknownOperationException", "message": f"harness stand-in does not serve {target}"}) |
| + try: |
| + req = json.loads(raw or b"{}") |
| + return self.send_json(200, fn(req)) |
| + except APIError as e: |
| + return self.send_json(e.status, {"__type": e.typ, "message": e.msg}) |
| + except Exception as e: |
| + log({"op": op, "error": repr(e)}) |
| + return self.send_json(500, {"__type": "InternalServerError", "message": str(e)}) |
| + |
| + def log_message(self, *a): |
| + pass |
| + |
| + |
| +def main(): |
| + os.makedirs(os.path.dirname(LOG), exist_ok=True) |
| + os.makedirs(CMDLOG, mode=0o700, exist_ok=True) |
| + instances() |
| + srv = http.server.ThreadingHTTPServer(BIND, H) |
| + print(f"ssm stand-in on {BIND[0]}:{BIND[1]}", flush=True) |
| + srv.serve_forever() |
| + |
| + |
| +if __name__ == "__main__": |
| + sys.exit(main()) |