amadan
design-deployment-harness: spec rev 1, a whole fake deployment for runbook rehearsals
A throwaway dev-account EC2 host running incus system containers built by the real host-init user_data and bootstrap-box.sh, MinIO for the buckets, SSM, IMDS and Keymail stand-ins, test release and sums keys, Pebble and dnsmasq. Box identity from donor instance documents so boxid runs unmodified. IAM checked offline from the rehearsal's action log. Scenario 1 is the claim trial's Option B (steps 1 to 7 through the platform's own commands, hand-seeded items listed); phase 1 is sized for tonight.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+427
−0
docs/superpowers/specs/2026-10-20-deployment-harness-design.md+427 −0
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 318.716s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.680s
ok github.com/carlosframework/platform/cmd/moveprobe 93.323s
ok github.com/carlosframework/platform/cmd/zddprobe 0.706s
ok github.com/carlosframework/platform/internal/activator 251.689s
ok github.com/carlosframework/platform/internal/agentsock 0.019s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.018s
ok github.com/carlosframework/platform/internal/appconfig 2.438s
ok github.com/carlosframework/platform/internal/apperr 0.072s
ok github.com/carlosframework/platform/internal/applog 0.128s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 0.784s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.012s
ok github.com/carlosframework/platform/internal/certwarm 0.007s
ok github.com/carlosframework/platform/internal/cgroup 0.691s
ok github.com/carlosframework/platform/internal/changelog 0.314s
ok github.com/carlosframework/platform/internal/claimactivation 0.768s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 406.711s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.784s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.348s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.004s
ok github.com/carlosframework/platform/internal/edge 45.061s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.130s
ok github.com/carlosframework/platform/internal/gate 1.654s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.016s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/historyreadback 0.149s
ok github.com/carlosframework/platform/internal/host 182.712s
ok github.com/carlosframework/platform/internal/hostactors 6.058s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.013s
ok github.com/carlosframework/platform/internal/hostbox 2.696s
ok github.com/carlosframework/platform/internal/hostclaim 0.542s
ok github.com/carlosframework/platform/internal/hostenv 0.013s
ok github.com/carlosframework/platform/internal/hostevidence 0.013s
ok github.com/carlosframework/platform/internal/hostexec 5.000s
ok github.com/carlosframework/platform/internal/hostfence 0.033s
ok github.com/carlosframework/platform/internal/hostfiles 10.567s
ok github.com/carlosframework/platform/internal/hostfloor 0.013s
ok github.com/carlosframework/platform/internal/hostguard 0.996s
ok github.com/carlosframework/platform/internal/hostops 1.610s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.013s
ok github.com/carlosframework/platform/internal/hostquarantine 0.014s
ok github.com/carlosframework/platform/internal/hostrelocate 0.433s
ok github.com/carlosframework/platform/internal/hostresolve 0.146s
ok github.com/carlosframework/platform/internal/hostwrite 0.013s
ok github.com/carlosframework/platform/internal/infraview 0.014s
ok github.com/carlosframework/platform/internal/instancectl 5.256s
ok github.com/carlosframework/platform/internal/instanceexec 2.763s
ok github.com/carlosframework/platform/internal/instancefs 0.005s
ok github.com/carlosframework/platform/internal/instancekey 0.002s
ok github.com/carlosframework/platform/internal/instanceupdates 0.018s
ok github.com/carlosframework/platform/internal/layout 1.546s
ok github.com/carlosframework/platform/internal/lbhealth 0.002s
ok github.com/carlosframework/platform/internal/lease 9.823s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.005s
ok github.com/carlosframework/platform/internal/ledger 0.011s
ok github.com/carlosframework/platform/internal/mail 0.163s
ok github.com/carlosframework/platform/internal/maintenance 0.093s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.528s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.072s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 21.058s
ok github.com/carlosframework/platform/internal/moves/archive 0.787s
ok github.com/carlosframework/platform/internal/objstore 0.188s
ok github.com/carlosframework/platform/internal/ops 0.180s
ok github.com/carlosframework/platform/internal/park 0.199s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 36.087s
ok github.com/carlosframework/platform/internal/publicrelease 0.010s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.495s
ok github.com/carlosframework/platform/internal/relay 7.280s
ok github.com/carlosframework/platform/internal/release 23.741s
ok github.com/carlosframework/platform/internal/rollout 3.820s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.058s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.561s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.525s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.659s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.643s
ok github.com/carlosframework/platform/internal/status 0.058s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.230s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.563s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 49414306 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.031s
=== RUN TestT4StorageProbeMinio
2026/10/20 00:28:07 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E015507832FA61, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.025s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-20-deployment-harness-design.md b/docs/superpowers/specs/2026-10-20-deployment-harness-design.md| new file mode 100644 |
| index 000000000..177c8aaf4 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-20-deployment-harness-design.md |
| @@ -0,0 +1,427 @@ |
| +# Deployment harness: a whole fake Carlos deployment for rehearsals |
| + |
| +Design, 2026-10-20, **rev 1**. Paul asked: "Can we create a test harness to simulate this that would allow much faster iteration?" He approved a spec for review in the morning, and later the same evening approved spinning the harness up tonight in the dev account, harness only: no writes to the real dev boxes, buckets or signer. Nothing here is built. |
| + |
| +Code is read at main `96db0efe`. The infrastructure is read at `carlosframework/platform-infrastructure` `origin/main` `a0b3f5a` (and its unapplied branch `host-root-owned-opt`). The claim runbook is read at `claim-dev-cohort1-runbook` `8a08fca5` (Option B, delta v2). The offline move plan is read at `moves-offline` `96ea271b`. |
| + |
| +## 1. The problem |
| + |
| +The dev cohort-1 claim trial took days because each defect showed up only on a real box, one stop per attended signing session: |
| + |
| +1. host-init's `chown -R carlos:carlos /data /opt/carlos` left `/opt/carlos` owned by `carlos`, so `guard-arm` refused with `hostfloor: authenticated floor required` (`hostfloor.OpenProtected`, `internal/hostfloor/image_linux.go:16`, needs every parent root-owned). |
| +2. A runbook block's `umask 077` made the guard directories `0700`; `console.service` (User=carlos) could not exec `guard/carlos`, and the dev console went into a 502 loop. |
| +3. The signer's role policy (`signing-host.md`, `ReadControl`) lacked one read, `status/consoles/dev-console.json`, which `publish-floor`'s census reads (`internal/hostactors/census.go:349`). |
| +4. `console-report` one second after a console restart was refused: it takes the console's PID from the console's own report, which still named the old process (`activationConsolePID`, `internal/host/activation_console_report_linux.go:26`). |
| +5. Instance records created with claims off carry no `acq`, so `backfill-collect` refuses them (`claimactivation/backfill.go:95` on the window branch). |
| + |
| +None of these is a logic bug that unit tests catch. The tests use in-process fakes (`leasetest.NewS3Server`, `release.DirStore`, scratch roots). The root runner (`root-runner.py`, a throwaway arm64 EC2 driven over SSM) gave real root and systemd, but its tests build their own scratch trees, so it missed defect 1, and its rehearsal ran the signer phases with other credentials, so it missed defect 3. The defects live in host-init, systemd units, root-owned paths, IAM policy and timing, which is to say in the deployment, not the code. |
| + |
| +The lesson the trial recorded four times: a production-like witness must start from the **infrastructure's** box layout (host-init and bootstrap-box), not only the platform's bootstrap. |
| + |
| +## 2. Goals and non-goals |
| + |
| +### 2.1 Goals |
| + |
| +- A whole deployment (boxes, buckets, console, signer, DNS, TLS) that a runbook or rehearsal runs against end to end, repeatedly, and that resets to a known state in seconds. |
| +- Boxes built by the **real** user_data from the infrastructure repos, then bootstrapped by the **real** `bootstrap-box.sh`, running the same arm64 binaries and the same embedded units as production. |
| +- Runbooks as code whose box commands go through one transport. The same script runs against real boxes over SSM and against the harness. |
| +- IAM checked offline from what the rehearsal actually did, so a missing grant is found before a signing session opens. |
| +- Fault injection for the cases moves and claims care about. |
| +- No production credential or key anywhere in the harness. |
| + |
| +### 2.2 The five defects: would the harness have caught them? |
| + |
| +| # | Defect | Caught? | Why | Phase | |
| +|---|---|---|---|---| |
| +| 1 | `/opt/carlos` carlos-owned | **Yes** | Boxes are provisioned by `dev/host-init.sh` as cloud-init user_data and bootstrapped by `scripts/bootstrap-box.sh`, unmodified. Step 6.2 runs the real `guard-arm`, which calls `OpenProtected` on the real tree and refuses exactly as on leo. | P1 | |
| +| 2 | `umask 077` makes guard dirs `0700`, console 502 | **Yes** | The runbook's blocks run verbatim through the SSM stand-in, which runs them as root under `/bin/sh` with the agent's `umask 022`, so the block's own `umask 077` takes effect as it did on sophie. Every step ends with the harness's after-step check: `console.service` and `carlos-edge.service` active, and the console answering 200 through sophie's edge. | P1 | |
| +| 3 | Signer policy lacks `status/consoles/<id>.json` | **Yes, offline; not by MinIO** | The action log shows the signer principal's `GetObject` on `status/consoles/dev-console.json`. Simulated against the policy in `signing-host.md`, it is `implicitDeny`, and the harness fails the rehearsal before any real session opens. MinIO's own policy engine is not relied on. | P1 by hand, P2 automated | |
| +| 4 | `console-report` one second after restart refused | **Likely** | Same code, same order, same units. The risk is speed: MinIO answers in well under a millisecond where S3 takes tens, which narrows race windows. P2's proxy adds S3-like latency. | P1 | |
| +| 5 | Records without `acq` refused by `backfill-collect` | **Yes, once the harness holds records created claims-off** | Option B's zero-host path deletes all records, so a harness started empty never meets it. The harness meets it when its state includes a claims-off app (step 4 through the console doors, P1 stretch) or a populated fixture (P2); `backfill-collect` then refuses identically. | P1 stretch, P2 | |
| + |
| +A sixth, found in the same trial, is worth naming because the harness changes it: the window package's S2 (the migration census walking leo's real `/data`, `/opt/carlos` and `/etc/systemd/system`). A harness box carries its own history's litter, and P2 can replay a real box's tree listing (`leo-tree-*.tsv`: paths, types, modes, owners, no contents) into a harness box, so the census meets a box-shaped tree. |
| + |
| +### 2.3 What it cannot catch |
| + |
| +- **Real IAM enforcement.** Roles, trust policies, resource policies keyed on `aws:PrincipalArn`, KMS key policies and grants, SCPs, session policies, VPC endpoint policies and IAM's eventual consistency. The offline check (§3.8) covers identity policies and the resource policies it is given; it is a simulation. **The real window still watches every IAM-touching step for `AccessDenied`.** |
| +- **AWS-specific S3 behaviour** (table in §3.6): 403 instead of 404 without `s3:ListBucket`, 409 `ConditionalRequestConflict`, `PermanentRedirect`, throttling, second-resolution timestamps, default SSE. |
| +- **EC2 lifecycle.** Termination fences (`internal/termfence`), EIP moves, auto-recover, real reboots on real hardware, IMDS hop limits. |
| +- **Real power loss.** Killing a container keeps the host's page cache, so unsynced writes survive; on a real box they would not. |
| +- **Scale.** One host and three to five boxes. |
| +- **Per-box clock skew** until P3's VM box (§6). |
| +- **Real networks.** Partitions, cross-region latency and packet loss are simulated on one host's bridge. |
| +- **The outside world.** Let's Encrypt, Route 53, Keymail and SES are stood in for. |
| + |
| +### 2.4 Non-goals |
| + |
| +- Replacing the root runner. The root runner proves kernel and unit behaviour of one test binary; the harness proves a deployment. |
| +- Load or soak testing. |
| +- A permanent shared environment. The harness is created, used and destroyed; snapshots make re-creation cheap. |
| +- Tito, carloku or any customer data. Nothing is copied from production buckets. |
| + |
| +## 3. Architecture |
| + |
| +### 3.1 Shape |
| + |
| +``` |
| +DOCS-T1 (operator) dev account, eu-west-1 |
| + harness up / run / down -- SSM --> harness host (t4g.xlarge, dev AMI, no inbound, TTL) |
| + iam-check (read-only IAM) incus, btrfs pool, bridge hbr0 10.77.0.0/24 |
| + host services, bound to 10.77.0.1 only: |
| + ssm stand-in :8100 imds stand-in :8169 (DNAT 169.254.169.254) |
| + containers (system containers, systemd as PID 1): |
| + infra 10.77.0.10 MinIO (+ audit log), Pebble, dnsmasq, Keymail stub |
| + leo 10.77.0.21 box, home ie, cohort1 (1 vCPU-pair, 1 GiB, as t4g.micro) |
| + sophie 10.77.0.22 box, home ie, console box (as t4g.micro) |
| + dennis 10.77.0.23 box, CARLOS_REGION=uk (as t4g.micro) |
| + signer 10.77.0.30 trusted signing host stand-in |
| + harness bucket carlos-harness-640012953448 (artefacts, evidence) |
| + two throwaway t4g.nano "donors" (minutes; §3.5) |
| +``` |
| + |
| +### 3.2 The host: a throwaway EC2 instance |
| + |
| +DOCS-T1 cannot host it: there is no `/dev/kvm` and no sudo, and rootless podman gives neither real uid 0, real ownership, nor a systemd PID 1 that can run the units' sandboxing. Defects 1 and 2 are ownership and mode defects, so real root is the whole point. |
| + |
| +The host copies the root runner's pattern (`root-runner.py`): dev account, eu-west-1, the dev boxes' Debian 13 arm64 AMI (`ami-071babd89cea53fa3`), SSM only, no inbound, `instance-initiated-shutdown-behavior=terminate`, IMDSv2 required, a 60 GiB gp3 root, and a driver that always terminates and prints `LEAKED` if it cannot. Using the dev AMI means the containers run on the dev boxes' own kernel. |
| + |
| +It differs from the root runner in two ways. It has its own role and bucket (§9), because tonight's scope forbids writes to the dev deployment bucket, which is where `root-runner/*` lives. And it is sized for several boxes: `t4g.xlarge` (4 vCPU, 16 GiB). |
| + |
| +### 3.3 Boxes: incus system containers, provisioned by the real user_data |
| + |
| +Choice: **incus**, with privileged system containers (`security.privileged=true`, `security.nesting=true`) on a btrfs storage pool. |
| + |
| +| Need | incus | systemd-nspawn | podman with systemd | |
| +|---|---|---|---| |
| +| Real uid 0 and ownership | privileged container: uid 0 is root | yes, without `--private-users` | rootful only; rootless maps uids | |
| +| systemd PID 1 running the units' sandboxing (`TemporaryFileSystem` x50, `BindReadOnlyPaths`, `PrivateDevices`, `ProtectKernel*`, `IPAddressDeny`, `Delegate`) | yes, with nesting | yes | partial; cgroup and mount limits | |
| +| user_data delivered the way EC2 delivers it | `cloud-init.user-data` on `images:debian/13/cloud` | hand-built; no cloud-init | no | |
| +| Snapshot and restore in seconds | `incus snapshot create/restore` on btrfs | btrfs subvolumes, by hand | commit images; slow | |
| +| Per-box CPU and memory limits like t4g.micro | `limits.cpu`, `limits.memory` | `--property` | yes | |
| +| Exec transport, file push and pull | `incus exec`, `incus file` | `machinectl shell`, `systemd-run -M` | `podman exec` | |
| +| Static IPs, bridge, ACLs for partitions | managed bridge, `ipv4.address`, network ACLs | networkd by hand | CNI or netavark | |
| +| Packaged on Debian 13 | yes (`incus` in trixie) | yes (`systemd-container`) | yes | |
| + |
| +nspawn has the best systemd compatibility, but tonight it means hand-building rootfs, networking, snapshots and user_data delivery. incus gives all of them, and cloud-init means host-init runs as it does on EC2 (as root, at first boot, with cloud-init's environment and `umask`). Fidelity risk to verify on day one: that every embedded unit starts under its real hardening inside a privileged nested container. A unit that will not start there is a harness defect to fix, never a unit to relax. |
| + |
| +Each box is launched with the infrastructure repo's user_data, byte for byte. Two sources, chosen per run: |
| +- **as-merged:** `dev/host-init.sh` (or `host-init.sh.tftpl` rendered with its one variable, `ssm_region`) at a given ref of `platform-infrastructure`, for testing an infrastructure change before it is applied (for example `host-root-owned-opt`, which is not applied). |
| +- **as-live:** the user_data the real instance was launched with, read once with a read-only `ec2 describe-instance-attribute --attribute userData`. `user_data_replace_on_change = false` (`infra/modules/region-host/main.tf:492`), so a live box may have run an older script than the file on main. |
| + |
| +Hand changes made to real boxes (leo's and sophie's `/opt/carlos` made root-owned by hand in session 3; leo's `CARLOS_POOLS`; the console drop-ins) are recorded as **drift scripts** in the scenario, each citing the runbook record it reproduces. Without them the harness models a box that does not exist. |
| + |
| +Containers are sized as the tofu roots declare the real boxes (`t4g.micro` for all three dev boxes: `limits.cpu=2`, `limits.memory=1GiB`, a 20 GiB root quota). zram is absent (§3.4), so memory pressure behaves differently; that is stated, not hidden. |
| + |
| +### 3.4 Shims, each declared |
| + |
| +The harness changes as little as it can, and lists every change. A run's evidence bundle records which shims were active. |
| + |
| +| Shim | Where | Why | What it hides | |
| +|---|---|---|---| |
| +| `systemd-zram-setup@.service` no-op unit and a `sysctl` wrapper that skips `vm.*` | each box, before cloud-init runs | host-init's last block starts zram and sets `vm.swappiness`; a container cannot, and `set -e` would stop the script there | zram and the swappiness setting | |
| +| `/etc/systemd/system.conf.d/90-harness.conf`: `DefaultEnvironment=CARLOS_S3_ENDPOINT=...` (strict) plus `AWS_ENDPOINT_URL_S3/STS=...` (lenient only) | each box and the signer | there is no AWS; every unit and every `systemd-run` inherits it | nothing the platform reads from `host.env` | |
| +| IMDS stand-in (§3.5) | host, DNAT from the bridge | box identity, addresses and role credentials | the real instance profile | |
| +| SSM stand-in (§3.9) | host | the one transport | Session Manager's attended shells | |
| +| Pebble's root in each box's trust store, `CARLOS_ACME_DIRECTORY` in `host.env` | each box | no public ACME | Let's Encrypt's policy and rate limits | |
| +| Egress allowlist: during provisioning only apt, the SSM agent package URL and the incus image server; after the `base` snapshot nothing but `10.77.0.0/24` | host nftables | boxes must not reach real AWS | nothing; it makes leaks loud | |
| + |
| +### 3.5 Box identity: donor documents |
| + |
| +Claim activation verifies each box by AWS's signed instance identity document: `boxid.Verifier.Enroll` (`internal/boxid/verify.go`) checks the signature under the certificate embedded for the document's region (`internal/boxid/certs/*.pem`, `EmbeddedCerts`), the account, the AWS region and the instance id. There is no override, and `claimactivation.Runner` hardcodes region `ie`, AWS region `eu-west-1` and a 12-digit account (`internal/claimactivation/runner.go:179`, `:190`, `:203`, `:207`, `:242`). A harness box cannot sign its own document. |
| + |
| +But `Enroll` has no freshness or liveness check: a document stays valid evidence for its instance forever. So the harness uses **donor documents**: |
| +1. Launch two `t4g.nano` instances in eu-west-1 in the dev account, no instance profile, with user-data that prints `/latest/dynamic/instance-identity/document` and `/signature` to the serial console and powers off (terminate on shutdown). |
| +2. Read them with `ec2 get-console-output` (read-only). Store them in the harness bucket. They are identifiers, not secrets. |
| +3. The IMDS stand-in serves each box its donor's document and signature byte for byte, plus `instance-id`, `placement/region`, `mac`, `network/interfaces/macs/...`, `local-ipv4` and `public-ipv4` (both the container's bridge address, so edges and backhaul work), and `iam/security-credentials/<role>` returning that box's MinIO keys. It keys every answer by source address, behind an IMDSv2 token. |
| + |
| +`publish-record` (`internal/host/boxpublish.go`), `invite-box`, `enroll-box` and every later verification then run unmodified, against `eu-west-1` and the real certificate. Donors are captured once and reused by every run. The roster's account is the dev account, `640012953448`, because that is what the documents say; the roster itself is signed by the harness's test key and lives only in harness MinIO. |
| + |
| +The alternative, a build of every box binary with a harness certificate added to `certs/`, changes the binary under test. Open question 1. |
| + |
| +The harness host's own IMDS (its real role) must stay unreachable from containers: the DNAT rule catches `169.254.169.254:80` from the bridge, and the host's hop limit stays at 1 so a container could not use an IMDSv2 token from the real service even if a rule were wrong. |
| + |
| +### 3.6 Buckets: MinIO |
| + |
| +MinIO, the release CI already pins and verifies by sha256 (`.amadan/ci.d/30-minio-contract`, `RELEASE.2025-09-07T16-13-09Z`), runs in the `infra` container. Buckets, versioning on for all, as the `deployment-buckets` module makes them: the deployment bucket, `data-ie` and `data-uk`. One endpoint serves all of them, because the code has one: `CARLOS_S3_ENDPOINT` (`internal/s3compat/s3compat.go`). MinIO's region is left unset so it accepts any signing region; `AWS_REGION=eu-west-1` and `CARLOS_HIBERNATE_REGION=eu-west-2` work unchanged. `MINIO_DOMAIN` is set and dnsmasq answers `*.s3.harness.test`, so virtual-hosted addressing works too, for clients that do not set path style. |
| + |
| +What the code assumes of S3, and what MinIO gives: |
| + |
| +| Assumption | Where | MinIO | |
| +|---|---|---| |
| +| `If-None-Match: *` and `If-Match` on `PutObject`, 412 on loss | `release.S3Store`, `boxid.Publish`, claim markers, leases | yes (exercised by `TestDeliveredContractMinio`) | |
| +| 409 `ConditionalRequestConflict` on racing conditional writes | `s3CASConflict`, `internal/release/store.go` | **no**: the loser gets 412. P2's proxy injects 409. | |
| +| Versioning: version ids, delete markers, `ListObjectVersions`, `GetObject`/`DeleteObject` by version | claim storage (Plan 2), cleanup scripts | yes | |
| +| Object lock | none on main | not needed | |
| +| SSE / KMS on objects | none on main | not needed; `GetEncryptionConfiguration` (infraview) answers "none", which differs from AWS's default SSE-S3 | |
| +| 404 for a missing key | `s3NotFound` | **always 404**. AWS answers 403 when the caller lacks `s3:ListBucket` on the prefix (the signer policy's `ListForNotFound` exists for this). The IAM check adds a `ListBucket` check for every 404 it sees (§3.8). | |
| +| Tagging, `x-amz-tagging` on put | objstore scanning | yes, but AGENTS.md records that MinIO accepted a pre-tagged upload an `s3:PutObjectTagging` deny did not stop; harness MinIO policies prove nothing about tagging | |
| +| `LastModified` precision | PITR (`db --at`) | **differs**: MinIO keeps milliseconds; S3 lists second resolution | |
| +| Latency, throttling (503 SlowDown), 500s, lost responses | everything | **no**; P2's proxy injects them | |
| +| `PermanentRedirect` for a wrong-region client | bootstrap-box.sh's comment | **no** | |
| +| Strong read-after-write | everything | yes | |
| + |
| +### 3.7 Endpoints: strict and lenient |
| + |
| +Two S3 clients on main ignore `CARLOS_S3_ENDPOINT`: |
| +- the claim's generation-marker client, `internal/host/claimproduction_marker.go:26` (already filed as #298); |
| +- the fence writer, `internal/hostfence/writer.go:63`, which also assumes a role through STS. |
| + |
| +So the harness has two modes. **Strict** sets only `CARLOS_S3_ENDPOINT`; a client that ignores it tries real AWS, egress blocks it, and the run reports a provider-neutrality finding naming the call. **Lenient** also sets `AWS_ENDPOINT_URL_S3` and `AWS_ENDPOINT_URL_STS` (honoured by `aws-sdk-go-v2/config` v1.32.6, and by `rootaws.LoadDefaultConfig`, which clears only shared files), pointing both at MinIO, whose STS `AssumeRole` issues temporary keys. Steps 1 to 7 run strict. The window (claims on: markers, fences) needs lenient until #298 and the fence writer are fixed, and each lenient run says so in its evidence. |
| + |
| +### 3.8 IAM: an offline check from what the rehearsal did |
| + |
| +MinIO cannot enforce AWS IAM faithfully: it has no roles, no trust policies, no `aws:PrincipalArn`, no KMS key policies. The harness gives each principal its own MinIO user with a broad policy (so runs do not stop on MinIO's approximations), and checks IAM offline from a log of what each principal actually did. |
| + |
| +**Capture.** P1 uses MinIO's audit log (a webhook target to a JSONL sink in `infra`): each entry has the access key, API name, bucket, object, query (`versionId`, `prefix`, `list-type`, `versions`), request headers (`If-Match`, `If-None-Match`, `x-amz-tagging`, `x-amz-copy-source`) and status. P2 replaces it with a small Go S3 proxy in front of MinIO, which records the same fields plus the source box, parses `DeleteObjects` bodies, and is also the fault injector (§6). The SSM stand-in logs `GetParameter` (with decryption: also a `kms:Decrypt` with `PARAMETER_ARN` context), `PutParameter` and `SendCommand`; MinIO's STS log gives `AssumeRole`. |
| + |
| +**Mapping.** A table in code, with tests, from each logged request to IAM actions, resource ARNs (with the real bucket names substituted for the harness names) and condition context: |
| + |
| +| Request | Action(s) | Context | |
| +|---|---|---| |
| +| `PutObject` | `s3:PutObject` | `s3:if-none-match`, `s3:if-match`, `s3:ObjectCreationOperation`, `s3:RequestObjectTagKeys` | |
| +| `GetObject`, `HeadObject` | `s3:GetObject`, or `s3:GetObjectVersion` with `versionId`; a 404 adds `s3:ListBucket` on the key's prefix | | |
| +| `DeleteObject`, `DeleteObjects` | `s3:DeleteObject`, or `s3:DeleteObjectVersion` per key | | |
| +| `ListObjectsV2` | `s3:ListBucket` | `s3:prefix` | |
| +| `ListObjectVersions` | `s3:ListBucketVersions` | `s3:prefix` | |
| +| `CopyObject` | `s3:GetObject` on the source, `s3:PutObject` on the target | | |
| +| multipart | `s3:PutObject`, `s3:AbortMultipartUpload`, `s3:ListBucketMultipartUploads` | | |
| +| SSM `GetParameter --with-decryption` | `ssm:GetParameter`, `kms:Decrypt` | `kms:EncryptionContext:PARAMETER_ARN` | |
| +| STS `AssumeRole` | `sts:AssumeRole` on the role | | |
| + |
| +Every request also carries `aws:PrincipalArn` = the real role the principal stands for (leo `carlos-dev-host-b`, sophie `carlos-dev-host`, dennis `carlos-dev-host-uk`, signer `carlos-claim-signer`, fence writer `carlos-dev-ie-b-fence-writer`). Step 3 of the runbook learned why: a negated condition matches a missing key. |
| + |
| +**Simulation**, from DOCS-T1 with the operator's read-only IAM rights, exactly as runbook step 3 ran it on 2026-10-18: |
| +- for applied policy: `aws iam simulate-principal-policy` on the real role ARN, with `--resource-policy` set to the live bucket policy or key policy (with `"Resource": "*"` replaced by the key ARN in a scratch copy, as step 3 had to); |
| +- for proposed policy (a PR not applied, or `signing-host.md`'s JSON before a session): `aws iam simulate-custom-policy` with the proposed documents, and permissions boundaries when the role has one. |
| + |
| +A run fails on any `implicitDeny` or `explicitDeny` for an action the rehearsal performed successfully, and lists them. Defect 3 is exactly this case. |
| + |
| +**What it cannot do.** SCPs, KMS grants, VPC endpoint policies and session policies are not evaluated. Every new grant still meets IAM's eventual consistency on the real account. The check sees only actions the harness performed, so a code path the harness did not reach is unchecked. Hence the rule this spec keeps: **box IAM is simulated offline, so the real window still has its IAM-touching steps watched for `AccessDenied`.** |
| + |
| +### 3.9 SSM, and one transport for runbooks |
| + |
| +The coordinator suggested `machinectl shell` or `nsenter` in the harness and SSM on real boxes. This spec goes one step further: the harness serves the **SSM API itself**, so the runbook's own helpers (`ssm.sh`, `ssm_run`, `box_get`, `box_put`, `box_inspect`), `bootstrap-box.sh` and `update-platform-binary.sh` run unchanged. The only difference is `AWS_ENDPOINT_URL_SSM`, set in the harness shell. |
| + |
| +The SSM stand-in runs on the host (it needs the incus socket), bound to the bridge address. It implements `SendCommand` (`AWS-RunShellScript`), `GetCommandInvocation`, `ListCommandInvocations`, `DescribeInstanceInformation`, and Parameter Store `GetParameter(s)`, `GetParametersByPath`, `PutParameter`, `DeleteParameter`, `DescribeParameters`. It maps instance ids (the donors' ids) to containers. A command runs as root through `incus exec`, under `/bin/sh` (dash, as the real agent's script does), in a per-command working directory, with `umask 022` and the agent's minimal environment, and its results reproduce the real limits: `StandardOutputContent` truncated at 24,000 characters and `StandardErrorContent` at 8,000, `Status` `Failed` on a non-zero exit, `TimedOut` at the execution timeout. Those limits are the reason `box_get` compresses and checks digests, so faking them generously would hide real failures. |
| + |
| +Attended root shells (`ssm start-session`) are not served. Runbook-as-code replaces them with block scripts: each attended block becomes one file sent through `SendCommand`, sourcing a per-window variables file on the box, so later blocks see earlier blocks' variables. The driver has an `--attended` mode that stops after each block and waits for the operator, which is how the same script runs in a real window. |
| + |
| +### 3.10 The console, and a Keymail stand-in |
| + |
| +The real console binary (`cmd/console`, built at the commit under test) runs on sophie, as on dev. Its unit is a problem: on dev, `console.service` and its drop-ins (`release-key.conf`, `apps-domain.conf`) are hand-written on sophie and in no repository. The harness needs a copy; until one is captured read-only, it uses a reconstruction from the operator notes and marks the console's unit as unverified in every run (open question 5). |
| + |
| +Console sign-in goes through Keymail (`CONSOLE_KEYMAIL`, default `https://keymail.dev`; `internal/console/keymail.go`). The harness runs a Keymail stand-in in `infra` serving the five endpoints the console calls (`/.well-known/keymail`, `/oauth/authorize`, `/api/oauth/token`, `/api/oauth/revoke`, the account lookup) and auto-approving one harness operator address, which is on `CARLOS_SYSTEM_OPERATORS`. A curl script signs in, then approves `carlos auth login`'s device code, so the CLI gets an ordinary console token. `/api/app/send` (console mail) answers 503, matching P9's "console mail off". |
| + |
| +P1 needs this: `install-channel --policy --identity` needs a system channel pointer, and system pointers are written only by the console's system update door (`release.PromoteSystem`, called from `internal/console/hostactivation.go:203`), which needs a signed-in operator. |
| + |
| +### 3.11 Release and signing with test keys |
| + |
| +Two production trust anchors are involved, and the harness replaces both with keys it generates on the host, in tmpfs, for the host's lifetime: |
| + |
| +- **The root release key** (`/carlos/release-key`, `CARLOS_RELEASE_PUBKEY`): a test key from `carlos release-keygen`. Its public half goes into each box's `host.env` through `bootstrap-box.sh`'s `pubkey-file` argument, and into the console's `release.env`. The fake Parameter Store holds the private half at `/carlos/release-key`, so the signer fetches it with `signing-host.md`'s own "Getting the key there" commands. |
| +- **The CLI update key**, which signs `SHA256SUMS.txt`. `publish-identity` verifies the sums with `publicrelease.Default().PubKey` (`internal/host/activation_command_linux.go:305`), a literal compiled into the binary (`internal/publicrelease/feed.go:35`, pinned equal to `cliUpdatePubKey` by `claims_sums_key_test.go`). An unreleased build cannot have production-signed sums, and the harness never holds that key. So the **signer's** `carlos` binary is built with a one-line harness patch that replaces that literal with the test sums key. Only the signer runs it; `publish-identity` is the only reader of that key on the claim path. Every box binary is unpatched. Later (P3) the literal becomes a build-time variable on main so no patch is needed (open question 2). |
| + |
| +A harness release is the README's release recipe with test keys: `carlos-linux-arm64` and `console-linux-arm64` built at the commit under test (`CGO_ENABLED=0`, `-s -w -X .../internal/version.Build=<tag>`), `SHA256SUMS.txt` over them, signed with the test sums key. The tag is the one the real release will carry (for example `v0.28.5`), so the runbook's literal values match. Nothing signed by a test key can install on a real box, because no real box trusts it. |
| + |
| +The signer is a container with the trusted host's role (its MinIO user), `jq` and the `carlos` CLI checked by sha256, and its key on `/run`. It runs `claims run` phases exactly as `signing-host.md` describes. "Closing the signing session" becomes: shred the key, `incus delete --force signer`, remove the exchange objects, and check the SSM stand-in's log shows exactly one read of `/carlos/release-key` (the stand-in for the CloudTrail check). |
| + |
| +### 3.12 DNS, TLS and the edge |
| + |
| +- **DNS:** dnsmasq in `infra`, the bridge's resolver. A reserved test domain, `dev.harness.test`, with explicit records for the console, each app host, and `*.s3.harness.test` for MinIO. `CARLOS_DNS_PROVIDER` is unset: DNS is the operator's, as on a self-hosted deployment. |
| +- **TLS:** Pebble, Let's Encrypt's test ACME server, in `infra`, resolving through dnsmasq. Boxes point `CARLOS_ACME_DIRECTORY` (already on host.env's allowlist, `internal/hostenv/hostenv.go`) at it and trust its root. This exercises the real ACME path, including runbook step 4.5's trap: only the housekeeper-lease holder orders certificates, and only when a TLS hello reaches it. |
| +- **The edge:** each box's real `carlos-edge.service`. Backhaul works because the IMDS stand-in gives each box its bridge address as its public address. Clients on the host use `curl --resolve` and Pebble's root, the same commands the runbook uses with real addresses. |
| + |
| +## 4. Scenario 1: the claim trial, Option B |
| + |
| +### 4.1 Commits |
| + |
| +- **Bring-up and the pre-window steps:** main `96db0efe` (v0.28.4 plus #317's `guard-arm` directory modes). |
| +- **The window onwards:** `claim-empty-cohort-done` (sha to come): 8.1b `backfill-collect`, 8.2 `journal-migrate --window`, 8.2b `backfill-done`, 8.4, step 9, block B4 and acceptance. The harness rolls leo to it with the console's system update door, as G4 does on dev, from the `s7` snapshot. |
| + |
| +### 4.2 Steps 1 to 7, through the platform's own commands |
| + |
| +The claim owner's requirement: the harness rebuilds steps 1 to 7 through the platform's own commands, from a test signer; nothing is hand-made, and every hand-seeded step is listed with its reason (§4.3). |
| + |
| +| Runbook step | In the harness | |
| +|---|---| |
| +| 1. Release and roll, claims off | Harness release (§3.11). Boxes bootstrapped by `bootstrap-box.sh` over the SSM stand-in, then the console shipped and promoted direct-bucket (`carlos ship --app console --kind binary`, `carlos promote ... canary/console`) and started. The system channel pointers come from `carlos system update --console <harness> --file ... --platform linux-arm64 --version <tag>`, the "new road". Checks as in the runbook: `system status` current, `x-carlos-version`, `claims help` text. | |
| +| 2. Empty app segment cleanup | Not applicable: no `apps/bab//` history. B5's check still runs and passes. | |
| +| 3. IAM | Hand-seeded: MinIO users stand in for the roles (§4.3). The step's denied-operation witnesses are replaced by the offline check (§3.8), which covers the same rows. | |
| +| 4. Pool and disposable app | `CARLOS_POOLS=cohort1` appended on leo by the runbook's own block. **Stretch for P1:** the app through the console doors (`apps create`, `ship`, `promote`, store-mode `instances enable --pool cohort1`, signed-in `instances create`) using `cmd/moveprobe` at the same commit instead of `wc4`, which exists only in the dev bucket. This is what gives the harness defect 5 (records without `acq`). | |
| +| 5.0 Box identity records | `systemd-run --wait --pipe -p EnvironmentFile=/etc/carlos/host.env -p User=carlos /opt/carlos/carlos system box publish-record` on leo and sophie through the SSM stand-in; the re-run's "nothing was written" message is checked verbatim. | |
| +| 5.0a Release-stamped console, digests | Pointer digests on for the harness account (`carlos channels digests`, release key in `CARLOS_RELEASE_KEY`), then ship and promote the release-stamped console to `canary/console`. | |
| +| 5.1 Roster and enrollment | On the signer: `claims run` phases `initialize`, `configure` (regions `{ie: eu-west-1, account 640012953448, namespaces, pools cohort1 and console}`), `invite-box` and `enroll-box` for leo and sophie (the donor instance ids as `instance_id`), `invite-console` and `enroll-console`. `box_inspect` and the staging helpers as written. | |
| +| 6.1 Policy and identity | `sign-policy` and `publish-identity` (`--out` files) on the signer, with the harness release's sums. | |
| +| 6.1a `.prev` | `install-channel --policy --identity` on leo, then sophie. | |
| +| 6.2, 6.3 | `guard-arm` on leo; P9 (console mail off); `install-console`, `guard-arm --console`, `console-config`, the console restart; the wait for both boxes' `floor_guard_ready`. | |
| +| 7 | P3 readers; `console-report` on sophie then `publish-floor` on the signer at once; `home ie`; `evidence-key` on leo, then `bind-key`; `console-config --enable-console-runtime`; the `jq -e` roster check. The session closes (§3.11). | |
| +| 7a Fence credentials | The fence writer's MinIO user keys go into the fake Parameter Store as `/carlos/cohort1/fence/<leo>`; 7a.2's own commands fetch them to tmpfs and run `claims fence-credentials --source - ...`. | |
| + |
| +Each step ends with the after-step check: every expected unit active, the console 200 through sophie's edge, each box's latest report verifying, no new `AccessDenied` or panic in any journal, and a mode and owner listing of `/opt/carlos`, `/etc/carlos` and `/data` saved to the evidence. |
| + |
| +### 4.3 Hand-seeded, with reasons |
| + |
| +| # | What | Reason | |
| +|---|---|---| |
| +| H1 | MinIO buckets, versioning on | stand-in for the `deployment-buckets` module, which creates AWS buckets | |
| +| H2 | MinIO users, one per role, plus the signer and fence writer | there is no IAM; enforcement is the offline check | |
| +| H3 | Donor identity documents, served by the IMDS stand-in | AWS signs identity documents and `boxid` trusts only AWS's certificates | |
| +| H4 | `90-harness.conf` endpoint defaults; the zram and sysctl no-ops; Pebble's root | §3.4: no AWS, and containers cannot do zram | |
| +| H5 | Test release key and test sums key; the signer's one-line patch | no production key ever; `publish-identity` trusts a compiled-in key | |
| +| H6 | `console.service` and its drop-ins | written by hand on sophie and in no repository | |
| +| H7 | Console environment: `CONSOLE_ORIGIN`, `CONSOLE_KEYMAIL` (the stand-in), `PLATFORM_APPS_DOMAIN`, `CARLOS_SYSTEM_OPERATORS` | sophie's real values name real hosts and a real IdP | |
| +| H8 | Drift scripts: hand changes recorded in the runbook (for example session 3's `chown`), applied only in as-live mode | the real boxes are not what host-init alone produces | |
| +| H9 | Account names: the harness account's sqid replaces `bab` | sqids come from a counter; the scenario takes the account as a parameter | |
| + |
| +Nothing else is written by hand. A step that cannot be reproduced without a new item stops the scenario and adds a row here with its reason, rather than being worked around. |
| + |
| +### 4.4 Snapshots for this scenario |
| + |
| +`base` (host-init done), `boot` (step 1), `pre5` (steps 2 to 4 and prechecks), `s7` (step 7 and 7a done, session closed), `pre8` (G3 and G4 on the window build, blocks B1, B2 and B5 done). The window is rehearsed from `pre8` as often as needed. |
| + |
| +## 5. Lifecycle |
| + |
| +| Stage | What happens | Time (estimate) | |
| +|---|---|---| |
| +| `harness up` | Launch the host (TTL tag, self-terminating shutdown timer); install incus and btrfs-progs; create the pool and bridge; pull `images:debian/13/cloud`; start `infra`; fetch artefacts and donor documents from the harness bucket | 10 to 15 min, once per host | |
| +| provision | Launch the boxes with the chosen user_data; cloud-init runs host-init (apt dominates); snapshot `base` | about 5 min, parallel | |
| +| bring-up | Bootstrap, console, the claims-off checks; snapshot `boot` | about 3 min | |
| +| steps 5 to 7 | §4.2; the agent's report ticks dominate (first about a minute after the edge starts, then every 5 minutes plus jitter, so up to about 6 minutes after `guard-arm`) | about 12 min | |
| +| **reset** | Stop all containers, `incus snapshot restore` each to the same named snapshot (MinIO's data is in `infra`, so bucket and boxes agree), start them | **20 to 40 s** | |
| +| run | One scenario from a snapshot | window from `pre8`: about 6 to 8 min | |
| +| collect | Evidence bundle to the harness bucket, then to DOCS-T1's `$TMPDIR` | under 1 min | |
| +| `harness down` | Terminate the host (retries, `LEAKED` if it fails); check no tagged instance remains | 1 min | |
| + |
| +**Reset plus a window run is under 10 minutes.** Steps 5 to 7 from scratch are about 12, so they run once per host and are kept as `s7`. |
| + |
| +Snapshots are taken with every container stopped, so they are consistent across boxes and buckets. A restore is a power cycle of the whole deployment at a later wall-clock time: leases and short-lived evidence may have expired. That is real behaviour (it is what #312 is about), so scenarios snapshot just before a time-sensitive pair, never between its halves. |
| + |
| +**The evidence bundle** for every run: the scenario transcript (each block, its stdout, stderr and exit), the S3 action log, the SSM and IMDS stand-ins' logs, `journalctl -o export` from each box, mode and owner listings of `/data`, `/opt/carlos`, `/etc/carlos` and `/etc/systemd/system`, the bucket listing with versions, the IAM check's report, and the list of active shims and the run's mode (strict or lenient). It never contains a key: keys live on `/run` and are excluded by path. |
| + |
| +## 6. Fault injection |
| + |
| +| Fault | Mechanism | Phase | The case it serves | |
| +|---|---|---|---| |
| +| Kill a box mid-step | `incus stop --force` (all processes killed; disk intact, page cache kept) or `systemctl kill -s KILL carlos-edge` for an agent crash; triggered by time, by a log line, or (P2) by an S3 request ("when leo puts `.../seal.json`, kill leo") | P1 by hand, P2 triggered | moves: a box restarts mid-step, the sealer dies mid-tombstone, the target dies after the flip; claims: crash points between CAS and effect | |
| +| Restart the agent before reconcile | restart `carlos-edge` after a record change and before its next pass, with the trigger above | P2 | moves' B1 (the hold decided from the record alone); a restarted box's startup latch | |
| +| Partition a box from a bucket | P1: host nftables dropping a box's traffic to `infra`. P2: the proxy refuses or blackholes one box's requests to one bucket, since every bucket shares one endpoint | P1 coarse, P2 per bucket | moves: partition the sealer from `data-ie` during the final mirror; a source box down at `open` | |
| +| Slow S3 | P2 proxy: a latency distribution per box and bucket, bandwidth caps | P2 | defect 4's race windows; lease TTL margins | |
| +| S3 errors | P2 proxy: 500, 503 SlowDown, 409 on conditional writes, a reset mid-body, and the **lost response** (forward the PUT, drop the answer) | P2 | every CAS writer's unknown-outcome path | |
| +| Box down for a while | `incus stop`, later `incus start` | P1 | moves: a box down at `start` or after `open`; claims: a box returning to a moved roster | |
| +| Clock skew | not possible per container: `CLOCK_REALTIME` has no namespace, privileged containers lack `CAP_SYS_TIME`, and `libfaketime` does not reach static Go binaries' vDSO calls. P3 adds a **VM box**: a separate small EC2 instance in the harness security group with chrony off, which joins the harness network over the VPC | P3 | lease expiry, evidence freshness, `written_at` checks | |
| + |
| +## 7. How it is used |
| + |
| +### 7.1 The claim rehearsal before every real window |
| + |
| +Before Paul is asked for a window's go, the exact runbook revision runs in the harness against the exact release candidate, from `boot` through acceptance, in `--attended` off mode, and its evidence bundle (with the IAM check's report) is posted on #250. Proposed as a gate (open question 4). The attended real window then runs the same block scripts through real SSM, one block at a time. The window's IAM-touching steps are still watched for `AccessDenied`. |
| + |
| +What it would have changed this week: defects 1, 2 and 4 found in one evening instead of three sessions; defect 3 found before session 4; S1 and S2 found on a box-shaped tree. |
| + |
| +### 7.2 The offline move rehearsal |
| + |
| +The offline move plan's runbook P1 ("sophie to dennis, claims off on both") cannot run on dev as-is: leo is a home-placement box with claims on, so the start door refuses (MV-O21), and the plan leaves that gate to a decision. The harness has no such conflict: its topology is chosen per scenario. It also reproduces the flagship's shape (one home box, the console's, plus one UK box), which dev cannot, because dev has two home boxes (#286). |
| + |
| +Scenario `moves-offline`, from a snapshot with `moveprobe` deployed (a store, a secret, a schedule, a once-timer) and woken on sophie only: |
| +- the happy path, every verb by hand with `status` between, and `/report` checks on dennis after `open`; |
| +- abort from `offline-landed` (the plan's rollback rehearsal); |
| +- the fault matrix from §6: the sealer killed between tombstone and final mirror, the target killed after `flip`, a source box down at `seal` and at `open`, the agent restarted between `start` and its next reconcile, the sealer partitioned from `data-ie` during the final round, slow S3 during landing, a lost response on the tombstone write; |
| +- each ending with the plan's own assertions (the record diff, the tombstone's owner, parked paths), read through the CLI and the bucket. |
| + |
| +This is the spawned-app proof the plan's Task 8 leaves to the dev rehearsal, with real units, real root passes and real ownership, plus faults the in-process witness cannot inject. |
| + |
| +### 7.3 CI |
| + |
| +- **Per merge (amadan CI, no AWS):** the harness's own code only: the action-to-IAM mapping table's tests, the proxy's tests (P2), the stand-ins' tests, and a lint that every scenario block parses. Cost: seconds. The harness itself never runs per merge: it needs AWS credentials, and CI has none. |
| +- **Nightly (P3), driven from DOCS-T1 or a scheduled job with the harness role:** a fresh host, main's head, scenarios `cohort1` (bring-up to step 7, strict) and `moves-offline` (happy path and abort). About 45 to 60 minutes of host time. |
| +- **Weekly:** the fault matrix. |
| +- **On demand:** any branch, before any window. |
| + |
| +## 8. Phases, effort and cost |
| + |
| +### 8.1 Phase 1, tonight: hours, rough |
| + |
| +**P1a: boxes, MinIO, host-init (about 3 agent hours).** Enough to reproduce defects 1 and 2's ground. |
| +- The approval packet below; the harness bucket, role and security group; the host; the two donors, captured and terminated. |
| +- incus, the bridge, `infra` with MinIO (audit log on), dnsmasq, Pebble. |
| +- IMDS and SSM stand-ins, Python 3 standard library only, about 150 and 250 lines. |
| +- leo, sophie, dennis and signer launched with `dev/host-init.sh` from `platform-infrastructure` main as cloud-init user_data; the shims; snapshot `base`. |
| +- Harness release at `96db0efe` built on DOCS-T1 (cross-compiled, as the root runner does) and copied through the harness bucket; `bootstrap-box.sh` run unchanged through the SSM stand-in; snapshot `boot`. |
| +- A read-only layout probe: owners and modes of `/opt/carlos` and its parents, `/etc/carlos/host.env`, against `OpenProtected`'s rule. This shows defect 1's cause at once, before step 6.2 shows its effect. |
| + |
| +**P1b: the console and steps 5 to 7 (about 4 agent hours).** |
| +- Test keys; the signer's patched build; the Keymail stand-in and the sign-in script; the console on sophie. |
| +- The runbook's steps 5.0 to 7a transcribed into block scripts, values as parameters, helpers verbatim. |
| +- A first IAM check by hand: the audit log filtered to the signer, each action simulated against `signing-host.md`'s policy with `simulate-custom-policy`. |
| +- Snapshot `s7`. Stretch: step 4's app (defect 5). |
| + |
| +**Approval packet** (dev account `640012953448`, eu-west-1, AGENTS.md's scoped-approval rule): |
| + |
| +| Resource | Detail | Cost | Lifetime and cleanup | |
| +|---|---|---|---| |
| +| S3 bucket `carlos-harness-640012953448` | versioning off, lifecycle: expire everything after 3 days, block public access, tag `purpose=carlos-harness` | pennies | deleted when the harness is retired | |
| +| IAM role and instance profile `carlos-harness` | `AmazonSSMManagedInstanceCore`; `s3:GetObject`/`PutObject`/`ListBucket` on the harness bucket only; explicit Deny on `arn:aws:s3:::carlos-dev-*` and on `ssm:GetParameter*` for `/carlos/*` | none | deleted with the bucket | |
| +| Security group `carlos-harness` | no inbound; egress 443 and 80 | none | deleted with the role | |
| +| One `t4g.xlarge` harness host | dev AMI, 60 GiB gp3, IMDSv2, hop limit 1, terminate on shutdown, tags `purpose=carlos-harness`, `expires=<UTC>` | about $0.15 an hour | at most 24 hours; driver terminate, self-shutdown timer, leftover check | |
| +| Two `t4g.nano` donors | no instance profile, no inbound, terminate on shutdown | under $0.01 | minutes | |
| + |
| +Read-only calls also made: `ec2 get-console-output` (donors), optionally `ec2 describe-instance-attribute userData` (as-live mode), and the IAM simulations. Nothing touches the dev boxes, the dev buckets, the real signer, or `/carlos/*` parameters. |
| + |
| +### 8.2 Later phases |
| + |
| +| Phase | Contents | Effort | |
| +|---|---|---| |
| +| **P2** | The Go S3 proxy (per box and bucket faults, the lost response, latency, 409s, the action log with source box); the automated IAM check (`harness iam-check`); the window scenario on `claim-empty-cohort-done` (8.1b to acceptance); step 4 and a populated-state fixture (defect 5); tree replay from `leo-tree-*.tsv` (S2); triggered kills and restarts; evidence bundle tooling | 3 days | |
| +| **P3** | The `moves-offline` scenario and its fault matrix; the VM box for clock skew; nightly and weekly runs; an AMI-derived container rootfs instead of the linuxcontainers image; upstream seams so no patch is needed (the sums key as a build-time variable, `bootstrap-box.sh` extra `host.env` lines, #298 and the fence writer honouring the endpoint, the console unit moved into `platform-infrastructure`); porting the stand-ins to Go with tests; a reaper (EventBridge Scheduler plus a small function terminating `purpose=carlos-harness` instances past `expires`) | 3 days | |
| +| **P4, optional** | Shadow IAM: harness-named real buckets and roles from the same tofu modules in a sandbox account, with the action log replayed live, so resource policies, KMS key policies and SCPs are enforced for real | 2 days, plus approvals | |
| + |
| +### 8.3 Cost |
| + |
| +| Use | Host time | Cost | |
| +|---|---|---| |
| +| Tonight | up to 12 hours | about $2 | |
| +| A rehearsal before a window | 1 to 2 hours | about $0.30 | |
| +| Nightly | about 1 hour a night | about $5 a month | |
| +| A working day kept up | 10 hours | about $1.50 | |
| +| Keeping a stopped host with built snapshots (skips 20 minutes of provisioning) | 60 GiB gp3 only | about $5 a month | |
| + |
| +## 9. Security |
| + |
| +- **No production credential or key.** Both signing keys are generated on the host, live on its tmpfs and die with it. The Parameter Store the signer reads is the stand-in. The harness role cannot read `/carlos/*` or touch `carlos-dev-*` buckets, by explicit Deny, so a mistaken endpoint variable fails rather than reaching dev. |
| +- **No real AWS from the boxes.** Boxes hold only MinIO keys. The real IMDS is unreachable from containers (DNAT plus hop limit 1). Egress from boxes is closed after provisioning. A client that ignores the endpoint fails loudly (§3.7). |
| +- **Test artefacts cannot escape.** Everything the harness signs is signed by test keys no real box or console trusts. Harness releases live only in harness MinIO and the harness bucket. |
| +- **Donor documents** are identifiers (account, instance id, addresses, AMI), not secrets; the donors had no role. |
| +- **Scoped account use.** The operator's `kass` credentials are used for the harness's own tagged resources, read-only console output, and read-only IAM simulation; never for dev boxes or buckets. |
| +- **Teardown is guaranteed three ways tonight.** The driver terminates in a `finally` with retries and prints `LEAKED` if it cannot (the root runner's 2026-10-14 lesson). The host's user-data arms `shutdown -h +<TTL>` with terminate-on-shutdown, so it dies even if the driver does. Every `harness up` first lists `purpose=carlos-harness` instances and refuses to start while an expired one exists. P3 adds the account-side reaper. |
| +- **Bindings.** Every stand-in binds the bridge address or loopback, never all interfaces, and the security group has no inbound rule. |
| +- **Logs.** The evidence excludes `/run` and never prints a key; key checks print presence and a sha256 prefix, as the runbook does. |
| + |
| +## 10. Provider neutrality |
| + |
| +The harness is half of the self-hosting proof: MinIO, hosts and DNS, with AWS removed underneath the platform. The other half is what it shows still depends on AWS. |
| + |
| +**Runs without AWS today** (P3 can run these with no donor and no AWS at all, using `rootaws`'s off-cloud static credentials file `/etc/carlos/credentials` instead of the IMDS stand-in): deploys, wakes, hibernation, leases, the offline move, the console's ordinary doors, ACME with any directory. |
| + |
| +**Hard AWS dependencies the harness has to stand in for or route around:** |
| +- Box identity is AWS's signed document (`internal/boxid`), so claim mode and multi-box regions are AWS-only (Paul's H26, #299). The donors keep the harness on AWS for claims. |
| +- `claimactivation.Runner` hardcodes `ie`, `eu-west-1` and a 12-digit account (`runner.go:179` to `:242`). |
| +- `publish-identity` trusts a compiled-in sums key (`publicrelease.Default`), so no deployment can publish an identity for its own build. |
| +- Claim markers and the fence writer ignore `CARLOS_S3_ENDPOINT` (#298; `hostfence/writer.go:63` also needs STS). |
| +- Termination fences read EC2 (`internal/termfence`, `awsinv.NewEC2Reader`); region HA's host.env parameters are SSM plus KMS (`internal/hostenv/awsssm.go`); store provisioning creates IAM users even with an endpoint (`internal/objstore/aws.go`). |
| +- Route 53 (DNS, steering), SES, Cost Explorer, CloudWatch and the SQS termination queue are adapters a non-AWS deployment simply lacks. |
| +- Infrastructure, not platform: host-init installs `awscli` and the SSM agent from an S3 URL, and `bootstrap-box.sh` copies the binary from S3 with the AWS CLI. |
| + |
| +Each strict-mode run lists which of these it met, so the list stays current. |
| + |
| +## 11. Where the code lives |
| + |
| +`harness/` in this repository: `up`, `down`, `run` (the driver, Python like `root-runner.py`, tonight), `host-setup.sh`, `stand-ins/` (IMDS, SSM, Keymail), `proxy/` (Go, P2), `iam/` (the mapping table and the simulation driver), `scenarios/cohort1/` and `scenarios/moves-offline/` (block scripts, parameters, after-step checks). Infrastructure files are fetched by ref from `platform-infrastructure`, never copied in, so the harness always runs the user_data under test. The scenarios cite the runbook revision they were transcribed from, and a runbook change without a scenario change is flagged in review. |
| + |
| +## 12. Open questions for Paul |
| + |
| +1. **Donor identities.** Use real dev-account identity documents from two throwaway `t4g.nano` instances, replayed by the harness's IMDS stand-in, so `boxid` verification runs unmodified on every box? The alternative is a harness certificate compiled into every box binary. |
| +2. **The signer's build.** Accept a harness-only signer binary carrying the test sums key tonight (a one-line local patch, never merged), and a build-time variable on main later? Box binaries stay unpatched either way. |
| +3. **Tonight's packet.** Approve §8.1's resources in dev: the harness bucket, role and security group, one `t4g.xlarge` for at most 24 hours, two `t4g.nano` for minutes. |
| +4. **A gate.** Make a green harness rehearsal of the exact runbook revision and release, with its IAM report, a precondition for every claim window and for the first offline move? |
| +5. **The dev console's unit.** May an agent read `console.service` and its drop-ins from sophie once (read-only SSM), and should they then move into `platform-infrastructure`? |
| +6. **Live IAM (P4).** Is offline simulation plus watching the real window for `AccessDenied` enough, or do you want shadow IAM in a sandbox account? |