carlos / platform Public

design-deployment-harness: spec rev 2, Fable's review applied

B1: a //go:build harness override file (endpoint fallback, the two root
clients, the sums key) with a no-tag test, replacing env-based lenient
mode that rootaws.ScrubEnvironment defeats (platform-1e's ruling; listed
as the one binary deviation). B2: bridge in 203.0.113.0/24, outside the
instance units' IPAddressDeny. B3: operator shell on the harness host.
B4: aws CLI endpoint injection, SSM agent masked. B5: donors captured
over SSM. Day-one checks, an ordered 11-step phase 1 with checks, the
safety additions (Denies, tag-filtered driver, donors in the leak check,
12 h TTL), fidelity holes as known limits, and proposed answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 5 days ago a438766b07243cc244b9de28b47154f5adc1d646 parent 85bd6c0

1 file changed, +203 −157

docs/superpowers/specs/2026-10-20-deployment-harness-design.md +203 −157

CI passed — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	317.596s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.957s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.268s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.727s
ok  	github.com/carlosframework/platform/internal/activator	251.515s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.004s
ok  	github.com/carlosframework/platform/internal/analytics	0.021s
ok  	github.com/carlosframework/platform/internal/appconfig	2.457s
ok  	github.com/carlosframework/platform/internal/apperr	0.072s
ok  	github.com/carlosframework/platform/internal/applog	0.142s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.005s
ok  	github.com/carlosframework/platform/internal/boxid	0.545s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.696s
ok  	github.com/carlosframework/platform/internal/changelog	0.324s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.793s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	407.233s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.442s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.005s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.005s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.378s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	44.938s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.673s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.031s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.154s
ok  	github.com/carlosframework/platform/internal/host	182.640s
ok  	github.com/carlosframework/platform/internal/hostactors	6.149s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.014s
ok  	github.com/carlosframework/platform/internal/hostbox	2.704s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.559s
ok  	github.com/carlosframework/platform/internal/hostenv	0.014s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.014s
ok  	github.com/carlosframework/platform/internal/hostexec	5.021s
ok  	github.com/carlosframework/platform/internal/hostfence	0.033s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.475s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.015s
ok  	github.com/carlosframework/platform/internal/hostguard	0.999s
ok  	github.com/carlosframework/platform/internal/hostops	1.617s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.014s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.476s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.132s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.016s
ok  	github.com/carlosframework/platform/internal/infraview	0.018s
ok  	github.com/carlosframework/platform/internal/instancectl	5.227s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.742s
ok  	github.com/carlosframework/platform/internal/instancefs	0.020s
ok  	github.com/carlosframework/platform/internal/instancekey	0.016s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.020s
ok  	github.com/carlosframework/platform/internal/layout	1.577s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.017s
ok  	github.com/carlosframework/platform/internal/lease	9.820s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.010s
ok  	github.com/carlosframework/platform/internal/ledger	0.013s
ok  	github.com/carlosframework/platform/internal/mail	0.174s
ok  	github.com/carlosframework/platform/internal/maintenance	0.079s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.538s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.074s
ok  	github.com/carlosframework/platform/internal/metrics	0.007s
ok  	github.com/carlosframework/platform/internal/moves	21.032s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.823s
ok  	github.com/carlosframework/platform/internal/objstore	0.188s
ok  	github.com/carlosframework/platform/internal/ops	0.172s
ok  	github.com/carlosframework/platform/internal/park	0.206s
ok  	github.com/carlosframework/platform/internal/privatefile	0.006s
ok  	github.com/carlosframework/platform/internal/provision	37.394s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.008s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.526s
ok  	github.com/carlosframework/platform/internal/relay	7.223s
ok  	github.com/carlosframework/platform/internal/release	23.989s
ok  	github.com/carlosframework/platform/internal/rollout	3.863s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.057s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.564s
ok  	github.com/carlosframework/platform/internal/s3compat	0.004s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.507s
ok  	github.com/carlosframework/platform/internal/seeds	0.013s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.821s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.686s
ok  	github.com/carlosframework/platform/internal/status	0.063s
ok  	github.com/carlosframework/platform/internal/steering	0.007s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.217s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.578s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 49414306 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
  minio: downloaded, digest verified
  mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
  mc: downloaded, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.023s
=== RUN   TestT4StorageProbeMinio
2026/10/20 00:44:57 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0163BD3EAB343, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
minio contract: PASS

diff --git a/docs/superpowers/specs/2026-10-20-deployment-harness-design.md b/docs/superpowers/specs/2026-10-20-deployment-harness-design.md

index 177c8aaf4..ceec5ed17 100644

--- a/docs/superpowers/specs/2026-10-20-deployment-harness-design.md

+++ b/docs/superpowers/specs/2026-10-20-deployment-harness-design.md

@@ -1,8 +1,16 @@

# Deployment harness: a whole fake Carlos deployment for rehearsals

-Design, 2026-10-20, **rev 1**. Paul asked: "Can we create a test harness to simulate this that would allow much faster iteration?" He approved a spec for review in the morning, and later the same evening approved spinning the harness up tonight in the dev account, harness only: no writes to the real dev boxes, buckets or signer. Nothing here is built.

+Design, 2026-10-20, **rev 2**. Paul asked: "Can we create a test harness to simulate this that would allow much faster iteration?" He approved a spec for review in the morning, and later the same evening approved spinning the harness up tonight in the dev account, harness only: no writes to the real dev boxes, buckets or signer. Nothing here is built.

-Code is read at main `96db0efe`. The infrastructure is read at `carlosframework/platform-infrastructure` `origin/main` `a0b3f5a` (and its unapplied branch `host-root-owned-opt`). The claim runbook is read at `claim-dev-cohort1-runbook` `8a08fca5` (Option B, delta v2). The offline move plan is read at `moves-offline` `96ea271b`.

+**Rev 2** applies Fable's review of rev 1 (`after236/harness-spec-fable-review.md`, verdict "build it tonight, with the changes below") in full, and platform-1e's ruling on endpoints:

+- **B1:** rev 1's "lenient mode" cannot work: `rootaws.ScrubEnvironment()` (`main.go:26`, `internal/rootaws/environment.go:49-75`) unsets every `AWS_ENDPOINT_URL_*` for euid 0. Replaced by one `//go:build harness` file (platform-1e's ruling), with a no-tag test; listed as the harness's one binary deviation (§3.7).

+- **B2:** the bridge moves to `203.0.113.0/24`, because every instance unit's `IPAddressDeny` covers `10.0.0.0/8` (§3.1).

+- **B3:** the operator shell runs on the harness host, not DOCS-T1 (§3.9).

+- **B4:** the SSM stand-in injects the aws CLI's endpoint; host-init's SSM agent is masked (§3.4, §3.9).

+- **B5:** donor documents are captured over SSM, not serial console output (§3.5).

+- The "verify on day one" checks (§3.13), the reordered 11-step phase 1 (§8.1), the safety additions (§9), the fidelity holes (§2.3) and the open questions' proposed answers (§12).

+Code is read at main `96db0efe`. The infrastructure is read at `carlosframework/platform-infrastructure` `origin/main` `a0b3f5a` and its unapplied branch `host-root-owned-opt`. The claim runbook is read at `claim-dev-cohort1-runbook` (Option B, delta v2). The offline move plan is read at `moves-offline` `96ea271b`.

## 1. The problem

@@ -14,7 +22,7 @@ The dev cohort-1 claim trial took days because each defect showed up only on a r

4. `console-report` one second after a console restart was refused: it takes the console's PID from the console's own report, which still named the old process (`activationConsolePID`, `internal/host/activation_console_report_linux.go:26`).

5. Instance records created with claims off carry no `acq`, so `backfill-collect` refuses them (`claimactivation/backfill.go:95` on the window branch).

-None of these is a logic bug that unit tests catch. The tests use in-process fakes (`leasetest.NewS3Server`, `release.DirStore`, scratch roots). The root runner (`root-runner.py`, a throwaway arm64 EC2 driven over SSM) gave real root and systemd, but its tests build their own scratch trees, so it missed defect 1, and its rehearsal ran the signer phases with other credentials, so it missed defect 3. The defects live in host-init, systemd units, root-owned paths, IAM policy and timing, which is to say in the deployment, not the code.

+None of these is a logic bug that unit tests catch. The tests use in-process fakes (`leasetest.NewS3Server`, `release.DirStore`, scratch roots). The root runner (`root-runner.py`, a throwaway arm64 EC2 driven over SSM) gave real root and systemd, but its tests build their own scratch trees, so it missed defect 1, and its rehearsal ran the signer phases with other credentials, so it missed defect 3. The defects live in host-init, systemd units, root-owned paths, IAM policy and timing: in the deployment, not the code.

The lesson the trial recorded four times: a production-like witness must start from the **infrastructure's** box layout (host-init and bootstrap-box), not only the platform's bootstrap.

@@ -22,9 +30,9 @@ The lesson the trial recorded four times: a production-like witness must start f

### 2.1 Goals

-- A whole deployment (boxes, buckets, console, signer, DNS, TLS) that a runbook or rehearsal runs against end to end, repeatedly, and that resets to a known state in seconds.

-- Boxes built by the **real** user_data from the infrastructure repos, then bootstrapped by the **real** `bootstrap-box.sh`, running the same arm64 binaries and the same embedded units as production.

-- Runbooks as code whose box commands go through one transport. The same script runs against real boxes over SSM and against the harness.

+- A whole deployment (boxes, buckets, console, signer, DNS, later TLS) that a runbook or rehearsal runs against end to end, repeatedly, and that resets to a known state in seconds.

+- Boxes built by the **real** user_data from the infrastructure repos, then bootstrapped by the **real** `bootstrap-box.sh`, running the same embedded units as production and arm64 binaries built from the commit under test **with `-tags harness`** (§3.7). That tag is the one binary deviation, and every run's report says so.

+- Runbooks as code whose box commands go through one transport. The same block scripts run against real boxes over SSM and against the harness.

- IAM checked offline from what the rehearsal actually did, so a missing grant is found before a signing session opens.

- Fault injection for the cases moves and claims care about.

- No production credential or key anywhere in the harness.

@@ -33,24 +41,27 @@ The lesson the trial recorded four times: a production-like witness must start f

|---|---|---|---|---|

-| 1 | `/opt/carlos` carlos-owned | **Yes** | Boxes are provisioned by `dev/host-init.sh` as cloud-init user_data and bootstrapped by `scripts/bootstrap-box.sh`, unmodified. Step 6.2 runs the real `guard-arm`, which calls `OpenProtected` on the real tree and refuses exactly as on leo. | P1 |

-| 2 | `umask 077` makes guard dirs `0700`, console 502 | **Yes** | The runbook's blocks run verbatim through the SSM stand-in, which runs them as root under `/bin/sh` with the agent's `umask 022`, so the block's own `umask 077` takes effect as it did on sophie. Every step ends with the harness's after-step check: `console.service` and `carlos-edge.service` active, and the console answering 200 through sophie's edge. | P1 |

-| 3 | Signer policy lacks `status/consoles/<id>.json` | **Yes, offline; not by MinIO** | The action log shows the signer principal's `GetObject` on `status/consoles/dev-console.json`. Simulated against the policy in `signing-host.md`, it is `implicitDeny`, and the harness fails the rehearsal before any real session opens. MinIO's own policy engine is not relied on. | P1 by hand, P2 automated |

-| 4 | `console-report` one second after restart refused | **Likely** | Same code, same order, same units. The risk is speed: MinIO answers in well under a millisecond where S3 takes tens, which narrows race windows. P2's proxy adds S3-like latency. | P1 |

-| 5 | Records without `acq` refused by `backfill-collect` | **Yes, once the harness holds records created claims-off** | Option B's zero-host path deletes all records, so a harness started empty never meets it. The harness meets it when its state includes a claims-off app (step 4 through the console doors, P1 stretch) or a populated fixture (P2); `backfill-collect` then refuses identically. | P1 stretch, P2 |

+| 1 | `/opt/carlos` carlos-owned | **Yes** | Boxes are provisioned by `dev/host-init.sh` as cloud-init user_data and bootstrapped by `scripts/bootstrap-box.sh`, unmodified. Step 6.2 runs the real `guard-arm`, which calls `OpenProtected` on the real tree and refuses as on leo. Run against both refs (main's host-init, then `host-root-owned-opt`), the harness says "caught, and fixed by the branch", not just "passes". | P1 |

+| 2 | `umask 077` makes guard dirs `0700`, console 502 | **Yes** | The runbook's blocks run verbatim through the SSM stand-in, so the block's own `umask 077` takes effect as on sophie (the stand-in runs `/bin/sh` like the agent, but `ssm.sh` pipes into `bash` itself, so the shell is the runbook's choice either way). After each step: `console.service` and `carlos-edge.service` active, and the console's own listener on sophie answering 200 (203/EXEC and a restart loop fail it). | P1 |

+| 3 | Signer policy lacks `status/consoles/<id>.json` | **Yes, offline; not by MinIO** | The audit log shows the signer principal's `GetObject` on `status/consoles/dev-console.json`. Read against `signing-host.md`'s policy it is `implicitDeny`. **Tonight a person reads the filtered log; P2 automates it** with the mapping table and `simulate-custom-policy`. | P1 by hand, P2 automated |

+| 4 | `console-report` one second after restart refused | **Likely** | Same code, same order, same units. MinIO answers in well under a millisecond where S3 takes tens, which narrows race windows; P2's proxy adds S3-like latency. Best effort tonight. | P1 |

+| 5 | Records without `acq` refused by `backfill-collect` | **Yes, once the harness holds records created claims-off** | Option B's zero-host path deletes all records, so an empty harness never meets it. It does once its state includes a claims-off app (step 4 through the console doors) or a populated fixture; `backfill-collect` then refuses identically. | P2 |

+A sixth from the same trial: the window package's S2 (the migration census walking leo's real `/data`, `/opt/carlos` and `/etc/systemd/system`). A harness box carries its own history's litter, and P2 can replay a real box's tree listing (`leo-tree-*.tsv`: paths, types, modes, owners, no contents) into a harness box, so the census meets a box-shaped tree.

-A sixth, found in the same trial, is worth naming because the harness changes it: the window package's S2 (the migration census walking leo's real `/data`, `/opt/carlos` and `/etc/systemd/system`). A harness box carries its own history's litter, and P2 can replay a real box's tree listing (`leo-tree-*.tsv`: paths, types, modes, owners, no contents) into a harness box, so the census meets a box-shaped tree.

+### 2.3 What it cannot catch: known limits

-### 2.3 What it cannot catch

+These are the fidelity holes: places where the harness passes and dev can fail.

-- **Real IAM enforcement.** Roles, trust policies, resource policies keyed on `aws:PrincipalArn`, KMS key policies and grants, SCPs, session policies, VPC endpoint policies and IAM's eventual consistency. The offline check (§3.8) covers identity policies and the resource policies it is given; it is a simulation. **The real window still watches every IAM-touching step for `AccessDenied`.**

-- **AWS-specific S3 behaviour** (table in §3.6): 403 instead of 404 without `s3:ListBucket`, 409 `ConditionalRequestConflict`, `PermanentRedirect`, throttling, second-resolution timestamps, default SSE.

-- **EC2 lifecycle.** Termination fences (`internal/termfence`), EIP moves, auto-recover, real reboots on real hardware, IMDS hop limits.

-- **Real power loss.** Killing a container keeps the host's page cache, so unsynced writes survive; on a real box they would not.

-- **Scale.** One host and three to five boxes.

-- **Per-box clock skew** until P3's VM box (§6).

-- **Real networks.** Partitions, cross-region latency and packet loss are simulated on one host's bridge.

-- **The outside world.** Let's Encrypt, Route 53, Keymail and SES are stood in for.

+1. **IAM is not enforced at all tonight.** MinIO users have broad policies. MinIO answers 404 where S3 answers 403 to a caller without `s3:ListBucket`. There is no `aws:PrincipalArn`, no KMS key policy or grant, no trust policy, no SCP, session or VPC endpoint policy, and no IAM eventual consistency. Defect 3 is caught only by a person reading the audit log until P2's automated check exists. **Box IAM is simulated offline, so the real window still has its IAM-touching steps watched for `AccessDenied`**, as a per-step checklist by action and object derived from the harness's audit log, not as a sentence.

+2. **Time and speed.** One clock for every box, sub-millisecond MinIO, a page cache shared with the host, leases that never meet a slow 503. Defect 4 reproduces only if the race is lost in the harness too. A green `console-report`/`publish-floor` pair says little about dev's five-minute ticks and tens-of-millisecond S3, so the window's time-sensitive pairs (#312) look safer than they are until P2's latency proxy. All containers share one `boot_id` (no lxcfs); `console-report` compares within one report, so it is fine there, but anything comparing boot ids across boxes is not exercised.

+3. **The binaries and the trust chain are harness builds.** Box, console and signer binaries carry the harness tag (§3.7); channels, identities, floors and the roster are test-signed. Untested: credential expiry and refresh (real instance-profile keys rotate; the stand-in's are static), `PermanentRedirect` (bootstrap's `AWS_REGION` pin, since MinIO accepts any region), 409 `ConditionalRequestConflict`, second-resolution `LastModified` (`db --at`), default SSE and `GetEncryptionConfiguration`, ListObjectsV2 page limits, Route 53.

+4. **EC2 lifecycle.** Termination fences (`internal/termfence`), EIP moves, auto-recover, real reboots on real hardware, IMDS hop limits.

+5. **Power loss and restore.** Killing a container keeps the host's page cache, so unsynced writes survive. A snapshot restore power-cycles every box at once, which the real fleet never does; scenarios never snapshot between the halves of a time-sensitive pair.

+6. **Scale and networks.** One host, three to five boxes; partitions, cross-region latency and loss are simulated on one bridge.

+7. **Per-box clock skew** until P3's VM box (§6).

+8. **The outside world.** Let's Encrypt, Route 53, Keymail and SES are stood in for (Pebble is P2).

+9. **Ref choice.** The ownership defect reproduces only in as-merged mode from main's host-init. Without drift scripts, the harness models a box that was never hand-changed.

### 2.4 Non-goals

@@ -64,53 +75,58 @@ A sixth, found in the same trial, is worth naming because the harness changes it

### 3.1 Shape

```

-DOCS-T1 (operator) dev account, eu-west-1

- harness up / run / down -- SSM --> harness host (t4g.xlarge, dev AMI, no inbound, TTL)

- iam-check (read-only IAM) incus, btrfs pool, bridge hbr0 10.77.0.0/24

- host services, bound to 10.77.0.1 only:

- ssm stand-in :8100 imds stand-in :8169 (DNAT 169.254.169.254)

+DOCS-T1 (driver only) dev account, eu-west-1

+ harness up / run / down -- SSM --> harness host (t4g.xlarge, dev AMI, no inbound, 12 h TTL)

+ iam-check (read-only IAM, P2) user "operator": the operator shell (§3.9)

+ incus, btrfs pool, bridge hbr0 203.0.113.0/24

+ host services, bound to 203.0.113.1 only:

+ ssm stand-in :8100 imds stand-in :8169 (DNAT 169.254.169.254:80)

containers (system containers, systemd as PID 1):

- infra 10.77.0.10 MinIO (+ audit log), Pebble, dnsmasq, Keymail stub

- leo 10.77.0.21 box, home ie, cohort1 (1 vCPU-pair, 1 GiB, as t4g.micro)

- sophie 10.77.0.22 box, home ie, console box (as t4g.micro)

- dennis 10.77.0.23 box, CARLOS_REGION=uk (as t4g.micro)

- signer 10.77.0.30 trusted signing host stand-in

- harness bucket carlos-harness-640012953448 (artefacts, evidence)

- two throwaway t4g.nano "donors" (minutes; §3.5)

+ infra 203.0.113.10 MinIO (+ audit log), dnsmasq, Keymail stub (P2: Pebble)

+ leo 203.0.113.21 box, home ie, cohort1 (as t4g.micro)

+ sophie 203.0.113.22 box, home ie, console box (as t4g.micro)

+ dennis 203.0.113.23 box, CARLOS_REGION=uk (as t4g.micro)

+ signer 203.0.113.30 trusted signing host stand-in

+ harness bucket carlos-harness-640012953448 (artefacts, donors, evidence)

+ two throwaway t4g.nano donors (minutes; §3.5)

```

+The bridge is in TEST-NET-3 on purpose. `carlos-instance@.service:142`, `-gen@`, `-cmd@`, `-seed@`, `-sidecar@`, `carlos-instance-probe.service` and `claimunit_template.go:48` all `IPAddressDeny` `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `100.64.0.0/10`, `169.254.0.0/16` and others, because real boxes reach S3 on public addresses. On a private range, no app instance could reach MinIO.

### 3.2 The host: a throwaway EC2 instance

DOCS-T1 cannot host it: there is no `/dev/kvm` and no sudo, and rootless podman gives neither real uid 0, real ownership, nor a systemd PID 1 that can run the units' sandboxing. Defects 1 and 2 are ownership and mode defects, so real root is the whole point.

-The host copies the root runner's pattern (`root-runner.py`): dev account, eu-west-1, the dev boxes' Debian 13 arm64 AMI (`ami-071babd89cea53fa3`), SSM only, no inbound, `instance-initiated-shutdown-behavior=terminate`, IMDSv2 required, a 60 GiB gp3 root, and a driver that always terminates and prints `LEAKED` if it cannot. Using the dev AMI means the containers run on the dev boxes' own kernel.

+The host copies the root runner's pattern (`root-runner.py`): dev account, eu-west-1, the dev boxes' Debian 13 arm64 AMI (`ami-071babd89cea53fa3`), SSM only, no inbound, `instance-initiated-shutdown-behavior=terminate`, IMDSv2 required with hop limit 1, a 60 GiB gp3 root, and a driver that always terminates and prints `LEAKED` if it cannot. Using the dev AMI means the containers run on the dev boxes' own kernel.

It differs from the root runner in two ways. It has its own role and bucket (§9), because tonight's scope forbids writes to the dev deployment bucket, which is where `root-runner/*` lives. And it is sized for several boxes: `t4g.xlarge` (4 vCPU, 16 GiB).

### 3.3 Boxes: incus system containers, provisioned by the real user_data

-Choice: **incus**, with privileged system containers (`security.privileged=true`, `security.nesting=true`) on a btrfs storage pool.

+Choice: **incus**, with privileged system containers on a btrfs pool: `security.privileged=true`, `security.nesting=true`, and `raw.lxc: lxc.apparmor.profile=unconfined` from the start. The default nested AppArmor profile's mount rules are unchecked against `nosymfollow` bind mounts (`data-instances.mount`), `ProtectProc=invisible`, `PrivatePIDs` and `TemporaryFileSystem`; the host is throwaway, so unconfined costs nothing.

|---|---|---|---|

-| systemd PID 1 running the units' sandboxing (`TemporaryFileSystem` x50, `BindReadOnlyPaths`, `PrivateDevices`, `ProtectKernel*`, `IPAddressDeny`, `Delegate`) | yes, with nesting | yes | partial; cgroup and mount limits |

+| systemd PID 1 running the units' sandboxing (`TemporaryFileSystem` x50, `BindReadOnlyPaths`, `PrivateDevices`, `ProtectKernel*`, `IPAddressDeny`, `Delegate`) | yes, privileged and nested | yes | partial; cgroup and mount limits |

-nspawn has the best systemd compatibility, but tonight it means hand-building rootfs, networking, snapshots and user_data delivery. incus gives all of them, and cloud-init means host-init runs as it does on EC2 (as root, at first boot, with cloud-init's environment and `umask`). Fidelity risk to verify on day one: that every embedded unit starts under its real hardening inside a privileged nested container. A unit that will not start there is a harness defect to fix, never a unit to relax.

+nspawn has the best systemd compatibility, but tonight it means hand-building rootfs, networking, snapshots and user_data delivery. incus gives all of them, and cloud-init means host-init runs as it does on EC2. Whether every embedded unit starts under its real hardening is checked on day one (§3.13). A unit that will not start there is a harness defect to fix, never a unit to relax.

-Each box is launched with the infrastructure repo's user_data, byte for byte. Two sources, chosen per run:

-- **as-merged:** `dev/host-init.sh` (or `host-init.sh.tftpl` rendered with its one variable, `ssm_region`) at a given ref of `platform-infrastructure`, for testing an infrastructure change before it is applied (for example `host-root-owned-opt`, which is not applied).

-- **as-live:** the user_data the real instance was launched with, read once with a read-only `ec2 describe-instance-attribute --attribute userData`. `user_data_replace_on_change = false` (`infra/modules/region-host/main.tf:492`), so a live box may have run an older script than the file on main.

+Each box is created stopped (`incus init`), its shims pushed (§3.4), then started, so cloud-init runs the infrastructure repo's user_data byte for byte. Two sources, chosen per run:

+- **as-merged:** `dev/host-init.sh` (or `host-init.sh.tftpl` rendered with its one variable, `ssm_region`) at a given ref of `platform-infrastructure`, for testing an infrastructure change before it is applied (for example `host-root-owned-opt`).

+- **as-live (P2):** the user_data the real instance was launched with, read once with a read-only `ec2 describe-instance-attribute --attribute userData`. `user_data_replace_on_change = false` (`infra/modules/region-host/main.tf:492`), so a live box may have run an older script than the file on main.

-Hand changes made to real boxes (leo's and sophie's `/opt/carlos` made root-owned by hand in session 3; leo's `CARLOS_POOLS`; the console drop-ins) are recorded as **drift scripts** in the scenario, each citing the runbook record it reproduces. Without them the harness models a box that does not exist.

+host-init at `a0b3f5a` reads no IMDS field, no EBS volume, no hostname and no EIP. It needs apt egress and the SSM agent `.deb` from S3 during provisioning only.

-Containers are sized as the tofu roots declare the real boxes (`t4g.micro` for all three dev boxes: `limits.cpu=2`, `limits.memory=1GiB`, a 20 GiB root quota). zram is absent (§3.4), so memory pressure behaves differently; that is stated, not hidden.

+Hand changes made to real boxes (leo's and sophie's `/opt/carlos` made root-owned by hand in session 3; leo's `CARLOS_POOLS`; the console drop-ins) are recorded as **drift scripts** in the scenario (P2), each citing the runbook record it reproduces.

+Each box gets a fixed `/etc/machine-id`: the box label comes from it (`metrics_loop.go:727`), so names stay stable across snapshot restores. Harness boxes get their own names; the scenario takes box names as parameters and never uses the literal `leo-climbing`. Containers are sized as the tofu roots declare the real boxes (`t4g.micro` for all three dev boxes: `limits.cpu=2`, `limits.memory=1GiB`, a 20 GiB root quota). zram is absent (§3.4), so memory pressure behaves differently.

### 3.4 Shims, each declared

@@ -118,63 +134,74 @@ The harness changes as little as it can, and lists every change. A run's evidenc

|---|---|---|---|

-| `systemd-zram-setup@.service` no-op unit and a `sysctl` wrapper that skips `vm.*` | each box, before cloud-init runs | host-init's last block starts zram and sets `vm.swappiness`; a container cannot, and `set -e` would stop the script there | zram and the swappiness setting |

-| `/etc/systemd/system.conf.d/90-harness.conf`: `DefaultEnvironment=CARLOS_S3_ENDPOINT=...` (strict) plus `AWS_ENDPOINT_URL_S3/STS=...` (lenient only) | each box and the signer | there is no AWS; every unit and every `systemd-run` inherits it | nothing the platform reads from `host.env` |

+| `systemd-zram-setup@.service` no-op unit and a `sysctl` wrapper that skips `vm.*`, pushed before first boot | each box | host-init's last block starts zram and sets `vm.swappiness`; a container cannot, and `set -e` would stop the script there | zram and the swappiness setting |

+| `systemctl mask --now amazon-ssm-agent`, after cloud-init, before snapshot `base` | each box | host-init installs and enables it (`dev/host-init.sh:25-29`); in a box it would take MinIO keys from the IMDS stand-in and loop against real SSM through closed egress | nothing: the SSM stand-in replaces it |

+| `/etc/carlos/harness-endpoint` (root, `0644`): the MinIO URL, `http://` | each box and the signer | read only by the harness build's fallback (§3.7); no shell, unit or `systemd-run` needs an endpoint variable | nothing the default build reads |

+| The SSM stand-in adds `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL_SSM` and `AWS_REGION=eu-west-1` to each command's environment | every `SendCommand` | `bootstrap-box.sh`, `update-platform-binary.sh` and the signer's key fetch run the aws CLI as root on a box, and the aws CLI reads no `CARLOS_*` variable. The real agent's env is otherwise minimal, and so is the stand-in's. `systemd-run` inside a block drops it, which is right: the unit gets `host.env` and the tag's fallback. | nothing the platform reads |

-| Egress allowlist: during provisioning only apt, the SSM agent package URL and the incus image server; after the `base` snapshot nothing but `10.77.0.0/24` | host nftables | boxes must not reach real AWS | nothing; it makes leaks loud |

+| Egress allowlist: during provisioning only apt, the SSM agent package URL and the incus image server; after snapshot `base`, nothing but `203.0.113.0/24`. Container traffic to the host's VPC address and to `169.254.0.0/16` (outside the IMDS DNAT) is dropped. | host nftables | boxes must not reach real AWS | nothing; it makes leaks loud |

### 3.5 Box identity: donor documents

-Claim activation verifies each box by AWS's signed instance identity document: `boxid.Verifier.Enroll` (`internal/boxid/verify.go`) checks the signature under the certificate embedded for the document's region (`internal/boxid/certs/*.pem`, `EmbeddedCerts`), the account, the AWS region and the instance id. There is no override, and `claimactivation.Runner` hardcodes region `ie`, AWS region `eu-west-1` and a 12-digit account (`internal/claimactivation/runner.go:179`, `:190`, `:203`, `:207`, `:242`). A harness box cannot sign its own document.

+Claim activation verifies each box by AWS's signed instance identity document: `boxid.Verifier.Enroll` (`internal/boxid/verify.go:128-163`) checks the signature under the certificate embedded for the document's region (`internal/boxid/certs/*.pem`), the account, the AWS region, the instance id and roster uniqueness. There is no override, and `claimactivation.Runner` hardcodes region `ie`, AWS region `eu-west-1` and a 12-digit account (`internal/claimactivation/runner.go:179`, `:190`, `:203`, `:207`, `:242`). A harness box cannot sign its own document.

-But `Enroll` has no freshness or liveness check: a document stays valid evidence for its instance forever. So the harness uses **donor documents**:

-1. Launch two `t4g.nano` instances in eu-west-1 in the dev account, no instance profile, with user-data that prints `/latest/dynamic/instance-identity/document` and `/signature` to the serial console and powers off (terminate on shutdown).

-2. Read them with `ec2 get-console-output` (read-only). Store them in the harness bucket. They are identifiers, not secrets.

-3. The IMDS stand-in serves each box its donor's document and signature byte for byte, plus `instance-id`, `placement/region`, `mac`, `network/interfaces/macs/...`, `local-ipv4` and `public-ipv4` (both the container's bridge address, so edges and backhaul work), and `iam/security-credentials/<role>` returning that box's MinIO keys. It keys every answer by source address, behind an IMDSv2 token.

+But `Enroll` has no freshness or liveness check: a document stays valid evidence for its instance forever. So the harness uses **donor documents** (approved):

+1. Launch two `t4g.nano` instances in eu-west-1 in the dev account with the **harness** instance profile (which has no dev access, by explicit Deny) and the root runner's user-data (install the SSM agent), tagged `purpose=carlos-harness`.

+2. Read `/latest/dynamic/instance-identity/document` and `/signature` over `send-command`; write them to the harness bucket; terminate. Minutes, and no reliance on buffered Nitro serial output. The documents are identifiers, not secrets.

+3. The IMDS stand-in serves each box its donor's document and signature byte for byte (`boxpublish.go:81-85` reads them raw over IMDSv2), plus `instance-id`, `placement/region` (also the aws CLI's region fallback), `local-ipv4` and `public-ipv4` (both the container's bridge address; `agent.go:3010` reads the public one for the edge door and DNS checks), and `iam/security-credentials/` with its JSON leaf returning that box's MinIO keys. It keys every answer by source address, behind an IMDSv2 token. No reader of `mac` or `network/interfaces` was found, so those are not served.

-`publish-record` (`internal/host/boxpublish.go`), `invite-box`, `enroll-box` and every later verification then run unmodified, against `eu-west-1` and the real certificate. Donors are captured once and reused by every run. The roster's account is the dev account, `640012953448`, because that is what the documents say; the roster itself is signed by the harness's test key and lives only in harness MinIO.

+`publish-record` (`internal/host/boxpublish.go`), `invite-box`, `enroll-box` and every later verification then run unmodified, against `eu-west-1` and the real certificate. The roster's account is the dev account, `640012953448`, because that is what the documents say; the roster itself is signed by the harness's test key and lives only in harness MinIO.

-The alternative, a build of every box binary with a harness certificate added to `certs/`, changes the binary under test. Open question 1.

+Replayed documents cannot touch the real dev roster: enrolling into dev's `control/host-actors.json` needs the dev root release key, which the harness never holds, and dev's enrollment also checks `TofuInstanceID` against the region's tofu output. The donor ids are kept out of every dev bucket. A harness certificate compiled into the binaries would have changed `boxid`, the one thing the claim path exists to prove.

-The harness host's own IMDS (its real role) must stay unreachable from containers: the DNAT rule catches `169.254.169.254:80` from the bridge, and the host's hop limit stays at 1 so a container could not use an IMDSv2 token from the real service even if a rule were wrong.

+The harness host's own IMDS (its real role) stays unreachable from containers: the DNAT rule catches `169.254.169.254:80` from the bridge, other `169.254.0.0/16` traffic from the bridge is dropped, and the host's hop limit 1 means a container could not use an IMDSv2 token from the real service even if a rule were wrong.

### 3.6 Buckets: MinIO

-MinIO, the release CI already pins and verifies by sha256 (`.amadan/ci.d/30-minio-contract`, `RELEASE.2025-09-07T16-13-09Z`), runs in the `infra` container. Buckets, versioning on for all, as the `deployment-buckets` module makes them: the deployment bucket, `data-ie` and `data-uk`. One endpoint serves all of them, because the code has one: `CARLOS_S3_ENDPOINT` (`internal/s3compat/s3compat.go`). MinIO's region is left unset so it accepts any signing region; `AWS_REGION=eu-west-1` and `CARLOS_HIBERNATE_REGION=eu-west-2` work unchanged. `MINIO_DOMAIN` is set and dnsmasq answers `*.s3.harness.test`, so virtual-hosted addressing works too, for clients that do not set path style.

+MinIO, the release CI already pins and verifies by sha256 (`.amadan/ci.d/30-minio-contract`, `RELEASE.2025-09-07T16-13-09Z`), runs in the `infra` container on its bridge address, with no host port published. Buckets, versioning on for all, as the `deployment-buckets` module makes them: the deployment bucket, `data-ie` and `data-uk`. One endpoint serves all of them, because the code has one (`internal/s3compat/s3compat.go`). MinIO's region is left unset so it accepts any signing region. Plain `http://`: root's trust-store overrides are scrubbed (`rootaws.ScrubEnvironment`), so a private CA would be work for nothing.

What the code assumes of S3, and what MinIO gives:

| Assumption | Where | MinIO |

|---|---|---|

-| `If-None-Match: *` and `If-Match` on `PutObject`, 412 on loss | `release.S3Store`, `boxid.Publish`, claim markers, leases | yes (exercised by `TestDeliveredContractMinio`) |

+| `If-None-Match: *` and `If-Match` on `PutObject`, 412 on loss; conditional DELETE | `release.S3Store`, `boxid.Publish`, claim markers, leases | expected yes; **not** covered by `TestDeliveredContractMinio` (`contract_test.go` has no conditional case), so proven at bring-up (§3.13) and recorded in the evidence |

| 409 `ConditionalRequestConflict` on racing conditional writes | `s3CASConflict`, `internal/release/store.go` | **no**: the loser gets 412. P2's proxy injects 409. |

| Versioning: version ids, delete markers, `ListObjectVersions`, `GetObject`/`DeleteObject` by version | claim storage (Plan 2), cleanup scripts | yes |

-| Object lock | none on main | not needed |

-| SSE / KMS on objects | none on main | not needed; `GetEncryptionConfiguration` (infraview) answers "none", which differs from AWS's default SSE-S3 |

-| 404 for a missing key | `s3NotFound` | **always 404**. AWS answers 403 when the caller lacks `s3:ListBucket` on the prefix (the signer policy's `ListForNotFound` exists for this). The IAM check adds a `ListBucket` check for every 404 it sees (§3.8). |

-| Tagging, `x-amz-tagging` on put | objstore scanning | yes, but AGENTS.md records that MinIO accepted a pre-tagged upload an `s3:PutObjectTagging` deny did not stop; harness MinIO policies prove nothing about tagging |

+| Object lock, SSE/KMS on objects | none on main | not needed; `GetEncryptionConfiguration` (infraview) answers "none", unlike AWS's default SSE-S3 |

+| 404 for a missing key | `s3NotFound` | **always 404**. AWS answers 403 when the caller lacks `s3:ListBucket` on the prefix (the signer policy's `ListForNotFound` exists for this). P2's IAM check adds a `ListBucket` check for every 404. |

+| Tagging, `x-amz-tagging` on put | objstore scanning | AGENTS.md records that MinIO accepted a pre-tagged upload a `PutObjectTagging` deny did not stop; harness MinIO proves nothing about tagging |

| `LastModified` precision | PITR (`db --at`) | **differs**: MinIO keeps milliseconds; S3 lists second resolution |

| Latency, throttling (503 SlowDown), 500s, lost responses | everything | **no**; P2's proxy injects them |

| `PermanentRedirect` for a wrong-region client | bootstrap-box.sh's comment | **no** |

| Strong read-after-write | everything | yes |

-### 3.7 Endpoints: strict and lenient

+### 3.7 Endpoints and the sums key: the harness build (the one binary deviation)

+platform-1e's ruling: the endpoint override lives behind `//go:build harness` only, with a test that the default build has none, and the harness lists it as a deviation.

-Two S3 clients on main ignore `CARLOS_S3_ENDPOINT`:

-- the claim's generation-marker client, `internal/host/claimproduction_marker.go:26` (already filed as #298);

-- the fence writer, `internal/hostfence/writer.go:63`, which also assumes a role through STS.

+Why a build tag and not environment variables. `main.go:26` calls `rootaws.ScrubEnvironment()`, which for euid 0 unsets every `AWS_ENDPOINT_URL_*`, `AWS_CA_BUNDLE`, `SSL_CERT_FILE` and `SSL_CERT_DIR` (`internal/rootaws/environment.go:49-75`). Every claim command the runbook runs as root, and every root pass, would never see an SDK endpoint variable. And two root S3 clients ignore `CARLOS_S3_ENDPOINT`: the generation-marker client (`internal/host/claimproduction_marker.go:22-26`, #298) and the fence writer's S3 and STS clients (`internal/hostfence/writer.go:63`). The claim owner rules that those two stay as they are on main.

-So the harness has two modes. **Strict** sets only `CARLOS_S3_ENDPOINT`; a client that ignores it tries real AWS, egress blocks it, and the run reports a provider-neutrality finding naming the call. **Lenient** also sets `AWS_ENDPOINT_URL_S3` and `AWS_ENDPOINT_URL_STS` (honoured by `aws-sdk-go-v2/config` v1.32.6, and by `rootaws.LoadDefaultConfig`, which clears only shared files), pointing both at MinIO, whose STS `AssumeRole` issues temporary keys. Steps 1 to 7 run strict. The window (claims on: markers, fences) needs lenient until #298 and the fence writer are fixed, and each lenient run says so in its evidence.

+**One file, `//go:build harness`**, that:

+- (a) gives `s3compat.Endpoint()` a fallback when `CARLOS_S3_ENDPOINT` is empty: read `/etc/carlos/harness-endpoint`;

+- (b) applies the same endpoint, path style and when-required checksums to the marker client and to the fence writer's S3 and STS clients (MinIO's STS issues temporary keys for `AssumeRole`);

+- (c) replaces the `publicrelease.Default().PubKey` literal (`internal/publicrelease/feed.go:35`, read by `publish-identity` at `internal/host/activation_command_linux.go:305`) with the test sums key.

+A `harness_default_test.go` with **no tag** asserts that the default build has no fallback and keeps the vendor key; `claims_sums_key_test.go` keeps passing. Boxes, console and signer all run the tagged build, so there is no separate signer patch, and rev 1's "box binaries are unmodified" is withdrawn. Every run's report lists "endpoint resolution and sums key: harness build" as its one binary deviation.

+**A separate finding for platform-1e (#267 vs #298).** Root processes do honour `CARLOS_S3_ENDPOINT` today: `release.NewS3Store`/`NewS3StoreIn` (`internal/release/store.go:712`) apply `s3compat.Options()` for every caller, and the root callers include every claims command (`activation_command_linux.go:62`, `:87`, `:201`, `:278`), `relocatepass.go:251`, `:680`, `agent.go:4188`, `:4282`, `claimfiles.go:216` and `internal/lease/endpoint.go:113`. `hostenv.go:65-67` allows the variable in `host.env` because it "selects no box authority", yet it redirects every root bucket write to a store of the file writer's choosing. So "root ignores endpoint overrides" holds for the SDK's `AWS_*` family only, and the marker and fence writer are the exceptions, not the rule. Either #298 is a real bug (two clients drift from the rest), or #267 is incomplete (`s3compat` should refuse the override for euid 0). The harness is indifferent, because the tag covers all of them.

### 3.8 IAM: an offline check from what the rehearsal did

-MinIO cannot enforce AWS IAM faithfully: it has no roles, no trust policies, no `aws:PrincipalArn`, no KMS key policies. The harness gives each principal its own MinIO user with a broad policy (so runs do not stop on MinIO's approximations), and checks IAM offline from a log of what each principal actually did.

+MinIO cannot enforce AWS IAM faithfully. The harness gives each principal its own MinIO user with a broad policy (so runs do not stop on MinIO's approximations), and checks IAM offline from a log of what each principal actually did.

-**Capture.** P1 uses MinIO's audit log (a webhook target to a JSONL sink in `infra`): each entry has the access key, API name, bucket, object, query (`versionId`, `prefix`, `list-type`, `versions`), request headers (`If-Match`, `If-None-Match`, `x-amz-tagging`, `x-amz-copy-source`) and status. P2 replaces it with a small Go S3 proxy in front of MinIO, which records the same fields plus the source box, parses `DeleteObjects` bodies, and is also the fault injector (§6). The SSM stand-in logs `GetParameter` (with decryption: also a `kms:Decrypt` with `PARAMETER_ARN` context), `PutParameter` and `SendCommand`; MinIO's STS log gives `AssumeRole`.

+**Capture.** P1 uses MinIO's audit log (a webhook target to a JSONL sink in `infra`): each entry has the access key, API name, bucket, object, query (`versionId`, `prefix`, `list-type`, `versions`), request headers (`If-Match`, `If-None-Match`, `x-amz-tagging`, `x-amz-copy-source`) and status. P2 adds a small Go S3 proxy in front of MinIO that records the same fields plus the source box, parses `DeleteObjects` bodies, and is also the fault injector (§6). The SSM stand-in logs `GetParameter` (with decryption: also a `kms:Decrypt` with `PARAMETER_ARN` context), `PutParameter` and `SendCommand`; MinIO's STS log gives `AssumeRole`.

-**Mapping.** A table in code, with tests, from each logged request to IAM actions, resource ARNs (with the real bucket names substituted for the harness names) and condition context:

+**Tonight (P1):** the audit log filtered to the signer's key, listed by action and object, read by a person against `signing-host.md`'s policy. Defect 3's row shows as `GetObject status/consoles/dev-console.json`.

+**P2: mapping and simulation.** A table in code, with tests, from each logged request to IAM actions, resource ARNs (real bucket names substituted for the harness names) and condition context:

| Request | Action(s) | Context |

|---|---|---|

@@ -188,55 +215,59 @@ MinIO cannot enforce AWS IAM faithfully: it has no roles, no trust policies, no

| SSM `GetParameter --with-decryption` | `ssm:GetParameter`, `kms:Decrypt` | `kms:EncryptionContext:PARAMETER_ARN` |

| STS `AssumeRole` | `sts:AssumeRole` on the role | |

-Every request also carries `aws:PrincipalArn` = the real role the principal stands for (leo `carlos-dev-host-b`, sophie `carlos-dev-host`, dennis `carlos-dev-host-uk`, signer `carlos-claim-signer`, fence writer `carlos-dev-ie-b-fence-writer`). Step 3 of the runbook learned why: a negated condition matches a missing key.

-**Simulation**, from DOCS-T1 with the operator's read-only IAM rights, exactly as runbook step 3 ran it on 2026-10-18:

-- for applied policy: `aws iam simulate-principal-policy` on the real role ARN, with `--resource-policy` set to the live bucket policy or key policy (with `"Resource": "*"` replaced by the key ARN in a scratch copy, as step 3 had to);

-- for proposed policy (a PR not applied, or `signing-host.md`'s JSON before a session): `aws iam simulate-custom-policy` with the proposed documents, and permissions boundaries when the role has one.

+Every request also carries `aws:PrincipalArn` = the real role the principal stands for (leo `carlos-dev-host-b`, sophie `carlos-dev-host`, dennis `carlos-dev-host-uk`, signer `carlos-claim-signer`, fence writer `carlos-dev-ie-b-fence-writer`), because a negated condition matches a missing key (runbook step 3).

-A run fails on any `implicitDeny` or `explicitDeny` for an action the rehearsal performed successfully, and lists them. Defect 3 is exactly this case.

-**What it cannot do.** SCPs, KMS grants, VPC endpoint policies and session policies are not evaluated. Every new grant still meets IAM's eventual consistency on the real account. The check sees only actions the harness performed, so a code path the harness did not reach is unchecked. Hence the rule this spec keeps: **box IAM is simulated offline, so the real window still has its IAM-touching steps watched for `AccessDenied`.**

+Simulation runs from DOCS-T1 with the operator's read-only IAM rights, as runbook step 3 ran it on 2026-10-18: `simulate-principal-policy` on the real role ARN with `--resource-policy` for applied policy, `simulate-custom-policy` for proposed policy (a PR not applied, or `signing-host.md` before a session). A run fails on any deny for an action the rehearsal performed successfully. Limits are in §2.3, item 1.

### 3.9 SSM, and one transport for runbooks

-The coordinator suggested `machinectl shell` or `nsenter` in the harness and SSM on real boxes. This spec goes one step further: the harness serves the **SSM API itself**, so the runbook's own helpers (`ssm.sh`, `ssm_run`, `box_get`, `box_put`, `box_inspect`), `bootstrap-box.sh` and `update-platform-binary.sh` run unchanged. The only difference is `AWS_ENDPOINT_URL_SSM`, set in the harness shell.

+The harness serves the **SSM API itself**, so the runbook's own helpers (`ssm.sh`, `ssm_run`, `box_get`, `box_put`, `box_inspect`), `bootstrap-box.sh` and `update-platform-binary.sh` run unchanged.

+**The operator shell is on the harness host**, not DOCS-T1. The stand-ins, MinIO, dnsmasq and the console all live on the host's bridge, and DOCS-T1 reaches the host only through real SSM. So the host has an unprivileged user `operator` with the harness `carlos` CLI (linux-arm64, harness build), `awscli`, `jq` and `python3`, and the scenario's block scripts. Its AWS environment is exactly `AWS_ENDPOINT_URL_SSM`, `AWS_ENDPOINT_URL_S3`, the operator's MinIO keys and `AWS_REGION=eu-west-1`. The DOCS-T1 driver sends "run scenario X" through real `send-command`, polls, and pulls the evidence bundle from the harness bucket. Attended mode is `aws ssm start-session` to the host and the same driver with `--attended`, which stops after each block and waits for the operator: that is also how the same scripts run in a real window.

-The SSM stand-in runs on the host (it needs the incus socket), bound to the bridge address. It implements `SendCommand` (`AWS-RunShellScript`), `GetCommandInvocation`, `ListCommandInvocations`, `DescribeInstanceInformation`, and Parameter Store `GetParameter(s)`, `GetParametersByPath`, `PutParameter`, `DeleteParameter`, `DescribeParameters`. It maps instance ids (the donors' ids) to containers. A command runs as root through `incus exec`, under `/bin/sh` (dash, as the real agent's script does), in a per-command working directory, with `umask 022` and the agent's minimal environment, and its results reproduce the real limits: `StandardOutputContent` truncated at 24,000 characters and `StandardErrorContent` at 8,000, `Status` `Failed` on a non-zero exit, `TimedOut` at the execution timeout. Those limits are the reason `box_get` compresses and checks digests, so faking them generously would hide real failures.

+The SSM stand-in runs on the host (it needs the incus socket), bound to `203.0.113.1`. Tonight it implements `SendCommand` (`AWS-RunShellScript`), `GetCommandInvocation`, and Parameter Store `GetParameter(s)`, `PutParameter`, `DeleteParameter`, `DescribeParameters`: all the runbook touches. `ListCommandInvocations` and `DescribeInstanceInformation` wait for P2. It parses the `X-Amz-Target: AmazonSSM.<Op>` header and ignores signatures. It maps instance ids (the donors' ids) to containers. A command runs as root through `incus exec`, under `/bin/sh`, in a per-command working directory, with `umask 022`, the agent's minimal environment plus the three injected variables (§3.4). Results reproduce the real limits: `StandardOutputContent` truncated at 24,000 characters and `StandardErrorContent` at 8,000, `Status` `Failed` on a non-zero exit, `TimedOut` at the execution timeout. `box_get` compresses and checks digests because of those limits, so faking them generously would hide real failures.

-Attended root shells (`ssm start-session`) are not served. Runbook-as-code replaces them with block scripts: each attended block becomes one file sent through `SendCommand`, sourcing a per-window variables file on the box, so later blocks see earlier blocks' variables. The driver has an `--attended` mode that stops after each block and waits for the operator, which is how the same script runs in a real window.

+Attended root shells into boxes (`ssm start-session`) are not served. Runbook-as-code replaces them with block scripts: each attended block becomes one file sent through `SendCommand`, sourcing a per-window variables file on the box, so later blocks see earlier blocks' variables.

### 3.10 The console, and a Keymail stand-in

-The real console binary (`cmd/console`, built at the commit under test) runs on sophie, as on dev. Its unit is a problem: on dev, `console.service` and its drop-ins (`release-key.conf`, `apps-domain.conf`) are hand-written on sophie and in no repository. The harness needs a copy; until one is captured read-only, it uses a reconstruction from the operator notes and marks the console's unit as unverified in every run (open question 5).

+The real console binary (`cmd/console`, harness build) runs on sophie, as on dev. On dev its unit is hand-written on sophie and in no repository (`ReadWritePaths=/data`; drop-ins `release-key.conf` loading `/etc/carlos/release.env`, and `apps-domain.conf`). `install-console` and `console-report` need the unit named `console.service`, `ControlGroup=/system.slice/console.service`, and the drop-ins `90-host-floor.conf` and `91-claim-runtime.conf` (`activation_console_report_linux.go:97-127`). Required environment: `CONSOLE_ORIGIN`, `CARLOS_RELEASE_PUBKEY`, `CARLOS_DEPLOYMENT_BUCKET`, `CARLOS_RELEASE_KEY` (in `release.env`), `CARLOS_CONSOLE_ID`, `CONSOLE_KEYMAIL`, `CONSOLE_DB`, `CARLOS_SYSTEM_OPERATORS`. Until the real unit is read (open question 5), the harness uses a reconstruction and marks the console's unit as unverified in every run.

-Console sign-in goes through Keymail (`CONSOLE_KEYMAIL`, default `https://keymail.dev`; `internal/console/keymail.go`). The harness runs a Keymail stand-in in `infra` serving the five endpoints the console calls (`/.well-known/keymail`, `/oauth/authorize`, `/api/oauth/token`, `/api/oauth/revoke`, the account lookup) and auto-approving one harness operator address, which is on `CARLOS_SYSTEM_OPERATORS`. A curl script signs in, then approves `carlos auth login`'s device code, so the CLI gets an ordinary console token. `/api/app/send` (console mail) answers 503, matching P9's "console mail off".

+Console sign-in goes through Keymail (`CONSOLE_KEYMAIL`; `internal/console/keymail.go:145-215`). The Keymail stand-in in `infra` (about 150 lines) serves `GET /.well-known/keymail` (`{"reauth":false}`), `/oauth/authorize` (auto-approve one harness operator address, redirect to `<origin>/auth/callback?code=&state=`), `POST /api/oauth/token` (`access_token`, `token_type: "bearer"`, `address`), `POST /api/oauth/revoke`, and answers `/api/app/send` with 503 (P9's "console mail off"). A sign-in script drives one browser-less cookie session through the console, then approves `carlos auth login`'s code through the console's own `/auth/cli/start`, `/auth/cli?code=` and `/auth/cli/poll` (`auth.go:695-730`), so the CLI gets an ordinary token.

-P1 needs this: `install-channel --policy --identity` needs a system channel pointer, and system pointers are written only by the console's system update door (`release.PromoteSystem`, called from `internal/console/hostactivation.go:203`), which needs a signed-in operator.

+Phase 1 needs this path: `install-channel --policy --identity` needs system channel pointers, which only the console's system update door writes (`release.PromoteSystem`, from `internal/console/hostactivation.go:203`). It is ordered late in phase 1 with a fallback (H10, §4.3), so a slip there costs the stretch, not the witness.

### 3.11 Release and signing with test keys

Two production trust anchors are involved, and the harness replaces both with keys it generates on the host, in tmpfs, for the host's lifetime:

- **The root release key** (`/carlos/release-key`, `CARLOS_RELEASE_PUBKEY`): a test key from `carlos release-keygen`. Its public half goes into each box's `host.env` through `bootstrap-box.sh`'s `pubkey-file` argument, and into the console's `release.env`. The fake Parameter Store holds the private half at `/carlos/release-key`, so the signer fetches it with `signing-host.md`'s own "Getting the key there" commands.

-- **The CLI update key**, which signs `SHA256SUMS.txt`. `publish-identity` verifies the sums with `publicrelease.Default().PubKey` (`internal/host/activation_command_linux.go:305`), a literal compiled into the binary (`internal/publicrelease/feed.go:35`, pinned equal to `cliUpdatePubKey` by `claims_sums_key_test.go`). An unreleased build cannot have production-signed sums, and the harness never holds that key. So the **signer's** `carlos` binary is built with a one-line harness patch that replaces that literal with the test sums key. Only the signer runs it; `publish-identity` is the only reader of that key on the claim path. Every box binary is unpatched. Later (P3) the literal becomes a build-time variable on main so no patch is needed (open question 2).

+- **The CLI update key**, which signs `SHA256SUMS.txt` and which `publish-identity` checks. The harness build carries a test sums key instead (§3.7).

-A harness release is the README's release recipe with test keys: `carlos-linux-arm64` and `console-linux-arm64` built at the commit under test (`CGO_ENABLED=0`, `-s -w -X .../internal/version.Build=<tag>`), `SHA256SUMS.txt` over them, signed with the test sums key. The tag is the one the real release will carry (for example `v0.28.5`), so the runbook's literal values match. Nothing signed by a test key can install on a real box, because no real box trusts it.

+A harness release: `carlos-linux-arm64` and `console-linux-arm64` built at the commit under test with `-tags harness` (`CGO_ENABLED=0`, `-s -w -X .../internal/version.Build=<tag>`), `SHA256SUMS.txt` over them, signed with the test sums key per the README's recipe. The tag is the one the real release will carry (for example `v0.28.5`), so the runbook's literal values match. Nothing signed by a test key can install on a real box, because no real box trusts it.

-The signer is a container with the trusted host's role (its MinIO user), `jq` and the `carlos` CLI checked by sha256, and its key on `/run`. It runs `claims run` phases exactly as `signing-host.md` describes. "Closing the signing session" becomes: shred the key, `incus delete --force signer`, remove the exchange objects, and check the SSM stand-in's log shows exactly one read of `/carlos/release-key` (the stand-in for the CloudTrail check).

+The signer is a container with the harness build, `jq`, the key on `/run` (tmpfs), and its MinIO user through the IMDS stand-in. `claims run` as root reaches MinIO through `NewS3StoreIn` and the tag's fallback. It runs the phases exactly as `signing-host.md` describes. "Closing the signing session" becomes: shred the key, `incus delete --force signer`, remove the exchange objects, and check the SSM stand-in's log shows exactly one read of `/carlos/release-key` (the stand-in for the CloudTrail check).

### 3.12 DNS, TLS and the edge

-- **DNS:** dnsmasq in `infra`, the bridge's resolver. A reserved test domain, `dev.harness.test`, with explicit records for the console, each app host, and `*.s3.harness.test` for MinIO. `CARLOS_DNS_PROVIDER` is unset: DNS is the operator's, as on a self-hosted deployment.

-- **TLS:** Pebble, Let's Encrypt's test ACME server, in `infra`, resolving through dnsmasq. Boxes point `CARLOS_ACME_DIRECTORY` (already on host.env's allowlist, `internal/hostenv/hostenv.go`) at it and trust its root. This exercises the real ACME path, including runbook step 4.5's trap: only the housekeeper-lease holder orders certificates, and only when a TLS hello reaches it.

-- **The edge:** each box's real `carlos-edge.service`. Backhaul works because the IMDS stand-in gives each box its bridge address as its public address. Clients on the host use `curl --resolve` and Pebble's root, the same commands the runbook uses with real addresses.

+- **DNS:** dnsmasq in `infra`, the bridge's resolver. A reserved test domain, `dev.harness.test`, with explicit records for the console and each app host. `CARLOS_DNS_PROVIDER` is unset: DNS is the operator's, as on a self-hosted deployment.

+- **TLS, P2:** Pebble in `infra`, resolving through dnsmasq; boxes point `CARLOS_ACME_DIRECTORY` (on host.env's allowlist, `internal/hostenv/hostenv.go`) at it and trust its root. This exercises the real ACME path, including runbook step 4.5's trap (only the housekeeper-lease holder orders certificates, and only when a TLS hello reaches it).

+- **The check in P1:** the edge's `:80` serves only ACME HTTP-01 and a redirect to https (`edge.go:3360`), so without a certificate there is no 200 through sophie's edge. Tonight's after-step check hits the console's own listener on sophie (`CONSOLE_ADDR`, loopback) through the SSM stand-in, plus `systemctl is-active console carlos-edge`. That still catches defect 2 (203/EXEC and a restart loop). The edge-path 200 is P2.

+- **The edge:** each box's real `carlos-edge.service`. Backhaul works because the IMDS stand-in gives each box its bridge address as its public address.

+### 3.13 Verified on day one, before anything is built on it

+- **Nested hardening.** After bootstrap, on every box: `systemctl --failed` is empty; `systemctl show -p Result,NRestarts carlos-edge data-instances.mount run-carlos-instances.mount run-carlos-logs.mount` is clean; the journal has no "Failed to set up mount namespacing".

+- **cgroup BPF for `IPAddressDeny`.** The same check: systemd logs `IPAddressDeny` as unsupported if a privileged nested container lacks it.

+- **MinIO conditional writes.** At bring-up: PUT with `If-None-Match: *` twice (the second answers 412), PUT with `If-Match` and a wrong ETag (412), a conditional DELETE. The answers go into the evidence.

+- **cloud-init.** `cloud-init status` says `done` and `/etc/motd` says "carlos host ready" before snapshot `base`.

## 4. Scenario 1: the claim trial, Option B

### 4.1 Commits

-- **Bring-up and the pre-window steps:** main `96db0efe` (v0.28.4 plus #317's `guard-arm` directory modes).

-- **The window onwards:** `claim-empty-cohort-done` (sha to come): 8.1b `backfill-collect`, 8.2 `journal-migrate --window`, 8.2b `backfill-done`, 8.4, step 9, block B4 and acceptance. The harness rolls leo to it with the console's system update door, as G4 does on dev, from the `s7` snapshot.

+- **Bring-up and the pre-window steps:** main `96db0efe` (v0.28.4 plus #317's `guard-arm` directory modes), harness build.

+- **The window onwards:** `claim-empty-cohort-done` (tip `cc90d542` at review; ruled, still moving): 8.1b `backfill-collect`, 8.2 `journal-migrate --window`, 8.2b `backfill-done`, 8.4, step 9, block B4 and acceptance. The harness rolls leo to it with the console's system update door, as G4 does on dev, from the `s7` snapshot. P2.

### 4.2 Steps 1 to 7, through the platform's own commands

@@ -244,10 +275,10 @@ The claim owner's requirement: the harness rebuilds steps 1 to 7 through the pla

| Runbook step | In the harness |

|---|---|

-| 1. Release and roll, claims off | Harness release (§3.11). Boxes bootstrapped by `bootstrap-box.sh` over the SSM stand-in, then the console shipped and promoted direct-bucket (`carlos ship --app console --kind binary`, `carlos promote ... canary/console`) and started. The system channel pointers come from `carlos system update --console <harness> --file ... --platform linux-arm64 --version <tag>`, the "new road". Checks as in the runbook: `system status` current, `x-carlos-version`, `claims help` text. |

+| 1. Release and roll, claims off | Harness release (§3.11). Boxes bootstrapped by `bootstrap-box.sh` over the SSM stand-in, then the console shipped and promoted direct-bucket (`carlos ship --app console --kind binary`, `carlos promote ... canary/console`) and started. System channel pointers from `carlos system update --console <harness> --file ... --platform linux-arm64 --version <tag>` (or H10). Checks as in the runbook: `system status` current, `x-carlos-version` from the console listener, `claims help` text. |

| 2. Empty app segment cleanup | Not applicable: no `apps/bab//` history. B5's check still runs and passes. |

-| 3. IAM | Hand-seeded: MinIO users stand in for the roles (§4.3). The step's denied-operation witnesses are replaced by the offline check (§3.8), which covers the same rows. |

-| 4. Pool and disposable app | `CARLOS_POOLS=cohort1` appended on leo by the runbook's own block. **Stretch for P1:** the app through the console doors (`apps create`, `ship`, `promote`, store-mode `instances enable --pool cohort1`, signed-in `instances create`) using `cmd/moveprobe` at the same commit instead of `wc4`, which exists only in the dev bucket. This is what gives the harness defect 5 (records without `acq`). |

+| 3. IAM | Hand-seeded: MinIO users stand in for the roles (H2). The step's denied-operation witnesses are replaced by the offline check (§3.8). |

+| 4. Pool and disposable app | `CARLOS_POOLS=cohort1` appended on leo by the runbook's own block. The app (`apps create`, `ship`, `promote`, store-mode `instances enable --pool cohort1`, signed-in `instances create`), using `cmd/moveprobe` at the same commit instead of `wc4` (which exists only in the dev bucket), is P2; it is what gives the harness defect 5. |

| 5.0 Box identity records | `systemd-run --wait --pipe -p EnvironmentFile=/etc/carlos/host.env -p User=carlos /opt/carlos/carlos system box publish-record` on leo and sophie through the SSM stand-in; the re-run's "nothing was written" message is checked verbatim. |

| 5.0a Release-stamped console, digests | Pointer digests on for the harness account (`carlos channels digests`, release key in `CARLOS_RELEASE_KEY`), then ship and promote the release-stamped console to `canary/console`. |

| 5.1 Roster and enrollment | On the signer: `claims run` phases `initialize`, `configure` (regions `{ie: eu-west-1, account 640012953448, namespaces, pools cohort1 and console}`), `invite-box` and `enroll-box` for leo and sophie (the donor instance ids as `instance_id`), `invite-console` and `enroll-console`. `box_inspect` and the staging helpers as written. |

@@ -257,7 +288,7 @@ The claim owner's requirement: the harness rebuilds steps 1 to 7 through the pla

| 7 | P3 readers; `console-report` on sophie then `publish-floor` on the signer at once; `home ie`; `evidence-key` on leo, then `bind-key`; `console-config --enable-console-runtime`; the `jq -e` roster check. The session closes (§3.11). |

| 7a Fence credentials | The fence writer's MinIO user keys go into the fake Parameter Store as `/carlos/cohort1/fence/<leo>`; 7a.2's own commands fetch them to tmpfs and run `claims fence-credentials --source - ...`. |

-Each step ends with the after-step check: every expected unit active, the console 200 through sophie's edge, each box's latest report verifying, no new `AccessDenied` or panic in any journal, and a mode and owner listing of `/opt/carlos`, `/etc/carlos` and `/data` saved to the evidence.

+Each step ends with the after-step check: every expected unit active, the console listener answering 200, each box's latest report verifying, no new `AccessDenied` or panic in any journal, and a mode and owner listing of `/opt/carlos`, `/etc/carlos` and `/data` saved to the evidence.

### 4.3 Hand-seeded, with reasons

@@ -266,114 +297,125 @@ Each step ends with the after-step check: every expected unit active, the consol

| H1 | MinIO buckets, versioning on | stand-in for the `deployment-buckets` module, which creates AWS buckets |

| H2 | MinIO users, one per role, plus the signer and fence writer | there is no IAM; enforcement is the offline check |

| H3 | Donor identity documents, served by the IMDS stand-in | AWS signs identity documents and `boxid` trusts only AWS's certificates |

-| H4 | `90-harness.conf` endpoint defaults; the zram and sysctl no-ops; Pebble's root | §3.4: no AWS, and containers cannot do zram |

-| H5 | Test release key and test sums key; the signer's one-line patch | no production key ever; `publish-identity` trusts a compiled-in key |

+| H4 | The shims in §3.4: zram and sysctl no-ops, the masked SSM agent, `/etc/carlos/harness-endpoint`, the aws CLI env injection, fixed machine-ids | no AWS, and containers cannot do zram |

+| H5 | Test release key and test sums key | no production key ever |

| H6 | `console.service` and its drop-ins | written by hand on sophie and in no repository |

| H7 | Console environment: `CONSOLE_ORIGIN`, `CONSOLE_KEYMAIL` (the stand-in), `PLATFORM_APPS_DOMAIN`, `CARLOS_SYSTEM_OPERATORS` | sophie's real values name real hosts and a real IdP |

-| H8 | Drift scripts: hand changes recorded in the runbook (for example session 3's `chown`), applied only in as-live mode | the real boxes are not what host-init alone produces |

-| H9 | Account names: the harness account's sqid replaces `bab` | sqids come from a counter; the scenario takes the account as a parameter |

+| H8 | Drift scripts (P2): hand changes recorded in the runbook, applied only in as-live mode | the real boxes are not what host-init alone produces |

+| H9 | Account names: the harness account's sqid replaces `bab`; box names are parameters | sqids come from a counter; box labels come from machine-ids |

+| H10 | **Fallback only:** the two system channel pointers written by a 30-line tagged helper calling `release.PromoteSystem` with the test key | used only if the Keymail and system update path (§8.1 step 8) is not done by two thirds of the evening; the stand-in then comes first the next day |

+**The binary deviation** is not a hand-seeded step but is listed beside them: every binary is the harness build (§3.7).

Nothing else is written by hand. A step that cannot be reproduced without a new item stops the scenario and adds a row here with its reason, rather than being worked around.

### 4.4 Snapshots for this scenario

-`base` (host-init done), `boot` (step 1), `pre5` (steps 2 to 4 and prechecks), `s7` (step 7 and 7a done, session closed), `pre8` (G3 and G4 on the window build, blocks B1, B2 and B5 done). The window is rehearsed from `pre8` as often as needed.

+`base` (host-init done, as-merged main), `base-opt` (the same from `host-root-owned-opt`), `boot` (step 1), `pre5` (prechecks), `s7` (step 7 and 7a done, session closed), `pre8` (P2: G3 and G4 on the window build, blocks B1, B2 and B5 done). The window is rehearsed from `pre8` as often as needed.

## 5. Lifecycle

| Stage | What happens | Time (estimate) |

|---|---|---|

-| `harness up` | Launch the host (TTL tag, self-terminating shutdown timer); install incus and btrfs-progs; create the pool and bridge; pull `images:debian/13/cloud`; start `infra`; fetch artefacts and donor documents from the harness bucket | 10 to 15 min, once per host |

-| provision | Launch the boxes with the chosen user_data; cloud-init runs host-init (apt dominates); snapshot `base` | about 5 min, parallel |

+| `harness up` | Leak check; launch the host (tags, `shutdown -h +720`); install incus, btrfs-progs and nftables; pool and bridge; pull `images:debian/13/cloud`; start `infra`; fetch artefacts and donor documents from the harness bucket | 10 to 15 min, once per host |

+| provision | `incus init` the boxes with the chosen user_data, push shims, start; cloud-init runs host-init (apt dominates); mask the SSM agent; snapshot `base` | about 5 min, parallel |

| bring-up | Bootstrap, console, the claims-off checks; snapshot `boot` | about 3 min |

| steps 5 to 7 | §4.2; the agent's report ticks dominate (first about a minute after the edge starts, then every 5 minutes plus jitter, so up to about 6 minutes after `guard-arm`) | about 12 min |

| **reset** | Stop all containers, `incus snapshot restore` each to the same named snapshot (MinIO's data is in `infra`, so bucket and boxes agree), start them | **20 to 40 s** |

| run | One scenario from a snapshot | window from `pre8`: about 6 to 8 min |

| collect | Evidence bundle to the harness bucket, then to DOCS-T1's `$TMPDIR` | under 1 min |

-| `harness down` | Terminate the host (retries, `LEAKED` if it fails); check no tagged instance remains | 1 min |

+| `harness down` | Terminate the host and any donor (filtered by `purpose=carlos-harness`, printed first; retries, `LEAKED` if it fails); check nothing tagged remains | 1 min |

**Reset plus a window run is under 10 minutes.** Steps 5 to 7 from scratch are about 12, so they run once per host and are kept as `s7`.

Snapshots are taken with every container stopped, so they are consistent across boxes and buckets. A restore is a power cycle of the whole deployment at a later wall-clock time: leases and short-lived evidence may have expired. That is real behaviour (it is what #312 is about), so scenarios snapshot just before a time-sensitive pair, never between its halves.

-**The evidence bundle** for every run: the scenario transcript (each block, its stdout, stderr and exit), the S3 action log, the SSM and IMDS stand-ins' logs, `journalctl -o export` from each box, mode and owner listings of `/data`, `/opt/carlos`, `/etc/carlos` and `/etc/systemd/system`, the bucket listing with versions, the IAM check's report, and the list of active shims and the run's mode (strict or lenient). It never contains a key: keys live on `/run` and are excluded by path.

+**The evidence bundle** for every run: the scenario transcript (each block, its stdout, stderr and exit), the S3 audit log, the SSM and IMDS stand-ins' logs, `journalctl -o export` from each box, mode and owner listings of `/data`, `/opt/carlos`, `/etc/carlos` and `/etc/systemd/system`, the bucket listing with versions, the conditional-write probe's answers, the IAM review, the active shims and the binary deviation. It never contains a key: keys live on `/run` and are excluded by path, and key checks print presence and a sha256 prefix.

## 6. Fault injection

+All of it is P2 or later; tonight has none.

|---|---|---|---|

-| Kill a box mid-step | `incus stop --force` (all processes killed; disk intact, page cache kept) or `systemctl kill -s KILL carlos-edge` for an agent crash; triggered by time, by a log line, or (P2) by an S3 request ("when leo puts `.../seal.json`, kill leo") | P1 by hand, P2 triggered | moves: a box restarts mid-step, the sealer dies mid-tombstone, the target dies after the flip; claims: crash points between CAS and effect |

+| Kill a box mid-step | `incus stop --force` (all processes killed; disk intact, page cache kept) or `systemctl kill -s KILL carlos-edge` for an agent crash; triggered by time, a log line, or an S3 request through the proxy ("when leo puts `.../seal.json`, kill leo") | P2 | moves: a box restarts mid-step, the sealer dies mid-tombstone, the target dies after the flip; claims: crash points between CAS and effect |

| Restart the agent before reconcile | restart `carlos-edge` after a record change and before its next pass, with the trigger above | P2 | moves' B1 (the hold decided from the record alone); a restarted box's startup latch |

-| Partition a box from a bucket | P1: host nftables dropping a box's traffic to `infra`. P2: the proxy refuses or blackholes one box's requests to one bucket, since every bucket shares one endpoint | P1 coarse, P2 per bucket | moves: partition the sealer from `data-ie` during the final mirror; a source box down at `open` |

-| Clock skew | not possible per container: `CLOCK_REALTIME` has no namespace, privileged containers lack `CAP_SYS_TIME`, and `libfaketime` does not reach static Go binaries' vDSO calls. P3 adds a **VM box**: a separate small EC2 instance in the harness security group with chrony off, which joins the harness network over the VPC | P3 | lease expiry, evidence freshness, `written_at` checks |

+| Partition a box from a bucket | host nftables (coarse), then the proxy refusing or blackholing one box's requests to one bucket, since every bucket shares one endpoint | P2 | moves: the sealer cut from `data-ie` during the final mirror; a source box down at `open` |

+| Clock skew | not possible per container: `CLOCK_REALTIME` has no namespace, privileged containers lack `CAP_SYS_TIME`, and `libfaketime` does not reach static Go binaries' vDSO calls. A **VM box**: a separate small EC2 instance in the harness security group with chrony off, joining the harness network over the VPC | P3 | lease expiry, evidence freshness, `written_at` checks |

## 7. How it is used

-### 7.1 The claim rehearsal before every real window

+### 7.1 The claim rehearsal before a real window

-Before Paul is asked for a window's go, the exact runbook revision runs in the harness against the exact release candidate, from `boot` through acceptance, in `--attended` off mode, and its evidence bundle (with the IAM check's report) is posted on #250. Proposed as a gate (open question 4). The attended real window then runs the same block scripts through real SSM, one block at a time. The window's IAM-touching steps are still watched for `AccessDenied`.

+Before Paul is asked for a window's go, the exact runbook revision runs in the harness against the exact release candidate, from `boot` through acceptance, and its evidence bundle (with the IAM review) is posted on #250. **Advisory** for the next claim window; a **gate** once P2's automated IAM check exists and has caught one planted denial (open question 4). The attended real window then runs the same block scripts through real SSM, one block at a time, with its IAM-touching steps watched for `AccessDenied`.

What it would have changed this week: defects 1, 2 and 4 found in one evening instead of three sessions; defect 3 found before session 4; S1 and S2 found on a box-shaped tree.

### 7.2 The offline move rehearsal

-The offline move plan's runbook P1 ("sophie to dennis, claims off on both") cannot run on dev as-is: leo is a home-placement box with claims on, so the start door refuses (MV-O21), and the plan leaves that gate to a decision. The harness has no such conflict: its topology is chosen per scenario. It also reproduces the flagship's shape (one home box, the console's, plus one UK box), which dev cannot, because dev has two home boxes (#286).

+The offline move plan's runbook P1 ("sophie to dennis, claims off on both") cannot run on dev as-is: leo is a home-placement box with claims on, so the start door refuses (MV-O21), and the plan leaves that gate to a decision. The harness has no such conflict: its topology is chosen per scenario. It also reproduces the flagship's shape (one home box, the console's, plus one UK box), which dev cannot, because dev has two home boxes (#286). The bridge range (§3.1) is what lets `moveprobe` reach MinIO through its instance unit's `IPAddressDeny`.

-Scenario `moves-offline`, from a snapshot with `moveprobe` deployed (a store, a secret, a schedule, a once-timer) and woken on sophie only:

+Scenario `moves-offline` (P3), from a snapshot with `moveprobe` deployed (a store, a secret, a schedule, a once-timer) and woken on sophie only:

- the happy path, every verb by hand with `status` between, and `/report` checks on dennis after `open`;

- abort from `offline-landed` (the plan's rollback rehearsal);

- the fault matrix from §6: the sealer killed between tombstone and final mirror, the target killed after `flip`, a source box down at `seal` and at `open`, the agent restarted between `start` and its next reconcile, the sealer partitioned from `data-ie` during the final round, slow S3 during landing, a lost response on the tombstone write;

- each ending with the plan's own assertions (the record diff, the tombstone's owner, parked paths), read through the CLI and the bucket.

-This is the spawned-app proof the plan's Task 8 leaves to the dev rehearsal, with real units, real root passes and real ownership, plus faults the in-process witness cannot inject.

+This is the spawned-app proof the plan's Task 8 leaves to the dev rehearsal, with real units, real root passes and real ownership, plus faults the in-process witness cannot inject. Advisory for the first offline move, a gate once it runs reliably.

### 7.3 CI

-- **Per merge (amadan CI, no AWS):** the harness's own code only: the action-to-IAM mapping table's tests, the proxy's tests (P2), the stand-ins' tests, and a lint that every scenario block parses. Cost: seconds. The harness itself never runs per merge: it needs AWS credentials, and CI has none.

-- **Nightly (P3), driven from DOCS-T1 or a scheduled job with the harness role:** a fresh host, main's head, scenarios `cohort1` (bring-up to step 7, strict) and `moves-offline` (happy path and abort). About 45 to 60 minutes of host time.

+- **Per merge (amadan CI, no AWS):** the harness's own code only: `harness_default_test.go` (no tag), the tagged file compiling (`go vet -tags harness`), the mapping table's tests and the proxy's tests (P2), the stand-ins' tests, and a lint that every scenario block parses. Seconds. The harness itself never runs per merge: it needs AWS credentials, and CI has none.

+- **Nightly (P3), driven from DOCS-T1 or a scheduled job:** a fresh host, main's head, scenarios `cohort1` (bring-up to step 7) and `moves-offline` (happy path and abort). About 45 to 60 minutes of host time.

- **Weekly:** the fault matrix.

- **On demand:** any branch, before any window.

## 8. Phases, effort and cost

-### 8.1 Phase 1, tonight: hours, rough

+### 8.1 Phase 1, tonight: an ordered checklist

-**P1a: boxes, MinIO, host-init (about 3 agent hours).** Enough to reproduce defects 1 and 2's ground.

-- The approval packet below; the harness bucket, role and security group; the host; the two donors, captured and terminated.

-- incus, the bridge, `infra` with MinIO (audit log on), dnsmasq, Pebble.

-- IMDS and SSM stand-ins, Python 3 standard library only, about 150 and 250 lines.

-- leo, sophie, dennis and signer launched with `dev/host-init.sh` from `platform-infrastructure` main as cloud-init user_data; the shims; snapshot `base`.

-- Harness release at `96db0efe` built on DOCS-T1 (cross-compiled, as the root runner does) and copied through the harness bucket; `bootstrap-box.sh` run unchanged through the SSM stand-in; snapshot `boot`.

-- A read-only layout probe: owners and modes of `/opt/carlos` and its parents, `/etc/carlos/host.env`, against `OpenProtected`'s rule. This shows defect 1's cause at once, before step 6.2 shows its effect.

+What is **verified for real** tonight: host-init's tree and ownership; `bootstrap-box.sh` and the embedded units under their own hardening; `publish-record` against AWS-signed documents and the real certificate; `claims run` phases from a separate signer with a root-owned key; `install-channel --policy --identity`; `guard-arm`'s refusal on a carlos-owned `/opt/carlos` and its directory modes; the console's 203/EXEC under the umask block; `console-report`/`publish-floor` ordering. What is **seeded**: H1 to H9, the binary deviation, and H10 if step 8 slips.

-**P1b: the console and steps 5 to 7 (about 4 agent hours).**

-- Test keys; the signer's patched build; the Keymail stand-in and the sign-in script; the console on sophie.

-- The runbook's steps 5.0 to 7a transcribed into block scripts, values as parameters, helpers verbatim.

-- A first IAM check by hand: the audit log filtered to the signer, each action simulated against `signing-host.md`'s policy with `simulate-custom-policy`.

-- Snapshot `s7`. Stretch: step 4's app (defect 5).

+| # | Step | Its check |

+|---|---|---|

+| 1 | **Approval packet and the harness build** (in parallel with everything until step 6): the `//go:build harness` file and `harness_default_test.go`; `go build -tags harness` for linux-arm64 (`carlos`, `console`) | `go test ./internal/s3compat ./internal/publicrelease -run Harness` green in both builds; the default build's `claims_sums_key_test.go` still passes; the tagged binary reports the test key's prefix (`carlos harness-info` or equivalent) |

+| 2 | **Host up:** launch (root-runner pattern, `t4g.xlarge`, 60 GiB, IMDSv2 hop 1, terminate on shutdown, tags, `shutdown -h +720`); install `incus btrfs-progs nftables`; pool, bridge `203.0.113.0/24`, egress allowlist, IMDS DNAT from the bridge | `incus list` empty; `shutdown --show` prints the deadline; from a scratch container, the token PUT to `169.254.169.254` reaches the stand-in (or is refused), never the host's role |

+| 3 | **infra:** MinIO (versioning on all three buckets, audit webhook to a JSONL sink), dnsmasq | the conditional-write probe (§3.13) answers 412 where S3 would; one audit line per request, `If-*` headers present |

+| 4 | **Donors and the IMDS stand-in** (§3.5) | from a box: token PUT, then `document`, `signature`, `instance-id`, `placement/region`, `public-ipv4`, `iam/security-credentials/` and its leaf; the signature verifies under `internal/boxid/certs/eu-west-1.pem` with `openssl`; both donors terminated |

+| 5 | **Boxes from host-init** (as-merged main; shims pushed before first boot), the signer container, SSM agent masked; snapshot `base` | `cloud-init status` done; the layout probe prints `/opt/carlos carlos:carlos` (defect 1's ground, on purpose); `systemctl --failed` empty |

+| 6 | **SSM stand-in, then `bootstrap-box.sh` unchanged**; snapshot `boot` | `carlos-edge` active with all mounts; the §3.13 hardening checks; `status/boxes/<box>.json` lands in MinIO within about a minute; a 30,000-byte stdout comes back truncated to 24,000 |

+| 7 | **Test keys, harness release, pointer digests, the console shipped and promoted direct-bucket** from the operator shell on the host | the pointer and manifest verify with the test release public key; the digests marker is present |

+| 8 | **The console on sophie** (reconstructed unit, `release.env`), **the Keymail stand-in, `carlos auth login`, `carlos system update`** for the system pointers. If not done by two thirds of the evening: H10, and carry on | the console listener on sophie answers 200 with `x-carlos-version` = the harness tag; `carlos system status` says `current` for all three; `status/consoles/dev-console.json` exists |

+| 9 | **Blocks 5.0, 5.0a, 5.1, 6.1, 6.1a, 6.2** as scripts with parameters, helpers verbatim | 5.0's re-run prints "nothing was written"; `box_inspect` shows both boxes and the console; 6.1a prints its expected lines; **6.2 `guard-arm` refuses with `hostfloor: authenticated floor required`** on main's host-init; then `base-opt` from `host-root-owned-opt`, and 6.2 passes. That pair is tonight's witness of defect 1. |

+| 10 | **6.3, 7, 7a**; snapshot `s7` | the guard directories are `755` after `guard-arm --console`; a deliberate replay of the pre-fix 6.3 block (`umask 077`) gives `700` and 203/EXEC (defect 2); `console-report` within a second of the restart is refused and succeeds once the report moves (defect 4, best effort); `publish-floor` straight after `console-report` succeeds; the `jq -e` roster check prints `true`; 7a's `fence-credentials` installs `root 600` |

+| 11 | **Evidence bundle** to the harness bucket; the audit log filtered to the signer's key, listed by action and object, for the hand IAM review against `signing-host.md` | defect 3's row shows as `GetObject status/consoles/dev-console.json`; the bundle contains no key |

+**Cut tonight**, in order of least regret: Pebble, TLS and the edge-path 200 (step 8's check replaces it); step 4's app (defect 5 and the window need the still-moving branch anyway); as-live mode and drift scripts; `ListCommandInvocations`/`DescribeInstanceInformation`; every fault; the automated IAM check.

+Effort: steps 1 to 6 and blocks 5.0 to 6.2 are a realistic evening; 6.3 to 7a are plausible and sit behind step 8.

**Approval packet** (dev account `640012953448`, eu-west-1, AGENTS.md's scoped-approval rule):

|---|---|---|---|

| S3 bucket `carlos-harness-640012953448` | versioning off, lifecycle: expire everything after 3 days, block public access, tag `purpose=carlos-harness` | pennies | deleted when the harness is retired |

-| IAM role and instance profile `carlos-harness` | `AmazonSSMManagedInstanceCore`; `s3:GetObject`/`PutObject`/`ListBucket` on the harness bucket only; explicit Deny on `arn:aws:s3:::carlos-dev-*` and on `ssm:GetParameter*` for `/carlos/*` | none | deleted with the bucket |

+| IAM role and instance profile `carlos-harness` | `AmazonSSMManagedInstanceCore`; `s3:*` on the harness bucket only; explicit Deny on `arn:aws:s3:::carlos-dev-*`, on `ssm:GetParameter*` for `/carlos/*`, on `ec2:*`, on `iam:*`, and on `ssm:SendCommand` and `ssm:StartSession` | none | deleted with the bucket |

-| One `t4g.xlarge` harness host | dev AMI, 60 GiB gp3, IMDSv2, hop limit 1, terminate on shutdown, tags `purpose=carlos-harness`, `expires=<UTC>` | about $0.15 an hour | at most 24 hours; driver terminate, self-shutdown timer, leftover check |

+| One `t4g.xlarge` harness host | dev AMI, 60 GiB gp3, IMDSv2, hop limit 1, terminate on shutdown, tags `purpose=carlos-harness`, `expires=<UTC>` | about $0.15 an hour | **12 hours**; driver terminate, self-shutdown, leak check |

-Read-only calls also made: `ec2 get-console-output` (donors), optionally `ec2 describe-instance-attribute userData` (as-live mode), and the IAM simulations. Nothing touches the dev boxes, the dev buckets, the real signer, or `/carlos/*` parameters.

+Read-only calls also made: `send-command` to the donors for their documents, optionally `ec2 describe-instance-attribute userData` (as-live mode, P2), and the IAM simulations (P2). Nothing touches the dev boxes, the dev buckets, the real signer, or `/carlos/*` parameters.

### 8.2 Later phases

| Phase | Contents | Effort |

|---|---|---|

-| **P2** | The Go S3 proxy (per box and bucket faults, the lost response, latency, 409s, the action log with source box); the automated IAM check (`harness iam-check`); the window scenario on `claim-empty-cohort-done` (8.1b to acceptance); step 4 and a populated-state fixture (defect 5); tree replay from `leo-tree-*.tsv` (S2); triggered kills and restarts; evidence bundle tooling | 3 days |

-| **P3** | The `moves-offline` scenario and its fault matrix; the VM box for clock skew; nightly and weekly runs; an AMI-derived container rootfs instead of the linuxcontainers image; upstream seams so no patch is needed (the sums key as a build-time variable, `bootstrap-box.sh` extra `host.env` lines, #298 and the fence writer honouring the endpoint, the console unit moved into `platform-infrastructure`); porting the stand-ins to Go with tests; a reaper (EventBridge Scheduler plus a small function terminating `purpose=carlos-harness` instances past `expires`) | 3 days |

-| **P4, optional** | Shadow IAM: harness-named real buckets and roles from the same tofu modules in a sandbox account, with the action log replayed live, so resource policies, KMS key policies and SCPs are enforced for real | 2 days, plus approvals |

+| **P2** | The Go S3 proxy (per box and bucket faults, the lost response, latency, 409s, the action log with source box); the automated IAM check (`harness iam-check`); the window scenario on `claim-empty-cohort-done` (8.1b to acceptance); step 4 and a populated-state fixture (defect 5); Pebble and the edge-path check; tree replay from `leo-tree-*.tsv` (S2); as-live mode and drift scripts; triggered kills and restarts; the account-side **reaper** (EventBridge Scheduler plus a small function terminating `purpose=carlos-harness` instances past `expires`), the only guard that survives a dead driver and a wedged host | 3 days |

+| **P3** | The `moves-offline` scenario and its fault matrix; the VM box for clock skew; nightly and weekly runs; an AMI-derived container rootfs instead of the linuxcontainers image; the console unit moved into `platform-infrastructure`; the stand-ins ported to Go with tests | 3 days |

+| **P4** | Shadow IAM: harness-named real buckets and roles from the same tofu modules in a sandbox account, with the action log replayed live, so resource policies, KMS key policies and SCPs are enforced for real. Before the first production cohort, not before. | 2 days, plus approvals |

### 8.3 Cost

@@ -387,14 +429,15 @@ Read-only calls also made: `ec2 get-console-output` (donors), optionally `ec2 de

## 9. Security

-- **No production credential or key.** Both signing keys are generated on the host, live on its tmpfs and die with it. The Parameter Store the signer reads is the stand-in. The harness role cannot read `/carlos/*` or touch `carlos-dev-*` buckets, by explicit Deny, so a mistaken endpoint variable fails rather than reaching dev.

-- **No real AWS from the boxes.** Boxes hold only MinIO keys. The real IMDS is unreachable from containers (DNAT plus hop limit 1). Egress from boxes is closed after provisioning. A client that ignores the endpoint fails loudly (§3.7).

+- **No production credential or key.** Both signing keys are generated on the host's tmpfs and die with it. The Parameter Store the signer reads is the stand-in.

+- **The host role cannot reach dev.** `s3:*` only on the harness bucket; explicit Deny on `arn:aws:s3:::carlos-dev-*`, `ssm:GetParameter*` for `/carlos/*`, `ec2:*`, `iam:*`, `ssm:SendCommand` and `ssm:StartSession`. The host never needs EC2, IAM or SSM-as-a-client; the driver does those with the operator's profile from DOCS-T1.

+- **The driver acts only on `purpose=carlos-harness`.** The operator's `kass` profile is broad, so every launch, terminate and leak check filters by that tag and prints what it is about to terminate before it does.

+- **No real AWS from the boxes.** Boxes hold only MinIO keys. The real IMDS is unreachable from containers (DNAT, a drop on the rest of `169.254.0.0/16`, hop limit 1). Container traffic to the host's VPC address is dropped. Egress from boxes is closed after provisioning.

- **Test artefacts cannot escape.** Everything the harness signs is signed by test keys no real box or console trusts. Harness releases live only in harness MinIO and the harness bucket.

-- **Donor documents** are identifiers (account, instance id, addresses, AMI), not secrets; the donors had no role.

-- **Scoped account use.** The operator's `kass` credentials are used for the harness's own tagged resources, read-only console output, and read-only IAM simulation; never for dev boxes or buckets.

-- **Teardown is guaranteed three ways tonight.** The driver terminates in a `finally` with retries and prints `LEAKED` if it cannot (the root runner's 2026-10-14 lesson). The host's user-data arms `shutdown -h +<TTL>` with terminate-on-shutdown, so it dies even if the driver does. Every `harness up` first lists `purpose=carlos-harness` instances and refuses to start while an expired one exists. P3 adds the account-side reaper.

-- **Bindings.** Every stand-in binds the bridge address or loopback, never all interfaces, and the security group has no inbound rule.

-- **Logs.** The evidence excludes `/run` and never prints a key; key checks print presence and a sha256 prefix, as the runbook does.

+- **Donors** carry the harness profile (no dev access), their documents are identifiers, and their ids stay out of every dev bucket.

+- **Teardown is guaranteed three ways tonight.** The driver terminates in a `finally` with retries and prints `LEAKED` if it cannot (the root runner's 2026-10-14 lesson). The host's user-data arms `shutdown -h +720` (a **12-hour TTL**) with terminate-on-shutdown, so it dies even if the driver does. Every `harness up` first lists `purpose=carlos-harness` instances, **donors included**, and refuses to start while an expired one exists. P2 adds the account-side reaper.

+- **Bindings.** Every stand-in binds the bridge address or loopback, never all interfaces; MinIO publishes no host port; the security group has no inbound rule.

+- **Logs.** The evidence excludes `/run` and never prints a key.

## 10. Provider neutrality

@@ -402,26 +445,29 @@ The harness is half of the self-hosting proof: MinIO, hosts and DNS, with AWS re

**Runs without AWS today** (P3 can run these with no donor and no AWS at all, using `rootaws`'s off-cloud static credentials file `/etc/carlos/credentials` instead of the IMDS stand-in): deploys, wakes, hibernation, leases, the offline move, the console's ordinary doors, ACME with any directory.

-**Hard AWS dependencies the harness has to stand in for or route around:**

+**Hard AWS dependencies the harness stands in for or routes around:**

- Box identity is AWS's signed document (`internal/boxid`), so claim mode and multi-box regions are AWS-only (Paul's H26, #299). The donors keep the harness on AWS for claims.

- `claimactivation.Runner` hardcodes `ie`, `eu-west-1` and a 12-digit account (`runner.go:179` to `:242`).

-- `publish-identity` trusts a compiled-in sums key (`publicrelease.Default`), so no deployment can publish an identity for its own build.

-- Claim markers and the fence writer ignore `CARLOS_S3_ENDPOINT` (#298; `hostfence/writer.go:63` also needs STS).

+- `publish-identity` trusts a compiled-in sums key (`publicrelease.Default`), so no deployment can publish an identity for its own build. The harness build swaps it.

+- The claim marker client and the fence writer ignore `CARLOS_S3_ENDPOINT` (#298; `hostfence/writer.go:63` also needs STS). The harness build covers them; see §3.7's finding.

- Termination fences read EC2 (`internal/termfence`, `awsinv.NewEC2Reader`); region HA's host.env parameters are SSM plus KMS (`internal/hostenv/awsssm.go`); store provisioning creates IAM users even with an endpoint (`internal/objstore/aws.go`).

- Route 53 (DNS, steering), SES, Cost Explorer, CloudWatch and the SQS termination queue are adapters a non-AWS deployment simply lacks.

- Infrastructure, not platform: host-init installs `awscli` and the SSM agent from an S3 URL, and `bootstrap-box.sh` copies the binary from S3 with the AWS CLI.

-Each strict-mode run lists which of these it met, so the list stays current.

## 11. Where the code lives

-`harness/` in this repository: `up`, `down`, `run` (the driver, Python like `root-runner.py`, tonight), `host-setup.sh`, `stand-ins/` (IMDS, SSM, Keymail), `proxy/` (Go, P2), `iam/` (the mapping table and the simulation driver), `scenarios/cohort1/` and `scenarios/moves-offline/` (block scripts, parameters, after-step checks). Infrastructure files are fetched by ref from `platform-infrastructure`, never copied in, so the harness always runs the user_data under test. The scenarios cite the runbook revision they were transcribed from, and a runbook change without a scenario change is flagged in review.

+- `harness/` in this repository: `up`, `down`, `run` (the driver, Python like `root-runner.py`, tonight), `host-setup.sh`, `stand-ins/` (IMDS, SSM, Keymail), `proxy/` (Go, P2), `iam/` (the mapping table and the simulation driver, P2), `scenarios/cohort1/` and `scenarios/moves-offline/` (block scripts, parameters, after-step checks).

+- The tagged override file and its no-tag test sit beside the code they override (`internal/s3compat`, `internal/publicrelease`, and the two client sites), so a reviewer sees the deviation next to what it changes.

+- Infrastructure files are fetched by ref from `platform-infrastructure`, never copied in, so the harness always runs the user_data under test.

+- The scenarios cite the runbook revision they were transcribed from, and a runbook change without a scenario change is flagged in review.

+## 12. Open questions: proposed answers

-## 12. Open questions for Paul

+Each answer is Fable's recommendation, **PROPOSED, Paul to confirm**.

-1. **Donor identities.** Use real dev-account identity documents from two throwaway `t4g.nano` instances, replayed by the harness's IMDS stand-in, so `boxid` verification runs unmodified on every box? The alternative is a harness certificate compiled into every box binary.

-2. **The signer's build.** Accept a harness-only signer binary carrying the test sums key tonight (a one-line local patch, never merged), and a build-time variable on main later? Box binaries stay unpatched either way.

-3. **Tonight's packet.** Approve §8.1's resources in dev: the harness bucket, role and security group, one `t4g.xlarge` for at most 24 hours, two `t4g.nano` for minutes.

-4. **A gate.** Make a green harness rehearsal of the exact runbook revision and release, with its IAM report, a precondition for every claim window and for the first offline move?

-5. **The dev console's unit.** May an agent read `console.service` and its drop-ins from sophie once (read-only SSM), and should they then move into `platform-infrastructure`?

-6. **Live IAM (P4).** Is offline simulation plus watching the real window for `AccessDenied` enough, or do you want shadow IAM in a sandbox account?

+1. **Donor identities.** PROPOSED, Paul to confirm (already approved in the ruling relayed during review): yes. Captured over SSM from donors launched with the harness profile (§3.5), stored in the harness bucket, donors terminated and in the leak check. A harness certificate compiled into boxes would change `boxid`, the one thing the claim path exists to prove.

+2. **The signer's build.** PROPOSED, Paul to confirm: superseded by platform-1e's ruling. One harness build with `-tags harness` for boxes, console and signer; the override file carries endpoint resolution and the sums key; a no-tag test proves the default build has neither; the report lists the deviation. No separate signer patch, and no claim that box binaries are unmodified.

+3. **Tonight's packet.** PROPOSED, Paul to confirm: approve §8.1's packet, with the `203.0.113.0/24` bridge, the host role's extra Denies, donors with the harness profile, a 12-hour TTL, and the bucket lifecycle at 3 days. `t4g.xlarge` is right for four containers plus MinIO.

+4. **A gate.** PROPOSED, Paul to confirm: not yet. A green rehearsal plus the hand IAM review is **advisory** for the next claim window, and becomes a gate once P2's automated IAM check exists and has caught one planted denial. Gating now on a harness that cannot catch defect 3 unattended would be a false floor. The same for the first offline move once its scenario runs.

+5. **The dev console's unit.** PROPOSED, Paul to confirm: yes, tonight if possible. One read-only `systemctl cat console.service` over SSM on sophie removes H6's "unverified" mark. Then the unit belongs in `platform-infrastructure`, `dev/`, beside `host-init.sh`.

+6. **Live IAM.** PROPOSED, Paul to confirm: offline simulation plus the watched window is enough for dev cohort 1. Shadow IAM (P4) comes before the first production cohort, not before. Until then, the window's IAM-touching steps are a checklist by action and object, derived from the harness's audit log.