claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.288 · DOCS-T1
Why: a read-only audit of main on 2026-10-03 found that tenants on a shared box weren't isolated. Every exec tenant ran as one user with /data writable, so it could read other tenants' data and the platform's secrets. There were also gaps in domains, mail, storage and credentials. Nobody outside is on the platform yet; the goal is to have all of it fixed before anyone is. Paul's decisions are recorded on each discussion.
Merged to main
| Discussion | Fix | Commit | |---|---|---| | #218, #191 | Bucket and IAM user names can't collide across stores | 2129e26 | | #220 | Release pointers and secret envelopes bound to their account; relay deny list | b94ad23 | | #219 | Purge revokes tokens and door keys and sweeps ledgers; invite accept by POST | f36ad36 | | #216 | Members can't adopt an SES identity they don't own; unverified claims lapse | be3338d | | #126 | Plan 4: the nested layout and layout migrate/revert, with holds and quarantine | 01f890c | | #214 (1–3), #178 | CPU/IO limits, disk floor, own-build-only /opt/carlos, no loopback or private egress, private IPC, Landlock | 09814f7 | | #215 | Opt-in proof of domain control; account namespaces reserved; fleet boxes see only their own claim bodies | 9bed97b | | #214 (4–5), #50 | Only operators add upstreams; the console trusts forwarded IPs only with the edge's signature | 1c43c2c |
Each was Fable-reviewed, with every finding fixed or recorded as an accepted risk in the branch description. The user-facing copy was approved by Paul.
In flight
#126 plan 5, branch tenant-iso-plan5: the console door, carlos system layout migrate|revert|status, and minting nested on nested boxes. LayoutFloor is v0.25.0. Its review fixes are in progress.
Steps to production (each needs operator approval where marked)
Before cutting v0.25.0: check the flagship and Tito console audit logs to confirm no app or service token adds instance upstreams (#50), since they would now get 403.
Cut v0.25.0 from a main containing 01f890c, after plan 5 lands. It is the layout floor.
Roll v0.25.0, fleet first, then the console. The domain proof (#215) and the edge's XFF signature (#214) both depend on this order. Approval.
Before the first layout migrate, check on one dev box that: - the isolation probe is ready; - Landlock is in the booted kernel's lsm= list; - resolv.conf names a non-loopback upstream; - the carlos-app@/opt/carlos change works; - carlos-instance-control reports its mountpoint-* lines as ok.
carlos system layout migrate --box <dev box>, then dev, then flagship one box at a time, watching carlos status --layout until no box shows the mixed warning. Approval per §8.6.
Flagship console settings:CONSOLE_RESERVE_PLATFORM_DOMAINS=true (believed set but unverified) and CONSOLE_DOMAIN_PROOF=true. Set CONSOLE_SECRETS_ENVELOPE_V2=true after the boxes roll, then re-set existing secrets (#220). Approval.
#217: add the *.oncarlos.com Public Suffix List entry. This needs a _psl TXT record and a public pull request from Paul. Approval.
#245: give /data its own volume with per-instance project quotas, and alert on low disk. Approval.
Open, not blocking launch
#216 item 3: shared SES reputation. SES tenants with ses:TenantName enforced is the cheap option.
#219: door keys left behind when an app is renamed (same account only).
#220: phase out v1 envelopes and pointers without an account, after step 6.
#50: the own-fleet upstream allowance and the manifest opt-in.
#126: Region HA plan 1b. The placer must refuse a relocation target whose layout differs from the intent, and /data/.installed and /data/hostguard can be forged by flat tenants.
#226: tests read ~/.carlos/config, and a test deletes TMPDIR during a run.
/data has no nosuid mount (pinfra/bootstrap).
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.288 · DOCS-T1
Overnight 2026-10-06/07: status and the migration runbook
Ready to merge once Paul approves the strings (copy review open; every branch below is Fable-approved and CI green or pending):
tenant-iso-plan5 @5f9186b2: the console door and carlos system layout migrate|revert|status; LayoutFloor v0.25.0; the downgrade guards (console and box side); new instances minted nested on nested boxes.
tenant-iso-default-nested (on plan5): with no intent recorded, a ready box is nested by default; CARLOS_LAYOUT_DEFAULT=flat opts a box out; a recorded intent always wins.
tenant-iso-cookies @3fd49738 (#217): the edge makes any tenant cookie whose Domain reaches beyond its own account host-only; the cache never replays a trailer Set-Cookie.
Done tonight (#217):
oncarlos.com renewed to 2029-08-01.
abuse@ and psl@carloku.com forward to the operator (carloku-infrastructure PR #5, applied).
The abuse contact is on carloku.com (c0123a7).
The PSL pull request is drafted but held: below the PSL's 2,000–3,000-user threshold.
Migration to nested: the plan for approval (flagship, AWS account 271376211898, boxes ie/us/uk/au plus the console)
Merge plan5, then default-nested, then cookies (in that order, each CI-green).
Confirm in the flagship and Tito console audit logs that no app or service token adds upstreams (#50).
Cut v0.25.0 from that main, following the README recipe. This is public: GitHub release, Homebrew, apt. Release notes go through copy review first.
Dev first: roll v0.25.0 to the dev fleet. Check: - carlos-instance-probe is ready; - Landlock is in the booted kernel's lsm= list; - resolv.conf names a non-loopback upstream; - the carlos-app@/opt/carlos view works; - the mountpoint-* lines are ok; - instances move to nested on their own, and carlos status --layout shows (nested by default) with no mixed warning.
Flagship: with the default on, boxes migrate unattended once v0.25.0 lands and the probe passes. Awake instances get 1 hour of grace, then are stopped and moved. For box-by-box control, record carlos system layout revert (or set CARLOS_LAYOUT_DEFAULT=flat) before rolling, then migrate --box one box at a time. Roll order: boxes first, then the console (the domain-proof and XFF-signature rules depend on it). barbara-tinkering's failing roll gate needs looking at first.
Verify: carlos status --layout shows every activator row nested and no WARNING lines; spot-check a tenant can't read /data or reach 127.0.0.1:8090.
Then the flagship console settings: CONSOLE_RESERVE_PLATFORM_DOMAINS=true (confirm), CONSOLE_DOMAIN_PROOF=true, CONSOLE_SECRETS_ENVELOPE_V2=true.
Rollback: before any box is nested, an ordinary rollback works. After that: carlos system layout revert, wait for every row to show flat, then roll back. The console and box guards enforce that order.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.292 · DOCS-T1
v0.25.0 published (tag 95ad32cf → main 2b2cc879): GitHub release, Homebrew (tap #26) and apt (v0.25.0-1 on canary/apt), all verified from the public side.
Dev rolled to v0.25.0 (carlos-dev-ie, carlos-dev-ie-b, through update-platform-binary.sh). The edge is active; the layout mounts and control socket are active; Landlock is in the kernel's LSM list; resolved's upstream is 172.31.0.2; the boxes log "nested by default".
Blocker found: the probe reports ready=false on both boxes. The new resolver-ports check lists interface addresses over netlink, which the probe unit's RestrictAddressFamilies refuses ("address family not supported by protocol"). Fail-closed: the boxes stay flat and nothing breaks, but nothing migrates either. A fix is on fix-probe-resolver-ports and will ship as v0.25.1. The flagship roll waits for v0.25.1.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.292 · DOCS-T1
Dev end-to-end test on platform 3de0b0fc (v0.25.0 + probe fix), 2026-10-07: pass.
The probe is ready on both dev boxes.
A throwaway on-demand app moved to nested on its own, after a 1h grace, deferred, then hibernating. It woke nested and served through the edge.
From inside, every escape failed: /data, the registry, litestream.env, acme, other instances, other pids, loopback, private ranges, IMDS and other versions. DNS and outbound HTTPS worked.
A new instance was minted nested from the start.
Found and being fixed before the flagship roll:
A nested tenant is PID 1 in its namespace, so a Go app with no SIGTERM handler ignores SIGTERM. Every stop becomes a 20s timeout plus SIGKILL, and the unit is left failed. Branch fix-instance-init adds a minimal init (signal forwarding and reaping).
Minor, follow-up: a layout refusal for a row whose db doesn't exist yet doesn't arm the retry timer, so it waits for the next serial move.
v0.25.1 will carry the probe fix and the init fix.
amadan