amadan
Mandate nested layout in runbooks; add flat->nested migration plan + security assessment
Human
Prompt
Spin up an agent to scope the nested-layout migration, and as part of that make sure all runbooks are updated so this can never happen again. Push the docs now.
From this commit's own message. The prompt ledger has no entries for it — which is what a squashed, rebased or amended commit looks like, and what every commit looks like in a repo without a ledger.
Summary
Finding #1 of the 2026-10-06 assessment: every instance on the flagship runs FLAT (child of the edge agent, uid carlos, no isolation), so one hostile app reads every tenant's DB/secrets and the control-plane credentials. Documents the fix. Runbooks (AGENTS.md, infra/modules/region-host/README.md, infra/remote-host.md, docs/site/cli.md) now state nested is mandatory for every box and every minted instance, flat is a security defect, and each box/instance procedure ends with an on-box layout verification. Adds docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md (mechanism, per-host runbook, rollback, the prerequisite build and four code guardrails) and the full assessment report it references.
Agent notes
IMPORTANT: the runbooks assert a mandate the platform cannot yet ENFORCE. The migration machinery (isolated-exec plans 4-5: converge/journal, SetInstancePaths, the operator verb) is not on main, and the reconciler still mints flat. Migration cannot run until that lands. Target layout is NESTED (carlos-instance@, User=carlos), NOT unit-backed DynamicUser carlos-app@. Enforcement guardrails are specced in the plan doc section 8, implemented by nobody yet.
6 files changed,
+731
−0
AGENTS.md+20 −0docs/security/2026-10-06-access-assessment.md+205 −0docs/site/cli.md+20 −0docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md+445 −0infra/modules/region-host/README.md+27 −0infra/remote-host.md+14 −0
CI passed — run details
diff --git a/AGENTS.md b/AGENTS.md| index 6a925d2c7..520a674b1 100644 |
| --- a/AGENTS.md |
| +++ b/AGENTS.md |
| @@ -319,6 +319,26 @@ anything. |
| byte-for-byte; new apps get a one-channel `edge` pipeline at create; |
| per-channel rules are console-enforced, bake windows box-enforced through |
| the `FirstSeen.Holds` ratchet. |
| +- **A flat instance is a tenant-isolation DEFECT, not a valid layout.** The |
| + mandated layout for every box and every minted instance is nested |
| + (isolated-exec): each instance runs as `carlos-instance@<iid>.service` under |
| + `carlos-instances-<iid>.slice`, in a mount namespace with |
| + `TemporaryFileSystem=/data:ro`, a single `BindPaths` to its own directory, |
| + and `InaccessiblePaths=/etc/carlos`. A FLAT instance — a plain child of |
| + `carlos-edge` with `--socket /run/carlos/<host>.sock --db /data/<host>.db` |
| + and no unit — can read every sibling tenant's `/data/<host>.db` and |
| + `<host>.env` plus the control-plane credentials in `/etc/carlos` and |
| + `/data/litestream.env`: a single hostile app owns the whole control plane |
| + (`docs/security/2026-10-06-access-assessment.md` finding #1, chain A). The |
| + layout is derived from a row's `DBPath`/`Socket` by |
| + `internal/activator/layout.go` `Classify`; `provision.NestedDBPath` / |
| + `NestedSocket` are the nested pair. Every box-provisioning and |
| + instance-create procedure ends with the layout check: no flat managed |
| + instance, every host backed by a `carlos-instance@*` unit (`systemctl |
| + list-units 'carlos-instance@*'` lists it; `pstree` of `carlos-edge` does NOT |
| + show the tenant as a direct child). The flat->nested conversion and the |
| + guardrails that keep flat from recurring are |
| + `docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md`. |
| ## Deploy reality (2026-08-02; convergence rewritten 2026-08-08) |
diff --git a/docs/security/2026-10-06-access-assessment.md b/docs/security/2026-10-06-access-assessment.md| new file mode 100644 |
| index 000000000..8e95cc64f |
| --- /dev/null |
| +++ b/docs/security/2026-10-06-access-assessment.md |
| @@ -0,0 +1,205 @@ |
| +# CARLOS access & tenant-isolation security assessment |
| + |
| +- **Date:** 2026-10-06 |
| +- **Target:** carlos.exciting.io (operator's own deployment — Hetzner, 91.98.31.237) |
| +- **Branch reviewed:** `deploy-exciting-20261006` |
| +- **Scope:** Attempt to gain access to the CARLOS control plane or hosted apps |
| + using only the source (no credentials); and whether one hosted app can escape |
| + tenant isolation. |
| +- **Method:** White-box, source-only, operator-authorized. Read-only — no files |
| + changed, no exploits fired, no credentials used. Four parallel Opus review |
| + passes (signing/bucket, console API, edge, tenant isolation) plus a fifth |
| + pass on sessions/CSRF/invite; top findings re-verified by hand in-code. |
| +- **Live recon performed (read-only):** DNS, TLS cert, HTTP headers, sign-in |
| + page. One active probe (spoofed `Host:` header) was blocked by the local |
| + harness; no deeper live probing was done. |
| + |
| +--- |
| + |
| +## Bottom line |
| + |
| +**No clean unauthenticated remote break was found.** The internet-facing surface |
| +— the edge (gate / door / routing) and the console API — is unusually |
| +well-defended. No gate bypass, no host-confusion routing bypass, no door |
| +privilege escalation, and no cross-tenant IDOR in the console authorization |
| +model (verified across every handler family, including the secrets door |
| +`apiconfig.go` and the customer-database door `apidb.go`). |
| + |
| +**The serious risk is a hostile app escaping tenant isolation and chaining into |
| +control-plane takeover.** The signing scheme resists *forgery* well but is weak |
| +against *replay / rollback* of legitimately-signed records, and the bucket is |
| +the control plane. Whether that chain is live today depends on three box-side |
| +runtime facts listed at the end — they could not be checked from source. |
| + |
| +--- |
| + |
| +## Ranked findings |
| + |
| +Severity key: Critical / High / Medium / Low. Confidence: "confirmed in code" |
| +means verified by reading the exact lines; "runtime check" means source |
| +confirms the mechanism but live exploitability depends on box configuration. |
| + |
| +### 1. Flat instance layout provides no isolation |
| +- **Severity:** Critical · **Confidence:** CONFIRMED LIVE on carlos.exciting.io 2026-10-06 |
| +- **LIVE STATUS (2026-10-06):** every instance on the box is flat. `pstree` of |
| + `carlos-edge.service` (PID 3041305) shows all tenants (`jam`, `jelly-manager`, |
| + `dev-jelly-manager`, `apple-testers-2`, `james-test-1`, `alpha-bravo-123/456`) |
| + as direct child processes of the `carlos` agent, each `--socket |
| + /run/carlos/<host>.sock --db /data/<host>.db` (flat defaults). Zero |
| + `carlos-instance@*` / `carlos-app@*` units. Chain A is live: any one hostile |
| + or compromised app owns every sibling's database and the control-plane |
| + credentials. |
| +- **Where:** `internal/activator/backend_exec.go:812` (`Start` → `startChild` for flat/foreign layout = plain `exec.Command`, no namespace). Credentials file `0600` owned by the `carlos` service user (`infra/remote-host.md:45`). |
| +- **Flaw:** A flat instance is a plain child of the edge agent, uid `carlos`, with full view of `/data` and `/etc/carlos`. A hostile flat tenant can read `/data/litestream.env` (box-wide replicator bucket creds), `/etc/carlos/credentials`, and every sibling's `<host>.db` / `<host>.env` (admin token) / `<host>.config.env` directly. The nested layout closes it (`TemporaryFileSystem=/data:ro`, single `BindPaths`, `InaccessiblePaths=/etc/carlos`), but no minting path produces a nested row — a box runs flat until `carlos system layout migrate` is run and converges it. |
| +- **Exploit:** hostile app → `cat /data/litestream.env` → deployment/replication bucket credentials → see finding #3/#5/#8 for control-plane takeover. |
| +- **Note:** This is the documented transition state the nested-layout effort exists to end (design §8.1); it is the live risk today, not a new bug. |
| +- **Fix:** complete and enforce the nested-layout migration on the flagship. |
| + |
| +### 2. Tenant environment leak (`CARLOS_`-denylist, not allowlist) |
| +- **Severity:** Critical if creds inlined · **Confidence:** mechanism confirmed in code; host.env staging = runtime check |
| +- **Where:** `internal/activator/backend_exec.go:1203` (`childEnv`) — the only env composer; `internal/release/store.go:557` (`NewS3Store` uses `awsconfig.LoadDefaultConfig`, the stock AWS chain that honours inline `AWS_ACCESS_KEY_ID`/`SECRET`/`SESSION_TOKEN`). |
| +- **Flaw:** `childEnv`'s only filter is `strings.HasPrefix(kv, "CARLOS_")`. Every non-`CARLOS_` variable in the edge agent's own environment is copied verbatim into every exec-backed tenant's `/proc/self/environ`. If `/etc/carlos/host.env` stages inline AWS keys, the tenant reads deployment-bucket write credentials from its own environment. This defeats the two namespace defenses the nested design leans on (`InaccessiblePaths=/etc/carlos`, IMDS `IPAddressDeny`), because the secret is already resident in the process environment. |
| +- **Confirmed inert on this deployment (2026-10-06):** `grep` of `/etc/carlos/host.env` returned only `AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials` — no inline `AWS_ACCESS_KEY_ID`/`SECRET`. So the AWS-key-via-env takeover path does NOT fire here; only a path pointer leaks (inert, and useless to a nested instance). The finding remains a latent sharp edge: any future non-`CARLOS_` secret added to `host.env` (a `DATABASE_URL`, SMTP cred, `SENTRY_DSN`) would leak into every tenant. The design works by operator discipline that nothing enforces. `CARLOS_SECRETS_KEY` / `CARLOS_RELEASE_KEY` are correctly caught because they are `CARLOS_`-named; the gap is purely the non-`CARLOS_` namespace. |
| +- **Fix (highest leverage):** make `childEnv` an explicit allowlist, or strip `AWS_*`/known-sensitive prefixes, or hard-fail bootstrap if `host.env` inlines credential keys. One change that makes the credential boundary hold regardless of staging, for both flat and nested. |
| + |
| +### 3. Channel pointers have no anti-rollback guard |
| +- **Severity:** High · **Confidence:** confirmed in code |
| +- **Where:** `internal/release/release.go:432-485` (`verifyPointer`). Contrast `internal/host/systemapply.go:277` which *does* floor system pointers on `PromotedAt`. |
| +- **Flaw:** `verifyPointer` checks signature, grant chain, channel match, and root-only fast-track, but applies no revision / monotonic / promoted-at floor. The bake clock (`bake.go`) is keyed by version, not promoted_at, and deliberately makes a previously-seen version instantly eligible. CAS/If-Match is concurrency control, not authorization — a raw PUT ignores it. |
| +- **Exploit:** a bucket-writer re-PUTs an old, still-validly-signed `channels/stable.json` pointing at a version later patched for an RCE/auth bug (immutable manifest+artifacts remain in the bucket within the ~20-release retention window). Every box accepts it (valid root sig) and adopts it immediately with no bake window → fleet-wide downgrade to a vulnerable build. |
| +- **Fix:** add a promoted-at/revision floor to `verifyPointer`, mirroring `systemapply.go:277`. |
| + |
| +### 4. Public ingress forwards spoofable client-IP headers to tenant apps |
| +- **Severity:** High (impact tenant-app-dependent) · **Confidence:** confirmed in code |
| +- **Where:** `internal/edge/door.go:120-121` strips `doorAddressHeaders` (`X-Real-IP`, `Client-IP`, `True-Client-IP`, `CF-Connecting-IP`, `Fastly-Client-IP`, `X-Client-IP`, `X-Cluster-Client-IP`, `X-Originating-IP`, `X-Forwarded`, `Forwarded-For`) plus `Forwarded`/`X-Forwarded-*` on the door path. The public path (`internal/edge/edge.go` `multiProxy`/`tcpProxy`/`unixProxy`/`fleetProxy` Rewrite → `setForwarded` → `pr.SetXForwarded()`) strips none of them and only manages `X-Forwarded-For/-Host/-Proto`. `SetXForwarded` *appends* the real peer, so a client-supplied leftmost `X-Forwarded-For` survives. |
| +- **Exploit (unauthenticated, no preconditions):** send any of those headers to `https://<tenant>.carlos.exciting.io/`. A tenant framework that reads them as the client address (stock Rails honours `Client-IP`; nginx-habit apps trust `X-Real-IP`; Cloudflare-expecting apps trust `CF-Connecting-IP`) sees an attacker-chosen source IP — defeating in-app IP allowlists, per-IP rate limits, geo-blocks, and audit attribution. |
| +- **Fix:** apply the door's strip list (plus `Forwarded`, and consider `X-ProxyUser-Ip`) on the public path too, and set `X-Forwarded-For` from the real peer with `Set`, not append — so the edge is the sole author of the client-IP headers it vouches for. |
| + |
| +### 5. Cross-record type confusion: a signed Pipeline validates as `production:false` |
| +- **Severity:** Medium · **Confidence:** confirmed in code |
| +- **Where:** `internal/release/production.go:177-191` (`validateProductionFlag`). All root-signed records sign the bare base64 payload with the same root key (`release.go:391`, `grant.go:143`, `pipeline.go:190`, `production.go:117`, `workercontrol.go:123`, `rebind.go:68`, `systempointer.go`); only doorkey is domain-separated. |
| +- **Flaw:** `Pipeline` and `ProductionFlag` both live under `apps/<acct>/<app>/`, both carry `schema`/`account`/`app`/`set_at`. `validateProductionFlag` requires only `schema==1`, matching account/app, non-zero `set_at`; `Production` defaults to `false` and unknown JSON keys are ignored. A root-signed pipeline payload satisfies all of it. |
| +- **Exploit:** copy the app's own `pipeline.json` bytes onto `production.json`. It verifies as a legitimate root-signed `production:false` flag (subject to the box `SetAt` ratchet) → turns off the supply-chain bake window for a production app without the root key. Only this direction is live (Production→Pipeline fails the "at least one channel" check). |
| +- **Fix:** domain-separate root-signed records by kind, as doorkey already does. |
| + |
| +### 6. Door-key revocation is bypassable by replay |
| +- **Severity:** Medium → Low · **Confidence:** confirmed in code |
| +- **Where:** `internal/doorkey/doorkey.go:199-213` (`Verify`) never compares `Revision` against a floor; `VerifyRevision` exists only for the publisher. |
| +- **Flaw/Exploit:** console revokes a leaked door key by publishing a higher-revision record dropping its entry; attacker re-PUTs the previous, still-signed record, which `Verify` re-admits until its ≤72h expiry. Bounded because the door is identity, not a privilege boundary. |
| +- **Fix:** enforce a revision floor on the read side. |
| + |
| +### 7. Open ledgers are unsigned |
| +- **Severity:** Medium → Low · **Confidence:** confirmed in code |
| +- **Where:** `internal/ledger/verify.go:50-99` proves only internal hash-chain consistency; `Head` carries no signature tied to the root key. |
| +- **Flaw:** a bucket-writer can replace HEAD + all entries with a fabricated but internally-consistent history and it verifies clean. Tamper-evidence against corruption, not authenticity against a bucket writer. Impact depends on whether anything trusts ledgers for authority vs. display. |
| +- **Fix:** sign the ledger head, or anchor it to the root key. |
| + |
| +### 8. Box-record immutability relies on AWS-only IAM conditions |
| +- **Severity:** Medium · **Confidence:** terraform confirmed AWS-specific; Hetzner behaviour = runtime check |
| +- **Where:** `infra/modules/deployment-buckets/main.tf` — the `control/boxes/*` create-only / never-deleted / operator-only-bind policy uses `s3:if-none-match`, `s3:ObjectCreationOperation`, and `aws:PrincipalArn`. |
| +- **Flaw:** these are AWS-specific IAM condition keys. Production runs on Hetzner Object Storage, which almost certainly does not enforce them — so the policy that guarantees "no stolen box credential can rebind a box name to another machine" may be a no-op in production. (The box-identity verifier `internal/boxid/verify.go` is itself sound — it requires an operator bind whose `RecordSHA256` matches — so this is a defense-in-depth loss, not a standalone full break.) |
| +- **Fix:** enforce box-record immutability in code / at the storage layer in a way that does not depend on AWS-only policy conditions. |
| + |
| +### 9. Accepted / documented residuals (Low) |
| +- v1 (unbound) secret envelopes decrypt under any app binding; v2 (`CSB2`, AAD-bound) is correct — residual shrinks to zero once v1 is sunset (`internal/appconfig/crypto.go:233`). |
| +- Account-less legacy channel pointers verify under any account's key for the same app/channel (`release.go:406`). |
| +- Scoped-key expiry is backdatable by the holder (controls `promoted_at`); scoped keys are unrevocable short of root rotation; `maxGrantTTL` is 365 days (`grant.go`). "Expiry is hygiene, not revocation." |
| + |
| +### 10. Session token reused as CSRF token + no `no-store` on HTML |
| +- **Severity:** Low → Medium · **Confidence:** confirmed in code; cache behaviour = runtime check |
| +- **Where:** `internal/console/server.go:1800` (`basePage` returns `page{Address: addr, CSRF: raw}`, `raw` = session cookie value); `server.go:1776` (`render()` sets no `Cache-Control`; the `no-store` at `:3259` is a separate secret-showing path). `SessionTTL` 30 days. |
| +- **Flaw:** the 30-day session bearer secret is reflected verbatim into the hidden `csrf` field of every authenticated page and those pages are not marked `no-store`. HttpOnly is undermined — any cached / saved / proxy-logged copy of an authenticated page body, or any future HTML-injection/XSS, yields the literal session token, which can be set as the session cookie for full account takeover. |
| +- **Fix:** derive the CSRF token from the session via HMAC/separate random, and set `Cache-Control: no-store` + `Vary: Cookie` on authenticated HTML. |
| + |
| +### 11. Invite tokens stored in cleartext |
| +- **Severity:** Low · **Confidence:** confirmed in code |
| +- **Where:** `internal/console/store.go:1204` inserts the raw token into `invites.token`; every sibling credential stores only sha256. |
| +- **Flaw:** a read of the console DB yields working invite links. Bounded: single-use (in-tx), 7-day TTL, member-only (never owner), and bound to the invited keymail address (attacker must also sign in as that address). |
| +- **Fix:** store `hashToken(tok)` and look up by hash, matching confirmations. |
| + |
| +--- |
| + |
| +## The two attack chains that matter |
| + |
| +**Chain A — tenant → fleet takeover (the serious one).** |
| +Hostile app on a box → obtain a bucket-write credential (finding #1 sibling-file |
| +read on a flat box, or #2 env-leak if creds are inlined) → finding #3 roll the |
| +fleet back to an old vulnerable release, or #5 disable production bake, or #8 |
| +rebind a box name. A real end-to-end control-plane compromise, gated entirely on |
| +the runtime facts below. |
| + |
| +**Chain B — unauthenticated external.** |
| +The edge and console are solid. The only unauthenticated finding is #4 |
| +(client-IP spoofing), which subverts an app's own IP-based controls but does not |
| +itself breach the platform. Finding #10 is a browser-side account-takeover |
| +primitive but requires first obtaining an authenticated page-body copy or an XSS. |
| + |
| +--- |
| + |
| +## Runtime checks that decide Chain A |
| + |
| +These need the box; they cannot be answered from source. Run them (or an |
| +approved on-box session) to confirm or close the Critical/High items. |
| + |
| +1. **Layout (decides #1).** There is no `carlos` CLI verb for this; layout is a |
| + per-route property derived from each instance's socket/DB path |
| + (`internal/activator/layout.go:78`, `SocketDir = /run/carlos`). Observe it on |
| + the box three ways: |
| + - `ls -la /run/carlos/ ; ls -la /run/carlos/instances/ 2>/dev/null` |
| + — a top-level `<host>.sock` = flat; a socket under `instances/<id>/app.sock` = nested. |
| + - `systemctl list-units --all 'carlos-instance@*' 'carlos-app@*'` |
| + — per-host units present = nested / unit-backed (isolated); absent = flat. |
| + - `pstree -ap "$(systemctl show -p MainPID --value carlos-edge.service)"` |
| + — tenants appearing as children of `carlos-edge` = flat (no isolation). |
| + Flat ⇒ #1 is live-critical today; nested/unit-backed ⇒ #1 closed. |
| +2. **Credential staging (decides #2): CLOSED 2026-10-06.** `/etc/carlos/host.env` |
| + sets only `AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials`, no inline |
| + keys — so the env-leak-of-AWS-keys path does not fire. (Re-check if any |
| + non-`CARLOS_` secret is ever added to `host.env`.) |
| +3. **Hetzner immutability (decides #8):** using a box credential, PUT a probe |
| + object to `control/boxes/__immutability-probe.json` twice and then DELETE it |
| + — if the second PUT or the DELETE succeeds, the AWS-only policy is not being |
| + enforced and #8 is live. Use the probe key, never a real `control/boxes/*` |
| + record. |
| + |
| +--- |
| + |
| +## Confirmed solid (so effort is not re-spent) |
| + |
| +- **Console authz:** single default-deny gate; deliberate byte-identical 404 |
| + membership-oracle defense; constant-time hashed-token compare; service token |
| + blast radius fixed at mint; db-handle HMAC re-bound to the resolved |
| + account/app/host; parameterized SQL throughout; operator doors fail closed to |
| + 404; no SSRF from attacker input. Verified across every handler family, |
| + including `apiconfig.go` (secrets) and `apidb.go` (customer DB). |
| +- **Sessions/CSRF/auth:** every browser POST does a constant-time CSRF compare; |
| + session cookie HttpOnly + Secure(https) + SameSite=Lax; step-up OAuth is |
| + server-side, single-use, state constant-time compared; keymail sign-in is PKCE |
| + S256 with single-use confirmations; CLI device-grant rate-limited and |
| + human-approved; invite account comes from the row (not a body param), single-use, |
| + address-bound. |
| +- **Edge:** gate cookie is per-host HMAC with expiry inside the signed region, |
| + constant-time; gate runs before cache/activator; `SafeNext` re-resolution is |
| + path-only, same-host, no cross-route smuggling; mesh-hop MAC unforgeable from |
| + the internet; door scoping fails closed; static path containment via `openat2` |
| + RESOLVE_BENEATH; purge is response-header-driven and tenancy-confined. |
| +- **Isolation:** control/agent sockets authenticated via `SO_PEERCRED` + pid / |
| + per-wake bearer token (not request headers); `instancefs` uses |
| + `openat2` with RESOLVE_BENEATH/NO_SYMLINKS on every path component; per-app |
| + bucket + scoped IAM user (no shared prefix to escape); host-shape validation |
| + blocks systemd/path injection. |
| +- **Signing:** everything anchored to one pinned root pubkey; grant chain not a |
| + confused deputy; worker-control canonicalized (rejects reorder/dup/unknown |
| + fields, base64 malleability); system & production box ratchets hold; no |
| + ed25519 panic-DoS (pubkey length checked before Verify). |
| + |
| +--- |
| + |
| +## Did the assessment gain access? |
| + |
| +No. Using only the source and no credentials, no turnkey unauthenticated break |
| +was achieved or demonstrated. Several genuine, code-confirmed defects were found; |
| +the most serious (Chain A) is a real tenant-isolation → control-plane takeover |
| +whose live exploitability depends on the three box facts above. The one defect |
| +directly reachable by an unauthenticated external attacker with no preconditions |
| +is #4 (client-IP spoofing), and its impact depends on the hosted app. |
diff --git a/docs/site/cli.md b/docs/site/cli.md| index c4e6a485b..802793b4e 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -1508,6 +1508,18 @@ tenant. `--unit console.service` names the systemd unit that owns the route's |
| process, which is what lets adoption cycle it; leave it off for an ordinary |
| exec child. |
| +An instance route must end up in the **nested** layout, where the instance |
| +runs as its own isolated `carlos-instance@<iid>.service` rather than as a plain |
| +child of the edge. A **flat** instance — a bare exec child sharing the edge's |
| +uid with a full view of `/data` and `/etc/carlos` — has no tenant isolation and |
| +is treated as a security defect, not a valid state: one hostile tenant can read |
| +every sibling's database and the control-plane credentials. After adding |
| +instances to a box, confirm each is backed by a `carlos-instance@*` unit and |
| +none shows as a direct child of `carlos-edge` (`systemctl list-units |
| +'carlos-instance@*'`; `pstree` of the edge). The conversion of an existing flat |
| +box and the guardrails that keep flat from recurring are in |
| +`docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md`. |
| + |
| ### carlos remove |
| Drops one of this box's routes. It stops routing the hostname and never |
| @@ -1649,6 +1661,14 @@ fleet-wide while the account each one is attributed to comes from the |
| console's own box's route registry, so on a multi-box deployment an instance |
| running elsewhere can show up under `(platform)` rather than under its tenant. |
| +What this does not yet surface is each instance's **layout** — whether it runs |
| +nested (isolated, the mandated state) or flat (a plain edge child with no |
| +tenant isolation, a security defect). Until a layout column lands, check it on |
| +the box: `systemctl list-units 'carlos-instance@*'` must list a unit per hosted |
| +instance, and `pstree` of `carlos-edge` must show no tenant as a direct child. |
| +A flat instance is a defect to converge, not a configuration to accept — see |
| +`docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md`. |
| + |
| `--dry-run <dir>` is the self-hoster's leak check. It builds a throwaway fleet |
| stuffed with fixture secrets, runs it through the real publish path into a |
| scratch directory, reads every published object back, and fails if any |
diff --git a/docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md b/docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md| new file mode 100644 |
| index 000000000..c4335b2e3 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md |
| @@ -0,0 +1,445 @@ |
| +# Flat -> nested migration plan and runbook (tenant isolation) |
| + |
| +- **Date:** 2026-10-06 |
| +- **Target:** the flagship box `carlos.exciting.io` (Hetzner, 91.98.31.237), |
| + and the fleet after it. |
| +- **Driver:** finding #1 of `docs/security/2026-10-06-access-assessment.md` |
| + ("Flat instance layout provides no isolation"), confirmed live. Every one of |
| + the seven instances on the box (`jam`, `jelly-manager`, `dev-jelly-manager`, |
| + `apple-testers-2`, `james-test-1`, `alpha-bravo-123`, `alpha-bravo-456`) runs |
| + flat: a plain child of `carlos-edge.service`, uid `carlos`, with a full view |
| + of `/data` and `/etc/carlos`. Any one hostile tenant reads every sibling's |
| + database, every sibling's admin token, and the control-plane credentials |
| + (`/etc/carlos/credentials`, `/data/litestream.env`). Attack chain A. |
| +- **Scope of this document:** investigation + plan + runbook. It changes no Go |
| + source. It states precisely what must be BUILT before the migration can run, |
| + because — see "The honest status" below — **no code path on this branch |
| + produces a nested row or moves a flat instance into one today.** |
| + |
| +--- |
| + |
| +## The honest status: the migration cannot be run today |
| + |
| +The nested layout the fix needs is half-built. The box can *run* a nested |
| +instance, but nothing can *create* one or *convert* a flat one. |
| + |
| +**What exists (isolated-exec plans 1-3, live on this branch):** |
| + |
| +- Classification. `internal/activator/layout.go` `Classify` reads a route and |
| + decides `flat` / `nested` / `fenced` / `conflict` / `foreign` purely from its |
| + `DBPath` and `Socket` columns, a migration-journal probe, and a peer scan. |
| +- The nested spawn. `internal/activator/backend_exec.go` `Start` |
| + (line 811) dispatches a `LayoutNested` row to `startUnit` (line 1066) / |
| + `seedUnit` (line 611), which brings the instance up as |
| + `carlos-instance@<iid>.service` through the root instance-control channel. |
| +- The path helpers. `internal/provision/provision.go` defines every nested |
| + path: `InstanceID`, `InstanceDir`, `NestedDBPath` (line 1277), |
| + `InstanceRunDir`, `NestedSocket` (line 1288), and `LayoutJournalPath` |
| + (line 1303). |
| +- The runtime contract. `internal/host/units/carlos-instance@.service` is the |
| + isolated unit: `TemporaryFileSystem=/data:ro` and `/run/carlos:ro`, a single |
| + `BindPaths=/data/instances/%I /run/carlos/instances/%I`, |
| + `InaccessiblePaths=/etc/carlos /run/carlos-control`, its own |
| + `/dev/shm`, read-only cgroupfs, `User=carlos` inside a private mount |
| + namespace, under `carlos-instances-%i.slice`. |
| +- The fence seam. `internal/host/agent.go` (lines 330-341) wires |
| + `LayoutFacts.Journal` to stat `LayoutJournalPath(dd, host)` and |
| + `LayoutFacts.Peers` to `reg.List`, so a journal file makes a host `fenced` |
| + and an `InstanceID` collision makes it `conflict`. |
| +- `FenceAndStop` (`internal/activator/maintenance.go:50`), the quiesce used to |
| + stop an instance before a move. |
| + |
| +**What does NOT exist (isolated-exec plans 4 and 5, UNIMPLEMENTED here):** |
| + |
| +Verified absent by grep over `internal/`, `cli.go`, `main.go` (non-test): |
| + |
| +- `internal/layout/` — the whole package. No `PlanFor`, no `Journal` writer, no |
| + `Preflight`, no `Pass.Converge`, no `Pass.Recover`. |
| +- `registry.SetInstancePaths` / `ErrPathsChanged` — the compare-and-swap that |
| + flips both path columns atomically. Not in `internal/registry/`. |
| +- `activator.Unfence` / `activator.Service.Fenced` — the release of the fence |
| + `FenceAndStop` installs. |
| +- `release.LayoutIntent`, `LayoutIntentKey` (`control/layout.json`), |
| + `LoadLayoutIntent`, `PutLayoutIntent` — the fleet intent object. Not in |
| + `internal/release/`. |
| +- Any agent wiring that calls `Recover` at start or `Converge` on a serial |
| + move. |
| +- Any operator surface: no `carlos system layout` verb, no `carlos status |
| + --layout`, no `Box.Layout` in `internal/status`. |
| + |
| +The reconciler mint writes **flat** paths unconditionally |
| +(`internal/provision/reconcile.go:390-391`: `rt.Socket = DefaultSocket(...)`, |
| +`rt.DBPath = DefaultDBPath(...)`). So every box runs flat from first mint and |
| +stays flat forever, exactly as the assessment states. Plan 4 of the |
| +isolated-exec series (`docs/superpowers/plans/2026-09-12-isolated-exec-4-layout.md`) |
| +is the design for the missing pass; it has a complete task-by-task spec but is |
| +not landed on `land-ica`. |
| + |
| +**Conclusion:** the migration described below REQUIRES a prerequisite code |
| +change (the "Prerequisite build" section). This document is the plan and the |
| +runbook you run *after* that change lands and is reviewed. Do not attempt a |
| +hand migration with `sqlite3` + `mv` on the live box: moving a flat database |
| +to a nested path and flipping the row by hand, without the journal, the fence, |
| +and the compare-and-swap, races the activator's own wake and the box |
| +replicator, and leaves no recoverable record if it is interrupted. |
| + |
| +--- |
| + |
| +## 1. The exact mechanism: what "nested" is, and how a flat row becomes one |
| + |
| +A route's layout is not a stored field. It is *derived*, every wake, from two |
| +columns of its registry row (`internal/activator/layout.go` `Classify`, lines |
| +103-110): |
| + |
| +``` |
| +flat <=> DBPath == provision.DefaultDBPath(dataDir, host) (/data/<host>.db) |
| + AND Socket == /run/carlos/<host>.sock |
| +nested <=> DBPath == provision.NestedDBPath(dataDir, host) |
| + (/data/instances/<iid>/<host>.db) |
| + AND Socket == provision.NestedSocket(host) |
| + (/run/carlos/instances/<iid>/app.sock) |
| +``` |
| + |
| +`<iid>` is `provision.InstanceID(host)` — the host itself for names <= 40 |
| +bytes (all seven live hosts qualify), otherwise a bounded hash. A row whose two |
| +columns match neither pair is `foreign` and is never touched. |
| + |
| +So **the migration is, at its core, two column edits plus the file moves that |
| +make those columns true**: |
| + |
| +1. Move the on-disk files from the flat names to the nested instance directory. |
| +2. Compare-and-swap the row's `DBPath` and `Socket` from the flat pair to the |
| + nested pair (`registry.SetInstancePaths` — to be built). |
| +3. On the next request, `Classify` now returns `LayoutNested`, `Start` |
| + dispatches to `startUnit`, and the instance comes up as |
| + `carlos-instance@<iid>.service` — isolated. |
| + |
| +Nothing else changes: the live binary symlink, the channel, the environment, |
| +the admin token, the replication rung all stay. The only things that move are |
| +the files and the two columns. This is why the reverse (nested -> flat) is the |
| +exact same operation with the endpoints swapped. |
| + |
| +**Functions involved, in order of the pass (the ones marked (build) do not |
| +exist yet):** |
| + |
| +| Step | Function | Status | |
| +|---|---|---| |
| +| Quiesce the instance | `activator.Service.FenceAndStop(ctx, host, gen)` | exists (`maintenance.go:50`) | |
| +| Derive source/dest paths | `layout.PlanFor(dataDir, socketDir, host, mode)` | (build) | |
| +| Write the journal (O_EXCL) | `layout.WriteJournal` -> `provision.LayoutJournalPath` | (build) / helper exists | |
| +| Move files through the raw dir | `layout.Pass.migrateHost` via `/data/.instances/<iid>/` | (build) | |
| +| Flip the two columns | `registry.SetInstancePaths(host, fromDB, fromSock, toDB, toSock)` | (build) | |
| +| Remove journal, release fence | `layout.RemoveJournal`, `activator.Service.Unfence(host, gen)` | (build) / Unfence (build) | |
| +| Next wake classifies nested | `activator.Classify` -> `ExecBackend.Start` -> `startUnit`/`seedUnit` | exists | |
| + |
| +The trigger that decides *which* mode a box converges to is the fleet intent |
| +object `control/layout.json` (`release.LayoutIntent`, to be built): the agent |
| +reads it on every serial move and runs `layout.Pass.Converge(ctx, nested)` over |
| +every managed row not already nested. An absent object means every box behaves |
| +exactly as it does today — flat — which is why the migration does not start |
| +until an operator writes that object (or runs a direct migrate verb; see the |
| +prerequisite section). |
| + |
| +--- |
| + |
| +## 2. Must the data move? Yes — and the WAL makes ordering load-bearing |
| + |
| +The flat database `/data/<host>.db` and the nested database |
| +`/data/instances/<iid>/<host>.db` are **different files at different paths**. |
| +`NestedDBPath` is deliberately not equal to `DefaultDBPath` in this release |
| +(see the `NestedDBPath` doc comment). The migration must physically move: |
| + |
| +- `<host>.db`, `<host>.db-wal`, `<host>.db-shm` (the database and its WAL |
| + sidecars), |
| +- `<host>.env`, `<host>-sidecar.env`, `<host>.config.env`. |
| + |
| +**Two mechanical constraints the pass must honour (from isolated-exec design |
| +§3.2 / plan 4):** |
| + |
| +- **Move through the RAW directory `/data/.instances/<iid>/`, never through the |
| + bind mount `/data/instances/<iid>/`.** The bind mount is a different mount; |
| + `rename(2)` across it answers `EXDEV`. The nested instance sees its files |
| + through the bind mount at runtime, but the pass must write them through the |
| + raw path. `layout.PlanFor` already encodes raw destinations; a hand `mv` to |
| + `/data/instances/...` would fail or silently copy. |
| +- **The WAL must be checkpointed into the database before the move, or the |
| + `-wal`/`-shm` moved atomically with it.** A live SQLite database in WAL mode |
| + holds uncheckpointed pages in `<host>.db-wal`. Moving `<host>.db` alone, while |
| + the instance is still writing, loses those pages or tears the file. This is |
| + why the instance must be **stopped** (quiesced) first: `FenceAndStop` sends |
| + the instance its SIGTERM drain and proves the process is gone, at which point |
| + SQLite has closed cleanly and `-wal`/`-shm` are either checkpointed away or |
| + safe to move alongside the database. |
| + |
| +**Litestream / replication implications of the path change:** |
| + |
| +A hibernating exec instance's database is replicated by the *activator's own* |
| +per-instance lease (`Acquire`/`Replicate` on `ExecBackend`), keyed on the |
| +route's `DBPath`. After the row flips to `NestedDBPath`, the activator |
| +replicates the new path on the next wake — correct, but it means the box's |
| +litestream view of that database has changed. `carlos ops litestream` |
| +regenerates the replication config from the registry and the data dir; run it |
| +(and restart replication if it prints `changed`) after the row flip so the |
| +box-wide replicator and the registry agree on what each database is, rather |
| +than leaving a stale generation pointed at the vanished flat path. Note the |
| +nested instance stays `User=carlos` (NOT `DynamicUser`), so the agent can still |
| +read and replicate the file — this is the key reason nested, not unit-backed, |
| +is the right target (see §6). |
| + |
| +**Safe ordering for one live instance (the pass enforces this):** |
| + |
| +``` |
| +1. hibernate / drain FenceAndStop(host, gen): SIGTERM drain, prove stop. |
| + An unproven stop (ErrStopUnproven) aborts the move and |
| + leaves the in-process fence installed. Never move a |
| + database whose writer might still be alive. |
| +2. journal WriteJournal(O_EXCL): the host is now `fenced`. |
| +3. move rename the six files through /data/.instances/<iid>/. |
| +4. flip SetInstancePaths(host, flatDB, flatSock, nestedDB, |
| + nestedSock): atomic CAS on both columns. |
| +5. point carlos ops litestream (restart replication if changed). |
| +6. release RemoveJournal + Unfence(host, gen). |
| +7. verify next request wakes it as a unit (see §4 checks). |
| +``` |
| + |
| +--- |
| + |
| +## 3. Fence, journal, and conflict: what a user sees mid-migration |
| + |
| +**While a journal exists** at `/data/layout/<host>.json`, `Classify` returns |
| +`LayoutFenced` with the reason `"layout migration in flight; see <path>"` |
| +(`layout.go:79-81`). In that window: |
| + |
| +- `ExecBackend.Start` refuses with `"<host>: refusing to wake: layout migration |
| + in flight..."` (`backend_exec.go:816-817`). |
| +- `ExecBackend.Seed` refuses the same way (`backend_exec.go:410-411`). |
| +- The edge, unable to wake the upstream, returns a clean **503** to any request |
| + that arrives in the window. It is not a hang and not a crash: the instance is |
| + deliberately held down while its files move. |
| + |
| +The fence survives an agent restart, because it is a file, not in-memory state: |
| +a fresh agent that finds a journal keeps the host fenced until the layout pass's |
| +`Recover` either completes the move or reverts it. The journal is written |
| +`O_EXCL` and validated on read (`layout.Validate`): a journal whose endpoints |
| +are not exactly what `PlanFor` derives for the host keeps the host fenced rather |
| +than acting on a forged or stale record. The window per instance is short |
| +(file renames on the same filesystem are near-instant); the 503s are bounded to |
| +that window plus the health-gated wake on the next request. |
| + |
| +**Conflict detection (`InstanceID` collisions).** `Classify` scans peers |
| +(`LayoutFacts.Peers = reg.List`) and, if two managed hosts resolve to the same |
| +`InstanceID`, returns `LayoutConflict`: **neither host wakes** until one is |
| +removed. A collision is only reachable for two hosts whose *full* names both |
| +exceed 40 bytes and share a 22-byte prefix and a 64-bit sha256 truncation; for |
| +any host <= 40 bytes the `InstanceID` is the host itself, so no collision is |
| +possible. The seven live tenants have short labels, but their registry rows key |
| +on full hostnames (FQDNs), which may be longer — so record this as **checked, |
| +not assumed**: before migrating, list the box's managed rows and confirm no two |
| +resolve to the same `provision.InstanceID` (expected trivially true here, but |
| +verify the actual FQDNs, do not infer it from the short labels). |
| + |
| +--- |
| + |
| +## 4. Per-instance runbook for the seven live hosts |
| + |
| +**Prerequisite:** the "Prerequisite build" section has landed and is reviewed, |
| +the box runs that binary, and the fleet intent `control/layout.json` is either |
| +absent (you drive per-host with a direct migrate verb) or scoped to this box |
| +only — never fleet-wide on the first run. |
| + |
| +**Order — throwaways first, flagship last.** Prove the whole mechanism on an |
| +instance whose downtime nobody notices before you touch a real tenant: |
| + |
| +1. `alpha-bravo-123` (throwaway) |
| +2. `alpha-bravo-456` (throwaway) |
| +3. `james-test-1` (test) |
| +4. `apple-testers-2` (low-stakes) |
| +5. `dev-jelly-manager` (staging for the flagship) |
| +6. `jelly-manager` (flagship) |
| +7. `jam` (flagship — last) |
| + |
| +Stop and run the full verification after #1. Do not proceed to #2 until #1 |
| +wakes nested and serves. Treat #5 as the dress rehearsal for #6/#7. |
| + |
| +**Per host, the steps:** |
| + |
| +1. **Announce / low-traffic window.** Each instance takes a short 503 window |
| + (see §3). Pick a quiet moment for `jam` and `jelly-manager`. |
| +2. **Pre-checks (read-only, on the box):** |
| + - `ls -la /run/carlos/<host>.sock` exists (flat today). |
| + - `test ! -e /data/instances/<iid>/<host>.db` — the nested destination is |
| + empty (the pass refuses two databases). |
| + - no peer shares this host's `InstanceID` (true for all seven). |
| +3. **Run the migrate** for this one host (the built verb / scoped intent), |
| + which performs the §2 ordered pass: quiesce -> journal -> move -> flip -> |
| + point -> release. |
| +4. **Expected downtime:** the quiesce drain (bounded by the 20s stop budget, |
| + usually far less) + the file renames (sub-second on one filesystem) + the |
| + health-gated wake on the first request after release. In practice a few |
| + seconds to low tens of seconds of 503s for that one host; zero for every |
| + other host on the box (the migration is per-host and serial). |
| +5. **Post-migration verification (all three must pass):** |
| + - **Socket moved:** `ls -la /run/carlos/instances/<iid>/app.sock` exists AND |
| + `test ! -e /run/carlos/<host>.sock` (no flat socket left). |
| + - **Backed by a unit:** `systemctl list-units --all 'carlos-instance@*'` |
| + lists `carlos-instance@<iid>.service` as loaded/active (it was absent for |
| + every host before). `systemctl show -p MainPID --value |
| + carlos-instance@<iid>.service` is a real pid. |
| + - **No longer an edge child:** `pstree -ap "$(systemctl show -p MainPID |
| + --value carlos-edge.service)"` no longer shows this tenant as a direct |
| + child of the edge. It now lives under `carlos-instances-<iid>.slice`. |
| + - **Isolation holds (spot-check once, on the first real tenant):** from |
| + inside the instance's namespace the probe `carlos instance-probe` already |
| + asserts: only its own directories, read-only cgroupfs, **no |
| + `/etc/carlos`**, metadata service refused. `carlos-instance-control` |
| + refuses to start isolated instances until the probe passes, so a nested |
| + instance that is up has already proven it cannot read `/etc/carlos` or a |
| + sibling's `/data`. |
| + - **Serves:** `curl -sI --resolve <host>:443:<eip> https://<host>/` returns |
| + the app, and the nested database at `/data/instances/<iid>/<host>.db` is |
| + the live one (`ls -la`). |
| +6. **Record** the host as migrated before starting the next. |
| + |
| +When all seven pass, the box has zero flat managed instances and finding #1 is |
| +closed on this box. Only then widen `control/layout.json` to the fleet. |
| + |
| +--- |
| + |
| +## 5. Rollback for a single instance whose nested wake fails |
| + |
| +The move is reversible by construction: nested -> flat is `PlanFor(..., flat)`, |
| +the same pass with endpoints swapped, and the CAS flips the two columns back. |
| + |
| +- **Mid-move failure (interrupted before or during the row flip).** The journal |
| + is still present, so the host stays fenced. On the next agent start (or pass), |
| + `layout.Pass.Recover` reads the journal, validates its endpoints, sees whether |
| + the row is at `from` (CAS had not run -> finish or revert the file move) or at |
| + `to` (CAS ran -> the move is effectively complete), and converges to a single |
| + consistent state, then removes the journal and unfences. The operator does not |
| + hand-edit anything; they re-run the pass / restart the agent and let `Recover` |
| + settle it. |
| +- **Nested wake fails health (binary refuses the namespace, probe fails, app |
| + will not boot isolated).** Run the reverse converge for that one host |
| + (`mode=flat`): it moves the six files back to the flat names and CASes the row |
| + back to the flat pair. The instance then wakes exactly as it did before the |
| + migration — a plain edge child — and serves. You have lost nothing: same |
| + files, same database, same token. Investigate the isolation failure off the |
| + critical path. |
| +- **Hard floor:** do not roll the box binary below the first release that |
| + understands the layout columns while any journal is unresolved or any row sits |
| + nested — an older binary does not know the nested paths and would treat a |
| + nested row as `foreign` and refuse it. (Plan 4 specifies this guard in |
| + `internal/host/systemapply.go`; see §7.) |
| + |
| +--- |
| + |
| +## 6. Blast radius and risk |
| + |
| +- **This is root-adjacent and fleet-converging.** The nested instance is spawned |
| + by `carlos-instance-control` (root) through the instance-control channel, and |
| + the layout pass runs in the agent but moves files under `/data` and flips |
| + registry rows. The fleet intent object `control/layout.json` converges **every |
| + box** on every serial move. Scope it to one box (`LayoutScope.Box`) for the |
| + flagship run; do not write a fleet-wide `nested` intent until the flagship is |
| + fully migrated and observed. A fleet-wide intent is a fleet-wide change nobody |
| + gets to approve box by box. |
| +- **Nested is `User=carlos`, so there is NO chown and NO data-ownership |
| + change.** This is the single most important risk distinction. The |
| + `carlos-instance@.service` isolation (TemporaryFileSystem + BindPaths + |
| + InaccessiblePaths) keeps the instance uid `carlos`, so the files stay |
| + `carlos`-owned and the agent keeps full read/replicate access. The migration |
| + does not re-own a single byte. |
| +- **Do NOT confuse this with the unit-backed / `DynamicUser` layout.** |
| + `carlos-app@<host>.service` (`Backing == unit`) runs each tenant under a |
| + *DynamicUser* with `StateDirectory=/var/lib/carlos-app/<host>`, chowned to a |
| + fresh ephemeral uid on every start. That is a different, heavier isolation |
| + with real ownership implications: the agent (uid `carlos`) **cannot read** a |
| + DynamicUser's 0700 state dir, so it can neither restore nor replicate that |
| + database — replication must move to the root box-wide `carlos-litestream` |
| + unit, and the route MUST be added with an explicit `-db` or it is replicated |
| + by nobody (a data-loss risk the moment it sleeps; see |
| + `internal/activator/backend_unit.go`'s package comment). **The seven live |
| + hosts should migrate to NESTED (isolated-exec), not to unit-backed.** Choosing |
| + unit-backed here would introduce a chown-per-start and a replication re-wiring |
| + that nested avoids entirely. |
| +- **The 503 window** is real but per-host and bounded (§4). Flagship hosts |
| + (`jam`, `jelly-manager`) go last and in a quiet window. |
| +- **Interrupted migration leaves a fenced host**, which serves 503 until |
| + `Recover` runs. That is fail-safe (no corruption, no cross-tenant exposure) |
| + but it is downtime — so the agent must be healthy and `Recover` must be wired |
| + (it is part of the prerequisite build) before you start. |
| + |
| +--- |
| + |
| +## 7. Prerequisite build (SPEC ONLY — do not implement from this doc) |
| + |
| +Before any of the above can run, land isolated-exec plan 4 |
| +(`docs/superpowers/plans/2026-09-12-isolated-exec-4-layout.md`) plus a trigger. |
| +Minimum set, by file/function: |
| + |
| +1. **`internal/release/layoutintent.go` (create)** — `LayoutIntent`, |
| + `LayoutIntentKey()` = `control/layout.json`, `LoadLayoutIntent`, |
| + `PutLayoutIntent`, `LayoutModeNone/Flat/Nested`, `LayoutScope`. The fleet |
| + intent the agent converges to. |
| +2. **`internal/registry/registry.go` (modify)** — `SetInstancePaths(host, |
| + fromDB, fromSock, toDB, toSock)` + `ErrPathsChanged`: one UPDATE guarded by |
| + both current column values. |
| +3. **`internal/activator/maintenance.go` (modify)** — `Unfence(host, |
| + generation)` and `Fenced(host)`: release the fence `FenceAndStop` installs. |
| + *(Owned by the activator; coordinate — this doc does not edit |
| + `internal/activator`.)* |
| +4. **`internal/layout/` (create)** — `plan.go` (`PlanFor`, raw-dir helpers), |
| + `journal.go` (`WriteJournal` O_EXCL, `ReadJournal`, `Validate`, |
| + `UpdatePhase`, `RemoveJournal`), `preflight.go` (`Preflight`, |
| + `DestinationCheck`, `EnsureDirs`), `pass.go` (`Pass.Converge`, |
| + `Pass.Recover`, `migrateHost`). |
| +5. **`internal/host/agent.go` (modify)** — call `Pass.Recover` at agent start |
| + and `Pass.Converge` on a serial move, reading `control/layout.json`. |
| +6. **A trigger surface** — either plan 5's console door that writes the intent |
| + object plus a `carlos system layout` CLI verb, OR a root-only direct |
| + `carlos system layout migrate --nested [--host <h>]` oneshot for the operator |
| + driving the flagship by hand. Without one of these, nothing writes the intent |
| + and nothing converges. |
| + |
| +--- |
| + |
| +## 8. "Never again": the code-level guardrails (SPEC for a reviewed follow-up) |
| + |
| +A runbook cannot guarantee flat never returns; enforcement must live in code. |
| +Name and place each guard; implement in the reviewed follow-up, not here. |
| + |
| +1. **Mint nested by default.** `internal/provision/reconcile.go` (the mint, |
| + lines ~388-391) currently writes `DefaultSocket`/`DefaultDBPath` |
| + unconditionally. Guard: when the box/fleet layout intent is `nested` (and, |
| + once nested is the fleet default, unconditionally), the mint must write |
| + `NestedSocket(host)`/`NestedDBPath(dataDir, host)` and ensure the raw |
| + instance directory first (`layout.EnsureDirs`). Equivalently, mint flat then |
| + let the first `Converge` nest it — but the mint default is the cleaner place, |
| + so a freshly minted instance is never flat even for one wake. |
| +2. **Converge refuses to leave a managed row flat.** The agent's serial-move |
| + `Pass.Converge(nested)` (`internal/host/agent.go`) must run on every box with |
| + a `nested` intent, and a managed row still flat after a converge attempt must |
| + be surfaced loudly (a logged drift line naming the host and the layout), not |
| + silently tolerated. |
| +3. **Status/doctor reports any flat managed instance as a FAILURE.** Plan 4's |
| + `internal/status/snapshot.go` gains `Box.Layout` (mode + per-row |
| + classification) and `CollectBox` takes a classifier; `carlos status --layout` |
| + (`internal/console/apistatusinstances.go` + root `status_instances.go`) |
| + surfaces it. The guard: a flat *managed* row (`ActivatorManaged` true) must |
| + render as a failing check, in red, not as neutral info — so an operator or a |
| + CI gate sees "box X has N flat instances" as a defect. **Proposed shape (do |
| + not implement here):** a `carlos status --layout` row per managed host with a |
| + `LAYOUT` column reading `nested` (ok) / `flat` (FAIL) / `fenced` / `foreign`, |
| + and a non-zero exit when any managed host reads `flat`. This is the single |
| + smallest check that makes the regression visible everywhere; it belongs in |
| + the plan-4/5 surface, not as a standalone hack. |
| +4. **Downgrade guards (plan 4).** `cli.go` refuses `carlos route |
| + --hibernate=false` on a nested row (clearing hibernation would hand the DB to |
| + the box replicator at the wrong path); `internal/provision/reconcile.go` |
| + refuses the same via flag application; `internal/host/systemapply.go` refuses |
| + a candidate binary that predates the layout while any reversal is incomplete. |
| + These stop an older binary or a careless flag from ever meeting a nested row. |
| + |
| +The property to hold, stated as the assessment states finding #1: **flat is a |
| +security defect, not a valid state.** The mint must not produce it, the converge |
| +must not leave it, and the status surface must fail on it. |
diff --git a/infra/modules/region-host/README.md b/infra/modules/region-host/README.md| index 5088fe84b..97cc2a6f2 100644 |
| --- a/infra/modules/region-host/README.md |
| +++ b/infra/modules/region-host/README.md |
| @@ -116,6 +116,33 @@ curl -sI --resolve <host>:443:<eip> https://<host>/ | grep -i x-carlos-version |
| Use `--resolve` against the box's own EIP rather than trusting DNS: a stale |
| answer from a resolver has produced more than one false "verified" report. |
| +## Layout: every instance must be nested, none flat |
| + |
| +The mandated layout for every box is **nested** (isolated-exec): each hosted |
| +instance runs as its own `carlos-instance@<iid>.service` under a mount |
| +namespace that hides `/data` and `/etc/carlos` from it. A **flat** instance — |
| +a plain child of `carlos-edge` on `/run/carlos/<host>.sock` + `/data/<host>.db` |
| +with no unit — has no isolation at all: one hostile tenant reads every other |
| +tenant's database and the control-plane credentials |
| +(`docs/security/2026-10-06-access-assessment.md` finding #1). Flat is a |
| +security defect, not a valid state for a box that hosts more than one tenant. |
| + |
| +After the box has adopted any instances, verify none is flat: |
| + |
| +```sh |
| +# Per-host units must exist (absent = flat, no isolation): |
| +systemctl list-units --all 'carlos-instance@*' |
| +# No tenant may be a direct child of the edge: |
| +pstree -ap "$(systemctl show -p MainPID --value carlos-edge.service)" |
| +# Nested sockets live under instances/<iid>/, never top-level: |
| +ls -la /run/carlos/ /run/carlos/instances/ |
| +``` |
| + |
| +A top-level `/run/carlos/<host>.sock`, a tenant showing as an edge child, or an |
| +empty `carlos-instance@*` unit list all mean the box is flat and must be |
| +converged to nested before it carries real tenants — see |
| +`docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md`. |
| + |
| ## Termination evidence |
| The module creates a queue, `<iam_name>-terminations`, that receives an |
diff --git a/infra/remote-host.md b/infra/remote-host.md| index f08826c0a..e07bd1533 100644 |
| --- a/infra/remote-host.md |
| +++ b/infra/remote-host.md |
| @@ -51,6 +51,20 @@ GOOS-GOARCH — linux-arm64 or linux-amd64; a 32-bit OS resolves nothing). |
| - `journalctl -u carlos-edge` shows "tunnel connected". |
| - The box appears in `carlos status -live` within one heartbeat |
| (5 minutes + jitter). |
| + - **Layout:** once the box hosts any instance, confirm it is nested, not |
| + flat. The mandated layout is nested (isolated-exec): each instance runs |
| + as its own `carlos-instance@<iid>.service` under a mount namespace that |
| + hides `/data` and `/etc/carlos`. A flat instance — a plain child of |
| + `carlos-edge` on `/run/carlos/<host>.sock` + `/data/<host>.db`, no unit — |
| + has no tenant isolation: one hostile app reads every sibling's database |
| + and the control-plane credentials |
| + (`docs/security/2026-10-06-access-assessment.md` finding #1). Flat is a |
| + security defect, not a valid state. Check: |
| + `systemctl list-units --all 'carlos-instance@*'` lists a unit per host, |
| + and `pstree -ap "$(systemctl show -p MainPID --value carlos-edge.service)"` |
| + shows no tenant as a direct child of the edge. If a box is flat, converge |
| + it before it carries real tenants — see |
| + `docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md`. |
| ## Edge side (once per edge box, not per remote host) |