amadan
Isolated exec spec: revision 4 after codex round 3
Round 3 findings committed unedited. All 7 accepted: the control service authenticates the edge's MainPID instead of a token and the edge refuses root peers; the exit record is advisory and lives in the run directory (ExecStopPost=+ keeps the unit's tmpfs); UDP EPERM probe for the IMDS deny; 150s seed deadline; nothing minted nested in this release; no manual memory.oom.group.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZ1ZvWj7naf73GjTX8MBv5
2 files changed,
+701
−470
docs/superpowers/specs/2026-09-11-isolated-exec-design.md+610 −470docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R3.md+91 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md| index 62ee58737..dfb734168 100644 |
| --- a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| +++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| @@ -1,8 +1,9 @@ |
| # Isolated exec — every exec-backed instance in its own namespace |
| -**Date:** 2026-09-11 · **Status:** revision 3, after codex rounds 1 and 2 |
| -(19 + 14 findings, all accepted; §13) · **Requested by:** discussion #126 |
| -(Tito, with box evidence), approved in direction by Paul on 2026-09-11 |
| +**Date:** 2026-09-11 · **Status:** revision 4, after codex rounds 1 to 3 |
| +(19 + 14 + 7 findings, all accepted; §13) · **Requested by:** discussion |
| +#126 (Tito, with box evidence), approved in direction by Paul on |
| +2026-09-11 |
| **Builds on:** discussion #126 and its companion #127; titogo's |
| `deploy/systemd/titogo@.service` (the namespace recipe, verified on |
| @@ -35,7 +36,10 @@ concurred with the control service and showed why it must authenticate |
| its caller (§5.4). Round 2 turned the migration from an agent-start side |
| effect into an operator-triggered fleet verb (§8), because the system |
| updater's own rollback runs the *old* binary and cannot reverse a layout |
| -it never knew. |
| +it never knew. Round 3 drew the transition boundary: while a box still |
| +runs unisolated tenants, nothing that runs as `carlos` is a trustworthy |
| +caller, so the root channel authenticates the agent *process*, not a |
| +secret the agent holds (§5.4). |
| The shape in one sentence: **the activator keeps owning every exec-backed |
| instance exactly as it does today, but the process it owns is launched as |
| @@ -120,7 +124,7 @@ above implies: |
| carries `IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128` for exactly |
| this reason; exec children carry nothing. |
| -### 1.3 The primitive under all of it: a symlink is a read |
| +### 1.3 The primitive under all of it: same uid, same paths |
| A namespace hides a sibling's files from the instance. It does not hide |
| them from the **agent**, which reads and writes the instance's directory |
| @@ -134,12 +138,16 @@ the instance as its own environment (`backend_exec.go:785-794, 966-971`; |
| sibling's database and the agent's replicator uploads the sibling's data |
| under the attacker's replica. Replace the listening socket with a symlink |
| to any other socket on the box and the edge dials it as this host |
| -(codex R1-1, R1-3). And any *file* the agent reads as instructions from |
| -the instance's directory is the same primitive without a symlink |
| -(codex R2-1). A design that bounds the instance but not the paths the |
| -agent follows on its behalf, or the records it trusts from the |
| -instance's own directory, has not bounded anything. §3.2 closes the |
| -first at the kernel; §3.3 states the rule for the second. |
| +(codex R1-1, R1-3). Any *file* the agent reads as instructions from the |
| +instance's directory is the same primitive without a symlink (R2-1). |
| +And for as long as a box still runs one unisolated tenant, every |
| +`carlos`-readable secret and every `carlos`-writable directory on the |
| +box is that tenant's too, so nothing that runs as `carlos` can be told |
| +apart from it by what it knows or where it writes (R3-1). A design that |
| +bounds the instance but not the paths the agent follows on its behalf, |
| +the records it trusts, or the callers it believes, has not bounded |
| +anything. §3.2 closes the symlinks at the kernel; §3.3 states the rule |
| +for records; §5.4 states the rule for callers. |
| ### 1.4 What the unit backing already gets right, and why it is not the answer |
| @@ -178,15 +186,14 @@ scoped to the three instance templates and does not extend to |
| creates the flat files today. The registry's `DBPath` and `Socket` |
| columns point into the new layout once a host is migrated. |
| -2. **No symlink is ever followed under the instance trees.** |
| - `/data/instances` and `/run/carlos/instances` are bind mounts of |
| - themselves with `nosymfollow` (§3.2). The kernel refuses to traverse a |
| - symlink there for every process on the box, agent and edge included, |
| - with no check-then-use race. Files the instance must never touch (the |
| - log sockets, the exit records, the control channel, its token, the |
| - migration journal) live outside those two trees, where no tenant mount |
| - reaches. Nothing the agent does is ever directed by a file it read |
| - from an instance directory (§3.3). |
| +2. **No symlink is ever followed under the trees the agent binds or |
| + opens for a tenant.** `/data/instances`, `/run/carlos/instances` and |
| + `/run/carlos/logs` are bind mounts of themselves with `nosymfollow` |
| + (§3.2). The kernel refuses to traverse a symlink there for every |
| + process on the box, agent, edge and PID 1 included, with no |
| + check-then-use race. Nothing the agent does is ever directed by a file |
| + it read from an instance directory, and a migration journal is honoured |
| + only when it says what the fleet's recorded intent already says (§3.3). |
| 3. **A template unit per role.** The instance runs as |
| `carlos-instance@<host>.service`; its seed as |
| @@ -197,32 +204,33 @@ scoped to the three instance templates and does not extend to |
| with exactly the instance's own two directories bound back in, |
| `PrivatePIDs=yes`, `ProtectProc=invisible`, `RestrictNamespaces=yes`, |
| `IPAddressDeny=` for IMDS), and all three sit in one per-host slice |
| - that carries the host's memory and swap tier and its whole-family OOM |
| - rule. A sibling's database does not exist in this namespace; a |
| - sibling's socket does not exist; a sibling's pid does not exist. |
| + that carries the host's memory and swap tier. A sibling's database does |
| + not exist in this namespace; a sibling's socket does not exist; a |
| + sibling's pid does not exist. |
| 4. **The agent stays the owner.** Lease, restore, seed ordering, |
| replication, hibernation, the control socket, `carlos logs`, the |
| next-due tick, sidecar co-lifecycle and residence moves are unchanged |
| - in `internal/activator`, with one bounded change in the sweeper (§6.7). |
| - What changes is the `Proc` the exec backend hands back: it is a unit, |
| - not an `*exec.Cmd` (§6). |
| - |
| -5. **One root channel, authenticated.** The agent cannot start a system |
| - unit. A small root service, `carlos-instance-control`, socket-activated |
| - on a unix socket only uid `carlos` can open, starts and stops units |
| - named `carlos-instance*@<host>` and nothing else, and accepts a request |
| - only when it carries a token the agent reads from a path no tenant |
| - mount contains (§5). This is the synchronous form of the |
| - `carlos-stage.path` pattern. |
| - |
| -6. **Launch through a trampoline; exit through a record.** Each unit's |
| - `ExecStart` is `/opt/carlos/carlos instance-exec <role> <host>`, which |
| - runs inside the namespace, reads the launch description the agent |
| - wrote, and `execve`s the app binary with the exact argv and environment |
| - the agent composed (§6.2). Each unit's `ExecStopPost=+` runs outside |
| - the namespace as root and writes the invocation's exit record where |
| - the agent, and only the agent, reads it (§6.5). |
| + in `internal/activator`, with two bounded changes in `activator.go` |
| + (§6.3, §6.7). What changes is the `Proc` the exec backend hands back: |
| + it is a unit, not an `*exec.Cmd` (§6). |
| + |
| +5. **One root channel, authenticating the process.** The agent cannot |
| + start a system unit. A small root service, `carlos-instance-control`, |
| + socket-activated on a unix socket in a root-owned directory, starts and |
| + stops units named `carlos-instance*@<host>` and nothing else, and |
| + accepts a request only from the edge unit's main process, as reported |
| + by systemd, never from a child of it (§5). This is the synchronous |
| + form of the `carlos-stage.path` pattern. |
| + |
| +6. **Launch through a trampoline; exit through the unit's own record |
| + plus the kernel's word.** Each unit's `ExecStart` is |
| + `/opt/carlos/carlos instance-exec <role> <host>`, which runs inside the |
| + namespace, reads the launch description the agent wrote, and `execve`s |
| + the app binary with the exact argv and environment the agent composed |
| + (§6.2). Each unit's `ExecStopPost=` leaves an advisory exit record in |
| + the instance's own run directory; termination itself is proven by |
| + systemd's unit state and an empty cgroup, never by a file (§6.5). |
| 7. **Resource walls become slices.** `carlos.slice` carries the |
| 2026-08-28 outer wall; the edge unit and every instance sit under it; |
| @@ -231,20 +239,21 @@ scoped to the three instance templates and does not extend to |
| 8. **One release, two layouts, one operator verb.** This release's |
| agent runs a *flat* host exactly as today (unisolated, `childProc`) |
| - and a *nested* host isolated. Nothing moves on upgrade, so the system |
| - updater's own rollback (which runs the previous binary) always lands |
| - on a layout that binary understands. `carlos system layout migrate` |
| - moves a fleet's hosts across under a journal the tenant cannot touch; |
| - `carlos system layout revert` moves them back; both are console |
| - verbs, reported per host in status (§8). The next release drops the |
| - flat path. |
| - |
| -9. **Addr routes are untouched, and now invisible.** A TCP-addr route's |
| - `<host>.config.env` stays flat in `/data`, because the app's own |
| - supervisor reads it there (titogo's `EnvironmentFile=-/data/%i.config.env`). |
| - Isolated instances can no longer see it. That closes the read #127 |
| - leads with; #127's ask for root-owned delivery stays open as its own |
| - change (§11). |
| + and a *nested* host isolated. Nothing moves on upgrade and no new row |
| + is minted nested, so the system updater's own rollback (which runs the |
| + previous binary) always lands on a layout that binary understands. |
| + `carlos system layout migrate` records the fleet's intent; every box |
| + converges its eligible hosts to it on every serial move; `revert` |
| + records the opposite; status reports each host (§8). The next release |
| + drops the flat spawn path for activator-managed hosts. |
| + |
| +9. **Addr routes and externally supervised routes are untouched, and now |
| + invisible.** A TCP-addr route's `<host>.config.env` stays flat in |
| + `/data`, because the app's own supervisor reads it there (titogo's |
| + `EnvironmentFile=-/data/%i.config.env`); an unflagged socket route keeps |
| + its flat paths because its supervisor owns them. Isolated instances can |
| + no longer see either. That closes the read #127 leads with; #127's ask |
| + for root-owned delivery stays open as its own change (§11). |
| --- |
| @@ -264,20 +273,19 @@ scoped to the three instance templates and does not extend to |
| launch.seed.json agent-written before a seed |
| launch.sidecar.json agent-written before a sidecar start |
| <host>.db.corrupt a quarantined restore, as today |
| -/data/layout/ 0700 carlos:carlos; NOT under any tenant mount |
| +/data/layout/ 0700 carlos:carlos |
| <host>.json the migration journal for a host in flight (§8.2) |
| /run/carlos/instances/ bind mount of itself, nosymfollow; 2750 carlos:carlos-sockets |
| /run/carlos/instances/<host>/ 2750 carlos:carlos-sockets, created by the agent |
| app.sock the instance's listener (rt.Socket); 0660 after health, as today |
| app.sock.control the agent's control listener (rt.Socket + ".control"); 0600 |
| -/run/carlos/logs/ 0700 carlos:carlos; NOT bound into any namespace |
| + exit.<role>.<invocation>.json the unit's advisory exit record (§6.5); tenant-writable, and treated so |
| +/run/carlos/logs/ bind mount of itself, nosymfollow; 0700 carlos:carlos; NOT bound into any namespace |
| <host>.run.sock the agent's log listener for the instance (§6.4) |
| <host>.seed.sock ... for the seed |
| <host>.sidecar.sock ... for the sidecar |
| -/run/carlos/exits/ 0750 root:carlos; NOT bound into any namespace |
| - <host>.<role>.<invocation>.json root-written exit record (§6.5) |
| -/run/carlos/instance-control.sock the root channel (§5); not in any namespace |
| -/run/carlos/instance-control.token 0600 carlos:carlos, root-written; not in any namespace |
| +/run/carlos-control/ 0750 root:carlos; NOT under /run/carlos, NOT in any namespace |
| + control.sock the root channel (§5); SocketUser=carlos, 0600 |
| ``` |
| The socket tree keeps the #103 contract: `/run/carlos/instances` and |
| @@ -291,33 +299,35 @@ its run directory and can loosen it; that changes nothing outside, |
| because outside is where the mode is set and checked. |
| **One helper, one owner.** `internal/provision` gains |
| -`InstanceDir(dataDir, host)` beside `DefaultDBPath`, and |
| -`DefaultDBPath(dataDir, host)` and `DefaultSocket(host)` change to return |
| -the nested paths for exec routes: |
| +`InstanceDir(dataDir, host)` beside `DefaultDBPath`, and the nested |
| +defaults exist alongside the flat ones. In this release **no minting |
| +path uses the nested defaults** (§8.1, codex R3-6): `routeFor` |
| +(`reconcile.go:299-301`) and `carlos add` (`cli.go:118-136`) keep minting |
| +flat paths, and the layout pass is the only thing that ever writes a |
| +nested path into a row. |
| ```go |
| func InstanceDir(dataDir, host string) string { |
| return filepath.Join(dataDir, "instances", host) |
| } |
| -func DefaultDBPath(dataDir, host string) string { |
| +func NestedDBPath(dataDir, host string) string { |
| return filepath.Join(InstanceDir(dataDir, host), host+".db") |
| } |
| func InstanceRunDir(host string) string { |
| return filepath.Join(SocketDir, "instances", host) |
| } |
| -func DefaultSocket(host string) string { |
| +func NestedSocket(host string) string { |
| return filepath.Join(InstanceRunDir(host), "app.sock") |
| } |
| func LogSocket(host, role string) string { |
| return filepath.Join(SocketDir, "logs", host+"."+role+".sock") |
| } |
| -func FlatDBPath(dataDir, host string) string { return filepath.Join(dataDir, host+".db") } |
| -func FlatSocket(host string) string { return filepath.Join(SocketDir, host+".sock") } |
| +// DefaultDBPath and DefaultSocket keep their flat meaning for this |
| +// release; the next release, which drops the flat spawn path, points |
| +// them at the nested helpers for activator-managed routes and keeps |
| +// them flat for externally supervised ones (§8.6). |
| ``` |
| -`FlatDBPath`/`FlatSocket` exist for this release only: they are how a row |
| -is classified (§8.1) and how the flat spawn path finds its files. Every |
| -DataDir-keyed builder in the tree moves onto these (the list is §9). |
| `ExecBackend.defaultDBPath` (`backend_exec.go:160-162`) is deleted in |
| favour of the provision helpers, so there is one definition, not two. |
| @@ -336,18 +346,11 @@ discussion asks for ("absent, not permission-denied") without any of |
| that, and leaves option B open as a later hardening on the same layout |
| (§11). |
| -**Custom `DBPath`.** `carlos add --db` may stamp any absolute path on an |
| -exec route (`cli.go:157-159` refuses only a relative one). Under this |
| -design an exec route's `DBPath` must be inside `InstanceDir`, because |
| -that is the only data path the namespace reveals. `carlos add` refuses a |
| -nested-layout exec route whose `--db` is elsewhere, naming the directory. |
| -Existing rows are handled by the migration (§8.3). |
| +### 3.2 Symlinks are never followed under the trees the platform binds or opens for a tenant (codex R1-1, R1-2, R1-3, R3-1) |
| -### 3.2 Symlinks are never followed under the instance trees (codex R1-1, R1-2, R1-3) |
| - |
| -Two mount units, embedded and installed like every other unit |
| +Three mount units, embedded and installed like every other unit |
| (`internal/host/bootstrap.go:33-34`), and tmpfiles lines beside the |
| -existing `/run/carlos` one (`bootstrap.go:61`). Both mounts carry |
| +existing `/run/carlos` one (`bootstrap.go:61`). All three carry |
| `DefaultDependencies=no`: a mount unit's default ordering places it |
| before `local-fs.target`, and `systemd-tmpfiles-setup.service`, which |
| creates the directories, runs after that target, so the defaults would |
| @@ -373,35 +376,21 @@ Options=bind,nosymfollow,nosuid,nodev,noexec |
| WantedBy=multi-user.target |
| ``` |
| -```ini |
| -# run-carlos-instances.mount |
| -[Unit] |
| -Description=carlos instance sockets (bind, nosymfollow) |
| -DefaultDependencies=no |
| -After=systemd-tmpfiles-setup.service |
| -Before=carlos-edge.service carlos-instance-control.service |
| -Conflicts=umount.target |
| -Before=umount.target |
| -[Mount] |
| -What=/run/carlos/instances |
| -Where=/run/carlos/instances |
| -Type=none |
| -Options=bind,nosymfollow,nosuid,nodev,noexec |
| -[Install] |
| -WantedBy=multi-user.target |
| -``` |
| +`run-carlos-instances.mount` and `run-carlos-logs.mount` are the same |
| +text for `/run/carlos/instances` and `/run/carlos/logs`, with |
| +`After=systemd-tmpfiles-setup.service` and no `RequiresMountsFor=`. |
| ``` |
| d /data/instances 0755 carlos carlos - |
| d /data/layout 0700 carlos carlos - |
| d /run/carlos/instances 2750 carlos carlos-sockets - |
| d /run/carlos/logs 0700 carlos carlos - |
| -d /run/carlos/exits 0750 root carlos - |
| +d /run/carlos-control 0750 root carlos - |
| ``` |
| `carlos-edge.service`, `carlos-instance-control.service` and the three |
| -templates gain `Requires=` and `After=` on both mounts, so an edge that |
| -starts without them does not start at all rather than starting |
| +templates gain `Requires=` and `After=` on the mounts they touch, so an |
| +edge that starts without them does not start at all rather than starting |
| unprotected. The plan's first box task is a **cold boot** with the full |
| unit set installed, then `systemctl list-dependencies --before |
| data-instances.mount` and a `systemd-analyze verify` over the embedded |
| @@ -412,12 +401,13 @@ set, pinned as a CI test that renders the units and runs |
| 2.41) makes path resolution on that mount return `ELOOP` at any symlink, |
| for every process, in every namespace that sees the mount. It is a |
| per-mount flag, so it holds through the `BindPaths=` bind into the |
| -instance's namespace and through the agent's and the edge's ordinary |
| -lookups from outside. `readlink` still works; nothing the platform runs |
| -needs a symlink under these trees (SQLite, litestream, the app's listener |
| -and the agent's env files are all plain files). Hard links cannot escape |
| -either: a `link()` needs a source path the caller can resolve, and inside |
| -the namespace no sibling path resolves. |
| +instance's namespace, through the agent's and the edge's ordinary |
| +lookups from outside, and through PID 1's open of a `StandardOutput=file:` |
| +path. `readlink` still works; nothing the platform runs needs a symlink |
| +under these trees (SQLite, litestream, the app's listener and the |
| +agent's env files are all plain files). Hard links cannot escape either: |
| +a `link()` needs a source path the caller can resolve, and inside the |
| +namespace no sibling path resolves. |
| What this closes, race-free and for every reader at once: |
| @@ -429,7 +419,19 @@ What this closes, race-free and for every reader at once: |
| agent-readable path, and having the agent read it into the instance's |
| environment or replicate it into the instance's replica (R1-3); |
| - the instance planting a symlink at a predictable temp name so the |
| - agent's write truncates the target (R1-3). |
| + agent's write truncates the target (R1-3); |
| +- a still-unisolated flat tenant, during the transition, planting a |
| + symlink at a log socket path so that PID 1 opens a root-owned file as |
| + some unit's stdout (R3-1). `/run/carlos/logs` is `carlos`-owned because |
| + the agent must bind there, so a same-uid process can write it; the |
| + mount flag is what makes that write inert. |
| + |
| +The flat tenants' own sockets under `/run/carlos` itself are not on such |
| +a mount (the directory holds flat sockets, the apps subtree and more, and |
| +this release does not change it). A flat tenant can therefore still point |
| +its socket at any socket on the box and have the edge dial it. §5.4 says |
| +why that no longer reaches the control service, and §6.1 adds the one |
| +check the edge makes on every unix dial. |
| Belt and braces in the agent, cheap and testable: |
| @@ -438,9 +440,9 @@ Belt and braces in the agent, cheap and testable: |
| `openat2(2)` with `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS` |
| (Linux 5.6+), so a symlink is refused with a named error even on a |
| box where the mount option is missing; the same helper is what the |
| - agent uses to verify, at start, that both mounts carry `nosymfollow` |
| - (`/proc/self/mountinfo`), and it **refuses to wake any nested host** |
| - until they do (§8.5); |
| + agent uses to verify, at start, that all three mounts carry |
| + `nosymfollow` (`/proc/self/mountinfo`), and it **refuses to wake any |
| + nested host** until they do (§8.5); |
| - every agent write into an instance directory creates its temp file |
| with `O_CREAT|O_EXCL` and a random suffix, then `renameat`s over the |
| target (rename replaces the directory entry and never follows the |
| @@ -448,26 +450,36 @@ Belt and braces in the agent, cheap and testable: |
| - launch files (§6.2) are written the same way, and `instance-exec` |
| reads them through the same helper. |
| -### 3.3 Nothing in an instance directory is an instruction (codex R2-1) |
| +### 3.3 Nothing in an instance directory is an instruction, and a journal is not one either (codex R2-1, R3-1) |
| The agent reads three kinds of thing from an instance directory: the |
| database (to replicate, restore and checkpoint), the two env files (to |
| -compose the instance's own environment), and nothing else. It never |
| -reads a path, a host name, a file list or a phase from there. In |
| -particular: |
| +compose the instance's own environment), and the advisory exit record |
| +(to decorate an event line, §6.5). It never reads a path, a host name, a |
| +file list or a phase from there. In particular: |
| - the migration journal lives in `/data/layout/<host>.json`, outside |
| every tenant mount, and every source and destination path in it is |
| *derived from the host name and the registry row* by the §3.1 helpers |
| - at plan time, never read back from a file (§8.2); |
| + at plan time. On recovery the pass **recomputes** the plan from the row |
| + and honours the journal only if its paths equal the recomputation and |
| + its direction equals the fleet's recorded intent (`control/layout.json`, |
| + a bucket object no tenant can write, §8.2). During the transition a |
| + flat tenant can write `/data/layout` (same uid); a journal it forges can |
| + therefore cause the agent to do exactly what the next layout pass would |
| + do for that host anyway, and nothing else (R3-1); |
| - the launch file is agent-written and then read by the trampoline |
| inside the namespace as the instance itself; it directs nothing the |
| agent does (§6.2); |
| -- the exit record is root-written outside every tenant mount (§6.5). |
| +- the exit record is written inside the instance's run directory by the |
| + unit's own `ExecStopPost=`, is therefore tenant-writable, and is treated |
| + as the tenant's claim about its own exit status: it never decides |
| + whether the process is gone (§6.5). |
| -A test plants a forged journal-shaped file in an instance directory and |
| -a forged launch file naming a sibling, and asserts the sibling is |
| -untouched by a full migrate, revert, wake and stop cycle. |
| +A test plants a forged journal (a sibling's paths under a host's name; a |
| +direction opposite to the fleet's intent), a forged launch file naming a |
| +sibling, and a forged exit record for a running unit, and asserts the |
| +sibling is untouched and the running unit is not treated as exited. |
| --- |
| @@ -477,7 +489,7 @@ Unit files under `internal/host/units/`, embedded and installed exactly as |
| the existing nineteen are (`bootstrap.go:33-34`, `systemapply.go:339-398`): |
| `carlos-instance@.service`, `carlos-instance-seed@.service`, |
| `carlos-instance-sidecar@.service`, `carlos.slice`, |
| -`carlos-instances.slice`, the two mount units of §3.2, the probe of |
| +`carlos-instances.slice`, the three mount units of §3.2, the probe of |
| §8.5, and the two control units of §5.2. Nothing enables the templates; |
| an instance unit exists only while the agent has started it. |
| @@ -500,15 +512,19 @@ Description=carlos exec-backed instance %I |
| # PartOf= would propagate the RESTART and race the agent's own startup |
| # (codex R1-7). |
| BindsTo=carlos-edge.service |
| -Requires=data-instances.mount run-carlos-instances.mount |
| -After=carlos-edge.service data-instances.mount run-carlos-instances.mount |
| +Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount |
| +After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount |
| [Service] |
| Type=simple |
| ExecStart=/opt/carlos/carlos instance-exec run %I |
| -# Runs as ROOT, outside the namespace ("+"), after the last process of |
| -# this invocation is gone: writes the exit record the agent reads (§6.5). |
| -ExecStopPost=+/opt/carlos/carlos instance-exit run %I |
| +# Runs inside the same namespace, as carlos, after this invocation's |
| +# processes have been signalled: leaves an ADVISORY exit record in the |
| +# instance's own run directory (§6.5). Not "+": the "+" prefix drops |
| +# User= and the Private*/Protect* settings but keeps the explicit |
| +# TemporaryFileSystem=/BindPaths= mounts (codex R3-2), so nothing run |
| +# from here can see the host's /run/carlos anyway. |
| +ExecStopPost=/opt/carlos/carlos instance-exit run %I |
| User=carlos |
| Group=carlos |
| # One slice per host holds the instance, its seed and its sidecar, and |
| @@ -523,8 +539,8 @@ Restart=no |
| KillMode=mixed |
| TimeoutStopSec=20 |
| # The log stream: stdout and stderr connect to the agent's per-role |
| -# listener (§6.4). PID 1 opens this as root, outside the namespace, at a |
| -# path no tenant mount contains. |
| +# listener (§6.4). PID 1 opens this as root, outside the namespace, on |
| +# the nosymfollow logs mount, at a path no tenant mount contains. |
| StandardOutput=file:/run/carlos/logs/%I.run.sock |
| StandardError=inherit |
| @@ -557,11 +573,15 @@ RestrictSUIDSGID=yes |
| MemoryDenyWriteExecute=yes |
| SystemCallArchitectures=native |
| -# --- per-service resources; the tier and the family OOM rule live on the |
| -# host slice (§4.3) --- |
| +# --- per-service resources; the tier lives on the host slice (§4.3) --- |
| TasksMax=256 |
| # The kernel picks a tenant before it picks anything else in carlos.slice. |
| OOMScoreAdjust=500 |
| +# One OOM kill takes every process of THIS service (systemd sets |
| +# memory.oom.group on the service cgroup for OOMPolicy=kill). The sidecar |
| +# is a sibling service with its own family rule; §4.3 says why the two are |
| +# not one family any more. |
| +OOMPolicy=kill |
| ``` |
| Lines that are load-bearing and why: |
| @@ -593,23 +613,19 @@ Lines that are load-bearing and why: |
| - **`IPAddressDeny=`** for IMDS. The nft rule in pinfra allows uid |
| `carlos` because the agent needs IMDS; this line is what stops an |
| instance, also uid `carlos`, from fetching the box's role credentials. |
| - It is a BPF filter, so a denied connect fails with `EPERM`, which is |
| - how the probe tells enforcement from an unreachable network (§8.5). |
| + It is a cgroup BPF **packet** filter: a denied datagram send fails with |
| + `EPERM`, a denied TCP `connect()` loses its SYN and times out. §8.5's |
| + probe is written for that (R3-4). |
| - **`BindsTo=carlos-edge.service`** keeps today's "instances die with the |
| agent": when the edge stops, so do they, and nothing restarts them but |
| the agent's next wake. §6.7 adds the startup sweep for the case where |
| the coupling did not fire. |
| -- **`ExecStopPost=+`**: the `+` prefix runs the command with full |
| - privileges and *without* the unit's `User=` and file-system namespacing, |
| - so it can write under `/run/carlos/exits`, which no tenant mount |
| - contains. systemd runs it after the service's own processes have been |
| - signalled and reaped, with `$SERVICE_RESULT`, `$EXIT_CODE`, |
| - `$EXIT_STATUS` and `$INVOCATION_ID` in its environment. §6.5 says what |
| - the agent does with that and why it still checks the cgroup. |
| - **`StandardOutput=file:` to an `AF_UNIX` socket** is documented |
| systemd behaviour: a stream connection is made and used as stdout. It |
| is opened by PID 1 outside the namespace, so the path is the real one, |
| - and the path is in `/run/carlos/logs`, which no tenant mount contains. |
| + the path is in `/run/carlos/logs`, which no tenant mount contains, and |
| + that directory is on a `nosymfollow` mount, so a same-uid process |
| + outside any namespace cannot turn it into a symlink either. |
| - **`/opt/carlos`** stays readable through `ProtectSystem=strict`; the |
| trampoline and the live symlink are all the instance needs from it. |
| @@ -623,16 +639,16 @@ Differences: |
| |---|---|---| |
| | `Type=` | `oneshot` | `simple` | |
| | `ExecStart=` | `/opt/carlos/carlos instance-exec seed %I` | `/opt/carlos/carlos instance-exec sidecar %I` | |
| -| `ExecStopPost=` | `+/opt/carlos/carlos instance-exit seed %I` | `+/opt/carlos/carlos instance-exit sidecar %I` | |
| +| `ExecStopPost=` | `/opt/carlos/carlos instance-exit seed %I` | `/opt/carlos/carlos instance-exit sidecar %I` | |
| | `StandardOutput=` | `file:/run/carlos/logs/%I.seed.sock` | `file:/run/carlos/logs/%I.sidecar.sock` | |
| -| timeouts | `TimeoutStartSec=120` (today's seed budget, `backend_exec.go:368-408`; a oneshot has no start timeout by default, R1-5), `TimeoutStopSec=20` | `TimeoutStopSec=20` | |
| +| timeouts | `TimeoutStartSec=120` (today's seed budget, `backend_exec.go:368-408, 466`; a oneshot has no start timeout by default, R1-5), `TimeoutStopSec=20` | `TimeoutStopSec=20` | |
| A Go test over the embedded FS asserts that the three files' namespace, |
| hardening and coupling lines are byte-identical (§10). Three files rather |
| than one because seed and instance never run together but sidecar and |
| instance do, and a template's instance is the host, not a role. |
| -### 4.3 Slices and the memory walls (codex R1-8, R1-10, R2-11) |
| +### 4.3 Slices and the memory walls (codex R1-8, R1-10, R2-11, R3-3) |
| ```ini |
| # carlos.slice — the outer wall of the 2026-08-28 design, as a slice |
| @@ -676,15 +692,22 @@ Seed, instance and sidecar all charge that one cgroup, which is what |
| today's shared `carlos-exec@<host>` gives them |
| (`backend_exec.go:149-155, 373-375, 890-895`). |
| -**The family OOM rule** is `memory.oom.group=1` on the host slice's |
| -cgroup, written by the control service (root) directly to |
| -`/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/memory.oom.group` |
| -after `set-property`, because systemd has no directive for it on a |
| -slice. With it, an OOM kill anywhere in the family takes the instance and |
| -its sidecar together, which is what `internal/cgroup` gives them today |
| -and what a per-service `OOMPolicy=kill` would not (R1-8, R2 disposition). |
| -systemd does not manage that attribute on slices, so the write does not |
| -fight it; the plan's box task confirms it survives a `daemon-reload`. |
| +**The family OOM rule changes, deliberately.** Today `memory.oom.group=1` |
| +on the shared `carlos-exec@<host>` cgroup means an OOM kill in the |
| +sidecar takes the instance with it. systemd writes `memory.oom.group` |
| +itself, for slices too, from the unit's cgroup context (default `0`, |
| +`cgroup_context_apply` in `src/core/cgroup.c`), so a value written |
| +behind its back is overwritten on the next property application, which |
| +`/start` for the sidecar performs on a slice whose instance is already |
| +running (codex R3-3). There is no supported way to keep it set on the |
| +slice. So the rule becomes: **each service is its own OOM family** |
| +(`OOMPolicy=kill`, which sets the attribute on the service's cgroup), and |
| +a sidecar killed by the OOM killer is a sidecar that died, which the |
| +sweeper already restarts with backoff (`activator.go:1165-1200`) while |
| +the instance keeps serving. An instance killed by the OOM killer still |
| +stands its entry down and co-stops its sidecar as today. The instance |
| +surviving its sidecar's OOM is the better of the two behaviours; the |
| +spec records it as a difference rather than claiming parity. |
| The cgroup path is |
| `/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/` |
| @@ -710,7 +733,7 @@ waiting HTTP request. |
| [Unit] |
| Description=carlos instance control socket (agent → root) |
| [Socket] |
| -ListenStream=/run/carlos/instance-control.sock |
| +ListenStream=/run/carlos-control/control.sock |
| SocketUser=carlos |
| SocketGroup=carlos |
| SocketMode=0600 |
| @@ -723,15 +746,14 @@ WantedBy=sockets.target |
| # carlos-instance-control.service |
| [Unit] |
| Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent) |
| -Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount |
| -After=data-instances.mount run-carlos-instances.mount |
| +Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount run-carlos-logs.mount |
| +After=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount |
| [Service] |
| ExecStart=/opt/carlos/carlos instance-control |
| # Root deliberately: the whole job is systemctl start/stop, and the |
| # caller is the unprivileged agent. Same rationale as carlos-stage.service. |
| ProtectSystem=strict |
| -ReadWritePaths=/run/carlos /sys/fs/cgroup/carlos.slice |
| ProtectHome=yes |
| PrivateTmp=yes |
| PrivateDevices=yes |
| @@ -743,30 +765,29 @@ MemoryDenyWriteExecute=yes |
| MemoryMax=64M |
| ``` |
| -The socket is owned by `carlos` with mode 0600 and the unit is socket- |
| -activated, so only the agent's uid can open it and it runs only while |
| -there is work. It is not in any instance's namespace: `/run/carlos` |
| -inside an instance is an empty tmpfs with the instance's own directory |
| -bound in. At start the service writes a fresh random token to |
| -`/run/carlos/instance-control.token` (`0600 carlos:carlos`, `O_EXCL` |
| -temp and rename) and runs the probe of §8.5. |
| +The socket lives in `/run/carlos-control`, `0750 root:carlos`: any |
| +`carlos` process can connect (search on the directory), none can create, |
| +rename or replace anything in it. The unit is socket-activated and runs |
| +only while there is work. It is not in any instance's namespace: |
| +`/run/carlos` inside an instance is an empty tmpfs with the instance's |
| +own directory bound in, and `/run/carlos-control` is not under it. |
| ### 5.3 Protocol |
| HTTP/1.1 over the unix socket, the shape `edge.ListenControl` already |
| -uses for the per-instance control socket. Every request carries |
| -`Authorization: Bearer <token>`; every response to a request without the |
| -right token is 401 and nothing runs (§5.4 says why). Five verbs, no |
| -shell, no argument passes through to `systemctl` except a unit name the |
| -service built itself: |
| - |
| -| Verb | What it runs | Returns | |
| -|---|---|---| |
| -| `POST /start?host=H&role=run\|sidecar&memory_max=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N`; write `memory.oom.group=1` on the slice cgroup; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID <unit>`, read immediately after the start returns; or 502 with systemctl's stderr | |
| -| `POST /seed?host=H&memory_max=N` | the same, on `carlos-instance-seed@<esc>.service` (oneshot: `start` blocks until exit or `TimeoutStartSec`) | 200 with `{invocation_id}`; the seed's result reaches the agent as an exit record (§6.5), not in this response | |
| -| `POST /stop?host=H&role=run\|seed\|sidecar` | `systemctl stop <unit>` (blocking; also cancels a pending start job) | 204 | |
| -| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>`, tolerating "not loaded" | 204 | |
| -| `GET /ready` | nothing | 204 once the probe verdict (§8.5) is `ready`; 503 with the verdict otherwise | |
| +uses for the per-instance control socket. Before reading a request line |
| +the service authenticates the **connection** (§5.4); a connection that |
| +fails is closed without a response. Five verbs, no shell, no argument |
| +passes through to `systemctl` except a unit name the service built |
| +itself: |
| + |
| +| Verb | What it runs | Returns | Agent deadline | |
| +|---|---|---|---| |
| +| `POST /start?host=H&role=run\|sidecar&memory_max=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N`; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID <unit>`, read immediately after the start returns; or 502 with systemctl's stderr | 30s | |
| +| `POST /seed?host=H&memory_max=N` | the same, on `carlos-instance-seed@<esc>.service` (oneshot: `start` blocks until exit or `TimeoutStartSec`); then `systemctl show -p InvocationID,Result,ExecMainStatus,ExecMainCode` | 200 with `{invocation_id, result, status, code}` if the unit is still loaded, else 200 with `{invocation_id: ""}` and the agent reads the exit record (§6.5) | 150s: 120s seed budget plus `TimeoutStopSec` and slack (R3-5) | |
| +| `POST /stop?host=H&role=run\|seed\|sidecar` | `systemctl stop <unit>` (blocking; also cancels a pending start job) | 204 | 30s | |
| +| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>`, tolerating "not loaded" | 204 | 30s | |
| +| `GET /ready` | nothing | 204 once the probe verdict (§8.5) is `ready`; 503 with the verdict otherwise | 5s | |
| There is deliberately **no `reset-failed` inside `/start`**: `ResetFailedUnit` |
| refuses a unit that is not loaded, and a template instance that has never |
| @@ -774,11 +795,12 @@ started is not loaded, so the first wake of every host would fail (codex |
| R2-3). `systemctl start` on a `failed` unit already clears the failed |
| state; `/reset` exists for the sweeper's housekeeping only. |
| -Validation, in order: token; `host` must satisfy `provision.ValidHostShape` |
| -(`provision.go:262`); `role` must be one of the three literals; the |
| -memory value must parse as a `Limit` the agent already knows how to |
| -produce (`internal/cgroup`, `TierDefault`/`TierChildren`). Anything else |
| -is a 400 with no systemctl call. The unit name is |
| +Validation, in order: connection (§5.4); `host` must satisfy |
| +`provision.ValidHostShape` (`provision.go:262`); `role` must be one of |
| +the three literals; the memory value must parse as a `Limit` the agent |
| +already knows how to produce (`internal/cgroup`, |
| +`TierDefault`/`TierChildren`). Anything else is a 400 with no systemctl |
| +call. The unit name is |
| `"carlos-instance" + suffix + "@" + escape(host) + ".service"` and the |
| slice name `"carlos-instances-" + escape(host) + ".slice"`, built from |
| the validated pieces, never from the request string. |
| @@ -796,54 +818,67 @@ writes a runtime drop-in the following `start` honours |
| R1-1). The plan's first task exercises it on a dev box for a slice that |
| does not yet exist. |
| -Every call the agent makes to this service carries a deadline |
| -(`controlTimeout`, 30s; a blocking `start` that exceeds it is answered |
| -by a `stop` and reported as a failed wake), because §6.7 puts one such |
| -call on a path that must never stall the activator. |
| +A `/start` or `/stop` that exceeds its deadline is answered by the agent |
| +with a `/stop` and reported as a failed wake or an unproven stop |
| +(§6.3); the deadlines exist because §6.7 puts one such call on a path |
| +that must never stall the activator, and the seed's is long because a |
| +seed is allowed to be slow (R3-5). |
| -### 5.4 The choice put to codex, and the ruling |
| +### 5.4 Who may call it, and the ruling (codex R1-1, R3-1) |
| The other way to let uid `carlos` start these units is polkit: a rules |
| file granting `org.freedesktop.systemd1.manage-units` to `carlos` when |
| the unit name matches `carlos-instance*@*.service`. Codex R1-1 confirmed |
| -that polkit can express the unit and verb restrictions (systemd passes |
| -`unit` and `verb`, including `set-property` and `reset-failed`, under |
| -that one action; the draft's claim that properties need a separate |
| -action was wrong), and also that it cannot constrain property values, |
| -that a uid-only grant is reachable by every tenant through the system |
| -bus, and that a wildcard on names also authorises transient units. |
| - |
| -Codex's advisement, concurring with the draft's: **the control service, |
| -provided it authenticates the request and not the socket's uid.** The |
| -reason is the proxy attack in §1.3: the edge dials `rt.Socket` as uid |
| -`carlos` in its own namespace, so before §3.2 an instance could point its |
| -own socket path at the control socket and have the edge deliver an |
| -attacker-written `POST /stop?host=<sibling>` with a perfectly authorised |
| -peer uid. A peer-uid check cannot tell that request from the agent's |
| -own. `nosymfollow` removes the redirection; the token removes the class: |
| -a request is the agent's only if it carries the token, and the token is |
| -readable only from `/run/carlos/instance-control.token`, root-written at |
| -service start, `0600 carlos:carlos`, at a path no instance namespace |
| -contains and the edge never serves. The agent reads it once at start and |
| -on 401. A tenant that could learn it would already have to be reading |
| -arbitrary agent-readable files, which is the property §3.2 denies. |
| - |
| -Ruled, 2026-09-11: control service with bearer token; polkit is not |
| +that polkit can express the unit and verb restrictions, and also that it |
| +cannot constrain property values, that a uid-only grant is reachable by |
| +every tenant through the system bus, and that a wildcard on names also |
| +authorises transient units. Codex's advisement, concurring with the |
| +draft's: **the control service, provided it authenticates the caller and |
| +not the socket's uid.** Ruled, 2026-09-11: control service; polkit is not |
| installed and not relied on. |
| +Revision 2 authenticated the caller with a bearer token in a |
| +`carlos`-readable file. Round 3 showed why that is not a boundary during |
| +the transition: a flat tenant is uid `carlos` in the host namespace and |
| +reads that file as easily as the agent does (R3-1). A secret the agent |
| +holds cannot distinguish the agent from an unisolated child of it. What |
| +can is **systemd's own record of which process is the agent**: |
| + |
| +- On every accepted connection the service reads `SO_PEERCRED` and |
| + requires `uid == carlos` **and** `pid == MainPID` of |
| + `carlos-edge.service` as `systemctl show -p MainPID` reports it at that |
| + moment (root, one call, cached for one second). A flat tenant is a |
| + child of that pid, never the pid itself; a nested tenant cannot reach |
| + the socket at all. |
| +- The edge is the agent, so a request the edge *proxies* for a tenant |
| + (a flat tenant pointing its own socket at `control.sock`, §3.2) would |
| + pass that check. It does not get that far: **the edge refuses any unix |
| + dial whose peer is uid 0** (`SO_PEERCRED` after `connect`, before a |
| + byte is sent; `edge.go:2000-2014`'s `unixProxy` and the activator's |
| + health and control dials, §6.1). Instances are `carlos` or a |
| + DynamicUser; the only root-owned listener the platform ever wants a |
| + tenant's traffic near is none. A proxied request therefore never |
| + reaches the service, and the service never sees a connection from |
| + anything but the agent's own goroutines. |
| + |
| +No token. The file it lived in is gone, and with it the question of who |
| +can read it. The two checks are each one syscall and one cached |
| +`systemctl show`, both testable with a fake peer, and both hold on a box |
| +that still runs flat tenants, which is the whole transition. |
| + |
| --- |
| ## 6. The activator on units |
| -`internal/activator/activator.go` changes in exactly one place, the |
| -sidecar restart in `sweepOnce` (§6.7, codex R2-6). `doWake` |
| -(`activator.go:772`), `stop` (`:962-1047`), `Bounce` (`:946-956`) and the |
| -`Proc`/`Rep` seams (`:194-207`) keep their contracts. `Exited()` stays a |
| -non-blocking read of a flag (R1-4); everything that talks to systemd |
| -runs in the proc's own goroutine or in the sidecar-start goroutine. The |
| -rest of the change is inside `ExecBackend`, which keeps its current |
| -`childProc` path for flat hosts for this one release (§8.1) and adds the |
| -unit path for nested hosts. |
| +`internal/activator/activator.go` changes in exactly two places: the |
| +sidecar restart in `sweepOnce` (§6.7, codex R2-6) and the lease release |
| +in `stop` (§6.3, codex R2-5). `doWake` (`activator.go:772`), `Bounce` |
| +(`:946-956`) and the `Proc`/`Rep` seams (`:194-207`) keep their contracts. |
| +`Exited()` stays a non-blocking read of a flag (R1-4); everything that |
| +talks to systemd runs in the proc's own goroutine or in the |
| +sidecar-start goroutine. The rest of the change is inside `ExecBackend`, |
| +which keeps its current `childProc` path for flat hosts for this one |
| +release (§8.1) and adds the unit path for nested hosts. |
| ### 6.1 Preparing a role, and `Start` |
| @@ -860,16 +895,16 @@ included (R1-5, R2-12): |
| as today. For `run` only: bind the control listener at |
| `rt.Socket + ".control"` (unchanged, `edge.ListenControl`) and clear |
| any stale `app.sock` (nothing is running: the unit is not active, or |
| - we would not be here). |
| + we would not be here). Remove any stale `exit.<role>.*.json`. |
| 3. Bind the role's log listener at `LogSocket(host, role)` (§6.4). |
| 4. Write `launch.<role>.json` (§6.2). |
| `Start` is then: `prepare(rt, "run")`; `POST /start` with the route's |
| -tier; `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` (unchanged); |
| -the `0660` normalisation of `app.sock` (unchanged); on failure |
| -`POST /stop`, wait for the proc's `done`, and clean up, exactly as the |
| -current failure path stops the child; return a `unitProc` carrying the |
| -invocation id. |
| +tier; `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` (unchanged, |
| +with the peer-uid check of §5.4 on its dial); the `0660` normalisation of |
| +`app.sock` (unchanged); on failure `POST /stop`, wait for the proc's |
| +`done`, and clean up, exactly as the current failure path stops the |
| +child; return a `unitProc` carrying the invocation id. |
| ### 6.2 The trampoline and `launch.<role>.json` |
| @@ -925,18 +960,18 @@ reads the file (§3.3). |
| Seed: `launch.seed.json` carries the argv from `rt.Seed` |
| (`backend_exec.go:332-338`) with the DB path substituted as today; `POST /seed` |
| -blocks up to `TimeoutStartSec`; the seed's exit status arrives as its |
| -exit record (§6.5); a non-zero status, a timeout or a cancelled `ctx` |
| -runs `POST /stop?role=seed`, waits for the seed proc's `done`, and only |
| -then runs the same cleanup as today (`backend_exec.go:438`), so the |
| -lease is never released (`activator.go:824-828`) while a seed can still |
| -write (R1-5). |
| +blocks up to its 150s deadline; the seed's exit status arrives in the |
| +response or in its exit record (§6.5); a non-zero status, a timeout or a |
| +cancelled `ctx` runs `POST /stop?role=seed`, waits for the seed proc's |
| +`done`, and only then runs the same cleanup as today |
| +(`backend_exec.go:438`), so the lease is never released |
| +(`activator.go:824-828`) while a seed can still write (R1-5). |
| Sidecar: `launch.sidecar.json` carries `[live/<host>, "sidecar", "run"]` |
| and the sidecar env from `<host>-sidecar.env` only, as today |
| (`backend_exec.go:885-911`); `POST /start?role=sidecar`. |
| -### 6.3 `unitProc` |
| +### 6.3 `unitProc`, and a stop that could not be proven (codex R2-5) |
| ```go |
| type unitProc struct { |
| @@ -956,6 +991,18 @@ first, close the listener only after the process is gone) is preserved: |
| `Stop` posts the stop, waits for `done`, then closes `ctl` and `logs`. |
| `done` closes only on proven termination (§6.5), never on log EOF. |
| +If `ctx` expires before `done` closes, `Stop` returns `ErrStillRunning` |
| +without closing the listeners. Today `stop()` then calls `rep.Close` |
| +regardless (`activator.go:1014-1033`), which releases the lease over a |
| +process that may still be writing. **The second `activator.go` change**: |
| +on `ErrStillRunning`, `stop()` does not call `rep.Close`; it marks the |
| +entry's outcome uncertain (the field exists, `outcome.uncertain`), leaves |
| +the entry in `stopping`, and `sweepOnce` retries `proc.Stop` on its |
| +cadence until `done` closes, after which the normal `rep.Close` runs. |
| +The lease's TTL bounds the worst case, as it does today when an agent |
| +dies mid-stop; what changes is that the agent never *voluntarily* |
| +releases over an unproven termination. |
| + |
| ### 6.4 Logs |
| Today `teeStd` (`backend_exec.go:123-137`) pipes the child's stdout and |
| @@ -968,13 +1015,16 @@ Each role's unit connects its stdout and stderr to that role's socket in |
| `/run/carlos/logs` (`StandardOutput=file:` on an `AF_UNIX` path; R1-2, |
| R1-6). The agent's `logListener` for a role accepts one connection per |
| start (the unit connects exactly once, at exec; a second connection |
| -during the same invocation is closed and logged, and can only come from |
| -PID 1 since no tenant mount contains the path), and feeds it to the same |
| -`applog.NewLineWriter` and the same `io.MultiWriter` to the agent's |
| +during the same invocation is closed and logged), and feeds it to the |
| +same `applog.NewLineWriter` and the same `io.MultiWriter` to the agent's |
| stdout. Same bytes, same cap, same sink, same `carlos logs`, and the |
| instance and its sidecar no longer contend for one connection. The |
| listener is bound in `prepare` before the start request, so PID 1's |
| -connect cannot race it. |
| +connect cannot race it. A flat tenant, during the transition, can |
| +connect to a log socket (same uid, host namespace) and write lines into |
| +another host's log stream; that is a nuisance the transition accepts and |
| +the next release ends, and it cannot make anything *read* wrong, because |
| +nothing reads the log stream as an instruction. |
| Grandchildren: a process the app forked that outlives it keeps the |
| socket open, as it keeps the pipe open today. `KillMode=mixed` plus |
| @@ -982,62 +1032,72 @@ socket open, as it keeps the pipe open today. `KillMode=mixed` plus |
| when systemd reaps it. EOF on the log connection ends **log collection |
| for that invocation**; it does not mean the process is gone (R1-4). |
| -### 6.5 Exit observation (codex R1-4, R2-4, R2-5) |
| +### 6.5 Exit observation (codex R1-4, R2-4, R2-5, R3-2, R3-7) |
| `childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`), |
| which is process death. A unit proc needs the same certainty, attributed |
| to the right invocation, and it must not depend on systemd keeping an |
| inactive unit loaded: systemd garbage-collects an unreferenced inactive |
| unit and its `ExecMainStatus` with it, so a `show` after a quick clean |
| -exit can find nothing to show (R2-4). So the record is written by the |
| -unit itself, on the way out: |
| - |
| -**`carlos instance-exit <role> <host>`** runs as `ExecStopPost=+` (root, |
| -outside the namespace, §4.1). It writes |
| -`/run/carlos/exits/<host>.<role>.<INVOCATION_ID>.json` with `O_EXCL` |
| -(`0640 root:carlos`), containing `{invocation, result, code, status, at}` |
| -from `$INVOCATION_ID`, `$SERVICE_RESULT`, `$EXIT_CODE`, `$EXIT_STATUS`. |
| -It runs whether the service exited, was stopped, timed out or was |
| -OOM-killed, and it runs after the invocation's processes have been |
| -signalled and reaped by systemd's stop sequence. The directory is |
| -root-owned and outside every tenant mount, so nothing but this command |
| -writes there. |
| - |
| -Each `unitProc` runs one goroutine that watches that directory with |
| -inotify (`IN_CLOSE_WRITE`/`IN_MOVED_TO` on the file for its invocation) |
| -and, as fallback, polls every 2s. A record alone is not termination: a |
| -service enters `failed` after its final `SIGKILL` timeout **with processes |
| -still present** if one is unkillable (`src/core/service.c:4020-4022`), |
| -and the lease must not be released over a writer (R2-5). So the |
| -goroutine, on seeing the record (or on the fallback finding the unit |
| -`inactive`/`failed` with an empty or absent `InvocationID`, which is a |
| -unit already unloaded), then requires **cgroup emptiness**: the unit's |
| -cgroup directory under the host slice is absent, or its `cgroup.events` |
| -says `populated 0`. Only then does it emit the member-facing event line |
| -with today's wording (`instance exited: ...` with the status or signal |
| -from the record, or `instance exited (exit 0)`; `instance exited (status |
| -unknown: no exit record)` in the unloaded-without-record case), set |
| -`exited`, and close `done`. A populated cgroup after a record is logged |
| +exit can find nothing to show (R2-4). Revision 3 tried to write the |
| +record from `ExecStopPost=+` into a root-owned directory; the `+` prefix |
| +drops `User=` and the `Private*`/`Protect*` settings but keeps the |
| +explicit `TemporaryFileSystem=`/`BindPaths=` mounts (`apply_mount_namespace` |
| +in `src/core/exec-invoke.c`), so that command could not see the |
| +directory (R3-2). So the record is **advisory, and truth is elsewhere**: |
| + |
| +**`carlos instance-exit <role> <host>`** runs as `ExecStopPost=` inside |
| +the namespace as `carlos`, after the invocation's processes have been |
| +signalled and reaped by systemd's stop sequence, with `$INVOCATION_ID`, |
| +`$SERVICE_RESULT`, `$EXIT_CODE` and `$EXIT_STATUS` in its environment. |
| +It writes `/run/carlos/instances/<host>/exit.<role>.<INVOCATION_ID>.json` |
| +(`O_EXCL`) with those four values. The directory is the instance's own, |
| +so the instance could have written that file itself; the record is |
| +therefore the tenant's claim about its own exit and is used for one |
| +thing only: the wording of the member-facing event line. It never |
| +decides whether the process is gone, and a record for a running unit is |
| +ignored (§3.3). The agent, as `carlos`, removes it once consumed, and |
| +`prepare` removes stale ones (R3-7). |
| + |
| +Each `unitProc` runs one goroutine that establishes termination from |
| +two facts the tenant cannot forge: |
| + |
| +1. **systemd's unit state**: `systemctl show -p ActiveState,SubState,Result,InvocationID <unit>` |
| + (read-only, unprivileged), polled every 2s and triggered early by |
| + inotify on the run directory (the record's arrival) and on the unit's |
| + `cgroup.events` file. Terminal is `inactive` or `failed` with the |
| + proc's own `InvocationID`, **or** `inactive` with an empty |
| + `InvocationID` (the unit was already unloaded; the proc's invocation |
| + is gone). |
| +2. **cgroup emptiness**: the unit's cgroup directory under the host |
| + slice is absent, or its `cgroup.events` says `populated 0`. A service |
| + enters `failed` after its final `SIGKILL` timeout **with processes |
| + still present** if one is unkillable (`src/core/service.c:4020-4022`), |
| + and the lease must not be released over a writer (R2-5). |
| + |
| +Only when both hold does the goroutine emit the event line with today's |
| +wording (`instance exited: ...` with the status or signal from the |
| +record if one exists for this invocation, or `instance exited (exit 0)`, |
| +or `instance exited (status unknown: no exit record)`), set `exited`, |
| +and close `done`. A populated cgroup after a terminal state is logged |
| every 2s as `instance <host>: processes remain after SIGKILL; not |
| -releasing`, and `done` stays open; that is a box-level incident, not a |
| -lease release. |
| - |
| -A record whose invocation is not this proc's (someone restarted the |
| -unit outside the activator, which `Restart=no` does not forbid) is |
| -logged as `unit restarted outside the activator`, the proc's own |
| -invocation is treated as gone (terminal, status unknown), and the |
| +releasing`, and `done` stays open (§6.3); that is a box-level incident, |
| +not a lease release. |
| + |
| +An `InvocationID` that is neither the proc's own nor empty means someone |
| +restarted the unit outside the activator, which `Restart=no` does not |
| +forbid. The goroutine logs `unit restarted outside the activator`, |
| +treats its own invocation as gone (terminal, status unknown) once its |
| +own cgroup emptiness cannot be told apart from the newcomer's, and the |
| sweeper's stand-down then `POST /stop`s whatever is running; the agent |
| owns these units and any other invocation is an anomaly to end, not to |
| adopt. |
| -The agent removes a record once it has consumed it; on start it removes |
| -every record older than its own start, since no proc can be waiting on |
| -them. `sweepOnce`'s existing dead-child check (`activator.go:1105-1122`) |
| -sees a flag, never a blocking call. |
| - |
| -`Restart=no` in the template is what makes this sound: systemd never |
| -restarts the unit behind the activator's back; a failed unit stays |
| -`failed` until the agent's next `start`, which clears it. |
| +`sweepOnce`'s existing dead-child check (`activator.go:1105-1122`) sees a |
| +flag, never a blocking call. `Restart=no` in the template is what makes |
| +this sound: systemd never restarts the unit behind the activator's back; |
| +a failed unit stays `failed` until the agent's next `start`, which clears |
| +it. |
| ### 6.6 cgroups, tiers and the sampler (codex R1-8, R1-9, R2-13) |
| @@ -1051,11 +1111,11 @@ directory. |
| Under units, systemd owns the cgroup. So: |
| - **Tier** travels in `POST /start` / `POST /seed` and lands on the host |
| - slice as the three properties of §4.3 plus the `memory.oom.group` |
| - write. The `Tier func(registry.Route) cgroup.Limit` seam and its box |
| - wiring (`stagedManifestOfLiveVersion`) are unchanged; only the consumer |
| - changes. The agent logs `instance: <host>: memory.max=…` on every start |
| - as it does today. Seed, instance and sidecar share the one cap. |
| + slice as the three properties of §4.3. The `Tier func(registry.Route) |
| + cgroup.Limit` seam and its box wiring (`stagedManifestOfLiveVersion`) |
| + are unchanged; only the consumer changes. The agent logs |
| + `instance: <host>: memory.max=…` on every start as it does today. Seed, |
| + instance and sidecar share the one cap. |
| - **Per-host sampling** gains |
| `/sys/fs/cgroup/carlos.slice/carlos-instances.slice/` as a scan root |
| and reads each `carlos-instances-<esc>.slice/memory.current` (the |
| @@ -1079,7 +1139,7 @@ Under units, systemd owns the cgroup. So: |
| platform graph is continuous across the roll. |
| - **`Manager.Start`, `CleanupHost`, the clone3 path and the delegation |
| dance** stay in use for flat hosts during this release and are deleted |
| - with the flat path in the next (§8.6). **`Delegate=` on the edge |
| + with the flat spawn path in the next (§8.6). **`Delegate=` on the edge |
| unit** goes with them. |
| ### 6.7 Agent start and restart; the sidecar restart (codex R1-7, R2-6) |
| @@ -1095,8 +1155,8 @@ start, before anything else: |
| 2. `POST /stop` for every unit that is not `inactive`, whatever its |
| role, and for every unit with a pending or running job (`stop` cancels |
| a queued start); |
| -3. wait for each to reach `inactive`/`failed` **and** for its cgroup to |
| - be empty (§6.5's rule), then remove stale exit records. |
| +3. wait for each to reach a terminal state **and** for its cgroup to be |
| + empty (§6.5's rule), then remove stale exit records. |
| Every instance is therefore asleep, and no job is queued, when the |
| activator's state machine begins, which is the invariant the activator |
| @@ -1104,14 +1164,14 @@ holds today. Adopting a running unit instead (re-acquiring its lease, |
| re-attaching replication) is a later improvement with its own review; it |
| is not needed for correctness. |
| -**The one change in `activator.go`.** `sweepOnce` restarts a dead |
| +**The first `activator.go` change.** `sweepOnce` restarts a dead |
| sidecar by calling `StartSidecar` under `a.mu` |
| (`activator.go:1165-1200`), justified there by "an exec is quick". A |
| unit start is an HTTP call and a `systemctl start`, and a stall there |
| would hold every other host's wake and stop. So the sidecar restart |
| becomes: under `a.mu`, mark `e.sidecarStarting = true`, capture the |
| entry's wake generation, and schedule the backoff exactly as now; then |
| -in a goroutine, `StartSidecar` with `controlTimeout`; then re-take |
| +in a goroutine, `StartSidecar` with its 30s deadline; then re-take |
| `a.mu`, and if the entry is still awake with the same generation, |
| install the proc as today, otherwise `Stop` it. The lock still makes a |
| restart impossible to race with a stop, because the install step checks |
| @@ -1166,7 +1226,7 @@ is derived from the same rule. |
| ## 8. Two layouts, one release, and the verb that moves between them |
| -### 8.1 Why nothing moves on upgrade (codex R2-9) |
| +### 8.1 Why nothing moves on upgrade, and nothing is minted nested (codex R2-9, R3-6) |
| `carlos system apply` is run by the **installed** binary: it stages the |
| candidate, restarts the edge, gates it, and on a failed gate restores the |
| @@ -1175,35 +1235,47 @@ previous files and restarts the edge again (`systemapply.go:360-372, |
| journal exists (`units/carlos-system-recover.service:17`); and a channel |
| rollback from the console (`system.go:466`) is an ordinary downgrade |
| install. In every one of those paths the code that runs after the |
| -downgrade is the *old* binary, which knows one layout. A migration that |
| -ran on the new agent's first start would therefore leave every failed |
| -first roll on a layout the recovering binary cannot serve. So: |
| +downgrade is the *old* binary, which knows one layout. So: |
| + |
| +**This release changes nothing about a host on upgrade, and mints |
| +nothing nested.** `routeFor` (`reconcile.go:299-301`, whose |
| +`DerivesPaths` predicate covers unflagged socket instances too, |
| +`provision.go:280`) and `carlos add` (`cli.go:118-136`) keep the flat |
| +defaults; a provisioning pass that runs during the candidate's gate |
| +creates a flat row the old binary serves (R3-6). Unflagged socket routes, |
| +whose supervisor owns their paths, keep flat paths in this release and |
| +every later one. The only writer of a nested path into a row is the |
| +layout pass, and it runs only on a recorded fleet intent. |
| -**This release changes nothing about a host on upgrade.** Its exec |
| -backend classifies each activator-managed route on every `Start`: |
| +The exec backend classifies each activator-managed route on every |
| +`Start`: |
| | Row | Classification | Spawn path | |
| |---|---|---| |
| -| `DBPath == FlatDBPath`, `Socket == FlatSocket` | `flat` | today's `childProc`, unchanged, unisolated | |
| -| `DBPath == DefaultDBPath` (nested), `Socket == DefaultSocket` (nested) | `nested` | `unitProc` (§6) | |
| +| `DBPath == DefaultDBPath` (flat), `Socket == DefaultSocket` (flat) | `flat` | today's `childProc`, unchanged, unisolated | |
| +| `DBPath == NestedDBPath`, `Socket == NestedSocket` | `nested` | `unitProc` (§6) | |
| | a journal exists in `/data/layout/<host>.json` | `fenced` | refuses to wake; reason from the journal | |
| -| anything else | `custom` | today's `childProc`; reported for the operator (§8.3) | |
| +| `DBPath` under `DataDir` and `Socket` under `SocketDir`, otherwise | `custom` | today's `childProc`; migrated from its actual paths by the pass (§8.3) | |
| +| anything else | `foreign` | today's `childProc`; reported for the operator (§8.3) | |
| A fleet rolled to this release behaves byte-for-byte as before until an |
| -operator runs the verb. A rollback before the verb is a rollback of |
| +operator records an intent. A rollback before that is a rollback of |
| nothing. The classification is published per route in the status |
| snapshot (§8.4) and is what every other consumer keys on. |
| -### 8.2 `carlos system layout migrate` and `revert` (codex R1-11, R1-12, R2-1, R2-7, R2-8) |
| +### 8.2 `carlos system layout migrate` and `revert` (codex R1-11, R1-12, R2-1, R2-7, R2-8, R2-9, R3-1) |
| The verb has the shape of `carlos restart` (#113): the console writes |
| -`control/layout.json` (`{id, requested_at, direction: nested|flat, scope}`, |
| -an append log like `restart-scope.json`), bumps the serial, and every box |
| -in scope acts on the serial move; boxes remember the last id they acted |
| -on. An old box never reads the object; an old console answers 404. Scope |
| -is the fleet, or `--box <label>` for one box, which is how a fleet |
| -migrates one box at a time with the console showing each box's rows |
| -turning `nested`. |
| +`control/layout.json` (`{id, requested_at, mode: nested|flat, scope}`, |
| +the newest entry of an append log like `restart-scope.json`), bumps the |
| +serial, and every box in scope converges to it. Scope is the fleet, or |
| +`--box <label>` for one box, which is how a fleet migrates one box at a |
| +time with the console showing each box's rows turning `nested`. **The |
| +recorded mode is the fleet's intent, and boxes converge to it on every |
| +serial move**, not once: a row that becomes eligible later (a route |
| +flagged for hibernation after the fleet went nested, a freshly |
| +provisioned flat row) is migrated on the next pass when it is asleep. An |
| +old box never reads the object; an old console answers 404. |
| On a box, the layout pass runs in the agent (uid `carlos`, no root |
| needed) and, per host in scope, in this order: |
| @@ -1214,17 +1286,23 @@ needed) and, per host in scope, in this order: |
| route with a socket is an externally supervised process (titogo's |
| router family is the live example); its files and rows are **not |
| touched**, it is not isolated by this change, and status reports it |
| - `external`. Only a `flat` row migrates; only a `nested` row reverts; |
| - `custom` and `fenced` rows are reported and skipped. |
| + `external`. Under mode `nested`, `flat` and `custom` rows migrate; |
| + under mode `flat`, `nested` rows revert; `fenced` rows recover first; |
| + `foreign` rows are reported and skipped. |
| 2. **Quiesce.** The pass asks the activator to stop the host |
| (`Bounce`-class stop with replication closed and the lease released, |
| - the same sequence `stop()` runs) and to refuse wakes for it until the |
| - pass is done. Nothing is writing. |
| -3. **Plan, in memory, from the row and the host name only.** Sources and |
| - destinations for `<host>.db`, `-wal`, `-shm`, the litestream shadow |
| - dir, `<host>.env`, `<host>-sidecar.env`, `<host>.config.env`, |
| - `<host>.db.corrupt`, computed by the §3.1 helpers. Nothing is read |
| - from the instance directory to build the plan (§3.3). |
| + the same sequence `stop()` runs, including §6.3's refusal to release |
| + over an unproven stop) and to refuse wakes for it until the pass is |
| + done. Nothing is writing. |
| +3. **Plan, in memory, from the row and the host name only.** Sources are |
| + the row's actual `DBPath` and `Socket` (for a `flat` row these are the |
| + flat defaults; for a `custom` row whatever the operator stamped, as |
| + long as they are under `DataDir` and `SocketDir`) and the siblings |
| + derived from them (`-wal`, `-shm`, the litestream shadow dir, |
| + `<host>.env`, `<host>-sidecar.env`, `<host>.config.env`, |
| + `<host>.db.corrupt` in the source directory); destinations are the |
| + §3.1 helpers. Nothing is read from the instance directory to build |
| + the plan (§3.3). |
| 4. **Ownership preflight** (R2-8). For every flat source name, compute |
| every route on the box that could own that name under the flat |
| rules: `<host>.config.env` is `host`'s sync-owned bundle **unless** a |
| @@ -1235,22 +1313,26 @@ needed) and, per host in scope, in this order: |
| on another route's file is a refusal, never an overwrite. A plan with |
| any ambiguous name is refused for that host with the two owners |
| named. |
| -5. **Destination check.** A destination that already exists is a |
| - refusal (`two databases (flat and nested); refusing to choose`), |
| - never a choice. Recorded in status; an operator resolves it on the |
| - box, which is the one box-local step this design keeps, for a state |
| - it never creates itself. |
| -6. **Journal.** Write `/data/layout/<host>.json` (`O_EXCL`), outside |
| - every tenant mount: `{host, direction, from: {db, socket}, to: {db, |
| - socket}, artifacts: [{name, present}], phase: "files"}`. Presence is |
| - recorded so recovery knows an absent optional artifact (a sidecar |
| - env, a quarantine) was absent, not lost (R2-7). From this moment the |
| - host is `fenced`: `Start`, `Seed`, `Restore`, the residence-move pass, |
| - `SyncExec`'s writer and the litestream generator all refuse it, with |
| - the journal's contents as the reason. The `MoveCutting` path in |
| - `movespass.go:285-305`, which would otherwise treat an absent DB as |
| - "nothing to checkpoint" and complete a flip, is the case the fence |
| - exists for (R1-12). |
| +5. **Destination check.** A destination `<host>.db` that already exists |
| + is a refusal (`two databases; refusing to choose`), never a choice. |
| + For the two env files, which an older binary running between a |
| + migration and a revert can have minted fresh at the flat path |
| + (`backend_exec.go:579`; R2-9 disposition), an existing destination is |
| + moved aside as `<name>.aside-<timestamp>` in the destination |
| + directory and the source takes its place: the source is the file the |
| + instance ran with for longer, the aside copy is kept, and the event |
| + line names both. `<host>.config.env` at the destination is simply |
| + replaced: sync re-renders it on its next pass regardless. |
| +6. **Journal.** Write `/data/layout/<host>.json` (`O_EXCL`): `{host, |
| + mode, from: {db, socket}, to: {db, socket}, artifacts: [{name, |
| + present}], phase: "files"}`. Presence is recorded so recovery knows |
| + an absent optional artifact (a sidecar env, a quarantine) was absent, |
| + not lost (R2-7). From this moment the host is `fenced`: `Start`, |
| + `Seed`, `Restore`, the residence-move pass, `SyncExec`'s writer and |
| + the litestream generator all refuse it, with the journal's contents |
| + as the reason. The `MoveCutting` path in `movespass.go:285-305`, which |
| + would otherwise treat an absent DB as "nothing to checkpoint" and |
| + complete a flip, is the case the fence exists for (R1-12). |
| 7. **Files.** `os.Rename` each artifact recorded present. Same |
| filesystem, same owner, atomic per file, no root. Then journal |
| `phase: "row"`. |
| @@ -1262,8 +1344,13 @@ needed) and, per host in scope, in this order: |
| reverse); release the host to the activator. The edge picks the new |
| `Socket` up on its next registry poll (`edge.go:449-451`, ~2s). |
| -**Recovery is the journal, and it is exact** (R2-7). On agent start and |
| -at the top of every layout pass, for each journal: |
| +**Recovery is the journal, and it is exact, and it is checked** (R2-7, |
| +R3-1). On agent start and at the top of every layout pass, for each |
| +journal: recompute the plan from the row (step 3); if the journal's |
| +`from`/`to` differ from the recomputation, or its `mode` differs from |
| +the fleet's recorded intent, the journal is refused (`layout: <host>: |
| +journal does not match row or intent; not acting`), the host stays |
| +`fenced`, and status says so. Otherwise: |
| - `phase: files`: for each artifact recorded present: at destination → |
| done; at source → rename; at neither → error, fence stays, status |
| @@ -1276,47 +1363,51 @@ at the top of every layout pass, for each journal: |
| A second clean agent start finds `nested` rows and no journals and does |
| nothing (the classification table runs before any legacy-default test). |
| -A forged journal-shaped file inside an instance directory is not a |
| -journal: the pass reads `/data/layout` only. |
| -**Revert** is the same pass with `direction: flat`, the same journal, |
| -the same fence, the same preflight in reverse. It exists so that a fleet |
| +**Revert** is the same pass under mode `flat`, the same journal, the |
| +same fence, the same preflight in reverse. It exists so that a fleet |
| can be taken back to a layout the previous release serves **before** |
| that release is installed: the operator's rollback order is `layout |
| revert` (this release, verified `flat` in status) then `system rollback`. |
| A `system rollback` or a failed-gate rollback that happens while hosts |
| are `nested` lands the old binary on rows it honours (`rt.DBPath`, |
| -`rt.Socket`; `backend_exec.go:220-225, 557-562`) but on a run directory |
| -it never creates, so those hosts fail to bind their socket, loudly and |
| -losslessly, until the new binary is back or the revert has run from it. |
| -No data is ever in two places: the fence and the destination check see |
| -to that. |
| - |
| -### 8.3 `custom` rows |
| - |
| -A row whose `DBPath` or `Socket` is neither the flat nor the nested |
| -default was set by hand (`carlos add --db`; note `carlos route` has no |
| -`--db` flag, `cli.go:469-475`, so it was set at add time). The pass does |
| -not guess: the row is reported `custom` in status with both values, and |
| -the host keeps running on today's flat path. `carlos add` on the box, |
| -re-adding the route with the flat defaults, is the repair, and it is |
| -box-local because the row was made box-locally; a console-provisioned |
| -row (`reconcile.go:300-301`) is never `custom`. |
| +`rt.Socket`; `backend_exec.go:220-225, 557-562`). On the same boot the |
| +nested run directories still exist, so the old binary can bind the |
| +nested socket and serve the nested database while minting flat env |
| +files; after a reboot the run directories are gone and those hosts fail |
| +to bind, loudly and losslessly. Either way the database has one history, |
| +the nested one, and the revert's step 5 handles the stray env files when |
| +the new binary is back. The layout is never two databases: the fence |
| +and the destination check see to that. |
| + |
| +### 8.3 `custom` and `foreign` rows |
| + |
| +A row whose `DBPath` or `Socket` is neither default was set by hand |
| +(`carlos add --db`; `carlos route` has no `--db` flag, `cli.go:469-475`, |
| +so it was set at add time). If both paths are still under `DataDir` and |
| +`SocketDir` the row is `custom` and **the pass migrates it from the |
| +paths it actually names** (step 3), through the console verb like every |
| +other row (R1-18). Only a row naming a path outside those trees is |
| +`foreign`: it is reported with both values, keeps running on today's |
| +flat path, and `carlos add` on the box, re-adding the route with the |
| +defaults, is the repair, box-local because the row was made box-locally. |
| +A console-provisioned row (`reconcile.go:300-301`) is never either. |
| ### 8.4 Status, through the API (codex R1-18) |
| `internal/status/snapshot.go` gains a per-route `layout` field with one |
| of `flat`, `nested`, `fenced` (with the journal's phase and reason), |
| -`custom` (with both values), `external`. `carlos status`, `carlos |
| -instances --layout` and the console's instance page render it; a box |
| -that predates the field shows `unknown (box predates layout reporting)`. |
| -It is the pre-roll inventory (every row `flat` or `custom` before the |
| -verb), the progress view during a migration, the member-visible proof |
| -that an instance is isolated (`nested`), and the reason a `fenced` host |
| -does not wake. `docs/site/cli.md` names the new verbs and flags (the |
| -docsite gate fails otherwise). |
| - |
| -### 8.5 Gate: no isolation, no nested tenants (codex R1-15, R2-14) |
| +`custom`, `foreign` (with both values), `external`, and a box-level |
| +`layout_mode` (`flat`, `nested`, or `none` before any intent). `carlos |
| +status`, `carlos instances --layout` and the console's instance page |
| +render them; a box that predates the field shows `unknown (box predates |
| +layout reporting)`. It is the pre-roll inventory, the progress view |
| +during a migration, the member-visible proof that an instance is |
| +isolated (`nested`), and the reason a `fenced` host does not wake. |
| +`docs/site/cli.md` names the new verbs and flags (the docsite gate fails |
| +otherwise). |
| + |
| +### 8.5 Gate: no isolation, no nested tenants (codex R1-15, R2-14, R3-4) |
| A box that cannot establish the namespace **does not start nested |
| tenants**. There is no fallback unit, no "proceed without |
| @@ -1333,37 +1424,49 @@ verdict line: |
| - `/data` and `/run/carlos` list nothing but the probe's own bound |
| directories; |
| - `/proc` lists only itself, and its own pid is 1; |
| -- `connect()` to `169.254.169.254:80` **and** to `[fd00:ec2::254]:80` |
| - fails with **`EPERM`** (the BPF filter's answer), while `connect()` to |
| - `127.0.0.1:1` fails with `ECONNREFUSED` (the network is live and the |
| - filter, not the network, said no). Any other errno on the metadata |
| - addresses, including a timeout, is `not enforced` (R2-14); |
| -- a symlink it creates in its directory is refused with `ELOOP`; |
| -- `/proc/self/mountinfo` shows `nosymfollow` on both bound directories. |
| +- a symlink it creates in its directory is refused with `ELOOP`, and |
| + `/proc/self/mountinfo` shows `nosymfollow` on both bound directories; |
| +- **IMDS, for a packet filter** (R3-4). `IPAddressDeny=` is a cgroup BPF |
| + egress filter; it drops packets, and a TCP `connect()` whose SYN is |
| + dropped retries and times out rather than returning `EPERM` |
| + (`tcp_connect`, `net/ipv4/tcp_output.c`). So the probe sends a UDP |
| + datagram, where the filter's verdict comes straight back from |
| + `sendmsg`: `sendto(169.254.169.254:80)` must fail with `EPERM`, and |
| + `sendto([fd00:ec2::254]:80)` must fail with `EPERM` **or**, if the |
| + host has no IPv6 route at all (the control service reads `ip -6 route |
| + show default` and passes the answer in as the probe's environment), |
| + with `ENETUNREACH`, which is then reported as `v6: no route, deny not |
| + exercised` rather than `enforced`. As a control, `sendto(127.0.0.1:9)` |
| + must succeed (the network stack is live and the filter, not the |
| + stack, said no). A 2s TCP `connect()` to the IPv4 address must not |
| + succeed. Any success on a metadata address, or any errno other than |
| + those named, is `not enforced`. |
| The service records the verdict; `GET /ready` returns it; the agent |
| refuses every nested wake with the verdict's failing line until it is |
| 204. The probe cannot pass while an instance would not be isolated, |
| because it runs under the same unit text (a test asserts the probe's |
| namespace block equals the templates'), and it fails only on a line |
| -that names the property that is missing. |
| +that names the property that is missing. It can fail while instances |
| +would be isolated only on a box with no IPv6 route whose operator has |
| +somehow made `ENETUNREACH` not the answer, which the verdict line names. |
| ### 8.6 Rollout order |
| 1. **Roll this release** to the dev fleet (`kass`, two boxes), boxes |
| before console, with `update-platform-binary.sh` enabling the new |
| units by name before it restarts the edge (§9). Every row stays |
| - `flat`; the probe verdict is `ready`; nothing else changes. Cold-boot |
| - one dev box and confirm both mounts, the control socket and the probe |
| - come up (R2-2). |
| + `flat`; `layout_mode` is `none`; the probe verdict is `ready`; nothing |
| + else changes. Cold-boot one dev box and confirm the three mounts, the |
| + control socket and the probe come up (R2-2). |
| 2. **`carlos system layout migrate --box <dev box>`**, watch the rows |
| turn `nested`, then the second box. The test that matters is from |
| inside: the plan's verification task provisions a throwaway instance |
| whose seed lists `/data` and `/run/carlos`, reads `/proc/1/status`, |
| - connects to both metadata addresses and to loopback, and plants and |
| - follows a symlink, and reads the results back through `carlos logs`. |
| - Run `layout revert` on one box and `migrate` again: both directions |
| - are exercised before the flagship sees either. |
| + sends the three datagrams, and plants and follows a symlink, and |
| + reads the results back through `carlos logs`. Run `layout revert` on |
| + one box and `migrate` again: both directions are exercised before the |
| + flagship sees either. |
| 3. **Flagship**: roll the release region by region, console box last; |
| then `layout migrate` one box at a time. The AU box has failed every |
| gate since 2026-09-07 for an unrelated reason; do not read its failure |
| @@ -1372,14 +1475,16 @@ that names the property that is missing. |
| the flagship has run a day, then migrates. The seven flagged instances |
| move; the router family (unflagged) is reported `external` and left |
| alone. Reply on #126 with the build and the check list above. |
| -5. **The next release** removes the flat spawn path, `FlatDBPath`, |
| - `FlatSocket`, the `custom`/`flat` classifications, the spawn half of |
| +5. **The next release** removes the flat spawn path for activator-managed |
| + hosts, points `DefaultDBPath`/`DefaultSocket` at the nested helpers |
| + for flagged routes (and keeps them flat for unflagged ones, whose |
| + supervisors own their paths), removes the spawn half of |
| `internal/cgroup` and `Delegate=` on the edge, and refuses to start on |
| - a box with a `flat` row. |
| + a box whose `layout_mode` is not `nested`. |
| **The roll script and the planner.** `update-platform-binary.sh:52-70` |
| (pinfra) enables units by name; it gains `carlos-instance-control.socket`, |
| -both mount units and the probe, before it restarts the edge, and |
| +the three mount units and the probe, before it restarts the edge, and |
| `bootstrap.go`'s `systemctlPlan` (`:165-179`) starts `.socket` and |
| `.mount` units it enables, which it does not today (R1-14). |
| `ensureSystemUnits` (`systemunits.go:34-83`) checks the control socket |
| @@ -1394,21 +1499,27 @@ unaffected, which is the point of the two-layout release. |
| The rule (AGENTS.md): the consumer list is everything that parses the |
| shape, not everything that calls the API. From the sweep of 2026-09-11, |
| -extended by codex R1-9, R1-14, R1-17, R1-18, R2-8, R2-9, R2-10, R2-13: |
| +extended by codex R1-9, R1-14, R1-17, R1-18, R2-8, R2-9, R2-10, R2-13, |
| +R3-1, R3-6: |
| **Path builders that move onto the helpers** |
| -- `internal/provision/provision.go:1011-1012` — `DefaultSocket`, `DefaultDBPath` (the definitions), plus `InstanceDir`, `InstanceRunDir`, `LogSocket`, `FlatDBPath`, `FlatSocket` |
| -- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); `:744-750` (socket normalisation, unchanged); every nested read through `instancefs`; the classification of §8.1 |
| +- `internal/provision/provision.go:280` (`DerivesPaths`, unchanged, but |
| + named: it is why unflagged routes mint flat), `:1011-1012` |
| + (`DefaultSocket`, `DefaultDBPath`, flat in this release), plus |
| + `InstanceDir`, `InstanceRunDir`, `NestedDBPath`, `NestedSocket`, |
| + `LogSocket` |
| +- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); `:744-750` (socket normalisation, unchanged); every nested read through `instancefs`; the classification of §8.1; the peer-uid check on the health and control dials (§5.4) |
| +- `internal/edge/edge.go:2000-2014` (`unixProxy`: the peer-uid check on every dial, §5.4) |
| - `internal/appconfig/syncexec.go:389-395, 465, 872, 921-926` (nested routes into the directory; flat and addr routes flat; `O_EXCL` temp); `:977-1024` `reapConfigEnv` walks both |
| - `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path, by classification) |
| - `internal/host/movespass.go:285-305` — rebuilds `<DataDir>/<host>.db` from the record; reads `rt.DBPath` from the row instead, and refuses a `fenced` host |
| -- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir" for nested rows), the new `carlos instances --layout` and `carlos system layout migrate|revert`; `docs/site/cli.md` names them |
| -- `internal/provision/reconcile.go:300-301` (`routeFor` mint) — new rows mint nested; `:461-464` and `:873-881` compare derived paths and log a `flat` row as drift for this one release (the message names the verb) |
| +- `cli.go:118-136` (`carlos add` defaults, unchanged and named), `:157-159` (relative-path refusal), the new `carlos instances --layout` and `carlos system layout migrate|revert`; `docs/site/cli.md` names them |
| +- `internal/provision/reconcile.go:299-301` (`routeFor` mint, unchanged and named); `:461-464` and `:873-881` compare derived paths and log a `nested` row as drift for this one release (the message names the layout mode) |
| **Readers of `rt.Socket` (no change; the value changes per host)** |
| -- `internal/edge/edge.go:536-538, 2000-2014` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| +- `internal/edge/edge.go:536-538` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| **The socket consumers group** (R2-10): `internal/host/bootstrap.go:41-62` |
| (`socketsGroup`, `tmpfilesConf` gains the lines of §3.2), the per-host |
| @@ -1424,16 +1535,16 @@ instance as an enrolled non-`carlos` uid |
| **Metrics and status** |
| - `internal/host/metrics_probes.go:49` (discovery: the two new roots), `internal/host/metrics_loop.go:659` (the edge's published value becomes the slice total), `internal/console/metricsview.go:728-762` (the key allowlist: unchanged because the sampler publishes under `carlos-exec@<host>`) |
| -- `internal/status/snapshot.go` gains the per-route `layout` field of §8.4; `carlos status` and the console instance page render it |
| +- `internal/status/snapshot.go` gains the per-route `layout` field and the box `layout_mode` of §8.4; `carlos status` and the console instance page render them |
| **The updater and the control plane** |
| -- `internal/host/systemapply.go:360-372, 738-748`, `carlos-system-recover.service`, `system.go:466`: unchanged, by design (§8.1); the layout verbs are a separate console door and bucket object (`control/layout.json`), a separate box pass, and a separate mark in `restart-marks.json`'s sibling `layout-marks.json` |
| +- `internal/host/systemapply.go:360-372, 738-748`, `carlos-system-recover.service`, `system.go:466`: unchanged, by design (§8.1); the layout verbs are a separate console door and bucket object (`control/layout.json`), a separate box pass that runs on every serial move, and a separate mark in `restart-marks.json`'s sibling `layout-marks.json` |
| - `internal/console`: the `POST /api/cli/system/layout` door (fleet-operator gate, fresh auth), `carlos system layout` in `system.go` |
| **Units and installers** |
| -- `internal/host/units/` gains: three templates, two slices, two mounts, the probe, the control socket and service (§4, §5, §8.5); `carlos-edge.service` gains `Slice=`, `Requires=`/`After=` on the mounts, and drops its memory lines; `carlos-litestream.service` gains the `After=` |
| +- `internal/host/units/` gains: three templates, two slices, three mounts, the probe, the control socket and service (§4, §5, §8.5); `carlos-edge.service` gains `Slice=`, `Requires=`/`After=` on the mounts, and drops its memory lines; `carlos-litestream.service` gains the `After=` |
| - `internal/host/bootstrap.go:61` (tmpfiles lines), `:116-122` `enabledUnits` (control socket, mounts, probe), `:165-179` `systemctlPlan` (starts `.socket`/`.mount`) |
| - `internal/host/systemunits.go:34-83` (active check on the socket) |
| - `update-platform-binary.sh:52-70` (pinfra): enables the new units by name before restarting the edge |
| @@ -1451,63 +1562,76 @@ sentinel `.path` units name fixed files under `/data`, never a host. |
| Unit level, all runnable in CI (no systemd needed): |
| -- `internal/provision`: `InstanceDir`, the nested defaults, the flat |
| - helpers, `LogSocket`, the escape function against `systemd-escape` |
| - fixtures, `carlos add` refusal of a nested `--db` outside the directory. |
| +- `internal/provision`: `InstanceDir`, the nested helpers, the flat |
| + defaults unchanged, `LogSocket`, the escape function against |
| + `systemd-escape` fixtures. |
| - `instancefs`: refuses a symlink at every component, refuses `..`, |
| - `O_EXCL` temp plus rename, the mountinfo check. |
| -- `internal/activator`: classification of the four row shapes; `prepare` |
| - mints the token for seed and run, writes `launch.<role>.json` with the |
| - deduplicated env and the exact argv (the existing env-composition tests |
| - move their assertion to the effective environment of an executed stub); |
| - the control client sends the tier and the token with a deadline, and |
| - retries once on 401 after re-reading the token; `unitProc.Stop` orders |
| - stop → done → close listeners; `Exited()` only after an exit record (or |
| - an unloaded unit) **and** an empty cgroup, never on log EOF, with a |
| - populated cgroup holding `done` open; a foreign invocation's record; |
| - the failed-health path posts `stop` and waits; seed timeout and |
| - cancellation stop-and-wait before cleanup; the startup sweep stops |
| - activating units, cancels jobs and waits for empty cgroups; the sidecar |
| - restart reserves under the lock and installs or stops after, with a |
| - stalled start while another host wakes and stops. The control channel |
| - is a seam (`Control interface { Start; Seed; Stop; Reset; Ready }`) |
| - with a fake, as `Systemctl` is today; the exit observer takes a fake |
| - records directory and a fake cgroup tree. |
| + `O_EXCL` temp plus rename, the mountinfo check for all three mounts. |
| +- `internal/edge` and `internal/activator` dials: a listener whose peer |
| + is uid 0 is refused before a byte is sent; a `carlos` or DynamicUser |
| + peer is accepted. |
| +- `instance-control`: a connection whose peer pid is not the edge's |
| + `MainPID`, or whose uid is not `carlos`, is closed without a response; |
| + every 400 (bad host, bad role, bad limit); the unit and slice names it |
| + builds for a host with a dash; `set-property` (three properties) → |
| + `start` ordering with no `reset-failed`; `/reset` tolerating "not |
| + loaded"; `/seed` returning the show fields when loaded and an empty id |
| + when not; `/ready` from the probe verdict; with a fake `systemctl` |
| + runner and a fake peer. |
| +- `internal/activator`: classification of the five row shapes; |
| + `prepare` mints the token for seed and run, writes `launch.<role>.json` |
| + with the deduplicated env and the exact argv (the existing |
| + env-composition tests move their assertion to the effective |
| + environment of an executed stub); the control client's per-verb |
| + deadlines (30s, 150s for seed); `unitProc.Stop` orders stop → done → |
| + close listeners and returns `ErrStillRunning` on expiry without |
| + closing; `stop()` skips `rep.Close` on `ErrStillRunning` and the |
| + sweeper retries; `Exited()` only after a terminal unit state **and** an |
| + empty cgroup, never on log EOF or on a record alone, with a populated |
| + cgroup holding `done` open; a forged exit record for a running unit |
| + ignored; a foreign invocation; the failed-health path posts `stop` and |
| + waits; seed timeout and cancellation stop-and-wait before cleanup; the |
| + startup sweep stops activating units, cancels jobs and waits for empty |
| + cgroups; the sidecar restart reserves under the lock and installs or |
| + stops after, with a stalled start while another host wakes and stops. |
| + The control channel is a seam (`Control interface { Start; Seed; Stop; |
| + Reset; Ready }`) with a fake; the exit observer takes a fake `show`, a |
| + fake run directory and a fake cgroup tree. |
| - `instance-exec`: reads the file through `instancefs`, refuses schema |
| ≠ 1, refuses an `argv[0]` that is not the live link, execs. |
| - `instance-exit`: writes the record with `O_EXCL` from the four |
| - variables; refuses to overwrite. |
| -- `instance-control`: 401 without the token; every 400 (bad host, bad |
| - role, bad limit); the unit and slice names it builds for a host with a |
| - dash; `set-property` (three properties) → `oom.group` write → `start` |
| - ordering with no `reset-failed`; `/reset` tolerating "not loaded"; |
| - `/ready` from the probe verdict; with a fake `systemctl` runner and a |
| - fake cgroup tree. |
| + variables into the run directory; refuses to overwrite. |
| - `instance-probe`: each verdict line, including `EPERM` versus |
| - `ECONNREFUSED` versus timeout on the metadata addresses, with a fake |
| - dialer. |
| + `ENETUNREACH` (with and without a v6 route) versus other errnos on the |
| + datagram sends, the loopback control, and the TCP connect bound, with |
| + a fake socket layer. |
| - `logListener`: one connection per invocation, a second refused and |
| logged, line cap, EOF ends collection without marking exit. |
| - Layout pass: eligibility by the activator's predicate (flagged only; |
| - external, tunnel, unit-backed untouched); quiesce before plan; the |
| - ownership preflight for `x`/`x.config` in both directions; destination |
| - collision refusal; the journal's phases and every crash boundary |
| - (before files, mid-files, after files, after CAS, before journal |
| - removal) recovered exactly, with absent optional artifacts; a CAS |
| + external, tunnel, unit-backed untouched); convergence on every serial |
| + move; a later-flagged row migrating on the next pass; `custom` rows |
| + migrated from their actual paths; `foreign` rows reported; quiesce |
| + before plan; the ownership preflight for `x`/`x.config` in both |
| + directions; DB destination collision refused, env destination moved |
| + aside; the journal's phases and every crash boundary (before files, |
| + mid-files, after files, after CAS, before journal removal) recovered |
| + exactly, with absent optional artifacts; a journal whose paths or mode |
| + do not match the recomputed plan and the fleet's intent refused; a CAS |
| conflict keeps the fence; a second clean start does nothing; every |
| fenced consumer (`Start`, `Seed`, `Restore`, moves pass, `SyncExec`, |
| litestream generator) refuses a fenced host; a partial rename followed |
| by a `MoveCutting` pass does not complete the flip; revert restores the |
| - flat layout byte for byte; a forged journal-shaped file and a forged |
| - launch file in an instance directory move nothing (§3.3). |
| + flat layout byte for byte, aside files included; a forged journal, a |
| + forged launch file and a forged exit record move nothing (§3.3). |
| - Units: a test over `unitsFS` asserts the three templates' and the |
| probe's namespace, hardening and coupling blocks are identical, that |
| `PrivatePIDs=yes`, `TemporaryFileSystem=/data:ro`, |
| `IPAddressDeny=169.254.169.254/32`, `Restart=no`, |
| `BindsTo=carlos-edge.service`, `Slice=carlos-instances-%i.slice`, |
| - `ExecStopPost=+` and the `%I` paths are present, that the seed carries |
| - `TimeoutStartSec=120`, that the control socket is `SocketMode=0600 |
| - SocketUser=carlos`, that both mounts carry `nosymfollow` and |
| + `OOMPolicy=kill`, an `ExecStopPost=` **without** `+`, and the `%I` |
| + paths are present, that the seed carries `TimeoutStartSec=120`, that |
| + the control socket is `SocketMode=0600 SocketUser=carlos` under |
| + `/run/carlos-control`, that all three mounts carry `nosymfollow` and |
| `DefaultDependencies=no` with the ordering of §3.2, and that |
| `carlos-edge.service` carries `Slice=carlos.slice` and no `MemoryMax=`; |
| and, where `systemd-analyze` exists on the CI host, `systemd-analyze |
| @@ -1518,10 +1642,10 @@ Unit level, all runnable in CI (no systemd needed): |
| sampler roots publish per-host under `carlos-exec@<host>` and the edge |
| as the slice total, flat hosts still from the delegated subtree; the |
| #103 consumer dial through the nested run directory. |
| -- `internal/status`: the `layout` field for each of the five states and |
| - the pre-field box. |
| +- `internal/status`: the `layout` field for each of the six states, the |
| + box mode, and the pre-field box. |
| - `internal/console` and `system.go`: the layout door's gate and |
| - refusal order, the bucket object, the scope forms. |
| + refusal order, the bucket object, the scope forms, convergence marks. |
| Box level, in the plan's verification task and in the rollout: |
| @@ -1531,18 +1655,19 @@ Box level, in the plan's verification task and in the rollout: |
| `IPAddressDeny=`) turns it `not enforced` and blocks nested wakes. |
| - Migrate one host; wake it; `systemctl status carlos-instance@<esc>` |
| active in `carlos-instances-<esc>.slice` with the three memory |
| - properties and `memory.oom.group=1`; from inside via the throwaway |
| - instance's seed: `/data` and `/run/carlos` list one directory each, |
| - `/proc/1` is the app, both metadata addresses `EPERM`, loopback |
| - `ECONNREFUSED`, a planted symlink `ELOOP`; `carlos logs` shows all of |
| - it for both instance and sidecar at once; a quick-exiting seed's record |
| - arrives after the unit is unloaded; hibernate, unit gone, record |
| - consumed; `carlos restart` cycles it; `systemctl restart carlos-edge` |
| - stops it and does not start it; an enrolled `carlos-sockets` uid dials |
| - `app.sock` and cannot open the control socket; `carlos instances |
| - --layout` says `nested`; the console's memory graphs for the app and |
| - for the platform are continuous across the migration; revert, and the |
| - previous release serves the host again. |
| + properties; from inside via the throwaway instance's seed: `/data` and |
| + `/run/carlos` list one directory each, `/proc/1` is the app, the three |
| + datagrams answer as §8.5 says, a planted symlink `ELOOP`; `carlos logs` |
| + shows all of it for both instance and sidecar at once; a quick-exiting |
| + seed's status arrives; hibernate, unit gone, record consumed; `carlos |
| + restart` cycles it; `systemctl restart carlos-edge` stops it and does |
| + not start it; an enrolled `carlos-sockets` uid dials `app.sock` and |
| + cannot open the control socket; a flat tenant on the same box cannot |
| + get a response from `control.sock` directly or through its own |
| + redirected socket; `carlos instances --layout` says `nested`; the |
| + console's memory graphs for the app and for the platform are |
| + continuous across the migration; revert, and the previous release |
| + serves the host again. |
| --- |
| @@ -1554,8 +1679,8 @@ Box level, in the plan's verification task and in the rollout: |
| layout with a root chown in the control service. Not in this change. |
| - **Externally supervised exec routes** (unflagged instance routes with |
| a socket, §8.2 step 1): not isolated by this change; their supervisor |
| - owns their process. Whether the platform should offer them the same |
| - namespace is a discussion to open once this lands. |
| + owns their process and their flat paths. Whether the platform should |
| + offer them the same namespace is a discussion to open once this lands. |
| - **Unit-backed console instances.** Not needed for isolation, and a |
| hibernation rewrite. Whether `backing=unit` hibernation works on the |
| fleet today (`UnitBackend` runs `systemctl` unprivileged, §1.4) is its |
| @@ -1566,8 +1691,12 @@ Box level, in the plan's verification task and in the rollout: |
| invisible to isolated instances but not to the agent. |
| - **Adopting running units on agent start** instead of stopping them |
| (§6.7). |
| -- **The next release** (§8.6 step 5): drop the flat path and the spawn |
| - half of `internal/cgroup`. |
| +- **The next release** (§8.6 step 5): drop the flat spawn path and the |
| + spawn half of `internal/cgroup`; with it, the transition-only |
| + exposures §3.2 and §6.4 name (a flat tenant redirecting its own socket, |
| + a flat tenant writing to a log socket) end. |
| +- **Whole-family OOM across instance and sidecar** (§4.3): not |
| + reproduced; recorded as a deliberate difference. |
| - **Discussion #1 (memory pressure)**: no new numbers here (§4.3); the |
| slice shape makes a later change a unit-file edit, never a drop-in. |
| @@ -1590,12 +1719,14 @@ Box level, in the plan's verification task and in the rollout: |
| Within 3: trampoline over `EnvironmentFile=` (§6.2, one env composer); |
| three templates over one (§4.2, the instance is the host); the control |
| -service with a token over polkit (§5.4, ruled with codex); |
| -`nosymfollow` mounts over per-read symlink checks (§3.2, kernel-enforced |
| -and race-free where a check-then-open is neither); an exit record from |
| -`ExecStopPost=+` over polling unit state (§6.5, survives unit unload); |
| -and an operator verb over an agent-start migration (§8.1, because the |
| -updater's rollback runs the old binary). |
| +service authenticating the agent's pid over polkit and over a token |
| +(§5.4, ruled with codex across two rounds); `nosymfollow` mounts over |
| +per-read symlink checks (§3.2, kernel-enforced and race-free where a |
| +check-then-open is neither); termination from unit state plus cgroup |
| +emptiness, with the unit's own record as decoration, over any record as |
| +proof (§6.5); and an operator verb recording a fleet intent that boxes |
| +converge to over an agent-start migration (§8.1, because the updater's |
| +rollback runs the old binary). |
| --- |
| @@ -1603,14 +1734,14 @@ updater's rollback runs the old binary). |
| - **Round 1** (codex, GPT-6, `CODEX-FINDINGS-ISOLATED-EXEC.md`, 19 |
| findings, verdict request changes). All accepted: |
| - R1-1 control service with token, `nosymfollow` (§3.2, §5.4); |
| + R1-1 control service, `nosymfollow` (§3.2, §5.4); |
| R1-2 log sockets outside tenant mounts, per role (§3.1, §6.4); |
| R1-3 `nosymfollow` plus `instancefs` and `O_EXCL` temps (§3.2, §7); |
| - R1-4 exit from a record plus cgroup emptiness, never from EOF (§6.5); |
| + R1-4 exit from unit state plus cgroup emptiness, never from EOF (§6.5); |
| R1-5 seed timeout, stop verb, wait before cleanup (§4.2, §5.3, §6.2); |
| R1-6 per-role log listeners (§6.4); |
| R1-7 `BindsTo=` not `PartOf=`, sweep covers jobs (§4.1, §6.7); |
| - R1-8 per-host slice carries the tier and `memory.oom.group` for all three roles (§4.3, §6.6); |
| + R1-8 per-host slice carries the tier for all three roles; whole-family OOM recorded as a difference after R3-3 (§4.3); |
| R1-9 cgroup path under `carlos.slice`, key compatibility, `metrics_probes.go` (§6.6, §9); |
| R1-10 one common wall in `carlos.slice`, no new numbers (§4.3); |
| R1-11 eligibility is the activator's predicate (§8.2); |
| @@ -1620,21 +1751,30 @@ updater's rollback runs the old binary). |
| R1-15 no `PrivatePIDs=` fallback; behavioural gate and probe (§8.5); |
| R1-16 env deduplication, effective-value test (§6.2); |
| R1-17 nested glob is discovery, not reporting (§9); |
| - R1-18 `layout` status through the API and CLI, `docs/site/cli.md`; repair is `carlos add`, not a `--db` flag `carlos route` does not have (§8.3, §8.4); |
| + R1-18 `layout` status through the API and CLI; `custom` rows migrated through the verb; `docs/site/cli.md` (§8.3, §8.4); |
| R1-19 `UnitBackend` claim removed (§1.4, §11). |
| - **Round 2** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R2.md`, 14 new |
| findings, verdict request changes). All accepted: |
| - R2-1 journal outside tenant mounts, plan from row and host only, nothing in an instance directory is an instruction (§3.3, §8.2); |
| - R2-2 mounts with `DefaultDependencies=no` and explicit ordering, cold-boot verification, `systemd-analyze verify` in CI (§3.2); |
| + R2-1 journal outside tenant mounts, plan from row and host only (§3.3, §8.2); |
| + R2-2 mounts with `DefaultDependencies=no` and explicit ordering, cold-boot verification (§3.2); |
| R2-3 no `reset-failed` in `/start` (§5.3); |
| - R2-4 exit record from `ExecStopPost=+`, survives unit unload; foreign invocation handling (§6.5); |
| - R2-5 cgroup emptiness required on every path before `done` (§6.5); |
| - R2-6 sidecar restart reserved under the lock, started outside it with a deadline (§5.3, §6.7); |
| + R2-4 termination not dependent on unit retention; `/seed` correlation (§5.3, §6.5); |
| + R2-5 cgroup emptiness before `done`; `stop()` does not release over `ErrStillRunning` (§6.3, §6.5); |
| + R2-6 sidecar restart reserved under the lock, started outside it with a deadline (§6.7); |
| R2-7 journal records artifact presence and phase; committed-row recognition; classification before legacy test (§8.1, §8.2); |
| R2-8 ownership preflight in both directions from `SyncExec`'s rule (§8.2); |
| - R2-9 two-layout release and operator verb; the updater's rollback lands on a layout the old binary serves (§8.1, §8.2, §8.6); |
| + R2-9 two-layout release and operator verb; same-boot rollback behaviour stated; env aside rule (§8.1, §8.2); |
| R2-10 `carlos-sockets` traversal preserved on the run directories (§1.1, §3.1); |
| R2-11 `MemorySwapMax` on the host slice (§4.3); |
| R2-12 seed prepares the admin token and merged env (§6.1); |
| R2-13 edge discovered in `carlos.slice`, published as the slice total (§6.6); |
| - R2-14 probe distinguishes `EPERM` from unreachable, tests both metadata addresses (§8.5). |
| + R2-14 probe distinguishes enforcement from unreachability (§8.5, reworked by R3-4). |
| +- **Round 3** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R3.md`, 7 new |
| + findings, verdict request changes). All accepted: |
| + R3-1 no token; the control service authenticates the edge's `MainPID`; the edge refuses root peers; `/run/carlos/logs` on a `nosymfollow` mount; journals checked against the recomputed plan and the fleet's intent (§3.2, §3.3, §5.4); |
| + R3-2 exit record from `ExecStopPost=` without `+`, advisory only, in the run directory (§4.1, §6.5); |
| + R3-3 no manual `memory.oom.group`; per-service families, difference recorded (§4.3); |
| + R3-4 UDP `EPERM` probe with a loopback control and a no-route rule for v6 (§8.5); |
| + R3-5 150s seed deadline (§5.3); |
| + R3-6 nothing minted nested in this release; unflagged routes flat by contract (§3.1, §8.1, §8.6); |
| + R3-7 the record lives where the agent can delete it (§6.5). |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R3.md| new file mode 100644 |
| index 000000000..17b1c8200 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R3.md |
| @@ -0,0 +1,91 @@ |
| +| Prior finding | Disposition at `bcf3fcbb` | |
| +|---|---| |
| +| R2-1 | **Resolved for the reported forged-journal attack.** Authoritative journals move outside nested tenant mounts. The newly retained flat execution path creates a separate exposure, R3-1. | |
| +| R2-2 | **Resolved.** Explicit mount ordering removes the reported cycle; cold-boot verification is specified. | |
| +| R2-3 | **Resolved.** `/start` no longer resets an unloaded service. | |
| +| R2-4 | **Partially resolved.** Persistent records are specified, but `/seed` still obtains its invocation ID through `show` **after** the oneshot finishes and can unload ([§5.3:765](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:765)). Retain request-to-invocation correlation before completion. A foreign invocation’s exit record also cannot detect that invocation while it remains running. The new record writer has an additional defect, R3-2. | |
| +| R2-5 | **Partially resolved.** `done` requires emptiness, but `Stop` may return on context expiry ([§6.3:950](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:950)). The unchanged activator then calls `rep.Close`, releasing the lease despite unsuccessful termination ([activator.go:1015](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/activator.go:1015), [1033](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/activator.go:1033)). Preserve the lease until termination is proven. | |
| +| R2-6 | **Resolved at spec level.** Reservation, bounded work outside the mutex, generation checks and stop coordination are specified. | |
| +| R2-7 | **Resolved.** Presence recording, committed-row recognition and settled-nested classification cover the reported recovery states. | |
| +| R2-8 | **Resolved.** Ownership preflight covers both migration directions. | |
| +| R2-9 | **Partially resolved.** Explicit migration protects existing rows during the initial upgrade, but new-row creation contradicts that protection, R3-6. The claimed lossless refusal after downgrade is also false: a same-boot rollback retains nested run directories, so the old backend can serve nested DB/socket paths while minting a flat env file ([backend_exec.go:557](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/backend_exec.go:557), [579](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/backend_exec.go:579)). That creates a destination collision which the subsequent revert refuses. The downgrade protocol must prevent this mixed-layout execution. | |
| +| R2-10 | **Resolved.** The run-directory traversal and socket-group contract are preserved. | |
| +| R2-11 | **Resolved.** The shared host slice receives `MemorySwapMax`. | |
| +| R2-12 | **Resolved.** Seed and run both prepare the admin token before composing their environment. | |
| +| R2-13 | **Resolved.** Relocated discovery and preservation of the published aggregate are explicit. | |
| +| R2-14 | **Partially resolved.** Both address families are tested and unrelated failures cannot produce readiness, but the replacement assertion rejects effective filtering, R3-4. | |
| +| R1-4 | **Partially resolved.** EOF and mutex defects are addressed; the remaining completion defects are recorded under R2-4, R2-5 and R3-2. | |
| +| R1-5 | **Partially resolved.** Preparation, cancellation and cleanup ordering are specified. Completion remains affected by the findings above, and the new control deadline shortens the seed budget, R3-5. | |
| +| R1-8 | **Partially resolved.** The shared tier is restored, but systemd can overwrite the proposed family OOM setting, R3-3. | |
| +| R1-12 | **Resolved at spec level.** The journal, recovery protocol and named consumer fences address the reported partial-migration failure. | |
| +| R1-13 | **Partially resolved.** Reverse migration exists; the remaining downgrade defects are recorded under R2-9. | |
| +| R1-15 | **Partially resolved.** The unsafe fallback is removed, but the replacement gate cannot establish readiness as written, R3-4. | |
| +| R1-18 | **Partially resolved.** Remote reporting and compatibility with pre-field boxes are specified. Repair remains explicitly box-local in [§8.3:1301](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1301); the requested API-backed repair operation remains absent. | |
| + |
| +**R3-1 — P1 — §§5.4, 8.1–8.3: Retain the authenticated control service, but protect it from the flat tenants this revision keeps running.** |
| + |
| +The two-layout release runs `flat` and `custom` hosts as ordinary agent children. Those processes can read `/run/carlos/instance-control.token`, connect to the control socket, and write `/run/carlos/logs`. The assertion that the token is unavailable to tenants therefore becomes false during the specified transition. |
| + |
| +This also exposes a new root-write route. A flat tenant can replace a role’s log pathname with a symlink to a root-owned file, then authenticate a start request. It can create the required nested directories and launch description for its own host. Systemd opens the specified stdout pathname before applying the tenant sandbox, handing the app a writable descriptor to the target. The `nosymfollow` mounts do not cover `/run/carlos/logs`. |
| + |
| +Evidence: [flat/custom execution:1187](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1187), [token:746](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:746), [log directory:273](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:273), [systemd stdout setup](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/exec-invoke.c). |
| + |
| +**Correction:** retain the bearer-authenticated service, but define a transition boundary that excludes every unisolated tenant from its credentials and root-opened paths. Do not activate this authority alongside unchanged same-uid flat children. Include `custom` and excluded same-uid processes in that boundary. |
| + |
| +**R3-2 — P1 — §§4.1, 6.5: `ExecStopPost=+` still receives the explicit tmpfs mounts and cannot write the exit record.** |
| + |
| +The `+` prefix disables privilege restrictions and several sandbox settings. It does **not** remove the explicit `TemporaryFileSystem=` and `BindPaths=` configuration in systemd 257. Its execution code passes both into namespace setup independently of `needs_sandboxing`. |
| + |
| +Consequently, `instance-exit` runs as root but still sees the empty `/run/carlos` tmpfs with only the instance directory bound back in. `/run/carlos/exits` is absent. The helper cannot write its record; its failure can leave the unit failed with an invocation ID, which also excludes the spec’s unloaded-unit fallback. `done` can remain open permanently. |
| + |
| +Evidence: [template:509](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:509), [record writer:995](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:995); systemd’s `apply_mount_namespace`, particularly lines 2993 and 3093–3097, in [exec-invoke.c](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/exec-invoke.c). |
| + |
| +**Correction:** use a completion-record mechanism that actually operates in the host mount namespace. Keep its writable endpoint unavailable to the app; `+` alone does not establish that separation. |
| + |
| +**R3-3 — P2 — §§4.3, 6.6: systemd manages the slice’s `memory.oom.group` and can clear the manual setting.** |
| + |
| +The spec explicitly relies on systemd leaving this attribute alone. In v257, the common memory-controller application code writes `memory.oom.group` from the unit’s `CGroupContext`, including for slices. Its default is false. |
| + |
| +Reapplying memory properties therefore writes `0`. The specified sidecar-start sequence does exactly that on a slice whose main process is already running, before the broker writes `1` again. Other systemd reapplications have no corresponding repair step. The family OOM guarantee does not hold continuously. |
| + |
| +Evidence: [§4.3:679](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:679); `cgroup_context_apply`, lines 2030–2052, in [systemd cgroup.c](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/cgroup.c), and [slice property application](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/dbus-slice.c). |
| + |
| +**Correction:** give the family OOM setting a supported ownership and update mechanism that preserves it while tenants run, including across property changes and reloads. Remove the claim that systemd does not manage it. |
| + |
| +**R3-4 — P1 — §8.5: The readiness probe rejects correctly enforced IMDS filtering.** |
| + |
| +`IPAddressDeny=` attaches packet filters, not a connect-address rejection hook. For an ordinary TCP connection, dropping the outgoing SYN does not guarantee that `connect()` returns `EPERM`. Linux’s TCP connect path retains most transmission failures and retries the SYN; the caller can time out. An unavailable IPv6 route can fail before filtering is exercised. |
| + |
| +The probe classifies both outcomes as `not enforced`, so a correctly isolated box can never become ready and its nested tenants cannot wake. |
| + |
| +Evidence: [required errno:1336](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1336), [systemd ingress/egress filter attachment](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/bpf-firewall.c), and `tcp_connect`, lines 3862–3881, in [Linux 6.12 tcp_output.c](https://raw.githubusercontent.com/torvalds/linux/v6.12/net/ipv4/tcp_output.c). |
| + |
| +**Correction:** prove effective filtering using an observation appropriate to packet filtering, with controlled reachability and evidence for both address families. Preserve the refusal on unavailable evidence; do not substitute either “any connection failure” or “must return EPERM.” |
| + |
| +**R3-5 — P1 — §§5.3, 6.2: The universal 30-second control deadline makes valid seeds permanently unwakeable.** |
| + |
| +`/seed` blocks until its oneshot finishes, but every agent control request now has a 30-second deadline. A seed that normally completes in 45 seconds is cancelled and stopped on every wake, despite the explicitly retained 120-second seed budget. Its partial database is then removed and the next wake repeats the failure. |
| + |
| +Evidence: [blocking seed request:766](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:766), [universal deadline:799](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:799), [seed cleanup:927](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:927), [existing budget:466](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/backend_exec.go:466). |
| + |
| +**Correction:** give the blocking seed request a budget covering seed execution and termination, or return its invocation promptly and observe completion separately. Keep sidecar control calls bounded independently. |
| + |
| +**R3-6 — P1 — §§8.1, 8.2, 9: Unconditional nested defaults bypass the migration boundary.** |
| + |
| +Section 9 requires newly provisioned rows to use nested paths. A provisioning pass during the candidate’s gate can therefore create a nested host before any `layout migrate` command. Failed-gate rollback then encounters the incompatible layout that the two-layout release claims cannot exist yet. |
| + |
| +The same default change also reaches externally supervised routes. `routeFor` uses `DerivesPaths()`, whose predicate includes socket instances with `Hibernate=false`; `carlos add` likewise assigns defaults independently of hibernation. Those new routes receive nested paths although their unchanged supervisors still use the flat layout. |
| + |
| +Evidence: [new-row requirement:1407](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1407), [compatibility promise:1182](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1182), [routeFor:299](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/provision/reconcile.go:299), [DerivesPaths:280](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/provision/provision.go:280), [CLI defaults:118](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/cli.go:118). |
| + |
| +**Correction:** select new-row defaults using both activator eligibility and explicit box migration state. Before migration is enabled, retain flat defaults; externally supervised routes retain their existing path contract. |
| + |
| +**R3-7 — P2 — §§3.1, 6.5: The agent cannot delete consumed exit records.** |
| + |
| +`/run/carlos/exits` is `0750 root:carlos`, while the agent runs as uid `carlos`. Reading a record is permitted; unlinking it requires write permission on the directory, which the agent lacks. Both per-record acknowledgement and startup cleanup therefore fail, leaving records to accumulate in `/run`. |
| + |
| +Evidence: [directory ownership:277](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:277), [required cleanup:1033](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1033). |
| + |
| +**Correction:** assign acknowledgement and cleanup to the root broker through a bounded operation, preserving the protection against forged records. |
| + |
| +**Verdict: request changes.** |
| \ No newline at end of file |