| new file mode 100644 |
| index 000000000..7edd2597e |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| @@ -0,0 +1,1000 @@ |
| +# Isolated exec — every exec-backed instance in its own namespace |
| + |
| +**Date:** 2026-09-11 · **Status:** draft, awaiting codex review round 1 · |
| +**Requested by:** discussion #126 (Tito, with box evidence), approved in |
| +direction by Paul on 2026-09-11 |
| + |
| +**Builds on:** discussion #126 and its companion #127; titogo's |
| +`deploy/systemd/titogo@.service` (the namespace recipe, verified on |
| +systemd 257 and on carlos-tito-ie); `internal/edge/purge.go:53-107` and |
| +`internal/activator/admintoken.go:6-14`, which already describe the gap; |
| +the exec-child accounting design (2026-08-07, pinfra |
| +`platform-design/superpowers/specs/`), the memory-pressure containment |
| +design (2026-08-28, same place), and the root-relay pattern |
| +(`carlos-stage.path` → `carlos adopt --stage-only`). |
| + |
| +--- |
| + |
| +## 0. How this spec was arrived at |
| + |
| +Tito filed #126 after verifying on carlos-tito-ie that an exec-backed |
| +instance can read every other instance's database, admin token and |
| +delivered config, including the relay route's platform credentials, and |
| +including instances of other accounts' apps. The discussion offered two |
| +shapes: make console-provisioned instances unit-backed, or isolate exec |
| +children in place with a per-host directory plus a uid or namespace. |
| + |
| +Three approaches were compared against the code on 2026-09-11 (§12). |
| +Paul approved the third on the same day and asked that the one open |
| +mechanism choice, how the unprivileged agent starts a root-managed unit, |
| +be settled between Fable and codex during review (§5.4). |
| + |
| +The shape in one sentence: **the activator keeps owning every exec-backed |
| +instance exactly as it does today, but the process it owns is launched as |
| +a systemd template unit, still uid `carlos`, inside a mount namespace in |
| +which nothing but that instance's own directory exists.** No uid change, |
| +no console change, no new backing, and the migration is a rename the |
| +agent performs itself. |
| + |
| +--- |
| + |
| +## 1. What exists, and what is wrong |
| + |
| +### 1.1 The agent and its children |
| + |
| +`carlos agent` runs as `User=carlos` under `carlos-edge.service` |
| +(`internal/host/units/carlos-edge.service`), with |
| +`CapabilityBoundingSet=CAP_NET_BIND_SERVICE`, `RestrictNamespaces=yes`, |
| +`ReadWritePaths=/data /run/carlos` and `Delegate=cpu memory io pids`. It |
| +cannot change a child's uid and cannot build a mount namespace for one. |
| + |
| +Every exec-backed instance is a plain child of that process: |
| +`internal/activator/backend_exec.go:690-692` builds |
| +`exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)` with |
| +`cmd.Env = childEnv(childVars)`; no `cmd.Dir`, no `SysProcAttr.Credential` |
| +(the only `SysProcAttr` fields set anywhere in the tree are the cgroup |
| +manager's `Setpgid`, `Pdeathsig`, `UseCgroupFD` and `CgroupFD`, |
| +`internal/cgroup/strict_linux.go:256`, `cgroup_linux.go:20-24`). Seed |
| +(`backend_exec.go:377-410`) and sidecars (`:885-911`) are spawned the same |
| +way. |
| + |
| +Everything the instance owns sits flat in one directory, `DataDir` |
| +(`/data`; `backend_exec.go:52-53`, `internal/host/agent.go:186-188`): |
| + |
| +| File | Written by | Read by | |
| +|---|---|---| |
| +| `<host>.db` (+ `-wal`, `-shm`, `.activator-<host>.db-litestream/`) | the instance; restore, seed and replication in the agent | the instance, the agent's in-process replicator (`internal/lease/lease.go:257-299`) | |
| +| `<host>.env` (admin token) | `EnsureAdminToken` (`admintoken.go:111-195`) | the agent (`backend_exec.go:507, 607, 790`; `internal/schedule/runner.go:461`) | |
| +| `<host>-sidecar.env` | hand-staged | `backend_exec.go:865-877` | |
| +| `<host>.config.env` | `internal/appconfig/syncexec.go:872-926` | `backend_exec.go:786` | |
| + |
| +All of it is `carlos:carlos`, and every instance is uid `carlos`. Mode |
| +0600 restricts nothing between siblings. `purge.go:53-95` says so in |
| +prose, and names per-instance uids as "the real fix". |
| + |
| +### 1.2 What a sibling can reach today |
| + |
| +Everything below is a read (or write) by any exec-backed instance of any |
| +app on the box, as verified by Tito on 2026-09-10 and as the file table |
| +above implies: |
| + |
| +- every other exec instance's database, read and write; |
| +- every other exec instance's admin token, which drives that host's |
| + schedule verbs (`admintoken.go:6-14`); |
| +- every `<host>.config.env` in `/data`, including the ones for TCP-addr |
| + routes CARLOS never spawns (`carlos env exec-delivery file`), which for |
| + Tito carry every platform credential the relay holds (#127); |
| +- every other exec instance's control socket, `rt.Socket + ".control"` |
| + (`backend_exec.go:594`; `purge.go:56-68`): a purge for another host, or |
| + a schedule tick with a token read from that host's env file; |
| +- every other exec instance's **listening socket file** under |
| + `/run/carlos`: same uid, so a sibling can unlink `<other>.sock` and bind |
| + its own listener there, and the edge (`internal/edge/edge.go:536-538`) |
| + will dial it as that host. This is not in #126; it follows from the |
| + same uid and the same directory; |
| +- `/data/registry.db`, `/data/litestream.env` (the box-wide replicator's |
| + bucket credentials, `carlos-litestream.service`) and, on a console box |
| + where `console.service` runs as `carlos` with `ReadWritePaths=/data`, |
| + `/data/console/console.db`. **Verify the console unit's user on the |
| + flagship before treating the last item as fact; the unit is |
| + hand-written and not in git.** |
| +- the instance metadata service. `harden-host.sh:36-44` (pinfra) allows |
| + IMDS to uid 0 and uid `carlos` and drops everyone else, because the |
| + agent reads IMDS. Exec children are uid `carlos`. `carlos-app@.service` |
| + carries `IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128` for exactly |
| + this reason; exec children carry nothing. |
| + |
| +### 1.3 What the unit backing already gets right, and why it is not the answer |
| + |
| +`carlos-app@.service` (`DynamicUser=yes`, `StateDirectory=carlos-app/%i`, |
| +root-0600 config in `/etc/carlos/apps`) has the property #126 asks for: a |
| +peer's state is absent, not permission-denied. But `UnitBackend` |
| +(`internal/activator/backend_unit.go`) has none of the machinery the |
| +exec backend gives a self-serve instance. Its `Acquire`, `Restore` and |
| +`Seed` are inert (`:113-132`), replication belongs to the root |
| +`carlos-litestream` unit because the agent cannot read a DynamicUser's |
| +state directory (`:12-19`), `NextDue` returns nothing (`:174-179`), and |
| +sidecars are refused (`:181-188`). `provision.Record` has no backing |
| +field (`internal/provision/provision.go:36-135`) and the reconciler |
| +refuses to adopt a row that carries one (`reconcile.go:426-436`). The |
| +always-awake design already concluded that moving exec instances to |
| +units "needs a separately reviewed root/unit/socket maintenance protocol |
| +and data migration". Making console-provisioned instances unit-backed is |
| +a rewrite of hibernation, not an isolation fix. |
| + |
| +One more thing the sweep found: `UnitBackend.systemctl` |
| +(`backend_unit.go:94-103`) runs `systemctl start` as the unprivileged |
| +agent. Nothing in this tree or in pinfra grants uid `carlos` the |
| +`org.freedesktop.systemd1.manage-units` action, and polkit is not |
| +referenced anywhere. §5 gives the agent a root channel for unit |
| +start/stop; wiring `UnitBackend` to it is a follow-up (§11). |
| + |
| +--- |
| + |
| +## 2. The design in one page |
| + |
| +1. **Per-instance directory.** An exec-backed instance's files move from |
| + `<DataDir>/<host>.*` to `<DataDir>/instances/<host>/` (§3). The agent |
| + creates the directory and everything in it, as `carlos`, exactly as it |
| + creates the flat files today. The registry's `DBPath` and `Socket` |
| + columns point into the new layout. |
| + |
| +2. **A template unit per role.** The instance runs as |
| + `carlos-instance@<host>.service`; its seed as |
| + `carlos-instance-seed@<host>.service`; its sidecar as |
| + `carlos-instance-sidecar@<host>.service` (§4). All three run as |
| + `User=carlos`, and all three carry titogo's namespace: |
| + `TemporaryFileSystem=/data:ro` and `TemporaryFileSystem=/run/carlos:ro` |
| + with exactly the instance's own two directories bound back in, |
| + `PrivatePIDs=yes`, `ProtectProc=invisible`, `RestrictNamespaces=yes`, |
| + `IPAddressDeny=` for IMDS. A sibling's database does not exist in this |
| + namespace; a sibling's socket does not exist; a sibling's pid does not |
| + exist. |
| + |
| +3. **The agent stays the owner.** Lease, restore, seed ordering, |
| + replication, hibernation, the control socket, `carlos logs`, the |
| + next-due tick, sidecar co-lifecycle and residence moves are unchanged |
| + in `internal/activator`. What changes is the `Proc` the exec backend |
| + hands back: it is a unit, not an `*exec.Cmd` (§6). |
| + |
| +4. **One root channel.** The agent cannot start a system unit. A small |
| + root service, `carlos-instance-control`, socket-activated on a unix |
| + socket only uid `carlos` can open, starts and stops units named |
| + `carlos-instance*@<host>` and nothing else (§5). This is the |
| + synchronous form of the `carlos-stage.path` pattern. §5.4 records the |
| + alternative (polkit) and the question put to codex. |
| + |
| +5. **Launch through a trampoline.** Each unit's `ExecStart` is |
| + `/opt/carlos/carlos instance-exec <role> <host>`, which runs inside the |
| + namespace, reads the launch description the agent wrote into the |
| + instance directory, and `execve`s the app binary with the exact argv and |
| + environment the agent composed (§6.2). Environment composition stays in |
| + Go, in one place, testable, with no dependence on systemd's |
| + `EnvironmentFile=` parser agreeing with the agent's. |
| + |
| +6. **Resource walls move to a slice.** Instances leave the edge unit's |
| + cgroup tree. `carlos-instances.slice` becomes the outer wall; the tier |
| + (`512M` / `1G`) becomes a runtime property the control service sets on |
| + the unit before starting it; the sampler gains the slice as a scan |
| + root (§6.6). |
| + |
| +7. **The migration is a rename.** On the first agent start on the new |
| + build every exec instance is asleep (they died with the agent). Before |
| + the activator starts, the agent moves each exec route's flat files into |
| + its directory and updates the two registry columns in one compare-and- |
| + swap. No root step, no chown, no operator action (§8). |
| + |
| +8. **Addr routes are untouched, and now invisible.** A TCP-addr route's |
| + `<host>.config.env` stays flat in `/data`, because the app's own |
| + supervisor reads it there (titogo's `EnvironmentFile=-/data/%i.config.env`). |
| + Exec children can no longer see it. That closes the read #127 leads |
| + with; #127's ask for root-owned delivery stays open as its own change |
| + (§11). |
| + |
| +--- |
| + |
| +## 3. The instance directory |
| + |
| +``` |
| +/data/instances/<host>/ 0700 carlos:carlos, created by the agent |
| + <host>.db the database (+ -wal, -shm) |
| + .activator-<host>.db-litestream/ the in-process replicator's shadow |
| + <host>.env admin token (EnsureAdminToken, unchanged) |
| + <host>-sidecar.env hand-staged sidecar env (unchanged) |
| + <host>.config.env syncexec's rendered bundle (unchanged) |
| + launch.run.json agent-written before each instance start (§6.2) |
| + launch.seed.json agent-written before a seed |
| + launch.sidecar.json agent-written before a sidecar start |
| + <host>.db.corrupt a quarantined restore, as today |
| +/run/carlos/instances/<host>/ 0700 carlos:carlos, created by the agent |
| + app.sock the instance's listener (rt.Socket) |
| + app.sock.control the agent's control listener (rt.Socket + ".control") |
| + log.sock the agent's log listener (§6.4) |
| +``` |
| + |
| +**One helper, one owner.** `internal/provision` gains |
| +`InstanceDir(dataDir, host)` beside `DefaultDBPath`, and |
| +`DefaultDBPath(dataDir, host)` and `DefaultSocket(host)` change to return |
| +the nested paths for exec routes: |
| + |
| +```go |
| +func InstanceDir(dataDir, host string) string { |
| + return filepath.Join(dataDir, "instances", host) |
| +} |
| +func DefaultDBPath(dataDir, host string) string { |
| + return filepath.Join(InstanceDir(dataDir, host), host+".db") |
| +} |
| +func InstanceRunDir(host string) string { |
| + return filepath.Join(SocketDir, "instances", host) |
| +} |
| +func DefaultSocket(host string) string { |
| + return filepath.Join(InstanceRunDir(host), "app.sock") |
| +} |
| +``` |
| + |
| +Every DataDir-keyed builder in the tree moves onto these (the list is |
| +§9). `ExecBackend.defaultDBPath` (`backend_exec.go:160-162`) is deleted in |
| +favour of the provision helper, so there is one definition, not two. |
| + |
| +**File names inside the directory keep the host prefix.** It costs |
| +nothing, it keeps every existing test fixture recognisable, and it means |
| +a file found outside its directory (a stray rename, a half-migration) is |
| +still attributable. The one thing the directory changes about names is |
| +that the `x.config.env`-versus-host-`x.config` ambiguity `reapConfigEnv` |
| +guards against (`syncexec.go:1009-1015`) can no longer arise for exec |
| +routes: there is exactly one host per directory. |
| + |
| +**Why nested and not per-instance uids.** titogo's unit comment says it |
| +best: "A flat `/data/titogo/%i.db` cannot be isolated — the bind would |
| +have to reveal all of `/data/titogo`, which is the regression itself." |
| +Per-instance uids (the discussion's option 2, titogo's "Option B") would |
| +additionally need a uid allocator, a chown as root on every migration and |
| +every restore, and a group arrangement that keeps the agent able to open |
| +the database for replication. The namespace gives the property the |
| +discussion asks for ("absent, not permission-denied") without any of |
| +that, and leaves option B open as a later hardening on the same layout |
| +(§11). |
| + |
| +**Custom `DBPath`.** `carlos add --db` may stamp any absolute path on an |
| +exec route (`cli.go:157-159` refuses only a relative one). Under this |
| +design an exec route's `DBPath` must be inside `InstanceDir`, because |
| +that is the only data path the namespace reveals. `carlos add` and |
| +`carlos route` refuse an exec route whose `--db` is elsewhere, naming the |
| +directory. Existing rows are handled by the migration (§8.2). |
| + |
| +--- |
| + |
| +## 4. The units |
| + |
| +Three template files under `internal/host/units/`, embedded and installed |
| +exactly as the existing nineteen are (`internal/host/bootstrap.go:33-34`, |
| +`systemapply.go:339-398`): `carlos-instance@.service`, |
| +`carlos-instance-seed@.service`, `carlos-instance-sidecar@.service`, plus |
| +`carlos-instances.slice`. Nothing enables the templates; an instance unit |
| +exists only while the agent has started it. |
| + |
| +### 4.1 `carlos-instance@.service` |
| + |
| +```ini |
| +[Unit] |
| +Description=carlos exec-backed instance %i |
| +# Stop and restart of the edge propagate here: an agent restart cycles |
| +# every instance, exactly as it did when instances were its children. |
| +PartOf=carlos-edge.service |
| +After=carlos-edge.service |
| + |
| +[Service] |
| +Type=simple |
| +ExecStart=/opt/carlos/carlos instance-exec run %i |
| +User=carlos |
| +Group=carlos |
| +Slice=carlos-instances.slice |
| +# The activator owns recycling: a crash stands the entry down and the |
| +# next request wakes it. systemd must not race that. |
| +Restart=no |
| +# SIGTERM to the app, SIGKILL to anything it left behind after |
| +# TimeoutStopSec; today's actStopTimeout is 20s. |
| +KillMode=mixed |
| +TimeoutStopSec=20 |
| +# The log stream: stdout and stderr connect to the agent's per-instance |
| +# listener (§6.4). systemd opens this as root, outside the namespace. |
| +StandardOutput=file:/run/carlos/instances/%i/log.sock |
| +StandardError=inherit |
| + |
| +# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) --- |
| +ProtectSystem=strict |
| +TemporaryFileSystem=/data:ro |
| +TemporaryFileSystem=/run/carlos:ro |
| +BindPaths=/data/instances/%i /run/carlos/instances/%i |
| +ReadWritePaths=/data/instances/%i /run/carlos/instances/%i |
| +ProtectHome=yes |
| +PrivateTmp=yes |
| +PrivateDevices=yes |
| +ProtectProc=invisible |
| +ProcSubset=pid |
| +PrivatePIDs=yes |
| +RestrictNamespaces=yes |
| +NoNewPrivileges=yes |
| +CapabilityBoundingSet= |
| +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX |
| +IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 |
| +ProtectKernelTunables=yes |
| +ProtectKernelModules=yes |
| +ProtectKernelLogs=yes |
| +ProtectClock=yes |
| +ProtectHostname=yes |
| +RemoveIPC=yes |
| +LockPersonality=yes |
| +RestrictRealtime=yes |
| +RestrictSUIDSGID=yes |
| +MemoryDenyWriteExecute=yes |
| +SystemCallArchitectures=native |
| + |
| +# --- resources (memory-pressure design 2026-08-28 §2.2-2.3, moved here) --- |
| +# Defaults; the control service overrides MemoryMax/MemoryHigh per start |
| +# with the route's tier (§6.6). |
| +MemoryMax=512M |
| +MemoryHigh=460M |
| +TasksMax=256 |
| +# The kernel picks a tenant before it picks anything else on the box. |
| +OOMScoreAdjust=500 |
| +# One OOM kill takes the whole family (memory.oom.group, as the exec |
| +# cgroup sets today). |
| +OOMPolicy=kill |
| +``` |
| + |
| +Lines that are load-bearing and why: |
| + |
| +- **`TemporaryFileSystem=/data:ro` + `BindPaths=/data/instances/%i`** is |
| + the isolation. The tmpfs is empty; the bind reveals one directory. A |
| + sibling's database is not permission-denied, it does not exist. The |
| + same for `/run/carlos`: a sibling's listening socket and control socket |
| + do not exist, which closes the socket-file impersonation in §1.2. |
| +- **The bind sources must exist before the unit starts**, or systemd |
| + fails at namespace setup before any `ExecStartPre=` runs (titogo |
| + verified this; it is why `titogo-instance-dir@.service` exists). Here |
| + the agent is the supervisor and creates both directories before it |
| + asks for a start (§6.1), so no helper unit is needed. |
| +- **`PrivatePIDs=yes` + `ProtectProc=invisible`.** Every instance is uid |
| + `carlos`, and `ProtectProc=invisible` does not hide same-uid peers |
| + (hidepid's test is `ptrace_may_access`, which same-uid passes). With its |
| + own pid namespace an instance's `/proc` lists only itself: no sibling |
| + pid to traverse into `/proc/<pid>/root/`, no sibling to signal, nothing |
| + for a same-uid ptrace to attach to. `PrivatePIDs=` needs systemd 257 |
| + (Debian 13, which `infra/modules/region-host/variables.tf:134` pins); |
| + §8.4 verifies it on every box before the roll. |
| +- **`RestrictNamespaces=yes`**: without it an unprivileged user namespace |
| + could bind-mount over a fresh mountpoint and re-reveal what the tmpfs |
| + hid. |
| +- **`IPAddressDeny=`** for IMDS. The nft rule in pinfra allows uid |
| + `carlos` because the agent needs IMDS; this line is what stops an |
| + instance, also uid `carlos`, from fetching the box's role credentials. |
| +- **`PartOf=carlos-edge.service`** keeps today's semantics: restarting |
| + the edge cycles every instance. Without it, an agent restart would leave |
| + units running that no `actEntry` knows about. §6.7 adds the startup |
| + sweep that covers the case where `PartOf=` did not fire (a crashed |
| + agent, `OOMPolicy=continue` keeping the edge up). |
| +- **`StandardOutput=file:` to an `AF_UNIX` socket** is documented |
| + systemd behaviour: a stream connection is made and used as stdout. It |
| + is opened by PID 1 outside the namespace, so the path is the real one. |
| + §6.4. |
| +- **`/opt/carlos`** stays readable through `ProtectSystem=strict`; the |
| + trampoline and the live symlink are all the instance needs from it. |
| + |
| +### 4.2 `carlos-instance-seed@.service` and `carlos-instance-sidecar@.service` |
| + |
| +Identical `[Service]` hardening and namespace blocks. Differences: |
| + |
| +| | seed | sidecar | |
| +|---|---|---| |
| +| `Type=` | `oneshot` | `simple` | |
| +| `ExecStart=` | `/opt/carlos/carlos instance-exec seed %i` | `/opt/carlos/carlos instance-exec sidecar %i` | |
| +| `StandardOutput=` | `file:/run/carlos/instances/%i/log.sock` | same | |
| +| `PartOf=` | `carlos-edge.service` | `carlos-edge.service` | |
| +| memory | shares the instance's tier via the slice; no per-unit override | same | |
| + |
| +A Go test over the embedded FS asserts that the three files' namespace |
| +and hardening lines are byte-identical (§10). Three files rather than one |
| +because seed and instance never run together but sidecar and instance |
| +do, and a template's `%i` is the host, not a role. |
| + |
| +### 4.3 `carlos-instances.slice` |
| + |
| +```ini |
| +[Unit] |
| +Description=carlos exec-backed instances (outer memory wall) |
| + |
| +[Slice] |
| +MemoryHigh=50% |
| +MemoryMax=60% |
| +``` |
| + |
| +And `carlos-edge.service` drops from `MemoryHigh=70% MemoryMax=80%` to |
| +`MemoryHigh=15% MemoryMax=20%`. The 2026-08-28 design set 80% for |
| +"agent plus every awake instance" so that 20% remained for the box's |
| +other units. Instances now sit in a sibling cgroup, so the wall splits: |
| +60% for the instances, 20% for the agent (about 110 MB resident; 20% of |
| +a t4g.micro is 200 MB), the same 20% outside both. **This split is a |
| +review target**, not a measurement: the agent's ceiling should be |
| +checked against its RSS on the flagship before the roll. `OOMPolicy=continue` |
| +stays on the edge unit (it no longer matters for tenants, and it costs |
| +nothing). The `oom_score_adj` write in `internal/cgroup` |
| +(`ExecOOMScoreAdj`) becomes `OOMScoreAdjust=500` in the template. |
| + |
| +--- |
| + |
| +## 5. The root channel: `carlos-instance-control` |
| + |
| +### 5.1 Why a channel at all |
| + |
| +The agent is uid `carlos`. `systemctl start` on a system unit needs |
| +either root or a polkit grant; read-only `systemctl show` / `is-active` / |
| +`list-units` do not. The existing root relays (`carlos-stage.path`, |
| +`carlos-restart.path`) are asynchronous sentinel files, fine for a |
| +minute-scale adoption pass and wrong for a wake that must answer a |
| +waiting HTTP request. |
| + |
| +### 5.2 Units |
| + |
| +```ini |
| +# carlos-instance-control.socket |
| +[Socket] |
| +ListenStream=/run/carlos/instance-control.sock |
| +SocketUser=carlos |
| +SocketGroup=carlos |
| +SocketMode=0600 |
| +Accept=no |
| + |
| +[Install] |
| +WantedBy=sockets.target |
| +``` |
| + |
| +```ini |
| +# carlos-instance-control.service |
| +[Unit] |
| +Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent) |
| +Requires=carlos-instance-control.socket |
| + |
| +[Service] |
| +ExecStart=/opt/carlos/carlos instance-control |
| +# Root deliberately: the whole job is systemctl start/stop, and the |
| +# caller is the unprivileged agent. Same rationale as carlos-stage.service. |
| +ProtectSystem=strict |
| +ProtectHome=yes |
| +PrivateTmp=yes |
| +PrivateDevices=yes |
| +ProtectKernelTunables=yes |
| +ProtectKernelModules=yes |
| +ProtectKernelLogs=yes |
| +RestrictAddressFamilies=AF_UNIX |
| +MemoryDenyWriteExecute=yes |
| +MemoryMax=64M |
| +``` |
| + |
| +The socket is owned by `carlos` with mode 0600 and the unit is socket- |
| +activated, so only the agent can reach it and it runs only while there is |
| +work. Instances cannot reach it: `/run/carlos` inside their namespace is |
| +an empty tmpfs with their own directory bound in; the control socket is |
| +not in it. |
| + |
| +### 5.3 Protocol |
| + |
| +HTTP/1.1 over the unix socket, the shape `edge.ListenControl` already |
| +uses for the per-instance control socket. Four verbs, no shell, no |
| +argument passes through to `systemctl` except a unit name the service |
| +built itself: |
| + |
| +| Verb | What it runs | Returns | |
| +|---|---|---| |
| +| `POST /start?host=H&role=run\|sidecar&memory_max=N&memory_high=N` | `systemctl set-property --runtime carlos-instance[-sidecar]@H.service MemoryMax=N MemoryHigh=N` then `systemctl start` (blocking: the start job completes) | 204, or 502 with systemctl's stderr | |
| +| `POST /seed?host=H` | `systemctl start carlos-instance-seed@H.service` (oneshot: blocks until exit), then `systemctl show -p Result,ExecMainStatus` | 200 with `{result, status}`; a failed seed is a 200 whose result says so, not a 502 | |
| +| `POST /stop?host=H&role=run\|sidecar` | `systemctl stop` (blocking) | 204 | |
| +| `POST /reset?host=H&role=…` | `systemctl reset-failed` | 204 | |
| + |
| +Validation, in order: `host` must satisfy `provision.ValidHostShape` |
| +(`provision.go:262`); `role` must be one of the three literals; the |
| +memory values must parse as `Limit` values the agent already knows how to |
| +produce (`internal/cgroup`, `TierDefault`/`TierChildren`). Anything else |
| +is a 400 with no systemctl call. The unit name is |
| +`"carlos-instance" + suffix + "@" + host + ".service"`, built from the |
| +validated pieces, never from the request string. |
| + |
| +What this grants the agent: start, stop and memory-tier control of units |
| +named `carlos-instance*@<host>`. That is exactly what it has today over |
| +its own children. It grants nothing else, and a host that does not exist |
| +is a unit whose bind source does not exist, which fails at namespace |
| +setup and starts nothing. |
| + |
| +`systemctl set-property --runtime` on a template instance that has not |
| +started yet writes a runtime drop-in under `/run/systemd/system.control/` |
| +that the following `start` honours; the control service runs the two in |
| +that order and the plan's first task verifies it on a box. |
| + |
| +### 5.4 The choice put to codex: this service, or polkit |
| + |
| +The other way to let uid `carlos` start these units is polkit: a rules |
| +file granting `org.freedesktop.systemd1.manage-units` to `carlos` when |
| +the unit name matches `carlos-instance*@*.service`, after which the |
| +agent's existing bare `systemctl` calls would work. |
| + |
| +Fable's advisement is the control service, for four reasons: |
| + |
| +1. **Polkit is not part of the platform.** Nothing in this tree or in |
| + pinfra references it, and no install step puts it on a box (whether |
| + the Debian 13 cloud image happens to carry `polkitd` is to be checked, |
| + not assumed). Relying on it means a daemon with a JavaScript rules |
| + engine on an image whose every other privileged path is a `carlos` |
| + verb in one binary. |
| +2. **A rules file is a second install path.** Units reach the box through |
| + `embed.FS` and `installUnits` (`systemapply.go:517-589`) and through |
| + `update-platform-binary.sh`'s `cp` of the staged units. A file under |
| + `/etc/polkit-1/rules.d` is neither; it would need its own install, |
| + its own drift check and its own rollback. |
| +3. **Start needs a tier and seed needs a wait.** `set-property` is a |
| + separate polkit action (`manage-unit-files` / `set-environment` |
| + family); the seed's oneshot result needs a `show` after the start. |
| + The control service carries all of that in one Go handler with one |
| + test file; polkit would carry it in a rules file plus the same Go. |
| +4. **It is the existing pattern, made synchronous.** `carlos-stage.service` |
| + and `carlos-restart.service` are root oneshots that run a `carlos` verb |
| + on the agent's request. Operators already know what those are for. |
| + |
| +The honest cost: a root-owned HTTP server whose input is a host name |
| +from the agent. The mitigations are the grammar check, the fixed verb |
| +set, the absence of any shell, the 0600 socket, and the fact that the |
| +agent is already the box's control plane. **Codex is asked, in the review |
| +brief, to advise for or against this choice with reasons, and to look for |
| +a way an instance could reach the control socket or make the agent send |
| +a request it did not intend.** |
| + |
| +--- |
| + |
| +## 6. The activator on units |
| + |
| +`internal/activator/activator.go` does not change. `doWake` |
| +(`activator.go:772`), `stop` (`:962-1047`), `Bounce` (`:946-956`), |
| +`sweepOnce` (`:1082`) and the `Proc`/`Rep` seams (`:194-207`) keep their |
| +contracts. The changes are inside `ExecBackend`. |
| + |
| +### 6.1 `Start` |
| + |
| +1. Ensure `InstanceDir` and `InstanceRunDir` exist, 0700. Clear any stale |
| + `app.sock`, `app.sock.control` and `log.sock` in the run dir (an |
| + earlier process is gone: the unit is not active, or we would not be |
| + here). |
| +2. `EnsureAdminToken(InstanceDir/<host>.env)` — unchanged. |
| +3. `instanceEnv` — unchanged, reading the two files from `InstanceDir`. |
| +4. Bind the control listener at `rt.Socket + ".control"` — unchanged |
| + (`edge.ListenControl`), now inside the run dir. |
| +5. Bind the log listener at `InstanceRunDir/log.sock` (§6.4). |
| +6. Write `launch.run.json` (§6.2). |
| +7. `POST /start` on the control socket with the route's tier. |
| +8. `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` — unchanged. On |
| + failure, `POST /stop` and clean up, exactly as the current failure |
| + path stops the child. |
| +9. Return a `unitProc`. |
| + |
| +### 6.2 The trampoline and `launch.<role>.json` |
| + |
| +```json |
| +{ |
| + "schema": 1, |
| + "argv": ["/opt/carlos/live/<host>", "--socket", "/run/carlos/instances/<host>/app.sock", "--db", "/data/instances/<host>/<host>.db"], |
| + "env": ["PATH=...", "CARLOS_ADMIN_TOKEN=...", "CARLOS_CONTROL_SOCKET=...", "..."] |
| +} |
| +``` |
| + |
| +`carlos instance-exec <role> <host>` runs as `ExecStart`, inside the |
| +namespace, as `carlos`. It reads `/data/instances/<host>/launch.<role>.json`, |
| +refuses a schema it does not know, refuses an `argv[0]` that is not |
| +`/opt/carlos/live/<host>` (defence in depth: the file is agent-written and |
| +0600, but the check is one line), and calls `syscall.Exec(argv[0], argv, env)`. |
| +It never parses env syntax: the agent's `childEnv` (`backend_exec.go:802-814`) |
| +composes the list exactly as it does today, including the agent-owned |
| +removal of tenant-supplied `CARLOS_CONTROL_SOCKET` / `CARLOS_PURGE` |
| +(`backend_exec.go:636-660`), and the trampoline passes it through. |
| + |
| +Why not `EnvironmentFile=`: systemd's parser and the agent's |
| +`parseEnvBytes` differ on quoting and continuation, and the agent-owned |
| +variables would then have to be written into a file the tenant's |
| +`<host>.env` overlay must not win over. One composer, one format, one |
| +test. |
| + |
| +The launch file is written fresh before every start and holds the same |
| +material the child could already read from `/proc/self/environ`. It is |
| +inside the instance's own directory, visible to that instance and the |
| +agent, nobody else. |
| + |
| +Seed: `launch.seed.json` carries the argv from `rt.Seed` |
| +(`backend_exec.go:332-338`) with the DB path substituted as today; `POST /seed` |
| +blocks; the result's `status` is the seed's exit code; a non-zero exit |
| +runs the same cleanup as today (`backend_exec.go:438`). |
| + |
| +Sidecar: `launch.sidecar.json` carries `[live/<host>, "sidecar", "run"]` |
| +and the sidecar env from `<host>-sidecar.env` only, as today |
| +(`backend_exec.go:885-911`); `POST /start?role=sidecar`. |
| + |
| +### 6.3 `unitProc` |
| + |
| +```go |
| +type unitProc struct { |
| + host string |
| + role string |
| + ctl *edge.ControlListener // as childProc.ctl today |
| + logs *logListener // §6.4 |
| + exited atomic.Bool |
| + done chan struct{} |
| +} |
| +func (p *unitProc) Stop(ctx context.Context) error // POST /stop, then wait for done or ctx |
| +func (p *unitProc) Exited() bool // §6.5 |
| +``` |
| + |
| +The control listener's close ordering (`backend_exec.go:992-1009`: signal |
| +first, close the listener only after the process is gone) is preserved: |
| +`Stop` posts the stop, waits for `done`, then closes `ctl` and `logs`. |
| + |
| +### 6.4 Logs |
| + |
| +Today `teeStd` (`backend_exec.go:123-137`) pipes the child's stdout and |
| +stderr through the agent: straight to the agent's own stdout (journald) |
| +and, with `Logs` set, line-wise into `Logs.App` for `carlos logs`, capped |
| +at `execLogLineMax`, with `WaitDelay` bounding a grandchild that holds |
| +the pipe. |
| + |
| +The unit connects its stdout and stderr to `log.sock` |
| +(`StandardOutput=file:` on an `AF_UNIX` path). The agent's `logListener` |
| +accepts exactly one connection per start (a second is closed: the unit |
| +is the only thing that should connect, and it connects once), and feeds |
| +it to the same `applog.NewLineWriter` and the same `io.MultiWriter` to |
| +the agent's stdout. Same bytes, same cap, same sink, same `carlos logs`. |
| +The listener is bound before `POST /start` so PID 1's connect cannot |
| +race it; it lives in the run dir, so only this instance can reach it, |
| +and a connect from inside the namespace is harmless (it would only be |
| +writing its own log). |
| + |
| +Grandchildren: a process the app forked that outlives it keeps the |
| +socket open, as it keeps the pipe open today. `KillMode=mixed` plus |
| +`TimeoutStopSec=20` kills it on stop; on a crash the unit's cgroup empties |
| +when systemd reaps it, and EOF follows. The `WaitDelay` grace becomes the |
| +unit's stop timeout. |
| + |
| +### 6.5 Exit observation |
| + |
| +`childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`). |
| +`unitProc` has two signals and uses both: |
| + |
| +- **EOF on the log connection**, which fires when the last process |
| + holding the unit's stdout closes it: the fast path, and the one that |
| + matches today's "pipe closed" semantics. |
| +- **`systemctl show -p ActiveState,SubState,Result,ExecMainStatus,ExecMainCode <unit>`**, |
| + read-only and unprivileged, polled by `sweepOnce` for every awake entry |
| + (the sweeper already visits each entry every `Idle/4`; one `show` per |
| + visit is cheap and can be batched with `systemctl show` on several |
| + units in one call). |
| + |
| +Either signal marks the proc exited, emits the member-facing event line |
| +with the same wording as today (`instance exited: ...` with the exit |
| +status or signal from `ExecMainStatus`/`ExecMainCode`, or |
| +`instance exited (exit 0)`), and closes `done`. The activator's existing |
| +"an exited line with no stopping line before it is a crash" reading is |
| +unchanged. |
| + |
| +`Restart=no` in the template is what makes this sound: systemd never |
| +restarts the unit behind the activator's back; a failed unit stays |
| +`failed` until the next `start`, and `POST /start` runs `reset-failed` |
| +first so a failed state is never a reason a wake cannot proceed. |
| + |
| +### 6.6 cgroups, tiers and the sampler |
| + |
| +`internal/cgroup` today: `Manager.Start` (`cgroup.go:242-285`) clones the |
| +child into `carlos-exec@<host>` under the edge unit's delegated subtree, |
| +writes `memory.max`/`memory.high` from the tier and `memory.oom.group=1`, |
| +and the sampler reads the subtree as a scan root (exec-child accounting |
| +design §4). `CleanupHost` (`:398-403`) removes the directory. |
| + |
| +Under units, systemd owns the cgroup: |
| +`/sys/fs/cgroup/carlos-instances.slice/carlos-instance@<host>.service/` |
| +and `.../carlos-instance-sidecar@<host>.service/`. So: |
| + |
| +- **Tier** travels in `POST /start` and lands as `MemoryMax`/`MemoryHigh` |
| + via `set-property --runtime`. The `Tier func(registry.Route) cgroup.Limit` |
| + seam and its box wiring (`stagedManifestOfLiveVersion`) are unchanged; |
| + only the consumer changes. The agent logs `instance: <host>: memory.max=…` |
| + on every start as it does today. |
| +- **Sampler** gains the slice as a scan root: a host's charge is the sum |
| + of its `carlos-instance@` and `carlos-instance-sidecar@` units, which |
| + keeps "a sidecar's load is that instance's work". cgroup v2 files are |
| + world-readable, so the agent reads them as `carlos` exactly as it reads |
| + the delegated subtree now. |
| +- **`Manager.Start`, `CleanupHost`, the clone3 path and the delegation |
| + dance** are no longer used by the exec backend. They stay in the tree |
| + until the last exec spawn is gone (`Seed` and sidecars move in the same |
| + change, so nothing uses them after it); deleting the package's spawn |
| + half is a follow-up once a release has run without it. |
| +- **`Delegate=` on the edge unit** stays for one release for the same |
| + reason, then goes. |
| + |
| +### 6.7 Agent start and restart |
| + |
| +Today an agent restart kills every child (`Pdeathsig`, the cgroup tree |
| +dies with the unit). `PartOf=carlos-edge.service` reproduces that for a |
| +`systemctl restart carlos-edge`. For a crash, or a stop that did not |
| +propagate, the agent on start runs |
| +`systemctl list-units --plain --no-legend 'carlos-instance*@*'` |
| +(unprivileged) and `POST /stop`s every active one before the activator |
| +starts. Every instance is therefore asleep when the activator's state |
| +machine begins, which is the invariant it holds today. Adopting a running |
| +unit instead (re-acquiring its lease, re-attaching replication) is a |
| +later improvement with its own review; it is not needed for correctness. |
| + |
| +The migration (§8) runs in this same window, after the sweep and before |
| +the activator starts, for the same reason: nothing is running. |
| + |
| +### 6.8 What does not change |
| + |
| +`Acquire`, `Restore` (including the `.corrupt` quarantine and |
| +`PurgeLocalDBArtifacts`, all keyed off `dbPath`), `lease.Replicate` (the |
| +meta dir is derived from `dbPath`, `lease.go:286`, so it lands inside the |
| +instance directory), `NextDue`, the schedule runner's token read, the |
| +control socket's purge and schedule mounts, `Bounce`, the sweeper's |
| +hibernation policy, residence moves (`movespass.go`, once it reads the |
| +DB path from the row instead of rebuilding it, §9), `carlos restart` in |
| +all three scopes, `X-Carlos-Version`, the caps pass, config-ack. All of |
| +these see a path change and nothing else. |
| + |
| +--- |
| + |
| +## 7. Config delivery |
| + |
| +`SyncExec` (`internal/appconfig/syncexec.go`) writes an exec route's |
| +bundle to `InstanceDir/<host>.config.env` instead of |
| +`<DataDir>/<host>.config.env`. Its ownership rule ("sync owns |
| +`*.config.env` outright, never touches `<host>.env` or |
| +`<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged and now trivially |
| +true per directory. `reapConfigEnv` (`:977-1024`) walks |
| +`<DataDir>/instances/*/` for exec routes as well as the flat `DataDir` |
| +for addr routes, and the bounce-on-change path is unchanged (the agent |
| +bounces the entry; the next wake reads the new file). |
| + |
| +**Opted-in TCP-addr routes stay flat.** Their file is read by the app's |
| +own supervisor at `/data/<host>.config.env` (titogo's |
| +`EnvironmentFile=-/data/%i.config.env`), and moving it would break every |
| +such drop-in. Exec instances cannot see it any more, which is the read |
| +#127 opens with. #127's request, root-rendered delivery into |
| +`/etc/carlos/apps` for addr routes so that the file is not readable by |
| +the agent either, is a separate, smaller change and stays open (§11). |
| + |
| +`reapConfigEnv`'s host-`x.config` ambiguity guard stays for the flat |
| +directory, where it still applies. |
| + |
| +--- |
| + |
| +## 8. Migration and rollout |
| + |
| +### 8.1 What moves, and when |
| + |
| +On agent start, after the unit sweep (§6.7) and before the activator |
| +starts, for every route with `Kind` instance, `Backing == ""`, and a |
| +`Socket` (an exec-backed route), in one pass: |
| + |
| +1. If `rt.DBPath == filepath.Join(DataDir, host+".db")` and |
| + `rt.Socket == filepath.Join(SocketDir, host+".sock")` (the flat |
| + defaults), the row is **eligible**. |
| +2. `mkdir -p InstanceDir(host)` 0700. |
| +3. `os.Rename` each of `<host>.db`, `<host>.db-wal`, `<host>.db-shm`, |
| + `.activator-<host>.db-litestream`, `<host>.env`, `<host>-sidecar.env`, |
| + `<host>.config.env`, `<host>.db.corrupt` that exists, into the |
| + directory. Same filesystem, same owner, atomic per file, no root. |
| +4. `registry.SetInstancePaths(host, fromDB, fromSocket, toDB, toSocket)`: |
| + one transaction, compare-and-swap on both columns, a new sibling of |
| + `SetDBPath` (`registry.go:1057-1075`). |
| +5. Log one line per host: `migrate: <host>: flat → instances/`. |
| + |
| +Order matters: files first, then the row. If the agent dies between 3 |
| +and 4, the next start finds a row still naming the flat paths and files |
| +already in the directory; step 3 is idempotent (a rename whose source is |
| +gone is skipped when the destination exists), so it completes and the |
| +row follows. If it dies between 4 and 5, nothing is missing. If a rename |
| +fails midway (a full disk cannot fail a same-filesystem rename, but a |
| +permission or I/O error can), the pass logs it, leaves that host's row |
| +untouched, and moves on; that host **does not wake** until it is |
| +migrated (§8.2), because a flat spawn is the bug. |
| + |
| +The edge picks the new `Socket` up on its next registry poll |
| +(`edge.go:449-451`, ~2s); nothing is awake to be affected. |
| + |
| +### 8.2 Rows that are not eligible |
| + |
| +A row whose `DBPath` or `Socket` is not the flat default was set by hand |
| +(`carlos add --db`). The migration does not guess. It logs |
| +`migrate: <host>: db_path/socket not at the flat default; run carlos route --instance-dir <host>` |
| +and the route's wake refuses with the same sentence as its reason, so a |
| +member sees it in `carlos logs` and an operator sees it in the journal. |
| +`carlos route --instance-dir <host>` (a new flag on the existing verb) |
| +performs steps 2 to 5 for that one host from the paths the row actually |
| +names, as long as they are under `DataDir` and `SocketDir`. The pre-roll |
| +check (§8.4) lists these rows before any box runs the new build, so on a |
| +fleet whose rows are all at the default (which is every console- |
| +provisioned row, `reconcile.go:300-301`) this branch never fires. |
| + |
| +### 8.3 Rollback |
| + |
| +An older binary on a migrated box computes the flat `DefaultDBPath`, |
| +finds no file, and restores from the replica into the flat path: correct |
| +data as of the last checkpoint at hibernation, which is what a fresh box |
| +gets today. The nested directory is left in place and is re-adopted, not |
| +re-restored, if the new build returns (step 3's idempotence handles a |
| +flat file that reappeared: the nested copy wins and the flat one is |
| +renamed to `<host>.db.rollback-<ts>` inside the directory, never |
| +deleted). The registry rows name the nested paths, which the old |
| +reconciler logs as shape drift (`reconcile.go:873-881`) and does not |
| +revert. So a rollback costs one restore per instance and no data. |
| + |
| +### 8.4 Rollout order |
| + |
| +1. **Pre-roll check**, from a laptop, per fleet: a new read-only |
| + `carlos ops instances --check` against the box's registry copy (or, |
| + until that exists, an SSM `carlos routes` read) listing every exec |
| + route whose paths are not at the flat default. Fix them with |
| + `carlos route --db` before the roll, or accept that they will refuse |
| + to wake until `carlos route --instance-dir` runs. |
| +2. **systemd version.** `systemctl --version` on every box, over the |
| + existing update path; `PrivatePIDs=` needs 257. Debian 13 ships 257; |
| + a box that answers lower drops the directive from all three templates |
| + **by decision, not by improvisation**: the fleet then runs with |
| + `ProtectProc=invisible` alone and the same-uid `/proc/<pid>/root` |
| + traversal is back until per-instance uids close it (§11). titogo's |
| + unit comment made the same contingency decision in advance. |
| +3. **Dev fleet first** (`kass`, two boxes), the usual boxes-before-console |
| + order. After the roll: `systemctl list-units 'carlos-instance*'` shows |
| + units coming and going with requests; `ls /data/instances` matches |
| + `carlos routes`. The test that matters is from inside an instance, |
| + not from a root shell: the plan's verification task provisions a |
| + throwaway instance whose seed argv lists `/data` and `/run/carlos`, |
| + reads `/proc/1/status`, and curls IMDS, and reads the output back |
| + through `carlos logs`. A planted `/data/probe.txt` must be absent |
| + from the listing. |
| +4. **Flagship**, region by region, console box last. The AU box has failed |
| + every gate since 2026-09-07 for an unrelated reason; do not read its |
| + failure as this change's. |
| +5. **carlos-tito-ie**: Tito's operator rolls it from Tito's console once |
| + the flagship has run a day. The seven flat instances migrate on the |
| + first agent start. Reply on #126 with the build and the check list |
| + above. |
| + |
| +--- |
| + |
| +## 9. Every consumer of the shape |
| + |
| +The rule (AGENTS.md): the consumer list is everything that parses the |
| +shape, not everything that calls the API. From the sweep of 2026-09-11: |
| + |
| +**Path builders that move onto `InstanceDir` / the new defaults** |
| + |
| +- `internal/provision/provision.go:1011-1012` — `DefaultSocket`, `DefaultDBPath` (the definitions) |
| +- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`) |
| +- `internal/appconfig/syncexec.go:465, 872` (exec routes only; addr routes stay flat); `:977-1024` `reapConfigEnv` walks both |
| +- `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path) |
| +- `internal/host/movespass.go:285` — rebuilds `<DataDir>/<host>.db` from the record; must read `rt.DBPath` from the row instead |
| +- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir") |
| +- `internal/provision/reconcile.go:300-301` (`routeFor` mint) — follows the helpers unchanged; `:461-464` and `:873-881` compare derived paths and follow too |
| + |
| +**Readers of `rt.Socket` (no change; the value changes)** |
| + |
| +- `internal/edge/edge.go:536-538` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| + |
| +**Litestream and replication** |
| + |
| +- `internal/lease/lease.go:286` — meta dir derived from `dbPath`; follows |
| +- `internal/ops/litestream.go:354` — the box-wide generator's flat `*.db` glob does not see nested files, **but** `FleetDBsWithSkipped` (`:323-428`) claims every route's `DBPath` from the registry, which is nested after migration. The glob is the proposer, the registry disposes. Add `instances/*/*.db` to the glob anyway so an unrouted nested file is reported under `skipped` rather than silently ignored, and pin it in `ops_test.go` |
| +- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine |
| + |
| +**Units and installers** |
| + |
| +- `internal/host/units/` gains four files (§4) and two (§5.2); `carlos-edge.service` changes its memory lines |
| +- `internal/host/bootstrap.go:116-122` `enabledUnits` gains `carlos-instance-control.socket`; `systemctlPlan` (`:165-180`) already starts sockets it enables |
| +- `update-platform-binary.sh:52-70` (pinfra) enables units **by name**: add `carlos-instance-control.socket` there, or the roll leaves the channel unarmed and every wake fails with "connection refused" on the control socket. `ensureSystemUnits` (`internal/host/systemunits.go:34-83`) is the belt for that: it gains the socket too |
| +- `harden-host.sh` (pinfra) writes hardening drop-ins for `$TENANT_UNITS`; the instance templates carry their own and must not be listed there (a drop-in that adds `ProtectProc=invisible` is harmless; one that adds `PrivateNetwork` or narrows `RestrictAddressFamilies` is not) |
| + |
| +**Tests that pin the flat layout** (§10): `internal/activator/activator_test.go` (20 fixture sites), `backend_exec_test.go` (4), `admintoken_test.go` (8, path-agnostic), `internal/appconfig/syncexec_test.go` (50), `internal/ops/ops_test.go` (52), `internal/provision/reconcile_test.go` (20), `internal/host/movespass_test.go` (1), `schedule_wiring_test.go` (1), `schedule_token_fence_test.go` (2), root `e2e_test.go` (24), `instances_e2e_test.go`, `config_e2e_test.go`, `deploy_e2e_test.go`. |
| + |
| +**Not consumers**: the console never reads box paths (`internal/console` |
| +has no `DBPath`/`DataDir` reader); `readDiskFrac` is whole-volume; the |
| +sentinel `.path` units name fixed files under `/data`, never a host. |
| + |
| +--- |
| + |
| +## 10. Tests |
| + |
| +Unit level, all runnable in CI (no systemd needed): |
| + |
| +- `internal/provision`: `InstanceDir`, new defaults, `carlos add` refusal |
| + of an exec `--db` outside the directory. |
| +- `internal/activator`: `Start` writes `launch.run.json` with the exact |
| + argv and env `childEnv` produces today (the existing env-composition |
| + tests move their assertion from `cmd.Env` to the file); the control |
| + client sends `start` with the tier; `unitProc.Stop` orders stop → done |
| + → close listeners; `Exited()` from log EOF and from a `show` answer; |
| + the failed-health path posts `stop`. The control channel is a seam |
| + (`Control interface { Start; Seed; Stop; Reset }`) with a fake, as |
| + `Systemctl` is today. |
| +- `instance-exec`: reads the file, refuses schema ≠ 1, refuses an |
| + `argv[0]` that is not the live link, execs (tested with a stub argv). |
| +- `instance-control`: every 400 (bad host, bad role, bad limit), the unit |
| + name it builds, the `set-property` before `start` ordering, the seed |
| + result mapping, with a fake `systemctl` runner. |
| +- `logListener`: one connection accepted, second refused, line cap, |
| + EOF → exited. |
| +- Migration: eligible row moves eight files and CAS-updates two columns; |
| + ineligible row is logged and left; idempotent re-run after a simulated |
| + crash between files and row; rollback re-adoption renames the flat |
| + reappearance. |
| +- Units: a test over `unitsFS` asserts the three templates' namespace and |
| + hardening blocks are identical, that `PrivatePIDs=yes`, |
| + `TemporaryFileSystem=/data:ro`, `IPAddressDeny=169.254.169.254/32` and |
| + `Restart=no` are present in each, and that the control socket is |
| + `SocketMode=0600` `SocketUser=carlos`. |
| +- `internal/ops`: the glob change and the skipped report. |
| +- `internal/cgroup`: the sampler's slice scan root. |
| + |
| +Box level, in the plan's verification task and in the pre-roll check: |
| + |
| +- On a dev box: wake an instance, `systemctl status carlos-instance@<host>` |
| + active; `nsenter`-free check from inside via a throwaway instance whose |
| + seed lists `/data` and `/run/carlos`; `cat /proc/1/status` from inside |
| + shows the app as pid 1 (PrivatePIDs); IMDS curl from inside fails; |
| + `carlos logs` shows all of it; hibernate, unit gone; `carlos restart` |
| + cycles it; `systemctl restart carlos-edge` stops it (`PartOf=`). |
| + |
| +--- |
| + |
| +## 11. Out of scope, and follow-ups |
| + |
| +- **Per-instance uids (option B).** The nested layout is its |
| + prerequisite; the namespace closes every path the discussion names |
| + without it. If a box cannot run `PrivatePIDs=`, or a later finding |
| + needs a uid boundary, it lands on this layout with a root chown in the |
| + control service. Not in this change. |
| +- **Unit-backed console instances.** Not needed for isolation, and a |
| + hibernation rewrite. `UnitBackend` gets a working `start` for free from |
| + §5 (it can call the control service instead of bare `systemctl`); that |
| + wiring, and whether `backing=unit` hibernation works on the fleet today, |
| + is a follow-up discussion. |
| +- **#127**: root-rendered config for opted-in addr routes into |
| + `/etc/carlos/apps`. Smaller and independent; this change makes the file |
| + invisible to instances but not to the agent. |
| +- **Adopting running units on agent start** instead of stopping them |
| + (§6.7). |
| +- **Deleting the spawn half of `internal/cgroup`** and `Delegate=` from |
| + the edge unit, one release after this lands. |
| +- **Discussion #1 (memory pressure)**: the slice split in §4.3 should be |
| + measured on the flagship after the roll, and the numbers adjusted in the |
| + unit files, never by drop-in. |
| + |
| +--- |
| + |
| +## 12. Approaches considered |
| + |
| +1. **Console-provisioned instances become unit-backed** (#126 option 1). |
| + Rejected: `UnitBackend` has no restore, seed, replication, hibernate- |
| + from-replica, sidecar or move; `provision.Record` cannot express a |
| + backing and the reconciler refuses one. A rewrite of self-serve |
| + hibernation, and the always-awake design already said so. |
| +2. **Per-instance uid from the agent** (#126 option 2 as written). |
| + Impossible without `CAP_SETUID` on the agent, which is root-equivalent |
| + for a compromised agent; and a uid alone does not hide a sibling's |
| + socket file or pid. |
| +3. **Isolated exec: the agent's instance, systemd's namespace** (this |
| + spec). Keeps every activator invariant, needs one root channel, and |
| + migrates with a rename. |
| + |
| +Within 3, two sub-choices were made and one is put to codex: trampoline |
| +over `EnvironmentFile=` (§6.2, one env composer), three templates over |
| +one (§4.2, `%i` is the host), and the control service over polkit |
| +(§5.4). |
| + |
| +--- |
| + |
| +## 13. Review log |
| + |
| +- Round 1: pending (`CODEX-REVIEW-BRIEF-ISOLATED-EXEC.md`). |
amadan