carlos / platform Public

Download this file

1	# Isolated exec — every exec-backed instance in its own namespace
2
3	**Date:** 2026-09-11 · **Status:** revision 17; codex verdict after
4	round 11 is **accept with corrections**, and the one correction is
5	applied here (54 findings over 11 rounds, all accepted; §13); a
6	whole-branch final review (plan 2) found six further items, fixed in
7	revision 13 (§13); plan 3's task 4 review settled the requested-stop
8	close, recorded in revision 14 (§6.5, §13 R13); the first roll's probe
9	found two defects, fixed in revision 15 (§4.1, §8.5, §13 R15); awaiting Paul's review · **Requested by:** discussion
10	#126 (Tito, with box evidence), approved in direction by Paul on
11	2026-09-11
12
13	**Builds on:** discussion #126 and its companion #127; titogo's
14	`deploy/systemd/titogo@.service` (the namespace recipe, verified on
15	systemd 257 and on carlos-tito-ie); `internal/edge/purge.go:53-107` and
16	`internal/activator/admintoken.go:6-14`, which already describe the gap;
17	the exec-child accounting design (2026-08-07, pinfra
18	`platform-design/superpowers/specs/`), the memory-pressure containment
19	design (2026-08-28, same place), the cross-user socket consumers group
20	(issue #103, `internal/host/bootstrap.go:41-62`), the system-update
21	design (2026-09-06) whose apply/rollback path the transition must
22	survive, and the root-relay pattern (`carlos-stage.path` →
23	`carlos adopt --stage-only`).
24
25	---
26
27	## 0. How this spec was arrived at
28
29	Tito filed #126 after verifying on carlos-tito-ie that an exec-backed
30	instance can read every other instance's database, admin token and
31	delivered config, including the relay route's platform credentials, and
32	including instances of other accounts' apps. The discussion offered two
33	shapes: make console-provisioned instances unit-backed, or isolate exec
34	children in place with a per-host directory plus a uid or namespace.
35
36	Three approaches were compared against the code on 2026-09-11 (§12).
37	Paul approved the third on the same day and asked that the one open
38	mechanism choice, how the unprivileged agent starts a root-managed unit,
39	be settled between Fable and codex during review. Codex's round 1
40	concurred with the control service and showed why it must authenticate
41	its caller (§5.4). Round 2 turned the migration from an agent-start side
42	effect into an operator-triggered fleet verb (§8), because the system
43	updater's own rollback runs the *old* binary and cannot reverse a layout
44	it never knew. Round 3 drew the transition boundary: while a box still
45	runs unisolated tenants, nothing that runs as `carlos` is a trustworthy
46	caller, so the root channel authenticates the agent *process*, not a
47	secret the agent holds (§5.4).
48
49	The shape in one sentence: **the activator keeps owning every exec-backed
50	instance exactly as it does today, but the process it owns is launched as
51	a systemd template unit, still uid `carlos`, inside a mount namespace in
52	which nothing but that instance's own directory exists, on a mount where
53	no symlink is ever followed.** No uid change, no console change, no new
54	backing. One release carries both layouts; an operator verb moves each
55	fleet across; the same verb moves it back.
56
57	---
58
59	## 1. What exists, and what is wrong
60
61	### 1.1 The agent and its children
62
63	`carlos agent` runs as `User=carlos` under `carlos-edge.service`
64	(`internal/host/units/carlos-edge.service`), with
65	`CapabilityBoundingSet=CAP_NET_BIND_SERVICE`, `RestrictNamespaces=yes`,
66	`ReadWritePaths=/data /run/carlos` and `Delegate=cpu memory io pids`. It
67	cannot change a child's uid and cannot build a mount namespace for one.
68
69	Every exec-backed instance is a plain child of that process:
70	`internal/activator/backend_exec.go:690-692` builds
71	`exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)` with
72	`cmd.Env = childEnv(childVars)`; no `cmd.Dir`, no `SysProcAttr.Credential`
73	(the only `SysProcAttr` fields set anywhere in the tree are the cgroup
74	manager's `Setpgid`, `Pdeathsig`, `UseCgroupFD` and `CgroupFD`,
75	`internal/cgroup/strict_linux.go:256`, `cgroup_linux.go:20-24`). Seed
76	(`backend_exec.go:377-410`) and sidecars (`:885-911`) are spawned the same
77	way.
78
79	Everything the instance owns sits flat in one directory, `DataDir`
80	(`/data`; `backend_exec.go:52-53`, `internal/host/agent.go:186-188`):
81
82	| File | Written by | Read by |
83	|---|---|---|
84	| `<host>.db` (+ `-wal`, `-shm`, `.activator-<host>.db-litestream/`) | the instance; restore, seed and replication in the agent | the instance, the agent's in-process replicator (`internal/lease/lease.go:257-299`) |
85	| `<host>.env` (admin token) | `EnsureAdminToken` (`admintoken.go:111-195`) | the agent (`backend_exec.go:350, 507, 579, 607, 790`; `internal/schedule/runner.go:461`) |
86	| `<host>-sidecar.env` | hand-staged | `backend_exec.go:865-877` |
87	| `<host>.config.env` | `internal/appconfig/syncexec.go:872-926` | `backend_exec.go:786` |
88
89	All of it is `carlos:carlos`, and every instance is uid `carlos`. Mode
90	0600 restricts nothing between siblings. `purge.go:53-95` says so in
91	prose, and names per-instance uids as "the real fix".
92
93	The sockets are shared on purpose with one more party: `/run/carlos` is
94	`2750 carlos:carlos-sockets` (`bootstrap.go:61`), and after the health
95	wait the activator normalises each instance socket to `0660`
96	(`backend_exec.go:744-750`), so a deployment's enrolled cross-user
97	consumer (titogo's router, uid `titogo`, issue #103) can dial every
98	instance socket on the box. That contract survives this change (§3.1).
99
100	### 1.2 What a sibling can reach today
101
102	Everything below is a read (or write) by any exec-backed instance of any
103	app on the box, as verified by Tito on 2026-09-10 and as the file table
104	above implies:
105
106	- every other exec instance's database, read and write;
107	- every other exec instance's admin token, which drives that host's
108	schedule verbs (`admintoken.go:6-14`);
109	- every `<host>.config.env` in `/data`, including the ones for TCP-addr
110	routes CARLOS never spawns (`carlos env exec-delivery file`), which for
111	Tito carry every platform credential the relay holds (#127);
112	- every other exec instance's control socket, `rt.Socket + ".control"`
113	(`backend_exec.go:594`; `purge.go:56-68`): a purge for another host, or
114	a schedule tick with a token read from that host's env file;
115	- every other exec instance's **listening socket file** under
116	`/run/carlos`: same uid, so a sibling can unlink `<other>.sock` and bind
117	its own listener there, and the edge (`internal/edge/edge.go:536-538`)
118	will dial it as that host. This is not in #126; it follows from the
119	same uid and the same directory;
120	- `/data/registry.db`, `/data/litestream.env` (the box-wide replicator's
121	bucket credentials, `carlos-litestream.service`) and, on a console box
122	where `console.service` runs as `carlos` with `ReadWritePaths=/data`,
123	`/data/console/console.db`. **Verify the console unit's user on the
124	flagship before treating the last item as fact; the unit is
125	hand-written and not in git.**
126	- the instance metadata service. `harden-host.sh:36-44` (pinfra) allows
127	IMDS to uid 0 and uid `carlos` and drops everyone else, because the
128	agent reads IMDS. Exec children are uid `carlos`. `carlos-app@.service`
129	carries `IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128` for exactly
130	this reason; exec children carry nothing.
131
132	### 1.3 The primitive under all of it: same uid, same paths
133
134	A namespace hides a sibling's files from the instance. It does not hide
135	them from the **agent**, which reads and writes the instance's directory
136	from outside, as the same uid, by path. Any path the instance can write
137	and the agent later opens is therefore a request the instance makes on
138	the agent's behalf: replace `<host>.env` with a symlink to
139	`/data/litestream.env` and the next wake hands the bucket credentials to
140	the instance as its own environment (`backend_exec.go:785-794, 966-971`;
141	`EnsureAdminToken` only refuses the *chmod* through a symlink,
142	`admintoken.go:141-150`). Replace `<host>.db` with a symlink to a
143	sibling's database and the agent's replicator uploads the sibling's data
144	under the attacker's replica. Replace the listening socket with a symlink
145	to any other socket on the box and the edge dials it as this host
146	(codex R1-1, R1-3). Any *file* the agent reads as instructions from the
147	instance's directory is the same primitive without a symlink (R2-1).
148	And for as long as a box still runs one unisolated tenant, every
149	`carlos`-readable secret and every `carlos`-writable directory on the
150	box is that tenant's too, so nothing that runs as `carlos` can be told
151	apart from it by what it knows or where it writes (R3-1). A design that
152	bounds the instance but not the paths the agent follows on its behalf,
153	the records it trusts, or the callers it believes, has not bounded
154	anything. §3.2 closes the symlinks at the kernel; §3.3 states the rule
155	for records; §5.4 states the rule for callers.
156
157	### 1.4 What the unit backing already gets right, and why it is not the answer
158
159	`carlos-app@.service` (`DynamicUser=yes`, `StateDirectory=carlos-app/%i`,
160	root-0600 config in `/etc/carlos/apps`) has the property #126 asks for: a
161	peer's state is absent, not permission-denied. But `UnitBackend`
162	(`internal/activator/backend_unit.go`) has none of the machinery the
163	exec backend gives a self-serve instance. Its `Acquire`, `Restore` and
164	`Seed` are inert (`:113-132`), replication belongs to the root
165	`carlos-litestream` unit because the agent cannot read a DynamicUser's
166	state directory (`:12-19`), `NextDue` returns nothing (`:174-179`), and
167	sidecars are refused (`:181-188`). `provision.Record` has no backing
168	field (`internal/provision/provision.go:36-135`) and the reconciler
169	refuses to adopt a row that carries one (`reconcile.go:426-436`). The
170	always-awake design already concluded that moving exec instances to
171	units "needs a separately reviewed root/unit/socket maintenance protocol
172	and data migration". Making console-provisioned instances unit-backed is
173	a rewrite of hibernation, not an isolation fix.
174
175	One more thing the sweep found: `UnitBackend.systemctl`
176	(`backend_unit.go:94-103`) runs `systemctl start` as the unprivileged
177	agent. Nothing in this tree or in pinfra grants uid `carlos` the
178	`org.freedesktop.systemd1.manage-units` action, and polkit is not
179	referenced anywhere. Whether `backing=unit` hibernation works on the
180	fleet today is a follow-up discussion (§11); this spec's channel is
181	scoped to the three instance templates and does not extend to
182	`carlos-app@` (codex R1-19).
183
184	---
185
186	## 2. The design in one page
187
188	1. **Per-instance directory.** An exec-backed instance's files move from
189	`<DataDir>/<host>.*` to `<DataDir>/instances/<host>/` (§3). The agent
190	creates the directory and everything in it, as `carlos`, exactly as it
191	creates the flat files today. The registry's `DBPath` and `Socket`
192	columns point into the new layout once a host is migrated.
193
194	2. **No symlink is ever followed under the trees the agent binds or
195	opens for a tenant.** `/data/instances`, `/run/carlos/instances` and
196	`/run/carlos/logs` are bind mounts of themselves with `nosymfollow`
197	(§3.2). The kernel refuses to traverse a symlink there for every
198	process on the box, agent, edge and PID 1 included, with no
199	check-then-use race. Nothing the agent does is ever directed by a file
200	it read from an instance directory, and a migration journal is honoured
201	only when it says what the fleet's recorded intent already says (§3.3).
202
203	3. **A template unit per role.** The instance runs as
204	`carlos-instance@<host>.service`; its seed as
205	`carlos-instance-seed@<host>.service`; its sidecar as
206	`carlos-instance-sidecar@<host>.service` (§4). All three run as
207	`User=carlos`, all three carry titogo's namespace
208	(`TemporaryFileSystem=/data:ro` and `TemporaryFileSystem=/run/carlos:ro`
209	with exactly the instance's own two directories bound back in,
210	`PrivatePIDs=yes`, `ProtectProc=invisible`, `RestrictNamespaces=yes`,
211	`IPAddressDeny=` for IMDS), and all three sit in one per-host slice
212	that carries the host's memory and swap tier. A sibling's database does
213	not exist in this namespace; a sibling's socket does not exist; a
214	sibling's pid does not exist.
215
216	4. **The agent stays the owner.** Lease, restore, seed ordering,
217	replication, hibernation, the control socket, `carlos logs`, the
218	next-due tick, sidecar co-lifecycle and residence moves are unchanged
219	in `internal/activator`, with two bounded changes in `activator.go`
220	(§6.3, §6.7). What changes is the `Proc` the exec backend hands back:
221	it is a unit, not an `*exec.Cmd` (§6).
222
223	5. **One root channel, authenticating the process.** The agent cannot
224	start a system unit. A small root service, `carlos-instance-control`,
225	socket-activated on a unix socket in a root-owned directory, starts and
226	stops units named `carlos-instance*@<host>` and nothing else, and
227	accepts a request only from the edge unit's main process, as reported
228	by systemd, never from a child of it (§5). This is the synchronous
229	form of the `carlos-stage.path` pattern.
230
231	6. **Launch through a trampoline; exit through the unit's own record
232	plus the kernel's word.** Each unit's `ExecStart` is
233	`/opt/carlos/carlos instance-exec <role> <host>`, which runs inside the
234	namespace, reads the launch description the agent wrote, and `execve`s
235	the app binary with the exact argv and environment the agent composed
236	(§6.2). Each unit's `ExecStopPost=` leaves an advisory exit record in
237	the instance's own run directory; termination itself is proven by
238	systemd's unit state and an empty cgroup, never by a file (§6.5).
239
240	7. **Resource walls become slices.** `carlos.slice` carries the
241	2026-08-28 outer wall; the edge unit and every instance sit under it;
242	each host has its own slice inside that, carrying the tier (§4.3,
243	§6.6).
244
245	8. **One release, two layouts, one operator verb.** This release's
246	agent runs a *flat* host exactly as today (unisolated, `childProc`)
247	and a *nested* host isolated. Nothing moves on upgrade and no new row
248	is minted nested, so the system updater's own rollback (which runs the
249	previous binary) always lands on a layout that binary understands.
250	`carlos system layout migrate` records the fleet's intent; every box
251	converges its eligible hosts to it on every serial move; `revert`
252	records the opposite; status reports each host (§8). The next release
253	drops the flat spawn path for activator-managed hosts.
254
255	9. **Addr routes and externally supervised routes are untouched, and now
256	invisible.** A TCP-addr route's `<host>.config.env` stays flat in
257	`/data`, because the app's own supervisor reads it there (titogo's
258	`EnvironmentFile=-/data/%i.config.env`); an unflagged socket route keeps
259	its flat paths because its supervisor owns them. Isolated instances can
260	no longer see either. That closes the read #127 leads with; #127's ask
261	for root-owned delivery stays open as its own change (§11).
262
263	---
264
265	## 3. The instance directory
266
267	### 3.1 Layout
268
269	```
270	/data/.instances/ the RAW directory: 0700 carlos:carlos; opened by the layout pass only (§3.2, §8.2)
271	/data/instances/ nosymfollow bind mount OF /data/.instances (§3.2); the view everything else uses
272	/data/instances/<iid>/ 0700 carlos:carlos, created by the agent
273	<host>.db the database (+ -wal, -shm)
274	.activator-<host>.db-litestream/ the in-process replicator's shadow
275	<host>.env admin token (EnsureAdminToken, unchanged)
276	<host>-sidecar.env hand-staged sidecar env (unchanged)
277	<host>.config.env syncexec's rendered bundle (unchanged)
278	launch.run.json agent-written before each instance start (§6.2)
279	launch.seed.json agent-written before a seed
280	launch.sidecar.json agent-written before a sidecar start
281	<host>.db.corrupt a quarantined restore, as today
282	/data/layout/ 0700 carlos:carlos
283	<host>.json the migration journal for a host in flight (§8.2)
284	/run/carlos/instances/ nosymfollow bind mount OF /run/carlos/.instances; 2750 carlos:carlos-sockets
285	/run/carlos/instances/<iid>/ 2750 carlos:carlos-sockets, created by the agent
286	app.sock the instance's listener (rt.Socket); 0660 after health, as today
287	app.sock.control the agent's control listener (rt.Socket + ".control"); 0600
288	exit.<role>.<invocation>.json the unit's advisory exit record (§6.5); tenant-writable, and treated so
289	/run/carlos/logs/ nosymfollow bind mount OF /run/carlos/.logs; 0700 carlos:carlos; NOT bound into any namespace
290	<iid>.run.sock the agent's log listener for the instance (§6.4)
291	<iid>.seed.sock ... for the seed
292	<iid>.sidecar.sock ... for the sidecar
293	/run/carlos-control/ 0750 root:carlos; NOT under /run/carlos, NOT in any namespace
294	control.sock the root channel (§5); SocketUser=carlos, 0600
295	```
296
297	**`<iid>` is the instance identifier, bounded to 40 bytes** (codex
298	R5-3). A host may be 253 bytes (`controlname.ValidHostShape`), and a
299	unix socket address holds 107 bytes of path, so a directory named after
300	the host cannot carry `app.sock.control` for every valid host.
301	`provision.InstanceID(host)` is the host itself when `len(host) <= 40`,
302	and otherwise `_` + the first 22 bytes of the host + `-` + the first 16
303	hex digits of `sha256(host)`, which is also 40 bytes. **The two domains
304	cannot meet** (codex R6-1): no valid host contains `_`
305	(`controlname` labels are `[a-z0-9-]`), so a hashed identifier is never
306	equal to any short host, and two hashed identifiers are equal only on a
307	64-bit hash collision between two long hosts sharing a 22-byte prefix.
308	It is the directory name under `/data/instances` and
309	`/run/carlos/instances`, the log socket prefix, and, escaped, the unit
310	instance and slice suffix (§4; `_` needs no escaping). File names
311	*inside* the directory keep the full host (a file name may be 255
312	bytes). The longest address this produces,
313	`/run/carlos/instances/<40>/app.sock.control`, is 79 bytes.
314
315	Two rows on one box whose hosts nonetheless collide in `InstanceID`
316	classify as **`conflict`**, which **refuses to wake** with both hosts
317	named and is never the unisolated fallback: a host that is already
318	`nested` must never be moved to the `childProc` path by the arrival of
319	a second row (R6-1). The layout pass skips a `conflict` host, status
320	reports it, and the console's create-instance door refuses a new host
321	whose identifier collides with an existing row on the target box. The
322	pass also preflights every socket address it is about to commit
323	against 107 bytes before it moves a file.
324
325	The socket tree keeps the #103 contract: `/run/carlos/instances` and
326	each host's run directory are `2750 carlos:carlos-sockets` (setgid, so
327	the socket an instance binds inherits the group, exactly as
328	`tmpfilesConf` arranges for `/run/carlos` today), the activator's
329	post-health `0660` normalisation is unchanged, and the control socket
330	stays `0600`. An enrolled consumer can traverse to and dial `app.sock`
331	and nothing else (codex R2-10). Inside the namespace the instance owns
332	its run directory and can loosen it; that changes nothing outside,
333	because outside is where the mode is set and checked.
334
335	**One helper, one owner.** `internal/provision` gains
336	`InstanceDir(dataDir, host)` beside `DefaultDBPath`, and the nested
337	defaults exist alongside the flat ones. In this release **no minting
338	path uses the nested defaults** (§8.1, codex R3-6): `routeFor`
339	(`reconcile.go:299-301`) and `carlos add` (`cli.go:118-136`) keep minting
340	flat paths, and the layout pass is the only thing that ever writes a
341	nested path into a row.
342
343	```go
344	func InstanceID(host string) string { // ≤ 40 bytes; see §3.1
345	if len(host) <= 40 { return host }
346	sum := sha256.Sum256([]byte(host))
347	return "_" + host[:22] + "-" + hex.EncodeToString(sum[:8]) // '_' never occurs in a host
348	}
349	func InstanceDir(dataDir, host string) string {
350	return filepath.Join(dataDir, "instances", InstanceID(host))
351	}
352	func NestedDBPath(dataDir, host string) string {
353	return filepath.Join(InstanceDir(dataDir, host), host+".db")
354	}
355	func InstanceRunDir(host string) string {
356	return filepath.Join(SocketDir, "instances", InstanceID(host))
357	}
358	func NestedSocket(host string) string {
359	return filepath.Join(InstanceRunDir(host), "app.sock")
360	}
361	func LogSocket(host, role string) string {
362	return filepath.Join(SocketDir, "logs", InstanceID(host)+"."+role+".sock")
363	}
364	// DefaultDBPath and DefaultSocket keep their flat meaning for this
365	// release; the next release, which drops the flat spawn path, points
366	// them at the nested helpers for activator-managed routes and keeps
367	// them flat for externally supervised ones (§8.6).
368	```
369
370	`ExecBackend.defaultDBPath` (`backend_exec.go:160-162`) is deleted in
371	favour of the provision helpers, so there is one definition, not two.
372
373	**File names inside the directory keep the host prefix.** It costs
374	nothing, it keeps every existing test fixture recognisable, and it means
375	a file found outside its directory is still attributable.
376
377	**Why nested and not per-instance uids.** titogo's unit comment says it
378	best: "A flat `/data/titogo/%i.db` cannot be isolated — the bind would
379	have to reveal all of `/data/titogo`, which is the regression itself."
380	Per-instance uids (the discussion's option 2, titogo's "Option B") would
381	additionally need a uid allocator, a chown as root on every migration and
382	every restore, and a group arrangement that keeps the agent able to open
383	the database for replication. The namespace gives the property the
384	discussion asks for ("absent, not permission-denied") without any of
385	that, and leaves option B open as a later hardening on the same layout
386	(§11).
387
388	### 3.2 Symlinks are never followed under the trees the platform binds or opens for a tenant (codex R1-1, R1-2, R1-3, R3-1)
389
390	Three mount units, embedded and installed like every other unit
391	(`internal/host/bootstrap.go:33-34`), and tmpfiles lines beside the
392	existing `/run/carlos` one (`bootstrap.go:61`). All three carry
393	`DefaultDependencies=no`: a mount unit's default ordering places it
394	before `local-fs.target`, and `systemd-tmpfiles-setup.service`, which
395	creates the directories, runs after that target, so the defaults would
396	form a cycle systemd breaks by dropping the mount (codex R2-2). The
397	ordering is therefore written out in full, including shutdown:
398
399	```ini
400	# data-instances.mount
401	[Unit]
402	Description=carlos instance data (bind, nosymfollow)
403	DefaultDependencies=no
404	RequiresMountsFor=/data
405	After=local-fs.target systemd-tmpfiles-setup.service
406	Before=carlos-edge.service carlos-instance-control.service carlos-litestream.service
407	Conflicts=umount.target
408	Before=umount.target
409	[Mount]
410	What=/data/.instances
411	Where=/data/instances
412	Type=none
413	Options=bind,nosymfollow,nosuid,nodev,noexec
414	[Install]
415	WantedBy=multi-user.target
416	```
417
418	**The source is a different directory from the mountpoint** (codex
419	R4-2). `rename(2)` refuses to cross a mount boundary with `EXDEV` even
420	when both sides are one filesystem, so a bind of `/data/instances` onto
421	itself would make the migration's rename from `/data/<host>.db` into it
422	impossible. Binding `/data/.instances` onto `/data/instances` keeps the
423	raw directory reachable on the `/data` mount: the layout pass renames
424	`/data/<host>.db` → `/data/.instances/<host>/<host>.db` on one mount,
425	atomically, and the file appears at `/data/instances/<host>/<host>.db`
426	through the protected view. The raw path is opened by **exactly one
427	caller**, the layout pass, and only as `openat2(RESOLVE_NO_SYMLINKS |
428	RESOLVE_BENEATH)` directory fds for `renameat`, so no lookup through it
429	ever follows a symlink either; a CI test greps the tree for the literal
430	`.instances` and fails on any other occurrence.
431
432	`run-carlos-instances.mount` and `run-carlos-logs.mount` are the same
433	shape for `/run/carlos/.instances` → `/run/carlos/instances` and
434	`/run/carlos/.logs` → `/run/carlos/logs`, with
435	`After=systemd-tmpfiles-setup.service` and no `RequiresMountsFor=`;
436	nothing is ever renamed into those, so their raw directories have no
437	caller at all.
438
439	```
440	d /data/.instances 0700 carlos carlos -
441	d /data/instances 0700 carlos carlos -
442	d /data/layout 0700 carlos carlos -
443	d /run/carlos/.instances 2750 carlos carlos-sockets -
444	d /run/carlos/.logs 0700 carlos carlos -
445	f /etc/carlos/probe-secret 0640 root carlos - probe
446	d /run/carlos/instances 2750 carlos carlos-sockets -
447	d /run/carlos/logs 0700 carlos carlos -
448	d /run/carlos-control 0750 root carlos -
449	```
450
451	`carlos-edge.service`, `carlos-instance-control.service` and the three
452	templates gain `Requires=` and `After=` on the mounts they touch, so an
453	edge that starts without them does not start at all rather than starting
454	unprotected. The plan's first box task is a **cold boot** with the full
455	unit set installed, then `systemctl list-dependencies --before
456	data-instances.mount` and a `systemd-analyze verify` over the embedded
457	set, pinned as a CI test that renders the units and runs
458	`systemd-analyze verify` where the tool exists.
459
460	`nosymfollow` (Linux 5.10+, util-linux 2.36+; Debian 13 ships 6.12 and
461	2.41) makes path resolution on that mount return `ELOOP` at any symlink,
462	for every process, in every namespace that sees the mount. It is a
463	per-mount flag, so it holds through the `BindPaths=` bind into the
464	instance's namespace, through the agent's and the edge's ordinary
465	lookups from outside, and through PID 1's open of a `StandardOutput=file:`
466	path. `readlink` still works; nothing the platform runs needs a symlink
467	under these trees (SQLite, litestream, the app's listener and the
468	agent's env files are all plain files). Hard links cannot escape either:
469	a `link()` needs a source path the caller can resolve, and inside the
470	namespace no sibling path resolves.
471
472	What this closes, race-free and for every reader at once:
473
474	- the instance replacing its own `app.sock` with a symlink to the root
475	control socket, a sibling's socket, or a sibling's control socket, and
476	having the edge dial it (R1-1);
477	- the instance replacing `<host>.env`, `<host>.config.env` or `<host>.db`
478	with a symlink to a sibling's file, `/data/litestream.env` or any
479	agent-readable path, and having the agent read it into the instance's
480	environment or replicate it into the instance's replica (R1-3);
481	- the instance planting a symlink at a predictable temp name so the
482	agent's write truncates the target (R1-3);
483	- a still-unisolated flat tenant, during the transition, planting a
484	symlink at a log socket path so that PID 1 opens a root-owned file as
485	some unit's stdout (R3-1). `/run/carlos/logs` is `carlos`-owned because
486	the agent must bind there, so a same-uid process can write it; the
487	mount flag is what makes that write inert.
488
489	The flat tenants' own sockets under `/run/carlos` itself are not on such
490	a mount (the directory holds flat sockets, the apps subtree and more, and
491	this release does not change it). A flat tenant can therefore still point
492	its socket at any socket on the box, by symlink or by editing the
493	registry row it can reach as `carlos`, and have the edge dial it. §5.4
494	says why that no longer reaches the control service, and §6.1 names the
495	two checks the edge and the agent make on every unix dial.
496
497	Belt and braces in the agent, cheap and testable:
498
499	- every agent open of a file under an instance directory goes through
500	one helper, `instancefs.Open(dir, name, flags)`, built on
501	`openat2(2)` with `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS`
502	(Linux 5.6+), so a symlink is refused with a named error even on a
503	box where the mount option is missing; the same helper is what the
504	agent uses to verify, at start, that all three mounts carry
505	`nosymfollow` (`/proc/self/mountinfo`), and it **refuses to wake any
506	nested host** until they do (§8.5);
507	- every agent write into an instance directory creates its temp file
508	with `O_CREAT|O_EXCL` and a random suffix, then `renameat`s over the
509	target (rename replaces the directory entry and never follows the
510	destination);
511	- launch files (§6.2) are written the same way, and `instance-exec`
512	reads them through the same helper.
513
514	### 3.3 Nothing in an instance directory is an instruction, and a journal is not one either (codex R2-1, R3-1)
515
516	The agent reads three kinds of thing from an instance directory: the
517	database (to replicate, restore and checkpoint), the two env files (to
518	compose the instance's own environment), and the advisory exit record
519	(to decorate an event line, §6.5). It never reads a path, a host name, a
520	file list or a phase from there. In particular:
521
522	- the migration journal lives in `/data/layout/<host>.json`, outside
523	every tenant mount, and every source and destination path in it is
524	*derived from the host name and the registry row* by the §3.1 helpers
525	at plan time. On recovery the pass **recomputes** both endpoints for
526	the journal's own direction and honours the journal only if its paths
527	equal the recomputation and the row stands at one of them (§8.2); a
528	journal that passes is completed to its own end and the host is then
529	converged to the fleet's recorded intent (`control/layout.json`, a
530	bucket object no tenant can write) before it is released. During the
531	transition a flat tenant can write `/data/layout` (same uid); a journal
532	it forges can
533	therefore cause the agent to do exactly what the next layout pass would
534	do for that host anyway, and nothing else (R3-1);
535	- the launch file is agent-written and then read by the trampoline
536	inside the namespace as the instance itself; it directs nothing the
537	agent does (§6.2);
538	- the exit record is written inside the instance's run directory by the
539	unit's own `ExecStopPost=`, is therefore tenant-writable, and is treated
540	as the tenant's claim about its own exit status: it never decides
541	whether the process is gone (§6.5).
542
543	A test plants a forged journal (a sibling's paths under a host's name; a
544	direction opposite to the fleet's intent), a forged launch file naming a
545	sibling, and a forged exit record for a running unit, and asserts the
546	sibling is untouched and the running unit is not treated as exited.
547
548	---
549
550	## 4. The units
551
552	Unit files under `internal/host/units/`, embedded and installed exactly as
553	the existing nineteen are (`bootstrap.go:33-34`, `systemapply.go:339-398`):
554	`carlos-instance@.service`, `carlos-instance-seed@.service`,
555	`carlos-instance-sidecar@.service`, `carlos.slice`,
556	`carlos-instances.slice`, the three mount units of §3.2, the probe of
557	§8.5, and the two control units of §5.2. Nothing enables the templates;
558	an instance unit exists only while the agent has started it.
559
560	**Instance names are escaped instance identifiers.** The unit instance
561	is `systemd-escape(InstanceID(host))` (§3.1): for a valid identifier
562	(`[a-z0-9.-]`) that only turns `-` into `\x2d`. `%I` (unescaped) is used
563	wherever the identifier is a path component; `%i` (escaped) wherever it
564	is part of a unit name, which is what lets a per-host slice be spelled
565	in the template (R1-8). The agent and the control service share one Go
566	escape function (a copy of systemd's rules for this character set,
567	pinned by a test against `systemd-escape` output for the hosts in the
568	fixture set). The templates never need the full host: every path they
569	name is keyed by `<iid>`, and the trampoline learns the host from the
570	launch file (§6.2).
571
572	### 4.1 `carlos-instance@.service`
573
574	```ini
575	[Unit]
576	Description=carlos exec-backed instance %I
577	# Stop-only coupling: when the edge stops (including the stop half of a
578	# restart) this unit stops, and nothing starts it again but the agent.
579	# PartOf= would propagate the RESTART and race the agent's own startup
580	# (codex R1-7).
581	BindsTo=carlos-edge.service
582	Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
583	After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
584
585	[Service]
586	Type=simple
587	ExecStart=/opt/carlos/carlos instance-exec run %I
588	# Runs inside the same namespace, as carlos, after this invocation's
589	# processes have been signalled: leaves an ADVISORY exit record in the
590	# instance's own run directory (§6.5). Not "+": the "+" prefix drops
591	# User= and the Private*/Protect* settings but keeps the explicit
592	# TemporaryFileSystem=/BindPaths= mounts (codex R3-2), so nothing run
593	# from here can see the host's /run/carlos anyway.
594	ExecStopPost=/opt/carlos/carlos instance-exit run %I
595	User=carlos
596	Group=carlos
597	# One slice per host holds the instance, its seed and its sidecar, and
598	# carries the tier (§4.3, §6.6). Slices referenced here are created on
599	# demand; no per-host unit file exists.
600	Slice=carlos-instances-%i.slice
601	# The activator owns recycling: a crash stands the entry down and the
602	# next request wakes it. systemd must not race that.
603	Restart=no
604	# SIGTERM to the app, SIGKILL to anything it left behind after
605	# TimeoutStopSec; today's actStopTimeout is 20s.
606	KillMode=mixed
607	TimeoutStopSec=20
608	# The log stream: stdout and stderr connect to the agent's per-role
609	# listener (§6.4). PID 1 opens this as root, outside the namespace, on
610	# the nosymfollow logs mount, at a path no tenant mount contains.
611	StandardOutput=file:/run/carlos/logs/%I.run.sock
612	StandardError=inherit
613
614	# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
615	ProtectSystem=strict
616	TemporaryFileSystem=/data:ro
617	TemporaryFileSystem=/run/carlos:ro
618	# Every other tree a carlos-uid unit writes on the box is hidden the same
619	# way (codex R9-2): the restore drills' cache under /var/cache, and
620	# /var/lib as a class. An instance needs nothing from either.
621	TemporaryFileSystem=/var/cache:ro
622	TemporaryFileSystem=/var/lib:ro
623	# Root-written credentials a carlos-GROUP process may read live here:
624	# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
625	# and an instance's Group=carlos would read it too (codex R10-1). An
626	# instance gets its configuration through its launch file and needs
627	# nothing from /etc/carlos.
628	InaccessiblePaths=/etc/carlos
629	BindPaths=/data/instances/%I /run/carlos/instances/%I
630	ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
631	ProtectHome=yes
632	PrivateTmp=yes
633	PrivateDevices=yes
634	# Its own /dev/shm; PrivateDevices= would bind the host's (R15).
635	TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
636	# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
637	# and its cgroup.kill / cgroup.freeze would otherwise be one write away
638	# for a same-uid tenant (codex R9-1).
639	ProtectControlGroups=yes
640	ProtectProc=invisible
641	ProcSubset=pid
642	PrivatePIDs=yes
643	RestrictNamespaces=yes
644	NoNewPrivileges=yes
645	CapabilityBoundingSet=
646	RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
647	IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
648	ProtectKernelTunables=yes
649	ProtectKernelModules=yes
650	ProtectKernelLogs=yes
651	ProtectClock=yes
652	ProtectHostname=yes
653	RemoveIPC=yes
654	LockPersonality=yes
655	RestrictRealtime=yes
656	# No RestrictSUIDSGID=: it refuses openat2 outright (R15).
657	MemoryDenyWriteExecute=yes
658	SystemCallArchitectures=native
659
660	# --- per-service resources; the tier lives on the host slice (§4.3) ---
661	TasksMax=256
662	# The kernel picks a tenant before it picks anything else in carlos.slice.
663	OOMScoreAdjust=500
664	# One OOM kill takes every process of THIS service (systemd sets
665	# memory.oom.group on the service cgroup for OOMPolicy=kill). The sidecar
666	# is a sibling service with its own family rule; §4.3 says why the two are
667	# not one family any more.
668	OOMPolicy=kill
669	```
670
671	Lines that are load-bearing and why:
672
673	- **`TemporaryFileSystem=/data:ro` + `BindPaths=/data/instances/%I`** is
674	the isolation. The tmpfs is empty; the bind reveals one directory. A
675	sibling's database is not permission-denied, it does not exist. The
676	same for `/run/carlos`: a sibling's listening socket and control socket
677	do not exist, which closes the socket-file impersonation in §1.2. The
678	bound directory is inside the `nosymfollow` mount (§3.2), so a symlink
679	the instance plants there is dead for every reader.
680	- **The bind sources must exist before the unit starts**, or systemd
681	fails at namespace setup before any `ExecStartPre=` runs (titogo
682	verified this; it is why `titogo-instance-dir@.service` exists). Here
683	the agent is the supervisor and creates both directories before it
684	asks for a start (§6.1), so no helper unit is needed.
685	- **`PrivatePIDs=yes` + `ProtectProc=invisible`.** Every instance is uid
686	`carlos`, and `ProtectProc=invisible` does not hide same-uid peers
687	(hidepid's test is `ptrace_may_access`, which same-uid passes). With its
688	own pid namespace an instance's `/proc` lists only itself: no sibling
689	pid to traverse into `/proc/<pid>/root/`, no sibling to signal, nothing
690	for a same-uid ptrace to attach to. `PrivatePIDs=` needs systemd 257
691	(Debian 13, which `infra/modules/region-host/variables.tf:134` pins)
692	and is **ignored, not refused**, on a kernel that cannot provide it, so
693	§8.5 gates on observed behaviour, not on the directive (R1-15).
694	- **`RestrictNamespaces=yes`**: without it an unprivileged user namespace
695	could bind-mount over a fresh mountpoint and re-reveal what the tmpfs
696	hid.
697	- **`IPAddressDeny=`** for IMDS. The nft rule in pinfra allows uid
698	`carlos` because the agent needs IMDS; this line is what stops an
699	instance, also uid `carlos`, from fetching the box's role credentials.
700	It is a cgroup BPF **packet** filter: a denied datagram send fails with
701	`EPERM`, a denied TCP `connect()` loses its SYN and times out. §8.5's
702	probe is written for that (R3-4).
703	- **`BindsTo=carlos-edge.service`** keeps today's "instances die with the
704	agent": when the edge stops, so do they, and nothing restarts them but
705	the agent's next wake. §6.7 adds the startup sweep for the case where
706	the coupling did not fire.
707	- **`StandardOutput=file:` to an `AF_UNIX` socket** is documented
708	systemd behaviour: a stream connection is made and used as stdout. It
709	is opened by PID 1 outside the namespace, so the path is the real one,
710	the path is in `/run/carlos/logs`, which no tenant mount contains, and
711	that directory is on a `nosymfollow` mount, so a same-uid process
712	outside any namespace cannot turn it into a symlink either.
713	- **`ProtectControlGroups=yes`** (codex R9-1). `ProtectSystem=strict`
714	leaves kernel API filesystems alone and `ProtectKernelTunables=` leaves
715	`/sys/fs/cgroup` writable; the edge unit's `Delegate=` subtree,
716	including the `supervise/` cgroup the agent moves itself into
717	(`internal/cgroup/cgroup.go:172`), is created by `carlos` and its
718	`cgroup.kill` and `cgroup.freeze` are owner-writable. A same-uid
719	tenant with a writable cgroupfs could kill or freeze the agent with
720	no pid and no socket. Read-only cgroupfs closes that; an instance has
721	no business in cgroupfs at all.
722	- **`TemporaryFileSystem=/var/cache:ro` and `/var/lib:ro`** (codex
723	R9-2). The isolation property is "no same-uid path on the box but its
724	own", and two trees outside `/data` and `/run/carlos` are written by
725	`carlos`-uid units: `carlos-hibernate-verify.service` and
726	`carlos-restore-verify.service` set `CacheDirectory=carlos-verify` and
727	restore every host's database into `/var/cache/carlos-verify` during a
728	drill (`internal/ops/ops.go:274`, `internal/lease/check.go:79`), and
729	`/var/lib` holds unit tenants' state. Both are hidden. §9 carries the
730	audit rule that found them, so the next such unit is caught at review.
731	- **`InaccessiblePaths=/etc/carlos`** (codex R10-1). `ProtectSystem=strict`
732	makes `/etc` read-only, not unreadable. `carlos-relay.service` runs as
733	`carlos:carlos` and reads `/etc/carlos/relay-masters`
734	(`internal/relay/masters.go:101`), which the relay design stages as
735	`root:carlos 0640` (`2026-09-07-relay-stun-turn-design.md:325`): a
736	file every `Group=carlos` process can read, and the secret every
737	app's relay key is derived from (`internal/relay/server.go:665`).
738	The instance's group is `carlos` for the socket contract (§3.1), so
739	the directory is hidden outright. `host.env` and `apps/*.env` there
740	are root-0600 and were already unreadable; hiding the directory is
741	what makes the next `0640` file safe by default.
742	- **`/opt/carlos`** stays readable through `ProtectSystem=strict`; the
743	trampoline and the live symlink are all the instance needs from it.
744
745	### 4.2 `carlos-instance-seed@.service` and `carlos-instance-sidecar@.service`
746
747	Identical `[Unit]` coupling, `[Service]` namespace, hardening and
748	resource blocks, and the same `Slice=carlos-instances-%i.slice`.
749	Differences:
750
751	| | seed | sidecar |
752	|---|---|---|
753	| `Type=` | `oneshot` with `RemainAfterExit=yes`: after `ExecStart` exits the unit stays `active (exited)` or `failed`, loaded, with its `InvocationID`, `Result` and `ExecMainStatus` retained until the control service stops it, so the seed's result can never be garbage-collected before it is read (codex R2-4, R5 disposition) | `simple` |
754	| `ExecStart=` | `/opt/carlos/carlos instance-exec seed %I` | `/opt/carlos/carlos instance-exec sidecar %I` |
755	| `ExecStopPost=` | `/opt/carlos/carlos instance-exit seed %I` | `/opt/carlos/carlos instance-exit sidecar %I` |
756	| `StandardOutput=` | `file:/run/carlos/logs/%I.seed.sock` | `file:/run/carlos/logs/%I.sidecar.sock` |
757	| timeouts | `TimeoutStartSec=120` (today's seed budget, `backend_exec.go:368-408, 466`; a oneshot has no start timeout by default, R1-5), `TimeoutStopSec=20` | `TimeoutStopSec=20` |
758
759	A Go test over the embedded FS asserts that the three files' namespace,
760	hardening and coupling lines are byte-identical (§10). Three files rather
761	than one because seed and instance never run together but sidecar and
762	instance do, and a template's instance is the host, not a role.
763
764	### 4.3 Slices and the memory walls (codex R1-8, R1-10, R2-11, R3-3)
765
766	```ini
767	# carlos.slice — the outer wall of the 2026-08-28 design, as a slice
768	[Unit]
769	Description=carlos (edge, activator and every exec-backed instance)
770	[Slice]
771	MemoryHigh=70%
772	MemoryMax=80%
773	```
774
775	```ini
776	# carlos-instances.slice — the parent of every per-host slice
777	[Unit]
778	Description=carlos exec-backed instances
779	[Slice]
780	# No cap of its own: the outer wall is carlos.slice, the inner walls are
781	# the per-host slices. This level exists so the sampler has one root to
782	# scan and so an operator can `systemctl status carlos-instances.slice`.
783	```
784
785	`carlos-edge.service` gains `Slice=carlos.slice` and **drops** its own
786	`MemoryHigh=70% MemoryMax=80%`: the wall moves up one level and now
787	bounds exactly what it bounded before, the agent plus every awake
788	instance, so borrowing between them and the OOM candidate set (a tenant
789	at `+500` is always picked before the agent) are unchanged. The
790	2026-08-28 reasoning, including the measured 1.2 GB leaf that was mostly
791	reclaimable page cache, carries over without new numbers.
792	`OOMPolicy=continue` stays on the edge unit.
793
794	The per-host slice `carlos-instances-<esc>.slice` is created on demand
795	by the first unit that names it. Before every start the control service
796	sets three properties on it from the route's tier (§5.3, §6.6):
797	`MemoryMax`, `MemoryHigh` (90% of max) and **`MemorySwapMax` equal to
798	`MemoryMax`**, the last because with zram on the box a cgroup at
799	`memory.max` otherwise spills into compressed swap and carries on, which
800	`internal/cgroup/cgroup.go:313-318` learned live on the flagship
801	(R2-11). The values are `512M`/`460M`/`512M` by default and
802	`1G`/`920M`/`1G` for an app whose manifest declares
803	`capabilities.children`, exactly what `internal/cgroup` writes today.
804	Seed, instance and sidecar all charge that one cgroup, which is what
805	today's shared `carlos-exec@<host>` gives them
806	(`backend_exec.go:149-155, 373-375, 890-895`).
807
808	**The family OOM rule changes, deliberately.** Today `memory.oom.group=1`
809	on the shared `carlos-exec@<host>` cgroup means an OOM kill in the
810	sidecar takes the instance with it. systemd writes `memory.oom.group`
811	itself, for slices too, from the unit's cgroup context (default `0`,
812	`cgroup_context_apply` in `src/core/cgroup.c`), so a value written
813	behind its back is overwritten on the next property application, which
814	`/start` for the sidecar performs on a slice whose instance is already
815	running (codex R3-3). There is no supported way to keep it set on the
816	slice. So the rule becomes: **each service is its own OOM family**
817	(`OOMPolicy=kill`, which sets the attribute on the service's cgroup), and
818	a sidecar killed by the OOM killer is a sidecar that died, which the
819	sweeper already restarts with backoff (`activator.go:1165-1200`) while
820	the instance keeps serving. An instance killed by the OOM killer still
821	stands its entry down and co-stops its sidecar as today. The instance
822	surviving its sidecar's OOM is the better of the two behaviours; the
823	spec records it as a difference rather than claiming parity.
824
825	The cgroup path is
826	`/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/`
827	(R1-9).
828
829	---
830
831	## 5. The root channel: `carlos-instance-control`
832
833	### 5.1 Why a channel at all
834
835	The agent is uid `carlos`. `systemctl start` on a system unit needs
836	either root or a polkit grant; read-only `systemctl show` / `is-active` /
837	`list-units` do not. The existing root relays (`carlos-stage.path`,
838	`carlos-restart.path`) are asynchronous sentinel files, fine for a
839	minute-scale adoption pass and wrong for a wake that must answer a
840	waiting HTTP request.
841
842	### 5.2 Units
843
844	```ini
845	# carlos-instance-control.socket
846	[Unit]
847	Description=carlos instance control socket (agent → root)
848	[Socket]
849	ListenStream=/run/carlos-control/control.sock
850	SocketUser=carlos
851	SocketGroup=carlos
852	SocketMode=0600
853	Accept=no
854	[Install]
855	WantedBy=sockets.target
856	```
857
858	```ini
859	# carlos-instance-control.service
860	[Unit]
861	Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent)
862	Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
863	After=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
864
865	[Service]
866	ExecStart=/opt/carlos/carlos instance-control
867	# Root deliberately: the whole job is systemctl start/stop, and the
868	# caller is the unprivileged agent. Same rationale as carlos-stage.service.
869	ProtectSystem=strict
870	ProtectHome=yes
871	PrivateTmp=yes
872	PrivateDevices=yes
873	ProtectKernelTunables=yes
874	ProtectKernelModules=yes
875	ProtectKernelLogs=yes
876	RestrictAddressFamilies=AF_UNIX
877	MemoryDenyWriteExecute=yes
878	MemoryMax=64M
879	```
880
881	The socket lives in `/run/carlos-control`, `0750 root:carlos`: any
882	`carlos` process can connect (search on the directory), none can create,
883	rename or replace anything in it. The unit is socket-activated and runs
884	only while there is work. It is hidden from every instance's namespace
885	by `InaccessiblePaths=` in the shared block (§4.2): a connect from
886	inside one is impossible, and the peer check refuses everything else.
887	`/run/carlos` inside an instance is an empty tmpfs with the instance's
888	own directory bound in.
889
890	### 5.3 Protocol
891
892	HTTP/1.1 over the unix socket, the shape `edge.ListenControl` already
893	uses for the per-instance control socket. Before reading a request line
894	the service authenticates the **connection** (§5.4); a connection that
895	fails is closed without a response. Five verbs, no shell, no argument
896	passes through to `systemctl` except a unit name the service built
897	itself:
898
899	| Verb | What it runs | Returns | Agent deadline |
900	|---|---|---|---|
901	| `POST /start?host=H&role=run\|sidecar&memory_max=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N`; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID <unit>`, read immediately after the start returns; or 502 with systemctl's stderr | 30s |
902	| `POST /seed?host=H&memory_max=N` | the same `set-property`; `systemctl start` on `carlos-instance-seed@<esc>.service` (blocking: a oneshot's start job completes when `ExecStart` exits, or fails at `TimeoutStartSec`); if the InvocationID it left is the SAME one that was running before this start, an earlier request's stop never ran, so the stale unit is stopped and started again; the unit is now `active (exited)` or `failed` and **stays loaded** (`RemainAfterExit=yes`, §4.2); `systemctl show -p InvocationID,Result,ExecMainStatus,ExecMainCode` reads the retained result; `systemctl stop` (instant for an exited oneshot; clears it), on a context detached from the request so a disconnected client cannot leave it running; then wait for the cgroup to be empty (§6.5's rule, applied by the service as root) | 200 with `{invocation_id, result, status, code}`, always complete | 150s: 120s seed budget plus `TimeoutStopSec` and slack (R3-5) |
903	| `POST /stop?host=H&role=run\|seed\|sidecar` | `systemctl stop <unit>` (blocking; also cancels a pending start job) | 204 | 30s |
904	| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>`, tolerating "not loaded" | 204 | 30s |
905	| `POST /sweep` | `systemctl list-units --all` and `list-jobs` over the three templates' instances; `systemctl stop` each that is not `inactive` or has a job; wait for terminal state and empty cgroup on each | 200 with the list of unit names swept, or 502 naming the unit that would not terminate | 60s |
906	| `GET /ready` | nothing | 204 once the probe verdict (§8.5) is `ready`; 503 with the verdict otherwise | 5s |
907
908	There is deliberately **no `reset-failed` inside `/start`**: `ResetFailedUnit`
909	refuses a unit that is not loaded, and a template instance that has never
910	started is not loaded, so the first wake of every host would fail (codex
911	R2-3). `systemctl start` on a `failed` unit already clears the failed
912	state; `/reset` exists for the sweeper's housekeeping only.
913
914	Validation, in order: connection (§5.4); `host` must satisfy
915	`provision.ValidHostShape` (`provision.go:262`); `role` must be one of
916	the three literals; the memory value must parse as a `Limit` the agent
917	already knows how to produce (`internal/cgroup`,
918	`TierDefault`/`TierChildren`). Anything else is a 400 with no systemctl
919	call. The unit name is
920	`"carlos-instance" + suffix + "@" + escape(InstanceID(host)) + ".service"`
921	and the slice name `"carlos-instances-" + escape(InstanceID(host)) + ".slice"`,
922	built from the validated host by the same `InstanceID` and escape
923	functions the agent used to create the directories (§3.1, §4), never
924	from the request string (codex R5-3 disposition).
925
926	What this grants the agent: start, stop, reset and tier control of units
927	named `carlos-instance*@<host>` and their host slice. That is exactly
928	what it has today over its own children. It grants nothing else, and a
929	host that does not exist is a unit whose bind source does not exist,
930	which fails at namespace setup and starts nothing.
931
932	`systemctl set-property --runtime` on a slice or on an instance of an
933	installed template that has never been started loads the unit first and
934	writes a runtime drop-in the following `start` honours
935	(`SetUnitProperties` in `src/core/dbus-manager.c`; confirmed by codex
936	R1-1). The plan's first task exercises it on a dev box for a slice that
937	does not yet exist.
938
939	A `/start` or `/stop` that exceeds its deadline is answered by the agent
940	with a `/stop` and reported as a failed wake or an unproven stop
941	(§6.3); the deadlines exist because §6.7 puts one such call on a path
942	that must never stall the activator, and the seed's is long because a
943	seed is allowed to be slow (R3-5).
944
945	### 5.4 Who may call it, and the ruling (codex R1-1, R3-1)
946
947	The other way to let uid `carlos` start these units is polkit: a rules
948	file granting `org.freedesktop.systemd1.manage-units` to `carlos` when
949	the unit name matches `carlos-instance*@*.service`. Codex R1-1 confirmed
950	that polkit can express the unit and verb restrictions, and also that it
951	cannot constrain property values, that a uid-only grant is reachable by
952	every tenant through the system bus, and that a wildcard on names also
953	authorises transient units. Codex's advisement, concurring with the
954	draft's: **the control service, provided it authenticates the caller and
955	not the socket's uid.** Ruled, 2026-09-11: control service; polkit is not
956	installed and not relied on.
957
958	Revision 2 authenticated the caller with a bearer token in a
959	`carlos`-readable file. Round 3 showed why that is not a boundary during
960	the transition: a flat tenant is uid `carlos` in the host namespace and
961	reads that file as easily as the agent does (R3-1). A secret the agent
962	holds cannot distinguish the agent from an unisolated child of it. What
963	can is **systemd's own record of which process is the agent**:
964
965	- On every accepted connection the service reads `SO_PEERCRED` and
966	requires `uid == carlos` **and** `pid == MainPID` of
967	`carlos-edge.service` as `systemctl show -p MainPID` reports it at that
968	moment (root, one call, cached for one second). A flat tenant is a
969	child of that pid, never the pid itself; a nested tenant cannot reach
970	the socket at all.
971	- The edge is the agent, so a request the edge *proxies* for a tenant
972	(a flat tenant pointing its own socket at `control.sock`, §3.2) would
973	pass that check. It does not get that far, because of two checks on
974	**every unix dial the edge and the agent make** (`edge.go:2000-2014`'s
975	`unixProxy`, the activator's health and control dials, §6.1):
976	1. **the reserved endpoint is refused before the dial**: a path under
977	`/run/carlos-control/` is never dialled as an upstream, so a
978	registry row pointing at `control.sock` is refused before
979	`connect()`. Nothing else is refused by path: a route whose
980	operator stamped `--socket /run/titogo/example.sock` keeps working
981	exactly as today (codex R5-1);
982	2. **the listener answers to the path that was dialled**: after
983	`connect()` and before a byte is sent, `getpeername()` on the
984	connected socket returns the address the *listener* bound, which is
985	its real path however it was reached; if that is not byte-equal to
986	the path the edge dialled, the connection is closed. A symlink at
987	`/run/carlos/<host>.sock` pointing anywhere, including a sibling's
988	socket or the control socket, therefore yields a mismatch.
989	Revision 4's rule, "refuse a peer whose uid is 0", is withdrawn: a
990	listener created by PID 1 for `carlos-app@.socket` reports root
991	credentials in `SO_PEERCRED` however the fd is later passed
992	(codex R4-1), so that rule would have refused every unit-backed app.
993	The two checks above do not look at credentials at all.
994
995	A proxied request therefore never reaches the service, and the service
996	never sees a connection from anything but the agent's own goroutines.
997	No token: the file it lived in is gone, and with it the question of who
998	can read it. Each check is one syscall or one cached `systemctl show`,
999	each is testable with a real socket pair, and all of them hold on a box
1000	that still runs flat tenants, which is the whole transition. They also
1001	end, as a side effect, the flat-tenant socket impersonation §1.2 names,
1002	which a symlink could otherwise still perform during this release.
1003
1004	---
1005
1006	## 6. The activator on units
1007
1008	`internal/activator/activator.go` changes in exactly three places: the
1009	sidecar restart in `sweepOnce` (§6.7, codex R2-6), the lease release
1010	in `stop` (§6.3, codex R2-5), and the route the wake runs on (§6.8,
1011	codex R8-1). `doWake` (`activator.go:772`), `Bounce` (`:946-956`) and
1012	the `Proc`/`Rep` seams (`:194-207`) keep their contracts.
1013	`Exited()` stays a non-blocking read of a flag (R1-4); everything that
1014	talks to systemd runs in the proc's own goroutine or in the
1015	sidecar-start goroutine. The rest of the change is inside `ExecBackend`,
1016	which keeps its current `childProc` path for flat hosts for this one
1017	release (§8.1) and adds the unit path for nested hosts.
1018
1019	### 6.1 Preparing a role, and `Start`
1020
1021	One helper, `prepare(rt, role)`, runs before **any** role starts, seed
1022	included (R1-5, R2-12):
1023
1024	1. Ensure `InstanceDir` and `InstanceRunDir` exist (0700 and 2750 as in
1025	§3.1), and `/run/carlos/logs`.
1026	2. `EnsureAdminToken(InstanceDir/<host>.env)` for `run` **and** `seed`
1027	(a seed is the first thing a fresh instance runs and must see the
1028	token the instance will, `backend_exec.go:346-352`); `instanceEnv`
1029	for both, reading the two files from `InstanceDir` through
1030	`instancefs`, with the seed's removal of the control-channel variables
1031	as today. For `run` only: bind the control listener at
1032	`rt.Socket + ".control"` (unchanged, `edge.ListenControl`) and clear
1033	any stale `app.sock` (nothing is running: the unit is not active, or
1034	we would not be here). Remove any stale `exit.<role>.*.json`.
1035	3. Bind the role's log listener at `LogSocket(host, role)` (§6.4).
1036	4. Write `launch.<role>.json` (§6.2).
1037
1038	`Start` is then: `prepare(rt, "run")`; `POST /start` with the route's
1039	tier; `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` (unchanged,
1040	with the path and `getpeername` checks of §5.4 on its dial); the `0660` normalisation of
1041	`app.sock` (unchanged); on failure `POST /stop`, wait for the proc's
1042	`done`, and clean up, exactly as the current failure path stops the
1043	child; return a `unitProc` carrying the invocation id.
1044
1045	### 6.2 The trampoline and `launch.<role>.json`
1046
1047	```json
1048	{
1049	"schema": 1,
1050	"host": "<host>",
1051	"argv": ["/opt/carlos/live/<host>", "--socket", "/run/carlos/instances/<iid>/app.sock", "--db", "/data/instances/<iid>/<host>.db"],
1052	"env": ["PATH=...", "CARLOS_ADMIN_TOKEN=...", "CARLOS_CONTROL_SOCKET=...", "..."]
1053	}
1054	```
1055
1056	`carlos instance-exec <role> <iid>` runs as `ExecStart`, inside the
1057	namespace, as `carlos`. It reads `/data/instances/<iid>/launch.<role>.json`
1058	through `instancefs`, refuses a schema it does not know, refuses an
1059	`argv[0]` that is not directly under `/opt/carlos/live/` or that does
1060	not equal `/opt/carlos/live/` plus the file's own `host` (defence in
1061	depth: the file is agent-written; the prefix check is the one that
1062	holds even against a rewritten file, since only the platform's live
1063	links live there), and calls
1064	`syscall.Exec(argv[0], argv, env)`. It never parses env syntax: the
1065	agent's `childEnv` (`backend_exec.go:802-814`) composes the list exactly
1066	as it does today, including the agent-owned removal of tenant-supplied
1067	`CARLOS_CONTROL_SOCKET` / `CARLOS_PURGE` (`backend_exec.go:636-660`), and
1068	the trampoline passes it through.
1069
1070	**One entry per key, last value wins** (R1-16). `os/exec` deduplicates
1071	`cmd.Env` in favour of the last occurrence; `syscall.Exec` passes
1072	duplicates through and a Go child keeps the first. `childEnv` therefore
1073	gains the deduplication itself (a map keyed on the name, insertion order
1074	preserved), and the test asserts the effective value *inside* an
1075	executed stub, not the JSON array.
1076
1077	What the current `exec.Command` path does that the trampoline must
1078	reproduce, and how: working directory (none is set today, and none is
1079	set; the app starts in `/`), inherited fds (none beyond stdio today;
1080	`syscall.Exec` inherits whatever the trampoline holds, which is stdio
1081	only), `WaitDelay` (replaced by `TimeoutStopSec`, §6.4), `Pdeathsig`
1082	(replaced by `BindsTo=` plus the startup sweep, §6.7),
1083	`oom_score_adj` (`OOMScoreAdjust=`), the cgroup placement (`Slice=`).
1084
1085	Why not `EnvironmentFile=`: systemd's parser and the agent's
1086	`parseEnvBytes` differ on quoting and continuation, and the agent-owned
1087	variables would then have to be written into a file the tenant's
1088	`<host>.env` overlay must not win over. One composer, one format, one
1089	test.
1090
1091	The launch file is written fresh before every start, `O_EXCL` temp plus
1092	rename (§3.2), and holds the same material the child could already read
1093	from `/proc/self/environ`. It is inside the instance's own directory,
1094	visible to that instance and the agent, nobody else. An instance can
1095	rewrite its own launch file while it runs; the next start reads it again
1096	through `instancefs`, refuses any `argv[0]` but its own live link, and
1097	executes only what a tenant could execute anyway: its own binary with an
1098	argv and env it chose. That is not a privilege, and the agent never
1099	reads the file (§3.3).
1100
1101	Seed: `launch.seed.json` carries the argv from `rt.Seed`
1102	(`backend_exec.go:332-338`) with the DB path substituted as today; `POST /seed`
1103	blocks up to its 150s deadline; the seed's exit status arrives in the
1104	response or in its exit record (§6.5); a non-zero status, a timeout or a
1105	cancelled `ctx` runs `POST /stop?role=seed`, waits for the seed proc's
1106	`done`, and only then runs the same cleanup as today
1107	(`backend_exec.go:438`), so the lease is never released
1108	(`activator.go:824-828`) while a seed can still write (R1-5).
1109
1110	Sidecar: `launch.sidecar.json` carries `[live/<host>, "sidecar", "run"]`
1111	and the sidecar env from `<host>-sidecar.env` only, as today
1112	(`backend_exec.go:885-911`); `POST /start?role=sidecar`.
1113
1114	### 6.3 `unitProc`, and a stop that could not be proven (codex R2-5)
1115
1116	```go
1117	type unitProc struct {
1118	host, role string
1119	invocation string // from the start response
1120	ctl *edge.ControlListener // run only; as childProc.ctl today
1121	logs *logListener // §6.4
1122	exited atomic.Bool
1123	done chan struct{}
1124	}
1125	func (p *unitProc) Stop(ctx context.Context) error // POST /stop, then wait for done or ctx
1126	func (p *unitProc) Exited() bool // p.exited.Load()
1127	```
1128
1129	The control listener's close ordering (`backend_exec.go:992-1009`: signal
1130	first, close the listener only after the process is gone) is preserved:
1131	`Stop` posts the stop, waits for `done`, then closes `ctl` and `logs`.
1132	`done` closes only on proven termination (§6.5), never on log EOF.
1133
1134	If `ctx` expires before `done` closes, `Stop` returns `ErrStillRunning`
1135	without closing the listeners. Today `stop()` then calls `rep.Close`
1136	regardless (`activator.go:1014-1033`), which releases the lease over a
1137	process that may still be writing. **The second `activator.go` change**:
1138	on `ErrStillRunning`, `stop()` does not call `rep.Close`; it marks the
1139	entry's outcome uncertain (the field exists, `outcome.uncertain`), leaves
1140	the entry in `stopping`, and `sweepOnce` retries `proc.Stop` on its
1141	cadence until `done` closes, after which the normal `rep.Close` runs.
1142	The lease's TTL bounds the worst case, as it does today when an agent
1143	dies mid-stop; what changes is that the agent never *voluntarily*
1144	releases over an unproven termination.
1145
1146	### 6.4 Logs
1147
1148	Today `teeStd` (`backend_exec.go:123-137`) pipes the child's stdout and
1149	stderr through the agent: straight to the agent's own stdout (journald)
1150	and, with `Logs` set, line-wise into `Logs.App` for `carlos logs`, capped
1151	at `execLogLineMax`, with `WaitDelay` bounding a grandchild that holds
1152	the pipe.
1153
1154	Each role's unit connects its stdout and stderr to that role's socket in
1155	`/run/carlos/logs` (`StandardOutput=file:` on an `AF_UNIX` path; R1-2,
1156	R1-6). The agent's `logListener` for a role accepts one connection per
1157	start (the unit connects exactly once, at exec; a second connection
1158	during the same invocation is closed and logged), and feeds it to the
1159	same `applog.NewLineWriter` and the same `io.MultiWriter` to the agent's
1160	stdout. Same bytes, same cap, same sink, same `carlos logs`, and the
1161	instance and its sidecar no longer contend for one connection. The
1162	listener is bound in `prepare` before the start request, so PID 1's
1163	connect cannot race it. A flat tenant, during the transition, can
1164	connect to a log socket (same uid, host namespace) and write lines into
1165	another host's log stream; that is a nuisance the transition accepts and
1166	the next release ends, and it cannot make anything *read* wrong, because
1167	nothing reads the log stream as an instruction.
1168
1169	Grandchildren: a process the app forked that outlives it keeps the
1170	socket open, as it keeps the pipe open today. `KillMode=mixed` plus
1171	`TimeoutStopSec=20` kills it on stop; on a crash the unit's cgroup empties
1172	when systemd reaps it. EOF on the log connection ends **log collection
1173	for that invocation**; it does not mean the process is gone (R1-4).
1174
1175	### 6.5 Exit observation (codex R1-4, R2-4, R2-5, R3-2, R3-7)
1176
1177	`childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`),
1178	which is process death. A unit proc needs the same certainty, attributed
1179	to the right invocation, and it must not depend on systemd keeping an
1180	inactive unit loaded: systemd garbage-collects an unreferenced inactive
1181	unit and its `ExecMainStatus` with it, so a `show` after a quick clean
1182	exit can find nothing to show (R2-4). Revision 3 tried to write the
1183	record from `ExecStopPost=+` into a root-owned directory; the `+` prefix
1184	drops `User=` and the `Private*`/`Protect*` settings but keeps the
1185	explicit `TemporaryFileSystem=`/`BindPaths=` mounts (`apply_mount_namespace`
1186	in `src/core/exec-invoke.c`), so that command could not see the
1187	directory (R3-2). So the record is **advisory, and truth is elsewhere**:
1188
1189	**`carlos instance-exit <role> <host>`** runs as `ExecStopPost=` inside
1190	the namespace as `carlos`, after the invocation's processes have been
1191	signalled and reaped by systemd's stop sequence, with `$INVOCATION_ID`,
1192	`$SERVICE_RESULT`, `$EXIT_CODE` and `$EXIT_STATUS` in its environment.
1193	It writes `/run/carlos/instances/<host>/exit.<role>.<INVOCATION_ID>.json`
1194	(`O_EXCL`) with those four values. The directory is the instance's own,
1195	so the instance could have written that file itself; the record is
1196	therefore the tenant's claim about its own exit and is used for one
1197	thing only: the wording of the member-facing event line. It never
1198	decides whether the process is gone, and a record for a running unit is
1199	ignored (§3.3). The agent, as `carlos`, removes it once consumed, and
1200	`prepare` removes stale ones (R3-7).
1201
1202	Each `unitProc` runs one goroutine that establishes termination from
1203	two facts the tenant cannot forge:
1204
1205	1. **systemd's unit state**: `systemctl show -p ActiveState,SubState,Result,InvocationID <unit>`
1206	(read-only, unprivileged), polled every 2s and triggered early by
1207	inotify on the run directory (the record's arrival) and on the unit's
1208	`cgroup.events` file. Terminal is `inactive` or `failed` with the
1209	proc's own `InvocationID`, **or** `inactive` with an empty
1210	`InvocationID` (the unit was already unloaded; the proc's invocation
1211	is gone).
1212	2. **cgroup emptiness**: the unit's cgroup directory under the host
1213	slice is absent, or its `cgroup.events` says `populated 0`. A service
1214	enters `failed` after its final `SIGKILL` timeout **with processes
1215	still present** if one is unkillable (`src/core/service.c:4020-4022`),
1216	and the lease must not be released over a writer (R2-5).
1217
1218	Only when both hold does the goroutine emit the event line with today's
1219	wording (`instance exited: ...` with the status or signal from the
1220	record if one exists for this invocation, or `instance exited (exit 0)`,
1221	or `instance exited (status unknown: no exit record)`), set `exited`,
1222	and close `done`. A populated cgroup after a terminal state is logged
1223	every 2s as `instance <host>: processes remain after SIGKILL; not
1224	releasing`, and `done` stays open (§6.3); that is a box-level incident,
1225	not a lease release.
1226
1227	An `InvocationID` that is neither the proc's own nor empty means someone
1228	restarted the unit outside the activator, which `Restart=no` does not
1229	forbid. The goroutine logs `unit restarted outside the activator` and
1230	**`POST /stop`s the unit itself**, then keeps waiting: `done` closes
1231	only when the unit is terminal and its cgroup is empty, whichever
1232	invocation the processes belonged to (codex R2-5 disposition). The
1233	agent owns these units and any other invocation is an anomaly to end,
1234	not to adopt; the invocation id decides the event line's wording and
1235	nothing about whether a writer is gone.
1236
1237	One exception, after the activator itself has asked for the stop
1238	(`Stop` was called, or a failed health check stopped a fresh unit): a
1239	unit that then reports an invocation that is not the proc's own, or
1240	none at all while not terminal, has been replaced. systemd starts a new
1241	invocation only once the previous stop job completed, which is its own
1242	termination proof for the old one, so the goroutine emits `instance
1243	exited (status unknown: replaced after stop)` and closes `done` without
1244	the cgroup gate (the cgroup now belongs to the new invocation and would
1245	never read empty). The goroutine does not stop the replacement: a stop
1246	it did not request is not its anomaly to end, and a discarded proc
1247	must never be able to stop a later, legitimate wake (plan 3, task 4
1248	review).
1249
1250	**The seed's outcome comes from systemd, always.** The seed unit
1251	carries `RemainAfterExit=yes` (§4.2), so it stays loaded, with its
1252	`Result` and `ExecMainStatus`, until the control service reads them and
1253	stops it (§5.3). `/seed` therefore always returns a complete result and
1254	the seed's exit record is, like every other record, wording only.
1255
1256	`sweepOnce`'s existing dead-child check (`activator.go:1105-1122`) sees a
1257	flag, never a blocking call. `Restart=no` in the template is what makes
1258	this sound: systemd never restarts the unit behind the activator's back;
1259	a failed unit stays `failed` until the agent's next `start`, which clears
1260	it.
1261
1262	### 6.6 cgroups, tiers and the sampler (codex R1-8, R1-9, R2-13)
1263
1264	`internal/cgroup` today: `Manager.Start` (`cgroup.go:242-285`) clones the
1265	child into `carlos-exec@<host>` under the edge unit's delegated subtree,
1266	writes `memory.max`/`memory.high`/`memory.swap.max` from the tier and
1267	`memory.oom.group=1`, and the sampler reads the subtree as a scan root
1268	(exec-child accounting design §4). `CleanupHost` (`:398-403`) removes the
1269	directory.
1270
1271	Under units, systemd owns the cgroup. So:
1272
1273	- **Tier** travels in `POST /start` / `POST /seed` and lands on the host
1274	slice as the three properties of §4.3. The `Tier func(registry.Route)
1275	cgroup.Limit` seam and its box wiring (`stagedManifestOfLiveVersion`)
1276	are unchanged; only the consumer changes. The agent logs
1277	`instance: <host>: memory.max=…` on every start as it does today. Seed,
1278	instance and sidecar share the one cap.
1279	- **Per-host sampling** gains
1280	`/sys/fs/cgroup/carlos.slice/carlos-instances.slice/` as a scan root
1281	and reads each `carlos-instances-<esc>.slice/memory.current` (the
1282	slice's charge is the family's total, so no per-role summation is
1283	needed). It **publishes under the existing key**
1284	`activator.ExecName(host)` (`carlos-exec@<host>`), because that is the
1285	key `internal/console/metricsview.go:728-762` recognises and everything
1286	else is discarded at the app boundary there. Flat hosts, for this one
1287	release, are still sampled from the delegated subtree as today; the
1288	two roots never both hold a given host.
1289	- **The edge's own reading.** Platform-unit discovery scans
1290	`/sys/fs/cgroup/system.slice` (`internal/host/metrics_probes.go:49`),
1291	where `carlos-edge.service` no longer lives once it moves into
1292	`carlos.slice`, and the accounting design defines the edge's reading
1293	as the hierarchical total including its children. So discovery adds
1294	`/sys/fs/cgroup/carlos.slice` as a root for the edge unit, and the
1295	value published under `carlos-edge.service` becomes **`carlos.slice`'s
1296	`memory.current`**: the agent plus every instance, which is what the
1297	key has always meant (R2-13). The edge unit's own leaf is published
1298	additionally under `carlos-edge.service/self` for the box view; the
1299	platform graph is continuous across the roll.
1300	- **`Manager.Start`, `CleanupHost`, the clone3 path and the delegation
1301	dance** stay in use for flat hosts during this release and are deleted
1302	with the flat spawn path in the next (§8.6). **`Delegate=` on the edge
1303	unit** goes with them.
1304
1305	### 6.7 Agent start and restart; the sidecar restart (codex R1-7, R2-6)
1306
1307	Today an agent restart kills every child (`Pdeathsig`, the cgroup tree
1308	dies with the unit). `BindsTo=carlos-edge.service` reproduces the stop
1309	half of that for a `systemctl restart carlos-edge` and does not propagate
1310	the start. For a crash, or a stop that did not propagate, the agent on
1311	start, before anything else:
1312
1313	1. `POST /sweep` (§5.3). The control service, as root, enumerates every
1314	`carlos-instance@*`, `carlos-instance-seed@*` and
1315	`carlos-instance-sidecar@*` unit and every pending or running job on
1316	them, `systemctl stop`s each (which also cancels a queued start),
1317	waits for each to reach a terminal state **and** for its cgroup to be
1318	empty (§6.5's rule), and returns the list. The sweep is keyed on the
1319	unit names systemd reports, not on hosts, so it covers a unit whose
1320	identifier is hashed and whose registry row is gone: the agent could
1321	not name such a unit through `/stop`, which takes a host (codex
1322	R7-1).
1323	2. The agent removes stale exit records under every run directory.
1324
1325	Every instance is therefore asleep, and no job is queued, when the
1326	activator's state machine begins, which is the invariant the activator
1327	holds today. Adopting a running unit instead (re-acquiring its lease,
1328	re-attaching replication) is a later improvement with its own review; it
1329	is not needed for correctness.
1330
1331	**The first `activator.go` change.** `sweepOnce` restarts a dead
1332	sidecar by calling `StartSidecar` under `a.mu`
1333	(`activator.go:1165-1200`), justified there by "an exec is quick". A
1334	unit start is an HTTP call and a `systemctl start`, and a stall there
1335	would hold every other host's wake and stop. So the sidecar restart
1336	becomes: under `a.mu`, mark `e.sidecarStarting = true`, capture the
1337	entry's wake generation, and schedule the backoff exactly as now; then
1338	in a goroutine, `StartSidecar` with its 30s deadline; then re-take
1339	`a.mu`, and if the entry is still awake with the same generation,
1340	install the proc as today, otherwise `Stop` it. The lock still makes a
1341	restart impossible to race with a stop, because the install step checks
1342	the state under the same lock the stop flips it under, and a stop that
1343	arrives during the start finds `sidecarStarting` and waits for the
1344	install step to see it. One test drives a stalled sidecar start while
1345	another host wakes and stops.
1346
1347	### 6.8 The wake runs on the committed row, not on a copy (codex R8-1)
1348
1349	The edge hands the activator the route it proxied for, and the
1350	activator copies it into the entry (`activator.go:481`); `doWake` then
1351	restores, seeds, spawns and replicates on that copy (`:806, :870`). The
1352	copy can be older than the registry: for the interval between a layout
1353	pass's row CAS and the edge's reload, the edge still holds the flat
1354	paths, and a request in that interval would carry a flat route to a
1355	host the registry already calls `nested`. On that copy the exec backend
1356	would classify the host `flat`, restore a second database at the
1357	vacated flat path, and run `childProc`, and `RetireExcept`
1358	(`activator.go:1260`) would keep the entry because its host name did
1359	not change.
1360
1361	**The third `activator.go` change**: at the top of `doWake`, before
1362	`Acquire`, the activator **re-reads the row** (`reg.Get(host)`, one
1363	indexed read) and uses that row for the whole wake, restore through
1364	replication, storing it back into the entry. If the row's `DBPath` or
1365	`Socket` differ from the copy the edge supplied, the wake logs
1366	`route refreshed: <old> → <new>` and proceeds on the fresh one; if the
1367	host has no row, or the row is `fenced` or `conflict`, the wake refuses
1368	as §8.1 says. The exec backend's classification (§8.1) therefore always
1369	sees the committed row. One test delivers a saved flat route for a
1370	migrated host and asserts no flat artifact is created and no
1371	`childProc` starts.
1372
1373	### 6.9 What does not change
1374
1375	`Acquire`, `Restore` (including the `.corrupt` quarantine and
1376	`PurgeLocalDBArtifacts`, all keyed off `dbPath`), `lease.Replicate` (the
1377	meta dir is derived from `dbPath`, `lease.go:286`, so it lands inside the
1378	instance directory), `NextDue`, the schedule runner's token read, the
1379	control socket's purge and schedule mounts, `Bounce`, the sweeper's
1380	hibernation policy, residence moves (`movespass.go`, once it reads the
1381	DB path from the row and honours the layout journal, §8.2 and §9),
1382	`carlos restart` in all three scopes, `X-Carlos-Version`, the caps pass,
1383	config-ack. All of these see a path change and nothing else.
1384
1385	---
1386
1387	## 7. Config delivery
1388
1389	`SyncExec` (`internal/appconfig/syncexec.go`) writes a nested exec
1390	route's bundle to `InstanceDir/<host>.config.env` instead of
1391	`<DataDir>/<host>.config.env`, through the `O_EXCL`-temp-and-rename
1392	helper of §3.2 rather than `os.WriteFile` on a predictable `.tmp` name
1393	(`syncexec.go:921-926`; R1-3), and a flat route's exactly as today. Its
1394	ownership rule ("sync owns `*.config.env` outright, never touches
1395	`<host>.env` or `<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged
1396	and, for a nested host, trivially true per directory. `reapConfigEnv`
1397	(`:977-1024`) walks `<DataDir>/instances/*/` for nested routes as well as
1398	the flat `DataDir` for flat and addr routes, and the bounce-on-change
1399	path is unchanged (the agent bounces the entry; the next wake reads the
1400	new file).
1401
1402	**Opted-in TCP-addr routes stay flat.** Their file is read by the app's
1403	own supervisor at `/data/<host>.config.env` (titogo's
1404	`EnvironmentFile=-/data/%i.config.env`), and moving it would break every
1405	such drop-in. Isolated instances cannot see it any more, which is the
1406	read #127 opens with. #127's request, root-rendered delivery into
1407	`/etc/carlos/apps` for addr routes so that the file is not readable by
1408	the agent either, is a separate, smaller change and stays open (§11).
1409
1410	`reapConfigEnv`'s host-`x.config` ambiguity guard and `SyncExec`'s
1411	collision refusal (`syncexec.go:389-395`) stay for the flat directory,
1412	where they still apply, and the migration's ownership preflight (§8.2)
1413	is derived from the same rule.
1414
1415	---
1416
1417	## 8. Two layouts, one release, and the verb that moves between them
1418
1419	### 8.1 Why nothing moves on upgrade, and nothing is minted nested (codex R2-9, R3-6)
1420
1421	`carlos system apply` is run by the **installed** binary: it stages the
1422	candidate, restarts the edge, gates it, and on a failed gate restores the
1423	previous files and restarts the edge again (`systemapply.go:360-372,
1424	738-748`); `carlos-system-recover.service` runs `carlos.prev` when a
1425	journal exists (`units/carlos-system-recover.service:17`); and a channel
1426	rollback from the console (`system.go:466`) is an ordinary downgrade
1427	install. In every one of those paths the code that runs after the
1428	downgrade is the *old* binary, which knows one layout. So:
1429
1430	**This release changes nothing about a host on upgrade, and mints
1431	nothing nested.** `routeFor` (`reconcile.go:299-301`, whose
1432	`DerivesPaths` predicate covers unflagged socket instances too,
1433	`provision.go:280`) and `carlos add` (`cli.go:118-136`) keep the flat
1434	defaults; a provisioning pass that runs during the candidate's gate
1435	creates a flat row the old binary serves (R3-6). Unflagged socket routes,
1436	whose supervisor owns their paths, keep flat paths in this release and
1437	every later one. The only writer of a nested path into a row is the
1438	layout pass, and it runs only on a recorded fleet intent.
1439
1440	The exec backend classifies each activator-managed route on every
1441	`Start`:
1442
1443	| Row | Classification | Spawn path |
1444	|---|---|---|
1445	| `DBPath == DefaultDBPath` (flat), `Socket == DefaultSocket` (flat) | `flat` | today's `childProc`, unchanged, unisolated |
1446	| `DBPath == NestedDBPath`, `Socket == NestedSocket` | `nested` | `unitProc` (§6) |
1447	| a journal exists in `/data/layout/<host>.json` | `fenced` | refuses to wake; reason from the journal |
1448	| another row on the box has the same `InstanceID` | `conflict` | refuses to wake; both hosts named (§3.1) |
1449	| anything else | `foreign` | today's `childProc`; reported for the operator (§8.3) |
1450
1451	A fleet rolled to this release behaves byte-for-byte as before until an
1452	operator records an intent. A rollback before that is a rollback of
1453	nothing. The classification is published per route in the status
1454	snapshot (§8.4) and is what every other consumer keys on.
1455
1456	### 8.2 `carlos system layout migrate` and `revert` (codex R1-11, R1-12, R2-1, R2-7, R2-8, R2-9, R3-1)
1457
1458	The verb has the shape of `carlos restart` (#113): the console writes
1459	`control/layout.json` (`{id, requested_at, mode: nested|flat, scope}`,
1460	the newest entry of an append log like `restart-scope.json`), bumps the
1461	serial, and every box in scope converges to it. Scope is the fleet, or
1462	`--box <label>` for one box, which is how a fleet migrates one box at a
1463	time with the console showing each box's rows turning `nested`. **The
1464	recorded mode is the fleet's intent, and boxes converge to it on every
1465	serial move**, not once: a row that becomes eligible later (a route
1466	flagged for hibernation after the fleet went nested, a freshly
1467	provisioned flat row) is migrated on the next pass when it is asleep. An
1468	old box never reads the object; an old console answers 404.
1469
1470	On a box, the layout pass runs in the agent (uid `carlos`, no root
1471	needed) and, per host in scope, in this order:
1472
1473	1. **Eligibility is the activator's own predicate.** `Kind == "instance"`,
1474	`Hibernate`, `Backing == ""`, `Tunnel == ""`, `Socket != ""`
1475	(`activator.go:1379-1384`, `edge.go:630-638`). An unflagged instance
1476	route with a socket is an externally supervised process (titogo's
1477	router family is the live example); its files and rows are **not
1478	touched**, it is not isolated by this change, and status reports it
1479	`external`. Under mode `nested`, `flat` rows migrate; under mode
1480	`flat`, `nested` rows revert; `fenced` rows recover first; `foreign`
1481	rows are reported and skipped.
1482	2. **Quiesce.** The pass asks the activator to stop the host
1483	(`Bounce`-class stop with replication closed and the lease released,
1484	the same sequence `stop()` runs, including §6.3's refusal to release
1485	over an unproven stop) and to refuse wakes for it until the pass is
1486	done. Nothing is writing.
1487	3. **Plan, in memory, from the row and the host name only, and only
1488	for files named after the host.** Each source follows its existing
1489	contract (codex R4-3): the database is the row's `DBPath`, and
1490	`-wal`, `-shm`, the litestream shadow dir and `.corrupt` are derived
1491	from it (`backend_exec.go:273, 1041-1057`, `lease.go:286`); the three
1492	env files are `DataDir`-relative regardless of `DBPath`
1493	(`backend_exec.go:785-794, 865-868`). The socket is the row's
1494	`Socket`. And every source must be **a file named `<host>` plus a
1495	platform suffix, directly under `DataDir` or `SocketDir`**: the row's
1496	`DBPath` must be `<DataDir>/<host>.db` and its `Socket`
1497	`<SocketDir>/<host>.sock`, which is what `flat` means (§8.1). A row
1498	naming any other path is `foreign` (§8.3) and the pass does not touch
1499	it, because during the transition the registry is writable by every
1500	flat tenant (same uid, §1.3) and a row is therefore not a trustworthy
1501	instruction to move an arbitrary file: a tenant that could point its
1502	`DBPath` at `/data/console/console.db` and have the pass move it
1503	into the tenant's directory would have the platform's own database.
1504	Destinations are the §3.1 helpers. Nothing is read from the instance
1505	directory to build the plan (§3.3).
1506	4. **Ownership preflight** (R2-8). For every flat source name, compute
1507	every route on the box that could own that name under the flat
1508	rules: `<host>.config.env` is `host`'s sync-owned bundle **unless** a
1509	route `<host>.config` exists, in which case it is that route's
1510	instance env and is not moved with `host` (this is `SyncExec`'s own
1511	collision rule, `syncexec.go:389-395`). Reverting into a flat
1512	directory applies the same table in reverse: a name that would land
1513	on another route's file is a refusal, never an overwrite. A plan with
1514	any ambiguous name is refused for that host with the two owners
1515	named.
1516	5. **Destination check.** A destination `<host>.db` that already exists
1517	is a refusal (`two databases; refusing to choose`), never a choice.
1518	For the two env files, which an older binary running between a
1519	migration and a revert can have minted fresh at the flat path
1520	(`backend_exec.go:579`; R2-9 disposition), an existing destination is
1521	moved aside as `<name>.aside-<timestamp>` in the destination
1522	directory and the source takes its place: the source is the file the
1523	instance ran with for longer, the aside copy is kept, and the event
1524	line names both. `<host>.config.env` at the destination is simply
1525	replaced: sync re-renders it on its next pass regardless.
1526	6. **Journal.** Write `/data/layout/<host>.json` (`O_EXCL`): `{host,
1527	mode, from: {db, socket}, to: {db, socket}, artifacts: [{name,
1528	present}], phase: "files"}`. Presence is recorded so recovery knows
1529	an absent optional artifact (a sidecar env, a quarantine) was absent,
1530	not lost (R2-7). From this moment the host is `fenced`: `Start`,
1531	`Seed`, `Restore`, the residence-move pass, `SyncExec`'s writer and
1532	the litestream generator all refuse it, with the journal's contents
1533	as the reason. The `MoveCutting` path in `movespass.go:285-305`, which
1534	would otherwise treat an absent DB as "nothing to checkpoint" and
1535	complete a flip, is the case the fence exists for (R1-12).
1536	7. **Files.** `os.Rename` each artifact recorded present. Same
1537	filesystem, same owner, atomic per file, no root. Then journal
1538	`phase: "row"`.
1539	8. **Row.** `registry.SetInstancePaths(host, from.db, from.socket,
1540	to.db, to.socket)`: one transaction, compare-and-swap on both columns,
1541	a new sibling of `SetDBPath` (`registry.go:1057-1075`). Then journal
1542	`phase: "done"`.
1543	9. Remove the journal; log `layout: <host>: flat → nested` (or the
1544	reverse); **reload routing synchronously** (the agent owns the edge
1545	and calls its `Reload()`, `edge.go:449-451`, and the activator's
1546	`RetireExcept`/route refresh, rather than waiting for the 2s poll);
1547	only then release the host to the activator. A request that copied
1548	the old route before that reload is caught by §6.8 (codex R8-1).
1549
1550	**Recovery is the journal, and it is exact, and it is checked** (R2-7,
1551	R3-1, R4 disposition). On agent start and at the top of every layout
1552	pass, for each journal, the check is against **both endpoints**, because
1553	the row is at `from` before step 8 and at `to` after it, and it is
1554	derived from **the journal's own direction** (codex R5 disposition): for
1555	`mode: nested`, `from` must be the host-named flat paths of step 3 and
1556	`to` the nested paths the §3.1 helpers derive; for `mode: flat` the two
1557	are swapped, artifact sources included. The row must currently equal
1558	either `from` or `to`.
1559	A journal that fails any of those is refused (`layout: <host>: journal
1560	does not match row; not acting`), the host stays `fenced`, and status
1561	says so. **A journal that passes is completed to its own endpoint even
1562	if the fleet's intent has since changed**: a transaction runs to its
1563	recorded end, and the host is then converged to the current intent in
1564	the same pass, before it is released to the activator, so a forged or
1565	stale journal can at most add one round trip a host never runs
1566	unisolated through. Otherwise:
1567
1568	- `phase: files`: for each artifact recorded present: at destination →
1569	done; at source → rename; at neither → error, fence stays, status
1570	says which file. Artifacts recorded absent are skipped. Then continue
1571	from step 7's end.
1572	- `phase: row`: if the row already equals `to` → the CAS committed
1573	before the crash; continue. If it equals `from` → CAS now. Anything
1574	else → error, fence stays, both values in status.
1575	- `phase: done`: remove the journal.
1576
1577	A second clean agent start finds `nested` rows and no journals and does
1578	nothing (the classification table runs before any legacy-default test).
1579
1580	**Revert** is the same pass under mode `flat`, the same journal, the
1581	same fence, the same preflight in reverse. It exists so that a fleet
1582	can be taken back to a layout the previous release serves **before**
1583	that release is installed: the operator's rollback order is `layout
1584	revert` (this release, verified `flat` in status) then `system rollback`.
1585	**That order is enforced, not advised** (codex R5-2). A downgrade
1586	removes this release's unit files: `installUnits` runs `systemctl
1587	disable --now` on every unit the candidate lacks (`systemapply.go:522`),
1588	so the bind mounts go with the binary, `/data/instances/<iid>/` becomes
1589	an empty mountpoint directory, and the old backend, honouring the nested
1590	`rt.DBPath` (`backend_exec.go:220-225`), would find no database and
1591	restore a second copy into the uncovered directory, which the next
1592	mount would hide. So:
1593
1594	- **This release's `carlos system apply` refuses to install a candidate
1595	that predates the layout** (its version is below the release that
1596	carries the mount units) unless reversal is **complete**: **no row on
1597	the box, whatever its flags**, has a `DBPath` or `Socket` under
1598	`/data/instances` or `/run/carlos/instances`, no row is `fenced` or
1599	`conflict`, and no journal exists. The check is by path shape and not
1600	by activator eligibility, because a row can leave management while
1601	keeping nested paths: `SetHibernate(false)` (`registry.go:1565`),
1602	including when the reconciler applies a record's flag
1603	(`reconcile.go:828`), does not touch them (codex R7 disposition of
1604	R5-2). `layout_mode` is not consulted either, because it records
1605	intent, and a revert refused at quiescence or at the destination check
1606	leaves a nested row under a flat intent. The reason names the rows;
1607	the installer is this binary for every downgrade from here, because a
1608	downgrade is an install performed by the installed binary (§8.1).
1609	- **Unflagging a nested row is refused.** `carlos route --hibernate=false`
1610	and the reconciler's flag application both refuse to clear `Hibernate`
1611	on a row whose paths are nested, with `layout revert first` as the
1612	reason (the reconciler logs it as it logs shape drift,
1613	`reconcile.go:873-881`), so a nested row is always activator-managed
1614	and the revert pass can always quiesce it. The classification table
1615	(§8.1) therefore never meets a nested row it cannot own.
1616	- **The console's rollback door** (`system.go:466`) refuses a channel
1617	rollback to a pre-layout release while any box in the fleet reports
1618	any `nested`, `fenced` or `conflict` row, or no layout field at all,
1619	and names the boxes and rows.
1620	- A failed-gate rollback of the upgrade *to* this release runs the old
1621	binary, but at that moment nothing is nested (§8.1). A failed-gate
1622	rollback of a later upgrade runs this binary, which knows the layout.
1623	- **Recovery for a layout that was unmounted anyway** (an operator
1624	removing units by hand): at start the control service (root)
1625	inspects the underlying mountpoint directories in a private mount
1626	namespace (`unshare -m`, `umount` the bind there, list) and, if
1627	`/data/instances` or `/run/carlos/instances` holds anything under the
1628	mount, records the verdict `mountpoint held files while unmounted`
1629	and `GET /ready` refuses nested wakes until an operator has moved or
1630	removed them; the layout pass's destination check (step 5) then sees
1631	whatever was moved into the raw directory. Nothing is chosen
1632	silently; the databases are never two.
1633
1634	The database therefore has one history in every path that is allowed
1635	to happen, and the two paths that are not allowed are refused with the
1636	revert named as the fix.
1637
1638	### 8.3 `foreign` rows
1639
1640	A row whose `DBPath` or `Socket` is neither the flat nor the nested
1641	default was set by hand (`carlos add --db`; `carlos route` has no `--db`
1642	flag, `cli.go:469-475`, so it was set at add time). The pass does not
1643	move it, for the reason step 3 gives: while flat tenants share the
1644	agent's uid, a row is not a trustworthy instruction to move an arbitrary
1645	file, and the only files the pass will move are the ones named after
1646	the host at the platform's own paths. A `foreign` row is reported with
1647	both values, keeps running on today's flat path, and `carlos add` on
1648	the box, re-adding the route with the defaults, is the repair: box-local
1649	because the row was made box-locally, and because a remote verb that
1650	moved operator-named files would be the primitive this section refuses
1651	to build (R1-18, accepted as the residual). A console-provisioned row
1652	(`reconcile.go:300-301`) is never `foreign`.
1653
1654	### 8.4 Status, through the API (codex R1-18)
1655
1656	`internal/status/snapshot.go` gains a per-route `layout` field with one
1657	of `flat`, `nested`, `fenced` (with the journal's phase and reason),
1658	`conflict` (with the other host), `foreign` (with both values),
1659	`external`, and a box-level
1660	`layout_mode` (`flat`, `nested`, or `none` before any intent). `carlos
1661	status --layout` (beside `--instances`, under the same operator gate;
1662	R14) and the console's instance page render them; a box that predates the field shows `unknown (box predates
1663	layout reporting)`. It is the pre-roll inventory, the progress view
1664	during a migration, the member-visible proof that an instance is
1665	isolated (`nested`), and the reason a `fenced` host does not wake.
1666	`docs/site/cli.md` names the new verbs and flags (the docsite gate fails
1667	otherwise).
1668
1669	### 8.5 Gate: no isolation, no nested tenants (codex R1-15, R2-14, R3-4)
1670
1671	A box that cannot establish the namespace **does not start nested
1672	tenants**. There is no fallback unit, no "proceed without
1673	`PrivatePIDs=`". Running the fleet with the same-uid `/proc/<pid>/root`
1674	traversal open is the defect, not a degraded mode.
1675
1676	The gate is observed behaviour, not a version string. At start, and on
1677	`daemon-reload`, the control service (root) runs
1678	`carlos-instance-probe.service`, a oneshot with the exact namespace and
1679	`IPAddressDeny=` block of §4.1 whose `ExecStart` is `carlos
1680	instance-probe`. From inside it asserts, and reports each as a separate
1681	verdict line:
1682
1683	- `/data` and `/run/carlos` list nothing but the probe's own bound
1684	directories, and `/var/cache` and `/var/lib` list nothing at all;
1685	- `access(W_OK)` on the edge's own delegated `cgroup.procs`
1686	(`/sys/fs/cgroup/carlos.slice/carlos-edge.service/cgroup.procs`, chowned
1687	to carlos by `Delegate=` on `carlos-edge.service`) answers `EROFS`,
1688	never `EACCES` (the kernel checks DAC before the mount flag, so `EACCES`
1689	there means the file is not delegated, not a pass) or `ENOENT` (the edge
1690	unit is not loaded); and `/proc/self/mountinfo` shows `/sys/fs/cgroup`
1691	itself carrying `ro`, so the `EROFS` answer is the mount flag's doing
1692	and not some other cause;
1693	- `/etc/carlos` cannot be opened (`EACCES` or `ENOENT`, never a
1694	listing), and the fixture `/etc/carlos/probe-secret` cannot be read.
1695	The fixture is **not** written by the control service, whose
1696	`ProtectSystem=strict` makes `/etc` read-only to it (codex R11-1); it
1697	is a tmpfiles line (§3.2, `f /etc/carlos/probe-secret 0640 root
1698	carlos - probe`) applied by bootstrap and by `update-platform-binary.sh`'s
1699	`systemd-tmpfiles --create`, before any probe runs. The control
1700	service, before it accepts a probe verdict, `stat`s the fixture and
1701	requires `root:carlos 0640`; a missing or differently-owned fixture is
1702	itself a `not ready` verdict line (`probe fixture absent`), never a
1703	vacuous pass;
1704	- `/proc` lists only itself, and its own pid is 1;
1705	- a symlink it creates in its directory is refused with `ELOOP`, and
1706	`/proc/self/mountinfo` shows `nosymfollow` on both bound directories;
1707	- **IMDS, for a packet filter** (R3-4). `IPAddressDeny=` is a cgroup BPF
1708	egress filter; it drops packets, and a TCP `connect()` whose SYN is
1709	dropped retries and times out rather than returning `EPERM`
1710	(`tcp_connect`, `net/ipv4/tcp_output.c`). So the probe sends a UDP
1711	datagram, where the filter's verdict comes straight back from
1712	`sendmsg`: `sendto(169.254.169.254:80)` must fail with `EPERM`;
1713	`sendto([fd00:ec2::254]:80)` must fail with `EPERM` (`v6: enforced`)
1714	or with `ENETUNREACH` (`v6: unverified, no route to the address`):
1715	the kernel's own answer for that exact destination, so no route query
1716	is needed and none is made (codex R4-4; the service's
1717	`RestrictAddressFamilies=AF_UNIX` stands). As a control,
1718	`sendto(127.0.0.1:9)` must succeed (the network stack is live and the
1719	filter, not the stack, said no). A 2s TCP `connect()` to the IPv4
1720	address must not succeed. Any success on a metadata address, or any
1721	errno other than those named, is `not enforced`.
1722
1723	The service records the verdict; `GET /ready` returns it; the agent
1724	refuses every nested wake with the verdict's failing line until it is
1725	204. `v6: unverified` does not block readiness on its own, because the
1726	boxes have no IPv6 route to that address today and a box with no route
1727	cannot reach it; it is **bounded and watched**: the probe re-runs every
1728	five minutes and on `daemon-reload`, the verdict is part of the status
1729	snapshot (§8.4), and the moment a v6 route appears and the answer is not
1730	`EPERM` the verdict flips to `not enforced` and nested wakes stop
1731	(codex R2-14 disposition). The probe cannot pass while an instance
1732	would not be isolated, because it runs under the same unit text (a test
1733	asserts the probe's namespace block equals the templates'), and it
1734	fails only on a line that names the property that is missing.
1735
1736	### 8.6 Rollout order
1737
1738	1. **Roll this release** to the dev fleet (`kass`, two boxes), boxes
1739	before console, with `update-platform-binary.sh` enabling the new
1740	units by name before it restarts the edge (§9). Every row stays
1741	`flat`; `layout_mode` is `none`; the probe verdict is `ready`; nothing
1742	else changes. Cold-boot one dev box and confirm the three mounts, the
1743	control socket and the probe come up (R2-2).
1744	2. **`carlos system layout migrate --box <dev box>`**, watch the rows
1745	turn `nested`, then the second box. The test that matters is from
1746	inside: the plan's verification task provisions a throwaway instance
1747	whose seed lists `/data` and `/run/carlos`, reads `/proc/1/status`,
1748	sends the three datagrams, and plants and follows a symlink, and
1749	reads the results back through `carlos logs`. Run `layout revert` on
1750	one box and `migrate` again: both directions are exercised before the
1751	flagship sees either.
1752	3. **Flagship**: roll the release region by region, console box last;
1753	then `layout migrate` one box at a time. The AU box has failed every
1754	gate since 2026-09-07 for an unrelated reason; do not read its failure
1755	as this change's.
1756	4. **carlos-tito-ie**: Tito's operator rolls it from Tito's console once
1757	the flagship has run a day, then migrates. The seven flagged instances
1758	move; the router family (unflagged) is reported `external` and left
1759	alone. Reply on #126 with the build and the check list above.
1760	5. **The next release** removes the flat spawn path for activator-managed
1761	hosts, points `DefaultDBPath`/`DefaultSocket` at the nested helpers
1762	for flagged routes (and keeps them flat for unflagged ones, whose
1763	supervisors own their paths), removes the spawn half of
1764	`internal/cgroup` and `Delegate=` on the edge, and refuses to start on
1765	a box whose `layout_mode` is not `nested`. Brought forward in plan 5
1766	(revision 16): on a box whose intent is `nested`, the reconciler and
1767	`carlos add` mint a fresh activator-managed row at the nested paths
1768	(`routeFor`'s `nested` argument), so a new tenant never runs flat
1769	there, not even for one pass. `DefaultDBPath`/`DefaultSocket` stay
1770	flat for every other caller until the next release.
1771
1772	**Nested by default (revision 17, Paul's decision 2026-10-06).** From
1773	the release that carries plan 5 (v0.25.0), nested is the default, not an
1774	opt-in. When `control/layout.json` is absent, the effective intent on a
1775	box is `nested` **if and only if the box is ready**: the deployment's
1776	default is nested (`CARLOS_LAYOUT_DEFAULT` in host.env is unset or
1777	`nested`; `flat` is the per-deployment opt-out, for a self-hoster whose
1778	boxes are not ready), the running build is at or above `LayoutFloor`
1779	when it can be placed (a build that evaluates the rule carries the pass
1780	by construction, so an unplaceable dev build passes and a mis-tagged
1781	release below the floor does not), and the nested gate of §8.5 answers:
1782	`instancefs.SelfCheck`, the three `nosymfollow` mounts, and the control
1783	service's probe verdict. A box that is not ready behaves exactly as
1784	before the default (flat, mints flat) and says why in its status
1785	report (`layout.default_blocked`); a box that is nested by default says
1786	so (`layout.default`), so `carlos status --layout` can tell a box that
1787	was migrated on purpose from one that moved on its own.
1788
1789	A recorded intent always wins: an explicit `flat` is the opt-out, an
1790	explicit `nested` behaves as today, and `--box` scoping is how an
1791	operator keeps box-by-box control. The default never forces: the awake
1792	grace, the holds and the quarantine of plan 4 apply as they do under a
1793	recorded intent. The console's guards need no new rule: the downgrade
1794	guard and the migrate guard judge what each box REPORTS (its rows and
1795	its running release), so a box that is nested by default is treated
1796	exactly like one under an explicit intent wherever it reports nested
1797	rows. `carlos add` cannot reach the gate from an operator's shell, so
1798	with no intent recorded it mints nested only when the box's registry
1799	already holds an activator-managed row at the nested paths; a fresh
1800	box's first hand-added row is minted flat and moved by the pass on the
1801	next serial move.
1802
1803	Rollout, restated: the flagship cuts v0.25.0 with this in, rolls it, and
1804	each box migrates on its own once its probe reports ready. An operator
1805	who wants box-by-box control records `carlos system layout migrate
1806	--box` (or `revert`) first; the recorded intent then governs.
1807
1808	Why a default and not a verb: the §1.2 list is what every flat tenant
1809	can reach, and a fleet that stays flat because nobody ran a verb is a
1810	fleet that stays exposed by omission. The gate is what makes the default
1811	safe to have: a box that cannot isolate does nothing, loudly.
1812
1813	**The roll script and the planner.** `update-platform-binary.sh:52-70`
1814	(pinfra) enables units by name; it gains `carlos-instance-control.socket`,
1815	the three mount units and the probe, before it restarts the edge, and
1816	`bootstrap.go`'s `systemctlPlan` (`:165-179`) starts `.socket` and
1817	`.mount` units it enables, which it does not today (R1-14).
1818	`ensureSystemUnits` (`systemunits.go:34-83`) checks the control socket
1819	is *active*, not just enabled. An agent that finds the channel absent
1820	refuses nested wakes with `instance control socket not listening`, which
1821	is what a forgotten enable looks like from `carlos logs`; flat hosts are
1822	unaffected, which is the point of the two-layout release.
1823
1824	---
1825
1826	## 9. Every consumer of the shape
1827
1828	The rule (AGENTS.md): the consumer list is everything that parses the
1829	shape, not everything that calls the API. From the sweep of 2026-09-11,
1830	extended by codex R1-9, R1-14, R1-17, R1-18, R2-8, R2-9, R2-10, R2-13,
1831	R3-1, R3-6:
1832
1833	**Path builders that move onto the helpers**
1834
1835	- `internal/provision/provision.go:280` (`DerivesPaths`, unchanged, but
1836	named: it is why unflagged routes mint flat), `:1011-1012`
1837	(`DefaultSocket`, `DefaultDBPath`, flat in this release), plus
1838	`InstanceDir`, `InstanceRunDir`, `NestedDBPath`, `NestedSocket`,
1839	`LogSocket`
1840	- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); `:744-750` (socket normalisation, unchanged); `:273, 1041-1057` (quarantine and artifact paths, the migration's source contract); every nested read through `instancefs`; the classification of §8.1; the path and `getpeername` checks on the health and control dials (§5.4)
1841	- `internal/edge/edge.go:2000-2014` (`unixProxy`: the path and `getpeername` checks on every dial, §5.4)
1842	- the layout pass's raw-directory renames (`/data/.instances`, §3.2): the one caller, pinned by a grep test
1843	- `internal/appconfig/syncexec.go:389-395, 465, 872, 921-926` (nested routes into the directory; flat and addr routes flat; `O_EXCL` temp); `:977-1024` `reapConfigEnv` walks both
1844	- `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path, by classification)
1845	- `internal/host/movespass.go:285-305` — rebuilds `<DataDir>/<host>.db` from the record; reads `rt.DBPath` from the row instead, and refuses a `fenced` host
1846	- `cli.go:118-136` (`carlos add` defaults, unchanged and named), `:157-159` (relative-path refusal), the new `carlos instances --layout` and `carlos system layout migrate|revert`; `docs/site/cli.md` names them
1847	- `internal/provision/reconcile.go:299-301` (`routeFor` mint, unchanged and named); `:461-464` and `:873-881` compare derived paths and log a `nested` row as drift for this one release (the message names the layout mode)
1848
1849	**Readers of `rt.Socket` (no change; the value changes per host)**
1850
1851	- `internal/edge/edge.go:536-538` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go`
1852
1853	**The socket consumers group** (R2-10): `internal/host/bootstrap.go:41-62`
1854	(`socketsGroup`, `tmpfilesConf` gains the lines of §3.2), the per-host
1855	run directory mode in `prepare`, and the #103 test that dials a migrated
1856	instance as an enrolled non-`carlos` uid
1857
1858	**Litestream and replication**
1859
1860	- `internal/lease/lease.go:286` — meta dir derived from `dbPath`; follows
1861	- `internal/ops/litestream.go:354-388` — the box-wide generator **discovers by glob and filters by registry claim**; it does not append claimed paths (R1-17). The glob gains `instances/*/*.db`, or nested non-hibernating databases are never replicated. Hibernating rows stay excluded at `:420-421` as today; a `fenced` host is excluded. Pinned in `ops_test.go` for both coverage and the `skipped` report
1862	- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine; ordered after `data-instances.mount`
1863
1864	**Metrics and status**
1865
1866	- `internal/host/metrics_probes.go:49` (discovery: the two new roots), `internal/host/metrics_loop.go:659` (the edge's published value becomes the slice total), `internal/console/metricsview.go:728-762` (the key allowlist: unchanged because the sampler publishes under `carlos-exec@<host>`)
1867	- `internal/status/snapshot.go` gains the per-route `layout` field and the box `layout_mode` of §8.4; `carlos status` and the console instance page render them
1868
1869	**The updater and the control plane**
1870
1871	- `internal/host/systemapply.go:360-372, 738-748`, `carlos-system-recover.service`, `system.go:466`: unchanged, by design (§8.1); the layout verbs are a separate console door and bucket object (`control/layout.json`), a separate box pass that runs on every serial move, and a separate mark in `restart-marks.json`'s sibling `layout-marks.json`
1872	- `internal/console`: the `POST /api/cli/system/layout` door (fleet-operator gate, fresh auth), `carlos system layout` in `system.go`
1873
1874	**Units and installers**
1875
1876	- `internal/host/units/` gains: three templates, two slices, three mounts, the probe, the control socket and service (§4, §5, §8.5); `carlos-edge.service` gains `Slice=`, `Requires=`/`After=` on the mounts, and drops its memory lines; `carlos-litestream.service` gains the `After=`
1877	- `internal/host/bootstrap.go:61` (tmpfiles lines), `:116-122` `enabledUnits` (control socket, mounts, probe), `:165-179` `systemctlPlan` (starts `.socket`/`.mount`)
1878	- `internal/host/systemunits.go:34-83` (active check on the socket)
1879	- `update-platform-binary.sh:52-70` (pinfra): enables the new units by name before restarting the edge
1880	- `harden-host.sh` (pinfra) writes hardening drop-ins for `$TENANT_UNITS`; the instance templates carry their own and must not be listed there
1881	- **The same-uid audit** (codex R9-1, R9-2): every unit in `internal/host/units/` that runs as `User=carlos` or delegates a `carlos`-owned tree, and every path it writes, is a path an isolated instance must not see. Today: `carlos-edge.service` (`/data`, `/run/carlos`, its delegated cgroup subtree), `carlos-litestream.service` (`/data`), `carlos-hibernate-verify.service` and `carlos-restore-verify.service` (`/var/cache/carlos-verify`). A test over `unitsFS` lists every `User=carlos` unit's `ReadWritePaths=`, `CacheDirectory=`, `StateDirectory=`, `RuntimeDirectory=` and `Delegate=` and asserts each resulting tree is covered by a `TemporaryFileSystem=`, `InaccessiblePaths=` or `ProtectControlGroups=` line in the templates, so a future unit that writes somewhere new fails the build until the templates hide it.
1882	**Reads count too** (codex R10-1): the same audit covers every file a `User=carlos` or `Group=carlos` unit *reads* that is not world-readable by design, which today is `/etc/carlos/relay-masters` (`root:carlos 0640`, `internal/relay/masters.go:101`) and the alternate path the relay design allows for it; the audit's second table lists each such file, its owner and mode, and the template line that hides it (`InaccessiblePaths=/etc/carlos`). A new credential input for a `carlos`-group reader is added to that table, or the test fails
1883
1884	**Tests that pin the flat layout** (§10): `internal/activator/activator_test.go` (20 fixture sites), `backend_exec_test.go` (4), `admintoken_test.go` (8, path-agnostic), `internal/appconfig/syncexec_test.go` (50), `internal/ops/ops_test.go` (52), `internal/provision/reconcile_test.go` (20), `internal/host/movespass_test.go` (1), `schedule_wiring_test.go` (1), `schedule_token_fence_test.go` (2), root `e2e_test.go` (24), `instances_e2e_test.go`, `config_e2e_test.go`, `deploy_e2e_test.go`. For this release they keep passing on the flat path; each gains a nested twin.
1885
1886	**Not consumers**: the console never reads box paths (`internal/console`
1887	has no `DBPath`/`DataDir` reader); `readDiskFrac` is whole-volume; the
1888	sentinel `.path` units name fixed files under `/data`, never a host.
1889
1890	---
1891
1892	## 10. Tests
1893
1894	Unit level, all runnable in CI (no systemd needed):
1895
1896	- `internal/provision`: `InstanceID` (identity below 41 bytes,
1897	truncation plus hash above, never longer than 40, stable), the nested
1898	helpers keyed by it, the flat defaults unchanged, `LogSocket`, the
1899	escape function against `systemd-escape` fixtures, and the socket
1900	address preflight: a host that fits today's flat paths but whose
1901	identifier-keyed paths would exceed 107 bytes cannot exist, pinned by
1902	computing the longest address for a 253-byte host; a short host and a
1903	hashed identifier can never be equal (the `_` prefix); two long hosts
1904	colliding in `InstanceID` classify as `conflict` and refuse to wake,
1905	including a host that was already `nested` when the second row
1906	arrived.
1907	- `instancefs`: refuses a symlink at every component, refuses `..`,
1908	`O_EXCL` temp plus rename, the mountinfo check for all three mounts.
1909	- `internal/edge` and `internal/activator` dials, with real socket
1910	pairs: a path under `/run/carlos-control/` is refused before
1911	`connect()`; a custom path elsewhere (`/run/titogo/example.sock`) is
1912	dialled as today; a symlink to another listener yields a `getpeername`
1913	mismatch and is closed before a byte is sent; a listener created by
1914	another process and passed over (the `carlos-app@.socket` shape) is
1915	accepted when its bound path is the one dialled.
1916	- `internal/host/systemapply`: a pre-layout candidate is refused while
1917	`layout_mode` is `nested` or a journal exists, and installed when the
1918	box is `flat`; the control service's start-time inspection of an
1919	underlying mountpoint that holds files (with a fake namespace runner).
1920	- `internal/console`: the rollback door refuses a pre-layout release
1921	while any box reports `nested`, `fenced` or no layout field, naming
1922	them.
1923	- `instance-control`: a connection whose peer pid is not the edge's
1924	`MainPID`, or whose uid is not `carlos`, is closed without a response;
1925	every 400 (bad host, bad role, bad limit); the unit and slice names it
1926	builds for a host with a dash; `set-property` (three properties) →
1927	`start` ordering with no `reset-failed`; `/reset` tolerating "not
1928	loaded"; `/seed` reading a retained result from an `active (exited)`
1929	and from a `failed` oneshot, waiting for the empty cgroup, then
1930	stopping it; `/ready` from the probe verdict; with a fake `systemctl`
1931	runner and a fake peer.
1932	- `internal/activator`: classification of the four row shapes;
1933	`prepare` mints the token for seed and run, writes `launch.<role>.json`
1934	with the deduplicated env and the exact argv (the existing
1935	env-composition tests move their assertion to the effective
1936	environment of an executed stub); the control client's per-verb
1937	deadlines (30s, 150s for seed); `unitProc.Stop` orders stop → done →
1938	close listeners and returns `ErrStillRunning` on expiry without
1939	closing; `stop()` skips `rep.Close` on `ErrStillRunning` and the
1940	sweeper retries; `Exited()` only after a terminal unit state **and** an
1941	empty cgroup, never on log EOF or on a record alone, with a populated
1942	cgroup holding `done` open; a forged exit record for a running unit
1943	ignored; a foreign invocation; the failed-health path posts `stop` and
1944	waits; seed timeout and cancellation stop-and-wait before cleanup; the
1945	startup sweep stops activating units, cancels jobs and waits for empty
1946	cgroups; the sidecar restart reserves under the lock and installs or
1947	stops after, with a stalled start while another host wakes and stops.
1948	The control channel is a seam (`Control interface { Start; Seed; Stop;
1949	Reset; Ready }`) with a fake; the exit observer takes a fake `show`, a
1950	fake run directory and a fake cgroup tree.
1951	- `instance-exec`: reads the file through `instancefs`, refuses schema
1952	≠ 1, refuses an `argv[0]` that is not the live link, execs.
1953	- `instance-exit`: writes the record with `O_EXCL` from the four
1954	variables into the run directory; refuses to overwrite.
1955	- `instance-probe`: each verdict line, including `EPERM` versus
1956	`ENETUNREACH` (with and without a v6 route) versus other errnos on the
1957	datagram sends, the loopback control, and the TCP connect bound, with
1958	a fake socket layer.
1959	- `logListener`: one connection per invocation, a second refused and
1960	logged, line cap, EOF ends collection without marking exit.
1961	- Layout pass: eligibility by the activator's predicate (flagged only;
1962	external, tunnel, unit-backed untouched); convergence on every serial
1963	move; a later-flagged row migrating on the next pass; `foreign` rows
1964	(a custom basename, a subdirectory, a path outside the trees, a row
1965	pointing at the console database) reported and never moved; quiesce
1966	before plan; the ownership preflight for `x`/`x.config` in both
1967	directions; DB destination collision refused, env destination moved
1968	aside; the rename performed through the raw `.instances` directory
1969	fds and never through the bind mount (a test with a real bind mount
1970	where `EXDEV` would otherwise fire, skipped where the CI host cannot
1971	mount); the journal's phases and every crash boundary (before files,
1972	mid-files, after files, after CAS, before journal removal) recovered
1973	exactly, with absent optional artifacts, with the row at either
1974	endpoint; a journal whose `to` is not the host's derived destination
1975	or whose `from` is not the host's flat paths refused; a journal
1976	completed to its own endpoint and the host then converged to a changed
1977	intent before release; a CAS conflict keeps the fence; a second clean
1978	start does nothing; every
1979	fenced consumer (`Start`, `Seed`, `Restore`, moves pass, `SyncExec`,
1980	litestream generator) refuses a fenced host; a partial rename followed
1981	by a `MoveCutting` pass does not complete the flip; revert restores the
1982	flat layout byte for byte, aside files included; a forged journal, a
1983	forged launch file and a forged exit record move nothing (§3.3).
1984	- Units: a test over `unitsFS` asserts the three templates' and the
1985	probe's namespace, hardening and coupling blocks are identical, that
1986	`PrivatePIDs=yes`, `TemporaryFileSystem=/data:ro`,
1987	`IPAddressDeny=169.254.169.254/32`, `Restart=no`,
1988	`BindsTo=carlos-edge.service`, `Slice=carlos-instances-%i.slice`,
1989	`OOMPolicy=kill`, an `ExecStopPost=` **without** `+`, and the `%I`
1990	paths are present, that the seed carries `TimeoutStartSec=120`, that
1991	the control socket is `SocketMode=0600 SocketUser=carlos` under
1992	`/run/carlos-control`, that all three mounts carry `nosymfollow` and
1993	`DefaultDependencies=no` with the ordering of §3.2, and that
1994	`carlos-edge.service` carries `Slice=carlos.slice` and no `MemoryMax=`;
1995	and, where `systemd-analyze` exists on the CI host, `systemd-analyze
1996	verify` over the rendered set.
1997	- `internal/ops`: the nested glob, coverage of a nested non-hibernating
1998	row, the `skipped` report, the fence.
1999	- `internal/host`: `systemctlPlan` starts sockets and mounts; the two
2000	sampler roots publish per-host under `carlos-exec@<host>` and the edge
2001	as the slice total, flat hosts still from the delegated subtree; the
2002	#103 consumer dial through the nested run directory.
2003	- `internal/status`: the `layout` field for each of the six states, the
2004	box mode, and the pre-field box.
2005	- `internal/console` and `system.go`: the layout door's gate and
2006	refusal order, the bucket object, the scope forms, convergence marks.
2007
2008	Box level, in the plan's verification task and in the rollout:
2009
2010	- Cold boot with the full unit set; `systemd-analyze verify`;
2011	`list-dependencies` shows the mounts before the edge.
2012	- Probe verdict `ready`; a deliberately broken probe (a drop-in removing
2013	`IPAddressDeny=`) turns it `not enforced` and blocks nested wakes.
2014	- Migrate one host; wake it; `systemctl status carlos-instance@<esc>`
2015	active in `carlos-instances-<esc>.slice` with the three memory
2016	properties; from inside via the throwaway instance's seed: `/data` and
2017	`/run/carlos` list one directory each, `/proc/1` is the app, the three
2018	datagrams answer as §8.5 says, a planted symlink `ELOOP`; `carlos logs`
2019	shows all of it for both instance and sidecar at once; a quick-exiting
2020	seed's status arrives; hibernate, unit gone, record consumed; `carlos
2021	restart` cycles it; `systemctl restart carlos-edge` stops it and does
2022	not start it; an enrolled `carlos-sockets` uid dials `app.sock` and
2023	cannot open the control socket; a flat tenant on the same box cannot
2024	get a response from `control.sock` directly or through its own
2025	redirected socket; `carlos instances --layout` says `nested`; the
2026	console's memory graphs for the app and for the platform are
2027	continuous across the migration; revert, and the previous release
2028	serves the host again.
2029
2030	---
2031
2032	## 11. Out of scope, and follow-ups
2033
2034	- **Per-instance uids (option B).** The nested layout is its
2035	prerequisite; the namespace closes every path the discussion names
2036	without it. If a later finding needs a uid boundary, it lands on this
2037	layout with a root chown in the control service. Not in this change.
2038	- **Externally supervised exec routes** (unflagged instance routes with
2039	a socket, §8.2 step 1): not isolated by this change; their supervisor
2040	owns their process and their flat paths. Whether the platform should
2041	offer them the same namespace is a discussion to open once this lands.
2042	- **Unit-backed console instances.** Not needed for isolation, and a
2043	hibernation rewrite. Whether `backing=unit` hibernation works on the
2044	fleet today (`UnitBackend` runs `systemctl` unprivileged, §1.4) is its
2045	own discussion; the control channel of §5 is deliberately not widened
2046	to `carlos-app@` (R1-19).
2047	- **#127**: root-rendered config for opted-in addr routes into
2048	`/etc/carlos/apps`. Smaller and independent; this change makes the file
2049	invisible to isolated instances but not to the agent.
2050	- **Adopting running units on agent start** instead of stopping them
2051	(§6.7).
2052	- **The next release** (§8.6 step 5): drop the flat spawn path and the
2053	spawn half of `internal/cgroup`; with it, the transition-only
2054	exposures §3.2 and §6.4 name (a flat tenant redirecting its own socket,
2055	a flat tenant writing to a log socket) end.
2056	- **Whole-family OOM across instance and sidecar** (§4.3): not
2057	reproduced; recorded as a deliberate difference.
2058	- **Discussion #1 (memory pressure)**: no new numbers here (§4.3); the
2059	slice shape makes a later change a unit-file edit, never a drop-in.
2060
2061	### 11.1 Hardening after the layout (#214 items 1-3, #178), and what it leaves
2062
2063	Added to the templates, the control service and the probe (each property
2064	has a probe line, so `ready` reflects it):
2065
2066	- **CPU and IO.** The control service writes the tier's `CPUWeight=`,
2067	`CPUQuota=` and `IOWeight=` beside its memory caps on the host slice
2068	(`cgroup.SharesOf`: 100/100%/100, children tier 100/200%/100). Nothing
2069	new on the wire: the shares follow from the memory tier.
2070	*Residual:* `IOWeight=` is enforced only by io.cost or BFQ; the probe
2071	says `unverified` on a box with neither, which is the default on an
2072	EBS `none` scheduler. Enabling io.cost is a box change, not made here.
2073	- **Disk.** No per-tenant quota: `/data` is a directory on the root ext4
2074	filesystem, and project quotas need `tune2fs -O project,quota` on an
2075	unmounted filesystem plus a `prjquota` mount; every writer is uid
2076	carlos, so reserved blocks cannot tell a tenant from the agent. The
2077	guard refuses rather than stops: below max(1 GiB, 10%) free, starts
2078	and seeds answer 507, and the control service logs a warning every
2079	30s while it lasts. Stopping the "biggest" tenant was tried and
2080	dropped in review: space held by unlinked-but-open files appears in
2081	no directory, so attribution can be gamed into stopping honest
2082	tenants, and a stop from the control service bypasses the activator's
2083	fence and holds. *Residual:* tenants already awake can keep writing
2084	past the floor until the disk is full. A dedicated `/data` volume
2085	with project quotas (or systemd's quota directives on top of them)
2086	is the real fix.
2087	- **The version store.** Each instance sees an empty `/opt/carlos` with
2088	`/opt/carlos/carlos` and its own build put back: the control service
2089	writes `/run/carlos-control/exe/<iid>/{app -> /opt/carlos/live/<host>,
2090	live/<host> -> ../app}` before every start, and the unit binds them at
2091	`/opt/carlos/app` and `/opt/carlos/live`. `carlos-app@` gets the same
2092	view by binding `/opt/carlos/live/%i` over an empty `/opt/carlos`.
2093	- **Egress.** `IPAddressDeny=` covers loopback, RFC 1918, CGNAT,
2094	link-local, ULA, NAT64, site-local, multicast and reserved ranges.
2095	An allow is per address, both ways and on every port, so no loopback
2096	resolver is ever opened. That includes resolved's 127.0.0.53/54
2097	stubs, because a tenant could bind the stub address and reach every
2098	wildcard listener through it. The namespace binds
2099	`/run/systemd/resolve/resolv.conf` (resolved's upstreams, when
2100	present) over `/etc/resolv.conf`. Before every probe, the control
2101	service re-opens the nameservers in that same file with
2102	`IPAddressAllow=` on `carlos-instances.slice`. Link-local resolvers
2103	are refused except 169.254.169.253: an allow carries no zone, so
2104	`fe80::1` would open that address on every link. A nameserver that
2105	is one of the box's own addresses is never opened either: the
2106	control service reads them from procfs (`/proc/net/fib_trie` LOCAL
2107	routes, `/proc/net/if_inet6`) as it writes the allow, closes every
2108	resolver if it cannot, and reports a refusal as a failing
2109	`resolver-local` line. The probe's `dns` and `resolver-ports` lines
2110	check from inside that each allowed resolver is reachable and does
2111	not answer on port 22. (The probe made the local-address check
2112	itself until v0.25.0, with `net.InterfaceAddrs`; that is netlink,
2113	which the unit's `RestrictAddressFamilies=` refuses, so no box was
2114	ever ready; #126.) *Residuals:* an allowed upstream is open on every port (it
2115	is a remote resolver, not the box), and a resolver change reaches
2116	running instances only at their next start. Loopback inside one instance is
2117	closed too, so a family that talks to itself over TCP loopback must
2118	use a unix socket in its own directory. Services the box exposes on a
2119	public interface address remain reachable, as from the internet.
2120	- **IPC (#178).** `PrivateIPC=yes` and a private `/dev/mqueue`. Abstract
2121	unix sockets are scoped with Landlock (`LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET`,
2122	ABI 6, Linux 6.12) by the trampoline before it execs; a kernel without
2123	it fails the probe and the trampoline refuses to exec. *Residual:* a
2124	sibling can still bind an abstract name first, which is a refusal of
2125	service to the owner, never a connection to it. A per-unit network
2126	namespace would close that too, at the cost of a veth or proxy for
2127	egress; not done.
2128
2129	---
2130
2131	## 12. Approaches considered
2132
2133	1. **Console-provisioned instances become unit-backed** (#126 option 1).
2134	Rejected: `UnitBackend` has no restore, seed, replication, hibernate-
2135	from-replica, sidecar or move; `provision.Record` cannot express a
2136	backing and the reconciler refuses one. A rewrite of self-serve
2137	hibernation, and the always-awake design already said so.
2138	2. **Per-instance uid from the agent** (#126 option 2 as written).
2139	Impossible without `CAP_SETUID` on the agent, which is root-equivalent
2140	for a compromised agent; and a uid alone does not hide a sibling's
2141	socket file or pid.
2142	3. **Isolated exec: the agent's instance, systemd's namespace** (this
2143	spec). Keeps every activator invariant, needs one root channel, and
2144	migrates with a rename.
2145
2146	Within 3: trampoline over `EnvironmentFile=` (§6.2, one env composer);
2147	three templates over one (§4.2, the instance is the host); the control
2148	service authenticating the agent's pid over polkit and over a token
2149	(§5.4, ruled with codex across two rounds); `nosymfollow` mounts over
2150	per-read symlink checks (§3.2, kernel-enforced and race-free where a
2151	check-then-open is neither); termination from unit state plus cgroup
2152	emptiness, with the unit's own record as decoration, over any record as
2153	proof (§6.5); and an operator verb recording a fleet intent that boxes
2154	converge to over an agent-start migration (§8.1, because the updater's
2155	rollback runs the old binary).
2156
2157	---
2158
2159	## 13. Review log
2160
2161	- **Round 1** (codex, GPT-6, `CODEX-FINDINGS-ISOLATED-EXEC.md`, 19
2162	findings, verdict request changes). All accepted:
2163	R1-1 control service, `nosymfollow` (§3.2, §5.4);
2164	R1-2 log sockets outside tenant mounts, per role (§3.1, §6.4);
2165	R1-3 `nosymfollow` plus `instancefs` and `O_EXCL` temps (§3.2, §7);
2166	R1-4 exit from unit state plus cgroup emptiness, never from EOF (§6.5);
2167	R1-5 seed timeout, stop verb, wait before cleanup (§4.2, §5.3, §6.2);
2168	R1-6 per-role log listeners (§6.4);
2169	R1-7 `BindsTo=` not `PartOf=`, sweep covers jobs (§4.1, §6.7);
2170	R1-8 per-host slice carries the tier for all three roles; whole-family OOM recorded as a difference after R3-3 (§4.3);
2171	R1-9 cgroup path under `carlos.slice`, key compatibility, `metrics_probes.go` (§6.6, §9);
2172	R1-10 one common wall in `carlos.slice`, no new numbers (§4.3);
2173	R1-11 eligibility is the activator's predicate (§8.2);
2174	R1-12 the journal fence and its consumers (§8.2);
2175	R1-13 explicit reverse migration; no "restore" claim (§8.2);
2176	R1-14 planner starts sockets and mounts; roll script; active check (§8.6);
2177	R1-15 no `PrivatePIDs=` fallback; behavioural gate and probe (§8.5);
2178	R1-16 env deduplication, effective-value test (§6.2);
2179	R1-17 nested glob is discovery, not reporting (§9);
2180	R1-18 `layout` status through the API and CLI; `docs/site/cli.md`; the box-local repair of a `foreign` row accepted as the residual, with the reason (§8.3, §8.4);
2181	R1-19 `UnitBackend` claim removed (§1.4, §11).
2182	- **Round 2** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R2.md`, 14 new
2183	findings, verdict request changes). All accepted:
2184	R2-1 journal outside tenant mounts, plan from row and host only (§3.3, §8.2);
2185	R2-2 mounts with `DefaultDependencies=no` and explicit ordering, cold-boot verification (§3.2);
2186	R2-3 no `reset-failed` in `/start` (§5.3);
2187	R2-4 termination not dependent on unit retention; `/seed` correlation (§5.3, §6.5);
2188	R2-5 cgroup emptiness before `done`; `stop()` does not release over `ErrStillRunning` (§6.3, §6.5);
2189	R2-6 sidecar restart reserved under the lock, started outside it with a deadline (§6.7);
2190	R2-7 journal records artifact presence and phase; committed-row recognition; classification before legacy test (§8.1, §8.2);
2191	R2-8 ownership preflight in both directions from `SyncExec`'s rule (§8.2);
2192	R2-9 two-layout release and operator verb; same-boot rollback behaviour stated; env aside rule (§8.1, §8.2);
2193	R2-10 `carlos-sockets` traversal preserved on the run directories (§1.1, §3.1);
2194	R2-11 `MemorySwapMax` on the host slice (§4.3);
2195	R2-12 seed prepares the admin token and merged env (§6.1);
2196	R2-13 edge discovered in `carlos.slice`, published as the slice total (§6.6);
2197	R2-14 probe distinguishes enforcement from unreachability (§8.5, reworked by R3-4).
2198	- **Round 3** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R3.md`, 7 new
2199	findings, verdict request changes). All accepted:
2200	R3-1 no token; the control service authenticates the edge's `MainPID`; every dial checks path and `getpeername` (the root-peer rule of revision 4 withdrawn by R4-1); `/run/carlos/logs` on a `nosymfollow` mount; journals checked against both endpoints (§3.2, §3.3, §5.4);
2201	R3-2 exit record from `ExecStopPost=` without `+`, advisory only, in the run directory (§4.1, §6.5);
2202	R3-3 no manual `memory.oom.group`; per-service families, difference recorded (§4.3);
2203	R3-4 UDP `EPERM` probe with a loopback control and a no-route rule for v6 (§8.5);
2204	R3-5 150s seed deadline (§5.3);
2205	R3-6 nothing minted nested in this release; unflagged routes flat by contract (§3.1, §8.1, §8.6);
2206	R3-7 the record lives where the agent can delete it (§6.5).
2207	- **Round 4** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R4.md`, 4 new
2208	findings plus partials, verdict request changes). All accepted:
2209	R4-1 root-peer rule withdrawn; path allowlist before the dial and `getpeername` equality after it (§5.4, §6.1);
2210	R4-2 `/data/.instances` bound onto `/data/instances`; the pass renames on one mount through raw directory fds (§3.1, §3.2);
2211	R4-3 sources follow each file's existing contract; only host-named files at the platform's paths move; everything else is `foreign` (§8.2, §8.3);
2212	R4-4 no route query; the v6 answer comes from `sendto` itself; the probe re-runs and the verdict is watched (§8.5);
2213	R2-4 `/seed` correlates by `--no-block` then `InvocationID`; a seed record may decide a seed's outcome because it can deceive only its author (§5.3, §6.5);
2214	R2-5 a foreign invocation is stopped and `done` waits for the cgroup (§6.5);
2215	R2-7 journals validated against both endpoints and completed to their own end before convergence (§8.2);
2216	R2-14 `v6: unverified` bounded by the five-minute re-run and reported in status (§8.5);
2217	R1-18 the `foreign` residual accepted with its reason (§8.3).
2218	- **Round 5** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R5.md`, 3 new
2219	findings plus 2 partials, verdict request changes). All accepted:
2220	R5-1 only the control endpoint is refused by path; custom sockets elsewhere dial as today (§5.4);
2221	R5-2 `system apply` and the console's rollback door refuse a pre-layout release while any box is nested; underlying mountpoints inspected at start (§8.2);
2222	R5-3 `InstanceID`, bounded to 40 bytes, keys every directory, socket and unit name; addresses preflighted against 107 bytes (§3.1, §4, §6.2);
2223	R2-4 the seed unit carries `RemainAfterExit=yes`, so its result is retained until read; the record is wording again (§4.2, §5.3, §6.5);
2224	R2-7 recovery endpoints derived from the journal's own direction; §3.3 reconciled with §8.2 (§3.3, §8.2).
2225	- **Round 6** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R6.md`, 1 new
2226	finding plus 2 partials, verdict request changes). All accepted:
2227	R6-1 hashed identifiers carry a `_` prefix no host can have, 64-bit hash; a collision is `conflict`, which refuses to wake and is never the unisolated fallback; the create door refuses a colliding host (§3.1, §8.1);
2228	R5-2 the downgrade guards require completed reversal (no `nested`, `fenced` or `conflict` row, no journal), not intent (§8.2);
2229	R5-3 the control service builds unit and slice names from `InstanceID(host)` (§5.3).
2230	- **Round 7** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R7.md`, 1 new
2231	finding plus 1 partial, verdict request changes). All accepted:
2232	R7-1 the startup sweep is `POST /sweep`, performed by the control service on the unit names systemd reports, so a hashed identifier without a row is still stopped (§5.3, §6.7);
2233	R5-2 the downgrade guard checks every row's path shape regardless of flags; unflagging a nested row is refused (§8.2);
2234	§10 aligned with the `conflict` rule.
2235	- **Round 8** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R8.md`, 1 new
2236	finding, every prior item resolved, verdict request changes). Accepted:
2237	R8-1 the wake re-reads the row before `Acquire` and runs on it; the layout pass reloads routing before releasing the fence (§6.8, §8.2).
2238	- **Round 9** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R9.md`, 2 new
2239	findings, every prior item resolved, verdict request changes). Both accepted:
2240	R9-1 `ProtectControlGroups=yes` in the templates and the probe; the probe writes `cgroup.kill` and expects `EROFS` (§4.1, §8.5);
2241	R9-2 `/var/cache` and `/var/lib` hidden; the same-uid audit rule and its test added to §9 (§4.1, §8.5, §9).
2242	- **Round 10** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R10.md`, 1 new
2243	finding, every prior item resolved, verdict request changes). Accepted:
2244	R10-1 `InaccessiblePaths=/etc/carlos` in the templates and the probe; the audit extends to credential inputs a `carlos`-group process reads (§4.1, §8.5, §9).
2245	- **Round 11** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R11.md`, 1 new
2246	finding, every prior item resolved, **verdict accept with
2247	corrections**). The correction is applied in this revision:
2248	R11-1 the probe fixture comes from tmpfiles at bootstrap, and the control service verifies its ownership and mode before accepting a verdict (§3.2, §8.5).
2249	- R12 (final review, plan 2): cgroup-ro targets the edge's delegated cgroup.procs; DAC precedes the mount flag so a root-owned file answers EACCES.
2250	- R13 (task 4 review, plan 3): after a stop the activator requested, a unit reporting a foreign or absent invocation proves the old invocation over without the cgroup gate (§6.5); the observer never stops an invocation it did not start once its own stop was requested.
2251	- R14 (plan 4, 2026-10-04; recorded in revision 16): the per-host layout view is `carlos status --layout`, beside `--instances` and under the same operator gate, not `carlos instances --layout`: the instances verb lists the local registry, and the layout report lives in each box's private status snapshot, which only the console reads (§8.4).
2252	- R17 (plan 5 follow-up, Paul's decision 2026-10-06; revision 17): nested is the default when no intent is recorded, gated on the box's readiness; `CARLOS_LAYOUT_DEFAULT=flat` is the deployment opt-out and a recorded `flat` intent the fleet opt-out (§8.6, "Nested by default").
2253	- R15 (first roll, 2026-09-12; fixed 2026-09-25): the probe on dev's boxes answered `ready: false` twice, and the review of the fix found a third gap. (a) `RestrictSUIDSGID=yes` makes `openat2` fail with ENOSYS on systemd 257, because systemd cannot read the mode inside `open_how` and so blocks the whole call; every `instancefs` open in the namespace is an `openat2`. The line is dropped from all four templates. What it guarded is covered another way: with no capabilities an instance can only make a set-id file owned by `carlos`, which gains nothing for another `carlos` process; `NoNewPrivileges=yes` voids it inside the namespace; and no other uid can reach one (the tenant trees are `nosuid`, and `PrivateTmp=`'s `/tmp` and `/var/tmp` sit under a root 0700 directory, `/var/tmp` being on the root filesystem and not `nosuid`). (b) `control-unreachable` checked `Lstat`, but `InaccessiblePaths=` leaves a mode 000 root-owned node at the path. The check is now what a tenant could do, a listing of the directory and a dial of the socket, and passes only when both are refused for permission or absence; a refused connection or a timeout proves the path reachable and fails. (c) `PrivateDevices=` bind-mounts the host's `/dev/shm`, so every instance, all uid `carlos`, shared one; each unit now gets its own tmpfs there, and the probe's `shm-private` check plants a file in the host's `/dev/shm` (the control service's tmpfiles fixture) and fails if it is visible. POSIX message queues and abstract unix sockets are still box-wide; they are a follow-up (§11). (d) 2026-10-04, plan 4: `carlos-edge.service` carried the same line, and the agent runs there: its nested `instancefs` opens and every rename of the layout pass are `openat2` calls, so the first `layout migrate` would have met ENOSYS on its first host. The line is dropped from the edge unit too. The argument in (a) carries: the edge's one capability, `CAP_NET_BIND_SERVICE`, does not bear on set-id files (that takes `CAP_CHOWN`, `CAP_FSETID` or `CAP_SETUID`), so anything the agent could mark set-id is owned by `carlos`, and the agent and its flat children run under `NoNewPrivileges=`. An `openat` fallback in `instancefs` was considered and rejected: it would be a second resolution path, exercised in production only when the hardening had regressed, and it would route around the regression rather than report it. Instead the agent runs `instancefs.SelfCheck` in its own unit at start, and the nested-wake gate (§8.5) asks it first, so a blocked `openat2` refuses nested wakes and the layout pass with the directive named.
2254	- R16 (plan 4 review, 2026-10-04): three gaps in the pass while flat tenants still share the agent's uid. (a) A tenant could make its own future directory under `/data/.instances` ahead of the pass and hard-link a sibling's database or `litestream.env` into it (its own uid's files, so `protected_hardlinks` allows it); `EnsureDirs` accepted the directory and the destination check looked only at the planned names. Now a migration refuses an instance directory holding any entry the plan does not name, refuses any file about to move that has more than one name (the shadow directory walked), and checks both again after the renames and before the row flips; a failure there keeps the journal at `files` and the host fenced. (b) Deferring an awake host (plan 4 amendment, "when it is asleep") was unbounded, so a tenant that never idled kept raw-path access to its nested siblings for good. The deferral is bounded: an hour of grace (`layout.DefaultAwakeGrace`), after which the pass stops the host as §8.2 step 2 always allowed and moves it; the serial leg re-runs on its own ten minutes after any deferral so the bound holds on a quiet fleet; the intent gains `force` for an operator who wants no grace; and `carlos status --layout` warns on every box with nested rows beside flat ones, because that box is not isolated yet. (c) Files moved aside at a destination (`<name>.aside-<unix>`) were kept inside the instance directory, readable by the tenant that gets the directory; they are kept under `/data/layout/aside` now, which no instance namespace contains. Still open: a flat tenant that is running while another host moves can plant between the check and the rename; the check is against what was planted before the pass arrived, and the closure is the bound in (b). (d) Re-check: (a)'s refusals were themselves escapable: a tenant could keep its own move refused with one syscall (`ln /data/B.db /data/.x`, or a file in its own future directory), and because the pass lifted its fence on a refusal the host woke flat again on its next request, for as long as it kept the refusal alive. The rule is now fail-closed on the way to nested: a flat host that has been stopped for its move and is then refused stays fenced and down (the pass keeps the fence, as it already did after a failed handover), status reports it `fenced` with the refusal (`held down after a refused move to nested`), every later pass retries from the kept fence and releases the host once it has moved, and a flat intent lifts the fence. A nested host refused its revert is not held: it is already isolated. The mixed-layout warning counts `foreign` and `external` rows beside `flat` ones, since all three run as uid carlos with no namespace; and the second-name refusal reports the inode and the `find <data> -xdev -samefile <path>` that lists the other names. (e) Landing follow-ups: a planted ENTRY in a host's future directory is no longer a refusal (one `touch` by a sibling would have held the host down until an operator acted, and the hold outlived the sibling's own migration); it is renamed into `/data/layout/aside/<host>/` with a random, no-replace name and the move goes on, logged and shown in status as a note. The hold stays for a second name (`nlink > 1`, no safe automatic fix) and for a symlink at the instance directory. The hold is persisted as `/data/layout/<host>.hold` and restored at agent start before anything wakes, by re-deriving it from the files: the marker is a hint to re-check, not the authority, because every file under `/data` is uid carlos's and a flat tenant can unlink it; a deleted marker therefore costs one grace after one restart, after which the next pass meets the same refusal and holds again, and a marker whose reason is gone is dropped rather than honoured. The serial leg re-runs on its own after a held result, as it does after a deferral, so a cleared reason is acted on within ten minutes on a quiet fleet.
2255