carlos / platform Public

Download this file

1	# Isolated Exec, Plan 2 of 4: Control Service, Trampoline, Exit, Probe, Templates
2
3	> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5	**Goal:** Land the root control channel and the three in-namespace programs, with their unit files, so that a box can start, stop and sweep `carlos-instance*@` units on the agent's behalf and can prove its own isolation with a probe. Nothing in the agent calls the channel yet (plan 3).
6
7	**Architecture:** One shared package `internal/instancectl` holds the names, escaping, launch-file and exit-record shapes, the HTTP protocol types, and the root server (`carlos instance-control`), which is socket-activated, authenticates the edge's main pid through `SO_PEERCRED`, drives `systemctl` through a seam, and runs the readiness probe. A second package `internal/instanceexec` holds the three programs that run inside a tenant's namespace as `carlos`: the trampoline (`instance-exec`), the advisory exit-record writer (`instance-exit`) and the probe (`instance-probe`). Six unit files carry titogo's namespace recipe with the spec's additions.
8
9	**Tech Stack:** Go 1.26, `golang.org/x/sys/unix` (`GetsockoptUcred`, `Openat2`, `Access`), systemd 257 (`PrivatePIDs=`, `StandardOutput=truncate:`), the repo's `embed.FS` unit set, `internal/instancefs`, `internal/provision`, `internal/controlsock`, `internal/cgroup` tiers.
10
11	**Spec:** `docs/superpowers/specs/2026-09-11-isolated-exec-design.md` (revision 12): §3.1, §3.3, §4, §5, §6.2, §6.5 (the record and the cgroup rule), §8.5, §9 (units and installers, the same-uid audit). Plan 1 (`docs/superpowers/plans/2026-09-11-isolated-exec-1-foundation.md`) landed the helpers this plan consumes: `provision.InstanceID`, `InstanceDir`, `InstanceRunDir`, `NestedSocket`, `LogSocket`, `MaxUnixPath`, `controlsock.Dir`/`Path`, `instancefs.OpenBeneath`/`ReadBeneath`/`WriteAtomic`/`MountNoSymfollow`, the mount units, `carlos.slice`, `carlos-instances.slice`, the tmpfiles lines (including `f /etc/carlos/probe-secret 0640 root carlos - probe`), and `systemctlPlan` starting `.socket` units.
12
13	## Global Constraints
14
15	- Gate before every commit: `go build ./...`, `go vet ./...`, `gofmt -l .` empty, `go test ./... -count=1`; env `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; socket-binding tests need `TMPDIR=/tmp/g` outside the sandbox. Cross-compile gate: darwin, windows, linux-arm64 must build; every Linux-only syscall is in a `_linux.go` with a `!linux` stub returning `errors.ErrUnsupported`.
16	- Unit-name escaping is systemd's: `/`→`-`, a leading `.`→`\x2e`, any byte outside `[a-zA-Z0-9:_.]`→`\xNN` (lowercase hex); so `-`→`\x2d` and `_` is unchanged (spec §4).
17	- Unit names: `carlos-instance@<esc>.service`, `carlos-instance-seed@<esc>.service`, `carlos-instance-sidecar@<esc>.service`, slice `carlos-instances-<esc>.slice`, where `<esc> = Escape(provision.InstanceID(host))`. Paths use `%I` (the unescaped identifier).
18	- The control service accepts a connection only when `SO_PEERCRED` reports uid `carlos` AND pid equal to `MainPID` of `carlos-edge.service` as `systemctl show` reports it (cached for one second); otherwise the connection is closed with no response (spec §5.4). No token.
19	- Verbs and their exact `systemctl` sequences are spec §5.3: `/start` = `set-property --runtime <slice> MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N`, `start <unit>`, `show -p InvocationID --value <unit>`; no `reset-failed` inside `/start`. `/seed` = the same `set-property`, blocking `start` of the seed unit, `show -p InvocationID,Result,ExecMainStatus,ExecMainCode`, wait for the unit's cgroup to be empty, `stop`. `/stop` = `stop <unit>`. `/reset` = `reset-failed <unit>` tolerating "not loaded". `/sweep` = enumerate the three templates' instances and jobs, stop each, wait terminal and empty. `/ready` = 204 when the probe verdict is ready, else 503 with the verdict.
20	- `memory_max` must be one of `cgroup.TierDefault.MemoryMax` (536870912) or `cgroup.TierChildren.MemoryMax` (1073741824); `MemoryHigh` is `cgroup.highOf` (max/10*9); anything else is 400.
21	- The exit record is advisory: `exit.<role>.<INVOCATION_ID>.json` in the instance's run directory, written `O_EXCL`, never overwritten (spec §6.5).
22	- The probe's verdict lines and their rules are spec §8.5: directory listings, pid 1, `EROFS` on cgroupfs, `/etc/carlos` unopenable and the fixture unreadable, `ELOOP` on a planted symlink, `nosymfollow` in mountinfo, UDP `sendto` `EPERM` on IMDS v4 (required) and v6 (`EPERM` enforced or `ENETUNREACH` unverified), loopback `sendto` succeeds, a 2s TCP connect to v4 IMDS must not succeed. `v6: unverified` does not block readiness; anything else failing does.
23	- `docs/site/cli.md` gets a `### carlos <verb>` heading for each new verb or the docsite gate fails. New verbs are `case` arms in `main.go`'s switch.
24	- Copy rules: no em dashes in any string; log lines `verb: subject: detail`; comments are full sentences explaining why.
25	- Commit after every task with the session's attribution trailer. Branch `worktree-isolated-exec-spec`.
26
27	---
28
29	## File Structure
30
31	| File | Responsibility |
32	|---|---|
33	| `internal/instancectl/names.go` (create) | `Role`, `Escape`, `UnitName`, `SliceName`, `ParseUnitName`, `LaunchFileName`, `ExitRecordName`, `Launch`, `ExitRecord`, protocol types |
34	| `internal/instancectl/names_test.go` (create) | tests for the above |
35	| `internal/instancectl/server.go` (create) | `Server`: listener, peer check seam, handlers, systemctl seam, cgroup-empty wait, probe runner, `Run(args)` |
36	| `internal/instancectl/peer_linux.go` (create) | `SO_PEERCRED` via `SyscallConn`, `LISTEN_FDS` listener |
37	| `internal/instancectl/peer_other.go` (create) | stubs |
38	| `internal/instancectl/server_test.go` (create) | handler tests over a real unix socket with fakes |
39	| `internal/instanceexec/exec.go` (create) | `RunExec`: the trampoline |
40	| `internal/instanceexec/exit.go` (create) | `RunExit`: the exit record |
41	| `internal/instanceexec/probe.go` (create) | `RunProbe`: the checks, with an injectable environment |
42	| `internal/instanceexec/probe_linux.go` / `probe_other.go` (create) | the `sendto`/`Access` syscalls and stubs |
43	| `internal/instanceexec/*_test.go` (create) | tests |
44	| `internal/host/units/carlos-instance@.service`, `carlos-instance-seed@.service`, `carlos-instance-sidecar@.service`, `carlos-instance-probe.service`, `carlos-instance-control.socket`, `carlos-instance-control.service` (create) | the units |
45	| `internal/host/bootstrap.go` (modify) | `enabledUnits` gains the control socket; tmpfiles gains the `_probe` directories |
46	| `internal/host/systemunits.go` (modify) | the hourly belt also arms the control socket |
47	| `internal/host/units_isolation_test.go` (modify) | template identity, the same-uid audit, control unit shape |
48	| `main.go` (modify) | four `case` arms |
49	| `docs/site/cli.md` (modify) | four headings under "On the host" |
50
51	---
52
53	### Task 1: Names, escaping and protocol types
54
55	**Files:**
56	- Create: `internal/instancectl/names.go`
57	- Test: `internal/instancectl/names_test.go`
58
59	**Interfaces:**
60	- Consumes: `provision.InstanceID`, `provision.ValidHostShape`.
61	- Produces:
62	```go
63	package instancectl
64	type Role string
65	const ( RoleRun Role = "run"; RoleSeed Role = "seed"; RoleSidecar Role = "sidecar" )
66	func ParseRole(s string) (Role, bool)
67	func Escape(s string) string
68	func UnitName(role Role, host string) string
69	func SliceName(host string) string
70	const ( UnitPrefixRun = "carlos-instance@"; UnitPrefixSeed = "carlos-instance-seed@"; UnitPrefixSidecar = "carlos-instance-sidecar@" )
71	func ParseUnitName(unit string) (role Role, escapedID string, ok bool)
72	const LaunchSchema = 1
73	type Launch struct { Schema int `json:"schema"`; Host string `json:"host"`; Argv []string `json:"argv"`; Env []string `json:"env"` }
74	func LaunchFileName(role Role) string // "launch.<role>.json"
75	type ExitRecord struct { Invocation string `json:"invocation"`; Result string `json:"result"`; Code string `json:"code"`; Status string `json:"status"`; At time.Time `json:"at"` }
76	func ExitRecordName(role Role, invocation string) string // "exit.<role>.<invocation>.json"
77	const ProbeDirName = "_probe" // the probe's own instance directory name; no host can start with '_'
78	type VerdictLine struct { Check string `json:"check"`; Verdict string `json:"verdict"`; Detail string `json:"detail,omitempty"` }
79	type ProbeReport struct { Ready bool `json:"ready"`; Lines []VerdictLine `json:"lines"` }
80	type StartResponse struct { InvocationID string `json:"invocation_id"` }
81	type SeedResponse struct { InvocationID string `json:"invocation_id"`; Result string `json:"result"`; Status string `json:"status"`; Code string `json:"code"` }
82	type SweepResponse struct { Units []string `json:"units"` }
83	```
84
85	- [ ] **Step 1: Write the failing tests**
86
87	Create `internal/instancectl/names_test.go`:
88
89	```go
90	package instancectl
91
92	import (
93	"strings"
94	"testing"
95
96	"github.com/carlosframework/platform/internal/provision"
97	)
98
99	// Escape follows systemd-escape(1) for unit instance names. The cases are
100	// the ones our identifiers can produce (hosts are [a-z0-9.-]; a hashed
101	// identifier adds a leading underscore), plus the general rules so a
102	// surprise character can never yield an unloadable unit name.
103	func TestEscapeFollowsSystemd(t *testing.T) {
104	cases := map[string]string{
105	"bill.go.tito.io": "bill.go.tito.io",
106	"bill-2.example.com": `bill\x2d2.example.com`,
107	"_aaaa-1234abcd": `_aaaa\x2d1234abcd`,
108	"a/b": "a-b",
109	".hidden": `\x2ehidden`,
110	"x y": `x\x20y`,
111	"": "",
112	"UPPER_ok:1": "UPPER_ok:1",
113	}
114	for in, want := range cases {
115	if got := Escape(in); got != want {
116	t.Errorf("Escape(%q) = %q, want %q", in, got, want)
117	}
118	}
119	}
120
121	func TestUnitAndSliceNamesAreKeyedByEscapedInstanceID(t *testing.T) {
122	host := "bill-2.example.com"
123	esc := `bill\x2d2.example.com`
124	if got := UnitName(RoleRun, host); got != "carlos-instance@"+esc+".service" {
125	t.Errorf("run unit = %q", got)
126	}
127	if got := UnitName(RoleSeed, host); got != "carlos-instance-seed@"+esc+".service" {
128	t.Errorf("seed unit = %q", got)
129	}
130	if got := UnitName(RoleSidecar, host); got != "carlos-instance-sidecar@"+esc+".service" {
131	t.Errorf("sidecar unit = %q", got)
132	}
133	if got := SliceName(host); got != "carlos-instances-"+esc+".slice" {
134	t.Errorf("slice = %q", got)
135	}
136	long := strings.Repeat("a", 60) + ".example.com"
137	if got := UnitName(RoleRun, long); !strings.HasPrefix(got, "carlos-instance@_") || !strings.HasSuffix(got, ".service") {
138	t.Errorf("long host unit = %q, want the hashed identifier", got)
139	}
140	if got, want := UnitName(RoleRun, long), "carlos-instance@"+Escape(provision.InstanceID(long))+".service"; got != want {
141	t.Errorf("long host unit = %q, want %q", got, want)
142	}
143	}
144
145	func TestParseUnitNameRoundTrips(t *testing.T) {
146	for _, role := range []Role{RoleRun, RoleSeed, RoleSidecar} {
147	u := UnitName(role, "bill-2.example.com")
148	gotRole, esc, ok := ParseUnitName(u)
149	if !ok || gotRole != role || esc != `bill\x2d2.example.com` {
150	t.Errorf("ParseUnitName(%q) = %q %q %v", u, gotRole, esc, ok)
151	}
152	}
153	for _, bad := range []string{"carlos-edge.service", "carlos-app@x.service", "carlos-instance@x.socket", "carlos-instance@.service", "carlos-instances-x.slice"} {
154	if _, _, ok := ParseUnitName(bad); ok {
155	t.Errorf("ParseUnitName(%q) accepted", bad)
156	}
157	}
158	}
159
160	func TestParseRole(t *testing.T) {
161	for _, s := range []string{"run", "seed", "sidecar"} {
162	if r, ok := ParseRole(s); !ok || string(r) != s {
163	t.Errorf("ParseRole(%q) = %q %v", s, r, ok)
164	}
165	}
166	for _, s := range []string{"", "Run", "exec", "run "} {
167	if _, ok := ParseRole(s); ok {
168	t.Errorf("ParseRole(%q) accepted", s)
169	}
170	}
171	}
172
173	func TestFileNames(t *testing.T) {
174	if got := LaunchFileName(RoleSidecar); got != "launch.sidecar.json" {
175	t.Errorf("LaunchFileName = %q", got)
176	}
177	if got := ExitRecordName(RoleRun, "abc123"); got != "exit.run.abc123.json" {
178	t.Errorf("ExitRecordName = %q", got)
179	}
180	if provision.ValidHostShape(ProbeDirName) {
181	t.Errorf("ProbeDirName %q is a valid host; it must never collide with one", ProbeDirName)
182	}
183	}
184	```
185
186	- [ ] **Step 2: Run the tests to verify they fail**
187
188	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instancectl -count=1`
189	Expected: build failure (`undefined: Escape` and the rest; or no Go files).
190
191	- [ ] **Step 3: Write `names.go`**
192
193	```go
194	// Package instancectl is the root instance-control channel of the
195	// isolated-exec design (§5) and the names everything on both sides of it
196	// agrees on: unit and slice names keyed by the bounded instance identifier,
197	// the launch file the trampoline reads (§6.2), the advisory exit record
198	// (§6.5) and the probe's verdict (§8.5).
199	//
200	// The names live in one package because three programs and two services
201	// build them: the agent (plan 3) composes launch files and requests, the
202	// control service builds unit names from validated hosts, and the
203	// in-namespace programs read files keyed by the same identifier. A
204	// disagreement between any two of them would be a unit that never starts
205	// or a record nobody finds.
206	package instancectl
207
208	import (
209	"fmt"
210	"strings"
211	"time"
212
213	"github.com/carlosframework/platform/internal/provision"
214	)
215
216	// Role is which of the three template units a request or a file is about.
217	type Role string
218
219	const (
220	RoleRun Role = "run"
221	RoleSeed Role = "seed"
222	RoleSidecar Role = "sidecar"
223	)
224
225	// ParseRole accepts exactly the three literals.
226	func ParseRole(s string) (Role, bool) {
227	switch Role(s) {
228	case RoleRun, RoleSeed, RoleSidecar:
229	return Role(s), true
230	}
231	return "", false
232	}
233
234	// Escape is systemd-escape(1)'s rule for a unit instance name: "/" becomes
235	// "-", a leading "." becomes "\x2e", and every byte outside [a-zA-Z0-9:_.]
236	// becomes "\x" plus two lowercase hex digits. For the identifiers this
237	// platform produces (hosts are [a-z0-9.-], a hashed identifier adds a
238	// leading underscore) that means "-" becomes "\x2d" and nothing else
239	// changes; the general rule is kept so an unexpected byte can never yield
240	// a name systemd refuses to load.
241	func Escape(s string) string {
242	var b strings.Builder
243	for i := 0; i < len(s); i++ {
244	c := s[i]
245	switch {
246	case c == '/':
247	b.WriteByte('-')
248	case i == 0 && c == '.':
249	b.WriteString(`\x2e`)
250	case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == ':', c == '_', c == '.':
251	b.WriteByte(c)
252	default:
253	fmt.Fprintf(&b, `\x%02x`, c)
254	}
255	}
256	return b.String()
257	}
258
259	// The three template prefixes. A unit under any of them is the platform's
260	// to start, stop and sweep; nothing else is (design §5.3).
261	const (
262	UnitPrefixRun = "carlos-instance@"
263	UnitPrefixSeed = "carlos-instance-seed@"
264	UnitPrefixSidecar = "carlos-instance-sidecar@"
265	)
266
267	func prefixFor(role Role) string {
268	switch role {
269	case RoleSeed:
270	return UnitPrefixSeed
271	case RoleSidecar:
272	return UnitPrefixSidecar
273	}
274	return UnitPrefixRun
275	}
276
277	// UnitName is the template instance for a role and a host:
278	// carlos-instance[-seed|-sidecar]@<Escape(InstanceID(host))>.service.
279	func UnitName(role Role, host string) string {
280	return prefixFor(role) + Escape(provision.InstanceID(host)) + ".service"
281	}
282
283	// SliceName is the per-host slice every role of a host runs in (design
284	// §4.3): carlos-instances-<Escape(InstanceID(host))>.slice.
285	func SliceName(host string) string {
286	return "carlos-instances-" + Escape(provision.InstanceID(host)) + ".slice"
287	}
288
289	// ParseUnitName recognises the three templates' instances and returns the
290	// role and the ESCAPED identifier (systemd reports names escaped). Anything
291	// else, including a template with no instance, is not ours.
292	func ParseUnitName(unit string) (Role, string, bool) {
293	if !strings.HasSuffix(unit, ".service") {
294	return "", "", false
295	}
296	body := strings.TrimSuffix(unit, ".service")
297	for _, p := range []struct {
298	prefix string
299	role Role
300	}{{UnitPrefixSeed, RoleSeed}, {UnitPrefixSidecar, RoleSidecar}, {UnitPrefixRun, RoleRun}} {
301	if strings.HasPrefix(body, p.prefix) {
302	esc := strings.TrimPrefix(body, p.prefix)
303	if esc == "" {
304	return "", "", false
305	}
306	return p.role, esc, true
307	}
308	}
309	return "", "", false
310	}
311
312	// LaunchSchema is the only launch-file schema this release understands.
313	const LaunchSchema = 1
314
315	// Launch is what the agent writes into the instance directory before a
316	// start and what the trampoline execs (design §6.2). Env is the complete
317	// environment, one entry per key, exactly as the agent composed it; the
318	// trampoline parses nothing.
319	type Launch struct {
320	Schema int `json:"schema"`
321	Host string `json:"host"`
322	Argv []string `json:"argv"`
323	Env []string `json:"env"`
324	}
325
326	// LaunchFileName is "launch.<role>.json" inside the instance directory.
327	func LaunchFileName(role Role) string { return "launch." + string(role) + ".json" }
328
329	// ExitRecord is the advisory record the unit's ExecStopPost leaves in the
330	// instance's run directory (design §6.5). It decides the wording of the
331	// member-facing event line and nothing else.
332	type ExitRecord struct {
333	Invocation string `json:"invocation"`
334	Result string `json:"result"`
335	Code string `json:"code"`
336	Status string `json:"status"`
337	At time.Time `json:"at"`
338	}
339
340	// ExitRecordName is "exit.<role>.<invocation>.json".
341	func ExitRecordName(role Role, invocation string) string {
342	return "exit." + string(role) + "." + invocation + ".json"
343	}
344
345	// ProbeDirName is the instance directory the readiness probe runs in. It
346	// starts with an underscore, which no valid host does, so it can never
347	// collide with a real instance's directory.
348	const ProbeDirName = "_probe"
349
350	// VerdictLine is one check of the readiness probe (design §8.5). Verdict is
351	// "ok", "unverified" or "fail"; only "fail" blocks readiness.
352	type VerdictLine struct {
353	Check string `json:"check"`
354	Verdict string `json:"verdict"`
355	Detail string `json:"detail,omitempty"`
356	}
357
358	// ProbeReport is what the probe prints and what /ready answers with.
359	type ProbeReport struct {
360	Ready bool `json:"ready"`
361	Lines []VerdictLine `json:"lines"`
362	}
363
364	// Responses of the control verbs (design §5.3).
365	type StartResponse struct {
366	InvocationID string `json:"invocation_id"`
367	}
368
369	type SeedResponse struct {
370	InvocationID string `json:"invocation_id"`
371	Result string `json:"result"`
372	Status string `json:"status"`
373	Code string `json:"code"`
374	}
375
376	type SweepResponse struct {
377	Units []string `json:"units"`
378	}
379	```
380
381	- [ ] **Step 4: Run the tests to verify they pass**
382
383	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instancectl -count=1`
384	Expected: PASS.
385
386	- [ ] **Step 5: gofmt and commit**
387
388	```bash
389	gofmt -l internal/instancectl
390	git add internal/instancectl
391	git commit -m "instancectl: unit names, escaping, launch and exit shapes, protocol types (plan 2, task 1)"
392	```
393
394	---
395
396	### Task 2: The trampoline, `carlos instance-exec`
397
398	**Files:**
399	- Create: `internal/instanceexec/exec.go`
400	- Create: `internal/instanceexec/exec_test.go`
401
402	**Interfaces:**
403	- Consumes: `instancectl.Launch`, `LaunchSchema`, `LaunchFileName`, `ParseRole`; `instancefs.ReadBeneath`; `provision.ValidHostShape`.
404	- Produces:
405	```go
406	package instanceexec
407	// RunExec is `carlos instance-exec <role> <iid>`: read the launch file, validate, execve.
408	func RunExec(args []string)
409	// execLaunch is the testable core; execFn defaults to syscall.Exec.
410	func execLaunch(dataDir string, role instancectl.Role, iid string, execFn func(argv0 string, argv, env []string) error) error
411	const LiveDir = "/opt/carlos/live"
412	```
413
414	- [ ] **Step 1: Write the failing tests**
415
416	Create `internal/instanceexec/exec_test.go`:
417
418	```go
419	package instanceexec
420
421	import (
422	"encoding/json"
423	"errors"
424	"os"
425	"path/filepath"
426	"testing"
427
428	"github.com/carlosframework/platform/internal/instancectl"
429	)
430
431	func writeLaunch(t *testing.T, dir string, role instancectl.Role, l instancectl.Launch) {
432	t.Helper()
433	if err := os.MkdirAll(dir, 0o700); err != nil {
434	t.Fatal(err)
435	}
436	b, err := json.Marshal(l)
437	if err != nil {
438	t.Fatal(err)
439	}
440	if err := os.WriteFile(filepath.Join(dir, instancectl.LaunchFileName(role)), b, 0o600); err != nil {
441	t.Fatal(err)
442	}
443	}
444
445	type captured struct {
446	argv0 string
447	argv []string
448	env []string
449	calls int
450	}
451
452	func (c *captured) exec(argv0 string, argv, env []string) error {
453	c.calls++
454	c.argv0, c.argv, c.env = argv0, argv, env
455	return nil
456	}
457
458	func TestExecLaunchExecsExactlyWhatTheAgentWrote(t *testing.T) {
459	data := t.TempDir()
460	iid := "bill.example.com"
461	dir := filepath.Join(data, "instances", iid)
462	writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{
463	Schema: 1, Host: "bill.example.com",
464	Argv: []string{"/opt/carlos/live/bill.example.com", "--socket", "/run/carlos/instances/bill.example.com/app.sock", "--db", "/data/instances/bill.example.com/bill.example.com.db"},
465	Env: []string{"PATH=/usr/bin", "CARLOS_ADMIN_TOKEN=abc"},
466	})
467	var c captured
468	if err := execLaunch(data, instancectl.RoleRun, iid, c.exec); err != nil {
469	t.Fatalf("execLaunch: %v", err)
470	}
471	if c.calls != 1 || c.argv0 != "/opt/carlos/live/bill.example.com" || len(c.argv) != 5 || c.argv[2] != "/run/carlos/instances/bill.example.com/app.sock" {
472	t.Errorf("exec = %+v", c)
473	}
474	if len(c.env) != 2 || c.env[1] != "CARLOS_ADMIN_TOKEN=abc" {
475	t.Errorf("env = %v", c.env)
476	}
477	}
478
479	func TestExecLaunchRefusesAnArgv0OutsideTheLiveDir(t *testing.T) {
480	data := t.TempDir()
481	iid := "h.example.com"
482	dir := filepath.Join(data, "instances", iid)
483	for _, argv0 := range []string{"/bin/sh", "/opt/carlos/live/../carlos", "/opt/carlos/live/other.example.com", "opt/carlos/live/h.example.com", ""} {
484	writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{Schema: 1, Host: "h.example.com", Argv: []string{argv0}, Env: nil})
485	var c captured
486	err := execLaunch(data, instancectl.RoleRun, iid, c.exec)
487	if err == nil || c.calls != 0 {
488	t.Errorf("argv0 %q: err = %v, calls = %d; want a refusal and no exec", argv0, err, c.calls)
489	}
490	}
491	}
492
493	func TestExecLaunchRefusesUnknownSchemaAndBadEnv(t *testing.T) {
494	data := t.TempDir()
495	iid := "h.example.com"
496	dir := filepath.Join(data, "instances", iid)
497	good := []string{"/opt/carlos/live/h.example.com"}
498	writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{Schema: 2, Host: "h.example.com", Argv: good})
499	var c captured
500	if err := execLaunch(data, instancectl.RoleRun, iid, c.exec); err == nil || c.calls != 0 {
501	t.Errorf("schema 2: err = %v, calls = %d", err, c.calls)
502	}
503	writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{Schema: 1, Host: "h.example.com", Argv: good, Env: []string{"NOEQUALS"}})
504	if err := execLaunch(data, instancectl.RoleRun, iid, c.exec); err == nil || c.calls != 0 {
505	t.Errorf("bad env: err = %v, calls = %d", err, c.calls)
506	}
507	writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{Schema: 1, Host: "not a host", Argv: []string{"/opt/carlos/live/not a host"}})
508	if err := execLaunch(data, instancectl.RoleRun, iid, c.exec); err == nil || c.calls != 0 {
509	t.Errorf("bad host: err = %v, calls = %d", err, c.calls)
510	}
511	}
512
513	func TestExecLaunchRefusesASymlinkedLaunchFile(t *testing.T) {
514	data := t.TempDir()
515	iid := "h.example.com"
516	dir := filepath.Join(data, "instances", iid)
517	if err := os.MkdirAll(dir, 0o700); err != nil {
518	t.Fatal(err)
519	}
520	other := filepath.Join(t.TempDir(), "launch.run.json")
521	l := instancectl.Launch{Schema: 1, Host: "h.example.com", Argv: []string{"/opt/carlos/live/h.example.com"}}
522	b, _ := json.Marshal(l)
523	if err := os.WriteFile(other, b, 0o600); err != nil {
524	t.Fatal(err)
525	}
526	if err := os.Symlink(other, filepath.Join(dir, "launch.run.json")); err != nil {
527	t.Fatal(err)
528	}
529	var c captured
530	err := execLaunch(data, instancectl.RoleRun, iid, c.exec)
531	if err == nil || c.calls != 0 || !errors.Is(err, errSymlinkOrEscape) {
532	t.Errorf("symlinked launch file: err = %v, calls = %d", err, c.calls)
533	}
534	}
535	```
536
537	- [ ] **Step 2: Run the tests to verify they fail**
538
539	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -run TestExecLaunch -count=1`
540	Expected: build failure, `undefined: execLaunch`.
541
542	- [ ] **Step 3: Write `exec.go`**
543
544	```go
545	// Package instanceexec holds the three programs that run INSIDE an isolated
546	// instance's namespace, as carlos, launched by the template units of the
547	// isolated-exec design: the trampoline (instance-exec, §6.2), the advisory
548	// exit-record writer (instance-exit, §6.5) and the readiness probe
549	// (instance-probe, §8.5). None of them is for operators; each is an
550	// ExecStart= or ExecStopPost= of a unit.
551	package instanceexec
552
553	import (
554	"encoding/json"
555	"errors"
556	"flag"
557	"fmt"
558	"log"
559	"os"
560	"path/filepath"
561	"strings"
562	"syscall"
563
564	"github.com/carlosframework/platform/internal/cliflag"
565	"github.com/carlosframework/platform/internal/instancectl"
566	"github.com/carlosframework/platform/internal/instancefs"
567	"github.com/carlosframework/platform/internal/provision"
568	)
569
570	// LiveDir is where the adoption pass keeps each host's live symlink; the
571	// trampoline execs nothing that is not directly under it.
572	const LiveDir = "/opt/carlos/live"
573
574	var errSymlinkOrEscape = errors.New("instance-exec: launch file refused")
575
576	// dataDir is the agent's own resolver (host/agent.go), repeated here because
577	// this program runs in a namespace where only the instance's directory is
578	// bound, and it must agree with the agent on where that directory is.
579	func dataDir() string {
580	if d := os.Getenv("CARLOS_DATA_DIR"); d != "" {
581	return d
582	}
583	return "/data"
584	}
585
586	// RunExec is `carlos instance-exec <role> <iid>`.
587	func RunExec(args []string) {
588	fs := flag.NewFlagSet("instance-exec", flag.ExitOnError)
589	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exec <run|seed|sidecar> <instance-id>")
590	_ = fs.Parse(args)
591	if fs.NArg() != 2 {
592	fs.Usage()
593	os.Exit(2)
594	}
595	role, ok := instancectl.ParseRole(fs.Arg(0))
596	if !ok {
597	log.Fatalf("instance-exec: role %q is not run, seed or sidecar", fs.Arg(0))
598	}
599	if err := execLaunch(dataDir(), role, fs.Arg(1), syscall.Exec); err != nil {
600	log.Fatalf("instance-exec: %v", err)
601	}
602	}
603
604	// execLaunch reads launch.<role>.json from the instance directory through
605	// instancefs (never through a symlink), validates it, and execs. It returns
606	// only on refusal; a successful exec never returns.
607	//
608	// The validation is defence in depth: the agent wrote the file, but the
609	// directory is the instance's own, so the instance can rewrite it. What
610	// the checks guarantee is that whatever is executed is directly under
611	// LiveDir (the platform's live links, root-written) and named for the host
612	// in the file, so a rewritten file can execute only what the tenant could
613	// execute anyway: its own binary with an argv and env it chose.
614	func execLaunch(dataDir string, role instancectl.Role, iid string, execFn func(argv0 string, argv, env []string) error) error {
615	dir := filepath.Join(dataDir, "instances", iid)
616	raw, err := instancefs.ReadBeneath(dir, instancectl.LaunchFileName(role))
617	if err != nil {
618	if errors.Is(err, instancefs.ErrSymlink) || errors.Is(err, instancefs.ErrEscapes) {
619	return fmt.Errorf("%w: %v", errSymlinkOrEscape, err)
620	}
621	return err
622	}
623	var l instancectl.Launch
624	if err := json.Unmarshal(raw, &l); err != nil {
625	return fmt.Errorf("launch file: %v", err)
626	}
627	if l.Schema != instancectl.LaunchSchema {
628	return fmt.Errorf("launch file schema %d is not %d", l.Schema, instancectl.LaunchSchema)
629	}
630	if !provision.ValidHostShape(l.Host) {
631	return fmt.Errorf("launch file host %q is not a host", l.Host)
632	}
633	if len(l.Argv) == 0 {
634	return errors.New("launch file has no argv")
635	}
636	argv0 := l.Argv[0]
637	if !filepath.IsAbs(argv0) || filepath.Clean(argv0) != argv0 || filepath.Dir(argv0) != LiveDir {
638	return fmt.Errorf("argv[0] %q is not directly under %s", argv0, LiveDir)
639	}
640	if filepath.Base(argv0) != l.Host {
641	return fmt.Errorf("argv[0] %q is not the live link of host %s", argv0, l.Host)
642	}
643	for _, kv := range l.Env {
644	if k, _, ok := strings.Cut(kv, "="); !ok || k == "" {
645	return fmt.Errorf("launch file env entry %q is not KEY=value", kv)
646	}
647	}
648	return execFn(argv0, l.Argv, l.Env)
649	}
650	```
651
652	- [ ] **Step 4: Run the tests to verify they pass**
653
654	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -run TestExecLaunch -count=1`
655	Expected: PASS (the symlink test needs Linux; on the CI's Linux hosts it runs).
656
657	- [ ] **Step 5: gofmt and commit**
658
659	```bash
660	gofmt -l internal/instanceexec
661	git add internal/instanceexec
662	git commit -m "instanceexec: the trampoline reads the launch file through instancefs and execs the live link (plan 2, task 2)"
663	```
664
665	---
666
667	### Task 3: The exit record, `carlos instance-exit`
668
669	**Files:**
670	- Create: `internal/instanceexec/exit.go`
671	- Create: `internal/instanceexec/exit_test.go`
672
673	**Interfaces:**
674	- Consumes: `instancectl.ExitRecord`, `ExitRecordName`, `ParseRole`; `instancefs.OpenBeneath`.
675	- Produces:
676	```go
677	func RunExit(args []string)
678	// writeExitRecord writes exit.<role>.<invocation>.json into runDir/instances/<iid>, O_EXCL; env is the process environment lookup.
679	func writeExitRecord(runDir string, role instancectl.Role, iid string, env func(string) string, now time.Time) error
680	```
681
682	- [ ] **Step 1: Write the failing tests**
683
684	Create `internal/instanceexec/exit_test.go`:
685
686	```go
687	package instanceexec
688
689	import (
690	"encoding/json"
691	"os"
692	"path/filepath"
693	"testing"
694	"time"
695
696	"github.com/carlosframework/platform/internal/instancectl"
697	)
698
699	func envOf(m map[string]string) func(string) string {
700	return func(k string) string { return m[k] }
701	}
702
703	func TestWriteExitRecordWritesTheFourVariables(t *testing.T) {
704	run := t.TempDir()
705	iid := "h.example.com"
706	if err := os.MkdirAll(filepath.Join(run, "instances", iid), 0o700); err != nil {
707	t.Fatal(err)
708	}
709	now := time.Date(2026, 9, 11, 12, 0, 0, 0, time.UTC)
710	env := envOf(map[string]string{"INVOCATION_ID": "deadbeef", "SERVICE_RESULT": "exit-code", "EXIT_CODE": "exited", "EXIT_STATUS": "3"})
711	if err := writeExitRecord(run, instancectl.RoleRun, iid, env, now); err != nil {
712	t.Fatalf("writeExitRecord: %v", err)
713	}
714	b, err := os.ReadFile(filepath.Join(run, "instances", iid, "exit.run.deadbeef.json"))
715	if err != nil {
716	t.Fatal(err)
717	}
718	var rec instancectl.ExitRecord
719	if err := json.Unmarshal(b, &rec); err != nil {
720	t.Fatal(err)
721	}
722	if rec.Invocation != "deadbeef" || rec.Result != "exit-code" || rec.Code != "exited" || rec.Status != "3" || !rec.At.Equal(now) {
723	t.Errorf("record = %+v", rec)
724	}
725	// O_EXCL: a second write for the same invocation is refused and the
726	// first record stands.
727	env2 := envOf(map[string]string{"INVOCATION_ID": "deadbeef", "SERVICE_RESULT": "success", "EXIT_CODE": "exited", "EXIT_STATUS": "0"})
728	if err := writeExitRecord(run, instancectl.RoleRun, iid, env2, now); err == nil {
729	t.Error("second write for the same invocation succeeded")
730	}
731	b2, _ := os.ReadFile(filepath.Join(run, "instances", iid, "exit.run.deadbeef.json"))
732	if string(b2) != string(b) {
733	t.Error("first record was overwritten")
734	}
735	}
736
737	func TestWriteExitRecordWithoutInvocationIsANoop(t *testing.T) {
738	run := t.TempDir()
739	iid := "h.example.com"
740	if err := os.MkdirAll(filepath.Join(run, "instances", iid), 0o700); err != nil {
741	t.Fatal(err)
742	}
743	if err := writeExitRecord(run, instancectl.RoleSeed, iid, envOf(nil), time.Now()); err != nil {
744	t.Fatalf("no INVOCATION_ID: %v (must not fail the unit's stop)", err)
745	}
746	entries, _ := os.ReadDir(filepath.Join(run, "instances", iid))
747	if len(entries) != 0 {
748	t.Errorf("wrote %v without an invocation id", entries)
749	}
750	}
751
752	func TestWriteExitRecordRefusesAnInvocationThatIsNotAName(t *testing.T) {
753	run := t.TempDir()
754	iid := "h.example.com"
755	if err := os.MkdirAll(filepath.Join(run, "instances", iid), 0o700); err != nil {
756	t.Fatal(err)
757	}
758	env := envOf(map[string]string{"INVOCATION_ID": "../../etc/x", "SERVICE_RESULT": "success"})
759	if err := writeExitRecord(run, instancectl.RoleRun, iid, env, time.Now()); err == nil {
760	t.Error("an invocation id with a path separator was accepted")
761	}
762	}
763	```
764
765	- [ ] **Step 2: Run the tests to verify they fail**
766
767	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -run TestWriteExitRecord -count=1`
768	Expected: build failure, `undefined: writeExitRecord`.
769
770	- [ ] **Step 3: Write `exit.go`**
771
772	```go
773	package instanceexec
774
775	import (
776	"encoding/json"
777	"errors"
778	"flag"
779	"fmt"
780	"log"
781	"os"
782	"path/filepath"
783	"regexp"
784	"time"
785
786	"github.com/carlosframework/platform/internal/cliflag"
787	"github.com/carlosframework/platform/internal/instancectl"
788	"github.com/carlosframework/platform/internal/instancefs"
789	"github.com/carlosframework/platform/internal/provision"
790	)
791
792	// invocationShape is systemd's 128-bit invocation id as 32 hex digits. The
793	// id becomes part of a file name, so anything else is refused before it can
794	// name a path.
795	var invocationShape = regexp.MustCompile(`^[0-9a-f]{32}$`)
796
797	// RunExit is `carlos instance-exit <role> <iid>`, the ExecStopPost= of every
798	// template unit. It runs inside the namespace as carlos with systemd's
799	// $INVOCATION_ID, $SERVICE_RESULT, $EXIT_CODE and $EXIT_STATUS, and leaves
800	// the advisory exit record in the instance's own run directory (design
801	// §6.5). It never fails the unit: a stop that could not be recorded is
802	// still a stop, and the agent proves termination from unit state and the
803	// cgroup, not from this file.
804	func RunExit(args []string) {
805	fs := flag.NewFlagSet("instance-exit", flag.ExitOnError)
806	runDir := fs.String("run-dir", provision.SocketDir, "run directory holding instances/<id>")
807	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exit <run|seed|sidecar> <instance-id>")
808	_ = fs.Parse(args)
809	if fs.NArg() != 2 {
810	fs.Usage()
811	os.Exit(2)
812	}
813	role, ok := instancectl.ParseRole(fs.Arg(0))
814	if !ok {
815	log.Printf("instance-exit: role %q is not run, seed or sidecar", fs.Arg(0))
816	return
817	}
818	if err := writeExitRecord(*runDir, role, fs.Arg(1), os.Getenv, time.Now()); err != nil {
819	log.Printf("instance-exit: %v", err)
820	}
821	}
822
823	// writeExitRecord writes exit.<role>.<invocation>.json with O_EXCL into
824	// <runDir>/instances/<iid>. A missing invocation id is a no-op (the
825	// variables are systemd's to provide); a malformed one is an error.
826	func writeExitRecord(runDir string, role instancectl.Role, iid string, env func(string) string, now time.Time) error {
827	inv := env("INVOCATION_ID")
828	if inv == "" {
829	return nil
830	}
831	if !invocationShape.MatchString(inv) {
832	return fmt.Errorf("INVOCATION_ID %q is not 32 hex digits", inv)
833	}
834	rec := instancectl.ExitRecord{
835	Invocation: inv,
836	Result: env("SERVICE_RESULT"),
837	Code: env("EXIT_CODE"),
838	Status: env("EXIT_STATUS"),
839	At: now.UTC(),
840	}
841	raw, err := json.MarshalIndent(rec, "", " ")
842	if err != nil {
843	return err
844	}
845	raw = append(raw, '\n')
846	dir := filepath.Join(runDir, "instances", iid)
847	f, err := instancefs.OpenBeneath(dir, instancectl.ExitRecordName(role, inv), os.O_WRONLY|os.O_CREAT|os.O_EXCL, 0o644)
848	if err != nil {
849	if errors.Is(err, os.ErrExist) {
850	return fmt.Errorf("record for invocation %s already exists", inv)
851	}
852	return err
853	}
854	if _, err := f.Write(raw); err != nil {
855	f.Close()
856	return err
857	}
858	return f.Close()
859	}
860	```
861
862	Note: `instancefs.OpenBeneath` wraps `EEXIST` in an `*os.PathError`; `errors.Is(err, os.ErrExist)` matches it through `syscall.Errno.Is`. On non-Linux `OpenBeneath` returns `errors.ErrUnsupported`, so the tests are Linux-only: put `//go:build linux` at the top of `exit_test.go` and `exec_test.go` (the symlink test needs it anyway), and add a trivial `exec_other_test.go`? No: leave non-Linux without tests; the package still builds there.
863
864	- [ ] **Step 4: Run the tests to verify they pass**
865
866	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -count=1 && GOOS=darwin go build ./internal/instanceexec && GOOS=windows go build ./internal/instanceexec`
867	Expected: PASS and both builds.
868
869	- [ ] **Step 5: gofmt and commit**
870
871	```bash
872	gofmt -l internal/instanceexec
873	git add internal/instanceexec
874	git commit -m "instanceexec: the advisory exit record, written O_EXCL from systemd's four variables (plan 2, task 3)"
875	```
876
877	---
878
879	### Task 4: The probe, `carlos instance-probe`
880
881	**Files:**
882	- Create: `internal/instanceexec/probe.go`
883	- Create: `internal/instanceexec/probe_linux.go`
884	- Create: `internal/instanceexec/probe_other.go`
885	- Create: `internal/instanceexec/probe_test.go`
886
887	**Interfaces:**
888	- Consumes: `instancectl.ProbeReport`, `VerdictLine`, `ProbeDirName`; `instancefs.MountNoSymfollow`.
889	- Produces:
890	```go
891	func RunProbe(args []string) // prints a ProbeReport as JSON on stdout; exit 0 always
892	type probeEnv struct {
893	dataRoot, runRoot, varCache, varLib, etcCarlos, procRoot, cgroupProcs string
894	pid func() int
895	writable func(path string) error // access(W_OK); production: unix.Access
896	sendUDP func(network, addr string) error // one datagram; production: a UDP socket write
897	connectTCP func(addr string, timeout time.Duration) error // production: net.DialTimeout
898	mountNoSym func(mountpoint string) (bool, error)
899	}
900	func runChecks(e probeEnv) instancectl.ProbeReport
901	```
902
903	- [ ] **Step 1: Write the failing tests**
904
905	Create `internal/instanceexec/probe_test.go`:
906
907	```go
908	//go:build linux
909
910	package instanceexec
911
912	import (
913	"errors"
914	"os"
915	"path/filepath"
916	"syscall"
917	"testing"
918	"time"
919
920	"github.com/carlosframework/platform/internal/instancectl"
921	)
922
923	// goodEnv is a fixture that passes every check: the trees an isolated
924	// instance sees, pid 1, a read-only cgroupfs, an unopenable /etc/carlos, a
925	// nosymfollow mount, EPERM on both metadata addresses, a live loopback and
926	// a TCP connect that never succeeds.
927	func goodEnv(t *testing.T) probeEnv {
928	t.Helper()
929	root := t.TempDir()
930	mk := func(p string) string {
931	if err := os.MkdirAll(filepath.Join(root, p), 0o700); err != nil {
932	t.Fatal(err)
933	}
934	return filepath.Join(root, p)
935	}
936	data := mk("data")
937	mk("data/instances/_probe")
938	run := mk("run/carlos")
939	mk("run/carlos/instances/_probe")
940	varCache := mk("var/cache")
941	varLib := mk("var/lib")
942	etc := filepath.Join(root, "etc", "carlos") // deliberately absent: opening it fails
943	proc := mk("proc")
944	mk("proc/1")
945	mk("proc/self")
946	return probeEnv{
947	dataRoot: data, runRoot: run, varCache: varCache, varLib: varLib, etcCarlos: etc, procRoot: proc,
948	cgroupProcs: filepath.Join(root, "sys", "cgroup.procs"),
949	pid: func() int { return 1 },
950	writable: func(string) error { return syscall.EROFS },
951	sendUDP: func(network, addr string) error {
952	switch addr {
953	case "169.254.169.254:80", "[fd00:ec2::254]:80":
954	return syscall.EPERM
955	}
956	return nil
957	},
958	connectTCP: func(string, time.Duration) error { return syscall.ETIMEDOUT },
959	mountNoSym: func(string) (bool, error) { return true, nil },
960	}
961	}
962
963	func verdict(r instancectl.ProbeReport, check string) string {
964	for _, l := range r.Lines {
965	if l.Check == check {
966	return l.Verdict
967	}
968	}
969	return "<missing>"
970	}
971
972	func TestProbePassesOnAnIsolatedNamespace(t *testing.T) {
973	r := runChecks(goodEnv(t))
974	if !r.Ready {
975	t.Fatalf("not ready: %+v", r.Lines)
976	}
977	for _, c := range []string{"data-tree", "run-tree", "var-empty", "pid1", "cgroup-ro", "etc-carlos", "symlink-eloop", "nosymfollow", "imds-v4", "imds-v6", "loopback", "imds-tcp"} {
978	if v := verdict(r, c); v != "ok" {
979	t.Errorf("%s = %s", c, v)
980	}
981	}
982	}
983
984	func TestProbeFailsWhenASiblingIsVisible(t *testing.T) {
985	e := goodEnv(t)
986	if err := os.MkdirAll(filepath.Join(e.dataRoot, "instances", "other.example.com"), 0o700); err != nil {
987	t.Fatal(err)
988	}
989	r := runChecks(e)
990	if r.Ready || verdict(r, "data-tree") != "fail" {
991	t.Errorf("sibling visible yet ready=%v data-tree=%s", r.Ready, verdict(r, "data-tree"))
992	}
993	}
994
995	func TestProbeFailsWhenCgroupfsIsWritable(t *testing.T) {
996	e := goodEnv(t)
997	e.writable = func(string) error { return nil }
998	r := runChecks(e)
999	if r.Ready || verdict(r, "cgroup-ro") != "fail" {
1000	t.Errorf("writable cgroupfs yet ready=%v", r.Ready)
1001	}
1002	}
1003
1004	func TestProbeFailsWhenEtcCarlosIsReadable(t *testing.T) {
1005	e := goodEnv(t)
1006	if err := os.MkdirAll(e.etcCarlos, 0o755); err != nil {
1007	t.Fatal(err)
1008	}
1009	if err := os.WriteFile(filepath.Join(e.etcCarlos, "probe-secret"), []byte("probe"), 0o644); err != nil {
1010	t.Fatal(err)
1011	}
1012	r := runChecks(e)
1013	if r.Ready || verdict(r, "etc-carlos") != "fail" {
1014	t.Errorf("readable /etc/carlos yet ready=%v", r.Ready)
1015	}
1016	}
1017
1018	func TestProbeIMDSVerdicts(t *testing.T) {
1019	// v4 must be EPERM; anything else is a failure, including a timeout.
1020	e := goodEnv(t)
1021	e.sendUDP = func(_, addr string) error {
1022	if addr == "169.254.169.254:80" {
1023	return syscall.ETIMEDOUT
1024	}
1025	return syscall.EPERM
1026	}
1027	if r := runChecks(e); r.Ready || verdict(r, "imds-v4") != "fail" {
1028	t.Errorf("v4 timeout: ready=%v v4=%s", r.Ready, verdict(r, "imds-v4"))
1029	}
1030	// v6 ENETUNREACH is unverified and does not block readiness.
1031	e = goodEnv(t)
1032	e.sendUDP = func(_, addr string) error {
1033	if addr == "[fd00:ec2::254]:80" {
1034	return syscall.ENETUNREACH
1035	}
1036	if addr == "169.254.169.254:80" {
1037	return syscall.EPERM
1038	}
1039	return nil
1040	}
1041	if r := runChecks(e); !r.Ready || verdict(r, "imds-v6") != "unverified" {
1042	t.Errorf("v6 no route: ready=%v v6=%s", r.Ready, verdict(r, "imds-v6"))
1043	}
1044	// A successful send to either address is a failure.
1045	e = goodEnv(t)
1046	e.sendUDP = func(string, string) error { return nil }
1047	if r := runChecks(e); r.Ready || verdict(r, "imds-v4") != "fail" {
1048	t.Errorf("v4 success: ready=%v", r.Ready)
1049	}
1050	// A dead loopback means the stack, not the filter, said no.
1051	e = goodEnv(t)
1052	e.sendUDP = func(_, addr string) error {
1053	if addr == "127.0.0.1:9" {
1054	return syscall.ENETUNREACH
1055	}
1056	return syscall.EPERM
1057	}
1058	if r := runChecks(e); r.Ready || verdict(r, "loopback") != "fail" {
1059	t.Errorf("dead loopback: ready=%v", r.Ready)
1060	}
1061	// A TCP connect that succeeds is a failure.
1062	e = goodEnv(t)
1063	e.connectTCP = func(string, time.Duration) error { return nil }
1064	if r := runChecks(e); r.Ready || verdict(r, "imds-tcp") != "fail" {
1065	t.Errorf("tcp success: ready=%v", r.Ready)
1066	}
1067	}
1068
1069	func TestProbeSymlinkCheckUsesARealSymlink(t *testing.T) {
1070	// The fixture's data tree is a plain temp dir with no nosymfollow, so
1071	// the real symlink check FAILS here: that proves the check is real.
1072	e := goodEnv(t)
1073	r := runChecks(e)
1074	_ = r
1075	if !errors.Is(realSymlinkCheck(filepath.Join(e.dataRoot, "instances", "_probe")), errNotELOOP) {
1076	t.Error("a followable symlink was not reported")
1077	}
1078	}
1079	```
1080
1081	- [ ] **Step 2: Run the tests to verify they fail**
1082
1083	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -run TestProbe -count=1`
1084	Expected: build failure, `undefined: probeEnv`.
1085
1086	- [ ] **Step 3: Write `probe.go`, `probe_linux.go`, `probe_other.go`**
1087
1088	`probe.go`:
1089
1090	```go
1091	package instanceexec
1092
1093	import (
1094	"encoding/json"
1095	"errors"
1096	"flag"
1097	"fmt"
1098	"os"
1099	"path/filepath"
1100	"sort"
1101	"strconv"
1102	"syscall"
1103	"time"
1104
1105	"github.com/carlosframework/platform/internal/cliflag"
1106	"github.com/carlosframework/platform/internal/instancectl"
1107	"github.com/carlosframework/platform/internal/instancefs"
1108	)
1109
1110	// probeEnv is everything the checks touch, so the verdict logic is testable
1111	// against a fixture tree and fake syscalls while production runs against
1112	// the real namespace.
1113	type probeEnv struct {
1114	dataRoot string // /data as the probe sees it
1115	runRoot string // /run/carlos as the probe sees it
1116	varCache string // /var/cache
1117	varLib string // /var/lib
1118	etcCarlos string // /etc/carlos
1119	procRoot string // /proc
1120	cgroupProcs string // a cgroupfs file to test writability on
1121	pid func() int
1122	writable func(path string) error
1123	sendUDP func(network, addr string) error
1124	connectTCP func(addr string, timeout time.Duration) error
1125	mountNoSym func(mountpoint string) (bool, error)
1126	}
1127
1128	var errNotELOOP = errors.New("symlink was followed")
1129
1130	// RunProbe is `carlos instance-probe`: it runs inside the probe unit's
1131	// namespace (the same text as the tenant templates) and prints a
1132	// ProbeReport on stdout, which the unit's StandardOutput=truncate: lands in
1133	// /run/carlos-control/probe.out for the control service to read (design
1134	// §8.5). It exits 0 whatever the verdict: the verdict is the output, and a
1135	// failed unit would hide it.
1136	func RunProbe(args []string) {
1137	fs := flag.NewFlagSet("instance-probe", flag.ExitOnError)
1138	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-probe")
1139	_ = fs.Parse(args)
1140	r := runChecks(productionProbeEnv())
1141	enc := json.NewEncoder(os.Stdout)
1142	enc.SetIndent("", " ")
1143	_ = enc.Encode(r)
1144	}
1145
1146	func productionProbeEnv() probeEnv {
1147	return probeEnv{
1148	dataRoot: "/data", runRoot: "/run/carlos", varCache: "/var/cache", varLib: "/var/lib",
1149	etcCarlos: "/etc/carlos", procRoot: "/proc", cgroupProcs: "/sys/fs/cgroup/cgroup.procs",
1150	pid: os.Getpid,
1151	writable: accessWritable,
1152	sendUDP: sendOneDatagram,
1153	connectTCP: connectWithTimeout,
1154	mountNoSym: instancefs.MountNoSymfollow,
1155	}
1156	}
1157
1158	func listNames(dir string) ([]string, error) {
1159	entries, err := os.ReadDir(dir)
1160	if err != nil {
1161	return nil, err
1162	}
1163	names := make([]string, 0, len(entries))
1164	for _, e := range entries {
1165	names = append(names, e.Name())
1166	}
1167	sort.Strings(names)
1168	return names, nil
1169	}
1170
1171	func exactly(dir string, want ...string) (string, bool) {
1172	got, err := listNames(dir)
1173	if err != nil {
1174	return err.Error(), false
1175	}
1176	sort.Strings(want)
1177	if fmt.Sprint(got) != fmt.Sprint(want) {
1178	return fmt.Sprintf("%s lists %v, want %v", dir, got, want), false
1179	}
1180	return "", true
1181	}
1182
1183	// runChecks is the verdict. Each line names the property it checks; "fail"
1184	// blocks readiness, "unverified" does not, "ok" is what a ready box shows
1185	// on every line.
1186	func runChecks(e probeEnv) instancectl.ProbeReport {
1187	var r instancectl.ProbeReport
1188	add := func(check, verdict, detail string) {
1189	r.Lines = append(r.Lines, instancectl.VerdictLine{Check: check, Verdict: verdict, Detail: detail})
1190	}
1191	pass := func(check string, detail string, ok bool) {
1192	if ok {
1193	add(check, "ok", "")
1194	} else {
1195	add(check, "fail", detail)
1196	}
1197	}
1198
1199	// The two trees hold exactly the probe's own directory.
1200	d1, ok1 := exactly(e.dataRoot, "instances")
1201	d2, ok2 := exactly(filepath.Join(e.dataRoot, "instances"), instancectl.ProbeDirName)
1202	pass("data-tree", d1+d2, ok1 && ok2)
1203	r1, ok1 := exactly(e.runRoot, "instances")
1204	r2, ok2 := exactly(filepath.Join(e.runRoot, "instances"), instancectl.ProbeDirName)
1205	pass("run-tree", r1+r2, ok1 && ok2)
1206
1207	// The two trees other carlos-uid units write are empty.
1208	v1, ok1 := exactly(e.varCache)
1209	v2, ok2 := exactly(e.varLib)
1210	pass("var-empty", v1+v2, ok1 && ok2)
1211
1212	// A private pid namespace: this process is pid 1 and /proc lists no
1213	// other pid.
1214	pids, err := listNames(e.procRoot)
1215	others := ""
1216	for _, n := range pids {
1217	if v, err := strconv.Atoi(n); err == nil && v != 1 {
1218	others += n + " "
1219	}
1220	}
1221	pass("pid1", fmt.Sprintf("pid=%d others=[%s] err=%v", e.pid(), others, err), err == nil && e.pid() == 1 && others == "")
1222
1223	// cgroupfs is read-only: access(W_OK) answers EROFS without writing.
1224	werr := e.writable(e.cgroupProcs)
1225	pass("cgroup-ro", fmt.Sprintf("access(W_OK) = %v", werr), errors.Is(werr, syscall.EROFS))
1226
1227	// /etc/carlos cannot be opened and its fixture cannot be read.
1228	_, derr := os.ReadDir(e.etcCarlos)
1229	_, ferr := os.ReadFile(filepath.Join(e.etcCarlos, "probe-secret"))
1230	pass("etc-carlos", fmt.Sprintf("readdir=%v readfile=%v", derr, ferr), derr != nil && ferr != nil)
1231
1232	// A symlink planted in the probe's own directory is refused with ELOOP.
1233	serr := realSymlinkCheck(filepath.Join(e.dataRoot, "instances", instancectl.ProbeDirName))
1234	pass("symlink-eloop", fmt.Sprint(serr), serr == nil)
1235
1236	// Both bound directories are on nosymfollow mounts.
1237	n1, err1 := e.mountNoSym(filepath.Join(e.dataRoot, "instances", instancectl.ProbeDirName))
1238	n2, err2 := e.mountNoSym(filepath.Join(e.runRoot, "instances", instancectl.ProbeDirName))
1239	pass("nosymfollow", fmt.Sprintf("data=%v/%v run=%v/%v", n1, err1, n2, err2), n1 && n2 && err1 == nil && err2 == nil)
1240
1241	// IMDS, for a packet filter: a datagram's verdict comes straight back
1242	// from sendmsg. v4 must be EPERM. v6 is EPERM (enforced) or ENETUNREACH
1243	// (no route to that address: unverified, not blocking). The loopback
1244	// control proves the stack is live. A TCP connect must not succeed.
1245	v4 := e.sendUDP("udp4", "169.254.169.254:80")
1246	pass("imds-v4", fmt.Sprintf("sendto = %v", v4), errors.Is(v4, syscall.EPERM))
1247	v6 := e.sendUDP("udp6", "[fd00:ec2::254]:80")
1248	switch {
1249	case errors.Is(v6, syscall.EPERM):
1250	add("imds-v6", "ok", "")
1251	case errors.Is(v6, syscall.ENETUNREACH):
1252	add("imds-v6", "unverified", "no route to the address; deny not exercised")
1253	default:
1254	add("imds-v6", "fail", fmt.Sprintf("sendto = %v", v6))
1255	}
1256	lo := e.sendUDP("udp4", "127.0.0.1:9")
1257	pass("loopback", fmt.Sprintf("sendto = %v", lo), lo == nil)
1258	tcp := e.connectTCP("169.254.169.254:80", 2*time.Second)
1259	pass("imds-tcp", fmt.Sprintf("connect = %v", tcp), tcp != nil)
1260
1261	r.Ready = true
1262	for _, l := range r.Lines {
1263	if l.Verdict == "fail" {
1264	r.Ready = false
1265	}
1266	}
1267	return r
1268	}
1269
1270	// realSymlinkCheck plants a symlink inside dir and tries to open it through
1271	// its name; on a nosymfollow mount the open fails with ELOOP. It returns nil
1272	// for ELOOP, errNotELOOP if the link was followed, and any other error.
1273	func realSymlinkCheck(dir string) error {
1274	target := filepath.Join(dir, "probe-target")
1275	link := filepath.Join(dir, "probe-link")
1276	os.Remove(link)
1277	os.Remove(target)
1278	if err := os.WriteFile(target, []byte("x"), 0o600); err != nil {
1279	return err
1280	}
1281	defer os.Remove(target)
1282	if err := os.Symlink(target, link); err != nil {
1283	return err
1284	}
1285	defer os.Remove(link)
1286	f, err := os.Open(link)
1287	if err == nil {
1288	f.Close()
1289	return errNotELOOP
1290	}
1291	if errors.Is(err, syscall.ELOOP) {
1292	return nil
1293	}
1294	return err
1295	}
1296	```
1297
1298	`probe_linux.go`:
1299
1300	```go
1301	//go:build linux
1302
1303	package instanceexec
1304
1305	import (
1306	"net"
1307	"time"
1308
1309	"golang.org/x/sys/unix"
1310	)
1311
1312	// accessWritable asks the kernel whether path could be opened for writing
1313	// without opening it: on a read-only mount access(2) answers EROFS, which
1314	// is exactly the property ProtectControlGroups= must give a tenant.
1315	func accessWritable(path string) error {
1316	return unix.Access(path, unix.W_OK)
1317	}
1318
1319	// sendOneDatagram sends one empty UDP datagram. With a cgroup BPF egress
1320	// filter (IPAddressDeny=) the filter's verdict is sendmsg's return value,
1321	// EPERM; with no route it is ENETUNREACH; on a live path it is nil.
1322	func sendOneDatagram(network, addr string) error {
1323	c, err := net.Dial(network, addr)
1324	if err != nil {
1325	return unwrapErrno(err)
1326	}
1327	defer c.Close()
1328	_, err = c.Write([]byte{})
1329	return unwrapErrno(err)
1330	}
1331
1332	func connectWithTimeout(addr string, timeout time.Duration) error {
1333	c, err := net.DialTimeout("tcp4", addr, timeout)
1334	if err != nil {
1335	return unwrapErrno(err)
1336	}
1337	c.Close()
1338	return nil
1339	}
1340
1341	// unwrapErrno returns the syscall.Errno inside a net.OpError so callers can
1342	// errors.Is against EPERM and ENETUNREACH; anything else is returned as is.
1343	func unwrapErrno(err error) error {
1344	var errno unix.Errno
1345	if err != nil && errorsAs(err, &errno) {
1346	return errno
1347	}
1348	return err
1349	}
1350	```
1351
1352	Add a tiny helper in the same file to avoid importing `errors` twice:
1353
1354	```go
1355	func errorsAs(err error, target *unix.Errno) bool {
1356	for err != nil {
1357	if e, ok := err.(unix.Errno); ok {
1358	*target = e
1359	return true
1360	}
1361	u, ok := err.(interface{ Unwrap() error })
1362	if !ok {
1363	return false
1364	}
1365	err = u.Unwrap()
1366	}
1367	return false
1368	}
1369	```
1370
1371	(`unix.Errno` is `syscall.Errno`; `errors.Is(errno, syscall.EPERM)` holds.)
1372
1373	`probe_other.go`:
1374
1375	```go
1376	//go:build !linux
1377
1378	package instanceexec
1379
1380	import (
1381	"errors"
1382	"time"
1383	)
1384
1385	func accessWritable(string) error { return errors.ErrUnsupported }
1386	func sendOneDatagram(string, string) error { return errors.ErrUnsupported }
1387	func connectWithTimeout(string, time.Duration) error { return errors.ErrUnsupported }
1388	```
1389
1390	- [ ] **Step 4: Run the tests to verify they pass, and cross-compile**
1391
1392	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instanceexec -count=1 && GOOS=darwin go build ./internal/instanceexec && GOOS=windows go build ./internal/instanceexec`
1393	Expected: PASS and both builds. (`net.Dial("udp4", ...)` to a link-local address in `sendOneDatagram` is never called by the tests; they inject `sendUDP`.)
1394
1395	- [ ] **Step 5: gofmt and commit**
1396
1397	```bash
1398	gofmt -l internal/instanceexec
1399	git add internal/instanceexec
1400	git commit -m "instanceexec: the readiness probe with injectable checks and the EPERM rule for a packet filter (plan 2, task 4)"
1401	```
1402
1403	---
1404
1405	### Task 5: The control service, `carlos instance-control`
1406
1407	**Files:**
1408	- Create: `internal/instancectl/server.go`
1409	- Create: `internal/instancectl/peer_linux.go`
1410	- Create: `internal/instancectl/peer_other.go`
1411	- Create: `internal/instancectl/server_test.go`
1412
1413	**Interfaces:**
1414	- Consumes: Task 1's names and types; `controlsock.Path`; `cgroup.TierDefault`, `cgroup.TierChildren`; `provision.ValidHostShape`; `instancectl.ProbeReport`.
1415	- Produces:
1416	```go
1417	// Run is `carlos instance-control` (root, socket-activated).
1418	func Run(args []string)
1419	type Server struct {
1420	Systemctl func(ctx context.Context, args ...string) (string, error) // production: exec systemctl, CombinedOutput
1421	PeerOK func(conn net.Conn) error // production: SO_PEERCRED uid carlos and pid == MainPID(carlos-edge.service)
1422	CgroupRoot string // "/sys/fs/cgroup"
1423	ProbeOut string // "/run/carlos-control/probe.out"
1424	FixtureOK func() error // stat /etc/carlos/probe-secret is root:carlos 0640
1425	ProbeEvery time.Duration // 5m
1426	Logf func(string, ...any)
1427	}
1428	func (s *Server) Handler() http.Handler
1429	func (s *Server) Serve(ctx context.Context, ln net.Listener) error // accept loop with PeerOK before HTTP
1430	func (s *Server) RunProbe(ctx context.Context) // start the probe unit, read ProbeOut, store the verdict
1431	func (s *Server) Ready() (ProbeReport, bool)
1432	func WaitCgroupEmpty(ctx context.Context, root, slice, unit string, every time.Duration) error
1433	```
1434
1435	- [ ] **Step 1: Write the failing tests**
1436
1437	Create `internal/instancectl/server_test.go`:
1438
1439	```go
1440	package instancectl
1441
1442	import (
1443	"context"
1444	"encoding/json"
1445	"errors"
1446	"net"
1447	"net/http"
1448	"os"
1449	"path/filepath"
1450	"strings"
1451	"sync"
1452	"testing"
1453	"time"
1454	)
1455
1456	type fakeCtl struct {
1457	mu sync.Mutex
1458	calls []string
1459	// answers maps a leading argument pattern ("show -p InvocationID") to output.
1460	answers map[string]string
1461	fail map[string]error
1462	}
1463
1464	func (f *fakeCtl) run(_ context.Context, args ...string) (string, error) {
1465	f.mu.Lock()
1466	defer f.mu.Unlock()
1467	line := strings.Join(args, " ")
1468	f.calls = append(f.calls, line)
1469	for k, err := range f.fail {
1470	if strings.HasPrefix(line, k) {
1471	return "", err
1472	}
1473	}
1474	for k, out := range f.answers {
1475	if strings.HasPrefix(line, k) {
1476	return out, nil
1477	}
1478	}
1479	return "", nil
1480	}
1481
1482	func (f *fakeCtl) joined() string {
1483	f.mu.Lock()
1484	defer f.mu.Unlock()
1485	return strings.Join(f.calls, "\n")
1486	}
1487
1488	func newServer(t *testing.T, ctl *fakeCtl) (*Server, *http.Client) {
1489	t.Helper()
1490	root := t.TempDir()
1491	s := &Server{
1492	Systemctl: ctl.run,
1493	PeerOK: func(net.Conn) error { return nil },
1494	CgroupRoot: filepath.Join(root, "cgroup"),
1495	ProbeOut: filepath.Join(root, "probe.out"),
1496	FixtureOK: func() error { return nil },
1497	ProbeEvery: time.Hour,
1498	Logf: t.Logf,
1499	}
1500	dir, err := os.MkdirTemp("", "ic")
1501	if err != nil {
1502	t.Fatal(err)
1503	}
1504	t.Cleanup(func() { os.RemoveAll(dir) })
1505	sock := filepath.Join(dir, "c.sock")
1506	ln, err := net.Listen("unix", sock)
1507	if err != nil {
1508	t.Skipf("cannot bind a unix socket here (%v); run with TMPDIR=/tmp/g outside the sandbox", err)
1509	}
1510	ctx, cancel := context.WithCancel(context.Background())
1511	t.Cleanup(cancel)
1512	go s.Serve(ctx, ln) //nolint:errcheck
1513	t.Cleanup(func() { ln.Close() })
1514	client := &http.Client{Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
1515	return (&net.Dialer{}).DialContext(ctx, "unix", sock)
1516	}}}
1517	return s, client
1518	}
1519
1520	func post(t *testing.T, c *http.Client, path string) *http.Response {
1521	t.Helper()
1522	resp, err := c.Post("http://unix"+path, "", nil)
1523	if err != nil {
1524	t.Fatalf("POST %s: %v", path, err)
1525	}
1526	return resp
1527	}
1528
1529	func TestStartSetsTheTierThenStartsThenReadsTheInvocation(t *testing.T) {
1530	ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "abc123\n"}}
1531	_, c := newServer(t, ctl)
1532	resp := post(t, c, "/start?host=bill-2.example.com&role=run&memory_max=536870912")
1533	if resp.StatusCode != 200 {
1534	t.Fatalf("status %d", resp.StatusCode)
1535	}
1536	var out StartResponse
1537	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || out.InvocationID != "abc123" {
1538	t.Errorf("response = %+v, %v", out, err)
1539	}
1540	want := "set-property --runtime carlos-instances-bill\\x2d2.example.com.slice MemoryMax=536870912 MemoryHigh=483183820 MemorySwapMax=536870912\n" +
1541	"start carlos-instance@bill\\x2d2.example.com.service\n" +
1542	"show -p InvocationID --value carlos-instance@bill\\x2d2.example.com.service"
1543	if got := ctl.joined(); got != want {
1544	t.Errorf("systemctl calls:\n%s\nwant:\n%s", got, want)
1545	}
1546	}
1547
1548	func TestStartRefusesBadInputsWithoutCallingSystemctl(t *testing.T) {
1549	ctl := &fakeCtl{}
1550	_, c := newServer(t, ctl)
1551	for _, q := range []string{
1552	"host=Bad_Host&role=run&memory_max=536870912",
1553	"host=h.example.com&role=exec&memory_max=536870912",
1554	"host=h.example.com&role=run&memory_max=1",
1555	"host=h.example.com&role=seed&memory_max=536870912", // seed is not a /start role
1556	"role=run&memory_max=536870912",
1557	} {
1558	resp := post(t, c, "/start?"+q)
1559	if resp.StatusCode != 400 {
1560	t.Errorf("%s: status %d, want 400", q, resp.StatusCode)
1561	}
1562	}
1563	if ctl.joined() != "" {
1564	t.Errorf("systemctl was called: %s", ctl.joined())
1565	}
1566	}
1567
1568	func TestStartReportsSystemctlFailureAs502(t *testing.T) {
1569	ctl := &fakeCtl{fail: map[string]error{"start ": errors.New("Job for carlos-instance@h.example.com.service failed")}}
1570	_, c := newServer(t, ctl)
1571	resp := post(t, c, "/start?host=h.example.com&role=sidecar&memory_max=1073741824")
1572	if resp.StatusCode != 502 {
1573	t.Errorf("status %d, want 502", resp.StatusCode)
1574	}
1575	if !strings.Contains(ctl.joined(), "MemoryMax=1073741824 MemoryHigh=966367641 MemorySwapMax=1073741824") {
1576	t.Errorf("children tier not applied: %s", ctl.joined())
1577	}
1578	}
1579
1580	func TestSeedBlocksReadsTheRetainedResultWaitsAndStops(t *testing.T) {
1581	ctl := &fakeCtl{answers: map[string]string{
1582	"show -p InvocationID,Result,ExecMainStatus,ExecMainCode": "InvocationID=feed\nResult=success\nExecMainStatus=0\nExecMainCode=exited\n",
1583	}}
1584	s, c := newServer(t, ctl)
1585	// An empty cgroup: cgroup.events says populated 0.
1586	ev := filepath.Join(s.CgroupRoot, "carlos.slice", "carlos-instances.slice", "carlos-instances-h.example.com.slice", "carlos-instance-seed@h.example.com.service", "cgroup.events")
1587	if err := os.MkdirAll(filepath.Dir(ev), 0o755); err != nil {
1588	t.Fatal(err)
1589	}
1590	if err := os.WriteFile(ev, []byte("populated 0\nfrozen 0\n"), 0o644); err != nil {
1591	t.Fatal(err)
1592	}
1593	resp := post(t, c, "/seed?host=h.example.com&memory_max=536870912")
1594	if resp.StatusCode != 200 {
1595	t.Fatalf("status %d", resp.StatusCode)
1596	}
1597	var out SeedResponse
1598	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || out.InvocationID != "feed" || out.Result != "success" || out.Status != "0" || out.Code != "exited" {
1599	t.Errorf("response = %+v, %v", out, err)
1600	}
1601	calls := ctl.joined()
1602	for _, want := range []string{"set-property --runtime carlos-instances-h.example.com.slice", "start carlos-instance-seed@h.example.com.service", "show -p InvocationID,Result,ExecMainStatus,ExecMainCode carlos-instance-seed@h.example.com.service", "stop carlos-instance-seed@h.example.com.service"} {
1603	if !strings.Contains(calls, want) {
1604	t.Errorf("missing %q in\n%s", want, calls)
1605	}
1606	}
1607	if strings.Index(calls, "show -p") > strings.Index(calls, "stop ") {
1608	t.Errorf("stop before show:\n%s", calls)
1609	}
1610	}
1611
1612	func TestSeedFailsWhenTheCgroupStaysPopulated(t *testing.T) {
1613	ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID,Result": "InvocationID=feed\nResult=success\nExecMainStatus=0\nExecMainCode=exited\n"}}
1614	s, c := newServer(t, ctl)
1615	s.cgroupWaitBudget = 200 * time.Millisecond
1616	ev := filepath.Join(s.CgroupRoot, "carlos.slice", "carlos-instances.slice", "carlos-instances-h.example.com.slice", "carlos-instance-seed@h.example.com.service", "cgroup.events")
1617	if err := os.MkdirAll(filepath.Dir(ev), 0o755); err != nil {
1618	t.Fatal(err)
1619	}
1620	if err := os.WriteFile(ev, []byte("populated 1\n"), 0o644); err != nil {
1621	t.Fatal(err)
1622	}
1623	resp := post(t, c, "/seed?host=h.example.com&memory_max=536870912")
1624	if resp.StatusCode != 502 {
1625	t.Errorf("status %d, want 502 while processes remain", resp.StatusCode)
1626	}
1627	}
1628
1629	func TestStopResetAndTheirTolerances(t *testing.T) {
1630	ctl := &fakeCtl{fail: map[string]error{"reset-failed carlos-instance-sidecar@": errors.New("Unit carlos-instance-sidecar@h.example.com.service not loaded.")}}
1631	_, c := newServer(t, ctl)
1632	if resp := post(t, c, "/stop?host=h.example.com&role=seed"); resp.StatusCode != 204 {
1633	t.Errorf("stop status %d", resp.StatusCode)
1634	}
1635	if resp := post(t, c, "/reset?host=h.example.com&role=sidecar"); resp.StatusCode != 204 {
1636	t.Errorf("reset of an unloaded unit: status %d, want 204 (tolerated)", resp.StatusCode)
1637	}
1638	if resp := post(t, c, "/reset?host=h.example.com&role=nope"); resp.StatusCode != 400 {
1639	t.Errorf("bad role: status %d", resp.StatusCode)
1640	}
1641	if !strings.Contains(ctl.joined(), "stop carlos-instance-seed@h.example.com.service") {
1642	t.Errorf("stop not issued: %s", ctl.joined())
1643	}
1644	}
1645
1646	func TestSweepStopsEveryInstanceUnitAndJob(t *testing.T) {
1647	ctl := &fakeCtl{answers: map[string]string{
1648	"list-units": "carlos-instance@a.example.com.service loaded active running carlos exec-backed instance a.example.com\n" +
1649	"carlos-instance-sidecar@a.example.com.service loaded inactive dead ...\n" +
1650	"carlos-instance-seed@_aaaa\\x2d0123456789abcdef.service loaded activating start ...\n",
1651	"list-jobs": "12 carlos-instance@b.example.com.service start waiting\n",
1652	"show -p ActiveState --value": "inactive\n",
1653	}}
1654	s, c := newServer(t, ctl)
1655	s.cgroupWaitBudget = time.Second
1656	resp := post(t, c, "/sweep")
1657	if resp.StatusCode != 200 {
1658	t.Fatalf("status %d", resp.StatusCode)
1659	}
1660	var out SweepResponse
1661	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
1662	t.Fatal(err)
1663	}
1664	got := strings.Join(out.Units, " ")
1665	for _, want := range []string{"carlos-instance@a.example.com.service", "carlos-instance-seed@_aaaa\\x2d0123456789abcdef.service", "carlos-instance@b.example.com.service"} {
1666	if !strings.Contains(got, want) {
1667	t.Errorf("sweep did not stop %s: %v", want, out.Units)
1668	}
1669	}
1670	if strings.Contains(got, "sidecar@a") {
1671	t.Errorf("sweep stopped an inactive unit with no job: %v", out.Units)
1672	}
1673	calls := ctl.joined()
1674	if !strings.Contains(calls, "stop carlos-instance@a.example.com.service") || !strings.Contains(calls, "stop carlos-instance@b.example.com.service") {
1675	t.Errorf("stops not issued:\n%s", calls)
1676	}
1677	}
1678
1679	func TestReadyReflectsTheProbeVerdict(t *testing.T) {
1680	ctl := &fakeCtl{}
1681	s, c := newServer(t, ctl)
1682	// The fake "start carlos-instance-probe.service" writes the report the
1683	// unit's StandardOutput=truncate: would.
1684	report := ProbeReport{Ready: false, Lines: []VerdictLine{{Check: "pid1", Verdict: "fail", Detail: "pid=4242"}}}
1685	ctl.answers = map[string]string{}
1686	ctl.fail = map[string]error{}
1687	writeReport := func(r ProbeReport) {
1688	b, _ := json.Marshal(r)
1689	if err := os.WriteFile(s.ProbeOut, b, 0o644); err != nil {
1690	t.Fatal(err)
1691	}
1692	}
1693	writeReport(report)
1694	s.RunProbe(context.Background())
1695	resp, err := c.Get("http://unix/ready")
1696	if err != nil || resp.StatusCode != 503 {
1697	t.Fatalf("not-ready: %v %v", resp, err)
1698	}
1699	var got ProbeReport
1700	if err := json.NewDecoder(resp.Body).Decode(&got); err != nil || got.Ready || len(got.Lines) != 1 {
1701	t.Errorf("body = %+v, %v", got, err)
1702	}
1703	writeReport(ProbeReport{Ready: true, Lines: []VerdictLine{{Check: "pid1", Verdict: "ok"}}})
1704	s.RunProbe(context.Background())
1705	if resp, err := c.Get("http://unix/ready"); err != nil || resp.StatusCode != 204 {
1706	t.Fatalf("ready: %v %v", resp, err)
1707	}
1708	if !strings.Contains(ctl.joined(), "start carlos-instance-probe.service") {
1709	t.Errorf("probe unit not started: %s", ctl.joined())
1710	}
1711	// A missing fixture is never a pass.
1712	s.FixtureOK = func() error { return errors.New("probe fixture absent") }
1713	s.RunProbe(context.Background())
1714	if resp, _ := c.Get("http://unix/ready"); resp.StatusCode != 503 {
1715	t.Errorf("fixture absent yet ready")
1716	}
1717	}
1718
1719	func TestPeerCheckClosesTheConnectionWithoutAResponse(t *testing.T) {
1720	ctl := &fakeCtl{}
1721	s, c := newServer(t, ctl)
1722	s.PeerOK = func(net.Conn) error { return errors.New("not the agent") }
1723	_, err := c.Get("http://unix/ready")
1724	if err == nil {
1725	t.Fatal("a refused peer got an HTTP response")
1726	}
1727	if ctl.joined() != "" {
1728	t.Errorf("systemctl ran for a refused peer: %s", ctl.joined())
1729	}
1730	}
1731
1732	func TestWaitCgroupEmptyTreatsAnAbsentCgroupAsEmpty(t *testing.T) {
1733	root := t.TempDir()
1734	if err := WaitCgroupEmpty(context.Background(), root, "carlos-instances-h.slice", "carlos-instance@h.service", 10*time.Millisecond); err != nil {
1735	t.Errorf("absent cgroup: %v", err)
1736	}
1737	}
1738	```
1739
1740	- [ ] **Step 2: Run the tests to verify they fail**
1741
1742	Run: `TMPDIR=/tmp/g GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instancectl -run 'TestStart|TestSeed|TestStop|TestSweep|TestReady|TestPeer|TestWaitCgroup' -count=1`
1743	Expected: build failure, `undefined: Server`.
1744
1745	- [ ] **Step 3: Write `server.go`**
1746
1747	```go
1748	package instancectl
1749
1750	import (
1751	"bufio"
1752	"context"
1753	"encoding/json"
1754	"errors"
1755	"flag"
1756	"fmt"
1757	"log"
1758	"net"
1759	"net/http"
1760	"os"
1761	"os/exec"
1762	"path/filepath"
1763	"strconv"
1764	"strings"
1765	"sync"
1766	"time"
1767
1768	"github.com/carlosframework/platform/internal/cgroup"
1769	"github.com/carlosframework/platform/internal/cliflag"
1770	"github.com/carlosframework/platform/internal/controlsock"
1771	"github.com/carlosframework/platform/internal/provision"
1772	)
1773
1774	// Server is the root instance-control service (design §5). It does four
1775	// things and nothing else: starts, stops and sweeps the three templates'
1776	// instances, sets a host slice's memory tier before a start, runs the
1777	// readiness probe, and answers /ready. Every systemctl argument it passes
1778	// is built from a validated host by UnitName/SliceName; nothing from a
1779	// request reaches systemctl as a string.
1780	type Server struct {
1781	// Systemctl runs one systemctl invocation and returns its combined
1782	// output; production is exec.Command, tests record.
1783	Systemctl func(ctx context.Context, args ...string) (string, error)
1784	// PeerOK is asked once per accepted connection, before a byte is read.
1785	// Production requires SO_PEERCRED uid carlos and pid == MainPID of
1786	// carlos-edge.service (design §5.4). A refused connection is closed
1787	// with no response.
1788	PeerOK func(conn net.Conn) error
1789	// CgroupRoot is /sys/fs/cgroup; tests point it at a fixture tree.
1790	CgroupRoot string
1791	// ProbeOut is where the probe unit's StandardOutput=truncate: lands.
1792	ProbeOut string
1793	// FixtureOK verifies /etc/carlos/probe-secret is root:carlos 0640
1794	// before a probe verdict is accepted (design §8.5, codex R11-1).
1795	FixtureOK func() error
1796	// ProbeEvery is the re-run cadence (five minutes in production).
1797	ProbeEvery time.Duration
1798	Logf func(string, ...any)
1799
1800	// cgroupWaitBudget bounds a wait for an empty cgroup; tests shorten it.
1801	cgroupWaitBudget time.Duration
1802
1803	mu sync.Mutex
1804	report ProbeReport
1805	probed bool
1806	}
1807
1808	const (
1809	probeUnit = "carlos-instance-probe.service"
1810	edgeUnit = "carlos-edge.service"
1811	parentSlices = "carlos.slice/carlos-instances.slice"
1812	)
1813
1814	// Run is `carlos instance-control`.
1815	func Run(args []string) {
1816	fs := flag.NewFlagSet("instance-control", flag.ExitOnError)
1817	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-control (socket-activated by carlos-instance-control.socket)")
1818	_ = fs.Parse(args)
1819	if os.Geteuid() != 0 {
1820	log.Fatal("instance-control: this is the root service behind carlos-instance-control.socket; it is not for operators")
1821	}
1822	s := &Server{
1823	Systemctl: func(ctx context.Context, args ...string) (string, error) {
1824	out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
1825	return string(out), err
1826	},
1827	PeerOK: productionPeerOK(),
1828	CgroupRoot: "/sys/fs/cgroup",
1829	ProbeOut: filepath.Join(controlsock.Dir, "probe.out"),
1830	FixtureOK: fixtureIsRootCarlos0640,
1831	ProbeEvery: 5 * time.Minute,
1832	Logf: log.Printf,
1833	}
1834	ln, activated, err := activatedListener()
1835	if err != nil {
1836	log.Fatalf("instance-control: %v", err)
1837	}
1838	if !activated {
1839	// Not socket-activated (a hand run): bind the path ourselves with
1840	// the same ownership the socket unit gives it.
1841	ln, err = bindControlSocket(controlsock.Path)
1842	if err != nil {
1843	log.Fatalf("instance-control: %v", err)
1844	}
1845	}
1846	ctx, cancel := context.WithCancel(context.Background())
1847	defer cancel()
1848	go s.probeLoop(ctx)
1849	if err := s.Serve(ctx, ln); err != nil {
1850	log.Fatalf("instance-control: %v", err)
1851	}
1852	}
1853
1854	// Serve accepts connections, asks PeerOK about each, and hands the accepted
1855	// ones to the HTTP server one at a time through a listener that only ever
1856	// yields vetted connections.
1857	func (s *Server) Serve(ctx context.Context, ln net.Listener) error {
1858	vetted := &vettedListener{Listener: ln, ok: s.PeerOK, logf: s.logf}
1859	srv := &http.Server{Handler: s.Handler(), ReadHeaderTimeout: 5 * time.Second}
1860	go func() {
1861	<-ctx.Done()
1862	srv.Close()
1863	}()
1864	err := srv.Serve(vetted)
1865	if errors.Is(err, http.ErrServerClosed) || ctx.Err() != nil {
1866	return nil
1867	}
1868	return err
1869	}
1870
1871	// vettedListener runs the peer check inside Accept so a refused peer is
1872	// closed before net/http ever reads from it.
1873	type vettedListener struct {
1874	net.Listener
1875	ok func(net.Conn) error
1876	logf func(string, ...any)
1877	}
1878
1879	func (v *vettedListener) Accept() (net.Conn, error) {
1880	for {
1881	c, err := v.Listener.Accept()
1882	if err != nil {
1883	return nil, err
1884	}
1885	if err := v.ok(c); err != nil {
1886	v.logf("instance-control: refused connection: %v", err)
1887	c.Close()
1888	continue
1889	}
1890	return c, nil
1891	}
1892	}
1893
1894	func (s *Server) logf(format string, args ...any) {
1895	if s.Logf != nil {
1896	s.Logf(format, args...)
1897	}
1898	}
1899
1900	func (s *Server) Handler() http.Handler {
1901	mux := http.NewServeMux()
1902	mux.HandleFunc("POST /start", s.handleStart)
1903	mux.HandleFunc("POST /seed", s.handleSeed)
1904	mux.HandleFunc("POST /stop", s.handleStop)
1905	mux.HandleFunc("POST /reset", s.handleReset)
1906	mux.HandleFunc("POST /sweep", s.handleSweep)
1907	mux.HandleFunc("GET /ready", s.handleReady)
1908	return mux
1909	}
1910
1911	// request is the validated form of a verb's query: nothing else reaches a
1912	// unit name.
1913	type request struct {
1914	host string
1915	role Role
1916	tier cgroup.Limit
1917	}
1918
1919	func parseRequest(r *http.Request, roles []Role, wantTier bool) (request, error) {
1920	q := r.URL.Query()
1921	host := q.Get("host")
1922	if !provision.ValidHostShape(host) {
1923	return request{}, fmt.Errorf("host %q is not a host", host)
1924	}
1925	req := request{host: host}
1926	if len(roles) > 0 {
1927	role, ok := ParseRole(q.Get("role"))
1928	if !ok {
1929	return request{}, fmt.Errorf("role %q is not run, seed or sidecar", q.Get("role"))
1930	}
1931	allowed := false
1932	for _, r := range roles {
1933	allowed = allowed || r == role
1934	}
1935	if !allowed {
1936	return request{}, fmt.Errorf("role %s is not valid for this verb", role)
1937	}
1938	req.role = role
1939	}
1940	if wantTier {
1941	n, err := strconv.ParseInt(q.Get("memory_max"), 10, 64)
1942	if err != nil {
1943	return request{}, fmt.Errorf("memory_max %q is not a number", q.Get("memory_max"))
1944	}
1945	switch n {
1946	case cgroup.TierDefault.MemoryMax, cgroup.TierChildren.MemoryMax:
1947	req.tier = cgroup.Limit{MemoryMax: n}
1948	default:
1949	return request{}, fmt.Errorf("memory_max %d is not a platform tier", n)
1950	}
1951	}
1952	return req, nil
1953	}
1954
1955	func (s *Server) setTier(ctx context.Context, host string, tier cgroup.Limit) error {
1956	max := strconv.FormatInt(tier.MemoryMax, 10)
1957	high := strconv.FormatInt(tier.MemoryMax/10*9, 10)
1958	_, err := s.Systemctl(ctx, "set-property", "--runtime", SliceName(host),
1959	"MemoryMax="+max, "MemoryHigh="+high, "MemorySwapMax="+max)
1960	return err
1961	}
1962
1963	func writeJSON(w http.ResponseWriter, status int, v any) {
1964	w.Header().Set("Content-Type", "application/json")
1965	w.WriteHeader(status)
1966	_ = json.NewEncoder(w).Encode(v)
1967	}
1968
1969	func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {
1970	req, err := parseRequest(r, []Role{RoleRun, RoleSidecar}, true)
1971	if err != nil {
1972	http.Error(w, err.Error(), http.StatusBadRequest)
1973	return
1974	}
1975	ctx := r.Context()
1976	unit := UnitName(req.role, req.host)
1977	if err := s.setTier(ctx, req.host, req.tier); err != nil {
1978	http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
1979	return
1980	}
1981	if out, err := s.Systemctl(ctx, "start", unit); err != nil {
1982	http.Error(w, "start: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
1983	return
1984	}
1985	inv, err := s.Systemctl(ctx, "show", "-p", "InvocationID", "--value", unit)
1986	if err != nil {
1987	http.Error(w, "show: "+err.Error(), http.StatusBadGateway)
1988	return
1989	}
1990	s.logf("instance-control: %s: started %s memory.max=%d", req.host, unit, req.tier.MemoryMax)
1991	writeJSON(w, http.StatusOK, StartResponse{InvocationID: strings.TrimSpace(inv)})
1992	}
1993
1994	// parseShow reads systemctl show's KEY=value lines.
1995	func parseShow(out string) map[string]string {
1996	m := map[string]string{}
1997	sc := bufio.NewScanner(strings.NewReader(out))
1998	for sc.Scan() {
1999	if k, v, ok := strings.Cut(sc.Text(), "="); ok {
2000	m[k] = strings.TrimSpace(v)
2001	}
2002	}
2003	return m
2004	}
2005
2006	func (s *Server) handleSeed(w http.ResponseWriter, r *http.Request) {
2007	req, err := parseRequest(r, nil, true)
2008	if err != nil {
2009	http.Error(w, err.Error(), http.StatusBadRequest)
2010	return
2011	}
2012	ctx := r.Context()
2013	unit := UnitName(RoleSeed, req.host)
2014	if err := s.setTier(ctx, req.host, req.tier); err != nil {
2015	http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
2016	return
2017	}
2018	// A oneshot's start job completes when ExecStart exits; RemainAfterExit
2019	// keeps the unit loaded with its result until the stop below.
2020	startOut, startErr := s.Systemctl(ctx, "start", unit)
2021	out, err := s.Systemctl(ctx, "show", "-p", "InvocationID,Result,ExecMainStatus,ExecMainCode", unit)
2022	if err != nil {
2023	http.Error(w, "show: "+err.Error(), http.StatusBadGateway)
2024	return
2025	}
2026	m := parseShow(out)
2027	if err := WaitCgroupEmpty(ctx, s.CgroupRoot, SliceName(req.host), unit, s.waitBudget()); err != nil {
2028	http.Error(w, "seed: "+err.Error(), http.StatusBadGateway)
2029	return
2030	}
2031	if _, err := s.Systemctl(ctx, "stop", unit); err != nil {
2032	s.logf("instance-control: %s: stop after seed: %v", req.host, err)
2033	}
2034	if startErr != nil && m["Result"] == "" {
2035	http.Error(w, "start: "+strings.TrimSpace(startOut+" "+startErr.Error()), http.StatusBadGateway)
2036	return
2037	}
2038	writeJSON(w, http.StatusOK, SeedResponse{InvocationID: m["InvocationID"], Result: m["Result"], Status: m["ExecMainStatus"], Code: m["ExecMainCode"]})
2039	}
2040
2041	func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {
2042	req, err := parseRequest(r, []Role{RoleRun, RoleSeed, RoleSidecar}, false)
2043	if err != nil {
2044	http.Error(w, err.Error(), http.StatusBadRequest)
2045	return
2046	}
2047	if out, err := s.Systemctl(r.Context(), "stop", UnitName(req.role, req.host)); err != nil {
2048	http.Error(w, "stop: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
2049	return
2050	}
2051	w.WriteHeader(http.StatusNoContent)
2052	}
2053
2054	func (s *Server) handleReset(w http.ResponseWriter, r *http.Request) {
2055	req, err := parseRequest(r, []Role{RoleRun, RoleSeed, RoleSidecar}, false)
2056	if err != nil {
2057	http.Error(w, err.Error(), http.StatusBadRequest)
2058	return
2059	}
2060	if out, err := s.Systemctl(r.Context(), "reset-failed", UnitName(req.role, req.host)); err != nil && !strings.Contains(out+err.Error(), "not loaded") {
2061	http.Error(w, "reset-failed: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
2062	return
2063	}
2064	w.WriteHeader(http.StatusNoContent)
2065	}
2066
2067	// handleSweep stops every instance of the three templates that is not
2068	// inactive or has a job, keyed on the names systemd reports, so a unit
2069	// whose identifier is hashed and whose row is gone is still stopped
2070	// (design §6.7, codex R7-1).
2071	func (s *Server) handleSweep(w http.ResponseWriter, r *http.Request) {
2072	ctx := r.Context()
2073	targets := map[string]bool{}
2074	out, err := s.Systemctl(ctx, "list-units", "--all", "--plain", "--no-legend", "--no-pager",
2075	UnitPrefixRun+"*.service", UnitPrefixSeed+"*.service", UnitPrefixSidecar+"*.service")
2076	if err != nil {
2077	http.Error(w, "list-units: "+err.Error(), http.StatusBadGateway)
2078	return
2079	}
2080	for _, line := range strings.Split(out, "\n") {
2081	f := strings.Fields(line)
2082	if len(f) < 3 {
2083	continue
2084	}
2085	if _, _, ok := ParseUnitName(f[0]); !ok {
2086	continue
2087	}
2088	if f[2] != "inactive" {
2089	targets[f[0]] = true
2090	}
2091	}
2092	jobs, err := s.Systemctl(ctx, "list-jobs", "--plain", "--no-legend", "--no-pager")
2093	if err != nil {
2094	http.Error(w, "list-jobs: "+err.Error(), http.StatusBadGateway)
2095	return
2096	}
2097	for _, line := range strings.Split(jobs, "\n") {
2098	f := strings.Fields(line)
2099	if len(f) < 2 {
2100	continue
2101	}
2102	if _, _, ok := ParseUnitName(f[1]); ok {
2103	targets[f[1]] = true
2104	}
2105	}
2106	var stopped []string
2107	for unit := range targets {
2108	if _, err := s.Systemctl(ctx, "stop", unit); err != nil {
2109	s.logf("instance-control: sweep: stop %s: %v", unit, err)
2110	}
2111	_, esc, _ := ParseUnitName(unit)
2112	if err := WaitCgroupEmpty(ctx, s.CgroupRoot, "carlos-instances-"+esc+".slice", unit, s.waitBudget()); err != nil {
2113	http.Error(w, "sweep: "+unit+": "+err.Error(), http.StatusBadGateway)
2114	return
2115	}
2116	stopped = append(stopped, unit)
2117	}
2118	writeJSON(w, http.StatusOK, SweepResponse{Units: stopped})
2119	}
2120
2121	func (s *Server) handleReady(w http.ResponseWriter, r *http.Request) {
2122	rep, ready := s.Ready()
2123	if ready {
2124	w.WriteHeader(http.StatusNoContent)
2125	return
2126	}
2127	writeJSON(w, http.StatusServiceUnavailable, rep)
2128	}
2129
2130	// Ready returns the last probe verdict. Before the first probe, or when
2131	// the last one could not be read, the box is not ready.
2132	func (s *Server) Ready() (ProbeReport, bool) {
2133	s.mu.Lock()
2134	defer s.mu.Unlock()
2135	return s.report, s.probed && s.report.Ready
2136	}
2137
2138	func (s *Server) waitBudget() time.Duration {
2139	if s.cgroupWaitBudget > 0 {
2140	return s.cgroupWaitBudget
2141	}
2142	return 30 * time.Second
2143	}
2144
2145	// WaitCgroupEmpty waits until the unit's cgroup under the host slice is
2146	// absent or reports "populated 0" in cgroup.events, polling every `every`,
2147	// bounded by ctx and the caller's budget (design §6.5, codex R2-5).
2148	func WaitCgroupEmpty(ctx context.Context, root, slice, unit string, every time.Duration) error {
2149	events := filepath.Join(root, parentSlices, slice, unit, "cgroup.events")
2150	deadline, hasDeadline := ctx.Deadline()
2151	_ = deadline
2152	_ = hasDeadline
2153	for {
2154	b, err := os.ReadFile(events)
2155	if errors.Is(err, os.ErrNotExist) {
2156	return nil
2157	}
2158	if err == nil && strings.Contains(string(b), "populated 0") {
2159	return nil
2160	}
2161	select {
2162	case <-ctx.Done():
2163	return fmt.Errorf("processes remain in %s", unit)
2164	case <-time.After(every):
2165	}
2166	}
2167	}
2168
2169	// probeLoop runs the probe at start and every ProbeEvery.
2170	func (s *Server) probeLoop(ctx context.Context) {
2171	s.RunProbe(ctx)
2172	t := time.NewTicker(s.ProbeEvery)
2173	defer t.Stop()
2174	for {
2175	select {
2176	case <-ctx.Done():
2177	return
2178	case <-t.C:
2179	s.RunProbe(ctx)
2180	}
2181	}
2182	}
2183
2184	// RunProbe starts the probe unit (a oneshot: start blocks until it exits),
2185	// reads the report it left, and stores the verdict. A fixture that is not
2186	// root:carlos 0640, a probe that could not be started, or a report that
2187	// could not be read all leave the box not ready, with the reason on a
2188	// verdict line; readiness is never the default.
2189	func (s *Server) RunProbe(ctx context.Context) {
2190	set := func(r ProbeReport) {
2191	s.mu.Lock()
2192	s.report, s.probed = r, true
2193	s.mu.Unlock()
2194	}
2195	if err := s.FixtureOK(); err != nil {
2196	set(ProbeReport{Lines: []VerdictLine{{Check: "fixture", Verdict: "fail", Detail: err.Error()}}})
2197	return
2198	}
2199	if out, err := s.Systemctl(ctx, "start", probeUnit); err != nil {
2200	set(ProbeReport{Lines: []VerdictLine{{Check: "probe-unit", Verdict: "fail", Detail: strings.TrimSpace(out + " " + err.Error())}}})
2201	return
2202	}
2203	b, err := os.ReadFile(s.ProbeOut)
2204	if err != nil {
2205	set(ProbeReport{Lines: []VerdictLine{{Check: "probe-output", Verdict: "fail", Detail: err.Error()}}})
2206	return
2207	}
2208	var r ProbeReport
2209	if err := json.Unmarshal(b, &r); err != nil {
2210	set(ProbeReport{Lines: []VerdictLine{{Check: "probe-output", Verdict: "fail", Detail: err.Error()}}})
2211	return
2212	}
2213	for _, l := range r.Lines {
2214	if l.Verdict != "ok" {
2215	s.logf("instance-control: probe: %s %s %s", l.Check, l.Verdict, l.Detail)
2216	}
2217	}
2218	set(r)
2219	}
2220	```
2221
2222	Remove the two `_ = deadline` lines and the `deadline, hasDeadline := ctx.Deadline()` line from `WaitCgroupEmpty` before committing; they are shown only so the function is complete if an implementer wants a deadline log. The seed and sweep handlers must wrap their `ctx` with the budget: in `handleSeed` and `handleSweep`, before calling `WaitCgroupEmpty`, do
2223
2224	```go
2225	wctx, cancel := context.WithTimeout(ctx, s.waitBudget())
2226	defer cancel()
2227	```
2228
2229	and pass `wctx`, with `every` = `200 * time.Millisecond` (not the budget). Update the two call sites accordingly: `WaitCgroupEmpty(wctx, s.CgroupRoot, SliceName(req.host), unit, 200*time.Millisecond)`.
2230
2231	`peer_linux.go`:
2232
2233	```go
2234	//go:build linux
2235
2236	package instancectl
2237
2238	import (
2239	"context"
2240	"errors"
2241	"fmt"
2242	"net"
2243	"os"
2244	"os/exec"
2245	"os/user"
2246	"strconv"
2247	"strings"
2248	"sync"
2249	"time"
2250
2251	"golang.org/x/sys/unix"
2252	)
2253
2254	// peerCred reads SO_PEERCRED from a connected unix socket.
2255	func peerCred(conn net.Conn) (*unix.Ucred, error) {
2256	uc, ok := conn.(*net.UnixConn)
2257	if !ok {
2258	return nil, errors.New("not a unix connection")
2259	}
2260	raw, err := uc.SyscallConn()
2261	if err != nil {
2262	return nil, err
2263	}
2264	var cred *unix.Ucred
2265	var cerr error
2266	if err := raw.Control(func(fd uintptr) {
2267	cred, cerr = unix.GetsockoptUcred(int(fd), unix.SOL_SOCKET, unix.SO_PEERCRED)
2268	}); err != nil {
2269	return nil, err
2270	}
2271	return cred, cerr
2272	}
2273
2274	// productionPeerOK requires the peer to be uid carlos AND the main process
2275	// of carlos-edge.service as systemd reports it (design §5.4): a flat tenant
2276	// is a child of that pid, never the pid itself, and a nested tenant cannot
2277	// reach the socket at all. MainPID is asked of systemctl and cached for one
2278	// second, because the agent restarts and the pid changes.
2279	func productionPeerOK() func(net.Conn) error {
2280	u, err := user.Lookup("carlos")
2281	if err != nil {
2282	return func(net.Conn) error { return fmt.Errorf("no carlos user: %v", err) }
2283	}
2284	carlosUID, _ := strconv.Atoi(u.Uid)
2285	var mu sync.Mutex
2286	var mainPID int
2287	var at time.Time
2288	edgePID := func() (int, error) {
2289	mu.Lock()
2290	defer mu.Unlock()
2291	if time.Since(at) < time.Second && mainPID > 0 {
2292	return mainPID, nil
2293	}
2294	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
2295	defer cancel()
2296	out, err := exec.CommandContext(ctx, "systemctl", "show", "-p", "MainPID", "--value", edgeUnit).Output()
2297	if err != nil {
2298	return 0, err
2299	}
2300	pid, err := strconv.Atoi(strings.TrimSpace(string(out)))
2301	if err != nil || pid <= 0 {
2302	return 0, fmt.Errorf("carlos-edge.service has no MainPID")
2303	}
2304	mainPID, at = pid, time.Now()
2305	return pid, nil
2306	}
2307	return func(conn net.Conn) error {
2308	cred, err := peerCred(conn)
2309	if err != nil {
2310	return err
2311	}
2312	if int(cred.Uid) != carlosUID {
2313	return fmt.Errorf("peer uid %d is not carlos", cred.Uid)
2314	}
2315	pid, err := edgePID()
2316	if err != nil {
2317	return err
2318	}
2319	if int(cred.Pid) != pid {
2320	return fmt.Errorf("peer pid %d is not the edge's main pid %d", cred.Pid, pid)
2321	}
2322	return nil
2323	}
2324	}
2325
2326	// activatedListener returns the socket-activated listener when systemd
2327	// passed one (LISTEN_PID is us, LISTEN_FDS is 1, the fd is 3).
2328	func activatedListener() (net.Listener, bool, error) {
2329	if os.Getenv("LISTEN_PID") != strconv.Itoa(os.Getpid()) {
2330	return nil, false, nil
2331	}
2332	if os.Getenv("LISTEN_FDS") != "1" {
2333	return nil, false, fmt.Errorf("LISTEN_FDS=%q, want 1", os.Getenv("LISTEN_FDS"))
2334	}
2335	f := os.NewFile(3, "control.sock")
2336	defer f.Close()
2337	ln, err := net.FileListener(f)
2338	if err != nil {
2339	return nil, false, err
2340	}
2341	return ln, true, nil
2342	}
2343
2344	// bindControlSocket is the hand-run fallback: bind the path with the
2345	// ownership the socket unit would give it (carlos, 0600).
2346	func bindControlSocket(path string) (net.Listener, error) {
2347	_ = os.Remove(path)
2348	ln, err := net.Listen("unix", path)
2349	if err != nil {
2350	return nil, err
2351	}
2352	u, err := user.Lookup("carlos")
2353	if err != nil {
2354	ln.Close()
2355	return nil, err
2356	}
2357	uid, _ := strconv.Atoi(u.Uid)
2358	gid, _ := strconv.Atoi(u.Gid)
2359	if err := os.Chown(path, uid, gid); err != nil {
2360	ln.Close()
2361	return nil, err
2362	}
2363	if err := os.Chmod(path, 0o600); err != nil {
2364	ln.Close()
2365	return nil, err
2366	}
2367	return ln, nil
2368	}
2369
2370	// fixtureIsRootCarlos0640 checks the probe fixture tmpfiles stages.
2371	func fixtureIsRootCarlos0640() error {
2372	fi, err := os.Stat("/etc/carlos/probe-secret")
2373	if err != nil {
2374	return fmt.Errorf("probe fixture absent: %v", err)
2375	}
2376	st, ok := fi.Sys().(*unix.Stat_t)
2377	if !ok {
2378	return errors.New("probe fixture: no stat")
2379	}
2380	g, err := user.LookupGroup("carlos")
2381	if err != nil {
2382	return err
2383	}
2384	gid, _ := strconv.Atoi(g.Gid)
2385	if st.Uid != 0 || int(st.Gid) != gid || fi.Mode().Perm() != 0o640 {
2386	return fmt.Errorf("probe fixture is uid %d gid %d mode %o, want root:carlos 0640", st.Uid, st.Gid, fi.Mode().Perm())
2387	}
2388	return nil
2389	}
2390	```
2391
2392	Note: `fi.Sys().(*unix.Stat_t)` is a `*syscall.Stat_t`; `unix.Stat_t` and `syscall.Stat_t` are distinct types. Use `fi.Sys().(*syscall.Stat_t)` with `import "syscall"` instead. Adjust the import block accordingly.
2393
2394	`peer_other.go`:
2395
2396	```go
2397	//go:build !linux
2398
2399	package instancectl
2400
2401	import (
2402	"errors"
2403	"net"
2404	)
2405
2406	func productionPeerOK() func(net.Conn) error {
2407	return func(net.Conn) error { return errors.ErrUnsupported }
2408	}
2409	func activatedListener() (net.Listener, bool, error) { return nil, false, errors.ErrUnsupported }
2410	func bindControlSocket(string) (net.Listener, error) { return nil, errors.ErrUnsupported }
2411	func fixtureIsRootCarlos0640() error { return errors.ErrUnsupported }
2412	```
2413
2414	- [ ] **Step 4: Run the tests to verify they pass, and cross-compile**
2415
2416	Run: `TMPDIR=/tmp/g GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/instancectl -count=1 -v` (outside the sandbox so the sockets bind)
2417	Expected: PASS, none skipped.
2418
2419	Run: `GOOS=darwin go build ./internal/instancectl && GOOS=windows go build ./internal/instancectl`
2420	Expected: both build.
2421
2422	- [ ] **Step 5: gofmt and commit**
2423
2424	```bash
2425	gofmt -l internal/instancectl
2426	git add internal/instancectl
2427	git commit -m "instancectl: the root control service, peer-authenticated, with the five verbs and the probe loop (plan 2, task 5)"
2428	```
2429
2430	---
2431
2432	### Task 6: Unit files, verb dispatch, bootstrap arming, docs
2433
2434	**Files:**
2435	- Create: `internal/host/units/carlos-instance@.service`, `carlos-instance-seed@.service`, `carlos-instance-sidecar@.service`, `carlos-instance-probe.service`, `carlos-instance-control.socket`, `carlos-instance-control.service`
2436	- Modify: `internal/host/bootstrap.go` (`enabledUnits`; `tmpfilesConf` gains the two `_probe` lines; `dirs`)
2437	- Modify: `internal/host/systemunits.go` (`systemUnitFiles` and the arming list gain the control socket)
2438	- Modify: `main.go` (four `case` arms)
2439	- Modify: `docs/site/cli.md` (four headings under "On the host")
2440	- Test: `internal/host/units_isolation_test.go` (append)
2441
2442	**Interfaces:**
2443	- Consumes: `instancectl.Run`, `instanceexec.RunExec`, `RunExit`, `RunProbe`.
2444	- Produces: the unit names plan 3 starts through the control service.
2445
2446	- [ ] **Step 1: Write the failing tests**
2447
2448	Append to `internal/host/units_isolation_test.go`:
2449
2450	```go
2451	// The three tenant templates and the probe carry one namespace: byte-
2452	// identical hardening blocks, so a property proven by the probe is the
2453	// property a tenant gets (design §4.2, §8.5).
2454	func TestInstanceTemplatesShareOneNamespace(t *testing.T) {
2455	block := func(name string) string {
2456	u := readUnit(t, name)
2457	start := strings.Index(u, "# --- the namespace")
2458	end := strings.Index(u, "# --- per-service resources")
2459	if start < 0 || end < 0 || end < start {
2460	t.Fatalf("%s: namespace block markers missing", name)
2461	}
2462	return u[start:end]
2463	}
2464	ref := block("carlos-instance@.service")
2465	for _, name := range []string{"carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {
2466	if block(name) != ref {
2467	t.Errorf("%s namespace block differs from carlos-instance@.service", name)
2468	}
2469	}
2470	for _, want := range []string{
2471	"ProtectSystem=strict", "TemporaryFileSystem=/data:ro", "TemporaryFileSystem=/run/carlos:ro",
2472	"TemporaryFileSystem=/var/cache:ro", "TemporaryFileSystem=/var/lib:ro", "InaccessiblePaths=/etc/carlos",
2473	"ProtectControlGroups=yes", "ProtectProc=invisible", "ProcSubset=pid", "PrivatePIDs=yes",
2474	"RestrictNamespaces=yes", "NoNewPrivileges=yes", "CapabilityBoundingSet=",
2475	"IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128", "MemoryDenyWriteExecute=yes",
2476	} {
2477	if !strings.Contains(ref, want) {
2478	t.Errorf("namespace block lacks %q", want)
2479	}
2480	}
2481	for name, wants := range map[string][]string{
2482	"carlos-instance@.service": {"ExecStart=/opt/carlos/carlos instance-exec run %I", "ExecStopPost=/opt/carlos/carlos instance-exit run %I",
2483	"BindPaths=/data/instances/%I /run/carlos/instances/%I", "StandardOutput=file:/run/carlos/logs/%I.run.sock",
2484	"Slice=carlos-instances-%i.slice", "Restart=no", "KillMode=mixed", "TimeoutStopSec=20", "BindsTo=carlos-edge.service",
2485	"Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount", "OOMScoreAdjust=500", "OOMPolicy=kill", "TasksMax=256", "Type=simple"},
2486	"carlos-instance-seed@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=120", "ExecStart=/opt/carlos/carlos instance-exec seed %I",
2487	"ExecStopPost=/opt/carlos/carlos instance-exit seed %I", "StandardOutput=file:/run/carlos/logs/%I.seed.sock"},
2488	"carlos-instance-sidecar@.service": {"Type=simple", "ExecStart=/opt/carlos/carlos instance-exec sidecar %I",
2489	"ExecStopPost=/opt/carlos/carlos instance-exit sidecar %I", "StandardOutput=file:/run/carlos/logs/%I.sidecar.sock"},
2490	"carlos-instance-probe.service": {"Type=oneshot", "ExecStart=/opt/carlos/carlos instance-probe",
2491	"BindPaths=/data/instances/_probe /run/carlos/instances/_probe", "StandardOutput=truncate:/run/carlos-control/probe.out"},
2492	} {
2493	u := readUnit(t, name)
2494	for _, w := range wants {
2495	if !strings.Contains(u, w) {
2496	t.Errorf("%s lacks %q", name, w)
2497	}
2498	}
2499	if strings.Contains(u, "ExecStopPost=+") {
2500	t.Errorf("%s uses ExecStopPost=+: the plus prefix keeps the unit's tmpfs (codex R3-2)", name)
2501	}
2502	if name != "carlos-instance-probe.service" && !strings.Contains(u, "User=carlos") {
2503	t.Errorf("%s does not run as carlos", name)
2504	}
2505	}
2506	}
2507
2508	func TestControlUnitsShape(t *testing.T) {
2509	s := readUnit(t, "carlos-instance-control.socket")
2510	for _, want := range []string{"ListenStream=/run/carlos-control/control.sock", "SocketUser=carlos", "SocketGroup=carlos", "SocketMode=0600", "Accept=no", "WantedBy=sockets.target"} {
2511	if !strings.Contains(s, want) {
2512	t.Errorf("socket unit lacks %q", want)
2513	}
2514	}
2515	u := readUnit(t, "carlos-instance-control.service")
2516	for _, want := range []string{"ExecStart=/opt/carlos/carlos instance-control", "Requires=carlos-instance-control.socket",
2517	"data-instances.mount", "ProtectSystem=strict", "ReadWritePaths=/run/carlos-control", "RestrictAddressFamilies=AF_UNIX", "MemoryMax=64M"} {
2518	if !strings.Contains(u, want) {
2519	t.Errorf("control service lacks %q", want)
2520	}
2521	}
2522	for _, never := range []string{"User=", "NoNewPrivileges"} {
2523	if strings.Contains(u, never) {
2524	t.Errorf("control service must run as root for systemctl; found %q", never)
2525	}
2526	}
2527	have := map[string]bool{}
2528	for _, e := range enabledUnits {
2529	have[e] = true
2530	}
2531	if !have["carlos-instance-control.socket"] {
2532	t.Error("enabledUnits lacks the control socket")
2533	}
2534	if have["carlos-instance-control.service"] || have["carlos-instance-probe.service"] || have["carlos-instance@.service"] {
2535	t.Error("a template, the probe or the socket-activated service must not be enabled")
2536	}
2537	for _, want := range []string{"d /data/.instances/_probe 0700 carlos carlos -", "d /run/carlos/.instances/_probe 2750 carlos carlos-sockets -"} {
2538	if !strings.Contains(tmpfilesConf, want+"\n") {
2539	t.Errorf("tmpfilesConf lacks %q", want)
2540	}
2541	}
2542	}
2543
2544	// The same-uid audit (design §9, codex R9-1, R9-2, R10-1): every tree a
2545	// carlos-uid unit writes, every carlos-owned cgroup, and every credential a
2546	// carlos-group process reads must be hidden from the tenant templates.
2547	func TestSameUIDAuditIsCoveredByTheTemplates(t *testing.T) {
2548	ns := readUnit(t, "carlos-instance@.service")
2549	hidden := func(path string) bool {
2550	for _, line := range strings.Split(ns, "\n") {
2551	for _, key := range []string{"TemporaryFileSystem=", "InaccessiblePaths="} {
2552	if strings.HasPrefix(line, key) {
2553	for _, p := range strings.Fields(strings.TrimPrefix(line, key)) {
2554	p = strings.TrimSuffix(p, ":ro")
2555	if path == p || strings.HasPrefix(path, p+"/") {
2556	return true
2557	}
2558	}
2559	}
2560	}
2561	}
2562	return false
2563	}
2564	names, err := unitNames()
2565	if err != nil {
2566	t.Fatal(err)
2567	}
2568	for _, name := range names {
2569	u := readUnit(t, name)
2570	if !strings.Contains(u, "User=carlos") {
2571	continue
2572	}
2573	for _, line := range strings.Split(u, "\n") {
2574	for _, key := range []string{"ReadWritePaths=", "CacheDirectory=", "StateDirectory=", "RuntimeDirectory="} {
2575	if !strings.HasPrefix(line, key) {
2576	continue
2577	}
2578	for _, p := range strings.Fields(strings.TrimPrefix(line, key)) {
2579	switch key {
2580	case "CacheDirectory=":
2581	p = "/var/cache/" + p
2582	case "StateDirectory=":
2583	p = "/var/lib/" + p
2584	case "RuntimeDirectory=":
2585	p = "/run/" + p
2586	}
2587	if !hidden(p) && !strings.HasPrefix(p, "/run/carlos/instances/") && !strings.HasPrefix(p, "/data/instances/") {
2588	t.Errorf("%s writes %s, which the tenant template does not hide", name, p)
2589	}
2590	}
2591	}
2592	}
2593	if strings.Contains(u, "Delegate=") && !strings.Contains(ns, "ProtectControlGroups=yes") {
2594	t.Errorf("%s delegates a cgroup subtree and the template leaves cgroupfs writable", name)
2595	}
2596	}
2597	// Credential inputs a carlos-group process reads (the second table).
2598	for _, cred := range []string{"/etc/carlos/relay-masters"} {
2599	if !hidden(cred) {
2600	t.Errorf("credential input %s is not hidden from tenants", cred)
2601	}
2602	}
2603	}
2604	```
2605
2606	- [ ] **Step 2: Run the tests to verify they fail**
2607
2608	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/host -run 'TestInstanceTemplates|TestControlUnits|TestSameUIDAudit' -count=1`
2609	Expected: FAIL (units missing).
2610
2611	- [ ] **Step 3: Write the six unit files**
2612
2613	`internal/host/units/carlos-instance@.service`:
2614
2615	```ini
2616	# One isolated exec-backed instance (isolated-exec design §4.1). The
2617	# activator owns it exactly as it owned the exec child this replaces: lease,
2618	# restore, seed, replication, hibernation, control socket, logs. What
2619	# changed is that the process is a unit, still uid carlos, inside a mount
2620	# namespace in which nothing but the instance's own two directories exists,
2621	# on mounts where no symlink is ever followed. The instance name is the
2622	# escaped bounded identifier (%i); paths use the unescaped one (%I).
2623	[Unit]
2624	Description=carlos exec-backed instance %I
2625	# Stop-only coupling: when the edge stops (including the stop half of a
2626	# restart) this unit stops, and nothing starts it again but the agent.
2627	# PartOf= would propagate the RESTART and race the agent's own startup
2628	# (codex R1-7).
2629	BindsTo=carlos-edge.service
2630	Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
2631	After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
2632
2633	[Service]
2634	Type=simple
2635	ExecStart=/opt/carlos/carlos instance-exec run %I
2636	# Runs inside the same namespace, as carlos, after this invocation's
2637	# processes have been signalled: leaves an ADVISORY exit record in the
2638	# instance's own run directory (§6.5). Not "+": the "+" prefix drops User=
2639	# and the Private*/Protect* settings but keeps the explicit
2640	# TemporaryFileSystem=/BindPaths= mounts (codex R3-2), so nothing run from
2641	# here can see the host's /run/carlos anyway.
2642	ExecStopPost=/opt/carlos/carlos instance-exit run %I
2643	User=carlos
2644	Group=carlos
2645	# One slice per host holds the instance, its seed and its sidecar, and
2646	# carries the tier (§4.3, §6.6). Slices referenced here are created on
2647	# demand; no per-host unit file exists.
2648	Slice=carlos-instances-%i.slice
2649	# The activator owns recycling: a crash stands the entry down and the next
2650	# request wakes it. systemd must not race that.
2651	Restart=no
2652	# SIGTERM to the app, SIGKILL to anything it left behind after
2653	# TimeoutStopSec; the activator's stop budget is 20s.
2654	KillMode=mixed
2655	TimeoutStopSec=20
2656	# The log stream: stdout and stderr connect to the agent's per-role
2657	# listener (§6.4). PID 1 opens this as root, outside the namespace, on the
2658	# nosymfollow logs mount, at a path no tenant mount contains.
2659	StandardOutput=file:/run/carlos/logs/%I.run.sock
2660	StandardError=inherit
2661
2662	# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
2663	ProtectSystem=strict
2664	TemporaryFileSystem=/data:ro
2665	TemporaryFileSystem=/run/carlos:ro
2666	# Every other tree a carlos-uid unit writes on the box is hidden the same
2667	# way (codex R9-2): the restore drills' cache under /var/cache, and
2668	# /var/lib as a class. An instance needs nothing from either.
2669	TemporaryFileSystem=/var/cache:ro
2670	TemporaryFileSystem=/var/lib:ro
2671	# Root-written credentials a carlos-GROUP process may read live here:
2672	# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
2673	# and an instance's Group=carlos would read it too (codex R10-1). An
2674	# instance gets its configuration through its launch file and needs
2675	# nothing from /etc/carlos.
2676	InaccessiblePaths=/etc/carlos
2677	BindPaths=/data/instances/%I /run/carlos/instances/%I
2678	ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
2679	ProtectHome=yes
2680	PrivateTmp=yes
2681	PrivateDevices=yes
2682	# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
2683	# and its cgroup.kill / cgroup.freeze would otherwise be one write away
2684	# for a same-uid tenant (codex R9-1).
2685	ProtectControlGroups=yes
2686	ProtectProc=invisible
2687	ProcSubset=pid
2688	PrivatePIDs=yes
2689	RestrictNamespaces=yes
2690	NoNewPrivileges=yes
2691	CapabilityBoundingSet=
2692	RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
2693	IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
2694	ProtectKernelTunables=yes
2695	ProtectKernelModules=yes
2696	ProtectKernelLogs=yes
2697	ProtectClock=yes
2698	ProtectHostname=yes
2699	RemoveIPC=yes
2700	LockPersonality=yes
2701	RestrictRealtime=yes
2702	RestrictSUIDSGID=yes
2703	MemoryDenyWriteExecute=yes
2704	SystemCallArchitectures=native
2705
2706	# --- per-service resources; the tier lives on the host slice (§4.3) ---
2707	TasksMax=256
2708	# The kernel picks a tenant before it picks anything else in carlos.slice.
2709	OOMScoreAdjust=500
2710	# One OOM kill takes every process of THIS service (systemd sets
2711	# memory.oom.group on the service cgroup for OOMPolicy=kill). The sidecar
2712	# is a sibling service with its own family rule (§4.3).
2713	OOMPolicy=kill
2714	```
2715
2716	`carlos-instance-seed@.service`: the same file with these differences, and the same two marker comment lines verbatim (`# --- the namespace ...` and `# --- per-service resources; ...`) so the test's block comparison holds:
2717	- `Description=carlos exec-backed instance %I (seed)`
2718	- `Type=oneshot`, followed by `RemainAfterExit=yes` and a comment: `# After ExecStart exits the unit stays active (exited) or failed, loaded, with its InvocationID, Result and ExecMainStatus retained until the control service stops it, so the seed's result can never be garbage-collected before it is read (codex R2-4).`, then `TimeoutStartSec=120` with the comment `# Today's seed budget; a oneshot has no start timeout by default (codex R1-5).`
2719	- `ExecStart=/opt/carlos/carlos instance-exec seed %I`, `ExecStopPost=/opt/carlos/carlos instance-exit seed %I`, `StandardOutput=file:/run/carlos/logs/%I.seed.sock`.
2720	- No `Restart=`, `KillMode=` and `TimeoutStopSec=20` kept.
2721
2722	`carlos-instance-sidecar@.service`: the run file with `Description=carlos exec-backed instance %I (sidecar)`, `ExecStart=/opt/carlos/carlos instance-exec sidecar %I`, `ExecStopPost=/opt/carlos/carlos instance-exit sidecar %I`, `StandardOutput=file:/run/carlos/logs/%I.sidecar.sock`; everything else identical.
2723
2724	`carlos-instance-probe.service`:
2725
2726	```ini
2727	# The readiness probe (isolated-exec design §8.5): a oneshot under the SAME
2728	# namespace text as the tenant templates (the test pins the block byte for
2729	# byte), run by carlos-instance-control at its start and every five
2730	# minutes. From inside it asserts what a tenant would see and prints a
2731	# verdict; the unit's StandardOutput=truncate: lands it where the control
2732	# service reads it. It exits 0 whatever the verdict: the verdict is the
2733	# output. Its instance directory is _probe, a name no host can have; the
2734	# tmpfiles lines create it.
2735	[Unit]
2736	Description=carlos instance isolation probe
2737	Requires=data-instances.mount run-carlos-instances.mount
2738	After=data-instances.mount run-carlos-instances.mount
2739
2740	[Service]
2741	Type=oneshot
2742	ExecStart=/opt/carlos/carlos instance-probe
2743	User=carlos
2744	Group=carlos
2745	Slice=carlos-instances.slice
2746	StandardOutput=truncate:/run/carlos-control/probe.out
2747	StandardError=journal
2748	TimeoutStartSec=30
2749
2750	# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
2751	ProtectSystem=strict
2752	TemporaryFileSystem=/data:ro
2753	TemporaryFileSystem=/run/carlos:ro
2754	# Every other tree a carlos-uid unit writes on the box is hidden the same
2755	# way (codex R9-2): the restore drills' cache under /var/cache, and
2756	# /var/lib as a class. An instance needs nothing from either.
2757	TemporaryFileSystem=/var/cache:ro
2758	TemporaryFileSystem=/var/lib:ro
2759	# Root-written credentials a carlos-GROUP process may read live here:
2760	# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
2761	# and an instance's Group=carlos would read it too (codex R10-1). An
2762	# instance gets its configuration through its launch file and needs
2763	# nothing from /etc/carlos.
2764	InaccessiblePaths=/etc/carlos
2765	BindPaths=/data/instances/_probe /run/carlos/instances/_probe
2766	ReadWritePaths=/data/instances/_probe /run/carlos/instances/_probe
2767	ProtectHome=yes
2768	PrivateTmp=yes
2769	PrivateDevices=yes
2770	# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
2771	# and its cgroup.kill / cgroup.freeze would otherwise be one write away
2772	# for a same-uid tenant (codex R9-1).
2773	ProtectControlGroups=yes
2774	ProtectProc=invisible
2775	ProcSubset=pid
2776	PrivatePIDs=yes
2777	RestrictNamespaces=yes
2778	NoNewPrivileges=yes
2779	CapabilityBoundingSet=
2780	RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
2781	IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
2782	ProtectKernelTunables=yes
2783	ProtectKernelModules=yes
2784	ProtectKernelLogs=yes
2785	ProtectClock=yes
2786	ProtectHostname=yes
2787	RemoveIPC=yes
2788	LockPersonality=yes
2789	RestrictRealtime=yes
2790	RestrictSUIDSGID=yes
2791	MemoryDenyWriteExecute=yes
2792	SystemCallArchitectures=native
2793
2794	# --- per-service resources; the tier lives on the host slice (§4.3) ---
2795	TasksMax=64
2796	MemoryMax=64M
2797	```
2798
2799	**The test's block comparison requires the namespace block to be byte-identical across the four files, and the probe's `BindPaths=`/`ReadWritePaths=` name `_probe` where the templates name `%I`.** Resolve this by moving the two `BindPaths=`/`ReadWritePaths=` lines OUT of the marked block in all four files: place them immediately after `StandardError=` (templates) / `TimeoutStartSec=30` (probe), before the `# --- the namespace` marker, with a one-line comment `# The instance's own two directories, the only writable data paths.` The marked block then starts at `ProtectSystem=strict` and is identical in all four files. Update the test's `wants` for `carlos-instance@.service` and the probe accordingly (they check `strings.Contains` on the whole file, so they still pass).
2800
2801	`carlos-instance-control.socket`:
2802
2803	```ini
2804	# The root instance-control channel's socket (isolated-exec design §5.2).
2805	# In /run/carlos-control, a root-owned directory beside /run/carlos, never
2806	# under it: an instance's namespace replaces /run/carlos with an empty
2807	# tmpfs, and nothing an instance can reach must resolve to this path. Owned
2808	# by carlos so the agent can connect; the service authenticates the peer
2809	# process itself (§5.4).
2810	[Unit]
2811	Description=carlos instance control socket (agent to root)
2812
2813	[Socket]
2814	ListenStream=/run/carlos-control/control.sock
2815	SocketUser=carlos
2816	SocketGroup=carlos
2817	SocketMode=0600
2818	Accept=no
2819
2820	[Install]
2821	WantedBy=sockets.target
2822	```
2823
2824	`carlos-instance-control.service`:
2825
2826	```ini
2827	# The root instance-control service (isolated-exec design §5). Socket-
2828	# activated, so it runs only while there is work. Its whole job is
2829	# systemctl start/stop/set-property on the three instance templates' units
2830	# and their host slices, on the agent's request, plus running the readiness
2831	# probe. Every unit name it passes to systemctl is built from a validated
2832	# host by the platform's own naming; nothing from a request reaches
2833	# systemctl as a string.
2834	#
2835	# Runs as ROOT deliberately, with no NoNewPrivileges= and no syscall
2836	# filter, for the systemctl child (the same rationale as
2837	# carlos-stage.service). It accepts a connection only from the edge unit's
2838	# main process (SO_PEERCRED uid carlos, pid == MainPID of
2839	# carlos-edge.service): a flat tenant is a child of that pid, never the pid
2840	# itself, and a nested tenant cannot reach the socket at all.
2841	[Unit]
2842	Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent)
2843	Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
2844	After=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
2845
2846	[Service]
2847	ExecStart=/opt/carlos/carlos instance-control
2848	ProtectSystem=strict
2849	ReadWritePaths=/run/carlos-control
2850	ProtectHome=yes
2851	PrivateTmp=yes
2852	PrivateDevices=yes
2853	ProtectKernelTunables=yes
2854	ProtectKernelModules=yes
2855	ProtectKernelLogs=yes
2856	RestrictAddressFamilies=AF_UNIX
2857	MemoryDenyWriteExecute=yes
2858	MemoryMax=64M
2859	```
2860
2861	- [ ] **Step 4: Bootstrap, systemunits, main.go, docs**
2862
2863	`internal/host/bootstrap.go`:
2864	- `enabledUnits` gains `"carlos-instance-control.socket"` after the three mounts, with the comment `// The root instance-control channel (design §5.2); socket-activated, so only the socket is enabled and started.`
2865	- `tmpfilesConf` gains, after the `/run/carlos/.instances` and `/run/carlos/instances` lines:
2866	```go
2867	// The readiness probe's own instance directory (design §8.5): a name
2868	// no host can have, created every boot so the probe unit's bind
2869	// sources exist before it starts.
2870	"d /data/.instances/_probe 0700 " + serviceUser + " " + serviceUser + " -\n" +
2871	"d /run/carlos/.instances/_probe 2750 " + serviceUser + " " + socketsGroup + " -\n" +
2872	```
2873	- `dirs` gains `{filepath.Join(dataDir, ".instances", "_probe"), 0o700}` and `{filepath.Join(runDir, ".instances", "_probe"), 0o750 | os.ModeSetgid}` after the `.instances` entries.
2874
2875	`internal/host/systemunits.go`: add `const instanceControlSocket = "carlos-instance-control.socket"`; extend `systemUnitFiles` with `instanceControlSocket` and `"carlos-instance-control.service"`; in `ensureSystemUnits`, probe `is-active` on `instanceControlSocket` as well (`ctlActive := parseIsActive(...)`), treat a nil like the existing nil case, and include `!*ctlActive` in the "needs arming" condition; pass `instanceControlSocket` in the `systemctlPlan([...])` list so the belt enables and starts it. Update `TestSystemUnitsAreShapedLikeTheStageRelay` only if it enumerates `systemUnitFiles` (read it first).
2876
2877	`main.go`: add after `case "adopt":` arm:
2878
2879	```go
2880	case "instance-control":
2881	instancectl.Run(args)
2882	case "instance-exec":
2883	instanceexec.RunExec(args)
2884	case "instance-exit":
2885	instanceexec.RunExit(args)
2886	case "instance-probe":
2887	instanceexec.RunProbe(args)
2888	```
2889
2890	with the two imports `"github.com/carlosframework/platform/internal/instancectl"` and `"github.com/carlosframework/platform/internal/instanceexec"`.
2891
2892	`docs/site/cli.md`, under `## On the host` after the `### carlos adopt` section, add four sections (plain prose, no em dashes):
2893
2894	```markdown
2895	### carlos instance-control
2896
2897	The root half of isolated exec. `carlos-instance-control.socket` activates
2898	it when the agent connects; it starts, stops and sweeps the
2899	`carlos-instance*@` units for the agent, sets each host's memory tier on
2900	its slice before a start, and runs the isolation probe at start and every
2901	five minutes. It accepts a connection only from the edge unit's own main
2902	process. Not for operators: there is nothing to type.
2903
2904	### carlos instance-exec
2905
2906	The first thing an isolated instance unit runs. It reads the launch file
2907	the agent wrote into the instance's directory and execs the host's live
2908	link with exactly the arguments and environment in it, refusing anything
2909	that is not the platform's own live link. `ExecStart=` of the three
2910	instance templates.
2911
2912	### carlos instance-exit
2913
2914	`ExecStopPost=` of the three instance templates. It leaves an advisory
2915	exit record in the instance's run directory from the four variables
2916	systemd provides; the agent proves termination from the unit's state and
2917	its cgroup, not from this file.
2918
2919	### carlos instance-probe
2920
2921	The isolation probe, run under the same namespace text as the instance
2922	templates. From inside it checks what a tenant would see: only its own
2923	directories, only its own pid, a read-only cgroupfs, no `/etc/carlos`, a
2924	symlink that is not followed, and a metadata service that a packet filter
2925	refuses. It prints a verdict; `carlos-instance-control` reads it and
2926	refuses to start isolated instances until every line passes.
2927	```
2928
2929	- [ ] **Step 5: Run the tests, the docsite gate, and the full gate**
2930
2931	Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/host ./internal/docsite -count=1` (with `TMPDIR=/tmp/g` outside the sandbox for `internal/host`)
2932	Expected: PASS, including `TestCLICommandsAreDocumented`.
2933
2934	Then the full gate:
2935
2936	```bash
2937	export GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp/g
2938	go build ./... && go vet ./... && gofmt -l . && go test ./... -count=1
2939	GOOS=darwin go build ./... && GOOS=windows go build ./... && GOARCH=arm64 go build ./...
2940	```
2941
2942	Expected: green.
2943
2944	- [ ] **Step 6: Commit**
2945
2946	```bash
2947	git add internal/host/units internal/host/bootstrap.go internal/host/systemunits.go internal/host/units_isolation_test.go main.go docs/site/cli.md
2948	git commit -m "units and verbs: the instance templates, the probe, the control socket and service, and their docs (plan 2, task 6)"
2949	```
2950
2951	---
2952
2953	## Self-review
2954
2955	**Spec coverage for this plan's scope.** §4.1, §4.2 templates: Task 6. §4 escaping and `%I`/`%i`: Tasks 1 and 6. §5.2 units: Task 6. §5.3 verbs and sequences, no `reset-failed` in `/start`, `/seed` with `RemainAfterExit`, `/sweep` by reported names, `/ready`: Task 5. §5.4 peer check by `SO_PEERCRED` uid and `MainPID`, no token: Task 5. §6.2 trampoline and its checks: Task 2. §6.5 exit record, `O_EXCL`, advisory: Task 3. §6.5 cgroup emptiness as the termination rule: Task 5's `WaitCgroupEmpty` (the agent-side observer is plan 3). §8.5 probe lines and rules, fixture verification, re-run every five minutes: Tasks 4 and 5. §9 same-uid audit test, `enabledUnits`, `ensureSystemUnits` active check, `docs/site/cli.md`: Task 6. Deferred by design to plan 3: everything in the agent (prepare, launch-file writing, log listeners, `unitProc`, the exit observer, the client for this channel, `/sweep` at agent start). Deferred to plan 4: the layout verb, status field, downgrade guards. `update-platform-binary.sh` (pinfra) must enable `carlos-instance-control.socket` by name; noted for the roll.
2956
2957	**Placeholder scan.** Task 6's seed and sidecar files are described as deltas from the run file with every differing line spelled out; the block-move instruction is explicit. No "TBD".
2958
2959	**Type consistency.** `instancectl.Role`, `RoleRun/Seed/Sidecar`, `ParseRole`, `Escape`, `UnitName`, `SliceName`, `ParseUnitName`, `LaunchFileName`, `ExitRecordName`, `Launch`, `ExitRecord`, `ProbeDirName`, `VerdictLine`, `ProbeReport`, `StartResponse`, `SeedResponse`, `SweepResponse` are used with the same spelling in Tasks 2 to 6. `probeEnv` fields are used identically in Task 4's test and code. `Server` fields match between Task 5's tests and code (`cgroupWaitBudget` is unexported and set directly by the tests in the same package).
2960