carlos / platform Public

Download this file

1	# `carlos run`, Plan 1: Box Foundation Implementation Plan
2
3	> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5	**Goal:** Give the box everything one `carlos run` command needs to run as its instance: an approved copy deck, a fourth instance role and unit, a control-service door that starts it and a long-poll that follows it, a launch file with the run's environment, two output streams, and a place on the activator entry that every stop path settles before replication closes.
6
7	**Architecture:** `instancectl` gains `RoleCmd`, the `carlos-instance-cmd@.service` template (the seed template with six lines changed), `POST /cmd` (start, answer at once) and `GET /cmd/{host}` (a 30-second long-poll). `activator.ExecBackend.StartCmd` prepares the instance's directories and env, binds two raw stream listeners, writes `launch.cmd.json` with `cmdVars`, and starts the unit. `activator.Service.RunCmd` wakes the instance (up to 60 s), attaches a `cmdHold` to the entry, and starts the unit only while the entry is awake. `finishStop`, the one place every stop path ends, stops and settles the hold's unit before `rep.Close`; the idle sweep skips an entry with a hold. Nothing in this plan picks a run up: Plan 3 calls `RunCmd`.
8
9	**Tech Stack:** Go 1.25, systemd via `systemctl` (`instancectl.Server.Systemctl`), `golang.org/x/sys/unix` (openat2, unlinkat), the repo's `instancectl`, `instancefs`, `activator`, `provision` packages.
10
11	**Spec:** `docs/superpowers/specs/2026-10-08-carlos-run-design.md` rev 4: §3.1, §4, §5.3, §5.4, §5.2 step 3 (the cleanup primitive), §9 "Writer lifecycle" and "Unit and control"; dispositions codex 1, 2, 4, 6, 7. Index: `2026-10-08-carlos-run-0-index.md`. Copy: `2026-10-08-carlos-run-copy.md`.
12
13	**Build target:** `main` at 4a51a6da or later. Every name below was read there.
14
15	## Global Constraints
16
17	- Every user-facing string (CLI output, errors, console messages, docs prose) is copy-reviewed by Paul before it is written into code. The deck is `docs/superpowers/plans/2026-10-08-carlos-run-copy.md` and its gate is `approved_run_copy_test.go` (Task 1). Implementation tasks use only strings from the deck; a task that writes one adds its id to `runCopyBuilt` in the same commit. Copy voice: no em dashes; full stops and new sentences instead.
18	- A deck string appears in Go as one string literal, or one `fmt.Sprintf`/`fmt.Errorf`/`errors.New` format, equal to the row with `%s`/`%d`/`%q` where the row has a `<placeholder>`, and with no trailing newline inside the literal.
19	- CI cross-compiles darwin and windows: linux-only syscalls/units code needs `//go:build linux` plus a `!linux` stub.
20	- Local test commands: run targeted tests (`go test ./internal/<pkg>/ -run TestX`), not the full suite in a loop. Inside the sandbox the default Go build cache may be read-only; if so use `GOCACHE=$TMPDIR/gocache` re-exported per command. Socket tests need a short TMPDIR (`TMPDIR=/tmp/g`). The local gate may need `GOFLAGS=-buildvcs=false`.
21	- Heavy loops run under `nice`; no -race full suite until the end of a plan.
22	- Commits end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
23	- The cmd template is the seed template outside exactly these lines (spec §5.4): `Description=carlos exec-backed instance %I (cmd)`, `ExecStart=/opt/carlos/carlos instance-exec cmd %I`, `ExecStopPost=/opt/carlos/carlos instance-exit cmd %I`, `TimeoutStartSec=6h5m`, `StandardOutput=file:/run/carlos/logs/%I.cmd.sock`, `StandardError=file:/run/carlos/logs/%I.cmd-err.sock`. `Type=oneshot` and `RemainAfterExit=yes` stay.
24	- `GET /cmd/{host}` waits at most 30 seconds. `POST /cmd` never waits for the command.
25	- The box waits up to 60 seconds for a wake before a run (spec §5.3). Signals: SIGTERM, then SIGKILL 20 seconds later (`KillMode=mixed`, `TimeoutStopSec=20`, already in the seed template).
26	- `cmdVars` (spec §4): the instance's file environment; `CARLOS_CONTROL_SOCKET` and `CARLOS_PURGE` removed; `CARLOS_REGION` set from `ExecBackend.Region`, or removed when it is empty; `CARLOS_HOST`; `CARLOS_ADMIN_TOKEN` kept; `CARLOS_DB`, `CARLOS_RUN_ID`, `CARLOS_RUN_DIR` added. The environment is never logged.
27	- Gate before the plan's last commit: `go build ./...`, `go vet ./...`, `gofmt -l .` clean, `make test`, and `GOOS=darwin go build ./...`, `GOOS=windows go build ./...`, `GOOS=linux GOARCH=arm64 go build ./...`.
28
29	## Review Focus
30
31	1. A stop that begins while `RunCmd` is between its attach and `StartCmd` (the files are still downloading) must never leave a cmd unit started after `rep.Close`: either the run is refused with the stop's reason before the unit starts, or `finishStop` waits for the start to return and stops the unit before it closes replication. Test in Task 8 (`TestRunCmdNeverStartsUnderAStoppingEntry` and `TestAStopDuringTheStartWaitsForIt`).
32	2. A cmd unit still loaded from an earlier run (`active (exited)` because the agent died before stopping it) must make `POST /cmd` refuse with 409, not silently no-op the start and hand back the old invocation. Test in Task 4 (`TestCmdRefusesAUnitStillLoaded`).
33	3. A cmd unit whose `StopAndSettle` cannot prove its cgroup empty must leave the stop uncertain (`stopObservation.uncertain`), so the maintenance fence never vouches for a stop with a writer possibly alive. Test in Task 8 (`TestAnUnsettledRunMakesTheStopUncertain`).
34	4. A tenant that plants a symlink at `.carlos-run` or inside a run directory must not make the cleanup delete anything outside the instance's tree. Test in Task 5 (`TestRemoveAllBeneathNeverFollowsASymlink`).
35	5. Output written by the command in its last moment (bytes still in the socket buffer when the main process exits) must reach the run's writer before `StopAndSettle` returns. Test in Task 6 (`TestStopAndSettleDrainsBothStreams`).
36
37	---
38
39	## File structure
40
41	| path | responsibility |
42	|---|---|
43	| `approved_run_copy_test.go` | the run deck's gate: inventory, approval before use, applied text, no em dash |
44	| `approved_moves_copy_test.go` | `movesCopyIDPattern` accepts `RUN-C` ids (the shared parser) |
45	| `testdata/run-copy/record.md`, `testdata/run-copy/applied.go` | the gate's negative fixtures |
46	| `internal/instancectl/names.go` | `RoleCmd`, `UnitPrefixCmd`, `CmdStatus` |
47	| `internal/instancectl/server.go` | the two routes, the roles `/stop` and `/reset` accept, the sweep pattern, the cmd timing fields |
48	| `internal/instancectl/cmd.go` | `handleCmd`, `handleCmdStatus`, `cmdStatusOf` |
49	| `internal/instancectl/client.go` | `Client.Cmd`, `Client.CmdStatus`, `DeadlineCmdPoll` |
50	| `internal/instanceexec/exec.go`, `exit.go` | usage and role lines name cmd |
51	| `internal/host/units/carlos-instance-cmd@.service` | the fourth template |
52	| `internal/host/units_isolation_test.go`, `units_hardening_test.go`, `host_test.go` | the template's tests and lists |
53	| `internal/instancefs/dirs_linux.go`, `instancefs_other.go` | `RemoveAllBeneath` |
54	| `internal/activator/control.go` | `Control.Cmd`, `Control.CmdStatus` |
55	| `internal/activator/prepare.go` | env for the cmd role, no app-log listener for it, `logSocketNamed` |
56	| `internal/activator/cmdunit.go` | `CmdSpec`, `CmdUnit`, `CmdBackend`, `cmdVars`, `cmdStream`, `ExecBackend.StartCmd`, `cmdUnit` |
57	| `internal/activator/backend_exec.go` | `waitRoleCgroupEmpty` |
58	| `internal/activator/backend_unit.go` | `SplitBackend.StartCmd` |
59	| `internal/activator/cmd.go` | stop reasons, `CmdHandle`, `cmdHold`, `Service.RunCmd`, errors |
60	| `internal/activator/activator.go`, `maintenance.go` | `beginStopLocked(e, why)`, `actEntry.cmd`/`stopWhy`, `finishStop` ordering, the idle hold |
61	| `docs/site/cli.md` | "four instance templates" (RUN-C11, RUN-C12) |
62
63	---
64
65	### Task 1: The copy gate and Paul's review
66
67	This is the series' first task. Nothing else in the series writes a user-facing string until it is merged with every row approved.
68
69	**Files:**
70	- Create: `approved_run_copy_test.go`, `testdata/run-copy/record.md`, `testdata/run-copy/applied.go`
71	- Modify: `approved_moves_copy_test.go:91` (`movesCopyIDPattern`) and the error text at `:150`
72	- Modify: `docs/superpowers/plans/2026-10-08-carlos-run-copy.md` (statuses and Paul's edits)
73
74	**Interfaces:**
75	- Consumes: `parseMovesCopyRecord(path string) ([]copyRow, error)`, `checkMovesInventory(rows []copyRow, want []string) []string`, `checkMovesApplied(rows []copyRow, root string) []string`, `checkMovesNoEmDash(rows []copyRow) []string`, `reportAll(t *testing.T, problems []string)`, `copyRow{ID, File, Text, Status string}` (all in `approved_moves_copy_test.go`, package `main`).
76	- Produces: `runCopyIDs []string` (RUN-C1 to RUN-C83), `runCopyBuilt []string` (empty here; every later task appends), `checkRunBuilt(rows []copyRow, built []string) []string`, `builtRows(rows []copyRow, built []string) []copyRow`.
77
78	- [ ] **Step 1: Write the failing test**
79
80	Create `approved_run_copy_test.go`:
81
82	```go
83	// approved_run_copy_test.go
84	package main
85
86	import (
87	"fmt"
88	"os"
89	"path/filepath"
90	"strconv"
91	"strings"
92	"testing"
93	)
94
95	// The copy gate of the carlos run series (run Plan 1 Task 1). The deck is a
96	// markdown table in the moves record's format, read with the moves gate's
97	// parser and checked with its helpers, so "applied" means the same thing in
98	// both gates:
99	//
100	// - TestRunCopyRecordIsComplete: the deck names exactly RUN-C1 to RUN-C83;
101	// - TestRunBuiltCopyIsApproved: an id in runCopyBuilt is an approved row
102	// (a string reaches code only after Paul has approved it);
103	// - TestRunBuiltCopyIsApplied: a built row's text is in the file it names;
104	// - TestRunCopyHasNoEmDash.
105	const runCopyRecordPath = "docs/superpowers/plans/2026-10-08-carlos-run-copy.md"
106
107	// runCopyIDs is the deck's inventory: RUN-C1 to RUN-C83, pinned.
108	var runCopyIDs = func() []string {
109	ids := make([]string, 0, 83)
110	for i := 1; i <= 83; i++ {
111	ids = append(ids, "RUN-C"+strconv.Itoa(i))
112	}
113	return ids
114	}()
115
116	// runCopyBuilt is every deck id whose text the code or docs now carry. The
117	// task that writes a row's text adds its id here in the same commit.
118	var runCopyBuilt = []string{}
119
120	func mustParseRunRecord(t *testing.T) []copyRow {
121	t.Helper()
122	rows, err := parseMovesCopyRecord(runCopyRecordPath)
123	if err != nil {
124	t.Fatal(err)
125	}
126	if len(rows) == 0 {
127	t.Fatalf("%s has no rows", runCopyRecordPath)
128	}
129	for _, r := range rows {
130	if !strings.HasPrefix(r.ID, "RUN-C") {
131	t.Fatalf("%s: %s is not a RUN-C id", runCopyRecordPath, r.ID)
132	}
133	}
134	return rows
135	}
136
137	// checkRunBuilt: every built id is named once, names a row, and that row is
138	// approved.
139	func checkRunBuilt(rows []copyRow, built []string) []string {
140	byID := map[string]copyRow{}
141	for _, r := range rows {
142	byID[r.ID] = r
143	}
144	var out []string
145	seen := map[string]bool{}
146	for _, id := range built {
147	if seen[id] {
148	out = append(out, fmt.Sprintf("runCopyBuilt names %s twice", id))
149	}
150	seen[id] = true
151	r, ok := byID[id]
152	switch {
153	case !ok:
154	out = append(out, fmt.Sprintf("runCopyBuilt names %s, which the deck has no row for", id))
155	case r.Status != "approved":
156	out = append(out, fmt.Sprintf("%s is built but %s: a string reaches code only after Paul approves it", id, r.Status))
157	}
158	}
159	return out
160	}
161
162	// builtRows is the rows runCopyBuilt names, in deck order.
163	func builtRows(rows []copyRow, built []string) []copyRow {
164	want := map[string]bool{}
165	for _, id := range built {
166	want[id] = true
167	}
168	var out []copyRow
169	for _, r := range rows {
170	if want[r.ID] {
171	out = append(out, r)
172	}
173	}
174	return out
175	}
176
177	func TestRunCopyRecordIsComplete(t *testing.T) {
178	reportAll(t, checkMovesInventory(mustParseRunRecord(t), runCopyIDs))
179	}
180
181	func TestRunCopyHasNoEmDash(t *testing.T) {
182	reportAll(t, checkMovesNoEmDash(mustParseRunRecord(t)))
183	}
184
185	func TestRunBuiltCopyIsApproved(t *testing.T) {
186	reportAll(t, checkRunBuilt(mustParseRunRecord(t), runCopyBuilt))
187	}
188
189	// runFragment reports whether a row's text is a piece of a larger literal
190	// (its file cell says "a fragment"): the top-level usage text, which no
191	// whole-expression match can find.
192	func runFragment(r copyRow) bool { return strings.Contains(r.File, "(a fragment") }
193
194	// checkRunApplied is checkMovesApplied, plus the fragment rows: those must
195	// sit inside some string literal of their file (goLiteralMatches,
196	// approved_copy_test.go), never only in a comment.
197	func checkRunApplied(rows []copyRow, root string) []string {
198	var whole []copyRow
199	var out []string
200	for _, r := range rows {
201	if !runFragment(r) {
202	whole = append(whole, r)
203	continue
204	}
205	if r.Status != "approved" {
206	continue
207	}
208	body, err := os.ReadFile(filepath.Join(root, r.Path()))
209	if err != nil {
210	out = append(out, fmt.Sprintf("%s: %v", r.ID, err))
211	continue
212	}
213	if !goLiteralMatches(body, func(v string) bool { return strings.Contains(v, r.Text) }) {
214	out = append(out, fmt.Sprintf("%s: %s has no string literal holding the approved text %q", r.ID, r.Path(), r.Text))
215	}
216	}
217	return append(checkMovesApplied(whole, root), out...)
218	}
219
220	func TestRunBuiltCopyIsApplied(t *testing.T) {
221	reportAll(t, checkRunApplied(builtRows(mustParseRunRecord(t), runCopyBuilt), "."))
222	}
223
224	// Each fixture case is a state the gate must refuse, for the right reason.
225	func TestRunCopyGateRefusesTheNegativeFixtures(t *testing.T) {
226	dir := filepath.Join("testdata", "run-copy")
227	rows, err := parseMovesCopyRecord(filepath.Join(dir, "record.md"))
228	if err != nil {
229	t.Fatal(err)
230	}
231	t.Run("a built row still pending", func(t *testing.T) {
232	got := checkRunBuilt(rows, []string{"RUN-C2"})
233	if len(got) != 1 || !strings.Contains(got[0], "RUN-C2 is built but pending") {
234	t.Errorf("problems = %q", got)
235	}
236	})
237	t.Run("a built id with no row", func(t *testing.T) {
238	got := checkRunBuilt(rows, []string{"RUN-C9"})
239	if len(got) != 1 || !strings.Contains(got[0], "no row for") {
240	t.Errorf("problems = %q", got)
241	}
242	})
243	t.Run("a built id named twice", func(t *testing.T) {
244	got := checkRunBuilt(rows, []string{"RUN-C1", "RUN-C1"})
245	if len(got) != 1 || !strings.Contains(got[0], "twice") {
246	t.Errorf("problems = %q", got)
247	}
248	})
249	t.Run("an approved row whose text is only in a comment", func(t *testing.T) {
250	got := checkMovesApplied(builtRows(rows, []string{"RUN-C3"}), dir)
251	if len(got) != 1 || !strings.Contains(got[0], "RUN-C3") {
252	t.Errorf("problems = %q", got)
253	}
254	})
255	t.Run("an approved row that is applied", func(t *testing.T) {
256	if got := checkRunApplied(builtRows(rows, []string{"RUN-C1"}), dir); len(got) != 0 {
257	t.Errorf("problems = %q", got)
258	}
259	})
260	t.Run("a fragment inside a larger literal", func(t *testing.T) {
261	if got := checkRunApplied(builtRows(rows, []string{"RUN-C4"}), dir); len(got) != 0 {
262	t.Errorf("problems = %q", got)
263	}
264	})
265	t.Run("a fragment only in a comment", func(t *testing.T) {
266	got := checkRunApplied(builtRows(rows, []string{"RUN-C5"}), dir)
267	if len(got) != 1 || !strings.Contains(got[0], "RUN-C5") {
268	t.Errorf("problems = %q", got)
269	}
270	})
271	}
272	```
273
274	Create `testdata/run-copy/record.md`:
275
276	```markdown
277	| id | file | text | status |
278	|---|---|---|---|
279	| RUN-C1 | applied.go | <host> has no run <id> | approved |
280	| RUN-C2 | applied.go | the instance hibernated | pending |
281	| RUN-C3 | applied.go | the box's agent stopped | approved |
282	| RUN-C4 | applied.go (a fragment of the usage text) | run one command | approved |
283	| RUN-C5 | applied.go (a fragment of the usage text) | only in a comment | approved |
284	```
285
286	Create `testdata/run-copy/applied.go` (never compiled: `testdata` is outside the build):
287
288	```go
289	package fixture
290
291	import "fmt"
292
293	func say(host, id string) string { return fmt.Sprintf("%s has no run %s", host, id) }
294
295	const usage = `usage: x
296	run run one command, somewhere`
297
298	// the box's agent stopped: in this comment only, which never counts.
299	// only in a comment
300	```
301
302	- [ ] **Step 2: Run to verify it fails**
303
304	Run: `nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt' -count=1`
305	Expected: FAIL. `parseMovesCopyTable` refuses every row with `"RUN-C1" is not an M-C<n>, MV-N<n> or HC5-N<n> id`.
306
307	- [ ] **Step 3: Accept RUN-C ids in the shared parser**
308
309	In `approved_moves_copy_test.go:91`:
310
311	```go
312	var movesCopyIDPattern = regexp.MustCompile(`^(M-C|MV-N|HC5-N|RUN-C)[0-9]+$`)
313	```
314
315	and at `:150` the refusal names the new id family (the `"bad id"` case of `TestParseMovesCopyTable` still matches `"is not an M-C"`):
316
317	```go
318	return nil, fmt.Errorf("%s: %q is not an M-C<n>, MV-N<n>, HC5-N<n> or RUN-C<n> id", at, r.ID)
319	```
320
321	A `RUN-C` row in the moves record is still caught: `checkMovesInventory` reports any id `movesCopyIDs` does not pin.
322
323	- [ ] **Step 4: Run to verify it passes**
324
325	Run: `nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt|TestParseMovesCopyTable|TestMovesCopy' -count=1`
326	Expected: PASS (every deck row is pending and `runCopyBuilt` is empty).
327
328	- [ ] **Step 5: Commit the gate**
329
330	```bash
331	git add approved_run_copy_test.go approved_moves_copy_test.go testdata/run-copy/record.md testdata/run-copy/applied.go
332	git commit -m "run: the carlos run copy gate (Plan 1 Task 1)
333
334	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
335	```
336
337	- [ ] **Step 6: Paul's copy review**
338
339	Invoke the `copy-review` skill with every row of `docs/superpowers/plans/2026-10-08-carlos-run-copy.md` (83 strings, grouped by the deck's three headings, with the deck's placeholder key as context). **Stop here until Paul has submitted the review.** Read the review's `result.json` before restarting the review server for any reason: a relaunch deletes it.
340
341	- [ ] **Step 7: Apply the review verbatim**
342
343	For every row: replace the text cell with Paul's text exactly, and set the status to `approved`. A row Paul strikes is deleted from the deck and from `runCopyIDs` (replace the generator with a literal list naming the ids that remain), and the plan task that names it drops the string. A row Paul adds gets the next free id and is added to `runCopyIDs`. Keep no em dash.
344
345	Run: `nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt' -count=1`
346	Expected: PASS.
347
348	- [ ] **Step 8: Commit the approved deck**
349
350	```bash
351	git add docs/superpowers/plans/2026-10-08-carlos-run-copy.md approved_run_copy_test.go
352	git commit -m "run: the copy deck, approved by Paul
353
354	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
355	```
356
357	---
358
359	### Task 2: `RoleCmd` in `instancectl`
360
361	**Files:**
362	- Modify: `internal/instancectl/names.go:23-39` (roles, `ParseRole`), `:66-82` (prefixes, `prefixFor`), `:104-107` (`ParseUnitName`'s table)
363	- Modify: `internal/instancectl/server.go:283` (the role error), `:474` (`handleStop` roles), `:487` (`handleReset` roles), `:506-507` (the sweep's patterns)
364	- Modify: `internal/instanceexec/exec.go:44,52`, `internal/instanceexec/exit.go:35,43`
365	- Test: `internal/instancectl/names_test.go:56-82`, `internal/instancectl/server_test.go` (a sweep case)
366
367	**Interfaces:**
368	- Produces: `instancectl.RoleCmd Role = "cmd"`, `instancectl.UnitPrefixCmd = "carlos-instance-cmd@"`; `UnitName(RoleCmd, host)` is `carlos-instance-cmd@<escaped id>.service`; `/stop` and `/reset` accept `role=cmd`; `/sweep` lists and stops cmd units.
369
370	- [ ] **Step 1: Write the failing tests**
371
372	In `names_test.go`, `TestParseUnitNameRoundTrips` iterates the four roles and `TestParseRole` accepts `cmd`:
373
374	```go
375	func TestParseUnitNameRoundTrips(t *testing.T) {
376	for _, role := range []Role{RoleRun, RoleSeed, RoleSidecar, RoleCmd} {
377	u := UnitName(role, "bill-2.example.com")
378	gotRole, esc, ok := ParseUnitName(u)
379	if !ok || gotRole != role || esc != `bill\x2d2.example.com` {
380	t.Errorf("ParseUnitName(%q) = %q %q %v", u, gotRole, esc, ok)
381	}
382	}
383	if got := UnitName(RoleCmd, "bill-2.example.com"); got != `carlos-instance-cmd@bill\x2d2.example.com.service` {
384	t.Errorf("cmd unit = %q", got)
385	}
386	for _, bad := range []string{"carlos-edge.service", "carlos-app@x.service", "carlos-instance@x.socket", "carlos-instance@.service", "carlos-instance-cmd@.service", "carlos-instances-x.slice"} {
387	if _, _, ok := ParseUnitName(bad); ok {
388	t.Errorf("ParseUnitName(%q) accepted", bad)
389	}
390	}
391	}
392
393	func TestParseRole(t *testing.T) {
394	for _, s := range []string{"run", "seed", "sidecar", "cmd"} {
395	if r, ok := ParseRole(s); !ok || string(r) != s {
396	t.Errorf("ParseRole(%q) = %q %v", s, r, ok)
397	}
398	}
399	for _, s := range []string{"", "Run", "exec", "run ", "Cmd", "cmd-err"} {
400	if _, ok := ParseRole(s); ok {
401	t.Errorf("ParseRole(%q) accepted", s)
402	}
403	}
404	}
405	```
406
407	Append to `server_test.go`:
408
409	```go
410	func TestStopResetAndSweepCoverTheCmdRole(t *testing.T) {
411	ctl := &fakeCtl{answers: map[string]string{
412	"list-units": "carlos-instance-cmd@a.example.com.service loaded active exited carlos exec-backed instance a.example.com (cmd)\n",
413	"show -p ActiveState --value": "inactive\n",
414	}}
415	s, c := newServer(t, ctl)
416	s.cgroupWaitBudget = time.Second
417	if resp := post(t, c, "/stop?host=a.example.com&role=cmd"); resp.StatusCode != http.StatusNoContent {
418	t.Fatalf("stop status %d", resp.StatusCode)
419	}
420	if resp := post(t, c, "/reset?host=a.example.com&role=cmd"); resp.StatusCode/100 != 2 {
421	t.Fatalf("reset status %d", resp.StatusCode)
422	}
423	if resp := post(t, c, "/sweep"); resp.StatusCode != http.StatusOK {
424	t.Fatalf("sweep status %d", resp.StatusCode)
425	}
426	calls := ctl.joined()
427	for _, want := range []string{
428	"stop carlos-instance-cmd@a.example.com.service",
429	"reset-failed carlos-instance-cmd@a.example.com.service",
430	"carlos-instance-cmd@*.service",
431	} {
432	if !strings.Contains(calls, want) {
433	t.Errorf("missing %q in\n%s", want, calls)
434	}
435	}
436	}
437	```
438
439	- [ ] **Step 2: Run to verify they fail**
440
441	Run: `nice -n 10 go test ./internal/instancectl/ -run 'TestParseUnitNameRoundTrips|TestParseRole|TestStopResetAndSweepCoverTheCmdRole' -count=1`
442	Expected: FAIL, `undefined: RoleCmd`.
443
444	- [ ] **Step 3: Implement**
445
446	`names.go`:
447
448	```go
449	type Role string
450
451	const (
452	RoleRun Role = "run"
453	RoleSeed Role = "seed"
454	RoleSidecar Role = "sidecar"
455	// RoleCmd is one `carlos run` command (run spec §5.4). "run" is the
456	// serving role already, so the command's role is cmd.
457	RoleCmd Role = "cmd"
458	)
459
460	// ParseRole accepts exactly the four literals.
461	func ParseRole(s string) (Role, bool) {
462	switch Role(s) {
463	case RoleRun, RoleSeed, RoleSidecar, RoleCmd:
464	return Role(s), true
465	}
466	return "", false
467	}
468	```
469
470	```go
471	// The four template prefixes. A unit under any of them is the platform's
472	// to start, stop and sweep; nothing else is (design §5.3).
473	const (
474	UnitPrefixRun = "carlos-instance@"
475	UnitPrefixSeed = "carlos-instance-seed@"
476	UnitPrefixSidecar = "carlos-instance-sidecar@"
477	UnitPrefixCmd = "carlos-instance-cmd@"
478	)
479
480	func prefixFor(role Role) string {
481	switch role {
482	case RoleSeed:
483	return UnitPrefixSeed
484	case RoleSidecar:
485	return UnitPrefixSidecar
486	case RoleCmd:
487	return UnitPrefixCmd
488	}
489	return UnitPrefixRun
490	}
491	```
492
493	`ParseUnitName`'s table, the run prefix still last:
494
495	```go
496	}{{UnitPrefixSeed, RoleSeed}, {UnitPrefixSidecar, RoleSidecar}, {UnitPrefixCmd, RoleCmd}, {UnitPrefixRun, RoleRun}} {
497	```
498
499	`server.go:283`:
500
501	```go
502	return request{}, fmt.Errorf("role %q is not run, seed, sidecar or cmd", q.Get("role"))
503	```
504
505	`server.go:474` and `:487`: the role lists become `[]Role{RoleRun, RoleSeed, RoleSidecar, RoleCmd}`. `server.go:506-507`:
506
507	```go
508	out, err := s.Systemctl(ctx, "list-units", "--all", "--plain", "--no-legend", "--no-pager",
509	UnitPrefixRun+"*.service", UnitPrefixSeed+"*.service", UnitPrefixSidecar+"*.service", UnitPrefixCmd+"*.service")
510	```
511
512	`internal/instanceexec/exec.go:44` and `:52` (RUN-C9, RUN-C13):
513
514	```go
515	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exec <run|seed|sidecar|cmd> <instance-id>")
516	```
517
518	```go
519	log.Fatalf("instance-exec: role %q is not run, seed, sidecar or cmd", fs.Arg(0))
520	```
521
522	`internal/instanceexec/exit.go:35` and `:43` (RUN-C10, RUN-C14):
523
524	```go
525	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exit <run|seed|sidecar|cmd> <instance-id>")
526	```
527
528	```go
529	log.Printf("instance-exit: role %q is not run, seed, sidecar or cmd", fs.Arg(0))
530	```
531
532	The trampoline needs nothing else: `execLaunch` (`exec.go:72`) reads `launch.<role>.json` for whatever role parsed, and validates argv0 under `/opt/carlos/live`.
533
534	In `approved_run_copy_test.go`: `var runCopyBuilt = []string{"RUN-C9", "RUN-C10", "RUN-C13", "RUN-C14"}`.
535
536	- [ ] **Step 4: Run to verify they pass**
537
538	Run: `nice -n 10 go test ./internal/instancectl/ ./internal/instanceexec/ -count=1 && nice -n 10 go test . -run 'TestRunBuilt' -count=1`
539	Expected: PASS. (`internal/instanceexec`'s own tests are linux-tagged; they run on this box.)
540
541	- [ ] **Step 5: Commit**
542
543	```bash
544	git add internal/instancectl/names.go internal/instancectl/names_test.go internal/instancectl/server.go internal/instancectl/server_test.go internal/instanceexec/exec.go internal/instanceexec/exit.go approved_run_copy_test.go
545	git commit -m "instancectl: RoleCmd and the carlos-instance-cmd@ prefix (run spec §5.4)
546
547	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
548	```
549
550	---
551
552	### Task 3: The `carlos-instance-cmd@.service` template
553
554	**Files:**
555	- Create: `internal/host/units/carlos-instance-cmd@.service`
556	- Test: `internal/host/units_isolation_test.go` (new test, and the lists at `:241` and `:266-286`, the seed-only rule at `:303`), `internal/host/units_hardening_test.go:45` and `:106-111`, `internal/host/host_test.go:32-64` (`expectedUnits`)
557	- Modify: `docs/site/cli.md:2076-2077` and `:2081` (RUN-C11, RUN-C12)
558
559	**Interfaces:**
560	- Consumes: `readUnit(t, name)` (`units_isolation_test.go:10`), `hasDirective` (`units_hardening_test.go`), `unitsFS` (`bootstrap.go:34-35`; the embed pattern `units/*.service` picks the new file up, and the bootstrap loop at `bootstrap.go:512-529` installs it).
561	- Produces: the template systemd starts for `UnitName(RoleCmd, host)`.
562
563	- [ ] **Step 1: Write the failing test**
564
565	Append to `units_isolation_test.go`:
566
567	```go
568	// The cmd template is the seed template outside six named lines (run spec
569	// §5.4, codex 4): every sandbox line is the seed's, so a command has
570	// exactly the reach a seed has. Comments and blank lines are free; every
571	// directive is compared, in order.
572	func TestCmdTemplateEqualsSeedOutsideTheNamedLines(t *testing.T) {
573	directives := func(name string) []string {
574	var out []string
575	for _, line := range strings.Split(readUnit(t, name), "\n") {
576	line = strings.TrimSpace(line)
577	if line == "" || strings.HasPrefix(line, "#") {
578	continue
579	}
580	out = append(out, line)
581	}
582	return out
583	}
584	swap := map[string]string{
585	"Description=carlos exec-backed instance %I (seed)": "Description=carlos exec-backed instance %I (cmd)",
586	"TimeoutStartSec=120": "TimeoutStartSec=6h5m",
587	"ExecStart=/opt/carlos/carlos instance-exec seed %I": "ExecStart=/opt/carlos/carlos instance-exec cmd %I",
588	"ExecStopPost=/opt/carlos/carlos instance-exit seed %I": "ExecStopPost=/opt/carlos/carlos instance-exit cmd %I",
589	"StandardOutput=file:/run/carlos/logs/%I.seed.sock": "StandardOutput=file:/run/carlos/logs/%I.cmd.sock",
590	"StandardError=inherit": "StandardError=file:/run/carlos/logs/%I.cmd-err.sock",
591	}
592	seed, cmd := directives("carlos-instance-seed@.service"), directives("carlos-instance-cmd@.service")
593	want := make([]string, len(seed))
594	used := map[string]bool{}
595	for i, l := range seed {
596	if c, ok := swap[l]; ok {
597	want[i], used[l] = c, true
598	continue
599	}
600	want[i] = l
601	}
602	for l := range swap {
603	if !used[l] {
604	t.Errorf("the seed template no longer has %q; the run spec names it as a cmd difference", l)
605	}
606	}
607	for i := 0; i < len(want) || i < len(cmd); i++ {
608	var w, g string
609	if i < len(want) {
610	w = want[i]
611	}
612	if i < len(cmd) {
613	g = cmd[i]
614	}
615	if w != g {
616	t.Fatalf("directive %d: cmd template has %q, want %q (the seed's line with the named swaps)", i, g, w)
617	}
618	}
619	for _, keep := range []string{"Type=oneshot", "RemainAfterExit=yes", "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"} {
620	if !hasDirective(readUnit(t, "carlos-instance-cmd@.service"), keep) {
621	t.Errorf("cmd template lacks %q", keep)
622	}
623	}
624	}
625	```
626
627	Extend the existing lists:
628
629	- `units_isolation_test.go:241`: `[]string{"carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-cmd@.service", "carlos-instance-probe.service"}`.
630	- the `wants` map at `:266-286` gains:
631
632	```go
633	"carlos-instance-cmd@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=6h5m", "ExecStart=/opt/carlos/carlos instance-exec cmd %I",
634	"ExecStopPost=/opt/carlos/carlos instance-exit cmd %I", "StandardOutput=file:/run/carlos/logs/%I.cmd.sock",
635	"StandardError=file:/run/carlos/logs/%I.cmd-err.sock", "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"},
636	```
637
638	- `:303`: the seed-only rule covers cmd too: `if (name == "carlos-instance-seed@.service" || name == "carlos-instance-cmd@.service") && strings.Contains(u, "\nSuccessExitStatus=") {`.
639	- `units_hardening_test.go:45`: the list gains `"carlos-instance-cmd@.service"`.
640	- `units_hardening_test.go:106-111`: the map gains `"carlos-instance-cmd@.service": "%I",`.
641	- `host_test.go` `expectedUnits`: add `"carlos-instance-cmd@.service",` before `"carlos-instance-control.service"` (the list is sorted).
642
643	- [ ] **Step 2: Run to verify it fails**
644
645	Run: `nice -n 10 go test ./internal/host/ -run 'TestCmdTemplateEqualsSeedOutsideTheNamedLines|TestInstanceTemplatesShareOneNamespace|TestUnitsEmbedComplete|TestNamespaceBlockCarriesTheHardening|TestInstanceTemplatesBindOnlyTheirOwnBuild' -count=1`
646	Expected: FAIL, `units/carlos-instance-cmd@.service: file does not exist`.
647
648	- [ ] **Step 3: Write the template**
649
650	Copy the seed template, then replace its head (from the first line through `StandardError=inherit`) with the head below. Everything from `# The instance's own two directories, the only writable data paths.` to the end of the file stays exactly as copied.
651
652	```bash
653	cp 'internal/host/units/carlos-instance-seed@.service' 'internal/host/units/carlos-instance-cmd@.service'
654	```
655
656	The head of `carlos-instance-cmd@.service`:
657
658	```ini
659	# One `carlos run` command against an isolated exec-backed instance (run
660	# spec §5.4). It is the seed template outside six lines, and a test holds
661	# it there (TestCmdTemplateEqualsSeedOutsideTheNamedLines): the same uid,
662	# namespace, bind mounts, slice and working directory as the instance, so
663	# the command has exactly the instance's reach. The agent starts it
664	# through the control service's POST /cmd and follows it with GET
665	# /cmd/<host>; the agent, not this file, enforces the run's own timeout.
666	[Unit]
667	Description=carlos exec-backed instance %I (cmd)
668	# Stop-only coupling, as for the seed: when the edge stops, the command
669	# stops, and only the agent starts one. A run picked before an agent
670	# restart is never run again (run spec §5.2).
671	BindsTo=carlos-edge.service
672	Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
673	After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
674
675	[Service]
676	Type=oneshot
677	RemainAfterExit=yes
678	# The result stays loaded, active (exited) or failed, until the agent has
679	# read it and stopped the unit (codex 4).
680	# A backstop past the run cap of 6 hours; the agent stops the unit at the
681	# run's own timeout.
682	TimeoutStartSec=6h5m
683	ExecStart=/opt/carlos/carlos instance-exec cmd %I
684	# The advisory exit record, as for the other templates. Not "+" (codex R3-2).
685	ExecStopPost=/opt/carlos/carlos instance-exit cmd %I
686	User=carlos
687	Group=carlos
688	Slice=carlos-instances-%i.slice
689	# SIGTERM to the command, SIGKILL to anything left 20 s later.
690	KillMode=mixed
691	TimeoutStopSec=20
692	# No SuccessExitStatus=: the agent reads the exit status itself.
693	# Two listeners, so stdout and stderr stay apart (run spec §5.5). PID 1
694	# opens both as root, outside the namespace, on the nosymfollow logs mount.
695	StandardOutput=file:/run/carlos/logs/%I.cmd.sock
696	StandardError=file:/run/carlos/logs/%I.cmd-err.sock
697	```
698
699	In `docs/site/cli.md`, the last sentence of `### carlos instance-exec` (`:2076-2077`, today wrapped across two lines) becomes `` `ExecStart=` of the four instance templates. `` on one line (RUN-C11; the gate matches docs raw, so the sentence must not wrap), and the first sentence of `### carlos instance-exit` (`:2081`) becomes `` `ExecStopPost=` of the four instance templates. `` (RUN-C12); the rest of each paragraph is unchanged. `runCopyBuilt` gains `"RUN-C11", "RUN-C12"`.
700
701	- [ ] **Step 4: Run to verify it passes**
702
703	Run: `nice -n 10 go test ./internal/host/ -run 'TestCmdTemplate|TestInstanceTemplates|TestUnitsEmbedComplete|TestNamespaceBlock|TestSameUIDAudit|TestControlUnitsShape' -count=1 && nice -n 10 go test . -run 'TestRunBuilt' -count=1 && nice -n 10 go test ./internal/docsite/ -count=1`
704	Expected: PASS. `TestSameUIDAuditIsCoveredByTheTemplates` iterates `unitNames()` and so covers the new file; `claimUnitCensus` (`claimcensus.go:86`) adds it to the census with no edit.
705
706	- [ ] **Step 5: Commit**
707
708	```bash
709	git add 'internal/host/units/carlos-instance-cmd@.service' internal/host/units_isolation_test.go internal/host/units_hardening_test.go internal/host/host_test.go docs/site/cli.md approved_run_copy_test.go
710	git commit -m "units: carlos-instance-cmd@, the seed template outside six lines (run spec §5.4)
711
712	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
713	```
714
715	---
716
717	### Task 4: `POST /cmd` and the `GET /cmd/{host}` long-poll
718
719	**Files:**
720	- Create: `internal/instancectl/cmd.go`, `internal/instancectl/cmd_test.go`
721	- Modify: `internal/instancectl/names.go` (`CmdStatus` beside `SeedResponse` at `:184`), `internal/instancectl/server.go:35-75` (three timing fields), `:249-263` (two routes), `internal/instancectl/client.go:23-28` (`DeadlineCmdPoll`) and after `:161` (`Cmd`, `CmdStatus`), `internal/instancectl/claim_test.go:11`
722
723	**Interfaces:**
724	- Consumes: `parseRequest(r, nil, true)`, `s.refuseStart(w, host, unit) bool`, `s.setTier(ctx, host, tier)`, `s.showValue(ctx, prop, unit) string`, `s.stopUnit(ctx, unit)`, `parseShow(out) map[string]string`, `writeJSON`, `s.claimAwareLegacy`, `StartResponse{InvocationID}`, `Client.do`, `hostRole`.
725	- Produces:
726	- `type CmdStatus struct { State, InvocationID, Result, Status, Code string }` with json `state`, `invocation_id`, `result`, `status`, `code`; `CmdRunning = "running"`, `CmdDone = "done"`, `CmdGone = "gone"`.
727	- `POST /cmd?host=&memory_max=` answers `StartResponse` at once; 409 while the unit is still loaded; 503 with claim control armed.
728	- `GET /cmd/{host}?invocation=<id>` answers `CmdStatus` within 30 s; 503 with claim control armed.
729	- `func (c *Client) Cmd(ctx context.Context, host string, memoryMax int64) (string, error)`; `func (c *Client) CmdStatus(ctx context.Context, host, invocation string) (CmdStatus, error)`; `DeadlineCmdPoll = 35 * time.Second`.
730	- `Code` is systemd's `ExecMainCode` as `systemctl show` prints it: a number (`1` exited, `2` killed, `3` dumped). Plan 3 reads both the numbers and the words.
731
732	- [ ] **Step 1: Write the failing tests**
733
734	`internal/instancectl/cmd_test.go`:
735
736	```go
737	package instancectl
738
739	import (
740	"context"
741	"encoding/json"
742	"net/http"
743	"net/http/httptest"
744	"strings"
745	"testing"
746	"time"
747	)
748
749	// set changes one answer under the fake's lock, for a test whose unit
750	// moves on while a handler polls.
751	func (f *fakeCtl) set(prefix, out string) {
752	f.mu.Lock()
753	f.answers[prefix] = out
754	f.mu.Unlock()
755	}
756
757	const cmdShowAll = "show -p InvocationID,ActiveState,SubState,Result,ExecMainStatus,ExecMainCode"
758
759	func fastCmd(s *Server) {
760	s.cmdPollEvery = 5 * time.Millisecond
761	s.cmdWaitBound = 300 * time.Millisecond
762	s.cmdStartBound = 300 * time.Millisecond
763	}
764
765	func TestCmdStartsWithoutBlockingAndAnswersTheNewInvocation(t *testing.T) {
766	ctl := &fakeCtl{answers: map[string]string{
767	"show -p ActiveState --value": "inactive\n",
768	"show -p InvocationID --value": "old\n",
769	}}
770	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
771	_, c := newServer(t, ctl, fastCmd)
772	resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912")
773	if resp.StatusCode != 200 {
774	t.Fatalf("status %d", resp.StatusCode)
775	}
776	var out StartResponse
777	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || out.InvocationID != "new" {
778	t.Fatalf("response %+v, %v", out, err)
779	}
780	calls := ctl.joined()
781	for _, want := range []string{"set-property --runtime carlos-instances-h.example.com.slice", "start --no-block carlos-instance-cmd@h.example.com.service"} {
782	if !strings.Contains(calls, want) {
783	t.Errorf("missing %q in\n%s", want, calls)
784	}
785	}
786	if strings.Contains(calls, "stop ") {
787	t.Errorf("a started command was stopped:\n%s", calls)
788	}
789	}
790
791	func TestCmdRefusesAUnitStillLoaded(t *testing.T) {
792	for _, state := range []string{"active", "activating", "deactivating"} {
793	ctl := &fakeCtl{answers: map[string]string{"show -p ActiveState --value": state + "\n"}}
794	_, c := newServer(t, ctl, fastCmd)
795	resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912")
796	if resp.StatusCode != http.StatusConflict {
797	t.Errorf("%s: status %d, want 409", state, resp.StatusCode)
798	}
799	if strings.Contains(ctl.joined(), "start ") {
800	t.Errorf("%s: a loaded unit was started again:\n%s", state, ctl.joined())
801	}
802	}
803	}
804
805	func TestCmdResetsAFailedUnitBeforeTheStart(t *testing.T) {
806	ctl := &fakeCtl{answers: map[string]string{
807	"show -p ActiveState --value": "failed\n",
808	"show -p InvocationID --value": "old\n",
809	}}
810	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
811	_, c := newServer(t, ctl, fastCmd)
812	if resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912"); resp.StatusCode != 200 {
813	t.Fatalf("status %d", resp.StatusCode)
814	}
815	calls := ctl.joined()
816	if r, s := strings.Index(calls, "reset-failed carlos-instance-cmd@"), strings.Index(calls, "start --no-block"); r < 0 || s < 0 || r > s {
817	t.Errorf("reset-failed must come before the start:\n%s", calls)
818	}
819	}
820
821	func TestCmdStopsAUnitThatNeverReportsAnInvocation(t *testing.T) {
822	ctl := &fakeCtl{answers: map[string]string{
823	"show -p ActiveState --value": "inactive\n",
824	"show -p InvocationID --value": "old\n",
825	}}
826	_, c := newServer(t, ctl, fastCmd)
827	if resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912"); resp.StatusCode != http.StatusBadGateway {
828	t.Fatalf("status %d, want 502", resp.StatusCode)
829	}
830	if !strings.Contains(ctl.joined(), "stop carlos-instance-cmd@h.example.com.service") {
831	t.Errorf("the unit was not stopped:\n%s", ctl.joined())
832	}
833	}
834
835	func getStatus(t *testing.T, c *http.Client, path string) CmdStatus {
836	t.Helper()
837	resp, err := c.Get("http://unix" + path)
838	if err != nil {
839	t.Fatal(err)
840	}
841	defer resp.Body.Close()
842	if resp.StatusCode != 200 {
843	t.Fatalf("GET %s: status %d", path, resp.StatusCode)
844	}
845	var st CmdStatus
846	if err := json.NewDecoder(resp.Body).Decode(&st); err != nil {
847	t.Fatal(err)
848	}
849	return st
850	}
851
852	func TestCmdStatusLongPollsUntilDone(t *testing.T) {
853	ctl := &fakeCtl{answers: map[string]string{
854	cmdShowAll: "InvocationID=inv1\nActiveState=activating\nSubState=start\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
855	}}
856	_, c := newServer(t, ctl, fastCmd)
857	go func() {
858	time.Sleep(50 * time.Millisecond)
859	ctl.set(cmdShowAll, "InvocationID=inv1\nActiveState=active\nSubState=exited\nResult=success\nExecMainStatus=3\nExecMainCode=1\n")
860	}()
861	st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1")
862	if st.State != CmdDone || st.Status != "3" || st.Code != "1" || st.InvocationID != "inv1" {
863	t.Fatalf("status = %+v", st)
864	}
865	}
866
867	func TestCmdStatusAnswersRunningAtTheBound(t *testing.T) {
868	ctl := &fakeCtl{answers: map[string]string{
869	cmdShowAll: "InvocationID=inv1\nActiveState=activating\nSubState=start\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
870	}}
871	_, c := newServer(t, ctl, fastCmd)
872	start := time.Now()
873	if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdRunning {
874	t.Fatalf("status = %+v", st)
875	}
876	if took := time.Since(start); took < 250*time.Millisecond {
877	t.Errorf("answered running after %s, before the bound", took)
878	}
879	}
880
881	func TestCmdStatusDoneCoversFailedAndInactive(t *testing.T) {
882	for _, show := range []string{
883	"InvocationID=inv1\nActiveState=failed\nSubState=failed\nResult=exit-code\nExecMainStatus=1\nExecMainCode=1\n",
884	"InvocationID=inv1\nActiveState=inactive\nSubState=dead\nResult=success\nExecMainStatus=0\nExecMainCode=1\n",
885	} {
886	ctl := &fakeCtl{answers: map[string]string{cmdShowAll: show}}
887	_, c := newServer(t, ctl, fastCmd)
888	if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdDone {
889	t.Errorf("%q: state %s, want done", show, st.State)
890	}
891	}
892	}
893
894	func TestCmdStatusIsGoneForAnotherInvocation(t *testing.T) {
895	for _, show := range []string{
896	"InvocationID=inv2\nActiveState=active\nSubState=exited\nResult=success\nExecMainStatus=0\nExecMainCode=1\n",
897	"InvocationID=\nActiveState=inactive\nSubState=dead\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
898	} {
899	ctl := &fakeCtl{answers: map[string]string{cmdShowAll: show}}
900	_, c := newServer(t, ctl, fastCmd)
901	if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdGone {
902	t.Errorf("%q: state %s, want gone", show, st.State)
903	}
904	}
905	}
906
907	func TestCmdRoutesRefuseWithClaimControlArmed(t *testing.T) {
908	called := false
909	s := &Server{ClaimStop: http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}), Systemctl: func(context.Context, ...string) (string, error) { called = true; return "", nil }}
910	for _, req := range []*http.Request{
911	httptest.NewRequest(http.MethodPost, "/cmd?host=app.example&memory_max=536870912", nil),
912	httptest.NewRequest(http.MethodGet, "/cmd/app.example?invocation=x", nil),
913	} {
914	w := httptest.NewRecorder()
915	s.Handler().ServeHTTP(w, req)
916	if w.Code != http.StatusServiceUnavailable {
917	t.Errorf("%s %s: status %d", req.Method, req.URL, w.Code)
918	}
919	}
920	if called {
921	t.Fatal("a cmd route reached systemctl with claim control armed")
922	}
923	}
924
925	func TestClientCmdAndCmdStatus(t *testing.T) {
926	ctl := &fakeCtl{answers: map[string]string{
927	"show -p ActiveState --value": "inactive\n",
928	"show -p InvocationID --value": "old\n",
929	cmdShowAll: "InvocationID=new\nActiveState=failed\nSubState=failed\nResult=exit-code\nExecMainStatus=2\nExecMainCode=1\n",
930	}}
931	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
932	_, cl := newClient(t, ctl, fastCmd)
933	inv, err := cl.Cmd(context.Background(), "h.example.com", 536870912)
934	if err != nil || inv != "new" {
935	t.Fatalf("Cmd = %q, %v", inv, err)
936	}
937	st, err := cl.CmdStatus(context.Background(), "h.example.com", inv)
938	if err != nil || st.State != CmdDone || st.Status != "2" {
939	t.Fatalf("CmdStatus = %+v, %v", st, err)
940	}
941	}
942	```
943
944	Add `"cmd"` to the verb list in `claim_test.go:11`'s loop is not enough for the GET route, which `TestCmdRoutesRefuseWithClaimControlArmed` covers; add it anyway so the legacy-verb test names every POST: `[]string{"start", "seed", "stop", "reset", "sweep", "cmd"}`.
945
946	- [ ] **Step 2: Run to verify they fail**
947
948	Run: `TMPDIR=/tmp/g nice -n 10 go test ./internal/instancectl/ -run 'TestCmd|TestClientCmd|TestClaimRootRefuses' -count=1`
949	Expected: FAIL, `s.cmdPollEvery undefined`.
950
951	- [ ] **Step 3: Implement**
952
953	`names.go`, after `SeedResponse`:
954
955	```go
956	// CmdStatus is GET /cmd/{host}'s answer (run spec §5.4). State is running,
957	// done once the named invocation's main process has exited (active
958	// (exited), failed, or inactive after a stop), or gone when the unit's
959	// invocation is another one or none. Result, Status and Code are systemd's
960	// Result, ExecMainStatus and ExecMainCode as `systemctl show` prints them.
961	type CmdStatus struct {
962	State string `json:"state"`
963	InvocationID string `json:"invocation_id"`
964	Result string `json:"result"`
965	Status string `json:"status"`
966	Code string `json:"code"`
967	}
968
969	const (
970	CmdRunning = "running"
971	CmdDone = "done"
972	CmdGone = "gone"
973	)
974	```
975
976	`server.go`, in `Server` after `DiskEvery`:
977
978	```go
979	// cmdPollEvery, cmdWaitBound and cmdStartBound pace the cmd routes:
980	// how often GET /cmd/{host} re-reads the unit (250 ms), how long it
981	// holds a request (30 s, run spec §5.4), and how long POST /cmd waits
982	// for the queued start to show a new invocation (10 s). Zero is the
983	// default; tests shorten them.
984	cmdPollEvery, cmdWaitBound, cmdStartBound time.Duration
985	```
986
987	`server.go:249-263`, after the `/sweep` line:
988
989	```go
990	mux.HandleFunc("POST /cmd", s.claimAwareLegacy(s.handleCmd))
991	mux.HandleFunc("GET /cmd/{host}", s.claimAwareLegacy(s.handleCmdStatus))
992	```
993
994	`internal/instancectl/cmd.go`:
995
996	```go
997	package instancectl
998
999	import (
1000	"context"
1001	"fmt"
1002	"net/http"
1003	"strings"
1004	"time"
1005
1006	"github.com/carlosframework/platform/internal/provision"
1007	)
1008
1009	func (s *Server) cmdPoll() time.Duration {
1010	if s.cmdPollEvery > 0 {
1011	return s.cmdPollEvery
1012	}
1013	return 250 * time.Millisecond
1014	}
1015
1016	func (s *Server) cmdWait() time.Duration {
1017	if s.cmdWaitBound > 0 {
1018	return s.cmdWaitBound
1019	}
1020	return 30 * time.Second
1021	}
1022
1023	func (s *Server) cmdStartWait() time.Duration {
1024	if s.cmdStartBound > 0 {
1025	return s.cmdStartBound
1026	}
1027	return 10 * time.Second
1028	}
1029
1030	// handleCmd starts host's cmd unit and answers at once with the new
1031	// invocation (run spec §5.4); /seed blocks, this never does. The agent
1032	// stops a cmd unit after every run, so a unit still loaded (active,
1033	// activating, deactivating) belongs to a run this request must not end: it
1034	// is refused, never restarted. A failed one is reset first, or start would
1035	// keep its old result.
1036	func (s *Server) handleCmd(w http.ResponseWriter, r *http.Request) {
1037	req, err := parseRequest(r, nil, true)
1038	if err != nil {
1039	http.Error(w, err.Error(), http.StatusBadRequest)
1040	return
1041	}
1042	ctx := r.Context()
1043	unit := UnitName(RoleCmd, req.host)
1044	switch state := s.showValue(ctx, "ActiveState", unit); state {
1045	case "active", "activating", "deactivating", "reloading", "refreshing":
1046	http.Error(w, "cmd: "+unit+" is "+state, http.StatusConflict)
1047	return
1048	case "failed":
1049	if out, err := s.Systemctl(ctx, "reset-failed", unit); err != nil {
1050	http.Error(w, "reset-failed: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
1051	return
1052	}
1053	}
1054	if s.refuseStart(w, req.host, unit) {
1055	return
1056	}
1057	if err := s.setTier(ctx, req.host, req.tier); err != nil {
1058	http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
1059	return
1060	}
1061	before := s.showValue(ctx, "InvocationID", unit)
1062	if out, err := s.Systemctl(ctx, "start", "--no-block", unit); err != nil {
1063	http.Error(w, "start: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
1064	return
1065	}
1066	// --no-block returns once the job is queued; the invocation exists
1067	// once the job runs.
1068	deadline := time.Now().Add(s.cmdStartWait())
1069	for {
1070	if inv := s.showValue(ctx, "InvocationID", unit); inv != "" && inv != before {
1071	s.logf("instance-control: %s: started %s memory.max=%d", req.host, unit, req.tier.MemoryMax)
1072	writeJSON(w, http.StatusOK, StartResponse{InvocationID: inv})
1073	return
1074	}
1075	if time.Now().After(deadline) || ctx.Err() != nil {
1076	dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
1077	s.stopUnit(dctx, unit)
1078	cancel()
1079	http.Error(w, "cmd: "+unit+" showed no new invocation", http.StatusBadGateway)
1080	return
1081	}
1082	time.Sleep(s.cmdPoll())
1083	}
1084	}
1085
1086	// handleCmdStatus is the bounded long-poll: it answers as soon as the named
1087	// invocation is done or gone, and running at the bound.
1088	func (s *Server) handleCmdStatus(w http.ResponseWriter, r *http.Request) {
1089	host := r.PathValue("host")
1090	if !provision.ValidHostShape(host) {
1091	http.Error(w, fmt.Sprintf("host %q is not a host", host), http.StatusBadRequest)
1092	return
1093	}
1094	want := r.URL.Query().Get("invocation")
1095	if want == "" {
1096	http.Error(w, "invocation is required", http.StatusBadRequest)
1097	return
1098	}
1099	ctx := r.Context()
1100	unit := UnitName(RoleCmd, host)
1101	deadline := time.Now().Add(s.cmdWait())
1102	for {
1103	out, err := s.Systemctl(ctx, "show", "-p", "InvocationID,ActiveState,SubState,Result,ExecMainStatus,ExecMainCode", unit)
1104	if err != nil {
1105	http.Error(w, "show: "+err.Error(), http.StatusBadGateway)
1106	return
1107	}
1108	st := cmdStatusOf(parseShow(out), want)
1109	if st.State != CmdRunning || !time.Now().Before(deadline) {
1110	writeJSON(w, http.StatusOK, st)
1111	return
1112	}
1113	select {
1114	case <-ctx.Done():
1115	return
1116	case <-time.After(s.cmdPoll()):
1117	}
1118	}
1119	}
1120
1121	// cmdStatusOf reads one show for the named invocation.
1122	func cmdStatusOf(m map[string]string, want string) CmdStatus {
1123	st := CmdStatus{InvocationID: m["InvocationID"], Result: m["Result"], Status: m["ExecMainStatus"], Code: m["ExecMainCode"]}
1124	active, sub := m["ActiveState"], m["SubState"]
1125	switch {
1126	case st.InvocationID != want:
1127	st.State = CmdGone
1128	case active == "failed", active == "inactive", active == "active" && sub == "exited":
1129	st.State = CmdDone
1130	default:
1131	st.State = CmdRunning
1132	}
1133	return st
1134	}
1135	```
1136
1137	`client.go`, in the deadline block:
1138
1139	```go
1140	// DeadlineCmdPoll bounds GET /cmd/{host}; the server answers within 30 s.
1141	DeadlineCmdPoll = 35 * time.Second
1142	```
1143
1144	and after `Stop`:
1145
1146	```go
1147	// Cmd starts host's cmd unit and returns its invocation (run spec §5.4).
1148	func (c *Client) Cmd(ctx context.Context, host string, memoryMax int64) (string, error) {
1149	q := hostRole(host, "")
1150	q.Set("memory_max", strconv.FormatInt(memoryMax, 10))
1151	var out StartResponse
1152	if _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/cmd", q, &out); err != nil {
1153	return "", err
1154	}
1155	return out.InvocationID, nil
1156	}
1157
1158	// CmdStatus long-polls the named invocation of host's cmd unit.
1159	func (c *Client) CmdStatus(ctx context.Context, host, invocation string) (CmdStatus, error) {
1160	var out CmdStatus
1161	_, _, err := c.do(ctx, DeadlineCmdPoll, http.MethodGet, "/cmd/"+url.PathEscape(host), url.Values{"invocation": {invocation}}, &out)
1162	return out, err
1163	}
1164	```
1165
1166	`newClient(t, ctl, configure...)` (`client_test.go:16`) already forwards configure hooks to `serveOnSocket`, so `fastCmd` reaches the server.
1167
1168	- [ ] **Step 4: Run to verify they pass**
1169
1170	Run: `TMPDIR=/tmp/g nice -n 10 go test ./internal/instancectl/ -count=1`
1171	Expected: PASS.
1172
1173	- [ ] **Step 5: Commit**
1174
1175	```bash
1176	git add internal/instancectl/cmd.go internal/instancectl/cmd_test.go internal/instancectl/names.go internal/instancectl/server.go internal/instancectl/client.go internal/instancectl/claim_test.go
1177	git commit -m "instancectl: POST /cmd and the GET /cmd/{host} long-poll (run spec §5.4, codex 4)
1178
1179	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
1180	```
1181
1182	---
1183
1184	### Task 5: `instancefs.RemoveAllBeneath`
1185
1186	Plan 3 deletes `.carlos-run/<id>/` after each run and every `.carlos-run/` at agent start. The instance shares uid `carlos` with the agent and can plant symlinks and subdirectories there, and `RemoveBeneath` (`dirs_linux.go:118`) removes only a non-directory entry.
1187
1188	**Files:**
1189	- Modify: `internal/instancefs/dirs_linux.go` (append), `internal/instancefs/instancefs_other.go` (stub)
1190	- Test: `internal/instancefs/dirs_linux_test.go` (append)
1191
1192	**Interfaces:**
1193	- Consumes: `openDir(dir string) (int, error)` (`instancefs_linux.go:30`), `bareName(name string) error` (`instancefs.go:72`), `ErrSymlink`, `ErrEscapes`.
1194	- Produces: `func RemoveAllBeneath(dir, name string) error`: removes `name` directly under `dir` and everything below it, following no symlink anywhere; a symlink is unlinked, never its target; a missing `name` is nil; `dir` reached through a symlink is `ErrSymlink`; a `name` with a slash is `ErrEscapes`. The `!linux` stub returns `errors.ErrUnsupported` after `bareName`.
1195
1196	- [ ] **Step 1: Write the failing tests**
1197
1198	Append to `dirs_linux_test.go`:
1199
1200	```go
1201	func TestRemoveAllBeneathRemovesATree(t *testing.T) {
1202	dir := t.TempDir()
1203	for _, p := range []string{"run/a", "run/sub/b", "run/sub/deeper/c"} {
1204	full := filepath.Join(dir, p)
1205	if err := os.MkdirAll(filepath.Dir(full), 0o700); err != nil {
1206	t.Fatal(err)
1207	}
1208	if err := os.WriteFile(full, []byte("x"), 0o600); err != nil {
1209	t.Fatal(err)
1210	}
1211	}
1212	if err := RemoveAllBeneath(dir, "run"); err != nil {
1213	t.Fatal(err)
1214	}
1215	if _, err := os.Lstat(filepath.Join(dir, "run")); !os.IsNotExist(err) {
1216	t.Fatalf("run survived: %v", err)
1217	}
1218	if err := RemoveAllBeneath(dir, "run"); err != nil {
1219	t.Fatalf("a missing name must be nil: %v", err)
1220	}
1221	}
1222
1223	func TestRemoveAllBeneathNeverFollowsASymlink(t *testing.T) {
1224	dir, outside := t.TempDir(), t.TempDir()
1225	keep := filepath.Join(outside, "keep")
1226	if err := os.WriteFile(keep, []byte("x"), 0o600); err != nil {
1227	t.Fatal(err)
1228	}
1229	if err := os.MkdirAll(filepath.Join(dir, "run"), 0o700); err != nil {
1230	t.Fatal(err)
1231	}
1232	if err := os.Symlink(outside, filepath.Join(dir, "run", "link")); err != nil {
1233	t.Fatal(err)
1234	}
1235	if err := os.Symlink(outside, filepath.Join(dir, "top")); err != nil {
1236	t.Fatal(err)
1237	}
1238	if err := RemoveAllBeneath(dir, "run"); err != nil {
1239	t.Fatal(err)
1240	}
1241	if err := RemoveAllBeneath(dir, "top"); err != nil {
1242	t.Fatal(err)
1243	}
1244	if _, err := os.Stat(keep); err != nil {
1245	t.Fatalf("a symlink's target was removed: %v", err)
1246	}
1247	if _, err := os.Lstat(filepath.Join(dir, "top")); !os.IsNotExist(err) {
1248	t.Fatalf("the top-level symlink itself survived: %v", err)
1249	}
1250	}
1251
1252	func TestRemoveAllBeneathRefusesPathsAndSymlinkedParents(t *testing.T) {
1253	dir, outside := t.TempDir(), t.TempDir()
1254	if err := RemoveAllBeneath(dir, "a/b"); !errors.Is(err, ErrEscapes) {
1255	t.Errorf("a/b: %v, want ErrEscapes", err)
1256	}
1257	link := filepath.Join(dir, "parent")
1258	if err := os.Symlink(outside, link); err != nil {
1259	t.Fatal(err)
1260	}
1261	if err := os.Mkdir(filepath.Join(outside, "run"), 0o700); err != nil {
1262	t.Fatal(err)
1263	}
1264	if err := RemoveAllBeneath(link, "run"); !errors.Is(err, ErrSymlink) {
1265	t.Errorf("symlinked parent: %v, want ErrSymlink", err)
1266	}
1267	if _, err := os.Stat(filepath.Join(outside, "run")); err != nil {
1268	t.Errorf("removed through a symlinked parent: %v", err)
1269	}
1270	}
1271	```
1272
1273	(`dirs_linux_test.go` already imports `errors`, `os`, `path/filepath` and `testing`; add any it lacks.)
1274
1275	- [ ] **Step 2: Run to verify they fail**
1276
1277	Run: `nice -n 10 go test ./internal/instancefs/ -run TestRemoveAllBeneath -count=1`
1278	Expected: FAIL, `undefined: RemoveAllBeneath`.
1279
1280	- [ ] **Step 3: Implement**
1281
1282	Append to `dirs_linux.go`:
1283
1284	```go
1285	// RemoveAllBeneath removes name, directly under dir, and everything below
1286	// it. No symlink is followed at any depth: dir itself is opened without
1287	// following one (ErrSymlink), every entry is examined with fstatat
1288	// AT_SYMLINK_NOFOLLOW, and a symlink is unlinked, never its target. A
1289	// tenant sharing the agent's uid can plant links and directories in a run
1290	// directory (run spec §5.2); this is the only way the agent deletes one.
1291	// A missing name is not an error.
1292	func RemoveAllBeneath(dir, name string) error {
1293	if err := bareName(name); err != nil {
1294	return err
1295	}
1296	dfd, err := openDir(dir)
1297	if err != nil {
1298	return err
1299	}
1300	defer unix.Close(dfd)
1301	return removeAt(dfd, name, filepath.Join(dir, name), 0)
1302	}
1303
1304	// removeAt removes name under the directory fd parent; path is for errors.
1305	func removeAt(parent int, name, path string, depth int) error {
1306	if depth > 64 {
1307	return fmt.Errorf("instancefs: %s is nested more than 64 deep", path)
1308	}
1309	var st unix.Stat_t
1310	if err := unix.Fstatat(parent, name, &st, unix.AT_SYMLINK_NOFOLLOW); err != nil {
1311	if errors.Is(err, unix.ENOENT) {
1312	return nil
1313	}
1314	return &os.PathError{Op: "fstatat", Path: path, Err: err}
1315	}
1316	if st.Mode&unix.S_IFMT != unix.S_IFDIR {
1317	if err := unix.Unlinkat(parent, name, 0); err != nil && !errors.Is(err, unix.ENOENT) {
1318	return &os.PathError{Op: "unlinkat", Path: path, Err: err}
1319	}
1320	return nil
1321	}
1322	fd, err := unix.Openat(parent, name, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
1323	if err != nil {
1324	return &os.PathError{Op: "openat", Path: path, Err: err}
1325	}
1326	f := os.NewFile(uintptr(fd), path)
1327	names, err := f.Readdirnames(-1)
1328	if err != nil {
1329	f.Close()
1330	return err
1331	}
1332	for _, n := range names {
1333	if err := removeAt(fd, n, filepath.Join(path, n), depth+1); err != nil {
1334	f.Close()
1335	return err
1336	}
1337	}
1338	f.Close()
1339	if err := unix.Unlinkat(parent, name, unix.AT_REMOVEDIR); err != nil && !errors.Is(err, unix.ENOENT) {
1340	return &os.PathError{Op: "unlinkat", Path: path, Err: err}
1341	}
1342	return nil
1343	}
1344	```
1345
1346	Append to `instancefs_other.go`:
1347
1348	```go
1349	func RemoveAllBeneath(dir, name string) error {
1350	if err := bareName(name); err != nil {
1351	return err
1352	}
1353	return errors.ErrUnsupported
1354	}
1355	```
1356
1357	- [ ] **Step 4: Run to verify they pass**
1358
1359	Run: `nice -n 10 go test ./internal/instancefs/ -count=1 && GOOS=darwin go build ./internal/instancefs/ && GOOS=windows go build ./internal/instancefs/`
1360	Expected: PASS, and both cross builds succeed.
1361
1362	- [ ] **Step 5: Commit**
1363
1364	```bash
1365	git add internal/instancefs/dirs_linux.go internal/instancefs/instancefs_other.go internal/instancefs/dirs_linux_test.go
1366	git commit -m "instancefs: RemoveAllBeneath, a tree removal that follows no symlink
1367
1368	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
1369	```
1370
1371	---
1372
1373	### Task 6: `ExecBackend.StartCmd`: env, launch file, streams, unit
1374
1375	**Files:**
1376	- Create: `internal/activator/cmdunit.go`, `internal/activator/cmdunit_test.go`
1377	- Modify: `internal/activator/control.go:17-24` (two methods), `internal/activator/prepare.go:44-57` (env for cmd), `:93-99` (no app-log listener for cmd), `:131-137` (`logSocketNamed`), `internal/activator/backend_exec.go:1024-1056` (`waitRoleCgroupEmpty`), `internal/activator/backend_unit.go` (after `:341`, `SplitBackend.StartCmd`), `internal/activator/unitproc_test.go:24-70` (`fakeControl`)
1378
1379	**Interfaces:**
1380	- Consumes: `b.prepare(rt, role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error)`, `b.nestedGate(rt)`, `b.ensureLiveLink(ctx, host)`, `b.Layout(rt)`, `b.liveLink(host)`, `b.tier(rt)`, `b.writeLaunch(p, role, host, argv, env)`, `childEnv`, `withVar`, `withoutVars`, `b.Region`, `edge.SecureUnixListen`, `instancectl.CmdStatus`, `Client.Cmd`, `Client.CmdStatus`.
1381	- Produces:
1382	- `Control` gains `Cmd(ctx, host string, memoryMax int64) (string, error)` and `CmdStatus(ctx, host, invocation string) (instancectl.CmdStatus, error)`.
1383	- `type CmdSpec struct { RunID string; Args []string; RunDir string; Stdout, Stderr io.Writer; Prepare func(ctx context.Context) error }` (`Prepare` is used by Task 8).
1384	- `type CmdUnit interface { Invocation() string; Poll(ctx context.Context) (instancectl.CmdStatus, error); StopAndSettle(ctx context.Context) error }`.
1385	- `type CmdBackend interface { StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) }`; `*ExecBackend` and `*SplitBackend` implement it.
1386	- `const CmdRunDirName = ".carlos-run"`; `StartCmd` refuses a `spec.RunDir` other than `<instance dir>/.carlos-run/<RunID>`.
1387	- `var ErrCmdNotNested = errors.New("activator: the instance is not on the nested layout")`.
1388	- `func (b *ExecBackend) waitRoleCgroupEmpty(ctx context.Context, rt registry.Route, role instancectl.Role) error`.
1389
1390	- [ ] **Step 1: Write the failing tests**
1391
1392	Extend `fakeControl` in `unitproc_test.go` with two fields and two methods:
1393
1394	```go
1395	cmdInv string
1396	cmdErr error
1397	cmdStatus instancectl.CmdStatus
1398	```
1399
1400	```go
1401	func (f *fakeControl) Cmd(_ context.Context, host string, max int64) (string, error) {
1402	f.rec("cmd " + host)
1403	return f.cmdInv, f.cmdErr
1404	}
1405	func (f *fakeControl) CmdStatus(_ context.Context, host, inv string) (instancectl.CmdStatus, error) {
1406	f.rec("cmdstatus " + host + " " + inv)
1407	return f.cmdStatus, nil
1408	}
1409	```
1410
1411	`internal/activator/cmdunit_test.go`:
1412
1413	```go
1414	package activator
1415
1416	import (
1417	"bytes"
1418	"context"
1419	"encoding/json"
1420	"errors"
1421	"net"
1422	"os"
1423	"path/filepath"
1424	"strings"
1425	"sync"
1426	"testing"
1427	"time"
1428
1429	"github.com/carlosframework/platform/internal/instancectl"
1430	"github.com/carlosframework/platform/internal/provision"
1431	"github.com/carlosframework/platform/internal/registry"
1432	)
1433
1434	// syncBuffer is a bytes.Buffer two goroutines may write.
1435	type syncBuffer struct {
1436	mu sync.Mutex
1437	b bytes.Buffer
1438	}
1439
1440	func (s *syncBuffer) Write(p []byte) (int, error) { s.mu.Lock(); defer s.mu.Unlock(); return s.b.Write(p) }
1441	func (s *syncBuffer) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }
1442
1443	func TestCmdVarsComposeTheRunEnvironment(t *testing.T) {
1444	b := &ExecBackend{Region: "ie"}
1445	rt := registry.Route{Host: "h.example.com"}
1446	file := map[string]string{
1447	"APP_KEY": "k", "CARLOS_ADMIN_TOKEN": "tok", "CARLOS_CONTROL_SOCKET": "/x", "CARLOS_PURGE": "header",
1448	"CARLOS_REGION": "us", "CARLOS_HOST": "liar.example.com", "CARLOS_RUN_ID": "planted",
1449	}
1450	got := b.cmdVars(rt, file, "/data/instances/h.example.com/app.db", "run_a", "/data/instances/h.example.com/.carlos-run/run_a")
1451	want := map[string]string{
1452	"APP_KEY": "k", "CARLOS_ADMIN_TOKEN": "tok", "CARLOS_REGION": "ie", "CARLOS_HOST": "h.example.com",
1453	"CARLOS_DB": "/data/instances/h.example.com/app.db", "CARLOS_RUN_ID": "run_a",
1454	"CARLOS_RUN_DIR": "/data/instances/h.example.com/.carlos-run/run_a",
1455	}
1456	if len(got) != len(want) {
1457	t.Fatalf("env = %v", got)
1458	}
1459	for k, v := range want {
1460	if got[k] != v {
1461	t.Errorf("%s = %q, want %q", k, got[k], v)
1462	}
1463	}
1464	b.Region = ""
1465	if _, ok := b.cmdVars(rt, file, "/db", "run_a", "/d")["CARLOS_REGION"]; ok {
1466	t.Error("an empty backend region must remove CARLOS_REGION (codex 7)")
1467	}
1468	}
1469
1470	func startTestCmd(t *testing.T) (*ExecBackend, registry.Route, *fakeControl, *fakeUnit, CmdUnit, *syncBuffer, *syncBuffer) {
1471	t.Helper()
1472	ctl := &fakeControl{cmdInv: "inv1"}
1473	unit := &fakeUnit{}
1474	b, rt := nestedBackend(t, ctl, unit)
1475	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutNested, "" }
1476	out, errs := &syncBuffer{}, &syncBuffer{}
1477	dir := filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), CmdRunDirName, "run_a")
1478	u, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a", Args: []string{"import", "--dry-run"}, RunDir: dir, Stdout: out, Stderr: errs})
1479	if err != nil {
1480	t.Fatalf("StartCmd: %v", err)
1481	}
1482	return b, rt, ctl, unit, u, out, errs
1483	}
1484
1485	func TestStartCmdWritesTheLaunchFileAndStartsTheUnit(t *testing.T) {
1486	b, rt, ctl, _, u, _, _ := startTestCmd(t)
1487	if u.Invocation() != "inv1" || !strings.Contains(ctl.joined(), "cmd "+rt.Host) {
1488	t.Fatalf("invocation %q, calls\n%s", u.Invocation(), ctl.joined())
1489	}
1490	raw, err := os.ReadFile(filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), "launch.cmd.json"))
1491	if err != nil {
1492	t.Fatal(err)
1493	}
1494	var l instancectl.Launch
1495	if err := json.Unmarshal(raw, &l); err != nil {
1496	t.Fatal(err)
1497	}
1498	if len(l.Argv) != 3 || l.Argv[0] != b.liveLink(rt.Host) || l.Argv[1] != "import" || l.Argv[2] != "--dry-run" {
1499	t.Errorf("argv = %q: exactly the binary and what followed --", l.Argv)
1500	}
1501	env := strings.Join(l.Env, "\n")
1502	for _, want := range []string{"CARLOS_RUN_ID=run_a", "CARLOS_HOST=" + rt.Host, "CARLOS_DB=" + rt.DBPath} {
1503	if !strings.Contains(env, want) {
1504	t.Errorf("env lacks %q", want)
1505	}
1506	}
1507	if strings.Contains(env, "CARLOS_CONTROL_SOCKET=") {
1508	t.Error("a command has no control socket")
1509	}
1510	}
1511
1512	func TestStartCmdRefusesAFlatInstanceAndAForeignRunDir(t *testing.T) {
1513	ctl := &fakeControl{cmdInv: "inv1"}
1514	b, rt := nestedBackend(t, ctl, &fakeUnit{})
1515	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutFlat, "test" }
1516	if _, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdNotNested) {
1517	t.Errorf("flat: %v, want ErrCmdNotNested", err)
1518	}
1519	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutNested, "" }
1520	if _, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a", RunDir: "/elsewhere"}); err == nil {
1521	t.Error("a run directory outside the instance's .carlos-run was accepted")
1522	}
1523	if strings.Contains(ctl.joined(), "cmd ") {
1524	t.Errorf("a refused start reached the control service:\n%s", ctl.joined())
1525	}
1526	}
1527
1528	func TestStopAndSettleDrainsBothStreams(t *testing.T) {
1529	b, rt, ctl, unit, u, out, errs := startTestCmd(t)
1530	unit.set("active", "inv1", true, true)
1531	write := func(path, s string) {
1532	c, err := net.Dial("unix", path)
1533	if err != nil {
1534	t.Fatal(err)
1535	}
1536	if _, err := c.Write([]byte(s)); err != nil {
1537	t.Fatal(err)
1538	}
1539	c.Close()
1540	}
1541	write(b.logSocketNamed(rt.Host, string(instancectl.RoleCmd)), "out-1\nout-2")
1542	write(b.logSocketNamed(rt.Host, cmdErrSocket), "err-1")
1543	go func() {
1544	time.Sleep(30 * time.Millisecond)
1545	unit.set("inactive", "inv1", false, false)
1546	}()
1547	if err := u.StopAndSettle(context.Background()); err != nil {
1548	t.Fatalf("StopAndSettle: %v", err)
1549	}
1550	if out.String() != "out-1\nout-2" || errs.String() != "err-1" {
1551	t.Errorf("stdout %q stderr %q", out.String(), errs.String())
1552	}
1553	if !strings.Contains(ctl.joined(), "stop "+rt.Host+" cmd") {
1554	t.Errorf("the cmd unit was not stopped:\n%s", ctl.joined())
1555	}
1556	if err := u.StopAndSettle(context.Background()); err != nil {
1557	t.Errorf("a second StopAndSettle: %v", err)
1558	}
1559	}
1560	```
1561
1562	`b.layoutFn` does not exist on `ExecBackend` yet: Step 3 adds the seam (nil keeps `Classify`). `nestedBackend` (`unitproc_test.go`) makes its root with `os.MkdirTemp("", "nb")`, so the stream sockets need `TMPDIR=/tmp/g`.
1563
1564	- [ ] **Step 2: Run to verify they fail**
1565
1566	Run: `TMPDIR=/tmp/g nice -n 10 go test ./internal/activator/ -run 'TestCmdVars|TestStartCmd|TestStopAndSettle' -count=1`
1567	Expected: FAIL, `b.cmdVars undefined`.
1568
1569	- [ ] **Step 3: Implement**
1570
1571	`control.go`, the interface:
1572
1573	```go
1574	type Control interface {
1575	Start(ctx context.Context, host string, role instancectl.Role, memoryMax int64) (string, error)
1576	Seed(ctx context.Context, host string, memoryMax int64) (instancectl.SeedResponse, error)
1577	// Cmd starts host's cmd unit and returns its invocation; CmdStatus
1578	// long-polls it (run spec §5.4).
1579	Cmd(ctx context.Context, host string, memoryMax int64) (string, error)
1580	CmdStatus(ctx context.Context, host, invocation string) (instancectl.CmdStatus, error)
1581	Stop(ctx context.Context, host string, role instancectl.Role) error
1582	Reset(ctx context.Context, host string, role instancectl.Role) error
1583	Sweep(ctx context.Context) (instancectl.SweepResponse, error)
1584	Ready(ctx context.Context) error
1585	}
1586	```
1587
1588	`prepare.go:44`: the instance environment for the cmd role too:
1589
1590	```go
1591	if role == instancectl.RoleRun || role == instancectl.RoleSeed || role == instancectl.RoleCmd {
1592	```
1593
1594	`prepare.go:93-99`: no app-log listener for cmd (its output goes only to its own chunks, never to the journal or the app log):
1595
1596	```go
1597	var logs *logListener
1598	if role != instancectl.RoleCmd {
1599	var err error
1600	logs, err = listenLogs(b.logSocket(rt.Host, role), b.logEmitter(rt))
1601	if err != nil {
1602	if ctl != nil {
1603	ctl.Close()
1604	}
1605	return p, nil, nil, nil, fmt.Errorf("log listener for %s %s: %w", rt.Host, role, err)
1606	}
1607	}
1608	return p, ctl, logs, env, nil
1609	```
1610
1611	`prepare.go:131-137`:
1612
1613	```go
1614	// logSocket is the role's log socket path, honouring the test override of
1615	// the run directory (RunDirOverride).
1616	func (b *ExecBackend) logSocket(host string, role instancectl.Role) string {
1617	return b.logSocketNamed(host, string(role))
1618	}
1619
1620	// logSocketNamed is logSocket for any listener name: the cmd role has two,
1621	// "cmd" and "cmd-err" (run spec §5.4).
1622	func (b *ExecBackend) logSocketNamed(host, name string) string {
1623	if b.RunDirOverride != "" {
1624	return filepath.Join(b.RunDirOverride, "logs", provision.InstanceID(host)+"."+name+".sock")
1625	}
1626	return provision.LogSocket(host, name)
1627	}
1628	```
1629
1630	`backend_exec.go:1024-1056`: the body moves to a role-taking function and the seed wrapper calls it:
1631
1632	```go
1633	func (b *ExecBackend) waitSeedCgroupEmpty(ctx context.Context, rt registry.Route) error {
1634	return b.waitRoleCgroupEmpty(ctx, rt, instancectl.RoleSeed)
1635	}
1636
1637	// waitRoleCgroupEmpty polls role's unit's cgroup.events until it reports
1638	// empty or actStopTimeout elapses, honouring ctx and b.ObservePoll.
1639	func (b *ExecBackend) waitRoleCgroupEmpty(ctx context.Context, rt registry.Route, role instancectl.Role) error {
1640	events := b.CgroupEvents
1641	if events == nil {
1642	events = cgroupEventsUnder(b.cgroupRoot())
1643	}
1644	poll := b.ObservePoll
1645	if poll <= 0 {
1646	poll = time.Second
1647	}
1648	slice := instancectl.SliceName(rt.Host)
1649	unit := instancectl.UnitName(role, rt.Host)
1650	deadline := time.Now().Add(actStopTimeout)
1651	for {
1652	exists, populated, err := events(slice, unit)
1653	if err != nil {
1654	return err
1655	}
1656	if !exists || !populated {
1657	return nil
1658	}
1659	if time.Now().After(deadline) {
1660	return fmt.Errorf("processes remained after %s", actStopTimeout)
1661	}
1662	select {
1663	case <-ctx.Done():
1664	return ctx.Err()
1665	case <-time.After(poll):
1666	}
1667	}
1668	}
1669	```
1670
1671	`ExecBackend` gains a test seam beside `RunDirOverride` (`backend_exec.go`, in the struct):
1672
1673	```go
1674	// layoutFn, when set, replaces Classify in StartCmd's layout check; a
1675	// unit test's way to stand a temp directory in for a nested host.
1676	layoutFn func(rt registry.Route) (Layout, string)
1677	```
1678
1679	`internal/activator/cmdunit.go`:
1680
1681	```go
1682	package activator
1683
1684	import (
1685	"context"
1686	"errors"
1687	"fmt"
1688	"io"
1689	"log"
1690	"net"
1691	"path/filepath"
1692	"sync"
1693	"sync/atomic"
1694	"time"
1695
1696	"github.com/carlosframework/platform/internal/edge"
1697	"github.com/carlosframework/platform/internal/instancectl"
1698	"github.com/carlosframework/platform/internal/registry"
1699	)
1700
1701	// CmdRunDirName is the directory under an instance's own data directory
1702	// that holds each run's --file uploads (run spec §5.2 step 3).
1703	const CmdRunDirName = ".carlos-run"
1704
1705	// cmdErrSocket is the stderr listener's name; stdout's is the role's.
1706	const cmdErrSocket = "cmd-err"
1707
1708	// cmdDrainBound is how long a stream may take to reach EOF once its unit
1709	// has stopped.
1710	const cmdDrainBound = 5 * time.Second
1711
1712	// ErrCmdNotNested: the instance is not on the nested layout, so a command
1713	// would run with the agent's reach (run spec §5.2 step 1, refused "flat").
1714	var ErrCmdNotNested = errors.New("activator: the instance is not on the nested layout")
1715
1716	// CmdSpec is one run's launch, as the box's run pass hands it over.
1717	type CmdSpec struct {
1718	RunID string
1719	// Args is exactly what followed `--`; the binary is the host's live link.
1720	Args []string
1721	// RunDir is CARLOS_RUN_DIR: <instance dir>/.carlos-run/<RunID>, made and
1722	// filled by the caller.
1723	RunDir string
1724	// Stdout and Stderr receive the unit's two streams, byte for byte.
1725	Stdout, Stderr io.Writer
1726	// Prepare runs after the run is attached to the awake entry and before
1727	// the unit starts (Service.RunCmd); an error ends the run before launch.
1728	Prepare func(ctx context.Context) error
1729	}
1730
1731	// CmdUnit is a started cmd unit.
1732	type CmdUnit interface {
1733	Invocation() string
1734	// Poll long-polls the invocation, at most about 30 seconds.
1735	Poll(ctx context.Context) (instancectl.CmdStatus, error)
1736	// StopAndSettle stops the unit (SIGTERM, then SIGKILL after 20 s),
1737	// proves its cgroup empty and reads both streams to their end. It is
1738	// safe to call more than once; later calls return the first's error.
1739	StopAndSettle(ctx context.Context) error
1740	}
1741
1742	// CmdBackend is the backend capability Service.RunCmd needs.
1743	type CmdBackend interface {
1744	StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error)
1745	}
1746
1747	var _ CmdBackend = (*ExecBackend)(nil)
1748
1749	// cmdVars is a run's environment over fileEnv (run spec §4, codex 7): the
1750	// control-channel variables stripped as seedVars strips them, CARLOS_REGION
1751	// set or removed from the backend's region as agentOwnedVars does,
1752	// CARLOS_HOST from the row, and the three run variables. fileEnv carries
1753	// CARLOS_ADMIN_TOKEN through.
1754	func (b *ExecBackend) cmdVars(rt registry.Route, fileEnv map[string]string, dbPath, runID, runDir string) map[string]string {
1755	v := withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE")
1756	if b.Region != "" {
1757	v = withVar(v, "CARLOS_REGION", b.Region)
1758	} else {
1759	v = withoutVars(v, "CARLOS_REGION")
1760	}
1761	v = withVar(v, "CARLOS_HOST", rt.Host)
1762	v = withVar(v, "CARLOS_DB", dbPath)
1763	v = withVar(v, "CARLOS_RUN_ID", runID)
1764	return withVar(v, "CARLOS_RUN_DIR", runDir)
1765	}
1766
1767	func (b *ExecBackend) cmdLayout(rt registry.Route) (Layout, string) {
1768	if b.layoutFn != nil {
1769	return b.layoutFn(rt)
1770	}
1771	return b.Layout(rt)
1772	}
1773
1774	// StartCmd starts one run's cmd unit (run spec §5.4): the layout check, the
1775	// nested gate, the live link, the instance's directories and environment,
1776	// the two stream listeners, launch.cmd.json, then POST /cmd. It returns
1777	// once the unit has an invocation. The environment is never logged.
1778	func (b *ExecBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
1779	if l, why := b.cmdLayout(rt); l != LayoutNested {
1780	return nil, fmt.Errorf("%w: %s is %s (%s)", ErrCmdNotNested, rt.Host, l, why)
1781	}
1782	if err := b.nestedGate(rt); err != nil {
1783	return nil, err
1784	}
1785	if err := b.ensureLiveLink(ctx, rt.Host); err != nil {
1786	return nil, err
1787	}
1788	p, _, _, fileEnv, err := b.prepare(rt, instancectl.RoleCmd)
1789	if err != nil {
1790	return nil, err
1791	}
1792	if want := filepath.Join(p.dataDir, CmdRunDirName, spec.RunID); spec.RunDir != want {
1793	return nil, fmt.Errorf("%s: run directory %q is not %q", rt.Host, spec.RunDir, want)
1794	}
1795	out, err := listenCmdStream(b.logSocketNamed(rt.Host, string(instancectl.RoleCmd)), spec.Stdout)
1796	if err != nil {
1797	return nil, fmt.Errorf("stdout listener for %s: %w", rt.Host, err)
1798	}
1799	errs, err := listenCmdStream(b.logSocketNamed(rt.Host, cmdErrSocket), spec.Stderr)
1800	if err != nil {
1801	out.Close()
1802	return nil, fmt.Errorf("stderr listener for %s: %w", rt.Host, err)
1803	}
1804	fail := func(err error) (CmdUnit, error) {
1805	out.Close()
1806	errs.Close()
1807	return nil, err
1808	}
1809	argv := append([]string{b.liveLink(rt.Host)}, spec.Args...)
1810	env := childEnv(b.cmdVars(rt, fileEnv, p.dbPath, spec.RunID, spec.RunDir))
1811	if err := b.writeLaunch(p, instancectl.RoleCmd, rt.Host, argv, env); err != nil {
1812	return fail(err)
1813	}
1814	inv, err := b.Control.Cmd(ctx, rt.Host, b.tier(rt).MemoryMax)
1815	if err != nil {
1816	return fail(fmt.Errorf("%s: start cmd unit: %w", rt.Host, err))
1817	}
1818	log.Printf("activator: %s: started %s for run %s", rt.Host, instancectl.UnitName(instancectl.RoleCmd, rt.Host), spec.RunID)
1819	return &cmdUnit{b: b, rt: rt, inv: inv, out: out, errs: errs}, nil
1820	}
1821
1822	type cmdUnit struct {
1823	b *ExecBackend
1824	rt registry.Route
1825	inv string
1826	out, errs *cmdStream
1827	once sync.Once
1828	err error
1829	}
1830
1831	func (u *cmdUnit) Invocation() string { return u.inv }
1832
1833	func (u *cmdUnit) Poll(ctx context.Context) (instancectl.CmdStatus, error) {
1834	return u.b.Control.CmdStatus(ctx, u.rt.Host, u.inv)
1835	}
1836
1837	func (u *cmdUnit) StopAndSettle(ctx context.Context) error {
1838	u.once.Do(func() {
1839	sctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), instancectl.DeadlineControl)
1840	stopErr := u.b.Control.Stop(sctx, u.rt.Host, instancectl.RoleCmd)
1841	cancel()
1842	wctx, wcancel := context.WithTimeout(context.WithoutCancel(ctx), actStopTimeout)
1843	emptyErr := u.b.waitRoleCgroupEmpty(wctx, u.rt, instancectl.RoleCmd)
1844	wcancel()
1845	u.err = errors.Join(stopErr, emptyErr, u.out.Drain(cmdDrainBound), u.errs.Drain(cmdDrainBound))
1846	})
1847	return u.err
1848	}
1849
1850	// cmdStream is the agent's end of one cmd stream (run spec §5.5): the
1851	// unit's StandardOutput=file: or StandardError=file: connects once, at
1852	// exec, and every byte is copied to w unchanged until EOF. Nothing goes to
1853	// the journal or the app log. One connection is accepted; a second is
1854	// closed unread.
1855	type cmdStream struct {
1856	ln net.Listener
1857	w io.Writer
1858	accepted atomic.Bool
1859	closed atomic.Bool
1860	mu sync.Mutex
1861	conn net.Conn
1862	copied chan struct{}
1863	wg sync.WaitGroup
1864	}
1865
1866	func listenCmdStream(path string, w io.Writer) (*cmdStream, error) {
1867	ln, err := edge.SecureUnixListen(path)
1868	if err != nil {
1869	return nil, err
1870	}
1871	s := &cmdStream{ln: ln, w: w, copied: make(chan struct{})}
1872	s.wg.Add(1)
1873	go s.accept()
1874	return s, nil
1875	}
1876
1877	func (s *cmdStream) accept() {
1878	defer s.wg.Done()
1879	for {
1880	c, err := s.ln.Accept()
1881	if err != nil {
1882	return
1883	}
1884	if s.accepted.Swap(true) {
1885	c.Close()
1886	continue
1887	}
1888	s.mu.Lock()
1889	if s.closed.Load() {
1890	s.mu.Unlock()
1891	c.Close()
1892	continue
1893	}
1894	s.conn = c
1895	s.wg.Add(1)
1896	s.mu.Unlock()
1897	go s.copy(c)
1898	}
1899	}
1900
1901	func (s *cmdStream) copy(c net.Conn) {
1902	defer s.wg.Done()
1903	defer close(s.copied)
1904	defer c.Close()
1905	_, _ = io.Copy(s.w, c)
1906	}
1907
1908	// Drain waits up to bound for the stream to reach EOF (its unit has
1909	// stopped, so every writer has closed), then closes the listener. A stream
1910	// that was never connected has nothing to drain.
1911	func (s *cmdStream) Drain(bound time.Duration) error {
1912	var err error
1913	if s.accepted.Load() {
1914	select {
1915	case <-s.copied:
1916	case <-time.After(bound):
1917	err = fmt.Errorf("a cmd stream did not reach its end within %s", bound)
1918	}
1919	}
1920	s.Close()
1921	return err
1922	}
1923
1924	func (s *cmdStream) Close() error {
1925	if s.closed.Swap(true) {
1926	return nil
1927	}
1928	err := s.ln.Close()
1929	s.mu.Lock()
1930	if s.conn != nil {
1931	s.conn.Close()
1932	}
1933	s.mu.Unlock()
1934	s.wg.Wait()
1935	return err
1936	}
1937	```
1938
1939	`backend_unit.go`, after `StartWorker`:
1940
1941	```go
1942	// StartCmd dispatches like every other call; a unit-backed row has no
1943	// nested instance to run a command in.
1944	func (b *SplitBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
1945	if c, ok := b.pick(rt).(CmdBackend); ok {
1946	return c.StartCmd(ctx, rt, spec)
1947	}
1948	return nil, fmt.Errorf("%w: %s is unit-backed", ErrCmdNotNested, rt.Host)
1949	}
1950	```
1951
1952	(Add `"fmt"` to `backend_unit.go`'s imports if it lacks it.)
1953
1954	- [ ] **Step 4: Run to verify they pass**
1955
1956	Run: `TMPDIR=/tmp/g nice -n 10 go test ./internal/activator/ -run 'TestCmdVars|TestStartCmd|TestStopAndSettle|TestStartUnit|TestSeed' -count=1 && GOOS=windows go build ./internal/activator/ && GOOS=darwin go build ./internal/activator/`
1957	Expected: PASS; both cross builds succeed (`edge.SecureUnixListen` has a windows body).
1958
1959	- [ ] **Step 5: Commit**
1960
1961	```bash
1962	git add internal/activator/cmdunit.go internal/activator/cmdunit_test.go internal/activator/control.go internal/activator/prepare.go internal/activator/backend_exec.go internal/activator/backend_unit.go internal/activator/unitproc_test.go
1963	git commit -m "activator: StartCmd, cmdVars and the two cmd streams (run spec §4, §5.4, §5.5)
1964
1965	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
1966	```
1967
1968	---
1969
1970	### Task 7: Every stop says why
1971
1972	A run interrupted by an instance stop ends `interrupted: "<reason>"` (spec §5.3). The reason is decided where the stop is decided, so `beginStopLocked` takes it.
1973
1974	**Files:**
1975	- Create: `internal/activator/cmd.go` (the reasons; Task 8 adds the rest)
1976	- Modify: `internal/activator/activator.go:346-457` (`actEntry.stopWhy`), `:1373` (`beginStopLocked`), and its 17 callers; `internal/activator/maintenance.go:133`
1977	- Test: `internal/activator/cmd_test.go` (new)
1978
1979	**Interfaces:**
1980	- Produces: `StopIdle`, `StopRestart`, `StopRelease`, `StopExited`, `StopLeaseLost`, `StopFenced`, `StopRetired`, `StopShutdown` (string constants, RUN-C1 to RUN-C8); `func (a *Service) beginStopLocked(e *actEntry, why string)`; `actEntry.stopWhy string`, written under `a.mu`.
1981
1982	- [ ] **Step 1: Write the failing test**
1983
1984	`internal/activator/cmd_test.go`:
1985
1986	```go
1987	package activator
1988
1989	import (
1990	"context"
1991	"testing"
1992	"time"
1993	)
1994
1995	func (a *Service) stopWhyOf(host string) string {
1996	a.mu.Lock()
1997	defer a.mu.Unlock()
1998	if e := a.entries[host]; e != nil {
1999	return e.stopWhy
2000	}
2001	return ""
2002	}
2003
2004	func TestStopsRecordTheirReason(t *testing.T) {
2005	a, b, _, _ := testActivator(t, time.Second)
2006	rt := testRoute("why.example.com")
2007	arrangeAwake(t, a, rt)
2008	gate := make(chan struct{})
2009	b.lastProc().setStopGate(gate) // hold the entry in stopping
2010	defer close(gate)
2011	a.Bounce(rt.Host)
2012	if got := a.stopWhyOf(rt.Host); got != StopRestart {
2013	t.Fatalf("Bounce: stopWhy %q, want %q", got, StopRestart)
2014	}
2015	}
2016	```
2017
2018	(`setStopGate(gate chan struct{})` is at `activator_test.go:143`; drop the `context` import this file does not need yet.)
2019
2020	- [ ] **Step 2: Run to verify it fails**
2021
2022	Run: `nice -n 10 go test ./internal/activator/ -run TestStopsRecordTheirReason -count=1`
2023	Expected: FAIL, `undefined: StopRestart`.
2024
2025	- [ ] **Step 3: Implement**
2026
2027	`internal/activator/cmd.go`:
2028
2029	```go
2030	package activator
2031
2032	// Why an instance stop interrupted a run (run spec §5.3). The text is what
2033	// the member reads after "the run was interrupted:" (RUN-C1 to RUN-C8);
2034	// beginStopLocked records one on every stop.
2035	const (
2036	StopIdle = "the instance hibernated"
2037	StopRestart = "the instance was restarted"
2038	StopRelease = "the instance moved to a new release"
2039	StopExited = "the instance's process exited"
2040	StopLeaseLost = "the instance lost its database lease"
2041	StopFenced = "the instance was stopped for a move or a layout change"
2042	StopRetired = "the instance was taken off this box"
2043	StopShutdown = "the box's agent stopped"
2044	)
2045	```
2046
2047	`actEntry` gains, after `stopOutcome`:
2048
2049	```go
2050	// stopWhy is why this entry is stopping (beginStopLocked), written under
2051	// a.mu; finishStop hands it to a run it interrupts (run spec §5.3).
2052	stopWhy string
2053	```
2054
2055	`activator.go:1373`:
2056
2057	```go
2058	func (a *Service) beginStopLocked(e *actEntry, why string) {
2059	e.state = actStopping
2060	e.stopWhy = why
2061	e.stopped = make(chan struct{})
2062	go a.stop(e)
2063	}
2064	```
2065
2066	Every caller passes its reason (`grep -n 'beginStopLocked(' internal/activator/*.go`; the compiler refuses any left out):
2067
2068	| site | function | reason |
2069	|---|---|---|
2070	| `activator.go:765` | `begin` | `StopLeaseLost` when `repErr != nil`, else `StopExited` (compute `why` inside the `if` beside the log line) |
2071	| `activator.go:850` | `BeginSidecarRequest` | as `:765` |
2072	| `activator.go:997` | `wake`, fence installed while waking | `StopFenced` |
2073	| `activator.go:1453` | `bounceLocked` (Bounce, BounceAdopted, the deferred bounce in `wake`) | `StopRestart` |
2074	| `activator.go:1566` | `ConvergeAdopted` | `StopRelease` |
2075	| `activator.go:1721` | `WakeWorker` | as `:765` |
2076	| `activator.go:1957` | `respawnWorkerLocked`, refused | `StopRestart` |
2077	| `activator.go:2072` | `SleepWorker` | `StopIdle` |
2078	| `activator.go:2465` | `sweepOnce`, lease lost | `StopLeaseLost` |
2079	| `activator.go:2507` | `sweepOnce`, process exited | `StopExited` |
2080	| `activator.go:2547` | `sweepOnce`, idle | `StopIdle` |
2081	| `activator.go:2574` | `sweepOnce`, worker `max_run` | `StopIdle` |
2082	| `activator.go:2611` | `sweepOnce`, convergence | `StopRelease` |
2083	| `activator.go:2841`, `:2861` | `RetireExcept` | `StopRetired` |
2084	| `activator.go:2925` | `Shutdown` | `StopShutdown` |
2085	| `maintenance.go:133` | `fenceAndStop` | `StopFenced` |
2086
2087	The `:765` shape, for example:
2088
2089	```go
2090	if repErr := e.rep.Err(); repErr != nil || e.proc.Exited() {
2091	why := StopExited
2092	if repErr != nil {
2093	why = StopLeaseLost
2094	log.Printf("activator: LEASE LOST for %s (%v) — standing down", e.host, repErr)
2095	} else {
2096	log.Printf("activator: instance %s exited unexpectedly — recycling", e.host)
2097	}
2098	a.beginStopLocked(e, why)
2099	```
2100
2101	(The log lines are main's bytes, journal-only, and stay as they are.) Test files that construct entries or call `beginStopLocked` directly get a reason too: `grep -n 'beginStopLocked(' internal/activator/*_test.go`.
2102
2103	`runCopyBuilt` gains `"RUN-C1", "RUN-C2", "RUN-C3", "RUN-C4", "RUN-C5", "RUN-C6", "RUN-C7", "RUN-C8"`.
2104
2105	- [ ] **Step 4: Run to verify it passes**
2106
2107	Run: `nice -n 10 go test ./internal/activator/ -count=1 && nice -n 10 go test . -run TestRunBuilt -count=1`
2108	Expected: PASS (the whole package: every stop path was edited).
2109
2110	- [ ] **Step 5: Commit**
2111
2112	```bash
2113	git add internal/activator/cmd.go internal/activator/cmd_test.go internal/activator/activator.go internal/activator/maintenance.go internal/activator/*_test.go approved_run_copy_test.go
2114	git commit -m "activator: every stop records why (run spec §5.3)
2115
2116	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
2117	```
2118
2119	---
2120
2121	### Task 8: The run joins the writer lifecycle: `cmd`, `RunCmd`, `finishStop`, the idle hold
2122
2123	**Files:**
2124	- Modify: `internal/activator/cmd.go` (append), `internal/activator/activator.go` (`actEntry.cmd`, `finishStop` at `:2355`, the idle check at `:2533`)
2125	- Test: `internal/activator/cmd_test.go` (append)
2126
2127	**Interfaces:**
2128	- Consumes: `a.begin(ctx, rt, stamp bool) (func(), error)` (`:725`), `errWakeTimeout`, `memberSafeWakeReason(err) string` (`logsink.go:49`), `a.cfg.HostWrites.Enabled`, `Classifier`, `CmdBackend`, `CmdSpec`, `CmdUnit`, `StopIdle`.
2129	- Produces:
2130	- `type CmdHandle interface { Invocation() string; Poll(ctx) (instancectl.CmdStatus, error); Interrupted() string; Stop(ctx) error; Release() }`.
2131	- `func (a *Service) RunCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdHandle, error)`.
2132	- Errors: `ErrCmdClaim`, `ErrCmdBusy`, `ErrCmdNotNested` (Task 6), `*CmdWakeError{Reason string; Err error}` (Reason is member-safe), `*CmdStoppedError{Reason string}`, and whatever `spec.Prepare` returned, unwrapped.
2133	- `cmdWakeBound = 60 * time.Second` (a var, for tests).
2134	- `actEntry.cmd *cmdHold`; while it is set the idle sweep never stops the entry; every stop path stops and settles its unit before `rep.Close`, and an unsettled unit marks the stop uncertain.
2135
2136	- [ ] **Step 1: Write the failing tests**
2137
2138	Append to `cmd_test.go` (its imports grow to `context`, `errors`, `path/filepath`, `sync`, `testing`, `time`, and the repo's `hostwrite`, `instancectl` and `registry`):
2139
2140	```go
2141	// fakeCmdUnit records the stop and whether replication was already closed
2142	// when it came.
2143	type fakeCmdUnit struct {
2144	mu sync.Mutex
2145	stops int
2146	repWasClosed bool
2147	settleErr error
2148	rep func() *fakeRep
2149	status instancectl.CmdStatus
2150	}
2151
2152	func (u *fakeCmdUnit) Invocation() string { return "inv1" }
2153	func (u *fakeCmdUnit) Poll(context.Context) (instancectl.CmdStatus, error) {
2154	u.mu.Lock()
2155	defer u.mu.Unlock()
2156	return u.status, nil
2157	}
2158	func (u *fakeCmdUnit) StopAndSettle(context.Context) error {
2159	u.mu.Lock()
2160	defer u.mu.Unlock()
2161	u.stops++
2162	if u.rep != nil {
2163	u.repWasClosed = u.rep().isClosed()
2164	}
2165	return u.settleErr
2166	}
2167	func (u *fakeCmdUnit) stopped() (int, bool) {
2168	u.mu.Lock()
2169	defer u.mu.Unlock()
2170	return u.stops, u.repWasClosed
2171	}
2172
2173	// cmdFakeBackend is fakeBackend with the run capability and a nested layout.
2174	type cmdFakeBackend struct {
2175	*fakeBackend
2176	cmdMu sync.Mutex
2177	unit *fakeCmdUnit
2178	startErr error
2179	cmdStarts int
2180	gate chan struct{} // when set, StartCmd blocks until it closes
2181	}
2182
2183	func (b *cmdFakeBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
2184	if b.gate != nil {
2185	<-b.gate
2186	}
2187	b.cmdMu.Lock()
2188	defer b.cmdMu.Unlock()
2189	b.cmdStarts++
2190	if b.startErr != nil {
2191	return nil, b.startErr
2192	}
2193	return b.unit, nil
2194	}
2195
2196	func testCmdActivator(t *testing.T, cfg Config) (*Service, *cmdFakeBackend, *fakeClock) {
2197	t.Helper()
2198	fb := &fakeBackend{layoutFn: func(registry.Route) (Layout, string) { return LayoutNested, "" }}
2199	cb := &cmdFakeBackend{fakeBackend: fb}
2200	cb.unit = &fakeCmdUnit{rep: fb.lastRep}
2201	clk := &fakeClock{t: time.Unix(1_700_000_000, 0)}
2202	reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
2203	if err != nil {
2204	t.Fatal(err)
2205	}
2206	if cfg.Bucket == "" {
2207	cfg.Bucket = "test-bucket"
2208	}
2209	if cfg.Idle == 0 {
2210	cfg.Idle = time.Second
2211	}
2212	a := New(cfg, cb, reg)
2213	a.now = clk.now
2214	return a, cb, clk
2215	}
2216
2217	func TestRunCmdRefusesWithClaimControlArmed(t *testing.T) {
2218	a, cb, _ := testCmdActivator(t, Config{HostWrites: hostwrite.Config{Enabled: true}})
2219	if _, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdClaim) {
2220	t.Fatalf("err = %v, want ErrCmdClaim", err)
2221	}
2222	if acquires, _, _ := cb.counts(); acquires != 0 {
2223	t.Fatal("a claim-armed refusal woke the instance")
2224	}
2225	}
2226
2227	func TestRunCmdRefusesAFlatInstance(t *testing.T) {
2228	a, cb, _ := testCmdActivator(t, Config{})
2229	cb.layoutFn = func(registry.Route) (Layout, string) { return LayoutFlat, "" }
2230	if _, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdNotNested) {
2231	t.Fatalf("err = %v, want ErrCmdNotNested", err)
2232	}
2233	}
2234
2235	func TestRunCmdWakesAHibernatedInstanceFirst(t *testing.T) {
2236	a, cb, _ := testCmdActivator(t, Config{})
2237	h, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"})
2238	if err != nil {
2239	t.Fatal(err)
2240	}
2241	if _, _, wakes := cb.counts(); h.Invocation() != "inv1" || wakes != 1 || cb.cmdStarts != 1 {
2242	t.Fatalf("wake starts %d, cmd starts %d", wakes, cb.cmdStarts)
2243	}
2244	h.Release()
2245	}
2246
2247	func TestRunCmdWakeFailureIsAWakeRefusal(t *testing.T) {
2248	a, cb, _ := testCmdActivator(t, Config{})
2249	cb.acquireErr = errors.New("restore failed")
2250	_, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"})
2251	var we *CmdWakeError
2252	if !errors.As(err, &we) || we.Reason == "" {
2253	t.Fatalf("err = %v, want a CmdWakeError with a member-safe reason", err)
2254	}
2255	if cb.cmdStarts != 0 {
2256	t.Fatal("a failed wake started the command")
2257	}
2258	}
2259
2260	func TestRunCmdRefusesASecondRun(t *testing.T) {
2261	a, _, _ := testCmdActivator(t, Config{})
2262	rt := testRoute("c.example.com")
2263	h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
2264	if err != nil {
2265	t.Fatal(err)
2266	}
2267	defer h.Release()
2268	if _, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_b"}); !errors.Is(err, ErrCmdBusy) {
2269	t.Fatalf("err = %v, want ErrCmdBusy", err)
2270	}
2271	}
2272
2273	func TestRunCmdHoldsTheIdleTimer(t *testing.T) {
2274	a, _, clk := testCmdActivator(t, Config{})
2275	rt := testRoute("c.example.com")
2276	h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
2277	if err != nil {
2278	t.Fatal(err)
2279	}
2280	clk.advance(time.Hour)
2281	a.sweepOnce()
2282	if a.entryCount() != 1 || a.stopWhyOf(rt.Host) != "" {
2283	t.Fatal("the idle timer stopped an instance with a run open")
2284	}
2285	h.Release()
2286	clk.advance(time.Hour)
2287	a.sweepOnce()
2288	waitFor(t, "hibernation after the run", func() bool { return a.entryCount() == 0 })
2289	}
2290
2291	// Every stop path stops and settles the run before replication closes, and
2292	// the run learns why (run spec §5.3, codex 1).
2293	func TestEveryStopPathSettlesTheRunBeforeReplicationCloses(t *testing.T) {
2294	cases := []struct {
2295	name string
2296	why string
2297	stop func(a *Service, cb *cmdFakeBackend, host string)
2298	}{
2299	{"bounce", StopRestart, func(a *Service, _ *cmdFakeBackend, h string) { a.Bounce(h) }},
2300	{"process exit", StopExited, func(a *Service, cb *cmdFakeBackend, _ string) { cb.lastProc().exit(); a.sweepOnce() }},
2301	{"lease lost", StopLeaseLost, func(a *Service, cb *cmdFakeBackend, _ string) { cb.lastRep().setErr(errors.New("lost")); a.sweepOnce() }},
2302	{"convergence", StopRelease, func(a *Service, _ *cmdFakeBackend, h string) {
2303	a.liveVer = func(string) (string, error) { return "v2", nil }
2304	a.ConvergeAdopted(h, "v2")
2305	}},
2306	{"move fence", StopFenced, func(a *Service, _ *cmdFakeBackend, h string) {
2307	go a.FenceAndStop(context.Background(), h, 1) //nolint:errcheck
2308	}},
2309	{"retire", StopRetired, func(a *Service, _ *cmdFakeBackend, _ string) { a.RetireExcept(map[string]bool{}) }},
2310	{"agent shutdown", StopShutdown, func(a *Service, _ *cmdFakeBackend, _ string) {
2311	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
2312	defer cancel()
2313	a.Shutdown(ctx)
2314	}},
2315	}
2316	for _, c := range cases {
2317	t.Run(c.name, func(t *testing.T) {
2318	a, cb, _ := testCmdActivator(t, Config{})
2319	cb.execVerFn = func() string { return "v1" }
2320	rt := testRoute("c.example.com")
2321	h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
2322	if err != nil {
2323	t.Fatal(err)
2324	}
2325	c.stop(a, cb, rt.Host)
2326	waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
2327	stops, repWasClosed := cb.unit.stopped()
2328	if stops != 1 {
2329	t.Fatalf("the cmd unit was stopped %d times, want 1", stops)
2330	}
2331	if repWasClosed {
2332	t.Fatal("replication closed before the run settled")
2333	}
2334	if !cb.lastRep().isClosed() {
2335	t.Fatal("replication never closed")
2336	}
2337	if got := h.Interrupted(); got != c.why {
2338	t.Fatalf("Interrupted() = %q, want %q", got, c.why)
2339	}
2340	})
2341	}
2342	}
2343
2344	func TestRunCmdNeverStartsUnderAStoppingEntry(t *testing.T) {
2345	a, cb, _ := testCmdActivator(t, Config{})
2346	rt := testRoute("c.example.com")
2347	_, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a", Prepare: func(context.Context) error {
2348	a.Bounce(rt.Host) // the instance starts stopping while the files download
2349	return nil
2350	}})
2351	var se *CmdStoppedError
2352	if !errors.As(err, &se) || se.Reason != StopRestart {
2353	t.Fatalf("err = %v, want CmdStoppedError{%q}", err, StopRestart)
2354	}
2355	if cb.cmdStarts != 0 {
2356	t.Fatal("the cmd unit started under a stopping entry")
2357	}
2358	}
2359
2360	func TestAStopDuringTheStartWaitsForIt(t *testing.T) {
2361	a, cb, _ := testCmdActivator(t, Config{})
2362	cb.gate = make(chan struct{})
2363	rt := testRoute("c.example.com")
2364	done := make(chan error, 1)
2365	go func() {
2366	_, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
2367	done <- err
2368	}()
2369	waitFor(t, "the run attached", func() bool {
2370	a.mu.Lock()
2371	defer a.mu.Unlock()
2372	e := a.entries[rt.Host]
2373	return e != nil && e.cmd != nil
2374	})
2375	a.Bounce(rt.Host)
2376	time.Sleep(50 * time.Millisecond)
2377	if cb.lastRep().isClosed() {
2378	t.Fatal("replication closed while the cmd unit's start was still in flight")
2379	}
2380	close(cb.gate)
2381	if err := <-done; err != nil {
2382	t.Fatalf("RunCmd: %v", err)
2383	}
2384	waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
2385	if stops, repWasClosed := cb.unit.stopped(); stops != 1 || repWasClosed {
2386	t.Fatalf("stops %d, replication closed first %v", stops, repWasClosed)
2387	}
2388	}
2389
2390	func TestAnUnsettledRunMakesTheStopUncertain(t *testing.T) {
2391	for _, c := range []struct {
2392	settleErr error
2393	uncertain bool
2394	}{{nil, false}, {errors.New("processes remained"), true}} {
2395	a, cb, _ := testCmdActivator(t, Config{})
2396	cb.unit.settleErr = c.settleErr
2397	rt := testRoute("c.example.com")
2398	if _, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"}); err != nil {
2399	t.Fatal(err)
2400	}
2401	a.Bounce(rt.Host)
2402	waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
2403	if got := a.StopUncertain(rt.Host); got != c.uncertain {
2404	t.Fatalf("settle error %v: StopUncertain = %v, want %v", c.settleErr, got, c.uncertain)
2405	}
2406	}
2407	}
2408	```
2409
2410	`counts()` answers `(acquires, restores, starts int)` (`activator_test.go:563`); `StopUncertain(host string) bool` is `maintenance.go:225`, and the fakes' ordinary stop is a complete one (`fakeProc.Stop` sets `dead`), which is why the settled case must read `false`.
2411
2412	- [ ] **Step 2: Run to verify they fail**
2413
2414	Run: `nice -n 10 go test ./internal/activator/ -run 'TestRunCmd|TestEveryStopPath|TestAStopDuringTheStart|TestAnUnsettledRun' -count=1`
2415	Expected: FAIL, `a.RunCmd undefined`.
2416
2417	- [ ] **Step 3: Implement**
2418
2419	`actEntry` gains, after `stopWhy`:
2420
2421	```go
2422	// cmd is the run attached to this instance (run spec §5.3), written
2423	// under a.mu. While it is set the idle sweep never stops the entry;
2424	// every other stop stops and settles its unit in finishStop, before
2425	// replication closes.
2426	cmd *cmdHold
2427	```
2428
2429	Append to `cmd.go` (imports: `context`, `errors`, `sync`, `time`, `instancectl`, `registry`):
2430
2431	```go
2432	// ErrCmdClaim: claim control is armed on this box. Runs need the isolated
2433	// root protocol's reservations (#250), so they are refused (run spec §3.1).
2434	var ErrCmdClaim = errors.New("activator: claim control is armed; runs wait for the reservation protocol (#250)")
2435
2436	// ErrCmdBusy: a run is already attached to the instance.
2437	var ErrCmdBusy = errors.New("activator: a run is already attached to this instance")
2438
2439	// CmdWakeError: the wake failed or did not finish within cmdWakeBound.
2440	// Reason is member-safe (memberSafeWakeReason).
2441	type CmdWakeError struct {
2442	Reason string
2443	Err error
2444	}
2445
2446	func (e *CmdWakeError) Error() string { return e.Reason }
2447	func (e *CmdWakeError) Unwrap() error { return e.Err }
2448
2449	// CmdStoppedError: the instance began stopping before the unit started. The
2450	// run is interrupted with Reason, never queued for later (run spec §5.3).
2451	type CmdStoppedError struct{ Reason string }
2452
2453	func (e *CmdStoppedError) Error() string {
2454	return "activator: the instance stopped before the run started: " + e.Reason
2455	}
2456
2457	// cmdWakeBound is how long RunCmd waits for a wake (run spec §5.3).
2458	var cmdWakeBound = 60 * time.Second
2459
2460	// cmdStopBound bounds finishStop's wait for a run: the start in flight
2461	// (DeadlineControl), the stop and the cgroup (actStopTimeout) and both
2462	// streams (cmdDrainBound), with room.
2463	const cmdStopBound = 2 * time.Minute
2464
2465	// CmdHandle is a started run, as the box's run pass sees it.
2466	type CmdHandle interface {
2467	Invocation() string
2468	Poll(ctx context.Context) (instancectl.CmdStatus, error)
2469	// Interrupted is the instance stop's reason when one stopped the unit,
2470	// "" otherwise.
2471	Interrupted() string
2472	// Stop stops the unit and waits for it to settle and its streams to
2473	// end, leaving the instance running: a cancel, a timeout, or the end
2474	// of a command that exited. Safe to call more than once.
2475	Stop(ctx context.Context) error
2476	// Release clears the run from the entry once its result is written.
2477	Release()
2478	}
2479
2480	// cmdHold is the entry's cmd field.
2481	type cmdHold struct {
2482	runID string
2483	started chan struct{} // closed once StartCmd returned, either way
2484	unit CmdUnit // written before started is closed; nil on a failed start
2485	mu sync.Mutex
2486	reason string
2487	stopOnce sync.Once
2488	settled chan struct{}
2489	stopErr error
2490	}
2491
2492	func newCmdHold(runID string) *cmdHold {
2493	return &cmdHold{runID: runID, started: make(chan struct{}), settled: make(chan struct{})}
2494	}
2495
2496	// stop waits for the start to return, records reason (the first one
2497	// wins; "" records none), and stops and settles the unit once.
2498	func (h *cmdHold) stop(ctx context.Context, reason string) error {
2499	select {
2500	case <-h.started:
2501	case <-ctx.Done():
2502	return ctx.Err()
2503	}
2504	h.mu.Lock()
2505	if reason != "" && h.reason == "" {
2506	h.reason = reason
2507	}
2508	h.mu.Unlock()
2509	if h.unit == nil {
2510	return nil
2511	}
2512	h.stopOnce.Do(func() {
2513	go func() {
2514	h.stopErr = h.unit.StopAndSettle(context.Background())
2515	close(h.settled)
2516	}()
2517	})
2518	select {
2519	case <-h.settled:
2520	return h.stopErr
2521	case <-ctx.Done():
2522	return ctx.Err()
2523	}
2524	}
2525
2526	type cmdRun struct {
2527	a *Service
2528	e *actEntry
2529	h *cmdHold
2530	once sync.Once
2531	}
2532
2533	func (r *cmdRun) Invocation() string { return r.h.unit.Invocation() }
2534	func (r *cmdRun) Poll(ctx context.Context) (instancectl.CmdStatus, error) {
2535	return r.h.unit.Poll(ctx)
2536	}
2537	func (r *cmdRun) Interrupted() string {
2538	r.h.mu.Lock()
2539	defer r.h.mu.Unlock()
2540	return r.h.reason
2541	}
2542	func (r *cmdRun) Stop(ctx context.Context) error { return r.h.stop(ctx, "") }
2543
2544	// Release clears the hold and stamps activity, so the instance idles a
2545	// full window after a run rather than hibernating at the next sweep.
2546	func (r *cmdRun) Release() {
2547	r.once.Do(func() {
2548	r.a.mu.Lock()
2549	if r.e.cmd == r.h {
2550	r.e.cmd = nil
2551	r.e.lastActive = r.a.now()
2552	}
2553	r.a.mu.Unlock()
2554	})
2555	}
2556
2557	// RunCmd runs one command against rt's instance (run spec §5.2 and §5.3):
2558	// refused with claim control armed or on a layout other than nested; the
2559	// instance woken through the ordinary wake path, waiting up to 60 s; the
2560	// run attached to the awake entry; spec.Prepare run under the hold; and
2561	// the unit started only while the entry is still awake. A stop that begins
2562	// after the attach is either seen here (CmdStoppedError, no unit) or
2563	// waits in finishStop for the start to return and then stops the unit
2564	// before replication closes.
2565	func (a *Service) RunCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdHandle, error) {
2566	if a.cfg.HostWrites.Enabled {
2567	return nil, ErrCmdClaim
2568	}
2569	cb, ok := a.backend.(CmdBackend)
2570	if !ok {
2571	return nil, ErrCmdNotNested
2572	}
2573	if c, ok := a.backend.(Classifier); !ok {
2574	return nil, ErrCmdNotNested
2575	} else if l, _ := c.Layout(rt); l != LayoutNested {
2576	return nil, ErrCmdNotNested
2577	}
2578	wctx, cancel := context.WithTimeout(ctx, cmdWakeBound)
2579	defer cancel()
2580	var release func()
2581	for {
2582	r, err := a.begin(wctx, rt, true)
2583	if err == nil {
2584	release = r
2585	break
2586	}
2587	if errors.Is(err, errWakeTimeout) && wctx.Err() == nil {
2588	continue // begin's own bound is 30 s; a run waits 60
2589	}
2590	if wctx.Err() != nil && ctx.Err() == nil {
2591	err = errWakeTimeout
2592	}
2593	return nil, &CmdWakeError{Reason: memberSafeWakeReason(err), Err: err}
2594	}
2595	h := newCmdHold(spec.RunID)
2596	a.mu.Lock()
2597	e := a.entries[rt.Host]
2598	switch {
2599	case e == nil || e.state != actAwake:
2600	why := StopIdle
2601	if e != nil && e.stopWhy != "" {
2602	why = e.stopWhy
2603	}
2604	a.mu.Unlock()
2605	release()
2606	return nil, &CmdStoppedError{Reason: why}
2607	case e.cmd != nil:
2608	a.mu.Unlock()
2609	release()
2610	return nil, ErrCmdBusy
2611	}
2612	e.cmd = h
2613	a.mu.Unlock()
2614	release() // the hold, not the in-flight count, keeps the idle timer off now
2615	run := &cmdRun{a: a, e: e, h: h}
2616	abandon := func(err error) (CmdHandle, error) {
2617	close(h.started)
2618	run.Release()
2619	return nil, err
2620	}
2621	if spec.Prepare != nil {
2622	if err := spec.Prepare(ctx); err != nil {
2623	return abandon(err)
2624	}
2625	}
2626	a.mu.Lock()
2627	if e.state != actAwake {
2628	why := e.stopWhy
2629	a.mu.Unlock()
2630	return abandon(&CmdStoppedError{Reason: why})
2631	}
2632	a.mu.Unlock()
2633	u, err := cb.StartCmd(ctx, rt, spec)
2634	h.unit = u
2635	close(h.started)
2636	if err != nil {
2637	run.Release()
2638	return nil, err
2639	}
2640	return run, nil
2641	}
2642	```
2643
2644	`finishStop` (`activator.go:2355`), its first lines:
2645
2646	```go
2647	func (a *Service) finishStop(e *actEntry, outcome *stopObservation) {
2648	// A run attached to this instance is stopped and settled before
2649	// replication closes (run spec §5.3, codex 1): everything it committed
2650	// is in the final checkpoint, and nothing writes the database after.
2651	// A run that will not settle leaves the stop unproven.
2652	a.mu.Lock()
2653	h, why := e.cmd, e.stopWhy
2654	a.mu.Unlock()
2655	if h != nil {
2656	cctx, ccancel := context.WithTimeout(context.Background(), cmdStopBound)
2657	if err := h.stop(cctx, why); err != nil {
2658	log.Printf("activator: %s: run %s did not settle before the final checkpoint: %v", e.host, h.runID, err)
2659	outcome.uncertain = true
2660	}
2661	ccancel()
2662	}
2663	// Close = final sync + lease release. ...
2664	```
2665
2666	(The rest of `finishStop` is unchanged; its later `outcome.uncertain = !outcome.complete()` keeps `true`, because `complete()` requires `!o.uncertain`.)
2667
2668	The idle check at `activator.go:2533`:
2669
2670	```go
2671	if hibernates && e.inflight == 0 && e.cmd == nil && now.Sub(e.lastActive) >= idleWindow {
2672	```
2673
2674	- [ ] **Step 4: Run to verify they pass**
2675
2676	Run: `nice -n 10 go test ./internal/activator/ -count=1`
2677	Expected: PASS, the whole package.
2678
2679	- [ ] **Step 5: Commit**
2680
2681	```bash
2682	git add internal/activator/cmd.go internal/activator/cmd_test.go internal/activator/activator.go
2683	git commit -m "activator: RunCmd, the cmd hold and the stop ordering in finishStop (run spec §5.3, codex 1, 2)
2684
2685	Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
2686	```
2687
2688	---
2689
2690	### Task 9: The plan's gate
2691
2692	**Files:** none new.
2693
2694	- [ ] **Step 1: Run the gate**
2695
2696	```bash
2697	export GOFLAGS=-buildvcs=false
2698	go build ./... && go vet ./... && test -z "$(gofmt -l .)"
2699	TMPDIR=/tmp/g nice -n 10 make test
2700	GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...
2701	TMPDIR=/tmp/g nice -n 10 go test -race ./internal/activator/ ./internal/instancectl/ ./internal/instancefs/ -count=1
2702	```
2703
2704	Expected: every command exits 0. If the sandbox's Go cache is read-only, prefix each `go` command with `GOCACHE=$TMPDIR/gocache`.
2705
2706	- [ ] **Step 2: Post the #250 note**
2707
2708	Post on amadan carlos/platform #250 (as this session's agent account, per the `amadan` skill): the claim activation must add `cmd` to the reservation protocol beside `main`, `seed` and `sidecar`, settle the cmd unit's producer before a generation is published, and then lift `activator.ErrCmdClaim` and the console's RUN-C42 refusal (Plan 2). Link this plan.
2709
2710	- [ ] **Step 3: Commit anything the gate changed**
2711
2712	```bash
2713	git status --short
2714	```
2715
2716	Expected: clean. If `gofmt` rewrote a file, commit it with `run: gofmt` and the trailer.
2717
2718	---
2719
2720	## Self-review notes
2721
2722	- Spec coverage: §3.1 (Task 8 `ErrCmdClaim`, Task 4 the 503s); §4 (Task 6 `cmdVars`, the launch file, the working directory from the template); §5.3 (Tasks 7 and 8, every stop path in the spec's list: hibernation is held, `Bounce`, convergence, a failed process, the move freeze, agent shutdown, plus lease loss and retirement); §5.4 (Tasks 2 to 4, 6); §9 "Writer lifecycle" and "Unit and control" (Tasks 3, 4, 8). §5.2 step 3's directory and §5.8's box sweep use Task 5's `RemoveAllBeneath` in Plan 3.
2723	- The spec's "a write the command commits just before the stop reaching the replica" is shown here by ordering (`repWasClosed`); the real write is a dev-box check (Plan 4 Task 6).
2724	- `ExecMainCode` arrives as a number from real systemd; Task 4 passes it through, Plan 3 maps it.
2725