carlos run, Plan 1: Box Foundation Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give the box everything one carlos run command needs to run as its instance: an approved copy deck, a fourth instance role and unit, a control-service door that starts it and a long-poll that follows it, a launch file with the run's environment, two output streams, and a place on the activator entry that every stop path settles before replication closes.

Architecture: instancectl gains RoleCmd, the carlos-instance-cmd@.service template (the seed template with six lines changed), POST /cmd (start, answer at once) and GET /cmd/{host} (a 30-second long-poll). activator.ExecBackend.StartCmd prepares the instance's directories and env, binds two raw stream listeners, writes launch.cmd.json with cmdVars, and starts the unit. activator.Service.RunCmd wakes the instance (up to 60 s), attaches a cmdHold to the entry, and starts the unit only while the entry is awake. finishStop, the one place every stop path ends, stops and settles the hold's unit before rep.Close; the idle sweep skips an entry with a hold. Nothing in this plan picks a run up: Plan 3 calls RunCmd.

Tech Stack: Go 1.25, systemd via systemctl (instancectl.Server.Systemctl), golang.org/x/sys/unix (openat2, unlinkat), the repo's instancectl, instancefs, activator, provision packages.

Spec: docs/superpowers/specs/2026-10-08-carlos-run-design.md rev 4: §3.1, §4, §5.3, §5.4, §5.2 step 3 (the cleanup primitive), §9 "Writer lifecycle" and "Unit and control"; dispositions codex 1, 2, 4, 6, 7. Index: 2026-10-08-carlos-run-0-index.md. Copy: 2026-10-08-carlos-run-copy.md.

Build target: main at 4a51a6da or later. Every name below was read there.

Global Constraints

• Every user-facing string (CLI output, errors, console messages, docs prose) is copy-reviewed by Paul before it is written into code. The deck is docs/superpowers/plans/2026-10-08-carlos-run-copy.md and its gate is approved_run_copy_test.go (Task 1). Implementation tasks use only strings from the deck; a task that writes one adds its id to runCopyBuilt in the same commit. Copy voice: no em dashes; full stops and new sentences instead.
• A deck string appears in Go as one string literal, or one fmt.Sprintf/fmt.Errorf/errors.New format, equal to the row with %s/%d/%q where the row has a <placeholder>, and with no trailing newline inside the literal.
• CI cross-compiles darwin and windows: linux-only syscalls/units code needs //go:build linux plus a !linux stub.
• Local test commands: run targeted tests (go test ./internal/<pkg>/ -run TestX), not the full suite in a loop. Inside the sandbox the default Go build cache may be read-only; if so use GOCACHE=$TMPDIR/gocache re-exported per command. Socket tests need a short TMPDIR (TMPDIR=/tmp/g). The local gate may need GOFLAGS=-buildvcs=false.
• Heavy loops run under nice; no -race full suite until the end of a plan.
• Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.
• The cmd template is the seed template outside exactly these lines (spec §5.4): Description=carlos exec-backed instance %I (cmd), ExecStart=/opt/carlos/carlos instance-exec cmd %I, ExecStopPost=/opt/carlos/carlos instance-exit cmd %I, TimeoutStartSec=6h5m, StandardOutput=file:/run/carlos/logs/%I.cmd.sock, StandardError=file:/run/carlos/logs/%I.cmd-err.sock. Type=oneshot and RemainAfterExit=yes stay.
• GET /cmd/{host} waits at most 30 seconds. POST /cmd never waits for the command.
• The box waits up to 60 seconds for a wake before a run (spec §5.3). Signals: SIGTERM, then SIGKILL 20 seconds later (KillMode=mixed, TimeoutStopSec=20, already in the seed template).
• cmdVars (spec §4): the instance's file environment; CARLOS_CONTROL_SOCKET and CARLOS_PURGE removed; CARLOS_REGION set from ExecBackend.Region, or removed when it is empty; CARLOS_HOST; CARLOS_ADMIN_TOKEN kept; CARLOS_DB, CARLOS_RUN_ID, CARLOS_RUN_DIR added. The environment is never logged.
• Gate before the plan's last commit: go build ./..., go vet ./..., gofmt -l . clean, make test, and GOOS=darwin go build ./..., GOOS=windows go build ./..., GOOS=linux GOARCH=arm64 go build ./....

Review Focus

1. A stop that begins while RunCmd is between its attach and StartCmd (the files are still downloading) must never leave a cmd unit started after rep.Close: either the run is refused with the stop's reason before the unit starts, or finishStop waits for the start to return and stops the unit before it closes replication. Test in Task 8 (TestRunCmdNeverStartsUnderAStoppingEntry and TestAStopDuringTheStartWaitsForIt).
2. A cmd unit still loaded from an earlier run (active (exited) because the agent died before stopping it) must make POST /cmd refuse with 409, not silently no-op the start and hand back the old invocation. Test in Task 4 (TestCmdRefusesAUnitStillLoaded).
3. A cmd unit whose StopAndSettle cannot prove its cgroup empty must leave the stop uncertain (stopObservation.uncertain), so the maintenance fence never vouches for a stop with a writer possibly alive. Test in Task 8 (TestAnUnsettledRunMakesTheStopUncertain).
4. A tenant that plants a symlink at .carlos-run or inside a run directory must not make the cleanup delete anything outside the instance's tree. Test in Task 5 (TestRemoveAllBeneathNeverFollowsASymlink).
5. Output written by the command in its last moment (bytes still in the socket buffer when the main process exits) must reach the run's writer before StopAndSettle returns. Test in Task 6 (TestStopAndSettleDrainsBothStreams).

---

File structure

| path | responsibility | |---|---| | approved_run_copy_test.go | the run deck's gate: inventory, approval before use, applied text, no em dash | | approved_moves_copy_test.go | movesCopyIDPattern accepts RUN-C ids (the shared parser) | | testdata/run-copy/record.md, testdata/run-copy/applied.go | the gate's negative fixtures | | internal/instancectl/names.go | RoleCmd, UnitPrefixCmd, CmdStatus | | internal/instancectl/server.go | the two routes, the roles /stop and /reset accept, the sweep pattern, the cmd timing fields | | internal/instancectl/cmd.go | handleCmd, handleCmdStatus, cmdStatusOf | | internal/instancectl/client.go | Client.Cmd, Client.CmdStatus, DeadlineCmdPoll | | internal/instanceexec/exec.go, exit.go | usage and role lines name cmd | | internal/host/units/carlos-instance-cmd@.service | the fourth template | | internal/host/units_isolation_test.go, units_hardening_test.go, host_test.go | the template's tests and lists | | internal/instancefs/dirs_linux.go, instancefs_other.go | RemoveAllBeneath | | internal/activator/control.go | Control.Cmd, Control.CmdStatus | | internal/activator/prepare.go | env for the cmd role, no app-log listener for it, logSocketNamed | | internal/activator/cmdunit.go | CmdSpec, CmdUnit, CmdBackend, cmdVars, cmdStream, ExecBackend.StartCmd, cmdUnit | | internal/activator/backend_exec.go | waitRoleCgroupEmpty | | internal/activator/backend_unit.go | SplitBackend.StartCmd | | internal/activator/cmd.go | stop reasons, CmdHandle, cmdHold, Service.RunCmd, errors | | internal/activator/activator.go, maintenance.go | beginStopLocked(e, why), actEntry.cmd/stopWhy, finishStop ordering, the idle hold | | docs/site/cli.md | "four instance templates" (RUN-C11, RUN-C12) |

---

Task 1: The copy gate and Paul's review

This is the series' first task. Nothing else in the series writes a user-facing string until it is merged with every row approved.

Files:

• Create: approved_run_copy_test.go, testdata/run-copy/record.md, testdata/run-copy/applied.go
• Modify: approved_moves_copy_test.go:91 (movesCopyIDPattern) and the error text at :150
• Modify: docs/superpowers/plans/2026-10-08-carlos-run-copy.md (statuses and Paul's edits)

Interfaces:

• Consumes: parseMovesCopyRecord(path string) ([]copyRow, error), checkMovesInventory(rows []copyRow, want []string) []string, checkMovesApplied(rows []copyRow, root string) []string, checkMovesNoEmDash(rows []copyRow) []string, reportAll(t *testing.T, problems []string), copyRow{ID, File, Text, Status string} (all in approved_moves_copy_test.go, package main).
• Produces: runCopyIDs []string (RUN-C1 to RUN-C83), runCopyBuilt []string (empty here; every later task appends), checkRunBuilt(rows []copyRow, built []string) []string, builtRows(rows []copyRow, built []string) []copyRow.

• [ ] Step 1: Write the failing test

Create approved_run_copy_test.go:

// approved_run_copy_test.go
package main

import (
	"fmt"
	"os"
	"path/filepath"
	"strconv"
	"strings"
	"testing"
)

// The copy gate of the carlos run series (run Plan 1 Task 1). The deck is a
// markdown table in the moves record's format, read with the moves gate's
// parser and checked with its helpers, so "applied" means the same thing in
// both gates:
//
//   - TestRunCopyRecordIsComplete: the deck names exactly RUN-C1 to RUN-C83;
//   - TestRunBuiltCopyIsApproved: an id in runCopyBuilt is an approved row
//     (a string reaches code only after Paul has approved it);
//   - TestRunBuiltCopyIsApplied: a built row's text is in the file it names;
//   - TestRunCopyHasNoEmDash.
const runCopyRecordPath = "docs/superpowers/plans/2026-10-08-carlos-run-copy.md"

// runCopyIDs is the deck's inventory: RUN-C1 to RUN-C83, pinned.
var runCopyIDs = func() []string {
	ids := make([]string, 0, 83)
	for i := 1; i <= 83; i++ {
		ids = append(ids, "RUN-C"+strconv.Itoa(i))
	}
	return ids
}()

// runCopyBuilt is every deck id whose text the code or docs now carry. The
// task that writes a row's text adds its id here in the same commit.
var runCopyBuilt = []string{}

func mustParseRunRecord(t *testing.T) []copyRow {
	t.Helper()
	rows, err := parseMovesCopyRecord(runCopyRecordPath)
	if err != nil {
		t.Fatal(err)
	}
	if len(rows) == 0 {
		t.Fatalf("%s has no rows", runCopyRecordPath)
	}
	for _, r := range rows {
		if !strings.HasPrefix(r.ID, "RUN-C") {
			t.Fatalf("%s: %s is not a RUN-C id", runCopyRecordPath, r.ID)
		}
	}
	return rows
}

// checkRunBuilt: every built id is named once, names a row, and that row is
// approved.
func checkRunBuilt(rows []copyRow, built []string) []string {
	byID := map[string]copyRow{}
	for _, r := range rows {
		byID[r.ID] = r
	}
	var out []string
	seen := map[string]bool{}
	for _, id := range built {
		if seen[id] {
			out = append(out, fmt.Sprintf("runCopyBuilt names %s twice", id))
		}
		seen[id] = true
		r, ok := byID[id]
		switch {
		case !ok:
			out = append(out, fmt.Sprintf("runCopyBuilt names %s, which the deck has no row for", id))
		case r.Status != "approved":
			out = append(out, fmt.Sprintf("%s is built but %s: a string reaches code only after Paul approves it", id, r.Status))
		}
	}
	return out
}

// builtRows is the rows runCopyBuilt names, in deck order.
func builtRows(rows []copyRow, built []string) []copyRow {
	want := map[string]bool{}
	for _, id := range built {
		want[id] = true
	}
	var out []copyRow
	for _, r := range rows {
		if want[r.ID] {
			out = append(out, r)
		}
	}
	return out
}

func TestRunCopyRecordIsComplete(t *testing.T) {
	reportAll(t, checkMovesInventory(mustParseRunRecord(t), runCopyIDs))
}

func TestRunCopyHasNoEmDash(t *testing.T) {
	reportAll(t, checkMovesNoEmDash(mustParseRunRecord(t)))
}

func TestRunBuiltCopyIsApproved(t *testing.T) {
	reportAll(t, checkRunBuilt(mustParseRunRecord(t), runCopyBuilt))
}

// runFragment reports whether a row's text is a piece of a larger literal
// (its file cell says "a fragment"): the top-level usage text, which no
// whole-expression match can find.
func runFragment(r copyRow) bool { return strings.Contains(r.File, "(a fragment") }

// checkRunApplied is checkMovesApplied, plus the fragment rows: those must
// sit inside some string literal of their file (goLiteralMatches,
// approved_copy_test.go), never only in a comment.
func checkRunApplied(rows []copyRow, root string) []string {
	var whole []copyRow
	var out []string
	for _, r := range rows {
		if !runFragment(r) {
			whole = append(whole, r)
			continue
		}
		if r.Status != "approved" {
			continue
		}
		body, err := os.ReadFile(filepath.Join(root, r.Path()))
		if err != nil {
			out = append(out, fmt.Sprintf("%s: %v", r.ID, err))
			continue
		}
		if !goLiteralMatches(body, func(v string) bool { return strings.Contains(v, r.Text) }) {
			out = append(out, fmt.Sprintf("%s: %s has no string literal holding the approved text %q", r.ID, r.Path(), r.Text))
		}
	}
	return append(checkMovesApplied(whole, root), out...)
}

func TestRunBuiltCopyIsApplied(t *testing.T) {
	reportAll(t, checkRunApplied(builtRows(mustParseRunRecord(t), runCopyBuilt), "."))
}

// Each fixture case is a state the gate must refuse, for the right reason.
func TestRunCopyGateRefusesTheNegativeFixtures(t *testing.T) {
	dir := filepath.Join("testdata", "run-copy")
	rows, err := parseMovesCopyRecord(filepath.Join(dir, "record.md"))
	if err != nil {
		t.Fatal(err)
	}
	t.Run("a built row still pending", func(t *testing.T) {
		got := checkRunBuilt(rows, []string{"RUN-C2"})
		if len(got) != 1 || !strings.Contains(got[0], "RUN-C2 is built but pending") {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("a built id with no row", func(t *testing.T) {
		got := checkRunBuilt(rows, []string{"RUN-C9"})
		if len(got) != 1 || !strings.Contains(got[0], "no row for") {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("a built id named twice", func(t *testing.T) {
		got := checkRunBuilt(rows, []string{"RUN-C1", "RUN-C1"})
		if len(got) != 1 || !strings.Contains(got[0], "twice") {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("an approved row whose text is only in a comment", func(t *testing.T) {
		got := checkMovesApplied(builtRows(rows, []string{"RUN-C3"}), dir)
		if len(got) != 1 || !strings.Contains(got[0], "RUN-C3") {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("an approved row that is applied", func(t *testing.T) {
		if got := checkRunApplied(builtRows(rows, []string{"RUN-C1"}), dir); len(got) != 0 {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("a fragment inside a larger literal", func(t *testing.T) {
		if got := checkRunApplied(builtRows(rows, []string{"RUN-C4"}), dir); len(got) != 0 {
			t.Errorf("problems = %q", got)
		}
	})
	t.Run("a fragment only in a comment", func(t *testing.T) {
		got := checkRunApplied(builtRows(rows, []string{"RUN-C5"}), dir)
		if len(got) != 1 || !strings.Contains(got[0], "RUN-C5") {
			t.Errorf("problems = %q", got)
		}
	})
}

Create testdata/run-copy/record.md:

| id | file | text | status |
|---|---|---|---|
| RUN-C1 | applied.go | <host> has no run <id> | approved |
| RUN-C2 | applied.go | the instance hibernated | pending |
| RUN-C3 | applied.go | the box's agent stopped | approved |
| RUN-C4 | applied.go (a fragment of the usage text) | run one command | approved |
| RUN-C5 | applied.go (a fragment of the usage text) | only in a comment | approved |

Create testdata/run-copy/applied.go (never compiled: testdata is outside the build):

package fixture

import "fmt"

func say(host, id string) string { return fmt.Sprintf("%s has no run %s", host, id) }

const usage = `usage: x
  run   run one command, somewhere`

// the box's agent stopped: in this comment only, which never counts.
// only in a comment

• [ ] Step 2: Run to verify it fails

Run: nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt' -count=1 Expected: FAIL. parseMovesCopyTable refuses every row with "RUN-C1" is not an M-C<n>, MV-N<n> or HC5-N<n> id.

• [ ] Step 3: Accept RUN-C ids in the shared parser

In approved_moves_copy_test.go:91:

var movesCopyIDPattern = regexp.MustCompile(`^(M-C|MV-N|HC5-N|RUN-C)[0-9]+$`)

and at :150 the refusal names the new id family (the "bad id" case of TestParseMovesCopyTable still matches "is not an M-C"):

			return nil, fmt.Errorf("%s: %q is not an M-C<n>, MV-N<n>, HC5-N<n> or RUN-C<n> id", at, r.ID)

A RUN-C row in the moves record is still caught: checkMovesInventory reports any id movesCopyIDs does not pin.

• [ ] Step 4: Run to verify it passes

Run: nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt|TestParseMovesCopyTable|TestMovesCopy' -count=1 Expected: PASS (every deck row is pending and runCopyBuilt is empty).

• [ ] Step 5: Commit the gate

git add approved_run_copy_test.go approved_moves_copy_test.go testdata/run-copy/record.md testdata/run-copy/applied.go
git commit -m "run: the carlos run copy gate (Plan 1 Task 1)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

• [ ] Step 6: Paul's copy review

Invoke the copy-review skill with every row of docs/superpowers/plans/2026-10-08-carlos-run-copy.md (83 strings, grouped by the deck's three headings, with the deck's placeholder key as context). Stop here until Paul has submitted the review. Read the review's result.json before restarting the review server for any reason: a relaunch deletes it.

• [ ] Step 7: Apply the review verbatim

For every row: replace the text cell with Paul's text exactly, and set the status to approved. A row Paul strikes is deleted from the deck and from runCopyIDs (replace the generator with a literal list naming the ids that remain), and the plan task that names it drops the string. A row Paul adds gets the next free id and is added to runCopyIDs. Keep no em dash.

Run: nice -n 10 go test . -run 'TestRunCopy|TestRunBuilt' -count=1 Expected: PASS.

• [ ] Step 8: Commit the approved deck

git add docs/superpowers/plans/2026-10-08-carlos-run-copy.md approved_run_copy_test.go
git commit -m "run: the copy deck, approved by Paul

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 2: RoleCmd in instancectl

Files:

• Modify: internal/instancectl/names.go:23-39 (roles, ParseRole), :66-82 (prefixes, prefixFor), :104-107 (ParseUnitName's table)
• Modify: internal/instancectl/server.go:283 (the role error), :474 (handleStop roles), :487 (handleReset roles), :506-507 (the sweep's patterns)
• Modify: internal/instanceexec/exec.go:44,52, internal/instanceexec/exit.go:35,43
• Test: internal/instancectl/names_test.go:56-82, internal/instancectl/server_test.go (a sweep case)

Interfaces:

• Produces: instancectl.RoleCmd Role = "cmd", instancectl.UnitPrefixCmd = "carlos-instance-cmd@"; UnitName(RoleCmd, host) is carlos-instance-cmd@<escaped id>.service; /stop and /reset accept role=cmd; /sweep lists and stops cmd units.

• [ ] Step 1: Write the failing tests

In names_test.go, TestParseUnitNameRoundTrips iterates the four roles and TestParseRole accepts cmd:

func TestParseUnitNameRoundTrips(t *testing.T) {
	for _, role := range []Role{RoleRun, RoleSeed, RoleSidecar, RoleCmd} {
		u := UnitName(role, "bill-2.example.com")
		gotRole, esc, ok := ParseUnitName(u)
		if !ok || gotRole != role || esc != `bill\x2d2.example.com` {
			t.Errorf("ParseUnitName(%q) = %q %q %v", u, gotRole, esc, ok)
		}
	}
	if got := UnitName(RoleCmd, "bill-2.example.com"); got != `carlos-instance-cmd@bill\x2d2.example.com.service` {
		t.Errorf("cmd unit = %q", got)
	}
	for _, bad := range []string{"carlos-edge.service", "carlos-app@x.service", "carlos-instance@x.socket", "carlos-instance@.service", "carlos-instance-cmd@.service", "carlos-instances-x.slice"} {
		if _, _, ok := ParseUnitName(bad); ok {
			t.Errorf("ParseUnitName(%q) accepted", bad)
		}
	}
}

func TestParseRole(t *testing.T) {
	for _, s := range []string{"run", "seed", "sidecar", "cmd"} {
		if r, ok := ParseRole(s); !ok || string(r) != s {
			t.Errorf("ParseRole(%q) = %q %v", s, r, ok)
		}
	}
	for _, s := range []string{"", "Run", "exec", "run ", "Cmd", "cmd-err"} {
		if _, ok := ParseRole(s); ok {
			t.Errorf("ParseRole(%q) accepted", s)
		}
	}
}

Append to server_test.go:

func TestStopResetAndSweepCoverTheCmdRole(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		"list-units":                  "carlos-instance-cmd@a.example.com.service loaded active exited carlos exec-backed instance a.example.com (cmd)\n",
		"show -p ActiveState --value": "inactive\n",
	}}
	s, c := newServer(t, ctl)
	s.cgroupWaitBudget = time.Second
	if resp := post(t, c, "/stop?host=a.example.com&role=cmd"); resp.StatusCode != http.StatusNoContent {
		t.Fatalf("stop status %d", resp.StatusCode)
	}
	if resp := post(t, c, "/reset?host=a.example.com&role=cmd"); resp.StatusCode/100 != 2 {
		t.Fatalf("reset status %d", resp.StatusCode)
	}
	if resp := post(t, c, "/sweep"); resp.StatusCode != http.StatusOK {
		t.Fatalf("sweep status %d", resp.StatusCode)
	}
	calls := ctl.joined()
	for _, want := range []string{
		"stop carlos-instance-cmd@a.example.com.service",
		"reset-failed carlos-instance-cmd@a.example.com.service",
		"carlos-instance-cmd@*.service",
	} {
		if !strings.Contains(calls, want) {
			t.Errorf("missing %q in\n%s", want, calls)
		}
	}
}

• [ ] Step 2: Run to verify they fail

Run: nice -n 10 go test ./internal/instancectl/ -run 'TestParseUnitNameRoundTrips|TestParseRole|TestStopResetAndSweepCoverTheCmdRole' -count=1 Expected: FAIL, undefined: RoleCmd.

• [ ] Step 3: Implement

names.go:

type Role string

const (
	RoleRun     Role = "run"
	RoleSeed    Role = "seed"
	RoleSidecar Role = "sidecar"
	// RoleCmd is one `carlos run` command (run spec §5.4). "run" is the
	// serving role already, so the command's role is cmd.
	RoleCmd Role = "cmd"
)

// ParseRole accepts exactly the four literals.
func ParseRole(s string) (Role, bool) {
	switch Role(s) {
	case RoleRun, RoleSeed, RoleSidecar, RoleCmd:
		return Role(s), true
	}
	return "", false
}

// The four template prefixes. A unit under any of them is the platform's
// to start, stop and sweep; nothing else is (design §5.3).
const (
	UnitPrefixRun     = "carlos-instance@"
	UnitPrefixSeed    = "carlos-instance-seed@"
	UnitPrefixSidecar = "carlos-instance-sidecar@"
	UnitPrefixCmd     = "carlos-instance-cmd@"
)

func prefixFor(role Role) string {
	switch role {
	case RoleSeed:
		return UnitPrefixSeed
	case RoleSidecar:
		return UnitPrefixSidecar
	case RoleCmd:
		return UnitPrefixCmd
	}
	return UnitPrefixRun
}

ParseUnitName's table, the run prefix still last:

	}{{UnitPrefixSeed, RoleSeed}, {UnitPrefixSidecar, RoleSidecar}, {UnitPrefixCmd, RoleCmd}, {UnitPrefixRun, RoleRun}} {

server.go:283:

			return request{}, fmt.Errorf("role %q is not run, seed, sidecar or cmd", q.Get("role"))

server.go:474 and :487: the role lists become []Role{RoleRun, RoleSeed, RoleSidecar, RoleCmd}. server.go:506-507:

	out, err := s.Systemctl(ctx, "list-units", "--all", "--plain", "--no-legend", "--no-pager",
		UnitPrefixRun+"*.service", UnitPrefixSeed+"*.service", UnitPrefixSidecar+"*.service", UnitPrefixCmd+"*.service")

internal/instanceexec/exec.go:44 and :52 (RUN-C9, RUN-C13):

	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exec <run|seed|sidecar|cmd> <instance-id>")

		log.Fatalf("instance-exec: role %q is not run, seed, sidecar or cmd", fs.Arg(0))

internal/instanceexec/exit.go:35 and :43 (RUN-C10, RUN-C14):

	fs.Usage = cliflag.Usage(fs, "usage: carlos instance-exit <run|seed|sidecar|cmd> <instance-id>")

		log.Printf("instance-exit: role %q is not run, seed, sidecar or cmd", fs.Arg(0))

The trampoline needs nothing else: execLaunch (exec.go:72) reads launch.<role>.json for whatever role parsed, and validates argv0 under /opt/carlos/live.

In approved_run_copy_test.go: var runCopyBuilt = []string{"RUN-C9", "RUN-C10", "RUN-C13", "RUN-C14"}.

• [ ] Step 4: Run to verify they pass

Run: nice -n 10 go test ./internal/instancectl/ ./internal/instanceexec/ -count=1 && nice -n 10 go test . -run 'TestRunBuilt' -count=1 Expected: PASS. (internal/instanceexec's own tests are linux-tagged; they run on this box.)

• [ ] Step 5: Commit

git add internal/instancectl/names.go internal/instancectl/names_test.go internal/instancectl/server.go internal/instancectl/server_test.go internal/instanceexec/exec.go internal/instanceexec/exit.go approved_run_copy_test.go
git commit -m "instancectl: RoleCmd and the carlos-instance-cmd@ prefix (run spec §5.4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 3: The carlos-instance-cmd@.service template

Files:

• Create: internal/host/units/carlos-instance-cmd@.service
• Test: internal/host/units_isolation_test.go (new test, and the lists at :241 and :266-286, the seed-only rule at :303), internal/host/units_hardening_test.go:45 and :106-111, internal/host/host_test.go:32-64 (expectedUnits)
• Modify: docs/site/cli.md:2076-2077 and :2081 (RUN-C11, RUN-C12)

Interfaces:

• Consumes: readUnit(t, name) (units_isolation_test.go:10), hasDirective (units_hardening_test.go), unitsFS (bootstrap.go:34-35; the embed pattern units/*.service picks the new file up, and the bootstrap loop at bootstrap.go:512-529 installs it).
• Produces: the template systemd starts for UnitName(RoleCmd, host).

• [ ] Step 1: Write the failing test

Append to units_isolation_test.go:

// The cmd template is the seed template outside six named lines (run spec
// §5.4, codex 4): every sandbox line is the seed's, so a command has
// exactly the reach a seed has. Comments and blank lines are free; every
// directive is compared, in order.
func TestCmdTemplateEqualsSeedOutsideTheNamedLines(t *testing.T) {
	directives := func(name string) []string {
		var out []string
		for _, line := range strings.Split(readUnit(t, name), "\n") {
			line = strings.TrimSpace(line)
			if line == "" || strings.HasPrefix(line, "#") {
				continue
			}
			out = append(out, line)
		}
		return out
	}
	swap := map[string]string{
		"Description=carlos exec-backed instance %I (seed)":     "Description=carlos exec-backed instance %I (cmd)",
		"TimeoutStartSec=120":                                   "TimeoutStartSec=6h5m",
		"ExecStart=/opt/carlos/carlos instance-exec seed %I":    "ExecStart=/opt/carlos/carlos instance-exec cmd %I",
		"ExecStopPost=/opt/carlos/carlos instance-exit seed %I": "ExecStopPost=/opt/carlos/carlos instance-exit cmd %I",
		"StandardOutput=file:/run/carlos/logs/%I.seed.sock":     "StandardOutput=file:/run/carlos/logs/%I.cmd.sock",
		"StandardError=inherit":                                 "StandardError=file:/run/carlos/logs/%I.cmd-err.sock",
	}
	seed, cmd := directives("carlos-instance-seed@.service"), directives("carlos-instance-cmd@.service")
	want := make([]string, len(seed))
	used := map[string]bool{}
	for i, l := range seed {
		if c, ok := swap[l]; ok {
			want[i], used[l] = c, true
			continue
		}
		want[i] = l
	}
	for l := range swap {
		if !used[l] {
			t.Errorf("the seed template no longer has %q; the run spec names it as a cmd difference", l)
		}
	}
	for i := 0; i < len(want) || i < len(cmd); i++ {
		var w, g string
		if i < len(want) {
			w = want[i]
		}
		if i < len(cmd) {
			g = cmd[i]
		}
		if w != g {
			t.Fatalf("directive %d: cmd template has %q, want %q (the seed's line with the named swaps)", i, g, w)
		}
	}
	for _, keep := range []string{"Type=oneshot", "RemainAfterExit=yes", "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"} {
		if !hasDirective(readUnit(t, "carlos-instance-cmd@.service"), keep) {
			t.Errorf("cmd template lacks %q", keep)
		}
	}
}

Extend the existing lists:

• units_isolation_test.go:241: []string{"carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-cmd@.service", "carlos-instance-probe.service"}.
• the wants map at :266-286 gains:

		"carlos-instance-cmd@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=6h5m", "ExecStart=/opt/carlos/carlos instance-exec cmd %I",
			"ExecStopPost=/opt/carlos/carlos instance-exit cmd %I", "StandardOutput=file:/run/carlos/logs/%I.cmd.sock",
			"StandardError=file:/run/carlos/logs/%I.cmd-err.sock", "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"},

• :303: the seed-only rule covers cmd too: if (name == "carlos-instance-seed@.service" || name == "carlos-instance-cmd@.service") && strings.Contains(u, "\nSuccessExitStatus=") {.
• units_hardening_test.go:45: the list gains "carlos-instance-cmd@.service".
• units_hardening_test.go:106-111: the map gains "carlos-instance-cmd@.service": "%I",.
• host_test.go expectedUnits: add "carlos-instance-cmd@.service", before "carlos-instance-control.service" (the list is sorted).

• [ ] Step 2: Run to verify it fails

Run: nice -n 10 go test ./internal/host/ -run 'TestCmdTemplateEqualsSeedOutsideTheNamedLines|TestInstanceTemplatesShareOneNamespace|TestUnitsEmbedComplete|TestNamespaceBlockCarriesTheHardening|TestInstanceTemplatesBindOnlyTheirOwnBuild' -count=1 Expected: FAIL, units/carlos-instance-cmd@.service: file does not exist.

• [ ] Step 3: Write the template

Copy the seed template, then replace its head (from the first line through StandardError=inherit) with the head below. Everything from # The instance's own two directories, the only writable data paths. to the end of the file stays exactly as copied.

cp 'internal/host/units/carlos-instance-seed@.service' 'internal/host/units/carlos-instance-cmd@.service'

The head of carlos-instance-cmd@.service:

# One `carlos run` command against an isolated exec-backed instance (run
# spec §5.4). It is the seed template outside six lines, and a test holds
# it there (TestCmdTemplateEqualsSeedOutsideTheNamedLines): the same uid,
# namespace, bind mounts, slice and working directory as the instance, so
# the command has exactly the instance's reach. The agent starts it
# through the control service's POST /cmd and follows it with GET
# /cmd/<host>; the agent, not this file, enforces the run's own timeout.
[Unit]
Description=carlos exec-backed instance %I (cmd)
# Stop-only coupling, as for the seed: when the edge stops, the command
# stops, and only the agent starts one. A run picked before an agent
# restart is never run again (run spec §5.2).
BindsTo=carlos-edge.service
Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount

[Service]
Type=oneshot
RemainAfterExit=yes
# The result stays loaded, active (exited) or failed, until the agent has
# read it and stopped the unit (codex 4).
# A backstop past the run cap of 6 hours; the agent stops the unit at the
# run's own timeout.
TimeoutStartSec=6h5m
ExecStart=/opt/carlos/carlos instance-exec cmd %I
# The advisory exit record, as for the other templates. Not "+" (codex R3-2).
ExecStopPost=/opt/carlos/carlos instance-exit cmd %I
User=carlos
Group=carlos
Slice=carlos-instances-%i.slice
# SIGTERM to the command, SIGKILL to anything left 20 s later.
KillMode=mixed
TimeoutStopSec=20
# No SuccessExitStatus=: the agent reads the exit status itself.
# Two listeners, so stdout and stderr stay apart (run spec §5.5). PID 1
# opens both as root, outside the namespace, on the nosymfollow logs mount.
StandardOutput=file:/run/carlos/logs/%I.cmd.sock
StandardError=file:/run/carlos/logs/%I.cmd-err.sock

In docs/site/cli.md, the last sentence of ### carlos instance-exec (:2076-2077, today wrapped across two lines) becomes ` ExecStart= of the four instance templates. on one line (RUN-C11; the gate matches docs raw, so the sentence must not wrap), and the first sentence of ### carlos instance-exit (:2081) becomes ExecStopPost= of the four instance templates. (RUN-C12); the rest of each paragraph is unchanged. runCopyBuilt gains "RUN-C11", "RUN-C12"`.

• [ ] Step 4: Run to verify it passes

Run: nice -n 10 go test ./internal/host/ -run 'TestCmdTemplate|TestInstanceTemplates|TestUnitsEmbedComplete|TestNamespaceBlock|TestSameUIDAudit|TestControlUnitsShape' -count=1 && nice -n 10 go test . -run 'TestRunBuilt' -count=1 && nice -n 10 go test ./internal/docsite/ -count=1 Expected: PASS. TestSameUIDAuditIsCoveredByTheTemplates iterates unitNames() and so covers the new file; claimUnitCensus (claimcensus.go:86) adds it to the census with no edit.

• [ ] Step 5: Commit

git add 'internal/host/units/carlos-instance-cmd@.service' internal/host/units_isolation_test.go internal/host/units_hardening_test.go internal/host/host_test.go docs/site/cli.md approved_run_copy_test.go
git commit -m "units: carlos-instance-cmd@, the seed template outside six lines (run spec §5.4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 4: POST /cmd and the GET /cmd/{host} long-poll

Files:

• Create: internal/instancectl/cmd.go, internal/instancectl/cmd_test.go
• Modify: internal/instancectl/names.go (CmdStatus beside SeedResponse at :184), internal/instancectl/server.go:35-75 (three timing fields), :249-263 (two routes), internal/instancectl/client.go:23-28 (DeadlineCmdPoll) and after :161 (Cmd, CmdStatus), internal/instancectl/claim_test.go:11

Interfaces:

• Consumes: parseRequest(r, nil, true), s.refuseStart(w, host, unit) bool, s.setTier(ctx, host, tier), s.showValue(ctx, prop, unit) string, s.stopUnit(ctx, unit), parseShow(out) map[string]string, writeJSON, s.claimAwareLegacy, StartResponse{InvocationID}, Client.do, hostRole.
• Produces: - type CmdStatus struct { State, InvocationID, Result, Status, Code string } with json state, invocation_id, result, status, code; CmdRunning = "running", CmdDone = "done", CmdGone = "gone". - POST /cmd?host=&memory_max= answers StartResponse at once; 409 while the unit is still loaded; 503 with claim control armed. - GET /cmd/{host}?invocation=<id> answers CmdStatus within 30 s; 503 with claim control armed. - func (c *Client) Cmd(ctx context.Context, host string, memoryMax int64) (string, error); func (c *Client) CmdStatus(ctx context.Context, host, invocation string) (CmdStatus, error); DeadlineCmdPoll = 35 * time.Second. - Code is systemd's ExecMainCode as systemctl show prints it: a number (1 exited, 2 killed, 3 dumped). Plan 3 reads both the numbers and the words.

• [ ] Step 1: Write the failing tests

internal/instancectl/cmd_test.go:

package instancectl

import (
	"context"
	"encoding/json"
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
	"time"
)

// set changes one answer under the fake's lock, for a test whose unit
// moves on while a handler polls.
func (f *fakeCtl) set(prefix, out string) {
	f.mu.Lock()
	f.answers[prefix] = out
	f.mu.Unlock()
}

const cmdShowAll = "show -p InvocationID,ActiveState,SubState,Result,ExecMainStatus,ExecMainCode"

func fastCmd(s *Server) {
	s.cmdPollEvery = 5 * time.Millisecond
	s.cmdWaitBound = 300 * time.Millisecond
	s.cmdStartBound = 300 * time.Millisecond
}

func TestCmdStartsWithoutBlockingAndAnswersTheNewInvocation(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		"show -p ActiveState --value":  "inactive\n",
		"show -p InvocationID --value": "old\n",
	}}
	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
	_, c := newServer(t, ctl, fastCmd)
	resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912")
	if resp.StatusCode != 200 {
		t.Fatalf("status %d", resp.StatusCode)
	}
	var out StartResponse
	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || out.InvocationID != "new" {
		t.Fatalf("response %+v, %v", out, err)
	}
	calls := ctl.joined()
	for _, want := range []string{"set-property --runtime carlos-instances-h.example.com.slice", "start --no-block carlos-instance-cmd@h.example.com.service"} {
		if !strings.Contains(calls, want) {
			t.Errorf("missing %q in\n%s", want, calls)
		}
	}
	if strings.Contains(calls, "stop ") {
		t.Errorf("a started command was stopped:\n%s", calls)
	}
}

func TestCmdRefusesAUnitStillLoaded(t *testing.T) {
	for _, state := range []string{"active", "activating", "deactivating"} {
		ctl := &fakeCtl{answers: map[string]string{"show -p ActiveState --value": state + "\n"}}
		_, c := newServer(t, ctl, fastCmd)
		resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912")
		if resp.StatusCode != http.StatusConflict {
			t.Errorf("%s: status %d, want 409", state, resp.StatusCode)
		}
		if strings.Contains(ctl.joined(), "start ") {
			t.Errorf("%s: a loaded unit was started again:\n%s", state, ctl.joined())
		}
	}
}

func TestCmdResetsAFailedUnitBeforeTheStart(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		"show -p ActiveState --value":  "failed\n",
		"show -p InvocationID --value": "old\n",
	}}
	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
	_, c := newServer(t, ctl, fastCmd)
	if resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912"); resp.StatusCode != 200 {
		t.Fatalf("status %d", resp.StatusCode)
	}
	calls := ctl.joined()
	if r, s := strings.Index(calls, "reset-failed carlos-instance-cmd@"), strings.Index(calls, "start --no-block"); r < 0 || s < 0 || r > s {
		t.Errorf("reset-failed must come before the start:\n%s", calls)
	}
}

func TestCmdStopsAUnitThatNeverReportsAnInvocation(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		"show -p ActiveState --value":  "inactive\n",
		"show -p InvocationID --value": "old\n",
	}}
	_, c := newServer(t, ctl, fastCmd)
	if resp := post(t, c, "/cmd?host=h.example.com&memory_max=536870912"); resp.StatusCode != http.StatusBadGateway {
		t.Fatalf("status %d, want 502", resp.StatusCode)
	}
	if !strings.Contains(ctl.joined(), "stop carlos-instance-cmd@h.example.com.service") {
		t.Errorf("the unit was not stopped:\n%s", ctl.joined())
	}
}

func getStatus(t *testing.T, c *http.Client, path string) CmdStatus {
	t.Helper()
	resp, err := c.Get("http://unix" + path)
	if err != nil {
		t.Fatal(err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != 200 {
		t.Fatalf("GET %s: status %d", path, resp.StatusCode)
	}
	var st CmdStatus
	if err := json.NewDecoder(resp.Body).Decode(&st); err != nil {
		t.Fatal(err)
	}
	return st
}

func TestCmdStatusLongPollsUntilDone(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		cmdShowAll: "InvocationID=inv1\nActiveState=activating\nSubState=start\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
	}}
	_, c := newServer(t, ctl, fastCmd)
	go func() {
		time.Sleep(50 * time.Millisecond)
		ctl.set(cmdShowAll, "InvocationID=inv1\nActiveState=active\nSubState=exited\nResult=success\nExecMainStatus=3\nExecMainCode=1\n")
	}()
	st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1")
	if st.State != CmdDone || st.Status != "3" || st.Code != "1" || st.InvocationID != "inv1" {
		t.Fatalf("status = %+v", st)
	}
}

func TestCmdStatusAnswersRunningAtTheBound(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		cmdShowAll: "InvocationID=inv1\nActiveState=activating\nSubState=start\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
	}}
	_, c := newServer(t, ctl, fastCmd)
	start := time.Now()
	if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdRunning {
		t.Fatalf("status = %+v", st)
	}
	if took := time.Since(start); took < 250*time.Millisecond {
		t.Errorf("answered running after %s, before the bound", took)
	}
}

func TestCmdStatusDoneCoversFailedAndInactive(t *testing.T) {
	for _, show := range []string{
		"InvocationID=inv1\nActiveState=failed\nSubState=failed\nResult=exit-code\nExecMainStatus=1\nExecMainCode=1\n",
		"InvocationID=inv1\nActiveState=inactive\nSubState=dead\nResult=success\nExecMainStatus=0\nExecMainCode=1\n",
	} {
		ctl := &fakeCtl{answers: map[string]string{cmdShowAll: show}}
		_, c := newServer(t, ctl, fastCmd)
		if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdDone {
			t.Errorf("%q: state %s, want done", show, st.State)
		}
	}
}

func TestCmdStatusIsGoneForAnotherInvocation(t *testing.T) {
	for _, show := range []string{
		"InvocationID=inv2\nActiveState=active\nSubState=exited\nResult=success\nExecMainStatus=0\nExecMainCode=1\n",
		"InvocationID=\nActiveState=inactive\nSubState=dead\nResult=success\nExecMainStatus=0\nExecMainCode=0\n",
	} {
		ctl := &fakeCtl{answers: map[string]string{cmdShowAll: show}}
		_, c := newServer(t, ctl, fastCmd)
		if st := getStatus(t, c, "/cmd/h.example.com?invocation=inv1"); st.State != CmdGone {
			t.Errorf("%q: state %s, want gone", show, st.State)
		}
	}
}

func TestCmdRoutesRefuseWithClaimControlArmed(t *testing.T) {
	called := false
	s := &Server{ClaimStop: http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}), Systemctl: func(context.Context, ...string) (string, error) { called = true; return "", nil }}
	for _, req := range []*http.Request{
		httptest.NewRequest(http.MethodPost, "/cmd?host=app.example&memory_max=536870912", nil),
		httptest.NewRequest(http.MethodGet, "/cmd/app.example?invocation=x", nil),
	} {
		w := httptest.NewRecorder()
		s.Handler().ServeHTTP(w, req)
		if w.Code != http.StatusServiceUnavailable {
			t.Errorf("%s %s: status %d", req.Method, req.URL, w.Code)
		}
	}
	if called {
		t.Fatal("a cmd route reached systemctl with claim control armed")
	}
}

func TestClientCmdAndCmdStatus(t *testing.T) {
	ctl := &fakeCtl{answers: map[string]string{
		"show -p ActiveState --value":  "inactive\n",
		"show -p InvocationID --value": "old\n",
		cmdShowAll:                     "InvocationID=new\nActiveState=failed\nSubState=failed\nResult=exit-code\nExecMainStatus=2\nExecMainCode=1\n",
	}}
	ctl.onStart = func() { ctl.set("show -p InvocationID --value", "new\n") }
	_, cl := newClient(t, ctl, fastCmd)
	inv, err := cl.Cmd(context.Background(), "h.example.com", 536870912)
	if err != nil || inv != "new" {
		t.Fatalf("Cmd = %q, %v", inv, err)
	}
	st, err := cl.CmdStatus(context.Background(), "h.example.com", inv)
	if err != nil || st.State != CmdDone || st.Status != "2" {
		t.Fatalf("CmdStatus = %+v, %v", st, err)
	}
}

Add "cmd" to the verb list in claim_test.go:11's loop is not enough for the GET route, which TestCmdRoutesRefuseWithClaimControlArmed covers; add it anyway so the legacy-verb test names every POST: []string{"start", "seed", "stop", "reset", "sweep", "cmd"}.

• [ ] Step 2: Run to verify they fail

Run: TMPDIR=/tmp/g nice -n 10 go test ./internal/instancectl/ -run 'TestCmd|TestClientCmd|TestClaimRootRefuses' -count=1 Expected: FAIL, s.cmdPollEvery undefined.

• [ ] Step 3: Implement

names.go, after SeedResponse:

// CmdStatus is GET /cmd/{host}'s answer (run spec §5.4). State is running,
// done once the named invocation's main process has exited (active
// (exited), failed, or inactive after a stop), or gone when the unit's
// invocation is another one or none. Result, Status and Code are systemd's
// Result, ExecMainStatus and ExecMainCode as `systemctl show` prints them.
type CmdStatus struct {
	State        string `json:"state"`
	InvocationID string `json:"invocation_id"`
	Result       string `json:"result"`
	Status       string `json:"status"`
	Code         string `json:"code"`
}

const (
	CmdRunning = "running"
	CmdDone    = "done"
	CmdGone    = "gone"
)

server.go, in Server after DiskEvery:

	// cmdPollEvery, cmdWaitBound and cmdStartBound pace the cmd routes:
	// how often GET /cmd/{host} re-reads the unit (250 ms), how long it
	// holds a request (30 s, run spec §5.4), and how long POST /cmd waits
	// for the queued start to show a new invocation (10 s). Zero is the
	// default; tests shorten them.
	cmdPollEvery, cmdWaitBound, cmdStartBound time.Duration

server.go:249-263, after the /sweep line:

	mux.HandleFunc("POST /cmd", s.claimAwareLegacy(s.handleCmd))
	mux.HandleFunc("GET /cmd/{host}", s.claimAwareLegacy(s.handleCmdStatus))

internal/instancectl/cmd.go:

package instancectl

import (
	"context"
	"fmt"
	"net/http"
	"strings"
	"time"

	"github.com/carlosframework/platform/internal/provision"
)

func (s *Server) cmdPoll() time.Duration {
	if s.cmdPollEvery > 0 {
		return s.cmdPollEvery
	}
	return 250 * time.Millisecond
}

func (s *Server) cmdWait() time.Duration {
	if s.cmdWaitBound > 0 {
		return s.cmdWaitBound
	}
	return 30 * time.Second
}

func (s *Server) cmdStartWait() time.Duration {
	if s.cmdStartBound > 0 {
		return s.cmdStartBound
	}
	return 10 * time.Second
}

// handleCmd starts host's cmd unit and answers at once with the new
// invocation (run spec §5.4); /seed blocks, this never does. The agent
// stops a cmd unit after every run, so a unit still loaded (active,
// activating, deactivating) belongs to a run this request must not end: it
// is refused, never restarted. A failed one is reset first, or start would
// keep its old result.
func (s *Server) handleCmd(w http.ResponseWriter, r *http.Request) {
	req, err := parseRequest(r, nil, true)
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}
	ctx := r.Context()
	unit := UnitName(RoleCmd, req.host)
	switch state := s.showValue(ctx, "ActiveState", unit); state {
	case "active", "activating", "deactivating", "reloading", "refreshing":
		http.Error(w, "cmd: "+unit+" is "+state, http.StatusConflict)
		return
	case "failed":
		if out, err := s.Systemctl(ctx, "reset-failed", unit); err != nil {
			http.Error(w, "reset-failed: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
			return
		}
	}
	if s.refuseStart(w, req.host, unit) {
		return
	}
	if err := s.setTier(ctx, req.host, req.tier); err != nil {
		http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
		return
	}
	before := s.showValue(ctx, "InvocationID", unit)
	if out, err := s.Systemctl(ctx, "start", "--no-block", unit); err != nil {
		http.Error(w, "start: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
		return
	}
	// --no-block returns once the job is queued; the invocation exists
	// once the job runs.
	deadline := time.Now().Add(s.cmdStartWait())
	for {
		if inv := s.showValue(ctx, "InvocationID", unit); inv != "" && inv != before {
			s.logf("instance-control: %s: started %s memory.max=%d", req.host, unit, req.tier.MemoryMax)
			writeJSON(w, http.StatusOK, StartResponse{InvocationID: inv})
			return
		}
		if time.Now().After(deadline) || ctx.Err() != nil {
			dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
			s.stopUnit(dctx, unit)
			cancel()
			http.Error(w, "cmd: "+unit+" showed no new invocation", http.StatusBadGateway)
			return
		}
		time.Sleep(s.cmdPoll())
	}
}

// handleCmdStatus is the bounded long-poll: it answers as soon as the named
// invocation is done or gone, and running at the bound.
func (s *Server) handleCmdStatus(w http.ResponseWriter, r *http.Request) {
	host := r.PathValue("host")
	if !provision.ValidHostShape(host) {
		http.Error(w, fmt.Sprintf("host %q is not a host", host), http.StatusBadRequest)
		return
	}
	want := r.URL.Query().Get("invocation")
	if want == "" {
		http.Error(w, "invocation is required", http.StatusBadRequest)
		return
	}
	ctx := r.Context()
	unit := UnitName(RoleCmd, host)
	deadline := time.Now().Add(s.cmdWait())
	for {
		out, err := s.Systemctl(ctx, "show", "-p", "InvocationID,ActiveState,SubState,Result,ExecMainStatus,ExecMainCode", unit)
		if err != nil {
			http.Error(w, "show: "+err.Error(), http.StatusBadGateway)
			return
		}
		st := cmdStatusOf(parseShow(out), want)
		if st.State != CmdRunning || !time.Now().Before(deadline) {
			writeJSON(w, http.StatusOK, st)
			return
		}
		select {
		case <-ctx.Done():
			return
		case <-time.After(s.cmdPoll()):
		}
	}
}

// cmdStatusOf reads one show for the named invocation.
func cmdStatusOf(m map[string]string, want string) CmdStatus {
	st := CmdStatus{InvocationID: m["InvocationID"], Result: m["Result"], Status: m["ExecMainStatus"], Code: m["ExecMainCode"]}
	active, sub := m["ActiveState"], m["SubState"]
	switch {
	case st.InvocationID != want:
		st.State = CmdGone
	case active == "failed", active == "inactive", active == "active" && sub == "exited":
		st.State = CmdDone
	default:
		st.State = CmdRunning
	}
	return st
}

client.go, in the deadline block:

	// DeadlineCmdPoll bounds GET /cmd/{host}; the server answers within 30 s.
	DeadlineCmdPoll = 35 * time.Second

and after Stop:

// Cmd starts host's cmd unit and returns its invocation (run spec §5.4).
func (c *Client) Cmd(ctx context.Context, host string, memoryMax int64) (string, error) {
	q := hostRole(host, "")
	q.Set("memory_max", strconv.FormatInt(memoryMax, 10))
	var out StartResponse
	if _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/cmd", q, &out); err != nil {
		return "", err
	}
	return out.InvocationID, nil
}

// CmdStatus long-polls the named invocation of host's cmd unit.
func (c *Client) CmdStatus(ctx context.Context, host, invocation string) (CmdStatus, error) {
	var out CmdStatus
	_, _, err := c.do(ctx, DeadlineCmdPoll, http.MethodGet, "/cmd/"+url.PathEscape(host), url.Values{"invocation": {invocation}}, &out)
	return out, err
}

newClient(t, ctl, configure...) (client_test.go:16) already forwards configure hooks to serveOnSocket, so fastCmd reaches the server.

• [ ] Step 4: Run to verify they pass

Run: TMPDIR=/tmp/g nice -n 10 go test ./internal/instancectl/ -count=1 Expected: PASS.

• [ ] Step 5: Commit

git add internal/instancectl/cmd.go internal/instancectl/cmd_test.go internal/instancectl/names.go internal/instancectl/server.go internal/instancectl/client.go internal/instancectl/claim_test.go
git commit -m "instancectl: POST /cmd and the GET /cmd/{host} long-poll (run spec §5.4, codex 4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 5: instancefs.RemoveAllBeneath

Plan 3 deletes .carlos-run/<id>/ after each run and every .carlos-run/ at agent start. The instance shares uid carlos with the agent and can plant symlinks and subdirectories there, and RemoveBeneath (dirs_linux.go:118) removes only a non-directory entry.

Files:

• Modify: internal/instancefs/dirs_linux.go (append), internal/instancefs/instancefs_other.go (stub)
• Test: internal/instancefs/dirs_linux_test.go (append)

Interfaces:

• Consumes: openDir(dir string) (int, error) (instancefs_linux.go:30), bareName(name string) error (instancefs.go:72), ErrSymlink, ErrEscapes.
• Produces: func RemoveAllBeneath(dir, name string) error: removes name directly under dir and everything below it, following no symlink anywhere; a symlink is unlinked, never its target; a missing name is nil; dir reached through a symlink is ErrSymlink; a name with a slash is ErrEscapes. The !linux stub returns errors.ErrUnsupported after bareName.

• [ ] Step 1: Write the failing tests

Append to dirs_linux_test.go:

func TestRemoveAllBeneathRemovesATree(t *testing.T) {
	dir := t.TempDir()
	for _, p := range []string{"run/a", "run/sub/b", "run/sub/deeper/c"} {
		full := filepath.Join(dir, p)
		if err := os.MkdirAll(filepath.Dir(full), 0o700); err != nil {
			t.Fatal(err)
		}
		if err := os.WriteFile(full, []byte("x"), 0o600); err != nil {
			t.Fatal(err)
		}
	}
	if err := RemoveAllBeneath(dir, "run"); err != nil {
		t.Fatal(err)
	}
	if _, err := os.Lstat(filepath.Join(dir, "run")); !os.IsNotExist(err) {
		t.Fatalf("run survived: %v", err)
	}
	if err := RemoveAllBeneath(dir, "run"); err != nil {
		t.Fatalf("a missing name must be nil: %v", err)
	}
}

func TestRemoveAllBeneathNeverFollowsASymlink(t *testing.T) {
	dir, outside := t.TempDir(), t.TempDir()
	keep := filepath.Join(outside, "keep")
	if err := os.WriteFile(keep, []byte("x"), 0o600); err != nil {
		t.Fatal(err)
	}
	if err := os.MkdirAll(filepath.Join(dir, "run"), 0o700); err != nil {
		t.Fatal(err)
	}
	if err := os.Symlink(outside, filepath.Join(dir, "run", "link")); err != nil {
		t.Fatal(err)
	}
	if err := os.Symlink(outside, filepath.Join(dir, "top")); err != nil {
		t.Fatal(err)
	}
	if err := RemoveAllBeneath(dir, "run"); err != nil {
		t.Fatal(err)
	}
	if err := RemoveAllBeneath(dir, "top"); err != nil {
		t.Fatal(err)
	}
	if _, err := os.Stat(keep); err != nil {
		t.Fatalf("a symlink's target was removed: %v", err)
	}
	if _, err := os.Lstat(filepath.Join(dir, "top")); !os.IsNotExist(err) {
		t.Fatalf("the top-level symlink itself survived: %v", err)
	}
}

func TestRemoveAllBeneathRefusesPathsAndSymlinkedParents(t *testing.T) {
	dir, outside := t.TempDir(), t.TempDir()
	if err := RemoveAllBeneath(dir, "a/b"); !errors.Is(err, ErrEscapes) {
		t.Errorf("a/b: %v, want ErrEscapes", err)
	}
	link := filepath.Join(dir, "parent")
	if err := os.Symlink(outside, link); err != nil {
		t.Fatal(err)
	}
	if err := os.Mkdir(filepath.Join(outside, "run"), 0o700); err != nil {
		t.Fatal(err)
	}
	if err := RemoveAllBeneath(link, "run"); !errors.Is(err, ErrSymlink) {
		t.Errorf("symlinked parent: %v, want ErrSymlink", err)
	}
	if _, err := os.Stat(filepath.Join(outside, "run")); err != nil {
		t.Errorf("removed through a symlinked parent: %v", err)
	}
}

(dirs_linux_test.go already imports errors, os, path/filepath and testing; add any it lacks.)

• [ ] Step 2: Run to verify they fail

Run: nice -n 10 go test ./internal/instancefs/ -run TestRemoveAllBeneath -count=1 Expected: FAIL, undefined: RemoveAllBeneath.

• [ ] Step 3: Implement

Append to dirs_linux.go:

// RemoveAllBeneath removes name, directly under dir, and everything below
// it. No symlink is followed at any depth: dir itself is opened without
// following one (ErrSymlink), every entry is examined with fstatat
// AT_SYMLINK_NOFOLLOW, and a symlink is unlinked, never its target. A
// tenant sharing the agent's uid can plant links and directories in a run
// directory (run spec §5.2); this is the only way the agent deletes one.
// A missing name is not an error.
func RemoveAllBeneath(dir, name string) error {
	if err := bareName(name); err != nil {
		return err
	}
	dfd, err := openDir(dir)
	if err != nil {
		return err
	}
	defer unix.Close(dfd)
	return removeAt(dfd, name, filepath.Join(dir, name), 0)
}

// removeAt removes name under the directory fd parent; path is for errors.
func removeAt(parent int, name, path string, depth int) error {
	if depth > 64 {
		return fmt.Errorf("instancefs: %s is nested more than 64 deep", path)
	}
	var st unix.Stat_t
	if err := unix.Fstatat(parent, name, &st, unix.AT_SYMLINK_NOFOLLOW); err != nil {
		if errors.Is(err, unix.ENOENT) {
			return nil
		}
		return &os.PathError{Op: "fstatat", Path: path, Err: err}
	}
	if st.Mode&unix.S_IFMT != unix.S_IFDIR {
		if err := unix.Unlinkat(parent, name, 0); err != nil && !errors.Is(err, unix.ENOENT) {
			return &os.PathError{Op: "unlinkat", Path: path, Err: err}
		}
		return nil
	}
	fd, err := unix.Openat(parent, name, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
	if err != nil {
		return &os.PathError{Op: "openat", Path: path, Err: err}
	}
	f := os.NewFile(uintptr(fd), path)
	names, err := f.Readdirnames(-1)
	if err != nil {
		f.Close()
		return err
	}
	for _, n := range names {
		if err := removeAt(fd, n, filepath.Join(path, n), depth+1); err != nil {
			f.Close()
			return err
		}
	}
	f.Close()
	if err := unix.Unlinkat(parent, name, unix.AT_REMOVEDIR); err != nil && !errors.Is(err, unix.ENOENT) {
		return &os.PathError{Op: "unlinkat", Path: path, Err: err}
	}
	return nil
}

Append to instancefs_other.go:

func RemoveAllBeneath(dir, name string) error {
	if err := bareName(name); err != nil {
		return err
	}
	return errors.ErrUnsupported
}

• [ ] Step 4: Run to verify they pass

Run: nice -n 10 go test ./internal/instancefs/ -count=1 && GOOS=darwin go build ./internal/instancefs/ && GOOS=windows go build ./internal/instancefs/ Expected: PASS, and both cross builds succeed.

• [ ] Step 5: Commit

git add internal/instancefs/dirs_linux.go internal/instancefs/instancefs_other.go internal/instancefs/dirs_linux_test.go
git commit -m "instancefs: RemoveAllBeneath, a tree removal that follows no symlink

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 6: ExecBackend.StartCmd: env, launch file, streams, unit

Files:

• Create: internal/activator/cmdunit.go, internal/activator/cmdunit_test.go
• Modify: internal/activator/control.go:17-24 (two methods), internal/activator/prepare.go:44-57 (env for cmd), :93-99 (no app-log listener for cmd), :131-137 (logSocketNamed), internal/activator/backend_exec.go:1024-1056 (waitRoleCgroupEmpty), internal/activator/backend_unit.go (after :341, SplitBackend.StartCmd), internal/activator/unitproc_test.go:24-70 (fakeControl)

Interfaces:

• Consumes: b.prepare(rt, role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error), b.nestedGate(rt), b.ensureLiveLink(ctx, host), b.Layout(rt), b.liveLink(host), b.tier(rt), b.writeLaunch(p, role, host, argv, env), childEnv, withVar, withoutVars, b.Region, edge.SecureUnixListen, instancectl.CmdStatus, Client.Cmd, Client.CmdStatus.
• Produces: - Control gains Cmd(ctx, host string, memoryMax int64) (string, error) and CmdStatus(ctx, host, invocation string) (instancectl.CmdStatus, error). - type CmdSpec struct { RunID string; Args []string; RunDir string; Stdout, Stderr io.Writer; Prepare func(ctx context.Context) error } (Prepare is used by Task 8). - type CmdUnit interface { Invocation() string; Poll(ctx context.Context) (instancectl.CmdStatus, error); StopAndSettle(ctx context.Context) error }. - type CmdBackend interface { StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) }; *ExecBackend and *SplitBackend implement it. - const CmdRunDirName = ".carlos-run"; StartCmd refuses a spec.RunDir other than <instance dir>/.carlos-run/<RunID>. - var ErrCmdNotNested = errors.New("activator: the instance is not on the nested layout"). - func (b *ExecBackend) waitRoleCgroupEmpty(ctx context.Context, rt registry.Route, role instancectl.Role) error.

• [ ] Step 1: Write the failing tests

Extend fakeControl in unitproc_test.go with two fields and two methods:

	cmdInv    string
	cmdErr    error
	cmdStatus instancectl.CmdStatus

func (f *fakeControl) Cmd(_ context.Context, host string, max int64) (string, error) {
	f.rec("cmd " + host)
	return f.cmdInv, f.cmdErr
}
func (f *fakeControl) CmdStatus(_ context.Context, host, inv string) (instancectl.CmdStatus, error) {
	f.rec("cmdstatus " + host + " " + inv)
	return f.cmdStatus, nil
}

internal/activator/cmdunit_test.go:

package activator

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"net"
	"os"
	"path/filepath"
	"strings"
	"sync"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/instancectl"
	"github.com/carlosframework/platform/internal/provision"
	"github.com/carlosframework/platform/internal/registry"
)

// syncBuffer is a bytes.Buffer two goroutines may write.
type syncBuffer struct {
	mu sync.Mutex
	b  bytes.Buffer
}

func (s *syncBuffer) Write(p []byte) (int, error) { s.mu.Lock(); defer s.mu.Unlock(); return s.b.Write(p) }
func (s *syncBuffer) String() string              { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }

func TestCmdVarsComposeTheRunEnvironment(t *testing.T) {
	b := &ExecBackend{Region: "ie"}
	rt := registry.Route{Host: "h.example.com"}
	file := map[string]string{
		"APP_KEY": "k", "CARLOS_ADMIN_TOKEN": "tok", "CARLOS_CONTROL_SOCKET": "/x", "CARLOS_PURGE": "header",
		"CARLOS_REGION": "us", "CARLOS_HOST": "liar.example.com", "CARLOS_RUN_ID": "planted",
	}
	got := b.cmdVars(rt, file, "/data/instances/h.example.com/app.db", "run_a", "/data/instances/h.example.com/.carlos-run/run_a")
	want := map[string]string{
		"APP_KEY": "k", "CARLOS_ADMIN_TOKEN": "tok", "CARLOS_REGION": "ie", "CARLOS_HOST": "h.example.com",
		"CARLOS_DB": "/data/instances/h.example.com/app.db", "CARLOS_RUN_ID": "run_a",
		"CARLOS_RUN_DIR": "/data/instances/h.example.com/.carlos-run/run_a",
	}
	if len(got) != len(want) {
		t.Fatalf("env = %v", got)
	}
	for k, v := range want {
		if got[k] != v {
			t.Errorf("%s = %q, want %q", k, got[k], v)
		}
	}
	b.Region = ""
	if _, ok := b.cmdVars(rt, file, "/db", "run_a", "/d")["CARLOS_REGION"]; ok {
		t.Error("an empty backend region must remove CARLOS_REGION (codex 7)")
	}
}

func startTestCmd(t *testing.T) (*ExecBackend, registry.Route, *fakeControl, *fakeUnit, CmdUnit, *syncBuffer, *syncBuffer) {
	t.Helper()
	ctl := &fakeControl{cmdInv: "inv1"}
	unit := &fakeUnit{}
	b, rt := nestedBackend(t, ctl, unit)
	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutNested, "" }
	out, errs := &syncBuffer{}, &syncBuffer{}
	dir := filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), CmdRunDirName, "run_a")
	u, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a", Args: []string{"import", "--dry-run"}, RunDir: dir, Stdout: out, Stderr: errs})
	if err != nil {
		t.Fatalf("StartCmd: %v", err)
	}
	return b, rt, ctl, unit, u, out, errs
}

func TestStartCmdWritesTheLaunchFileAndStartsTheUnit(t *testing.T) {
	b, rt, ctl, _, u, _, _ := startTestCmd(t)
	if u.Invocation() != "inv1" || !strings.Contains(ctl.joined(), "cmd "+rt.Host) {
		t.Fatalf("invocation %q, calls\n%s", u.Invocation(), ctl.joined())
	}
	raw, err := os.ReadFile(filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), "launch.cmd.json"))
	if err != nil {
		t.Fatal(err)
	}
	var l instancectl.Launch
	if err := json.Unmarshal(raw, &l); err != nil {
		t.Fatal(err)
	}
	if len(l.Argv) != 3 || l.Argv[0] != b.liveLink(rt.Host) || l.Argv[1] != "import" || l.Argv[2] != "--dry-run" {
		t.Errorf("argv = %q: exactly the binary and what followed --", l.Argv)
	}
	env := strings.Join(l.Env, "\n")
	for _, want := range []string{"CARLOS_RUN_ID=run_a", "CARLOS_HOST=" + rt.Host, "CARLOS_DB=" + rt.DBPath} {
		if !strings.Contains(env, want) {
			t.Errorf("env lacks %q", want)
		}
	}
	if strings.Contains(env, "CARLOS_CONTROL_SOCKET=") {
		t.Error("a command has no control socket")
	}
}

func TestStartCmdRefusesAFlatInstanceAndAForeignRunDir(t *testing.T) {
	ctl := &fakeControl{cmdInv: "inv1"}
	b, rt := nestedBackend(t, ctl, &fakeUnit{})
	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutFlat, "test" }
	if _, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdNotNested) {
		t.Errorf("flat: %v, want ErrCmdNotNested", err)
	}
	b.layoutFn = func(registry.Route) (Layout, string) { return LayoutNested, "" }
	if _, err := b.StartCmd(context.Background(), rt, CmdSpec{RunID: "run_a", RunDir: "/elsewhere"}); err == nil {
		t.Error("a run directory outside the instance's .carlos-run was accepted")
	}
	if strings.Contains(ctl.joined(), "cmd ") {
		t.Errorf("a refused start reached the control service:\n%s", ctl.joined())
	}
}

func TestStopAndSettleDrainsBothStreams(t *testing.T) {
	b, rt, ctl, unit, u, out, errs := startTestCmd(t)
	unit.set("active", "inv1", true, true)
	write := func(path, s string) {
		c, err := net.Dial("unix", path)
		if err != nil {
			t.Fatal(err)
		}
		if _, err := c.Write([]byte(s)); err != nil {
			t.Fatal(err)
		}
		c.Close()
	}
	write(b.logSocketNamed(rt.Host, string(instancectl.RoleCmd)), "out-1\nout-2")
	write(b.logSocketNamed(rt.Host, cmdErrSocket), "err-1")
	go func() {
		time.Sleep(30 * time.Millisecond)
		unit.set("inactive", "inv1", false, false)
	}()
	if err := u.StopAndSettle(context.Background()); err != nil {
		t.Fatalf("StopAndSettle: %v", err)
	}
	if out.String() != "out-1\nout-2" || errs.String() != "err-1" {
		t.Errorf("stdout %q stderr %q", out.String(), errs.String())
	}
	if !strings.Contains(ctl.joined(), "stop "+rt.Host+" cmd") {
		t.Errorf("the cmd unit was not stopped:\n%s", ctl.joined())
	}
	if err := u.StopAndSettle(context.Background()); err != nil {
		t.Errorf("a second StopAndSettle: %v", err)
	}
}

b.layoutFn does not exist on ExecBackend yet: Step 3 adds the seam (nil keeps Classify). nestedBackend (unitproc_test.go) makes its root with os.MkdirTemp("", "nb"), so the stream sockets need TMPDIR=/tmp/g.

• [ ] Step 2: Run to verify they fail

Run: TMPDIR=/tmp/g nice -n 10 go test ./internal/activator/ -run 'TestCmdVars|TestStartCmd|TestStopAndSettle' -count=1 Expected: FAIL, b.cmdVars undefined.

• [ ] Step 3: Implement

control.go, the interface:

type Control interface {
	Start(ctx context.Context, host string, role instancectl.Role, memoryMax int64) (string, error)
	Seed(ctx context.Context, host string, memoryMax int64) (instancectl.SeedResponse, error)
	// Cmd starts host's cmd unit and returns its invocation; CmdStatus
	// long-polls it (run spec §5.4).
	Cmd(ctx context.Context, host string, memoryMax int64) (string, error)
	CmdStatus(ctx context.Context, host, invocation string) (instancectl.CmdStatus, error)
	Stop(ctx context.Context, host string, role instancectl.Role) error
	Reset(ctx context.Context, host string, role instancectl.Role) error
	Sweep(ctx context.Context) (instancectl.SweepResponse, error)
	Ready(ctx context.Context) error
}

prepare.go:44: the instance environment for the cmd role too:

	if role == instancectl.RoleRun || role == instancectl.RoleSeed || role == instancectl.RoleCmd {

prepare.go:93-99: no app-log listener for cmd (its output goes only to its own chunks, never to the journal or the app log):

	var logs *logListener
	if role != instancectl.RoleCmd {
		var err error
		logs, err = listenLogs(b.logSocket(rt.Host, role), b.logEmitter(rt))
		if err != nil {
			if ctl != nil {
				ctl.Close()
			}
			return p, nil, nil, nil, fmt.Errorf("log listener for %s %s: %w", rt.Host, role, err)
		}
	}
	return p, ctl, logs, env, nil

prepare.go:131-137:

// logSocket is the role's log socket path, honouring the test override of
// the run directory (RunDirOverride).
func (b *ExecBackend) logSocket(host string, role instancectl.Role) string {
	return b.logSocketNamed(host, string(role))
}

// logSocketNamed is logSocket for any listener name: the cmd role has two,
// "cmd" and "cmd-err" (run spec §5.4).
func (b *ExecBackend) logSocketNamed(host, name string) string {
	if b.RunDirOverride != "" {
		return filepath.Join(b.RunDirOverride, "logs", provision.InstanceID(host)+"."+name+".sock")
	}
	return provision.LogSocket(host, name)
}

backend_exec.go:1024-1056: the body moves to a role-taking function and the seed wrapper calls it:

func (b *ExecBackend) waitSeedCgroupEmpty(ctx context.Context, rt registry.Route) error {
	return b.waitRoleCgroupEmpty(ctx, rt, instancectl.RoleSeed)
}

// waitRoleCgroupEmpty polls role's unit's cgroup.events until it reports
// empty or actStopTimeout elapses, honouring ctx and b.ObservePoll.
func (b *ExecBackend) waitRoleCgroupEmpty(ctx context.Context, rt registry.Route, role instancectl.Role) error {
	events := b.CgroupEvents
	if events == nil {
		events = cgroupEventsUnder(b.cgroupRoot())
	}
	poll := b.ObservePoll
	if poll <= 0 {
		poll = time.Second
	}
	slice := instancectl.SliceName(rt.Host)
	unit := instancectl.UnitName(role, rt.Host)
	deadline := time.Now().Add(actStopTimeout)
	for {
		exists, populated, err := events(slice, unit)
		if err != nil {
			return err
		}
		if !exists || !populated {
			return nil
		}
		if time.Now().After(deadline) {
			return fmt.Errorf("processes remained after %s", actStopTimeout)
		}
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(poll):
		}
	}
}

ExecBackend gains a test seam beside RunDirOverride (backend_exec.go, in the struct):

	// layoutFn, when set, replaces Classify in StartCmd's layout check; a
	// unit test's way to stand a temp directory in for a nested host.
	layoutFn func(rt registry.Route) (Layout, string)

internal/activator/cmdunit.go:

package activator

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log"
	"net"
	"path/filepath"
	"sync"
	"sync/atomic"
	"time"

	"github.com/carlosframework/platform/internal/edge"
	"github.com/carlosframework/platform/internal/instancectl"
	"github.com/carlosframework/platform/internal/registry"
)

// CmdRunDirName is the directory under an instance's own data directory
// that holds each run's --file uploads (run spec §5.2 step 3).
const CmdRunDirName = ".carlos-run"

// cmdErrSocket is the stderr listener's name; stdout's is the role's.
const cmdErrSocket = "cmd-err"

// cmdDrainBound is how long a stream may take to reach EOF once its unit
// has stopped.
const cmdDrainBound = 5 * time.Second

// ErrCmdNotNested: the instance is not on the nested layout, so a command
// would run with the agent's reach (run spec §5.2 step 1, refused "flat").
var ErrCmdNotNested = errors.New("activator: the instance is not on the nested layout")

// CmdSpec is one run's launch, as the box's run pass hands it over.
type CmdSpec struct {
	RunID string
	// Args is exactly what followed `--`; the binary is the host's live link.
	Args []string
	// RunDir is CARLOS_RUN_DIR: <instance dir>/.carlos-run/<RunID>, made and
	// filled by the caller.
	RunDir string
	// Stdout and Stderr receive the unit's two streams, byte for byte.
	Stdout, Stderr io.Writer
	// Prepare runs after the run is attached to the awake entry and before
	// the unit starts (Service.RunCmd); an error ends the run before launch.
	Prepare func(ctx context.Context) error
}

// CmdUnit is a started cmd unit.
type CmdUnit interface {
	Invocation() string
	// Poll long-polls the invocation, at most about 30 seconds.
	Poll(ctx context.Context) (instancectl.CmdStatus, error)
	// StopAndSettle stops the unit (SIGTERM, then SIGKILL after 20 s),
	// proves its cgroup empty and reads both streams to their end. It is
	// safe to call more than once; later calls return the first's error.
	StopAndSettle(ctx context.Context) error
}

// CmdBackend is the backend capability Service.RunCmd needs.
type CmdBackend interface {
	StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error)
}

var _ CmdBackend = (*ExecBackend)(nil)

// cmdVars is a run's environment over fileEnv (run spec §4, codex 7): the
// control-channel variables stripped as seedVars strips them, CARLOS_REGION
// set or removed from the backend's region as agentOwnedVars does,
// CARLOS_HOST from the row, and the three run variables. fileEnv carries
// CARLOS_ADMIN_TOKEN through.
func (b *ExecBackend) cmdVars(rt registry.Route, fileEnv map[string]string, dbPath, runID, runDir string) map[string]string {
	v := withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE")
	if b.Region != "" {
		v = withVar(v, "CARLOS_REGION", b.Region)
	} else {
		v = withoutVars(v, "CARLOS_REGION")
	}
	v = withVar(v, "CARLOS_HOST", rt.Host)
	v = withVar(v, "CARLOS_DB", dbPath)
	v = withVar(v, "CARLOS_RUN_ID", runID)
	return withVar(v, "CARLOS_RUN_DIR", runDir)
}

func (b *ExecBackend) cmdLayout(rt registry.Route) (Layout, string) {
	if b.layoutFn != nil {
		return b.layoutFn(rt)
	}
	return b.Layout(rt)
}

// StartCmd starts one run's cmd unit (run spec §5.4): the layout check, the
// nested gate, the live link, the instance's directories and environment,
// the two stream listeners, launch.cmd.json, then POST /cmd. It returns
// once the unit has an invocation. The environment is never logged.
func (b *ExecBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
	if l, why := b.cmdLayout(rt); l != LayoutNested {
		return nil, fmt.Errorf("%w: %s is %s (%s)", ErrCmdNotNested, rt.Host, l, why)
	}
	if err := b.nestedGate(rt); err != nil {
		return nil, err
	}
	if err := b.ensureLiveLink(ctx, rt.Host); err != nil {
		return nil, err
	}
	p, _, _, fileEnv, err := b.prepare(rt, instancectl.RoleCmd)
	if err != nil {
		return nil, err
	}
	if want := filepath.Join(p.dataDir, CmdRunDirName, spec.RunID); spec.RunDir != want {
		return nil, fmt.Errorf("%s: run directory %q is not %q", rt.Host, spec.RunDir, want)
	}
	out, err := listenCmdStream(b.logSocketNamed(rt.Host, string(instancectl.RoleCmd)), spec.Stdout)
	if err != nil {
		return nil, fmt.Errorf("stdout listener for %s: %w", rt.Host, err)
	}
	errs, err := listenCmdStream(b.logSocketNamed(rt.Host, cmdErrSocket), spec.Stderr)
	if err != nil {
		out.Close()
		return nil, fmt.Errorf("stderr listener for %s: %w", rt.Host, err)
	}
	fail := func(err error) (CmdUnit, error) {
		out.Close()
		errs.Close()
		return nil, err
	}
	argv := append([]string{b.liveLink(rt.Host)}, spec.Args...)
	env := childEnv(b.cmdVars(rt, fileEnv, p.dbPath, spec.RunID, spec.RunDir))
	if err := b.writeLaunch(p, instancectl.RoleCmd, rt.Host, argv, env); err != nil {
		return fail(err)
	}
	inv, err := b.Control.Cmd(ctx, rt.Host, b.tier(rt).MemoryMax)
	if err != nil {
		return fail(fmt.Errorf("%s: start cmd unit: %w", rt.Host, err))
	}
	log.Printf("activator: %s: started %s for run %s", rt.Host, instancectl.UnitName(instancectl.RoleCmd, rt.Host), spec.RunID)
	return &cmdUnit{b: b, rt: rt, inv: inv, out: out, errs: errs}, nil
}

type cmdUnit struct {
	b         *ExecBackend
	rt        registry.Route
	inv       string
	out, errs *cmdStream
	once      sync.Once
	err       error
}

func (u *cmdUnit) Invocation() string { return u.inv }

func (u *cmdUnit) Poll(ctx context.Context) (instancectl.CmdStatus, error) {
	return u.b.Control.CmdStatus(ctx, u.rt.Host, u.inv)
}

func (u *cmdUnit) StopAndSettle(ctx context.Context) error {
	u.once.Do(func() {
		sctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), instancectl.DeadlineControl)
		stopErr := u.b.Control.Stop(sctx, u.rt.Host, instancectl.RoleCmd)
		cancel()
		wctx, wcancel := context.WithTimeout(context.WithoutCancel(ctx), actStopTimeout)
		emptyErr := u.b.waitRoleCgroupEmpty(wctx, u.rt, instancectl.RoleCmd)
		wcancel()
		u.err = errors.Join(stopErr, emptyErr, u.out.Drain(cmdDrainBound), u.errs.Drain(cmdDrainBound))
	})
	return u.err
}

// cmdStream is the agent's end of one cmd stream (run spec §5.5): the
// unit's StandardOutput=file: or StandardError=file: connects once, at
// exec, and every byte is copied to w unchanged until EOF. Nothing goes to
// the journal or the app log. One connection is accepted; a second is
// closed unread.
type cmdStream struct {
	ln       net.Listener
	w        io.Writer
	accepted atomic.Bool
	closed   atomic.Bool
	mu       sync.Mutex
	conn     net.Conn
	copied   chan struct{}
	wg       sync.WaitGroup
}

func listenCmdStream(path string, w io.Writer) (*cmdStream, error) {
	ln, err := edge.SecureUnixListen(path)
	if err != nil {
		return nil, err
	}
	s := &cmdStream{ln: ln, w: w, copied: make(chan struct{})}
	s.wg.Add(1)
	go s.accept()
	return s, nil
}

func (s *cmdStream) accept() {
	defer s.wg.Done()
	for {
		c, err := s.ln.Accept()
		if err != nil {
			return
		}
		if s.accepted.Swap(true) {
			c.Close()
			continue
		}
		s.mu.Lock()
		if s.closed.Load() {
			s.mu.Unlock()
			c.Close()
			continue
		}
		s.conn = c
		s.wg.Add(1)
		s.mu.Unlock()
		go s.copy(c)
	}
}

func (s *cmdStream) copy(c net.Conn) {
	defer s.wg.Done()
	defer close(s.copied)
	defer c.Close()
	_, _ = io.Copy(s.w, c)
}

// Drain waits up to bound for the stream to reach EOF (its unit has
// stopped, so every writer has closed), then closes the listener. A stream
// that was never connected has nothing to drain.
func (s *cmdStream) Drain(bound time.Duration) error {
	var err error
	if s.accepted.Load() {
		select {
		case <-s.copied:
		case <-time.After(bound):
			err = fmt.Errorf("a cmd stream did not reach its end within %s", bound)
		}
	}
	s.Close()
	return err
}

func (s *cmdStream) Close() error {
	if s.closed.Swap(true) {
		return nil
	}
	err := s.ln.Close()
	s.mu.Lock()
	if s.conn != nil {
		s.conn.Close()
	}
	s.mu.Unlock()
	s.wg.Wait()
	return err
}

backend_unit.go, after StartWorker:

// StartCmd dispatches like every other call; a unit-backed row has no
// nested instance to run a command in.
func (b *SplitBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
	if c, ok := b.pick(rt).(CmdBackend); ok {
		return c.StartCmd(ctx, rt, spec)
	}
	return nil, fmt.Errorf("%w: %s is unit-backed", ErrCmdNotNested, rt.Host)
}

(Add "fmt" to backend_unit.go's imports if it lacks it.)

• [ ] Step 4: Run to verify they pass

Run: TMPDIR=/tmp/g nice -n 10 go test ./internal/activator/ -run 'TestCmdVars|TestStartCmd|TestStopAndSettle|TestStartUnit|TestSeed' -count=1 && GOOS=windows go build ./internal/activator/ && GOOS=darwin go build ./internal/activator/ Expected: PASS; both cross builds succeed (edge.SecureUnixListen has a windows body).

• [ ] Step 5: Commit

git add internal/activator/cmdunit.go internal/activator/cmdunit_test.go internal/activator/control.go internal/activator/prepare.go internal/activator/backend_exec.go internal/activator/backend_unit.go internal/activator/unitproc_test.go
git commit -m "activator: StartCmd, cmdVars and the two cmd streams (run spec §4, §5.4, §5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 7: Every stop says why

A run interrupted by an instance stop ends interrupted: "<reason>" (spec §5.3). The reason is decided where the stop is decided, so beginStopLocked takes it.

Files:

• Create: internal/activator/cmd.go (the reasons; Task 8 adds the rest)
• Modify: internal/activator/activator.go:346-457 (actEntry.stopWhy), :1373 (beginStopLocked), and its 17 callers; internal/activator/maintenance.go:133
• Test: internal/activator/cmd_test.go (new)

Interfaces:

• Produces: StopIdle, StopRestart, StopRelease, StopExited, StopLeaseLost, StopFenced, StopRetired, StopShutdown (string constants, RUN-C1 to RUN-C8); func (a *Service) beginStopLocked(e *actEntry, why string); actEntry.stopWhy string, written under a.mu.

• [ ] Step 1: Write the failing test

internal/activator/cmd_test.go:

package activator

import (
	"context"
	"testing"
	"time"
)

func (a *Service) stopWhyOf(host string) string {
	a.mu.Lock()
	defer a.mu.Unlock()
	if e := a.entries[host]; e != nil {
		return e.stopWhy
	}
	return ""
}

func TestStopsRecordTheirReason(t *testing.T) {
	a, b, _, _ := testActivator(t, time.Second)
	rt := testRoute("why.example.com")
	arrangeAwake(t, a, rt)
	gate := make(chan struct{})
	b.lastProc().setStopGate(gate) // hold the entry in stopping
	defer close(gate)
	a.Bounce(rt.Host)
	if got := a.stopWhyOf(rt.Host); got != StopRestart {
		t.Fatalf("Bounce: stopWhy %q, want %q", got, StopRestart)
	}
}

(setStopGate(gate chan struct{}) is at activator_test.go:143; drop the context import this file does not need yet.)

• [ ] Step 2: Run to verify it fails

Run: nice -n 10 go test ./internal/activator/ -run TestStopsRecordTheirReason -count=1 Expected: FAIL, undefined: StopRestart.

• [ ] Step 3: Implement

internal/activator/cmd.go:

package activator

// Why an instance stop interrupted a run (run spec §5.3). The text is what
// the member reads after "the run was interrupted:" (RUN-C1 to RUN-C8);
// beginStopLocked records one on every stop.
const (
	StopIdle      = "the instance hibernated"
	StopRestart   = "the instance was restarted"
	StopRelease   = "the instance moved to a new release"
	StopExited    = "the instance's process exited"
	StopLeaseLost = "the instance lost its database lease"
	StopFenced    = "the instance was stopped for a move or a layout change"
	StopRetired   = "the instance was taken off this box"
	StopShutdown  = "the box's agent stopped"
)

actEntry gains, after stopOutcome:

	// stopWhy is why this entry is stopping (beginStopLocked), written under
	// a.mu; finishStop hands it to a run it interrupts (run spec §5.3).
	stopWhy string

activator.go:1373:

func (a *Service) beginStopLocked(e *actEntry, why string) {
	e.state = actStopping
	e.stopWhy = why
	e.stopped = make(chan struct{})
	go a.stop(e)
}

Every caller passes its reason (grep -n 'beginStopLocked(' internal/activator/*.go; the compiler refuses any left out):

| site | function | reason | |---|---|---| | activator.go:765 | begin | StopLeaseLost when repErr != nil, else StopExited (compute why inside the if beside the log line) | | activator.go:850 | BeginSidecarRequest | as :765 | | activator.go:997 | wake, fence installed while waking | StopFenced | | activator.go:1453 | bounceLocked (Bounce, BounceAdopted, the deferred bounce in wake) | StopRestart | | activator.go:1566 | ConvergeAdopted | StopRelease | | activator.go:1721 | WakeWorker | as :765 | | activator.go:1957 | respawnWorkerLocked, refused | StopRestart | | activator.go:2072 | SleepWorker | StopIdle | | activator.go:2465 | sweepOnce, lease lost | StopLeaseLost | | activator.go:2507 | sweepOnce, process exited | StopExited | | activator.go:2547 | sweepOnce, idle | StopIdle | | activator.go:2574 | sweepOnce, worker max_run | StopIdle | | activator.go:2611 | sweepOnce, convergence | StopRelease | | activator.go:2841, :2861 | RetireExcept | StopRetired | | activator.go:2925 | Shutdown | StopShutdown | | maintenance.go:133 | fenceAndStop | StopFenced |

The :765 shape, for example:

			if repErr := e.rep.Err(); repErr != nil || e.proc.Exited() {
				why := StopExited
				if repErr != nil {
					why = StopLeaseLost
					log.Printf("activator: LEASE LOST for %s (%v) — standing down", e.host, repErr)
				} else {
					log.Printf("activator: instance %s exited unexpectedly — recycling", e.host)
				}
				a.beginStopLocked(e, why)

(The log lines are main's bytes, journal-only, and stay as they are.) Test files that construct entries or call beginStopLocked directly get a reason too: grep -n 'beginStopLocked(' internal/activator/*_test.go.

runCopyBuilt gains "RUN-C1", "RUN-C2", "RUN-C3", "RUN-C4", "RUN-C5", "RUN-C6", "RUN-C7", "RUN-C8".

• [ ] Step 4: Run to verify it passes

Run: nice -n 10 go test ./internal/activator/ -count=1 && nice -n 10 go test . -run TestRunBuilt -count=1 Expected: PASS (the whole package: every stop path was edited).

• [ ] Step 5: Commit

git add internal/activator/cmd.go internal/activator/cmd_test.go internal/activator/activator.go internal/activator/maintenance.go internal/activator/*_test.go approved_run_copy_test.go
git commit -m "activator: every stop records why (run spec §5.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 8: The run joins the writer lifecycle: cmd, RunCmd, finishStop, the idle hold

Files:

• Modify: internal/activator/cmd.go (append), internal/activator/activator.go (actEntry.cmd, finishStop at :2355, the idle check at :2533)
• Test: internal/activator/cmd_test.go (append)

Interfaces:

• Consumes: a.begin(ctx, rt, stamp bool) (func(), error) (:725), errWakeTimeout, memberSafeWakeReason(err) string (logsink.go:49), a.cfg.HostWrites.Enabled, Classifier, CmdBackend, CmdSpec, CmdUnit, StopIdle.
• Produces: - type CmdHandle interface { Invocation() string; Poll(ctx) (instancectl.CmdStatus, error); Interrupted() string; Stop(ctx) error; Release() }. - func (a *Service) RunCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdHandle, error). - Errors: ErrCmdClaim, ErrCmdBusy, ErrCmdNotNested (Task 6), *CmdWakeError{Reason string; Err error} (Reason is member-safe), *CmdStoppedError{Reason string}, and whatever spec.Prepare returned, unwrapped. - cmdWakeBound = 60 * time.Second (a var, for tests). - actEntry.cmd *cmdHold; while it is set the idle sweep never stops the entry; every stop path stops and settles its unit before rep.Close, and an unsettled unit marks the stop uncertain.

• [ ] Step 1: Write the failing tests

Append to cmd_test.go (its imports grow to context, errors, path/filepath, sync, testing, time, and the repo's hostwrite, instancectl and registry):

// fakeCmdUnit records the stop and whether replication was already closed
// when it came.
type fakeCmdUnit struct {
	mu            sync.Mutex
	stops         int
	repWasClosed  bool
	settleErr     error
	rep           func() *fakeRep
	status        instancectl.CmdStatus
}

func (u *fakeCmdUnit) Invocation() string { return "inv1" }
func (u *fakeCmdUnit) Poll(context.Context) (instancectl.CmdStatus, error) {
	u.mu.Lock()
	defer u.mu.Unlock()
	return u.status, nil
}
func (u *fakeCmdUnit) StopAndSettle(context.Context) error {
	u.mu.Lock()
	defer u.mu.Unlock()
	u.stops++
	if u.rep != nil {
		u.repWasClosed = u.rep().isClosed()
	}
	return u.settleErr
}
func (u *fakeCmdUnit) stopped() (int, bool) {
	u.mu.Lock()
	defer u.mu.Unlock()
	return u.stops, u.repWasClosed
}

// cmdFakeBackend is fakeBackend with the run capability and a nested layout.
type cmdFakeBackend struct {
	*fakeBackend
	cmdMu     sync.Mutex
	unit      *fakeCmdUnit
	startErr  error
	cmdStarts int
	gate      chan struct{} // when set, StartCmd blocks until it closes
}

func (b *cmdFakeBackend) StartCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdUnit, error) {
	if b.gate != nil {
		<-b.gate
	}
	b.cmdMu.Lock()
	defer b.cmdMu.Unlock()
	b.cmdStarts++
	if b.startErr != nil {
		return nil, b.startErr
	}
	return b.unit, nil
}

func testCmdActivator(t *testing.T, cfg Config) (*Service, *cmdFakeBackend, *fakeClock) {
	t.Helper()
	fb := &fakeBackend{layoutFn: func(registry.Route) (Layout, string) { return LayoutNested, "" }}
	cb := &cmdFakeBackend{fakeBackend: fb}
	cb.unit = &fakeCmdUnit{rep: fb.lastRep}
	clk := &fakeClock{t: time.Unix(1_700_000_000, 0)}
	reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
	if err != nil {
		t.Fatal(err)
	}
	if cfg.Bucket == "" {
		cfg.Bucket = "test-bucket"
	}
	if cfg.Idle == 0 {
		cfg.Idle = time.Second
	}
	a := New(cfg, cb, reg)
	a.now = clk.now
	return a, cb, clk
}

func TestRunCmdRefusesWithClaimControlArmed(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{HostWrites: hostwrite.Config{Enabled: true}})
	if _, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdClaim) {
		t.Fatalf("err = %v, want ErrCmdClaim", err)
	}
	if acquires, _, _ := cb.counts(); acquires != 0 {
		t.Fatal("a claim-armed refusal woke the instance")
	}
}

func TestRunCmdRefusesAFlatInstance(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{})
	cb.layoutFn = func(registry.Route) (Layout, string) { return LayoutFlat, "" }
	if _, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"}); !errors.Is(err, ErrCmdNotNested) {
		t.Fatalf("err = %v, want ErrCmdNotNested", err)
	}
}

func TestRunCmdWakesAHibernatedInstanceFirst(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{})
	h, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"})
	if err != nil {
		t.Fatal(err)
	}
	if _, _, wakes := cb.counts(); h.Invocation() != "inv1" || wakes != 1 || cb.cmdStarts != 1 {
		t.Fatalf("wake starts %d, cmd starts %d", wakes, cb.cmdStarts)
	}
	h.Release()
}

func TestRunCmdWakeFailureIsAWakeRefusal(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{})
	cb.acquireErr = errors.New("restore failed")
	_, err := a.RunCmd(context.Background(), testRoute("c.example.com"), CmdSpec{RunID: "run_a"})
	var we *CmdWakeError
	if !errors.As(err, &we) || we.Reason == "" {
		t.Fatalf("err = %v, want a CmdWakeError with a member-safe reason", err)
	}
	if cb.cmdStarts != 0 {
		t.Fatal("a failed wake started the command")
	}
}

func TestRunCmdRefusesASecondRun(t *testing.T) {
	a, _, _ := testCmdActivator(t, Config{})
	rt := testRoute("c.example.com")
	h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
	if err != nil {
		t.Fatal(err)
	}
	defer h.Release()
	if _, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_b"}); !errors.Is(err, ErrCmdBusy) {
		t.Fatalf("err = %v, want ErrCmdBusy", err)
	}
}

func TestRunCmdHoldsTheIdleTimer(t *testing.T) {
	a, _, clk := testCmdActivator(t, Config{})
	rt := testRoute("c.example.com")
	h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
	if err != nil {
		t.Fatal(err)
	}
	clk.advance(time.Hour)
	a.sweepOnce()
	if a.entryCount() != 1 || a.stopWhyOf(rt.Host) != "" {
		t.Fatal("the idle timer stopped an instance with a run open")
	}
	h.Release()
	clk.advance(time.Hour)
	a.sweepOnce()
	waitFor(t, "hibernation after the run", func() bool { return a.entryCount() == 0 })
}

// Every stop path stops and settles the run before replication closes, and
// the run learns why (run spec §5.3, codex 1).
func TestEveryStopPathSettlesTheRunBeforeReplicationCloses(t *testing.T) {
	cases := []struct {
		name string
		why  string
		stop func(a *Service, cb *cmdFakeBackend, host string)
	}{
		{"bounce", StopRestart, func(a *Service, _ *cmdFakeBackend, h string) { a.Bounce(h) }},
		{"process exit", StopExited, func(a *Service, cb *cmdFakeBackend, _ string) { cb.lastProc().exit(); a.sweepOnce() }},
		{"lease lost", StopLeaseLost, func(a *Service, cb *cmdFakeBackend, _ string) { cb.lastRep().setErr(errors.New("lost")); a.sweepOnce() }},
		{"convergence", StopRelease, func(a *Service, _ *cmdFakeBackend, h string) {
			a.liveVer = func(string) (string, error) { return "v2", nil }
			a.ConvergeAdopted(h, "v2")
		}},
		{"move fence", StopFenced, func(a *Service, _ *cmdFakeBackend, h string) {
			go a.FenceAndStop(context.Background(), h, 1) //nolint:errcheck
		}},
		{"retire", StopRetired, func(a *Service, _ *cmdFakeBackend, _ string) { a.RetireExcept(map[string]bool{}) }},
		{"agent shutdown", StopShutdown, func(a *Service, _ *cmdFakeBackend, _ string) {
			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
			defer cancel()
			a.Shutdown(ctx)
		}},
	}
	for _, c := range cases {
		t.Run(c.name, func(t *testing.T) {
			a, cb, _ := testCmdActivator(t, Config{})
			cb.execVerFn = func() string { return "v1" }
			rt := testRoute("c.example.com")
			h, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
			if err != nil {
				t.Fatal(err)
			}
			c.stop(a, cb, rt.Host)
			waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
			stops, repWasClosed := cb.unit.stopped()
			if stops != 1 {
				t.Fatalf("the cmd unit was stopped %d times, want 1", stops)
			}
			if repWasClosed {
				t.Fatal("replication closed before the run settled")
			}
			if !cb.lastRep().isClosed() {
				t.Fatal("replication never closed")
			}
			if got := h.Interrupted(); got != c.why {
				t.Fatalf("Interrupted() = %q, want %q", got, c.why)
			}
		})
	}
}

func TestRunCmdNeverStartsUnderAStoppingEntry(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{})
	rt := testRoute("c.example.com")
	_, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a", Prepare: func(context.Context) error {
		a.Bounce(rt.Host) // the instance starts stopping while the files download
		return nil
	}})
	var se *CmdStoppedError
	if !errors.As(err, &se) || se.Reason != StopRestart {
		t.Fatalf("err = %v, want CmdStoppedError{%q}", err, StopRestart)
	}
	if cb.cmdStarts != 0 {
		t.Fatal("the cmd unit started under a stopping entry")
	}
}

func TestAStopDuringTheStartWaitsForIt(t *testing.T) {
	a, cb, _ := testCmdActivator(t, Config{})
	cb.gate = make(chan struct{})
	rt := testRoute("c.example.com")
	done := make(chan error, 1)
	go func() {
		_, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"})
		done <- err
	}()
	waitFor(t, "the run attached", func() bool {
		a.mu.Lock()
		defer a.mu.Unlock()
		e := a.entries[rt.Host]
		return e != nil && e.cmd != nil
	})
	a.Bounce(rt.Host)
	time.Sleep(50 * time.Millisecond)
	if cb.lastRep().isClosed() {
		t.Fatal("replication closed while the cmd unit's start was still in flight")
	}
	close(cb.gate)
	if err := <-done; err != nil {
		t.Fatalf("RunCmd: %v", err)
	}
	waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
	if stops, repWasClosed := cb.unit.stopped(); stops != 1 || repWasClosed {
		t.Fatalf("stops %d, replication closed first %v", stops, repWasClosed)
	}
}

func TestAnUnsettledRunMakesTheStopUncertain(t *testing.T) {
	for _, c := range []struct {
		settleErr error
		uncertain bool
	}{{nil, false}, {errors.New("processes remained"), true}} {
		a, cb, _ := testCmdActivator(t, Config{})
		cb.unit.settleErr = c.settleErr
		rt := testRoute("c.example.com")
		if _, err := a.RunCmd(context.Background(), rt, CmdSpec{RunID: "run_a"}); err != nil {
			t.Fatal(err)
		}
		a.Bounce(rt.Host)
		waitFor(t, "the instance stopped", func() bool { return a.entryCount() == 0 })
		if got := a.StopUncertain(rt.Host); got != c.uncertain {
			t.Fatalf("settle error %v: StopUncertain = %v, want %v", c.settleErr, got, c.uncertain)
		}
	}
}

counts() answers (acquires, restores, starts int) (activator_test.go:563); StopUncertain(host string) bool is maintenance.go:225, and the fakes' ordinary stop is a complete one (fakeProc.Stop sets dead), which is why the settled case must read false.

• [ ] Step 2: Run to verify they fail

Run: nice -n 10 go test ./internal/activator/ -run 'TestRunCmd|TestEveryStopPath|TestAStopDuringTheStart|TestAnUnsettledRun' -count=1 Expected: FAIL, a.RunCmd undefined.

• [ ] Step 3: Implement

actEntry gains, after stopWhy:

	// cmd is the run attached to this instance (run spec §5.3), written
	// under a.mu. While it is set the idle sweep never stops the entry;
	// every other stop stops and settles its unit in finishStop, before
	// replication closes.
	cmd *cmdHold

Append to cmd.go (imports: context, errors, sync, time, instancectl, registry):

// ErrCmdClaim: claim control is armed on this box. Runs need the isolated
// root protocol's reservations (#250), so they are refused (run spec §3.1).
var ErrCmdClaim = errors.New("activator: claim control is armed; runs wait for the reservation protocol (#250)")

// ErrCmdBusy: a run is already attached to the instance.
var ErrCmdBusy = errors.New("activator: a run is already attached to this instance")

// CmdWakeError: the wake failed or did not finish within cmdWakeBound.
// Reason is member-safe (memberSafeWakeReason).
type CmdWakeError struct {
	Reason string
	Err    error
}

func (e *CmdWakeError) Error() string { return e.Reason }
func (e *CmdWakeError) Unwrap() error { return e.Err }

// CmdStoppedError: the instance began stopping before the unit started. The
// run is interrupted with Reason, never queued for later (run spec §5.3).
type CmdStoppedError struct{ Reason string }

func (e *CmdStoppedError) Error() string {
	return "activator: the instance stopped before the run started: " + e.Reason
}

// cmdWakeBound is how long RunCmd waits for a wake (run spec §5.3).
var cmdWakeBound = 60 * time.Second

// cmdStopBound bounds finishStop's wait for a run: the start in flight
// (DeadlineControl), the stop and the cgroup (actStopTimeout) and both
// streams (cmdDrainBound), with room.
const cmdStopBound = 2 * time.Minute

// CmdHandle is a started run, as the box's run pass sees it.
type CmdHandle interface {
	Invocation() string
	Poll(ctx context.Context) (instancectl.CmdStatus, error)
	// Interrupted is the instance stop's reason when one stopped the unit,
	// "" otherwise.
	Interrupted() string
	// Stop stops the unit and waits for it to settle and its streams to
	// end, leaving the instance running: a cancel, a timeout, or the end
	// of a command that exited. Safe to call more than once.
	Stop(ctx context.Context) error
	// Release clears the run from the entry once its result is written.
	Release()
}

// cmdHold is the entry's cmd field.
type cmdHold struct {
	runID    string
	started  chan struct{} // closed once StartCmd returned, either way
	unit     CmdUnit       // written before started is closed; nil on a failed start
	mu       sync.Mutex
	reason   string
	stopOnce sync.Once
	settled  chan struct{}
	stopErr  error
}

func newCmdHold(runID string) *cmdHold {
	return &cmdHold{runID: runID, started: make(chan struct{}), settled: make(chan struct{})}
}

// stop waits for the start to return, records reason (the first one
// wins; "" records none), and stops and settles the unit once.
func (h *cmdHold) stop(ctx context.Context, reason string) error {
	select {
	case <-h.started:
	case <-ctx.Done():
		return ctx.Err()
	}
	h.mu.Lock()
	if reason != "" && h.reason == "" {
		h.reason = reason
	}
	h.mu.Unlock()
	if h.unit == nil {
		return nil
	}
	h.stopOnce.Do(func() {
		go func() {
			h.stopErr = h.unit.StopAndSettle(context.Background())
			close(h.settled)
		}()
	})
	select {
	case <-h.settled:
		return h.stopErr
	case <-ctx.Done():
		return ctx.Err()
	}
}

type cmdRun struct {
	a    *Service
	e    *actEntry
	h    *cmdHold
	once sync.Once
}

func (r *cmdRun) Invocation() string { return r.h.unit.Invocation() }
func (r *cmdRun) Poll(ctx context.Context) (instancectl.CmdStatus, error) {
	return r.h.unit.Poll(ctx)
}
func (r *cmdRun) Interrupted() string {
	r.h.mu.Lock()
	defer r.h.mu.Unlock()
	return r.h.reason
}
func (r *cmdRun) Stop(ctx context.Context) error { return r.h.stop(ctx, "") }

// Release clears the hold and stamps activity, so the instance idles a
// full window after a run rather than hibernating at the next sweep.
func (r *cmdRun) Release() {
	r.once.Do(func() {
		r.a.mu.Lock()
		if r.e.cmd == r.h {
			r.e.cmd = nil
			r.e.lastActive = r.a.now()
		}
		r.a.mu.Unlock()
	})
}

// RunCmd runs one command against rt's instance (run spec §5.2 and §5.3):
// refused with claim control armed or on a layout other than nested; the
// instance woken through the ordinary wake path, waiting up to 60 s; the
// run attached to the awake entry; spec.Prepare run under the hold; and
// the unit started only while the entry is still awake. A stop that begins
// after the attach is either seen here (CmdStoppedError, no unit) or
// waits in finishStop for the start to return and then stops the unit
// before replication closes.
func (a *Service) RunCmd(ctx context.Context, rt registry.Route, spec CmdSpec) (CmdHandle, error) {
	if a.cfg.HostWrites.Enabled {
		return nil, ErrCmdClaim
	}
	cb, ok := a.backend.(CmdBackend)
	if !ok {
		return nil, ErrCmdNotNested
	}
	if c, ok := a.backend.(Classifier); !ok {
		return nil, ErrCmdNotNested
	} else if l, _ := c.Layout(rt); l != LayoutNested {
		return nil, ErrCmdNotNested
	}
	wctx, cancel := context.WithTimeout(ctx, cmdWakeBound)
	defer cancel()
	var release func()
	for {
		r, err := a.begin(wctx, rt, true)
		if err == nil {
			release = r
			break
		}
		if errors.Is(err, errWakeTimeout) && wctx.Err() == nil {
			continue // begin's own bound is 30 s; a run waits 60
		}
		if wctx.Err() != nil && ctx.Err() == nil {
			err = errWakeTimeout
		}
		return nil, &CmdWakeError{Reason: memberSafeWakeReason(err), Err: err}
	}
	h := newCmdHold(spec.RunID)
	a.mu.Lock()
	e := a.entries[rt.Host]
	switch {
	case e == nil || e.state != actAwake:
		why := StopIdle
		if e != nil && e.stopWhy != "" {
			why = e.stopWhy
		}
		a.mu.Unlock()
		release()
		return nil, &CmdStoppedError{Reason: why}
	case e.cmd != nil:
		a.mu.Unlock()
		release()
		return nil, ErrCmdBusy
	}
	e.cmd = h
	a.mu.Unlock()
	release() // the hold, not the in-flight count, keeps the idle timer off now
	run := &cmdRun{a: a, e: e, h: h}
	abandon := func(err error) (CmdHandle, error) {
		close(h.started)
		run.Release()
		return nil, err
	}
	if spec.Prepare != nil {
		if err := spec.Prepare(ctx); err != nil {
			return abandon(err)
		}
	}
	a.mu.Lock()
	if e.state != actAwake {
		why := e.stopWhy
		a.mu.Unlock()
		return abandon(&CmdStoppedError{Reason: why})
	}
	a.mu.Unlock()
	u, err := cb.StartCmd(ctx, rt, spec)
	h.unit = u
	close(h.started)
	if err != nil {
		run.Release()
		return nil, err
	}
	return run, nil
}

finishStop (activator.go:2355), its first lines:

func (a *Service) finishStop(e *actEntry, outcome *stopObservation) {
	// A run attached to this instance is stopped and settled before
	// replication closes (run spec §5.3, codex 1): everything it committed
	// is in the final checkpoint, and nothing writes the database after.
	// A run that will not settle leaves the stop unproven.
	a.mu.Lock()
	h, why := e.cmd, e.stopWhy
	a.mu.Unlock()
	if h != nil {
		cctx, ccancel := context.WithTimeout(context.Background(), cmdStopBound)
		if err := h.stop(cctx, why); err != nil {
			log.Printf("activator: %s: run %s did not settle before the final checkpoint: %v", e.host, h.runID, err)
			outcome.uncertain = true
		}
		ccancel()
	}
	// Close = final sync + lease release. ...

(The rest of finishStop is unchanged; its later outcome.uncertain = !outcome.complete() keeps true, because complete() requires !o.uncertain.)

The idle check at activator.go:2533:

		if hibernates && e.inflight == 0 && e.cmd == nil && now.Sub(e.lastActive) >= idleWindow {

• [ ] Step 4: Run to verify they pass

Run: nice -n 10 go test ./internal/activator/ -count=1 Expected: PASS, the whole package.

• [ ] Step 5: Commit

git add internal/activator/cmd.go internal/activator/cmd_test.go internal/activator/activator.go
git commit -m "activator: RunCmd, the cmd hold and the stop ordering in finishStop (run spec §5.3, codex 1, 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 9: The plan's gate

Files: none new.

• [ ] Step 1: Run the gate

export GOFLAGS=-buildvcs=false
go build ./... && go vet ./... && test -z "$(gofmt -l .)"
TMPDIR=/tmp/g nice -n 10 make test
GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...
TMPDIR=/tmp/g nice -n 10 go test -race ./internal/activator/ ./internal/instancectl/ ./internal/instancefs/ -count=1

Expected: every command exits 0. If the sandbox's Go cache is read-only, prefix each go command with GOCACHE=$TMPDIR/gocache.

• [ ] Step 2: Post the #250 note

Post on amadan carlos/platform #250 (as this session's agent account, per the amadan skill): the claim activation must add cmd to the reservation protocol beside main, seed and sidecar, settle the cmd unit's producer before a generation is published, and then lift activator.ErrCmdClaim and the console's RUN-C42 refusal (Plan 2). Link this plan.

• [ ] Step 3: Commit anything the gate changed

git status --short

Expected: clean. If gofmt rewrote a file, commit it with run: gofmt and the trailer.

---

Self-review notes

• Spec coverage: §3.1 (Task 8 ErrCmdClaim, Task 4 the 503s); §4 (Task 6 cmdVars, the launch file, the working directory from the template); §5.3 (Tasks 7 and 8, every stop path in the spec's list: hibernation is held, Bounce, convergence, a failed process, the move freeze, agent shutdown, plus lease loss and retirement); §5.4 (Tasks 2 to 4, 6); §9 "Writer lifecycle" and "Unit and control" (Tasks 3, 4, 8). §5.2 step 3's directory and §5.8's box sweep use Task 5's RemoveAllBeneath in Plan 3.
• The spec's "a write the command commits just before the stop reaching the replica" is shown here by ordering (repWasClosed); the real write is a dev-box check (Plan 4 Task 6).
• ExecMainCode arrives as a number from real systemd; Task 4 passes it through, Plan 3 maps it.