carlos run: a member runs an app command against one instance (#259)

Status: rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Rev 1 (c66792a3) was approved by Paul. Rev 2 answers Astra's round 1 (CODEX-FINDINGS-RUN-AND-SEEDED-R1.md; findings 1 to 7 are this spec's). Each change is marked (codex N), and §11 lists the dispositions.

Discussion: amadan carlos/platform #259. Companion spec: 2026-10-08-seeded-instances-design.md (#260, #261).

1. The problem

Every one-off command against a live instance goes over aws ssm send-command as root on the box today. On 2026-10-06 a titogo session imported 22,462 check-ins that way (titogo PR #4048). It:

• ran the app's own binary against the instance's database;
• read a credential from the app's config;
• staged the input file through presigned URLs.

The canary recipe (titogo add, auth grant, seed) has the same shape.

Two members of Tito's team can sign in to the console as members of tito, but they can't do any of this without root on a box every tenant shares. The platform has no door for "run this app's command against that instance".

2. Rulings (Paul, 2026-10-08)

1. Who may: any person who is a member of the account, the same rule as carlos db pull. Service credentials are refused. A member can already read every instance's database and publish to non-stable channels, so this grants nothing new. Amended 2026-10-24: a run service credential may, for the commands it was granted. See 2026-10-24-service-runs-design.md.
2. Output transport: the run's own chunks in the deployment bucket, polled by the CLI through the console. There is no new network path; the console still never connects to a box.
3. Platform-run instances only. controld-run (titogo@) instances are converted (#261) rather than served.

3. What it does

carlos run --app <app> --instance <host> [--file <path>]... [--timeout 30m] -- <arg>...
carlos run --follow <id>

• It runs the release the instance follows, as /opt/carlos/live/<host> <arg>.... The caller never names the binary.
• It runs as the instance: the same uid, namespace, bind mounts, slice and environment. Never root, and never the agent.
• It streams stdout and stderr to the terminal, about one second behind, and exits with the command's own status.
• It records who ran what, where, and how it ended.

Defaults:

| | | |---|---| | hibernated instance | woken first, so its database is restored and replicating | | live instance | left running; the command opens the same database beside it (WAL) | | stdin | none; argv only | | Ctrl-C | cancels the run on the box | | --timeout | 30 minutes by default, 6 hours at most | | --file | repeatable; 64 MB in total | | concurrency | one run at a time per instance | | layout | refused unless the instance is nested | | host lifecycle claim | refused while the box's claim control is armed (§3.1) |

Not in this version:

• --version. The trampoline execs only /opt/carlos/live/<host> (internal/instanceexec/exec.go:72), and the control service binds that link to the instance's own build.
• --stopped (stop, run, start).
• --record (keep the output).
• An interactive shell.

3.1 With the host lifecycle claim armed (codex 2)

The claim code is on main but not active anywhere. Once a box arms claim control, two things stop working:

• instancectl answers 503 to every legacy start, seed and stop route (claimAwareLegacy, internal/instancectl/server.go:840);
• RoleLaunch.Start refuses every launch until the isolated root protocol carries reservations (internal/activator/rolelaunch_linux.go:23-28).

So on such a box carlos run refuses with refused: "claim", the same way it refuses on a flat box. Supporting it there is part of the claim's activation work (#250), not this spec. That work must add cmd to the reservation protocol beside main, seed and sidecar, and must settle the cmd unit's producer before the generation is published. #250 gets a note saying so.

4. The command's contract

This is what the app's binary sees. It goes into the docs/site/cli.md section "What an instance is started with".

• argv: exactly what followed --, after the binary. The platform appends nothing, unlike a seed's --db (seedArgv).
• Environment: a new composer, cmdVars (codex 7). It takes the instance's file environment (the config bundle with <host>.env on top), then: - removes CARLOS_CONTROL_SOCKET and CARLOS_PURGE, as seedVars does (backend_exec.go:1359); - sets or removes CARLOS_REGION from the backend's region, exactly as the serving composer does (backend_exec.go:1340-1344); - sets CARLOS_HOST, keeps CARLOS_ADMIN_TOKEN, and adds the three variables below.

| Variable | Value | |---|---| | CARLOS_DB | the instance's database path, the one --db names when the instance starts | | CARLOS_RUN_ID | this run's identifier | | CARLOS_RUN_DIR | a directory holding the --file uploads under their base names. Always present, even with no files. Deleted after the run |

• Working directory: the instance's data directory, as for every template (#256 item D).
• Signals: SIGTERM on cancel, timeout or an instance stop (§5.3), then SIGKILL 20 seconds later (KillMode=mixed, TimeoutStopSec=20).

Secrets reach the command only through its environment. The platform never puts them in argv and never prints the environment.

5. The flow

5.1 Request (CLI to console)

POST /api/cli/apps/{account}/{app}/instances/{host}/runs, multipart: a meta part (argv, timeout, file names and sizes), then one part per file. The console streams the file parts straight to the bucket (the F8 rule: bytes go through the console, never through a presigned URL). The whole request is capped by http.MaxBytesReader at 64 MB plus the meta allowance.

The console refuses, in this order, each with its own status and message:

1. A service credential (cliRoute with the zero svcrole.Set).
2. Not a member of the account: the byte-identical 404 apiAccountApp gives.
3. Over budget (429), on the instance verbs' per-token window.
4. No record for the host, or a record of another app: 404.
5. The record isn't platform-run exec, or is pooled, or is placed on a customer's fleet: 409.
6. A move is in flight (move state other than none): 409.
7. A run is already open on the instance: 409, naming its id and caller.
8. Bad input, 400: - argv empty, more than 256 arguments, over 64 KB in total, or containing a NUL byte; - a timeout outside 1 second to 6 hours; - files over the cap, or a file name that isn't a plain base name.

1. The instance is an import that isn't committed yet (seeded spec §5.4): 409, "finish or abort the import first" (codex R2-2).

If none apply, it writes these under the app's own subtree (release.AppPrefix, so the owner gate maps onto them and app trash and purge sweep them):

• runs/<id>/in/<name> for each file;
• runs/<id>/request.json: host, the record's incarnation (hostops.Record.Incarnation: account, app, host and CreatedAt), argv, timeout, the file list with sizes and sha256, the caller, and the time requested.

Then it writes the run onto the instance record itself, in one CAS on the record's ETag (codex 1, R2-6):

ActiveRun *RunState `json:"run,omitempty"`

type RunState struct {
    ID       string    `json:"id"`
    State    string    `json:"state"`    // queued | picked | cancel | done | cancelled | expired
    Box      string    `json:"box,omitempty"`
    BootID   string    `json:"boot,omitempty"`
    Deadline time.Time `json:"deadline"` // pickup deadline, then run deadline
}

Ordinary record writers carry ActiveRun through unchanged, the way they carry Maintenance today (provision.go:40-42).

It then bumps control/serial, writes the run-start audit row (§6), and answers 202 with the run id.

One record, one CAS. Runs, moves and imports all change the same object, so they serialize on the record's ETag:

• a run is admitted only when there is no non-terminal ActiveRun, no move in flight, and no uncommitted import;
• a move request (provision.go:1129) is refused while ActiveRun is non-terminal;
• an import's abort is refused while ActiveRun is non-terminal, although run admission already refuses uncommitted imports.

Whichever write lands first wins. The loser's CAS fails, it re-reads the record, and it refuses.

Deleting and recreating the host makes a new record with no ActiveRun, so an old request can never run against the replacement. The box also checks that request.json's incarnation matches the record before it launches (codex 3).

5.2 Pickup on the box (codex 3)

Restart's pattern (internal/host/restart.go) records the first sight of an object and never acts on it (:114, :165). That's right for restart and wrong here: it would drop runs queued before a box started, or before it came to own a host. So pickup works like this instead:

• When the box looks. Reconcile already reads the instance records in its scope on every serial bump and at agent start, so the box reads ActiveRun from those same reads. A record the box has just come to own is covered by the same pass, and an idle box pays nothing extra.
• Which runs it takes. It acts on a queued run only for a host whose exec row it owns. Every other box ignores it.
• Taking a run. The box moves the run from queued to picked, with its box name, agent boot id and the run's deadline, in one CAS on the record. - If the CAS fails, the box re-reads. If the run is no longer queued (taken, cancelled or expired), the box leaves it alone. - Only the box whose CAS succeeded launches, and it checks the deadline and cancel state again right before launch.
• A run this box picked but never finished. At agent start, a run marked picked by this box under an earlier boot id has lost its unit: BindsTo=carlos-edge.service stopped it with the edge. The box writes its result.json as interrupted: "agent restart" and CASes the run to done. A picked run is never executed again.
• No pickup (codex R2-6). Sixty seconds after the request, the CLI asks the console to expire the run. The console CASes it from queued to expired, which is terminal. - If that CAS succeeds, the CLI reports "not picked up" and exits 3. An expired run can never be picked, so a box that turns up late can't run it. - If the CAS fails because a box picked the run in the meantime, the CLI goes on following it.

Once the box has taken the run:

1. The layout check. It refuses unless activator.Classify answers LayoutNested for the row and nestedGate passes. A refusal is a result.json with refused: "flat", never a fallback to running the command as the agent. With claim control armed it is refused: "claim" (§3.1).
2. The wake. It wakes the instance if it is hibernated, through the ordinary wake path, so the database is restored and replication is running. A wake failure is refused: "wake", with the wake's error.
3. The files. It downloads the files into provision.InstanceDir(dataDir, host)/.carlos-run/<id>/ (codex 6: long hosts hash, provision.go:2032-2045). Each one is checked against its size and sha256 in request.json, written mode 0700 by carlos through instancefs, and no symlink is followed.
4. The launch. It attaches the run to the instance's activator entry (§5.3), then starts the unit.

5.3 The run is part of the instance's writer lifecycle (codex 1)

The command writes the instance's database, so it has to be inside every rule that protects those writes, not beside them. The activator entry for the host gains a cmd field: the run id, the unit's InvocationID, and a done channel. Then:

• Every way the instance stops first stops the cmd unit and waits for it to settle, before replication is closed or the entry is forgotten. That covers: - hibernation (stop, activator.go:2108); - Bounce (:1425), which config sync and carlos restart use; - convergence that stops an instance; - a serving process that fails; - the move freeze; - agent shutdown.

The ordering lives in finishStop (:2355), so a new stop path can't skip it. A run stopped this way ends as interrupted: "<reason>", and everything it committed before the stop was replicated before replication closed.

• Hibernation's idle timer doesn't fire while cmd is set. That's the run hold. The other stops above aren't held off; they interrupt the run.
• A restore never runs under the command. The cmd unit starts only for an entry that is awake and replicating. While the entry is waking or stopping it isn't started, and the box waits up to 60 seconds for the wake.
• A wake failure, or a stop before launch, ends the run as refused: "wake" or interrupted. It is never queued for later.

5.4 The unit (codex 4)

A fourth template, carlos-instance-cmd@.service, is copied from carlos-instance-seed@.service. Every sandbox line stays the same, and a test fails if the two ever differ outside these lines:

• Description=carlos exec-backed instance %I (cmd)
• ExecStart=/opt/carlos/carlos instance-exec cmd %I and ExecStopPost=/opt/carlos/carlos instance-exit cmd %I
• TimeoutStartSec=6h5m, a backstop past the 6-hour cap. The agent enforces the run's own timeout.
• StandardOutput=file:/run/carlos/logs/%I.cmd.sock and StandardError=file:/run/carlos/logs/%I.cmd-err.sock, two listeners so the streams stay separate.

Type=oneshot and RemainAfterExit=yes stay, so the result is kept until the agent reads it, as for the seed.

instancectl gains RoleCmd = "cmd" and UnitPrefixCmd = "carlos-instance-cmd@" (internal/instancectl/names.go; run is already the serving role). The control service starts, stops and sweeps the new unit like the other three, through two routes:

• POST /cmd starts the unit and answers at once with its InvocationID. (/seed blocks for up to 150 seconds; this one doesn't.)
• GET /cmd/{host}?invocation=<id> is a bounded long-poll, at most 30 seconds. It answers {state, InvocationID, Result, ExecMainStatus, Code}, where state is running, or done once that invocation's main process has exited. done covers both "active (exited)" and "failed".

When GET answers done, the agent records the result, then calls the existing /stop for the host's cmd role and waits for the unit to settle, exactly as /seed does (server.go:431-448). An invocation id that doesn't match is gone, and the run ends interrupted.

The agent writes launch.cmd.json through writeLaunch, with argv [/opt/carlos/live/<host>, <arg>...] and the environment from §4.

5.5 Output (codex 5)

The agent reads the two sockets and writes runs/<id>/out/<seq>-<stream>, where stream is o or e. The seq is zero-padded and shared across both streams, so the CLI can interleave them in the order the agent read them. It writes a chunk every second or every 256 KB, whichever comes first.

Total output is capped at 64 MB. Past the cap the agent stops writing chunks and records truncated; the command keeps running.

result.json is written only after every chunk is durably written, and it names the last seq:

{"exit": 0, "signal": "", "timed_out": false, "cancelled": false,
 "interrupted": "", "refused": "", "started": "...", "ended": "...",
 "box": "...", "last_seq": 17, "truncated": false}

The agent then deletes the run directory and clears the entry's cmd.

5.6 Following (CLI and console) (codex 5)

GET …/runs/{id}?after=<seq> answers the chunks after seq (a bounded page) and result.json once it exists. The CLI:

• polls every second;
• writes o chunks to stdout and e chunks to stderr;
• keeps reading pages until it has every chunk up to result.last_seq;
• then sends POST …/runs/{id}/ack with last_seq.

The console refuses an ack whose last_seq doesn't match the result. The CLI exits with:

| case | exit | |---|---| | command exited | its own status | | command killed by a signal | 128 + the signal number | | timed out | 124 (timeout(1)'s) | | interrupted by an instance stop or agent restart | 125 | | refused by the box (flat, claim, wake) or not picked up in 60 s | 3 | | refused by the console | 2 |

5.7 Cancel and timeout

When you press Ctrl-C:

1. The CLI sends POST …/runs/{id}/cancel.
2. The console changes the run on the record by CAS: - queued becomes cancelled, which is terminal, so no box ever launches it (codex R2-6); - picked becomes cancel, and the console bumps the serial.
3. On cancel, the owning box stops the unit. If it hasn't launched yet, it doesn't launch at all.

A second Ctrl-C exits the CLI without waiting; the cancel still happens. At the run's timeout the agent stops the unit itself. All of these end in an ordinary result.json. Once the box has written result.json, it CASes the run to done.

If the CLI goes away without cancelling, the run carries on to its end or its timeout. carlos run --follow <id> attaches to an open or finished run and prints every chunk not yet acknowledged.

5.8 Cleanup

• The ack. After a valid ack (§5.6), the console deletes runs/<id>/ and clears ActiveRun from the record by CAS, but only if the record still names this run in a terminal state.
• The console sweep. It runs in the existing deadline-sweep loop and: - deletes any run whose result is more than 24 hours old, whether or not it was acknowledged; - expires any run still queued 60 seconds past its request (codex R3-3). It CASes queued to expired (losing to a pickup is fine), writes the run-end audit row as expired, and clears it. So a run whose caller disconnected never blocks the instance; - deals with any run still picked or cancel an hour past its deadline: it marks the run done with the audit end lost, and clears it.

A run left in a terminal state is cleared the same way, so the next run is never blocked by one nobody acknowledged.

The 24-hour bucket lifecycle rule from chat is replaced by this sweep. The runs live under each app's subtree, which a prefix rule can't match, and the sweep needs no infrastructure change.

• The box sweep. At agent start, the box deletes any .carlos-run/ directory with no open run.

6. The record

Two audit rows are written through Server.auditActivity, with the instance as the target:

• run-start: the caller, app, host, argv, file names and sizes, the release version the instance follows, and the run id.
• run-end: the run id, exit, signal, timed out, cancelled, interrupted, refused, started, ended and box; or lost. The console writes it the first time it reads result.json, whether from the CLI's poll or from the sweep.

Output never goes in the audit log, but argv does. A member who puts a secret in argv has put it in the log, and docs/site/cli.md says so and says to pass secrets through --file.

7. Security

• Reach. The command has exactly the instance's reach. That's why it is refused on a flat box, where it would have the agent's.
• No widening for members. A member can already pull the database and publish a build to a non-stable channel that the instance runs. run only executes the build the instance follows.
• Box-written objects can be forged. Any box can write anywhere in the deployment bucket today (#171), so a compromised box could forge another box's run state, output or result. That is the existing record-authority boundary, not a new one, and run adds no path from a forged object to any action.
• File names are plain base names, checked at the console and again by the agent.

8. Errors the member sees

Each has its own message, and every one names the instance:

• not a member, or no such instance;
• not platform-run, pooled, or on a customer's fleet;
• moving, or (for a move) a run is open;
• a run is already open (with its id and who started it);
• the box isn't nested yet;
• the box has claim control armed;
• not picked up within 60 seconds;
• the wake failed (with the wake's error);
• interrupted (with the reason);
• timed out;
• a file is too large, or the output was truncated.

Every CLI and console string goes through copy review before it is written into the code.

9. Testing

• Console door: - each refusal in §5.1, in order; - the member-only rule; - the audit rows; - the record CAS under two concurrent run requests; - the run/move serialization in both directions; - cancel, ack (including a wrong last_seq) and the sweep.
• Pickup: - only the owning box acts; - a run queued before agent start is picked up; - a run queued before the box owned the host is picked up; - the pickup CAS with two boxes racing; - expiry winning over a late box, and a late pickup winning over expiry; - the sweep expiring a queued run whose CLI went away; - cancel before pickup never launching; - a request whose incarnation doesn't match the record being refused; - run, move and import abort racing on the record's ETag; - an uncommitted import refused at admission and again at launch; - a run picked before an agent restart ends interrupted and is never run again.
• Writer lifecycle: - each stop path in §5.3 stops and settles cmd before replication closes, shown by a write the command commits just before the stop reaching the replica; - the idle timer is held while a run is open; - no start while the entry is waking or stopping.
• Unit and control: - the cmd template equals the seed template outside the named lines; - RoleCmd parsing; - /cmd start, the long-poll for running, done and gone; - stop after done; - the sweep; - refusal while claim control is armed.
• Output: - chunk ordering across both streams; - the cap; - result.json only after the last chunk; - several pages of output after the result exists, all delivered before the ack.
• CLI: - interleaving; - every exit code; - Ctrl-C once and twice; - --follow.
• On the dev box once built: - a real run against a nested instance, including a write the instance then serves; - a run across a hibernation timer; - a run interrupted by carlos restart.
• Cross-compile: GOOS=darwin and windows builds stay green. The unit and control code is linux-only, behind build tags with stubs.

10. Rollout

It ships in the next release. It does nothing on a box until a member runs it, and it refuses on flat boxes and on claim-armed boxes, so it needs no switch. Tito's boxes need the nested layout before Doc and Vicky can use it (#247). The claim's activation (#250) carries the work in §3.1.

11. Round 1 dispositions

| Finding | Disposition | |---|---| | codex 1 (P1) the hold misses stop paths | Accepted. §5.3: the run joins the instance's writer lifecycle, and every stop settles cmd before replication closes; run and move are serialized at the console (§5.1) | | codex 2 (P2) claim-armed launch is missing | Accepted. §3.1: refused while claim control is armed; the work moves to #250 | | codex 3 (P2) the ETag pattern drops or replays runs | Accepted. §5.2: pickup from reconcile's record reads (at start and on ownership); a CAS from queued to picked; a picked run is never run again | | codex 4 (P2) oneshot success never becomes inactive | Accepted. §5.4: completion is the invocation's main process exiting; the agent records the result, then stops and settles the unit | | codex 5 (P2) ack can delete unread output | Accepted. §5.5 and §5.6: the result comes after the last chunk and names last_seq; the ack must match | | codex 6 (P2) the long-host path | Accepted. §5.2 step 3: provision.InstanceDir | | codex 7 (P3) CARLOS_REGION | Accepted. §4: cmdVars |

12. Round 2 dispositions

| Finding | Disposition | |---|---| | R1-1, still open: run and move don't share a reservation | Accepted. §5.1: the run state lives on the instance record, so run, move and import abort all CAS the same ETag | | R1-3, still open: no incarnation | Accepted. §5.1: the request carries the record's incarnation, and recreating the host drops ActiveRun | | R2-2 (P1) run gets past the import gate | Accepted. §5.1 step 9: refused while an import is uncommitted, checked under the record CAS and again at launch | | R2-6 (P2) "not picked up" leaves a runnable request | Accepted. §5.2 and §5.7: queued, picked, cancel, done, cancelled and expired, all moved by CAS on the record; expired and cancelled are terminal before launch |

13. Round 3 dispositions

| Finding | Disposition | |---|---| | R3-3 (P2) a disconnected caller leaves the run queued | Accepted. §5.8: the console sweep expires queued runs 60 seconds past the request, by CAS against pickup |