| 1 | # `carlos run`: a member runs an app command against one instance (#259) |
| 2 | |
| 3 | **Status:** rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Rev 1 (c66792a3) was approved by Paul. |
| 4 | Rev 2 answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`; |
| 5 | findings 1 to 7 are this spec's). Each change is marked **(codex N)**, and |
| 6 | §11 lists the dispositions. |
| 7 | |
| 8 | **Discussion:** amadan carlos/platform #259. Companion spec: |
| 9 | `2026-10-08-seeded-instances-design.md` (#260, #261). |
| 10 | |
| 11 | ## 1. The problem |
| 12 | |
| 13 | Every one-off command against a live instance goes over |
| 14 | `aws ssm send-command` as root on the box today. On 2026-10-06 a titogo |
| 15 | session imported 22,462 check-ins that way (titogo PR #4048). It: |
| 16 | |
| 17 | - ran the app's own binary against the instance's database; |
| 18 | - read a credential from the app's config; |
| 19 | - staged the input file through presigned URLs. |
| 20 | |
| 21 | The canary recipe (`titogo add`, `auth grant`, `seed`) has the same shape. |
| 22 | |
| 23 | Two members of Tito's team can sign in to the console as members of `tito`, |
| 24 | but they can't do any of this without root on a box every tenant shares. |
| 25 | The platform has no door for "run this app's command against that |
| 26 | instance". |
| 27 | |
| 28 | ## 2. Rulings (Paul, 2026-10-08) |
| 29 | |
| 30 | 1. **Who may:** any person who is a member of the account, the same rule as |
| 31 | `carlos db pull`. Service credentials are refused. A member can already |
| 32 | read every instance's database and publish to non-stable channels, so |
| 33 | this grants nothing new. |
| 34 | *Amended 2026-10-24: a `run` service credential may, for the commands it |
| 35 | was granted. See `2026-10-24-service-runs-design.md`.* |
| 36 | 2. **Output transport:** the run's own chunks in the deployment bucket, |
| 37 | polled by the CLI through the console. There is no new network path; the |
| 38 | console still never connects to a box. |
| 39 | 3. **Platform-run instances only.** controld-run (`titogo@`) instances are |
| 40 | converted (#261) rather than served. |
| 41 | |
| 42 | ## 3. What it does |
| 43 | |
| 44 | ``` |
| 45 | carlos run --app <app> --instance <host> [--file <path>]... [--timeout 30m] -- <arg>... |
| 46 | carlos run --follow <id> |
| 47 | ``` |
| 48 | |
| 49 | - It runs the release the instance follows, as |
| 50 | `/opt/carlos/live/<host> <arg>...`. The caller never names the binary. |
| 51 | - It runs as the instance: the same uid, namespace, bind mounts, slice and |
| 52 | environment. Never root, and never the agent. |
| 53 | - It streams stdout and stderr to the terminal, about one second behind, |
| 54 | and exits with the command's own status. |
| 55 | - It records who ran what, where, and how it ended. |
| 56 | |
| 57 | **Defaults:** |
| 58 | |
| 59 | | | | |
| 60 | |---|---| |
| 61 | | hibernated instance | woken first, so its database is restored and replicating | |
| 62 | | live instance | left running; the command opens the same database beside it (WAL) | |
| 63 | | stdin | none; argv only | |
| 64 | | Ctrl-C | cancels the run on the box | |
| 65 | | `--timeout` | 30 minutes by default, 6 hours at most | |
| 66 | | `--file` | repeatable; 64 MB in total | |
| 67 | | concurrency | one run at a time per instance | |
| 68 | | layout | refused unless the instance is nested | |
| 69 | | host lifecycle claim | refused while the box's claim control is armed (§3.1) | |
| 70 | |
| 71 | **Not in this version:** |
| 72 | |
| 73 | - `--version`. The trampoline execs only `/opt/carlos/live/<host>` |
| 74 | (`internal/instanceexec/exec.go:72`), and the control service binds that |
| 75 | link to the instance's own build. |
| 76 | - `--stopped` (stop, run, start). |
| 77 | - `--record` (keep the output). |
| 78 | - An interactive shell. |
| 79 | |
| 80 | ### 3.1 With the host lifecycle claim armed (codex 2) |
| 81 | |
| 82 | The claim code is on main but not active anywhere. Once a box arms claim |
| 83 | control, two things stop working: |
| 84 | |
| 85 | - instancectl answers 503 to every legacy start, seed and stop route |
| 86 | (`claimAwareLegacy`, `internal/instancectl/server.go:840`); |
| 87 | - `RoleLaunch.Start` refuses every launch until the isolated root protocol |
| 88 | carries reservations (`internal/activator/rolelaunch_linux.go:23-28`). |
| 89 | |
| 90 | So on such a box `carlos run` refuses with `refused: "claim"`, the same way |
| 91 | it refuses on a flat box. Supporting it there is part of the claim's |
| 92 | activation work (#250), not this spec. That work must add `cmd` to the |
| 93 | reservation protocol beside `main`, `seed` and `sidecar`, and must settle |
| 94 | the cmd unit's producer before the generation is published. #250 gets a |
| 95 | note saying so. |
| 96 | |
| 97 | ## 4. The command's contract |
| 98 | |
| 99 | This is what the app's binary sees. It goes into the `docs/site/cli.md` |
| 100 | section "What an instance is started with". |
| 101 | |
| 102 | - **argv:** exactly what followed `--`, after the binary. The platform |
| 103 | appends nothing, unlike a seed's `--db` (`seedArgv`). |
| 104 | - **Environment:** a new composer, `cmdVars` (codex 7). It takes the |
| 105 | instance's file environment (the config bundle with `<host>.env` on |
| 106 | top), then: |
| 107 | - removes `CARLOS_CONTROL_SOCKET` and `CARLOS_PURGE`, as `seedVars` does |
| 108 | (`backend_exec.go:1359`); |
| 109 | - sets or removes `CARLOS_REGION` from the backend's region, exactly as |
| 110 | the serving composer does (`backend_exec.go:1340-1344`); |
| 111 | - sets `CARLOS_HOST`, keeps `CARLOS_ADMIN_TOKEN`, and adds the three |
| 112 | variables below. |
| 113 | |
| 114 | | Variable | Value | |
| 115 | |---|---| |
| 116 | | `CARLOS_DB` | the instance's database path, the one `--db` names when the instance starts | |
| 117 | | `CARLOS_RUN_ID` | this run's identifier | |
| 118 | | `CARLOS_RUN_DIR` | a directory holding the `--file` uploads under their base names. Always present, even with no files. Deleted after the run | |
| 119 | |
| 120 | - **Working directory:** the instance's data directory, as for every |
| 121 | template (#256 item D). |
| 122 | - **Signals:** SIGTERM on cancel, timeout or an instance stop (§5.3), then |
| 123 | SIGKILL 20 seconds later (`KillMode=mixed`, `TimeoutStopSec=20`). |
| 124 | |
| 125 | Secrets reach the command only through its environment. The platform never |
| 126 | puts them in argv and never prints the environment. |
| 127 | |
| 128 | ## 5. The flow |
| 129 | |
| 130 | ### 5.1 Request (CLI to console) |
| 131 | |
| 132 | `POST /api/cli/apps/{account}/{app}/instances/{host}/runs`, multipart: a |
| 133 | `meta` part (`argv`, `timeout`, file names and sizes), then one part per |
| 134 | file. The console streams the file parts straight to the bucket (the F8 |
| 135 | rule: bytes go through the console, never through a presigned URL). The |
| 136 | whole request is capped by `http.MaxBytesReader` at 64 MB plus the meta |
| 137 | allowance. |
| 138 | |
| 139 | The console refuses, in this order, each with its own status and message: |
| 140 | |
| 141 | 1. A service credential (`cliRoute` with the zero `svcrole.Set`). |
| 142 | 2. Not a member of the account: the byte-identical 404 `apiAccountApp` |
| 143 | gives. |
| 144 | 3. Over budget (429), on the instance verbs' per-token window. |
| 145 | 4. No record for the host, or a record of another app: 404. |
| 146 | 5. The record isn't platform-run exec, or is pooled, or is placed on a |
| 147 | customer's fleet: 409. |
| 148 | 6. A move is in flight (`move` state other than none): 409. |
| 149 | 7. A run is already open on the instance: 409, naming its id and caller. |
| 150 | 8. Bad input, 400: |
| 151 | - argv empty, more than 256 arguments, over 64 KB in total, or |
| 152 | containing a NUL byte; |
| 153 | - a timeout outside 1 second to 6 hours; |
| 154 | - files over the cap, or a file name that isn't a plain base name. |
| 155 | |
| 156 | 9. The instance is an import that isn't committed yet (seeded spec §5.4): |
| 157 | 409, "finish or abort the import first" (codex R2-2). |
| 158 | |
| 159 | If none apply, it writes these under the app's own subtree |
| 160 | (`release.AppPrefix`, so the owner gate maps onto them and app trash and |
| 161 | purge sweep them): |
| 162 | |
| 163 | - `runs/<id>/in/<name>` for each file; |
| 164 | - `runs/<id>/request.json`: host, the record's incarnation |
| 165 | (`hostops.Record.Incarnation`: account, app, host and `CreatedAt`), argv, |
| 166 | timeout, the file list with sizes and sha256, the caller, and the time |
| 167 | requested. |
| 168 | |
| 169 | Then it writes the run onto the **instance record** itself, in one CAS on |
| 170 | the record's ETag (codex 1, R2-6): |
| 171 | |
| 172 | ```go |
| 173 | ActiveRun *RunState `json:"run,omitempty"` |
| 174 | |
| 175 | type RunState struct { |
| 176 | ID string `json:"id"` |
| 177 | State string `json:"state"` // queued | picked | cancel | done | cancelled | expired |
| 178 | Box string `json:"box,omitempty"` |
| 179 | BootID string `json:"boot,omitempty"` |
| 180 | Deadline time.Time `json:"deadline"` // pickup deadline, then run deadline |
| 181 | } |
| 182 | ``` |
| 183 | |
| 184 | Ordinary record writers carry `ActiveRun` through unchanged, the way they |
| 185 | carry `Maintenance` today (`provision.go:40-42`). |
| 186 | |
| 187 | It then bumps `control/serial`, writes the `run-start` audit row (§6), and |
| 188 | answers `202` with the run id. |
| 189 | |
| 190 | **One record, one CAS.** Runs, moves and imports all change the same |
| 191 | object, so they serialize on the record's ETag: |
| 192 | |
| 193 | - a run is admitted only when there is no non-terminal `ActiveRun`, no |
| 194 | move in flight, and no uncommitted import; |
| 195 | - a move request (`provision.go:1129`) is refused while `ActiveRun` is |
| 196 | non-terminal; |
| 197 | - an import's abort is refused while `ActiveRun` is non-terminal, although |
| 198 | run admission already refuses uncommitted imports. |
| 199 | |
| 200 | Whichever write lands first wins. The loser's CAS fails, it re-reads the |
| 201 | record, and it refuses. |
| 202 | |
| 203 | Deleting and recreating the host makes a new record with no `ActiveRun`, |
| 204 | so an old request can never run against the replacement. The box also |
| 205 | checks that `request.json`'s incarnation matches the record before it |
| 206 | launches (codex 3). |
| 207 | |
| 208 | ### 5.2 Pickup on the box (codex 3) |
| 209 | |
| 210 | Restart's pattern (`internal/host/restart.go`) records the first sight of |
| 211 | an object and never acts on it (`:114`, `:165`). That's right for restart |
| 212 | and wrong here: it would drop runs queued before a box started, or before |
| 213 | it came to own a host. So pickup works like this instead: |
| 214 | |
| 215 | - **When the box looks.** Reconcile already reads the instance records in |
| 216 | its scope on every serial bump and at agent start, so the box reads |
| 217 | `ActiveRun` from those same reads. A record the box has just come to own |
| 218 | is covered by the same pass, and an idle box pays nothing extra. |
| 219 | - **Which runs it takes.** It acts on a `queued` run only for a host whose |
| 220 | exec row it owns. Every other box ignores it. |
| 221 | - **Taking a run.** The box moves the run from `queued` to `picked`, with |
| 222 | its box name, agent boot id and the run's deadline, in one CAS on the |
| 223 | record. |
| 224 | - If the CAS fails, the box re-reads. If the run is no longer `queued` |
| 225 | (taken, cancelled or expired), the box leaves it alone. |
| 226 | - Only the box whose CAS succeeded launches, and it checks the deadline |
| 227 | and cancel state again right before launch. |
| 228 | - **A run this box picked but never finished.** At agent start, a run |
| 229 | marked `picked` by this box under an earlier boot id has lost its unit: |
| 230 | `BindsTo=carlos-edge.service` stopped it with the edge. The box writes |
| 231 | its `result.json` as `interrupted: "agent restart"` and CASes the run to |
| 232 | `done`. **A picked run is never executed again.** |
| 233 | - **No pickup** (codex R2-6). Sixty seconds after the request, the CLI asks |
| 234 | the console to expire the run. The console CASes it from `queued` to |
| 235 | `expired`, which is terminal. |
| 236 | - If that CAS succeeds, the CLI reports "not picked up" and exits 3. An |
| 237 | expired run can never be picked, so a box that turns up late can't |
| 238 | run it. |
| 239 | - If the CAS fails because a box picked the run in the meantime, the CLI |
| 240 | goes on following it. |
| 241 | |
| 242 | Once the box has taken the run: |
| 243 | |
| 244 | 1. **The layout check.** It refuses unless `activator.Classify` answers |
| 245 | `LayoutNested` for the row and `nestedGate` passes. A refusal is a |
| 246 | `result.json` with `refused: "flat"`, never a fallback to running the |
| 247 | command as the agent. With claim control armed it is |
| 248 | `refused: "claim"` (§3.1). |
| 249 | 2. **The wake.** It wakes the instance if it is hibernated, through the |
| 250 | ordinary wake path, so the database is restored and replication is |
| 251 | running. A wake failure is `refused: "wake"`, with the wake's error. |
| 252 | 3. **The files.** It downloads the files into |
| 253 | `provision.InstanceDir(dataDir, host)/.carlos-run/<id>/` (codex 6: |
| 254 | long hosts hash, `provision.go:2032-2045`). Each one is checked against |
| 255 | its size and sha256 in `request.json`, written mode 0700 by `carlos` |
| 256 | through `instancefs`, and no symlink is followed. |
| 257 | 4. **The launch.** It attaches the run to the instance's activator entry |
| 258 | (§5.3), then starts the unit. |
| 259 | |
| 260 | ### 5.3 The run is part of the instance's writer lifecycle (codex 1) |
| 261 | |
| 262 | The command writes the instance's database, so it has to be inside every |
| 263 | rule that protects those writes, not beside them. The activator entry for |
| 264 | the host gains a `cmd` field: the run id, the unit's InvocationID, and a |
| 265 | done channel. Then: |
| 266 | |
| 267 | - **Every way the instance stops** first stops the cmd unit and waits for |
| 268 | it to settle, before replication is closed or the entry is forgotten. |
| 269 | That covers: |
| 270 | - hibernation (`stop`, `activator.go:2108`); |
| 271 | - `Bounce` (`:1425`), which config sync and `carlos restart` use; |
| 272 | - convergence that stops an instance; |
| 273 | - a serving process that fails; |
| 274 | - the move freeze; |
| 275 | - agent shutdown. |
| 276 | |
| 277 | The ordering lives in `finishStop` (`:2355`), so a new stop path can't |
| 278 | skip it. A run stopped this way ends as |
| 279 | `interrupted: "<reason>"`, and everything it committed before the stop |
| 280 | was replicated before replication closed. |
| 281 | - **Hibernation's idle timer doesn't fire** while `cmd` is set. That's |
| 282 | the run hold. The other stops above aren't held off; they interrupt the |
| 283 | run. |
| 284 | - **A restore never runs under the command.** The cmd unit starts only for |
| 285 | an entry that is awake and replicating. While the entry is waking or |
| 286 | stopping it isn't started, and the box waits up to 60 seconds for the |
| 287 | wake. |
| 288 | - **A wake failure, or a stop before launch,** ends the run as |
| 289 | `refused: "wake"` or `interrupted`. It is never queued for later. |
| 290 | |
| 291 | ### 5.4 The unit (codex 4) |
| 292 | |
| 293 | A fourth template, `carlos-instance-cmd@.service`, is copied from |
| 294 | `carlos-instance-seed@.service`. Every sandbox line stays the same, and a |
| 295 | test fails if the two ever differ outside these lines: |
| 296 | |
| 297 | - `Description=carlos exec-backed instance %I (cmd)` |
| 298 | - `ExecStart=/opt/carlos/carlos instance-exec cmd %I` and |
| 299 | `ExecStopPost=/opt/carlos/carlos instance-exit cmd %I` |
| 300 | - `TimeoutStartSec=6h5m`, a backstop past the 6-hour cap. The agent |
| 301 | enforces the run's own timeout. |
| 302 | - `StandardOutput=file:/run/carlos/logs/%I.cmd.sock` and |
| 303 | `StandardError=file:/run/carlos/logs/%I.cmd-err.sock`, two listeners so |
| 304 | the streams stay separate. |
| 305 | |
| 306 | `Type=oneshot` and `RemainAfterExit=yes` stay, so the result is kept until |
| 307 | the agent reads it, as for the seed. |
| 308 | |
| 309 | `instancectl` gains `RoleCmd = "cmd"` and |
| 310 | `UnitPrefixCmd = "carlos-instance-cmd@"` (`internal/instancectl/names.go`; |
| 311 | `run` is already the serving role). The control service starts, stops and |
| 312 | sweeps the new unit like the other three, through two routes: |
| 313 | |
| 314 | - **`POST /cmd`** starts the unit and answers at once with its |
| 315 | InvocationID. (`/seed` blocks for up to 150 seconds; this one doesn't.) |
| 316 | - **`GET /cmd/{host}?invocation=<id>`** is a bounded long-poll, at most 30 |
| 317 | seconds. It answers `{state, InvocationID, Result, ExecMainStatus, |
| 318 | Code}`, where `state` is `running`, or `done` once that invocation's |
| 319 | main process has exited. `done` covers both "active (exited)" and |
| 320 | "failed". |
| 321 | |
| 322 | When `GET` answers `done`, the agent records the result, then calls the |
| 323 | existing `/stop` for the host's cmd role and waits for the unit to settle, |
| 324 | exactly as `/seed` does (`server.go:431-448`). An invocation id that |
| 325 | doesn't match is `gone`, and the run ends `interrupted`. |
| 326 | |
| 327 | The agent writes `launch.cmd.json` through `writeLaunch`, with argv |
| 328 | `[/opt/carlos/live/<host>, <arg>...]` and the environment from §4. |
| 329 | |
| 330 | ### 5.5 Output (codex 5) |
| 331 | |
| 332 | The agent reads the two sockets and writes `runs/<id>/out/<seq>-<stream>`, |
| 333 | where stream is `o` or `e`. The seq is zero-padded and shared across both |
| 334 | streams, so the CLI can interleave them in the order the agent read them. |
| 335 | It writes a chunk every second or every 256 KB, whichever comes first. |
| 336 | |
| 337 | Total output is capped at 64 MB. Past the cap the agent stops writing |
| 338 | chunks and records `truncated`; the command keeps running. |
| 339 | |
| 340 | `result.json` is written **only after every chunk is durably written**, |
| 341 | and it names the last seq: |
| 342 | |
| 343 | ```json |
| 344 | {"exit": 0, "signal": "", "timed_out": false, "cancelled": false, |
| 345 | "interrupted": "", "refused": "", "started": "...", "ended": "...", |
| 346 | "box": "...", "last_seq": 17, "truncated": false} |
| 347 | ``` |
| 348 | |
| 349 | The agent then deletes the run directory and clears the entry's `cmd`. |
| 350 | |
| 351 | ### 5.6 Following (CLI and console) (codex 5) |
| 352 | |
| 353 | `GET …/runs/{id}?after=<seq>` answers the chunks after `seq` (a bounded |
| 354 | page) and `result.json` once it exists. The CLI: |
| 355 | |
| 356 | - polls every second; |
| 357 | - writes `o` chunks to stdout and `e` chunks to stderr; |
| 358 | - keeps reading pages until it has every chunk up to `result.last_seq`; |
| 359 | - then sends `POST …/runs/{id}/ack` with `last_seq`. |
| 360 | |
| 361 | The console refuses an ack whose `last_seq` doesn't match the result. The |
| 362 | CLI exits with: |
| 363 | |
| 364 | | case | exit | |
| 365 | |---|---| |
| 366 | | command exited | its own status | |
| 367 | | command killed by a signal | 128 + the signal number | |
| 368 | | timed out | 124 (`timeout(1)`'s) | |
| 369 | | interrupted by an instance stop or agent restart | 125 | |
| 370 | | refused by the box (flat, claim, wake) or not picked up in 60 s | 3 | |
| 371 | | refused by the console | 2 | |
| 372 | |
| 373 | ### 5.7 Cancel and timeout |
| 374 | |
| 375 | When you press Ctrl-C: |
| 376 | |
| 377 | 1. The CLI sends `POST …/runs/{id}/cancel`. |
| 378 | 2. The console changes the run on the record by CAS: |
| 379 | - `queued` becomes `cancelled`, which is terminal, so no box ever |
| 380 | launches it (codex R2-6); |
| 381 | - `picked` becomes `cancel`, and the console bumps the serial. |
| 382 | 3. On `cancel`, the owning box stops the unit. If it hasn't launched yet, |
| 383 | it doesn't launch at all. |
| 384 | |
| 385 | A second Ctrl-C exits the CLI without waiting; the cancel still happens. |
| 386 | At the run's timeout the agent stops the unit itself. All of these end in |
| 387 | an ordinary `result.json`. Once the box has written `result.json`, it CASes |
| 388 | the run to `done`. |
| 389 | |
| 390 | If the CLI goes away without cancelling, the run carries on to its end or |
| 391 | its timeout. `carlos run --follow <id>` attaches to an open or finished run |
| 392 | and prints every chunk not yet acknowledged. |
| 393 | |
| 394 | ### 5.8 Cleanup |
| 395 | |
| 396 | - **The ack.** After a valid ack (§5.6), the console deletes `runs/<id>/` |
| 397 | and clears `ActiveRun` from the record by CAS, but only if the record |
| 398 | still names this run in a terminal state. |
| 399 | - **The console sweep.** It runs in the existing deadline-sweep loop and: |
| 400 | - deletes any run whose result is more than 24 hours old, whether or not |
| 401 | it was acknowledged; |
| 402 | - expires any run still `queued` 60 seconds past its request (codex |
| 403 | R3-3). It CASes `queued` to `expired` (losing to a pickup is fine), |
| 404 | writes the `run-end` audit row as `expired`, and clears it. So a run |
| 405 | whose caller disconnected never blocks the instance; |
| 406 | - deals with any run still `picked` or `cancel` an hour past its |
| 407 | deadline: it marks the run `done` with the audit end `lost`, and |
| 408 | clears it. |
| 409 | |
| 410 | A run left in a terminal state is cleared the same way, so the next run |
| 411 | is never blocked by one nobody acknowledged. |
| 412 | |
| 413 | The 24-hour bucket lifecycle rule from chat is replaced by this sweep. |
| 414 | The runs live under each app's subtree, which a prefix rule can't match, |
| 415 | and the sweep needs no infrastructure change. |
| 416 | - **The box sweep.** At agent start, the box deletes any `.carlos-run/` |
| 417 | directory with no open run. |
| 418 | |
| 419 | ## 6. The record |
| 420 | |
| 421 | Two audit rows are written through `Server.auditActivity`, with the |
| 422 | instance as the target: |
| 423 | |
| 424 | - `run-start`: the caller, app, host, argv, file names and sizes, the |
| 425 | release version the instance follows, and the run id. |
| 426 | - `run-end`: the run id, exit, signal, timed out, cancelled, interrupted, |
| 427 | refused, started, ended and box; or `lost`. The console writes it the |
| 428 | first time it reads `result.json`, whether from the CLI's poll or from |
| 429 | the sweep. |
| 430 | |
| 431 | Output never goes in the audit log, but argv does. A member who puts a |
| 432 | secret in argv has put it in the log, and `docs/site/cli.md` says so and |
| 433 | says to pass secrets through `--file`. |
| 434 | |
| 435 | ## 7. Security |
| 436 | |
| 437 | - **Reach.** The command has exactly the instance's reach. That's why it |
| 438 | is refused on a flat box, where it would have the agent's. |
| 439 | - **No widening for members.** A member can already pull the database and |
| 440 | publish a build to a non-stable channel that the instance runs. `run` |
| 441 | only executes the build the instance follows. |
| 442 | - **Box-written objects can be forged.** Any box can write anywhere in the |
| 443 | deployment bucket today (#171), so a compromised box could forge another |
| 444 | box's run state, output or result. That is the existing |
| 445 | record-authority boundary, not a new one, and `run` adds no path from a |
| 446 | forged object to any action. |
| 447 | - **File names** are plain base names, checked at the console and again by |
| 448 | the agent. |
| 449 | |
| 450 | ## 8. Errors the member sees |
| 451 | |
| 452 | Each has its own message, and every one names the instance: |
| 453 | |
| 454 | - not a member, or no such instance; |
| 455 | - not platform-run, pooled, or on a customer's fleet; |
| 456 | - moving, or (for a move) a run is open; |
| 457 | - a run is already open (with its id and who started it); |
| 458 | - the box isn't nested yet; |
| 459 | - the box has claim control armed; |
| 460 | - not picked up within 60 seconds; |
| 461 | - the wake failed (with the wake's error); |
| 462 | - interrupted (with the reason); |
| 463 | - timed out; |
| 464 | - a file is too large, or the output was truncated. |
| 465 | |
| 466 | Every CLI and console string goes through copy review before it is written |
| 467 | into the code. |
| 468 | |
| 469 | ## 9. Testing |
| 470 | |
| 471 | - **Console door:** |
| 472 | - each refusal in §5.1, in order; |
| 473 | - the member-only rule; |
| 474 | - the audit rows; |
| 475 | - the record CAS under two concurrent run requests; |
| 476 | - the run/move serialization in both directions; |
| 477 | - cancel, ack (including a wrong `last_seq`) and the sweep. |
| 478 | - **Pickup:** |
| 479 | - only the owning box acts; |
| 480 | - a run queued before agent start is picked up; |
| 481 | - a run queued before the box owned the host is picked up; |
| 482 | - the pickup CAS with two boxes racing; |
| 483 | - expiry winning over a late box, and a late pickup winning over expiry; |
| 484 | - the sweep expiring a queued run whose CLI went away; |
| 485 | - cancel before pickup never launching; |
| 486 | - a request whose incarnation doesn't match the record being refused; |
| 487 | - run, move and import abort racing on the record's ETag; |
| 488 | - an uncommitted import refused at admission and again at launch; |
| 489 | - a run picked before an agent restart ends `interrupted` and is never |
| 490 | run again. |
| 491 | - **Writer lifecycle:** |
| 492 | - each stop path in §5.3 stops and settles cmd before replication |
| 493 | closes, shown by a write the command commits just before the stop |
| 494 | reaching the replica; |
| 495 | - the idle timer is held while a run is open; |
| 496 | - no start while the entry is waking or stopping. |
| 497 | - **Unit and control:** |
| 498 | - the cmd template equals the seed template outside the named lines; |
| 499 | - `RoleCmd` parsing; |
| 500 | - `/cmd` start, the long-poll for `running`, `done` and `gone`; |
| 501 | - stop after done; |
| 502 | - the sweep; |
| 503 | - refusal while claim control is armed. |
| 504 | - **Output:** |
| 505 | - chunk ordering across both streams; |
| 506 | - the cap; |
| 507 | - `result.json` only after the last chunk; |
| 508 | - several pages of output after the result exists, all delivered before |
| 509 | the ack. |
| 510 | - **CLI:** |
| 511 | - interleaving; |
| 512 | - every exit code; |
| 513 | - Ctrl-C once and twice; |
| 514 | - `--follow`. |
| 515 | - **On the dev box once built:** |
| 516 | - a real run against a nested instance, including a write the instance |
| 517 | then serves; |
| 518 | - a run across a hibernation timer; |
| 519 | - a run interrupted by `carlos restart`. |
| 520 | - **Cross-compile:** `GOOS=darwin` and `windows` builds stay green. The |
| 521 | unit and control code is linux-only, behind build tags with stubs. |
| 522 | |
| 523 | ## 10. Rollout |
| 524 | |
| 525 | It ships in the next release. It does nothing on a box until a member runs |
| 526 | it, and it refuses on flat boxes and on claim-armed boxes, so it needs no |
| 527 | switch. Tito's boxes need the nested layout before Doc and Vicky can use it |
| 528 | (#247). The claim's activation (#250) carries the work in §3.1. |
| 529 | |
| 530 | ## 11. Round 1 dispositions |
| 531 | |
| 532 | | Finding | Disposition | |
| 533 | |---|---| |
| 534 | | codex 1 (P1) the hold misses stop paths | **Accepted.** §5.3: the run joins the instance's writer lifecycle, and every stop settles cmd before replication closes; run and move are serialized at the console (§5.1) | |
| 535 | | codex 2 (P2) claim-armed launch is missing | **Accepted.** §3.1: refused while claim control is armed; the work moves to #250 | |
| 536 | | codex 3 (P2) the ETag pattern drops or replays runs | **Accepted.** §5.2: pickup from reconcile's record reads (at start and on ownership); a CAS from queued to picked; a picked run is never run again | |
| 537 | | codex 4 (P2) oneshot success never becomes inactive | **Accepted.** §5.4: completion is the invocation's main process exiting; the agent records the result, then stops and settles the unit | |
| 538 | | codex 5 (P2) ack can delete unread output | **Accepted.** §5.5 and §5.6: the result comes after the last chunk and names `last_seq`; the ack must match | |
| 539 | | codex 6 (P2) the long-host path | **Accepted.** §5.2 step 3: `provision.InstanceDir` | |
| 540 | | codex 7 (P3) `CARLOS_REGION` | **Accepted.** §4: `cmdVars` | |
| 541 | |
| 542 | ## 12. Round 2 dispositions |
| 543 | |
| 544 | | Finding | Disposition | |
| 545 | |---|---| |
| 546 | | R1-1, still open: run and move don't share a reservation | **Accepted.** §5.1: the run state lives on the instance record, so run, move and import abort all CAS the same ETag | |
| 547 | | R1-3, still open: no incarnation | **Accepted.** §5.1: the request carries the record's incarnation, and recreating the host drops `ActiveRun` | |
| 548 | | R2-2 (P1) run gets past the import gate | **Accepted.** §5.1 step 9: refused while an import is uncommitted, checked under the record CAS and again at launch | |
| 549 | | R2-6 (P2) "not picked up" leaves a runnable request | **Accepted.** §5.2 and §5.7: queued, picked, cancel, done, cancelled and expired, all moved by CAS on the record; expired and cancelled are terminal before launch | |
| 550 | |
| 551 | ## 13. Round 3 dispositions |
| 552 | |
| 553 | | Finding | Disposition | |
| 554 | |---|---| |
| 555 | | R3-3 (P2) a disconnected caller leaves the run queued | **Accepted.** §5.8: the console sweep expires queued runs 60 seconds past the request, by CAS against pickup | |
| 556 | |
amadan