| 1 | # Runs started by a service credential |
| 2 | |
| 3 | **Date:** 2026-10-24 |
| 4 | **Status:** Design, awaiting approval |
| 5 | **Owner:** Paul |
| 6 | **Asked for by:** Tito (Linear PRO-6214). Fleet's Classic import and recurring |
| 7 | sync need to run an app command against an instance without a person at a |
| 8 | terminal. |
| 9 | **Amends:** `2026-10-08-carlos-run-design.md`, § 2 ruling 1. |
| 10 | |
| 11 | ## Why |
| 12 | |
| 13 | `carlos run` lets a member of an account run the app's own command against an |
| 14 | instance, as the instance. Fleet, Tito's back office, needs exactly that for two |
| 15 | jobs: |
| 16 | |
| 17 | - **The Classic import.** A person presses a button and Fleet copies one |
| 18 | account's data in. |
| 19 | - **The recurring sync.** It keeps an imported account up to date on a |
| 20 | schedule. |
| 21 | |
| 22 | Fleet holds a server credential and no person's token. Today every run route |
| 23 | refuses a service principal: `runGate` and `handleAPIRunCreate` check for |
| 24 | `PrincipalUser` (`internal/console/apiruns.go:156`, |
| 25 | `internal/console/apirunsfollow.go:36`). The work-arounds on the table are both |
| 26 | unacceptable: |
| 27 | |
| 28 | - **A personal operator token in Tito's router.** A person's whole membership |
| 29 | would sit in a service, and nobody could tell that from the audit. |
| 30 | - **Root or systemd jobs on the box.** That goes around the platform, which is |
| 31 | what `carlos run` exists to end. |
| 32 | |
| 33 | ## Decisions (Paul, 2026-10-24) |
| 34 | |
| 35 | - **Service credentials may start and follow runs.** This reverses run spec |
| 36 | § 2 ruling 1 ("service credentials are refused"), for the one role below |
| 37 | only. |
| 38 | - **A `run` credential names the commands it may run** (option A). It does |
| 39 | not get any command the binary has. A leaked credential can then run only |
| 40 | those, not arbitrary commands against every instance of the app. |
| 41 | - **The person a run is "for" is recorded on the service's word.** When Fleet |
| 42 | starts a run because someone pressed a button, the run records the name |
| 43 | Fleet gives, labelled as stated by that service. It grants nothing and is |
| 44 | not verified. A scheduled run records no person. |
| 45 | |
| 46 | ## Goals |
| 47 | |
| 48 | - A service can create, read (progress and output), cancel, expire and ack |
| 49 | runs on its own app's instances, and nothing else, once it has been granted |
| 50 | that on purpose. |
| 51 | - An ungranted service, a service from the wrong app, and a revoked service |
| 52 | are all refused. |
| 53 | - Deleting and recreating an account, or moving it, can never redirect a run. |
| 54 | - A caller that loses an HTTP response can find out what happened and carry |
| 55 | on without starting a second run. |
| 56 | - Everything above goes through the supported console API, with no box access. |
| 57 | |
| 58 | ## Non-goals |
| 59 | |
| 60 | - Host-claim consoles. They refuse runs for everyone until the reservation |
| 61 | protocol (#250) adds `cmd` (see § 6). |
| 62 | - Verifying who the person was. That is a later upgrade (§ 4). |
| 63 | - A scheduler. The caller decides when to run, and Fleet owns its schedule. |
| 64 | - Anything about what Fleet's command does: the Classic DB credentials, how |
| 65 | snapshots are cut. See § 7 for the caller's side. |
| 66 | |
| 67 | ## Anti-goals |
| 68 | |
| 69 | - **A role that quietly widens.** No existing role, `Admin` included, gains |
| 70 | runs. A service that can provision instances cannot run commands on them |
| 71 | unless it is separately minted for that. |
| 72 | - **A second run from a retry.** A lost response retried must never start the |
| 73 | command twice. |
| 74 | - **An audit line that misleads.** A service's run must never read as if a |
| 75 | person ran it. |
| 76 | |
| 77 | ## 1. The capability: a new role, `run` |
| 78 | |
| 79 | - **`svcrole.Run` joins the closed catalogue** (`internal/svcrole/svcrole.go`). |
| 80 | It is app-scoped, so `App` is required, refused account-wide the way |
| 81 | `Service` is (`ErrAccountScopedService`). It carries no rungs. |
| 82 | - **It grants exactly the five run routes** for instances of its own app: |
| 83 | create, get (with `?after=`), cancel, expire and ack |
| 84 | (`authcli.go:736-740`). Those routes are registered with `svcrole.Set{Run}` |
| 85 | instead of the empty set. People keep the routes on the existing member |
| 86 | rule, unchanged. |
| 87 | - **No other role implies it, Admin included** (my call, holding the |
| 88 | anti-goal). Today `svcrole.Allow` always adds Admin to a route's set |
| 89 | (`internal/svcrole/svcrole.go:126`, pinned by |
| 90 | `TestAdminPermitsEveryNonEmptySet`), so this IS new code. Add a constructor |
| 91 | for an exact set (e.g. `svcrole.Exactly(Run)`) that Admin does not join, |
| 92 | and amend that test and the Admin doc to name the one exception. A test |
| 93 | asserts that every role except `Run`, Admin included, is refused on all five |
| 94 | routes. |
| 95 | - **Minted, rotated and revoked like every other service:** |
| 96 | `carlos services create --role run --app <app> --commands "<prefix>,<prefix>" <name>`, |
| 97 | by an account owner, through `handleAPIServiceCreate`. Revocation is the |
| 98 | existing soft revoke, and a revoked token is refused at the door. |
| 99 | - **The command list (Paul, 2026-10-24, option A).** A `run` credential |
| 100 | carries a non-empty list of NON-EMPTY allowed command prefixes. A |
| 101 | zero-word prefix would match every argv, so it is refused at mint, refused |
| 102 | when the credential is read back, and never matches in the matcher. The |
| 103 | list is stored on its |
| 104 | `cli_tokens` row the way `Deploy` carries rungs: an additive column, parsed |
| 105 | only for this role, and refused at mint when it is empty |
| 106 | (`ErrRunNeedsCommands`). |
| 107 | - Each prefix is a sequence of argv words, written space-separated on the |
| 108 | CLI. `import classic` is the words `import`, `classic`. |
| 109 | - A run is allowed only when its argv starts with every word of at least |
| 110 | one prefix, compared word for word and exactly: no globbing, no case |
| 111 | folding, no substring. |
| 112 | - Create refuses anything else with 403 `command_not_granted`, before any |
| 113 | object is written, and with no key state either: an unkeyed refusal. |
| 114 | - `carlos services list` shows the list. Changing it means minting a new |
| 115 | credential (rotation keeps the list, and there is no edit door; my call). |
| 116 | - Prefixes contain no empty word, and each word is at most 128 bytes, with |
| 117 | at most 32 prefixes (the builder may tune these). |
| 118 | - People are not affected: members keep running any command. |
| 119 | - **A service sees and acts on only the runs it started** (my call). The run |
| 120 | records the CREDENTIAL's id (its `cli_tokens` row), not its name, because a |
| 121 | revoked name can be minted again (`authstore.go:250`) and must not inherit |
| 122 | the old credential's runs. Rotation keeps the row, so a rotated credential |
| 123 | keeps its runs. Get, cancel, expire and ack check that id. A run it didn't |
| 124 | start answers 404, the same as an unknown id, so its existence doesn't leak. |
| 125 | Members still see every run on their account, the service's included. |
| 126 | - **Create's "already has a run open" refusal is redacted for a service** |
| 127 | (my call). Today it names the open run's id and the member's email |
| 128 | (`apiruns.go:32`, `:202`). A service gets the refusal and its code without |
| 129 | either, unless the open run is its own. |
| 130 | - **Rate budgets are unchanged:** create spends `s.cli.instances` and the |
| 131 | follow routes spend `s.cli.runs`, both keyed on the token hash |
| 132 | (`apiruns.go:168`, `apirunsfollow.go:48`). A service's runs therefore draw on |
| 133 | that credential's own budget, which is separate from Fleet's provisioning |
| 134 | credential. |
| 135 | |
| 136 | ## 2. What a service's run records |
| 137 | |
| 138 | `runs.Request` (`internal/runs/runs.go:166`) gains fields, additively, so old |
| 139 | readers ignore them: |
| 140 | |
| 141 | | Field | For a person's run | For a service's run | |
| 142 | |---|---|---| |
| 143 | | `by` | the member's email (today) | what `auditActor` writes for a service today, `service:<account sqid>/<name>` (`apicli.go:105`) | |
| 144 | | `by_service` | empty | the service name, for display | |
| 145 | | `by_credential` | empty | the credential's id, which the own-runs check uses | |
| 146 | | `on_behalf_of` | empty | the email the service gave, or empty for a scheduled run | |
| 147 | | `idempotency_key` | the key's scoped hash (§ 3), when one was sent | the key's scoped hash, expected from Fleet | |
| 148 | |
| 149 | Every place the run is shown or audited (the CLI's follow output, the |
| 150 | console's run history, the account's audit log) shows a service's run as |
| 151 | "run by service `<name>` for `<email>` (as stated by the service)", or "run by |
| 152 | service `<name>` (scheduled)". It is never shown as if the email ran it. |
| 153 | |
| 154 | `on_behalf_of` is validated for shape only: it must be an email, at most 254 |
| 155 | bytes. It is not checked against membership, because it grants nothing. |
| 156 | |
| 157 | ## 3. Lost responses: idempotent create |
| 158 | |
| 159 | Run ids are minted by the console (`runs.MintID`), so today a caller who loses |
| 160 | the create response can't tell "started" from "never arrived". So: |
| 161 | |
| 162 | - **Create accepts an `Idempotency-Key` header**: 1–128 bytes of |
| 163 | `[A-Za-z0-9._:-]`, chosen by the caller. |
| 164 | - **A key is scoped to its caller** (my call): to the credential id for a |
| 165 | service, to the user for a person, and within that to the instance. One |
| 166 | caller can never find, probe or collide with another's key. |
| 167 | - **The proof that a key was admitted lives in the instance record, written |
| 168 | by the same compare-and-swap that admits the run** (my call, after Astra's |
| 169 | review). The record gains `run_keys`: a short list of recent admissions, |
| 170 | each with a hash of (caller scope, key), the run id, the request digest, a |
| 171 | hash of the caller scope (which the per-caller cap counts by) and the time. `provision.RunAdmit` refuses to admit when the list already holds |
| 172 | this caller's key, and returns that run instead. |
| 173 | - **Admission and dedupe are therefore one atomic write.** Two identical |
| 174 | creates at once can both write their objects, but only one is admitted. |
| 175 | The other finds its key already in the record, deletes its own objects |
| 176 | and answers with the winner's run. |
| 177 | - **There is no separate key object,** so there is no pending or abandoned |
| 178 | state, no sweep, and no crash window between "admitted" and "key |
| 179 | recorded". |
| 180 | - **Ack, the run sweep and the next run replacing a finished one all keep |
| 181 | `run_keys`,** so a retry after any of them still finds the run. |
| 182 | - **The digest covers the meta part only:** argv, timeout, `on_behalf_of`, |
| 183 | and each file's name, size and sha256. A keyed create must declare each |
| 184 | file's sha256 in the meta, and the console verifies the uploaded bytes |
| 185 | against it. The meta is the first multipart part, so the key is checked |
| 186 | against the record straight after parsing it, before the open-run, moving |
| 187 | and import refusals and before any file byte is read. A retry of an |
| 188 | admitted key is answered without reading the files. |
| 189 | - **The same key with a different digest** is `409` `idempotency_key_reused`, |
| 190 | and nothing is written. |
| 191 | - **Answers to a keyed create:** |
| 192 | - a fresh admission and a replay of an admitted key both return `202`, the |
| 193 | status create gives today, with the same body, plus `"replayed": true` on |
| 194 | a replay; |
| 195 | - `GET .../instances/{host}/runs?key=<key>` returns the run id from the |
| 196 | record, or 404. |
| 197 | - **Refusals are not remembered** (my call). Only an admitted run is. |
| 198 | - A 4xx means THIS attempt admitted nothing. An earlier attempt with the |
| 199 | same key may already have run: the shared gates (401, the rate budget's |
| 200 | 429) come before the key lookup, so the caller confirms with |
| 201 | `GET ...?key=` before treating a 4xx on a retry as "never ran". |
| 202 | - A retry with the same key is evaluated afresh: a `run_open` refusal can |
| 203 | succeed once the other run has finished, which is what the caller asked |
| 204 | for. |
| 205 | - A 5xx, a timeout or a dropped connection means unknown, and the caller |
| 206 | retries with the same key. |
| 207 | - **How long a key lasts:** each admission prunes `run_keys` to entries younger |
| 208 | than 24 hours (`runs.KeyTTL`), then keeps the newest 32 entries of each |
| 209 | caller (`runs.MaxRunKeysPerCaller`). The cap is per caller, so one caller's |
| 210 | admissions can never evict another's key. A key is honoured for 24 hours, |
| 211 | or until that same caller has made 32 newer keyed admissions on this |
| 212 | instance, whichever comes first. A caller must not retry a key past that. |
| 213 | Fleet retries within minutes. |
| 214 | - **`request.json` records the key** as `idempotency_key`: its scoped hash, |
| 215 | never the caller's own words. It links the run to its key after the record |
| 216 | has pruned the entry. |
| 217 | - **Mixed versions:** an older box binary rewriting the record would drop |
| 218 | `run_keys`. Keyed creates stay off (`503` `keys_not_ready`) until the console |
| 219 | is told, by config, that every record writer preserves the field. |
| 220 | - **A known limit:** deleting an instance deletes its record, `run_keys` with |
| 221 | it. A retry that arrives after the host was deleted and created again is a |
| 222 | fresh run on the new incarnation. It is still only a command the credential |
| 223 | was granted, on its own app. |
| 224 | - **Refusal codes:** the run handlers' own refusals carry a stable `code` |
| 225 | beside `error`, and the API docs list them. The shared gates before the |
| 226 | handler keep their bodies exactly as today, deliberately identical across |
| 227 | doors (AGENTS.md): not signed in, role mismatch, and the canonical |
| 228 | membership 404. Those already distinguish themselves by status. |
| 229 | |
| 230 | People may send the key too, and the CLI gains `--idempotency-key`. Nothing |
| 231 | changes for a person who doesn't. |
| 232 | |
| 233 | ## 4. Upgrading "on the service's word" later |
| 234 | |
| 235 | If the person ever needs to be verified, the request gains a signed assertion |
| 236 | from the caller's identity system, and the console adds `on_behalf_verified: |
| 237 | true` when it checks out. The fields above stay, so nothing already recorded |
| 238 | changes meaning. Not built now. |
| 239 | |
| 240 | ## 5. What stays exactly as it is |
| 241 | |
| 242 | These were already built in the run series, and must be shown to hold for a |
| 243 | service's run too, each by a test that starts the run with a `Run` service: |
| 244 | |
| 245 | - **Incarnation binding.** The incarnation is taken at create |
| 246 | (`apiruns.go:250`) and checked by the box before launch. A run created |
| 247 | before an account is deleted and recreated, or moved, never launches |
| 248 | against the new one. |
| 249 | - **One run per instance**, through the `ActiveRun` compare-and-swap. This |
| 250 | covers a service's run and a person's run on the same instance too. |
| 251 | - **The 64 MB file cap** (`runs.MaxFileBytes`), with the same 413. |
| 252 | - **Runs execute beside a live instance** (the same database, WAL). Live sync |
| 253 | relies on this. |
| 254 | - **Cancel, expire, timeout, poll-and-ack, and the 24-hour sweep.** |
| 255 | |
| 256 | ## 6. Host-claim consoles |
| 257 | |
| 258 | A console with `HostClaim` on refuses every run route with |
| 259 | `claimRunUnsupportedMsg` (`apiruns.go:26`, `:164`; `apirunsfollow.go:44`). |
| 260 | That stays true for services. A granted `Run` service gets the same refusal |
| 261 | a person does. An ungranted service gets 403 first, because the role check |
| 262 | runs in `cliRoute` (`authcli.go:455`) before any handler, and that is fine. The API docs say so: a service run works only |
| 263 | on a console without host-claim until #250 adds `cmd` to the reservation |
| 264 | protocol. Nothing here bypasses or pre-empts #250. |
| 265 | |
| 266 | Tito's console (`carlos.tito.io`) serves runs today: a follow of an unknown |
| 267 | run there answers "no such run", not the host-claim refusal (checked |
| 268 | 2026-10-24). So Fleet is not blocked by #250. |
| 269 | |
| 270 | ## 7. The caller's side (Tito; recorded here, built there) |
| 271 | |
| 272 | - **Fleet keeps the run id and key in its own store before calling,** so a |
| 273 | deploy or restart mid-run resumes by key, then follows by id, until the |
| 274 | result is acked. |
| 275 | - **The Classic DB credentials stay on the source reader**, Fleet's side. They |
| 276 | never go in argv (argv is audited) or in uploaded files, and never into a |
| 277 | customer instance. Fleet reads Classic and uploads an account-specific |
| 278 | snapshot. |
| 279 | - **A snapshot over 64 MB** is Fleet's problem to split into several runs, or |
| 280 | to refuse with a clear message. The platform's cap does not change. |
| 281 | |
| 282 | ## Tests and fences |
| 283 | |
| 284 | - The command list: |
| 285 | - an allowed prefix runs; |
| 286 | - a longer argv under it runs; |
| 287 | - an argv that only shares a substring of a word is refused; |
| 288 | - a missing word is refused; |
| 289 | - an empty list is refused at mint; |
| 290 | - rotation keeps the list; |
| 291 | - the refusal writes nothing. |
| 292 | - A role-by-route matrix: `Run` is allowed on the five routes and refused |
| 293 | everywhere else. Every other role is refused on all five. |
| 294 | - Wrong app, wrong account, revoked, and a run that isn't this service's own: |
| 295 | all refused or 404. So is a revoked name minted again: the new credential |
| 296 | cannot see the old one's runs. |
| 297 | - Idempotency: |
| 298 | - the same key and the same request give the same run with no second |
| 299 | launch, after ack, after the sweep, and after a later run has replaced it |
| 300 | in the record; |
| 301 | - a different request is a 409; |
| 302 | - two concurrent identical creates make one run, and the loser's objects |
| 303 | are deleted; |
| 304 | - another caller's key is invisible; |
| 305 | - an admitted key is answered without reading the files; |
| 306 | - a refusal is re-evaluated on retry; |
| 307 | - `run_keys` is pruned by age and count. |
| 308 | - Incarnation, one-run and move tests repeated with a service principal. |
| 309 | - Audit: a service's run renders as the service's, never as the email's. |
| 310 | - `docs/site/cli.md` gains `--idempotency-key`, and the services docs gain |
| 311 | the `run` role (AGENTS.md: a command isn't done until the docs name it). |
| 312 | |
| 313 | ## Left to the builder |
| 314 | |
| 315 | - The exact shape of `run_keys` entries. |
| 316 | - The 4xx code list. |
| 317 | - The 24-hour and 32-per-caller bounds, if the builder finds a reason to change |
| 318 | them, stated in the docs. |
| 319 | |
amadan