amadan
docs: design for runs started by a scoped service credential (PRO-6214)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+306
−0
docs/superpowers/specs/2026-10-24-service-runs-design.md+306 −0
diff --git a/docs/superpowers/specs/2026-10-24-service-runs-design.md b/docs/superpowers/specs/2026-10-24-service-runs-design.md| new file mode 100644 |
| index 000000000..dba24b645 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-24-service-runs-design.md |
| @@ -0,0 +1,306 @@ |
| +# Runs started by a service credential |
| + |
| +**Date:** 2026-10-24 |
| +**Status:** Design, awaiting approval |
| +**Owner:** Paul |
| +**Asked for by:** Tito (Linear PRO-6214). Fleet's Classic import and recurring |
| +sync need to run an app command against an instance without a person at a |
| +terminal. |
| +**Amends:** `2026-10-08-carlos-run-design.md`, § 2 ruling 1. |
| + |
| +## Why |
| + |
| +`carlos run` lets a member of an account run the app's own command against an |
| +instance, as the instance. Fleet, Tito's back office, needs exactly that for two |
| +jobs: |
| + |
| +- **The Classic import.** A person presses a button and Fleet copies one |
| + account's data in. |
| +- **The recurring sync.** It keeps an imported account up to date on a |
| + schedule. |
| + |
| +Fleet holds a server credential and no person's token. Today every run route |
| +refuses a service principal: `runGate` and `handleAPIRunCreate` check for |
| +`PrincipalUser` (`internal/console/apiruns.go:156`, |
| +`internal/console/apirunsfollow.go:36`). The work-arounds on the table are both |
| +unacceptable: |
| + |
| +- **A personal operator token in Tito's router.** A person's whole membership |
| + would sit in a service, and nobody could tell that from the audit. |
| +- **Root or systemd jobs on the box.** That goes around the platform, which is |
| + what `carlos run` exists to end. |
| + |
| +## Decisions (Paul, 2026-10-24) |
| + |
| +- **Service credentials may start and follow runs.** This reverses run spec |
| + § 2 ruling 1 ("service credentials are refused"), for the one role below |
| + only. |
| +- **A `run` credential names the commands it may run** (option A). It does |
| + not get any command the binary has. A leaked credential can then run only |
| + those, not arbitrary commands against every instance of the app. |
| +- **The person a run is "for" is recorded on the service's word.** When Fleet |
| + starts a run because someone pressed a button, the run records the name |
| + Fleet gives, labelled as stated by that service. It grants nothing and is |
| + not verified. A scheduled run records no person. |
| + |
| +## Goals |
| + |
| +- A service can create, read (progress and output), cancel, expire and ack |
| + runs on its own app's instances, and nothing else, once it has been granted |
| + that on purpose. |
| +- An ungranted service, a service from the wrong app, and a revoked service |
| + are all refused. |
| +- Deleting and recreating an account, or moving it, can never redirect a run. |
| +- A caller that loses an HTTP response can find out what happened and carry |
| + on without starting a second run. |
| +- Everything above goes through the supported console API, with no box access. |
| + |
| +## Non-goals |
| + |
| +- Host-claim consoles. They refuse runs for everyone until the reservation |
| + protocol (#250) adds `cmd` (see § 6). |
| +- Verifying who the person was. That is a later upgrade (§ 4). |
| +- A scheduler. The caller decides when to run, and Fleet owns its schedule. |
| +- Anything about what Fleet's command does: the Classic DB credentials, how |
| + snapshots are cut. See § 7 for the caller's side. |
| + |
| +## Anti-goals |
| + |
| +- **A role that quietly widens.** No existing role, `Admin` included, gains |
| + runs. A service that can provision instances cannot run commands on them |
| + unless it is separately minted for that. |
| +- **A second run from a retry.** A lost response retried must never start the |
| + command twice. |
| +- **An audit line that misleads.** A service's run must never read as if a |
| + person ran it. |
| + |
| +## 1. The capability: a new role, `run` |
| + |
| +- **`svcrole.Run` joins the closed catalogue** (`internal/svcrole/svcrole.go`). |
| + It is app-scoped, so `App` is required, refused account-wide the way |
| + `Service` is (`ErrAccountScopedService`). It carries no rungs. |
| +- **It grants exactly the five run routes** for instances of its own app: |
| + create, get (with `?after=`), cancel, expire and ack |
| + (`authcli.go:736-740`). Those routes are registered with `svcrole.Set{Run}` |
| + instead of the empty set. People keep the routes on the existing member |
| + rule, unchanged. |
| +- **No other role implies it, Admin included** (my call, holding the |
| + anti-goal). Today `svcrole.Allow` always adds Admin to a route's set |
| + (`internal/svcrole/svcrole.go:126`, pinned by |
| + `TestAdminPermitsEveryNonEmptySet`), so this IS new code. Add a constructor |
| + for an exact set (e.g. `svcrole.Exactly(Run)`) that Admin does not join, |
| + and amend that test and the Admin doc to name the one exception. A test |
| + asserts that every role except `Run`, Admin included, is refused on all five |
| + routes. |
| +- **Minted, rotated and revoked like every other service:** |
| + `carlos services create --role run --app <app> --commands "<prefix>,<prefix>" <name>`, |
| + by an account owner, through `handleAPIServiceCreate`. Revocation is the |
| + existing soft revoke, and a revoked token is refused at the door. |
| +- **The command list (Paul, 2026-10-24, option A).** A `run` credential |
| + carries a non-empty list of allowed command prefixes, stored on its |
| + `cli_tokens` row the way `Deploy` carries rungs: an additive column, parsed |
| + only for this role, and refused at mint when it is empty |
| + (`ErrRunNeedsCommands`). |
| + - Each prefix is a sequence of argv words, written space-separated on the |
| + CLI. `import classic` is the words `import`, `classic`. |
| + - A run is allowed only when its argv starts with every word of at least |
| + one prefix, compared word for word and exactly: no globbing, no case |
| + folding, no substring. |
| + - Create refuses anything else with 403 `command_not_granted`, before any |
| + object is written, and with no key state either: an unkeyed refusal. |
| + - `carlos services list` shows the list. Changing it means minting a new |
| + credential (rotation keeps the list, and there is no edit door; my call). |
| + - Prefixes contain no empty word, and each word is at most 128 bytes, with |
| + at most 32 prefixes (the builder may tune these). |
| + - People are not affected: members keep running any command. |
| +- **A service sees and acts on only the runs it started** (my call). The run |
| + records the CREDENTIAL's id (its `cli_tokens` row), not its name, because a |
| + revoked name can be minted again (`authstore.go:250`) and must not inherit |
| + the old credential's runs. Rotation keeps the row, so a rotated credential |
| + keeps its runs. Get, cancel, expire and ack check that id. A run it didn't |
| + start answers 404, the same as an unknown id, so its existence doesn't leak. |
| + Members still see every run on their account, the service's included. |
| +- **Create's "already has a run open" refusal is redacted for a service** |
| + (my call). Today it names the open run's id and the member's email |
| + (`apiruns.go:32`, `:202`). A service gets the refusal and its code without |
| + either, unless the open run is its own. |
| +- **Rate budgets are unchanged:** create spends `s.cli.instances` and the |
| + follow routes spend `s.cli.runs`, both keyed on the token hash |
| + (`apiruns.go:168`, `apirunsfollow.go:48`). A service's runs therefore draw on |
| + that credential's own budget, which is separate from Fleet's provisioning |
| + credential. |
| + |
| +## 2. What a service's run records |
| + |
| +`runs.Request` (`internal/runs/runs.go:166`) gains fields, additively, so old |
| +readers ignore them: |
| + |
| +| Field | For a person's run | For a service's run | |
| +|---|---|---| |
| +| `by` | the member's email (today) | what `auditActor` writes for a service today, `service:<account sqid>/<name>` (`apicli.go:105`) | |
| +| `by_service` | empty | the service name, for display | |
| +| `by_credential` | empty | the credential's id, which the own-runs check uses | |
| +| `on_behalf_of` | empty | the email the service gave, or empty for a scheduled run | |
| +| `idempotency_key` | optional (§ 3) | optional, and expected from Fleet | |
| + |
| +Every place the run is shown or audited (the CLI's follow output, the |
| +console's run history, the account's audit log) shows a service's run as |
| +"run by service `<name>` for `<email>` (as stated by the service)", or "run by |
| +service `<name>` (scheduled)". It is never shown as if the email ran it. |
| + |
| +`on_behalf_of` is validated for shape only: it must be an email, at most 254 |
| +bytes. It is not checked against membership, because it grants nothing. |
| + |
| +## 3. Lost responses: idempotent create |
| + |
| +Run ids are minted by the console (`runs.MintID`), so today a caller who loses |
| +the create response can't tell "started" from "never arrived". So: |
| + |
| +- **Create accepts an `Idempotency-Key` header**: 1–128 bytes of |
| + `[A-Za-z0-9._:-]`, chosen by the caller. |
| +- **A key is scoped to its caller** (my call): to the credential id for a |
| + service, to the user for a person, and within that to account, app and |
| + instance. One caller can never find, probe or collide with another's key. |
| +- **The digest covers the meta part only:** argv, timeout, `on_behalf_of`, |
| + and each file's name, size and sha256. A keyed create MUST declare each |
| + file's sha256 in the meta. The console verifies the uploaded bytes against |
| + it and refuses a mismatch. The meta is the first multipart part, so the key |
| + is checked straight after parsing it, before any other refusal and before |
| + any file byte is read. A retry of a known key is answered without reading |
| + the files (the console stops reading the body). Fleet doesn't re-upload |
| + 64 MB to learn that its run exists. |
| +- **The key object carries its own state,** changed only by conditional |
| + writes: |
| + - **`pending` → `admitted`.** The create handler writes `pending` (with the |
| + run id it minted and the digest) create-only, before any run object. It |
| + moves to `admitted` once the admission CAS commits. Admission refused for a |
| + reason (a 4xx) moves it to `refused`, recording the code. |
| + - **The key keeps that answer** (my call, the usual idempotency semantics). |
| + An `admitted` key returns that run, `200`, whatever state the run is in |
| + later, acked or swept included: the command never runs again under that |
| + key. A `refused` key returns the same refusal and code; the caller uses a |
| + new key to try again. |
| + - **The same key with a different digest** is `409` |
| + `idempotency_key_reused`, and nothing is written. |
| + - **A duplicate that finds `pending`** gets `409` `in_progress`, with |
| + `Retry-After`. Two identical creates at once therefore produce one run, |
| + never two. |
| + - **Abandoned `pending`.** A create handler runs under a deadline (the |
| + builder sets it, about 2 minutes) shorter than the abandon threshold (the |
| + builder sets it, about 5 minutes). So a `pending` key older than the |
| + threshold belongs to a handler that has certainly stopped. The next |
| + caller to find it settles it the way `apiruns.go:279-291` settles a lost |
| + admission: |
| + - if the record shows the key's run admitted, mark it `admitted`; |
| + - otherwise CAS it to `abandoned`, and proceed as a fresh create, with a |
| + new run id, overwriting the key by CAS from `abandoned`. |
| + |
| + The record alone can't tell "never admitted" from "admitted then acked", |
| + which is why the key's own state, not the record, is what answers a |
| + retry. |
| +- **Finding a run by key** without retrying the write: |
| + `GET .../instances/{host}/runs?key=<key>` returns the key's state and run id, |
| + or 404. Same caller scoping. |
| +- **How long a key lasts:** seven days from the create (the builder may tune |
| + it; record it in the API docs). After that a retry is a new run, so a |
| + caller must not retry a week-old key. The key objects get their own sweep, |
| + fenced by age, which removes only keys older than that. |
| +- **Rejected is distinguishable from unknown:** |
| + - every refusal is a 4xx with a stable `code` in the JSON body. Unkeyed, it |
| + means nothing was written. Keyed, it is also what the key will answer |
| + from now on. |
| + - a 5xx, a timeout or a dropped connection means unknown: retry with the |
| + same key. |
| + |
| + The codes are listed in the API docs and pinned by a test. |
| + |
| +People may send the key too, and the CLI gains `--idempotency-key`. Nothing |
| +changes for a person who doesn't. |
| + |
| +## 4. Upgrading "on the service's word" later |
| + |
| +If the person ever needs to be verified, the request gains a signed assertion |
| +from the caller's identity system, and the console adds `on_behalf_verified: |
| +true` when it checks out. The fields above stay, so nothing already recorded |
| +changes meaning. Not built now. |
| + |
| +## 5. What stays exactly as it is |
| + |
| +These were already built in the run series, and must be shown to hold for a |
| +service's run too, each by a test that starts the run with a `Run` service: |
| + |
| +- **Incarnation binding.** The incarnation is taken at create |
| + (`apiruns.go:250`) and checked by the box before launch. A run created |
| + before an account is deleted and recreated, or moved, never launches |
| + against the new one. |
| +- **One run per instance**, through the `ActiveRun` compare-and-swap. This |
| + covers a service's run and a person's run on the same instance too. |
| +- **The 64 MB file cap** (`runs.MaxFileBytes`), with the same 413. |
| +- **Runs execute beside a live instance** (the same database, WAL). Live sync |
| + relies on this. |
| +- **Cancel, expire, timeout, poll-and-ack, and the 24-hour sweep.** |
| + |
| +## 6. Host-claim consoles |
| + |
| +A console with `HostClaim` on refuses every run route with |
| +`claimRunUnsupportedMsg` (`apiruns.go:26`, `:164`; `apirunsfollow.go:44`). |
| +That stays true for services. A granted `Run` service gets the same refusal |
| +a person does. An ungranted service gets 403 first, because the role check |
| +runs in `cliRoute` (`authcli.go:455`) before any handler, and that is fine. The API docs say so: a service run works only |
| +on a console without host-claim until #250 adds `cmd` to the reservation |
| +protocol. Nothing here bypasses or pre-empts #250. |
| + |
| +Tito's console (`carlos.tito.io`) serves runs today: a follow of an unknown |
| +run there answers "no such run", not the host-claim refusal (checked |
| +2026-10-24). So Fleet is not blocked by #250. |
| + |
| +## 7. The caller's side (Tito; recorded here, built there) |
| + |
| +- **Fleet keeps the run id and key in its own store before calling,** so a |
| + deploy or restart mid-run resumes by key, then follows by id, until the |
| + result is acked. |
| +- **The Classic DB credentials stay on the source reader**, Fleet's side. They |
| + never go in argv (argv is audited) or in uploaded files, and never into a |
| + customer instance. Fleet reads Classic and uploads an account-specific |
| + snapshot. |
| +- **A snapshot over 64 MB** is Fleet's problem to split into several runs, or |
| + to refuse with a clear message. The platform's cap does not change. |
| + |
| +## Tests and fences |
| + |
| +- The command list: |
| + - an allowed prefix runs; |
| + - a longer argv under it runs; |
| + - an argv that only shares a substring of a word is refused; |
| + - a missing word is refused; |
| + - an empty list is refused at mint; |
| + - rotation keeps the list; |
| + - the refusal writes nothing. |
| +- A role-by-route matrix: `Run` is allowed on the five routes and refused |
| + everywhere else. Every other role is refused on all five. |
| +- Wrong app, wrong account, revoked, and a run that isn't this service's own: |
| + all refused or 404. So is a revoked name minted again: the new credential |
| + cannot see the old one's runs. |
| +- Idempotency: |
| + - same key and same request gives the same run with no second launch, even |
| + after ack and after the sweep; |
| + - a different request is a 409; |
| + - two concurrent identical creates make one run; |
| + - an abandoned `pending` key is settled both ways (admitted, and never |
| + admitted); |
| + - a refused key repeats its refusal; |
| + - another caller's key is invisible; |
| + - a known key is answered without reading the files; |
| + - a key past its lifetime is swept and then starts a new run. |
| +- Incarnation, one-run and move tests repeated with a service principal. |
| +- Audit: a service's run renders as the service's, never as the email's. |
| +- `docs/site/cli.md` gains `--idempotency-key`, and the services docs gain |
| + the `run` role (AGENTS.md: a command isn't done until the docs name it). |
| + |
| +## Left to the builder |
| + |
| +- Where key objects live in the bucket, and their exact shape (create-only, |
| + swept by time). |
| +- The 4xx code list. |
| +- The seven-day lifetime, if the builder finds a reason to change it, stated |
| + in the docs. |