Runs started by a service credential

Date: 2026-10-24 Status: Design, awaiting approval Owner: Paul Asked for by: Tito (Linear PRO-6214). Fleet's Classic import and recurring sync need to run an app command against an instance without a person at a terminal. Amends: 2026-10-08-carlos-run-design.md, § 2 ruling 1.

Why

carlos run lets a member of an account run the app's own command against an instance, as the instance. Fleet, Tito's back office, needs exactly that for two jobs:

• The Classic import. A person presses a button and Fleet copies one account's data in.
• The recurring sync. It keeps an imported account up to date on a schedule.

Fleet holds a server credential and no person's token. Today every run route refuses a service principal: runGate and handleAPIRunCreate check for PrincipalUser (internal/console/apiruns.go:156, internal/console/apirunsfollow.go:36). The work-arounds on the table are both unacceptable:

• A personal operator token in Tito's router. A person's whole membership would sit in a service, and nobody could tell that from the audit.
• Root or systemd jobs on the box. That goes around the platform, which is what carlos run exists to end.

Decisions (Paul, 2026-10-24)

• Service credentials may start and follow runs. This reverses run spec § 2 ruling 1 ("service credentials are refused"), for the one role below only.
• A run credential names the commands it may run (option A). It does not get any command the binary has. A leaked credential can then run only those, not arbitrary commands against every instance of the app.
• The person a run is "for" is recorded on the service's word. When Fleet starts a run because someone pressed a button, the run records the name Fleet gives, labelled as stated by that service. It grants nothing and is not verified. A scheduled run records no person.

Goals

• A service can create, read (progress and output), cancel, expire and ack runs on its own app's instances, and nothing else, once it has been granted that on purpose.
• An ungranted service, a service from the wrong app, and a revoked service are all refused.
• Deleting and recreating an account, or moving it, can never redirect a run.
• A caller that loses an HTTP response can find out what happened and carry on without starting a second run.
• Everything above goes through the supported console API, with no box access.

Non-goals

• Host-claim consoles. They refuse runs for everyone until the reservation protocol (#250) adds cmd (see § 6).
• Verifying who the person was. That is a later upgrade (§ 4).
• A scheduler. The caller decides when to run, and Fleet owns its schedule.
• Anything about what Fleet's command does: the Classic DB credentials, how snapshots are cut. See § 7 for the caller's side.

Anti-goals

• A role that quietly widens. No existing role, Admin included, gains runs. A service that can provision instances cannot run commands on them unless it is separately minted for that.
• A second run from a retry. A lost response retried must never start the command twice.
• An audit line that misleads. A service's run must never read as if a person ran it.

1. The capability: a new role, run

• svcrole.Run joins the closed catalogue (internal/svcrole/svcrole.go). It is app-scoped, so App is required, refused account-wide the way Service is (ErrAccountScopedService). It carries no rungs.
• It grants exactly the five run routes for instances of its own app: create, get (with ?after=), cancel, expire and ack (authcli.go:736-740). Those routes are registered with svcrole.Set{Run} instead of the empty set. People keep the routes on the existing member rule, unchanged.
• No other role implies it, Admin included (my call, holding the anti-goal). Today svcrole.Allow always adds Admin to a route's set (internal/svcrole/svcrole.go:126, pinned by TestAdminPermitsEveryNonEmptySet), so this IS new code. Add a constructor for an exact set (e.g. svcrole.Exactly(Run)) that Admin does not join, and amend that test and the Admin doc to name the one exception. A test asserts that every role except Run, Admin included, is refused on all five routes.
• Minted, rotated and revoked like every other service: carlos services create --role run --app <app> --commands "<prefix>,<prefix>" <name>, by an account owner, through handleAPIServiceCreate. Revocation is the existing soft revoke, and a revoked token is refused at the door.
• The command list (Paul, 2026-10-24, option A). A run credential carries a non-empty list of NON-EMPTY allowed command prefixes. A zero-word prefix would match every argv, so it is refused at mint, refused when the credential is read back, and never matches in the matcher. The list is stored on its cli_tokens row the way Deploy carries rungs: an additive column, parsed only for this role, and refused at mint when it is empty (ErrRunNeedsCommands). - Each prefix is a sequence of argv words, written space-separated on the CLI. import classic is the words import, classic. - A run is allowed only when its argv starts with every word of at least one prefix, compared word for word and exactly: no globbing, no case folding, no substring. - Create refuses anything else with 403 command_not_granted, before any object is written, and with no key state either: an unkeyed refusal. - carlos services list shows the list. Changing it means minting a new credential (rotation keeps the list, and there is no edit door; my call). - Prefixes contain no empty word, and each word is at most 128 bytes, with at most 32 prefixes (the builder may tune these). - People are not affected: members keep running any command.
• A service sees and acts on only the runs it started (my call). The run records the CREDENTIAL's id (its cli_tokens row), not its name, because a revoked name can be minted again (authstore.go:250) and must not inherit the old credential's runs. Rotation keeps the row, so a rotated credential keeps its runs. Get, cancel, expire and ack check that id. A run it didn't start answers 404, the same as an unknown id, so its existence doesn't leak. Members still see every run on their account, the service's included.
• Create's "already has a run open" refusal is redacted for a service (my call). Today it names the open run's id and the member's email (apiruns.go:32, :202). A service gets the refusal and its code without either, unless the open run is its own.
• Rate budgets are unchanged: create spends s.cli.instances and the follow routes spend s.cli.runs, both keyed on the token hash (apiruns.go:168, apirunsfollow.go:48). A service's runs therefore draw on that credential's own budget, which is separate from Fleet's provisioning credential.

2. What a service's run records

runs.Request (internal/runs/runs.go:166) gains fields, additively, so old readers ignore them:

| Field | For a person's run | For a service's run | |---|---|---| | by | the member's email (today) | what auditActor writes for a service today, service:<account sqid>/<name> (apicli.go:105) | | by_service | empty | the service name, for display | | by_credential | empty | the credential's id, which the own-runs check uses | | on_behalf_of | empty | the email the service gave, or empty for a scheduled run | | idempotency_key | the key's scoped hash (§ 3), when one was sent | the key's scoped hash, expected from Fleet |

Every place the run is shown or audited (the CLI's follow output, the console's run history, the account's audit log) shows a service's run as "run by service <name> for <email> (as stated by the service)", or "run by service <name> (scheduled)". It is never shown as if the email ran it.

on_behalf_of is validated for shape only: it must be an email, at most 254 bytes. It is not checked against membership, because it grants nothing.

3. Lost responses: idempotent create

Run ids are minted by the console (runs.MintID), so today a caller who loses the create response can't tell "started" from "never arrived". So:

• Create accepts an Idempotency-Key header: 1–128 bytes of [A-Za-z0-9._:-], chosen by the caller.
• A key is scoped to its caller (my call): to the credential id for a service, to the user for a person, and within that to the instance. One caller can never find, probe or collide with another's key.
• The proof that a key was admitted lives in the instance record, written by the same compare-and-swap that admits the run (my call, after Astra's review). The record gains run_keys: a short list of recent admissions, each with a hash of (caller scope, key), the run id, the request digest, a hash of the caller scope (which the per-caller cap counts by) and the time. provision.RunAdmit refuses to admit when the list already holds this caller's key, and returns that run instead. - Admission and dedupe are therefore one atomic write. Two identical creates at once can both write their objects, but only one is admitted. The other finds its key already in the record, deletes its own objects and answers with the winner's run. - There is no separate key object, so there is no pending or abandoned state, no sweep, and no crash window between "admitted" and "key recorded". - Ack, the run sweep and the next run replacing a finished one all keep run_keys, so a retry after any of them still finds the run.
• The digest covers the meta part only: argv, timeout, on_behalf_of, and each file's name, size and sha256. A keyed create must declare each file's sha256 in the meta, and the console verifies the uploaded bytes against it. The meta is the first multipart part, so the key is checked against the record straight after parsing it, before the open-run, moving and import refusals and before any file byte is read. A retry of an admitted key is answered without reading the files.
• The same key with a different digest is 409 idempotency_key_reused, and nothing is written.
• Answers to a keyed create: - a fresh admission and a replay of an admitted key both return 202, the status create gives today, with the same body, plus "replayed": true on a replay; - GET .../instances/{host}/runs?key=<key> returns the run id from the record, or 404.
• Refusals are not remembered (my call). Only an admitted run is. - A 4xx means THIS attempt admitted nothing. An earlier attempt with the same key may already have run: the shared gates (401, the rate budget's 429) come before the key lookup, so the caller confirms with GET ...?key= before treating a 4xx on a retry as "never ran". - A retry with the same key is evaluated afresh: a run_open refusal can succeed once the other run has finished, which is what the caller asked for. - A 5xx, a timeout or a dropped connection means unknown, and the caller retries with the same key.
• How long a key lasts: each admission prunes run_keys to entries younger than 24 hours (runs.KeyTTL), then keeps the newest 32 entries of each caller (runs.MaxRunKeysPerCaller). The cap is per caller, so one caller's admissions can never evict another's key. A key is honoured for 24 hours, or until that same caller has made 32 newer keyed admissions on this instance, whichever comes first. A caller must not retry a key past that. Fleet retries within minutes.
• request.json records the key as idempotency_key: its scoped hash, never the caller's own words. It links the run to its key after the record has pruned the entry.
• Mixed versions: an older box binary rewriting the record would drop run_keys. Keyed creates stay off (503 keys_not_ready) until the console is told, by config, that every record writer preserves the field.
• A known limit: deleting an instance deletes its record, run_keys with it. A retry that arrives after the host was deleted and created again is a fresh run on the new incarnation. It is still only a command the credential was granted, on its own app.
• Refusal codes: the run handlers' own refusals carry a stable code beside error, and the API docs list them. The shared gates before the handler keep their bodies exactly as today, deliberately identical across doors (AGENTS.md): not signed in, role mismatch, and the canonical membership 404. Those already distinguish themselves by status.

People may send the key too, and the CLI gains --idempotency-key. Nothing changes for a person who doesn't.

4. Upgrading "on the service's word" later

If the person ever needs to be verified, the request gains a signed assertion from the caller's identity system, and the console adds on_behalf_verified: true when it checks out. The fields above stay, so nothing already recorded changes meaning. Not built now.

5. What stays exactly as it is

These were already built in the run series, and must be shown to hold for a service's run too, each by a test that starts the run with a Run service:

• Incarnation binding. The incarnation is taken at create (apiruns.go:250) and checked by the box before launch. A run created before an account is deleted and recreated, or moved, never launches against the new one.
• One run per instance, through the ActiveRun compare-and-swap. This covers a service's run and a person's run on the same instance too.
• The 64 MB file cap (runs.MaxFileBytes), with the same 413.
• Runs execute beside a live instance (the same database, WAL). Live sync relies on this.
• Cancel, expire, timeout, poll-and-ack, and the 24-hour sweep.

6. Host-claim consoles

A console with HostClaim on refuses every run route with claimRunUnsupportedMsg (apiruns.go:26, :164; apirunsfollow.go:44). That stays true for services. A granted Run service gets the same refusal a person does. An ungranted service gets 403 first, because the role check runs in cliRoute (authcli.go:455) before any handler, and that is fine. The API docs say so: a service run works only on a console without host-claim until #250 adds cmd to the reservation protocol. Nothing here bypasses or pre-empts #250.

Tito's console (carlos.tito.io) serves runs today: a follow of an unknown run there answers "no such run", not the host-claim refusal (checked 2026-10-24). So Fleet is not blocked by #250.

7. The caller's side (Tito; recorded here, built there)

• Fleet keeps the run id and key in its own store before calling, so a deploy or restart mid-run resumes by key, then follows by id, until the result is acked.
• The Classic DB credentials stay on the source reader, Fleet's side. They never go in argv (argv is audited) or in uploaded files, and never into a customer instance. Fleet reads Classic and uploads an account-specific snapshot.
• A snapshot over 64 MB is Fleet's problem to split into several runs, or to refuse with a clear message. The platform's cap does not change.

Tests and fences

• The command list: - an allowed prefix runs; - a longer argv under it runs; - an argv that only shares a substring of a word is refused; - a missing word is refused; - an empty list is refused at mint; - rotation keeps the list; - the refusal writes nothing.
• A role-by-route matrix: Run is allowed on the five routes and refused everywhere else. Every other role is refused on all five.
• Wrong app, wrong account, revoked, and a run that isn't this service's own: all refused or 404. So is a revoked name minted again: the new credential cannot see the old one's runs.
• Idempotency: - the same key and the same request give the same run with no second launch, after ack, after the sweep, and after a later run has replaced it in the record; - a different request is a 409; - two concurrent identical creates make one run, and the loser's objects are deleted; - another caller's key is invisible; - an admitted key is answered without reading the files; - a refusal is re-evaluated on retry; - run_keys is pruned by age and count.
• Incarnation, one-run and move tests repeated with a service principal.
• Audit: a service's run renders as the service's, never as the email's.
• docs/site/cli.md gains --idempotency-key, and the services docs gain the run role (AGENTS.md: a command isn't done until the docs name it).

Left to the builder

• The exact shape of run_keys entries.
• The 4xx code list.
• The 24-hour and 32-per-caller bounds, if the builder finds a reason to change them, stated in the docs.