🤖 CARLOS platform

> 🤖 This document was written by an LLM (Claude), on the ideas, > instruction, and editing of humans, per > factor X.

A black-box deployment platform for CARLOS apps: carlos ship publishes an immutable, content-addressed release; the platform places it on instances, routes to it through a shared edge, and surrounds it with platform-level glue. One deployment is one object-store bucket + N hosts running the carlos binary + DNS — the whole self-hosting story in one sentence. See the 2026-08-01-carlos-platform-design.md design spec (private platform-infrastructure repo) for the full design.

Build & test

make build # CGO_ENABLED=0 build of the carlos binary make test # go vet, gofmt -l, go test ./...

One static binary plays every role, selected by subcommand: carlos edge, carlos agent (edge + activator + release adoption + fleet status, one process per host), carlos ship/promote/rollback, carlos vet. Run carlos with no arguments for the full command list.

Platforms

Linux and macOS run every role — box and workstation alike — and install from apt (sudo apt install carlos) or Homebrew (brew install carlosframework/tap/carlos).

Windows is a client-only target. A Windows workstation can carlos login, carlos ship, carlos deploy, carlos logs — everything that talks to a console or a bucket over the network — against a Linux fleet. It cannot BE a box: carlos agent and the root-side passes need unix sockets, flock and systemd, and the packages that provide them compile on Windows (the CLI is the same binary) but refuse at their per-OS boundary rather than pretending. See internal/edge/purge_listen_windows.go and internal/host/{statfs,flock}_windows.go for what that refusal means in each case.

Two consequences worth knowing before you install it:

• carlos update does not self-replace on Windows. It still checks the release feed and tells you when you are behind, then hands you the URL of the zip to unpack over your existing binary. Windows cannot rename a new binary over a running .exe, and a self-replace nobody here can test is worse than an honest manual step.
• Your credentials file is not permission-protected on Windows. On Linux and macOS ~/.carlos/credentials is written 0600; the equivalent mode on %USERPROFILE%\.carlos\credentials means nothing, so any other user of that machine can read your session. Do not carlos login on a shared Windows machine.

Install by downloading carlos-windows-amd64.zip (or -arm64) from carlosframework/releases, unpacking carlos.exe somewhere on your PATH, and verifying its checksum against the release's signed SHA256SUMS.txt.

Accounts

An account is the tenancy primitive: a project or team that owns apps and has members. Kass, Keymail and Woodstar each get their own rather than sharing one flat list, and a self-hosted single-app deployment can give team A and team B separate accounts that deploy independently.

An account is {id, sqid, name}. The id is the next free integer, allocated at creation by a create-only write of accounts/<sqid>/account.json in the deployment bucket; the registry only ever materializes what the bucket says, never assigns. That is what makes account identity survive a registry rebuild and be identical on every box: minted once, recorded in the bucket, never reassigned. Sequential ids keep sqids short enough to live in a hostname you can read out (kass.bac.oncarlos.com); they are not secret, and never were — every platform hostname's certificate is published in Certificate Transparency. The sqid is the frozen encoder over that id and is the account's public identity, in every platform domain and every bucket key. The name is free text and deliberately not unique: two accounts may both be called "Blog", because identity is the sqid and there is nothing to squat.

ops is the built-in operator account — id 1, sqid bab — created by migration on every deployment. CLI-added apps live there by default, so a console-less self-hoster is unaffected by any of this.

Roles. Exactly two:

| Role | Can | |---|---| | owner | everything a member can, plus invite, remove members, rename | | member | deploy, promote, rollback, create/retire canaries, view everything |

Every account has at least one owner; the last one can be neither removed nor demoted (transfer first).

In the console. The account switcher sits top left on every account page, where the wordmark used to be: a dropdown whose trigger names the account you are in, listing the accounts you are a member of and linking to the New account screen at /accounts/new. It is a <details> disclosure, so it is keyboard-operable and works with scripting off. URLs are explicit and shareable — /a/<sqid>/apps/<app> — so a link a teammate opens lands where you meant; / redirects to the account you were last in (or your first membership, or one minted on the spot). A signed-in non-member asking for an account URL gets the same 404 a nonexistent page gets: account existence is not probeable.

Invites go into the addressee's Keymail Apps inbox — the platform never emails anyone — as a single-use link that expires in 7 days and is revocable. Clicking it while signed in as that address creates the membership.

From the CLI. Every app verb takes --account, a sqid or a name, defaulting to ops:

carlos auth login --console https://platform.carlosframework.com carlos accounts create --name kass # you own it carlos accounts create --name kass --owner paul@keymail.dev # headless carlos accounts list # sqid, name, app count carlos apps create --account kass --app blog # claim the name (rate limited; # the console's New app form is not) carlos ship --account kass --app blog ... carlos add --account kass --app blog blog.example.com

Colour. The CLI colours its output when — and only when — it is writing to a terminal: a write that happened reads green, a dry run or a bypassed rail yellow, a refusal red, and context you did not ask for (the using <console> as <you> banner, "nothing to do" verdicts, follow-up hints) is dimmed. Redirect it, pipe it, or run it in CI and every byte is what it always was, so parsing scripts and test assertions are unaffected. Override with CARLOS_COLOR=always|never; NO_COLOR=1 and TERM=dumb are honoured.

Who the CLI is. carlos auth login gives the command line an identity, and that identity owns the accounts it creates. It authenticates against the deployment's own console — already a registered Keymail OAuth client and already the authority on who belongs to what — rather than minting a second OAuth client, so there is one consent screen, one place to revoke, and a self-hoster's CLI trusts their own box.

It prints a code rather than opening a browser or listening on a local port, because the machine that needs the credential is usually an SSH session where neither works. You open /auth/cli in whatever browser you are already signed in on and type the code the terminal printed.

Typing it is the point, not a nicety: a link that arrives one click from an Approve button is the device-code phishing attack (the one run at scale against Microsoft tenants) — an attacker starts the flow, sends you their link, and one click hands them a credential that acts as you. So the console never approves from a prefilled code, and the CLI never prints the ready-made link. When a token is issued, a note goes to your Keymail inbox naming the terminal, where it connected from, and how to revoke it.

The token lands in ~/.carlos/credentials (mode 0600, one entry per console origin), expires in 90 days, and is listed with a revoke button at /auth/cli/tokens.

carlos auth whoami # who this terminal is, per the console carlos auth logout # revokes the token, then forgets it

Shipping from a logged-in laptop. Once you're logged in, ship, promote, rollback, channels, and env/secrets (issue #43) speak through the console instead of the bucket directly — no CARLOS_DEPLOYMENT_BUCKET/_DIR on a developer's machine at all:

carlos ship --app blog --version $(git rev-parse --short HEAD) --target linux-arm64 blog-linux-arm64 carlos promote --app blog abc1234 canary/my-branch carlos channels --app blog

Canary and edge promote exactly as before; promoting onto stable is refused — the console names itself in the refusal, because moving stable needs a passkey step-up only a browser can do. Finish it at the URL the refusal prints.

CI uses the console too. Mint a deploy credential for the app, limited to the rungs CI may move, and give CI the secret and the console's address:

carlos services create --role deploy --app blog --rungs canary,edge ci # in CI: export CARLOSSERVICETOKEN=... CARLOSCONSOLE=https://console.example.com carlos deploy --app blog --version "$GITHUBSHA" --channel edge blog-linux-arm64

carlos deploy ships, promotes and watches the app's hosts, the same as from a laptop. The first deploy of a new app still needs a person: a deploy credential cannot create the app or set up its first host. Every promote is audited in the console and reaches the boxes within seconds. A deploy credential can ship, and promote or roll back only its rungs. It cannot reach stable, hotfix, delete the app, or change retention or domains.

Direct bucket mode is for operators. With CARLOS_DEPLOYMENT_BUCKET (or _DIR) set, every verb goes straight to the bucket and the console never sees it. Use it to set up a deployment before its console exists, to run one that has no console, to recover when the console is down, and for the platform's own releases. A scoped CARLOS_RELEASE_KEY limits that path only. The CLI refuses to run with both CARLOS_SERVICE_TOKEN and a bucket variable set.

On a box, or in CI, don't log in. There is no browser and no human, and a long-lived token sitting on a box is a standing credential nobody is watching. Name the owner outright instead — it needs no identity of its own and takes precedence over any logged-in one:

carlos accounts create --name kass --owner paul@keymail.dev

Either way an owner is named. carlos accounts create refuses with no login and no --owner: an account with no owner materializes into the console with no members, which means nobody can open it, invite to it, or see the apps migrated into it.

An exact sqid wins; otherwise a unique name match; an ambiguous name lists the candidates and refuses rather than guessing which tenant you meant.

--owner is required and is recorded in the account manifest, which the console materializes into the account's first owner membership. Membership is authorization here, so an account created without one exists, holds its apps, and cannot be opened by anybody — a state the flagship reached for real during the multi-account migration.

Bucket layout, and the migration. App objects live under apps/<account-sqid>/<app>/…, and that key path is the app→account linkage — there is no separate manifest. Every read path resolves those qualified keys and nothing else (the legacy read-fallback was retired on 2026-08-03, once both deployments were migrated). Moving an app is a verb:

carlos accounts migrate --app kass --to <sqid> --dry-run # prints the plan carlos accounts migrate --app kass --to <sqid> console migrate-app --app kass --to <sqid> # on the console's box

It copies the objects and re-stamps the app's routes. The console keeps its own app→account row — console state is the console's authority, and the carlos binary deliberately never opens its database — so console migrate-app completes the move; carlos accounts migrate prints that exact command when it finishes. Both halves are idempotent.

Upgrading from a pre-accounts binary? Your objects are all at the old account-less apps/<app>/… keys, which nothing reads any more — so the apps will go quiet rather than serve something stale, and every miss tells you so by name:

carlos accounts migrate --app <app> --from legacy --to bab # --to <sqid> for a tenant

That --from legacy source is permanent. However long a deployment sits on an old build, this is the way off those keys.

It never deletes a legacy object. Deleting them is a separate operator step, and an unhurried one now that nothing resolves through them.

Releasing the platform

A release is a vX.Y.Z tag on this repo plus signed binaries on carlosframework/releases. Everything downstream reads that one set of files: carlos update, the Homebrew tap, apt, and every deployment's carlos system update. Cut it from a green main and verify it from the public side, never from your build directory.

1. Tag. git tag -a vX.Y.Z <sha> -m "carlos vX.Y.Z" on the main commit you are releasing, then git push origin vX.Y.Z to the forge.
2. Build. Six targets, darwin, linux and windows on amd64 and arm64, each with make build GOOS=<os> GOARCH=<arch> SHA=vX.Y.Z from a clean checkout of the tag. Build with TMPDIR and GOCACHE on a real disk: /tmp is a shared tmpfs, and a full one fails the link with "no space left on device". Zip each Windows binary as a lone carlos.exe.
3. Debs. One per architecture, built with dpkg-deb --root-owner-group, holding usr/bin/carlos. Copy the last release's control and change the version and the description's last sentence. Wrap the description without breaking on hyphens, or apt shows "step- by-step".
4. Sums and signature. Write SHA256SUMS.txt over the eight files. Sign it with the ed25519 seed in SSM /carlos/cli-update-key, passed on stdin and never in argv, and write the signature base64-encoded to SHA256SUMS.txt.sig. Before signing, check that the key's public half equals cliUpdatePubKey in update.go. A mismatch publishes a release nobody can update to.
5. Release notes. Draft them, and put them through copy review before anything is published. Lead with what changed for people using the CLI. Add a short section for operators, taken from the release notes below this recipe, about anything to check before rolling.
6. Publish. gh release create vX.Y.Z -R carlosframework/releases --latest --title vX.Y.Z --notes-file NOTES.md with all ten files. Confirm it is the latest with gh api repos/carlosframework/releases/releases/latest -q .tag_name.
7. Homebrew. In carlosframework/homebrew-tap, bump Formula/carlos.rb: the version, the four url and sha256 pairs from the published SHA256SUMS.txt, and the carlos vX.Y.Z test string. Open a PR, check each hash against the sums file, and squash-merge it.
8. apt. Download the live tree, apps/bab/apt-carlosframework/releases/<last>/site.tar.gz from the flagship bucket, and unpack it. Add the two debs under pool/main/c/carlos/, and regenerate each architecture's Packages with dpkg-scanpackages --multiversion. Delete Release, InRelease and Release.gpg, then rebuild Release with apt-ftparchive release, passing Origin and Label CARLOS, Suite and Codename stable, the architectures and the component. Write it outside the tree first, or it lists itself. Sign InRelease and Release.gpg with the SSM key /carlos/apt-signing-key in a throwaway GNUPGHOME, set latest.txt to the tag, and check both signatures with gpgv against the tree's own keyring.
9. Ship apt. Ship the directory and let carlos ship build the tarball. A hand-made tar with a leading ./ entry took apt down once. With AWS_PROFILE=keymail and CARLOS_DEPLOYMENT_BUCKET set to the flagship bucket, run carlos ship -app apt-carlosframework -account-id bab -kind static -version vX.Y.Z-1 site/, then carlos promote -app apt-carlosframework -account-id bab vX.Y.Z-1 canary/apt with CARLOS_RELEASE_KEY from SSM /carlos/release-key set for that one command. The channel is the literal canary/apt.
10. Changelog. In a full checkout of main, with gh and amadan signed in, run git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*', then go run ./cmd/changelog. Map each of the new release's bullets to the branches behind it in docs/site/changelog-links.json, keyed by the bullet's bold lead, and run it again. Land docs/site/changelog.json and the links file on a branch. Then, in carlosframework/website, run node hack/sync-docs.mjs <checkout> and ship the site the way its AGENTS.md describes. The page is carlosframework.com/platform/changelog.

Then verify from the public side:

v=vX.Y.Z; rel=https://github.com/carlosframework/releases/releases/download/$v
curl -sSLO $rel/carlos-linux-arm64; curl -sSLO $rel/SHA256SUMS.txt; curl -sSLO $rel/SHA256SUMS.txt.sig
printf '\x30\x2a\x30\x05\x06\x03\x2b\x65\x70\x03\x21\x00' > pub.der
echo 'R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=' | base64 -d >> pub.der
base64 -d SHA256SUMS.txt.sig > sig.bin
openssl pkeyutl -verify -pubin -inkey pub.der -keyform DER -rawin -in SHA256SUMS.txt -sigfile sig.bin
sha256sum -c --ignore-missing SHA256SUMS.txt
curl -sS https://apt.carlosframework.com/latest.txt          # prints $v
carlos system update --console <console> --version $v         # then: carlos system status

Last, roll it, and roll dev before flagship. The order is fixed: dev boxes, then the dev console, then check dev end to end, then flagship boxes, then the flagship console. Do not start on flagship until dev has been exercised on the new build.

carlos system update --console <console> --version vX.Y.Z, then carlos system status. Every box should read current within the spread. A box that reads rolled-back or refused says why on the same line; fix that box, and do not re-cut the release.

Checking dev. The check is wakecheck, a standing hibernating app on dev. Each request writes one SQLite row and replies with its build, the box's host and the row count (hits=). It has no region, so both dev boxes serve it: carlos-dev-ie, which is the console box, and carlos-dev-ie-b. Once dev is rolled:

1. Confirm it is asleep. carlos logs --app wakecheck --console https://platform.dev.carlosframework.com --since 30m should end in "instance exited". Do not use instances list --health for this, because it always reads Asleep (#287). Idle is 15 minutes and bots can keep it awake, so wait it out.
2. Wake it on carlos-dev-ie through DNS: curl -sS https://wakecheck.dev.carlosframework.com/. A pass is a 200 whose host= is that box, in about 0.6 s to first byte.
3. Wake it on carlos-dev-ie-b by pinning its address: curl -sS --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210 https://wakecheck.dev.carlosframework.com/. A pass is a 200 whose host= is that box. It takes about 1.6 s, because it restores from S3.
4. carlos status --layout shows wakecheck as nested on both boxes.

On dev, hits= going backwards from one box to the other is not a failure. With the claim off, a wake that switches box can serve a stale local DB. That is the known hazard in #286, and it stays until the claim ruling lands. Judge the check on the 200s and the host= values, not on the count.

Three things catch people out on the roll:

• system update rolls a whole channel. It rolls every box on the channel within the spread. It cannot canary one box. To put a release on one box first, roll that box by hand with scripts/update-platform-binary.sh in platform-infrastructure.
• Old updaters need carlos-instance-control restarted. A box whose updater predates v0.26.0 does not restart carlos-instance-control when it rolls. The moment the new edge serves, restart it by hand. Until then the new agent refuses isolated wakes as "running an older build". Use a one-off watcher to do the restart, not memory.
• The dev console needs its release key first. Before dev can be rolled with system update, its console must hold the release key. That is /etc/carlos/release.env (root:root, 0600) and a console.service.d/release-key.conf drop-in that loads it.

• Refresh the embedded timezone tables when a new IANA tzdata release is out: scripts/update-tzgeo.sh <version>, then go test ./internal/geo.

A release-specific note goes here when a release changes something an operator needs to check or act on before rolling it. Read the note for the release you are about to roll.

Binary size (#275). The linux/arm64 platform binary doubled to 77.7 MB in earlier builds because SDK clients were kept whole by the linker. This release is 45.7 MB again, so boxes download about 32 MB less. CI now fails a build over 52 MB.

Pointer digests (#253): writers before readers. Once a box has verified a channel pointer that carries manifest digests, it refuses pointers for that channel without them. So digests are turned on in order. First upgrade every writer: every console, and every CLI or deploy script that promotes, rolls back or fast-tracks. Then run carlos channels digests on. Then watch carlos channels digests: a channel marked as written by an old writer means one is still in use. Rolling a writer back below this release stops it writing pointers boxes accept; carlos channels digests off suspends digests while it is fixed, but does not clear the boxes that have already seen them. When carlos channels digests lists nothing, set CARLOS_REQUIRE_POINTER_DIGEST=on in every box's host.env, so a new box refuses old pointers too.

Instance update settings: the roll order. Roll every box to the release that applies update settings before anyone turns on manual updates, a major hold or a pin: the console refuses those until every box reports it can protect the builds they name from pruning, and says how many boxes it is waiting on. Before rolling any box back below that release, turn retention off for the deployment (carlos releases retention --off for each app that has one), because an older housekeeper prunes builds a pin or a held instance still needs. After such a rollback the console shows settings not enforced for instances on the older boxes: they follow their channel until the box is rolled forward again.

v0.25.0: instances move to the nested layout on their own. This is the first release on which an isolated (nested) instance wakes, and nested is the default. On each box, migration starts unattended on the first serial move after the release lands and the box's isolation probe passes: every activator-managed instance that is asleep is moved into its own directory and namespace at once; one that is awake gets an hour and is then stopped for its move. An instance whose move is refused for a reason a tenant could have planted (a file in its new directory, a second name for its database) is held down automatically, and stays down across agent restarts until the reason is cleared or the layout is reverted. An operator who wants control over any of that acts BEFORE rolling: set CARLOS_LAYOUT_DEFAULT=flat in each box's host.env to keep the fleet flat, or record carlos system layout migrate --box <label> to move one box at a time, or carlos system layout revert to opt the fleet out; a recorded intent always wins over the default. Watch with carlos status --layout, which marks a box (nested by default) and says why a box the default has not reached is still flat. Cut v0.25.0 from a main that contains 7b7d3026 (plan 4's edge openat2 fix); without it nested instances cannot wake.

App regions rollback floor. Once any app holds a region other than the home, do not roll the console binary below the first release that understands app regions: an older console places a no-region create at the home. That release's version is named here, in its release notes, and in the console roll recipe in carloku-infrastructure; check all three name the same version before turning CONSOLE_APP_REGIONS on. If a rollback below the floor happens anyway, write down when it started and ended: on its next start the console logs "possible placement discrepancy" lines, and the interval tells you which to review. Turning the switch off is the feature's own rollback: it stops new non-home regions and keeps existing ones in force.

The first release that understands app regions is v0.23.0.

Turning app regions on. In order:

1. Stage the deployment's CARLOS_SECRETS_KEY on every box in every region that should take placed instances, and set CARLOS_SECRETS_PUBKEY on the console. A box without the key refuses to mint or wake placed exec instances whose config contains sealed secrets.
2. Roll the box release that publishes serves_shared_pool, accepts_inbound and secrets_key_fp. Check every region's control/edges/*.json carries them, and that each secrets_key_fp equals the console's.
3. Roll the console with the switch off.
4. On dev, turn the switch on. Create an app and an instance, read their region, and run the service gate below for every region.
5. On production, with the switch still off, create one probe instance per eligible region with an explicit --region, deploy a known build, and run scripts/region-service-gate.sh <probe-host> <build> <every other edge EIP> for each. Every region must pass. Delete the probes.
6. Turn the switch on. Re-run step 5 after any period with the switch off, and whenever a new region becomes eligible.

#292: instance-control changes. The release carrying #292 changes carlos-instance-control: it orders a host's legacy run starts against its stops. So the service must restart on roll. A box whose updater is v0.26.0 or later does that itself, so no manual step is needed; an older updater needs the hand restart described under "Old updaters" above.

Tenant email senders, plan 2 of 5 (2026-09-21)

This release turns the one implicit per-app email sender into an explicit sender record (design spec 2026-09-15-tenant-email-senders-design.md).

• SES event counts now attribute to the sender that sent them. an app sharing a domain with a sibling app stops being charged for the sibling's sends. a tenant watching their own numbers will see them move.
• the app's email record moved to apps/<acct>/<app>/email/senders/app.json, with a read fallback to the old email.json. a console rolled back after the first sender write loses any pause or rotation made in between. that has to be redone by hand.
• two prerequisites from the design spec (§4.2) before any release sends with a tenant header. - every domain provisioned before this release has no SES tenant association, and nothing will create one on its own. a backfill, or a re-run of carlos email domains add against every existing domain, is needed before any release that sends with a tenant header. - the control-plane role needs ses:CreateTenantResourceAssociation and ses:DeleteTenantResourceAssociation before this rolls.
• CARLOS_MAIL_LEGACY_SWEEP=1 is off by default. it does two separate things. it deletes the bucket's legacy email.json objects, and it drops the database's mail_counts table. do not set it until the release after this one has been live for a while.

Tenant email senders, plan 4 of 5 (2026-09-22)

This release adds the per-instance config layer (design spec 2026-09-15-tenant-email-senders-design.md §6.2): a third config bundle, layered over an app's default and named-environment config, scoped to one route's host rather than the whole app.

• a new failure mode comes with this layer. every config sweep now also lists apps/<acct>/<app>/instances-config/ before it converges that app's routes. s3 throttling, an iam policy change that adds a prefix condition to s3:ListBucket, or a transient 503 on that call all look the same from a box: the listing fails, and every route of that app freezes on its last-good config file until the listing succeeds again. this holds for the whole app, not just routes with an overlay of their own, because the listing runs once per app regardless of who uses the layer.
• grep the box's log for instance config unresolved to find it. it prints once per app, naming the account, the app and one route the pass reached. every other route of that app is covered by the same line and the same cause.
• it clears itself. once the listing succeeds again, the next pass converges normally, with nothing to clean up by hand.
• the in-repo iam policies grant s3:ListBucket on the whole deployment bucket with no prefix condition, so a platform box is unaffected by this on its own account's policy. check a hand-edited or self-hosted deployment's policy before this rolls there, since a narrower grant is exactly what would trip it.

Availability, and why the edge had no answer (2026-09-24)

This release moves every 502, 503 and 504 the edge sees from the Errors tab to a new Availability tab, grouped by cause (errors spec 2026-09-07-errors-and-analytics-design.md §11), and adds replica placements (#409).

• roll the boxes and the console together. the boxes record causes and the console shows them. the console alone gives the Availability tab, and 502s already stored regroup there under "cause not recorded"; until the boxes roll, new 502s land there too. boxes alone, with an old console, file new records the old console lists on Errors, one row per cause, named like "502 · the instance refused the connection".
• rolling the console back is safe. nothing in the bucket is rewritten; the regrouping happens on read.
• the Errors tab and carlos errors show fewer rows. edge 502/503/504 and every platform-stream line (failed wakes, exits, sidecar restarts) are on Availability now. anyone watching Errors for wake failures should watch Availability instead.
• a request whose visitor gives up after waiting five seconds or more, before the instance has sent response headers, is now filed as an outage, as "the visitor gave up waiting for the instance". the five seconds count from when the edge received the request, so on a hibernating route they include the wake. an endpoint that holds its response headers longer than that by design, like a long poll, will show a standing row for it. that row is expected; it is not a regression. a slow export that sends its headers straight away does not.
• a visitor who leaves within five seconds is no longer filed at all. previously those were filed as a 502 on whatever path they were loading, so expect fewer single-count 502s on static assets.
• on a route this box proxies over a tunnel or to a fleet, the front box now files only its own hop failures. the box serving the app files the rest, so a tunnelled 500 is filed once, not twice. two edges follow from that: a visitor who leaves after five seconds can be filed as "gave up waiting" on both boxes, and a far agent whose status store failed to open files nothing, which the front box no longer covers.
• a new CLI against a console that has not been updated: - carlos availability fails with not found: check --account and --app, and that your token's address is a member. until the console rolls, that message means the console predates the command, not that the app or your membership is missing. - carlos instances create --role replica is silently ignored by an old console, which creates an ordinary home instance in that region instead. update the console before creating a replica.
• replica placements add two registry columns (routes.replicas, routes.replica) on each box's start, by an idempotent ALTER TABLE. there is no migration to run, and a box rolled back keeps two unused columns harmlessly. do not create a replica until every box is on this release: only this release's DNS steering leaves replicas out of latency routing.

Seven days of history for sleeping instances (2026-10-08)

This release keeps seven days of database snapshots for instances that sleep, the same as instances that stay awake (#263). carlos db pull --at (#262) can then reach back a week on them, not a day.

• boxes only. the console is unaffected.
• the window fills in over the week after the roll. snapshots older than a day were already deleted under the old setting, so on the day of the roll a sleeping instance still reaches back about a day, and a week only seven days later.
• an instance's history moves only while it is awake. one that sleeps for several days takes no snapshot in that time, and loses none either: retention also runs only while it is awake.
• storage grows by up to six more daily snapshots per sleeping instance, plus the hourly files back to the oldest one. there are no extra s3 requests.

Bucket reads only when something changed (2026-10-22)

This release stops every box and the console from reading the whole deployment bucket on every pass (#336). Writers now record which app changed in control/changes.json, and readers re-read only those apps. A full read of the bucket still runs every 30 minutes as a safety net.

• roll every box and the console before you arm any reader. writers update the index as soon as they run the new build, and readers stay off until you arm them.
• arm a box by adding CARLOS_CHANGE_INDEX=read to its host.env and restarting it, and the console the same way in its environment. do dev first, and boxes before the console.
• to turn it off, remove the variable and restart. readers go back to reading the whole bucket, and writers keep updating the index, which does no harm.
• anyone using an older carlos CLI that writes straight to the bucket (promote, env set, trash) should update it. until they do, their changes reach boxes at the next 30-minute read, not within seconds.
• customer fleet boxes keep reading the way they do today.
• to see the effect, look at daily S3 ListBucket and GetObject requests for the deployment bucket in Cost Explorer. they should fall within a day of arming.

Email sending users get a permissions boundary (2026-10-23)

This release creates every email sending user (IAM path /carlos-mail/) under a permissions boundary, the way store users already are. Without one, anything that can act as the console's role can create a user on that path, give it an administrator policy and mint it a key.

A console with CARLOS_MAIL_REGION set refuses to start on this release without CARLOS_MAIL_USER_BOUNDARY. Its value is the ARN of a customer managed policy in the same account, arn:aws:iam::<12-digit account>:policy/<name>, that allows ses:SendRawEmail and nothing else. A console without CARLOS_MAIL_REGION needs nothing.

Roll it in this order:

1. create the boundary policy. set CARLOS_MAIL_USER_BOUNDARY to its ARN in every console's environment before any console gets this release. an older console ignores the variable. this release refuses to start without it.
2. apply IAM step 1 to the console role. add iam:PutUserPermissionsBoundary on user/carlos-mail/*, conditioned on iam:PermissionsBoundary equal to the boundary ARN. without that condition the role can set any boundary it likes. add iam:ListUsers for the sweep in step 4. AWS does not scope ListUsers to a path, so it is granted on Resource "*". it only reveals user names and paths, and the sweep asks only for /carlos-mail/. the role already has iam:GetUser on the path. never grant iam:DeleteUserPermissionsBoundary. nothing calls it.
3. roll the console. from now on every new sending user gets the boundary. an existing user gets it the next time the console touches it: enable, a standalone credential, a domain change, and key rotation too, since no key is minted for a user without the boundary.
4. run the sweep, which also reaches users nothing will touch again. set CARLOS_MAIL_BOUNDARY_SWEEP=1 on one console and restart it. it logs set the boundary on <user> for each user it fixes. its last line is either all N mail users bounded or the users that are not, with why. fix any failure and run it again. then remove the variable.
5. check it yourself. for every user in aws iam list-users --path-prefix /carlos-mail/, aws iam get-user --user-name <user> must show PermissionsBoundary with the boundary ARN.
6. apply IAM step 2. condition the role's iam:CreateUser and iam:PutUserPolicy on iam:PermissionsBoundary equal to the boundary ARN. do this only after every console runs this release, because an older console creates users without the boundary and step 2 refuses them.
7. wait a few minutes for IAM to propagate, then run the check in step 5 again with your own operator credentials, not the console role. iam:CreateAccessKey cannot be conditioned on the boundary, so anyone holding the console role could have made an unbounded user between step 5 and step 2. delete any user without the boundary, or set the boundary on it. the gap is closed only when this second check is clean.

App config & secrets

Per-app config lives in the deployment bucket (apps/<acct>/<app>/config.json): plain vars via carlos env, secrets sealed client-side via carlos secrets — the bucket never sees plaintext.

carlos secrets genkey # mint CARLOSSECRETSKEY / CARLOSSECRETSPUBKEY carlos env set --app keymail LOGLEVEL=debug carlos secrets set --app keymail SESSECRET=... # sealed with CARLOSSECRETSPUBKEY before upload carlos secrets list --app keymail # key names only — a value is never readable again

Like ship/promote (see "From the CLI" above), env and secrets work through a logged-in console with no CARLOS_DEPLOYMENT_BUCKET/_DIR on your machine at all (issue #43) — carlos auth login alone is enough, including for secrets set: with no CARLOS_SECRETS_PUBKEY in your own shell, it seals against the console's own served public key instead (set via CARLOS_SECRETS_PUBKEY in the console's environment; the private half never leaves the hosts that hold it either way). Direct-bucket mode (CARLOS_DEPLOYMENT_BUCKET/_DIR set) is unchanged.

The keypair is the mirror image of the release signing pair: CARLOS_SECRETS_PUBKEY only encrypts and can be held widely (CI, laptops — write-only credentials); CARLOS_SECRETS_KEY decrypts and belongs in exactly two places, your secret manager and every host's /etc/carlos/host.env.

Hosts converge during the hourly root carlos adopt pass: with CARLOS_SECRETS_KEY set, each pass materializes the manifest into /etc/carlos/apps/<route.Host>.env (the file carlos-app@.service already reads) and restarts routes whose file changed. With it unset, config sync is inert — the same convention as every other optional loop. A write also reaches exec-backed instances in seconds rather than waiting for that hourly pass (issue #44) via the agent's own fast poll loop; run carlos env sync --app <a> to request convergence on demand.

Operational notes:

• Delivery is split by backing: unit-backed routes (Backing: "unit") materialize to /etc/carlos/apps/<host>.env via the hourly root carlos adopt; exec-backed instance routes (socket-upstream) materialize to /data/<host>.config.env via the agent's in-process hourly loop, which also bounces a running instance when its config changes. At spawn the instance's own /data/<host>.env is layered on top — it wins every conflict, so CARLOS_ADMIN_TOKEN and box-local overrides are never clobbered from the bucket. Addr-upstream (proxy-only) routes get no delivery — the process behind them is externally managed.
• Migrate an app's config into the bucket before enabling CARLOS_SECRETS_KEY on its hosts: once a manifest object exists for an app, the bucket is authoritative for its env files (fully unset ⇒ converges to empty). An app with no manifest at all is left alone.
• Values are write-only. Rotating the keypair means re-entering every secret of every app from its source of record — there is no rewrap yet.

Domains

Every app has domains as a first-class concept: a platform domain it is born with, and any number of custom domains layered on top — each with DNS and TLS health visible in the console, and point-your-DNS-here guidance when a custom domain isn't connected yet. The scheme:

<app>.<account-sqid>.<apps-domain> kass.bab.oncarlos.com <canary>.<app>.<account-sqid>.<apps-domain> pricing.kass.bab.oncarlos.com

The account carries the sqid (base-23 over a frozen no-lookalike alphabet, encoding the account's minted id — deliberately not configurable, so nobody can squat a vanity label; custom domains are the vanity mechanism), the app carries its name — app names are unique per account, not globally. Account #1 is ops, the deployment operator's built-in account, created by migration; CLI-added apps live there, and its sqid (bab) is identical on every deployment. The console writes apps/<app>/account.json at claim time; the agent's reconcile pass (riding the 5-minute tick) reads those manifests and mints platform routes, and removes only routes it minted itself — custom domains are never touched by reconciliation. DNS needs one wildcard A record (*.<apps-domain> → edge; RFC 4592 wildcards match multiple labels); certs come per-host from the edge's existing ACME machinery.

A platform domain is an alias: a second name for the app's canonical route, pointing at the same backend. It is not a lifecycle host — release adoption and the fleet status counts both skip minted routes, so a deployment that turns this feature on does not double-stage artifacts or publish an inflated instance count.

Three honest limits, none of them papered over:

• Minting is additive — an app is never re-homed. Once an app has a platform apex, changing its account does not mint a new one or remove the old: one app, one apex, and moving an app between accounts is a deliberate operator migration (delete the old apex route by hand, let the next pass mint the new one), never a silent side effect of editing a manifest.
• A hibernating app's platform domain 502s while the app sleeps. The activator arms on the canonical route only; minted routes deliberately do not copy the hibernate flag, because a minted route carrying it would make the activator spawn a second process for the same app against an empty DBPath — meaningfully worse than a 502. Wake the app through its canonical host; the platform domain serves normally once it's up.
• Canary hosts are minted for static apps only. For a static app the channel genuinely is the backend (the edge resolves the channel pointer to find that release's tar), so a canary host serves exactly the canary. An instance app's minted canary would only be able to alias the app's stable process — a URL claiming to serve the canary while proxying stable — so none is minted; spawning a real per-host canary instance is future work (carlos canary), and until it exists the pass logs the gap rather than faking it.

Config surface:

• PLATFORM_APPS_DOMAIN — the deployment's apps domain (flagship: oncarlos.com), read by carlos agent, the edge, and the console (the member-facing carlos instances enable pins manifests to <sqid>.<this domain>; the console refuses a malformed value at boot). Unset = the whole feature is inert: no minting, no manifests read, the enable door answers 503 — the same litestream convention as everything else here.
• CONSOLE_RESERVE_PLATFORM_DOMAINS — set to true on a deployment that hosts apps for other people (the flagship, carloku). carlos domains attach then refuses every name under PLATFORM_APPS_DOMAIN and under the console's own hostname. Unset = false: a self-hosted deployment can attach any name except the console's own hostname and CONSOLE_DENY_DOMAINS. A value that is not a boolean stops the console at boot.
• CONSOLE_DENY_DOMAINS: comma-separated hostnames in the deployment's own domains (flagship: carloku.com). carlos domains attach refuses each one, and every name under it, to members. An owner of the ops account, or an address on CARLOS_SYSTEM_OPERATORS, can still attach one to an app with carlos domains attach --platform, and after that only they can change or detach it. The console's own hostnames, the steering host, the tunnel host and, with CONSOLE_RESERVE_PLATFORM_DOMAINS, everything under the apps domain stay refused to everyone.
• CONSOLE_DOMAIN_PROOF — set to true to require proof of control before an attached hostname is served: a TXT record at _carlos-challenge.<host> (carlos domains verify). A deployment that hosts apps for other people must set it, after the fleet runs a build that knows pending claims. Unset = false: attach is first-writer-wins, as before. A value that is not a boolean stops the console at boot.
• PLATFORM_EDGE_IPS — comma-separated edge IPs the DNS health checks compare against, read by both the agent (checks) and the console (guidance text), so the two can never name different addresses. Default: the box's public IP via IMDS, best-effort; if neither is knowable the checks stand down and health reads unknown, never a guess.
• CONSOLE_APP_REGIONS: on lets new apps be given a region other than the home. It needs CONSOLE_HOME_REGION set too. With the home unnamed, the console logs once and leaves app regions off. Default off. Automatic placement outside the home also needs CARLOS_SECRETS_PUBKEY, so the console can check that a box holds the deployment's secrets key.
• CONSOLE_GEOIP_DB: the path to an operator-supplied MaxMind-format .mmdb. DB-IP Lite is recommended, and it requires the attribution the new-app page shows. Unset or unusable turns the IP step of the region guess off, and the console starts either way. Replace the file by atomic rename, then restart. For the platform-wide GeoIP that every edge serves to apps, use carlos system geoip publish instead (docs/site/cli.md, "Platform GeoIP"). Uploads are staged in geoip-staging/ next to CONSOLE_DB, which the console clears at startup.
• CARLOS_SIDECAR_ACCOUNTS: comma-separated account sqids (not names) that may still enable and create native sidecars. Set, every other account gets 403 sidecars-paused from carlos sidecars enable and carlos sidecars create. List, get, wake and delete still work, and existing sidecars keep running. Native sidecars run on the flat tier, outside the nested tenants' isolation, until nested sidecars land (discussion #370). Unset or empty = no gate. An entry that is not a sqid stops the console at boot.

Canonical-host caveat, stated honestly: an app that pins its own host (e.g. keymail's --host keymail.dev) will redirect or 404 on its platform domain — correct behavior. Rastrillo apps accept any granted host from day one; for legacy apps the platform domain is the pre-custom-domain bootstrap URL.

Fleet status

Every deployment computes a fleet status view: how many instances are running, how busy the boxes are, whether replication is healthy — the CARLOS analogue of Eleven's own status pages. Each box's agent (carlos agent) writes its own snapshot to the deployment bucket every 5 minutes; whichever box holds the status lease that tick merges every snapshot into status/fleet.json plus the embedded viewer (status/index.html, status/status.css) and republishes all three. The edge serves them back out through a kind=status route, bucket-backed with a short cache — no release machinery involved, no CAS churn, just "read the key, cache it briefly, serve it."

Three commitments hold end to end, enforced in the writer rather than the page: the page publishes only fleet-wide aggregates and band words (comfortable/busy/critical, plenty/getting-full/at-capacity) — never raw ratios, hostnames, or machine ids. An unknown reading publishes loudly as null, and a box that has stopped reporting renders stale, never silently dropped. App names are opt-in only, via carlos add --status-named, and everything else folds into one honest apps_unlisted count, so the aggregate never lies by omission.

A route added this way is described by no bucket record, so nothing converges it: its channel and environment are box-local until an instance record adopts it. A provisioned route's channel is repointed through the record, not here.

Config surface — every env var this feature reads, gathered here since no single reader owns the whole contract:

• CARLOS_STATUS_BUCKET / CARLOS_STATUS_PREFIX / CARLOS_STATUS_REGION — read by carlos agent's status loop: the bucket each box writes its snapshot to and, when it wins the lease, publishes the merged feed to; region also feeds the cross-box status-lease election. The serving side (the edge's kind=status route and the console's operator view) must be pointed at this same bucket — nothing enforces that agreement automatically, publishing and serving are separately configured. To make the common case foolproof, an unset CARLOS_STATUS_BUCKET defaults to CARLOS_DEPLOYMENT_BUCKET (an explicit CARLOS_STATUS_BUCKET still always wins), so a single-bucket deployment needs only one env var for both release adoption and status. Both unset disables the whole loop — inert until configured, the same convention litestream replication uses. Honest limitation: that default only ever resolves to an S3 bucket — a deployment adopting from CARLOS_DEPLOYMENT_DIR (no bucket at all) does not publish fleet status yet; give it its own CARLOS_STATUS_BUCKET if it wants the page. CARLOS_STATUS_PREFIX does double duty: it namespaces the status-lease key, AND (customer-fleets plan, Task 5) rebases every key the status loop's AND the metrics loop's store touches — a fleet box set to CARLOS_STATUS_PREFIX=fleets/<name>/ writes its own status snapshot at fleets/<name>/status/boxes/<label>.json and its raw/rollup metrics and alert state under fleets/<name>/metrics/boxes/… and fleets/<name>/alerts/boxes/…, instead of the shared paths a platform box uses — so neither leaks into the platform's own fleet.json, monitoring page, or alert webhook. Empty (every platform box today) is a pure passthrough. A publishing box with a prefix set would relocate status/fleet.json itself under that prefix too — moot today, since nothing sets a prefix on a publishing (non-fleet) box, and a fleet box never publishes at all (see CARLOS_FLEET, below).
• A box with CARLOS_HIBERNATE_BUCKET set but no deployment store (CARLOS_DEPLOYMENT_BUCKET or CARLOS_DEPLOYMENT_DIR) refuses placed exec instances, because it cannot tell whether their config is sealed. Set CARLOS_NO_SEALED_CONFIG=1 to say this deployment has none.
• CARLOS_FLEET — read by carlos agent's status loop (customer-fleets plan, Task 5). Set (to the fleet's name) marks this agent as a fleet box: it still self-reports its own snapshot every tick, but never contends for the cross-box publish lease and never merges/publishes status/fleet.json — that feed belongs to the platform's own boxes only. Unset (every platform box today) is unaffected.
• CARLOS_FLEET_BOX — this box's operator-chosen fleet_boxes.label (customer-fleets plan, Task 6 fix round). This — NOT the public status page's hashed machine-id pseudonym — is the box's real identity: the channel name the edge dials is <fleet>/<label>, so both the fleet binding a box publishes on wake and its own heartbeat filename (fleets/<fleet>/status/boxes/<label>.json) use it. Optional: defaults to CARLOS_TUNNEL_NAME with the <fleet>/ prefix stripped, since a fleet box's tunnel name already has this shape. Both the fleet name and the resolved label are validated with registry.ValidTunnelName; a fleet box whose identity fails to resolve refuses (loudly) to arm binding-publish at all, and its heartbeat falls back to the hashed pseudonym instead of going dark.
• STATUS_BOX_REGION — also read by carlos agent's status loop: an operator-set, coarse region word (e.g. "EU · Ireland") carried onto this box's snapshot verbatim; unset omits the field. Free text the operator chooses, never derived from cloud metadata. Keep it geographically consistent with the region's place name in the console's region picker (for example "EU · Ireland" beside "Ireland · Dublin"); it may stay coarser.
• STATUS_HOST / STATUS_PUBLIC — not read by the agent process itself; consumed by ops/tofu to decide whether the edge fronts the bucket's status/ prefix with a public hostname. STATUS_PUBLIC=1 publishes the page at STATUS_HOST; leaving it unset keeps the feed private, visible only to console operators (console grant --operator <address>).

Dry run. Before pointing a binary at a real bucket, carlos status --dry-run <dir> runs the whole publish pipeline — CollectBox → WriteBox → BuildFleet → PublishFleet — against a scratch directory stuffed with fixture secrets (hostnames, a machine id, mid-band fractions), then reads every object it just published back out and fails loudly if any of them leaked. There is no live-publish CLI verb; the 5-minute loop inside carlos agent is the only publisher.

Incident fallback, and its actual limits. The page footer links "if this page is unreachable" straight at the bucket's own copy of fleet.json, but that link is built client-side from location.origin (see internal/status/assets/index.html) — it is only meaningful once you are already viewing the page served from the bucket's own origin (the public STATUS_HOST, or the raw bucket URL). It is not a link that rescues you from a down deployment; if the edge/console are unreachable you won't be looking at this page's footer in the first place. The real operational fallback for a down deployment is fetching the objects directly, either the raw bucket URL in a browser or:

aws s3 cp s3://<bucket>/status/fleet.json - aws s3 cp s3://<bucket>/status/index.html -

One current wrinkle: index.html fetched straight from S3 currently downloads instead of rendering in a browser, because the store's Put sets no Content-Type and S3 falls back to application/octet-stream. It still opens fine as plain text in an editor, and aws s3 cp ... - | less works, but "open the bucket URL in a browser and read it" doesn't work yet for the HTML — see the design spec's rollout list (§5) for the queued fix (setting content-type on publish). fleet.json is unaffected (any client reading raw JSON already expects to view its own headers).

A fully-private deployment (STATUS_PUBLIC unset) whose console is down has no page at all — the aws s3 cp fallback above is the entire story there; nothing papers over it.

Host & process monitoring

Operator-grade monitoring layered on top of fleet status: real CPU, memory, and disk numbers per host and per process, with history, alerting, and a console dashboard — needles where the public status page shows only band words. See the 2026-08-02-host-process-monitoring-design.md design spec (private platform-infrastructure repo) for the full design.

Bucket layout:

• metrics/boxes/<box>/raw/<YYYY-MM-DD-HH>.json — 60-second host + per-unit samples, one object per UTC hour, retained 48 hours.
• metrics/boxes/<box>/days/<YYYY-MM-DD>.json — hourly min/avg/max rollups, kept forever (the capacity/trend history).
• alerts/boxes/<box>.json — on-box alert state (fire/clear, hysteresis).

Config surface — read by the box agent from /etc/carlos/host.env:

• CARLOS_ALERT_WEBHOOK — optional; the agent POSTs the alert JSON here on fire and on clear. Unset simply skips the webhook channel (keymail delivery and the console UI channel are unaffected).
• CARLOS_ALERT_SNS_TOPIC_ARN — optional; the agent publishes each fire and clear to this SNS topic as a plain-text email-sized message, signed with the box's instance role (which needs sns:Publish on the topic). The region is read from the ARN, not from AWS_REGION. Set it to the box's own region's alerts topic and anyone subscribed to that topic gets the email. Either sink works alone and both may be set. A failed send is logged and counted, never fatal. The SDK tries a retryable failure up to three times within ten seconds. After that, a later tick does not send that alert again, unless the same tick's bucket write also failed, which repeats the whole transition on the next tick. A value that is not an SNS topic ARN (a subscription ARN, for example) is logged and the sink stays off. So does a client that cannot be built, which carlos status --live shows as alerts=sns-unavailable.
• With neither sink set, the box says so: the agent logs NO SINK CONFIGURED at start, and the box's private status snapshot carries metrics.alerts with both sinks false, which carlos status --live prints as alerts=not-delivered.
• CARLOS_ALERT_DISK_PCT / CARLOS_ALERT_MEM_PCT / CARLOS_ALERT_CPU_PCT — optional threshold overrides for the built-in disk / memory / host-CPU rules, as percent integers 0-100 (e.g. 95). Anything else is ignored with a log line, and that rule keeps its built-in default.

Console surfaces:

• GET /fleet/metrics — operator-only dashboard: per-box needles and 48-hour sparklines, a per-box per-unit table (ordered by CPU, busiest first), and active alerts at the top.
• A "Resources" section on each app's own page — owner-facing, scoped to that app's own instances only (current CPU and memory plus 48-hour sparklines); never host-level totals or other tenants' units.

The days/ rollups are written from the first flush onward but nothing renders them yet — they are the history a later trend view will read.

App capabilities

A handful of apps need more from the platform than the carlos-app@<host> template gives every instance for free — a subcommand other than serve, a privileged port, the box's cloud instance role, or the resource headroom a document-conversion engine needs. Rather than hand-provision a bespoke unit per app, a release declares what it needs and the platform decides how — and whether — to provide it, from a closed vocabulary of eight names. See the 2026-08-03-app-capabilities-design.md design spec (private platform-infrastructure repo) for the full design; this section is the operator-facing summary.

The manifest shape — an optional capabilities object on the release manifest, additive to schema 1 (a manifest with no capabilities key parses as "none", today's behaviour byte for byte):

"capabilities": { "v": 1, "command": "relay", "daemon": true, "children": true, "runtimedir": "keymail", "ports": [ { "port": 25, "name": "mx" } ], "data": { "replicate": "app" }, "hostrole": true }

A document-conversion engine (Gotenberg, an OCR service) declares a different shape — host_role and doc_engine are mutually exclusive and a manifest carrying both is refused at ship, because host_role re-opens the cloud metadata address and doc_engine exists to fence the engine's egress away from private networks:

"capabilities": { "v": 1, "daemon": true, "children": true, "doc_engine": true }

command, daemon, children, runtime_dir and data are self-service — they shape only the app's own sandbox and materialize from the manifest alone. ports, host_role and doc_engine are operator-consented: declaring one asks for it, but nothing materializes until an operator has granted it on that specific box, because the resource (a privileged port, the box's IAM role, the private-network egress fence that pays for a conversion engine's JIT and memory headroom) is box-local, not fleet-wide.

Declaring a capability rides carlos ship --capabilities <file.json>, where the file holds exactly the capabilities object above (not a whole manifest). The same strict decoder and validation gate ship and adopt both run: an unknown capability name or field, a bad charset, a reserved runtime_dir, or an out-of-range/reserved port is refused at the ship terminal, where a human is standing to read it — no per-capability flags, because a flag vocabulary would drift from internal/caps's own.

Granting a capability is carlos route, per route, per box:

carlos route --host keymail.dev --backing unit # first: grants need it carlos route --host keymail.dev --grant port:25 carlos route --host keymail.dev --grant host-role carlos route --host gotenberg.docs.example.com --grant doc-engine carlos route --host keymail.dev --revoke port:25 carlos route --host keymail.dev --hibernate=false

Each of --backing, --grant/--revoke and --hibernate is its own operation: they require --host (never --app — a bulk grant is precisely the operation nobody should be able to type by accident), and none of them combines with another, or with --channel/--environment, in one invocation.

The order above is not cosmetic. A grant requires unit backing: a route that is not a kind=instance, backing=unit row has nothing for a grant to materialize into, so --grant/--revoke are refused there, and grants cannot be staged ahead of a migration. --backing unit is the switch that moves a route from an agent-run exec child onto a carlos-app@<host> systemd instance; an exec-backed route that declares capabilities materializes nothing and is advised once per pass until it is switched over (see the design's §5 migration steps). Stamping it arms the hourly carlos-adopt.timer, and carlos adopt for a daemon route enables and STARTS the service — so the data copy comes before the stamp, not after it.

A declared-but-ungranted capability does not fail the app or fail adoption: the drop-in simply is not written, and the hourly adopt pass logs one line naming the route, the capability, and the exact grant command. The mirror line is there too — a token this box has granted that the live release no longer declares materializes nothing, and is named with the revoke command rather than left sitting in the registry looking like live authority.

doc-engine carries two box preconditions the platform cannot check for you, which is what makes granting it an operator decision rather than a checkbox. The grant buys the app JIT memory and a raised resource tier, and in the same drop-in fences its egress away from every private network range — RFC1918, link-local, CGNAT, IPv6 ULA and the NAT64 prefix — because a conversion engine fetches whatever URLs the documents handed to it name, and an unfenced one is an SSRF proxy into the VPC. Two consequences:

• The box must hand the tenant a loopback DNS resolver. The fence deliberately leaves 127.0.0.0/8 open — the engine drives its own Chromium and LibreOffice over loopback — but it closes 10.0.0.0/8, and on AWS the VPC resolver lives inside that range (VPC base + 2). A box running systemd-resolved on 127.0.0.53 resolves through loopback and the fence is transparent; a box pointing tenants straight at the VPC resolver gives them an engine that cannot resolve a hostname.
• The box should be dedicated. Loopback staying open means an SSRF'd engine can still reach anything listening on 127.0.0.1 on its own box — a co-tenant console, a co-tenant's loopback-bound upstream. The fence closes the network half of that risk; not co-locating tenants closes the rest. Run doc-engine routes on a box of their own.

Deploy order is pinned, same rule and reason as scoped signing keys: the platform binaries that understand a capability name have to be on the box before a release declaring it, never after. An older binary meeting a manifest it does not understand refuses the version on that box — not staged, not pointed at — rather than materialize a unit missing something the app says it needs.

"Platform binaries" includes the console binary, and on the flagship that is the one that matters: carlos ship in API mode posts through the console, so the console is what either understands the capabilities key or silently drops it as an unknown JSON field — storing a manifest that declares nothing, which on a box is the instruction to REMOVE the drop-ins. The ship response therefore echoes the capability block it stored, and carlos ship fails hard when it sent a block and gets no echo back ("this console does not understand capability declarations — update the console binary first, or ship without --capabilities"). Update the console before shipping a capability-bearing release.

What refuses vs what leaves alone — the distinction that matters most:

• An unreadable or unresolved staged manifest leaves every drop-in exactly as it is and logs once per pass. A bucket hiccup must never strip :25 off a running mail server.
• A manifest that actually resolves and no longer declares a capability (or a grant that gets revoked) removes the generated drop-in.
• A hand-edited drop-in — anything at the generated path without the # GENERATED by carlos adopt — do not hand-edit marker on its first line — is never overwritten and never deleted; it is reported and that unit's half is skipped while the other half still converges.
• An unknown capability name or an unsupported "v" refuses the version on that box, not the whole fleet — the box keeps serving what it already runs.

Console gets, at most, a read-only line beside each route's other facts — "granted: host-role, port:25" for a route with grants, nothing at all otherwise, and only on routes belonging to the account whose page it is. There is no console UI for granting or revoking; grants stay a carlos route operation on the box, and nothing about the console's sign-in/session/CSRF/owner gates changes.