| 1 | # 🤖 CARLOS platform |
| 2 | |
| 3 | > 🤖 This document was written by an LLM (Claude), on the ideas, |
| 4 | > instruction, and editing of humans, per |
| 5 | > [factor X](https://11factor.org/#x). |
| 6 | |
| 7 | A black-box deployment platform for CARLOS apps: `carlos ship` publishes an |
| 8 | immutable, content-addressed release; the platform places it on instances, |
| 9 | routes to it through a shared edge, and surrounds it with platform-level |
| 10 | glue. One deployment is **one object-store bucket + N hosts running the |
| 11 | `carlos` binary + DNS** — the whole self-hosting story in one sentence. See |
| 12 | the `2026-08-01-carlos-platform-design.md` design spec (private `platform-infrastructure` repo) |
| 13 | for the full design. |
| 14 | |
| 15 | ## Build & test |
| 16 | |
| 17 | make build # CGO_ENABLED=0 build of the carlos binary |
| 18 | make test # go vet, gofmt -l, go test ./... |
| 19 | |
| 20 | One static binary plays every role, selected by subcommand: `carlos edge`, |
| 21 | `carlos agent` (edge + activator + release adoption + fleet status, one |
| 22 | process per host), `carlos ship`/`promote`/`rollback`, `carlos vet`. Run |
| 23 | `carlos` with no arguments for the full command list. |
| 24 | |
| 25 | ### Platforms |
| 26 | |
| 27 | Linux and macOS run every role — box and workstation alike — and install |
| 28 | from apt (`sudo apt install carlos`) or Homebrew |
| 29 | (`brew install carlosframework/tap/carlos`). |
| 30 | |
| 31 | **Windows is a client-only target.** A Windows workstation can `carlos |
| 32 | login`, `carlos ship`, `carlos deploy`, `carlos logs` — everything that |
| 33 | talks to a console or a bucket over the network — against a Linux fleet. |
| 34 | It cannot BE a box: `carlos agent` and the root-side passes need unix |
| 35 | sockets, `flock` and systemd, and the packages that provide them compile |
| 36 | on Windows (the CLI is the same binary) but refuse at their per-OS |
| 37 | boundary rather than pretending. See `internal/edge/purge_listen_windows.go` |
| 38 | and `internal/host/{statfs,flock}_windows.go` for what that refusal means |
| 39 | in each case. |
| 40 | |
| 41 | Two consequences worth knowing before you install it: |
| 42 | |
| 43 | - **`carlos update` does not self-replace on Windows.** It still checks the |
| 44 | release feed and tells you when you are behind, then hands you the URL of |
| 45 | the zip to unpack over your existing binary. Windows cannot rename a new |
| 46 | binary over a running `.exe`, and a self-replace nobody here can test is |
| 47 | worse than an honest manual step. |
| 48 | - **Your credentials file is not permission-protected on Windows.** On Linux |
| 49 | and macOS `~/.carlos/credentials` is written `0600`; the equivalent mode |
| 50 | on `%USERPROFILE%\.carlos\credentials` means nothing, so any other user |
| 51 | of that machine can read your session. Do not `carlos login` on a shared |
| 52 | Windows machine. |
| 53 | |
| 54 | Install by downloading `carlos-windows-amd64.zip` (or `-arm64`) from |
| 55 | [carlosframework/releases](https://github.com/carlosframework/releases/releases), |
| 56 | unpacking `carlos.exe` somewhere on your `PATH`, and verifying its checksum |
| 57 | against the release's signed `SHA256SUMS.txt`. |
| 58 | |
| 59 | ## Accounts |
| 60 | |
| 61 | An **account** is the tenancy primitive: a project or team that owns apps and |
| 62 | has members. Kass, Keymail and Woodstar each get their own rather than sharing |
| 63 | one flat list, and a self-hosted single-app deployment can give team A and |
| 64 | team B separate accounts that deploy independently. |
| 65 | |
| 66 | An account is `{id, sqid, name}`. The **id** is the next free integer, |
| 67 | allocated at creation by a create-only write of `accounts/<sqid>/account.json` |
| 68 | in the deployment bucket; the registry only ever *materializes* what the |
| 69 | bucket says, never assigns. That is what makes account identity survive a |
| 70 | registry rebuild and be identical on every box: minted once, recorded in the |
| 71 | bucket, never reassigned. Sequential ids keep sqids short enough to live in a |
| 72 | hostname you can read out (`kass.bac.oncarlos.com`); they are not secret, and |
| 73 | never were — every platform hostname's certificate is published in |
| 74 | Certificate Transparency. The **sqid** is the frozen encoder over that id and |
| 75 | is the account's public identity, in every platform domain and every bucket |
| 76 | key. The **name** is free text and deliberately not unique: two accounts may |
| 77 | both be called "Blog", because identity is the sqid and there is nothing to |
| 78 | squat. |
| 79 | |
| 80 | `ops` is the built-in operator account — id 1, sqid `bab` — created by |
| 81 | migration on every deployment. CLI-added apps live there by default, so a |
| 82 | console-less self-hoster is unaffected by any of this. |
| 83 | |
| 84 | **Roles.** Exactly two: |
| 85 | |
| 86 | | Role | Can | |
| 87 | |---|---| |
| 88 | | **owner** | everything a member can, plus invite, remove members, rename | |
| 89 | | **member** | deploy, promote, rollback, create/retire canaries, view everything | |
| 90 | |
| 91 | Every account has at least one owner; the last one can be neither removed nor |
| 92 | demoted (transfer first). |
| 93 | |
| 94 | **In the console.** The account switcher sits top left on every account page, |
| 95 | where the wordmark used to be: a dropdown whose trigger names the account you |
| 96 | are in, listing the accounts you are a member of and linking to the |
| 97 | **New account** screen at `/accounts/new`. It is a `<details>` disclosure, so |
| 98 | it is keyboard-operable and works with scripting off. |
| 99 | URLs are explicit and shareable — `/a/<sqid>/apps/<app>` — so a link a |
| 100 | teammate opens lands where you meant; `/` redirects to the account you were |
| 101 | last in (or your first membership, or one minted on the spot). A signed-in |
| 102 | non-member asking for an account URL gets the **same 404 a nonexistent page |
| 103 | gets**: account existence is not probeable. |
| 104 | |
| 105 | **Invites** go into the addressee's Keymail Apps inbox — the platform never |
| 106 | emails anyone — as a single-use link that expires in 7 days and is revocable. |
| 107 | Clicking it while signed in as that address creates the membership. |
| 108 | |
| 109 | **From the CLI.** Every app verb takes `--account`, a sqid or a name, |
| 110 | defaulting to `ops`: |
| 111 | |
| 112 | carlos auth login --console https://platform.carlosframework.com |
| 113 | carlos accounts create --name kass # you own it |
| 114 | carlos accounts create --name kass --owner paul@keymail.dev # headless |
| 115 | carlos accounts list # sqid, name, app count |
| 116 | carlos apps create --account kass --app blog # claim the name (rate limited; |
| 117 | # the console's New app form is not) |
| 118 | carlos ship --account kass --app blog ... |
| 119 | carlos add --account kass --app blog blog.example.com |
| 120 | |
| 121 | **Colour.** The CLI colours its output when — and only when — it is writing |
| 122 | to a terminal: a write that happened reads green, a dry run or a bypassed |
| 123 | rail yellow, a refusal red, and context you did not ask for (the `using |
| 124 | <console> as <you>` banner, "nothing to do" verdicts, follow-up hints) is |
| 125 | dimmed. Redirect it, pipe it, or run it in CI and every byte is what it |
| 126 | always was, so parsing scripts and test assertions are unaffected. Override |
| 127 | with `CARLOS_COLOR=always|never`; `NO_COLOR=1` and `TERM=dumb` are honoured. |
| 128 | |
| 129 | **Who the CLI is.** `carlos auth login` gives the command line an identity, |
| 130 | and that identity owns the accounts it creates. It authenticates against the |
| 131 | **deployment's own console** — already a registered Keymail OAuth client and |
| 132 | already the authority on who belongs to what — rather than minting a second |
| 133 | OAuth client, so there is one consent screen, one place to revoke, and a |
| 134 | self-hoster's CLI trusts their own box. |
| 135 | |
| 136 | It prints a code rather than opening a browser or listening on a local port, |
| 137 | because the machine that needs the credential is usually an SSH session where |
| 138 | neither works. You open `/auth/cli` in whatever browser you are already signed |
| 139 | in on and **type the code** the terminal printed. |
| 140 | |
| 141 | Typing it is the point, not a nicety: a link that arrives one click from an |
| 142 | Approve button is the device-code phishing attack (the one run at scale against |
| 143 | Microsoft tenants) — an attacker starts the flow, sends you their link, and one |
| 144 | click hands them a credential that acts as you. So the console never approves |
| 145 | from a prefilled code, and the CLI never prints the ready-made link. When a |
| 146 | token is issued, a note goes to your Keymail inbox naming the terminal, where |
| 147 | it connected from, and how to revoke it. |
| 148 | |
| 149 | The token lands in `~/.carlos/credentials` (mode 0600, one entry per console |
| 150 | origin), expires in 90 days, and is listed with a revoke button at |
| 151 | `/auth/cli/tokens`. |
| 152 | |
| 153 | carlos auth whoami # who this terminal is, per the console |
| 154 | carlos auth logout # revokes the token, then forgets it |
| 155 | |
| 156 | **Shipping from a logged-in laptop.** Once you're logged in, `ship`, |
| 157 | `promote`, `rollback`, `channels`, and `env`/`secrets` (issue #43) speak |
| 158 | through the console instead of the bucket directly — no |
| 159 | `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` on a developer's machine at all: |
| 160 | |
| 161 | carlos ship --app blog --version $(git rev-parse --short HEAD) --target linux-arm64 blog-linux-arm64 |
| 162 | carlos promote --app blog abc1234 canary/my-branch |
| 163 | carlos channels --app blog |
| 164 | |
| 165 | Canary and edge promote exactly as before; **promoting onto `stable` is |
| 166 | refused** — the console names itself in the refusal, because moving `stable` |
| 167 | needs a passkey step-up only a browser can do. Finish it at the URL the |
| 168 | refusal prints. |
| 169 | |
| 170 | **CI uses the console too.** Mint a deploy credential for the app, limited |
| 171 | to the rungs CI may move, and give CI the secret and the console's address: |
| 172 | |
| 173 | carlos services create --role deploy --app blog --rungs canary,edge ci |
| 174 | # in CI: |
| 175 | export CARLOS_SERVICE_TOKEN=... CARLOS_CONSOLE=https://console.example.com |
| 176 | carlos deploy --app blog --version "$GITHUB_SHA" --channel edge blog-linux-arm64 |
| 177 | |
| 178 | `carlos deploy` ships, promotes and watches the app's hosts, the same as |
| 179 | from a laptop. The first deploy of a new app still needs a person: a deploy |
| 180 | credential cannot create the app or set up its first host. Every promote is |
| 181 | audited in the console and reaches the boxes within seconds. A deploy |
| 182 | credential can ship, and promote or roll back only its rungs. It cannot |
| 183 | reach stable, hotfix, delete the app, or change retention or domains. |
| 184 | |
| 185 | **Direct bucket mode is for operators.** With `CARLOS_DEPLOYMENT_BUCKET` |
| 186 | (or `_DIR`) set, every verb goes straight to the bucket and the console |
| 187 | never sees it. Use it to set up a deployment before its console exists, to |
| 188 | run one that has no console, to recover when the console is down, and for |
| 189 | the platform's own releases. A scoped `CARLOS_RELEASE_KEY` limits that path |
| 190 | only. The CLI refuses to run with both `CARLOS_SERVICE_TOKEN` and a bucket |
| 191 | variable set. |
| 192 | |
| 193 | **On a box, or in CI, don't log in.** There is no browser and no human, and a |
| 194 | long-lived token sitting on a box is a standing credential nobody is watching. |
| 195 | Name the owner outright instead — it needs no identity of its own and takes |
| 196 | precedence over any logged-in one: |
| 197 | |
| 198 | carlos accounts create --name kass --owner paul@keymail.dev |
| 199 | |
| 200 | Either way an owner is named. `carlos accounts create` **refuses** with no |
| 201 | login and no `--owner`: an account with no owner materializes into the console |
| 202 | with no members, which means nobody can open it, invite to it, or see the apps |
| 203 | migrated into it. |
| 204 | |
| 205 | An exact sqid wins; otherwise a unique name match; an ambiguous name lists the |
| 206 | candidates and refuses rather than guessing which tenant you meant. |
| 207 | |
| 208 | `--owner` is required and is recorded in the account manifest, which the console |
| 209 | materializes into the account's first owner membership. Membership *is* |
| 210 | authorization here, so an account created without one exists, holds its apps, |
| 211 | and cannot be opened by anybody — a state the flagship reached for real during |
| 212 | the multi-account migration. |
| 213 | |
| 214 | **Bucket layout, and the migration.** App objects live under |
| 215 | `apps/<account-sqid>/<app>/…`, and that key path *is* the app→account linkage |
| 216 | — there is no separate manifest. **Every read path resolves those qualified |
| 217 | keys and nothing else** (the legacy read-fallback was retired on 2026-08-03, |
| 218 | once both deployments were migrated). Moving an app is a verb: |
| 219 | |
| 220 | carlos accounts migrate --app kass --to <sqid> --dry-run # prints the plan |
| 221 | carlos accounts migrate --app kass --to <sqid> |
| 222 | console migrate-app --app kass --to <sqid> # on the console's box |
| 223 | |
| 224 | It copies the objects and re-stamps the app's routes. The console keeps its |
| 225 | own app→account row — console state is the console's authority, and the |
| 226 | carlos binary deliberately never opens its database — so `console |
| 227 | migrate-app` completes the move; `carlos accounts migrate` prints that exact |
| 228 | command when it finishes. Both halves are idempotent. |
| 229 | |
| 230 | **Upgrading from a pre-accounts binary?** Your objects are all at the old |
| 231 | account-less `apps/<app>/…` keys, which nothing reads any more — so the apps |
| 232 | will go quiet rather than serve something stale, and every miss tells you so |
| 233 | by name: |
| 234 | |
| 235 | carlos accounts migrate --app <app> --from legacy --to bab # --to <sqid> for a tenant |
| 236 | |
| 237 | That `--from legacy` source is permanent. However long a deployment sits on an |
| 238 | old build, this is the way off those keys. |
| 239 | |
| 240 | **It never deletes a legacy object.** Deleting them is a separate operator |
| 241 | step, and an unhurried one now that nothing resolves through them. |
| 242 | |
| 243 | ## Releasing the platform |
| 244 | |
| 245 | A release is a `vX.Y.Z` tag on this repo plus signed binaries on [carlosframework/releases](https://github.com/carlosframework/releases). Everything downstream reads that one set of files: `carlos update`, the Homebrew tap, apt, and every deployment's `carlos system update`. Cut it from a green `main` and verify it from the public side, never from your build directory. |
| 246 | |
| 247 | 1. **Tag.** `git tag -a vX.Y.Z <sha> -m "carlos vX.Y.Z"` on the `main` commit you are releasing, then `git push origin vX.Y.Z` to the forge. |
| 248 | 2. **Build.** Six targets, darwin, linux and windows on amd64 and arm64, each with `make build GOOS=<os> GOARCH=<arch> SHA=vX.Y.Z` from a clean checkout of the tag. Build with `TMPDIR` and `GOCACHE` on a real disk: `/tmp` is a shared tmpfs, and a full one fails the link with "no space left on device". Zip each Windows binary as a lone `carlos.exe`. |
| 249 | 3. **Debs.** One per architecture, built with `dpkg-deb --root-owner-group`, holding `usr/bin/carlos`. Copy the last release's `control` and change the version and the description's last sentence. Wrap the description without breaking on hyphens, or apt shows "step- by-step". |
| 250 | 4. **Sums and signature.** Write `SHA256SUMS.txt` over the eight files. Sign it with the ed25519 seed in SSM `/carlos/cli-update-key`, passed on stdin and never in argv, and write the signature base64-encoded to `SHA256SUMS.txt.sig`. Before signing, check that the key's public half equals `cliUpdatePubKey` in `update.go`. A mismatch publishes a release nobody can update to. |
| 251 | 5. **Release notes.** Draft them, and put them through copy review before anything is published. Lead with what changed for people using the CLI. Add a short section for operators, taken from the release notes below this recipe, about anything to check before rolling. |
| 252 | 6. **Publish.** `gh release create vX.Y.Z -R carlosframework/releases --latest --title vX.Y.Z --notes-file NOTES.md` with all ten files. Confirm it is the latest with `gh api repos/carlosframework/releases/releases/latest -q .tag_name`. |
| 253 | 7. **Homebrew.** In `carlosframework/homebrew-tap`, bump `Formula/carlos.rb`: the version, the four url and sha256 pairs from the published `SHA256SUMS.txt`, and the `carlos vX.Y.Z` test string. Open a PR, check each hash against the sums file, and squash-merge it. |
| 254 | 8. **apt.** Download the live tree, `apps/bab/apt-carlosframework/releases/<last>/site.tar.gz` from the flagship bucket, and unpack it. Add the two debs under `pool/main/c/carlos/`, and regenerate each architecture's `Packages` with `dpkg-scanpackages --multiversion`. Delete `Release`, `InRelease` and `Release.gpg`, then rebuild `Release` with `apt-ftparchive release`, passing Origin and Label `CARLOS`, Suite and Codename `stable`, the architectures and the component. Write it outside the tree first, or it lists itself. Sign `InRelease` and `Release.gpg` with the SSM key `/carlos/apt-signing-key` in a throwaway `GNUPGHOME`, set `latest.txt` to the tag, and check both signatures with `gpgv` against the tree's own keyring. |
| 255 | 9. **Ship apt.** Ship the directory and let `carlos ship` build the tarball. A hand-made tar with a leading `./` entry took apt down once. With `AWS_PROFILE=keymail` and `CARLOS_DEPLOYMENT_BUCKET` set to the flagship bucket, run `carlos ship -app apt-carlosframework -account-id bab -kind static -version vX.Y.Z-1 site/`, then `carlos promote -app apt-carlosframework -account-id bab vX.Y.Z-1 canary/apt` with `CARLOS_RELEASE_KEY` from SSM `/carlos/release-key` set for that one command. The channel is the literal `canary/apt`. |
| 256 | 10. **Changelog.** In a full checkout of `main`, with `gh` and `amadan` signed in, run `git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*'`, then `go run ./cmd/changelog`. Map each of the new release's bullets to the branches behind it in `docs/site/changelog-links.json`, keyed by the bullet's bold lead, and run it again. Land `docs/site/changelog.json` and the links file on a branch. Then, in `carlosframework/website`, run `node hack/sync-docs.mjs <checkout>` and ship the site the way its AGENTS.md describes. The page is carlosframework.com/platform/changelog. |
| 257 | |
| 258 | Then verify from the public side: |
| 259 | |
| 260 | ```sh |
| 261 | v=vX.Y.Z; rel=https://github.com/carlosframework/releases/releases/download/$v |
| 262 | curl -sSLO $rel/carlos-linux-arm64; curl -sSLO $rel/SHA256SUMS.txt; curl -sSLO $rel/SHA256SUMS.txt.sig |
| 263 | printf '\x30\x2a\x30\x05\x06\x03\x2b\x65\x70\x03\x21\x00' > pub.der |
| 264 | echo 'R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=' | base64 -d >> pub.der |
| 265 | base64 -d SHA256SUMS.txt.sig > sig.bin |
| 266 | openssl pkeyutl -verify -pubin -inkey pub.der -keyform DER -rawin -in SHA256SUMS.txt -sigfile sig.bin |
| 267 | sha256sum -c --ignore-missing SHA256SUMS.txt |
| 268 | curl -sS https://apt.carlosframework.com/latest.txt # prints $v |
| 269 | carlos system update --console <console> --version $v # then: carlos system status |
| 270 | ``` |
| 271 | |
| 272 | Last, roll it, and roll dev before flagship. The order is fixed: dev boxes, then the dev console, then check dev end to end, then flagship boxes, then the flagship console. Do not start on flagship until dev has been exercised on the new build. |
| 273 | |
| 274 | `carlos system update --console <console> --version vX.Y.Z`, then `carlos system status`. Every box should read `current` within the spread. A box that reads `rolled-back` or `refused` says why on the same line; fix that box, and do not re-cut the release. |
| 275 | |
| 276 | **Checking dev.** The check is `wakecheck`, a standing hibernating app on dev. Each request writes one SQLite row and replies with its build, the box's host and the row count (`hits=`). It has no region, so both dev boxes serve it: `carlos-dev-ie`, which is the console box, and `carlos-dev-ie-b`. Once dev is rolled: |
| 277 | |
| 278 | 1. Confirm it is asleep. `carlos logs --app wakecheck --console https://platform.dev.carlosframework.com --since 30m` should end in "instance exited". Do not use `instances list --health` for this, because it always reads Asleep (#287). Idle is 15 minutes and bots can keep it awake, so wait it out. |
| 279 | 2. Wake it on `carlos-dev-ie` through DNS: `curl -sS https://wakecheck.dev.carlosframework.com/`. A pass is a 200 whose `host=` is that box, in about 0.6 s to first byte. |
| 280 | 3. Wake it on `carlos-dev-ie-b` by pinning its address: `curl -sS --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210 https://wakecheck.dev.carlosframework.com/`. A pass is a 200 whose `host=` is that box. It takes about 1.6 s, because it restores from S3. |
| 281 | 4. `carlos status --layout` shows `wakecheck` as `nested` on both boxes. |
| 282 | |
| 283 | On dev, `hits=` going backwards from one box to the other is not a failure. With the claim off, a wake that switches box can serve a stale local DB. That is the known hazard in #286, and it stays until the claim ruling lands. Judge the check on the 200s and the `host=` values, not on the count. |
| 284 | |
| 285 | Three things catch people out on the roll: |
| 286 | |
| 287 | - **`system update` rolls a whole channel.** It rolls every box on the channel within the spread. It cannot canary one box. To put a release on one box first, roll that box by hand with `scripts/update-platform-binary.sh` in platform-infrastructure. |
| 288 | - **Old updaters need `carlos-instance-control` restarted.** A box whose updater predates v0.26.0 does not restart `carlos-instance-control` when it rolls. The moment the new edge serves, restart it by hand. Until then the new agent refuses isolated wakes as "running an older build". Use a one-off watcher to do the restart, not memory. |
| 289 | - **The dev console needs its release key first.** Before dev can be rolled with `system update`, its console must hold the release key. That is `/etc/carlos/release.env` (root:root, 0600) and a `console.service.d/release-key.conf` drop-in that loads it. |
| 290 | |
| 291 | - Refresh the embedded timezone tables when a new IANA tzdata release is out: |
| 292 | `scripts/update-tzgeo.sh <version>`, then `go test ./internal/geo`. |
| 293 | |
| 294 | A release-specific note goes here when a release changes something an |
| 295 | operator needs to check or act on before rolling it. Read the note for |
| 296 | the release you are about to roll. |
| 297 | |
| 298 | **Binary size (#275).** The linux/arm64 platform binary doubled to 77.7 MB in earlier builds because SDK clients were kept whole by the linker. This release is 45.7 MB again, so boxes download about 32 MB less. CI now fails a build over 52 MB. |
| 299 | |
| 300 | **Pointer digests (#253): writers before readers.** Once a box has verified a channel pointer that carries manifest digests, it refuses pointers for that channel without them. So digests are turned on in order. First upgrade every writer: every console, and every CLI or deploy script that promotes, rolls back or fast-tracks. Then run `carlos channels digests on`. Then watch `carlos channels digests`: a channel marked as written by an old writer means one is still in use. Rolling a writer back below this release stops it writing pointers boxes accept; `carlos channels digests off` suspends digests while it is fixed, but does not clear the boxes that have already seen them. When `carlos channels digests` lists nothing, set `CARLOS_REQUIRE_POINTER_DIGEST=on` in every box's host.env, so a new box refuses old pointers too. |
| 301 | |
| 302 | **Instance update settings: the roll order.** Roll every box to the release that applies update settings before anyone turns on manual updates, a major hold or a pin: the console refuses those until every box reports it can protect the builds they name from pruning, and says how many boxes it is waiting on. Before rolling any box back below that release, turn retention off for the deployment (`carlos releases retention --off` for each app that has one), because an older housekeeper prunes builds a pin or a held instance still needs. After such a rollback the console shows *settings not enforced* for instances on the older boxes: they follow their channel until the box is rolled forward again. |
| 303 | |
| 304 | **v0.25.0: instances move to the nested layout on their own.** This is |
| 305 | the first release on which an isolated (nested) instance wakes, and nested |
| 306 | is the default. On each box, migration starts unattended on the first |
| 307 | serial move after the release lands and the box's isolation probe passes: |
| 308 | every activator-managed instance that is asleep is moved into its own |
| 309 | directory and namespace at once; one that is awake gets an hour and is |
| 310 | then stopped for its move. An instance whose move is refused for a reason |
| 311 | a tenant could have planted (a file in its new directory, a second name |
| 312 | for its database) is held down automatically, and stays down across agent |
| 313 | restarts until the reason is cleared or the layout is reverted. An |
| 314 | operator who wants control over any of that acts BEFORE rolling: set |
| 315 | `CARLOS_LAYOUT_DEFAULT=flat` in each box's host.env to keep the fleet flat, |
| 316 | or record `carlos system layout migrate --box <label>` to move one box at |
| 317 | a time, or `carlos system layout revert` to opt the fleet out; a recorded |
| 318 | intent always wins over the default. Watch with `carlos status --layout`, |
| 319 | which marks a box `(nested by default)` and says why a box the default |
| 320 | has not reached is still flat. Cut v0.25.0 from a main that contains |
| 321 | 7b7d3026 (plan 4's edge openat2 fix); without it nested instances cannot |
| 322 | wake. |
| 323 | |
| 324 | **App regions rollback floor.** Once any app holds a region other than the |
| 325 | home, do not roll the console binary below the first release that |
| 326 | understands app regions: an older console places a no-region create at the |
| 327 | home. That release's version is named here, in its release notes, and in the |
| 328 | console roll recipe in carloku-infrastructure; check all three name the same |
| 329 | version before turning `CONSOLE_APP_REGIONS` on. If a rollback below the |
| 330 | floor happens anyway, write down when it started and ended: on its next |
| 331 | start the console logs "possible placement discrepancy" lines, and the |
| 332 | interval tells you which to review. Turning the switch off is the feature's |
| 333 | own rollback: it stops new non-home regions and keeps existing ones in force. |
| 334 | |
| 335 | The first release that understands app regions is v0.23.0. |
| 336 | |
| 337 | **Turning app regions on.** In order: |
| 338 | |
| 339 | 1. Stage the deployment's `CARLOS_SECRETS_KEY` on every box in every region |
| 340 | that should take placed instances, and set `CARLOS_SECRETS_PUBKEY` on the |
| 341 | console. A box without the key refuses to mint or wake placed exec instances |
| 342 | whose config contains sealed secrets. |
| 343 | 2. Roll the box release that publishes `serves_shared_pool`, |
| 344 | `accepts_inbound` and `secrets_key_fp`. Check every region's |
| 345 | `control/edges/*.json` carries them, and that each `secrets_key_fp` equals |
| 346 | the console's. |
| 347 | 3. Roll the console with the switch off. |
| 348 | 4. On dev, turn the switch on. Create an app and an instance, read their |
| 349 | region, and run the service gate below for every region. |
| 350 | 5. On production, with the switch still off, create one probe instance per |
| 351 | eligible region with an explicit `--region`, deploy a known build, and run |
| 352 | `scripts/region-service-gate.sh <probe-host> <build> <every other edge EIP>` |
| 353 | for each. Every region must pass. Delete the probes. |
| 354 | 6. Turn the switch on. Re-run step 5 after any period with the switch off, |
| 355 | and whenever a new region becomes eligible. |
| 356 | |
| 357 | **#292: instance-control changes.** The release carrying #292 changes |
| 358 | `carlos-instance-control`: it orders a host's legacy run starts against |
| 359 | its stops. So the service must restart on roll. A box whose updater is |
| 360 | v0.26.0 or later does that itself, so no manual step is needed; an older |
| 361 | updater needs the hand restart described under "Old updaters" above. |
| 362 | |
| 363 | ### Tenant email senders, plan 2 of 5 (2026-09-21) |
| 364 | |
| 365 | This release turns the one implicit per-app email sender into an explicit |
| 366 | sender record (design spec `2026-09-15-tenant-email-senders-design.md`). |
| 367 | |
| 368 | - SES event counts now attribute to the sender that sent them. an app |
| 369 | sharing a domain with a sibling app stops being charged for the |
| 370 | sibling's sends. a tenant watching their own numbers will see them move. |
| 371 | - the app's email record moved to |
| 372 | `apps/<acct>/<app>/email/senders/app.json`, with a read fallback to the |
| 373 | old `email.json`. a console rolled back after the first sender write |
| 374 | loses any pause or rotation made in between. that has to be redone by |
| 375 | hand. |
| 376 | - two prerequisites from the design spec (§4.2) before any release sends |
| 377 | with a tenant header. |
| 378 | - every domain provisioned before this release has no SES tenant |
| 379 | association, and nothing will create one on its own. a backfill, or |
| 380 | a re-run of `carlos email domains add` against every existing |
| 381 | domain, is needed before any release that sends with a tenant |
| 382 | header. |
| 383 | - the control-plane role needs `ses:CreateTenantResourceAssociation` |
| 384 | and `ses:DeleteTenantResourceAssociation` before this rolls. |
| 385 | - `CARLOS_MAIL_LEGACY_SWEEP=1` is off by default. it does two separate |
| 386 | things. it deletes the bucket's legacy `email.json` objects, and it |
| 387 | drops the database's `mail_counts` table. do not set it until the |
| 388 | release after this one has been live for a while. |
| 389 | |
| 390 | ### Tenant email senders, plan 4 of 5 (2026-09-22) |
| 391 | |
| 392 | This release adds the per-instance config layer (design spec |
| 393 | `2026-09-15-tenant-email-senders-design.md` §6.2): a third config bundle, |
| 394 | layered over an app's default and named-environment config, scoped to one |
| 395 | route's host rather than the whole app. |
| 396 | |
| 397 | - a new failure mode comes with this layer. every config sweep now also |
| 398 | lists `apps/<acct>/<app>/instances-config/` before it converges that |
| 399 | app's routes. s3 throttling, an iam policy change that adds a prefix |
| 400 | condition to `s3:ListBucket`, or a transient 503 on that call all look |
| 401 | the same from a box: the listing fails, and every route of that app |
| 402 | freezes on its last-good config file until the listing succeeds again. |
| 403 | this holds for the whole app, not just routes with an overlay of their |
| 404 | own, because the listing runs once per app regardless of who uses the |
| 405 | layer. |
| 406 | - grep the box's log for `instance config unresolved` to find it. it |
| 407 | prints once per app, naming the account, the app and one route the pass |
| 408 | reached. every other route of that app is covered by the same line and |
| 409 | the same cause. |
| 410 | - it clears itself. once the listing succeeds again, the next pass |
| 411 | converges normally, with nothing to clean up by hand. |
| 412 | - the in-repo iam policies grant `s3:ListBucket` on the whole deployment |
| 413 | bucket with no prefix condition, so a platform box is unaffected by this |
| 414 | on its own account's policy. check a hand-edited or self-hosted |
| 415 | deployment's policy before this rolls there, since a narrower grant is |
| 416 | exactly what would trip it. |
| 417 | |
| 418 | ### Availability, and why the edge had no answer (2026-09-24) |
| 419 | |
| 420 | This release moves every 502, 503 and 504 the edge sees from the Errors |
| 421 | tab to a new Availability tab, grouped by cause (errors spec |
| 422 | `2026-09-07-errors-and-analytics-design.md` §11), and adds replica |
| 423 | placements (#409). |
| 424 | |
| 425 | - roll the boxes and the console together. the boxes record causes and the |
| 426 | console shows them. the console alone gives the Availability tab, and |
| 427 | 502s already stored regroup there under "cause not recorded"; until the |
| 428 | boxes roll, new 502s land there too. boxes alone, with an old console, |
| 429 | file new records the old console lists on Errors, one row per cause, |
| 430 | named like "502 · the instance refused the connection". |
| 431 | - rolling the console back is safe. nothing in the bucket is rewritten; |
| 432 | the regrouping happens on read. |
| 433 | - the Errors tab and `carlos errors` show fewer rows. edge 502/503/504 and |
| 434 | every platform-stream line (failed wakes, exits, sidecar restarts) are |
| 435 | on Availability now. anyone watching Errors for wake failures should |
| 436 | watch Availability instead. |
| 437 | - a request whose visitor gives up after waiting five seconds or more, |
| 438 | before the instance has sent response headers, is now filed as an |
| 439 | outage, as "the visitor gave up waiting for the instance". the five |
| 440 | seconds count from when the edge received the request, so on a |
| 441 | hibernating route they include the wake. an endpoint that holds its |
| 442 | response headers longer than that by design, like a long poll, will |
| 443 | show a standing row for it. that row is expected; it is not a |
| 444 | regression. a slow export that sends its headers straight away does |
| 445 | not. |
| 446 | - a visitor who leaves within five seconds is no longer filed at all. |
| 447 | previously those were filed as a 502 on whatever path they were |
| 448 | loading, so expect fewer single-count 502s on static assets. |
| 449 | - on a route this box proxies over a tunnel or to a fleet, the front box |
| 450 | now files only its own hop failures. the box serving the app files the |
| 451 | rest, so a tunnelled 500 is filed once, not twice. two edges follow from |
| 452 | that: a visitor who leaves after five seconds can be filed as |
| 453 | "gave up waiting" on both boxes, and a far agent whose status store |
| 454 | failed to open files nothing, which the front box no longer covers. |
| 455 | - a new CLI against a console that has not been updated: |
| 456 | - `carlos availability` fails with `not found: check --account and |
| 457 | --app, and that your token's address is a member`. until the |
| 458 | console rolls, that message means the console predates the |
| 459 | command, not that the app or your membership is missing. |
| 460 | - `carlos instances create --role replica` is silently ignored by an |
| 461 | old console, which creates an ordinary home instance in that region |
| 462 | instead. update the console before creating a replica. |
| 463 | - replica placements add two registry columns (`routes.replicas`, |
| 464 | `routes.replica`) on each box's start, by an idempotent `ALTER TABLE`. |
| 465 | there is no migration to run, and a box rolled back keeps two unused |
| 466 | columns harmlessly. do not create a replica until every box is on this |
| 467 | release: only this release's DNS steering leaves replicas out of |
| 468 | latency routing. |
| 469 | |
| 470 | ### Seven days of history for sleeping instances (2026-10-08) |
| 471 | |
| 472 | This release keeps seven days of database snapshots for instances that |
| 473 | sleep, the same as instances that stay awake (#263). `carlos db pull --at` |
| 474 | (#262) can then reach back a week on them, not a day. |
| 475 | |
| 476 | - boxes only. the console is unaffected. |
| 477 | - the window fills in over the week after the roll. snapshots older than a |
| 478 | day were already deleted under the old setting, so on the day of the roll a |
| 479 | sleeping instance still reaches back about a day, and a week only seven |
| 480 | days later. |
| 481 | - an instance's history moves only while it is awake. one that sleeps for |
| 482 | several days takes no snapshot in that time, and loses none either: |
| 483 | retention also runs only while it is awake. |
| 484 | - storage grows by up to six more daily snapshots per sleeping instance, |
| 485 | plus the hourly files back to the oldest one. there are no extra s3 |
| 486 | requests. |
| 487 | |
| 488 | ### Bucket reads only when something changed (2026-10-22) |
| 489 | |
| 490 | This release stops every box and the console from reading the whole deployment bucket on every pass (#336). Writers now record which app changed in `control/changes.json`, and readers re-read only those apps. A full read of the bucket still runs every 30 minutes as a safety net. |
| 491 | |
| 492 | - roll every box and the console before you arm any reader. writers update the index as soon as they run the new build, and readers stay off until you arm them. |
| 493 | - arm a box by adding `CARLOS_CHANGE_INDEX=read` to its `host.env` and restarting it, and the console the same way in its environment. do dev first, and boxes before the console. |
| 494 | - to turn it off, remove the variable and restart. readers go back to reading the whole bucket, and writers keep updating the index, which does no harm. |
| 495 | - anyone using an older carlos CLI that writes straight to the bucket (promote, env set, trash) should update it. until they do, their changes reach boxes at the next 30-minute read, not within seconds. |
| 496 | - customer fleet boxes keep reading the way they do today. |
| 497 | - to see the effect, look at daily S3 ListBucket and GetObject requests for the deployment bucket in Cost Explorer. they should fall within a day of arming. |
| 498 | |
| 499 | ### Email sending users get a permissions boundary (2026-10-23) |
| 500 | |
| 501 | This release creates every email sending user (IAM path `/carlos-mail/`) under a permissions boundary, the way store users already are. Without one, anything that can act as the console's role can create a user on that path, give it an administrator policy and mint it a key. |
| 502 | |
| 503 | A console with `CARLOS_MAIL_REGION` set refuses to start on this release without `CARLOS_MAIL_USER_BOUNDARY`. Its value is the ARN of a customer managed policy in the same account, `arn:aws:iam::<12-digit account>:policy/<name>`, that allows `ses:SendRawEmail` and nothing else. A console without `CARLOS_MAIL_REGION` needs nothing. |
| 504 | |
| 505 | Roll it in this order: |
| 506 | |
| 507 | 1. create the boundary policy. set `CARLOS_MAIL_USER_BOUNDARY` to its ARN in every console's environment before any console gets this release. an older console ignores the variable. this release refuses to start without it. |
| 508 | 2. apply IAM step 1 to the console role. add `iam:PutUserPermissionsBoundary` on `user/carlos-mail/*`, conditioned on `iam:PermissionsBoundary` equal to the boundary ARN. without that condition the role can set any boundary it likes. add `iam:ListUsers` for the sweep in step 4. AWS does not scope `ListUsers` to a path, so it is granted on `Resource "*"`. it only reveals user names and paths, and the sweep asks only for `/carlos-mail/`. the role already has `iam:GetUser` on the path. never grant `iam:DeleteUserPermissionsBoundary`. nothing calls it. |
| 509 | 3. roll the console. from now on every new sending user gets the boundary. an existing user gets it the next time the console touches it: enable, a standalone credential, a domain change, and key rotation too, since no key is minted for a user without the boundary. |
| 510 | 4. run the sweep, which also reaches users nothing will touch again. set `CARLOS_MAIL_BOUNDARY_SWEEP=1` on one console and restart it. it logs `set the boundary on <user>` for each user it fixes. its last line is either `all N mail users bounded` or the users that are not, with why. fix any failure and run it again. then remove the variable. |
| 511 | 5. check it yourself. for every user in `aws iam list-users --path-prefix /carlos-mail/`, `aws iam get-user --user-name <user>` must show `PermissionsBoundary` with the boundary ARN. |
| 512 | 6. apply IAM step 2. condition the role's `iam:CreateUser` and `iam:PutUserPolicy` on `iam:PermissionsBoundary` equal to the boundary ARN. do this only after every console runs this release, because an older console creates users without the boundary and step 2 refuses them. |
| 513 | 7. wait a few minutes for IAM to propagate, then run the check in step 5 again with your own operator credentials, not the console role. `iam:CreateAccessKey` cannot be conditioned on the boundary, so anyone holding the console role could have made an unbounded user between step 5 and step 2. delete any user without the boundary, or set the boundary on it. the gap is closed only when this second check is clean. |
| 514 | |
| 515 | ## App config & secrets |
| 516 | |
| 517 | Per-app config lives in the deployment bucket |
| 518 | (`apps/<acct>/<app>/config.json`): plain vars via `carlos env`, secrets |
| 519 | sealed client-side via `carlos secrets` — the bucket never sees plaintext. |
| 520 | |
| 521 | carlos secrets genkey # mint CARLOS_SECRETS_KEY / CARLOS_SECRETS_PUBKEY |
| 522 | carlos env set --app keymail LOG_LEVEL=debug |
| 523 | carlos secrets set --app keymail SES_SECRET=... # sealed with CARLOS_SECRETS_PUBKEY before upload |
| 524 | carlos secrets list --app keymail # key names only — a value is never readable again |
| 525 | |
| 526 | Like `ship`/`promote` (see "From the CLI" above), `env` and `secrets` work |
| 527 | through a logged-in console with no `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` on your |
| 528 | machine at all (issue #43) — `carlos auth login` alone is enough, including |
| 529 | for `secrets set`: with no `CARLOS_SECRETS_PUBKEY` in your own shell, it |
| 530 | seals against the console's own served public key instead (set via |
| 531 | `CARLOS_SECRETS_PUBKEY` in *the console's* environment; the private half |
| 532 | never leaves the hosts that hold it either way). Direct-bucket mode |
| 533 | (`CARLOS_DEPLOYMENT_BUCKET`/`_DIR` set) is unchanged. |
| 534 | |
| 535 | The keypair is the mirror image of the release signing pair: |
| 536 | `CARLOS_SECRETS_PUBKEY` only encrypts and can be held widely (CI, laptops |
| 537 | — write-only credentials); `CARLOS_SECRETS_KEY` decrypts and belongs in |
| 538 | exactly two places, your secret manager and every host's |
| 539 | `/etc/carlos/host.env`. |
| 540 | |
| 541 | Hosts converge during the hourly root `carlos adopt` pass: with |
| 542 | `CARLOS_SECRETS_KEY` set, each pass materializes the manifest into |
| 543 | `/etc/carlos/apps/<route.Host>.env` (the file `carlos-app@.service` |
| 544 | already reads) and restarts routes whose file changed. With it unset, |
| 545 | config sync is inert — the same convention as every other optional loop. |
| 546 | A write also reaches exec-backed instances in seconds rather than waiting |
| 547 | for that hourly pass (issue #44) via the agent's own fast poll loop; run |
| 548 | `carlos env sync --app <a>` to request convergence on demand. |
| 549 | |
| 550 | Operational notes: |
| 551 | |
| 552 | - Delivery is split by backing: unit-backed routes (`Backing: "unit"`) |
| 553 | materialize to `/etc/carlos/apps/<host>.env` via the hourly root |
| 554 | `carlos adopt`; exec-backed instance routes (socket-upstream) |
| 555 | materialize to `/data/<host>.config.env` via the agent's in-process |
| 556 | hourly loop, which also bounces a running instance when its config |
| 557 | changes. At spawn the instance's own `/data/<host>.env` is layered on |
| 558 | top — it wins every conflict, so `CARLOS_ADMIN_TOKEN` and box-local |
| 559 | overrides are never clobbered from the bucket. Addr-upstream |
| 560 | (proxy-only) routes get no delivery — the process behind them is |
| 561 | externally managed. |
| 562 | - Migrate an app's config into the bucket *before* enabling |
| 563 | `CARLOS_SECRETS_KEY` on its hosts: once a manifest object exists for an |
| 564 | app, the bucket is authoritative for its env files (fully unset ⇒ |
| 565 | converges to empty). An app with no manifest at all is left alone. |
| 566 | - Values are write-only. Rotating the keypair means re-entering every |
| 567 | secret of every app from its source of record — there is no rewrap yet. |
| 568 | |
| 569 | ## Domains |
| 570 | |
| 571 | Every app has domains as a first-class concept: a **platform domain** it is |
| 572 | born with, and any number of **custom domains** layered on top — each with |
| 573 | DNS and TLS health visible in the console, and point-your-DNS-here guidance |
| 574 | when a custom domain isn't connected yet. The scheme: |
| 575 | |
| 576 | <app>.<account-sqid>.<apps-domain> kass.bab.oncarlos.com |
| 577 | <canary>.<app>.<account-sqid>.<apps-domain> pricing.kass.bab.oncarlos.com |
| 578 | |
| 579 | The account carries the sqid (base-23 over a frozen no-lookalike alphabet, |
| 580 | encoding the account's minted id — deliberately not configurable, so nobody |
| 581 | can squat a vanity label; custom domains are the vanity mechanism), the app |
| 582 | carries its name — app names are unique per account, not globally. Account #1 |
| 583 | is `ops`, the deployment operator's built-in account, created by migration; |
| 584 | CLI-added apps live there, and its sqid (`bab`) is identical on every |
| 585 | deployment. The |
| 586 | console writes `apps/<app>/account.json` at claim time; the agent's |
| 587 | reconcile pass (riding the 5-minute tick) reads those manifests and mints |
| 588 | platform routes, and removes only routes it minted itself — **custom |
| 589 | domains are never touched by reconciliation**. DNS needs one wildcard A |
| 590 | record (`*.<apps-domain>` → edge; RFC 4592 wildcards match multiple |
| 591 | labels); certs come per-host from the edge's existing ACME machinery. |
| 592 | |
| 593 | A platform domain is an **alias**: a second name for the app's canonical |
| 594 | route, pointing at the same backend. It is not a lifecycle host — release |
| 595 | adoption and the fleet status counts both skip minted routes, so a |
| 596 | deployment that turns this feature on does not double-stage artifacts or |
| 597 | publish an inflated instance count. |
| 598 | |
| 599 | **Three honest limits**, none of them papered over: |
| 600 | |
| 601 | - **Minting is additive — an app is never re-homed.** Once an app has a |
| 602 | platform apex, changing its account does *not* mint a new one or remove |
| 603 | the old: one app, one apex, and moving an app between accounts is a |
| 604 | deliberate operator migration (delete the old apex route by hand, let the |
| 605 | next pass mint the new one), never a silent side effect of editing a |
| 606 | manifest. |
| 607 | - **A hibernating app's platform domain 502s while the app sleeps.** The |
| 608 | activator arms on the canonical route only; minted routes deliberately do |
| 609 | not copy the `hibernate` flag, because a minted route carrying it would |
| 610 | make the activator spawn a *second* process for the same app against an |
| 611 | empty `DBPath` — meaningfully worse than a 502. Wake the app through its |
| 612 | canonical host; the platform domain serves normally once it's up. |
| 613 | - **Canary hosts are minted for static apps only.** For a static app the |
| 614 | channel genuinely is the backend (the edge resolves the channel pointer |
| 615 | to find that release's tar), so a canary host serves exactly the canary. |
| 616 | An instance app's minted canary would only be able to alias the app's |
| 617 | *stable* process — a URL claiming to serve the canary while proxying |
| 618 | stable — so none is minted; spawning a real per-host canary instance is |
| 619 | future work (`carlos canary`), and until it exists the pass logs the gap |
| 620 | rather than faking it. |
| 621 | |
| 622 | **Config surface:** |
| 623 | |
| 624 | - `PLATFORM_APPS_DOMAIN` — the deployment's apps domain (flagship: |
| 625 | `oncarlos.com`), read by `carlos agent`, the edge, and the console (the |
| 626 | member-facing `carlos instances enable` pins manifests to |
| 627 | `<sqid>.<this domain>`; the console refuses a malformed value at boot). |
| 628 | **Unset = the whole feature is inert**: no minting, no manifests read, |
| 629 | the enable door answers 503 — the same litestream convention as |
| 630 | everything else here. |
| 631 | - `CONSOLE_RESERVE_PLATFORM_DOMAINS` — set to `true` on a deployment that |
| 632 | hosts apps for other people (the flagship, carloku). `carlos domains |
| 633 | attach` then refuses every name under `PLATFORM_APPS_DOMAIN` and under the |
| 634 | console's own hostname. **Unset = false**: a self-hosted deployment can |
| 635 | attach any name except the console's own hostname and |
| 636 | `CONSOLE_DENY_DOMAINS`. A value that is not a boolean stops the console at |
| 637 | boot. |
| 638 | - `CONSOLE_DENY_DOMAINS`: comma-separated hostnames in the deployment's own domains (flagship: `carloku.com`). `carlos domains attach` refuses each one, and every name under it, to members. An owner of the ops account, or an address on `CARLOS_SYSTEM_OPERATORS`, can still attach one to an app with `carlos domains attach --platform`, and after that only they can change or detach it. The console's own hostnames, the steering host, the tunnel host and, with `CONSOLE_RESERVE_PLATFORM_DOMAINS`, everything under the apps domain stay refused to everyone. |
| 639 | - `CONSOLE_DOMAIN_PROOF` — set to `true` to require proof of control before |
| 640 | an attached hostname is served: a TXT record at `_carlos-challenge.<host>` |
| 641 | (`carlos domains verify`). A deployment that hosts apps for other people |
| 642 | must set it, after the fleet runs a build that knows pending claims. |
| 643 | **Unset = false**: attach is first-writer-wins, as before. A value that is |
| 644 | not a boolean stops the console at boot. |
| 645 | - `PLATFORM_EDGE_IPS` — comma-separated edge IPs the DNS health checks |
| 646 | compare against, read by both the agent (checks) and the console |
| 647 | (guidance text), so the two can never name different addresses. Default: |
| 648 | the box's public IP via IMDS, best-effort; if neither is knowable the |
| 649 | checks stand down and health reads unknown, never a guess. |
| 650 | - `CONSOLE_APP_REGIONS`: `on` lets new apps be given a region other than the |
| 651 | home. It needs `CONSOLE_HOME_REGION` set too. With the home unnamed, the |
| 652 | console logs once and leaves app regions off. Default off. Automatic |
| 653 | placement outside the home also needs `CARLOS_SECRETS_PUBKEY`, so the |
| 654 | console can check that a box holds the deployment's secrets key. |
| 655 | - `CONSOLE_GEOIP_DB`: the path to an operator-supplied MaxMind-format `.mmdb`. |
| 656 | DB-IP Lite is recommended, and it requires the attribution the new-app page |
| 657 | shows. Unset or unusable turns the IP step of the region guess off, and the |
| 658 | console starts either way. Replace the file by atomic rename, then restart. |
| 659 | For the platform-wide GeoIP that every edge serves to apps, use |
| 660 | `carlos system geoip publish` instead (docs/site/cli.md, "Platform GeoIP"). |
| 661 | Uploads are staged in `geoip-staging/` next to `CONSOLE_DB`, which the |
| 662 | console clears at startup. |
| 663 | - `CARLOS_SIDECAR_ACCOUNTS`: comma-separated account sqids (not names) that |
| 664 | may still enable and create native sidecars. Set, every other account gets |
| 665 | 403 `sidecars-paused` from `carlos sidecars enable` and `carlos sidecars |
| 666 | create`. List, get, wake and delete still work, and existing sidecars keep |
| 667 | running. Native sidecars run on the flat tier, outside the nested tenants' |
| 668 | isolation, until nested sidecars land (discussion #370). **Unset or empty = |
| 669 | no gate.** An entry that is not a sqid stops the console at boot. |
| 670 | |
| 671 | **Canonical-host caveat**, stated honestly: an app that pins its own host |
| 672 | (e.g. keymail's `--host keymail.dev`) will redirect or 404 on its platform |
| 673 | domain — correct behavior. Rastrillo apps accept any granted host from day |
| 674 | one; for legacy apps the platform domain is the pre-custom-domain bootstrap |
| 675 | URL. |
| 676 | |
| 677 | ## Fleet status |
| 678 | |
| 679 | Every deployment computes a fleet status view: how many instances are |
| 680 | running, how busy the boxes are, whether replication is healthy — the |
| 681 | CARLOS analogue of Eleven's own status pages. Each box's agent (`carlos |
| 682 | agent`) writes its own snapshot to the deployment bucket every 5 minutes; |
| 683 | whichever box holds the status lease that tick merges every snapshot into |
| 684 | `status/fleet.json` plus the embedded viewer (`status/index.html`, |
| 685 | `status/status.css`) and republishes all three. The edge serves them back |
| 686 | out through a `kind=status` route, bucket-backed with a short cache — no |
| 687 | release machinery involved, no CAS churn, just "read the key, cache it |
| 688 | briefly, serve it." |
| 689 | |
| 690 | Three commitments hold end to end, enforced in the writer rather than the |
| 691 | page: the page publishes only fleet-wide aggregates and band words |
| 692 | (`comfortable`/`busy`/`critical`, `plenty`/`getting-full`/`at-capacity`) — |
| 693 | never raw ratios, hostnames, or machine ids. An unknown reading publishes |
| 694 | loudly as `null`, and a box that has stopped reporting renders `stale`, |
| 695 | never silently dropped. App names are opt-in only, via `carlos add |
| 696 | --status-named`, and everything else folds into one honest `apps_unlisted` |
| 697 | count, so the aggregate never lies by omission. |
| 698 | |
| 699 | A route added this way is described by no bucket record, so nothing converges |
| 700 | it: its channel and environment are box-local until an instance record adopts |
| 701 | it. A provisioned route's channel is repointed through the record, not here. |
| 702 | |
| 703 | **Config surface** — every env var this feature reads, gathered here since |
| 704 | no single reader owns the whole contract: |
| 705 | |
| 706 | - `CARLOS_STATUS_BUCKET` / `CARLOS_STATUS_PREFIX` / `CARLOS_STATUS_REGION` |
| 707 | — read by `carlos agent`'s status loop: the bucket each box writes its |
| 708 | snapshot to and, when it wins the lease, publishes the merged feed to; |
| 709 | region also feeds the cross-box status-lease election. **The serving side |
| 710 | (the edge's `kind=status` route and the console's operator view) must be |
| 711 | pointed at this same bucket** — nothing enforces that agreement |
| 712 | automatically, publishing and serving are separately configured. To make |
| 713 | the common case foolproof, an unset `CARLOS_STATUS_BUCKET` defaults to |
| 714 | `CARLOS_DEPLOYMENT_BUCKET` (an explicit `CARLOS_STATUS_BUCKET` still |
| 715 | always wins), so a single-bucket deployment needs only one env var for |
| 716 | both release adoption and status. Both unset disables the whole loop — |
| 717 | inert until configured, the same convention litestream replication uses. |
| 718 | **Honest limitation**: that default only ever resolves to an S3 bucket — |
| 719 | a deployment adopting from `CARLOS_DEPLOYMENT_DIR` (no bucket at all) |
| 720 | does not publish fleet status yet; give it its own `CARLOS_STATUS_BUCKET` |
| 721 | if it wants the page. `CARLOS_STATUS_PREFIX` does double duty: it |
| 722 | namespaces the status-lease key, AND (customer-fleets plan, Task 5) |
| 723 | rebases every key the status loop's AND the metrics loop's store |
| 724 | touches — a fleet box set to `CARLOS_STATUS_PREFIX=fleets/<name>/` writes |
| 725 | its own status snapshot at `fleets/<name>/status/boxes/<label>.json` and |
| 726 | its raw/rollup metrics and alert state under `fleets/<name>/metrics/boxes/…` |
| 727 | and `fleets/<name>/alerts/boxes/…`, instead of the shared paths a platform |
| 728 | box uses — so neither leaks into the platform's own `fleet.json`, |
| 729 | monitoring page, or alert webhook. Empty (every platform box today) is a |
| 730 | pure passthrough. A **publishing** box with a prefix set would relocate |
| 731 | `status/fleet.json` itself under that prefix too — moot today, since |
| 732 | nothing sets a prefix on a publishing (non-fleet) box, and a fleet box |
| 733 | never publishes at all (see `CARLOS_FLEET`, below). |
| 734 | - A box with `CARLOS_HIBERNATE_BUCKET` set but no deployment store |
| 735 | (`CARLOS_DEPLOYMENT_BUCKET` or `CARLOS_DEPLOYMENT_DIR`) refuses placed exec |
| 736 | instances, because it cannot tell whether their config is sealed. Set |
| 737 | `CARLOS_NO_SEALED_CONFIG=1` to say this deployment has none. |
| 738 | - `CARLOS_FLEET` — read by `carlos agent`'s status loop (customer-fleets |
| 739 | plan, Task 5). Set (to the fleet's name) marks this agent as a fleet box: |
| 740 | it still self-reports its own snapshot every tick, but never contends for |
| 741 | the cross-box publish lease and never merges/publishes `status/fleet.json` |
| 742 | — that feed belongs to the platform's own boxes only. Unset (every |
| 743 | platform box today) is unaffected. |
| 744 | - `CARLOS_FLEET_BOX` — this box's operator-chosen `fleet_boxes.label` |
| 745 | (customer-fleets plan, Task 6 fix round). This — NOT the public status |
| 746 | page's hashed machine-id pseudonym — is the box's real identity: the |
| 747 | channel name the edge dials is `<fleet>/<label>`, so both the fleet |
| 748 | binding a box publishes on wake and its own heartbeat filename |
| 749 | (`fleets/<fleet>/status/boxes/<label>.json`) use it. Optional: defaults to |
| 750 | `CARLOS_TUNNEL_NAME` with the `<fleet>/` prefix stripped, since a fleet |
| 751 | box's tunnel name already has this shape. Both the fleet name and the |
| 752 | resolved label are validated with `registry.ValidTunnelName`; a fleet box |
| 753 | whose identity fails to resolve refuses (loudly) to arm binding-publish at |
| 754 | all, and its heartbeat falls back to the hashed pseudonym instead of going |
| 755 | dark. |
| 756 | - `STATUS_BOX_REGION` — also read by `carlos agent`'s status loop: an |
| 757 | operator-set, coarse region word (e.g. `"EU · Ireland"`) carried onto |
| 758 | this box's snapshot verbatim; unset omits the field. Free text the |
| 759 | operator chooses, never derived from cloud metadata. |
| 760 | Keep it geographically consistent with the region's place name in the |
| 761 | console's region picker (for example "EU · Ireland" beside |
| 762 | "Ireland · Dublin"); it may stay coarser. |
| 763 | - `STATUS_HOST` / `STATUS_PUBLIC` — not read by the agent process itself; |
| 764 | consumed by ops/tofu to decide whether the edge fronts the bucket's |
| 765 | `status/` prefix with a public hostname. `STATUS_PUBLIC=1` publishes the |
| 766 | page at `STATUS_HOST`; leaving it unset keeps the feed private, visible |
| 767 | only to console operators (`console grant --operator <address>`). |
| 768 | |
| 769 | **Dry run.** Before pointing a binary at a real bucket, `carlos status |
| 770 | --dry-run <dir>` runs the whole publish pipeline — `CollectBox` → |
| 771 | `WriteBox` → `BuildFleet` → `PublishFleet` — against a scratch directory |
| 772 | stuffed with fixture secrets (hostnames, a machine id, mid-band fractions), |
| 773 | then reads every object it just published back out and fails loudly if any |
| 774 | of them leaked. There is no live-publish CLI verb; the 5-minute loop inside |
| 775 | `carlos agent` is the only publisher. |
| 776 | |
| 777 | **Incident fallback, and its actual limits.** The page footer links "if this |
| 778 | page is unreachable" straight at the bucket's own copy of `fleet.json`, but |
| 779 | that link is built client-side from `location.origin` (see |
| 780 | `internal/status/assets/index.html`) — it is only meaningful once you are |
| 781 | *already* viewing the page served from the bucket's own origin (the public |
| 782 | `STATUS_HOST`, or the raw bucket URL). It is not a link that rescues you from |
| 783 | a *down* deployment; if the edge/console are unreachable you won't be looking |
| 784 | at this page's footer in the first place. The real operational fallback for |
| 785 | a down deployment is fetching the objects directly, either the raw bucket |
| 786 | URL in a browser or: |
| 787 | |
| 788 | aws s3 cp s3://<bucket>/status/fleet.json - |
| 789 | aws s3 cp s3://<bucket>/status/index.html - |
| 790 | |
| 791 | One current wrinkle: `index.html` fetched straight from S3 currently |
| 792 | **downloads instead of rendering** in a browser, because the store's `Put` |
| 793 | sets no `Content-Type` and S3 falls back to `application/octet-stream`. It |
| 794 | still opens fine as plain text in an editor, and `aws s3 cp ... - | less` |
| 795 | works, but "open the bucket URL in a browser and read it" doesn't work yet |
| 796 | for the HTML — see the design spec's rollout list (§5) for the queued fix |
| 797 | (setting content-type on publish). `fleet.json` is unaffected (any client |
| 798 | reading raw JSON already expects to view its own headers). |
| 799 | |
| 800 | A fully-private deployment (`STATUS_PUBLIC` unset) whose console is down has |
| 801 | no page at all — the `aws s3 cp` fallback above is the entire story there; |
| 802 | nothing papers over it. |
| 803 | |
| 804 | ## Host & process monitoring |
| 805 | |
| 806 | Operator-grade monitoring layered on top of fleet status: real CPU, memory, |
| 807 | and disk numbers per host and per process, with history, alerting, and a |
| 808 | console dashboard — needles where the public status page shows only band |
| 809 | words. See |
| 810 | the `2026-08-02-host-process-monitoring-design.md` design spec (private `platform-infrastructure` repo) |
| 811 | for the full design. |
| 812 | |
| 813 | **Bucket layout:** |
| 814 | |
| 815 | - `metrics/boxes/<box>/raw/<YYYY-MM-DD-HH>.json` — 60-second host + per-unit |
| 816 | samples, one object per UTC hour, retained 48 hours. |
| 817 | - `metrics/boxes/<box>/days/<YYYY-MM-DD>.json` — hourly min/avg/max rollups, |
| 818 | kept forever (the capacity/trend history). |
| 819 | - `alerts/boxes/<box>.json` — on-box alert state (fire/clear, hysteresis). |
| 820 | |
| 821 | **Config surface** — read by the box agent from `/etc/carlos/host.env`: |
| 822 | |
| 823 | - `CARLOS_ALERT_WEBHOOK` — optional; the agent POSTs the alert JSON here on |
| 824 | fire and on clear. Unset simply skips the webhook channel (keymail |
| 825 | delivery and the console UI channel are unaffected). |
| 826 | - `CARLOS_ALERT_SNS_TOPIC_ARN` — optional; the agent publishes each fire and |
| 827 | clear to this SNS topic as a plain-text email-sized message, signed with |
| 828 | the box's instance role (which needs `sns:Publish` on the topic). The |
| 829 | region is read from the ARN, not from `AWS_REGION`. Set it to the box's |
| 830 | own region's alerts topic and anyone subscribed to that topic gets the |
| 831 | email. Either sink works alone and both may be set. A failed send is |
| 832 | logged and counted, never fatal. The SDK tries a retryable failure up to |
| 833 | three times within ten seconds. After that, a later tick does not send |
| 834 | that alert again, unless the same tick's bucket write also failed, which |
| 835 | repeats the whole transition on the next tick. A value that is not an SNS |
| 836 | topic ARN (a subscription ARN, for example) is logged and the sink stays |
| 837 | off. So does a client that cannot be built, which `carlos status --live` |
| 838 | shows as `alerts=sns-unavailable`. |
| 839 | - With neither sink set, the box says so: the agent logs `NO SINK |
| 840 | CONFIGURED` at start, and the box's private status snapshot carries |
| 841 | `metrics.alerts` with both sinks false, which `carlos status --live` prints |
| 842 | as `alerts=not-delivered`. |
| 843 | - `CARLOS_ALERT_DISK_PCT` / `CARLOS_ALERT_MEM_PCT` / `CARLOS_ALERT_CPU_PCT` — |
| 844 | optional threshold overrides for the built-in disk / memory / host-CPU |
| 845 | rules, as percent integers 0-100 (e.g. `95`). Anything else is ignored |
| 846 | with a log line, and that rule keeps its built-in default. |
| 847 | |
| 848 | **Console surfaces:** |
| 849 | |
| 850 | - `GET /fleet/metrics` — operator-only dashboard: per-box needles and |
| 851 | 48-hour sparklines, a per-box per-unit table (ordered by CPU, busiest |
| 852 | first), and active alerts at the top. |
| 853 | - A "Resources" section on each app's own page — owner-facing, scoped to |
| 854 | that app's own instances only (current CPU and memory plus 48-hour |
| 855 | sparklines); never host-level totals or other tenants' units. |
| 856 | |
| 857 | The `days/` rollups are written from the first flush onward but nothing |
| 858 | renders them yet — they are the history a later trend view will read. |
| 859 | |
| 860 | ## App capabilities |
| 861 | |
| 862 | A handful of apps need more from the platform than the `carlos-app@<host>` |
| 863 | template gives every instance for free — a subcommand other than `serve`, a |
| 864 | privileged port, the box's cloud instance role, or the resource headroom a |
| 865 | document-conversion engine needs. Rather than hand-provision a bespoke unit |
| 866 | per app, a release **declares** what it needs and the platform decides how — |
| 867 | and whether — to provide it, from a closed vocabulary of eight names. See |
| 868 | the `2026-08-03-app-capabilities-design.md` design spec (private `platform-infrastructure` repo) |
| 869 | for the full design; this section is the operator-facing summary. |
| 870 | |
| 871 | **The manifest shape** — an optional `capabilities` object on the release |
| 872 | manifest, additive to schema 1 (a manifest with no `capabilities` key parses |
| 873 | as "none", today's behaviour byte for byte): |
| 874 | |
| 875 | "capabilities": { |
| 876 | "v": 1, |
| 877 | "command": "relay", |
| 878 | "daemon": true, |
| 879 | "children": true, |
| 880 | "runtime_dir": "keymail", |
| 881 | "ports": [ { "port": 25, "name": "mx" } ], |
| 882 | "data": { "replicate": "app" }, |
| 883 | "host_role": true |
| 884 | } |
| 885 | |
| 886 | A document-conversion engine (Gotenberg, an OCR service) declares a different |
| 887 | shape — `host_role` and `doc_engine` are mutually exclusive and a manifest |
| 888 | carrying both is refused at ship, because `host_role` re-opens the cloud |
| 889 | metadata address and `doc_engine` exists to fence the engine's egress away |
| 890 | from private networks: |
| 891 | |
| 892 | "capabilities": { |
| 893 | "v": 1, |
| 894 | "daemon": true, |
| 895 | "children": true, |
| 896 | "doc_engine": true |
| 897 | } |
| 898 | |
| 899 | `command`, `daemon`, `children`, `runtime_dir` and `data` are self-service — |
| 900 | they shape only the app's own sandbox and materialize from the manifest |
| 901 | alone. `ports`, `host_role` and `doc_engine` are operator-consented: declaring |
| 902 | one asks for it, but nothing materializes until an operator has granted it |
| 903 | **on that specific box**, because the resource (a privileged port, the box's |
| 904 | IAM role, the private-network egress fence that pays for a conversion |
| 905 | engine's JIT and memory headroom) is box-local, not fleet-wide. |
| 906 | |
| 907 | **Declaring a capability** rides `carlos ship --capabilities <file.json>`, |
| 908 | where the file holds exactly the `capabilities` object above (not a whole |
| 909 | manifest). The same strict decoder and validation gate ship and adopt both |
| 910 | run: an unknown capability name or field, a bad charset, a reserved |
| 911 | `runtime_dir`, or an out-of-range/reserved port is refused at the ship |
| 912 | terminal, where a human is standing to read it — no per-capability flags, |
| 913 | because a flag vocabulary would drift from `internal/caps`'s own. |
| 914 | |
| 915 | **Granting a capability** is `carlos route`, per route, per box: |
| 916 | |
| 917 | carlos route --host keymail.dev --backing unit # first: grants need it |
| 918 | carlos route --host keymail.dev --grant port:25 |
| 919 | carlos route --host keymail.dev --grant host-role |
| 920 | carlos route --host gotenberg.docs.example.com --grant doc-engine |
| 921 | carlos route --host keymail.dev --revoke port:25 |
| 922 | carlos route --host keymail.dev --hibernate=false |
| 923 | |
| 924 | Each of `--backing`, `--grant`/`--revoke` and `--hibernate` is its own operation: |
| 925 | they require `--host` (never `--app` — a bulk grant is precisely the operation |
| 926 | nobody should be able to type by accident), and none of them combines with |
| 927 | another, or with `--channel`/`--environment`, in one invocation. |
| 928 | |
| 929 | The order above is not cosmetic. **A grant requires unit backing**: a route |
| 930 | that is not a `kind=instance`, `backing=unit` row has nothing for a grant to |
| 931 | materialize into, so `--grant`/`--revoke` are refused there, and grants cannot |
| 932 | be staged ahead of a migration. `--backing unit` is the switch that moves a |
| 933 | route from an agent-run exec child onto a `carlos-app@<host>` systemd |
| 934 | instance; an exec-backed route that declares capabilities materializes |
| 935 | nothing and is advised once per pass until it is switched over (see the |
| 936 | design's §5 migration steps). Stamping it arms the **hourly** |
| 937 | `carlos-adopt.timer`, and `carlos adopt` for a `daemon` route enables and |
| 938 | STARTS the service — so the data copy comes before the stamp, not after it. |
| 939 | |
| 940 | A declared-but-ungranted capability does not fail the app or fail adoption: |
| 941 | the drop-in simply is not written, and the hourly adopt pass logs one line |
| 942 | naming the route, the capability, and the exact grant command. The mirror |
| 943 | line is there too — a token this box has granted that the live release no |
| 944 | longer declares materializes nothing, and is named with the revoke command |
| 945 | rather than left sitting in the registry looking like live authority. |
| 946 | |
| 947 | **`doc-engine` carries two box preconditions** the platform cannot check for |
| 948 | you, which is what makes granting it an operator decision rather than a |
| 949 | checkbox. The grant buys the app JIT memory and a raised resource tier, and |
| 950 | in the same drop-in fences its egress away from every private network range — |
| 951 | RFC1918, link-local, CGNAT, IPv6 ULA and the NAT64 prefix — because a |
| 952 | conversion engine fetches whatever URLs the documents handed to it name, and |
| 953 | an unfenced one is an SSRF proxy into the VPC. Two consequences: |
| 954 | |
| 955 | - **The box must hand the tenant a loopback DNS resolver.** The fence |
| 956 | deliberately leaves 127.0.0.0/8 open — the engine drives its own Chromium |
| 957 | and LibreOffice over loopback — but it closes 10.0.0.0/8, and on AWS the |
| 958 | VPC resolver lives inside that range (VPC base + 2). A box running |
| 959 | systemd-resolved on 127.0.0.53 resolves through loopback and the fence is |
| 960 | transparent; a box pointing tenants straight at the VPC resolver gives them |
| 961 | an engine that cannot resolve a hostname. |
| 962 | - **The box should be dedicated.** Loopback staying open means an SSRF'd |
| 963 | engine can still reach anything listening on 127.0.0.1 on its own box — a |
| 964 | co-tenant console, a co-tenant's loopback-bound upstream. The fence closes |
| 965 | the network half of that risk; not co-locating tenants closes the rest. |
| 966 | Run `doc-engine` routes on a box of their own. |
| 967 | |
| 968 | **Deploy order is pinned**, same rule and reason as scoped signing keys: the |
| 969 | platform binaries that understand a capability name have to be on the box |
| 970 | *before* a release declaring it, never after. An older binary meeting a |
| 971 | manifest it does not understand refuses the version on that box — not |
| 972 | staged, not pointed at — rather than materialize a unit missing something |
| 973 | the app says it needs. |
| 974 | |
| 975 | **"Platform binaries" includes the console binary**, and on the flagship that |
| 976 | is the one that matters: `carlos ship` in API mode posts through the console, |
| 977 | so the console is what either understands the `capabilities` key or silently |
| 978 | drops it as an unknown JSON field — storing a manifest that declares nothing, |
| 979 | which on a box is the instruction to REMOVE the drop-ins. The ship response |
| 980 | therefore echoes the capability block it stored, and `carlos ship` fails hard |
| 981 | when it sent a block and gets no echo back ("this console does not understand |
| 982 | capability declarations — update the console binary first, or ship without |
| 983 | `--capabilities`"). Update the console before shipping a capability-bearing |
| 984 | release. |
| 985 | |
| 986 | **What refuses vs what leaves alone** — the distinction that matters most: |
| 987 | |
| 988 | - An unreadable or unresolved staged manifest **leaves every drop-in exactly |
| 989 | as it is** and logs once per pass. A bucket hiccup must never strip `:25` |
| 990 | off a running mail server. |
| 991 | - A manifest that actually resolves and no longer declares a capability (or |
| 992 | a grant that gets revoked) **removes** the generated drop-in. |
| 993 | - A hand-edited drop-in — anything at the generated path without the |
| 994 | `# GENERATED by carlos adopt — do not hand-edit` marker on its first |
| 995 | line — is never overwritten and never deleted; it is reported and that |
| 996 | unit's half is skipped while the other half still converges. |
| 997 | - An unknown capability name or an unsupported `"v"` refuses the *version on |
| 998 | that box*, not the whole fleet — the box keeps serving what it already |
| 999 | runs. |
| 1000 | |
| 1001 | Console gets, at most, a read-only line beside each route's other facts — |
| 1002 | "granted: host-role, port:25" for a route with grants, nothing at all |
| 1003 | otherwise, and only on routes belonging to the account whose page it is. |
| 1004 | There is no console UI for granting or revoking; grants stay a `carlos route` |
| 1005 | operation on the box, and nothing about the console's |
| 1006 | sign-in/session/CSRF/owner gates changes. |
| 1007 | |
amadan