| 1 | # 🤖 The CARLOS platform, member-side |
| 2 | |
| 3 | What `carlosframework/platform` does for an app, and the `carlos` CLI |
| 4 | surface a member drives it with. Everything here works with **zero |
| 5 | infrastructure access** — no SSH, no AWS console, no box commands. That is |
| 6 | a design law, not a convenience: every command is built for a user who has |
| 7 | nothing but the CLI and a browser. If a task seems to need a box, either |
| 8 | you are self-hosting and operating the platform itself, or you have found |
| 9 | a product gap to file — never a workaround to build. |
| 10 | |
| 11 | Snapshot date 2026-10-18, describing what Carloku serves: platform |
| 12 | v0.28.0, released 2026-10-17. Verbs are stable, flag details evolve — trust |
| 13 | `carlos <verb> -h` over this file, and remember a CLI can offer a flag the |
| 14 | console you are talking to has switched off or does not have yet |
| 15 | ("Switched off, or not released" at the end). The CLI ships for macOS/Linux (brew, |
| 16 | apt, static binaries) and Windows (client-only zip — no self-replace, |
| 17 | `carlos update` defers to a fresh download). |
| 18 | |
| 19 | **Flags are written `--flag`.** Single-dash forms still parse and always |
| 20 | will, but as of 2026-08-27 the documented face of every verb is the |
| 21 | double dash, and the aliases are `-a` (`--app`), `-c` (`--console`), `-e` |
| 22 | (`--environment`), `-v` (`--version`). Three flags turn up nearly |
| 23 | everywhere and are not repeated below: `--app`, `--account` (or |
| 24 | `--account-id` when you want a sqid matched as a sqid and nothing else), |
| 25 | and `--console`, which you need only when this terminal is logged in to |
| 26 | more than one deployment. |
| 27 | |
| 28 | ## Start with `carlos auth whoami` |
| 29 | |
| 30 | Before concluding you cannot reach a CARLOS deployment, run it: |
| 31 | |
| 32 | ``` |
| 33 | carlos auth whoami # the default console |
| 34 | carlos auth whoami --console https://<console> # a specific one |
| 35 | ``` |
| 36 | |
| 37 | It prints the identity, the terminal's label, the expiry, and every |
| 38 | account with its sqid — which is also where you find the sqid your app's |
| 39 | hostname needs. |
| 40 | |
| 41 | This is first because the obvious check is the misleading one. An agent's |
| 42 | instinct is to `curl` the console and read the answer: the console will |
| 43 | `302` you to a login page every time, because a browser session is not |
| 44 | what the CLI holds. That redirect says nothing about whether you have |
| 45 | access, and reading it as "no access" has already cost one session an |
| 46 | afternoon and ended in a filed issue. **The CLI's credential is the only |
| 47 | thing that answers the question.** |
| 48 | |
| 49 | Same for the deployment's identity: a self-hosted console is not |
| 50 | `carloku.com`, and the console origin and the apps domain are different |
| 51 | hostnames (Tito's are `carlos.tito.io` and `platform.tito.io`). If a |
| 52 | project commits a `.carlos/config`, that names the console it belongs to; |
| 53 | otherwise `--console` does. |
| 54 | |
| 55 | ## What the platform owns (never hand-roll these) |
| 56 | |
| 57 | - **Routing and TLS** — the edge is the only process on :443; the route |
| 58 | table doubles as the ACME allowlist; per-host certs auto-obtained and |
| 59 | renewed, including customer domains and (via ACME delegation) wildcards. |
| 60 | - **Replication** — Litestream on every instance database, run by the |
| 61 | platform's host agent; restore drills are the platform's job too. A |
| 62 | hibernating route's rung is tunable (`stream` — the default, about once |
| 63 | a second — / `batched`, five minutes / `daily`), a loss-window trade an |
| 64 | operator makes on a large, rarely-written database. |
| 65 | - **Hibernation** — provisioned instances doze when idle and wake on |
| 66 | request; live and default-on, not a future feature. Cents-per-month |
| 67 | idle cost is the platform's economic story. |
| 68 | - **Process supervision and restarts** — instances are converged from |
| 69 | bucket records; `carlos restart` cycles them in seconds, console-side. |
| 70 | - **Config delivery** — `carlos env` / `carlos secrets` writes converge |
| 71 | onto boxes within seconds (a serial bump, 2s poll). No ssh-delivered |
| 72 | EnvironmentFiles. |
| 73 | - **Deploy verification** — `carlos deploy` follows the rollout box by |
| 74 | box where boxes report it (they do on Carloku), and otherwise watches |
| 75 | the URL: the app's build probe if it declares one |
| 76 | (`carlos apps build-probe`), else the `X-Carlos-Version` header, which |
| 77 | the edge stamps from the route row the moment adoption lands, whether |
| 78 | or not the process has restarted yet. |
| 79 | - **Outbound email** — the platform mints a sending identity on a domain, |
| 80 | publishes DKIM/SPF/DMARC where it controls the zone, and delivers working |
| 81 | SMTP credentials as config. Apps never hold an AWS key and nobody runs an |
| 82 | MTA. See "Sending email" below. |
| 83 | - **Scheduled work** — the platform keeps the clock: a declared schedule |
| 84 | POSTs a tick to a path in your app at the time it is due, waking a |
| 85 | hibernating instance to run it. No in-app scheduler, no cron line on a |
| 86 | box. See "Scheduled work" below. |
| 87 | - **Errors, availability and analytics** — the edge and the log stream |
| 88 | feed all three, with no SDK, no JavaScript, no cookie and no stored IP. |
| 89 | Errors and availability collect from the day an app ships; analytics is |
| 90 | opt-in. See "Watching an app" below. |
| 91 | - **Password gate** — `carlos gate` puts one shared password in front of |
| 92 | an app's hostnames, enforced at the edge ahead of the cache and the |
| 93 | activator, so the app behind it never sees the cookie. A curtain over a |
| 94 | hostname, not accounts for the app's own users. |
| 95 | - **DNS steering** — `carlos steering ... latency` opts a host into a |
| 96 | latency record per armed edge so the nearest one answers. A static site |
| 97 | on a platform hostname is steered from creation without asking; a |
| 98 | custom domain is not, because its DNS is yours. |
| 99 | |
| 100 | ## Concepts |
| 101 | |
| 102 | - **Account** — the tenancy unit. Public short id (**sqid**, e.g. three |
| 103 | letters) appears in default hostnames; it is not a secret. Roles are |
| 104 | owner/member. `carlos auth whoami` shows yours. |
| 105 | - **App** — named per account (not globally). Claimed with |
| 106 | `carlos apps create`; deleted apps sit in a 30-day trash. |
| 107 | - **Release** — immutable, content-addressed, produced by the ship half |
| 108 | of `carlos deploy` (or by `carlos ship` on its own). |
| 109 | Kinds: `binary` (default) and `static`. Versions are free-form; git |
| 110 | short sha is the house convention. |
| 111 | - **Channels and pipelines** (release pipelines v2, live 2026-08-22) — |
| 112 | **a new app is born with one channel, named `edge` by default** |
| 113 | (renameable at creation): that is its production, and `carlos deploy` |
| 114 | lands on it. Multi-channel ceremony is opt-in: a console-mediated |
| 115 | **pipeline** declares an ordered channel list (names are app-defined; |
| 116 | promotion onto channel N must come from N−1; the first is the entry |
| 117 | channel) with per-channel, default-permissive policy — `bake` |
| 118 | duration, `passkey` step-up, `promote_approvals`, and |
| 119 | `change_approvals` (which also guards editing/removing the channel |
| 120 | and fast-tracking through its bake). `canary/<slug>` stays a reserved |
| 121 | platform namespace outside any pipeline: always allowed, zero bake, |
| 122 | per-session dead ends. Apps with no pipeline keep the legacy frozen |
| 123 | ladder (`edge → beta → stable`, holds 0/24h/72h, unconditional |
| 124 | passkey on stable — reaching stable cuts a semver tag). `--hotfix`, the |
| 125 | bake bypass, is an operator act on the direct-bucket path only: the |
| 126 | console refuses it by name, so a member never has it. Box-side, bake |
| 127 | changes **ratchet**: a shorter window is honoured only after the previously-known window has elapsed |
| 128 | once on the box's own clock, so a compromised console session cannot |
| 129 | collapse a hold and ship in the same hour. |
| 130 | - **Production flag** — legacy: superseded by per-channel pipeline rules |
| 131 | for pipelined apps, still honoured by legacy boxes/apps. Its sharp |
| 132 | edge is recorded: a stable promote plus the console's default-checked |
| 133 | safety delay once left a hibernating app unwakeable for days — |
| 134 | ceremony belongs on channels you chose, not on defaults. |
| 135 | - **Instance** — one account's running process for an app on a host. |
| 136 | Declared console-side (`carlos instances enable` once per app, then |
| 137 | `instances create --host …`); a box reconciler mints the actual route. |
| 138 | Exec-backed and hibernating by default. The process contract is |
| 139 | `<bin> --socket <path> --db <path>` on a unix socket — there is no |
| 140 | `$PORT` — and every instance serves `GET /healthz` and |
| 141 | `GET /api/version`. Every exec child is handed `CARLOS_HOST`, its |
| 142 | hostname exactly as `carlos instances list` shows it — **read it, never |
| 143 | the socket's file name**, which says nothing about the host on an |
| 144 | isolated box — and `CARLOS_REGION`, the placement code of the box it |
| 145 | runs on (`ie`, `au`, …), so an app that wants to know where it is can |
| 146 | read it rather than guess. Since v0.26.0 the platform also writes a |
| 147 | `CARLOS_INSTANCE_KEY` into the instance's own env before its first |
| 148 | start: a per-instance secret key for sealing the instance's own data. |
| 149 | It is not an API token and opens no platform door. On a box that has |
| 150 | isolated its instances (v0.25.0), each instance runs in a box of its |
| 151 | own: its own directory and namespaces, seeing only its own app's build, |
| 152 | under a memory ceiling set from the staged manifest plus CPU and IO |
| 153 | limits, and |
| 154 | with **no access to loopback or private addresses on the box** except |
| 155 | its DNS resolver — so an app cannot reach a sibling over `127.0.0.1`, |
| 156 | and one that grows past its tier is OOM-killed and restarted rather |
| 157 | than allowed to take the box's other tenants with it. On such a box the |
| 158 | working directory is the instance's data directory, the one its |
| 159 | database is in, so a relative path like `./data/blobs` lands beside |
| 160 | the database. **A static site has no |
| 161 | instance** — the edge serves it off the channel pointer, and giving one |
| 162 | an instance produces a route that can never wake ("Deploying a static |
| 163 | site" below). |
| 164 | - **Config environments (bundles)** — `carlos env` / `carlos secrets` |
| 165 | write the app's **default** bundle; `--environment <name>` writes a |
| 166 | named one, layered on top of the default when config is materialized. |
| 167 | A route is *bound* to a bundle, and on a provisioned route that binding |
| 168 | lives in the instance record: `carlos instances create --environment` |
| 169 | at birth, `carlos instances set-environment` afterwards. (The box-local |
| 170 | `carlos route --environment` is reverted by the reconciler within |
| 171 | seconds and refuses such a row by name.) `carlos env environments` |
| 172 | lists an app's bundles, and `carlos restart --environment <name>` |
| 173 | cycles exactly the routes that read one. |
| 174 | - **Canary** — `carlos canary` stands a second instance beside production |
| 175 | on a channel and hostname belonging to one branch: `canary/<slug>` and |
| 176 | `<slug>-<app>.<sqid>.<domain>` (one label since v0.24.0, so slug and |
| 177 | app together are at most 63 characters), the slug defaulting to the git |
| 178 | branch. Promotion into a canary is never refused and costs no rung — canary |
| 179 | channels sit outside every pipeline, so a build that has ridden one |
| 180 | still enters the pipeline fresh. Seven-day lease, refreshed on every |
| 181 | re-promote. It is a URL, not a traffic split. |
| 182 | - **Service credential** — a bearer token a *server* holds rather than a |
| 183 | person: a CI job that ships, a signup flow that creates an instance per |
| 184 | customer. `carlos services create --role <r> --app <app> <name>` mints |
| 185 | one; reach is fixed at mint and bound to one account and (with `--app`) |
| 186 | one app. Roles, narrowest first: `deploy` (for CI: ships, and promotes |
| 187 | or rolls back only the rungs named at mint — |
| 188 | `services create --role deploy --app <app> --rungs canary,edge ci`, |
| 189 | never `stable`, and only `canary/<name>` channels on an app that |
| 190 | declares a pipeline; no hotfix, no app creation, so a new app's first deploy |
| 191 | still needs a person; CI sets `CARLOS_SERVICE_TOKEN` and |
| 192 | `CARLOS_CONSOLE` and runs `carlos deploy` as usual), `publish` (ship |
| 193 | releases), `instance` (create, delete, repoint and move the app's |
| 194 | instances, attach its domains, write its path routes, read its logs, |
| 195 | errors and analytics), `operate`, and `admin` (which includes minting |
| 196 | store IAM users — a real handover). No role can run |
| 197 | `carlos instances enable` or mint another credential. "Provisioning |
| 198 | instances from your own app" below. |
| 199 | - **Path routes** — one hostname split by its first path segment: |
| 200 | `example.com/acme` served by one route, `example.com/globex` by another, |
| 201 | everything else by the hostname's own route. A prefix *points at* an |
| 202 | existing route and borrows its channel, config, hibernation and billing; |
| 203 | it defines nothing of its own. "One hostname, many instances" below. |
| 204 | - **`.carlos/config`** — two layers, global `~/.carlos/config` and |
| 205 | per-project `./.carlos/config` (committed; nearest wins walking up). |
| 206 | Holds console, account, app, kind, artifact — the reason zero-argument |
| 207 | `carlos deploy` works — and optionally named `destinations`, one per |
| 208 | console or app a checkout deploys to, picked with `-d <name>` on any |
| 209 | verb that takes `--console` (`carlos config destinations` edits them). Project values are honored only when the file's |
| 210 | console matches the session's, so a committed config cannot silently |
| 211 | redirect someone else's credentials. |
| 212 | - **Default addresses** — `<app>.<sqid>.<apps-domain>` (on the hosted |
| 213 | platform, `oncarlos.com`); canary form `<canary>-<app>.<sqid>.<domain>`. |
| 214 | Minted **alias** hosts intentionally carry no `X-Carlos-Version` — |
| 215 | verify on the canonical host. |
| 216 | |
| 217 | ## Describe changes in Activity |
| 218 | |
| 219 | Agents pass `--message` explicitly for `ship`, `deploy`, `promote`, |
| 220 | `rollback`, `restart`, `instances create`, `instances delete`, `features set`, |
| 221 | `env set`, `env unset`, and `env exec-delivery`. |
| 222 | Use a concise sentence describing the intended change or why it is needed: |
| 223 | `carlos deploy --message "Fix invitation links after an email address changes"`. |
| 224 | The console records the exact action, version, channel, or instance separately; |
| 225 | repeating only a commit hash or "deploy app" adds no useful context. Do not |
| 226 | invent an outcome you have not verified, or include secrets or config values. |
| 227 | |
| 228 | Messages are optional, one line, at most 240 Unicode characters. They are |
| 229 | separate from release labels and release notes. Interactive CLI commands ask |
| 230 | for a description by default; an empty answer skips it. Noninteractive commands |
| 231 | do not prompt. `--no-prompt` suppresses the question for a command, and |
| 232 | `"message_prompt": "off"` in `.carlos/config` disables prompts by default. |
| 233 | Agents should still supply `--message` when |
| 234 | prompts are disabled. Explicit `--message` still applies with `--no-prompt` or |
| 235 | config suppression. `deploy` asks once and uses the same description for its |
| 236 | ship and promote steps. |
| 237 | |
| 238 | ## The member CLI |
| 239 | |
| 240 | | Verb | What it does | |
| 241 | |---|---| |
| 242 | | `carlos auth login\|whoami\|logout\|default` | Device-code login (approve in any signed-in browser); identity + memberships; per-project default console | |
| 243 | | `carlos apps create\|list\|place\|set\|build-probe\|delete\|restore` | Claim an app (a name whose platform address is already held is refused, and `tmp-` names are reserved); `list` every app, the Trash included; place it on a customer fleet; `set --swap overlap\|hold` and `--swap-concurrency` choose how its instances take a new build ("Two builds at once"); `build-probe /api/version` makes `deploy` ask the process which build it runs rather than trust the header — worth setting on anything you deploy often; trash/restore | |
| 244 | | `carlos deploy` | **Release new code — reach for this one.** ship + promote + watch until the new build serves; zero-arg with a saved project config, `-d <name>` for a named destination. Follows the rollout box by box and names a box where the build failed. `--channel` to land somewhere other than the entry channel (a canary); `--swap hold` for a release that cannot run beside its predecessor. Also the WHOLE story for a static site: `--kind static --host <h> <dir>` declares the route as well as shipping it ("Deploying a static site") | |
| 245 | | `carlos ship` | The ship half alone: publish a release *without* releasing it (`--kind binary\|static`, `--version`, `--notes`); refuses a Go binary whose main package differs from the app's last release unless you pass `--program-change` (deploy has no such flag: ship, then promote); rate-limited per app (~2/minute — a 429 carries `Retry-After`) | |
| 246 | | `carlos promote` | The promote half alone: move an **already-shipped** version onto a channel — a ladder step or a re-point, not how you release new code. The channel is positional, and optional when the app has only one. `--retry` re-tries a build that failed on some box, once you have fixed what it depended on. (`--hotfix` is refused by the console: an operator's direct-bucket act) | |
| 247 | | `carlos canary` | Deploy onto a branch-only channel + hostname, waiting on the canary URL's own `X-Carlos-Version`; `ls` / `rm`. **Reach for `--environment`** if the app carries config — see "Standing a canary beside production" | |
| 248 | | `carlos rollback` | Point a channel back at an earlier version | |
| 249 | | `carlos pipeline` | Show or shape the app's release channels; `init --template edge-production\|full-ladder` replaces the single default channel with a starter pipeline | |
| 250 | | `carlos channels` / `carlos releases` | What each channel serves / every shipped version; `releases retention` prunes old ones | |
| 251 | | `carlos version target` | The semver family ships auto-increment under | |
| 252 | | `carlos env` / `carlos secrets` | Plain vars / sealed secrets, layered per environment; `env sync` forces convergence; `secrets genkey` mints keypairs locally | |
| 253 | | `carlos instances enable\|create\|list\|delete\|set-upstreams\|set-channel\|set-environment\|set\|update\|retry\|door-key` | Opt an app in (`enable --private-storage` gives every later instance its own environment and bucket); declare/inspect/remove instances; reorder or remove upstreams (**adding** an upstream address is operator-only since v0.25.0); move a route onto another channel or config bundle. `list` shows INSTALLED and SERVING builds and a REASON when they differ; `retry <host>` clears a failed build's hold on one instance; `set <host> --updates manual\|--pin <build>` and `update <host>` steer how one instance takes builds; `create --role replica --region <r>` makes a replica near buyers, holding a one-way copy of the live instance's data (the live instance names allowed regions with `--replicas` at its own create); `door-key` manages keys for the edge-local door, which a box operator must turn on. `--health running\|asleep\|not-responding` narrows a listing, which is usually what you want when something is wrong. For **binaries only** — a static site needs none of these, and `carlos deploy` declares what it does need | |
| 254 | | `carlos restart` | Cycle an app's processes — no version or config change. `--host` narrows to one route, `--environment` to every route reading that bundle (the request that follows a `secrets set`), `--dry-run` prints the set. Sleeping instances are left asleep | |
| 255 | | `carlos routes` | The app's routes through the console: upstream, channel, config environment, replication rung. It reads the console's own box, and says so on a multi-box fleet | |
| 256 | | `carlos steering` | `latency` opts a host into a per-edge latency DNS record; `off` returns it to one static answer. Shared-pool instances only in v1 | |
| 257 | | `carlos features set\|list` | The app's own feature flags — the platform stores and serves them, only the app interprets them. Lands on the instance's next `deployment`-block poll: no restart, no env edit, no box file. `key=` clears | |
| 258 | | `carlos gate set\|list\|clear` | One shared password in front of the app's hostnames, custom domains included. `set` reads stdin when `--password` is absent; changing or clearing signs every visitor out | |
| 259 | | `carlos errors` | One row per distinct error, newest first — `error`-tagged log lines plus the 500s the app itself answered, no wiring at all. `--fp <fingerprint>` expands a group; `--set-level warn` widens what counts | |
| 260 | | `carlos availability` | Every 502, 503 and 504 the edge saw (the instance could not answer), grouped by cause, plus failed wakes and exits; one outage is one row. `--fp` expands a cause to each request. The Availability tab beside Errors | |
| 261 | | `carlos analytics` | Page views, daily visitors, bytes, top pages/referers/events, counted at the edge. `--set-count edge` turns it on; `--range today\|7d\|30d` | |
| 262 | | `carlos logs` | Merged app + platform + edge timeline (`-f` follows, `--grep`, `--since`) — no box access | |
| 263 | | `carlos db pull\|query` | A read-only copy of an instance's database, restored from its replica on your machine; the live one is never touched. `--at <RFC 3339 time>` reads it as it was then (sleeping instances keep seven days of history). `--instance` when the app has several | |
| 264 | | `carlos run --app <a> --instance <host> -- <command>` | Run one of your app's own commands as that instance, with its environment and database, output streamed as it arrives (v0.28.0). `--file` uploads files with the run (64 MB in total), `--timeout` (30 minutes by default, up to 6 hours), `--follow <id>` reattaches. A sleeping instance is woken; a live one keeps serving beside it; one run at a time per instance, refused while it moves or if its box is not yet on the nested layout. The way to run a one-off task against production without a box | |
| 265 | | `carlos geo <address>` | What the `X-Carlos-Geo-*` headers would say for any address, with the credit the database requires | |
| 266 | | `carlos sidecars enable\|create\|ls\|wake\|logs\|rm` | A process the platform runs for the app with no socket, hostname or DNS: a bot holding a connection out, a worker. Woken only by its schedule or `wake` | |
| 267 | | `carlos config destinations` | Name the places one checkout deploys to in `.carlos/config`; `-d <name>` picks one | |
| 268 | | `carlos domains attach\|detach\|verify\|list\|paths` | `attach --app <a> '*.<parent>'` (quoted) gives every instance a name `<label>.<parent>` with no per-instance claim; `<parent>` must be attached first and a `*` DNS record must point at the deployment. `verify` finds the TXT record a deployment that requires domain proof asks for. `paths <host>` prints the hostname's published path-route map (written through the API — "One hostname, many instances"). Claim customer hostnames (`--wildcard`, `--catchall`); prints the DNS records to create; certs follow automatically. Since 2026-08-27 the platform orders and renews those certs itself, and `list` joins the fleet's readings sweep, so DNS state, certificate expiry and delegation are readable from the terminal — a certificate the platform has given up on now says so instead of sitting pending forever | |
| 269 | | `carlos store apply\|create\|status\|rotate\|scan` | Object storage. **Once per account, first:** an owner runs `store apply --account <a> --note "<why>"` and a deployment operator approves it — until then `create` has nothing to provision into, and `status` says the account is unapproved or pending. After approval `create` is the whole member workflow (unlimited stores, usage metered); `--environment <e>` gives one config bundle its own bucket and credential; credentials arrive as env; `rotate` is two-phase (`--finish` retires the old key). `scan` is opt-in virus scanning over the bucket's objects (`request`/`status` are yours, `grant`/`revoke`/`sweep`/`enforce` a deployment operator's) — and a scanner cannot read ciphertext, so a client-encrypted store is marked not applicable rather than given a coverage claim | |
| 270 | | `carlos email enable\|status\|test\|domains\|events\|credentials\|rotate` | Declare sending; provision a verified domain (`domains rm` frees one); SMTP credentials arrive as env; `events` shows what became of each message for 30 days, `events push --path` posts them to the app, signed (`pause`/`resume` are a deployment operator's) | |
| 271 | | `carlos relay enable\|status\|disable` | The platform's own STUN/TURN relay for an app's WebRTC calls. `enable` delivers `CARLOS_RELAY` (base64 JSON: per-region ICE URLs, a key, a username suffix) and the app mints a TURN credential per call from the key; `status` shows issue, pickup and per-region health. `enable` refuses, with a sentence, on a deployment that runs no relay. Never run coturn or buy a TURN service | |
| 272 | | `carlos services create\|ls\|rotate\|revoke` | Server-held credentials (`--role deploy\|publish\|instance\|operate\|service\|admin`, `--app` to bind one app, `--rungs` for `deploy`); the secret prints once; `rotate` refuses the old secret on its next call, `revoke` stops the credential outright | |
| 273 | | `carlos schedule ls\|set\|rm\|run` | Declare recurring work per app (`--every 6h` or `--cron "0 8 * * *"` → a `POST` to a path your app serves); `ls` shows next/last per instance; `run` fires one now | |
| 274 | | `carlos ledger append\|publish\|verify` | Open hash-chained per-app ledgers (the transparency machinery) | |
| 275 | | `carlos accounts create\|list\|migrate` | Mint/list accounts; move an app between them. An account is *your* tenancy on the platform — never create one per customer of your app (customers are instances: "Provisioning instances from your own app") | |
| 276 | | `carlos skills` | List the deployment's published agent skills (`/.well-known/agent-skills/index.json`) and fetch one, digest-verified, to stdout | |
| 277 | | `carlos fleets create\|add-box\|rotate-token\|…` | Bring-your-own-boxes fleets that dial the console | |
| 278 | | `carlos update` | Update the CLI binary itself (signature-verified; defers to brew/apt) | |
| 279 | | `carlos vet` | Check a release against the platform contract | |
| 280 | |
| 281 | Box-side verbs exist (`edge`, `agent`, `adopt`, `route`, `add`, `ops`, |
| 282 | `bootstrap`) but they are the *operator's* surface for running a platform |
| 283 | deployment — a member never types them, and an agent reaching for them on |
| 284 | a member task has taken a wrong turn. |
| 285 | |
| 286 | ## Provisioning instances from your own app |
| 287 | |
| 288 | A product where every customer gets their own instance creates those |
| 289 | instances itself, at signup, through the same console API the CLI uses. |
| 290 | Nothing here needs a box, and nothing here is a workaround: |
| 291 | |
| 292 | 1. **A person enables the app once** — `carlos instances enable --app |
| 293 | <tenant-app>`. This writes the manifest that bounds which domain, pool |
| 294 | and regions any instance may be created in, and no credential can do |
| 295 | it. Add `--private-storage` and every instance created afterwards |
| 296 | gets a config environment named after its host's first label with an |
| 297 | object store bound to it, provisioned by the ordinary create call (the |
| 298 | account needs an approved `store apply` first, and nothing is torn |
| 299 | down when an instance is deleted). |
| 300 | 2. **Mint the credential** — `carlos services create --role instance --app |
| 301 | <tenant-app> signup`. Store it with `carlos secrets set` on the app |
| 302 | that runs signup, **never on the tenant app itself**: every tenant |
| 303 | instance would then hold a token able to delete its siblings. |
| 304 | 3. **Create on signup** — `POST /api/cli/apps/<account>/<tenant-app>/instances` |
| 305 | with `Authorization: Bearer <credential>` and a JSON body; `host` is |
| 306 | required (one DNS label plus the manifest's domain), `region`, |
| 307 | `channel`, `environment` and `message` (the Activity description) are |
| 308 | optional. The console writes the |
| 309 | record and announces it, and a box mints the route within seconds. |
| 310 | 4. **Wait for the real thing** before sending the person there: poll |
| 311 | `https://<host>/api/version` for a 200 (a sub-500 answer is not |
| 312 | evidence), behind a zero-JS "getting your space ready" page that |
| 313 | refreshes itself. |
| 314 | |
| 315 | Give every label a random suffix (`<slug>-<6 random base32 chars>`, the |
| 316 | label being the host's first DNS segment). That makes a collision with |
| 317 | somebody else's instance a first-call event, so the one retry below can |
| 318 | read its 409 without guessing. |
| 319 | |
| 320 | The answers to handle, from the apps that run this: |
| 321 | |
| 322 | | Status | Meaning | Do | |
| 323 | |---|---|---| |
| 324 | | `201` | Created | Wait for `/api/version`, then hand off | |
| 325 | | `409` | That host already exists (or a sibling app owns the label) | Never treat it as yours: pick a new label. On a *retry of your own timed-out call*, a 409 means the first call worked | |
| 326 | | `429` | The credential's own rate limit (it is separate from any person's, so a signup burst never starves your promotes) | Show a sentence and a retry; never loop | |
| 327 | | `502` / timeout | The console could not complete the write | Retry once with the **same** host | |
| 328 | | `400` | Bad body, or a host that is not one label plus the manifest's domain | A bug on your side; fail loudly | |
| 329 | | `401` / `404` | Bad credential, or the wrong account or app (every miss is the same 404) | A configuration bug; fail loudly | |
| 330 | |
| 331 | Also on the same credential: `GET .../instances` (list), `GET`, `PATCH` |
| 332 | (reorder or remove addrs, set the probe — adding an upstream address is |
| 333 | operator-only since v0.25.0) and `DELETE .../instances/<host>`, and |
| 334 | `POST/PUT .../domains` for customer hostnames. **Deleting an instance |
| 335 | removes its record, never its data** — the parked database and blobs are |
| 336 | left in place on purpose, and the call is idempotent. |
| 337 | |
| 338 | **The known gap:** the `instance` role is app-wide. One credential can |
| 339 | create, delete or move *every* instance of the app, not only the ones it |
| 340 | made. That is why it lives on the signup app alone, and why a signup app |
| 341 | compromise must be named in your published trade-offs. A narrower, |
| 342 | create-only credential is a filed platform request — do not build your own |
| 343 | token broker around it. |
| 344 | |
| 345 | ## One hostname, many instances |
| 346 | |
| 347 | Path routes let one hostname — your product's own domain — serve each |
| 348 | customer from their own instance, so `example.com/acme` and |
| 349 | `example.com/globex` are two processes and two databases behind one |
| 350 | address. The hostname's own route keeps everything else (`/`, sign-in, |
| 351 | marketing). |
| 352 | |
| 353 | - The hostname must be one the app holds: `carlos domains attach --app |
| 354 | <hub-app> example.com` first. |
| 355 | - The map is written **whole** through the API, by a person or by the |
| 356 | hub app's `instance`/`publish` credential: |
| 357 | `PUT /api/cli/apps/<account>/<hub-app>/domains/example.com/paths` with |
| 358 | `{"paths": {"acme": "acme.<tenant-domain>", "globex": "globex.<tenant-domain>"}}`. |
| 359 | Keys are one bare path segment (no slash); values are hostnames of |
| 360 | existing routes. Publish the full map from your own records every time — |
| 361 | a lost delta desyncs silently, a whole republish repairs. `{"paths": {}}` |
| 362 | clears it; a body without the `paths` wrapper is refused, so a typo can |
| 363 | never wipe the map. |
| 364 | - **Targets must belong to the same account** (they may be another app of |
| 365 | that account — a hub app pointing at tenant-app instances is the |
| 366 | intended shape). |
| 367 | - **Targets must live on the same box as the hostname's own route.** A |
| 368 | target homed elsewhere is silently ignored and its prefix falls through |
| 369 | to the hostname's route. That caps a path-routed product at one box |
| 370 | today; cross-box path targets are a filed platform request. |
| 371 | - A write goes live in about five seconds. `carlos domains paths --app |
| 372 | <hub-app> example.com` prints what was published; `carlos routes` |
| 373 | shows what the box actually applied. |
| 374 | - The upstream receives the original path (`/acme/...`) unchanged, and the |
| 375 | target route is what is billed, versioned and woken. |
| 376 | - **One hostname is one browser origin.** Cookies, storage and script |
| 377 | are shared across every prefix, so every tenant's pages can act on the |
| 378 | hub's session. That is fine while every tenant runs your app's code; if |
| 379 | tenants serve HTML their own customers wrote, give each tenant its own |
| 380 | hostname instead. |
| 381 | |
| 382 | ## Deploying a static site |
| 383 | |
| 384 | A static site is one command, and the command is the same one binaries use: |
| 385 | |
| 386 | ``` |
| 387 | carlos apps create --app mysite |
| 388 | carlos deploy --app mysite --kind static --host mysite.<sqid>.oncarlos.com ./site |
| 389 | ``` |
| 390 | |
| 391 | That ships the directory as an immutable release, promotes it to the app's |
| 392 | entry channel, declares the route, and watches the URL until |
| 393 | `X-Carlos-Version` reports the build. Nothing else is needed — not |
| 394 | `carlos instances enable`, not `carlos instances create`, not a separate |
| 395 | `carlos domains attach`. |
| 396 | |
| 397 | `--host` is required on the **first** static deploy and only that one. A |
| 398 | static app has no instance record yet, so there is nothing to resolve the |
| 399 | host from; that deploy writes the record, and afterwards plain `carlos |
| 400 | deploy` resolves host and channel by itself. Give it a custom domain and |
| 401 | the deploy attaches that too, printing the DNS records to publish. |
| 402 | |
| 403 | Expect the URL to take up to a minute after `promoted` prints. A static |
| 404 | route trusts its resolved channel pointer for about 60 seconds, so the old |
| 405 | page can outlive the promote by that long — `carlos deploy` already waits |
| 406 | past one such window, and says so while it waits. |
| 407 | |
| 408 | ### A static site has no instance, and must not be given one |
| 409 | |
| 410 | **This is the trap.** The verbs read like the binary recipe — enable an |
| 411 | app for instances, create one, point it at a channel — and every step |
| 412 | succeeds. The result cannot work. |
| 413 | |
| 414 | A static site is served by the edge straight off the channel pointer: |
| 415 | there is no process, nothing hibernates, nothing wakes. An **instance** is |
| 416 | a process the platform execs. A static release's manifest carries a site |
| 417 | tarball and no build for any box's architecture, so an instance route on |
| 418 | that channel finds nothing to run and fails every wake, permanently. |
| 419 | |
| 420 | Older platform builds reported that failure as *"nothing is promoted for |
| 421 | this instance's channel"* — while `carlos channels` showed the promote |
| 422 | sitting right there. Both statements were true and neither named the |
| 423 | problem, which is the shape of the route, not the state of the channel. |
| 424 | Current builds name it and refuse the create; if you meet the old message, |
| 425 | this is what it means. |
| 426 | |
| 427 | The way out, on an app already in that state: |
| 428 | |
| 429 | ``` |
| 430 | carlos instances delete --app mysite --host mysite.<sqid>.oncarlos.com |
| 431 | carlos instances create --app mysite --host mysite.<sqid>.oncarlos.com --kind static --channel <ch> |
| 432 | ``` |
| 433 | |
| 434 | The already-shipped release needs no re-ship — a static route reads the |
| 435 | channel pointer, so the site serves within seconds of the record landing. |
| 436 | |
| 437 | ### Channels belong to the app, not to the platform |
| 438 | |
| 439 | A new app is born with **one** channel, `edge` by default, and that one |
| 440 | channel is its production. More are opt-in, declared by a pipeline, and |
| 441 | named whatever the app calls them. |
| 442 | |
| 443 | So do not guess a channel name from the `canary → edge → beta → stable` |
| 444 | ladder in `--help`: that is the frozen legacy ladder an app inherits only |
| 445 | when it declares no pipeline. `carlos channels` lists what this app |
| 446 | actually has and `carlos pipeline` shows the order. `carlos promote |
| 447 | <version>` with no channel resolves it when the app has one, and names the |
| 448 | app's real channels when it has several. |
| 449 | |
| 450 | ## Standing a canary beside production |
| 451 | |
| 452 | Review happens on a deployed canary, never on localhost, and one command |
| 453 | does it: |
| 454 | |
| 455 | ``` |
| 456 | carlos canary --app jam ./jam |
| 457 | ``` |
| 458 | |
| 459 | The name defaults to the slugified git branch, so branch `new-queue` |
| 460 | gives channel `canary/new-queue` and host |
| 461 | `new-queue-jam.<sqid>.oncarlos.com`. An explicit `--name` is validated |
| 462 | rather than quietly repaired, because the same string has to serve as |
| 463 | both a channel segment and part of a hostname label: lowercase letters, |
| 464 | digits, dashes, and at most 63 characters with the app name. `carlos canary ls` shows each canary's host, what it serves, |
| 465 | which config bundle it is bound to and what is left of its seven-day |
| 466 | lease; `rm` deletes the instance record and the host stops being served. |
| 467 | The channel lapses on its own. |
| 468 | |
| 469 | **If the app carries config, pass `--environment`.** This is the trap, |
| 470 | and it is worth stating in full because the failure looks like an app |
| 471 | bug. Config in CARLOS is per-app, so a canary with no bundle of its own |
| 472 | serves the app's *default* bundle — where every origin-shaped value names |
| 473 | the app's **main** host. Anything keyed to that single scalar refuses |
| 474 | requests on the canary hostname: the `Origin` header the app compares |
| 475 | against, magic-link and callback URLs, a WebAuthn RP ID, the cookie |
| 476 | `Secure` flag. The result is a canary that serves every GET perfectly and |
| 477 | fails every form submission. Write the bundle first, then bind it: |
| 478 | |
| 479 | ``` |
| 480 | carlos env --app jam --environment jam-canary JAM_ORIGIN=https://new-queue-jam.<sqid>.oncarlos.com |
| 481 | carlos canary --app jam --environment jam-canary ./jam |
| 482 | ``` |
| 483 | |
| 484 | The same applies to any second instance standing beside a live one, not |
| 485 | just a canary — `carlos instances create --environment`, or |
| 486 | `carlos instances set-environment` on a route that already exists, |
| 487 | followed by a restart to pick the values up. |
| 488 | |
| 489 | ## Watching an app: errors, availability and analytics |
| 490 | |
| 491 | All three read from the terminal or the app's dashboard tabs. **Errors |
| 492 | and availability collect from the day the app ships** — nothing to turn |
| 493 | on. **Analytics is off |
| 494 | until you switch it on** (a deliberate ruling, 2026-09-07: counting |
| 495 | visitors is the app's call, not the platform's). |
| 496 | |
| 497 | ``` |
| 498 | carlos errors --app hello --since 24h |
| 499 | carlos availability --app hello --since 24h |
| 500 | carlos analytics --app hello --set-count edge |
| 501 | carlos analytics --app hello --range 7d |
| 502 | ``` |
| 503 | |
| 504 | `carlos errors` needs no wiring at all: it groups `error`-tagged log |
| 505 | lines (or `warn` too, with `--set-level warn`) together with the 500s |
| 506 | the app itself answered, one row per distinct error with a count and a |
| 507 | location. `--fp <fingerprint>` expands one group to its occurrences. |
| 508 | |
| 509 | `carlos availability` (since v0.22.0) holds what used to sit in errors: |
| 510 | every 502, 503 and 504 the edge saw, meaning the instance could not |
| 511 | answer, grouped by cause (refused, closed mid-request, timed out, no box |
| 512 | connected, or the app's own 503), with failed wakes and exits beside |
| 513 | them. One outage is one row however many pages it hit; `--fp` expands a |
| 514 | cause to each request and path. A visitor who gives up after five |
| 515 | seconds or more without response headers is filed as an outage, so an |
| 516 | endpoint that holds its headers by design, like a long poll, shows a |
| 517 | standing row there, and that row is expected. |
| 518 | |
| 519 | `carlos analytics` counts at the edge as it serves the app: page views, |
| 520 | daily visitors, bytes served, top pages, top referers, named events — |
| 521 | **no JavaScript, no cookie, no stored IP**, which is why an app on CARLOS |
| 522 | has no business reaching for a third-party analytics script. Visitors are |
| 523 | a sketch, so a multi-day total is daily visitors summed rather than a |
| 524 | deduplicated unique count. The switch is per-app or account-wide. |
| 525 | Where the deployment publishes a GeoIP database, page views are counted |
| 526 | by country too. |
| 527 | |
| 528 | Neither replaces `carlos logs`, still the merged app + platform + edge |
| 529 | timeline (`-f`, `--grep`, `--since`, `--stream app`, `--host`) with no |
| 530 | box access. |
| 531 | |
| 532 | ## Sending email |
| 533 | |
| 534 | The platform issues SMTP credentials. An app declares that it sends, the |
| 535 | console provisions an SES identity for a domain, publishes the DNS records |
| 536 | where it controls the zone, and delivers working credentials as config. The |
| 537 | app never holds a cloud key and nobody runs an MTA. |
| 538 | |
| 539 | One command does the whole walk — declare, provision the domain, wait for |
| 540 | verification, deliver: |
| 541 | |
| 542 | ``` |
| 543 | carlos email enable --app <app> |
| 544 | ``` |
| 545 | |
| 546 | The identity defaults to the app's **own host** |
| 547 | (`<app>.<sqid>.oncarlos.com`), so the signing domain matches the domain of |
| 548 | the links inside the mail. Credentials arrive as env — |
| 549 | `CARLOS_SMTP_HOST`, `_PORT`, `_USER`, `_PASS`, `_FROM`. Transport is |
| 550 | submission on 587 with STARTTLS. `--env-prefix MAILER` renames the whole set |
| 551 | to `MAILER_HOST`, `MAILER_PORT`, and so on, for an app that already reads |
| 552 | its own names. |
| 553 | |
| 554 | For a customer's own domain: |
| 555 | |
| 556 | ``` |
| 557 | carlos email enable --app <app> --domain mail.example.com |
| 558 | ``` |
| 559 | |
| 560 | On the platform domain CARLOS publishes the records itself and verification |
| 561 | is usually seconds. On a custom domain it prints the record set for the |
| 562 | zone's owner to publish, then polls until SES confirms it (`--timeout`, |
| 563 | default 10m; a gave-up wait exits non-zero and names what SES is still |
| 564 | waiting for). `carlos email domains add` is the same provisioning step on |
| 565 | its own, for adding a second sending domain to an app that already sends. |
| 566 | |
| 567 | The record set a customer has to publish is always these five, for |
| 568 | `<domain>` and the region the identity was first verified in: |
| 569 | |
| 570 | | Name | Type | Value | |
| 571 | |---|---|---| |
| 572 | | `<token>._domainkey.<domain>` × 3 | CNAME | `<token>.dkim.amazonses.com` | |
| 573 | | `mail-<region>.<domain>` | MX | `10 feedback-smtp.<region>.amazonses.com` | |
| 574 | | `mail-<region>.<domain>` | TXT | `v=spf1 include:amazonses.com ~all` | |
| 575 | | `_dmarc.<domain>` | TXT | `v=DMARC1; p=none; adkim=s; aspf=s` (plus `rua=` if the deployment sets one) | |
| 576 | |
| 577 | The SPF record sits on the MAIL FROM subdomain rather than the identity |
| 578 | domain because that is the domain SPF is checked against. |
| 579 | |
| 580 | **The published DMARC policy is `p=none`**, deliberately: a tenant whose |
| 581 | very first send is rejected by its own DMARC record learns nothing useful |
| 582 | from the failure. There is no `carlos email` verb that changes it. On a |
| 583 | custom domain the zone is the customer's, so they can tighten it themselves |
| 584 | whenever they choose — read the DKIM note below first. |
| 585 | |
| 586 | Then: |
| 587 | |
| 588 | | Verb | What it does | |
| 589 | |---|---| |
| 590 | | `carlos email status --app <app>` | What is declared, which domains verified, what was delivered | |
| 591 | | `carlos email test --app <app> --to me@example.com` | Sends a REAL message through a throwaway credential, then revokes it | |
| 592 | | `carlos email credentials create\|list\|revoke` | Standalone credentials for something not running on CARLOS; the password is shown **once** | |
| 593 | | `carlos email rotate --app <app> [--finish]` | Two-phase key rotation — new key delivered, then the old one retired | |
| 594 | |
| 595 | ### Three things that will bite |
| 596 | |
| 597 | - **A freshly minted credential does not work for the first minute or |
| 598 | two.** SES's SMTP endpoint does not see a new IAM access key |
| 599 | immediately, and the symptom is `535 Authentication Credentials Invalid` |
| 600 | — which reads exactly like a wrong password and is not one. Wait and |
| 601 | retry. An app's first sends straight after `enable` hit this too, so do |
| 602 | not diagnose a broken credential from the first minute of logs. |
| 603 | **A 535 that outlasts a few minutes is a different problem** and is worth |
| 604 | treating as real: check the key is still `Active` and not paused |
| 605 | (`carlos email status`), and that the password was derived for the same |
| 606 | region as the SMTP host — the derivation is region-scoped, so a |
| 607 | credential minted for one region never authenticates against another's |
| 608 | endpoint. |
| 609 | - **DMARC alignment rests on DKIM alone.** Records are published with |
| 610 | `adkim=s; aspf=s`, and that strictness is what stops one tenant's |
| 611 | signature authenticating another tenant's From address on a shared |
| 612 | domain. SPF authenticates but does **not** align: the envelope sender is |
| 613 | a region-qualified subdomain (`mail-<region>.<domain>`), which strict |
| 614 | alignment treats as a different domain. DMARC therefore passes on the |
| 615 | DKIM rail only. That is normal for SES — but it means DKIM is the single |
| 616 | rail, so never turn off DKIM signing on a sending identity. If a customer |
| 617 | tightens their own zone to `p=reject`, that is safe but unforgiving: a |
| 618 | DKIM CNAME later dropped from their zone stops being a downgrade and |
| 619 | becomes total delivery failure. |
| 620 | - **The MAIL FROM subdomain carries its region forever.** It is fixed at |
| 621 | the first verification (`mail-eu-west-1.<domain>`). On a custom domain |
| 622 | that record lives in the customer's zone — the one record CARLOS cannot |
| 623 | republish for them — so moving regions later means going back to the |
| 624 | customer for a DNS change. |
| 625 | |
| 626 | ## Scheduled work |
| 627 | |
| 628 | The platform keeps the clock. A hibernating instance cannot run an |
| 629 | in-process timer, so recurring work is declared from outside the app and |
| 630 | delivered as a **tick**: a `POST` to a path you declared, at the time it |
| 631 | is due, waking the instance first and holding it awake while your handler |
| 632 | runs. Nothing else crosses the boundary — no payload, no job state, no |
| 633 | queue. Live 2026-08-24. |
| 634 | |
| 635 | Declare it with the CLI, not in code: |
| 636 | |
| 637 | ``` |
| 638 | carlos schedule set --app <app> --name sync --every 6h --path /jobs/sync |
| 639 | carlos schedule set --app <app> --name reminders --cron "0 8 * * *" --path /jobs/reminders |
| 640 | ``` |
| 641 | |
| 642 | `--every` is a Go duration, whole minutes, 1m to 30d. `--cron` is five UTC |
| 643 | fields (`min hour dom month dow`, with `*`, `n`, `a-b`, `*/n`, `a,b`) — |
| 644 | no names, no `@daily`, no seconds. Names are |
| 645 | `^[a-z0-9][a-z0-9-]{0,31}$`, at most 20 per app. `carlos schedule ls` |
| 646 | prints the declaration and then what each instance reports (`next`, |
| 647 | `last`, `last_status`); `run` requests an out-of-band fire on top of the |
| 648 | normal rhythm; `rm` removes the declaration. The verbs say **recorded** |
| 649 | and **requested**, never "fired": they write one console object, and each |
| 650 | box acts on it within seconds. |
| 651 | |
| 652 | The app's whole part is a handler on that path: |
| 653 | |
| 654 | ```go |
| 655 | // POST /jobs/sync |
| 656 | func handleSync(w http.ResponseWriter, r *http.Request) { |
| 657 | if !carlos.Tick(r) { // amadan.net/rastrillo/rastrillo/carlos |
| 658 | http.Error(w, "forbidden", http.StatusForbidden) |
| 659 | return |
| 660 | } |
| 661 | if occ, ok := carlos.TickOccurrence(r); ok && alreadyDone(occ) { |
| 662 | w.WriteHeader(http.StatusNoContent) |
| 663 | return |
| 664 | } |
| 665 | if err := syncer.RunOnce(r.Context()); err != nil { |
| 666 | http.Error(w, err.Error(), http.StatusInternalServerError) // 5xx = retry |
| 667 | return |
| 668 | } |
| 669 | w.WriteHeader(http.StatusNoContent) |
| 670 | } |
| 671 | ``` |
| 672 | |
| 673 | `carlos.Tick` is the constant-time compare of the bearer against |
| 674 | `$CARLOS_ADMIN_TOKEN`, which the platform mints into the instance's |
| 675 | environment; use it rather than hand-rolling the check. A tick arrives on |
| 676 | an ordinary public route, so a tick nobody authenticated is an internet |
| 677 | request to that path — and the `X-Carlos-*` headers are not evidence, |
| 678 | anyone can set those. The status is your whole reply: **2xx** done, |
| 679 | **5xx** retry with backoff (five attempts, 1m doubling to 15m, then the |
| 680 | schedule advances), **4xx** don't bother. |
| 681 | |
| 682 | One-shots are the app's own call, over the control socket the agent binds |
| 683 | before your process starts: |
| 684 | |
| 685 | ```go |
| 686 | err := carlos.ScheduleAt(ctx, "remind-"+id, when, "/jobs/remind") // carlos.ScheduleCancel drops it |
| 687 | ``` |
| 688 | |
| 689 | Same tick, same handler, same name namespace as declared schedules (a |
| 690 | collision returns `ErrDeclaredSchedule`). Up to 400 days ahead, 1000 |
| 691 | pending per host. Reach for one only when the work genuinely has a |
| 692 | specific time; a daily schedule whose handler asks its own database |
| 693 | "what falls today?" beats one timer per row. |
| 694 | |
| 695 | ### Things that bite |
| 696 | |
| 697 | - **Do the work inside the request.** The instance is held awake for |
| 698 | exactly as long as the request is open (30-minute ceiling) and the idle |
| 699 | clock starts the moment you return. Reply 202 and finish in a goroutine |
| 700 | and that goroutine gets hibernated mid-job. |
| 701 | - **Dedupe on `X-Carlos-Schedule-At`** — unix seconds, read it with |
| 702 | `carlos.TickOccurrence`. It is the instant the delivery is *for*, so |
| 703 | every retry of one failed occurrence carries the same value, as does a |
| 704 | redelivery after a box crash or a deploy that cut a long job. Never key |
| 705 | on the wall clock: a retry twenty minutes later is the same occurrence. |
| 706 | Keep `Tick` as the guard — a request with no occurrence header means |
| 707 | "no dedupe key, run it", which is what your own "Sync now" button looks |
| 708 | like. |
| 709 | - **A 4xx is never retried.** A path with no handler, or a handler that |
| 710 | refuses a token it cannot see, records `app refused` and the schedule |
| 711 | advances to its next occurrence. That failure is quiet — `next` still |
| 712 | moves, so `ls` reads like a success. Check `last_status`. |
| 713 | - **An instance running since before the platform roll refuses its own |
| 714 | ticks.** The token is minted at spawn, so a process older than the roll |
| 715 | has no `$CARLOS_ADMIN_TOKEN`, `Tick` is always false, and |
| 716 | `carlos.ScheduleAt` returns `ErrUnauthorized`. `carlos restart` clears |
| 717 | it; nothing else does. |
| 718 | - **A schedule tighter than the idle window keeps the instance resident, |
| 719 | and residency is billed.** Since 2026-08-31 a tick no longer stamps |
| 720 | activity, but the sweeper deliberately will not put a host to sleep |
| 721 | when its next tick is due sooner than the idle window (15 minutes by |
| 722 | default) — staying awake is genuinely cheaper than a wake/sleep cycle |
| 723 | per tick, each of which is a full SQLite checkpoint-and-restore. So a |
| 724 | 5-minute schedule against a 15-minute window never sleeps and accrues |
| 725 | instance-hours around the clock; an hourly one sleeps about three |
| 726 | quarters of each hour. `carlos schedule ls` and `set` now say so in one |
| 727 | line rather than leaving it to be found on a bill. The platform warns |
| 728 | and never refuses: a tight schedule is a legitimate thing to want. |
| 729 | - **Unit-backed instances are not delivered to** in this release — `ls` |
| 730 | says `unsupported` against them. Exec-backed instances (the default) |
| 731 | and sidecars work. |
| 732 | - **Once-timers outlive the process.** They live in the box registry, so |
| 733 | a restart loses none, and re-asserting them at boot is safe because a |
| 734 | repeated name replaces rather than adds. Off-platform there is no |
| 735 | control socket at all: `ScheduleAt` returns `ErrNotOnCarlos`, which |
| 736 | boot code should treat as "skip", not as a failure. |
| 737 | |
| 738 | ## Two builds at once |
| 739 | |
| 740 | Since v0.26.0, where the deployment has switched it on (Carloku did on |
| 741 | 2026-10-14; the release default is off, and then a build is taken by |
| 742 | restart and `deploy` watches the URL instead of each box's rollout), a |
| 743 | new build reaching an awake instance starts beside the old one and takes over once it passes |
| 744 | its health check, so no request waits (`--swap overlap`, the default). |
| 745 | For a few seconds both builds run against the same database and the same |
| 746 | scheduled work, and the new build's migrations run while the old one |
| 747 | still serves. That makes two rules part of every release: |
| 748 | |
| 749 | - **Migrations stay additive across two consecutive releases.** Add a |
| 750 | column, table or index in one release; drop the old one in a later |
| 751 | release, once no running build reads it. A rename is an add now and a |
| 752 | drop later. |
| 753 | - **Schedule names stay the same across the two builds** |
| 754 | (`carlos schedule`). |
| 755 | |
| 756 | A release that cannot follow them ships with `--swap hold` on |
| 757 | `carlos deploy` or `carlos ship`: each instance stops before its new |
| 758 | build starts, and requests wait a few seconds for it. |
| 759 | `carlos apps set <app> --swap hold` makes that the app's rule, and |
| 760 | `--swap-concurrency` is how many instances on one box swap at once, |
| 761 | after the first has stayed up for ten seconds. If a build fails on a box, |
| 762 | that box stops rolling it out, its other instances stay on the build they |
| 763 | were serving, and `carlos deploy` names the failure and exits non-zero; |
| 764 | `carlos instances list` shows it under REASON. Fix the cause, then |
| 765 | `carlos promote --retry <version> <channel>` (every box) or |
| 766 | `carlos instances retry <host>` (one instance). |
| 767 | |
| 768 | ## Visitor location headers |
| 769 | |
| 770 | On a deployment that publishes a GeoIP database (since v0.25.1), |
| 771 | every request the edge passes to an app carries the visitor's rough |
| 772 | location: `X-Carlos-Geo-Country` (ISO 3166-1, `IE`), |
| 773 | `X-Carlos-Geo-Subdivision` (ISO 3166-2, only some databases), |
| 774 | `X-Carlos-Geo-Lat` / `-Lon` (two decimals: an area, not a position) and |
| 775 | `X-Carlos-Geo-Source` (which database answered). Any of them can be |
| 776 | missing: treat that as unknown, never as an error. The edge strips any a |
| 777 | visitor sends, so the values are always the platform's. A cached page |
| 778 | that differs by country sends `Vary: X-Carlos-Geo-Country`; never vary on |
| 779 | the coordinates. |
| 780 | |
| 781 | **An app that shows a visitor anything worked out from these headers |
| 782 | must credit the database `X-Carlos-Geo-Source` names** — `DBIP-…`: |
| 783 | "IP Geolocation by DB-IP", linked to https://db-ip.com; `GeoLite2-…`: |
| 784 | "This product includes GeoLite2 data created by MaxMind, available from |
| 785 | https://www.maxmind.com". For an address that is not the current |
| 786 | request (a log line, a webhook), `carlos geo <address> --app <a>`, or |
| 787 | from an app `GET /api/cli/apps/<account>/<app>/geo?ip=<address>`, at |
| 788 | most 60 a minute. That call needs an `instance`-role token, which is |
| 789 | app-wide (it can also create, repoint and delete every sibling |
| 790 | instance), so make it only from a central app that already holds one; |
| 791 | never deliver that token to a tenant app's instances just to look up an |
| 792 | address. Tenant instances use the request headers. |
| 793 | |
| 794 | ## Deploy truths (each paid for at least once) |
| 795 | |
| 796 | - **`carlos deploy` is the release motion**: ship, promote to the entry |
| 797 | channel (or the channel the app's instances follow), then follow each |
| 798 | box's rollout (or, where a box does not report one, watch the URL) |
| 799 | until the shipped build is serving. "Held for bake" on a |
| 800 | channel that declares one is the system working, not failing. |
| 801 | - **A promote is not a deploy** until the process cycles. An awake |
| 802 | instance starts the new build within seconds of its box adopting it |
| 803 | (beside the old one where two builds at once is on), an asleep one |
| 804 | takes it on its next request, and unit-stamped routes are restarted. |
| 805 | A bespoke long-running unit |
| 806 | outside the platform's knowledge stays old until *something* restarts |
| 807 | it — know which of the three your app is. |
| 808 | - **Verify against the thing you changed, with the binary you built.** |
| 809 | A 200 is not proof: an app-shell route returns 200 HTML for |
| 810 | `/api/version` and will happily "verify" any build ever shipped. The |
| 811 | proof is the build the process itself reports: set |
| 812 | `carlos apps build-probe`, or read `X-Carlos-Version` on the canonical |
| 813 | host knowing the edge stamps it before the process has cycled; stale |
| 814 | DNS and stale local binaries have both produced false "verified" |
| 815 | reports. |
| 816 | - **Review on a canary, not on localhost.** `carlos canary` cannot |
| 817 | disturb production: its channel sits outside every pipeline and its |
| 818 | hostname is the branch's own. Bring a config bundle with it if the app |
| 819 | has one. |
| 820 | - **When CI ships for you, verify by ancestry, not equality** — a PR |
| 821 | merging behind yours cancels your run and ships a commit *containing* |
| 822 | yours. |
| 823 | - **Stamp your build version.** For rastrillo apps, |
| 824 | `-ldflags "-X amadan.net/rastrillo/rastrillo.BuildVersion=<sha>"` |
| 825 | — or every release's `/api/version` reports `dev`. |
| 826 | |
| 827 | ## Switched off, or not released |
| 828 | |
| 829 | Released, switched off on Carloku — the CLI has the flag, the console |
| 830 | declines or ignores it, and nothing should be built to stand in for |
| 831 | them: app residency (`--region` on `apps create`, `deploy` and `canary` |
| 832 | since v0.23.0; the console starts the app in its home region and says |
| 833 | so), instance moves (`carlos move`, v0.26.0; it sends nothing), and |
| 834 | temporary static sites (`deploy --temporary` / `--ttl`, `apps ttl|keep`, |
| 835 | v0.25.1). Names starting `tmp-` are reserved whether or not temporary |
| 836 | sites are on. |
| 837 | |
| 838 | Released in v0.28.0 but refused until an operator turns them on: |
| 839 | seeded instances (`instances create --from <host>` starts a new |
| 840 | instance from a copy of another's database; `instances import` brings |
| 841 | in one the platform never ran). Not built: an |
| 842 | instance-bound token in place of the app-wide `instance` role, a |
| 843 | create-only provisioning role, and cross-box path targets. Check |
| 844 | `carlos <verb> -h` against the console and wait. |
| 845 | |
| 846 | ## Self-hosting the platform |
| 847 | |
| 848 | The platform is the same software members deploy onto: a self-hosted |
| 849 | deployment (Tito's path) runs its own console, boxes, and bucket in its |
| 850 | own cloud account, and its members use the identical `carlos` CLI pointed |
| 851 | at that console. Operating it is real work — box provisioning, the |
| 852 | adopt/restart cadence, platform rolls — and it is the one context where |
| 853 | box verbs and cloud access are legitimate. Commit a `.carlos/config` |
| 854 | naming the deployment's console so sessions cannot fall back to the wrong |
| 855 | one; the CLI's account default is not your account (`ops` exists on every |
| 856 | deployment and is empty — a `not found` against it once cost an hour of |
| 857 | confident wrong diagnosis). |
| 858 | |
| 859 | The deployment's own binary is a channel like any app's: |
| 860 | `carlos system update --channel canary|stable` promotes a release, and |
| 861 | every box on that channel fetches it, verifies it against the signed |
| 862 | pointer, installs it, restarts its edge, and rolls itself back if the new |
| 863 | one does not come up and stay up. `carlos system status` shows each box's |
| 864 | running build and last apply outcome; `carlos system rollback` undoes it; |
| 865 | `--spread 10m` staggers a stable roll. The console is an app — update it |
| 866 | with `carlos deploy --app console`. Owner of the ops account only. |
| 867 | |
| 868 | For the underlying machinery — registry, router, replication, hibernation |
| 869 | internals — see blueprint.md, which is the reference for what the |
| 870 | platform does on your behalf and for hand-rolling outside it. |
| 871 | |
amadan