🤖 The CARLOS platform, member-side

What carlosframework/platform does for an app, and the carlos CLI surface a member drives it with. Everything here works with zero infrastructure access — no SSH, no AWS console, no box commands. That is a design law, not a convenience: every command is built for a user who has nothing but the CLI and a browser. If a task seems to need a box, either you are self-hosting and operating the platform itself, or you have found a product gap to file — never a workaround to build.

Snapshot date 2026-10-18, describing what Carloku serves: platform v0.28.0, released 2026-10-17. Verbs are stable, flag details evolve — trust carlos <verb> -h over this file, and remember a CLI can offer a flag the console you are talking to has switched off or does not have yet ("Switched off, or not released" at the end). The CLI ships for macOS/Linux (brew, apt, static binaries) and Windows (client-only zip — no self-replace, carlos update defers to a fresh download).

Flags are written --flag. Single-dash forms still parse and always will, but as of 2026-08-27 the documented face of every verb is the double dash, and the aliases are -a (--app), -c (--console), -e (--environment), -v (--version). Three flags turn up nearly everywhere and are not repeated below: --app, --account (or --account-id when you want a sqid matched as a sqid and nothing else), and --console, which you need only when this terminal is logged in to more than one deployment.

Start with carlos auth whoami

Before concluding you cannot reach a CARLOS deployment, run it:

carlos auth whoami                              # the default console
carlos auth whoami --console https://<console>  # a specific one

It prints the identity, the terminal's label, the expiry, and every account with its sqid — which is also where you find the sqid your app's hostname needs.

This is first because the obvious check is the misleading one. An agent's instinct is to curl the console and read the answer: the console will 302 you to a login page every time, because a browser session is not what the CLI holds. That redirect says nothing about whether you have access, and reading it as "no access" has already cost one session an afternoon and ended in a filed issue. The CLI's credential is the only thing that answers the question.

Same for the deployment's identity: a self-hosted console is not carloku.com, and the console origin and the apps domain are different hostnames (Tito's are carlos.tito.io and platform.tito.io). If a project commits a .carlos/config, that names the console it belongs to; otherwise --console does.

What the platform owns (never hand-roll these)

• Routing and TLS — the edge is the only process on :443; the route table doubles as the ACME allowlist; per-host certs auto-obtained and renewed, including customer domains and (via ACME delegation) wildcards.
• Replication — Litestream on every instance database, run by the platform's host agent; restore drills are the platform's job too. A hibernating route's rung is tunable (stream — the default, about once a second — / batched, five minutes / daily), a loss-window trade an operator makes on a large, rarely-written database.
• Hibernation — provisioned instances doze when idle and wake on request; live and default-on, not a future feature. Cents-per-month idle cost is the platform's economic story.
• Process supervision and restarts — instances are converged from bucket records; carlos restart cycles them in seconds, console-side.
• Config delivery — carlos env / carlos secrets writes converge onto boxes within seconds (a serial bump, 2s poll). No ssh-delivered EnvironmentFiles.
• Deploy verification — carlos deploy follows the rollout box by box where boxes report it (they do on Carloku), and otherwise watches the URL: the app's build probe if it declares one (carlos apps build-probe), else the X-Carlos-Version header, which the edge stamps from the route row the moment adoption lands, whether or not the process has restarted yet.
• Outbound email — the platform mints a sending identity on a domain, publishes DKIM/SPF/DMARC where it controls the zone, and delivers working SMTP credentials as config. Apps never hold an AWS key and nobody runs an MTA. See "Sending email" below.
• Scheduled work — the platform keeps the clock: a declared schedule POSTs a tick to a path in your app at the time it is due, waking a hibernating instance to run it. No in-app scheduler, no cron line on a box. See "Scheduled work" below.
• Errors, availability and analytics — the edge and the log stream feed all three, with no SDK, no JavaScript, no cookie and no stored IP. Errors and availability collect from the day an app ships; analytics is opt-in. See "Watching an app" below.
• Password gate — carlos gate puts one shared password in front of an app's hostnames, enforced at the edge ahead of the cache and the activator, so the app behind it never sees the cookie. A curtain over a hostname, not accounts for the app's own users.
• DNS steering — carlos steering ... latency opts a host into a latency record per armed edge so the nearest one answers. A static site on a platform hostname is steered from creation without asking; a custom domain is not, because its DNS is yours.

Concepts

• Account — the tenancy unit. Public short id (sqid, e.g. three letters) appears in default hostnames; it is not a secret. Roles are owner/member. carlos auth whoami shows yours.
• App — named per account (not globally). Claimed with carlos apps create; deleted apps sit in a 30-day trash.
• Release — immutable, content-addressed, produced by the ship half of carlos deploy (or by carlos ship on its own). Kinds: binary (default) and static. Versions are free-form; git short sha is the house convention.
• Channels and pipelines (release pipelines v2, live 2026-08-22) — a new app is born with one channel, named edge by default (renameable at creation): that is its production, and carlos deploy lands on it. Multi-channel ceremony is opt-in: a console-mediated pipeline declares an ordered channel list (names are app-defined; promotion onto channel N must come from N−1; the first is the entry channel) with per-channel, default-permissive policy — bake duration, passkey step-up, promote_approvals, and change_approvals (which also guards editing/removing the channel and fast-tracking through its bake). canary/<slug> stays a reserved platform namespace outside any pipeline: always allowed, zero bake, per-session dead ends. Apps with no pipeline keep the legacy frozen ladder (edge → beta → stable, holds 0/24h/72h, unconditional passkey on stable — reaching stable cuts a semver tag). --hotfix, the bake bypass, is an operator act on the direct-bucket path only: the console refuses it by name, so a member never has it. Box-side, bake changes ratchet: a shorter window is honoured only after the previously-known window has elapsed once on the box's own clock, so a compromised console session cannot collapse a hold and ship in the same hour.
• Production flag — legacy: superseded by per-channel pipeline rules for pipelined apps, still honoured by legacy boxes/apps. Its sharp edge is recorded: a stable promote plus the console's default-checked safety delay once left a hibernating app unwakeable for days — ceremony belongs on channels you chose, not on defaults.
• Instance — one account's running process for an app on a host. Declared console-side (carlos instances enable once per app, then instances create --host …); a box reconciler mints the actual route. Exec-backed and hibernating by default. The process contract is <bin> --socket <path> --db <path> on a unix socket — there is no $PORT — and every instance serves GET /healthz and GET /api/version. Every exec child is handed CARLOS_HOST, its hostname exactly as carlos instances list shows it — read it, never the socket's file name, which says nothing about the host on an isolated box — and CARLOS_REGION, the placement code of the box it runs on (ie, au, …), so an app that wants to know where it is can read it rather than guess. Since v0.26.0 the platform also writes a CARLOS_INSTANCE_KEY into the instance's own env before its first start: a per-instance secret key for sealing the instance's own data. It is not an API token and opens no platform door. On a box that has isolated its instances (v0.25.0), each instance runs in a box of its own: its own directory and namespaces, seeing only its own app's build, under a memory ceiling set from the staged manifest plus CPU and IO limits, and with no access to loopback or private addresses on the box except its DNS resolver — so an app cannot reach a sibling over 127.0.0.1, and one that grows past its tier is OOM-killed and restarted rather than allowed to take the box's other tenants with it. On such a box the working directory is the instance's data directory, the one its database is in, so a relative path like ./data/blobs lands beside the database. A static site has no instance — the edge serves it off the channel pointer, and giving one an instance produces a route that can never wake ("Deploying a static site" below).
• Config environments (bundles) — carlos env / carlos secrets write the app's default bundle; --environment <name> writes a named one, layered on top of the default when config is materialized. A route is bound to a bundle, and on a provisioned route that binding lives in the instance record: carlos instances create --environment at birth, carlos instances set-environment afterwards. (The box-local carlos route --environment is reverted by the reconciler within seconds and refuses such a row by name.) carlos env environments lists an app's bundles, and carlos restart --environment <name> cycles exactly the routes that read one.
• Canary — carlos canary stands a second instance beside production on a channel and hostname belonging to one branch: canary/<slug> and <slug>-<app>.<sqid>.<domain> (one label since v0.24.0, so slug and app together are at most 63 characters), the slug defaulting to the git branch. Promotion into a canary is never refused and costs no rung — canary channels sit outside every pipeline, so a build that has ridden one still enters the pipeline fresh. Seven-day lease, refreshed on every re-promote. It is a URL, not a traffic split.
• Service credential — a bearer token a server holds rather than a person: a CI job that ships, a signup flow that creates an instance per customer. carlos services create --role <r> --app <app> <name> mints one; reach is fixed at mint and bound to one account and (with --app) one app. Roles, narrowest first: deploy (for CI: ships, and promotes or rolls back only the rungs named at mint — services create --role deploy --app <app> --rungs canary,edge ci, never stable, and only canary/<name> channels on an app that declares a pipeline; no hotfix, no app creation, so a new app's first deploy still needs a person; CI sets CARLOS_SERVICE_TOKEN and CARLOS_CONSOLE and runs carlos deploy as usual), publish (ship releases), instance (create, delete, repoint and move the app's instances, attach its domains, write its path routes, read its logs, errors and analytics), operate, and admin (which includes minting store IAM users — a real handover). No role can run carlos instances enable or mint another credential. "Provisioning instances from your own app" below.
• Path routes — one hostname split by its first path segment: example.com/acme served by one route, example.com/globex by another, everything else by the hostname's own route. A prefix points at an existing route and borrows its channel, config, hibernation and billing; it defines nothing of its own. "One hostname, many instances" below.
• .carlos/config — two layers, global ~/.carlos/config and per-project ./.carlos/config (committed; nearest wins walking up). Holds console, account, app, kind, artifact — the reason zero-argument carlos deploy works — and optionally named destinations, one per console or app a checkout deploys to, picked with -d <name> on any verb that takes --console (carlos config destinations edits them). Project values are honored only when the file's console matches the session's, so a committed config cannot silently redirect someone else's credentials.
• Default addresses — <app>.<sqid>.<apps-domain> (on the hosted platform, oncarlos.com); canary form <canary>-<app>.<sqid>.<domain>. Minted alias hosts intentionally carry no X-Carlos-Version — verify on the canonical host.

Describe changes in Activity

Agents pass --message explicitly for ship, deploy, promote, rollback, restart, instances create, instances delete, features set, env set, env unset, and env exec-delivery. Use a concise sentence describing the intended change or why it is needed: carlos deploy --message "Fix invitation links after an email address changes". The console records the exact action, version, channel, or instance separately; repeating only a commit hash or "deploy app" adds no useful context. Do not invent an outcome you have not verified, or include secrets or config values.

Messages are optional, one line, at most 240 Unicode characters. They are separate from release labels and release notes. Interactive CLI commands ask for a description by default; an empty answer skips it. Noninteractive commands do not prompt. --no-prompt suppresses the question for a command, and "message_prompt": "off" in .carlos/config disables prompts by default. Agents should still supply --message when prompts are disabled. Explicit --message still applies with --no-prompt or config suppression. deploy asks once and uses the same description for its ship and promote steps.

The member CLI

| Verb | What it does | |---|---| | carlos auth login\|whoami\|logout\|default | Device-code login (approve in any signed-in browser); identity + memberships; per-project default console | | carlos apps create\|list\|place\|set\|build-probe\|delete\|restore | Claim an app (a name whose platform address is already held is refused, and tmp- names are reserved); list every app, the Trash included; place it on a customer fleet; set --swap overlap\|hold and --swap-concurrency choose how its instances take a new build ("Two builds at once"); build-probe /api/version makes deploy ask the process which build it runs rather than trust the header — worth setting on anything you deploy often; trash/restore | | carlos deploy | Release new code — reach for this one. ship + promote + watch until the new build serves; zero-arg with a saved project config, -d <name> for a named destination. Follows the rollout box by box and names a box where the build failed. --channel to land somewhere other than the entry channel (a canary); --swap hold for a release that cannot run beside its predecessor. Also the WHOLE story for a static site: --kind static --host <h> <dir> declares the route as well as shipping it ("Deploying a static site") | | carlos ship | The ship half alone: publish a release without releasing it (--kind binary\|static, --version, --notes); refuses a Go binary whose main package differs from the app's last release unless you pass --program-change (deploy has no such flag: ship, then promote); rate-limited per app (~2/minute — a 429 carries Retry-After) | | carlos promote | The promote half alone: move an already-shipped version onto a channel — a ladder step or a re-point, not how you release new code. The channel is positional, and optional when the app has only one. --retry re-tries a build that failed on some box, once you have fixed what it depended on. (--hotfix is refused by the console: an operator's direct-bucket act) | | carlos canary | Deploy onto a branch-only channel + hostname, waiting on the canary URL's own X-Carlos-Version; ls / rm. Reach for --environment if the app carries config — see "Standing a canary beside production" | | carlos rollback | Point a channel back at an earlier version | | carlos pipeline | Show or shape the app's release channels; init --template edge-production\|full-ladder replaces the single default channel with a starter pipeline | | carlos channels / carlos releases | What each channel serves / every shipped version; releases retention prunes old ones | | carlos version target | The semver family ships auto-increment under | | carlos env / carlos secrets | Plain vars / sealed secrets, layered per environment; env sync forces convergence; secrets genkey mints keypairs locally | | carlos instances enable\|create\|list\|delete\|set-upstreams\|set-channel\|set-environment\|set\|update\|retry\|door-key | Opt an app in (enable --private-storage gives every later instance its own environment and bucket); declare/inspect/remove instances; reorder or remove upstreams (adding an upstream address is operator-only since v0.25.0); move a route onto another channel or config bundle. list shows INSTALLED and SERVING builds and a REASON when they differ; retry <host> clears a failed build's hold on one instance; set <host> --updates manual\|--pin <build> and update <host> steer how one instance takes builds; create --role replica --region <r> makes a replica near buyers, holding a one-way copy of the live instance's data (the live instance names allowed regions with --replicas at its own create); door-key manages keys for the edge-local door, which a box operator must turn on. --health running\|asleep\|not-responding narrows a listing, which is usually what you want when something is wrong. For binaries only — a static site needs none of these, and carlos deploy declares what it does need | | carlos restart | Cycle an app's processes — no version or config change. --host narrows to one route, --environment to every route reading that bundle (the request that follows a secrets set), --dry-run prints the set. Sleeping instances are left asleep | | carlos routes | The app's routes through the console: upstream, channel, config environment, replication rung. It reads the console's own box, and says so on a multi-box fleet | | carlos steering | latency opts a host into a per-edge latency DNS record; off returns it to one static answer. Shared-pool instances only in v1 | | carlos features set\|list | The app's own feature flags — the platform stores and serves them, only the app interprets them. Lands on the instance's next deployment-block poll: no restart, no env edit, no box file. key= clears | | carlos gate set\|list\|clear | One shared password in front of the app's hostnames, custom domains included. set reads stdin when --password is absent; changing or clearing signs every visitor out | | carlos errors | One row per distinct error, newest first — error-tagged log lines plus the 500s the app itself answered, no wiring at all. --fp <fingerprint> expands a group; --set-level warn widens what counts | | carlos availability | Every 502, 503 and 504 the edge saw (the instance could not answer), grouped by cause, plus failed wakes and exits; one outage is one row. --fp expands a cause to each request. The Availability tab beside Errors | | carlos analytics | Page views, daily visitors, bytes, top pages/referers/events, counted at the edge. --set-count edge turns it on; --range today\|7d\|30d | | carlos logs | Merged app + platform + edge timeline (-f follows, --grep, --since) — no box access | | carlos db pull\|query | A read-only copy of an instance's database, restored from its replica on your machine; the live one is never touched. --at <RFC 3339 time> reads it as it was then (sleeping instances keep seven days of history). --instance when the app has several | | carlos run --app <a> --instance <host> -- <command> | Run one of your app's own commands as that instance, with its environment and database, output streamed as it arrives (v0.28.0). --file uploads files with the run (64 MB in total), --timeout (30 minutes by default, up to 6 hours), --follow <id> reattaches. A sleeping instance is woken; a live one keeps serving beside it; one run at a time per instance, refused while it moves or if its box is not yet on the nested layout. The way to run a one-off task against production without a box | | carlos geo <address> | What the X-Carlos-Geo-* headers would say for any address, with the credit the database requires | | carlos sidecars enable\|create\|ls\|wake\|logs\|rm | A process the platform runs for the app with no socket, hostname or DNS: a bot holding a connection out, a worker. Woken only by its schedule or wake | | carlos config destinations | Name the places one checkout deploys to in .carlos/config; -d <name> picks one | | carlos domains attach\|detach\|verify\|list\|paths | attach --app <a> '*.<parent>' (quoted) gives every instance a name <label>.<parent> with no per-instance claim; <parent> must be attached first and a * DNS record must point at the deployment. verify finds the TXT record a deployment that requires domain proof asks for. paths <host> prints the hostname's published path-route map (written through the API — "One hostname, many instances"). Claim customer hostnames (--wildcard, --catchall); prints the DNS records to create; certs follow automatically. Since 2026-08-27 the platform orders and renews those certs itself, and list joins the fleet's readings sweep, so DNS state, certificate expiry and delegation are readable from the terminal — a certificate the platform has given up on now says so instead of sitting pending forever | | carlos store apply\|create\|status\|rotate\|scan | Object storage. Once per account, first: an owner runs store apply --account <a> --note "<why>" and a deployment operator approves it — until then create has nothing to provision into, and status says the account is unapproved or pending. After approval create is the whole member workflow (unlimited stores, usage metered); --environment <e> gives one config bundle its own bucket and credential; credentials arrive as env; rotate is two-phase (--finish retires the old key). scan is opt-in virus scanning over the bucket's objects (request/status are yours, grant/revoke/sweep/enforce a deployment operator's) — and a scanner cannot read ciphertext, so a client-encrypted store is marked not applicable rather than given a coverage claim | | carlos email enable\|status\|test\|domains\|events\|credentials\|rotate | Declare sending; provision a verified domain (domains rm frees one); SMTP credentials arrive as env; events shows what became of each message for 30 days, events push --path posts them to the app, signed (pause/resume are a deployment operator's) | | carlos relay enable\|status\|disable | The platform's own STUN/TURN relay for an app's WebRTC calls. enable delivers CARLOS_RELAY (base64 JSON: per-region ICE URLs, a key, a username suffix) and the app mints a TURN credential per call from the key; status shows issue, pickup and per-region health. enable refuses, with a sentence, on a deployment that runs no relay. Never run coturn or buy a TURN service | | carlos services create\|ls\|rotate\|revoke | Server-held credentials (--role deploy\|publish\|instance\|operate\|service\|admin, --app to bind one app, --rungs for deploy); the secret prints once; rotate refuses the old secret on its next call, revoke stops the credential outright | | carlos schedule ls\|set\|rm\|run | Declare recurring work per app (--every 6h or --cron "0 8 * * *" → a POST to a path your app serves); ls shows next/last per instance; run fires one now | | carlos ledger append\|publish\|verify | Open hash-chained per-app ledgers (the transparency machinery) | | carlos accounts create\|list\|migrate | Mint/list accounts; move an app between them. An account is your tenancy on the platform — never create one per customer of your app (customers are instances: "Provisioning instances from your own app") | | carlos skills | List the deployment's published agent skills (/.well-known/agent-skills/index.json) and fetch one, digest-verified, to stdout | | carlos fleets create\|add-box\|rotate-token\|… | Bring-your-own-boxes fleets that dial the console | | carlos update | Update the CLI binary itself (signature-verified; defers to brew/apt) | | carlos vet | Check a release against the platform contract |

Box-side verbs exist (edge, agent, adopt, route, add, ops, bootstrap) but they are the operator's surface for running a platform deployment — a member never types them, and an agent reaching for them on a member task has taken a wrong turn.

Provisioning instances from your own app

A product where every customer gets their own instance creates those instances itself, at signup, through the same console API the CLI uses. Nothing here needs a box, and nothing here is a workaround:

1. A person enables the app once — carlos instances enable --app <tenant-app>. This writes the manifest that bounds which domain, pool and regions any instance may be created in, and no credential can do it. Add --private-storage and every instance created afterwards gets a config environment named after its host's first label with an object store bound to it, provisioned by the ordinary create call (the account needs an approved store apply first, and nothing is torn down when an instance is deleted).
2. Mint the credential — carlos services create --role instance --app <tenant-app> signup. Store it with carlos secrets set on the app that runs signup, never on the tenant app itself: every tenant instance would then hold a token able to delete its siblings.
3. Create on signup — POST /api/cli/apps/<account>/<tenant-app>/instances with Authorization: Bearer <credential> and a JSON body; host is required (one DNS label plus the manifest's domain), region, channel, environment and message (the Activity description) are optional. The console writes the record and announces it, and a box mints the route within seconds.
4. Wait for the real thing before sending the person there: poll https://<host>/api/version for a 200 (a sub-500 answer is not evidence), behind a zero-JS "getting your space ready" page that refreshes itself.

Give every label a random suffix (<slug>-<6 random base32 chars>, the label being the host's first DNS segment). That makes a collision with somebody else's instance a first-call event, so the one retry below can read its 409 without guessing.

The answers to handle, from the apps that run this:

| Status | Meaning | Do | |---|---|---| | 201 | Created | Wait for /api/version, then hand off | | 409 | That host already exists (or a sibling app owns the label) | Never treat it as yours: pick a new label. On a retry of your own timed-out call, a 409 means the first call worked | | 429 | The credential's own rate limit (it is separate from any person's, so a signup burst never starves your promotes) | Show a sentence and a retry; never loop | | 502 / timeout | The console could not complete the write | Retry once with the same host | | 400 | Bad body, or a host that is not one label plus the manifest's domain | A bug on your side; fail loudly | | 401 / 404 | Bad credential, or the wrong account or app (every miss is the same 404) | A configuration bug; fail loudly |

Also on the same credential: GET .../instances (list), GET, PATCH (reorder or remove addrs, set the probe — adding an upstream address is operator-only since v0.25.0) and DELETE .../instances/<host>, and POST/PUT .../domains for customer hostnames. Deleting an instance removes its record, never its data — the parked database and blobs are left in place on purpose, and the call is idempotent.

The known gap: the instance role is app-wide. One credential can create, delete or move every instance of the app, not only the ones it made. That is why it lives on the signup app alone, and why a signup app compromise must be named in your published trade-offs. A narrower, create-only credential is a filed platform request — do not build your own token broker around it.

One hostname, many instances

Path routes let one hostname — your product's own domain — serve each customer from their own instance, so example.com/acme and example.com/globex are two processes and two databases behind one address. The hostname's own route keeps everything else (/, sign-in, marketing).

• The hostname must be one the app holds: carlos domains attach --app <hub-app> example.com first.
• The map is written whole through the API, by a person or by the hub app's instance/publish credential: PUT /api/cli/apps/<account>/<hub-app>/domains/example.com/paths with {"paths": {"acme": "acme.<tenant-domain>", "globex": "globex.<tenant-domain>"}}. Keys are one bare path segment (no slash); values are hostnames of existing routes. Publish the full map from your own records every time — a lost delta desyncs silently, a whole republish repairs. {"paths": {}} clears it; a body without the paths wrapper is refused, so a typo can never wipe the map.
• Targets must belong to the same account (they may be another app of that account — a hub app pointing at tenant-app instances is the intended shape).
• Targets must live on the same box as the hostname's own route. A target homed elsewhere is silently ignored and its prefix falls through to the hostname's route. That caps a path-routed product at one box today; cross-box path targets are a filed platform request.
• A write goes live in about five seconds. carlos domains paths --app <hub-app> example.com prints what was published; carlos routes shows what the box actually applied.
• The upstream receives the original path (/acme/...) unchanged, and the target route is what is billed, versioned and woken.
• One hostname is one browser origin. Cookies, storage and script are shared across every prefix, so every tenant's pages can act on the hub's session. That is fine while every tenant runs your app's code; if tenants serve HTML their own customers wrote, give each tenant its own hostname instead.

Deploying a static site

A static site is one command, and the command is the same one binaries use:

carlos apps create --app mysite
carlos deploy --app mysite --kind static --host mysite.<sqid>.oncarlos.com ./site

That ships the directory as an immutable release, promotes it to the app's entry channel, declares the route, and watches the URL until X-Carlos-Version reports the build. Nothing else is needed — not carlos instances enable, not carlos instances create, not a separate carlos domains attach.

--host is required on the first static deploy and only that one. A static app has no instance record yet, so there is nothing to resolve the host from; that deploy writes the record, and afterwards plain carlos deploy resolves host and channel by itself. Give it a custom domain and the deploy attaches that too, printing the DNS records to publish.

Expect the URL to take up to a minute after promoted prints. A static route trusts its resolved channel pointer for about 60 seconds, so the old page can outlive the promote by that long — carlos deploy already waits past one such window, and says so while it waits.

A static site has no instance, and must not be given one

This is the trap. The verbs read like the binary recipe — enable an app for instances, create one, point it at a channel — and every step succeeds. The result cannot work.

A static site is served by the edge straight off the channel pointer: there is no process, nothing hibernates, nothing wakes. An instance is a process the platform execs. A static release's manifest carries a site tarball and no build for any box's architecture, so an instance route on that channel finds nothing to run and fails every wake, permanently.

Older platform builds reported that failure as "nothing is promoted for this instance's channel" — while carlos channels showed the promote sitting right there. Both statements were true and neither named the problem, which is the shape of the route, not the state of the channel. Current builds name it and refuse the create; if you meet the old message, this is what it means.

The way out, on an app already in that state:

carlos instances delete --app mysite --host mysite.<sqid>.oncarlos.com
carlos instances create --app mysite --host mysite.<sqid>.oncarlos.com --kind static --channel <ch>

The already-shipped release needs no re-ship — a static route reads the channel pointer, so the site serves within seconds of the record landing.

Channels belong to the app, not to the platform

A new app is born with one channel, edge by default, and that one channel is its production. More are opt-in, declared by a pipeline, and named whatever the app calls them.

So do not guess a channel name from the canary → edge → beta → stable ladder in --help: that is the frozen legacy ladder an app inherits only when it declares no pipeline. carlos channels lists what this app actually has and carlos pipeline shows the order. carlos promote <version> with no channel resolves it when the app has one, and names the app's real channels when it has several.

Standing a canary beside production

Review happens on a deployed canary, never on localhost, and one command does it:

carlos canary --app jam ./jam

The name defaults to the slugified git branch, so branch new-queue gives channel canary/new-queue and host new-queue-jam.<sqid>.oncarlos.com. An explicit --name is validated rather than quietly repaired, because the same string has to serve as both a channel segment and part of a hostname label: lowercase letters, digits, dashes, and at most 63 characters with the app name. carlos canary ls shows each canary's host, what it serves, which config bundle it is bound to and what is left of its seven-day lease; rm deletes the instance record and the host stops being served. The channel lapses on its own.

If the app carries config, pass --environment. This is the trap, and it is worth stating in full because the failure looks like an app bug. Config in CARLOS is per-app, so a canary with no bundle of its own serves the app's default bundle — where every origin-shaped value names the app's main host. Anything keyed to that single scalar refuses requests on the canary hostname: the Origin header the app compares against, magic-link and callback URLs, a WebAuthn RP ID, the cookie Secure flag. The result is a canary that serves every GET perfectly and fails every form submission. Write the bundle first, then bind it:

carlos env --app jam --environment jam-canary JAM_ORIGIN=https://new-queue-jam.<sqid>.oncarlos.com
carlos canary --app jam --environment jam-canary ./jam

The same applies to any second instance standing beside a live one, not just a canary — carlos instances create --environment, or carlos instances set-environment on a route that already exists, followed by a restart to pick the values up.

Watching an app: errors, availability and analytics

All three read from the terminal or the app's dashboard tabs. Errors and availability collect from the day the app ships — nothing to turn on. Analytics is off until you switch it on (a deliberate ruling, 2026-09-07: counting visitors is the app's call, not the platform's).

carlos errors --app hello --since 24h
carlos availability --app hello --since 24h
carlos analytics --app hello --set-count edge
carlos analytics --app hello --range 7d

carlos errors needs no wiring at all: it groups error-tagged log lines (or warn too, with --set-level warn) together with the 500s the app itself answered, one row per distinct error with a count and a location. --fp <fingerprint> expands one group to its occurrences.

carlos availability (since v0.22.0) holds what used to sit in errors: every 502, 503 and 504 the edge saw, meaning the instance could not answer, grouped by cause (refused, closed mid-request, timed out, no box connected, or the app's own 503), with failed wakes and exits beside them. One outage is one row however many pages it hit; --fp expands a cause to each request and path. A visitor who gives up after five seconds or more without response headers is filed as an outage, so an endpoint that holds its headers by design, like a long poll, shows a standing row there, and that row is expected.

carlos analytics counts at the edge as it serves the app: page views, daily visitors, bytes served, top pages, top referers, named events — no JavaScript, no cookie, no stored IP, which is why an app on CARLOS has no business reaching for a third-party analytics script. Visitors are a sketch, so a multi-day total is daily visitors summed rather than a deduplicated unique count. The switch is per-app or account-wide. Where the deployment publishes a GeoIP database, page views are counted by country too.

Neither replaces carlos logs, still the merged app + platform + edge timeline (-f, --grep, --since, --stream app, --host) with no box access.

Sending email

The platform issues SMTP credentials. An app declares that it sends, the console provisions an SES identity for a domain, publishes the DNS records where it controls the zone, and delivers working credentials as config. The app never holds a cloud key and nobody runs an MTA.

One command does the whole walk — declare, provision the domain, wait for verification, deliver:

carlos email enable --app <app>

The identity defaults to the app's own host (<app>.<sqid>.oncarlos.com), so the signing domain matches the domain of the links inside the mail. Credentials arrive as env — CARLOS_SMTP_HOST, _PORT, _USER, _PASS, _FROM. Transport is submission on 587 with STARTTLS. --env-prefix MAILER renames the whole set to MAILER_HOST, MAILER_PORT, and so on, for an app that already reads its own names.

For a customer's own domain:

carlos email enable --app <app> --domain mail.example.com

On the platform domain CARLOS publishes the records itself and verification is usually seconds. On a custom domain it prints the record set for the zone's owner to publish, then polls until SES confirms it (--timeout, default 10m; a gave-up wait exits non-zero and names what SES is still waiting for). carlos email domains add is the same provisioning step on its own, for adding a second sending domain to an app that already sends.

The record set a customer has to publish is always these five, for <domain> and the region the identity was first verified in:

| Name | Type | Value | |---|---|---| | <token>._domainkey.<domain> × 3 | CNAME | <token>.dkim.amazonses.com | | mail-<region>.<domain> | MX | 10 feedback-smtp.<region>.amazonses.com | | mail-<region>.<domain> | TXT | v=spf1 include:amazonses.com ~all | | _dmarc.<domain> | TXT | v=DMARC1; p=none; adkim=s; aspf=s (plus rua= if the deployment sets one) |

The SPF record sits on the MAIL FROM subdomain rather than the identity domain because that is the domain SPF is checked against.

The published DMARC policy is p=none, deliberately: a tenant whose very first send is rejected by its own DMARC record learns nothing useful from the failure. There is no carlos email verb that changes it. On a custom domain the zone is the customer's, so they can tighten it themselves whenever they choose — read the DKIM note below first.

Then:

| Verb | What it does | |---|---| | carlos email status --app <app> | What is declared, which domains verified, what was delivered | | carlos email test --app <app> --to me@example.com | Sends a REAL message through a throwaway credential, then revokes it | | carlos email credentials create\|list\|revoke | Standalone credentials for something not running on CARLOS; the password is shown once | | carlos email rotate --app <app> [--finish] | Two-phase key rotation — new key delivered, then the old one retired |

Three things that will bite

• A freshly minted credential does not work for the first minute or two. SES's SMTP endpoint does not see a new IAM access key immediately, and the symptom is 535 Authentication Credentials Invalid — which reads exactly like a wrong password and is not one. Wait and retry. An app's first sends straight after enable hit this too, so do not diagnose a broken credential from the first minute of logs. A 535 that outlasts a few minutes is a different problem and is worth treating as real: check the key is still Active and not paused (carlos email status), and that the password was derived for the same region as the SMTP host — the derivation is region-scoped, so a credential minted for one region never authenticates against another's endpoint.
• DMARC alignment rests on DKIM alone. Records are published with adkim=s; aspf=s, and that strictness is what stops one tenant's signature authenticating another tenant's From address on a shared domain. SPF authenticates but does not align: the envelope sender is a region-qualified subdomain (mail-<region>.<domain>), which strict alignment treats as a different domain. DMARC therefore passes on the DKIM rail only. That is normal for SES — but it means DKIM is the single rail, so never turn off DKIM signing on a sending identity. If a customer tightens their own zone to p=reject, that is safe but unforgiving: a DKIM CNAME later dropped from their zone stops being a downgrade and becomes total delivery failure.
• The MAIL FROM subdomain carries its region forever. It is fixed at the first verification (mail-eu-west-1.<domain>). On a custom domain that record lives in the customer's zone — the one record CARLOS cannot republish for them — so moving regions later means going back to the customer for a DNS change.

Scheduled work

The platform keeps the clock. A hibernating instance cannot run an in-process timer, so recurring work is declared from outside the app and delivered as a tick: a POST to a path you declared, at the time it is due, waking the instance first and holding it awake while your handler runs. Nothing else crosses the boundary — no payload, no job state, no queue. Live 2026-08-24.

Declare it with the CLI, not in code:

carlos schedule set --app <app> --name sync      --every 6h         --path /jobs/sync
carlos schedule set --app <app> --name reminders --cron "0 8 * * *" --path /jobs/reminders

--every is a Go duration, whole minutes, 1m to 30d. --cron is five UTC fields (min hour dom month dow, with *, n, a-b, */n, a,b) — no names, no @daily, no seconds. Names are ^[a-z0-9][a-z0-9-]{0,31}$, at most 20 per app. carlos schedule ls prints the declaration and then what each instance reports (next, last, last_status); run requests an out-of-band fire on top of the normal rhythm; rm removes the declaration. The verbs say recorded and requested, never "fired": they write one console object, and each box acts on it within seconds.

The app's whole part is a handler on that path:

// POST /jobs/sync
func handleSync(w http.ResponseWriter, r *http.Request) {
	if !carlos.Tick(r) {           // amadan.net/rastrillo/rastrillo/carlos
		http.Error(w, "forbidden", http.StatusForbidden)
		return
	}
	if occ, ok := carlos.TickOccurrence(r); ok && alreadyDone(occ) {
		w.WriteHeader(http.StatusNoContent)
		return
	}
	if err := syncer.RunOnce(r.Context()); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError) // 5xx = retry
		return
	}
	w.WriteHeader(http.StatusNoContent)
}

carlos.Tick is the constant-time compare of the bearer against $CARLOS_ADMIN_TOKEN, which the platform mints into the instance's environment; use it rather than hand-rolling the check. A tick arrives on an ordinary public route, so a tick nobody authenticated is an internet request to that path — and the X-Carlos-* headers are not evidence, anyone can set those. The status is your whole reply: 2xx done, 5xx retry with backoff (five attempts, 1m doubling to 15m, then the schedule advances), 4xx don't bother.

One-shots are the app's own call, over the control socket the agent binds before your process starts:

err := carlos.ScheduleAt(ctx, "remind-"+id, when, "/jobs/remind")  // carlos.ScheduleCancel drops it

Same tick, same handler, same name namespace as declared schedules (a collision returns ErrDeclaredSchedule). Up to 400 days ahead, 1000 pending per host. Reach for one only when the work genuinely has a specific time; a daily schedule whose handler asks its own database "what falls today?" beats one timer per row.

Things that bite

• Do the work inside the request. The instance is held awake for exactly as long as the request is open (30-minute ceiling) and the idle clock starts the moment you return. Reply 202 and finish in a goroutine and that goroutine gets hibernated mid-job.
• Dedupe on X-Carlos-Schedule-At — unix seconds, read it with carlos.TickOccurrence. It is the instant the delivery is for, so every retry of one failed occurrence carries the same value, as does a redelivery after a box crash or a deploy that cut a long job. Never key on the wall clock: a retry twenty minutes later is the same occurrence. Keep Tick as the guard — a request with no occurrence header means "no dedupe key, run it", which is what your own "Sync now" button looks like.
• A 4xx is never retried. A path with no handler, or a handler that refuses a token it cannot see, records app refused and the schedule advances to its next occurrence. That failure is quiet — next still moves, so ls reads like a success. Check last_status.
• An instance running since before the platform roll refuses its own ticks. The token is minted at spawn, so a process older than the roll has no $CARLOS_ADMIN_TOKEN, Tick is always false, and carlos.ScheduleAt returns ErrUnauthorized. carlos restart clears it; nothing else does.
• A schedule tighter than the idle window keeps the instance resident, and residency is billed. Since 2026-08-31 a tick no longer stamps activity, but the sweeper deliberately will not put a host to sleep when its next tick is due sooner than the idle window (15 minutes by default) — staying awake is genuinely cheaper than a wake/sleep cycle per tick, each of which is a full SQLite checkpoint-and-restore. So a 5-minute schedule against a 15-minute window never sleeps and accrues instance-hours around the clock; an hourly one sleeps about three quarters of each hour. carlos schedule ls and set now say so in one line rather than leaving it to be found on a bill. The platform warns and never refuses: a tight schedule is a legitimate thing to want.
• Unit-backed instances are not delivered to in this release — ls says unsupported against them. Exec-backed instances (the default) and sidecars work.
• Once-timers outlive the process. They live in the box registry, so a restart loses none, and re-asserting them at boot is safe because a repeated name replaces rather than adds. Off-platform there is no control socket at all: ScheduleAt returns ErrNotOnCarlos, which boot code should treat as "skip", not as a failure.

Two builds at once

Since v0.26.0, where the deployment has switched it on (Carloku did on 2026-10-14; the release default is off, and then a build is taken by restart and deploy watches the URL instead of each box's rollout), a new build reaching an awake instance starts beside the old one and takes over once it passes its health check, so no request waits (--swap overlap, the default). For a few seconds both builds run against the same database and the same scheduled work, and the new build's migrations run while the old one still serves. That makes two rules part of every release:

• Migrations stay additive across two consecutive releases. Add a column, table or index in one release; drop the old one in a later release, once no running build reads it. A rename is an add now and a drop later.
• Schedule names stay the same across the two builds (carlos schedule).

A release that cannot follow them ships with --swap hold on carlos deploy or carlos ship: each instance stops before its new build starts, and requests wait a few seconds for it. carlos apps set <app> --swap hold makes that the app's rule, and --swap-concurrency is how many instances on one box swap at once, after the first has stayed up for ten seconds. If a build fails on a box, that box stops rolling it out, its other instances stay on the build they were serving, and carlos deploy names the failure and exits non-zero; carlos instances list shows it under REASON. Fix the cause, then carlos promote --retry <version> <channel> (every box) or carlos instances retry <host> (one instance).

Visitor location headers

On a deployment that publishes a GeoIP database (since v0.25.1), every request the edge passes to an app carries the visitor's rough location: X-Carlos-Geo-Country (ISO 3166-1, IE), X-Carlos-Geo-Subdivision (ISO 3166-2, only some databases), X-Carlos-Geo-Lat / -Lon (two decimals: an area, not a position) and X-Carlos-Geo-Source (which database answered). Any of them can be missing: treat that as unknown, never as an error. The edge strips any a visitor sends, so the values are always the platform's. A cached page that differs by country sends Vary: X-Carlos-Geo-Country; never vary on the coordinates.

An app that shows a visitor anything worked out from these headers must credit the database X-Carlos-Geo-Source names — DBIP-…: "IP Geolocation by DB-IP", linked to https://db-ip.com; GeoLite2-…: "This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com". For an address that is not the current request (a log line, a webhook), carlos geo <address> --app <a>, or from an app GET /api/cli/apps/<account>/<app>/geo?ip=<address>, at most 60 a minute. That call needs an instance-role token, which is app-wide (it can also create, repoint and delete every sibling instance), so make it only from a central app that already holds one; never deliver that token to a tenant app's instances just to look up an address. Tenant instances use the request headers.

Deploy truths (each paid for at least once)

• carlos deploy is the release motion: ship, promote to the entry channel (or the channel the app's instances follow), then follow each box's rollout (or, where a box does not report one, watch the URL) until the shipped build is serving. "Held for bake" on a channel that declares one is the system working, not failing.
• A promote is not a deploy until the process cycles. An awake instance starts the new build within seconds of its box adopting it (beside the old one where two builds at once is on), an asleep one takes it on its next request, and unit-stamped routes are restarted. A bespoke long-running unit outside the platform's knowledge stays old until something restarts it — know which of the three your app is.
• Verify against the thing you changed, with the binary you built. A 200 is not proof: an app-shell route returns 200 HTML for /api/version and will happily "verify" any build ever shipped. The proof is the build the process itself reports: set carlos apps build-probe, or read X-Carlos-Version on the canonical host knowing the edge stamps it before the process has cycled; stale DNS and stale local binaries have both produced false "verified" reports.
• Review on a canary, not on localhost. carlos canary cannot disturb production: its channel sits outside every pipeline and its hostname is the branch's own. Bring a config bundle with it if the app has one.
• When CI ships for you, verify by ancestry, not equality — a PR merging behind yours cancels your run and ships a commit containing yours.
• Stamp your build version. For rastrillo apps, -ldflags "-X amadan.net/rastrillo/rastrillo.BuildVersion=<sha>" — or every release's /api/version reports dev.

Switched off, or not released

Released, switched off on Carloku — the CLI has the flag, the console declines or ignores it, and nothing should be built to stand in for them: app residency (--region on apps create, deploy and canary since v0.23.0; the console starts the app in its home region and says so), instance moves (carlos move, v0.26.0; it sends nothing), and temporary static sites (deploy --temporary / --ttl, apps ttl|keep, v0.25.1). Names starting tmp- are reserved whether or not temporary sites are on.

Released in v0.28.0 but refused until an operator turns them on: seeded instances (instances create --from <host> starts a new instance from a copy of another's database; instances import brings in one the platform never ran). Not built: an instance-bound token in place of the app-wide instance role, a create-only provisioning role, and cross-box path targets. Check carlos <verb> -h against the console and wait.

Self-hosting the platform

The platform is the same software members deploy onto: a self-hosted deployment (Tito's path) runs its own console, boxes, and bucket in its own cloud account, and its members use the identical carlos CLI pointed at that console. Operating it is real work — box provisioning, the adopt/restart cadence, platform rolls — and it is the one context where box verbs and cloud access are legitimate. Commit a .carlos/config naming the deployment's console so sessions cannot fall back to the wrong one; the CLI's account default is not your account (ops exists on every deployment and is empty — a not found against it once cost an hour of confident wrong diagnosis).

The deployment's own binary is a channel like any app's: carlos system update --channel canary|stable promotes a release, and every box on that channel fetches it, verifies it against the signed pointer, installs it, restarts its edge, and rolls itself back if the new one does not come up and stay up. carlos system status shows each box's running build and last apply outcome; carlos system rollback undoes it; --spread 10m staggers a stable roll. The console is an app — update it with carlos deploy --app console. Owner of the ops account only.

For the underlying machinery — registry, router, replication, hibernation internals — see blueprint.md, which is the reference for what the platform does on your behalf and for hand-rolling outside it.