amadan
spec: app capabilities — a closed vocabulary, provided by the platform
Two production apps run as bespoke hand-provisioned units the infra repo does not even have the text of. Seven capability names — command, daemon, children, data, runtime_dir self-service; ports and host_role operator-granted — materialize as vetted drop-ins on the one carlos-app@ template, declared in the release manifest so a rollback rolls back the unit shape with the binary. Migration paths for console.service and keymail-relay.service; monitoring then attributes both with zero special cases. Directive (Paul): avoid special cases for CARLOS apps — app capabilities should be provided by the platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+734
−0
docs/superpowers/specs/2026-08-03-app-capabilities-design.md+734 −0
diff --git a/docs/superpowers/specs/2026-08-03-app-capabilities-design.md b/docs/superpowers/specs/2026-08-03-app-capabilities-design.md| new file mode 100644 |
| index 000000000..484c2cedc |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-03-app-capabilities-design.md |
| @@ -0,0 +1,734 @@ |
| +# App capabilities — design |
| + |
| +Date: 2026-08-03. Directive (Paul, verbatim): *"We want to start avoiding |
| +special cases for CARLOS apps. App capabilities should be provided by the |
| +platform."* |
| + |
| +Two production apps run as bespoke, hand-provisioned systemd units instead |
| +of platform-managed `carlos-app@<host>.service` instances: |
| + |
| +- **console** (`console.service`, both boxes) — the platform console. Long |
| + running, owns `console.db`, reads the deployment bucket, signs channel |
| + pointers when it holds a key. |
| +- **keymail relay** (`keymail-relay.service`, flagship) — binds the MX port |
| + `:25` directly, serves HTTP on `127.0.0.1:8446` behind the edge, spawns |
| + one `keymail instance` child per mailbox over sockets in `/run/keymail`, |
| + and replicates its own SQLite files with embedded litestream. |
| + |
| +Neither unit exists as text in the infra repo. `platform-infrastructure` |
| +contains operational references only (`systemctl restart console.service` |
| +in `scripts/rollout-status-flagship.sh`, a *rehearsal* unit heredoc in |
| +`scripts/rehearse-keymail.sh`) — the production units live on the boxes and |
| +nowhere else. That is the sharpest statement of the problem: the two most |
| +important long-running processes in the fleet are configured by memory. |
| + |
| +What the special-casing costs today, concretely: |
| + |
| +- The metrics sampler scans `carlos-*.service` cgroups |
| + (`internal/host/metrics_probes.go`), so neither app appears in the |
| + operator per-unit table, and `keymail.dev`'s owner page renders "no |
| + per-instance readings" while the relay is serving mail. The monitoring |
| + spec (`2026-08-02-host-process-monitoring-design.md` §1) already *claims* |
| + "the console and keymail are themselves `carlos-app@` instances". This |
| + design makes that sentence true. |
| +- Adoption's restart is hand-scripted (`systemctl restart console.service` |
| + over SSM) rather than owned by the promote → adopt → restart order. |
| +- Config delivery, litestream coverage, and every future platform feature |
| + has to remember these two by name. |
| + |
| +Scope: the unit shape of an app. Nothing here touches the console's |
| +security gates or their ordering, the channel ladder, or the pinned |
| +production channel names (`internal/release/laddercheck_test.go` — `canary/*` |
| +keeps working exactly as today; the ladder is not consulted anywhere in |
| +this design). |
| + |
| +## 1. What the two apps actually need |
| + |
| +Derived, not imagined. Each row is something the bespoke unit does today |
| +that the template cannot express. |
| + |
| +| Need | console | keymail relay | Template today | |
| +|---|---|---|---| |
| +| A verb other than `serve` on the ExecStart line | no (`serve`) | yes (`relay`) | fixed `serve` | |
| +| Long-running: enabled, started at boot, restarted on adopt, never hibernated | yes | yes | nothing enables an instance at all | |
| +| Direct bind of a privileged port (`:25`) | no | yes | `CapabilityBoundingSet=` empty; no listener but the unix socket | |
| +| A persistent data dir the platform provisions, owns, and knows how to replicate | yes (`console.db`) | yes (app replicates itself) | `StateDirectory` exists; the DB filename is unknown to the platform | |
| +| A runtime dir at a fixed absolute path (`/run/keymail`) | no | yes | none | |
| +| Forks a process family into its own cgroup | no | yes (1 + N mailboxes) | `TasksMax=256`, `MemoryMax=512M` — a single-process budget | |
| +| Uses the box's cloud instance role from inside the process (IMDS) | yes (S3: releases, metrics, pointers) | yes (litestream, sealed blobs, SES) | `DynamicUser=yes` mints an ephemeral uid; the box's IMDS firewall gate admits root and `carlos` only | |
| + |
| +The last row is not theory. During the keymail rehearsal the IMDS uid gate |
| +starved the relay's in-process AWS clients, restore-on-boot silently created |
| +empty databases, and the relay served them |
| +(`docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md`, finding F1). |
| +Any design that ignores it does not migrate keymail. |
| + |
| +Everything else both apps need — origins, RP IDs, bucket names, PoW cost, |
| +listen addresses — is **environment**, which the config plane already |
| +delivers to unit-backed routes (`appconfig.Sync` writes |
| +`/etc/carlos/apps/<host>.env`, which the template already reads). No |
| +capability is needed for any of it, and none is defined for any of it. |
| + |
| +## 2. The capability model |
| + |
| +An app **declares** what it needs from the platform; the platform decides |
| +how (and whether) to provide it. The vocabulary is **closed**: seven names, |
| +each mapping to a fixed set of vetted systemd directives that live in this |
| +repo, never in a manifest. A manifest never names a path, a user, a uid, a |
| +resource number, or a systemd directive. |
| + |
| +**Self-service** — affects only the app's own sandbox, materializes from |
| +the manifest alone: |
| + |
| +- **`command`** — the single subcommand token appended to the template's |
| + ExecStart in place of `serve` (keymail: `relay`). Charset |
| + `^[a-z][a-z0-9-]{0,31}$`, one token, no flags. Flags and values are |
| + environment; keymail already reads all of its own from `KEYMAIL_*`. |
| +- **`daemon`** — long-running. The platform `enable`s the instance (so it |
| + survives reboot), starts it if inactive, **restarts** rather than |
| + try-restarts it on adopt, and refuses hibernation for the route. This is |
| + the capability that makes "adoption swaps a symlink; it does not restart |
| + processes" survivable: the platform owns the restart, in the documented |
| + promote → adopt → restart order, instead of an operator remembering it. |
| +- **`children`** — the app forks a family. The unit is sized for a family |
| + rather than a process (platform-chosen `TasksMax` / `MemoryMax` tier, not |
| + manifest-chosen numbers), `KillMode=control-group` is pinned explicitly so |
| + a stop takes the whole family, and monitoring reads the cgroup as one |
| + honest row for the family. |
| +- **`data`** — a persistent per-app data dir. The path is **always** |
| + `StateDirectory=carlos-app/%i` (`/var/lib/carlos-app/<host>`); the |
| + manifest may not name it. What the manifest adds is the part the platform |
| + cannot guess: which files under it are databases, and who replicates them. |
| +- **`runtime_dir`** — `/run/<name>`, `RuntimeDirectory=<name>`, |
| + `RuntimeDirectoryMode=0700`, `RuntimeDirectoryPreserve=restart`. The name |
| + matters for keymail because its restored `registry.db` stores **absolute** |
| + socket paths (`/run/keymail/<local>.sock`; finding F2), so the platform |
| + must let the app name the directory — inside a closed charset and a |
| + reserved-name list (§7). |
| + |
| +**Operator-consented** — consumes a shared, scarce, or box-authority |
| +resource. The manifest declares the need; nothing materializes until an |
| +operator has granted it *on this box*: |
| + |
| +- **`ports`** — direct binds of privileged ports (`< 1024`). keymail: `25`. |
| +- **`host_role`** — the app's processes may use the box's cloud instance |
| + role. This is a real transfer of authority (the flagship role can read and |
| + write the deployment bucket, keymail's blob bucket, and send SES mail), so |
| + it can never be self-granted by tenant input. |
| + |
| +The split is the whole security model in one sentence: **a capability that |
| +only shapes the app's own sandbox is the app's to declare; a capability that |
| +reaches anything shared is the operator's to grant.** |
| + |
| +### Grants |
| + |
| +Grants are a new registry column, per route, per box — the operator's door |
| +is `carlos route`, which already carries safety rails and narrow setters: |
| + |
| +``` |
| +carlos route -grant port:25 keymail.dev |
| +carlos route -grant host-role platform.carlosframework.com |
| +carlos route -revoke port:25 keymail.dev |
| +``` |
| + |
| +Per box, not per fleet, because the resource is box-local: `:25` can be held |
| +by exactly one process on one machine, and the IMDS gate is that box's |
| +firewall. Two boxes running the same app is two grants, deliberately. |
| + |
| +A declared-but-ungranted capability does not fail the app and does not fail |
| +adoption: the drop-in simply is not written, and the pass logs one line per |
| +pass naming the route, the capability, and the exact `carlos route -grant` |
| +command. keymail with no `port:25` grant serves HTTP and does not bind the |
| +MX — senders retry, which is what SMTP is for — instead of refusing to start |
| +and taking the web UI down with it. |
| + |
| +## 3. Wire and storage shapes |
| + |
| +### 3a. The manifest field (additive, schema stays 1) |
| + |
| +`release.Manifest` gains one optional object. Precedent: |
| +`appconfig.Manifest.ExecDelivery` (issue #18) added a field to a schema-1 |
| +manifest without bumping the schema, because an old binary must still parse |
| +what a new one writes. |
| + |
| +```json |
| +{ |
| + "schema": 1, |
| + "app": "keymail", |
| + "version": "6927be7", |
| + "kind": "binary", |
| + "artifacts": { "linux-arm64": { "file": "linux-arm64", "sha256": "…", "size": 14680064 } }, |
| + "capabilities": { |
| + "v": 1, |
| + "command": "relay", |
| + "daemon": true, |
| + "children": true, |
| + "runtime_dir": "keymail", |
| + "ports": [ { "port": 25, "name": "mx" } ], |
| + "data": { "replicate": "app" }, |
| + "host_role": true |
| + } |
| +} |
| +``` |
| + |
| +```json |
| + "capabilities": { |
| + "v": 1, |
| + "daemon": true, |
| + "data": { "replicate": "platform", "dbs": ["console.db"] }, |
| + "host_role": true |
| + } |
| +``` |
| + |
| +Field rules: |
| + |
| +- `"v"` is the capability block's own version, matching the `"v": 1` |
| + discipline the metrics objects use. `schema` stays 1 forever for this |
| + change; every manifest already in a production bucket has no |
| + `capabilities` key, parses unchanged, and means "no capabilities" — which |
| + is today's behaviour byte for byte. |
| +- `data.replicate` is `"platform"` (the box-wide root `carlos-litestream` |
| + unit replicates the named `dbs`) or `"app"` (the app streams its own WAL; |
| + the platform provisions the directory and replicates **nothing**). Two |
| + replicators on one WAL fight over checkpointing and corrupt the replica — |
| + the hazard `ops.FleetDBs` already documents for exec children. `dbs` names |
| + bare filenames inside the state dir (`^[A-Za-z0-9._-]{1,64}$`, no |
| + separators, no `..`), and is meaningless with `"app"`. |
| +- `ports[].name` is a label for operators and logs. It is not a systemd |
| + `FileDescriptorName` (see §4c) and carries no behaviour. |
| + |
| +**Why the release manifest and not the config manifest.** `exec_delivery` |
| +lives in the config plane because it describes delivery to a supervisor the |
| +platform does not own, and it is a property of an app, not a build. |
| +Capabilities describe the **unit the platform does own**, and they change |
| +with the code: the version that starts binding `:25` is a specific version. |
| +Putting them in the immutable, content-addressed release manifest means (a) |
| +a rollback rolls back the unit shape with the binary, (b) the declaration |
| +rides the same trust chain as the bytes — signed pointer → immutable |
| +manifest → sha256 — with no second CAS object to keep in step, and (c) there |
| +is no window in which a running binary and its unit disagree about what the |
| +app is. |
| + |
| +### 3b. Forward compatibility, and which way each failure falls |
| + |
| +- Unknown **top-level** manifest key: ignored (Go's decoder already does |
| + this). Old binaries keep parsing new manifests. |
| +- Unknown **field inside a known capability**: ignored, one log line. Room |
| + to grow a capability additively. |
| +- Unknown **capability name**, or `"v"` greater than this binary knows: the |
| + **version is refused on this box** — not staged, not pointed at, one log |
| + line per pass naming the capability and the fix ("update the platform |
| + binary"). Fail closed, because the alternative is materializing a unit |
| + that is missing something the app says it needs and looks healthy anyway — |
| + the config-that-lies failure mode that shaped issue #18. The box keeps |
| + serving the version it has, which for a daemon is the safe direction. |
| +- Ship-time validation refuses anything this binary does not understand, |
| + where a human is present to read the refusal. Same value set, one |
| + `ValidCapabilities` function shared by ship and adopt, mirroring |
| + `ValidExecDelivery`. |
| + |
| +**Deploy order, pinned:** platform binaries first, capability-bearing |
| +releases second — the same rule and the same reason as scoped signing keys |
| +(`signedWrapper.Grant`: an old binary meeting a new document must fail in |
| +the direction that changes nothing). |
| + |
| +### 3c. On-box storage |
| + |
| +- **Staged manifest**: `ensureStaged` writes the manifest it already fetched |
| + to `<opt>/versions/<ver>/[<acct>/]<app>.manifest.json` — a sibling file of |
| + the staged binary, so no directory shape changes and an old box's tree |
| + stays valid. This is what lets the converge pass read a *running* version's |
| + capabilities without a bucket round-trip on every tick. |
| +- **Registry**: one additive column, |
| + `ALTER TABLE routes ADD COLUMN caps TEXT NOT NULL DEFAULT ''` — the |
| + granted tokens, comma-separated (`port:25,host-role`), written only by the |
| + narrow setter behind `carlos route -grant/-revoke`. Default `''` = nothing |
| + granted = today's behaviour for every row already on every box. It is |
| + deliberately **not** added to `Fingerprint()`: grants change unit shape, |
| + not routing, and the edge has no reason to reload for one. |
| +- **Drop-ins**, generated, root-owned, `0644`: |
| + |
| +``` |
| +/etc/systemd/system/carlos-app@<host>.service.d/10-carlos-capabilities.conf |
| +/etc/systemd/system/carlos-app@<host>.socket.d/10-carlos-capabilities.conf |
| +/etc/systemd/system/carlos-app@<host>.service.d/90-operator.conf ← never written or read by the platform |
| +``` |
| + |
| +Each generated file opens with `# GENERATED by carlos adopt — do not |
| +hand-edit`, the marker `carlos ops litestream` already uses. A file at that |
| +path **without** the marker is never overwritten and never deleted; it is |
| +reported and skipped. `10-` sorts before `90-`, so an operator drop-in wins |
| +every conflict, which is the escape hatch for anything this vocabulary |
| +cannot say. |
| + |
| +One generated file per unit, not one per capability. Removal semantics are |
| +the reason: a capability disappearing from a manifest is then one rewrite of |
| +one file, with no orphan-file reaping pass to get wrong, and "which |
| +capability wrote this line" is answered by the manifest, which is on disk |
| +next to the binary. |
| + |
| +## 4. How capabilities materialize |
| + |
| +### 4a. One template, drop-ins per instance |
| + |
| +`internal/host/units/carlos-app@.service` and `carlos-app@.socket` stay |
| +exactly as they are — one template, no variants, no per-app unit files. Every |
| +capability is expressed as directives in the per-instance drop-in. Rendered |
| +deterministically (fixed directive order, sorted lists) so an unchanged pass |
| +writes no bytes, triggers no `daemon-reload`, and restarts nothing. |
| + |
| +Worked example — `carlos-app@keymail.dev.service.d/10-carlos-capabilities.conf`: |
| + |
| +```ini |
| +# GENERATED by carlos adopt — do not hand-edit |
| +[Service] |
| +ExecStart= |
| +ExecStart=/opt/carlos/live/%i relay |
| +RuntimeDirectory=keymail |
| +RuntimeDirectoryMode=0700 |
| +RuntimeDirectoryPreserve=restart |
| +KillMode=control-group |
| +TasksMax=1024 |
| +MemoryMax=1G |
| +DynamicUser=no |
| +User=carlos-keymail |
| +Group=carlos-keymail |
| +``` |
| + |
| +and `carlos-app@keymail.dev.socket.d/10-carlos-capabilities.conf`: |
| + |
| +```ini |
| +# GENERATED by carlos adopt — do not hand-edit |
| +[Socket] |
| +ListenStream=25 |
| +``` |
| + |
| +console's is three lines plus the header: the template's `ExecStart` already |
| +says `serve`, `daemon` adds no directives (it is enable/start/restart |
| +behaviour, not unit text), and only `host_role` writes anything — |
| +`DynamicUser=no`, `User=carlos-console`, `Group=carlos-console`. |
| + |
| +The empty `ExecStart=` before the real one is systemd's documented way to |
| +reset a list-valued directive from a drop-in; without it the unit would have |
| +two ExecStart lines and refuse to start. |
| + |
| +### 4b. `host_role`: a static user, and why |
| + |
| +`DynamicUser=yes` mints an ephemeral uid per start. The box's IMDS gate is |
| +an nftables rule keyed on `skuid`, so an ephemeral uid can never be on its |
| +allowlist — and the observed consequence is not an error but silence |
| +(keymail's restore came up empty and served). Matching the rule on the |
| +unit's cgroup instead was considered and rejected: nftables resolves a |
| +`cgroupv2` path to a numeric id when the rule is loaded, and every service |
| +restart creates a new cgroup id, so the rule would go stale on the first |
| +restart — a firewall that silently stops matching is worse than no firewall. |
| + |
| +So `host_role` drops `DynamicUser` and runs the unit as a platform-created |
| +system user `carlos-<app>` (locked, no home, `nologin`, created idempotently |
| +by the adopt pass exactly as `ensureServiceUser` creates `carlos`). The name |
| +is derived by the platform from the app name, never taken from the manifest; |
| +an app name long enough to overflow the 32-character account limit is |
| +refused at ship time. |
| + |
| +The nft allowlist entry for that uid is **not** written by the platform. The |
| +firewall is operator territory (and classifier-blocked for agents); the |
| +adopt pass prints the exact command once per pass while the grant exists and |
| +the uid is absent from the gate. Naming the missing step every hour is the |
| +whole lesson of F1. |
| + |
| +**Pinned sharp edge:** with `DynamicUser=yes`, systemd materialises |
| +`StateDirectory` under `/var/lib/private/` with a symlink at |
| +`/var/lib/carlos-app/<host>`; with a static `User=` it expects a real |
| +directory. Flipping `host_role` on an app whose state dir already exists in |
| +the other shape is therefore **refused in place** — the converge pass logs |
| +and skips, and the operator does it as a stop → move → start. Neither |
| +console nor keymail is a DynamicUser tenant today, so both migrate directly |
| +onto the static shape with no conversion. |
| + |
| +### 4c. `ports`: socket units, never `CAP_NET_BIND_SERVICE` |
| + |
| +The port is bound by **systemd, as root, in the socket unit**, and the |
| +already-bound fd is handed to the tenant. The tenant never holds a |
| +capability. |
| + |
| +Rejected alternative: `AmbientCapabilities=CAP_NET_BIND_SERVICE` on the |
| +service. It is one line and needs no app change, and it is the wrong grant: |
| +the capability is "bind any port below 1024", on a box where the edge owns |
| +`:80` and `:443`. A tenant that restarts at the wrong moment — or is |
| +compromised — could bind the edge's ports and serve the fleet's traffic. |
| +"Port 25" and "every privileged port" are not the same request, and only one |
| +of them is what keymail asked for. Reopening the template's |
| +`CapabilityBoundingSet=` per tenant is not a trade this platform should make |
| +to save an app one flag. |
| + |
| +Socket activation is also the house pattern already: `carlos-app@.socket` |
| +exists precisely so root binds and the tenant inherits, "the simplest design |
| +that gets the PERMISSIONS right". `ports` adds `ListenStream=` lines to that |
| +same per-instance socket unit rather than inventing a second family of unit |
| +files. |
| + |
| +Two operational consequences, both good: the listener is bound at boot |
| +before the service starts, so an MX restart never refuses a connection (it |
| +queues in the backlog) — strictly better than today, where a relay restart |
| +drops `:25` entirely; and a connection to `:25` can start a stopped unit, |
| +though `daemon` means it should never be stopped. |
| + |
| +**App-side contract**: the app enumerates `LISTEN_FDS` (from fd 3) and |
| +identifies each listener by its own local address — a `*net.TCPAddr` on port |
| +25 is the MX, a `*net.UnixAddr` is the HTTP upstream. Deliberately **not** |
| +an ordering or `FileDescriptorName` contract: `FileDescriptorName=` names a |
| +whole socket unit, not an individual `ListenStream=`, so a name could not |
| +distinguish two listeners in one unit, and an order contract would be a |
| +silent-corruption bug the first time a drop-in is re-rendered. Matching on |
| +the address the kernel reports cannot drift. |
| + |
| +For keymail this is one flag, the sibling of K0's `-public-origin`: accept |
| +the MX listener from `LISTEN_FDS` when present, keep binding `:25` when not. |
| +Until it lands, keymail migrates everything except `ports` — see §5. |
| + |
| +### 4d. Where the pass runs, and in what order |
| + |
| +`/etc/systemd/system` is root, so this belongs to `RunAdopt` |
| +(`carlos-adopt.service`, hourly plus the operator's "converge now") — the |
| +same placement-follows-capability argument that keeps the unit half of |
| +config sync there and the exec half in the agent. `RunAdopt` becomes: |
| + |
| +1. `appconfig.Sync` (unchanged, first: a route's env exists before its |
| + process starts). |
| +2. **Capability converge** for every adoptable unit route, from the |
| + **staged manifest of its live version** — repairs drift: a deleted |
| + drop-in, a fresh grant, a revoked grant, a unit that is not enabled, a |
| + daemon that is not running. |
| +3. `release.Pass` (unchanged in shape). Inside `adoptHost`, on a version |
| + change, the order is now: `ensureStaged` (binary + manifest) → **converge |
| + capabilities for the new version** → `PointLive` → restart. A restart |
| + failure rolls back the symlink *and* restores the previous drop-ins, then |
| + poisons the version for the rest of the pass, exactly as today. |
| +4. Advisories (`adviseBackingMismatches`, plus one new line — §5). |
| + |
| +Both entry points call the same converge function; step 2 is what a *grant* |
| +change moves through, step 3 is what a *version* change moves through, and |
| +running both is idempotent. |
| + |
| +**The converge pass never removes a drop-in because a manifest is |
| +unreadable.** Absent or unparseable staged manifest ⇒ leave the drop-ins |
| +exactly as they are and log, every pass. The same three-state rule |
| +`appconfig.Sync` uses (converge / never-configured / failed-leave-alone), |
| +for the same reason: a bucket hiccup must not strip `:25` off a running mail |
| +server. A capability is only removed when a manifest that *actually |
| +resolved* no longer declares it, or a grant was revoked. |
| + |
| +Restart ownership: `restartRoute` keeps `systemctl try-restart` for every |
| +route it handles today — byte-identical behaviour — and uses `restart` for |
| +`daemon` routes, so a daemon that crashed into a stopped state comes back on |
| +the new version instead of being left down by a try-restart no-op. |
| + |
| +### 4e. Backing, and the rails around it |
| + |
| +Capabilities are honoured only for `Kind == "instance"`, `Backing == |
| +"unit"`, `!Platform` — the same filter `release.adoptable` and |
| +`appconfig.syncable` already apply. So both apps become `Backing="unit"` |
| +routes, and everything the unit path already provides (env files, restart on |
| +config change, per-unit metrics, the `-db` replication hook) arrives with |
| +them. |
| + |
| +- A **platform alias** route (`<app>.<sqid>.<apps-domain>`) never |
| + materialises anything. An alias is a second name for one process; a second |
| + `carlos-app@` unit claiming `:25` would break the real one. `adoptable` |
| + already excludes them; this design depends on that and says so. |
| +- Additional hostnames for one daemon are `kind=service` proxy routes, never |
| + a second `kind=instance` route. `status.keymail.dev` is exactly this case: |
| + one relay process, two names, one unit. |
| +- An **exec-backed** route (`Backing == ""`) whose release declares |
| + capabilities materialises nothing and gets an advisory line — the same |
| + shape as the issue #14 backing-mismatch advisory, and the guided path |
| + through the migration in §5. |
| +- `daemon` + `hibernate` is refused by `carlos add` / `carlos route`. A |
| + daemon the platform is asked to keep running and to stop when idle is a |
| + contradiction, and hibernation for these apps means an MX that sleeps. |
| +- `ports` conflicts are refused before anything is written: a port already |
| + claimed by another route's drop-in on this box, or in the reserved set |
| + (§7), does not materialise. |
| + |
| +## 5. Migration |
| + |
| +Both migrations are per box, independent, and reversible. Neither is |
| +literally zero-downtime — a single-writer SQLite app cannot have two |
| +processes on one database, so the cutover is stop → copy → start. The |
| +honest number is **seconds**, and the design makes the risky part (the data) |
| +a copy rather than a move, so rollback has a body to return to. |
| + |
| +### 5a. Prerequisites (both apps) |
| + |
| +1. Platform binary with capability support on the box first |
| + (`update-platform-binary.sh` — it already re-stages the unit family from |
| + `carlos bootstrap -root /tmp/carlos-unitstage`). Deploy order is not |
| + optional: an older binary meeting a capability-bearing manifest refuses |
| + the version (§3b). |
| +2. Grants staged on the route (`carlos route -grant …`). |
| +3. For `host_role`: the operator's nft allowlist entry for the new |
| + `carlos-<app>` uid, and the same uid added to `harden-host.sh` so a |
| + hardening re-run does not strip it (F1's second half). |
| + |
| +### 5b. console |
| + |
| +console is the thing that ships and promotes, so the chicken-and-egg |
| +deserves naming — and then defusing: |
| + |
| +- **The restarter is not the console.** Adoption runs in `carlos adopt` |
| + (root, `carlos-adopt.service`) or in the agent, never inside the console |
| + process. `systemctl restart carlos-app@platform.carlosframework.com` |
| + restarting the console is no more circular than the SSM script that does |
| + it today. |
| +- **A promote in flight is a CAS write.** Worst case for a restart landing |
| + mid-promote is that the pointer was not written; nothing is half-written, |
| + because `signPointer` + `Put(…, etag)` is atomic. |
| +- **A console build that will not boot cannot be rolled back from the |
| + console.** The escape hatch is and stays the CLI: `carlos rollback` over |
| + SSM, which repoints and — on `canary/console`, which has no bake window — |
| + is picked up by the canary fast loop within seconds. |
| +- **Once the platform owns the restart, promoting console *is* deploying |
| + console.** Today an operator chooses the moment by running the restart by |
| + hand. That choice moves into the promotion. On `canary/console` with zero |
| + hold that is immediate; it is the right default and it should be a |
| + deliberate one (open question 3). |
| + |
| +Steps, per box: |
| + |
| +1. `carlos env set -app console …` — move console's environment from the |
| + bespoke unit's `EnvironmentFile` into the bucket, including |
| + `CONSOLE_DB=/var/lib/carlos-app/<host>/console.db`. Nothing reads it yet. |
| +2. Ship a console release whose manifest declares `daemon`, `data` |
| + (`replicate: platform`, `dbs: ["console.db"]`), `host_role`. Promote it |
| + to `canary/console`. The route is still exec-backed, so the box stages |
| + the binary, changes nothing about the running unit, and logs the |
| + capability/backing advisory. This is the reversible dress rehearsal. |
| +3. Cutover (one operator session, seconds): |
| + `systemctl disable --now console.service` → copy `console.db`, |
| + `-wal`, `-shm` to `/var/lib/carlos-app/<host>/`, owned `carlos-console`, |
| + `0700` → `carlos route -backing unit <host>` (a narrow setter; a |
| + `carlos add` re-upsert is the wrong tool — it resets the channel, the |
| + issue #14 lesson) → `carlos adopt` (writes the drop-ins, creates the |
| + user, enables and starts `carlos-app@<host>.service`) → verify: unit |
| + active, `/healthz`-equivalent through the edge, sign-in round trip. |
| +4. Remove `/etc/systemd/system/console.service`. Confirm the per-unit |
| + metrics table now lists `carlos-app@<host>.service`. |
| + |
| +Rollback: stop and disable the templated unit, `carlos route -backing ""`, |
| +re-enable `console.service` (its data is untouched at `/data/console`). |
| +Console writes made after cutover are lost by rolling back — keep the window |
| +short, the same posture the keymail cutover plan takes about its fork |
| +window. |
| + |
| +### 5c. keymail relay |
| + |
| +Gated on one keymail change: accept the MX listener from `LISTEN_FDS` |
| +(§4c). Until it merges, keymail migrates **without** `ports` — which means |
| +`:25` cannot move, which means the relay cannot move. There is no half |
| +migration for a single process, and pretending otherwise would put an MX |
| +gap into a plan (open question 1). |
| + |
| +Steps, once the flag is merged and shipped: |
| + |
| +1. `carlos env set -app keymail …` — `KEYMAIL_RELAY_ADDR=127.0.0.1:8446`, |
| + `KEYMAIL_HOST`, `KEYMAIL_RP_ID`, `KEYMAIL_PUBLIC_ORIGIN`, |
| + `KEYMAIL_DATA=/var/lib/carlos-app/keymail.dev`, |
| + `KEYMAIL_SOCK_DIR=/run/keymail`, the bucket names, the SES pair; |
| + `carlos secrets set` for the Google client secret. |
| +2. Ship a release declaring `command: relay`, `daemon`, `children`, |
| + `runtime_dir: keymail`, `ports: [25]`, `data: {replicate: "app"}`, |
| + `host_role`. Promote on the canary channel. Nothing changes yet. |
| +3. Grants: `port:25`, `host-role`. Operator: nft allowance for |
| + `carlos-keymail`. |
| +4. Cutover: `systemctl disable --now keymail-relay.service` (releases `:25`) |
| + → `carlos route -backing unit keymail.dev` → `carlos adopt` (the socket |
| + unit binds `:25` immediately — inbound connections queue rather than |
| + bounce) → copy `/opt/keymail/data` to `/var/lib/carlos-app/keymail.dev`, |
| + owned `carlos-keymail`, `0700` → start. Verify **before declaring |
| + success**: the `backup: restored …` log lines, a federation lookup 200, |
| + the instance children present, `/run/keymail/*.sock` populated. F4's |
| + restore-succeeds-empty failure mode makes "the unit is active" an |
| + insufficient check. |
| +5. `status.keymail.dev` becomes (or stays) a `kind=service` proxy route to |
| + the same upstream. |
| + |
| +Rollback: stop the templated unit, `carlos route -backing ""`, re-enable |
| +`keymail-relay.service` — its data is still at `/opt/keymail/data` and its |
| +own embedded litestream resumes from local files. Mail accepted during the |
| +window exists only in the streamed lineage; short window, quiet hour, same |
| +caveat the cutover plan already carries. |
| + |
| +### 5d. What the infra repo loses |
| + |
| +- `scripts/rollout-status-flagship.sh` — the `systemctl restart |
| + console.service` / `is-active` pair, and the two-guess |
| + `CONSOLE_DB`/binary-path fallback for the operator grant, whose existence |
| + is itself the argument for this design. |
| +- `scripts/rehearse-keymail.sh` — the unit heredoc, the `useradd`, the |
| + `install -d` calls for the data and runtime dirs, and the hand-written |
| + `EnvironmentFile`; a rehearsal becomes a route plus a channel. |
| +- `scripts/harden-host.sh` — the per-unit hardening drop-in retrofit for |
| + these two units; the template already carries the sandbox. The IMDS uid |
| + allowance stays (it is a firewall, not a unit shape) and gains the |
| + `carlos-<app>` uids. |
| +- The `flagship/README.md` prose that has to remember what is running. |
| + |
| +What does **not** shrink: `ship-app.sh` / `promote-app.sh` / |
| +`promote-console.sh`. Shipping and promoting are orthogonal to unit shape, |
| +and the signing key keeps coming from SSM inside the script. |
| + |
| +## 6. What monitoring gets for free |
| + |
| +This design exists because `keymail.dev` rendered "asleep" on its owner page |
| +while it was serving mail. Once both apps are `carlos-app@<host>.service` |
| +instances, with no change at all to the monitoring code: |
| + |
| +- `listCarlosUnits` finds them — instantiated units land in |
| + `system.slice/system-carlos\x2dapp.slice/`, which the sampler already |
| + scans one level down for. Both appear in the operator per-unit table with |
| + real CPU, memory, and io. |
| +- `OwnerAppMetrics` matches `carlos-app@<host>.service` exactly, derived |
| + from the app's own routes filtered by account. keymail.dev's owner page |
| + shows per-instance readings instead of "no per-instance readings". |
| +- keymail's mailbox children are inside the unit's cgroup, so the family |
| + reports as one honest row — relay plus 18 instances, attributed to the app |
| + that spawns them, which is precisely the noisy-neighbour question the |
| + per-unit table exists to answer. |
| +- `InstanceMetrics.Hibernate` is false for a daemon, so absence renders as |
| + the state-neutral "no per-instance readings" rather than a sleep claim |
| + nobody observed — and with `daemon` the platform now knows that absence is |
| + a **fault**, not a state. Turning that into an alert is a follow-on (the |
| + monitoring spec puts per-unit alerts out of v1); the data is now there to |
| + do it. |
| +- `ops.FleetDBs` learns console's database from `data.dbs` — stamped onto |
| + the route's `DBPath` at converge time — instead of depending on an |
| + operator having typed `-db` at `carlos add`, which the current code |
| + documents as "silently replicated by nobody". |
| +- The backing-mismatch and external-socket advisories stop firing for these |
| + two hosts, because the thing they were warning about is fixed. |
| +- Restart on adopt is the platform's, in the documented promote → adopt → |
| + restart order, on every box, every pass — not a line in an SSM script. |
| + |
| +## 7. Security posture |
| + |
| +The manifest is **tenant input**. Anyone who can ship to an app writes it. |
| +The design assumes a malicious one and holds by construction: |
| + |
| +- **No systemd directives from manifests.** The vocabulary is seven names; |
| + the directives they map to are Go string constants in this repo. There is |
| + no passthrough, no `extra`, no escape hatch — the escape hatch is |
| + `90-operator.conf`, which only root can write and which the platform never |
| + reads. |
| +- **No paths.** `data` cannot name a directory; the state dir is always |
| + `/var/lib/carlos-app/<host>`. A manifest saying `"path": "/etc"` has |
| + nowhere to say it, so the platform can never be induced to chown, create, |
| + or expose a path of the tenant's choosing. `data.dbs` entries are bare |
| + filenames, validated against `^[A-Za-z0-9._-]{1,64}$`, rejected on `.`, |
| + `..`, or any separator, and used only as `filepath.Join(stateDir, name)`. |
| +- **No users, uids, or groups.** `host_role` runs as `carlos-<app>`, derived |
| + by the platform. A manifest cannot ask to be `root`, `carlos`, or another |
| + tenant's user. |
| +- **No numbers.** `children` selects a platform-chosen tier; |
| + `TasksMax=infinity` and `MemoryMax=infinity` are unsayable. A tenant |
| + cannot size itself into a box-wide denial of service. |
| +- **`runtime_dir` is namespace-guarded twice.** Charset |
| + `^[a-z][a-z0-9-]{0,31}$` (so `../etc`, `carlos/apps`, and absolute paths |
| + do not parse), a reserved list that includes `carlos` — a tenant owning |
| + `/run/carlos` would own every tenant socket the edge dials, the single |
| + most valuable directory on the box — plus `systemd`, `lock`, `log`, |
| + `user`, `dbus`, `sshd`, `udev`, and anything matching `systemd-*`; and a |
| + box-wide uniqueness check, so the second app to claim `keymail` is refused |
| + rather than handed a directory another app's sockets live in. |
| +- **`ports` is grant-gated, conflict-checked, and reserve-guarded.** `22`, |
| + `80`, `443` are reserved absolutely and are **not grantable** — an |
| + operator typo cannot hand a tenant the edge's ports or the box's SSH. |
| + Ports already claimed by another route on this box are refused. Only |
| + privileged ports are accepted at all: a high port needs no help from the |
| + platform, and letting the vocabulary cover them would blur what the |
| + capability is for. |
| +- **`host_role` is the one real transfer of authority, and it is |
| + operator-only, per box.** It hands the app the box's IAM role. It cannot |
| + be self-granted, its grant lives in the box's registry rather than in |
| + anything a tenant can write, and the firewall entry that actually makes it |
| + work is a separate operator action the platform only ever *prints*. Two |
| + independent humans-in-the-loop for the capability that matters most. |
| +- **A grant is not a promise.** Revoking `port:25` rewrites the drop-in and |
| + restarts the unit on the next pass; there is no path by which a manifest |
| + re-grants itself. |
| +- **Nothing here touches the console's gates.** Session → CSRF → owner |
| + (byte-identical 404) → `requireFreshAuth` last, unchanged. The console |
| + gains, at most, a read-only rendering of which capabilities a route has |
| + been granted; grants are written by `carlos route` on the box. |
| + |
| +Residual risk, stated plainly: `host_role` means a compromised console or |
| +relay binary has the box's IAM role — as both do today, because that is how |
| +they already work. This design does not widen it; it makes it a named, |
| +grantable, revocable, per-box thing instead of a property of a unit file |
| +nobody has a copy of. |
| + |
| +## 8. Testing |
| + |
| +- Pure-function tests for capability validation (unknown name refused at |
| + ship, unknown field ignored, `"v"` bump refused, charset and reserved-name |
| + refusals for `runtime_dir`, reserved and non-privileged port refusals, |
| + `daemon`+`hibernate` refusal). |
| +- Golden-file tests for the rendered drop-ins — console's and keymail's, |
| + byte for byte — plus a determinism test (render twice, identical bytes, |
| + no write, no `daemon-reload`). |
| +- A parses-yesterday's-manifest test: a manifest with no `capabilities` key |
| + round-trips and yields no capabilities, joining `golden_test.go`'s |
| + existing byte-identical guarantees. |
| +- Converge-pass tests at the `Systemctl`-style seam the activator already |
| + uses, so nothing touches systemd: grant added → drop-in written and unit |
| + restarted; grant revoked → drop-in rewritten; manifest unreadable → |
| + drop-ins untouched, logged; hand-written drop-in (no marker) → never |
| + overwritten; version change → capabilities converge before `PointLive`, |
| + and a restart failure restores both the symlink and the previous drop-ins. |
| +- Port-conflict and `runtime_dir`-collision tests across two routes on one |
| + box. |
| +- A `laddercheck_test.go`-style assertion that no path in this design |
| + consults or constrains channel names — `canary/*` in, `canary/*` out. |
| +- `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l` |
| + clean before anything lands. |
| + |
| +## 9. Non-goals |
| + |
| +- Free-form systemd directives, free argv, or per-app resource numbers from |
| + a manifest. Ever, not "not yet". |
| +- Capabilities for exec-backed routes. They get an advisory, not a |
| + materialization. |
| +- Hibernation for daemon apps. |
| +- Multi-box placement. Which box runs a daemon is still the route row's |
| + answer, as it is today. |
| +- Agent-managed sidecars for unit tenants (`UnitBackend` refuses them; that |
| + is unchanged). |
| +- UDP sockets, non-TCP protocols, and edge termination for anything but |
| + HTTP. `:25` stays a direct bind because the edge does not speak SMTP. |
| +- Automatic firewall/IMDS allowlisting. The platform prints the command. |
| +- Cross-tenant shared data directories, or one static user shared by two |
| + apps. |
| +- Migrating any *third* app. The vocabulary was derived from two; the next |
| + app that needs something absent from it is a design conversation, not an |
| + `extra` field. |
| + |
| +## 10. Open questions for Paul |
| + |
| +1. **keymail's `LISTEN_FDS` change is the gate on the whole keymail |
| + migration.** Take it (recommended — sibling of K0's `-public-origin`, |
| + and it also removes the MX gap on restart), or accept an interim |
| + `AmbientCapabilities=CAP_NET_BIND_SERVICE` grant knowing it means "any |
| + privileged port" on the box that owns `:443`? |
| +2. **`carlos-<app>` static users and the nft gate.** Accept a per-app system |
| + user plus an operator-run allowlist entry per box, or is there an |
| + appetite for moving the IMDS gate onto something that survives restarts |
| + better than a uid (nothing I found does — cgroup matching goes stale)? |
| +3. **Console: promotion becomes deployment.** On `canary/console` (zero |
| + bake) a promote restarts the console within seconds of the fast-adopt |
| + loop noticing. Keep that, or give console a bake window / an explicit |
| + hold so the restart moment stays a human's? |
| +4. **Grant scope.** Per route, per box (proposed — the resources are |
| + box-local), or per app fleet-wide with the box merely enforcing? |
| +5. **Console's data location.** Move `console.db` into |
| + `/var/lib/carlos-app/<host>/` (proposed — the platform then owns and |
| + replicates it), or keep `/data/console` via an operator drop-in and |
| + accept that the platform does not know where console's database is? |