amadan
spec: fleet-internal engines (#335), rev 1 draft, and its review brief
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+445
−0
docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md+385 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md+60 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.012s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md b/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md| new file mode 100644 |
| index 000000000..abe272058 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md |
| @@ -0,0 +1,385 @@ |
| +# Fleet-internal engines: a doc engine nobody on the internet can call |
| + |
| +**Date:** 2026-10-21 |
| +**Revision:** 1 (draft, not reviewed). |
| +**Discussion:** amadan carlos/platform #335. Related: #328 (per-app egress allow). |
| + |
| +## 1. Intent |
| + |
| +The `doc_engine` capability (`internal/caps/render.go`, README "App |
| +capabilities") fences a conversion engine's **egress** away from private |
| +networks. It says nothing about who can **call** the engine. A doc-engine |
| +route is a route like any other: the engine box's public edge serves its |
| +hostname on :443, any other box backhauls a public request for it, and the |
| +only gate is whatever auth the engine applies itself (Gotenberg: basic auth, |
| +often none). An operator who skips that auth, or leaks its password, |
| +publishes a Chromium and LibreOffice renderer to the internet: free compute |
| +for anyone, and an SSRF engine whose remaining reach is the public internet |
| +and its own box's loopback. |
| + |
| +This spec makes an engine **fleet-internal**. Its hostname answers only |
| +calls the platform carried from an app the operator granted, and an |
| +isolated tenant reaches it through a socket inside its own instance, with no |
| +private address opened in its fence. |
| + |
| +## 2. Goals, non-goals, anti-goals |
| + |
| +**Goals** |
| + |
| +- G1. A registered engine's hostname is refused to every public request, on |
| + every box, on every listener and every path except the ACME challenge. |
| +- G2. An operator grants an app the use of a named engine, per app. Every |
| + instance of the app, on any box and in any unit kind the app's code runs |
| + as, can then call it. A revoke takes effect within the freshness bounds |
| + of §6.4. |
| +- G3. A nested (isolated) instance calls an engine through a unix socket in |
| + its own run directory. Its `IPAddressDeny=` fence is unchanged. |
| +- G4. The engine box knows which app made each call, and refuses a call |
| + from an app without a grant even if the calling box was wrong. |
| +- G5. One app cannot starve the others: the engine box caps each app's |
| + calls in flight. |
| +- G6. Provider-neutral. Nothing here uses a cloud API; see §10. |
| + |
| +**Non-goals** |
| + |
| +- Metering or billing engine use. |
| +- Members requesting a grant. Grants are operator-only, as `carlos store |
| + grant` is. |
| +- A general "internal route" product for arbitrary services. The mechanism |
| + in §5 is not specific to conversion, but the vocabulary, the CLI and the |
| + docs speak only of engines. Widening it is a later spec. |
| +- Reaching an engine over TCP from inside a nested instance. Its fence |
| + denies loopback and every private range, and `IPAddressAllow=` cannot |
| + name a port (#328's limit 1). The socket is the only way in. |
| +- Replacing #328. An operator's engine that lives outside the platform (as |
| + Tito's Gotenberg does) still needs #328's egress allow. |
| + |
| +**Anti-goals** (would make this a failure even if it works) |
| + |
| +- A window in which an engine is public again: a stale mirror, a restarted |
| + edge or a box that cannot reach the bucket must never reopen a hostname |
| + the platform has said is internal (§6.3). |
| +- A caller identity a tenant can forge. Who is calling comes from which |
| + socket the request arrived on, or from a door key, never from a header |
| + the app wrote. |
| +- Opening any private address in an isolated instance's fence. |
| + |
| +## 3. What exists today, and what this builds on |
| + |
| +Cited from main at `3d444a1b`. |
| + |
| +- **The backhaul hop** (`internal/edge/edge.go` `backhaul`/`backhaulTo`, |
| + `homemap.go`): one hop, ever, from the box a request reached to the box |
| + that homes the host. It dials the home box's `private_ip` when region HA |
| + is on and both boxes share a region and placement, and its public address |
| + otherwise (`homemap.go` `dialAddress`), on :443 with public-CA TLS and SNI |
| + set to the real hostname. It is authenticated by `X-Carlos-Hop` and |
| + `X-Carlos-Hop-Auth`, an HMAC-SHA256 over host and timestamp (±30 s) under |
| + the deployment's mesh key `control/edge-mesh-key` (`mesh.go`). **The hop |
| + carries no caller**: the receiving edge knows "some edge in this |
| + deployment" and nothing else (`acceptHop`, `homemap.go`). |
| +- **The edge-local door** (#175 P7, `internal/edge/door.go`): a unix socket |
| + (`CARLOS_EDGE_DOOR`) through which an app's own processes reach the app's |
| + hosts on any box, with an app-scoped door key (`internal/doorkey`). The |
| + key is a console-minted, root-signed, revisioned, expiring record under |
| + `control/door-keys/`, mirrored by every edge every 30 s with a ten-minute |
| + stale bound (`doorkeys.go`). The door marks the request with a context |
| + value, never a header, and `ServeHTTP` scopes it to the caller's own |
| + routes. The mark does not cross a hop. |
| +- **The per-instance control socket** (`internal/edge/purge.go` |
| + `ListenControl`): one listener per host, opened by the edge, with the host |
| + fixed when the listener is built. Nested path |
| + `/run/carlos/instances/<iid>/app.sock.control`; the instance sees it |
| + because every instance unit binds its whole run directory |
| + (`internal/host/units/carlos-instance@.service`, `BindPaths` of |
| + `/run/carlos/instances/%I`). Generations get their own under `g<n>/`. |
| + The app learns the path from `CARLOS_CONTROL_SOCKET`, which tenant config |
| + cannot set (`backend_exec.go`). |
| +- **The doc-engine grant** (`carlos route --host X --grant doc-engine`): |
| + box-local, on a `kind=instance, backing=unit` route, materialized as the |
| + `10-carlos-capabilities.conf` drop-in on `carlos-app@<host>`. The README |
| + asks for a dedicated box. |
| +- **Operator-only per-app grants**: `carlos store grant` is the precedent |
| + (gated by store operators, recorded in the console, projected to the |
| + bucket). |
| + |
| +## 4. The shape, in one paragraph |
| + |
| +An operator **registers** an engine: a name (`gotenberg`) bound to the |
| +hostname of a route that already serves it. The console records it and |
| +publishes a root-signed `control/engines.json` that every edge mirrors. |
| +From then on every edge refuses that hostname to public requests. The |
| +operator **grants** apps the engine by name. A granted instance's code sends |
| +plain HTTP to `$CARLOS_ENGINES_SOCKET` with `Host: gotenberg`. Its box's |
| +edge knows the caller from the socket, checks the grant, and forwards the |
| +request to the engine's home box over the existing backhaul hop, now signed |
| +with the caller's account, app and engine. The engine box verifies the |
| +signature, checks the grant again, applies the per-app concurrency cap and |
| +serves the route. App processes outside an instance (the door's callers) |
| +use the door with `Host: <engine name>` in the same way. |
| + |
| +## 5. The parts |
| + |
| +### 5.1 Registration and grants (console) |
| + |
| +New operator-only CLI, gated like `domains attach --platform`: an owner of |
| +the ops account or an address on `CARLOS_SYSTEM_OPERATORS` |
| +(`Server.systemOperator`). |
| + |
| + carlos engines add <name> --host <hostname> [--per-app <n>] |
| + carlos engines remove <name> |
| + carlos engines grant <name> --app <account>/<app> |
| + carlos engines revoke <name> --app <account>/<app> |
| + carlos engines list |
| + |
| +- `<name>` matches `^[a-z][a-z0-9-]{0,31}$` and is unique in the |
| + deployment. It is what apps put in `Host:`. |
| +- `--host` is the engine route's hostname as the edge serves it: exact, no |
| + wildcard, lower-cased. The console does not check that a box serves it |
| + (the console holds no route registry); §5.4 reports the mismatch from the |
| + box. |
| +- `--per-app` is the engine box's in-flight cap per calling app (§5.5). |
| + Default 4, range 1 to 64. |
| +- `remove` refuses while grants remain. Removing an engine makes its |
| + hostname public again (§6.3), so it is a deliberate two-step act. |
| +- Every change is audited with `store.RecordEvent`, like the store grant. |
| +- `list` shows each engine, its host, its cap and its grantees to an |
| + operator. A member sees only the engines granted to apps in accounts they |
| + belong to, with name and status, never the host or the other grantees. |
| + |
| +The console keeps engines and grants in its own database (the authority) |
| +and publishes the whole set as one record, `control/engines.json`, |
| +signed with the deployment's root release key under its own signing domain |
| +(`carlos engines v1\n`), with a revision and `expires_at` 72 hours out, |
| +exactly as door keys are (`internal/doorkey`). The console re-signs well |
| +inside the lifetime. One file, because a box needs the whole set to decide |
| +G1, and a deployment has a handful of engines. |
| + |
| + { |
| + "v": 1, "revision": 17, "signed_at": "...", "expires_at": "...", |
| + "engines": [ |
| + { "name": "gotenberg", "host": "gotenberg.example.com", "per_app": 4, |
| + "grants": [ { "account": "acct1", "app": "shop" } ] } |
| + ] |
| + } |
| + |
| +### 5.2 The engine mirror (every edge) |
| + |
| +A new mirror beside the door-key mirror, same loop shape and same rules for |
| +a signature, shape, expiry or revision failure (`doorkeys.go`). It feeds two |
| +views with **different failure rules**: |
| + |
| +- **The internal set** (hostnames that must be refused publicly) fails |
| + closed. See §6.3. |
| +- **The grant table** (which app may call which engine) fails closed the |
| + other way: stale or expired means no grants, so calls are refused, never |
| + admitted on old authority. |
| + |
| +### 5.3 Refusing the public (every edge) |
| + |
| +For a host in the internal set, a request that is not an engine call |
| +(§5.4) or the engine app's own door caller gets `unknownHost`, the same |
| +answer as a host nobody serves. Concretely: |
| + |
| +- `ServeHTTP`: after the route lookup and before backhaul, the gate, the |
| + meter, the cache and the dispatch. A box that does not home the host |
| + refuses without dialling. The home box refuses a hop that carries no |
| + verified engine caller (a backhauled public request). |
| +- The :80 handler (`httpFallbackHandler`) serves the ACME challenge for the |
| + host and nothing else: no redirect to https. |
| +- `Allowed` (certificate issuance) is unchanged. The home box needs a |
| + publicly trusted certificate for the hop's TLS (§3). The hostname is not |
| + a secret: certificate transparency publishes it anyway. The refusal is |
| + about serving, not hiding. |
| +- The engine app's own door callers keep today's door rule (their own |
| + routes). The engine's operator can still reach it from their own app's |
| + processes. |
| + |
| +### 5.4 The call path |
| + |
| +**From a nested instance.** For every instance the edge prepares in the |
| +nested layout, it opens `engines.sock` in the instance's run directory: |
| +`/run/carlos/instances/<iid>/engines.sock`, mode 0600, owner carlos. It |
| +opens it beside the control socket, with the same per-host construction |
| +(the host fixed at build time, never read from the request). One socket per |
| +instance, not per generation: the run directory is bound into every unit |
| +kind (run, generation, seed, `carlos run` command, legacy sidecar), so all |
| +of them reach it. Each gets `CARLOS_ENGINES_SOCKET` in its launch |
| +environment, set the way `CARLOS_CONTROL_SOCKET` is, so tenant config cannot |
| +set or override it. The socket exists whether or not the app holds a grant, |
| +so a later grant needs no restart. |
| + |
| +**From the door.** A door caller (`X-Carlos-Door-Key`) whose `Host:` is an |
| +engine *name* in the grant table, granted to the key's app, is an engine |
| +call. Today a door caller's `Host:` is a real hostname; engine names cannot |
| +collide with one because a hostname registered in a deployment always has a |
| +dot and a name never does. |
| + |
| +**On the calling box.** The listener: |
| + |
| +1. strips every client-address header and every `X-Carlos-*` header, as the |
| + door does (`doorRequest`), and presents the box's own address; |
| +2. resolves the caller: the instance's route gives account and app |
| + (`release.RouteAccount`, `Route.App`); |
| +3. reads `Host:` as an engine name, looks it up in the grant table, and |
| + answers 404 `engine not granted` for an unknown name, an ungranted app, |
| + or a stale or expired table (one body for all three); |
| +4. puts an engine-call mark (account, app, engine) on the request context |
| + and rewrites `Host:` to the engine's hostname; |
| +5. serves the engine route locally if this box homes it (allowed, but the |
| + README's dedicated-box advice stands), or backhauls it to the home box. |
| + |
| +**The hop.** An engine call's hop carries a new header, |
| +`X-Carlos-Hop-Caller: <account>/<app>/<engine>`, and its MAC covers it: |
| +`hopMAC` gains a versioned form over `host\nts\ncaller`. The receiving edge |
| +verifies it with the same dual-key rule (`verifyHopEitherKey`), deletes it |
| +with the other hop headers, and turns it into the same engine-call context |
| +mark. A hop without a caller is an ordinary hop and stays exactly as today. |
| +The caller header is the mesh's vocabulary: the door and the engine socket |
| +delete it on the way in, and `acceptHop` deletes it on every request. |
| + |
| +**On the engine box.** An engine call (local or hopped) is admitted to the |
| +engine route only when the mark's engine names this host in the internal |
| +set and the grant table grants it to the mark's app. Otherwise |
| +`unknownHost`. Then §5.5's cap, then today's dispatch. The cache is |
| +skipped for engine calls. |
| + |
| +### 5.5 Limits |
| + |
| +- **Per-app in flight, on the engine box:** at most `per_app` concurrent |
| + calls per calling app, summed across its instances and boxes. Over the |
| + cap: 429 with `Retry-After: 1` and a log line naming the app, rate |
| + limited. The engine box is the one place that sees every caller. |
| +- **Request body:** 64 MiB, refused with 413 at the calling box before the |
| + hop. |
| +- **Time:** the hop and the socket allow a call up to 5 minutes end to end. |
| + Longer work belongs in the engine's async mode, whose webhook reaches the |
| + caller's public URL over the internet (the engine's egress allows the |
| + public internet). |
| + |
| +### 5.6 What the engine sees |
| + |
| +The request as the app sent it, with `Host:` set to the engine's hostname, |
| +the client-address headers replaced by the calling box's address, and the |
| +app's own `Authorization` passed through. An engine that keeps basic auth |
| +keeps it as a second lock. The edge's access log on the engine box records |
| +the caller app for each call. No caller header reaches the engine. |
| + |
| +### 5.7 Box-side advisories |
| + |
| +The hourly capability pass on the engine's home box logs one line per pass |
| +when a registered engine host is served by a route without the |
| +`doc-engine` grant, naming the grant command. It does not refuse: the |
| +internal set protects any route registered, and the grant is about the |
| +engine's own egress. |
| + |
| +## 6. Correctness rules |
| + |
| +### 6.1 Identity |
| + |
| +Who calls is decided by the socket the request arrived on, or by a verified |
| +door key, on the calling box; and by a verified hop caller on the engine |
| +box. No other input names a caller. Every box holds the mesh key, so any |
| +box can sign any caller: a compromised box can call any engine as any app. |
| +That is the trust the hop already places in boxes, and stated here rather |
| +than solved. |
| + |
| +### 6.2 Flat layout |
| + |
| +A flat instance gets **no engine socket**. In the flat layout every |
| +instance runs as uid carlos with the same view of `/run/carlos`, so a |
| +sibling could dial another host's socket and the socket would prove |
| +nothing. A flat app reaches an engine through the door with a door key, |
| +like any app process; the door key says which app it is. (My call; see §11.) |
| + |
| +### 6.3 The internal set never reopens by accident |
| + |
| +- The edge persists the last verified internal set to the box's data |
| + directory and loads it at start, before any listener opens. |
| +- A stale, expired, missing or unverifiable `engines.json` leaves the |
| + internal set as it was. Only a verified record with a higher revision |
| + can remove a host from it. |
| +- A host leaves the set only through `carlos engines remove`, whose record |
| + no longer lists it. |
| +- A box that has never read a verified record has an empty set: nothing it |
| + serves was ever internal on its watch. |
| + |
| +### 6.4 Freshness |
| + |
| +The grant table follows the door-key contract: a grant or revoke lands on |
| +every edge within one 30 s interval of the record; a box that cannot list |
| +the bucket drops every grant ten minutes after its last good read; an |
| +expired record (a console down for 72 hours) drops every grant. The two |
| +edges in a call (calling and engine box) check independently, so a revoke |
| +is in force once either has read it. |
| + |
| +## 7. Deploy order and rollout |
| + |
| +- Every box and the console must run a build carrying this before the first |
| + `carlos engines add`. A box on an older build ignores `engines.json`. If |
| + the old box homes the engine, it keeps serving the host publicly; if it |
| + only receives a public request for it, it backhauls as today and the new |
| + home box refuses the hop (no caller). So **the engine's home box is the |
| + one that must be current**; the runbook checks its version first. |
| +- Registering an engine cuts public callers within 30 s. Move callers onto |
| + the socket or door before `add`, not after. |
| +- Dormant until the first `add`: with no `engines.json`, the internal set |
| + is empty, the grant table is empty, and every engine socket answers 404. |
| +- Roll order as usual: dev boxes, dev console, check, flagship boxes, |
| + flagship console. |
| + |
| +## 8. Tests that prove it |
| + |
| +Production-wired, through real listeners and real hops between two edges in |
| +one test process, not handler fixtures: |
| + |
| +- A public request for a registered host, on the home box and on a |
| + non-home box, gets `unknownHost`, on :443 and :80 (no redirect), and the |
| + ACME challenge path still answers. |
| +- An instance socket call with a grant reaches the engine on another edge; |
| + without a grant it gets 404 and the engine sees nothing. |
| +- A forged `X-Carlos-Hop-Caller` from a client, a door caller or an app |
| + through the socket never survives to the engine box's decision. |
| +- A hop signed with a caller for an ungranted app is refused at the engine |
| + box even though the calling box admitted it (G4: inject the calling |
| + box's table out of date). |
| +- A stale mirror and a restarted edge with no bucket both keep refusing a |
| + registered host (§6.3). |
| +- The per-app cap: the n+1th concurrent call from one app gets 429 while a |
| + second app's call proceeds. |
| +- Every unit kind of a nested instance (run, generation, seed, command) can |
| + reach the socket and sees `CARLOS_ENGINES_SOCKET`; tenant config cannot |
| + set it. |
| + |
| +## 9. Docs |
| + |
| +`docs/site/cli.md`: a `carlos engines` section; the README's "App |
| +capabilities" doc-engine passage gains the public-refusal story and drops |
| +"behind basic auth" as the only gate; a short "Calling an engine from your |
| +app" passage with a Go example using a unix-socket transport and |
| +`http://gotenberg/forms/chromium/convert/html`. All user-facing strings go |
| +through copy review before they ship. |
| + |
| +## 10. Provider boundary |
| + |
| +Core only. The mirror reads a `release.Store` (S3 or a directory); signing |
| +uses the root release key; the hop is the existing mesh. On AWS with region |
| +HA, a hop inside one region uses private addresses; everywhere else it |
| +crosses the public internet under TLS and the hop MAC, as backhaul does |
| +today. A non-AWS deployment loses nothing. "Fleet-internal" means "only the |
| +platform can call it", not "only over a private network". |
| + |
| +## 11. Calls I made (for review) |
| + |
| +- Explicit registration, not "every doc-engine grant is internal". A |
| + doc-engine grant is box-local, and other boxes cannot see it; a |
| + console-side registration gives every box the same answer, and no |
| + existing public engine changes behaviour until an operator registers it. |
| +- Flat instances get no socket and use the door (§6.2). |
| +- Engine selection by `Host:` name on one socket, so a client's base URL is |
| + just `http://<name>`. |
| +- Any route may be registered; a missing doc-engine grant is advised, not |
| + refused (§5.7). |
| +- Limits: 64 MiB body, 5 minutes, per-app cap default 4. |
| +- The engine box re-checks the grant (G4) rather than trusting the calling |
| + box. |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md| new file mode 100644 |
| index 000000000..5314c02b3 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md |
| @@ -0,0 +1,60 @@ |
| +# Review brief: fleet-internal engines (#335) |
| + |
| +You are an independent second reader of a design spec for the CARLOS |
| +platform (this repository). Read it cold and adversarially. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md`, |
| +at the commit named in the prompt. |
| + |
| +## What it claims |
| + |
| +A registered doc engine's hostname is refused to every public request on |
| +every box, and granted apps reach it through a per-instance unix socket (or |
| +the edge-local door), relayed over the existing backhaul hop, which now |
| +signs the caller. Read §2's anti-goals first: the spec fails if an engine |
| +can become public again by accident, if a tenant can forge a caller, or if |
| +an isolated instance's fence opens. |
| + |
| +## What to check |
| + |
| +Check against the code on `main`, not the spec's description of it. Cite |
| +`file:line` for every claim about existing code. |
| + |
| +- **The edge:** `internal/edge/edge.go` (`ServeHTTP`, `backhaul`, |
| + `backhaulTo`, `httpFallbackHandler`, `Allowed`, `unknownHost`, the |
| + cache path), `homemap.go` (`acceptHop`, `dialAddress`, `Owner`, |
| + `ExactHome`), `mesh.go` (hop MAC), `door.go`, `doorkeys.go`, |
| + `purge.go` / `purge_listen_unix.go` (`ListenControl`). |
| +- **Instances:** `internal/activator/backend_exec.go` (launch env, |
| + `CARLOS_CONTROL_SOCKET`), `provision.go`, `prepare.go`, |
| + `generations.go`, `cmdunit.go`, `internal/host/units/carlos-instance*@.service`, |
| + `internal/instancectl/gen.go`, `internal/layout/preflight.go`. |
| +- **Grants and records:** `internal/doorkey`, `internal/caps`, |
| + `internal/console/apistore.go` (store grant), `apisystemupdate.go` |
| + (`systemOperator`). |
| + |
| +## Questions to answer |
| + |
| +1. Can a public request reach a registered engine by any path: a spelling |
| + of the host, a wildcard route, the :80 handler, a door caller, a |
| + backhaul from an old box, the login re-resolution, an upgrade, the |
| + cache, a hibernating wake? |
| +2. Can a tenant forge or borrow a caller identity: through headers, through |
| + another instance's socket, across generations, in the flat layout, via |
| + the door? |
| +3. Does §6.3's "never reopens" hold across restarts, stale mirrors, revision |
| + rollback by a bucket writer, and `engines remove`? |
| +4. Can the per-instance socket be built as described for every unit kind, |
| + within the 107-byte socket path limit, and is the "one socket per |
| + instance" claim true of the binds? |
| +5. Does the hop extension stay compatible with boxes on the old build in |
| + both directions? |
| +6. Anything §5.5's limits break (streaming, upgrades, long conversions). |
| + |
| +## Output |
| + |
| +Findings ranked P1 (spec fails a goal or anti-goal, or cannot be built), P2 |
| +(a real gap a builder would hit), P3 (worth fixing). For each: the section, |
| +the failure scenario as concrete steps, the code citation, and a suggested |
| +fix. End with an overall verdict: accept, accept with fixes, or reject. |
| +Do not report wording or style. |