amadan
spec: fleet-internal engines rev 2 (Astra R1 accepted; Paul's rulings R1, R2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+415
−281
docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md+401 −281docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md+14 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m ok github.com/carlosframework/platform 307.721s ? github.com/carlosframework/platform/cmd/changelog [no test files] ok github.com/carlosframework/platform/cmd/console 3.045s ok github.com/carlosframework/platform/cmd/moveprobe 93.265s ok github.com/carlosframework/platform/cmd/zddprobe 0.710s ok github.com/carlosframework/platform/internal/activator 250.574s ok github.com/carlosframework/platform/internal/agentsock 0.015s ok github.com/carlosframework/platform/internal/alerting 0.003s ok github.com/carlosframework/platform/internal/analytics 0.019s ok github.com/carlosframework/platform/internal/appconfig 2.446s ok github.com/carlosframework/platform/internal/apperr 0.071s ok github.com/carlosframework/platform/internal/applog 0.139s ok github.com/carlosframework/platform/internal/awsinv 0.003s ok github.com/carlosframework/platform/internal/boxchoice 0.005s ok github.com/carlosframework/platform/internal/boxid 0.731s ? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files] ? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files] ok github.com/carlosframework/platform/internal/boxvolumes 0.008s ok github.com/carlosframework/platform/internal/caps 0.013s ok github.com/carlosframework/platform/internal/certwarm 0.002s ok github.com/carlosframework/platform/internal/cgroup 0.692s ok github.com/carlosframework/platform/internal/changelog 0.324s ok github.com/carlosframework/platform/internal/claimactivation 0.760s ok github.com/carlosframework/platform/internal/clamd 0.054s ? github.com/carlosframework/platform/internal/cliflag [no test files] amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md b/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md| index abe272058..55770cbd4 100644 |
| --- a/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md |
| @@ -1,69 +1,92 @@ |
| # Fleet-internal engines: a doc engine nobody on the internet can call |
| **Date:** 2026-10-21 |
| -**Revision:** 1 (draft, not reviewed). |
| +**Revision:** 2. |
| **Discussion:** amadan carlos/platform #335. Related: #328 (per-app egress allow). |
| +- **Rev 2** answers Astra round 1 (`CODEX-FINDINGS-ENGINES-R1.md`, reject, |
| + four P1s, all accepted) and the builder check, and records Paul's two |
| + rulings of 2026-10-21 (§2.1). What makes a route internal moves from a |
| + console record to the route itself (R1), which closes R1-3 (a new box |
| + starts public) and R1-2 (aliases and path routes) by construction. The |
| + socket's caller is proved by the peer's cgroup, not the socket's path |
| + (R1-1). The grant table keeps a durable revision floor (R1-4). The door's |
| + owner exception is dropped and the door-to-engine transition is defined |
| + (R1-5). The socket gets an instance-level owner (R1-6) and engine calls |
| + their own transport (R1-7). Each change is marked **(codex R1-n)**. |
| + |
| ## 1. Intent |
| The `doc_engine` capability (`internal/caps/render.go`, README "App |
| capabilities") fences a conversion engine's **egress** away from private |
| networks. It says nothing about who can **call** the engine. A doc-engine |
| -route is a route like any other: the engine box's public edge serves its |
| -hostname on :443, any other box backhauls a public request for it, and the |
| -only gate is whatever auth the engine applies itself (Gotenberg: basic auth, |
| -often none). An operator who skips that auth, or leaks its password, |
| -publishes a Chromium and LibreOffice renderer to the internet: free compute |
| -for anyone, and an SSRF engine whose remaining reach is the public internet |
| -and its own box's loopback. |
| - |
| -This spec makes an engine **fleet-internal**. Its hostname answers only |
| -calls the platform carried from an app the operator granted, and an |
| -isolated tenant reaches it through a socket inside its own instance, with no |
| -private address opened in its fence. |
| +route is a route like any other: its box's public edge serves it on :443, |
| +any other box backhauls a public request for it, and the only gate is |
| +whatever auth the engine applies itself (Gotenberg: basic auth, often |
| +none). An operator who skips that auth, or leaks its password, publishes a |
| +Chromium and LibreOffice renderer to the internet: free compute for anyone, |
| +and an SSRF engine whose remaining reach is the public internet and its own |
| +box's loopback. |
| + |
| +This spec makes every doc engine **fleet-internal**. Its routes answer only |
| +calls the platform carried from an app an operator granted, and an isolated |
| +tenant reaches it through a socket inside its own instance, with no private |
| +address opened in its fence. |
| ## 2. Goals, non-goals, anti-goals |
| **Goals** |
| -- G1. A registered engine's hostname is refused to every public request, on |
| - every box, on every listener and every path except the ACME challenge. |
| +- G1. A route whose live release declares `doc_engine`, or which holds the |
| + `doc-engine` grant, is refused to every public request, through every |
| + hostname, alias, wildcard or path that resolves to it, on every listener, |
| + except the ACME challenge. |
| - G2. An operator grants an app the use of a named engine, per app. Every |
| - instance of the app, on any box and in any unit kind the app's code runs |
| - as, can then call it. A revoke takes effect within the freshness bounds |
| - of §6.4. |
| -- G3. A nested (isolated) instance calls an engine through a unix socket in |
| - its own run directory. Its `IPAddressDeny=` fence is unchanged. |
| -- G4. The engine box knows which app made each call, and refuses a call |
| - from an app without a grant even if the calling box was wrong. |
| -- G5. One app cannot starve the others: the engine box caps each app's |
| - calls in flight. |
| -- G6. Provider-neutral. Nothing here uses a cloud API; see §10. |
| + nested instance of the app, on any box and in every unit kind its code |
| + runs as (run, generation, seed, `carlos run` command, legacy sidecar), |
| + can then call it, and so can the app's processes that hold a door key. A |
| + revoke takes effect within §6.4's bounds. |
| +- G3. A nested instance calls an engine through a unix socket in its own run |
| + directory. Its `IPAddressDeny=` fence is unchanged. |
| +- G4. The engine box knows which app made each call and refuses a call from |
| + an app without a grant, even if the calling box admitted it. |
| +- G5. One app cannot starve the others: the engine box caps each app's calls |
| + in flight. |
| +- G6. Provider-neutral (§10). |
| **Non-goals** |
| -- Metering or billing engine use. |
| -- Members requesting a grant. Grants are operator-only, as `carlos store |
| - grant` is. |
| -- A general "internal route" product for arbitrary services. The mechanism |
| - in §5 is not specific to conversion, but the vocabulary, the CLI and the |
| - docs speak only of engines. Widening it is a later spec. |
| -- Reaching an engine over TCP from inside a nested instance. Its fence |
| - denies loopback and every private range, and `IPAddressAllow=` cannot |
| - name a port (#328's limit 1). The socket is the only way in. |
| -- Replacing #328. An operator's engine that lives outside the platform (as |
| - Tito's Gotenberg does) still needs #328's egress allow. |
| - |
| -**Anti-goals** (would make this a failure even if it works) |
| - |
| -- A window in which an engine is public again: a stale mirror, a restarted |
| - edge or a box that cannot reach the bucket must never reopen a hostname |
| - the platform has said is internal (§6.3). |
| -- A caller identity a tenant can forge. Who is calling comes from which |
| - socket the request arrived on, or from a door key, never from a header |
| - the app wrote. |
| +- **Unit-backed callers** (`carlos-app@`, DynamicUser). Ruling R2: later, if |
| + a caller appears. They have no instance run directory and cannot be |
| + enrolled for the door. Their routes can still *be* engines. |
| +- **Flat-layout instances as socket callers** (§6.2). A flat app uses the |
| + door. |
| +- Metering or billing engine use. Member-requested grants. |
| +- A general internal-route product for arbitrary services. |
| +- Reaching an engine over TCP from a nested instance: its fence denies |
| + loopback and every private range, and `IPAddressAllow=` cannot name a |
| + port (#328's limit 1). |
| +- Replacing #328. An engine outside the platform still needs #328's |
| + egress allow. |
| + |
| +**Anti-goals** |
| + |
| +- A window in which an engine route is public: a new or replacement box, a |
| + restarted edge, a stale mirror or a box that cannot reach the bucket must |
| + never serve one to the public. |
| +- A caller identity a tenant can forge or borrow: through a header, another |
| + instance's socket, a sibling's process or the door. |
| - Opening any private address in an isolated instance's fence. |
| +### 2.1 Rulings (Paul, 2026-10-21; do not reopen) |
| + |
| +- **R1. Declaring `doc_engine` makes a route internal.** No opt-out and no |
| + separate registration step for it. A doc-engine route in public use today |
| + goes dark when its home box takes this build; its callers move to the |
| + socket or door first (§7). |
| +- **R2. Unit-backed apps are not callers in this spec.** Later, if a caller |
| + appears. |
| + |
| ## 3. What exists today, and what this builds on |
| Cited from main at `3d444a1b`. |
| @@ -71,60 +94,113 @@ Cited from main at `3d444a1b`. |
| - **The backhaul hop** (`internal/edge/edge.go` `backhaul`/`backhaulTo`, |
| `homemap.go`): one hop, ever, from the box a request reached to the box |
| that homes the host. It dials the home box's `private_ip` when region HA |
| - is on and both boxes share a region and placement, and its public address |
| + is on and both boxes share a region and placement, its public address |
| otherwise (`homemap.go` `dialAddress`), on :443 with public-CA TLS and SNI |
| - set to the real hostname. It is authenticated by `X-Carlos-Hop` and |
| - `X-Carlos-Hop-Auth`, an HMAC-SHA256 over host and timestamp (±30 s) under |
| - the deployment's mesh key `control/edge-mesh-key` (`mesh.go`). **The hop |
| - carries no caller**: the receiving edge knows "some edge in this |
| - deployment" and nothing else (`acceptHop`, `homemap.go`). |
| + set to the real hostname, with a 30 s response-header timeout and a |
| + transport shared per destination (`edge.go:2825`, `:2938`). It is |
| + authenticated by `X-Carlos-Hop` and `X-Carlos-Hop-Auth`, an HMAC-SHA256 |
| + over host and timestamp (±30 s) under the deployment's mesh key |
| + (`mesh.go`). **The hop carries no caller** (`acceptHop`). |
| - **The edge-local door** (#175 P7, `internal/edge/door.go`): a unix socket |
| (`CARLOS_EDGE_DOOR`) through which an app's own processes reach the app's |
| - hosts on any box, with an app-scoped door key (`internal/doorkey`). The |
| - key is a console-minted, root-signed, revisioned, expiring record under |
| + hosts on any box with an app-scoped door key (`internal/doorkey`): a |
| + console-minted, root-signed, revisioned, expiring record under |
| `control/door-keys/`, mirrored by every edge every 30 s with a ten-minute |
| - stale bound (`doorkeys.go`). The door marks the request with a context |
| - value, never a header, and `ServeHTTP` scopes it to the caller's own |
| - routes. The mark does not cross a hop. |
| + stale bound and an in-memory revision floor (`doorkeys.go:22`). The door's |
| + mark is a context value; it does not cross a hop. |
| - **The per-instance control socket** (`internal/edge/purge.go` |
| - `ListenControl`): one listener per host, opened by the edge, with the host |
| - fixed when the listener is built. Nested path |
| - `/run/carlos/instances/<iid>/app.sock.control`; the instance sees it |
| - because every instance unit binds its whole run directory |
| - (`internal/host/units/carlos-instance@.service`, `BindPaths` of |
| - `/run/carlos/instances/%I`). Generations get their own under `g<n>/`. |
| - The app learns the path from `CARLOS_CONTROL_SOCKET`, which tenant config |
| - cannot set (`backend_exec.go`). |
| -- **The doc-engine grant** (`carlos route --host X --grant doc-engine`): |
| - box-local, on a `kind=instance, backing=unit` route, materialized as the |
| - `10-carlos-capabilities.conf` drop-in on `carlos-app@<host>`. The README |
| - asks for a dedicated box. |
| -- **Operator-only per-app grants**: `carlos store grant` is the precedent |
| - (gated by store operators, recorded in the console, projected to the |
| - bucket). |
| + `ListenControl`), opened per generation for the run role only |
| + (`prepare.go:111`) and closed with the generation (`unitproc.go:203`). |
| + Every nested unit binds its whole run directory |
| + `/run/carlos/instances/<iid>` (`carlos-instance@.service:53`; generations |
| + through `instancectl/gen.go:48`). Every nested unit kind runs in |
| + `Slice=carlos-instances-<iid>.slice` (`instancectl.SliceName`; the four |
| + templates and the generation drop-in). The instance ID is at most 40 |
| + bytes (`provision.go:2918`). |
| +- **Peer checks on a unix socket:** the root control service accepts a |
| + connection only from SO_PEERCRED uid carlos and the edge's MainPID |
| + (`instancectl/server.go` `PeerOK`). |
| +- **Capabilities:** the release's manifest declares them; adoption stages |
| + the manifest beside the live version (`release/adopt.go` |
| + `stageAuthorized`, `capability_pass` reads it via |
| + `stagedManifestOfLiveVersion`); grants are box-local in `routes.caps` |
| + (`registry.SetCaps`), set with `carlos route --host X --grant doc-engine`. |
| +- **Flat tenants share uid carlos** with nested ones and can write their |
| + siblings' directories (`layout/preflight.go:181`). |
| ## 4. The shape, in one paragraph |
| -An operator **registers** an engine: a name (`gotenberg`) bound to the |
| -hostname of a route that already serves it. The console records it and |
| -publishes a root-signed `control/engines.json` that every edge mirrors. |
| -From then on every edge refuses that hostname to public requests. The |
| -operator **grants** apps the engine by name. A granted instance's code sends |
| -plain HTTP to `$CARLOS_ENGINES_SOCKET` with `Host: gotenberg`. Its box's |
| -edge knows the caller from the socket, checks the grant, and forwards the |
| -request to the engine's home box over the existing backhaul hop, now signed |
| -with the caller's account, app and engine. The engine box verifies the |
| -signature, checks the grant again, applies the per-app concurrency cap and |
| -serves the route. App processes outside an instance (the door's callers) |
| -use the door with `Host: <engine name>` in the same way. |
| +A route becomes **internal** on the box that serves it when its live |
| +release declares `doc_engine` or it holds the `doc-engine` grant; the box |
| +records that in its own registry and refuses the route to every public |
| +request. An operator **names** an engine (`gotenberg` → its hostname) and |
| +**grants** it to apps; the console publishes the set as a root-signed |
| +`control/engines.json`. A granted nested instance's code sends plain HTTP |
| +to `$CARLOS_ENGINES_SOCKET` with `Host: gotenberg`. Its box's edge proves |
| +the caller from the connecting process's cgroup, checks the grant, and |
| +relays the request to the engine's home box over the backhaul hop, now |
| +signed with the caller's account, app and engine. The engine box verifies |
| +the signature, checks the grant again, applies the per-app cap and serves |
| +the route. A door caller does the same with `Host: gotenberg`. |
| ## 5. The parts |
| -### 5.1 Registration and grants (console) |
| - |
| -New operator-only CLI, gated like `domains attach --platform`: an owner of |
| -the ops account or an address on `CARLOS_SYSTEM_OPERATORS` |
| -(`Server.systemOperator`). |
| +### 5.1 The internal mark (every box, on its own routes) |
| + |
| +**(codex R1-2, R1-3)** Internal is a property of a route on the box that |
| +serves it, never of a hostname and never of a bucket record. |
| + |
| +- A new registry column `routes.internal` (0 or 1, default 0). |
| +- **Adoption stamps it.** Whenever adoption points a route's live version at |
| + a release, it has that release's manifest in hand; it sets |
| + `internal = 1` if the manifest declares `doc_engine`, and `0` otherwise, |
| + in the same registry write that records the new live version, before the |
| + edge can serve the new version. A version whose manifest adoption cannot |
| + read is not adopted (today's rule for manifests it does not understand |
| + extends to unreadable ones for this column). |
| +- **The grant stamps it too.** `carlos route --grant doc-engine` sets |
| + `internal = 1`; `--revoke doc-engine` leaves it to the next adoption |
| + stamp. A granted route is internal even if its release stopped declaring |
| + the capability. |
| +- **First pass after upgrade.** Routes adopted before this build have no |
| + stamp from adoption. The first capability pass of the new build stamps |
| + each from its staged live manifest. Until that pass has run, the edge |
| + treats a route as internal if it holds the grant; the residual (a route |
| + that declares `doc_engine`, holds no grant, and whose staged manifest is |
| + missing) is an engine running without its JIT and memory band, which |
| + Chromium does not survive, and the pass logs it by name. |
| +- A replacement or new box stamps at its own first adoption of the route, |
| + before it serves it. Nothing it reads from the bucket after that can |
| + unstamp it except a newer adopted release. |
| + |
| +**What the edge does with it** (`ServeHTTP`): |
| + |
| +- The check runs on the **final resolved upstream's route**: after the |
| + route lookup and after the login POST's re-resolution, and before the |
| + import gate, the password gate, the meter, the cache, an upgrade and the |
| + activator's wake. Aliases (`edge.go:929`) and path-mapped hosts |
| + (`edge.go:1364`) copy the target's upstream, so every name and path that |
| + resolves to an internal route is covered, in any spelling the lookup |
| + accepts. |
| +- An internal route admits only an engine call (§5.4). Anything else gets |
| + `unknownHost`. There is no owner exception: the engine app's own |
| + processes reach it by granting the engine to their own app. |
| +- The :80 handler serves the ACME challenge for an internal route's |
| + hostnames and nothing else: no redirect. |
| +- `Allowed` (certificate issuance) is unchanged: the hop's TLS needs a |
| + publicly trusted certificate on the home box. The hostname is not a |
| + secret (certificate transparency publishes it); the refusal is about |
| + serving. |
| +- A box that does not home the host learns nothing new. It backhauls a |
| + public request as today, as an ordinary hop with no caller, and the home |
| + box refuses it. A home box on an older build is the one gap, closed by |
| + deploy order (§7). |
| + |
| +### 5.2 Engine names and grants (console) |
| + |
| +Operator-only CLI, gated like `domains attach --platform` (an owner of the |
| +ops account or an address on `CARLOS_SYSTEM_OPERATORS`, |
| +`Server.systemOperator`): |
| carlos engines add <name> --host <hostname> [--per-app <n>] |
| carlos engines remove <name> |
| @@ -132,28 +208,25 @@ the ops account or an address on `CARLOS_SYSTEM_OPERATORS` |
| carlos engines revoke <name> --app <account>/<app> |
| carlos engines list |
| -- `<name>` matches `^[a-z][a-z0-9-]{0,31}$` and is unique in the |
| - deployment. It is what apps put in `Host:`. |
| -- `--host` is the engine route's hostname as the edge serves it: exact, no |
| - wildcard, lower-cased. The console does not check that a box serves it |
| - (the console holds no route registry); §5.4 reports the mismatch from the |
| - box. |
| -- `--per-app` is the engine box's in-flight cap per calling app (§5.5). |
| - Default 4, range 1 to 64. |
| -- `remove` refuses while grants remain. Removing an engine makes its |
| - hostname public again (§6.3), so it is a deliberate two-step act. |
| -- Every change is audited with `store.RecordEvent`, like the store grant. |
| -- `list` shows each engine, its host, its cap and its grantees to an |
| - operator. A member sees only the engines granted to apps in accounts they |
| - belong to, with name and status, never the host or the other grantees. |
| - |
| -The console keeps engines and grants in its own database (the authority) |
| -and publishes the whole set as one record, `control/engines.json`, |
| -signed with the deployment's root release key under its own signing domain |
| -(`carlos engines v1\n`), with a revision and `expires_at` 72 hours out, |
| -exactly as door keys are (`internal/doorkey`). The console re-signs well |
| -inside the lifetime. One file, because a box needs the whole set to decide |
| -G1, and a deployment has a handful of engines. |
| +- `<name>` matches `^[a-z][a-z0-9-]{0,31}$`, unique in the deployment, never |
| + contains a dot. It is what apps put in `Host:`. |
| +- `--host` is an exact hostname the engine route serves on its home box. |
| + The console cannot check it (it holds no route registry); §5.7 reports |
| + mismatches from the box. |
| +- `--per-app` is the engine box's in-flight cap per calling app. Default 4, |
| + range 1 to 64. |
| +- `remove` refuses while grants remain. Removing a name changes nothing |
| + about whether the route is internal (§5.1). |
| +- Every change is audited with `store.RecordEvent`, like `store grant`. |
| +- `list` shows each engine's name, host, cap and grantees to an operator. A |
| + member sees only the engine names granted to apps in their accounts. |
| + |
| +The console's database is the authority. It publishes the whole set as one |
| +record, `control/engines.json`, signed with the root release key under its |
| +own domain (`carlos engines v1\n`), with a strictly increasing revision and |
| +`expires_at` 72 hours out, re-signed well inside that, as door keys are. It |
| +publishes an empty set (revision 1) the first time the feature is on, so a |
| +box can tell "no engines" from "not read yet". |
| { |
| "v": 1, "revision": 17, "signed_at": "...", "expires_at": "...", |
| @@ -163,223 +236,270 @@ G1, and a deployment has a handful of engines. |
| ] |
| } |
| -### 5.2 The engine mirror (every edge) |
| - |
| -A new mirror beside the door-key mirror, same loop shape and same rules for |
| -a signature, shape, expiry or revision failure (`doorkeys.go`). It feeds two |
| -views with **different failure rules**: |
| - |
| -- **The internal set** (hostnames that must be refused publicly) fails |
| - closed. See §6.3. |
| -- **The grant table** (which app may call which engine) fails closed the |
| - other way: stale or expired means no grants, so calls are refused, never |
| - admitted on old authority. |
| - |
| -### 5.3 Refusing the public (every edge) |
| - |
| -For a host in the internal set, a request that is not an engine call |
| -(§5.4) or the engine app's own door caller gets `unknownHost`, the same |
| -answer as a host nobody serves. Concretely: |
| - |
| -- `ServeHTTP`: after the route lookup and before backhaul, the gate, the |
| - meter, the cache and the dispatch. A box that does not home the host |
| - refuses without dialling. The home box refuses a hop that carries no |
| - verified engine caller (a backhauled public request). |
| -- The :80 handler (`httpFallbackHandler`) serves the ACME challenge for the |
| - host and nothing else: no redirect to https. |
| -- `Allowed` (certificate issuance) is unchanged. The home box needs a |
| - publicly trusted certificate for the hop's TLS (§3). The hostname is not |
| - a secret: certificate transparency publishes it anyway. The refusal is |
| - about serving, not hiding. |
| -- The engine app's own door callers keep today's door rule (their own |
| - routes). The engine's operator can still reach it from their own app's |
| - processes. |
| +### 5.3 The grant mirror (every edge) |
| + |
| +**(codex R1-4)** The mirror verifies signature, shape, domain and expiry as |
| +the door-key mirror does, and adds a **durable revision floor**: |
| + |
| +- The edge persists the highest revision it has admitted, with the record |
| + itself, to its data directory (write to a temporary file, fsync, rename). |
| + It loads both at start, before any engine socket opens. |
| +- A record whose revision is below the floor is refused, at runtime and |
| + after a restart. An equal revision must be byte-identical. |
| +- Stale (no good read for ten minutes), expired, or never read: the grant |
| + table is **empty**, so every engine call is refused. A persisted record |
| + that has since expired counts as expired. |
| +- If the floor cannot be persisted, the edge keeps refusing engine calls |
| + until it can, and logs it once per minute. |
| +- **Residual, stated:** a box with no persisted floor (new or rebuilt) will |
| + admit any validly signed, unexpired record. A bucket writer who replays a |
| + record revoked less than 72 hours ago can restore that grant on such a |
| + box until a newer record is read. The engine box's own floor still refuses |
| + it unless the engine box is itself new. The same residual the door-key |
| + plan accepted. |
| ### 5.4 The call path |
| -**From a nested instance.** For every instance the edge prepares in the |
| -nested layout, it opens `engines.sock` in the instance's run directory: |
| -`/run/carlos/instances/<iid>/engines.sock`, mode 0600, owner carlos. It |
| -opens it beside the control socket, with the same per-host construction |
| -(the host fixed at build time, never read from the request). One socket per |
| -instance, not per generation: the run directory is bound into every unit |
| -kind (run, generation, seed, `carlos run` command, legacy sidecar), so all |
| -of them reach it. Each gets `CARLOS_ENGINES_SOCKET` in its launch |
| -environment, set the way `CARLOS_CONTROL_SOCKET` is, so tenant config cannot |
| -set or override it. The socket exists whether or not the app holds a grant, |
| -so a later grant needs no restart. |
| - |
| -**From the door.** A door caller (`X-Carlos-Door-Key`) whose `Host:` is an |
| -engine *name* in the grant table, granted to the key's app, is an engine |
| -call. Today a door caller's `Host:` is a real hostname; engine names cannot |
| -collide with one because a hostname registered in a deployment always has a |
| -dot and a name never does. |
| - |
| -**On the calling box.** The listener: |
| - |
| -1. strips every client-address header and every `X-Carlos-*` header, as the |
| - door does (`doorRequest`), and presents the box's own address; |
| -2. resolves the caller: the instance's route gives account and app |
| - (`release.RouteAccount`, `Route.App`); |
| -3. reads `Host:` as an engine name, looks it up in the grant table, and |
| - answers 404 `engine not granted` for an unknown name, an ungranted app, |
| - or a stale or expired table (one body for all three); |
| -4. puts an engine-call mark (account, app, engine) on the request context |
| - and rewrites `Host:` to the engine's hostname; |
| -5. serves the engine route locally if this box homes it (allowed, but the |
| - README's dedicated-box advice stands), or backhauls it to the home box. |
| - |
| -**The hop.** An engine call's hop carries a new header, |
| -`X-Carlos-Hop-Caller: <account>/<app>/<engine>`, and its MAC covers it: |
| -`hopMAC` gains a versioned form over `host\nts\ncaller`. The receiving edge |
| -verifies it with the same dual-key rule (`verifyHopEitherKey`), deletes it |
| -with the other hop headers, and turns it into the same engine-call context |
| -mark. A hop without a caller is an ordinary hop and stays exactly as today. |
| -The caller header is the mesh's vocabulary: the door and the engine socket |
| -delete it on the way in, and `acceptHop` deletes it on every request. |
| - |
| -**On the engine box.** An engine call (local or hopped) is admitted to the |
| -engine route only when the mark's engine names this host in the internal |
| -set and the grant table grants it to the mark's app. Otherwise |
| -`unknownHost`. Then §5.5's cap, then today's dispatch. The cache is |
| -skipped for engine calls. |
| - |
| -### 5.5 Limits |
| - |
| -- **Per-app in flight, on the engine box:** at most `per_app` concurrent |
| - calls per calling app, summed across its instances and boxes. Over the |
| - cap: 429 with `Retry-After: 1` and a log line naming the app, rate |
| - limited. The engine box is the one place that sees every caller. |
| -- **Request body:** 64 MiB, refused with 413 at the calling box before the |
| - hop. |
| -- **Time:** the hop and the socket allow a call up to 5 minutes end to end. |
| - Longer work belongs in the engine's async mode, whose webhook reaches the |
| - caller's public URL over the internet (the engine's egress allows the |
| - public internet). |
| +**The engine socket (nested instances; codex R1-1, R1-6).** An |
| +instance-level listener manager in the edge owns one socket per nested |
| +instance: `/run/carlos/instances/<iid>/engines.sock` (at most 75 bytes). |
| + |
| +- **Lifetime:** opened when the instance's run directory is prepared for |
| + its first unit of any kind (a seed, a run, a command), kept across |
| + generations and restarts of the instance's units, closed only when the |
| + route leaves the box (removal, a move's commit, a reset). It is not tied |
| + to a generation and never closed by one. It exists whether or not the |
| + app holds a grant, so a later grant needs no restart. Answering on a |
| + sleeping instance's socket does not wake the instance. |
| +- **Mode 0666**, because the connecting process's identity is checked |
| + below, not by file permission. |
| +- **Who is calling:** on every accepted connection, before a byte is read, |
| + the edge takes the peer's pidfd (SO_PEERPIDFD), reads the process's |
| + cgroup, and confirms through the pidfd that the process is still the |
| + same one. The connection is admitted only if the cgroup is inside |
| + `carlos-instances-<iid>.slice` for the instance this socket belongs to. |
| + A flat tenant (in `carlos-edge.service`'s cgroup), a sibling instance, or |
| + anything else is closed with no response. The socket's path proves |
| + nothing; the cgroup is the identity, and every nested unit kind runs in |
| + that slice (§3). A box whose kernel lacks SO_PEERPIDFD (before 6.5) opens |
| + no engine sockets and says so once. |
| +- **Environment:** `CARLOS_ENGINES_SOCKET` is set for every nested unit kind |
| + the way `CARLOS_CONTROL_SOCKET` is for the run role (tenant config cannot |
| + set it): the run launch file, each generation's launch file, the seed's |
| + variables and the command unit's environment. |
| + |
| +**The door (codex R1-5).** A door caller whose `Host:` has no dot and names |
| +an engine is an engine call: after `doorRequest` admits the key, the door |
| +mark is replaced by an engine mark for the key's account and app, and the |
| +door's own scope check (`doorOwnsRoute`) does not run for it; the engine |
| +grant check (step 3 below) replaces it. A door caller whose `Host:` is an |
| +engine's real hostname gets `unknownHost` like any other non-engine call. |
| + |
| +**On the calling box.** |
| + |
| +1. The listener strips every client-address header and every `X-Carlos-*` |
| + header, as `doorRequest` does, and presents the box's own address. |
| +2. The caller is the instance's route (`release.RouteAccount`, `Route.App`), |
| + or the door key's account and app. |
| +3. `Host:` names the engine. Unknown name, ungranted app, or an empty grant |
| + table: 404 `engine not granted`, one body for all three. |
| +4. The edge puts an engine mark (account, app, engine) on the request |
| + context, sets `Host:` to the engine's hostname, and refuses `Upgrade` |
| + requests with 400. |
| +5. If this box homes the hostname it dispatches locally; otherwise it |
| + backhauls to the home box (§5.5's transport). A home reached through a |
| + customer-fleet tunnel is not supported: 502 `engine unreachable`. |
| + |
| +**The hop.** An engine call's hop adds `X-Carlos-Hop-Caller: |
| +<account>/<app>/<engine>`, and its MAC covers it: a second MAC form over |
| +`"v2\n" + host + "\n" + ts + "\n" + caller`, verified with the same |
| +dual-key rule. The receiver deletes the header with the other hop headers |
| +on every request, verified or not, and turns a verified caller into the |
| +same engine mark. A hop without a caller is an ordinary hop, unchanged. |
| +The door, the engine socket and `acceptHop` all delete the header on the |
| +way in, so only an edge can speak it. |
| + |
| +**On the engine box.** An engine-marked request reaches a route only if the |
| +route is internal, the mark's engine names the request's hostname in the |
| +grant table, and the table grants that engine to the mark's app. Otherwise |
| +`unknownHost`. Then: |
| + |
| +- the per-app cap (§5.5); |
| +- the import gate, unchanged; |
| +- **no password gate** (the platform authenticated the caller; a login page |
| + means nothing to a program); |
| +- the meter and analytics, unchanged: calls count against the engine's own |
| + app, as any request to it does today; |
| +- **no cache**; |
| +- the activator's dispatch, which may wake a hibernating engine. |
| + |
| +### 5.5 Limits and transport (codex R1-7) |
| + |
| +- **Per-app in flight**, on the engine box: at most `per_app` concurrent |
| + calls per calling app. Over the cap: 429, `Retry-After: 1`, and a |
| + rate-limited log line naming the app. The cap is per serving box: an |
| + engine served by more than one box (replica placements) allows `per_app` |
| + on each. |
| +- **Transport:** engine hops use their own transport, never the backhaul |
| + transport shared per destination: a 5-minute response-header timeout, |
| + and one 5-minute deadline over the whole call (upload, wake, conversion, |
| + response). The socket's server side carries the same deadline. |
| +- **Body:** 64 MiB. A declared `Content-Length` over it is refused with 413 |
| + before anything is forwarded. A body of unknown length is streamed and |
| + counted; at 64 MiB the edge aborts the upstream request and answers 413, |
| + so the engine may receive a truncated request it then discards. |
| +- **Async work:** longer conversions use the engine's own async mode, whose |
| + webhook reaches the caller's public URL over the internet. |
| ### 5.6 What the engine sees |
| The request as the app sent it, with `Host:` set to the engine's hostname, |
| the client-address headers replaced by the calling box's address, and the |
| -app's own `Authorization` passed through. An engine that keeps basic auth |
| -keeps it as a second lock. The edge's access log on the engine box records |
| -the caller app for each call. No caller header reaches the engine. |
| +app's own `Authorization` passed through, so an engine that keeps basic auth |
| +keeps it as a second lock. No caller header reaches the engine; the engine |
| +box's access log records the caller app for each call. |
| ### 5.7 Box-side advisories |
| -The hourly capability pass on the engine's home box logs one line per pass |
| -when a registered engine host is served by a route without the |
| -`doc-engine` grant, naming the grant command. It does not refuse: the |
| -internal set protects any route registered, and the grant is about the |
| -engine's own egress. |
| +The hourly capability pass on each box logs once per pass: |
| + |
| +- an internal route whose hostname is not named by any engine in the grant |
| + table (nobody can call it); |
| +- an engine whose hostname this box serves on a route that is not internal |
| + (the engine is still public; calls to it are refused by §5.4's engine-box |
| + rule, so the name is useless until the route declares `doc_engine`); |
| +- an internal route that declares `doc_engine` without the grant (no JIT |
| + or memory band; it will not render). |
| ## 6. Correctness rules |
| ### 6.1 Identity |
| -Who calls is decided by the socket the request arrived on, or by a verified |
| -door key, on the calling box; and by a verified hop caller on the engine |
| -box. No other input names a caller. Every box holds the mesh key, so any |
| -box can sign any caller: a compromised box can call any engine as any app. |
| -That is the trust the hop already places in boxes, and stated here rather |
| -than solved. |
| +On the calling box the caller is the kernel's answer about the connecting |
| +process's cgroup, or a verified door key. On the engine box it is a verified |
| +hop caller, or the local engine mark. No header the app wrote names a |
| +caller. Every box holds the mesh key, so a compromised box can call any |
| +engine as any app; that is the trust the hop already places in boxes, stated |
| +and not solved. |
| ### 6.2 Flat layout |
| -A flat instance gets **no engine socket**. In the flat layout every |
| -instance runs as uid carlos with the same view of `/run/carlos`, so a |
| -sibling could dial another host's socket and the socket would prove |
| -nothing. A flat app reaches an engine through the door with a door key, |
| -like any app process; the door key says which app it is. (My call; see §11.) |
| +A flat instance has no engine socket of its own, and cannot use a nested |
| +sibling's (its process is not in that sibling's slice). A flat app reaches |
| +an engine through the door with a door key. Enabling the door on a box |
| +(`CARLOS_EDGE_DOOR`) and minting the app's key are operator steps in the |
| +runbook, not part of this build. |
| -### 6.3 The internal set never reopens by accident |
| +### 6.3 The public never reaches an internal route |
| -- The edge persists the last verified internal set to the box's data |
| - directory and loads it at start, before any listener opens. |
| -- A stale, expired, missing or unverifiable `engines.json` leaves the |
| - internal set as it was. Only a verified record with a higher revision |
| - can remove a host from it. |
| -- A host leaves the set only through `carlos engines remove`, whose record |
| - no longer lists it. |
| -- A box that has never read a verified record has an empty set: nothing it |
| - serves was ever internal on its watch. |
| +- The mark lives in the box's own registry, written by adoption or the |
| + grant before the route is served (§5.1). It needs no bucket read at |
| + serve time and survives restarts. |
| +- The check runs on the final upstream's route, after every re-resolution, |
| + before every dispatch branch (§5.1). |
| +- Old non-home boxes backhaul public requests as ordinary hops; the home box |
| + refuses every hop without a verified caller for an internal route. |
| ### 6.4 Freshness |
| -The grant table follows the door-key contract: a grant or revoke lands on |
| -every edge within one 30 s interval of the record; a box that cannot list |
| -the bucket drops every grant ten minutes after its last good read; an |
| -expired record (a console down for 72 hours) drops every grant. The two |
| -edges in a call (calling and engine box) check independently, so a revoke |
| -is in force once either has read it. |
| +A grant or revoke reaches every edge within one 30 s interval of the |
| +record. A box that cannot read the bucket drops every grant ten minutes |
| +after its last good read; an expired record (a console down for 72 hours) |
| +drops every grant. The calling box and the engine box check independently, |
| +so a revoke is in force once either has read it. |
| ## 7. Deploy order and rollout |
| -- Every box and the console must run a build carrying this before the first |
| - `carlos engines add`. A box on an older build ignores `engines.json`. If |
| - the old box homes the engine, it keeps serving the host publicly; if it |
| - only receives a public request for it, it backhauls as today and the new |
| - home box refuses the hop (no caller). So **the engine's home box is the |
| - one that must be current**; the runbook checks its version first. |
| -- Registering an engine cuts public callers within 30 s. Move callers onto |
| - the socket or door before `add`, not after. |
| -- Dormant until the first `add`: with no `engines.json`, the internal set |
| - is empty, the grant table is empty, and every engine socket answers 404. |
| +- **Every engine's home box takes this build first.** From then on its |
| + doc-engine routes are internal (R1). Before that, the operator moves each |
| + public caller of a doc-engine route onto the socket or the door. On the |
| + flagship and dev, list doc-engine routes and their callers before the |
| + roll. |
| +- Calling boxes and the console follow. With no `engines.json` the grant |
| + table is empty and every engine socket answers 404; nothing else changes. |
| +- A calling box on an older build cannot make engine calls (it has no |
| + socket and sends no caller). A home box on an older build serves its |
| + engine publicly, which is today's behaviour and the reason for the order |
| + above. |
| - Roll order as usual: dev boxes, dev console, check, flagship boxes, |
| flagship console. |
| ## 8. Tests that prove it |
| -Production-wired, through real listeners and real hops between two edges in |
| -one test process, not handler fixtures: |
| - |
| -- A public request for a registered host, on the home box and on a |
| - non-home box, gets `unknownHost`, on :443 and :80 (no redirect), and the |
| - ACME challenge path still answers. |
| -- An instance socket call with a grant reaches the engine on another edge; |
| - without a grant it gets 404 and the engine sees nothing. |
| +Production-wired: real listeners, real hops between two edges in one test |
| +process, real unix sockets and a real cgroup check where the kernel allows |
| +(the cgroup reader takes a root path for fixtures, as `instancectl` does). |
| + |
| +- Public requests to an internal route through its host, an alias, a path |
| + map, a wildcard, a mixed-case spelling and a login POST whose `next` |
| + re-resolves to it: all `unknownHost`, on :443 and :80 (no redirect); the |
| + ACME challenge still answers. |
| +- A public request reaching a non-home box is backhauled and refused by the |
| + home box. |
| +- A route adopted from a release declaring `doc_engine` is internal before |
| + its first request, on a fresh registry, with the bucket unreachable after |
| + adoption. |
| +- A socket call from a process in the instance's slice reaches the engine on |
| + another edge; from a process outside the slice (a flat child, a sibling |
| + slice) the connection is closed unread. |
| - A forged `X-Carlos-Hop-Caller` from a client, a door caller or an app |
| - through the socket never survives to the engine box's decision. |
| -- A hop signed with a caller for an ungranted app is refused at the engine |
| - box even though the calling box admitted it (G4: inject the calling |
| - box's table out of date). |
| -- A stale mirror and a restarted edge with no bucket both keep refusing a |
| - registered host (§6.3). |
| + through the socket never reaches the engine box's decision. |
| +- A hop signed with a caller the engine box's table does not grant is |
| + refused even though the calling box admitted it. |
| +- A restart with a persisted floor refuses a replayed older record. |
| - The per-app cap: the n+1th concurrent call from one app gets 429 while a |
| second app's call proceeds. |
| -- Every unit kind of a nested instance (run, generation, seed, command) can |
| - reach the socket and sees `CARLOS_ENGINES_SOCKET`; tenant config cannot |
| - set it. |
| +- The socket survives a generation swap and is up before a seed runs; every |
| + nested unit kind sees `CARLOS_ENGINES_SOCKET`; tenant config cannot set it. |
| +- A 45-second conversion completes; a declared 65 MiB body gets 413 before |
| + the engine sees a byte. |
| ## 9. Docs |
| -`docs/site/cli.md`: a `carlos engines` section; the README's "App |
| -capabilities" doc-engine passage gains the public-refusal story and drops |
| -"behind basic auth" as the only gate; a short "Calling an engine from your |
| -app" passage with a Go example using a unix-socket transport and |
| -`http://gotenberg/forms/chromium/convert/html`. All user-facing strings go |
| +`docs/site/cli.md`: a `carlos engines` section. The README's doc-engine |
| +passage gains the public refusal and the deploy order. A short "Calling an |
| +engine from your app" passage with a Go example (a unix-socket transport, |
| +`http://gotenberg/forms/chromium/convert/html`). All user-facing strings go |
| through copy review before they ship. |
| ## 10. Provider boundary |
| Core only. The mirror reads a `release.Store` (S3 or a directory); signing |
| -uses the root release key; the hop is the existing mesh. On AWS with region |
| -HA, a hop inside one region uses private addresses; everywhere else it |
| -crosses the public internet under TLS and the hop MAC, as backhaul does |
| -today. A non-AWS deployment loses nothing. "Fleet-internal" means "only the |
| -platform can call it", not "only over a private network". |
| +uses the root release key; the hop is the existing mesh; the peer check is |
| +Linux, as the whole nested layout already is. On AWS with region HA a hop |
| +within one region uses private addresses; everywhere else it crosses the |
| +public internet under TLS and the hop MAC, as backhaul does today. A non-AWS |
| +deployment loses nothing. "Fleet-internal" means only the platform can call |
| +it, not only over a private network. |
| ## 11. Calls I made (for review) |
| -- Explicit registration, not "every doc-engine grant is internal". A |
| - doc-engine grant is box-local, and other boxes cannot see it; a |
| - console-side registration gives every box the same answer, and no |
| - existing public engine changes behaviour until an operator registers it. |
| -- Flat instances get no socket and use the door (§6.2). |
| - Engine selection by `Host:` name on one socket, so a client's base URL is |
| - just `http://<name>`. |
| -- Any route may be registered; a missing doc-engine grant is advised, not |
| - refused (§5.7). |
| -- Limits: 64 MiB body, 5 minutes, per-app cap default 4. |
| + `http://<name>`. |
| +- Limits: 64 MiB body, 5 minutes end to end, per-app cap default 4, per |
| + serving box. |
| +- Engine calls skip the password gate and the cache; the import gate, the |
| + meter and analytics run as for any request. |
| +- No owner exception on internal routes: the engine app grants itself. |
| +- Tunnel-homed engines are refused (502), not supported. |
| +- The flat layout's door is an operator step, not built here. |
| - The engine box re-checks the grant (G4) rather than trusting the calling |
| box. |
| + |
| +## 12. Disposition of review findings |
| + |
| +| Finding | Disposition | |
| +| --- | --- | |
| +| R1-1 flat tenant borrows a nested socket | Accepted: peer cgroup check (§5.4) | |
| +| R1-2 aliases and path routes stay public | Accepted: mark on the route, checked on the final upstream (§5.1) | |
| +| R1-3 new box starts public | Accepted: mark from adoption, no bucket at serve time (§5.1, R1) | |
| +| R1-4 revoked grant resurrected after restart | Accepted: durable floor; fresh-box residual stated (§5.3) | |
| +| R1-5 door exceptions | Accepted: owner exception dropped, door-to-engine transition defined (§5.4) | |
| +| R1-6 socket lifecycle | Accepted: instance-level owner (§5.4) | |
| +| R1-7 transport and body | Accepted: engine transport, streamed body with abort (§5.5) | |
| +| Builder Q1 unit-backed callers | Ruling R2: out of scope | |
| +| Builder Q2 door for flat apps | Operator step (§6.2) | |
| +| Builder Q3 engine owner's own processes | Grant itself (§5.1) | |
| +| Builder Q4 gate, meter | §5.4 | |
| +| Builder Q5 socket lifetime | §5.4 | |
| +| Builder Q6 several homes, tunnels | Per-box cap; tunnels refused (§5.4, §5.5) | |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md| index 5314c02b3..78e6dae46 100644 |
| --- a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-ENGINES.md |
| @@ -58,3 +58,17 @@ Findings ranked P1 (spec fails a goal or anti-goal, or cannot be built), P2 |
| the failure scenario as concrete steps, the code citation, and a suggested |
| fix. End with an overall verdict: accept, accept with fixes, or reject. |
| Do not report wording or style. |
| + |
| +## Round 2 and later |
| + |
| +Read your previous findings (`CODEX-FINDINGS-ENGINES-R<n-1>.md`) and the |
| +spec's §12 disposition table first. Start your answer with a table: each |
| +previous finding, and whether the new revision actually fixes it (cite the |
| +section). Then report only NEW findings. |
| + |
| +§2.1's rulings are fixed: do not argue for registration-based internal |
| +marking or for unit-backed callers. Report only a mechanism that fails to |
| +honour a ruling, fails a goal or anti-goal, breaks something else in the |
| +platform, or cannot be built against `main`. A mechanism the spec names, |
| +bounds and assigns is complete at spec level; do not ask for plan-level |
| +detail. |