Fleet-internal engines: a doc engine nobody on the internet can call

Date: 2026-10-21 Revision: 14. Approved by Paul, 2026-10-21. Discussion: amadan carlos/platform #335. Related: #328 (per-app egress allow).

• Rev 14 applies Astra round 13's two P1s (CODEX-FINDINGS-ENGINES-R13.md) as small fixes within R3, per R4 with no further round. The ingress precondition covers private sources too: another box reaching the engine's backend directly (R13-1). The fence on an engine's box is box-wide, cancelling every non-engine proxy request admitted before it, including one held by a row since deleted or replaced (R13-2). Marked (codex R13-n).
• Rev 13 answers Astra round 12 (CODEX-FINDINGS-ENGINES-R12.md, reject, two P1s: an engine binding its own high port, and a separate public route row pointing at the engine's backend) with Paul's ruling R3: an engine's home box is dedicated, and the platform enforces it (§5.9). Ruling R4 sets the review end: one more round, then Paul's read. Marked (codex R12-n).
• Rev 12 answers Astra round 11 (CODEX-FINDINGS-ENGINES-R11.md, reject, one P1: a declaring release that also declares ports keeps a granted TCP listener the edge never sees). doc_engine and ports are refused together at ship, as doc_engine and host_role already are, and a box renders no port listener for an internal group (R11-1). §5.1 now also says why systemd's own starts (socket activation, Restart=always) keep I1. Marked (codex R11-1).
• Rev 11 answers Astra round 10 (CODEX-FINDINGS-ENGINES-R10.md, reject, one P1: a mark written but never fenced, after a failed reload and a timed-out adoption, let a recovery launch start a declaring build while a public request was admitted). I1 now speaks of the fenced mark, not the registry's, and every declaring launch requires the fence whether or not it finds 1 already written (R10-1). Marked (codex R10-1).
• Rev 10 answers Astra round 9 (CODEX-FINDINGS-ENGINES-R9.md, reject, one P1: after --public, a failed first wake falls back to a declaring GoodVersion). Instead of a fourth path-specific rule, §5.1 now states the invariant every path must keep, I1, and enforces it at the one place every path passes through: the launch. A launch of a declaring build sets the mark and completes the barrier first, cancelling the public request that caused it (R9-1). Marked (codex R9-1).
• Rev 9 answers Astra round 8 (CODEX-FINDINGS-ENGINES-R8.md, reject, a P1 and a P2, both accepted) by changing approach rather than adding a third proof: a route never becomes public automatically. Clearing the mark is an operator verb that stops the group under the adoption lock and checks its cgroups are empty (R7-1, R8-1, R8-2). Upgrade classification also marks a route internal when any staged version of its app declares doc_engine. Marked (codex R8-n).
• Rev 8 answers Astra round 7 (CODEX-FINDINGS-ENGINES-R7.md, reject, a P1 and a P2, both accepted). Clearing the mark waits for proof that no process of a declaring build is left, not for adoption's return, which under generation rollouts only schedules the swap (R7-1). A minted row records its owner explicitly; the fallback match needs a unique non-empty backend, TCP included (R7-2). Marked (codex R7-n).
• Rev 7 answers Astra round 6 (CODEX-FINDINGS-ENGINES-R6.md, reject, a P1 and a P2 in rev 6's minted-row rule, both accepted). Inheritance is scoped to minted instance rows, static and service platform rows keep 0 (R6-2), and a doc-engine grant on a minted row marks its execution owner, so the whole group goes internal together (R6-1). Marked (codex R6-n).
• Rev 6 answers Astra round 5 (CODEX-FINDINGS-ENGINES-R5.md, accept with fixes, three P2s, all accepted). Platform-minted rows take their execution owner's mark and barrier (R5-1). Arming judges exec-backed routes by layout and admits unit-backed ones (R5-2). The watch ticket signs the probe path the watcher uses (R5-3). Marked (codex R5-n).
• Rev 5 answers Astra round 4 (CODEX-FINDINGS-ENGINES-R4.md, reject, two P1s and a P2, all accepted). "No flat route" becomes a positive, operator-armed box state that also excludes foreign instances and workers, and refuses unisolated launches while armed (R4-1). An engine call is dispatched only to the engine hostname's own route, never through a path map (R4-2). Watch tickets are renewed for as long as the watch runs (R4-3). Marked (codex R4-n).
• Rev 4 answers Astra round 3 (CODEX-FINDINGS-ENGINES-R3.md, reject, two P1s and a P2, all accepted). The barrier's acknowledgement now also covers public requests admitted before the mark: the edge cancels them and waits for them to finish before it acknowledges (R3-1). Engine sockets open only on a box with no flat instance at all, because a flat tenant can write a nested sibling's files and run code inside its slice (R3-2). Every deploy target gets a watch ticket, not only declaring releases (R3-3). Marked (codex R3-n).
• Rev 3 answers Astra round 2 (CODEX-FINDINGS-ENGINES-R2.md, reject, three P1s and two P2s, all accepted). The mark gains an "unclassified" state and a serving barrier: set before the release switch, cleared after it, and observed by the edge before adoption proceeds (R2-1). Every route is classified before the upgraded edge opens a listener (R2-2). The door carries engine calls only for callers with their own uid, so flat instances cannot call engines at all (R2-3). carlos deploy watches an internal engine with a console-signed watch ticket (R2-4). The refusal is a plain 404 that never redirects, and the no-redirect promise on :80 is scoped to the home box (R2-5). Marked (codex R2-n).

• Rev 2 answers Astra round 1 (CODEX-FINDINGS-ENGINES-R1.md, reject, four P1s, all accepted) and the builder check, and records Paul's two rulings of 2026-10-21 (§2.1). What makes a route internal moves from a console record to the route itself (R1), which closes R1-3 (a new box starts public) and R1-2 (aliases and path routes) by construction. The socket's caller is proved by the peer's cgroup, not the socket's path (R1-1). The grant table keeps a durable revision floor (R1-4). The door's owner exception is dropped and the door-to-engine transition is defined (R1-5). The socket gets an instance-level owner (R1-6) and engine calls their own transport (R1-7). Each change is marked (codex R1-n).

1. Intent

The doc_engine capability (internal/caps/render.go, README "App capabilities") fences a conversion engine's egress away from private networks. It says nothing about who can call the engine. A doc-engine route is a route like any other: its box's public edge serves it on :443, any other box backhauls a public request for it, and the only gate is whatever auth the engine applies itself (Gotenberg: basic auth, often none). An operator who skips that auth, or leaks its password, publishes a Chromium and LibreOffice renderer to the internet: free compute for anyone, and an SSRF engine whose remaining reach is the public internet and its own box's loopback.

This spec makes every doc engine fleet-internal. Its routes answer only calls the platform carried from an app an operator granted, and an isolated tenant reaches it through a socket inside its own instance, with no private address opened in its fence.

2. Goals, non-goals, anti-goals

Goals

• G1. A route whose live release declares doc_engine, or which holds the doc-engine grant, is refused to every public request, through every hostname, alias, wildcard or path that resolves to it, on every listener, except the ACME challenge.
• G2. An operator grants an app the use of a named engine, per app. Every nested instance of the app, on any box and in every unit kind its code runs as (run, generation, seed, carlos run command, legacy sidecar), can then call it, and so can the app's own host processes that hold a door key and run under their own uid (§5.4). A revoke takes effect within §6.4's bounds.
• G3. A nested instance calls an engine through a unix socket in its own run directory. Its IPAddressDeny= fence is unchanged.
• G4. The engine box knows which app made each call and refuses a call from an app without a grant, even if the calling box admitted it.
• G5. One app cannot starve the others: the engine box caps each app's calls in flight.
• G6. Provider-neutral (§10).

Non-goals

• Unit-backed callers (carlos-app@, DynamicUser). Ruling R2: later, if a caller appears. They have no instance run directory and cannot be enrolled for the door. Their routes can still be engines.
• Flat-layout instances as callers (§6.2). They share uid carlos with every other flat tenant, so nothing they hold can prove which app they are. An app that needs an engine moves to the nested layout.
• Metering or billing engine use. Member-requested grants.
• A general internal-route product for arbitrary services.
• Reaching an engine over TCP from a nested instance: its fence denies loopback and every private range, and IPAddressAllow= cannot name a port (#328's limit 1).
• Replacing #328. An engine outside the platform still needs #328's egress allow.

Anti-goals

• A window in which an engine route is public: a new or replacement box, a restarted edge, a stale mirror or a box that cannot reach the bucket must never serve one to the public.
• A caller identity a tenant can forge or borrow: through a header, another instance's socket, a sibling's process or the door.
• Opening any private address in an isolated instance's fence.

2.1 Rulings (Paul, 2026-10-21; do not reopen)

• R1. Declaring doc_engine makes a route internal. No opt-out and no separate registration step for it. A doc-engine route in public use today goes dark when its home box takes this build; its callers move to the socket or door first (§7).
• R2. Unit-backed apps are not callers in this spec. Later, if a caller appears.
• R3. An engine's home box is dedicated, and the platform enforces it (§5.9). Public ingress beyond the edge's own ports is a stated box precondition, not a platform-installed firewall.
• R4. Review end: one Astra round on rev 13, small fixes, then Paul's read. The plan's first task is a simulation of I1 and §5.9 across every launch and ingress path.
• R5. Grant records go stale ten minutes after signing (Paul, 2026-10-21). An edge treats an engines.json record whose signed_at is more than 10 minutes old as stale, so its grant table is empty; the console re-signs every 2 minutes and on every change (§5.2, §5.3) (sim, plan 1, final review).

3. What exists today, and what this builds on

Cited from main at 3d444a1b.

• The backhaul hop (internal/edge/edge.go backhaul/backhaulTo, homemap.go): one hop, ever, from the box a request reached to the box that homes the host. It dials the home box's private_ip when region HA is on and both boxes share a region and placement, its public address otherwise (homemap.go dialAddress), on :443 with public-CA TLS and SNI set to the real hostname, with a 30 s response-header timeout and a transport shared per destination (edge.go:2825, :2938). It is authenticated by X-Carlos-Hop and X-Carlos-Hop-Auth, an HMAC-SHA256 over host and timestamp (±30 s) under the deployment's mesh key (mesh.go). The hop carries no caller (acceptHop).
• The edge-local door (#175 P7, internal/edge/door.go): a unix socket (CARLOS_EDGE_DOOR) through which an app's own processes reach the app's hosts on any box with an app-scoped door key (internal/doorkey): a console-minted, root-signed, revisioned, expiring record under control/door-keys/, mirrored by every edge every 30 s with a ten-minute stale bound and an in-memory revision floor (doorkeys.go:22). The door's mark is a context value; it does not cross a hop.
• The per-instance control socket (internal/edge/purge.go ListenControl), opened per generation for the run role only (prepare.go:111) and closed with the generation (unitproc.go:203). Every nested unit binds its whole run directory /run/carlos/instances/<iid> (carlos-instance@.service:53; generations through instancectl/gen.go:48). Every nested unit kind runs in Slice=carlos-instances-<iid>.slice (instancectl.SliceName; the four templates and the generation drop-in). The instance ID is at most 40 bytes (provision.go:2918).
• Peer checks on a unix socket: the root control service accepts a connection only from SO_PEERCRED uid carlos and the edge's MainPID (instancectl/server.go PeerOK).
• Capabilities: the release's manifest declares them; adoption stages the manifest beside the live version (release/adopt.go stageAuthorized, capability_pass reads it via stagedManifestOfLiveVersion); grants are box-local in routes.caps (registry.SetCaps), set with carlos route --host X --grant doc-engine.
• Flat tenants share uid carlos with nested ones and can write their siblings' directories (layout/preflight.go:181).

4. The shape, in one paragraph

A route becomes internal on the box that serves it when its live release declares doc_engine or it holds the doc-engine grant; the box records that in its own registry and refuses the route to every public request. An operator names an engine (gotenberg → its hostname) and grants it to apps; the console publishes the set as a root-signed control/engines.json. A granted nested instance's code sends plain HTTP to $CARLOS_ENGINES_SOCKET with Host: gotenberg. Its box's edge proves the caller from the connecting process's cgroup, checks the grant, and relays the request to the engine's home box over the backhaul hop, now signed with the caller's account, app and engine. The engine box verifies the signature, checks the grant again, applies the per-app cap and serves the route. A door caller does the same with Host: gotenberg.

5. The parts

5.1 The internal mark (every box, on its own routes)

(codex R1-2, R1-3, R2-1, R2-2) Internal is a property of a route on the box that serves it, never of a hostname and never of a bucket record.

The column. routes.internal has three states: NULL (unclassified), 0 (public) and 1 (internal). Routes of a kind that serves no release (static, service, the status host, sidecars) are 0 by construction. An instance route is classified from the manifest of its live release: 1 if it declares doc_engine, 0 otherwise. The doc-engine grant also sets 1, whatever the manifest says. The edge treats NULL like 1.

Platform-minted rows (codex R5-1, R6-1, R6-2). A platform domain is its own registry row (Platform: true, domains/reconcile.go:881), copying the source's kind and backend but no version, and adoption skips it (release/adopt.go:1210).

• A minted static or service row (including a static canary on its own channel, reconcile.go:368) is 0 by construction, like any row of those kinds. The rest of this paragraph does not apply to it.
• A minted instance row's execution owner is recorded on the row: the domains reconciler writes the source's host into a new routes.owner column when it mints (mintedRoute has the source in hand) and backfills it on its next pass for rows minted earlier (codex R7-2). Until a row has an owner recorded, the edge falls back to matching the non-platform instance row with the same account and app and the same non-empty backend: Socket, Addr (TCP-backed rows, cli.go:172, domains/reconcile.go:835), or a fleet-placed Region. Empty values never match, and a match that is not unique counts as no match. The edge resolves owners at Reload.
• The owner and every minted row that shares it form one group with one mark and one admission epoch. The group is internal when the owner's own classification is 1 or any row in the group holds the doc-engine grant. The grant CLI already permits a grant on a plain minted row (cli.go:1432); such a grant marks the whole group, and --grant and --revoke on any row of the group run §5.1's barrier for the group.
• A minted instance row whose owner the edge cannot find, or finds ambiguously, is treated as NULL and refused, and logged by name, until the reconciler records its owner. --grant doc-engine on a minted row whose owner is unresolved or ambiguous is refused; the row stays unclassified and refused (sim, plan 1).

The invariant (codex R9-1).

> I1. A process of a build whose manifest declares doc_engine starts > only after the edge has fenced its group as internal: the edge's > fenced counter (below) has reached a registry change that set the > group's mark to 1, and no later change has cleared it.

What protects the public is the edge's serving map, not the registry row (codex R10-1): a 1 written while the edge's Reload fails (edge.go:925, :993) refuses nobody. So the registry mark is the intent, and the fence is the fact I1 is about. A group closed as unclassified (NULL) counts as fenced only once the edge has loaded that NULL: the change that closed it is recorded for the group, and the reload that reached it drained every older public request on the box (the upgrade's restart does both). A newly created or unseen row never counts as fenced, so a first declaring launch writes 1 and waits for the fence like any other (sim, plan 1, final review).

Rounds 2 to 9 each found one more path by which a declaring build could run under a public mark: adoption's order, requests admitted early, a pending generation swap, an un-cycled hibernating route, a sleeping unit, and wake recovery falling back to GoodVersion (rollout/wakeplan.go:76, :111; activator/wakeplan.go:448). So I1 is enforced where every one of those paths ends: the launch gate. Every activator launch path (an exec child, a nested run or generation unit through the control service, a unit-backed systemctl start, a seed, a carlos run command, a swap's incoming generation, a wake plan's fallback or recovery build) reads the manifest of the build it is about to start. If that build declares doc_engine, the launch writes 1 if it is not already written, and then always waits for the edge's fence to cover the group's current mark (below), which cancels every public request admitted before it, including the one whose wake triggered the launch. Only then does it start the process. A launch that finds 1 written but not yet fenced (a timed-out adoption, a reload that keeps failing) waits the same way; if the fence does not arrive within the bound, the launch is refused, the wake fails, and the request gets the activator's ordinary wake-failure answer. A launch whose build manifest it cannot read is treated as declaring. Adoption's set-before is the same gate, reached earlier. The gate holds a per-group lock from its check through process creation, and --public holds the same lock for its whole run, so no launch can pass the gate between --public's empty check and its write of 0 (sim, plan 1, final review).

--public (below) is the only writer of 0, and it checks I1 directly: the group is stopped and its cgroups empty. Any later declaring launch re-marks the group through the gate. The one place I1 is not enforced is stated as a residual under "Classification before serving".

Starts systemd makes on its own. A unit-backed route's unit is also started by socket activation (carlos-app@.socket) and restarted by Restart=always (carlos-app@.service:50), without the activator. Both execute /opt/carlos/live/<host> (:42), the live version. Adoption passes the launch gate before it repoints the live link to a declaring build, and --public clears only when the live build does not declare, so whatever systemd starts is either fenced or non-declaring. The builder keeps that true by construction: nothing but adoption (through the gate) and --public (through its checks) may move a live link.

The edge is the only ingress (codex R11-1). I1 protects the public only if every way in goes through the edge. A granted ports capability is a ListenStream=<port> socket the edge never sees (caps/render.go:286, :319), and a hibernating route's port starts the unit without the activator (caps/converge.go:241). So:

• At ship, doc_engine and ports are refused together, beside the existing host_role refusal (caps/caps.go:382): a conversion engine is reached over HTTP through the platform, never on a port of its own.
• On the box, the capability pass renders no ListenStream for any route of an internal group, whatever its grants, and removes an existing one (stopping the socket unit) before the launch gate's fence completes for that group; it logs the grant it is ignoring with the revoke command. A ports listener is removed on every write of 1 or NULL (a grant, the upgrade, the launch gate), not only at the launch gate (sim, plan 1).

Setting the mark: the serving barrier (codex R2-1). Adoption repoints the live symlink (release/adopt.go:1751), restarts (adopt.go:1831) and records the version afterwards (adopt.go:2051), and the edge serves an in-memory map refreshed on a 2 s registry poll (edge.go:2298). So the order is explicit:

• Becoming internal (the incoming release declares doc_engine, the route is 0): adoption writes 1, then waits for the edge's acknowledgement, then repoints and restarts. The edge publishes the registry change it has fenced (a counter in a file under /run/carlos), and it advances that counter only after both: 1. Reload has swapped in a map carrying the mark (a failed Reload advances nothing); and 2. every request admitted to that route as a non-engine call before the swap has finished (codex R3-1). Each such request holds a reference on the route's public-admission epoch from the moment it passes the internal check until its response ends, including while it waits in the activator's queue for a wake (activator.go:1554, :5404). Loading a 0→1 change bumps the epoch and cancels the contexts of every request holding the old one, which also drops it from the activator's queue; the counter advances when the last of them has released. A request admitted after the swap sees 1 and is refused.

The wait is bounded (30 s); on timeout adoption does not repoint this pass, the mark stays 1, and the next pass retries. The old, non-engine release is refused publicly for that interval, which costs availability, never exposure.

• Becoming public is never automatic (codex R7-1, R8-1, R8-2). No adoption, revoke, rollback or pass ever writes 0 over 1. Rounds 6 to 8 each found a new interleaving in which an automatic clear raced a process of the declaring build that was still running (a generation swap still pending, an awake hibernating unit adopted without a restart, a sleeping unit with no main process to vouch for). So clearing is an operator act whose preconditions are checked, not inferred:

carlos route --host <host> --public

On the box, holding the group's lock for its whole run (the adoption lock, so the live version cannot move under it, and the launch gate's, so nothing starts between step 3 and step 4; sim, plan 1, final review), it: 1. refuses unless the live release's manifest is readable and does not declare doc_engine, and no row of the group holds the grant; 2. stops the whole group (every unit and generation of the owner, through the activator and the control service); 3. confirms every cgroup of the group is empty (carlos-instances-<iid>.slice, carlos-app@<host>.service, carlos-exec@<host>), and refuses if it cannot; 4. writes 0 and releases the lock.

Whatever starts afterwards passes the launch gate: the live version, which step 1 proved non-declaring, starts under 0; a declaring build chosen by wake recovery (GoodVersion) re-marks the group first and the triggering public request is cancelled (I1). The route's next request wakes it. The cost is a short stop on a rare operation; a log line on the capability pass names every internal route whose live release no longer declares doc_engine and holds no grant, with this command.

• Rollback to a declaring release is an adoption like any other: set before. Rollback away from one leaves the mark set.
• The same barrier applies to a grant: carlos route --grant doc-engine writes 1 and returns once the edge has loaded it (or says it timed out).

Classification before serving (codex R2-2, R8-1). The new column arrives NULL for every existing route. Before the upgraded agent opens any edge listener, it classifies every NULL instance route (minted rows follow their owner):

• 1 if any row of the group holds the grant, or the live version's manifest declares doc_engine, or the manifest of any other version of the app staged on the box declares it. The last clause covers a box whose live symlink has moved off a declaring build that is still running (an awake hibernating route is repointed without a restart, release/adopt.go:1174).
• 0 only if the live manifest is readable, none of the above holds.
• Otherwise (the live manifest is unreadable from the staged sibling and from the bucket, as adoption's back-fill reads it) the route stays NULL, is refused publicly, and is named in a log line every pass until the capability pass classifies it. That is the fail-closed choice.

A route that turns out 1 only because an old staged version declared is cleared with --public like any other. Residual, stated: a declaring version whose staged manifest failed to write, still running across the upgrade after the live symlink moved to a non-declaring version, would be classified 0; it needs a sibling write failure on a doc-engine release and a pending, un-cycled switch away from it at the moment of the upgrade. A route the agent creates later (provisioning, a move) is classified by its first adoption, which runs before the route is served.

What the edge does with it (ServeHTTP):

• The check runs on the final resolved upstream's route: after the route lookup and after the login POST's re-resolution, and before the import gate, the password gate, the meter, the cache, an upgrade and the activator's wake. Aliases (edge.go:929) and path-mapped hosts (edge.go:1364) copy the target's upstream, so every name and path that resolves to an internal route is covered, in any spelling the lookup accepts.
• An internal route admits only an engine call (§5.4). Everything else gets the internal refusal: a plain 404 not found, Cache-Control: no-store, never the pending page and never the fallback redirect (unknownHost does both, edge.go:2205, so it is not reused).
• There is no owner exception: the engine app's own processes reach it by granting the engine to their own app.
• :80 on the home box serves the ACME challenge for an internal route's hostnames and the internal refusal for everything else: no redirect. :80 on another box knows only the home map, so it redirects to https as it does for any host homed elsewhere (edge.go:2262), and the https request is then backhauled and refused (codex R2-5). The redirect names nothing DNS and certificate transparency do not already publish.
• Allowed (certificate issuance) is unchanged: the hop's TLS needs a publicly trusted certificate on the home box.
• A box that does not home the host backhauls a public request as an ordinary hop with no caller, and the home box refuses it. A home box on an older build is the one gap, closed by deploy order (§7).

5.2 Engine names and grants (console)

Operator-only CLI, gated like domains attach --platform (an owner of the ops account or an address on CARLOS_SYSTEM_OPERATORS, Server.systemOperator):

carlos engines add <name> --host <hostname> [--per-app <n>] carlos engines remove <name> carlos engines grant <name> --app <account>/<app> carlos engines revoke <name> --app <account>/<app> carlos engines list

• <name> matches ^[a-z][a-z0-9-]{0,31}$, unique in the deployment, never contains a dot. It is what apps put in Host:.
• --host is an exact hostname the engine route serves on its home box. The console cannot check it (it holds no route registry); §5.7 reports mismatches from the box.
• --per-app is the engine box's in-flight cap per calling app. Default 4, range 1 to 64.
• remove refuses while grants remain. Removing a name changes nothing about whether the route is internal (§5.1).
• Every change is audited with store.RecordEvent, like store grant.
• list shows each engine's name, host, cap and grantees to an operator. A member sees only the engine names granted to apps in their accounts.

The console's database is the authority. It publishes the whole set as one record, control/engines.json, signed with the root release key under its own domain (carlos engines v1\n), with a strictly increasing revision and expires_at 72 hours out. It re-signs every 2 minutes and on every change, and an edge treats a record whose signed_at is more than 10 minutes old as stale (ruling R5; sim, plan 1, final review). It publishes an empty set (revision 1) the first time the feature is on, so a box can tell "no engines" from "not read yet".

{ "v": 1, "revision": 17, "signedat": "...", "expiresat": "...", "engines": [ { "name": "gotenberg", "host": "gotenberg.example.com", "per_app": 4, "grants": [ { "account": "acct1", "app": "shop" } ] } ] }

5.3 The grant mirror (every edge)

(codex R1-4) The mirror verifies signature, shape, domain and expiry as the door-key mirror does, and adds a durable revision floor:

• The edge persists the highest revision it has admitted, with the record itself, to its data directory (write to a temporary file, fsync, rename). It loads both at start, before any engine socket opens. The mirror also persists the time of its last good read, so a restart is not a fresh read (sim, plan 1).
• A record whose revision is below the floor is refused, at runtime and after a restart. An equal revision must be byte-identical.
• Stale (no good read for ten minutes, or a record signed more than ten minutes ago), expired, or never read: the grant table is empty, so every engine call is refused. A persisted record that has since expired or gone stale counts as expired. The mirror refuses a stale record like an expired one (ruling R5; sim, plan 1, final review).
• If the floor cannot be persisted, the edge keeps refusing engine calls until it can, and logs it once per minute.
• The freshness bound (ruling R5; sim, plan 1, final review). A bucket writer can replay an older signed record: before any edge has read a revoke, or to a box with no persisted floor (new or rebuilt). Every record that grants a revoked key was signed before the revoke, and goes stale ten minutes after its signing, so a replay keeps a revoked grant at most until the replayed record's signed_at plus ten minutes. The worst-case revoke delay is therefore about ten minutes plus one poll, replay or not; a box that has read the revoke refuses the replay at once (its floor). The cost: engine calls fail once the console has been down for ten minutes.

5.4 The call path

The engine socket (nested instances; codex R1-1, R1-6). An instance-level listener manager in the edge owns one socket per nested instance: /run/carlos/instances/<iid>/engines.sock (at most 75 bytes).

• Lifetime: opened when the instance's run directory is prepared for its first unit of any kind (a seed, a run, a command), kept across generations and restarts of the instance's units, closed only when the route leaves the box (removal, a move's commit, a reset). It is not tied to a generation and never closed by one. It exists whether or not the app holds a grant, so a later grant needs no restart. Answering on a sleeping instance's socket does not wake the instance. All of this holds only on an armed box (§6.2); elsewhere no engine socket is open, and CARLOS_ENGINES_SOCKET still names the path, so an app finds the socket missing rather than the variable.
• Mode 0666, because the connecting process's identity is checked below, not by file permission.
• Who is calling: on every accepted connection, before a byte is read, the edge takes the peer's pidfd (SOPEERPIDFD), reads the process's cgroup, and confirms through the pidfd that the process is still the same one. The connection is admitted only if the cgroup is inside carlos-instances-<iid>.slice for the instance this socket belongs to. A flat tenant (in carlos-edge.service's cgroup), a sibling instance, or anything else is closed with no response. The socket's path proves nothing; the cgroup is the identity, and every nested unit kind runs in that slice (§3). A box whose kernel lacks SOPEERPIDFD (before 6.5) opens no engine sockets and says so once.
• Environment: CARLOS_ENGINES_SOCKET is set for every nested unit kind the way CARLOS_CONTROL_SOCKET is for the run role (tenant config cannot set it): the run launch file, each generation's launch file, the seed's variables and the command unit's environment.

The door (codex R1-5, R2-3). A door caller whose Host: has no dot and names an engine is an engine call only if the connecting process runs under a uid other than carlos (SO_PEERCRED): an enrolled host-app user, whose door key sits in files only that user can read. A caller running as carlos (a flat instance, or anything else sharing the edge's uid) gets 404 engine not granted, because every flat tenant can read every other's config (layout/preflight.go:181, activator/paths.go:35), so a door key in a flat app's hands proves nothing. When the uid passes and doorRequest admits the key, the door mark is replaced by an engine mark for the key's account and app, and the door's own scope check (doorOwnsRoute) does not run for it; the engine grant check (step 3 below) replaces it. A door caller whose Host: is an engine's real hostname reaches the internal route without an engine mark, so it gets the internal refusal (§5.1) like any other non-engine call.

On the calling box.

1. The listener strips every client-address header and every X-Carlos-* header, as doorRequest does, and presents the box's own address.
2. The caller is the instance's route (release.RouteAccount, Route.App), or the door key's account and app.
3. Host: names the engine. Unknown name, ungranted app, or an empty grant table: 404 engine not granted, one body for all three.
4. The edge puts an engine mark (account, app, engine) on the request context, sets Host: to the engine's hostname, and refuses Upgrade requests with 400.
5. If this box homes the hostname it dispatches locally; otherwise it backhauls to the home box (§5.5's transport). A home reached through a customer-fleet tunnel is not supported: 502 engine unreachable.

The hop. An engine call's hop adds X-Carlos-Hop-Caller: <account>/<app>/<engine>, and its MAC covers it: a second MAC form over "v2\n" + host + "\n" + ts + "\n" + caller, verified with the same dual-key rule. The receiver deletes the header with the other hop headers on every request, verified or not, and turns a verified caller into the same engine mark. A hop without a caller is an ordinary hop, unchanged. The door, the engine socket and acceptHop all delete the header on the way in, so only an edge can speak it.

On the engine box. An engine-marked request reaches a route only if the route is internal, the mark's engine names the request's hostname in the grant table, the table grants that engine to the mark's app, and the route is the engine hostname's own route (codex R4-2). An engine call is resolved by exact hostname only, never through a path map or a wildcard (edge.go:1364; path targets may cross apps within an account, domains/aliases.go:395), so a grant for one engine can never dispatch to another. An alias of the engine's hostname resolves to the same route and is fine. Otherwise the internal refusal (§5.1). Then:

• the per-app cap (§5.5);
• the import gate, unchanged;
• no password gate (the platform authenticated the caller; a login page means nothing to a program);
• the meter and analytics, unchanged: calls count against the engine's own app, as any request to it does today;
• no cache;
• the activator's dispatch, which may wake a hibernating engine.

5.5 Limits and transport (codex R1-7)

• Per-app in flight, on the engine box: at most per_app concurrent calls per calling app. Over the cap: 429, Retry-After: 1, and a rate-limited log line naming the app. The cap is per serving box: an engine served by more than one box (replica placements) allows per_app on each.
• Transport: engine hops use their own transport, never the backhaul transport shared per destination: a 5-minute response-header timeout, and one 5-minute deadline over the whole call (upload, wake, conversion, response). The socket's server side carries the same deadline.
• Body: 64 MiB. A declared Content-Length over it is refused with 413 before anything is forwarded. A body of unknown length is streamed and counted; at 64 MiB the edge aborts the upstream request and answers 413, so the engine may receive a truncated request it then discards.
• Async work: longer conversions use the engine's own async mode, whose webhook reaches the caller's public URL over the internet.

5.6 What the engine sees

The request as the app sent it, with Host: set to the engine's hostname, the client-address headers replaced by the calling box's address, and the app's own Authorization passed through, so an engine that keeps basic auth keeps it as a second lock. No caller header reaches the engine; the engine box's access log records the caller app for each call.

5.7 Box-side advisories

The hourly capability pass on each box logs once per pass:

• an internal route whose hostname is not named by any engine in the grant table (nobody can call it);
• an engine whose hostname this box serves on a route that is not internal (the engine is still public; calls to it are refused by §5.4's engine-box rule, so the name is useless until the route declares doc_engine);
• an internal route that declares doc_engine without the grant (no JIT or memory band; it will not render).

5.8 Watching an engine deploy (codex R2-4)

carlos deploy verifies a release by polling its public URL (deploy.go:1426, :1536), which an internal route now refuses. Whether a route is internal is box-local (a grant can make it so whatever the release declares), so every deploy target's watch carries a watch ticket (codex R3-3). The edge honours it only on an internal route and strips it everywhere else, where the watch proceeds exactly as today:

• The console mints it on the deploy request it already authorizes: signed with the root release key under its own domain (carlos watch v1\n), bound to one hostname, one version and the probe path the watcher will poll (codex R5-3), expiring 5 minutes after minting. The probe comes from the same app setting the console already returns with the ship response (console/apicli.go:2114, :2285; used at deploy.go:404); no probe means / only. It is returned to the CLI with the promote response.
• In bucket mode, where no console is involved, the CLI signs the ticket itself with the root release key it already holds to sign the promote, over the probe path it is about to poll.
• Renewal (codex R4-3): the CLI asks for a fresh ticket for the same hostname and version every 4 minutes for as long as the watch runs (--timeout may be longer than one ticket's life, deploy.go:75). The console re-authorizes the request with the same deploy credential and refuses a hostname or version the deploy did not name; in bucket mode the CLI re-signs.
• The CLI sends it as X-Carlos-Watch-Ticket on each poll. The home box's edge admits a request carrying a valid ticket to the internal route only for GET or HEAD with no body, on / or the ticket's signed probe path, strips the header, and dispatches as usual (no cache, as the watch already busts it). Anything else with a ticket gets the internal refusal.
• The ticket crosses a backhaul as a header and is verified only on the home box. It names no app and grants no engine call.
• A ticket is a credential for reading a version, not for converting a document: the probe paths are the app's own choice, and Gotenberg's /health is a fine probe.

5.9 The engine box is dedicated (codex R12-1, R12-2; ruling R3)

I1 keeps a declaring build from starting before the edge refuses the public, but the edge is not the only way to a process on its box. Round 12 found two others: the engine binding a port of its own on every interface (the template allows it, carlos-app@.service:21, :66; the capability validator leaves high ports to the app, caps/caps.go:401), and another route row whose upstream is the engine's backend (carlos add with a service kind and an explicit address, cli.go:126, :276; the edge proxies by that row's own address, edge.go:760, :889, :1790). The README already tells operators to run a doc engine on a box of its own. This spec makes that a rule the box enforces:

• One internal group per box, and nothing else that proxies. On a box that homes an internal group, every route row whose upstream is a local process (any instance, service or sidecar row: a socket path or an address) must belong to that group. Rows the edge answers without a local process (static, the status host, the tunnel and steering handlers) are unaffected. An unclassified (NULL) route homes its box like an internal one for this rule, since otherwise a declaring carlos run on a shared box is publicly reachable; the cost is that an unclassifiable route on a shared box refuses its proxying neighbours until it is classified (sim, plan 1).
• The fence is box-wide on an engine's box (codex R13-2). The fence a declaring launch waits for (§5.1) cancels every non-engine proxy request the box's edge admitted before it, on any row, not only the group's. That includes a request admitted through a row since deleted or replaced, which still holds its old proxy (registry.go:988, edge.go:1790): the epoch is the box's, not the row's. On a dedicated box that is cheap.
• The launch gate checks it. Before it waits for the fence, a declaring launch confirms the rule holds on the box. If any other proxying row exists, the launch is refused (the wake fails), and one line per pass names each offending row.
• The edge keeps it. While an internal group is homed on the box, the edge refuses to serve any proxying row outside the group: it gets the internal refusal, and the pass logs it with the move or remove command. A row placed on the box later (provisioning, a move, carlos add) is therefore never served there; the engine keeps running.
• Ingress precondition, stated (ruling R3; codex R13-1). The engine's box admits connections from anywhere, public or private, including the deployment's other boxes, only on the edge's ports (80 and 443, plus whatever the operator keeps for administration), by its security group or host firewall. Engine traffic between boxes enters through the home edge's backhaul, never a direct connection to the engine's port: a service row on another box aimed at the engine box's address is a violation of this precondition, and the survey looks for such rows. The platform cannot see that boundary from inside the box. The rollout survey (§7) checks it for every engine box, and the README's doc-engine preconditions gain it as a third, beside the loopback resolver and the dedicated box.

With the box dedicated, a port the engine binds on its own can be reached only through that firewall, and no other row on the box can point at it.

6. Correctness rules

6.1 Identity

On the calling box the caller is the kernel's answer about the connecting process's cgroup, or a verified door key. On the engine box it is a verified hop caller, or the local engine mark. No header the app wrote names a caller. Every box holds the mesh key, so a compromised box can call any engine as any app; that is the trust the hop already places in boxes, stated and not solved.

6.2 Flat layout

Engine sockets need an armed box (codex R3-2, R4-1). Any tenant code running as uid carlos outside an instance slice can write its nested siblings' directories (layout/preflight.go:181), including the environment file a nested launch reads (activator/backend_exec.go:2701), plant a library and an LD_PRELOAD, and so run code inside a granted sibling's slice, which the cgroup check then accepts as that sibling. Three launch paths run tenant code that way today: a flat instance, a foreign one (activator/layout.go:103; both go through the exec-child path, backend_exec.go:1826), and an app's worker, which spawns into carlos-exec@<host> whatever its layout (StartWorker, opset.go). The predicate is therefore positive: the only tenant code on the box runs in instance slices or as unit-backed DynamicUser services (which cannot write carlos-owned directories).

• Arming is an operator act on the box: carlos engines arm. It refuses unless the box's layout intent is nested, every exec-backed instance route classifies nested, no route on the box spawns workers, and every carlos-exec@ cgroup on the box is empty (codex R5-2). Unit-backed routes (backing=unit, carlos-app@, DynamicUser) are admitted: their socket is /run/carlos/apps/<host>.sock, which Classify reports as foreign (activator/layout.go:103), so they are judged by backing, not layout. Platform-minted rows are judged through their execution owner. An exec-backed route that classifies foreign still blocks arming. It records the armed state in the box registry. The edge opens engine sockets only on an armed box.
• While armed, the activator refuses every unisolated launch: a flat or foreign start and a worker spawn are refused with one log line naming the route and carlos engines disarm. The check sits in the launch path, so no unisolated process can start on an armed box, rather than being noticed afterwards. Provisioning a route that would classify flat or foreign on an armed box leaves it unlaunched until the box is disarmed.
• Disarming (carlos engines disarm) closes every engine socket first, then clears the state; launches follow. Re-arming is the same operator act as arming, with the same checks.
• Residual, stated: arming vouches for the present, not the past. A nested instance's files on a box that ran unisolated tenants before it was armed are only as trustworthy as that period, which is true of isolation as a whole. carlos engines arm says so when it succeeds; an operator who doubts a box's past rebuilds its instances from the bucket first.
• Consequence, stated: an app that uses workers cannot run on an armed box until workers move into instance slices. That is a separate change.

(codex R2-3) A flat instance cannot call an engine. It has no engine socket, it cannot use a nested sibling's (its process is not in that sibling's slice), and the door refuses engine calls from uid carlos. Flat tenants share one uid and can read each other's files, so no credential a flat app holds can say which app it is. An app that needs an engine moves to the nested layout, the default since v0.25.0. A host-app process under its own uid uses the door; enabling the door on a box (CARLOS_EDGE_DOOR), enrolling the user (carlos bootstrap --socket-consumer) and minting the key are operator steps in the runbook, not part of this build.

6.3 The public never reaches an internal route

• The mark lives in the box's own registry and survives restarts. It is set before a declaring release can serve, with the edge's fenced state as the barrier, and cleared only by the operator's --public, which stops the group first; an unclassified route is refused (§5.1). No bucket read happens at serve time.
• The check runs on the final upstream's route, after every re-resolution, before every dispatch branch (§5.1).
• Old non-home boxes backhaul public requests as ordinary hops; the home box refuses every hop without a verified caller for an internal route.

6.4 Freshness

A grant or revoke reaches every edge within one 30 s interval of the record. A box that cannot read the bucket drops every grant ten minutes after its last good read; an expired record (a console down for 72 hours) drops every grant. The calling box and the engine box check independently, so a revoke is in force once either has read it.

7. Deploy order and rollout

• Engine boxes. For each box that will home an engine: confirm its public ingress is the edge's ports only (security group or firewall), and move every other app off it. The launch gate refuses a declaring build on a box that still has one.
• Survey first. On dev and the flagship, list every route whose live release declares doc_engine or holds the grant, and who calls it. A route with public callers is cut off the moment its home box takes this build (R1), and its callers can only switch once the console and their own boxes have it too, so that route's cut is scheduled with its owner. A route with no public callers needs nothing.
• Boxes, then the console, in the usual order: dev boxes, dev console, check, flagship boxes, flagship console. Engine calls need the calling box, the home box and the console all on this build.
• Arm the calling boxes. On each box whose instances will call an engine, the operator runs carlos engines arm once the box is fully nested and runs no workers. The survey lists, per box, what blocks arming.
• Then names and grants. With no engines.json the grant table is empty and every engine socket answers 404; nothing else changes. carlos engines add and grant follow the roll, and callers switch to the socket.
• A calling box on an older build cannot make engine calls (no socket, no caller on the hop). A home box on an older build serves its engine publicly, which is today's behaviour and why boxes roll before anything is named.
• A box whose kernel lacks SO_PEERPIDFD opens no engine sockets; the survey checks kernels (Debian 13's 6.12 has it).

8. Tests that prove it

Production-wired: real listeners, real hops between two edges in one test process, real unix sockets and a real cgroup check where the kernel allows (the cgroup reader takes a root path for fixtures, as instancectl does).

• Public requests to an internal route through its host, an alias, a path map, a wildcard, a mixed-case spelling and a login POST whose next re-resolves to it: all the internal refusal (a plain 404, never a redirect, even with a fallback URL set), on :443 and on the home box's :80; the ACME challenge still answers.
• A public request reaching a non-home box is backhauled and refused by the home box.
• A route adopted from a release declaring doc_engine is internal before its first request, on a fresh registry, with the bucket unreachable after adoption.
• Becoming internal: with the edge's reload held back, adoption of a declaring release does not repoint; once the edge loads the mark, it does, and no public request reaches the new release at any point. Becoming public: the mark clears only after the non-declaring release is live; a failed adoption leaves it set.
• An upgraded box with an unclassified declaring route and a missing staged manifest refuses it publicly before its first request, and classifies it from the bucket.
• A door engine call from uid carlos is refused with a valid, granted key; the same key from an enrolled host-app uid succeeds.
• A deploy watch with a valid ticket reads the version on / and the probe path; the ticket on a POST, another path, an expired ticket or one for another host gets the internal refusal.
• A public request admitted before a 0→1 mark and held (mid-proxy, and queued behind a wake) is cancelled before the edge acknowledges, and never reaches the declaring release.
• carlos engines arm refuses with a flat route, a foreign route, a route with workers, or a non-empty carlos-exec@ cgroup on the box; on an armed box a flat start, a foreign start and a worker spawn are refused before any process starts; disarm closes every engine socket before launches resume.
• An engine call with Host: A whose path maps to engine B's route is refused, though the caller holds A's grant.
• A platform-minted hostname of an internal route is refused publicly and admits engine calls exactly as the owner does; a minted hostname of a public route keeps working across the upgrade; a static platform domain and a static canary keep serving.
• Deploying a non-declaring release over a declaring one, with and without generation rollouts, leaves the route internal; --public refuses while the live manifest declares or a grant remains, stops the group, refuses if a cgroup stays non-empty, and otherwise clears; the next request wakes the non-declaring build.
• I1 per launch path: for each of an exec child, a nested run unit, a generation swap, a seed, a command, a unit-backed start and a wake plan's fallback to a declaring GoodVersion after a failed first wake, a launch of a declaring build under 0 sets 1, completes the fence, and cancels the public request that caused it before the process starts.
• With the edge's reload held failing: an adoption of a declaring build times out with 1 written; a later recovery wake that selects a declaring GoodVersion is refused, no declaring process starts, and the public request gets the wake-failure answer.
• carlos ship refuses a capability block with both doc_engine and ports; a box with an existing port grant on a route that becomes internal removes the listener before the fence completes, and a direct connection to the port is refused afterwards.
• A public request admitted through a service row that is then deleted, held before dispatch, is cancelled by the next declaring launch's fence and never reaches the engine.
• On a box homing an internal group, a service row pointing at the engine's address, or any other app's instance, gets the internal refusal; a declaring launch on a box that has one is refused; static and status rows keep serving.
• Upgrade classification marks a route internal when its live version is non-declaring but another staged version of the app declares.
• A TCP-backed instance's minted hostname keeps serving across the upgrade; two TCP instances of one app with an unrecorded owner are refused until the reconciler records it.
• A doc-engine grant on a minted row of a non-declaring release makes the owner and every minted row internal, through the barrier.
• carlos engines arm succeeds on a box with nested callers and a unit-backed engine.
• A watch with a probe of /healthz reads it with a ticket; the same ticket on /other gets the internal refusal.
• A watch with a 10-minute timeout whose target turns healthy at minute 6 succeeds on a renewed ticket.
• A watch ticket on a public route is stripped and the watch behaves as today; on a grant-only internal route it reads the version.
• A socket call from a process in the instance's slice reaches the engine on another edge; from a process outside the slice (a flat child, a sibling slice) the connection is closed unread.
• A forged X-Carlos-Hop-Caller from a client, a door caller or an app through the socket never reaches the engine box's decision.
• A hop signed with a caller the engine box's table does not grant is refused even though the calling box admitted it.
• A restart with a persisted floor refuses a replayed older record.
• The per-app cap: the n+1th concurrent call from one app gets 429 while a second app's call proceeds.
• The socket survives a generation swap and is up before a seed runs; every nested unit kind sees CARLOS_ENGINES_SOCKET; tenant config cannot set it.
• A 45-second conversion completes; a declared 65 MiB body gets 413 before the engine sees a byte.

Simulation (#335 plan 1)

internal/sim/engines is a test-only model of this spec: 40 named rules, each with a mutation that turns it off, checked against the invariants INV-P (no public request is delivered to a declaring build), INV-I1, INV-AV (at the end, every row of a 0 group answers a public request), INV-G (an engine call reaches only the engine's exact route; no box admits one on a table that was never read, has expired, has had no good read for ten minutes or holds a record signed ten minutes ago; none is granted more than ten minutes plus one poll after a revoke, replay or not, nor by a box that has read the revoke), INV-ID (the caller the engine box records is the app whose code made the call) and INV-ARM (no unisolated tenant process on a box while an engine socket is open there). internal/sim/engines/mutate.sh 3000 runs each mutation through a 3000-seed random walk and through the rule's hand-scheduled witnesses; expected-outcomes.txt lists the rules caught only in combination.

Sweep after the final review, 2026-10-21, on the tree committed as "enginesim: clean baseline env, immediate listener checks, completed-transition coverage" (a337fb28). It adds three rules (new-group-fenced, clear-launch-lock, grant-freshness-bound), the freshness bound of ruling R5 (the walk now replays any older record at any time) and the oracle changes of the final review, so its counts are not comparable with the first sweep's (parent 4648305e). The unmutated sweep: 0 of 3000 seeds violated, 0 residuals. Result: 36 rules CAUGHT, 4 EXPECTED-MISS, none MISSED, ERROR or STALE (sim, plan 1, final review).

| Rule | Verdict | Sweep catches (of 3000) | First seed | Witness | |---|---|---|---|---| | set-before-repoint | CAUGHT | 2448 | 2 | violated | | fence-needs-reload | CAUGHT | 412 | 6 | violated | | fence-drains-admitted | CAUGHT | 217 | 31 | violated | | fence-box-wide | CAUGHT | 13 | 481 | violated | | gate-adopt | EXPECTED-MISS | 0 | - | held | | gate-swap | CAUGHT | 2294 | 2 | violated | | gate-fallback | CAUGHT | 265 | 18 | violated | | gate-recovery | CAUGHT | 262 | 5 | violated | | gate-seed | CAUGHT | 1711 | 2 | violated | | gate-cmd | CAUGHT | 1734 | 2 | violated | | gate-waits-when-written | CAUGHT | 287 | 6 | violated | | unreadable-declares | CAUGHT | 2012 | 3 | violated | | live-link-only-adoption | CAUGHT | 857 | 2 | violated | | mark-on-final-upstream | CAUGHT | 2832 | 1 | violated | | classify-null-refused | CAUGHT | 476 | 13 | violated | | classify-staged | CAUGHT | 66 | 8 | violated | | minted-follow-owner | CAUGHT | 802 | 1 | violated | | grant-marks-group | CAUGHT | 484 | 549 | violated | | static-minted-public | CAUGHT | 2981 | 1 | violated | | owner-nonempty-unique | CAUGHT | 42 | 2359 | held | | public-checks-live | CAUGHT | 851 | 2 | violated | | public-stops-group | EXPECTED-MISS | 0 | - | held | | public-holds-lock | CAUGHT | 62 | 37 | violated | | public-confirms-empty | CAUGHT | 49 | 8 | violated | | ports-excluded | CAUGHT | 1995 | 1 | violated | | dedicated-launch-check | EXPECTED-MISS | 0 | - | held | | dedicated-edge-refusal | CAUGHT | 1138 | 2 | violated | | ingress-precondition | CAUGHT | 1248 | 4 | violated | | cgroup-identity | CAUGHT | 171 | 32 | violated | | door-uid | CAUGHT | 85 | 43 | violated | | hop-caller-signed | CAUGHT | 137 | 41 | violated | | engine-box-recheck | CAUGHT | 44 | 79 | violated | | exact-route-engine | CAUGHT | 206 | 13 | violated | | floor-persisted | CAUGHT | 4 | 13 | violated | | stale-clears-grants | EXPECTED-MISS | 0 | - | held | | armed-refuses-unisolated | CAUGHT | 2738 | 2 | violated | | arm-checks-box | CAUGHT | 2321 | 1 | violated | | new-group-fenced | CAUGHT | 0 | - | violated | | clear-launch-lock | CAUGHT | 65 | 42 | violated | | grant-freshness-bound | CAUGHT | 455 | 2 | violated |

The four EXPECTED-MISS rules are each caught in combination, by their witness and by the sweep (200 seeds: 187, 12, 125 and 59 catches): gate-adopt with set-before-repoint (adoption's set-before is the same gate, reached earlier), public-stops-group with public-confirms-empty (step 3 refuses while the declaring generation still runs), dedicated-launch-check with dedicated-edge-refusal (the edge refuses the proxying row once the group is internal), and stale-clears-grants with grant-freshness-bound (a table's record was signed no later than its last good read, so the signed_at bound empties a stale table first). owner-nonempty-unique's witness needs dedicated-edge-refusal gone too, so it holds alone; the sweep catches the rule alone through INV-I1 (a misgrouped 0 row makes the owner's group read public). new-group-fenced is caught by its witness only: the walk's dedicated launch check refuses every declaring launch in a new group beside the starting routes, so only a box that homes nothing reaches it.

9. Docs

docs/site/cli.md: a carlos engines section. The README's doc-engine passage gains the public refusal and the deploy order. A short "Calling an engine from your app" passage with a Go example (a unix-socket transport, http://gotenberg/forms/chromium/convert/html). All user-facing strings go through copy review before they ship.

10. Provider boundary

Core only. The mirror reads a release.Store (S3 or a directory); signing uses the root release key; the hop is the existing mesh; the peer check is Linux, as the whole nested layout already is. On AWS with region HA a hop within one region uses private addresses; everywhere else it crosses the public internet under TLS and the hop MAC, as backhaul does today. A non-AWS deployment loses nothing. "Fleet-internal" means only the platform can call it, not only over a private network.

11. Calls I made (for review)

• Engine selection by Host: name on one socket, so a client's base URL is http://<name>.
• Limits: 64 MiB body, 5 minutes end to end, per-app cap default 4, per serving box.
• Engine calls skip the password gate and the cache; the import gate, the meter and analytics run as for any request.
• No owner exception on internal routes: the engine app grants itself.
• Tunnel-homed engines are refused (502), not supported.
• Flat instances cannot call engines at all (§6.2); host-app processes under their own uid use the door, enabled by the operator.
• Engine sockets only on boxes an operator armed; an armed box refuses flat, foreign and worker launches rather than closing sockets afterwards.
• A route never becomes public automatically; carlos route --public is the only way back, and it stops the group.
• One internal group per box, enforced: a second engine needs a second box.
• Unclassified routes are refused publicly (fail closed) even though that can make a route with no readable manifest unreachable while the bucket is down.
• The deploy watch ticket rather than skipping verification for engines.
• The engine box re-checks the grant (G4) rather than trusting the calling box.

12. Disposition of review findings

| Finding | Disposition | | --- | --- | | R1-1 flat tenant borrows a nested socket | Accepted: peer cgroup check (§5.4) | | R1-2 aliases and path routes stay public | Accepted: mark on the route, checked on the final upstream (§5.1) | | R1-3 new box starts public | Accepted: mark from adoption, no bucket at serve time (§5.1, R1) | | R1-4 revoked grant resurrected after restart | Accepted: durable floor; fresh-box residual stated (§5.3) | | R1-5 door exceptions | Accepted: owner exception dropped, door-to-engine transition defined (§5.4) | | R1-6 socket lifecycle | Accepted: instance-level owner (§5.4) | | R1-7 transport and body | Accepted: engine transport, streamed body with abort (§5.5) | | Builder Q1 unit-backed callers | Ruling R2: out of scope | | Builder Q2 door for flat apps | Superseded by R2-3: flat instances cannot call engines; host-app door is an operator step (§6.2) | | Builder Q3 engine owner's own processes | Grant itself (§5.1) | | Builder Q4 gate, meter | §5.4 | | Builder Q5 socket lifetime | §5.4 | | Builder Q6 several homes, tunnels | Per-box cap; tunnels refused (§5.4, §5.5) | | R2-1 stamp does not precede serving | Accepted: set before, clear after, edge-loaded barrier (§5.1) | | R2-2 upgrade serves unclassified engines | Accepted: NULL state, classification before listeners, NULL refused (§5.1) | | R2-3 flat callers borrow a door key | Accepted: door engine calls need a non-carlos uid; flat cannot call (§5.4, §6.2) | | R2-4 deploy watch cannot verify | Accepted: watch ticket (§5.8) | | R2-5 :80 on non-home boxes; unknownHost redirects | Accepted: plain 404 refusal; no-redirect scoped to the home box (§5.1) | | R3-1 requests admitted before the mark | Accepted: the acknowledgement waits for cancelled pre-mark requests to finish (§5.1) | | R3-2 flat tenant injects code into a nested caller | Accepted, then widened by R4-1: armed boxes (§6.2) | | R3-3 grant-only routes lack a watch ticket | Accepted: every deploy target carries one (§5.8) | | R4-1 foreign instances and workers are unisolated too | Accepted: positive armed state, unisolated launches refused while armed (§6.2) | | R4-2 a path map widens a grant to another engine | Accepted: engine calls resolve by exact hostname to the engine's own route (§5.4) | | R4-3 watch tickets expire mid-watch | Accepted: renewal every 4 minutes (§5.8) | | R5-1 platform-minted rows never classify | Accepted: execution owner's mark and epoch (§5.1) | | R5-2 arming rejects unit-backed routes | Accepted: exec-backed routes by layout, unit-backed by backing (§6.2) | | R5-3 no source for the probe path | Accepted: the ticket signs the watcher's probe path (§5.8) | | R6-1 grant on a minted row ignored | Accepted: grants anywhere in the group mark the group (§5.1) | | R6-2 static platform domains refused | Accepted: inheritance only for minted instance rows (§5.1) | | R7-1 clear-after reopens a still-serving engine under rollouts | Accepted in rev 8, superseded in rev 9 by the operator clear (§5.1) | | R7-2 TCP-backed minted rows have no owner | Accepted: owner recorded on the row; unique non-empty fallback, TCP included (§5.1) | | R8-1 upgrade classifies a running declaring build public | Accepted: staged declaring versions classify 1; residual stated (§5.1) | | R8-2 a sleeping unit-backed route cannot clear | Accepted, with R7-1: no automatic clear; --public stops the group and checks cgroups (§5.1) | | R9-1 --public then a fallback to a declaring GoodVersion | Accepted: invariant I1 enforced at every launch (§5.1) | | R10-1 a written but unfenced 1 lets a declaring launch through | Accepted: I1 is about the fenced mark; every declaring launch waits for the fence (§5.1) | | R11-1 a granted port bypasses the edge | Accepted: doc_engine + ports refused at ship; no listener for internal groups (§5.1) | | R12-1 an engine binds its own public port | Ruling R3: dedicated box; ingress beyond the edge's ports is a stated, surveyed precondition (§5.9) | | R12-2 a separate service row proxies to the engine | Ruling R3: no proxying row outside the group on an engine's box, enforced by the launch gate and the edge (§5.9) | | R13-1 another box proxies privately to the engine | Accepted within R3: ingress precondition covers private sources; survey looks for such rows (§5.9) | | R13-2 a request admitted via a deleted row | Accepted: the fence is box-wide on an engine's box (§5.9) |