carlos / platform Public

branch:

Clone

git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1	# Fleet-internal engines: a doc engine nobody on the internet can call
2
3	Date: 2026-10-21
4	Revision: 14. Approved by Paul, 2026-10-21.
5	Discussion: amadan carlos/platform #335. Related: #328 (per-app egress allow).
6
7	- Rev 14 applies Astra round 13's two P1s (`CODEX-FINDINGS-ENGINES-R13.md`)
8	as small fixes within R3, per R4 with no further round. The ingress
9	precondition covers private sources too: another box reaching the
10	engine's backend directly (R13-1). The fence on an engine's box is
11	box-wide, cancelling every non-engine proxy request admitted before it,
12	including one held by a row since deleted or replaced (R13-2). Marked
13	(codex R13-n).
14	- Rev 13 answers Astra round 12 (`CODEX-FINDINGS-ENGINES-R12.md`,
15	reject, two P1s: an engine binding its own high port, and a separate
16	public route row pointing at the engine's backend) with Paul's ruling R3:
17	an engine's home box is dedicated, and the platform enforces it (§5.9).
18	Ruling R4 sets the review end: one more round, then Paul's read. Marked
19	(codex R12-n).
20	- Rev 12 answers Astra round 11 (`CODEX-FINDINGS-ENGINES-R11.md`,
21	reject, one P1: a declaring release that also declares `ports` keeps a
22	granted TCP listener the edge never sees). `doc_engine` and `ports` are
23	refused together at ship, as `doc_engine` and `host_role` already are, and
24	a box renders no port listener for an internal group (R11-1). §5.1 now
25	also says why systemd's own starts (socket activation, `Restart=always`)
26	keep I1. Marked (codex R11-1).
27	- Rev 11 answers Astra round 10 (`CODEX-FINDINGS-ENGINES-R10.md`,
28	reject, one P1: a mark written but never fenced, after a failed reload
29	and a timed-out adoption, let a recovery launch start a declaring build
30	while a public request was admitted). I1 now speaks of the fenced
31	mark, not the registry's, and every declaring launch requires the fence
32	whether or not it finds `1` already written (R10-1). Marked
33	(codex R10-1).
34	- Rev 10 answers Astra round 9 (`CODEX-FINDINGS-ENGINES-R9.md`, reject,
35	one P1: after `--public`, a failed first wake falls back to a declaring
36	`GoodVersion`). Instead of a fourth path-specific rule, §5.1 now states
37	the invariant every path must keep, I1, and enforces it at the one
38	place every path passes through: the launch. A launch of a declaring
39	build sets the mark and completes the barrier first, cancelling the
40	public request that caused it (R9-1). Marked (codex R9-1).
41	- Rev 9 answers Astra round 8 (`CODEX-FINDINGS-ENGINES-R8.md`, reject,
42	a P1 and a P2, both accepted) by changing approach rather than adding a
43	third proof: a route never becomes public automatically. Clearing the
44	mark is an operator verb that stops the group under the adoption lock and
45	checks its cgroups are empty (R7-1, R8-1, R8-2). Upgrade classification
46	also marks a route internal when any staged version of its app declares
47	`doc_engine`. Marked (codex R8-n).
48	- Rev 8 answers Astra round 7 (`CODEX-FINDINGS-ENGINES-R7.md`, reject,
49	a P1 and a P2, both accepted). Clearing the mark waits for proof that no
50	process of a declaring build is left, not for adoption's return, which
51	under generation rollouts only schedules the swap (R7-1). A minted row
52	records its owner explicitly; the fallback match needs a unique non-empty
53	backend, TCP included (R7-2). Marked (codex R7-n).
54	- Rev 7 answers Astra round 6 (`CODEX-FINDINGS-ENGINES-R6.md`, reject,
55	a P1 and a P2 in rev 6's minted-row rule, both accepted). Inheritance is
56	scoped to minted `instance` rows, static and service platform rows keep
57	`0` (R6-2), and a `doc-engine` grant on a minted row marks its execution
58	owner, so the whole group goes internal together (R6-1). Marked
59	(codex R6-n).
60	- Rev 6 answers Astra round 5 (`CODEX-FINDINGS-ENGINES-R5.md`, accept
61	with fixes, three P2s, all accepted). Platform-minted rows take their
62	execution owner's mark and barrier (R5-1). Arming judges exec-backed
63	routes by layout and admits unit-backed ones (R5-2). The watch ticket
64	signs the probe path the watcher uses (R5-3). Marked (codex R5-n).
65	- Rev 5 answers Astra round 4 (`CODEX-FINDINGS-ENGINES-R4.md`, reject,
66	two P1s and a P2, all accepted). "No flat route" becomes a positive,
67	operator-armed box state that also excludes `foreign` instances and
68	workers, and refuses unisolated launches while armed (R4-1). An engine
69	call is dispatched only to the engine hostname's own route, never through
70	a path map (R4-2). Watch tickets are renewed for as long as the watch runs
71	(R4-3). Marked (codex R4-n).
72	- Rev 4 answers Astra round 3 (`CODEX-FINDINGS-ENGINES-R3.md`, reject,
73	two P1s and a P2, all accepted). The barrier's acknowledgement now also
74	covers public requests admitted before the mark: the edge cancels them and
75	waits for them to finish before it acknowledges (R3-1). Engine sockets
76	open only on a box with no flat instance at all, because a flat tenant can
77	write a nested sibling's files and run code inside its slice (R3-2). Every
78	deploy target gets a watch ticket, not only declaring releases (R3-3).
79	Marked (codex R3-n).
80	- Rev 3 answers Astra round 2 (`CODEX-FINDINGS-ENGINES-R2.md`, reject,
81	three P1s and two P2s, all accepted). The mark gains an "unclassified"
82	state and a serving barrier: set before the release switch, cleared after
83	it, and observed by the edge before adoption proceeds (R2-1). Every route
84	is classified before the upgraded edge opens a listener (R2-2). The door
85	carries engine calls only for callers with their own uid, so flat
86	instances cannot call engines at all (R2-3). `carlos deploy` watches an
87	internal engine with a console-signed watch ticket (R2-4). The refusal is
88	a plain 404 that never redirects, and the no-redirect promise on :80 is
89	scoped to the home box (R2-5). Marked (codex R2-n).
90
91	- Rev 2 answers Astra round 1 (`CODEX-FINDINGS-ENGINES-R1.md`, reject,
92	four P1s, all accepted) and the builder check, and records Paul's two
93	rulings of 2026-10-21 (§2.1). What makes a route internal moves from a
94	console record to the route itself (R1), which closes R1-3 (a new box
95	starts public) and R1-2 (aliases and path routes) by construction. The
96	socket's caller is proved by the peer's cgroup, not the socket's path
97	(R1-1). The grant table keeps a durable revision floor (R1-4). The door's
98	owner exception is dropped and the door-to-engine transition is defined
99	(R1-5). The socket gets an instance-level owner (R1-6) and engine calls
100	their own transport (R1-7). Each change is marked (codex R1-n).
101
102	## 1. Intent
103
104	The `doc_engine` capability (`internal/caps/render.go`, README "App
105	capabilities") fences a conversion engine's egress away from private
106	networks. It says nothing about who can call the engine. A doc-engine
107	route is a route like any other: its box's public edge serves it on :443,
108	any other box backhauls a public request for it, and the only gate is
109	whatever auth the engine applies itself (Gotenberg: basic auth, often
110	none). An operator who skips that auth, or leaks its password, publishes a
111	Chromium and LibreOffice renderer to the internet: free compute for anyone,
112	and an SSRF engine whose remaining reach is the public internet and its own
113	box's loopback.
114
115	This spec makes every doc engine fleet-internal. Its routes answer only
116	calls the platform carried from an app an operator granted, and an isolated
117	tenant reaches it through a socket inside its own instance, with no private
118	address opened in its fence.
119
120	## 2. Goals, non-goals, anti-goals
121
122	Goals
123
124	- G1. A route whose live release declares `doc_engine`, or which holds the
125	`doc-engine` grant, is refused to every public request, through every
126	hostname, alias, wildcard or path that resolves to it, on every listener,
127	except the ACME challenge.
128	- G2. An operator grants an app the use of a named engine, per app. Every
129	nested instance of the app, on any box and in every unit kind its code
130	runs as (run, generation, seed, `carlos run` command, legacy sidecar),
131	can then call it, and so can the app's own host processes that hold a
132	door key and run under their own uid (§5.4). A revoke takes effect within
133	§6.4's bounds.
134	- G3. A nested instance calls an engine through a unix socket in its own run
135	directory. Its `IPAddressDeny=` fence is unchanged.
136	- G4. The engine box knows which app made each call and refuses a call from
137	an app without a grant, even if the calling box admitted it.
138	- G5. One app cannot starve the others: the engine box caps each app's calls
139	in flight.
140	- G6. Provider-neutral (§10).
141
142	Non-goals
143
144	- Unit-backed callers (`carlos-app@`, DynamicUser). Ruling R2: later, if
145	a caller appears. They have no instance run directory and cannot be
146	enrolled for the door. Their routes can still be engines.
147	- Flat-layout instances as callers (§6.2). They share uid carlos with
148	every other flat tenant, so nothing they hold can prove which app they
149	are. An app that needs an engine moves to the nested layout.
150	- Metering or billing engine use. Member-requested grants.
151	- A general internal-route product for arbitrary services.
152	- Reaching an engine over TCP from a nested instance: its fence denies
153	loopback and every private range, and `IPAddressAllow=` cannot name a
154	port (#328's limit 1).
155	- Replacing #328. An engine outside the platform still needs #328's
156	egress allow.
157
158	Anti-goals
159
160	- A window in which an engine route is public: a new or replacement box, a
161	restarted edge, a stale mirror or a box that cannot reach the bucket must
162	never serve one to the public.
163	- A caller identity a tenant can forge or borrow: through a header, another
164	instance's socket, a sibling's process or the door.
165	- Opening any private address in an isolated instance's fence.
166
167	### 2.1 Rulings (Paul, 2026-10-21; do not reopen)
168
169	- R1. Declaring `doc_engine` makes a route internal. No opt-out and no
170	separate registration step for it. A doc-engine route in public use today
171	goes dark when its home box takes this build; its callers move to the
172	socket or door first (§7).
173	- R2. Unit-backed apps are not callers in this spec. Later, if a caller
174	appears.
175	- R3. An engine's home box is dedicated, and the platform enforces it
176	(§5.9). Public ingress beyond the edge's own ports is a stated box
177	precondition, not a platform-installed firewall.
178	- R4. Review end: one Astra round on rev 13, small fixes, then Paul's
179	read. The plan's first task is a simulation of I1 and §5.9 across every
180	launch and ingress path.
181	- **R5. Grant records go stale ten minutes after signing (Paul,
182	2026-10-21).** An edge treats an `engines.json` record whose `signed_at`
183	is more than 10 minutes old as stale, so its grant table is empty; the
184	console re-signs every 2 minutes and on every change (§5.2, §5.3) (sim,
185	plan 1, final review).
186
187	## 3. What exists today, and what this builds on
188
189	Cited from main at `3d444a1b`.
190
191	- The backhaul hop (`internal/edge/edge.go` `backhaul`/`backhaulTo`,
192	`homemap.go`): one hop, ever, from the box a request reached to the box
193	that homes the host. It dials the home box's `private_ip` when region HA
194	is on and both boxes share a region and placement, its public address
195	otherwise (`homemap.go` `dialAddress`), on :443 with public-CA TLS and SNI
196	set to the real hostname, with a 30 s response-header timeout and a
197	transport shared per destination (`edge.go:2825`, `:2938`). It is
198	authenticated by `X-Carlos-Hop` and `X-Carlos-Hop-Auth`, an HMAC-SHA256
199	over host and timestamp (±30 s) under the deployment's mesh key
200	(`mesh.go`). The hop carries no caller (`acceptHop`).
201	- The edge-local door (#175 P7, `internal/edge/door.go`): a unix socket
202	(`CARLOS_EDGE_DOOR`) through which an app's own processes reach the app's
203	hosts on any box with an app-scoped door key (`internal/doorkey`): a
204	console-minted, root-signed, revisioned, expiring record under
205	`control/door-keys/`, mirrored by every edge every 30 s with a ten-minute
206	stale bound and an in-memory revision floor (`doorkeys.go:22`). The door's
207	mark is a context value; it does not cross a hop.
208	- The per-instance control socket (`internal/edge/purge.go`
209	`ListenControl`), opened per generation for the run role only
210	(`prepare.go:111`) and closed with the generation (`unitproc.go:203`).
211	Every nested unit binds its whole run directory
212	`/run/carlos/instances/<iid>` (`carlos-instance@.service:53`; generations
213	through `instancectl/gen.go:48`). Every nested unit kind runs in
214	`Slice=carlos-instances-<iid>.slice` (`instancectl.SliceName`; the four
215	templates and the generation drop-in). The instance ID is at most 40
216	bytes (`provision.go:2918`).
217	- Peer checks on a unix socket: the root control service accepts a
218	connection only from SO_PEERCRED uid carlos and the edge's MainPID
219	(`instancectl/server.go` `PeerOK`).
220	- Capabilities: the release's manifest declares them; adoption stages
221	the manifest beside the live version (`release/adopt.go`
222	`stageAuthorized`, `capability_pass` reads it via
223	`stagedManifestOfLiveVersion`); grants are box-local in `routes.caps`
224	(`registry.SetCaps`), set with `carlos route --host X --grant doc-engine`.
225	- Flat tenants share uid carlos with nested ones and can write their
226	siblings' directories (`layout/preflight.go:181`).
227
228	## 4. The shape, in one paragraph
229
230	A route becomes internal on the box that serves it when its live
231	release declares `doc_engine` or it holds the `doc-engine` grant; the box
232	records that in its own registry and refuses the route to every public
233	request. An operator names an engine (`gotenberg` → its hostname) and
234	grants it to apps; the console publishes the set as a root-signed
235	`control/engines.json`. A granted nested instance's code sends plain HTTP
236	to `$CARLOS_ENGINES_SOCKET` with `Host: gotenberg`. Its box's edge proves
237	the caller from the connecting process's cgroup, checks the grant, and
238	relays the request to the engine's home box over the backhaul hop, now
239	signed with the caller's account, app and engine. The engine box verifies
240	the signature, checks the grant again, applies the per-app cap and serves
241	the route. A door caller does the same with `Host: gotenberg`.
242
243	## 5. The parts
244
245	### 5.1 The internal mark (every box, on its own routes)
246
247	(codex R1-2, R1-3, R2-1, R2-2) Internal is a property of a route on the
248	box that serves it, never of a hostname and never of a bucket record.
249
250	The column. `routes.internal` has three states: `NULL` (unclassified),
251	`0` (public) and `1` (internal). Routes of a kind that serves no release
252	(`static`, `service`, the status host, sidecars) are `0` by construction.
253	An `instance` route is classified from the manifest of its live release:
254	`1` if it declares `doc_engine`, `0` otherwise. The `doc-engine` grant also
255	sets `1`, whatever the manifest says. The edge treats `NULL` like `1`.
256
257	Platform-minted rows (codex R5-1, R6-1, R6-2). A platform domain is
258	its own registry row (`Platform: true`, `domains/reconcile.go:881`),
259	copying the source's kind and backend but no version, and adoption skips it
260	(`release/adopt.go:1210`).
261
262	- A minted `static` or `service` row (including a static canary on its own
263	channel, `reconcile.go:368`) is `0` by construction, like any row of
264	those kinds. The rest of this paragraph does not apply to it.
265	- A minted `instance` row's execution owner is recorded on the row: the
266	domains reconciler writes the source's host into a new `routes.owner`
267	column when it mints (`mintedRoute` has the source in hand) and backfills
268	it on its next pass for rows minted earlier (codex R7-2). Until a row
269	has an owner recorded, the edge falls back to matching the non-platform
270	`instance` row with the same account and app and the same non-empty
271	backend: `Socket`, `Addr` (TCP-backed rows, `cli.go:172`,
272	`domains/reconcile.go:835`), or a fleet-placed `Region`. Empty values never
273	match, and a match that is not unique counts as no match. The edge
274	resolves owners at `Reload`.
275	- The owner and every minted row that shares it form one group with
276	one mark and one admission epoch. The group is internal when the owner's
277	own classification is `1` **or any row in the group holds the
278	`doc-engine` grant**. The grant CLI already permits a grant on a plain
279	minted row (`cli.go:1432`); such a grant marks the whole group, and
280	`--grant` and `--revoke` on any row of the group run §5.1's barrier for
281	the group.
282	- A minted `instance` row whose owner the edge cannot find, or finds
283	ambiguously, is treated as `NULL` and refused, and logged by name, until
284	the reconciler records its owner. `--grant doc-engine` on a minted row
285	whose owner is unresolved or ambiguous is refused; the row stays
286	unclassified and refused (sim, plan 1).
287
288	The invariant (codex R9-1).
289
290	> I1. A process of a build whose manifest declares `doc_engine` starts
291	> only after the edge has fenced its group as internal: the edge's
292	> fenced counter (below) has reached a registry change that set the
293	> group's mark to `1`, and no later change has cleared it.
294
295	What protects the public is the edge's serving map, not the registry row
296	(codex R10-1): a `1` written while the edge's `Reload` fails
297	(`edge.go:925`, `:993`) refuses nobody. So the registry mark is the
298	intent, and the fence is the fact I1 is about. A group closed as
299	unclassified (`NULL`) counts as fenced only once the edge has loaded that
300	`NULL`: the change that closed it is recorded for the group, and the reload
301	that reached it drained every older public request on the box (the
302	upgrade's restart does both). A newly created or unseen row never counts as
303	fenced, so a first declaring launch writes `1` and waits for the fence like
304	any other (sim, plan 1, final review).
305
306	Rounds 2 to 9 each found one more path by which a declaring build could
307	run under a public mark: adoption's order, requests admitted early, a
308	pending generation swap, an un-cycled hibernating route, a sleeping unit,
309	and wake recovery falling back to `GoodVersion`
310	(`rollout/wakeplan.go:76`, `:111`; `activator/wakeplan.go:448`). So I1 is
311	enforced where every one of those paths ends: the launch gate. Every
312	activator launch path (an exec child, a nested run or generation unit
313	through the control service, a unit-backed `systemctl start`, a seed, a
314	`carlos run` command, a swap's incoming generation, a wake plan's
315	fallback or recovery build) reads the manifest of the build it is about to
316	start. If that build declares `doc_engine`, the launch writes `1` if it is
317	not already written, and then always waits for the edge's fence to
318	cover the group's current mark (below), which cancels every public request
319	admitted before it, including the one whose wake triggered the launch.
320	Only then does it start the process. A launch that finds `1` written but
321	not yet fenced (a timed-out adoption, a reload that keeps failing) waits
322	the same way; if the fence does not arrive within the bound, the launch is
323	refused, the wake fails, and the request gets the activator's ordinary
324	wake-failure answer. A launch whose build manifest it cannot read is
325	treated as declaring. Adoption's
326	set-before is the same gate, reached earlier. The gate holds a per-group
327	lock from its check through process creation, and `--public` holds the same
328	lock for its whole run, so no launch can pass the gate between `--public`'s
329	empty check and its write of `0` (sim, plan 1, final review).
330
331	`--public` (below) is the only writer of `0`, and it checks I1 directly:
332	the group is stopped and its cgroups empty. Any later declaring launch
333	re-marks the group through the gate. The one place I1 is not enforced is
334	stated as a residual under "Classification before serving".
335
336	Starts systemd makes on its own. A unit-backed route's unit is also
337	started by socket activation (`carlos-app@.socket`) and restarted by
338	`Restart=always` (`carlos-app@.service:50`), without the activator. Both
339	execute `/opt/carlos/live/<host>` (`:42`), the live version. Adoption
340	passes the launch gate before it repoints the live link to a declaring
341	build, and `--public` clears only when the live build does not declare, so
342	whatever systemd starts is either fenced or non-declaring. The builder
343	keeps that true by construction: nothing but adoption (through the gate)
344	and `--public` (through its checks) may move a live link.
345
346	The edge is the only ingress (codex R11-1). I1 protects the public only
347	if every way in goes through the edge. A granted `ports` capability is a
348	`ListenStream=<port>` socket the edge never sees (`caps/render.go:286`,
349	`:319`), and a hibernating route's port starts the unit without the
350	activator (`caps/converge.go:241`). So:
351
352	- At ship, `doc_engine` and `ports` are refused together, beside the
353	existing `host_role` refusal (`caps/caps.go:382`): a conversion engine is
354	reached over HTTP through the platform, never on a port of its own.
355	- On the box, the capability pass renders no `ListenStream` for any
356	route of an internal group, whatever its grants, and removes an existing
357	one (stopping the socket unit) before the launch gate's fence completes
358	for that group; it logs the grant it is ignoring with the revoke command.
359	A `ports` listener is removed on every write of `1` or `NULL` (a grant,
360	the upgrade, the launch gate), not only at the launch gate (sim, plan 1).
361
362	Setting the mark: the serving barrier (codex R2-1). Adoption
363	repoints the live symlink (`release/adopt.go:1751`), restarts
364	(`adopt.go:1831`) and records the version afterwards (`adopt.go:2051`),
365	and the edge serves an in-memory map refreshed on a 2 s registry poll
366	(`edge.go:2298`). So the order is explicit:
367
368	- Becoming internal (the incoming release declares `doc_engine`, the
369	route is `0`): adoption writes `1`, then waits for the edge's
370	acknowledgement, then repoints and restarts. The edge publishes the
371	registry change it has fenced (a counter in a file under
372	`/run/carlos`), and it advances that counter only after both:
373	1. `Reload` has swapped in a map carrying the mark (a failed `Reload`
374	advances nothing); and
375	2. **every request admitted to that route as a non-engine call before the
376	swap has finished (codex R3-1).** Each such request holds a reference
377	on the route's public-admission epoch from the moment it passes the
378	internal check until its response ends, including while it waits in
379	the activator's queue for a wake (`activator.go:1554`, `:5404`). Loading
380	a `0`→`1` change bumps the epoch and cancels the contexts of every
381	request holding the old one, which also drops it from the activator's
382	queue; the counter advances when the last of them has released. A
383	request admitted after the swap sees `1` and is refused.
384
385	The wait is
386	bounded (30 s); on timeout adoption does not repoint this pass, the mark
387	stays `1`, and the next pass retries. The old, non-engine release is
388	refused publicly for that interval, which costs availability, never
389	exposure.
390	- Becoming public is never automatic (codex R7-1, R8-1, R8-2). No
391	adoption, revoke, rollback or pass ever writes `0` over `1`. Rounds 6 to
392	8 each found a new interleaving in which an automatic clear raced a
393	process of the declaring build that was still running (a generation
394	swap still pending, an awake hibernating unit adopted without a restart,
395	a sleeping unit with no main process to vouch for). So clearing is an
396	operator act whose preconditions are checked, not inferred:
397
398	carlos route --host <host> --public
399
400	On the box, holding the group's lock for its whole run (the adoption
401	lock, so the live version cannot move under it, and the launch gate's,
402	so nothing starts between step 3 and step 4; sim, plan 1, final review),
403	it:
404	1. refuses unless the live release's manifest is readable and does not
405	declare `doc_engine`, and no row of the group holds the grant;
406	2. stops the whole group (every unit and generation of the owner, through
407	the activator and the control service);
408	3. confirms every cgroup of the group is empty (`carlos-instances-<iid>.slice`,
409	`carlos-app@<host>.service`, `carlos-exec@<host>`), and refuses if it
410	cannot;
411	4. writes `0` and releases the lock.
412
413	Whatever starts afterwards passes the launch gate: the live version,
414	which step 1 proved non-declaring, starts under `0`; a declaring build
415	chosen by wake recovery (`GoodVersion`) re-marks the group first and the
416	triggering public request is cancelled (I1). The route's next request
417	wakes it. The cost is a short stop on a rare operation; a log line on the
418	capability pass names every internal route whose live release no longer
419	declares `doc_engine` and holds no grant, with this command.
420	- Rollback to a declaring release is an adoption like any other: set
421	before. Rollback away from one leaves the mark set.
422	- The same barrier applies to a grant: `carlos route --grant doc-engine`
423	writes `1` and returns once the edge has loaded it (or says it timed out).
424
425	Classification before serving (codex R2-2, R8-1). The new column arrives
426	`NULL` for every existing route. Before the upgraded agent opens any edge
427	listener, it classifies every `NULL` instance route (minted rows follow their
428	owner):
429
430	- `1` if any row of the group holds the grant, or the live version's
431	manifest declares `doc_engine`, or **the manifest of any other version of
432	the app staged on the box declares it**. The last clause covers a box
433	whose live symlink has moved off a declaring build that is still running
434	(an awake hibernating route is repointed without a restart,
435	`release/adopt.go:1174`).
436	- `0` only if the live manifest is readable, none of the above holds.
437	- Otherwise (the live manifest is unreadable from the staged sibling and
438	from the bucket, as adoption's back-fill reads it) the route stays `NULL`,
439	is refused publicly, and is named in a log line every pass until the
440	capability pass classifies it. That is the fail-closed choice.
441
442	A route that turns out `1` only because an old staged version declared is
443	cleared with `--public` like any other. Residual, stated: a declaring
444	version whose staged manifest failed to write, still running across the
445	upgrade after the live symlink moved to a non-declaring version, would be
446	classified `0`; it needs a sibling write failure on a doc-engine release
447	and a pending, un-cycled switch away from it at the moment of the upgrade.
448	A route the agent creates later (provisioning, a move) is classified by its
449	first adoption, which runs before the route is served.
450
451	What the edge does with it (`ServeHTTP`):
452
453	- The check runs on the final resolved upstream's route: after the
454	route lookup and after the login POST's re-resolution, and before the
455	import gate, the password gate, the meter, the cache, an upgrade and the
456	activator's wake. Aliases (`edge.go:929`) and path-mapped hosts
457	(`edge.go:1364`) copy the target's upstream, so every name and path that
458	resolves to an internal route is covered, in any spelling the lookup
459	accepts.
460	- An internal route admits only an engine call (§5.4). Everything else gets
461	the internal refusal: a plain `404 not found`, `Cache-Control:
462	no-store`, never the pending page and never the fallback redirect
463	(`unknownHost` does both, `edge.go:2205`, so it is not reused).
464	- There is no owner exception: the engine app's own processes reach it by
465	granting the engine to their own app.
466	- :80 on the home box serves the ACME challenge for an internal route's
467	hostnames and the internal refusal for everything else: no redirect.
468	:80 on another box knows only the home map, so it redirects to https
469	as it does for any host homed elsewhere (`edge.go:2262`), and the https
470	request is then backhauled and refused (codex R2-5). The redirect names
471	nothing DNS and certificate transparency do not already publish.
472	- `Allowed` (certificate issuance) is unchanged: the hop's TLS needs a
473	publicly trusted certificate on the home box.
474	- A box that does not home the host backhauls a public request as an
475	ordinary hop with no caller, and the home box refuses it. A home box on
476	an older build is the one gap, closed by deploy order (§7).
477
478	### 5.2 Engine names and grants (console)
479
480	Operator-only CLI, gated like `domains attach --platform` (an owner of the
481	ops account or an address on `CARLOS_SYSTEM_OPERATORS`,
482	`Server.systemOperator`):
483
484	carlos engines add <name> --host <hostname> [--per-app <n>]
485	carlos engines remove <name>
486	carlos engines grant <name> --app <account>/<app>
487	carlos engines revoke <name> --app <account>/<app>
488	carlos engines list
489
490	- `<name>` matches `^[a-z][a-z0-9-]{0,31}$`, unique in the deployment, never
491	contains a dot. It is what apps put in `Host:`.
492	- `--host` is an exact hostname the engine route serves on its home box.
493	The console cannot check it (it holds no route registry); §5.7 reports
494	mismatches from the box.
495	- `--per-app` is the engine box's in-flight cap per calling app. Default 4,
496	range 1 to 64.
497	- `remove` refuses while grants remain. Removing a name changes nothing
498	about whether the route is internal (§5.1).
499	- Every change is audited with `store.RecordEvent`, like `store grant`.
500	- `list` shows each engine's name, host, cap and grantees to an operator. A
501	member sees only the engine names granted to apps in their accounts.
502
503	The console's database is the authority. It publishes the whole set as one
504	record, `control/engines.json`, signed with the root release key under its
505	own domain (`carlos engines v1\n`), with a strictly increasing revision and
506	`expires_at` 72 hours out. It re-signs every 2 minutes and on every change,
507	and an edge treats a record whose `signed_at` is more than 10 minutes old as
508	stale (ruling R5; sim, plan 1, final review). It
509	publishes an empty set (revision 1) the first time the feature is on, so a
510	box can tell "no engines" from "not read yet".
511
512	{
513	"v": 1, "revision": 17, "signed_at": "...", "expires_at": "...",
514	"engines": [
515	{ "name": "gotenberg", "host": "gotenberg.example.com", "per_app": 4,
516	"grants": [ { "account": "acct1", "app": "shop" } ] }
517	]
518	}
519
520	### 5.3 The grant mirror (every edge)
521
522	(codex R1-4) The mirror verifies signature, shape, domain and expiry as
523	the door-key mirror does, and adds a durable revision floor:
524
525	- The edge persists the highest revision it has admitted, with the record
526	itself, to its data directory (write to a temporary file, fsync, rename).
527	It loads both at start, before any engine socket opens. The mirror also
528	persists the time of its last good read, so a restart is not a fresh
529	read (sim, plan 1).
530	- A record whose revision is below the floor is refused, at runtime and
531	after a restart. An equal revision must be byte-identical.
532	- Stale (no good read for ten minutes, or a record signed more than ten
533	minutes ago), expired, or never read: the grant table is empty, so
534	every engine call is refused. A persisted record that has since expired
535	or gone stale counts as expired. The mirror refuses a stale record like
536	an expired one (ruling R5; sim, plan 1, final review).
537	- If the floor cannot be persisted, the edge keeps refusing engine calls
538	until it can, and logs it once per minute.
539	- The freshness bound (ruling R5; sim, plan 1, final review). A bucket
540	writer can replay an older signed record: before any edge has read a
541	revoke, or to a box with no persisted floor (new or rebuilt). Every
542	record that grants a revoked key was signed before the revoke, and goes
543	stale ten minutes after its signing, so a replay keeps a revoked grant at
544	most until the replayed record's `signed_at` plus ten minutes. The
545	worst-case revoke delay is therefore about ten minutes plus one poll,
546	replay or not; a box that has read the revoke refuses the replay at once
547	(its floor). The cost: engine calls fail once the console has been down
548	for ten minutes.
549
550	### 5.4 The call path
551
552	The engine socket (nested instances; codex R1-1, R1-6). An
553	instance-level listener manager in the edge owns one socket per nested
554	instance: `/run/carlos/instances/<iid>/engines.sock` (at most 75 bytes).
555
556	- Lifetime: opened when the instance's run directory is prepared for
557	its first unit of any kind (a seed, a run, a command), kept across
558	generations and restarts of the instance's units, closed only when the
559	route leaves the box (removal, a move's commit, a reset). It is not tied
560	to a generation and never closed by one. It exists whether or not the
561	app holds a grant, so a later grant needs no restart. Answering on a
562	sleeping instance's socket does not wake the instance. All of this holds
563	only on an armed box (§6.2); elsewhere no engine socket is
564	open, and `CARLOS_ENGINES_SOCKET` still names the path, so an app finds
565	the socket missing rather than the variable.
566	- Mode 0666, because the connecting process's identity is checked
567	below, not by file permission.
568	- Who is calling: on every accepted connection, before a byte is read,
569	the edge takes the peer's pidfd (SO_PEERPIDFD), reads the process's
570	cgroup, and confirms through the pidfd that the process is still the
571	same one. The connection is admitted only if the cgroup is inside
572	`carlos-instances-<iid>.slice` for the instance this socket belongs to.
573	A flat tenant (in `carlos-edge.service`'s cgroup), a sibling instance, or
574	anything else is closed with no response. The socket's path proves
575	nothing; the cgroup is the identity, and every nested unit kind runs in
576	that slice (§3). A box whose kernel lacks SO_PEERPIDFD (before 6.5) opens
577	no engine sockets and says so once.
578	- Environment: `CARLOS_ENGINES_SOCKET` is set for every nested unit kind
579	the way `CARLOS_CONTROL_SOCKET` is for the run role (tenant config cannot
580	set it): the run launch file, each generation's launch file, the seed's
581	variables and the command unit's environment.
582
583	The door (codex R1-5, R2-3). A door caller whose `Host:` has no dot and
584	names an engine is an engine call only if the connecting process runs under
585	a uid other than carlos (SO_PEERCRED): an enrolled host-app user, whose door
586	key sits in files only that user can read. A caller running as carlos (a
587	flat instance, or anything else sharing the edge's uid) gets 404 `engine
588	not granted`, because every flat tenant can read every other's config
589	(`layout/preflight.go:181`, `activator/paths.go:35`), so a door key in a
590	flat app's hands proves nothing. When the uid passes and `doorRequest`
591	admits the key, the door mark is replaced by an engine mark for the key's account and app, and the
592	door's own scope check (`doorOwnsRoute`) does not run for it; the engine
593	grant check (step 3 below) replaces it. A door caller whose `Host:` is an
594	engine's real hostname reaches the internal route without an engine mark,
595	so it gets the internal refusal (§5.1) like any other non-engine call.
596
597	On the calling box.
598
599	1. The listener strips every client-address header and every `X-Carlos-*`
600	header, as `doorRequest` does, and presents the box's own address.
601	2. The caller is the instance's route (`release.RouteAccount`, `Route.App`),
602	or the door key's account and app.
603	3. `Host:` names the engine. Unknown name, ungranted app, or an empty grant
604	table: 404 `engine not granted`, one body for all three.
605	4. The edge puts an engine mark (account, app, engine) on the request
606	context, sets `Host:` to the engine's hostname, and refuses `Upgrade`
607	requests with 400.
608	5. If this box homes the hostname it dispatches locally; otherwise it
609	backhauls to the home box (§5.5's transport). A home reached through a
610	customer-fleet tunnel is not supported: 502 `engine unreachable`.
611
612	The hop. An engine call's hop adds `X-Carlos-Hop-Caller:
613	<account>/<app>/<engine>`, and its MAC covers it: a second MAC form over
614	`"v2\n" + host + "\n" + ts + "\n" + caller`, verified with the same
615	dual-key rule. The receiver deletes the header with the other hop headers
616	on every request, verified or not, and turns a verified caller into the
617	same engine mark. A hop without a caller is an ordinary hop, unchanged.
618	The door, the engine socket and `acceptHop` all delete the header on the
619	way in, so only an edge can speak it.
620
621	On the engine box. An engine-marked request reaches a route only if the
622	route is internal, the mark's engine names the request's hostname in the
623	grant table, the table grants that engine to the mark's app, **and the
624	route is the engine hostname's own route (codex R4-2)**. An engine call is
625	resolved by exact hostname only, never through a path map or a wildcard
626	(`edge.go:1364`; path targets may cross apps within an account,
627	`domains/aliases.go:395`), so a grant for one engine can never dispatch to
628	another. An alias of the engine's hostname resolves to the same route and
629	is fine. Otherwise the internal refusal (§5.1). Then:
630
631	- the per-app cap (§5.5);
632	- the import gate, unchanged;
633	- no password gate (the platform authenticated the caller; a login page
634	means nothing to a program);
635	- the meter and analytics, unchanged: calls count against the engine's own
636	app, as any request to it does today;
637	- no cache;
638	- the activator's dispatch, which may wake a hibernating engine.
639
640	### 5.5 Limits and transport (codex R1-7)
641
642	- Per-app in flight, on the engine box: at most `per_app` concurrent
643	calls per calling app. Over the cap: 429, `Retry-After: 1`, and a
644	rate-limited log line naming the app. The cap is per serving box: an
645	engine served by more than one box (replica placements) allows `per_app`
646	on each.
647	- Transport: engine hops use their own transport, never the backhaul
648	transport shared per destination: a 5-minute response-header timeout,
649	and one 5-minute deadline over the whole call (upload, wake, conversion,
650	response). The socket's server side carries the same deadline.
651	- Body: 64 MiB. A declared `Content-Length` over it is refused with 413
652	before anything is forwarded. A body of unknown length is streamed and
653	counted; at 64 MiB the edge aborts the upstream request and answers 413,
654	so the engine may receive a truncated request it then discards.
655	- Async work: longer conversions use the engine's own async mode, whose
656	webhook reaches the caller's public URL over the internet.
657
658	### 5.6 What the engine sees
659
660	The request as the app sent it, with `Host:` set to the engine's hostname,
661	the client-address headers replaced by the calling box's address, and the
662	app's own `Authorization` passed through, so an engine that keeps basic auth
663	keeps it as a second lock. No caller header reaches the engine; the engine
664	box's access log records the caller app for each call.
665
666	### 5.7 Box-side advisories
667
668	The hourly capability pass on each box logs once per pass:
669
670	- an internal route whose hostname is not named by any engine in the grant
671	table (nobody can call it);
672	- an engine whose hostname this box serves on a route that is not internal
673	(the engine is still public; calls to it are refused by §5.4's engine-box
674	rule, so the name is useless until the route declares `doc_engine`);
675	- an internal route that declares `doc_engine` without the grant (no JIT
676	or memory band; it will not render).
677
678	### 5.8 Watching an engine deploy (codex R2-4)
679
680	`carlos deploy` verifies a release by polling its public URL
681	(`deploy.go:1426`, `:1536`), which an internal route now refuses. Whether a
682	route is internal is box-local (a grant can make it so whatever the release
683	declares), so **every deploy target's watch carries a watch ticket (codex
684	R3-3)**. The edge honours it only on an internal route and strips it
685	everywhere else, where the watch proceeds exactly as today:
686
687	- The console mints it on the deploy request it already authorizes: signed
688	with the root release key under its own domain (`carlos watch v1\n`),
689	bound to one hostname, one version and **the probe path the watcher will
690	poll (codex R5-3)**, expiring 5 minutes after minting. The probe comes
691	from the same app setting the console already returns with the ship
692	response (`console/apicli.go:2114`, `:2285`; used at `deploy.go:404`);
693	no probe means `/` only.
694	It is returned to the CLI with the promote response.
695	- In bucket mode, where no console is involved, the CLI signs the ticket
696	itself with the root release key it already holds to sign the promote,
697	over the probe path it is about to poll.
698	- Renewal (codex R4-3): the CLI asks for a fresh ticket for the same
699	hostname and version every 4 minutes for as long as the watch runs
700	(`--timeout` may be longer than one ticket's life, `deploy.go:75`). The
701	console re-authorizes the request with the same deploy credential and
702	refuses a hostname or version the deploy did not name; in bucket mode the
703	CLI re-signs.
704	- The CLI sends it as `X-Carlos-Watch-Ticket` on each poll. The home box's
705	edge admits a request carrying a valid ticket to the internal route only
706	for `GET` or `HEAD` with no body, on `/` or the ticket's signed probe
707	path, strips the header, and dispatches as usual (no cache, as the watch
708	already busts it). Anything else with a ticket gets the internal refusal.
709	- The ticket crosses a backhaul as a header and is verified only on the home
710	box. It names no app and grants no engine call.
711	- A ticket is a credential for reading a version, not for converting a
712	document: the probe paths are the app's own choice, and Gotenberg's
713	`/health` is a fine probe.
714
715	### 5.9 The engine box is dedicated (codex R12-1, R12-2; ruling R3)
716
717	I1 keeps a declaring build from starting before the edge refuses the
718	public, but the edge is not the only way to a process on its box. Round 12
719	found two others: the engine binding a port of its own on every interface
720	(the template allows it, `carlos-app@.service:21`, `:66`; the capability
721	validator leaves high ports to the app, `caps/caps.go:401`), and another
722	route row whose upstream is the engine's backend (`carlos add` with a
723	`service` kind and an explicit address, `cli.go:126`, `:276`; the edge
724	proxies by that row's own address, `edge.go:760`, `:889`, `:1790`). The
725	README already tells operators to run a doc engine on a box of its own.
726	This spec makes that a rule the box enforces:
727
728	- One internal group per box, and nothing else that proxies. On a box
729	that homes an internal group, every route row whose upstream is a local
730	process (any `instance`, `service` or sidecar row: a socket path or an
731	address) must belong to that group. Rows the edge answers without a
732	local process (`static`, the status host, the tunnel and steering
733	handlers) are unaffected. An unclassified (`NULL`) route homes its box
734	like an internal one for this rule, since otherwise a declaring `carlos
735	run` on a shared box is publicly reachable; the cost is that an
736	unclassifiable route on a shared box refuses its proxying neighbours
737	until it is classified (sim, plan 1).
738	- The fence is box-wide on an engine's box (codex R13-2). The fence a
739	declaring launch waits for (§5.1) cancels every non-engine proxy request
740	the box's edge admitted before it, on any row, not only the group's. That
741	includes a request admitted through a row since deleted or replaced, which
742	still holds its old proxy (`registry.go:988`, `edge.go:1790`): the
743	epoch is the box's, not the row's. On a dedicated box that is cheap.
744	- The launch gate checks it. Before it waits for the fence, a declaring
745	launch confirms the rule holds on the box. If any other proxying row
746	exists, the launch is refused (the wake fails), and one line per pass
747	names each offending row.
748	- The edge keeps it. While an internal group is homed on the box, the
749	edge refuses to serve any proxying row outside the group: it gets the
750	internal refusal, and the pass logs it with the move or remove command.
751	A row placed on the box later (provisioning, a move, `carlos add`) is
752	therefore never served there; the engine keeps running.
753	- Ingress precondition, stated (ruling R3; codex R13-1). The engine's
754	box admits connections **from anywhere, public or private, including the
755	deployment's other boxes**, only on the edge's ports (80 and 443, plus
756	whatever the operator keeps for administration), by its security group or
757	host firewall. Engine traffic between boxes enters through the home
758	edge's backhaul, never a direct connection to the engine's port: a
759	`service` row on another box aimed at the engine box's address is a
760	violation of this precondition, and the survey looks for such rows. The platform cannot see that boundary from inside the box. The
761	rollout survey (§7) checks it for every engine box, and the README's
762	doc-engine preconditions gain it as a third, beside the loopback resolver
763	and the dedicated box.
764
765	With the box dedicated, a port the engine binds on its own can be reached
766	only through that firewall, and no other row on the box can point at it.
767
768	## 6. Correctness rules
769
770	### 6.1 Identity
771
772	On the calling box the caller is the kernel's answer about the connecting
773	process's cgroup, or a verified door key. On the engine box it is a verified
774	hop caller, or the local engine mark. No header the app wrote names a
775	caller. Every box holds the mesh key, so a compromised box can call any
776	engine as any app; that is the trust the hop already places in boxes, stated
777	and not solved.
778
779	### 6.2 Flat layout
780
781	Engine sockets need an armed box (codex R3-2, R4-1). Any tenant code
782	running as uid carlos outside an instance slice can write its nested
783	siblings' directories (`layout/preflight.go:181`), including the
784	environment file a nested launch reads (`activator/backend_exec.go:2701`),
785	plant a library and an `LD_PRELOAD`, and so run code inside a granted
786	sibling's slice, which the cgroup check then accepts as that sibling. Three
787	launch paths run tenant code that way today: a `flat` instance, a `foreign`
788	one (`activator/layout.go:103`; both go through the exec-child path,
789	`backend_exec.go:1826`), and an app's worker, which spawns into
790	`carlos-exec@<host>` whatever its layout (`StartWorker`, `opset.go`). The
791	predicate is therefore positive: **the only tenant code on the box runs in
792	instance slices or as unit-backed DynamicUser services** (which cannot write
793	carlos-owned directories).
794
795	- Arming is an operator act on the box: `carlos engines arm`. It
796	refuses unless the box's layout intent is nested, every exec-backed
797	`instance` route classifies `nested`, no route on the box spawns workers,
798	and every `carlos-exec@` cgroup on the box is empty (codex R5-2).
799	Unit-backed routes (`backing=unit`, `carlos-app@`, DynamicUser) are
800	admitted: their socket is `/run/carlos/apps/<host>.sock`, which
801	`Classify` reports as `foreign` (`activator/layout.go:103`), so they are
802	judged by backing, not layout. Platform-minted rows are judged through
803	their execution owner. An exec-backed route that classifies `foreign`
804	still blocks arming. It records the armed state in
805	the box registry. The edge opens engine sockets only on an armed box.
806	- While armed, the activator refuses every unisolated launch: a flat or
807	foreign start and a worker spawn are refused with one log line naming the
808	route and `carlos engines disarm`. The check sits in the launch path, so
809	no unisolated process can start on an armed box, rather than being
810	noticed afterwards. Provisioning a route that would classify flat or
811	foreign on an armed box leaves it unlaunched until the box is disarmed.
812	- Disarming (`carlos engines disarm`) closes every engine socket first,
813	then clears the state; launches follow. Re-arming is the same operator act
814	as arming, with the same checks.
815	- Residual, stated: arming vouches for the present, not the past. A
816	nested instance's files on a box that ran unisolated tenants before it was
817	armed are only as trustworthy as that period, which is true of isolation
818	as a whole. `carlos engines arm` says so when it succeeds; an operator who
819	doubts a box's past rebuilds its instances from the bucket first.
820	- Consequence, stated: an app that uses workers cannot run on an armed
821	box until workers move into instance slices. That is a separate change.
822
823	(codex R2-3) A flat instance cannot call an engine. It has no engine
824	socket, it cannot use a nested sibling's (its process is not in that
825	sibling's slice), and the door refuses engine calls from uid carlos. Flat
826	tenants share one uid and can read each other's files, so no credential a
827	flat app holds can say which app it is. An app that needs an engine moves to
828	the nested layout, the default since v0.25.0. A host-app process under its
829	own uid uses the door; enabling the door on a box (`CARLOS_EDGE_DOOR`),
830	enrolling the user (`carlos bootstrap --socket-consumer`) and minting the
831	key are operator steps in the runbook, not part of this build.
832
833	### 6.3 The public never reaches an internal route
834
835	- The mark lives in the box's own registry and survives restarts. It is set
836	before a declaring release can serve, with the edge's fenced state as the
837	barrier, and cleared only by the operator's `--public`, which stops the
838	group first; an unclassified route is refused (§5.1). No bucket read
839	happens at serve time.
840	- The check runs on the final upstream's route, after every re-resolution,
841	before every dispatch branch (§5.1).
842	- Old non-home boxes backhaul public requests as ordinary hops; the home box
843	refuses every hop without a verified caller for an internal route.
844
845	### 6.4 Freshness
846
847	A grant or revoke reaches every edge within one 30 s interval of the
848	record. A box that cannot read the bucket drops every grant ten minutes
849	after its last good read; an expired record (a console down for 72 hours)
850	drops every grant. The calling box and the engine box check independently,
851	so a revoke is in force once either has read it.
852
853	## 7. Deploy order and rollout
854
855	- Engine boxes. For each box that will home an engine: confirm its
856	public ingress is the edge's ports only (security group or firewall), and
857	move every other app off it. The launch gate refuses a declaring build on
858	a box that still has one.
859	- Survey first. On dev and the flagship, list every route whose live
860	release declares `doc_engine` or holds the grant, and who calls it. A
861	route with public callers is cut off the moment its home box takes this
862	build (R1), and its callers can only switch once the console and their own
863	boxes have it too, so that route's cut is scheduled with its owner. A
864	route with no public callers needs nothing.
865	- Boxes, then the console, in the usual order: dev boxes, dev console,
866	check, flagship boxes, flagship console. Engine calls need the calling
867	box, the home box and the console all on this build.
868	- Arm the calling boxes. On each box whose instances will call an
869	engine, the operator runs `carlos engines arm` once the box is fully
870	nested and runs no workers. The survey lists, per box, what blocks
871	arming.
872	- Then names and grants. With no `engines.json` the grant table is empty
873	and every engine socket answers 404; nothing else changes. `carlos engines
874	add` and `grant` follow the roll, and callers switch to the socket.
875	- A calling box on an older build cannot make engine calls (no socket, no
876	caller on the hop). A home box on an older build serves its engine
877	publicly, which is today's behaviour and why boxes roll before anything is
878	named.
879	- A box whose kernel lacks SO_PEERPIDFD opens no engine sockets; the survey
880	checks kernels (Debian 13's 6.12 has it).
881
882	## 8. Tests that prove it
883
884	Production-wired: real listeners, real hops between two edges in one test
885	process, real unix sockets and a real cgroup check where the kernel allows
886	(the cgroup reader takes a root path for fixtures, as `instancectl` does).
887
888	- Public requests to an internal route through its host, an alias, a path
889	map, a wildcard, a mixed-case spelling and a login POST whose `next`
890	re-resolves to it: all the internal refusal (a plain 404, never a redirect,
891	even with a fallback URL set), on :443 and on the home box's :80; the ACME
892	challenge still answers.
893	- A public request reaching a non-home box is backhauled and refused by the
894	home box.
895	- A route adopted from a release declaring `doc_engine` is internal before
896	its first request, on a fresh registry, with the bucket unreachable after
897	adoption.
898	- Becoming internal: with the edge's reload held back, adoption of a
899	declaring release does not repoint; once the edge loads the mark, it does,
900	and no public request reaches the new release at any point. Becoming
901	public: the mark clears only after the non-declaring release is live; a
902	failed adoption leaves it set.
903	- An upgraded box with an unclassified declaring route and a missing staged
904	manifest refuses it publicly before its first request, and classifies it
905	from the bucket.
906	- A door engine call from uid carlos is refused with a valid, granted key;
907	the same key from an enrolled host-app uid succeeds.
908	- A deploy watch with a valid ticket reads the version on `/` and the probe
909	path; the ticket on a POST, another path, an expired ticket or one for
910	another host gets the internal refusal.
911	- A public request admitted before a `0`→`1` mark and held (mid-proxy, and
912	queued behind a wake) is cancelled before the edge acknowledges, and never
913	reaches the declaring release.
914	- `carlos engines arm` refuses with a flat route, a foreign route, a route
915	with workers, or a non-empty `carlos-exec@` cgroup on the box; on an
916	armed box a flat start, a foreign start and a worker spawn are refused
917	before any process starts; `disarm` closes every engine socket before
918	launches resume.
919	- An engine call with `Host: A` whose path maps to engine B's route is
920	refused, though the caller holds A's grant.
921	- A platform-minted hostname of an internal route is refused publicly and
922	admits engine calls exactly as the owner does; a minted hostname of a
923	public route keeps working across the upgrade; a static platform domain
924	and a static canary keep serving.
925	- Deploying a non-declaring release over a declaring one, with and without
926	generation rollouts, leaves the route internal; `--public` refuses while
927	the live manifest declares or a grant remains, stops the group, refuses if
928	a cgroup stays non-empty, and otherwise clears; the next request wakes the
929	non-declaring build.
930	- I1 per launch path: for each of an exec child, a nested run unit, a
931	generation swap, a seed, a command, a unit-backed start and a wake plan's
932	fallback to a declaring `GoodVersion` after a failed first wake, a launch
933	of a declaring build under `0` sets `1`, completes the fence, and cancels
934	the public request that caused it before the process starts.
935	- With the edge's reload held failing: an adoption of a declaring build
936	times out with `1` written; a later recovery wake that selects a
937	declaring `GoodVersion` is refused, no declaring process starts, and the
938	public request gets the wake-failure answer.
939	- `carlos ship` refuses a capability block with both `doc_engine` and
940	`ports`; a box with an existing port grant on a route that becomes
941	internal removes the listener before the fence completes, and a direct
942	connection to the port is refused afterwards.
943	- A public request admitted through a `service` row that is then deleted,
944	held before dispatch, is cancelled by the next declaring launch's fence
945	and never reaches the engine.
946	- On a box homing an internal group, a `service` row pointing at the
947	engine's address, or any other app's instance, gets the internal refusal;
948	a declaring launch on a box that has one is refused; static and status
949	rows keep serving.
950	- Upgrade classification marks a route internal when its live version is
951	non-declaring but another staged version of the app declares.
952	- A TCP-backed instance's minted hostname keeps serving across the upgrade;
953	two TCP instances of one app with an unrecorded owner are refused until
954	the reconciler records it.
955	- A `doc-engine` grant on a minted row of a non-declaring release makes the
956	owner and every minted row internal, through the barrier.
957	- `carlos engines arm` succeeds on a box with nested callers and a
958	unit-backed engine.
959	- A watch with a probe of `/healthz` reads it with a ticket; the same
960	ticket on `/other` gets the internal refusal.
961	- A watch with a 10-minute timeout whose target turns healthy at minute 6
962	succeeds on a renewed ticket.
963	- A watch ticket on a public route is stripped and the watch behaves as
964	today; on a grant-only internal route it reads the version.
965	- A socket call from a process in the instance's slice reaches the engine on
966	another edge; from a process outside the slice (a flat child, a sibling
967	slice) the connection is closed unread.
968	- A forged `X-Carlos-Hop-Caller` from a client, a door caller or an app
969	through the socket never reaches the engine box's decision.
970	- A hop signed with a caller the engine box's table does not grant is
971	refused even though the calling box admitted it.
972	- A restart with a persisted floor refuses a replayed older record.
973	- The per-app cap: the n+1th concurrent call from one app gets 429 while a
974	second app's call proceeds.
975	- The socket survives a generation swap and is up before a seed runs; every
976	nested unit kind sees `CARLOS_ENGINES_SOCKET`; tenant config cannot set it.
977	- A 45-second conversion completes; a declared 65 MiB body gets 413 before
978	the engine sees a byte.
979
980	### Simulation (#335 plan 1)
981
982	`internal/sim/engines` is a test-only model of this spec: 40 named rules,
983	each with a mutation that turns it off, checked against the invariants
984	INV-P (no public request is delivered to a declaring build), INV-I1,
985	INV-AV (at the end, every row of a `0` group answers a public request),
986	INV-G (an engine call reaches only the engine's exact route; no box admits
987	one on a table that was never read, has expired, has had no good read for
988	ten minutes or holds a record signed ten minutes ago; none is granted more
989	than ten minutes plus one poll after a revoke, replay or not, nor by a box
990	that has read the revoke), INV-ID (the caller the engine box records is the
991	app whose code made the call) and INV-ARM (no unisolated tenant process on
992	a box while an engine socket is open there).
993	`internal/sim/engines/mutate.sh 3000` runs each
994	mutation through a 3000-seed random walk and through the rule's
995	hand-scheduled witnesses; `expected-outcomes.txt` lists the rules caught
996	only in combination.
997
998	Sweep after the final review, 2026-10-21, on the tree committed as
999	"enginesim: clean baseline env, immediate listener checks,
1000	completed-transition coverage" (a337fb28). It adds three rules
1001	(`new-group-fenced`, `clear-launch-lock`, `grant-freshness-bound`), the
1002	freshness bound of ruling R5 (the walk now replays any older record at any
1003	time) and the oracle changes of the final review, so its counts are not
1004	comparable with the first sweep's (parent 4648305e). The unmutated sweep:
1005	0 of 3000 seeds violated, 0 residuals. Result: 36 rules CAUGHT, 4
1006	EXPECTED-MISS, none MISSED, ERROR or STALE (sim, plan 1, final review).
1007
1008	\| Rule \| Verdict \| Sweep catches (of 3000) \| First seed \| Witness \|
1009	\|---\|---\|---\|---\|---\|
1010	\| `set-before-repoint` \| CAUGHT \| 2448 \| 2 \| violated \|
1011	\| `fence-needs-reload` \| CAUGHT \| 412 \| 6 \| violated \|
1012	\| `fence-drains-admitted` \| CAUGHT \| 217 \| 31 \| violated \|
1013	\| `fence-box-wide` \| CAUGHT \| 13 \| 481 \| violated \|
1014	\| `gate-adopt` \| EXPECTED-MISS \| 0 \| - \| held \|
1015	\| `gate-swap` \| CAUGHT \| 2294 \| 2 \| violated \|
1016	\| `gate-fallback` \| CAUGHT \| 265 \| 18 \| violated \|
1017	\| `gate-recovery` \| CAUGHT \| 262 \| 5 \| violated \|
1018	\| `gate-seed` \| CAUGHT \| 1711 \| 2 \| violated \|
1019	\| `gate-cmd` \| CAUGHT \| 1734 \| 2 \| violated \|
1020	\| `gate-waits-when-written` \| CAUGHT \| 287 \| 6 \| violated \|
1021	\| `unreadable-declares` \| CAUGHT \| 2012 \| 3 \| violated \|
1022	\| `live-link-only-adoption` \| CAUGHT \| 857 \| 2 \| violated \|
1023	\| `mark-on-final-upstream` \| CAUGHT \| 2832 \| 1 \| violated \|
1024	\| `classify-null-refused` \| CAUGHT \| 476 \| 13 \| violated \|
1025	\| `classify-staged` \| CAUGHT \| 66 \| 8 \| violated \|
1026	\| `minted-follow-owner` \| CAUGHT \| 802 \| 1 \| violated \|
1027	\| `grant-marks-group` \| CAUGHT \| 484 \| 549 \| violated \|
1028	\| `static-minted-public` \| CAUGHT \| 2981 \| 1 \| violated \|
1029	\| `owner-nonempty-unique` \| CAUGHT \| 42 \| 2359 \| held \|
1030	\| `public-checks-live` \| CAUGHT \| 851 \| 2 \| violated \|
1031	\| `public-stops-group` \| EXPECTED-MISS \| 0 \| - \| held \|
1032	\| `public-holds-lock` \| CAUGHT \| 62 \| 37 \| violated \|
1033	\| `public-confirms-empty` \| CAUGHT \| 49 \| 8 \| violated \|
1034	\| `ports-excluded` \| CAUGHT \| 1995 \| 1 \| violated \|
1035	\| `dedicated-launch-check` \| EXPECTED-MISS \| 0 \| - \| held \|
1036	\| `dedicated-edge-refusal` \| CAUGHT \| 1138 \| 2 \| violated \|
1037	\| `ingress-precondition` \| CAUGHT \| 1248 \| 4 \| violated \|
1038	\| `cgroup-identity` \| CAUGHT \| 171 \| 32 \| violated \|
1039	\| `door-uid` \| CAUGHT \| 85 \| 43 \| violated \|
1040	\| `hop-caller-signed` \| CAUGHT \| 137 \| 41 \| violated \|
1041	\| `engine-box-recheck` \| CAUGHT \| 44 \| 79 \| violated \|
1042	\| `exact-route-engine` \| CAUGHT \| 206 \| 13 \| violated \|
1043	\| `floor-persisted` \| CAUGHT \| 4 \| 13 \| violated \|
1044	\| `stale-clears-grants` \| EXPECTED-MISS \| 0 \| - \| held \|
1045	\| `armed-refuses-unisolated` \| CAUGHT \| 2738 \| 2 \| violated \|
1046	\| `arm-checks-box` \| CAUGHT \| 2321 \| 1 \| violated \|
1047	\| `new-group-fenced` \| CAUGHT \| 0 \| - \| violated \|
1048	\| `clear-launch-lock` \| CAUGHT \| 65 \| 42 \| violated \|
1049	\| `grant-freshness-bound` \| CAUGHT \| 455 \| 2 \| violated \|
1050
1051	The four EXPECTED-MISS rules are each caught in combination, by their
1052	witness and by the sweep (200 seeds: 187, 12, 125 and 59 catches):
1053	`gate-adopt` with `set-before-repoint` (adoption's set-before is the same
1054	gate, reached earlier), `public-stops-group` with `public-confirms-empty`
1055	(step 3 refuses while the declaring generation still runs),
1056	`dedicated-launch-check` with `dedicated-edge-refusal` (the edge refuses
1057	the proxying row once the group is internal), and `stale-clears-grants`
1058	with `grant-freshness-bound` (a table's record was signed no later than
1059	its last good read, so the `signed_at` bound empties a stale table first).
1060	`owner-nonempty-unique`'s witness needs `dedicated-edge-refusal` gone too,
1061	so it holds alone; the sweep catches the rule alone through INV-I1 (a
1062	misgrouped `0` row makes the owner's group read public).
1063	`new-group-fenced` is caught by its witness only: the walk's dedicated
1064	launch check refuses every declaring launch in a new group beside the
1065	starting routes, so only a box that homes nothing reaches it.
1066
1067	## 9. Docs
1068
1069	`docs/site/cli.md`: a `carlos engines` section. The README's doc-engine
1070	passage gains the public refusal and the deploy order. A short "Calling an
1071	engine from your app" passage with a Go example (a unix-socket transport,
1072	`http://gotenberg/forms/chromium/convert/html`). All user-facing strings go
1073	through copy review before they ship.
1074
1075	## 10. Provider boundary
1076
1077	Core only. The mirror reads a `release.Store` (S3 or a directory); signing
1078	uses the root release key; the hop is the existing mesh; the peer check is
1079	Linux, as the whole nested layout already is. On AWS with region HA a hop
1080	within one region uses private addresses; everywhere else it crosses the
1081	public internet under TLS and the hop MAC, as backhaul does today. A non-AWS
1082	deployment loses nothing. "Fleet-internal" means only the platform can call
1083	it, not only over a private network.
1084
1085	## 11. Calls I made (for review)
1086
1087	- Engine selection by `Host:` name on one socket, so a client's base URL is
1088	`http://<name>`.
1089	- Limits: 64 MiB body, 5 minutes end to end, per-app cap default 4, per
1090	serving box.
1091	- Engine calls skip the password gate and the cache; the import gate, the
1092	meter and analytics run as for any request.
1093	- No owner exception on internal routes: the engine app grants itself.
1094	- Tunnel-homed engines are refused (502), not supported.
1095	- Flat instances cannot call engines at all (§6.2); host-app processes
1096	under their own uid use the door, enabled by the operator.
1097	- Engine sockets only on boxes an operator armed; an armed box refuses
1098	flat, foreign and worker launches rather than closing sockets afterwards.
1099	- A route never becomes public automatically; `carlos route --public` is
1100	the only way back, and it stops the group.
1101	- One internal group per box, enforced: a second engine needs a second
1102	box.
1103	- Unclassified routes are refused publicly (fail closed) even though that
1104	can make a route with no readable manifest unreachable while the bucket
1105	is down.
1106	- The deploy watch ticket rather than skipping verification for engines.
1107	- The engine box re-checks the grant (G4) rather than trusting the calling
1108	box.
1109
1110	## 12. Disposition of review findings
1111
1112	\| Finding \| Disposition \|
1113	\| --- \| --- \|
1114	\| R1-1 flat tenant borrows a nested socket \| Accepted: peer cgroup check (§5.4) \|
1115	\| R1-2 aliases and path routes stay public \| Accepted: mark on the route, checked on the final upstream (§5.1) \|
1116	\| R1-3 new box starts public \| Accepted: mark from adoption, no bucket at serve time (§5.1, R1) \|
1117	\| R1-4 revoked grant resurrected after restart \| Accepted: durable floor; fresh-box residual stated (§5.3) \|
1118	\| R1-5 door exceptions \| Accepted: owner exception dropped, door-to-engine transition defined (§5.4) \|
1119	\| R1-6 socket lifecycle \| Accepted: instance-level owner (§5.4) \|
1120	\| R1-7 transport and body \| Accepted: engine transport, streamed body with abort (§5.5) \|
1121	\| Builder Q1 unit-backed callers \| Ruling R2: out of scope \|
1122	\| Builder Q2 door for flat apps \| Superseded by R2-3: flat instances cannot call engines; host-app door is an operator step (§6.2) \|
1123	\| Builder Q3 engine owner's own processes \| Grant itself (§5.1) \|
1124	\| Builder Q4 gate, meter \| §5.4 \|
1125	\| Builder Q5 socket lifetime \| §5.4 \|
1126	\| Builder Q6 several homes, tunnels \| Per-box cap; tunnels refused (§5.4, §5.5) \|
1127	\| R2-1 stamp does not precede serving \| Accepted: set before, clear after, edge-loaded barrier (§5.1) \|
1128	\| R2-2 upgrade serves unclassified engines \| Accepted: NULL state, classification before listeners, NULL refused (§5.1) \|
1129	\| R2-3 flat callers borrow a door key \| Accepted: door engine calls need a non-carlos uid; flat cannot call (§5.4, §6.2) \|
1130	\| R2-4 deploy watch cannot verify \| Accepted: watch ticket (§5.8) \|
1131	\| R2-5 :80 on non-home boxes; unknownHost redirects \| Accepted: plain 404 refusal; no-redirect scoped to the home box (§5.1) \|
1132	\| R3-1 requests admitted before the mark \| Accepted: the acknowledgement waits for cancelled pre-mark requests to finish (§5.1) \|
1133	\| R3-2 flat tenant injects code into a nested caller \| Accepted, then widened by R4-1: armed boxes (§6.2) \|
1134	\| R3-3 grant-only routes lack a watch ticket \| Accepted: every deploy target carries one (§5.8) \|
1135	\| R4-1 foreign instances and workers are unisolated too \| Accepted: positive armed state, unisolated launches refused while armed (§6.2) \|
1136	\| R4-2 a path map widens a grant to another engine \| Accepted: engine calls resolve by exact hostname to the engine's own route (§5.4) \|
1137	\| R4-3 watch tickets expire mid-watch \| Accepted: renewal every 4 minutes (§5.8) \|
1138	\| R5-1 platform-minted rows never classify \| Accepted: execution owner's mark and epoch (§5.1) \|
1139	\| R5-2 arming rejects unit-backed routes \| Accepted: exec-backed routes by layout, unit-backed by backing (§6.2) \|
1140	\| R5-3 no source for the probe path \| Accepted: the ticket signs the watcher's probe path (§5.8) \|
1141	\| R6-1 grant on a minted row ignored \| Accepted: grants anywhere in the group mark the group (§5.1) \|
1142	\| R6-2 static platform domains refused \| Accepted: inheritance only for minted instance rows (§5.1) \|
1143	\| R7-1 clear-after reopens a still-serving engine under rollouts \| Accepted in rev 8, superseded in rev 9 by the operator clear (§5.1) \|
1144	\| R7-2 TCP-backed minted rows have no owner \| Accepted: owner recorded on the row; unique non-empty fallback, TCP included (§5.1) \|
1145	\| R8-1 upgrade classifies a running declaring build public \| Accepted: staged declaring versions classify `1`; residual stated (§5.1) \|
1146	\| R8-2 a sleeping unit-backed route cannot clear \| Accepted, with R7-1: no automatic clear; `--public` stops the group and checks cgroups (§5.1) \|
1147	\| R9-1 `--public` then a fallback to a declaring `GoodVersion` \| Accepted: invariant I1 enforced at every launch (§5.1) \|
1148	\| R10-1 a written but unfenced `1` lets a declaring launch through \| Accepted: I1 is about the fenced mark; every declaring launch waits for the fence (§5.1) \|
1149	\| R11-1 a granted port bypasses the edge \| Accepted: `doc_engine` + `ports` refused at ship; no listener for internal groups (§5.1) \|
1150	\| R12-1 an engine binds its own public port \| Ruling R3: dedicated box; ingress beyond the edge's ports is a stated, surveyed precondition (§5.9) \|
1151	\| R12-2 a separate service row proxies to the engine \| Ruling R3: no proxying row outside the group on an engine's box, enforced by the launch gate and the edge (§5.9) \|
1152	\| R13-1 another box proxies privately to the engine \| Accepted within R3: ingress precondition covers private sources; survey looks for such rows (§5.9) \|
1153	\| R13-2 a request admitted via a deleted row \| Accepted: the fence is box-wide on an engine's box (§5.9) \|
1154