carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Instance catch-alls: `<label>.<parent>` serves the app's instance `<label>`
2
3**Date:** 2026-10-05
4**Status:** branch `instance-catchall` on amadan, not landed. Revised after
5the review on [#233](https://amadan.net/carlos/platform/-/discussion/233)
6(§8).
7
8## 1. Problem
9
10Jelly (lazyatom/jam) runs one instance of `dev-managed-jelly` per team, at
11`<team>.bad.carlos.exciting.io`. Teams should also answer at
12`<team>.letsjelly.dev`. Today the manager makes that happen with one exact
13claim per team at signup (`AttachDomain` in `manager/carlos.go`), plus a wait
14for DNS. The existing catch-all (`Wildcard`+`Catchall` on a claim) cannot do
15it, because it sends every subdomain to ONE route
16(`domains.ReconcileAliases`, the `*.<host>` → primary row). It cannot fan
17out to per-team instances, and it belongs to whoever holds the apex claim.
18Here that is the static site `dev-jelly-site`.
19
20## 2. Design
21
22**The claim.** An instance catch-all is a `domains.Claim` at the key
23`domains/*.<parent>.json`, with `Host: "*.<parent>"`, `Account`, `App`,
24`Instances: true` and nothing else: no Route, Wildcard, Catchall or Paths.
25
26- The star in the key means there is one catch-all per parent, held by the
27 same create-only bucket-key gate that every claim uses.
28- No attach door accepts a `*` (`NormalizeAliasHost`), so a catch-all can
29 never collide with an ordinary claim.
30
31**The door.** `carlos domains attach --app <a> '*.<parent>'` (API: the
32existing attach body with `host: "*.<parent>"`). `detach` removes it the
33same way. Writing a catch-all requires all of these:
34
351. A **person** (`apiPerson`), like `carlos instances enable`. A catch-all
36 is standing policy over every unclaimed name under the parent. A
37 credential the app holds must not be able to set it for itself. Without
38 this, an app token could also put its instances under a sibling app's
39 parent.
402. No route, kind, channel, wildcard, catchall or platform field in the
41 request. They are refused, not ignored.
423. The app has an instance manifest. Without one it has nothing to serve.
43 An app without one is also the only kind of app a rename can still
44 move, which would leave the claim naming a dead app.
454. The parent passes `catchallParentRefused`. It refuses a parent when:
46 - the parent is in a deny tier (reserved or platform, so operators are
47 refused too);
48 - any console, reserved or platform hostname is equal to the parent,
49 under it, or above it. The box cannot see the deny lists, so a
50 derived name could otherwise land on the tunnel host or a console
51 host;
52 - the parent is equal to the apps domain, under it, or above it. That
53 space holds the account-sqid and app-apex names. This is checked
54 whatever `CONSOLE_RESERVE_PLATFORM_DOMAINS` says: on a self-hosted box
55 the apps domain is not reserved, and `edge.<apps-domain>` would
56 otherwise be derivable;
57 - the parent is a public suffix (`co.uk`, `github.io`);
58 - the parent is under the sidecar suffix.
595. **The parent's own claim is held by the same account, by any of its
60 apps** (`domains.AttachInstanceCatchall`). With wildcard DNS at the edge,
61 a catch-all covers every unclaimed name under the parent at once, so only
62 whoever already holds the parent may make one. If the parent has no claim
63 at all, the request is refused: claim the parent first (an alias to any
64 route of the account will do). Holding a catch-all in the same account
65 but under a different app from the apex is the point of the design
66 (static site at the apex, teams below it).
67
68**On the box.** `ReconcileAliases` gains `deriveInstanceAliases`. For each
69catch-all whose parent is still claimed by the same account, it walks the
70box's local routes and derives one ordinary `route_aliases` row,
71`<rt.Slug>.<parent>` → `rt.Host`, for each route that meets every one of
72these:
73
74- `Provisioned`, not a sidecar, and not a replica;
75- the claim's own app, and the claim's own account (`RouteAccount`);
76- a `Slug` that is a single DNS label and is the first label of the host.
77 The instance create door makes every record this shape.
78
79A derived name loses to, in this order:
80
811. **any claim on that exact name, whoever holds it**. This covers the
82 manager's existing per-team claims, `www`, `auth`, and another
83 account's claim made before the catch-all existed;
842. a real route on this box;
853. a real route on any live box (`deploymentRoutes`).
86
87The desired set is rebuilt on every pass:
88
89- a deleted or moved instance drops its name on the next pass;
90- a detached catch-all drops every name;
91- a released exact claim hands its name back to the catch-all.
92
93Rows are written only when they change, so a steady state causes no edge
94reload. At most 1000 derived names per box (`maxInstanceAliases`), in a
95deterministic order, with the overflow logged (§4.11).
96
97**What the derived rows are.** They are the same rows an exact alias claim
98for each name produces today: `Alias`, `Host`, `Account`, `App`. That is the
99property the rest of this note relies on. Every reader of `route_aliases`
100already handles such rows correctly, and none of them looks for a claim
101behind a row. The only code that deletes alias rows is the pass's own
102desired-set sweep. The edge itself needed only one change, made after
103review: the cross-box precedence fix in §8, item 5.
104
105**The attach guard.** While a catch-all is live, the attach doors refuse
106**other accounts** the one-label names under its parent that it reserves
107(`domains.ForeignInstanceCatchall`, then `catchallInstanceName` in the
108console). A refused attach gets the neutral 409 from the same place
109`deploymentRouteTaken` gives it, in all three attach paths (API alias, form
110alias, first-route). What it reserves (§8, review P1s):
111
112- **Proved parent: every label.** The account proves it controls the parent
113 in DNS with TXT `_carlos-catchall.<parent>` =
114 `carlos-catchall-account=<sqid>`. The value names the account, the same
115 shape as the mail ownership proof (#216).
116 - The guard checks the record live, so it follows the domain's DNS rather
117 than a stamp taken once.
118 - A lookup failure counts as "not proved": the narrower reservation, never
119 a refusal.
120 - `carlos domains list` reports `reserves: all` or `reserves: instances`
121 on the catch-all's row, and names the record when it is missing.
122- **Unproved parent: the labels of the app's existing instances**, under
123 any instance domain the app has used. These are counted from the app's
124 instance record keys, so a manifest-domain change does not orphan them.
125
126- **Why it exists.** Without it, any account could attach
127 `acme.letsjelly.dev`. Its exact claim would beat the derived name and
128 take a live team's address. Attach needs no DNS proof, so nothing else
129 stops that.
130- **Why an unproved parent holds less (first review, finding 3).** Attach
131 needs no DNS proof, so a parent claim proves nothing. Holding every label
132 for an unproved parent would let one squatted apex claim plus a
133 catch-all block the real domain owner from every name under it.
134- **One label only**, because a catch-all serves one label only. Deeper
135 names stay as open as they are today.
136- **The caller's own existing claim is never refused (review finding 1).**
137 A claim made before the catch-all keeps re-attaching and converging.
138 titogo's router re-attaches every name on every pass.
139 `deploymentRouteTaken` makes the same allowance.
140- **"Live"** is the box's rule: the catch-all's parent is still claimed by
141 its account. A catch-all whose parent was released, or has moved to
142 another account, holds nothing. The parent's new holder may replace it
143 (`AttachInstanceCatchall` overwrites a dead one).
144- **Read failures.** A bucket read failure in the guard is a retryable
145 502. It is never a 409 (which would read as "held") and never a skipped
146 guard.
147
148**Precedence is decided from the listing (review finding 2).** The box
149decides "a claim on this name exists" from the bucket listing's keys, not
150from the claim bodies that decoded this pass. A body whose GET fails is
151still a claim, so a spurious store error cannot hand another account's name
152to a derived alias for a pass.
153
154**Certificates.** Each derived name gets per-host HTTP-01/ALPN certificates
155exactly as exact aliases do: the cert warm orders them once DNS points
156here, and the first handshake orders one otherwise. A delegated wildcard
157certificate is optional on top, by setting `--wildcard` on the parent's
158own claim. It then covers every derived name on every box (§4.10).
159
160## 3. Mixed versions
161
162- **An old box or old console reading a catch-all** decodes a claim with
163 no Route, no Wildcard and no Paths. Every pass already skips that shape.
164 It logs nothing and makes no DNS lookup or ACME order for the literal
165 `*.<parent>`. Certwarm drops `*.` hosts anyway.
166- **An old console** lists the row with an empty route and "checking"
167 state. Its Detach button answers 404, because `NormalizeAliasHost` refuses
168 the star. No old door can rewrite the claim, for the same reason, so the
169 unknown `instances` field cannot be stripped.
170- **Keys:** `*` is safe in S3 keys, in DirStore on Linux, and through the
171 fleet store door. `canonicalKey` accepts it, and `aclRead` and
172 `aclListScope` already grant `domains/**` to fleet boxes. DirStore on
173 Windows would refuse it, but no deployment bucket runs there.
174- **A new console with old boxes:** the catch-all is inert until the boxes
175 upgrade. The attach guard is already live at that point, so other
176 accounts are refused names the boxes do not serve yet. That is harmless,
177 and it is what the owner asked for.
178
179## 4. Deployment types
180
181Each item says whether it is affected and what guards it.
182
1831. **Flagship (console as an app, carloku.com, #209).** Not affected unless
184 a catch-all is made, and none can be made on the deployment's own
185 zones: refused for everyone, operators included. The console's names,
186 steering, tunnel and the apps domain are refused at, under and above the
187 parent. There is no `--platform` form; it can be added if the flagship
188 ever wants one.
1892. **Tito / titogo.** Its router re-attaches hostnames with an instance
190 token, and it cannot create or remove a catch-all: those are
191 person-only. Store-mode custom instance domains (go.tito.io) are
192 unaffected, since nothing derives without a catch-all. The re-attach
193 loop pays one extra GET per attach (`*.<parent>` of the host, usually a
194 404). A Hetzner spurious 403 there is a retryable 502. Re-attach is
195 idempotent, so the next pass succeeds.
1963. **Pages-hub one-route catch-all, and woodstar.app** (apex plus
197 `Wildcard`+`Catchall`, same app). Unaffected unless an instance
198 catch-all is added on the same parent. If one is, derived exact names
199 beat the route catch-all name by name (`lookupIn`: exact key first, then
200 `*.`). That is same-account by construction, and it is the useful
201 combination: unknown labels fall to the site, known ones to their team.
202 - On woodstar it would take a PDS handle whose label matches an instance
203 label, so do not combine the two there.
204 - The ceecb3a9 self-removal loop cannot recur. Derived rows are exact
205 names published with `Alias: true`, and `DeploymentRoutes` counts only
206 `Role==home && !Alias`.
2074. **Self-hosted single box, and this hand-staged box.** Same as above:
208 nothing changes without a catch-all. The apps domain is refused as a
209 parent even though it is not reserved there. Hand-added routes
210 (`carlos route add`) are not `Provisioned` and derive nothing. That also
211 keeps a legacy account-name column, which reads as ops, out of reach.
2125. **Customer fleets / `carlos apps place`.** Fleet boxes read `domains/**`
213 through the store door, so they derive for their own instances. The store
214 door does not grant `control/edges/`, so `deploymentRoutes` is nil
215 (permissive) there. This gap already exists for plain aliases, and the
216 exact-claim skip still applies.
2176. **Multi-box, multi-region edges, the edges file.** Derived rows publish
218 exactly like exact aliases: only once `StampHealth` has stamped them
219 `pointing`, and backhauled by remote edges to the home box. A new or
220 moved name can take up to about 12 minutes to reach other edges (one
221 stamp period plus one edges-file period). Exact claims have the same
222 delay today; a single box serves at once. Duplicate publication
223 (everywhere-class instances) folds to the first-sorted box, as for exact
224 aliases.
2257. **Edge replicas, steering.**
226 - Replica rows are excluded explicitly (`rt.Replica`). A door-created
227 replica carries the label as its Slug, so the Slug shape alone would
228 not exclude it.
229 - Steering looks at platform names only, and never at aliases or claims.
2308. **Instance moves.** During the cut the source row is `moving`, and the
231 edge and edges file skip it. After the flip the target box derives the
232 name. The name follows the instance, as an exact alias claim would.
2339. **Hibernation and activation.** An alias entry is its primary's upstream
234 copied whole, `route` and `hibernating` included. A request on a derived
235 name therefore wakes the instance itself, and metering, the gate and
236 analytics are keyed on `rt.Host`.
237 - The response cache keys on the requested host, so the app purges a
238 derived name's entries with `Carlos-Purge-Host`. That is pre-existing
239 for exact aliases.
240 - The backend sees `Host: <label>.<parent>`, so the app must accept it.
241 Jelly does today, via the exact claims.
24210. **Isolated exec, path-prefix maps.**
243 - Isolated exec: units and sockets are keyed on `rt.Host`, so it is
244 unaffected.
245 - Path-prefix maps: a derived name inherits its instance's map, as any
246 alias does. A Paths claim on a derived name itself is an exact claim
247 and suppresses the derived row.
24811. **Platform apps-domain wildcard, platform-domain claims (#209),
249 reserved/deny lists, readings sweep, cert sweep, LE limits, alias-table
250 growth, `maxWildcardDomains`.**
251 - `parentsFor` takes only platform/provisioned non-alias hosts, so
252 derived names cannot mint platform parents.
253 - The deny tiers are covered in §2.
254 - The readings sweep (`StampHealth`) does one serial `LookupHost` per
255 alias (3s timeout) every 5–6 minutes. Each alias adds about 140 bytes
256 to the box's status snapshot, and readers skip a snapshot above 1 MiB.
257 A skipped box also drops out of `DeploymentRoutes` and the edge-IP
258 fold. That is why there is a **per-box cap of 1000 derived names**,
259 logged when it is hit.
260 - The cert sweep is expiry-only, and `ownsCertName` checks
261 `deploymentHost`, not claims.
262 - **Let's Encrypt:** without a wildcard, N instances means N new
263 CT-logged certs against the registered domain's limit (50 per 7 days
264 as of writing). About 200 teams take about four weeks of issuance;
265 past the limit, certwarm backs off 12h per host. That is identical to
266 today's per-team exact claims. The remedy is `--wildcard` on the
267 parent claim, which needs `CARLOS_ACME_DELEGATION_ZONE` and a DNS
268 provider. The testbed has neither.
269 - The instance catch-all is not a Wildcard claim, so it takes no slot
270 in `maxWildcardDomains` or `maxDelegatedParents`.
27112. **Trash, purge, rename and move of apps; instance delete with parked
272 data.**
273 - Trash drops the app's routes, so the derived names go with them. The
274 catch-all claim stays, as every claim does, and comes back to life on
275 restore.
276 - Purge never deletes `domains/` (pre-existing for all claims). An
277 orphaned catch-all keeps holding names under its parent for its
278 account, and passes to a later app of the same name in the same
279 account. Detach it before purging.
280 - Rename only applies to never-shipped apps, and the door requires an
281 instance manifest.
282 - Account move leaves claims on the old account, which derive nothing
283 (pre-existing).
284 - Instance delete removes the route but keeps the parked data. The name
285 drops, and comes back if the instance is recreated.
286
287## 5. Security cases
288
289- **(a) A label colliding with another account's exact claim:** the exact
290 claim wins, on every box. A derived name is skipped whenever any claim
291 exists for it. New claims of that kind are refused by the attach guard
292 while the catch-all is live. Claims made before the catch-all keep
293 working, so it cannot take a name away from anyone.
294- **(b) A sibling app's apex label:** cannot happen.
295 - Only the claim's own app's routes derive.
296 - Sibling apex hosts are not under the parent.
297 - The instance create door refuses a label that names a live or trashed
298 sibling app, and `apex.go` refuses the name arriving afterwards.
299- **(c) A team's name taken before the team exists.**
300 - **Proved parent:** impossible while the catch-all is live, for every
301 label, including a deleted team's.
302 - **Unproved parent:** the name is protected once the instance exists. A
303 foreign claim made before that wins, which is the first-come rule every
304 claim has.
305 - A same-account exact claim wins on purpose.
306- **(d) Races between the claim pass and the instance reconcile:** both
307 sides are rebuilt from scratch each pass, and every precedence check is
308 made against the same pass's claim list and route table. A catch-all
309 write racing a foreign attach can let both land; the exact claim then
310 wins, which is the same outcome as (c). A parent released between the
311 door's check and its write is caught by the box's parent re-check.
312- **(e) The parent changes hands:** the old catch-all goes inert on the
313 box, stops holding names at the door, and the new holder can replace it.
314- **(f) A derived alias pointing at another account's route:** prevented by
315 three filters, all of which stay: `rt.App == c.App`,
316 `RouteAccount(rt) == c.Account` and `rt.Provisioned`.
317- **Abuse of the guard as a squat.**
318 - **Unproved parent:** the guard holds only names of instances that
319 exist, each a real, rate-limited instance. That is the same class of
320 squat as claiming those names directly.
321 - **Proved parent:** holding every label takes write access to the
322 parent's DNS, which is what owning the domain means.
323 - Public suffixes are refused as parents either way.
324
325## 7. Review
326
327Reviewed by an Opus agent before landing. Three medium findings were all
328fixed, each with a test that was watched to fail first where practical:
329
3301. The guard refused an account's re-attach of its own older claim.
3312. Precedence was decided from decoded bodies, which fails open on a read
332 error.
3333. The guard held every label, which amplified a squat.
334
335Lows taken:
336
337- the browser form now refuses a channel on a catch-all.
338
339Lows left as they are:
340
341- **Member, not owner, on the API door.** The API door admits any person
342 in the account, the browser door only owners. That is the same split
343 ordinary attach has.
344- **A reassigned name keeps its old `dns_status` until the next stamp.**
345 Pre-existing for exact claims changing hands.
346- **Once-per-process refusal logs** do not repeat if a catch-all recovers
347 and fails again.
348
349## 8. Second review ([#233](https://amadan.net/carlos/platform/-/discussion/233), Paul's agent with Astra)
350
351Two P1s and three P2s. Every one was checked against the code and found
352real, and all five are fixed with tests that were watched to fail first.
353
3541. **P1: a team's name could be taken before the team exists.** Wildcard
355 DNS points every label at the edge, and attach needs no proof.
356 **Fixed:** a proved parent reserves every label (§2, the attach guard).
3572. **P1: the guard checked only the current manifest domain.** **Fixed:**
358 the guard counts the app's instance record keys by first label, across
359 every instance domain (`domains.InstanceLabelCounts`), with one listing
360 and no record bodies.
3613. **P2: a duplicate label routed differently per box.** Two instances
362 sharing a label, possible only after an instance-domain change, were
363 each named by the box that held them. **Fixed:** a label shared by two
364 records derives nothing, on every box alike, and the refusal is logged.
365 - Seen from the same record listing, asked once per catch-all and only
366 when the box has an instance to name.
367 - If the listing fails, the catch-all's existing rows are kept for that
368 pass, never dropped.
369 - The earlier "first in host order wins" text is withdrawn.
3704. **P2: an old owner's detach could delete the new owner's catch-all.**
371 This happened after the parent changed hands and the catch-all was
372 replaced in place. **Fixed:** `DetachClaim` deletes an instance
373 catch-all only at the revision it read (`release.DeleteIfMatch`).
374 - Ordinary claims are unchanged. They cannot be replaced while held, so
375 the race does not exist for them.
376 - Both production stores (S3, DirStore) support the conditional delete.
3775. **P2: a route catch-all on one box beat an instance name on another.**
378 The local `*.<parent>` matched before backhaul was considered.
379 **Fixed in the edge:** a local match made only through a star key now
380 yields to another box's exact host (`homeMap.ExactHome`, then backhaul).
381 - That is the single-box precedence, exact then star, extended across
382 the deployment.
383 - An arrived hop and a door call keep the local answer.
384 - A two-box test covers it.
385
386The review's answers to this note's asks:
387
388- **`--platform` catch-alls:** keep refusing them.
389- **CLI spelling:** agreed.
390- **"Delete now" leftovers:** filed as their own discussion.
391
392## 6. Surfaces
393
394- **CLI.** The catch-all is spelled `carlos domains attach --app <a>
395 '*.<parent>'` and `carlos domains detach --app <a> '*.<parent>'`.
396 - The star is the mode, and no other flag combines with it.
397 - This keeps the meaning of `attach <host>` as "this hostname now
398 serves this app", and needs no new flag grammar.
399 - `--catchall` stays the bool capability it is; `--catchall instances`
400 would have made it a value flag and broken bare `--catchall`.
401 - `carlos domains list` shows the catch-all row with mode `instances`,
402 then each derived name the fleet reports, marked `via *.<parent>`.
403- **API.** Same endpoints. The attach answer and list rows gain
404 `instances: true`, and the list gains `via` on derived rows. Both are
405 additive.
406- **Web UI.** The app's Domains & TLS table shows the catch-all and its
407 derived names. The page does not edit wildcard or catchall today, and
408 the catch-all is person-only, so the page lets an owner attach and detach
409 it the same way as any hostname: typing `*.<parent>` into the attach
410 form.
411