amadan
spec: unit-tenant purge channel — Carlos-Purge response header design (#19)
* spec: a purge channel unit tenants can actually reach PR #12 reverted the unit-tenant purge listener because a DynamicUser tenant cannot connect to a 0600 carlos-owned socket in a 0750 carlos-owned directory — two independent denials, and an env var naming an unreachable listener is an inert feature pretending to work. This design doc enumerates the five ways out (per-route group + SupplementaryGroups, OpenFile= fd passing, loopback endpoint + bearer token, RuntimeDirectory signalling, and a response header the edge already parses), including both sketches PR #12's body recorded, and recommends the response header: the only candidate that closes the reachability half of the tenant-isolation invariant by construction, and the only one needing no unit file, no group, no restart and no secret. Docs only — no production code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * spec: align the purge-channel contract with what shipped Four amendments from PR #22's review, all of them the spec catching up to the implementation rather than the other way round. §6.1 rejection granularity — the spec said a malformed target meant "the whole header line is ignored"; the implementation rejects per-target and honours the neighbours, which is the right failure direction. Under-purging an entire declaration because one entry was mistyped would let a single typo silently disable an app's invalidation. §6.1 query-variant semantics, new — a bare path now purges EVERY query variant under it (Cache.PurgePath over the byPath index, both method groups), because after a write to /events/42 the ?utm_source= and ?lang=fr copies are exactly as stale as the bare one and were being left to expire on their TTLs. A target with "?" stays exact; "/events/42?" is how an app names the unqueried variant alone. §6.1 percent-encoding, new — the path half is decoded and the query half is raw, because that is how the key is STORED (r.URL.Path is already decoded, r.URL.RawQuery is verbatim). Also records the double refusal gate and the one deliberate asymmetry: a decoded space is allowed where a raw one is not. §6.1 final-response-only narrowing — 1xx blocks and trailers are neither honoured nor stripped, pinned by tests rather than fixed, because the fix would mean wrapping the raw ResponseWriter the upgrade path passes straight through. Adds the Connection footgun (hop-by-hop stripping eats the header before the hook), the inbound request scrub and its rationale, and the honest per-response scope of "purge before store". §6.1 bounds cost — corrected: degrading is ~32x MORE work than honouring, not less. The caps bound per-response work; the degrade is the simple path. §7.3 — unit-tenant CARLOS_PURGE delivery via the appconfig sync is DEFERRED and marked exec-only as shipped; harmless precisely because the var is advisory. Also records that both purge vars are agent-owned in BOTH directions, since the env layers are tenant-supplied. §8.8 — the on-box DynamicUser acceptance run (and §8.1's foreign-uid test) need root and a real carlos-app@.service, so they are deploy-time acceptance steps, not merge gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> ---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+697
−0
docs/superpowers/specs/2026-08-03-unit-tenant-purge-channel.md+697 −0
diff --git a/docs/superpowers/specs/2026-08-03-unit-tenant-purge-channel.md b/docs/superpowers/specs/2026-08-03-unit-tenant-purge-channel.md| new file mode 100644 |
| index 000000000..887f7c2bb |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-03-unit-tenant-purge-channel.md |
| @@ -0,0 +1,697 @@ |
| +# 🤖 Unit-Tenant Purge Channel — Reaching a `DynamicUser` Tenant Without Inventing a Static User |
| + |
| +> 🤖 This document was written by an LLM (Claude), on the ideas, instruction, |
| +> and editing of humans, per [factor X](https://11factor.org/#x). |
| + |
| +**Status:** Concept spec, 2026-08-03. Not yet approved. |
| + |
| +**Relationship to the cache spec:** this is a direct follow-on to |
| +[2026-08-02-edge-cache-before-wake-design.md](2026-08-02-edge-cache-before-wake-design.md). |
| +That spec's §3 introduced "a local purge channel" and left its transport as |
| +open question §10.1 ("unix socket vs. an extension of an existing local |
| +channel… a plan-level decision"). The exec-backed half of that answer |
| +shipped in PR #10 (`edge.ListenPurge` + `CARLOS_CONTROL_SOCKET`). The |
| +unit-backed half did not: it was attempted and reverted in |
| +[PR #12](https://github.com/carlosframework/platform/pull/12), and this |
| +document is the design that revert asked for. Everything in the parent |
| +spec — cache keys, TTL ceiling, purge-on-write, never-cache-non-2xx, |
| +local-only — is unchanged and still wins where the two could diverge. |
| + |
| +## 1. The gap, stated plainly |
| + |
| +Exec-backed instances have a purge channel. `backend_exec.go` spawns the |
| +child with `CARLOS_CONTROL_SOCKET=<rt.Socket>.control` in its environment |
| +and calls `edge.ListenPurge` on that path; the app POSTs `/purge` to it |
| +after a write, and the edge drops the affected cache keys immediately. |
| + |
| +Unit-backed tenants (`carlos-app@<host>.service`, `backend_unit.go`) have |
| +nothing. Their cached pages are bounded by `DefaultCacheMaxTTL` alone — up |
| +to five minutes of staleness after a write, with no way to shorten it. |
| +That is not a regression (it is how things have always been) but it is a |
| +hole in a platform capability that is supposed to be general to any |
| +`routed` app, and it is the difference between "cache everything public, |
| +safely" and "cache everything public, and accept five-minute staleness on |
| +writes." |
| + |
| +The obvious fix — set `CARLOS_CONTROL_SOCKET` in the unit tenant's |
| +environment too — was written, reviewed, and reverted, because the |
| +listener it names is structurally unreachable from the tenant. The |
| +reviewer verified it empirically: `connect: permission denied`, against |
| +both the socket and its parent directory. Shipping it would have made the |
| +platform's most load-bearing environment-variable convention lie: the var |
| +is set everywhere else exactly when the listener behind it is reachable. |
| + |
| +## 2. Why the exec design does not transfer |
| + |
| +### 2.1 What `DynamicUser=yes` actually guarantees |
| + |
| +`carlos-app@.service` sets `DynamicUser=yes` (and the template's own |
| +comment is right about why: apps share no identity with each other or the |
| +platform, and nobody manages user accounts). Mechanically, systemd: |
| + |
| +- allocates a UID/GID pair from the transient range (61184–65519) at unit |
| + **start**, registered through `nss-systemd` for as long as the unit |
| + runs, and released when it stops; |
| +- picks a **different** pair on the next start — including the next |
| + `Restart=always` restart after a crash. Nothing about the uid is |
| + predictable or stable; |
| +- **recycles** released uids. A uid that belonged to tenant A this morning |
| + can belong to tenant B this afternoon. systemd's own documentation warns |
| + against leaving anything uid-owned behind for exactly this reason, which |
| + is why the template also carries `RemoveIPC=yes`, `PrivateTmp=yes`, and |
| + a `StateDirectory=` that systemd re-chowns to the fresh uid on every |
| + start; |
| +- implies `ProtectSystem=strict` and a read-only view of the filesystem |
| + hierarchy outside the state dir (the template sets it explicitly too). |
| + |
| +So the tenant's identity is: an ephemeral, unpredictable, recycled uid, |
| +with **no supplementary groups at all** as the template stands today, and |
| +in particular never `carlos` and never another tenant's. |
| + |
| +### 2.2 Why 0600/0750 fails |
| + |
| +`ListenPurge` creates its socket under umask 0177 and chmods it 0600. It |
| +lives beside the tenant's listener in `/run/carlos/apps/`, which is |
| +`0750 carlos:carlos`. For the tenant to `connect()` it needs, in order: |
| + |
| +1. search (`+x`) on `/run/carlos` and `/run/carlos/apps` — denied: the |
| + tenant is neither owner `carlos` nor in group `carlos`, and the dir has |
| + no other-execute bit; |
| +2. write permission on the socket inode itself — denied: mode 0600, owner |
| + `carlos`. |
| + |
| +Two independent denials. The exec case sidesteps both by not having the |
| +problem: exec children are `exec.Command` children of the agent with no |
| +uid change, so they *are* `carlos` (which is also the source of the |
| +same-uid sibling gap `purge.go`'s comment documents honestly). |
| + |
| +Note what does **not** block a unix-socket connect: the tenant's |
| +`ProtectSystem=strict` read-only view of `/run`. The kernel's read-only |
| +superblock check (`sb_permission`) returns `-EROFS` only for regular |
| +files, directories, and symlinks — sockets are exempt, so `connect()` to a |
| +socket on a read-only mount succeeds if the ordinary permission bits allow |
| +it. This is a claim the reachability test in §8 must prove rather than |
| +assume; it is stated here because it rules *in* the socket family of |
| +designs, which would otherwise be dead on arrival. |
| + |
| +### 2.3 What `SupplementaryGroups=` can do |
| + |
| +`SupplementaryGroups=` is compatible with `DynamicUser=yes` — systemd sets |
| +the supplementary group list before dropping to the dynamic uid, and the |
| +unit's empty `CapabilityBoundingSet=` and `NoNewPrivileges=yes` do not |
| +interfere (both apply to the executed process, after the manager has |
| +already done the `setgroups`). This is the standard pattern for giving a |
| +`DynamicUser` service access to a device or directory. |
| + |
| +What it cannot do is be *shared*. A group is a static, box-wide, |
| +named identity. Adding one line to the shared template |
| +`carlos-app@.service` puts **every** tenant in the **same** group, which |
| +converts a per-tenant boundary into a per-box one — strictly worse than |
| +the exec gap, because it crosses the isolation boundary `DynamicUser` |
| +exists to create. Keeping the invariant requires one group **per route**, |
| +created by root, referenced from a per-instance drop-in. That is real, and |
| +it is candidate (a) below — but it is machinery, not a one-liner. |
| + |
| +### 2.4 What POSIX ACLs cannot do |
| + |
| +`setfacl -m u:<uid>:rw` on the socket looks like the surgical answer and is |
| +not usable here: |
| + |
| +- the uid does not exist until the unit starts, so the ACL can only be |
| + applied *after* start, from something that has learned the uid; |
| +- `Restart=always` mints a **new** uid on every restart, and the activator |
| + deliberately never notices unit restarts (`unitProc.Exited()` returns |
| + false until `Stop`, documented in `backend_unit.go` as a hot-path |
| + choice) — so nothing would re-apply it; |
| +- uid recycling makes a stale ACL entry actively dangerous: an entry left |
| + for uid 61200 grants access to whatever tenant systemd hands 61200 next. |
| + |
| +An ACL keyed on an ephemeral, recycled identifier is a permission that |
| +drifts onto the wrong subject over time. Ruled out. |
| + |
| +### 2.5 What the environment can do |
| + |
| +Unit tenants get their environment from `/etc/carlos/apps/<host>.env`, |
| +which is `0600 root` and written by the **root** adopt pass |
| +(`internal/appconfig` sync, fired by `carlos-adopt.timer`). The agent, as |
| +`carlos`, cannot read it — this is already a documented, load-bearing |
| +constraint elsewhere: `UnitBackend.NextDue` returns "nothing scheduled" |
| +precisely because the admin token it would need lives in a file the agent |
| +cannot read. Any design whose security depends on a shared secret the |
| +agent must also know has to solve that distribution problem first. |
| + |
| +## 3. Constraints any candidate must satisfy |
| + |
| +1. **The invariant.** A tenant may purge only its own host's entries. |
| + `ListenPurge` closes the confused-deputy half by construction (`host` |
| + is fixed at construction; `purgeRequest` carries only `Path` and |
| + `Query`, never a host). Any new channel must close it the same way — |
| + and additionally must not be *reachable* by another tenant, which is |
| + the half exec-backed tenants do not have. |
| +2. **Env-var-only app contract.** No new CLI flags, ever. Every app binary |
| + built before this feature parses flags with `flag.ExitOnError`, which |
| + exits 2 on an unknown flag — a new flag is a fleet-wide outage on agent |
| + upgrade. An unknown environment variable is ignored. (This rule is |
| + already written into `backend_exec.go`'s comment; it applies here |
| + verbatim.) |
| +3. **Old binaries keep working, unchanged.** An app that knows nothing |
| + about purging must behave exactly as it does today: TTL-bounded |
| + staleness, no errors, no failed starts. |
| +4. **Wire-shape changes are additive.** Anything written into a bucket or |
| + a registry before this change must still parse afterwards. |
| +5. **Adoption swaps a symlink; it does not restart processes.** Order is |
| + promote → adopt → restart. Long-running units need the restart; |
| + hibernating tenants pick it up on wake. A design that requires a unit |
| + file change, a `daemon-reload`, or a tenant restart to become |
| + *correct* is paying a materially higher rollout cost than one that does |
| + not. |
| +6. **No inert features.** Whatever ships must have a test that proves |
| + reachability from a process with the tenant's real privileges — see §8. |
| + |
| +## 4. Candidates |
| + |
| +### (a) Per-route group on the control socket |
| + |
| +Root, at adopt time, creates a group per route and a drop-in: |
| + |
| +``` |
| +/etc/systemd/system/carlos-app@<host>.service.d/10-purge.conf |
| + [Service] |
| + SupplementaryGroups=carlos-p-<hash> |
| +``` |
| + |
| +with `/run/carlos/purge/<host>/` at `0710 carlos:carlos-p-<hash>` and the |
| +control socket inside it at `0660 carlos:carlos-p-<hash>`. |
| +`CARLOS_CONTROL_SOCKET` points at it; the existing `ListenPurge` serves it |
| +unchanged. |
| + |
| +- **Boundary:** holds exactly. Only that route's group can traverse the |
| + directory; another tenant is in a different group and is denied at the |
| + directory before it ever reaches the socket. Strictly *better* than the |
| + exec-backed status quo, which has the same-uid sibling gap. |
| +- **Failure modes:** the drop-in only takes effect after |
| + `systemctl daemon-reload` **and** a restart of the unit, so a tenant |
| + adopted before the drop-in existed has `CARLOS_CONTROL_SOCKET` set and |
| + no group until it next restarts — the exact inert-feature shape PR #12 |
| + refused, in miniature. Group lifecycle needs GC on route removal or the |
| + box accumulates groups forever. |
| +- **Warts:** group names cannot be `%i`. Linux caps group names at 32 |
| + bytes, and Debian's default `NAME_REGEX` rejects the dots in an FQDN, so |
| + the name has to be a hash (`carlos-p-<8 hex of sha256(host)>`), which |
| + means an operator reading `/etc/group` cannot tell which route a group |
| + belongs to without a lookup table. |
| +- **Blast radius of a bug:** a mis-assigned group grants one tenant purge |
| + rights over one other host. Bounded (a forced re-fetch from that host's |
| + own backend), but it is a real cross-tenant grant. |
| +- **Operational complexity:** high — new root-side state (groups), |
| + per-instance drop-ins, `daemon-reload`, restart, GC. It is the only |
| + candidate that requires the platform to manage Unix accounts, which the |
| + unit template exists to avoid. |
| +- **Old binaries:** fine (env-var-only). |
| + |
| +**Sub-variant, from PR #12's second sketch — a static per-tenant user |
| +tier.** Same shape, but with a real `useradd`ed user per route instead of |
| +a group, abandoning `DynamicUser` for tenants that want purge. This gives |
| +up the template's central property (nobody manages user accounts; a |
| +compromised tenant's uid is unpredictable and short-lived) to buy a cache |
| +invalidation channel. Rejected: the price is the isolation model, and the |
| +thing being bought is at most five minutes of freshness. |
| + |
| +### (b) Socket activation / FD passing |
| + |
| +PR #12's first sketch: "a systemd-bound socket unit mirroring |
| +`carlos-app@.socket`'s root-binds-as-carlos trick in reverse." The trick |
| +does not reverse. `.socket` units pass **listening** fds; purge needs the |
| +tenant to be the **client**. |
| + |
| +The closest real mechanism is `OpenFile=` (systemd ≥ 253, so available on |
| +the fleet's Debian 13 boxes): the manager opens the path and, when it is |
| +an `AF_UNIX` socket, `connect()`s it and passes the **connected** fd to |
| +the service. It works, and it dies on three counts: |
| + |
| +- **It breaks old binaries.** `OpenFile=` fds arrive through the same |
| + `LISTEN_FDS`/`sd_listen_fds` channel as the listener. `carlos-app@.socket`'s |
| + contract is "`LISTEN_FDS=1`, fd 3 is the listener," and shipped app |
| + binaries assume it. Bumping it to 2 is precisely the class of fatal |
| + change constraint 2 exists to prevent — the same outage as a new flag, |
| + arriving through a different door. |
| +- **One connection, established once, never re-established.** If it |
| + breaks, the tenant has no channel until it restarts. `carlos-edge.service` |
| + is `Restart=always` and gets restarted on every platform-binary adopt, |
| + so the fd would go dead as a matter of routine operation, and the tenant |
| + would not know. |
| +- **It is a unit-file change**, with the same `daemon-reload` + restart |
| + rollout cost as (a), plus a systemd version floor that a self-hosted |
| + CARLOS elsewhere may not meet. |
| + |
| +Rejected. |
| + |
| +### (c) Loopback HTTP endpoint on the edge, per-instance bearer token |
| + |
| +The edge binds `127.0.0.1:<port>`; each tenant gets |
| +`CARLOS_PURGE_TOKEN=<opaque>` in its environment; the edge maps token → |
| +host and purges only that host's keys. |
| + |
| +- **Boundary:** the confused-deputy half is closed the same way as |
| + `ListenPurge` (host comes from the token, never the body). The |
| + reachability half rests **entirely** on the token: loopback is |
| + connectable by every uid on the box, so any tenant that learns another's |
| + token can purge that host. |
| +- **Failure modes:** token leakage is the whole risk surface, and env vars |
| + leak in ordinary ways — an app that dumps its environment on a debug |
| + page, a crash reporter, a subprocess that inherits it. There is no |
| + rotation story short of a restart. |
| +- **Distribution is the blocker.** The agent needs the token; the tenant |
| + gets it from `/etc/carlos/apps/<host>.env`, which is `0600 root` and |
| + which the agent cannot read (§2.5). So the token must be minted by the |
| + agent, persisted somewhere durable (it must survive an agent restart, |
| + because the env file is only rewritten at adopt time and only takes |
| + effect on the tenant's next restart), and copied into the env file by |
| + the root adopt pass. That is a new long-lived secret at rest, on every |
| + box, for cache invalidation. |
| +- **Blast radius:** a leaked token = forced re-fetches on one host. |
| + Bounded. But it also means the platform now has a credential whose |
| + compromise is silent. |
| +- **Complexity:** no unit-file change (the template already permits |
| + `AF_INET`), which is its one genuine advantage over (a) and (b). Against |
| + that: a new listener, a new port to reason about on every box, a new |
| + secret, and a new persistence file. |
| +- **Old binaries:** fine (env-var-only). |
| + |
| +Not rejected outright — it is the best of the "new channel" family — but |
| +it buys a channel by introducing a secret, and §4(e) gets the same |
| +capability with no secret at all. |
| + |
| +### (d) Filesystem signalling |
| + |
| +The tenant writes a file naming the paths to purge; the agent watches for |
| +it. Needs `RuntimeDirectory=carlos-purge/%i` on the template (the tenant's |
| +view of `/run` is otherwise read-only under `ProtectSystem=strict`), |
| +which systemd creates owned by the dynamic uid and cleans up on stop. |
| + |
| +- **Boundary:** holds. `RuntimeDirectory=` is per-instance, so a tenant |
| + can only write in its own directory, and the agent attributes intent by |
| + path. But the directory must be world-readable (`0755`) for the agent to |
| + read it at all, because it is owned by an ephemeral uid **and** an |
| + ephemeral group — `carlos` cannot be given group access to something |
| + whose group does not persist. Every tenant can then read every other |
| + tenant's purge intents. Low sensitivity (they are the tenant's own URLs) |
| + but it is a new, permanent information leak across the tenant boundary. |
| +- **Failure modes:** `RuntimeDirectory` is cleaned on stop, so a purge |
| + written moments before hibernation is silently lost. The agent needs |
| + either inotify (a new dependency and a new watch per route) or a poll |
| + loop (latency traded against CPU on an idle box, which is the resource |
| + this whole project exists to conserve). |
| +- **Blast radius — the worst of any candidate.** A tenant can create |
| + files as fast as it likes. A buggy or hostile one can drive the agent's |
| + watcher to consume CPU and inodes **box-wide**, degrading every other |
| + tenant and the edge itself. Every other candidate's worst case is |
| + bounded to one host's cache; this one is not. |
| +- **Complexity:** unit-file change (adopt + `daemon-reload` + restart), |
| + new watcher machinery, file GC, symlink and file-count defences. |
| +- **Old binaries:** fine. |
| + |
| +Rejected on blast radius. |
| + |
| +### (e) Purge as a response header the edge already reads |
| + |
| +The app signals a purge by setting a response header on the response it is |
| +*already* returning from the write: |
| + |
| +``` |
| +HTTP/1.1 302 Found |
| +Location: /events/42 |
| +Carlos-Purge: /events/42, /events/42?lang=fr |
| +``` |
| + |
| +The edge is already in this path. `Edge.ServeHTTP` already wraps every |
| +proxied response in a `recordingResponseWriter` and already reads |
| +`Cache-Control` and `Vary` off it via `joinHeader`; both `unixProxy` and |
| +`tcpProxy` are plain `httputil.ReverseProxy` values with an unused |
| +`ModifyResponse` seam. `Cache.PurgeURL` and `Cache.PurgeHost` already |
| +exist and are already called from the spawn path. |
| + |
| +- **Boundary: closed by construction, and more tightly than any socket.** |
| + The host is the host of the request the edge is *currently serving* — |
| + `r.Host`, which the edge resolved from its own routing table before it |
| + ever dialled the upstream. The header never names a host and the edge |
| + never accepts one. A tenant cannot address another host because there is |
| + no field in which to address it. There is also nothing to *reach*: no |
| + socket, no port, no path, no token, so "which tenants can reach this |
| + channel" has no answer beyond "each tenant's own responses." |
| +- **Failure modes:** the channel only exists during a request/response |
| + cycle. A write that does not happen inside an HTTP request the edge |
| + proxied — a background job, a systemd timer, a scheduled sales close, an |
| + operator running the app's CLI — cannot purge. Those fall back to the |
| + TTL ceiling, exactly as today. This is the design's real cost and §10 |
| + turns it into the decision Paul has to make. |
| +- **Blast radius of a bug:** an app that emits the header on every |
| + response effectively disables its own cache. Its own host, nobody |
| + else's, bounded by re-fetching from its own backend. |
| +- **Complexity: the lowest of the five.** No unit file change, no |
| + drop-in, no group, no user, no listener, no port, no secret, no watcher, |
| + no new file anywhere on disk. The entire platform-side change is inside |
| + `internal/edge`. |
| +- **Old binaries:** they emit no header and nothing changes for them. An |
| + app that emits it while deployed behind something that is not a CARLOS |
| + edge leaks a header naming its own URLs — negligible, and the edge |
| + strips it before the client sees it. |
| +- **Prior art:** this is the `Surrogate-Key`/`Surrogate-Control` family |
| + (Fastly, Varnish) and nginx's `X-Accel-*` — an upstream steering its own |
| + cache through response metadata is a well-trodden pattern, not a novel |
| + trick. |
| + |
| +## 5. Recommendation |
| + |
| +**Ship (e): purge as a response header, for both backends.** |
| + |
| +It is the only candidate that satisfies the tenant-isolation invariant by |
| +construction rather than by permission bookkeeping, and the only one that |
| +requires no unit file change, no drop-in, no daemon-reload, no restart, no |
| +new Unix account, no new listening endpoint, and no new secret. It is also |
| +the only one that closes the *reachability* half of the invariant, which |
| +even the existing exec-backed control socket does not (`purge.go` is |
| +honest that a same-uid sibling can dial another instance's control |
| +socket). A unit tenant would get a better isolation story than an exec one |
| +— and over time exec tenants can migrate to the same channel and |
| +`CARLOS_CONTROL_SOCKET` can be deprecated, though that is not proposed |
| +here. |
| + |
| +Its cost — no out-of-band purge — is real and is stated in §10 rather than |
| +argued away. Keep candidate (a) on the shelf: if out-of-band purge turns |
| +out to be required, (a) is the design to build for it, and (e) does not |
| +conflict with it. |
| + |
| +## 6. Contract |
| + |
| +### 6.1 Response header (app → edge) |
| + |
| +- **Name:** `Carlos-Purge`. |
| +- **Value:** a comma-separated list of targets. The header is |
| + **list-valued and may be repeated**; the edge reads *every line* via the |
| + existing `joinHeader`, never `Header.Get`. (This repo has paid for that |
| + lesson four times on `Cache-Control`, `Vary`, and `Connection`.) |
| +- **Target:** an absolute-path URL with optional query — `/events/42`, |
| + `/events/42?lang=fr` — or the single token `*`. |
| +- **`*`** means "every cache entry for this host": `Cache.PurgeHost`, the |
| + same primitive the exec spawn path already invokes. |
| +- **Query-variant semantics.** A target's shape decides its breadth: |
| + - **Bare path, no `?`** (`/events/42`) purges **every cached entry under |
| + that path regardless of query string** — `?utm_source=…`, `?lang=fr`, |
| + and the unqueried one, in one shot, via `Cache.PurgePath(host, path)`. |
| + This is what an app almost always means: after a write to `/events/42` |
| + the campaign-link and locale-switcher copies are exactly as stale as the |
| + bare one, and an exact-query purge left them serving the pre-write body |
| + until their TTLs expired. `PurgePath` rides the cache's existing |
| + `byPath` index (both the GET and HEAD method groups, mirroring |
| + `PurgeURL`), so it is two map lookups, not a scan. |
| + - **With a `?`** (`/events/42?lang=fr`) is an **exact-variant** purge — |
| + `Cache.PurgeURL(host, path, rawQuery)`, unchanged — for an app that |
| + really does mean "only the French one". |
| + - **`/events/42?`** (a `?` with an empty query) is the exact empty-query |
| + variant, and is how an app asks for "just the unqueried one" now that |
| + the bare form means "all of them". |
| +- **Percent-encoding: the path is decoded, the query is raw.** This is not a |
| + choice; it is what the store side does, and purge must key identically or |
| + it silently purges nothing. `edge.go` builds every key as |
| + `CacheKey{…, Path: r.URL.Path, RawQuery: r.URL.RawQuery}`, and `net/url` |
| + gives `URL.Path` **already decoded** while `URL.RawQuery` is the query |
| + **byte-for-byte as received**. So the edge `url.PathUnescape`s a target's |
| + path half and compares its query half verbatim: `/caf%C3%A9` purges the |
| + entry stored under `/café`, and `?q=caf%C3%A9` matches a request whose |
| + query was literally `q=caf%C3%A9` — raw against raw. A path that will not |
| + decode is refused rather than guessed at. |
| +- **No host field exists.** The edge purges only `r.Host` for the request |
| + it is serving. A target containing a scheme or an authority is |
| + **rejected** and a warning logged, never interpreted. |
| +- **Rejection is per-target, not per-line.** One malformed target does |
| + **not** discard the valid targets beside it — the edge ignores the bad one |
| + and honours the rest. Under-purging the whole declaration because one |
| + entry was mistyped is the wrong failure direction (§6.1's over-purge |
| + principle), and it would let a single bad target silently disable an app's |
| + entire invalidation. The refusal gate is applied twice: once to the target |
| + as declared, and again to the percent-decoded path, because decoding |
| + manufactures the very shapes the gate exists to refuse (`/%2Fhost` |
| + decodes to `//host`). The one deliberate asymmetry is that a **decoded** |
| + space is allowed where a raw one is refused — `%20` decodes to a byte |
| + `r.URL.Path` genuinely contains, so refusing it would make `/a b` |
| + permanently unpurgeable. |
| +- **Any status.** The purge is honoured on 2xx, 3xx, 4xx, and 5xx alike — |
| + a POST-redirect-GET write signals on its 302. This is deliberately |
| + decoupled from cacheability, which remains 2xx-only. |
| +- **Ordering:** the edge applies the purge **before** `maybeStore` for the |
| + same response, so a cacheable GET that also purges stores its fresh self |
| + rather than purging what it just stored. |
| +- **Final-response headers only.** The edge reads and strips the header off |
| + the **final** response's header block, in `ModifyResponse`. Two header |
| + blocks pass through without reaching that seam, and neither is a purge |
| + channel: |
| + - **1xx informational responses.** A `Carlos-Purge` on a `103 Early |
| + Hints` block is neither honoured **nor stripped** — the stdlib forwards |
| + 1xx to the client on a path this code never sees, so the client *does* |
| + see it. |
| + - **Trailers.** A header announced via `Trailer` and sent after the body |
| + is not in `resp.Header` when the hook runs, so it is likewise neither |
| + honoured nor stripped. |
| + |
| + Both are pinned by tests rather than fixed. Closing them would mean |
| + wrapping the client's `ResponseWriter` to filter 1xx and trailer writes, |
| + and the upgrade path deliberately hands the **raw** `ResponseWriter` to |
| + the stdlib — a wrapper there re-opens the WebSocket-breakage class this |
| + repo has already paid for. Narrowing the promise is the honest trade: |
| + **apps must declare purges in final-response headers, and nowhere else.** |
| +- **Never name `Carlos-Purge` in `Connection`.** That marks it hop-by-hop, |
| + and `httputil.ReverseProxy` deletes hop-by-hop headers **before** |
| + `ModifyResponse` runs, so the declaration is silently eaten and nothing is |
| + purged — with no signal to the app author. Pinned by a test. |
| +- **Stripping:** the header is removed from the final response before it |
| + reaches the client. It is never forwarded, never cached as part of a |
| + stored response, and **never read from a request**. It is also |
| + **scrubbed off every proxied request on the way in** (both the unix- and |
| + TCP-proxy `Rewrite` funcs), so the app never sees a client-supplied one |
| + either. That second half matters because the app is the other half of the |
| + contract: an app that reflects request headers onto its response — a debug |
| + echo, a framework pass-through list, a CORS mirror — would launder a |
| + client's header into an app-*declared* one, which **is** honoured. The |
| + scrub makes that impossible regardless of how any app behaves. |
| +- **Ordering is per-response, not global.** "Purge before store" holds for |
| + the response carrying the header and no other. A concurrent request for |
| + the same URL whose response is already in flight will still be stored |
| + *after* the purge runs, resurrecting a body that predates the write. The |
| + staleness this reintroduces is bounded by the TTL ceiling rather than |
| + eliminated. Pre-existing and identical for `purge.go`'s socket channel; a |
| + fix needs store-time fencing (a per-URL purge generation checked in `Put`) |
| + and is deferred. |
| +- **Bounds:** at most 64 targets and 8 KiB of joined header per response. |
| + Beyond either bound the edge degrades to `*` for that host. The failure |
| + direction is deliberately over-purge, never under-purge: an extra |
| + re-fetch is cheap, and the parent spec's decision 3 puts freshness above |
| + cleverness. |
| +- **Vary:** whichever breadth a target selects, it drops *all* `Vary` |
| + variants of everything it matches — the existing `PurgeURL` semantics, |
| + and `PurgePath`'s by construction (it sweeps whole `byPath` groups). |
| +- **Bounds cost, honestly.** Degrading to a whole-host purge is the |
| + **simple** path, not the cheap one: `PurgeHost` takes the cache's global |
| + lock and scans *every* entry (4096 at `DefaultCacheMaxEntries`), where |
| + honouring 64 targets is 128 map lookups — roughly **32× more** work at a |
| + full cache. The caps exist to give per-response work a *ceiling that does |
| + not scale with header length*, not because degrading is cheaper. An app |
| + gains nothing by exceeding them except a larger blast radius on its own |
| + cache. |
| + |
| +### 6.2 Environment (edge → app), advisory only |
| + |
| +- **`CARLOS_PURGE=header`** is set in the tenant's environment when this |
| + box's edge has a cache and understands the header. It exists so an app |
| + can log or branch; **the channel works whether or not the app ever sees |
| + it**, because the header is inert when unread. This is what makes the |
| + rollout ordering in §7 harmless — unlike `CARLOS_CONTROL_SOCKET`, an |
| + advisory var cannot become a lie. |
| +- **`CARLOS_CONTROL_SOCKET`** is unchanged for exec-backed instances and |
| + is still **not set** for unit-backed tenants. Nothing about PR #12's |
| + refusal is reversed here. |
| +- No new flags. No changes to `carlos-app@.service`, `carlos-app@.socket`, |
| + or `carlos-edge.service`. |
| + |
| +### 6.3 Registry and bucket wire shapes |
| + |
| +Unchanged. No manifest field, no route field, no pointer field. Nothing |
| +written by an older `carlos` needs to parse differently, because nothing |
| +new is written. |
| + |
| +## 7. Migration and rollout |
| + |
| +1. **Platform binary.** The change lives entirely in `internal/edge`. |
| + Ship it through the normal channel (`carlos ship` → `promote`). |
| +2. **Adopt stages it; adopt does not activate it.** Adoption swaps the |
| + `/opt/carlos/live` symlink and does not restart processes. |
| + `carlos-edge.service` is a long-running unit, so it needs the explicit |
| + restart: **promote → adopt → restart `carlos-edge.service`**. Until |
| + that restart, the box's edge does not read the header and apps that |
| + emit it are simply not purging — which is exactly today's behaviour, so |
| + the intermediate state is safe. |
| +3. **`CARLOS_PURGE=header`** — **DEFERRED, and exec-only as shipped.** The |
| + var is currently set by `internal/activator`'s exec backend for |
| + exec-backed instances only. Delivery to **unit** tenants via the |
| + `internal/appconfig` sync into `/etc/carlos/apps/<host>.env` on the root |
| + adopt path is *not* built yet; it is tracked as follow-on work. |
| + |
| + This costs the feature nothing, which is the point of the var being |
| + advisory: **the channel works whether or not the app ever sees it**, so a |
| + unit tenant that emits `Carlos-Purge` today is honoured today, variable |
| + or no variable. When the sync does land it takes effect on that unit's |
| + next restart — hibernating tenants pick it up on wake, per the standard |
| + rule — and because the var is advisory there is no window in which it |
| + means something untrue. |
| + |
| + Where the var *is* set, it is **agent-owned in both directions**: set |
| + when there is a cache, and **removed** from the child's environment when |
| + there is not. The removal half is load-bearing, not decoration. The env |
| + layers (`<host>.config.env` from the bucket, `<host>.env` on the |
| + instance) are **tenant-supplied**, so gating only on "does the agent set |
| + it" would let a tenant set it itself and advertise a channel that does |
| + not exist on a cacheless box — an app trusting the var would then skip |
| + its own invalidation and serve stale content indefinitely. |
| + `CARLOS_CONTROL_SOCKET` is stripped in the same branch for the same |
| + reason, and a stronger one: a tenant-chosen socket path is a name the |
| + agent never vouched for. |
| +4. **Apps opt in independently and in any order.** An app that starts |
| + emitting `Carlos-Purge` before its box's edge has restarted loses |
| + nothing; an edge that reads the header before any app emits one changes |
| + nothing. There is no coordinated flag day and no rollback coupling |
| + between the two sides. |
| +5. **Rollback** is a binary rollback of the edge. Apps keep emitting a |
| + header nobody reads, which is inert. |
| +6. **First consumer:** titogo's write paths (parent spec §6.3) switch from |
| + "call the local purge channel" to "set `Carlos-Purge` on the response |
| + they are already returning," which is strictly less code on the app |
| + side — no client, no socket path, no error handling, no fire-and-forget |
| + goroutine. |
| + |
| +## 8. Testing |
| + |
| +The test that matters is the one PR #12 did not have: a test that proves |
| +the channel is *reachable by the tenant*, not merely that the platform set |
| +a variable naming it. The rule this spec proposes, beyond its own feature: |
| +**an environment variable that names a channel must have a test that |
| +exercises that channel from a process holding the tenant's real |
| +privileges.** |
| + |
| +1. **`TestPurgeChannelReachableFromForeignUid`** — the regression test for |
| + the PR #12 failure, written so it would have failed then. The test |
| + process forks a helper via `os/exec` with |
| + `SysProcAttr.Credential{Uid, Gid}` set to an unprivileged id that |
| + shares nothing with the test process (skipped unless running as root; |
| + run in the privileged e2e lane). The helper — not the test — performs |
| + the purge, and the test asserts the cache entry is gone. Under the |
| + reverted socket design the helper would have died on |
| + `connect: permission denied`, which is the whole point. Under the |
| + recommended design the helper *is* the upstream: it serves an HTTP |
| + response carrying `Carlos-Purge`, the edge proxies to it, and the |
| + assertion is that a foreign-uid upstream successfully invalidated its |
| + own host's entry. |
| +2. **`TestPurgeHeaderCannotTouchAnotherHost`** — the isolation half. Two |
| + hosts are cached; host B's upstream returns `Carlos-Purge: /x` (and |
| + variants attempting `https://a.example/x`, `a.example/x`, `//a.example/x`). |
| + Host A's `/x` must survive every one of them, and the malformed targets |
| + must be ignored rather than reinterpreted. |
| +3. **`TestPurgeHeaderStrippedFromClientResponse`** — the client never sees |
| + `Carlos-Purge`, on a cached-and-then-served response as well as a |
| + pass-through one. |
| +4. **`TestPurgeHeaderOnNon2xx`** — a 302 and a 404 both purge; neither is |
| + stored. Guards the deliberate decoupling in §6.1. |
| +5. **`TestPurgeBeforeStore`** — a cacheable GET whose response carries |
| + `Carlos-Purge` for its own URL ends with the *new* response cached, not |
| + with an empty entry. Ordering regression. |
| +6. **`TestPurgeHeaderBoundsDegradeToHost`** — 65 targets, and an |
| + oversized header, each purge the whole host rather than partially |
| + applying. Asserts the over-purge failure direction explicitly. |
| +7. **`TestListValuedPurgeHeader`** — the header split across two `Add()` |
| + calls is fully honoured. The fourth instance of this bug class in this |
| + package would be the first one caught before merge. |
| +8. **End-to-end, on a box with a real `carlos-app@<host>.service`** — |
| + the parent spec §8's three-step sequence (cold miss wakes and populates |
| + → repeat hits without waking → write plus purge makes the next request |
| + miss and reflect the change), run against a **unit-backed, hibernating, |
| + `DynamicUser` route**. This is the only test that exercises a genuine |
| + systemd-minted ephemeral uid, and it is the acceptance gate for the |
| + feature. |
| + |
| + **This runs at deploy time, not in CI**, and is not a merge gate for the |
| + edge change. It needs root and a real `carlos-app@.service`, neither of |
| + which the unit-test lane has; the same is true of |
| + `TestPurgeChannelReachableFromForeignUid` in item 1. Both are on-box |
| + acceptance steps, to be run on the first box that takes the restart in |
| + §7.2 — the feature is not "accepted" until they pass there, but the code |
| + merges on the in-process coverage above. |
| +9. **A no-inert-feature assertion** — `CARLOS_PURGE` is set only when |
| + `Edge.cache != nil`, mirroring `backend_exec.go`'s existing "only set |
| + when there's actually a purge listener to talk to" discipline. |
| + |
| +## 9. Deliberately not built |
| + |
| +- **Out-of-band purge for unit tenants.** A write that does not ride an |
| + HTTP response the edge proxied cannot purge. Scheduled transitions, |
| + background jobs, and app-CLI writes stay TTL-bounded. This is the |
| + design's headline cost, accepted here and escalated in §10. |
| +- **Cross-host purge, in any form.** No syntax for it, no privileged |
| + tenant, no admin override. If a legitimate need appears it is a |
| + different feature with a different authorisation story. |
| +- **`Carlos-Purge` on requests.** Read only from upstream responses, and |
| + **deleted from every proxied request before the app sees it** (§6.1). A |
| + client-supplied purge header is exactly the externally-reachable purge |
| + endpoint the parent spec §7 rejects as a wake-storm vector; ignoring it at |
| + the edge is necessary but not sufficient, because an app that echoes |
| + request headers would re-declare it on the way back out. |
| +- **A static per-tenant user tier** (PR #12's second sketch). Rejected in |
| + §4(a): it trades the unit template's isolation model for cache |
| + freshness. |
| +- **Retiring `CARLOS_CONTROL_SOCKET`.** Exec tenants keep it, gap and all. |
| + Converging them onto the header is attractive and is deferred. |
| +- **Purge acknowledgement or delivery guarantees.** Purge stays |
| + best-effort and fire-and-forget, bounded by the TTL ceiling — parent |
| + spec decision 4, unchanged. Riding a response the app is returning |
| + anyway makes it *more* reliable than a socket call the app might fail to |
| + make, but it is still not a guarantee. |
| +- **Purge on hibernation.** A tenant going to sleep does not flush its |
| + cache; that is the cache doing its job while the tenant is parked. |
| + |
| +## 10. Open questions |
| + |
| +1. **The decision.** Is request-coupled purge enough for unit tenants, or |
| + do they need an out-of-band channel too? Everything above hinges on |
| + this. The one concrete case in the parent spec that request-coupling |
| + cannot serve is a **scheduled** sales open/close — a state transition |
| + driven by wall-clock time, not by a request — and the parent spec's own |
| + `SalesEndAt` default change reduces but does not eliminate it. If "the |
| + scheduled transition is stale for up to the TTL ceiling" is acceptable, |
| + ship (e) alone and nothing else is needed. If it is not, (e) still |
| + ships (it is cheap and strictly better isolation) and candidate (a)'s |
| + per-route group machinery has to be built alongside it. **This is |
| + Paul's call and it is the only thing blocking a plan.** |
| +2. **Header name.** `Carlos-Purge` versus reusing the de-facto |
| + `Surrogate-Key` vocabulary, which some app frameworks already emit and |
| + which would make an app portable between CARLOS and a Fastly-style CDN. |
| + Against: inheriting `Surrogate-Key`'s tag semantics means inheriting a |
| + tag index in the cache, which is more machinery than URL targets need |
| + today. |
| +3. **Whether exec tenants converge.** Deprecating `CARLOS_CONTROL_SOCKET` |
| + in favour of the header would close the same-uid sibling gap |
| + `purge.go` documents, at the cost of a deprecation cycle across app |
| + binaries. Not proposed here; worth a decision once (e) has run in |
| + production. |
| +4. **Bounds tuning** — the 64-target and 8 KiB numbers in §6.1 are |
| + plausible starting points, not measured ones, exactly like the parent |
| + spec's TTL and cache-size figures. |
| + |
| +## 11. Provenance |
| + |
| +Written 2026-08-03, directly out of |
| +[PR #12](https://github.com/carlosframework/platform/pull/12)'s revert of |
| +the unit-tenant purge listener and the two rough sketches its body |
| +recorded ("a systemd-bound socket unit mirroring `carlos-app@.socket`'s |
| +root-binds-as-carlos trick in reverse, or a static per-tenant user tier"), |
| +both of which are enumerated and disposed of above. The honest isolation |
| +comment in `internal/edge/purge.go` — which states plainly that a |
| +`DynamicUser` tenant "can't reach ANY control socket, including one built |
| +for itself" — is the problem statement this document started from. |