| new file mode 100644 |
| index 000000000..2dea7b998 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-07-named-unit-config-delivery-design.md |
| @@ -0,0 +1,437 @@ |
| +# Config delivery for routes that name their own unit |
| + |
| +**Date:** 2026-09-07 |
| +**Status:** approved in chat (Paul, 2026-09-07), written up for a codex |
| +second read before the plan. Every choice below is marked **(decision)** |
| +and can be overturned cheaply; the two in §8 are Paul's **(ruling)**s of |
| +2026-09-07 and are marked where they land. Drafted by Claude Fable from |
| +amadan discussion #114 (carlos/platform), which follows #16 (delivered |
| +but not bounced) and issue #110 (the `Unit` column). Line numbers cite |
| +`origin/main` at `8e16153`. |
| + |
| +## 1. The problem, as found |
| + |
| +Discussion #114 reports the Woodstar community-relays release: a secret |
| +set through the door with `carlos secrets set -app woodstar-relay` is |
| +listed by `secrets list`, and the live relay runs without it. The relay's |
| +process is a hand-built unit, `woodstar-relay.service`, installed by a |
| +script staged in S3 and never committed anywhere. Its route is what the |
| +console shows today: |
| + |
| +``` |
| +host woodstar-relay.bdm.oncarlos.com |
| +kind instance |
| +upstream 127.0.0.1:8451 |
| +environment (unbound) |
| +unit woodstar-relay.service |
| +backing "" (an exec child, as far as the column knows) |
| +``` |
| + |
| +The reporter looked in `/etc/carlos/apps/` and found nothing rendered. |
| +That is correct, and it is only half the picture. The app's default bundle |
| +carries `exec-delivery: file` (`carlos env list -app woodstar-relay` prints |
| +it), so the route matches `execAddrCandidate` |
| +(`internal/appconfig/syncexec.go:509`) plus the opt-in, and the agent's |
| +in-process sweep should be writing the merged bundle to |
| +`/data/woodstar-relay.bdm.oncarlos.com.config.env`. The unit does not read |
| +that file either. Nothing on the box reads any file carlos writes for this |
| +route, and nothing told the person who ran `secrets set`. The only way the |
| +value has ever reached the process is a root drop-in written over SSM, |
| +which is the shape Paul has ruled out: **we do not want to be using SSM |
| +for any of this as much as possible.** |
| + |
| +Three asks came with the report. This spec builds the first two and |
| +defers the third (§5, ruling 2): |
| + |
| +1. deliver config to a route whose process is a unit that is not |
| + `carlos-app@<host>`; |
| +2. say so at set time when a write reaches nothing; |
| +3. a way to ask which config generation a process is actually running. |
| + |
| +## 2. What is already right, and must stay right |
| + |
| +**The root pass is the only writer of `/etc/carlos/apps` and the only |
| +thing that restarts units.** `appconfig.Sync` (`internal/appconfig/ |
| +sync.go:42`) runs from `carlos adopt`, root, out of `carlos-adopt.service` |
| +hourly and `carlos-config-sync.service` on the agent's request |
| +(`internal/host/agent.go:2839-2860`). The agent's own unit is |
| +`ProtectSystem=strict` and cannot write there. `carlos-config-sync.service` |
| +(`internal/host/units/carlos-config-sync.service`) has no `ProtectSystem` |
| +line, so it can write under `/etc/systemd/system` as well; nothing new is |
| +needed for the drop-in in §3.2. |
| + |
| +**Restart dispatch already has the named-unit arm.** `makeRestartRoute` |
| +(`agent.go:2543`) tries `rt.Unit` first, through `systemctlTryRestartUnit` |
| +(`agent.go:2637`), which returns its failure so that adoption can roll a |
| +symlink back (`makeStartRoute`, `agent.go:2593`). Sync is handed the same |
| +callback (`agent.go:2950`), and a comment there records that the arm is |
| +unreachable from Sync today only because Sync skips every route whose |
| +`Backing` is not `unit` (`sync.go:164-166`). This spec makes it reachable. |
| + |
| +**`Unit` and `Backing=unit` are exclusive.** `carlos route --unit` refuses |
| +a unit-backed row and `--backing unit` refuses a stamped one |
| +(`cli.go:483`, `cli.go:612-630`); the column's doc comment |
| +(`internal/registry/registry.go:148-170`) says a stamped route is an |
| +addr-backed route whose process no backing-dispatched restart could reach. |
| +`ValidUnitName` (`registry.go:301-316`) pins the name to |
| +`^[A-Za-z0-9@_][A-Za-z0-9@_.\-]{0,63}\.service$`, so a stamp can never |
| +carry a path separator into a root-written path. |
| + |
| +**The two config halves share one set of predicates, in one package.** |
| +`syncable` (`sync.go:284`) is `release.Adoptable` |
| +(`internal/release/adopt.go:816`); `execRouteBase` (`syncexec.go:478`) |
| +narrows it to non-unit backing; `UnitConfigApps` (`sync.go:299`), |
| +`ExecConfigApps` (`syncexec.go:533`), `ConfigRouteFingerprints` |
| +(`internal/appconfig/routefingerprint.go:58`) and `AwaitingRestart` |
| +(`syncexec.go:679`) all compose those two. Each says why it lives beside |
| +its predicate: an app the fast loop watched but the sweep skipped burns |
| +reads forever, and one it did not watch falls back to the hourly timer. |
| +The change in §3.1 is one predicate edit on each side for that reason. |
| + |
| +**The set-time notice has three states, not two.** `RestartPending` |
| +(`syncexec.go:647`) separates hosts delivered to from hosts it could not |
| +decide about, and the CLI renders "could not check" as its own line |
| +(`appconfig_cli.go:94-120`, copy at `:152-156`); a console predating the |
| +field sends no `restart` key and the CLI prints nothing. #16's history is |
| +the reason: "nothing needs a restart" and "I could not tell" rendered |
| +identically is how that discussion got filed. §3.4 adds a state and keeps |
| +the discipline. |
| + |
| +**Systemd reads `EnvironmentFile` as pid 1.** The rendered file is |
| +root-owned `0600` (`sync.go:425`), which is fine for a unit running as any |
| +user, because systemd reads the file before it drops privileges. And |
| +`EnvironmentFile=` settings override `Environment=` lines in the same |
| +unit, so a rendered file wins over the hand-copied |
| +`Environment=WOODSTAR_APPLE_APP_ID=…` the Woodstar units carry today. |
| + |
| +## 3. The change |
| + |
| +### 3.1 Ownership rule: a route that names its unit is the root pass's |
| + |
| +**(decision)** A route with `Unit != ""` is delivered to, and restarted, |
| +by the root pass, exactly as a `Backing=unit` route is. The agent's exec |
| +sweep never touches it. |
| + |
| +Concretely, in `internal/appconfig`: |
| + |
| +- a new predicate `unitDelivered(rt)` = `syncable(rt) && (rt.Backing == |
| + registry.BackingUnit || rt.Unit != "")`, used by `Sync`'s per-route skip |
| + (`sync.go:164`), by `UnitConfigApps` (`sync.go:303`) and by the unit |
| + half of `ConfigRouteFingerprints` (`routefingerprint.go:71`); |
| +- `execRouteBase` (`syncexec.go:478-482`) gains `&& rt.Unit == ""`, which |
| + removes stamped routes from `execSyncable`, `execAddrCandidate`, |
| + `ExecConfigApps`, the exec fingerprint and `AwaitingRestart` at once. |
| + |
| +`Sync`'s skip message (`sync.go:165`) stays for the routes it still skips, |
| +which after this are exec routes with no stamp. The comment at |
| +`agent.go:2899-2907` that explains why Sync never reaches the named-unit |
| +arm is rewritten to say that it now does. |
| + |
| +`release.Adoptable` is untouched; a stamped route was adoptable before and |
| +still is. `restartApps` (`internal/host/restart.go:72`) and the |
| +`-restart-only` pass are untouched: they already admit stamped routes and |
| +dispatch through the same arm. |
| + |
| +### 3.2 Root delivery for a named-unit route |
| + |
| +**(decision)** For a `unitDelivered` route with a stamp, `Sync` renders |
| +`<envDir>/<host>.env` through `syncHost` (`sync.go:331`) unchanged: the |
| +same resolution, the same key-loss guard for a bound route whose app has no |
| +default bundle, the same atomic write, the same `applyAgentVars` line. The |
| +only addition is that, before the restart, the pass makes sure the unit |
| +reads the file: |
| + |
| +``` |
| +/etc/systemd/system/<unit>.d/carlos-env.conf |
| + |
| +# Managed by carlos: config delivery for the route(s) naming this unit. |
| +# Edit the route's config with `carlos env` / `carlos secrets`, not this file. |
| +[Service] |
| +EnvironmentFile=-/etc/carlos/apps/<host>.env |
| +``` |
| + |
| +**(decision)** The drop-in file name is fixed, `carlos-env.conf`, and the |
| +directory is `<unit>.d` beside the unit, which is how systemd finds |
| +drop-ins for a unit whether the unit itself lives in `/etc/systemd/system` |
| +or `/usr/lib/systemd/system`. The pass never reads or edits the unit file |
| +itself and never touches any other drop-in. |
| + |
| +**(decision)** One drop-in per unit, one `EnvironmentFile=` line per route |
| +naming that unit, sorted by host. Two routes on one unit is not a shape |
| +anything provisions today, but the file must be deterministic if it |
| +happens, and the alternative, refusing the second route, would leave one |
| +of them silently undelivered, which is the failure this spec exists to |
| +end. The drop-in's content is a pure function of the set of stamped, |
| +unit-delivered routes naming the unit, so it is computed once per pass per |
| +unit, not per route. |
| + |
| +**(decision)** The drop-in is written only when a file is being rendered |
| +for at least one route naming the unit: an app that has never been |
| +configured (`Sync`'s "no manifest object at all" state, `sync.go:114` and `:153`) |
| +gets no drop-in, so a console route stamped under #110 for an app with no |
| +config is left exactly as it is. Once a route has config, its drop-in |
| +exists for as long as the route does; the `-` prefix makes an absent env |
| +file harmless in between. |
| + |
| +**(decision)** Writes are atomic (temp file, rename) and mode `0644`: the |
| +drop-in carries a path, not a secret. A changed or new drop-in is followed |
| +by one `systemctl daemon-reload`, batched once per pass after all drop-ins |
| +are written and before any restart, so a pass that touches several units |
| +reloads once. `daemon-reload` failing is logged per unit and the restart |
| +is skipped for that unit, because a restart that does not load the new |
| +file would stamp nothing true. |
| + |
| +**(decision)** The restart fires when the env file changed, as today, and |
| +also when the drop-in was created or changed while the env file did not, |
| +because a unit that has never loaded the file is running without it |
| +however old the file is. `syncHost` therefore reports whether it wrote, |
| +and the caller combines that with the drop-in result before calling |
| +`restart(rt.Host)`. The callback is the existing `rootRestartRoute` |
| +(`agent.go:2684`); its first arm reaches `try-restart <unit>` and returns |
| +the failure, which Sync already prints as "config updated but restart |
| +failed" (`sync.go:436-438`). |
| + |
| +**(decision)** Reaping. After the route loop, the pass lists |
| +`/etc/systemd/system/*.d/carlos-env.conf`, and removes every one whose |
| +unit is not named by any route row on the box. The same rule as `reap` |
| +for env files (`sync.go:196`): keyed off the full route list, not the |
| +delivered subset, so a route mid-provisioning or unresolved keeps its |
| +drop-in; and an empty route list is never authority to delete anything |
| +(`sync.go:175-178`). A removed drop-in also triggers the batched |
| +daemon-reload. A unit that was unstamped (`carlos route --unit -`) or |
| +whose route was removed is therefore left with no carlos line in it on the |
| +next pass; the unit keeps running on whatever it loaded until something |
| +else cycles it, which is right, because an unstamp says carlos no longer |
| +owns that process. Only files named exactly `carlos-env.conf` inside a |
| +`*.service.d` directory are ever candidates; the pass never removes the |
| +directory. |
| + |
| +Hibernation needs no special case: `try-restart` is a no-op on an |
| +inactive unit, as `syncHost` already notes, and provisioning refuses the |
| +addr-with-hibernate shape (`agent.go:2588-2590`). |
| + |
| +### 3.3 The exec side lets go |
| + |
| +**(decision)** `SyncExec` (`syncexec.go:65`) removes |
| +`<dataDir>/<host>.config.env` for a stamped route it would otherwise have |
| +delivered to: the route is in the box's registry with a `Unit`, and the |
| +file in sync's own namespace holds decrypted secrets that nothing on the |
| +agent's side will ever refresh again. `reapConfigEnv` (`syncexec.go:934`) |
| +only reaps hosts with no route row, so a route that moved halves would |
| +keep a stale secrets file forever without this. The removal goes through |
| +the same path the "never configured" state uses, which already removes |
| +rather than truncates; no bounce follows, because there is no child to |
| +bounce for an addr route and the root pass restarts the unit itself. |
| + |
| +The `exec-delivery: file` opt-in on `woodstar-relay`'s bundle becomes |
| +inert for its stamped routes and stays where it is; an operator may turn |
| +it off, and nothing depends on either. |
| + |
| +### 3.4 The set-time notice: a write that reached nothing |
| + |
| +**(decision)** `RestartPending` gains `Undelivered []string`: the hosts of |
| +routes the write reached whose process reads no file carlos writes. The |
| +predicate is `execAddrCandidate(rt)` (which after §3.1 excludes stamped |
| +routes) with the opt-in resolved to off, including the `Unknown` case |
| +`resolveExecDelivery` (`syncexec.go:596`) already decides as off. The |
| +loop in `AwaitingRestart` (`syncexec.go:722-770`) already computes exactly |
| +this verdict per route and `continue`s on `!v.On` with the comment |
| +"nothing to say"; that branch appends to `Undelivered` instead. Routes |
| +whose bundle would not load stay in `Unresolved`, never in `Undelivered`: |
| +"could not tell" is still never rendered as "not delivered". |
| + |
| +The same environment and account narrowing applies as for `Hosts`: a |
| +named-bundle write reaches the routes bound to it, a default-bundle write |
| +reaches every route of the app. The shadowing test does not apply, since |
| +the point is that no file exists to be unchanged. |
| + |
| +**(decision)** On the wire, `apiConfigRestart` (`internal/console/ |
| +apiconfig.go:495`) gains `undelivered []string, omitempty`. Additive: an |
| +older CLI ignores it and prints what it printed before; an older console |
| +sends no field and the newer CLI prints nothing for it. `Empty()` |
| +(`syncexec.go:655`) counts it. |
| + |
| +**(decision)** The CLI prints one line per host under the same `note:` |
| +prefix, then the existing scope caveat once. Copy direction, to be fixed |
| +by copy-review before the branch lands, not here: |
| + |
| +``` |
| +note: woodstar-relay.bdm.oncarlos.com has no file to read. The value is stored, not delivered. Name the unit that runs it (carlos route --unit, on the box) or opt the app into file delivery (carlos env exec-delivery --app woodstar-relay file). |
| +(this machine's routes only) |
| +``` |
| + |
| +**(decision)** With §3.1, `AwaitingRestart` can no longer return a host |
| +with a `Unit`, because a stamped route is not an `execAddrCandidate`. The |
| +`Unit` field on `RestartHost` and `apiConfigRestartHost`, the |
| +`restartCopyUnit` line (`appconfig_cli.go:153`) and |
| +`TestAwaitingRestartCarriesUnit` are removed. Dropping an `omitempty` |
| +field from a response is additive in the sense that matters: an older CLI |
| +that still reads `unit` sees it absent and prints the plain line, which is |
| +now the true one. A stamped route needs no remedy line because the root |
| +pass restarts it within seconds of the write, like a `carlos-app@` tenant. |
| + |
| +`carlos env sync`'s output already says files land in seconds and |
| +anything carlos does not own keeps its old values until restarted; it is |
| +still true and is not changed. |
| + |
| +### 3.5 Fingerprints and the fast loop |
| + |
| +**(decision)** The unit row of `ConfigRouteFingerprints` |
| +(`routefingerprint.go:67`) appends `rt.Unit`. A re-stamp to a different |
| +unit changes which drop-in the route belongs in without changing the env |
| +file's bytes, and the fast loop's registry half (`agent.go:834-841`) is |
| +what turns a registry change into a root pass request within a tick |
| +rather than an hour. The exec row does not carry it: a stamped route is |
| +not in that set. A stamp being added or removed already moves the route |
| +between the two sets and so changes both digests. |
| + |
| +`genPollPass` (`agent.go:829`) is otherwise untouched: `UnitConfigApps` |
| +now lists apps with stamped routes, so their generation object is polled |
| +and a write requests a root pass through `configSyncMarker` |
| +(`agent.go:888`) exactly as for `carlos-app@` tenants. |
| + |
| +### 3.6 Documentation |
| + |
| +`docs/site/cli.md`: the `carlos route --unit` paragraph (line 910) says |
| +that naming the unit also makes the root pass deliver the route's config |
| +to it, through a drop-in the pass manages, and restart it on change; the |
| +`carlos add --unit` paragraph (line 884) says the same; the `carlos env` |
| +and `carlos secrets` sections gain one sentence on the "no file to read" |
| +note and its two remedies. `internal/docsite` checks headings and |
| +examples, not this prose, so the gate does not force these, but a change |
| +that alters what a flag does is not done until the page says so. |
| + |
| +AGENTS.md's "Deploy reality" bullet on #110 (line 282) gains one clause: |
| +a stamped route also receives its config from the root pass. |
| + |
| +## 4. Consequences to be explicit about |
| + |
| +**One restart per stamped, configured route on the first pass after the |
| +roll.** Every such route gets a drop-in it did not have, and §3.2 restarts |
| +on a new drop-in. On today's fleet that is Woodstar's stamped units on |
| +their box and any console route stamped under #110 whose app has config. |
| +This is the same trade the `CARLOS_PURGE` roll accepted (`sync.go:401- |
| +420`): the restart is the feature landing, and a route that stays on the |
| +old file would be exactly the bug being fixed. A route whose app has no |
| +config is not restarted. |
| + |
| +**A stamped console cycles itself on its own config change.** After the |
| +roll, `carlos env set` on the console's app bounces `console.service` |
| +within seconds, as it would any tenant. That is what stamping means, and |
| +it is the behaviour #110 asked for on adoption; config joins it. An |
| +operator who wants a console's config delivered but not applied until a |
| +chosen moment has no verb for that today and did not before either. |
| + |
| +**The drop-in is a root write into a directory carlos does not own.** It |
| +is additive to the unit, named so that it reads as carlos's, carries no |
| +secret, is removed when the stamp goes, and is written only for a unit an |
| +operator named on the route. A hand-written drop-in with the same file |
| +name would be overwritten; the name is chosen to make that collision |
| +unlikely and the marker line makes it visible. |
| + |
| +**The stale `/data/<host>.config.env` disappears.** For a stamped route |
| +that was opted into file delivery, the agent's next sweep removes the |
| +file (§3.3). Nothing reads it, by the premise of #114, so nothing |
| +changes for the process; the point is that a decrypted secret is not left |
| +in a file no pass refreshes. |
| + |
| +**Two units, one process, is still one arm.** `makeRestartRoute` tries |
| +`Unit` and returns; it never falls through to `carlos-app@<host>` or the |
| +activator for a stamped route. Unchanged, and relied on. |
| + |
| +**Multi-box fleets.** The console reads its own box's registry for the |
| +notice, as today (#145's blind spot); the caveat line stays. Delivery |
| +itself runs on every box that serves the route, each from its own |
| +registry, so a stamped route on two boxes is delivered on both. |
| + |
| +## 5. What is not built |
| + |
| +- **Ask 3, delivered-generation readback.** Deferred **(ruling 2)**. |
| + #11, #13 and #16 want the same absent field; it needs an app-side |
| + signal and deserves one decision across all four threads. |
| +- **A remote door for `--unit`.** Stamping is `carlos route --unit` against |
| + the registry on the box (`cli.go:470-483`). AGENTS.md's rule makes a |
| + box-only step a product gap, and this one remains; it is not needed for |
| + Woodstar, whose route is already stamped, and provisioning a stamp from |
| + an instance record is a design of its own (the record cannot express |
| + `Backing` today either, `internal/provision/reconcile.go`). |
| +- **`UNIT` and `BACKING` columns in `carlos routes`.** The API already |
| + carries both (`internal/console/apiroutes.go:61-62`); printing them is |
| + a small change that belongs with ask 3's column, not here. |
| +- **A post-delivery hook** for the un-owned addr shape (#16's second |
| + ask). The stamp is the answer for a process a unit runs; the hook would |
| + be for processes nothing on the box supervises. |
| + |
| +## 6. Testing |
| + |
| +`internal/appconfig/sync_test.go`, beside `TestSyncSkipsExecBackedRoutes`, |
| +with the same registry-and-tempdir harness and a fake `restart` recorder; |
| +the drop-in root and the daemon-reload runner are parameters of the |
| +function under test so that no test touches `/etc` or runs `systemctl`: |
| + |
| +- a stamped route renders `<host>.env`, writes the drop-in with the |
| + expected bytes, calls daemon-reload once, restarts the host; |
| +- a second pass with nothing changed writes nothing, reloads nothing, |
| + restarts nothing; |
| +- an env-file change alone restarts; a drop-in change alone (a re-stamp |
| + to a new unit) restarts and the old unit's drop-in is reaped; |
| +- two stamped routes naming one unit produce one drop-in with two sorted |
| + lines and one reload; |
| +- an app with no manifest object writes no file and no drop-in; |
| +- the key-loss guard still refuses a bound route with no default bundle |
| + when the file on disk holds keys the render drops; |
| +- a drop-in for a unit named by no route is removed and reload follows; |
| + an empty route list removes nothing; |
| +- daemon-reload failure logs and skips that unit's restart; |
| +- `UnitConfigApps` and the unit fingerprint include a stamped route; |
| + the fingerprint changes on re-stamp. |
| + |
| +`internal/appconfig/syncexec_test.go` and `awaitingrestart_test.go`: |
| + |
| +- `execRouteBase` rejects a stamped route; `ExecConfigApps` and the exec |
| + fingerprint exclude it; |
| +- `SyncExec` removes an existing `<host>.config.env` for a stamped route |
| + and does not bounce; |
| +- `AwaitingRestart` lists an un-opted addr route in `Undelivered`, keeps |
| + an unreadable bundle in `Unresolved`, and never lists a stamped route |
| + anywhere; a named-bundle write lists only routes bound to it. |
| + |
| +`internal/console` and the CLI: the wire field round-trips; the CLI |
| +prints the note for `Undelivered`, prints nothing for a console that |
| +sends no field, and the copy tests are updated for the removed unit line. |
| + |
| +`internal/host`: the comment change at `agent.go:2899` has no test; the |
| +restart arm's existing tests (`restart_test.go`) already cover a stamped |
| +route. |
| + |
| +Everything runs under `go test ./... -count=1`; socket-binding tests in |
| +this repo need `TMPDIR=/tmp/g`. |
| + |
| +## 7. Rollout |
| + |
| +1. Land the branch on the forge; CI green on the amadan remote; push |
| + `main` to the GitHub mirror. |
| +2. Roll the platform binary. The first root pass on each box writes the |
| + drop-ins and cycles each stamped, configured route once (§4). |
| +3. Verify on Woodstar with no box access: `carlos secrets set -app |
| + woodstar-relay WOODSTAR_COMMUNITY_PROVISION_TOKEN=…` prints no note; |
| + within seconds `GET /api/communities` on the live relay answers |
| + `"creation_enabled": true`. That is the verification the platform can |
| + offer today; ask 3 is what would make it a column. |
| +4. Reply on #114 with what shipped, the one-time restart, and that no |
| + box step is needed. The Woodstar setup script can drop its |
| + `Environment=WOODSTAR_APPLE_APP_ID` line at its next edit, since the |
| + bundle carries the var and the rendered file overrides it anyway; that |
| + edit is in the private repo and is not part of this change. |
| + |
| +## 8. Rulings (Paul, 2026-09-07) |
| + |
| +1. **The root pass owns a named-unit route fully.** Render the env file, |
| + write the `EnvironmentFile=` drop-in, daemon-reload, try-restart on |
| + change. Stamping the unit is the consent. No SSM step, including for |
| + `console.service`, which cycles on its own config changes like any |
| + `carlos-app@` tenant. **(ruling 1)**, lands in §3.2 and §4. |
| +2. **Ask 3 is deferred to its own thread**, to be decided once across |
| + #11, #13, #16 and #114. **(ruling 2)**, lands in §5. |
| +3. The design as presented in chat (§3.1 to §3.6, §4, §6) was approved |
| + without change. |