amadan
host: unit-backed apps join config fast convergence (issue #44)
PR #45 gave exec-backed instances seconds-not-hours config convergence and left carlos-app@ routes on the hourly carlos-adopt.timer floor, because the agent cannot write /etc/carlos/apps or restart units. It can ask, though. genPollPass now polls the UNION of appconfig.ExecConfigApps and the new appconfig.UnitConfigApps — one generation read per app, however its routes are backed — and reports two bools. execChanged runs SyncExec exactly as before; unitChanged touches <dataDir>/config-sync-requested, which the new carlos-adopt.path relays into a root carlos-adopt.service run that already owns the unit half (appconfig.Sync + try-restart). No new privileged code: this process still writes nothing under /etc and restarts no unit. bootstrap embeds and arms the path unit the same way it arms timers — a path unit that is enabled but never started watches nothing until the next boot, which is the carlos-adopt.timer bug in a new suit. Two consequences, recorded in the spec as accepted: a PathChanged event landing while the adopt service is mid-run can be lost (the next write re-touches, the timer floor delivers within the hour), and the loop's CARLOS_SECRETS_KEY gate now covers the unit half too (both units read the same host.env in practice).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
9 files changed,
+428
−73
appconfig_cli.go+9 −5docs/superpowers/specs/2026-08-04-config-console-api-design.md+33 −5internal/appconfig/sync.go+34 −0internal/appconfig/syncexec_test.go+61 −0internal/host/agent.go+95 −25internal/host/agent_test.go+149 −18internal/host/bootstrap.go+23 −18internal/host/host_test.go+8 −2internal/host/units/carlos-adopt.path+16 −0
diff --git a/appconfig_cli.go b/appconfig_cli.go| index 784f8c240..de2bcb704 100644 |
| --- a/appconfig_cli.go |
| +++ b/appconfig_cli.go |
| @@ -44,8 +44,9 @@ when set. |
| sync touches apps/<acct>/<app>/config-gen.json — the agent's fast poll loop |
| (host package) notices within ~3s and re-renders exec-backed instances; |
| -unit-backed routes still wait for the next root adopt pass. It never |
| -restarts a box's shared edge service, only the one app's own delivery. |
| +unit-backed routes converge in seconds too, via a root adopt pass the same |
| +loop requests (the agent cannot write /etc or restart units itself). It |
| +never restarts a box's shared edge service, only the one app's own delivery. |
| Every command above also works through a logged-in console |
| (carlos auth login) when CARLOS_DEPLOYMENT_BUCKET/CARLOS_DEPLOYMENT_DIR are |
| @@ -448,8 +449,11 @@ func runEnvEnvironments(args []string) { |
| // config write that needs to reach a running instance faster than |
| // configSyncLoop's hourly floor: it touches apps/<acct>/<app>/config-gen.json |
| // (appconfig.GenKey), which the agent's configGenLoop polls every ~3s and |
| -// treats a moved etag as "run SyncExec now". It never restarts a box's |
| -// shared edge service and never targets a bundle directly — the generation |
| +// treats a moved etag as "run SyncExec now" for the app's exec-backed |
| +// instances and "ask the root adopt pass to run now" (carlos-adopt.path) for |
| +// its unit-backed ones, so both shapes converge in seconds rather than one |
| +// waiting out the hour. It never restarts a box's shared edge service and |
| +// never targets a bundle directly — the generation |
| // object covers every bundle an app has (default plus every environment) in |
| // one touch, matching TouchGeneration's own "app granularity, not |
| // per-bundle" design (generation.go). |
| @@ -476,7 +480,7 @@ func runEnvSync(args []string) { |
| log.Fatal(err) |
| } |
| fmt.Printf("config sync requested for %s — exec-backed instances converge within seconds; "+ |
| - "unit-backed routes on the next root adopt pass\n", *app) |
| + "unit-backed routes within seconds too, on the root adopt pass the agent requests\n", *app) |
| } |
| const secretsUsage = `usage: carlos secrets <command> |
diff --git a/docs/superpowers/specs/2026-08-04-config-console-api-design.md b/docs/superpowers/specs/2026-08-04-config-console-api-design.md| index b8e1fe040..90dc09cfd 100644 |
| --- a/docs/superpowers/specs/2026-08-04-config-console-api-design.md |
| +++ b/docs/superpowers/specs/2026-08-04-config-console-api-design.md |
| @@ -161,8 +161,30 @@ silently (boot must not re-bounce the world; `configSyncLoop`'s ~1min first |
| pass owns boot convergence). Any etag change — including absent→present — |
| triggers ONE `SyncExec` pass for the tick, not one per app. Absent stays |
| absent silently. Hourly `configSyncLoop` stays as the floor (old CLIs, |
| -missed ticks, #27's boot-retry gap). Unit-backed routes keep the root adopt |
| -path — the agent cannot restart units, and this spec does not try. |
| +missed ticks, #27's boot-retry gap). |
| + |
| +Unit-backed apps join the same fast path (amended 2026-08-04, approved). |
| +The poll set is the UNION of `appconfig.ExecConfigApps` and the new |
| +`appconfig.UnitConfigApps`, read once per app (one generation object per |
| +app however its routes are backed), and `genPollPass` reports two bools: |
| +`execChanged` runs `SyncExec` as before; `unitChanged` touches |
| +`<dataDir>/config-sync-requested`, which a new `carlos-adopt.path` |
| +(`PathChanged=/data/config-sync-requested` → `Unit=carlos-adopt.service`) |
| +relays into the ROOT pass that already owns the unit half — `appconfig.Sync` |
| +into `/etc/carlos/apps` plus try-restart. No new privileged code: the agent |
| +still writes nothing under `/etc` and restarts no unit; it only asks. An app |
| +holding both route shapes sets both halves off the one read. |
| + |
| +Two consequences, both accepted: |
| + |
| +- A `PathChanged` event landing while `carlos-adopt.service` is mid-run can |
| + be lost (systemd does not queue it). The next config write re-touches the |
| + marker, the exec half converged in-process regardless, and |
| + `carlos-adopt.timer` delivers within the hour. |
| +- The loop stays gated on `CARLOS_SECRETS_KEY`, so a box whose AGENT lacks |
| + the key gets no fast path for its unit apps either, even though the root |
| + pass that would run holds its own key. In practice both units read the |
| + same `/etc/carlos/host.env`; the timer floor stands regardless. |
| Cardinality note: unlike canary channels this polls every exec-config app, |
| but one small GET per app per 3s against the store the canary loop already |
| @@ -171,7 +193,9 @@ the read is the cheapest object in the bucket. |
| ## §5 Out of scope (recorded, deliberate) |
| -- Unit-backed route fast convergence (root adopt path owns it). |
| +- ~~Unit-backed route fast convergence (root adopt path owns it).~~ Closed |
| + by the §4 amendment above: the root adopt path still owns the delivery, |
| + the agent just asks it to run now. |
| - `#27` (configSyncLoop boot retry) — narrowed by this spec (the gen loop |
| re-arms convergence on the next write) but not fixed by it. |
| - Browser console config UI — CLI-only surface today, same as ledgers. |
| @@ -192,7 +216,11 @@ the read is the cheapest object in the bucket. |
| order (env → console → fatal). |
| - agent: the per-tick decision extracted as a pure-ish helper |
| (`genPollPass`) and tested (first-sight silent, change triggers, absent |
| - stays silent, one sync per tick); the loop itself stays untested by |
| - convention like adoptLoop/statusLoop. |
| + stays silent, one sync per tick; and per half — unit movement requests the |
| + root pass only, exec movement syncs only, a both-shapes app sets both off |
| + ONE read, cache pruning covers the union); the loop itself stays untested |
| + by convention like adoptLoop/statusLoop. `UnitConfigApps` mirrors |
| + `ExecConfigApps`' own test; the bootstrap unit-list and arm-the-timers |
| + tests cover `carlos-adopt.path` being installed, enabled AND started. |
| - e2e (auth_e2e_test.go pattern): login → env set → list → secrets set |
| (console-served pubkey) → list names → unset → sync → gen object moved. |
diff --git a/internal/appconfig/sync.go b/internal/appconfig/sync.go| index 8a471e7e0..083a29443 100644 |
| --- a/internal/appconfig/sync.go |
| +++ b/internal/appconfig/sync.go |
| @@ -285,6 +285,40 @@ func syncable(rt registry.Route) bool { |
| (rt.Status == "" || rt.Status == "ready") |
| } |
| +// UnitConfigApps lists the distinct (account, app) pairs having at least |
| +// one unit-backed syncable route on this box — the apps whose config the |
| +// ROOT adopt pass delivers (Sync + try-restart), which the agent itself |
| +// cannot. The fast loop uses it to decide when to request a root pass |
| +// (see configGenLoop): same dedupe-and-sort shape as ExecConfigApps, and |
| +// the two may overlap — an app can have both route shapes at once. |
| +// |
| +// Exported from THIS file, beside the predicate Sync itself gates on, for |
| +// the reason ExecConfigApps gives for living beside its own: an app the |
| +// loop watched but this file's sweep skipped would burn reads forever, and |
| +// one it didn't watch would silently fall back to the hourly timer. |
| +func UnitConfigApps(routes []registry.Route) [][2]string { |
| + seen := map[[2]string]bool{} |
| + var out [][2]string |
| + for _, rt := range routes { |
| + if !syncable(rt) || rt.Backing != registry.BackingUnit { |
| + continue |
| + } |
| + k := [2]string{release.RouteAccount(rt), rt.App} |
| + if seen[k] { |
| + continue |
| + } |
| + seen[k] = true |
| + out = append(out, k) |
| + } |
| + sort.Slice(out, func(i, j int) bool { |
| + if out[i][0] != out[j][0] { |
| + return out[i][0] < out[j][0] |
| + } |
| + return out[i][1] < out[j][1] |
| + }) |
| + return out |
| +} |
| + |
| // syncHost converges one route's env file. It is called only with a manifest |
| // that really resolved — the object (default bundle, named environment, or |
| // both) exists in the bucket — so an empty manifest here means "every var |
diff --git a/internal/appconfig/syncexec_test.go b/internal/appconfig/syncexec_test.go| index 9db2fda22..53d5519d9 100644 |
| --- a/internal/appconfig/syncexec_test.go |
| +++ b/internal/appconfig/syncexec_test.go |
| @@ -1567,3 +1567,64 @@ func TestExecConfigAppsNoRoutes(t *testing.T) { |
| t.Fatalf("ExecConfigApps(nil) = %v, want empty", got) |
| } |
| } |
| + |
| +// TestUnitConfigApps is TestExecConfigApps' mirror for the other half of the |
| +// fast loop's poll set: the apps whose config only the ROOT adopt pass can |
| +// deliver. Same fixture shapes, opposite inclusion on Backing — and the |
| +// both-shapes app below is the case that matters most, because an app can |
| +// hold a socket-exec route and a unit route at once and must appear in BOTH |
| +// functions' outputs (the loop then syncs exec in-process AND asks for a |
| +// root pass off one generation read). |
| +func TestUnitConfigApps(t *testing.T) { |
| + routes := []registry.Route{ |
| + // In: the carlos-app@ unit shape the root adopt pass renders. |
| + {Host: "unit.example.com", Kind: "instance", App: "unitapp", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/unit.sock", Backing: registry.BackingUnit}, |
| + // In, and deduped: one app, two unit routes, one generation object. |
| + {Host: "unit2.example.com", Kind: "instance", App: "unitapp", Account: "bab", |
| + Status: "", Socket: "/run/carlos/unit2.sock", Backing: registry.BackingUnit}, |
| + // In under release.OpsAccount, exactly as ExecConfigApps resolves it. |
| + {Host: "legacy.example.com", Kind: "instance", App: "kass", Account: "", |
| + Status: "ready", Socket: "/run/carlos/legacy.sock", Backing: registry.BackingUnit}, |
| + // In BOTH sets: this app's unit route is here, its exec route is in |
| + // ExecConfigApps (asserted below). |
| + {Host: "mixed-unit.example.com", Kind: "instance", App: "mixed", Account: "zed", |
| + Status: "ready", Socket: "/run/carlos/mixed-unit.sock", Backing: registry.BackingUnit}, |
| + {Host: "mixed-exec.example.com", Kind: "instance", App: "mixed", Account: "zed", |
| + Status: "ready", Socket: "/run/carlos/mixed-exec.sock"}, |
| + // Out: exec-backed only — the agent delivers this one itself. |
| + {Host: "sock.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/sock.sock"}, |
| + // Out: platform routes are not app config. |
| + {Host: "plat.example.com", Kind: "instance", App: "platapp", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/plat.sock", Platform: true, |
| + Backing: registry.BackingUnit}, |
| + // Out: a draining route is on its way out; nothing to converge. |
| + {Host: "drain.example.com", Kind: "instance", App: "drainapp", Account: "bab", |
| + Status: "draining", Socket: "/run/carlos/drain.sock", Backing: registry.BackingUnit}, |
| + // Out: no app means no config manifest to key on. |
| + {Host: "noapp.example.com", Kind: "instance", App: "", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/noapp.sock", Backing: registry.BackingUnit}, |
| + // Out: not an instance route at all. |
| + {Host: "proxy.example.com", Kind: "proxy", App: "proxyapp", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/proxy.sock", Backing: registry.BackingUnit}, |
| + } |
| + got := UnitConfigApps(routes) |
| + want := [][2]string{ |
| + {"bab", "kass"}, |
| + {"bab", "unitapp"}, |
| + {"zed", "mixed"}, |
| + } |
| + if !slices.Equal(got, want) { |
| + t.Fatalf("UnitConfigApps = %v, want %v (distinct, sorted by account then app)", got, want) |
| + } |
| + if gotExec := ExecConfigApps(routes); !slices.Contains(gotExec, [2]string{"zed", "mixed"}) { |
| + t.Fatalf("ExecConfigApps = %v, want it to also hold {zed mixed} — an app with both route shapes is in both sets", gotExec) |
| + } |
| +} |
| + |
| +func TestUnitConfigAppsNoRoutes(t *testing.T) { |
| + if got := UnitConfigApps(nil); len(got) != 0 { |
| + t.Fatalf("UnitConfigApps(nil) = %v, want empty", got) |
| + } |
| +} |
diff --git a/internal/host/agent.go b/internal/host/agent.go| index bfef5ab14..41133a5ac 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -314,9 +314,10 @@ func bounce(svc *activator.Service) func(string) error { |
| // DirStore's is hex). |
| const genAbsent = "\x00absent" |
| -// genPollPass is one tick of configGenLoop, split out for tests: read every |
| -// exec-config app's generation etag and report whether ANY moved since the |
| -// cache was built. |
| +// genPollPass is one tick of configGenLoop, split out for tests: read the |
| +// generation etag of every app the box can converge — exec-config apps and |
| +// unit-backed apps alike — and report which HALVES moved since the cache |
| +// was built. |
| // |
| // Semantics, each of which is a deliberate choice: |
| // |
| @@ -333,9 +334,17 @@ const genAbsent = "\x00absent" |
| // created and overwritten, never deleted), but it's treated as a change |
| // too — one extra SyncExec pass is cheap, and pretending we didn't see |
| // it would be a lie about the only thing this function reports. |
| -// - ONE bool for the whole pass, however many apps moved: the caller runs |
| -// one SyncExec, which resolves every route itself. Per-app syncs would |
| -// multiply bucket reads to deliver exactly the same files. |
| +// - ONE bool PER HALF for the whole pass, however many apps moved: the |
| +// caller runs one SyncExec (which resolves every route itself) and asks |
| +// for at most one root adopt pass (which sweeps every unit route |
| +// itself). Per-app syncs would multiply bucket reads to deliver exactly |
| +// the same files, and per-app root requests would ask the same pass to |
| +// run several times over. |
| +// - The halves are independent: an app whose routes are all exec-backed |
| +// sets execChanged alone, an all-unit app sets unitChanged alone, and an |
| +// app holding both shapes at once sets both — off the SAME single read, |
| +// because its whole config lives in one generation object however its |
| +// routes are backed. |
| // - Apps that vanish from the route set lose their cache entries, so a |
| // re-added app is a first sight again rather than a comparison against |
| // an etag remembered from before it left (a route being re-added gets |
| @@ -349,11 +358,28 @@ const genAbsent = "\x00absent" |
| // The cache is mutated in place — it lives across ticks in the loop, which |
| // is the whole point (there is nowhere else to remember what "changed" |
| // means). |
| -func genPollPass(ctx context.Context, routes []registry.Route, store release.Store, cache map[[2]string]string) (bool, []error) { |
| - apps := appconfig.ExecConfigApps(routes) |
| +func genPollPass(ctx context.Context, routes []registry.Route, store release.Store, cache map[[2]string]string) (execChanged, unitChanged bool, readErrs []error) { |
| + execApps := appconfig.ExecConfigApps(routes) |
| + unitApps := appconfig.UnitConfigApps(routes) |
| + // The poll set is the UNION, read once per app: the two sets overlap |
| + // whenever an app has both route shapes, and its config is one object |
| + // either way. inExec/inUnit then say which half a movement belongs to. |
| + inExec := make(map[[2]string]bool, len(execApps)) |
| + for _, k := range execApps { |
| + inExec[k] = true |
| + } |
| + inUnit := make(map[[2]string]bool, len(unitApps)) |
| + for _, k := range unitApps { |
| + inUnit[k] = true |
| + } |
| + apps := make([][2]string, 0, len(execApps)+len(unitApps)) |
| + apps = append(apps, execApps...) |
| + for _, k := range unitApps { |
| + if !inExec[k] { |
| + apps = append(apps, k) |
| + } |
| + } |
| live := make(map[[2]string]bool, len(apps)) |
| - changed := false |
| - var readErrs []error |
| for _, k := range apps { |
| live[k] = true |
| _, etag, err := store.Get(ctx, appconfig.GenKey(k[0], k[1])) |
| @@ -367,7 +393,8 @@ func genPollPass(ctx context.Context, routes []registry.Route, store release.Sto |
| prev, seen := cache[k] |
| cache[k] = etag |
| if seen && prev != etag { |
| - changed = true |
| + execChanged = execChanged || inExec[k] |
| + unitChanged = unitChanged || inUnit[k] |
| } |
| } |
| for k := range cache { |
| @@ -375,27 +402,53 @@ func genPollPass(ctx context.Context, routes []registry.Route, store release.Sto |
| delete(cache, k) |
| } |
| } |
| - return changed, readErrs |
| + return execChanged, unitChanged, readErrs |
| +} |
| + |
| +// configSyncMarker is the file configGenLoop touches to ask the ROOT adopt |
| +// pass to run now: carlos-adopt.path watches it (PathChanged) and starts |
| +// carlos-adopt.service, which owns the unit half of config delivery |
| +// (/etc/carlos/apps + try-restart) that this unprivileged process cannot. |
| +// It lives in the data dir, deliberately outside the *.config.env namespace |
| +// reapConfigEnv sweeps — the reaper matches that suffix alone and must go |
| +// on ignoring this. |
| +const configSyncMarker = "config-sync-requested" |
| + |
| +// requestRootSync touches the marker. The content is a timestamp only so an |
| +// operator reading /data can see when the last request went out; systemd |
| +// watches the path, not the bytes. Errors are the caller's to log and |
| +// swallow: a request that cannot be written costs latency, not correctness, |
| +// because carlos-adopt.timer's hourly pass is the floor underneath it. |
| +func requestRootSync(dd string) error { |
| + return os.WriteFile(filepath.Join(dd, configSyncMarker), |
| + []byte(time.Now().UTC().Format(time.RFC3339)+"\n"), 0o644) |
| } |
| // configGenLoop is canaryAdoptLoop's shape applied to issue #44: a config |
| // set should converge like a canary adopts — seconds, not the next hour. |
| -// Every ~3s it reads one small object per exec-config app (the generation |
| -// object every config write touches) and, on any movement, runs ONE |
| -// SyncExec pass for the whole tick. Spurious movement costs a pass that |
| -// finds nothing to write: SyncExec bounces only routes whose rendered file |
| +// Every ~3s it reads one small object per config-carrying app (the |
| +// generation object every config write touches) and, on movement, delivers |
| +// the halves it can: ONE SyncExec pass for the whole tick for exec-backed |
| +// instances, and one touch of the root-sync marker for unit-backed ones. |
| +// Spurious movement costs a pass that finds nothing to write: SyncExec (and |
| +// the root pass's appconfig.Sync) bounces only routes whose rendered file |
| // actually changed, so a redundant touch can never cost a restart. |
| // |
| // Gated on CARLOS_SECRETS_KEY exactly as configSyncLoop is, for a reason |
| // beyond convention: without the key SyncExec cannot render an app's |
| // environment at all, so this loop would poll the bucket every 3 seconds |
| // forever to signal no one. openDeployStore failing is the same story with |
| -// nothing to read. |
| +// nothing to read. The gate covers the unit half too, which is an accepted |
| +// asymmetry rather than a considered one: a box whose agent lacks the key |
| +// gets no fast path for its unit apps either, even though the root pass |
| +// that would run holds its own key. In practice both units read the same |
| +// /etc/carlos/host.env, so the case is theoretical — and the hourly timer |
| +// floor stands regardless. |
| // |
| // Cardinality: unlike canaryAdoptLoop, which is bounded to the deliberately |
| -// rare canary/* routes, this watches EVERY exec-config app on the box. That |
| -// is accepted knowingly — one GET of the smallest object in the bucket per |
| -// app per 3s, against the store canaryAdoptLoop already hits every 2s. What |
| +// rare canary/* routes, this watches EVERY config-carrying app on the box. |
| +// That is accepted knowingly — one GET of the smallest object in the bucket |
| +// per app per 3s, against the store canaryAdoptLoop already hits every 2s. What |
| // keeps it bounded is that the pass reads generation objects only: no |
| // manifests, no pointers, no per-environment resolution. All of that |
| // happens inside the SyncExec a real change triggers. |
| @@ -407,8 +460,13 @@ func genPollPass(ctx context.Context, routes []registry.Route, store release.Sto |
| // convergence belongs to configSyncLoop's ~1min first pass, and a box |
| // coming up must not bounce every instance it has because it has never |
| // seen their etags before. |
| -// - No unit-backed routes. This process cannot restart units (see |
| -// RunAdopt); the root adopt path keeps that half, at its own cadence. |
| +// - No unit-backed rendering or restarting. This process still writes |
| +// nothing under /etc and restarts no unit (see RunAdopt): a unit app's |
| +// movement only touches <dataDir>/config-sync-requested, which |
| +// carlos-adopt.path relays into a root carlos-adopt.service run that |
| +// does both. The relay can drop an event that lands while that service |
| +// is mid-run — accepted: the next config write re-touches the marker, |
| +// and the timer floor delivers within the hour regardless. |
| // - No hourly floor replaced. configSyncLoop stays exactly as it was: it |
| // catches apps written by older CLIs (no generation object at all), |
| // ticks this loop missed, and the boot-retry gap of issue #27. |
| @@ -435,7 +493,7 @@ func configGenLoop(reg *registry.Registry, svc *activator.Service, dd string) { |
| // One cache for the life of the process: what "changed" means is |
| // exactly "differs from the last tick that could read it". |
| cache := map[[2]string]string{} |
| - log.Printf("agent: config generation loop armed (3s + jitter, exec-config apps)") |
| + log.Printf("agent: config generation loop armed (3s + jitter, exec-config and unit-backed apps)") |
| for { |
| time.Sleep(3*time.Second + rand.N(time.Second)) |
| routes, err := reg.List() |
| @@ -447,7 +505,7 @@ func configGenLoop(reg *registry.Registry, svc *activator.Service, dd string) { |
| log.Printf("agent: config generation poll: list routes: %v", err) |
| continue |
| } |
| - changed, readErrs := genPollPass(ctx, routes, store, cache) |
| + execChanged, unitChanged, readErrs := genPollPass(ctx, routes, store, cache) |
| if len(readErrs) > 0 { |
| // ONE line per pass, never one per app: a bucket having a bad |
| // minute would otherwise flood the journal at 20 lines a minute |
| @@ -455,7 +513,19 @@ func configGenLoop(reg *registry.Registry, svc *activator.Service, dd string) { |
| // the cause; they are the same cause in practice. |
| log.Printf("agent: config generation poll: %d apps unreadable: %v", len(readErrs), readErrs[0]) |
| } |
| - if !changed { |
| + if unitChanged { |
| + // Requested first and never fatal: the write is instant while |
| + // syncExecSerialized can block on a serialized pass, so the root |
| + // run gets to start alongside it rather than behind it. A failed |
| + // marker write costs those apps the wait until |
| + // carlos-adopt.timer's next hourly pass — never correctness. |
| + if err := requestRootSync(dd); err != nil { |
| + log.Printf("agent: request root config sync: %v", err) |
| + } else { |
| + log.Printf("agent: config changed for unit-backed app(s), requested root adopt pass") |
| + } |
| + } |
| + if !execChanged { |
| continue |
| } |
| if err := syncExecSerialized(ctx, reg, store, priv, dd, svc); err != nil { |
diff --git a/internal/host/agent_test.go b/internal/host/agent_test.go| index d76f88694..3bddab9a9 100644 |
| --- a/internal/host/agent_test.go |
| +++ b/internal/host/agent_test.go |
| @@ -31,6 +31,32 @@ func execRoute(host, acct, app string) registry.Route { |
| } |
| } |
| +// unitRoute is the carlos-app@ shape appconfig.UnitConfigApps accepts: the |
| +// same syncable instance, backed by a systemd unit whose env file only the |
| +// root adopt pass can write. The agent's half of a change to one of these |
| +// is the marker file, never a render. |
| +func unitRoute(host, acct, app string) registry.Route { |
| + rt := execRoute(host, acct, app) |
| + rt.Backing = registry.BackingUnit |
| + return rt |
| +} |
| + |
| +// countingStore records how many times each key was read, so a test can |
| +// pin that an app carrying BOTH route shapes still costs ONE generation |
| +// read per tick — the whole economy of the poll set being a union. |
| +type countingStore struct { |
| + release.Store |
| + gets map[string]int |
| +} |
| + |
| +func (c *countingStore) Get(ctx context.Context, key string) ([]byte, string, error) { |
| + if c.gets == nil { |
| + c.gets = map[string]int{} |
| + } |
| + c.gets[key]++ |
| + return c.Store.Get(ctx, key) |
| +} |
| + |
| // genEtag reads the app's generation etag straight from the store, so a |
| // test can assert the cache holds what the bucket actually reports rather |
| // than a value the test made up. |
| @@ -74,8 +100,8 @@ func TestGenPollPassFirstSightIsSilent(t *testing.T) { |
| routes := []registry.Route{execRoute("a.example.com", "bab", "eleven")} |
| cache := map[[2]string]string{} |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if changed { |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if execChanged || unitChanged { |
| t.Fatal("first sight of an app must not trigger a sync (boot would re-bounce the world)") |
| } |
| if len(errs) != 0 { |
| @@ -94,8 +120,8 @@ func TestGenPollPassUnchangedIsSilent(t *testing.T) { |
| cache := map[[2]string]string{} |
| genPollPass(ctx, routes, s, cache) |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if changed { |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if execChanged || unitChanged { |
| t.Fatal("an unmoved generation etag must not trigger a sync") |
| } |
| if len(errs) != 0 { |
| @@ -111,10 +137,13 @@ func TestGenPollPassTouchedGenerationTriggersSync(t *testing.T) { |
| genPollPass(ctx, routes, s, cache) |
| touch(t, ctx, s, "bab", "eleven") |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if !changed { |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if !execChanged { |
| t.Fatal("a moved generation etag must trigger a sync") |
| } |
| + if unitChanged { |
| + t.Fatal("an exec-only app must never ask for a root adopt pass") |
| + } |
| if len(errs) != 0 { |
| t.Fatalf("unexpected read errors: %v", errs) |
| } |
| @@ -122,7 +151,7 @@ func TestGenPollPassTouchedGenerationTriggersSync(t *testing.T) { |
| if got := cache[[2]string{"bab", "eleven"}]; got != want { |
| t.Fatalf("cache etag = %q, want the new %q (a stale cache would re-sync forever)", got, want) |
| } |
| - if changed, _ := genPollPass(ctx, routes, s, cache); changed { |
| + if execChanged, unitChanged, _ := genPollPass(ctx, routes, s, cache); execChanged || unitChanged { |
| t.Fatal("the pass after a change must be silent again") |
| } |
| } |
| @@ -135,8 +164,8 @@ func TestGenPollPassAbsentGenerationStaysSilent(t *testing.T) { |
| cache := map[[2]string]string{} |
| for i := range 2 { |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if changed { |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if execChanged || unitChanged { |
| t.Fatalf("pass %d: an app with no generation object must never trigger a sync", i) |
| } |
| if len(errs) != 0 { |
| @@ -155,8 +184,8 @@ func TestGenPollPassAbsentToPresentTriggersSync(t *testing.T) { |
| genPollPass(ctx, routes, s, cache) // records known-absent |
| touch(t, ctx, s, "bab", "eleven") // the app's first ever config write |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if !changed { |
| + execChanged, _, errs := genPollPass(ctx, routes, s, cache) |
| + if !execChanged { |
| t.Fatal("absent -> present is the first write to an old app: it must sync") |
| } |
| if len(errs) != 0 { |
| @@ -176,8 +205,8 @@ func TestGenPollPassOneChangedAppSyncsOnce(t *testing.T) { |
| genPollPass(ctx, routes, s, cache) |
| touch(t, ctx, s, "bab", "titogo") |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if !changed { |
| + execChanged, _, errs := genPollPass(ctx, routes, s, cache) |
| + if !execChanged { |
| t.Fatal("one moved app in the set must report the pass as changed") |
| } |
| if len(errs) != 0 { |
| @@ -189,7 +218,7 @@ func TestGenPollPassOneChangedAppSyncsOnce(t *testing.T) { |
| if got, want := cache[[2]string{"bab", "eleven"}], genEtag(t, ctx, s, "bab", "eleven"); got != want { |
| t.Fatalf("unchanged app's cache = %q, want %q", got, want) |
| } |
| - if changed, _ := genPollPass(ctx, routes, s, cache); changed { |
| + if execChanged, _, _ := genPollPass(ctx, routes, s, cache); execChanged { |
| t.Fatal("the next pass must be silent for both apps") |
| } |
| } |
| @@ -215,8 +244,8 @@ func TestGenPollPassDropsVanishedApps(t *testing.T) { |
| // again), not fire a sync off a remembered etag — a route being |
| // re-added already gets fresh config from the spawn path. |
| touch(t, ctx, s, "bab", "titogo") |
| - changed, errs := genPollPass(ctx, both, s, cache) |
| - if changed { |
| + execChanged, unitChanged, errs := genPollPass(ctx, both, s, cache) |
| + if execChanged || unitChanged { |
| t.Fatal("a re-added app is a first sight, not a change") |
| } |
| if len(errs) != 0 { |
| @@ -241,8 +270,8 @@ func TestGenPollPassReadErrorLeavesCacheAndKeepsGoing(t *testing.T) { |
| s := &failingKeyStore{Store: dir, failKey: appconfig.GenKey("bab", "eleven"), err: boom} |
| touch(t, ctx, dir, "bab", "titogo") |
| - changed, errs := genPollPass(ctx, routes, s, cache) |
| - if !changed { |
| + execChanged, _, errs := genPollPass(ctx, routes, s, cache) |
| + if !execChanged { |
| t.Fatal("one app failing must not stop the others being evaluated") |
| } |
| if len(errs) != 1 { |
| @@ -255,3 +284,105 @@ func TestGenPollPassReadErrorLeavesCacheAndKeepsGoing(t *testing.T) { |
| t.Fatalf("unreadable app's cache = %q, want it untouched at %q (a cleared entry would re-fire on recovery)", got, elevenEtag) |
| } |
| } |
| + |
| +func TestGenPollPassUnitFirstSightIsSilent(t *testing.T) { |
| + ctx, s := genKit(t) |
| + touch(t, ctx, s, "bab", "unitapp") |
| + routes := []registry.Route{unitRoute("u.example.com", "bab", "unitapp")} |
| + cache := map[[2]string]string{} |
| + |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if execChanged || unitChanged { |
| + t.Fatal("first sight of a unit-backed app must not request a root pass (boot would restart every tenant)") |
| + } |
| + if len(errs) != 0 { |
| + t.Fatalf("unexpected read errors: %v", errs) |
| + } |
| + want := genEtag(t, ctx, s, "bab", "unitapp") |
| + if got := cache[[2]string{"bab", "unitapp"}]; got != want { |
| + t.Fatalf("cache etag = %q, want the store's %q", got, want) |
| + } |
| +} |
| + |
| +func TestGenPollPassUnitMovementRequestsRootPassOnly(t *testing.T) { |
| + ctx, s := genKit(t) |
| + touch(t, ctx, s, "bab", "unitapp") |
| + touch(t, ctx, s, "bab", "eleven") |
| + routes := []registry.Route{ |
| + unitRoute("u.example.com", "bab", "unitapp"), |
| + execRoute("a.example.com", "bab", "eleven"), |
| + } |
| + cache := map[[2]string]string{} |
| + genPollPass(ctx, routes, s, cache) |
| + |
| + touch(t, ctx, s, "bab", "unitapp") |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if !unitChanged { |
| + t.Fatal("a moved generation for a unit-backed app must request the root adopt pass") |
| + } |
| + if execChanged { |
| + t.Fatal("the exec half must stay quiet: no exec-config app moved, so SyncExec would find nothing to write") |
| + } |
| + if len(errs) != 0 { |
| + t.Fatalf("unexpected read errors: %v", errs) |
| + } |
| + if execChanged, unitChanged, _ := genPollPass(ctx, routes, s, cache); execChanged || unitChanged { |
| + t.Fatal("the pass after a change must be silent again (a stale cache would re-touch the marker forever)") |
| + } |
| +} |
| + |
| +func TestGenPollPassBothShapesSetBothHalvesOnOneRead(t *testing.T) { |
| + ctx, dir := genKit(t) |
| + touch(t, ctx, dir, "zed", "mixed") |
| + // One app, two routes: the activator spawns one, systemd owns the other. |
| + // Its config lives in ONE generation object, so the union must read that |
| + // object once per tick and report both halves off it. |
| + routes := []registry.Route{ |
| + execRoute("mixed-exec.example.com", "zed", "mixed"), |
| + unitRoute("mixed-unit.example.com", "zed", "mixed"), |
| + } |
| + s := &countingStore{Store: dir} |
| + cache := map[[2]string]string{} |
| + genPollPass(ctx, routes, s, cache) |
| + |
| + touch(t, ctx, dir, "zed", "mixed") |
| + execChanged, unitChanged, errs := genPollPass(ctx, routes, s, cache) |
| + if !execChanged || !unitChanged { |
| + t.Fatalf("execChanged=%v unitChanged=%v, want both — the app has both route shapes", execChanged, unitChanged) |
| + } |
| + if len(errs) != 0 { |
| + t.Fatalf("unexpected read errors: %v", errs) |
| + } |
| + if got := s.gets[appconfig.GenKey("zed", "mixed")]; got != 2 { |
| + t.Fatalf("generation object read %d times over 2 ticks, want 2 — an app in both sets must cost one read per tick", got) |
| + } |
| +} |
| + |
| +func TestGenPollPassPrunesUnitAppsToo(t *testing.T) { |
| + ctx, s := genKit(t) |
| + touch(t, ctx, s, "bab", "unitapp") |
| + both := []registry.Route{ |
| + execRoute("a.example.com", "bab", "eleven"), |
| + unitRoute("u.example.com", "bab", "unitapp"), |
| + } |
| + one := both[:1] |
| + cache := map[[2]string]string{} |
| + genPollPass(ctx, both, s, cache) |
| + |
| + // The live set the cache is pruned against is the UNION: a unit app whose |
| + // last route left must lose its entry exactly as an exec app does, or its |
| + // remembered etag would fire a bogus root-pass request when the route |
| + // comes back (adoption already delivers fresh config on the way in). |
| + genPollPass(ctx, one, s, cache) |
| + if _, ok := cache[[2]string{"bab", "unitapp"}]; ok { |
| + t.Fatal("a unit-backed route that left the registry must leave no cache entry behind") |
| + } |
| + touch(t, ctx, s, "bab", "unitapp") |
| + execChanged, unitChanged, errs := genPollPass(ctx, both, s, cache) |
| + if execChanged || unitChanged { |
| + t.Fatal("a re-added unit-backed app is a first sight, not a change") |
| + } |
| + if len(errs) != 0 { |
| + t.Fatalf("unexpected read errors: %v", errs) |
| + } |
| +} |
diff --git a/internal/host/bootstrap.go b/internal/host/bootstrap.go| index 024587fd8..488d52ccb 100644 |
| --- a/internal/host/bootstrap.go |
| +++ b/internal/host/bootstrap.go |
| @@ -26,7 +26,7 @@ import ( |
| // no separate files to copy. Ported from Eleven's deploy/systemd (renamed, |
| // same hardening) and titogo's deploy/systemd (litestream + restore-verify). |
| // |
| -//go:embed units/*.service units/*.timer units/*.socket |
| +//go:embed units/*.service units/*.timer units/*.socket units/*.path |
| var unitsFS embed.FS |
| // serviceUser is the unprivileged account the edge unit runs as; bootstrap |
| @@ -42,25 +42,30 @@ const serviceUser = "carlos" |
| const tmpfilesConf = "d /run/carlos 0750 " + serviceUser + " " + serviceUser + " -\n" + |
| "d /run/carlos/apps 0750 " + serviceUser + " " + serviceUser + " -\n" |
| -// enabledUnits are what a live bootstrap turns on: the always-on front door |
| -// and the hourly adoption timer. The litestream + restore-verify units install |
| +// enabledUnits are what a live bootstrap turns on: the always-on front door, |
| +// the hourly adoption timer, and the path unit that lets the agent ask for an |
| +// adoption pass NOW (carlos-adopt.path — the unit-backed half of config fast |
| +// convergence, issue #44). The litestream + restore-verify units install |
| // everywhere but stay inert until an operator stages /data/litestream.env and |
| // generates /etc/litestream.yml (`carlos ops litestream`) — a box with no |
| // replication bucket is unaffected. |
| -var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer"} |
| +var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path"} |
| // systemctlPlan is the exact systemctl sequence a live bootstrap runs — and |
| // the same sequence a staging run (-root <dir>) PRINTS instead of running, so |
| // the two can never drift into describing different behaviour. |
| // |
| -// Timers get an explicit `start` after the `enable`, and that line is the |
| -// point of this function: `systemctl enable` writes a .wants symlink and |
| -// stops there, which arms the timer at the NEXT boot and not a moment sooner. |
| -// carlos-adopt.timer sat enabled-but-inactive on the flagship for weeks for |
| -// exactly this reason — symlink present, nothing rebooted, no release ever |
| -// adopted itself, every deploy silently needing a hand-run adoption pass. A |
| -// re-run is harmless: starting an already-running timer is a no-op, which |
| -// keeps bootstrap idempotent. |
| +// Timers and path units get an explicit `start` after the `enable`, and that |
| +// line is the point of this function: `systemctl enable` writes a .wants |
| +// symlink and stops there, which arms the unit at the NEXT boot and not a |
| +// moment sooner. carlos-adopt.timer sat enabled-but-inactive on the flagship |
| +// for weeks for exactly this reason — symlink present, nothing rebooted, no |
| +// release ever adopted itself, every deploy silently needing a hand-run |
| +// adoption pass. A path unit fails the same way and just as quietly: an |
| +// unstarted carlos-adopt.path watches nothing, so the agent's config-sync |
| +// marker lands in a file no one is reading. A re-run is harmless: starting an |
| +// already-running timer or path unit is a no-op, which keeps bootstrap |
| +// idempotent. |
| // |
| // Services are deliberately NOT started. carlos-edge needs its binary at |
| // /opt/carlos/carlos and /etc/carlos/host.env staged first — neither of which |
| @@ -73,14 +78,14 @@ func systemctlPlan(units []string) [][]string { |
| {"daemon-reload"}, |
| append([]string{"enable"}, units...), |
| } |
| - var timers []string |
| + var armed []string |
| for _, u := range units { |
| - if strings.HasSuffix(u, ".timer") { |
| - timers = append(timers, u) |
| + if strings.HasSuffix(u, ".timer") || strings.HasSuffix(u, ".path") { |
| + armed = append(armed, u) |
| } |
| } |
| - if len(timers) > 0 { |
| - plan = append(plan, append([]string{"start"}, timers...)) |
| + if len(armed) > 0 { |
| + plan = append(plan, append([]string{"start"}, armed...)) |
| } |
| return plan |
| } |
| @@ -194,7 +199,7 @@ func bootstrap(root string) error { |
| return err |
| } |
| } |
| - fmt.Printf("bootstrap: units installed, %s enabled (timers armed, not just symlinked)\n", strings.Join(enabledUnits, " + ")) |
| + fmt.Printf("bootstrap: units installed, %s enabled (timers and path units armed, not just symlinked)\n", strings.Join(enabledUnits, " + ")) |
| fmt.Println(" next: copy the carlos binary to /opt/carlos/carlos, stage /etc/carlos/host.env, then: systemctl start carlos-edge") |
| return nil |
| } |
diff --git a/internal/host/host_test.go b/internal/host/host_test.go| index a94cc4d65..c15f0cb29 100644 |
| --- a/internal/host/host_test.go |
| +++ b/internal/host/host_test.go |
| @@ -24,6 +24,7 @@ import ( |
| // must be added here (and a removed one taken out) or TestUnitsEmbedComplete |
| // fails, so the embed directive and the fleet's unit set can't drift apart. |
| var expectedUnits = []string{ |
| + "carlos-adopt.path", |
| "carlos-adopt.service", |
| "carlos-adopt.timer", |
| "carlos-app@.service", |
| @@ -401,6 +402,11 @@ func TestIsCanaryRoute(t *testing.T) { |
| // Every timer bootstrap enables must therefore also be started, and no |
| // SERVICE may be (carlos-edge needs its binary and host.env staged first — |
| // bootstrap's own closing line tells the operator to start it). |
| +// |
| +// PATH units are the same trap with the same fix: an enabled-but-unstarted |
| +// carlos-adopt.path watches nothing until the next reboot, so the agent's |
| +// config-sync marker would be written to a file no one reads and unit-backed |
| +// apps would silently keep only the hourly floor. |
| func TestSystemctlPlanArmsTimers(t *testing.T) { |
| plan := systemctlPlan(enabledUnits) |
| if len(plan) == 0 || plan[0][0] != "daemon-reload" { |
| @@ -423,8 +429,8 @@ func TestSystemctlPlanArmsTimers(t *testing.T) { |
| if !enabled[u] { |
| t.Errorf("%s: enabled by the plan? no — bootstrap must still enable everything it did before", u) |
| } |
| - if strings.HasSuffix(u, ".timer") && !started[u] { |
| - t.Errorf("%s: enabled but never started — that is exactly the bug (enable only symlinks; the timer arms at the next boot)", u) |
| + if (strings.HasSuffix(u, ".timer") || strings.HasSuffix(u, ".path")) && !started[u] { |
| + t.Errorf("%s: enabled but never started — that is exactly the bug (enable only symlinks; the unit arms at the next boot)", u) |
| } |
| if strings.HasSuffix(u, ".service") && started[u] { |
| t.Errorf("%s: bootstrap must not start a service — the binary and /etc/carlos/host.env may not be staged yet", u) |
diff --git a/internal/host/units/carlos-adopt.path b/internal/host/units/carlos-adopt.path| new file mode 100644 |
| index 000000000..823eabfbd |
| --- /dev/null |
| +++ b/internal/host/units/carlos-adopt.path |
| @@ -0,0 +1,16 @@ |
| +# Fast config convergence for unit-backed apps (issue #44). The agent's |
| +# configGenLoop touches this file when a config write moves the generation |
| +# object of an app with carlos-app@ routes on this box: the agent renders |
| +# exec-backed instances itself, but it cannot write /etc/carlos/apps or |
| +# restart units, so it asks the root pass to do both. /data is hardcoded |
| +# like every path in these unit files — a box with a nonstandard data dir |
| +# simply never fast-triggers and keeps carlos-adopt.timer's hourly floor. |
| +[Unit] |
| +Description=CARLOS root adopt pass on agent request (config fast convergence) |
| + |
| +[Path] |
| +PathChanged=/data/config-sync-requested |
| +Unit=carlos-adopt.service |
| + |
| +[Install] |
| +WantedBy=multi-user.target |