carlos env/secrets: exec-backend delivery — the instance-fleet shape (#16)
appconfig.SyncExec materializes per-app config into a sync-owned /data/<host>.config.env for activator-spawned routes, environments-aware ((acct,app,env) grouping, LoadMerged for bound routes, error-never-means-absent, mass-removal guard). ExecBackend.Start layers it under the instance's own env file (instance wins; CARLOS_ADMIN_TOKEN structurally unclobberable, pinned at child-process level). Service.Bounce restarts running instances on config change; configSyncLoop in carlos agent delivers hourly, inert without CARLOS_SECRETS_KEY. Plus the issue-#14 adopt advisory and Backing round-trip pin.
15 files changed,
+2443
−20
README.md+10 −3appconfig_cli.go+3 −2docs/superpowers/plans/2026-08-02-exec-config-delivery.md+945 −0docs/superpowers/specs/2026-08-02-app-config-secrets-design.md+3 −0docs/superpowers/specs/2026-08-02-exec-config-delivery-design.md+200 −0internal/activator/activator.go+32 −7internal/activator/activator_test.go+158 −0internal/activator/backend_exec.go+26 −2internal/appconfig/sync.go+3 −1internal/appconfig/syncexec.go+349 −0internal/appconfig/syncexec_test.go+569 −0internal/host/adopt_advice.go+43 −0internal/host/adopt_advice_test.go+25 −0internal/host/agent.go+60 −5internal/registry/registry_test.go+17 −0
diff --git a/README.md b/README.md| index f7dbddf35..071fc93a4 100644 |
| --- a/README.md |
| +++ b/README.md |
| @@ -177,9 +177,16 @@ config sync is inert — the same convention as every other optional loop. |
| Operational notes: |
| -- Config sync reaches **unit-backed routes only** (`Backing: "unit"`). |
| - Exec-backed routes read `/data/<host>.env` and keep their hand-staged |
| - files untouched; the adopt log says so per pass. |
| +- Delivery is split by backing: unit-backed routes (`Backing: "unit"`) |
| + materialize to `/etc/carlos/apps/<host>.env` via the hourly root |
| + `carlos adopt`; exec-backed instance routes (socket-upstream) |
| + materialize to `/data/<host>.config.env` via the agent's in-process |
| + hourly loop, which also bounces a running instance when its config |
| + changes. At spawn the instance's own `/data/<host>.env` is layered on |
| + top — it wins every conflict, so `CARLOS_ADMIN_TOKEN` and box-local |
| + overrides are never clobbered from the bucket. Addr-upstream |
| + (proxy-only) routes get no delivery — the process behind them is |
| + externally managed. |
| - Migrate an app's config into the bucket *before* enabling |
| `CARLOS_SECRETS_KEY` on its hosts: once a manifest object exists for an |
| app, the bucket is authoritative for its env files (fully unset ⇒ |
diff --git a/appconfig_cli.go b/appconfig_cli.go| index 4ee80d1b5..a00157af2 100644 |
| --- a/appconfig_cli.go |
| +++ b/appconfig_cli.go |
| @@ -379,6 +379,7 @@ func runSecretsGenkey(args []string) { |
| fmt.Printf("CARLOS_SECRETS_KEY=%s\nCARLOS_SECRETS_PUBKEY=%s\n", priv, pub) |
| fmt.Fprintln(os.Stderr, "store the KEY in your secret manager and in every host's host.env; "+ |
| "the PUBKEY can be held widely — it only encrypts") |
| - fmt.Fprintln(os.Stderr, "note: config sync currently reaches unit-backed routes (carlos-app@) only; "+ |
| - "exec-backed routes keep their hand-staged env files") |
| + fmt.Fprintln(os.Stderr, "note: config sync reaches unit-backed routes (carlos-app@) via the root adopt pass, "+ |
| + "and exec-backed instance routes via the agent's in-process loop (materialized to /data/<host>.config.env; "+ |
| + "the instance's own /data/<host>.env always wins conflicts)") |
| } |
diff --git a/docs/superpowers/plans/2026-08-02-exec-config-delivery.md b/docs/superpowers/plans/2026-08-02-exec-config-delivery.md| new file mode 100644 |
| index 000000000..1c99a2088 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-02-exec-config-delivery.md |
| @@ -0,0 +1,945 @@ |
| +# Exec-Backend Config Delivery Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Deliver `carlos env`/`carlos secrets` config to exec-backed (activator-spawned) routes via a sync-owned `/data/<host>.config.env`, composed under the instance's own env file at spawn, with bounce-on-change. |
| + |
| +**Architecture:** A new `appconfig.SyncExec` sweep (mirroring the shipped unit-path `Sync`) materializes manifests into `<DataDir>/<host>.config.env` for socket-upstream exec routes; the activator's `ExecBackend.Start` layers that file UNDER the instance's `<host>.env` (instance wins every conflict); a new in-process `configSyncLoop` in `carlos agent` runs the sweep hourly and bounces running instances via a new `Service.Bounce`. Spec: `docs/superpowers/specs/2026-08-02-exec-config-delivery-design.md`. |
| + |
| +**Tech Stack:** Go stdlib only. Existing `internal/appconfig` core (manifest/crypto/store) is reused untouched. |
| + |
| +## Global Constraints |
| + |
| +- The instance's own `<DataDir>/<host>.env` is NEVER read, written, merged into, or reaped by sync — it stays activator/operator property (it carries `CARLOS_ADMIN_TOKEN`). |
| +- Config file namespace is exactly `<host>.config.env` (+ `.tmp` during atomic write), `0600`, tmp+rename. |
| +- Manifest object absent → the `.config.env` file is REMOVED if present (sync owns the namespace outright; this is the opposite of the unit path's leave-alone rule, and deliberate — spec §2). |
| +- An empty route list never reaps and never removes anything (`registry.Open` auto-creates empty DBs — same guard as unit-path `Sync`). |
| +- In-scope routes: `Kind == "instance"`, `App != ""`, `!Platform`, `Status` `""` or `"ready"`, `Backing != registry.BackingUnit`, `Socket != ""`. Addr-upstream and unit routes are untouched. |
| +- Bounce is best-effort: log and continue, never abort the sweep; never START an instance to deliver config. |
| +- Loop is inert without `CARLOS_SECRETS_KEY` (one log line), matching every optional loop. |
| +- Build/test discipline: `gofmt` clean; sandbox needs `GOCACHE=/tmp/claude-1001/go-cache-pr9` (or run tests with sandbox disabled); run the focused test while iterating, full `go build ./... && go vet ./... && go test ./...` once before each commit. |
| +- Commit messages end with `Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>`. |
| + |
| +## File Structure |
| + |
| +- Create: `internal/appconfig/syncexec.go` — the exec-route sweep (Task 1) |
| +- Create: `internal/appconfig/syncexec_test.go` (Task 1) |
| +- Modify: `internal/activator/backend_exec.go` — spawn merge (Task 2) |
| +- Modify: `internal/activator/activator_test.go` — merge tests (Task 2) |
| +- Modify: `internal/activator/activator.go` — `Service.Bounce` (Task 3) |
| +- Modify: `internal/host/agent.go` — `configSyncLoop` + wiring (Task 4) |
| +- Modify: `internal/host/adopt_advice.go` (create) + `internal/host/agent.go` — #14 advisory (Task 5) |
| +- Modify: `internal/registry/registry_test.go`, `appconfig_cli.go`, `README.md`, `docs/superpowers/specs/2026-08-02-app-config-secrets-design.md` (Task 5) |
| + |
| +--- |
| + |
| +### Task 1: `appconfig.SyncExec` — exec-route materialization sweep |
| + |
| +**Files:** |
| +- Create: `internal/appconfig/syncexec.go` |
| +- Create: `internal/appconfig/syncexec_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes (all already in package `appconfig` on main): `getIfExists(ctx, s, acct, app) (*Manifest, bool, error)`, `RenderEnv(m, priv) ([]byte, error)`, `release.RouteAccount(rt)`, `registry.BackingUnit`. |
| +- Produces: `func SyncExec(ctx context.Context, reg *registry.Registry, s release.Store, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error) error` — Task 4 calls exactly this. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Create `internal/appconfig/syncexec_test.go`. The package already has `testRegistry(t)` (sync_test.go) and key helpers; reuse them. |
| + |
| +```go |
| +package appconfig |
| + |
| +import ( |
| + "context" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// execKit is the shared setup: store, registry, dataDir, keypair. |
| +func execKit(t *testing.T) (ctx context.Context, s release.Store, reg *registry.Registry, dataDir, priv, pub string) { |
| + t.Helper() |
| + s = release.NewDirStore(t.TempDir()) |
| + ctx = context.Background() |
| + reg = testRegistry(t) |
| + dataDir = t.TempDir() |
| + var err error |
| + priv, pub, err = GenerateKey() |
| + if err != nil { |
| + t.Fatalf("GenerateKey: %v", err) |
| + } |
| + return |
| +} |
| + |
| +func TestSyncExecWritesConfigFileAndBouncesOnChange(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, err := ParsePrivateKey(priv, "test") |
| + if err != nil { |
| + t.Fatalf("ParsePrivateKey: %v", err) |
| + } |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"LOG_LEVEL": "debug"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + var bounced []string |
| + bounce := func(h string) error { bounced = append(bounced, h); return nil } |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + got, err := os.ReadFile(filepath.Join(dd, "inst.example.com.config.env")) |
| + if err != nil { |
| + t.Fatalf("read config env: %v", err) |
| + } |
| + if string(got) != "LOG_LEVEL=debug\n" { |
| + t.Fatalf("config env = %q, want LOG_LEVEL=debug", got) |
| + } |
| + if fi, _ := os.Stat(filepath.Join(dd, "inst.example.com.config.env")); fi.Mode().Perm() != 0o600 { |
| + t.Fatalf("config env mode = %v, want 0600", fi.Mode().Perm()) |
| + } |
| + if len(bounced) != 1 || bounced[0] != "inst.example.com" { |
| + t.Fatalf("bounced = %v, want exactly [inst.example.com]", bounced) |
| + } |
| + // Second pass, nothing changed: no bounce. |
| + bounced = nil |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec (second): %v", err) |
| + } |
| + if len(bounced) != 0 { |
| + t.Fatalf("unchanged pass bounced %v, want none", bounced) |
| + } |
| +} |
| + |
| +func TestSyncExecRemovesFileWhenManifestDeleted(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + // A config file sync wrote earlier, whose manifest object is now gone: |
| + // only sync writes this namespace, so it must be removed — and the |
| + // running instance bounced, or it keeps stale config until next wake. |
| + path := filepath.Join(dd, "inst.example.com.config.env") |
| + if err := os.WriteFile(path, []byte("OLD=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(path); !os.IsNotExist(err) { |
| + t.Fatalf("config file survived manifest deletion, stat err = %v", err) |
| + } |
| + if len(bounced) != 1 { |
| + t.Fatalf("bounced = %v, want the stale-config instance bounced once", bounced) |
| + } |
| +} |
| + |
| +func TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"A": "1"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + // The instance's own env file must never be read, changed, or reaped. |
| + own := filepath.Join(dd, "inst.example.com.env") |
| + if err := os.WriteFile(own, []byte("CARLOS_ADMIN_TOKEN=tok\n"), 0o600); err != nil { |
| + t.Fatalf("stage own env: %v", err) |
| + } |
| + routes := []registry.Route{ |
| + {Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/inst.sock"}, |
| + {Host: "unit.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/u.sock", Backing: registry.BackingUnit}, |
| + {Host: "proxy.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Addr: "127.0.0.1:8444"}, |
| + {Host: "alias.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/a.sock", Platform: true}, |
| + {Host: "pending.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "provisioning", Socket: "/run/carlos/p.sock"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if got, _ := os.ReadFile(own); string(got) != "CARLOS_ADMIN_TOKEN=tok\n" { |
| + t.Fatalf("instance env file was touched: %q", got) |
| + } |
| + if _, err := os.Stat(filepath.Join(dd, "inst.example.com.config.env")); err != nil { |
| + t.Fatalf("in-scope route got no config file: %v", err) |
| + } |
| + for _, h := range []string{"unit.example.com", "proxy.example.com", "alias.example.com", "pending.example.com"} { |
| + if _, err := os.Stat(filepath.Join(dd, h+".config.env")); !os.IsNotExist(err) { |
| + t.Fatalf("out-of-scope route %s got a config file, stat err = %v", h, err) |
| + } |
| + } |
| +} |
| + |
| +func TestSyncExecReapsOnlyItsOwnNamespace(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "live.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/l.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + files := map[string]bool{ // name -> should survive |
| + "live.example.com.config.env": true, // has a route |
| + "gone.example.com.config.env": false, // no route: reaped |
| + "gone.example.com.config.env.tmp": false, // stale tmp: reaped |
| + "gone.example.com.env": true, // NOT ours: instance file, never reaped |
| + "gone.example.com-sidecar.env": true, // NOT ours: sidecar file, never reaped |
| + } |
| + for name := range files { |
| + if err := os.WriteFile(filepath.Join(dd, name), []byte("X=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage %s: %v", name, err) |
| + } |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + for name, keep := range files { |
| + _, err := os.Stat(filepath.Join(dd, name)) |
| + if keep && err != nil { |
| + t.Fatalf("%s: want kept, got %v", name, err) |
| + } |
| + if !keep && !os.IsNotExist(err) { |
| + t.Fatalf("%s: want reaped, stat err = %v", name, err) |
| + } |
| + } |
| +} |
| + |
| +func TestSyncExecEmptyRegistryRemovesNothing(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + path := filepath.Join(dd, "orphan.example.com.config.env") |
| + if err := os.WriteFile(path, []byte("X=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(path); err != nil { |
| + t.Fatalf("empty registry removed a config file: %v", err) |
| + } |
| +} |
| + |
| +func TestSyncExecRefusesNonBareHostname(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + _ = reg // routes injected directly via the per-route helper below |
| + m := &Manifest{Schema: 1, App: "eleven", Vars: map[string]Var{"A": {Kind: "env", Value: "1"}}} |
| + for _, host := range []string{"../escape", "a/b", "", ".", ".."} { |
| + syncExecHost(registry.Route{Host: host}, m, pk, dd, func(string) error { return nil }) |
| + } |
| + ents, err := os.ReadDir(dd) |
| + if err != nil { |
| + t.Fatalf("read dataDir: %v", err) |
| + } |
| + if len(ents) != 0 { |
| + t.Fatalf("dataDir = %v, want nothing written for non-bare hostnames", ents) |
| + } |
| + _ = ctx |
| + _ = s |
| +} |
| +``` |
| + |
| +`execKit` is the only helper — do not invent further indirection. |
| + |
| +- [ ] **Step 2: Run tests to verify they fail** |
| + |
| +Run: `go test ./internal/appconfig/ -run TestSyncExec -v` |
| +Expected: FAIL — `undefined: SyncExec`, `undefined: syncExecHost`. |
| + |
| +- [ ] **Step 3: Implement `internal/appconfig/syncexec.go`** |
| + |
| +```go |
| +// SyncExec is the exec-backend half of config delivery: for every |
| +// activator-spawned route (socket-upstream, non-unit), materialize the |
| +// app's manifest into <dataDir>/<host>.config.env — the file |
| +// ExecBackend.Start layers UNDER the instance's own <host>.env at spawn. |
| +// Sync owns this namespace outright (nothing else writes *.config.env), |
| +// which is why an absent manifest REMOVES the file here, where the |
| +// unit-path Sync leaves /etc/carlos/apps files alone: there is no |
| +// hand-staged-file migration hazard by construction. See spec |
| +// 2026-08-02-exec-config-delivery-design.md. |
| +package appconfig |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "crypto/ecdh" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// configEnvSuffix names sync's file namespace under dataDir. Everything |
| +// with this suffix belongs to sync; nothing else in dataDir does. |
| +const configEnvSuffix = ".config.env" |
| + |
| +// SyncExec fetches config once per distinct (Account, App) among the |
| +// box's exec-syncable routes and converges every such route's |
| +// <dataDir>/<host>.config.env, calling bounce(host) whenever a file's |
| +// content changed (including removal) so a running instance respawns |
| +// with fresh env on its next wake. Die-free per app, like Sync. |
| +func SyncExec(ctx context.Context, reg *registry.Registry, s release.Store, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error) error { |
| + routes, err := reg.List() |
| + if err != nil { |
| + return err |
| + } |
| + type groupKey struct{ acct, app string } |
| + manifests := map[groupKey]*Manifest{} |
| + found := map[groupKey]bool{} |
| + for _, rt := range routes { |
| + if !execSyncable(rt) { |
| + continue |
| + } |
| + k := groupKey{release.RouteAccount(rt), rt.App} |
| + if _, done := found[k]; done { |
| + continue |
| + } |
| + m, ok, err := getIfExists(ctx, s, k.acct, k.app) |
| + if err != nil { |
| + fmt.Printf(" %s/%s: config fetch failed: %v\n", k.acct, k.app, err) |
| + // A fetch failure is NOT "absent": leave this app's files alone |
| + // this pass rather than removing config on a bucket blip. |
| + continue |
| + } |
| + found[k] = true |
| + if ok { |
| + manifests[k] = m |
| + } |
| + } |
| + for _, rt := range routes { |
| + if !execSyncable(rt) { |
| + continue |
| + } |
| + k := groupKey{release.RouteAccount(rt), rt.App} |
| + if !found[k] { |
| + continue // fetch failed this pass; converge next time |
| + } |
| + m := manifests[k] // nil = manifest object absent -> remove |
| + syncExecHost(rt, m, priv, dataDir, bounce) |
| + } |
| + // An empty route list is never authority to delete anything — |
| + // registry.Open silently creates an empty database on a wrong path or |
| + // an unmounted /data (same guard, same reason, as the unit-path Sync). |
| + if len(routes) == 0 { |
| + return nil |
| + } |
| + reapConfigEnv(routes, dataDir) |
| + return nil |
| +} |
| + |
| +// execSyncable: lifecycle-managed instances the ACTIVATOR spawns — and |
| +// therefore whose environment this sweep can actually deliver. Mirrors |
| +// the unit path's syncable() exclusions, then narrows to exec routes |
| +// with a socket upstream: addr-upstream routes are proxy-only (the |
| +// process behind them is externally managed) and unit routes belong to |
| +// the root adopt path. |
| +func execSyncable(rt registry.Route) bool { |
| + return rt.Kind == "instance" && rt.App != "" && !rt.Platform && |
| + (rt.Status == "" || rt.Status == "ready") && |
| + rt.Backing != registry.BackingUnit && rt.Socket != "" |
| +} |
| + |
| +// syncExecHost converges one route's config file. m == nil means the |
| +// manifest object is absent from the bucket: remove the file (sync owns |
| +// the namespace; a file with no manifest is a leftover, and leaving it |
| +// would keep dead config flowing into spawns). |
| +func syncExecHost(rt registry.Route, m *Manifest, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error) { |
| + if rt.Host == "" || rt.Host == "." || rt.Host == ".." || rt.Host != filepath.Base(rt.Host) { |
| + fmt.Printf(" %q: not a bare hostname, refusing to derive a config file path from it\n", rt.Host) |
| + return |
| + } |
| + path := filepath.Join(dataDir, rt.Host+configEnvSuffix) |
| + if m == nil { |
| + if err := os.Remove(path); err != nil { |
| + if !os.IsNotExist(err) { |
| + fmt.Printf(" %s: remove config for deleted manifest: %v\n", rt.Host, err) |
| + } |
| + return |
| + } |
| + fmt.Printf(" %s: manifest deleted, removed its config file\n", rt.Host) |
| + bounceLogged(rt.Host, bounce) |
| + return |
| + } |
| + cur, err := os.ReadFile(path) |
| + existed := err == nil |
| + if err != nil && !os.IsNotExist(err) { |
| + fmt.Printf(" %s: read %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + rendered, err := RenderEnv(m, priv) |
| + if err != nil { |
| + fmt.Printf(" %s: render config: %v\n", rt.Host, err) |
| + return |
| + } |
| + if existed && bytes.Equal(cur, rendered) { |
| + return // up to date — stay quiet on the periodic tick |
| + } |
| + tmp := path + ".tmp" |
| + if err := os.WriteFile(tmp, rendered, 0o600); err != nil { |
| + fmt.Printf(" %s: write %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + if err := os.Rename(tmp, path); err != nil { |
| + os.Remove(tmp) // don't leave rendered secrets behind |
| + fmt.Printf(" %s: rename %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + fmt.Printf(" %s: config updated\n", rt.Host) |
| + bounceLogged(rt.Host, bounce) |
| +} |
| + |
| +// bounceLogged: bounce is best-effort — a failure is logged, never |
| +// propagated; the file on disk is already correct and the next natural |
| +// wake picks it up. |
| +func bounceLogged(host string, bounce func(host string) error) { |
| + if bounce == nil { |
| + return |
| + } |
| + if err := bounce(host); err != nil { |
| + fmt.Printf(" %s: bounce after config change: %v\n", host, err) |
| + } |
| +} |
| + |
| +// reapConfigEnv deletes *.config.env (and stale *.config.env.tmp) for |
| +// hosts with no route row at all. It matches ONLY sync's own suffix: |
| +// <host>.env and <host>-sidecar.env are other owners' files and are |
| +// never candidates, whatever their host's route state. |
| +func reapConfigEnv(routes []registry.Route, dataDir string) { |
| + ents, err := os.ReadDir(dataDir) |
| + if err != nil { |
| + return |
| + } |
| + live := make(map[string]bool, len(routes)) |
| + for _, rt := range routes { |
| + live[rt.Host] = true |
| + } |
| + for _, e := range ents { |
| + if e.IsDir() { |
| + continue |
| + } |
| + if strings.HasSuffix(e.Name(), configEnvSuffix+".tmp") { |
| + // A crashed pass's leftover holds decrypted secrets and is |
| + // stale by construction once reap runs. |
| + if err := os.Remove(filepath.Join(dataDir, e.Name())); err != nil { |
| + fmt.Printf(" %s: remove stale config temp file: %v\n", e.Name(), err) |
| + } |
| + continue |
| + } |
| + if !strings.HasSuffix(e.Name(), configEnvSuffix) { |
| + continue |
| + } |
| + host := strings.TrimSuffix(e.Name(), configEnvSuffix) |
| + if live[host] { |
| + continue |
| + } |
| + if err := os.Remove(filepath.Join(dataDir, e.Name())); err != nil { |
| + fmt.Printf(" %s: remove orphaned config file: %v\n", host, err) |
| + continue |
| + } |
| + fmt.Printf(" %s: no route on this box, removed its config file\n", host) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests, then the package** |
| + |
| +Run: `go test ./internal/appconfig/ -run TestSyncExec -v` → PASS, then `go test ./internal/appconfig/` → all pass (the unit-path tests must be untouched). |
| + |
| +- [ ] **Step 5: Full suite + commit** |
| + |
| +```bash |
| +go build ./... && go vet ./... && go test ./... |
| +git add internal/appconfig/syncexec.go internal/appconfig/syncexec_test.go |
| +git commit -m "appconfig: SyncExec — exec-route config materialization into /data/<host>.config.env" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: Spawn-time merge in `ExecBackend.Start` |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go` (the `Start` method, ~line 218; `childEnv`/`withVar` at ~311) |
| +- Modify: `internal/activator/activator_test.go` (append tests near the existing `childEnv` tests, ~line 1812) |
| + |
| +**Interfaces:** |
| +- Consumes: `appconfig.SyncExec`'s file contract — `<DataDir>/<host>.config.env`, optional, `readEnvFile`-parseable. |
| +- Produces: `func mergeEnv(base, overlay map[string]string) map[string]string` (package-private) — overlay wins conflicts. Task 1's file feeds `base`; today's `<host>.env` feeds `overlay`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `internal/activator/activator_test.go`: |
| + |
| +```go |
| +// mergeEnv layers the sync-owned <host>.config.env UNDER the instance's |
| +// own <host>.env: the instance file wins every conflict, which is what |
| +// makes CARLOS_ADMIN_TOKEN structurally unclobberable from the bucket. |
| +func TestMergeEnvInstanceFileWins(t *testing.T) { |
| + base := map[string]string{"A": "bucket", "B": "bucket-only", "CARLOS_ADMIN_TOKEN": "forged"} |
| + overlay := map[string]string{"A": "instance", "CARLOS_ADMIN_TOKEN": "real"} |
| + got := mergeEnv(base, overlay) |
| + if got["A"] != "instance" || got["B"] != "bucket-only" || got["CARLOS_ADMIN_TOKEN"] != "real" { |
| + t.Fatalf("mergeEnv = %v, want instance values winning and bucket-only surviving", got) |
| + } |
| + // Inputs must not be mutated (withVar's same discipline). |
| + if base["A"] != "bucket" || len(overlay) != 2 { |
| + t.Fatal("mergeEnv mutated an input map") |
| + } |
| +} |
| + |
| +func TestMergeEnvNilInputs(t *testing.T) { |
| + if got := mergeEnv(nil, map[string]string{"A": "1"}); got["A"] != "1" { |
| + t.Fatalf("nil base: got %v", got) |
| + } |
| + if got := mergeEnv(map[string]string{"A": "1"}, nil); got["A"] != "1" { |
| + t.Fatalf("nil overlay: got %v", got) |
| + } |
| + if got := mergeEnv(nil, nil); len(got) != 0 { |
| + t.Fatalf("nil/nil: got %v", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/activator/ -run TestMergeEnv -v` |
| +Expected: FAIL — `undefined: mergeEnv`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +In `backend_exec.go`, add next to `withVar`: |
| + |
| +```go |
| +// mergeEnv composes the child's env-file layers: base (the sync-owned |
| +// <host>.config.env, bucket-derived) under overlay (the instance's own |
| +// <host>.env). The overlay wins every conflict — CARLOS_ADMIN_TOKEN |
| +// lives only in the instance file and must never be clobberable from |
| +// the bucket, and a box-local override deliberately beats central |
| +// config. Returns a fresh map; inputs are never mutated. |
| +func mergeEnv(base, overlay map[string]string) map[string]string { |
| + out := make(map[string]string, len(base)+len(overlay)) |
| + for k, v := range base { |
| + out[k] = v |
| + } |
| + for k, v := range overlay { |
| + out[k] = v |
| + } |
| + return out |
| +} |
| +``` |
| + |
| +In `Start` (the block reading the env file — currently: |
| + |
| +```go |
| + // The env file is optional, like the unit's "-" prefix. |
| + fileEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".env")) |
| + if err != nil && !os.IsNotExist(err) { |
| + return nil, "", fmt.Errorf("read env file: %w", err) |
| + } |
| +``` |
| + |
| +) change to: |
| + |
| +```go |
| + // Two env layers, both optional like the unit's "-" prefix: the |
| + // sync-owned bucket config underneath, the instance's own file on |
| + // top (it wins conflicts — see mergeEnv). |
| + cfgEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".config.env")) |
| + if err != nil && !os.IsNotExist(err) { |
| + return nil, "", fmt.Errorf("read config env file: %w", err) |
| + } |
| + ownEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".env")) |
| + if err != nil && !os.IsNotExist(err) { |
| + return nil, "", fmt.Errorf("read env file: %w", err) |
| + } |
| + fileEnv := mergeEnv(cfgEnv, ownEnv) |
| +``` |
| + |
| +Leave everything downstream (`childVars := fileEnv`, `withVar`, `childEnv`) exactly as is. Do NOT touch `StartSidecar`, `NextDue`, or any other `readEnvFile` call site — the token paths keep reading the instance file only. |
| + |
| +- [ ] **Step 4: Run tests** |
| + |
| +Run: `go test ./internal/activator/ -run 'TestMergeEnv|TestChildEnv' -v` → PASS, then `go test ./internal/activator/` → all pass. |
| + |
| +- [ ] **Step 5: Full suite + commit** |
| + |
| +```bash |
| +go build ./... && go vet ./... && go test ./... |
| +git add internal/activator/backend_exec.go internal/activator/activator_test.go |
| +git commit -m "activator: layer sync-owned <host>.config.env under the instance env at spawn" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: `Service.Bounce` |
| + |
| +**Files:** |
| +- Modify: `internal/activator/activator.go` (near `beginStopLocked`, ~line 680) |
| +- Modify: `internal/activator/activator_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: existing `a.mu`, `a.entries map[string]*actEntry`, `actAwake`, `beginStopLocked`. |
| +- Produces: `func (a *Service) Bounce(host string) bool` — Task 4 calls it; `true` means a running instance is now stopping. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +The file has an existing harness for waking instances through a fake backend — find the test that drives a wake to `actAwake` (search `actAwake` in activator_test.go) and follow its setup exactly. The test to add: |
| + |
| +```go |
| +// Bounce stops a running instance so its next wake respawns with fresh |
| +// env (config delivery). It must be a no-op for unknown hosts and for |
| +// entries not currently awake — never a spurious wake, never a panic. |
| +func TestBounceStopsAwakeInstanceOnly(t *testing.T) { |
| + // [Use the existing wake-to-awake harness from this file: construct |
| + // Service with the fake backend, drive one wake for host |
| + // "b.example.com" to actAwake — copy the arrange block from the |
| + // nearest existing test that asserts an awake state.] |
| + a := newTestServiceAwake(t, "b.example.com") // helper extracted in this step if one doesn't exist |
| + if !a.Bounce("b.example.com") { |
| + t.Fatal("Bounce(awake host) = false, want true") |
| + } |
| + a.mu.Lock() |
| + st := a.entries["b.example.com"].state |
| + a.mu.Unlock() |
| + if st != actStopping { |
| + t.Fatalf("state after Bounce = %v, want actStopping", st) |
| + } |
| + if a.Bounce("b.example.com") { |
| + t.Fatal("Bounce(already stopping) = true, want false") |
| + } |
| + if a.Bounce("never-woken.example.com") { |
| + t.Fatal("Bounce(unknown host) = true, want false") |
| + } |
| +} |
| +``` |
| + |
| +If extracting `newTestServiceAwake` from an existing test requires more than moving the arrange block into a helper, STOP and report DONE_WITH_CONCERNS instead of inventing a new harness — the reviewer will judge. |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/activator/ -run TestBounce -v` |
| +Expected: FAIL — `undefined: (*Service).Bounce` (or the helper). |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +In `activator.go`, directly after `beginStopLocked`: |
| + |
| +```go |
| +// Bounce initiates a graceful stop of host's RUNNING instance so that |
| +// its next wake — the next request, or its schedule — respawns it with |
| +// a freshly composed environment. This is config delivery's restart |
| +// half: the analogue of the unit path's try-restart, with the same |
| +// caveat that in-flight requests are not waited for. It never starts |
| +// anything: not-awake (waking, stopping, asleep, unknown) is a no-op, |
| +// and the instance picks the new config up at its natural next wake. |
| +func (a *Service) Bounce(host string) bool { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + e := a.entries[host] |
| + if e == nil || e.state != actAwake { |
| + return false |
| + } |
| + a.beginStopLocked(e) |
| + return true |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests** |
| + |
| +Run: `go test ./internal/activator/ -run TestBounce -v` → PASS, then `go test ./internal/activator/` → all pass. |
| + |
| +- [ ] **Step 5: Full suite + commit** |
| + |
| +```bash |
| +go build ./... && go vet ./... && go test ./... |
| +git add internal/activator/activator.go internal/activator/activator_test.go |
| +git commit -m "activator: Service.Bounce — stop a running instance so its next wake sees fresh env" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: `configSyncLoop` in `carlos agent` |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (new function near `adoptLoop` ~line 187; wiring in `RunAgent` ~line 145) |
| + |
| +**Interfaces:** |
| +- Consumes: `appconfig.SyncExec` (Task 1), `(*activator.Service).Bounce` (Task 3), existing `openDeployStore`, `appconfig.ParsePrivateKey`, `dataDir()`. |
| +- Produces: the running loop; nothing downstream consumes it. |
| + |
| +- [ ] **Step 1: Implement the loop** (no isolated unit test — the sweep and bounce are tested in Tasks 1 and 3; this is thin wiring following `adoptLoop`'s exact shape, which also has no loop test. The compile + full suite is the gate; say so in your report.) |
| + |
| +Add after `adoptLoop`: |
| + |
| +```go |
| +// configSyncLoop delivers app config to exec-backed routes: hourly (plus |
| +// a first pass ~1min after boot, like adoptLoop), materialize every |
| +// exec-syncable route's <dataDir>/<host>.config.env from the bucket and |
| +// bounce running instances whose config changed. Runs IN-PROCESS on |
| +// purpose — /data is inside this unit's sandbox and the exec children |
| +// are this process's own, so it can both write the files and restart |
| +// the readers; the root adopt path keeps the unit-backed half |
| +// (/etc/carlos/apps needs root). Inert until CARLOS_SECRETS_KEY is set, |
| +// the same convention as every other optional loop. |
| +func configSyncLoop(reg *registry.Registry, svc *activator.Service, dd string) { |
| + raw := os.Getenv("CARLOS_SECRETS_KEY") |
| + if raw == "" { |
| + log.Printf("agent: config sync loop disabled: CARLOS_SECRETS_KEY not set") |
| + return |
| + } |
| + priv, err := appconfig.ParsePrivateKey(raw, "CARLOS_SECRETS_KEY") |
| + if err != nil { |
| + log.Printf("agent: config sync loop disabled: %v", err) |
| + return |
| + } |
| + ctx := context.Background() |
| + store, err := openDeployStore(ctx) |
| + if err != nil { |
| + log.Printf("agent: config sync loop disabled: %v", err) |
| + return |
| + } |
| + bounce := func(host string) error { |
| + if svc == nil { |
| + return nil // no activator: files still converge for future spawns |
| + } |
| + if svc.Bounce(host) { |
| + log.Printf("agent: config changed, bounced %s", host) |
| + } |
| + return nil |
| + } |
| + log.Printf("agent: config sync loop armed (hourly + jitter)") |
| + delay := time.Minute |
| + for { |
| + time.Sleep(delay) |
| + delay = time.Hour + rand.N(10*time.Minute) |
| + if err := appconfig.SyncExec(ctx, reg, store, priv, dd, bounce); err != nil { |
| + log.Printf("agent: config sync: %v", err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +In `RunAgent`, after `go domainsLoop(...)`: |
| + |
| +```go |
| + go configSyncLoop(reg, svc, dd) |
| +``` |
| + |
| +(`svc` is in scope and may be nil when the activator isn't armed — the bounce closure handles it.) |
| + |
| +- [ ] **Step 2: Build, vet, full test** |
| + |
| +Run: `go build ./... && go vet ./... && go test ./...` — all green. `go vet` will catch a forgotten import (this function needs nothing new: `context`, `log`, `os`, `time`, `rand`, `appconfig`, `activator`, `registry` are already imported by agent.go). |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add internal/host/agent.go |
| +git commit -m "host: configSyncLoop — in-process exec-route config delivery in carlos agent" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: #14 advisory, registry Backing pin, docs |
| + |
| +**Files:** |
| +- Create: `internal/host/adopt_advice.go` |
| +- Create: `internal/host/adopt_advice_test.go` |
| +- Modify: `internal/host/agent.go` (`RunAdopt`, after the unit-path config sync block) |
| +- Modify: `internal/registry/registry_test.go` |
| +- Modify: `appconfig_cli.go` (genkey hint, ~line 240) |
| +- Modify: `README.md` ("App config & secrets" section) |
| +- Modify: `docs/superpowers/specs/2026-08-02-app-config-secrets-design.md` (§8 first bullet) |
| + |
| +**Interfaces:** |
| +- Produces: `func warnBackingMismatch(routes []registry.Route, unitActive func(host string) bool) []string` — returns the warned hosts (for the test); `RunAdopt` logs each. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Create `internal/host/adopt_advice_test.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// A route claiming exec backing while a carlos-app@<host> unit is ACTIVE |
| +// is exactly the #14 shape (kass): adopt keeps working, but config sync |
| +// silently skips the route. The adopt pass warns so the mismatch is |
| +// visible in the hourly log instead of discovered by absence. |
| +func TestWarnBackingMismatch(t *testing.T) { |
| + routes := []registry.Route{ |
| + {Host: "mismatch.example.com", Kind: "instance", App: "kass"}, // exec-claimed, unit active -> warn |
| + {Host: "honest-unit.example.com", Kind: "instance", App: "kass", Backing: registry.BackingUnit}, // consistent |
| + {Host: "honest-exec.example.com", Kind: "instance", App: "kass"}, // exec, no unit |
| + {Host: "alias.example.com", Kind: "instance", App: "kass", Platform: true}, // aliases never warn |
| + } |
| + active := map[string]bool{"mismatch.example.com": true, "honest-unit.example.com": true} |
| + got := warnBackingMismatch(routes, func(h string) bool { return active[h] }) |
| + if len(got) != 1 || got[0] != "mismatch.example.com" { |
| + t.Fatalf("warned = %v, want exactly [mismatch.example.com]", got) |
| + } |
| +} |
| +``` |
| + |
| +Append to `internal/registry/registry_test.go`: |
| + |
| +```go |
| +// Backing must survive an Upsert/List round trip — the #14 incident was |
| +// a route whose backing stamp went missing, and everything that branches |
| +// on rt.Backing (activator dispatch, config sync scope) reasons from it. |
| +func TestUpsertListRoundTripsBacking(t *testing.T) { |
| + r := openTestRegistry(t) // use this file's existing open helper; match its actual name |
| + if err := r.Upsert(Route{Host: "u.example.com", Kind: "instance", App: "a", Backing: BackingUnit}); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + routes, err := r.List() |
| + if err != nil { |
| + t.Fatalf("List: %v", err) |
| + } |
| + if len(routes) != 1 || routes[0].Backing != BackingUnit { |
| + t.Fatalf("routes = %+v, want one route with Backing=unit", routes) |
| + } |
| +} |
| +``` |
| + |
| +(Adapt the helper name to whatever registry_test.go actually uses to open a test registry; if a Backing round-trip assertion already exists, extend it rather than duplicating and say so in the report.) |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/host/ -run TestWarnBackingMismatch -v` → FAIL `undefined: warnBackingMismatch`. `go test ./internal/registry/ -run TestUpsertListRoundTripsBacking -v` → PASS or FAIL depending on prior coverage; if it passes immediately, note that in the report (the pin is still worth keeping). |
| + |
| +- [ ] **Step 3: Implement `internal/host/adopt_advice.go`** |
| + |
| +```go |
| +// Adopt-pass advisories: warnings about registry rows whose claims |
| +// disagree with what systemd is actually running. Advisory only — the |
| +// pass never mutates a route to "fix" it; the operator decides (see |
| +// platform issue #14 for the kass incident that motivated this). |
| +package host |
| + |
| +import ( |
| + "log" |
| + "os/exec" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// warnBackingMismatch reports hosts whose route claims exec backing |
| +// ("" ) while a carlos-app@<host> unit is actively running — the state |
| +// where release adoption still works but config sync silently skips the |
| +// route. Returns the warned hosts; the caller logs them. |
| +func warnBackingMismatch(routes []registry.Route, unitActive func(host string) bool) []string { |
| + var warned []string |
| + for _, rt := range routes { |
| + if rt.Kind != "instance" || rt.Platform || rt.Backing == registry.BackingUnit { |
| + continue |
| + } |
| + if unitActive(rt.Host) { |
| + warned = append(warned, rt.Host) |
| + } |
| + } |
| + return warned |
| +} |
| + |
| +// unitIsActive: the real systemd probe for warnBackingMismatch. |
| +func unitIsActive(host string) bool { |
| + return exec.Command("systemctl", "is-active", "--quiet", "carlos-app@"+host+".service").Run() == nil |
| +} |
| + |
| +// adviseBackingMismatches runs the advisory against live systemd and |
| +// logs each finding — called from RunAdopt, root-side, where systemctl |
| +// is reachable and the hourly log is read. |
| +func adviseBackingMismatches(routes []registry.Route) { |
| + for _, h := range warnBackingMismatch(routes, unitIsActive) { |
| + log.Printf("adopt: route %s claims exec backing but carlos-app@%s is ACTIVE — stamp backing='unit' (see platform issue #14); config sync is skipping this route until then", h, h) |
| + } |
| +} |
| +``` |
| + |
| +In `RunAdopt` (agent.go), after the existing config-sync block (before the closing brace of the function), add: |
| + |
| +```go |
| + if routes, err := reg.List(); err == nil { |
| + adviseBackingMismatches(routes) |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Docs** |
| + |
| +1. `appconfig_cli.go` genkey hint — replace the second stderr line: |
| + |
| +```go |
| + fmt.Fprintln(os.Stderr, "note: config sync reaches unit-backed routes (carlos-app@) via the root adopt pass, "+ |
| + "and exec-backed instance routes via the agent's in-process loop (materialized to /data/<host>.config.env; "+ |
| + "the instance's own /data/<host>.env always wins conflicts)") |
| +``` |
| + |
| +2. `README.md` — in "App config & secrets", replace the first operational note (unit-backed-only) with: |
| + |
| +```markdown |
| +- Delivery is split by backing: unit-backed routes (`Backing: "unit"`) |
| + materialize to `/etc/carlos/apps/<host>.env` via the hourly root |
| + `carlos adopt`; exec-backed instance routes (socket-upstream) |
| + materialize to `/data/<host>.config.env` via the agent's in-process |
| + hourly loop, which also bounces a running instance when its config |
| + changes. At spawn the instance's own `/data/<host>.env` is layered on |
| + top — it wins every conflict, so `CARLOS_ADMIN_TOKEN` and box-local |
| + overrides are never clobbered from the bucket. Addr-upstream |
| + (proxy-only) routes get no delivery — the process behind them is |
| + externally managed. |
| +``` |
| + |
| +3. Parent spec §8 first bullet — append to the end of the "Exec-backed routes are out of scope" bullet: |
| + |
| +```markdown |
| + *Addressed 2026-08-02 by `2026-08-02-exec-config-delivery-design.md`: |
| + exec delivery ships via a sync-owned `/data/<host>.config.env` layered |
| + under the instance's own env file at spawn.* |
| +``` |
| + |
| +- [ ] **Step 5: Run tests, full suite, commit** |
| + |
| +```bash |
| +go test ./internal/host/ ./internal/registry/ -v -run 'TestWarnBackingMismatch|TestUpsertListRoundTripsBacking' |
| +go build ./... && go vet ./... && go test ./... |
| +git add internal/host/adopt_advice.go internal/host/adopt_advice_test.go internal/host/agent.go internal/registry/registry_test.go appconfig_cli.go README.md docs/superpowers/specs/2026-08-02-app-config-secrets-design.md |
| +git commit -m "host: adopt-pass backing-mismatch advisory (#14), Backing round-trip pin, exec-delivery docs" |
| +``` |
| + |
| +--- |
| + |
| +## Post-merge integration (controller, not a subagent task) |
| + |
| +After the whole-branch review, fixes, and merge (merge-and-deploy-to-dev is pre-authorized; production is NOT): |
| + |
| +1. Build linux-arm64 at the merge commit, stage to `s3://carlos-dev-640012953448/ops/`, install on `carlos-dev-ie` (i-05498eada6f0afd23, profile `kass`) via SSM: binary swap + `systemctl restart carlos-edge` — the same allowed pattern as tonight. |
| +2. Live proof on dev's exec-backed `helloworld` route: `carlos env set -app helloworld CONFIG_PROOF=exec-shape` + `carlos secrets set -app helloworld PROOF_SECRET=...` (pubkey `BLsM2F4v+qdjgzFsJ8s6R87xHiVrDUdsv/zZ1i8guv6n/zBCSzcWJRNHcQdW6YqPSpgr8rY/RW3W2CWDRHmPGTU=`, bucket `carlos-dev-640012953448`); wait for/trigger the loop's first pass (restart of carlos-edge re-arms it with a ~1min first pass); verify `/data/helloworld.dev.oncarlos.com.config.env` exists 0600 with both values; hit `https://helloworld.dev.oncarlos.com` to wake the instance, then verify the child env via `sudo cat /proc/$(pgrep -f 'live/helloworld')/environ | tr '\0' '\n' | grep -E 'CONFIG_PROOF|PROOF_SECRET'` (SSM, root). Unset both vars after; verify the file converges to empty; delete the manifest test object is NOT needed (unset suffices — deletion path is unit-tested). |
| +3. Stage `fix-kass-backing.sh` to `s3://carlos-flagship-271376211898/ops/` for the operator (surgical `UPDATE routes SET backing='unit' WHERE host='home.kass.training' AND backing=''` + verify SELECT) — do NOT run it; classifier + morning approval. |
| +4. Update shared memory (session-exec-state) and comment on issue #14 that the advisory + pin shipped and the ops script is staged; production rollout of the new binary queued for the operator's morning. |
diff --git a/docs/superpowers/specs/2026-08-02-app-config-secrets-design.md b/docs/superpowers/specs/2026-08-02-app-config-secrets-design.md| index f8596abef..40f427030 100644 |
| --- a/docs/superpowers/specs/2026-08-02-app-config-secrets-design.md |
| +++ b/docs/superpowers/specs/2026-08-02-app-config-secrets-design.md |
| @@ -194,6 +194,9 @@ Recorded after the whole-branch review, before merge: |
| Until that's designed, `Sync` skips exec-backed routes with a log line |
| rather than writing a file nothing reads. Extending config sync to the |
| exec backend is explicitly future work. |
| + *Addressed 2026-08-02 by `2026-08-02-exec-config-delivery-design.md`: |
| + exec delivery ships via a sync-owned `/data/<host>.config.env` layered |
| + under the instance's own env file at spawn.* |
| - **Sync runs before release adoption in the pass**, so a newly adopted |
| route's first start already sees its env file and a pass carrying both |
| a release and a config change never starts the new binary against stale |
diff --git a/docs/superpowers/specs/2026-08-02-exec-config-delivery-design.md b/docs/superpowers/specs/2026-08-02-exec-config-delivery-design.md| new file mode 100644 |
| index 000000000..51c15fff0 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-02-exec-config-delivery-design.md |
| @@ -0,0 +1,200 @@ |
| +# Exec-backend config delivery — design |
| + |
| +**Date:** 2026-08-02 |
| +**Status:** Approved (design approved in-session; merge-and-deploy-to-dev |
| +authorized; production rollout queued for operator approval) |
| + |
| +Extends `carlos env` / `carlos secrets` (spec |
| +`2026-08-02-app-config-secrets-design.md`, shipped as `68d0662`) to |
| +exec-backed routes — the instance-fleet shape — closing the limitation |
| +recorded in that spec's §8. App-level config currently reaches only |
| +unit-backed routes (`carlos-app@<host>`, `EnvironmentFile= |
| +/etc/carlos/apps/%i.env`); exec-backed routes (`Backing == ""`, the fleet |
| +default) get their environment from the activator at spawn and receive |
| +nothing from the bucket. |
| + |
| +## 1. Scope |
| + |
| +- Routes the **activator spawns**: `Kind == "instance"`, `Backing == ""`, |
| + with a **Socket upstream** — plus the unit-path `syncable` exclusions |
| + (no platform-minted aliases, `Status` empty or `"ready"`, `App` set). |
| + These are the Eleven-shaped instances — the activator execs |
| + `live/<host>` and composes the child's environment. |
| +- **Out of scope:** routes with an `Addr` upstream (proxy-only — the |
| + process behind them is externally managed, e.g. titogo's router fleet; |
| + there is nothing for the activator to configure or bounce). Sidecar |
| + processes (their env file is a separate, activator-owned contract). |
| + Unit-backed routes (already served by the root adopt path; unchanged). |
| +- Same manifest, same CLI, same crypto, same bucket object as the unit |
| + path — only the materialization target and delivery loop are new. |
| + |
| +## 2. Ownership model: a second file, never a merge-in-place |
| + |
| +New file namespace: **`<DataDir>/<host>.config.env`** (production: |
| +`/data/<host>.config.env`), `0600`, owned by the agent (edge) user. |
| + |
| +- Written **solely** by config sync. Nothing else creates these files, so |
| + sync owns the whole lifecycle by construction — the hand-staged-file |
| + migration hazard that shaped the unit path does not exist here. |
| +- Convergence: manifest exists → file converges to `RenderEnv` output |
| + (including to empty, when every var is unset). Manifest object **absent** |
| + → the file is **removed** if present (unlike the unit path, absence |
| + cannot mean "hand-staged": only sync writes this namespace, so a file |
| + with no manifest is a leftover, and leaving it would keep dead config |
| + flowing into spawns). |
| +- Reap: `<host>.config.env` for hosts with zero route rows is deleted, |
| + under the same guards the unit path got post-review: an **empty route |
| + list never reaps** (registry.Open auto-creates empty DBs), and stale |
| + `*.config.env.tmp` files are removed. Atomic write via tmp+rename. |
| +- The instance's own **`<DataDir>/<host>.env` is never touched** — not |
| + read by sync, not merged into, not reaped. It remains the box-local, |
| + instance-owned file carrying `CARLOS_ADMIN_TOKEN` and hand-staged vars. |
| + |
| +**Accepted risk — no tenant isolation on disk.** Every exec-backed |
| +instance on a box runs as the same agent uid and shares the same `/data` |
| +directory, so `0600` only keeps *other users* out — it does not stop one |
| +instance from reading another instance's `.config.env` (or `.env`, or any |
| +other tenant's files). Any app process on the box can read every other |
| +app's rendered config, secrets included. This matches the pre-existing |
| +exposure of `<host>.env`, which already carries `CARLOS_ADMIN_TOKEN` under |
| +the same shared-uid model — `.config.env` doesn't newly weaken anything. |
| +Accepted for v1. Unit-backed routes don't have this property: each |
| +`carlos-app@<host>.service` can in principle be sandboxed per-unit, so |
| +that path has room to add real isolation later that the exec path |
| +structurally cannot. |
| + |
| +## 3. Spawn-time merge, instance file wins |
| + |
| +`ExecBackend`'s spawn path composes the child environment as: |
| + |
| +1. base: `readEnvFile(<host>.config.env)` — optional, like today's file |
| +2. overlay: `readEnvFile(<host>.env)` — optional, exactly today's file |
| +3. `childEnv(merged)` where **the overlay (instance file) wins every |
| + conflict**. |
| + |
| +Consequences, both deliberate: `CARLOS_ADMIN_TOKEN` can never be |
| +clobbered from the bucket (it lives only in the instance file), and a |
| +box-local override in `<host>.env` beats central config. Operators |
| +migrate a var to the bucket by deleting it from `<host>.env`. The token |
| +lookup paths (`NextDue` etc.) keep reading `<host>.env` only — the token |
| +is not config. |
| + |
| +## 4. Delivery: an edge-side sync loop |
| + |
| +A new **in-process loop in `carlos agent`** (`configSyncLoop`, shaped |
| +after `adoptLoop`: first pass ~1 minute after boot, then hourly + |
| +jitter): |
| + |
| +- Gated on `CARLOS_SECRETS_KEY` — inert until configured, one log line, |
| + the same convention as every other optional loop. The edge already |
| + receives `host.env` via its unit's `EnvironmentFile`; **no unit changes, |
| + no new privileges** — `/data` is already inside the edge sandbox's |
| + `ReadWritePaths`. |
| +- Resolves once per distinct `(Account, App, Environment)` among in-scope |
| + routes, renders once, converges every in-scope route's `.config.env`. |
| + The grouping and the resolution mirror the unit-path `Sync` exactly |
| + (pipelines design §4a): the default bundle is read once per |
| + `(Account, App)` via `getIfExists`, and a route with a **bound** |
| + environment resolves through `LoadMerged` — the default bundle overlaid |
| + by the named bundle, environment winning per key. Two exec instances of |
| + one app bound to different environments therefore get different files |
| + in the same pass. |
| +- **Three-state resolution**, with this path's own file semantics for the |
| + middle state: |
| + - *resolved* — converge the file, bounce on change. |
| + - *never configured* (`Environment == ""` and the app has no |
| + `config.json` object at all) — **remove** the file. Legitimate here, |
| + where the unit path skips, because sync owns the `*.config.env` |
| + namespace outright: nothing hand-staged can live in it, so a file |
| + with no manifest behind it is a leftover feeding dead config into |
| + spawns. Subject to the mass-removal guard below. |
| + - *resolution failed* (bucket fetch error, or a bound environment |
| + missing from the bucket — `LoadMerged` treats that as an error, not |
| + an empty overlay) — leave the file **exactly as-is**, log loudly per |
| + route, every pass, and do not bounce. |
| + |
| + Error and absence are not merely different on this path, they are |
| + opposite: absence removes. Folding a missing bundle into absence would |
| + strip a live instance's entire config, and bounce it onto the stripped |
| + file, because an operator deleted a bundle. |
| +- A mass-removal guard sits in front of the *absent* state: when more |
| + than one resolvable group exists and **every** one of them resolved |
| + absent, no removals happen this pass (the shape a mangled registry |
| + account column produces, where every bucket key 404s at once). |
| +- No key-loss guard, deliberately — the unit path's `lostKeys` / |
| + `defaultMissing` refusal exists because `/etc/carlos/apps` files may be |
| + hand-staged, and has no counterpart here for the ownership reason |
| + above. §2's accepted risk is the same argument. |
| +- Die-free per group, like `release.Pass` and the unit-path `Sync`. |
| +- The root adopt path (`carlos adopt`) is **not** taught about exec |
| + routes; the two writers never share a namespace. (The root one-shot |
| + also can't bounce the edge's children — placement follows capability.) |
| + |
| +## 5. Bounce on change |
| + |
| +When a route's `.config.env` **content changes** and the activator has a |
| +**running** instance for that host, the sync asks the activator to stop |
| +it (a new small `Service` method wrapping the existing stop machinery). |
| +The next request — or the schedule wake — respawns it through the normal |
| +wake path, which now composes the fresh env. Semantics match the unit |
| +path's restart-on-change: converged within one pass fleet-wide. |
| + |
| +- Not running / hibernated → no bounce; the file is picked up at next |
| + wake. No instance is ever *started* just to receive config. |
| +- Bounce is best-effort and logged; a bounce failure never aborts the |
| + sweep. |
| +- Activator disabled (no hibernate bucket) → no exec children exist to |
| + bounce; the loop still materializes files so spawns pick them up when |
| + the activator is enabled. |
| + |
| +## 6. Non-goals |
| + |
| +- Rewriting or merging `<host>.env` in place (rejected: fragile ownership |
| + tracking; the token lives there). |
| +- Fetch-at-spawn from the bucket (rejected: bucket latency/availability |
| + in the wake path, no on-disk artifact to debug). |
| +- Per-sidecar config, addr-route delivery, and key rotation/rewrap — all |
| + unchanged from the parent spec's §6/§8. |
| + |
| +The pipelines `environments/` reconciliation is **no longer** a non-goal |
| +here: §4's resolution *is* that reconciliation for the exec path. The |
| +unit path was reconciled first, in the environments slice itself (main's |
| +`Sync`); this spec's implementation follows it, deliberately mirroring |
| +its grouping and three-state resolution so the two halves of delivery |
| +never drift apart on what a route's config *is* — only on what a file |
| +absence *means*, which is a property of who owns the namespace. |
| + |
| +## 7. Companion track: issue #14 prep (kass backing stamp) |
| + |
| +- Stage `fix-kass-backing.sh` (surgical |
| + `UPDATE routes SET backing='unit' WHERE host='home.kass.training'` + |
| + verify select) to the flagship bucket's `ops/` prefix for the |
| + operator's one-command morning run. A registry-row `carlos add` upsert |
| + is explicitly the wrong tool (it would reset the route's |
| + `canary/rehearsal` channel mid-bake). |
| +- Find where the registry rebuild path dropped `Backing` and fix it to |
| + carry the column through, with a regression test. |
| +- Amend the parent spec's §8 to record that the exec limitation is |
| + addressed by this design. |
| + |
| +## 8. Testing |
| + |
| +- **Merge precedence** (the load-bearing test): a var in both files |
| + resolves to the instance file's value; `CARLOS_ADMIN_TOKEN` present in |
| + `<host>.env` and (maliciously) in the bucket manifest still yields the |
| + instance file's token in the child env. |
| +- Lifecycle: converge on set; converge-to-empty on full unset; **file |
| + removed when the manifest object is deleted**; reap on route removal; |
| + empty-registry guard holds; `.tmp` hygiene. |
| +- Bounce: content change + running instance → exactly one stop; unchanged |
| + content → no stop; not-running → no stop, file still written. |
| +- Inert: no `CARLOS_SECRETS_KEY` → loop exits after one log line, no |
| + files touched. |
| +- Scope filter: addr-upstream and unit-backed routes are untouched by the |
| + exec sweep. |
| +- **Live proof on dev** (merge-and-deploy-to-dev authorized): dev's |
| + `helloworld` is exec-backed — `carlos env set`/`secrets set`, watch the |
| + agent loop materialize `/data/helloworld.dev.oncarlos.com.config.env` |
| + and bounce, verify the child's real environment via |
| + `/proc/<pid>/environ`, then unset/cleanup. Production rollout queued |
| + for operator approval. |
diff --git a/internal/activator/activator.go b/internal/activator/activator.go| index e47ef06d5..6c98be5ea 100644 |
| --- a/internal/activator/activator.go |
| +++ b/internal/activator/activator.go |
| @@ -674,15 +674,36 @@ func (a *Service) doWake(ctx context.Context, rt registry.Route) (*wokenInstance |
| const actStopTimeout = 20 * time.Second |
| // beginStopLocked flips an awake entry to stopping and kicks the stop off; the |
| -// caller holds a.mu. In-flight requests are not waited for — a stop is only |
| -// initiated when the instance is idle, dead, or has lost its lease, and in the |
| -// latter two cases open requests are already doomed. |
| +// caller holds a.mu. In-flight requests are not waited for (the caller's |
| +// SIGTERM drain inside actStopTimeout is what covers them). Callers: the |
| +// idle sweep (instance has been idle past cfg.Idle), a dead child, a lost |
| +// lease (open requests on either of the last two are already doomed), and |
| +// Bounce (config changed — the instance may be mid-request, but the drain |
| +// still gives it actStopTimeout to finish before SIGKILL). |
| func (a *Service) beginStopLocked(e *actEntry) { |
| e.state = actStopping |
| e.stopped = make(chan struct{}) |
| go a.stop(e) |
| } |
| +// Bounce initiates a graceful stop of host's RUNNING instance so that |
| +// its next wake — the next request, or its schedule — respawns it with |
| +// a freshly composed environment. This is config delivery's restart |
| +// half: the analogue of the unit path's try-restart, with the same |
| +// caveat that in-flight requests are not waited for. It never starts |
| +// anything: not-awake (waking, stopping, asleep, unknown) is a no-op, |
| +// and the instance picks the new config up at its natural next wake. |
| +func (a *Service) Bounce(host string) bool { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + e := a.entries[host] |
| + if e == nil || e.state != actAwake { |
| + return false |
| + } |
| + a.beginStopLocked(e) |
| + return true |
| +} |
| + |
| // stop drives awake→asleep: stop the child (its SIGTERM drain is the final |
| // write), checkpoint what's on disk to S3 and release the lease, then forget |
| // the entry. The local DB stays in place — it's the warm cache for the next |
| @@ -708,10 +729,14 @@ func (a *Service) stop(e *actEntry) { |
| // failure records 0, i.e. "sleeps until a request": a missed probe must |
| // degrade to pre-index behavior, never block or crash the sleep. A due time |
| // already in the past is recorded as-is and the sweeper wakes the host on |
| - // its next tick — there is deliberately no minimum-sleep guard beyond that, |
| - // because thrash is already impossible: an awake instance only sleeps after |
| - // a full idle window, so wake/sleep cycles can never be spaced closer than |
| - // cfg.Idle. |
| + // its next tick — there is deliberately no minimum-sleep guard beyond |
| + // that. The idle sweep alone would make thrash impossible (an awake |
| + // instance only sleeps after a full idle window), but Bounce breaks that |
| + // bound on purpose: it can stop an instance shortly after it woke, so |
| + // consecutive wake/sleep cycles are bounded by config-change frequency, |
| + // not cfg.Idle. That's accepted — a bounce is rare relative to request |
| + // traffic, and each cycle is still a bounded SIGTERM drain, not a busy |
| + // spin. |
| dueCtx, cancelDue := context.WithTimeout(context.Background(), 5*time.Second) |
| due, err := a.backend.NextDue(dueCtx, e.route) |
| cancelDue() |
diff --git a/internal/activator/activator_test.go b/internal/activator/activator_test.go| index 8b48b26c0..164c9d137 100644 |
| --- a/internal/activator/activator_test.go |
| +++ b/internal/activator/activator_test.go |
| @@ -316,6 +316,21 @@ func testRoute(host string) registry.Route { |
| DBPath: filepath.Join("/tmp", host+".db"), Hibernate: true} |
| } |
| +// newTestServiceAwake builds a Service against the fake backend and drives |
| +// host to actAwake via a real beginRequest/release round trip — the same |
| +// wake arrange block used throughout this file (e.g. |
| +// TestActivatorShutdownStopsChildren, TestActivatorConvergesOnSymlinkRepoint). |
| +func newTestServiceAwake(t *testing.T, host string) *Service { |
| + t.Helper() |
| + a, _, _, _ := testActivator(t, time.Hour) |
| + release, err := a.beginRequest(context.Background(), testRoute(host)) |
| + if err != nil { |
| + t.Fatalf("beginRequest %s: %v", host, err) |
| + } |
| + release() |
| + return a |
| +} |
| + |
| // waitFor polls until a condition holds — the stop path runs on its own |
| // goroutine, so tests observe its completion rather than assuming it. |
| func waitFor(t *testing.T, what string, cond func() bool) { |
| @@ -1171,6 +1186,28 @@ func TestActivatorShutdownStopsChildren(t *testing.T) { |
| } |
| } |
| +// Bounce stops a running instance so its next wake respawns with fresh |
| +// env (config delivery). It must be a no-op for unknown hosts and for |
| +// entries not currently awake — never a spurious wake, never a panic. |
| +func TestBounceStopsAwakeInstanceOnly(t *testing.T) { |
| + a := newTestServiceAwake(t, "b.example.com") |
| + if !a.Bounce("b.example.com") { |
| + t.Fatal("Bounce(awake host) = false, want true") |
| + } |
| + a.mu.Lock() |
| + st := a.entries["b.example.com"].state |
| + a.mu.Unlock() |
| + if st != actStopping { |
| + t.Fatalf("state after Bounce = %v, want actStopping", st) |
| + } |
| + if a.Bounce("b.example.com") { |
| + t.Fatal("Bounce(already stopping) = true, want false") |
| + } |
| + if a.Bounce("never-woken.example.com") { |
| + t.Fatal("Bounce(unknown host) = true, want false") |
| + } |
| +} |
| + |
| // --- due-time index -------------------------------------------------------------- |
| // entryAwake reports whether host has a live entry in the awake state — how the |
| @@ -1848,6 +1885,34 @@ func splitEnv(kv string) (k, v string, ok bool) { |
| return "", "", false |
| } |
| +// mergeEnv layers the sync-owned <host>.config.env UNDER the instance's |
| +// own <host>.env: the instance file wins every conflict, which is what |
| +// makes CARLOS_ADMIN_TOKEN structurally unclobberable from the bucket. |
| +func TestMergeEnvInstanceFileWins(t *testing.T) { |
| + base := map[string]string{"A": "bucket", "B": "bucket-only", "CARLOS_ADMIN_TOKEN": "forged"} |
| + overlay := map[string]string{"A": "instance", "CARLOS_ADMIN_TOKEN": "real"} |
| + got := mergeEnv(base, overlay) |
| + if got["A"] != "instance" || got["B"] != "bucket-only" || got["CARLOS_ADMIN_TOKEN"] != "real" { |
| + t.Fatalf("mergeEnv = %v, want instance values winning and bucket-only surviving", got) |
| + } |
| + // Inputs must not be mutated (withVar's same discipline). |
| + if base["A"] != "bucket" || len(overlay) != 2 { |
| + t.Fatal("mergeEnv mutated an input map") |
| + } |
| +} |
| + |
| +func TestMergeEnvNilInputs(t *testing.T) { |
| + if got := mergeEnv(nil, map[string]string{"A": "1"}); got["A"] != "1" { |
| + t.Fatalf("nil base: got %v", got) |
| + } |
| + if got := mergeEnv(map[string]string{"A": "1"}, nil); got["A"] != "1" { |
| + t.Fatalf("nil overlay: got %v", got) |
| + } |
| + if got := mergeEnv(nil, nil); len(got) != 0 { |
| + t.Fatalf("nil/nil: got %v", got) |
| + } |
| +} |
| + |
| // The sleep-time next-due probe: GET /api/next-due over the instance's unix |
| // socket, authorized with the CARLOS_ADMIN_TOKEN from its env file. |
| func TestExecBackendNextDue(t *testing.T) { |
| @@ -1943,6 +2008,39 @@ func stageRecordingLive(t *testing.T, b *ExecBackend, host, ver string) (argvPat |
| return argvPath, envPath |
| } |
| +// stageRecordingLiveFullEnv is stageRecordingLive's sibling: instead of |
| +// picking out CARLOS_CONTROL_SOCKET alone, it dumps the child's ENTIRE |
| +// real environment (`env`) to envPath. TestExecBackendStartMergesConfigEnvUnderOwnEnv |
| +// needs the full dump — it isn't just checking that a var made it into the |
| +// map Start builds, it's checking what the exec'd process itself actually |
| +// saw. |
| +func stageRecordingLiveFullEnv(t *testing.T, b *ExecBackend, host, ver string) (argvPath, envPath string) { |
| + t.Helper() |
| + dir := t.TempDir() |
| + argvPath = filepath.Join(dir, "argv") |
| + envPath = filepath.Join(dir, "env") |
| + script := "#!/bin/sh\n" + |
| + "printf '%s\\n' \"$@\" > '" + argvPath + "'\n" + |
| + "env > '" + envPath + "'\n" |
| + |
| + vdir := filepath.Join(b.OptDir, "versions", ver) |
| + if err := os.MkdirAll(vdir, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + bin := filepath.Join(vdir, "app") |
| + if err := os.WriteFile(bin, []byte(script), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(b.OptDir, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + os.Remove(b.liveLink(host)) |
| + if err := os.Symlink(bin, b.liveLink(host)); err != nil { |
| + t.Fatal(err) |
| + } |
| + return argvPath, envPath |
| +} |
| + |
| // waitForFile polls until path exists (the child writes it concurrently with |
| // Start's health wait) and returns its contents. |
| func waitForFile(t *testing.T, path string, within time.Duration) string { |
| @@ -2061,6 +2159,66 @@ func TestExecBackendStartWithNilCacheSkipsPurgeListener(t *testing.T) { |
| } |
| } |
| +// TestExecBackendStartMergesConfigEnvUnderOwnEnv is the load-bearing |
| +// merge-precedence regression: unlike a unit test of mergeEnv in isolation, |
| +// this goes through the real ExecBackend.Start spawn path and inspects the |
| +// spawned process's ACTUAL environment (via `env`, dumped by the recorder |
| +// script) — the only way to see what the child was really handed. |
| +// |
| +// It stages both <host>.config.env (sync-owned, bucket-derived — |
| +// CARLOS_ADMIN_TOKEN=forged, SHARED=bucket, BUCKET_ONLY=1) and <host>.env |
| +// (the instance's own file — CARLOS_ADMIN_TOKEN=real, SHARED=instance) and |
| +// asserts the child saw the instance's own values winning the conflicts |
| +// while the bucket-only var still flowed through. This must fail if |
| +// Start's `mergeEnv(cfgEnv, ownEnv)` call is ever transposed to |
| +// `mergeEnv(ownEnv, cfgEnv)`: that would let bucket-owned config forge |
| +// CARLOS_ADMIN_TOKEN, a secret held only in the instance's own file. |
| +// Verified empirically by transposing the call and watching this test |
| +// fail, then restoring it (see the fix report for the RED output). |
| +func TestExecBackendStartMergesConfigEnvUnderOwnEnv(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "merge-precedence.example.com" |
| + _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1") |
| + |
| + cfgEnv := "CARLOS_ADMIN_TOKEN=forged\nSHARED=bucket\nBUCKET_ONLY=1\n" |
| + if err := os.WriteFile(filepath.Join(b.DataDir, host+".config.env"), []byte(cfgEnv), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + ownEnv := "CARLOS_ADMIN_TOKEN=real\nSHARED=instance\n" |
| + if err := os.WriteFile(filepath.Join(b.DataDir, host+".env"), []byte(ownEnv), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + rt := registry.Route{ |
| + Host: host, |
| + Kind: "instance", |
| + Socket: filepath.Join(t.TempDir(), "instance.sock"), |
| + } |
| + |
| + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| + defer cancel() |
| + proc, _, _ := b.Start(ctx, rt) // health-wait error from the fake binary is expected and irrelevant here |
| + if proc != nil { |
| + defer proc.Stop(context.Background()) |
| + } |
| + |
| + child := map[string]string{} |
| + for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") { |
| + if k, v, ok := strings.Cut(line, "="); ok { |
| + child[k] = v |
| + } |
| + } |
| + if got := child["CARLOS_ADMIN_TOKEN"]; got != "real" { |
| + t.Fatalf("child CARLOS_ADMIN_TOKEN = %q, want %q — the instance's own env must win over bucket config", got, "real") |
| + } |
| + if got := child["SHARED"]; got != "instance" { |
| + t.Fatalf("child SHARED = %q, want %q — the instance's own env must win over bucket config", got, "instance") |
| + } |
| + if got := child["BUCKET_ONLY"]; got != "1" { |
| + t.Fatalf("child BUCKET_ONLY = %q, want %q — bucket config must still flow through when not overridden", got, "1") |
| + } |
| +} |
| + |
| // Spawning a fresh process for a host is a deploy boundary: the code behind |
| // the host just changed, so every response cached from the old process is |
| // stale. Start drops them all before the new process serves anything — |
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go| index f04447ee9..68f6393bb 100644 |
| --- a/internal/activator/backend_exec.go |
| +++ b/internal/activator/backend_exec.go |
| @@ -230,11 +230,18 @@ func (b *ExecBackend) Start(ctx context.Context, rt registry.Route) (Proc, strin |
| return nil, "", err |
| } |
| - // The env file is optional, like the unit's "-" prefix. |
| - fileEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".env")) |
| + // Two env layers, both optional like the unit's "-" prefix: the |
| + // sync-owned bucket config underneath, the instance's own file on |
| + // top (it wins conflicts — see mergeEnv). |
| + cfgEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".config.env")) |
| + if err != nil && !os.IsNotExist(err) { |
| + return nil, "", fmt.Errorf("read config env file: %w", err) |
| + } |
| + ownEnv, err := readEnvFile(filepath.Join(b.DataDir, host+".env")) |
| if err != nil && !os.IsNotExist(err) { |
| return nil, "", fmt.Errorf("read env file: %w", err) |
| } |
| + fileEnv := mergeEnv(cfgEnv, ownEnv) |
| // The control socket reaches the child through its ENVIRONMENT, never as a |
| // CLI flag: every app binary built before this feature existed parses its |
| @@ -333,6 +340,23 @@ func withVar(env map[string]string, key, value string) map[string]string { |
| return out |
| } |
| +// mergeEnv composes the child's env-file layers: base (the sync-owned |
| +// <host>.config.env, bucket-derived) under overlay (the instance's own |
| +// <host>.env). The overlay wins every conflict — CARLOS_ADMIN_TOKEN |
| +// lives only in the instance file and must never be clobberable from |
| +// the bucket, and a box-local override deliberately beats central |
| +// config. Returns a fresh map; inputs are never mutated. |
| +func mergeEnv(base, overlay map[string]string) map[string]string { |
| + out := make(map[string]string, len(base)+len(overlay)) |
| + for k, v := range base { |
| + out[k] = v |
| + } |
| + for k, v := range overlay { |
| + out[k] = v |
| + } |
| + return out |
| +} |
| + |
| // sidecarEnvPath: the host's sidecar EnvironmentFile, |
| // <DataDir>/<host>-sidecar.env (Eleven's <host>-bot.env, renamed). |
| func (b *ExecBackend) sidecarEnvPath(host string) string { |
diff --git a/internal/appconfig/sync.go b/internal/appconfig/sync.go| index 7be8313f3..ce177e02a 100644 |
| --- a/internal/appconfig/sync.go |
| +++ b/internal/appconfig/sync.go |
| @@ -159,8 +159,10 @@ func Sync(ctx context.Context, reg *registry.Registry, s release.Store, priv *ec |
| // overwritten here). Materializing for one would write a file |
| // nothing reads and log a restart that restarted nothing; say so |
| // instead of succeeding silently. Spec §5 records this limitation. |
| + // Delivery still happens for these routes — SyncExec, called from |
| + // the agent's in-process configSyncLoop, not this pass. |
| if rt.Backing != registry.BackingUnit { |
| - fmt.Printf(" %s: has bucket config but is exec-backed — config sync only reaches carlos-app@ unit routes, skipped\n", rt.Host) |
| + fmt.Printf(" %s: exec-backed — config delivered by the agent's in-process loop to /data/<host>.config.env, not this pass\n", rt.Host) |
| continue |
| } |
| syncHost(rt, m, r.defaultMissing, priv, envDir, restart) |
diff --git a/internal/appconfig/syncexec.go b/internal/appconfig/syncexec.go| new file mode 100644 |
| index 000000000..b6040135c |
| --- /dev/null |
| +++ b/internal/appconfig/syncexec.go |
| @@ -0,0 +1,349 @@ |
| +// SyncExec is the exec-backend half of config delivery: for every |
| +// activator-spawned route (socket-upstream, non-unit), materialize the |
| +// route's merged manifest into <dataDir>/<host>.config.env — the file |
| +// ExecBackend.Start layers UNDER the instance's own <host>.env at spawn. |
| +// Sync owns this namespace outright (nothing else writes *.config.env), |
| +// which is why an absent manifest REMOVES the file here, where the |
| +// unit-path Sync leaves /etc/carlos/apps files alone: there is no |
| +// hand-staged-file migration hazard by construction. See spec |
| +// 2026-08-02-exec-config-delivery-design.md. |
| +// |
| +// Resolution mirrors sync.go's exactly — the group is (Account, App, |
| +// Environment) and a route's bound environment is an overlay on the app's |
| +// default bundle (environments.go, pipelines design §4a). Only the file |
| +// semantics differ, and only where sync's ownership of the namespace |
| +// makes them differ: absence removes rather than skips, and there is no |
| +// key-loss guard (see syncExecHost). |
| +package appconfig |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "crypto/ecdh" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// configEnvSuffix names sync's file namespace under dataDir. Everything |
| +// with this suffix belongs to sync; nothing else in dataDir does. |
| +const configEnvSuffix = ".config.env" |
| + |
| +// SyncExec resolves config once per distinct (Account, App, Environment) |
| +// among the box's exec-syncable routes and converges every such route's |
| +// <dataDir>/<host>.config.env, calling bounce(host) whenever a file's |
| +// content changed (including removal) so a running instance respawns |
| +// with fresh env on its next wake. Die-free per group, like Sync. |
| +// |
| +// The group carries Environment for the same reason Sync's does |
| +// (pipelines design §4a): two routes of ONE app bound to different |
| +// environments resolve to different manifests, and keying on (acct, app) |
| +// alone would render whichever route was seen first into both files. |
| +func SyncExec(ctx context.Context, reg *registry.Registry, s release.Store, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error) error { |
| + routes, err := reg.List() |
| + if err != nil { |
| + return err |
| + } |
| + type groupKey struct{ acct, app, env string } |
| + // The same three states Sync resolves to, with this path's own file |
| + // semantics for the middle one: a manifest to converge to; "never |
| + // configured at all" (m == nil, err == nil), which REMOVES the file |
| + // here because sync owns the *.config.env namespace outright (nothing |
| + // hand-staged can live in it); and "resolution failed" (err != nil), |
| + // which leaves the file exactly as it is and says so, loudly, per |
| + // route. An error is never absence — folding the two together would |
| + // delete a live route's config on a bucket blip or a deleted bundle. |
| + type resolution struct { |
| + m *Manifest |
| + err error |
| + } |
| + resolved := map[groupKey]resolution{} |
| + // The default bundle is read once per (acct, app) and reused across |
| + // that app's environments — same cache, same reason, as Sync's. |
| + type appKey struct{ acct, app string } |
| + type defaultRes struct { |
| + m *Manifest |
| + found bool |
| + err error |
| + } |
| + defaults := map[appKey]defaultRes{} |
| + // getIfExists, not Get: an app with no manifest object in the bucket |
| + // at all must be distinguishable from one whose every var was unset. |
| + // Get folds those together into an empty manifest; here the first |
| + // removes the route's config file and the second converges it to an |
| + // empty one. |
| + loadDefault := func(acct, app string) defaultRes { |
| + ak := appKey{acct, app} |
| + if d, done := defaults[ak]; done { |
| + return d |
| + } |
| + m, found, err := getIfExists(ctx, s, acct, app) |
| + if err != nil { |
| + err = fmt.Errorf("%s/%s: config fetch failed: %w", acct, app, err) |
| + } |
| + d := defaultRes{m: m, found: found, err: err} |
| + defaults[ak] = d |
| + return d |
| + } |
| + for _, rt := range routes { |
| + if !execSyncable(rt) { |
| + continue |
| + } |
| + k := groupKey{release.RouteAccount(rt), rt.App, rt.Environment} |
| + if _, done := resolved[k]; done { |
| + continue |
| + } |
| + def := loadDefault(k.acct, k.app) |
| + if def.err != nil { |
| + // A fetch failure is NOT "absent": leave this app's files |
| + // alone this pass rather than removing config on a blip. |
| + resolved[k] = resolution{err: def.err} |
| + continue |
| + } |
| + if k.env == "" { |
| + if !def.found { |
| + resolved[k] = resolution{} // absent -> remove |
| + continue |
| + } |
| + resolved[k] = resolution{m: def.m} |
| + continue |
| + } |
| + // A bound environment asserts the bundle exists, so LoadMerged |
| + // treats its absence as an error rather than an empty overlay — |
| + // and that error stays contained to the routes bound to it. There |
| + // is no never-configured case here: an app can lack a default |
| + // bundle and still resolve (environment-only), and a missing |
| + // environment is the error above, not silence. Critically for |
| + // THIS path, that error must not read as absence: removing the |
| + // file would strip a live instance's config because someone |
| + // deleted a bundle. |
| + m, err := LoadMerged(ctx, s, k.acct, k.app, k.env) |
| + if err != nil || m == nil { |
| + if err == nil { |
| + err = fmt.Errorf("config: resolved a nil manifest for environment %q", k.env) |
| + } |
| + resolved[k] = resolution{err: err} |
| + continue |
| + } |
| + resolved[k] = resolution{m: m} |
| + } |
| + |
| + // Mass-removal misconfiguration guard. A registry rebuild that mangles |
| + // the account column makes release.RouteAccount fall back to "ops" for |
| + // every app at once, so every group's bucket key 404s and every group |
| + // in this pass looks "absent" — indistinguishable, from here, from |
| + // every manifest object having genuinely been deleted. An unguarded |
| + // pass would then remove every config file on the box in one sweep. |
| + // Require MORE THAN ONE group before refusing: a single-app box (a |
| + // self-hoster who really did delete their one manifest) must still |
| + // converge normally, and a lone group can't produce a false "mass" |
| + // signal — only a registry-wide fault spanning multiple apps can look |
| + // like this. |
| + // |
| + // Groups in the ERROR state are counted on neither side: they are |
| + // neither absent nor present, they never remove anything anyway, and |
| + // letting one sit in the denominator would silently disarm the guard |
| + // on a box where one app's bundle happens to be broken. That choice |
| + // has a direction: a PARTIALLY errored mangled box can still remove |
| + // the remaining absent groups' files, which counting errors as absent |
| + // would have prevented — but counting them as absent would also freeze |
| + // legitimate removals for as long as any one app's bundle stays |
| + // broken, indefinitely and silently. Bounded, recoverable over-removal |
| + // beats an unbounded silent freeze; routes bound to an environment are |
| + // protected by the error path regardless, since an unresolved group |
| + // never reaches a removal at all. |
| + eligible, absent := 0, 0 |
| + for _, r := range resolved { |
| + if r.err != nil { |
| + continue |
| + } |
| + eligible++ |
| + if r.m == nil { |
| + absent++ |
| + } |
| + } |
| + skipRemovals := eligible > 1 && absent == eligible |
| + if skipRemovals { |
| + // Name what was actually checked — "every app" would overclaim, |
| + // since errored groups were never in this count. |
| + fmt.Printf(" config sync: all %d resolvable (non-errored) config groups on this box resolved absent — refusing to remove any config files this pass; check the registry's account column and the deployment bucket\n", eligible) |
| + } |
| + |
| + for _, rt := range routes { |
| + if !execSyncable(rt) { |
| + continue |
| + } |
| + k := groupKey{release.RouteAccount(rt), rt.App, rt.Environment} |
| + // Insist the group was really resolved rather than taking the map's |
| + // zero value: a zero resolution reads as "absent", which on this |
| + // path REMOVES the file. The two loops filter on the same |
| + // execSyncable today, so this cannot fire — but if they ever |
| + // diverge, this turns a silent config deletion into a skip. |
| + r, ok := resolved[k] |
| + if !ok { |
| + continue |
| + } |
| + if r.err != nil { |
| + // Loud, per route, every pass: this route is running on a |
| + // stale config file until an operator fixes it. Its file is |
| + // left exactly as it is — nothing below this line runs for it. |
| + env := k.env |
| + if env == "" { |
| + env = "default" |
| + } |
| + fmt.Printf(" %s: config unresolved (environment %q), config file left as-is: %v\n", rt.Host, env, r.err) |
| + continue |
| + } |
| + if r.m == nil && skipRemovals { |
| + continue |
| + } |
| + // The bucket key whose absence would drive a removal, named in the |
| + // removal log so cause and effect land on one line. Always the |
| + // DEFAULT bundle's key: removal is reachable only from the absent |
| + // state, and only an unbound group can resolve absent (a bound one |
| + // resolves through LoadMerged, which errors on a missing bundle |
| + // rather than returning nothing). A bound group's key would be |
| + // dead in every branch that reads this. |
| + syncExecHost(rt, r.m, Key(k.acct, k.app), priv, dataDir, bounce) |
| + } |
| + // An empty route list is never authority to delete anything — |
| + // registry.Open silently creates an empty database on a wrong path or |
| + // an unmounted /data (same guard, same reason, as the unit-path Sync). |
| + if len(routes) == 0 { |
| + return nil |
| + } |
| + reapConfigEnv(routes, dataDir) |
| + return nil |
| +} |
| + |
| +// execSyncable: lifecycle-managed instances the ACTIVATOR spawns — and |
| +// therefore whose environment this sweep can actually deliver. Mirrors |
| +// the unit path's syncable() exclusions, then narrows to exec routes |
| +// with a socket upstream: addr-upstream routes are proxy-only (the |
| +// process behind them is externally managed) and unit routes belong to |
| +// the root adopt path. |
| +func execSyncable(rt registry.Route) bool { |
| + return rt.Kind == "instance" && rt.App != "" && !rt.Platform && |
| + (rt.Status == "" || rt.Status == "ready") && |
| + rt.Backing != registry.BackingUnit && rt.Socket != "" |
| +} |
| + |
| +// syncExecHost converges one route's config file. m == nil means the |
| +// manifest object is absent from the bucket: remove the file (sync owns |
| +// the namespace; a file with no manifest is a leftover, and leaving it |
| +// would keep dead config flowing into spawns). key is the default |
| +// bundle's key, Key(acct, app) — the only key whose absence can reach |
| +// this branch — included in the removal log line so cause (which bucket |
| +// key came back absent) and effect (which file got removed) land |
| +// together on one line, instead of requiring a reader to cross-reference |
| +// the fetch-side log. |
| +func syncExecHost(rt registry.Route, m *Manifest, key string, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error) { |
| + if rt.Host == "" || rt.Host == "." || rt.Host == ".." || rt.Host != filepath.Base(rt.Host) { |
| + fmt.Printf(" %q: not a bare hostname, refusing to derive a config file path from it\n", rt.Host) |
| + return |
| + } |
| + path := filepath.Join(dataDir, rt.Host+configEnvSuffix) |
| + if m == nil { |
| + if err := os.Remove(path); err != nil { |
| + if !os.IsNotExist(err) { |
| + fmt.Printf(" %s: remove config for %s (manifest absent): %v\n", rt.Host, key, err) |
| + } |
| + return |
| + } |
| + fmt.Printf(" %s: manifest %s absent, removed its config file\n", rt.Host, key) |
| + bounceLogged(rt.Host, bounce) |
| + return |
| + } |
| + cur, err := os.ReadFile(path) |
| + existed := err == nil |
| + if err != nil && !os.IsNotExist(err) { |
| + fmt.Printf(" %s: read %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + // Where sync.go has its defaultMissing/lostKeys guard, this path has |
| + // nothing — deliberately. That guard exists because /etc/carlos/apps |
| + // files may be HAND-STAGED, so a bound-environment-only merged view |
| + // dropping keys could delete config no one put in the bucket. This |
| + // file is sync-owned by construction: every byte in it was rendered by |
| + // a previous pass of this function from a manifest. A merged view that |
| + // drops keys is therefore correct convergence, not truncation — and |
| + // the instance's own <host>.env, the one file here that IS |
| + // hand-staged, is a separate file layered on top at spawn and never |
| + // read or written by this sweep. |
| + rendered, err := RenderEnv(m, priv) |
| + if err != nil { |
| + fmt.Printf(" %s: render config: %v\n", rt.Host, err) |
| + return |
| + } |
| + if existed && bytes.Equal(cur, rendered) { |
| + return // up to date — stay quiet on the periodic tick |
| + } |
| + tmp := path + ".tmp" |
| + if err := os.WriteFile(tmp, rendered, 0o600); err != nil { |
| + fmt.Printf(" %s: write %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + if err := os.Rename(tmp, path); err != nil { |
| + os.Remove(tmp) // don't leave rendered secrets behind |
| + fmt.Printf(" %s: rename %s: %v\n", rt.Host, path, err) |
| + return |
| + } |
| + fmt.Printf(" %s: config updated\n", rt.Host) |
| + bounceLogged(rt.Host, bounce) |
| +} |
| + |
| +// bounceLogged: bounce is best-effort — a failure is logged, never |
| +// propagated; the file on disk is already correct and the next natural |
| +// wake picks it up. |
| +func bounceLogged(host string, bounce func(host string) error) { |
| + if bounce == nil { |
| + return |
| + } |
| + if err := bounce(host); err != nil { |
| + fmt.Printf(" %s: bounce after config change: %v\n", host, err) |
| + } |
| +} |
| + |
| +// reapConfigEnv deletes *.config.env (and stale *.config.env.tmp) for |
| +// hosts with no route row at all. It matches ONLY sync's own suffix: |
| +// <host>.env and <host>-sidecar.env are other owners' files and are |
| +// never candidates, whatever their host's route state. |
| +func reapConfigEnv(routes []registry.Route, dataDir string) { |
| + ents, err := os.ReadDir(dataDir) |
| + if err != nil { |
| + return |
| + } |
| + live := make(map[string]bool, len(routes)) |
| + for _, rt := range routes { |
| + live[rt.Host] = true |
| + } |
| + for _, e := range ents { |
| + if e.IsDir() { |
| + continue |
| + } |
| + if strings.HasSuffix(e.Name(), configEnvSuffix+".tmp") { |
| + // A crashed pass's leftover holds decrypted secrets and is |
| + // stale by construction once reap runs. |
| + if err := os.Remove(filepath.Join(dataDir, e.Name())); err != nil { |
| + fmt.Printf(" %s: remove stale config temp file: %v\n", e.Name(), err) |
| + } |
| + continue |
| + } |
| + if !strings.HasSuffix(e.Name(), configEnvSuffix) { |
| + continue |
| + } |
| + host := strings.TrimSuffix(e.Name(), configEnvSuffix) |
| + if live[host] { |
| + continue |
| + } |
| + if err := os.Remove(filepath.Join(dataDir, e.Name())); err != nil { |
| + fmt.Printf(" %s: remove orphaned config file: %v\n", host, err) |
| + continue |
| + } |
| + fmt.Printf(" %s: no route on this box, removed its config file\n", host) |
| + } |
| +} |
diff --git a/internal/appconfig/syncexec_test.go b/internal/appconfig/syncexec_test.go| new file mode 100644 |
| index 000000000..1c863c076 |
| --- /dev/null |
| +++ b/internal/appconfig/syncexec_test.go |
| @@ -0,0 +1,569 @@ |
| +package appconfig |
| + |
| +import ( |
| + "context" |
| + "os" |
| + "path/filepath" |
| + "slices" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// execKit is the shared setup: store, registry, dataDir, keypair. |
| +func execKit(t *testing.T) (ctx context.Context, s release.Store, reg *registry.Registry, dataDir, priv, pub string) { |
| + t.Helper() |
| + s = release.NewDirStore(t.TempDir()) |
| + ctx = context.Background() |
| + reg = testRegistry(t) |
| + dataDir = t.TempDir() |
| + var err error |
| + priv, pub, err = GenerateKey() |
| + if err != nil { |
| + t.Fatalf("GenerateKey: %v", err) |
| + } |
| + return |
| +} |
| + |
| +func TestSyncExecWritesConfigFileAndBouncesOnChange(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, err := ParsePrivateKey(priv, "test") |
| + if err != nil { |
| + t.Fatalf("ParsePrivateKey: %v", err) |
| + } |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"LOG_LEVEL": "debug"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + var bounced []string |
| + bounce := func(h string) error { bounced = append(bounced, h); return nil } |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + got, err := os.ReadFile(filepath.Join(dd, "inst.example.com.config.env")) |
| + if err != nil { |
| + t.Fatalf("read config env: %v", err) |
| + } |
| + if string(got) != "LOG_LEVEL=debug\n" { |
| + t.Fatalf("config env = %q, want LOG_LEVEL=debug", got) |
| + } |
| + if fi, _ := os.Stat(filepath.Join(dd, "inst.example.com.config.env")); fi.Mode().Perm() != 0o600 { |
| + t.Fatalf("config env mode = %v, want 0600", fi.Mode().Perm()) |
| + } |
| + if len(bounced) != 1 || bounced[0] != "inst.example.com" { |
| + t.Fatalf("bounced = %v, want exactly [inst.example.com]", bounced) |
| + } |
| + // Second pass, nothing changed: no bounce. |
| + bounced = nil |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec (second): %v", err) |
| + } |
| + if len(bounced) != 0 { |
| + t.Fatalf("unchanged pass bounced %v, want none", bounced) |
| + } |
| +} |
| + |
| +func TestSyncExecRemovesFileWhenManifestDeleted(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + // A config file sync wrote earlier, whose manifest object is now gone: |
| + // only sync writes this namespace, so it must be removed — and the |
| + // running instance bounced, or it keeps stale config until next wake. |
| + path := filepath.Join(dd, "inst.example.com.config.env") |
| + if err := os.WriteFile(path, []byte("OLD=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(path); !os.IsNotExist(err) { |
| + t.Fatalf("config file survived manifest deletion, stat err = %v", err) |
| + } |
| + if len(bounced) != 1 { |
| + t.Fatalf("bounced = %v, want the stale-config instance bounced once", bounced) |
| + } |
| +} |
| + |
| +// TestSyncExecMassRemovalGuardRefusesWhenEveryGroupResolvesAbsent covers |
| +// F4's misconfiguration guard: two distinct apps' groups both resolve |
| +// "manifest absent" in the same pass (the shape a mangled registry account |
| +// column produces — release.RouteAccount falls back to "ops" for every |
| +// route at once, so every group's bucket key 404s together). Both |
| +// existing config files must survive, and neither instance is bounced — |
| +// an unguarded pass would otherwise wipe every config file on the box in |
| +// one sweep from what might be a registry fault, not a real deletion. |
| +func TestSyncExecMassRemovalGuardRefusesWhenEveryGroupResolvesAbsent(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + // Neither app ever had SetEnv called: both groups' manifest objects are |
| + // genuinely absent from the bucket. |
| + routes := []registry.Route{ |
| + {Host: "a.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/a.sock"}, |
| + {Host: "b.example.com", Kind: "instance", App: "titogo", Account: "bab", Status: "ready", Socket: "/run/carlos/b.sock"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + pathA := filepath.Join(dd, "a.example.com.config.env") |
| + pathB := filepath.Join(dd, "b.example.com.config.env") |
| + if err := os.WriteFile(pathA, []byte("OLD=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage a: %v", err) |
| + } |
| + if err := os.WriteFile(pathB, []byte("OLD=2\n"), 0o600); err != nil { |
| + t.Fatalf("stage b: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(pathA); err != nil { |
| + t.Fatalf("a's config file was removed despite the mass-removal guard: %v", err) |
| + } |
| + if _, err := os.Stat(pathB); err != nil { |
| + t.Fatalf("b's config file was removed despite the mass-removal guard: %v", err) |
| + } |
| + if len(bounced) != 0 { |
| + t.Fatalf("bounced = %v, want none — the guard must hold back both the removal and the bounce", bounced) |
| + } |
| +} |
| + |
| +// TestSyncExecMassRemovalGuardDoesNotBlockNormalConvergence covers the |
| +// guard's other edge: with two apps in the pass but only ONE absent, that |
| +// one's file is still removed normally — the guard only fires when EVERY |
| +// group is absent, so a real single-app deletion on a multi-app box must |
| +// keep converging. |
| +func TestSyncExecMassRemovalGuardDoesNotBlockNormalConvergence(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"LOG_LEVEL": "debug"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + // titogo's manifest is never set: absent, but eleven's is present, so |
| + // this pass is NOT unanimous. |
| + routes := []registry.Route{ |
| + {Host: "a.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/a.sock"}, |
| + {Host: "b.example.com", Kind: "instance", App: "titogo", Account: "bab", Status: "ready", Socket: "/run/carlos/b.sock"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + pathA := filepath.Join(dd, "a.example.com.config.env") |
| + pathB := filepath.Join(dd, "b.example.com.config.env") |
| + if err := os.WriteFile(pathB, []byte("OLD=2\n"), 0o600); err != nil { |
| + t.Fatalf("stage b: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(pathA); err != nil { |
| + t.Fatalf("a's config file wasn't written: %v", err) |
| + } |
| + if _, err := os.Stat(pathB); !os.IsNotExist(err) { |
| + t.Fatalf("b's config file survived its manifest's absence, stat err = %v", err) |
| + } |
| + // a's file is a fresh write (its manifest has real content, unlike |
| + // b's), so it bounces too — both changes converge normally. |
| + if !slices.Contains(bounced, "a.example.com") || !slices.Contains(bounced, "b.example.com") || len(bounced) != 2 { |
| + t.Fatalf("bounced = %v, want exactly [a.example.com b.example.com] (order not guaranteed)", bounced) |
| + } |
| +} |
| + |
| +// TestSyncExecRendersEachRoutesBoundEnvironment is the exec-path twin of |
| +// sync.go's TestSyncRendersEachRoutesBoundEnvironment: two instances of ONE |
| +// app, one bound to a named environment and one unbound, must land |
| +// DIFFERENT files in the same pass — the bound one gets the default bundle |
| +// overlaid by its environment (environment wins per key), the unbound one |
| +// gets the default bundle alone. Grouping on (acct, app) alone would render |
| +// whichever route came first into both. |
| +func TestSyncExecRendersEachRoutesBoundEnvironment(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, err := ParsePrivateKey(priv, "test") |
| + if err != nil { |
| + t.Fatalf("ParsePrivateKey: %v", err) |
| + } |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{ |
| + "LOG_LEVEL": "info", |
| + "BASE_ONLY": "1", |
| + }); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := SaveEnvironment(ctx, s, "bab", "eleven", "demo", map[string]Var{ |
| + "LOG_LEVEL": {Kind: "env", Value: "debug"}, // overrides the default |
| + "DEMO_ONLY": {Kind: "env", Value: "yes"}, // adds |
| + }, nil); err != nil { |
| + t.Fatalf("SaveEnvironment: %v", err) |
| + } |
| + routes := []registry.Route{ |
| + {Host: "demo.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/demo.sock", Environment: "demo"}, |
| + {Host: "plain.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/plain.sock"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + want := map[string]string{ |
| + "demo.example.com": "BASE_ONLY=1\nDEMO_ONLY=yes\nLOG_LEVEL=debug\n", |
| + "plain.example.com": "BASE_ONLY=1\nLOG_LEVEL=info\n", |
| + } |
| + for host, w := range want { |
| + got, err := os.ReadFile(filepath.Join(dd, host+".config.env")) |
| + if err != nil { |
| + t.Fatalf("read %s config env: %v", host, err) |
| + } |
| + if string(got) != w { |
| + t.Fatalf("%s config env = %q, want %q", host, got, w) |
| + } |
| + } |
| +} |
| + |
| +// TestSyncExecLeavesFileUntouchedWhenBoundEnvironmentIsMissing pins the |
| +// distinction that matters most on this path: a bound environment that is |
| +// not in the bucket is an ERROR, never absence. Absence removes the config |
| +// file here (sync owns the namespace) — so folding the two together would |
| +// strip a live instance's whole config, and bounce it onto the stripped |
| +// file, because someone deleted a bundle. The file must survive untouched |
| +// and the route must not be bounced. |
| +func TestSyncExecLeavesFileUntouchedWhenBoundEnvironmentIsMissing(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"LOG_LEVEL": "info"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "ghost.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/g.sock", Environment: "never-created", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + // The last good render, from before the bundle went missing. |
| + path := filepath.Join(dd, "ghost.example.com.config.env") |
| + const lastGood = "LOG_LEVEL=trace\n" |
| + if err := os.WriteFile(path, []byte(lastGood), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + got, err := os.ReadFile(path) |
| + if err != nil { |
| + t.Fatalf("config file removed on an unresolved environment: %v", err) |
| + } |
| + if string(got) != lastGood { |
| + t.Fatalf("config env = %q, want it left exactly as-is (%q)", got, lastGood) |
| + } |
| + if len(bounced) != 0 { |
| + t.Fatalf("bounced = %v, want none — an unresolved route is not a changed route", bounced) |
| + } |
| +} |
| + |
| +// TestSyncExecOverlayOnlyAppDropsStaleKeys pins the deliberate divergence |
| +// from the unit path: SyncExec has NO lostKeys/defaultMissing guard, and |
| +// must not grow one. |
| +// |
| +// This is the exact shape sync.go REFUSES (2026-08-02 review, CRITICAL 1): |
| +// a route bound to an environment whose app has no default config.json at |
| +// all, so the merged view is the overlay bundle ALONE, dropping a key the |
| +// file on disk already holds. On the unit path that refusal is right — |
| +// /etc/carlos/apps/<host>.env may have been hand-staged by an operator |
| +// before the app's config was ever imported into the bucket, so those keys |
| +// are not sync's to delete, and the documented migration step (create a |
| +// small override, bind the routes) would otherwise wipe them as root. |
| +// |
| +// Here it is wrong. Every byte of a .config.env was rendered by a previous |
| +// pass of SyncExec from a manifest — the namespace is sync-owned by |
| +// construction, nothing hand-staged can be in it — so a key the merged |
| +// view no longer contains is stale config that was removed from the |
| +// bucket, and dropping it IS the convergence. The instance's own |
| +// <host>.env, the one hand-staged file in this directory, is a separate |
| +// file layered on top at spawn and never read or written by this sweep. |
| +// |
| +// Import the guard and this test fails, which is the point. |
| +func TestSyncExecOverlayOnlyAppDropsStaleKeys(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + // No SetEnv for eleven: the app has NO default bundle, only an |
| + // environment overlay. STALE is in neither. |
| + if err := SaveEnvironment(ctx, s, "bab", "eleven", "demo", map[string]Var{ |
| + "KEPT": {Kind: "env", Value: "1"}, |
| + }, nil); err != nil { |
| + t.Fatalf("SaveEnvironment: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", Environment: "demo", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + // A file an earlier pass rendered, back when STALE was still in the |
| + // bundle. Sync wrote it, so sync gets to drop it. |
| + path := filepath.Join(dd, "inst.example.com.config.env") |
| + if err := os.WriteFile(path, []byte("KEPT=1\nSTALE=old\n"), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + got, err := os.ReadFile(path) |
| + if err != nil { |
| + t.Fatalf("read config env: %v", err) |
| + } |
| + if string(got) != "KEPT=1\n" { |
| + t.Fatalf("config env = %q, want the overlay ALONE (KEPT=1) — a key-loss guard must not be imported onto this path", got) |
| + } |
| + if len(bounced) != 1 || bounced[0] != "inst.example.com" { |
| + t.Fatalf("bounced = %v, want exactly one bounce for the dropped key", bounced) |
| + } |
| +} |
| + |
| +// TestSyncExecMassRemovalGuardIgnoresErroredGroups: an app whose bound |
| +// environment is missing is in neither the absent nor the present column, |
| +// so it must not sit in the guard's denominator and disarm it. Two |
| +// genuinely-absent groups plus one errored group is still "every resolvable |
| +// group is absent" — the guard fires, and the errored route's file is left |
| +// alone by the error path regardless. |
| +func TestSyncExecMassRemovalGuardIgnoresErroredGroups(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + // keymail has a default bundle but its bound environment is missing: |
| + // that group resolves to an error, not to absence. |
| + if err := SetEnv(ctx, s, "bab", "keymail", map[string]string{"K": "1"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + routes := []registry.Route{ |
| + {Host: "a.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/a.sock"}, |
| + {Host: "b.example.com", Kind: "instance", App: "titogo", Account: "bab", Status: "ready", Socket: "/run/carlos/b.sock"}, |
| + {Host: "c.example.com", Kind: "instance", App: "keymail", Account: "bab", Status: "ready", Socket: "/run/carlos/c.sock", Environment: "gone"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + for _, h := range []string{"a.example.com", "b.example.com", "c.example.com"} { |
| + if err := os.WriteFile(filepath.Join(dd, h+".config.env"), []byte("OLD=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage %s: %v", h, err) |
| + } |
| + } |
| + var bounced []string |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(h string) error { |
| + bounced = append(bounced, h) |
| + return nil |
| + }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + for _, h := range []string{"a.example.com", "b.example.com", "c.example.com"} { |
| + if _, err := os.Stat(filepath.Join(dd, h+".config.env")); err != nil { |
| + t.Fatalf("%s: config file removed despite the mass-removal guard: %v", h, err) |
| + } |
| + } |
| + if len(bounced) != 0 { |
| + t.Fatalf("bounced = %v, want none", bounced) |
| + } |
| +} |
| + |
| +func TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"A": "1"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + // The instance's own env file must never be read, changed, or reaped. |
| + own := filepath.Join(dd, "inst.example.com.env") |
| + if err := os.WriteFile(own, []byte("CARLOS_ADMIN_TOKEN=tok\n"), 0o600); err != nil { |
| + t.Fatalf("stage own env: %v", err) |
| + } |
| + routes := []registry.Route{ |
| + {Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/inst.sock"}, |
| + {Host: "unit.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/u.sock", Backing: registry.BackingUnit}, |
| + {Host: "proxy.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Addr: "127.0.0.1:8444"}, |
| + {Host: "alias.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "ready", Socket: "/run/carlos/a.sock", Platform: true}, |
| + {Host: "pending.example.com", Kind: "instance", App: "eleven", Account: "bab", Status: "provisioning", Socket: "/run/carlos/p.sock"}, |
| + } |
| + for _, rt := range routes { |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatalf("Upsert %s: %v", rt.Host, err) |
| + } |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if got, _ := os.ReadFile(own); string(got) != "CARLOS_ADMIN_TOKEN=tok\n" { |
| + t.Fatalf("instance env file was touched: %q", got) |
| + } |
| + if _, err := os.Stat(filepath.Join(dd, "inst.example.com.config.env")); err != nil { |
| + t.Fatalf("in-scope route got no config file: %v", err) |
| + } |
| + for _, h := range []string{"unit.example.com", "proxy.example.com", "alias.example.com", "pending.example.com"} { |
| + if _, err := os.Stat(filepath.Join(dd, h+".config.env")); !os.IsNotExist(err) { |
| + t.Fatalf("out-of-scope route %s got a config file, stat err = %v", h, err) |
| + } |
| + } |
| +} |
| + |
| +func TestSyncExecReapsOnlyItsOwnNamespace(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"X": "1"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "live.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/l.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + files := map[string]bool{ // name -> should survive |
| + "live.example.com.config.env": true, // has a route |
| + "gone.example.com.config.env": false, // no route: reaped |
| + "gone.example.com.config.env.tmp": false, // stale tmp: reaped |
| + "gone.example.com.env": true, // NOT ours: instance file, never reaped |
| + "gone.example.com-sidecar.env": true, // NOT ours: sidecar file, never reaped |
| + } |
| + for name := range files { |
| + if err := os.WriteFile(filepath.Join(dd, name), []byte("X=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage %s: %v", name, err) |
| + } |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + for name, keep := range files { |
| + _, err := os.Stat(filepath.Join(dd, name)) |
| + if keep && err != nil { |
| + t.Fatalf("%s: want kept, got %v", name, err) |
| + } |
| + if !keep && !os.IsNotExist(err) { |
| + t.Fatalf("%s: want reaped, stat err = %v", name, err) |
| + } |
| + } |
| +} |
| + |
| +func TestSyncExecEmptyRegistryRemovesNothing(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + path := filepath.Join(dd, "orphan.example.com.config.env") |
| + if err := os.WriteFile(path, []byte("X=1\n"), 0o600); err != nil { |
| + t.Fatalf("stage: %v", err) |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }); err != nil { |
| + t.Fatalf("SyncExec: %v", err) |
| + } |
| + if _, err := os.Stat(path); err != nil { |
| + t.Fatalf("empty registry removed a config file: %v", err) |
| + } |
| +} |
| + |
| +func TestSyncExecRefusesNonBareHostname(t *testing.T) { |
| + _, _, _, dd, priv, _ := execKit(t) // routes injected directly via the per-route helper below |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + m := &Manifest{Schema: 1, App: "eleven", Vars: map[string]Var{"A": {Kind: "env", Value: "1"}}} |
| + for _, host := range []string{"../escape", "a/b", "", ".", ".."} { |
| + syncExecHost(registry.Route{Host: host}, m, "apps/bab/eleven/config.json", pk, dd, func(string) error { return nil }) |
| + } |
| + ents, err := os.ReadDir(dd) |
| + if err != nil { |
| + t.Fatalf("read dataDir: %v", err) |
| + } |
| + if len(ents) != 0 { |
| + t.Fatalf("dataDir = %v, want nothing written for non-bare hostnames", ents) |
| + } |
| +} |
| + |
| +// TestSyncExecConvergesToEmptyFileWhenAllVarsUnset covers the lifecycle |
| +// F8 flagged as missing: a manifest object that still EXISTS but has had |
| +// every var unset must converge its route to an EMPTY file, not have the |
| +// file removed — removal is reserved for a manifest object that is |
| +// entirely ABSENT from the bucket (see syncExecHost's m == nil branch). |
| +// Blurring those two states would make "unset every var" behave like |
| +// deleting the manifest, which it structurally is not. |
| +func TestSyncExecConvergesToEmptyFileWhenAllVarsUnset(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, err := ParsePrivateKey(priv, "test") |
| + if err != nil { |
| + t.Fatalf("ParsePrivateKey: %v", err) |
| + } |
| + if err := SetEnv(ctx, s, "bab", "eleven", map[string]string{"LOG_LEVEL": "debug"}); err != nil { |
| + t.Fatalf("SetEnv: %v", err) |
| + } |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "inst.example.com", Kind: "instance", App: "eleven", Account: "bab", |
| + Status: "ready", Socket: "/run/carlos/inst.sock", |
| + }); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + path := filepath.Join(dd, "inst.example.com.config.env") |
| + |
| + var bounced []string |
| + bounce := func(h string) error { bounced = append(bounced, h); return nil } |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec (set): %v", err) |
| + } |
| + got, err := os.ReadFile(path) |
| + if err != nil { |
| + t.Fatalf("read config env: %v", err) |
| + } |
| + if string(got) != "LOG_LEVEL=debug\n" { |
| + t.Fatalf("config env = %q, want LOG_LEVEL=debug", got) |
| + } |
| + |
| + // Unset every var — the manifest object itself still exists (Unset |
| + // rewrites it with an empty Vars map; it does not delete the object). |
| + if _, err := Unset(ctx, s, "bab", "eleven", []string{"LOG_LEVEL"}); err != nil { |
| + t.Fatalf("Unset (all vars): %v", err) |
| + } |
| + bounced = nil |
| + if err := SyncExec(ctx, reg, s, pk, dd, bounce); err != nil { |
| + t.Fatalf("SyncExec (unset): %v", err) |
| + } |
| + got, err = os.ReadFile(path) |
| + if err != nil { |
| + t.Fatalf("config file removed instead of converged to empty: %v", err) |
| + } |
| + if string(got) != "" { |
| + t.Fatalf("config env = %q, want empty", got) |
| + } |
| + if len(bounced) != 1 || bounced[0] != "inst.example.com" { |
| + t.Fatalf("bounced = %v, want exactly one bounce for the change to empty", bounced) |
| + } |
| +} |
diff --git a/internal/host/adopt_advice.go b/internal/host/adopt_advice.go| new file mode 100644 |
| index 000000000..4e0a3691a |
| --- /dev/null |
| +++ b/internal/host/adopt_advice.go |
| @@ -0,0 +1,43 @@ |
| +// Adopt-pass advisories: warnings about registry rows whose claims |
| +// disagree with what systemd is actually running. Advisory only — the |
| +// pass never mutates a route to "fix" it; the operator decides (see |
| +// platform issue #14 for the kass incident that motivated this). |
| +package host |
| + |
| +import ( |
| + "log" |
| + "os/exec" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// warnBackingMismatch reports hosts whose route claims exec backing |
| +// ("") while a carlos-app@<host> unit is actively running — the state |
| +// where release adoption still works but config sync silently skips the |
| +// route. Returns the warned hosts; the caller logs them. |
| +func warnBackingMismatch(routes []registry.Route, unitActive func(host string) bool) []string { |
| + var warned []string |
| + for _, rt := range routes { |
| + if rt.Kind != "instance" || rt.Platform || rt.Backing == registry.BackingUnit { |
| + continue |
| + } |
| + if unitActive(rt.Host) { |
| + warned = append(warned, rt.Host) |
| + } |
| + } |
| + return warned |
| +} |
| + |
| +// unitIsActive: the real systemd probe for warnBackingMismatch. |
| +func unitIsActive(host string) bool { |
| + return exec.Command("systemctl", "is-active", "--quiet", "carlos-app@"+host+".service").Run() == nil |
| +} |
| + |
| +// adviseBackingMismatches runs the advisory against live systemd and |
| +// logs each finding — called from RunAdopt, root-side, where systemctl |
| +// is reachable and the hourly log is read. |
| +func adviseBackingMismatches(routes []registry.Route) { |
| + for _, h := range warnBackingMismatch(routes, unitIsActive) { |
| + log.Printf("adopt: route %s claims exec backing but carlos-app@%s is ACTIVE — stamp backing='unit' (see platform issue #14), then remove the stray /data/%s.config.env the exec loop materialized — the unit reads /etc/carlos/apps instead", h, h, h) |
| + } |
| +} |
diff --git a/internal/host/adopt_advice_test.go b/internal/host/adopt_advice_test.go| new file mode 100644 |
| index 000000000..8a11c8ef4 |
| --- /dev/null |
| +++ b/internal/host/adopt_advice_test.go |
| @@ -0,0 +1,25 @@ |
| +package host |
| + |
| +import ( |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// A route claiming exec backing while a carlos-app@<host> unit is ACTIVE |
| +// is exactly the #14 shape (kass): adopt keeps working, but config sync |
| +// silently skips the route. The adopt pass warns so the mismatch is |
| +// visible in the hourly log instead of discovered by absence. |
| +func TestWarnBackingMismatch(t *testing.T) { |
| + routes := []registry.Route{ |
| + {Host: "mismatch.example.com", Kind: "instance", App: "kass"}, // exec-claimed, unit active -> warn |
| + {Host: "honest-unit.example.com", Kind: "instance", App: "kass", Backing: registry.BackingUnit}, // consistent |
| + {Host: "honest-exec.example.com", Kind: "instance", App: "kass"}, // exec, no unit |
| + {Host: "alias.example.com", Kind: "instance", App: "kass", Platform: true}, // aliases never warn |
| + } |
| + active := map[string]bool{"mismatch.example.com": true, "honest-unit.example.com": true} |
| + got := warnBackingMismatch(routes, func(h string) bool { return active[h] }) |
| + if len(got) != 1 || got[0] != "mismatch.example.com" { |
| + t.Fatalf("warned = %v, want exactly [mismatch.example.com]", got) |
| + } |
| +} |
diff --git a/internal/host/agent.go b/internal/host/agent.go| index 07a1d6f55..9e42508c9 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -146,6 +146,7 @@ func RunAgent(args []string) { |
| go canaryAdoptLoop(reg) |
| go statusLoop(reg) |
| go domainsLoop(reg, peekAcmeCache(args)) |
| + go configSyncLoop(reg, svc, dd) |
| // The same serving loop as `carlos edge`, with the activator injected. |
| // Caching applies here too: cached hits still avoid the activator, which |
| @@ -210,6 +211,52 @@ func adoptLoop(reg *registry.Registry) { |
| } |
| } |
| +// configSyncLoop delivers app config to exec-backed routes: hourly (plus |
| +// a first pass ~1min after boot, like adoptLoop), materialize every |
| +// exec-syncable route's <dataDir>/<host>.config.env from the bucket and |
| +// bounce running instances whose config changed. Runs IN-PROCESS on |
| +// purpose — /data is inside this unit's sandbox and the exec children |
| +// are this process's own, so it can both write the files and restart |
| +// the readers; the root adopt path keeps the unit-backed half |
| +// (/etc/carlos/apps needs root). Inert until CARLOS_SECRETS_KEY is set, |
| +// the same convention as every other optional loop. |
| +func configSyncLoop(reg *registry.Registry, svc *activator.Service, dd string) { |
| + raw := os.Getenv("CARLOS_SECRETS_KEY") |
| + if raw == "" { |
| + log.Printf("agent: config sync loop disabled: CARLOS_SECRETS_KEY not set") |
| + return |
| + } |
| + priv, err := appconfig.ParsePrivateKey(raw, "CARLOS_SECRETS_KEY") |
| + if err != nil { |
| + log.Printf("agent: config sync loop disabled: %v", err) |
| + return |
| + } |
| + ctx := context.Background() |
| + store, err := openDeployStore(ctx) |
| + if err != nil { |
| + log.Printf("agent: config sync loop disabled: %v", err) |
| + return |
| + } |
| + bounce := func(host string) error { |
| + if svc == nil { |
| + return nil // no activator: files still converge for future spawns |
| + } |
| + if svc.Bounce(host) { |
| + log.Printf("agent: config changed, bounced %s", host) |
| + } |
| + return nil |
| + } |
| + log.Printf("agent: config sync loop armed (hourly + jitter)") |
| + delay := time.Minute |
| + for { |
| + time.Sleep(delay) |
| + delay = time.Hour + rand.N(10*time.Minute) |
| + if err := appconfig.SyncExec(ctx, reg, store, priv, dd, bounce); err != nil { |
| + log.Printf("agent: config sync: %v", err) |
| + } |
| + } |
| +} |
| + |
| // canaryAdoptLoop mirrors adoptLoop but sweeps only routes following a |
| // canary/* channel, every few seconds instead of hourly — "I promoted a |
| // canary, why is my box still on the old version for the next 40 minutes" |
| @@ -604,11 +651,15 @@ func openDeployStore(ctx context.Context) (release.Store, error) { |
| // RunAdopt is the `carlos adopt` verb: one immediate adoption pass — the |
| // carlos-adopt.timer unit's ExecStart, and an operator's "converge now". |
| -// Runs as root (see carlos-adopt.service): this is the ONLY place config |
| -// sync may run, because it writes /etc/carlos/apps and restarts units — |
| -// the same filesystem-write shape release staging needs root for, and |
| -// exactly what the sandboxed carlos-edge.service's in-process adopt loop |
| -// (ProtectSystem=strict, ReadWritePaths=/data /run/carlos only) cannot do. |
| +// Runs as root (see carlos-adopt.service): this is the ONLY place the |
| +// UNIT-BACKED half of config sync (appconfig.Sync) may run, because it |
| +// writes /etc/carlos/apps and restarts units — the same filesystem-write |
| +// shape release staging needs root for, and exactly what the sandboxed |
| +// carlos-edge.service's in-process adopt loop (ProtectSystem=strict, |
| +// ReadWritePaths=/data /run/carlos only) cannot do. The EXEC-BACKED half |
| +// (appconfig.SyncExec) runs elsewhere, in the agent's in-process |
| +// configSyncLoop: it needs neither root nor /etc access, because it only |
| +// ever writes under /data and bounces its own children. |
| func RunAdopt(args []string) { |
| fs := flag.NewFlagSet("adopt", flag.ExitOnError) |
| regPath := fs.String("registry", filepath.Join(dataDir(), "registry.db"), "registry database") |
| @@ -643,4 +694,8 @@ func RunAdopt(args []string) { |
| if err := release.Pass(ctx, reg, store, pub, release.OptDir(), restartRoute); err != nil { |
| log.Fatalf("adopt: %v", err) |
| } |
| + |
| + if routes, err := reg.List(); err == nil { |
| + adviseBackingMismatches(routes) |
| + } |
| } |
diff --git a/internal/registry/registry_test.go b/internal/registry/registry_test.go| index 7a943f0fd..aba9dc0b4 100644 |
| --- a/internal/registry/registry_test.go |
| +++ b/internal/registry/registry_test.go |
| @@ -199,6 +199,23 @@ func TestSetVersionsAndHibernateAreNarrow(t *testing.T) { |
| } |
| } |
| +// Backing must survive an Upsert/List round trip — the #14 incident was |
| +// a route whose backing stamp went missing, and everything that branches |
| +// on rt.Backing (activator dispatch, config sync scope) reasons from it. |
| +func TestUpsertListRoundTripsBacking(t *testing.T) { |
| + r := openTestRegistry(t) |
| + if err := r.Upsert(Route{Host: "u.example.com", Kind: "instance", App: "a", Backing: BackingUnit}); err != nil { |
| + t.Fatalf("Upsert: %v", err) |
| + } |
| + routes, err := r.List() |
| + if err != nil { |
| + t.Fatalf("List: %v", err) |
| + } |
| + if len(routes) != 1 || routes[0].Backing != BackingUnit { |
| + t.Fatalf("routes = %+v, want one route with Backing=unit", routes) |
| + } |
| +} |
| + |
| // openTestRegistry opens a fresh registry under a per-test temp dir — the |
| // shared open helper for tests that don't care about the path. |
| func openTestRegistry(t *testing.T) *Registry { |