carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Tenant Email, Plan 4 of 5: The per-instance config layer and the delivery ack
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Give an app a third config bundle, `apps/<acct>/<app>/instances-config/<host>.json`, resolved after the named environment, so one instance can be handed a value none of its siblings ever see, and make the agent say out loud which version of that layer it rendered: `ConfigAck` gains `Schema` and `InstanceConfigETag`, and the agent rewrites the ack on every pass rather than only when a file changes, so a console can tell "this box has not got the layer yet" apart from "this box has nothing to render". Nothing in this plan writes an overlay for anyone: there is no CLI verb, no console door, and no email change. Step 5 is the layer's first caller.
6
7**Architecture:** `internal/appconfig` gains `instances.go`: the key, the host rule, the CAS read/write/delete, `LoadMergedHost`, and the two functions the sweeps reach the same answer through (`instanceOverlays`, one prefix listing per app, and `applyInstanceOverlay`, the pure merge). `Sync` and `SyncExec` keep their group caches exactly as they are — the group is still (Account, App, Environment) and the default bundle is still read once per (account, app) — and each route's own merge happens in the materialization loop, so a pass costs one `List` per app plus one `Get` per route that actually has an overlay. Four guards written against a two-layer merge are revisited one per task, each with the case that would have been wrong: `lostKeys`/`defaultMissing` (`sync.go:260`, `:388`), the mass-removal guard (`syncexec.go:341-380`), `AwaitingRestart` (`syncexec.go:771`) and `allShadowed` (`syncexec.go:872`). `ConfigAck` grows two fields, `Schema` and `InstanceConfigETag`, and keeps its single change-only writer. An earlier draft had the agent rewrite the ack on every pass so that absence became decidable; the spec now rules that out (§6.5), because it is a Get and a Put per converged route per pass on every box, and on a fleet-door box `control/config-ack/` is outside both the read and the write scope (`internal/edge/storedoor.go:389`, `:489`), so every one of those passes would fail, log, and still leave the console with nothing. Delivery stays decidable where it matters: writing an instance's credential always changes that instance's file, so it always produces an ack.
8
9**Tech Stack:** Go 1.26, `internal/appconfig` (bundles, merge, the two sync sweeps, the ack), `internal/host` (the agent's two config loops and `configAckFunc`), `internal/release` (`Store`, `NewDirStore`, `AppPrefix`, `ErrCASConflict`), `internal/registry` (route rows), `internal/console` (reads acks; unchanged by this plan).
10
11**Spec:** `docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md` — §11 step 4 is the whole mandate: §6.2 (the per-instance config layer) and §6.5 (confirming delivery). Everything else in §6 is step 5's: no instance senders, no `PerInstance` transition, no instance email doors, no `svcrole.Mail`, no uid check, no `claimDomain` change. §5 and §7 landed in plans 2 and 3 and are not touched.
12
13**`ExecDelivery` is not read from this layer, and that is the existing ruling, not a new one.** `appconfig.go:44-71` records why the flag is bundle-level and why per-route was refused: "a per-route flag would need a registry column and — since a Provisioned route's Channel and Environment belong to its instance record — a record field and reconcile convergence too: three doors onto one bit", and "the config-that-lies hazard the original ruling named is real at that granularity, where two routes resolving through the SAME bundle could disagree about whether it is delivered" (`:66-71`). A per-HOST bundle is exactly that granularity. So `resolveExecDelivery` (`syncexec.go:688`) is never handed the instance layer, `checkOptedIn` never reads it, and an `exec_delivery` field that somehow appears in one of these objects is inert — pinned by a test in task 3. The layer carries vars and sealed secrets, nothing else.
14
15**The sweeps do not literally call `LoadMergedHost`, and that is deliberate.** `LoadMergedHost(ctx, s, acct, app, env, host)` is the single-shot API — the one function anything outside this package asks "what does this host actually get" — and it fetches all three layers itself. Calling it per route inside `Sync`/`SyncExec` would re-fetch `config.json` and the named bundle once per route and throw away the group caches those loops exist for: a box with forty routes of one app would go from two bucket reads to eighty. The loops instead reach the same answer through the same two pieces `LoadMergedHost` is built from — `instanceOverlays` for the per-app listing and `applyInstanceOverlay` for the merge — and task 3 pins the two paths against each other with a test that resolves one host both ways and compares.
16
17## Global Constraints
18
19- **Run `gofmt -w` on every file you touched before running the gate.** `gofmt -l .` only NAMES an unformatted file; it fixes nothing. Several blocks below are pasted with aligned trailing comments (task 1's key constants, task 9's `ConfigAck` fields) that gofmt will realign, so a plan followed literally goes red on the gate's last step unless `gofmt -w` ran first.
20- **The gate before every commit:** `go build ./... && go vet ./... && go test ./... -count=1` all clean, and `gofmt -l .` empty over the whole tree. Env: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off` — Go may need `GOCACHE` pointed somewhere writable, and "read-only file system" from the toolchain is that tell. Socket-binding tests need `TMPDIR=/tmp/g`.
21- **This repo lands through `amadan branch merge`, never a GitHub PR.** Branch `worktree-tenant-email-senders`. CI is `.amadan/ci.d/`, triggered by a push to the `amadan` remote; a branch pushed only to `origin` has been gated by nothing.
22- **No new capability for a member, and no email change.** This plan ships alone. After every task an app that gets config today gets the same bytes, in the same file, with the same restart behaviour, because nothing writes an overlay: the tests are the only writers in the tree when this plan ends.
23- **Wire-shape changes are additive.** Live `ConfigAck` objects exist in production buckets, written by agents that have never heard of `Schema`, and must still parse. `Schema == 0` therefore means "an agent older than this release" and is a first-class state, never an error.
24- **Absence is never a removal.** Every branch this plan touches in the two sweeps has the same rule the existing code has: a fetch failure leaves the file exactly as it is, and only a determination can remove one. The overlay adds a second way a route can be configured, so it also adds a second way an "absent" verdict can be wrong — tasks 5 and 7 are that, and nothing else.
25- Copy: no em dashes. Full stop and a new sentence, lower case. Comments are full sentences saying why, never process history; log lines use `->` between values.
26- No member-facing strings are added by this plan, so no `copy-review` gate is due. If a task grows one, run it.
27- Commit after every task with the trailer `Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>`.
28
29---
30
31## File Structure
32
33| File | Responsibility |
34|---|---|
35| `internal/appconfig/instances.go` (create) | `InstancesConfigPrefix`, `InstanceConfigKey`, `ValidInstanceHost`, `LoadInstanceConfig`, `ApplyInstance`, `DeleteInstanceConfig`, `LoadMergedHost`, `applyInstanceOverlay`, `instanceOverlays` |
36| `internal/appconfig/instances_test.go` (create) | the key and host rules, the CAS write, the generation bump, three-layer precedence, the sweeps-vs-`LoadMergedHost` equivalence |
37| `internal/appconfig/store.go` (modify) | `getIfExistsAtETag` (the etag-returning read `getIfExistsAt` delegates to); `applyAt` returns the new etag |
38| `internal/appconfig/sync.go` (modify) | the per-app overlay index, the per-route merge, `defaultMissing` extended to the overlay-alone case |
39| `internal/appconfig/syncexec.go` (modify) | the same index and merge; absent base with an overlay is not a removal; the mass-removal guard; `AwaitingRestart` and `allShadowed` |
40| `internal/appconfig/configack.go` (modify) | `Schema`, `InstanceConfigETag`, `ConfigAckSchema`, `ConfigAckPrefix`, `LatestConfigAck`, `InstanceConfigDelivery` |
41| `internal/appconfig/routefingerprint.go` (modify) | doc only: the layer is bucket-side, so the generation bump is its signal |
42| `internal/host/agent.go` (modify) | `configAckFunc` stamps `ConfigAckSchema` on every ack it writes |
43
44---
45
46### Task 1: The layer's key, its host rule and its reader
47
48**Files:**
49- Create: `internal/appconfig/instances.go`, `internal/appconfig/instances_test.go`
50- Modify: `internal/appconfig/store.go` (`getIfExistsAt` at `:54`)
51
52**Interfaces:**
53- Consumes: `release.AppPrefix` (`internal/release/release.go:212`), `release.ValidAccount` (`internal/release/qualified.go:53`), `release.ValidApp` (`internal/release/release.go:240`), `Manifest` (`appconfig.go:21`), `getIfExistsAt` (`store.go:54`).
54- Produces:
55 ```go
56 func InstancesConfigPrefix(acct, app string) string // apps/<acct>/<app>/instances-config/
57 func InstanceConfigKey(acct, app, host string) string // .../instances-config/<host>.json
58 func ValidInstanceHost(host string) bool
59 func LoadInstanceConfig(ctx context.Context, s release.Store, acct, app, host string) (*Manifest, string, bool, error) // manifest, etag, found
60 func getIfExistsAtETag(ctx context.Context, s release.Store, key string) (*Manifest, string, bool, error)
61 ```
62
63- [ ] **Step 1: Write the failing test**
64
65`internal/appconfig/instances_test.go` (create):
66
67```go
68package appconfig
69
70import (
71 "context"
72 "encoding/json"
73 "strings"
74 "testing"
75
76 "github.com/carlosframework/platform/internal/release"
77)
78
79// putInstanceRaw writes an instance-layer object without going through the
80// package's own writer, which does not exist until task 2. Tests in this
81// file that only READ the layer use it, so task 1 can be verified on its
82// own.
83func putInstanceRaw(t *testing.T, ctx context.Context, s release.Store, acct, app, host string, m *Manifest) string {
84 t.Helper()
85 body, err := json.MarshalIndent(m, "", " ")
86 if err != nil {
87 t.Fatalf("marshal instance manifest: %v", err)
88 }
89 etag, err := s.Put(ctx, InstanceConfigKey(acct, app, host), body, "")
90 if err != nil {
91 t.Fatalf("put instance manifest: %v", err)
92 }
93 return etag
94}
95
96// The layer sits under the app's own prefix, beside config.json and
97// environments/, so release.PurgeApp's prefix sweep reaches it and a purged
98// app leaves no instance overlay behind holding sealed secrets.
99func TestInstanceConfigKeysSitUnderTheAppsOwnPrefix(t *testing.T) {
100 if got, want := InstancesConfigPrefix("bab", "titogo"), "apps/bab/titogo/instances-config/"; got != want {
101 t.Fatalf("InstancesConfigPrefix = %q, want %q", got, want)
102 }
103 if got, want := InstanceConfigKey("bab", "titogo", "book.customer.example.com"),
104 "apps/bab/titogo/instances-config/book.customer.example.com.json"; got != want {
105 t.Fatalf("InstanceConfigKey = %q, want %q", got, want)
106 }
107}
108
109// A host becomes a bucket key segment here, so the one gate every caller
110// passes through is where a separator, a traversal or an empty name is
111// refused. The shape is a bare lowercase hostname, the same thing a route
112// row's Host column holds and the same thing sync.go's own path guard
113// (rt.Host != filepath.Base(rt.Host)) already insists on before it derives
114// a file name from one.
115func TestValidInstanceHostRefusesAnythingThatIsNotABareHost(t *testing.T) {
116 for _, bad := range []string{
117 "", ".", "..", "a/b", "../etc", "Book.example.com", "book..example.com",
118 "-lead.example.com", "trail-.example.com", "book.example.com/",
119 strings.Repeat("a", 254),
120 } {
121 if ValidInstanceHost(bad) {
122 t.Errorf("ValidInstanceHost(%q) = true", bad)
123 }
124 }
125 for _, good := range []string{"book.customer.example.com", "inst1", "a-b.c-d.example.com", "127-0-0-1.example.com"} {
126 if !ValidInstanceHost(good) {
127 t.Errorf("ValidInstanceHost(%q) = false", good)
128 }
129 }
130}
131
132// The reader answers three ways, and the caller needs all three apart: a
133// manifest and the etag it was read at, "no object" without an error (an
134// instance with no overlay is the normal case, not a fault), and an error
135// for anything else. The etag is what the delivery ack carries (design
136// §6.5), so a read that dropped it would leave the agent unable to say
137// which version of the layer it rendered.
138func TestLoadInstanceConfigReadsTheLayerAndItsETag(t *testing.T) {
139 ctx := context.Background()
140 s := release.NewDirStore(t.TempDir())
141
142 m, etag, found, err := LoadInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com")
143 if err != nil || found || m != nil || etag != "" {
144 t.Fatalf("absent overlay = %+v %q %v %v, want (nil, \"\", false, nil)", m, etag, found, err)
145 }
146
147 want := putInstanceRaw(t, ctx, s, "bab", "titogo", "book.customer.example.com",
148 &Manifest{Schema: 1, App: "titogo", Vars: map[string]Var{"TENANT_SMTP_USER": {Kind: "env", Value: "book"}}})
149 m, etag, found, err = LoadInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com")
150 if err != nil || !found || m == nil {
151 t.Fatalf("present overlay = %+v %v %v", m, found, err)
152 }
153 if etag == "" || etag != want {
154 t.Fatalf("etag = %q, want the object's own %q", etag, want)
155 }
156 if m.Vars["TENANT_SMTP_USER"].Value != "book" {
157 t.Fatalf("overlay lost its var: %+v", m.Vars)
158 }
159
160 // The same newer-schema refusal every other bundle read makes: acting on
161 // a manifest whose unknown fields json.Unmarshal has already dropped
162 // would silently discard whatever they meant.
163 putInstanceRaw(t, ctx, s, "bab", "titogo", "newer.example.com", &Manifest{Schema: 2, App: "titogo"})
164 if _, _, _, err := LoadInstanceConfig(ctx, s, "bab", "titogo", "newer.example.com"); err == nil {
165 t.Fatal("a schema-2 overlay was accepted, want a refusal naming the upgrade")
166 }
167
168 // A host that is not a bare hostname never reaches the bucket at all.
169 if _, _, _, err := LoadInstanceConfig(ctx, s, "bab", "titogo", "../config"); err == nil {
170 t.Fatal("LoadInstanceConfig accepted a traversal host")
171 }
172}
173```
174
175- [ ] **Step 2: Run it to verify it fails**
176
177```bash
178go test -run 'TestInstanceConfigKeys|TestValidInstanceHost|TestLoadInstanceConfig' ./internal/appconfig/
179```
180
181Expected: build failure, `undefined: InstancesConfigPrefix`.
182
183- [ ] **Step 3: Write the code**
184
185`internal/appconfig/instances.go` (create):
186
187```go
188// The instance config layer (tenant-email senders design §6.2): a third
189// bundle, one object per route host, resolved after the named environment.
190// Same Manifest shape, same sealed envelopes, same name and value rules as
191// the other two — a per-instance bundle is not a new format, just a third
192// bucket key the same shape is written to.
193//
194// Bucket layout addition:
195//
196// apps/<acct>/<app>/instances-config/<host>.json CAS-guarded, schema-1 manifest
197//
198// It exists so a credential can be scoped to ONE instance. #148's shape is
199// one instance per customer under a single app, and everything the platform
200// delivers today lands in a bundle every instance of the app renders, so a
201// per-customer secret delivered through the default bundle reaches every
202// other customer's process.
203//
204// Manifest.ExecDelivery is NOT read from this layer. That is the ruling at
205// appconfig.go:66-71, unchanged: a per-route delivery flag would need a
206// registry column, a record field and reconcile convergence, and two routes
207// resolving through the same bundle could disagree about whether they are
208// delivered. A per-host bundle is exactly the granularity that ruling
209// refused, so resolveExecDelivery is never handed one of these manifests
210// and a field that somehow appears in one is inert.
211package appconfig
212
213import (
214 "context"
215 "fmt"
216 "regexp"
217 "strings"
218
219 "github.com/carlosframework/platform/internal/release"
220)
221
222// InstancesConfigPrefix is the list prefix behind "which instances of this
223// app have an overlay" — every instances-config/<host>.json key shares it.
224// Inside apps/<acct>/<app>/ on purpose: release.PurgeApp's prefix sweep
225// reaches it, so a purged app cannot leave an object full of sealed
226// secrets behind.
227func InstancesConfigPrefix(acct, app string) string {
228 return release.AppPrefix(acct, app) + "instances-config/"
229}
230
231// InstanceConfigKey is the bucket key of one instance's overlay. host is a
232// route's Host column verbatim, which is what makes the key derivable from
233// a registry row with nothing to look up.
234func InstanceConfigKey(acct, app, host string) string {
235 return InstancesConfigPrefix(acct, app) + host + ".json"
236}
237
238// instanceHostRe is a bare lowercase hostname: labels of letters, digits
239// and hyphens, joined by dots, never starting or ending with a hyphen or a
240// dot. Case-restricted like ValidEnvironmentName and for the same reason,
241// with one more: this string is also the file name sync derives on the box
242// (sync.go's filepath.Base guard), so one convention keeps a bucket key and
243// a file name from disagreeing about which host they belong to.
244var instanceHostRe = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$`)
245
246// ValidInstanceHost reports whether host is usable as an instance overlay
247// key. A separator or a traversal is refused at the one gate every caller
248// passes through, because this string is concatenated into a bucket key: a
249// host of "../config" would otherwise name the app's default bundle.
250func ValidInstanceHost(host string) bool {
251 if host == "" || len(host) > 253 || strings.Contains(host, "..") {
252 return false
253 }
254 // Per-label bound as well as the whole-name one, matching DNS and
255 // launchctx/controlname's label rule. Without it a single 200-character
256 // label passes here and is refused everywhere else that handles the same
257 // host, which is a disagreement between two gates rather than one rule.
258 for _, label := range strings.Split(host, ".") {
259 if len(label) > 63 {
260 return false
261 }
262 }
263 return instanceHostRe.MatchString(host)
264}
265
266// LoadInstanceConfig reads one instance's overlay, returning the manifest,
267// the etag it was read at, and whether the object exists at all.
268//
269// The etag is the point of this signature. The delivery ack (design §6.5)
270// carries the etag the agent actually rendered, and the console compares it
271// with the one its own Put returned: that comparison is decidable, where a
272// map of delivered key names cannot tell an old agent apart from a new one
273// rendering a route with no overlay vars.
274//
275// Not-found is (nil, "", false, nil): an instance with no overlay is the
276// normal case and the state every instance is in until something writes one.
277func LoadInstanceConfig(ctx context.Context, s release.Store, acct, app, host string) (*Manifest, string, bool, error) {
278 if !release.ValidAccount(acct) {
279 return nil, "", false, fmt.Errorf("config: invalid account %q", acct)
280 }
281 if !release.ValidApp(app) {
282 return nil, "", false, fmt.Errorf("config: invalid app %q", app)
283 }
284 if !ValidInstanceHost(host) {
285 return nil, "", false, fmt.Errorf("config: invalid instance host %q", host)
286 }
287 return getIfExistsAtETag(ctx, s, InstanceConfigKey(acct, app, host))
288}
289```
290
291`internal/appconfig/store.go`, replacing `getIfExistsAt` (`:49-73`) with an etag-returning read and a wrapper that keeps every existing caller's signature:
292
293```go
294// getIfExistsAt is getIfExists generalized to any manifest key — Key(acct,
295// app) for the default bundle, EnvironmentKey(acct, app, name) for a named
296// one (see environments.go), InstanceConfigKey(acct, app, host) for an
297// instance overlay (instances.go). Same shape either way: all three are the
298// identical manifest format at sibling bucket paths, so one read path
299// serves them.
300func getIfExistsAt(ctx context.Context, s release.Store, key string) (*Manifest, bool, error) {
301 m, _, found, err := getIfExistsAtETag(ctx, s, key)
302 return m, found, err
303}
304
305// getIfExistsAtETag is getIfExistsAt plus the object's etag, which the
306// instance layer's reader needs and the other two bundles' readers do not:
307// the delivery ack names the etag of the overlay the agent rendered
308// (senders design §6.5), and an etag dropped here could not be recovered
309// later without a second read racing the first.
310func getIfExistsAtETag(ctx context.Context, s release.Store, key string) (*Manifest, string, bool, error) {
311 raw, etag, err := s.Get(ctx, key)
312 if err != nil {
313 if errors.Is(err, os.ErrNotExist) {
314 return nil, "", false, nil
315 }
316 return nil, "", false, err
317 }
318 var m Manifest
319 if err := json.Unmarshal(raw, &m); err != nil {
320 return nil, "", false, fmt.Errorf("config %s: %w", key, err)
321 }
322 // json.Unmarshal has already dropped any fields this binary doesn't
323 // know about, so acting on a newer-schema manifest would silently
324 // discard whatever those fields meant. Refuse instead.
325 if m.Schema > 1 {
326 return nil, "", false, fmt.Errorf("config %s: schema %d is newer than this binary understands — upgrade carlos", key, m.Schema)
327 }
328 return &m, etag, true, nil
329}
330```
331
332- [ ] **Step 4: Run it to verify it passes**
333
334```bash
335go test ./internal/appconfig/ -count=1
336```
337
338Expected: PASS, including every existing bundle test unchanged.
339
340- [ ] **Step 5: Gate and commit**
341
342```bash
343gofmt -w internal/appconfig/instances.go internal/appconfig/instances_test.go internal/appconfig/store.go
344go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
345git add internal/appconfig
346git commit -m "appconfig: the instance config layer's key, host rule and etag-returning reader (plan 4, task 1)"
347```
348
349---
350
351### Task 2: Writing and deleting the layer, and the generation bump
352
353**Files:**
354- Modify: `internal/appconfig/instances.go`, `internal/appconfig/instances_test.go`, `internal/appconfig/store.go` (`applyAt` at `:146`, `apply` at `:115`), `internal/appconfig/environments.go` (`applyEnvironment` at `:108`)
355
356**Interfaces:**
357- Consumes: `applyAt` (`store.go:146`), `touchLogged` (`generation.go:107`), `GenKey` (`generation.go:51`).
358- Produces:
359 ```go
360 func ApplyInstance(ctx context.Context, s release.Store, acct, app, host string, sets map[string]Var, unsets []string) (removed int, etag string, err error)
361 func DeleteInstanceConfig(ctx context.Context, s release.Store, acct, app, host string) error
362 func applyAt(ctx context.Context, s release.Store, key, app, bundle string, sets map[string]Var, unsets []string) (int, string, error) // now returns the new etag
363 ```
364
365- [ ] **Step 1: Write the failing test**
366
367`internal/appconfig/instances_test.go` (append):
368
369```go
370// The write door. It returns the etag it wrote at, which is the whole
371// mechanism behind the delivery check (design §6.5): the caller keeps that
372// string and compares it with what an agent's ack says it rendered.
373func TestApplyInstanceWritesTheLayerAndReturnsItsETag(t *testing.T) {
374 ctx := context.Background()
375 s := release.NewDirStore(t.TempDir())
376
377 removed, etag, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com",
378 map[string]Var{"TENANT_SMTP_USER": {Kind: "env", Value: "book"}}, nil)
379 if err != nil {
380 t.Fatalf("ApplyInstance: %v", err)
381 }
382 if removed != 0 {
383 t.Fatalf("removed = %d on a create, want 0", removed)
384 }
385 if etag == "" {
386 t.Fatal("ApplyInstance returned an empty etag, so nothing can check delivery against it")
387 }
388 m, readETag, found, err := LoadInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com")
389 if err != nil || !found {
390 t.Fatalf("LoadInstanceConfig after the write = %+v %v %v", m, found, err)
391 }
392 if readETag != etag {
393 t.Fatalf("the reader saw etag %q, the writer returned %q — the delivery check compares these two", readETag, etag)
394 }
395 if m.Schema != 1 || m.App != "titogo" || m.Vars["TENANT_SMTP_USER"].Value != "book" {
396 t.Fatalf("the written overlay is not a schema-1 manifest of this app: %+v", m)
397 }
398
399 // A second write moves the etag, which is what makes a stale ack
400 // detectable at all.
401 _, etag2, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com",
402 map[string]Var{"TENANT_SMTP_USER": {Kind: "env", Value: "book2"}}, nil)
403 if err != nil {
404 t.Fatalf("ApplyInstance (second): %v", err)
405 }
406 if etag2 == etag {
407 t.Fatalf("etag did not move on a changed write: %q", etag2)
408 }
409
410 // Unset reports how many of the keys were actually there, the same
411 // contract every other bundle's unset has.
412 removed, _, err = ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com", nil,
413 []string{"TENANT_SMTP_USER", "NEVER_SET"})
414 if err != nil || removed != 1 {
415 t.Fatalf("unset = %d %v, want 1 and no error", removed, err)
416 }
417
418 // The var-name and var-value rules are the bundle's, not this layer's:
419 // applyAt refuses before anything reaches the bucket.
420 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com",
421 map[string]Var{"not a name": {Kind: "env", Value: "x"}}, nil); err == nil {
422 t.Fatal("an invalid var name was accepted into the instance layer")
423 }
424 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "../config", nil, nil); err == nil {
425 t.Fatal("ApplyInstance accepted a traversal host")
426 }
427}
428
429// An unset against an overlay that does not exist is refused rather than
430// minting an empty one, the same refusal applyAt already makes for the
431// other two bundles and for the same reason: an empty manifest that EXISTS
432// means "configured, then fully unset" by this package's contract, which is
433// a state the sweeps converge a file to.
434func TestApplyInstanceRefusesUnsetOnlyAgainstAMissingOverlay(t *testing.T) {
435 ctx := context.Background()
436 s := release.NewDirStore(t.TempDir())
437 _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com", nil, []string{"K"})
438 if err == nil {
439 t.Fatal("unsetting from a nonexistent overlay was turned into a create")
440 }
441 if !strings.Contains(err.Error(), "book.customer.example.com") {
442 t.Fatalf("the refusal does not name the instance: %v", err)
443 }
444}
445
446// Writing the layer bumps the app's generation object, the way
447// SetExecDeliveryAt does. It has to: ConfigRouteFingerprints digests
448// registry columns only (routefingerprint.go:58) and cannot see a
449// bucket-side change, so without the bump an overlay would wait out the
450// agent's hourly floor with nothing to signal it.
451func TestWritingTheInstanceLayerBumpsTheGeneration(t *testing.T) {
452 ctx := context.Background()
453 s := release.NewDirStore(t.TempDir())
454 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"LOG_LEVEL": "debug"}); err != nil {
455 t.Fatalf("SetEnv: %v", err)
456 }
457 genETag := func() string {
458 t.Helper()
459 _, etag, err := s.Get(ctx, GenKey("bab", "titogo"))
460 if err != nil {
461 t.Fatalf("read generation: %v", err)
462 }
463 return etag
464 }
465 before := genETag()
466 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com",
467 map[string]Var{"TENANT_SMTP_USER": {Kind: "env", Value: "book"}}, nil); err != nil {
468 t.Fatalf("ApplyInstance: %v", err)
469 }
470 afterWrite := genETag()
471 if afterWrite == before {
472 t.Fatal("writing the instance layer did not bump the generation, so the fast loop would never see it")
473 }
474 if err := DeleteInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com"); err != nil {
475 t.Fatalf("DeleteInstanceConfig: %v", err)
476 }
477 if genETag() == afterWrite {
478 t.Fatal("removing the instance layer did not bump the generation, so the route would keep its last merge")
479 }
480 if _, _, found, err := LoadInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com"); err != nil || found {
481 t.Fatalf("the overlay survived its delete: found=%v err=%v", found, err)
482 }
483 // Deleting again is a no-op success, so a retried teardown converges.
484 if err := DeleteInstanceConfig(ctx, s, "bab", "titogo", "book.customer.example.com"); err != nil {
485 t.Fatalf("second DeleteInstanceConfig: %v", err)
486 }
487}
488```
489
490- [ ] **Step 2: Run it to verify it fails**
491
492```bash
493go test -run 'TestApplyInstance|TestWritingTheInstanceLayer' ./internal/appconfig/
494```
495
496Expected: build failure, `undefined: ApplyInstance`.
497
498- [ ] **Step 3: Write the code**
499
500`internal/appconfig/store.go`, `applyAt` (`:146`) returns the new etag. The body is unchanged except the three early returns and the final `Put`:
501
502```go
503func applyAt(ctx context.Context, s release.Store, key, app, bundle string, sets map[string]Var, unsets []string) (int, string, error) {
504 for k, v := range sets {
505 if !ValidVarName(k) {
506 return 0, "", fmt.Errorf("config: invalid var name %q", k)
507 }
508 if v.Kind == "env" {
509 if err := ValidVarValue(v.Value); err != nil {
510 return 0, "", fmt.Errorf("config: var %s: %w", k, err)
511 }
512 }
513 }
514 m, etag, found, err := getForWriteAt(ctx, s, key, app)
515 if err != nil {
516 return 0, "", err
517 }
518 if !found && len(sets) == 0 {
519 return 0, "", fmt.Errorf("config: %s has no %s bundle — nothing to unset", app, bundle)
520 }
521 if m.Vars == nil {
522 m.Vars = map[string]Var{}
523 }
524 removed := 0
525 for _, k := range unsets {
526 if _, ok := m.Vars[k]; ok {
527 removed++
528 }
529 delete(m.Vars, k)
530 }
531 for k, v := range sets {
532 m.Vars[k] = v
533 }
534 m.Schema, m.App = 1, app
535 raw, err := json.MarshalIndent(m, "", " ")
536 if err != nil {
537 return 0, "", err
538 }
539 // The new etag is returned because the instance layer's writer (design
540 // §6.5) has to hand it to whoever will later compare it with an agent's
541 // delivery ack. The other two bundles' callers ignore it.
542 newETag, err := s.Put(ctx, key, raw, etag)
543 if err != nil {
544 return 0, "", err
545 }
546 return removed, newETag, nil
547}
548```
549
550Its two existing callers drop the new value: `apply` (`store.go:122`) becomes
551
552```go
553 removed, _, err := applyAt(ctx, s, Key(acct, app), app, bundle, sets, unsets)
554```
555
556and `applyEnvironment` (`environments.go:118`) becomes
557
558```go
559 removed, _, err := applyAt(ctx, s, EnvironmentKey(acct, app, envName), app, envName, sets, unsets)
560```
561
562`internal/appconfig/instances.go` (append):
563
564```go
565// ApplyInstance writes sets and removes unsets from ONE instance's overlay
566// in one CAS write, returning how many of unsets were actually present and
567// THE ETAG IT WROTE AT.
568//
569// That etag is the contract with the box. The console keeps it, the agent
570// reports the etag it rendered in its config ack, and the two are compared
571// (design §6.5, InstanceConfigDelivery). A map of delivered key names would
572// not do: it cannot tell an agent too old to render this layer apart from a
573// new agent rendering a route whose overlay sets nothing it recognises.
574//
575// There is no CLI verb and no console door onto this function, by design
576// (§6.2, "only the platform writes this layer"). It exists so a credential
577// can be scoped to one instance, and the only caller that scopes one is the
578// email walk in step 5.
579//
580// Unset-only against an overlay that does not exist is refused, not turned
581// into a create, for the reason applyAt's own doc comment gives: an empty
582// manifest that EXISTS is a state the sweeps converge a file to, and
583// minting one to hold nothing would deliver an empty file to a live
584// instance.
585func ApplyInstance(ctx context.Context, s release.Store, acct, app, host string, sets map[string]Var, unsets []string) (int, string, error) {
586 if !release.ValidAccount(acct) {
587 return 0, "", fmt.Errorf("config: invalid account %q", acct)
588 }
589 if !release.ValidApp(app) {
590 return 0, "", fmt.Errorf("config: invalid app %q", app)
591 }
592 if !ValidInstanceHost(host) {
593 return 0, "", fmt.Errorf("config: invalid instance host %q", host)
594 }
595 removed, etag, err := applyAt(ctx, s, InstanceConfigKey(acct, app, host), app, "instance "+host, sets, unsets)
596 if err != nil {
597 return 0, "", err
598 }
599 // The generation object is the ONLY signal this layer has.
600 // ConfigRouteFingerprints (routefingerprint.go) digests registry columns
601 // and cannot see a bucket-side change, so an overlay written without
602 // this touch would wait out configSyncLoop's hourly floor. Same
603 // best-effort demotion every other write here makes: the manifest has
604 // landed, so a failed touch is logged, never returned.
605 touchLogged(ctx, s, acct, app)
606 return removed, etag, nil
607}
608
609// DeleteInstanceConfig removes one instance's overlay outright, which
610// converges that route back to its group's merge on the next pass. Absent
611// is success: a retried teardown must converge rather than fail, and
612// release.Store.Delete is already idempotent.
613//
614// The generation is touched even when nothing was removed. Branching on it
615// would mean a Get before every delete to find out, and one redundant touch
616// costs a tiny object write where a missed one costs an hour of a live
617// instance holding config nothing meant it to have.
618func DeleteInstanceConfig(ctx context.Context, s release.Store, acct, app, host string) error {
619 if !release.ValidAccount(acct) {
620 return fmt.Errorf("config: invalid account %q", acct)
621 }
622 if !release.ValidApp(app) {
623 return fmt.Errorf("config: invalid app %q", app)
624 }
625 if !ValidInstanceHost(host) {
626 return fmt.Errorf("config: invalid instance host %q", host)
627 }
628 if err := s.Delete(ctx, InstanceConfigKey(acct, app, host)); err != nil {
629 return err
630 }
631 touchLogged(ctx, s, acct, app)
632 return nil
633}
634```
635
636- [ ] **Step 4: Run it to verify it passes**
637
638```bash
639go test ./internal/appconfig/ -count=1
640```
641
642Expected: PASS.
643
644- [ ] **Step 5: Gate and commit**
645
646```bash
647gofmt -w internal/appconfig/instances.go internal/appconfig/instances_test.go internal/appconfig/store.go internal/appconfig/environments.go
648go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
649git add internal/appconfig
650git commit -m "appconfig: write and delete an instance overlay, bumping the app's generation (plan 4, task 2)"
651```
652
653---
654
655### Task 3: `LoadMergedHost`, the three-layer resolve
656
657**Files:**
658- Modify: `internal/appconfig/instances.go`, `internal/appconfig/instances_test.go`
659
660**Interfaces:**
661- Consumes: `LoadMerged` (`environments.go:169`), unchanged in signature.
662- Produces:
663 ```go
664 func LoadMergedHost(ctx context.Context, s release.Store, acct, app, envName, host string) (*Manifest, string, error) // merged, instance overlay etag
665 func applyInstanceOverlay(app string, base, inst *Manifest) *Manifest
666 func instanceOverlays(ctx context.Context, s release.Store, acct, app string) (map[string]bool, error)
667 ```
668
669- [ ] **Step 1: Write the failing test**
670
671`internal/appconfig/instances_test.go` (append):
672
673```go
674// The precedence the design states: default < environment < instance
675// (§6.2). Three layers, each winning per key over the one below it, and the
676// instance layer's etag handed back so the caller can later check what was
677// delivered.
678func TestLoadMergedHostResolvesDefaultThenEnvironmentThenInstance(t *testing.T) {
679 ctx := context.Background()
680 s := release.NewDirStore(t.TempDir())
681 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{
682 "FROM_DEFAULT": "default", "FROM_ENV": "default", "FROM_INSTANCE": "default"}); err != nil {
683 t.Fatalf("SetEnv: %v", err)
684 }
685 if _, err := ApplyEnvironment(ctx, s, "bab", "titogo", "prod", map[string]Var{
686 "FROM_ENV": {Kind: "env", Value: "environment"},
687 "FROM_INSTANCE": {Kind: "env", Value: "environment"}}, nil); err != nil {
688 t.Fatalf("ApplyEnvironment: %v", err)
689 }
690 _, want, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com", map[string]Var{
691 "FROM_INSTANCE": {Kind: "env", Value: "instance"}}, nil)
692 if err != nil {
693 t.Fatalf("ApplyInstance: %v", err)
694 }
695
696 m, etag, err := LoadMergedHost(ctx, s, "bab", "titogo", "prod", "book.customer.example.com")
697 if err != nil {
698 t.Fatalf("LoadMergedHost: %v", err)
699 }
700 if etag != want {
701 t.Fatalf("etag = %q, want the overlay's own %q", etag, want)
702 }
703 for k, wantVal := range map[string]string{
704 "FROM_DEFAULT": "default", "FROM_ENV": "environment", "FROM_INSTANCE": "instance"} {
705 if got := m.Vars[k].Value; got != wantVal {
706 t.Errorf("%s = %q, want %q", k, got, wantVal)
707 }
708 }
709
710 // A sibling with no overlay of its own gets the two-layer merge and an
711 // empty etag, which is what says "this host has no overlay" rather than
712 // "this agent did not render one".
713 m2, etag2, err := LoadMergedHost(ctx, s, "bab", "titogo", "prod", "other.customer.example.com")
714 if err != nil {
715 t.Fatalf("LoadMergedHost (sibling): %v", err)
716 }
717 if etag2 != "" {
718 t.Fatalf("a host with no overlay reported etag %q", etag2)
719 }
720 if m2.Vars["FROM_INSTANCE"].Value != "environment" {
721 t.Fatalf("the sibling saw another instance's value: %+v", m2.Vars)
722 }
723}
724
725// With no overlay, LoadMergedHost is LoadMerged, to the field. That matters
726// for the unbound case in particular: LoadMerged returns the default bundle
727// OBJECT there, exec_delivery and all, and a merge that quietly rebuilt it
728// would strip a flag other code reads off the same manifest.
729func TestLoadMergedHostWithNoOverlayIsExactlyLoadMerged(t *testing.T) {
730 ctx := context.Background()
731 s := release.NewDirStore(t.TempDir())
732 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"K": "v"}); err != nil {
733 t.Fatalf("SetEnv: %v", err)
734 }
735 if err := SetExecDelivery(ctx, s, "bab", "titogo", ExecDeliveryFile); err != nil {
736 t.Fatalf("SetExecDelivery: %v", err)
737 }
738 want, err := LoadMerged(ctx, s, "bab", "titogo", "")
739 if err != nil {
740 t.Fatalf("LoadMerged: %v", err)
741 }
742 got, etag, err := LoadMergedHost(ctx, s, "bab", "titogo", "", "book.customer.example.com")
743 if err != nil || etag != "" {
744 t.Fatalf("LoadMergedHost = %q %v", etag, err)
745 }
746 if got.ExecDelivery != want.ExecDelivery || got.Vars["K"] != want.Vars["K"] || len(got.Vars) != len(want.Vars) {
747 t.Fatalf("LoadMergedHost diverged from LoadMerged with no overlay: %+v vs %+v", got, want)
748 }
749}
750
751// exec_delivery is never read from this layer (the ruling at
752// appconfig.go:66-71). An overlay that carries the field does not put it in
753// the merged view, so nothing downstream can act on it.
754func TestInstanceLayerNeverCarriesExecDelivery(t *testing.T) {
755 ctx := context.Background()
756 s := release.NewDirStore(t.TempDir())
757 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"K": "v"}); err != nil {
758 t.Fatalf("SetEnv: %v", err)
759 }
760 putInstanceRaw(t, ctx, s, "bab", "titogo", "book.customer.example.com", &Manifest{
761 Schema: 1, App: "titogo", Vars: map[string]Var{"OWN": {Kind: "env", Value: "1"}},
762 ExecDelivery: ExecDeliveryFile})
763 m, _, err := LoadMergedHost(ctx, s, "bab", "titogo", "", "book.customer.example.com")
764 if err != nil {
765 t.Fatalf("LoadMergedHost: %v", err)
766 }
767 if m.ExecDelivery != "" {
768 t.Fatalf("exec_delivery = %q: the instance layer must never decide delivery", m.ExecDelivery)
769 }
770 if m.Vars["OWN"].Value != "1" {
771 t.Fatalf("the overlay's vars were dropped along with its flag: %+v", m.Vars)
772 }
773}
774
775// A route bound to a bundle that does not exist is still an ERROR, overlay
776// or no overlay. The binding asserted the bundle exists; an overlay does not
777// make that assertion true, and answering with a partial merge would hand a
778// live route a file built from a binding nobody honoured.
779func TestLoadMergedHostStillRefusesAMissingBoundBundle(t *testing.T) {
780 ctx := context.Background()
781 s := release.NewDirStore(t.TempDir())
782 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"K": "v"}); err != nil {
783 t.Fatalf("SetEnv: %v", err)
784 }
785 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.customer.example.com",
786 map[string]Var{"OWN": {Kind: "env", Value: "1"}}, nil); err != nil {
787 t.Fatalf("ApplyInstance: %v", err)
788 }
789 if _, _, err := LoadMergedHost(ctx, s, "bab", "titogo", "gone", "book.customer.example.com"); err == nil {
790 t.Fatal("a missing bound bundle resolved anyway because the host had an overlay")
791 }
792}
793
794// The index the sweeps use: one listing per app answers "which of this
795// app's instances have an overlay at all", so a pass costs a Get only for
796// the instances that do.
797func TestInstanceOverlaysListsOnlyThisAppsHosts(t *testing.T) {
798 ctx := context.Background()
799 s := release.NewDirStore(t.TempDir())
800 have, err := instanceOverlays(ctx, s, "bab", "titogo")
801 if err != nil || len(have) != 0 {
802 t.Fatalf("empty app = %v %v, want an empty set and no error", have, err)
803 }
804 for _, host := range []string{"book.customer.example.com", "cart.customer.example.com"} {
805 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", host,
806 map[string]Var{"OWN": {Kind: "env", Value: host}}, nil); err != nil {
807 t.Fatalf("ApplyInstance %s: %v", host, err)
808 }
809 }
810 if _, _, err := ApplyInstance(ctx, s, "bab", "eleven", "other.example.com",
811 map[string]Var{"OWN": {Kind: "env", Value: "x"}}, nil); err != nil {
812 t.Fatalf("ApplyInstance (other app): %v", err)
813 }
814 have, err = instanceOverlays(ctx, s, "bab", "titogo")
815 if err != nil {
816 t.Fatalf("instanceOverlays: %v", err)
817 }
818 if len(have) != 2 || !have["book.customer.example.com"] || !have["cart.customer.example.com"] {
819 t.Fatalf("instanceOverlays = %v, want just this app's two hosts", have)
820 }
821 if have["other.example.com"] {
822 t.Fatal("another app's overlay leaked into this app's index")
823 }
824}
825```
826
827- [ ] **Step 2: Run it to verify it fails**
828
829```bash
830go test -run 'TestLoadMergedHost|TestInstanceLayerNever|TestInstanceOverlays' ./internal/appconfig/
831```
832
833Expected: build failure, `undefined: LoadMergedHost`.
834
835- [ ] **Step 3: Write the code**
836
837`internal/appconfig/instances.go` (append; add `sort` to the imports only if a later task needs it — it does not here):
838
839```go
840// LoadMergedHost is LoadMerged plus the instance layer: default <
841// environment < instance, the instance winning per key, and the overlay's
842// etag returned alongside so a caller can check delivery against it later
843// (design §6.2, §6.5). An empty etag means this host has no overlay, which
844// is a different statement from an agent that never rendered one.
845//
846// LoadMerged keeps its signature and its meaning. Everything that asks
847// "what does this APP's environment resolve to" still gets two layers, and
848// only a caller that names a host gets three.
849//
850// The two sync sweeps do NOT call this function, deliberately: they resolve
851// one group base for every route that shares it and would re-fetch
852// config.json per route if they came through here. They reach the same
853// answer through instanceOverlays and applyInstanceOverlay instead, and the
854// equivalence is pinned by a test rather than by hope.
855func LoadMergedHost(ctx context.Context, s release.Store, acct, app, envName, host string) (*Manifest, string, error) {
856 if !ValidInstanceHost(host) {
857 return nil, "", fmt.Errorf("config: invalid instance host %q", host)
858 }
859 base, err := LoadMerged(ctx, s, acct, app, envName)
860 if err != nil {
861 return nil, "", err
862 }
863 inst, etag, found, err := LoadInstanceConfig(ctx, s, acct, app, host)
864 if err != nil {
865 return nil, "", err
866 }
867 if !found {
868 return base, "", nil
869 }
870 return applyInstanceOverlay(app, base, inst), etag, nil
871}
872
873// applyInstanceOverlay lays inst over base, per key. base may be nil (the
874// app has no default bundle and the route is unbound), in which case the
875// merged view is the overlay ALONE — which the callers treat as the
876// wholesale replacement it is, see sync.go's defaultMissing guard.
877//
878// A nil overlay returns base untouched, pointer and all: that is the path
879// every instance without an overlay takes, and rebuilding the manifest
880// there would strip exec_delivery off the default bundle object LoadMerged
881// hands back for an unbound route.
882//
883// The merged view never carries exec_delivery FROM the overlay. See this
884// file's own doc comment: a per-host delivery flag is the granularity the
885// #195 ruling refused, and the merged manifest is what gets rendered into a
886// file, where a delivery flag has no business appearing anyway.
887func applyInstanceOverlay(app string, base, inst *Manifest) *Manifest {
888 if inst == nil {
889 return base
890 }
891 merged := &Manifest{Schema: 1, App: app, Vars: map[string]Var{}}
892 if base != nil {
893 for k, v := range base.Vars {
894 merged.Vars[k] = v
895 }
896 }
897 for k, v := range inst.Vars {
898 merged.Vars[k] = v
899 }
900 return merged
901}
902
903// instanceOverlays lists which of this app's instances have an overlay at
904// all: one listing per (account, app), cached by the caller for a whole
905// pass, so a sweep pays one List per app and then one Get per instance that
906// actually has one. Reading every route's key blind would cost a bucket
907// round trip per instance on a box where nothing has an overlay, which is
908// every box in the fleet the day this ships.
909//
910// A key that does not end in .json, or whose host does not survive
911// ValidInstanceHost, is skipped rather than reported: nothing can have
912// written it through this package, so it is not this app's overlay
913// whatever it is.
914func instanceOverlays(ctx context.Context, s release.Store, acct, app string) (map[string]bool, error) {
915 prefix := InstancesConfigPrefix(acct, app)
916 keys, err := s.List(ctx, prefix)
917 if err != nil {
918 return nil, err
919 }
920 out := make(map[string]bool, len(keys))
921 for _, k := range keys {
922 host := strings.TrimSuffix(strings.TrimPrefix(k, prefix), ".json")
923 if host == strings.TrimPrefix(k, prefix) || !ValidInstanceHost(host) {
924 continue
925 }
926 out[host] = true
927 }
928 return out, nil
929}
930```
931
932- [ ] **Step 4: Run it to verify it passes**
933
934```bash
935go test ./internal/appconfig/ -count=1
936```
937
938Expected: PASS.
939
940- [ ] **Step 5: Gate and commit**
941
942```bash
943gofmt -w internal/appconfig/instances.go internal/appconfig/instances_test.go
944go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
945git add internal/appconfig
946git commit -m "appconfig: LoadMergedHost resolves default < environment < instance (plan 4, task 3)"
947```
948
949---
950
951### Task 4: `Sync` applies the overlay per route, at one listing per app
952
953**Files:**
954- Modify: `internal/appconfig/instances.go`, `internal/appconfig/sync.go` (the materialization loop at `:143-176`), `internal/appconfig/instances_test.go`
955
956**Interfaces:**
957- Consumes: `instanceOverlays`, `applyInstanceOverlay` (task 3), `Sync`'s existing group caches (`sync.go:72-105`).
958- Produces:
959 ```go
960 func instanceOverlayLoader(ctx context.Context, s release.Store) func(acct, app, host string) (*Manifest, string, error)
961 ```
962 One closure per pass, shared by both sweeps: one `List` per (account, app), then a `Get` only for a host the listing named.
963
964- [ ] **Step 1: Write the failing test**
965
966`internal/appconfig/instances_test.go` (append; add `bytes`, `crypto/ecdh`, `os`, `path/filepath`, `sync` and `github.com/carlosframework/platform/internal/registry` to the file's imports):
967
968```go
969// countingStore counts the reads and writes a pass makes, per key, so a
970// test can pin the COST of a feature and not just its output. The instance
971// layer's promise is one extra listing per app and one extra Get per
972// instance that actually has an overlay, which is a claim only a counter
973// can keep honest.
974type countingStore struct {
975 release.Store
976 mu sync.Mutex
977 gets map[string]int
978 lists map[string]int
979 puts map[string]int
980}
981
982func newCountingStore(s release.Store) *countingStore {
983 return &countingStore{Store: s, gets: map[string]int{}, lists: map[string]int{}, puts: map[string]int{}}
984}
985
986func (c *countingStore) Get(ctx context.Context, key string) ([]byte, string, error) {
987 c.mu.Lock()
988 c.gets[key]++
989 c.mu.Unlock()
990 return c.Store.Get(ctx, key)
991}
992
993func (c *countingStore) List(ctx context.Context, prefix string) ([]string, error) {
994 c.mu.Lock()
995 c.lists[prefix]++
996 c.mu.Unlock()
997 return c.Store.List(ctx, prefix)
998}
999
1000func (c *countingStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
1001 c.mu.Lock()
1002 c.puts[key]++
1003 c.mu.Unlock()
1004 return c.Store.Put(ctx, key, body, ifMatch)
1005}
1006
1007func (c *countingStore) n(m map[string]int, k string) int {
1008 c.mu.Lock()
1009 defer c.mu.Unlock()
1010 return m[k]
1011}
1012
1013// syncInstanceKit is the unit-path fixture these tests share: a store, a
1014// registry, an env dir and a parsed key pair.
1015func syncInstanceKit(t *testing.T) (context.Context, release.Store, *registry.Registry, string, *ecdh.PrivateKey) {
1016 t.Helper()
1017 ctx := context.Background()
1018 s := release.NewDirStore(t.TempDir())
1019 reg := testRegistry(t)
1020 priv, _, err := GenerateKey()
1021 if err != nil {
1022 t.Fatalf("GenerateKey: %v", err)
1023 }
1024 privKey, err := ParsePrivateKey(priv, "test")
1025 if err != nil {
1026 t.Fatalf("ParsePrivateKey: %v", err)
1027 }
1028 return ctx, s, reg, t.TempDir(), privKey
1029}
1030
1031func unitRoute(t *testing.T, reg *registry.Registry, host, app, acct, env string) {
1032 t.Helper()
1033 if err := reg.Upsert(registry.Route{
1034 Host: host, Kind: "instance", App: app, Account: acct, Status: "ready",
1035 Environment: env, Backing: registry.BackingUnit,
1036 }); err != nil {
1037 t.Fatalf("Upsert %s: %v", host, err)
1038 }
1039}
1040
1041// The delivery this layer exists for: one instance of an app gets a value
1042// its siblings never see. The sibling's file is the app merge, untouched.
1043func TestSyncRendersTheInstanceOverlayForItsOwnRouteOnly(t *testing.T) {
1044 ctx, s, reg, envDir, priv := syncInstanceKit(t)
1045 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes", "TENANT": "none"}); err != nil {
1046 t.Fatalf("SetEnv: %v", err)
1047 }
1048 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
1049 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1050 t.Fatalf("ApplyInstance: %v", err)
1051 }
1052 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1053 unitRoute(t, reg, "cart.example.com", "titogo", "bab", "")
1054
1055 if err := Sync(ctx, reg, s, priv, envDir, func(string) error { return nil }, nil); err != nil {
1056 t.Fatalf("Sync: %v", err)
1057 }
1058 book, err := os.ReadFile(filepath.Join(envDir, "book.example.com.env"))
1059 if err != nil {
1060 t.Fatalf("read book env: %v", err)
1061 }
1062 if !bytes.Contains(book, []byte("TENANT=book\n")) || !bytes.Contains(book, []byte("APP_WIDE=yes\n")) {
1063 t.Fatalf("the overlaid route's file is wrong:\n%s", book)
1064 }
1065 cart, err := os.ReadFile(filepath.Join(envDir, "cart.example.com.env"))
1066 if err != nil {
1067 t.Fatalf("read cart env: %v", err)
1068 }
1069 if bytes.Contains(cart, []byte("book")) || !bytes.Contains(cart, []byte("TENANT=none\n")) {
1070 t.Fatalf("a sibling saw another instance's config:\n%s", cart)
1071 }
1072
1073 // The sweep and the single-shot API must agree about what a host gets.
1074 // The sweeps resolve through the group cache and never call
1075 // LoadMergedHost, so this is the only thing holding the two together.
1076 want, _, err := LoadMergedHost(ctx, s, "bab", "titogo", "", "book.example.com")
1077 if err != nil {
1078 t.Fatalf("LoadMergedHost: %v", err)
1079 }
1080 rendered, err := RenderEnv(want, priv)
1081 if err != nil {
1082 t.Fatalf("RenderEnv: %v", err)
1083 }
1084 if !bytes.Contains(book, rendered) {
1085 t.Fatalf("the pass and LoadMergedHost disagree:\npass:\n%s\nLoadMergedHost:\n%s", book, rendered)
1086 }
1087}
1088
1089// The cost. One listing per app, a Get for the instance that has an
1090// overlay, and NOTHING for the instance that does not.
1091func TestSyncCostsOneListingPerAppAndOneGetPerOverlay(t *testing.T) {
1092 ctx, base, reg, envDir, priv := syncInstanceKit(t)
1093 if err := SetEnv(ctx, base, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
1094 t.Fatalf("SetEnv: %v", err)
1095 }
1096 if _, _, err := ApplyInstance(ctx, base, "bab", "titogo", "book.example.com",
1097 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1098 t.Fatalf("ApplyInstance: %v", err)
1099 }
1100 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1101 unitRoute(t, reg, "cart.example.com", "titogo", "bab", "")
1102 unitRoute(t, reg, "desk.example.com", "titogo", "bab", "")
1103
1104 s := newCountingStore(base)
1105 if err := Sync(ctx, reg, s, priv, envDir, func(string) error { return nil }, nil); err != nil {
1106 t.Fatalf("Sync: %v", err)
1107 }
1108 if got := s.n(s.lists, InstancesConfigPrefix("bab", "titogo")); got != 1 {
1109 t.Fatalf("instance overlay listings = %d, want exactly 1 for three routes of one app", got)
1110 }
1111 if got := s.n(s.gets, InstanceConfigKey("bab", "titogo", "book.example.com")); got != 1 {
1112 t.Fatalf("overlay Gets for book = %d, want 1", got)
1113 }
1114 for _, host := range []string{"cart.example.com", "desk.example.com"} {
1115 if got := s.n(s.gets, InstanceConfigKey("bab", "titogo", host)); got != 0 {
1116 t.Fatalf("overlay Gets for %s = %d, want none: the listing already said it has no overlay", host, got)
1117 }
1118 }
1119}
1120
1121// A listing that fails is not a determination. The route keeps its file and
1122// the pass says so, the same contract a bundle fetch failure already has.
1123func TestSyncLeavesTheFileAloneWhenTheOverlayListingFails(t *testing.T) {
1124 ctx, base, reg, envDir, priv := syncInstanceKit(t)
1125 if err := SetEnv(ctx, base, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
1126 t.Fatalf("SetEnv: %v", err)
1127 }
1128 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1129 path := filepath.Join(envDir, "book.example.com.env")
1130 if err := os.WriteFile(path, []byte("APP_WIDE=stale\n"), 0o600); err != nil {
1131 t.Fatalf("stage file: %v", err)
1132 }
1133 failing := &failingListStore{Store: base, failPrefix: InstancesConfigPrefix("bab", "titogo"),
1134 err: errors.New("s3: transient 503 SlowDown")}
1135 out := captureStdout(t, func() {
1136 if err := Sync(ctx, reg, failing, priv, envDir, func(string) error { return nil }, nil); err != nil {
1137 t.Fatalf("Sync: %v", err)
1138 }
1139 })
1140 cur, err := os.ReadFile(path)
1141 if err != nil || string(cur) != "APP_WIDE=stale\n" {
1142 t.Fatalf("the file moved on a listing failure: %q %v", cur, err)
1143 }
1144 if !strings.Contains(out, "book.example.com") || !strings.Contains(out, "left as-is") {
1145 t.Fatalf("the pass was silent about an unresolved instance layer:\n%s", out)
1146 }
1147}
1148
1149// failingListStore makes List fail for exactly one prefix, the listing
1150// shape of failingKeyStore (syncexec_test.go).
1151type failingListStore struct {
1152 release.Store
1153 failPrefix string
1154 err error
1155}
1156
1157func (f *failingListStore) List(ctx context.Context, prefix string) ([]string, error) {
1158 if prefix == f.failPrefix {
1159 return nil, f.err
1160 }
1161 return f.Store.List(ctx, prefix)
1162}
1163```
1164
1165Add `"errors"` and `"strings"` to `instances_test.go`'s imports.
1166
1167- [ ] **Step 2: Run it to verify it fails**
1168
1169```bash
1170go test -run 'TestSyncRendersTheInstanceOverlay|TestSyncCostsOneListing|TestSyncLeavesTheFileAlone' ./internal/appconfig/
1171```
1172
1173Expected: `TestSyncRendersTheInstanceOverlayForItsOwnRouteOnly` fails on `TENANT=none` in the overlaid route's file (the overlay is not applied yet), after a build failure for `failingListStore`'s first use is resolved by the block above.
1174
1175- [ ] **Step 3: Write the code**
1176
1177`internal/appconfig/instances.go` (append):
1178
1179```go
1180// instanceOverlayLoader is one pass's instance-layer reader, shared by both
1181// sweeps so neither can drift from the other about what a host gets or what
1182// it costs. One List per (account, app), cached for the pass including its
1183// failure, then a Get only for a host that listing named.
1184//
1185// A listing FAILURE is returned to the caller, never folded into "no
1186// overlay". The two sweeps treat it the way they treat a bundle fetch
1187// failure: leave the file exactly as it is, say so per route, retry next
1188// pass. Folding it away would silently deliver the app-wide merge to an
1189// instance that is meant to have its own credential, which is the one
1190// outcome this layer exists to prevent.
1191//
1192// A host the listing named but whose object is gone by the time of the Get
1193// is simply "no overlay": a delete racing the pass converges the route back
1194// to its group's merge, which is what the delete asked for.
1195func instanceOverlayLoader(ctx context.Context, s release.Store) func(acct, app, host string) (*Manifest, string, error) {
1196 type appKey struct{ acct, app string }
1197 type indexRes struct {
1198 hosts map[string]bool
1199 err error
1200 }
1201 index := map[appKey]indexRes{}
1202 return func(acct, app, host string) (*Manifest, string, error) {
1203 ak := appKey{acct, app}
1204 idx, done := index[ak]
1205 if !done {
1206 hosts, err := instanceOverlays(ctx, s, acct, app)
1207 if err != nil {
1208 err = fmt.Errorf("%s/%s: instance config listing failed: %w", acct, app, err)
1209 }
1210 idx = indexRes{hosts: hosts, err: err}
1211 index[ak] = idx
1212 }
1213 if idx.err != nil {
1214 return nil, "", idx.err
1215 }
1216 if !idx.hosts[host] {
1217 return nil, "", nil
1218 }
1219 m, etag, found, err := LoadInstanceConfig(ctx, s, acct, app, host)
1220 if err != nil {
1221 return nil, "", fmt.Errorf("%s/%s: instance config for %s: %w", acct, app, host, err)
1222 }
1223 if !found {
1224 return nil, "", nil
1225 }
1226 return m, etag, nil
1227 }
1228}
1229```
1230
1231`internal/appconfig/sync.go`, in `Sync`: build the loader once, just before the materialization loop (after the resolution loop ends at `:142`):
1232
1233```go
1234 // The instance layer (senders design §6.2), resolved per route AFTER the
1235 // group base above. The group is still (Account, App, Environment) and
1236 // the default bundle is still read once per app: only the third layer is
1237 // per host, and it costs one listing per app plus a Get for each
1238 // instance that actually has an overlay.
1239 overlayFor := instanceOverlayLoader(ctx, s)
1240 // One line per app when its listing fails, not one per route: every
1241 // route of that app fails identically and for the same reason.
1242 said := map[groupKey]bool{}
1243```
1244
1245Then in the materialization loop (`:143-176`), name the group key once and apply the overlay between the `m == nil` skip and the backing check:
1246
1247```go
1248 for _, rt := range routes {
1249 if !syncable(rt) {
1250 continue
1251 }
1252 k := groupKey{release.RouteAccount(rt), rt.App, rt.Environment}
1253 r := resolved[k]
1254 if r.err != nil {
1255 env := rt.Environment
1256 if env == "" {
1257 env = "default"
1258 }
1259 fmt.Printf(" %s: config unresolved (environment %q), env file left as-is: %v\n", rt.Host, env, r.err)
1260 continue
1261 }
1262 m := r.m
1263 if m == nil {
1264 continue
1265 }
1266 // The instance overlay wins per key over the group's merge. An
1267 // unresolved overlay is NOT an absent one: this route keeps its file
1268 // and the pass says which route and why, exactly as it does for a
1269 // group whose bundle could not be read.
1270 inst, _, err := overlayFor(k.acct, k.app, rt.Host)
1271 if err != nil {
1272 // Said ONCE per (account, app), not once per route. The listing
1273 // this failed on is per app, so every route of that app fails
1274 // identically and for the same reason; a line each would bury the
1275 // cause under its own repetitions on a box with many routes. The
1276 // loader already holds the error, so said is only about output.
1277 if !said[k] {
1278 said[k] = true
1279 fmt.Printf(" %s/%s: instance config unresolved, env files left as-is: %v\n", k.acct, k.app, err)
1280 }
1281 continue
1282 }
1283 m = applyInstanceOverlay(k.app, m, inst)
1284 if rt.Backing != registry.BackingUnit {
1285 fmt.Printf(" %s: exec-backed — config delivered by the agent's in-process loop to /data/<host>.config.env, not this pass\n", rt.Host)
1286 continue
1287 }
1288 syncHost(rt, m, r.defaultMissing, priv, envDir, restart, ack)
1289 }
1290```
1291
1292- [ ] **Step 4: Run it to verify it passes**
1293
1294```bash
1295go test ./internal/appconfig/ -count=1
1296```
1297
1298Expected: PASS.
1299
1300- [ ] **Step 5: Gate and commit**
1301
1302```bash
1303gofmt -w internal/appconfig/instances.go internal/appconfig/instances_test.go internal/appconfig/sync.go
1304go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
1305git add internal/appconfig
1306git commit -m "appconfig: the unit sync pass applies each route's instance overlay (plan 4, task 4)"
1307```
1308
1309---
1310
1311### Task 5: The `lostKeys` / `defaultMissing` guard, revisited for an overlay alone
1312
1313**Files:**
1314- Modify: `internal/appconfig/sync.go` (`lostKeys` at `:260`, `syncHost`'s guard at `:388-407`, the materialization loop from task 4), `internal/appconfig/instances_test.go`
1315
1316**Interfaces:**
1317- Consumes: `lostKeys` (`sync.go:260`), `agentOwned` (`agentvars.go`).
1318- Produces: no new exported surface. `syncHost`'s `defaultMissing` parameter keeps its name and gains a second way to be true.
1319
1320**Why this is a guard change and not a merge change.** `sync.go:388` refuses to overwrite a hand-staged `/etc/carlos/apps/<host>.env` when the merged view is an overlay ALONE, because the app has no `config.json` and the render is a wholesale replacement rather than an overlay (2026-08-02 review, CRITICAL 1). Before this plan there was exactly one way into that state: a route bound to an environment. The instance layer adds a second, and it does not go through a binding at all — an UNBOUND route of an app with no default bundle used to resolve to "never configured" and have its file left alone (`sync.go:120-123`), and with an overlay it now resolves to something. Without this task, that route's hand-staged file would be replaced by the overlay's vars alone, as root, and the unit restarted.
1321
1322- [ ] **Step 1: Write the failing test**
1323
1324`internal/appconfig/instances_test.go` (append):
1325
1326```go
1327// The CRITICAL 1 guard's second door. An app with no config.json, a route
1328// bound to nothing, and an overlay: the merged view is that overlay ALONE,
1329// which is a wholesale replacement of whatever a hand-staged file holds.
1330// Refuse it, loudly, every pass, and name the way out.
1331func TestSyncRefusesToDropHandStagedKeysForAnOverlayAlone(t *testing.T) {
1332 ctx, s, reg, envDir, priv := syncInstanceKit(t)
1333 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
1334 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1335 t.Fatalf("ApplyInstance: %v", err)
1336 }
1337 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1338 path := filepath.Join(envDir, "book.example.com.env")
1339 staged := "HAND_STAGED=1\nTENANT=old\n"
1340 if err := os.WriteFile(path, []byte(staged), 0o600); err != nil {
1341 t.Fatalf("stage file: %v", err)
1342 }
1343
1344 var restarted []string
1345 out := captureStdout(t, func() {
1346 if err := Sync(ctx, reg, s, priv, envDir, func(h string) error {
1347 restarted = append(restarted, h)
1348 return nil
1349 }, nil); err != nil {
1350 t.Fatalf("Sync: %v", err)
1351 }
1352 })
1353 cur, err := os.ReadFile(path)
1354 if err != nil || string(cur) != staged {
1355 t.Fatalf("a hand-staged file was replaced by an overlay-alone render: %q %v", cur, err)
1356 }
1357 if len(restarted) != 0 {
1358 t.Fatalf("restarted %v for a route this pass refused", restarted)
1359 }
1360 if !strings.Contains(out, "HAND_STAGED") || !strings.Contains(out, "book.example.com") {
1361 t.Fatalf("the refusal does not name the route and the key it would have dropped:\n%s", out)
1362 }
1363}
1364
1365// The same shape, self-healing: nothing is lost, so the route converges.
1366// An app with no default bundle and an instance overlay is a legitimate
1367// state (step 5 creates exactly that for a fresh instance), and this guard
1368// must not freeze it forever.
1369func TestSyncConvergesAnOverlayAloneWhenNothingIsLost(t *testing.T) {
1370 ctx, s, reg, envDir, priv := syncInstanceKit(t)
1371 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
1372 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1373 t.Fatalf("ApplyInstance: %v", err)
1374 }
1375 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1376 if err := Sync(ctx, reg, s, priv, envDir, func(string) error { return nil }, nil); err != nil {
1377 t.Fatalf("Sync: %v", err)
1378 }
1379 cur, err := os.ReadFile(filepath.Join(envDir, "book.example.com.env"))
1380 if err != nil {
1381 t.Fatalf("read env file: %v", err)
1382 }
1383 if !bytes.Contains(cur, []byte("TENANT=book\n")) {
1384 t.Fatalf("the overlay-alone render did not land:\n%s", cur)
1385 }
1386}
1387
1388// And the case that must NOT change: no default bundle, no overlay, no
1389// binding is still "never configured at all", and a never-configured app's
1390// hand-staged file is not this pass's to touch.
1391func TestSyncStillLeavesUnconfiguredAppsAlone(t *testing.T) {
1392 ctx, s, reg, envDir, priv := syncInstanceKit(t)
1393 unitRoute(t, reg, "book.example.com", "titogo", "bab", "")
1394 path := filepath.Join(envDir, "book.example.com.env")
1395 staged := "HAND_STAGED=1\n"
1396 if err := os.WriteFile(path, []byte(staged), 0o600); err != nil {
1397 t.Fatalf("stage file: %v", err)
1398 }
1399 if err := Sync(ctx, reg, s, priv, envDir, func(string) error { return nil }, nil); err != nil {
1400 t.Fatalf("Sync: %v", err)
1401 }
1402 cur, err := os.ReadFile(path)
1403 if err != nil || string(cur) != staged {
1404 t.Fatalf("an unconfigured app's file moved: %q %v", cur, err)
1405 }
1406}
1407```
1408
1409- [ ] **Step 2: Run it to verify it fails**
1410
1411```bash
1412go test -run 'TestSyncRefusesToDropHandStaged|TestSyncConvergesAnOverlayAlone|TestSyncStillLeavesUnconfigured' ./internal/appconfig/
1413```
1414
1415Expected: `TestSyncRefusesToDropHandStagedKeysForAnOverlayAlone` fails — the file has been replaced by `TENANT=book` — and `TestSyncConvergesAnOverlayAloneWhenNothingIsLost` fails because the route resolves as never configured today.
1416
1417- [ ] **Step 3: Write the code**
1418
1419`internal/appconfig/sync.go`, the materialization loop from task 4: fetch the overlay BEFORE the `m == nil` skip, and let an overlay stand alone.
1420
1421```go
1422 // The instance overlay wins per key over the group's merge. An
1423 // unresolved overlay is NOT an absent one: this route keeps its file
1424 // and the pass says which route and why, exactly as it does for a
1425 // group whose bundle could not be read.
1426 inst, _, err := overlayFor(k.acct, k.app, rt.Host)
1427 if err != nil {
1428 if !said[k] {
1429 said[k] = true
1430 fmt.Printf(" %s/%s: instance config unresolved, env files left as-is: %v\n", k.acct, k.app, err)
1431 }
1432 continue
1433 }
1434 m := r.m
1435 if m == nil && inst == nil {
1436 continue // never configured at all: this file is not ours to touch
1437 }
1438 // An overlay with no group base behind it is a wholesale
1439 // replacement, not an overlay — the same state a bound route with no
1440 // default bundle is in, reached without a binding. It gets the same
1441 // guard: see syncHost.
1442 defaultMissing := r.defaultMissing
1443 if m == nil {
1444 defaultMissing = true
1445 }
1446 m = applyInstanceOverlay(k.app, m, inst)
1447```
1448
1449The `syncHost` call further down the same loop takes the local rather than the resolution's field, or this compiles to "declared and not used" and the guard never sees the new case:
1450
1451```go
1452 syncHost(rt, m, defaultMissing, priv, envDir, restart, instETag, ack)
1453```
1454
1455`internal/appconfig/sync.go`, `syncHost`'s guard (`:388-407`): the condition is unchanged, the message names which overlay it is.
1456
1457```go
1458 if defaultMissing && existed && len(cur) > 0 {
1459 lost, parsed := lostKeys(cur, m)
1460 // Which overlay is standing alone. A bound route names its
1461 // environment; an unbound one is here because this instance has its
1462 // own config layer (senders design §6.2) and its app has no default
1463 // bundle, and naming an empty environment would be a riddle.
1464 alone := "this instance's own config layer"
1465 if rt.Environment != "" {
1466 alone = fmt.Sprintf("environment %q", rt.Environment)
1467 }
1468 switch {
1469 case !parsed:
1470 // Unparseable content has unknown keys, so nothing here can prove
1471 // the render drops none of them. Assume the worst — this guard's
1472 // whole job is to be wrong in the safe direction.
1473 fmt.Printf(" %s: %s has no default config bundle, so the merged view is %s ALONE — "+
1474 "refusing to replace %s, which this pass cannot parse as KEY=value and so cannot prove is safe to overwrite; "+
1475 "import its config (`carlos env set --app %s KEY=value ...`) or remove the file\n",
1476 rt.Host, rt.App, alone, path, rt.App)
1477 return
1478 case len(lost) > 0:
1479 fmt.Printf(" %s: %s has no default config bundle, so the merged view is %s ALONE — "+
1480 "refusing to drop %d var(s) already in %s (%s); import them (`carlos env set --app %s KEY=value ...`) "+
1481 "and this route converges on the next pass, or remove the file if it is dead\n",
1482 rt.Host, rt.App, alone, len(lost), path, strings.Join(lost, ", "), rt.App)
1483 return
1484 }
1485 }
1486```
1487
1488`internal/appconfig/sync.go`, `lostKeys`'s doc comment (`:241-259`) gains the third layer, and `resolution.defaultMissing`'s (`:66-70`) gains the second door:
1489
1490```go
1491 // defaultMissing marks a bound route's merged view as the named
1492 // bundle ALONE, because the app has no config.json at all. Such a
1493 // render is not an overlay, it is a wholesale replacement, so
1494 // syncHost refuses to let it drop keys the file on disk already holds
1495 // (see its doc comment; 2026-08-02 review, CRITICAL 1 and round 2).
1496 // The materialization loop sets the same flag for an UNBOUND route
1497 // whose only config is its instance overlay (senders design §6.2),
1498 // which is the same replacement reached by a different door.
1499 defaultMissing bool
1500```
1501
1502and, on `lostKeys`:
1503
1504```go
1505// It compares against m.Vars rather than the rendered bytes on purpose: the
1506// render is what the caller is deciding whether to WRITE, and this decision
1507// has to be made before rendering (a refused route must never reach the
1508// render-diff-write-restart path at all). m is the route's FULL merge, all
1509// three layers of it (default, environment, instance), because the question
1510// is what the file would lose, not which bundle a key came from.
1511```
1512
1513- [ ] **Step 4: Run it to verify it passes**
1514
1515```bash
1516go test ./internal/appconfig/ -count=1
1517```
1518
1519Expected: PASS.
1520
1521- [ ] **Step 5: Gate and commit**
1522
1523```bash
1524gofmt -w internal/appconfig/sync.go internal/appconfig/instances_test.go
1525go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
1526git add internal/appconfig
1527git commit -m "appconfig: the key-loss guard covers an instance overlay standing alone (plan 4, task 5)"
1528```
1529
1530---
1531
1532### Task 6: `SyncExec` applies the overlay per route
1533
1534**Files:**
1535- Modify: `internal/appconfig/syncexec.go` (the materialization loop at `:382-470`), `internal/appconfig/instances_test.go`
1536
1537**Interfaces:**
1538- Consumes: `instanceOverlayLoader`, `applyInstanceOverlay` (tasks 3 and 4), `SyncExec`'s existing group and opt-in caches (`syncexec.go:121-268`).
1539- Produces: no new exported surface.
1540
1541- [ ] **Step 1: Write the failing test**
1542
1543`internal/appconfig/instances_test.go` (append):
1544
1545```go
1546// The exec half of the same delivery: one activator-spawned instance gets
1547// its own value, its sibling does not, and the listing is paid for once.
1548func TestSyncExecRendersTheInstanceOverlayForItsOwnRouteOnly(t *testing.T) {
1549 ctx, base, reg, dd, privB64, _ := execKit(t)
1550 priv, err := ParsePrivateKey(privB64, "test")
1551 if err != nil {
1552 t.Fatalf("ParsePrivateKey: %v", err)
1553 }
1554 if err := SetEnv(ctx, base, "bab", "titogo", map[string]string{"APP_WIDE": "yes", "TENANT": "none"}); err != nil {
1555 t.Fatalf("SetEnv: %v", err)
1556 }
1557 if _, _, err := ApplyInstance(ctx, base, "bab", "titogo", "book.example.com",
1558 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1559 t.Fatalf("ApplyInstance: %v", err)
1560 }
1561 for _, host := range []string{"book.example.com", "cart.example.com"} {
1562 if err := reg.Upsert(registry.Route{
1563 Host: host, Kind: "instance", App: "titogo", Account: "bab", Status: "ready",
1564 Socket: "/run/carlos/" + host + ".sock",
1565 }); err != nil {
1566 t.Fatalf("Upsert %s: %v", host, err)
1567 }
1568 }
1569
1570 s := newCountingStore(base)
1571 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", nil); err != nil {
1572 t.Fatalf("SyncExec: %v", err)
1573 }
1574 book, err := os.ReadFile(filepath.Join(dd, "book.example.com.config.env"))
1575 if err != nil {
1576 t.Fatalf("read book config: %v", err)
1577 }
1578 if !bytes.Contains(book, []byte("TENANT=book\n")) || !bytes.Contains(book, []byte("APP_WIDE=yes\n")) {
1579 t.Fatalf("the overlaid route's config file is wrong:\n%s", book)
1580 }
1581 cart, err := os.ReadFile(filepath.Join(dd, "cart.example.com.config.env"))
1582 if err != nil {
1583 t.Fatalf("read cart config: %v", err)
1584 }
1585 if bytes.Contains(cart, []byte("book")) || !bytes.Contains(cart, []byte("TENANT=none\n")) {
1586 t.Fatalf("a sibling saw another instance's config:\n%s", cart)
1587 }
1588 if got := s.n(s.lists, InstancesConfigPrefix("bab", "titogo")); got != 1 {
1589 t.Fatalf("instance overlay listings = %d, want exactly 1 for two routes of one app", got)
1590 }
1591 if got := s.n(s.gets, InstanceConfigKey("bab", "titogo", "cart.example.com")); got != 0 {
1592 t.Fatalf("read an overlay for a host the listing did not name (%d times)", got)
1593 }
1594}
1595
1596// exec_delivery is a BUNDLE-level flag and the instance layer is not one of
1597// the bundles it is read from (the ruling at appconfig.go:66-71). A TCP-addr
1598// route whose app has never opted in gets no file, whatever its overlay
1599// says — a per-host delivery flag is exactly the granularity that ruling
1600// refused, and honouring one here would be the config-that-lies hazard it
1601// names.
1602func TestSyncExecNeverReadsExecDeliveryFromTheInstanceLayer(t *testing.T) {
1603 ctx, s, reg, dd, privB64, _ := execKit(t)
1604 priv, err := ParsePrivateKey(privB64, "test")
1605 if err != nil {
1606 t.Fatalf("ParsePrivateKey: %v", err)
1607 }
1608 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
1609 t.Fatalf("SetEnv: %v", err)
1610 }
1611 putInstanceRaw(t, ctx, s, "bab", "titogo", "router.example.com", &Manifest{
1612 Schema: 1, App: "titogo", Vars: map[string]Var{"TENANT": {Kind: "env", Value: "book"}},
1613 ExecDelivery: ExecDeliveryFile})
1614 if err := reg.Upsert(registry.Route{
1615 Host: "router.example.com", Kind: "instance", App: "titogo", Account: "bab",
1616 Status: "ready", Addr: "127.0.0.1:8080",
1617 }); err != nil {
1618 t.Fatalf("Upsert: %v", err)
1619 }
1620 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", nil); err != nil {
1621 t.Fatalf("SyncExec: %v", err)
1622 }
1623 if _, err := os.Stat(filepath.Join(dd, "router.example.com.config.env")); !os.IsNotExist(err) {
1624 t.Fatalf("an instance overlay opted a route into file delivery: %v", err)
1625 }
1626
1627 // Opt the APP in, and the same route gets the same overlay merged in the
1628 // ordinary way — the flag's door is the bundle, the overlay's payload is
1629 // vars.
1630 if err := SetExecDelivery(ctx, s, "bab", "titogo", ExecDeliveryFile); err != nil {
1631 t.Fatalf("SetExecDelivery: %v", err)
1632 }
1633 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", nil); err != nil {
1634 t.Fatalf("SyncExec (opted in): %v", err)
1635 }
1636 cur, err := os.ReadFile(filepath.Join(dd, "router.example.com.config.env"))
1637 if err != nil {
1638 t.Fatalf("read router config: %v", err)
1639 }
1640 if !bytes.Contains(cur, []byte("TENANT=book\n")) {
1641 t.Fatalf("the overlay's vars did not land once the app opted in:\n%s", cur)
1642 }
1643}
1644```
1645
1646- [ ] **Step 2: Run it to verify it fails**
1647
1648```bash
1649go test -run 'TestSyncExecRendersTheInstanceOverlay|TestSyncExecNeverReadsExecDelivery' ./internal/appconfig/
1650```
1651
1652Expected: `TestSyncExecRendersTheInstanceOverlayForItsOwnRouteOnly` fails with `TENANT=none` in the overlaid route's file.
1653
1654- [ ] **Step 3: Write the code**
1655
1656`internal/appconfig/syncexec.go`: build the loader once, just after the mass-removal guard's `skipRemovals` block (`:380`) and before the materialization loop:
1657
1658```go
1659 // The instance layer (senders design §6.2), resolved per route after the
1660 // group base. Same loader, same cost, same failure contract as the unit
1661 // pass's: one listing per app, a Get only for a host it named, and an
1662 // unresolved overlay leaves the file exactly as it is.
1663 overlayFor := instanceOverlayLoader(ctx, s)
1664 // One line per app when its listing fails, not one per route: every
1665 // route of that app fails identically and for the same reason.
1666 said := map[groupKey]bool{}
1667```
1668
1669Then, inside the materialization loop, fetch it immediately after the `r.err` branch (`:397-407`) — BEFORE any branch that can remove a file — and apply it at the call:
1670
1671```go
1672 // Fetched here, used at the bottom, because a failure must be able to
1673 // stop a REMOVAL too: deciding a file is orphaned while unable to see
1674 // whether this instance has an overlay would delete a live tenant's
1675 // delivered secrets on a listing blip.
1676 inst, _, instErr := overlayFor(k.acct, k.app, rt.Host)
1677 if instErr != nil {
1678 if !said[k] {
1679 said[k] = true
1680 fmt.Printf(" %s/%s: instance config unresolved, config files left as-is: %v\n", k.acct, k.app, instErr)
1681 }
1682 continue
1683 }
1684 if r.m == nil && skipRemovals {
1685 continue
1686 }
1687```
1688
1689and, at the call site (`:469`):
1690
1691```go
1692 // The group's merge, overlaid by this instance's own layer. A nil
1693 // base stays nil: an absent manifest is a REMOVAL on this path, and
1694 // task 7 is the one case where an overlay changes that.
1695 m := r.m
1696 if m != nil {
1697 m = applyInstanceOverlay(k.app, m, inst)
1698 }
1699 syncExecHost(rt, m, Key(k.acct, k.app), priv, dataDir, hostBounce, header, ack)
1700```
1701
1702- [ ] **Step 4: Run it to verify it passes**
1703
1704```bash
1705go test ./internal/appconfig/ -count=1
1706```
1707
1708Expected: PASS.
1709
1710- [ ] **Step 5: Gate and commit**
1711
1712```bash
1713gofmt -w internal/appconfig/syncexec.go internal/appconfig/instances_test.go
1714go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
1715git add internal/appconfig
1716git commit -m "appconfig: the exec sync pass applies each route's instance overlay (plan 4, task 6)"
1717```
1718
1719---
1720
1721### Task 7: An overlay is not an absence, and the mass-removal guard says so
1722
1723**Files:**
1724- Modify: `internal/appconfig/syncexec.go` (the guard at `:341-380`, the removal branches at `:408-424`, the call site), `internal/appconfig/instances_test.go`
1725
1726**Interfaces:**
1727- Consumes: `skipRemovals` (`syncexec.go:375`), `release.HostAccountLabel` (`internal/release/qualified.go:188`).
1728- Produces: no new exported surface.
1729
1730**What this task decides, and why the guard's counting does not move.** `SyncExec` owns the `*.config.env` namespace, so an ABSENT manifest removes the file (`syncexec.go:954-963`). "Absent" was a group fact — no `config.json`, and no bound bundle — and with a third layer it is not: an instance with an overlay and an app with no default bundle is a configured route, and removing its file would strip the very credential this layer exists to deliver. So the per-route removal branches consult the overlay, while the mass-removal guard's COUNT stays exactly where it is, over group bases. That is deliberate: the guard exists for a registry-wide account mangle, where every key of every app 404s at once — the overlay listing included, since it hangs off the same mangled prefix — so counting overlays would not make it more sensitive, and a route with an overlay never reaches a removal branch anyway.
1731
1732- [ ] **Step 1: Write the failing test**
1733
1734`internal/appconfig/instances_test.go` (append):
1735
1736```go
1737// An app with no default bundle at all, and one instance with an overlay.
1738// The overlay is the whole configuration of that route, so its file is
1739// written, not removed — while its sibling, which has nothing anywhere, is
1740// reaped exactly as it is today.
1741func TestSyncExecConvergesAnOverlayAloneInsteadOfRemoving(t *testing.T) {
1742 ctx, s, reg, dd, privB64, _ := execKit(t)
1743 priv, err := ParsePrivateKey(privB64, "test")
1744 if err != nil {
1745 t.Fatalf("ParsePrivateKey: %v", err)
1746 }
1747 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
1748 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1749 t.Fatalf("ApplyInstance: %v", err)
1750 }
1751 for _, host := range []string{"book.example.com", "cart.example.com"} {
1752 if err := reg.Upsert(registry.Route{
1753 Host: host, Kind: "instance", App: "titogo", Account: "bab", Status: "ready",
1754 Socket: "/run/carlos/" + host + ".sock",
1755 }); err != nil {
1756 t.Fatalf("Upsert %s: %v", host, err)
1757 }
1758 if err := os.WriteFile(filepath.Join(dd, host+".config.env"), []byte("OLD=1\n"), 0o600); err != nil {
1759 t.Fatalf("stage %s: %v", host, err)
1760 }
1761 }
1762 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", nil); err != nil {
1763 t.Fatalf("SyncExec: %v", err)
1764 }
1765 book, err := os.ReadFile(filepath.Join(dd, "book.example.com.config.env"))
1766 if err != nil {
1767 t.Fatalf("the overlaid route's file was removed: %v", err)
1768 }
1769 if !bytes.Contains(book, []byte("TENANT=book\n")) || bytes.Contains(book, []byte("OLD=1")) {
1770 t.Fatalf("the overlay-alone render did not converge:\n%s", book)
1771 }
1772 if _, err := os.Stat(filepath.Join(dd, "cart.example.com.config.env")); !os.IsNotExist(err) {
1773 t.Fatalf("a route with no config anywhere kept its file: %v", err)
1774 }
1775}
1776
1777// The mass-removal guard, with an overlay in the middle of it. Two apps
1778// whose manifests all read absent still freeze every removal on the box,
1779// and the one route that has an overlay still converges: writing a file a
1780// bundle asked for is not a removal, and the guard exists to stop
1781// deletions, not delivery.
1782func TestSyncExecMassRemovalGuardHoldsRemovalsButNotOverlays(t *testing.T) {
1783 ctx, s, reg, dd, privB64, _ := execKit(t)
1784 priv, err := ParsePrivateKey(privB64, "test")
1785 if err != nil {
1786 t.Fatalf("ParsePrivateKey: %v", err)
1787 }
1788 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
1789 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
1790 t.Fatalf("ApplyInstance: %v", err)
1791 }
1792 routes := map[string]string{
1793 "book.example.com": "titogo",
1794 "cart.example.com": "titogo",
1795 "desk.example.com": "eleven",
1796 }
1797 for host, app := range routes {
1798 if err := reg.Upsert(registry.Route{
1799 Host: host, Kind: "instance", App: app, Account: "bab", Status: "ready",
1800 Socket: "/run/carlos/" + host + ".sock",
1801 }); err != nil {
1802 t.Fatalf("Upsert %s: %v", host, err)
1803 }
1804 if err := os.WriteFile(filepath.Join(dd, host+".config.env"), []byte("OLD=1\n"), 0o600); err != nil {
1805 t.Fatalf("stage %s: %v", host, err)
1806 }
1807 }
1808 out := captureStdout(t, func() {
1809 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", nil); err != nil {
1810 t.Fatalf("SyncExec: %v", err)
1811 }
1812 })
1813 if !strings.Contains(out, "refusing to remove any config files this pass") {
1814 t.Fatalf("the mass-removal guard did not fire:\n%s", out)
1815 }
1816 for _, host := range []string{"cart.example.com", "desk.example.com"} {
1817 cur, err := os.ReadFile(filepath.Join(dd, host+".config.env"))
1818 if err != nil || string(cur) != "OLD=1\n" {
1819 t.Fatalf("%s lost its file while the guard was up: %q %v", host, cur, err)
1820 }
1821 }
1822 book, err := os.ReadFile(filepath.Join(dd, "book.example.com.config.env"))
1823 if err != nil {
1824 t.Fatalf("the overlaid route was frozen by a guard about removals: %v", err)
1825 }
1826 if !bytes.Contains(book, []byte("TENANT=book\n")) {
1827 t.Fatalf("the overlay did not converge under the guard:\n%s", book)
1828 }
1829}
1830```
1831
1832- [ ] **Step 2: Run it to verify it fails**
1833
1834```bash
1835go test -run 'TestSyncExecConvergesAnOverlayAlone|TestSyncExecMassRemovalGuard' ./internal/appconfig/
1836```
1837
1838Expected: both fail — `book.example.com.config.env` is removed in the first and left at `OLD=1` in the second, because an absent base still decides the route on its own.
1839
1840- [ ] **Step 3: Write the code**
1841
1842`internal/appconfig/syncexec.go`, the two removal branches (`:408-424`) consult the overlay:
1843
1844```go
1845 // An overlay is configuration, so a route that has one is never
1846 // "absent" however its group base resolved. Both branches below decide
1847 // REMOVAL, and neither may fire for a route whose own layer is
1848 // standing right there.
1849 if r.m == nil && inst == nil && skipRemovals {
1850 continue
1851 }
1852 // The #344 guard: an absent manifest removes the config file ONLY
1853 // when the account it was looked up under agrees with the account
1854 // the host itself names. A platform host's <sqid> label is not a
1855 // moving part; the row's account column is (the domains reconcile
1856 // stamps it from bucket listings, and a mis-stamp deleted a live
1857 // tenant's delivered secrets on 2026-08-28). Held, loudly, like an
1858 // unresolved group — the fix is the row's account, not this file.
1859 if r.m == nil && inst == nil {
1860 if sq := release.HostAccountLabel(rt.Host, appsDomain); sq != "" && sq != k.acct {
1861 fmt.Printf(" %s: no config manifest under account %q, but the host names account %q — config file left as-is; fix the route row's account column (#344)\n",
1862 rt.Host, k.acct, sq)
1863 continue
1864 }
1865 }
1866```
1867
1868and the call site takes the overlay even with no base:
1869
1870```go
1871 // The group's merge, overlaid by this instance's own layer. A nil
1872 // base with an overlay is the overlay ALONE, which is a real
1873 // configuration and not the absence that drives a removal; nil and
1874 // nil together stay nil, which still removes the file.
1875 m := r.m
1876 if m != nil || inst != nil {
1877 m = applyInstanceOverlay(k.app, m, inst)
1878 }
1879 syncExecHost(rt, m, Key(k.acct, k.app), priv, dataDir, hostBounce, header, ack)
1880```
1881
1882`internal/appconfig/syncexec.go`, the mass-removal guard's comment (`:341-364`) gains the paragraph that says why its arithmetic did not move:
1883
1884```go
1885 // The instance layer does not enter this count, and does not need to.
1886 // The fault this guard exists for is registry-wide — a mangled account
1887 // column makes EVERY key of every app 404 at once, the overlay listing
1888 // included, since it hangs off the same mangled prefix — so an overlay
1889 // could not survive the fault it is counting. A route that does have an
1890 // overlay never reaches a removal branch at all (see below), so it is
1891 // neither protected nor endangered by this number.
1892```
1893
1894- [ ] **Step 4: Run it to verify it passes**
1895
1896```bash
1897go test ./internal/appconfig/ -count=1
1898```
1899
1900Expected: PASS.
1901
1902- [ ] **Step 5: Gate and commit**
1903
1904```bash
1905gofmt -w internal/appconfig/syncexec.go internal/appconfig/instances_test.go
1906go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
1907git add internal/appconfig
1908git commit -m "appconfig: an instance overlay is configuration, so it is never a removal (plan 4, task 7)"
1909```
1910
1911---
1912
1913### Task 8: `AwaitingRestart` and `allShadowed`, revisited for a third layer
1914
1915**Files:**
1916- Modify: `internal/appconfig/syncexec.go` (`AwaitingRestart` at `:771`, `allShadowed` at `:872`), `internal/appconfig/awaitingrestart_test.go`
1917
1918**Interfaces:**
1919- Consumes: `instanceOverlayLoader` (task 4), `resolveExecDelivery` (`syncexec.go:688`), `execAddrCandidate` (`syncexec.go:574`).
1920- Produces:
1921 ```go
1922 func allShadowed(keys []string, above ...*Manifest) bool // every key stated by at least one layer above the written one
1923 ```
1924 `AwaitingRestart` keeps its signature exactly: `(ctx, s, routes, acct, app, envName, keys) RestartPending`.
1925
1926**On exported surface that arrives before its caller.** This plan lands three readers whose only callers are step 5's: `LoadMergedHost` (task 3), and `LatestConfigAck` with `InstanceConfigDelivery` (task 10). That is deliberate — they are the decidable half of a question step 5 has to answer at a door, and proving an etag comparison means what it says belongs with the layer, not with the door. The ruling below is narrower than it reads: it is about inventing a NEW question nothing has asked yet, not about landing the answer to one step 5 is already known to ask.
1927
1928**Reporting an INSTANCE-layer write is step 5's, and that is a ruling.** `AwaitingRestart` answers "a write to app's `envName` bundle landed a file nothing will read": its whole vocabulary is (app, environment), and the CLI and console callers (`appconfig_cli.go:83`, `internal/console/apiconfig.go:544`) pass exactly what a member just wrote. Nothing in this plan writes an instance overlay, so there is no call site that would need the instance-layer question answered, and inventing one now would mean an exported function with no caller and no test but its own. What this task owes is the other direction, which is live the moment task 6 shipped: a write to the default or the named bundle can now be SHADOWED by an instance overlay, and naming such a route would send someone to restart a process for a value it was never going to see. Step 5 adds the instance-write reporting alongside the door that does the writing.
1929
1930- [ ] **Step 1: Write the failing test**
1931
1932`internal/appconfig/awaitingrestart_test.go` (append; add `context`, `os`, `strings` and `github.com/carlosframework/platform/internal/release` to the imports as needed):
1933
1934```go
1935// The third layer shadows a default-bundle write the same way a named
1936// bundle does, and for the same reason: the merged manifest does not move,
1937// so syncExecHost's bytes.Equal writes nothing and there is no file for
1938// anyone to restart over.
1939func TestAwaitingRestartInstanceOverlayShadowsADefaultWrite(t *testing.T) {
1940 ctx, s, _, _, _, _ := execKit(t)
1941 if err := SetExecDelivery(ctx, s, "bab", "relay", ExecDeliveryFile); err != nil {
1942 t.Fatalf("SetExecDelivery: %v", err)
1943 }
1944 if _, _, err := ApplyInstance(ctx, s, "bab", "relay", "connect.example.com",
1945 map[string]Var{"OWN_KEY": {Kind: "env", Value: "instance"}}, nil); err != nil {
1946 t.Fatalf("ApplyInstance: %v", err)
1947 }
1948 routes := []registry.Route{addrRoute("connect.example.com", "relay", "bab", "")}
1949
1950 if got := AwaitingRestart(ctx, s, routes, "bab", "relay", "", []string{"OWN_KEY"}); !got.Empty() {
1951 t.Fatalf("got %+v, want nothing: OWN_KEY is shadowed by this instance's own layer", got)
1952 }
1953 got := AwaitingRestart(ctx, s, routes, "bab", "relay", "", []string{"OTHER_KEY"})
1954 if len(got.Hosts) != 1 || got.Hosts[0].Host != "connect.example.com" {
1955 t.Fatalf("Hosts = %+v, want the route for a key its overlay does not state", got.Hosts)
1956 }
1957 got = AwaitingRestart(ctx, s, routes, "bab", "relay", "", []string{"OWN_KEY", "OTHER_KEY"})
1958 if len(got.Hosts) != 1 {
1959 t.Fatalf("Hosts = %+v, want the route when any key is unshadowed", got.Hosts)
1960 }
1961}
1962
1963// The case that did not exist before the third layer: a write to the
1964// route's OWN bound bundle used to be unshadowable, because nothing sat
1965// above it. Something does now.
1966func TestAwaitingRestartInstanceOverlayShadowsANamedBundleWrite(t *testing.T) {
1967 ctx, s, _, _, _, _ := execKit(t)
1968 if err := SetExecDelivery(ctx, s, "bab", "relay", ExecDeliveryFile); err != nil {
1969 t.Fatalf("SetExecDelivery: %v", err)
1970 }
1971 if _, err := ApplyEnvironment(ctx, s, "bab", "relay", "prod",
1972 map[string]Var{"K": {Kind: "env", Value: "environment"}}, nil); err != nil {
1973 t.Fatalf("ApplyEnvironment: %v", err)
1974 }
1975 if _, _, err := ApplyInstance(ctx, s, "bab", "relay", "connect.example.com",
1976 map[string]Var{"K": {Kind: "env", Value: "instance"}}, nil); err != nil {
1977 t.Fatalf("ApplyInstance: %v", err)
1978 }
1979 routes := []registry.Route{addrRoute("connect.example.com", "relay", "bab", "prod")}
1980
1981 if got := AwaitingRestart(ctx, s, routes, "bab", "relay", "prod", []string{"K"}); !got.Empty() {
1982 t.Fatalf("got %+v, want nothing: K is shadowed by this instance's own layer", got)
1983 }
1984 if got := AwaitingRestart(ctx, s, routes, "bab", "relay", "prod", []string{"OTHER"}); len(got.Hosts) != 1 {
1985 t.Fatalf("Hosts = %+v, want the route for an unshadowed key", got.Hosts)
1986 }
1987}
1988
1989// A read this function could not make is not a determination, on this layer
1990// as on the other two: the route is named as unresolved rather than
1991// reported either way.
1992func TestAwaitingRestartUnresolvedWhenTheOverlayCannotBeRead(t *testing.T) {
1993 ctx, s, _, _, _, _ := execKit(t)
1994 if err := SetExecDelivery(ctx, s, "bab", "relay", ExecDeliveryFile); err != nil {
1995 t.Fatalf("SetExecDelivery: %v", err)
1996 }
1997 if _, _, err := ApplyInstance(ctx, s, "bab", "relay", "connect.example.com",
1998 map[string]Var{"OWN_KEY": {Kind: "env", Value: "instance"}}, nil); err != nil {
1999 t.Fatalf("ApplyInstance: %v", err)
2000 }
2001 blip := &failingKeyStore{Store: s, failKey: InstanceConfigKey("bab", "relay", "connect.example.com"),
2002 err: errors.New("s3: transient 503 SlowDown")}
2003 routes := []registry.Route{addrRoute("connect.example.com", "relay", "bab", "")}
2004 got := AwaitingRestart(ctx, blip, routes, "bab", "relay", "", []string{"OWN_KEY"})
2005 if len(got.Hosts) != 0 || len(got.Unresolved) != 1 || got.Unresolved[0] != "connect.example.com" {
2006 t.Fatalf("got %+v, want the host as unresolved and nothing claimed either way", got)
2007 }
2008}
2009```
2010
2011- [ ] **Step 2: Run it to verify it fails**
2012
2013```bash
2014go test -run 'TestAwaitingRestartInstanceOverlay|TestAwaitingRestartUnresolvedWhenTheOverlay' ./internal/appconfig/
2015```
2016
2017Expected: all three fail — the route is named in every case, because nothing consults the third layer.
2018
2019- [ ] **Step 3: Write the code**
2020
2021`internal/appconfig/syncexec.go`, `AwaitingRestart` (`:771`): one more loader beside the two caches, and the shadow test takes every layer above the bundle that was written.
2022
2023```go
2024 // The instance layer, through the same loader the sweeps use, so this
2025 // notice and the pass that writes the files agree about what sits above
2026 // a bundle. One listing for the app, then a Get per route that has an
2027 // overlay; both live for one call, like the caches above.
2028 overlayFor := instanceOverlayLoader(ctx, s)
2029 // One line per app when its listing fails, not one per route: every
2030 // route of that app fails identically and for the same reason.
2031 said := map[groupKey]bool{}
2032```
2033
2034Inside the route loop, after the opt-in verdict's `if !v.On { continue }` (`:841-843`):
2035
2036```go
2037 // Which layers sit ABOVE the bundle that was written, for this
2038 // route. A write cannot change what a route resolves to for a key
2039 // some higher layer states, and the third layer sits above both of
2040 // the other two.
2041 inst, _, err := overlayFor(acct, app, rt.Host)
2042 if err != nil {
2043 out.Unresolved = append(out.Unresolved, rt.Host)
2044 continue
2045 }
2046 var above []*Manifest
2047 if envName == "" && rt.Environment != "" {
2048 above = append(above, eb.m)
2049 }
2050 above = append(above, inst)
2051 // The shadowing test, and the reason keys is a parameter: where every
2052 // touched key is shadowed, the merged manifest is byte-identical,
2053 // syncExecHost's bytes.Equal writes nothing, and naming the route
2054 // would send someone to restart a process for a value it was never
2055 // going to see.
2056 if len(keys) > 0 && allShadowed(keys, above...) {
2057 continue
2058 }
2059 out.Hosts = append(out.Hosts, RestartHost{Host: rt.Host, Unit: rt.Unit})
2060```
2061
2062The old shadow test (`:844-857`) is replaced by the block above, including its comment: the "only for a default-bundle write reaching a BOUND route" carve-out is gone, because a named write can now be shadowed too.
2063
2064`internal/appconfig/syncexec.go`, `allShadowed` (`:868-882`):
2065
2066```go
2067// allShadowed reports whether every one of keys is STATED by at least one
2068// of the layers above the bundle a write touched — the test that decides
2069// whether that write can have changed what a route actually resolves to. An
2070// unset key counts the same as a set one: removing a key a higher layer
2071// shadows changes the merge by nothing.
2072//
2073// above is variadic because how many layers sit over a write depends on
2074// which bundle was written: a default-bundle write to a bound route is
2075// under both the named bundle and the instance overlay, a named-bundle
2076// write is under the overlay alone, and a write to the overlay itself has
2077// nothing over it. A nil layer (no bundle of that kind for this route) is
2078// skipped rather than treated as stating nothing, so an empty list answers
2079// false and the route is named.
2080func allShadowed(keys []string, above ...*Manifest) bool {
2081 for _, k := range keys {
2082 stated := false
2083 for _, m := range above {
2084 if m == nil {
2085 continue
2086 }
2087 if _, ok := m.Vars[k]; ok {
2088 stated = true
2089 break
2090 }
2091 }
2092 if !stated {
2093 return false
2094 }
2095 }
2096 return true
2097}
2098```
2099
2100- [ ] **Step 4: Run it to verify it passes**
2101
2102```bash
2103go test ./internal/appconfig/ -count=1 && go test ./internal/console/ ./ -count=1
2104```
2105
2106Expected: PASS, including `TestAwaitingRestartDefaultWriteReachesBoundRoute` and `TestAPIConfigPutNamesRoutesAwaitingRestart` unchanged.
2107
2108- [ ] **Step 5: Gate and commit**
2109
2110```bash
2111gofmt -w internal/appconfig/syncexec.go internal/appconfig/awaitingrestart_test.go
2112go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
2113git add internal/appconfig
2114git commit -m "appconfig: the restart notice knows an instance overlay shadows a bundle write (plan 4, task 8)"
2115```
2116
2117---
2118
2119### Task 9: `ConfigAck` says which agent wrote it and which overlay it rendered
2120
2121**Files:**
2122- Modify: `internal/appconfig/configack.go` (`ConfigAck` at `:36-64`), `internal/appconfig/sync.go` (`syncHost` at `:343`, its ack at `:451`), `internal/appconfig/syncexec.go` (`syncExecHost` at `:948`, its ack at `:1046`), `internal/appconfig/configack_test.go`
2123- Modify, because they call the two functions this task re-signs and the package will not compile without them: `internal/appconfig/sync_test.go` (`syncHost` at `:403`, `:1057`, `:1075`) and `internal/appconfig/syncexec_test.go` (`syncExecHost` at `:776`). Each gets `""` for the new `instETag`, in the position shown in the Produces block below: after `restart` for `syncHost`, after `header` for `syncExecHost`. These are four one-word edits, and skipping them makes Step 4's "Expected: PASS" impossible to reach.
2124
2125**Interfaces:**
2126- Produces:
2127 ```go
2128 const ConfigAckSchema = 1
2129 type ConfigAck struct { ...; Schema int `json:"schema"`; InstanceConfigETag string `json:"instance_config_etag,omitempty"` }
2130 func syncHost(rt registry.Route, m *Manifest, defaultMissing bool, priv *ecdh.PrivateKey, envDir string, restart func(host string) error, instETag string, ack func(ConfigAck))
2131 func syncExecHost(rt registry.Route, m *Manifest, key string, priv *ecdh.PrivateKey, dataDir string, bounce func(host string) error, header, instETag string, ack func(ConfigAck))
2132 ```
2133
2134- [ ] **Step 1: Write the failing test**
2135
2136`internal/appconfig/configack_test.go` (append):
2137
2138```go
2139// The ack says which version of the instance layer the agent rendered
2140// (design §6.5). The console holds the etag its own write returned and
2141// compares the two, which is decidable in a way a map of delivered key
2142// names is not: an old agent and a new agent rendering an overlay that sets
2143// nothing look identical by key, and not at all by etag.
2144func TestConfigAckNamesTheInstanceOverlayItRendered(t *testing.T) {
2145 ctx := context.Background()
2146 s := release.NewDirStore(t.TempDir())
2147 reg := testRegistry(t)
2148 envDir := t.TempDir()
2149 priv, _ := syncRelayTestKeys(t)
2150
2151 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
2152 t.Fatalf("SetEnv: %v", err)
2153 }
2154 _, want, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
2155 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil)
2156 if err != nil {
2157 t.Fatalf("ApplyInstance: %v", err)
2158 }
2159 for _, host := range []string{"book.example.com", "cart.example.com"} {
2160 if err := reg.Upsert(registry.Route{
2161 Host: host, Kind: "instance", App: "titogo", Account: "bab", Status: "ready",
2162 Backing: registry.BackingUnit,
2163 }); err != nil {
2164 t.Fatalf("Upsert %s: %v", host, err)
2165 }
2166 }
2167 acks := map[string]ConfigAck{}
2168 if err := Sync(ctx, reg, s, priv, envDir, func(string) error { return nil },
2169 func(a ConfigAck) { acks[a.Host] = a }); err != nil {
2170 t.Fatalf("Sync: %v", err)
2171 }
2172 book, ok := acks["book.example.com"]
2173 if !ok {
2174 t.Fatal("no ack for the overlaid route")
2175 }
2176 if book.Schema != ConfigAckSchema {
2177 t.Fatalf("Schema = %d, want %d — absence is how an older agent is recognised", book.Schema, ConfigAckSchema)
2178 }
2179 if book.InstanceConfigETag != want {
2180 t.Fatalf("InstanceConfigETag = %q, want the etag the write returned %q", book.InstanceConfigETag, want)
2181 }
2182 cart, ok := acks["cart.example.com"]
2183 if !ok {
2184 t.Fatal("no ack for the sibling")
2185 }
2186 if cart.Schema != ConfigAckSchema || cart.InstanceConfigETag != "" {
2187 t.Fatalf("sibling ack = %+v, want the schema and an empty etag: it has no overlay to render", cart)
2188 }
2189}
2190
2191// The same statement from the exec half, which is the half an instance
2192// sender's box actually runs (design §6.4's delivery check).
2193func TestSyncExecConfigAckNamesTheInstanceOverlayItRendered(t *testing.T) {
2194 ctx, s, reg, dd, privB64, _ := execKit(t)
2195 priv, err := ParsePrivateKey(privB64, "test")
2196 if err != nil {
2197 t.Fatalf("ParsePrivateKey: %v", err)
2198 }
2199 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
2200 t.Fatalf("SetEnv: %v", err)
2201 }
2202 _, want, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
2203 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil)
2204 if err != nil {
2205 t.Fatalf("ApplyInstance: %v", err)
2206 }
2207 if err := reg.Upsert(registry.Route{
2208 Host: "book.example.com", Kind: "instance", App: "titogo", Account: "bab",
2209 Status: "ready", Socket: "/run/carlos/book.sock",
2210 }); err != nil {
2211 t.Fatalf("Upsert: %v", err)
2212 }
2213 var got []ConfigAck
2214 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "",
2215 func(a ConfigAck) { got = append(got, a) }); err != nil {
2216 t.Fatalf("SyncExec: %v", err)
2217 }
2218 if len(got) != 1 {
2219 t.Fatalf("acks = %+v, want exactly one", got)
2220 }
2221 if got[0].Schema != ConfigAckSchema || got[0].InstanceConfigETag != want {
2222 t.Fatalf("ack = %+v, want schema %d and etag %q", got[0], ConfigAckSchema, want)
2223 }
2224}
2225```
2226
2227- [ ] **Step 2: Run it to verify it fails**
2228
2229```bash
2230go test -run 'ConfigAckNamesTheInstanceOverlay' ./internal/appconfig/
2231```
2232
2233Expected: build failure, `undefined: ConfigAckSchema`.
2234
2235- [ ] **Step 3: Write the code**
2236
2237`internal/appconfig/configack.go`, on `ConfigAck` (`:36`):
2238
2239```go
2240// ConfigAckSchema is the ack schema THIS binary writes. It exists so a
2241// reader can tell an agent that renders the instance config layer (senders
2242// design §6.2) from one that does not, and the only honest way to do that
2243// is a positive statement: an ack written by an older agent decodes with
2244// Schema 0, because the field was not there to write.
2245//
2246// It is not the manifest schema and is not tied to it. Bump it when an
2247// agent starts stating something a reader may need to depend on.
2248const ConfigAckSchema = 1
2249```
2250
2251and, inside the struct:
2252
2253```go
2254 // Schema is ConfigAckSchema as written by the agent that produced this
2255 // ack, and 0 for every ack written before this field existed. A reader
2256 // asking "does this box render the instance config layer" reads this and
2257 // nothing else: absence of an ack, or an ack at schema 0, both mean the
2258 // question cannot be answered from the box's own statements yet.
2259 Schema int `json:"schema"`
2260 // InstanceConfigETag is the etag of the instance overlay
2261 // (apps/<acct>/<app>/instances-config/<host>.json) this pass rendered
2262 // into the file, empty when the route has no overlay at all. The writer
2263 // keeps the etag its own Put returned and compares the two
2264 // (InstanceConfigDelivery): equal is delivered, different is stale. A
2265 // map of delivered var names could not make that distinction — an agent
2266 // too old to read the layer and a new agent rendering an overlay whose
2267 // vars the reader does not recognise produce the same map.
2268 InstanceConfigETag string `json:"instance_config_etag,omitempty"`
2269```
2270
2271`internal/appconfig/sync.go`: `syncHost` takes `instETag string` before `ack`, the materialization loop keeps the etag it already fetched (`inst, instETag, err := overlayFor(...)`) and passes it, and the ack at `:451` becomes
2272
2273```go
2274 a := ConfigAck{Schema: ConfigAckSchema, Host: rt.Host, Shape: shapeOf(rt), At: time.Now(), InstanceConfigETag: instETag}
2275```
2276
2277`internal/appconfig/syncexec.go`: `syncExecHost` takes `instETag string` after `header`, the materialization loop passes the etag from its own `overlayFor` call, and the ack at `:1046` becomes
2278
2279```go
2280 a := ConfigAck{Schema: ConfigAckSchema, Host: rt.Host, Shape: shapeOf(rt), At: time.Now(), InstanceConfigETag: instETag}
2281```
2282
2283- [ ] **Step 4: Run it to verify it passes**
2284
2285```bash
2286go test ./internal/appconfig/ ./internal/console/ -count=1
2287```
2288
2289Expected: PASS. The console's `relayConfigAcks` decodes the two new fields without being touched, since they are additive.
2290
2291- [ ] **Step 5: Gate and commit**
2292
2293```bash
2294gofmt -w internal/appconfig/configack.go internal/appconfig/sync.go internal/appconfig/syncexec.go internal/appconfig/configack_test.go
2295go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
2296git add internal/appconfig
2297git commit -m "appconfig: the config ack carries its schema and the instance overlay it rendered (plan 4, task 9)"
2298```
2299
2300---
2301
2302### Task 10: Reading the ack: delivered, stale, or not yet delivered
2303
2304**Files:**
2305- Modify: `internal/appconfig/configack.go` (`ConfigAckKey` at `:69`), `internal/appconfig/configack_test.go`
2306
2307**Interfaces:**
2308- Produces:
2309 ```go
2310 const ConfigAckPrefix = "control/config-ack/"
2311 const (
2312 InstanceConfigDelivered = "delivered"
2313 InstanceConfigStale = "stale"
2314 InstanceConfigUnknownAgent = "unknown-agent"
2315 InstanceConfigNotYetDelivered = "not-yet-delivered"
2316 )
2317 func LatestConfigAck(ctx context.Context, s release.Store, host string) (*ConfigAck, error)
2318 func InstanceConfigDelivery(a *ConfigAck, want string) string
2319 ```
2320
2321**Who calls this, and when.** Nothing in this plan. Step 5's instance `enable` refuses a box that cannot render the layer, and `carlos email status --instance <host>` prints delivered, stale or not yet delivered (design §6.4, §6.5); both are that step's doors. What lands here is the decidable part, with the four states named and tested, so step 5 spends its judgement on the refusal's wording rather than on re-deriving what an etag comparison means. The console's own `configAckPrefix` (`internal/console/relayops.go:73`) is the same string and is left alone: `relayConfigAcks` batches a whole app's routes through one listing and has no reason to change for this.
2322
2323- [ ] **Step 1: Write the failing test**
2324
2325`internal/appconfig/configack_test.go` (append):
2326
2327```go
2328// The four answers, and they are four rather than two on purpose. "This
2329// agent is too old to render the layer" and "this agent rendered an older
2330// version of it" call for different sentences from whoever asks: one is a
2331// box to roll, the other is a pass to wait for.
2332func TestInstanceConfigDeliveryVerdicts(t *testing.T) {
2333 cases := []struct {
2334 name string
2335 ack *ConfigAck
2336 want string
2337 got string
2338 }{
2339 {"not yet acked", nil, "etag-1", InstanceConfigNotYetDelivered},
2340 {"agent older than the layer", &ConfigAck{Schema: 0, InstanceConfigETag: ""}, "etag-1", InstanceConfigUnknownAgent},
2341 {"rendered this overlay", &ConfigAck{Schema: ConfigAckSchema, InstanceConfigETag: "etag-1"}, "etag-1", InstanceConfigDelivered},
2342 {"rendered an older one", &ConfigAck{Schema: ConfigAckSchema, InstanceConfigETag: "etag-0"}, "etag-1", InstanceConfigStale},
2343 {"no overlay, and none expected", &ConfigAck{Schema: ConfigAckSchema}, "", InstanceConfigDelivered},
2344 {"no overlay, but one was written", &ConfigAck{Schema: ConfigAckSchema}, "etag-1", InstanceConfigStale},
2345 }
2346 for _, c := range cases {
2347 if got := InstanceConfigDelivery(c.ack, c.want); got != c.got {
2348 t.Errorf("%s: InstanceConfigDelivery = %q, want %q", c.name, got, c.got)
2349 }
2350 }
2351}
2352
2353// One route can have acks from more than one box across a migration, so the
2354// reader takes the freshest and ignores every other host's.
2355func TestLatestConfigAckPicksTheFreshestForTheHost(t *testing.T) {
2356 ctx := context.Background()
2357 s := release.NewDirStore(t.TempDir())
2358 older := ConfigAck{Schema: ConfigAckSchema, Box: "box-1", Host: "book.example.com", Shape: "unit",
2359 At: time.Date(2026, 9, 15, 10, 0, 0, 0, time.UTC), InstanceConfigETag: "etag-0"}
2360 newer := ConfigAck{Schema: ConfigAckSchema, Box: "box-2", Host: "book.example.com", Shape: "unit",
2361 At: time.Date(2026, 9, 15, 12, 0, 0, 0, time.UTC), InstanceConfigETag: "etag-1"}
2362 other := ConfigAck{Schema: ConfigAckSchema, Box: "box-2", Host: "cart.example.com", Shape: "unit",
2363 At: time.Date(2026, 9, 15, 13, 0, 0, 0, time.UTC), InstanceConfigETag: "etag-9"}
2364 for _, a := range []ConfigAck{older, newer, other} {
2365 if err := WriteConfigAck(ctx, s, a); err != nil {
2366 t.Fatalf("WriteConfigAck %s/%s: %v", a.Box, a.Host, err)
2367 }
2368 }
2369 got, err := LatestConfigAck(ctx, s, "book.example.com")
2370 if err != nil || got == nil {
2371 t.Fatalf("LatestConfigAck = %+v %v", got, err)
2372 }
2373 if got.Box != "box-2" || got.InstanceConfigETag != "etag-1" {
2374 t.Fatalf("LatestConfigAck = %+v, want box-2's etag-1", got)
2375 }
2376 if v := InstanceConfigDelivery(got, "etag-1"); v != InstanceConfigDelivered {
2377 t.Fatalf("delivery = %q, want %q", v, InstanceConfigDelivered)
2378 }
2379
2380 none, err := LatestConfigAck(ctx, s, "desk.example.com")
2381 if err != nil || none != nil {
2382 t.Fatalf("LatestConfigAck for a host with no ack = %+v %v, want (nil, nil)", none, err)
2383 }
2384 if v := InstanceConfigDelivery(none, "etag-1"); v != InstanceConfigNotYetDelivered {
2385 t.Fatalf("delivery = %q, want %q", v, InstanceConfigNotYetDelivered)
2386 }
2387}
2388```
2389
2390- [ ] **Step 2: Run it to verify it fails**
2391
2392```bash
2393go test -run 'TestInstanceConfigDeliveryVerdicts|TestLatestConfigAck' ./internal/appconfig/
2394```
2395
2396Expected: build failure, `undefined: InstanceConfigDelivery`.
2397
2398- [ ] **Step 3: Write the code**
2399
2400`internal/appconfig/configack.go`, `ConfigAckKey` (`:69`) gains a named prefix, and the reader follows it:
2401
2402```go
2403// ConfigAckPrefix is where every route's config-adoption ack lives, one
2404// object per (box, route host).
2405const ConfigAckPrefix = "control/config-ack/"
2406
2407// ConfigAckKey is where a route's config-adoption ack lives: additive, one
2408// object per (box, route host), rewritten (never appended to) on every sync
2409// pass, by WriteConfigAck, when what it delivers changes
2410// it did not.
2411func ConfigAckKey(box, host string) string {
2412 return ConfigAckPrefix + box + "/" + host + ".json"
2413}
2414
2415// The four answers to "is this instance running the overlay I wrote", named
2416// because three of them are not "no" and a caller that flattened them would
2417// tell an operator to roll a box that is merely one pass behind.
2418const (
2419 // InstanceConfigDelivered: the agent rendered exactly this version of
2420 // the overlay. A route with no overlay, whose caller expects none, is
2421 // also delivered: there is nothing outstanding.
2422 InstanceConfigDelivered = "delivered"
2423 // InstanceConfigStale: the agent rendered a different version. One more
2424 // pass, or a generation that never got read, is the usual cause.
2425 InstanceConfigStale = "stale"
2426 // InstanceConfigUnknownAgent: this box's agent predates the instance
2427 // config layer, so its ack says nothing about it. Not "no" — the answer
2428 // cannot be had until the box is rolled.
2429 InstanceConfigUnknownAgent = "unknown-agent"
2430 // InstanceConfigNotYetDelivered: no ack for this host at all. Acks are
2431 // change-only, so this is the honest answer for a route whose file has
2432 // not moved since its agent rolled. It never means "old agent": that
2433 // question is the box record's agent version, which step 5 reads.
2434 InstanceConfigNotYetDelivered = "not-yet-delivered"
2435)
2436
2437// InstanceConfigDelivery compares an ack with the etag a write returned and
2438// says which of the four states the route is in. want is the etag of the
2439// overlay the caller wrote, or "" when the caller wrote none.
2440//
2441// The comparison is on etags rather than on delivered var names because
2442// only etags are decidable: an agent too old to read the layer and a new
2443// agent rendering an overlay whose vars the reader does not recognise
2444// deliver the same set of names (design §6.5).
2445func InstanceConfigDelivery(a *ConfigAck, want string) string {
2446 switch {
2447 case a == nil:
2448 return InstanceConfigNotYetDelivered
2449 case a.Schema < ConfigAckSchema:
2450 return InstanceConfigUnknownAgent
2451 case a.InstanceConfigETag == want:
2452 return InstanceConfigDelivered
2453 default:
2454 return InstanceConfigStale
2455 }
2456}
2457
2458// LatestConfigAck reads the freshest ack naming host, across every box that
2459// has written one, and nil (with no error) when nothing has. A route can
2460// legitimately have acks from two boxes at once across a migration, and the
2461// freshest is the one describing what is running now.
2462//
2463// It lists the whole ack prefix, which is the same limit relayConfigAcks
2464// accepts (internal/console/relayops.go): there is no per-host box list to
2465// narrow a listing by. The Gets are narrowed, though — only keys whose
2466// basename is this host are read.
2467func LatestConfigAck(ctx context.Context, s release.Store, host string) (*ConfigAck, error) {
2468 keys, err := s.List(ctx, ConfigAckPrefix)
2469 if err != nil {
2470 return nil, err
2471 }
2472 var best *ConfigAck
2473 for _, key := range keys {
2474 if !strings.HasSuffix(key, "/"+host+".json") {
2475 continue
2476 }
2477 body, _, err := s.Get(ctx, key)
2478 if err != nil {
2479 if errors.Is(err, os.ErrNotExist) {
2480 continue
2481 }
2482 return nil, err
2483 }
2484 var a ConfigAck
2485 if err := json.Unmarshal(body, &a); err != nil {
2486 // One unreadable object must not hide the box that answered
2487 // properly, and there is nothing in it to report either way.
2488 log.Printf("config: config ack %s: %v", key, err)
2489 continue
2490 }
2491 if best == nil || a.At.After(best.At) {
2492 latest := a
2493 best = &latest
2494 }
2495 }
2496 return best, nil
2497}
2498```
2499
2500Add `"log"` and `"strings"` to `configack.go`'s imports.
2501
2502- [ ] **Step 4: Run it to verify it passes**
2503
2504```bash
2505go test ./internal/appconfig/ -count=1
2506```
2507
2508Expected: PASS.
2509
2510- [ ] **Step 5: Gate and commit**
2511
2512```bash
2513gofmt -w internal/appconfig/configack.go internal/appconfig/configack_test.go
2514go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
2515git add internal/appconfig
2516git commit -m "appconfig: read a route's ack and say delivered, stale or not yet delivered (plan 4, task 10)"
2517```
2518
2519---
2520
2521### Task 11: The signal, and one pass end to end
2522
2523**Files:**
2524- Modify: `internal/appconfig/appconfig.go` (the package's bucket layout at `:6-11`), `internal/appconfig/routefingerprint.go` (`ConfigRouteFingerprints`'s doc at `:34-57`), `internal/appconfig/instances_test.go`
2525
2526**Interfaces:** none new. This task pins the two facts the whole layer rests on and walks one pass over a box that has every shape of it.
2527
2528**Why the fingerprint does not move, and must not.** `ConfigRouteFingerprints` digests registry columns — host, account, app, environment (`routefingerprint.go:67`) — and the instance layer is a bucket object. It cannot see one being written and should not try: reading the layer per route per tick is exactly the cost the fast loop exists to avoid, the same reasoning the doc already gives for leaving `exec_delivery` out (`:47-52`). The generation object is the signal, which is why task 2 bumps it on every write and delete.
2529
2530- [ ] **Step 1: Write the failing test**
2531
2532`internal/appconfig/instances_test.go` (append):
2533
2534```go
2535// The layer's only convergence signal is the generation object. The route
2536// fingerprint cannot see a bucket write and must not start paying to try:
2537// this pins both halves of that at once.
2538func TestWritingAnOverlayMovesTheGenerationAndNotTheFingerprint(t *testing.T) {
2539 ctx := context.Background()
2540 s := release.NewDirStore(t.TempDir())
2541 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes"}); err != nil {
2542 t.Fatalf("SetEnv: %v", err)
2543 }
2544 routes := []registry.Route{{
2545 Host: "book.example.com", Kind: "instance", App: "titogo", Account: "bab",
2546 Status: "ready", Socket: "/run/carlos/book.sock",
2547 }}
2548 execBefore, unitBefore := ConfigRouteFingerprints(routes)
2549 _, genBefore, err := s.Get(ctx, GenKey("bab", "titogo"))
2550 if err != nil {
2551 t.Fatalf("read generation: %v", err)
2552 }
2553
2554 if _, _, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
2555 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil); err != nil {
2556 t.Fatalf("ApplyInstance: %v", err)
2557 }
2558
2559 execAfter, unitAfter := ConfigRouteFingerprints(routes)
2560 if execAfter != execBefore || unitAfter != unitBefore {
2561 t.Fatal("the route fingerprint moved for a bucket-side write, which would cost a read per route per tick")
2562 }
2563 _, genAfter, err := s.Get(ctx, GenKey("bab", "titogo"))
2564 if err != nil {
2565 t.Fatalf("read generation: %v", err)
2566 }
2567 if genAfter == genBefore {
2568 t.Fatal("the generation did not move, so the fast loop has nothing to notice and delivery waits out the hourly floor")
2569 }
2570}
2571
2572// One pass over a box holding every shape this layer comes in: an instance
2573// with an overlay over its app's bundle, a sibling with none, and an
2574// instance whose app has no bundle at all so its overlay stands alone.
2575// Then the overlay is removed and the route converges back, with the ack
2576// saying so.
2577func TestOnePassOverEveryInstanceLayerShape(t *testing.T) {
2578 ctx, s, reg, dd, privB64, _ := execKit(t)
2579 priv, err := ParsePrivateKey(privB64, "test")
2580 if err != nil {
2581 t.Fatalf("ParsePrivateKey: %v", err)
2582 }
2583 if err := SetEnv(ctx, s, "bab", "titogo", map[string]string{"APP_WIDE": "yes", "TENANT": "none"}); err != nil {
2584 t.Fatalf("SetEnv: %v", err)
2585 }
2586 _, bookETag, err := ApplyInstance(ctx, s, "bab", "titogo", "book.example.com",
2587 map[string]Var{"TENANT": {Kind: "env", Value: "book"}}, nil)
2588 if err != nil {
2589 t.Fatalf("ApplyInstance (book): %v", err)
2590 }
2591 _, deskETag, err := ApplyInstance(ctx, s, "bab", "solo", "desk.example.com",
2592 map[string]Var{"TENANT": {Kind: "env", Value: "desk"}}, nil)
2593 if err != nil {
2594 t.Fatalf("ApplyInstance (desk): %v", err)
2595 }
2596 for host, app := range map[string]string{
2597 "book.example.com": "titogo", "cart.example.com": "titogo", "desk.example.com": "solo"} {
2598 if err := reg.Upsert(registry.Route{
2599 Host: host, Kind: "instance", App: app, Account: "bab", Status: "ready",
2600 Socket: "/run/carlos/" + host + ".sock",
2601 }); err != nil {
2602 t.Fatalf("Upsert %s: %v", host, err)
2603 }
2604 }
2605
2606 acks := map[string]ConfigAck{}
2607 collect := func(a ConfigAck) { acks[a.Host] = a }
2608 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", collect); err != nil {
2609 t.Fatalf("SyncExec: %v", err)
2610 }
2611 read := func(host string) string {
2612 t.Helper()
2613 body, err := os.ReadFile(filepath.Join(dd, host+".config.env"))
2614 if err != nil {
2615 t.Fatalf("read %s: %v", host, err)
2616 }
2617 return string(body)
2618 }
2619 if got := read("book.example.com"); !strings.Contains(got, "TENANT=book\n") || !strings.Contains(got, "APP_WIDE=yes\n") {
2620 t.Fatalf("book: %q", got)
2621 }
2622 if got := read("cart.example.com"); !strings.Contains(got, "TENANT=none\n") {
2623 t.Fatalf("cart saw another instance's config: %q", got)
2624 }
2625 if got := read("desk.example.com"); got != "TENANT=desk\n" {
2626 t.Fatalf("desk, whose app has no bundle, should render its overlay alone: %q", got)
2627 }
2628 for host, want := range map[string]string{
2629 "book.example.com": bookETag, "cart.example.com": "", "desk.example.com": deskETag} {
2630 a, ok := acks[host]
2631 if !ok {
2632 t.Fatalf("no ack for %s", host)
2633 }
2634 if a.Schema != ConfigAckSchema || a.InstanceConfigETag != want {
2635 t.Fatalf("%s ack = %+v, want schema %d and etag %q", host, a, ConfigAckSchema, want)
2636 }
2637 if v := InstanceConfigDelivery(&a, want); v != InstanceConfigDelivered {
2638 t.Fatalf("%s delivery = %q, want %q", host, v, InstanceConfigDelivered)
2639 }
2640 }
2641
2642 // Remove one overlay: that route converges back to its app's merge, and
2643 // its ack stops naming an overlay, which is what tells a reader the
2644 // removal actually reached the box.
2645 if err := DeleteInstanceConfig(ctx, s, "bab", "titogo", "book.example.com"); err != nil {
2646 t.Fatalf("DeleteInstanceConfig: %v", err)
2647 }
2648 acks = map[string]ConfigAck{}
2649 if err := SyncExec(ctx, reg, s, priv, dd, func(string) error { return nil }, "", collect); err != nil {
2650 t.Fatalf("SyncExec (after delete): %v", err)
2651 }
2652 if got := read("book.example.com"); !strings.Contains(got, "TENANT=none\n") {
2653 t.Fatalf("book did not converge back to its app's merge: %q", got)
2654 }
2655 if a := acks["book.example.com"]; a.InstanceConfigETag != "" || a.Schema != ConfigAckSchema {
2656 t.Fatalf("book ack after the delete = %+v, want the schema and no overlay", a)
2657 }
2658 if got := read("desk.example.com"); got != "TENANT=desk\n" {
2659 t.Fatalf("desk was disturbed by a sibling app's delete: %q", got)
2660 }
2661}
2662```
2663
2664- [ ] **Step 2: Run it to verify it fails**
2665
2666```bash
2667go test -run 'TestWritingAnOverlayMovesTheGeneration|TestOnePassOverEveryInstanceLayerShape' ./internal/appconfig/
2668```
2669
2670Expected: PASS, both of them, on the first run. Tasks 1 to 11 built everything these assert; this task exists to pin the composition and the two facts nothing else states, and a test that fails here means an earlier task's behaviour is not what that task claimed. If either fails, fix the earlier task rather than this test.
2671
2672- [ ] **Step 3: Write the code**
2673
2674Documentation only. `internal/appconfig/appconfig.go`, the package doc's bucket layout (`:6-11`):
2675
2676```go
2677// Bucket layout (deployment bucket, account-scoped like every other app key
2678// internal/release writes):
2679//
2680// apps/<acct>/<app>/config.json CAS-guarded config manifest
2681// apps/<acct>/<app>/environments/<name>.json named environment overlay
2682// apps/<acct>/<app>/instances-config/<host>.json one instance's overlay
2683//
2684// The three resolve default < environment < instance, per key, and only a
2685// caller that names a host gets the third (LoadMergedHost, instances.go).
2686```
2687
2688`internal/appconfig/routefingerprint.go`, `ConfigRouteFingerprints`'s doc (after the `exec_delivery` paragraph at `:47-52`):
2689
2690```go
2691// The INSTANCE CONFIG LAYER is deliberately not here either, and for the
2692// same reason with a sharper edge: apps/<acct>/<app>/instances-config/
2693// <host>.json is a bucket object, so seeing a change to one would cost a
2694// listing per app per tick at best and a read per route at worst. It does
2695// not need to be seen here — writing or deleting an overlay bumps the app's
2696// generation object (ApplyInstance, DeleteInstanceConfig), which this loop
2697// already watches, and that is the layer's whole signal.
2698```
2699
2700- [ ] **Step 4: Run the whole package**
2701
2702```bash
2703go test ./internal/appconfig/ ./internal/host/ ./internal/console/ -count=1
2704```
2705
2706Expected: PASS.
2707
2708- [ ] **Step 5: Gate and commit**
2709
2710```bash
2711gofmt -w internal/appconfig/appconfig.go internal/appconfig/routefingerprint.go internal/appconfig/instances_test.go
2712go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l .
2713git add internal/appconfig
2714git commit -m "appconfig: pin the instance layer's signal and one pass over every shape of it (plan 4, task 11)"
2715```
2716
2717---
2718
2719## Self-review
2720
2721**Spec coverage.** §6.2's third bundle at `apps/<acct>/<app>/instances-config/<host>.json`, same manifest shape, same sealed envelopes, same name and value rules: tasks 1 and 2. §6.2's "`LoadMerged` keeps its signature; `LoadMergedHost` adds the third overlay": task 3. §6.2's "`Sync`/`SyncExec` call it per route after resolving the group base, so a pass costs one extra bucket read per instance that has an overlay and none for instances that do not": tasks 4 and 6, with the cost pinned by a counting store and the one deviation (the sweeps use the two functions `LoadMergedHost` is built from, not `LoadMergedHost` itself) stated in the header and pinned by an equivalence test. §6.2's "writing the layer bumps the app's generation object, as `SetExecDeliveryAt` does": task 2, with task 12 pinning that the route fingerprint does not move. §6.2's four guards: `lostKeys`/`defaultMissing` in task 5, the mass-removal guard in task 7, `AwaitingRestart` and `allShadowed` in task 8. §6.2's "`ExecDelivery` is not read from this layer": tasks 3 and 6. §6.2's "only the platform writes this layer: no CLI verb, no console door": nothing in this plan adds either, and the only callers of `ApplyInstance` in the tree when this plan ends are tests. §6.5's `Schema` and `InstanceConfigETag`: task 9. §6.5's "acks stay change-only, and the capability question belongs to the box record's agent version": nothing here writes an extra ack, and the agent-version refusal is step 5's. §6.5's "the console compares the etag with the one its own `Put` returned": task 10 lands the comparison and the states; the door that calls it is step 5's.
2722
2723**Out of scope and deliberately absent.** No instance senders, no `PerInstance` flag, no per-instance mode transition, no instance email doors, no `svcrole.Mail`, no uid or backing check at enable, no `claimDomain` change, no CLI verb, no console door, no mail package change of any kind. Nothing in this plan writes an overlay outside a test, so nothing in the fleet behaves differently after it rolls except that every converged route now leaves an ack behind on every pass.
2724
2725**Four rulings this plan takes.** (1) The sweeps do not call `LoadMergedHost`; they compose `instanceOverlays` and `applyInstanceOverlay`, because calling it per route would refetch the group bundles per route and discard the caches the group loop exists for. The two paths are pinned against each other in task 4. (2) `AwaitingRestart` keeps its (app, environment) vocabulary and gains only the shadowing half; reporting an instance-layer write arrives with step 5's door, because an exported reporting function with no writer and no caller is a guess about what step 5 needs. (3) An every-pass ack carries the previous ack's delivery facts forward rather than restating this pass's, or the hourly pass would walk `carlos relay status` back from "adopted, restarted" to "awaiting delivery" for every route on the fleet. (4) The mass-removal guard's count stays over group bases: the fault it guards against is registry-wide and takes the overlay listing down with everything else, and a route with an overlay never reaches a removal branch anyway.
2726
2727**The orphan question, and why it is step 5's.** Nothing deletes `instances-config/<host>.json` when an instance is retired. `release.PurgeApp`'s prefix sweep covers a deleted APP, since the key sits under `apps/<acct>/<app>/`, but a deleted INSTANCE leaves its overlay behind, holding sealed secrets, ready to be re-applied to whatever takes its host name next. That is a real hazard and it is not reachable from this plan: nothing here writes an overlay, so nothing here can orphan one. The sweep belongs beside the writer — step 5's instance teardown calls `DeleteInstanceConfig`, and the reconcile pass that retires a record is where a belt-and-braces sweep goes. This paragraph is the note that says so, and step 5's plan owes the task.
2728
2729**Cost, stated plainly.** Per sync pass, per (account, app) with routes on the box: one `List` of `instances-config/`. Per route with an overlay: one `Get`. Per converged route whose delivered file CHANGED: one `Put` of its ack, exactly as today. A pass that changes nothing writes nothing and reads nothing extra beyond the one `List` per app, so a box with two hundred settled routes pays one listing per app per pass and no writes at all.
2730
2731**Placeholder scan.** None. Every task carries its test code and its implementation code in full, including the three fixtures the tests share (`putInstanceRaw`, `countingStore`, `syncInstanceKit`), each defined in the task that first uses it. `execKit` (`internal/appconfig/syncexec_test.go:24`), `testRegistry` (`sync_test.go:13`), `captureStdout` (`syncexec_test.go:46`), `failingKeyStore` (`syncexec_test.go:71`), `addrRoute` (`awaitingrestart_test.go:14`) and `syncRelayTestKeys` (`configack_test.go:17`) already exist and are used as they are.
2732
2733**Type consistency.** `InstancesConfigPrefix`, `InstanceConfigKey`, `ValidInstanceHost`, `LoadInstanceConfig`, `ApplyInstance`, `DeleteInstanceConfig`, `LoadMergedHost`, `applyInstanceOverlay`, `instanceOverlays`, `instanceOverlayLoader`, `getIfExistsAtETag`, `allShadowed`, `ConfigAckSchema`, `ConfigAckPrefix`, `LatestConfigAck`, `InstanceConfigDelivery`, `InstanceConfigDelivered`, `InstanceConfigStale` and `InstanceConfigNotYetDelivered` are spelled identically in every task that produces or consumes them.
2734
2735**Two things worth saying plainly, because a reader will wonder.** The etag this plan puts in the ack is an S3 etag over sealed JSON, so it names a version without revealing anything about the values inside it; it is safe in a reply a member can read. And a flat exec route still shares the agent's uid, so every sibling on that box can read its `<host>.config.env` whatever this layer does — per-instance delivery is a scoping mechanism, not an isolation one, and the isolation is step 5's uid refusal plus #126's per-instance uids.
2736
2737**The roll note this plan owes.** Four lines.
2738
27391. **Every box's store policy needs `s3:ListBucket` on the deployment bucket before this rolls.** A pass now lists `instances-config/` once per app, and a listing that fails leaves every route of that app with the file it already has — correct, but it is a freeze, and it would hit routes that have no overlay at all. In-repo policies grant it unconditionally (`infra/modules/region-host/main.tf:174`) and boxes already list `apps/` during adoption (`internal/provision/reconcile.go:651`), so platform boxes are safe; a hand-edited or self-hosted policy might not be. Check before rolling, not after.
27402. **A fleet-door box cannot write config acks at all.** `control/config-ack/` is outside the door's read and write scopes (`internal/edge/storedoor.go:389`, `:489`), so delivery to an instance on such a box reads as unconfirmed however well it actually went. Granting the door `control/config-ack/<box>/` is its own piece of work; until then, instance senders should not be placed there.
27413. An ack written by a box still on an older agent decodes with `schema: 0`. Step 5's enable refuses on the box record's agent version rather than on that, but the field is what makes an old ack legible rather than confusing.
27424. Acks stay change-only, so this plan adds no new write pressure: a settled route writes nothing, exactly as today.
2743