amadan
activator: the replica gate's refusal is logged once an hour per host and reason, and the rollout lists keyless replicas' sidecars and workers (final review I5, minor 5)
Before any app adopts the helper every replica is keyless, and after the roll its legacy sidecar and any worker row of its own are refused at every retry of the sidecar sweep. The gate's log line now goes through logPerFile, keyed by host and reason, so one line is one refused host; logPerFile takes its own mutex because the gate runs outside adminTokenMu. The bucket's error is logged as a separate argument after the fixed reason (minor 5), never folded into the reason's text. The plan's rollout gains the pre-roll query for replica rows with a legacy sidecar or a worker row, what happens to them at the roll and how they come back, and step 4's grep rule names them and the new deferral line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 files changed,
+83
−15
docs/superpowers/plans/2026-10-10-instance-key-provision.md+12 −2internal/activator/admintoken.go+8 −2internal/activator/replica_instancekey_test.go+44 −0internal/activator/replicakey.go+19 −11
diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md| index 80f1939c6..dfaa8d80b 100644 |
| --- a/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| +++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| @@ -5191,7 +5191,16 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| For each replica: |
| - **Its home has a key and its digest equals the replica's:** add `replica-host=<digest>`, one per line, to the replica box's list, `/etc/carlos/replica-home-keys`, before that box is upgraded. Write the whole list to a temporary file in `/etc/carlos`, set it to 0640 root:carlos, and rename it into place (ruling 30). |
| - **The two digests differ:** the replica will be refused after the roll. Tell the owner, who sets the replica's key to the home's, or accepts the refusal. |
| - - **The replica has no key:** list nothing. It starts as before wherever it has a control socket. An adopting app without its own key refuses there itself (`ErrNoInstanceKey`) until O8. |
| + - **The replica has no key:** list nothing. Its instance starts as before wherever it has a control socket. An adopting app without its own key refuses there itself (`ErrNoInstanceKey`) until O8. **Its legacy sidecar and any worker row of its own do not run** (next paragraph). |
| + |
| + **Pre-roll check: keyless replicas with a legacy sidecar or a worker row** (Task 3b's build; final review I5). A launch with no control socket is gated as a seed is: it runs only with the home's verified key, and a keyless one is refused outright, because the helper would otherwise mint over the home's data. Two such launches exist on a replica today: the legacy co-spawned sidecar (`<data>/<host>-sidecar.env` exists for the replica row) and a worker row flagged replica. On every box with a replica row, list both: |
| + |
| + ```sh |
| + sqlite3 <data>/registry.db "select host from routes where replica=1" | while read -r h; do [ -f "<data>/$h-sidecar.env" ] && echo "legacy sidecar: $h"; done |
| + sqlite3 <data>/registry.db "select host from routes where replica=1 and kind='sidecar'" |
| + ``` |
| + |
| + Each one listed whose replica is keyless stops at the roll: the instance keeps serving, its sidecar's wake logs `SIDECAR for <host> failed to start ... (the sweep retries)` once, and the sweep's retries log `activator: replica <host> not started: it has no key of its home's and no control socket` once an hour per host (the line is rate-limited). They run again when the home's key is listed for the replica (a configured key, this check's first case) or when O8 delivers the home's platform key. Tell the owner before the roll, and record each host for step 4. |
| Only this check writes the list; nothing refreshes it from a replica's configuration (ruling 27). The agent reads it again at every replica launch and keeps no copy, so a change takes effect at the next launch with no restart (ruling 30). The digests go only into that file, never into a log. |
| @@ -5214,7 +5223,8 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| - **Roll every box before any configured key matters** (task 1 review, minor 6). While boxes run mixed versions, a move's readiness check (`movesreport.go:487`) fails for an instance whose own key is empty but which has a configured key: the old source reports the empty own key, and the new target reports the configured one. That is unlikely until keys are minted, but finish the roll before an owner configures a key on an app that moves. |
| 4. **Verify on each box.** |
| - `journalctl -u carlos-edge --since '<roll time>' | grep 'agent: instance keys:'` must show `0 failed`. For each failure, read the per-host line above it, fix the file, and restart the instance (its start mints the key, or fails loudly with the same error). `never started here` hosts mint at their first start. Claim boxes print no backfill line: their instances mint at their next cold admission. |
| - - `grep 'activator: replica .* not started'` must name only the replicas step 1 expected to be refused. |
| + - `grep 'activator: replica .* not started'` must name only the hosts step 1 expected to be refused: replicas whose key differs from their home's, and the keyless replicas' legacy sidecars and worker rows it listed. The line is said once an hour per host and reason, so one line is one refused host, not one refused retry; a host missing from the list is a question for its owner, not a count. |
| + - `grep 'instance key not minted: the instance's configuration is not delivered'` names first starts held back while a configured key was on its way (ruling 5's delivery rule, final review I2). Each clears at the config sync's next pass; one that persists past an hour means the sync cannot render that app's configuration, which the config guard's line for the host explains. |
| 5. **Post the rastrillo ask and its addendum** (above) on #66, and wait for a rastrillo release that makes the call with `source`. |
| 6. **Only then tell app owners to adopt** that rastrillo release: `carlos.InstanceKey(own)`. Before a box has this build, an adopting app with no key of its own refuses to start on it. After the roll, an app on the platform key or a configured key runs on its home, and on every other host where it is delivered a key. A replica runs only with its home's key, verified (ruling 26). None of these apps relocates until O6 and O7 are in (step 8). |
| 7. **App owners, before adopting:** |
diff --git a/internal/activator/admintoken.go b/internal/activator/admintoken.go| index 6b4960122..5b79338e3 100644 |
| --- a/internal/activator/admintoken.go |
| +++ b/internal/activator/admintoken.go |
| @@ -58,6 +58,10 @@ var adminTokenMu sync.Mutex |
| // enough that a condition which comes back is heard again. |
| var adminTokenLogged = map[string]time.Time{} |
| +// adminTokenLogMu guards adminTokenLogged: logPerFile is called under |
| +// adminTokenMu by the mints and outside it by the replica gate. |
| +var adminTokenLogMu sync.Mutex |
| + |
| // adminTokenLogRepeat is how long one (condition, file) complaint stays |
| // deduplicated before it is said again. |
| const adminTokenLogRepeat = time.Hour |
| @@ -65,10 +69,12 @@ const adminTokenLogRepeat = time.Hour |
| // logPerFile emits msg unless this (kind, path) pair has already said it |
| // within adminTokenLogRepeat. Keyed by kind as well as path so a file that |
| // is both un-tightenable AND holding a malformed token still reports both — |
| -// one condition must not swallow the other's only line. Caller holds |
| -// adminTokenMu. |
| +// one condition must not swallow the other's only line. The replica gate |
| +// keys it by host instead of path, to the same end. |
| func logPerFile(kind, path, msg string) { |
| key := kind + "\x00" + path |
| + adminTokenLogMu.Lock() |
| + defer adminTokenLogMu.Unlock() |
| if last, seen := adminTokenLogged[key]; seen && time.Since(last) < adminTokenLogRepeat { |
| return |
| } |
diff --git a/internal/activator/replica_instancekey_test.go b/internal/activator/replica_instancekey_test.go| index 41b4023f3..2bc52c544 100644 |
| --- a/internal/activator/replica_instancekey_test.go |
| +++ b/internal/activator/replica_instancekey_test.go |
| @@ -415,3 +415,47 @@ func TestReplicaGateNamesWhyItRefused(t *testing.T) { |
| }) |
| } |
| } |
| + |
| +// Final review I5 and minor 5: the gate's refusal is logged once an hour |
| +// per host and reason, so a keyless replica's legacy sidecar, refused at |
| +// every retry of the sidecar sweep, does not fill the log; a new reason or |
| +// another host is still heard. A bucket error is logged apart from the |
| +// fixed reason. |
| +func TestReplicaGateLogsARefusalOncePerHostAndReason(t *testing.T) { |
| + buf := captureLog(t) |
| + b := &ExecBackend{DataDir: t.TempDir()} |
| + rt := registry.Route{Host: "ratelimit-au.example.com", Kind: "instance", Replica: true} |
| + for i := 0; i < 3; i++ { |
| + if err := b.replicaKeyGate(rt, map[string]string{}, instancekey.LayerNone, false); !errors.Is(err, ErrReplicaKey) { |
| + t.Fatalf("gate %d = %v", i, err) |
| + } |
| + } |
| + other := rt |
| + other.Host = "ratelimit-eu.example.com" |
| + if err := b.replicaKeyGate(other, map[string]string{}, instancekey.LayerNone, false); !errors.Is(err, ErrReplicaKey) { |
| + t.Fatal(err) |
| + } |
| + b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest("home"), true } |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return "", errors.New("bucket unreachable") } |
| + env := map[string]string{instancekey.Var: "home"} |
| + for i := 0; i < 2; i++ { |
| + if err := b.replicaKeyGate(rt, env, instancekey.LayerConfig, false); !errors.Is(err, ErrReplicaKey) || !strings.Contains(err.Error(), "bucket unreachable") { |
| + t.Fatalf("gate with the bucket down = %v", err) |
| + } |
| + } |
| + lines := strings.Split(strings.TrimSpace(buf.String()), "\n") |
| + var au, eu, bucket int |
| + for _, l := range lines { |
| + switch { |
| + case strings.Contains(l, "replica ratelimit-au.example.com not started: the bucket's configuration could not be read: bucket unreachable"): |
| + bucket++ |
| + case strings.Contains(l, "replica ratelimit-au.example.com not started"): |
| + au++ |
| + case strings.Contains(l, "replica ratelimit-eu.example.com not started"): |
| + eu++ |
| + } |
| + } |
| + if au != 1 || eu != 1 || bucket != 1 { |
| + t.Fatalf("refusal lines: au %d, eu %d, bucket %d; want one each:\n%s", au, eu, bucket, buf.String()) |
| + } |
| +} |
diff --git a/internal/activator/replicakey.go b/internal/activator/replicakey.go| index 186b3163b..74545df80 100644 |
| --- a/internal/activator/replicakey.go |
| +++ b/internal/activator/replicakey.go |
| @@ -41,7 +41,6 @@ import ( |
| "context" |
| "errors" |
| "fmt" |
| - "log" |
| "github.com/carlosframework/platform/internal/instancekey" |
| "github.com/carlosframework/platform/internal/registry" |
| @@ -85,7 +84,12 @@ var replicaGateHook func() |
| // env (the composed env files), layer (where env's key came from, decided |
| // by the same read) and, when haveControl, a bound control socket. It reads |
| // no env file. It is nil for every row that is not a replica. Its errors and |
| -// log lines name the host and the reason, never a key or a digest. |
| +// log lines name the host and the reason, never a key or a digest. The |
| +// log line is said once an hour per host and reason (logPerFile): a |
| +// keyless replica's legacy sidecar is refused at every retry of the |
| +// sidecar sweep, and the line is for the operator, who needs it once. A |
| +// bucket error is logged as its own argument, apart from the fixed |
| +// reason, so a reader can see where the text came from. |
| func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, layer instancekey.Layer, haveControl bool) error { |
| if !rt.Replica { |
| return nil |
| @@ -93,8 +97,12 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, l |
| if replicaGateHook != nil { |
| replicaGateHook() |
| } |
| - refuse := func(why string) error { |
| - log.Printf("activator: replica %s not started: %s", rt.Host, why) |
| + refuse := func(why string, cause error) error { |
| + if cause != nil { |
| + logPerFile("replica-gate\x00"+why, rt.Host, fmt.Sprintf("activator: replica %s not started: %s: %v", rt.Host, why, cause)) |
| + return fmt.Errorf("%w: %s: %s: %w", ErrReplicaKey, rt.Host, why, cause) |
| + } |
| + logPerFile("replica-gate\x00"+why, rt.Host, fmt.Sprintf("activator: replica %s not started: %s", rt.Host, why)) |
| return fmt.Errorf("%w: %s: %s", ErrReplicaKey, rt.Host, why) |
| } |
| key := env[instancekey.Var] |
| @@ -102,7 +110,7 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, l |
| if haveControl { |
| return nil |
| } |
| - return refuse("it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own") |
| + return refuse("it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own", nil) |
| } |
| var home string |
| var ok bool |
| @@ -111,27 +119,27 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, l |
| } |
| switch { |
| case !ok: |
| - return refuse(instancekey.Var + " cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)") |
| + return refuse(instancekey.Var+" cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)", nil) |
| case instancekey.Digest(key) != home: |
| - return refuse(instancekey.Var + " is not its home's key") |
| + return refuse(instancekey.Var+" is not its home's key", nil) |
| } |
| switch layer { |
| case instancekey.LayerOwn: |
| return nil // the replica's own file: it changes only by hand |
| case instancekey.LayerConfig: |
| default: |
| - return refuse("where its " + instancekey.Var + " came from is unknown") |
| + return refuse("where its "+instancekey.Var+" came from is unknown", nil) |
| } |
| configured := b.configuredKeyFor(rt) |
| if configured == nil { |
| - return refuse("its configured " + instancekey.Var + " cannot be checked against the bucket's configuration") |
| + return refuse("its configured "+instancekey.Var+" cannot be checked against the bucket's configuration", nil) |
| } |
| fresh, err := configured(context.Background()) |
| switch { |
| case err != nil: |
| - return refuse("the bucket's configuration could not be read: " + err.Error()) |
| + return refuse("the bucket's configuration could not be read", err) |
| case fresh != instancekey.Digest(key): |
| - return refuse("its configured " + instancekey.Var + " is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)") |
| + return refuse("its configured "+instancekey.Var+" is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)", nil) |
| } |
| return nil |
| } |