carlos / platform Public

Instance key provision plan rev 5.1: one read for the replica gate, the list read at every launch (rulings 29, 30)

The replica gate takes the key's layer from the same read that composed
the launch env (instanceEnvLayer) and reads no env file; a deterministic
interleaving test rewrites the config between the read and the gate. The
verified home-key list moves to /etc/carlos/replica-home-keys, read at
every replica launch, so the coordinated procedure takes effect with no
agent restart and is verified before the key changes. Task 1's blocks now
match the reviewed branch code (c568c13b..e8487c3a); a mixed-version roll
note joins the rollout. IK-1 unchanged. Answers Astra round 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Paul Campbell pushed by claude 1 week ago 7a2e469672a6dceea9b8bd77f9e8d44785210e8c parent e8487c3

2 files changed, +433 −76

docs/superpowers/plans/2026-10-10-instance-key-provision.md +413 −76
docs/superpowers/plans/INSTANCE-KEY-PLAN-REPORT-R3.md +20 −0

diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md

index ed2e34b4b..bb0845aa9 100644

--- a/docs/superpowers/plans/2026-10-10-instance-key-provision.md

+++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md

@@ -1,4 +1,4 @@

-# Instance Key Provision (Region HA L21) Implementation Plan, rev 5

+# Instance Key Provision (Region HA L21) Implementation Plan, rev 5.1

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

@@ -13,7 +13,25 @@

- Region HA spec `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`: H6 (line 131) and §5.6 (line 539): `instance.key`, `report.token`, `<host>.env` are always continuous.

- Framework contract: rastrillo/rastrillo discussion #66 (agreed; landed on rastrillo main as e3ea08d). `carlos.InstanceKey(own string) (string, error)` resolves, in order: (1) `own`, the app's own key; (2) `$CARLOS_INSTANCE_KEY`, verbatim; (3) on CARLOS (`carlos.Running()`, which checks `$CARLOS_CONTROL_SOCKET`) with no key, `ErrNoInstanceKey`; (4) off CARLOS, `ErrNotOnCarlos`, today's behaviour. Read at e3ea08de in `~/amadan.net/rastrillo/rastrillo` (`carlos/instancekey.go`, `carlos/schedule.go:28`).

- Stage B plan `docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md` on `region-ha-1b-next-build` (919662d2): rulings 25 and 55, Task 8b step 0 (its O2' variant with a `KeyEvidence` reader) and Task 8c (O3). This plan's O2' replaces that step.

-- Review: Astra rounds 1 to 5, `docs/superpowers/plans/CODEX-PLAN-INSTANCE-KEY-R1.md` to `-R5.md` (verdict each time: not ready; rounds 3 to 5 found no P1). Their findings are answered in the five tables below, newest first.

+- Review: Astra rounds 1 to 6, `docs/superpowers/plans/CODEX-PLAN-INSTANCE-KEY-R1.md` to `-R6.md` (verdict each time: not ready; rounds 3 to 6 found no P1). Their findings are answered in the six tables below, newest first.

+- Task 1 is built and reviewed on this branch (c568c13b, afa89057, e8487c3a); its code below is the branch's.

+## Rev 5.1: round 6 dispositions

+Astra's round 6 (`CODEX-PLAN-INSTANCE-KEY-R6.md`, reviewed at 2c39819c) closed R5-2 and R5-3, and judged R5-1's approach sound. It found two P2s. Task 1 is now built and reviewed on this branch (c568c13b, afa89057, e8487c3a). Rev 5.1 changes none of Task 1's code; it copies the reviewed code into Task 1 so the plan and the branch match.

+|---|---|---|---|---|

+| R6-1 | P2. A concurrent config removal bypasses the fresh check. The launch captures configured key A, but the gate re-read the files to find the source: a sync that removed the key in between makes `Effective` say `LayerNone`, and the gate admitted the launch without asking the bucket, while the child still got A. | **Accepted** (ruling 29). Verified: rev 5's gate called `instanceEnvParts` again. | One read decides both the env and the key's source: `instanceEnvLayer` returns `mergeEnv`'s env and `Effective`'s layer from the same two maps. Every gate call (Seed, Start, the nested prepare, the claim snapshot) passes that layer. The gate reads no env file, and refuses a key whose layer is unknown. | `TestReplicaGateDecidesOnTheLaunchesOwnRead`: a deterministic interleaving through `Start`, where a hook rewrites `<host>.config.env` without the key between the launch's read and the gate. It is refused; with rev 5's second read restored, the app ran with A. Also `TestReplicaGateReadsNoEnvFile` |

+| R6-2 | P2. The procedure never applied list changes to the running agent: `replicaHomeKeys` parsed the environment once at startup. | **Accepted** (ruling 30): **read on each launch.** | The list moves from the agent's environment to a file, `/etc/carlos/replica-home-keys`, written whole by the pre-roll check (temporary file, then rename). The agent reads it again at every replica launch and keeps no copy. An invalidation takes effect at the next launch, with no agent restart. The procedure checks the file before the key changes, with the replicas kept stopped throughout. | `TestReplicaHomeKeysReadsTheListAtEveryLaunch` (a removed entry stops verifying at the next call; a replaced one verifies the new digest); `TestReplicaHomeKeysLogsAMalformedListWithoutItsText` (also an unreadable list) |

+**Counts:** 2 accepted, 0 declined.

+**Also in rev 5.1:**

+- Task 1's code is the reviewed branch's, from c568c13b to e8487c3a. `ParseAdoption` accepts only the exact bytes `AdoptionRecord` writes; `ParseEnv`'s behaviour is pinned by a test; and the two activator doc comments name the empty-key exception. Task 1 also gains `TestEmptyOwnInstanceKeyShadowsNothing`.

+- A rollout note from Task 1's review: a mixed-version roll and the moves readiness check (rollout step 3).

+**Effect on stage B (IK-1):** none in this revision. The reviewed `ParseAdoption` is stricter: it accepts only the canonical bytes, which is all `AdoptionRecord` ever writes. O2' and its test were re-run against stage B's current tip (b9a753f0) with the reviewed `instancekey`, and pass.

## Rev 5: round 5 dispositions

@@ -277,7 +295,7 @@ Source (c) and `ConfiguredInstanceKey` are gone: a configured key is covered by

- **The gate, `replicaKeyGate`,** runs on every replica launch before anything starts: Start, Seed, the nested prepare's run and seed roles, and the claim launch source.

- If the replica's env files deliver a key, the replica starts only when that key's digest is its home's (`ExecBackend.ReplicaHomeKeys`). A key that differs is refused, with the log line `activator: replica <host> not started: CARLOS_INSTANCE_KEY is not its home's key`. A key that cannot be checked is refused too.

- If the env files deliver no key, the replica starts only with a bound control socket. There an app on rastrillo's `InstanceKey` with no key of its own gets `ErrNoInstanceKey` and refuses (step 3), and an app that never asks for a key runs as before. Without a socket (every seed, every claim launch before O6, a flat launch whose bind failed) the helper would fall to step 4 and mint over the home's data. So the agent refuses the launch itself: it does not leave the refusal to the helper.

-- **Where the home's key comes from.** The home's key reaches a replica through O8: a replica record that names its home, and a read of the home's published env item, verified by digest against the home's adoption record or env item. Until O8, nothing can hand a keyless replica its home's key, so it gets none and is gated as above. Verification uses the digests the pre-roll check records in the box's `CARLOS_REPLICA_HOME_KEYS` (`replica-host=<sha256>`, comma-separated, read by `replicaHomeKeys`). A value that does not parse verifies nothing.

+- **Where the home's key comes from.** The home's key reaches a replica through O8: a replica record that names its home, and a read of the home's published env item, verified by digest against the home's adoption record or env item. Until O8, nothing can hand a keyless replica its home's key, so it gets none and is gated as above. Verification uses the digests the pre-roll check records in the box's list, `/etc/carlos/replica-home-keys` (`replica-host=<sha256>` entries, read by `replicaHomeKeys` at every launch; ruling 30). A list that does not parse verifies nothing.

- **No regression on upgrade.** A replica that shares a configured key that is its home's effective key keeps it: the pre-roll check lists it, and the gate passes it. A replica whose configured key differs is refused, and the pre-roll check shows it before any box is upgraded (rollout step 1).

- **The adoption verb** still answers 409 on a replica: a replica's record would bind a writer whose data is the home's.

@@ -289,7 +307,7 @@ Source (c) and `ConfiguredInstanceKey` are gone: a configured key is covered by

- The list's freshness, and any change to a configured key, are ruling 27 (rev 5). O8's scope is ruling 28.

**27. A configured key is never changed in place (rev 5; Paul's ruling; Astra round 5 finding 1).** Changing a configured `CARLOS_INSTANCE_KEY` on an app that has data is a key rotation, and this plan does not support rotation. `SyncExec` reconciles each box on its own schedule, hourly, so during any configured-key change the home and a replica can briefly hold different keys.

-- **The procedure** (rollout step 1 and the operator note): stop the app's replicas; remove their entries from `CARLOS_REPLICA_HOME_KEYS`; change the key; let every box sync; re-run the pre-roll check, which writes the new entries; then start the replicas.

+- **The procedure** (rollout step 1 and the operator note): stop the app's replicas; remove their entries from the box's list and check that the file no longer names them (the agent keeps no copy, ruling 30); change the key; let every box sync; re-run the pre-roll check, which writes the new entries; then start the replicas.

- **The safety net.** On every replica launch whose key comes from its configuration, the agent checks two things. The key's digest must equal the replica's list entry, which only the pre-roll check writes and nothing refreshes from the replica's configuration. And the bucket's configuration must render that same key now (`ReplicaConfigKey`, a fresh read on every such launch). Any mismatch, or a read that fails, refuses the launch with a log line naming the host and the reason.

- **The fresh read is the cheap fresh evidence that works without O8.** A shared configuration change reaches the bucket before any box syncs it, so a replica whose box lags the home is refused.

@@ -299,6 +317,15 @@ Source (c) and `ConfiguredInstanceKey` are gone: a configured key is covered by

**28. O8 delivers platform keys only (rev 5; Paul's ruling; Astra round 5 finding 2).** O8 hands a keyless replica only a key the platform minted into the home's `<host>.env`, read from the home's published env item and verified against it. A home whose key exists only in its configuration is explicitly deferred: its value is not in any published item, and the adoption record holds only its digest. Its keyless replica keeps refusing (ruling 26), and a replica configured with the same key follows ruling 27. *Cost if wrong:* an app that keys its home by configuration must also configure its replicas with that key (the shared app-level key, listed by the pre-roll check) for them to run.

+**29. The replica gate decides on the launch's own read (rev 5.1; Astra round 6 finding 1).** Configuration sync shares no lock with a launch, so the env files can change between the read that composes a child's env and any later read. The source of the key the child is handed is therefore taken from the same read: `instanceEnvLayer` returns `mergeEnv(cfg, own)` and `Effective(cfg, own)`'s layer together. Every gate call passes that layer, and the gate never calls `Effective` on a second read. A non-empty key with no known layer refuses. *Cost if wrong:* none at run time, since it is one read where there were two. If a future caller composed the env some other way and passed a stale layer, the gate would still compare the delivered key's digest with the list and the bucket, and an unknown layer still refuses.

+**30. The verified list is read at every replica launch (rev 5.1; Astra round 6 finding 2).**

+- **Where it lives.** The list (still called `CARLOS_REPLICA_HOME_KEYS` in errors and docs) moves from the agent's environment, which a running agent never re-reads, to a file: `/etc/carlos/replica-home-keys`. It holds `replica-host=<sha256>` entries separated by newlines or commas, root-owned and readable by the agent (0640 root:carlos).

+- **How it is written.** The pre-roll check writes it whole, to a temporary file renamed into place, so a reader sees the old list or the new one, never part of one. The agent writes nothing to it.

+- **How it is read.** `replicaHomeKeys` reads the file at every call and keeps no copy. A removed entry stops verifying at the very next replica launch, and a replacement verifies from the next launch on, with no agent restart (which would restart every instance on the box). An absent file verifies nothing. An unreadable or malformed one verifies nothing and is logged without its text.

+*Cost if wrong:* one read of a small file per replica launch. A list half-written by hand, without the rename, can be read mid-write; it then fails to parse and refuses every replica on the box until it is whole, which is safe.

## Obligations on other plans

Rev 3 rewrites O1', O2', O3', O4, O5 and O6, and adds O7. Rev 4 adds O8.

@@ -706,6 +733,7 @@ Two changes to the ask above, and only the second is new:

6. **An empty `CARLOS_INSTANCE_KEY=` in `<host>.env` with a configured key.** The child gets the configured key and nothing is minted (Task 3).

7. **A replica without a control socket.** A keyless replica is refused by the agent, before its app could mint; a replica whose key is not its home's is refused; its home's key runs (Task 3b, through the helper's real resolution).

8. **A configured key in flight (ruling 27).** The bucket says B while the replica's box still holds A, and the replica is refused. The documented residual, a change only on the home, runs with A (Task 3b).

+9. **A sync between the launch's read and the gate (ruling 29),** and **a list change while the agent runs (ruling 30).** The gate decides on the launch's own read, and the list is read at every launch (Task 3b).

## File structure

@@ -713,7 +741,7 @@ Two changes to the ask above, and only the second is new:

- Modify `internal/activator/admintoken.go` (`parseEnvBytes` calls `instancekey.ParseEnv`) and `internal/activator/backend_exec.go` (`mergeEnv`, `LaunchEnvironment`'s shadow list).

- Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `ensureInstanceKeyFor`, `BackfillInstanceKey`, `settleClaimInstanceEnv`, `InstanceKeyVar`, `InstanceKeyOutcome`).

- Create `internal/activator/instancekey_test.go`, `instancekey_linux_test.go`, `replica_instancekey_test.go`, `replica_instancekey_linux_test.go`, `claim_instancekey_test.go` and `claimretry_instancekey_linux_test.go`.

-- Create `internal/activator/replicakey.go` (`replicaKeyGate`, `ErrReplicaKey`), `internal/host/replicahomekeys.go` (`replicaHomeKeys`, `replicaConfigKey`) and `internal/appconfig/instancekey.go` (`ConfiguredKeyDigest`), with `internal/host/replicahomekeys_test.go` and `internal/appconfig/instancekey_test.go`. Modify `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` and `internal/host/agent.go` (Task 3b).

+- Create `internal/activator/replicakey.go` (`replicaKeyGate`, `ErrReplicaKey`), `internal/host/replicahomekeys.go` (`replicaHomeKeys`, `replicaConfigKey`) and `internal/appconfig/instancekey.go` (`ConfiguredKeyDigest`), with `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` and `internal/activator/replicakey_race_test.go`. Modify `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` and `internal/host/agent.go` (Task 3b).

- Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369), `internal/activator/prepare.go` (:60) and `internal/activator/claim_snapshot.go` (the snapshot's fields, `EnsureClaimAdminToken` :40-67, `SnapshotClaimLaunch`'s env read).

- Create `internal/activator/keyadoption.go`, `keyadoption_test.go` and `keyadoption_linux_test.go`: the adoption verb, mounted by O6.

- Create `internal/hostfiles/relay.go` and `relay_test.go`: `PlatformRelay`.

@@ -810,13 +838,42 @@ func TestAdoptionRecordBindsOneWriter(t *testing.T) {

"an unknown field": `{"v":2,"var":"CARLOS_INSTANCE_KEY","host":"h","incarnation":"i","session":1,"generation":"g","source":"config","sha256":"` + Digest(key) + `","key":"x"}`,

"trailing data": string(rec) + `{}`,

"garbage": "not json",

+ // The canonical bytes are the only record (review minor 1):

+ // encoding/json alone accepts each of these.

+ "a trailing }": string(rec) + `}`,

+ "a trailing ]": string(rec) + `]`,

+ "an upper-case key": strings.Replace(string(rec), `"sha256"`, `"SHA256"`, 1),

+ "a key given twice": strings.Replace(string(rec), `"source":"platform"`, `"source":"config","source":"platform"`, 1),

+ "reordered fields": strings.Replace(string(rec), `{"v":2,"var":"CARLOS_INSTANCE_KEY",`, `{"var":"CARLOS_INSTANCE_KEY","v":2,`, 1),

+ "an upper-case digest": strings.Replace(string(rec), Digest(key), strings.ToUpper(Digest(key)), 1),

+ "a short digest": strings.Replace(string(rec), Digest(key), Digest(key)[:63], 1),

+ "another variable": strings.Replace(string(rec), `"var":"CARLOS_INSTANCE_KEY"`, `"var":"OTHER"`, 1),

} {

+ if raw == string(rec) {

+ t.Fatalf("%s: the case did not change the record", name)

+ }

if _, err := ParseAdoption([]byte(raw)); err == nil {

t.Errorf("%s: parsed", name)

}

+// ParseEnv is exactly the activator's old parseEnvBytes, so what a reader

+// parses is what a child received: quotes kept, whole-line comments only,

+// the last duplicate winning, CRLF and surrounding whitespace trimmed.

+func TestParseEnvMatchesWhatAChildReceives(t *testing.T) {

+ got := ParseEnv([]byte("A=\"quoted\"\nB='single'\n# C=1\n # D=1\nE=1 # kept\nF=1\nF=2\n G = spaced \nH=a=b\nI=1\r\nnoequals\n\n"))

+ want := map[string]string{"A": `"quoted"`, "B": "'single'", "E": "1 # kept", "F": "2", "G": "spaced", "H": "a=b", "I": "1"}

+ if len(got) != len(want) {

+ t.Fatalf("ParseEnv = %q, want %q", got, want)

+ }

+ for k, v := range want {

+ if got[k] != v {

+ t.Fatalf("ParseEnv = %q, want %q", got, want)

+ }

// O3': the install refuses env files that would deliver any key but the

// pinned one, including a configuration changed since the pin.

func TestVerifyDeliveredRefusesAnyOtherKey(t *testing.T) {

@@ -1027,8 +1084,11 @@ func AdoptionRecord(b Binding, source, sha string) []byte {

}

// ParseAdoption reads a record strictly: version 2, this variable, a

-// complete binding, a known source and a digest. Anything else is an

-// error, never a record.

+// complete binding, a known source and a digest, in exactly the bytes

+// AdoptionRecord writes for them. Anything else is an error, never a

+// record: trailing data, a key in another case, a key given twice, or any

+// other spelling of the same fields (encoding/json alone accepts the

+// last three).

func ParseAdoption(record []byte) (Adoption, error) {

var a Adoption

dec := json.NewDecoder(bytes.NewReader(record))

@@ -1039,6 +1099,9 @@ func ParseAdoption(record []byte) (Adoption, error) {

if dec.More() || a.V != 2 || a.Var != Var || !a.Binding.valid() || (a.Source != SourcePlatform && a.Source != SourceConfig) || !ValidDigest(a.SHA256) {

return Adoption{}, errors.New("instancekey: adoption record: not a version 2 record for " + Var)

}

+ if !bytes.Equal(record, AdoptionRecord(a.Binding, a.Source, a.SHA256)) {

+ return Adoption{}, errors.New("instancekey: adoption record: not in the canonical form AdoptionRecord writes")

+ }

return a, nil

}

@@ -1154,16 +1217,79 @@ In `LaunchEnvironment`, an empty own key is not a shadow:

}

```

+Name the exception in both doc comments (task 1 review):

+```go

+// mergeEnv composes the child's env-file layers: base (the sync-owned

+// <host>.config.env, bucket-derived) under overlay (the instance's own

+// <host>.env). The overlay wins every conflict but one — CARLOS_ADMIN_TOKEN

+// lives only in the instance file and must never be clobberable from

+// the bucket, and a box-local override deliberately beats central

+// config. The exception: an empty CARLOS_INSTANCE_KEY in the overlay

+// supplies no key, so a non-empty one in base stands (instancekey.Effective).

+// Returns a fresh map; inputs are never mutated.

+func mergeEnv(base, overlay map[string]string) map[string]string {

+```

+```go

+// shadowed is the sorted names present in both the rendered configuration

+// file and the instance's own <host>.env: the override mergeEnv applies

+// (moves spec §7.3 T3 reports it), collected here because this is the only

+// layer that sees both files. The one exception is mergeEnv's: an empty

+// CARLOS_INSTANCE_KEY in the own file overrides nothing, so it is not

+// listed. The move's launch report (§7.3, codex R2-8)

+// is computed over exactly this env, so a target compares what would run,

+// not a reconstruction of it. The values are raw: the caller HMACs them

+// and must never log or return them.

+func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, []string, error) {

+```

+Add to `internal/activator/backend_exec_test.go`, after `TestLaunchEnvironmentComposesTheSpawnEnvironment`:

+```go

+// An empty CARLOS_INSTANCE_KEY in the instance's own file supplies no key

+// (instance-key plan ruling 13): the merge keeps the configured key, and

+// the launch report does not list the empty one as shadowing it. A

+// non-empty own key still wins and is still reported.

+func TestEmptyOwnInstanceKeyShadowsNothing(t *testing.T) {

+ cfg := map[string]string{"CARLOS_INSTANCE_KEY": "cfg"}

+ if got := mergeEnv(cfg, map[string]string{"CARLOS_INSTANCE_KEY": ""}); got["CARLOS_INSTANCE_KEY"] != "cfg" {

+ t.Fatalf("mergeEnv with an empty own key: CARLOS_INSTANCE_KEY = %q, want the configured key", got["CARLOS_INSTANCE_KEY"])

+ }

+ b := &ExecBackend{DataDir: t.TempDir(), Region: "ie"}

+ rt := registry.Route{Host: "k.go.example.com", Socket: filepath.Join(b.DataDir, "k.sock")}

+ write := func(name, body string) {

+ t.Helper()

+ if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+name), []byte(body), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ write(".config.env", "CARLOS_INSTANCE_KEY=cfg\nMINE=0\n")

+ write(".env", "CARLOS_INSTANCE_KEY=\nMINE=1\n")

+ env, shadowed, err := b.LaunchEnvironment(rt)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if env["CARLOS_INSTANCE_KEY"] != "cfg" || len(shadowed) != 1 || shadowed[0] != "MINE" {

+ t.Fatalf("empty own key: CARLOS_INSTANCE_KEY is the configured key: %v; shadowed %v, want [MINE]", env["CARLOS_INSTANCE_KEY"] == "cfg", shadowed)

+ }

+ write(".env", "CARLOS_INSTANCE_KEY=own\nMINE=1\n")

+ if env, shadowed, err = b.LaunchEnvironment(rt); err != nil || env["CARLOS_INSTANCE_KEY"] != "own" || len(shadowed) != 2 {

+ t.Fatalf("non-empty own key: CARLOS_INSTANCE_KEY is the own key: %v; shadowed %v; err %v", env["CARLOS_INSTANCE_KEY"] == "own", shadowed, err)

+ }

+```

- [ ] **Step 5: Run the tests**

-Run: `nice -n 19 go test ./internal/instancekey/ -count=1 && nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartMergesConfigEnvUnderOwnEnv|LaunchEnvironment|TestEnsureAdminToken' -count=1` (sandbox off for the second: it starts children)

+Run: `nice -n 19 go test ./internal/instancekey/ -count=1 && nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartMergesConfigEnvUnderOwnEnv|LaunchEnvironment|TestEmptyOwnInstanceKeyShadowsNothing|TestEnsureAdminToken' -count=1` (sandbox off for the second: it starts children)

Expected: PASS.

- [ ] **Step 6: Commit**

```bash

gofmt -l internal/instancekey internal/activator

-git add internal/instancekey internal/activator/admintoken.go internal/activator/backend_exec.go

+git add internal/instancekey internal/activator/admintoken.go internal/activator/backend_exec.go internal/activator/backend_exec_test.go

git commit -m "instancekey: one rule for the instance key's parse, effective layer and adoption record (L21)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

@@ -2728,6 +2854,105 @@ func TestConfiguredKeyDigestReadsTheBucketNow(t *testing.T) {

}

```

+Create `internal/activator/replicakey_race_test.go`, ruling 29's deterministic interleaving:

+```go

+package activator

+import (

+ "context"

+ "errors"

+ "os"

+ "path/filepath"

+ "strings"

+ "testing"

+ "time"

+ "github.com/carlosframework/platform/internal/instancekey"

+ "github.com/carlosframework/platform/internal/registry"

+// Ruling 29 (Astra round 6 finding 1), deterministic: the launch reads a

+// configured key A, and then a config sync removes the key from this box's

+// <host>.config.env (the bucket no longer holds it) before the replica

+// gate runs. The gate decides on the layer the launch's own read found, so

+// it still consults the bucket, which now renders no key, and refuses. A

+// second read of the files would have found no key, skipped the bucket

+// and admitted a child that is handed A.

+func TestReplicaGateDecidesOnTheLaunchesOwnRead(t *testing.T) {

+ tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"

+ keyA := strings.Repeat("A", 43) + "="

+ b := testExecBackend(t)

+ rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true, Socket: noControlSocket(t)}

+ cfgPath := filepath.Join(b.DataDir, rt.Host+".config.env")

+ if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte(tok), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(cfgPath, []byte(InstanceKeyVar+"="+keyA+"\n"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true }

+ b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed

+ syncs := 0

+ replicaGateHook = func() {

+ syncs++

+ if err := os.WriteFile(cfgPath, []byte("OTHER=1\n"), 0o600); err != nil {

+ t.Error(err)

+ }

+ t.Cleanup(func() { replicaGateHook = nil })

+ out := stageHelperApp(t, b, rt.Host, "v1")

+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)

+ defer cancel()

+ proc, _, err := b.Start(ctx, rt)

+ if proc != nil {

+ defer proc.Stop(context.Background())

+ }

+ if syncs != 1 {

+ t.Fatalf("the sync ran %d times between the read and the gate, want once", syncs)

+ }

+ if got := outcomeOf(out); !errors.Is(err, ErrReplicaKey) || got != "" {

+ t.Fatalf("Start = %v, app outcome %q; want ErrReplicaKey and no app", err, got)

+ }

+// The gate reads no env file: given the env and layer the launch's read

+// captured, it decides the same with the files gone.

+func TestReplicaGateReadsNoEnvFile(t *testing.T) {

+ keyA := strings.Repeat("A", 43) + "="

+ b := &ExecBackend{DataDir: t.TempDir()}

+ rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true}

+ if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(InstanceKeyVar+"="+keyA+"\n"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ env, layer, err := b.instanceEnvLayer(rt)

+ if err != nil || layer != instancekey.LayerConfig || env[InstanceKeyVar] != keyA {

+ t.Fatalf("instanceEnvLayer = %v, %v", layer, err)

+ }

+ if err := os.RemoveAll(b.DataDir); err != nil {

+ t.Fatal(err)

+ }

+ b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true }

+ for _, tc := range []struct {

+ bucket string

+ ok bool

+ }{{keyA, true}, {"", false}} {

+ b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) {

+ if tc.bucket == "" {

+ return "", nil

+ }

+ return instancekey.Digest(tc.bucket), nil

+ }

+ if err := b.replicaKeyGate(rt, env, layer, false); (err == nil) != tc.ok {

+ t.Fatalf("bucket %q: gate = %v, want ok %v", tc.bucket, err, tc.ok)

+ }

+ if err := b.replicaKeyGate(rt, env, instancekey.LayerNone, false); !errors.Is(err, ErrReplicaKey) {

+ t.Fatalf("a key with no known layer = %v, want a refusal", err)

+ }

+```

Create `internal/host/replicahomekeys_test.go`:

```go

@@ -2739,6 +2964,7 @@ import (

"errors"

"log"

"os"

+ "path/filepath"

"strings"

"testing"

@@ -2748,14 +2974,42 @@ import (

"github.com/carlosframework/platform/internal/release"

)

-func TestReplicaHomeKeysFromTheEnvironment(t *testing.T) {

- sha := instancekey.Digest("home-key")

- env := map[string]string{"CARLOS_REPLICA_HOME_KEYS": "shop-au.example.com=" + sha}

- keys := replicaHomeKeys(func(k string) string { return env[k] })

- if got, ok := keys("shop-au.example.com"); !ok || got != sha {

+// Ruling 30 (Astra round 6 finding 2): the list is read again at every

+// call, so the procedure's invalidation and its new entries take effect at

+// the next replica launch, with no restart. An absent list verifies

+// nothing.

+func TestReplicaHomeKeysReadsTheListAtEveryLaunch(t *testing.T) {

+ a, b := instancekey.Digest("home-key-a"), instancekey.Digest("home-key-b")

+ path := filepath.Join(t.TempDir(), "replica-home-keys")

+ keys := replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(path) })

+ if _, ok := keys("shop-au.example.com"); ok {

+ t.Fatal("an absent list verified a key")

+ }

+ write := func(body string) {

+ t.Helper()

+ tmp := path + ".tmp"

+ if err := os.WriteFile(tmp, []byte(body), 0o640); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.Rename(tmp, path); err != nil {

+ t.Fatal(err)

+ }

+ write("shop-au.example.com=" + a + "\nshop-uk.example.com=" + a + "\n")

+ if got, ok := keys("shop-au.example.com"); !ok || got != a {

t.Fatalf("listed replica = %q, %v", got, ok)

}

- if _, ok := keys("shop-uk.example.com"); ok {

+ // The procedure's invalidation: the entry is gone at the next call.

+ write("shop-uk.example.com=" + a + "\n")

+ if _, ok := keys("shop-au.example.com"); ok {

+ t.Fatal("a removed entry still verified: the list was cached")

+ }

+ // And its replacement, after the key change and the re-run check.

+ write("shop-au.example.com=" + b + ",shop-uk.example.com=" + a)

+ if got, ok := keys("shop-au.example.com"); !ok || got != b {

+ t.Fatalf("the replaced entry = %q, %v; want the new digest", got, ok)

+ }

+ if _, ok := keys("shop-nz.example.com"); ok {

t.Fatal("an unlisted replica verified")

}

@@ -2768,7 +3022,7 @@ func TestReplicaHomeKeysLogsAMalformedListWithoutItsText(t *testing.T) {

var buf bytes.Buffer

log.SetOutput(&buf)

defer log.SetOutput(os.Stderr)

- keys := replicaHomeKeys(func(string) string { return "shop-au.example.com=" + sha + "," + sha })

+ keys := replicaHomeKeys(func() ([]byte, error) { return []byte("shop-au.example.com=" + sha + "\n" + sha + "\n"), nil })

if _, ok := keys("shop-au.example.com"); ok {

t.Fatal("a malformed list verified a key")

}

@@ -2776,6 +3030,13 @@ func TestReplicaHomeKeysLogsAMalformedListWithoutItsText(t *testing.T) {

if !strings.Contains(line, "CARLOS_REPLICA_HOME_KEYS entry 2") || strings.Contains(line, sha[:16]) || strings.Contains(line, "shop-au") {

t.Fatalf("log line %q: want the entry's position and reason, and none of its text", line)

}

+ buf.Reset()

+ if _, ok := replicaHomeKeys(func() ([]byte, error) { return nil, errors.New("permission denied") })("shop-au.example.com"); ok {

+ t.Fatal("an unreadable list verified a key")

+ }

+ if !strings.Contains(buf.String(), "could not be read") {

+ t.Fatalf("log line %q", buf.String())

+ }

}

// Ruling 27's fresh read, as the agent wires it: the bucket's current

@@ -2820,8 +3081,8 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) {

- [ ] **Step 2: Run them to verify they fail**

-Run: `nice -n 19 go test ./internal/activator/ -run 'TestReplicaLaunchWithoutAControlSocket|TestReplicaSeedIsGated|TestKeylessReplica|TestNestedReplicaPrepareIsGated' -count=1 -v; nice -n 19 go test ./internal/host/ -run 'TestReplicaHomeKeys|TestReplicaConfigKey' -count=1; nice -n 19 go test ./internal/appconfig/ -run TestConfiguredKeyDigest -count=1`

-Expected: build failures, `undefined: ErrReplicaKey` (and `ReplicaHomeKeys`, `ReplicaConfigKey`, `replicaHomeKeys`, `replicaConfigKey`, `ConfiguredKeyDigest`). In the scratch build, with the gate present but disabled, the keyless and differing-key cases failed: the app reported "minted", or ran with the wrong key. That is Astra's round 4 failure. With only the fresh read disabled, "the home updated, this replica's configuration delayed" failed: the app ran with the old key. That is Astra's round 5 failure.

+Run: `nice -n 19 go test ./internal/activator/ -run 'TestReplicaLaunchWithoutAControlSocket|TestReplicaSeedIsGated|TestReplicaGate|TestKeylessReplica|TestNestedReplicaPrepareIsGated' -count=1 -v; nice -n 19 go test ./internal/host/ -run 'TestReplicaHomeKeys|TestReplicaConfigKey' -count=1; nice -n 19 go test ./internal/appconfig/ -run TestConfiguredKeyDigest -count=1`

+Expected: build failures, `undefined: ErrReplicaKey` (and `ReplicaHomeKeys`, `ReplicaConfigKey`, `replicaHomeKeys`, `replicaConfigKey`, `ConfiguredKeyDigest`). In the scratch build, with the gate present but disabled, the keyless and differing-key cases failed: the app reported "minted", or ran with the wrong key. That is Astra's round 4 failure. With only the fresh read disabled, "the home updated, this replica's configuration delayed" failed: the app ran with the old key. That is Astra's round 5 failure. With rev 5's second read of the files restored in the gate, `TestReplicaGateDecidesOnTheLaunchesOwnRead` failed: the app ran with A. That is Astra's round 6 failure.

- [ ] **Step 3: Write the gate**

@@ -2838,9 +3099,11 @@ package activator

// - a replica whose env files deliver a key starts only when that key's

// digest is its entry in the box's verified list (ReplicaHomeKeys, which

-// only the pre-roll check writes) and, when the key comes from the

-// replica's configuration, the configuration in the bucket renders that

-// same key now (ReplicaConfigKey). A key that differs from either, or

+// only the pre-roll check writes, read again at every launch) and, when

+// the key comes from the replica's configuration, the configuration in

+// the bucket renders that same key now (ReplicaConfigKey). Where the key

+// came from is decided by the same read that composed the env the child

+// gets (instanceEnvLayer, ruling 29), never by reading the files again. A key that differs from either, or

// that cannot be checked, refuses the launch with a log line. The fresh

// read catches a configuration change that this box has not synced yet

// while the home already holds the new key (ruling 27);

@@ -2874,14 +3137,36 @@ var ErrReplicaKey = errors.New("activator: replica launch refused: its instance

// replicaConfigTimeout bounds replicaKeyGate's fresh read of the bucket.

const replicaConfigTimeout = 10 * time.Second

-// replicaKeyGate is rulings 26 and 27's check for a launch of rt with env

-// (the composed env files) and, when haveControl, a bound control socket.

-// It is nil for every row that is not a replica. Its errors and log lines

-// name the host and the reason, never a key or a digest.

-func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, haveControl bool) error {

+// instanceEnvLayer is instanceEnv returning, from the same read, the layer

+// its CARLOS_INSTANCE_KEY came from (ruling 29). mergeEnv delivers exactly

+// the key instancekey.Effective names, so env[Var] and the layer always

+// describe the same bytes, however the files change afterwards.

+func (b *ExecBackend) instanceEnvLayer(rt registry.Route) (map[string]string, instancekey.Layer, error) {

+ cfgEnv, ownEnv, err := b.instanceEnvParts(rt)

+ if err != nil {

+ return nil, instancekey.LayerNone, err

+ }

+ _, layer := instancekey.Effective(cfgEnv, ownEnv)

+ return mergeEnv(cfgEnv, ownEnv), layer, nil

+// replicaGateHook runs at the start of every replica gate, after the env

+// was read. A test seam: it lets a test change the env files between the

+// read and the gate, as a concurrent config sync can. Nil in production.

+var replicaGateHook func()

+// replicaKeyGate is rulings 26, 27 and 29's check for a launch of rt with

+// env (the composed env files), layer (where env's key came from, decided

+// by the same read) and, when haveControl, a bound control socket. It reads

+// no env file. It is nil for every row that is not a replica. Its errors and

+// log lines name the host and the reason, never a key or a digest.

+func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, layer instancekey.Layer, haveControl bool) error {

if !rt.Replica {

return nil

}

+ if replicaGateHook != nil {

+ replicaGateHook()

+ }

refuse := func(why string) error {

log.Printf("activator: replica %s not started: %s", rt.Host, why)

return fmt.Errorf("%w: %s: %s", ErrReplicaKey, rt.Host, why)

@@ -2904,12 +3189,12 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, h

case instancekey.Digest(key) != home:

return refuse(instancekey.Var + " is not its home's key")

}

- cfgEnv, ownEnv, err := b.instanceEnvParts(rt)

- if err != nil {

- return refuse("its env files could not be read again: " + err.Error())

- }

- if _, layer := instancekey.Effective(cfgEnv, ownEnv); layer != instancekey.LayerConfig {

+ switch layer {

+ case instancekey.LayerOwn:

return nil // the replica's own file: it changes only by hand

+ case instancekey.LayerConfig:

+ default:

+ return refuse("where its " + instancekey.Var + " came from is unknown")

}

if b.ReplicaConfigKey == nil {

return refuse("its configured " + instancekey.Var + " cannot be checked against the bucket's configuration")

@@ -2931,12 +3216,12 @@ In `internal/activator/backend_exec.go`, add the two fields to `ExecBackend`, af

```go

// ReplicaHomeKeys answers, for a replica row's host, the SHA-256 of its

- // home's effective CARLOS_INSTANCE_KEY (instance-key plan ruling 26).

- // O8 will supply it from the home's published env item; until then

- // the agent reads CARLOS_REPLICA_HOME_KEYS, which the pre-roll check

- // fills. Nil, or false for a host, means the home key cannot be

- // verified, and a replica launch that would be handed a key is refused

- // (replicaKeyGate).

+ // home's effective CARLOS_INSTANCE_KEY (instance-key plan rulings 26 and

+ // 30). O8 will supply it from the home's published env item; until then

+ // the agent reads the box's list (/etc/carlos/replica-home-keys), which

+ // only the pre-roll check writes, again at every call. Nil, or false for

+ // a host, means the home key cannot be verified, and a replica launch

+ // that would be handed a key is refused (replicaKeyGate).

ReplicaHomeKeys func(host string) (sha256 string, ok bool)

// ReplicaConfigKey reads rt's configuration from the bucket now and

// returns the SHA-256 of the CARLOS_INSTANCE_KEY it renders, "" for

@@ -2946,27 +3231,38 @@ In `internal/activator/backend_exec.go`, add the two fields to `ExecBackend`, af

ReplicaConfigKey func(ctx context.Context, rt registry.Route) (sha256 string, err error)

```

-In `seedChild`, after the env is read:

+In `seedChild`, read the env and its key's layer together (`instanceEnvLayer`, in `replicakey.go`), then gate:

```go

- fileEnv, err := b.instanceEnv(rt)

+ fileEnv, keyLayer, err := b.instanceEnvLayer(rt)

if err != nil {

return err

}

// A seed has no control socket, so a replica's seed is gated as a

- // launch without one (ruling 26).

- if err := b.replicaKeyGate(rt, fileEnv, false); err != nil {

+ // launch without one (ruling 26), on the layer this same read found.

+ if err := b.replicaKeyGate(rt, fileEnv, keyLayer, false); err != nil {

return err

}

dbPath := paths.dbPath

```

+In `spawnChildFrom`, replace the `instanceEnv` read after the key mint with one read of the env and its key's layer:

+```go

+ // One read gives the env and the layer its CARLOS_INSTANCE_KEY came

+ // from, for the replica gate below (ruling 29).

+ fileEnv, keyLayer, err := b.instanceEnvLayer(rt)

+ if err != nil {

+ return nil, "", err

+ }

+```

In `spawnChildFrom`, once the control listener is bound or has failed, before the child's env is composed:

```go

// A replica is started only with its home's key, verified, or with

// none and a control socket its helper can refuse on (ruling 26).

- if err := b.replicaKeyGate(rt, fileEnv, ctl != nil); err != nil {

+ if err := b.replicaKeyGate(rt, fileEnv, keyLayer, ctl != nil); err != nil {

if ctl != nil {

ctl.Close()

}

@@ -2975,6 +3271,25 @@ In `spawnChildFrom`, once the control listener is bound or has failed, before th

childVars := b.agentOwnedVars(rt, controlSocket, fileEnv, ctl != nil)

```

+In `internal/activator/prepare.go`, add the `instancekey` import; in `prepareAt`, declare the layer beside `env` and read them together (the lines between are unchanged):

+```go

+ var env map[string]string

+ var keyLayer instancekey.Layer // the layer env's key came from, read with it (ruling 29)

+ if role == instancectl.RoleRun || role == instancectl.RoleSeed {

+ // p.dataDir is the instance directory here, tenant-writable ground

+ // (design §3.2), so the token file is minted and read through

+ // instancefs rather than a plain path-based open.

+ if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil {

+ return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err)

+ }

+ if _, err := b.ensureInstanceKeyFor(rt, p); err != nil {

+ return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err)

+ }

+ var err error

+ env, keyLayer, err = b.instanceEnvLayer(rt)

+```

In `internal/activator/prepare.go`, `prepareAt`, before the log listener:

```go

@@ -2982,7 +3297,7 @@ In `internal/activator/prepare.go`, `prepareAt`, before the log listener:

// seed has no control socket, the run role has the one just bound or

// none (ruling 26). The sidecar role reads no instance env.

if role == instancectl.RoleRun || role == instancectl.RoleSeed {

- if err := b.replicaKeyGate(rt, env, ctl != nil); err != nil {

+ if err := b.replicaKeyGate(rt, env, keyLayer, ctl != nil); err != nil {

if ctl != nil {

ctl.Close()

}

@@ -3041,6 +3356,8 @@ import (

"crypto/ecdh"

"errors"

"log"

+ "os"

+ "strings"

"sync"

"github.com/carlosframework/platform/internal/appconfig"

@@ -3049,21 +3366,36 @@ import (

"github.com/carlosframework/platform/internal/release"

)

-// replicaHomeKeys is ExecBackend.ReplicaHomeKeys from the box's

-// CARLOS_REPLICA_HOME_KEYS (instance-key plan rulings 26 and 27): the

-// pre-roll check's verified home-key digests for this box's replica rows.

-// Only the pre-roll check writes the list; nothing refreshes it from a

-// replica's own configuration. A value that does not parse is logged by

-// entry position and reason, never by its text, and verifies nothing, so

-// every replica launch that would be handed a key is refused until it is

-// fixed.

-func replicaHomeKeys(getenv func(string) string) func(host string) (string, bool) {

- keys, err := instancekey.ParseHomeKeys(getenv("CARLOS_REPLICA_HOME_KEYS"))

- if err != nil {

- log.Printf("agent: %v (no replica home key verifies until it is fixed)", err)

- keys = nil

- }

+// replicaHomeKeysPath is the box's list of verified home keys for its

+// replica rows (the list the plan calls CARLOS_REPLICA_HOME_KEYS):

+// "replica-host=sha256" entries, separated by commas or newlines. Only the

+// pre-roll check writes it, as root, replacing it whole (write a temporary

+// file, then rename it into place); the agent only reads it.

+const replicaHomeKeysPath = "/etc/carlos/replica-home-keys"

+// replicaHomeKeys is ExecBackend.ReplicaHomeKeys (instance-key plan rulings

+// 26, 27 and 30): it reads the list again at every call, so an entry the

+// procedure removes or replaces takes effect at the next replica launch,

+// with no agent restart and no cached copy to go stale. Nothing refreshes

+// the list from a replica's own configuration. An absent list verifies

+// nothing. A list that cannot be read or parsed is logged by the error's

+// position and reason, never its text, and verifies nothing, so every

+// replica launch that would be handed a key is refused until it is fixed.

+func replicaHomeKeys(read func() ([]byte, error)) func(host string) (string, bool) {

return func(host string) (string, bool) {

+ raw, err := read()

+ if errors.Is(err, os.ErrNotExist) {

+ return "", false

+ }

+ if err != nil {

+ log.Printf("agent: replica home keys: the list could not be read (no replica home key verifies): %v", err)

+ return "", false

+ }

+ keys, err := instancekey.ParseHomeKeys(strings.ReplaceAll(string(raw), "\n", ","))

+ if err != nil {

+ log.Printf("agent: replica home keys: %v (no replica home key verifies until it is fixed)", err)

+ return "", false

+ }

sha, ok := keys[host]

return sha, ok

}

@@ -3110,9 +3442,10 @@ func replicaConfigKey(getenv func(string) string, open func(context.Context) (re

In `internal/host/agent.go`, after the `ExecBackend` literal (beside `eb.Logs`):

```go

- // The verified home keys of this box's replica rows (L21 ruling 26),

- // until O8 reads them from each home's published env item.

- eb.ReplicaHomeKeys = replicaHomeKeys(os.Getenv)

+ // The verified home keys of this box's replica rows (L21 rulings 26

+ // and 30), read again at every replica launch, until O8 reads them

+ // from each home's published env item.

+ eb.ReplicaHomeKeys = replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(replicaHomeKeysPath) })

// The bucket's configuration, read fresh at each replica launch

// whose key is configured (ruling 27).

eb.ReplicaConfigKey = replicaConfigKey(os.Getenv, openDeployStore)

@@ -3127,7 +3460,7 @@ Expected: PASS. (With the sandbox on, the control-socket case skips and the nest

```bash

gofmt -l internal/activator internal/host internal/appconfig

-git add internal/activator/replicakey.go internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/replica_instancekey_test.go internal/activator/replica_instancekey_linux_test.go internal/host/replicahomekeys.go internal/host/replicahomekeys_test.go internal/host/agent.go internal/appconfig/instancekey.go internal/appconfig/instancekey_test.go

+git add internal/activator/replicakey.go internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/replica_instancekey_test.go internal/activator/replica_instancekey_linux_test.go internal/host/replicahomekeys.go internal/host/replicahomekeys_test.go internal/host/agent.go internal/appconfig/instancekey.go internal/appconfig/instancekey_test.go internal/activator/replicakey_race_test.go

git commit -m "activator: a replica launches only with its home's key, verified, or none and a socket its helper refuses on (L21)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

@@ -3557,7 +3890,7 @@ Replace the body of `EnsureClaimAdminToken` from the early return to its end:

}

```

-In `SnapshotClaimLaunch`, read the two layers, keep `instanceEnv`'s merge, pass a replica through Task 3b's gate (a claim launch has no control socket before O6), and record the delivered key. The identity is computed from `Env` as before, so it does not change:

+In `SnapshotClaimLaunch`, read the two layers, keep `instanceEnv`'s merge, pass a replica through Task 3b's gate with the layer from this same read (ruling 29; a claim launch has no control socket before O6), and record the delivered key. The identity is computed from `Env` as before, so it does not change:

```go

cfgEnv, ownEnv, err := b.instanceEnvParts(rt)

@@ -3568,15 +3901,17 @@ In `SnapshotClaimLaunch`, read the two layers, keep `instanceEnv`'s merge, pass

if vars["CARLOS_ADMIN_TOKEN"] == "" {

return out, hostwrite.ErrUnavailable

}

+ // The key this launch delivers and the file that supplied it, from

+ // the same read as vars: what the replica gate decides on (ruling 29)

+ // and what the process's adoption report is checked against (ruling

+ // 21, O6). Not in the identity: the env already is.

+ key, layer := instancekey.Effective(cfgEnv, ownEnv)

// A claim launch has no control socket until O6, so a replica is

// selected only with its home's key, verified (ruling 26).

- if err := b.replicaKeyGate(rt, vars, false); err != nil {

+ if err := b.replicaKeyGate(rt, vars, layer, false); err != nil {

return out, err

}

- // The key this launch delivers and the file that supplied it: what the

- // process's adoption report is checked against (ruling 21, O6). Not in

- // the identity: the env already is.

- if key, layer := instancekey.Effective(cfgEnv, ownEnv); key != "" {

+ if key != "" {

out.InstanceKeySHA256, out.InstanceKeySource = instancekey.Digest(key), instancekey.SourceOf(layer)

}

```

@@ -4644,23 +4979,25 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

```

For each replica:

- - **Its home has a key and its digest equals the replica's:** add `replica-host=<digest>` to the replica box's `CARLOS_REPLICA_HOME_KEYS`, in the agent's environment drop-in, before that box is upgraded.

+ - **Its home has a key and its digest equals the replica's:** add `replica-host=<digest>`, one per line, to the replica box's list, `/etc/carlos/replica-home-keys`, before that box is upgraded. Write the whole list to a temporary file in `/etc/carlos`, set it to 0640 root:carlos, and rename it into place (ruling 30).

- **The two digests differ:** the replica will be refused after the roll. Tell the owner, who sets the replica's key to the home's, or accepts the refusal.

- **The replica has no key:** list nothing. It starts as before wherever it has a control socket. An adopting app without its own key refuses there itself (`ErrNoInstanceKey`) until O8.

- Only this check writes `CARLOS_REPLICA_HOME_KEYS`; nothing refreshes it from a replica's configuration (ruling 27). The digests go only into the agent's environment, never into a log.

+ Only this check writes the list; nothing refreshes it from a replica's configuration (ruling 27). The agent reads it again at every replica launch and keeps no copy, so a change takes effect at the next launch with no restart (ruling 30). The digests go only into that file, never into a log.

**Operator note: changing a configured key on an app with replicas (ruling 27).** It is a key rotation, which this plan does not support for an app with data. If it must be done, coordinate it:

- 1. stop the app's replicas;

- 2. remove their entries from `CARLOS_REPLICA_HOME_KEYS` on their boxes;

- 3. change the key;

- 4. let every box sync its configuration;

- 5. re-run this check, which writes the new entries;

- 6. start the replicas.

+ 1. stop the app's replicas, and keep them stopped until the last step;

+ 2. remove their entries from the list on their boxes (the whole list rewritten and renamed into place);

+ 3. verify the invalidation took effect before touching the key: on each replica's box, `grep -c '^<replica-host>=' /etc/carlos/replica-home-keys` prints 0. The agent keeps no copy (ruling 30), so the file is what the next launch reads. Then confirm the replicas are still stopped;

+ 4. change the key;

+ 5. let every box sync its configuration;

+ 6. re-run this check, which writes the new entries;

+ 7. start the replicas.

Do the same after editing a home's own `<host>.env` by hand, or configuring a per-host key on either side. Those are the changes the agent's fresh read cannot see.

2. **Merge and release.** Merging is Paul's call, as a squash on the hub, after CI is green on the tip. Cut a platform release by the README's "Releasing the platform" recipe.

3. **Ship to every box, every region.** Use the platform roll recipe (build arm64, upload, `update-platform-binary.sh` per box, `roll-console.sh` after). The console needs nothing new, but rolls with the release as usual.

+ - **Roll every box before any configured key matters** (task 1 review, minor 6). While boxes run mixed versions, a move's readiness check (`movesreport.go:487`) fails for an instance whose own key is empty but which has a configured key: the old source reports the empty own key, and the new target reports the configured one. That is unlikely until keys are minted, but finish the roll before an owner configures a key on an app that moves.

4. **Verify on each box.**

- `journalctl -u carlos-edge --since '<roll time>' | grep 'agent: instance keys:'` must show `0 failed`. For each failure, read the per-host line above it, fix the file, and restart the instance (its start mints the key, or fails loudly with the same error). `never started here` hosts mint at their first start. Claim boxes print no backfill line: their instances mint at their next cold admission.

- `grep 'activator: replica .* not started'` must name only the replicas step 1 expected to be refused.

@@ -4672,7 +5009,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

- **An app with replica placements must not adopt with an empty `own` and only the platform key until O8.** Its replicas would have no key: an adopting replica refuses (`ErrNoInstanceKey`) wherever it has a control socket, and the agent refuses it where it has none. Such an app either passes its own key as `own` everywhere, or configures one key at app level (not per host). The pre-roll check then lists that key, and its replicas run with it.

8. **Region HA.** Do not enable relocation on a region until this plan is rolled there and O1', O2', O3', O5 and O7 are in. Platform-key and configured-key apps relocate only after O6 as well (rulings 23 and 24); until then their relocation refuses, explicitly. This is stage B Task 10's checklist item 9, as O4 rewrites it.

-**Rollback:** roll the previous binary. Keys already written stay. That is harmless: no app reads the key until it adopts, and the next forward roll finds it present. No adoption record is written before O6. `CARLOS_REPLICA_HOME_KEYS` is ignored by an older agent.

+**Rollback:** roll the previous binary. Keys already written stay. That is harmless: no app reads the key until it adopts, and the next forward roll finds it present. No adoption record is written before O6. An older agent ignores `/etc/carlos/replica-home-keys`.

## Open questions for Paul

diff --git a/docs/superpowers/plans/INSTANCE-KEY-PLAN-REPORT-R3.md b/docs/superpowers/plans/INSTANCE-KEY-PLAN-REPORT-R3.md

index 39672451a..3c294f15e 100644

--- a/docs/superpowers/plans/INSTANCE-KEY-PLAN-REPORT-R3.md

+++ b/docs/superpowers/plans/INSTANCE-KEY-PLAN-REPORT-R3.md

@@ -114,3 +114,23 @@ All three findings are accepted: two P2s and a P3, all about replicas.

- **Code blocks:** the plan's blocks, rebuilt from the document, match the tested scratch tree byte for byte. `go vet` and gofmt are clean, and builds for darwin, windows and linux/arm64 are clean.

- **Tests:** the activator launch families, run with the sandbox off, pass. So do the targeted host tests and all of `instancekey`, `hostfiles`, `appconfig`, `moves` and `moves/archive`.

- Scratch copies are deleted, and `$TMPDIR/gocache` is kept.

+## Rev 5.1 (Astra round 6)

+Both P2s are accepted. Task 1's code is unchanged; it is built and reviewed on this branch.

+- **R6-1, ruling 29:** the replica gate decides on the launch's own read. `instanceEnvLayer` returns the env and its key's layer from one read, every gate call passes that layer, and the gate reads no env file.

+ - **The interleaving test** (`TestReplicaGateDecidesOnTheLaunchesOwnRead`) has a hook rewrite `<host>.config.env` between the launch's read and the gate, as a config sync can. The launch is refused.

+ - **Red check:** with rev 5's second read restored, the app ran with the stale key.

+- **R6-2, ruling 30: read on each launch.** The list moves from the agent's environment (never re-read by a running agent) to `/etc/carlos/replica-home-keys`.

+ - The pre-roll check writes the list whole and renames it into place. The agent reads it at every replica launch and keeps no copy, so no restart is needed. A restart would bounce every instance on the box.

+ - The procedure keeps replicas stopped throughout, and checks the file before the key changes.

+- **Coordinator notes:**

+ - Task 1's blocks are now the branch's code, from c568c13b to e8487c3a: the canonical-only `ParseAdoption`, the `ParseEnv` test, the two doc comments and `TestEmptyOwnInstanceKeyShadowsNothing`.

+ - Rollout step 3 has the mixed-version moves readiness note: roll every box before any configured key matters.

+- **IK-1:** unchanged by this revision. O2' and its test were re-run against stage B's tip b9a753f0 with the reviewed `instancekey`, and pass.

+**How it was checked:**

+- **Code blocks:** rebuilt from the document on top of the branch, they match the tested tree byte for byte, and Task 1's blocks equal the branch's files. `go vet` and gofmt are clean, and builds for darwin, windows and linux/arm64 are clean.

+- **Tests:** the activator launch families, run with the sandbox off, pass. So do the targeted host tests and all of `instancekey`, `hostfiles`, `appconfig`, `moves` and `moves/archive`.

+- Scratch copies are deleted, and `$TMPDIR/gocache` is kept.