amadan
activator: nothing is minted until the row's configuration is delivered (final review I2, Astra P2; minor 2)
A first wake can land before the config sync's first render: the fast-converge pass reconciles the row ready and syncs config last, so a request in between found no <host>.config.env, Effective saw no key, and the platform minted one that shadowed the configured key for good once it arrived. SyncExec removes the file for a route that was never configured, so its existence cannot be the delivery signal; the mint now asks the bucket, as the replica gate does. ExecBackend.ConfiguredKey (the field the gate called ReplicaConfigKey) is read once, under adminTokenMu, only when a mint is next: a key the configuration sets and the box has not delivered defers the launch with ErrInstanceKeyDeferred, so does a bucket that cannot be read, and a configuration that sets no key lets the mint go ahead. Nil is a box with no configuration source. The launch fails and is retried; the next start finds the key configured. Each read is bounded by configReadTimeout and the caller's context; the startup backfill runs its whole pass under one budget and counts a deferred row apart from a failure. The host's reader follows the config guard's rule for which boxes have a configuration to read: no secrets key still renders plain values, no deployment store is an error unless CARLOS_NO_SEALED_CONFIG says there is no configuration at all. Minor 2: the zero outcome is InstanceKeyFailed, what every ensure answers beside its error, so no caller can count a refused row by reading the outcome first. The exported ensure wrappers, used only by tests, are now ensureInstanceKeyNoConfig and ensureInstanceKeyBeneathNoConfig, named for what they skip. Tests: the ensure's three answers and that the bucket is read only at a mint; the read's bounds; a first start with delivery delayed, end to end, then the configured key delivered and no key minted beside it; the same on the claim path; the reader's guard semantics; the backfill's budget and deferred count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
21 files changed,
+582
−185
docs/superpowers/plans/2026-10-10-instance-key-provision.md+29 −29internal/activator/backend_exec.go+14 −7internal/activator/claim_snapshot.go+1 −1internal/activator/instancekey.go+106 −34internal/activator/instancekey_delivery_test.go+186 −0internal/activator/instancekey_linux_test.go+4 −4internal/activator/instancekey_test.go+30 −30internal/activator/prepare.go+2 −1internal/activator/replica_instancekey_linux_test.go+4 −4internal/activator/replica_instancekey_test.go+8 −8internal/activator/replicakey.go+4 −9internal/activator/replicakey_race_test.go+2 −2internal/host/agent.go+1 −1internal/host/claim_instancekey_test.go+32 −0internal/host/claim_replica_instancekey_test.go+1 −1internal/host/configguard.go+11 −4internal/host/instancekeys.go+39 −16internal/host/instancekeys_test.go+40 −8internal/host/layoutleg.go+1 −1internal/host/replicahomekeys.go+24 −14internal/host/replicahomekeys_test.go+43 −11
diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md| index 1e16b5bbc..80f1939c6 100644 |
| --- a/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| +++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| @@ -46,7 +46,7 @@ Astra's round 5 (`CODEX-PLAN-INSTANCE-KEY-R5.md`, reviewed at 37c48a40) accepted |
| | # | Finding | Disposition | Change | Test | |
| |---|---|---|---|---| |
| -| R5-1 | P2. A stale digest admits the wrong key after an ordinary configuration change. `SyncExec` reconciles each box separately and hourly, so the home can hold B while the replica still holds A and still matches A's entry. | **Accepted** (ruling 27). Verified: `configSyncLoop` runs hourly per box (agent.go:1078). | A configured key is never changed in place; that would be a rotation, which this plan does not support. The rollout gives the procedure for any configured-key change on an app with replicas. There is also a cheap fresh-evidence check that needs no O8: a replica whose key comes from its configuration is started only if the bucket's configuration renders that same key now (`appconfig.ConfiguredKeyDigest`, through `ExecBackend.ReplicaConfigKey`). That catches a shared configuration change the replica's box has not synced. The list itself is written only by the pre-roll check, never refreshed from a replica's configuration. | `TestReplicaLaunchWithoutAControlSocket`: "the home updated, this replica's configuration delayed: refused" (it fails with the fresh check removed); "the bucket cannot be read: refused"; "no fresh reader: refused"; "residual: a change only on the home is not seen". Also `TestConfiguredKeyDigestReadsTheBucketNow` and `TestReplicaConfigKeyReadsTheBucketNow` | |
| +| R5-1 | P2. A stale digest admits the wrong key after an ordinary configuration change. `SyncExec` reconciles each box separately and hourly, so the home can hold B while the replica still holds A and still matches A's entry. | **Accepted** (ruling 27). Verified: `configSyncLoop` runs hourly per box (agent.go:1078). | A configured key is never changed in place; that would be a rotation, which this plan does not support. The rollout gives the procedure for any configured-key change on an app with replicas. There is also a cheap fresh-evidence check that needs no O8: a replica whose key comes from its configuration is started only if the bucket's configuration renders that same key now (`appconfig.ConfiguredKeyDigest`, through `ExecBackend.ConfiguredKey`). That catches a shared configuration change the replica's box has not synced. The list itself is written only by the pre-roll check, never refreshed from a replica's configuration. | `TestReplicaLaunchWithoutAControlSocket`: "the home updated, this replica's configuration delayed: refused" (it fails with the fresh check removed); "the bucket cannot be read: refused"; "no fresh reader: refused"; "residual: a change only on the home is not seen". Also `TestConfiguredKeyDigestReadsTheBucketNow` and `TestReplicaConfigKeyReadsTheBucketNow` | |
| | R5-2 | P2. O8 has no source for a configured key's value: the published `<host>.env` leaves it out (ruling 5), and the adoption record holds only its digest. | **Accepted** (ruling 28). | O8 delivers only a key the platform minted into the home's `<host>.env`. A home whose key exists only in configuration is explicitly deferred: its keyless replica keeps refusing, and its configured-key replica follows ruling 27. | `TestReplicaLaunchWithoutAControlSocket`: "a config-only home's keyless replica is refused" | |
| | R5-3 | P3. A malformed list entry can reach the log unredacted: a bare digest with no `replica-host=` lands in the error. | **Accepted.** | `ParseHomeKeys` reports the entry's position and the reason it was rejected, never the entry's text. The agent logs that error and nothing else. | `TestParseHomeKeys` (no error echoes its input); `TestReplicaHomeKeysLogsAMalformedListWithoutItsText` (the logging path) | |
| @@ -315,7 +315,7 @@ Source (c) and `ConfiguredInstanceKey` are gone: a configured key is covered by |
| **27. A configured key is never changed in place (rev 5; Paul's ruling; Astra round 5 finding 1).** Changing a configured `CARLOS_INSTANCE_KEY` on an app that has data is a key rotation, and this plan does not support rotation. `SyncExec` reconciles each box on its own schedule, hourly, so during any configured-key change the home and a replica can briefly hold different keys. |
| - **The procedure** (rollout step 1 and the operator note): stop the app's replicas; remove their entries from the box's list and check that the file no longer names them (the agent keeps no copy, ruling 30); change the key; let every box sync; re-run the pre-roll check, which writes the new entries; then start the replicas. |
| -- **The safety net.** On every replica launch whose key comes from its configuration, the agent checks two things. The key's digest must equal the replica's list entry, which only the pre-roll check writes and nothing refreshes from the replica's configuration. And the bucket's configuration must render that same key now (`ReplicaConfigKey`, a fresh read on every such launch). Any mismatch, or a read that fails, refuses the launch with a log line naming the host and the reason. |
| +- **The safety net.** On every replica launch whose key comes from its configuration, the agent checks two things. The key's digest must equal the replica's list entry, which only the pre-roll check writes and nothing refreshes from the replica's configuration. And the bucket's configuration must render that same key now (`ConfiguredKey`, a fresh read on every such launch). Any mismatch, or a read that fails, refuses the launch with a log line naming the host and the reason. |
| - **The fresh read is the cheap fresh evidence that works without O8.** A shared configuration change reaches the bucket before any box syncs it, so a replica whose box lags the home is refused. |
| *Residual (procedure not followed):* a change the bucket does not hold is invisible to the replica's box. Examples are the home's own `<host>.env` edited by hand, or a per-host key configured on the home alone. The replica's list entry and the bucket still agree, so it runs with the old key. A test documents this case; only the procedure, or O8's fresh evidence for platform keys, closes it. |
| @@ -750,7 +750,7 @@ Two changes to the ask above, and only the second is new: |
| - Modify `internal/activator/admintoken.go` (`parseEnvBytes` calls `instancekey.ParseEnv`) and `internal/activator/backend_exec.go` (`mergeEnv`, `LaunchEnvironment`'s shadow list). |
| - Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `ensureInstanceKeyFor`, `BackfillInstanceKey`, `settleClaimInstanceEnv`, `InstanceKeyVar`, `InstanceKeyOutcome`). |
| - Create `internal/activator/instancekey_test.go`, `instancekey_linux_test.go`, `replica_instancekey_test.go`, `replica_instancekey_linux_test.go`, `claim_instancekey_test.go` and `claimretry_instancekey_linux_test.go`. |
| -- Create `internal/activator/replicakey.go` (`replicaKeyGate`, `ErrReplicaKey`), `internal/host/replicahomekeys.go` (`replicaHomeKeys`, `replicaConfigKey`) and `internal/appconfig/instancekey.go` (`ConfiguredKeyDigest`), with `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` and `internal/activator/replicakey_race_test.go`. Modify `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` and `internal/host/agent.go` (Task 3b). |
| +- Create `internal/activator/replicakey.go` (`replicaKeyGate`, `ErrReplicaKey`), `internal/host/replicahomekeys.go` (`replicaHomeKeys`, `configuredKey`) and `internal/appconfig/instancekey.go` (`ConfiguredKeyDigest`), with `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` and `internal/activator/replicakey_race_test.go`. Modify `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ConfiguredKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` and `internal/host/agent.go` (Task 3b). |
| - Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369), `internal/activator/prepare.go` (:60) and `internal/activator/claim_snapshot.go` (the snapshot's fields, `EnsureClaimAdminToken` :40-67, `SnapshotClaimLaunch`'s env read). |
| - Create `internal/activator/keyadoption.go`, `keyadoption_test.go` and `keyadoption_linux_test.go`: the adoption verb, mounted by O6. |
| - Create `internal/hostfiles/relay.go` and `relay_test.go`: `PlatformRelay`. |
| @@ -2573,12 +2573,12 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Files:** |
| - Create: `internal/activator/replicakey.go`, `internal/host/replicahomekeys.go`, `internal/appconfig/instancekey.go` |
| -- Modify: `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild`, `spawnChildFrom`, `StartWorkerStaged` and `startSidecarChildFrom`), `internal/activator/prepare.go` (`prepareAt`), `internal/host/agent.go` (the wiring) |
| +- Modify: `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ConfiguredKey`; the gate in `seedChild`, `spawnChildFrom`, `StartWorkerStaged` and `startSidecarChildFrom`), `internal/activator/prepare.go` (`prepareAt`), `internal/host/agent.go` (the wiring) |
| - Test: `internal/activator/replica_instancekey_test.go`, `internal/activator/replica_instancekey_linux_test.go`, `internal/activator/replicakey_race_test.go`, `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` |
| **Interfaces:** |
| - Consumes: Task 1's `instancekey.Digest`, `Effective` and `ParseHomeKeys`; appconfig's `LoadMergedHostVersions`, `RenderEnv`, `routeBinding`, and test kit `execKit`, `SetEnv`, `ApplyInstance`, `Seal`; Task 2's `InstanceKeyReplica` (never minted); test helpers `testExecBackend`, `tmpSock`, `nestedBackend`, `readEnvFile`. |
| -- Produces: `ExecBackend.ReplicaHomeKeys func(host string) (sha256 string, ok bool)` and `ReplicaConfigKey func(ctx, rt) (sha256 string, err error)`; `(*ExecBackend).replicaKeyGate(rt, env, layer, haveControl) error`, with `layer` the one the env's key came from, decided by the same read that composed the env (`instanceEnvLayer`, ruling 29); `activator.ErrReplicaKey`; `appconfig.ConfiguredKeyDigest(ctx, s, priv, rt) (string, error)`; `host.replicaHomeKeys(read)`, which parses the list the reader returns (`/etc/carlos/replica-home-keys`, read again at every replica launch; ruling 30), and `host.replicaConfigKey(getenv, open)`, the fresh read (ruling 27). Task 4's snapshot calls the gate. |
| +- Produces: `ExecBackend.ReplicaHomeKeys func(host string) (sha256 string, ok bool)` and `ConfiguredKey func(ctx, rt) (sha256 string, err error)`; `(*ExecBackend).replicaKeyGate(rt, env, layer, haveControl) error`, with `layer` the one the env's key came from, decided by the same read that composed the env (`instanceEnvLayer`, ruling 29); `activator.ErrReplicaKey`; `appconfig.ConfiguredKeyDigest(ctx, s, priv, rt) (string, error)`; `host.replicaHomeKeys(read)`, which parses the list the reader returns (`/etc/carlos/replica-home-keys`, read again at every replica launch; ruling 30), and `host.configuredKey(getenv, open)`, the fresh read (ruling 27). Task 4's snapshot calls the gate. |
| **Built:** two launch paths rev 5.1 left ungated pass the gate as a seed does (no control socket: the home's verified key or none, refused otherwise), by the ledger's rulings on the 3b build and its review. The sidecar worker (`StartWorkerStaged`) reads the env and its key's layer in one read, and when the agent-issued `extra` carries `CARLOS_INSTANCE_KEY` the layer is set to none, so the gate refuses it as a key of unknown origin: a worker never mints and `extra` has no business carrying a key. The legacy co-spawned sidecar (`startSidecarChildFrom`) gates its own env file's key the same way (`sidecarKeyLayer`). Refusal and log text name the file path, not the variable the plan calls the list by. The bucket read in the gate runs on its own bounded context (`prepareAt` has none to thread). |
| @@ -2714,13 +2714,13 @@ type replicaCase struct { |
| start string // "refused" (no app runs) or the helper outcome wanted |
| wantKeyFromOwnOK bool // a home row: the outcome is the key minted into <host>.env |
| // bucket is the key the replica's configuration in the bucket renders |
| - // now (ReplicaConfigKey; "" renders none). bucketDown makes that read |
| - // fail; noBucket leaves ReplicaConfigKey nil. |
| + // now (ConfiguredKey; "" renders none). bucketDown makes that read |
| + // fail; noBucket leaves ConfiguredKey nil. |
| bucket string |
| bucketDown, noBucket bool |
| } |
| -// freshConfig is ExecBackend.ReplicaConfigKey for one case. |
| +// freshConfig is ExecBackend.ConfiguredKey for one case. |
| func freshConfig(tc replicaCase) func(context.Context, registry.Route) (string, error) { |
| if tc.noBucket { |
| return nil |
| @@ -2787,7 +2787,7 @@ func TestReplicaLaunchWithoutAControlSocket(t *testing.T) { |
| return instancekey.Digest(tc.homeKey), h == rt.Host |
| } |
| } |
| - b.ReplicaConfigKey = freshConfig(tc) |
| + b.ConfiguredKey = freshConfig(tc) |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| defer cancel() |
| @@ -2841,7 +2841,7 @@ func TestReplicaSeedIsGated(t *testing.T) { |
| } |
| } |
| b.ReplicaHomeKeys = func(h string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - b.ReplicaConfigKey = freshConfig(replicaCase{bucket: homeKey}) |
| + b.ConfiguredKey = freshConfig(replicaCase{bucket: homeKey}) |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| err := b.Seed(context.Background(), rt) |
| got := outcomeOf(out) |
| @@ -2938,7 +2938,7 @@ func TestNestedReplicaPrepareIsGated(t *testing.T) { |
| b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| rt.Replica = true |
| b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| dir := provision.InstanceDir(b.DataDir, rt.Host) |
| if err := os.MkdirAll(dir, 0o700); err != nil { |
| t.Fatal(err) |
| @@ -3055,7 +3055,7 @@ func TestReplicaGateDecidesOnTheLaunchesOwnRead(t *testing.T) { |
| t.Fatal(err) |
| } |
| b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true } |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed |
| syncs := 0 |
| replicaGateHook = func() { |
| syncs++ |
| @@ -3100,7 +3100,7 @@ func TestReplicaGateReadsNoEnvFile(t *testing.T) { |
| bucket string |
| ok bool |
| }{{keyA, true}, {"", false}} { |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { |
| if tc.bucket == "" { |
| return "", nil |
| } |
| @@ -3218,7 +3218,7 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", App: "shop", Account: "acct", Replica: true} |
| opened := 0 |
| open := func(context.Context) (release.Store, error) { opened++; return s, nil } |
| - read := replicaConfigKey(func(k string) string { |
| + read := configuredKey(func(k string) string { |
| if k == "CARLOS_SECRETS_KEY" { |
| return priv |
| } |
| @@ -3232,11 +3232,11 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| if opened != 1 { |
| t.Fatalf("the store was opened %d times, want once", opened) |
| } |
| - failing := replicaConfigKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { return nil, errors.New("bucket unreachable") }) |
| + failing := configuredKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { return nil, errors.New("bucket unreachable") }) |
| if _, err := failing(ctx, rt); err == nil { |
| t.Fatal("an unreachable bucket was not an error") |
| } |
| - if _, err := replicaConfigKey(func(string) string { return "" }, open)(ctx, rt); err == nil { |
| + if _, err := configuredKey(func(string) string { return "" }, open)(ctx, rt); err == nil { |
| t.Fatal("no secrets key was not an error") |
| } |
| } |
| @@ -3245,7 +3245,7 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| - [ ] **Step 2: Run them to verify they fail** |
| Run: `nice -n 19 go test ./internal/activator/ -run 'TestReplicaLaunchWithoutAControlSocket|TestReplicaSeedIsGated|TestReplicaGate|TestKeylessReplica|TestNestedReplicaPrepareIsGated' -count=1 -v; nice -n 19 go test ./internal/host/ -run 'TestReplicaHomeKeys|TestReplicaConfigKey' -count=1; nice -n 19 go test ./internal/appconfig/ -run TestConfiguredKeyDigest -count=1` |
| -Expected: build failures, `undefined: ErrReplicaKey` (and `ReplicaHomeKeys`, `ReplicaConfigKey`, `replicaHomeKeys`, `replicaConfigKey`, `ConfiguredKeyDigest`). In the scratch build, with the gate present but disabled, the keyless and differing-key cases failed: the app reported "minted", or ran with the wrong key. That is Astra's round 4 failure. With only the fresh read disabled, "the home updated, this replica's configuration delayed" failed: the app ran with the old key. That is Astra's round 5 failure. With rev 5's second read of the files restored in the gate, `TestReplicaGateDecidesOnTheLaunchesOwnRead` failed: the app ran with A. That is Astra's round 6 failure. |
| +Expected: build failures, `undefined: ErrReplicaKey` (and `ReplicaHomeKeys`, `ConfiguredKey`, `replicaHomeKeys`, `configuredKey`, `ConfiguredKeyDigest`). In the scratch build, with the gate present but disabled, the keyless and differing-key cases failed: the app reported "minted", or ran with the wrong key. That is Astra's round 4 failure. With only the fresh read disabled, "the home updated, this replica's configuration delayed" failed: the app ran with the old key. That is Astra's round 5 failure. With rev 5's second read of the files restored in the gate, `TestReplicaGateDecidesOnTheLaunchesOwnRead` failed: the app ran with A. That is Astra's round 6 failure. |
| - [ ] **Step 3: Write the gate** |
| @@ -3264,7 +3264,7 @@ package activator |
| // digest is its entry in the box's verified list (ReplicaHomeKeys, which |
| // only the pre-roll check writes, read again at every launch) and, when |
| // the key comes from the replica's configuration, the configuration in |
| -// the bucket renders that same key now (ReplicaConfigKey). Where the key |
| +// the bucket renders that same key now (ConfiguredKey). Where the key |
| // came from is decided by the same read that composed the env the child |
| // gets (instanceEnvLayer, ruling 29), never by reading the files again. A key that differs from either, or |
| // that cannot be checked, refuses the launch with a log line. The fresh |
| @@ -3297,8 +3297,8 @@ import ( |
| // ErrReplicaKey is a replica launch refused by replicaKeyGate. |
| var ErrReplicaKey = errors.New("activator: replica launch refused: its instance key is not its home's") |
| -// replicaConfigTimeout bounds replicaKeyGate's fresh read of the bucket. |
| -const replicaConfigTimeout = 10 * time.Second |
| +// configReadTimeout bounds replicaKeyGate's fresh read of the bucket. |
| +const configReadTimeout = 10 * time.Second |
| // instanceEnvLayer is instanceEnv returning, from the same read, the layer |
| // its CARLOS_INSTANCE_KEY came from (ruling 29). mergeEnv delivers exactly |
| @@ -3359,12 +3359,12 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, l |
| default: |
| return refuse("where its " + instancekey.Var + " came from is unknown") |
| } |
| - if b.ReplicaConfigKey == nil { |
| + if b.ConfiguredKey == nil { |
| return refuse("its configured " + instancekey.Var + " cannot be checked against the bucket's configuration") |
| } |
| - ctx, cancel := context.WithTimeout(context.Background(), replicaConfigTimeout) |
| + ctx, cancel := context.WithTimeout(context.Background(), configReadTimeout) |
| defer cancel() |
| - fresh, err := b.ReplicaConfigKey(ctx, rt) |
| + fresh, err := b.ConfiguredKey(ctx, rt) |
| switch { |
| case err != nil: |
| return refuse("the bucket's configuration could not be read: " + err.Error()) |
| @@ -3386,12 +3386,12 @@ In `internal/activator/backend_exec.go`, add the two fields to `ExecBackend`, af |
| // a host, means the home key cannot be verified, and a replica launch |
| // that would be handed a key is refused (replicaKeyGate). |
| ReplicaHomeKeys func(host string) (sha256 string, ok bool) |
| - // ReplicaConfigKey reads rt's configuration from the bucket now and |
| + // ConfiguredKey reads rt's configuration from the bucket now and |
| // returns the SHA-256 of the CARLOS_INSTANCE_KEY it renders, "" for |
| // none (appconfig.ConfiguredKeyDigest). replicaKeyGate refuses a |
| // replica whose configured key is not that digest: a configuration |
| // change is in flight (ruling 27). Nil refuses every configured key. |
| - ReplicaConfigKey func(ctx context.Context, rt registry.Route) (sha256 string, err error) |
| + ConfiguredKey func(ctx context.Context, rt registry.Route) (sha256 string, err error) |
| ``` |
| In `seedChild`, read the env and its key's layer together (`instanceEnvLayer`, in `replicakey.go`), then gate: |
| @@ -3564,12 +3564,12 @@ func replicaHomeKeys(read func() ([]byte, error)) func(host string) (string, boo |
| } |
| } |
| -// replicaConfigKey is ExecBackend.ReplicaConfigKey (ruling 27): the digest |
| +// configuredKey is ExecBackend.ConfiguredKey (ruling 27): the digest |
| // of the key rt's configuration in the bucket renders now, read with the |
| // box's secrets key through the deploy store. The store and key are opened |
| // on first use and kept once both open; a failure is returned, so the |
| // replica is refused, and is tried again at its next launch. |
| -func replicaConfigKey(getenv func(string) string, open func(context.Context) (release.Store, error)) func(context.Context, registry.Route) (string, error) { |
| +func configuredKey(getenv func(string) string, open func(context.Context) (release.Store, error)) func(context.Context, registry.Route) (string, error) { |
| var ( |
| mu sync.Mutex |
| store release.Store |
| @@ -3611,7 +3611,7 @@ In `internal/host/agent.go`, after the `ExecBackend` literal (beside `eb.Logs`): |
| eb.ReplicaHomeKeys = replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(replicaHomeKeysPath) }) |
| // The bucket's configuration, read fresh at each replica launch |
| // whose key is configured (ruling 27). |
| - eb.ReplicaConfigKey = replicaConfigKey(os.Getenv, openDeployStore) |
| + eb.ConfiguredKey = configuredKey(os.Getenv, openDeployStore) |
| ``` |
| - [ ] **Step 4: Run the tests, then the launch tests around them** |
| @@ -3965,7 +3965,7 @@ func TestClaimReplicaLaunchIsGated(t *testing.T) { |
| f := newAdminTokenFixture(t, "box-1") |
| f.rt.Replica = true |
| f.backend.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - f.backend.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| + f.backend.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| own := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n" |
| if err := os.WriteFile(f.envPath(), []byte(own), 0o600); err != nil { |
| t.Fatal(err) |
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go| index 7566a90b3..f327f208e 100644 |
| --- a/internal/activator/backend_exec.go |
| +++ b/internal/activator/backend_exec.go |
| @@ -104,12 +104,19 @@ type ExecBackend struct { |
| // a host, means the home key cannot be verified, and a replica launch |
| // that would be handed a key is refused (replicaKeyGate). |
| ReplicaHomeKeys func(host string) (sha256 string, ok bool) |
| - // ReplicaConfigKey reads rt's configuration from the bucket now and |
| + // ConfiguredKey reads rt's configuration from the bucket now and |
| // returns the SHA-256 of the CARLOS_INSTANCE_KEY it renders, "" for |
| - // none (appconfig.ConfiguredKeyDigest). replicaKeyGate refuses a |
| - // replica whose configured key is not that digest: a configuration |
| - // change is in flight (ruling 27). Nil refuses every configured key. |
| - ReplicaConfigKey func(ctx context.Context, rt registry.Route) (sha256 string, err error) |
| + // none (appconfig.ConfiguredKeyDigest). Two readers ask it. The mint |
| + // asks before it mints, so a key the configuration sets is never |
| + // shadowed by one minted while that configuration was still on its |
| + // way to the box (ensureInstanceKey; ruling 5): a key the bucket sets |
| + // and the box has not delivered defers the launch, and nil is a box |
| + // with no configuration source, where nothing is on its way. The |
| + // replica gate asks at every launch whose key is configured, and |
| + // refuses a replica whose configured key is not that digest: a |
| + // configuration change is in flight (ruling 27); there nil refuses |
| + // every configured key. |
| + ConfiguredKey func(ctx context.Context, rt registry.Route) (sha256 string, err error) |
| // Cache, when non-nil, adds the purge verb to each spawned instance's |
| // control socket so it can invalidate its own cached responses right |
| // after a write. Nil disables that ONE verb, not the socket: the control |
| @@ -848,7 +855,7 @@ func (b *ExecBackend) seedChild(ctx context.Context, rt registry.Route) error { |
| } |
| // The instance key, before the env is read for the same reason: a seed |
| // is usually the first process of a fresh instance (L21). |
| - if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil { |
| + if _, err := b.ensureInstanceKeyFor(ctx, rt, paths); err != nil { |
| return fmt.Errorf("instance key for %s: %w", host, err) |
| } |
| fileEnv, keyLayer, err := b.instanceEnvLayer(rt) |
| @@ -1398,7 +1405,7 @@ func (b *ExecBackend) spawnChildFrom(ctx context.Context, rt registry.Route, soc |
| // wake the same way: an app that takes its key from the platform |
| // refuses to start without one (L21). It returns only once the key is |
| // durable, so no child starts with a key a power cut could take back. |
| - if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil { |
| + if _, err := b.ensureInstanceKeyFor(ctx, rt, paths); err != nil { |
| return nil, "", fmt.Errorf("instance key for %s: %w", host, err) |
| } |
diff --git a/internal/activator/claim_snapshot.go b/internal/activator/claim_snapshot.go| index 1e9b45835..21dfe1332 100644 |
| --- a/internal/activator/claim_snapshot.go |
| +++ b/internal/activator/claim_snapshot.go |
| @@ -108,7 +108,7 @@ func (b *ExecBackend) EnsureClaimAdminToken(ctx context.Context, rt registry.Rou |
| // The one place the claim launch source names the host around the |
| // key's error: the refusal itself does not, and every other launch |
| // path wraps it the same way. |
| - if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil { |
| + if _, err := b.ensureInstanceKeyFor(ctx, rt, paths); err != nil { |
| return fmt.Errorf("instance key for %s: %w", rt.Host, err) |
| } |
| return nil |
diff --git a/internal/activator/instancekey.go b/internal/activator/instancekey.go| index 72e9744dc..79a6629eb 100644 |
| --- a/internal/activator/instancekey.go |
| +++ b/internal/activator/instancekey.go |
| @@ -18,6 +18,15 @@ package activator |
| // the rule mergeEnv applies): a key the rendered configuration |
| // supplies is the app's key and nothing is minted; an empty own value |
| // is no key; |
| +// - nothing is minted until the row's configuration is here. The files |
| +// say what this box has delivered, not what the configuration sets: a |
| +// first wake can land before the config sync's first render, and a |
| +// key minted then would shadow the configured one for good once it |
| +// arrived. So before it mints, the ensure asks the bucket |
| +// (ExecBackend.ConfiguredKey, the replica gate's reader) whether the |
| +// configuration sets a key now; if it does, or the bucket cannot say, |
| +// the launch is deferred (ErrInstanceKeyDeferred) and retried once the |
| +// sync has delivered the file; |
| // - durable every time: before an ensure reports success the file is |
| // 0600 (fchmod on its descriptor), the file is fsynced and so is its |
| // directory, when the key was already there too. A sync that failed |
| @@ -38,6 +47,7 @@ package activator |
| // gets the same from instancefs. |
| import ( |
| + "context" |
| "crypto/rand" |
| "encoding/base64" |
| "errors" |
| @@ -46,6 +56,7 @@ import ( |
| "os" |
| "path/filepath" |
| "strings" |
| + "time" |
| "github.com/carlosframework/platform/internal/instancefs" |
| "github.com/carlosframework/platform/internal/instancekey" |
| @@ -61,13 +72,21 @@ const InstanceKeyVar = instancekey.Var |
| // every key minted with it; changing it is a rotation. |
| const instanceKeyBytes = 32 |
| -// InstanceKeyOutcome is what one ensure did. |
| +// InstanceKeyOutcome is what one ensure did. Its zero value is |
| +// InstanceKeyFailed, what every ensure answers beside an error, so a |
| +// caller that reads the outcome before the error can never count a |
| +// refused row as anything. |
| type InstanceKeyOutcome int |
| const ( |
| + // InstanceKeyFailed: the ensure returned an error and reports nothing. |
| + // It refused the row (fenced, conflicting, a configuration not yet |
| + // delivered) or failed part way (an open, a write, a sync); the error |
| + // says which. Nothing it may have written is reported as durable. |
| + InstanceKeyFailed InstanceKeyOutcome = iota |
| // InstanceKeyPresent: the own env file already carried a key. No byte |
| // was written; the file was made durable. |
| - InstanceKeyPresent InstanceKeyOutcome = iota |
| + InstanceKeyPresent |
| // InstanceKeyMinted: a key was appended and made durable. |
| InstanceKeyMinted |
| // InstanceKeyFromConfig: the rendered configuration supplies the key. |
| @@ -84,6 +103,8 @@ const ( |
| func (o InstanceKeyOutcome) String() string { |
| switch o { |
| + case InstanceKeyFailed: |
| + return "failed" |
| case InstanceKeyPresent: |
| return "present" |
| case InstanceKeyMinted: |
| @@ -157,22 +178,57 @@ func configEnvName(envName string) string { |
| return strings.TrimSuffix(envName, ".env") + ".config.env" |
| } |
| -// EnsureInstanceKey makes sure path (an instance's own <host>.env on the |
| -// flat layout) carries CARLOS_INSTANCE_KEY, creating the file if needed. |
| -// Launch paths call it right after EnsureAdminToken. |
| -func EnsureInstanceKey(path string) (InstanceKeyOutcome, error) { |
| - return ensureInstanceKey(filepath.Dir(path), filepath.Base(path), false, true) |
| +// ErrInstanceKeyDeferred is a first start held back because the row's |
| +// configuration is not here yet: the bucket's configuration sets |
| +// CARLOS_INSTANCE_KEY and this box has not delivered it, or the bucket |
| +// could not be read to tell. Nothing is minted (ruling 5: a configured key |
| +// is the app's, and one minted while it was on its way would shadow it for |
| +// good). The launch fails and is retried: the config sync's next pass |
| +// delivers the file, and the next start finds the key configured. |
| +var ErrInstanceKeyDeferred = errors.New("activator: instance key not minted: the instance's configuration is not delivered") |
| + |
| +// configReadTimeout bounds each read of the bucket's configuration, by |
| +// the mint and by the replica gate. |
| +const configReadTimeout = 10 * time.Second |
| + |
| +// configuredKeyReader answers, for one row, the SHA-256 of the |
| +// CARLOS_INSTANCE_KEY the bucket's configuration renders now, "" for |
| +// none. nil is a box with no configuration source. |
| +type configuredKeyReader func(context.Context) (string, error) |
| + |
| +// configuredKeyFor is b.ConfiguredKey for rt, each read bounded by |
| +// configReadTimeout; nil when the backend has no reader. |
| +func (b *ExecBackend) configuredKeyFor(rt registry.Route) configuredKeyReader { |
| + if b.ConfiguredKey == nil { |
| + return nil |
| + } |
| + return func(ctx context.Context) (string, error) { |
| + ctx, cancel := context.WithTimeout(ctx, configReadTimeout) |
| + defer cancel() |
| + return b.ConfiguredKey(ctx, rt) |
| + } |
| } |
| -// EnsureInstanceKeyBeneath is EnsureInstanceKey for a nested row's own env |
| -// file, read and written through instancefs because the directory is |
| -// tenant-writable (isolated-exec design §3.2). |
| -func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) { |
| - return ensureInstanceKey(dir, name, true, true) |
| +// ensureInstanceKeyNoConfig makes sure path (an instance's own <host>.env |
| +// on the flat layout) carries CARLOS_INSTANCE_KEY, creating the file if |
| +// needed, for a row whose configuration can set no key: there is no |
| +// reader, so nothing is asked before the mint. Tests use it; every launch |
| +// path goes through ensureInstanceKeyFor, which asks the bucket first. |
| +func ensureInstanceKeyNoConfig(path string) (InstanceKeyOutcome, error) { |
| + return ensureInstanceKey(context.Background(), filepath.Dir(path), filepath.Base(path), false, true, nil) |
| +} |
| + |
| +// ensureInstanceKeyBeneathNoConfig is ensureInstanceKeyNoConfig for a |
| +// nested row's own env file, read and written through instancefs because |
| +// the directory is tenant-writable (isolated-exec design §3.2). |
| +func ensureInstanceKeyBeneathNoConfig(dir, name string) (InstanceKeyOutcome, error) { |
| + return ensureInstanceKey(context.Background(), dir, name, true, true, nil) |
| } |
| // ensureInstanceKeyFor is the launch paths' ensure for rt: none for a |
| // replica row, the nested body for a nested row, the flat one otherwise. |
| +// ctx bounds the one bucket read a mint makes; a launch with none passes |
| +// context.Background(). |
| // |
| // A fenced or conflicting row is refused and nothing is written, as |
| // BackfillInstanceKey refuses it. Every launch dispatcher (Start, |
| @@ -184,18 +240,15 @@ func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) { |
| // (isolated-exec design §8.2). The refusal is decided on p, the paths the |
| // mint would use. Every caller names the host around the error ("instance |
| // key for <host>: ..."), so the refusal does not. |
| -func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) { |
| +func (b *ExecBackend) ensureInstanceKeyFor(ctx context.Context, rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) { |
| if rt.Replica { |
| return InstanceKeyReplica, nil |
| } |
| switch p.layout { |
| case LayoutFenced, LayoutConflict: |
| - return InstanceKeyPresent, fmt.Errorf("not minted: %s", p.reason) |
| - } |
| - if p.layout == LayoutNested { |
| - return EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env") |
| + return InstanceKeyFailed, fmt.Errorf("not minted: %s", p.reason) |
| } |
| - return EnsureInstanceKey(p.envFile) |
| + return ensureInstanceKey(ctx, p.dataDir, rt.Host+".env", p.layout == LayoutNested, true, b.configuredKeyFor(rt)) |
| } |
| // BackfillInstanceKey is the startup backfill's step for one row: a key is |
| @@ -206,18 +259,16 @@ func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) ( |
| // refuse to start (fenced, conflict) is an error and nothing is written. |
| // The agent's backfill names the host when it logs the error, so the |
| // refusal does not. |
| -func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error) { |
| +func (b *ExecBackend) BackfillInstanceKey(ctx context.Context, rt registry.Route) (InstanceKeyOutcome, error) { |
| if rt.Replica { |
| return InstanceKeyReplica, nil |
| } |
| p := b.pathsFor(rt) |
| switch p.layout { |
| - case LayoutNested: |
| - return ensureInstanceKey(p.dataDir, rt.Host+".env", true, false) |
| case LayoutFenced, LayoutConflict: |
| - return InstanceKeyPresent, fmt.Errorf("not backfilled: %s", p.reason) |
| + return InstanceKeyFailed, fmt.Errorf("not backfilled: %s", p.reason) |
| } |
| - return ensureInstanceKey(p.dataDir, rt.Host+".env", false, false) |
| + return ensureInstanceKey(ctx, p.dataDir, rt.Host+".env", p.layout == LayoutNested, false, b.configuredKeyFor(rt)) |
| } |
| // openInstanceEnv opens an own env file for append. Only a create passes |
| @@ -295,8 +346,10 @@ func settleInstanceEnv(f *os.File, dir, path string) error { |
| // ensureInstanceKey is the shared body. It holds adminTokenMu, the lock |
| // every mint into <host>.env takes, so a key append and a token append can |
| // never interleave. create false is the backfill: an absent file is |
| -// InstanceKeyNoEnvFile and nothing is created. |
| -func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutcome, error) { |
| +// InstanceKeyNoEnvFile and nothing is created. configured is asked, under |
| +// the lock, only when nothing delivered here sets a key and a mint is |
| +// next: the one bucket read a row ever costs. |
| +func ensureInstanceKey(ctx context.Context, dir, name string, beneath, create bool, configured configuredKeyReader) (InstanceKeyOutcome, error) { |
| path := filepath.Join(dir, name) |
| adminTokenMu.Lock() |
| defer adminTokenMu.Unlock() |
| @@ -316,11 +369,11 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| return InstanceKeyNoEnvFile, nil |
| } |
| case err != nil: |
| - return InstanceKeyPresent, fmt.Errorf("read %s: %w", path, err) |
| + return InstanceKeyFailed, fmt.Errorf("read %s: %w", path, err) |
| } |
| cfg, err := read(configEnvName(name)) |
| if err != nil && !errors.Is(err, os.ErrNotExist) { |
| - return InstanceKeyPresent, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err) |
| + return InstanceKeyFailed, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err) |
| } |
| // The decision is the effective env's: the key the launch will hand |
| // the app, by the same rule mergeEnv applies. |
| @@ -341,17 +394,36 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| } |
| f, err := openInstanceEnv(dir, name, beneath, false) |
| if err != nil { |
| - return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err) |
| + return InstanceKeyFailed, fmt.Errorf("open %s: %w", path, err) |
| } |
| defer f.Close() |
| if err := settleInstanceEnv(f, dir, path); err != nil { |
| - return InstanceKeyPresent, err |
| + return InstanceKeyFailed, err |
| } |
| return InstanceKeyPresent, nil |
| } |
| + // Nothing delivered here sets a key. The configuration may still: a |
| + // first wake can run before the config sync's first render of |
| + // <host>.config.env (the fast-converge pass reconciles the row ready |
| + // and syncs config last), and a key minted in that gap would shadow |
| + // the configured one for good once it arrived (Effective: the own file |
| + // wins). So ask the bucket now, and mint only when the configuration |
| + // sets no key. A key it sets is not here yet: defer, and the next |
| + // start finds it delivered. A bucket that cannot be read cannot say, |
| + // and defers the same way. With no reader there is no configuration |
| + // source, and nothing can be on its way. |
| + if configured != nil { |
| + digest, err := configured(ctx) |
| + switch { |
| + case err != nil: |
| + return InstanceKeyFailed, fmt.Errorf("%w: its configuration could not be read: %w", ErrInstanceKeyDeferred, err) |
| + case digest != "": |
| + return InstanceKeyFailed, fmt.Errorf("%w: its configuration sets %s and this box has not delivered it yet (the next config sync does, and the next start finds it)", ErrInstanceKeyDeferred, InstanceKeyVar) |
| + } |
| + } |
| minted, err := mintInstanceKey() |
| if err != nil { |
| - return InstanceKeyPresent, fmt.Errorf("mint instance key for %s: %w", path, err) |
| + return InstanceKeyFailed, fmt.Errorf("mint instance key for %s: %w", path, err) |
| } |
| if instanceKeyBeforeOpen != nil { |
| instanceKeyBeforeOpen(path) |
| @@ -361,7 +433,7 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill |
| } |
| if err != nil { |
| - return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err) |
| + return InstanceKeyFailed, fmt.Errorf("open %s: %w", path, err) |
| } |
| defer f.Close() |
| // The length the append starts from, taken from the open descriptor: |
| @@ -370,7 +442,7 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| // appended since the read. |
| st, err := f.Stat() |
| if err != nil { |
| - return InstanceKeyPresent, fmt.Errorf("stat %s: %w", path, err) |
| + return InstanceKeyFailed, fmt.Errorf("stat %s: %w", path, err) |
| } |
| prefix := "" |
| if len(raw) > 0 && raw[len(raw)-1] != '\n' { |
| @@ -381,10 +453,10 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| if err == nil { |
| err = io.ErrShortWrite |
| } |
| - return InstanceKeyPresent, undoInstanceKeyAppend(f, st.Size(), path, err) |
| + return InstanceKeyFailed, undoInstanceKeyAppend(f, st.Size(), path, err) |
| } |
| if err := settleInstanceEnv(f, dir, path); err != nil { |
| - return InstanceKeyPresent, err |
| + return InstanceKeyFailed, err |
| } |
| return InstanceKeyMinted, nil |
| } |
diff --git a/internal/activator/instancekey_delivery_test.go b/internal/activator/instancekey_delivery_test.go| new file mode 100644 |
| index 000000000..a928e5b5d |
| --- /dev/null |
| +++ b/internal/activator/instancekey_delivery_test.go |
| @@ -0,0 +1,186 @@ |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/instancekey" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// bucketKey is ExecBackend.ConfiguredKey for a bucket whose configuration |
| +// renders key ("" for none), counting its reads; down makes every read |
| +// fail. |
| +type bucketKey struct { |
| + key string |
| + down bool |
| + reads int |
| +} |
| + |
| +func (b *bucketKey) read(context.Context, registry.Route) (string, error) { |
| + b.reads++ |
| + if b.down { |
| + return "", errors.New("bucket unreachable") |
| + } |
| + if b.key == "" { |
| + return "", nil |
| + } |
| + return instancekey.Digest(b.key), nil |
| +} |
| + |
| +// Final review I2 (Astra's P2), ruling 5: nothing is minted until the |
| +// row's configuration is delivered. The files say what this box has |
| +// delivered; the bucket says what the configuration sets. A key the bucket |
| +// sets and the box has not delivered defers the mint; a bucket that cannot |
| +// be read defers it the same way; a configuration that sets no key lets |
| +// the mint proceed. The bucket is asked only when a mint is next: a key |
| +// already delivered, in either file, costs no read. |
| +func TestEnsureInstanceKeyAsksTheBucketBeforeItMints(t *testing.T) { |
| + tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n" |
| + for _, tc := range []struct { |
| + name string |
| + own, config string |
| + bucket *bucketKey |
| + want InstanceKeyOutcome |
| + deferred bool |
| + reads int |
| + }{ |
| + {"the configuration sets a key the box has not delivered: deferred", tok, "", &bucketKey{key: "key-A"}, InstanceKeyFailed, true, 1}, |
| + {"the bucket cannot be read: deferred", tok, "", &bucketKey{down: true}, InstanceKeyFailed, true, 1}, |
| + {"the configuration sets no key: minted", tok, "", &bucketKey{}, InstanceKeyMinted, false, 1}, |
| + {"no configuration source: minted, nothing asked", tok, "", nil, InstanceKeyMinted, false, 0}, |
| + {"the configured key is delivered: from config, nothing asked", tok, InstanceKeyVar + "=key-A\n", &bucketKey{key: "key-A"}, InstanceKeyFromConfig, false, 0}, |
| + {"an own key: present, nothing asked", tok + InstanceKeyVar + "=" + strings.Repeat("K", 43) + "=\n", "", &bucketKey{key: "key-A"}, InstanceKeyPresent, false, 0}, |
| + } { |
| + t.Run(tc.name, func(t *testing.T) { |
| + b := &ExecBackend{DataDir: t.TempDir()} |
| + if tc.bucket != nil { |
| + b.ConfiguredKey = tc.bucket.read |
| + } |
| + rt := registry.Route{Host: "shop.example.com", Kind: "instance"} |
| + own := filepath.Join(b.DataDir, rt.Host+".env") |
| + if err := os.WriteFile(own, []byte(tc.own), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if tc.config != "" { |
| + if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(tc.config), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + for _, call := range []string{"launch", "backfill"} { |
| + var got InstanceKeyOutcome |
| + var err error |
| + if call == "launch" { |
| + got, err = b.ensureInstanceKeyFor(t.Context(), rt, b.pathsFor(rt)) |
| + } else { |
| + if err := os.WriteFile(own, []byte(tc.own), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if tc.bucket != nil { |
| + tc.bucket.reads = 0 |
| + } |
| + got, err = b.BackfillInstanceKey(t.Context(), rt) |
| + } |
| + if got != tc.want || errors.Is(err, ErrInstanceKeyDeferred) != tc.deferred || (err != nil) != tc.deferred { |
| + t.Fatalf("%s = %v, %v; want %v, deferred %v", call, got, err, tc.want, tc.deferred) |
| + } |
| + raw, _ := os.ReadFile(own) |
| + if minted := len(keyLines(raw)) > len(keyLines([]byte(tc.own))); minted != (tc.want == InstanceKeyMinted) { |
| + t.Fatalf("%s: a key line was appended: %v, want %v", call, minted, tc.want == InstanceKeyMinted) |
| + } |
| + if tc.bucket != nil && tc.bucket.reads != tc.reads { |
| + t.Fatalf("%s: the bucket was read %d times, want %d", call, tc.bucket.reads, tc.reads) |
| + } |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// The mint's bucket read is bounded by configReadTimeout and by the |
| +// launch's own context, whichever ends first. |
| +func TestEnsureInstanceKeyBoundsTheBucketRead(t *testing.T) { |
| + b := &ExecBackend{DataDir: t.TempDir()} |
| + var deadline time.Time |
| + b.ConfiguredKey = func(ctx context.Context, _ registry.Route) (string, error) { |
| + deadline, _ = ctx.Deadline() |
| + return "", ctx.Err() |
| + } |
| + rt := registry.Route{Host: "shop.example.com", Kind: "instance"} |
| + if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("A=1\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := b.ensureInstanceKeyFor(context.Background(), rt, b.pathsFor(rt)); err != nil { |
| + t.Fatal(err) |
| + } |
| + if until := time.Until(deadline); until <= 0 || until > configReadTimeout { |
| + t.Fatalf("the read's deadline is %v away, want within %v", until, configReadTimeout) |
| + } |
| + if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("A=1\n"), 0o600); err != nil { |
| + t.Fatal(err) // keyless again, so the backfill reaches the read |
| + } |
| + ctx, cancel := context.WithCancel(context.Background()) |
| + cancel() |
| + if _, err := b.BackfillInstanceKey(ctx, rt); !errors.Is(err, ErrInstanceKeyDeferred) || !errors.Is(err, context.Canceled) { |
| + t.Fatalf("a cancelled read = %v, want deferred by the cancellation", err) |
| + } |
| +} |
| + |
| +// A first start whose configuration is still on its way, end to end: the |
| +// start is deferred and no app runs; nothing is minted into <host>.env; |
| +// once the config sync has delivered <host>.config.env with the configured |
| +// key, the start runs and the app seals with that key, the configured one, |
| +// never a minted one. |
| +func TestFirstStartWaitsForTheConfiguredKey(t *testing.T) { |
| + b := testExecBackend(t) |
| + bucket := &bucketKey{key: "key-A"} |
| + b.ConfiguredKey = bucket.read |
| + rt := registry.Route{Host: "shop.example.com", Kind: "instance", Socket: noControlSocket(t)} |
| + own := filepath.Join(b.DataDir, rt.Host+".env") |
| + tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n" |
| + if err := os.WriteFile(own, []byte(tok), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + out := stageHelperApp(t, b, rt.Host, "v1") |
| + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| + defer cancel() |
| + proc, _, err := b.Start(ctx, rt) |
| + if proc != nil { |
| + proc.Stop(context.Background()) |
| + } |
| + if !errors.Is(err, ErrInstanceKeyDeferred) || !strings.Contains(err.Error(), rt.Host) { |
| + t.Fatalf("Start before delivery = %v, want ErrInstanceKeyDeferred naming the host", err) |
| + } |
| + if got := outcomeOf(out); got != "" { |
| + t.Fatalf("an app ran before delivery: %q", got) |
| + } |
| + if raw, _ := os.ReadFile(own); string(raw) != tok { |
| + t.Fatalf("the own file changed before delivery: %d bytes", len(raw)) |
| + } |
| + // The config sync delivers the file. |
| + if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(InstanceKeyVar+"=key-A\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The helper exits once it has recorded its outcome, so Start's health |
| + // wait never sees a live socket: the outcome is the proof, as in the |
| + // replica tests, and the error only must not be the deferral. |
| + ctx2, cancel2 := context.WithTimeout(context.Background(), 2*time.Second) // ctx was spent waiting for the first outcome |
| + defer cancel2() |
| + proc, _, err = b.Start(ctx2, rt) |
| + if proc != nil { |
| + defer proc.Stop(context.Background()) |
| + } |
| + if errors.Is(err, ErrInstanceKeyDeferred) { |
| + t.Fatalf("Start after delivery still deferred: %v", err) |
| + } |
| + if got := outcomeOf(out); got != "key "+instancekey.Digest("key-A") { |
| + t.Fatalf("the app's outcome = %q, want the configured key", got) |
| + } |
| + if raw, _ := os.ReadFile(own); string(raw) != tok { |
| + t.Fatal("a key was minted beside the configured one") |
| + } |
| +} |
diff --git a/internal/activator/instancekey_linux_test.go b/internal/activator/instancekey_linux_test.go| index 095cdac08..4158427df 100644 |
| --- a/internal/activator/instancekey_linux_test.go |
| +++ b/internal/activator/instancekey_linux_test.go |
| @@ -55,7 +55,7 @@ func TestBackfillInstanceKeyAppendsToANestedEnvFile(t *testing.T) { |
| if err := os.WriteFile(envPath, []byte(tok), 0o644); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyMinted { |
| + if got, err := b.BackfillInstanceKey(t.Context(), rt); err != nil || got != InstanceKeyMinted { |
| t.Fatalf("nested backfill = %v, %v; want minted", got, err) |
| } |
| raw, _ := os.ReadFile(envPath) |
| @@ -65,7 +65,7 @@ func TestBackfillInstanceKeyAppendsToANestedEnvFile(t *testing.T) { |
| if st, _ := os.Stat(envPath); st.Mode().Perm() != 0o600 { |
| t.Fatalf("mode = %o, want 600", st.Mode().Perm()) |
| } |
| - if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyPresent { |
| + if got, err := b.BackfillInstanceKey(t.Context(), rt); err != nil || got != InstanceKeyPresent { |
| t.Fatalf("second nested backfill = %v, %v; want present", got, err) |
| } |
| if raw2, _ := os.ReadFile(envPath); string(raw2) != string(raw) { |
| @@ -84,7 +84,7 @@ func TestEnsureInstanceKeyBeneathRefusesASymlink(t *testing.T) { |
| if err := os.Symlink(target, filepath.Join(dir, "shop.example.com.env")); err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := EnsureInstanceKeyBeneath(dir, "shop.example.com.env"); err == nil { |
| + if _, err := ensureInstanceKeyBeneathNoConfig(dir, "shop.example.com.env"); err == nil { |
| t.Fatal("ensure followed a symlink planted at the env file's name") |
| } |
| if raw, _ := os.ReadFile(target); string(raw) != "X=1\n" { |
| @@ -137,7 +137,7 @@ func TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking(t *testing.T) { |
| } |
| done := make(chan error, 1) |
| go func() { |
| - _, err := b.BackfillInstanceKey(rt) |
| + _, err := b.BackfillInstanceKey(t.Context(), rt) |
| done <- err |
| }() |
| select { |
diff --git a/internal/activator/instancekey_test.go b/internal/activator/instancekey_test.go| index aab0d36a0..18bafbfb8 100644 |
| --- a/internal/activator/instancekey_test.go |
| +++ b/internal/activator/instancekey_test.go |
| @@ -67,7 +67,7 @@ func probeInstanceKeySync(t *testing.T) *syncProbe { |
| func TestEnsureInstanceKeyMintsOnceInTheAgreedFormat(t *testing.T) { |
| p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| - got, err := EnsureInstanceKey(p) |
| + got, err := ensureInstanceKeyNoConfig(p) |
| if err != nil || got != InstanceKeyMinted { |
| t.Fatalf("first ensure = %v, %v; want minted", got, err) |
| } |
| @@ -91,7 +91,7 @@ func TestEnsureInstanceKeyMintsOnceInTheAgreedFormat(t *testing.T) { |
| if env[InstanceKeyVar] != keys[0] { |
| t.Fatal("readEnvFile does not yield the minted key verbatim") |
| } |
| - again, err := EnsureInstanceKey(p) |
| + again, err := ensureInstanceKeyNoConfig(p) |
| if err != nil || again != InstanceKeyPresent { |
| t.Fatalf("second ensure = %v, %v; want present", again, err) |
| } |
| @@ -114,7 +114,7 @@ func TestEnsureInstanceKeyNeverRewritesAnExistingKey(t *testing.T) { |
| } |
| var buf bytes.Buffer |
| log.SetOutput(&buf) |
| - got, err := EnsureInstanceKey(p) |
| + got, err := ensureInstanceKeyNoConfig(p) |
| log.SetOutput(os.Stderr) |
| if err != nil || got != InstanceKeyPresent { |
| t.Fatalf("ensure over %q = %v, %v; want present", existing, got, err) |
| @@ -134,7 +134,7 @@ func TestEnsureInstanceKeyAppendsOnAFreshLineAndKeepsEveryOtherLine(t *testing.T |
| if err := os.WriteFile(p, []byte(before), 0o644); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted { |
| + if got, err := ensureInstanceKeyNoConfig(p); err != nil || got != InstanceKeyMinted { |
| t.Fatalf("ensure = %v, %v", got, err) |
| } |
| raw, _ := os.ReadFile(p) |
| @@ -164,7 +164,7 @@ func TestEnsureInstanceKeyDefersToTheRenderedConfiguration(t *testing.T) { |
| if err := os.WriteFile(p, []byte(own), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - got, err := EnsureInstanceKey(p) |
| + got, err := ensureInstanceKeyNoConfig(p) |
| if err != nil || got != InstanceKeyFromConfig { |
| t.Fatalf("own %q: ensure = %v, %v; want from-config", own, got, err) |
| } |
| @@ -177,7 +177,7 @@ func TestEnsureInstanceKeyDefersToTheRenderedConfiguration(t *testing.T) { |
| func TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile(t *testing.T) { |
| b := &ExecBackend{DataDir: t.TempDir()} |
| fresh := registry.Route{Host: "new.example.com", Socket: "/run/carlos/new.sock"} |
| - got, err := b.BackfillInstanceKey(fresh) |
| + got, err := b.BackfillInstanceKey(t.Context(), fresh) |
| if err != nil || got != InstanceKeyNoEnvFile { |
| t.Fatalf("backfill with no env file = %v, %v; want no-env-file", got, err) |
| } |
| @@ -190,7 +190,7 @@ func TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile(t *testing.T) { |
| if err := os.WriteFile(envPath, []byte(tok), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyMinted { |
| + if got, err := b.BackfillInstanceKey(t.Context(), old); err != nil || got != InstanceKeyMinted { |
| t.Fatalf("backfill = %v, %v; want minted", got, err) |
| } |
| raw, _ := os.ReadFile(envPath) |
| @@ -198,7 +198,7 @@ func TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile(t *testing.T) { |
| t.Fatalf("file = %q", raw) |
| } |
| // Idempotent: a second backfill (the next agent start) writes nothing. |
| - if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyPresent { |
| + if got, err := b.BackfillInstanceKey(t.Context(), old); err != nil || got != InstanceKeyPresent { |
| t.Fatalf("second backfill = %v, %v; want present", got, err) |
| } |
| if raw2, _ := os.ReadFile(envPath); !bytes.Equal(raw, raw2) { |
| @@ -214,7 +214,7 @@ func TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone(t *testing.T) { |
| if err := os.WriteFile(p, []byte(body), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyPresent { |
| + if got, err := ensureInstanceKeyNoConfig(p); err != nil || got != InstanceKeyPresent { |
| t.Fatalf("ensure = %v, %v; want present", got, err) |
| } |
| if raw, _ := os.ReadFile(p); string(raw) != body { |
| @@ -232,7 +232,7 @@ func TestEnsureInstanceKeyTreatsAnEmptyValueAsAbsent(t *testing.T) { |
| if err := os.WriteFile(p, []byte(InstanceKeyVar+"=\n"), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted { |
| + if got, err := ensureInstanceKeyNoConfig(p); err != nil || got != InstanceKeyMinted { |
| t.Fatalf("ensure = %v, %v; want minted", got, err) |
| } |
| env, _ := readEnvFile(p) |
| @@ -250,7 +250,7 @@ func TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers(t *testing.T) { |
| wg.Add(1) |
| go func() { |
| defer wg.Done() |
| - if _, err := EnsureInstanceKey(p); err != nil { |
| + if _, err := ensureInstanceKeyNoConfig(p); err != nil { |
| t.Error(err) |
| } |
| }() |
| @@ -268,7 +268,7 @@ func TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty(t *testing.T) { |
| if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=\n"), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := EnsureInstanceKey(filepath.Join(dir, "shop.example.com.env")); err != nil || got != InstanceKeyMinted { |
| + if got, err := ensureInstanceKeyNoConfig(filepath.Join(dir, "shop.example.com.env")); err != nil || got != InstanceKeyMinted { |
| t.Fatalf("ensure = %v, %v; want minted", got, err) |
| } |
| } |
| @@ -289,14 +289,14 @@ func TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess(t *testing.T) |
| probe := probeInstanceKeySync(t) |
| p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| tc.fail(probe, errors.New("injected: EIO")) |
| - if _, err := EnsureInstanceKey(p); err == nil { |
| + if _, err := ensureInstanceKeyNoConfig(p); err == nil { |
| t.Fatal("a mint whose sync failed reported success") |
| } |
| raw, _ := os.ReadFile(p) |
| if len(keyLines(raw)) != 1 { |
| t.Fatalf("the failed mint left %d keys, want its 1", len(keyLines(raw))) |
| } |
| - if _, err := EnsureInstanceKey(p); err == nil { |
| + if _, err := ensureInstanceKeyNoConfig(p); err == nil { |
| t.Fatal("a retry reported success while the sync still fails") |
| } |
| if err := os.Chmod(p, 0o644); err != nil { |
| @@ -304,7 +304,7 @@ func TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess(t *testing.T) |
| } |
| tc.fail(probe, nil) |
| before := *probe |
| - got, err := EnsureInstanceKey(p) |
| + got, err := ensureInstanceKeyNoConfig(p) |
| if err != nil || got != InstanceKeyPresent { |
| t.Fatalf("retry = %v, %v; want present", got, err) |
| } |
| @@ -330,10 +330,10 @@ func TestReplicaRowsGetNoPlatformKey(t *testing.T) { |
| if err := os.WriteFile(envPath, []byte("CARLOS_ADMIN_TOKEN=t\n"), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - if got, err := b.ensureInstanceKeyFor(rt, b.pathsFor(rt)); err != nil || got != InstanceKeyReplica { |
| + if got, err := b.ensureInstanceKeyFor(t.Context(), rt, b.pathsFor(rt)); err != nil || got != InstanceKeyReplica { |
| t.Fatalf("launch ensure = %v, %v; want replica", got, err) |
| } |
| - if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyReplica { |
| + if got, err := b.BackfillInstanceKey(t.Context(), rt); err != nil || got != InstanceKeyReplica { |
| t.Fatalf("backfill = %v, %v; want replica", got, err) |
| } |
| if raw, _ := os.ReadFile(envPath); len(keyLines(raw)) != 0 { |
| @@ -354,19 +354,19 @@ func TestEnsureInstanceKeyRefusesToFollowASymlink(t *testing.T) { |
| call func(b *ExecBackend, rt registry.Route, link string) error |
| }{ |
| {"mint", "A=1\n", 0o644, func(_ *ExecBackend, _ registry.Route, link string) error { |
| - _, err := EnsureInstanceKey(link) |
| + _, err := ensureInstanceKeyNoConfig(link) |
| return err |
| }}, |
| {"existing key", wellFormed, 0o644, func(_ *ExecBackend, _ registry.Route, link string) error { |
| - _, err := EnsureInstanceKey(link) |
| + _, err := ensureInstanceKeyNoConfig(link) |
| return err |
| }}, |
| {"backfill mint", "A=1\n", 0o644, func(b *ExecBackend, rt registry.Route, _ string) error { |
| - _, err := b.BackfillInstanceKey(rt) |
| + _, err := b.BackfillInstanceKey(t.Context(), rt) |
| return err |
| }}, |
| {"backfill existing key", wellFormed, 0o644, func(b *ExecBackend, rt registry.Route, _ string) error { |
| - _, err := b.BackfillInstanceKey(rt) |
| + _, err := b.BackfillInstanceKey(t.Context(), rt) |
| return err |
| }}, |
| {"claim settle", wellFormed, 0o600, func(b *ExecBackend, rt registry.Route, _ string) error { |
| @@ -417,7 +417,7 @@ func TestEnsureInstanceKeyRefusesADanglingSymlink(t *testing.T) { |
| if err := os.Symlink(target, link); err != nil { |
| t.Skipf("this filesystem will not make symlinks: %v", err) |
| } |
| - if _, err := EnsureInstanceKey(link); !errors.Is(err, errOwnEnvSymlink) { |
| + if _, err := ensureInstanceKeyNoConfig(link); !errors.Is(err, errOwnEnvSymlink) { |
| t.Fatalf("err = %v, want errOwnEnvSymlink", err) |
| } |
| if _, err := os.Lstat(target); !os.IsNotExist(err) { |
| @@ -516,7 +516,7 @@ func TestEnsureInstanceKeyUndoesAPartialAppend(t *testing.T) { |
| } |
| } |
| setInstanceKeySeams(t, tc.write, nil) |
| - if _, err := EnsureInstanceKey(p); err == nil { |
| + if _, err := ensureInstanceKeyNoConfig(p); err == nil { |
| t.Fatal("a mint whose append failed reported success") |
| } |
| raw, err := os.ReadFile(p) |
| @@ -534,7 +534,7 @@ func TestEnsureInstanceKeyUndoesAPartialAppend(t *testing.T) { |
| t.Fatalf("the undo synced the file %d times, want once", probe.file) |
| } |
| setInstanceKeySeams(t, func(f *os.File, b []byte) (int, error) { return f.Write(b) }, nil) |
| - got, err := EnsureInstanceKey(p) |
| + got, err := ensureInstanceKeyNoConfig(p) |
| if err != nil || got != InstanceKeyMinted { |
| t.Fatalf("next ensure = %v, %v; want minted", got, err) |
| } |
| @@ -564,15 +564,15 @@ func TestEnsureInstanceKeyOpenRefusesALinkSwappedInAfterTheRead(t *testing.T) { |
| call func(b *ExecBackend, rt registry.Route, path string) error |
| }{ |
| {"mint", "A=1\n", func(_ *ExecBackend, _ registry.Route, path string) error { |
| - _, err := EnsureInstanceKey(path) |
| + _, err := ensureInstanceKeyNoConfig(path) |
| return err |
| }}, |
| {"existing key", wellFormed, func(_ *ExecBackend, _ registry.Route, path string) error { |
| - _, err := EnsureInstanceKey(path) |
| + _, err := ensureInstanceKeyNoConfig(path) |
| return err |
| }}, |
| {"backfill mint", "A=1\n", func(b *ExecBackend, rt registry.Route, _ string) error { |
| - _, err := b.BackfillInstanceKey(rt) |
| + _, err := b.BackfillInstanceKey(t.Context(), rt) |
| return err |
| }}, |
| } { |
| @@ -636,7 +636,7 @@ func TestEnsureInstanceKeySymlinkErrorNamesThePathOnce(t *testing.T) { |
| if err := os.Symlink(target, link); err != nil { |
| t.Skipf("this filesystem will not make symlinks: %v", err) |
| } |
| - _, err := EnsureInstanceKey(link) |
| + _, err := ensureInstanceKeyNoConfig(link) |
| if !errors.Is(err, errOwnEnvSymlink) { |
| t.Fatalf("err = %v, want errOwnEnvSymlink", err) |
| } |
| @@ -957,7 +957,7 @@ func TestEnsureInstanceKeyForRefusesFencedAndConflictRows(t *testing.T) { |
| if p.layout != layout { |
| t.Fatalf("paths classify as %s, want %s", p.layout, layout) |
| } |
| - if _, err := b.ensureInstanceKeyFor(rt, p); err == nil || !strings.Contains(err.Error(), "not minted") { |
| + if _, err := b.ensureInstanceKeyFor(t.Context(), rt, p); err == nil || !strings.Contains(err.Error(), "not minted") { |
| t.Fatalf("ensure for a %s row: err = %v, want a refusal", layout, err) |
| } else if strings.HasPrefix(err.Error(), rt.Host) { |
| t.Fatalf("the refusal names the host itself (%v); its callers do", err) |
| @@ -991,7 +991,7 @@ func TestBackfillInstanceKeyRefusesFencedAndConflictRows(t *testing.T) { |
| } |
| refuseAfterFirstClassification(b, layout) |
| _, _ = b.Layout(rt) |
| - if _, err := b.BackfillInstanceKey(rt); err == nil || !strings.Contains(err.Error(), "not backfilled") { |
| + if _, err := b.BackfillInstanceKey(t.Context(), rt); err == nil || !strings.Contains(err.Error(), "not backfilled") { |
| t.Fatalf("backfill of a %s row: err = %v, want a refusal", layout, err) |
| } else if strings.HasPrefix(err.Error(), host) { |
| t.Fatalf("the refusal names the host itself (%v); the agent's backfill does", err) |
diff --git a/internal/activator/prepare.go b/internal/activator/prepare.go| index 09ca41981..ef63d3349 100644 |
| --- a/internal/activator/prepare.go |
| +++ b/internal/activator/prepare.go |
| @@ -6,6 +6,7 @@ |
| package activator |
| import ( |
| + "context" |
| "encoding/json" |
| "errors" |
| "fmt" |
| @@ -62,7 +63,7 @@ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock s |
| if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil { |
| return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err) |
| } |
| - if _, err := b.ensureInstanceKeyFor(rt, p); err != nil { |
| + if _, err := b.ensureInstanceKeyFor(context.Background(), rt, p); err != nil { // prepareAt has no ctx to bound the mint's bucket read; configReadTimeout does |
| return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err) |
| } |
| var err error |
diff --git a/internal/activator/replica_instancekey_linux_test.go b/internal/activator/replica_instancekey_linux_test.go| index d9f8bf65e..66c5cfe36 100644 |
| --- a/internal/activator/replica_instancekey_linux_test.go |
| +++ b/internal/activator/replica_instancekey_linux_test.go |
| @@ -77,7 +77,7 @@ func TestNestedReplicaPrepareIsGated(t *testing.T) { |
| b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| rt.Replica = true |
| b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { |
| if tc.bucket == "" { |
| return "", nil |
| } |
| @@ -130,7 +130,7 @@ func TestNestedReplicaSidecarPrepareIsGated(t *testing.T) { |
| b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| rt.Replica = true |
| b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest("configured-B"), nil } |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest("configured-B"), nil } |
| dir := provision.InstanceDir(b.DataDir, rt.Host) |
| if err := os.MkdirAll(dir, 0o700); err != nil { |
| t.Fatal(err) |
| @@ -185,7 +185,7 @@ func TestReplicaRefusalClosesTheControlListener(t *testing.T) { |
| if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(InstanceKeyVar+"=configured-B\n"), 0o600); err != nil { |
| t.Fatal(err) |
| } |
| - b.ReplicaHomeKeys, b.ReplicaConfigKey = homes, bucketB |
| + b.ReplicaHomeKeys, b.ConfiguredKey = homes, bucketB |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| defer cancel() |
| @@ -201,7 +201,7 @@ func TestReplicaRefusalClosesTheControlListener(t *testing.T) { |
| t.Run("nested run role", func(t *testing.T) { |
| b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| rt.Replica = true |
| - b.ReplicaHomeKeys, b.ReplicaConfigKey = homes, bucketB |
| + b.ReplicaHomeKeys, b.ConfiguredKey = homes, bucketB |
| dir := provision.InstanceDir(b.DataDir, rt.Host) |
| if err := os.MkdirAll(dir, 0o700); err != nil { |
| t.Fatal(err) |
diff --git a/internal/activator/replica_instancekey_test.go b/internal/activator/replica_instancekey_test.go| index 21f28fb1e..41b4023f3 100644 |
| --- a/internal/activator/replica_instancekey_test.go |
| +++ b/internal/activator/replica_instancekey_test.go |
| @@ -123,13 +123,13 @@ type replicaCase struct { |
| start string // "refused" (no app runs) or the helper outcome wanted |
| wantKeyFromOwnOK bool // a home row: the outcome is the key minted into <host>.env |
| // bucket is the key the replica's configuration in the bucket renders |
| - // now (ReplicaConfigKey; "" renders none). bucketDown makes that read |
| - // fail; noBucket leaves ReplicaConfigKey nil. |
| + // now (ConfiguredKey; "" renders none). bucketDown makes that read |
| + // fail; noBucket leaves ConfiguredKey nil. |
| bucket string |
| bucketDown, noBucket bool |
| } |
| -// freshConfig is ExecBackend.ReplicaConfigKey for one case. |
| +// freshConfig is ExecBackend.ConfiguredKey for one case. |
| func freshConfig(tc replicaCase) func(context.Context, registry.Route) (string, error) { |
| if tc.noBucket { |
| return nil |
| @@ -196,7 +196,7 @@ func TestReplicaLaunchWithoutAControlSocket(t *testing.T) { |
| return instancekey.Digest(tc.homeKey), h == rt.Host |
| } |
| } |
| - b.ReplicaConfigKey = freshConfig(tc) |
| + b.ConfiguredKey = freshConfig(tc) |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| defer cancel() |
| @@ -250,7 +250,7 @@ func TestReplicaSeedIsGated(t *testing.T) { |
| } |
| } |
| b.ReplicaHomeKeys = func(h string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - b.ReplicaConfigKey = freshConfig(replicaCase{bucket: homeKey}) |
| + b.ConfiguredKey = freshConfig(replicaCase{bucket: homeKey}) |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| err := b.Seed(context.Background(), rt) |
| got := outcomeOf(out) |
| @@ -307,7 +307,7 @@ func TestReplicaWorkerIsGatedLikeASeed(t *testing.T) { |
| } |
| if tc.replica { |
| b.ReplicaHomeKeys = func(h string) (string, bool) { return instancekey.Digest(homeKey), h == rt.Host } |
| - b.ReplicaConfigKey = freshConfig(replicaCase{bucket: tc.bucket}) |
| + b.ConfiguredKey = freshConfig(replicaCase{bucket: tc.bucket}) |
| } |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| extra := map[string]string{"CARLOS_SIDECAR_NAME": "bot"} |
| @@ -362,7 +362,7 @@ func TestReplicaLegacySidecarIsGatedLikeTheWorker(t *testing.T) { |
| b.ReplicaHomeKeys = func(h string) (string, bool) { return instancekey.Digest(homeKey), h == rt.Host } |
| // The bucket agrees with whatever the case holds, so only the |
| // home-key check decides. |
| - b.ReplicaConfigKey = freshConfig(replicaCase{bucket: "configured-B"}) |
| + b.ConfiguredKey = freshConfig(replicaCase{bucket: "configured-B"}) |
| } |
| out := stageHelperApp(t, b, rt.Host, "v1") |
| proc, err := b.StartSidecar(context.Background(), rt) |
| @@ -407,7 +407,7 @@ func TestReplicaGateNamesWhyItRefused(t *testing.T) { |
| }, "the bucket's configuration could not be read: bucket unreachable"}, |
| } { |
| t.Run(tc.name, func(t *testing.T) { |
| - b := &ExecBackend{DataDir: t.TempDir(), ReplicaHomeKeys: tc.homes, ReplicaConfigKey: tc.bucket} |
| + b := &ExecBackend{DataDir: t.TempDir(), ReplicaHomeKeys: tc.homes, ConfiguredKey: tc.bucket} |
| err := b.replicaKeyGate(rt, env, instancekey.LayerConfig, false) |
| if !errors.Is(err, ErrReplicaKey) || !strings.Contains(err.Error(), tc.want) { |
| t.Fatalf("gate = %v; want ErrReplicaKey naming %q", err, tc.want) |
diff --git a/internal/activator/replicakey.go b/internal/activator/replicakey.go| index d171798b4..515cd8f18 100644 |
| --- a/internal/activator/replicakey.go |
| +++ b/internal/activator/replicakey.go |
| @@ -14,7 +14,7 @@ package activator |
| // digest is its entry in the box's verified list (ReplicaHomeKeys, which |
| // only the pre-roll check writes, read again at every launch) and, when |
| // the key comes from the replica's configuration, the configuration in |
| -// the bucket renders that same key now (ReplicaConfigKey). Where the key |
| +// the bucket renders that same key now (ConfiguredKey). Where the key |
| // came from is decided by the same read that composed the env the child |
| // gets (instanceEnvLayer, ruling 29), never by reading the files again. |
| // A key that differs from either, or that cannot be checked, refuses |
| @@ -40,7 +40,6 @@ import ( |
| "errors" |
| "fmt" |
| "log" |
| - "time" |
| "github.com/carlosframework/platform/internal/instancekey" |
| "github.com/carlosframework/platform/internal/registry" |
| @@ -49,9 +48,6 @@ import ( |
| // ErrReplicaKey is a replica launch refused by replicaKeyGate. |
| var ErrReplicaKey = errors.New("activator: replica launch refused: its instance key is not its home's") |
| -// replicaConfigTimeout bounds replicaKeyGate's fresh read of the bucket. |
| -const replicaConfigTimeout = 10 * time.Second |
| - |
| // instanceEnvLayer is instanceEnv returning, from the same read, the layer |
| // its CARLOS_INSTANCE_KEY came from (ruling 29). mergeEnv delivers exactly |
| // the key instancekey.Effective names, so env[Var] and the layer always |
| @@ -124,12 +120,11 @@ func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, l |
| default: |
| return refuse("where its " + instancekey.Var + " came from is unknown") |
| } |
| - if b.ReplicaConfigKey == nil { |
| + configured := b.configuredKeyFor(rt) |
| + if configured == nil { |
| return refuse("its configured " + instancekey.Var + " cannot be checked against the bucket's configuration") |
| } |
| - ctx, cancel := context.WithTimeout(context.Background(), replicaConfigTimeout) |
| - defer cancel() |
| - fresh, err := b.ReplicaConfigKey(ctx, rt) |
| + fresh, err := configured(context.Background()) |
| switch { |
| case err != nil: |
| return refuse("the bucket's configuration could not be read: " + err.Error()) |
diff --git a/internal/activator/replicakey_race_test.go b/internal/activator/replicakey_race_test.go| index 931666206..b7dfd5bb4 100644 |
| --- a/internal/activator/replicakey_race_test.go |
| +++ b/internal/activator/replicakey_race_test.go |
| @@ -33,7 +33,7 @@ func TestReplicaGateDecidesOnTheLaunchesOwnRead(t *testing.T) { |
| t.Fatal(err) |
| } |
| b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true } |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed |
| syncs := 0 |
| replicaGateHook = func() { |
| syncs++ |
| @@ -78,7 +78,7 @@ func TestReplicaGateReadsNoEnvFile(t *testing.T) { |
| bucket string |
| ok bool |
| }{{keyA, true}, {"", false}} { |
| - b.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { |
| + b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { |
| if tc.bucket == "" { |
| return "", nil |
| } |
diff --git a/internal/host/agent.go b/internal/host/agent.go| index 9ab848b86..bf3211f1e 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -366,7 +366,7 @@ func runAgent(args []string, buildClaims ClaimFactory) { |
| eb.ReplicaHomeKeys = replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(replicaHomeKeysPath) }) |
| // The bucket's configuration, read fresh at each replica launch |
| // whose key is configured (ruling 27). |
| - eb.ReplicaConfigKey = replicaConfigKey(os.Getenv, openDeployStore) |
| + eb.ConfiguredKey = configuredKey(os.Getenv, openDeployStore) |
| // The root instance-control channel (design §5.3) and the box facts |
| // a nested wake needs beyond the row (design §8.1, Task 6): the |
| // journal check and the peer list both read this SAME registry |
diff --git a/internal/host/claim_instancekey_test.go b/internal/host/claim_instancekey_test.go| index 71c46b918..4674ecc3f 100644 |
| --- a/internal/host/claim_instancekey_test.go |
| +++ b/internal/host/claim_instancekey_test.go |
| @@ -2,6 +2,7 @@ package host |
| import ( |
| "context" |
| + "errors" |
| "os" |
| "path/filepath" |
| "slices" |
| @@ -14,6 +15,7 @@ import ( |
| "github.com/carlosframework/platform/internal/hostwrite" |
| "github.com/carlosframework/platform/internal/instancekey" |
| "github.com/carlosframework/platform/internal/lease/leasetest" |
| + "github.com/carlosframework/platform/internal/registry" |
| ) |
| // keyCapturingFiles is a C8 provider that records the instance's own env |
| @@ -78,6 +80,36 @@ func TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest(t *testing.T) { |
| } |
| } |
| +// Final review I2 on the claim path: a first wake whose configuration is |
| +// still on its way is deferred by the launch source and nothing is minted; |
| +// once the config sync has delivered the configured key, the launch |
| +// delivers that key and the own file still holds none. |
| +func TestClaimFirstWakeWaitsForTheConfiguredKey(t *testing.T) { |
| + f := newAdminTokenFixture(t, "box-1") |
| + f.backend.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest("key-A"), nil } |
| + launch := claimServingLaunchSource(f.backend, f.tokens) |
| + if _, err := launch(t.Context(), f.rt); !errors.Is(err, activator.ErrInstanceKeyDeferred) || !strings.Contains(err.Error(), adminTokenHost) { |
| + t.Fatalf("launch before delivery = %v, want ErrInstanceKeyDeferred naming the host", err) |
| + } |
| + raw, _ := os.ReadFile(f.envPath()) |
| + if strings.Contains(string(raw), activator.InstanceKeyVar) { |
| + t.Fatal("a key was minted while the configured one was on its way") |
| + } |
| + if err := os.WriteFile(filepath.Join(f.data, adminTokenHost+".config.env"), []byte(activator.InstanceKeyVar+"=key-A\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := launch(t.Context(), f.rt) |
| + if err != nil { |
| + t.Fatalf("launch after delivery: %v", err) |
| + } |
| + if !slices.Contains(got.Env, activator.InstanceKeyVar+"=key-A") { |
| + t.Fatal("the launch does not deliver the configured key") |
| + } |
| + if raw, _ := os.ReadFile(f.envPath()); strings.Contains(string(raw), activator.InstanceKeyVar) { |
| + t.Fatal("a key was minted beside the configured one") |
| + } |
| +} |
| + |
| // An instance from before this build has a token and no key. Its next |
| // claim launch mints the key under the same guarded write, leaves the |
| // token alone, and a retry keeps both. |
diff --git a/internal/host/claim_replica_instancekey_test.go b/internal/host/claim_replica_instancekey_test.go| index f3178c053..aa5bb0ff3 100644 |
| --- a/internal/host/claim_replica_instancekey_test.go |
| +++ b/internal/host/claim_replica_instancekey_test.go |
| @@ -30,7 +30,7 @@ func TestClaimReplicaLaunchIsGated(t *testing.T) { |
| f := newAdminTokenFixture(t, "box-1") |
| f.rt.Replica = true |
| f.backend.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true } |
| - f.backend.ReplicaConfigKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| + f.backend.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil } |
| own := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n" |
| if err := os.WriteFile(f.envPath(), []byte(own), 0o600); err != nil { |
| t.Fatal(err) |
diff --git a/internal/host/configguard.go b/internal/host/configguard.go| index 8c4e8fee3..742e0a748 100644 |
| --- a/internal/host/configguard.go |
| +++ b/internal/host/configguard.go |
| @@ -106,10 +106,7 @@ func newConfigGuard(getenv func(string) string, openStore func(context.Context) |
| verdicts: map[string]guardVerdict{}, |
| refused: map[string]*guardRefusal{}, |
| } |
| - switch strings.ToLower(strings.TrimSpace(getenv("CARLOS_NO_SEALED_CONFIG"))) { |
| - case "1", "true": |
| - g.noSealedConfig = true |
| - } |
| + g.noSealedConfig = noSealedConfigFlag(getenv) |
| raw := strings.TrimSpace(getenv("CARLOS_SECRETS_KEY")) |
| switch { |
| case raw == "": |
| @@ -126,6 +123,16 @@ func newConfigGuard(getenv func(string) string, openStore func(context.Context) |
| return g |
| } |
| +// noSealedConfigFlag is the operator's CARLOS_NO_SEALED_CONFIG, read the |
| +// one way the guard and the instance key's configuration reader share. |
| +func noSealedConfigFlag(getenv func(string) string) bool { |
| + switch strings.ToLower(strings.TrimSpace(getenv("CARLOS_NO_SEALED_CONFIG"))) { |
| + case "1", "true": |
| + return true |
| + } |
| + return false |
| +} |
| + |
| // needsBoxConfig reports whether a route is one this box would spawn with a |
| // config.env of its own: a socket-backed instance or a sidecar worker that |
| // belongs to an app. Static sites, addr-backed routes, Backing=unit routes |
diff --git a/internal/host/instancekeys.go b/internal/host/instancekeys.go| index e68775afa..d79ea90f4 100644 |
| --- a/internal/host/instancekeys.go |
| +++ b/internal/host/instancekeys.go |
| @@ -1,7 +1,10 @@ |
| package host |
| import ( |
| + "context" |
| + "errors" |
| "log" |
| + "time" |
| "github.com/carlosframework/platform/internal/activator" |
| "github.com/carlosframework/platform/internal/registry" |
| @@ -20,11 +23,20 @@ import ( |
| // that guard before every cold admission (claim_snapshot.go |
| // EnsureClaimAdminToken), so a claim box is not backfilled here. |
| -// instanceKeyBackfiller is the activator surface the backfill needs. |
| +// instanceKeyBackfiller is the activator surface the backfill needs. The |
| +// context bounds the one bucket read a mint makes (ruling 5: nothing is |
| +// minted until the row's configuration is delivered). |
| type instanceKeyBackfiller interface { |
| - BackfillInstanceKey(registry.Route) (activator.InstanceKeyOutcome, error) |
| + BackfillInstanceKey(context.Context, registry.Route) (activator.InstanceKeyOutcome, error) |
| } |
| +// instanceKeyBackfillBudget bounds one pass. Each keyless row costs one |
| +// read of the bucket's configuration before its key is minted, and the |
| +// pass runs before the activator starts, so with the bucket unreachable |
| +// the rows after the budget fail fast rather than each waiting out its |
| +// own read; every one of them mints at its next start. |
| +const instanceKeyBackfillBudget = time.Minute |
| + |
| // instanceKeyRoute selects the rows the backfill writes for: ready, |
| // activator-spawned instances, the rows execSyncable delivers config to, |
| // without sidecars and without replicas (a replica is never minted a key, |
| @@ -37,28 +49,39 @@ func instanceKeyRoute(rt registry.Route) bool { |
| rt.Backing != registry.BackingUnit && rt.Tunnel == "" && rt.Unit == "" |
| } |
| -// instanceKeyBackfill counts one pass's outcomes. |
| +// instanceKeyBackfill counts one pass's outcomes. Deferred is a row whose |
| +// configuration sets a key this box has not delivered yet, or whose |
| +// configuration could not be read: not minted, by design, and minted or |
| +// found configured at its next start. |
| type instanceKeyBackfill struct { |
| - Minted, Present, FromConfig, NoEnvFile, Failed, Skipped int |
| + Minted, Present, FromConfig, NoEnvFile, Deferred, Failed, Skipped int |
| } |
| -// backfillInstanceKeys runs the backfill over routes. It never fails the |
| -// agent: a host it cannot write is logged and counted, and that host's next |
| -// start mints its key (or refuses, loudly, if the same fault is still there). |
| -// Nothing logged names a key. |
| +// backfillInstanceKeys runs the backfill over routes, within |
| +// instanceKeyBackfillBudget. It never fails the agent: a host it cannot |
| +// write is logged and counted, and that host's next start mints its key (or |
| +// refuses, loudly, if the same fault is still there). Nothing logged names |
| +// a key. |
| // |
| -// The error is read before the outcome: a refusal (a fenced or conflicting |
| -// row) answers InstanceKeyPresent together with its error, and counting it |
| -// as present would call a host ready whose key was never written. |
| -func backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) instanceKeyBackfill { |
| +// The error is read before the outcome: a refused or failed row answers |
| +// InstanceKeyFailed (the zero outcome) beside its error, and counting an |
| +// outcome first would call a host ready whose key was never written. |
| +func backfillInstanceKeys(ctx context.Context, routes []registry.Route, b instanceKeyBackfiller) instanceKeyBackfill { |
| + ctx, cancel := context.WithTimeout(ctx, instanceKeyBackfillBudget) |
| + defer cancel() |
| var st instanceKeyBackfill |
| for _, rt := range routes { |
| if !instanceKeyRoute(rt) { |
| st.Skipped++ |
| continue |
| } |
| - got, err := b.BackfillInstanceKey(rt) |
| - if err != nil { |
| + got, err := b.BackfillInstanceKey(ctx, rt) |
| + switch { |
| + case errors.Is(err, activator.ErrInstanceKeyDeferred): |
| + st.Deferred++ |
| + log.Printf("agent: instance key for %s: %v", rt.Host, err) |
| + continue |
| + case err != nil: |
| st.Failed++ |
| log.Printf("agent: instance key for %s: %v (its next start mints it, or refuses for the same reason)", rt.Host, err) |
| continue |
| @@ -78,7 +101,7 @@ func backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) inst |
| st.Skipped++ |
| } |
| } |
| - log.Printf("agent: instance keys: %d minted, %d already present, %d set by app configuration, %d never started here, %d failed, %d rows skipped", |
| - st.Minted, st.Present, st.FromConfig, st.NoEnvFile, st.Failed, st.Skipped) |
| + log.Printf("agent: instance keys: %d minted, %d already present, %d set by app configuration, %d never started here, %d waiting for their configuration, %d failed, %d rows skipped", |
| + st.Minted, st.Present, st.FromConfig, st.NoEnvFile, st.Deferred, st.Failed, st.Skipped) |
| return st |
| } |
diff --git a/internal/host/instancekeys_test.go b/internal/host/instancekeys_test.go| index 950dd7ccc..1b6c63339 100644 |
| --- a/internal/host/instancekeys_test.go |
| +++ b/internal/host/instancekeys_test.go |
| @@ -3,8 +3,10 @@ package host |
| import ( |
| "context" |
| "errors" |
| + "fmt" |
| "reflect" |
| "testing" |
| + "time" |
| "github.com/carlosframework/platform/internal/activator" |
| "github.com/carlosframework/platform/internal/layout" |
| @@ -17,7 +19,7 @@ type fakeKeyBackfiller struct { |
| asked []string |
| } |
| -func (f *fakeKeyBackfiller) BackfillInstanceKey(rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| +func (f *fakeKeyBackfiller) BackfillInstanceKey(_ context.Context, rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| f.asked = append(f.asked, rt.Host) |
| return f.got[rt.Host], f.err[rt.Host] |
| } |
| @@ -46,7 +48,7 @@ func TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances(t *testing.T) { |
| got: map[string]activator.InstanceKeyOutcome{"shop.example.com": activator.InstanceKeyMinted, "old.example.com": activator.InstanceKeyPresent, "cfg.example.com": activator.InstanceKeyFromConfig}, |
| err: map[string]error{"bad.example.com": errors.New("read: permission denied")}, |
| } |
| - st := backfillInstanceKeys([]registry.Route{ready("shop.example.com"), ready("old.example.com"), ready("cfg.example.com"), ready("bad.example.com"), held, landing, provisioning, unit, alias, sidecar, addr, replica}, f) |
| + st := backfillInstanceKeys(t.Context(), []registry.Route{ready("shop.example.com"), ready("old.example.com"), ready("cfg.example.com"), ready("bad.example.com"), held, landing, provisioning, unit, alias, sidecar, addr, replica}, f) |
| if len(f.asked) != 4 { |
| t.Fatalf("asked %v, want only the four ready activator instances", f.asked) |
| } |
| @@ -65,7 +67,7 @@ func TestBackfillInstanceKeysCountsARefusalAsFailedNotPresent(t *testing.T) { |
| got: map[string]activator.InstanceKeyOutcome{rt.Host: activator.InstanceKeyPresent}, |
| err: map[string]error{rt.Host: errors.New("not backfilled: layout migration in flight")}, |
| } |
| - st := backfillInstanceKeys([]registry.Route{rt}, f) |
| + st := backfillInstanceKeys(t.Context(), []registry.Route{rt}, f) |
| if st != (instanceKeyBackfill{Failed: 1}) { |
| t.Fatalf("stats = %+v, want the refusal counted as failed only", st) |
| } |
| @@ -76,17 +78,47 @@ func TestBackfillInstanceKeysCountsARefusalAsFailedNotPresent(t *testing.T) { |
| func TestBackfillInstanceKeysCountsAHostNeverStartedHere(t *testing.T) { |
| rt := registry.Route{Host: "shop.example.com", Kind: "instance", Status: "ready", Socket: "/run/carlos/shop.example.com.sock"} |
| f := &fakeKeyBackfiller{got: map[string]activator.InstanceKeyOutcome{rt.Host: activator.InstanceKeyNoEnvFile}} |
| - st := backfillInstanceKeys([]registry.Route{rt}, f) |
| + st := backfillInstanceKeys(t.Context(), []registry.Route{rt}, f) |
| if st != (instanceKeyBackfill{NoEnvFile: 1}) { |
| t.Fatalf("stats = %+v", st) |
| } |
| } |
| +// A row whose configuration is on its way (activator.ErrInstanceKeyDeferred) |
| +// is counted apart from a failure: not minted by design, and found at its |
| +// next start. The whole pass runs under one budget, so a bucket that is |
| +// down costs the pass at most instanceKeyBackfillBudget, not one read's |
| +// timeout per row. |
| +func TestBackfillInstanceKeysCountsADeferredRowApartWithinOneBudget(t *testing.T) { |
| + ready := func(host string) registry.Route { |
| + return registry.Route{Host: host, Kind: "instance", Status: "ready", Socket: "/run/carlos/" + host + ".sock"} |
| + } |
| + var deadlines []time.Time |
| + f := keyBackfillFunc(func(ctx context.Context, rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| + d, ok := ctx.Deadline() |
| + if !ok { |
| + t.Fatalf("%s: the backfill's context has no deadline", rt.Host) |
| + } |
| + deadlines = append(deadlines, d) |
| + if rt.Host == "waiting.example.com" { |
| + return activator.InstanceKeyFailed, fmt.Errorf("%w: its configuration sets CARLOS_INSTANCE_KEY", activator.ErrInstanceKeyDeferred) |
| + } |
| + return activator.InstanceKeyMinted, nil |
| + }) |
| + st := backfillInstanceKeys(context.Background(), []registry.Route{ready("waiting.example.com"), ready("shop.example.com")}, f) |
| + if st != (instanceKeyBackfill{Minted: 1, Deferred: 1}) { |
| + t.Fatalf("stats = %+v, want one minted and one waiting", st) |
| + } |
| + if len(deadlines) != 2 || !deadlines[0].Equal(deadlines[1]) || time.Until(deadlines[0]) > instanceKeyBackfillBudget { |
| + t.Fatalf("deadlines %v, want one budget shared by both rows", deadlines) |
| + } |
| +} |
| + |
| // keyBackfillFunc is a backfiller made of one function. |
| -type keyBackfillFunc func(registry.Route) (activator.InstanceKeyOutcome, error) |
| +type keyBackfillFunc func(context.Context, registry.Route) (activator.InstanceKeyOutcome, error) |
| -func (f keyBackfillFunc) BackfillInstanceKey(rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| - return f(rt) |
| +func (f keyBackfillFunc) BackfillInstanceKey(ctx context.Context, rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| + return f(ctx, rt) |
| } |
| // Task 6 ruling (K1): the start sequence is activatorStart's, so the two |
| @@ -116,7 +148,7 @@ func TestActivatorStartBackfillsOnlyALegacyBoxBetweenRecoveryAndPublish(t *testi |
| Sweep: func(context.Context) error { note("sweep"); return nil }, |
| Recover: func(context.Context, bool) *layout.Pass { note("recover"); return nil }, |
| Routes: func() ([]registry.Route, error) { return []registry.Route{ready}, tc.listErr }, |
| - Keys: keyBackfillFunc(func(registry.Route) (activator.InstanceKeyOutcome, error) { |
| + Keys: keyBackfillFunc(func(context.Context, registry.Route) (activator.InstanceKeyOutcome, error) { |
| note("backfill") |
| return activator.InstanceKeyMinted, nil |
| }), |
diff --git a/internal/host/layoutleg.go b/internal/host/layoutleg.go| index c3b7821aa..2abec0341 100644 |
| --- a/internal/host/layoutleg.go |
| +++ b/internal/host/layoutleg.go |
| @@ -380,7 +380,7 @@ func (s activatorStart) Run(ctx context.Context) { |
| if routes, err := s.Routes(); err != nil { |
| log.Printf("agent: instance keys: list routes: %v (each host's next start mints its key)", err) |
| } else { |
| - backfillInstanceKeys(routes, s.Keys) |
| + backfillInstanceKeys(ctx, routes, s.Keys) |
| } |
| s.Publish(pass) |
| }, s.Start) |
diff --git a/internal/host/replicahomekeys.go b/internal/host/replicahomekeys.go| index c1fbd2e2e..74e349afa 100644 |
| --- a/internal/host/replicahomekeys.go |
| +++ b/internal/host/replicahomekeys.go |
| @@ -52,12 +52,19 @@ func replicaHomeKeys(read func() ([]byte, error)) func(host string) (string, boo |
| } |
| } |
| -// replicaConfigKey is ExecBackend.ReplicaConfigKey (ruling 27): the digest |
| -// of the key rt's configuration in the bucket renders now, read with the |
| -// box's secrets key through the deploy store. The store and key are opened |
| -// on first use and kept once both open; a failure is returned, so the |
| -// replica is refused, and is tried again at its next launch. |
| -func replicaConfigKey(getenv func(string) string, open func(context.Context) (release.Store, error)) func(context.Context, registry.Route) (string, error) { |
| +// configuredKey is ExecBackend.ConfiguredKey (rulings 5 and 27): the |
| +// digest of the key rt's configuration in the bucket renders now, read |
| +// through the deploy store with the box's secrets key. Which boxes have a |
| +// configuration to read is decided as the config guard decides whether a |
| +// box can provide a route's config at all (configguard.go): a box with no |
| +// deployment store and CARLOS_NO_SEALED_CONFIG set has no configuration |
| +// source and renders no key; one with no store and no such flag is an |
| +// error; one with a store and no secrets key renders what it can, and a |
| +// sealed value fails the render here as it fails the guard. The store is |
| +// opened on first use and kept once open; a failed open is returned, so |
| +// the mint is deferred or the replica refused, and is tried again at the |
| +// next launch. |
| +func configuredKey(getenv func(string) string, open func(context.Context) (release.Store, error)) func(context.Context, registry.Route) (string, error) { |
| var ( |
| mu sync.Mutex |
| store release.Store |
| @@ -66,17 +73,20 @@ func replicaConfigKey(getenv func(string) string, open func(context.Context) (re |
| return func(ctx context.Context, rt registry.Route) (string, error) { |
| mu.Lock() |
| if store == nil { |
| - raw := getenv("CARLOS_SECRETS_KEY") |
| - if raw == "" { |
| - mu.Unlock() |
| - return "", errors.New("CARLOS_SECRETS_KEY is not set") |
| + var k *ecdh.PrivateKey |
| + if raw := getenv("CARLOS_SECRETS_KEY"); raw != "" { |
| + var err error |
| + k, err = appconfig.ParsePrivateKey(raw, "CARLOS_SECRETS_KEY") |
| + if err != nil { |
| + mu.Unlock() |
| + return "", err |
| + } |
| } |
| - k, err := appconfig.ParsePrivateKey(raw, "CARLOS_SECRETS_KEY") |
| - if err != nil { |
| + st, err := open(ctx) |
| + if errors.Is(err, errNoDeployStore) && noSealedConfigFlag(getenv) { |
| mu.Unlock() |
| - return "", err |
| + return "", nil // no configuration source: nothing is on its way |
| } |
| - st, err := open(ctx) |
| if err != nil { |
| mu.Unlock() |
| return "", err |
diff --git a/internal/host/replicahomekeys_test.go b/internal/host/replicahomekeys_test.go| index c2939f532..c3823dc9b 100644 |
| --- a/internal/host/replicahomekeys_test.go |
| +++ b/internal/host/replicahomekeys_test.go |
| @@ -81,13 +81,18 @@ func TestReplicaHomeKeysLogsAMalformedListWithoutItsText(t *testing.T) { |
| } |
| } |
| -// Ruling 27's fresh read, as the agent wires it: the bucket's current |
| -// configured key, through the deploy store and the box's secrets key. With |
| -// no secrets key it is an error, so the replica is refused. |
| -func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| +// Rulings 5 and 27's fresh read, as the agent wires it: the bucket's |
| +// current configured key, through the deploy store and the box's secrets |
| +// key. Which boxes have a configuration to read follows the config guard: |
| +// no secrets key still reads the bucket and renders a plain value (a |
| +// sealed one fails the render, as it fails the guard); no deployment |
| +// store is an error, unless CARLOS_NO_SEALED_CONFIG says this deployment |
| +// has no configuration at all, where nothing is on its way and no key is |
| +// rendered. |
| +func TestConfiguredKeyReadsTheBucketNow(t *testing.T) { |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| - priv, _, err := appconfig.GenerateKey() |
| + priv, pubB64, err := appconfig.GenerateKey() |
| if err != nil { |
| t.Fatal(err) |
| } |
| @@ -97,7 +102,7 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", App: "shop", Account: "acct", Replica: true} |
| opened := 0 |
| open := func(context.Context) (release.Store, error) { opened++; return s, nil } |
| - read := replicaConfigKey(func(k string) string { |
| + read := configuredKey(func(k string) string { |
| if k == "CARLOS_SECRETS_KEY" { |
| return priv |
| } |
| @@ -111,18 +116,45 @@ func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) { |
| if opened != 1 { |
| t.Fatalf("the store was opened %d times, want once", opened) |
| } |
| - failing := replicaConfigKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { return nil, errors.New("bucket unreachable") }) |
| + failing := configuredKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { return nil, errors.New("bucket unreachable") }) |
| if _, err := failing(ctx, rt); err == nil { |
| t.Fatal("an unreachable bucket was not an error") |
| } |
| - if _, err := replicaConfigKey(func(string) string { return "" }, open)(ctx, rt); err == nil { |
| - t.Fatal("no secrets key was not an error") |
| + if got, err := configuredKey(func(string) string { return "" }, open)(ctx, rt); err != nil || got != instancekey.Digest("key-A") { |
| + t.Fatalf("no secrets key, a plain configured key = %q, %v; want the digest (the guard admits plain config on a keyless box)", got, err) |
| + } |
| + pub, err := appconfig.ParsePublicKey(pubB64, "test") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + envelope, err := appconfig.SealUnbound(pub, []byte("x")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := appconfig.SetSecret(ctx, s, "acct", "shop", map[string]string{"SEALED": envelope}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := configuredKey(func(string) string { return "" }, open)(ctx, rt); err == nil { |
| + t.Fatal("no secrets key beside a sealed value was not an error") |
| + } |
| + noStore := func(context.Context) (release.Store, error) { return nil, errNoDeployStore } |
| + if _, err := configuredKey(func(string) string { return "" }, noStore)(ctx, rt); err == nil { |
| + t.Fatal("no deployment store was not an error") |
| + } |
| + bypass := func(k string) string { |
| + if k == "CARLOS_NO_SEALED_CONFIG" { |
| + return "1" |
| + } |
| + return "" |
| + } |
| + if got, err := configuredKey(bypass, noStore)(ctx, rt); err != nil || got != "" { |
| + t.Fatalf("no store under CARLOS_NO_SEALED_CONFIG = %q, %v; want no key and no error", got, err) |
| } |
| } |
| // A failed first open is not kept: the same closure opens the store again |
| // at the next replica launch, and reads the bucket once it opens. |
| -func TestReplicaConfigKeyRetriesAFailedOpen(t *testing.T) { |
| +func TestConfiguredKeyRetriesAFailedOpen(t *testing.T) { |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| priv, _, err := appconfig.GenerateKey() |
| @@ -134,7 +166,7 @@ func TestReplicaConfigKeyRetriesAFailedOpen(t *testing.T) { |
| } |
| rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", App: "shop", Account: "acct", Replica: true} |
| opens := 0 |
| - read := replicaConfigKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { |
| + read := configuredKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { |
| opens++ |
| if opens == 1 { |
| return nil, errors.New("bucket unreachable") |