carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

File too large to show in full — download it above (showing a truncated preview below).

1# Instance Key Provision (Region HA L21) Implementation Plan, rev 5.1
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Built (branch `instance-key-build`, tasks 1 to 7 and 3b, 2026-10-11).** The code blocks below are the branch's. Where the build diverged from rev 5.1, each by a ledger ruling (`.superpowers/sdd/2026-10-10-instance-key-provision/progress.md`):
6- Task 2: the flat own file is opened with `O_NOFOLLOW` (`openOwnEnvNoFollow`, with an `Lstat` stand-in for windows), a failed append is truncated back to the descriptor's size before the error returns, and `ensureInstanceKeyFor` refuses a fenced or conflicting row itself (the launch bodies classify again after the dispatcher's check). A symlinked flat `<host>.env` now fails the launch; the rollout checks for one first.
7- Task 3b: the sidecar worker (`StartWorkerStaged`) and the legacy co-spawned sidecar (`startSidecarChildFrom`) pass the replica gate too, as a seed does (no control socket); a key in a worker's agent-issued `extra` is refused as of unknown origin. `replicaKeyGate` takes the key's layer; the home-keys list is a file the agent reads at every launch, not an environment variable.
8- Task 4: the claim's guarded write mints the admin token only when the effective env has none, so a configured token is never shadowed by a second one; a missing own file with both values configured proceeds as before; the key's refusal is wrapped with the host once, in the launch source.
9- Task 6: the activator's start sequence is a named, tested function (`activatorStart.Run`), with the backfill between layout recovery and the pass's publication on a legacy box and nothing on a claim box.
10- Final fix wave (the two final reviews, 2026-10-11; rulings I1 to I5 and P3 in the ledger): nothing is minted until the row's configuration is delivered, by a fresh bucket read before the mint (`ExecBackend.ConfiguredKey`, the field Task 3b called `ReplicaConfigKey`; the delivery rule is under ruling 5; `ErrInstanceKeyDeferred`; the backfill's one-minute budget and its "waiting for their configuration" count); the flat own file and its configuration are opened `O_NONBLOCK` and must be regular files (a FIFO is refused, never waited on), and the flat configuration read is capped at 1 MiB as the nested one is; `SnapshotClaimLaunch` is the gateless read the claim's input policy uses and `SelectClaimLaunch` the launch sources' read with the gate, so an identity check never reads the bucket and keeps `ErrChanged`; the claim path's backend is built by `claimExecBackend` with the same two readers the agent's backend gets (`instanceKeyReaders`); the gate's refusal line is said once an hour per host and reason; the adoption verb refuses a body with anything after its one object; the zero `InstanceKeyOutcome` is `InstanceKeyFailed`, every error's companion; `ensureInstanceKeyFor` and `BackfillInstanceKey` take a context, and the exported ensure wrappers are the unexported `ensureInstanceKeyNoConfig` pair, used by tests. The rollout gained the keyless-replica sidecar and worker check.
11- Known gaps, obligations and follow-ups are at the end, with the rollout's pre-roll checks.
12
13**Goal:** The platform mints every exec-backed instance's `CARLOS_INSTANCE_KEY` on the box, create-only and durably, into the instance's own `<host>.env` before any process of the instance starts, and backfills it into every existing instance. Relocation accepts a key only with evidence that the app actually seals with it: its own `instance.key`, or an adoption record bound to the writer whose data is pinned, for the exact key the target will deliver, pinned by digest and checked again at install.
14
15**Architecture:** A small leaf package, `internal/instancekey`, holds the one rule for the key: the env parser, which file's key the app receives, and the adoption record. The activator mints by it (`instancekey.go`), right after the existing `CARLOS_ADMIN_TOKEN` mint on every launch path and under the same lock, and makes the file durable on every call. A control-socket verb, `PUT /instance-key/adoption`, records a process's adoption beside `<host>.env`, bound to the writer the agent admitted it as, and publishes it on the running session through `hostfiles.PlatformRelay`; claim launches mount it once they bind a control socket (O6). A startup backfill in `internal/host` appends a key to every existing `<host>.env` on boxes where host claims are off. On claim boxes the launch source mints under the claim's guard before `Materialize`. Stage B's pinner (O2'), the C8 provider (O1'), the relocation install (O3') and moves (O7) carry the evidence across boxes and retire it. No flag.
16
17**Tech Stack:** Go, `crypto/rand`, `encoding/base64`, `crypto/sha256`, `internal/instancefs` (openat2), `internal/safefile`, the registry, the claim launch source, the control socket.
18
19**Spec:**
20- Decision: the L21 row of `docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md` on branch `region-ha-1b-next-build` (line 357; Paul, 2026-10-09).
21- Region HA spec `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`: H6 (line 131) and §5.6 (line 539): `instance.key`, `report.token`, `<host>.env` are always continuous.
22- Framework contract: rastrillo/rastrillo discussion #66 (agreed; landed on rastrillo main as e3ea08d). `carlos.InstanceKey(own string) (string, error)` resolves, in order: (1) `own`, the app's own key; (2) `$CARLOS_INSTANCE_KEY`, verbatim; (3) on CARLOS (`carlos.Running()`, which checks `$CARLOS_CONTROL_SOCKET`) with no key, `ErrNoInstanceKey`; (4) off CARLOS, `ErrNotOnCarlos`, today's behaviour. Read at e3ea08de in `~/amadan.net/rastrillo/rastrillo` (`carlos/instancekey.go`, `carlos/schedule.go:28`).
23- Stage B plan `docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md` on `region-ha-1b-next-build` (919662d2): rulings 25 and 55, Task 8b step 0 (its O2' variant with a `KeyEvidence` reader) and Task 8c (O3). This plan's O2' replaces that step.
24- Review: Astra rounds 1 to 6, `docs/superpowers/plans/CODEX-PLAN-INSTANCE-KEY-R1.md` to `-R6.md` (verdict each time: not ready; rounds 3 to 6 found no P1). Their findings are answered in the six tables below, newest first.
25- Task 1 is built and reviewed on this branch (c568c13b, afa89057, e8487c3a); its code below is the branch's.
26
27## Rev 5.1: round 6 dispositions
28
29Astra's round 6 (`CODEX-PLAN-INSTANCE-KEY-R6.md`, reviewed at 2c39819c) closed R5-2 and R5-3, and judged R5-1's approach sound. It found two P2s. Task 1 is now built and reviewed on this branch (c568c13b, afa89057, e8487c3a). Rev 5.1 changes none of Task 1's code; it copies the reviewed code into Task 1 so the plan and the branch match.
30
31| # | Finding | Disposition | Change | Test |
32|---|---|---|---|---|
33| R6-1 | P2. A concurrent config removal bypasses the fresh check. The launch captures configured key A, but the gate re-read the files to find the source: a sync that removed the key in between makes `Effective` say `LayerNone`, and the gate admitted the launch without asking the bucket, while the child still got A. | **Accepted** (ruling 29). Verified: rev 5's gate called `instanceEnvParts` again. | One read decides both the env and the key's source: `instanceEnvLayer` returns `mergeEnv`'s env and `Effective`'s layer from the same two maps. Every gate call (Seed, Start, the nested prepare, the claim snapshot) passes that layer. The gate reads no env file, and refuses a key whose layer is unknown. | `TestReplicaGateDecidesOnTheLaunchesOwnRead`: a deterministic interleaving through `Start`, where a hook rewrites `<host>.config.env` without the key between the launch's read and the gate. It is refused; with rev 5's second read restored, the app ran with A. Also `TestReplicaGateReadsNoEnvFile` |
34| R6-2 | P2. The procedure never applied list changes to the running agent: `replicaHomeKeys` parsed the environment once at startup. | **Accepted** (ruling 30): **read on each launch.** | The list moves from the agent's environment to a file, `/etc/carlos/replica-home-keys`, written whole by the pre-roll check (temporary file, then rename). The agent reads it again at every replica launch and keeps no copy. An invalidation takes effect at the next launch, with no agent restart. The procedure checks the file before the key changes, with the replicas kept stopped throughout. | `TestReplicaHomeKeysReadsTheListAtEveryLaunch` (a removed entry stops verifying at the next call; a replaced one verifies the new digest); `TestReplicaHomeKeysLogsAMalformedListWithoutItsText` (also an unreadable list) |
35
36**Counts:** 2 accepted, 0 declined.
37
38**Also in rev 5.1:**
39- Task 1's code is the reviewed branch's, from c568c13b to e8487c3a. `ParseAdoption` accepts only the exact bytes `AdoptionRecord` writes; `ParseEnv`'s behaviour is pinned by a test; and the two activator doc comments name the empty-key exception. Task 1 also gains `TestEmptyOwnInstanceKeyShadowsNothing`.
40- A rollout note from Task 1's review: a mixed-version roll and the moves readiness check (rollout step 3).
41
42**Effect on stage B (IK-1):** none in this revision. The reviewed `ParseAdoption` is stricter: it accepts only the canonical bytes, which is all `AdoptionRecord` ever writes. O2' and its test were re-run against stage B's current tip (b9a753f0) with the reviewed `instancekey`, and pass.
43
44## Rev 5: round 5 dispositions
45
46Astra's round 5 (`CODEX-PLAN-INSTANCE-KEY-R5.md`, reviewed at 37c48a40) accepted the rest of the plan, including the keyless gate on every launch path, and `childEnv` stripping the list from inherited tenant environments. It found two P2s and a P3, all about replicas. Paul's rulings 27 and 28 answer the P2s. The replica scope stays narrow.
47
48| # | Finding | Disposition | Change | Test |
49|---|---|---|---|---|
50| R5-1 | P2. A stale digest admits the wrong key after an ordinary configuration change. `SyncExec` reconciles each box separately and hourly, so the home can hold B while the replica still holds A and still matches A's entry. | **Accepted** (ruling 27). Verified: `configSyncLoop` runs hourly per box (agent.go:1078). | A configured key is never changed in place; that would be a rotation, which this plan does not support. The rollout gives the procedure for any configured-key change on an app with replicas. There is also a cheap fresh-evidence check that needs no O8: a replica whose key comes from its configuration is started only if the bucket's configuration renders that same key now (`appconfig.ConfiguredKeyDigest`, through `ExecBackend.ConfiguredKey`). That catches a shared configuration change the replica's box has not synced. The list itself is written only by the pre-roll check, never refreshed from a replica's configuration. | `TestReplicaLaunchWithoutAControlSocket`: "the home updated, this replica's configuration delayed: refused" (it fails with the fresh check removed); "the bucket cannot be read: refused"; "no fresh reader: refused"; "residual: a change only on the home is not seen". Also `TestConfiguredKeyDigestReadsTheBucketNow` and `TestReplicaConfigKeyReadsTheBucketNow` |
51| R5-2 | P2. O8 has no source for a configured key's value: the published `<host>.env` leaves it out (ruling 5), and the adoption record holds only its digest. | **Accepted** (ruling 28). | O8 delivers only a key the platform minted into the home's `<host>.env`. A home whose key exists only in configuration is explicitly deferred: its keyless replica keeps refusing, and its configured-key replica follows ruling 27. | `TestReplicaLaunchWithoutAControlSocket`: "a config-only home's keyless replica is refused" |
52| R5-3 | P3. A malformed list entry can reach the log unredacted: a bare digest with no `replica-host=` lands in the error. | **Accepted.** | `ParseHomeKeys` reports the entry's position and the reason it was rejected, never the entry's text. The agent logs that error and nothing else. | `TestParseHomeKeys` (no error echoes its input); `TestReplicaHomeKeysLogsAMalformedListWithoutItsText` (the logging path) |
53
54**Counts:** 3 accepted, 0 declined.
55
56**Effect on stage B (IK-1):** none. IK-1's symbols (`ParseEnv`, `Effective`, `Digest`, `Binding`, `Adoption`, `AdoptionRecord`, `ParseAdoption`, `Binds`, `VerifyDelivered`, `AdoptionItem`, `AdoptionSuffix`) are unchanged in name, signature and behaviour. The only `instancekey` change is the error text of `ParseHomeKeys`, which stage B does not use. O2' and its test are unchanged.
57
58## Rev 4: round 4 dispositions
59
60Astra's round 4 (`CODEX-PLAN-INSTANCE-KEY-R4.md`, reviewed at cba8ae4e) found two P2s in the rev 3.1 replica fix. Paul decided the follow-up instead of deferring it: **a replica gets its home's key** (ruling 26, which replaces rulings 17 and 25).
61
62| # | Finding | Disposition | Change | Test |
63|---|---|---|---|---|
64| R4-1 | P2. The unconditional strip breaks existing replicas that share a configured key matching the home's. "Runs everywhere" is wrong. | **Accepted** (ruling 26). | The strip is reverted: Task 3b's `fileEnvFor` is gone, and `mergeEnv` composes a replica's env as any other. A replica whose key is exactly its home's keeps it. One whose key differs is refused with a log line. A pre-roll check (rollout step 1) lists every replica's key against its home's before any box is upgraded, and records the matches in `CARLOS_REPLICA_HOME_KEYS`. The wording at rollout step 6 is corrected. | `TestReplicaLaunchWithoutAControlSocket` (the home's configured key runs; a pre-existing env key that is the home's runs; a differing key is refused; an unverifiable key is refused); `TestClaimReplicaLaunchIsGated`; `TestNestedReplicaPrepareIsGated`; `TestParseHomeKeys`, `TestReplicaHomeKeysFromTheEnvironment` |
65| R4-2 | P2. Removing the variable does not refuse on a claim launch: with no control socket, `InstanceKey("")` returns `ErrNotOnCarlos` and the app mints its own key over the home's data. | **Accepted** (ruling 26). Verified: rastrillo e3ea08de's step 4, and `SnapshotClaimLaunch`'s env has no socket (claim_snapshot.go:116). | An agent-side refusal, `replicaKeyGate`, runs on every replica launch before anything starts. A replica with no key of its home's starts only with a control socket, where the helper refuses at step 3. Without one (a seed, a claim launch before O6, a failed bind) the agent refuses the launch itself. | `TestReplicaLaunchWithoutAControlSocket` and `TestReplicaSeedIsGated` run the app as rastrillo's resolution. A keyless replica is refused and no app runs; with the gate disabled the app reports "minted". `TestKeylessReplicaWithAControlSocketLeavesTheHelperToRefuse` (the helper reports `ErrNoInstanceKey`) |
66
67**Counts:** 2 accepted, 0 declined.
68
69**New obligation O8 (replica placements, #162).** The platform has no channel today for a replica to read its home's key. A replica record names no home: `provision.Replica` copies the app, account, pool and channel, not the home's host. The one-way mirror between home and replica is the app's own (titogo's), not the platform's. A home's `<host>.env` is published only on claim boxes, once O1' is in, into the home region's data bucket. So obtaining the home key's *value* for a replica is O8, with this plan's fallback: a replica that cannot be handed its home's key gets none, and refuses to start reliably (ruling 26). Until O8, verification uses the digests the pre-roll check records.
70
71**Effect on stage B and O2':**
72- O2' and its test are unchanged.
73- `SnapshotClaimLaunch` can now return `activator.ErrReplicaKey` for a replica row. A relocated replica's launch passes the same gate, so it needs its home's key, verified, or O6's socket.
74- `LaunchEnvironment` is as in rev 3 again, so the moves env report lists a replica's key like any other.
75
76## Rev 3.1: round 3 dispositions
77
78Astra's round 3 (`CODEX-PLAN-INSTANCE-KEY-R3.md`, reviewed at f649d972) found no P1. It closed round 2's five findings at the contract level and judged the `source` check sound. Two fixes remain.
79
80| # | Finding | Disposition | Change | Test |
81|---|---|---|---|---|
82| R3-1 | P2. The replica exclusion stops minting but not delivery. A replica whose configuration supplies B, or whose own env already holds a key, is handed it, and the helper's step 2 returns it over the home's replicated data. The adoption 409 cannot stop that, because the helper ignores failures. | **Accepted** (ruling 25). Verified: `InstanceKeyReplica` only skipped the mint; `mergeEnv` still delivered the configured or existing key. | No launch of a replica row is handed `CARLOS_INSTANCE_KEY`: `fileEnvFor` strips it from the composed env on every launch path (Start, Seed, the nested prepare, `LaunchEnvironment`, the claim snapshot). With the variable absent, the helper's step 2 has nothing to return. No rastrillo change. | `TestReplicaLaunchIsHandedNoInstanceKey` (a real replica Start and Seed: a configured key, a pre-existing env key, both; the home row delivers the key as the control case); `TestNestedReplicaPrepareIsHandedNoInstanceKey` (run and seed roles); `TestClaimReplicaLaunchIsHandedNoInstanceKey` (the claim launch env and the snapshot) |
83| R3-2 | P3. The follow-ups said "O6 and O7, in that order", contradicting ruling 24. | **Accepted.** | The follow-up now says O7 ships before O6, and why. | none |
84
85**Counts:** 2 accepted, 0 declined.
86
87**Effect on stage B and O2':** none. O2' and its test are unchanged. Stage B consumes `instancekey`, `ClaimLaunchSnapshot` and `PinObject` as before. One behaviour visible elsewhere: the moves env report for a replica (`LaunchEnvironment`) no longer lists `CARLOS_INSTANCE_KEY`, on the source and the target alike, so the two still compare equal.
88
89## Rev 3: round 2 dispositions
90
91Astra's round 2 (`CODEX-PLAN-INSTANCE-KEY-R2.md`, reviewed at a21e8225) found three P1s and two P2s. It also withdrew round 1's finding 6, and accepted the missing claim socket as an explicit refusal. Each finding was checked against the code, and rulings 19 to 24 (Paul's R-A to R-F) answer them. Source (c) and `appconfig.ConfiguredInstanceKey` are gone. The rev 2 table below stays as history; where it and this table disagree, this one stands.
92
93| # | Finding | Disposition | Change | Test |
94|---|---|---|---|---|
95| R2-1 | P1. A configured-key source still bypasses the refusal: configuration supplies A, the app's own unpublished B wins (step 1), and (c) passes. | **Accepted** (ruling 20). Verified: rev 2's O2' consulted `ConfigKey` with no word from the app. | Source (c) and `ConfiguredInstanceKey` are dropped. A configured key counts only through an adoption record, reported by the helper with `source: config`, exactly as a platform key. An app on its own key B sends no report, so there is no record and the pin refuses. | O2' `TestStorePinnerKeyEvidence`, row "configured A, the app's own unpublished B: no report, no record" |
96| R2-2 | P1. Configuration evidence neither identifies nor pins the key: a configuration changed from A to B before the box dies, or between pin and install, delivers B over data sealed with A. | **Accepted** (ruling 20). Verified: `ConfiguredInstanceKey` returned a boolean from the current configuration. | The record names the key by digest. The pinner pins it (`PinObject.InstanceKey`). The install checks that the env files it is about to launch with deliver exactly that key (`instancekey.VerifyDelivered`), and refuses otherwise (O3'). | `TestVerifyDeliveredRefusesAnyOtherKey` ("the configuration changed since the pin"); O2' config rows; O3''s install test |
97| R2-3 | P1. An adoption record outlives the writer it describes: a later release on its own key, or a rollback, inherits an earlier session's record. | **Accepted** (rulings 20, 21). Verified: rev 2's record held only a version, a name and a digest. | The record binds host, incarnation, writer session and generation, all taken from the agent's own launch state for that process. Every new writer admission starts with no record (O1'). The pinner accepts only a record that binds the pinned writer. | `TestAdoptionRecordBindsOneWriter`; O2' rows "a stale record from a previous release", "a rollback", "a record from another installation"; `TestInstanceKeyAdoptionRecordsTheLaunchesWriter` (a session in the request is a 400) |
98| R2-4 | P2. A first adoption has no publication path during the running session, and O6 is not gated in the HA checklist. | **Accepted** (rulings 22, 23). | The verb hands the durable record to the session's publisher through `hostfiles.PlatformRelay`, and the publisher's continuous loop commits it. 204 means durable on disk; publication follows the publisher's bound. O6 is a gate in O4's checklist. | `TestAdoptionSurvivesBoxLossWithoutAnotherAdmissionOrRelease`, `TestPlatformRelayStagesTheLatestBodyWithoutBlockingItsCaller`, `TestInstanceKeyAdoptionIsDurableBeforeItAnswers` |
99| R2-5 | P2. Deferring the record's retirement breaks a later install on the source box: the census refuses the leftover. | **Accepted** (ruling 24). Verified: `installArtifactPaths` (movesinstall.go:751) does not name it, and `EmptyInstallationCensus.CheckLocal` refuses an unclassified host artifact. | O7 puts the record in moves' journalled artifact lifecycle, with the exact lists. Its writer (O6) does not ship before O7. | O7's `TestMoveAwayRetiresTheAdoptionRecordSoTheHostCanReturn` (in the moves change) |
100| note | The record is tenant-forgeable. | **Answered** (ruling 19). | The record is a trusted-helper assertion against accidental loss, not independent evidence. A tenant who forges it can lose only its own data. | none |
101| R1-6 | Round 1's merge option. | **Closed**: Astra withdrew it. Reconciling AGENTS.md's local-squash text with the hub rule is Paul's call. | none | none |
102| gap | Claim launches have no control socket. | **Accepted as an explicit refusal** (ruling 23). | O4's checklist gates relocation for apps on a platform or configured key on O6. | none |
103
104**Counts:** findings R2-1 to R2-5: 5 accepted, 0 declined. The forgeability note is answered by ruling 19, and R1-6 is closed.
105
106**Found while revising:**
107- **The helper cannot tell the source itself.** A platform key and a configured key both arrive as `$CARLOS_INSTANCE_KEY` (ruling 13). So O6 delivers the source beside the key, as an agent-owned `CARLOS_INSTANCE_KEY_SOURCE`, and the helper echoes it. The agent still checks the echo against its own launch state, which is authoritative.
108- **Stage B's rev 3 (919662d2) already changed its O2.** It moved to a `StorePinner.KeyEvidence` reader (its ruling 55) that still accepts a configured key on presence, and still parses `export` lines. This plan's O2' replaces that reader. The evidence is in the pinned manifest itself, so the pinner needs no outside reader.
109
110## Rev 2: dispositions of Astra round 1
111
112Each finding was checked against the code on this branch (cut from origin/main 0901e835) before it was answered. "Provisional default" rows are Paul's open questions 1 to 4: they are the defaults this plan builds on, and **Paul has not yet confirmed them**.
113
114| # | Finding | Disposition | Change | Test |
115|---|---|---|---|---|
116| 1 | P1. O2 accepts a platform key the app may never have used: the app's own `instance.key` wins in rastrillo, and a missing `instance.key` beside a backed-up platform key would pass. | **Accepted.** Verified: rastrillo e3ea08de `InstanceKey` returns `own` first, and rev 1's mint ran whatever the app used. | Ruling 15: relocation accepts a key only with positive evidence, (a) `instance.key`, (b) the platform key plus an adoption record for that exact key, or (c) a key from the configuration. The control socket has no verb for (b) today, so Task 5 adds `PUT /instance-key/adoption` and the record beside `<host>.env`. O1' publishes the record, O2' requires it, and the rastrillo ask below asks the helper to call it. | O2' `TestStorePinnerKeyEvidence` (compiled and run on a stage B scratch), row "a backed-up platform key, the app's own key missing" refuses; `TestInstanceKeyAdoptionRecordsOnlyTheOwnFilesKey`; `TestNestedControlSocketRecordsAdoptionDurably` |
117| 2 | P1. A failed durability step becomes success on retry: after the append, a failed chmod or sync leaves a key the next ensure returns early on; the claim early return skips it too. | **Accepted.** Verified at rev 1's lines 585-647 and claim_snapshot.go:56-58. | Ruling 4: every ensure ends with fchmod 0600, file fsync and directory fsync, also when the key is already there; the bytes are never rewritten. The claim early return fsyncs the file and its directory without writing (ruling 18), and takes the guarded path when the mode needs setting. | `TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess`, `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced`, `TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers` |
118| 3 | P2. Every nested backfill that appends fails with `EINVAL`: `create=false` drops `O_CREAT` but still passes mode 0600 to openat2. | **Accepted.** Verified: `instancefs.OpenBeneath` puts `perm` in `OpenHow.Mode` (instancefs_linux.go:93-97), and openat2 refuses a non-zero mode without `O_CREAT`. | `openInstanceEnv` passes mode 0 unless it creates. The fchmod on the descriptor sets 0600. | `TestBackfillInstanceKeyAppendsToANestedEnvFile` |
119| 4 | P2. An empty `CARLOS_INSTANCE_KEY=` in `<host>.env` defeats a configured key: the mint defers to config, but `mergeEnv` hands the child the empty override. | **Accepted.** Verified: `mergeEnv` lets the overlay win every conflict, empty values included (backend_exec.go:1980-1990). | Ruling 13: the mint decides on the effective env (`instancekey.Effective`), and `mergeEnv` applies the same rule, so an empty own key never shadows a non-empty configured one. The child never gets an empty key while a non-empty one exists. `LaunchEnvironment` stops reporting that name as shadowed. | `TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists` (both files, through `Start`); `TestEffectiveNeverHandsOutAnEmptyKeyWhileOneExists`; `TestEnsureInstanceKeyDefersToTheRenderedConfiguration` |
120| 5 | P2. O2 refuses a key that comes from the configuration, which ruling 5 supports. | **Accepted.** | Ruling 15 (c): the configuration in the bucket is the evidence, and the target renders the same configuration. `appconfig.ConfiguredInstanceKey` renders it with SyncExec's own loader and renderer; O2' calls it through `StorePinner.ConfigKey`. | `TestConfiguredInstanceKeyReadsTheRenderedConfiguration`; O2' rows "(c) ..." |
121| 6 | P3. The rollout's `amadan branch merge -squash` contradicts AGENTS.md's local-squash workflow. | **Declined.** The hub has `-squash`, and Paul's standing rule (2026-09-22) is squash-merge on the hub. AGENTS.md's local-squash text predates that rule. | Rollout step 1 now says: merging is Paul's call, as a squash on the hub, after CI is green on the tip. | none |
122| 7 | Open question 5: claim-session restarts can mint outside the C8 publish point. Astra: the premise is not established. | **Accepted: the premise was wrong.** Verified: `startAdmitted` calls `Prepare` once (claimwake.go:355) and loops over attempts, calling `Materialize` before each (:405-412); the assembly's inputs are captured once (claimwakeassembly.go:353, :377); production claim swaps refuse (`claimsSwapRefusal`, claimgen.go:30). | Ruling 16 states the corrected premise. Question 5 is removed. | `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` |
123| 8 | Coverage gaps: a first flat seed, the nested seed role, a real hibernation restore (a second `Start` does not exercise `Restore`). | **Accepted.** | Tests added. | `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`; `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile/seed`; `TestHibernationRestoreKeepsTheInstanceKey` |
124| 9 | O1's capturing fake proves ordering only, not committed publication or refusal on a failed publication. | **Accepted, moved.** Those need the provider. | O1' names the provider's integration tests, in the C8 plan. Task 4's test keeps proving the ordering. | in the C8 provider's plan (O1') |
125| 10 | Legacy durability is disk plus move archives: no off-box recovery after box loss. | **Acknowledged, no change.** | Ruling 6 already says so. Relocation cannot run on a legacy box. | none |
126| Q1 | Replicas get their own key; should a replica get its home's key? | **Provisional default, Paul not yet confirmed.** Deferred to a follow-up. | Ruling 17: until then replicas are excluded from platform-key adoption. No path mints a key on a replica row, the backfill skips replicas, and the adoption verb refuses on one. An adopting app on a replica refuses to start unless it passes its own key. | `TestReplicaRowsGetNoPlatformKey`; the backfill test's replica row; the adoption test's replica case |
127| Q2 | A key set in configuration: defer to it, or reserve the name? | **Provisional default, Paul not yet confirmed.** | Configuration wins (ruling 5). The name is not reserved. | `TestEnsureInstanceKeyDefersToTheRenderedConfiguration` |
128| Q3 | Should the platform provision `report.token`? | **Provisional default, Paul not yet confirmed.** | No `report.token`. | none |
129| Q4 | Workers and sidecars: should they get their instance's key? | **Provisional default, Paul not yet confirmed.** | A follow-up; out of scope (ruling 12). | none |
130
131**Counts:** findings 1 to 10: 8 accepted, 1 declined (6), 1 acknowledged with no change (10). Open questions 1 to 4: 4 provisional defaults.
132
133**Found while revising (not in Astra's list):**
134- Stage B's O2 parses the env item with its own rule, accepting `export CARLOS_INSTANCE_KEY="k1"`. The activator's parser reads that line as the variable `export CARLOS_INSTANCE_KEY`, so the app never receives it. O2' parses with `instancekey.ParseEnv`, the activator's own parser, and its test pins the exported line as no key.
135- Claim launches carry no control socket. `SnapshotClaimLaunch` composes the env with `agentOwnedVars(rt, "", vars, false)` (claim_snapshot.go:116), which strips `CARLOS_CONTROL_SOCKET`. An app first adopting the platform key on a claim box cannot record adoption until claim launches bind one (obligation O6).
136
137## Global Constraints
138
139- **Format (rastrillo #66):** 32 CSPRNG bytes, standard padded base64 (44 characters). The key is the exact bytes the env file yields: no quotes, no trailing newline in the value, no trimming. Changing the encoding later is a key rotation.
140- **Create-only:** a `CARLOS_INSTANCE_KEY` already in `<host>.env` is never rewritten, by any path, whatever its shape.
141- **Durable before success:** an ensure that reports success has set 0600 on the file, fsynced it and fsynced its directory, in that call.
142- **Variable name:** `CARLOS_INSTANCE_KEY`, exactly.
143- **File mode:** `<host>.env` and the adoption record are 0600.
144- **Never logged, never in an error, never in the console.** Errors and log lines name paths, never values. The adoption record holds a SHA-256 digest, never the key.
145- **Rollout order (rastrillo #66):** the key is on every box, existing instances included, before any app adopts the helper.
146- **Neutral examples:** `shop`, `example.com`, `acct`. No tenant or app names in code, tests or comments.
147- **Cross-compile:** `internal/host`, `internal/activator`, `internal/appconfig` and `internal/instancekey` build for darwin, windows and linux/arm64 (`20-cross`). Tests that need openat2, `mkfifo` or a unix socket go in `*_linux_test.go`.
148- **Gate (AGENTS.md "Build & test"):** `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean.
149- **No user-facing copy.** Log lines are operator-facing, in the surrounding style. The adoption verb answers with `http.StatusText` bodies only.
150
151## What the code does today (branch `instance-key-build`, cut from origin/main 0901e835)
152
153**Two things are called "overlay".** The decision's "overlay env file, alongside `$CARLOS_ADMIN_TOKEN`" is `<host>.env`, the instance's own env file:
154- `ExecBackend.pathsFor` names it `<dataDir>/<host>.env` beside `<host>.config.env` (internal/activator/paths.go:50-52); nested, `dataDir` is the instance directory (paths.go:36-37).
155- `mergeEnv` calls it the overlay: "base (the sync-owned `<host>.config.env`, bucket-derived) under overlay (the instance's own `<host>.env`). The overlay wins every conflict" (internal/activator/backend_exec.go:1974-1990). It wins with an empty value too: `CARLOS_INSTANCE_KEY=` in `<host>.env` hands the child an empty key over a configured one (finding 4).
156- The moves archive calls it "the `<host>.env` overlay" (internal/moves/archive/archive_test.go:36-37), and region HA's H6 names `<host>.env` as always continuous.
157
158The other overlay is appconfig's instance config layer, `apps/<acct>/<app>/instances-config/<host>.json`, written by `appconfig.ApplyInstance` (internal/appconfig/instances.go:1-24, 371-376). It is app configuration, sealed with the deployment's secrets key, rendered into `<host>.config.env` by `SyncExec` (internal/appconfig/syncexec.go:1-9). `LoadMergedHostVersions` (instances.go:222) and `RenderEnv` (materialize.go:37) are the one loader and renderer that SyncExec, the held render and `CheckRenderedIdentity` (claim_identity.go:13) share. Sync never reads, writes or reaps `<host>.env` (internal/appconfig/sync.go:270-275, pinned by `TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes`, syncexec_test.go:683).
159
160**The admin token is the precedent** (internal/activator/admintoken.go):
161- minted by `ensureAdminToken` (:142-238): one read of the raw bytes, `parseEnvBytes` (:245-257), 32 `crypto/rand` bytes hex-encoded, appended with `O_APPEND|O_CREATE` at 0600 on a fresh line (:205-229), then `chmod 0600` (:234). It never fsyncs;
162- create-only, malformed tokens logged once an hour per file (`logPerFile`, :70-77) and left in place (:189-203);
163- serialized by the process-wide `adminTokenMu` (:43);
164- nested files go through `instancefs.ReadBeneathLimit` / `OpenBeneath` (:150-216). `OpenBeneath` passes its `perm` straight into openat2's `OpenHow.Mode` (internal/instancefs/instancefs_linux.go:93-97); the kernel refuses a non-zero mode without `O_CREAT` with `EINVAL` (finding 3).
165
166Its callers are every launch path, each before the env is read: `seedChild` (backend_exec.go:831), `spawnChildFrom`, which flat `Start` and `StartGen` share (:1369), the nested `prepareAt` for the run and seed roles (internal/activator/prepare.go:60), the claim's `EnsureClaimAdminToken` (internal/activator/claim_snapshot.go:40-67), called by the production launch source `claimServingLaunchSource` under `claimAdminTokens.With` (internal/host/claimproduction.go:356-428), and the scheduler tick's `scheduleToken` (internal/host/schedule_wiring.go:150-180).
167
168**Creation against first start.** The console writes the instance record. The box's reconcile mints the registry row with `Status: "ready"` and starts nothing (internal/provision/reconcile.go:425-435). The first process is started by the first wake, `doWake`: `Restore` (internal/activator/activator.go:1797), then `Seed` when the database is fresh (:1817), then `Start`. `Restore` restores the database only (backend_exec.go:663-757). Both seed and start mint before they read the env.
169
170**The control socket.** `edge.ListenControl` (internal/edge/purge.go:126) binds `<socket>.control` before every exec spawn and serves `POST /purge` (when there is a cache) plus whatever the activator mounts: today `schedule.MountGated`, which serves `GET /schedule-capabilities`, `PUT` and `DELETE /schedule/{name}` and `GET /schedule`, each bearer-gated on the host's own `CARLOS_ADMIN_TOKEN` (internal/schedule/control.go:97-225). It is mounted in `spawnChildFrom` (backend_exec.go:1406-1414) and in the nested `listenControl` (prepare.go:135-155), and only when `b.Timers != nil`. Rastrillo e3ea08de's `carlos` package calls only `PUT` and `DELETE /schedule/{name}` (`carlos/schedule.go`). **Nothing on the socket lets an app tell the agent which key it uses,** so source (b') needs a new verb here and a call in rastrillo (the ask below).
171
172**Claim launches have no control socket.** `SnapshotClaimLaunch` composes the launch env with `agentOwnedVars(rt, "", vars, false)` (claim_snapshot.go:116), which strips `CARLOS_CONTROL_SOCKET`. A claim-launched app is not `carlos.Running()`, but rastrillo's step 2 still returns `$CARLOS_INSTANCE_KEY`. It cannot call the adoption verb (O6).
173
174**Claim retries.** `startAdmitted` reads the claim, calls `Prepare` once (internal/activator/claimwake.go:355), restores, then loops over attempts: `Reserve`, then `Materialize`, then the start (:405-412). The assembly captures the launch inputs once, from the selection (`_, in := a.snapshot()`, internal/host/claimwakeassembly.go:353), and `Materialize` is bound to them (:377). `ClaimStart` documents that "process retries reuse it". Production claim swaps refuse before they raise anything (`claimsSwapRefusal`, internal/activator/claimgen.go:30). So no claim attempt reaches the launch source again, and none runs before its `Materialize`.
175
176**Backup of `<host>.env` today.**
177- **Legacy boxes** (host claims off; every production box today): nothing publishes `<host>.env`. `internal/hostfiles` is imported by no production package (internal/hostfiles/contract.go:13-15). `hostfiles.EnvItem` exists only on `region-ha-1b-next-build` (internal/hostfiles/platform.go:13 there). The file's only off-box copy is a move's sealed archive (`archive.Build`, internal/moves/archive/archive.go:395).
178- **Claim boxes:** the serving owners call the launch source in `Select` (internal/host/claimowners.go:375-416) and then `Materialize`, which calls `ClaimFilesManifest.Establish` "before the app executes" (claimowners.go:125-138, 425-438). No C8 provider exists on main: `Files == nil` refuses every serving wake (`claimFilesRefusal`, :140-143).
179
180**Moves (L16's shared install routine on main).** `installFiles` extracts the archive and `placeStaged` moves the env into place only when `<host>.env` is absent or is this install's own (internal/host/moveslandinstall.go:130-280). The move's env report HMACs every value and classes any name it does not know as `equal` (internal/moves/envreport.go:24-38, 90-103, 117-150). The archive carries the data directory and `<host>.env` only. The source's finalization parks a fixed, journalled list of flat paths (`installArtifactPaths`, internal/host/movesinstall.go:751; `sourceArtifactPaths`, movesfinal.go:586).
181
182**The layout pass** moves every flat entry the flat naming rules give a host (`<host>` plus a dot or a dash and more) into the instance directory at the same name, and back on a revert (internal/layout/extras.go:14-60, `belongsTo`). An adoption record named `<host>.instance-key-adoption.json` travels with it.
183
184**Relocation's pinner (stage B branch).** `StorePinner.Pin` runs on the region runtime, which holds the control store and the box's secrets key (`RelocationAssemblyConfig.SecretsKey`, from `CARLOS_SECRETS_KEY`). With a nonempty database written by an admitted writer it requires `instance.key` in the pinned manifest (`needKey`, pin.go:250 there). The target renders its held config from the same bucket configuration (`appconfig.RenderHeldUnder` in Task 8c), over the route `relocationRecord` mints.
185
186## Design and rulings
187
188Each ruling ends with its cost if wrong. Rulings 1 to 12 are rev 1's, amended where marked. Rulings 13 to 18 are new in rev 2.
189
190**1. The key lives in `<host>.env`, and is minted on the box.** `<host>.env` is already the platform-managed per-instance secret file, H6 makes it always continuous, moves carry it, and stage B restores it (`EnvItem`). Minting in the console, into appconfig's instance config layer, is rejected: that layer is the app's own editable configuration (`carlos env unset` would delete the key), and it renders into `<host>.config.env`, which sits *under* `<host>.env` in `mergeEnv`. *Cost if wrong:* if Paul wanted the key in the bucket before the box first sees the instance, O1' would be unnecessary. Moving the key later is a copy of the same bytes, not a rotation.
191
192**2. Format: `base64.StdEncoding` of 32 bytes from `crypto/rand`, 44 characters, one line `CARLOS_INSTANCE_KEY=<44 chars>\n`.** Every reader of `<host>.env` yields these bytes exactly: the parser cuts at the first `=` and trims whitespace, which base64 never contains, and the nested launch file passes `KEY=value` strings to execve unchanged (prepare.go:204). *Cost if wrong:* the encoding is fixed by the first mint. Changing it later is a rotation for every adopted app.
193
194**3. Every launch path mints, right after the admin token, under the same lock.** These are `seedChild`, `spawnChildFrom`, `prepareAt` (run and seed roles) and `EnsureClaimAdminToken`, through one helper, `ensureInstanceKeyFor(rt, paths)`. The scheduler tick does not mint: any tick that wakes the instance goes through `Start`. A failed mint fails the wake, as a failed token mint does. *Cost if wrong:* a missed launch path makes an adopting app refuse to start there (`ErrNoInstanceKey`), loudly.
195
196**4. Create-only, append-only, durable every time (amended for finding 2).** A non-empty `CARLOS_INSTANCE_KEY` in `<host>.env`, of any shape, is left exactly as it is: two lines stay two, and readers take the last. The key is appended on a fresh line, never rewritten. Every ensure that reports success, whether it minted or found the key, has done three things on that call: `fchmod 0600` on a descriptor it opened, fsync of the file, and fsync of its directory. A mint whose sync failed returns the error and leaves its line in place. The next ensure finds the key present, keeps its bytes and redoes all three steps. It succeeds only when they succeed, so no process is started with a key a power cut could take back. An existing file is opened with mode 0 (finding 3). *Cost if wrong:* one open, fchmod and two fsyncs per launch, on a file of a few hundred bytes.
197
198**5. A key from the app's configuration wins, and the platform mints none (amended for finding 4; the delivery rule added by the final fix wave).** If the effective env (ruling 13) takes its key from `<host>.config.env`, the mint writes nothing and logs once an hour. Minting would put a key into `<host>.env`, which overrides the configured one, and change the app's key under its data. *Cost if wrong:* an app that sets the name and later unsets it gets a fresh platform key at its next start, by its own action. Reserving the name instead was open question 2; the provisional default is not to.
199
200*The delivery rule (final review I2, Astra's P2).* The files say what this box has delivered, not what the configuration sets: the fast-converge pass reconciles a row ready and syncs its configuration last, so a first wake can run before `<host>.config.env` is rendered, and a key minted then shadows the configured one for good once it arrives. `SyncExec` removes the file for a route that was never configured, so the file's existence cannot be the delivery signal. The mint therefore asks the bucket, as the replica gate does (`ExecBackend.ConfiguredKey`, `appconfig.ConfiguredKeyDigest`), once, only when nothing delivered here sets a key and a mint is next: a key the configuration sets and this box has not delivered defers the launch (`ErrInstanceKeyDeferred`), so does a bucket that cannot be read, and a configuration that sets no key lets the mint go ahead. The read is two-phase (the re-review's finding): outside `adminTokenMu`, the files are read and, if a mint looks next, the bucket is asked; under the lock the files are read again and the decision is that read's alone. A key that appeared in either file meanwhile is present or configured and nothing is minted; a mint goes ahead only when the locked read still finds no key and the bucket was asked and said none; a mint that looks next under the lock without a verdict defers, and the next attempt asks. So a bucket that is slow or down for one row never holds the lock every other row's mint waits on. The launch fails and is retried; the config sync's next pass delivers the file and the next start finds the key configured. A key already delivered, in either file, costs no read. Which boxes have a configuration to read follows the config guard (`configguard.go`): no deployment store under `CARLOS_NO_SEALED_CONFIG` is no configuration source, and the mint proceeds; no store otherwise is an error. *Cost:* a delayed first start while the configuration is in flight (seconds, one pass), and one bucket read per instance, ever. The window the read cannot close is a key configured after the read and before the append, the same in-flight window ruling 27 names for replicas; a key configured after an instance has minted is a rotation, which this plan does not support.
201
202**6. Durability before the first start.**
203- *Legacy boxes:* nothing can publish today, and region HA cannot be on, because it requires host claims. The key's durability is the box's disk, now fsynced, and a move's sealed archive. A dead legacy box loses its keys with its data; that is today's state, unchanged (row 10).
204- *Claim boxes* are region HA's only path. The launch source mints in `Select`. `Materialize` then calls `ClaimFilesManifest.Establish`, and the app executes only after that returns. O1' binds the provider to publish the env file, committed, before it returns, and to start the new generation with no adoption record.
205- *Bucket unreachable at first start:* `Establish` fails, so the wake fails and the app does not start. The key stays on disk unchanged. The next wake publishes the same bytes.
206
207*Cost if wrong:* a C8 provider that ignores O1' reopens the window. O2' is the backstop: without evidence, the host goes `stuck` and is never relocated without its key.
208
209**7. The backfill: the agent at startup, legacy boxes only, existing files only.**
210- *Who and when:* `runAgent`, inside the startup closure that runs after layout recovery has settled every journal, and before the layout pass is published and before `svc.Start` wakes anything (internal/host/agent.go:476-484).
211- *Which rows:* ready, activator-spawned instances (`Kind == "instance"`, a socket, not `Platform`, not unit-backed, no tunnel, not a replica, `Status` empty or `ready`). Held rows (`moving`, `landing`, `restoring`, `releasing`), provisioning rows, sidecars, replicas and TCP-addr routes are skipped.
212- *Existing files only:* a missing `<host>.env` means the host was never started here, and its first start mints. Creating one could occupy the path a landing's `placeStaged` needs empty.
213- *Idempotent:* a present key writes nothing (it is made durable), so every agent start re-runs it harmlessly.
214- *Visible:* it logs one count line, which the rollout reads.
215- *Claim boxes are not backfilled.* The claim's launch source mints under the claim's guard before every cold admission.
216
217*Cost if wrong:* a box whose registry cannot be listed at startup logs it, and its instances get keys at their next start instead.
218
219**8. Never overwrite, on every path:**
220
221| Path | What guarantees it | Test |
222|---|---|---|
223| create (first seed or start) | create-only mint | Task 2 `TestEnsureInstanceKeyMintsOnceInTheAgreedFormat`; Task 3 `TestExecBackendStartHandsTheChildTheMintedInstanceKey`, `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`, `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` (run and seed) |
224| a retry after a failed sync | the present path keeps the bytes and redoes the durability | Task 2 `TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess`; Task 3 `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced` |
225| backfill | same body, existing files only | Task 2 `TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile`; Task 3 `TestBackfillInstanceKeyAppendsToANestedEnvFile`; Task 6 `TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances` |
226| any existing key | create-only, any shape | Task 2 `TestEnsureInstanceKeyNeverRewritesAnExistingKey`, `...LeavesAFileWithTwoKeysAlone` |
227| claim launch, retried | the launch source mints create-only; a session's attempts reuse their inputs | Task 4 `TestClaimLaunchMintsAKeyBesideAnExistingToken`, `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` |
228| re-render (config change) | sync never touches `<host>.env` | Task 7 `TestSyncExecLeavesTheInstanceKeyAlone` |
229| restore (hibernation wake) | `Restore` restores the database only; `Start` mints create-only after it | Task 3 `TestHibernationRestoreKeepsTheInstanceKey` |
230| move | landing installs env only on an empty path; archive carries bytes exactly; env report fails a missing or different key | Task 7 `TestArchiveCarriesTheInstanceKeyByteForByte`, `TestEnvReportComparesTheInstanceKey`; existing `TestLandingNeverDeletesFilesItDidNotCreate` |
231| relocation install | stage B writes `EnvItem`'s bytes; the next launch finds the key present | O3' (stage B branch) |
232| layout move (flat to nested) | `<host>.env` is one of the pass's named artifacts; the adoption record is one of its extras | existing layout pass tests |
233| adoption record | written only for the launch's own writer and delivered key; identical bytes are not rewritten; a new admission starts with none (O1') | Task 5 `TestInstanceKeyAdoptionRecordsTheLaunchesWriter` |
234
235**9. Secrecy.**
236- No log line or error takes the value. The shape warning names the path only, and a test checks the value is not in the log.
237- `<host>.env` is 0600 (ruling 4), as is the adoption record and the nested launch file (prepare.go:204).
238- The adoption record holds `sha256(key)`, never the key. For a platform key it is published beside the env item, which holds the key itself, so it gives a bucket reader nothing new. For a configured key (rev 3, ruling 20) it publishes the digest of a configuration secret: nothing for a random key, a guessing target for a weak one (follow-ups).
239- The console never reads a box's `<host>.env`. `carlos env list` reads the bucket configuration through the console API (appconfig_cli.go:462-507), never `<host>.env`. The moves env report carries HMACs only.
240- Once region HA publishes the env item, the key sits in the data bucket beside the database it seals. That is inherent in making the key recoverable with the data.
241
242*Cost if wrong:* a leak of one instance's key exposes only what that app sealed with it.
243
244**10. Moves and relocation carry the key inside `<host>.env`.**
245- A move archives the file, sealed (archive.go:395), extracts it byte for byte (extract.go:35), and places it only on an empty path (moveslandinstall.go:212-280).
246- If the source never had a key, the target's first start mints one. That is correct: no source process ever held a key.
247- The move's env report classes `CARLOS_INSTANCE_KEY` as `equal`. A target that would start with a different key, or none, is a `differs` or `missing` difference, never a note.
248- A move's probe launches with the env its readiness pin checks against the source's report (internal/host/movesprobeowners.go:106-120). So the probe path never mints.
249- A move does not carry the adoption record (the archive holds the data directory and `<host>.env` only). The target's sessions are new writers and start with none (ruling 20); the app re-reports at its first process start there. The source's record is retired with the move's artifacts (O7).
250- Relocation's install (stage B Task 8c) writes the pinned manifest's `EnvItem` verbatim and checks the pinned key before anything launches (O3'). The next launch's mint finds the key present.
251
252*Cost if wrong:* a key lost in transit is the L21 failure itself. The env report makes it visible on moves.
253
254**11. No switch.** Nothing reads `CARLOS_INSTANCE_KEY` until an app adopts the helper, so writing it is harmless. The new verb answers only a caller that holds the host's admin token. The claim's launch identity, a digest over the env, changes once per host. *Cost if wrong:* the mint is on every launch path, so a bug in it fails wakes across the fleet. The tests, and an error shape identical to the token mint's, are the mitigation. Rollback is reverting the binary. Keys and records already written stay, and are harmless.
255
256**12. Scope: exec-backed activator instances (amended for replicas).** These get no platform key:
257- unit-backed tenants: their env is `/etc/carlos/apps/<host>.env`, root-only and unreadable to the agent (backend_unit.go:174-178);
258- TCP-addr routes: an app supervisor, not the agent, starts them;
259- sidecars and workers: they have their own rows and env files (open question 4, provisional default: a follow-up);
260- replica rows: ruling 26 (only the home's key, never a minted one).
261
262The agent sets `CARLOS_CONTROL_SOCKET` only for instance children (`agentOwnedVars`), so `carlos.Running()` is false for unit tenants, TCP-addr routes, sidecars and workers, and the helper keeps today's behaviour. *Cost if wrong:* an excluded process that runs with `CARLOS_CONTROL_SOCKET` set and adopts the helper refuses to start. That is loud, not silent.
263
264**13. One rule for the effective key (new; finding 4).** `instancekey.Effective(config, own)` is the key a launched app receives: the own file's when non-empty, else the configuration's when non-empty, else none. An empty own value never shadows a non-empty configured one. `mergeEnv` applies it on top of its merge, so the child's env follows the rule; the mint decides by it; the adoption verb checks by it; and the pinner reads the env item with the same parser (`instancekey.ParseEnv`, which `parseEnvBytes` now calls). *Cost if wrong:* a deliberately empty `CARLOS_INSTANCE_KEY=` in `<host>.env` can no longer hide a configured key. Nobody can rely on that today: an empty key makes an adopting app refuse to start.
265
266**14. The adoption record (rev 2; superseded by rulings 20 to 22).** Rev 2's record named only a key, was written for any launch with a control socket, and was not bound to a writer. Rev 3's record, its writer and its publication are rulings 20 to 22.
267
268**15. Relocation's key evidence (rev 2; superseded by ruling 20).** Rev 2 accepted three sources: (a) `instance.key`, (b) an adopted platform key, and (c) a configured key on presence. Rev 3 keeps (a), replaces (b) with the writer-bound (b'), and drops (c).
269
270**16. Claim retries reuse their inputs (new; question 5's premise corrected).** On a claim box every serving start is a claim wake. The launch source, which mints, runs in `Select`. A session's retries reuse the captured inputs and call `Materialize`, and so `Establish`, before every attempt. The legacy spawn path does not serve a claim host, and production claim swaps refuse. No claim attempt can run with a key minted after its files manifest was established. Any future path that mints before an asynchronous publish needs a synchronous publication gate. *Cost if wrong:* `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` fails on the change that breaks it.
271
272**17. Replicas are excluded from platform-key adoption (rev 2; superseded by ruling 26).** Rev 2 deferred a replica's key and minted none; ruling 26 decides it.
273
274**18. The claim early return syncs without writing (new; finding 2).** When the token and the key are both present, the claim launch source writes nothing and needs no authority, as before. It still opens the env file read-only, checks the mode, fsyncs the file and fsyncs its directory before it answers. fsync changes no byte, mode or name, so it is not a host write and runs outside the claim's guard. A file whose mode is not 0600 needs a write, so it takes the guarded path, whose ensure sets it. *Cost if wrong:* if fsync were judged a host write, this step would move under `authorize`, which re-reads the claim on every selection.
275
276**19. Threat model: the adoption record is a trusted-helper assertion (rev 3; Paul's R-A).** The record exists to stop *accidental* loss, such as a relocation that restores a database without the key that sealed it. It is not independent platform evidence. The tenant holds the bearer and the key, the nested record sits in tenant-writable storage, and the helper is the tenant's code. A tenant who forges a record, or reports a key it does not seal with, can lose only its own data. Authentication pins the request to the listener's host. Writer binding (ruling 20) stops a record outliving its writer. Neither proves which key seals the database, and nothing here claims to. *Cost if wrong:* if the record were ever treated as a security boundary between tenants, it would be the wrong tool. It is not used as one.
277
278**20. One evidence rule, two sources (rev 3; Paul's R-B; findings R2-1 to R2-3).** For a nonempty database written by an admitted writer, the pinner (O2') accepts the pinned manifest only with:
279- **(a)** a live `instance.key` entry, unchanged; or
280- **(b')** an adoption record, the platform item `instance-key-adoption.json`, **bound to the writer whose data is pinned**. That writer is the session admitted as the pinned database's generation: the holder's own session when that generation is the holder's, else the matching admission (`pinnedWriter`). All three of these must hold:
281 - the record binds exactly that writer's host, incarnation, session and generation;
282 - its digest is the digest of the key the target will deliver. A `platform` record must match the pinned env item's own key, which the install restores byte for byte. A `config` record needs an env item with no own key, because one would override the configuration at launch;
283 - the key is pinned by digest (`PinObject.InstanceKey`) and verified again at install. The install refuses if the env files it would launch with deliver any other key, for example a configuration changed since the pin (O3', `instancekey.VerifyDelivered`).
284
285Source (c) and `ConfiguredInstanceKey` are gone: a configured key is covered by (b'). **Invalidation:** every new writer admission starts with no record (O1'). A later release that picks its own key (step 1) sends no report, so it has no record and never inherits an earlier session's. A rollback is a new session and is the same case. Without (a) or (b'), relocation refuses (stuck), as before L21. *Cost if wrong:* an app that adopts but whose report never landed for the pinned session goes stuck rather than relocating. That is safe and visible, and the next process start re-reports.
286
287**21. The agent binds the record from its own launch state (rev 3; Paul's R-B).** The helper reports on every process start, at most once per process: the digest of the key it resolved at step 2, and its source, `platform` or `config`. The agent checks both against the launch state it holds for that process (`AdoptionLaunch`). That state is the writer (holder incarnation, session, generation at admission), plus the digest and source of the key the launch env delivered, which `SnapshotClaimLaunch` now records (`InstanceKeySHA256`, `InstanceKeySource`, computed with `instancekey.Effective` from the same two files as the env). It then writes the record: `{"v":2,"var":"CARLOS_INSTANCE_KEY","host","incarnation","session","generation","source","sha256"}`. Nothing in the record comes from the request; a body naming a session is a 400. The verb is mounted only on launches that have a writer session, which means claim launches once they bind a control socket (O6). A legacy launch has no session to bind, so nothing mounts the verb there, and the helper's call is the 404 it ignores. Rev 2 mounted it on legacy launches; that is withdrawn. *Cost if wrong:* if a launch's state were taken from anything but its admission, a record could bind the wrong writer. `AdoptionLaunch` is built once per launch, by O6, from the admission.
288
289**22. Adoption is published on the running session (rev 3; Paul's R-C; finding R2-4).**
290- The verb writes the record beside `<host>.env`: atomically, 0600, file and directory fsynced.
291- It then hands the record to the session's files publisher through `hostfiles.PlatformRelay` and answers 204. 204 means the record is durable on disk; it does not wait on the bucket.
292- The relay stages the record as the platform item (`StagePlatform`), retrying while the store refuses. The publisher's continuous loop (`hostfiles.Run`) commits it within its deadline, as it commits every staged change. The item is always continuous, whatever the app's files durability.
293- An adoption followed by box loss, with no further admission or release, is therefore in the bucket for the pinner.
294
295*Cost if wrong:* a box lost inside the publisher's bound (one deadline, a second by default) after the first adoption loses that report, and the pin refuses until a later session re-reports. That is safe.
296
297**23. O6 is a gate in the HA rollout (rev 3; Paul's R-D).** Claim launches bind no control socket today (claim_snapshot.go:116 strips `CARLOS_CONTROL_SOCKET`). Until O6, an app on the platform key or a configured key cannot report adoption on a claim box. Its relocation refuses: an explicit refusal, not platform-key HA support. O4's checklist says so. *Cost if wrong:* none beyond the refusal, which is the point.
298
299**24. The move lifecycle includes the record (rev 3; Paul's R-E; finding R2-5).** The record is an installation artifact. Nested, it lives in the instance directory and is retired with it. Flat, it must be in moves' journalled artifact lists, so a move away retires it and a later empty-installation census on that box finds nothing of the host's. That is O7, with the exact lists. The writer (O6) does not ship before O7. The census classification does not change: the record stays an unclassified host artifact, so a leftover still refuses, which is the check that keeps O7 honest. *Cost if wrong:* a host that moved away could not be installed on its old box again until an operator removed the file.
300
301**25. A replica is handed no key at all (rev 3.1; superseded by ruling 26).** Rev 3.1 stripped `CARLOS_INSTANCE_KEY` from every replica launch. Astra's round 4 showed that broke replicas that already share their home's configured key, and did not refuse on a launch without a control socket. The strip is reverted.
302
303**26. A replica gets exactly its home's key (rev 4; Paul's decision, 2026-10-11; replaces rulings 17 and 25; Astra round 4).** A replica (#162) mirrors its home's data, which is sealed under the home's effective key. A replica is handed exactly that key or none, and it never mints.
304- **Never minted.** No path mints on a replica row (`InstanceKeyReplica`), and the backfill skips replicas.
305- **The gate, `replicaKeyGate`,** runs on every replica launch before anything starts: Start, Seed, the nested prepare's run and seed roles, and the claim launch source.
306 - If the replica's env files deliver a key, the replica starts only when that key's digest is its home's (`ExecBackend.ReplicaHomeKeys`). A key that differs is refused, with the log line `activator: replica <host> not started: CARLOS_INSTANCE_KEY is not its home's key`. A key that cannot be checked is refused too.
307 - If the env files deliver no key, the replica starts only with a bound control socket. There an app on rastrillo's `InstanceKey` with no key of its own gets `ErrNoInstanceKey` and refuses (step 3), and an app that never asks for a key runs as before. Without a socket (every seed, every claim launch before O6, a flat launch whose bind failed) the helper would fall to step 4 and mint over the home's data. So the agent refuses the launch itself: it does not leave the refusal to the helper.
308- **Where the home's key comes from.** The home's key reaches a replica through O8: a replica record that names its home, and a read of the home's published env item, verified by digest against the home's adoption record or env item. Until O8, nothing can hand a keyless replica its home's key, so it gets none and is gated as above. Verification uses the digests the pre-roll check records in the box's list, `/etc/carlos/replica-home-keys` (`replica-host=<sha256>` entries, read by `replicaHomeKeys` at every launch; ruling 30). A list that does not parse verifies nothing.
309- **No regression on upgrade.** A replica that shares a configured key that is its home's effective key keeps it: the pre-roll check lists it, and the gate passes it. A replica whose configured key differs is refused, and the pre-roll check shows it before any box is upgraded (rollout step 1).
310- **The adoption verb** still answers 409 on a replica: a replica's record would bind a writer whose data is the home's.
311
312*Cost if wrong:*
313- A keyless replica of an app with a seed command cannot seed until O8 or a verified configured key, because seeds have no control socket.
314- A keyless replica on a claim box does not start before O6. Claim launches start nothing before Group D anyway.
315- A flat replica whose control bind fails does not start.
316- A replica whose key the operator never listed is refused after the roll. That is loud and fixed by the list.
317- The list's freshness, and any change to a configured key, are ruling 27 (rev 5). O8's scope is ruling 28.
318
319**27. A configured key is never changed in place (rev 5; Paul's ruling; Astra round 5 finding 1).** Changing a configured `CARLOS_INSTANCE_KEY` on an app that has data is a key rotation, and this plan does not support rotation. `SyncExec` reconciles each box on its own schedule, hourly, so during any configured-key change the home and a replica can briefly hold different keys.
320- **The procedure** (rollout step 1 and the operator note): stop the app's replicas; remove their entries from the box's list and check that the file no longer names them (the agent keeps no copy, ruling 30); change the key; let every box sync; re-run the pre-roll check, which writes the new entries; then start the replicas.
321- **The safety net.** On every replica launch whose key comes from its configuration, the agent checks two things. The key's digest must equal the replica's list entry, which only the pre-roll check writes and nothing refreshes from the replica's configuration. And the bucket's configuration must render that same key now (`ConfiguredKey`, a fresh read on every such launch). Any mismatch, or a read that fails, refuses the launch with a log line naming the host and the reason.
322- **The fresh read is the cheap fresh evidence that works without O8.** A shared configuration change reaches the bucket before any box syncs it, so a replica whose box lags the home is refused.
323
324*Residual (procedure not followed):* a change the bucket does not hold is invisible to the replica's box. Examples are the home's own `<host>.env` edited by hand, or a per-host key configured on the home alone. The replica's list entry and the bucket still agree, so it runs with the old key. A test documents this case; only the procedure, or O8's fresh evidence for platform keys, closes it.
325
326*Cost if wrong:* a bucket read on every launch of a replica whose key is configured. A bucket that cannot be reached refuses those launches, which is safe but costs availability while the bucket is down.
327
328**28. O8 delivers platform keys only (rev 5; Paul's ruling; Astra round 5 finding 2).** O8 hands a keyless replica only a key the platform minted into the home's `<host>.env`, read from the home's published env item and verified against it. A home whose key exists only in its configuration is explicitly deferred: its value is not in any published item, and the adoption record holds only its digest. Its keyless replica keeps refusing (ruling 26), and a replica configured with the same key follows ruling 27. *Cost if wrong:* an app that keys its home by configuration must also configure its replicas with that key (the shared app-level key, listed by the pre-roll check) for them to run.
329
330**29. The replica gate decides on the launch's own read (rev 5.1; Astra round 6 finding 1).** Configuration sync shares no lock with a launch, so the env files can change between the read that composes a child's env and any later read. The source of the key the child is handed is therefore taken from the same read: `instanceEnvLayer` returns `mergeEnv(cfg, own)` and `Effective(cfg, own)`'s layer together. Every gate call passes that layer, and the gate never calls `Effective` on a second read. A non-empty key with no known layer refuses. *Cost if wrong:* none at run time, since it is one read where there were two. If a future caller composed the env some other way and passed a stale layer, the gate would still compare the delivered key's digest with the list and the bucket, and an unknown layer still refuses.
331
332**30. The verified list is read at every replica launch (rev 5.1; Astra round 6 finding 2).**
333- **Where it lives.** The list (still called `CARLOS_REPLICA_HOME_KEYS` in errors and docs) moves from the agent's environment, which a running agent never re-reads, to a file: `/etc/carlos/replica-home-keys`. It holds `replica-host=<sha256>` entries separated by newlines or commas, root-owned and readable by the agent (0640 root:carlos).
334- **How it is written.** The pre-roll check writes it whole, to a temporary file renamed into place, so a reader sees the old list or the new one, never part of one. The agent writes nothing to it.
335- **How it is read.** `replicaHomeKeys` reads the file at every call and keeps no copy. A removed entry stops verifying at the very next replica launch, and a replacement verifies from the next launch on, with no agent restart (which would restart every instance on the box). An absent file verifies nothing. An unreadable or malformed one verifies nothing and is logged without its text.
336
337*Cost if wrong:* one read of a small file per replica launch. A list written by hand without the rename can be read mid-write. A cut inside an entry fails to parse and refuses every keyed replica until the file is whole. A cut after a complete entry still parses, so the entries before the cut keep verifying and those after it are refused until the file is whole. Both outcomes refuse rather than admit a wrong key, and the prescribed write-then-rename avoids either (Astra round 7, P3).
338
339## Obligations on other plans
340
341Rev 3 rewrites O1', O2', O3', O4, O5 and O6, and adds O7. Rev 4 adds O8.
342
343- **O1' (the C8 files-manifest provider, `ClaimFilesManifest`):** `Establish` leaves a committed manifest whose `hostfiles.EnvItem` holds exactly the bytes of `<host>.env` when it runs. A new admission starts with no adoption record (ruling 20): the new generation's manifest carries no `instancekey.AdoptionItem` (`StagePlatformDelete`), and `Establish` removes the local `<host>.instance-key-adoption.json` under the start's captured work. It never writes `<host>.env`. Any error (store unreachable, `If-Match` lost) is returned, and the wake fails before the app executes. Tests in the provider's plan, run against the provider itself rather than a fake:
344 - `TestEstablishPublishesTheEnvFileWithItsInstanceKey`: a dir store, a predecessor manifest whose env item lacks the key and which carries an adoption item, and a local `<host>.env` with the key plus a local record. After `Establish`, `ReadPlatform(EnvItem)` equals the local bytes, the new manifest has no adoption item, the local record is gone, and the manifest is the committed revision.
345 - `TestEstablishFailsTheWakeWhenThePublishFails`: a store that refuses the PUT. `Establish` errors, and through `claimServingOwners.Materialize` and `startAdmitted` no start is reserved.
346- **O2' (stage B, `hostrelocate/pin.go` on `region-ha-1b-next-build`):** the pinner's check for a nonempty database is **(a) or (b')** (ruling 20). It replaces rev 2's O2' and stage B's own rev 3 variant: ruling 55, its `StorePinner.KeyEvidence` reader and its `KeyQuestion`, `KeySource` and `envHasInstanceKey`. The evidence is in the pinned manifest, so the pinner needs no outside reader and no wiring. The diff and the test are below ("O2' for stage B"). Stage B's ruling 55 becomes: "The instance key is evidenced by (a) `instance.key` or (b') an adoption record bound to the pinned writer for the key the target delivers, pinned by digest (instance-key plan rev 3, ruling 20). The env item is parsed with `instancekey.ParseEnv`."
347- **O3' (stage B Task 8c, relocation install):**
348 - The install writes the pinned manifest's `EnvItem` byte for byte with `writeFileSynced`, or removes `<host>.env` when the item is absent. It never mints.
349 - It does not restore the adoption item: the target's sessions are new writers, which start with no record (ruling 20).
350 - When the pin carries `InstanceKey`, then after the env file and the held render (`RenderHeldUnder`) are on disk and before anything launches, it reads both files and calls `instancekey.VerifyDelivered(config, env, pin.InstanceKey.SHA256)`. On an error it refuses (the operation goes stuck) and starts nothing.
351 - Tests: `TestRelocationInstallKeepsThePublishedEnvByteForByte` gains "after `Install`, `activator.EnsureInstanceKeyBeneath(dir, host+".env")` returns `InstanceKeyPresent` and leaves the bytes unchanged". New `TestRelocationInstallDeliversThePinnedKey` has three cases: a pinned platform key installs; a pinned configured key whose configuration is unchanged installs; a configuration changed between the pin and the install refuses with nothing launched.
352- **O4 (stage B Task 10 checklist item 9)** becomes: "L21. The instance-key plan is rolled to the region's boxes, and O1', O2', O3', O5 and O7 are in. **O6 gates platform-key and configured-key relocation:** until claim launches bind a control socket and mount the adoption verb, an app whose key is not its own `instance.key` cannot report adoption on a claim box, and its relocation refuses (stuck). That is an explicit refusal, not HA support for those apps. Apps must also run a rastrillo release with the adoption call (#66 ask and addendum)."
353- **O5 (stage B's release flush, `internal/host/relocationflush.go` on that branch):** stages the local adoption record as `AdoptionItem` when it parses and binds the releasing writer (the holder's incarnation, session and generation); otherwise it stages `StagePlatformDelete`. Both go in the same commit as the env item. Test: a flush with the releasing writer's record publishes it; with a stale record, or none, publishes none.
354- **O6 (claim Plan 9 Group D, the isolated-root launcher), the gate of ruling 23:**
355 - When claim launches bind a control socket, mount `ExecBackend.MountInstanceKeyAdoption` on it.
356 - Thread `ClaimLaunchSnapshot.InstanceKeySHA256` and `InstanceKeySource` through `ClaimServingLaunch` and `ClaimLaunchInputs`. Today `claimServingLaunchSource` (claimproduction.go) drops both; the snapshot computes them and nothing carries them further.
357 - Its `launch` returns the process's `AdoptionLaunch`: `Writer` from the admission (holder incarnation, session and generation as admitted); `KeySHA256` and `Source` from the threaded fields; and the instance directory. **It must reflect the current admission:** the verb binds whatever `launch()` returns at the moment of the request, so once the session has ended or been superseded `launch` returns `false` (a 404), never the old session's writer. A late helper report after a re-admission then writes nothing. The verb refuses a writer binding the pinner could not read (500), but it cannot tell a stale complete one from a current one; that is the callback's lifetime, which O6's review checks against the session's.
358 - `Publish` is a `hostfiles.PlatformRelay` over the session's files publisher. O6 runs **both** `hostfiles.Run` and `relay.Run` on the session's publisher: the relay only stages, and a staged item nobody commits never reaches the bucket (`TestAdoptionSurvivesBoxLossWithoutAnotherAdmissionOrRelease` is the shape).
359 - The record's write holds `adminTokenMu` across its read and two fsyncs (once per helper report, once per process start); on a slow disk that stalls every token mint and key ensure on the box for that long. Known and accepted; O6's review should know the cost.
360 - The record's write runs under the claim's guard (a hostwrite selection for the host, as the token's mint does).
361 - The launch env gains the agent-owned `CARLOS_INSTANCE_KEY_SOURCE` (`platform` or `config`, from `InstanceKeySource`), set only when the socket is bound. It is stripped from tenant files like `CARLOS_CONTROL_SOCKET` (`agentOwnedVars`) and added to appconfig's `agentOwned` list.
362 - `SnapshotClaimLaunch`'s replica gate (ruling 26) then takes the bound socket as `haveControl`, so a keyless replica runs and its helper refuses at step 3.
363 - **O7 ships before O6; O6 does not ship before O7** (ruling 24). Until O6, ruling 23's refusal stands.
364- **O7 (moves, the journalled artifact lifecycle; ruling 24).** These lists change:
365 - `installArtifactPaths` (internal/host/movesinstall.go:751-754): append `host + instancekey.AdoptionSuffix` after `host + ".env"`. It flows into the dirty return's pre-move park (movesinstall.go:852), into `sourceArtifactPaths` (movesfinal.go:586-587) for the source finalization's park (movesfinal.go:527), and into the rollback's candidates (movesrollback.go:1314).
366 - The landing's flat inventory (movesinstall.go:664): `paths` gains `host + instancekey.AdoptionSuffix`. The installation's inventory then claims the path its app's sessions will write.
367 - `rollbackOwnership.owns` (movesrollback.go:1449-1470): a case for the record. It is the target's when it parses (`instancekey.ParseAdoption`) and binds the move's incarnation. Anything else is not ours.
368 - The census classification does not change (ruling 24). A nested record lives in the instance directory and is retired with it.
369 - Journals: the lists are journalled in order, and a journal written before O7 keeps its recorded list. No record exists anywhere before O6 ships, so an older journal cannot miss one.
370 - Tests (in the moves change): `TestMoveAwayRetiresTheAdoptionRecordSoTheHostCanReturn`. A flat source holds a record; the move completes; the source finalization parks the record; `EmptyInstallationCensus.CheckLocal` for a reinstall of the host on the source box passes; a move back lands. Also the dirty-return variant: a return over a source that still holds a stale record parks it before placing anything.
371
372- **O8 (replica placements, #162; rulings 26 and 28).** It hands a keyless replica its home's **platform-minted** key, and only that. Today's fallback: a replica that cannot be handed its home's key gets none, and refuses to start reliably (`replicaKeyGate`).
373 - The replica record names its home. `provision.Replica` and the console's create door record the home's host (and so its incarnation, through the home's claim) in an immutable field. Today a replica record names only the app, account, region and role.
374 - The replica's box reads the home's published `EnvItem` from the home's files manifest, in the home region's data bucket (O1' publishes it). That needs read access for the replica's region to the home region's bucket, which no box has today.
375 - It takes the env item's own key, the one the platform minted, and verifies it by digest against the home's adoption record (`instancekey.Binds` to the home's current writer) or the env item itself.
376 - It writes the key, create-only, into the replica's own `<host>.env`, and supplies `ExecBackend.ReplicaHomeKeys` from the same read, replacing `CARLOS_REPLICA_HOME_KEYS` for that replica.
377 - **A home whose key exists only in configuration is out of O8's scope (ruling 28).** Its value is in no published item. Its keyless replica keeps refusing, and a configured-key replica follows ruling 27.
378 - Test, in O8's plan: a keyless replica of an adopting app whose home holds a platform key starts with exactly that key, and its helper resolves it. A home whose env item is unreadable, unverifiable or keyless leaves the replica refused.
379
380### O2' for stage B
381
382Apply to `internal/hostrelocate/pin.go` on `region-ha-1b-next-build` after merging main, which brings `internal/instancekey`. It goes in place of Task 8b step 0's diff, and that step's `KeyEvidence` reader goes with it. It was verified in a scratch copy of `origin/region-ha-1b-next-build` (919662d2) with `internal/instancekey` added: `go build ./internal/...` and `go vet` clean, and `go test ./internal/hostrelocate/ -count=1` passes, every existing pinner test included.
383
384```diff
385--- a/internal/hostrelocate/pin.go
386+++ b/internal/hostrelocate/pin.go
387@@ -15,6 +15,7 @@
388 "github.com/carlosframework/platform/internal/genselect"
389 "github.com/carlosframework/platform/internal/hostclaim"
390 "github.com/carlosframework/platform/internal/hostfiles"
391+ "github.com/carlosframework/platform/internal/instancekey"
392 "github.com/carlosframework/platform/internal/release"
393 )
394
395@@ -37,6 +38,18 @@
396 // Release is the app release version the relocation installs, read
397 // at pinning from StorePinner.Release (R9); a reuse carries it.
398 Release string `json:"release"`
399+ // InstanceKey is the key a nonempty database is sealed with when the
400+ // pinned manifest evidences it by an adoption record rather than by
401+ // instance.key (O2', instance-key plan rev 3): its digest and source.
402+ // The install delivers exactly this key or refuses (O3').
403+ InstanceKey *KeyPin `json:"instance_key,omitempty"`
404+}
405+
406+// KeyPin is a pinned instance key: the SHA-256 of the key and the file
407+// that supplies it at launch (instancekey.SourcePlatform or SourceConfig).
408+type KeyPin struct {
409+ Source string `json:"source"`
410+ SHA256 string `json:"sha256"`
411 }
412
413 // FilesPin names one generation's files manifest by prefix, sequence and
414@@ -142,11 +155,15 @@
415 // that writer's instance.key: the pinned manifest must carry it (rev
416 // 18, round 17 finding 1).
417 needKey := c.Holder.Session != 0 && obj.DB != nil && obj.DB.MaxTXID > 0
418- files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0, needKey)
419+ // The writer whose data is pinned: the session admitted as the pinned
420+ // database's generation (O2', instance-key plan ruling 20). Asked only
421+ // when instance.key is missing.
422+ writer := func() (instancekey.Binding, error) { return pinnedWriter(c, obj.DB.Generation) }
423+ files, keyPin, retained, err := p.filesPin(ctx, writer, c.Holder, released && rel.ManifestSeq == 0, needKey)
424 if err != nil {
425 return hostclaim.RecoveryPin{}, err
426 }
427- obj.Files = files
428+ obj.Files, obj.InstanceKey = files, keyPin
429 if released && obj.Files.Seq < rel.ManifestSeq {
430 return hostclaim.RecoveryPin{}, fmt.Errorf("%w: the files manifest is at %d, the release at %d", ErrNotYet, obj.Files.Seq, rel.ManifestSeq)
431 }
432@@ -221,7 +238,7 @@
433 // stays restorable at g10. Every referenced object must lie under this
434 // holder's location (its incarnation) in this region's data bucket, at
435 // files/<generation>/objects/; any other reference refuses the pin.
436-func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, []byte, error) {
437+func (p StorePinner) filesPin(ctx context.Context, writer func() (instancekey.Binding, error), h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, *KeyPin, []byte, error) {
438 chain := []string{h.Generation}
439 for i := len(h.Admissions) - 1; i >= 0; i-- {
440 chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor)
441@@ -241,24 +258,33 @@
442 continue
443 }
444 if err != nil {
445- return FilesPin{}, nil, err
446+ return FilesPin{}, nil, nil, err
447 }
448 var m hostfiles.Manifest // decoded from the bytes the digest names
449 if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion {
450- return FilesPin{}, nil, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err)
451- }
452- if needKey && !liveEntry(m, InstanceKeyPath) {
453- // The manifest was copied before the app started and the
454- // key it minted afterwards never published (§5.6's
455- // continuous-sync window, L21): installing this set would mint
456- // a new key over data sealed with the old one.
457- return FilesPin{}, nil, fmt.Errorf("%w: %s's pinned manifest has no %s for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath)
458+ return FilesPin{}, nil, nil, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err)
459 }
460 if origins, err = referencedOrigins(h.Location, origins, m); err != nil {
461- return FilesPin{}, nil, err
462+ return FilesPin{}, nil, nil, err
463+ }
464+ var keyPin *KeyPin
465+ if needKey && !liveEntry(m, InstanceKeyPath) {
466+ // (a) failed: no instance.key. (b'): an adoption record bound
467+ // to the pinned writer, for the key the target will deliver
468+ // (O2', instance-key plan rev 3). Without it the manifest may
469+ // have been copied before the app minted a key of its own
470+ // (§5.6's continuous-sync window), and installing this set
471+ // would mint a new key over data sealed with the old one.
472+ w, err := writer()
473+ if err != nil {
474+ return FilesPin{}, nil, nil, err
475+ }
476+ if keyPin, err = p.adoptedKey(ctx, w, prefix, m, origins); err != nil {
477+ return FilesPin{}, nil, nil, err
478+ }
479 }
480 sum := sha256.Sum256(raw)
481- return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:]), Origins: origins}, raw, nil
482+ return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:]), Origins: origins}, keyPin, raw, nil
483 }
484 // No manifest anywhere in the chain. An empty files pin needs the
485 // same kind of evidence an empty database pin does (ruling 15): no
486@@ -272,14 +298,14 @@
487 // 17, finding F2), never an install that mints a fresh key over data
488 // sealed with the old one.
489 if h.Session != 0 && !releasedEmpty {
490- return FilesPin{}, nil, fmt.Errorf("%w: %s has admitted a writer but no files manifest exists in its chain", ErrFilesUnevidenced, h.Location)
491+ return FilesPin{}, nil, nil, fmt.Errorf("%w: %s has admitted a writer but no files manifest exists in its chain", ErrFilesUnevidenced, h.Location)
492 }
493 gen := h.Generation
494 if gen == "" {
495 gen = hostclaim.RootGeneration
496 }
497 prefix := h.Location + "/files/" + gen + "/"
498- return FilesPin{Prefix: prefix, Origins: []string{prefix}}, nil, nil
499+ return FilesPin{Prefix: prefix, Origins: []string{prefix}}, nil, nil, nil
500 }
501
502 // referencedOrigins adds to origins the files prefix of every generation
503@@ -395,3 +421,72 @@
504 }
505 return pinned, nil
506 }
507+
508+// pinnedWriter is the writer whose data a pin of generation gen holds: the
509+// holder's own session when gen is the holder's generation, else the
510+// admitted session recorded for gen. A generation no admission names has
511+// no writer to bind evidence to, and refuses.
512+func pinnedWriter(c *hostclaim.Claim, gen string) (instancekey.Binding, error) {
513+ h := c.Holder
514+ w := instancekey.Binding{Host: c.Host, Incarnation: h.Incarnation, Generation: gen}
515+ if gen == h.Generation {
516+ w.Session = h.Session
517+ }
518+ for i := len(h.Admissions) - 1; w.Session == 0 && i >= 0; i-- {
519+ if h.Admissions[i].Generation == gen {
520+ w.Session = h.Admissions[i].Session
521+ }
522+ }
523+ if w.Session == 0 || w.Incarnation == "" || gen == "" {
524+ return instancekey.Binding{}, fmt.Errorf("%w: no admitted writer for generation %q", ErrFilesUnevidenced, gen)
525+ }
526+ return w, nil
527+}
528+
529+// adoptedKey is O2' source (b') (instance-key plan rev 3, ruling 20): the
530+// pinned manifest's adoption record must bind exactly the pinned writer,
531+// and name the key the target will deliver. A platform-sourced record must
532+// match the env item's own key, which the install restores byte for byte.
533+// A config-sourced record needs an env item with no own key (one would
534+// override the configuration at launch); its digest is pinned, and the
535+// install verifies the configuration it renders still delivers it (O3').
536+// The env item is parsed with instancekey.ParseEnv, the activator's own
537+// rule. A named item that cannot be read is an error, never "no evidence".
538+func (p StorePinner) adoptedKey(ctx context.Context, writer instancekey.Binding, prefix string, m hostfiles.Manifest, origins []string) (*KeyPin, error) {
539+ read := func(name string) ([]byte, bool, error) {
540+ if e, ok := m.Platform[name]; !ok || e.Deleted {
541+ return nil, false, nil
542+ }
543+ raw, err := hostfiles.ReadPlatform(ctx, p.Data, prefix, m, name, origins)
544+ return raw, err == nil, err
545+ }
546+ rec, ok, err := read(instancekey.AdoptionItem)
547+ if err != nil {
548+ return nil, err
549+ }
550+ refuse := func(why string) (*KeyPin, error) {
551+ return nil, fmt.Errorf("%w: %s's pinned manifest has no %s and %s, for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath, why)
552+ }
553+ if !ok {
554+ return refuse("no adoption record")
555+ }
556+ a, err := instancekey.ParseAdoption(rec)
557+ if err != nil {
558+ return refuse("its adoption record is malformed")
559+ }
560+ if !a.Binds(writer) {
561+ return refuse("its adoption record is not the pinned writer's")
562+ }
563+ env, _, err := read(hostfiles.EnvItem)
564+ if err != nil {
565+ return nil, err
566+ }
567+ own, _ := instancekey.Effective(nil, instancekey.ParseEnv(env))
568+ switch {
569+ case a.Source == instancekey.SourcePlatform && own != "" && instancekey.Digest(own) == a.SHA256:
570+ case a.Source == instancekey.SourceConfig && own == "":
571+ default:
572+ return refuse("its adoption record does not name the key the env delivers")
573+ }
574+ return &KeyPin{Source: a.Source, SHA256: a.SHA256}, nil
575+}
576```
577
578`internal/hostrelocate/pin_envkey_test.go` (in place of stage B's `TestStorePinnerNeedsPositiveEvidenceForAnEnvKey`):
579
580```go
581package hostrelocate
582
583import (
584 "context"
585 "crypto/sha256"
586 "encoding/hex"
587 "encoding/json"
588 "errors"
589 "fmt"
590 "testing"
591 "time"
592
593 "github.com/carlosframework/platform/internal/hostclaim"
594 "github.com/carlosframework/platform/internal/hostfiles"
595 "github.com/carlosframework/platform/internal/instancekey"
596 "github.com/carlosframework/platform/internal/release"
597)
598
599// O2' (instance-key plan rev 3, replacing stage B's O2 and ruling 55's
600// KeyEvidence reader): a nonempty database's pinned manifest evidences its
601// key by (a) instance.key, or (b') an adoption record bound to the pinned
602// writer for the key the target will deliver. Presence of a key, in the env
603// or in the configuration, is never evidence (Astra rounds 1 and 2).
604func TestStorePinnerKeyEvidence(t *testing.T) {
605 ctx := context.Background()
606 at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)
607 const key = "q+/Zk3vJ0aW1Yx9sR7tUe2oPl4mN8bC6dF5gH0iJ1kA="
608 writer := instancekey.Binding{Host: host, Incarnation: "inc-1", Session: 3, Generation: "g3"}
609 earlier := instancekey.Binding{Host: host, Incarnation: "inc-1", Session: 2, Generation: "g2"}
610 envWithKey := "CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=" + key + "\n"
611 rec := func(b instancekey.Binding, source, k string) string {
612 return string(instancekey.AdoptionRecord(b, source, instancekey.Digest(k)))
613 }
614 for _, tc := range []struct {
615 name string
616 appKey bool // (a) instance.key in the manifest
617 env, adoption string // "" is no item
618 unreadable string // a named item whose object is missing
619 holder func(*hostclaim.Holder)
620 want error
621 pinned *KeyPin
622 }{
623 {name: "(a) the app's own instance.key", appKey: true, env: envWithKey, want: nil},
624 {name: "(b') the platform key, adopted by the pinned writer", env: envWithKey, adoption: rec(writer, "platform", key), pinned: &KeyPin{Source: "platform", SHA256: instancekey.Digest(key)}},
625 {name: "(b') a configured key, adopted by the pinned writer", env: "CARLOS_ADMIN_TOKEN=t\n", adoption: rec(writer, "config", "configured-A"), pinned: &KeyPin{Source: "config", SHA256: instancekey.Digest("configured-A")}},
626 {name: "(b') a configured key under an empty override", env: "CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=\n", adoption: rec(writer, "config", "configured-A"), pinned: &KeyPin{Source: "config", SHA256: instancekey.Digest("configured-A")}},
627 {name: "a backed-up platform key, the app's own key missing", env: envWithKey, want: ErrFilesUnevidenced},
628 {name: "configured A, the app's own unpublished B: no report, no record", env: "CARLOS_ADMIN_TOKEN=t\n", want: ErrFilesUnevidenced},
629 {name: "a stale record from a previous release", env: envWithKey, adoption: rec(earlier, "platform", key), want: ErrFilesUnevidenced},
630 {name: "a rollback: a new session, the old session's record", env: envWithKey, adoption: rec(writer, "platform", key),
631 holder: func(h *hostclaim.Holder) {
632 h.Session, h.Generation = 4, "g4"
633 h.Admissions = append(h.Admissions, hostclaim.AdmissionRecord{Generation: "g4", Predecessor: "g3", Session: 4})
634 }, want: ErrFilesUnevidenced},
635 {name: "a record from another installation", env: envWithKey, adoption: rec(instancekey.Binding{Host: host, Incarnation: "inc-0", Session: 3, Generation: "g3"}, "platform", key), want: ErrFilesUnevidenced},
636 {name: "a platform record for another key", env: envWithKey, adoption: rec(writer, "platform", "another-key"), want: ErrFilesUnevidenced},
637 {name: "a config record while the env's own key overrides it", env: envWithKey, adoption: rec(writer, "config", key), want: ErrFilesUnevidenced},
638 {name: "a platform record without the platform key", env: "CARLOS_ADMIN_TOKEN=t\n", adoption: rec(writer, "platform", key), want: ErrFilesUnevidenced},
639 {name: "an exported key line is not what the app receives", env: "export CARLOS_INSTANCE_KEY=" + key + "\n", adoption: rec(writer, "platform", key), want: ErrFilesUnevidenced},
640 {name: "a malformed record", env: envWithKey, adoption: `{"v":1}`, want: ErrFilesUnevidenced},
641 {name: "no admitted writer for the pinned generation", env: envWithKey, adoption: rec(writer, "platform", key), holder: func(h *hostclaim.Holder) { h.Incarnation = "" }, want: ErrFilesUnevidenced},
642 {name: "a named adoption item that cannot be read", env: envWithKey, unreadable: instancekey.AdoptionItem, want: errUnreadable},
643 {name: "a named env item that cannot be read", adoption: rec(writer, "config", "configured-A"), unreadable: hostfiles.EnvItem, want: errUnreadable},
644 } {
645 t.Run(tc.name, func(t *testing.T) {
646 claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
647 obs := pinClaim()
648 obs.Claim.Holder.Incarnation = "inc-1"
649 gen := "g3"
650 if tc.holder != nil {
651 tc.holder(obs.Claim.Holder)
652 gen = obs.Claim.Holder.Generation
653 }
654 m := hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: 4, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}}
655 if tc.appKey {
656 m = keyed(m)
657 }
658 item := func(name, body string, put bool) {
659 sum := sha256.Sum256([]byte(body))
660 sha := hex.EncodeToString(sum[:])
661 obj := pinLocation + "/files/" + gen + "/objects/" + fmt.Sprintf("%016d-%s", 4, sha)
662 if put {
663 if _, err := data.Put(ctx, obj, []byte(body), ""); err != nil {
664 t.Fatal(err)
665 }
666 }
667 m.Platform[name] = hostfiles.Entry{Object: obj, SHA256: sha, Size: int64(len(body)), Mode: 0o600, Seq: 4}
668 }
669 if tc.env != "" {
670 item(hostfiles.EnvItem, tc.env, true)
671 }
672 if tc.adoption != "" {
673 item(instancekey.AdoptionItem, tc.adoption, true)
674 }
675 if tc.unreadable != "" {
676 item(tc.unreadable, "never stored\n", false)
677 }
678 putManifest(t, data, gen, m)
679 p := StorePinner{Claims: claims, Data: data, Release: routeRelease, Bucket: "data", Region: "ie", SelectDB: selectAt(41)}
680 ref, err := p.Pin(ctx, obs, at)
681 switch {
682 case tc.want == errUnreadable:
683 if err == nil || errors.Is(err, ErrFilesUnevidenced) {
684 t.Fatalf("got %v, want a read error, never a refusal for no evidence", err)
685 }
686 return
687 case !errors.Is(err, tc.want):
688 t.Fatalf("got %v, want %v", err, tc.want)
689 }
690 if err != nil {
691 return
692 }
693 raw, _, err := claims.Get(ctx, ref.Key)
694 if err != nil {
695 t.Fatal(err)
696 }
697 var obj PinObject
698 if err := json.Unmarshal(raw, &obj); err != nil {
699 t.Fatal(err)
700 }
701 if (obj.InstanceKey == nil) != (tc.pinned == nil) || (tc.pinned != nil && *obj.InstanceKey != *tc.pinned) {
702 t.Fatalf("pinned key %+v, want %+v", obj.InstanceKey, tc.pinned)
703 }
704 })
705 }
706}
707
708var errUnreadable = errors.New("sentinel: any read error")
709```
710
711## Ask for rastrillo #66
712
713*For Paul to post on rastrillo/rastrillo discussion #66. Rev 2's ask is below, updated; the rev 3 addendum follows it.*
714
715**The ask.** When `carlos.InstanceKey(own)` returns `$CARLOS_INSTANCE_KEY` (step 2), have it tell the agent so, on every process start, so the platform can record that this app's current writer seals with that key. Failure is non-fatal to the app.
716
717**Why.** Region HA relocates an instance's data to another box. It may hand the app the delivered key there only if that key is the one the data is sealed with. A key being present proves nothing: an app with its own key keeps it (step 1). So the platform needs positive word, and only the app's helper knows which step it took.
718
719**The call.**
720- `PUT http://carlos/instance-key/adoption` over `$CARLOS_CONTROL_SOCKET`, the same transport and placeholder host `ScheduleAt` uses.
721- Header `Authorization: Bearer $CARLOS_ADMIN_TOKEN`; `Content-Type: application/json`.
722- Body `{"sha256":"<hex>","source":"<platform|config>"}`. `sha256` is the lowercase hex SHA-256 of the key string exactly as resolved; never send the key. `source` is `$CARLOS_INSTANCE_KEY_SOURCE` verbatim, which the platform sets beside the key.
723- When: inside `InstanceKey`, only on the step 2 return, only when `Running()` and the token are set, at most once per process (a `sync.Once`), on every process start.
724- How long: synchronous, before `InstanceKey` returns, so the record is on disk before the app seals anything. It is bounded by a short timeout of its own (2 seconds suggested), since `InstanceKey` takes no context.
725- Answers: 204 means recorded. 404 or 405 means an agent or a launch without the verb: ignore it silently. 409 means the agent's launch delivered another key or source, or the host is a replica: worth one log line. 400, 401, 5xx, a timeout or a dial error: one log line at most. Never retry in a loop, and never log the key.
726- Whatever the answer, `InstanceKey` returns the key and a nil error.
727
728**Not called** at step 1 (the app's own key), step 3 or step 4, or with no `$CARLOS_ADMIN_TOKEN`.
729
730**What changes for apps.** Nothing visible. An app on a rastrillo release without the call still runs. Its data does not relocate (it is held as `stuck`) until it runs a release with the call.
731
732### Addendum (rev 3), for Paul to post
733
734Two changes to the ask above, and only the second is new:
7351. **Every process start.** The helper reports once per process, on every process start, not once per agent or per deployment. The platform binds each report to the writer session that started the process, and a new session starts with no record.
7362. **New: `source`.** The body adds `"source": "platform"` or `"source": "config"`, copied verbatim from `$CARLOS_INSTANCE_KEY_SOURCE`, which the platform sets beside `$CARLOS_INSTANCE_KEY`. Both keys arrive as the same variable, so the helper cannot tell them apart without it. If `$CARLOS_INSTANCE_KEY_SOURCE` is unset, skip the call: that agent cannot record adoption.
737
738## Review Focus
739
7401. **O2''s evidence table.** Astra's configured-A, own-B case refuses. So do a stale record from a previous release, a rollback, another installation, and a record naming a key the env does not deliver. The pinned key is in the pin object.
7412. **The install's last check (O3').** A configuration changed between the pin and the install refuses (`TestVerifyDeliveredRefusesAnyOtherKey`).
7423. **The verb takes nothing from the request but the digest and the source.** A session in the body is a 400. The record binds the launch's writer. 204 means durable on disk (Task 5).
7434. **Adoption, then box loss.** The record is in the bucket with no admission or release after it (`TestAdoptionSurvivesBoxLossWithoutAnotherAdmissionOrRelease`).
7445. **A failed sync, then a retry.** The retry keeps the bytes, redoes the fchmod and both syncs, and only then succeeds; no child starts in between (Tasks 2 to 4).
7456. **An empty `CARLOS_INSTANCE_KEY=` in `<host>.env` with a configured key.** The child gets the configured key and nothing is minted (Task 3).
7467. **A replica without a control socket.** A keyless replica is refused by the agent, before its app could mint; a replica whose key is not its home's is refused; its home's key runs (Task 3b, through the helper's real resolution).
7478. **A configured key in flight (ruling 27).** The bucket says B while the replica's box still holds A, and the replica is refused. The documented residual, a change only on the home, runs with A (Task 3b).
7489. **A sync between the launch's read and the gate (ruling 29),** and **a list change while the agent runs (ruling 30).** The gate decides on the launch's own read, and the list is read at every launch (Task 3b).
749
750## File structure
751
752- Create `internal/instancekey/instancekey.go` and `instancekey_test.go`. They hold the parser, `Effective`, `Digest`, the sources, the writer-bound adoption record (`Binding`, `Adoption`, `AdoptionRecord`, `ParseAdoption`, `Binds`) and `VerifyDelivered`. Standard library only.
753- Modify `internal/activator/admintoken.go` (`parseEnvBytes` calls `instancekey.ParseEnv`) and `internal/activator/backend_exec.go` (`mergeEnv`, `LaunchEnvironment`'s shadow list).
754- Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `ensureInstanceKeyFor`, `BackfillInstanceKey`, `settleClaimInstanceEnv`, `InstanceKeyVar`, `InstanceKeyOutcome`).
755- Create `internal/activator/instancekey_test.go`, `instancekey_linux_test.go`, `replica_instancekey_test.go`, `replica_instancekey_linux_test.go`, `claim_instancekey_test.go` and `claimretry_instancekey_linux_test.go`.
756- Create `internal/activator/replicakey.go` (`replicaKeyGate`, `ErrReplicaKey`), `internal/host/replicahomekeys.go` (`replicaHomeKeys`, `configuredKey`) and `internal/appconfig/instancekey.go` (`ConfiguredKeyDigest`), with `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` and `internal/activator/replicakey_race_test.go`. Modify `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ConfiguredKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` and `internal/host/agent.go` (Task 3b).
757- Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369), `internal/activator/prepare.go` (:60) and `internal/activator/claim_snapshot.go` (the snapshot's fields, `EnsureClaimAdminToken` :40-67, `SnapshotClaimLaunch`'s env read).
758- Create `internal/activator/keyadoption.go`, `keyadoption_test.go` and `keyadoption_linux_test.go`: the adoption verb, mounted by O6.
759- Create `internal/hostfiles/relay.go` and `relay_test.go`: `PlatformRelay`.
760- Modify `internal/host/claimowners.go` (:125-138), for O1' on `ClaimFilesManifest`. Create `internal/host/claim_instancekey_test.go`, `claim_replica_instancekey_test.go` and `instancekey_adoption_test.go`.
761- Create `internal/host/instancekeys.go` and `instancekeys_test.go`: the backfill. Modify `internal/host/agent.go` (:476-484).
762- Create `internal/appconfig/syncexec_instancekey_test.go`, `internal/moves/envreport_instancekey_test.go` and `internal/moves/archive/instancekey_test.go`.
763
764**Sandbox note for every command below:**
765- If the toolchain reports "read-only file system" for the Go cache, prefix commands with `GOCACHE=$TMPDIR/gocache`. Re-export it per command, and never delete that shared directory. In a worktree, also set `GOFLAGS=-buildvcs=false`.
766- On a shared machine, run commands under `nice -n 19`.
767- The sandbox refuses unix sockets ("socket: operation not permitted"). Tests that start a child or prepare a nested role must run with the sandbox off.
768- `go test ./internal/host/ -run Claim` takes longer than go test's default 10-minute timeout; pass `-timeout 30m`.
769
770---
771
772
773### Task 1: The shared rule
774
775**Files:**
776- Create: `internal/instancekey/instancekey.go`, `internal/instancekey/instancekey_test.go`
777- Modify: `internal/activator/admintoken.go:245-257` (`parseEnvBytes`), `internal/activator/backend_exec.go:1879-1905` (`LaunchEnvironment`), `:1980-1990` (`mergeEnv`)
778
779**Interfaces:**
780- Produces: `instancekey.Var`, `AdoptionSuffix`, `AdoptionItem`, `ParseEnv`, `Layer` (`LayerNone`, `LayerOwn`, `LayerConfig`), `Effective`, `Digest`, `SourcePlatform`, `SourceConfig`, `SourceOf`, `Binding`, `Adoption`, `AdoptionRecord`, `ParseAdoption`, `(Adoption).Binds`, `VerifyDelivered`, `ValidDigest`, `ParseHomeKeys` (ruling 26, for Task 3b).
781
782- [ ] **Step 1: Write the failing tests**
783
784Create `internal/instancekey/instancekey_test.go`:
785
786```go
787package instancekey
788
789import (
790 "strings"
791 "testing"
792)
793
794// Finding 4: an empty own value never shadows a configured key; a
795// non-empty own value always wins.
796func TestEffectiveNeverHandsOutAnEmptyKeyWhileOneExists(t *testing.T) {
797 for _, tc := range []struct {
798 name string
799 config, own string
800 want string
801 layer Layer
802 }{
803 {"own wins", Var + "=cfg\n", Var + "=own\n", "own", LayerOwn},
804 {"empty own, configured", Var + "=cfg\n", Var + "=\n", "cfg", LayerConfig},
805 {"no own, configured", Var + "=cfg\n", "A=1\n", "cfg", LayerConfig},
806 {"own only", "", Var + "=own\n", "own", LayerOwn},
807 {"both empty", Var + "=\n", Var + "=\n", "", LayerNone},
808 {"exported is not the key", "", "export " + Var + "=own\n", "", LayerNone},
809 } {
810 t.Run(tc.name, func(t *testing.T) {
811 got, layer := Effective(ParseEnv([]byte(tc.config)), ParseEnv([]byte(tc.own)))
812 if got != tc.want || layer != tc.layer {
813 t.Fatalf("Effective = %q, %v; want %q, %v", got, layer, tc.want, tc.layer)
814 }
815 })
816 }
817}
818
819// Ruling 20 (R-B): a record binds one writer. Another session, another
820// generation, another installation or another host never matches, so a
821// later release that picks its own key, a rollback and a reinstall never
822// inherit an earlier writer's adoption.
823func TestAdoptionRecordBindsOneWriter(t *testing.T) {
824 key := strings.Repeat("A", 43) + "="
825 w := Binding{Host: "shop.example.com", Incarnation: "inc-1", Session: 3, Generation: "g3"}
826 rec := AdoptionRecord(w, SourcePlatform, Digest(key))
827 if strings.Contains(string(rec), key) {
828 t.Fatal("the record holds the key itself")
829 }
830 a, err := ParseAdoption(rec)
831 if err != nil || !a.Binds(w) || a.Source != SourcePlatform || a.SHA256 != Digest(key) {
832 t.Fatalf("ParseAdoption = %+v, %v", a, err)
833 }
834 for name, other := range map[string]Binding{
835 "an earlier session (an earlier release)": {Host: w.Host, Incarnation: w.Incarnation, Session: 2, Generation: "g2"},
836 "a later session (a rollback)": {Host: w.Host, Incarnation: w.Incarnation, Session: 4, Generation: "g4"},
837 "the same session, another generation": {Host: w.Host, Incarnation: w.Incarnation, Session: 3, Generation: "g3b"},
838 "another installation (a reinstall)": {Host: w.Host, Incarnation: "inc-2", Session: 3, Generation: "g3"},
839 "another host": {Host: "other.example.com", Incarnation: w.Incarnation, Session: 3, Generation: "g3"},
840 "no writer": {},
841 } {
842 if a.Binds(other) {
843 t.Errorf("%s: the record matched", name)
844 }
845 }
846 for name, raw := range map[string]string{
847 "version 1": `{"v":1,"var":"CARLOS_INSTANCE_KEY","sha256":"` + Digest(key) + `"}`,
848 "no binding": `{"v":2,"var":"CARLOS_INSTANCE_KEY","source":"platform","sha256":"` + Digest(key) + `"}`,
849 "unknown source": `{"v":2,"var":"CARLOS_INSTANCE_KEY","host":"h","incarnation":"i","session":1,"generation":"g","source":"own","sha256":"` + Digest(key) + `"}`,
850 "an unknown field": `{"v":2,"var":"CARLOS_INSTANCE_KEY","host":"h","incarnation":"i","session":1,"generation":"g","source":"config","sha256":"` + Digest(key) + `","key":"x"}`,
851 "trailing data": string(rec) + `{}`,
852 "garbage": "not json",
853 // The canonical bytes are the only record (review minor 1):
854 // encoding/json alone accepts each of these.
855 "a trailing }": string(rec) + `}`,
856 "a trailing ]": string(rec) + `]`,
857 "an upper-case key": strings.Replace(string(rec), `"sha256"`, `"SHA256"`, 1),
858 "a key given twice": strings.Replace(string(rec), `"source":"platform"`, `"source":"config","source":"platform"`, 1),
859 "reordered fields": strings.Replace(string(rec), `{"v":2,"var":"CARLOS_INSTANCE_KEY",`, `{"var":"CARLOS_INSTANCE_KEY","v":2,`, 1),
860 "an upper-case digest": strings.Replace(string(rec), Digest(key), strings.ToUpper(Digest(key)), 1),
861 "a short digest": strings.Replace(string(rec), Digest(key), Digest(key)[:63], 1),
862 "another variable": strings.Replace(string(rec), `"var":"CARLOS_INSTANCE_KEY"`, `"var":"OTHER"`, 1),
863 } {
864 if raw == string(rec) {
865 t.Fatalf("%s: the case did not change the record", name)
866 }
867 if _, err := ParseAdoption([]byte(raw)); err == nil {
868 t.Errorf("%s: parsed", name)
869 }
870 }
871}
872
873// ParseEnv is exactly the activator's old parseEnvBytes, so what a reader
874// parses is what a child received: quotes kept, whole-line comments only,
875// the last duplicate winning, CRLF and surrounding whitespace trimmed.
876func TestParseEnvMatchesWhatAChildReceives(t *testing.T) {
877 got := ParseEnv([]byte("A=\"quoted\"\nB='single'\n# C=1\n # D=1\nE=1 # kept\nF=1\nF=2\n G = spaced \nH=a=b\nI=1\r\nnoequals\n\n"))
878 want := map[string]string{"A": `"quoted"`, "B": "'single'", "E": "1 # kept", "F": "2", "G": "spaced", "H": "a=b", "I": "1"}
879 if len(got) != len(want) {
880 t.Fatalf("ParseEnv = %q, want %q", got, want)
881 }
882 for k, v := range want {
883 if got[k] != v {
884 t.Fatalf("ParseEnv = %q, want %q", got, want)
885 }
886 }
887}
888
889// O3': the install refuses env files that would deliver any key but the
890// pinned one, including a configuration changed since the pin.
891func TestVerifyDeliveredRefusesAnyOtherKey(t *testing.T) {
892 a, b := strings.Repeat("A", 43)+"=", strings.Repeat("B", 43)+"="
893 for _, tc := range []struct {
894 name, config, own string
895 ok bool
896 }{
897 {"the configured key, unchanged", Var + "=" + a + "\n", "T=1\n", true},
898 {"the configuration changed since the pin", Var + "=" + b + "\n", "T=1\n", false},
899 {"the configured key removed", "", "T=1\n", false},
900 {"an empty override over the configured key", Var + "=" + a + "\n", Var + "=\n", true},
901 {"the platform key, restored", "", Var + "=" + a + "\n", true},
902 {"a platform key over the pinned configured one", Var + "=" + a + "\n", Var + "=" + b + "\n", false},
903 } {
904 if err := VerifyDelivered([]byte(tc.config), []byte(tc.own), Digest(a)); (err == nil) != tc.ok {
905 t.Errorf("%s: err = %v", tc.name, err)
906 }
907 }
908}
909
910// Ruling 26: the pre-roll check's verified home keys, as the box reads
911// them. A list that may be wrong verifies nothing, and its errors never
912// echo an entry, which may hold a digest (Astra round 5 finding 3).
913func TestParseHomeKeys(t *testing.T) {
914 a, b := Digest("home-key-a"), Digest("home-key-b")
915 got, err := ParseHomeKeys(" shop-au.example.com=" + a + ", shop-uk.example.com=" + b + ",")
916 if err != nil || len(got) != 2 || got["shop-au.example.com"] != a || got["shop-uk.example.com"] != b {
917 t.Fatalf("ParseHomeKeys = %v, %v", got, err)
918 }
919 if got, err := ParseHomeKeys(""); err != nil || len(got) != 0 {
920 t.Fatalf("empty = %v, %v", got, err)
921 }
922 for _, tc := range []struct{ value, want string }{
923 {a, "entry 1: no \"replica-host=\""},
924 {"shop-au.example.com=" + a + "," + b, "entry 2: no \"replica-host=\""},
925 {"shop-au.example.com=" + a[:40], "entry 1: the digest is not"},
926 {"=" + a, "entry 1: the host is empty"},
927 {a[:20] + "=" + a, "entry 1: the host is empty or not a host name"},
928 {"shop-au.example.com=" + a + ",shop-au.example.com=" + b, "entry 2: its host is listed"},
929 } {
930 _, err := ParseHomeKeys(tc.value)
931 if err == nil || !strings.Contains(err.Error(), tc.want) {
932 t.Errorf("%q: err = %v, want %q", tc.value, err, tc.want)
933 continue
934 }
935 if strings.Contains(err.Error(), a[:16]) || strings.Contains(err.Error(), b[:16]) || strings.Contains(err.Error(), "shop-") {
936 t.Errorf("the error echoes its input: %v", err)
937 }
938 }
939}
940```
941
942- [ ] **Step 2: Run them to verify they fail**
943
944Run: `nice -n 19 go test ./internal/instancekey/ -count=1`
945Expected: build failure, no non-test Go files.
946
947- [ ] **Step 3: Write the package**
948
949Create `internal/instancekey/instancekey.go`:
950
951```go
952// Package instancekey is the one rule for the platform's per-instance key
953// (region HA L21, rastrillo #66): how an env file is parsed, which file's
954// CARLOS_INSTANCE_KEY a launched app receives, and the adoption record, a
955// writer-bound assertion from the app's helper that it seals with that
956// key. The activator mints and
957// delivers the key by these rules, the control socket records adoption by
958// them, and the relocation pinner checks its evidence by them, so the three
959// can never disagree about what the app was given.
960//
961// It imports only the standard library, so the pinner can use it without
962// importing the activator.
963package instancekey
964
965import (
966 "bytes"
967 "crypto/sha256"
968 "encoding/hex"
969 "encoding/json"
970 "errors"
971 "fmt"
972 "strings"
973)
974
975// Var is the variable the key is delivered as.
976const Var = "CARLOS_INSTANCE_KEY"
977
978// AdoptionSuffix names the adoption record beside an instance's own env
979// file: <host>.env's sibling <host>.instance-key-adoption.json. It is an
980// installation artifact: moves retire it with <host>.env (O7), and nested
981// it lives in the instance directory, retired with it.
982const AdoptionSuffix = ".instance-key-adoption.json"
983
984// AdoptionItem is the adoption record's name among a files manifest's
985// platform items, beside hostfiles' EnvItem ("host.env").
986const AdoptionItem = "instance-key-adoption.json"
987
988// ParseEnv parses an EnvironmentFile: KEY=VALUE lines, blank lines and #
989// comments skipped, the key and the value trimmed of surrounding
990// whitespace, the last of two lines for one key winning. It is exactly
991// what the activator hands a child, so a reader that wants to know what an
992// app received must parse with it. "export KEY=..." is the key "export
993// KEY", and a quoted value keeps its quotes.
994func ParseEnv(raw []byte) map[string]string {
995 m := map[string]string{}
996 for _, line := range strings.Split(string(raw), "\n") {
997 line = strings.TrimSpace(line)
998 if line == "" || strings.HasPrefix(line, "#") {
999 continue
1000 }
1001 if k, v, ok := strings.Cut(line, "="); ok {
1002 m[strings.TrimSpace(k)] = strings.TrimSpace(v)
1003 }
1004 }
1005 return m
1006}
1007
1008// Layer is which env file supplied the effective key.
1009type Layer int
1010
1011const (
1012 // LayerNone: neither file supplies a non-empty key.
1013 LayerNone Layer = iota
1014 // LayerOwn: the instance's own <host>.env, where the platform mints.
1015 LayerOwn
1016 // LayerConfig: the rendered <host>.config.env, the app's configuration.
1017 LayerConfig
1018)
1019
1020// Effective is the key a launched app receives from its two env files, and
1021// which file supplied it. The instance's own file wins, as in every other
1022// variable (the activator's mergeEnv), but an empty value supplies no key:
1023// an empty CARLOS_INSTANCE_KEY= in the own file never shadows a non-empty
1024// configured key, so an app is never handed an empty key while a non-empty
1025// one exists.
1026func Effective(config, own map[string]string) (string, Layer) {
1027 if k := own[Var]; k != "" {
1028 return k, LayerOwn
1029 }
1030 if k := config[Var]; k != "" {
1031 return k, LayerConfig
1032 }
1033 return "", LayerNone
1034}
1035
1036// Digest is the lowercase hex SHA-256 of key: what an app's helper sends
1037// to say which key it resolved, and what the adoption record holds. The
1038// record is published beside the env file that holds a platform key
1039// itself, so the digest gives that file's reader nothing new.
1040func Digest(key string) string {
1041 sum := sha256.Sum256([]byte(key))
1042 return hex.EncodeToString(sum[:])
1043}
1044
1045// The two sources rastrillo's step 2 can have resolved. Both arrive as
1046// $CARLOS_INSTANCE_KEY once the env files are merged.
1047const (
1048 SourcePlatform = "platform" // the instance's own <host>.env, where the platform mints
1049 SourceConfig = "config" // the rendered <host>.config.env, the app's configuration
1050)
1051
1052// SourceOf names the source of a layer, "" for LayerNone.
1053func SourceOf(l Layer) string {
1054 switch l {
1055 case LayerOwn:
1056 return SourcePlatform
1057 case LayerConfig:
1058 return SourceConfig
1059 }
1060 return ""
1061}
1062
1063// Binding is the writer an adoption belongs to: the host, its installation
1064// (the claim holder's incarnation), and the writer session and generation
1065// the process was admitted as. The agent takes all four from its own
1066// launch state for the process, never from the request.
1067type Binding struct {
1068 Host string `json:"host"`
1069 Incarnation string `json:"incarnation"`
1070 Session uint64 `json:"session"`
1071 Generation string `json:"generation"`
1072}
1073
1074func (b Binding) valid() bool {
1075 return b.Host != "" && b.Incarnation != "" && b.Session != 0 && b.Generation != ""
1076}
1077
1078// Adoption is the record the agent writes when a process reports that its
1079// helper resolved the key at step 2: which writer, which source, which
1080// key (by digest). It is a trusted-helper assertion, not independent
1081// evidence: it stops accidental loss, and a tenant who forges one can lose
1082// only its own data (instance-key plan ruling 19).
1083type Adoption struct {
1084 V int `json:"v"`
1085 Var string `json:"var"`
1086 Binding
1087 Source string `json:"source"`
1088 SHA256 string `json:"sha256"`
1089}
1090
1091// AdoptionRecord is the canonical bytes of the record for writer b, which
1092// was handed the key with digest sha from source.
1093func AdoptionRecord(b Binding, source, sha string) []byte {
1094 raw, _ := json.Marshal(Adoption{V: 2, Var: Var, Binding: b, Source: source, SHA256: sha})
1095 return append(raw, '\n')
1096}
1097
1098// ParseAdoption reads a record strictly: version 2, this variable, a
1099// complete binding, a known source and a digest, in exactly the bytes
1100// AdoptionRecord writes for them. Anything else is an error, never a
1101// record: trailing data, a key in another case, a key given twice, or any
1102// other spelling of the same fields (encoding/json alone accepts the
1103// last three).
1104func ParseAdoption(record []byte) (Adoption, error) {
1105 var a Adoption
1106 dec := json.NewDecoder(bytes.NewReader(record))
1107 dec.DisallowUnknownFields()
1108 if err := dec.Decode(&a); err != nil {
1109 return Adoption{}, fmt.Errorf("instancekey: adoption record: %w", err)
1110 }
1111 if dec.More() || a.V != 2 || a.Var != Var || !a.Binding.valid() || (a.Source != SourcePlatform && a.Source != SourceConfig) || !ValidDigest(a.SHA256) {
1112 return Adoption{}, errors.New("instancekey: adoption record: not a version 2 record for " + Var)
1113 }
1114 if !bytes.Equal(record, AdoptionRecord(a.Binding, a.Source, a.SHA256)) {
1115 return Adoption{}, errors.New("instancekey: adoption record: not in the canonical form AdoptionRecord writes")
1116 }
1117 return a, nil
1118}
1119
1120// Binds reports whether the record belongs to exactly writer w. A record
1121// from another installation, another session or another generation (an
1122// earlier release, a rollback, a reinstall) never vouches for w.
1123func (a Adoption) Binds(w Binding) bool {
1124 return w.valid() && a.Binding == w
1125}
1126
1127// VerifyDelivered checks that the env files a relocation target is about
1128// to launch with deliver the key pinned by digest (O3'): the effective key
1129// of the rendered configuration and the restored own env file, by
1130// Effective's rule. A configuration changed since the pin, a key removed,
1131// or a platform key where a configured one was pinned, is an error; the
1132// install refuses rather than hand the app a different key.
1133func VerifyDelivered(configEnv, ownEnv []byte, pinnedSHA string) error {
1134 key, _ := Effective(ParseEnv(configEnv), ParseEnv(ownEnv))
1135 if key == "" || Digest(key) != pinnedSHA {
1136 return errors.New("instancekey: the env files would not deliver the pinned " + Var)
1137 }
1138 return nil
1139}
1140
1141// ValidDigest reports whether s is a lowercase hex SHA-256.
1142func ValidDigest(s string) bool {
1143 if len(s) != 64 {
1144 return false
1145 }
1146 for _, r := range s {
1147 if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
1148 return false
1149 }
1150 }
1151 return true
1152}
1153
1154// ParseHomeKeys reads CARLOS_REPLICA_HOME_KEYS, the box's list of verified
1155// home keys for its replica rows (instance-key plan rulings 26 and 27):
1156// "replica-host=sha256,replica-host=sha256", each digest the home's
1157// effective key as the pre-roll check computed it. Only the pre-roll check
1158// writes it; nothing refreshes it from a replica's own configuration.
1159// Whitespace around entries is ignored and "" is an empty list. A
1160// malformed entry, or a host named twice, fails the whole value: a list
1161// that may be wrong is not a list to verify keys against. Its errors name
1162// the entry by position and the reason, never the entry's text, which may
1163// hold a digest.
1164func ParseHomeKeys(s string) (map[string]string, error) {
1165 out := map[string]string{}
1166 n := 0
1167 for _, e := range strings.Split(s, ",") {
1168 e = strings.TrimSpace(e)
1169 if e == "" {
1170 continue
1171 }
1172 n++
1173 host, sha, ok := strings.Cut(e, "=")
1174 host, sha = strings.TrimSpace(host), strings.TrimSpace(sha)
1175 switch {
1176 case !ok:
1177 return nil, fmt.Errorf("instancekey: CARLOS_REPLICA_HOME_KEYS entry %d: no \"replica-host=\" before the digest", n)
1178 case host == "" || strings.ContainsAny(host, " /=") || !strings.Contains(host, "."):
1179 return nil, fmt.Errorf("instancekey: CARLOS_REPLICA_HOME_KEYS entry %d: the host is empty or not a host name", n)
1180 case !ValidDigest(sha):
1181 return nil, fmt.Errorf("instancekey: CARLOS_REPLICA_HOME_KEYS entry %d: the digest is not 64 lowercase hex characters", n)
1182 }
1183 if _, dup := out[host]; dup {
1184 return nil, fmt.Errorf("instancekey: CARLOS_REPLICA_HOME_KEYS entry %d: its host is listed by an earlier entry", n)
1185 }
1186 out[host] = sha
1187 }
1188 return out, nil
1189}
1190```
1191
1192- [ ] **Step 4: One parser, and the merge follows the rule**
1193
1194In `internal/activator/admintoken.go`, replace `parseEnvBytes`'s body (keep its doc comment), drop the now-unused `"strings"` import and add `"github.com/carlosframework/platform/internal/instancekey"`:
1195
1196```go
1197func parseEnvBytes(raw []byte) map[string]string {
1198 return instancekey.ParseEnv(raw)
1199}
1200```
1201
1202In `internal/activator/backend_exec.go`, add the same import, and end `mergeEnv` with the rule:
1203
1204```go
1205 for k, v := range overlay {
1206 out[k] = v
1207 }
1208 // An empty CARLOS_INSTANCE_KEY in the instance's own file supplies no
1209 // key, so it never shadows a configured one: the child is never handed
1210 // an empty key while a non-empty one exists (instance-key plan ruling
1211 // 13). instancekey.Effective is the rule the key's mint decides by.
1212 if key, _ := instancekey.Effective(base, overlay); key != "" {
1213 out[instancekey.Var] = key
1214 }
1215 return out
1216}
1217```
1218
1219In `LaunchEnvironment`, an empty own key is not a shadow:
1220
1221```go
1222 for k := range ownEnv {
1223 if _, rendered := cfgEnv[k]; rendered {
1224 if k == instancekey.Var && ownEnv[k] == "" {
1225 continue // an empty own key shadows nothing (mergeEnv)
1226 }
1227 shadowed = append(shadowed, k)
1228 }
1229 }
1230```
1231
1232Name the exception in both doc comments (task 1 review):
1233
1234```go
1235// mergeEnv composes the child's env-file layers: base (the sync-owned
1236// <host>.config.env, bucket-derived) under overlay (the instance's own
1237// <host>.env). The overlay wins every conflict but one — CARLOS_ADMIN_TOKEN
1238// lives only in the instance file and must never be clobberable from
1239// the bucket, and a box-local override deliberately beats central
1240// config. The exception: an empty CARLOS_INSTANCE_KEY in the overlay
1241// supplies no key, so a non-empty one in base stands (instancekey.Effective).
1242// Returns a fresh map; inputs are never mutated.
1243func mergeEnv(base, overlay map[string]string) map[string]string {
1244```
1245
1246```go
1247// shadowed is the sorted names present in both the rendered configuration
1248// file and the instance's own <host>.env: the override mergeEnv applies
1249// (moves spec §7.3 T3 reports it), collected here because this is the only
1250// layer that sees both files. The one exception is mergeEnv's: an empty
1251// CARLOS_INSTANCE_KEY in the own file overrides nothing, so it is not
1252// listed. The move's launch report (§7.3, codex R2-8)
1253// is computed over exactly this env, so a target compares what would run,
1254// not a reconstruction of it. The values are raw: the caller HMACs them
1255// and must never log or return them.
1256func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, []string, error) {
1257```
1258
1259Add to `internal/activator/backend_exec_test.go`, after `TestLaunchEnvironmentComposesTheSpawnEnvironment`:
1260
1261```go
1262// An empty CARLOS_INSTANCE_KEY in the instance's own file supplies no key
1263// (instance-key plan ruling 13): the merge keeps the configured key, and
1264// the launch report does not list the empty one as shadowing it. A
1265// non-empty own key still wins and is still reported.
1266func TestEmptyOwnInstanceKeyShadowsNothing(t *testing.T) {
1267 cfg := map[string]string{"CARLOS_INSTANCE_KEY": "cfg"}
1268 if got := mergeEnv(cfg, map[string]string{"CARLOS_INSTANCE_KEY": ""}); got["CARLOS_INSTANCE_KEY"] != "cfg" {
1269 t.Fatalf("mergeEnv with an empty own key: CARLOS_INSTANCE_KEY = %q, want the configured key", got["CARLOS_INSTANCE_KEY"])
1270 }
1271 b := &ExecBackend{DataDir: t.TempDir(), Region: "ie"}
1272 rt := registry.Route{Host: "k.go.example.com", Socket: filepath.Join(b.DataDir, "k.sock")}
1273 write := func(name, body string) {
1274 t.Helper()
1275 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+name), []byte(body), 0o600); err != nil {
1276 t.Fatal(err)
1277 }
1278 }
1279 write(".config.env", "CARLOS_INSTANCE_KEY=cfg\nMINE=0\n")
1280 write(".env", "CARLOS_INSTANCE_KEY=\nMINE=1\n")
1281 env, shadowed, err := b.LaunchEnvironment(rt)
1282 if err != nil {
1283 t.Fatal(err)
1284 }
1285 if env["CARLOS_INSTANCE_KEY"] != "cfg" || len(shadowed) != 1 || shadowed[0] != "MINE" {
1286 t.Fatalf("empty own key: CARLOS_INSTANCE_KEY is the configured key: %v; shadowed %v, want [MINE]", env["CARLOS_INSTANCE_KEY"] == "cfg", shadowed)
1287 }
1288 write(".env", "CARLOS_INSTANCE_KEY=own\nMINE=1\n")
1289 if env, shadowed, err = b.LaunchEnvironment(rt); err != nil || env["CARLOS_INSTANCE_KEY"] != "own" || strings.Join(shadowed, ",") != "CARLOS_INSTANCE_KEY,MINE" {
1290 t.Fatalf("non-empty own key: CARLOS_INSTANCE_KEY is the own key: %v; shadowed %v; err %v", env["CARLOS_INSTANCE_KEY"] == "own", shadowed, err)
1291 }
1292}
1293```
1294
1295- [ ] **Step 5: Run the tests**
1296
1297Run: `nice -n 19 go test ./internal/instancekey/ -count=1 && nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartMergesConfigEnvUnderOwnEnv|LaunchEnvironment|TestEmptyOwnInstanceKeyShadowsNothing|TestEnsureAdminToken' -count=1` (sandbox off for the second: it starts children)
1298Expected: PASS.
1299
1300- [ ] **Step 6: Commit**
1301
1302```bash
1303gofmt -l internal/instancekey internal/activator
1304git add internal/instancekey internal/activator/admintoken.go internal/activator/backend_exec.go internal/activator/backend_exec_test.go
1305git commit -m "instancekey: one rule for the instance key's parse, effective layer and adoption record (L21)
1306
1307Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
1308```
1309
1310---
1311
1312### Task 2: The mint, durable every time
1313
1314**Files:**
1315- Create: `internal/activator/instancekey.go`, `internal/activator/ownenv_unix.go` and `ownenv_other.go` (`openOwnEnvNoFollow`: `O_NOFOLLOW` on unix, an `Lstat` check-then-open on windows, which the agent never runs on)
1316- Test: `internal/activator/instancekey_test.go`, `internal/activator/instancekey_linux_test.go`
1317
1318**Built:** the code block in Step 3 is the branch's, with three things rev 5.1 did not have, each by ruling. The flat own file is opened with `O_NOFOLLOW` in the open itself (not check-then-open), mirroring `EnsureAdminToken`'s symlink refusal; a symlinked flat `<host>.env` fails the launch (the admin-token mint path has the same hole on main: #273). A failed or short append is undone: the file is truncated back to the size the open descriptor reported before the write (`undoInstanceKeyAppend`) and fsynced, so a short key line never survives to be read as present; under `adminTokenMu` that size is `len(raw)`. And `ensureInstanceKeyFor` refuses a fenced or conflicting row itself, deciding on the paths the mint would use, because the launch bodies classify again in `pathsFor` after the dispatcher's own check and a migration starting mid-wake would otherwise mint into the flat directory of a half-migrated host.
1319
1320**Interfaces:**
1321- Consumes: Task 1's `instancekey`; `adminTokenMu` (admintoken.go:43), `logPerFile` (:70), `parseEnvBytes`, `instancefs.ReadBeneathLimit`, `instancefs.OpenBeneath`, `(*ExecBackend).pathsFor` (paths.go:33), `Layout*` constants (layout.go:21-26).
1322- Produces:
1323 - `const InstanceKeyVar = instancekey.Var`
1324 - `type InstanceKeyOutcome int` with `InstanceKeyPresent`, `InstanceKeyMinted`, `InstanceKeyFromConfig`, `InstanceKeyNoEnvFile`, `InstanceKeyReplica`
1325 - `func EnsureInstanceKey(path string) (InstanceKeyOutcome, error)`
1326 - `func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error)`
1327 - `func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error)`
1328 - `func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error)`
1329 - `func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error`, `errInstanceEnvNeedsWrite`
1330 - the test seam `instanceKeySync` (file and directory sync), and `syncDir`
1331
1332- [ ] **Step 1: Write the failing tests**
1333
1334Create `internal/activator/instancekey_test.go`:
1335
1336```go
1337package activator
1338
1339import (
1340 "bytes"
1341 "encoding/base64"
1342 "errors"
1343 "log"
1344 "os"
1345 "path/filepath"
1346 "strings"
1347 "sync"
1348 "testing"
1349
1350 "github.com/carlosframework/platform/internal/registry"
1351)
1352
1353// keyLines returns every CARLOS_INSTANCE_KEY value in raw, in order.
1354func keyLines(raw []byte) []string {
1355 var out []string
1356 for _, line := range strings.Split(string(raw), "\n") {
1357 if v, ok := strings.CutPrefix(line, InstanceKeyVar+"="); ok {
1358 out = append(out, v)
1359 }
1360 }
1361 return out
1362}
1363
1364// syncProbe counts the key's syncs and fails them on demand. Set the
1365// error fields only between calls.
1366type syncProbe struct {
1367 file, dir int
1368 fileErr, dirErr error
1369}
1370
1371func probeInstanceKeySync(t *testing.T) *syncProbe {
1372 t.Helper()
1373 p := &syncProbe{}
1374 adminTokenMu.Lock()
1375 old := instanceKeySync
1376 instanceKeySync.file = func(f *os.File) error {
1377 p.file++
1378 if p.fileErr != nil {
1379 return p.fileErr
1380 }
1381 return f.Sync()
1382 }
1383 instanceKeySync.dir = func(d string) error {
1384 p.dir++
1385 if p.dirErr != nil {
1386 return p.dirErr
1387 }
1388 return syncDir(d)
1389 }
1390 adminTokenMu.Unlock()
1391 t.Cleanup(func() {
1392 adminTokenMu.Lock()
1393 instanceKeySync = old
1394 adminTokenMu.Unlock()
1395 })
1396 return p
1397}
1398
1399func TestEnsureInstanceKeyMintsOnceInTheAgreedFormat(t *testing.T) {
1400 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1401 got, err := EnsureInstanceKey(p)
1402 if err != nil || got != InstanceKeyMinted {
1403 t.Fatalf("first ensure = %v, %v; want minted", got, err)
1404 }
1405 raw, _ := os.ReadFile(p)
1406 keys := keyLines(raw)
1407 if len(keys) != 1 {
1408 t.Fatalf("file holds %d keys, want 1", len(keys))
1409 }
1410 dec, err := base64.StdEncoding.Strict().DecodeString(keys[0])
1411 if err != nil || len(dec) != 32 || len(keys[0]) != 44 {
1412 t.Fatalf("key is not 32 bytes in padded standard base64 (len %d, decode err %v)", len(keys[0]), err)
1413 }
1414 if !bytes.HasSuffix(raw, []byte("\n")) {
1415 t.Fatal("the key line has no newline: the next append would join it")
1416 }
1417 if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
1418 t.Fatalf("mode = %o, want 600", st.Mode().Perm())
1419 }
1420 // The parser every launch path uses yields the exact characters minted.
1421 env, _ := readEnvFile(p)
1422 if env[InstanceKeyVar] != keys[0] {
1423 t.Fatal("readEnvFile does not yield the minted key verbatim")
1424 }
1425 again, err := EnsureInstanceKey(p)
1426 if err != nil || again != InstanceKeyPresent {
1427 t.Fatalf("second ensure = %v, %v; want present", again, err)
1428 }
1429 raw2, _ := os.ReadFile(p)
1430 if !bytes.Equal(raw, raw2) {
1431 t.Fatal("second ensure changed the file")
1432 }
1433}
1434
1435// Never overwrite: a key already there, of any shape, is kept byte for byte
1436// and only reported.
1437func TestEnsureInstanceKeyNeverRewritesAnExistingKey(t *testing.T) {
1438 for _, existing := range []string{
1439 "CARLOS_INSTANCE_KEY=hand-written-not-base64\n",
1440 "A=1\nCARLOS_INSTANCE_KEY=" + strings.Repeat("A", 43) + "=", // well formed, no trailing newline
1441 } {
1442 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1443 if err := os.WriteFile(p, []byte(existing), 0o600); err != nil {
1444 t.Fatal(err)
1445 }
1446 var buf bytes.Buffer
1447 log.SetOutput(&buf)
1448 got, err := EnsureInstanceKey(p)
1449 log.SetOutput(os.Stderr)
1450 if err != nil || got != InstanceKeyPresent {
1451 t.Fatalf("ensure over %q = %v, %v; want present", existing, got, err)
1452 }
1453 if raw, _ := os.ReadFile(p); string(raw) != existing {
1454 t.Fatalf("file changed: %q -> %q", existing, raw)
1455 }
1456 if strings.Contains(buf.String(), "hand-written-not-base64") {
1457 t.Fatal("the shape warning logged the key's value")
1458 }
1459 }
1460}
1461
1462func TestEnsureInstanceKeyAppendsOnAFreshLineAndKeepsEveryOtherLine(t *testing.T) {
1463 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1464 before := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("a", 64) + "\n# comment\nDATABASE_URL=sqlite" // no trailing newline
1465 if err := os.WriteFile(p, []byte(before), 0o644); err != nil {
1466 t.Fatal(err)
1467 }
1468 if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted {
1469 t.Fatalf("ensure = %v, %v", got, err)
1470 }
1471 raw, _ := os.ReadFile(p)
1472 if !strings.HasPrefix(string(raw), before+"\n"+InstanceKeyVar+"=") {
1473 t.Fatalf("file = %q: the key must land on its own line after the untouched original", raw)
1474 }
1475 env, _ := readEnvFile(p)
1476 if env["DATABASE_URL"] != "sqlite" || len(env["CARLOS_ADMIN_TOKEN"]) != 64 {
1477 t.Fatal("an existing line was lost")
1478 }
1479 if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
1480 t.Fatalf("mode = %o, want the 0644 file tightened to 600", st.Mode().Perm())
1481 }
1482}
1483
1484// A key the app's configuration supplies is the app's key: minting one into
1485// <host>.env would override it (mergeEnv) and change the key under its data.
1486// The decision is the effective env's, so an empty CARLOS_INSTANCE_KEY= in
1487// the own file does not hide the configured key (finding 4).
1488func TestEnsureInstanceKeyDefersToTheRenderedConfiguration(t *testing.T) {
1489 for _, own := range []string{"A=1\n", "A=1\n" + InstanceKeyVar + "=\n"} {
1490 dir := t.TempDir()
1491 p := filepath.Join(dir, "shop.example.com.env")
1492 if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=from-config\n"), 0o600); err != nil {
1493 t.Fatal(err)
1494 }
1495 if err := os.WriteFile(p, []byte(own), 0o600); err != nil {
1496 t.Fatal(err)
1497 }
1498 got, err := EnsureInstanceKey(p)
1499 if err != nil || got != InstanceKeyFromConfig {
1500 t.Fatalf("own %q: ensure = %v, %v; want from-config", own, got, err)
1501 }
1502 if raw, _ := os.ReadFile(p); string(raw) != own {
1503 t.Fatalf("own file changed: %q", raw)
1504 }
1505 }
1506}
1507
1508func TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile(t *testing.T) {
1509 b := &ExecBackend{DataDir: t.TempDir()}
1510 fresh := registry.Route{Host: "new.example.com", Socket: "/run/carlos/new.sock"}
1511 got, err := b.BackfillInstanceKey(fresh)
1512 if err != nil || got != InstanceKeyNoEnvFile {
1513 t.Fatalf("backfill with no env file = %v, %v; want no-env-file", got, err)
1514 }
1515 if _, err := os.Lstat(filepath.Join(b.DataDir, fresh.Host+".env")); !os.IsNotExist(err) {
1516 t.Fatalf("backfill created an env file: %v", err)
1517 }
1518 old := registry.Route{Host: "shop.example.com", Socket: "/run/carlos/shop.sock"}
1519 envPath := filepath.Join(b.DataDir, old.Host+".env")
1520 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("b", 64) + "\n"
1521 if err := os.WriteFile(envPath, []byte(tok), 0o600); err != nil {
1522 t.Fatal(err)
1523 }
1524 if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyMinted {
1525 t.Fatalf("backfill = %v, %v; want minted", got, err)
1526 }
1527 raw, _ := os.ReadFile(envPath)
1528 if !strings.HasPrefix(string(raw), tok) || len(keyLines(raw)) != 1 {
1529 t.Fatalf("file = %q", raw)
1530 }
1531 // Idempotent: a second backfill (the next agent start) writes nothing.
1532 if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyPresent {
1533 t.Fatalf("second backfill = %v, %v; want present", got, err)
1534 }
1535 if raw2, _ := os.ReadFile(envPath); !bytes.Equal(raw, raw2) {
1536 t.Fatal("second backfill changed the file")
1537 }
1538}
1539
1540// Review focus 1: a hand-edited file with two keys. Readers take the last
1541// (parseEnvBytes), and the ensure writes nothing.
1542func TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone(t *testing.T) {
1543 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1544 body := InstanceKeyVar + "=first\n" + InstanceKeyVar + "=second\n"
1545 if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
1546 t.Fatal(err)
1547 }
1548 if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyPresent {
1549 t.Fatalf("ensure = %v, %v; want present", got, err)
1550 }
1551 if raw, _ := os.ReadFile(p); string(raw) != body {
1552 t.Fatalf("file changed: %q", raw)
1553 }
1554 if env, _ := readEnvFile(p); env[InstanceKeyVar] != "second" {
1555 t.Fatalf("readers see %q, want the last line's value", env[InstanceKeyVar])
1556 }
1557}
1558
1559// Review focus 2: an empty value is no key. One is appended after it, and
1560// readers take the appended one.
1561func TestEnsureInstanceKeyTreatsAnEmptyValueAsAbsent(t *testing.T) {
1562 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1563 if err := os.WriteFile(p, []byte(InstanceKeyVar+"=\n"), 0o600); err != nil {
1564 t.Fatal(err)
1565 }
1566 if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted {
1567 t.Fatalf("ensure = %v, %v; want minted", got, err)
1568 }
1569 env, _ := readEnvFile(p)
1570 if !wellFormedInstanceKey(env[InstanceKeyVar]) {
1571 t.Fatal("readers do not see the appended key")
1572 }
1573}
1574
1575// Review focus 3: a seed, a visitor's wake and the backfill can ask at
1576// once. Exactly one key is written.
1577func TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers(t *testing.T) {
1578 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1579 var wg sync.WaitGroup
1580 for i := 0; i < 8; i++ {
1581 wg.Add(1)
1582 go func() {
1583 defer wg.Done()
1584 if _, err := EnsureInstanceKey(p); err != nil {
1585 t.Error(err)
1586 }
1587 }()
1588 }
1589 wg.Wait()
1590 if raw, _ := os.ReadFile(p); len(keyLines(raw)) != 1 {
1591 t.Fatalf("%d keys written", len(keyLines(raw)))
1592 }
1593}
1594
1595// Review focus 5: an empty CARLOS_INSTANCE_KEY in the rendered
1596// configuration supplies nothing, so the platform mints one.
1597func TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty(t *testing.T) {
1598 dir := t.TempDir()
1599 if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=\n"), 0o600); err != nil {
1600 t.Fatal(err)
1601 }
1602 if got, err := EnsureInstanceKey(filepath.Join(dir, "shop.example.com.env")); err != nil || got != InstanceKeyMinted {
1603 t.Fatalf("ensure = %v, %v; want minted", got, err)
1604 }
1605}
1606
1607// Finding 2: a mint whose file or directory sync fails reports failure,
1608// and so does every retry until a sync succeeds. The retry finds the key
1609// present, keeps its bytes, and still completes the chmod and both syncs
1610// before it reports success.
1611func TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess(t *testing.T) {
1612 for _, tc := range []struct {
1613 name string
1614 fail func(*syncProbe, error)
1615 }{
1616 {"file sync", func(p *syncProbe, err error) { p.fileErr = err }},
1617 {"directory sync", func(p *syncProbe, err error) { p.dirErr = err }},
1618 } {
1619 t.Run(tc.name, func(t *testing.T) {
1620 probe := probeInstanceKeySync(t)
1621 p := filepath.Join(t.TempDir(), "shop.example.com.env")
1622 tc.fail(probe, errors.New("injected: EIO"))
1623 if _, err := EnsureInstanceKey(p); err == nil {
1624 t.Fatal("a mint whose sync failed reported success")
1625 }
1626 raw, _ := os.ReadFile(p)
1627 if len(keyLines(raw)) != 1 {
1628 t.Fatalf("the failed mint left %d keys, want its 1", len(keyLines(raw)))
1629 }
1630 if _, err := EnsureInstanceKey(p); err == nil {
1631 t.Fatal("a retry reported success while the sync still fails")
1632 }
1633 if err := os.Chmod(p, 0o644); err != nil {
1634 t.Fatal(err)
1635 }
1636 tc.fail(probe, nil)
1637 before := *probe
1638 got, err := EnsureInstanceKey(p)
1639 if err != nil || got != InstanceKeyPresent {
1640 t.Fatalf("retry = %v, %v; want present", got, err)
1641 }
1642 if probe.file != before.file+1 || probe.dir != before.dir+1 {
1643 t.Fatalf("the retry synced the file %d and the directory %d times, want once each", probe.file-before.file, probe.dir-before.dir)
1644 }
1645 if raw2, _ := os.ReadFile(p); !bytes.Equal(raw, raw2) {
1646 t.Fatal("a retry rewrote the key")
1647 }
1648 if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
1649 t.Fatalf("mode = %o, want the present path to set 600", st.Mode().Perm())
1650 }
1651 })
1652 }
1653}
1654
1655// Ruling 26: a replica row is never minted a key, on any path; the only
1656// key it may hold is its home's (replicaKeyGate).
1657func TestReplicaRowsGetNoPlatformKey(t *testing.T) {
1658 b := &ExecBackend{DataDir: t.TempDir()}
1659 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Socket: "/run/carlos/shop-au.sock", Replica: true}
1660 envPath := filepath.Join(b.DataDir, rt.Host+".env")
1661 if err := os.WriteFile(envPath, []byte("CARLOS_ADMIN_TOKEN=t\n"), 0o600); err != nil {
1662 t.Fatal(err)
1663 }
1664 if got, err := b.ensureInstanceKeyFor(rt, b.pathsFor(rt)); err != nil || got != InstanceKeyReplica {
1665 t.Fatalf("launch ensure = %v, %v; want replica", got, err)
1666 }
1667 if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyReplica {
1668 t.Fatalf("backfill = %v, %v; want replica", got, err)
1669 }
1670 if raw, _ := os.ReadFile(envPath); len(keyLines(raw)) != 0 {
1671 t.Fatal("a replica row was given a platform key")
1672 }
1673}
1674```
1675
1676- [ ] **Step 2: Run them to verify they fail**
1677
1678Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestReplicaRows' -count=1`
1679Expected: build failure, `undefined: EnsureInstanceKey` (and `InstanceKeyVar`, `instanceKeySync`, ...).
1680
1681- [ ] **Step 3: Write the implementation**
1682
1683Create `internal/activator/instancekey.go`:
1684
1685```go
1686package activator
1687
1688// The instance key (region HA request L21; rastrillo/rastrillo #66). An app
1689// on CARLOS seals its data with a key. Before this file the app minted that
1690// key itself after it started, so a box that died before the key reached a
1691// backup left a database nothing could unseal. The platform now mints the
1692// key on the box, into the instance's own <host>.env as
1693// CARLOS_INSTANCE_KEY, before the app's first start, and never rewrites it.
1694// rastrillo's carlos.InstanceKey reads it verbatim when the app has no key
1695// of its own.
1696//
1697// The rules, each a ruling in
1698// docs/superpowers/plans/2026-10-10-instance-key-provision.md:
1699// - create-only: a key already in the file is never rewritten, whatever
1700// its shape. A malformed one is reported and left in place, as
1701// EnsureAdminToken does with a malformed token;
1702// - the decision is made on the effective env (instancekey.Effective,
1703// the rule mergeEnv applies): a key the rendered configuration
1704// supplies is the app's key and nothing is minted; an empty own value
1705// is no key;
1706// - durable every time: before an ensure reports success the file is
1707// 0600 (fchmod on its descriptor), the file is fsynced and so is its
1708// directory, when the key was already there too. A sync that failed
1709// once is redone by the next ensure, never skipped because the bytes
1710// are already there. The known limit: once fsync has returned EIO,
1711// Linux may mark the pages clean, so a later fsync's success does not
1712// prove the bytes reached the disk. The retry narrows the gap; it
1713// cannot close it, and no create-only design can;
1714// - a mint that fails part way through its append is undone: the file
1715// is truncated back to its length before the append and fsynced, so a
1716// short key line never survives to be read as present. Those bytes
1717// were never reported, so removing them rewrites no key;
1718// - the value is never logged, never put in an error, and never leaves
1719// the file except in the launch environment;
1720// - a flat row's own file is opened O_NOFOLLOW on every path, read and
1721// append alike (openOwnEnvNoFollow): a symlink there is refused, never
1722// followed into a file a same-uid sibling chose. A nested row's file
1723// gets the same from instancefs.
1724
1725import (
1726 "crypto/rand"
1727 "encoding/base64"
1728 "errors"
1729 "fmt"
1730 "io"
1731 "os"
1732 "path/filepath"
1733 "strings"
1734
1735 "github.com/carlosframework/platform/internal/instancefs"
1736 "github.com/carlosframework/platform/internal/instancekey"
1737 "github.com/carlosframework/platform/internal/registry"
1738)
1739
1740// InstanceKeyVar is the variable the key is delivered as.
1741const InstanceKeyVar = instancekey.Var
1742
1743// instanceKeyBytes is the key's length before encoding: 32 CSPRNG bytes,
1744// standard base64 with padding, 44 characters. rastrillo uses the exact
1745// bytes the env file yields, so this encoding is fixed for the life of
1746// every key minted with it; changing it is a rotation.
1747const instanceKeyBytes = 32
1748
1749// InstanceKeyOutcome is what one ensure did.
1750type InstanceKeyOutcome int
1751
1752const (
1753 // InstanceKeyPresent: the own env file already carried a key. No byte
1754 // was written; the file was made durable.
1755 InstanceKeyPresent InstanceKeyOutcome = iota
1756 // InstanceKeyMinted: a key was appended and made durable.
1757 InstanceKeyMinted
1758 // InstanceKeyFromConfig: the rendered configuration supplies the key.
1759 // Nothing was written.
1760 InstanceKeyFromConfig
1761 // InstanceKeyNoEnvFile: the backfill found no own env file, so the host
1762 // has never been started here. Nothing was written; its first start
1763 // mints the key.
1764 InstanceKeyNoEnvFile
1765 // InstanceKeyReplica: a replica row (#162). A replica is never minted
1766 // a key: it may hold only its home's (ruling 26, replicaKeyGate).
1767 InstanceKeyReplica
1768)
1769
1770func (o InstanceKeyOutcome) String() string {
1771 switch o {
1772 case InstanceKeyPresent:
1773 return "present"
1774 case InstanceKeyMinted:
1775 return "minted"
1776 case InstanceKeyFromConfig:
1777 return "from-config"
1778 case InstanceKeyNoEnvFile:
1779 return "no-env-file"
1780 case InstanceKeyReplica:
1781 return "replica"
1782 }
1783 return fmt.Sprintf("outcome(%d)", int(o))
1784}
1785
1786// instanceKeySync is the durability step's two syncs, a seam so a test can
1787// fail each one. Read and replaced only under adminTokenMu.
1788var instanceKeySync = struct {
1789 file func(*os.File) error
1790 dir func(string) error
1791}{file: (*os.File).Sync, dir: syncDir}
1792
1793// syncDir fsyncs a directory, so an entry created in it survives a power
1794// cut.
1795func syncDir(dir string) error {
1796 d, err := os.Open(dir)
1797 if err != nil {
1798 return err
1799 }
1800 err = d.Sync()
1801 if cerr := d.Close(); err == nil {
1802 err = cerr
1803 }
1804 return err
1805}
1806
1807// instanceKeyWrite is the mint's append, a seam so a test can make it fail
1808// part way (a short write, ENOSPC). Read and replaced only under
1809// adminTokenMu.
1810var instanceKeyWrite = func(f *os.File, b []byte) (int, error) { return f.Write(b) }
1811
1812// instanceKeyBeforeOpen, when set, runs under adminTokenMu after the read
1813// and the decision and before the ensure opens the file to append or
1814// settle: a test hook to change the file between the two, so the open's
1815// own symlink refusal is pinned and not only the read's. nil outside
1816// tests.
1817var instanceKeyBeforeOpen func(path string)
1818
1819func mintInstanceKey() (string, error) {
1820 raw := make([]byte, instanceKeyBytes)
1821 if _, err := rand.Read(raw); err != nil {
1822 return "", err
1823 }
1824 return base64.StdEncoding.EncodeToString(raw), nil
1825}
1826
1827// wellFormedInstanceKey reports whether key has the shape this file mints.
1828// A key that fails it is not necessarily broken (an operator may have
1829// written one by hand), which is why a failure is a warning, never a
1830// rotation.
1831func wellFormedInstanceKey(key string) bool {
1832 if len(key) != base64.StdEncoding.EncodedLen(instanceKeyBytes) {
1833 return false
1834 }
1835 raw, err := base64.StdEncoding.Strict().DecodeString(key)
1836 return err == nil && len(raw) == instanceKeyBytes
1837}
1838
1839// configEnvName is the rendered configuration's file name beside an own env
1840// file: <host>.env's sibling <host>.config.env, the pair pathsFor names.
1841func configEnvName(envName string) string {
1842 return strings.TrimSuffix(envName, ".env") + ".config.env"
1843}
1844
1845// EnsureInstanceKey makes sure path (an instance's own <host>.env on the
1846// flat layout) carries CARLOS_INSTANCE_KEY, creating the file if needed.
1847// Launch paths call it right after EnsureAdminToken.
1848func EnsureInstanceKey(path string) (InstanceKeyOutcome, error) {
1849 return ensureInstanceKey(filepath.Dir(path), filepath.Base(path), false, true)
1850}
1851
1852// EnsureInstanceKeyBeneath is EnsureInstanceKey for a nested row's own env
1853// file, read and written through instancefs because the directory is
1854// tenant-writable (isolated-exec design §3.2).
1855func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) {
1856 return ensureInstanceKey(dir, name, true, true)
1857}
1858
1859// ensureInstanceKeyFor is the launch paths' ensure for rt: none for a
1860// replica row, the nested body for a nested row, the flat one otherwise.
1861//
1862// A fenced or conflicting row is refused and nothing is written, as
1863// BackfillInstanceKey refuses it. Every launch dispatcher (Start,
1864// StartGen, StartStaged, Seed) refuses those rows already, but on its own
1865// classification: the launch body classifies again in pathsFor, and p is
1866// that second answer. A layout journal written between the two (a
1867// migration starting mid-wake) makes p fenced, with the flat directory as
1868// its dataDir, and minting there would write to a half-migrated host
1869// (isolated-exec design §8.2). The refusal is decided on p, the paths the
1870// mint would use. Every caller names the host around the error ("instance
1871// key for <host>: ..."), so the refusal does not.
1872func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) {
1873 if rt.Replica {
1874 return InstanceKeyReplica, nil
1875 }
1876 switch p.layout {
1877 case LayoutFenced, LayoutConflict:
1878 return InstanceKeyPresent, fmt.Errorf("not minted: %s", p.reason)
1879 }
1880 if p.layout == LayoutNested {
1881 return EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env")
1882 }
1883 return EnsureInstanceKey(p.envFile)
1884}
1885
1886// BackfillInstanceKey is the startup backfill's step for one row: a key is
1887// appended only to an own env file that already exists. A host with no file
1888// has never been started on this box, and its first start mints the key;
1889// creating a file here could also occupy the path a move's install expects
1890// to be empty (moveslandinstall.go placeStaged). A row this backend would
1891// refuse to start (fenced, conflict) is an error and nothing is written.
1892// The agent's backfill names the host when it logs the error, so the
1893// refusal does not.
1894func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error) {
1895 if rt.Replica {
1896 return InstanceKeyReplica, nil
1897 }
1898 p := b.pathsFor(rt)
1899 switch p.layout {
1900 case LayoutNested:
1901 return ensureInstanceKey(p.dataDir, rt.Host+".env", true, false)
1902 case LayoutFenced, LayoutConflict:
1903 return InstanceKeyPresent, fmt.Errorf("not backfilled: %s", p.reason)
1904 }
1905 return ensureInstanceKey(p.dataDir, rt.Host+".env", false, false)
1906}
1907
1908// openInstanceEnv opens an own env file for append. Only a create passes
1909// a mode: openat2 refuses a non-zero mode without O_CREAT (EINVAL), and an
1910// existing file's mode is set by the fchmod that follows anyway.
1911func openInstanceEnv(dir, name string, beneath, create bool) (*os.File, error) {
1912 flags, perm := os.O_APPEND|os.O_WRONLY, os.FileMode(0)
1913 if create {
1914 flags, perm = flags|os.O_CREATE, 0o600
1915 }
1916 if beneath {
1917 return instancefs.OpenBeneath(dir, name, flags, perm)
1918 }
1919 return openOwnEnvNoFollow(filepath.Join(dir, name), flags, perm)
1920}
1921
1922// errOwnEnvSymlink is a flat row's own env file found to be a symlink. The
1923// ensure refuses it on every path (mint, existing key, backfill, the claim
1924// settle): following it would append to, chmod and fsync a file a same-uid
1925// sibling chose (spec §7.2).
1926var errOwnEnvSymlink = errors.New("activator: the instance env file is a symlink, refused")
1927
1928// readOwnEnv reads a flat row's own env file through openOwnEnvNoFollow,
1929// so the read refuses a link exactly as the append does.
1930func readOwnEnv(path string) ([]byte, error) {
1931 f, err := openOwnEnvNoFollow(path, os.O_RDONLY, 0)
1932 if err != nil {
1933 return nil, err
1934 }
1935 defer f.Close()
1936 return io.ReadAll(f)
1937}
1938
1939// settleInstanceEnv is the durability step every successful ensure ends
1940// with: 0600 on the descriptor, then the file, then its directory. Caller
1941// holds adminTokenMu.
1942func settleInstanceEnv(f *os.File, dir, path string) error {
1943 if err := f.Chmod(0o600); err != nil {
1944 return fmt.Errorf("chmod %s: %w", path, err)
1945 }
1946 if err := instanceKeySync.file(f); err != nil {
1947 return fmt.Errorf("sync %s: %w", path, err)
1948 }
1949 if err := instanceKeySync.dir(dir); err != nil {
1950 return fmt.Errorf("sync %s: %w", dir, err)
1951 }
1952 return nil
1953}
1954
1955// ensureInstanceKey is the shared body. It holds adminTokenMu, the lock
1956// every mint into <host>.env takes, so a key append and a token append can
1957// never interleave. create false is the backfill: an absent file is
1958// InstanceKeyNoEnvFile and nothing is created.
1959func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutcome, error) {
1960 path := filepath.Join(dir, name)
1961 adminTokenMu.Lock()
1962 defer adminTokenMu.Unlock()
1963 read := func(n string) ([]byte, error) {
1964 if beneath {
1965 return instancefs.ReadBeneathLimit(dir, n, 1<<20)
1966 }
1967 return os.ReadFile(filepath.Join(dir, n))
1968 }
1969 var raw []byte
1970 var err error
1971 if beneath {
1972 raw, err = read(name)
1973 } else {
1974 raw, err = readOwnEnv(path)
1975 }
1976 switch {
1977 case errors.Is(err, os.ErrNotExist):
1978 if !create {
1979 return InstanceKeyNoEnvFile, nil
1980 }
1981 case err != nil:
1982 return InstanceKeyPresent, fmt.Errorf("read %s: %w", path, err)
1983 }
1984 cfg, err := read(configEnvName(name))
1985 if err != nil && !errors.Is(err, os.ErrNotExist) {
1986 return InstanceKeyPresent, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err)
1987 }
1988 // The decision is the effective env's: the key the launch will hand
1989 // the app, by the same rule mergeEnv applies.
1990 key, layer := instancekey.Effective(parseEnvBytes(cfg), parseEnvBytes(raw))
1991 switch layer {
1992 case instancekey.LayerConfig:
1993 logPerFile("key-config", path, fmt.Sprintf("activator: %s is set by the app's configuration for %s: the platform mints none while it is", InstanceKeyVar, path))
1994 return InstanceKeyFromConfig, nil
1995 case instancekey.LayerOwn:
1996 if !wellFormedInstanceKey(key) {
1997 logPerFile("key-shape", path, fmt.Sprintf("activator: %s in %s is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it", InstanceKeyVar, path))
1998 }
1999 // Present is not a no-op: a mint whose sync failed left these
2000 // bytes behind, and this call is the retry that makes them
2001 // durable before anything is started with them.
2002 if instanceKeyBeforeOpen != nil {
2003 instanceKeyBeforeOpen(path)
2004 }
2005 f, err := openInstanceEnv(dir, name, beneath, false)
2006 if err != nil {
2007 return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err)
2008 }
2009 defer f.Close()
2010 if err := settleInstanceEnv(f, dir, path); err != nil {
2011 return InstanceKeyPresent, err
2012 }
2013 return InstanceKeyPresent, nil
2014 }
2015 minted, err := mintInstanceKey()
2016 if err != nil {
2017 return InstanceKeyPresent, fmt.Errorf("mint instance key for %s: %w", path, err)
2018 }
2019 if instanceKeyBeforeOpen != nil {
2020 instanceKeyBeforeOpen(path)
2021 }
2022 f, err := openInstanceEnv(dir, name, beneath, create)
2023 if errors.Is(err, os.ErrNotExist) && !create {
2024 return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill
2025 }
2026 if err != nil {
2027 return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err)
2028 }
2029 defer f.Close()
2030 // The length the append starts from, taken from the open descriptor:
2031 // what a failed append is undone to. Under adminTokenMu it is
2032 // len(raw); reading it here also keeps any byte another process
2033 // appended since the read.
2034 st, err := f.Stat()
2035 if err != nil {
2036 return InstanceKeyPresent, fmt.Errorf("stat %s: %w", path, err)
2037 }
2038 prefix := ""
2039 if len(raw) > 0 && raw[len(raw)-1] != '\n' {
2040 prefix = "\n"
2041 }
2042 line := []byte(prefix + InstanceKeyVar + "=" + minted + "\n")
2043 if n, err := instanceKeyWrite(f, line); err != nil || n != len(line) {
2044 if err == nil {
2045 err = io.ErrShortWrite
2046 }
2047 return InstanceKeyPresent, undoInstanceKeyAppend(f, st.Size(), path, err)
2048 }
2049 if err := settleInstanceEnv(f, dir, path); err != nil {
2050 return InstanceKeyPresent, err
2051 }
2052 return InstanceKeyMinted, nil
2053}
2054
2055// undoInstanceKeyAppend removes a failed append's bytes: it truncates f
2056// back to size, the length before the append, and fsyncs, so a short key
2057// line can never survive to be read as present. Nothing past size was
2058// ever reported to anyone. Returns the write error, joined with any
2059// failure to undo it. Caller holds adminTokenMu.
2060func undoInstanceKeyAppend(f *os.File, size int64, path string, werr error) error {
2061 err := fmt.Errorf("write %s: %w", path, werr)
2062 if terr := f.Truncate(size); terr != nil {
2063 return errors.Join(err, fmt.Errorf("undo the partial write to %s: %w", path, terr))
2064 }
2065 if serr := instanceKeySync.file(f); serr != nil {
2066 return errors.Join(err, fmt.Errorf("sync %s after undoing the partial write: %w", path, serr))
2067 }
2068 return err
2069}
2070
2071// errInstanceEnvNeedsWrite is settleClaimInstanceEnv's answer for a file
2072// whose mode is not 0600: making it so is a write, which the claim path
2073// does only under its guard.
2074var errInstanceEnvNeedsWrite = errors.New("activator: the instance env file needs its mode set")
2075
2076// settleClaimInstanceEnv is the claim launch source's durability step for
2077// an env file that already carries what the launch needs: it makes the
2078// file and its directory durable and writes nothing. fsync changes no
2079// byte, mode or name, so it is not a host write and runs without the
2080// claim's guard. A file that is not 0600 is errInstanceEnvNeedsWrite, and
2081// the caller takes the guarded path, whose ensure sets the mode when the
2082// own file holds the key (and leaves it when the key is configured or the
2083// row is a replica: EnsureClaimAdminToken).
2084func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error {
2085 p := b.pathsFor(rt)
2086 name := rt.Host + ".env"
2087 adminTokenMu.Lock()
2088 defer adminTokenMu.Unlock()
2089 var f *os.File
2090 var err error
2091 if p.layout == LayoutNested {
2092 f, err = instancefs.OpenBeneath(p.dataDir, name, os.O_RDONLY, 0)
2093 } else {
2094 f, err = openOwnEnvNoFollow(p.envFile, os.O_RDONLY, 0)
2095 }
2096 if err != nil {
2097 return fmt.Errorf("open %s: %w", p.envFile, err)
2098 }
2099 defer f.Close()
2100 st, err := f.Stat()
2101 if err != nil {
2102 return fmt.Errorf("stat %s: %w", p.envFile, err)
2103 }
2104 if st.Mode().Perm() != 0o600 {
2105 return errInstanceEnvNeedsWrite
2106 }
2107 if err := instanceKeySync.file(f); err != nil {
2108 return fmt.Errorf("sync %s: %w", p.envFile, err)
2109 }
2110 if err := instanceKeySync.dir(p.dataDir); err != nil {
2111 return fmt.Errorf("sync %s: %w", p.dataDir, err)
2112 }
2113 return nil
2114}
2115```
2116
2117Create `internal/activator/ownenv_unix.go` (the open that refuses a link in the open itself) and `ownenv_other.go` (the windows stand-in, so the cross-compiles build; the agent never runs there):
2118
2119```go
2120//go:build unix
2121
2122package activator
2123
2124import (
2125 "errors"
2126 "os"
2127
2128 "golang.org/x/sys/unix"
2129)
2130
2131// openOwnEnvNoFollow opens a flat row's own <host>.env and refuses a
2132// symlink at it in the open itself (O_NOFOLLOW), so there is no window
2133// between a check and the open for a same-uid sibling (spec §7.2) to plant
2134// a link in. A refused link is the bare errOwnEnvSymlink: the caller names
2135// the path, once. A nested row's file needs none of this: instancefs.OpenBeneath already refuses links.
2136func openOwnEnvNoFollow(path string, flag int, perm os.FileMode) (*os.File, error) {
2137 f, err := os.OpenFile(path, flag|unix.O_NOFOLLOW, perm)
2138 if errors.Is(err, unix.ELOOP) {
2139 return nil, errOwnEnvSymlink
2140 }
2141 return f, err
2142}
2143```
2144
2145```go
2146//go:build !unix
2147
2148package activator
2149
2150import (
2151 "os"
2152)
2153
2154// openOwnEnvNoFollow is the unix helper's stand-in where there is no
2155// O_NOFOLLOW (windows). The agent only runs on Linux; this keeps the
2156// cross-compiled builds working and still refuses a link it can see, by
2157// the same Lstat check EnsureAdminToken makes. It is check-then-open, so
2158// it is weaker than the unix open, which is the one that matters.
2159func openOwnEnvNoFollow(path string, flag int, perm os.FileMode) (*os.File, error) {
2160 if fi, err := os.Lstat(path); err == nil && fi.Mode()&os.ModeSymlink != 0 {
2161 return nil, errOwnEnvSymlink
2162 }
2163 return os.OpenFile(path, flag, perm)
2164}
2165```
2166
2167- [ ] **Step 4: Run the tests to verify they pass**
2168
2169Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestReplicaRows|TestEnsureAdminToken' -count=1`
2170Expected: PASS. The admin token tests are in the run because the key mint shares their lock and log throttle.
2171
2172- [ ] **Step 5: Commit**
2173
2174```bash
2175gofmt -l internal/activator
2176git add internal/activator/instancekey.go internal/activator/instancekey_test.go
2177git commit -m "activator: mint CARLOS_INSTANCE_KEY create-only into <host>.env, durable on every call (L21)
2178
2179Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
2180```
2181
2182---
2183
2184### Task 3: Every activator launch path mints the key
2185
2186**Files:**
2187- Modify: `internal/activator/backend_exec.go:831-833` (`seedChild`), `:1369-1371` (`spawnChildFrom`)
2188- Modify: `internal/activator/prepare.go:60-62` (`prepareAt`)
2189- Test: `internal/activator/instancekey_test.go` (append), `internal/activator/instancekey_linux_test.go` (create)
2190
2191**Interfaces:**
2192- Consumes: Task 2's `ensureInstanceKeyFor`, `BackfillInstanceKey`, `probeInstanceKeySync`, `wellFormedInstanceKey`; test helpers `testExecBackend` (activator_test.go:4262), `stageRecordingLiveFullEnv` (:4712), `waitForFile` (:4741), `tmpSock` (:954), `writeMinimalSQLite` (backend_exec_test.go:343), the `restoreReplica` seam (backend_exec.go:757), `nestedBackend`, `fakeControl`, `fakeUnit` (unitproc_test.go:24-160).
2193- Produces: nothing new; the launch env of every exec start and seed carries `CARLOS_INSTANCE_KEY`, and no child starts before it is durable.
2194
2195- [ ] **Step 1: Write the failing tests**
2196
2197Append to `internal/activator/instancekey_test.go`, and add `"context"` and `"time"` to its imports:
2198
2199```go
2200// startRecorded starts host on b with the recording binary and returns the
2201// environment the child saw, or nil when no child ran within a second.
2202func startRecorded(t *testing.T, b *ExecBackend, host string) (map[string]string, error) {
2203 t.Helper()
2204 _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
2205 rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)}
2206 ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
2207 defer cancel()
2208 proc, _, err := b.Start(ctx, rt) // the fake binary never binds: the health wait fails, as in the sibling tests
2209 if proc != nil {
2210 defer proc.Stop(context.Background())
2211 }
2212 deadline := time.Now().Add(time.Second)
2213 for time.Now().Before(deadline) {
2214 if raw, rerr := os.ReadFile(envPath); rerr == nil {
2215 child := map[string]string{}
2216 for _, line := range strings.Split(string(raw), "\n") {
2217 if k, v, ok := strings.Cut(line, "="); ok {
2218 child[k] = v
2219 }
2220 }
2221 return child, err
2222 }
2223 time.Sleep(10 * time.Millisecond)
2224 }
2225 return nil, err
2226}
2227
2228// Start hands the child the key it minted, and a second Start (a restart, a
2229// redeploy) hands it the same one.
2230func TestExecBackendStartHandsTheChildTheMintedInstanceKey(t *testing.T) {
2231 b := testExecBackend(t)
2232 host := "shop.example.com"
2233 first, _ := startRecorded(t, b, host)
2234 minted, err := readEnvFile(filepath.Join(b.DataDir, host+".env"))
2235 if err != nil {
2236 t.Fatal(err)
2237 }
2238 if first == nil || !wellFormedInstanceKey(minted[InstanceKeyVar]) || first[InstanceKeyVar] != minted[InstanceKeyVar] {
2239 t.Fatal("the child was not handed the key minted into <host>.env")
2240 }
2241 if second, _ := startRecorded(t, b, host); second == nil || second[InstanceKeyVar] != minted[InstanceKeyVar] {
2242 t.Fatal("a second start handed the child a different key")
2243 }
2244}
2245
2246// Finding 2 through a launch: while the key's sync fails, Start fails and
2247// no child runs. Once the sync succeeds, the child is handed the key the
2248// failed attempt wrote, never a second one.
2249func TestExecBackendStartStartsNoChildUntilTheKeyIsSynced(t *testing.T) {
2250 for _, tc := range []struct {
2251 name string
2252 fail func(*syncProbe, error)
2253 }{
2254 {"file sync", func(p *syncProbe, err error) { p.fileErr = err }},
2255 {"directory sync", func(p *syncProbe, err error) { p.dirErr = err }},
2256 } {
2257 t.Run(tc.name, func(t *testing.T) {
2258 probe := probeInstanceKeySync(t)
2259 b := testExecBackend(t)
2260 host := "shop.example.com"
2261 tc.fail(probe, errors.New("injected: EIO"))
2262 for i := 0; i < 2; i++ {
2263 child, err := startRecorded(t, b, host)
2264 if err == nil || !strings.Contains(err.Error(), "instance key for "+host) {
2265 t.Fatalf("start %d with a failing sync: err = %v, want the instance key's error", i, err)
2266 }
2267 if child != nil {
2268 t.Fatalf("start %d ran a child before the key was durable", i)
2269 }
2270 }
2271 written, _ := readEnvFile(filepath.Join(b.DataDir, host+".env"))
2272 tc.fail(probe, nil)
2273 child, _ := startRecorded(t, b, host)
2274 if child == nil || child[InstanceKeyVar] == "" || child[InstanceKeyVar] != written[InstanceKeyVar] {
2275 t.Fatal("after the sync succeeded the child was not handed the key the failed attempt wrote")
2276 }
2277 })
2278 }
2279}
2280
2281// Finding 4 through a launch: <host>.env and <host>.config.env together.
2282// An empty override never hands the child an empty key while the
2283// configuration supplies one, and no key is minted over the configured one.
2284// With neither supplying one, the child gets the minted key.
2285func TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists(t *testing.T) {
2286 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
2287 for _, tc := range []struct {
2288 name, own, config string
2289 configured bool
2290 }{
2291 {"empty override, configured key", tok + InstanceKeyVar + "=\n", InstanceKeyVar + "=from-config\n", true},
2292 {"no override, configured key", tok, InstanceKeyVar + "=from-config\n", true},
2293 {"empty override, empty configuration", tok + InstanceKeyVar + "=\n", InstanceKeyVar + "=\n", false},
2294 } {
2295 t.Run(tc.name, func(t *testing.T) {
2296 b := testExecBackend(t)
2297 host := "shop.example.com"
2298 own := filepath.Join(b.DataDir, host+".env")
2299 if err := os.WriteFile(own, []byte(tc.own), 0o600); err != nil {
2300 t.Fatal(err)
2301 }
2302 if err := os.WriteFile(filepath.Join(b.DataDir, host+".config.env"), []byte(tc.config), 0o600); err != nil {
2303 t.Fatal(err)
2304 }
2305 child, _ := startRecorded(t, b, host)
2306 if child == nil {
2307 t.Fatal("no child ran")
2308 }
2309 raw, _ := os.ReadFile(own)
2310 if tc.configured {
2311 if child[InstanceKeyVar] != "from-config" {
2312 t.Fatalf("the child was handed %q, want the configured key", child[InstanceKeyVar])
2313 }
2314 if string(raw) != tc.own {
2315 t.Fatal("a key was minted over the configured one")
2316 }
2317 return
2318 }
2319 env, _ := readEnvFile(own)
2320 if !wellFormedInstanceKey(child[InstanceKeyVar]) || child[InstanceKeyVar] != env[InstanceKeyVar] {
2321 t.Fatal("the child was not handed the minted key")
2322 }
2323 })
2324 }
2325}
2326
2327// Coverage (Astra round 1): the first process of a fresh flat instance is
2328// often its seed. The seed mints the key, sees it, and the first start
2329// hands the instance the same one.
2330func TestExecBackendSeedMintsTheKeyForAFreshFlatInstance(t *testing.T) {
2331 b := testExecBackend(t)
2332 host := "seeded.example.com"
2333 _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
2334 if err := b.Seed(context.Background(), registry.Route{Host: host, Kind: "instance", Seed: `["seed"]`}); err != nil {
2335 t.Fatalf("Seed: %v", err)
2336 }
2337 seed := map[string]string{}
2338 for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") {
2339 if k, v, ok := strings.Cut(line, "="); ok {
2340 seed[k] = v
2341 }
2342 }
2343 own, _ := readEnvFile(filepath.Join(b.DataDir, host+".env"))
2344 if !wellFormedInstanceKey(own[InstanceKeyVar]) || seed[InstanceKeyVar] != own[InstanceKeyVar] {
2345 t.Fatal("the seed did not mint the key and see it")
2346 }
2347 if child, _ := startRecorded(t, b, host); child == nil || child[InstanceKeyVar] != own[InstanceKeyVar] {
2348 t.Fatal("the first start handed the instance a different key from its seed's")
2349 }
2350}
2351
2352// Coverage (Astra round 1): a hibernation wake restores the database and
2353// leaves <host>.env alone, so the start after the restore hands the
2354// instance the key it had.
2355func TestHibernationRestoreKeepsTheInstanceKey(t *testing.T) {
2356 b := testExecBackend(t)
2357 host := "sleepy.example.com"
2358 if child, _ := startRecorded(t, b, host); child == nil {
2359 t.Fatal("no child ran")
2360 }
2361 envPath := filepath.Join(b.DataDir, host+".env")
2362 before, _ := os.ReadFile(envPath)
2363 key := parseEnvBytes(before)[InstanceKeyVar]
2364 if !wellFormedInstanceKey(key) {
2365 t.Fatal("the first start minted no key")
2366 }
2367 old := restoreReplica
2368 restoreReplica = func(_ context.Context, dbPath, bucket, prefix, region string) (bool, error) {
2369 return true, writeMinimalSQLite(dbPath)
2370 }
2371 t.Cleanup(func() { restoreReplica = old })
2372 _ = os.Remove(filepath.Join(b.DataDir, host+".db")) // hibernated: the database is gone
2373 rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)}
2374 if fresh, err := b.Restore(context.Background(), rt); err != nil || fresh {
2375 t.Fatalf("Restore = fresh %v, %v; want a restored database", fresh, err)
2376 }
2377 if after, _ := os.ReadFile(envPath); !bytes.Equal(before, after) {
2378 t.Fatal("the restore changed <host>.env")
2379 }
2380 if child, _ := startRecorded(t, b, host); child == nil || child[InstanceKeyVar] != key {
2381 t.Fatal("the start after a restore handed the instance a different key")
2382 }
2383}
2384```
2385
2386Create `internal/activator/instancekey_linux_test.go`:
2387
2388```go
2389package activator
2390
2391import (
2392 "os"
2393 "path/filepath"
2394 "strings"
2395 "testing"
2396 "time"
2397
2398 "golang.org/x/sys/unix"
2399
2400 "github.com/carlosframework/platform/internal/instancectl"
2401 "github.com/carlosframework/platform/internal/provision"
2402)
2403
2404// The nested prepare mints the key into the instance directory's own env
2405// file and hands it to the role in the same env it returns: the run role
2406// and the seed role (coverage, Astra round 1).
2407func TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile(t *testing.T) {
2408 for _, role := range []instancectl.Role{instancectl.RoleRun, instancectl.RoleSeed} {
2409 t.Run(string(role), func(t *testing.T) {
2410 b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
2411 _, ctl, logs, env, err := b.prepare(rt, role)
2412 if err != nil {
2413 t.Fatalf("prepare: %v", err)
2414 }
2415 logs.Close()
2416 if ctl != nil {
2417 ctl.Close()
2418 }
2419 own, err := readEnvFileBeneath(provision.InstanceDir(b.DataDir, rt.Host), rt.Host+".env")
2420 if err != nil {
2421 t.Fatal(err)
2422 }
2423 if !wellFormedInstanceKey(own[InstanceKeyVar]) || env[InstanceKeyVar] != own[InstanceKeyVar] {
2424 t.Fatal("prepare's env does not carry the key minted into the nested env file")
2425 }
2426 })
2427 }
2428}
2429
2430// Finding 3: the nested backfill opens an existing file with mode 0
2431// (openat2 refuses a mode without O_CREAT), appends, and sets 0600 on the
2432// descriptor.
2433func TestBackfillInstanceKeyAppendsToANestedEnvFile(t *testing.T) {
2434 b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
2435 dir := provision.InstanceDir(b.DataDir, rt.Host)
2436 if err := os.MkdirAll(dir, 0o700); err != nil {
2437 t.Fatal(err)
2438 }
2439 envPath := filepath.Join(dir, rt.Host+".env")
2440 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("e", 64) + "\n"
2441 if err := os.WriteFile(envPath, []byte(tok), 0o644); err != nil {
2442 t.Fatal(err)
2443 }
2444 if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyMinted {
2445 t.Fatalf("nested backfill = %v, %v; want minted", got, err)
2446 }
2447 raw, _ := os.ReadFile(envPath)
2448 if !strings.HasPrefix(string(raw), tok) || len(keyLines(raw)) != 1 || !wellFormedInstanceKey(keyLines(raw)[0]) {
2449 t.Fatalf("nested env file after the backfill: %d bytes, %d keys", len(raw), len(keyLines(raw)))
2450 }
2451 if st, _ := os.Stat(envPath); st.Mode().Perm() != 0o600 {
2452 t.Fatalf("mode = %o, want 600", st.Mode().Perm())
2453 }
2454 if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyPresent {
2455 t.Fatalf("second nested backfill = %v, %v; want present", got, err)
2456 }
2457 if raw2, _ := os.ReadFile(envPath); string(raw2) != string(raw) {
2458 t.Fatal("the second backfill changed the file")
2459 }
2460}
2461
2462// A nested own env file is on tenant-writable ground: a symlink planted at
2463// its name is refused, and the file it points at is not touched.
2464func TestEnsureInstanceKeyBeneathRefusesASymlink(t *testing.T) {
2465 dir := t.TempDir()
2466 target := filepath.Join(t.TempDir(), "elsewhere.env")
2467 if err := os.WriteFile(target, []byte("X=1\n"), 0o644); err != nil {
2468 t.Fatal(err)
2469 }
2470 if err := os.Symlink(target, filepath.Join(dir, "shop.example.com.env")); err != nil {
2471 t.Fatal(err)
2472 }
2473 if _, err := EnsureInstanceKeyBeneath(dir, "shop.example.com.env"); err == nil {
2474 t.Fatal("ensure followed a symlink planted at the env file's name")
2475 }
2476 if raw, _ := os.ReadFile(target); string(raw) != "X=1\n" {
2477 t.Fatalf("the symlink's target changed: %q", raw)
2478 }
2479}
2480
2481// Review focus 4: a FIFO planted at a nested env file's name fails the
2482// backfill fast; it never blocks agent start.
2483func TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking(t *testing.T) {
2484 b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
2485 dir := provision.InstanceDir(b.DataDir, rt.Host)
2486 if err := os.MkdirAll(dir, 0o700); err != nil {
2487 t.Fatal(err)
2488 }
2489 if err := unix.Mkfifo(filepath.Join(dir, rt.Host+".env"), 0o600); err != nil {
2490 t.Fatal(err)
2491 }
2492 done := make(chan error, 1)
2493 go func() {
2494 _, err := b.BackfillInstanceKey(rt)
2495 done <- err
2496 }()
2497 select {
2498 case err := <-done:
2499 if err == nil {
2500 t.Fatal("backfill over a FIFO succeeded, want an error")
2501 }
2502 case <-time.After(2 * time.Second):
2503 t.Fatal("backfill blocked on a FIFO")
2504 }
2505}
2506```
2507
2508- [ ] **Step 2: Run them to verify they fail**
2509
2510Run (sandbox off: these start children and bind sockets): `nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartHandsTheChild|TestExecBackendStartStartsNoChild|TestExecBackendStartNeverHands|TestExecBackendSeedMintsTheKey|TestHibernationRestoreKeeps|TestPrepareMintsTheInstanceKey|TestBackfillInstanceKeyAppendsToANested|TestEnsureInstanceKeyBeneathRefuses|TestBackfillInstanceKeyRefusesAFIFO' -count=1 -v`
2511Expected (checked in the scratch build): FAIL for `TestExecBackendStartHandsTheChildTheMintedInstanceKey`, both `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced` cases, the "empty override, empty configuration" case of `TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists` (its two configured cases already pass, from Task 1's `mergeEnv`), `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`, `TestHibernationRestoreKeepsTheInstanceKey` and both roles of `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile`. The nested backfill, symlink and FIFO tests already pass, from Task 2's body; they pin the nested path.
2512
2513- [ ] **Step 3: Call the mint on each launch path**
2514
2515In `internal/activator/backend_exec.go`, `seedChild`, after the admin token mint:
2516
2517```go
2518 if err := EnsureAdminToken(paths.envFile); err != nil {
2519 return fmt.Errorf("admin token for %s: %w", host, err)
2520 }
2521 // The instance key, before the env is read for the same reason: a seed
2522 // is usually the first process of a fresh instance (L21).
2523 if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil {
2524 return fmt.Errorf("instance key for %s: %w", host, err)
2525 }
2526 fileEnv, err := b.instanceEnv(rt)
2527```
2528
2529In `spawnChildFrom`, after the admin token mint:
2530
2531```go
2532 if err := EnsureAdminToken(paths.envFile); err != nil {
2533 return nil, "", fmt.Errorf("admin token for %s: %w", host, err)
2534 }
2535 // The instance key is minted here for the same reason and fails the
2536 // wake the same way: an app that takes its key from the platform
2537 // refuses to start without one (L21). It returns only once the key is
2538 // durable, so no child starts with a key a power cut could take back.
2539 if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil {
2540 return nil, "", fmt.Errorf("instance key for %s: %w", host, err)
2541 }
2542```
2543
2544In `internal/activator/prepare.go`, `prepareAt`, after `EnsureAdminTokenBeneath`:
2545
2546```go
2547 if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil {
2548 return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err)
2549 }
2550 if _, err := b.ensureInstanceKeyFor(rt, p); err != nil {
2551 return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err)
2552 }
2553```
2554
2555- [ ] **Step 4: Run the tests, then the package's launch tests**
2556
2557Run (sandbox off): the Step 2 command.
2558Expected: PASS.
2559
2560Run (sandbox off): `nice -n 19 go test ./internal/activator/ -run 'Seed|Start|Prepare|LaunchEnvironment|Control|InstanceKey|AdminToken' -count=1`
2561Expected: PASS. If an existing test compares a child's whole environment or an env file's exact bytes after a start or seed, it now sees one more line, `CARLOS_INSTANCE_KEY=...`. Update that assertion to allow the key line (assert it is well formed). Do not weaken any other check, and list each test you changed in the commit message. (None needed changing in the scratch build.)
2562
2563- [ ] **Step 5: Commit**
2564
2565```bash
2566gofmt -l internal/activator
2567git add internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/instancekey_test.go internal/activator/instancekey_linux_test.go
2568git commit -m "activator: seed, start and nested prepare mint the instance key durably before the env is read (L21)
2569
2570Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
2571```
2572
2573---
2574
2575### Task 3b: A replica gets exactly its home's key, or none and a refusal (ruling 26)
2576
2577**Files:**
2578- Create: `internal/activator/replicakey.go`, `internal/host/replicahomekeys.go`, `internal/appconfig/instancekey.go`
2579- Modify: `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ConfiguredKey`; the gate in `seedChild`, `spawnChildFrom`, `StartWorkerStaged` and `startSidecarChildFrom`), `internal/activator/prepare.go` (`prepareAt`), `internal/host/agent.go` (the wiring)
2580- Test: `internal/activator/replica_instancekey_test.go`, `internal/activator/replica_instancekey_linux_test.go`, `internal/activator/replicakey_race_test.go`, `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go`
2581
2582**Interfaces:**
2583- Consumes: Task 1's `instancekey.Digest`, `Effective` and `ParseHomeKeys`; appconfig's `LoadMergedHostVersions`, `RenderEnv`, `routeBinding`, and test kit `execKit`, `SetEnv`, `ApplyInstance`, `Seal`; Task 2's `InstanceKeyReplica` (never minted); test helpers `testExecBackend`, `tmpSock`, `nestedBackend`, `readEnvFile`.
2584- Produces: `ExecBackend.ReplicaHomeKeys func(host string) (sha256 string, ok bool)` and `ConfiguredKey func(ctx, rt) (sha256 string, err error)`; `(*ExecBackend).replicaKeyGate(rt, env, layer, haveControl) error`, with `layer` the one the env's key came from, decided by the same read that composed the env (`instanceEnvLayer`, ruling 29); `activator.ErrReplicaKey`; `appconfig.ConfiguredKeyDigest(ctx, s, priv, rt) (string, error)`; `host.replicaHomeKeys(read)`, which parses the list the reader returns (`/etc/carlos/replica-home-keys`, read again at every replica launch; ruling 30), and `host.configuredKey(getenv, open)`, the fresh read (ruling 27). Task 4's snapshot calls the gate.
2585
2586**Built:** two launch paths rev 5.1 left ungated pass the gate as a seed does (no control socket: the home's verified key or none, refused otherwise), by the ledger's rulings on the 3b build and its review. The sidecar worker (`StartWorkerStaged`) reads the env and its key's layer in one read, and when the agent-issued `extra` carries `CARLOS_INSTANCE_KEY` the layer is set to none, so the gate refuses it as a key of unknown origin: a worker never mints and `extra` has no business carrying a key. The legacy co-spawned sidecar (`startSidecarChildFrom`) gates its own env file's key the same way (`sidecarKeyLayer`). Refusal and log text name the file path, not the variable the plan calls the list by. The bucket read in the gate runs on its own bounded context (`prepareAt` has none to thread).
2587
2588- [ ] **Step 1: Write the failing tests**
2589
2590The app these tests launch is the test binary itself, running `TestInstanceKeyHelperProcess`: rastrillo e3ea08de's resolution and its documented caller. So each case records what an adopting app actually did ("key <sha256>", "refused" or "minted"), not only which variables it was handed. "No control socket" is a socket path too long to bind, so the launch withholds `CARLOS_CONTROL_SOCKET`, exactly as a claim launch before O6 does.
2591
2592Create `internal/activator/replica_instancekey_test.go`:
2593
2594```go
2595package activator
2596
2597import (
2598 "context"
2599 "errors"
2600 "fmt"
2601 "os"
2602 "path/filepath"
2603 "strings"
2604 "testing"
2605 "time"
2606
2607 "github.com/carlosframework/platform/internal/instancekey"
2608 "github.com/carlosframework/platform/internal/registry"
2609)
2610
2611// The app the replica tests launch is this test binary, running
2612// TestInstanceKeyHelperProcess: rastrillo e3ea08de's carlos.InstanceKey
2613// and its documented caller, so a test sees what an adopting app does, not
2614// only which variables it was handed.
2615const helperOutEnv = "GO_TEST_INSTANCE_KEY_HELPER_OUT"
2616
2617var (
2618 errHelperNoInstanceKey = errors.New("ErrNoInstanceKey")
2619 errHelperNotOnCarlos = errors.New("ErrNotOnCarlos")
2620)
2621
2622// helperInstanceKey is rastrillo e3ea08de's InstanceKey(own) with Running():
2623// own, then $CARLOS_INSTANCE_KEY verbatim, then ErrNoInstanceKey when
2624// $CARLOS_CONTROL_SOCKET is set, else ErrNotOnCarlos.
2625func helperInstanceKey(own string) (string, error) {
2626 if own != "" {
2627 return own, nil
2628 }
2629 if k := os.Getenv(InstanceKeyVar); k != "" {
2630 return k, nil
2631 }
2632 if os.Getenv("CARLOS_CONTROL_SOCKET") != "" {
2633 return "", errHelperNoInstanceKey
2634 }
2635 return "", errHelperNotOnCarlos
2636}
2637
2638// TestInstanceKeyHelperProcess is the app, not a test: with no output path
2639// set it does nothing. It resolves its key as an adopting app with no key
2640// of its own does, records the outcome, and exits:
2641// - "key <sha256>": it sealed with the key it was handed;
2642// - "refused": ErrNoInstanceKey, it did not start;
2643// - "minted": ErrNotOnCarlos, and the documented caller minted its own.
2644func TestInstanceKeyHelperProcess(t *testing.T) {
2645 out := os.Getenv(helperOutEnv)
2646 if out == "" {
2647 return
2648 }
2649 var outcome string
2650 switch key, err := helperInstanceKey(""); {
2651 case err == nil:
2652 outcome = "key " + instancekey.Digest(key)
2653 case errors.Is(err, errHelperNoInstanceKey):
2654 outcome = "refused"
2655 default:
2656 outcome = "minted"
2657 }
2658 _ = os.WriteFile(out+".tmp", []byte(outcome), 0o600)
2659 _ = os.Rename(out+".tmp", out)
2660 os.Exit(0)
2661}
2662
2663// stageHelperApp makes this test binary host's live build, running only
2664// TestInstanceKeyHelperProcess, and returns where its outcome lands.
2665func stageHelperApp(t *testing.T, b *ExecBackend, host, ver string) string {
2666 t.Helper()
2667 self, err := os.Executable()
2668 if err != nil {
2669 t.Fatal(err)
2670 }
2671 out := filepath.Join(t.TempDir(), "outcome")
2672 t.Setenv(helperOutEnv, out)
2673 vdir := filepath.Join(b.OptDir, "versions", ver)
2674 if err := os.MkdirAll(vdir, 0o755); err != nil {
2675 t.Fatal(err)
2676 }
2677 bin := filepath.Join(vdir, "app")
2678 script := fmt.Sprintf("#!/bin/sh\nexec '%s' -test.run='^TestInstanceKeyHelperProcess$'\n", self)
2679 if err := os.WriteFile(bin, []byte(script), 0o755); err != nil {
2680 t.Fatal(err)
2681 }
2682 if err := os.MkdirAll(filepath.Join(b.OptDir, "live"), 0o755); err != nil {
2683 t.Fatal(err)
2684 }
2685 os.Remove(b.liveLink(host))
2686 if err := os.Symlink(bin, b.liveLink(host)); err != nil {
2687 t.Fatal(err)
2688 }
2689 return out
2690}
2691
2692// outcomeOf waits up to two seconds for the helper's outcome; "" is no app
2693// ran at all.
2694func outcomeOf(path string) string {
2695 deadline := time.Now().Add(2 * time.Second)
2696 for time.Now().Before(deadline) {
2697 if raw, err := os.ReadFile(path); err == nil {
2698 return string(raw)
2699 }
2700 time.Sleep(10 * time.Millisecond)
2701 }
2702 return ""
2703}
2704
2705// noControlSocket is a socket path too long to bind, so the launch's
2706// control listener fails and CARLOS_CONTROL_SOCKET is withheld: the same
2707// launch a claim box runs before O6, and a flat launch whose bind failed.
2708func noControlSocket(t *testing.T) string {
2709 return filepath.Join(t.TempDir(), strings.Repeat("s", 120)+".sock")
2710}
2711
2712type replicaCase struct {
2713 name string
2714 replica bool
2715 own, config string // env file bodies ("" = no config file)
2716 homeKey string // the home's effective key the box lists ("" = none listed)
2717 start string // "refused" (no app runs) or the helper outcome wanted
2718 wantKeyFromOwnOK bool // a home row: the outcome is the key minted into <host>.env
2719 // bucket is the key the replica's configuration in the bucket renders
2720 // now (ConfiguredKey; "" renders none). bucketDown makes that read
2721 // fail; noBucket leaves ConfiguredKey nil.
2722 bucket string
2723 bucketDown, noBucket bool
2724}
2725
2726// freshConfig is ExecBackend.ConfiguredKey for one case.
2727func freshConfig(tc replicaCase) func(context.Context, registry.Route) (string, error) {
2728 if tc.noBucket {
2729 return nil
2730 }
2731 return func(context.Context, registry.Route) (string, error) {
2732 switch {
2733 case tc.bucketDown:
2734 return "", errors.New("bucket unreachable")
2735 case tc.bucket == "":
2736 return "", nil
2737 }
2738 return instancekey.Digest(tc.bucket), nil
2739 }
2740}
2741
2742// Rulings 26 to 28 through a real launch and the helper (Astra rounds 4 and
2743// 5): a replica runs only with exactly its home's key, listed by the
2744// pre-roll check and, when configured, still the bucket's configured key;
2745// or with none and a control socket its helper refuses on. Every launch
2746// here has no control socket, which is where the helper would otherwise
2747// mint.
2748func TestReplicaLaunchWithoutAControlSocket(t *testing.T) {
2749 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
2750 homeKey := strings.Repeat("H", 43) + "="
2751 for _, tc := range []replicaCase{
2752 {name: "a keyless replica is refused by the agent, not left to mint", replica: true, own: tok, start: "refused"},
2753 {name: "a configured key that is the home's runs and seals with it", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, bucket: homeKey, start: "key " + instancekey.Digest(homeKey)},
2754 {name: "a pre-existing env key that is the home's runs", replica: true, own: tok + InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, start: "key " + instancekey.Digest(homeKey)},
2755 {name: "a configured key that differs is refused", replica: true, own: tok, config: InstanceKeyVar + "=configured-B\n", homeKey: homeKey, bucket: "configured-B", start: "refused"},
2756 {name: "a configured key with no verified home key is refused", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", bucket: homeKey, start: "refused"},
2757 // Ruling 27, Astra round 5 finding 1: the key changed in the bucket
2758 // to B and the home already holds it, while this box's sync is
2759 // behind and the replica's configuration and list entry still say A.
2760 {name: "the home updated, this replica's configuration delayed: refused", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, bucket: "configured-B", start: "refused"},
2761 {name: "the bucket cannot be read: refused", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, bucketDown: true, start: "refused"},
2762 {name: "no fresh reader: refused", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, noBucket: true, start: "refused"},
2763 // Ruling 27's residual, documented: a change the bucket does not
2764 // hold (the home's own <host>.env edited by hand, or a per-host key
2765 // configured on the home only) is invisible here. The replica's
2766 // list entry and the bucket both still say A, so it runs with A.
2767 // Only the procedure, or O8's fresh evidence, closes this.
2768 {name: "residual: a change only on the home is not seen, and the replica runs with the old key", replica: true, own: tok, config: InstanceKeyVar + "=" + homeKey + "\n", homeKey: homeKey, bucket: homeKey, start: "key " + instancekey.Digest(homeKey)},
2769 // Ruling 28: a home whose key exists only in configuration, with a
2770 // replica that is handed none (its own overlay blanks it). O8 never
2771 // delivers a configured key, so the replica stays keyless and is
2772 // refused here, where it has no control socket.
2773 {name: "a config-only home's keyless replica is refused", replica: true, own: tok, config: InstanceKeyVar + "=\n", homeKey: homeKey, bucket: "", start: "refused"},
2774 {name: "a home row mints and its app seals with the minted key", own: tok, wantKeyFromOwnOK: true},
2775 } {
2776 t.Run(tc.name, func(t *testing.T) {
2777 b := testExecBackend(t)
2778 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: tc.replica, Socket: noControlSocket(t)}
2779 own := filepath.Join(b.DataDir, rt.Host+".env")
2780 if err := os.WriteFile(own, []byte(tc.own), 0o600); err != nil {
2781 t.Fatal(err)
2782 }
2783 if tc.config != "" {
2784 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(tc.config), 0o600); err != nil {
2785 t.Fatal(err)
2786 }
2787 }
2788 if tc.homeKey != "" {
2789 b.ReplicaHomeKeys = func(h string) (string, bool) {
2790 return instancekey.Digest(tc.homeKey), h == rt.Host
2791 }
2792 }
2793 b.ConfiguredKey = freshConfig(tc)
2794 out := stageHelperApp(t, b, rt.Host, "v1")
2795 ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
2796 defer cancel()
2797 proc, _, err := b.Start(ctx, rt)
2798 if proc != nil {
2799 defer proc.Stop(context.Background())
2800 }
2801 got := outcomeOf(out)
2802 if tc.wantKeyFromOwnOK {
2803 env, _ := readEnvFile(own)
2804 if got != "key "+instancekey.Digest(env[InstanceKeyVar]) {
2805 t.Fatalf("home row: outcome %q, want the minted key", got)
2806 }
2807 return
2808 }
2809 if tc.start == "refused" {
2810 if !errors.Is(err, ErrReplicaKey) || got != "" {
2811 t.Fatalf("Start = %v, app outcome %q; want ErrReplicaKey and no app", err, got)
2812 }
2813 } else if got != tc.start {
2814 t.Fatalf("app outcome %q, want %q (Start: %v)", got, tc.start, err)
2815 }
2816 if raw, _ := os.ReadFile(own); string(raw) != tc.own {
2817 t.Fatal("a replica launch changed its own env file: a replica never mints")
2818 }
2819 })
2820 }
2821}
2822
2823// A replica's seed has no control socket: a keyless replica's seed is
2824// refused before it runs; one handed its home's key runs.
2825func TestReplicaSeedIsGated(t *testing.T) {
2826 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
2827 homeKey := strings.Repeat("H", 43) + "="
2828 for _, tc := range []struct {
2829 name, config string
2830 want string
2831 }{
2832 {"keyless", "", ""},
2833 {"the home's key", InstanceKeyVar + "=" + homeKey + "\n", "key " + instancekey.Digest(homeKey)},
2834 } {
2835 t.Run(tc.name, func(t *testing.T) {
2836 b := testExecBackend(t)
2837 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true, Seed: `["seed"]`}
2838 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte(tok), 0o600); err != nil {
2839 t.Fatal(err)
2840 }
2841 if tc.config != "" {
2842 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(tc.config), 0o600); err != nil {
2843 t.Fatal(err)
2844 }
2845 }
2846 b.ReplicaHomeKeys = func(h string) (string, bool) { return instancekey.Digest(homeKey), true }
2847 b.ConfiguredKey = freshConfig(replicaCase{bucket: homeKey})
2848 out := stageHelperApp(t, b, rt.Host, "v1")
2849 err := b.Seed(context.Background(), rt)
2850 got := outcomeOf(out)
2851 if tc.want == "" {
2852 if !errors.Is(err, ErrReplicaKey) || got != "" {
2853 t.Fatalf("Seed = %v, outcome %q; want ErrReplicaKey and no seed process", err, got)
2854 }
2855 return
2856 }
2857 if err != nil || got != tc.want {
2858 t.Fatalf("Seed = %v, outcome %q; want %q", err, got, tc.want)
2859 }
2860 })
2861 }
2862}
2863```
2864
2865Create `internal/activator/replica_instancekey_linux_test.go`:
2866
2867```go
2868package activator
2869
2870import (
2871 "context"
2872 "errors"
2873 "net"
2874 "os"
2875 "path/filepath"
2876 "strings"
2877 "testing"
2878 "time"
2879
2880 "github.com/carlosframework/platform/internal/instancectl"
2881 "github.com/carlosframework/platform/internal/instancekey"
2882 "github.com/carlosframework/platform/internal/provision"
2883 "github.com/carlosframework/platform/internal/registry"
2884)
2885
2886// socketsAllowed reports whether this process may bind a unix socket (the
2887// sandbox refuses it).
2888func socketsAllowed(t *testing.T) bool {
2889 dir, err := os.MkdirTemp("", "ik") // short: a test's own temp dir can exceed a socket path's limit
2890 if err != nil {
2891 t.Fatal(err)
2892 }
2893 defer os.RemoveAll(dir)
2894 l, err := net.Listen("unix", filepath.Join(dir, "p.sock"))
2895 if err != nil {
2896 return false
2897 }
2898 l.Close()
2899 return true
2900}
2901
2902// Ruling 26 with a control socket: a keyless replica runs, and the app's
2903// helper refuses at step 3 (ErrNoInstanceKey) instead of minting.
2904func TestKeylessReplicaWithAControlSocketLeavesTheHelperToRefuse(t *testing.T) {
2905 if !socketsAllowed(t) {
2906 t.Skip("needs unix sockets (run with the sandbox off)")
2907 }
2908 b := testExecBackend(t)
2909 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true, Socket: tmpSock(t)}
2910 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("CARLOS_ADMIN_TOKEN="+strings.Repeat("c", 64)+"\n"), 0o600); err != nil {
2911 t.Fatal(err)
2912 }
2913 out := stageHelperApp(t, b, rt.Host, "v1")
2914 ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
2915 defer cancel()
2916 proc, _, _ := b.Start(ctx, rt)
2917 if proc != nil {
2918 defer proc.Stop(context.Background())
2919 }
2920 if got := outcomeOf(out); got != "refused" {
2921 t.Fatalf("app outcome %q, want the helper's refusal", got)
2922 }
2923}
2924
2925// Ruling 26 on the nested layout: the seed role has no control socket, so
2926// a keyless replica's seed is refused at prepare; a run role handed a key
2927// that is not the home's is refused too; one handed the home's key is
2928// prepared with it.
2929func TestNestedReplicaPrepareIsGated(t *testing.T) {
2930 homeKey := strings.Repeat("H", 43) + "="
2931 for _, tc := range []struct {
2932 name, config string
2933 role instancectl.Role
2934 refused bool
2935 }{
2936 {"a keyless seed", "", instancectl.RoleSeed, true},
2937 {"a run role with another key", InstanceKeyVar + "=configured-B\n", instancectl.RoleRun, true},
2938 {"a run role with the home's key", InstanceKeyVar + "=" + homeKey + "\n", instancectl.RoleRun, false},
2939 } {
2940 t.Run(tc.name, func(t *testing.T) {
2941 b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
2942 rt.Replica = true
2943 b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true }
2944 b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil }
2945 dir := provision.InstanceDir(b.DataDir, rt.Host)
2946 if err := os.MkdirAll(dir, 0o700); err != nil {
2947 t.Fatal(err)
2948 }
2949 if tc.config != "" {
2950 if err := os.WriteFile(filepath.Join(dir, rt.Host+".config.env"), []byte(tc.config), 0o600); err != nil {
2951 t.Fatal(err)
2952 }
2953 }
2954 _, ctl, logs, env, err := b.prepare(rt, tc.role)
2955 if tc.refused {
2956 if !errors.Is(err, ErrReplicaKey) {
2957 t.Fatalf("prepare = %v, want ErrReplicaKey", err)
2958 }
2959 return
2960 }
2961 if err != nil {
2962 t.Fatalf("prepare: %v", err)
2963 }
2964 logs.Close()
2965 if ctl != nil {
2966 ctl.Close()
2967 }
2968 if env[InstanceKeyVar] != homeKey {
2969 t.Fatal("the prepared env does not carry the home's key")
2970 }
2971 })
2972 }
2973}
2974```
2975
2976Create `internal/appconfig/instancekey_test.go`:
2977
2978```go
2979package appconfig
2980
2981import (
2982 "testing"
2983
2984 "github.com/carlosframework/platform/internal/instancekey"
2985 "github.com/carlosframework/platform/internal/registry"
2986)
2987
2988// Ruling 27's fresh check: the digest of the key the bucket's
2989// configuration renders now, from the app's bundle or the host's overlay,
2990// plain or sealed; "" when it renders none.
2991func TestConfiguredKeyDigestReadsTheBucketNow(t *testing.T) {
2992 ctx, s, _, _, priv, pub := execKit(t)
2993 pk, _ := ParsePrivateKey(priv, "test")
2994 pubKey, _ := ParsePublicKey(pub, "test")
2995 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", App: "shop", Account: "acct", Replica: true}
2996 check := func(want string) {
2997 t.Helper()
2998 got, err := ConfiguredKeyDigest(ctx, s, pk, rt)
2999 if err != nil || got != want {
3000 t.Fatalf("ConfiguredKeyDigest = %q, %v; want %q", got, err, want)
3001 }
3002 }
3003 check("")
3004 if err := SetEnv(ctx, s, "acct", "shop", map[string]string{"CARLOS_INSTANCE_KEY": "key-A"}); err != nil {
3005 t.Fatal(err)
3006 }
3007 check(instancekey.Digest("key-A"))
3008 if err := SetEnv(ctx, s, "acct", "shop", map[string]string{"CARLOS_INSTANCE_KEY": "key-B"}); err != nil {
3009 t.Fatal(err)
3010 }
3011 check(instancekey.Digest("key-B"))
3012 sealed, err := Seal(pubKey, Binding{Account: "acct", App: "shop"}, []byte("key-C"))
3013 if err != nil {
3014 t.Fatal(err)
3015 }
3016 if _, _, err := ApplyInstance(ctx, s, "acct", "shop", rt.Host, map[string]Var{"CARLOS_INSTANCE_KEY": {Kind: "secret", Envelope: sealed}}, nil); err != nil {
3017 t.Fatal(err)
3018 }
3019 check(instancekey.Digest("key-C"))
3020}
3021```
3022
3023Create `internal/activator/replicakey_race_test.go`, ruling 29's deterministic interleaving:
3024
3025```go
3026package activator
3027
3028import (
3029 "context"
3030 "errors"
3031 "os"
3032 "path/filepath"
3033 "strings"
3034 "testing"
3035 "time"
3036
3037 "github.com/carlosframework/platform/internal/instancekey"
3038 "github.com/carlosframework/platform/internal/registry"
3039)
3040
3041// Ruling 29 (Astra round 6 finding 1), deterministic: the launch reads a
3042// configured key A, and then a config sync removes the key from this box's
3043// <host>.config.env (the bucket no longer holds it) before the replica
3044// gate runs. The gate decides on the layer the launch's own read found, so
3045// it still consults the bucket, which now renders no key, and refuses. A
3046// second read of the files would have found no key, skipped the bucket
3047// and admitted a child that is handed A.
3048func TestReplicaGateDecidesOnTheLaunchesOwnRead(t *testing.T) {
3049 tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
3050 keyA := strings.Repeat("A", 43) + "="
3051 b := testExecBackend(t)
3052 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true, Socket: noControlSocket(t)}
3053 cfgPath := filepath.Join(b.DataDir, rt.Host+".config.env")
3054 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte(tok), 0o600); err != nil {
3055 t.Fatal(err)
3056 }
3057 if err := os.WriteFile(cfgPath, []byte(InstanceKeyVar+"="+keyA+"\n"), 0o600); err != nil {
3058 t.Fatal(err)
3059 }
3060 b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true }
3061 b.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return "", nil } // the bucket: key removed
3062 syncs := 0
3063 replicaGateHook = func() {
3064 syncs++
3065 if err := os.WriteFile(cfgPath, []byte("OTHER=1\n"), 0o600); err != nil {
3066 t.Error(err)
3067 }
3068 }
3069 t.Cleanup(func() { replicaGateHook = nil })
3070 out := stageHelperApp(t, b, rt.Host, "v1")
3071 ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
3072 defer cancel()
3073 proc, _, err := b.Start(ctx, rt)
3074 if proc != nil {
3075 defer proc.Stop(context.Background())
3076 }
3077 if syncs != 1 {
3078 t.Fatalf("the sync ran %d times between the read and the gate, want once", syncs)
3079 }
3080 if got := outcomeOf(out); !errors.Is(err, ErrReplicaKey) || got != "" {
3081 t.Fatalf("Start = %v, app outcome %q; want ErrReplicaKey and no app", err, got)
3082 }
3083}
3084
3085// The gate reads no env file: given the env and layer the launch's read
3086// captured, it decides the same with the files gone.
3087func TestReplicaGateReadsNoEnvFile(t *testing.T) {
3088 keyA := strings.Repeat("A", 43) + "="
3089 b := &ExecBackend{DataDir: t.TempDir()}
3090 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Replica: true}
3091 if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".config.env"), []byte(InstanceKeyVar+"="+keyA+"\n"), 0o600); err != nil {
3092 t.Fatal(err)
3093 }
3094 env, layer, err := b.instanceEnvLayer(rt)
3095 if err != nil || layer != instancekey.LayerConfig || env[InstanceKeyVar] != keyA {
3096 t.Fatalf("instanceEnvLayer = %v, %v", layer, err)
3097 }
3098 if err := os.RemoveAll(b.DataDir); err != nil {
3099 t.Fatal(err)
3100 }
3101 b.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(keyA), true }
3102 for _, tc := range []struct {
3103 bucket string
3104 ok bool
3105 }{{keyA, true}, {"", false}} {
3106 b.ConfiguredKey = func(context.Context, registry.Route) (string, error) {
3107 if tc.bucket == "" {
3108 return "", nil
3109 }
3110 return instancekey.Digest(tc.bucket), nil
3111 }
3112 if err := b.replicaKeyGate(rt, env, layer, false); (err == nil) != tc.ok {
3113 t.Fatalf("bucket %q: gate = %v, want ok %v", tc.bucket, err, tc.ok)
3114 }
3115 }
3116 if err := b.replicaKeyGate(rt, env, instancekey.LayerNone, false); !errors.Is(err, ErrReplicaKey) {
3117 t.Fatalf("a key with no known layer = %v, want a refusal", err)
3118 }
3119}
3120```
3121
3122Create `internal/host/replicahomekeys_test.go`:
3123
3124```go
3125package host
3126
3127import (
3128 "bytes"
3129 "context"
3130 "errors"
3131 "log"
3132 "os"
3133 "path/filepath"
3134 "strings"
3135 "testing"
3136
3137 "github.com/carlosframework/platform/internal/appconfig"
3138 "github.com/carlosframework/platform/internal/instancekey"
3139 "github.com/carlosframework/platform/internal/registry"
3140 "github.com/carlosframework/platform/internal/release"
3141)
3142
3143// Ruling 30 (Astra round 6 finding 2): the list is read again at every
3144// call, so the procedure's invalidation and its new entries take effect at
3145// the next replica launch, with no restart. An absent list verifies
3146// nothing.
3147func TestReplicaHomeKeysReadsTheListAtEveryLaunch(t *testing.T) {
3148 a, b := instancekey.Digest("home-key-a"), instancekey.Digest("home-key-b")
3149 path := filepath.Join(t.TempDir(), "replica-home-keys")
3150 keys := replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(path) })
3151 if _, ok := keys("shop-au.example.com"); ok {
3152 t.Fatal("an absent list verified a key")
3153 }
3154 write := func(body string) {
3155 t.Helper()
3156 tmp := path + ".tmp"
3157 if err := os.WriteFile(tmp, []byte(body), 0o640); err != nil {
3158 t.Fatal(err)
3159 }
3160 if err := os.Rename(tmp, path); err != nil {
3161 t.Fatal(err)
3162 }
3163 }
3164 write("shop-au.example.com=" + a + "\nshop-uk.example.com=" + a + "\n")
3165 if got, ok := keys("shop-au.example.com"); !ok || got != a {
3166 t.Fatalf("listed replica = %q, %v", got, ok)
3167 }
3168 // The procedure's invalidation: the entry is gone at the next call.
3169 write("shop-uk.example.com=" + a + "\n")
3170 if _, ok := keys("shop-au.example.com"); ok {
3171 t.Fatal("a removed entry still verified: the list was cached")
3172 }
3173 // And its replacement, after the key change and the re-run check.
3174 write("shop-au.example.com=" + b + ",shop-uk.example.com=" + a)
3175 if got, ok := keys("shop-au.example.com"); !ok || got != b {
3176 t.Fatalf("the replaced entry = %q, %v; want the new digest", got, ok)
3177 }
3178 if _, ok := keys("shop-nz.example.com"); ok {
3179 t.Fatal("an unlisted replica verified")
3180 }
3181}
3182
3183// Astra round 5 finding 3: a malformed list is logged by entry position and
3184// reason, never by its text (a bare digest, with no host before it, would
3185// otherwise reach the log), and verifies nothing.
3186func TestReplicaHomeKeysLogsAMalformedListWithoutItsText(t *testing.T) {
3187 sha := instancekey.Digest("home-key")
3188 var buf bytes.Buffer
3189 log.SetOutput(&buf)
3190 defer log.SetOutput(os.Stderr)
3191 keys := replicaHomeKeys(func() ([]byte, error) { return []byte("shop-au.example.com=" + sha + "\n" + sha + "\n"), nil })
3192 if _, ok := keys("shop-au.example.com"); ok {
3193 t.Fatal("a malformed list verified a key")
3194 }
3195 line := buf.String()
3196 if !strings.Contains(line, "CARLOS_REPLICA_HOME_KEYS entry 2") || strings.Contains(line, sha[:16]) || strings.Contains(line, "shop-au") {
3197 t.Fatalf("log line %q: want the entry's position and reason, and none of its text", line)
3198 }
3199 buf.Reset()
3200 if _, ok := replicaHomeKeys(func() ([]byte, error) { return nil, errors.New("permission denied") })("shop-au.example.com"); ok {
3201 t.Fatal("an unreadable list verified a key")
3202 }
3203 if !strings.Contains(buf.String(), "could not be read") {
3204 t.Fatalf("log line %q", buf.String())
3205 }
3206}
3207
3208// Ruling 27's fresh read, as the agent wires it: the bucket's current
3209// configured key, through the deploy store and the box's secrets key. With
3210// no secrets key it is an error, so the replica is refused.
3211func TestReplicaConfigKeyReadsTheBucketNow(t *testing.T) {
3212 ctx := context.Background()
3213 s := release.NewDirStore(t.TempDir())
3214 priv, _, err := appconfig.GenerateKey()
3215 if err != nil {
3216 t.Fatal(err)
3217 }
3218 if err := appconfig.SetEnv(ctx, s, "acct", "shop", map[string]string{instancekey.Var: "key-A"}); err != nil {
3219 t.Fatal(err)
3220 }
3221 rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", App: "shop", Account: "acct", Replica: true}
3222 opened := 0
3223 open := func(context.Context) (release.Store, error) { opened++; return s, nil }
3224 read := configuredKey(func(k string) string {
3225 if k == "CARLOS_SECRETS_KEY" {
3226 return priv
3227 }
3228 return ""
3229 }, open)
3230 for i := 0; i < 2; i++ {
3231 if got, err := read(ctx, rt); err != nil || got != instancekey.Digest("key-A") {
3232 t.Fatalf("read %d = %q, %v", i, got, err)
3233 }
3234 }
3235 if opened != 1 {
3236 t.Fatalf("the store was opened %d times, want once", opened)
3237 }
3238 failing := configuredKey(func(string) string { return priv }, func(context.Context) (release.Store, error) { return nil, errors.New("bucket unreachable") })
3239 if _, err := failing(ctx, rt); err == nil {
3240 t.Fatal("an unreachable bucket was not an error")
3241 }
3242 if _, err := configuredKey(func(string) string { return "" }, open)(ctx, rt); err == nil {
3243 t.Fatal("no secrets key was not an error")
3244 }
3245}
3246```
3247
3248- [ ] **Step 2: Run them to verify they fail**
3249
3250Run: `nice -n 19 go test ./internal/activator/ -run 'TestReplicaLaunchWithoutAControlSocket|TestReplicaSeedIsGated|TestReplicaGate|TestKeylessReplica|TestNestedReplicaPrepareIsGated' -count=1 -v; nice -n 19 go test ./internal/host/ -run 'TestReplicaHomeKeys|TestReplicaConfigKey' -count=1; nice -n 19 go test ./internal/appconfig/ -run TestConfiguredKeyDigest -count=1`
3251Expected: build failures, `undefined: ErrReplicaKey` (and `ReplicaHomeKeys`, `ConfiguredKey`, `replicaHomeKeys`, `configuredKey`, `ConfiguredKeyDigest`). In the scratch build, with the gate present but disabled, the keyless and differing-key cases failed: the app reported "minted", or ran with the wrong key. That is Astra's round 4 failure. With only the fresh read disabled, "the home updated, this replica's configuration delayed" failed: the app ran with the old key. That is Astra's round 5 failure. With rev 5's second read of the files restored in the gate, `TestReplicaGateDecidesOnTheLaunchesOwnRead` failed: the app ran with A. That is Astra's round 6 failure.
3252
3253- [ ] **Step 3: Write the gate**
3254
3255Create `internal/activator/replicakey.go`:
3256
3257```go
3258package activator
3259
3260// A replica gets its home's key (instance-key plan rulings 26 to 28). A
3261// replica (#162) mirrors its home's data, sealed with the home's key, so
3262// the only key it may be handed is exactly the home's effective key. The
3263// platform never mints one on a replica row (ensureInstanceKeyFor), and
3264// every replica launch passes replicaKeyGate before anything runs:
3265//
3266// - a replica whose env files deliver a key starts only when that key's
3267// digest is its entry in the box's verified list (ReplicaHomeKeys, which
3268// only the pre-roll check writes, read again at every launch) and, when
3269// the key comes from the replica's configuration, the configuration in
3270// the bucket renders that same key now (ConfiguredKey). Where the key
3271// came from is decided by the same read that composed the env the child
3272// gets (instanceEnvLayer, ruling 29), never by reading the files again. A key that differs from either, or
3273// that cannot be checked, refuses the launch with a log line. The fresh
3274// read catches a configuration change that this box has not synced yet
3275// while the home already holds the new key (ruling 27);
3276// - a replica whose env files deliver no key starts only with a control
3277// socket. There an app on rastrillo's InstanceKey with no key of its
3278// own gets ErrNoInstanceKey and refuses (step 3), and an app that never
3279// asks for a key runs as before. Without a socket (a seed, a claim
3280// launch before O6, a control bind that failed) the helper would fall
3281// to step 4 and mint its own key over the home's data, so the agent
3282// refuses the launch itself rather than leave that to the helper.
3283//
3284// Delivering the home's key to a replica that has none is O8's, for a
3285// platform-minted home key only (ruling 28): until then, and always for a
3286// home whose key exists only in configuration, a keyless replica gets
3287// nothing and is gated as above.
3288
3289import (
3290 "context"
3291 "errors"
3292 "fmt"
3293 "log"
3294 "time"
3295
3296 "github.com/carlosframework/platform/internal/instancekey"
3297 "github.com/carlosframework/platform/internal/registry"
3298)
3299
3300// ErrReplicaKey is a replica launch refused by replicaKeyGate.
3301var ErrReplicaKey = errors.New("activator: replica launch refused: its instance key is not its home's")
3302
3303// configReadTimeout bounds replicaKeyGate's fresh read of the bucket.
3304const configReadTimeout = 10 * time.Second
3305
3306// instanceEnvLayer is instanceEnv returning, from the same read, the layer
3307// its CARLOS_INSTANCE_KEY came from (ruling 29). mergeEnv delivers exactly
3308// the key instancekey.Effective names, so env[Var] and the layer always
3309// describe the same bytes, however the files change afterwards.
3310func (b *ExecBackend) instanceEnvLayer(rt registry.Route) (map[string]string, instancekey.Layer, error) {
3311 cfgEnv, ownEnv, err := b.instanceEnvParts(rt)
3312 if err != nil {
3313 return nil, instancekey.LayerNone, err
3314 }
3315 _, layer := instancekey.Effective(cfgEnv, ownEnv)
3316 return mergeEnv(cfgEnv, ownEnv), layer, nil
3317}
3318
3319// replicaGateHook runs at the start of every replica gate, after the env
3320// was read. A test seam: it lets a test change the env files between the
3321// read and the gate, as a concurrent config sync can. Nil in production.
3322var replicaGateHook func()
3323
3324// replicaKeyGate is rulings 26, 27 and 29's check for a launch of rt with
3325// env (the composed env files), layer (where env's key came from, decided
3326// by the same read) and, when haveControl, a bound control socket. It reads
3327// no env file. It is nil for every row that is not a replica. Its errors and
3328// log lines name the host and the reason, never a key or a digest.
3329func (b *ExecBackend) replicaKeyGate(rt registry.Route, env map[string]string, layer instancekey.Layer, haveControl bool) error {
3330 if !rt.Replica {
3331 return nil
3332 }
3333 if replicaGateHook != nil {
3334 replicaGateHook()
3335 }
3336 refuse := func(why string) error {
3337 log.Printf("activator: replica %s not started: %s", rt.Host, why)
3338 return fmt.Errorf("%w: %s: %s", ErrReplicaKey, rt.Host, why)
3339 }
3340 key := env[instancekey.Var]
3341 if key == "" {
3342 if haveControl {
3343 return nil
3344 }
3345 return refuse("it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own")
3346 }
3347 var home string
3348 var ok bool
3349 if b.ReplicaHomeKeys != nil {
3350 home, ok = b.ReplicaHomeKeys(rt.Host)
3351 }
3352 switch {
3353 case !ok:
3354 return refuse(instancekey.Var + " cannot be checked against its home's key (no entry in CARLOS_REPLICA_HOME_KEYS; O8)")
3355 case instancekey.Digest(key) != home:
3356 return refuse(instancekey.Var + " is not its home's key")
3357 }
3358 switch layer {
3359 case instancekey.LayerOwn:
3360 return nil // the replica's own file: it changes only by hand
3361 case instancekey.LayerConfig:
3362 default:
3363 return refuse("where its " + instancekey.Var + " came from is unknown")
3364 }
3365 if b.ConfiguredKey == nil {
3366 return refuse("its configured " + instancekey.Var + " cannot be checked against the bucket's configuration")
3367 }
3368 ctx, cancel := context.WithTimeout(context.Background(), configReadTimeout)
3369 defer cancel()
3370 fresh, err := b.ConfiguredKey(ctx, rt)
3371 switch {
3372 case err != nil:
3373 return refuse("the bucket's configuration could not be read: " + err.Error())
3374 case fresh != instancekey.Digest(key):
3375 return refuse("its configured " + instancekey.Var + " is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)")
3376 }
3377 return nil
3378}
3379```
3380
3381In `internal/activator/backend_exec.go`, add the two fields to `ExecBackend`, after `Region`:
3382
3383```go
3384 // ReplicaHomeKeys answers, for a replica row's host, the SHA-256 of its
3385 // home's effective CARLOS_INSTANCE_KEY (instance-key plan rulings 26 and
3386 // 30). O8 will supply it from the home's published env item; until then
3387 // the agent reads the box's list (/etc/carlos/replica-home-keys), which
3388 // only the pre-roll check writes, again at every call. Nil, or false for
3389 // a host, means the home key cannot be verified, and a replica launch
3390 // that would be handed a key is refused (replicaKeyGate).
3391 ReplicaHomeKeys func(host string) (sha256 string, ok bool)
3392 // ConfiguredKey reads rt's configuration from the bucket now and
3393 // returns the SHA-256 of the CARLOS_INSTANCE_KEY it renders, "" for
3394 // none (appconfig.ConfiguredKeyDigest). replicaKeyGate refuses a
3395 // replica whose configured key is not that digest: a configuration
3396 // change is in flight (ruling 27). Nil refuses every configured key.
3397 ConfiguredKey func(ctx context.Context, rt registry.Route) (sha256 string, err error)
3398```
3399
3400In `seedChild`, read the env and its key's layer together (`instanceEnvLayer`, in `replicakey.go`), then gate:
3401
3402```go
3403 fileEnv, keyLayer, err := b.instanceEnvLayer(rt)
3404 if err != nil {
3405 return err
3406 }
3407 // A seed has no control socket, so a replica's seed is gated as a
3408 // launch without one (ruling 26), on the layer this same read found.
3409 if err := b.replicaKeyGate(rt, fileEnv, keyLayer, false); err != nil {
3410 return err
3411 }
3412 dbPath := paths.dbPath
3413```
3414
3415In `spawnChildFrom`, replace the `instanceEnv` read after the key mint with one read of the env and its key's layer:
3416
3417```go
3418 // One read gives the env and the layer its CARLOS_INSTANCE_KEY came
3419 // from, for the replica gate below (ruling 29).
3420 fileEnv, keyLayer, err := b.instanceEnvLayer(rt)
3421 if err != nil {
3422 return nil, "", err
3423 }
3424```
3425
3426In `spawnChildFrom`, once the control listener is bound or has failed, before the child's env is composed:
3427
3428```go
3429 // A replica is started only with its home's key, verified, or with
3430 // none and a control socket its helper can refuse on (ruling 26).
3431 if err := b.replicaKeyGate(rt, fileEnv, keyLayer, ctl != nil); err != nil {
3432 if ctl != nil {
3433 ctl.Close()
3434 }
3435 return nil, "", err
3436 }
3437 childVars := b.agentOwnedVars(rt, controlSocket, fileEnv, ctl != nil)
3438```
3439
3440In `internal/activator/prepare.go`, add the `instancekey` import; in `prepareAt`, declare the layer beside `env` and read them together (the lines between are unchanged):
3441
3442```go
3443 var env map[string]string
3444 var keyLayer instancekey.Layer // the layer env's key came from, read with it (ruling 29)
3445 if role == instancectl.RoleRun || role == instancectl.RoleSeed {
3446 // p.dataDir is the instance directory here, tenant-writable ground
3447 // (design §3.2), so the token file is minted and read through
3448 // instancefs rather than a plain path-based open.
3449 if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil {
3450 return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err)
3451 }
3452 if _, err := b.ensureInstanceKeyFor(rt, p); err != nil {
3453 return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err)
3454 }
3455 var err error
3456 env, keyLayer, err = b.instanceEnvLayer(rt)
3457```
3458
3459In `internal/activator/prepare.go`, `prepareAt`, before the log listener:
3460
3461```go
3462 // A replica's run and seed roles are gated like a flat launch: the
3463 // seed has no control socket, the run role has the one just bound or
3464 // none (ruling 26). The sidecar role reads no instance env.
3465 if role == instancectl.RoleRun || role == instancectl.RoleSeed {
3466 if err := b.replicaKeyGate(rt, env, keyLayer, ctl != nil); err != nil {
3467 if ctl != nil {
3468 ctl.Close()
3469 }
3470 return p, nil, nil, nil, err
3471 }
3472 }
3473 logs, err := listenLogs(b.logSocketFor(rt.Host, role, gen), b.logEmitter(rt))
3474```
3475
3476Create `internal/appconfig/instancekey.go`, the fresh read:
3477
3478```go
3479package appconfig
3480
3481import (
3482 "context"
3483 "crypto/ecdh"
3484
3485 "github.com/carlosframework/platform/internal/instancekey"
3486 "github.com/carlosframework/platform/internal/registry"
3487 "github.com/carlosframework/platform/internal/release"
3488)
3489
3490// ConfiguredKeyDigest is the SHA-256 of the CARLOS_INSTANCE_KEY rt's
3491// configuration in the bucket renders now, "" when it renders none: the
3492// same merge and renderer SyncExec writes <host>.config.env with
3493// (LoadMergedHostVersions, RenderEnv), read fresh rather than from the
3494// box's last sync. A replica launch whose configured key is not this
3495// digest is refused (instance-key plan ruling 27): a configuration change
3496// is in flight, and the home may already hold the new key. The value
3497// never leaves this function.
3498func ConfiguredKeyDigest(ctx context.Context, s release.Store, priv *ecdh.PrivateKey, rt registry.Route) (string, error) {
3499 m, _, err := LoadMergedHostVersions(ctx, s, release.RouteAccount(rt), rt.App, rt.Environment, rt.Host)
3500 if err != nil || m == nil {
3501 return "", err
3502 }
3503 rendered, err := RenderEnv(m, priv, routeBinding(rt))
3504 if err != nil {
3505 return "", err
3506 }
3507 key, _ := instancekey.Effective(instancekey.ParseEnv(rendered), nil)
3508 if key == "" {
3509 return "", nil
3510 }
3511 return instancekey.Digest(key), nil
3512}
3513```
3514
3515Create `internal/host/replicahomekeys.go`:
3516
3517```go
3518package host
3519
3520import (
3521 "context"
3522 "crypto/ecdh"
3523 "errors"
3524 "log"
3525 "os"
3526 "strings"
3527 "sync"
3528
3529 "github.com/carlosframework/platform/internal/appconfig"
3530 "github.com/carlosframework/platform/internal/instancekey"
3531 "github.com/carlosframework/platform/internal/registry"
3532 "github.com/carlosframework/platform/internal/release"
3533)
3534
3535// replicaHomeKeysPath is the box's list of verified home keys for its
3536// replica rows (the list the plan calls CARLOS_REPLICA_HOME_KEYS):
3537// "replica-host=sha256" entries, separated by commas or newlines. Only the
3538// pre-roll check writes it, as root, replacing it whole (write a temporary
3539// file, then rename it into place); the agent only reads it.
3540const replicaHomeKeysPath = "/etc/carlos/replica-home-keys"
3541
3542// replicaHomeKeys is ExecBackend.ReplicaHomeKeys (instance-key plan rulings
3543// 26, 27 and 30): it reads the list again at every call, so an entry the
3544// procedure removes or replaces takes effect at the next replica launch,
3545// with no agent restart and no cached copy to go stale. Nothing refreshes
3546// the list from a replica's own configuration. An absent list verifies
3547// nothing. A list that cannot be read or parsed is logged by the error's
3548// position and reason, never its text, and verifies nothing, so every
3549// replica launch that would be handed a key is refused until it is fixed.
3550func replicaHomeKeys(read func() ([]byte, error)) func(host string) (string, bool) {
3551 return func(host string) (string, bool) {
3552 raw, err := read()
3553 if errors.Is(err, os.ErrNotExist) {
3554 return "", false
3555 }
3556 if err != nil {
3557 log.Printf("agent: replica home keys: the list could not be read (no replica home key verifies): %v", err)
3558 return "", false
3559 }
3560 keys, err := instancekey.ParseHomeKeys(strings.ReplaceAll(string(raw), "\n", ","))
3561 if err != nil {
3562 log.Printf("agent: replica home keys: %v (no replica home key verifies until it is fixed)", err)
3563 return "", false
3564 }
3565 sha, ok := keys[host]
3566 return sha, ok
3567 }
3568}
3569
3570// configuredKey is ExecBackend.ConfiguredKey (ruling 27): the digest
3571// of the key rt's configuration in the bucket renders now, read with the
3572// box's secrets key through the deploy store. The store and key are opened
3573// on first use and kept once both open; a failure is returned, so the
3574// replica is refused, and is tried again at its next launch.
3575func configuredKey(getenv func(string) string, open func(context.Context) (release.Store, error)) func(context.Context, registry.Route) (string, error) {
3576 var (
3577 mu sync.Mutex
3578 store release.Store
3579 priv *ecdh.PrivateKey
3580 )
3581 return func(ctx context.Context, rt registry.Route) (string, error) {
3582 mu.Lock()
3583 if store == nil {
3584 raw := getenv("CARLOS_SECRETS_KEY")
3585 if raw == "" {
3586 mu.Unlock()
3587 return "", errors.New("CARLOS_SECRETS_KEY is not set")
3588 }
3589 k, err := appconfig.ParsePrivateKey(raw, "CARLOS_SECRETS_KEY")
3590 if err != nil {
3591 mu.Unlock()
3592 return "", err
3593 }
3594 st, err := open(ctx)
3595 if err != nil {
3596 mu.Unlock()
3597 return "", err
3598 }
3599 store, priv = st, k
3600 }
3601 st, k := store, priv
3602 mu.Unlock()
3603 return appconfig.ConfiguredKeyDigest(ctx, st, k, rt)
3604 }
3605}
3606```
3607
3608In `internal/host/agent.go`, after the `ExecBackend` literal (beside `eb.Logs`):
3609
3610```go
3611 // The verified home keys of this box's replica rows (L21 rulings 26
3612 // and 30), read again at every replica launch, until O8 reads them
3613 // from each home's published env item.
3614 eb.ReplicaHomeKeys = replicaHomeKeys(func() ([]byte, error) { return os.ReadFile(replicaHomeKeysPath) })
3615 // The bucket's configuration, read fresh at each replica launch
3616 // whose key is configured (ruling 27).
3617 eb.ConfiguredKey = configuredKey(os.Getenv, openDeployStore)
3618```
3619
3620- [ ] **Step 4: Run the tests, then the launch tests around them**
3621
3622Run (sandbox off: the control-socket case and the nested prepare bind sockets): the Step 2 commands, then `nice -n 19 go test ./internal/activator/ -run 'Replica|InstanceKey|Seed|Start|Prepare|LaunchEnvironment|Control' -count=1`
3623Expected: PASS. (With the sandbox on, the control-socket case skips and the nested run-role case cannot bind its log socket.)
3624
3625- [ ] **Step 5: Commit**
3626
3627```bash
3628gofmt -l internal/activator internal/host internal/appconfig
3629git add internal/activator/replicakey.go internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/replica_instancekey_test.go internal/activator/replica_instancekey_linux_test.go internal/host/replicahomekeys.go internal/host/replicahomekeys_test.go internal/host/agent.go internal/appconfig/instancekey.go internal/appconfig/instancekey_test.go internal/activator/replicakey_race_test.go
3630git commit -m "activator: a replica launches only with its home's key, verified, or none and a socket its helper refuses on (L21)
3631
3632Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
3633```
3634
3635---
3636
3637### Task 4: The claim's launch source mints under the guard, syncs before it answers, and names the key it delivers
3638
3639**Files:**
3640- Modify: `internal/activator/claim_snapshot.go:17-25` (`ClaimLaunchSnapshot`), `:40-67` (`EnsureClaimAdminToken`), `:69-85` (`SnapshotClaimLaunch`'s env read)
3641- Modify: `internal/host/claimowners.go:125-138` (doc on `ClaimFilesManifest`, O1')
3642- Test: `internal/activator/claim_instancekey_test.go`, `internal/activator/claimretry_instancekey_linux_test.go`, `internal/host/claim_instancekey_test.go`, `internal/host/claim_replica_instancekey_test.go` (create all four)
3643
3644**Interfaces:**
3645- Consumes: Task 1's `instancekey.Effective`, `Digest`, `SourceOf`; Task 2's `ensureInstanceKeyFor`, `settleClaimInstanceEnv`, `errInstanceEnvNeedsWrite`, `probeInstanceKeySync`; `instanceEnvParts` (backend_exec.go:1836); `claimServingLaunchSource` (claimproduction.go:421), `claimServingOwners` (claimowners.go:293) with `Select` (:375) and `Materialize` (:425); test fixture `newAdminTokenFixture`, `adminTokenHost`, `f.envPath()`, `f.token()` (claimproduction_admintoken_test.go:20-118); `leasetest.NewVersionedBucket`; `Service.startAdmitted` (claimwake.go:330) as `claimloop_linux_test.go` drives it.
3646- Produces: `EnsureClaimAdminToken` initializes the token and the key inside the one authorized write, and syncs a present key before it answers. `ClaimLaunchSnapshot.InstanceKeySHA256` and `InstanceKeySource` name the key the snapshot's env delivers, for O6 (ruling 21).
3647
3648**Built:** the guarded write mints the admin token only when the effective env has none (rev 5.1 called `EnsureAdminToken` unconditionally, which would mint a second token into the own file over a configured one and change a working launch); the key is ensured after it either way. The early return has a third arm: no own env file with the token and the key both from the rendered configuration proceeds as it did before the key existed (pre-flight ruling A; a configured key is not the platform's to make durable, ruling 5, and an absent file is not created, ruling 7). The claim launch source is the one launch path that returns the key's refusal without a wrapper of its own, so it wraps it with the host, once (Task 6's ruling). `claimServingLaunchSource` does not yet carry `InstanceKeySHA256` and `InstanceKeySource` into `ClaimServingLaunch`: that is O6's.
3649
3650- [ ] **Step 1: Write the failing tests**
3651
3652Create `internal/activator/claim_instancekey_test.go`:
3653
3654```go
3655package activator
3656
3657import (
3658 "context"
3659 "errors"
3660 "os"
3661 "path/filepath"
3662 "strings"
3663 "testing"
3664
3665 "github.com/carlosframework/platform/internal/registry"
3666)
3667
3668// Finding 2 on the claim path: when the token and the key are already
3669// present, EnsureClaimAdminToken writes nothing and calls no authorizer,
3670// but it does not answer until the env file and its directory are synced.
3671// A file that is not 0600 goes through the authorized write, which sets it.
3672func TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers(t *testing.T) {
3673 probe := probeInstanceKeySync(t)
3674 const host = "claim-key.example"
3675 b := &ExecBackend{DataDir: t.TempDir()}
3676 rt := registry.Route{Host: host, Socket: "/run/carlos/claim-key.sock", DBPath: "/data/claim-key.db"}
3677 envPath := filepath.Join(b.DataDir, host+".env")
3678 body := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("d", 64) + "\n" + InstanceKeyVar + "=" + strings.Repeat("A", 43) + "=\n"
3679 if err := os.WriteFile(envPath, []byte(body), 0o600); err != nil {
3680 t.Fatal(err)
3681 }
3682 authorized := 0
3683 authorize := func(_ context.Context, write func() error) error { authorized++; return write() }
3684 for _, fail := range []func(error){func(e error) { probe.fileErr = e }, func(e error) { probe.dirErr = e }} {
3685 fail(errors.New("injected: EIO"))
3686 if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err == nil {
3687 t.Fatal("the launch source answered while the env file's sync fails")
3688 }
3689 fail(nil)
3690 }
3691 before := *probe
3692 if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err != nil {
3693 t.Fatal(err)
3694 }
3695 if authorized != 0 || probe.file != before.file+1 || probe.dir != before.dir+1 {
3696 t.Fatalf("authorized %d, synced file %d dir %d; want 0, 1, 1", authorized, probe.file-before.file, probe.dir-before.dir)
3697 }
3698 if err := os.Chmod(envPath, 0o644); err != nil {
3699 t.Fatal(err)
3700 }
3701 if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err != nil {
3702 t.Fatal(err)
3703 }
3704 if st, _ := os.Stat(envPath); authorized != 1 || st.Mode().Perm() != 0o600 {
3705 t.Fatalf("a 0644 file: authorized %d, mode %o; want the guarded path to set 600", authorized, st.Mode().Perm())
3706 }
3707 if raw, _ := os.ReadFile(envPath); string(raw) != body {
3708 t.Fatal("the claim path rewrote the env file")
3709 }
3710}
3711```
3712
3713Create `internal/activator/claimretry_instancekey_linux_test.go`:
3714
3715```go
3716//go:build linux
3717
3718package activator
3719
3720import (
3721 "context"
3722 "errors"
3723 "os/exec"
3724 "slices"
3725 "sync"
3726 "testing"
3727 "time"
3728
3729 "github.com/carlosframework/platform/internal/hostclaim"
3730 "github.com/carlosframework/platform/internal/hostwrite"
3731 "github.com/carlosframework/platform/internal/registry"
3732)
3733
3734// Open question 5's premise, corrected (instance-key plan rev 2): on a
3735// claim box a session's process retries do not go back to the launch
3736// source, which is where the key is minted. Prepare runs once per
3737// admitted session and captures the launch inputs, and every attempt
3738// calls Materialize (the C8 provider's Establish) before it reserves a
3739// start. So no attempt runs with a key minted after the files manifest
3740// was established. This pins that shape of startAdmitted: one Prepare,
3741// and Materialize before each attempt's start.
3742func TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst(t *testing.T) {
3743 var (
3744 mu sync.Mutex
3745 events []string
3746 prepares int
3747 )
3748 note := func(e string) { mu.Lock(); events = append(events, e); mu.Unlock() }
3749 obs := hostclaim.Observed{Host: "x.example", ETag: "one", Claim: &hostclaim.Claim{Holder: &hostclaim.Holder{Location: "instances/x.example/inc", Session: 1, Generation: "one"}, Op: hostclaim.Op{Kind: "landing", ID: "op", Phase: "probing", Exec: "box"}}}
3750 work, _ := (hostwrite.Config{}).Select(t.Context(), obs.Host, "spawn")
3751 a := &Service{cfg: Config{HostWrites: hostwrite.Config{Enabled: true}, Claims: &ClaimRuntime{Box: "box", Read: func(context.Context, string) (hostclaim.Observed, error) { return obs, nil }, Prepare: func(context.Context, registry.Route, hostclaim.Observed, startPolicy) (ClaimStart, error) {
3752 mu.Lock()
3753 prepares++
3754 mu.Unlock()
3755 return ClaimStart{RoleLauncher: &RoleLaunch{}, CurrentAttempt: func() hostclaim.WakeRetryAttempt { return hostclaim.WakeRetryAttempt{} }, Host: obs.Host, Work: work, Identity: hostwrite.Identity{Config: "c", Release: "r"},
3756 OpenProbe: func(context.Context, time.Time) (time.Time, error) {
3757 return time.Time{}, errors.New("unreachable while Start refuses")
3758 },
3759 Restore: func(context.Context) error { note("restore"); return nil },
3760 Materialize: func(context.Context) error {
3761 note("materialize")
3762 return nil
3763 },
3764 Build: func() *exec.Cmd { return nil },
3765 Ready: func(context.Context, *exec.Cmd) error { return nil },
3766 Reserve: func(context.Context, int) error { note("reserve"); return nil },
3767 Failed: func(context.Context, int) error { note("failed"); return nil },
3768 Healthy: func(context.Context, int, *exec.Cmd) error { return nil },
3769 Replicate: func(context.Context, string) (Rep, error) { return nil, nil },
3770 Publish: func(context.Context, hostclaim.Observed) (hostclaim.Observed, error) { return obs, nil },
3771 Retain: func(Proc, Rep, error) {},
3772 }, nil
3773 }}}}
3774 // Two starts: the first refused (RoleLaunch.Start refuses until claim
3775 // Plan 9 Group D), a 1 s backoff, then the second.
3776 err := a.startAdmitted(t.Context(), &actEntry{host: obs.Host}, obs, startPolicy{MaxStarts: 2, Window: 30 * time.Second, Deadline: time.Now().Add(time.Minute), Probe: true})
3777 if !errors.Is(err, ErrClaimIsolationUnavailable) {
3778 t.Fatalf("the loop should end on the last start's refusal: %v", err)
3779 }
3780 want := []string{"restore", "reserve", "materialize", "failed", "reserve", "materialize", "failed"}
3781 mu.Lock()
3782 defer mu.Unlock()
3783 if prepares != 1 || !slices.Equal(events, want) {
3784 t.Fatalf("prepares %d, events %v; want 1 and %v", prepares, events, want)
3785 }
3786}
3787```
3788
3789Create `internal/host/claim_instancekey_test.go`:
3790
3791```go
3792package host
3793
3794import (
3795 "context"
3796 "os"
3797 "path/filepath"
3798 "strings"
3799 "testing"
3800 "time"
3801
3802 "github.com/carlosframework/platform/internal/activator"
3803 "github.com/carlosframework/platform/internal/hostclaim"
3804 "github.com/carlosframework/platform/internal/hostwrite"
3805 "github.com/carlosframework/platform/internal/instancekey"
3806 "github.com/carlosframework/platform/internal/lease/leasetest"
3807)
3808
3809// keyCapturingFiles is a C8 provider that records the instance's own env
3810// file as it stands when Establish runs: the moment the provider must
3811// publish it (ruling 6).
3812type keyCapturingFiles struct {
3813 path string
3814 seen []byte
3815}
3816
3817func (k *keyCapturingFiles) Establish(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, work hostwrite.Work) error {
3818 raw, err := os.ReadFile(k.path)
3819 k.seen = raw
3820 return err
3821}
3822
3823type keyTestMarkers struct{}
3824
3825func (keyTestMarkers) PutMarker(context.Context, string, string, []byte, string, func(context.Context) error) (string, error) {
3826 return "v1", nil
3827}
3828
3829type passWork struct{}
3830
3831func (passWork) With(_ context.Context, _ hostwrite.Identity, fn func() error) error { return fn() }
3832
3833// The claim's first wake mints the key in Select (the launch source), so
3834// the env file the C8 provider publishes in Materialize already carries it,
3835// and the launch env the app will run with carries the same value.
3836func TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest(t *testing.T) {
3837 f := newAdminTokenFixture(t, "box-1")
3838 files := &keyCapturingFiles{path: f.envPath()}
3839 owners := &claimServingOwners{Data: leasetest.NewVersionedBucket(), Bucket: "data", Region: "ie", Box: "box-1",
3840 NewID: func() string { return "g1" }, Now: func() time.Time { return time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) },
3841 Sources: ClaimServingSources{Launch: claimServingLaunchSource(f.backend, f.tokens), Markers: keyTestMarkers{}, Files: files}}
3842 o, err := hostclaim.Read(t.Context(), f.claims, adminTokenHost)
3843 if err != nil {
3844 t.Fatal(err)
3845 }
3846 in, err := owners.Select(t.Context(), f.rt, o, activator.StartPolicy{})
3847 if err != nil {
3848 t.Fatalf("select: %v", err)
3849 }
3850 if err := owners.Materialize(t.Context(), o, in, passWork{}); err != nil {
3851 t.Fatalf("materialize: %v", err)
3852 }
3853 var key string
3854 for _, line := range strings.Split(string(files.seen), "\n") {
3855 if v, ok := strings.CutPrefix(line, activator.InstanceKeyVar+"="); ok {
3856 key = v
3857 }
3858 }
3859 if len(key) != 44 {
3860 t.Fatalf("the env file at Establish carries no minted key: %d bytes", len(files.seen))
3861 }
3862 found := false
3863 for _, kv := range in.Env {
3864 found = found || kv == activator.InstanceKeyVar+"="+key
3865 }
3866 if !found {
3867 t.Fatal("the launch env does not carry the key the provider published")
3868 }
3869}
3870
3871// An instance from before this build has a token and no key. Its next
3872// claim launch mints the key under the same guarded write, leaves the
3873// token alone, and a retry keeps both.
3874func TestClaimLaunchMintsAKeyBesideAnExistingToken(t *testing.T) {
3875 f := newAdminTokenFixture(t, "box-1")
3876 tok := strings.Repeat("c", 64)
3877 if err := os.WriteFile(f.envPath(), []byte("CARLOS_ADMIN_TOKEN="+tok+"\n"), 0o600); err != nil {
3878 t.Fatal(err)
3879 }
3880 launch := claimServingLaunchSource(f.backend, f.tokens)
3881 first, err := launch(t.Context(), f.rt)
3882 if err != nil {
3883 t.Fatal(err)
3884 }
3885 raw, _ := os.ReadFile(f.envPath())
3886 if f.token() != tok || !strings.HasPrefix(string(raw), "CARLOS_ADMIN_TOKEN="+tok+"\n"+activator.InstanceKeyVar+"=") {
3887 t.Fatalf("env file after the launch: %d bytes, token kept %v", len(raw), f.token() == tok)
3888 }
3889 again, err := launch(t.Context(), f.rt)
3890 if err != nil {
3891 t.Fatal(err)
3892 }
3893 raw2, _ := os.ReadFile(f.envPath())
3894 if string(raw2) != string(raw) || again.Identity != first.Identity {
3895 t.Fatal("a retried launch changed the env file or the launch identity")
3896 }
3897}
3898
3899// Ruling 21: the claim snapshot names the key its env delivers, by digest,
3900// and the file that supplied it, for O6 to check a process's report
3901// against. The identity is the env's alone, so it does not change.
3902func TestClaimSnapshotNamesTheDeliveredKeyAndItsSource(t *testing.T) {
3903 f := newAdminTokenFixture(t, "box-1")
3904 first, err := claimServingLaunchSource(f.backend, f.tokens)(t.Context(), f.rt)
3905 if err != nil {
3906 t.Fatal(err)
3907 }
3908 snap, err := f.backend.SnapshotClaimLaunch(t.Context(), f.rt)
3909 if err != nil {
3910 t.Fatal(err)
3911 }
3912 own, _ := os.ReadFile(f.envPath())
3913 var key string
3914 for _, line := range strings.Split(string(own), "\n") {
3915 if v, ok := strings.CutPrefix(line, activator.InstanceKeyVar+"="); ok {
3916 key = v
3917 }
3918 }
3919 if snap.InstanceKeySource != instancekey.SourcePlatform || snap.InstanceKeySHA256 != instancekey.Digest(key) || snap.Identity != first.Identity {
3920 t.Fatalf("platform key: source %q, digest match %v, identity kept %v", snap.InstanceKeySource, snap.InstanceKeySHA256 == instancekey.Digest(key), snap.Identity == first.Identity)
3921 }
3922 tok := "CARLOS_ADMIN_TOKEN=" + f.token() + "\n"
3923 if err := os.WriteFile(f.envPath(), []byte(tok+activator.InstanceKeyVar+"=\n"), 0o600); err != nil {
3924 t.Fatal(err)
3925 }
3926 if err := os.WriteFile(filepath.Join(f.data, adminTokenHost+".config.env"), []byte(activator.InstanceKeyVar+"=from-config\n"), 0o600); err != nil {
3927 t.Fatal(err)
3928 }
3929 snap, err = f.backend.SnapshotClaimLaunch(t.Context(), f.rt)
3930 if err != nil || snap.InstanceKeySource != instancekey.SourceConfig || snap.InstanceKeySHA256 != instancekey.Digest("from-config") {
3931 t.Fatalf("configured key: source %q, %v", snap.InstanceKeySource, err)
3932 }
3933}
3934```
3935
3936Create `internal/host/claim_replica_instancekey_test.go`, ruling 26 on the claim path:
3937
3938```go
3939package host
3940
3941import (
3942 "context"
3943 "errors"
3944 "os"
3945 "path/filepath"
3946 "strings"
3947 "testing"
3948
3949 "github.com/carlosframework/platform/internal/activator"
3950 "github.com/carlosframework/platform/internal/instancekey"
3951 "github.com/carlosframework/platform/internal/registry"
3952)
3953
3954// Ruling 26 on the claim path, which has no control socket before O6: a
3955// keyless replica is never selected (its helper would mint), a key that is
3956// not the home's is refused, and the home's key is selected and named.
3957func TestClaimReplicaLaunchIsGated(t *testing.T) {
3958 homeKey := strings.Repeat("H", 43) + "="
3959 for _, tc := range []struct {
3960 name, config string
3961 refused bool
3962 }{
3963 {"keyless", "", true},
3964 {"another key", activator.InstanceKeyVar + "=configured-B\n", true},
3965 {"the home's key", activator.InstanceKeyVar + "=" + homeKey + "\n", false},
3966 } {
3967 t.Run(tc.name, func(t *testing.T) {
3968 f := newAdminTokenFixture(t, "box-1")
3969 f.rt.Replica = true
3970 f.backend.ReplicaHomeKeys = func(string) (string, bool) { return instancekey.Digest(homeKey), true }
3971 f.backend.ConfiguredKey = func(context.Context, registry.Route) (string, error) { return instancekey.Digest(homeKey), nil }
3972 own := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
3973 if err := os.WriteFile(f.envPath(), []byte(own), 0o600); err != nil {
3974 t.Fatal(err)
3975 }
3976 if tc.config != "" {
3977 if err := os.WriteFile(filepath.Join(f.data, adminTokenHost+".config.env"), []byte(tc.config), 0o600); err != nil {
3978 t.Fatal(err)
3979 }
3980 }
3981 launch, err := claimServingLaunchSource(f.backend, f.tokens)(t.Context(), f.rt)
3982 if tc.refused {
3983 if !errors.Is(err, activator.ErrReplicaKey) {
3984 t.Fatalf("launch source = %v, want ErrReplicaKey", err)
3985 }
3986 } else {
3987 found := false
3988 for _, kv := range launch.Env {
3989 found = found || kv == activator.InstanceKeyVar+"="+homeKey
3990 }
3991 if err != nil || !found {
3992 t.Fatalf("launch source = %v, key delivered %v", err, found)
3993 }
3994 }
3995 if raw, _ := os.ReadFile(f.envPath()); string(raw) != own {
3996 t.Fatal("a replica's claim launch changed its own env file: a replica never mints")
3997 }
3998 })
3999 }
4000}
4001```
4002
4003- [ ] **Step 2: Run them to verify they fail**
4004
4005Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureClaimAdminTokenSyncs|TestStartAdmittedRetriesReuse' -count=1 -v && nice -n 19 go test ./internal/host/ -run 'TestClaimFirstWakeKey|TestClaimLaunchMintsAKey|TestClaimSnapshotNames|TestClaimReplicaLaunch' -count=1 -v`
4006Expected, as checked in the scratch build:
4007- `TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers` fails: the early return answers without a sync.
4008- `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` passes: it pins existing behaviour (ruling 16).
4009- The host package fails to build: `snap.InstanceKeySHA256 undefined`.
4010
4011- [ ] **Step 3: Mint, sync, and name the key in the launch source**
4012
4013In `internal/activator/claim_snapshot.go`, add `"errors"` and `"github.com/carlosframework/platform/internal/instancekey"` to the imports, and add the two fields to `ClaimLaunchSnapshot`:
4014
4015```go
4016 Roles []string
4017 Limit cgroup.Limit
4018 // InstanceKeySHA256 and InstanceKeySource name the CARLOS_INSTANCE_KEY
4019 // Env delivers (its digest) and the file that supplied it
4020 // (instancekey.SourcePlatform or SourceConfig); both empty when Env
4021 // delivers none. O6 hands them to the process's adoption verb.
4022 InstanceKeySHA256, InstanceKeySource string
4023}
4024```
4025
4026Replace the body of `EnsureClaimAdminToken` from the early return to its end:
4027
4028```go
4029 // The instance key is initialized the same way and in the same
4030 // authorized write (L21). vars is the effective env, so a key the
4031 // rendered configuration supplies counts as present, and a replica
4032 // gets none (ruling 26). When both are present nothing is written and
4033 // authorize is not called, but the file is still made durable before
4034 // the launch source answers: a key whose mint failed its sync is never
4035 // launched with until a sync succeeds (settleClaimInstanceEnv, ruling
4036 // 18). A file that is not 0600 takes the guarded path. The key's
4037 // ensure sets the mode when the own file holds the key; when the key
4038 // is the configuration's, or the row is a replica, the ensure writes
4039 // nothing, the mode stays as it is, and the next launch of that row
4040 // asks the guard again. The token's mode has never been the claim
4041 // path's to tighten.
4042 if vars["CARLOS_ADMIN_TOKEN"] != "" && (rt.Replica || vars[InstanceKeyVar] != "") {
4043 err := b.settleClaimInstanceEnv(rt)
4044 switch {
4045 case err == nil:
4046 return nil
4047 case errors.Is(err, os.ErrNotExist):
4048 // No own env file: the token and the key both come from the
4049 // rendered configuration. A configured key is the app's and
4050 // is not the platform's to make durable (ruling 5), and an
4051 // absent file is not created here (ruling 7), so the launch
4052 // proceeds as it did before the key existed.
4053 return nil
4054 case !errors.Is(err, errInstanceEnvNeedsWrite):
4055 return err
4056 }
4057 }
4058 if authorize == nil {
4059 return hostwrite.ErrUnavailable
4060 }
4061 paths := b.pathsFor(rt)
4062 return authorize(ctx, func() error {
4063 // Only a missing token is minted: one the rendered configuration
4064 // supplies is the one the launch delivers, and a second one in the
4065 // own file would shadow it.
4066 if vars["CARLOS_ADMIN_TOKEN"] == "" {
4067 if paths.layout == LayoutNested {
4068 if err := EnsureAdminTokenBeneath(paths.dataDir, rt.Host+".env"); err != nil {
4069 return err
4070 }
4071 } else if err := EnsureAdminToken(paths.envFile); err != nil {
4072 return err
4073 }
4074 }
4075 // The one place the claim launch source names the host around the
4076 // key's error: the refusal itself does not, and every other launch
4077 // path wraps it the same way.
4078 if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil {
4079 return fmt.Errorf("instance key for %s: %w", rt.Host, err)
4080 }
4081 return nil
4082 })
4083}
4084```
4085
4086(`"fmt"` and `"os"` join the imports.)
4087
4088In `SnapshotClaimLaunch`, read the two layers, keep `instanceEnv`'s merge, pass a replica through Task 3b's gate with the layer from this same read (ruling 29; a claim launch has no control socket before O6), and record the delivered key. The identity is computed from `Env` as before, so it does not change:
4089
4090```go
4091 cfgEnv, ownEnv, err := b.instanceEnvParts(rt)
4092 if err != nil {
4093 return out, err
4094 }
4095 vars := mergeEnv(cfgEnv, ownEnv) // instanceEnv, with its two layers kept
4096 if vars["CARLOS_ADMIN_TOKEN"] == "" {
4097 return out, hostwrite.ErrUnavailable
4098 }
4099 // The key this launch delivers and the file that supplied it, from
4100 // the same read as vars: what the replica gate decides on (ruling 29)
4101 // and what the process's adoption report is checked against (ruling
4102 // 21, O6). Not in the identity: the env already is.
4103 key, layer := instancekey.Effective(cfgEnv, ownEnv)
4104 // A claim launch has no control socket until O6, so a replica is
4105 // selected only with its home's key, verified (ruling 26).
4106 if err := b.replicaKeyGate(rt, vars, layer, false); err != nil {
4107 return out, err
4108 }
4109 if key != "" {
4110 out.InstanceKeySHA256, out.InstanceKeySource = instancekey.Digest(key), instancekey.SourceOf(layer)
4111 }
4112```
4113
4114In `internal/host/claimowners.go`, extend the `ClaimFilesManifest` doc comment (O1'):
4115
4116```go
4117// bootstrap; only an empty bootstrap (in.Empty, after EmptyAuthority) starts
4118// from no manifest.
4119//
4120// The instance key (L21, plan 2026-10-10-instance-key-provision.md
4121// obligation O1'): the launch source has already minted
4122// CARLOS_INSTANCE_KEY into the host's own <host>.env, durably, when
4123// Establish runs, and the app executes only after it returns. So the
4124// manifest Establish leaves must carry the env file (region HA §5.6's
4125// always-continuous platform item) with exactly the bytes on disk when it
4126// runs, staged and committed before it returns. A new admission starts
4127// with no adoption record: the manifest carries no instancekey.AdoptionItem
4128// (StagePlatformDelete), and the local <host>.instance-key-adoption.json is
4129// removed under work. An error, a bucket that cannot be reached included,
4130// fails the wake and the app does not start. Establish never writes the env
4131// file on disk.
4132type ClaimFilesManifest interface {
4133```
4134
4135- [ ] **Step 4: Run the tests, then the claim tests around them**
4136
4137Run: `nice -n 19 go test ./internal/host/ -run 'InstanceKey|ClaimSnapshot|AdminToken|Rev2Shape|SnapshotClaimLaunch' -count=1`
4138Expected: PASS, including `TestClaimProductionAdminTokenRefusesWithoutAuthority`: without authority no `<host>.env` is written, so no key either.
4139
4140Run: `nice -n 19 go test ./internal/activator/ -run 'Claim|StartAdmitted' -count=1`
4141Expected: PASS, including `TestEnsureClaimAdminTokenInspectsSidecarBeforeAuthorize`.
4142
4143- [ ] **Step 5: Commit**
4144
4145```bash
4146gofmt -l internal/activator internal/host
4147git add internal/activator/claim_snapshot.go internal/activator/claim_instancekey_test.go internal/activator/claimretry_instancekey_linux_test.go internal/host/claimowners.go internal/host/claim_instancekey_test.go internal/host/claim_replica_instancekey_test.go
4148git commit -m "claim: the launch source mints the instance key under the guard, syncs it before Establish, and names it (L21)
4149
4150Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
4151```
4152
4153---
4154
4155### Task 5: The adoption verb and its publication
4156
4157The verb is built and tested here and mounted nowhere on main (ruling 21). O6 mounts it on claim launches, after O7 (ruling 24).
4158
4159**Files:**
4160- Create: `internal/hostfiles/relay.go`, `internal/activator/keyadoption.go`
4161- Test: `internal/hostfiles/relay_test.go`, `internal/activator/keyadoption_test.go`, `internal/activator/keyadoption_linux_test.go`, `internal/host/instancekey_adoption_test.go`
4162
4163**Interfaces:**
4164- Consumes: Task 1's `instancekey` (`Binding`, `AdoptionRecord`, `ParseAdoption`, `ValidDigest`, the sources, `AdoptionSuffix`, `AdoptionItem`); Task 2's `syncDir`; `adminTokenMu`; `instancefs.WriteAtomic`, `instancefs.ReadBeneathLimit`, `safefile.WriteFile` (which fsyncs the file and its directory); `hostfiles.Open`, `Run`, `Options`, `ReadPlatform`, `ManifestKey`, `(*Publisher).StagePlatform`.
4165- Produces: `activator.AdoptionLaunch`, `(*ExecBackend).MountInstanceKeyAdoption`; `hostfiles.PlatformStager`, `hostfiles.PlatformRelay` (`Publish`, `Run`); the test seam `adoptionRecordSync`.
4166
4167- [ ] **Step 1: Write the failing tests**
4168
4169Create `internal/hostfiles/relay_test.go`:
4170
4171```go
4172package hostfiles
4173
4174import (
4175 "context"
4176 "errors"
4177 "sync"
4178 "testing"
4179 "time"
4180)
4181
4182type flakyStager struct {
4183 mu sync.Mutex
4184 fail int
4185 staged [][]byte
4186 block chan struct{}
4187}
4188
4189func (f *flakyStager) StagePlatform(_ context.Context, name string, body []byte) error {
4190 if f.block != nil {
4191 <-f.block
4192 }
4193 f.mu.Lock()
4194 defer f.mu.Unlock()
4195 if f.fail > 0 {
4196 f.fail--
4197 return errors.New("store unreachable")
4198 }
4199 f.staged = append(f.staged, append([]byte(nil), body...))
4200 return nil
4201}
4202
4203func (f *flakyStager) last() []byte {
4204 f.mu.Lock()
4205 defer f.mu.Unlock()
4206 if len(f.staged) == 0 {
4207 return nil
4208 }
4209 return f.staged[len(f.staged)-1]
4210}
4211
4212// Ruling 22: Publish never waits on the store; Run stages the latest body,
4213// retrying a failed stage, and stages nothing again once it has.
4214func TestPlatformRelayStagesTheLatestBodyWithoutBlockingItsCaller(t *testing.T) {
4215 st := &flakyStager{fail: 2, block: make(chan struct{})}
4216 r := &PlatformRelay{Name: "instance-key-adoption.json", Stage: st, Retry: 10 * time.Millisecond}
4217 ctx, cancel := context.WithCancel(context.Background())
4218 defer cancel()
4219 done := make(chan error, 1)
4220 go func() { done <- r.Run(ctx) }()
4221 published := make(chan struct{})
4222 go func() { r.Publish([]byte("one")); r.Publish([]byte("two")); close(published) }()
4223 select {
4224 case <-published:
4225 case <-time.After(time.Second):
4226 t.Fatal("Publish waited on a store that does not answer")
4227 }
4228 close(st.block)
4229 deadline := time.Now().Add(2 * time.Second)
4230 for string(st.last()) != "two" {
4231 if time.Now().After(deadline) {
4232 t.Fatalf("staged %q, want the latest body after two failed stages", st.last())
4233 }
4234 time.Sleep(5 * time.Millisecond)
4235 }
4236 time.Sleep(50 * time.Millisecond)
4237 st.mu.Lock()
4238 n := len(st.staged)
4239 st.mu.Unlock()
4240 if n != 1 {
4241 t.Fatalf("staged %d times, want once", n)
4242 }
4243 cancel()
4244 if err := <-done; !errors.Is(err, context.Canceled) {
4245 t.Fatalf("Run = %v", err)
4246 }
4247}
4248```
4249
4250Create `internal/activator/keyadoption_test.go`:
4251
4252```go
4253package activator
4254
4255import (
4256 "net/http"
4257 "net/http/httptest"
4258 "os"
4259 "path/filepath"
4260 "strings"
4261 "testing"
4262
4263 "github.com/carlosframework/platform/internal/instancekey"
4264 "github.com/carlosframework/platform/internal/registry"
4265)
4266
4267// adoptionFixture is one claim-launched process's control socket, served
4268// in memory: its launch state, and every record handed to the publisher.
4269type adoptionFixture struct {
4270 mux *http.ServeMux
4271 launch AdoptionLaunch
4272 hasLaunch bool
4273 published [][]byte
4274}
4275
4276func newAdoptionFixture(t *testing.T, rt registry.Route, tok, key, source string) *adoptionFixture {
4277 t.Helper()
4278 f := &adoptionFixture{hasLaunch: true}
4279 f.launch = AdoptionLaunch{
4280 Writer: instancekey.Binding{Host: rt.Host, Incarnation: "inc-1", Session: 3, Generation: "g3"},
4281 KeySHA256: instancekey.Digest(key), Source: source, Dir: t.TempDir(),
4282 Publish: func(rec []byte) { f.published = append(f.published, rec) },
4283 }
4284 f.mux = http.NewServeMux()
4285 (&ExecBackend{}).MountInstanceKeyAdoption(f.mux, rt, func() (string, error) { return tok, nil },
4286 func() (AdoptionLaunch, bool) { return f.launch, f.hasLaunch })
4287 return f
4288}
4289
4290func (f *adoptionFixture) put(token, body string) int {
4291 req := httptest.NewRequest(http.MethodPut, instanceKeyAdoptionPath, strings.NewReader(body))
4292 if token != "" {
4293 req.Header.Set("Authorization", "Bearer "+token)
4294 }
4295 w := httptest.NewRecorder()
4296 f.mux.ServeHTTP(w, req)
4297 return w.Code
4298}
4299
4300func adoptionBody(key, source string) string {
4301 return `{"sha256":"` + instancekey.Digest(key) + `","source":"` + source + `"}`
4302}
4303
4304// Rulings 20 and 21: the agent records the launch's own writer and the key
4305// the launch delivered, from its own launch state. A request cannot name a
4306// writer, a digest or a source the launch did not have.
4307func TestInstanceKeyAdoptionRecordsTheLaunchesWriter(t *testing.T) {
4308 tok := strings.Repeat("f", 64)
4309 key := strings.Repeat("K", 43) + "="
4310 rt := registry.Route{Host: "shop.example.com", Kind: "instance"}
4311 f := newAdoptionFixture(t, rt, tok, key, instancekey.SourcePlatform)
4312 recPath := filepath.Join(f.launch.Dir, rt.Host+instancekey.AdoptionSuffix)
4313 for _, tc := range []struct {
4314 name, token, body string
4315 want int
4316 }{
4317 {"no bearer", "", adoptionBody(key, "platform"), http.StatusUnauthorized},
4318 {"another host's bearer", strings.Repeat("0", 64), adoptionBody(key, "platform"), http.StatusUnauthorized},
4319 {"not a digest", tok, `{"sha256":"abc","source":"platform"}`, http.StatusBadRequest},
4320 {"an unknown source", tok, adoptionBody(key, "own"), http.StatusBadRequest},
4321 {"a session in the request", tok, `{"sha256":"` + instancekey.Digest(key) + `","source":"platform","session":9}`, http.StatusBadRequest},
4322 {"another key", tok, adoptionBody("some-other-key", "platform"), http.StatusConflict},
4323 {"another source", tok, adoptionBody(key, "config"), http.StatusConflict},
4324 } {
4325 if got := f.put(tc.token, tc.body); got != tc.want {
4326 t.Fatalf("%s: %d, want %d", tc.name, got, tc.want)
4327 }
4328 if _, err := os.Stat(recPath); !os.IsNotExist(err) || len(f.published) != 0 {
4329 t.Fatalf("%s: a record was written or published", tc.name)
4330 }
4331 }
4332 want := instancekey.AdoptionRecord(f.launch.Writer, instancekey.SourcePlatform, instancekey.Digest(key))
4333 for i := 0; i < 2; i++ { // the helper reports on every process start
4334 if got := f.put(tok, adoptionBody(key, "platform")); got != http.StatusNoContent {
4335 t.Fatalf("report %d: %d, want 204", i, got)
4336 }
4337 rec, err := os.ReadFile(recPath)
4338 if err != nil || string(rec) != string(want) {
4339 t.Fatalf("report %d: record = %q, %v", i, rec, err)
4340 }
4341 if st, _ := os.Stat(recPath); st.Mode().Perm() != 0o600 {
4342 t.Fatalf("record mode = %o, want 600", st.Mode().Perm())
4343 }
4344 if len(f.published) != i+1 || string(f.published[i]) != string(want) {
4345 t.Fatalf("report %d: published %d records", i, len(f.published))
4346 }
4347 }
4348
4349 // A configured key is recorded the same way, with its source (ruling 20).
4350 cfg := newAdoptionFixture(t, rt, tok, "from-config", instancekey.SourceConfig)
4351 if got := cfg.put(tok, adoptionBody("from-config", "config")); got != http.StatusNoContent {
4352 t.Fatalf("configured key: %d, want 204", got)
4353 }
4354 if a, err := instancekey.ParseAdoption(cfg.published[0]); err != nil || a.Source != instancekey.SourceConfig || !a.Binds(cfg.launch.Writer) {
4355 t.Fatalf("configured key's record = %+v, %v", a, err)
4356 }
4357
4358 // No launch state (a legacy launch) is the 404 an agent without the
4359 // verb answers; a replica refuses (ruling 26).
4360 none := newAdoptionFixture(t, rt, tok, key, instancekey.SourcePlatform)
4361 none.hasLaunch = false
4362 replicaRT := rt
4363 replicaRT.Replica = true
4364 replica := newAdoptionFixture(t, replicaRT, tok, key, instancekey.SourcePlatform)
4365 if got := none.put(tok, adoptionBody(key, "platform")); got != http.StatusNotFound {
4366 t.Fatalf("no launch state: %d, want 404", got)
4367 }
4368 if got := replica.put(tok, adoptionBody(key, "platform")); got != http.StatusConflict {
4369 t.Fatalf("replica: %d, want 409", got)
4370 }
4371 if len(none.published)+len(replica.published) != 0 {
4372 t.Fatal("a record was published without a writer")
4373 }
4374}
4375```
4376
4377Create `internal/activator/keyadoption_linux_test.go`:
4378
4379```go
4380package activator
4381
4382import (
4383 "errors"
4384 "net/http"
4385 "os"
4386 "path/filepath"
4387 "strings"
4388 "testing"
4389
4390 "github.com/carlosframework/platform/internal/instancekey"
4391 "github.com/carlosframework/platform/internal/registry"
4392)
4393
4394// Ruling 22 (R-C): 204 means the record is durable on disk. A nested
4395// record whose directory sync fails answers 500 and publishes nothing; the
4396// next report writes it and publishes it.
4397func TestInstanceKeyAdoptionIsDurableBeforeItAnswers(t *testing.T) {
4398 tok := strings.Repeat("f", 64)
4399 key := strings.Repeat("K", 43) + "="
4400 rt := registry.Route{Host: "shop.example.com", Kind: "instance"}
4401 f := newAdoptionFixture(t, rt, tok, key, instancekey.SourcePlatform)
4402 f.launch.Beneath = true
4403 adminTokenMu.Lock()
4404 old := adoptionRecordSync
4405 adoptionRecordSync = func(string) error { return errors.New("injected: EIO") }
4406 adminTokenMu.Unlock()
4407 if got := f.put(tok, adoptionBody(key, "platform")); got != http.StatusInternalServerError {
4408 t.Fatalf("a failing directory sync: %d, want 500", got)
4409 }
4410 if len(f.published) != 0 {
4411 t.Fatal("a record that is not durable was published")
4412 }
4413 adminTokenMu.Lock()
4414 adoptionRecordSync = old
4415 adminTokenMu.Unlock()
4416 if got := f.put(tok, adoptionBody(key, "platform")); got != http.StatusNoContent {
4417 t.Fatalf("retry: %d, want 204", got)
4418 }
4419 rec, err := os.ReadFile(filepath.Join(f.launch.Dir, rt.Host+instancekey.AdoptionSuffix))
4420 if err != nil || len(f.published) != 1 || string(rec) != string(f.published[0]) {
4421 t.Fatalf("after the retry: record %q (%v), %d published", rec, err, len(f.published))
4422 }
4423}
4424```
4425
4426Create `internal/host/instancekey_adoption_test.go`, the box-loss test of ruling 22:
4427
4428```go
4429package host
4430
4431import (
4432 "context"
4433 "encoding/json"
4434 "net/http"
4435 "net/http/httptest"
4436 "os"
4437 "path/filepath"
4438 "strings"
4439 "testing"
4440 "time"
4441
4442 "github.com/carlosframework/platform/internal/activator"
4443 "github.com/carlosframework/platform/internal/hostfiles"
4444 "github.com/carlosframework/platform/internal/instancekey"
4445 "github.com/carlosframework/platform/internal/registry"
4446 "github.com/carlosframework/platform/internal/release"
4447)
4448
4449// Ruling 22 (R-C): an adoption is published by the running session's
4450// publisher, with no admission or release after it. The box is then lost:
4451// every process and every local file is gone. The data bucket's manifest
4452// for the session's generation still holds the record, bound to the
4453// session's writer, which is what O2' pins.
4454func TestAdoptionSurvivesBoxLossWithoutAnotherAdmissionOrRelease(t *testing.T) {
4455 ctx, cancel := context.WithCancel(context.Background())
4456 defer cancel()
4457 data := release.NewDirStore(t.TempDir())
4458 writer := instancekey.Binding{Host: "shop.example.com", Incarnation: "inc-1", Session: 3, Generation: "g3"}
4459 prefix := "instances/" + writer.Host + "/" + writer.Incarnation + "/files/" + writer.Generation + "/"
4460 pub, err := hostfiles.Open(ctx, data, prefix)
4461 if err != nil {
4462 t.Fatal(err)
4463 }
4464 box := t.TempDir()
4465 filesDir, envDir := filepath.Join(box, "files"), filepath.Join(box, "env")
4466 for _, d := range []string{filesDir, envDir} {
4467 if err := os.MkdirAll(d, 0o700); err != nil {
4468 t.Fatal(err)
4469 }
4470 }
4471 // The session's continuous publisher and the adoption relay.
4472 running := make(chan error, 2)
4473 go func() {
4474 running <- hostfiles.Run(ctx, pub, filesDir, hostfiles.Options{Deadline: 40 * time.Millisecond, Scan: time.Minute, Poll: 40 * time.Millisecond})
4475 }()
4476 relay := &hostfiles.PlatformRelay{Name: instancekey.AdoptionItem, Stage: pub, Retry: 10 * time.Millisecond}
4477 go func() { running <- relay.Run(ctx) }()
4478
4479 key := strings.Repeat("K", 43) + "="
4480 tok := strings.Repeat("f", 64)
4481 launch := activator.AdoptionLaunch{Writer: writer, KeySHA256: instancekey.Digest(key), Source: instancekey.SourcePlatform, Dir: envDir, Publish: relay.Publish}
4482 mux := http.NewServeMux()
4483 (&activator.ExecBackend{}).MountInstanceKeyAdoption(mux, registry.Route{Host: writer.Host, Kind: "instance"},
4484 func() (string, error) { return tok, nil }, func() (activator.AdoptionLaunch, bool) { return launch, true })
4485 req := httptest.NewRequest(http.MethodPut, "/instance-key/adoption", strings.NewReader(`{"sha256":"`+instancekey.Digest(key)+`","source":"platform"}`))
4486 req.Header.Set("Authorization", "Bearer "+tok)
4487 w := httptest.NewRecorder()
4488 mux.ServeHTTP(w, req)
4489 if w.Code != http.StatusNoContent {
4490 t.Fatalf("adoption = %d, want 204", w.Code)
4491 }
4492
4493 // Within the publisher's continuous bound the record is committed.
4494 readRecord := func() ([]byte, error) {
4495 raw, _, err := data.Get(context.Background(), hostfiles.ManifestKey(prefix))
4496 if err != nil {
4497 return nil, err
4498 }
4499 var m hostfiles.Manifest
4500 if err := json.Unmarshal(raw, &m); err != nil {
4501 return nil, err
4502 }
4503 return hostfiles.ReadPlatform(context.Background(), data, prefix, m, instancekey.AdoptionItem, nil)
4504 }
4505 deadline := time.Now().Add(2 * time.Second)
4506 for {
4507 if _, err := readRecord(); err == nil {
4508 break
4509 }
4510 if time.Now().After(deadline) {
4511 t.Fatal("the adoption was not published within the continuous bound")
4512 }
4513 time.Sleep(10 * time.Millisecond)
4514 }
4515
4516 // Box loss: the processes stop and the disk is gone.
4517 cancel()
4518 <-running
4519 <-running
4520 if err := os.RemoveAll(box); err != nil {
4521 t.Fatal(err)
4522 }
4523 rec, err := readRecord()
4524 if err != nil {
4525 t.Fatalf("after box loss: %v", err)
4526 }
4527 a, err := instancekey.ParseAdoption(rec)
4528 if err != nil || !a.Binds(writer) || a.SHA256 != instancekey.Digest(key) || a.Source != instancekey.SourcePlatform {
4529 t.Fatalf("after box loss the bucket holds %+v (%v), want the session's record", a, err)
4530 }
4531}
4532```
4533
4534- [ ] **Step 2: Run them to verify they fail**
4535
4536Run: `nice -n 19 go test ./internal/hostfiles/ -run TestPlatformRelay -count=1; nice -n 19 go test ./internal/activator/ -run Adoption -count=1; nice -n 19 go test ./internal/host/ -run TestAdoptionSurvivesBoxLoss -count=1`
4537Expected: build failures: `undefined: PlatformRelay`, `undefined: MountInstanceKeyAdoption` (and `AdoptionLaunch`, `instanceKeyAdoptionPath`, `adoptionRecordSync`).
4538
4539- [ ] **Step 3: Write the relay**
4540
4541Create `internal/hostfiles/relay.go`:
4542
4543```go
4544package hostfiles
4545
4546import (
4547 "context"
4548 "log"
4549 "sync"
4550 "time"
4551)
4552
4553// PlatformStager is the part of a session's Publisher a PlatformRelay
4554// needs. *Publisher implements it.
4555type PlatformStager interface {
4556 StagePlatform(ctx context.Context, name string, body []byte) error
4557}
4558
4559// PlatformRelay hands one platform item to a running session's publisher
4560// without making its caller wait on the store (instance-key plan ruling
4561// 22). Publish records the latest body and returns at once; Run stages it
4562// with StagePlatform, retrying until it is staged, and the publisher's own
4563// continuous loop (Run in run.go) commits it within its deadline, as it
4564// commits every staged change. A body staged once is not staged again
4565// unless Publish is called with it again.
4566type PlatformRelay struct {
4567 Name string
4568 Stage PlatformStager
4569 Retry time.Duration // between failed stages; 0 is one second
4570
4571 mu sync.Mutex
4572 pending []byte
4573 wake chan struct{}
4574}
4575
4576func (r *PlatformRelay) signal() chan struct{} {
4577 r.mu.Lock()
4578 defer r.mu.Unlock()
4579 if r.wake == nil {
4580 r.wake = make(chan struct{}, 1)
4581 }
4582 return r.wake
4583}
4584
4585// Publish queues body as the item's next value. It never blocks.
4586func (r *PlatformRelay) Publish(body []byte) {
4587 w := r.signal()
4588 r.mu.Lock()
4589 r.pending = append([]byte(nil), body...)
4590 r.mu.Unlock()
4591 select {
4592 case w <- struct{}{}:
4593 default:
4594 }
4595}
4596
4597// Run stages queued bodies until ctx ends, and returns ctx.Err().
4598func (r *PlatformRelay) Run(ctx context.Context) error {
4599 retry := r.Retry
4600 if retry <= 0 {
4601 retry = time.Second
4602 }
4603 w := r.signal()
4604 for {
4605 r.mu.Lock()
4606 body := r.pending
4607 r.mu.Unlock()
4608 if body != nil {
4609 if err := r.Stage.StagePlatform(ctx, r.Name, body); err != nil {
4610 log.Printf("hostfiles: stage platform item %s: %v (retrying)", r.Name, err)
4611 select {
4612 case <-ctx.Done():
4613 return ctx.Err()
4614 case <-time.After(retry):
4615 }
4616 continue
4617 }
4618 r.mu.Lock()
4619 if string(r.pending) == string(body) {
4620 r.pending = nil
4621 }
4622 r.mu.Unlock()
4623 continue
4624 }
4625 select {
4626 case <-ctx.Done():
4627 return ctx.Err()
4628 case <-w:
4629 }
4630 }
4631}
4632```
4633
4634- [ ] **Step 4: Write the verb**
4635
4636Create `internal/activator/keyadoption.go`:
4637
4638```go
4639package activator
4640
4641// The instance key's adoption record (instance-key plan rev 3, rulings 19
4642// to 22). A key in <host>.env or <host>.config.env proves only that one
4643// was delivered: an app with a key of its own keeps it (rastrillo's step
4644// 1), and its data is sealed under that key. So relocation accepts a
4645// delivered key only when the app's helper, having resolved
4646// $CARLOS_INSTANCE_KEY (step 2), told the agent so, and the agent recorded
4647// it for the writer the process was admitted as. The record is a
4648// trusted-helper assertion: it stops accidental loss, and a tenant that
4649// forges one can lose only its own data (ruling 19).
4650//
4651// The verb is mounted only where a launch has a writer session: claim
4652// launches, once they bind a control socket (obligation O6). A legacy
4653// launch has no session to bind a record to, so nothing mounts it there and
4654// the helper's call is a 404 it ignores.
4655
4656import (
4657 "crypto/sha256"
4658 "cry