carlos / platform Public

Instance key provision plan rev 2: answers Astra round 1

Key-source evidence for relocation: instance.key, the platform key with an
adoption record the app's helper asks the agent to write, or a key from the
configuration (O2' replaces stage B's O2, compiled and run against stage B).
Every ensure is durable, present key included; the nested backfill opens
with mode 0; the mint and mergeEnv share one effective-key rule. The rollout
merge finding is declined (squash on the hub). Claim retries' premise is
corrected and pinned. Paul's open questions 1-4 carry provisional defaults,
not yet confirmed. Includes the ask for rastrillo #66.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude agent, operated by paul@keymail.dev f2c8149c4bc2087fe84ebb2fcf30451f9a1d40b3 parent 55b22cb
1 file changed, +1964 −296
  • docs/superpowers/plans/2026-10-10-instance-key-provision.md +1964 −296
diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md
index 2b63f255c..7b4097d20 100644
--- a/docs/superpowers/plans/2026-10-10-instance-key-provision.md
+++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md
@@ -1,87 +1,124 @@
-# Instance Key Provision (Region HA L21) Implementation Plan
+# Instance Key Provision (Region HA L21) Implementation Plan, rev 2
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-**Goal:** The platform mints every exec-backed instance's `CARLOS_INSTANCE_KEY` on the box, create-only, into the instance's own `<host>.env` before any process of the instance starts, and backfills it into every existing instance, so an app that adopts rastrillo's `carlos.InstanceKey` never mints a key the platform cannot recover.
+**Goal:** The platform mints every exec-backed instance's `CARLOS_INSTANCE_KEY` on the box, create-only and durably, into the instance's own `<host>.env` before any process of the instance starts, and backfills it into every existing instance. Relocation accepts a key only with evidence that the app actually seals with it: its own `instance.key`, the platform key plus an adoption record the app's helper asked the agent to write, or a key from the app's configuration.
-**Architecture:** One new mint in `internal/activator` (`instancekey.go`), called right after the existing `CARLOS_ADMIN_TOKEN` mint on every launch path (flat seed, flat start, nested prepare, the claim's launch source) and holding the same lock. A startup backfill in `internal/host` appends a key to every existing `<host>.env` on boxes where host claims are off. On claim boxes the launch source mints under the claim's guard before `Materialize`, and the C8 files-manifest provider's `Establish` is bound, by contract, to publish the env file before the app runs. No flag.
+**Architecture:** A small leaf package, `internal/instancekey`, holds the one rule for the key: the env parser, which file's key the app receives, and the adoption record. The activator mints by it (`instancekey.go`), right after the existing `CARLOS_ADMIN_TOKEN` mint on every launch path and under the same lock, and makes the file durable on every call. The control socket gains one verb, `PUT /instance-key/adoption`, that records adoption beside `<host>.env`. A startup backfill in `internal/host` appends a key to every existing `<host>.env` on boxes where host claims are off. On claim boxes the launch source mints under the claim's guard before `Materialize`. `appconfig.ConfiguredInstanceKey` answers the pinner's question about a configured key. Stage B's pinner (O2'), the C8 provider (O1') and the relocation install (O3') carry the evidence across boxes. No flag.
-**Tech Stack:** Go, `crypto/rand`, `encoding/base64`, `internal/instancefs` (openat2), the registry, the claim launch source.
+**Tech Stack:** Go, `crypto/rand`, `encoding/base64`, `crypto/sha256`, `internal/instancefs` (openat2), `internal/safefile`, the registry, the claim launch source, the control socket.
**Spec:**
- Decision: the L21 row of `docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md` on branch `region-ha-1b-next-build` (line 357; Paul, 2026-10-09).
- Region HA spec `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`: H6 (line 131) and §5.6 (line 539): `instance.key`, `report.token`, `<host>.env` are always continuous.
-- Framework contract: rastrillo/rastrillo discussion #66 (agreed; landed on rastrillo main as e3ea08d): `carlos.InstanceKey(own string) (string, error)` resolves (1) `own`, (2) `$CARLOS_INSTANCE_KEY` verbatim, (3) on CARLOS (`carlos.Running()`, which checks `$CARLOS_CONTROL_SOCKET`) `ErrNoInstanceKey`, (4) off CARLOS `ErrNotOnCarlos`.
-- Stage B plan `docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md` on `region-ha-1b-next-build`, ruling 40 and its "L21" section (lines 110-115), which this plan supersedes on where the key lives (see ruling 1).
+- Framework contract: rastrillo/rastrillo discussion #66 (agreed; landed on rastrillo main as e3ea08d). `carlos.InstanceKey(own string) (string, error)` resolves, in order: (1) `own`, the app's own key; (2) `$CARLOS_INSTANCE_KEY`, verbatim; (3) on CARLOS (`carlos.Running()`, which checks `$CARLOS_CONTROL_SOCKET`) with no key, `ErrNoInstanceKey`; (4) off CARLOS, `ErrNotOnCarlos`, today's behaviour. Read at e3ea08de in `~/amadan.net/rastrillo/rastrillo` (`carlos/instancekey.go`, `carlos/schedule.go:28`).
+- Stage B plan `docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md` on `region-ha-1b-next-build` (1651ff7d): rulings 25 and 54, Task 8b step 0 (O2) and Task 8c (O3). This plan's O2' replaces O2.
+- Review: Astra round 1, `docs/superpowers/plans/CODEX-PLAN-INSTANCE-KEY-R1.md` (verdict: not ready). Its findings are answered in the table below.
+
+## Rev 2: dispositions of Astra round 1
+
+Each finding was checked against the code on this branch (cut from origin/main 0901e835) before it was answered. "Provisional default" rows are Paul's open questions 1 to 4: they are the defaults this plan builds on, and **Paul has not yet confirmed them**.
+
+| # | Finding | Disposition | Change | Test |
+|---|---|---|---|---|
+| 1 | P1. O2 accepts a platform key the app may never have used: the app's own `instance.key` wins in rastrillo, and a missing `instance.key` beside a backed-up platform key would pass. | **Accepted.** Verified: rastrillo e3ea08de `InstanceKey` returns `own` first, and rev 1's mint ran whatever the app used. | Ruling 15: relocation accepts a key only with positive evidence, (a) `instance.key`, (b) the platform key plus an adoption record for that exact key, or (c) a key from the configuration. The control socket has no verb for (b) today, so Task 5 adds `PUT /instance-key/adoption` and the record beside `<host>.env`. O1' publishes the record, O2' requires it, and the rastrillo ask below asks the helper to call it. | O2' `TestStorePinnerKeyEvidence` (compiled and run on a stage B scratch), row "a backed-up platform key, the app's own key missing" refuses; `TestInstanceKeyAdoptionRecordsOnlyTheOwnFilesKey`; `TestNestedControlSocketRecordsAdoptionDurably` |
+| 2 | P1. A failed durability step becomes success on retry: after the append, a failed chmod or sync leaves a key the next ensure returns early on; the claim early return skips it too. | **Accepted.** Verified at rev 1's lines 585-647 and claim_snapshot.go:56-58. | Ruling 4: every ensure ends with fchmod 0600, file fsync and directory fsync, also when the key is already there; the bytes are never rewritten. The claim early return fsyncs the file and its directory without writing (ruling 18), and takes the guarded path when the mode needs setting. | `TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess`, `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced`, `TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers` |
+| 3 | P2. Every nested backfill that appends fails with `EINVAL`: `create=false` drops `O_CREAT` but still passes mode 0600 to openat2. | **Accepted.** Verified: `instancefs.OpenBeneath` puts `perm` in `OpenHow.Mode` (instancefs_linux.go:93-97), and openat2 refuses a non-zero mode without `O_CREAT`. | `openInstanceEnv` passes mode 0 unless it creates. The fchmod on the descriptor sets 0600. | `TestBackfillInstanceKeyAppendsToANestedEnvFile` |
+| 4 | P2. An empty `CARLOS_INSTANCE_KEY=` in `<host>.env` defeats a configured key: the mint defers to config, but `mergeEnv` hands the child the empty override. | **Accepted.** Verified: `mergeEnv` lets the overlay win every conflict, empty values included (backend_exec.go:1980-1990). | Ruling 13: the mint decides on the effective env (`instancekey.Effective`), and `mergeEnv` applies the same rule, so an empty own key never shadows a non-empty configured one. The child never gets an empty key while a non-empty one exists. `LaunchEnvironment` stops reporting that name as shadowed. | `TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists` (both files, through `Start`); `TestEffectiveNeverHandsOutAnEmptyKeyWhileOneExists`; `TestEnsureInstanceKeyDefersToTheRenderedConfiguration` |
+| 5 | P2. O2 refuses a key that comes from the configuration, which ruling 5 supports. | **Accepted.** | Ruling 15 (c): the configuration in the bucket is the evidence, and the target renders the same configuration. `appconfig.ConfiguredInstanceKey` renders it with SyncExec's own loader and renderer; O2' calls it through `StorePinner.ConfigKey`. | `TestConfiguredInstanceKeyReadsTheRenderedConfiguration`; O2' rows "(c) ..." |
+| 6 | P3. The rollout's `amadan branch merge -squash` contradicts AGENTS.md's local-squash workflow. | **Declined.** The hub has `-squash`, and Paul's standing rule (2026-09-22) is squash-merge on the hub. AGENTS.md's local-squash text predates that rule. | Rollout step 1 now says: merging is Paul's call, as a squash on the hub, after CI is green on the tip. | none |
+| 7 | Open question 5: claim-session restarts can mint outside the C8 publish point. Astra: the premise is not established. | **Accepted: the premise was wrong.** Verified: `startAdmitted` calls `Prepare` once (claimwake.go:355) and loops over attempts, calling `Materialize` before each (:405-412); the assembly's inputs are captured once (claimwakeassembly.go:353, :377); production claim swaps refuse (`claimsSwapRefusal`, claimgen.go:30). | Ruling 16 states the corrected premise. Question 5 is removed. | `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` |
+| 8 | Coverage gaps: a first flat seed, the nested seed role, a real hibernation restore (a second `Start` does not exercise `Restore`). | **Accepted.** | Tests added. | `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`; `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile/seed`; `TestHibernationRestoreKeepsTheInstanceKey` |
+| 9 | O1's capturing fake proves ordering only, not committed publication or refusal on a failed publication. | **Accepted, moved.** Those need the provider. | O1' names the provider's integration tests, in the C8 plan. Task 4's test keeps proving the ordering. | in the C8 provider's plan (O1') |
+| 10 | Legacy durability is disk plus move archives: no off-box recovery after box loss. | **Acknowledged, no change.** | Ruling 6 already says so. Relocation cannot run on a legacy box. | none |
+| Q1 | Replicas get their own key; should a replica get its home's key? | **Provisional default, Paul not yet confirmed.** Deferred to a follow-up. | Ruling 17: until then replicas are excluded from platform-key adoption. No path mints a key on a replica row, the backfill skips replicas, and the adoption verb refuses on one. An adopting app on a replica refuses to start unless it passes its own key. | `TestReplicaRowsGetNoPlatformKey`; the backfill test's replica row; the adoption test's replica case |
+| Q2 | A key set in configuration: defer to it, or reserve the name? | **Provisional default, Paul not yet confirmed.** | Configuration wins (ruling 5). The name is not reserved. | `TestEnsureInstanceKeyDefersToTheRenderedConfiguration` |
+| Q3 | Should the platform provision `report.token`? | **Provisional default, Paul not yet confirmed.** | No `report.token`. | none |
+| Q4 | Workers and sidecars: should they get their instance's key? | **Provisional default, Paul not yet confirmed.** | A follow-up; out of scope (ruling 12). | none |
+
+**Counts:** findings 1 to 10: 8 accepted, 1 declined (6), 1 acknowledged with no change (10). Open questions 1 to 4: 4 provisional defaults.
+
+**Found while revising (not in Astra's list):**
+- Stage B's O2 parses the env item with its own rule, accepting `export CARLOS_INSTANCE_KEY="k1"`. The activator's parser reads that line as the variable `export CARLOS_INSTANCE_KEY`, so the app never receives it. O2' parses with `instancekey.ParseEnv`, the activator's own parser, and its test pins the exported line as no key.
+- Claim launches carry no control socket. `SnapshotClaimLaunch` composes the env with `agentOwnedVars(rt, "", vars, false)` (claim_snapshot.go:116), which strips `CARLOS_CONTROL_SOCKET`. An app first adopting the platform key on a claim box cannot record adoption until claim launches bind one (obligation O6).
## Global Constraints
- **Format (rastrillo #66):** 32 CSPRNG bytes, standard padded base64 (44 characters). The key is the exact bytes the env file yields: no quotes, no trailing newline in the value, no trimming. Changing the encoding later is a key rotation.
- **Create-only:** a `CARLOS_INSTANCE_KEY` already in `<host>.env` is never rewritten, by any path, whatever its shape.
+- **Durable before success:** an ensure that reports success has set 0600 on the file, fsynced it and fsynced its directory, in that call.
- **Variable name:** `CARLOS_INSTANCE_KEY`, exactly.
-- **File mode:** `<host>.env` is 0600, as `EnsureAdminToken` leaves it (admintoken.go:104-117).
-- **Never logged, never in an error, never in the console.** Errors and log lines name paths, never values.
+- **File mode:** `<host>.env` and the adoption record are 0600.
+- **Never logged, never in an error, never in the console.** Errors and log lines name paths, never values. The adoption record holds a SHA-256 digest, never the key.
- **Rollout order (rastrillo #66):** the key is on every box, existing instances included, before any app adopts the helper.
- **Neutral examples:** `shop`, `example.com`, `acct`. No tenant or app names in code, tests or comments.
-- **Cross-compile:** `internal/host` and `internal/activator` build for darwin, windows and linux/arm64 (`20-cross`). Tests that need openat2 or `mkfifo` go in `*_linux_test.go`.
+- **Cross-compile:** `internal/host`, `internal/activator`, `internal/appconfig` and `internal/instancekey` build for darwin, windows and linux/arm64 (`20-cross`). Tests that need openat2, `mkfifo` or a unix socket go in `*_linux_test.go`.
- **Gate (AGENTS.md "Build & test"):** `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean.
-- **No user-facing copy.** Log lines are operator-facing, in the surrounding style.
+- **No user-facing copy.** Log lines are operator-facing, in the surrounding style. The adoption verb answers with `http.StatusText` bodies only.
-## What the code does today (branch `instance-key-provision`, cut from main 0d338c14)
+## What the code does today (branch `instance-key-build`, cut from origin/main 0901e835)
**Two things are called "overlay".** The decision's "overlay env file, alongside `$CARLOS_ADMIN_TOKEN`" is `<host>.env`, the instance's own env file:
- `ExecBackend.pathsFor` names it `<dataDir>/<host>.env` beside `<host>.config.env` (internal/activator/paths.go:50-52); nested, `dataDir` is the instance directory (paths.go:36-37).
-- `mergeEnv` calls it the overlay: "base (the sync-owned `<host>.config.env`, bucket-derived) under overlay (the instance's own `<host>.env`). The overlay wins every conflict — `CARLOS_ADMIN_TOKEN` lives only in the instance file" (internal/activator/backend_exec.go:1974-1990).
+- `mergeEnv` calls it the overlay: "base (the sync-owned `<host>.config.env`, bucket-derived) under overlay (the instance's own `<host>.env`). The overlay wins every conflict" (internal/activator/backend_exec.go:1974-1990). It wins with an empty value too: `CARLOS_INSTANCE_KEY=` in `<host>.env` hands the child an empty key over a configured one (finding 4).
- The moves archive calls it "the `<host>.env` overlay" (internal/moves/archive/archive_test.go:36-37), and region HA's H6 names `<host>.env` as always continuous.
-The other overlay is appconfig's instance config layer, `apps/<acct>/<app>/instances-config/<host>.json`, written by `appconfig.ApplyInstance` (internal/appconfig/instances.go:1-24, 371-376). It is app configuration, sealed with the deployment's secrets key, rendered into `<host>.config.env` by `SyncExec` (internal/appconfig/syncexec.go:1-9; the file suffix at :57). Sync never reads, writes or reaps `<host>.env` (internal/appconfig/sync.go:270-275, pinned by `TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes`, syncexec_test.go:683). The stage B plan's L21 section points at this layer (`ApplyInstance`, `syncexec.go`); that is the wrong file for this key (ruling 1).
+The other overlay is appconfig's instance config layer, `apps/<acct>/<app>/instances-config/<host>.json`, written by `appconfig.ApplyInstance` (internal/appconfig/instances.go:1-24, 371-376). It is app configuration, sealed with the deployment's secrets key, rendered into `<host>.config.env` by `SyncExec` (internal/appconfig/syncexec.go:1-9). `LoadMergedHostVersions` (instances.go:222) and `RenderEnv` (materialize.go:37) are the one loader and renderer that SyncExec, the held render and `CheckRenderedIdentity` (claim_identity.go:13) share. Sync never reads, writes or reaps `<host>.env` (internal/appconfig/sync.go:270-275, pinned by `TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes`, syncexec_test.go:683).
**The admin token is the precedent** (internal/activator/admintoken.go):
-- minted by `ensureAdminToken` (:142-238): one read of the raw bytes, `parseEnvBytes` (:245-257, which trims whitespace around keys and values), 32 `crypto/rand` bytes hex-encoded, appended with `O_APPEND|O_CREATE` at 0600 on a fresh line (:205-229), then `chmod 0600` (:234);
-- create-only: an existing token is never rewritten; a malformed one is logged once an hour per file (`logPerFile`, :70-77) and left in place (:189-203);
+- minted by `ensureAdminToken` (:142-238): one read of the raw bytes, `parseEnvBytes` (:245-257), 32 `crypto/rand` bytes hex-encoded, appended with `O_APPEND|O_CREATE` at 0600 on a fresh line (:205-229), then `chmod 0600` (:234). It never fsyncs;
+- create-only, malformed tokens logged once an hour per file (`logPerFile`, :70-77) and left in place (:189-203);
- serialized by the process-wide `adminTokenMu` (:43);
-- nested files go through `instancefs.ReadBeneathLimit` / `OpenBeneath` (:150-216).
+- nested files go through `instancefs.ReadBeneathLimit` / `OpenBeneath` (:150-216). `OpenBeneath` passes its `perm` straight into openat2's `OpenHow.Mode` (internal/instancefs/instancefs_linux.go:93-97); the kernel refuses a non-zero mode without `O_CREAT` with `EINVAL` (finding 3).
-Its callers are every launch path, each before the env is read: `seedChild` (backend_exec.go:831), `spawnChildFrom`, which flat `Start` and `StartGen` share (:1369), the nested `prepareAt` for the run and seed roles (internal/activator/prepare.go:60), the claim's `EnsureClaimAdminToken` (internal/activator/claim_snapshot.go:40-67), called by the production launch source `claimServingLaunchSource` under `claimAdminTokens.With`, the claim's guarded write (internal/host/claimproduction.go:356-428), and the scheduler tick's `scheduleToken` (internal/host/schedule_wiring.go:150-180).
+Its callers are every launch path, each before the env is read: `seedChild` (backend_exec.go:831), `spawnChildFrom`, which flat `Start` and `StartGen` share (:1369), the nested `prepareAt` for the run and seed roles (internal/activator/prepare.go:60), the claim's `EnsureClaimAdminToken` (internal/activator/claim_snapshot.go:40-67), called by the production launch source `claimServingLaunchSource` under `claimAdminTokens.With` (internal/host/claimproduction.go:356-428), and the scheduler tick's `scheduleToken` (internal/host/schedule_wiring.go:150-180).
-**Creation against first start.** The console writes the instance record. The box's reconcile mints the registry row with `Status: "ready"` and starts nothing (internal/provision/reconcile.go:425-435). The first process is started by the first wake, `doWake`: `Restore` (internal/activator/activator.go:1797), then `Seed` when the database is fresh (:1817), then `Start`. Both seed and start mint before they read the env. So every instance's first process already starts after the token mint, and after this plan's key mint too.
+**Creation against first start.** The console writes the instance record. The box's reconcile mints the registry row with `Status: "ready"` and starts nothing (internal/provision/reconcile.go:425-435). The first process is started by the first wake, `doWake`: `Restore` (internal/activator/activator.go:1797), then `Seed` when the database is fresh (:1817), then `Start`. `Restore` restores the database only (backend_exec.go:663-757). Both seed and start mint before they read the env.
+
+**The control socket.** `edge.ListenControl` (internal/edge/purge.go:126) binds `<socket>.control` before every exec spawn and serves `POST /purge` (when there is a cache) plus whatever the activator mounts: today `schedule.MountGated`, which serves `GET /schedule-capabilities`, `PUT` and `DELETE /schedule/{name}` and `GET /schedule`, each bearer-gated on the host's own `CARLOS_ADMIN_TOKEN` (internal/schedule/control.go:97-225). It is mounted in `spawnChildFrom` (backend_exec.go:1406-1414) and in the nested `listenControl` (prepare.go:135-155), and only when `b.Timers != nil`. Rastrillo e3ea08de's `carlos` package calls only `PUT` and `DELETE /schedule/{name}` (`carlos/schedule.go`). **Nothing on the socket lets an app tell the agent which key it uses,** so source (b) needs a new verb here and a call in rastrillo (the ask below).
+
+**Claim launches have no control socket.** `SnapshotClaimLaunch` composes the launch env with `agentOwnedVars(rt, "", vars, false)` (claim_snapshot.go:116), which strips `CARLOS_CONTROL_SOCKET`. A claim-launched app is not `carlos.Running()`, but rastrillo's step 2 still returns `$CARLOS_INSTANCE_KEY`. It cannot call the adoption verb (O6).
+
+**Claim retries.** `startAdmitted` reads the claim, calls `Prepare` once (internal/activator/claimwake.go:355), restores, then loops over attempts: `Reserve`, then `Materialize`, then the start (:405-412). The assembly captures the launch inputs once, from the selection (`_, in := a.snapshot()`, internal/host/claimwakeassembly.go:353), and `Materialize` is bound to them (:377). `ClaimStart` documents that "process retries reuse it". Production claim swaps refuse before they raise anything (`claimsSwapRefusal`, internal/activator/claimgen.go:30). So no claim attempt reaches the launch source again, and none runs before its `Materialize`.
**Backup of `<host>.env` today.**
-- **Legacy boxes** (host claims off; every production box today): nothing publishes `<host>.env`. `internal/hostfiles` (publisher, `StagePlatform` publisher.go:128-150, `Commit` :152) is imported by no production package; its own header says "Both are dormant: nothing in a production path calls them yet" (internal/hostfiles/contract.go:13-15). `hostfiles.EnvItem` does not exist on main; it is on `region-ha-1b-next-build` (internal/hostfiles/platform.go:8-13 there), used by stage B's release flush (internal/host/relocationflush.go:64-70 there). The file's only off-box copy is a move's sealed archive (`archive.Build`, internal/moves/archive/archive.go:395).
-- **Claim boxes:** the serving owners call the launch source in `Select` (internal/host/claimowners.go:375-416) and then `Materialize`, which calls `ClaimFilesManifest.Establish` "before the app executes" (claimowners.go:125-138, 425-438). No C8 provider exists on main: `Files == nil` refuses every serving wake (`claimFilesRefusal`, :140-143), and moves' install records the files manifest step as skipped "until C8 exists" (internal/host/movesinstall.go:700-712).
+- **Legacy boxes** (host claims off; every production box today): nothing publishes `<host>.env`. `internal/hostfiles` is imported by no production package (internal/hostfiles/contract.go:13-15). `hostfiles.EnvItem` exists only on `region-ha-1b-next-build` (internal/hostfiles/platform.go:13 there). The file's only off-box copy is a move's sealed archive (`archive.Build`, internal/moves/archive/archive.go:395).
+- **Claim boxes:** the serving owners call the launch source in `Select` (internal/host/claimowners.go:375-416) and then `Materialize`, which calls `ClaimFilesManifest.Establish` "before the app executes" (claimowners.go:125-138, 425-438). No C8 provider exists on main: `Files == nil` refuses every serving wake (`claimFilesRefusal`, :140-143).
+
+**Moves (L16's shared install routine on main).** `installFiles` extracts the archive and `placeStaged` moves the env into place only when `<host>.env` is absent or is this install's own (internal/host/moveslandinstall.go:130-280). The move's env report HMACs every value and classes any name it does not know as `equal` (internal/moves/envreport.go:24-38, 90-103, 117-150). The archive carries the data directory and `<host>.env` only. The source's finalization parks a fixed, journalled list of flat paths (`installArtifactPaths`, internal/host/movesinstall.go:751; `sourceArtifactPaths`, movesfinal.go:586).
-**Moves (L16's shared install routine on main).** `installFiles` extracts the archive into staging and `placeStaged` moves the env into place only when `<host>.env` is absent or is this install's own (internal/host/moveslandinstall.go:130-280; the refusal at :239-244). `TestLandingNeverDeletesFilesItDidNotCreate/only_an_env_file` (internal/host/movesland_finalfix_test.go:85-99) pins that a landing never installs over an existing env file. The move's env report HMACs every value and classes any name it does not know as `equal` (internal/moves/envreport.go:24-38, 90-103, 117-150).
+**The layout pass** moves every flat entry the flat naming rules give a host (`<host>` plus a dot or a dash and more) into the instance directory at the same name, and back on a revert (internal/layout/extras.go:14-60, `belongsTo`). An adoption record named `<host>.instance-key-adoption.json` travels with it.
+
+**Relocation's pinner (stage B branch).** `StorePinner.Pin` runs on the region runtime, which holds the control store and the box's secrets key (`RelocationAssemblyConfig.SecretsKey`, from `CARLOS_SECRETS_KEY`). With a nonempty database written by an admitted writer it requires `instance.key` in the pinned manifest (`needKey`, pin.go:250 there). The target renders its held config from the same bucket configuration (`appconfig.RenderHeldUnder` in Task 8c), over the route `relocationRecord` mints.
## Design and rulings
-Each ruling ends with its cost if wrong.
+Each ruling ends with its cost if wrong. Rulings 1 to 12 are rev 1's, amended where marked. Rulings 13 to 18 are new in rev 2.
-**1. The key lives in `<host>.env`, and is minted on the box.** This is the decision, and the code allows it: `<host>.env` is already the platform-managed per-instance secret file, H6 makes it always continuous, moves carry it, and stage B restores it (`EnvItem`). Minting in the console at create, into appconfig's instance config layer, is rejected. That layer is the app's own editable configuration (`carlos env unset` would delete the key), it exists only for instances given per-instance config, and it renders into `<host>.config.env`, which sits *under* `<host>.env` in `mergeEnv`. *Cost if wrong:* if Paul wanted the key in the bucket before the box first sees the instance, the C8 obligation in ruling 6 would be unnecessary. Moving the key later is a copy of the same bytes, not a rotation.
+**1. The key lives in `<host>.env`, and is minted on the box.** `<host>.env` is already the platform-managed per-instance secret file, H6 makes it always continuous, moves carry it, and stage B restores it (`EnvItem`). Minting in the console, into appconfig's instance config layer, is rejected: that layer is the app's own editable configuration (`carlos env unset` would delete the key), and it renders into `<host>.config.env`, which sits *under* `<host>.env` in `mergeEnv`. *Cost if wrong:* if Paul wanted the key in the bucket before the box first sees the instance, O1' would be unnecessary. Moving the key later is a copy of the same bytes, not a rotation.
-**2. Format: `base64.StdEncoding` of 32 bytes from `crypto/rand`, 44 characters, one line `CARLOS_INSTANCE_KEY=<44 chars>\n`.** #66 says "base64" unqualified, and the standard alphabet with padding is Go's default. Every reader of `<host>.env` yields these bytes exactly: `parseEnvBytes` cuts at the first `=` and trims whitespace, which base64 never contains, and the nested launch file passes `KEY=value` strings to execve unchanged (prepare.go:204). A malformed existing key is reported, never repaired (ruling 4). *Cost if wrong:* the encoding is fixed by the first mint. Changing it later is a rotation for every adopted app.
+**2. Format: `base64.StdEncoding` of 32 bytes from `crypto/rand`, 44 characters, one line `CARLOS_INSTANCE_KEY=<44 chars>\n`.** Every reader of `<host>.env` yields these bytes exactly: the parser cuts at the first `=` and trims whitespace, which base64 never contains, and the nested launch file passes `KEY=value` strings to execve unchanged (prepare.go:204). *Cost if wrong:* the encoding is fixed by the first mint. Changing it later is a rotation for every adopted app.
-**3. Every launch path mints, right after the admin token, under the same lock.** These are `seedChild`, `spawnChildFrom`, `prepareAt` (run and seed roles) and `EnsureClaimAdminToken`. The key mint takes `adminTokenMu`, so a token append and a key append never interleave. The scheduler tick does not mint the key: its ensure exists for the token it presents, and any tick that wakes the instance goes through `Start`, which mints. Creation needs no new step: no process of an instance starts before `doWake`'s seed or start, and both mint first. A failed mint fails the wake, the same way a failed token mint does. *Cost if wrong:* if a launch path were missed, an adopting app would refuse to start there (`ErrNoInstanceKey`), loudly. The tests in Tasks 2 and 3 cover each path.
+**3. Every launch path mints, right after the admin token, under the same lock.** These are `seedChild`, `spawnChildFrom`, `prepareAt` (run and seed roles) and `EnsureClaimAdminToken`, through one helper, `ensureInstanceKeyFor(rt, paths)`. The scheduler tick does not mint: any tick that wakes the instance goes through `Start`. A failed mint fails the wake, as a failed token mint does. *Cost if wrong:* a missed launch path makes an adopting app refuse to start there (`ErrNoInstanceKey`), loudly.
-**4. Create-only, append-only, durable.** A non-empty `CARLOS_INSTANCE_KEY` in `<host>.env`, of any shape, is left exactly as it is: two lines are left as two, and readers take the last, as `parseEnvBytes` does. An empty value is no key: one is appended after it, matching the token's `tok != ""`. The key is appended, never rewritten, on a fresh line. The file is set to 0600 by `fchmod` on the descriptor that wrote it. Then the file and its directory are `fsync`ed before the call returns. A key handed to a process must survive a power cut. Without the sync, a reboot could lose the line, and the next start would mint a second key over data sealed with the first. *Cost if wrong:* one fsync per instance lifetime.
+**4. Create-only, append-only, durable every time (amended for finding 2).** A non-empty `CARLOS_INSTANCE_KEY` in `<host>.env`, of any shape, is left exactly as it is: two lines stay two, and readers take the last. The key is appended on a fresh line, never rewritten. Every ensure that reports success, whether it minted or found the key, has done three things on that call: `fchmod 0600` on a descriptor it opened, fsync of the file, and fsync of its directory. A mint whose sync failed returns the error and leaves its line in place. The next ensure finds the key present, keeps its bytes and redoes all three steps. It succeeds only when they succeed, so no process is started with a key a power cut could take back. An existing file is opened with mode 0 (finding 3). *Cost if wrong:* one open, fchmod and two fsyncs per launch, on a file of a few hundred bytes.
-**5. A key from the app's configuration wins, and the platform mints none.** If the rendered `<host>.config.env` already carries a non-empty `CARLOS_INSTANCE_KEY` (the app set one with `carlos secrets set`), the mint writes nothing and logs once an hour. Minting would put a key into `<host>.env`, which overrides the configured one in `mergeEnv` and changes the app's key under its data. The configured key's durability is the bucket's. The claim's early return counts a configured key as present for the same reason. *Cost if wrong:* an app that sets the name and later unsets it gets a fresh platform key at its next start, by its own action. Reserving the name instead is open question 2.
+**5. A key from the app's configuration wins, and the platform mints none (amended for finding 4).** If the effective env (ruling 13) takes its key from `<host>.config.env`, the mint writes nothing and logs once an hour. Minting would put a key into `<host>.env`, which overrides the configured one, and change the app's key under its data. *Cost if wrong:* an app that sets the name and later unsets it gets a fresh platform key at its next start, by its own action. Reserving the name instead was open question 2; the provisional default is not to.
**6. Durability before the first start.**
-- *Legacy boxes:* nothing can publish today, and region HA cannot be on, because it requires host claims (stage B Global Constraints). The key's durability is the admin token's: the box's disk, now fsynced, and a move's sealed archive.
-- *Claim boxes* are region HA's only path. The launch source mints the key in `Select`. `Materialize` then calls `ClaimFilesManifest.Establish`, and the app executes only after that returns. The C8 provider is bound to obligation O1 below: the manifest `Establish` leaves carries the env item with exactly the bytes on disk, committed before it returns.
-- *Bucket unreachable at first start:* `Establish` fails, so the wake fails and the app does not start. The key stays on disk unchanged. The next wake publishes the same bytes, because the mint is create-only and a retry never makes a second key.
+- *Legacy boxes:* nothing can publish today, and region HA cannot be on, because it requires host claims. The key's durability is the box's disk, now fsynced, and a move's sealed archive. A dead legacy box loses its keys with its data; that is today's state, unchanged (row 10).
+- *Claim boxes* are region HA's only path. The launch source mints in `Select`. `Materialize` then calls `ClaimFilesManifest.Establish`, and the app executes only after that returns. O1' binds the provider to publish the env file and the adoption record, committed, before it returns.
+- *Bucket unreachable at first start:* `Establish` fails, so the wake fails and the app does not start. The key stays on disk unchanged. The next wake publishes the same bytes.
-This plan builds the part that is on main: the key is on disk when `Establish` runs (Task 3's test), and the obligation is written on the interface (Task 3). *Cost if wrong:* a C8 provider that ignores O1 reopens the window for every key. Stage B's pinner, amended per O2, is the backstop: such a host goes `stuck` and is never relocated without its key.
+*Cost if wrong:* a C8 provider that ignores O1' reopens the window. O2' is the backstop: without evidence, the host goes `stuck` and is never relocated without its key.
**7. The backfill: the agent at startup, legacy boxes only, existing files only.**
-- *Who and when:* `runAgent`, inside the startup closure that runs after layout recovery has settled every journal, and before the layout pass is published and before `svc.Start` wakes anything (internal/host/agent.go:476-484). Nothing moves a file under it.
-- *Which rows:* ready, activator-spawned instances (`Kind == "instance"`, a socket, not `Platform`, not unit-backed, no tunnel, `Status` empty or `ready`). Held rows (`moving`, `landing`, `restoring`, `releasing`; registry.go:1164), provisioning rows, sidecars and TCP-addr routes are skipped.
-- *Existing files only:* a missing `<host>.env` means the host was never started here, and its first start mints the key. Creating one could occupy the path a landing's `placeStaged` needs empty.
-- *Idempotent:* a present key writes nothing, so every agent start re-runs it harmlessly.
-- *Never overwrites:* it is the same create-only body.
+- *Who and when:* `runAgent`, inside the startup closure that runs after layout recovery has settled every journal, and before the layout pass is published and before `svc.Start` wakes anything (internal/host/agent.go:476-484).
+- *Which rows:* ready, activator-spawned instances (`Kind == "instance"`, a socket, not `Platform`, not unit-backed, no tunnel, not a replica, `Status` empty or `ready`). Held rows (`moving`, `landing`, `restoring`, `releasing`), provisioning rows, sidecars, replicas and TCP-addr routes are skipped.
+- *Existing files only:* a missing `<host>.env` means the host was never started here, and its first start mints. Creating one could occupy the path a landing's `placeStaged` needs empty.
+- *Idempotent:* a present key writes nothing (it is made durable), so every agent start re-runs it harmlessly.
- *Visible:* it logs one count line, which the rollout reads.
-- *Claim boxes are not backfilled.* Every write there runs under the claim's guard, and the claim's launch source already mints under that guard before every cold admission.
-- *The decision's "forced publish" is replaced by ruling 6:* every cold admission publishes the env, with its key, before the app runs. No process ever holds a platform key that its generation's manifest lacks. A running instance started before the roll has no key, and its app cannot be using one.
+- *Claim boxes are not backfilled.* The claim's launch source mints under the claim's guard before every cold admission.
*Cost if wrong:* a box whose registry cannot be listed at startup logs it, and its instances get keys at their next start instead.
@@ -89,93 +126,676 @@ This plan builds the part that is on main: the key is on disk when `Establish` r
| Path | What guarantees it | Test |
|---|---|---|
-| create (first seed or start) | create-only mint | Task 1 `TestEnsureInstanceKeyMintsOnceInTheAgreedFormat`; Task 2 `TestExecBackendStartHandsTheChildTheMintedInstanceKey` (second start, same key), `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` |
-| backfill | same body, existing files only | Task 1 `TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile`; Task 4 `TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances` |
-| any existing key | create-only, any shape | Task 1 `TestEnsureInstanceKeyNeverRewritesAnExistingKey`, `...LeavesAFileWithTwoKeysAlone` |
-| claim launch, retried | early return; create-only | Task 3 `TestClaimLaunchMintsAKeyBesideAnExistingToken` |
-| re-render (config change) | sync never touches `<host>.env` | Task 5 `TestSyncExecLeavesTheInstanceKeyAlone` (also a config that sets the same name) |
-| restore (hibernation wake) | `Restore` restores the database only; `Start` mints create-only after it | Task 2's second start |
-| move | landing installs env only on an empty path (existing test); archive carries bytes exactly; env report fails a missing or different key | Task 5 `TestArchiveCarriesTheInstanceKeyByteForByte`, `TestEnvReportComparesTheInstanceKey`; existing `TestLandingNeverDeletesFilesItDidNotCreate` |
-| relocation install | stage B writes `EnvItem`'s bytes; the next launch finds the key present | obligation O3 (stage B branch), test code below |
-| layout move (flat to nested) | `<host>.env` is one of the artifacts the pass moves by name | existing layout pass tests |
+| create (first seed or start) | create-only mint | Task 2 `TestEnsureInstanceKeyMintsOnceInTheAgreedFormat`; Task 3 `TestExecBackendStartHandsTheChildTheMintedInstanceKey`, `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`, `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` (run and seed) |
+| a retry after a failed sync | the present path keeps the bytes and redoes the durability | Task 2 `TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess`; Task 3 `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced` |
+| backfill | same body, existing files only | Task 2 `TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile`; Task 3 `TestBackfillInstanceKeyAppendsToANestedEnvFile`; Task 6 `TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances` |
+| any existing key | create-only, any shape | Task 2 `TestEnsureInstanceKeyNeverRewritesAnExistingKey`, `...LeavesAFileWithTwoKeysAlone` |
+| claim launch, retried | the launch source mints create-only; a session's attempts reuse their inputs | Task 4 `TestClaimLaunchMintsAKeyBesideAnExistingToken`, `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` |
+| re-render (config change) | sync never touches `<host>.env` | Task 7 `TestSyncExecLeavesTheInstanceKeyAlone` |
+| restore (hibernation wake) | `Restore` restores the database only; `Start` mints create-only after it | Task 3 `TestHibernationRestoreKeepsTheInstanceKey` |
+| move | landing installs env only on an empty path; archive carries bytes exactly; env report fails a missing or different key | Task 7 `TestArchiveCarriesTheInstanceKeyByteForByte`, `TestEnvReportComparesTheInstanceKey`; existing `TestLandingNeverDeletesFilesItDidNotCreate` |
+| relocation install | stage B writes `EnvItem`'s bytes; the next launch finds the key present | O3' (stage B branch) |
+| layout move (flat to nested) | `<host>.env` is one of the pass's named artifacts; the adoption record is one of its extras | existing layout pass tests |
+| adoption record | written only for the own file's key; identical bytes are not rewritten | Task 5 `TestInstanceKeyAdoptionRecordsOnlyTheOwnFilesKey` |
**9. Secrecy.**
- No log line or error takes the value. The shape warning names the path only, and a test checks the value is not in the log.
-- `<host>.env` is 0600 (ruling 4), as the token leaves it. The nested launch file that carries the env to the trampoline is 0600 (prepare.go:204).
-- The console never reads a box's `<host>.env`: no console API or view carries it.
-- `carlos env list` reads the bucket configuration through the console API (appconfig_cli.go:462-507), never `<host>.env`, so the platform key cannot appear in it. `carlos secrets list` shows names only (appconfig_cli.go:1039). A key an app owner chose to set with plain `carlos env set` is listed like any plain variable, by their choice.
-- The moves env report carries HMACs only.
-- Once region HA publishes the env item, the key sits in the data bucket beside the database it seals, so a reader of that bucket holds both. That is inherent in making the key recoverable with the data. It is recorded here, not changed.
+- `<host>.env` is 0600 (ruling 4), as is the adoption record and the nested launch file (prepare.go:204).
+- The adoption record holds `sha256(key)`, never the key. It is published beside the env item, which holds the key itself, so it gives a bucket reader nothing new. A configured key is never recorded (ruling 14), so no digest of a configuration secret is published.
+- The console never reads a box's `<host>.env`. `carlos env list` reads the bucket configuration through the console API (appconfig_cli.go:462-507), never `<host>.env`. The moves env report carries HMACs only.
+- Once region HA publishes the env item, the key sits in the data bucket beside the database it seals. That is inherent in making the key recoverable with the data.
*Cost if wrong:* a leak of one instance's key exposes only what that app sealed with it.
**10. Moves and relocation carry the key inside `<host>.env`.**
- A move archives the file, sealed (archive.go:395), extracts it byte for byte (extract.go:35), and places it only on an empty path (moveslandinstall.go:212-280).
-- If the source never had a key, the archived env has none, and the target's first start mints one. That is correct, because no source process ever held a key.
+- If the source never had a key, the target's first start mints one. That is correct: no source process ever held a key.
- The move's env report classes `CARLOS_INSTANCE_KEY` as `equal`. A target that would start with a different key, or none, is a `differs` or `missing` difference, never a note.
-- A move's probe launches with the env its readiness pin checks against the source's report (internal/host/movesprobeowners.go:106-120). So the probe path never mints, and a key minted there would fail the pin. This plan adds no mint to it.
-- Relocation's install (L16's shared routine, stage B Task 8c) writes the pinned manifest's `EnvItem` with `writeFileSynced`, or removes the env when the item is absent. That is a restore of the host's authoritative file onto a box that holds no installation of it, so it is the one write allowed to replace `<host>.env`. The next launch's mint finds the key present (O3).
+- A move's probe launches with the env its readiness pin checks against the source's report (internal/host/movesprobeowners.go:106-120). So the probe path never mints.
+- A move does not carry the adoption record (the archive holds the data directory and `<host>.env` only). The app re-records it at its first start on the target, because the helper calls the verb once per process. Until then a relocation of that host would refuse (O2'), which is safe.
+- Relocation's install (stage B Task 8c) writes the pinned manifest's `EnvItem` and adoption item verbatim (O3'). The next launch's mint finds the key present.
*Cost if wrong:* a key lost in transit is the L21 failure itself. The env report makes it visible on moves.
-**11. No switch.** Nothing reads `CARLOS_INSTANCE_KEY` until an app adopts the helper, so writing it is harmless:
-- an app that never calls `carlos.InstanceKey` never looks at it;
-- an app that prints its whole environment already prints `CARLOS_ADMIN_TOKEN`;
-- the claim's launch identity, a digest over the env, changes once per host, as any `carlos env set` does;
-- the backfill appends one line to files the activator already owns.
-
-*Cost if wrong:* the mint is on every launch path, so a bug in it fails wakes across the fleet. The tests, and an error shape identical to the token mint's, are the mitigation. Rollback is reverting the binary. Keys already written stay, and are harmless.
+**11. No switch.** Nothing reads `CARLOS_INSTANCE_KEY` until an app adopts the helper, so writing it is harmless. The new verb answers only a caller that holds the host's admin token. The claim's launch identity, a digest over the env, changes once per host. *Cost if wrong:* the mint is on every launch path, so a bug in it fails wakes across the fleet. The tests, and an error shape identical to the token mint's, are the mitigation. Rollback is reverting the binary. Keys and records already written stay, and are harmless.
-**12. Scope: exec-backed activator instances.** These do not get a platform key:
+**12. Scope: exec-backed activator instances (amended for replicas).** These get no platform key:
- unit-backed tenants: their env is `/etc/carlos/apps/<host>.env`, root-only and unreadable to the agent (backend_unit.go:174-178);
- TCP-addr routes: an app supervisor, not the agent, starts them;
-- sidecars and workers: they have their own rows and env files, and `StartWorkerStaged` (backend_exec.go:2104) mints no token either.
+- sidecars and workers: they have their own rows and env files (open question 4, provisional default: a follow-up);
+- replica rows: ruling 17.
+
+The agent sets `CARLOS_CONTROL_SOCKET` only for instance children (`agentOwnedVars`), so `carlos.Running()` is false for unit tenants, TCP-addr routes, sidecars and workers, and the helper keeps today's behaviour. *Cost if wrong:* an excluded process that runs with `CARLOS_CONTROL_SOCKET` set and adopts the helper refuses to start. That is loud, not silent.
+
+**13. One rule for the effective key (new; finding 4).** `instancekey.Effective(config, own)` is the key a launched app receives: the own file's when non-empty, else the configuration's when non-empty, else none. An empty own value never shadows a non-empty configured one. `mergeEnv` applies it on top of its merge, so the child's env follows the rule; the mint decides by it; the adoption verb checks by it; and the pinner reads the env item with the same parser (`instancekey.ParseEnv`, which `parseEnvBytes` now calls). *Cost if wrong:* a deliberately empty `CARLOS_INSTANCE_KEY=` in `<host>.env` can no longer hide a configured key. Nobody can rely on that today: an empty key makes an adopting app refuse to start.
+
+**14. The adoption record (new; finding 1, source b).**
+- *The verb:* `PUT /instance-key/adoption` on the instance's control socket, bearer `CARLOS_ADMIN_TOKEN`, body `{"sha256":"<hex>"}`, the SHA-256 of the key the app resolved. The host is the listener's, never the request's.
+- *What it checks:* the host's effective key (ruling 13) must have that digest. If it is the own file's key, the agent writes `<host>.instance-key-adoption.json` beside `<host>.env`: `{"v":1,"var":"CARLOS_INSTANCE_KEY","sha256":"<hex>"}`, atomically, 0600, file and directory fsynced, and only then answers 204. Identical bytes already there are made durable and not rewritten. If it is the configuration's key, it answers 204 and records nothing: source (c) evidences that key. A digest that matches no effective key, or a replica row, is 409 and records nothing.
+- *Who calls it:* rastrillo's `InstanceKey`, once per process, when it returns `$CARLOS_INSTANCE_KEY` (step 2). Failure is non-fatal to the app (the ask below).
+- *Where it travels:* the layout pass carries it as an extra. O1' publishes it as the platform item `instance-key-adoption.json`; O5 stages it in stage B's release flush; O3' restores it. A move does not carry it (ruling 10); the next process start re-records it.
+- *Claim boxes:* the verb is mounted wherever the agent binds a control socket. Claim launches bind none today (O6).
+
+*Cost if wrong:* an app that adopted on a release of rastrillo without the call has no record, and its relocation refuses (stuck) until it runs a release with the call. That is safe and visible.
+
+**15. Relocation's key evidence (new; findings 1 and 5).** For a nonempty database written by an admitted writer, the pinner (O2') accepts the pinned manifest only with positive evidence that it holds the key the app uses:
+- (a) a live `instance.key` entry, as today;
+- (b) the env item's `CARLOS_INSTANCE_KEY` is non-empty and the adoption item is a record for exactly that key;
+- (c) the env item has no non-empty key (or there is no env item), and the host's configuration renders a non-empty one (`appconfig.ConfiguredInstanceKey`, through `StorePinner.ConfigKey`).
+
+A non-empty env key overrides a configured key at launch, so (c) is never consulted for it. A named item that cannot be read is an error, never "no evidence". Without any of the three, relocation refuses, as today. That is not a regression. *Cost if wrong:* (c) carries a residual that (b) closes: an app that keeps its own `instance.key`, never published, and also has a `CARLOS_INSTANCE_KEY` in its configuration passes (c) though it seals with its own key. Requiring an adoption record for configured keys too would close it; the binding ruling accepts the configuration itself as the evidence (concern 3 in the report).
+
+**16. Claim retries reuse their inputs (new; question 5's premise corrected).** On a claim box every serving start is a claim wake. The launch source, which mints, runs in `Select`. A session's retries reuse the captured inputs and call `Materialize`, and so `Establish`, before every attempt. The legacy spawn path does not serve a claim host, and production claim swaps refuse. No claim attempt can run with a key minted after its files manifest was established. Any future path that mints before an asynchronous publish needs a synchronous publication gate. *Cost if wrong:* `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` fails on the change that breaks it.
-The agent sets `CARLOS_CONTROL_SOCKET` only for instance children (`agentOwnedVars`), so `carlos.Running()` is false in all three, and the helper keeps today's behaviour. Replica rows (#162, `Route.Replica`) are their own hosts and get their own key: see open question 1. *Cost if wrong:* an excluded process that runs with `CARLOS_CONTROL_SOCKET` set and adopts the helper would refuse to start. That is loud, not silent.
+**17. Replicas are excluded from platform-key adoption (new; open question 1, provisional default, Paul not yet confirmed).** A replica (#162) mirrors its home's data, which is sealed under the home's key. A replica's own platform key would let an adopting app seal under a different key, silently. So no path mints on a replica row (`InstanceKeyReplica`), the backfill skips replicas, and the adoption verb answers 409 on one. An app with replica placements passes its own key as `own`, or refuses to start on the replica (`ErrNoInstanceKey`), loudly. *Cost if wrong:* a replica of an adopting app with no own key does not start until the follow-up gives replicas their home's key.
+
+**18. The claim early return syncs without writing (new; finding 2).** When the token and the key are both present, the claim launch source writes nothing and needs no authority, as before. It still opens the env file read-only, checks the mode, fsyncs the file and fsyncs its directory before it answers. fsync changes no byte, mode or name, so it is not a host write and runs outside the claim's guard. A file whose mode is not 0600 needs a write, so it takes the guarded path, whose ensure sets it. *Cost if wrong:* if fsync were judged a host write, this step would move under `authorize`, which re-reads the claim on every selection.
## Obligations on other plans
-- **O1 (the C8 files-manifest provider, `ClaimFilesManifest`):** `Establish` leaves a committed manifest whose env item has exactly the bytes of the host's `<host>.env` as they are when `Establish` runs. A copied manifest whose env item differs gets a new revision with the file's bytes. It never writes the file on disk. Any error (store unreachable, `If-Match` lost) is returned, and the wake fails before the app executes. Its test, in the provider's plan: `TestEstablishPublishesTheEnvFileWithItsInstanceKey`. A dir store, a predecessor manifest whose `host.env` item lacks the key, and a local `<host>.env` that carries it; after `Establish`, `ReadPlatform(EnvItem)` equals the local bytes. A store that refuses the PUT returns an error and the app never starts.
-- **O2 (stage B, ruling 25's `needKey` in `hostrelocate/pin.go:250-256` on `region-ha-1b-next-build`):** evidence of a key is a live `instance.key` entry **or** a platform `EnvItem` whose parsed env carries a non-empty `CARLOS_INSTANCE_KEY`. Without this, every app that adopts the platform key and has a nonempty database goes `stuck` on every relocation, because it never writes `instance.key`. Test: `TestStorePinnerAcceptsAPlatformKeyInTheEnvItem`. A nonempty database, a manifest with no `instance.key` and an `EnvItem` of `CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=<44>\n` pins; the same `EnvItem` without the key line refuses with `ErrFilesUnevidenced`.
-- **O3 (stage B Task 8c, relocation install):** add `TestRelocationInstallKeepsThePublishedInstanceKey`. P's manifest `EnvItem` carries a key. After `Install`, `<host>.env` equals the item's bytes, and `activator.EnsureInstanceKey(envPath)` returns `InstanceKeyPresent` and leaves the bytes unchanged.
-- **O4 (stage B Task 10 checklist item 9):** satisfied once this plan is rolled to the region's boxes and O1 and O2 are in.
+- **O1' (the C8 files-manifest provider, `ClaimFilesManifest`; replaces O1):** `Establish` leaves a committed manifest whose platform items mirror the disk when it runs: `hostfiles.EnvItem` with exactly the bytes of `<host>.env`, and `instancekey.AdoptionItem` with exactly the bytes of `<host>.instance-key-adoption.json`, or no such item when the file is absent (`StagePlatformDelete`). It never writes either file on disk. Any error (store unreachable, `If-Match` lost) is returned, and the wake fails before the app executes. Tests, in the provider's plan, with the provider itself, not a fake: `TestEstablishPublishesTheEnvFileWithItsInstanceKey` (a dir store, a predecessor manifest whose env item lacks the key, a local `<host>.env` that carries it and an adoption record; after `Establish`, `ReadPlatform` of both items equals the local bytes, and the manifest is the committed revision); `TestEstablishFailsTheWakeWhenThePublishFails` (a store that refuses the PUT: `Establish` errors, and through `claimServingOwners.Materialize` and `startAdmitted` no start is reserved).
+- **O2' (stage B, ruling 25's `needKey` in `hostrelocate/pin.go` on `region-ha-1b-next-build`; replaces O2 and ruling 54):** ruling 15. The diff, the test and the wiring are below ("O2' for stage B"). Ruling 54's text becomes: "The instance key is evidenced by (a) `instance.key`, (b) the env item's key with an adoption record for it, or (c) no env key and a configured one (instance-key plan rev 2, ruling 15). The env item is parsed with `instancekey.ParseEnv`."
+- **O3' (stage B Task 8c, relocation install; replaces O3):** the install writes the pinned manifest's `EnvItem` and `AdoptionItem` byte for byte (`hostfiles.ReadPlatform`, then `writeFileSynced` to `<host>.env` and `<host>.instance-key-adoption.json` in the instance directory), and removes either file when its item is absent. It never mints. `TestRelocationInstallKeepsThePublishedEnvByteForByte` gains: the adoption item's bytes equal the record after `Install`, and `activator.EnsureInstanceKeyBeneath(dir, host+".env")` then returns `InstanceKeyPresent` and leaves the env bytes unchanged.
+- **O4 (stage B Task 10 checklist item 9):** becomes: "L21. The instance-key plan is rolled to the region's boxes, O1', O2', O3' and O5 are in, and every app that adopted the platform key runs a rastrillo release with the adoption call (#66 ask). A host whose manifest evidences its key by none of (a), (b), (c) stays `stuck`."
+- **O5 (stage B's release flush, `internal/host/relocationflush.go` on that branch):** stages `AdoptionItem` from disk beside `EnvItem` (or `StagePlatformDelete` when the file is absent), in the same commit. Test: the flush of a host with a record publishes it; without one, publishes none.
+- **O6 (claim Plan 9 Group D, the isolated-root launcher):** when claim launches bind a control socket, mount `mountInstanceKeyAdoption` on it, with the record's write run under the claim's guard (a hostwrite selection for the host, as the token's mint is). Until then, an app that first adopts the platform key on a claim box cannot record adoption, and its relocation refuses. A record written while the box was legacy stays on disk and is published by `Establish` (O1').
+
+### O2' for stage B
+
+Apply to `internal/hostrelocate/pin.go` on `region-ha-1b-next-build` (after merging main, which brings `internal/instancekey`), in place of Task 8b step 0's O2 diff. Verified in a scratch copy of `origin/region-ha-1b-next-build` (1651ff7d) with `internal/instancekey` added: `go vet` clean, and `go test ./internal/hostrelocate/ -count=1` passes, every existing pinner test included.
+
+```diff
+--- a/internal/hostrelocate/pin.go
++++ b/internal/hostrelocate/pin.go
+@@ -15,6 +15,7 @@
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfiles"
++ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/release"
+ )
+
+@@ -88,6 +89,12 @@
+ // Stage B wires it to the controller's registry. Nil is ErrUnsupported;
+ // an empty version refuses the pin, never a guess.
+ Release func(ctx context.Context, host string) (string, error)
++ // ConfigKey reports whether the host's configuration in the bucket
++ // renders a non-empty CARLOS_INSTANCE_KEY: appconfig.
++ // ConfiguredInstanceKey over the route a relocation target mints for
++ // the host (relocationRecord), with the box's secrets key. It is the
++ // key evidence (c) of O2'. Nil leaves only evidence (a) and (b).
++ ConfigKey func(ctx context.Context, host string) (bool, error)
+ }
+
+ func (p StorePinner) Pin(ctx context.Context, obs hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) {
+@@ -142,7 +149,7 @@
+ // that writer's instance.key: the pinned manifest must carry it (rev
+ // 18, round 17 finding 1).
+ needKey := c.Holder.Session != 0 && obj.DB != nil && obj.DB.MaxTXID > 0
+- files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0, needKey)
++ files, retained, err := p.filesPin(ctx, c.Host, c.Holder, released && rel.ManifestSeq == 0, needKey)
+ if err != nil {
+ return hostclaim.RecoveryPin{}, err
+ }
+@@ -221,7 +228,7 @@
+ // stays restorable at g10. Every referenced object must lie under this
+ // holder's location (its incarnation) in this region's data bucket, at
+ // files/<generation>/objects/; any other reference refuses the pin.
+-func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, []byte, error) {
++func (p StorePinner) filesPin(ctx context.Context, host string, h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, []byte, error) {
+ chain := []string{h.Generation}
+ for i := len(h.Admissions) - 1; i >= 0; i-- {
+ chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor)
+@@ -247,16 +254,20 @@
+ if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion {
+ return FilesPin{}, nil, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err)
+ }
+- if needKey && !liveEntry(m, InstanceKeyPath) {
+- // The manifest was copied before the app started and the
+- // key it minted afterwards never published (§5.6's
+- // continuous-sync window, L21): installing this set would mint
+- // a new key over data sealed with the old one.
+- return FilesPin{}, nil, fmt.Errorf("%w: %s's pinned manifest has no %s for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath)
+- }
+ if origins, err = referencedOrigins(h.Location, origins, m); err != nil {
+ return FilesPin{}, nil, err
+ }
++ if needKey && !liveEntry(m, InstanceKeyPath) {
++ // (a) failed: no instance.key. The key is accepted only with
++ // evidence the app seals with it (O2', instance-key plan rev
++ // 2): (b) or (c). Without either, the manifest may have been
++ // copied before the app minted a key of its own (§5.6's
++ // continuous-sync window), and installing this set would mint
++ // a new key over data sealed with the old one.
++ if err := p.keyEvidenced(ctx, host, prefix, m, origins); err != nil {
++ return FilesPin{}, nil, err
++ }
++ }
+ sum := sha256.Sum256(raw)
+ return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:]), Origins: origins}, raw, nil
+ }
+@@ -395,3 +406,54 @@
+ }
+ return pinned, nil
+ }
++
++// keyEvidenced is O2' (instance-key plan rev 2): a pinned manifest without
++// instance.key still evidences the key a nonempty database is sealed with
++// when either
++//
++// (b) its env item's CARLOS_INSTANCE_KEY is non-empty and its adoption
++// item is a record for exactly that key: the app told the agent it
++// resolved $CARLOS_INSTANCE_KEY and was handed this key;
++// (c) its env item carries no non-empty key (or there is none) and the
++// host's configuration renders one, which the target renders the
++// same way.
++//
++// The env item is parsed by instancekey.ParseEnv, the activator's own
++// rule, so the key read here is the key the launch hands the app. A
++// non-empty env key overrides a configured one at launch, so (c) is never
++// consulted for it. A named item that cannot be read is an error, never
++// "no evidence".
++func (p StorePinner) keyEvidenced(ctx context.Context, host, prefix string, m hostfiles.Manifest, origins []string) error {
++ read := func(name string) ([]byte, bool, error) {
++ if e, ok := m.Platform[name]; !ok || e.Deleted {
++ return nil, false, nil
++ }
++ raw, err := hostfiles.ReadPlatform(ctx, p.Data, prefix, m, name, origins)
++ return raw, err == nil, err
++ }
++ env, _, err := read(hostfiles.EnvItem)
++ if err != nil {
++ return err
++ }
++ own, _ := instancekey.Effective(nil, instancekey.ParseEnv(env))
++ if own != "" {
++ rec, ok, err := read(instancekey.AdoptionItem)
++ if err != nil {
++ return err
++ }
++ if ok && instancekey.Adopted(rec, own) {
++ return nil
++ }
++ return fmt.Errorf("%w: %s's pinned manifest has no %s, and no adoption record for the %s in its env, for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath, instancekey.Var)
++ }
++ if p.ConfigKey != nil {
++ ok, err := p.ConfigKey(ctx, host)
++ if err != nil {
++ return err
++ }
++ if ok {
++ return nil
++ }
++ }
++ return fmt.Errorf("%w: %s's pinned manifest has no %s, no adopted %s and no configured one, for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath, instancekey.Var)
++}
+```
+
+Replace `internal/hostrelocate/pin_envkey_test.go` (stage B's `TestStorePinnerAcceptsAnInstanceKeyInTheEnv`) with:
+
+```go
+package hostrelocate
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// O2' (instance-key plan rev 2, replacing stage B's O2): a nonempty
+// database's pinned manifest evidences its key by (a) instance.key, (b) an
+// env key with an adoption record for that very key, or (c) no env key and
+// a key in the host's configuration. A platform key that is merely present
+// is not evidence: an app with a key of its own keeps it, and its data is
+// sealed under that key (Astra round 1 finding 1).
+func TestStorePinnerKeyEvidence(t *testing.T) {
+ ctx := context.Background()
+ at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)
+ const key = "q+/Zk3vJ0aW1Yx9sR7tUe2oPl4mN8bC6dF5gH0iJ1kA="
+ envWithKey := "CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=" + key + "\n"
+ configured := func(context.Context, string) (bool, error) { return true, nil }
+ unconfigured := func(context.Context, string) (bool, error) { return false, nil }
+ for _, tc := range []struct {
+ name string
+ appKey bool // (a) instance.key in the manifest
+ env, adoption string // "" is no item
+ unreadable string // a named item whose object is missing
+ configKey func(context.Context, string) (bool, error)
+ want error
+ }{
+ {name: "(a) the app's own instance.key", appKey: true, env: envWithKey, want: nil},
+ {name: "(b) the env key, adopted", env: envWithKey, adoption: string(instancekey.AdoptionRecord(key)), want: nil},
+ {name: "a backed-up platform key, the app's own key missing", env: envWithKey, want: ErrFilesUnevidenced},
+ {name: "an adoption record for another key", env: envWithKey, adoption: string(instancekey.AdoptionRecord("another-key")), want: ErrFilesUnevidenced},
+ {name: "the env key overrides a configured one, unadopted", env: envWithKey, configKey: configured, want: ErrFilesUnevidenced},
+ {name: "an exported key line is not what the app receives", env: "export CARLOS_INSTANCE_KEY=" + key + "\n", adoption: string(instancekey.AdoptionRecord(key)), configKey: unconfigured, want: ErrFilesUnevidenced},
+ {name: "(c) no env key, a configured key", env: "CARLOS_ADMIN_TOKEN=t\n", configKey: configured, want: nil},
+ {name: "(c) an empty override, a configured key", env: "CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=\n", configKey: configured, want: nil},
+ {name: "(c) no env item, a configured key", configKey: configured, want: nil},
+ {name: "no key anywhere", env: "CARLOS_ADMIN_TOKEN=t\n", configKey: unconfigured, want: ErrFilesUnevidenced},
+ {name: "no configuration reader", env: "CARLOS_ADMIN_TOKEN=t\n", want: ErrFilesUnevidenced},
+ {name: "the configuration cannot be read", env: "CARLOS_ADMIN_TOKEN=t\n", configKey: func(context.Context, string) (bool, error) { return false, errBucket }, want: errBucket},
+ {name: "a named adoption item that cannot be read", env: envWithKey, unreadable: instancekey.AdoptionItem, want: errUnreadable},
+ {name: "a named env item that cannot be read", unreadable: hostfiles.EnvItem, configKey: configured, want: errUnreadable},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
+ m := hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: 4, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}}
+ if tc.appKey {
+ m = keyed(m)
+ }
+ item := func(name, body string, put bool) {
+ sum := sha256.Sum256([]byte(body))
+ sha := hex.EncodeToString(sum[:])
+ obj := pinLocation + "/files/g3/objects/" + fmt.Sprintf("%016d-%s", 4, sha)
+ if put {
+ if _, err := data.Put(ctx, obj, []byte(body), ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ m.Platform[name] = hostfiles.Entry{Object: obj, SHA256: sha, Size: int64(len(body)), Mode: 0o600, Seq: 4}
+ }
+ if tc.env != "" {
+ item(hostfiles.EnvItem, tc.env, true)
+ }
+ if tc.adoption != "" {
+ item(instancekey.AdoptionItem, tc.adoption, true)
+ }
+ if tc.unreadable != "" {
+ item(tc.unreadable, "never stored\n", false)
+ }
+ putManifest(t, data, "g3", m)
+ p := StorePinner{Claims: claims, Data: data, Release: routeRelease, Bucket: "data", Region: "ie", SelectDB: selectAt(41), ConfigKey: tc.configKey}
+ _, err := p.Pin(ctx, pinClaim(), at)
+ switch {
+ case tc.want == errUnreadable:
+ if err == nil || errors.Is(err, ErrFilesUnevidenced) {
+ t.Fatalf("got %v, want a read error, never a refusal for no evidence", err)
+ }
+ case !errors.Is(err, tc.want):
+ t.Fatalf("got %v, want %v", err, tc.want)
+ }
+ })
+ }
+}
+
+var (
+ errBucket = errors.New("bucket unreachable")
+ errUnreadable = errors.New("sentinel: any read error")
+)
+```
+
+Wiring, in `NewRelocationRuntime`'s worker literal (stage B Task 10), beside `Release: channelRelease(store, pub)`. This sketch is not compiled here, because `relocationRecord` exists only in stage B's plan text:
+
+```go
+// relocationConfigKey is O2' evidence (c): the host's configuration,
+// rendered for the route a relocation target mints (relocationRecord), so
+// the pinner asks exactly what the target's held render will deliver.
+func relocationConfigKey(dataDir string, store release.Store, priv *ecdh.PrivateKey) func(context.Context, string) (bool, error) {
+ return func(ctx context.Context, host string) (bool, error) {
+ o, err := hostclaim.Read(ctx, store, host)
+ if err != nil {
+ return false, err
+ }
+ rec, err := relocationRecord(ctx, dataDir, store, o)
+ if err != nil {
+ return false, err
+ }
+ return appconfig.ConfiguredInstanceKey(ctx, store, priv, rec.Route)
+ }
+}
+```
+
+and `ConfigKey: relocationConfigKey(cfg.DataDir, store, cfg.SecretsKey)` in `hostrelocate.StorePinner{...}`.
+
+## Ask for rastrillo #66
+
+*For Paul to post on rastrillo/rastrillo discussion #66.*
+
+**The ask.** When `carlos.InstanceKey(own)` returns the platform's key, that is `$CARLOS_INSTANCE_KEY` at step 2, have it tell the agent once, so the platform can record that this app seals with that key. Failure is non-fatal to the app.
+
+**Why.** Region HA relocates an instance's data to another box. It may hand the app the platform key there only if that key is the one the data is sealed with. A platform key being present proves nothing: an app with its own key keeps it (step 1). So the platform needs positive evidence, and only the app's helper knows which step it took.
+
+**The call.**
+- `PUT http://carlos/instance-key/adoption` over `$CARLOS_CONTROL_SOCKET`, the same transport and placeholder host `ScheduleAt` uses.
+- Header `Authorization: Bearer $CARLOS_ADMIN_TOKEN`; `Content-Type: application/json`.
+- Body `{"sha256":"<hex>"}`: the lowercase hex SHA-256 of the key string exactly as resolved. Never the key.
+- When: inside `InstanceKey`, only on the step 2 return, only when `Running()` and the token is set, at most once per process (a `sync.Once`).
+- How long: synchronous, before `InstanceKey` returns, so the record is on disk before the app seals anything. Bounded by a short timeout of its own (2 seconds suggested), since `InstanceKey` takes no context.
+- Answers: 204 recorded (or the key is the app's configured one, which needs no record). 404 or 405: an agent from before this verb; ignore it silently. 409: the agent's effective key is not the one the app holds, or the host is a replica; worth one log line. 401, 5xx, a timeout, a dial error: one log line at most. Never retry in a loop, never log the key.
+- Whatever the answer, `InstanceKey` returns the key and a nil error.
+
+**Not called** at step 1 (the app's own key), step 3 or step 4, or with no `$CARLOS_ADMIN_TOKEN`.
+
+**What changes for apps.** Nothing visible. An app on a rastrillo release without the call still runs; its data simply does not relocate (it is held as `stuck`) until it runs a release with the call. The reference page could say that `InstanceKey` may make one local call to the platform.
## Review Focus
-1. **A hand-edited `<host>.env` with two key lines.** The launch env takes the last line, and the mint writes nothing. Task 1, `TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone`.
-2. **An empty `CARLOS_INSTANCE_KEY=` line.** It is treated as absent: a key is appended, and readers take the appended one. Task 1, `TestEnsureInstanceKeyTreatsAnEmptyValueAsAbsent`.
-3. **A visitor's wake, a seed and the startup backfill racing on one fresh host.** Exactly one key is written. Task 1, `TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers`.
-4. **A FIFO planted at a nested `<host>.env`.** The startup backfill must fail fast, never block agent start. Task 2, `TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking`.
-5. **An empty `CARLOS_INSTANCE_KEY=` in the app's configuration.** It supplies nothing, so the platform mints one. Task 1, `TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty`.
+1. **A failed sync, then a retry.** The retry keeps the bytes, redoes fchmod and both syncs, and only then succeeds; no child starts in between. Task 2 `TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess`, Task 3 `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced`, Task 4 `TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers`.
+2. **An empty `CARLOS_INSTANCE_KEY=` in `<host>.env` with a configured key.** The child gets the configured key and nothing is minted. Task 3 `TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists`.
+3. **The adoption verb's checks.** Bearer, digest, own file only, replica refused, durable before 204. Task 5.
+4. **O2''s evidence table.** In particular "a backed-up platform key, the app's own key missing" refuses, and the exported line is not a key.
+5. **A FIFO planted at a nested `<host>.env`.** The startup backfill fails fast. Task 3 `TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking`.
+6. **A visitor's wake, a seed and the startup backfill racing on one fresh host.** Exactly one key is written. Task 2 `TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers`.
## File structure
-- Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `(*ExecBackend).BackfillInstanceKey`, `InstanceKeyVar`, `InstanceKeyOutcome`). One responsibility: the key line in `<host>.env`.
-- Create `internal/activator/instancekey_test.go` and `internal/activator/instancekey_linux_test.go`.
-- Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369), `internal/activator/prepare.go` (:60), `internal/activator/claim_snapshot.go` (:40-67): call the mint.
-- Modify `internal/host/claimowners.go` (:125-138): the obligation on `ClaimFilesManifest`. Create `internal/host/claim_instancekey_test.go`.
-- Create `internal/host/instancekeys.go` and `internal/host/instancekeys_test.go`: the backfill. Modify `internal/host/agent.go` (:476-484): call it.
-- Create `internal/appconfig/syncexec_instancekey_test.go`, `internal/moves/envreport_instancekey_test.go`, `internal/moves/archive/instancekey_test.go`: never-overwrite across re-render and moves.
+- Create `internal/instancekey/instancekey.go` and `instancekey_test.go`: the parser, `Effective`, `Digest`, the adoption record. Standard library only.
+- Modify `internal/activator/admintoken.go` (`parseEnvBytes` calls `instancekey.ParseEnv`) and `internal/activator/backend_exec.go` (`mergeEnv`, `LaunchEnvironment`'s shadow list).
+- Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `ensureInstanceKeyFor`, `BackfillInstanceKey`, `settleClaimInstanceEnv`, `InstanceKeyVar`, `InstanceKeyOutcome`).
+- Create `internal/activator/instancekey_test.go`, `instancekey_linux_test.go`, `claim_instancekey_test.go`, `claimretry_instancekey_linux_test.go`.
+- Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369 and its mount :1406), `internal/activator/prepare.go` (:60 and its mount :143), `internal/activator/claim_snapshot.go` (:40-67).
+- Create `internal/activator/keyadoption.go`, `keyadoption_test.go`, `keyadoption_linux_test.go`: the adoption verb.
+- Modify `internal/host/claimowners.go` (:125-138): O1' on `ClaimFilesManifest`. Create `internal/host/claim_instancekey_test.go`.
+- Create `internal/host/instancekeys.go` and `instancekeys_test.go`: the backfill. Modify `internal/host/agent.go` (:476-484).
+- Create `internal/appconfig/instancekey.go` (`ConfiguredInstanceKey`) and `syncexec_instancekey_test.go`, `internal/moves/envreport_instancekey_test.go`, `internal/moves/archive/instancekey_test.go`.
-**Sandbox note for every command below:** if the toolchain reports "read-only file system" for the Go cache, prefix commands with `GOCACHE=$TMPDIR/gocache` (re-export per command), and with `GOFLAGS=-buildvcs=false` in a worktree. On a shared machine run them under `nice -n 19`.
+**Sandbox note for every command below:** if the toolchain reports "read-only file system" for the Go cache, prefix commands with `GOCACHE=$TMPDIR/gocache` (re-export per command; never delete that shared directory), and with `GOFLAGS=-buildvcs=false` in a worktree. On a shared machine run them under `nice -n 19`. The sandbox refuses unix sockets ("socket: operation not permitted"): tests that start a child, prepare a nested role or dial a control socket must run with the sandbox off.
---
-### Task 1: The mint
+### Task 1: The shared rule
+
+**Files:**
+- Create: `internal/instancekey/instancekey.go`, `internal/instancekey/instancekey_test.go`
+- Modify: `internal/activator/admintoken.go:245-257` (`parseEnvBytes`), `internal/activator/backend_exec.go:1879-1905` (`LaunchEnvironment`), `:1980-1990` (`mergeEnv`)
+
+**Interfaces:**
+- Produces: `instancekey.Var`, `AdoptionSuffix`, `AdoptionItem`, `ParseEnv`, `Layer` (`LayerNone`, `LayerOwn`, `LayerConfig`), `Effective`, `Digest`, `Adoption`, `AdoptionRecord`, `Adopted`, `ValidDigest`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `internal/instancekey/instancekey_test.go`:
+
+```go
+package instancekey
+
+import (
+ "strings"
+ "testing"
+)
+
+// Finding 4: an empty own value never shadows a configured key; a
+// non-empty own value always wins.
+func TestEffectiveNeverHandsOutAnEmptyKeyWhileOneExists(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ config, own string
+ want string
+ layer Layer
+ }{
+ {"own wins", Var + "=cfg\n", Var + "=own\n", "own", LayerOwn},
+ {"empty own, configured", Var + "=cfg\n", Var + "=\n", "cfg", LayerConfig},
+ {"no own, configured", Var + "=cfg\n", "A=1\n", "cfg", LayerConfig},
+ {"own only", "", Var + "=own\n", "own", LayerOwn},
+ {"both empty", Var + "=\n", Var + "=\n", "", LayerNone},
+ {"exported is not the key", "", "export " + Var + "=own\n", "", LayerNone},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ got, layer := Effective(ParseEnv([]byte(tc.config)), ParseEnv([]byte(tc.own)))
+ if got != tc.want || layer != tc.layer {
+ t.Fatalf("Effective = %q, %v; want %q, %v", got, layer, tc.want, tc.layer)
+ }
+ })
+ }
+}
+
+func TestAdoptionRecordVouchesOnlyForItsOwnKey(t *testing.T) {
+ key := strings.Repeat("A", 43) + "="
+ rec := AdoptionRecord(key)
+ if !Adopted(rec, key) {
+ t.Fatal("the record does not vouch for its own key")
+ }
+ if strings.Contains(string(rec), key) {
+ t.Fatal("the record holds the key itself")
+ }
+ for _, tc := range []struct {
+ name string
+ record []byte
+ key string
+ }{
+ {"another key", rec, strings.Repeat("B", 43) + "="},
+ {"an empty key", AdoptionRecord(""), ""},
+ {"a wrong version", []byte(`{"v":2,"var":"CARLOS_INSTANCE_KEY","sha256":"` + Digest(key) + `"}`), key},
+ {"another variable", []byte(`{"v":1,"var":"OTHER","sha256":"` + Digest(key) + `"}`), key},
+ {"an unknown field", []byte(`{"v":1,"var":"CARLOS_INSTANCE_KEY","sha256":"` + Digest(key) + `","extra":1}`), key},
+ {"trailing data", append(append([]byte(nil), rec...), []byte(`{}`)...), key},
+ {"garbage", []byte("not json"), key},
+ } {
+ if Adopted(tc.record, tc.key) {
+ t.Errorf("%s: the record vouched", tc.name)
+ }
+ }
+}
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+Run: `nice -n 19 go test ./internal/instancekey/ -count=1`
+Expected: build failure, no non-test Go files.
+
+- [ ] **Step 3: Write the package**
+
+Create `internal/instancekey/instancekey.go`:
+
+```go
+// Package instancekey is the one rule for the platform's per-instance key
+// (region HA L21, rastrillo #66): how an env file is parsed, which file's
+// CARLOS_INSTANCE_KEY a launched app receives, and the adoption record that
+// is the evidence an app seals with that key. The activator mints and
+// delivers the key by these rules, the control socket records adoption by
+// them, and the relocation pinner checks its evidence by them, so the three
+// can never disagree about what the app was given.
+//
+// It imports only the standard library, so the pinner can use it without
+// importing the activator.
+package instancekey
+
+import (
+ "bytes"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "strings"
+)
+
+// Var is the variable the key is delivered as.
+const Var = "CARLOS_INSTANCE_KEY"
+
+// AdoptionSuffix names the adoption record beside an instance's own env
+// file: <host>.env's sibling <host>.instance-key-adoption.json. The flat
+// layout's naming rules make it the host's (layout.belongsTo), so the
+// layout pass carries it into the instance directory with the rest.
+const AdoptionSuffix = ".instance-key-adoption.json"
+
+// AdoptionItem is the adoption record's name among a files manifest's
+// platform items, beside hostfiles' EnvItem ("host.env").
+const AdoptionItem = "instance-key-adoption.json"
+
+// ParseEnv parses an EnvironmentFile: KEY=VALUE lines, blank lines and #
+// comments skipped, the key and the value trimmed of surrounding
+// whitespace, the last of two lines for one key winning. It is exactly
+// what the activator hands a child, so a reader that wants to know what an
+// app received must parse with it. "export KEY=..." is the key "export
+// KEY", and a quoted value keeps its quotes.
+func ParseEnv(raw []byte) map[string]string {
+ m := map[string]string{}
+ for _, line := range strings.Split(string(raw), "\n") {
+ line = strings.TrimSpace(line)
+ if line == "" || strings.HasPrefix(line, "#") {
+ continue
+ }
+ if k, v, ok := strings.Cut(line, "="); ok {
+ m[strings.TrimSpace(k)] = strings.TrimSpace(v)
+ }
+ }
+ return m
+}
+
+// Layer is which env file supplied the effective key.
+type Layer int
+
+const (
+ // LayerNone: neither file supplies a non-empty key.
+ LayerNone Layer = iota
+ // LayerOwn: the instance's own <host>.env, where the platform mints.
+ LayerOwn
+ // LayerConfig: the rendered <host>.config.env, the app's configuration.
+ LayerConfig
+)
+
+// Effective is the key a launched app receives from its two env files, and
+// which file supplied it. The instance's own file wins, as in every other
+// variable (the activator's mergeEnv), but an empty value supplies no key:
+// an empty CARLOS_INSTANCE_KEY= in the own file never shadows a non-empty
+// configured key, so an app is never handed an empty key while a non-empty
+// one exists.
+func Effective(config, own map[string]string) (string, Layer) {
+ if k := own[Var]; k != "" {
+ return k, LayerOwn
+ }
+ if k := config[Var]; k != "" {
+ return k, LayerConfig
+ }
+ return "", LayerNone
+}
+
+// Digest is the lowercase hex SHA-256 of key: what an app sends to say
+// which key it adopted, and what the adoption record holds. The record is
+// published beside the env file that holds the key itself, so the digest
+// reveals nothing the same reader lacks.
+func Digest(key string) string {
+ sum := sha256.Sum256([]byte(key))
+ return hex.EncodeToString(sum[:])
+}
+
+// Adoption is the record the agent writes when an app reports that it
+// resolved its key from $CARLOS_INSTANCE_KEY (rastrillo's step 2), and the
+// key it resolved is the one in the instance's own <host>.env.
+type Adoption struct {
+ V int `json:"v"`
+ Var string `json:"var"`
+ SHA256 string `json:"sha256"`
+}
+
+// AdoptionRecord is the canonical bytes of the record for key.
+func AdoptionRecord(key string) []byte {
+ raw, _ := json.Marshal(Adoption{V: 1, Var: Var, SHA256: Digest(key)})
+ return append(raw, '\n')
+}
+
+// Adopted reports whether record is an adoption record for exactly key. A
+// record for another key, a malformed record or an empty key is false: a
+// record never vouches for a key it does not name.
+func Adopted(record []byte, key string) bool {
+ if key == "" {
+ return false
+ }
+ var a Adoption
+ dec := json.NewDecoder(bytes.NewReader(record))
+ dec.DisallowUnknownFields()
+ if err := dec.Decode(&a); err != nil || dec.More() {
+ return false
+ }
+ return a.V == 1 && a.Var == Var && a.SHA256 == Digest(key)
+}
+
+// ValidDigest reports whether s is a lowercase hex SHA-256.
+func ValidDigest(s string) bool {
+ if len(s) != 64 {
+ return false
+ }
+ for _, r := range s {
+ if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
+ return false
+ }
+ }
+ return true
+}
+```
+
+- [ ] **Step 4: One parser, and the merge follows the rule**
+
+In `internal/activator/admintoken.go`, replace `parseEnvBytes`'s body (keep its doc comment), drop the now-unused `"strings"` import and add `"github.com/carlosframework/platform/internal/instancekey"`:
+
+```go
+func parseEnvBytes(raw []byte) map[string]string {
+ return instancekey.ParseEnv(raw)
+}
+```
+
+In `internal/activator/backend_exec.go`, add the same import, and end `mergeEnv` with the rule:
+
+```go
+ for k, v := range overlay {
+ out[k] = v
+ }
+ // An empty CARLOS_INSTANCE_KEY in the instance's own file supplies no
+ // key, so it never shadows a configured one: the child is never handed
+ // an empty key while a non-empty one exists (instance-key plan ruling
+ // 13). instancekey.Effective is the rule the key's mint decides by.
+ if key, _ := instancekey.Effective(base, overlay); key != "" {
+ out[instancekey.Var] = key
+ }
+ return out
+}
+```
+
+In `LaunchEnvironment`, an empty own key is not a shadow:
+
+```go
+ for k := range ownEnv {
+ if _, rendered := cfgEnv[k]; rendered {
+ if k == instancekey.Var && ownEnv[k] == "" {
+ continue // an empty own key shadows nothing (mergeEnv)
+ }
+ shadowed = append(shadowed, k)
+ }
+ }
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run: `nice -n 19 go test ./internal/instancekey/ -count=1 && nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartMergesConfigEnvUnderOwnEnv|LaunchEnvironment|TestEnsureAdminToken' -count=1` (sandbox off for the second: it starts children)
+Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+gofmt -l internal/instancekey internal/activator
+git add internal/instancekey internal/activator/admintoken.go internal/activator/backend_exec.go
+git commit -m "instancekey: one rule for the instance key's parse, effective layer and adoption record (L21)
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 2: The mint, durable every time
**Files:**
- Create: `internal/activator/instancekey.go`
- Test: `internal/activator/instancekey_test.go`
**Interfaces:**
-- Consumes: `adminTokenMu` (admintoken.go:43), `logPerFile` (:70), `parseEnvBytes` (:245), `instancefs.ReadBeneathLimit`, `instancefs.OpenBeneath`, `(*ExecBackend).pathsFor` (paths.go:33), `Layout*` constants (layout.go:21-26).
+- Consumes: Task 1's `instancekey`; `adminTokenMu` (admintoken.go:43), `logPerFile` (:70), `parseEnvBytes`, `instancefs.ReadBeneathLimit`, `instancefs.OpenBeneath`, `(*ExecBackend).pathsFor` (paths.go:33), `Layout*` constants (layout.go:21-26).
- Produces:
- - `const InstanceKeyVar = "CARLOS_INSTANCE_KEY"`
- - `type InstanceKeyOutcome int` with `InstanceKeyPresent`, `InstanceKeyMinted`, `InstanceKeyFromConfig`, `InstanceKeyNoEnvFile`
+ - `const InstanceKeyVar = instancekey.Var`
+ - `type InstanceKeyOutcome int` with `InstanceKeyPresent`, `InstanceKeyMinted`, `InstanceKeyFromConfig`, `InstanceKeyNoEnvFile`, `InstanceKeyReplica`
- `func EnsureInstanceKey(path string) (InstanceKeyOutcome, error)`
- `func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error)`
+ - `func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error)`
- `func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error)`
- - `func wellFormedInstanceKey(key string) bool` (package-private, used by later tests)
+ - `func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error`, `errInstanceEnvNeedsWrite`
+ - the test seam `instanceKeySync` (file and directory sync), and `syncDir`
- [ ] **Step 1: Write the failing tests**
@@ -187,6 +807,7 @@ package activator
import (
"bytes"
"encoding/base64"
+ "errors"
"log"
"os"
"path/filepath"
@@ -208,6 +829,41 @@ func keyLines(raw []byte) []string {
return out
}
+// syncProbe counts the key's syncs and fails them on demand. Set the
+// error fields only between calls.
+type syncProbe struct {
+ file, dir int
+ fileErr, dirErr error
+}
+
+func probeInstanceKeySync(t *testing.T) *syncProbe {
+ t.Helper()
+ p := &syncProbe{}
+ adminTokenMu.Lock()
+ old := instanceKeySync
+ instanceKeySync.file = func(f *os.File) error {
+ p.file++
+ if p.fileErr != nil {
+ return p.fileErr
+ }
+ return f.Sync()
+ }
+ instanceKeySync.dir = func(d string) error {
+ p.dir++
+ if p.dirErr != nil {
+ return p.dirErr
+ }
+ return syncDir(d)
+ }
+ adminTokenMu.Unlock()
+ t.Cleanup(func() {
+ adminTokenMu.Lock()
+ instanceKeySync = old
+ adminTokenMu.Unlock()
+ })
+ return p
+}
+
func TestEnsureInstanceKeyMintsOnceInTheAgreedFormat(t *testing.T) {
p := filepath.Join(t.TempDir(), "shop.example.com.env")
got, err := EnsureInstanceKey(p)
@@ -295,21 +951,25 @@ func TestEnsureInstanceKeyAppendsOnAFreshLineAndKeepsEveryOtherLine(t *testing.T
// A key the app's configuration supplies is the app's key: minting one into
// <host>.env would override it (mergeEnv) and change the key under its data.
+// The decision is the effective env's, so an empty CARLOS_INSTANCE_KEY= in
+// the own file does not hide the configured key (finding 4).
func TestEnsureInstanceKeyDefersToTheRenderedConfiguration(t *testing.T) {
- dir := t.TempDir()
- own := filepath.Join(dir, "shop.example.com.env")
- if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=from-config\n"), 0o600); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(own, []byte("A=1\n"), 0o600); err != nil {
- t.Fatal(err)
- }
- got, err := EnsureInstanceKey(own)
- if err != nil || got != InstanceKeyFromConfig {
- t.Fatalf("ensure = %v, %v; want from-config", got, err)
- }
- if raw, _ := os.ReadFile(own); string(raw) != "A=1\n" {
- t.Fatalf("own file changed: %q", raw)
+ for _, own := range []string{"A=1\n", "A=1\n" + InstanceKeyVar + "=\n"} {
+ dir := t.TempDir()
+ p := filepath.Join(dir, "shop.example.com.env")
+ if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=from-config\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(p, []byte(own), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ got, err := EnsureInstanceKey(p)
+ if err != nil || got != InstanceKeyFromConfig {
+ t.Fatalf("own %q: ensure = %v, %v; want from-config", own, got, err)
+ }
+ if raw, _ := os.ReadFile(p); string(raw) != own {
+ t.Fatalf("own file changed: %q", raw)
+ }
}
}
@@ -411,12 +1071,81 @@ func TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty(t *testing.T) {
t.Fatalf("ensure = %v, %v; want minted", got, err)
}
}
+
+// Finding 2: a mint whose file or directory sync fails reports failure,
+// and so does every retry until a sync succeeds. The retry finds the key
+// present, keeps its bytes, and still completes the chmod and both syncs
+// before it reports success.
+func TestEnsureInstanceKeyRedoesAFailedSyncBeforeReportingSuccess(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ fail func(*syncProbe, error)
+ }{
+ {"file sync", func(p *syncProbe, err error) { p.fileErr = err }},
+ {"directory sync", func(p *syncProbe, err error) { p.dirErr = err }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ probe := probeInstanceKeySync(t)
+ p := filepath.Join(t.TempDir(), "shop.example.com.env")
+ tc.fail(probe, errors.New("injected: EIO"))
+ if _, err := EnsureInstanceKey(p); err == nil {
+ t.Fatal("a mint whose sync failed reported success")
+ }
+ raw, _ := os.ReadFile(p)
+ if len(keyLines(raw)) != 1 {
+ t.Fatalf("the failed mint left %d keys, want its 1", len(keyLines(raw)))
+ }
+ if _, err := EnsureInstanceKey(p); err == nil {
+ t.Fatal("a retry reported success while the sync still fails")
+ }
+ if err := os.Chmod(p, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ tc.fail(probe, nil)
+ before := *probe
+ got, err := EnsureInstanceKey(p)
+ if err != nil || got != InstanceKeyPresent {
+ t.Fatalf("retry = %v, %v; want present", got, err)
+ }
+ if probe.file != before.file+1 || probe.dir != before.dir+1 {
+ t.Fatalf("the retry synced the file %d and the directory %d times, want once each", probe.file-before.file, probe.dir-before.dir)
+ }
+ if raw2, _ := os.ReadFile(p); !bytes.Equal(raw, raw2) {
+ t.Fatal("a retry rewrote the key")
+ }
+ if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
+ t.Fatalf("mode = %o, want the present path to set 600", st.Mode().Perm())
+ }
+ })
+ }
+}
+
+// Open question 1, provisional default: a replica row gets no platform key
+// on any path, so an app on a replica that adopts the helper refuses to
+// start rather than seal with a key its home does not hold.
+func TestReplicaRowsGetNoPlatformKey(t *testing.T) {
+ b := &ExecBackend{DataDir: t.TempDir()}
+ rt := registry.Route{Host: "shop-au.example.com", Kind: "instance", Socket: "/run/carlos/shop-au.sock", Replica: true}
+ envPath := filepath.Join(b.DataDir, rt.Host+".env")
+ if err := os.WriteFile(envPath, []byte("CARLOS_ADMIN_TOKEN=t\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if got, err := b.ensureInstanceKeyFor(rt, b.pathsFor(rt)); err != nil || got != InstanceKeyReplica {
+ t.Fatalf("launch ensure = %v, %v; want replica", got, err)
+ }
+ if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyReplica {
+ t.Fatalf("backfill = %v, %v; want replica", got, err)
+ }
+ if raw, _ := os.ReadFile(envPath); len(keyLines(raw)) != 0 {
+ t.Fatal("a replica row was given a platform key")
+ }
+}
```
- [ ] **Step 2: Run them to verify they fail**
-Run: `go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey' -count=1`
-Expected: build failure, `undefined: EnsureInstanceKey` (and `InstanceKeyVar`, `InstanceKeyMinted`, ...).
+Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestReplicaRows' -count=1`
+Expected: build failure, `undefined: EnsureInstanceKey` (and `InstanceKeyVar`, `instanceKeySync`, ...).
- [ ] **Step 3: Write the implementation**
@@ -439,14 +1168,17 @@ package activator
// - create-only: a key already in the file is never rewritten, whatever
// its shape. A malformed one is reported and left in place, as
// EnsureAdminToken does with a malformed token;
-// - the rendered configuration wins: when <host>.config.env already
-// supplies CARLOS_INSTANCE_KEY, nothing is minted, because minting one
-// into <host>.env would override it (mergeEnv) and change the app's key;
+// - the decision is made on the effective env (instancekey.Effective,
+// the rule mergeEnv applies): a key the rendered configuration
+// supplies is the app's key and nothing is minted; an empty own value
+// is no key;
+// - durable every time: before an ensure reports success the file is
+// 0600 (fchmod on its descriptor), the file is fsynced and so is its
+// directory, when the key was already there too. A sync that failed
+// once is redone by the next ensure, never skipped because the bytes
+// are already there;
// - the value is never logged, never put in an error, and never leaves
-// the file except in the launch environment;
-// - the file ends up 0600, set on the descriptor that wrote it, and the
-// write is fsynced before the call returns, so a key handed to a
-// process cannot vanish in a power cut.
+// the file except in the launch environment.
import (
"crypto/rand"
@@ -458,11 +1190,12 @@ import (
"strings"
"github.com/carlosframework/platform/internal/instancefs"
+ "github.com/carlosframework/platform/internal/instancekey"
"github.com/carlosframework/platform/internal/registry"
)
// InstanceKeyVar is the variable the key is delivered as.
-const InstanceKeyVar = "CARLOS_INSTANCE_KEY"
+const InstanceKeyVar = instancekey.Var
// instanceKeyBytes is the key's length before encoding: 32 CSPRNG bytes,
// standard base64 with padding, 44 characters. rastrillo uses the exact
@@ -474,10 +1207,10 @@ const instanceKeyBytes = 32
type InstanceKeyOutcome int
const (
- // InstanceKeyPresent: the own env file already carried a key. Nothing
- // was written.
+ // InstanceKeyPresent: the own env file already carried a key. No byte
+ // was written; the file was made durable.
InstanceKeyPresent InstanceKeyOutcome = iota
- // InstanceKeyMinted: a key was appended.
+ // InstanceKeyMinted: a key was appended and made durable.
InstanceKeyMinted
// InstanceKeyFromConfig: the rendered configuration supplies the key.
// Nothing was written.
@@ -486,6 +1219,10 @@ const (
// has never been started here. Nothing was written; its first start
// mints the key.
InstanceKeyNoEnvFile
+ // InstanceKeyReplica: a replica row (#162). The platform mints no key
+ // for a replica until replicas can carry their home's key (open
+ // question 1, provisional default).
+ InstanceKeyReplica
)
func (o InstanceKeyOutcome) String() string {
@@ -498,10 +1235,33 @@ func (o InstanceKeyOutcome) String() string {
return "from-config"
case InstanceKeyNoEnvFile:
return "no-env-file"
+ case InstanceKeyReplica:
+ return "replica"
}
return fmt.Sprintf("outcome(%d)", int(o))
}
+// instanceKeySync is the durability step's two syncs, a seam so a test can
+// fail each one. Read and replaced only under adminTokenMu.
+var instanceKeySync = struct {
+ file func(*os.File) error
+ dir func(string) error
+}{file: (*os.File).Sync, dir: syncDir}
+
+// syncDir fsyncs a directory, so an entry created in it survives a power
+// cut.
+func syncDir(dir string) error {
+ d, err := os.Open(dir)
+ if err != nil {
+ return err
+ }
+ err = d.Sync()
+ if cerr := d.Close(); err == nil {
+ err = cerr
+ }
+ return err
+}
+
func mintInstanceKey() (string, error) {
raw := make([]byte, instanceKeyBytes)
if _, err := rand.Read(raw); err != nil {
@@ -542,6 +1302,18 @@ func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) {
return ensureInstanceKey(dir, name, true, true)
}
+// ensureInstanceKeyFor is the launch paths' ensure for rt: none for a
+// replica row, the nested body for a nested row, the flat one otherwise.
+func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) {
+ if rt.Replica {
+ return InstanceKeyReplica, nil
+ }
+ if p.layout == LayoutNested {
+ return EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env")
+ }
+ return EnsureInstanceKey(p.envFile)
+}
+
// BackfillInstanceKey is the startup backfill's step for one row: a key is
// appended only to an own env file that already exists. A host with no file
// has never been started on this box, and its first start mints the key;
@@ -549,6 +1321,9 @@ func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) {
// to be empty (moveslandinstall.go placeStaged). A row this backend would
// refuse to start (fenced, conflict) is an error and nothing is written.
func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error) {
+ if rt.Replica {
+ return InstanceKeyReplica, nil
+ }
p := b.pathsFor(rt)
switch p.layout {
case LayoutNested:
@@ -559,6 +1334,36 @@ func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome
return ensureInstanceKey(p.dataDir, rt.Host+".env", false, false)
}
+// openInstanceEnv opens an own env file for append. Only a create passes
+// a mode: openat2 refuses a non-zero mode without O_CREAT (EINVAL), and an
+// existing file's mode is set by the fchmod that follows anyway.
+func openInstanceEnv(dir, name string, beneath, create bool) (*os.File, error) {
+ flags, perm := os.O_APPEND|os.O_WRONLY, os.FileMode(0)
+ if create {
+ flags, perm = flags|os.O_CREATE, 0o600
+ }
+ if beneath {
+ return instancefs.OpenBeneath(dir, name, flags, perm)
+ }
+ return os.OpenFile(filepath.Join(dir, name), flags, perm)
+}
+
+// settleInstanceEnv is the durability step every successful ensure ends
+// with: 0600 on the descriptor, then the file, then its directory. Caller
+// holds adminTokenMu.
+func settleInstanceEnv(f *os.File, dir, path string) error {
+ if err := f.Chmod(0o600); err != nil {
+ return fmt.Errorf("chmod %s: %w", path, err)
+ }
+ if err := instanceKeySync.file(f); err != nil {
+ return fmt.Errorf("sync %s: %w", path, err)
+ }
+ if err := instanceKeySync.dir(dir); err != nil {
+ return fmt.Errorf("sync %s: %w", dir, err)
+ }
+ return nil
+}
+
// ensureInstanceKey is the shared body. It holds adminTokenMu, the lock
// every mint into <host>.env takes, so a key append and a token append can
// never interleave. create false is the backfill: an absent file is
@@ -582,36 +1387,39 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco
case err != nil:
return InstanceKeyPresent, fmt.Errorf("read %s: %w", path, err)
}
- if key := parseEnvBytes(raw)[InstanceKeyVar]; key != "" {
+ cfg, err := read(configEnvName(name))
+ if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return InstanceKeyPresent, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err)
+ }
+ // The decision is the effective env's: the key the launch will hand
+ // the app, by the same rule mergeEnv applies.
+ key, layer := instancekey.Effective(parseEnvBytes(cfg), parseEnvBytes(raw))
+ switch layer {
+ case instancekey.LayerConfig:
+ logPerFile("key-config", path, fmt.Sprintf("activator: %s is set by the app's configuration for %s: the platform mints none while it is", InstanceKeyVar, path))
+ return InstanceKeyFromConfig, nil
+ case instancekey.LayerOwn:
if !wellFormedInstanceKey(key) {
logPerFile("key-shape", path, fmt.Sprintf("activator: %s in %s is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it", InstanceKeyVar, path))
}
- return InstanceKeyPresent, nil
- }
- cfg, err := read(configEnvName(name))
- switch {
- case err == nil:
- if parseEnvBytes(cfg)[InstanceKeyVar] != "" {
- logPerFile("key-config", path, fmt.Sprintf("activator: %s is set by the app's configuration for %s: the platform mints none while it is", InstanceKeyVar, path))
- return InstanceKeyFromConfig, nil
+ // Present is not a no-op: a mint whose sync failed left these
+ // bytes behind, and this call is the retry that makes them
+ // durable before anything is started with them.
+ f, err := openInstanceEnv(dir, name, beneath, false)
+ if err != nil {
+ return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err)
}
- case !errors.Is(err, os.ErrNotExist):
- return InstanceKeyPresent, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err)
+ defer f.Close()
+ if err := settleInstanceEnv(f, dir, path); err != nil {
+ return InstanceKeyPresent, err
+ }
+ return InstanceKeyPresent, nil
}
- key, err := mintInstanceKey()
+ minted, err := mintInstanceKey()
if err != nil {
return InstanceKeyPresent, fmt.Errorf("mint instance key for %s: %w", path, err)
}
- flags := os.O_APPEND | os.O_WRONLY
- if create {
- flags |= os.O_CREATE
- }
- var f *os.File
- if beneath {
- f, err = instancefs.OpenBeneath(dir, name, flags, 0o600)
- } else {
- f, err = os.OpenFile(path, flags, 0o600)
- }
+ f, err := openInstanceEnv(dir, name, beneath, create)
if errors.Is(err, os.ErrNotExist) && !create {
return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill
}
@@ -623,34 +1431,62 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco
if len(raw) > 0 && raw[len(raw)-1] != '\n' {
prefix = "\n"
}
- if _, err := fmt.Fprintf(f, "%s%s=%s\n", prefix, InstanceKeyVar, key); err != nil {
+ if _, err := fmt.Fprintf(f, "%s%s=%s\n", prefix, InstanceKeyVar, minted); err != nil {
return InstanceKeyPresent, fmt.Errorf("write %s: %w", path, err)
}
- if err := f.Chmod(0o600); err != nil {
- return InstanceKeyPresent, fmt.Errorf("chmod %s: %w", path, err)
+ if err := settleInstanceEnv(f, dir, path); err != nil {
+ return InstanceKeyPresent, err
}
- if err := f.Sync(); err != nil {
- return InstanceKeyPresent, fmt.Errorf("sync %s: %w", path, err)
+ return InstanceKeyMinted, nil
+}
+
+// errInstanceEnvNeedsWrite is settleClaimInstanceEnv's answer for a file
+// whose mode is not 0600: making it so is a write, which the claim path
+// does only under its guard.
+var errInstanceEnvNeedsWrite = errors.New("activator: the instance env file needs its mode set")
+
+// settleClaimInstanceEnv is the claim launch source's durability step for
+// an env file that already carries what the launch needs: it makes the
+// file and its directory durable and writes nothing. fsync changes no
+// byte, mode or name, so it is not a host write and runs without the
+// claim's guard. A file that is not 0600 is errInstanceEnvNeedsWrite, and
+// the caller takes the guarded path, whose ensure sets the mode.
+func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error {
+ p := b.pathsFor(rt)
+ name := rt.Host + ".env"
+ adminTokenMu.Lock()
+ defer adminTokenMu.Unlock()
+ var f *os.File
+ var err error
+ if p.layout == LayoutNested {
+ f, err = instancefs.OpenBeneath(p.dataDir, name, os.O_RDONLY, 0)
+ } else {
+ f, err = os.Open(p.envFile)
}
- // The directory entry too: the token mint just before this one may
- // have created the file without syncing its directory, and a key a
- // process was handed must still be here after a power cut.
- d, err := os.Open(dir)
if err != nil {
- return InstanceKeyPresent, fmt.Errorf("open %s: %w", dir, err)
+ return fmt.Errorf("open %s: %w", p.envFile, err)
}
- syncErr := d.Sync()
- d.Close()
- if syncErr != nil {
- return InstanceKeyPresent, fmt.Errorf("sync %s: %w", dir, syncErr)
+ defer f.Close()
+ st, err := f.Stat()
+ if err != nil {
+ return fmt.Errorf("stat %s: %w", p.envFile, err)
}
- return InstanceKeyMinted, nil
+ if st.Mode().Perm() != 0o600 {
+ return errInstanceEnvNeedsWrite
+ }
+ if err := instanceKeySync.file(f); err != nil {
+ return fmt.Errorf("sync %s: %w", p.envFile, err)
+ }
+ if err := instanceKeySync.dir(p.dataDir); err != nil {
+ return fmt.Errorf("sync %s: %w", p.dataDir, err)
+ }
+ return nil
}
```
- [ ] **Step 4: Run the tests to verify they pass**
-Run: `go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestEnsureAdminToken' -count=1`
+Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestReplicaRows|TestEnsureAdminToken' -count=1`
Expected: PASS. The admin token tests are in the run because the key mint shares their lock and log throttle.
- [ ] **Step 5: Commit**
@@ -658,14 +1494,14 @@ Expected: PASS. The admin token tests are in the run because the key mint shares
```bash
gofmt -l internal/activator
git add internal/activator/instancekey.go internal/activator/instancekey_test.go
-git commit -m "activator: mint CARLOS_INSTANCE_KEY create-only into <host>.env (L21)
+git commit -m "activator: mint CARLOS_INSTANCE_KEY create-only into <host>.env, durable on every call (L21)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
```
---
-### Task 2: Every activator launch path mints the key
+### Task 3: Every activator launch path mints the key
**Files:**
- Modify: `internal/activator/backend_exec.go:831-833` (`seedChild`), `:1369-1371` (`spawnChildFrom`)
@@ -673,48 +1509,198 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
- Test: `internal/activator/instancekey_test.go` (append), `internal/activator/instancekey_linux_test.go` (create)
**Interfaces:**
-- Consumes: Task 1's `EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `BackfillInstanceKey`, `wellFormedInstanceKey`, `InstanceKeyVar`; test helpers `testExecBackend` (activator_test.go:4262), `stageRecordingLiveFullEnv` (:4712), `waitForFile` (:4741), `tmpSock` (:954), `nestedBackend`, `fakeControl`, `fakeUnit` (unitproc_test.go:24-160).
-- Produces: nothing new; the launch env of every exec start and seed carries `CARLOS_INSTANCE_KEY`.
+- Consumes: Task 2's `ensureInstanceKeyFor`, `BackfillInstanceKey`, `probeInstanceKeySync`, `wellFormedInstanceKey`; test helpers `testExecBackend` (activator_test.go:4262), `stageRecordingLiveFullEnv` (:4712), `waitForFile` (:4741), `tmpSock` (:954), `writeMinimalSQLite` (backend_exec_test.go:343), the `restoreReplica` seam (backend_exec.go:757), `nestedBackend`, `fakeControl`, `fakeUnit` (unitproc_test.go:24-160).
+- Produces: nothing new; the launch env of every exec start and seed carries `CARLOS_INSTANCE_KEY`, and no child starts before it is durable.
- [ ] **Step 1: Write the failing tests**
Append to `internal/activator/instancekey_test.go`, and add `"context"` and `"time"` to its imports:
```go
+// startRecorded starts host on b with the recording binary and returns the
+// environment the child saw, or nil when no child ran within a second.
+func startRecorded(t *testing.T, b *ExecBackend, host string) (map[string]string, error) {
+ t.Helper()
+ _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
+ rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)}
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer cancel()
+ proc, _, err := b.Start(ctx, rt) // the fake binary never binds: the health wait fails, as in the sibling tests
+ if proc != nil {
+ defer proc.Stop(context.Background())
+ }
+ deadline := time.Now().Add(time.Second)
+ for time.Now().Before(deadline) {
+ if raw, rerr := os.ReadFile(envPath); rerr == nil {
+ child := map[string]string{}
+ for _, line := range strings.Split(string(raw), "\n") {
+ if k, v, ok := strings.Cut(line, "="); ok {
+ child[k] = v
+ }
+ }
+ return child, err
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ return nil, err
+}
+
// Start hands the child the key it minted, and a second Start (a restart, a
// redeploy) hands it the same one.
func TestExecBackendStartHandsTheChildTheMintedInstanceKey(t *testing.T) {
b := testExecBackend(t)
host := "shop.example.com"
- start := func() map[string]string {
- _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
- rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)}
- ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
- defer cancel()
- proc, _, _ := b.Start(ctx, rt) // the fake binary never binds: the health wait fails, as in the sibling tests
- child := map[string]string{}
- for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") {
- if k, v, ok := strings.Cut(line, "="); ok {
- child[k] = v
- }
- }
- if proc != nil {
- proc.Stop(context.Background())
- }
- return child
- }
- first := start()
+ first, _ := startRecorded(t, b, host)
minted, err := readEnvFile(filepath.Join(b.DataDir, host+".env"))
if err != nil {
t.Fatal(err)
}
- if !wellFormedInstanceKey(minted[InstanceKeyVar]) || first[InstanceKeyVar] != minted[InstanceKeyVar] {
+ if first == nil || !wellFormedInstanceKey(minted[InstanceKeyVar]) || first[InstanceKeyVar] != minted[InstanceKeyVar] {
t.Fatal("the child was not handed the key minted into <host>.env")
}
- if second := start(); second[InstanceKeyVar] != minted[InstanceKeyVar] {
+ if second, _ := startRecorded(t, b, host); second == nil || second[InstanceKeyVar] != minted[InstanceKeyVar] {
t.Fatal("a second start handed the child a different key")
}
}
+
+// Finding 2 through a launch: while the key's sync fails, Start fails and
+// no child runs. Once the sync succeeds, the child is handed the key the
+// failed attempt wrote, never a second one.
+func TestExecBackendStartStartsNoChildUntilTheKeyIsSynced(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ fail func(*syncProbe, error)
+ }{
+ {"file sync", func(p *syncProbe, err error) { p.fileErr = err }},
+ {"directory sync", func(p *syncProbe, err error) { p.dirErr = err }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ probe := probeInstanceKeySync(t)
+ b := testExecBackend(t)
+ host := "shop.example.com"
+ tc.fail(probe, errors.New("injected: EIO"))
+ for i := 0; i < 2; i++ {
+ child, err := startRecorded(t, b, host)
+ if err == nil || !strings.Contains(err.Error(), "instance key for "+host) {
+ t.Fatalf("start %d with a failing sync: err = %v, want the instance key's error", i, err)
+ }
+ if child != nil {
+ t.Fatalf("start %d ran a child before the key was durable", i)
+ }
+ }
+ written, _ := readEnvFile(filepath.Join(b.DataDir, host+".env"))
+ tc.fail(probe, nil)
+ child, _ := startRecorded(t, b, host)
+ if child == nil || child[InstanceKeyVar] == "" || child[InstanceKeyVar] != written[InstanceKeyVar] {
+ t.Fatal("after the sync succeeded the child was not handed the key the failed attempt wrote")
+ }
+ })
+ }
+}
+
+// Finding 4 through a launch: <host>.env and <host>.config.env together.
+// An empty override never hands the child an empty key while the
+// configuration supplies one, and no key is minted over the configured one.
+// With neither supplying one, the child gets the minted key.
+func TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists(t *testing.T) {
+ tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("c", 64) + "\n"
+ for _, tc := range []struct {
+ name, own, config string
+ configured bool
+ }{
+ {"empty override, configured key", tok + InstanceKeyVar + "=\n", InstanceKeyVar + "=from-config\n", true},
+ {"no override, configured key", tok, InstanceKeyVar + "=from-config\n", true},
+ {"empty override, empty configuration", tok + InstanceKeyVar + "=\n", InstanceKeyVar + "=\n", false},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ b := testExecBackend(t)
+ host := "shop.example.com"
+ own := filepath.Join(b.DataDir, host+".env")
+ if err := os.WriteFile(own, []byte(tc.own), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(b.DataDir, host+".config.env"), []byte(tc.config), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ child, _ := startRecorded(t, b, host)
+ if child == nil {
+ t.Fatal("no child ran")
+ }
+ raw, _ := os.ReadFile(own)
+ if tc.configured {
+ if child[InstanceKeyVar] != "from-config" {
+ t.Fatalf("the child was handed %q, want the configured key", child[InstanceKeyVar])
+ }
+ if string(raw) != tc.own {
+ t.Fatal("a key was minted over the configured one")
+ }
+ return
+ }
+ env, _ := readEnvFile(own)
+ if !wellFormedInstanceKey(child[InstanceKeyVar]) || child[InstanceKeyVar] != env[InstanceKeyVar] {
+ t.Fatal("the child was not handed the minted key")
+ }
+ })
+ }
+}
+
+// Coverage (Astra round 1): the first process of a fresh flat instance is
+// often its seed. The seed mints the key, sees it, and the first start
+// hands the instance the same one.
+func TestExecBackendSeedMintsTheKeyForAFreshFlatInstance(t *testing.T) {
+ b := testExecBackend(t)
+ host := "seeded.example.com"
+ _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
+ if err := b.Seed(context.Background(), registry.Route{Host: host, Kind: "instance", Seed: `["seed"]`}); err != nil {
+ t.Fatalf("Seed: %v", err)
+ }
+ seed := map[string]string{}
+ for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") {
+ if k, v, ok := strings.Cut(line, "="); ok {
+ seed[k] = v
+ }
+ }
+ own, _ := readEnvFile(filepath.Join(b.DataDir, host+".env"))
+ if !wellFormedInstanceKey(own[InstanceKeyVar]) || seed[InstanceKeyVar] != own[InstanceKeyVar] {
+ t.Fatal("the seed did not mint the key and see it")
+ }
+ if child, _ := startRecorded(t, b, host); child == nil || child[InstanceKeyVar] != own[InstanceKeyVar] {
+ t.Fatal("the first start handed the instance a different key from its seed's")
+ }
+}
+
+// Coverage (Astra round 1): a hibernation wake restores the database and
+// leaves <host>.env alone, so the start after the restore hands the
+// instance the key it had.
+func TestHibernationRestoreKeepsTheInstanceKey(t *testing.T) {
+ b := testExecBackend(t)
+ host := "sleepy.example.com"
+ if child, _ := startRecorded(t, b, host); child == nil {
+ t.Fatal("no child ran")
+ }
+ envPath := filepath.Join(b.DataDir, host+".env")
+ before, _ := os.ReadFile(envPath)
+ key := parseEnvBytes(before)[InstanceKeyVar]
+ if !wellFormedInstanceKey(key) {
+ t.Fatal("the first start minted no key")
+ }
+ old := restoreReplica
+ restoreReplica = func(_ context.Context, dbPath, bucket, prefix, region string) (bool, error) {
+ return true, writeMinimalSQLite(dbPath)
+ }
+ t.Cleanup(func() { restoreReplica = old })
+ _ = os.Remove(filepath.Join(b.DataDir, host+".db")) // hibernated: the database is gone
+ rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)}
+ if fresh, err := b.Restore(context.Background(), rt); err != nil || fresh {
+ t.Fatalf("Restore = fresh %v, %v; want a restored database", fresh, err)
+ }
+ if after, _ := os.ReadFile(envPath); !bytes.Equal(before, after) {
+ t.Fatal("the restore changed <host>.env")
+ }
+ if child, _ := startRecorded(t, b, host); child == nil || child[InstanceKeyVar] != key {
+ t.Fatal("the start after a restore handed the instance a different key")
+ }
+}
```
Create `internal/activator/instancekey_linux_test.go`:
@@ -725,6 +1711,7 @@ package activator
import (
"os"
"path/filepath"
+ "strings"
"testing"
"time"
@@ -735,23 +1722,60 @@ import (
)
// The nested prepare mints the key into the instance directory's own env
-// file and hands it to the role in the same env it returns.
+// file and hands it to the role in the same env it returns: the run role
+// and the seed role (coverage, Astra round 1).
func TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile(t *testing.T) {
- b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
- _, ctl, logs, env, err := b.prepare(rt, instancectl.RoleRun)
- if err != nil {
- t.Fatalf("prepare: %v", err)
+ for _, role := range []instancectl.Role{instancectl.RoleRun, instancectl.RoleSeed} {
+ t.Run(string(role), func(t *testing.T) {
+ b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
+ _, ctl, logs, env, err := b.prepare(rt, role)
+ if err != nil {
+ t.Fatalf("prepare: %v", err)
+ }
+ logs.Close()
+ if ctl != nil {
+ ctl.Close()
+ }
+ own, err := readEnvFileBeneath(provision.InstanceDir(b.DataDir, rt.Host), rt.Host+".env")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !wellFormedInstanceKey(own[InstanceKeyVar]) || env[InstanceKeyVar] != own[InstanceKeyVar] {
+ t.Fatal("prepare's env does not carry the key minted into the nested env file")
+ }
+ })
}
- logs.Close()
- if ctl != nil {
- ctl.Close()
+}
+
+// Finding 3: the nested backfill opens an existing file with mode 0
+// (openat2 refuses a mode without O_CREAT), appends, and sets 0600 on the
+// descriptor.
+func TestBackfillInstanceKeyAppendsToANestedEnvFile(t *testing.T) {
+ b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
+ dir := provision.InstanceDir(b.DataDir, rt.Host)
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ t.Fatal(err)
}
- own, err := readEnvFileBeneath(provision.InstanceDir(b.DataDir, rt.Host), rt.Host+".env")
- if err != nil {
+ envPath := filepath.Join(dir, rt.Host+".env")
+ tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("e", 64) + "\n"
+ if err := os.WriteFile(envPath, []byte(tok), 0o644); err != nil {
t.Fatal(err)
}
- if !wellFormedInstanceKey(own[InstanceKeyVar]) || env[InstanceKeyVar] != own[InstanceKeyVar] {
- t.Fatal("prepare's env does not carry the key minted into the nested env file")
+ if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyMinted {
+ t.Fatalf("nested backfill = %v, %v; want minted", got, err)
+ }
+ raw, _ := os.ReadFile(envPath)
+ if !strings.HasPrefix(string(raw), tok) || len(keyLines(raw)) != 1 || !wellFormedInstanceKey(keyLines(raw)[0]) {
+ t.Fatalf("nested env file after the backfill: %d bytes, %d keys", len(raw), len(keyLines(raw)))
+ }
+ if st, _ := os.Stat(envPath); st.Mode().Perm() != 0o600 {
+ t.Fatalf("mode = %o, want 600", st.Mode().Perm())
+ }
+ if got, err := b.BackfillInstanceKey(rt); err != nil || got != InstanceKeyPresent {
+ t.Fatalf("second nested backfill = %v, %v; want present", got, err)
+ }
+ if raw2, _ := os.ReadFile(envPath); string(raw2) != string(raw) {
+ t.Fatal("the second backfill changed the file")
}
}
@@ -803,8 +1827,8 @@ func TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking(t *testing.T) {
- [ ] **Step 2: Run them to verify they fail**
-Run: `go test ./internal/activator/ -run 'TestExecBackendStartHandsTheChildTheMintedInstanceKey|TestPrepareMintsTheInstanceKey|TestEnsureInstanceKeyBeneathRefusesASymlink|TestBackfillInstanceKeyRefusesAFIFO' -count=1`
-Expected: `TestExecBackendStartHandsTheChildTheMintedInstanceKey` and `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` FAIL (no key in the child's env or the nested file). The symlink and FIFO tests already pass, because Task 1's body uses instancefs. They are here to pin the nested path.
+Run (sandbox off: these start children and bind sockets): `nice -n 19 go test ./internal/activator/ -run 'TestExecBackendStartHandsTheChild|TestExecBackendStartStartsNoChild|TestExecBackendStartNeverHands|TestExecBackendSeedMintsTheKey|TestHibernationRestoreKeeps|TestPrepareMintsTheInstanceKey|TestBackfillInstanceKeyAppendsToANested|TestEnsureInstanceKeyBeneathRefuses|TestBackfillInstanceKeyRefusesAFIFO' -count=1 -v`
+Expected (checked in the scratch build): FAIL for `TestExecBackendStartHandsTheChildTheMintedInstanceKey`, both `TestExecBackendStartStartsNoChildUntilTheKeyIsSynced` cases, the "empty override, empty configuration" case of `TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExists` (its two configured cases already pass, from Task 1's `mergeEnv`), `TestExecBackendSeedMintsTheKeyForAFreshFlatInstance`, `TestHibernationRestoreKeepsTheInstanceKey` and both roles of `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile`. The nested backfill, symlink and FIFO tests already pass, from Task 2's body; they pin the nested path.
- [ ] **Step 3: Call the mint on each launch path**
@@ -816,7 +1840,7 @@ In `internal/activator/backend_exec.go`, `seedChild`, after the admin token mint
}
// The instance key, before the env is read for the same reason: a seed
// is usually the first process of a fresh instance (L21).
- if _, err := EnsureInstanceKey(paths.envFile); err != nil {
+ if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil {
return fmt.Errorf("instance key for %s: %w", host, err)
}
fileEnv, err := b.instanceEnv(rt)
@@ -830,8 +1854,9 @@ In `spawnChildFrom`, after the admin token mint:
}
// The instance key is minted here for the same reason and fails the
// wake the same way: an app that takes its key from the platform
- // refuses to start without one (L21).
- if _, err := EnsureInstanceKey(paths.envFile); err != nil {
+ // refuses to start without one (L21). It returns only once the key is
+ // durable, so no child starts with a key a power cut could take back.
+ if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil {
return nil, "", fmt.Errorf("instance key for %s: %w", host, err)
}
```
@@ -842,44 +1867,181 @@ In `internal/activator/prepare.go`, `prepareAt`, after `EnsureAdminTokenBeneath`
if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil {
return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err)
}
- if _, err := EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env"); err != nil {
+ if _, err := b.ensureInstanceKeyFor(rt, p); err != nil {
return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err)
}
```
-- [ ] **Step 4: Run the tests, then the whole package**
+- [ ] **Step 4: Run the tests, then the package's launch tests**
+
+Run (sandbox off): the Step 2 command.
+Expected: PASS.
+
+Run (sandbox off): `nice -n 19 go test ./internal/activator/ -run 'Seed|Start|Prepare|LaunchEnvironment|Control|InstanceKey|AdminToken' -count=1`
+Expected: PASS. If an existing test compares a child's whole environment or an env file's exact bytes after a start or seed, it now sees one more line, `CARLOS_INSTANCE_KEY=...`. Update that assertion to allow the key line (assert it is well formed). Do not weaken any other check, and list each test you changed in the commit message. (None needed changing in the scratch build.)
+
+- [ ] **Step 5: Commit**
+
+```bash
+gofmt -l internal/activator
+git add internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/instancekey_test.go internal/activator/instancekey_linux_test.go
+git commit -m "activator: seed, start and nested prepare mint the instance key durably before the env is read (L21)
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 4: The claim's launch source mints under the guard, and syncs before it answers
+
+**Files:**
+- Modify: `internal/activator/claim_snapshot.go:40-67` (`EnsureClaimAdminToken`)
+- Modify: `internal/host/claimowners.go:125-138` (doc on `ClaimFilesManifest`, O1')
+- Test: `internal/activator/claim_instancekey_test.go`, `internal/activator/claimretry_instancekey_linux_test.go`, `internal/host/claim_instancekey_test.go` (create all three)
+
+**Interfaces:**
+- Consumes: Task 2's `ensureInstanceKeyFor`, `settleClaimInstanceEnv`, `errInstanceEnvNeedsWrite`, `probeInstanceKeySync`; `claimServingLaunchSource` (claimproduction.go:421), `claimServingOwners` (claimowners.go:293) with `Select` (:375) and `Materialize` (:425); test fixture `newAdminTokenFixture`, `adminTokenHost`, `f.envPath()`, `f.token()` (claimproduction_admintoken_test.go:20-118); `leasetest.NewVersionedBucket`; `Service.startAdmitted` (claimwake.go:330) as `claimloop_linux_test.go` drives it.
+- Produces: `EnsureClaimAdminToken` initializes the token and the key inside the one authorized write, and syncs a present key before it answers.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `internal/activator/claim_instancekey_test.go`:
+
+```go
+package activator
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// Finding 2 on the claim path: when the token and the key are already
+// present, EnsureClaimAdminToken writes nothing and calls no authorizer,
+// but it does not answer until the env file and its directory are synced.
+// A file that is not 0600 goes through the authorized write, which sets it.
+func TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers(t *testing.T) {
+ probe := probeInstanceKeySync(t)
+ const host = "claim-key.example"
+ b := &ExecBackend{DataDir: t.TempDir()}
+ rt := registry.Route{Host: host, Socket: "/run/carlos/claim-key.sock", DBPath: "/data/claim-key.db"}
+ envPath := filepath.Join(b.DataDir, host+".env")
+ body := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("d", 64) + "\n" + InstanceKeyVar + "=" + strings.Repeat("A", 43) + "=\n"
+ if err := os.WriteFile(envPath, []byte(body), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ authorized := 0
+ authorize := func(_ context.Context, write func() error) error { authorized++; return write() }
+ for _, fail := range []func(error){func(e error) { probe.fileErr = e }, func(e error) { probe.dirErr = e }} {
+ fail(errors.New("injected: EIO"))
+ if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err == nil {
+ t.Fatal("the launch source answered while the env file's sync fails")
+ }
+ fail(nil)
+ }
+ before := *probe
+ if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err != nil {
+ t.Fatal(err)
+ }
+ if authorized != 0 || probe.file != before.file+1 || probe.dir != before.dir+1 {
+ t.Fatalf("authorized %d, synced file %d dir %d; want 0, 1, 1", authorized, probe.file-before.file, probe.dir-before.dir)
+ }
+ if err := os.Chmod(envPath, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := b.EnsureClaimAdminToken(context.Background(), rt, authorize); err != nil {
+ t.Fatal(err)
+ }
+ if st, _ := os.Stat(envPath); authorized != 1 || st.Mode().Perm() != 0o600 {
+ t.Fatalf("a 0644 file: authorized %d, mode %o; want the guarded path to set 600", authorized, st.Mode().Perm())
+ }
+ if raw, _ := os.ReadFile(envPath); string(raw) != body {
+ t.Fatal("the claim path rewrote the env file")
+ }
+}
+```
+
+Create `internal/activator/claimretry_instancekey_linux_test.go`:
-Run: `go test ./internal/activator/ -run 'InstanceKey' -count=1`
-Expected: PASS.
+```go
+//go:build linux
-Run: `go test ./internal/activator/ -count=1`
-Expected: PASS. If an existing test compares a child's whole environment or an env file's exact bytes after a start or seed, it now sees one more line, `CARLOS_INSTANCE_KEY=...`. Update that assertion to allow the key line (assert it is well formed). Do not weaken any other check, and list each test you changed in the commit message.
+package activator
-- [ ] **Step 5: Commit**
+import (
+ "context"
+ "errors"
+ "os/exec"
+ "slices"
+ "sync"
+ "testing"
+ "time"
-```bash
-gofmt -l internal/activator
-git add internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/instancekey_test.go internal/activator/instancekey_linux_test.go
-git commit -m "activator: seed, start and nested prepare mint the instance key before the env is read (L21)
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/registry"
+)
-Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+// Open question 5's premise, corrected (instance-key plan rev 2): on a
+// claim box a session's process retries do not go back to the launch
+// source, which is where the key is minted. Prepare runs once per
+// admitted session and captures the launch inputs, and every attempt
+// calls Materialize (the C8 provider's Establish) before it reserves a
+// start. So no attempt runs with a key minted after the files manifest
+// was established. This pins that shape of startAdmitted: one Prepare,
+// and Materialize before each attempt's start.
+func TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst(t *testing.T) {
+ var (
+ mu sync.Mutex
+ events []string
+ prepares int
+ )
+ note := func(e string) { mu.Lock(); events = append(events, e); mu.Unlock() }
+ obs := hostclaim.Observed{Host: "x.example", ETag: "one", Claim: &hostclaim.Claim{Holder: &hostclaim.Holder{Location: "instances/x.example/inc", Session: 1, Generation: "one"}, Op: hostclaim.Op{Kind: "landing", ID: "op", Phase: "probing", Exec: "box"}}}
+ work, _ := (hostwrite.Config{}).Select(t.Context(), obs.Host, "spawn")
+ a := &Service{cfg: Config{HostWrites: hostwrite.Config{Enabled: true}, Claims: &ClaimRuntime{Box: "box", Read: func(context.Context, string) (hostclaim.Observed, error) { return obs, nil }, Prepare: func(context.Context, registry.Route, hostclaim.Observed, startPolicy) (ClaimStart, error) {
+ mu.Lock()
+ prepares++
+ mu.Unlock()
+ return ClaimStart{RoleLauncher: &RoleLaunch{}, CurrentAttempt: func() hostclaim.WakeRetryAttempt { return hostclaim.WakeRetryAttempt{} }, Host: obs.Host, Work: work, Identity: hostwrite.Identity{Config: "c", Release: "r"},
+ OpenProbe: func(context.Context, time.Time) (time.Time, error) {
+ return time.Time{}, errors.New("unreachable while Start refuses")
+ },
+ Restore: func(context.Context) error { note("restore"); return nil },
+ Materialize: func(context.Context) error {
+ note("materialize")
+ return nil
+ },
+ Build: func() *exec.Cmd { return nil },
+ Ready: func(context.Context, *exec.Cmd) error { return nil },
+ Reserve: func(context.Context, int) error { note("reserve"); return nil },
+ Failed: func(context.Context, int) error { note("failed"); return nil },
+ Healthy: func(context.Context, int, *exec.Cmd) error { return nil },
+ Replicate: func(context.Context, string) (Rep, error) { return nil, nil },
+ Publish: func(context.Context, hostclaim.Observed) (hostclaim.Observed, error) { return obs, nil },
+ Retain: func(Proc, Rep, error) {},
+ }, nil
+ }}}}
+ // Two starts: the first refused (RoleLaunch.Start refuses until claim
+ // Plan 9 Group D), a 1 s backoff, then the second.
+ err := a.startAdmitted(t.Context(), &actEntry{host: obs.Host}, obs, startPolicy{MaxStarts: 2, Window: 30 * time.Second, Deadline: time.Now().Add(time.Minute), Probe: true})
+ if !errors.Is(err, ErrClaimIsolationUnavailable) {
+ t.Fatalf("the loop should end on the last start's refusal: %v", err)
+ }
+ want := []string{"restore", "reserve", "materialize", "failed", "reserve", "materialize", "failed"}
+ mu.Lock()
+ defer mu.Unlock()
+ if prepares != 1 || !slices.Equal(events, want) {
+ t.Fatalf("prepares %d, events %v; want 1 and %v", prepares, events, want)
+ }
+}
```
----
-
-### Task 3: The claim's launch source mints the key under the guard, before the files manifest
-
-**Files:**
-- Modify: `internal/activator/claim_snapshot.go:40-67` (`EnsureClaimAdminToken`)
-- Modify: `internal/host/claimowners.go:125-138` (doc on `ClaimFilesManifest`)
-- Test: `internal/host/claim_instancekey_test.go` (create)
-
-**Interfaces:**
-- Consumes: Task 1's mint; `claimServingLaunchSource` (claimproduction.go:421), `claimAdminTokens` (:368), `claimServingOwners` (claimowners.go:293) with `Select` (:375) and `Materialize` (:425); test fixture `newAdminTokenFixture`, `adminTokenHost`, `f.envPath()`, `f.token()` (claimproduction_admintoken_test.go:20-118); `leasetest.NewVersionedBucket`.
-- Produces: `EnsureClaimAdminToken` initializes both the token and the key, inside the one authorized write.
-
-- [ ] **Step 1: Write the failing tests**
-
Create `internal/host/claim_instancekey_test.go`:
```go
@@ -991,19 +2153,27 @@ func TestClaimLaunchMintsAKeyBesideAnExistingToken(t *testing.T) {
- [ ] **Step 2: Run them to verify they fail**
-Run: `go test ./internal/host/ -run 'TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest|TestClaimLaunchMintsAKeyBesideAnExistingToken' -count=1`
-Expected: FAIL. The first test reports "the env file at Establish carries no minted key". The second fails on the prefix check, because the token-only file is returned early and gets no key.
+Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureClaimAdminTokenSyncs|TestStartAdmittedRetriesReuse' -count=1 -v && nice -n 19 go test ./internal/host/ -run 'TestClaimFirstWakeKey|TestClaimLaunchMintsAKey' -count=1 -v`
+Expected (checked in the scratch build): FAIL for `TestEnsureClaimAdminTokenSyncsAPresentKeyBeforeItAnswers` (the early return answers without a sync), `TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest` ("the env file at Establish carries no minted key") and `TestClaimLaunchMintsAKeyBesideAnExistingToken` (the token-only file is returned early). `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` passes: it pins existing behaviour (ruling 16).
-- [ ] **Step 3: Mint the key in the authorized write**
+- [ ] **Step 3: Mint and sync in the launch source**
-In `internal/activator/claim_snapshot.go`, replace the body from the early return to the end of `EnsureClaimAdminToken`:
+In `internal/activator/claim_snapshot.go`, add `"errors"` to the imports and replace the body from the early return to the end of `EnsureClaimAdminToken`:
```go
// The instance key is initialized the same way and in the same
- // authorized write (L21). A key the rendered configuration supplies
- // counts as present, as ensureInstanceKey treats it.
- if vars["CARLOS_ADMIN_TOKEN"] != "" && vars[InstanceKeyVar] != "" {
- return nil
+ // authorized write (L21). vars is the effective env, so a key the
+ // rendered configuration supplies counts as present, and a replica
+ // gets none (open question 1). When both are present nothing is
+ // written, but the file is still made durable before the launch
+ // source answers: a key whose mint failed its sync is never launched
+ // with until a sync succeeds (settleClaimInstanceEnv). A file that
+ // needs its mode set takes the guarded path, which sets it.
+ if vars["CARLOS_ADMIN_TOKEN"] != "" && (rt.Replica || vars[InstanceKeyVar] != "") {
+ err := b.settleClaimInstanceEnv(rt)
+ if !errors.Is(err, errInstanceEnvNeedsWrite) {
+ return err
+ }
}
if authorize == nil {
return hostwrite.ErrUnavailable
@@ -1014,13 +2184,10 @@ In `internal/activator/claim_snapshot.go`, replace the body from the early retur
if err := EnsureAdminTokenBeneath(paths.dataDir, rt.Host+".env"); err != nil {
return err
}
- _, err := EnsureInstanceKeyBeneath(paths.dataDir, rt.Host+".env")
- return err
- }
- if err := EnsureAdminToken(paths.envFile); err != nil {
+ } else if err := EnsureAdminToken(paths.envFile); err != nil {
return err
}
- _, err := EnsureInstanceKey(paths.envFile)
+ _, err := b.ensureInstanceKeyFor(rt, paths)
return err
})
}
@@ -1028,58 +2195,451 @@ In `internal/activator/claim_snapshot.go`, replace the body from the early retur
Leave `SnapshotClaimLaunch` read-only and unchanged. It never mints, and existing tests pin that.
-In `internal/host/claimowners.go`, extend the `ClaimFilesManifest` doc comment (after "...starts from no manifest."):
+In `internal/host/claimowners.go`, extend the `ClaimFilesManifest` doc comment:
-```diff
-@@ -133,6 +133,17 @@ type ClaimMarkerStore interface {
- // region. Missing or malformed data is an error, never an empty-files
- // bootstrap; only an empty bootstrap (in.Empty, after EmptyAuthority) starts
- // from no manifest.
-+//
-+// The instance key (L21, plan 2026-10-10-instance-key-provision.md ruling
-+// 6): the launch source has already minted CARLOS_INSTANCE_KEY into the
-+// host's own <host>.env when Establish runs, and the app executes only
-+// after it returns. So the manifest Establish leaves must carry the env
-+// file (region HA §5.6's always-continuous platform item) with exactly the
-+// bytes on disk, staged and committed before Establish returns; an error
-+// here, a bucket that cannot be reached included, fails the wake and the
-+// app does not start. A copied manifest whose env item differs from the
-+// file on disk gets a new revision with the file's bytes; it never
-+// replaces the file on disk.
- type ClaimFilesManifest interface {
- Establish(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, work hostwrite.Work) error
- }
+```go
+// bootstrap; only an empty bootstrap (in.Empty, after EmptyAuthority) starts
+// from no manifest.
+//
+// The instance key (L21, plan 2026-10-10-instance-key-provision.md ruling
+// 6, obligation O1'): the launch source has already minted
+// CARLOS_INSTANCE_KEY into the host's own <host>.env, durably, when
+// Establish runs, and the app executes only after it returns. So the
+// manifest Establish leaves must carry two platform items with exactly the
+// bytes on disk when it runs, staged and committed before it returns: the
+// env file (region HA §5.6's always-continuous item) and the key's
+// adoption record, <host>.instance-key-adoption.json beside it
+// (instancekey.AdoptionItem; an absent file is an absent item). An error,
+// a bucket that cannot be reached included, fails the wake and the app
+// does not start. Establish never writes either file on disk.
+type ClaimFilesManifest interface {
```
- [ ] **Step 4: Run the tests, then the claim tests around them**
-Run: `go test ./internal/host/ -run 'InstanceKey|AdminToken|Rev2Shape|SnapshotClaimLaunch' -count=1`
-Expected: PASS, including the existing `TestClaimProductionAdminTokenRefusesWithoutAuthority`. Without authority no `<host>.env` is written, so no key is written either.
+Run: `nice -n 19 go test ./internal/host/ -run 'InstanceKey|AdminToken|Rev2Shape|SnapshotClaimLaunch' -count=1`
+Expected: PASS, including `TestClaimProductionAdminTokenRefusesWithoutAuthority`: without authority no `<host>.env` is written, so no key either.
-Run: `go test ./internal/activator/ -run 'Claim' -count=1`
+Run: `nice -n 19 go test ./internal/activator/ -run 'Claim|StartAdmitted' -count=1`
Expected: PASS, including `TestEnsureClaimAdminTokenInspectsSidecarBeforeAuthorize`.
- [ ] **Step 5: Commit**
```bash
gofmt -l internal/activator internal/host
-git add internal/activator/claim_snapshot.go internal/host/claimowners.go internal/host/claim_instancekey_test.go
-git commit -m "claim: the launch source mints the instance key under the guard, before Establish (L21)
+git add internal/activator/claim_snapshot.go internal/activator/claim_instancekey_test.go internal/activator/claimretry_instancekey_linux_test.go internal/host/claimowners.go internal/host/claim_instancekey_test.go
+git commit -m "claim: the launch source mints the instance key under the guard and syncs it before Establish (L21)
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 5: The adoption verb
+
+**Files:**
+- Create: `internal/activator/keyadoption.go`
+- Modify: `internal/activator/backend_exec.go:1406-1414` (`spawnChildFrom`'s mount), `internal/activator/prepare.go:143-148` (`listenControl`'s mount)
+- Test: `internal/activator/keyadoption_test.go`, `internal/activator/keyadoption_linux_test.go`
+
+**Interfaces:**
+- Consumes: Task 1's `instancekey`; Task 2's `configEnvName`, `syncDir`; `adminTokenMu`; `instancefs.WriteAtomic`, `safefile.WriteFile` (which fsyncs the file and its directory); `nestedBackend`.
+- Produces: `PUT /instance-key/adoption` on every control socket the activator binds; `(*ExecBackend).mountInstanceKeyAdoption`, `recordInstanceKeyAdoption`; the seam `adoptionRecordSync`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `internal/activator/keyadoption_test.go`:
+
+```go
+package activator
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// adoptionMux serves the adoption verb for rt on b, with the token read
+// from the host's own env file as the real listener reads it.
+func adoptionMux(b *ExecBackend, rt registry.Route) *http.ServeMux {
+ mux := http.NewServeMux()
+ b.mountInstanceKeyAdoption(mux, rt, func() (string, error) {
+ env, err := readEnvFile(filepath.Join(b.DataDir, rt.Host+".env"))
+ return env["CARLOS_ADMIN_TOKEN"], err
+ })
+ return mux
+}
+
+func putAdoption(mux *http.ServeMux, token, body string) int {
+ req := httptest.NewRequest(http.MethodPut, instanceKeyAdoptionPath, strings.NewReader(body))
+ if token != "" {
+ req.Header.Set("Authorization", "Bearer "+token)
+ }
+ w := httptest.NewRecorder()
+ mux.ServeHTTP(w, req)
+ return w.Code
+}
+
+// Finding 1, source (b): the agent records adoption only for the key in
+// the instance's own <host>.env, only for the key the app names, and only
+// with the instance's bearer. A configured key is acknowledged and not
+// recorded (source c is its evidence); a replica records nothing.
+func TestInstanceKeyAdoptionRecordsOnlyTheOwnFilesKey(t *testing.T) {
+ b := &ExecBackend{DataDir: t.TempDir()}
+ tok := strings.Repeat("f", 64)
+ key := strings.Repeat("K", 43) + "="
+ rt := registry.Route{Host: "shop.example.com", Kind: "instance", Socket: "/run/carlos/shop.sock"}
+ if err := os.WriteFile(filepath.Join(b.DataDir, rt.Host+".env"), []byte("CARLOS_ADMIN_TOKEN="+tok+"\n"+InstanceKeyVar+"="+key+"\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ mux := adoptionMux(b, rt)
+ recPath := filepath.Join(b.DataDir, rt.Host+instancekey.AdoptionSuffix)
+ body := func(k string) string { return `{"sha256":"` + instancekey.Digest(k) + `"}` }
+ for _, tc := range []struct {
+ name, token, body string
+ want int
+ }{
+ {"no bearer", "", body(key), http.StatusUnauthorized},
+ {"another host's bearer", strings.Repeat("0", 64), body(key), http.StatusUnauthorized},
+ {"not a digest", tok, `{"sha256":"abc"}`, http.StatusBadRequest},
+ {"an unknown field", tok, `{"sha256":"` + instancekey.Digest(key) + `","key":"x"}`, http.StatusBadRequest},
+ {"another key", tok, body("some-other-key"), http.StatusConflict},
+ } {
+ if got := putAdoption(mux, tc.token, tc.body); got != tc.want {
+ t.Fatalf("%s: %d, want %d", tc.name, got, tc.want)
+ }
+ if _, err := os.Stat(recPath); !os.IsNotExist(err) {
+ t.Fatalf("%s: a record was written", tc.name)
+ }
+ }
+ for i := 0; i < 2; i++ { // the app's helper calls once per process start
+ if got := putAdoption(mux, tok, body(key)); got != http.StatusNoContent {
+ t.Fatalf("adoption %d: %d, want 204", i, got)
+ }
+ rec, err := os.ReadFile(recPath)
+ if err != nil || string(rec) != string(instancekey.AdoptionRecord(key)) {
+ t.Fatalf("adoption %d: record = %q, %v", i, rec, err)
+ }
+ if st, _ := os.Stat(recPath); st.Mode().Perm() != 0o600 {
+ t.Fatalf("record mode = %o, want 600", st.Mode().Perm())
+ }
+ }
+
+ cfgHost := registry.Route{Host: "cfg.example.com", Kind: "instance", Socket: "/run/carlos/cfg.sock"}
+ if err := os.WriteFile(filepath.Join(b.DataDir, cfgHost.Host+".env"), []byte("CARLOS_ADMIN_TOKEN="+tok+"\n"+InstanceKeyVar+"=\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(b.DataDir, cfgHost.Host+".config.env"), []byte(InstanceKeyVar+"=from-config\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if got := putAdoption(adoptionMux(b, cfgHost), tok, body("from-config")); got != http.StatusNoContent {
+ t.Fatalf("configured key: %d, want 204", got)
+ }
+ if _, err := os.Stat(filepath.Join(b.DataDir, cfgHost.Host+instancekey.AdoptionSuffix)); !os.IsNotExist(err) {
+ t.Fatal("a configured key was recorded as a platform key adoption")
+ }
+
+ replica := rt
+ replica.Replica = true
+ if err := os.Remove(recPath); err != nil {
+ t.Fatal(err)
+ }
+ if got := putAdoption(adoptionMux(b, replica), tok, body(key)); got != http.StatusConflict {
+ t.Fatalf("replica: %d, want 409", got)
+ }
+ if _, err := os.Stat(recPath); !os.IsNotExist(err) {
+ t.Fatal("a replica recorded adoption")
+ }
+}
+```
+
+Create `internal/activator/keyadoption_linux_test.go`:
+
+```go
+package activator
+
+import (
+ "context"
+ "errors"
+ "net"
+ "net/http"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/instancectl"
+ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// The adoption verb is served on the nested control socket, and a nested
+// record is not acknowledged until its directory sync succeeds.
+func TestNestedControlSocketRecordsAdoptionDurably(t *testing.T) {
+ b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{})
+ _, ctl, logs, env, err := b.prepare(rt, instancectl.RoleRun)
+ if err != nil {
+ t.Fatalf("prepare: %v", err)
+ }
+ logs.Close()
+ if ctl == nil {
+ t.Fatal("no control listener")
+ }
+ defer ctl.Close()
+ client := &http.Client{Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ var d net.Dialer
+ return d.DialContext(ctx, "unix", rt.Socket+".control")
+ }}}
+ put := func() int {
+ req, _ := http.NewRequest(http.MethodPut, "http://carlos"+instanceKeyAdoptionPath, strings.NewReader(`{"sha256":"`+instancekey.Digest(env[InstanceKeyVar])+`"}`))
+ req.Header.Set("Authorization", "Bearer "+env["CARLOS_ADMIN_TOKEN"])
+ resp, err := client.Do(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ resp.Body.Close()
+ return resp.StatusCode
+ }
+ adminTokenMu.Lock()
+ old := adoptionRecordSync
+ adoptionRecordSync = func(string) error { return errors.New("injected: EIO") }
+ adminTokenMu.Unlock()
+ if code := put(); code != http.StatusInternalServerError {
+ t.Fatalf("PUT with a failing directory sync = %d, want 500", code)
+ }
+ adminTokenMu.Lock()
+ adoptionRecordSync = old
+ adminTokenMu.Unlock()
+ if code := put(); code != http.StatusNoContent {
+ t.Fatalf("PUT = %d, want 204", code)
+ }
+ rec, err := os.ReadFile(filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), rt.Host+instancekey.AdoptionSuffix))
+ if err != nil || !instancekey.Adopted(rec, env[InstanceKeyVar]) {
+ t.Fatalf("no adoption record for the instance's key beside the nested env file: %v", err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to verify they fail**
+
+Run: `nice -n 19 go test ./internal/activator/ -run 'Adoption' -count=1`
+Expected: build failure, `undefined: mountInstanceKeyAdoption` (and `instanceKeyAdoptionPath`, `adoptionRecordSync`).
+
+- [ ] **Step 3: Write the verb**
+
+Create `internal/activator/keyadoption.go`:
+
+```go
+package activator
+
+// The instance key's adoption record (instance-key plan rev 2, ruling 14).
+// A platform key in <host>.env proves only that the platform minted one: an
+// app with a key of its own keeps it (rastrillo's step 1), and its data is
+// sealed under that key, not the platform's. So relocation accepts the
+// platform key only with positive evidence that the app uses it: the app's
+// helper, having resolved $CARLOS_INSTANCE_KEY (step 2), tells the agent
+// over its control socket, and the agent records that, durably, beside the
+// env file. The files manifest publishes the record as a platform item and
+// the relocation pinner requires it (obligations O1' and O2').
+
+import (
+ "crypto/sha256"
+ "crypto/subtle"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net/http"
+ "os"
+ "path/filepath"
+
+ "github.com/carlosframework/platform/internal/instancefs"
+ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/safefile"
+)
+
+// instanceKeyAdoptionPath is the verb's path on the control socket.
+const instanceKeyAdoptionPath = "/instance-key/adoption"
+
+// mountInstanceKeyAdoption adds PUT /instance-key/adoption to an
+// instance's control socket. host is fixed by the listener, never read
+// from the request, and the bearer is the host's own CARLOS_ADMIN_TOKEN,
+// as for the schedule verbs (schedule.MountGated has the long form of what
+// that bearer does and does not buy).
+//
+// The body is {"sha256": "<hex>"}: the SHA-256 of the key the app
+// resolved. The answer:
+// - 204 when the key is the one in the instance's own <host>.env and its
+// adoption record is on disk and durable, or when the key is the one
+// the rendered configuration supplies (the configuration is that
+// key's evidence, and nothing is recorded);
+// - 409 when no effective key matches the digest (the app holds some
+// other key), or for a replica row (replicas do not adopt the platform
+// key: open question 1, provisional default). Nothing is recorded;
+// - 400 for a malformed body, 401 without the bearer, 500 when a read
+// or the durable write fails.
+//
+// Bodies are http.StatusText only: the app's helper treats every failure
+// as non-fatal and the status is all it reads.
+func (b *ExecBackend) mountInstanceKeyAdoption(mux *http.ServeMux, rt registry.Route, token func() (string, error)) {
+ fail := func(w http.ResponseWriter, code int) { http.Error(w, http.StatusText(code), code) }
+ mux.HandleFunc("PUT "+instanceKeyAdoptionPath, func(w http.ResponseWriter, r *http.Request) {
+ want, err := token()
+ if err != nil || want == "" {
+ fail(w, http.StatusUnauthorized)
+ return
+ }
+ got := sha256.Sum256([]byte(r.Header.Get("Authorization")))
+ exp := sha256.Sum256([]byte("Bearer " + want))
+ if subtle.ConstantTimeCompare(got[:], exp[:]) != 1 {
+ fail(w, http.StatusUnauthorized)
+ return
+ }
+ var body struct {
+ SHA256 string `json:"sha256"`
+ }
+ dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1024))
+ dec.DisallowUnknownFields()
+ if err := dec.Decode(&body); err != nil || dec.More() || !instancekey.ValidDigest(body.SHA256) {
+ fail(w, http.StatusBadRequest)
+ return
+ }
+ if rt.Replica {
+ fail(w, http.StatusConflict)
+ return
+ }
+ switch err := b.recordInstanceKeyAdoption(rt, body.SHA256); {
+ case errors.Is(err, errAdoptionMismatch):
+ fail(w, http.StatusConflict)
+ case err != nil:
+ fail(w, http.StatusInternalServerError)
+ default:
+ w.WriteHeader(http.StatusNoContent)
+ }
+ })
+}
+
+// errAdoptionMismatch: the digest names no effective key of this host.
+var errAdoptionMismatch = errors.New("activator: the adopted key is not this instance's effective key")
+
+// adoptionRecordSync is the record's directory sync, a seam for tests.
+// Read and replaced only under adminTokenMu.
+var adoptionRecordSync = syncDir
+
+// recordInstanceKeyAdoption checks digest against rt's effective key and,
+// when that key is the own env file's, writes the adoption record beside
+// it: atomically, 0600, the file and its directory fsynced before it
+// returns. An identical record already there is made durable and not
+// rewritten. It holds adminTokenMu, so it never reads an env file halfway
+// through a mint.
+func (b *ExecBackend) recordInstanceKeyAdoption(rt registry.Route, digest string) error {
+ p := b.pathsFor(rt)
+ beneath := p.layout == LayoutNested
+ switch p.layout {
+ case LayoutFenced, LayoutConflict:
+ return fmt.Errorf("%s: %s", rt.Host, p.reason)
+ }
+ envName, recName := rt.Host+".env", rt.Host+instancekey.AdoptionSuffix
+ read := func(n string) ([]byte, error) {
+ if beneath {
+ return instancefs.ReadBeneathLimit(p.dataDir, n, 1<<20)
+ }
+ return os.ReadFile(filepath.Join(p.dataDir, n))
+ }
+ adminTokenMu.Lock()
+ defer adminTokenMu.Unlock()
+ own, err := read(envName)
+ if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ cfg, err := read(configEnvName(envName))
+ if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ key, layer := instancekey.Effective(parseEnvBytes(cfg), parseEnvBytes(own))
+ if layer == instancekey.LayerNone || instancekey.Digest(key) != digest {
+ return errAdoptionMismatch
+ }
+ if layer == instancekey.LayerConfig {
+ return nil // the configuration is the evidence (O2' source c)
+ }
+ record := instancekey.AdoptionRecord(key)
+ if cur, err := read(recName); err == nil && string(cur) == string(record) {
+ return adoptionRecordSync(p.dataDir)
+ } else if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ if beneath {
+ // WriteAtomic fsyncs the file before its rename; the rename itself
+ // is made durable by the directory sync.
+ if err := instancefs.WriteAtomic(p.dataDir, recName, record, 0o600); err != nil {
+ return err
+ }
+ return adoptionRecordSync(p.dataDir)
+ }
+ // safefile.WriteFile fsyncs the file, renames it into place and fsyncs
+ // the directory.
+ return safefile.WriteFile(filepath.Join(p.dataDir, recName), record, 0o600)
+}
+```
+
+- [ ] **Step 4: Mount it on both control sockets**
+
+In `spawnChildFrom`, the mount is no longer conditional on `b.Timers`:
+
+```go
+ mount := func(mux *http.ServeMux) {
+ if b.Timers != nil {
+ schedule.MountGated(mux, rt.Host, b.Timers, tokenReader, time.Now, b.writeFence(rt.Host, n))
+ }
+ b.mountInstanceKeyAdoption(mux, rt, tokenReader)
+ }
+ ctl, err := edge.ListenControl(controlSocket, rt.Host, b.Cache, mount)
+```
+
+In `listenControl` (prepare.go):
+
+```go
+ mount := func(mux *http.ServeMux) {
+ if b.Timers != nil {
+ schedule.MountGated(mux, rt.Host, b.Timers, tokenReader, time.Now, b.writeFence(rt.Host, gen))
+ }
+ b.mountInstanceKeyAdoption(mux, rt, tokenReader)
+ }
+```
+
+- [ ] **Step 5: Run the tests**
+
+Run (sandbox off: the nested test dials the socket): `nice -n 19 go test ./internal/activator/ -run 'Adoption|Control|Purge|Schedule' -count=1`
+Expected: PASS, the existing control-socket and schedule-verb tests included.
+
+- [ ] **Step 6: Commit**
+
+```bash
+gofmt -l internal/activator
+git add internal/activator/keyadoption.go internal/activator/keyadoption_test.go internal/activator/keyadoption_linux_test.go internal/activator/backend_exec.go internal/activator/prepare.go
+git commit -m "activator: the control socket records an app's adoption of the platform instance key (L21)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
```
---
-### Task 4: The startup backfill
+### Task 6: The startup backfill
**Files:**
- Create: `internal/host/instancekeys.go`
-- Modify: `internal/host/agent.go:476-484` (the legacy branch's `startActivator` recovery closure)
+- Modify: `internal/host/agent.go:476-484` (the legacy branch's recovery closure)
- Test: `internal/host/instancekeys_test.go`
**Interfaces:**
-- Consumes: Task 1's `(*activator.ExecBackend).BackfillInstanceKey` and `activator.InstanceKeyOutcome` values; `registry.Route`, `registry.BackingUnit`; `(*registry.Registry).List` (registry.go:882).
+- Consumes: Task 2's `(*activator.ExecBackend).BackfillInstanceKey` and `activator.InstanceKeyOutcome` values; `registry.Route`, `registry.BackingUnit`; `(*registry.Registry).List` (registry.go:882).
- Produces: `backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) instanceKeyBackfill`; the log line `agent: instance keys: N minted, N already present, N set by app configuration, N never started here, N failed, N rows not instances`.
- [ ] **Step 1: Write the failing test**
@@ -1126,15 +2686,17 @@ func TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances(t *testing.T) {
sidecar.Kind = "sidecar"
addr := ready("addr.example.com")
addr.Socket, addr.Addr = "", "127.0.0.1:8444"
+ replica := ready("shop-au.example.com")
+ replica.Replica = true
f := &fakeKeyBackfiller{
got: map[string]activator.InstanceKeyOutcome{"shop.example.com": activator.InstanceKeyMinted, "old.example.com": activator.InstanceKeyPresent, "cfg.example.com": activator.InstanceKeyFromConfig},
err: map[string]error{"bad.example.com": errors.New("read: permission denied")},
}
- st := backfillInstanceKeys([]registry.Route{ready("shop.example.com"), ready("old.example.com"), ready("cfg.example.com"), ready("bad.example.com"), held, landing, provisioning, unit, alias, sidecar, addr}, f)
+ st := backfillInstanceKeys([]registry.Route{ready("shop.example.com"), ready("old.example.com"), ready("cfg.example.com"), ready("bad.example.com"), held, landing, provisioning, unit, alias, sidecar, addr, replica}, f)
if len(f.asked) != 4 {
t.Fatalf("asked %v, want only the four ready activator instances", f.asked)
}
- if st != (instanceKeyBackfill{Minted: 1, Present: 1, FromConfig: 1, Failed: 1, Skipped: 7}) {
+ if st != (instanceKeyBackfill{Minted: 1, Present: 1, FromConfig: 1, Failed: 1, Skipped: 8}) {
t.Fatalf("stats = %+v", st)
}
}
@@ -1142,7 +2704,7 @@ func TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances(t *testing.T) {
- [ ] **Step 2: Run it to verify it fails**
-Run: `go test ./internal/host/ -run TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances -count=1`
+Run: `nice -n 19 go test ./internal/host/ -run TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances -count=1`
Expected: build failure, `undefined: backfillInstanceKeys`.
- [ ] **Step 3: Write the backfill**
@@ -1179,11 +2741,12 @@ type instanceKeyBackfiller interface {
// instanceKeyRoute selects the rows the backfill writes for: ready,
// activator-spawned instances, the rows execSyncable delivers config to,
-// without sidecars. A held row (moving, landing, restoring, releasing)
-// belongs to its operation and is skipped, as is a provisioning row whose
-// first start will mint.
+// without sidecars and without replicas (open question 1, provisional
+// default). A held row (moving, landing, restoring, releasing) belongs to
+// its operation and is skipped, as is a provisioning row whose first start
+// will mint.
func instanceKeyRoute(rt registry.Route) bool {
- return rt.Kind == "instance" && rt.Host != "" && rt.Socket != "" && !rt.Platform &&
+ return rt.Kind == "instance" && rt.Host != "" && rt.Socket != "" && !rt.Platform && !rt.Replica &&
(rt.Status == "" || rt.Status == "ready") &&
rt.Backing != registry.BackingUnit && rt.Tunnel == "" && rt.Unit == ""
}
@@ -1245,12 +2808,12 @@ Wire it in `internal/host/agent.go`, in the legacy branch's recovery closure, be
Do not add it to the `claims.Writes.Enabled` branch (ruling 7).
-- [ ] **Step 4: Run the test, then the package and the cross-compiles**
+- [ ] **Step 4: Run the test, then the cross-compiles**
-Run: `go test ./internal/host/ -run 'TestBackfillInstanceKeys|TestStartActivator' -count=1`
+Run: `nice -n 19 go test ./internal/host/ -run 'TestBackfillInstanceKeys|TestStartActivator' -count=1`
Expected: PASS.
-Run: `go vet ./internal/host/ && GOOS=darwin go build ./internal/host/ && GOOS=windows go build ./internal/host/ && GOOS=linux GOARCH=arm64 go build ./internal/host/`
+Run: `nice -n 19 go vet ./internal/host/ && GOOS=darwin nice -n 19 go build ./internal/host/ && GOOS=windows nice -n 19 go build ./internal/host/ && GOOS=linux GOARCH=arm64 nice -n 19 go build ./internal/host/`
Expected: no output.
- [ ] **Step 5: Commit**
@@ -1265,18 +2828,17 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
---
-### Task 5: Never overwrite across re-render and moves
+### Task 7: The configured key's evidence, and never overwrite across re-render and moves
-These tests pin behaviour the code already has, which this plan now depends on (ruling 8). They should pass on first run. A failure means a premise of this plan is false: stop and report it rather than changing the code under test.
+`ConfiguredInstanceKey` is new. The other three tests pin behaviour the code already has, which this plan depends on (ruling 8). They should pass on first run. A failure means a premise of this plan is false: stop and report it rather than changing the code under test.
**Files:**
-- Create: `internal/appconfig/syncexec_instancekey_test.go`
-- Create: `internal/moves/envreport_instancekey_test.go`
-- Create: `internal/moves/archive/instancekey_test.go`
+- Create: `internal/appconfig/instancekey.go`
+- Create: `internal/appconfig/syncexec_instancekey_test.go`, `internal/moves/envreport_instancekey_test.go`, `internal/moves/archive/instancekey_test.go`
**Interfaces:**
-- Consumes: `execKit`, `SetEnv`, `ParsePrivateKey`, `SyncExec` (internal/appconfig); `Class`, `ReportEnv`, `CompareEnv` (internal/moves/envreport.go); `filesFixture`, `mustWrite`, `Build`, `Extract` (internal/moves/archive).
-- Produces: tests only.
+- Consumes: `LoadMergedHostVersions` (instances.go:222), `RenderEnv` (materialize.go:37), `routeBinding` (sync.go:469), `execKit`, `SetEnv`, `ApplyInstance`, `Seal`, `ParsePrivateKey`, `ParsePublicKey`, `ConfigEnvPath`, `SyncExec` (internal/appconfig); `Class`, `ReportEnv`, `CompareEnv` (internal/moves/envreport.go); `filesFixture`, `mustWrite`, `Build`, `Extract` (internal/moves/archive).
+- Produces: `appconfig.ConfiguredInstanceKey(ctx, s, priv, rt) (bool, error)`, for O2''s `ConfigKey`.
- [ ] **Step 1: Write the tests**
@@ -1319,6 +2881,56 @@ func TestSyncExecLeavesTheInstanceKeyAlone(t *testing.T) {
}
}
}
+
+// Finding 5, evidence (c): the pinner asks whether the host's configuration
+// renders a non-empty key. A plain variable, a sealed secret and an
+// instance overlay count; an empty value, an overlay that empties it, and
+// no configuration at all do not. The render is SyncExec's own: the file
+// it writes carries the same answer.
+func TestConfiguredInstanceKeyReadsTheRenderedConfiguration(t *testing.T) {
+ ctx, s, reg, dd, priv, pub := execKit(t)
+ pk, _ := ParsePrivateKey(priv, "test")
+ pubKey, _ := ParsePublicKey(pub, "test")
+ rt := registry.Route{Host: "shop.example.com", Kind: "instance", App: "shop", Account: "acct", Status: "ready", Socket: "/run/carlos/shop.sock"}
+ if err := reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ check := func(want bool) {
+ t.Helper()
+ got, err := ConfiguredInstanceKey(ctx, s, pk, rt)
+ if err != nil || got != want {
+ t.Fatalf("ConfiguredInstanceKey = %v, %v; want %v", got, err, want)
+ }
+ }
+ check(false) // no configuration at all
+ if err := SetEnv(ctx, s, "acct", "shop", map[string]string{"CARLOS_INSTANCE_KEY": ""}); err != nil {
+ t.Fatal(err)
+ }
+ check(false)
+ if err := SetEnv(ctx, s, "acct", "shop", map[string]string{"CARLOS_INSTANCE_KEY": "from-config"}); err != nil {
+ t.Fatal(err)
+ }
+ check(true)
+ if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }, "", nil); err != nil {
+ t.Fatal(err)
+ }
+ rendered, err := os.ReadFile(ConfigEnvPath(dd, rt))
+ if err != nil || string(rendered) != "CARLOS_INSTANCE_KEY=from-config\n" {
+ t.Fatalf("SyncExec rendered %q, %v", rendered, err)
+ }
+ if _, _, err := ApplyInstance(ctx, s, "acct", "shop", rt.Host, map[string]Var{"CARLOS_INSTANCE_KEY": {Kind: "env", Value: ""}}, nil); err != nil {
+ t.Fatal(err)
+ }
+ check(false) // the host's overlay empties it
+ sealed, err := Seal(pubKey, Binding{Account: "acct", App: "shop"}, []byte("sealed-key"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := ApplyInstance(ctx, s, "acct", "shop", rt.Host, map[string]Var{"CARLOS_INSTANCE_KEY": {Kind: "secret", Envelope: sealed}}, nil); err != nil {
+ t.Fatal(err)
+ }
+ check(true)
+}
```
`internal/moves/envreport_instancekey_test.go`:
@@ -1393,15 +3005,62 @@ func TestArchiveCarriesTheInstanceKeyByteForByte(t *testing.T) {
- [ ] **Step 2: Run them**
-Run: `go test ./internal/appconfig/ -run TestSyncExecLeavesTheInstanceKeyAlone -count=1 && go test ./internal/moves/ -run TestEnvReportComparesTheInstanceKey -count=1 && go test ./internal/moves/archive/ -run TestArchiveCarriesTheInstanceKeyByteForByte -count=1`
+Run: `nice -n 19 go test ./internal/appconfig/ -run 'TestSyncExecLeavesTheInstanceKeyAlone|TestConfiguredInstanceKey' -count=1 && nice -n 19 go test ./internal/moves/ -run TestEnvReportComparesTheInstanceKey -count=1 && nice -n 19 go test ./internal/moves/archive/ -run TestArchiveCarriesTheInstanceKeyByteForByte -count=1`
+Expected: build failure in `internal/appconfig`, `undefined: ConfiguredInstanceKey`; the moves tests PASS.
+
+- [ ] **Step 3: Write `ConfiguredInstanceKey`**
+
+Create `internal/appconfig/instancekey.go`:
+
+```go
+package appconfig
+
+import (
+ "context"
+ "crypto/ecdh"
+
+ "github.com/carlosframework/platform/internal/instancekey"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// ConfiguredInstanceKey reports whether rt's configuration in the bucket
+// renders a non-empty CARLOS_INSTANCE_KEY into <host>.config.env: the
+// app's default bundle, its bound environment and the host's instance
+// overlay, merged and rendered by the same loader and renderer SyncExec
+// and the held render use (LoadMergedHostVersions, RenderEnv). It is the
+// relocation pinner's evidence (c) for a key the app takes from its
+// configuration (instance-key plan rev 2, obligation O2'): the bucket holds
+// the configuration durably, and a relocation target renders the same one.
+//
+// The value never leaves this function. A secret is opened to tell an
+// empty value from a set one; priv may be nil only when no variable is a
+// secret, as for RenderEnv.
+func ConfiguredInstanceKey(ctx context.Context, s release.Store, priv *ecdh.PrivateKey, rt registry.Route) (bool, error) {
+ m, _, err := LoadMergedHostVersions(ctx, s, release.RouteAccount(rt), rt.App, rt.Environment, rt.Host)
+ if err != nil || m == nil {
+ return false, err
+ }
+ rendered, err := RenderEnv(m, priv, routeBinding(rt))
+ if err != nil {
+ return false, err
+ }
+ key, _ := instancekey.Effective(instancekey.ParseEnv(rendered), nil)
+ return key != "", nil
+}
+```
+
+- [ ] **Step 4: Run them again**
+
+Run: the Step 2 command.
Expected: PASS.
-- [ ] **Step 3: Commit**
+- [ ] **Step 5: Commit**
```bash
gofmt -l internal/appconfig internal/moves
-git add internal/appconfig/syncexec_instancekey_test.go internal/moves/envreport_instancekey_test.go internal/moves/archive/instancekey_test.go
-git commit -m "test: the instance key survives re-render and travels byte for byte through a move (L21)
+git add internal/appconfig/instancekey.go internal/appconfig/syncexec_instancekey_test.go internal/moves/envreport_instancekey_test.go internal/moves/archive/instancekey_test.go
+git commit -m "appconfig: answer whether a host's configuration supplies its instance key; pin re-render and moves (L21)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
```
@@ -1410,30 +3069,39 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
### Final verification (once, before handing off)
-- [ ] `go build ./... && go vet ./... && gofmt -l .` (no output)
-- [ ] `go test ./... -count=1` (all packages pass)
+- [ ] `nice -n 19 go build ./... && nice -n 19 go vet ./... && gofmt -l .` (no output)
+- [ ] `nice -n 19 go test ./... -count=1` with the sandbox off (all packages pass)
- [ ] `GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...`
-- [ ] `git grep -n 'CARLOS_INSTANCE_KEY' -- '*.go' | grep -i 'log\.\|Printf\|Errorf'`: every hit names the variable, never prints a value.
-- [ ] Push to amadan and read `amadan ci status carlos/platform -branch instance-key-provision` (exit 0) before asking for the merge.
+- [ ] `git grep -n 'CARLOS_INSTANCE_KEY\|InstanceKeyVar\|instancekey.Var' -- '*.go' | grep -i 'log\.\|Printf\|Errorf'`: every hit names the variable, never prints a value.
+- [ ] Push the branch to the hub and read its CI status on the tip (green) before asking Paul for the merge.
## Rollout
-1. **Merge and release.** Merge on amadan with `amadan branch merge -squash` after re-merging main and a green CI on the tip. Cut a platform release by the README's "Releasing the platform" recipe.
+1. **Merge and release.** Merging is Paul's call, as a squash on the hub, after CI is green on the tip. Cut a platform release by the README's "Releasing the platform" recipe.
2. **Ship to every box, every region.** Use the platform roll recipe (build arm64, upload, `update-platform-binary.sh` per box, `roll-console.sh` after). The console needs nothing new, but rolls with the release as usual.
3. **Verify the backfill on each box.** `journalctl -u carlos-edge --since '<roll time>' | grep 'agent: instance keys:'` must show `0 failed`. For each failure, read the per-host line above it, fix the file, and restart the instance (its start mints the key, or fails loudly with the same error). `never started here` hosts mint at their first start. Claim boxes print no backfill line: their instances mint at their next cold admission.
-4. **Only then tell app owners to adopt** rastrillo e3ea08d or later: `carlos.InstanceKey(own)`. Before a box has this build, an adopting app with no key of its own refuses to start on it.
-5. **App owners, before adopting:**
+4. **Post the rastrillo ask** (above) on #66, and wait for a rastrillo release that makes the adoption call.
+5. **Only then tell app owners to adopt** that rastrillo release: `carlos.InstanceKey(own)`. Before a box has this build, an adopting app with no key of its own refuses to start on it. An app on e3ea08d without the call runs, but its data does not relocate (O2' holds it `stuck`) until it moves to the release with the call.
+6. **App owners, before adopting:**
- An app with its own key passes it as `own`: its explicit setting, or the key file it already has, read without minting. It keeps that key, and nothing changes.
- An app that ran on CARLOS with a **development fallback key** moves to the platform key when it adopts. Anything sealed under the old key stops reading: sessions, sealed cookies, sign-in challenges, sealed columns. Check what is sealed under the old key first. Then either pass the old key as `own` and keep it, or accept the switch knowing what it invalidates.
- - An app with replica placements keeps its own key until open question 1 is answered: each replica host gets its own platform key.
-6. **Region HA.** Do not enable relocation on a region until this plan is rolled there and O1 (C8) and O2 (stage B pinner) are in. That is stage B Task 10's checklist item 9.
+ - An app with replica placements passes its own key as `own` (ruling 17, provisional): the platform mints no key on a replica, and an adopting app without its own key refuses to start there.
+7. **Region HA.** Do not enable relocation on a region until this plan is rolled there and O1', O2', O3' and O5 are in (stage B Task 10 checklist item 9, as O4 rewrites it).
-**Rollback:** roll the previous binary. Keys already written stay in `<host>.env`. That is harmless: no app reads them until it adopts, and the next forward roll finds them present.
+**Rollback:** roll the previous binary. Keys and adoption records already written stay. That is harmless: no app reads the key until it adopts, an older agent answers the adoption call 404 (ignored by the helper), and the next forward roll finds both present.
## Open questions for Paul
-1. **Replicas.** A replica host (#162) holds a one-way mirror of its home, but gets its own platform key. Should a replica receive its home's key instead? That needs a way to copy the key between boxes, which this plan does not build.
-2. **A key set in configuration.** This plan defers to it (ruling 5). The alternative is to reserve `CARLOS_INSTANCE_KEY`: refuse it in `carlos env set` / `secrets set` and strip it at render. That needs copy review for the refusal message.
-3. **`report.token`.** The L21 row's candidate contract also pre-provisioned `report.token`. The decision covers only the key. Should the platform provision `report.token` too?
-4. **Workers and sidecars.** Should a worker get its instance's key? Today they get none and behave as off CARLOS.
-5. **Restarts inside a claim session.** On claim boxes, a key minted by an activator restart inside a running session (not a cold admission) is published by the continuous publisher, not before the start. This only happens to an instance admitted before this build that then restarts in the same session. Accept it, or forbid minting outside the claim's launch source on claim boxes?
+Questions 1 to 4 have provisional defaults, which this plan builds on. **Paul has not yet confirmed them.**
+
+1. **Replicas.** *Provisional default:* deferred to a follow-up; replicas are excluded from platform-key adoption until then (ruling 17). The follow-up gives a replica its home's key, which needs a way to copy the key between boxes.
+2. **A key set in configuration.** *Provisional default:* configuration wins (ruling 5); the name is not reserved.
+3. **`report.token`.** *Provisional default:* not provisioned. The decision covers the key only.
+4. **Workers and sidecars.** *Provisional default:* a follow-up; out of scope here (ruling 12).
+
+Question 5 (restarts inside a claim session) is withdrawn: its premise was wrong (ruling 16).
+
+## Follow-ups this plan does not build
+
+- **Moves and the adoption record.** A move's archive does not carry `<host>.instance-key-adoption.json`, and the source's finalization does not park it (`installArtifactPaths` is a journalled list). The target re-records at the app's first start. The source keeps a stale flat record, which vouches only for the same key but is a host-named file that a later empty-installation census on that box would refuse. The fix adds the name to the moves artifact lists, with a journal version step.
+- **Configured keys and adoption.** Ruling 15's residual for (c) (an app with an unpublished own key and a configured key) closes if the pinner also requires an adoption record for configured keys. That needs the verb to record configured keys too, which publishes a digest of a configuration secret.