amadan
Instance key provision (region HA L21) implementation plan
The platform mints CARLOS_INSTANCE_KEY create-only into each exec instance's own <host>.env on every launch path, beside the admin token, backfills existing instances at agent start, and binds the C8 files-manifest provider to publish the env before the app runs. Twelve rulings, four obligations on other plans, five tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+1439
−0
docs/superpowers/plans/2026-10-10-instance-key-provision.md+1439 −0
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md| new file mode 100644 |
| index 000000000..2b63f255c |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| @@ -0,0 +1,1439 @@ |
| +# Instance Key Provision (Region HA L21) Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** The platform mints every exec-backed instance's `CARLOS_INSTANCE_KEY` on the box, create-only, into the instance's own `<host>.env` before any process of the instance starts, and backfills it into every existing instance, so an app that adopts rastrillo's `carlos.InstanceKey` never mints a key the platform cannot recover. |
| + |
| +**Architecture:** One new mint in `internal/activator` (`instancekey.go`), called right after the existing `CARLOS_ADMIN_TOKEN` mint on every launch path (flat seed, flat start, nested prepare, the claim's launch source) and holding the same lock. A startup backfill in `internal/host` appends a key to every existing `<host>.env` on boxes where host claims are off. On claim boxes the launch source mints under the claim's guard before `Materialize`, and the C8 files-manifest provider's `Establish` is bound, by contract, to publish the env file before the app runs. No flag. |
| + |
| +**Tech Stack:** Go, `crypto/rand`, `encoding/base64`, `internal/instancefs` (openat2), the registry, the claim launch source. |
| + |
| +**Spec:** |
| +- Decision: the L21 row of `docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md` on branch `region-ha-1b-next-build` (line 357; Paul, 2026-10-09). |
| +- Region HA spec `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`: H6 (line 131) and §5.6 (line 539): `instance.key`, `report.token`, `<host>.env` are always continuous. |
| +- Framework contract: rastrillo/rastrillo discussion #66 (agreed; landed on rastrillo main as e3ea08d): `carlos.InstanceKey(own string) (string, error)` resolves (1) `own`, (2) `$CARLOS_INSTANCE_KEY` verbatim, (3) on CARLOS (`carlos.Running()`, which checks `$CARLOS_CONTROL_SOCKET`) `ErrNoInstanceKey`, (4) off CARLOS `ErrNotOnCarlos`. |
| +- Stage B plan `docs/superpowers/plans/2026-10-10-region-ha-1b-next-stage-b.md` on `region-ha-1b-next-build`, ruling 40 and its "L21" section (lines 110-115), which this plan supersedes on where the key lives (see ruling 1). |
| + |
| +## Global Constraints |
| + |
| +- **Format (rastrillo #66):** 32 CSPRNG bytes, standard padded base64 (44 characters). The key is the exact bytes the env file yields: no quotes, no trailing newline in the value, no trimming. Changing the encoding later is a key rotation. |
| +- **Create-only:** a `CARLOS_INSTANCE_KEY` already in `<host>.env` is never rewritten, by any path, whatever its shape. |
| +- **Variable name:** `CARLOS_INSTANCE_KEY`, exactly. |
| +- **File mode:** `<host>.env` is 0600, as `EnsureAdminToken` leaves it (admintoken.go:104-117). |
| +- **Never logged, never in an error, never in the console.** Errors and log lines name paths, never values. |
| +- **Rollout order (rastrillo #66):** the key is on every box, existing instances included, before any app adopts the helper. |
| +- **Neutral examples:** `shop`, `example.com`, `acct`. No tenant or app names in code, tests or comments. |
| +- **Cross-compile:** `internal/host` and `internal/activator` build for darwin, windows and linux/arm64 (`20-cross`). Tests that need openat2 or `mkfifo` go in `*_linux_test.go`. |
| +- **Gate (AGENTS.md "Build & test"):** `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean. |
| +- **No user-facing copy.** Log lines are operator-facing, in the surrounding style. |
| + |
| +## What the code does today (branch `instance-key-provision`, cut from main 0d338c14) |
| + |
| +**Two things are called "overlay".** The decision's "overlay env file, alongside `$CARLOS_ADMIN_TOKEN`" is `<host>.env`, the instance's own env file: |
| +- `ExecBackend.pathsFor` names it `<dataDir>/<host>.env` beside `<host>.config.env` (internal/activator/paths.go:50-52); nested, `dataDir` is the instance directory (paths.go:36-37). |
| +- `mergeEnv` calls it the overlay: "base (the sync-owned `<host>.config.env`, bucket-derived) under overlay (the instance's own `<host>.env`). The overlay wins every conflict — `CARLOS_ADMIN_TOKEN` lives only in the instance file" (internal/activator/backend_exec.go:1974-1990). |
| +- The moves archive calls it "the `<host>.env` overlay" (internal/moves/archive/archive_test.go:36-37), and region HA's H6 names `<host>.env` as always continuous. |
| + |
| +The other overlay is appconfig's instance config layer, `apps/<acct>/<app>/instances-config/<host>.json`, written by `appconfig.ApplyInstance` (internal/appconfig/instances.go:1-24, 371-376). It is app configuration, sealed with the deployment's secrets key, rendered into `<host>.config.env` by `SyncExec` (internal/appconfig/syncexec.go:1-9; the file suffix at :57). Sync never reads, writes or reaps `<host>.env` (internal/appconfig/sync.go:270-275, pinned by `TestSyncExecNeverTouchesInstanceEnvOrOutOfScopeRoutes`, syncexec_test.go:683). The stage B plan's L21 section points at this layer (`ApplyInstance`, `syncexec.go`); that is the wrong file for this key (ruling 1). |
| + |
| +**The admin token is the precedent** (internal/activator/admintoken.go): |
| +- minted by `ensureAdminToken` (:142-238): one read of the raw bytes, `parseEnvBytes` (:245-257, which trims whitespace around keys and values), 32 `crypto/rand` bytes hex-encoded, appended with `O_APPEND|O_CREATE` at 0600 on a fresh line (:205-229), then `chmod 0600` (:234); |
| +- create-only: an existing token is never rewritten; a malformed one is logged once an hour per file (`logPerFile`, :70-77) and left in place (:189-203); |
| +- serialized by the process-wide `adminTokenMu` (:43); |
| +- nested files go through `instancefs.ReadBeneathLimit` / `OpenBeneath` (:150-216). |
| + |
| +Its callers are every launch path, each before the env is read: `seedChild` (backend_exec.go:831), `spawnChildFrom`, which flat `Start` and `StartGen` share (:1369), the nested `prepareAt` for the run and seed roles (internal/activator/prepare.go:60), the claim's `EnsureClaimAdminToken` (internal/activator/claim_snapshot.go:40-67), called by the production launch source `claimServingLaunchSource` under `claimAdminTokens.With`, the claim's guarded write (internal/host/claimproduction.go:356-428), and the scheduler tick's `scheduleToken` (internal/host/schedule_wiring.go:150-180). |
| + |
| +**Creation against first start.** The console writes the instance record. The box's reconcile mints the registry row with `Status: "ready"` and starts nothing (internal/provision/reconcile.go:425-435). The first process is started by the first wake, `doWake`: `Restore` (internal/activator/activator.go:1797), then `Seed` when the database is fresh (:1817), then `Start`. Both seed and start mint before they read the env. So every instance's first process already starts after the token mint, and after this plan's key mint too. |
| + |
| +**Backup of `<host>.env` today.** |
| +- **Legacy boxes** (host claims off; every production box today): nothing publishes `<host>.env`. `internal/hostfiles` (publisher, `StagePlatform` publisher.go:128-150, `Commit` :152) is imported by no production package; its own header says "Both are dormant: nothing in a production path calls them yet" (internal/hostfiles/contract.go:13-15). `hostfiles.EnvItem` does not exist on main; it is on `region-ha-1b-next-build` (internal/hostfiles/platform.go:8-13 there), used by stage B's release flush (internal/host/relocationflush.go:64-70 there). The file's only off-box copy is a move's sealed archive (`archive.Build`, internal/moves/archive/archive.go:395). |
| +- **Claim boxes:** the serving owners call the launch source in `Select` (internal/host/claimowners.go:375-416) and then `Materialize`, which calls `ClaimFilesManifest.Establish` "before the app executes" (claimowners.go:125-138, 425-438). No C8 provider exists on main: `Files == nil` refuses every serving wake (`claimFilesRefusal`, :140-143), and moves' install records the files manifest step as skipped "until C8 exists" (internal/host/movesinstall.go:700-712). |
| + |
| +**Moves (L16's shared install routine on main).** `installFiles` extracts the archive into staging and `placeStaged` moves the env into place only when `<host>.env` is absent or is this install's own (internal/host/moveslandinstall.go:130-280; the refusal at :239-244). `TestLandingNeverDeletesFilesItDidNotCreate/only_an_env_file` (internal/host/movesland_finalfix_test.go:85-99) pins that a landing never installs over an existing env file. The move's env report HMACs every value and classes any name it does not know as `equal` (internal/moves/envreport.go:24-38, 90-103, 117-150). |
| + |
| +## Design and rulings |
| + |
| +Each ruling ends with its cost if wrong. |
| + |
| +**1. The key lives in `<host>.env`, and is minted on the box.** This is the decision, and the code allows it: `<host>.env` is already the platform-managed per-instance secret file, H6 makes it always continuous, moves carry it, and stage B restores it (`EnvItem`). Minting in the console at create, into appconfig's instance config layer, is rejected. That layer is the app's own editable configuration (`carlos env unset` would delete the key), it exists only for instances given per-instance config, and it renders into `<host>.config.env`, which sits *under* `<host>.env` in `mergeEnv`. *Cost if wrong:* if Paul wanted the key in the bucket before the box first sees the instance, the C8 obligation in ruling 6 would be unnecessary. Moving the key later is a copy of the same bytes, not a rotation. |
| + |
| +**2. Format: `base64.StdEncoding` of 32 bytes from `crypto/rand`, 44 characters, one line `CARLOS_INSTANCE_KEY=<44 chars>\n`.** #66 says "base64" unqualified, and the standard alphabet with padding is Go's default. Every reader of `<host>.env` yields these bytes exactly: `parseEnvBytes` cuts at the first `=` and trims whitespace, which base64 never contains, and the nested launch file passes `KEY=value` strings to execve unchanged (prepare.go:204). A malformed existing key is reported, never repaired (ruling 4). *Cost if wrong:* the encoding is fixed by the first mint. Changing it later is a rotation for every adopted app. |
| + |
| +**3. Every launch path mints, right after the admin token, under the same lock.** These are `seedChild`, `spawnChildFrom`, `prepareAt` (run and seed roles) and `EnsureClaimAdminToken`. The key mint takes `adminTokenMu`, so a token append and a key append never interleave. The scheduler tick does not mint the key: its ensure exists for the token it presents, and any tick that wakes the instance goes through `Start`, which mints. Creation needs no new step: no process of an instance starts before `doWake`'s seed or start, and both mint first. A failed mint fails the wake, the same way a failed token mint does. *Cost if wrong:* if a launch path were missed, an adopting app would refuse to start there (`ErrNoInstanceKey`), loudly. The tests in Tasks 2 and 3 cover each path. |
| + |
| +**4. Create-only, append-only, durable.** A non-empty `CARLOS_INSTANCE_KEY` in `<host>.env`, of any shape, is left exactly as it is: two lines are left as two, and readers take the last, as `parseEnvBytes` does. An empty value is no key: one is appended after it, matching the token's `tok != ""`. The key is appended, never rewritten, on a fresh line. The file is set to 0600 by `fchmod` on the descriptor that wrote it. Then the file and its directory are `fsync`ed before the call returns. A key handed to a process must survive a power cut. Without the sync, a reboot could lose the line, and the next start would mint a second key over data sealed with the first. *Cost if wrong:* one fsync per instance lifetime. |
| + |
| +**5. A key from the app's configuration wins, and the platform mints none.** If the rendered `<host>.config.env` already carries a non-empty `CARLOS_INSTANCE_KEY` (the app set one with `carlos secrets set`), the mint writes nothing and logs once an hour. Minting would put a key into `<host>.env`, which overrides the configured one in `mergeEnv` and changes the app's key under its data. The configured key's durability is the bucket's. The claim's early return counts a configured key as present for the same reason. *Cost if wrong:* an app that sets the name and later unsets it gets a fresh platform key at its next start, by its own action. Reserving the name instead is open question 2. |
| + |
| +**6. Durability before the first start.** |
| +- *Legacy boxes:* nothing can publish today, and region HA cannot be on, because it requires host claims (stage B Global Constraints). The key's durability is the admin token's: the box's disk, now fsynced, and a move's sealed archive. |
| +- *Claim boxes* are region HA's only path. The launch source mints the key in `Select`. `Materialize` then calls `ClaimFilesManifest.Establish`, and the app executes only after that returns. The C8 provider is bound to obligation O1 below: the manifest `Establish` leaves carries the env item with exactly the bytes on disk, committed before it returns. |
| +- *Bucket unreachable at first start:* `Establish` fails, so the wake fails and the app does not start. The key stays on disk unchanged. The next wake publishes the same bytes, because the mint is create-only and a retry never makes a second key. |
| + |
| +This plan builds the part that is on main: the key is on disk when `Establish` runs (Task 3's test), and the obligation is written on the interface (Task 3). *Cost if wrong:* a C8 provider that ignores O1 reopens the window for every key. Stage B's pinner, amended per O2, is the backstop: such a host goes `stuck` and is never relocated without its key. |
| + |
| +**7. The backfill: the agent at startup, legacy boxes only, existing files only.** |
| +- *Who and when:* `runAgent`, inside the startup closure that runs after layout recovery has settled every journal, and before the layout pass is published and before `svc.Start` wakes anything (internal/host/agent.go:476-484). Nothing moves a file under it. |
| +- *Which rows:* ready, activator-spawned instances (`Kind == "instance"`, a socket, not `Platform`, not unit-backed, no tunnel, `Status` empty or `ready`). Held rows (`moving`, `landing`, `restoring`, `releasing`; registry.go:1164), provisioning rows, sidecars and TCP-addr routes are skipped. |
| +- *Existing files only:* a missing `<host>.env` means the host was never started here, and its first start mints the key. Creating one could occupy the path a landing's `placeStaged` needs empty. |
| +- *Idempotent:* a present key writes nothing, so every agent start re-runs it harmlessly. |
| +- *Never overwrites:* it is the same create-only body. |
| +- *Visible:* it logs one count line, which the rollout reads. |
| +- *Claim boxes are not backfilled.* Every write there runs under the claim's guard, and the claim's launch source already mints under that guard before every cold admission. |
| +- *The decision's "forced publish" is replaced by ruling 6:* every cold admission publishes the env, with its key, before the app runs. No process ever holds a platform key that its generation's manifest lacks. A running instance started before the roll has no key, and its app cannot be using one. |
| + |
| +*Cost if wrong:* a box whose registry cannot be listed at startup logs it, and its instances get keys at their next start instead. |
| + |
| +**8. Never overwrite, on every path:** |
| + |
| +| Path | What guarantees it | Test | |
| +|---|---|---| |
| +| create (first seed or start) | create-only mint | Task 1 `TestEnsureInstanceKeyMintsOnceInTheAgreedFormat`; Task 2 `TestExecBackendStartHandsTheChildTheMintedInstanceKey` (second start, same key), `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` | |
| +| backfill | same body, existing files only | Task 1 `TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile`; Task 4 `TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances` | |
| +| any existing key | create-only, any shape | Task 1 `TestEnsureInstanceKeyNeverRewritesAnExistingKey`, `...LeavesAFileWithTwoKeysAlone` | |
| +| claim launch, retried | early return; create-only | Task 3 `TestClaimLaunchMintsAKeyBesideAnExistingToken` | |
| +| re-render (config change) | sync never touches `<host>.env` | Task 5 `TestSyncExecLeavesTheInstanceKeyAlone` (also a config that sets the same name) | |
| +| restore (hibernation wake) | `Restore` restores the database only; `Start` mints create-only after it | Task 2's second start | |
| +| move | landing installs env only on an empty path (existing test); archive carries bytes exactly; env report fails a missing or different key | Task 5 `TestArchiveCarriesTheInstanceKeyByteForByte`, `TestEnvReportComparesTheInstanceKey`; existing `TestLandingNeverDeletesFilesItDidNotCreate` | |
| +| relocation install | stage B writes `EnvItem`'s bytes; the next launch finds the key present | obligation O3 (stage B branch), test code below | |
| +| layout move (flat to nested) | `<host>.env` is one of the artifacts the pass moves by name | existing layout pass tests | |
| + |
| +**9. Secrecy.** |
| +- No log line or error takes the value. The shape warning names the path only, and a test checks the value is not in the log. |
| +- `<host>.env` is 0600 (ruling 4), as the token leaves it. The nested launch file that carries the env to the trampoline is 0600 (prepare.go:204). |
| +- The console never reads a box's `<host>.env`: no console API or view carries it. |
| +- `carlos env list` reads the bucket configuration through the console API (appconfig_cli.go:462-507), never `<host>.env`, so the platform key cannot appear in it. `carlos secrets list` shows names only (appconfig_cli.go:1039). A key an app owner chose to set with plain `carlos env set` is listed like any plain variable, by their choice. |
| +- The moves env report carries HMACs only. |
| +- Once region HA publishes the env item, the key sits in the data bucket beside the database it seals, so a reader of that bucket holds both. That is inherent in making the key recoverable with the data. It is recorded here, not changed. |
| + |
| +*Cost if wrong:* a leak of one instance's key exposes only what that app sealed with it. |
| + |
| +**10. Moves and relocation carry the key inside `<host>.env`.** |
| +- A move archives the file, sealed (archive.go:395), extracts it byte for byte (extract.go:35), and places it only on an empty path (moveslandinstall.go:212-280). |
| +- If the source never had a key, the archived env has none, and the target's first start mints one. That is correct, because no source process ever held a key. |
| +- The move's env report classes `CARLOS_INSTANCE_KEY` as `equal`. A target that would start with a different key, or none, is a `differs` or `missing` difference, never a note. |
| +- A move's probe launches with the env its readiness pin checks against the source's report (internal/host/movesprobeowners.go:106-120). So the probe path never mints, and a key minted there would fail the pin. This plan adds no mint to it. |
| +- Relocation's install (L16's shared routine, stage B Task 8c) writes the pinned manifest's `EnvItem` with `writeFileSynced`, or removes the env when the item is absent. That is a restore of the host's authoritative file onto a box that holds no installation of it, so it is the one write allowed to replace `<host>.env`. The next launch's mint finds the key present (O3). |
| + |
| +*Cost if wrong:* a key lost in transit is the L21 failure itself. The env report makes it visible on moves. |
| + |
| +**11. No switch.** Nothing reads `CARLOS_INSTANCE_KEY` until an app adopts the helper, so writing it is harmless: |
| +- an app that never calls `carlos.InstanceKey` never looks at it; |
| +- an app that prints its whole environment already prints `CARLOS_ADMIN_TOKEN`; |
| +- the claim's launch identity, a digest over the env, changes once per host, as any `carlos env set` does; |
| +- the backfill appends one line to files the activator already owns. |
| + |
| +*Cost if wrong:* the mint is on every launch path, so a bug in it fails wakes across the fleet. The tests, and an error shape identical to the token mint's, are the mitigation. Rollback is reverting the binary. Keys already written stay, and are harmless. |
| + |
| +**12. Scope: exec-backed activator instances.** These do not get a platform key: |
| +- unit-backed tenants: their env is `/etc/carlos/apps/<host>.env`, root-only and unreadable to the agent (backend_unit.go:174-178); |
| +- TCP-addr routes: an app supervisor, not the agent, starts them; |
| +- sidecars and workers: they have their own rows and env files, and `StartWorkerStaged` (backend_exec.go:2104) mints no token either. |
| + |
| +The agent sets `CARLOS_CONTROL_SOCKET` only for instance children (`agentOwnedVars`), so `carlos.Running()` is false in all three, and the helper keeps today's behaviour. Replica rows (#162, `Route.Replica`) are their own hosts and get their own key: see open question 1. *Cost if wrong:* an excluded process that runs with `CARLOS_CONTROL_SOCKET` set and adopts the helper would refuse to start. That is loud, not silent. |
| + |
| +## Obligations on other plans |
| + |
| +- **O1 (the C8 files-manifest provider, `ClaimFilesManifest`):** `Establish` leaves a committed manifest whose env item has exactly the bytes of the host's `<host>.env` as they are when `Establish` runs. A copied manifest whose env item differs gets a new revision with the file's bytes. It never writes the file on disk. Any error (store unreachable, `If-Match` lost) is returned, and the wake fails before the app executes. Its test, in the provider's plan: `TestEstablishPublishesTheEnvFileWithItsInstanceKey`. A dir store, a predecessor manifest whose `host.env` item lacks the key, and a local `<host>.env` that carries it; after `Establish`, `ReadPlatform(EnvItem)` equals the local bytes. A store that refuses the PUT returns an error and the app never starts. |
| +- **O2 (stage B, ruling 25's `needKey` in `hostrelocate/pin.go:250-256` on `region-ha-1b-next-build`):** evidence of a key is a live `instance.key` entry **or** a platform `EnvItem` whose parsed env carries a non-empty `CARLOS_INSTANCE_KEY`. Without this, every app that adopts the platform key and has a nonempty database goes `stuck` on every relocation, because it never writes `instance.key`. Test: `TestStorePinnerAcceptsAPlatformKeyInTheEnvItem`. A nonempty database, a manifest with no `instance.key` and an `EnvItem` of `CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=<44>\n` pins; the same `EnvItem` without the key line refuses with `ErrFilesUnevidenced`. |
| +- **O3 (stage B Task 8c, relocation install):** add `TestRelocationInstallKeepsThePublishedInstanceKey`. P's manifest `EnvItem` carries a key. After `Install`, `<host>.env` equals the item's bytes, and `activator.EnsureInstanceKey(envPath)` returns `InstanceKeyPresent` and leaves the bytes unchanged. |
| +- **O4 (stage B Task 10 checklist item 9):** satisfied once this plan is rolled to the region's boxes and O1 and O2 are in. |
| + |
| +## Review Focus |
| + |
| +1. **A hand-edited `<host>.env` with two key lines.** The launch env takes the last line, and the mint writes nothing. Task 1, `TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone`. |
| +2. **An empty `CARLOS_INSTANCE_KEY=` line.** It is treated as absent: a key is appended, and readers take the appended one. Task 1, `TestEnsureInstanceKeyTreatsAnEmptyValueAsAbsent`. |
| +3. **A visitor's wake, a seed and the startup backfill racing on one fresh host.** Exactly one key is written. Task 1, `TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers`. |
| +4. **A FIFO planted at a nested `<host>.env`.** The startup backfill must fail fast, never block agent start. Task 2, `TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking`. |
| +5. **An empty `CARLOS_INSTANCE_KEY=` in the app's configuration.** It supplies nothing, so the platform mints one. Task 1, `TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty`. |
| + |
| +## File structure |
| + |
| +- Create `internal/activator/instancekey.go`: the mint (`EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `(*ExecBackend).BackfillInstanceKey`, `InstanceKeyVar`, `InstanceKeyOutcome`). One responsibility: the key line in `<host>.env`. |
| +- Create `internal/activator/instancekey_test.go` and `internal/activator/instancekey_linux_test.go`. |
| +- Modify `internal/activator/backend_exec.go` (`seedChild` :831, `spawnChildFrom` :1369), `internal/activator/prepare.go` (:60), `internal/activator/claim_snapshot.go` (:40-67): call the mint. |
| +- Modify `internal/host/claimowners.go` (:125-138): the obligation on `ClaimFilesManifest`. Create `internal/host/claim_instancekey_test.go`. |
| +- Create `internal/host/instancekeys.go` and `internal/host/instancekeys_test.go`: the backfill. Modify `internal/host/agent.go` (:476-484): call it. |
| +- Create `internal/appconfig/syncexec_instancekey_test.go`, `internal/moves/envreport_instancekey_test.go`, `internal/moves/archive/instancekey_test.go`: never-overwrite across re-render and moves. |
| + |
| +**Sandbox note for every command below:** if the toolchain reports "read-only file system" for the Go cache, prefix commands with `GOCACHE=$TMPDIR/gocache` (re-export per command), and with `GOFLAGS=-buildvcs=false` in a worktree. On a shared machine run them under `nice -n 19`. |
| + |
| +--- |
| + |
| +### Task 1: The mint |
| + |
| +**Files:** |
| +- Create: `internal/activator/instancekey.go` |
| +- Test: `internal/activator/instancekey_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `adminTokenMu` (admintoken.go:43), `logPerFile` (:70), `parseEnvBytes` (:245), `instancefs.ReadBeneathLimit`, `instancefs.OpenBeneath`, `(*ExecBackend).pathsFor` (paths.go:33), `Layout*` constants (layout.go:21-26). |
| +- Produces: |
| + - `const InstanceKeyVar = "CARLOS_INSTANCE_KEY"` |
| + - `type InstanceKeyOutcome int` with `InstanceKeyPresent`, `InstanceKeyMinted`, `InstanceKeyFromConfig`, `InstanceKeyNoEnvFile` |
| + - `func EnsureInstanceKey(path string) (InstanceKeyOutcome, error)` |
| + - `func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error)` |
| + - `func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error)` |
| + - `func wellFormedInstanceKey(key string) bool` (package-private, used by later tests) |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Create `internal/activator/instancekey_test.go`: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "bytes" |
| + "encoding/base64" |
| + "log" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "sync" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// keyLines returns every CARLOS_INSTANCE_KEY value in raw, in order. |
| +func keyLines(raw []byte) []string { |
| + var out []string |
| + for _, line := range strings.Split(string(raw), "\n") { |
| + if v, ok := strings.CutPrefix(line, InstanceKeyVar+"="); ok { |
| + out = append(out, v) |
| + } |
| + } |
| + return out |
| +} |
| + |
| +func TestEnsureInstanceKeyMintsOnceInTheAgreedFormat(t *testing.T) { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + got, err := EnsureInstanceKey(p) |
| + if err != nil || got != InstanceKeyMinted { |
| + t.Fatalf("first ensure = %v, %v; want minted", got, err) |
| + } |
| + raw, _ := os.ReadFile(p) |
| + keys := keyLines(raw) |
| + if len(keys) != 1 { |
| + t.Fatalf("file holds %d keys, want 1", len(keys)) |
| + } |
| + dec, err := base64.StdEncoding.Strict().DecodeString(keys[0]) |
| + if err != nil || len(dec) != 32 || len(keys[0]) != 44 { |
| + t.Fatalf("key is not 32 bytes in padded standard base64 (len %d, decode err %v)", len(keys[0]), err) |
| + } |
| + if !bytes.HasSuffix(raw, []byte("\n")) { |
| + t.Fatal("the key line has no newline: the next append would join it") |
| + } |
| + if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 { |
| + t.Fatalf("mode = %o, want 600", st.Mode().Perm()) |
| + } |
| + // The parser every launch path uses yields the exact characters minted. |
| + env, _ := readEnvFile(p) |
| + if env[InstanceKeyVar] != keys[0] { |
| + t.Fatal("readEnvFile does not yield the minted key verbatim") |
| + } |
| + again, err := EnsureInstanceKey(p) |
| + if err != nil || again != InstanceKeyPresent { |
| + t.Fatalf("second ensure = %v, %v; want present", again, err) |
| + } |
| + raw2, _ := os.ReadFile(p) |
| + if !bytes.Equal(raw, raw2) { |
| + t.Fatal("second ensure changed the file") |
| + } |
| +} |
| + |
| +// Never overwrite: a key already there, of any shape, is kept byte for byte |
| +// and only reported. |
| +func TestEnsureInstanceKeyNeverRewritesAnExistingKey(t *testing.T) { |
| + for _, existing := range []string{ |
| + "CARLOS_INSTANCE_KEY=hand-written-not-base64\n", |
| + "A=1\nCARLOS_INSTANCE_KEY=" + strings.Repeat("A", 43) + "=", // well formed, no trailing newline |
| + } { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + if err := os.WriteFile(p, []byte(existing), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + var buf bytes.Buffer |
| + log.SetOutput(&buf) |
| + got, err := EnsureInstanceKey(p) |
| + log.SetOutput(os.Stderr) |
| + if err != nil || got != InstanceKeyPresent { |
| + t.Fatalf("ensure over %q = %v, %v; want present", existing, got, err) |
| + } |
| + if raw, _ := os.ReadFile(p); string(raw) != existing { |
| + t.Fatalf("file changed: %q -> %q", existing, raw) |
| + } |
| + if strings.Contains(buf.String(), "hand-written-not-base64") { |
| + t.Fatal("the shape warning logged the key's value") |
| + } |
| + } |
| +} |
| + |
| +func TestEnsureInstanceKeyAppendsOnAFreshLineAndKeepsEveryOtherLine(t *testing.T) { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + before := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("a", 64) + "\n# comment\nDATABASE_URL=sqlite" // no trailing newline |
| + if err := os.WriteFile(p, []byte(before), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted { |
| + t.Fatalf("ensure = %v, %v", got, err) |
| + } |
| + raw, _ := os.ReadFile(p) |
| + if !strings.HasPrefix(string(raw), before+"\n"+InstanceKeyVar+"=") { |
| + t.Fatalf("file = %q: the key must land on its own line after the untouched original", raw) |
| + } |
| + env, _ := readEnvFile(p) |
| + if env["DATABASE_URL"] != "sqlite" || len(env["CARLOS_ADMIN_TOKEN"]) != 64 { |
| + t.Fatal("an existing line was lost") |
| + } |
| + if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 { |
| + t.Fatalf("mode = %o, want the 0644 file tightened to 600", st.Mode().Perm()) |
| + } |
| +} |
| + |
| +// A key the app's configuration supplies is the app's key: minting one into |
| +// <host>.env would override it (mergeEnv) and change the key under its data. |
| +func TestEnsureInstanceKeyDefersToTheRenderedConfiguration(t *testing.T) { |
| + dir := t.TempDir() |
| + own := filepath.Join(dir, "shop.example.com.env") |
| + if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=from-config\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(own, []byte("A=1\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := EnsureInstanceKey(own) |
| + if err != nil || got != InstanceKeyFromConfig { |
| + t.Fatalf("ensure = %v, %v; want from-config", got, err) |
| + } |
| + if raw, _ := os.ReadFile(own); string(raw) != "A=1\n" { |
| + t.Fatalf("own file changed: %q", raw) |
| + } |
| +} |
| + |
| +func TestBackfillInstanceKeyAppendsOnlyToAnExistingEnvFile(t *testing.T) { |
| + b := &ExecBackend{DataDir: t.TempDir()} |
| + fresh := registry.Route{Host: "new.example.com", Socket: "/run/carlos/new.sock"} |
| + got, err := b.BackfillInstanceKey(fresh) |
| + if err != nil || got != InstanceKeyNoEnvFile { |
| + t.Fatalf("backfill with no env file = %v, %v; want no-env-file", got, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(b.DataDir, fresh.Host+".env")); !os.IsNotExist(err) { |
| + t.Fatalf("backfill created an env file: %v", err) |
| + } |
| + old := registry.Route{Host: "shop.example.com", Socket: "/run/carlos/shop.sock"} |
| + envPath := filepath.Join(b.DataDir, old.Host+".env") |
| + tok := "CARLOS_ADMIN_TOKEN=" + strings.Repeat("b", 64) + "\n" |
| + if err := os.WriteFile(envPath, []byte(tok), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyMinted { |
| + t.Fatalf("backfill = %v, %v; want minted", got, err) |
| + } |
| + raw, _ := os.ReadFile(envPath) |
| + if !strings.HasPrefix(string(raw), tok) || len(keyLines(raw)) != 1 { |
| + t.Fatalf("file = %q", raw) |
| + } |
| + // Idempotent: a second backfill (the next agent start) writes nothing. |
| + if got, err := b.BackfillInstanceKey(old); err != nil || got != InstanceKeyPresent { |
| + t.Fatalf("second backfill = %v, %v; want present", got, err) |
| + } |
| + if raw2, _ := os.ReadFile(envPath); !bytes.Equal(raw, raw2) { |
| + t.Fatal("second backfill changed the file") |
| + } |
| +} |
| + |
| +// Review focus 1: a hand-edited file with two keys. Readers take the last |
| +// (parseEnvBytes), and the ensure writes nothing. |
| +func TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone(t *testing.T) { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + body := InstanceKeyVar + "=first\n" + InstanceKeyVar + "=second\n" |
| + if err := os.WriteFile(p, []byte(body), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyPresent { |
| + t.Fatalf("ensure = %v, %v; want present", got, err) |
| + } |
| + if raw, _ := os.ReadFile(p); string(raw) != body { |
| + t.Fatalf("file changed: %q", raw) |
| + } |
| + if env, _ := readEnvFile(p); env[InstanceKeyVar] != "second" { |
| + t.Fatalf("readers see %q, want the last line's value", env[InstanceKeyVar]) |
| + } |
| +} |
| + |
| +// Review focus 2: an empty value is no key. One is appended after it, and |
| +// readers take the appended one. |
| +func TestEnsureInstanceKeyTreatsAnEmptyValueAsAbsent(t *testing.T) { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + if err := os.WriteFile(p, []byte(InstanceKeyVar+"=\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := EnsureInstanceKey(p); err != nil || got != InstanceKeyMinted { |
| + t.Fatalf("ensure = %v, %v; want minted", got, err) |
| + } |
| + env, _ := readEnvFile(p) |
| + if !wellFormedInstanceKey(env[InstanceKeyVar]) { |
| + t.Fatal("readers do not see the appended key") |
| + } |
| +} |
| + |
| +// Review focus 3: a seed, a visitor's wake and the backfill can ask at |
| +// once. Exactly one key is written. |
| +func TestEnsureInstanceKeyMintsOneKeyUnderConcurrentCallers(t *testing.T) { |
| + p := filepath.Join(t.TempDir(), "shop.example.com.env") |
| + var wg sync.WaitGroup |
| + for i := 0; i < 8; i++ { |
| + wg.Add(1) |
| + go func() { |
| + defer wg.Done() |
| + if _, err := EnsureInstanceKey(p); err != nil { |
| + t.Error(err) |
| + } |
| + }() |
| + } |
| + wg.Wait() |
| + if raw, _ := os.ReadFile(p); len(keyLines(raw)) != 1 { |
| + t.Fatalf("%d keys written", len(keyLines(raw))) |
| + } |
| +} |
| + |
| +// Review focus 5: an empty CARLOS_INSTANCE_KEY in the rendered |
| +// configuration supplies nothing, so the platform mints one. |
| +func TestEnsureInstanceKeyMintsWhenTheConfigurationValueIsEmpty(t *testing.T) { |
| + dir := t.TempDir() |
| + if err := os.WriteFile(filepath.Join(dir, "shop.example.com.config.env"), []byte(InstanceKeyVar+"=\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := EnsureInstanceKey(filepath.Join(dir, "shop.example.com.env")); err != nil || got != InstanceKeyMinted { |
| + t.Fatalf("ensure = %v, %v; want minted", got, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey' -count=1` |
| +Expected: build failure, `undefined: EnsureInstanceKey` (and `InstanceKeyVar`, `InstanceKeyMinted`, ...). |
| + |
| +- [ ] **Step 3: Write the implementation** |
| + |
| +Create `internal/activator/instancekey.go`: |
| + |
| +```go |
| +package activator |
| + |
| +// The instance key (region HA request L21; rastrillo/rastrillo #66). An app |
| +// on CARLOS seals its data with a key. Before this file the app minted that |
| +// key itself after it started, so a box that died before the key reached a |
| +// backup left a database nothing could unseal. The platform now mints the |
| +// key on the box, into the instance's own <host>.env as |
| +// CARLOS_INSTANCE_KEY, before the app's first start, and never rewrites it. |
| +// rastrillo's carlos.InstanceKey reads it verbatim when the app has no key |
| +// of its own. |
| +// |
| +// The rules, each a ruling in |
| +// docs/superpowers/plans/2026-10-10-instance-key-provision.md: |
| +// - create-only: a key already in the file is never rewritten, whatever |
| +// its shape. A malformed one is reported and left in place, as |
| +// EnsureAdminToken does with a malformed token; |
| +// - the rendered configuration wins: when <host>.config.env already |
| +// supplies CARLOS_INSTANCE_KEY, nothing is minted, because minting one |
| +// into <host>.env would override it (mergeEnv) and change the app's key; |
| +// - the value is never logged, never put in an error, and never leaves |
| +// the file except in the launch environment; |
| +// - the file ends up 0600, set on the descriptor that wrote it, and the |
| +// write is fsynced before the call returns, so a key handed to a |
| +// process cannot vanish in a power cut. |
| + |
| +import ( |
| + "crypto/rand" |
| + "encoding/base64" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/instancefs" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// InstanceKeyVar is the variable the key is delivered as. |
| +const InstanceKeyVar = "CARLOS_INSTANCE_KEY" |
| + |
| +// instanceKeyBytes is the key's length before encoding: 32 CSPRNG bytes, |
| +// standard base64 with padding, 44 characters. rastrillo uses the exact |
| +// bytes the env file yields, so this encoding is fixed for the life of |
| +// every key minted with it; changing it is a rotation. |
| +const instanceKeyBytes = 32 |
| + |
| +// InstanceKeyOutcome is what one ensure did. |
| +type InstanceKeyOutcome int |
| + |
| +const ( |
| + // InstanceKeyPresent: the own env file already carried a key. Nothing |
| + // was written. |
| + InstanceKeyPresent InstanceKeyOutcome = iota |
| + // InstanceKeyMinted: a key was appended. |
| + InstanceKeyMinted |
| + // InstanceKeyFromConfig: the rendered configuration supplies the key. |
| + // Nothing was written. |
| + InstanceKeyFromConfig |
| + // InstanceKeyNoEnvFile: the backfill found no own env file, so the host |
| + // has never been started here. Nothing was written; its first start |
| + // mints the key. |
| + InstanceKeyNoEnvFile |
| +) |
| + |
| +func (o InstanceKeyOutcome) String() string { |
| + switch o { |
| + case InstanceKeyPresent: |
| + return "present" |
| + case InstanceKeyMinted: |
| + return "minted" |
| + case InstanceKeyFromConfig: |
| + return "from-config" |
| + case InstanceKeyNoEnvFile: |
| + return "no-env-file" |
| + } |
| + return fmt.Sprintf("outcome(%d)", int(o)) |
| +} |
| + |
| +func mintInstanceKey() (string, error) { |
| + raw := make([]byte, instanceKeyBytes) |
| + if _, err := rand.Read(raw); err != nil { |
| + return "", err |
| + } |
| + return base64.StdEncoding.EncodeToString(raw), nil |
| +} |
| + |
| +// wellFormedInstanceKey reports whether key has the shape this file mints. |
| +// A key that fails it is not necessarily broken (an operator may have |
| +// written one by hand), which is why a failure is a warning, never a |
| +// rotation. |
| +func wellFormedInstanceKey(key string) bool { |
| + if len(key) != base64.StdEncoding.EncodedLen(instanceKeyBytes) { |
| + return false |
| + } |
| + raw, err := base64.StdEncoding.Strict().DecodeString(key) |
| + return err == nil && len(raw) == instanceKeyBytes |
| +} |
| + |
| +// configEnvName is the rendered configuration's file name beside an own env |
| +// file: <host>.env's sibling <host>.config.env, the pair pathsFor names. |
| +func configEnvName(envName string) string { |
| + return strings.TrimSuffix(envName, ".env") + ".config.env" |
| +} |
| + |
| +// EnsureInstanceKey makes sure path (an instance's own <host>.env on the |
| +// flat layout) carries CARLOS_INSTANCE_KEY, creating the file if needed. |
| +// Launch paths call it right after EnsureAdminToken. |
| +func EnsureInstanceKey(path string) (InstanceKeyOutcome, error) { |
| + return ensureInstanceKey(filepath.Dir(path), filepath.Base(path), false, true) |
| +} |
| + |
| +// EnsureInstanceKeyBeneath is EnsureInstanceKey for a nested row's own env |
| +// file, read and written through instancefs because the directory is |
| +// tenant-writable (isolated-exec design §3.2). |
| +func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) { |
| + return ensureInstanceKey(dir, name, true, true) |
| +} |
| + |
| +// BackfillInstanceKey is the startup backfill's step for one row: a key is |
| +// appended only to an own env file that already exists. A host with no file |
| +// has never been started on this box, and its first start mints the key; |
| +// creating a file here could also occupy the path a move's install expects |
| +// to be empty (moveslandinstall.go placeStaged). A row this backend would |
| +// refuse to start (fenced, conflict) is an error and nothing is written. |
| +func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error) { |
| + p := b.pathsFor(rt) |
| + switch p.layout { |
| + case LayoutNested: |
| + return ensureInstanceKey(p.dataDir, rt.Host+".env", true, false) |
| + case LayoutFenced, LayoutConflict: |
| + return InstanceKeyPresent, fmt.Errorf("%s: not backfilled: %s", rt.Host, p.reason) |
| + } |
| + return ensureInstanceKey(p.dataDir, rt.Host+".env", false, false) |
| +} |
| + |
| +// ensureInstanceKey is the shared body. It holds adminTokenMu, the lock |
| +// every mint into <host>.env takes, so a key append and a token append can |
| +// never interleave. create false is the backfill: an absent file is |
| +// InstanceKeyNoEnvFile and nothing is created. |
| +func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutcome, error) { |
| + path := filepath.Join(dir, name) |
| + adminTokenMu.Lock() |
| + defer adminTokenMu.Unlock() |
| + read := func(n string) ([]byte, error) { |
| + if beneath { |
| + return instancefs.ReadBeneathLimit(dir, n, 1<<20) |
| + } |
| + return os.ReadFile(filepath.Join(dir, n)) |
| + } |
| + raw, err := read(name) |
| + switch { |
| + case errors.Is(err, os.ErrNotExist): |
| + if !create { |
| + return InstanceKeyNoEnvFile, nil |
| + } |
| + case err != nil: |
| + return InstanceKeyPresent, fmt.Errorf("read %s: %w", path, err) |
| + } |
| + if key := parseEnvBytes(raw)[InstanceKeyVar]; key != "" { |
| + if !wellFormedInstanceKey(key) { |
| + logPerFile("key-shape", path, fmt.Sprintf("activator: %s in %s is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it", InstanceKeyVar, path)) |
| + } |
| + return InstanceKeyPresent, nil |
| + } |
| + cfg, err := read(configEnvName(name)) |
| + switch { |
| + case err == nil: |
| + if parseEnvBytes(cfg)[InstanceKeyVar] != "" { |
| + logPerFile("key-config", path, fmt.Sprintf("activator: %s is set by the app's configuration for %s: the platform mints none while it is", InstanceKeyVar, path)) |
| + return InstanceKeyFromConfig, nil |
| + } |
| + case !errors.Is(err, os.ErrNotExist): |
| + return InstanceKeyPresent, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err) |
| + } |
| + key, err := mintInstanceKey() |
| + if err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("mint instance key for %s: %w", path, err) |
| + } |
| + flags := os.O_APPEND | os.O_WRONLY |
| + if create { |
| + flags |= os.O_CREATE |
| + } |
| + var f *os.File |
| + if beneath { |
| + f, err = instancefs.OpenBeneath(dir, name, flags, 0o600) |
| + } else { |
| + f, err = os.OpenFile(path, flags, 0o600) |
| + } |
| + if errors.Is(err, os.ErrNotExist) && !create { |
| + return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill |
| + } |
| + if err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err) |
| + } |
| + defer f.Close() |
| + prefix := "" |
| + if len(raw) > 0 && raw[len(raw)-1] != '\n' { |
| + prefix = "\n" |
| + } |
| + if _, err := fmt.Fprintf(f, "%s%s=%s\n", prefix, InstanceKeyVar, key); err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("write %s: %w", path, err) |
| + } |
| + if err := f.Chmod(0o600); err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("chmod %s: %w", path, err) |
| + } |
| + if err := f.Sync(); err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("sync %s: %w", path, err) |
| + } |
| + // The directory entry too: the token mint just before this one may |
| + // have created the file without syncing its directory, and a key a |
| + // process was handed must still be here after a power cut. |
| + d, err := os.Open(dir) |
| + if err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("open %s: %w", dir, err) |
| + } |
| + syncErr := d.Sync() |
| + d.Close() |
| + if syncErr != nil { |
| + return InstanceKeyPresent, fmt.Errorf("sync %s: %w", dir, syncErr) |
| + } |
| + return InstanceKeyMinted, nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests to verify they pass** |
| + |
| +Run: `go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestEnsureAdminToken' -count=1` |
| +Expected: PASS. The admin token tests are in the run because the key mint shares their lock and log throttle. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/activator |
| +git add internal/activator/instancekey.go internal/activator/instancekey_test.go |
| +git commit -m "activator: mint CARLOS_INSTANCE_KEY create-only into <host>.env (L21) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: Every activator launch path mints the key |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go:831-833` (`seedChild`), `:1369-1371` (`spawnChildFrom`) |
| +- Modify: `internal/activator/prepare.go:60-62` (`prepareAt`) |
| +- Test: `internal/activator/instancekey_test.go` (append), `internal/activator/instancekey_linux_test.go` (create) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `EnsureInstanceKey`, `EnsureInstanceKeyBeneath`, `BackfillInstanceKey`, `wellFormedInstanceKey`, `InstanceKeyVar`; test helpers `testExecBackend` (activator_test.go:4262), `stageRecordingLiveFullEnv` (:4712), `waitForFile` (:4741), `tmpSock` (:954), `nestedBackend`, `fakeControl`, `fakeUnit` (unitproc_test.go:24-160). |
| +- Produces: nothing new; the launch env of every exec start and seed carries `CARLOS_INSTANCE_KEY`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `internal/activator/instancekey_test.go`, and add `"context"` and `"time"` to its imports: |
| + |
| +```go |
| +// Start hands the child the key it minted, and a second Start (a restart, a |
| +// redeploy) hands it the same one. |
| +func TestExecBackendStartHandsTheChildTheMintedInstanceKey(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "shop.example.com" |
| + start := func() map[string]string { |
| + _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1") |
| + rt := registry.Route{Host: host, Kind: "instance", Socket: tmpSock(t)} |
| + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| + defer cancel() |
| + proc, _, _ := b.Start(ctx, rt) // the fake binary never binds: the health wait fails, as in the sibling tests |
| + child := map[string]string{} |
| + for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") { |
| + if k, v, ok := strings.Cut(line, "="); ok { |
| + child[k] = v |
| + } |
| + } |
| + if proc != nil { |
| + proc.Stop(context.Background()) |
| + } |
| + return child |
| + } |
| + first := start() |
| + minted, err := readEnvFile(filepath.Join(b.DataDir, host+".env")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !wellFormedInstanceKey(minted[InstanceKeyVar]) || first[InstanceKeyVar] != minted[InstanceKeyVar] { |
| + t.Fatal("the child was not handed the key minted into <host>.env") |
| + } |
| + if second := start(); second[InstanceKeyVar] != minted[InstanceKeyVar] { |
| + t.Fatal("a second start handed the child a different key") |
| + } |
| +} |
| +``` |
| + |
| +Create `internal/activator/instancekey_linux_test.go`: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + "time" |
| + |
| + "golang.org/x/sys/unix" |
| + |
| + "github.com/carlosframework/platform/internal/instancectl" |
| + "github.com/carlosframework/platform/internal/provision" |
| +) |
| + |
| +// The nested prepare mints the key into the instance directory's own env |
| +// file and hands it to the role in the same env it returns. |
| +func TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile(t *testing.T) { |
| + b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| + _, ctl, logs, env, err := b.prepare(rt, instancectl.RoleRun) |
| + if err != nil { |
| + t.Fatalf("prepare: %v", err) |
| + } |
| + logs.Close() |
| + if ctl != nil { |
| + ctl.Close() |
| + } |
| + own, err := readEnvFileBeneath(provision.InstanceDir(b.DataDir, rt.Host), rt.Host+".env") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !wellFormedInstanceKey(own[InstanceKeyVar]) || env[InstanceKeyVar] != own[InstanceKeyVar] { |
| + t.Fatal("prepare's env does not carry the key minted into the nested env file") |
| + } |
| +} |
| + |
| +// A nested own env file is on tenant-writable ground: a symlink planted at |
| +// its name is refused, and the file it points at is not touched. |
| +func TestEnsureInstanceKeyBeneathRefusesASymlink(t *testing.T) { |
| + dir := t.TempDir() |
| + target := filepath.Join(t.TempDir(), "elsewhere.env") |
| + if err := os.WriteFile(target, []byte("X=1\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(target, filepath.Join(dir, "shop.example.com.env")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := EnsureInstanceKeyBeneath(dir, "shop.example.com.env"); err == nil { |
| + t.Fatal("ensure followed a symlink planted at the env file's name") |
| + } |
| + if raw, _ := os.ReadFile(target); string(raw) != "X=1\n" { |
| + t.Fatalf("the symlink's target changed: %q", raw) |
| + } |
| +} |
| + |
| +// Review focus 4: a FIFO planted at a nested env file's name fails the |
| +// backfill fast; it never blocks agent start. |
| +func TestBackfillInstanceKeyRefusesAFIFOWithoutBlocking(t *testing.T) { |
| + b, rt := nestedBackend(t, &fakeControl{inv: "inv1"}, &fakeUnit{}) |
| + dir := provision.InstanceDir(b.DataDir, rt.Host) |
| + if err := os.MkdirAll(dir, 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := unix.Mkfifo(filepath.Join(dir, rt.Host+".env"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + done := make(chan error, 1) |
| + go func() { |
| + _, err := b.BackfillInstanceKey(rt) |
| + done <- err |
| + }() |
| + select { |
| + case err := <-done: |
| + if err == nil { |
| + t.Fatal("backfill over a FIFO succeeded, want an error") |
| + } |
| + case <-time.After(2 * time.Second): |
| + t.Fatal("backfill blocked on a FIFO") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `go test ./internal/activator/ -run 'TestExecBackendStartHandsTheChildTheMintedInstanceKey|TestPrepareMintsTheInstanceKey|TestEnsureInstanceKeyBeneathRefusesASymlink|TestBackfillInstanceKeyRefusesAFIFO' -count=1` |
| +Expected: `TestExecBackendStartHandsTheChildTheMintedInstanceKey` and `TestPrepareMintsTheInstanceKeyIntoTheNestedEnvFile` FAIL (no key in the child's env or the nested file). The symlink and FIFO tests already pass, because Task 1's body uses instancefs. They are here to pin the nested path. |
| + |
| +- [ ] **Step 3: Call the mint on each launch path** |
| + |
| +In `internal/activator/backend_exec.go`, `seedChild`, after the admin token mint: |
| + |
| +```go |
| + if err := EnsureAdminToken(paths.envFile); err != nil { |
| + return fmt.Errorf("admin token for %s: %w", host, err) |
| + } |
| + // The instance key, before the env is read for the same reason: a seed |
| + // is usually the first process of a fresh instance (L21). |
| + if _, err := EnsureInstanceKey(paths.envFile); err != nil { |
| + return fmt.Errorf("instance key for %s: %w", host, err) |
| + } |
| + fileEnv, err := b.instanceEnv(rt) |
| +``` |
| + |
| +In `spawnChildFrom`, after the admin token mint: |
| + |
| +```go |
| + if err := EnsureAdminToken(paths.envFile); err != nil { |
| + return nil, "", fmt.Errorf("admin token for %s: %w", host, err) |
| + } |
| + // The instance key is minted here for the same reason and fails the |
| + // wake the same way: an app that takes its key from the platform |
| + // refuses to start without one (L21). |
| + if _, err := EnsureInstanceKey(paths.envFile); err != nil { |
| + return nil, "", fmt.Errorf("instance key for %s: %w", host, err) |
| + } |
| +``` |
| + |
| +In `internal/activator/prepare.go`, `prepareAt`, after `EnsureAdminTokenBeneath`: |
| + |
| +```go |
| + if err := EnsureAdminTokenBeneath(p.dataDir, rt.Host+".env"); err != nil { |
| + return p, nil, nil, nil, fmt.Errorf("admin token for %s: %w", rt.Host, err) |
| + } |
| + if _, err := EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env"); err != nil { |
| + return p, nil, nil, nil, fmt.Errorf("instance key for %s: %w", rt.Host, err) |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests, then the whole package** |
| + |
| +Run: `go test ./internal/activator/ -run 'InstanceKey' -count=1` |
| +Expected: PASS. |
| + |
| +Run: `go test ./internal/activator/ -count=1` |
| +Expected: PASS. If an existing test compares a child's whole environment or an env file's exact bytes after a start or seed, it now sees one more line, `CARLOS_INSTANCE_KEY=...`. Update that assertion to allow the key line (assert it is well formed). Do not weaken any other check, and list each test you changed in the commit message. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/activator |
| +git add internal/activator/backend_exec.go internal/activator/prepare.go internal/activator/instancekey_test.go internal/activator/instancekey_linux_test.go |
| +git commit -m "activator: seed, start and nested prepare mint the instance key before the env is read (L21) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: The claim's launch source mints the key under the guard, before the files manifest |
| + |
| +**Files:** |
| +- Modify: `internal/activator/claim_snapshot.go:40-67` (`EnsureClaimAdminToken`) |
| +- Modify: `internal/host/claimowners.go:125-138` (doc on `ClaimFilesManifest`) |
| +- Test: `internal/host/claim_instancekey_test.go` (create) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's mint; `claimServingLaunchSource` (claimproduction.go:421), `claimAdminTokens` (:368), `claimServingOwners` (claimowners.go:293) with `Select` (:375) and `Materialize` (:425); test fixture `newAdminTokenFixture`, `adminTokenHost`, `f.envPath()`, `f.token()` (claimproduction_admintoken_test.go:20-118); `leasetest.NewVersionedBucket`. |
| +- Produces: `EnsureClaimAdminToken` initializes both the token and the key, inside the one authorized write. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Create `internal/host/claim_instancekey_test.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "os" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/lease/leasetest" |
| +) |
| + |
| +// keyCapturingFiles is a C8 provider that records the instance's own env |
| +// file as it stands when Establish runs: the moment the provider must |
| +// publish it (ruling 6). |
| +type keyCapturingFiles struct { |
| + path string |
| + seen []byte |
| +} |
| + |
| +func (k *keyCapturingFiles) Establish(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, work hostwrite.Work) error { |
| + raw, err := os.ReadFile(k.path) |
| + k.seen = raw |
| + return err |
| +} |
| + |
| +type keyTestMarkers struct{} |
| + |
| +func (keyTestMarkers) PutMarker(context.Context, string, string, []byte, string, func(context.Context) error) (string, error) { |
| + return "v1", nil |
| +} |
| + |
| +type passWork struct{} |
| + |
| +func (passWork) With(_ context.Context, _ hostwrite.Identity, fn func() error) error { return fn() } |
| + |
| +// The claim's first wake mints the key in Select (the launch source), so |
| +// the env file the C8 provider publishes in Materialize already carries it, |
| +// and the launch env the app will run with carries the same value. |
| +func TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest(t *testing.T) { |
| + f := newAdminTokenFixture(t, "box-1") |
| + files := &keyCapturingFiles{path: f.envPath()} |
| + owners := &claimServingOwners{Data: leasetest.NewVersionedBucket(), Bucket: "data", Region: "ie", Box: "box-1", |
| + NewID: func() string { return "g1" }, Now: func() time.Time { return time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) }, |
| + Sources: ClaimServingSources{Launch: claimServingLaunchSource(f.backend, f.tokens), Markers: keyTestMarkers{}, Files: files}} |
| + o, err := hostclaim.Read(t.Context(), f.claims, adminTokenHost) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + in, err := owners.Select(t.Context(), f.rt, o, activator.StartPolicy{}) |
| + if err != nil { |
| + t.Fatalf("select: %v", err) |
| + } |
| + if err := owners.Materialize(t.Context(), o, in, passWork{}); err != nil { |
| + t.Fatalf("materialize: %v", err) |
| + } |
| + var key string |
| + for _, line := range strings.Split(string(files.seen), "\n") { |
| + if v, ok := strings.CutPrefix(line, activator.InstanceKeyVar+"="); ok { |
| + key = v |
| + } |
| + } |
| + if len(key) != 44 { |
| + t.Fatalf("the env file at Establish carries no minted key: %d bytes", len(files.seen)) |
| + } |
| + found := false |
| + for _, kv := range in.Env { |
| + found = found || kv == activator.InstanceKeyVar+"="+key |
| + } |
| + if !found { |
| + t.Fatal("the launch env does not carry the key the provider published") |
| + } |
| +} |
| + |
| +// An instance from before this build has a token and no key. Its next |
| +// claim launch mints the key under the same guarded write, leaves the |
| +// token alone, and a retry keeps both. |
| +func TestClaimLaunchMintsAKeyBesideAnExistingToken(t *testing.T) { |
| + f := newAdminTokenFixture(t, "box-1") |
| + tok := strings.Repeat("c", 64) |
| + if err := os.WriteFile(f.envPath(), []byte("CARLOS_ADMIN_TOKEN="+tok+"\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + launch := claimServingLaunchSource(f.backend, f.tokens) |
| + first, err := launch(t.Context(), f.rt) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + raw, _ := os.ReadFile(f.envPath()) |
| + if f.token() != tok || !strings.HasPrefix(string(raw), "CARLOS_ADMIN_TOKEN="+tok+"\n"+activator.InstanceKeyVar+"=") { |
| + t.Fatalf("env file after the launch: %d bytes, token kept %v", len(raw), f.token() == tok) |
| + } |
| + again, err := launch(t.Context(), f.rt) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + raw2, _ := os.ReadFile(f.envPath()) |
| + if string(raw2) != string(raw) || again.Identity != first.Identity { |
| + t.Fatal("a retried launch changed the env file or the launch identity") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `go test ./internal/host/ -run 'TestClaimFirstWakeKeyIsOnDiskBeforeTheFilesManifest|TestClaimLaunchMintsAKeyBesideAnExistingToken' -count=1` |
| +Expected: FAIL. The first test reports "the env file at Establish carries no minted key". The second fails on the prefix check, because the token-only file is returned early and gets no key. |
| + |
| +- [ ] **Step 3: Mint the key in the authorized write** |
| + |
| +In `internal/activator/claim_snapshot.go`, replace the body from the early return to the end of `EnsureClaimAdminToken`: |
| + |
| +```go |
| + // The instance key is initialized the same way and in the same |
| + // authorized write (L21). A key the rendered configuration supplies |
| + // counts as present, as ensureInstanceKey treats it. |
| + if vars["CARLOS_ADMIN_TOKEN"] != "" && vars[InstanceKeyVar] != "" { |
| + return nil |
| + } |
| + if authorize == nil { |
| + return hostwrite.ErrUnavailable |
| + } |
| + paths := b.pathsFor(rt) |
| + return authorize(ctx, func() error { |
| + if paths.layout == LayoutNested { |
| + if err := EnsureAdminTokenBeneath(paths.dataDir, rt.Host+".env"); err != nil { |
| + return err |
| + } |
| + _, err := EnsureInstanceKeyBeneath(paths.dataDir, rt.Host+".env") |
| + return err |
| + } |
| + if err := EnsureAdminToken(paths.envFile); err != nil { |
| + return err |
| + } |
| + _, err := EnsureInstanceKey(paths.envFile) |
| + return err |
| + }) |
| +} |
| +``` |
| + |
| +Leave `SnapshotClaimLaunch` read-only and unchanged. It never mints, and existing tests pin that. |
| + |
| +In `internal/host/claimowners.go`, extend the `ClaimFilesManifest` doc comment (after "...starts from no manifest."): |
| + |
| +```diff |
| +@@ -133,6 +133,17 @@ type ClaimMarkerStore interface { |
| + // region. Missing or malformed data is an error, never an empty-files |
| + // bootstrap; only an empty bootstrap (in.Empty, after EmptyAuthority) starts |
| + // from no manifest. |
| ++// |
| ++// The instance key (L21, plan 2026-10-10-instance-key-provision.md ruling |
| ++// 6): the launch source has already minted CARLOS_INSTANCE_KEY into the |
| ++// host's own <host>.env when Establish runs, and the app executes only |
| ++// after it returns. So the manifest Establish leaves must carry the env |
| ++// file (region HA §5.6's always-continuous platform item) with exactly the |
| ++// bytes on disk, staged and committed before Establish returns; an error |
| ++// here, a bucket that cannot be reached included, fails the wake and the |
| ++// app does not start. A copied manifest whose env item differs from the |
| ++// file on disk gets a new revision with the file's bytes; it never |
| ++// replaces the file on disk. |
| + type ClaimFilesManifest interface { |
| + Establish(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, work hostwrite.Work) error |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests, then the claim tests around them** |
| + |
| +Run: `go test ./internal/host/ -run 'InstanceKey|AdminToken|Rev2Shape|SnapshotClaimLaunch' -count=1` |
| +Expected: PASS, including the existing `TestClaimProductionAdminTokenRefusesWithoutAuthority`. Without authority no `<host>.env` is written, so no key is written either. |
| + |
| +Run: `go test ./internal/activator/ -run 'Claim' -count=1` |
| +Expected: PASS, including `TestEnsureClaimAdminTokenInspectsSidecarBeforeAuthorize`. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/activator internal/host |
| +git add internal/activator/claim_snapshot.go internal/host/claimowners.go internal/host/claim_instancekey_test.go |
| +git commit -m "claim: the launch source mints the instance key under the guard, before Establish (L21) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: The startup backfill |
| + |
| +**Files:** |
| +- Create: `internal/host/instancekeys.go` |
| +- Modify: `internal/host/agent.go:476-484` (the legacy branch's `startActivator` recovery closure) |
| +- Test: `internal/host/instancekeys_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `(*activator.ExecBackend).BackfillInstanceKey` and `activator.InstanceKeyOutcome` values; `registry.Route`, `registry.BackingUnit`; `(*registry.Registry).List` (registry.go:882). |
| +- Produces: `backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) instanceKeyBackfill`; the log line `agent: instance keys: N minted, N already present, N set by app configuration, N never started here, N failed, N rows not instances`. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Create `internal/host/instancekeys_test.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "errors" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +type fakeKeyBackfiller struct { |
| + got map[string]activator.InstanceKeyOutcome |
| + err map[string]error |
| + asked []string |
| +} |
| + |
| +func (f *fakeKeyBackfiller) BackfillInstanceKey(rt registry.Route) (activator.InstanceKeyOutcome, error) { |
| + f.asked = append(f.asked, rt.Host) |
| + return f.got[rt.Host], f.err[rt.Host] |
| +} |
| + |
| +func TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances(t *testing.T) { |
| + ready := func(host string) registry.Route { |
| + return registry.Route{Host: host, Kind: "instance", App: "shop", Status: "ready", Socket: "/run/carlos/" + host + ".sock"} |
| + } |
| + held := ready("moving.example.com") |
| + held.Status = "moving" |
| + landing := ready("landing.example.com") |
| + landing.Status = "landing" |
| + provisioning := ready("new.example.com") |
| + provisioning.Status = "provisioning" |
| + unit := ready("unit.example.com") |
| + unit.Backing = registry.BackingUnit |
| + alias := ready("alias.example.com") |
| + alias.Platform = true |
| + sidecar := ready("worker.example.com") |
| + sidecar.Kind = "sidecar" |
| + addr := ready("addr.example.com") |
| + addr.Socket, addr.Addr = "", "127.0.0.1:8444" |
| + f := &fakeKeyBackfiller{ |
| + got: map[string]activator.InstanceKeyOutcome{"shop.example.com": activator.InstanceKeyMinted, "old.example.com": activator.InstanceKeyPresent, "cfg.example.com": activator.InstanceKeyFromConfig}, |
| + err: map[string]error{"bad.example.com": errors.New("read: permission denied")}, |
| + } |
| + st := backfillInstanceKeys([]registry.Route{ready("shop.example.com"), ready("old.example.com"), ready("cfg.example.com"), ready("bad.example.com"), held, landing, provisioning, unit, alias, sidecar, addr}, f) |
| + if len(f.asked) != 4 { |
| + t.Fatalf("asked %v, want only the four ready activator instances", f.asked) |
| + } |
| + if st != (instanceKeyBackfill{Minted: 1, Present: 1, FromConfig: 1, Failed: 1, Skipped: 7}) { |
| + t.Fatalf("stats = %+v", st) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run it to verify it fails** |
| + |
| +Run: `go test ./internal/host/ -run TestBackfillInstanceKeysWritesOnlyReadyActivatorInstances -count=1` |
| +Expected: build failure, `undefined: backfillInstanceKeys`. |
| + |
| +- [ ] **Step 3: Write the backfill** |
| + |
| +Create `internal/host/instancekeys.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "log" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// The instance-key backfill (L21, plan 2026-10-10-instance-key-provision.md |
| +// ruling 7). Every launch path mints CARLOS_INSTANCE_KEY before it reads the |
| +// env, so no process is ever started without one. The backfill gives every |
| +// instance already on a box its key now, at agent start, rather than at its |
| +// next start, so the rollout can be checked box by box before any app adopts |
| +// the key: a host that the backfill names "present" or "minted" is ready. |
| +// |
| +// It runs only where the legacy activator writes <host>.env itself. With |
| +// host claims on, every write to a host's installation runs under the |
| +// claim's guard, and the claim's launch source already mints the key under |
| +// that guard before every cold admission (claim_snapshot.go |
| +// EnsureClaimAdminToken), so a claim box is not backfilled here. |
| + |
| +// instanceKeyBackfiller is the activator surface the backfill needs. |
| +type instanceKeyBackfiller interface { |
| + BackfillInstanceKey(registry.Route) (activator.InstanceKeyOutcome, error) |
| +} |
| + |
| +// instanceKeyRoute selects the rows the backfill writes for: ready, |
| +// activator-spawned instances, the rows execSyncable delivers config to, |
| +// without sidecars. A held row (moving, landing, restoring, releasing) |
| +// belongs to its operation and is skipped, as is a provisioning row whose |
| +// first start will mint. |
| +func instanceKeyRoute(rt registry.Route) bool { |
| + return rt.Kind == "instance" && rt.Host != "" && rt.Socket != "" && !rt.Platform && |
| + (rt.Status == "" || rt.Status == "ready") && |
| + rt.Backing != registry.BackingUnit && rt.Tunnel == "" && rt.Unit == "" |
| +} |
| + |
| +// instanceKeyBackfill counts one pass's outcomes. |
| +type instanceKeyBackfill struct { |
| + Minted, Present, FromConfig, NoEnvFile, Failed, Skipped int |
| +} |
| + |
| +// backfillInstanceKeys runs the backfill over routes. It never fails the |
| +// agent: a host it cannot write is logged and counted, and that host's next |
| +// start mints its key (or refuses, loudly, if the same fault is still there). |
| +// Nothing logged names a key. |
| +func backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) instanceKeyBackfill { |
| + var st instanceKeyBackfill |
| + for _, rt := range routes { |
| + if !instanceKeyRoute(rt) { |
| + st.Skipped++ |
| + continue |
| + } |
| + got, err := b.BackfillInstanceKey(rt) |
| + if err != nil { |
| + st.Failed++ |
| + log.Printf("agent: instance key for %s: %v (its next start mints it)", rt.Host, err) |
| + continue |
| + } |
| + switch got { |
| + case activator.InstanceKeyMinted: |
| + st.Minted++ |
| + case activator.InstanceKeyPresent: |
| + st.Present++ |
| + case activator.InstanceKeyFromConfig: |
| + st.FromConfig++ |
| + case activator.InstanceKeyNoEnvFile: |
| + st.NoEnvFile++ |
| + } |
| + } |
| + log.Printf("agent: instance keys: %d minted, %d already present, %d set by app configuration, %d never started here, %d failed, %d rows not instances", |
| + st.Minted, st.Present, st.FromConfig, st.NoEnvFile, st.Failed, st.Skipped) |
| + return st |
| +} |
| +``` |
| + |
| +Wire it in `internal/host/agent.go`, in the legacy branch's recovery closure, between `recoverLayoutAtStart` and `armedLayoutPass.Store` (`eb` is the `*activator.ExecBackend` built at :331): |
| + |
| +```go |
| + recoverLayoutAtStart(ctx, pass, intent) |
| + // L21: every instance already on this box gets its key |
| + // after layout recovery has settled every journal and |
| + // before the layout pass is published or the activator |
| + // wakes anything, so nothing moves a file under it. |
| + if routes, err := reg.List(); err != nil { |
| + log.Printf("agent: instance keys: list routes: %v (each host's next start mints its key)", err) |
| + } else { |
| + backfillInstanceKeys(routes, eb) |
| + } |
| + armedLayoutPass.Store(pass) |
| +``` |
| + |
| +Do not add it to the `claims.Writes.Enabled` branch (ruling 7). |
| + |
| +- [ ] **Step 4: Run the test, then the package and the cross-compiles** |
| + |
| +Run: `go test ./internal/host/ -run 'TestBackfillInstanceKeys|TestStartActivator' -count=1` |
| +Expected: PASS. |
| + |
| +Run: `go vet ./internal/host/ && GOOS=darwin go build ./internal/host/ && GOOS=windows go build ./internal/host/ && GOOS=linux GOARCH=arm64 go build ./internal/host/` |
| +Expected: no output. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/host |
| +git add internal/host/instancekeys.go internal/host/instancekeys_test.go internal/host/agent.go |
| +git commit -m "agent: backfill the instance key into existing <host>.env files at startup (L21) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: Never overwrite across re-render and moves |
| + |
| +These tests pin behaviour the code already has, which this plan now depends on (ruling 8). They should pass on first run. A failure means a premise of this plan is false: stop and report it rather than changing the code under test. |
| + |
| +**Files:** |
| +- Create: `internal/appconfig/syncexec_instancekey_test.go` |
| +- Create: `internal/moves/envreport_instancekey_test.go` |
| +- Create: `internal/moves/archive/instancekey_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `execKit`, `SetEnv`, `ParsePrivateKey`, `SyncExec` (internal/appconfig); `Class`, `ReportEnv`, `CompareEnv` (internal/moves/envreport.go); `filesFixture`, `mustWrite`, `Build`, `Extract` (internal/moves/archive). |
| +- Produces: tests only. |
| + |
| +- [ ] **Step 1: Write the tests** |
| + |
| +`internal/appconfig/syncexec_instancekey_test.go`: |
| + |
| +```go |
| +package appconfig |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// A re-render never touches the instance's own env file, so the key minted |
| +// there survives every config change, including one that sets the same |
| +// name in the app's configuration (L21, never overwrite: re-render). |
| +func TestSyncExecLeavesTheInstanceKeyAlone(t *testing.T) { |
| + ctx, s, reg, dd, priv, _ := execKit(t) |
| + pk, _ := ParsePrivateKey(priv, "test") |
| + own := filepath.Join(dd, "shop.example.com.env") |
| + body := []byte("CARLOS_ADMIN_TOKEN=tok\nCARLOS_INSTANCE_KEY=q+/Zk3vJ0aW1Yx9sR7tUe2oPl4mN8bC6dF5gH0iJ1kA=\n") |
| + if err := os.WriteFile(own, body, 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.Upsert(registry.Route{Host: "shop.example.com", Kind: "instance", App: "shop", Account: "acct", Status: "ready", Socket: "/run/carlos/shop.sock"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, vars := range []map[string]string{{"A": "1"}, {"A": "2", "CARLOS_INSTANCE_KEY": "from-config"}} { |
| + if err := SetEnv(ctx, s, "acct", "shop", vars); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := SyncExec(ctx, reg, s, pk, dd, func(string) error { return nil }, "", nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, _ := os.ReadFile(own); string(got) != string(body) { |
| + t.Fatalf("a re-render changed the instance's own env file: %q", got) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`internal/moves/envreport_instancekey_test.go`: |
| + |
| +```go |
| +package moves |
| + |
| +import "testing" |
| + |
| +// A move compares the instance key like any instance variable: a target |
| +// that would start with a different key, or none, is a difference the |
| +// launch check reports, never a note (L21). |
| +func TestEnvReportComparesTheInstanceKey(t *testing.T) { |
| + const name = "CARLOS_INSTANCE_KEY" |
| + if got := Class(name); got != "equal" { |
| + t.Fatalf("Class(%s) = %q, want equal", name, got) |
| + } |
| + key := []byte("0123456789abcdef0123456789abcdef") |
| + src := ReportEnv(key, map[string]string{name: "a2V5LW9uZQ==", "CARLOS_REGION": "ie"}) |
| + for _, tc := range []struct { |
| + target map[string]string |
| + want string |
| + }{ |
| + {map[string]string{name: "a2V5LXR3bw==", "CARLOS_REGION": "uk"}, "differs"}, |
| + {map[string]string{"CARLOS_REGION": "uk"}, "missing"}, |
| + } { |
| + diffs := CompareEnv(src, ReportEnv(key, tc.target), false) |
| + if len(diffs) != 1 || diffs[0].Name != name || diffs[0].Kind != tc.want || diffs[0].Note() { |
| + t.Fatalf("diffs = %+v, want one %s on %s", diffs, tc.want, name) |
| + } |
| + } |
| + if diffs := CompareEnv(src, ReportEnv(key, map[string]string{name: "a2V5LW9uZQ==", "CARLOS_REGION": "uk"}), false); len(diffs) != 0 { |
| + t.Fatalf("the same key reported a difference: %+v", diffs) |
| + } |
| +} |
| +``` |
| + |
| +`internal/moves/archive/instancekey_test.go`: |
| + |
| +```go |
| +package archive |
| + |
| +import ( |
| + "bytes" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| +) |
| + |
| +// The move archive carries <host>.env byte for byte, so the instance key |
| +// lands on the target exactly as the source minted it (L21: the key is the |
| +// exact bytes the env file yields). |
| +func TestArchiveCarriesTheInstanceKeyByteForByte(t *testing.T) { |
| + dir, env := filesFixture(t) |
| + body := []byte("CARLOS_ADMIN_TOKEN=t\nCARLOS_INSTANCE_KEY=q+/Zk3vJ0aW1Yx9sR7tUe2oPl4mN8bC6dF5gH0iJ1kA=\n") |
| + mustWrite(t, env, body, 0o600) |
| + var built bytes.Buffer |
| + if _, _, _, err := Build(dir, env, &built); err != nil { |
| + t.Fatal(err) |
| + } |
| + out := filepath.Join(t.TempDir(), "out") |
| + outEnv := filepath.Join(t.TempDir(), "out.env") |
| + if err := Extract(bytes.NewReader(built.Bytes()), out, outEnv); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := os.ReadFile(outEnv) |
| + if err != nil || !bytes.Equal(got, body) { |
| + t.Fatalf("extracted env differs from the source's (err %v)", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them** |
| + |
| +Run: `go test ./internal/appconfig/ -run TestSyncExecLeavesTheInstanceKeyAlone -count=1 && go test ./internal/moves/ -run TestEnvReportComparesTheInstanceKey -count=1 && go test ./internal/moves/archive/ -run TestArchiveCarriesTheInstanceKeyByteForByte -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +gofmt -l internal/appconfig internal/moves |
| +git add internal/appconfig/syncexec_instancekey_test.go internal/moves/envreport_instancekey_test.go internal/moves/archive/instancekey_test.go |
| +git commit -m "test: the instance key survives re-render and travels byte for byte through a move (L21) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Final verification (once, before handing off) |
| + |
| +- [ ] `go build ./... && go vet ./... && gofmt -l .` (no output) |
| +- [ ] `go test ./... -count=1` (all packages pass) |
| +- [ ] `GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...` |
| +- [ ] `git grep -n 'CARLOS_INSTANCE_KEY' -- '*.go' | grep -i 'log\.\|Printf\|Errorf'`: every hit names the variable, never prints a value. |
| +- [ ] Push to amadan and read `amadan ci status carlos/platform -branch instance-key-provision` (exit 0) before asking for the merge. |
| + |
| +## Rollout |
| + |
| +1. **Merge and release.** Merge on amadan with `amadan branch merge -squash` after re-merging main and a green CI on the tip. Cut a platform release by the README's "Releasing the platform" recipe. |
| +2. **Ship to every box, every region.** Use the platform roll recipe (build arm64, upload, `update-platform-binary.sh` per box, `roll-console.sh` after). The console needs nothing new, but rolls with the release as usual. |
| +3. **Verify the backfill on each box.** `journalctl -u carlos-edge --since '<roll time>' | grep 'agent: instance keys:'` must show `0 failed`. For each failure, read the per-host line above it, fix the file, and restart the instance (its start mints the key, or fails loudly with the same error). `never started here` hosts mint at their first start. Claim boxes print no backfill line: their instances mint at their next cold admission. |
| +4. **Only then tell app owners to adopt** rastrillo e3ea08d or later: `carlos.InstanceKey(own)`. Before a box has this build, an adopting app with no key of its own refuses to start on it. |
| +5. **App owners, before adopting:** |
| + - An app with its own key passes it as `own`: its explicit setting, or the key file it already has, read without minting. It keeps that key, and nothing changes. |
| + - An app that ran on CARLOS with a **development fallback key** moves to the platform key when it adopts. Anything sealed under the old key stops reading: sessions, sealed cookies, sign-in challenges, sealed columns. Check what is sealed under the old key first. Then either pass the old key as `own` and keep it, or accept the switch knowing what it invalidates. |
| + - An app with replica placements keeps its own key until open question 1 is answered: each replica host gets its own platform key. |
| +6. **Region HA.** Do not enable relocation on a region until this plan is rolled there and O1 (C8) and O2 (stage B pinner) are in. That is stage B Task 10's checklist item 9. |
| + |
| +**Rollback:** roll the previous binary. Keys already written stay in `<host>.env`. That is harmless: no app reads them until it adopts, and the next forward roll finds them present. |
| + |
| +## Open questions for Paul |
| + |
| +1. **Replicas.** A replica host (#162) holds a one-way mirror of its home, but gets its own platform key. Should a replica receive its home's key instead? That needs a way to copy the key between boxes, which this plan does not build. |
| +2. **A key set in configuration.** This plan defers to it (ruling 5). The alternative is to reserve `CARLOS_INSTANCE_KEY`: refuse it in `carlos env set` / `secrets set` and strip it at render. That needs copy review for the refusal message. |
| +3. **`report.token`.** The L21 row's candidate contract also pre-provisioned `report.token`. The decision covers only the key. Should the platform provision `report.token` too? |
| +4. **Workers and sidecars.** Should a worker get its instance's key? Today they get none and behave as off CARLOS. |
| +5. **Restarts inside a claim session.** On claim boxes, a key minted by an activator restart inside a running session (not a cold admission) is published by the continuous publisher, not before the start. This only happens to an instance admitted before this build that then restarts in the same session. Accept it, or forbid minting outside the claim's launch source on claim boxes? |