amadan
docs: the instance-key plan matches what was built, with a Built note, the rollout's two pre-roll checks, the known gaps, O6's threading and loop obligations, and the follow-ups (end sweep)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+267
−44
docs/superpowers/plans/2026-10-10-instance-key-provision.md+267 −44
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/plans/2026-10-10-instance-key-provision.md b/docs/superpowers/plans/2026-10-10-instance-key-provision.md| index 2c23aba7e..1e16b5bbc 100644 |
| --- a/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| +++ b/docs/superpowers/plans/2026-10-10-instance-key-provision.md |
| @@ -2,6 +2,13 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| +**Built (branch `instance-key-build`, tasks 1 to 7 and 3b, 2026-10-11).** The code blocks below are the branch's. Where the build diverged from rev 5.1, each by a ledger ruling (`.superpowers/sdd/2026-10-10-instance-key-provision/progress.md`): |
| +- Task 2: the flat own file is opened with `O_NOFOLLOW` (`openOwnEnvNoFollow`, with an `Lstat` stand-in for windows), a failed append is truncated back to the descriptor's size before the error returns, and `ensureInstanceKeyFor` refuses a fenced or conflicting row itself (the launch bodies classify again after the dispatcher's check). A symlinked flat `<host>.env` now fails the launch; the rollout checks for one first. |
| +- Task 3b: the sidecar worker (`StartWorkerStaged`) and the legacy co-spawned sidecar (`startSidecarChildFrom`) pass the replica gate too, as a seed does (no control socket); a key in a worker's agent-issued `extra` is refused as of unknown origin. `replicaKeyGate` takes the key's layer; the home-keys list is a file the agent reads at every launch, not an environment variable. |
| +- Task 4: the claim's guarded write mints the admin token only when the effective env has none, so a configured token is never shadowed by a second one; a missing own file with both values configured proceeds as before; the key's refusal is wrapped with the host once, in the launch source. |
| +- Task 6: the activator's start sequence is a named, tested function (`activatorStart.Run`), with the backfill between layout recovery and the pass's publication on a legacy box and nothing on a claim box. |
| +- Known gaps, obligations and follow-ups are at the end, with the rollout's two pre-roll checks. |
| + |
| **Goal:** The platform mints every exec-backed instance's `CARLOS_INSTANCE_KEY` on the box, create-only and durably, into the instance's own `<host>.env` before any process of the instance starts, and backfills it into every existing instance. Relocation accepts a key only with evidence that the app actually seals with it: its own `instance.key`, or an adoption record bound to the writer whose data is pinned, for the exact key the target will deliver, pinned by digest and checked again at install. |
| **Architecture:** A small leaf package, `internal/instancekey`, holds the one rule for the key: the env parser, which file's key the app receives, and the adoption record. The activator mints by it (`instancekey.go`), right after the existing `CARLOS_ADMIN_TOKEN` mint on every launch path and under the same lock, and makes the file durable on every call. A control-socket verb, `PUT /instance-key/adoption`, records a process's adoption beside `<host>.env`, bound to the writer the agent admitted it as, and publishes it on the running session through `hostfiles.PlatformRelay`; claim launches mount it once they bind a control socket (O6). A startup backfill in `internal/host` appends a key to every existing `<host>.env` on boxes where host claims are off. On claim boxes the launch source mints under the claim's guard before `Materialize`. Stage B's pinner (O2'), the C8 provider (O1'), the relocation install (O3') and moves (O7) carry the evidence across boxes and retire it. No flag. |
| @@ -343,12 +350,14 @@ Rev 3 rewrites O1', O2', O3', O4, O5 and O6, and adds O7. Rev 4 adds O8. |
| - **O5 (stage B's release flush, `internal/host/relocationflush.go` on that branch):** stages the local adoption record as `AdoptionItem` when it parses and binds the releasing writer (the holder's incarnation, session and generation); otherwise it stages `StagePlatformDelete`. Both go in the same commit as the env item. Test: a flush with the releasing writer's record publishes it; with a stale record, or none, publishes none. |
| - **O6 (claim Plan 9 Group D, the isolated-root launcher), the gate of ruling 23:** |
| - When claim launches bind a control socket, mount `ExecBackend.MountInstanceKeyAdoption` on it. |
| - - Its `launch` returns the process's `AdoptionLaunch`: `Writer` from the admission (holder incarnation, session and generation as admitted); `KeySHA256` and `Source` from `ClaimLaunchSnapshot.InstanceKeySHA256` and `InstanceKeySource`, carried through `ClaimServingLaunch` and `ClaimLaunchInputs`; and the instance directory. |
| - - `Publish` is a `hostfiles.PlatformRelay` over the session's files publisher, whose `hostfiles.Run` loop runs for the session. |
| + - Thread `ClaimLaunchSnapshot.InstanceKeySHA256` and `InstanceKeySource` through `ClaimServingLaunch` and `ClaimLaunchInputs`. Today `claimServingLaunchSource` (claimproduction.go) drops both; the snapshot computes them and nothing carries them further. |
| + - Its `launch` returns the process's `AdoptionLaunch`: `Writer` from the admission (holder incarnation, session and generation as admitted); `KeySHA256` and `Source` from the threaded fields; and the instance directory. **It must reflect the current admission:** the verb binds whatever `launch()` returns at the moment of the request, so once the session has ended or been superseded `launch` returns `false` (a 404), never the old session's writer. A late helper report after a re-admission then writes nothing. The verb refuses a writer binding the pinner could not read (500), but it cannot tell a stale complete one from a current one; that is the callback's lifetime, which O6's review checks against the session's. |
| + - `Publish` is a `hostfiles.PlatformRelay` over the session's files publisher. O6 runs **both** `hostfiles.Run` and `relay.Run` on the session's publisher: the relay only stages, and a staged item nobody commits never reaches the bucket (`TestAdoptionSurvivesBoxLossWithoutAnotherAdmissionOrRelease` is the shape). |
| + - The record's write holds `adminTokenMu` across its read and two fsyncs (once per helper report, once per process start); on a slow disk that stalls every token mint and key ensure on the box for that long. Known and accepted; O6's review should know the cost. |
| - The record's write runs under the claim's guard (a hostwrite selection for the host, as the token's mint does). |
| - The launch env gains the agent-owned `CARLOS_INSTANCE_KEY_SOURCE` (`platform` or `config`, from `InstanceKeySource`), set only when the socket is bound. It is stripped from tenant files like `CARLOS_CONTROL_SOCKET` (`agentOwnedVars`) and added to appconfig's `agentOwned` list. |
| - `SnapshotClaimLaunch`'s replica gate (ruling 26) then takes the bound socket as `haveControl`, so a keyless replica runs and its helper refuses at step 3. |
| - - **O6 does not ship before O7.** Until O6, ruling 23's refusal stands. |
| + - **O7 ships before O6; O6 does not ship before O7** (ruling 24). Until O6, ruling 23's refusal stands. |
| - **O7 (moves, the journalled artifact lifecycle; ruling 24).** These lists change: |
| - `installArtifactPaths` (internal/host/movesinstall.go:751-754): append `host + instancekey.AdoptionSuffix` after `host + ".env"`. It flows into the dirty return's pre-move park (movesinstall.go:852), into `sourceArtifactPaths` (movesfinal.go:586-587) for the source finalization's park (movesfinal.go:527), and into the rollback's candidates (movesrollback.go:1314). |
| - The landing's flat inventory (movesinstall.go:664): `paths` gains `host + instancekey.AdoptionSuffix`. The installation's inventory then claims the path its app's sessions will write. |
| @@ -1300,8 +1309,10 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 2: The mint, durable every time |
| **Files:** |
| -- Create: `internal/activator/instancekey.go` |
| -- Test: `internal/activator/instancekey_test.go` |
| +- Create: `internal/activator/instancekey.go`, `internal/activator/ownenv_unix.go` and `ownenv_other.go` (`openOwnEnvNoFollow`: `O_NOFOLLOW` on unix, an `Lstat` check-then-open on windows, which the agent never runs on) |
| +- Test: `internal/activator/instancekey_test.go`, `internal/activator/instancekey_linux_test.go` |
| + |
| +**Built:** the code block in Step 3 is the branch's, with three things rev 5.1 did not have, each by ruling. The flat own file is opened with `O_NOFOLLOW` in the open itself (not check-then-open), mirroring `EnsureAdminToken`'s symlink refusal; a symlinked flat `<host>.env` fails the launch (the admin-token mint path has the same hole on main: #273). A failed or short append is undone: the file is truncated back to the size the open descriptor reported before the write (`undoInstanceKeyAppend`) and fsynced, so a short key line never survives to be read as present; under `adminTokenMu` that size is `len(raw)`. And `ensureInstanceKeyFor` refuses a fenced or conflicting row itself, deciding on the paths the mint would use, because the launch bodies classify again in `pathsFor` after the dispatcher's own check and a migration starting mid-wake would otherwise mint into the flat directory of a half-migrated host. |
| **Interfaces:** |
| - Consumes: Task 1's `instancekey`; `adminTokenMu` (admintoken.go:43), `logPerFile` (:70), `parseEnvBytes`, `instancefs.ReadBeneathLimit`, `instancefs.OpenBeneath`, `(*ExecBackend).pathsFor` (paths.go:33), `Layout*` constants (layout.go:21-26). |
| @@ -1693,15 +1704,27 @@ package activator |
| // 0600 (fchmod on its descriptor), the file is fsynced and so is its |
| // directory, when the key was already there too. A sync that failed |
| // once is redone by the next ensure, never skipped because the bytes |
| -// are already there; |
| +// are already there. The known limit: once fsync has returned EIO, |
| +// Linux may mark the pages clean, so a later fsync's success does not |
| +// prove the bytes reached the disk. The retry narrows the gap; it |
| +// cannot close it, and no create-only design can; |
| +// - a mint that fails part way through its append is undone: the file |
| +// is truncated back to its length before the append and fsynced, so a |
| +// short key line never survives to be read as present. Those bytes |
| +// were never reported, so removing them rewrites no key; |
| // - the value is never logged, never put in an error, and never leaves |
| -// the file except in the launch environment. |
| +// the file except in the launch environment; |
| +// - a flat row's own file is opened O_NOFOLLOW on every path, read and |
| +// append alike (openOwnEnvNoFollow): a symlink there is refused, never |
| +// followed into a file a same-uid sibling chose. A nested row's file |
| +// gets the same from instancefs. |
| import ( |
| "crypto/rand" |
| "encoding/base64" |
| "errors" |
| "fmt" |
| + "io" |
| "os" |
| "path/filepath" |
| "strings" |
| @@ -1778,6 +1801,18 @@ func syncDir(dir string) error { |
| return err |
| } |
| +// instanceKeyWrite is the mint's append, a seam so a test can make it fail |
| +// part way (a short write, ENOSPC). Read and replaced only under |
| +// adminTokenMu. |
| +var instanceKeyWrite = func(f *os.File, b []byte) (int, error) { return f.Write(b) } |
| + |
| +// instanceKeyBeforeOpen, when set, runs under adminTokenMu after the read |
| +// and the decision and before the ensure opens the file to append or |
| +// settle: a test hook to change the file between the two, so the open's |
| +// own symlink refusal is pinned and not only the read's. nil outside |
| +// tests. |
| +var instanceKeyBeforeOpen func(path string) |
| + |
| func mintInstanceKey() (string, error) { |
| raw := make([]byte, instanceKeyBytes) |
| if _, err := rand.Read(raw); err != nil { |
| @@ -1820,10 +1855,25 @@ func EnsureInstanceKeyBeneath(dir, name string) (InstanceKeyOutcome, error) { |
| // ensureInstanceKeyFor is the launch paths' ensure for rt: none for a |
| // replica row, the nested body for a nested row, the flat one otherwise. |
| +// |
| +// A fenced or conflicting row is refused and nothing is written, as |
| +// BackfillInstanceKey refuses it. Every launch dispatcher (Start, |
| +// StartGen, StartStaged, Seed) refuses those rows already, but on its own |
| +// classification: the launch body classifies again in pathsFor, and p is |
| +// that second answer. A layout journal written between the two (a |
| +// migration starting mid-wake) makes p fenced, with the flat directory as |
| +// its dataDir, and minting there would write to a half-migrated host |
| +// (isolated-exec design §8.2). The refusal is decided on p, the paths the |
| +// mint would use. Every caller names the host around the error ("instance |
| +// key for <host>: ..."), so the refusal does not. |
| func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) { |
| if rt.Replica { |
| return InstanceKeyReplica, nil |
| } |
| + switch p.layout { |
| + case LayoutFenced, LayoutConflict: |
| + return InstanceKeyPresent, fmt.Errorf("not minted: %s", p.reason) |
| + } |
| if p.layout == LayoutNested { |
| return EnsureInstanceKeyBeneath(p.dataDir, rt.Host+".env") |
| } |
| @@ -1836,6 +1886,8 @@ func (b *ExecBackend) ensureInstanceKeyFor(rt registry.Route, p instancePaths) ( |
| // creating a file here could also occupy the path a move's install expects |
| // to be empty (moveslandinstall.go placeStaged). A row this backend would |
| // refuse to start (fenced, conflict) is an error and nothing is written. |
| +// The agent's backfill names the host when it logs the error, so the |
| +// refusal does not. |
| func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome, error) { |
| if rt.Replica { |
| return InstanceKeyReplica, nil |
| @@ -1845,7 +1897,7 @@ func (b *ExecBackend) BackfillInstanceKey(rt registry.Route) (InstanceKeyOutcome |
| case LayoutNested: |
| return ensureInstanceKey(p.dataDir, rt.Host+".env", true, false) |
| case LayoutFenced, LayoutConflict: |
| - return InstanceKeyPresent, fmt.Errorf("%s: not backfilled: %s", rt.Host, p.reason) |
| + return InstanceKeyPresent, fmt.Errorf("not backfilled: %s", p.reason) |
| } |
| return ensureInstanceKey(p.dataDir, rt.Host+".env", false, false) |
| } |
| @@ -1861,7 +1913,24 @@ func openInstanceEnv(dir, name string, beneath, create bool) (*os.File, error) { |
| if beneath { |
| return instancefs.OpenBeneath(dir, name, flags, perm) |
| } |
| - return os.OpenFile(filepath.Join(dir, name), flags, perm) |
| + return openOwnEnvNoFollow(filepath.Join(dir, name), flags, perm) |
| +} |
| + |
| +// errOwnEnvSymlink is a flat row's own env file found to be a symlink. The |
| +// ensure refuses it on every path (mint, existing key, backfill, the claim |
| +// settle): following it would append to, chmod and fsync a file a same-uid |
| +// sibling chose (spec §7.2). |
| +var errOwnEnvSymlink = errors.New("activator: the instance env file is a symlink, refused") |
| + |
| +// readOwnEnv reads a flat row's own env file through openOwnEnvNoFollow, |
| +// so the read refuses a link exactly as the append does. |
| +func readOwnEnv(path string) ([]byte, error) { |
| + f, err := openOwnEnvNoFollow(path, os.O_RDONLY, 0) |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer f.Close() |
| + return io.ReadAll(f) |
| } |
| // settleInstanceEnv is the durability step every successful ensure ends |
| @@ -1894,7 +1963,13 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| } |
| return os.ReadFile(filepath.Join(dir, n)) |
| } |
| - raw, err := read(name) |
| + var raw []byte |
| + var err error |
| + if beneath { |
| + raw, err = read(name) |
| + } else { |
| + raw, err = readOwnEnv(path) |
| + } |
| switch { |
| case errors.Is(err, os.ErrNotExist): |
| if !create { |
| @@ -1921,6 +1996,9 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| // Present is not a no-op: a mint whose sync failed left these |
| // bytes behind, and this call is the retry that makes them |
| // durable before anything is started with them. |
| + if instanceKeyBeforeOpen != nil { |
| + instanceKeyBeforeOpen(path) |
| + } |
| f, err := openInstanceEnv(dir, name, beneath, false) |
| if err != nil { |
| return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err) |
| @@ -1935,6 +2013,9 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| if err != nil { |
| return InstanceKeyPresent, fmt.Errorf("mint instance key for %s: %w", path, err) |
| } |
| + if instanceKeyBeforeOpen != nil { |
| + instanceKeyBeforeOpen(path) |
| + } |
| f, err := openInstanceEnv(dir, name, beneath, create) |
| if errors.Is(err, os.ErrNotExist) && !create { |
| return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill |
| @@ -1943,12 +2024,24 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| return InstanceKeyPresent, fmt.Errorf("open %s: %w", path, err) |
| } |
| defer f.Close() |
| + // The length the append starts from, taken from the open descriptor: |
| + // what a failed append is undone to. Under adminTokenMu it is |
| + // len(raw); reading it here also keeps any byte another process |
| + // appended since the read. |
| + st, err := f.Stat() |
| + if err != nil { |
| + return InstanceKeyPresent, fmt.Errorf("stat %s: %w", path, err) |
| + } |
| prefix := "" |
| if len(raw) > 0 && raw[len(raw)-1] != '\n' { |
| prefix = "\n" |
| } |
| - if _, err := fmt.Fprintf(f, "%s%s=%s\n", prefix, InstanceKeyVar, minted); err != nil { |
| - return InstanceKeyPresent, fmt.Errorf("write %s: %w", path, err) |
| + line := []byte(prefix + InstanceKeyVar + "=" + minted + "\n") |
| + if n, err := instanceKeyWrite(f, line); err != nil || n != len(line) { |
| + if err == nil { |
| + err = io.ErrShortWrite |
| + } |
| + return InstanceKeyPresent, undoInstanceKeyAppend(f, st.Size(), path, err) |
| } |
| if err := settleInstanceEnv(f, dir, path); err != nil { |
| return InstanceKeyPresent, err |
| @@ -1956,6 +2049,22 @@ func ensureInstanceKey(dir, name string, beneath, create bool) (InstanceKeyOutco |
| return InstanceKeyMinted, nil |
| } |
| +// undoInstanceKeyAppend removes a failed append's bytes: it truncates f |
| +// back to size, the length before the append, and fsyncs, so a short key |
| +// line can never survive to be read as present. Nothing past size was |
| +// ever reported to anyone. Returns the write error, joined with any |
| +// failure to undo it. Caller holds adminTokenMu. |
| +func undoInstanceKeyAppend(f *os.File, size int64, path string, werr error) error { |
| + err := fmt.Errorf("write %s: %w", path, werr) |
| + if terr := f.Truncate(size); terr != nil { |
| + return errors.Join(err, fmt.Errorf("undo the partial write to %s: %w", path, terr)) |
| + } |
| + if serr := instanceKeySync.file(f); serr != nil { |
| + return errors.Join(err, fmt.Errorf("sync %s after undoing the partial write: %w", path, serr)) |
| + } |
| + return err |
| +} |
| + |
| // errInstanceEnvNeedsWrite is settleClaimInstanceEnv's answer for a file |
| // whose mode is not 0600: making it so is a write, which the claim path |
| // does only under its guard. |
| @@ -1966,7 +2075,9 @@ var errInstanceEnvNeedsWrite = errors.New("activator: the instance env file need |
| // file and its directory durable and writes nothing. fsync changes no |
| // byte, mode or name, so it is not a host write and runs without the |
| // claim's guard. A file that is not 0600 is errInstanceEnvNeedsWrite, and |
| -// the caller takes the guarded path, whose ensure sets the mode. |
| +// the caller takes the guarded path, whose ensure sets the mode when the |
| +// own file holds the key (and leaves it when the key is configured or the |
| +// row is a replica: EnsureClaimAdminToken). |
| func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error { |
| p := b.pathsFor(rt) |
| name := rt.Host + ".env" |
| @@ -1977,7 +2088,7 @@ func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error { |
| if p.layout == LayoutNested { |
| f, err = instancefs.OpenBeneath(p.dataDir, name, os.O_RDONLY, 0) |
| } else { |
| - f, err = os.Open(p.envFile) |
| + f, err = openOwnEnvNoFollow(p.envFile, os.O_RDONLY, 0) |
| } |
| if err != nil { |
| return fmt.Errorf("open %s: %w", p.envFile, err) |
| @@ -2000,6 +2111,56 @@ func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error { |
| } |
| ``` |
| +Create `internal/activator/ownenv_unix.go` (the open that refuses a link in the open itself) and `ownenv_other.go` (the windows stand-in, so the cross-compiles build; the agent never runs there): |
| + |
| +```go |
| +//go:build unix |
| + |
| +package activator |
| + |
| +import ( |
| + "errors" |
| + "os" |
| + |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +// openOwnEnvNoFollow opens a flat row's own <host>.env and refuses a |
| +// symlink at it in the open itself (O_NOFOLLOW), so there is no window |
| +// between a check and the open for a same-uid sibling (spec §7.2) to plant |
| +// a link in. A refused link is the bare errOwnEnvSymlink: the caller names |
| +// the path, once. A nested row's file needs none of this: instancefs.OpenBeneath already refuses links. |
| +func openOwnEnvNoFollow(path string, flag int, perm os.FileMode) (*os.File, error) { |
| + f, err := os.OpenFile(path, flag|unix.O_NOFOLLOW, perm) |
| + if errors.Is(err, unix.ELOOP) { |
| + return nil, errOwnEnvSymlink |
| + } |
| + return f, err |
| +} |
| +``` |
| + |
| +```go |
| +//go:build !unix |
| + |
| +package activator |
| + |
| +import ( |
| + "os" |
| +) |
| + |
| +// openOwnEnvNoFollow is the unix helper's stand-in where there is no |
| +// O_NOFOLLOW (windows). The agent only runs on Linux; this keeps the |
| +// cross-compiled builds working and still refuses a link it can see, by |
| +// the same Lstat check EnsureAdminToken makes. It is check-then-open, so |
| +// it is weaker than the unix open, which is the one that matters. |
| +func openOwnEnvNoFollow(path string, flag int, perm os.FileMode) (*os.File, error) { |
| + if fi, err := os.Lstat(path); err == nil && fi.Mode()&os.ModeSymlink != 0 { |
| + return nil, errOwnEnvSymlink |
| + } |
| + return os.OpenFile(path, flag, perm) |
| +} |
| +``` |
| + |
| - [ ] **Step 4: Run the tests to verify they pass** |
| Run: `nice -n 19 go test ./internal/activator/ -run 'TestEnsureInstanceKey|TestBackfillInstanceKey|TestReplicaRows|TestEnsureAdminToken' -count=1` |
| @@ -2412,12 +2573,14 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Files:** |
| - Create: `internal/activator/replicakey.go`, `internal/host/replicahomekeys.go`, `internal/appconfig/instancekey.go` |
| -- Modify: `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild` and `spawnChildFrom`), `internal/activator/prepare.go` (`prepareAt`), `internal/host/agent.go` (the wiring) |
| -- Test: `internal/activator/replica_instancekey_test.go`, `internal/activator/replica_instancekey_linux_test.go`, `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` |
| +- Modify: `internal/activator/backend_exec.go` (`ExecBackend.ReplicaHomeKeys` and `ReplicaConfigKey`; the gate in `seedChild`, `spawnChildFrom`, `StartWorkerStaged` and `startSidecarChildFrom`), `internal/activator/prepare.go` (`prepareAt`), `internal/host/agent.go` (the wiring) |
| +- Test: `internal/activator/replica_instancekey_test.go`, `internal/activator/replica_instancekey_linux_test.go`, `internal/activator/replicakey_race_test.go`, `internal/host/replicahomekeys_test.go`, `internal/appconfig/instancekey_test.go` |
| **Interfaces:** |
| - Consumes: Task 1's `instancekey.Digest`, `Effective` and `ParseHomeKeys`; appconfig's `LoadMergedHostVersions`, `RenderEnv`, `routeBinding`, and test kit `execKit`, `SetEnv`, `ApplyInstance`, `Seal`; Task 2's `InstanceKeyReplica` (never minted); test helpers `testExecBackend`, `tmpSock`, `nestedBackend`, `readEnvFile`. |
| -- Produces: `ExecBackend.ReplicaHomeKeys func(host string) (sha256 string, ok bool)` and `ReplicaConfigKey func(ctx, rt) (sha256 string, err error)`; `(*ExecBackend).replicaKeyGate(rt, env, haveControl) error`; `activator.ErrReplicaKey`; `appconfig.ConfiguredKeyDigest(ctx, s, priv, rt) (string, error)`; `host.replicaHomeKeys(getenv)`, which reads `CARLOS_REPLICA_HOME_KEYS`, and `host.replicaConfigKey(getenv, open)`, the fresh read (ruling 27). Task 4's snapshot calls the gate. |
| +- Produces: `ExecBackend.ReplicaHomeKeys func(host string) (sha256 string, ok bool)` and `ReplicaConfigKey func(ctx, rt) (sha256 string, err error)`; `(*ExecBackend).replicaKeyGate(rt, env, layer, haveControl) error`, with `layer` the one the env's key came from, decided by the same read that composed the env (`instanceEnvLayer`, ruling 29); `activator.ErrReplicaKey`; `appconfig.ConfiguredKeyDigest(ctx, s, priv, rt) (string, error)`; `host.replicaHomeKeys(read)`, which parses the list the reader returns (`/etc/carlos/replica-home-keys`, read again at every replica launch; ruling 30), and `host.replicaConfigKey(getenv, open)`, the fresh read (ruling 27). Task 4's snapshot calls the gate. |
| + |
| +**Built:** two launch paths rev 5.1 left ungated pass the gate as a seed does (no control socket: the home's verified key or none, refused otherwise), by the ledger's rulings on the 3b build and its review. The sidecar worker (`StartWorkerStaged`) reads the env and its key's layer in one read, and when the agent-issued `extra` carries `CARLOS_INSTANCE_KEY` the layer is set to none, so the gate refuses it as a key of unknown origin: a worker never mints and `extra` has no business carrying a key. The legacy co-spawned sidecar (`startSidecarChildFrom`) gates its own env file's key the same way (`sidecarKeyLayer`). Refusal and log text name the file path, not the variable the plan calls the list by. The bucket read in the gate runs on its own bounded context (`prepareAt` has none to thread). |
| - [ ] **Step 1: Write the failing tests** |
| @@ -3479,6 +3642,8 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| - Consumes: Task 1's `instancekey.Effective`, `Digest`, `SourceOf`; Task 2's `ensureInstanceKeyFor`, `settleClaimInstanceEnv`, `errInstanceEnvNeedsWrite`, `probeInstanceKeySync`; `instanceEnvParts` (backend_exec.go:1836); `claimServingLaunchSource` (claimproduction.go:421), `claimServingOwners` (claimowners.go:293) with `Select` (:375) and `Materialize` (:425); test fixture `newAdminTokenFixture`, `adminTokenHost`, `f.envPath()`, `f.token()` (claimproduction_admintoken_test.go:20-118); `leasetest.NewVersionedBucket`; `Service.startAdmitted` (claimwake.go:330) as `claimloop_linux_test.go` drives it. |
| - Produces: `EnsureClaimAdminToken` initializes the token and the key inside the one authorized write, and syncs a present key before it answers. `ClaimLaunchSnapshot.InstanceKeySHA256` and `InstanceKeySource` name the key the snapshot's env delivers, for O6 (ruling 21). |
| +**Built:** the guarded write mints the admin token only when the effective env has none (rev 5.1 called `EnsureAdminToken` unconditionally, which would mint a second token into the own file over a configured one and change a working launch); the key is ensured after it either way. The early return has a third arm: no own env file with the token and the key both from the rendered configuration proceeds as it did before the key existed (pre-flight ruling A; a configured key is not the platform's to make durable, ruling 5, and an absent file is not created, ruling 7). The claim launch source is the one launch path that returns the key's refusal without a wrapper of its own, so it wraps it with the host, once (Task 6's ruling). `claimServingLaunchSource` does not yet carry `InstanceKeySHA256` and `InstanceKeySource` into `ClaimServingLaunch`: that is O6's. |
| + |
| - [ ] **Step 1: Write the failing tests** |
| Create `internal/activator/claim_instancekey_test.go`: |
| @@ -3861,14 +4026,29 @@ Replace the body of `EnsureClaimAdminToken` from the early return to its end: |
| // The instance key is initialized the same way and in the same |
| // authorized write (L21). vars is the effective env, so a key the |
| // rendered configuration supplies counts as present, and a replica |
| - // gets none (ruling 26). When both are present nothing is |
| - // written, but the file is still made durable before the launch |
| - // source answers: a key whose mint failed its sync is never launched |
| - // with until a sync succeeds (settleClaimInstanceEnv). A file that |
| - // needs its mode set takes the guarded path, which sets it. |
| + // gets none (ruling 26). When both are present nothing is written and |
| + // authorize is not called, but the file is still made durable before |
| + // the launch source answers: a key whose mint failed its sync is never |
| + // launched with until a sync succeeds (settleClaimInstanceEnv, ruling |
| + // 18). A file that is not 0600 takes the guarded path. The key's |
| + // ensure sets the mode when the own file holds the key; when the key |
| + // is the configuration's, or the row is a replica, the ensure writes |
| + // nothing, the mode stays as it is, and the next launch of that row |
| + // asks the guard again. The token's mode has never been the claim |
| + // path's to tighten. |
| if vars["CARLOS_ADMIN_TOKEN"] != "" && (rt.Replica || vars[InstanceKeyVar] != "") { |
| err := b.settleClaimInstanceEnv(rt) |
| - if !errors.Is(err, errInstanceEnvNeedsWrite) { |
| + switch { |
| + case err == nil: |
| + return nil |
| + case errors.Is(err, os.ErrNotExist): |
| + // No own env file: the token and the key both come from the |
| + // rendered configuration. A configured key is the app's and |
| + // is not the platform's to make durable (ruling 5), and an |
| + // absent file is not created here (ruling 7), so the launch |
| + // proceeds as it did before the key existed. |
| + return nil |
| + case !errors.Is(err, errInstanceEnvNeedsWrite): |
| return err |
| } |
| } |
| @@ -3877,19 +4057,31 @@ Replace the body of `EnsureClaimAdminToken` from the early return to its end: |
| } |
| paths := b.pathsFor(rt) |
| return authorize(ctx, func() error { |
| - if paths.layout == LayoutNested { |
| - if err := EnsureAdminTokenBeneath(paths.dataDir, rt.Host+".env"); err != nil { |
| + // Only a missing token is minted: one the rendered configuration |
| + // supplies is the one the launch delivers, and a second one in the |
| + // own file would shadow it. |
| + if vars["CARLOS_ADMIN_TOKEN"] == "" { |
| + if paths.layout == LayoutNested { |
| + if err := EnsureAdminTokenBeneath(paths.dataDir, rt.Host+".env"); err != nil { |
| + return err |
| + } |
| + } else if err := EnsureAdminToken(paths.envFile); err != nil { |
| return err |
| } |
| - } else if err := EnsureAdminToken(paths.envFile); err != nil { |
| - return err |
| } |
| - _, err := b.ensureInstanceKeyFor(rt, paths) |
| - return err |
| + // The one place the claim launch source names the host around the |
| + // key's error: the refusal itself does not, and every other launch |
| + // path wraps it the same way. |
| + if _, err := b.ensureInstanceKeyFor(rt, paths); err != nil { |
| + return fmt.Errorf("instance key for %s: %w", rt.Host, err) |
| + } |
| + return nil |
| }) |
| } |
| ``` |
| +(`"fmt"` and `"os"` join the imports.) |
| + |
| In `SnapshotClaimLaunch`, read the two layers, keep `instanceEnv`'s merge, pass a replica through Task 3b's gate with the layer from this same read (ruling 29; a claim launch has no control socket before O6), and record the delivered key. The identity is computed from `Env` as before, so it does not change: |
| ```go |
| @@ -4624,8 +4816,10 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Files:** |
| - Create: `internal/host/instancekeys.go` |
| -- Modify: `internal/host/agent.go:476-484` (the legacy branch's recovery closure) |
| -- Test: `internal/host/instancekeys_test.go` |
| +- Modify: `internal/host/agent.go:476-484` (the activator's start, now assembled as an `activatorStart`), `internal/host/layoutleg.go` (`activatorStart` and its `Run`) |
| +- Test: `internal/host/instancekeys_test.go` (the counter, and `TestActivatorStartBackfillsOnlyALegacyBoxBetweenRecoveryAndPublish` for the start sequence) |
| + |
| +**Built:** rev 5.1 wired the backfill inline in `runAgent`'s legacy branch, where neither the branch (a claim box never backfills, ruling 7) nor the order (after layout recovery, before the pass is published and before `svc.Start`) could be tested. By the ledger's ruling the start sequence is a named function, `activatorStart.Run` in layoutleg.go, assembled in `runAgent` from the sweep, the recovery, the registry's `List`, the exec backend, `armedLayoutPass.Store` and `svc.Start`; its tests pin the claim-box branch and the order. The backfill's counter checks the error before the outcome (a refused row returns `InstanceKeyPresent` with its error). |
| **Interfaces:** |
| - Consumes: Task 2's `(*activator.ExecBackend).BackfillInstanceKey` and `activator.InstanceKeyOutcome` values; `registry.Route`, `registry.BackingUnit`; `(*registry.Registry).List` (registry.go:882). |
| @@ -4779,23 +4973,39 @@ func backfillInstanceKeys(routes []registry.Route, b instanceKeyBackfiller) inst |
| } |
| ``` |
| -Wire it in `internal/host/agent.go`, in the legacy branch's recovery closure, between `recoverLayoutAtStart` and `armedLayoutPass.Store` (`eb` is the `*activator.ExecBackend` built at :331): |
| +Wire it through the activator's start sequence, a named function in `internal/host/layoutleg.go` so the branch and the order are tested (built, by ruling): |
| ```go |
| - recoverLayoutAtStart(ctx, pass, intent) |
| - // L21: every instance already on this box gets its key |
| - // after layout recovery has settled every journal and |
| - // before the layout pass is published or the activator |
| - // wakes anything, so nothing moves a file under it. |
| - if routes, err := reg.List(); err != nil { |
| - log.Printf("agent: instance keys: list routes: %v (each host's next start mints its key)", err) |
| - } else { |
| - backfillInstanceKeys(routes, eb) |
| - } |
| - armedLayoutPass.Store(pass) |
| +// activatorStart is the activator's start sequence, assembled in runAgent |
| +// ... |
| +type activatorStart struct { |
| + Claims bool // claims.Writes.Enabled: a claim box |
| + Sweep func(context.Context) error // the instance-unit sweep |
| + Recover func(context.Context, bool) *layout.Pass // builds the pass and recovers under it; the bool is whether the sweep succeeded |
| + Routes func() ([]registry.Route, error) // the registry's List |
| + Keys instanceKeyBackfiller // the exec backend |
| + Publish func(*layout.Pass) // armedLayoutPass.Store |
| + Start func() // the service's Start |
| +} |
| + |
| +func (s activatorStart) Run(ctx context.Context) { |
| + if s.Claims { |
| + s.Start() |
| + return |
| + } |
| + startActivator(ctx, s.Sweep, func(ctx context.Context, swept bool) { |
| + pass := s.Recover(ctx, swept) |
| + if routes, err := s.Routes(); err != nil { |
| + log.Printf("agent: instance keys: list routes: %v (each host's next start mints its key)", err) |
| + } else { |
| + backfillInstanceKeys(routes, s.Keys) |
| + } |
| + s.Publish(pass) |
| + }, s.Start) |
| +} |
| ``` |
| -Do not add it to the `claims.Writes.Enabled` branch (ruling 7). |
| +In `internal/host/agent.go`, the former `if claims.Writes.Enabled { svc.Start() } else { startActivator(...) }` becomes one `activatorStart{...}.Run(context.Background())` literal: `Claims: claims.Writes.Enabled`, the sweep and recovery closures as they were, `Routes: reg.List`, `Keys: eb`, `Publish: armedLayoutPass.Store`, `Start: svc.Start`. A claim box runs `Start` alone (ruling 7); a legacy box backfills after layout recovery has settled every journal and before the pass is published or the activator wakes anything, so nothing moves a file under it. |
| - [ ] **Step 4: Run the test, then the cross-compiles** |
| @@ -4995,6 +5205,10 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 7. start the replicas. |
| Do the same after editing a home's own `<host>.env` by hand, or configuring a per-host key on either side. Those are the changes the agent's fresh read cannot see. |
| + |
| + **Pre-roll check: no flat `<host>.env` is a symlink** (Task 2's build). The mint opens the flat own file with `O_NOFOLLOW`, so a symlink at `<data>/<host>.env` now fails that host's launch, where the admin-token path only logged. On every box: `find <data> -maxdepth 1 -name '*.env' -type l` must print nothing. A link found there was planted by a same-uid process (spec §7.2) or by hand; replace it with the real file (the instance stopped), and treat the token in it as compromised. |
| + |
| + **Pre-roll check: the agent can read the list** (Task 3b's build). `/etc/carlos/replica-home-keys` is 0640 root:carlos and the agent runs in the `carlos` group, so on every box with a replica row: `sudo -u <agent user> test -r /etc/carlos/replica-home-keys && echo ok` must print `ok`. An unreadable list verifies nothing, and every replica with a key is refused with a log line that names the file. |
| 2. **Merge and release.** Merging is Paul's call, as a squash on the hub, after CI is green on the tip. Cut a platform release by the README's "Releasing the platform" recipe. |
| 3. **Ship to every box, every region.** Use the platform roll recipe (build arm64, upload, `update-platform-binary.sh` per box, `roll-console.sh` after). The console needs nothing new, but rolls with the release as usual. |
| - **Roll every box before any configured key matters** (task 1 review, minor 6). While boxes run mixed versions, a move's readiness check (`movesreport.go:487`) fails for an instance whose own key is empty but which has a configured key: the old source reports the empty own key, and the new target reports the configured one. That is unlikely until keys are minted, but finish the roll before an owner configures a key on an app that moves. |
| @@ -5022,8 +5236,17 @@ Question 1 is decided (Paul, 2026-10-11). Questions 2 to 4 have provisional defa |
| Question 5 (restarts inside a claim session) is withdrawn: its premise was wrong (ruling 16). |
| +## Known gaps |
| + |
| +- **fsync after EIO on Linux (Task 2's review).** A mint whose file sync fails returns the error and the launch fails; the next ensure finds the key present and syncs it again before anything is launched with it. On Linux a page whose writeback failed once is marked clean when the error is reported, so that second fsync can succeed without the bytes ever reaching the disk, and the ensure reports a present key that is not durable. A process can then be started with a key a power cut takes back. This is every fsync-based guarantee on Linux, not this plan's: a rare non-durable key after EIO, accepted by the ledger's ruling (the comment in `ensureInstanceKey` is softened to match; no code). |
| +- **The residual reclassify window (Task 3's review, #273).** `ensureInstanceKeyFor` refuses a fenced or conflicting row on the paths the launch body classified, but a journal written after `pathsFor` and before the append still mints flat, and `instanceEnv` classifies a third time. Both sit inside the layout pass's fence (`FenceAndStopIfUnfenced`). The admin-token mint has the same window. Closing it means refusing at the top of the launch body: #273's. |
| +- **A replica or configured-key row stays at its mode (Task 4's review).** On the claim path a 0644 own file that holds only the token is not tightened: the key's ensure writes nothing on those rows, and the token's mode has never been the claim path's to set. Every launch of such a row enters the guard to do nothing. Not a regression. |
| +- **The non-linux `safefile.Write` fsyncs the file, not the directory** (Task 5's review). Boxes are Linux; dev only. |
| + |
| ## Follow-ups this plan does not build |
| +- **`RoleLaunch.Start` refuses by name until claim Plan 9 Group D.** `TestStartAdmittedRetriesReuseTheirInputsAndMaterializeFirst` (claimretry_instancekey_linux_test.go) pins the key's survival across a refused start by relying on that refusal and asserting `ErrClaimIsolationUnavailable`. When Group D makes `Start` launch, the test's `want` list and that assertion change: a tripwire to expect, not a regression. |
| +- **#273, the admin token's own holes.** Three things this plan fixed for the key and left for the token, filed together: the flat mint path follows a symlink at `<host>.env` (Task 2 opens with `O_NOFOLLOW`; the token's `Lstat` guard is check-then-open on the present path and absent on the mint path); the token is minted before the fenced/conflict refusal on a row reclassified mid-wake (Task 3 refuses inside `ensureInstanceKeyFor`; the claim closure mints the token first); and the residual reclassify window above. One fix: refuse at the top of each launch body and share the key's open. |
| - **O8, the home's platform key for replicas** (replica placements, #162; platform keys only, ruling 28). Until it lands, replicas run only with a verified configured key, or keyless with a control socket. |
| - **O7, then O6.** Both are other plans' work. O7 (moves' artifact lifecycle) ships first; O6 (the writer, on claim launches) ships only after it (ruling 24), so no record is ever written that moves cannot retire. |
| - **Low-entropy configured keys.** A `config` record publishes the SHA-256 of a configured key in the data bucket (ruling 9). For a random key that reveals nothing. For a short key an owner typed by hand, a data-bucket reader could test guesses against it. Rastrillo's docs could say that a configured `CARLOS_INSTANCE_KEY` should be random. |