| index d9904217b..566430113 100644 |
| --- a/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md |
| +++ b/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md |
| @@ -1119,7 +1119,8 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| |
| **Files:** |
| - Create: `internal/activator/beforeexec.go`, `internal/activator/beforeexec_test.go` |
| -- Modify: `internal/activator/activator.go` (`Service`; `doWake` between the restore (step 2) and the seed (step 3) ~L1186, the seed call, and the spawn arms), `internal/activator/claimwake.go` (`doClaimWake`, the same points), `internal/activator/backend_exec.go` (`StartExe`, `SeedExe`; `seedChild` gains `exe`), `internal/activator/backend_unit.go` (`SplitBackend` forwards), and the zdd spawn points listed in Step 6 |
| +- Modify: `internal/activator/activator.go` (`Service`; `doWake` between the restore (step 2) and the seed (step 3) ~L1186, the seed call, and the spawn arms), `internal/activator/claimwake.go` (`startAdmitted`'s `beforeStart`, Step 5b), `internal/activator/backend_exec.go` (`LiveIdentity`, `StartExe`, `SeedExe`; `seedChild` and `seedUnit` gain `exe`; `startRole`'s generation-0 branch), `internal/activator/backend_unit.go` (`SplitBackend` forwards), and the zdd spawn points listed in Step 6 |
| +- Modify: `internal/instancectl/server.go` (the `/start` parser admits `exe` for the generation-0 run role; `handleSeed` parses and validates `exe`; `parseStagedExe`), `internal/instancectl/client.go` (`StartExe`, `SeedExe`), `internal/instancectl/stagedexe_test.go` (Step 4a) |
| - Modify: `internal/host/buildstate.go` (`beforeExec`), `internal/host/agent.go` (wire `svc.BeforeExec`) |
| |
| **Interfaces:** |
| @@ -1132,11 +1133,14 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| |---|---|---| |
| | planned boot (zdd-5 T9) | `plan.Boot` | `startPinned` (zdd-3 T6) | |
| | pinned / hold wake (zdd-3 T6) | `*e.pin` | `startPinned` | |
| -| worker wake | `release.ResolveLive(optDir, host)`, once | `startWorker(…, &id)` (zdd-4 T9 execs `onto` by path) | |
| -| flag-off instance wake, claimed or not | `release.ResolveLive(optDir, host)`, once | `ExecBackend.StartExe(ctx, rt, id)` (new, below): zdd-3's spawn-only `spawnChildFrom` (flat) or `spawnUnitFrom` with a `StagedExe` (nested) on the canonical socket with generation 0, plus `startChildAt`'s existing health tail | |
| -| the seed, whichever wake | the wake's identity | `ExecBackend.SeedExe(ctx, rt, id)` (new): `seedChild` with the staged path in place of `b.liveLink(host)` | |
| +| worker wake | materialize, then `release.ResolveLive(optDir, host)`, once | `startWorker(…, &id)` (zdd-4 T9 execs `onto` by path) | |
| +| unclaimed instance wake, **flat** | materialize (`ensureLiveLink`, which keeps today's `Materialize` hook and its root-stage relay), then `release.ResolveLive`, once | `ExecBackend.StartExe(ctx, rt, id)`: zdd-3's spawn-only `spawnChildFrom(ctx, rt, sock, 0, id.StagedPath, tier)` plus `startChildAt`'s health tail | |
| +| unclaimed instance wake, **nested** | the same | `StartExe`: the original run unit started with the staged executable named. Root validates it and points the unit's `app` mount at it, with zdd-3 T9's `validateStagedExe` and `ensureHostExeAt` (the mechanism zdd-3 built for the legacy sidecar). This task extends that mechanism to the run role at generation 0 and to the seed unit (Step 4a). | |
| +| the seed, flat or nested | the wake's identity | `ExecBackend.SeedExe(ctx, rt, id)`: flat `seedChild` with the staged path in place of `b.liveLink(host)`; nested, the seed unit started with the staged executable named (Step 4a) | |
| +| **claimed** primary start | the claim inputs' pinned executable: `{Version: in.Version, StagedPath: in.Executable, Digest: in.ExecutableSHA256}` (`internal/host/claimwakeassembly.go`, chosen by `Inputs.Select` and opened by device and inode in `RoleLaunch`) | `s.RoleLauncher.Start(…, beforeStart)` (`claimwake.go` ~L224), with the build-state write composed into `beforeStart`, which runs after materialization and strict preparation, under the admitted claim, immediately before the exec, on the first start and every retry (Step 5b) | |
| +| claimed first-wake fallback | `policy.Fallback` | zdd-3 T10's fallback arm, which already records (Step 6) | |
| |
| -`doWake` and `doClaimWake` record that identity **once**, after `Restore` and **before the seed** (so a planned or pinned wake records before seeding too, which the first plan skipped). The spawn functions below still call `beforeExec` for spawns that do not come from a wake (a swap's successor, a fallback, a recycle), and skip the write when the wake already recorded the same build (`recordedBuild(ctx)`): |
| +`doWake` records that identity **once**, after `Restore` and **before the seed** (so a planned or pinned wake records before seeding too). The claimed path has no seed (it restores or installs an empty database, `ColdRestore`) and records in `beforeStart` instead. The spawn functions below still call `beforeExec` for spawns that do not come from a wake (a swap's successor, a fallback, a recycle), and skip the write when the wake already recorded the same build (`recordedBuild(ctx)`): |
| |
| | Spawn | Function (plan) | Build | |
| |---|---|---| |
| @@ -1278,7 +1282,7 @@ func TestANilHookIsToday(t *testing.T) { |
| } |
| ``` |
| |
| -Add to `fakeBackend` (if it lacks them): a `liveVersion` it returns from `LiveVersion(host)` and resolves for `ResolveLive` (a staged path ending in the version); `fresh` for `Restore`; `StartExe`/`SeedExe` that append `"start <version>"`/`"seed <version>"` to an event log and remember the last identity; `Start` and `Seed` (the live-link forms) that count into `liveStarts()`; `startCount()` (all starts), `seedCount()`, `lastExeVersion()` and `events()`. The test hook in each test appends `"record <build>"` to the same log (wrap `l.hook`). `newPlannedTestService` and `seededRoute` wrap zdd-5 T9's `planActivator` and `testRoute` with a seed. Add one more test per zdd chokepoint, each driving that path with the fixture zdd's own tests use for it and asserting `BeforeExec` saw the path's build before the backend's spawn count rose: `TestBeforeExecRunsBeforeAHoldSwap` (`Swap` with `ModeHold`, zdd-3 T6's fixture), `TestBeforeExecRunsBeforeAnOverlapSuccessor` (zdd-3 T4/T10's `swapFixture`), `TestBeforeExecRunsBeforeAWorkerRecycle` (zdd-4 T9's `workerFixture`, `recycleWorkerLocked`), `TestBeforeExecRunsBeforeAPlannedFallback` (zdd-5 T9's `planActivator` with `FirstWake` and a `Fallback`), and in each, a failing hook leaves the spawn count unchanged and the result a failure or `SwapAbandoned`. |
| +Add to `fakeBackend` (if it lacks them): a `liveVersion` it returns from `LiveVersion(host)` (so `New` wires `a.liveVer` to it) and resolves in `LiveIdentity` (a staged path ending in the version); `fresh` for `Restore`; `StartExe`/`SeedExe` that append `"start <version>"`/`"seed <version>"` to an event log and remember the last identity; `Start` and `Seed` (the live-link forms) that count into `liveStarts()`; `startCount()` (all starts), `seedCount()`, `lastExeVersion()` and `events()`. The test hook in each test appends `"record <build>"` to the same log (wrap `l.hook`). `newPlannedTestService` and `seededRoute` wrap zdd-5 T9's `planActivator` and `testRoute` with a seed. Add one more test per zdd chokepoint, each driving that path with the fixture zdd's own tests use for it and asserting `BeforeExec` saw the path's build before the backend's spawn count rose: `TestBeforeExecRunsBeforeAHoldSwap` (`Swap` with `ModeHold`, zdd-3 T6's fixture), `TestBeforeExecRunsBeforeAnOverlapSuccessor` (zdd-3 T4/T10's `swapFixture`), `TestBeforeExecRunsBeforeAWorkerRecycle` (zdd-4 T9's `workerFixture`, `recycleWorkerLocked`), `TestBeforeExecRunsBeforeAPlannedFallback` (zdd-5 T9's `planActivator` with `FirstWake` and a `Fallback`), and in each, a failing hook leaves the spawn count unchanged and the result a failure or `SwapAbandoned`. |
| |
| - [ ] **Step 2: Run them and see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/activator/ -run 'BuildStateIsWritten|FailedWriteBlocks|ExecsTheIdentityItRecorded|SeedRunsTheRecordedIdentity|PlannedWakeRecordsBeforeTheSeed|NilHookIsToday|BeforeExecRuns' -count=1`. Expected: FAIL to compile, `a.BeforeExec undefined`. |
| |
| @@ -1328,6 +1332,7 @@ func recordedBuild(ctx context.Context) string { |
| // (Astra I1 #5): exec exactly id.StagedPath, verified against id.Digest, |
| // never live/<host>. |
| type identityBackend interface { |
| + LiveIdentity(ctx context.Context, rt registry.Route) (release.StagedIdentity, error) |
| StartExe(ctx context.Context, rt registry.Route, id release.StagedIdentity) (Proc, string, error) |
| SeedExe(ctx context.Context, rt registry.Route, id release.StagedIdentity) error |
| } |
| @@ -1344,9 +1349,53 @@ and to `Service`: |
| BeforeExec func(ctx context.Context, host, build string) error |
| ``` |
| |
| -- [ ] **Step 4: The exec backend execs an identity.** In `internal/activator/backend_exec.go`, add `StartExe` and `SeedExe` to `ExecBackend` (and forward them on `SplitBackend`, as zdd-4 T8 forwards `StartWorkerStaged`): |
| - - `StartExe(ctx, rt, id)`: `release.VerifyStaged(id)` first (a refusal is a failed start, never a fallback to the link); then the flat path calls zdd-3's `spawnChildFrom(ctx, rt, sock, 0, id.StagedPath, tier)` and the nested path `spawnUnitFrom(ctx, rt, sock, 0, &instancectl.StagedExe{…from id…}, tier)`, with `sock` and `tier` exactly as `Start` derives them today (the canonical socket; `b.TierOf(rt, id.Version)`), followed by the health tail `startChildAt`/`startUnitAt` already run after their spawn (zdd-3 split those functions at the health wait, so the tail is a function to call, not code to copy). It returns `id.Version` as the spawned version. |
| - - `SeedExe(ctx, rt, id)`: `release.VerifyStaged(id)`, then `seedChild`'s body with `id.StagedPath` where it passes `b.liveLink(host)` (`backend_exec.go` ~L770: `seedChild` gains an `exe string` parameter; `Seed` calls it with the live link, unchanged). For a nested row, pass the staged executable the way zdd-3 T9 passes one to `startRole` (`exe *instancectl.StagedExe`), which root validates. |
| +- [ ] **Step 4: The exec backend execs an identity.** In `internal/activator/backend_exec.go`, add to `ExecBackend` (and forward on `SplitBackend`, as zdd-4 T8 forwards `StartWorkerStaged`): |
| + - `LiveIdentity(ctx, rt) (release.StagedIdentity, error)`: today's `b.ensureLiveLink(ctx, rt.Host)` (`backend_exec.go:1106`), unchanged, so a missing link goes through the `Materialize` hook and its root-stage relay; then `release.ResolveLive(b.OptDir, rt.Host)`, once (Astra I2 #6: the identity is captured only after materializing, so a cold default instance still wakes). |
| + - `StartExe(ctx, rt, id)`: `release.VerifyStaged(id)` first (a refusal is a failed start, never a fallback to the link). The flat path calls zdd-3's `spawnChildFrom(ctx, rt, sock, 0, id.StagedPath, tier)` then `startChildAt`'s health tail. The nested path calls `spawnUnitFrom(ctx, rt, sock, 0, &instancectl.StagedExe{Path: id.StagedPath, Version: id.Version, Digest: id.Digest}, tier)` then `startUnitAt`'s health tail; Step 4a makes the generation-0 run role honour that executable. `sock` and `tier` are what `Start` derives today (the canonical socket; `b.TierOf(rt, id.Version)`). It returns `id.Version` as the spawned version. |
| + - `SeedExe(ctx, rt, id)`: `release.VerifyStaged(id)`, then by layout. Flat: `seedChild` with `id.StagedPath` where it passes `b.liveLink(host)` (`seedChild` gains an `exe string` parameter; `Seed` passes the live link, unchanged). Nested: `seedUnit` gains `exe *instancectl.StagedExe` and calls `b.Control.SeedExe(ctx, rt.Host, tier.MemoryMax, *exe)` when it is set (Step 4a); `Seed` passes nil. |
| + |
| +- [ ] **Step 4a: Root runs a staged executable for the original run and seed units (Astra I2 #2).** zdd-3 Task 9 built the mechanism for generation starts and the legacy sidecar: `instancectl.StagedExe`, root's `validateStagedExe` (the host's own app's `VersionPath`, opened without following a symlink, the digest equal) and `ensureHostExeAt`, which points the original templates' `<ExeRoot>/<id>/app` mount at the staged file instead of the live link. Its parser deliberately refuses `exe` for a generation-0 run start. Reuse that mechanism, do not build a second one: |
| + - `internal/instancectl/server.go`, zdd-3's `/start` parsing: change `if req.gen == 0 && req.role != RoleSidecar {` to `if req.gen == 0 && req.role != RoleSidecar && req.role != RoleRun {`. The original-template path already validates `req.exe` and lays out with `ensureHostExeAt` (zdd-3 Task 9 Step 4). In that path's tier condition, keep the populated-slice exemption for the sidecar only: `if req.role == RoleSidecar && req.exe.Path != "" && s.slicePopulated(req.host) {`. zdd-3's `TestSidecarStartMountsTheValidatedStagedExe` ends by asserting a run-role start with `exe` answers 400; that assertion is this change's intended reversal, so replace its last block (from `// A run-role start on the original template names no executable.`) with the same query and `q.Set("digest", strings.Repeat("0", 64))`, expecting `http.StatusForbidden`: the run role may now name an executable, and root still refuses one that is not the host's own validated build. The seed, run and sidecar units share the one `app` mount, and each binds it when its unit starts, so a seed and run started on identity A run A even if a later flag-off sidecar start points the mount back at the live link. |
| + - `handleSeed` (`server.go` ~L377): parse `exe`, `version` and `digest` the same way (extract zdd-3's three lines into `parseStagedExe(q url.Values) StagedExe` and call it from both parsers). Before `s.refuseStart(w, req.host, unit)`, when `req.exe.Path != ""`: `validateStagedExe`, answering 403 on a refusal; then `s.ensureHostExeAt(req.host, req.exe)` inside `refuseStart`'s callback, as the run path does, so the seed unit binds the staged file. |
| + - `internal/instancectl/client.go`: `func (c *Client) StartExe(ctx context.Context, host string, role Role, memoryMax int64, x StagedExe) (string, error)` (zdd-3's `StartSidecarExe` body with `role` in place of `RoleSidecar`; `StartSidecarExe` becomes `return c.StartExe(ctx, host, RoleSidecar, memoryMax, x)`), and `func (c *Client) SeedExe(ctx context.Context, host string, memoryMax int64, x StagedExe) (SeedResponse, error)` (`Seed` with the three query values; a 403 is `ErrStagedExe`). |
| + - `internal/activator`: the interface `ExecBackend.Control` satisfies gains `StartExe` and `SeedExe`; `startRole`'s generation-0 branch, when `exe != nil`, calls `b.Control.StartExe(ctx, rt.Host, role, tier.MemoryMax, *exe)` instead of `b.Control.Start`. |
| + |
| + Tests, added to zdd-3's `internal/instancectl/stagedexe_test.go` (its fixture builds a `Server` with a fake `Systemctl`, an `OptDir` and `ExeRoot`): |
| + |
| +```go |
| +// Astra I2 #2: the original run unit binds the recorded file, and a repoint |
| +// of the live link after the start does not change it. |
| +func TestTheRunUnitBindsAValidatedStagedExe(t *testing.T) { |
| + s, c, opt := stagedExeFixture(t) // zdd-3 T9's fixture: server, client, opt dir with genHost's app staged at v1 and v2 |
| + x := stagedExeOf(t, opt, genHost, "v1") |
| + if _, err := c.StartExe(context.Background(), genHost, RoleRun, cgroup.TierDefault.MemoryMax, x); err != nil { |
| + t.Fatal(err) |
| + } |
| + repointLive(t, opt, genHost, "v2") |
| + if got := appMountTarget(t, s, genHost); got != x.Path { |
| + t.Fatalf("the run unit's app mount = %q, want the recorded %q", got, x.Path) |
| + } |
| + bad := x |
| + bad.Digest = strings.Repeat("0", 64) |
| + if _, err := c.StartExe(context.Background(), genHost, RoleRun, cgroup.TierDefault.MemoryMax, bad); !errors.Is(err, ErrStagedExe) { |
| + t.Fatalf("a wrong digest = %v, want ErrStagedExe", err) |
| + } |
| +} |
| + |
| +func TestTheSeedUnitBindsAValidatedStagedExe(t *testing.T) { |
| + s, c, opt := stagedExeFixture(t) |
| + x := stagedExeOf(t, opt, genHost, "v1") |
| + if _, err := c.SeedExe(context.Background(), genHost, cgroup.TierDefault.MemoryMax, x); err != nil { |
| + t.Fatal(err) |
| + } |
| + repointLive(t, opt, genHost, "v2") |
| + if got := appMountTarget(t, s, genHost); got != x.Path { |
| + t.Fatalf("the seed unit's app mount = %q, want the recorded %q", got, x.Path) |
| + } |
| +} |
| +``` |
| + |
| +`stagedExeFixture`, `stagedExeOf`, `repointLive` and `appMountTarget` are thin wrappers over what zdd-3 Task 9's tests already do (it stages builds, builds a `StagedExe` from `release.FileSHA256`, and reads the `app` link with `os.Readlink`); write them beside those tests by those names. And in `internal/activator`'s nested backend tests (zdd-2 T11's `delegatedBackend` fixture with a recording `Control` fake): `TestNestedStartExeNamesTheRecordedFile` and `TestNestedSeedExeNamesTheRecordedFile`, each recording identity A, repointing the live link to B, then calling `StartExe`/`SeedExe` and asserting the fake `Control` was asked for A's path and digest, never a plain `Start`/`Seed`. |
| |
| - [ ] **Step 5: The wake records one identity, then seeds and spawns it.** In `doWake`, directly before the comment `// 3. First-boot seed, strictly between the restore and the spawn`, add: |
| |
| @@ -1354,23 +1403,32 @@ and to `Service`: |
| // 2b. One executable identity for this wake (settings spec §4.2, Astra |
| // I1 #5): chosen now, recorded now (before the seed and the spawn, under |
| // the lease taken in step 1), and the only file this wake runs. |
| - id, idErr := a.wakeIdentity(e, host) |
| - if idErr == nil { |
| - idErr = a.beforeExec(ctx, host, id.Version) |
| - } |
| - if idErr != nil { |
| - w.stopOutcome.leaseReleased = lease.Release(ctx) == nil |
| - return w, fmt.Errorf("%s: %w", host, idErr) |
| + // A backend that cannot pin an executable (UnitBackend) keeps today's |
| + // calls and records nothing. |
| + ib, _ := a.backend.(identityBackend) |
| + var id release.StagedIdentity |
| + if ib != nil { |
| + var idErr error |
| + id, idErr = a.wakeIdentity(ctx, ib, e, rt) |
| + if idErr == nil { |
| + idErr = a.beforeExec(ctx, host, id.Version) |
| + } |
| + if idErr != nil { |
| + w.stopOutcome.leaseReleased = lease.Release(ctx) == nil |
| + return w, fmt.Errorf("%s: %w", host, idErr) |
| + } |
| + ctx = withRecordedBuild(ctx, id.Version) |
| } |
| - ctx = withRecordedBuild(ctx, id.Version) |
| ``` |
| |
| with, in `beforeexec.go`: |
| |
| ```go |
| // wakeIdentity is the one file a wake will run: a planned boot's or a pinned |
| -// wake's staged build, else the live link resolved once (release.ResolveLive). |
| -func (a *Service) wakeIdentity(e *actEntry, host string) (release.StagedIdentity, error) { |
| +// wake's staged build, else the live link, materialized first exactly as |
| +// Start does today (the Materialize hook and its root-stage relay, Astra I2 |
| +// #6), then resolved once (release.ResolveLive). |
| +func (a *Service) wakeIdentity(ctx context.Context, ib identityBackend, e *actEntry, rt registry.Route) (release.StagedIdentity, error) { |
| a.mu.Lock() |
| pin, plan := e.pin, e.plan |
| a.mu.Unlock() |
| @@ -1380,19 +1438,80 @@ func (a *Service) wakeIdentity(e *actEntry, host string) (release.StagedIdentity |
| case pin != nil: |
| return release.StagedIdentity{Version: pin.Build, StagedPath: pin.StagedPath, Digest: pin.Digest, Demand: pin.Demand}, nil |
| } |
| - return release.ResolveLive(a.optDir(), host) |
| + return ib.LiveIdentity(ctx, rt) |
| } |
| ``` |
| |
| -Then: in step 3, when the backend is an `identityBackend`, replace `a.backend.Seed(ctx, rt)` with `ib.SeedExe(ctx, rt, id)`; in step 4, replace the flag-off instance arm's `proc, spawnVer, err = a.backend.Start(ctx, rt)` with `proc, spawnVer, err = ib.StartExe(ctx, rt, id)`, and the worker arm's `a.startWorker(ctx, rt, env, nil)` with `a.startWorker(ctx, rt, env, &id)`. The planned and pinned arms already exec `plan.Boot`/`*e.pin` through `startPinned`, which now skips its own write for the recorded build. A backend that is not an `identityBackend` (`UnitBackend`: systemd-unit routes, which plan 3 refuses settings on) keeps today's calls, and the wake records nothing for it (`wakeIdentity` is skipped when `ib` is nil), because nothing there could exec a pinned file. |
| +Then: in step 3, when the backend is an `identityBackend`, replace `a.backend.Seed(ctx, rt)` with `ib.SeedExe(ctx, rt, id)`; in step 4, replace the instance arm's `proc, spawnVer, err = a.backend.Start(ctx, rt)` with `proc, spawnVer, err = ib.StartExe(ctx, rt, id)` (flat and nested alike: `StartExe` dispatches by layout as `Start` does), and the worker arm's `a.startWorker(ctx, rt, env, nil)` with `a.startWorker(ctx, rt, env, &id)`. The planned and pinned arms already exec `plan.Boot`/`*e.pin` through `startPinned`, which now skips its own write for the recorded build. A backend that is not an `identityBackend` (`UnitBackend`: systemd-unit routes, which plan 3 refuses settings on) keeps today's calls, and the wake records nothing for it, because nothing there could exec a pinned file. |
| |
| -Make the same edits in `doClaimWake` at its restore-then-seed-then-spawn point. |
| +Add to Step 1's tests: |
| + |
| +```go |
| +// Astra I2 #6: a cold default instance with no live link still wakes: the |
| +// Materialize hook runs before the identity is captured, and the identity is |
| +// the build it materialized. |
| +func TestADefaultWakeMaterializesBeforeItCapturesTheIdentity(t *testing.T) { |
| + a, b := newTestService(t) |
| + b.liveVersion = "" // no live link yet |
| + b.materializeTo = "v1" |
| + var l execLog |
| + a.BeforeExec = l.hook |
| + done, err := a.beginRequest(context.Background(), testRoute("one.example.com")) |
| + if err != nil { |
| + t.Fatalf("a cold wake failed before materializing: %v", err) |
| + } |
| + done() |
| + if b.materializeCount() != 1 || len(l.calls) != 1 || l.calls[0] != "one.example.com@v1" || b.lastExeVersion() != "v1" { |
| + t.Fatalf("materialized %d, recorded %v, ran %q", b.materializeCount(), l.calls, b.lastExeVersion()) |
| + } |
| +} |
| +``` |
| + |
| +`fakeBackend.LiveIdentity` sets `liveVersion = materializeTo` when the link is absent, counts that as a materialize, and returns `{Version: liveVersion, StagedPath: "/opt/carlos/versions/" + liveVersion, Digest: "sha-" + liveVersion}`; with the I1 version (resolve without materializing) the wake fails on the missing link and the test fails. |
| + |
| +- [ ] **Step 5b: The claimed primary start records in `beforeStart` (Astra I2 #3).** `doClaimWake` has no restore-seed-spawn sequence of its own: it admits and delegates to `startAdmitted` (`claimwake.go` ~L131), whose loop runs `s.Materialize`, prepares the `RoleLauncher`, and execs through `s.RoleLauncher.Start(ctx, "main", s.CurrentAttempt(), s.Build, beforeStart)` (~L224). The executable it runs is already pinned: the claim inputs' `in.Executable` and `in.ExecutableSHA256` (`internal/host/claimwakeassembly.go`), with `ClaimStart.Version` naming its build, and `RoleLaunch` opens the file by device and inode. The launcher already execs exactly that file, so this path needs no new exec mechanism: the build state records `s.Version`, the build of the pinned file, and the write goes into the pre-start callback, after materialization and strict preparation, under the admitted claim, immediately before each exec (the first start and every retry): |
| + |
| +```go |
| + beforeStart := func() error { |
| + // Settings spec §4.2: the build state names this build before |
| + // it runs against the data; a failed write refuses the start. |
| + if err := a.beforeExec(ctx, e.host, s.Version); err != nil { |
| + return err |
| + } |
| + return budget.start(ctx, s.OpenProbe) |
| + } |
| +``` |
| + |
| +replacing `beforeStart := func() error { return budget.start(ctx, s.OpenProbe) }`. zdd-3 Task 10's generation and fallback spawns in the same loop reuse `beforeStart` for the primary build, and the fallback arm records its own build (Step 6). Test, beside the claimed-wake tests (`claimwake_test.go`'s fake `ClaimStart` and `RoleLaunch`): |
| + |
| +```go |
| +// Astra I2 #3: the claimed primary start records its build before every |
| +// exec, retries included, and a failed write refuses the exec. |
| +func TestAClaimedStartRecordsBeforeEachExec(t *testing.T) { |
| + a, start := claimedFixture(t, "v4") // claimwake_test.go's admitted fixture; ClaimStart.Version "v4"; first start fails, second succeeds |
| + var l execLog |
| + a.BeforeExec = l.hook |
| + if _, err := a.doClaimWake(context.Background(), a.entryFor("one.example.com")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(l.calls) != start.execs() || start.execs() != 2 || l.calls[0] != "one.example.com@v4" { |
| + t.Fatalf("records %v for %d execs; want one v4 record before each", l.calls, start.execs()) |
| + } |
| + b, start2 := claimedFixture(t, "v4") |
| + b.BeforeExec = (&execLog{fail: true}).hook |
| + if _, err := b.doClaimWake(context.Background(), b.entryFor("one.example.com")); err == nil || start2.execs() != 0 { |
| + t.Fatalf("a failed build-state write still execed (%d), err %v", start2.execs(), err) |
| + } |
| +} |
| +``` |
| + |
| +`claimedFixture` and `entryFor` wrap the admitted-claim setup `claimwake_test.go` uses; `start.execs()` counts calls into the fake launcher's exec after `beforeStart` returned nil. |
| |
| - [ ] **Step 6: The zdd chokepoints.** Add, as the first statement after each function's argument checks (each skips the write when the wake already recorded the same build): |
| - zdd-3 `startPinned(ctx, rt, pin, …)`: `if err := a.beforeExec(ctx, rt.Host, pin.Build); err != nil { return nil, "", pin, err }`. |
| - zdd-3 `spawnSuccessor(ctx, gb, e, rt, work, g)`: `if err := a.beforeExec(ctx, rt.Host, g.Build); err != nil { return g, nil, err }`. Its caller in `swapOverlap` already maps a spawn error to the abandoned or failed result the zdd plans define; check that an `errBuildStateUnwritten` maps to `SwapAbandoned` (not a build failure, which would write a failure record) and add `errors.Is(spawnErr, errBuildStateUnwritten)` to that mapping if not. |
| - zdd-3 `startAdmitted`'s fallback arm (`if policy.Fallback != nil && !ready && …`): before the fallback's spawn, `if err := a.beforeExec(ctx, rt.Host, policy.Fallback.Build); err != nil { … return the same error the arm returns for a failed fallback spawn … }`. |
| - - zdd-4 `startWorker(ctx, rt, env, onto)`: if `onto == nil`, resolve first: `id, err := release.ResolveLive(a.optDir(), rt.Host); if err == nil { onto = &id }`; then `build := ""; if onto != nil { build = onto.Version }; if err := a.beforeExec(ctx, rt.Host, build); err != nil { return nil, release.StagedIdentity{}, err }`, and pass `onto` on. (Passing the resolution as `onto` makes the exec use exactly the build recorded.) |
| + - zdd-4 `startWorker(ctx, rt, env, onto)`: if `onto == nil`, materialize and resolve first, the same order as `wakeIdentity` (Astra I2 #6): `if ib, ok := a.backend.(identityBackend); ok { id, err := ib.LiveIdentity(ctx, rt); if err != nil { return nil, release.StagedIdentity{}, err }; onto = &id }`; then `build := ""; if onto != nil { build = onto.Version }; if err := a.beforeExec(ctx, rt.Host, build); err != nil { return nil, release.StagedIdentity{}, err }`, and pass `onto` on. (Passing the resolution as `onto` makes the exec use exactly the build recorded.) |
| |
| zdd-5's `spawnPlanned`, `spawnPlannedWorker` and `ReplaceWorker` reach the exec only through `startPinned` and `startWorker`, so they need nothing of their own; the tests in Step 1 prove it. |
| |
| @@ -1446,13 +1565,13 @@ func (b *buildStates) beforeExec(ctx context.Context, host, build string) error |
| |
| In `runAgent`, where the activator `Service` is built, set `svc.BeforeExec = bs.beforeExec` when `bs != nil`. A box with no replica bucket runs no hibernating instances and leaves it nil. |
| |
| -- [ ] **Step 8: Run the packages.** Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test -race ./internal/activator/ ./internal/host/ -count=1`. Expected: PASS, every zdd test included. |
| +- [ ] **Step 8: Run the packages.** Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test -race ./internal/activator/ ./internal/host/ ./internal/instancectl/ -count=1`. Expected: PASS, every zdd test included, with Step 4a's one changed assertion in `TestSidecarStartMountsTheValidatedStagedExe`. |
| |
| - [ ] **Step 9: Commit.** |
| |
| ```bash |
| -cd $W && gofmt -l internal/activator internal/host && go vet ./internal/activator/ ./internal/host/ |
| -git add internal/activator/ internal/host/ |
| +cd $W && gofmt -l internal/activator internal/host internal/instancectl && go vet ./internal/activator/ ./internal/host/ ./internal/instancectl/ |
| +git add internal/activator/ internal/host/ internal/instancectl/ |
| git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 5: the build state is written under the lease before every spawn |
| |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| @@ -1785,16 +1904,81 @@ An unreadable tag list leaves the map empty: every head reads as untagged, so a |
| if stop { |
| return |
| } |
| - if ptr == nil { |
| - // The gates below read the pointer for its version and promoted |
| - // time; a head-less decision stands in the way heldProofAdmit's |
| - // does (materializeheld.go). |
| - ptr = &ChannelPointer{Version: ver} |
| + } |
| + // sel is the build this pass installs, as the gates below read it |
| + // (Astra I2 #8): the head itself when the decision is the head, else a |
| + // pointer naming only the selected build, the way heldProofAdmit's |
| + // stand-in does (materializeheld.go). The proof gate (admit and bind), |
| + // D1, D2 and the refusal records judge the build that would run, never |
| + // a head the decision passed over. ptr stays the real head, nil or not, |
| + // for the staging digest below. |
| + sel := ptr |
| + if ptr == nil || ver != ptr.Version { |
| + sel = &ChannelPointer{Channel: channel, Version: ver} |
| + } |
| +``` |
| + |
| +The `read live link` line is today's, kept verbatim. Every later use of `ptr` in `adoptHost` becomes `sel`: the `Hold` call's `ptr.FastTrack` (false for a selected non-head build, which is right: the waiver belongs to one promotion of the head), `r.Blocks(ptr)` in both D1 checks, `cfg.proof.admit(ctx, s, optDir, rt, sel, cur)`, the D2 refusal's `PointerPromotedAt: sel.PromotedAt`, and `cfg.proof.bind(optDir, rt, sel, verdict, cur, now)`. The canary nag stays inside the `ptr != nil` branch above. Also, in `PassFilter`'s two loops, call `adoptHost` for a route even when its group's pointer is nil (they already do: they look the pointer up and pass it, nil or not). |
| + |
| +Add to Step 1's tests (with `encoding/json`, `fmt`, `os` and `path/filepath` in the imports): |
| + |
| +```go |
| +// Astra I2 #8: the proof gate checks the build the decision selected, not |
| +// the channel head. Pinned to v1 with the head at v2: a proof for v2 alone |
| +// installs nothing, and a proof for v1 installs v1. |
| +func TestProofGateSeesTheSelectedBuild(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt, reg, bake := t.TempDir(), testRegistry(t), t.TempDir() |
| + for _, v := range []string{"v1", "v2"} { |
| + shipRelease(t, s, "bab", "km", v, adoptPlatform, []byte(v)) |
| + if err := Promote(ctx, s, priv, "bab", "km", "canary/x", v); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + settingsRoute(t, reg, "a.example.com", UpdateSettings{Updates: UpdatesAuto, Channel: "canary/x", Pin: "v1"}, BuildStateCopy{Known: true}) |
| + prover, _, proverPub := proverKey(t) // proofgate_test.go |
| + policy := filepath.Join(t.TempDir(), "proof-policy.json") |
| + if err := os.WriteFile(policy, []byte(fmt.Sprintf(`{"schema":1,"require":[{"app":"km","prover":"dev","key":%q}]}`, proverPub)), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + gate := NewProofGate(policy, bake) |
| + prove := func(v string) { |
| + sha := etagOf([]byte(v)) |
| + p, err := SignProof(prover, Proof{App: "km", Platform: adoptPlatform, SHA256: sha, Version: v, Prover: "dev", |
| + ActiveSince: time.Now().Add(-time.Hour), ProvenAt: time.Now()}) |
| + if err != nil { |
| + t.Fatal(err) |
| } |
| + b, _ := json.Marshal(p) |
| + if _, err := s.Put(ctx, ProofKey("bab", "km", sha, "dev"), b, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + pass := func() { |
| + var out bytes.Buffer |
| + withCapturedStdout(t, &out, func() { |
| + if err := Pass(ctx, reg, s, pub, opt, nil, BakeDir(bake), WithProofGate(gate), WithUpdateSettings(nil, true)); err != nil { |
| + t.Fatal(err) |
| + } |
| + }) |
| + } |
| + prove("v2") // the head is proven, the pinned build is not |
| + pass() |
| + if v, _ := LiveVersion(opt, "a.example.com"); v == "v1" { |
| + t.Fatal("the pinned v1 went live on the head's proof") |
| + } |
| + prove("v1") |
| + gate.clear("a.example.com") // drop the 30-second miss cache |
| + pass() |
| + if v, _ := LiveVersion(opt, "a.example.com"); v != "v1" { |
| + t.Fatalf("live %q, want the proven pin v1", v) |
| } |
| +} |
| ``` |
| |
| -The `read live link` line is today's, kept verbatim. Every later use of `ptr` then sees either the real head or that stand-in. The D1 refusal check, `Hold` and `ptr.FastTrack` already only apply when the decision is the head (next paragraph). Also, in `PassFilter`'s two loops, call `adoptHost` for a route even when its group's pointer is nil (they already do: they look the pointer up and pass it, nil or not). |
| +Without `sel`, the second pass admits on v2's proof and then `bind` hashes v2's path, which this box never staged, so v1 never goes live and the test fails; with the I1 stand-in only (applied when `ptr == nil`) the same happens, because the pointer here is readable. |
| |
| and add the helper at the end of the file: |
| |
| @@ -1881,7 +2065,7 @@ func settingsTarget(ctx context.Context, s Store, optDir string, bake *FirstSeen |
| |
| The `dec.Version == cur` return keeps a held instance's live link where it is with no stage and no restart. Then make the rest of `adoptHost` act on the decision: |
| - The existing `if hold := Hold(…ver…)` block runs only when `!cfg.settings || dec.Bake || dec.Reason == ""` (a pin, a resolved T and a recovery waive bake; the decision already applied it when it said `Bake`). Replace its condition accordingly. |
| - - The existing `D1` refusal check (`r.Blocks(ptr)`) compares against the pointer's version and `PromotedAt`; for a decision whose `ver != ptr.Version`, skip it (the failure book decided). |
| + - The existing `D1` refusal check (now `r.Blocks(sel)`) compares against the pointer's version and `PromotedAt`; when `sel != ptr` (the decision is not the head), skip it (the failure book decided). |
| - Replace the staging call with evidence: |
| |
| ```go |
| @@ -1890,7 +2074,11 @@ The `dec.Version == cur` return keeps a held instance's live link where it is wi |
| case cfg.settings && dec.Evidence == EvidenceRecord: |
| m, err = stageAuthorized(ctx, s, cfg.pub, optDir, RouteAccount(rt), rt.App, ver, rt.Host) |
| default: |
| - m, err = ensureStagedDigest(ctx, s, optDir, RouteAccount(rt), rt.App, ver, rt.Host, ptr.DigestOf(ver)) |
| + digest := "" |
| + if ptr != nil { |
| + digest = ptr.DigestOf(ver) // the real pointer: its head or history digest |
| + } |
| + m, err = ensureStagedDigest(ctx, s, optDir, RouteAccount(rt), rt.App, ver, rt.Host, digest) |
| } |
| ``` |
| |
| @@ -1976,10 +2164,10 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| - Modify: `internal/activator/activator.go` (`Service`; `doWake` and `claimwake.go` `doClaimWake` before Task 5's `beforeExec` block), `internal/host/buildstate.go` (`heldTarget`), `internal/host/agent.go` |
| |
| **Interfaces:** |
| -- Consumes: `release.ReadConsumed`, `ParseBuildStateCopy`, `ParseUpdateSettings`, `Watermark`, `LiveVersion`, `VersionPath`, `FileSHA256`; zdd-3 `actEntry.pin`, `Gen`, `GenerationsOn`; `stageRequester` (`agent.go:1357`). |
| -- Produces: `Service.HeldTarget func(host string) (build string, held bool, unknown bool)`; `Service.RequestStage func(host, build string)`; `func (a *Service) wakeCheck(ctx context.Context, e *actEntry, host string) error`; `var errBuildUnknown, errStaleLink error`; host `func (b *buildStates) heldTarget(host string) (string, bool, bool)`. |
| +- Consumes: `release.ReadConsumed`, `ParseBuildStateCopy`, `ParseUpdateSettings`, `Watermark`, `ReasonFailed`; `Service.liveVer`, `Service.Routes`; zdd-3 `actEntry.pin` (read only: the check never sets it); zdd-5 `Planner`, `WakePlan`, `planActivator`; Task 5's `LiveIdentity` and step 2b; `stageRequester` (`agent.go:1357`). |
| +- Produces: `Service.HeldTarget func(host string) (build string, held bool, unknown bool)`; `Service.RequestStage func(host, build string)`; `func (a *Service) wakeCheck(ctx context.Context, host string) (string, error)`; `func heldIdentityOK(target string, id release.StagedIdentity, plan WakePlan, rt registry.Route) bool`; `var errBuildUnknown, errStaleLink error`; host `func (b *buildStates) heldTarget(host string) (string, bool, bool)`. |
| |
| -§4.3 rule 3: before it spawns a held instance, the agent compares the installed build with its target (a pending request's T, else `last_executed`). If they differ it never runs the installed build. With generations on, it runs the target from its staged path (a pinned wake, zdd-3's `e.pin`); otherwise it touches the stage relay and waits, within the wake's own deadline, for root to repoint. An unknown build fails the wake with `settings.box.reason.unknown`. |
| +§4.3 rule 3: before it spawns a held instance, the agent compares the installed build with its target, in root's rule order (a pin, then a pending request's T, then `last_executed`; Astra I2 #7). If they differ it never runs the installed build: it touches the stage relay and waits, within the wake's own deadline, for root to install the target. It never pins the target itself (Astra I2 #4): root's `Target` applies the failure book before it installs anything, and the wake then goes through the scheduler's failure-aware decision (zdd-5's `Planner`, `decideWake`: `HeldDown` refuses, `Blocks` boots the good build or refuses) exactly as any other wake does. When root's own decision for the target is `failed`, the agent does not wait for an install root will not make: the wake is not held and the failure-aware planner decides. After the identity is chosen (Task 5), the wake refuses unless it is the target or the planner's recovery boot. An unknown build fails the wake with `settings.box.reason.unknown`. |
| |
| - [ ] **Step 1: Write the failing tests.** Create `internal/activator/wakecheck_test.go`, on `newTestService`: |
| |
| @@ -2051,11 +2239,72 @@ func TestAnAutoInstanceIsNotChecked(t *testing.T) { |
| t.Fatal("an auto instance's wake changed") |
| } |
| } |
| + |
| +// stampRoutes is the registry row the wake re-reads (RouteLookup); root's |
| +// repoint stamps Version, as adoption does. |
| +type stampRoutes struct { |
| + mu sync.Mutex |
| + rt registry.Route |
| +} |
| + |
| +func (s *stampRoutes) Get(string) (*registry.Route, error) { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + rt := s.rt |
| + return &rt, nil |
| +} |
| + |
| +func (s *stampRoutes) stamp(v string) { s.mu.Lock(); s.rt.Version = v; s.mu.Unlock() } |
| + |
| +// Astra I2 #4: the held wake never pins its target itself. Generations are |
| +// on and the target v1 is staged on this box (the case where the I1 wake |
| +// check set e.pin and skipped the planner), but the failure book blocks v1 |
| +// here: the planner, which sees root's install, boots the good build v0. |
| +func TestAHeldWakeGoesThroughTheFailureAwarePlan(t *testing.T) { |
| + good := release.StagedIdentity{Version: "v0", StagedPath: "/opt/carlos/versions/v0/blog", Digest: "sha-v0"} |
| + a, b, _ := planActivator(t, WakePlan{}) // zdd-5 T9's fixture: generations on, a recording GenBackend |
| + routes := &stampRoutes{rt: testRoute("one.example.com")} |
| + routes.rt.Version, routes.rt.GoodVersion = "v3", "v0" |
| + a.Routes = routes |
| + b.liveVersion = "v3" |
| + b.stage("v1") // the target's staged copy exists |
| + var planned []string |
| + a.Planner = func(_ context.Context, rt registry.Route) WakePlan { |
| + planned = append(planned, rt.Version) |
| + return WakePlan{Boot: good} // decideWake's recovery when Blocks(v1) |
| + } |
| + a.HeldTarget = func(string) (string, bool, bool) { return "v1", true, false } |
| + a.RequestStage = func(host, build string) { b.liveVersion = build; routes.stamp(build) } |
| + done, err := a.beginRequest(context.Background(), routes.rt) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + done() |
| + if len(planned) != 1 || planned[0] != "v1" || b.lastStartedVersion() != "v0" { |
| + t.Fatalf("planner saw %v, started %q; want one plan on root's install v1, booting the good v0", planned, b.lastStartedVersion()) |
| + } |
| +} |
| + |
| +// Astra I2 #4, the race half: root installs something other than the target |
| +// between the wait and the identity; the wake refuses rather than run it. |
| +func TestAHeldWakeRefusesAnIdentityThatIsNotItsTarget(t *testing.T) { |
| + a, b := newTestService(t) |
| + b.liveVersion = "v3" |
| + a.HeldTarget = func(string) (string, bool, bool) { return "v1", true, false } |
| + a.RequestStage = func(string, string) { b.liveVersion = "v1" } |
| + b.onLiveIdentity = func() { b.liveVersion = "v2" } // a repoint after the wait |
| + if _, err := a.beginRequest(context.Background(), testRoute("one.example.com")); !errors.Is(err, errStaleLink) { |
| + t.Fatalf("err = %v, want errStaleLink", err) |
| + } |
| + if b.startCount() != 0 { |
| + t.Fatal("the wake ran a build that is not its target") |
| + } |
| +} |
| ``` |
| |
| -`lastStartedVersion()` is Task 5's `lastExeVersion()` (the version of the last `StartExe` or `StartStaged`). |
| +`lastStartedVersion()` is Task 5's `lastExeVersion()` (the version of the last `StartExe` or `StartStaged`). Add to `fakeBackend`: `stage(v string)` (records v as staged; `StartStaged` and the I1 shortcut's existence check see it) and `onLiveIdentity func()`, run inside `LiveIdentity` before it resolves. Add `sync`, `registry` and `release` to the imports. |
| |
| -- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/activator/ -run 'HeldWake|UnknownBuildNeverWakes|AutoInstanceIsNotChecked' -count=1`. Expected: FAIL to compile. |
| +- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/activator/ -run 'HeldWake|UnknownBuildNeverWakes|AutoInstanceIsNotChecked' -count=1`. Expected: FAIL to compile. (`TestAHeldWakeGoesThroughTheFailureAwarePlan` needs zdd-5 T9's `planActivator`, which Task 0 confirms is on the base.) |
| |
| - [ ] **Step 3: Create `internal/activator/wakecheck.go`:** |
| |
| @@ -2077,33 +2326,26 @@ var ( |
| ) |
| |
| // wakeCheck is settings spec §4.3 rule 3's wake half. For a held instance it |
| -// refuses to run an installed build that differs from the target (a pending |
| -// request's T, else last_executed): with generations on it runs the target's |
| -// staged copy by path; otherwise it asks root to install it and waits within |
| -// the wake's own deadline. An unknown build never wakes. |
| -func (a *Service) wakeCheck(ctx context.Context, e *actEntry, host string) error { |
| +// returns the target (a pin, else a pending request's T, else last_executed) |
| +// once root has installed it: it asks root and waits within the wake's own |
| +// deadline. It never pins the target (Astra I2 #4): what boots is still |
| +// chosen by the failure-aware planner, or by the live identity Task 5 |
| +// captures, and checked against the target by heldIdentityOK. "" means the |
| +// instance is not held. An unknown build never wakes. |
| +func (a *Service) wakeCheck(ctx context.Context, host string) (string, error) { |
| if a.HeldTarget == nil { |
| - return nil |
| + return "", nil |
| } |
| target, held, unknown := a.HeldTarget(host) |
| if !held { |
| - return nil |
| + return "", nil |
| } |
| if unknown || target == "" { |
| - return errBuildUnknown |
| + return "", errBuildUnknown |
| } |
| - installed, _ := a.liveBuild(host) |
| + installed, _ := a.liveVer(host) |
| if installed == target { |
| - return nil |
| - } |
| - if a.GenerationsOn() { |
| - path := release.VersionPath(a.optDir(), e.route.Account, target, e.route.App) |
| - if sum, err := release.FileSHA256(path); err == nil { |
| - a.mu.Lock() |
| - e.pin = &Gen{Build: target, StagedPath: path, Digest: sum} |
| - a.mu.Unlock() |
| - return nil |
| - } |
| + return target, nil |
| } |
| if a.RequestStage != nil { |
| a.RequestStage(host, target) |
| @@ -2111,19 +2353,35 @@ func (a *Service) wakeCheck(ctx context.Context, e *actEntry, host string) error |
| tick := time.NewTicker(250 * time.Millisecond) |
| defer tick.Stop() |
| for { |
| - if v, _ := a.liveBuild(host); v == target { |
| - return nil |
| + if v, _ := a.liveVer(host); v == target { |
| + return target, nil |
| } |
| select { |
| case <-ctx.Done(): |
| - return fmt.Errorf("%w: %s is on %s, wants %s", errStaleLink, host, installed, target) |
| + return "", fmt.Errorf("%w: %s is on %s, wants %s", errStaleLink, host, installed, target) |
| case <-tick.C: |
| } |
| } |
| } |
| + |
| +// heldIdentityOK reports whether a held wake may run id: its target, or the |
| +// good build the planner chose as recovery because the failure book blocks |
| +// the target (zero-downtime failure records outrank every rule, §4.3). |
| +func heldIdentityOK(target string, id release.StagedIdentity, plan WakePlan, rt registry.Route) bool { |
| + if target == "" || id.Version == target { |
| + return true |
| + } |
| + return plan.planned() && !plan.FirstWake && id.Version == rt.GoodVersion |
| +} |
| ``` |
| |
| -`e.route.Account` must be the account sqid `VersionPath` takes; use `release.RouteAccount(e.route)` if `Account` can hold a name. Add `HeldTarget` and `RequestStage` to `Service` with doc comments, and call `if err := a.wakeCheck(ctx, e, host); err != nil { … return w, err }` in `doWake` and `doClaimWake` before Task 5's `beforeExec` block (after the lease, so a held instance's wake is serialized like any other; release the lease on error as the restore-failure path does). If the wake path sets `e.pin` itself before this point (zdd-5's planning), run the check before the planning block instead, so the check's pin wins. |
| +(add `registry` to the imports). `a.liveVer` is the `LiveVersion` reader `New` already wires (`activator.go` ~L707); the fakes set it through `fakeBackend.LiveVersion`. Add `HeldTarget` and `RequestStage` to `Service` with doc comments. |
| + |
| +Call sites in `doWake` (Astra I2 #4): |
| + 1. After the layout refusal (step 0.5) and **before** zdd-5's planning block, when `e.pin == nil` (a hold swap's pinned wake was chosen by the swap from root's install): `held, err := a.wakeCheck(ctx, host); if err != nil { return w, fmt.Errorf("%s: %w", host, err) }`. No lease is held yet. When `held != "" && a.Routes != nil`, re-read the row exactly as step 0 does (`fresh, err := a.Routes.Get(host)`, `rt = *fresh`, `e.route = rt` under `a.mu`) so the planner sees the build root just installed. |
| + 2. In Task 5's step 2b, after `wakeIdentity` succeeds: `if !heldIdentityOK(held, id, e.plan, rt) { idErr = fmt.Errorf("%w: %s would run %s, wants %s", errStaleLink, host, id.Version, held) }`, before `beforeExec`, so the refusal releases the lease the way 2b's other errors do and records nothing. |
| + |
| +In `doClaimWake`, the same: the check before zdd-5's claimed planning block, the route re-read, and in Task 5's Step 5b `beforeStart`, before `a.beforeExec`: `if held != "" && s.Version != held && !(e.plan.planned() && !e.plan.FirstWake && s.Version == rt.GoodVersion) { return fmt.Errorf("%w: %s would run %s, wants %s", errStaleLink, e.host, s.Version, held) }`. |
| |
| - [ ] **Step 4: The agent's answer.** Add to `internal/host/buildstate.go`: |
| |
| @@ -2142,17 +2400,30 @@ func (b *buildStates) heldTarget(host string) (string, bool, bool) { |
| } |
| bs, _ := release.ParseBuildStateCopy(rt.BuildState) |
| c, _ := release.ReadConsumed(b.optDir, host) |
| + // Root's order (Target, §4.3): a pin outranks a pending request (Astra |
| + // I2 #7). When root's own decision found the target failed, root will |
| + // not install it: the wake is not held, and the failure-aware planner |
| + // decides what runs (Astra I2 #4). |
| + rootFailed := func(v string) bool { |
| + return c != nil && c.Reason == string(release.ReasonFailed) && c.Detail == v |
| + } |
| + held := func(v string) (string, bool, bool) { |
| + if rootFailed(v) { |
| + return "", false, false |
| + } |
| + return v, true, false |
| + } |
| + if u.Pin != "" { |
| + return held(u.Pin) |
| + } |
| if u.InstallRequest > release.Watermark(c, bs.State) { |
| if c != nil && c.Resolved != nil && c.Resolved.ID == u.InstallRequest { |
| - return c.Resolved.Target, true, false |
| + return held(c.Resolved.Target) |
| } |
| if bs.State != nil && bs.State.Resolved != nil && bs.State.Resolved.ID == u.InstallRequest { |
| - return bs.State.Resolved.Target, true, false |
| + return held(bs.State.Resolved.Target) |
| } |
| } |
| - if u.Pin != "" { |
| - return u.Pin, true, false |
| - } |
| heldByRoot := c != nil && (c.Reason == string(release.ReasonManual) || c.Reason == string(release.ReasonMajor) || c.Reason == string(release.ReasonUnknown)) |
| if u.Updates != release.UpdatesManual && !heldByRoot { |
| return "", false, false |
| @@ -2161,7 +2432,7 @@ func (b *buildStates) heldTarget(host string) (string, bool, bool) { |
| return "", true, true |
| } |
| if bs.State != nil && bs.State.LastExecuted != "" { |
| - return bs.State.LastExecuted, true, false |
| + return held(bs.State.LastExecuted) |
| } |
| return "", false, false // no build state: whatever root installed is the instance's |
| } |
| @@ -2195,6 +2466,43 @@ func TestAMajorHoldStaysHeldWhileARequestBakes(t *testing.T) { |
| |
| With Task 6's fix (the baking head goes in `Baking`, `Reason` stays `major`) this passes; with the first plan's overwrite to `baking`, `heldByRoot` was false and the instance woke on its stale link. |
| |
| +```go |
| +// Astra I2 #7: a pin set after a request resolved outranks it, as in root's |
| +// Target. The wake targets the pin P, never the request's T. |
| +func TestAPinOutranksAPendingRequestAtTheWake(t *testing.T) { |
| + b, _, reg := testBuildStates(t) |
| + if err := reg.SetUpdateSettings("a.example.com", release.UpdateSettings{Updates: release.UpdatesManual, Channel: "stable", Pin: "v5", InstallRequest: 2}.Column()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetBuildState("a.example.com", release.BuildStateCopy{Known: true, Replica: true, State: &release.BuildState{LastExecuted: "v2", Completed: 1}}.Column()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := release.WriteConsumed(b.optDir, release.Consumed{Host: "a.example.com", Completed: 1, Resolved: &release.ResolvedRequest{ID: 2, Target: "v4"}}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if target, held, _ := b.heldTarget("a.example.com"); !held || target != "v5" { |
| + t.Fatalf("heldTarget = %q, %v; want the pin v5", target, held) |
| + } |
| +} |
| + |
| +// Astra I2 #4: root found the pin failed and will not install it; the wake |
| +// does not wait for it, and the failure-aware planner decides. |
| +func TestARootFailedTargetIsNotHeld(t *testing.T) { |
| + b, _, reg := testBuildStates(t) |
| + if err := reg.SetUpdateSettings("a.example.com", release.UpdateSettings{Updates: release.UpdatesAuto, Channel: "stable", Pin: "v5"}.Column()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := release.WriteConsumed(b.optDir, release.Consumed{Host: "a.example.com", Reason: string(release.ReasonFailed), Detail: "v5"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, held, _ := b.heldTarget("a.example.com"); held { |
| + t.Fatal("a pin root refused as failed held the wake for an install that never comes") |
| + } |
| +} |
| +``` |
| + |
| +The first fails against the I1 order (the resolved request's v4 came first); the second fails without `rootFailed` (the pin returned held). |
| + |
| - [ ] **Step 5: Run the packages.** Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test -race ./internal/activator/ ./internal/host/ -count=1`. Expected: PASS. |
| |
| - [ ] **Step 6: Commit.** |
| @@ -2362,9 +2670,9 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| |
| **Interfaces:** |
| - Consumes: `provision.ListRecords`, `release.PruneEligible`'s `alsoProtected`; `status.LoadBoxesComplete` (`internal/status/fleet.go:234`), `release.NewPrefixStore` (`store.go:111`), `Store.ListDirs`; the registry's `BuildState` column (Task 3). |
| -- Produces: `type status.BuildStatePins struct{ Account, App string; Builds []string }`; `BoxMetrics.BuildStates map[string]BuildStatePins` (`json:"build_states,omitempty"`, host-keyed, private); `func buildStatePins(reg *registry.Registry) map[string]status.BuildStatePins`; `func settingsProtected(ctx context.Context, store release.Store, boxes []status.Box, complete bool, now time.Time, acct, app string) (map[string]bool, error)`; `const buildStateReportMaxAge = 6 * time.Hour`; `func loadAllBoxes(ctx context.Context, store release.Store) ([]status.Box, bool, error)`. |
| +- Produces: `type status.BuildStatePins struct{ Account, App string; Builds []string }`; `BoxMetrics.BuildStates map[string]BuildStatePins` (`json:"build_states,omitempty"`, host-keyed, private); `BoxMetrics.BuildStatesErr string` (`json:"build_states_err,omitempty"`); `func buildStatePins(reg *registry.Registry) (map[string]status.BuildStatePins, error)`; `func settingsProtected(ctx context.Context, store release.Store, boxes []status.Box, complete bool, now time.Time, acct, app string) (map[string]bool, error)`; `const buildStateReportMaxAge = 6 * time.Hour`; `func loadAllBoxes(ctx context.Context, store release.Store) ([]status.Box, bool, error)`. |
| |
| -Spec rev 11 §4.4: build states live in **regional** data buckets the deployment housekeeper cannot read, so "each box adds the build-state builds of the instances it hosts (`last_executed`, and a resolved request's T) to its status report in the deployment bucket. The housekeeper protects every build named there." And: "The housekeeper prunes nothing for an app on that pass if it can't read every provision record of the app, or if any box hosting an instance of the app hasn't reported in the last 6 hours." A box "hosts an instance of the app" when its report names one of the app's hosts in `BuildStates`; when several reports name the same host (it moved), the newest report for that host is the one that counts, so a box retired after a move does not block pruning forever. Pins come from the provision records, as before. |
| +Spec rev 11 §4.4: build states live in **regional** data buckets the deployment housekeeper cannot read, so "each box adds the build-state builds of the instances it hosts (`last_executed`, and a resolved request's T) to its status report in the deployment bucket. The housekeeper protects every build named there." And: "The housekeeper prunes nothing for an app on that pass if it can't read every provision record of the app, or if any box hosting an instance of the app hasn't reported in the last 6 hours." A box "hosts an instance of the app" when its report names one of the app's hosts in `BuildStates`; when several reports name the same host (it moved), the newest report for that host is the one that counts, so a box retired after a move does not block pruning forever. Every provisioned instance of the app that keeps a build state (record `KindOrDefault() == "instance"`, no `Unit`, `HibernateOrDefault()`) must be named by some report, and a fresh report whose box could not collect its build states (`BuildStatesErr`) prunes nothing at all (Astra I2 #5). Pins come from the provision records, as before. |
| |
| - [ ] **Step 1: Write the failing tests.** Create `internal/host/retention_settings_test.go`: |
| |
| @@ -2401,13 +2709,14 @@ func TestAnIncompleteBuildStatePrunesNothing(t *testing.T) { |
| |
| func TestAnUnreadBuildStateIsReportedIncomplete(t *testing.T) { |
| _, _, reg := testBuildStates(t) // a.example.com, never refreshed: copy unknown |
| - if p, ok := buildStatePins(reg)["a.example.com"]; !ok || p.Complete { |
| - t.Fatalf("report = %+v, %v; want the host listed and incomplete", p, ok) |
| + pins, err := buildStatePins(reg) |
| + if p, ok := pins["a.example.com"]; err != nil || !ok || p.Complete { |
| + t.Fatalf("report = %+v, %v, %v; want the host listed and incomplete", p, ok, err) |
| } |
| if err := reg.SetBuildState("a.example.com", release.BuildStateCopy{Known: true, ReadErr: "timeout"}.Column()); err != nil { |
| t.Fatal(err) |
| } |
| - if p := buildStatePins(reg)["a.example.com"]; p.Complete { |
| + if pins, _ := buildStatePins(reg); pins["a.example.com"].Complete { |
| t.Fatal("a failed refresh was reported complete") |
| } |
| } |
| @@ -2430,7 +2739,8 @@ func TestPinsAndReportedBuildStatesAreProtected(t *testing.T) { |
| s := recordsStore(t, |
| provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable", Pin: "v1"}, |
| provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable", Region: "uk"}) |
| - boxes := []status.Box{report("uk-1", now0.Add(-time.Hour), map[string]status.BuildStatePins{"b.example.com": kmPins("v2", "v3")})} |
| + boxes := []status.Box{report("uk-1", now0.Add(-time.Hour), map[string]status.BuildStatePins{ |
| + "a.example.com": kmPins(), "b.example.com": kmPins("v2", "v3")})} |
| got, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km") |
| if err != nil { |
| t.Fatal(err) |
| @@ -2464,13 +2774,53 @@ func TestANewerReportForTheHostWins(t *testing.T) { |
| |
| func TestOtherAppsReportsDoNotBlock(t *testing.T) { |
| s := recordsStore(t, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| - boxes := []status.Box{report("uk-1", now0.Add(-72*time.Hour), map[string]status.BuildStatePins{ |
| - "x.example.com": {Account: "bab", App: "blog", Builds: []string{"b1"}}})} |
| + boxes := []status.Box{ |
| + report("uk-1", now0.Add(-72*time.Hour), map[string]status.BuildStatePins{ |
| + "x.example.com": {Account: "bab", App: "blog", Builds: []string{"b1"}}}), |
| + report("ie-1", now0.Add(-time.Minute), map[string]status.BuildStatePins{"a.example.com": kmPins("v1")}), |
| + } |
| if _, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km"); err != nil { |
| t.Fatalf("another app's silent box blocked km: %v", err) |
| } |
| } |
| |
| +// Astra I2 #5: every provisioned instance of the app needs an entry. A |
| +// fresh, readable report that simply omits the host (its map left out) |
| +// prunes nothing, not "nothing to protect". |
| +func TestAnOmittedHostPrunesNothing(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + boxes := []status.Box{report("uk-1", now0.Add(-time.Minute), nil)} |
| + if _, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km"); err == nil { |
| + t.Fatal("a host no report names was treated as having no builds to protect") |
| + } |
| +} |
| + |
| +// Astra I2 #5: a box that could not list its routes says so, and a fresh |
| +// report saying so stops pruning even when an older report from another |
| +// box still names the host (it may have moved onto the failing box). |
| +func TestACollectionFailurePrunesNothing(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + failed := report("uk-1", now0.Add(-time.Minute), nil) |
| + failed.Metrics.BuildStatesErr = "registry: database is locked" |
| + boxes := []status.Box{ |
| + report("ie-1", now0.Add(-2*time.Hour), map[string]status.BuildStatePins{"b.example.com": kmPins("v1")}), |
| + failed, |
| + } |
| + if _, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km"); err == nil { |
| + t.Fatal("a box that could not collect its build states did not stop pruning") |
| + } |
| +} |
| + |
| +func TestTheCollectorReportsItsFailure(t *testing.T) { |
| + _, _, reg := testBuildStates(t) |
| + if err := reg.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if pins, err := buildStatePins(reg); err == nil || pins != nil { |
| + t.Fatalf("a failed registry list returned %v, %v; want an error", pins, err) |
| + } |
| +} |
| + |
| func TestAnUnreadableRecordOrReportPrunesNothing(t *testing.T) { |
| s := recordsStore(t, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| if _, err := s.Put(context.Background(), provision.RecordKey("bab", "km", "z.example.com"), []byte("{"), ""); err != nil { |
| @@ -2491,7 +2841,11 @@ func TestTheBoxReportsItsBuildStates(t *testing.T) { |
| State: &release.BuildState{LastExecuted: "v1", Resolved: &release.ResolvedRequest{ID: 2, Target: "v2"}}}.Column()); err != nil { |
| t.Fatal(err) |
| } |
| - got := buildStatePins(reg)["a.example.com"] |
| + pins, err := buildStatePins(reg) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got := pins["a.example.com"] |
| if got.App != "km" || len(got.Builds) != 2 || got.Builds[0] != "v1" || got.Builds[1] != "v2" { |
| t.Fatalf("report = %+v", got) |
| } |
| @@ -2502,7 +2856,7 @@ Add `"time"` and `"github.com/carlosframework/platform/internal/status"` to the |
| |
| If `provision.CreateRecord` validates more than these fields (a slug, a domain), fill them from `provision_test.go`'s record fixture. |
| |
| -- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/host/ -run 'ReportedBuildStates|SilentForSixHours|NewerReportForTheHost|OtherAppsReports|UnreadableRecordOrReport|ReportsItsBuildStates|IncompleteBuildState|ReportedIncomplete' -count=1`. Expected: FAIL to compile (`status.BuildStatePins` undefined). |
| +- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/host/ -run 'ReportedBuildStates|SilentForSixHours|NewerReportForTheHost|OtherAppsReports|UnreadableRecordOrReport|ReportsItsBuildStates|IncompleteBuildState|ReportedIncomplete|OmittedHost|CollectionFailure|CollectorReportsItsFailure' -count=1`. Expected: FAIL to compile (`status.BuildStatePins` undefined). |
| |
| - [ ] **Step 3: The report.** In `internal/status/snapshot.go`, beside `HostUpdates`' neighbours in `BoxMetrics`: |
| |
| @@ -2513,6 +2867,11 @@ If `provision.CreateRecord` validates more than these fields (a slug, a domain), |
| // regional data buckets the objects live in. Host-keyed and private like |
| // RouteVersions; omitted when empty. |
| BuildStates map[string]BuildStatePins `json:"build_states,omitempty"` |
| + // BuildStatesErr is set when this box could not collect BuildStates at |
| + // all (its registry list failed). The housekeeper prunes nothing while a |
| + // fresh report says so (Astra I2 #5): an absent map is never "nothing to |
| + // protect". |
| + BuildStatesErr string `json:"build_states_err,omitempty"` |
| ``` |
| |
| ```go |
| @@ -2534,10 +2893,10 @@ In `internal/host/retention.go` add: |
| // buildStatePins is this box's contribution to retention (settings spec rev |
| // 11 §4.4): every hosted instance's build-state builds, from the registry |
| // copies the agent refreshes at reconcile and before every wake. |
| -func buildStatePins(reg *registry.Registry) map[string]status.BuildStatePins { |
| +func buildStatePins(reg *registry.Registry) (map[string]status.BuildStatePins, error) { |
| routes, err := reg.List() |
| if err != nil { |
| - return nil |
| + return nil, fmt.Errorf("listing routes for build states: %w", err) |
| } |
| out := map[string]status.BuildStatePins{} |
| for _, rt := range routes { |
| @@ -2561,13 +2920,13 @@ func buildStatePins(reg *registry.Registry) map[string]status.BuildStatePins { |
| out[rt.Host] = p |
| } |
| if len(out) == 0 { |
| - return nil |
| + return nil, nil |
| } |
| - return out |
| + return out, nil |
| } |
| ``` |
| |
| -and in `probes.go`, where `BoxMetrics` is filled, set `BuildStates: buildStatePins(reg)`. |
| +and in `probes.go`, where `BoxMetrics` is filled: `pins, pinsErr := buildStatePins(reg)`, set `BuildStates: pins`, and `if pinsErr != nil { m.BuildStatesErr = pinsErr.Error() }` (with `m` the `BoxMetrics` being filled; log the error too). |
| |
| - [ ] **Step 4: The housekeeper.** Add to `internal/host/retention.go`: |
| |
| @@ -2631,6 +2990,11 @@ func settingsProtected(ctx context.Context, store release.Store, boxes []status. |
| if b.Metrics == nil { |
| continue |
| } |
| + // A box that could not collect its build states may host any of |
| + // this app's instances (Astra I2 #5). |
| + if b.Metrics.BuildStatesErr != "" && now.Sub(b.GeneratedAt) <= buildStateReportMaxAge { |
| + return nil, fmt.Errorf("%s could not collect its build states: %s", b.Label, b.Metrics.BuildStatesErr) |
| + } |
| for host, p := range b.Metrics.BuildStates { |
| if p.Account != acct || p.App != app { |
| continue |
| @@ -2640,6 +3004,17 @@ func settingsProtected(ctx context.Context, store release.Store, boxes []status. |
| } |
| } |
| } |
| + // Every provisioned instance of the app that keeps a build state must |
| + // be named by some report (Astra I2 #5): a missing entry is unknown, |
| + // never "nothing to protect". |
| + for _, r := range recs { |
| + if r.KindOrDefault() != "instance" || r.Unit != "" || !r.HibernateOrDefault() { |
| + continue |
| + } |
| + if _, ok := byHost[r.Host]; !ok { |
| + return nil, fmt.Errorf("no box report names %s's build state", r.Host) |
| + } |
| + } |
| for host, n := range byHost { |
| if now.Sub(n.at) > buildStateReportMaxAge { |
| return nil, fmt.Errorf("%s, which hosts %s, has not reported since %s", n.box, host, n.at.UTC().Format(time.RFC3339)) |
| @@ -2670,7 +3045,7 @@ Use `ListRecords`' real `max` convention (read its doc: pass the value that list |
| |
| Promotion records go with their version only through `DeleteVersion` (plan 2 Task 5), so a protected version keeps its record by not being deleted. |
| |
| -A box from before this plan reports no `BuildStates`; it also writes no build state (the build-state writer is this plan's), so it has nothing to protect. The console refuses `pin`, `manual` and `major` until every box runs this plan (plan 3), and the roll note says to turn retention off before rolling a box back. |
| +A box from before this plan reports no `BuildStates`, so while it hosts an instance of an app, the coverage check prunes nothing for that app (Astra I2 #5 chose failing closed over trusting an absent map). Pruning therefore resumes app by app as the boxes hosting it roll; the hand-off note says so, and that retention should stay off before rolling a box back. A provisioned instance not yet placed on any box blocks its app's pruning the same way until a box reports it. The console refuses `pin`, `manual` and `major` until every box runs this plan (plan 3). |
| |
| - [ ] **Step 5: Run the packages.** Run: `cd $W && go build ./... && go test ./internal/status/ -count=1 && TMPDIR=/tmp/k go test -race ./internal/host/ -count=1`. Expected: PASS. |
| |
amadan