amadan
settings plans updated for spec rev 11 (56 tasks): retention from box reports, moves carry build state, held stage authorized, no default pin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+436
−121
docs/superpowers/plans/2026-10-07-instance-settings-0-index.md+69 −40docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md+367 −81
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-07-instance-settings-0-index.md b/docs/superpowers/plans/2026-10-07-instance-settings-0-index.md| index c47c34375..b4223cf88 100644 |
| --- a/docs/superpowers/plans/2026-10-07-instance-settings-0-index.md |
| +++ b/docs/superpowers/plans/2026-10-07-instance-settings-0-index.md |
| @@ -5,7 +5,7 @@ |
| > names they share. Read it before any plan, and read the spec before both. |
| **Spec:** `docs/superpowers/specs/2026-10-07-instance-update-settings-design.md` |
| -(rev 10). Rev 7 was approved by Paul and accepted by Astra (S5); §4.6 (#253) |
| +(rev 11; rev 11 adds the planning clarifications below). Rev 7 was approved by Paul and accepted by Astra (S5); §4.6 (#253) |
| was accepted by Astra in S7. The spec is the authority: where a plan and the |
| spec disagree, the spec wins and the plan is the bug. Its §7 and §9–§13 |
| record every review finding. None may be reopened. |
| @@ -31,9 +31,9 @@ gives beside the number. |
| | # | Plan | Branch | Lands | Depends on | |
| |---|------|--------|-------|------------| |
| | 1 | `…-1-pointer-digests.md`: #253. `ChannelPointer.Digest` and `HistoryDigests`; the provenance rule in promote, rollback, `FastTrack`, `RebindPointer` and (through them) the `cli.go` direct-store paths; the signed emission marker; root's digest check before staging; the static-site reader on the edge; per-box ratchets for root and the edge; `CARLOS_REQUIRE_POINTER_DIGEST`; `carlos channels resign`; the console's straggler report; its own copy review; roll notes for "writers before readers" | `settings-1-pointer-digests` | **ships first.** Inert until the marker is written: no writer emits a digest, and a ratchet only arms on a pointer that carries one | none (independent of every zdd plan) | |
| -| 2 | `…-2-promotion-records.md`: §4.1. The signed promotion record (account, app, version, manifest digest) written by every promotion; its verifier; `release.AuthorizedManifest`; `carlos releases authorize` and its console door; `DeleteVersion` removes a pruned build's record; the `internal/updatewords` package | `settings-2-promotion-records` | additive: records are written from merge on, and nothing reads them until plan 4 | 1 | |
| +| 2 | `…-2-promotion-records.md`: §4.1. The signed promotion record (account, app, version, manifest digest) written by every promotion; its verifier; `release.AuthorizedManifest`; `carlos releases authorize` and its console door; `DeleteVersion` removes a pruned build's record; a move's held stage installs only §4.1-authorized builds (rev 11 §4.3a); the `internal/updatewords` package | `settings-2-promotion-records` | additive: records are written from merge on, and nothing reads them until plan 4 | 1 | |
| | 3 | `…-3-console-settings.md`: §3, §5.1. The settings and `install_request` on the provision record; the app policy on the instances manifest; reconcile resolving defaults into one registry column; `GET`/`PUT /updates`, `POST /updates/install` and the app-policy door, for app members only (the instance-bound half is gated on #155 step 6); the console's refusal of `pin`, `manual` and `major` until every box reports the protecting binary; the control socket's read-only `GET /updates` | `settings-3-console-settings` | dormant on boxes: nothing reads the column until plan 4, and the refusal holds every non-default value until plan 4's boxes report | 2 | |
| -| 4 | `…-4-root-applies.md`: §4. `release.Target`; the consumed file; the build-state object and its registry copy; the `BeforeExec` write under the lease before every spawn path execs; `install_request` resolution, completion and the merged watermark; held-instance recovery and the wake check; failure records outranking every rule; evidence (pointer, promotion record, staged copy) per install; retention protection; the box report that lifts plan 3's refusal | `settings-4-root-applies` | the behaviour change. A record with no settings resolves to `auto` and rule 4, today's adoption, byte for byte | 1, 2, 3; zdd-1 to zdd-5 merged | |
| +| 4 | `…-4-root-applies.md`: §4. `release.Target`; the consumed file; the build-state object and its registry copy; the `BeforeExec` write under the lease before every spawn path execs; `install_request` resolution, completion and the merged watermark; held-instance recovery and the wake check; failure records outranking every rule; evidence (pointer, promotion record, staged copy) per install; moves carry the build state (rev 11 §4.3a); retention protects pins and the build-state builds boxes report (rev 11 §4.4); the box report that lifts plan 3's refusal | `settings-4-root-applies` | the behaviour change. A record with no settings resolves to `auto` and rule 4, today's adoption, byte for byte | 1, 2, 3; zdd-1 to zdd-5 merged | |
| | 5 | `…-5-surfaces.md`: §5. `carlos apps set` gains the policy flags; `carlos instances set`, `carlos instances update`; `carlos instances list` reasons; the console instance page's Updates section (Eleven-style); *settings not enforced*; docs; roll notes; and the copy review of every string plans 2–5 drafted, which stops for Paul | `settings-5-surfaces` | additive | 3, 4; zdd-6 merged (it adds `carlos apps set` and the instance list's INSTALLED/SERVING/REASON columns) | |
| Plans 1 and 2 can merge and roll before any zdd plan. Plan 3 can merge |
| @@ -172,7 +172,11 @@ func AuthorizedManifest(ctx context.Context, s Store, pub ed25519.PublicKey, acc |
| type AuthorizePlan struct { Account, App, Version, ManifestDigest string; Artifacts map[string]Artifact } |
| func PlanAuthorize(ctx context.Context, s Store, acct, app, version string) (*AuthorizePlan, error) |
| func (sg Signer) Authorize(ctx context.Context, s Store, plan *AuthorizePlan, actor string) (already bool, err error) // root only; re-reads, refuses if the digest moved |
| -func PromotedPrefix(acct, app string) string // every record of an app; PurgeApp sweeps it, plan 4's retention reads it |
| +func PromotedPrefix(acct, app string) string // every record of an app; PurgeApp sweeps it |
| +var ErrHeldNotAuthorized error |
| +func HeldAuthorized(ctx context.Context, s Store, pub ed25519.PublicKey, acct, app, version string) (want string, err error) // rev 11 §4.3a |
| +// StageHeld, MaterializeHeld and ServeHeldStageRequests each gain `pub ed25519.PublicKey` after `s Store`; |
| +// internal/host heldStagePass gains `pub` (runAdopt passes the parsed release key). |
| ``` |
| ### Plan 3 defines |
| @@ -210,6 +214,8 @@ type UpdatePolicy struct { |
| func EffectiveUpdates(rec Record, p *UpdatePolicy) release.UpdateSettings // drops major off stable |
| func ValidateUpdates(rec Record) error |
| func ValidatePolicy(p *UpdatePolicy) error |
| +// The app policy door accepts exactly instance_control, default_updates, default_channel, |
| +// default_major (rev 11: no default pin) and refuses any other key. |
| // channel stays on Record.Channel and the existing POST .../instances/{host}/channel door |
| // (pipeline passkeys and approvals); the /updates doors report it and refuse to change it. |
| @@ -336,6 +342,15 @@ type HostUpdates struct { |
| CompletedFailed bool `json:"completed_failed,omitempty"` |
| } |
| // BoxMetrics gains: Updates map[string]HostUpdates `json:"updates,omitempty"` (host-keyed, private) |
| +type BuildStatePins struct { Account, App string; Builds []string } // rev 11 §4.4 |
| +// BoxMetrics gains: BuildStates map[string]BuildStatePins `json:"build_states,omitempty"` (host-keyed, private) |
| + |
| +// internal/moves |
| +func BuildStateKey(host, opID string) string // moves/<host>/<op>/state/build-state.json: the phase-1 lane's copy |
| + |
| +// internal/host (retention, rev 11 §4.4) |
| +const buildStateReportMaxAge = 6 * time.Hour |
| +func settingsProtected(ctx context.Context, store release.Store, boxes []status.Box, complete bool, now time.Time, acct, app string) (map[string]bool, error) |
| ``` |
| ### Words (plans 2–5) |
| @@ -409,16 +424,23 @@ printed. Plan 1's strings are reviewed in plan 1. |
| Expected: the console refuses `pin`, `manual` and `major`; the housekeeper |
| prunes nothing it cannot prove unneeded. Plans 3 and 4 own it. |
| -## Spec questions found while planning (not resolved here; listed for Paul) |
| - |
| -- **Q1. Build-state objects live in regional data buckets.** §4.2 puts the |
| - build state "beside the instance's replica" (`CARLOS_HIBERNATE_BUCKET`, per |
| - region). §4.4 says the housekeeper prunes nothing for an app unless it can |
| - read every build-state object of the app. The deployment-wide housekeeper's |
| - role cannot read another region's data bucket (`infra/modules/region-host` |
| - grants sibling regions `moves/*` push only, never `GetObject`). Read |
| - literally, an app with instances in a second region is never pruned again. |
| - Plan 4 implements the literal, fail-closed reading and logs why. |
| +## Spec questions found while planning |
| + |
| +Q1, Q4, Q6, Q9 and Q10 are answered by spec rev 11 (the coordinator's |
| +numbering called them Q1, Q5, Q7, Q2 and Q9). The plans implement those |
| +answers, each with a test that fails without it. The readings recorded for |
| +Q2, Q3, Q5, Q7 and Q8 stand. |
| + |
| +- **Q1 (resolved, rev 11 §4.4). Build-state objects live in regional data |
| + buckets.** Each box puts its instances' build-state builds in its status |
| + report (`BoxMetrics.BuildStates`); the housekeeper protects those and every |
| + pin, and prunes nothing for an app if any provision record is unreadable or |
| + the newest report naming any of its hosts is more than 6 hours old. Plan 4 |
| + Task 9 (`TestPinsAndReportedBuildStatesAreProtected`, |
| + `TestABoxSilentForSixHoursPrunesNothing`, `TestANewerReportForTheHostWins`, |
| + `TestAnUnreadableRecordOrReportPrunesNothing`). The newest report for a host |
| + is the one that counts, so a box retired after a move does not block |
| + pruning for ever: a reading of "any box hosting an instance", recorded here. |
| - **Q2. `release.Target(route, pointer, settings)`** takes more than those |
| three: the build state, root's consumed file, the installed build, whether |
| a replica exists, this box's bake verdict and the failure records. Plan 4 |
| @@ -428,39 +450,46 @@ printed. Plan 1's strings are reviewed in plan 1. |
| The console has no `/api/v1`; every instance door is |
| `/api/cli/apps/{account}/{app}/instances/{host}/…`. Plan 3 uses the house |
| path. |
| -- **Q4. Default channel and default pin.** `Record.Channel` is required and |
| - set at create, so a "default channel" only ever reaches new instances; plan |
| - 3 maps `--default-channel` to the existing `instances.json` `channel`. §3.2's |
| - table implies a default pin, which §5 gives no flag; plan 3 offers none. |
| +- **Q4 (resolved, rev 11 §3.2). Default channel and default pin.** The |
| + defaults are `updates` and `major`, plus the channel for new instances |
| + only (`instances.json` `channel`). There is no default pin: plan 3's policy |
| + door refuses any key but its four (`TestPolicyDoorHasNoDefaultPin`) and |
| + pins that the channel default never moves an existing instance |
| + (`TestDefaultChannelReachesOnlyNewInstances`). |
| - **Q5. No command writes the emission marker.** §4.6 says the marker "is |
| written". Plan 1 adds `carlos channels digests on|off` (root key, |
| direct-store only, like other deployment-control writes). |
| -- **Q6. The moves held stage installs a version with no pointer.** |
| - `release.StageHeld` and `PublishHeld` (`materializeheld.go`) stage a |
| - version named by a move, checked only by the D3 proof gate. §4.5 says |
| - "nothing can make root install a build that §4.1 doesn't authorize". It is |
| - not an install this spec adds, so no plan changes it; it is listed. |
| +- **Q6 (resolved, rev 11 §4.3a). The moves held stage.** It installs only a |
| + build a promotion record or a signed pointer authorizes. Plan 2 Task 7 adds |
| + `release.HeldAuthorized` before `StageHeld` stages anything, binds staging |
| + to the record's (or pointer entry's) digest, and drops an unauthorized |
| + request (`TestHeldStageRefusesABuildNothingAuthorizes`, |
| + `TestHeldStageBindsTheRecordDigest`). `StageHeld`, `MaterializeHeld` and |
| + `ServeHeldStageRequests` gain a `pub` parameter. |
| - **Q7. `install_request` with FastTrack.** §4.3 lets a request ask for the |
| current head past bake. Today only the root key waives bake (`FastTrack`). |
| Plan 3 lets only a person who may fast-track the channel today set |
| `fast_track` on a request. |
| -- **Q9. A move does not carry the build state.** The moves protocol copies |
| - only the claim generation's LTX files (`moves.ReplicaSource`'s keep |
| - filter excludes everything else), so `build-state.json` stays behind at |
| - the source. A held instance landing in another region therefore has a |
| - replica and no build state, and rule 3 holds it as *build unknown* until a |
| - member presses Update now or sets a pin. That is the spec's safe answer, |
| - but it is a visible outage for `manual` instances that move. Plan 4 does |
| - not change the moves protocol; carrying the object needs a ruling. |
| -- **Q10. The flag-off wake execs the live link.** With `CARLOS_SWAP` unset, |
| - root can repoint between the build-state write and the exec. Plan 4 |
| - records the build the process actually ran before the wake completes, and |
| - stops the process if that write fails, so the build state is right by the |
| - time the wake answers, but a newer build can have opened the database for |
| - a moment first. With `CARLOS_SWAP` on every instance spawn execs a staged |
| - path and the window does not exist. Likewise the held wake check can run |
| - `last_executed` from its staged path only with generations on; flag off, |
| - it asks root to install it and waits within the wake's bound. |
| +- **Q9 (resolved, rev 11 §4.3a). Moves carry the build state.** The legacy |
| + move round already mirrors the whole replica prefix, `build-state.json` |
| + included, and the landing's `promoteStagedReplica` copies it over; plan 4 |
| + Task 8 pins that (`TestALandingMovePromotesTheBuildState`). The phase-1 |
| + lane mirrors only LTX files, so Task 8 adds a one-object mirror call to |
| + `moves.BuildStateKey(host, op)` (`TestThePhase1LaneCopiesTheBuildState`). |
| + Task 8 also closes a gap found on the way: between a legacy cut and the |
| + first wake, the staging is not yet promoted, so root saw no replica and no |
| + build state and would have treated a held instance as new. The agent now |
| + reads the staged copy (`TestAStagedMoveIsReadAsTheInstancesState`). |
| + **Still open:** the phase-1 *landing* is not on `main`. Task 8 Step 5 adds |
| + its copy if it has landed by then; otherwise whoever lands it must copy |
| + `moves.BuildStateKey` beside the destination's lease before the first wake. |
| +- **Q10 (resolved, rev 11 §4.2). The flag-off wake.** Rev 11 states that the |
| + legacy wake writes the build state before it execs, which plan 4 Task 5 |
| + does (`TestTheBuildStateIsWrittenBeforeTheWake`, |
| + `TestAFailedWriteBlocksTheSpawn`). The residual the plan already handles |
| + stays recorded: the flag-off wake execs `live/<host>`, so a repoint between |
| + the write and the exec is caught by comparing the build that ran and |
| + re-recording it before the wake completes (`TestAWakeThatRanAnotherBuildRecordsIt`). |
| - **Q8. Who may authorize and resign.** Both sign with the deployment's |
| key. Plan 2 and plan 1 make them root-key operations (the console signs |
| with its key for an app member, as it does for promote; a direct-store CLI |
diff --git a/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md b/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md| index 3c8f97f37..75d1740ce 100644 |
| --- a/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md |
| +++ b/docs/superpowers/plans/2026-10-07-instance-settings-4-root-applies.md |
| @@ -2,7 +2,7 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -**Goal:** Root's adoption pass installs, for every exec instance, the build its update settings call for (pin, a resolved "Update now", a held build, or today's channel head), each with the evidence §4.1 demands; the agent records which build last touched each instance's database before any process runs it; and retention never prunes a build a setting or a build state names. |
| +**Goal:** Root's adoption pass installs, for every exec instance, the build its update settings call for (pin, a resolved "Update now", a held build, or today's channel head), each with the evidence §4.1 demands; the agent records which build last touched each instance's database before any process runs it, and that record travels with a move; and retention never prunes a build a setting or a build state names. |
| **Architecture:** One pure decision function, `release.Target(TargetInput) TargetDecision`, implements §4.3's rules in order, with the zero-downtime failure records outranking all of them. `adoptHost` asks it which version to install instead of taking the pointer's head, then stages that version with the evidence the decision names: the signed pointer (today's head adoption, digest-checked by plan 1), the signed promotion record (plan 2), or the box's own staged and verified copy. Root alone resolves an install request, writing {R, T} to `/opt/carlos/consumed/<host>` before staging. The agent writes the **build state** (`last_executed`, the resolved request, the completed watermark) to `build-state.json` beside the instance's replica, under the instance's lease, before every spawn that runs a build against the database; a failed write refuses the spawn, and that write is what completes a request. The agent keeps a copy of the build state in the registry, which root reads. A held instance's wake refuses to run a stale live link. The housekeeper protects pins and build-state builds, and prunes nothing for an app it cannot fully read. Last, the box reports that it enforces settings, which lifts plan 3's console refusal. |
| @@ -25,8 +25,10 @@ |
| - Completion: a request completes when its T first runs against the data: the pre-exec build-state write records it. A cold install on a sleeper changes nothing. A T that fails (a failure record, or a hold failure) completes the request as failed, once. |
| - The watermark is the highest completed id from either the consumed file or the build state, never one in place of the other. |
| - The build state is written by the agent under the instance's lease, before the process (or its seed) runs; a failed write refuses the spawn. `build-state.json` sits at the same prefix as the lease's `lock.json`. |
| -- Retention: a version named by any provision record's `pin`, or by any build state's `last_executed` or resolved target, is protected, with its promotion record. An app whose records or build states cannot all be read is not pruned that pass (§4.4; index Q1 for regions). |
| -- Box report: `BoxMetrics.UpdateSettings = status.UpdateSettingsProtocol` only from this binary on, and only once retention protection (Task 9) is in. |
| +- Retention (rev 11 §4.4): a version named by any provision record's `pin`, or by any box report's build-state builds for the app (`BoxMetrics.BuildStates`: `last_executed` and a resolved T), is protected, with its promotion record. The housekeeper prunes nothing for an app if any of its provision records is unreadable, or if the newest report naming any of its hosts is more than 6 hours old. |
| +- Moves (rev 11 §4.3a): a move copies the build state with the replica (legacy round: already, pinned by a test; phase-1 lane: a second, one-object mirror call). Until a landed move's staging is promoted, the agent reads the staged build state as the instance's. The held stage's §4.1 check is plan 2 Task 7. |
| +- The legacy (flag-off) wake writes the build state before it execs (rev 11 §4.2), as Task 5 does. |
| +- Box report: `BoxMetrics.UpdateSettings = status.UpdateSettingsProtocol` only from this binary on, and only once moves carry the build state (Task 8) and retention protects it (Task 9). |
| - Member-facing strings: `internal/updatewords` drafts in `words_box.go` (`settings.box.*`); box journal lines are not drafts. No em dash. |
| - Concurrency: `-race` on every activator and host test run. The last functional task is an invariant simulation, run serially at `nice -n 19` with one 3000-seed sweep, plus mutation checks (CPU budget note: never deeper without Paul's say-so). |
| - Go: default build cache; `GOFLAGS=-mod=mod GOPROXY=off`; `TMPDIR=/tmp/k` for `internal/activator` and `internal/host`. Scratch under `$TMPDIR/settings-4/`, deleted in the last task. |
| @@ -55,9 +57,10 @@ |
| | `internal/host/agent.go`, `updatesroot.go` (new) | root and agent pass options | 6 | |
| | `internal/activator/wakecheck.go` (new), `_test.go`; `internal/host/buildstate.go` | the held wake check | 7 | |
| | `internal/updatewords/words_box.go` (new) | reasons a member reads | 2 | |
| -| `internal/host/retention.go`, `retention_settings_test.go` (new) | protected set, fail closed | 8 | |
| -| `internal/status/snapshot.go`; `internal/host/probes.go`; `internal/host/updatesreport.go` (new) | the box report; the control socket reader | 9 | |
| -| `internal/release/updatesim_test.go` (new) | the invariant simulation and mutation checks | 10 | |
| +| `internal/moves/staging.go`, `internal/host/movesphase1.go`, `internal/host/buildstate.go`; `internal/host/buildstate_moves_test.go`, `internal/activator/promote_buildstate_test.go` (new) | moves carry the build state; a landed move's staging is read before promotion | 8 | |
| +| `internal/status/snapshot.go` (`BuildStatePins`), `internal/host/retention.go`, `probes.go`, `retention_settings_test.go` (new) | boxes report build-state builds; the housekeeper protects them and pins, and fails closed | 9 | |
| +| `internal/status/snapshot.go`; `internal/host/probes.go`; `internal/host/updatesreport.go` (new) | the box report; the control socket reader | 10 | |
| +| `internal/release/updatesim_test.go` (new) | the invariant simulation and mutation checks | 11 | |
| --- |
| @@ -97,6 +100,10 @@ g 'func (a \*Service) persistGenerations() error' internal/activator |
| g 'GoodVersion' internal/registry |
| g 'ServingVersion' internal/registry |
| g 'func armedRollout()' internal/host |
| +g 'func HeldAuthorized(' internal/release |
| +g 'func LoadBoxesComplete(' internal/status |
| +g 'func MovesPrefix(host string) string' internal/activator |
| +g 'func ReplicaSource(' internal/moves |
| ``` |
| Expected: every line starts with `ok`. |
| @@ -1909,17 +1916,162 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| --- |
| -### Task 8: Retention protects pins and build-state builds |
| +### Task 8: Moves carry the build state |
| **Files:** |
| -- Modify: `internal/host/retention.go` (`retentionPass`) |
| +- Modify: `internal/host/buildstate.go` (`read`: a landed move's staged copy), `internal/host/movesphase1.go` (the replica lane, ~L316–335) |
| +- Modify: `internal/moves/staging.go` (`BuildStateKey`) |
| +- Create: `internal/host/buildstate_moves_test.go`, `internal/activator/promote_buildstate_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `activator.MovesPrefix(host)` (`backend_exec.go:1950`), `ExecBackend.promoteStagedReplica` (~L1981), `lease.MirrorPrefix`, `lease.MirrorPrefixKeep`, `moves.StagingPrefix`, `moves.ReplicaSource`; the activator test double `newFakeMirrorStore` (`backend_exec_test.go:264`); Task 4's `buildStates`. |
| +- Produces: `func moves.BuildStateKey(host, opID string) string` (`moves/<host>/<op>/state/build-state.json`); `buildStates.read` reads a staged build state when the own prefix has none. |
| + |
| +Spec rev 11 §4.3a: "A move or restore across regions copies the instance's build state object along with its replica (`internal/host/movespass.go`), so the destination holds the same `last_executed` and request state." There are two move paths on `main`: |
| + |
| +- **The legacy round** (`legacyMoveRound`, `movespass.go:627`) mirrors the whole `HibernatePrefix(host)` to `moves/<host>/` with no keep filter, and the destination's `promoteStagedReplica` copies everything staged over its own prefix at the first wake's restore. So `build-state.json` already travels. This task pins that with a test, because a keep filter added to either call would silently drop it. |
| +- **The phase-1 lane** (`movesphase1.go`) mirrors only the claim generation's LTX files (`moves.ReplicaSource`'s keep). It gains a second `MirrorPrefixKeep` call that copies exactly `build-state.json` from the holder's location to `moves.BuildStateKey(host, opID)`, in every round, so the byte-final round carries the final state. The phase-1 **landing** (the destination side that reads `moves/<host>/<op>/…`) is not on `main` yet; Task 0 Step 2 below checks, and if it has landed, Step 5 copies the staged state beside the destination's lease before the first wake. |
| + |
| +**Between the cut and the first wake**, the legacy staging sits at `moves/<host>/` and the destination's own prefix is empty: before this task, root's next pass saw "no replica, no build state" and would treat a held instance as **new** and install the head. `buildStates.read` therefore reads the staged copy when the own prefix has none: the staged `build-state.json` is the state, and any staged object counts as a replica. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/activator/promote_buildstate_test.go`, on the existing `fakeMirrorStore` and the `ExecBackend` test fixture `backend_exec_test.go` uses for `Restore` (read its tests at ~L140–240 for the constructor): |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "testing" |
| +) |
| + |
| +// Spec rev 11 §4.3a: a landing move carries the build state with the |
| +// replica. promoteStagedReplica copies every staged object, build-state.json |
| +// included; a keep filter added later must not drop it. |
| +func TestALandingMovePromotesTheBuildState(t *testing.T) { |
| + b, rt := restoreFixture(t) // the ExecBackend + route backend_exec_test.go's Restore tests build, with b.Mirror = newFakeMirrorStore() |
| + m := b.Mirror.(*fakeMirrorStore) |
| + m.put(b.Cfg.Bucket, MovesPrefix(rt.Host)+"generations/0000/a.ltx", "e1") |
| + m.put(b.Cfg.Bucket, MovesPrefix(rt.Host)+"build-state.json", "e2") |
| + if _, err := b.promoteStagedReplica(context.Background(), rt.Host); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := m.buckets[b.Cfg.Bucket][b.Cfg.hibernatePrefix(rt.Host)+"/build-state.json"]; !ok { |
| + t.Fatal("the landing did not carry build-state.json beside the replica") |
| + } |
| +} |
| +``` |
| + |
| +If `backend_exec_test.go` builds its `ExecBackend` inline rather than through a helper, extract that inline construction into `restoreFixture` in this file. Create `internal/host/buildstate_moves_test.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// Before the first wake promotes a landed move's staging, the own prefix is |
| +// empty. A held instance must not read as new. |
| +func TestAStagedMoveIsReadAsTheInstancesState(t *testing.T) { |
| + b, s, _ := testBuildStates(t) |
| + b.staged = func(h string) string { return "moves/" + h } |
| + ctx := context.Background() |
| + if _, err := s.Put(ctx, "moves/a.example.com/generations/0000/x.ltx", []byte("x"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Put(ctx, "moves/a.example.com/build-state.json", |
| + []byte(`{"schema":1,"host":"a.example.com","last_executed":"v1","at":"2026-10-07T12:00:00Z"}`), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + st, replica, err := b.read(ctx, "a.example.com") |
| + if err != nil || !replica || st == nil || st.LastExecuted != "v1" { |
| + t.Fatalf("read = %+v, %v, %v; want the staged state and a replica", st, replica, err) |
| + } |
| + // And Target therefore holds a manual instance on v1, not the head. |
| + d := release.Target(release.TargetInput{Pointer: &release.ChannelPointer{Version: "v3"}, |
| + Settings: release.UpdateSettings{Updates: release.UpdatesManual}, |
| + Build: release.BuildStateCopy{Known: true, Replica: replica, State: st}}) |
| + if d.Version != "v1" { |
| + t.Fatalf("Target = %+v", d) |
| + } |
| +} |
| + |
| +func TestThePhase1LaneCopiesTheBuildState(t *testing.T) { |
| + // movesphase1's existing round test fixture (movesphase1_test.go): run one |
| + // replica round for a host whose holder location has build-state.json, |
| + // and assert the target bucket holds moves.BuildStateKey(host, id). |
| + fx := phase1RoundFixture(t) |
| + fx.putSource(fx.holderLocation()+"/build-state.json", `{"schema":1,"host":"`+fx.host+`","last_executed":"v1","at":"2026-10-07T12:00:00Z"}`) |
| + fx.runRound(t) |
| + if !fx.targetHas(fx.buildStateKey()) { |
| + t.Fatal("the phase-1 lane did not stage the build state") |
| + } |
| +} |
| +``` |
| + |
| +`phase1RoundFixture` stands for the fixture `movesphase1_test.go` already uses to run a replica round against fake buckets: read that file and call its constructor and helpers by their real names (put an object in the source bucket at the holder's location, run the round, read the target bucket). Do not build a second fixture. |
| + |
| +- [ ] **Step 2: Check for the phase-1 landing.** |
| + |
| +```bash |
| +cd $W && git grep -n 'moves.ReplicaPrefix(\|moves.StagingPrefix(' -- internal/activator internal/host | grep -v _test | grep -v 'movesphase1.go\|movesreport.go' |
| +``` |
| + |
| +Expected on `main` at `9cf9f18f`: no output (no landing yet). If lines print, the landing exists, and Step 5 applies. |
| + |
| +- [ ] **Step 3: Run the tests, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/host/ -run 'StagedMove|Phase1LaneCopies' -count=1 && TMPDIR=/tmp/k go test ./internal/activator/ -run LandingMovePromotes -count=1`. Expected: the host tests FAIL (`b.staged` undefined; no staged state); the activator test PASSES already: it pins today's behaviour, and the commit says so. |
| + |
| +- [ ] **Step 4: Implement.** |
| + - `internal/moves/staging.go`: `func BuildStateKey(host, opID string) string { return StagingPrefix(host, opID) + "state/build-state.json" }`. |
| + - `buildStates` gains `staged func(host string) string` (set in `newBuildStates` to `func(h string) string { return strings.TrimSuffix(activator.MovesPrefix(h), "/") }`). In `read`, when the own prefix holds no `build-state.json` **and** lists no replica object, repeat both reads under `b.staged(host)`; a staged object counts as a replica, and a staged `build-state.json` is the state. Keep the own prefix authoritative when it holds anything, as `promoteStagedReplica` does the reverse at restore. |
| + - `movesphase1.go`: directly after the replica lane's retry loop succeeds (after `if err != nil { return fmt.Errorf("replica lane: %w", err) }`), add: |
| + |
| +```go |
| + // The build state travels with the replica (settings spec rev 11 §4.3a): |
| + // the destination must know which build last used this data. One object, |
| + // beside the lane, in every round so the byte-final round carries the |
| + // final state. |
| + stateKeep := func(rel string) bool { return rel == "build-state.json" } |
| + stateSrc := strings.TrimSuffix(holderLocation(*obs.Claim.Holder), "/") + "/" |
| + stateDst := strings.TrimSuffix(moves.BuildStateKey(rec.Host, id), "build-state.json") |
| + if _, err := lease.MirrorPrefixKeep(ctx, gm, cfg.OwnBucket, stateSrc, tgt.DataBucket, stateDst, |
| + lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(rec.Host, id+"-state")}, stateKeep); err != nil { |
| + return fmt.Errorf("build state lane: %w", err) |
| + } |
| +``` |
| + |
| +where `holderLocation` is the function `moves.ReplicaSource` uses to turn a `hostclaim.Holder` into its location (`instances/<host>` or `instances/<host>/<incarnation>`; read `internal/moves/source.go` and call it, or export it there if it is unexported). The build state is written beside the lease that location's wake takes (Task 0 Step 4); if Task 0 found it elsewhere, use that prefix here. |
| + |
| +- [ ] **Step 5 (only if Step 2 printed lines): The phase-1 landing.** Where the landing promotes `moves.ReplicaPrefix(host, op)` into the destination's lease prefix, copy `moves.BuildStateKey(host, op)` to `release.BuildStateKey(<that prefix>)` in the same promotion and before the first spawn, and add a test beside the landing's own tests asserting the destination prefix holds `build-state.json` after landing. If Step 2 printed nothing, record in the commit message that the landing must do this when it lands (index Q9). |
| + |
| +- [ ] **Step 6: Run the packages.** Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test -race ./internal/host/ ./internal/activator/ ./internal/moves/ -count=1`. Expected: PASS. |
| + |
| +- [ ] **Step 7: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/host internal/activator internal/moves && go vet ./internal/host/ ./internal/activator/ ./internal/moves/ |
| +git add internal/host/ internal/activator/promote_buildstate_test.go internal/moves/staging.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 8: moves carry the build state; a landed move's staging is read as the instance's state |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 9: Retention learns build-state builds from box reports |
| + |
| +**Files:** |
| +- Modify: `internal/status/snapshot.go` (`BuildStatePins`, `BoxMetrics.BuildStates`), `internal/host/probes.go` (where `BoxMetrics` is filled; find `GeoIP:`), `internal/host/retention.go` (`retentionPass`) |
| - Create: `internal/host/retention_settings_test.go` |
| **Interfaces:** |
| -- Consumes: `provision.ListRecords`, `release.PruneEligible`'s `alsoProtected`, `release.PromotionRecordKey`; Task 4's `buildStates.read`; `os.Getenv("CARLOS_REGION")`. |
| -- Produces: `func settingsProtected(ctx context.Context, store release.Store, bs *buildStates, region, acct, app string) (map[string]bool, error)`; `retentionPass` gains a `bs *buildStates` parameter (its one caller, `agent.go:2871`, passes the agent's). |
| +- Consumes: `provision.ListRecords`, `release.PruneEligible`'s `alsoProtected`; `status.LoadBoxesComplete` (`internal/status/fleet.go:234`), `release.NewPrefixStore` (`store.go:111`), `Store.ListDirs`; the registry's `BuildState` column (Task 3). |
| +- Produces: `type status.BuildStatePins struct{ Account, App string; Builds []string }`; `BoxMetrics.BuildStates map[string]BuildStatePins` (`json:"build_states,omitempty"`, host-keyed, private); `func buildStatePins(reg *registry.Registry) map[string]status.BuildStatePins`; `func settingsProtected(ctx context.Context, store release.Store, boxes []status.Box, complete bool, now time.Time, acct, app string) (map[string]bool, error)`; `const buildStateReportMaxAge = 6 * time.Hour`; `func loadAllBoxes(ctx context.Context, store release.Store) ([]status.Box, bool, error)`. |
| -§4.4: every build a record pins, and every build any build state names (installed or as a resolved target), is protected, with its promotion record; if any record or build state of the app cannot be read, nothing of the app is pruned that pass. A record placed in another region has its build state in that region's data bucket, which this box cannot read (index Q1): that app is skipped and the log says why. |
| +Spec rev 11 §4.4: build states live in **regional** data buckets the deployment housekeeper cannot read, so "each box adds the build-state builds of the instances it hosts (`last_executed`, and a resolved request's T) to its status report in the deployment bucket. The housekeeper protects every build named there." And: "The housekeeper prunes nothing for an app on that pass if it can't read every provision record of the app, or if any box hosting an instance of the app hasn't reported in the last 6 hours." A box "hosts an instance of the app" when its report names one of the app's hosts in `BuildStates`; when several reports name the same host (it moved), the newest report for that host is the one that counts, so a box retired after a move does not block pruning forever. Pins come from the provision records, as before. |
| - [ ] **Step 1: Write the failing tests.** Create `internal/host/retention_settings_test.go`: |
| @@ -1934,23 +2086,34 @@ import ( |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| -func TestPinsAndBuildStatesAreProtected(t *testing.T) { |
| - b, s, _ := testBuildStates(t) |
| - ctx := context.Background() |
| - deploy := release.NewDirStore(t.TempDir()) |
| - for _, r := range []provision.Record{ |
| - {Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable", Pin: "v1"}, |
| - {Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable"}, |
| - } { |
| - if err := provision.CreateRecord(ctx, deploy, r); err != nil { |
| +func report(label string, at time.Time, pins map[string]status.BuildStatePins) status.Box { |
| + return status.Box{Label: label, GeneratedAt: at, Metrics: &status.BoxMetrics{BuildStates: pins}} |
| +} |
| + |
| +func kmPins(builds ...string) status.BuildStatePins { |
| + return status.BuildStatePins{Account: "bab", App: "km", Builds: builds} |
| +} |
| + |
| +func recordsStore(t *testing.T, recs ...provision.Record) release.Store { |
| + t.Helper() |
| + s := release.NewDirStore(t.TempDir()) |
| + for _, r := range recs { |
| + if err := provision.CreateRecord(context.Background(), s, r); err != nil { |
| t.Fatal(err) |
| } |
| } |
| - if err := b.write(ctx, "b.example.com", release.BuildState{LastExecuted: "v2", Resolved: &release.ResolvedRequest{ID: 1, Target: "v3"}}); err != nil { |
| - t.Fatal(err) |
| - } |
| - _ = s |
| - got, err := settingsProtected(ctx, deploy, b, "", "bab", "km") |
| + return s |
| +} |
| + |
| +var now0 = time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC) |
| + |
| +// Spec rev 11 §4.4: pins from records, build-state builds from box reports. |
| +func TestPinsAndReportedBuildStatesAreProtected(t *testing.T) { |
| + s := recordsStore(t, |
| + provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable", Pin: "v1"}, |
| + provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable", Region: "uk"}) |
| + boxes := []status.Box{report("uk-1", now0.Add(-time.Hour), map[string]status.BuildStatePins{"b.example.com": kmPins("v2", "v3")})} |
| + got, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km") |
| if err != nil { |
| t.Fatal(err) |
| } |
| @@ -1961,47 +2124,161 @@ func TestPinsAndBuildStatesAreProtected(t *testing.T) { |
| } |
| } |
| -func TestAnUnreadableBuildStatePrunesNothing(t *testing.T) { |
| - b, s, _ := testBuildStates(t) |
| - ctx := context.Background() |
| - deploy := release.NewDirStore(t.TempDir()) |
| - if err := provision.CreateRecord(ctx, deploy, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}); err != nil { |
| - t.Fatal(err) |
| +func TestABoxSilentForSixHoursPrunesNothing(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + boxes := []status.Box{report("uk-1", now0.Add(-6*time.Hour-time.Minute), map[string]status.BuildStatePins{"b.example.com": kmPins("v2")})} |
| + if _, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km"); err == nil { |
| + t.Fatal("a box hosting the app that has not reported in 6h did not stop pruning") |
| + } |
| +} |
| + |
| +func TestANewerReportForTheHostWins(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "b.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + boxes := []status.Box{ |
| + report("uk-1", now0.Add(-48*time.Hour), map[string]status.BuildStatePins{"b.example.com": kmPins("v1")}), // the box it moved off |
| + report("ie-1", now0.Add(-time.Minute), map[string]status.BuildStatePins{"b.example.com": kmPins("v2")}), |
| } |
| - if _, err := s.Put(ctx, "inst/a.example.com/build-state.json", []byte("{"), ""); err != nil { |
| + got, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km") |
| + if err != nil || !got["v2"] { |
| + t.Fatalf("got %v, %v; want v2 from the newest report and no refusal", got, err) |
| + } |
| +} |
| + |
| +func TestOtherAppsReportsDoNotBlock(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + boxes := []status.Box{report("uk-1", now0.Add(-72*time.Hour), map[string]status.BuildStatePins{ |
| + "x.example.com": {Account: "bab", App: "blog", Builds: []string{"b1"}}})} |
| + if _, err := settingsProtected(context.Background(), s, boxes, true, now0, "bab", "km"); err != nil { |
| + t.Fatalf("another app's silent box blocked km: %v", err) |
| + } |
| +} |
| + |
| +func TestAnUnreadableRecordOrReportPrunesNothing(t *testing.T) { |
| + s := recordsStore(t, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + if _, err := s.Put(context.Background(), provision.RecordKey("bab", "km", "z.example.com"), []byte("{"), ""); err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := settingsProtected(ctx, deploy, b, "", "bab", "km"); err == nil { |
| - t.Fatal("an unreadable build state did not stop pruning") |
| + if _, err := settingsProtected(context.Background(), s, nil, true, now0, "bab", "km"); err == nil { |
| + t.Fatal("an unreadable record did not stop pruning") |
| + } |
| + clean := recordsStore(t, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable"}) |
| + if _, err := settingsProtected(context.Background(), clean, nil, false, now0, "bab", "km"); err == nil { |
| + t.Fatal("an incomplete read of box reports did not stop pruning") |
| } |
| } |
| -func TestAnotherRegionsInstancePrunesNothing(t *testing.T) { |
| - b, _, _ := testBuildStates(t) |
| - ctx := context.Background() |
| - deploy := release.NewDirStore(t.TempDir()) |
| - if err := provision.CreateRecord(ctx, deploy, provision.Record{Schema: 1, Host: "a.example.com", App: "km", Account: "bab", Channel: "stable", Region: "uk"}); err != nil { |
| +func TestTheBoxReportsItsBuildStates(t *testing.T) { |
| + _, _, reg := testBuildStates(t) |
| + if err := reg.SetBuildState("a.example.com", release.BuildStateCopy{Known: true, Replica: true, |
| + State: &release.BuildState{LastExecuted: "v1", Resolved: &release.ResolvedRequest{ID: 2, Target: "v2"}}}.Column()); err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := settingsProtected(ctx, deploy, b, "ie", "bab", "km"); err == nil { |
| - t.Fatal("a build state this box cannot read (another region's bucket) did not stop pruning") |
| + got := buildStatePins(reg)["a.example.com"] |
| + if got.App != "km" || len(got.Builds) != 2 || got.Builds[0] != "v1" || got.Builds[1] != "v2" { |
| + t.Fatalf("report = %+v", got) |
| } |
| } |
| ``` |
| +Add `"time"` and `"github.com/carlosframework/platform/internal/status"` to the imports. `testBuildStates` (Task 4) registers `a.example.com` for app `km`; set its `Account` to `bab` there if it does not already. |
| + |
| If `provision.CreateRecord` validates more than these fields (a slug, a domain), fill them from `provision_test.go`'s record fixture. |
| -- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/host/ -run 'PinsAndBuildStates|UnreadableBuildState|AnotherRegions' -count=1`. Expected: FAIL to compile. |
| +- [ ] **Step 2: Run them, see them fail.** Run: `cd $W && TMPDIR=/tmp/k go test ./internal/host/ -run 'ReportedBuildStates|SilentForSixHours|NewerReportForTheHost|OtherAppsReports|UnreadableRecordOrReport|ReportsItsBuildStates' -count=1`. Expected: FAIL to compile (`status.BuildStatePins` undefined). |
| + |
| +- [ ] **Step 3: The report.** In `internal/status/snapshot.go`, beside `HostUpdates`' neighbours in `BoxMetrics`: |
| + |
| +```go |
| + // BuildStates is, per instance this box hosts, the builds its build state |
| + // names: last_executed and a resolved request's target (settings spec rev |
| + // 11 §4.4). The housekeeper protects them, because it cannot read the |
| + // regional data buckets the objects live in. Host-keyed and private like |
| + // RouteVersions; omitted when empty. |
| + BuildStates map[string]BuildStatePins `json:"build_states,omitempty"` |
| +``` |
| + |
| +```go |
| +// BuildStatePins is one instance's build-state builds in a box report. |
| +type BuildStatePins struct { |
| + Account string `json:"account"` |
| + App string `json:"app"` |
| + Builds []string `json:"builds"` |
| +} |
| +``` |
| + |
| +In `internal/host/retention.go` add: |
| + |
| +```go |
| +// buildStatePins is this box's contribution to retention (settings spec rev |
| +// 11 §4.4): every hosted instance's build-state builds, from the registry |
| +// copies the agent refreshes at reconcile and before every wake. |
| +func buildStatePins(reg *registry.Registry) map[string]status.BuildStatePins { |
| + routes, err := reg.List() |
| + if err != nil { |
| + return nil |
| + } |
| + out := map[string]status.BuildStatePins{} |
| + for _, rt := range routes { |
| + c, err := release.ParseBuildStateCopy(rt.BuildState) |
| + if err != nil || c.State == nil { |
| + continue |
| + } |
| + var builds []string |
| + if c.State.LastExecuted != "" { |
| + builds = append(builds, c.State.LastExecuted) |
| + } |
| + if r := c.State.Resolved; r != nil && r.Target != "" && r.Target != c.State.LastExecuted { |
| + builds = append(builds, r.Target) |
| + } |
| + if len(builds) > 0 { |
| + out[rt.Host] = status.BuildStatePins{Account: release.RouteAccount(rt), App: rt.App, Builds: builds} |
| + } |
| + } |
| + if len(out) == 0 { |
| + return nil |
| + } |
| + return out |
| +} |
| +``` |
| + |
| +and in `probes.go`, where `BoxMetrics` is filled, set `BuildStates: buildStatePins(reg)`. |
| -- [ ] **Step 3: Implement.** Add to `internal/host/retention.go`: |
| +- [ ] **Step 4: The housekeeper.** Add to `internal/host/retention.go`: |
| ```go |
| -// settingsProtected is settings spec §4.4's addition to an app's protected |
| -// set: every pin, and every build any build state names. Any record or build |
| -// state it cannot read is an error, and the caller prunes nothing for the |
| -// app. A record placed in another region keeps its build state in that |
| -// region's data bucket, which this box cannot read (index Q1): an error too. |
| -func settingsProtected(ctx context.Context, store release.Store, bs *buildStates, region, acct, app string) (map[string]bool, error) { |
| +// buildStateReportMaxAge is §4.4's bound: an app is not pruned while any box |
| +// hosting one of its instances has been silent longer than this. |
| +const buildStateReportMaxAge = 6 * time.Hour |
| + |
| +// loadAllBoxes reads every box report: the deployment's and every fleet's. |
| +// complete is false if any report could not be read. |
| +func loadAllBoxes(ctx context.Context, store release.Store) ([]status.Box, bool, error) { |
| + boxes, complete, err := status.LoadBoxesComplete(ctx, store) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + fleets, err := store.ListDirs(ctx, "fleets/") |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + for _, f := range fleets { |
| + fb, ok, err := status.LoadBoxesComplete(ctx, release.NewPrefixStore(store, f)) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + complete = complete && ok |
| + boxes = append(boxes, fb...) |
| + } |
| + return boxes, complete, nil |
| +} |
| + |
| +// settingsProtected is settings spec rev 11 §4.4's addition to an app's |
| +// protected set: every pin a provision record names, and every build a box |
| +// report names for one of the app's instances. It refuses (the caller prunes |
| +// nothing for the app) when a record is unreadable, when the reports could |
| +// not all be read, or when the newest report naming any of the app's hosts |
| +// is older than six hours. |
| +func settingsProtected(ctx context.Context, store release.Store, boxes []status.Box, complete bool, now time.Time, acct, app string) (map[string]bool, error) { |
| recs, truncated, err := provision.ListRecords(ctx, store, acct, app, 0) |
| if err != nil { |
| return nil, err |
| @@ -2009,41 +2286,50 @@ func settingsProtected(ctx context.Context, store release.Store, bs *buildStates |
| if truncated { |
| return nil, fmt.Errorf("the records of %s/%s were not listed in full", acct, app) |
| } |
| + if !complete { |
| + return nil, fmt.Errorf("the box reports could not all be read") |
| + } |
| out := map[string]bool{} |
| for _, r := range recs { |
| if r.Pin != "" { |
| out[r.Pin] = true |
| } |
| - if r.KindOrDefault() != "instance" || r.Unit != "" { |
| + } |
| + type newest struct { |
| + at time.Time |
| + pins status.BuildStatePins |
| + box string |
| + } |
| + byHost := map[string]newest{} |
| + for _, b := range boxes { |
| + if b.Metrics == nil { |
| continue |
| } |
| - if r.Region != "" && r.Region != region { |
| - return nil, fmt.Errorf("%s lives in %s; its build state is not readable from %s", r.Host, r.Region, region) |
| - } |
| - if bs == nil { |
| - return nil, fmt.Errorf("this box cannot read build states (no replica bucket)") |
| - } |
| - st, _, err := bs.read(ctx, r.Host) |
| - if err != nil { |
| - return nil, fmt.Errorf("build state of %s: %w", r.Host, err) |
| - } |
| - if st != nil { |
| - if st.LastExecuted != "" { |
| - out[st.LastExecuted] = true |
| + for host, p := range b.Metrics.BuildStates { |
| + if p.Account != acct || p.App != app { |
| + continue |
| } |
| - if st.Resolved != nil { |
| - out[st.Resolved.Target] = true |
| + if cur, ok := byHost[host]; !ok || b.GeneratedAt.After(cur.at) { |
| + byHost[host] = newest{at: b.GeneratedAt, pins: p, box: b.Label} |
| } |
| } |
| } |
| + for host, n := range byHost { |
| + if now.Sub(n.at) > buildStateReportMaxAge { |
| + return nil, fmt.Errorf("%s, which hosts %s, has not reported since %s", n.box, host, n.at.UTC().Format(time.RFC3339)) |
| + } |
| + for _, v := range n.pins.Builds { |
| + out[v] = true |
| + } |
| + } |
| return out, nil |
| } |
| ``` |
| -Use `ListRecords`' real `max` convention (read its doc: `0` may mean "default cap"; pass the largest value it accepts so `truncated` is meaningful). In `retentionPass(ctx, store, pub, reg, bs)`, inside `for k, protected := range pins`, before `PruneEligible`: |
| +Use `ListRecords`' real `max` convention (read its doc: pass the value that lists every record, so `truncated` is meaningful). An unparseable record must make `ListRecords` (or the loop over its keys) return an error; if `ListRecords` skips unparseable records silently, list the keys with `RecordsPrefix` and count them against `len(recs)` here, refusing on a mismatch, so `TestAnUnreadableRecordOrReportPrunesNothing` holds. In `retentionPass`, read the reports once per pass with `boxes, complete, err := loadAllBoxes(ctx, store)` (on error, log and prune nothing at all this pass), then inside `for k, protected := range pins`, before `PruneEligible`: |
| ```go |
| - extra, err := settingsProtected(ctx, store, bs, strings.TrimSpace(os.Getenv("CARLOS_REGION")), k.acct, k.app) |
| + extra, err := settingsProtected(ctx, store, boxes, complete, time.Now(), k.acct, k.app) |
| if err != nil { |
| log.Printf("agent: retention %s/%s: pruning nothing this pass: %v", k.acct, k.app, err) |
| continue |
| @@ -2053,25 +2339,25 @@ Use `ListRecords`' real `max` convention (read its doc: `0` may mean "default ca |
| } |
| ``` |
| -Promotion records go with their version only through `DeleteVersion` (plan 2 Task 5), so a protected version keeps its record by not being deleted; nothing more is needed. |
| +Promotion records go with their version only through `DeleteVersion` (plan 2 Task 5), so a protected version keeps its record by not being deleted. |
| -The housekeeper's region-matching uses `Record.Region` as reconcile places it; a record with an empty region is the home region, which every box in it can read. If the housekeeper box's own region is not in `CARLOS_REGION` on some deployments, every placed record will block pruning; say so in the commit message. |
| +A box from before this plan reports no `BuildStates`; it also writes no build state (the build-state writer is this plan's), so it has nothing to protect. The console refuses `pin`, `manual` and `major` until every box runs this plan (plan 3), and the roll note says to turn retention off before rolling a box back. |
| -- [ ] **Step 4: Run the package.** Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test -race ./internal/host/ -count=1`. Expected: PASS. |
| +- [ ] **Step 5: Run the packages.** Run: `cd $W && go build ./... && go test ./internal/status/ -count=1 && TMPDIR=/tmp/k go test -race ./internal/host/ -count=1`. Expected: PASS. |
| -- [ ] **Step 5: Commit.** |
| +- [ ] **Step 6: Commit.** |
| ```bash |
| -cd $W && gofmt -l internal/host && go vet ./internal/host/ |
| -git add internal/host/retention.go internal/host/retention_settings_test.go internal/host/agent.go |
| -git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 8: retention protects pinned and build-state builds, and fails closed |
| +cd $W && gofmt -l internal/host internal/status && go vet ./internal/host/ ./internal/status/ |
| +git add internal/status/snapshot.go internal/host/retention.go internal/host/retention_settings_test.go internal/host/probes.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 9: boxes report build-state builds; retention protects them and pins, and fails closed |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ``` |
| --- |
| -### Task 9: The box says it enforces settings |
| +### Task 10: The box says it enforces settings |
| **Files:** |
| - Modify: `internal/status/snapshot.go` (`HostUpdates`, `BoxMetrics.Updates`, `Probes`), `internal/host/probes.go` (where `BoxMetrics` is filled; find `GeoIP:` there) |
| @@ -2081,7 +2367,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Produces: `type status.HostUpdates struct{ Desired, Reason, Detail string; Pending, Completed uint64; CompletedFailed bool }`; `BoxMetrics.Updates map[string]HostUpdates` (`json:"updates,omitempty"`); `func updatesReport(reg *registry.Registry, optDir string) map[string]status.HostUpdates`; `func updatesReader(reg *registry.Registry, optDir string) func(string) (instanceupdates.Report, error)`. |
| -Setting `UpdateSettings = status.UpdateSettingsProtocol` is what lets plan 3's console turn on `pin`, `manual` and `major`. It goes in only now, after retention protects them (Task 8). |
| +Setting `UpdateSettings = status.UpdateSettingsProtocol` is what lets plan 3's console turn on `pin`, `manual` and `major`. It goes in only now, after moves carry the build state (Task 8) and retention protects the builds it names (Task 9). |
| - [ ] **Step 1: Write the failing test.** Create `internal/host/updatesreport_test.go`: |
| @@ -2206,14 +2492,14 @@ In `probes.go`, where `BoxMetrics` is assembled, set `UpdateSettings: boxSetting |
| ```bash |
| cd $W && gofmt -l internal/status internal/host && go vet ./internal/status/ ./internal/host/ |
| git add internal/status/snapshot.go internal/host/ |
| -git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 9: boxes report that they enforce update settings, and why each instance runs what it does |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 10: boxes report that they enforce update settings, and why each instance runs what it does |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ``` |
| --- |
| -### Task 10: The invariant simulation and mutation checks |
| +### Task 11: The invariant simulation and mutation checks |
| **Files:** |
| - Create: `internal/release/updatesim_test.go` |
| @@ -2265,14 +2551,14 @@ Record each result in the commit message. Delete the copy: `rm -rf $TMPDIR/setti |
| ```bash |
| cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| git add internal/release/updatesim_test.go |
| -git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 10: request lifecycle simulation; 3000 seeds clean, M1-M5 each caught |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "settings-4 task 11: request lifecycle simulation; 3000 seeds clean, M1-M5 each caught |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ``` |
| --- |
| -### Task 11: Gate, push and hand-off |
| +### Task 12: Gate, push and hand-off |
| - [ ] **Step 1: The full gate.** |
| @@ -2291,7 +2577,7 @@ cd $W && git push amadan settings-4-root-applies 2>/dev/null || git push origin |
| amadan branch describe carlos/platform settings-4-root-applies -body - <<'EOF' |
| Root applies instance update settings (settings spec §4; plan 4). |
| -release.Target implements §4.3 (failure records first; pin; resolved request; held; today's adoption) as a pure function root's adoption pass asks per route; evidence per §4.1 (pointer for head adoption, promotion record otherwise, staged copy for pre-feature builds, refused elsewhere). Root resolves Update now into /opt/carlos/consumed/<host> before staging. The agent writes build-state.json beside the replica under the lease before every spawn (flag-off wake and seed, startPinned, spawnSuccessor, the claimed fallback, startWorker); a failed write refuses the spawn and the write completes requests. Held wakes never run a stale link. Retention protects pins and build-state builds and fails closed per app. Boxes report BoxMetrics.update_settings, which lifts plan 3's console refusal. Simulation: 3000 seeds clean; mutations M1-M5 caught. |
| +release.Target implements §4.3 (failure records first; pin; resolved request; held; today's adoption) as a pure function root's adoption pass asks per route; evidence per §4.1 (pointer for head adoption, promotion record otherwise, staged copy for pre-feature builds, refused elsewhere). Root resolves Update now into /opt/carlos/consumed/<host> before staging. The agent writes build-state.json beside the replica under the lease before every spawn (flag-off wake and seed, startPinned, spawnSuccessor, the claimed fallback, startWorker); a failed write refuses the spawn and the write completes requests. Held wakes never run a stale link. Moves carry build-state.json (legacy round pinned by a test; phase-1 lane copies it); a landed move's staging is read as the instance's state until promoted. Boxes report their instances' build-state builds; the housekeeper protects those and pins, and prunes nothing for an app with an unreadable record or a hosting box silent for 6h. Boxes report BoxMetrics.update_settings, which lifts plan 3's console refusal. Simulation: 3000 seeds clean; mutations M1-M5 caught. |
| Roll: every box before turning any setting on (the console enforces this). Before rolling any box back below this release, turn retention off for the deployment (README note in plan 5). |
| EOF |