| new file mode 100644 |
| index 000000000..db972141c |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md |
| @@ -0,0 +1,3373 @@ |
| +# Zero-Downtime Deploys, Plan 4: Release Side, Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** The release-store and staging half of zero-downtime deploys: a release's own `swap` demand, lossless staged manifests, a canonical-digest check for staged copies, a pin-set interface, sidecar adoption, worker recycling onto the build a worker actually runs, and the signed retry request. |
| + |
| +**Architecture:** Everything here is additive in `internal/release` and in the activator's worker path. `Manifest.Swap` and `CanonicalDemand` are read by nobody until plan 5. Root staging hashes a reused staged binary and compares it with the canonical manifest. A mismatch restages only when the pin set (`BoxPins` plus a `PinSource` that plan 5 fills) does not name the copy. Root's adoption pass gains a second admission predicate, `SidecarAdoptable`, and runs sidecars with no restart callback. The activator captures each worker's exec'd build (`release.StagedIdentity`) at spawn and recycles crashes and `max_run` onto it by path. The retry request is a new signed object in the release store, verified like a channel pointer. |
| + |
| +**Tech Stack:** Go (stdlib), the repo's `internal/release`, `internal/activator`, `internal/registry` and `internal/provision` packages. No new dependencies. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 13, accepted). Read §3, §4.2, §7.5 step 3, §7.9, §8.4 (the `carlos deploy --retry` paragraph), §8.5 and §12 before starting any task. The plan index, `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md`, fixes the shared names `release.CanonicalDemand` and `release.StagedIdentity`. They are used here exactly as the index spells them. |
| + |
| +**Branch:** `zdd-4-release-side`, cut from `origin/main`. Line numbers below are from `origin/main` at `289043db`. If main has moved, find each edit by the function name given beside the line number. |
| + |
| +**Behaviour changes on a box.** This plan reads no flag, and `CARLOS_SWAP` (plan 5) does not exist yet. With one exception class, a box behaves as it did before. The exceptions are mandated by the spec: |
| + |
| +1. **Worker identity (spec §7.9).** A worker (native sidecar) now execs the staged file its live link resolves to, not the link itself. `argv[0]` becomes `/opt/carlos/versions/<ver>/...` instead of `/opt/carlos/live/<host>`. A crash recycle and an always-on `max_run` recycle respawn that captured file, verified against its captured sha256, even if root has repointed the link since. A wake and a recycle no longer clear registry `Version`, and a worker's `/v1/ready` report no longer writes it. The report is kept in memory as display metadata (`Service.WorkerReportedVersion`). |
| +2. **`SidecarAdoptable` (spec §7.9).** Every adoption pass (root's hourly and relayed pass, the agent's hourly and fast loops, and cold materialization) now stages, repoints and stamps native sidecar rows from their channel. It never restarts them. The existing drift check moves the running worker, as it does today for a hand-repointed link. |
| +3. **Reused staged copies (spec §4.2).** Before this plan, a staged version directory was reused by name. Now a reused binary is hashed. If it matches the canonical manifest, nothing changes. If it differs, it is restaged when nothing refers to it, and refused when something does. This path is reachable only when a version name was shipped again with different bytes. |
| +4. **Sibling bytes (spec §4.2).** The staged `<app>.manifest.json` sibling is now the canonical manifest's exact bytes, not a re-marshal. Every reader parses it the same way. |
| + |
| +A consequence of 1 and 2 that the spec does not discuss: `carlos sidecars ls` shows a sidecar's registry `Version` as its "running version" (`internal/console/apisidecars.go` `sidecarListRecord`). After this plan, that column holds root's installed stamp. Plan 5 adds `ServingVersion` and plan 6 changes the surface. Until then the column can name an installed build that is not yet running. This is flagged for Paul in the review notes, not fixed here. |
| + |
| +## Global Constraints |
| + |
| +- Wire shapes are additive (AGENTS.md). `Manifest.Swap` is `json:"swap,omitempty"`. A manifest without it marshals byte-identically to before, and every manifest written before it still parses. |
| +- `swap` values: `"hold"` or empty (spec §4.2). `Ship` refuses any other value. Reads accept any value, and `NormalizeDemand` maps every non-empty value to `"hold"`, because "Missing or unverifiable policy data never permits an overlap" (spec §4.2). |
| +- `CanonicalDemand` reads the canonical manifest in the release store, never the staged sibling (spec §4.2). Unreadable manifest, no artifact for the platform, unreadable staged file: `"hold"` and a nil error. Digest mismatch: an error wrapping `ErrStagedDigestMismatch`. |
| +- The refusal for a referenced reused name is the spec's text, verbatim: `version name reused with different content, ship under a new name` (`release.ErrVersionReused`). |
| +- The pin set is: live links, `Version`, `PrevVersion`, `GoodVersion`, and every live or pending generation (spec §4.2, §8.5). This plan reads the first three itself (`BoxPins`). The last two arrive through `release.PinSource`, which plan 5 implements. An unknown pin set never permits a restage. |
| +- `Adoptable` is not widened. Its body and every caller outside `internal/release` stay byte-identical (spec §7.9). |
| +- A sidecar route is adopted with `restart == nil`, after every instance, awake or hibernating. |
| +- Crash and `max_run` recycles exec the captured `StagedIdentity.StagedPath` after `release.VerifyStaged`. A converge (drift) respawn still execs what the live link names; plan 5 turns that into a queued `release` attempt. |
| +- The retry request is signed in the channel pointer's `signedWrapper`, root key or scoped key with a grant, and the grant's scope is checked against the request's channel (spec §8.4, §11). |
| +- No member-facing string is added. New errors are sentinel values or operator journal lines. Plan 6 maps them to reviewed copy. New strings use no em dash. |
| +- `release.FileSHA256` already exists on main (`internal/release/proof.go:169`, D3). Reuse it. Do not add a second hashing helper. |
| +- Go: use the default build cache. Do not set `GOCACHE`, `GOMODCACHE` or `GOTMPDIR` under `/tmp`. Activator tests bind Unix sockets and need a short temp root: run them with `TMPDIR=/tmp/k`. Every `go` command below assumes `GOFLAGS=-mod=mod GOPROXY=off` exported in the same shell. |
| +- CI cross-compiles darwin, windows and linux-arm64. Nothing here uses a Linux-only syscall. |
| +- Commits in the worktree use `git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit` and end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. One commit per task, message naming the plan and task. |
| +- Worktree: `/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-4-release-side`, written `$W` below. Shell state does not persist between an agent's commands, so start each one with `export W=/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-4-release-side GOFLAGS=-mod=mod GOPROXY=off`. |
| + |
| +## Review Focus |
| + |
| +1. **A live link an operator made by hand: relative, or pointing outside `/opt/carlos/versions`.** Expected: the worker still spawns. Its identity carries the resolved path and the file's sha256, with an empty version, and a recycle verifies those bytes. Pinned in Task 3 (`TestResolveLiveOutsideTheStoreStillIdentifiesTheBytes`, `TestResolveLiveFollowsARelativeLink`). |
| +2. **A canonical manifest carrying a `swap` value this binary does not know** (a newer shipper's). Expected: the manifest still reads and stages, and its demand is `hold`, never an overlap. Pinned in Task 1 (`TestReadsAcceptAnUnknownSwapDemand`) and Task 4 (`TestCanonicalDemandIsHoldWhenItCannotCompare`, case "unknown demand"). |
| +3. **A stale staged copy while the bucket cannot answer for the manifest.** Expected: today's answer. The copy is used as staged, nothing is restaged or refused, and no new line claims a mismatch. Pinned in Task 6 (`TestUnreadableManifestKeepsTodaysAnswerForAStagedCopy`). |
| +4. **A worker that reports an empty or app-chosen version.** Expected: registry `Version` is untouched, the report is visible as display metadata, and a crash recycle still execs the captured build. Pinned in Task 9 (`TestNoteWorkerReadyIsDisplayOnly`, `TestWorkerReportingAnAppChosenVersionStillRecyclesItsCapturedBuild`). |
| +5. **Signed bytes moved to the wrong key:** a channel pointer copied to a retry key, a retry copied to a channel key, a retry copied to another build's key. Expected: each is refused on read. Pinned in Task 10 (`TestAChannelPointerIsNotARetryRequest`, `TestARetryRequestIsNotAChannelPointer`, `TestARetryCopiedToAnotherBuildIsRefused`). |
| + |
| +--- |
| + |
| +## File Structure |
| + |
| +| File | Responsibility | Tasks | |
| +| --- | --- | --- | |
| +| `internal/release/release.go` | `Manifest.Swap`; `Ship` refuses an unknown demand; `GetManifestRaw` | 1, 2 | |
| +| `internal/release/demand.go` (new) | `DemandHold`, `NormalizeDemand`, `ErrStagedDigestMismatch`, `CanonicalDemand` | 1, 4 | |
| +| `internal/release/staged.go` (new) | `StagedIdentity`, `ErrStagedCopyChanged`, `ErrVersionReused`, `ResolveLive`, `VerifyStaged` | 3 | |
| +| `internal/release/versions.go` | `versionOfTarget` split out of `LiveVersion`; sibling doc comments | 2, 3 | |
| +| `internal/release/pins.go` (new) | `StagedPaths`, `PinSource`, `PinSourceFunc`, `BoxPins` | 5 | |
| +| `internal/release/adopt.go` | raw-byte sibling; `stageCheck`, `ensureStagedChecked`, `WithPinSource`; `SidecarAdoptable` and the sidecar leg | 2, 6, 7 | |
| +| `internal/release/retry.go` (new) | `RetryRequest`, `RetryKey`, `RequestRetry`, `GetRetry`, `Clears` | 10 | |
| +| `internal/activator/backend_exec.go` | `StartWorkerStaged`; `StartWorker` delegates to it | 8 | |
| +| `internal/activator/backend_unit.go` | `SplitBackend.StartWorkerStaged` | 8 | |
| +| `internal/activator/workerbuild.go` (new) | `stagedWorkerStarter`, `startWorker`, `recycleWorkerLocked`, `WorkerReportedVersion` | 9 | |
| +| `internal/activator/activator.go` | captured serving identity, recycle call sites, `CoalesceRecycle`, display-only report, `DueStore` without `SetVersions` | 9 | |
| +| tests | one new `_test.go` per task, named in each task; five obsolete activator tests removed in Task 9 | all | |
| + |
| +--- |
| + |
| +### Task 0: The worktree |
| + |
| +- [ ] **Step 1: Cut the branch from main.** |
| + |
| +```bash |
| +cd /home/paulca/amadan.net/carlos/platform |
| +git fetch origin |
| +git worktree add -b zdd-4-release-side .claude/worktrees/zdd-4-release-side origin/main |
| +cd .claude/worktrees/zdd-4-release-side && git log --oneline -1 |
| +``` |
| + |
| +Expected: the worktree is created and `git log` prints main's tip (`289043db` or later). |
| + |
| +- [ ] **Step 2: Baseline.** Confirm the packages this plan touches are green before any change. |
| + |
| +```bash |
| +cd $W && export GOFLAGS=-mod=mod GOPROXY=off && mkdir -p /tmp/k |
| +go build ./... && go test ./internal/release/ ./internal/provision/ -count=1 && TMPDIR=/tmp/k go test ./internal/activator/ -count=1 |
| +``` |
| + |
| +Expected: `ok` for all three packages. A red baseline is reported before any task starts, never worked around. |
| + |
| +--- |
| + |
| +### Task 1: `Manifest.Swap`, the release's own demand |
| + |
| +**Files:** |
| +- Modify: `internal/release/release.go:47-100` (`Manifest`, after `ProgramChange` at line 69), `:595-626` (`Ship`) |
| +- Create: `internal/release/demand.go` |
| +- Create: `internal/release/demand_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `Manifest.Swap string` (`json:"swap,omitempty"`). |
| + - `const DemandHold = "hold"`. |
| + - `func NormalizeDemand(swap string) string`: `""` stays `""`, everything else is `DemandHold`. |
| + - Test helper `putRawRelease(t *testing.T, s Store, acct, app, ver string, body []byte, extra string) []byte` in `demand_test.go`, used by Tasks 2, 4 and 6. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/demand_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "fmt" |
| + "strings" |
| + "testing" |
| +) |
| + |
| +// putRawRelease writes a release the way a NEWER shipper would: the artifact, |
| +// then a manifest whose JSON this binary did not marshal. extra is spliced in |
| +// before "artifacts" and must end with a comma when non-empty, for example |
| +// `"swap":"hold",`. Returns the manifest bytes exactly as stored. |
| +func putRawRelease(t *testing.T, s Store, acct, app, ver string, body []byte, extra string) []byte { |
| + t.Helper() |
| + ctx := context.Background() |
| + if _, err := s.Put(ctx, ArtifactKey(acct, app, ver, adoptPlatform), body, ""); err != nil { |
| + t.Fatalf("put artifact: %v", err) |
| + } |
| + raw := []byte(fmt.Sprintf(`{"schema":1,"app":%q,"version":%q,"kind":"binary",%s"artifacts":{%q:{"file":%q,"sha256":%q,"size":%d}}}`, |
| + app, ver, extra, adoptPlatform, adoptPlatform, etagOf(body), len(body))) |
| + if _, err := s.Put(ctx, ManifestKey(acct, app, ver), raw, ""); err != nil { |
| + t.Fatalf("put manifest: %v", err) |
| + } |
| + return raw |
| +} |
| + |
| +func TestManifestSwapIsAdditive(t *testing.T) { |
| + old := []byte(`{"schema":1,"app":"km","version":"de2766f8","kind":"binary","artifacts":{"linux-arm64":{"file":"linux-arm64","sha256":"` + |
| + strings.Repeat("a", 64) + `","size":1}}}`) |
| + var m Manifest |
| + if err := json.Unmarshal(old, &m); err != nil { |
| + t.Fatalf("a manifest written before swap existed no longer parses: %v", err) |
| + } |
| + if m.Swap != "" { |
| + t.Fatalf("absent swap decoded as %q", m.Swap) |
| + } |
| + out, err := json.MarshalIndent(m, "", " ") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + // Byte-identical marshal is what keeps Ship's idempotent re-ship of an |
| + // old release working. |
| + if bytes.Contains(out, []byte(`"swap"`)) { |
| + t.Fatalf("an empty swap leaked into the marshal:\n%s", out) |
| + } |
| +} |
| + |
| +func TestShipRecordsAHoldDemand(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + m := Manifest{App: "km", Version: "v1", Kind: "binary", Swap: DemandHold, |
| + Artifacts: map[string]Artifact{adoptPlatform: {}}} |
| + if err := Ship(ctx, s, "bab", m, map[string][]byte{adoptPlatform: []byte("bin")}); err != nil { |
| + t.Fatalf("Ship: %v", err) |
| + } |
| + got, err := GetManifest(ctx, s, "bab", "km", "v1") |
| + if err != nil || got.Swap != DemandHold { |
| + t.Fatalf("GetManifest = %+v, %v; want swap %q", got, err, DemandHold) |
| + } |
| +} |
| + |
| +func TestShipRefusesAnUnknownSwapDemand(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + m := Manifest{App: "km", Version: "v1", Kind: "binary", Swap: "drain", |
| + Artifacts: map[string]Artifact{adoptPlatform: {}}} |
| + err := Ship(ctx, s, "bab", m, map[string][]byte{adoptPlatform: []byte("bin")}) |
| + if err == nil || !strings.Contains(err.Error(), `"drain"`) { |
| + t.Fatalf("Ship with swap drain = %v; want a refusal naming the value", err) |
| + } |
| + if _, _, gerr := s.Get(ctx, ManifestKey("bab", "km", "v1")); gerr == nil { |
| + t.Fatal("a refused ship still wrote its manifest") |
| + } |
| + if _, _, gerr := s.Get(ctx, ArtifactKey("bab", "km", "v1", adoptPlatform)); gerr == nil { |
| + t.Fatal("a refused ship still wrote its artifact") |
| + } |
| +} |
| + |
| +// Review Focus 2: a newer shipper's value must not stop a box reading the |
| +// manifest, and must act as hold. |
| +func TestReadsAcceptAnUnknownSwapDemand(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"swap":"drain",`) |
| + m, err := GetManifest(context.Background(), s, "bab", "km", "v1") |
| + if err != nil { |
| + t.Fatalf("a manifest from a newer shipper must still read: %v", err) |
| + } |
| + if got := NormalizeDemand(m.Swap); got != DemandHold { |
| + t.Fatalf("NormalizeDemand(%q) = %q; an unknown demand must act as hold", m.Swap, got) |
| + } |
| +} |
| + |
| +func TestNormalizeDemand(t *testing.T) { |
| + for in, want := range map[string]string{"": "", "hold": "hold", "overlap": "hold", "HOLD": "hold", "drain": "hold"} { |
| + if got := NormalizeDemand(in); got != want { |
| + t.Errorf("NormalizeDemand(%q) = %q, want %q", in, got, want) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'Swap|NormalizeDemand' -count=1` |
| +Expected: FAIL to compile, `m.Swap undefined` and `undefined: DemandHold`. |
| + |
| +- [ ] **Step 3: Add the field.** In `internal/release/release.go`, inside `Manifest`, directly after the `ProgramChange` line (69): |
| + |
| +```go |
| + // Swap is this release's own swap demand (zero-downtime deploys spec |
| + // §4.2). DemandHold says the build's boot must not share the instance's |
| + // database with the build before it, so an awake instance moves onto it |
| + // stop-then-start. Empty means the release demands nothing and the app's |
| + // policy decides. ADDITIVE to schema 1 like Label: absent from every |
| + // manifest written before it, omitted from marshal when empty, ignored by |
| + // old binaries. A box reads it from the CANONICAL manifest only |
| + // (CanonicalDemand), never from its staged copy, because a root older |
| + // than this field re-marshalled its copies without it. |
| + Swap string `json:"swap,omitempty"` |
| +``` |
| + |
| +- [ ] **Step 4: Refuse an unknown demand at ship.** In `Ship`, directly after `m.Schema = 1`: |
| + |
| +```go |
| + // Ship is where a person is present to read a refusal, so the one value |
| + // a release may demand is enforced here. Reads stay permissive |
| + // (NormalizeDemand): a value from a newer shipper must not stop a box |
| + // staging the release, only stop an overlap. |
| + if m.Swap != "" && m.Swap != DemandHold { |
| + return fmt.Errorf("ship: swap %q: a release can demand %q or nothing", m.Swap, DemandHold) |
| + } |
| +``` |
| + |
| +- [ ] **Step 5: Create `internal/release/demand.go`.** |
| + |
| +```go |
| +// The release's own swap demand (zero-downtime deploys spec §4.2): how an |
| +// awake instance may move onto a build, as the build's shipper said. |
| +package release |
| + |
| +// DemandHold is the one swap demand a release can make (`carlos deploy |
| +// --swap hold`, spec §9.1). |
| +const DemandHold = "hold" |
| + |
| +// NormalizeDemand is the demand a box acts on for a manifest's swap field. |
| +// "" is no demand. Anything else is DemandHold: "hold" itself, and any value |
| +// a newer shipper wrote that this binary does not know, because missing or |
| +// unverifiable policy data never permits an overlap (spec §4.2). |
| +func NormalizeDemand(swap string) string { |
| + if swap == "" { |
| + return "" |
| + } |
| + return DemandHold |
| +} |
| +``` |
| + |
| +- [ ] **Step 6: Run the package.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -count=1` |
| +Expected: PASS, including every existing test (`TestManifestLabelNotesAdditive`, `TestShipImmutable`, the golden tests). |
| + |
| +- [ ] **Step 7: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/release.go internal/release/demand.go internal/release/demand_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 1: Manifest.Swap, the release's own demand (additive) |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +Expected: `gofmt -l` prints nothing; vet is clean; one commit. |
| + |
| +--- |
| + |
| +### Task 2: Staging keeps the manifest's raw bytes |
| + |
| +**Files:** |
| +- Modify: `internal/release/release.go:663-683` (`GetManifest`) |
| +- Modify: `internal/release/adopt.go:1613-1683` (`ensureStaged`), `:1698-1715` (`backfillManifest`), imports (drop `encoding/json`) |
| +- Modify: `internal/release/versions.go:164-178` (`StageManifest` doc), `:236-242` (`ReadStagedManifest` doc) |
| +- Create: `internal/release/staged_raw_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `putRawRelease` (Task 1). |
| +- Produces: `func GetManifestRaw(ctx context.Context, s Store, acct, app, version string) (*Manifest, []byte, error)`. `GetManifest` keeps its signature and calls it. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/staged_raw_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "os" |
| + "testing" |
| +) |
| + |
| +func TestGetManifestRawReturnsTheBytesItParsed(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + raw := putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"swap":"hold","from_a_newer_shipper":{"x":1},`) |
| + m, got, err := GetManifestRaw(context.Background(), s, "bab", "km", "v1") |
| + if err != nil { |
| + t.Fatalf("GetManifestRaw: %v", err) |
| + } |
| + if !bytes.Equal(got, raw) || m.Swap != DemandHold { |
| + t.Fatalf("GetManifestRaw = %+v, %s; want the stored bytes and swap hold", m, got) |
| + } |
| +} |
| + |
| +// Spec §4.2: copies staged from now on are lossless. A field this binary does |
| +// not know survives in the sibling. |
| +func TestStagedSiblingKeepsTheCanonicalBytes(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + opt := t.TempDir() |
| + raw := putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"swap":"hold","from_a_newer_shipper":{"x":1},`) |
| + if _, err := ensureStaged(ctx, s, opt, "bab", "km", "v1", "h.example.com"); err != nil { |
| + t.Fatalf("ensureStaged: %v", err) |
| + } |
| + got, err := os.ReadFile(StagedManifestPath(opt, "bab", "v1", "km")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !bytes.Equal(got, raw) { |
| + t.Fatalf("staged sibling is not the canonical bytes:\n got %s\nwant %s", got, raw) |
| + } |
| + m, err := ReadStagedManifest(opt, "bab", "v1", "km") |
| + if err != nil || m == nil || m.Swap != DemandHold { |
| + t.Fatalf("ReadStagedManifest = %+v, %v; want the canonical manifest back", m, err) |
| + } |
| +} |
| + |
| +func TestBackfilledSiblingKeepsTheCanonicalBytes(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + opt := t.TempDir() |
| + raw := putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"from_a_newer_shipper":true,`) |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("bin")); err != nil { |
| + t.Fatal(err) |
| + } |
| + backfillManifest(ctx, s, opt, "bab", "km", "v1", "h.example.com") |
| + got, err := os.ReadFile(StagedManifestPath(opt, "bab", "v1", "km")) |
| + if err != nil || !bytes.Equal(got, raw) { |
| + t.Fatalf("back-filled sibling = %s, %v; want the canonical bytes %s", got, err, raw) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'GetManifestRaw|SiblingKeepsTheCanonicalBytes' -count=1` |
| +Expected: FAIL to compile, `undefined: GetManifestRaw`. |
| + |
| +- [ ] **Step 3: Add `GetManifestRaw`.** Replace the body of `GetManifest` in `release.go` and add the new function after it: |
| + |
| +```go |
| +func GetManifest(ctx context.Context, s Store, acct, app, version string) (*Manifest, error) { |
| + m, _, err := GetManifestRaw(ctx, s, acct, app, version) |
| + return m, err |
| +} |
| + |
| +// GetManifestRaw is GetManifest that also returns the exact bytes it read, so |
| +// a box can keep them (the staged sibling) rather than a re-marshal that |
| +// drops every field this binary does not know (spec §4.2). |
| +func GetManifestRaw(ctx context.Context, s Store, acct, app, version string) (*Manifest, []byte, error) { |
| + raw, _, err := GetQualified(ctx, s, ManifestKey(acct, app, version)) |
| + if err != nil { |
| + return nil, nil, err |
| + } |
| + var m Manifest |
| + if err := json.Unmarshal(raw, &m); err != nil { |
| + return nil, nil, fmt.Errorf("manifest %s/%s: %v", app, version, err) |
| + } |
| + if err := validateManifest(&m); err != nil { |
| + return nil, nil, fmt.Errorf("manifest %s/%s: %w", app, version, err) |
| + } |
| + if m.App != app || m.Version != version { |
| + return nil, nil, fmt.Errorf("manifest at %s names %s/%s", ManifestKey(acct, app, version), m.App, m.Version) |
| + } |
| + return &m, raw, nil |
| +} |
| +``` |
| + |
| +Keep `GetManifest`'s existing doc comment above it. |
| + |
| +- [ ] **Step 4: Stage the raw bytes.** In `adopt.go` `ensureStaged`, change `m, err := GetManifest(ctx, s, acct, app, ver)` to `m, raw, err := GetManifestRaw(ctx, s, acct, app, ver)`. Replace the block from the comment `// The sibling write is best-effort` to the end of the function with: |
| + |
| +```go |
| + // The sibling is the CANONICAL bytes (spec §4.2): what the bucket holds, |
| + // unknown fields and all, never a re-marshal by whichever binary staged |
| + // it. The write is best-effort: a failure here is not fatal to staging the |
| + // binary. A binary with an unreadable sibling is the leave-alone state, |
| + // which is safe (ReadStagedManifest's nil-nil-on-absence contract), and |
| + // backfillManifest gets another chance at it on every later pass. |
| + if err := StageManifest(optDir, acct, ver, app, raw); err != nil { |
| + fmt.Printf(" %s: stage manifest sibling for %s: %v\n", host, ver, err) |
| + } |
| + return m, nil |
| +} |
| +``` |
| + |
| +In `backfillManifest`, change `m, err := GetManifest(...)` to `_, raw, err := GetManifestRaw(ctx, s, acct, app, ver)`, and delete the `json.MarshalIndent` block so that `StageManifest(optDir, acct, ver, app, raw)` follows the error check. Remove `"encoding/json"` from `adopt.go`'s imports; nothing else in the file uses it. |
| + |
| +- [ ] **Step 5: Correct the sibling's doc comments.** In `versions.go`, in `StageManifest`'s comment, replace the paragraph starting "raw is whatever bytes the caller already has (ensureStaged's json.MarshalIndent of the fetched Manifest)" with: |
| + |
| +```go |
| +// raw is the canonical manifest's bytes as the store returned them |
| +// (GetManifestRaw), so the sibling is lossless (zero-downtime deploys spec |
| +// §4.2). This is the ONE write path that skips StageArtifact, so it repeats |
| +// StageArtifact's three charset gates itself rather than relying on a caller |
| +// that staged the binary first: the back-fill branch calls this with no |
| +// StageArtifact call in the same pass, and the reserved-word/traversal |
| +// protection those gates give the binary's path must cover this sibling path |
| +// too. |
| +``` |
| + |
| +In `ReadStagedManifest`'s comment, replace the paragraph starting "The sibling is a deliberately LOSSY re-marshal" with: |
| + |
| +```go |
| +// A sibling staged by this binary or later is the canonical manifest's exact |
| +// bytes. One staged by an older binary is a LOSSY re-marshal: it drops any |
| +// top-level key that binary's Manifest struct did not know, `swap` included. |
| +// So the bucket object stays the authority for anything a box acts on |
| +// (CanonicalDemand); this sibling exists so the converge pass can read a |
| +// running version's capabilities without a bucket round trip on every tick. |
| +``` |
| + |
| +- [ ] **Step 6: Run the package.** |
| + |
| +Run: `cd $W && go build ./... && go test ./internal/release/ -count=1` |
| +Expected: PASS, including `TestEnsureStagedWritesTheSiblingManifest`, `TestEnsureStagedBackfillsASiblingForAnAlreadyStagedBinary` and `TestPassBackfillsTheLiveVersionsSiblingOnAnUnchangedPass`. |
| + |
| +- [ ] **Step 7: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/release.go internal/release/adopt.go internal/release/versions.go internal/release/staged_raw_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 2: staged manifest siblings keep the canonical bytes |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: `StagedIdentity`, `ResolveLive` and `VerifyStaged` |
| + |
| +**Files:** |
| +- Create: `internal/release/staged.go` |
| +- Modify: `internal/release/versions.go:290-313` (`LiveVersion` gains a helper) |
| +- Create: `internal/release/staged_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `release.FileSHA256(path string) (string, error)` (`proof.go:169`, existing). |
| +- Produces: |
| + - `type StagedIdentity struct { Version, StagedPath, Digest, Demand string }`, exactly as the index declares. |
| + - `var ErrStagedCopyChanged error`, `var ErrVersionReused error`. |
| + - `func ResolveLive(optDir, host string) (StagedIdentity, error)`: one `Readlink`, then the target's path and sha256. `Demand` is always `""` here; plan 5 fills it from `CanonicalDemand`. |
| + - `func VerifyStaged(id StagedIdentity) error`: nil only when `id.StagedPath` hashes to `id.Digest`. Every refusal wraps `ErrStagedCopyChanged`. |
| + - unexported `func versionOfTarget(optDir, target string) string`. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/staged_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| +) |
| + |
| +func TestResolveLiveReadsTheLinkOnce(t *testing.T) { |
| + opt := t.TempDir() |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("build one")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := PointLive(opt, "h.example.com", "bab", "v1", "km"); err != nil { |
| + t.Fatal(err) |
| + } |
| + id, err := ResolveLive(opt, "h.example.com") |
| + if err != nil { |
| + t.Fatalf("ResolveLive: %v", err) |
| + } |
| + want := StagedIdentity{Version: "v1", StagedPath: VersionPath(opt, "bab", "v1", "km"), Digest: etagOf([]byte("build one"))} |
| + if id != want { |
| + t.Fatalf("ResolveLive = %+v, want %+v", id, want) |
| + } |
| + if id.StagedPath == LiveLink(opt, "h.example.com") { |
| + t.Fatal("the identity names the live link, which moves; it must name the staged file") |
| + } |
| +} |
| + |
| +// Review Focus 1: a link an operator made by hand, outside the version store. |
| +func TestResolveLiveOutsideTheStoreStillIdentifiesTheBytes(t *testing.T) { |
| + opt := t.TempDir() |
| + elsewhere := filepath.Join(t.TempDir(), "hand-built") |
| + if err := os.WriteFile(elsewhere, []byte("operator's build"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(elsewhere, LiveLink(opt, "h.example.com")); err != nil { |
| + t.Fatal(err) |
| + } |
| + id, err := ResolveLive(opt, "h.example.com") |
| + if err != nil { |
| + t.Fatalf("ResolveLive: %v", err) |
| + } |
| + want := StagedIdentity{Version: "", StagedPath: elsewhere, Digest: etagOf([]byte("operator's build"))} |
| + if id != want { |
| + t.Fatalf("ResolveLive = %+v, want %+v (no version, but the bytes still identified)", id, want) |
| + } |
| +} |
| + |
| +// Review Focus 1: a relative link resolves against live/, and its version |
| +// reads exactly as LiveVersion reads it. |
| +func TestResolveLiveFollowsARelativeLink(t *testing.T) { |
| + opt := t.TempDir() |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("b")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(filepath.Join("..", "versions", "v1", "km"), LiveLink(opt, "h.example.com")); err != nil { |
| + t.Fatal(err) |
| + } |
| + id, err := ResolveLive(opt, "h.example.com") |
| + if err != nil { |
| + t.Fatalf("ResolveLive: %v", err) |
| + } |
| + lv, _ := LiveVersion(opt, "h.example.com") |
| + if id.StagedPath != VersionPath(opt, "bab", "v1", "km") || id.Digest != etagOf([]byte("b")) || id.Version != lv { |
| + t.Fatalf("ResolveLive = %+v; want path %s, the file's digest, and version %q", id, VersionPath(opt, "bab", "v1", "km"), lv) |
| + } |
| +} |
| + |
| +func TestResolveLiveWithNoLinkIsAnError(t *testing.T) { |
| + if _, err := ResolveLive(t.TempDir(), "h.example.com"); err == nil { |
| + t.Fatal("ResolveLive with no live link returned no error") |
| + } |
| +} |
| + |
| +func TestVerifyStaged(t *testing.T) { |
| + opt := t.TempDir() |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("build one")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := PointLive(opt, "h.example.com", "bab", "v1", "km"); err != nil { |
| + t.Fatal(err) |
| + } |
| + id, err := ResolveLive(opt, "h.example.com") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := VerifyStaged(id); err != nil { |
| + t.Fatalf("VerifyStaged on untouched bytes: %v", err) |
| + } |
| + // Damaged in place, to a different length. |
| + if err := os.WriteFile(id.StagedPath, []byte("build one, damaged"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := VerifyStaged(id); !errors.Is(err, ErrStagedCopyChanged) { |
| + t.Fatalf("VerifyStaged on changed bytes = %v, want ErrStagedCopyChanged", err) |
| + } |
| + if err := os.Remove(id.StagedPath); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := VerifyStaged(id); !errors.Is(err, ErrStagedCopyChanged) { |
| + t.Fatalf("VerifyStaged on a missing file = %v, want ErrStagedCopyChanged", err) |
| + } |
| + if err := VerifyStaged(StagedIdentity{Version: "v1"}); !errors.Is(err, ErrStagedCopyChanged) { |
| + t.Fatalf("VerifyStaged with no retained path or digest = %v, want ErrStagedCopyChanged", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'ResolveLive|VerifyStaged' -count=1` |
| +Expected: FAIL to compile, `undefined: ResolveLive`. |
| + |
| +- [ ] **Step 3: Split `versionOfTarget` out of `LiveVersion`.** In `versions.go`, replace `LiveVersion`'s body after the `Readlink` error check with `return versionOfTarget(optDir, target), nil`, and add: |
| + |
| +```go |
| +// versionOfTarget is the version a live link's RAW target names, or "" when |
| +// the target is not under the version store. target is |
| +// <opt>/versions/<ver>/<app> (ops) or <opt>/versions/<ver>/<acct>/<app> |
| +// (everyone else), so the version is the FIRST segment under the versions |
| +// dir, not simply the parent directory, which would read an account sqid as |
| +// a version. A target outside the store, or a relative one, is not ours to |
| +// interpret: "" is the same answer an absent link gets. |
| +func versionOfTarget(optDir, target string) string { |
| + rel, err := filepath.Rel(versionsDir(optDir), target) |
| + if err != nil || strings.HasPrefix(rel, "..") { |
| + return "" |
| + } |
| + ver, _, ok := strings.Cut(filepath.ToSlash(rel), "/") |
| + if !ok { |
| + return "" |
| + } |
| + return ver |
| +} |
| +``` |
| + |
| +Move the existing comment from inside `LiveVersion` ("target is <opt>/versions/...") into this helper's doc as above, and delete it from `LiveVersion`. |
| + |
| +- [ ] **Step 4: Create `internal/release/staged.go`.** |
| + |
| +```go |
| +// The identity of a staged artifact (zero-downtime deploys spec §4.2, §7.9, |
| +// §8.5): which bytes a process was exec'd from, kept so that recovery runs |
| +// exactly those bytes again and can tell when they have changed. |
| +package release |
| + |
| +import ( |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| +) |
| + |
| +// StagedIdentity is the retained identity of a staged artifact (spec §4.2, |
| +// §8.5): the version name it was staged under, the immutable path it is |
| +// exec'd from, the sha256 of its bytes, and the release's own swap demand as |
| +// verified for those bytes ("hold" or ""). The agent captures it at exec, |
| +// never from the live link afterwards, because the link moves under a |
| +// running process. |
| +type StagedIdentity struct { |
| + Version string |
| + StagedPath string |
| + Digest string |
| + Demand string |
| +} |
| + |
| +// ErrStagedCopyChanged is recovery's refusal (spec §4.2): the staged file no |
| +// longer hashes to the digest retained for it, so it is not the build that |
| +// passed, and running it would be running something nobody verified. |
| +var ErrStagedCopyChanged = errors.New("staged copy no longer matches the digest it was verified with") |
| + |
| +// ErrVersionReused is root staging's refusal (spec §4.2): the store holds |
| +// different bytes under a version name this box already staged, and |
| +// something on the box still refers to the staged copy, so it is not |
| +// overwritten. The text is the spec's. |
| +var ErrVersionReused = errors.New("version name reused with different content, ship under a new name") |
| + |
| +// ResolveLive reads host's live link ONCE and reports the file it names and |
| +// that file's sha256. Version is the link's version as LiveVersion reads it, |
| +// "" for a link outside the version store; the path and digest identify the |
| +// bytes either way. Demand is left empty: only CanonicalDemand can vouch for |
| +// it. |
| +func ResolveLive(optDir, host string) (StagedIdentity, error) { |
| + link := LiveLink(optDir, host) |
| + raw, err := os.Readlink(link) |
| + if err != nil { |
| + return StagedIdentity{}, fmt.Errorf("read live link %s: %w", link, err) |
| + } |
| + path := raw |
| + if !filepath.IsAbs(path) { |
| + path = filepath.Join(filepath.Dir(link), path) |
| + } |
| + path = filepath.Clean(path) |
| + sum, err := FileSHA256(path) |
| + if err != nil { |
| + return StagedIdentity{}, fmt.Errorf("hash %s: %w", path, err) |
| + } |
| + return StagedIdentity{Version: versionOfTarget(optDir, raw), StagedPath: path, Digest: sum}, nil |
| +} |
| + |
| +// VerifyStaged checks the file at id.StagedPath against id.Digest, the digest |
| +// retained when the build was exec'd (spec §4.2's recovery check). It never |
| +// consults the canonical manifest: a later re-ship of the same name must not |
| +// revoke local recovery. |
| +func VerifyStaged(id StagedIdentity) error { |
| + if id.StagedPath == "" || id.Digest == "" { |
| + return fmt.Errorf("%w: no retained path and digest for %q", ErrStagedCopyChanged, id.Version) |
| + } |
| + sum, err := FileSHA256(id.StagedPath) |
| + if err != nil { |
| + return fmt.Errorf("%w: %s: %v", ErrStagedCopyChanged, id.StagedPath, err) |
| + } |
| + if !strings.EqualFold(sum, id.Digest) { |
| + return fmt.Errorf("%w: %s hashes to %s, retained %s", ErrStagedCopyChanged, id.StagedPath, sum, id.Digest) |
| + } |
| + return nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 5: Run the package.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -count=1` |
| +Expected: PASS, including every `LiveVersion` test in `versions_test.go` and `sweep_test.go`. |
| + |
| +- [ ] **Step 6: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/staged.go internal/release/staged_test.go internal/release/versions.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 3: StagedIdentity, ResolveLive and VerifyStaged |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: `CanonicalDemand` |
| + |
| +**Files:** |
| +- Modify: `internal/release/demand.go` |
| +- Modify: `internal/release/demand_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `GetManifestRaw` (Task 2), `NormalizeDemand`, `DemandHold` (Task 1), `FileSHA256`. |
| +- Produces (index signature, unchanged): |
| + - `func CanonicalDemand(ctx context.Context, s Store, acct, app, version, platform, stagedPath string) (demand string, digest string, err error)`. |
| + - `var ErrStagedDigestMismatch error`. |
| + - Results: match gives `(NormalizeDemand(m.Swap), stagedSHA, nil)`. Mismatch gives `("", stagedSHA, err)` with `errors.Is(err, ErrStagedDigestMismatch)`. Cannot compare gives `(DemandHold, stagedSHA or "", nil)`. Plan 5 refuses the attempt on a non-nil error and runs it in `hold` whenever `demand == DemandHold`. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Append to `internal/release/demand_test.go`, adding `"errors"` and `"os"` to its imports: |
| + |
| +```go |
| +// unreadableStore answers every Get with an error that is not "not found": a |
| +// bucket this box cannot read right now. |
| +type unreadableStore struct{ Store } |
| + |
| +func (unreadableStore) Get(context.Context, string) ([]byte, string, error) { |
| + return nil, "", errors.New("bucket unreachable") |
| +} |
| + |
| +// Spec §12 "Swap mode source": a staged copy without swap, from an older |
| +// root, cannot produce an overlap for a hold build. |
| +func TestCanonicalDemandReadsTheCanonicalManifestNotTheSibling(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + opt := t.TempDir() |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"swap":"hold",`) |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("bin")); err != nil { |
| + t.Fatal(err) |
| + } |
| + canon, err := GetManifest(ctx, s, "bab", "km", "v1") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + lossy := *canon |
| + lossy.Swap = "" // what an older root's re-marshal wrote |
| + b, err := json.MarshalIndent(lossy, "", " ") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := StageManifest(opt, "bab", "v1", "km", b); err != nil { |
| + t.Fatal(err) |
| + } |
| + demand, digest, err := CanonicalDemand(ctx, s, "bab", "km", "v1", adoptPlatform, VersionPath(opt, "bab", "v1", "km")) |
| + if err != nil || demand != DemandHold || digest != etagOf([]byte("bin")) { |
| + t.Fatalf("CanonicalDemand = %q, %q, %v; want hold from the canonical manifest", demand, digest, err) |
| + } |
| +} |
| + |
| +func TestCanonicalDemandWithNoDemand(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + opt := t.TempDir() |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), "") |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("bin")); err != nil { |
| + t.Fatal(err) |
| + } |
| + demand, digest, err := CanonicalDemand(context.Background(), s, "bab", "km", "v1", adoptPlatform, VersionPath(opt, "bab", "v1", "km")) |
| + if err != nil || demand != "" || digest != etagOf([]byte("bin")) { |
| + t.Fatalf("CanonicalDemand = %q, %q, %v; want no demand", demand, digest, err) |
| + } |
| +} |
| + |
| +// Spec §12 "Artifact binding": a re-shipped name with different bytes, plus |
| +// an old staged copy, is a staging error. |
| +func TestCanonicalDemandRefusesBytesTheReleaseDoesNotName(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + opt := t.TempDir() |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("new bytes"), "") |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("old bytes")); err != nil { |
| + t.Fatal(err) |
| + } |
| + demand, digest, err := CanonicalDemand(context.Background(), s, "bab", "km", "v1", adoptPlatform, VersionPath(opt, "bab", "v1", "km")) |
| + if !errors.Is(err, ErrStagedDigestMismatch) { |
| + t.Fatalf("err = %v, want ErrStagedDigestMismatch", err) |
| + } |
| + if demand != "" || digest != etagOf([]byte("old bytes")) { |
| + t.Fatalf("CanonicalDemand = %q, %q; want no demand and the staged file's own digest", demand, digest) |
| + } |
| +} |
| + |
| +// Spec §4.2: if the canonical manifest cannot be read, or the digests cannot |
| +// be compared, the attempt runs in hold. Review Focus 2 is the last case. |
| +func TestCanonicalDemandIsHoldWhenItCannotCompare(t *testing.T) { |
| + ctx := context.Background() |
| + staged := func(t *testing.T, opt string) string { |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("bin")); err != nil { |
| + t.Fatal(err) |
| + } |
| + return VersionPath(opt, "bab", "v1", "km") |
| + } |
| + t.Run("manifest missing", func(t *testing.T) { |
| + opt := t.TempDir() |
| + d, sum, err := CanonicalDemand(ctx, NewDirStore(t.TempDir()), "bab", "km", "v1", adoptPlatform, staged(t, opt)) |
| + if err != nil || d != DemandHold || sum != etagOf([]byte("bin")) { |
| + t.Fatalf("= %q, %q, %v; want hold", d, sum, err) |
| + } |
| + }) |
| + t.Run("store unreadable", func(t *testing.T) { |
| + opt := t.TempDir() |
| + s := NewDirStore(t.TempDir()) |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), "") |
| + d, _, err := CanonicalDemand(ctx, unreadableStore{s}, "bab", "km", "v1", adoptPlatform, staged(t, opt)) |
| + if err != nil || d != DemandHold { |
| + t.Fatalf("= %q, %v; want hold", d, err) |
| + } |
| + }) |
| + t.Run("no build for this platform", func(t *testing.T) { |
| + opt := t.TempDir() |
| + s := NewDirStore(t.TempDir()) |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), "") |
| + d, _, err := CanonicalDemand(ctx, s, "bab", "km", "v1", "plan9-mips", staged(t, opt)) |
| + if err != nil || d != DemandHold { |
| + t.Fatalf("= %q, %v; want hold", d, err) |
| + } |
| + }) |
| + t.Run("staged file missing", func(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), "") |
| + missing := VersionPath(t.TempDir(), "bab", "v1", "km") |
| + if _, err := os.Stat(missing); err == nil { |
| + t.Fatal("setup: the path exists") |
| + } |
| + d, sum, err := CanonicalDemand(ctx, s, "bab", "km", "v1", adoptPlatform, missing) |
| + if err != nil || d != DemandHold || sum != "" { |
| + t.Fatalf("= %q, %q, %v; want hold and no digest", d, sum, err) |
| + } |
| + }) |
| + t.Run("unknown demand", func(t *testing.T) { |
| + opt := t.TempDir() |
| + s := NewDirStore(t.TempDir()) |
| + putRawRelease(t, s, "bab", "km", "v1", []byte("bin"), `"swap":"drain",`) |
| + d, _, err := CanonicalDemand(ctx, s, "bab", "km", "v1", adoptPlatform, staged(t, opt)) |
| + if err != nil || d != DemandHold { |
| + t.Fatalf("= %q, %v; want hold", d, err) |
| + } |
| + }) |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run CanonicalDemand -count=1` |
| +Expected: FAIL to compile, `undefined: CanonicalDemand`. |
| + |
| +- [ ] **Step 3: Implement.** Append to `internal/release/demand.go`, with imports `context`, `errors`, `fmt`, `strings`: |
| + |
| +```go |
| +// ErrStagedDigestMismatch is an adoption attempt's staging error (spec §4.2): |
| +// the staged executable is not the artifact the canonical manifest names for |
| +// this platform, so the manifest's demand does not describe the bytes that |
| +// would run. The attempt is refused, with no failure record against the |
| +// build. |
| +var ErrStagedDigestMismatch = errors.New("staged artifact is not the release's canonical artifact") |
| + |
| +// CanonicalDemand reads build version's own swap demand from its CANONICAL |
| +// manifest in the release store, and checks the staged executable at |
| +// stagedPath against that manifest's digest for platform (spec §4.2). |
| +// |
| +// - Match: the manifest's demand (NormalizeDemand), the staged digest, nil. |
| +// - Mismatch: "", the staged digest, an error wrapping |
| +// ErrStagedDigestMismatch. |
| +// - Cannot compare (the manifest is unreadable, it has no artifact for |
| +// platform, or the staged file cannot be read): DemandHold and a nil |
| +// error, because missing or unverifiable policy data never permits an |
| +// overlap. digest is the staged file's when it could be hashed, else "". |
| +// |
| +// It is for ADOPTION attempts only (causes release, install, retry). Recovery |
| +// checks a retained digest with VerifyStaged and never asks the store. |
| +func CanonicalDemand(ctx context.Context, s Store, acct, app, version, platform, stagedPath string) (demand string, digest string, err error) { |
| + digest, herr := FileSHA256(stagedPath) |
| + if herr != nil { |
| + return DemandHold, "", nil |
| + } |
| + m, _, merr := GetManifestRaw(ctx, s, acct, app, version) |
| + if merr != nil { |
| + return DemandHold, digest, nil |
| + } |
| + art, ok := m.Artifacts[platform] |
| + if !ok { |
| + return DemandHold, digest, nil |
| + } |
| + if !strings.EqualFold(art.SHA256, digest) { |
| + return "", digest, fmt.Errorf("%w: %s/%s staged at %s hashes to %s; the release names %s", |
| + ErrStagedDigestMismatch, app, version, stagedPath, digest, art.SHA256) |
| + } |
| + return NormalizeDemand(m.Swap), digest, nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the package.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/demand.go internal/release/demand_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 4: CanonicalDemand reads the canonical manifest and checks the staged digest |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: The pin set and its provider interface |
| + |
| +**Files:** |
| +- Create: `internal/release/pins.go` |
| +- Create: `internal/release/pins_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type StagedPaths map[string]bool` with `func (sp StagedPaths) Has(p string) bool` (cleans `p`) and unexported `add(p string)`. |
| + - `type PinSource interface { StagedPins(ctx context.Context, optDir string) (StagedPaths, error) }`. **Plan 5 implements it** with `GoodVersion` (registry) and the activator's live and pending generations (current, successor, retiring, spawning: spec §8.5). |
| + - `type PinSourceFunc func(ctx context.Context, optDir string) (StagedPaths, error)`, which satisfies `PinSource`. |
| + - `func BoxPins(optDir string, routes []registry.Route) (StagedPaths, error)`: every live link's target (including `<host>.tmp` leftovers), plus `VersionPath(optDir, RouteAccount(rt), v, rt.App)` for each row's `Version` and `PrevVersion`. An unreadable live directory or link is an error, never a narrower set. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/pins_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "context" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +func TestBoxPinsNamesLiveTargetsAndRegistryRows(t *testing.T) { |
| + opt := t.TempDir() |
| + for _, v := range []string{"v1", "v2"} { |
| + if err := StageArtifact(opt, "bab", v, "km", []byte(v)); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := StageArtifact(opt, "kqmv", "v3", "km", []byte("v3")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := PointLive(opt, "a.example.com", "bab", "v1", "km"); err != nil { |
| + t.Fatal(err) |
| + } |
| + routes := []registry.Route{ |
| + {Host: "b.example.com", Kind: "instance", App: "km", Version: "v2", PrevVersion: "v1"}, |
| + {Host: "c.example.com", Kind: "sidecar", App: "km", Account: "kqmv", Version: "v3"}, |
| + {Host: "d.example.com", Kind: "instance", App: "", Version: "v9"}, // no app: names no staged path |
| + } |
| + pins, err := BoxPins(opt, routes) |
| + if err != nil { |
| + t.Fatalf("BoxPins: %v", err) |
| + } |
| + want := []string{VersionPath(opt, "bab", "v1", "km"), VersionPath(opt, "bab", "v2", "km"), VersionPath(opt, "kqmv", "v3", "km")} |
| + if len(pins) != len(want) { |
| + t.Fatalf("BoxPins = %v, want exactly %v", pins, want) |
| + } |
| + for _, p := range want { |
| + if !pins.Has(p) { |
| + t.Errorf("BoxPins is missing %s", p) |
| + } |
| + } |
| +} |
| + |
| +// A crashed PointLive leaves <host>.tmp naming a real staged tree. It is a |
| +// reference, as it is to LivePins. |
| +func TestBoxPinsKeepsATmpLeftover(t *testing.T) { |
| + opt := t.TempDir() |
| + if err := StageArtifact(opt, "bab", "v1", "km", []byte("v1")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(VersionPath(opt, "bab", "v1", "km"), LiveLink(opt, "a.example.com")+".tmp"); err != nil { |
| + t.Fatal(err) |
| + } |
| + pins, err := BoxPins(opt, nil) |
| + if err != nil || !pins.Has(VersionPath(opt, "bab", "v1", "km")) { |
| + t.Fatalf("BoxPins = %v, %v; want the leftover's target pinned", pins, err) |
| + } |
| +} |
| + |
| +// Losing a pin is how a running build gets overwritten, so an unreadable |
| +// entry stops the answer rather than narrowing it. |
| +func TestBoxPinsFailsOnAnUnreadableLink(t *testing.T) { |
| + opt := t.TempDir() |
| + if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(filepath.Join(opt, "live", "not-a-link"), []byte("x"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := BoxPins(opt, nil); err == nil { |
| + t.Fatal("BoxPins read past an entry it could not interpret") |
| + } |
| +} |
| + |
| +func TestBoxPinsWithNoLiveDir(t *testing.T) { |
| + pins, err := BoxPins(t.TempDir(), nil) |
| + if err != nil || len(pins) != 0 { |
| + t.Fatalf("BoxPins on a fresh box = %v, %v; want empty and no error", pins, err) |
| + } |
| +} |
| + |
| +func TestPinSourceFuncIsAPinSource(t *testing.T) { |
| + var src PinSource = PinSourceFunc(func(context.Context, string) (StagedPaths, error) { |
| + return StagedPaths{"/opt/carlos/versions/v1/km": true}, nil |
| + }) |
| + got, err := src.StagedPins(context.Background(), "/opt/carlos") |
| + if err != nil || !got.Has("/opt/carlos/versions/v1/./km") { |
| + t.Fatalf("StagedPins = %v, %v", got, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'BoxPins|PinSource' -count=1` |
| +Expected: FAIL to compile, `undefined: BoxPins`. |
| + |
| +- [ ] **Step 3: Create `internal/release/pins.go`.** |
| + |
| +```go |
| +// The pin set (zero-downtime deploys spec §4.2, §8.5): every staged copy |
| +// something on this box still refers to. A staged copy in it is never |
| +// overwritten, because staged paths are keyed by version name with no digest |
| +// and another instance's good build or running generation would change |
| +// underneath it. |
| +package release |
| + |
| +import ( |
| + "context" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// StagedPaths is a set of staged artifact paths (VersionPath values, held |
| +// cleaned) that something on this box refers to. |
| +type StagedPaths map[string]bool |
| + |
| +func (sp StagedPaths) add(p string) { sp[filepath.Clean(p)] = true } |
| + |
| +// Has reports whether p is referred to. |
| +func (sp StagedPaths) Has(p string) bool { return sp[filepath.Clean(p)] } |
| + |
| +// PinSource supplies the references this package cannot read for itself: |
| +// GoodVersion and every live or pending generation (current, successor, |
| +// retiring, spawning). Plan 5 implements it. A source that cannot answer |
| +// returns an error, and an error never permits a restage. |
| +type PinSource interface { |
| + StagedPins(ctx context.Context, optDir string) (StagedPaths, error) |
| +} |
| + |
| +// PinSourceFunc adapts a function to PinSource. |
| +type PinSourceFunc func(ctx context.Context, optDir string) (StagedPaths, error) |
| + |
| +func (f PinSourceFunc) StagedPins(ctx context.Context, optDir string) (StagedPaths, error) { |
| + return f(ctx, optDir) |
| +} |
| + |
| +// BoxPins is the part of the pin set this package reads itself: the target |
| +// of every entry under <opt>/live (a crashed swap's <host>.tmp included, as |
| +// LivePins includes it), and the staged path of every row's Version and |
| +// PrevVersion. routes is every registry row, unfiltered: a hibernating route, |
| +// an expired canary and a mid-provision row all still refer to their builds. |
| +// |
| +// An entry that cannot be read is an error, never a skipped entry: a pin set |
| +// that quietly narrowed is how a running build's bytes get replaced. |
| +func BoxPins(optDir string, routes []registry.Route) (StagedPaths, error) { |
| + pins := StagedPaths{} |
| + dir := liveDir(optDir) |
| + entries, err := os.ReadDir(dir) |
| + if err != nil && !os.IsNotExist(err) { |
| + return nil, fmt.Errorf("read %s: %w", dir, err) |
| + } |
| + for _, e := range entries { |
| + link := filepath.Join(dir, e.Name()) |
| + target, err := os.Readlink(link) |
| + if err != nil { |
| + return nil, fmt.Errorf("read live link %s: %w", link, err) |
| + } |
| + if !filepath.IsAbs(target) { |
| + target = filepath.Join(dir, target) |
| + } |
| + pins.add(target) |
| + } |
| + for _, rt := range routes { |
| + if rt.App == "" { |
| + continue |
| + } |
| + for _, v := range []string{rt.Version, rt.PrevVersion} { |
| + if ValidVersion(v) { |
| + pins.add(VersionPath(optDir, RouteAccount(rt), v, rt.App)) |
| + } |
| + } |
| + } |
| + return pins, nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the package.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/pins.go internal/release/pins_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 5: the staged-copy pin set and the PinSource plan 5 fills |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: Root staging checks a reused staged copy |
| + |
| +**Files:** |
| +- Modify: `internal/release/adopt.go:36-88` (`passConfig`), `:336-362` (`PassFilter`, after `cfg.selected = selected`), `:1087` (the `ensureStaged` call in `adoptHost`), `:1585-1683` (`ensureStaged`) |
| +- Create: `internal/release/stage_reuse_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `GetManifestRaw` (Task 2), `ErrVersionReused` (Task 3), `StagedPaths`, `PinSource`, `BoxPins` (Task 5), `FileSHA256`. |
| +- Produces: |
| + - `func WithPinSource(src PinSource) PassOption`. Plan 5 passes it to every pass. |
| + - unexported `type stageCheck struct { refs func() (StagedPaths, error); verified map[string]bool }`. |
| + - unexported `func ensureStagedChecked(ctx context.Context, s Store, optDir, acct, app, ver, host string, chk stageCheck) (*Manifest, error)`. `ensureStaged` keeps its signature for existing tests and calls this with `BoxPins(optDir, nil)`. |
| + - Refusal: `fmt.Errorf("%s/%s: %w", app, ver, ErrVersionReused)`, printed by `adoptHost` as ` <host>: <app>/<ver>: version name reused with different content, ship under a new name`. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/stage_reuse_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "crypto/ed25519" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// reuseFixture: h1 follows canary/x and serves v1. Then v2 is shipped as |
| +// "new v2" while a stale copy of v2 ("old v2", from an earlier release under |
| +// the same name that retention has since pruned) is already staged, and the |
| +// channel moves to v2. |
| +type reuseFixture struct { |
| + s Store |
| + priv ed25519.PrivateKey |
| + pub ed25519.PublicKey |
| + opt string |
| + reg *registry.Registry |
| +} |
| + |
| +func newReuseFixture(t *testing.T) reuseFixture { |
| + t.Helper() |
| + ctx := context.Background() |
| + f := reuseFixture{s: NewDirStore(t.TempDir()), opt: t.TempDir(), reg: testRegistry(t)} |
| + f.priv, f.pub = testKeys(t) |
| + shipRelease(t, f.s, "bab", "myapp", "v1", adoptPlatform, []byte("v1")) |
| + if err := Promote(ctx, f.s, f.priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, f.reg, "h1.example.com", "myapp", "canary/x", false) |
| + if err := Pass(ctx, f.reg, f.s, f.pub, f.opt, func(string) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if v, _ := LiveVersion(f.opt, "h1.example.com"); v != "v1" { |
| + t.Fatalf("setup: live = %q, want v1", v) |
| + } |
| + if err := StageArtifact(f.opt, "bab", "v2", "myapp", []byte("old v2")); err != nil { |
| + t.Fatal(err) |
| + } |
| + shipRelease(t, f.s, "bab", "myapp", "v2", adoptPlatform, []byte("new v2")) |
| + if err := Promote(ctx, f.s, f.priv, "bab", "myapp", "canary/x", "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + return f |
| +} |
| + |
| +func (f reuseFixture) pass(t *testing.T, opts ...PassOption) string { |
| + t.Helper() |
| + var out bytes.Buffer |
| + withCapturedStdout(t, &out, func() { |
| + if err := Pass(context.Background(), f.reg, f.s, f.pub, f.opt, func(string) error { return nil }, opts...); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + }) |
| + return out.String() |
| +} |
| + |
| +func (f reuseFixture) stagedV2(t *testing.T) string { |
| + t.Helper() |
| + b, err := os.ReadFile(VersionPath(f.opt, "bab", "v2", "myapp")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return string(b) |
| +} |
| + |
| +// assertRefused: h1 stays on v1, the stale copy is untouched, and the pass |
| +// said why. |
| +func (f reuseFixture) assertRefused(t *testing.T, out, why string) { |
| + t.Helper() |
| + if got := f.stagedV2(t); got != "old v2" { |
| + t.Fatalf("the referenced copy was overwritten: it now holds %q", got) |
| + } |
| + if v, _ := LiveVersion(f.opt, "h1.example.com"); v != "v1" { |
| + t.Fatalf("h1 live = %q, want v1: the reused name must not install", v) |
| + } |
| + if rt, _ := f.reg.Get("h1.example.com"); rt == nil || rt.Version != "v1" { |
| + t.Fatalf("h1 row = %+v, want Version v1", rt) |
| + } |
| + if !strings.Contains(out, why) { |
| + t.Fatalf("pass printed %q, want it to contain %q", out, why) |
| + } |
| +} |
| + |
| +// Spec §12 "Restage refusal": an unreferenced stale copy is restaged. |
| +func TestUnreferencedStaleCopyIsRestaged(t *testing.T) { |
| + f := newReuseFixture(t) |
| + out := f.pass(t) |
| + if got := f.stagedV2(t); got != "new v2" { |
| + t.Fatalf("staged v2 = %q, want the canonical bytes", got) |
| + } |
| + if v, _ := LiveVersion(f.opt, "h1.example.com"); v != "v2" { |
| + t.Fatalf("h1 live = %q, want v2", v) |
| + } |
| + raw, _, err := f.s.Get(context.Background(), ManifestKey("bab", "myapp", "v2")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + sib, err := os.ReadFile(StagedManifestPath(f.opt, "bab", "v2", "myapp")) |
| + if err != nil || !bytes.Equal(sib, raw) { |
| + t.Fatalf("sibling = %s, %v; want the canonical manifest bytes", sib, err) |
| + } |
| + if !strings.Contains(out, "restaging") { |
| + t.Fatalf("pass printed %q; want a line saying the copy was restaged", out) |
| + } |
| +} |
| + |
| +// A row's PrevVersion is a reference (spec §4.2's pin set). |
| +func TestStaleCopyARegistryRowNamesIsNotOverwritten(t *testing.T) { |
| + f := newReuseFixture(t) |
| + // h2 is mid-provision, so no pass adopts it, but its row still names v2. |
| + if err := f.reg.Upsert(registry.Route{Host: "h2.example.com", Kind: "instance", App: "myapp", Channel: "canary/x", Status: "creating"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := f.reg.SetVersions("h2.example.com", "v1", "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.assertRefused(t, f.pass(t), ErrVersionReused.Error()) |
| +} |
| + |
| +func TestStaleCopyALiveLinkNamesIsNotOverwritten(t *testing.T) { |
| + f := newReuseFixture(t) |
| + if err := PointLive(f.opt, "h2.example.com", "bab", "v2", "myapp"); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.assertRefused(t, f.pass(t), ErrVersionReused.Error()) |
| +} |
| + |
| +// GoodVersion and live generations arrive through a PinSource (plan 5). |
| +func TestStaleCopyAPinSourceNamesIsNotOverwritten(t *testing.T) { |
| + f := newReuseFixture(t) |
| + src := PinSourceFunc(func(context.Context, string) (StagedPaths, error) { |
| + return StagedPaths{VersionPath(f.opt, "bab", "v2", "myapp"): true}, nil |
| + }) |
| + f.assertRefused(t, f.pass(t, WithPinSource(src)), ErrVersionReused.Error()) |
| +} |
| + |
| +func TestUnknownReferencesNeverPermitARestage(t *testing.T) { |
| + f := newReuseFixture(t) |
| + src := PinSourceFunc(func(context.Context, string) (StagedPaths, error) { |
| + return nil, errors.New("generation report unreadable") |
| + }) |
| + f.assertRefused(t, f.pass(t, WithPinSource(src)), "what refers to it is unknown") |
| +} |
| + |
| +// Two hosts moving onto one stale build in one pass: restaged once, both move. |
| +func TestOneRestageServesEveryHostInThePass(t *testing.T) { |
| + f := newReuseFixture(t) |
| + addRoute(t, f.reg, "h3.example.com", "myapp", "canary/x", false) |
| + out := f.pass(t) |
| + if n := strings.Count(out, "restaging"); n != 1 { |
| + t.Fatalf("restaged %d times, want once:\n%s", n, out) |
| + } |
| + for _, h := range []string{"h1.example.com", "h3.example.com"} { |
| + if v, _ := LiveVersion(f.opt, h); v != "v2" { |
| + t.Fatalf("%s live = %q, want v2", h, v) |
| + } |
| + } |
| +} |
| + |
| +// Unchanged behaviour: a reused copy that matches is not downloaded again. |
| +func TestMatchingStagedCopyIsReusedWithoutADownload(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v2", adoptPlatform, []byte("v2 bytes")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := StageArtifact(opt, "bab", "v2", "myapp", []byte("v2 bytes")); err != nil { |
| + t.Fatal(err) |
| + } |
| + old := time.Unix(1_600_000_000, 0) |
| + bin := VersionPath(opt, "bab", "v2", "myapp") |
| + if err := os.Chtimes(bin, old, old); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, reg, "h1.example.com", "myapp", "canary/x", false) |
| + if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if v, _ := LiveVersion(opt, "h1.example.com"); v != "v2" { |
| + t.Fatalf("live = %q, want v2", v) |
| + } |
| + if info, err := os.Stat(bin); err != nil || !info.ModTime().Equal(old) { |
| + t.Fatalf("the matching copy was rewritten (mtime %v, %v)", info.ModTime(), err) |
| + } |
| +} |
| + |
| +// Review Focus 3: when the bucket cannot answer for the manifest, a staged |
| +// copy keeps today's answer. It is used, never restaged, and nothing claims a |
| +// mismatch the box could not establish. |
| +func TestUnreadableManifestKeepsTodaysAnswerForAStagedCopy(t *testing.T) { |
| + dir := t.TempDir() |
| + s := NewDirStore(dir) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v2", adoptPlatform, []byte("new v2")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := StageArtifact(opt, "bab", "v2", "myapp", []byte("old v2")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Remove(filepath.Join(dir, filepath.FromSlash(ManifestKey("bab", "myapp", "v2")))); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, reg, "h1.example.com", "myapp", "canary/x", false) |
| + var out bytes.Buffer |
| + withCapturedStdout(t, &out, func() { |
| + if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + }) |
| + if v, _ := LiveVersion(opt, "h1.example.com"); v != "v2" { |
| + t.Fatalf("live = %q, want v2 (today's answer)", v) |
| + } |
| + b, _ := os.ReadFile(VersionPath(opt, "bab", "v2", "myapp")) |
| + if string(b) != "old v2" { |
| + t.Fatalf("staged v2 = %q: nothing may be restaged from a manifest the box could not read", b) |
| + } |
| + if strings.Contains(out.String(), "restaging") || strings.Contains(out.String(), ErrVersionReused.Error()) { |
| + t.Fatalf("pass claimed a mismatch it could not establish:\n%s", out.String()) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'StaleCopy|UnknownReferences|OneRestage|MatchingStagedCopy|UnreadableManifestKeeps' -count=1` |
| +Expected: FAIL to compile, `undefined: WithPinSource`. |
| + |
| +- [ ] **Step 3: Add the option and the pass's check.** In `adopt.go`, add to `passConfig` after `proof *ProofGate`: |
| + |
| +```go |
| + // pins is WithPinSource's provider: staged copies that something outside |
| + // the registry and the live links still refers to (spec §8.5's |
| + // GoodVersion and generations; plan 5 supplies it). nil adds nothing. |
| + pins PinSource |
| + |
| + // stage is what ensureStagedChecked needs to judge a staged copy it is |
| + // about to reuse (spec §4.2). PassFilter sets it once per pass. |
| + stage stageCheck |
| +``` |
| + |
| +After the `WithProofGate` function, add: |
| + |
| +```go |
| +// WithPinSource adds src's references to the pin set a reused staged copy is |
| +// checked against (spec §4.2). Plan 5 passes it to every pass, root's and the |
| +// agent's, with GoodVersion and the activator's generations. |
| +func WithPinSource(src PinSource) PassOption { |
| + return func(c *passConfig) { c.pins = src } |
| +} |
| +``` |
| + |
| +In `PassFilter`, directly after `cfg.selected = selected`: |
| + |
| +```go |
| + // The reuse check's view of what refers to a staged copy (spec §4.2). It |
| + // is asked only when a staged copy turns out to differ from its canonical |
| + // release, so a healthy pass never pays for it, and it reads fresh then, |
| + // so a host this pass has already repointed counts. |
| + cfg.stage = stageCheck{ |
| + verified: map[string]bool{}, |
| + refs: func() (StagedPaths, error) { |
| + rows, err := reg.List() |
| + if err != nil { |
| + return nil, fmt.Errorf("list routes: %w", err) |
| + } |
| + pins, err := BoxPins(optDir, rows) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if cfg.pins != nil { |
| + more, err := cfg.pins.StagedPins(ctx, optDir) |
| + if err != nil { |
| + return nil, err |
| + } |
| + for p := range more { |
| + pins.add(p) |
| + } |
| + } |
| + return pins, nil |
| + }, |
| + } |
| +``` |
| + |
| +In `adoptHost`, change the call at line 1087 to: |
| + |
| +```go |
| + m, err := ensureStagedChecked(ctx, s, optDir, RouteAccount(rt), rt.App, ver, rt.Host, cfg.stage) |
| +``` |
| + |
| +- [ ] **Step 4: Replace `ensureStaged`.** Keep its whole doc comment, and add this paragraph at its end: |
| + |
| +```go |
| +// |
| +// A version directory that is already staged is no longer trusted by name |
| +// (zero-downtime deploys spec §4.2). A version name can be shipped again with |
| +// different bytes once retention has pruned it, and the staged path is keyed |
| +// by name alone. So a staged binary is hashed and compared with the canonical |
| +// manifest's digest for this platform: |
| +// - a match reuses it, as before; |
| +// - a mismatch restages over it only when nothing on the box refers to it |
| +// (chk.refs: live links, Version, PrevVersion, and whatever |
| +// WithPinSource adds); otherwise the version is refused with |
| +// ErrVersionReused, so a copy under another instance's build never |
| +// changes underneath it; |
| +// - a canonical manifest that cannot be read keeps today's answer: the copy |
| +// is used as staged, because a box cannot compare what it cannot read, |
| +// and the agent runs any attempt onto it in hold (CanonicalDemand). |
| +``` |
| + |
| +Then replace the function itself with these three functions: |
| + |
| +```go |
| +// stageCheck is what ensureStagedChecked needs to judge a staged copy it is |
| +// about to reuse (spec §4.2). |
| +type stageCheck struct { |
| + // refs reports the staged copies something on this box still refers to. |
| + // nil, or an error, means unknown, and unknown never permits a restage. |
| + refs func() (StagedPaths, error) |
| + // verified holds staged paths whose bytes already matched their canonical |
| + // release this pass, so N hosts moving onto one build hash it once. nil |
| + // remembers nothing. |
| + verified map[string]bool |
| +} |
| + |
| +// ensureStaged is ensureStagedChecked with the pin set this package can read |
| +// with no registry in hand: the live links. The tests that stage one version |
| +// directly call it. |
| +func ensureStaged(ctx context.Context, s Store, optDir, acct, app, ver, host string) (*Manifest, error) { |
| + return ensureStagedChecked(ctx, s, optDir, acct, app, ver, host, stageCheck{ |
| + refs: func() (StagedPaths, error) { return BoxPins(optDir, nil) }, |
| + }) |
| +} |
| + |
| +func ensureStagedChecked(ctx context.Context, s Store, optDir, acct, app, ver, host string, chk stageCheck) (*Manifest, error) { |
| + path := VersionPath(optDir, acct, ver, app) |
| + _, statErr := os.Stat(path) |
| + staged := statErr == nil |
| + m, raw, err := GetManifestRaw(ctx, s, acct, app, ver) |
| + if err != nil { |
| + if !staged { |
| + return nil, fmt.Errorf("manifest for %s/%s: %w", app, ver, err) |
| + } |
| + // The binary IS staged, which is this function's whole promise, and |
| + // the canonical manifest cannot be read to compare it with. A sibling |
| + // already on disk is the manifest in hand, exactly as before this |
| + // check existed. Without one, a fetch failure is only worth a line; |
| + // backfillManifest retries on every pass. |
| + sib, serr := ReadStagedManifest(optDir, acct, ver, app) |
| + if serr == nil && sib != nil { |
| + return sib, nil |
| + } |
| + if serr != nil { |
| + fmt.Printf(" %s: staged manifest for %s: %v\n", host, ver, serr) |
| + } |
| + fmt.Printf(" %s: back-fill manifest for %s: %v\n", host, ver, err) |
| + return nil, nil |
| + } |
| + art, hasBuild := m.Artifacts[adoptPlatform] |
| + if staged { |
| + if chk.verified[path] { |
| + repairSibling(optDir, acct, app, ver, host, raw) |
| + return m, nil |
| + } |
| + sum, herr := FileSHA256(path) |
| + if herr != nil { |
| + return nil, fmt.Errorf("staged %s/%s: %w", app, ver, herr) |
| + } |
| + if hasBuild && strings.EqualFold(sum, art.SHA256) { |
| + if chk.verified != nil { |
| + chk.verified[path] = true |
| + } |
| + repairSibling(optDir, acct, app, ver, host, raw) |
| + return m, nil |
| + } |
| + // Different bytes under the same name. Overwrite only a copy nothing |
| + // refers to: overwriting a referenced one would change the bytes under |
| + // another instance's build (spec §4.2). |
| + if chk.refs == nil { |
| + return nil, fmt.Errorf("%s/%s: the staged copy differs from the release and what refers to it is unknown, so it was not restaged", app, ver) |
| + } |
| + pins, perr := chk.refs() |
| + if perr != nil { |
| + return nil, fmt.Errorf("%s/%s: the staged copy differs from the release and what refers to it is unknown (%v), so it was not restaged", app, ver, perr) |
| + } |
| + if pins.Has(path) { |
| + return nil, fmt.Errorf("%s/%s: %w", app, ver, ErrVersionReused) |
| + } |
| + fmt.Printf(" %s: staged %s differs from the release in the store and nothing on this box refers to it; restaging\n", host, ver) |
| + } |
| + if !hasBuild { |
| + // A box facing a release with no build for its platform: skip, never |
| + // install the wrong arch. The next release ships it and the box |
| + // catches up. |
| + return nil, fmt.Errorf("release %s has no %s build — skipping", ver, adoptPlatform) |
| + } |
| + body, _, err := GetQualified(ctx, s, ArtifactKey(acct, app, ver, art.File)) |
| + if err != nil { |
| + return nil, fmt.Errorf("download %s: %w", ver, err) |
| + } |
| + if got := etagOf(body); got != art.SHA256 { |
| + return nil, fmt.Errorf("download %s: sha256 mismatch (refusing to stage)", ver) |
| + } |
| + if err := StageArtifact(optDir, acct, ver, app, body); err != nil { |
| + return nil, err |
| + } |
| + if chk.verified != nil { |
| + chk.verified[path] = true |
| + } |
| + // The sibling is the CANONICAL bytes (spec §4.2), and a restage replaces |
| + // whatever sibling the stale copy had. Best-effort: a failure here is not |
| + // fatal to staging the binary, and backfillManifest gets another chance |
| + // at it on every later pass. |
| + if err := StageManifest(optDir, acct, ver, app, raw); err != nil { |
| + fmt.Printf(" %s: stage manifest sibling for %s: %v\n", host, ver, err) |
| + } |
| + return m, nil |
| +} |
| + |
| +// repairSibling writes the canonical bytes beside a reused binary whose |
| +// sibling is absent or does not read: the back-fill for versions staged |
| +// before the sibling existed, and the repair for one a different binary |
| +// wrote. A sibling that reads is left alone, as before. Best-effort. |
| +func repairSibling(optDir, acct, app, ver, host string, raw []byte) { |
| + if sib, err := ReadStagedManifest(optDir, acct, ver, app); err == nil && sib != nil { |
| + return |
| + } |
| + if err := StageManifest(optDir, acct, ver, app, raw); err != nil { |
| + fmt.Printf(" %s: stage manifest sibling for %s: %v\n", host, ver, err) |
| + } |
| +} |
| +``` |
| + |
| +The string `release %s has no %s build — skipping` is the existing line, moved unchanged. Do not reword it here. |
| + |
| +- [ ] **Step 5: Run the package.** |
| + |
| +Run: `cd $W && go build ./... && go test ./internal/release/ -count=1` |
| +Expected: PASS, including `TestEnsureStagedBackfillFetchFailureIsNotFatal`, `TestArmedPassRefusesAVersionWhoseManifestItCouldNotObtain`, `TestPassRelaysBlockedLiveSwapForStagedRelease` and the D3 proof-gate tests. A failure in an existing test means this function changed behaviour on a path that is not a re-ship. Read it and report; do not edit the old test. |
| + |
| +- [ ] **Step 6: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/adopt.go internal/release/stage_reuse_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 6: root staging checks a reused staged copy's digest and never overwrites a referenced one |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: `SidecarAdoptable`, the sidecar leg of the pass |
| + |
| +**Files:** |
| +- Modify: `internal/release/adopt.go:348-350` (`adopts`), `:812-842` (the awake and hibernating loops), `:864-867` (after `Adoptable`), `:1497` (before `if rt.Hibernate {` at the end of `adoptHost`) |
| +- Modify: `internal/release/materialize.go:12-16` (doc only) |
| +- Create: `internal/release/sidecar_adopt_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func SidecarAdoptable(rt registry.Route) bool`: `rt.IsSidecar() && rt.App != "" && !rt.Platform && (rt.Status == "" || rt.Status == "ready")`. |
| +- `Adoptable` and `CapabilityRoute` are unchanged. Plan 5 relies on this leg: once root repoints a sidecar, the activator's drift check sees it, and plan 5 queues that as a `release` attempt. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/sidecar_adopt_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "reflect" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +func addSidecar(t *testing.T, reg *registry.Registry, host, app, channel string, hibernate bool) { |
| + t.Helper() |
| + if err := reg.Upsert(registry.Route{ |
| + Host: host, Kind: "sidecar", App: app, Channel: channel, Hibernate: hibernate, |
| + Args: `["sidecar","run"]`, Status: "ready", |
| + }); err != nil { |
| + t.Fatalf("upsert %s: %v", host, err) |
| + } |
| +} |
| + |
| +func TestSidecarAdoptableIsASecondPredicateNotAWiderOne(t *testing.T) { |
| + base := registry.Route{Host: "bot.example.com", Kind: "sidecar", App: "myapp", Status: "ready"} |
| + for _, c := range []struct { |
| + name string |
| + mut func(*registry.Route) |
| + sidecar, instance bool |
| + }{ |
| + {"ready sidecar", func(*registry.Route) {}, true, false}, |
| + {"status empty", func(r *registry.Route) { r.Status = "" }, true, false}, |
| + {"no app", func(r *registry.Route) { r.App = "" }, false, false}, |
| + {"platform", func(r *registry.Route) { r.Platform = true }, false, false}, |
| + {"creating", func(r *registry.Route) { r.Status = "creating" }, false, false}, |
| + {"an instance", func(r *registry.Route) { r.Kind = "instance" }, false, true}, |
| + {"static", func(r *registry.Route) { r.Kind = "static" }, false, false}, |
| + } { |
| + rt := base |
| + c.mut(&rt) |
| + if got := SidecarAdoptable(rt); got != c.sidecar { |
| + t.Errorf("%s: SidecarAdoptable = %v, want %v", c.name, got, c.sidecar) |
| + } |
| + if got := Adoptable(rt); got != c.instance { |
| + t.Errorf("%s: Adoptable = %v, want %v (Adoptable is not widened)", c.name, got, c.instance) |
| + } |
| + } |
| + unit := base |
| + unit.Backing = registry.BackingUnit |
| + if CapabilityRoute(unit) { |
| + t.Error("CapabilityRoute admitted a sidecar: no capability drop-ins are ever written for one") |
| + } |
| +} |
| + |
| +// Spec §7.9: sidecars go through the same selection, staging, repoint and |
| +// stamp, and the pass never restarts them; RestartHibernating changes nothing |
| +// for them. |
| +func TestPassAdoptsASidecarWithoutRestartingIt(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", adoptPlatform, []byte("bin")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, reg, "web.example.com", "myapp", "canary/x", false) |
| + addSidecar(t, reg, "bot.example.com", "myapp", "canary/x", false) |
| + addSidecar(t, reg, "nap.example.com", "myapp", "canary/x", true) |
| + var restarted []string |
| + restart := func(h string) error { restarted = append(restarted, h); return nil } |
| + if err := Pass(ctx, reg, s, pub, opt, restart, RestartHibernating(restart)); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + for _, h := range []string{"web.example.com", "bot.example.com", "nap.example.com"} { |
| + if v, _ := LiveVersion(opt, h); v != "v1" { |
| + t.Errorf("%s live = %q, want v1", h, v) |
| + } |
| + if rt, _ := reg.Get(h); rt == nil || rt.Version != "v1" { |
| + t.Errorf("%s row = %+v, want Version v1 (root's stamp)", h, rt) |
| + } |
| + } |
| + if !reflect.DeepEqual(restarted, []string{"web.example.com"}) { |
| + t.Fatalf("restarted = %v; the pass must never restart a sidecar", restarted) |
| + } |
| +} |
| + |
| +// Sidecars go after every instance, so a build an instance poisons this pass |
| +// never reaches a sidecar. |
| +func TestSidecarIsNotMovedOntoABuildAnInstancePoisoned(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", adoptPlatform, []byte("good")) |
| + shipRelease(t, s, "bab", "myapp", "v2", adoptPlatform, []byte("bad")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, reg, "web.example.com", "myapp", "canary/x", false) |
| + addSidecar(t, reg, "bot.example.com", "myapp", "canary/x", false) |
| + if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + restart := func(h string) error { |
| + if h == "web.example.com" { |
| + return errors.New("did not come up") |
| + } |
| + return nil |
| + } |
| + if err := Pass(ctx, reg, s, pub, opt, restart); err != nil { |
| + t.Fatal(err) |
| + } |
| + if v, _ := LiveVersion(opt, "bot.example.com"); v != "v1" { |
| + t.Fatalf("sidecar live = %q, want v1: a build that failed an instance this pass must not reach it", v) |
| + } |
| +} |
| + |
| +// Spec §7.9: sidecars take the same bake holds. |
| +func TestSidecarWaitsOutItsBakeWindow(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", adoptPlatform, []byte("bin")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "stable", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := SetProduction(ctx, s, priv, "bab", "myapp", true, "test"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addSidecar(t, reg, "bot.example.com", "myapp", "stable", false) |
| + if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if v, _ := LiveVersion(opt, "bot.example.com"); v != "" { |
| + t.Fatalf("sidecar live = %q, want unmoved inside stable's bake window", v) |
| + } |
| +} |
| + |
| +// Spec §7.9: cold materialization of a new sidecar uses the same leg. |
| +func TestMaterializeHostMaterializesANewSidecar(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", adoptPlatform, []byte("bin")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addSidecar(t, reg, "bot.example.com", "myapp", "canary/x", true) |
| + if err := MaterializeHost(ctx, reg, s, pub, opt, t.TempDir(), "bot.example.com"); err != nil { |
| + t.Fatalf("MaterializeHost: %v", err) |
| + } |
| + if _, err := os.Stat(LiveLink(opt, "bot.example.com")); err != nil { |
| + t.Fatalf("live link does not resolve: %v", err) |
| + } |
| +} |
| + |
| +// Unchanged behaviour: what happens to instances does not depend on whether |
| +// the box also carries a sidecar. |
| +func TestInstanceOutcomesDoNotDependOnTheSidecarLeg(t *testing.T) { |
| + type outcome struct { |
| + restarted []string |
| + live, row string |
| + } |
| + run := func(withSidecar bool) outcome { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + opt := t.TempDir() |
| + reg := testRegistry(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", adoptPlatform, []byte("bin")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + addRoute(t, reg, "web.example.com", "myapp", "canary/x", false) |
| + addRoute(t, reg, "nap.example.com", "myapp", "canary/x", true) |
| + if withSidecar { |
| + addSidecar(t, reg, "bot.example.com", "myapp", "canary/x", false) |
| + } |
| + var o outcome |
| + restart := func(h string) error { o.restarted = append(o.restarted, h); return nil } |
| + if err := Pass(ctx, reg, s, pub, opt, restart, RestartHibernating(restart)); err != nil { |
| + t.Fatal(err) |
| + } |
| + o.live, _ = LiveVersion(opt, "web.example.com") |
| + if rt, _ := reg.Get("nap.example.com"); rt != nil { |
| + o.row = rt.Version |
| + } |
| + return o |
| + } |
| + without, with := run(false), run(true) |
| + if !reflect.DeepEqual(without, with) { |
| + t.Fatalf("instance outcomes changed when a sidecar was present:\nwithout %+v\nwith %+v", without, with) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'SidecarAdoptable|Sidecar|MaterializeHostMaterializesANewSidecar|InstanceOutcomes' -count=1` |
| +Expected: FAIL to compile, `undefined: SidecarAdoptable`. |
| + |
| +- [ ] **Step 3: Add the predicate.** In `adopt.go`, directly after `Adoptable`: |
| + |
| +```go |
| +// SidecarAdoptable is the pass's SECOND admission predicate (zero-downtime |
| +// deploys spec §7.9): a native sidecar row (Kind "sidecar") of a known app, |
| +// not platform-minted, and ready. Its build follows the channel on its own |
| +// row, as sidecar_reconcile.go provisions it, through the same signed |
| +// selection, bake holds, staging, repoint and stamp an instance gets. |
| +// |
| +// Adoptable is deliberately NOT widened: capability drop-ins, the fast |
| +// sweep's membership, config sync, the restart passes and the console's |
| +// Restartable flag all keep their own answer for a sidecar. The pass opts in |
| +// by name, and only the pass. |
| +func SidecarAdoptable(rt registry.Route) bool { |
| + return rt.IsSidecar() && rt.App != "" && !rt.Platform && |
| + (rt.Status == "" || rt.Status == "ready") |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Admit sidecars in `PassFilter`, and run them last with no restart.** Change `adopts`: |
| + |
| +```go |
| + adopts := func(rt registry.Route) bool { |
| + return (Adoptable(rt) || SidecarAdoptable(rt)) && (extra == nil || extra(rt)) |
| + } |
| +``` |
| + |
| +In the loops at the end of `PassFilter`, change `var hibernating []registry.Route` to `var hibernating, sidecars []registry.Route`. Inside the first loop, directly after the `if !adopts(rt) { continue }` check, add: |
| + |
| +```go |
| + if rt.IsSidecar() { |
| + sidecars = append(sidecars, rt) |
| + continue |
| + } |
| +``` |
| + |
| +After the `for _, rt := range hibernating { ... }` loop and before `return nil`, add: |
| + |
| +```go |
| + // Native sidecars LAST, and never through a restart callback (spec |
| + // §7.9). The pass stages, repoints and stamps. The running worker moves |
| + // when the activator's drift check sees the link change, which plan 5 |
| + // queues as a paced release attempt; a Bounce here would stop an always-on |
| + // worker with nothing to start it. Last for the reason sleepers go after |
| + // awake hosts: a repoint has no health gate, so a build an instance |
| + // poisons this pass never reaches one. |
| + for _, rt := range sidecars { |
| + ptr := pointers[groupKey{RouteAccount(rt), rt.App, routeChannel(rt)}] |
| + adoptHost(ctx, selected[rt.Host], reg, s, optDir, bake, poisoned, rt, ptr, nil, now, cfg, routeHolds(rt), bakeIgnore(rt)) |
| + } |
| +``` |
| + |
| +- [ ] **Step 5: Say what happened to a sidecar.** In `adoptHost`, directly before the closing `if rt.Hibernate {` block that prints "repointed"/"updated": |
| + |
| +```go |
| + if rt.IsSidecar() { |
| + fmt.Printf(" %s: repointed %s -> %s (%s; sidecar, its worker moves onto it on the agent's drift check)\n", rt.Host, cur, ver, channel) |
| + return |
| + } |
| +``` |
| + |
| +- [ ] **Step 6: Update `MaterializeHost`'s doc.** In `materialize.go`, change the first sentence to: `// MaterializeHost stages host's channel head and points its live symlink — the cold-wake, on-demand form of the adopt pass (carlos-deploy spec §5), for an instance or a native sidecar (zero-downtime deploys spec §7.9).` Nothing else in that file changes. |
| + |
| +- [ ] **Step 7: Run the package.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -count=1` |
| +Expected: PASS, including `TestPassSkipsUnadoptableRoutes` and `TestPassSkipsPlatformMintedRoutes`. |
| + |
| +- [ ] **Step 8: Show `Adoptable`'s callers are unchanged.** |
| + |
| +```bash |
| +cd $W |
| +git grep -n -E '(^|[^A-Za-z])Adoptable\(' -- '*.go' ':!*_test.go' ':!internal/release/' |
| +git diff --quiet origin/main -- internal/appconfig internal/console internal/host && echo "callers unchanged" |
| +``` |
| + |
| +Expected, exactly these twelve lines and then `callers unchanged`: |
| + |
| +``` |
| +internal/appconfig/sync.go:430: return release.Adoptable(rt) |
| +internal/console/apirestartscope.go:167: if rt.App != app || release.RouteAccount(rt) != acct.Sqid || !release.Adoptable(rt) { |
| +internal/console/browserrestart.go:88: if rt.Host == host && rt.App == app && release.RouteAccount(rt) == acct.Sqid && release.Adoptable(rt) { |
| +internal/console/view.go:599: Restartable: release.Adoptable(rt), |
| +internal/host/agent.go:1693: if !release.Adoptable(rt) || seen[rt.Host] { |
| +internal/host/proofask.go:135: if !release.Adoptable(rt) || rt.Unit == "" || !pol.Proves(rt.App) { |
| +internal/host/proofroot.go:100: if !release.Adoptable(rt) || rt.Unit == "" { |
| +internal/host/proofroot.go:155: if !release.Adoptable(rt) || rt.Unit == "" { |
| +internal/host/restart.go:71: return release.Adoptable(rt) || rt.IsSidecar() |
| +internal/host/restart.go:216: if !release.Adoptable(rt) || !execBounceable(rt) { |
| +internal/host/restart.go:784: if !release.Adoptable(rt) || rt.App != k[1] || release.RouteAccount(rt) != k[0] { |
| +internal/host/restartscope.go:151: if !release.Adoptable(rt) || rt.App != app[1] || release.RouteAccount(rt) != app[0] { |
| +``` |
| + |
| +Inside `internal/release`, `Adoptable`'s body is unchanged, `CapabilityRoute` still calls it alone, and `PassFilter`'s `adopts` is the one place that ORs in `SidecarAdoptable`. If main has moved, the line numbers may differ but the set of call sites must not; a new call site is reported, not silently accepted. |
| + |
| +One consequence, recorded for review: `host.fastMembers` (agent.go:1693) keeps `Adoptable`, so a sidecar on a `canary/*` channel is not in the 2–3 s fast sweep. It moves on the serial-bump full pass and on the hourly passes. That is the spec's "callers untouched", not a bug in this task. |
| + |
| +- [ ] **Step 9: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release && go vet ./internal/release/ |
| +git add internal/release/adopt.go internal/release/materialize.go internal/release/sidecar_adopt_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 7: SidecarAdoptable, a second admission predicate; Adoptable not widened |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 8: The exec backend spawns workers by staged path |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go:1344-1407` (`StartWorker`) |
| +- Modify: `internal/activator/backend_unit.go:341-343` (`SplitBackend.StartWorker`, add a sibling) |
| +- Create: `internal/activator/worker_staged_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `release.StagedIdentity`, `release.ResolveLive`, `release.VerifyStaged`, `release.ErrStagedCopyChanged` (Task 3). |
| +- Produces: |
| + - `func (b *ExecBackend) StartWorkerStaged(ctx context.Context, rt registry.Route, extra map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error)`. With `onto == nil` it ensures the live link, resolves it once, and execs the resolved staged path. With `onto != nil` it verifies `onto` and execs `onto.StagedPath`. Either way it returns the identity it exec'd. |
| + - `func (b *SplitBackend) StartWorkerStaged(...)` with the same signature: it forwards to the picked backend when that backend has the method, and otherwise falls back to `StartWorker` with `StagedIdentity{Version: ver}`. |
| + - `ExecBackend.StartWorker` keeps its signature and returns `id.Version`. |
| + - Test doubles for Task 9: `type stagedFake struct{ *fakeBackend; ... }`, `func fakeIdentity(ver string) release.StagedIdentity`, and `func (b *stagedFake) calls() []*release.StagedIdentity`. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/activator/worker_staged_test.go`: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "sync" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// stagedFake is fakeBackend with spec §7.9's capability. It records which |
| +// build each worker spawn asked for (nil = "what the live link names") and |
| +// reports a synthetic identity, so a test can tell builds apart without real |
| +// files. |
| +type stagedFake struct { |
| + *fakeBackend |
| + smu sync.Mutex |
| + onto []*release.StagedIdentity |
| +} |
| + |
| +func fakeIdentity(ver string) release.StagedIdentity { |
| + return release.StagedIdentity{Version: ver, StagedPath: "/opt/carlos/versions/" + ver + "/app", Digest: "sha-of-" + ver} |
| +} |
| + |
| +func (b *stagedFake) StartWorkerStaged(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error) { |
| + b.smu.Lock() |
| + if onto != nil { |
| + c := *onto |
| + b.onto = append(b.onto, &c) |
| + } else { |
| + b.onto = append(b.onto, nil) |
| + } |
| + b.smu.Unlock() |
| + p, ver, err := b.fakeBackend.StartWorker(ctx, rt, env) |
| + if err != nil { |
| + return p, release.StagedIdentity{}, err |
| + } |
| + if onto != nil { |
| + return p, *onto, nil |
| + } |
| + return p, fakeIdentity(ver), nil |
| +} |
| + |
| +func (b *stagedFake) calls() []*release.StagedIdentity { |
| + b.smu.Lock() |
| + defer b.smu.Unlock() |
| + return append([]*release.StagedIdentity(nil), b.onto...) |
| +} |
| + |
| +// stageArgv0Worker stages a worker build at <OptDir>/versions/<ver>/app that |
| +// writes its own $0 to out and then sleeps, and points live/<host> at it. The |
| +// version is in the script, so each build hashes differently. |
| +func stageArgv0Worker(t *testing.T, b *ExecBackend, host, ver, out string) string { |
| + t.Helper() |
| + vdir := filepath.Join(b.OptDir, "versions", ver) |
| + if err := os.MkdirAll(vdir, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + bin := filepath.Join(vdir, "app") |
| + script := "#!/bin/sh\n# build " + ver + "\nprintf '%s\\n' \"$0\" > '" + out + ".tmp' && mv '" + out + ".tmp' '" + out + "'\nexec sleep 30\n" |
| + if err := os.WriteFile(bin, []byte(script), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(b.OptDir, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + os.Remove(b.liveLink(host)) |
| + if err := os.Symlink(bin, b.liveLink(host)); err != nil { |
| + t.Fatal(err) |
| + } |
| + return bin |
| +} |
| + |
| +func workerRoute(host string) registry.Route { |
| + return registry.Route{Host: host, Kind: "sidecar", Args: `["sidecar","run"]`} |
| +} |
| + |
| +// Spec §7.9: the serving identity is the build captured at exec, so the exec |
| +// must be of the file the identity names, never of live/<host>. |
| +func TestExecBackendWorkerExecsTheStagedPathItResolved(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "bot.x.internal" |
| + out := filepath.Join(t.TempDir(), "argv0") |
| + bin := stageArgv0Worker(t, b, host, "v1", out) |
| + p, id, err := b.StartWorkerStaged(context.Background(), workerRoute(host), nil, nil) |
| + if err != nil { |
| + t.Fatalf("StartWorkerStaged: %v", err) |
| + } |
| + defer p.Stop(context.Background()) |
| + if got := strings.TrimSpace(waitForFile(t, out, 5*time.Second)); got != bin { |
| + t.Fatalf("$0 = %q; want the staged path %q, never live/%s", got, bin, host) |
| + } |
| + sum, err := release.FileSHA256(bin) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if want := (release.StagedIdentity{Version: "v1", StagedPath: bin, Digest: sum}); id != want { |
| + t.Fatalf("identity = %+v, want %+v", id, want) |
| + } |
| +} |
| + |
| +// Spec §7.9: a recycle execs the captured build even after root has |
| +// repointed the link. |
| +func TestExecBackendWorkerRecycleExecsTheCapturedBuildAfterARepoint(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "bot.x.internal" |
| + out1 := filepath.Join(t.TempDir(), "argv0-v1") |
| + bin1 := stageArgv0Worker(t, b, host, "v1", out1) |
| + p1, id1, err := b.StartWorkerStaged(context.Background(), workerRoute(host), nil, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + waitForFile(t, out1, 5*time.Second) |
| + p1.Stop(context.Background()) |
| + os.Remove(out1) |
| + |
| + stageArgv0Worker(t, b, host, "v2", filepath.Join(t.TempDir(), "argv0-v2")) // root installs v2 |
| + |
| + p2, id2, err := b.StartWorkerStaged(context.Background(), workerRoute(host), nil, &id1) |
| + if err != nil { |
| + t.Fatalf("recycle onto v1: %v", err) |
| + } |
| + defer p2.Stop(context.Background()) |
| + if got := strings.TrimSpace(waitForFile(t, out1, 5*time.Second)); got != bin1 { |
| + t.Fatalf("recycle exec'd %q, want the captured %q", got, bin1) |
| + } |
| + if id2 != id1 { |
| + t.Fatalf("recycle reported %+v, want the captured %+v", id2, id1) |
| + } |
| +} |
| + |
| +// Spec §4.2: a recovery whose staged copy has changed is refused, and nothing |
| +// runs. |
| +func TestExecBackendWorkerRecycleRefusesAChangedStagedCopy(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "bot.x.internal" |
| + out := filepath.Join(t.TempDir(), "argv0") |
| + bin := stageArgv0Worker(t, b, host, "v1", out) |
| + p1, id1, err := b.StartWorkerStaged(context.Background(), workerRoute(host), nil, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + waitForFile(t, out, 5*time.Second) |
| + p1.Stop(context.Background()) |
| + os.Remove(out) |
| + if err := os.WriteFile(bin, []byte("#!/bin/sh\n# tampered, and longer than the original\nexec sleep 30\n"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + p2, _, err := b.StartWorkerStaged(context.Background(), workerRoute(host), nil, &id1) |
| + if !errors.Is(err, release.ErrStagedCopyChanged) { |
| + t.Fatalf("recycle onto changed bytes = %v, want ErrStagedCopyChanged", err) |
| + } |
| + if p2 != nil { |
| + p2.Stop(context.Background()) |
| + t.Fatal("a refused recycle still spawned a child") |
| + } |
| +} |
| + |
| +func TestSplitBackendForwardsStagedWorkerStarts(t *testing.T) { |
| + exec := &stagedFake{fakeBackend: &fakeBackend{workerSpawnVer: "v1"}} |
| + sb := &SplitBackend{Exec: exec, Unit: &fakeBackend{workerSpawnVer: "u1"}} |
| + rt := testWorkerRoute("bot.x.internal") |
| + onto := fakeIdentity("v0") |
| + if _, id, err := sb.StartWorkerStaged(context.Background(), rt, nil, &onto); err != nil || id != onto { |
| + t.Fatalf("exec route: identity %+v, %v; want %+v forwarded", id, err, onto) |
| + } |
| + if calls := exec.calls(); len(calls) != 1 || calls[0] == nil || *calls[0] != onto { |
| + t.Fatalf("exec backend saw %v, want one call onto v0", calls) |
| + } |
| + unit := rt |
| + unit.Backing = registry.BackingUnit |
| + if _, id, err := sb.StartWorkerStaged(context.Background(), unit, nil, &onto); err != nil || id != (release.StagedIdentity{Version: "u1"}) { |
| + t.Fatalf("a backend without the capability: %+v, %v; want today's StartWorker answer", id, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && TMPDIR=/tmp/k go test ./internal/activator/ -run 'StagedPath|CapturedBuild|ChangedStagedCopy|SplitBackendForwardsStaged' -count=1` |
| +Expected: FAIL to compile, `b.StartWorkerStaged undefined`. |
| + |
| +- [ ] **Step 3: Implement in `backend_exec.go`.** Replace `StartWorker` (doc comment included) with: |
| + |
| +```go |
| +// StartWorker is StartWorkerStaged onto whatever live/<host> names now. Its |
| +// string result is that build's version, as it always was. |
| +func (b *ExecBackend) StartWorker(ctx context.Context, rt registry.Route, extra map[string]string) (Proc, string, error) { |
| + p, id, err := b.StartWorkerStaged(ctx, rt, extra, nil) |
| + return p, id.Version, err |
| +} |
| + |
| +// StartWorkerStaged execs a worker with the route's declared argv (rt.Args, a |
| +// JSON array: a sidecar's `["sidecar","run"]` today) under the instance's own |
| +// env layers, CARLOS_*-clean like every other child this backend spawns, and |
| +// with NO health wait: a worker binds nothing, so there is nothing |
| +// waitHealthyCtx could dial. extra (the per-wake sidecar token) is applied |
| +// last and wins over the tenant's own <host>.env. |
| +// |
| +// What it execs is a staged build BY PATH, never live/<host> (zero-downtime |
| +// deploys spec §7.9): |
| +// - onto == nil (a wake, or a converge onto a new build): the live link is |
| +// ensured, read ONCE, and the file it names is hashed and exec'd; |
| +// - onto != nil (a crash or max_run recycle): onto.StagedPath is checked |
| +// against onto.Digest (release.VerifyStaged) and exec'd. A copy that has |
| +// changed is refused with release.ErrStagedCopyChanged and nothing runs. |
| +// |
| +// The identity returned is the build exec'd, the agent's serving identity for |
| +// this child. What the child later reports over the agent socket is display |
| +// metadata only. |
| +func (b *ExecBackend) StartWorkerStaged(ctx context.Context, rt registry.Route, extra map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error) { |
| + var argv []string |
| + if err := json.Unmarshal([]byte(rt.Args), &argv); err != nil || len(argv) == 0 { |
| + return nil, release.StagedIdentity{}, fmt.Errorf("worker %s: no runnable argv in args %q", rt.Host, rt.Args) |
| + } |
| + var id release.StagedIdentity |
| + if onto == nil { |
| + if err := b.ensureLiveLink(ctx, rt.Host); err != nil { |
| + return nil, release.StagedIdentity{}, err |
| + } |
| + got, err := release.ResolveLive(b.OptDir, rt.Host) |
| + if err != nil { |
| + return nil, release.StagedIdentity{}, fmt.Errorf("worker %s: %w", rt.Host, err) |
| + } |
| + id = got |
| + } else { |
| + if err := release.VerifyStaged(*onto); err != nil { |
| + return nil, release.StagedIdentity{}, fmt.Errorf("worker %s: not recycled onto %s: %w", rt.Host, onto.Version, err) |
| + } |
| + id = *onto |
| + } |
| + |
| + fileEnv, err := b.instanceEnv(rt) |
| + if err != nil { |
| + return nil, release.StagedIdentity{}, err |
| + } |
| + for k, v := range extra { |
| + fileEnv = withVar(fileEnv, k, v) |
| + } |
| + |
| + // Through spawnInCgroup, not Cgroups.Start directly: that seam is what |
| + // applies the route's resource tier, and a worker is a long-lived child |
| + // of a tenant's app like any other. Bypassing it compiles and silently |
| + // ships an uncapped process — the exact failure spawnInCgroup's own |
| + // comment names. |
| + cmd, err := b.spawnInCgroup(rt, func() *exec.Cmd { |
| + cmd := exec.Command(id.StagedPath, argv...) |
| + cmd.Env = childEnv(fileEnv) |
| + b.teeStd(cmd, rt) |
| + return cmd |
| + }) |
| + if err != nil { |
| + return nil, release.StagedIdentity{}, err |
| + } |
| + p := &childProc{cmd: cmd, done: make(chan struct{})} |
| + go func() { |
| + p.err = cmd.Wait() |
| + // Mirrors Start's exit record (#5 review, closed here): a worker's |
| + // crash is a sidecar's health word ("Crashed") the moment it lands, |
| + // so the member needs the SAME explanation an instance crash gets, |
| + // not silence. This fires once per actual exit, and the caller's |
| + // own doubling-backoff restart (2s up to a minute, activator.go) |
| + // already bounds how often an exit can happen at all — a |
| + // crash-looping worker cannot make this any noisier than the |
| + // restart log line it already earns internally. |
| + if p.err != nil { |
| + emitEvent(b.Logs, rt, "warn", "worker exited: "+RedactPaths(p.err.Error())) |
| + } else { |
| + emitEvent(b.Logs, rt, "info", "worker exited (exit 0)") |
| + } |
| + b.Cgroups.CleanupHost(rt.Host) |
| + close(p.done) |
| + }() |
| + return p, id, nil |
| +} |
| +``` |
| + |
| +The goroutine body and the two `emitEvent` strings are copied unchanged from today's `StartWorker`. Do not reword them. |
| + |
| +- [ ] **Step 4: Forward through `SplitBackend`.** In `backend_unit.go`, after `SplitBackend.StartWorker`: |
| + |
| +```go |
| +// StartWorkerStaged forwards spec §7.9's staged spawn to the backend the |
| +// route picks. One without the capability (UnitBackend, which refuses |
| +// workers anyway) gets today's StartWorker, and its identity is the version |
| +// alone, which no recycle can be verified against. |
| +func (b *SplitBackend) StartWorkerStaged(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error) { |
| + if s, ok := b.pick(rt).(stagedWorkerStarter); ok { |
| + return s.StartWorkerStaged(ctx, rt, env, onto) |
| + } |
| + p, ver, err := b.pick(rt).StartWorker(ctx, rt, env) |
| + return p, release.StagedIdentity{Version: ver}, err |
| +} |
| +``` |
| + |
| +`stagedWorkerStarter` is declared in Task 9's `workerbuild.go`. So that this task compiles on its own, create `internal/activator/workerbuild.go` now with only the interface: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "context" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// stagedWorkerStarter is the optional Backend capability behind spec §7.9: |
| +// spawn a worker from a NAMED staged build (onto != nil, a recycle) or from |
| +// whatever live/<host> names now (onto == nil, a wake or a converge), and say |
| +// exactly which bytes were exec'd. ExecBackend and SplitBackend have it. A |
| +// backend without it keeps today's StartWorker, and its workers have no |
| +// captured identity to recycle onto. |
| +type stagedWorkerStarter interface { |
| + StartWorkerStaged(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error) |
| +} |
| + |
| +var ( |
| + _ stagedWorkerStarter = (*ExecBackend)(nil) |
| + _ stagedWorkerStarter = (*SplitBackend)(nil) |
| +) |
| +``` |
| + |
| +- [ ] **Step 5: Run the package.** |
| + |
| +Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test ./internal/activator/ -count=1` |
| +Expected: PASS, including `TestExecBackendStartWorkerSpawnsWithoutAnyHealthProbe`, `TestExecBackendStartWorkerRefusesEmptyArgs`, `TestExecBackendStartWorkerRefusesMalformedArgs` and `TestExecBackendStartWorkerEnvWinsOverTenantFile`. |
| + |
| +- [ ] **Step 6: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/activator && go vet ./internal/activator/ |
| +git add internal/activator/backend_exec.go internal/activator/backend_unit.go internal/activator/workerbuild.go internal/activator/worker_staged_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 8: workers exec a staged build by path and report what they exec'd |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 9: The activator recycles the captured serving build |
| + |
| +**Files:** |
| +- Modify: `internal/activator/workerbuild.go` (add `startWorker`, `recycleWorkerLocked`, `WorkerReportedVersion`) |
| +- Modify: `internal/activator/activator.go`: imports (add `internal/release`); `:205-231` (`DueStore`); `:442` (`actEntry.workerVersion`, plus `workerServing`); `:617-678` (`Service`, add `CoalesceRecycle` after `OnWake`); `:994` (awake transition); `:1068-1077` (`wokenInstance`); `:1262-1281` (the worker branch of `doWake`); `:1801-1816` (`restartWorkerLocked`); `:1823-1826` (`convergeWorkerLocked`); `:1837-1983` (`respawnWorkerLocked`); `:1990-2038` (`NoteWorkerReady`); `:2567-2571` (`max_run`) |
| +- Modify: `internal/activator/activator_test.go`: delete `TestNoteWorkerReadyPersistsVersionToRegistry` (3329-3352), `TestNoteWorkerReadyEmptyVersionNeverStampsUnknown` (3354-3392), `TestFreshWorkerWakeClearsAPreviousIncarnationsVersion` (3394-3428), `TestAlwaysOnWorkerCrashRestartClearsVersionUntilTheNewChildReports` (3486-3523), `TestWorkerConvergeRespawnClearsVersionUntilTheNewChildReports` (4131-4178), each with its doc comment. They pin the behaviour spec §7.9 replaces. |
| +- Create: `internal/activator/worker_identity_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `stagedWorkerStarter`, `stagedFake`, `fakeIdentity` (Task 8); `release.StagedIdentity`. |
| +- Produces: |
| + - `Service.CoalesceRecycle func(host string) bool`: **plan 5's coalescing hook.** It is called under `a.mu` before a crash or `max_run` recycle. True means a queued, admissible `release` attempt will make the spawn, and the recycle is skipped. |
| + - `func (a *Service) WorkerReportedVersion(host string) string`: display metadata for plan 6. |
| + - unexported `actEntry.workerServing release.StagedIdentity`, `wokenInstance.spawnIdentity release.StagedIdentity`, `func (a *Service) startWorker(...) (Proc, release.StagedIdentity, error)`, `func (a *Service) recycleWorkerLocked(e *actEntry, stopOld Proc, why string)`. |
| + - `respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)`. |
| + - `DueStore` no longer has `SetVersions`. `*registry.Registry` still satisfies it. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/activator/worker_identity_test.go`: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "path/filepath" |
| + "sync/atomic" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// stagedActivator is shippingActivatorWithRegistry over stagedFake: ver is |
| +// what live/<host> names, and every spawn reports the identity it exec'd. |
| +func stagedActivator(t *testing.T) (*Service, *stagedFake, *fakeClock, *registry.Registry, *atomic.Value) { |
| + t.Helper() |
| + fb := &fakeBackend{} |
| + b := &stagedFake{fakeBackend: fb} |
| + clk := &fakeClock{t: time.Unix(1_700_000_000, 0)} |
| + reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatalf("open registry: %v", err) |
| + } |
| + a := New(Config{Bucket: "test-bucket", Idle: time.Hour}, b, reg) |
| + a.now = clk.now |
| + var ver atomic.Value |
| + ver.Store("v1") |
| + fb.execVerFn = func() string { return ver.Load().(string) } |
| + a.liveVer = func(string) (string, error) { return ver.Load().(string), nil } |
| + return a, b, clk, reg, &ver |
| +} |
| + |
| +func (a *Service) entryServing(host string) release.StagedIdentity { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + if e := a.entries[host]; e != nil { |
| + return e.workerServing |
| + } |
| + return release.StagedIdentity{} |
| +} |
| + |
| +// Spec §7.9: a crash recycle respawns the serving build, not the installed one. |
| +func TestWorkerCrashRecyclesTheCapturedServingBuild(t *testing.T) { |
| + a, b, _, _, ver := stagedActivator(t) |
| + host := "bot.x.internal" |
| + if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + if got := a.entryServing(host); got != fakeIdentity("v1") { |
| + t.Fatalf("serving after wake = %+v, want the captured v1", got) |
| + } |
| + ver.Store("v2") // root installs v2 under the running child |
| + b.lastWorker().kill() |
| + a.sweepOnce() |
| + waitForRestart(t, a, b.fakeBackend, host, 2) |
| + calls := b.calls() |
| + if len(calls) != 2 || calls[0] != nil { |
| + t.Fatalf("spawns = %v; want a wake from the live link, then a recycle", calls) |
| + } |
| + if calls[1] == nil || *calls[1] != fakeIdentity("v1") { |
| + t.Fatalf("crash recycle asked for %v; want the captured v1 build", calls[1]) |
| + } |
| + if got := a.entrySpawnVersion(host); got != "v1" { |
| + t.Fatalf("spawn version = %q, want v1", got) |
| + } |
| +} |
| + |
| +// Spec §7.9: so does an always-on worker's max_run recycle. |
| +func TestWorkerMaxRunRecyclesTheCapturedServingBuild(t *testing.T) { |
| + a, b, clk, _, ver := stagedActivator(t) |
| + host := "bot.x.internal" |
| + rt := alwaysOnWorkerRoute(host) |
| + rt.MaxRunSeconds = 30 |
| + if err := a.WakeWorker(rt); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + ver.Store("v2") |
| + clk.advance(31 * time.Second) |
| + a.sweepOnce() |
| + waitForRestart(t, a, b.fakeBackend, host, 2) |
| + if calls := b.calls(); len(calls) != 2 || calls[1] == nil || *calls[1] != fakeIdentity("v1") { |
| + t.Fatalf("max_run recycle asked for %v; want the captured v1 build", calls) |
| + } |
| +} |
| + |
| +// Until plan 5 queues it, the drift check is the one path that moves a |
| +// worker to a different installed build, and it reads the live link. |
| +func TestWorkerConvergeStillMovesOntoTheInstalledBuild(t *testing.T) { |
| + a, b, clk, _, ver := stagedActivator(t) |
| + host := "bot.x.internal" |
| + if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + ver.Store("v2") |
| + a.sweepOnce() |
| + clk.advance(ConvergeGrace + time.Minute) |
| + a.sweepOnce() |
| + waitFor(t, "the converge onto v2", func() bool { return a.entrySpawnVersion(host) == "v2" && !a.entryRespawning(host) }) |
| + if calls := b.calls(); len(calls) != 2 || calls[1] != nil { |
| + t.Fatalf("converge asked for %v; want the live link (nil)", calls) |
| + } |
| + if got := a.entryServing(host); got != fakeIdentity("v2") { |
| + t.Fatalf("serving after converge = %+v, want v2", got) |
| + } |
| +} |
| + |
| +// Plan 5's hook: an admissible queued attempt takes the spawn. |
| +func TestWorkerRecycleDefersToCoalesceRecycle(t *testing.T) { |
| + a, b, _, _, _ := stagedActivator(t) |
| + host := "bot.x.internal" |
| + var asked []string |
| + a.CoalesceRecycle = func(h string) bool { asked = append(asked, h); return true } |
| + if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + b.lastWorker().kill() |
| + a.sweepOnce() |
| + if a.entryRespawning(host) || b.workerStarts() != 1 { |
| + t.Fatalf("workerStarts = %d; the hook said a queued attempt makes the spawn", b.workerStarts()) |
| + } |
| + if len(asked) != 1 || asked[0] != host { |
| + t.Fatalf("hook asked about %v, want [%s]", asked, host) |
| + } |
| +} |
| + |
| +// Review Focus 4, and spec §7.9: the report is display metadata; registry |
| +// Version is root's install stamp. |
| +func TestNoteWorkerReadyIsDisplayOnly(t *testing.T) { |
| + a, b, _, reg := testActivator(t, time.Hour) |
| + host := "bot.x.internal" |
| + if err := reg.Upsert(testWorkerRoute(host)); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetVersions(host, "v1", ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := a.WakeWorker(testWorkerRoute(host)); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + noteReadyAsChild(t, a, b, host, "1.4.2-app-chosen") |
| + if rt, _ := reg.Get(host); rt == nil || rt.Version != "v1" { |
| + t.Fatalf("row = %+v; a worker's report must not overwrite root's stamp", rt) |
| + } |
| + if got := a.WorkerReportedVersion(host); got != "1.4.2-app-chosen" { |
| + t.Fatalf("WorkerReportedVersion = %q", got) |
| + } |
| + a.NoteWorkerReady(host, "") |
| + if rt, _ := reg.Get(host); rt == nil || rt.Version != "v1" { |
| + t.Fatalf("row = %+v after an empty report", rt) |
| + } |
| + if got := a.WorkerReportedVersion("nobody.x.internal"); got != "" { |
| + t.Fatalf("WorkerReportedVersion for no worker = %q", got) |
| + } |
| +} |
| + |
| +// Spec §7.9: no spawn path writes registry Version: not a wake, a crash |
| +// recycle or a converge. |
| +func TestWorkerSpawnsLeaveTheInstalledStampAlone(t *testing.T) { |
| + a, b, clk, reg, ver := shippingActivatorWithRegistry(t, time.Hour) |
| + host := "bot.x.internal" |
| + rt := alwaysOnWorkerRoute(host) |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetVersions(host, "v1", "v0"); err != nil { |
| + t.Fatal(err) |
| + } |
| + stamp := func(when string) { |
| + t.Helper() |
| + if got, _ := reg.Get(host); got == nil || got.Version != "v1" || got.PrevVersion != "v0" { |
| + t.Fatalf("%s: row = %+v; want root's stamp v1 (prev v0) untouched", when, got) |
| + } |
| + } |
| + if err := a.WakeWorker(rt); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + stamp("after a wake") |
| + noteReadyAsChild(t, a, b, host, "whatever-the-app-says") |
| + stamp("after a ready report") |
| + b.lastWorker().kill() |
| + a.sweepOnce() |
| + waitForRestart(t, a, b, host, 2) |
| + stamp("after a crash recycle") |
| + ver.Store("v2") |
| + a.sweepOnce() |
| + clk.advance(ConvergeGrace + time.Minute) |
| + a.sweepOnce() |
| + waitFor(t, "the converge onto v2", func() bool { return a.entrySpawnVersion(host) == "v2" && !a.entryRespawning(host) }) |
| + stamp("after a converge") |
| +} |
| + |
| +// Review Focus 4: an app-chosen report changes nothing about what recycles, |
| +// and a new child's report starts out unknown. |
| +func TestWorkerReportingAnAppChosenVersionStillRecyclesItsCapturedBuild(t *testing.T) { |
| + a, b, _, _, _ := stagedActivator(t) |
| + host := "bot.x.internal" |
| + if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil { |
| + t.Fatalf("WakeWorker: %v", err) |
| + } |
| + noteReadyAsChild(t, a, b.fakeBackend, host, "latest") |
| + b.lastWorker().kill() |
| + a.sweepOnce() |
| + waitForRestart(t, a, b.fakeBackend, host, 2) |
| + if calls := b.calls(); len(calls) != 2 || calls[1] == nil || *calls[1] != fakeIdentity("v1") { |
| + t.Fatalf("recycle asked for %v; want the captured v1, whatever the child said", calls) |
| + } |
| + if got := a.WorkerReportedVersion(host); got != "" { |
| + t.Fatalf("WorkerReportedVersion = %q after a recycle; the new child has not reported", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Delete the five obsolete tests** named under Files, with their doc comments. Keep the helpers they use (`noteReadyAsChild`, `waitForRestart`, `shippingActivatorWithRegistry`); other tests still call them. |
| + |
| +- [ ] **Step 3: Run the new tests and see them fail.** |
| + |
| +Run: `cd $W && TMPDIR=/tmp/k go test ./internal/activator/ -run 'Captured|InstalledBuild|CoalesceRecycle|DisplayOnly|InstalledStamp|AppChosen' -count=1` |
| +Expected: FAIL to compile, `e.workerServing undefined` and `a.CoalesceRecycle undefined`. |
| + |
| +- [ ] **Step 4: Add the helpers to `workerbuild.go`.** Add `"log"` to its imports and append: |
| + |
| +```go |
| +// startWorker spawns through the staged capability when the backend has it, |
| +// and through StartWorker otherwise. |
| +func (a *Service) startWorker(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error) { |
| + if s, ok := a.backend.(stagedWorkerStarter); ok { |
| + return s.StartWorkerStaged(ctx, rt, env, onto) |
| + } |
| + p, ver, err := a.backend.StartWorker(ctx, rt, env) |
| + return p, release.StagedIdentity{Version: ver}, err |
| +} |
| + |
| +// recycleWorkerLocked replaces a worker's child for a reason that is not a |
| +// build change: a crash, or an always-on worker's max_run. Automatic |
| +// recycling never changes a worker's build (spec §7.9): the child is |
| +// respawned from its SERVING build, the staged path and digest captured at |
| +// its exec, whatever live/<host> names by now. The caller holds a.mu. |
| +// |
| +// CoalesceRecycle is asked first. When it says a queued release attempt is |
| +// admissible now, that attempt makes the one spawn and the recycle is |
| +// skipped. A worker spawned by a backend without the staged capability has |
| +// no captured build, and keeps today's respawn from the live link. |
| +func (a *Service) recycleWorkerLocked(e *actEntry, stopOld Proc, why string) { |
| + if hook := a.CoalesceRecycle; hook != nil && hook(e.host) { |
| + log.Printf("activator: worker %s %s; a queued release attempt makes the spawn instead of a recycle", e.host, why) |
| + return |
| + } |
| + var onto *release.StagedIdentity |
| + if e.workerServing.StagedPath != "" { |
| + id := e.workerServing |
| + onto = &id |
| + } else { |
| + log.Printf("activator: worker %s %s; no captured build to recycle onto, so it respawns from live/%s", e.host, why, e.host) |
| + } |
| + a.respawnWorkerLocked(e, stopOld, why, onto) |
| +} |
| + |
| +// WorkerReportedVersion is what host's worker last said it runs, over the |
| +// agent socket: display metadata only (spec §7.9). It may be empty or an |
| +// app-chosen string, and it never identifies the executable. "" when no |
| +// worker entry exists. |
| +func (a *Service) WorkerReportedVersion(host string) string { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + if e := a.entries[host]; e != nil && e.worker { |
| + return e.workerVersion |
| + } |
| + return "" |
| +} |
| +``` |
| + |
| +- [ ] **Step 5: Edit `activator.go`.** |
| + |
| +1. Add `"github.com/carlosframework/platform/internal/release"` to the imports. |
| + |
| +2. `DueStore`: delete the paragraph beginning `// SetVersions rides along here rather than on a separate interface (Task 8):` (lines 210-215) and the method line `SetVersions(host, ver, prev string) error` (227). The activator no longer writes a route's version: root stamps it (spec §7.9). |
| + |
| +3. `actEntry`: replace the `workerVersion` line (442) with: |
| + |
| +```go |
| + // workerVersion is what the child last reported over the agent socket: |
| + // DISPLAY metadata only (spec §7.9). It can be empty or an app-chosen |
| + // string, so it never identifies the executable and is never written to |
| + // the registry, whose Version is root's install stamp. |
| + workerVersion string |
| + // workerServing is the build this worker's current child was exec'd |
| + // from, captured by the backend at the exec (spec §7.9): staged path and |
| + // sha256. Crash and max_run recycles respawn exactly this, never what |
| + // live/<host> names by then. Zero when the backend could not report it. |
| + // Written under a.mu at the awake transition and on every installed |
| + // respawn. |
| + workerServing release.StagedIdentity |
| +``` |
| + |
| +4. `Service`: after the `OnWake` field, add: |
| + |
| +```go |
| + // CoalesceRecycle, when set, is asked before a worker is recycled for a |
| + // crash or max_run (spec §7.9). True means a queued release attempt for |
| + // that host is admissible now and will make the one spawn itself, so the |
| + // recycle is skipped. The sweep asks again next time if the child is still |
| + // due. Nil, the default and the whole of plan 4, recycles every time. It |
| + // is called with a.mu held: it must answer from its own state, never |
| + // block, and never call back into Service. Set before Start, like OnWake. |
| + // Plan 5's scheduler fills it. |
| + CoalesceRecycle func(host string) bool |
| +``` |
| + |
| +5. `wokenInstance`: after `spawnVersion`, add `spawnIdentity release.StagedIdentity // the build exec'd, as the backend captured it (workers; spec §7.9)`. |
| + |
| +6. The awake transition (line 994): after `e.spawnVersion = w.spawnVersion`, add `e.workerServing = w.spawnIdentity`. |
| + |
| +7. `doWake`'s worker branch: replace everything from the comment `// Clear any version this row carries from a previous incarnation` through `proc, spawnVer, err = a.backend.StartWorker(ctx, rt, workerEnv(a.agentSocket(), host, token))` with: |
| + |
| +```go |
| + // No registry write (spec §7.9): the row's Version is root's install |
| + // stamp, and the build this child runs is the identity the backend |
| + // captures at the exec, not anything the child later says. |
| + var ident release.StagedIdentity |
| + proc, ident, err = a.startWorker(ctx, rt, workerEnv(a.agentSocket(), host, token), nil) |
| + spawnVer, w.spawnIdentity = ident.Version, ident |
| +``` |
| + |
| +8. `restartWorkerLocked`: change its last line to `a.recycleWorkerLocked(e, nil, "exited")`. |
| + |
| +9. `convergeWorkerLocked`: change its body to `a.respawnWorkerLocked(e, e.proc, "converging onto a new build", nil)`, and add to its doc comment: `nil: a converge execs what live/<host> names now. Until plan 5 queues it as a paced release attempt, this is the one path that moves a worker to a different installed build.` |
| + |
| +10. `respawnWorkerLocked`: change the signature to `func (a *Service) respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)`. Add this paragraph to its doc comment, after the first paragraph: |
| + |
| +```go |
| +// onto is the build to exec: the captured serving build for a recycle |
| +// (recycleWorkerLocked), or nil for a converge, which execs what live/<host> |
| +// names now. A recycle onto a staged copy whose bytes have changed is refused |
| +// by the backend (release.ErrStagedCopyChanged) and lands in the failed-spawn |
| +// arm below, which leaves the crash restart to retry with its backoff. |
| +``` |
| + |
| +In the goroutine's `var (...)` block, replace `spawnVer string` with `ident release.StagedIdentity`. In the `if wanted {` block, delete the comment beginning `// Clear the persisted running version BEFORE spawning` and the `a.due.SetVersions(host, "", rt.Version)` call with its `log.Printf`, and replace the spawn line with: |
| + |
| +```go |
| + // No registry write (spec §7.9): see doWake. |
| + if tok, err = newWorkerToken(); err == nil { |
| + proc, ident, err = a.startWorker(ctx, rt, workerEnv(a.agentSocket(), host, tok), onto) |
| + } |
| +``` |
| + |
| +In the `default:` arm, replace `e.spawnVersion = spawnVer` with: |
| + |
| +```go |
| + e.spawnVersion = ident.Version |
| + e.workerServing = ident |
| + e.workerVersion = "" // the new child has not reported yet |
| +``` |
| + |
| +and change the log line to `log.Printf("activator: worker %s respawned (%s), now running %s", host, why, ident.Version)`. |
| + |
| +11. The `max_run` arm (line 2570): change `a.respawnWorkerLocked(e, e.proc, "hit its max_run")` to `a.recycleWorkerLocked(e, e.proc, "hit its max_run")`. |
| + |
| +12. `NoteWorkerReady`: replace the function and its doc comment with: |
| + |
| +```go |
| +// NoteWorkerReady records a worker's own "I am up": last-activity, the end |
| +// of its settle window, and the build it SAYS it runs. That claim is display |
| +// metadata only (zero-downtime deploys spec §7.9): it can be empty or an |
| +// app-chosen string, so it never identifies the executable, never overwrites |
| +// registry Version (root's install stamp) and never decides a recycle. The |
| +// serving build is workerServing, captured at the exec. |
| +// |
| +// It acts on a WAKING entry as well as an awake one, which is the whole point |
| +// of the settle window: a child that boots fast says so and its wake resolves |
| +// immediately instead of waiting out sidecarSettle. |
| +func (a *Service) NoteWorkerReady(host, version string) { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + e := a.entries[host] |
| + if e == nil || !e.worker { |
| + return |
| + } |
| + e.lastActive = a.now() |
| + e.workerVersion = version |
| + if !e.readyNoted { |
| + e.readyNoted = true |
| + close(e.readyc) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 6: Confirm no activator code writes a route's version.** |
| + |
| +Run: `cd $W && grep -n "SetVersions" internal/activator/*.go | grep -v _test.go` |
| +Expected: no output. |
| + |
| +- [ ] **Step 7: Run the package and its neighbours.** |
| + |
| +Run: `cd $W && go build ./... && TMPDIR=/tmp/k go test ./internal/activator/ ./internal/agentsock/ -count=1` |
| +Expected: PASS, including `TestAlwaysOnWorkerCrashRestartsWithGrowingBackoff`, `TestAlwaysOnWorkerMaxRunRecyclesItsChild`, `TestAlwaysOnWorkerRespawnsOntoANewBuild`, `TestShutdownStopsARespawnFromOutlivingTheLease`, `TestRespawnShedsARacedChildBeforeTheLeaseIsReleased`, and the three config-gate respawn tests. These run on the plain `fakeBackend`, so they also prove that a backend without the capability keeps today's spawns. |
| + |
| +- [ ] **Step 8: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/activator && go vet ./internal/activator/ |
| +git add internal/activator/ |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 9: crash and max_run recycle the captured serving build; a worker's report is display-only |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 10: The signed retry request |
| + |
| +**Files:** |
| +- Create: `internal/release/retry.go` |
| +- Create: `internal/release/retry_test.go` |
| +- Create: `internal/provision/retrykey_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `signedWrapper`, `verifyGrant`, `ParsePublicKey`, `Signer`, `RootSigner`, `Channel`, `ValidChannel`, `ValidVersion`, `ValidAccount`, `ErrCASConflict` (existing); `scopedFixture`, `trapStore`, `testKeys`, `shipRelease` (existing test helpers). |
| +- Produces (plan 5 reads, plan 6 writes): |
| + - Bucket key `apps/<acct>/<app>/retries/<version>.json`: `func RetryKey(acct, app, version string) string`. |
| + - `type RetryRequest struct { Schema int; Kind string; Account, App, Version, Channel, ID string; Seq uint64; RequestedAt time.Time; Actor string }`. |
| + - `func (sg Signer) RequestRetry(ctx context.Context, s Store, acct, app, channel, version, actor string) (*RetryRequest, error)` and the root-key form `func RequestRetry(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, channel, version, actor string) (*RetryRequest, error)`. |
| + - `func GetRetry(ctx context.Context, s Store, pub ed25519.PublicKey, acct, app, version string) (*RetryRequest, error)`: absent gives `(nil, nil)`; anything that fails verification is an error, and a box treats it as no request. |
| + - `func (r *RetryRequest) Clears(mark uint64) bool`: true when `r.Seq > mark`. Plan 5's failure record stores the `Seq` it last saw as `mark`. |
| + - `var ErrRetryNotServing error`. |
| + |
| +- [ ] **Step 1: Write the failing tests.** Create `internal/release/retry_test.go`: |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "context" |
| + "crypto/ed25519" |
| + "errors" |
| + "strings" |
| + "testing" |
| + "time" |
| +) |
| + |
| +type retryEnv struct { |
| + s Store |
| + priv ed25519.PrivateKey |
| + pub ed25519.PublicKey |
| +} |
| + |
| +func newRetryEnv(t *testing.T) retryEnv { |
| + t.Helper() |
| + e := retryEnv{s: NewDirStore(t.TempDir())} |
| + e.priv, e.pub = testKeys(t) |
| + for _, v := range []string{"v1", "v2"} { |
| + shipRelease(t, e.s, "bab", "km", v, adoptPlatform, []byte(v)) |
| + } |
| + if err := Promote(context.Background(), e.s, e.priv, "bab", "km", "canary/x", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + return e |
| +} |
| + |
| +func (e retryEnv) request(t *testing.T) *RetryRequest { |
| + t.Helper() |
| + r, err := RequestRetry(context.Background(), e.s, e.priv, "bab", "km", "canary/x", "v1", "paul@example.test") |
| + if err != nil { |
| + t.Fatalf("RequestRetry: %v", err) |
| + } |
| + return r |
| +} |
| + |
| +func TestRetryRoundTripsAndGrows(t *testing.T) { |
| + e := newRetryEnv(t) |
| + ctx := context.Background() |
| + if got, err := GetRetry(ctx, e.s, e.pub, "bab", "km", "v1"); err != nil || got != nil { |
| + t.Fatalf("GetRetry before any request = %+v, %v; want nil, nil", got, err) |
| + } |
| + before := uint64(time.Now().UnixMilli()) |
| + first := e.request(t) |
| + got, err := GetRetry(ctx, e.s, e.pub, "bab", "km", "v1") |
| + if err != nil || got == nil || got.ID != first.ID || got.Seq != first.Seq || got.Channel != "canary/x" || got.Actor != "paul@example.test" { |
| + t.Fatalf("GetRetry = %+v, %v; want %+v", got, err, first) |
| + } |
| + if first.Seq < before || len(first.ID) != 32 { |
| + t.Fatalf("first request seq %d (want >= %d), id %q", first.Seq, before, first.ID) |
| + } |
| + second := e.request(t) |
| + if second.Seq <= first.Seq || second.ID == first.ID { |
| + t.Fatalf("second request %+v does not supersede %+v", second, first) |
| + } |
| + if !second.Clears(first.Seq) || second.Clears(second.Seq) { |
| + t.Fatal("Clears must be true only for a seq older than the request") |
| + } |
| + var none *RetryRequest |
| + if none.Clears(0) { |
| + t.Fatal("no request clears nothing") |
| + } |
| +} |
| + |
| +func TestRetryRefusesABuildTheChannelDoesNotServe(t *testing.T) { |
| + e := newRetryEnv(t) |
| + _, err := RequestRetry(context.Background(), e.s, e.priv, "bab", "km", "canary/x", "v2", "paul@example.test") |
| + if !errors.Is(err, ErrRetryNotServing) { |
| + t.Fatalf("RequestRetry for a build the channel does not serve = %v, want ErrRetryNotServing", err) |
| + } |
| +} |
| + |
| +// Spec §11: a box cannot forge a retry, because it has no release key. |
| +func TestARetrySignedByAnotherKeyIsRefused(t *testing.T) { |
| + e := newRetryEnv(t) |
| + other, _ := testKeys(t) |
| + w, err := signRetry(RootSigner(other), &RetryRequest{Schema: 1, Kind: retryKind, Account: "bab", App: "km", |
| + Version: "v1", Channel: "canary/x", ID: strings.Repeat("ab", 16), Seq: 1 << 62, RequestedAt: time.Now().UTC().Truncate(time.Second)}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := e.s.Put(context.Background(), RetryKey("bab", "km", "v1"), w, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := GetRetry(context.Background(), e.s, e.pub, "bab", "km", "v1"); err == nil { |
| + t.Fatal("a retry signed by a key the fleet does not trust verified") |
| + } |
| +} |
| + |
| +// Review Focus 5. |
| +func TestAChannelPointerIsNotARetryRequest(t *testing.T) { |
| + e := newRetryEnv(t) |
| + ctx := context.Background() |
| + ptr, _, err := e.s.Get(ctx, ChannelKey("bab", "km", "canary/x")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := e.s.Put(ctx, RetryKey("bab", "km", "v1"), ptr, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := GetRetry(ctx, e.s, e.pub, "bab", "km", "v1"); err == nil { |
| + t.Fatal("a signed channel pointer verified as a retry request") |
| + } |
| +} |
| + |
| +// Review Focus 5. |
| +func TestARetryRequestIsNotAChannelPointer(t *testing.T) { |
| + e := newRetryEnv(t) |
| + ctx := context.Background() |
| + e.request(t) |
| + retry, _, err := e.s.Get(ctx, RetryKey("bab", "km", "v1")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + _, etag, err := e.s.Get(ctx, ChannelKey("bab", "km", "canary/x")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := e.s.Put(ctx, ChannelKey("bab", "km", "canary/x"), retry, etag); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := Channel(ctx, e.s, e.pub, "bab", "km", "canary/x"); err == nil { |
| + t.Fatal("a signed retry request verified as a channel pointer") |
| + } |
| +} |
| + |
| +// Review Focus 5. |
| +func TestARetryCopiedToAnotherBuildIsRefused(t *testing.T) { |
| + e := newRetryEnv(t) |
| + ctx := context.Background() |
| + e.request(t) |
| + b, _, err := e.s.Get(ctx, RetryKey("bab", "km", "v1")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := e.s.Put(ctx, RetryKey("bab", "km", "v2"), b, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := GetRetry(ctx, e.s, e.pub, "bab", "km", "v2"); err == nil { |
| + t.Fatal("v1's retry verified at v2's key") |
| + } |
| +} |
| + |
| +// An unverifiable object at the key is overwritten, and the new seq is not |
| +// derived from it. |
| +func TestARetryReplacesAnUnverifiableObject(t *testing.T) { |
| + e := newRetryEnv(t) |
| + ctx := context.Background() |
| + if _, err := e.s.Put(ctx, RetryKey("bab", "km", "v1"), []byte("not json"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + before := uint64(time.Now().UnixMilli()) |
| + r := e.request(t) |
| + if r.Seq < before { |
| + t.Fatalf("seq %d is below the clock %d", r.Seq, before) |
| + } |
| + if got, err := GetRetry(ctx, e.s, e.pub, "bab", "km", "v1"); err != nil || got.ID != r.ID { |
| + t.Fatalf("GetRetry = %+v, %v", got, err) |
| + } |
| +} |
| + |
| +// Signed like a channel pointer: a scoped key reaches only its rungs, and is |
| +// refused before the store is touched. |
| +func TestARetryFromAScopedKeyIsBoundToItsRungs(t *testing.T) { |
| + root, rootPub, sg := scopedFixture(t, []string{RungEdge}, 90*24*time.Hour) |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + shipRelease(t, s, "bab", "km", "v1", adoptPlatform, []byte("b")) |
| + if err := Promote(ctx, s, root, "bab", "km", "edge", "v1"); err != nil { |
| + t.Fatal(err) |
| + } |
| + r, err := sg.RequestRetry(ctx, s, "bab", "km", "edge", "v1", "ci") |
| + if err != nil { |
| + t.Fatalf("scoped RequestRetry on its own rung: %v", err) |
| + } |
| + if got, err := GetRetry(ctx, s, rootPub, "bab", "km", "v1"); err != nil || got.ID != r.ID { |
| + t.Fatalf("a scoped retry did not verify against the root key: %+v, %v", got, err) |
| + } |
| + if _, err := sg.RequestRetry(ctx, trapStore{t}, "bab", "km", "beta", "v1", "ci"); err == nil { |
| + t.Fatal("an edge-only key requested a retry on beta") |
| + } |
| +} |
| + |
| +func TestRetryKeyLivesBesideTheReleasePlaneNotInsideIt(t *testing.T) { |
| + k := RetryKey("bab", "km", "v1") |
| + if k != "apps/bab/km/retries/v1.json" { |
| + t.Fatalf("RetryKey = %q", k) |
| + } |
| + if strings.HasPrefix(k, ChannelsPrefix("bab", "km")) || strings.HasPrefix(k, ReleasesPrefix("bab", "km")) { |
| + t.Fatalf("%s sits under a prefix the console and CLI parse as pointers or releases", k) |
| + } |
| +} |
| +``` |
| + |
| +Create `internal/provision/retrykey_test.go`. When a storage shape changes, the consumers to check are everything that PARSES keys under `apps/` (AGENTS.md): |
| + |
| +```go |
| +package provision |
| + |
| +import ( |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// A retry request lives under the app prefix. Neither record parser may read |
| +// it as an instance or a sidecar record. |
| +func TestRetryKeyIsNotARecord(t *testing.T) { |
| + known := map[string]bool{"bab": true} |
| + k := release.RetryKey("bab", "km", "v1") |
| + if _, _, _, ok := ParseRecordKey(k, known); ok { |
| + t.Fatalf("ParseRecordKey accepted %s", k) |
| + } |
| + if _, _, _, ok := ParseSidecarKey(k, known); ok { |
| + t.Fatalf("ParseSidecarKey accepted %s", k) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them and see them fail.** |
| + |
| +Run: `cd $W && go test ./internal/release/ -run 'Retry' -count=1` |
| +Expected: FAIL to compile, `undefined: RequestRetry`. |
| + |
| +- [ ] **Step 3: Create `internal/release/retry.go`.** |
| + |
| +```go |
| +// The retry request (zero-downtime deploys spec §8.4, §11): `carlos deploy |
| +// --retry` asks every box to clear its failure record for one build of one |
| +// app, so the build is tried again as a canary. Promoting a build a channel |
| +// already serves writes nothing (PromoteOnLadder's repeat path), so the |
| +// request is a separate object with its own id. |
| +// |
| +// It is signed exactly like a channel pointer (signedWrapper, root key or a |
| +// scoped key with its grant) because a box has bucket write access but no |
| +// release key: an unsigned request would let one box clear another box's |
| +// failure records. |
| +// |
| +// Bucket layout addition: |
| +// |
| +// apps/<acct>/<app>/retries/<version>.json SIGNED, CAS-guarded, schema 1 |
| +// |
| +// Beside channels/ and releases/, never under them: those listings are parsed |
| +// as pointers and releases by the console and the CLI. |
| +// |
| +// Plan 5 reads it (GetRetry, Clears); plan 6 writes it from the CLI. |
| +package release |
| + |
| +import ( |
| + "context" |
| + "crypto/ed25519" |
| + "crypto/rand" |
| + "encoding/base64" |
| + "encoding/hex" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "io" |
| + "os" |
| + "regexp" |
| + "time" |
| +) |
| + |
| +// retryKind domain-separates a retry's signed payload from a channel |
| +// pointer's, so signed bytes of one can never verify as the other. |
| +const retryKind = "retry" |
| + |
| +// RetryRequest is the signed payload. |
| +// |
| +// Seq orders requests for one build: every write is strictly greater than |
| +// the last verified one and never below the writer's clock in milliseconds, |
| +// so an object that will not verify (and whose seq cannot be trusted) never |
| +// resets the order. A box's failure record keeps the Seq it last saw; a |
| +// request with a greater Seq clears it (Clears). ID names this request in |
| +// events and status. |
| +// |
| +// Channel is the channel the retry was asked for, and it must serve Version |
| +// when the request is written. A scoped key is checked against it, as a |
| +// pointer's channel is checked. The effect on a box is still per build, app |
| +// wide (spec §8.4). |
| +type RetryRequest struct { |
| + Schema int `json:"schema"` // 1 |
| + Kind string `json:"kind"` // always "retry" |
| + Account string `json:"account"` |
| + App string `json:"app"` |
| + Version string `json:"version"` |
| + Channel string `json:"channel"` |
| + ID string `json:"id"` |
| + Seq uint64 `json:"seq"` |
| + RequestedAt time.Time `json:"requested_at"` |
| + Actor string `json:"actor,omitempty"` |
| +} |
| + |
| +// ErrRetryNotServing refuses a retry for a build the named channel does not |
| +// serve: a retry renews a build that failed, it never installs one. |
| +var ErrRetryNotServing = errors.New("the channel does not serve that build") |
| + |
| +var retryIDRe = regexp.MustCompile(`^[0-9a-f]{32}$`) |
| + |
| +// RetryKey is where app's retry request for version lives. |
| +func RetryKey(acct, app, version string) string { |
| + return appPrefix(acct, app) + "retries/" + version + ".json" |
| +} |
| + |
| +// Clears reports whether this request is newer than mark, the Seq a failure |
| +// record last saw. A nil request clears nothing. |
| +func (r *RetryRequest) Clears(mark uint64) bool { return r != nil && r.Seq > mark } |
| + |
| +// RequestRetry is Signer.RequestRetry with the deployment's root key. |
| +func RequestRetry(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, channel, version, actor string) (*RetryRequest, error) { |
| + return RootSigner(priv).RequestRetry(ctx, s, acct, app, channel, version, actor) |
| +} |
| + |
| +// RequestRetry writes a new retry request for version, which channel must |
| +// serve now. A scoped key that may not sign channel is refused before the |
| +// store is touched, exactly as its promote is. |
| +func (sg Signer) RequestRetry(ctx context.Context, s Store, acct, app, channel, version, actor string) (*RetryRequest, error) { |
| + if !ValidAccount(acct) || !ValidApp(app) || !ValidChannel(channel) || !ValidVersion(version) || version == "dev" { |
| + return nil, fmt.Errorf("retry: invalid %s/%s/%s@%s", acct, app, channel, version) |
| + } |
| + if err := sg.allows(channel, time.Now().UTC()); err != nil { |
| + return nil, fmt.Errorf("retry: %w", err) |
| + } |
| + pub := sg.verifyPub() |
| + ptr, err := Channel(ctx, s, pub, acct, app, channel) |
| + if err != nil { |
| + return nil, fmt.Errorf("retry: %w", err) |
| + } |
| + if ptr.Version != version { |
| + return nil, fmt.Errorf("retry %s %s: %w (%s serves %s)", app, version, ErrRetryNotServing, channel, ptr.Version) |
| + } |
| + key := RetryKey(acct, app, version) |
| + cur, etag, err := s.Get(ctx, key) |
| + var prev uint64 |
| + switch { |
| + case err == nil: |
| + // An object that will not verify is overwritten under its own etag, |
| + // and its seq is not trusted. |
| + if old, verr := verifyRetry(pub, cur, acct, app, version); verr == nil { |
| + prev = old.Seq |
| + } |
| + case errors.Is(err, os.ErrNotExist): |
| + etag = "" |
| + default: |
| + return nil, fmt.Errorf("retry: read %s: %w", key, err) |
| + } |
| + now := time.Now().UTC() |
| + seq := uint64(now.UnixMilli()) |
| + if seq <= prev { |
| + seq = prev + 1 |
| + } |
| + nonce := make([]byte, 16) |
| + if _, err := io.ReadFull(rand.Reader, nonce); err != nil { |
| + return nil, err |
| + } |
| + r := &RetryRequest{ |
| + Schema: 1, Kind: retryKind, Account: acct, App: app, Version: version, Channel: channel, |
| + ID: hex.EncodeToString(nonce), Seq: seq, RequestedAt: now.Truncate(time.Second), Actor: actor, |
| + } |
| + wrapper, err := signRetry(sg, r) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if _, err := s.Put(ctx, key, wrapper, etag); err != nil { |
| + if errors.Is(err, ErrCASConflict) { |
| + // Another retry of this build landed between the read and the |
| + // write. It is newer than anything read here, which is all a |
| + // retry needs, so it is the answer. |
| + if raw, _, gerr := s.Get(ctx, key); gerr == nil { |
| + if won, verr := verifyRetry(pub, raw, acct, app, version); verr == nil && won.Seq > prev { |
| + return won, nil |
| + } |
| + } |
| + } |
| + return nil, err |
| + } |
| + return r, nil |
| +} |
| + |
| +// GetRetry reads the retry request for one build. Absent is (nil, nil). An |
| +// object that does not verify is an error, and a box treats it as no request: |
| +// it clears nothing. |
| +func GetRetry(ctx context.Context, s Store, pub ed25519.PublicKey, acct, app, version string) (*RetryRequest, error) { |
| + if !ValidAccount(acct) || !ValidApp(app) || !ValidVersion(version) { |
| + return nil, fmt.Errorf("retry: invalid %s/%s@%s", acct, app, version) |
| + } |
| + raw, _, err := s.Get(ctx, RetryKey(acct, app, version)) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + return verifyRetry(pub, raw, acct, app, version) |
| +} |
| + |
| +// signRetry is signPointer's discipline for a retry: validate, check the |
| +// signer's scope against the request's own channel and time, sign the exact |
| +// base64 payload, attach the grant when the signer has one. |
| +func signRetry(sg Signer, r *RetryRequest) ([]byte, error) { |
| + if err := sg.usable(); err != nil { |
| + return nil, err |
| + } |
| + if err := validateRetry(r, r.Account, r.App, r.Version); err != nil { |
| + return nil, err |
| + } |
| + if err := sg.allows(r.Channel, r.RequestedAt); err != nil { |
| + return nil, err |
| + } |
| + payload, err := json.MarshalIndent(r, "", " ") |
| + if err != nil { |
| + return nil, err |
| + } |
| + b64 := base64.StdEncoding.EncodeToString(payload) |
| + return json.MarshalIndent(signedWrapper{ |
| + Payload: b64, |
| + Sig: base64.StdEncoding.EncodeToString(ed25519.Sign(sg.key, []byte(b64))), |
| + Grant: sg.grant, |
| + }, "", " ") |
| +} |
| + |
| +// verifyRetry is verifyPointer's chain for a retry: root, then grant when |
| +// present, then payload; decode only after the signature; validate against |
| +// the key it was read from; check a grant's scope against the SIGNED channel |
| +// and requested_at. |
| +func verifyRetry(pub ed25519.PublicKey, wrapper []byte, wantAcct, wantApp, wantVersion string) (*RetryRequest, error) { |
| + var sw signedWrapper |
| + if err := json.Unmarshal(wrapper, &sw); err != nil { |
| + return nil, fmt.Errorf("retry wrapper: %v", err) |
| + } |
| + signer, scope := pub, (*Grant)(nil) |
| + if sw.Grant != nil { |
| + g, err := verifyGrant(pub, sw.Grant) |
| + if err != nil { |
| + return nil, err |
| + } |
| + gpub, err := ParsePublicKey(g.Pub, "key grant pub") |
| + if err != nil { |
| + return nil, err |
| + } |
| + signer, scope = gpub, g |
| + } |
| + sig, err := base64.StdEncoding.DecodeString(sw.Sig) |
| + if err != nil { |
| + return nil, fmt.Errorf("retry sig: %v", err) |
| + } |
| + if len(signer) != ed25519.PublicKeySize { |
| + return nil, fmt.Errorf("retry request: no usable release public key") |
| + } |
| + if !ed25519.Verify(signer, []byte(sw.Payload), sig) { |
| + return nil, fmt.Errorf("retry request signature verification failed") |
| + } |
| + payload, err := base64.StdEncoding.DecodeString(sw.Payload) |
| + if err != nil { |
| + return nil, fmt.Errorf("retry payload: %v", err) |
| + } |
| + var r RetryRequest |
| + if err := json.Unmarshal(payload, &r); err != nil { |
| + return nil, fmt.Errorf("retry payload: %v", err) |
| + } |
| + if err := validateRetry(&r, wantAcct, wantApp, wantVersion); err != nil { |
| + return nil, err |
| + } |
| + if scope != nil { |
| + if err := scope.allows(r.Channel, r.RequestedAt); err != nil { |
| + return nil, fmt.Errorf("retry request %s/%s: %w", r.App, r.Version, err) |
| + } |
| + } |
| + return &r, nil |
| +} |
| + |
| +func validateRetry(r *RetryRequest, wantAcct, wantApp, wantVersion string) error { |
| + switch { |
| + case r.Schema != 1: |
| + return fmt.Errorf("unsupported retry request schema %d", r.Schema) |
| + case r.Kind != retryKind: |
| + return fmt.Errorf("retry request: kind %q is not %q", r.Kind, retryKind) |
| + case r.Account != wantAcct || !ValidAccount(r.Account): |
| + return fmt.Errorf("retry request was signed for account %q, expected %s", r.Account, wantAcct) |
| + case r.App != wantApp || r.Version != wantVersion: |
| + return fmt.Errorf("retry request names %s/%s, expected %s/%s", r.App, r.Version, wantApp, wantVersion) |
| + case !ValidApp(r.App) || !ValidVersion(r.Version) || r.Version == "dev": |
| + return fmt.Errorf("retry request has invalid app/version %s/%s", r.App, r.Version) |
| + case !ValidChannel(r.Channel): |
| + return fmt.Errorf("retry request has invalid channel %q", r.Channel) |
| + case !retryIDRe.MatchString(r.ID): |
| + return fmt.Errorf("retry request has invalid id %q", r.ID) |
| + case r.Seq == 0: |
| + return fmt.Errorf("retry request has no seq") |
| + case r.RequestedAt.IsZero(): |
| + return fmt.Errorf("retry request has no requested_at") |
| + } |
| + return nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run both packages.** |
| + |
| +Run: `cd $W && go test ./internal/release/ ./internal/provision/ -count=1` |
| +Expected: PASS, including the golden pointer tests: `signedWrapper` is shared and unchanged. |
| + |
| +- [ ] **Step 5: Check the other parsers of `apps/` keys.** |
| + |
| +Run: `cd $W && git grep -n 'List(ctx, "apps/")\|ListDirs(ctx, "apps/")' -- '*.go' ':!*_test.go'` |
| +Expected: `internal/provision/reconcile.go` (instance and sidecar records, covered by Step 1's provision test), `internal/domains/reconcile.go` (it reads keys only as `channels/canary/` pointers or as the app's owner, and an app with a retry already has releases) and `internal/console/materialize.go` (any key keeps a trashed row; trash purges the whole prefix). Confirm each by reading its parse function. A parser not on this list is reported before committing. |
| + |
| +- [ ] **Step 6: Commit.** |
| + |
| +```bash |
| +cd $W && gofmt -l internal/release internal/provision && go vet ./internal/release/ ./internal/provision/ |
| +git add internal/release/retry.go internal/release/retry_test.go internal/provision/retrykey_test.go |
| +git -c user.name=claude -c user.email='claude~paul@keymail.dev' commit -m "zdd-4 task 10: the signed retry request, verified like a channel pointer |
| + |
| +Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 11: The full gate, push and CI |
| + |
| +- [ ] **Step 1: Nothing reads a swap flag yet.** |
| + |
| +Run: `cd $W && git grep -n CARLOS_SWAP -- '*.go'` |
| +Expected: no output. Plan 5 introduces the flag. |
| + |
| +- [ ] **Step 2: Full gate.** Run it outside the sandbox, at low priority, with the `-exec` wrapper. Activator tests need a short temp root; every other package uses one on disk. |
| + |
| +```bash |
| +mkdir -p ~/.cache/zdd-4/gotmp |
| +cat > ~/.cache/zdd-4/texec.sh <<'EOF' |
| +#!/bin/sh |
| +case "$1" in |
| +*/activator.test) root=/tmp/k ;; |
| +*) root=/var/tmp/ck ;; |
| +esac |
| +mkdir -p "$root" |
| +export TMPDIR="$root" |
| +exec "$@" |
| +EOF |
| +chmod +x ~/.cache/zdd-4/texec.sh |
| +cd /home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-4-release-side |
| +go build ./... && GOTMPDIR=$HOME/.cache/zdd-4/gotmp nice -n 19 \ |
| + env GOFLAGS="-buildvcs=false -exec=$HOME/.cache/zdd-4/texec.sh" make test |
| +``` |
| + |
| +Expected: build, vet, gofmt and every package's tests pass. A failure is read first, never just re-run. A failure in `internal/host` or `internal/console` that mentions a sidecar's version names a test that pinned behaviour this plan changes on purpose (see the header). Report it with the spec section; do not edit it to pass. |
| + |
| +- [ ] **Step 3: Cross-compile.** `internal/host` and the activator are linked into the CLI. |
| + |
| +```bash |
| +cd $W && for p in darwin/arm64 windows/amd64 linux/arm64; do GOOS=${p%/*} GOARCH=${p#*/} go build ./... || echo "FAIL $p"; done |
| +``` |
| + |
| +Expected: no `FAIL` lines. |
| + |
| +- [ ] **Step 4: Clean up, push and describe.** |
| + |
| +```bash |
| +rm -rf /tmp/k /var/tmp/ck ~/.cache/zdd-4 |
| +cd $W && git push origin zdd-4-release-side |
| +amadan branch describe carlos/platform zdd-4-release-side -body - <<'EOF' |
| +Zero-downtime deploys, plan 4 (release side). Spec: docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md §4.2, §7.9, §8.4. Plan: docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md. |
| + |
| +Additive: Manifest.Swap, lossless staged siblings, CanonicalDemand, the pin set and PinSource, the signed retry request. Nothing reads swap until plan 5. |
| + |
| +Behaviour changes the spec mandates: workers exec their staged build by path and recycle (crash, max_run) onto the captured build; a worker's /v1/ready report no longer writes registry Version; SidecarAdoptable adopts native sidecars (stage, repoint, stamp, no restart); a reused staged copy is hashed and restaged only when nothing refers to it. |
| + |
| +Known display effect: `carlos sidecars ls` RUNNING VERSION now shows root's install stamp until plans 5 and 6. |
| +EOF |
| +amadan ci status carlos/platform -branch zdd-4-release-side; echo "exit=$?" |
| +``` |
| + |
| +Expected: the push succeeds; CI status eventually exits 0. Do not merge. Merging waits for review of this branch and Paul's call on the open points in the self-review notes. |
| + |
| +--- |
| + |
| +## Self-review notes |
| + |
| +- **Spec coverage.** §4.2 `Manifest.swap` additive: Task 1. Raw-byte staging: Task 2. Canonical read, digest compare, mismatch refusal, unverifiable gives hold: Task 4. Root restage check and the refusal text: Task 6. Pin set (live links, `Version`, `PrevVersion`): Task 5. `GoodVersion` and generations through `PinSource`: Task 5 defines it, plan 5 fills it. Retained identity (path, sha256, demand): `StagedIdentity`, Task 3. Recovery checks the retained digest: `VerifyStaged`, Tasks 3 and 8. §7.9 `SidecarAdoptable`, same selection, bake, staging, repoint, stamp, cold materialization, `Adoptable` not widened: Task 7. Crash and `max_run` respawn the serving build by immutable path: Tasks 8 and 9. Serving identity is the agent-captured spawn build: Tasks 8 and 9. Reported version display-only: Task 9. Coalescing: hook only (`CoalesceRecycle`), plan 5. Drift check queued as an attempt: plan 5. §8.4 signed retry request: Task 10; CLI is plan 6. §12 tests owned here: "Swap mode source" (Task 4), "Artifact binding" (Tasks 4 and 6), "Restage refusal" (Task 6), "Worker identity" (Task 9), "Sidecars" first, third and fourth clauses (Task 7; the "queued as a paced release attempt" clause is plan 5's). |
| +- **Not here, by the index:** the `swap` field's readers, failure records, `GoodVersion`, the scheduler, the stage-relay touch on an agent-side mismatch (plan 5); CLI and console surfaces (plan 6). |
| +- **Open points for Paul and plan 5** (also in the report that accompanied this plan): |
| + 1. Spec §4.2 says root "restages over it, as today". Today root never restages an existing staged directory: `ensureStaged` reuses it by name without comparing. The restage is new behaviour, implemented in Task 6. |
| + 2. Main now has `carlos promote --retry` (D1, `9e628024`), which re-signs a pointer with a fresh `promoted_at` to lift a box's refusal. Spec §8.4's reason for a separate retry object ("promoting a build that's already current is a no-op") is no longer quite true on main, and plan 6's `carlos deploy --retry` will sit beside a `carlos promote --retry` that does something else. D1's per-host refusal records also overlap plan 5's failure records. |
| + 3. Registry `Version` for a sidecar becomes root's install stamp, so `carlos sidecars ls` RUNNING VERSION shows it until plans 5 and 6. The sidecar-members design §9.2 promised a running version there. |
| + 4. The retry request is bound to a channel (it must serve the build; a scoped key must cover it), while its effect on a box is per build, app wide, as the spec says. The spec says only "signed like a channel pointer". |
| + 5. "Live or pending generations" live in the agent's memory, but root's adopt pass is a separate process (`carlos-adopt.service`). Plan 5's `PinSource` must persist them somewhere root can read. |
| + 6. When the canonical manifest is unreadable at root staging time, the staged copy is used as today (Review Focus 3). The spec is silent for root staging. `CanonicalDemand` treats "no artifact for this platform" as hold (cannot compare), while root staging treats the same condition on a reused copy as a mismatch. |
| + 7. `host.fastMembers` keeps `Adoptable`, so a canary sidecar is not in the 2–3 s fast sweep. |
amadan