amadan
plan: console release defences, plan 1 of 4 (D1)
Nine tasks: refusal records, the adoptHost refusal gate, the process-group rollback, the settle check, readiness on the effective upstream, wiring the checked restart and rollback restart, promote --retry, the alert and status --live line, and the full gate with an operator note.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+1952
−0
docs/superpowers/plans/2026-10-06-console-release-defences-plan-1-d1.md+1952 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-06-console-release-defences-plan-1-d1.md b/docs/superpowers/plans/2026-10-06-console-release-defences-plan-1-d1.md| new file mode 100644 |
| index 000000000..3953c4dbe |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-06-console-release-defences-plan-1-d1.md |
| @@ -0,0 +1,1952 @@ |
| +# Console release defences, Plan 1 of 4: D1 (roll back a named unit that will not come up) |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** A box that adopts a named-unit build (today: the console) checks that |
| +the new process actually comes up. If it doesn't, the box rolls back, refuses |
| +that promotion until `carlos promote --retry`, and alerts. |
| + |
| +**Architecture:** `adoptHost` (internal/release/adopt.go) already rolls back |
| +when its restart callback errors. This plan makes the named-unit restart |
| +return an honest error by settling the unit after `try-restart`, through a |
| +small `unitSystem` seam over systemctl. It adds a durable refusal record in the |
| +bake directory, scoped to the routes that share one process. It also adds a |
| +`--retry` promote, and an alert plus a `status --live` line fed from the |
| +refusal records. |
| + |
| +**Tech Stack:** Go (stdlib, `os/exec` systemctl, `net/http` over TCP or Unix |
| +sockets), the existing release/registry/host/status packages. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-06-console-release-defences-design.md` |
| +(rev 5): §2 (D1) and §7's D1 and retry tests. Plans 2 (D2), 3 (D3) and 4 (D4, |
| +process and scripts) follow; this plan stands alone. |
| + |
| +## Global Constraints |
| + |
| +- Settle window `CARLOS_UNIT_SETTLE`: default 30 s, floor 5 s, poll every 500 ms (spec §2.3). |
| +- Readiness follows the effective upstream: `Socket` if set, else `Addr` (spec §2.4). |
| +- No-probe readiness: the unit stays `active` within one invocation for 5 s (spec §2.4). |
| +- A unit not running before the restart is never checked or started (spec §2.3). |
| +- A binding mismatch (`ExecStart` not `live/<host>`) skips the settle check only; the `try-restart` error still propagates (spec §2.2, codex 12). |
| +- D1 rollback runs `reset-failed` then `restart`; every other `makeStartRoute` caller keeps `start` (spec §2.3, codex 1). |
| +- A rollback group is the bound route plus unstamped same-app, same-account routes with the same non-empty upstream; two empty upstreams never match (spec §2.3). |
| +- A refusal blocks while the pointer's `Version` and `PromotedAt` equal the record's (spec §2.5). See Task 1's note on keying. |
| +- If `systemctl show` fails (for example, the sandboxed agent may not query the unit), behave exactly as today: plain `try-restart`, return its error. |
| +- Gate locally with `GOFLAGS=-buildvcs=false`; in this sandbox set `GOCACHE=$TMPDIR/gocache` per command (AGENTS.md path is read-only here). |
| +- Commits go on branch `design-console-release-defences`'s implementation branch (named at execution), as the agent identity, ending with the Co-Authored-By line. |
| + |
| +## Review Focus |
| + |
| +1. **A console that takes longer than the window to start** (cold registry, slow disk). Expect: it is rolled back only after the full window, the window is configurable, and the log says "not ready after 30s", not "crashed". Test in Task 4 (`TestSettleDeadlineSaysNotReady`). |
| +2. **The sandboxed agent cannot run `systemctl show`.** Expect: today's behaviour exactly, with no false rollback. Test in Task 6 (`TestNamedRestartShowFailsFallsBack`). |
| +3. **An operator stopped the console on purpose during an adoption.** Expect: no check, no start, and it stays stopped. Test in Task 4 (`TestSettleSkipsAUnitThatWasNotRunning`). |
| +4. **A corrupt or half-written refusal file.** Expect: treated as absent and logged, never a permanent block or a panic. Test in Task 1 (`TestReadRefusalCorruptIsAbsent`). |
| +5. **Two instances of one app on different upstreams.** Expect: a rollback of one never touches the other. Test in Task 3 (`TestRollbackGroupLeavesOtherUpstreamAlone`). |
| + |
| +--- |
| + |
| +### Task 1: Refusal records |
| + |
| +**Files:** |
| +- Create: `internal/release/refusal.go` |
| +- Test: `internal/release/refusal_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type Refusal struct { Schema int; Host, App, Version, SHA256, Platform string; PointerPromotedAt, RefusedAt time.Time; Reason string }` (JSON keys below) |
| + - `func RefusalDir(bakeDir string) string` |
| + - `func WriteRefusal(bakeDir string, r Refusal) error` |
| + - `func ReadRefusal(bakeDir, host string) (*Refusal, error)` — `(nil, nil)` when absent or unparseable |
| + - `func ListRefusals(bakeDir string) ([]Refusal, error)` |
| + - `func ClearRefusal(bakeDir, host string) error` — nil when absent |
| + - `func (r *Refusal) Blocks(p *ChannelPointer) bool` |
| + |
| +**Keying note (for the reviewer):** spec §2.5 keys on the artifact sha256. This |
| +plan blocks on `(Version, PointerPromotedAt)` and records `SHA256` for display. |
| +That is equivalent: a version's manifest is immutable while a pointer names |
| +it, and reusing a name for new bytes needs a new promotion, which changes |
| +`PromotedAt` and clears the block. It also avoids a manifest GET on every |
| +fast-loop tick for a refused route. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + "time" |
| +) |
| + |
| +func TestRefusalRoundTripAndBlocks(t *testing.T) { |
| + dir := t.TempDir() |
| + at := time.Date(2026, 10, 6, 9, 30, 30, 0, time.UTC) |
| + r := Refusal{Host: "console.example.com", App: "console", Version: "v2", |
| + SHA256: "abc", Platform: "linux-arm64", PointerPromotedAt: at, |
| + RefusedAt: at.Add(10 * time.Second), Reason: "exited: status=2"} |
| + if err := WriteRefusal(dir, r); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := ReadRefusal(dir, "console.example.com") |
| + if err != nil || got == nil { |
| + t.Fatalf("ReadRefusal = %+v, %v", got, err) |
| + } |
| + if got.Schema != 1 || got.Version != "v2" || !got.PointerPromotedAt.Equal(at) { |
| + t.Fatalf("round trip = %+v", got) |
| + } |
| + if !got.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at}) { |
| + t.Fatal("the same promotion is not blocked") |
| + } |
| + if got.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at.Add(time.Second)}) { |
| + t.Fatal("a re-signed pointer (retry) is still blocked") |
| + } |
| + if got.Blocks(&ChannelPointer{Version: "v3", PromotedAt: at}) { |
| + t.Fatal("a different version is blocked") |
| + } |
| + var nilR *Refusal |
| + if nilR.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at}) { |
| + t.Fatal("a nil refusal blocks") |
| + } |
| +} |
| + |
| +func TestReadRefusalAbsentIsNil(t *testing.T) { |
| + got, err := ReadRefusal(t.TempDir(), "nobody.example.com") |
| + if got != nil || err != nil { |
| + t.Fatalf("absent = %+v, %v; want nil, nil", got, err) |
| + } |
| +} |
| + |
| +func TestReadRefusalCorruptIsAbsent(t *testing.T) { |
| + dir := t.TempDir() |
| + p := filepath.Join(RefusalDir(dir), "console.example.com.json") |
| + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(p, []byte(`{"schema":1,"host":`), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := ReadRefusal(dir, "console.example.com") |
| + if got != nil || err != nil { |
| + t.Fatalf("corrupt = %+v, %v; want nil, nil (a torn file must never block)", got, err) |
| + } |
| +} |
| + |
| +func TestRefusalRejectsAPathInTheHost(t *testing.T) { |
| + if err := WriteRefusal(t.TempDir(), Refusal{Host: "../etc/x"}); err == nil { |
| + t.Fatal("a host with a path separator was written") |
| + } |
| +} |
| + |
| +func TestListAndClearRefusals(t *testing.T) { |
| + dir := t.TempDir() |
| + for _, h := range []string{"a.example.com", "b.example.com"} { |
| + if err := WriteRefusal(dir, Refusal{Host: h, Version: "v2"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := ClearRefusal(dir, "a.example.com"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := ClearRefusal(dir, "never.example.com"); err != nil { |
| + t.Fatalf("clearing an absent record: %v", err) |
| + } |
| + all, err := ListRefusals(dir) |
| + if err != nil || len(all) != 1 || all[0].Host != "b.example.com" { |
| + t.Fatalf("ListRefusals = %+v, %v", all, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'Refusal' -count=1` |
| +Expected: FAIL, `undefined: Refusal`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +// Refusal records — D1 of the console release defences (spec |
| +// 2026-10-06-console-release-defences-design.md §2.5). |
| +// |
| +// When a named unit does not come up on a new build, the root pass rolls it |
| +// back and writes one of these per host in the rolled-back process group. |
| +// Every later pass, the agent's fast loop and root alike, reads it before |
| +// staging and skips the promotion it names. Without it the very next pass |
| +// would adopt the same build and bounce the service again. |
| +// |
| +// The record blocks a PROMOTION, not a version name: it matches while the |
| +// channel pointer still has the version and PromotedAt it had when the build |
| +// was refused. `carlos promote --retry` re-signs the pointer with a fresh |
| +// PromotedAt, and so does any move to another version; either clears it. |
| +// |
| +// Root writes these under the bake directory, which the sandboxed agent can |
| +// read. A file that does not parse reads as absent: a torn write must never |
| +// become a permanent block. |
| +package release |
| + |
| +import ( |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "log" |
| + "os" |
| + "path/filepath" |
| + "sort" |
| + "strings" |
| + "time" |
| +) |
| + |
| +const refusalSchema = 1 |
| + |
| +type Refusal struct { |
| + Schema int `json:"schema"` |
| + Host string `json:"host"` |
| + App string `json:"app"` |
| + Version string `json:"version"` |
| + SHA256 string `json:"sha256,omitempty"` |
| + Platform string `json:"platform,omitempty"` |
| + PointerPromotedAt time.Time `json:"pointer_promoted_at"` |
| + RefusedAt time.Time `json:"refused_at"` |
| + Reason string `json:"reason"` |
| +} |
| + |
| +func RefusalDir(bakeDir string) string { return filepath.Join(bakeDir, "refusals") } |
| + |
| +func refusalPath(bakeDir, host string) (string, error) { |
| + if host == "" || strings.ContainsAny(host, `/\`) || host == "." || host == ".." { |
| + return "", fmt.Errorf("refusal: invalid host %q", host) |
| + } |
| + return filepath.Join(RefusalDir(bakeDir), host+".json"), nil |
| +} |
| + |
| +// WriteRefusal writes atomically (temp file + rename) and world-readable, |
| +// because the reader is the sandboxed agent. |
| +func WriteRefusal(bakeDir string, r Refusal) error { |
| + p, err := refusalPath(bakeDir, r.Host) |
| + if err != nil { |
| + return err |
| + } |
| + r.Schema = refusalSchema |
| + b, err := json.MarshalIndent(r, "", " ") |
| + if err != nil { |
| + return err |
| + } |
| + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { |
| + return err |
| + } |
| + tmp, err := os.CreateTemp(filepath.Dir(p), ".refusal-*") |
| + if err != nil { |
| + return err |
| + } |
| + defer os.Remove(tmp.Name()) |
| + if _, err := tmp.Write(append(b, '\n')); err != nil { |
| + tmp.Close() |
| + return err |
| + } |
| + if err := tmp.Chmod(0o644); err != nil { |
| + tmp.Close() |
| + return err |
| + } |
| + if err := tmp.Close(); err != nil { |
| + return err |
| + } |
| + return os.Rename(tmp.Name(), p) |
| +} |
| + |
| +func ReadRefusal(bakeDir, host string) (*Refusal, error) { |
| + p, err := refusalPath(bakeDir, host) |
| + if err != nil { |
| + return nil, err |
| + } |
| + b, err := os.ReadFile(p) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + var r Refusal |
| + if err := json.Unmarshal(b, &r); err != nil || r.Schema != refusalSchema { |
| + log.Printf("refusal: %s does not parse (%v), treating it as absent", p, err) |
| + return nil, nil |
| + } |
| + return &r, nil |
| +} |
| + |
| +func ListRefusals(bakeDir string) ([]Refusal, error) { |
| + ents, err := os.ReadDir(RefusalDir(bakeDir)) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + var out []Refusal |
| + for _, e := range ents { |
| + name := e.Name() |
| + if e.IsDir() || !strings.HasSuffix(name, ".json") || strings.HasPrefix(name, ".") { |
| + continue |
| + } |
| + r, err := ReadRefusal(bakeDir, strings.TrimSuffix(name, ".json")) |
| + if err != nil || r == nil { |
| + continue |
| + } |
| + out = append(out, *r) |
| + } |
| + sort.Slice(out, func(i, j int) bool { return out[i].Host < out[j].Host }) |
| + return out, nil |
| +} |
| + |
| +func ClearRefusal(bakeDir, host string) error { |
| + p, err := refusalPath(bakeDir, host) |
| + if err != nil { |
| + return err |
| + } |
| + if err := os.Remove(p); err != nil && !errors.Is(err, os.ErrNotExist) { |
| + return err |
| + } |
| + return nil |
| +} |
| + |
| +// Blocks reports whether this record still refuses the promotion p names. |
| +func (r *Refusal) Blocks(p *ChannelPointer) bool { |
| + return r != nil && p != nil && r.Version == p.Version && r.PointerPromotedAt.Equal(p.PromotedAt) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests to verify they pass** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'Refusal' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/release/refusal.go internal/release/refusal_test.go |
| +git commit -m "release: refusal records for builds a box rolled back (console defences D1)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: adoptHost honours a refusal before it stages or relays |
| + |
| +**Files:** |
| +- Modify: `internal/release/adopt.go` (in `adoptHost`, after the `poisoned[ver]` check and before `ensureStaged`; and after the final `SetVersions` success) |
| +- Test: `internal/release/adopt_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `ReadRefusal`, `ClearRefusal`, `(*Refusal).Blocks`; `passConfig.bakeDir` (set by `BakeDir(dir)`). |
| +- Produces: nothing new; the printed line `refused: <ver> did not come up on this box at <time> (<reason>); retry with: carlos promote --retry <ver> <channel>` is what Task 8 and operators grep. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// TestRefusedPromotionIsNeitherStagedNorRelayed: a refusal naming the current |
| +// pointer stops adoptHost before ensureStaged, so the agent never relays a |
| +// root pass for it and nothing restarts. |
| +func TestRefusedPromotionIsNeitherStagedNorRelayed(t *testing.T) { |
| + host := "console.example.com" |
| + ctx, reg, s, pub, opt := startUnitFixture(t, host) |
| + bake := t.TempDir() |
| + ptr, err := Channel(ctx, s, pub, "bab", "console", "canary/x") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := WriteRefusal(bake, Refusal{Host: host, App: "console", Version: ptr.Version, |
| + PointerPromotedAt: ptr.PromotedAt, Reason: "exited: status=2"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + restart := func(h string) error { t.Fatalf("restart consulted for refused %s", h); return nil } |
| + relayed := false |
| + var out bytes.Buffer |
| + withCapturedStdout(t, &out, func() { |
| + if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake), |
| + OnStageBlocked(func(string, string, error) { relayed = true })); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + }) |
| + if v, _ := LiveVersion(opt, host); v != "old" { |
| + t.Fatalf("live = %q, want old", v) |
| + } |
| + if relayed { |
| + t.Fatal("a refused promotion was relayed to root") |
| + } |
| + if !strings.Contains(out.String(), "promote --retry") { |
| + t.Fatalf("printed %q, want the retry hint", out.String()) |
| + } |
| +} |
| + |
| +// TestRePromotionClearsTheBlock: a pointer re-signed for the same version |
| +// (what promote --retry does) is adopted, and the stale record is removed. |
| +func TestRePromotionClearsTheBlock(t *testing.T) { |
| + host := "console.example.com" |
| + ctx, reg, s, pub, opt := startUnitFixture(t, host) |
| + bake := t.TempDir() |
| + if err := WriteRefusal(bake, Refusal{Host: host, App: "console", Version: "new", |
| + PointerPromotedAt: time.Unix(1, 0).UTC(), Reason: "older promotion"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }, BakeDir(bake)); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + if v, _ := LiveVersion(opt, host); v != "new" { |
| + t.Fatalf("live = %q, want new (the record names an older promotion)", v) |
| + } |
| + if r, _ := ReadRefusal(bake, host); r != nil { |
| + t.Fatalf("stale refusal survived a successful adoption: %+v", r) |
| + } |
| +} |
| +``` |
| + |
| +`Channel` (internal/release/release.go:1047) is the verified pointer read. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'TestRefusedPromotion|TestRePromotionClears' -count=1` |
| +Expected: FAIL. The first test's restart fatals, and the second still finds the record. |
| + |
| +- [ ] **Step 3: Implement the gate** |
| + |
| +In `adoptHost`, immediately after: |
| + |
| +```go |
| + if poisoned[ver] { |
| + fmt.Printf(" %s: skipping %s (failed on another host this pass)\n", rt.Host, ver) |
| + return |
| + } |
| +``` |
| + |
| +insert: |
| + |
| +```go |
| + // D1's refusal (refusal.go): this box rolled this exact promotion back |
| + // because the unit did not come up. Checked before ensureStaged so a |
| + // refused build is neither downloaded nor relayed to a root pass. |
| + if cfg.bakeDir != "" { |
| + if r, err := ReadRefusal(cfg.bakeDir, rt.Host); err != nil { |
| + fmt.Printf(" %s: reading refusal record: %v\n", rt.Host, err) |
| + } else if r.Blocks(ptr) { |
| + fmt.Printf(" %s: refused: %s did not come up on this box at %s (%s); retry with: carlos promote --retry %s %s\n", |
| + rt.Host, ver, r.RefusedAt.Format(time.RFC3339), r.Reason, ver, channel) |
| + return |
| + } |
| + } |
| +``` |
| + |
| +and immediately after the final successful `reg.SetVersions(rt.Host, ver, cur)` |
| +(the one followed by the `updated %s -> %s` print), insert: |
| + |
| +```go |
| + // A record left from an older promotion is stale once a build is live. |
| + // Best-effort: the agent cannot remove root's file, the root pass can. |
| + if cfg.bakeDir != "" { |
| + _ = ClearRefusal(cfg.bakeDir, rt.Host) |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Run the release package tests** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -count=1` |
| +Expected: PASS (new tests and every existing adopt test). |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/release/adopt.go internal/release/adopt_test.go |
| +git commit -m "release: adoptHost skips a refused promotion before staging or relaying" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: The process-group rollback when a unit does not come up |
| + |
| +**Files:** |
| +- Create: `internal/release/rollback.go` |
| +- Modify: `internal/release/adopt.go` (`passConfig`, new `WithRollbackRestart`, the `rerr != nil` branch of `adoptHost`) |
| +- Test: `internal/release/rollback_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type UnitNotUpError struct { Unit, Reason, Journal string }` with `Error() string`. The host package's restart returns this (Task 6) and adoptHost tests it with `errors.As`. |
| + - `func upstream(rt registry.Route) string`: `rt.Socket` if non-empty, else `rt.Addr`. |
| + - `func processGroup(reg *registry.Registry, bound registry.Route) []registry.Route`: excludes `bound`. |
| + - `func WithRollbackRestart(fn func(host string) error) PassOption`: replaces `startUnit` on the D1 path only. |
| +- Consumes: Task 1's `WriteRefusal`; `LiveVersion`, `PointLive`, `RouteAccount`, `reg.List`, `reg.SetVersions`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "bytes" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +func groupFixtureRoute(t *testing.T, reg *registry.Registry, host, socket, addr, unit string) { |
| + t.Helper() |
| + if err := reg.Upsert(registry.Route{Host: host, Kind: "instance", App: "console", |
| + Channel: "canary/x", Socket: socket, Addr: addr, Unit: unit}); err != nil { |
| + t.Fatalf("upsert %s: %v", host, err) |
| + } |
| +} |
| + |
| +func TestProcessGroupKeysOnTheEffectiveUpstream(t *testing.T) { |
| + reg := testRegistry(t) |
| + groupFixtureRoute(t, reg, "platform.example.com", "", "127.0.0.1:8090", "console.service") |
| + groupFixtureRoute(t, reg, "console.example.com", "", "127.0.0.1:8090", "") |
| + groupFixtureRoute(t, reg, "other.example.com", "", "127.0.0.1:9000", "") |
| + groupFixtureRoute(t, reg, "sock.example.com", "/run/x.sock", "127.0.0.1:8090", "") |
| + bound, _ := reg.Get("platform.example.com") |
| + got := processGroup(reg, *bound) |
| + if len(got) != 1 || got[0].Host != "console.example.com" { |
| + t.Fatalf("group = %+v, want only console.example.com (socket wins over addr; 9000 is another process)", got) |
| + } |
| +} |
| + |
| +func TestProcessGroupNeverMatchesEmptyUpstreams(t *testing.T) { |
| + reg := testRegistry(t) |
| + groupFixtureRoute(t, reg, "a.example.com", "", "", "a.service") |
| + groupFixtureRoute(t, reg, "b.example.com", "", "", "") |
| + bound, _ := reg.Get("a.example.com") |
| + if got := processGroup(reg, *bound); len(got) != 0 { |
| + t.Fatalf("group = %+v, want none: two empty upstreams are not one process", got) |
| + } |
| +} |
| + |
| +// The flagship shape: three hosts, one process, the unit runs the LAST |
| +// host's link. Hosts before it were repointed and stamped; the bound host's |
| +// restart reports the unit did not come up. |
| +func TestNotUpRollsTheWholeGroupBackAndRefusesEveryHost(t *testing.T) { |
| + ctx, reg, s, pub, opt := startUnitFixture(t, "console.example.com") |
| + if err := reg.SetUnit("console.example.com", ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + groupFixtureRoute(t, reg, "platform.example.com", "", "127.0.0.1:8090", "console.service") |
| + if err := reg.SetUpstreams("console.example.com", "127.0.0.1:8090", "", ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := PointLive(opt, "platform.example.com", "bab", "old", "console"); err != nil { |
| + t.Fatal(err) |
| + } |
| + bake := t.TempDir() |
| + restart := func(h string) error { |
| + if h == "platform.example.com" { |
| + return &UnitNotUpError{Unit: "console.service", Reason: "exited: status=2"} |
| + } |
| + return nil |
| + } |
| + var rolledBackHost string |
| + rb := func(h string) error { rolledBackHost = h; return nil } |
| + var out bytes.Buffer |
| + withCapturedStdout(t, &out, func() { |
| + if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake), WithRollbackRestart(rb)); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + }) |
| + for _, h := range []string{"console.example.com", "platform.example.com"} { |
| + if v, _ := LiveVersion(opt, h); v != "old" { |
| + t.Fatalf("%s live = %q, want old", h, v) |
| + } |
| + rt, _ := reg.Get(h) |
| + if rt.Version == "new" { |
| + t.Fatalf("%s still stamped new: %+v", h, rt) |
| + } |
| + if r, _ := ReadRefusal(bake, h); r == nil || r.Version != "new" { |
| + t.Fatalf("%s refusal = %+v, want one naming new", h, r) |
| + } |
| + } |
| + if rolledBackHost != "platform.example.com" { |
| + t.Fatalf("rollback restart ran for %q, want the bound host", rolledBackHost) |
| + } |
| +} |
| + |
| +func TestRollbackGroupLeavesOtherUpstreamAlone(t *testing.T) { |
| + ctx, reg, s, pub, opt := startUnitFixture(t, "console.example.com") |
| + if err := reg.SetUnit("console.example.com", "console.service"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetUpstreams("console.example.com", "127.0.0.1:8090", "", ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + groupFixtureRoute(t, reg, "zz-other.example.com", "", "127.0.0.1:9000", "") |
| + if err := StageArtifact(opt, "bab", "new", "console", []byte("bad build")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := PointLive(opt, "zz-other.example.com", "bab", "new", "console"); err != nil { |
| + t.Fatal(err) |
| + } |
| + restart := func(h string) error { |
| + if h == "console.example.com" { |
| + return &UnitNotUpError{Unit: "console.service", Reason: "not ready after 30s"} |
| + } |
| + return nil |
| + } |
| + bake := t.TempDir() |
| + if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake), WithRollbackRestart(func(string) error { return nil })); err != nil { |
| + t.Fatalf("Pass: %v", err) |
| + } |
| + if v, _ := LiveVersion(opt, "zz-other.example.com"); v != "new" { |
| + t.Fatalf("another process's route was rolled back to %q", v) |
| + } |
| + if r, _ := ReadRefusal(bake, "zz-other.example.com"); r != nil { |
| + t.Fatalf("another process's route was refused: %+v", r) |
| + } |
| +} |
| +``` |
| + |
| +Check the fixture's assumptions before running: `reg.SetUnit` and |
| +`reg.SetUpstreams(host, addr, addrs, health)` exist at |
| +`internal/registry/registry.go:1231` and `:1315`, and `startUnitFixture`'s route |
| +starts with no `Unit`. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'ProcessGroup|NotUpRolls|RollbackGroup' -count=1` |
| +Expected: FAIL, `undefined: UnitNotUpError`. |
| + |
| +- [ ] **Step 3: Implement `rollback.go`** |
| + |
| +```go |
| +// The D1 rollback (spec 2026-10-06-console-release-defences-design.md §2.3): |
| +// what adoptHost does when a named unit's restart reports that the new build |
| +// did not come up, as opposed to systemctl itself failing. |
| +package release |
| + |
| +import ( |
| + "fmt" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// UnitNotUpError is a named unit that was restarted onto a new build and did |
| +// not settle: it failed, restarted again, or was not ready by the deadline. |
| +// The host package's settle check returns it (internal/host/unitsettle.go). |
| +type UnitNotUpError struct { |
| + Unit string |
| + Reason string |
| + Journal string // the unit's last journal lines, for the rollback line and the alert |
| +} |
| + |
| +func (e *UnitNotUpError) Error() string { |
| + return fmt.Sprintf("%s did not come up: %s", e.Unit, e.Reason) |
| +} |
| + |
| +// upstream is a route's effective upstream: Socket wins over Addr, as at the |
| +// edge (registry.Route.Socket's doc). |
| +func upstream(rt registry.Route) string { |
| + if rt.Socket != "" { |
| + return rt.Socket |
| + } |
| + return rt.Addr |
| +} |
| + |
| +// processGroup is every OTHER route served by bound's process: unstamped, |
| +// same account and app, same non-empty upstream. A bound route with no |
| +// upstream is a group of one. |
| +func processGroup(reg *registry.Registry, bound registry.Route) []registry.Route { |
| + up := upstream(bound) |
| + if up == "" { |
| + return nil |
| + } |
| + all, err := reg.List() |
| + if err != nil { |
| + fmt.Printf(" %s: listing routes for the rollback group: %v (rolling back this host only)\n", bound.Host, err) |
| + return nil |
| + } |
| + var out []registry.Route |
| + for _, r := range all { |
| + if r.Host == bound.Host || r.Unit != "" || r.App != bound.App || |
| + RouteAccount(r) != RouteAccount(bound) || upstream(r) != up { |
| + continue |
| + } |
| + out = append(out, r) |
| + } |
| + return out |
| +} |
| + |
| +// rollbackGroup puts every group route whose link names the rejected version |
| +// onto restored (what bound now runs), stamps it to match, and writes a |
| +// refusal for bound and every group host. |
| +func rollbackGroup(reg *registry.Registry, optDir string, cfg passConfig, bound registry.Route, |
| + ptr *ChannelPointer, rejected, restored, sha, platform string, nu *UnitNotUpError, now time.Time) { |
| + hosts := []string{bound.Host} |
| + for _, g := range processGroup(reg, bound) { |
| + hosts = append(hosts, g.Host) |
| + v, err := LiveVersion(optDir, g.Host) |
| + if err != nil || v != rejected || restored == "" { |
| + continue |
| + } |
| + if err := PointLive(optDir, g.Host, RouteAccount(g), restored, g.App); err != nil { |
| + fmt.Printf(" %s: could not roll the symlink back to %s with its process: %v\n", g.Host, restored, err) |
| + continue |
| + } |
| + if err := reg.SetVersions(g.Host, restored, g.PrevVersion); err != nil { |
| + fmt.Printf(" %s: rolled back to %s but recording the version failed: %v\n", g.Host, restored, err) |
| + } |
| + } |
| + if cfg.bakeDir == "" { |
| + return |
| + } |
| + reason := nu.Reason |
| + if j := strings.TrimSpace(nu.Journal); j != "" { |
| + lines := strings.Split(j, "\n") |
| + reason += " — " + lines[len(lines)-1] |
| + } |
| + for _, h := range hosts { |
| + r := Refusal{Host: h, App: bound.App, Version: rejected, SHA256: sha, Platform: platform, |
| + PointerPromotedAt: ptr.PromotedAt, RefusedAt: now.UTC(), Reason: reason} |
| + if err := WriteRefusal(cfg.bakeDir, r); err != nil { |
| + fmt.Printf(" %s: writing refusal record: %v\n", h, err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`now` is the `now time.Time` that `adoptHost` already receives. Add `"time"` |
| +to `rollback.go`'s imports. |
| + |
| +- [ ] **Step 4: Wire it into `adoptHost` and `passConfig`** |
| + |
| +In `passConfig` (adopt.go:34) add: |
| + |
| +```go |
| + // rollbackRestart replaces startUnit on D1's path only — a named unit |
| + // whose restart returned *UnitNotUpError. It must REPLACE whatever is |
| + // running (reset-failed + restart), because a deadline failure can leave |
| + // the rejected build running unready and `start` is a no-op on it. |
| + rollbackRestart func(host string) error |
| +``` |
| + |
| +Below `WithStartUnit`, add: |
| + |
| +```go |
| +// WithRollbackRestart sets D1's rollback restart (see passConfig). |
| +func WithRollbackRestart(fn func(host string) error) PassOption { |
| + return func(c *passConfig) { c.rollbackRestart = fn } |
| +} |
| +``` |
| + |
| +In the `rerr != nil` branch of `adoptHost`, right after |
| +`poisoned[ver] = true`, record whether this is D1's case: |
| + |
| +```go |
| + var notUp *UnitNotUpError |
| + isNotUp := errors.As(rerr, ¬Up) |
| +``` |
| + |
| +Replace the final start block |
| + |
| +```go |
| + if cfg.startUnit != nil && rolledBack && !daemonRestart { |
| + if serr := cfg.startUnit(rt.Host); serr != nil { |
| +``` |
| + |
| +with: |
| + |
| +```go |
| + start := cfg.startUnit |
| + if isNotUp && cfg.rollbackRestart != nil { |
| + start = cfg.rollbackRestart |
| + } |
| + if isNotUp { |
| + sha := "" |
| + if m != nil { |
| + sha = m.Artifacts[adoptPlatform].SHA256 |
| + } |
| + rollbackGroup(reg, optDir, cfg, rt, ptr, ver, cur, sha, adoptPlatform, notUp, now) |
| + if notUp.Journal != "" { |
| + fmt.Printf(" %s: %s's last lines:\n%s\n", rt.Host, notUp.Unit, indent(notUp.Journal, " ")) |
| + } |
| + } |
| + if start != nil && rolledBack && !daemonRestart { |
| + if serr := start(rt.Host); serr != nil { |
| +``` |
| + |
| +Keep the rest of that block (the "could not start it" print) unchanged. Add |
| +`indent` to `rollback.go` if the package has no such helper: |
| + |
| +```go |
| +func indent(s, pre string) string { |
| + return pre + strings.ReplaceAll(strings.TrimRight(s, "\n"), "\n", "\n"+pre) |
| +} |
| +``` |
| + |
| +`adoptPlatform` (adopt.go:26, `runtime.GOOS + "-" + runtime.GOARCH`) is the |
| +box's own artifact key. |
| + |
| +- [ ] **Step 5: Run the release package tests** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -count=1` |
| +Expected: PASS, including every pre-existing restart-failure test. Those |
| +return plain errors, so `isNotUp` is false and their behaviour is unchanged. |
| + |
| +- [ ] **Step 6: Commit** |
| + |
| +```bash |
| +git add internal/release/rollback.go internal/release/rollback_test.go internal/release/adopt.go |
| +git commit -m "release: roll back a unit's whole process group and refuse the build when it does not come up" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: The settle check |
| + |
| +**Files:** |
| +- Create: `internal/host/unitsettle.go` |
| +- Test: `internal/host/unitsettle_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type unitProps struct { ActiveState, InvocationID string; MainPID, NRestarts int; ExecPath string }` |
| + - `type unitSystem interface { Show(unit string) (unitProps, error); TryRestart(unit string) error; Restart(unit string) error; ResetFailed(unit string) error; JournalTail(unit string, n int) string; Exe(pid int) (string, error) }` |
| + - `type systemdUnits struct{}` implementing it with `systemctl`/`journalctl`/`os.Readlink("/proc/<pid>/exe")` |
| + - `func parseShow(out string) unitProps` |
| + - `type settleOpts struct { Timeout, Poll, Hold time.Duration; WantDir string; Ready func() error; Now func() time.Time; Sleep func(time.Duration) }` |
| + - `func settleUnit(sys unitSystem, unit, beforeInvocation string, o settleOpts) error`: nil or `*release.UnitNotUpError` |
| + - `func unitSettleTimeout(getenv func(string) string) time.Duration` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "errors" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type fakeUnits struct { |
| + shows []unitProps // returned in order; the last repeats |
| + n int |
| + exe map[int]string |
| + journal string |
| + restarts []string |
| +} |
| + |
| +func (f *fakeUnits) Show(string) (unitProps, error) { |
| + p := f.shows[min(f.n, len(f.shows)-1)] |
| + f.n++ |
| + return p, nil |
| +} |
| +func (f *fakeUnits) TryRestart(u string) error { f.restarts = append(f.restarts, "try-restart "+u); return nil } |
| +func (f *fakeUnits) Restart(u string) error { f.restarts = append(f.restarts, "restart "+u); return nil } |
| +func (f *fakeUnits) ResetFailed(u string) error { f.restarts = append(f.restarts, "reset-failed "+u); return nil } |
| +func (f *fakeUnits) JournalTail(string, int) string { return f.journal } |
| +func (f *fakeUnits) Exe(pid int) (string, error) { |
| + if e, ok := f.exe[pid]; ok { |
| + return e, nil |
| + } |
| + return "", errors.New("no such pid") |
| +} |
| + |
| +func fakeClock() (func() time.Time, func(time.Duration)) { |
| + t := time.Unix(0, 0) |
| + return func() time.Time { return t }, func(d time.Duration) { t = t.Add(d) } |
| +} |
| + |
| +func opts(ready func() error) settleOpts { |
| + now, sleep := fakeClock() |
| + return settleOpts{Timeout: 30 * time.Second, Poll: 500 * time.Millisecond, Hold: 5 * time.Second, |
| + WantDir: "/opt/carlos/versions/new", Ready: ready, Now: now, Sleep: sleep} |
| +} |
| + |
| +func TestParseShow(t *testing.T) { |
| + p := parseShow("ActiveState=active\nInvocationID=abc\nMainPID=42\nNRestarts=0\n" + |
| + "ExecStart={ path=/opt/carlos/live/platform.example.com ; argv[]=/opt/carlos/live/platform.example.com serve ; ignore_errors=no }\n") |
| + if p.ActiveState != "active" || p.InvocationID != "abc" || p.MainPID != 42 || |
| + p.ExecPath != "/opt/carlos/live/platform.example.com" { |
| + t.Fatalf("parseShow = %+v", p) |
| + } |
| +} |
| + |
| +func TestSettleHealthyNewInvocation(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{ |
| + {ActiveState: "activating", InvocationID: "new"}, |
| + {ActiveState: "active", InvocationID: "new", MainPID: 7}, |
| + }, exe: map[int]string{7: "/opt/carlos/versions/new/console"}} |
| + if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); err != nil { |
| + t.Fatalf("settle = %v, want healthy", err) |
| + } |
| +} |
| + |
| +// systemd resets NRestarts on a manual start: a healthy unit that had |
| +// auto-restarted before must still pass (spec §2.3, codex 4). |
| +func TestSettleIgnoresThePreviousInvocationsCounter(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7, NRestarts: 0}}, |
| + exe: map[int]string{7: "/opt/carlos/versions/new/console"}} |
| + if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); err != nil { |
| + t.Fatalf("settle = %v", err) |
| + } |
| +} |
| + |
| +// Today's outage: the process exits in 55 ms and systemd cycles it. |
| +func TestSettleCatchesARestartLoop(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{ |
| + {ActiveState: "active", InvocationID: "new1", MainPID: 7}, |
| + {ActiveState: "activating", InvocationID: "new2", NRestarts: 1}, |
| + }, exe: map[int]string{7: "/opt/carlos/versions/new/console"}, journal: "usage: carlos <command> [flags]"} |
| + err := settleUnit(f, "console.service", "old", opts(func() error { return errors.New("connection refused") })) |
| + var nu *release.UnitNotUpError |
| + if !errors.As(err, &nu) || !strings.Contains(nu.Journal, "usage: carlos") { |
| + t.Fatalf("settle = %v, want UnitNotUpError carrying the journal", err) |
| + } |
| +} |
| + |
| +func TestSettleCatchesFailed(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "failed", InvocationID: "new"}}} |
| + var nu *release.UnitNotUpError |
| + if err := settleUnit(f, "console.service", "old", opts(nil)); !errors.As(err, &nu) { |
| + t.Fatalf("settle = %v, want UnitNotUpError", err) |
| + } |
| +} |
| + |
| +func TestSettleDeadlineSaysNotReady(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}}, |
| + exe: map[int]string{7: "/opt/carlos/versions/new/console"}} |
| + err := settleUnit(f, "console.service", "old", opts(func() error { return errors.New("dial: refused") })) |
| + var nu *release.UnitNotUpError |
| + if !errors.As(err, &nu) || !strings.Contains(nu.Reason, "not ready after 30s") { |
| + t.Fatalf("settle = %v, want a not-ready deadline", err) |
| + } |
| +} |
| + |
| +func TestSettleRejectsTheOldBinary(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}}, |
| + exe: map[int]string{7: "/opt/carlos/versions/old/console"}} |
| + var nu *release.UnitNotUpError |
| + if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); !errors.As(err, &nu) { |
| + t.Fatalf("settle = %v, want not-up: the process runs the old build", err) |
| + } |
| +} |
| + |
| +func TestSettleWithoutReadyHoldsFiveSeconds(t *testing.T) { |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}}, |
| + exe: map[int]string{7: "/opt/carlos/versions/new/console"}} |
| + o := opts(nil) |
| + start := o.Now() |
| + if err := settleUnit(f, "console.service", "old", o); err != nil { |
| + t.Fatalf("settle = %v", err) |
| + } |
| + if got := o.Now().Sub(start); got < 5*time.Second { |
| + t.Fatalf("settled after %v, want at least the 5s hold", got) |
| + } |
| +} |
| + |
| +func TestSettleSkipsAUnitThatWasNotRunning(t *testing.T) { |
| + // beforeInvocation "" means the caller saw the unit inactive; settleUnit |
| + // is not called at all in that case (Task 6). This pins the caller's |
| + // contract from this side: an inactive unit that stays inactive after |
| + // try-restart is "nothing started", not a failure. |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "inactive"}}} |
| + if err := settleUnit(f, "console.service", "", opts(nil)); err != nil { |
| + t.Fatalf("settle on a stopped unit = %v, want nil", err) |
| + } |
| +} |
| + |
| +func TestUnitSettleTimeoutFloor(t *testing.T) { |
| + env := func(v string) func(string) string { return func(string) string { return v } } |
| + for in, want := range map[string]time.Duration{"": 30 * time.Second, "1s": 5 * time.Second, |
| + "45s": 45 * time.Second, "junk": 30 * time.Second} { |
| + if got := unitSettleTimeout(env(in)); got != want { |
| + t.Fatalf("CARLOS_UNIT_SETTLE=%q -> %v, want %v", in, got, want) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'Settle|ParseShow' -count=1` |
| +Expected: FAIL, `undefined: unitProps`. |
| + |
| +- [ ] **Step 3: Implement `unitsettle.go`** |
| + |
| +```go |
| +// The settle check — D1 of the console release defences (spec |
| +// 2026-10-06-console-release-defences-design.md §2.2-§2.4). |
| +// |
| +// `systemctl try-restart` of a Type=simple unit returns 0 as soon as the start |
| +// job is queued, so it says nothing about whether the new build runs. On |
| +// 2026-10-06 the console's new process printed the CLI's usage and exited 2 |
| +// after 55 ms while try-restart reported success, and adoptHost's rollback |
| +// never fired. settleUnit watches the unit after the restart and returns |
| +// *release.UnitNotUpError when the new build does not come up, which drives |
| +// that rollback. |
| +package host |
| + |
| +import ( |
| + "fmt" |
| + "os" |
| + "os/exec" |
| + "regexp" |
| + "strconv" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type unitProps struct { |
| + ActiveState string |
| + InvocationID string |
| + MainPID int |
| + NRestarts int |
| + ExecPath string // ExecStart's executable path |
| +} |
| + |
| +type unitSystem interface { |
| + Show(unit string) (unitProps, error) |
| + TryRestart(unit string) error |
| + Restart(unit string) error |
| + ResetFailed(unit string) error |
| + JournalTail(unit string, n int) string |
| + Exe(pid int) (string, error) |
| +} |
| + |
| +type systemdUnits struct{} |
| + |
| +func (systemdUnits) Show(unit string) (unitProps, error) { |
| + out, err := exec.Command("systemctl", "show", unit, |
| + "-p", "ActiveState", "-p", "InvocationID", "-p", "MainPID", "-p", "NRestarts", "-p", "ExecStart").Output() |
| + if err != nil { |
| + return unitProps{}, fmt.Errorf("systemctl show %s: %w", unit, err) |
| + } |
| + return parseShow(string(out)), nil |
| +} |
| + |
| +func (systemdUnits) TryRestart(unit string) error { return systemctlTryRestartUnit(unit) } |
| +func (systemdUnits) Restart(unit string) error { return systemctlVerb("restart", unit) } |
| +func (systemdUnits) ResetFailed(unit string) error { return systemctlVerb("reset-failed", unit) } |
| + |
| +func (systemdUnits) JournalTail(unit string, n int) string { |
| + out, _ := exec.Command("journalctl", "-u", unit, "-o", "cat", "-n", strconv.Itoa(n), "--no-pager").Output() |
| + return strings.TrimSpace(string(out)) |
| +} |
| + |
| +func (systemdUnits) Exe(pid int) (string, error) { return os.Readlink(fmt.Sprintf("/proc/%d/exe", pid)) } |
| + |
| +func systemctlVerb(verb, unit string) error { |
| + out, err := exec.Command("systemctl", verb, unit).CombinedOutput() |
| + if err != nil { |
| + return fmt.Errorf("systemctl %s %s: %w: %s", verb, unit, err, strings.TrimSpace(string(out))) |
| + } |
| + return nil |
| +} |
| + |
| +var execPathRE = regexp.MustCompile(`path=([^ ;]+)`) |
| + |
| +func parseShow(out string) unitProps { |
| + var p unitProps |
| + for _, line := range strings.Split(out, "\n") { |
| + k, v, ok := strings.Cut(strings.TrimSpace(line), "=") |
| + if !ok { |
| + continue |
| + } |
| + switch k { |
| + case "ActiveState": |
| + p.ActiveState = v |
| + case "InvocationID": |
| + p.InvocationID = v |
| + case "MainPID": |
| + p.MainPID, _ = strconv.Atoi(v) |
| + case "NRestarts": |
| + p.NRestarts, _ = strconv.Atoi(v) |
| + case "ExecStart": |
| + if m := execPathRE.FindStringSubmatch(v); m != nil { |
| + p.ExecPath = m[1] |
| + } |
| + } |
| + } |
| + return p |
| +} |
| + |
| +type settleOpts struct { |
| + Timeout time.Duration // the whole window (CARLOS_UNIT_SETTLE) |
| + Poll time.Duration // 500ms |
| + Hold time.Duration // no-probe readiness: active this long in one invocation |
| + WantDir string // the new version's directory; MainPID's exe must be under it |
| + Ready func() error // nil = no upstream to probe; Hold applies |
| + Now func() time.Time |
| + Sleep func(time.Duration) |
| +} |
| + |
| +// settleUnit watches unit after a try-restart. beforeInvocation is the |
| +// InvocationID before the restart; "" means the unit was not running, and an |
| +// inactive unit is then "nothing started", not a failure. |
| +func settleUnit(sys unitSystem, unit, beforeInvocation string, o settleOpts) error { |
| + deadline := o.Now().Add(o.Timeout) |
| + var inv string // the new invocation, once seen active |
| + var activeSince time.Time |
| + var lastReady error |
| + notUp := func(reason string) error { |
| + return &release.UnitNotUpError{Unit: unit, Reason: reason, Journal: sys.JournalTail(unit, 20)} |
| + } |
| + for { |
| + p, err := sys.Show(unit) |
| + if err != nil { |
| + return notUp(err.Error()) |
| + } |
| + if beforeInvocation == "" && p.ActiveState == "inactive" { |
| + return nil |
| + } |
| + switch { |
| + case p.ActiveState == "failed": |
| + return notUp("unit entered failed state") |
| + case inv != "" && p.InvocationID != inv: |
| + return notUp("systemd restarted it (new invocation)") |
| + case inv != "" && p.NRestarts > 0: |
| + return notUp(fmt.Sprintf("systemd restarted it %d time(s)", p.NRestarts)) |
| + case inv == "" && p.ActiveState == "active" && p.InvocationID != beforeInvocation: |
| + inv, activeSince = p.InvocationID, o.Now() |
| + } |
| + if inv != "" && p.ActiveState == "active" { |
| + exe, err := sys.Exe(p.MainPID) |
| + if err == nil && o.WantDir != "" && !strings.HasPrefix(exe, strings.TrimSuffix(o.WantDir, "/")+"/") { |
| + return notUp(fmt.Sprintf("running %s, not the new build in %s", exe, o.WantDir)) |
| + } |
| + if o.Ready != nil { |
| + if lastReady = o.Ready(); lastReady == nil { |
| + return nil |
| + } |
| + } else if o.Now().Sub(activeSince) >= o.Hold { |
| + return nil |
| + } |
| + } |
| + if !o.Now().Before(deadline) { |
| + if lastReady != nil { |
| + return notUp(fmt.Sprintf("not ready after %s: %v", o.Timeout, lastReady)) |
| + } |
| + return notUp(fmt.Sprintf("not ready after %s", o.Timeout)) |
| + } |
| + o.Sleep(o.Poll) |
| + } |
| +} |
| + |
| +func unitSettleTimeout(getenv func(string) string) time.Duration { |
| + d, err := time.ParseDuration(strings.TrimSpace(getenv("CARLOS_UNIT_SETTLE"))) |
| + switch { |
| + case err != nil || d <= 0: |
| + return 30 * time.Second |
| + case d < 5*time.Second: |
| + return 5 * time.Second |
| + } |
| + return d |
| +} |
| +``` |
| + |
| +`systemctlTryRestartUnit` already exists in agent.go and is reused, not |
| +duplicated. |
| + |
| +- [ ] **Step 4: Run the tests to verify they pass** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'Settle|ParseShow' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/unitsettle.go internal/host/unitsettle_test.go |
| +git commit -m "host: settle check for a named unit after try-restart" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: Readiness on the effective upstream |
| + |
| +**Files:** |
| +- Modify: `internal/host/unitsettle.go` (append) |
| +- Test: `internal/host/unitsettle_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Produces: `func routeReady(rt registry.Route, timeout time.Duration) func() error`, which returns nil when the route has no upstream. |
| +- Consumes: `provision.Health` and `.WithDefaults()` (internal/provision/provision.go:189). |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestRouteReadyProbesTheUnixSocket(t *testing.T) { |
| + sock := filepath.Join(t.TempDir(), "s.sock") |
| + ln, err := net.Listen("unix", sock) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + if r.URL.Path == "/healthz" && r.Host == "console.example.com" { |
| + w.WriteHeader(200) |
| + return |
| + } |
| + w.WriteHeader(500) |
| + })} |
| + go srv.Serve(ln) |
| + t.Cleanup(func() { srv.Close() }) |
| + rt := registry.Route{Host: "console.example.com", Socket: sock, Addr: "127.0.0.1:1", |
| + Health: `{"path":"/healthz","status":200}`} |
| + if err := routeReady(rt, time.Second)(); err != nil { |
| + t.Fatalf("ready over the socket = %v (Addr must be ignored when Socket is set)", err) |
| + } |
| +} |
| + |
| +func TestRouteReadyWrongStatus(t *testing.T) { |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(503) })) |
| + t.Cleanup(srv.Close) |
| + rt := registry.Route{Host: "x.example.com", Addr: strings.TrimPrefix(srv.URL, "http://"), |
| + Health: `{"path":"/healthz","status":200}`} |
| + if err := routeReady(rt, time.Second)(); err == nil || !strings.Contains(err.Error(), "503") { |
| + t.Fatalf("ready = %v, want a 503 failure", err) |
| + } |
| +} |
| + |
| +func TestRouteReadyNoProbeIsAConnect(t *testing.T) { |
| + ln, err := net.Listen("tcp", "127.0.0.1:0") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { ln.Close() }) |
| + if err := routeReady(registry.Route{Host: "x", Addr: ln.Addr().String()}, time.Second)(); err != nil { |
| + t.Fatalf("connect readiness = %v", err) |
| + } |
| +} |
| + |
| +func TestRouteReadyNoUpstreamIsNil(t *testing.T) { |
| + if routeReady(registry.Route{Host: "x"}, time.Second) != nil { |
| + t.Fatal("a route with no upstream got a ready check; settle must use the hold instead") |
| + } |
| +} |
| +``` |
| + |
| +Add `net`, `net/http`, `net/http/httptest`, `path/filepath` and |
| +`github.com/carlosframework/platform/internal/registry` to the test imports. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RouteReady' -count=1` |
| +Expected: FAIL, `undefined: routeReady`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +Append to `unitsettle.go` (add `context`, `encoding/json`, `io`, `net`, |
| +`net/http`, `registry` and `provision` imports): |
| + |
| +```go |
| +// routeReady is the readiness check for rt's effective upstream (spec §2.4): |
| +// Socket wins over Addr. With a configured probe it is an HTTP GET; without |
| +// one, a connect. nil when the route has no upstream: settle then requires |
| +// the unit to stay active for its Hold. |
| +func routeReady(rt registry.Route, timeout time.Duration) func() error { |
| + network, address := "unix", rt.Socket |
| + if address == "" { |
| + network, address = "tcp", rt.Addr |
| + } |
| + if address == "" { |
| + return nil |
| + } |
| + if rt.Health == "" { |
| + return func() error { |
| + c, err := net.DialTimeout(network, address, timeout) |
| + if err != nil { |
| + return fmt.Errorf("connect %s: %w", address, err) |
| + } |
| + return c.Close() |
| + } |
| + } |
| + var h provision.Health |
| + if err := json.Unmarshal([]byte(rt.Health), &h); err != nil { |
| + return func() error { return fmt.Errorf("route health probe does not parse: %w", err) } |
| + } |
| + h = h.WithDefaults() |
| + hostHdr := h.HostHeader |
| + if hostHdr == "" { |
| + hostHdr = rt.Host |
| + } |
| + client := &http.Client{Timeout: timeout, Transport: &http.Transport{ |
| + DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { |
| + var d net.Dialer |
| + return d.DialContext(ctx, network, address) |
| + }, |
| + }} |
| + return func() error { |
| + req, err := http.NewRequest(http.MethodGet, "http://upstream"+h.Path, nil) |
| + if err != nil { |
| + return err |
| + } |
| + req.Host = hostHdr |
| + resp, err := client.Do(req) |
| + if err != nil { |
| + return fmt.Errorf("dial: %v", err) |
| + } |
| + io.Copy(io.Discard, io.LimitReader(resp.Body, 4<<10)) |
| + resp.Body.Close() |
| + if resp.StatusCode != h.Status { |
| + return fmt.Errorf("status %d (want %d)", resp.StatusCode, h.Status) |
| + } |
| + return nil |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests to verify they pass** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RouteReady|Settle' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/unitsettle.go internal/host/unitsettle_test.go |
| +git commit -m "host: readiness on a route's effective upstream, socket before addr" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: Wire the checked restart and the rollback restart |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go`: `makeRestartRoute` (3101) gets `tryRestartUnit func(rt registry.Route) error`; new `checkedUnitRestart` and `rollbackUnitRestart`; callers at 820, 1931, 3243; `RunAdopt` pass options at 3660; the agent loop options at 871 and 1500. |
| +- Modify: `internal/host/agent_test.go`: the seven `makeRestartRoute(` call sites wrap their unit fake with `unitOnly(...)`. |
| +- Test: `internal/host/unitsettle_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 4 and 5 (`unitSystem`, `settleUnit`, `settleOpts`, `routeReady`, `unitSettleTimeout`); Task 3's `release.WithRollbackRestart`. |
| +- Produces: |
| + - `func unitOnly(f func(unit string) error) func(rt registry.Route) error` |
| + - `func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string) func(rt registry.Route) error` |
| + - `func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func boundRoute() registry.Route { |
| + return registry.Route{Host: "platform.example.com", App: "console", Unit: "console.service", Addr: "127.0.0.1:1"} |
| +} |
| + |
| +func liveTo(t *testing.T, opt, host, ver string) { |
| + t.Helper() |
| + dir := filepath.Join(opt, "versions", ver) |
| + if err := os.MkdirAll(dir, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(filepath.Join(dir, "console"), []byte("x"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + os.Remove(filepath.Join(opt, "live", host)) |
| + if err := os.Symlink(filepath.Join(dir, "console"), filepath.Join(opt, "live", host)); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func TestNamedRestartBindingMismatchKeepsTheError(t *testing.T) { |
| + opt := t.TempDir() |
| + liveTo(t, opt, "platform.example.com", "new") |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "old", |
| + ExecPath: filepath.Join(opt, "live", "console.example.com")}}} |
| + failing := &failingTry{fakeUnits: f} |
| + err := checkedUnitRestart(failing, opt, func(string) string { return "" })(boundRoute()) |
| + if err == nil || !strings.Contains(err.Error(), "try-restart failed") { |
| + t.Fatalf("err = %v, want the try-restart error propagated (codex 12)", err) |
| + } |
| +} |
| + |
| +type failingTry struct{ *fakeUnits } |
| + |
| +func (failingTry) TryRestart(string) error { return errors.New("try-restart failed") } |
| + |
| +func TestNamedRestartShowFailsFallsBack(t *testing.T) { |
| + opt := t.TempDir() |
| + sys := &showFails{fakeUnits: &fakeUnits{}} |
| + if err := checkedUnitRestart(sys, opt, func(string) string { return "" })(boundRoute()); err != nil { |
| + t.Fatalf("err = %v, want today's behaviour (plain try-restart, its nil)", err) |
| + } |
| + if len(sys.restarts) != 1 || sys.restarts[0] != "try-restart console.service" { |
| + t.Fatalf("restarts = %v", sys.restarts) |
| + } |
| +} |
| + |
| +type showFails struct{ *fakeUnits } |
| + |
| +func (showFails) Show(string) (unitProps, error) { return unitProps{}, errors.New("Access denied") } |
| + |
| +func TestNamedRestartStoppedUnitIsNotChecked(t *testing.T) { |
| + opt := t.TempDir() |
| + liveTo(t, opt, "platform.example.com", "new") |
| + f := &fakeUnits{shows: []unitProps{{ActiveState: "inactive", ExecPath: filepath.Join(opt, "live", "platform.example.com")}}} |
| + if err := checkedUnitRestart(f, opt, func(string) string { return "" })(boundRoute()); err != nil { |
| + t.Fatalf("err = %v, want nil for an operator-stopped unit", err) |
| + } |
| +} |
| + |
| +func TestRollbackRestartResetsThenRestarts(t *testing.T) { |
| + opt := t.TempDir() |
| + liveTo(t, opt, "platform.example.com", "old") |
| + reg := restartTestRegistry(t) |
| + if err := reg.Upsert(boundRoute()); err != nil { |
| + t.Fatal(err) |
| + } |
| + f := &fakeUnits{shows: []unitProps{ |
| + {ActiveState: "active", InvocationID: "bad"}, |
| + {ActiveState: "active", InvocationID: "restored", MainPID: 9}, |
| + }, exe: map[int]string{9: filepath.Join(opt, "versions", "old", "console")}} |
| + rb := rollbackUnitRestart(reg, f, opt, func(string) string { return "" }) |
| + if err := rb("platform.example.com"); err != nil { |
| + t.Fatalf("rollback = %v", err) |
| + } |
| + if strings.Join(f.restarts, ",") != "reset-failed console.service,restart console.service" { |
| + t.Fatalf("restarts = %v, want reset-failed then restart (never start)", f.restarts) |
| + } |
| +} |
| +``` |
| + |
| +`restartTestRegistry` is agent_test.go's existing constructor (line 539). |
| +In `TestRollbackRestartResetsThenRestarts`, upsert `boundRoute()` with |
| +`Addr` cleared (`r := boundRoute(); r.Addr = ""`). With no upstream, |
| +`routeReady` returns nil and settle uses the hold instead of dialling. |
| +`rollbackUnitRestart` uses the real clock, so this test takes about 5 s, |
| +which is acceptable for one test. The binding test keeps `Addr` because it |
| +returns before readiness. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'NamedRestart|RollbackRestart' -count=1` |
| +Expected: FAIL, `undefined: checkedUnitRestart`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +In agent.go, change `makeRestartRoute`'s signature and its first arm: |
| + |
| +```go |
| +func makeRestartRoute(reg *registry.Registry, tryRestart func(host string), tryRestartUnit func(rt registry.Route) error, bounce func(host string) bool) func(host string) error { |
| + return func(host string) error { |
| + rt, err := reg.Get(host) |
| + if err != nil { |
| + log.Printf("adopt: registry lookup %s: %v (restarting as a unit)", host, err) |
| + } |
| + if err == nil && rt != nil && rt.Unit != "" { |
| + return tryRestartUnit(*rt) |
| + } |
| +``` |
| + |
| +(the rest unchanged). Add, next to `systemctlTryRestartUnit`: |
| + |
| +```go |
| +// unitOnly adapts a unit-name restart to makeRestartRoute's route-aware arm. |
| +func unitOnly(f func(unit string) error) func(rt registry.Route) error { |
| + return func(rt registry.Route) error { return f(rt.Unit) } |
| +} |
| + |
| +// checkedUnitRestart is the named-unit arm with D1's settle check (spec |
| +// §2.2-§2.3). The binding is verified first: the check judges a restart only |
| +// when the unit's ExecStart runs THIS route's live link. Anything it cannot |
| +// establish (show denied, wrong binding, unit stopped) falls back to today's |
| +// plain try-restart and returns that command's own error. |
| +func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string) func(rt registry.Route) error { |
| + return func(rt registry.Route) error { |
| + before, err := sys.Show(rt.Unit) |
| + if err != nil { |
| + log.Printf("adopt: %s: cannot read %s (%v); restart not checked", rt.Host, rt.Unit, err) |
| + return sys.TryRestart(rt.Unit) |
| + } |
| + link := filepath.Join(optDir, "live", rt.Host) |
| + if before.ExecPath != link { |
| + log.Printf("adopt: binding: %s runs %s, but the unit stamp is on %s; restart not checked", rt.Unit, before.ExecPath, rt.Host) |
| + return sys.TryRestart(rt.Unit) |
| + } |
| + if before.ActiveState != "active" { |
| + return sys.TryRestart(rt.Unit) // stopped by an operator: stays a no-op |
| + } |
| + want, err := filepath.EvalSymlinks(link) |
| + if err != nil { |
| + return sys.TryRestart(rt.Unit) |
| + } |
| + if err := sys.TryRestart(rt.Unit); err != nil { |
| + return err |
| + } |
| + timeout := unitSettleTimeout(getenv) |
| + err = settleUnit(sys, rt.Unit, before.InvocationID, settleOpts{ |
| + Timeout: timeout, Poll: 500 * time.Millisecond, Hold: 5 * time.Second, |
| + WantDir: filepath.Dir(want), Ready: routeReady(rt, 2*time.Second), |
| + Now: time.Now, Sleep: time.Sleep}) |
| + if err == nil { |
| + log.Printf("adopt: %s: %s settled on %s", rt.Host, rt.Unit, filepath.Dir(want)) |
| + } |
| + return err |
| + } |
| +} |
| + |
| +// rollbackUnitRestart is D1's rollback restart (release.WithRollbackRestart): |
| +// the live link already names the restored build; replace whatever is |
| +// running (a deadline failure can leave the rejected build up) and check |
| +// the restored build comes up too. |
| +func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error { |
| + return func(host string) error { |
| + rt, err := reg.Get(host) |
| + if err != nil || rt == nil || rt.Unit == "" { |
| + return nil |
| + } |
| + before, _ := sys.Show(rt.Unit) |
| + if err := sys.ResetFailed(rt.Unit); err != nil { |
| + log.Printf("adopt: %s: reset-failed %s: %v (continuing)", host, rt.Unit, err) |
| + } |
| + if err := sys.Restart(rt.Unit); err != nil { |
| + return err |
| + } |
| + want, err := filepath.EvalSymlinks(filepath.Join(optDir, "live", host)) |
| + if err != nil { |
| + return err |
| + } |
| + if err := settleUnit(sys, rt.Unit, before.InvocationID, settleOpts{ |
| + Timeout: unitSettleTimeout(getenv), Poll: 500 * time.Millisecond, Hold: 5 * time.Second, |
| + WantDir: filepath.Dir(want), Ready: routeReady(*rt, 2*time.Second), |
| + Now: time.Now, Sleep: time.Sleep}); err != nil { |
| + return fmt.Errorf("rollback did not come up either: %w", err) |
| + } |
| + return nil |
| + } |
| +} |
| +``` |
| + |
| +`settleUnit` treats `beforeInvocation == ""` with an inactive unit as nothing |
| +started. In `rollbackUnitRestart` the unit has just been `restart`ed, so |
| +`before.InvocationID` is whatever ran before. A failed unit keeps its last |
| +InvocationID, and a new one appears on restart, which is what settle keys on. |
| + |
| +Callers: |
| +- line 820 and line 1931: replace `systemctlTryRestartUnit` with |
| + `checkedUnitRestart(systemdUnits{}, optDir, os.Getenv)`, using whatever |
| + opt-dir variable is in scope there (`d.optDir` on the fast path). |
| +- line 3243 `rootRestartRoute`: the same, with the opt dir its caller uses |
| + (`RunAdopt`'s). Add an `optDir string` parameter to `rootRestartRoute` and |
| + pass it from its one caller. |
| +- `RunAdopt`'s `passOpts` (3660): append |
| + `release.WithRollbackRestart(rollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv))`. |
| +- the agent loop options at 871 and 1500: append the same option. The |
| + sandboxed agent's `reset-failed`/`restart` may be denied; that is logged |
| + and the rollback line says "could not start it", the same as today's start |
| + path there. |
| + |
| +In agent_test.go, change each `makeRestartRoute(reg, X, Y, Z)` whose third |
| +argument is a `func(string) error` to `makeRestartRoute(reg, X, unitOnly(Y), Z)`. |
| + |
| +- [ ] **Step 4: Run the host and release suites** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host ./internal/release -count=1` |
| +Expected: PASS. Per `[[cpu-budget-docs-t1]]`, run them niced and one at a time if the box is busy: `nice -n 10`. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/agent.go internal/host/agent_test.go internal/host/unitsettle_test.go |
| +git commit -m "host: named-unit restarts settle and roll back by restart (console defences D1)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: `carlos promote --retry` |
| + |
| +**Files:** |
| +- Modify: `internal/release/release.go`: `PromoteOptions` (786) adds `Retry bool`; `Promotion` (144) adds `Retry bool \`json:"retry,omitempty"\``; `PromoteOnLadder`'s repeat branch. |
| +- Modify: `cli.go` (2356: flag; 2397 and 2417: pass it) |
| +- Modify: `apimode.go:947` (`promote` payload) |
| +- Modify: `internal/console/apicli.go` (743: request field; 899: option) |
| +- Test: `internal/release/release_test.go` (append), `internal/console/apicli_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Produces: `PromoteOptions.Retry`. With Retry, a repeat is re-signed with a fresh `PromotedAt`. Without a repeat it is an error: `promote --retry renews a promotion: <channel> names <cur>, not <version>`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestPromoteRetryRenewsPromotedAt(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "console", "v1", adoptPlatform, []byte("b")) |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, _ := Channel(ctx, s, pub, "bab", "console", "canary/x") |
| + time.Sleep(1100 * time.Millisecond) // PromotedAt is truncated to the second |
| + res, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{}) |
| + if err != nil || !res.NoOp { |
| + t.Fatalf("plain repeat = %+v, %v; want the existing no-op", res, err) |
| + } |
| + same, _ := Channel(ctx, s, pub, "bab", "console", "canary/x") |
| + if !same.PromotedAt.Equal(before.PromotedAt) { |
| + t.Fatal("a plain repeat rewrote the pointer") |
| + } |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{Retry: true}); err != nil { |
| + t.Fatalf("retry: %v", err) |
| + } |
| + after, _ := Channel(ctx, s, pub, "bab", "console", "canary/x") |
| + if after.Version != "v1" || !after.PromotedAt.After(before.PromotedAt) { |
| + t.Fatalf("after retry = %+v, want v1 with a later PromotedAt", after) |
| + } |
| +} |
| + |
| +func TestPromoteRetryNeverChangesTheVersion(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, _ := testKeys(t) |
| + shipRelease(t, s, "bab", "console", "v1", adoptPlatform, []byte("a")) |
| + shipRelease(t, s, "bab", "console", "v2", adoptPlatform, []byte("b")) |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v2", PromoteOptions{Retry: true}) |
| + if err == nil || !strings.Contains(err.Error(), "renews a promotion") { |
| + t.Fatalf("retry onto a different version = %v, want a refusal", err) |
| + } |
| +} |
| +``` |
| + |
| +For the API door, add one test beside the existing `handleAPIPromote` tests in |
| +`internal/console/apicli_test.go`. Copy the nearest existing promote test's |
| +setup verbatim, send `{"version":"v1","channel":"canary/x","retry":true}` |
| +after a first promote, and assert 200 and that the stored pointer's |
| +`PromotedAt` moved. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'PromoteRetry' -count=1` |
| +Expected: FAIL, `unknown field Retry`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +`PromoteOptions`: add |
| + |
| +```go |
| + // Retry re-signs a pointer that already names this version, with a |
| + // fresh PromotedAt: the way past a box's D1 refusal (refusal.go), which |
| + // blocks one promotion, not a version. Never changes the version. |
| + Retry bool |
| +``` |
| + |
| +`Promotion`: add `Retry bool \`json:"retry,omitempty"\``. |
| + |
| +In `PromoteOnLadder`'s repeat branch, before `res.NoOp = true`, insert: |
| + |
| +```go |
| + if opt.Retry { |
| + res.From = res.To |
| + rec := Promotion{From: res.To, To: res.To, Actor: opt.Actor, Retry: true} |
| + if !time.Now().UTC().Truncate(time.Second).After(p.PromotedAt) { |
| + time.Sleep(time.Until(p.PromotedAt.Add(time.Second))) |
| + } |
| + if err := sg.promote(ctx, s, acct, app, channel, version, opt.Authorize, rec); err != nil { |
| + return res, err |
| + } |
| + return res, nil |
| + } |
| +``` |
| + |
| +and after the repeat branch's closing brace (so it is reached only when the |
| +pointer is absent or names another version), insert: |
| + |
| +```go |
| + if opt.Retry { |
| + cur := "nothing" |
| + if c, _, err := s.Get(ctx, ChannelKey(acct, app, channel)); err == nil { |
| + if p, verr := verifyPointer(pub, c, acct, app, channel); verr == nil { |
| + cur = p.Version |
| + } |
| + } |
| + return res, fmt.Errorf("promote --retry renews a promotion: %s names %s, not %s", channel, cur, version) |
| + } |
| +``` |
| + |
| +`cli.go`: next to `hotfix := fs.Bool("hotfix", ...)` (2356) add |
| + |
| +```go |
| + retry := fs.Bool("retry", false, "re-sign the channel's current promotion of this version with a fresh time, so a box that refused it tries again") |
| +``` |
| + |
| +and pass it: `release.PromoteOptions{Hotfix: *hotfix, Retry: *retry, Actor: promoteActor()}` |
| +(2417), and in API mode `sess.promote(..., *hotfix, *retry, message)`. |
| + |
| +`apimode.go:947`: add a `retry bool` parameter after `hotfix`, and a |
| +`Retry bool \`json:"retry,omitempty"\`` field in the payload. Update its one |
| +other caller if any (grep `sess.promote(` and `.promote(ctx`). |
| + |
| +`internal/console/apicli.go`: add `Retry bool \`json:"retry"\`` to the request |
| +struct (743), and `Retry: req.Retry` to the `PromoteOptions` at 899. The retry |
| +goes through the same `Authorize` and scope checks as any promote. |
| + |
| +- [ ] **Step 4: Run the tests** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release ./internal/console -run 'Promote' -count=1 && GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./...` |
| +Expected: PASS, and the build succeeds. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/release/release.go internal/release/release_test.go cli.go apimode.go internal/console/apicli.go internal/console/apicli_test.go |
| +git commit -m "promote --retry: re-sign a promotion so a box retries a refused build" |
| +``` |
| + |
| +--- |
| + |
| +### Task 8: The alert and the `status --live` line |
| + |
| +**Files:** |
| +- Create: `internal/host/refusalwatch.go` |
| +- Modify: `internal/host/alertsink.go` (add `deliverRelease`) |
| +- Modify: `internal/status/snapshot.go` (`ReleaseRefusal` type; `Probes.ReleaseRefusals`; `BoxMetrics.ReleaseRefusals`; the copy at 719 and the emptiness check at 700) |
| +- Modify: `internal/host/agent.go:2939` (feed the probe), and the metrics loop (`internal/host/metrics_loop.go:511` neighbourhood) to call the watcher each tick |
| +- Modify: `cli.go:3418` `printLiveBoxes` |
| +- Test: `internal/host/refusalwatch_test.go`, `cli_test.go` (the existing `printLiveBoxes` test, extended) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `release.ListRefusals`. |
| +- Produces: |
| + - `type ReleaseRefusal struct { Host, Version, Reason string; RefusedAt time.Time }` (status package, JSON `host`, `version`, `reason`, `refused_at`) |
| + - `type refusalWatch struct { seen map[string]time.Time }` with `func (w *refusalWatch) tick(ctx context.Context, bakeDir string, sinks *alertSinks, now time.Time) []status.ReleaseRefusal` |
| + - `func (a *alertSinks) deliverRelease(ctx context.Context, r release.Refusal)` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestRefusalWatchAlertsOncePerRecord(t *testing.T) { |
| + bake := t.TempDir() |
| + now := time.Date(2026, 10, 6, 9, 31, 0, 0, time.UTC) |
| + if err := release.WriteRefusal(bake, release.Refusal{Host: "console.example.com", App: "console", |
| + Version: "v2", RefusedAt: now.Add(-time.Minute), Reason: "exited: status=2"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + var posts []string |
| + sinks := newAlertSinks("https://hook.example", func(_ string, b []byte) error { posts = append(posts, string(b)); return nil }, "", nil, &alertCounters{}) |
| + w := &refusalWatch{} |
| + got := w.tick(context.Background(), bake, sinks, now) |
| + w.tick(context.Background(), bake, sinks, now.Add(time.Minute)) |
| + if len(posts) != 1 || !strings.Contains(posts[0], `"release_refused"`) || !strings.Contains(posts[0], "v2") { |
| + t.Fatalf("posts = %v, want exactly one release_refused alert", posts) |
| + } |
| + if len(got) != 1 || got[0].Host != "console.example.com" { |
| + t.Fatalf("status rows = %+v", got) |
| + } |
| +} |
| + |
| +func TestRefusalWatchSkipsOldRecordsAtStartup(t *testing.T) { |
| + bake := t.TempDir() |
| + now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC) |
| + if err := release.WriteRefusal(bake, release.Refusal{Host: "a.example.com", Version: "v2", |
| + RefusedAt: now.Add(-2 * time.Hour)}); err != nil { |
| + t.Fatal(err) |
| + } |
| + var posts int |
| + sinks := newAlertSinks("https://hook.example", func(string, []byte) error { posts++; return nil }, "", nil, &alertCounters{}) |
| + (&refusalWatch{}).tick(context.Background(), bake, sinks, now) |
| + if posts != 0 { |
| + t.Fatalf("an agent restart re-alerted a refusal from two hours ago") |
| + } |
| +} |
| +``` |
| + |
| +In `cli_test.go`, extend the existing `printLiveBoxes` test (grep |
| +`printLiveBoxes(` in `*_test.go`) with a box whose `Metrics.ReleaseRefusals` |
| +has one row. Assert that the output contains |
| +`refused console.example.com v2 since 2026-10-06T09:30:40Z: exited: status=2`. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RefusalWatch' -count=1` |
| +Expected: FAIL, `undefined: refusalWatch`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +`internal/status/snapshot.go`, beside `ConfigRefusal`: |
| + |
| +```go |
| +// ReleaseRefusal is a build this box rolled back because its unit did not |
| +// come up (D1, release.Refusal). PRIVATE like the rest of Box.Metrics: Host |
| +// is a real route name. |
| +type ReleaseRefusal struct { |
| + Host string `json:"host"` |
| + Version string `json:"version"` |
| + Reason string `json:"reason"` |
| + RefusedAt time.Time `json:"refused_at"` |
| +} |
| +``` |
| + |
| +Add `ReleaseRefusals []ReleaseRefusal` to `Probes` and to the metrics struct |
| +that `ConfigRefusals` is copied into at line 719, with JSON key |
| +`release_refusals,omitempty`. Copy it there the same way, and add |
| +`len(p.ReleaseRefusals) > 0 ||` to the emptiness check at 700. |
| + |
| +`internal/host/alertsink.go`, after `deliver`: |
| + |
| +```go |
| +// deliverRelease sends a D1 refusal to every configured sink. The webhook |
| +// body is the record with kind "release_refused"; SNS gets a sentence. |
| +func (a *alertSinks) deliverRelease(ctx context.Context, r release.Refusal) { |
| + if a == nil { |
| + return |
| + } |
| + if a.webhookURL != "" && a.post != nil { |
| + body, err := json.Marshal(struct { |
| + Kind string `json:"kind"` |
| + release.Refusal |
| + }{"release_refused", r}) |
| + if err == nil { |
| + if err := a.post(a.webhookURL, body); err != nil { |
| + a.counts.webhook.Add(1) |
| + log.Printf("host: alert webhook post (release refusal %s): %v", r.Host, err) |
| + } |
| + } |
| + } |
| + if a.topicARN != "" && a.pub != nil { |
| + subject := fmt.Sprintf("%s: %s did not start, rolled back", r.Host, r.Version) |
| + if len(subject) > 100 { // SNS subject limit |
| + subject = subject[:100] |
| + } |
| + msg := fmt.Sprintf("%s refused %s at %s: %s\n\nRetry with: carlos promote --retry %s <channel>", |
| + r.Host, r.Version, r.RefusedAt.Format(time.RFC3339), r.Reason, r.Version) |
| + pctx, cancel := context.WithTimeout(ctx, alertPublishTimeout) |
| + defer cancel() |
| + if _, err := a.pub.Publish(pctx, &sns.PublishInput{TopicArn: &a.topicARN, Subject: &subject, Message: &msg}); err != nil { |
| + a.counts.sns.Add(1) |
| + log.Printf("host: alert sns publish (release refusal %s): %v", r.Host, err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`internal/host/refusalwatch.go`: |
| + |
| +```go |
| +// refusalWatch turns D1's refusal records (release.Refusal, written by the |
| +// root pass) into one alert each and a status row each. It runs on the |
| +// agent's metrics tick. A record older than refusalAlertWindow when first |
| +// seen is not alerted, so an agent restart does not re-send old news. |
| +package host |
| + |
| +import ( |
| + "context" |
| + "log" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/status" |
| +) |
| + |
| +const refusalAlertWindow = 30 * time.Minute |
| + |
| +type refusalWatch struct{ seen map[string]time.Time } |
| + |
| +func (w *refusalWatch) tick(ctx context.Context, bakeDir string, sinks *alertSinks, now time.Time) []status.ReleaseRefusal { |
| + if w.seen == nil { |
| + w.seen = map[string]time.Time{} |
| + } |
| + recs, err := release.ListRefusals(bakeDir) |
| + if err != nil { |
| + log.Printf("agent: refusals: %v", err) |
| + return nil |
| + } |
| + var rows []status.ReleaseRefusal |
| + for _, r := range recs { |
| + rows = append(rows, status.ReleaseRefusal{Host: r.Host, Version: r.Version, Reason: r.Reason, RefusedAt: r.RefusedAt}) |
| + if last, ok := w.seen[r.Host]; ok && !r.RefusedAt.After(last) { |
| + continue |
| + } |
| + w.seen[r.Host] = r.RefusedAt |
| + if now.Sub(r.RefusedAt) <= refusalAlertWindow { |
| + sinks.deliverRelease(ctx, r) |
| + } |
| + } |
| + return rows |
| +} |
| +``` |
| + |
| +Wire it. In the metrics loop, keep one `refusalWatch` for the life of the |
| +loop. Each tick, call `rows := watch.tick(ctx, dataDir(), sinks, now)` and |
| +store the rows where the probe at agent.go:2939 can read them. Follow the |
| +`configRefusalProbe` pattern: a package-level `atomic.Pointer[[]status.ReleaseRefusal]` |
| +written by the tick and read by a `releaseRefusalProbe()` that fills |
| +`p.ReleaseRefusals`. |
| + |
| +`cli.go` `printLiveBoxes`: after the box's main `Fprintf`, add |
| + |
| +```go |
| + for _, r := range m.ReleaseRefusals { |
| + fmt.Fprintf(w, " refused %s %s since %s: %s\n", r.Host, r.Version, r.RefusedAt.UTC().Format(time.RFC3339), r.Reason) |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Run the tests** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host ./internal/status . -run 'RefusalWatch|LiveBoxes|Snapshot' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/refusalwatch.go internal/host/refusalwatch_test.go internal/host/alertsink.go internal/host/agent.go internal/host/metrics_loop.go internal/status/snapshot.go cli.go cli_test.go |
| +git commit -m "host: alert on a refused build and show it in status --live" |
| +``` |
| + |
| +--- |
| + |
| +### Task 9: Full gate and the README operator note |
| + |
| +**Files:** |
| +- Modify: `README.md`: under "Releasing the platform", in the release-notes area that lists operator checks, add the note below. |
| + |
| +- [ ] **Step 1: Add the note** |
| + |
| +```markdown |
| +**Named units now settle (console release defences, D1).** After adopting a |
| +build for a route with a named unit (the console), the box waits up to 30 s |
| +(`CARLOS_UNIT_SETTLE`) for the unit to stay up on the new build and pass its |
| +health probe. If it does not, the box rolls back, refuses that promotion and |
| +alerts. Retry with `carlos promote --retry <version> <channel>`. The check |
| +only runs when the unit's `ExecStart` names the stamped route's own live |
| +link; `adopt` logs `binding: … restart not checked` where it does not, and |
| +that box needs its stamp moved (2026-08-23 console design §4c). |
| +``` |
| + |
| +This is operator documentation, not member-facing copy. Keep its sentences |
| +plain and avoid em dashes in new prose (`[[copy-voice-no-em-dashes]]`). |
| + |
| +- [ ] **Step 2: Run the full gate** |
| + |
| +Run, niced and serial (`[[cpu-budget-docs-t1]]`): |
| + |
| +```bash |
| +GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false nice -n 10 go vet ./... && \ |
| +GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false nice -n 10 go test ./... -count=1 && \ |
| +GOOS=darwin GOARCH=arm64 GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./... && \ |
| +GOOS=windows GOARCH=amd64 GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./... |
| +``` |
| + |
| +Expected: all pass. `unitsettle.go` uses no linux-only syscalls (exec and |
| +`/proc` reads only), so it needs no build tag (`[[cross-compile-gate]]`). If |
| +the cross build fails on it anyway, split the `systemdUnits` methods into |
| +`unitsettle_linux.go` with a `!linux` stub returning an error from `Show`. |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add README.md |
| +git commit -m "README: operator note for the named-unit settle check" |
| +``` |