carlos / platform Public
Console release defences, Plan 1 of 4: D1 (roll back a named unit that will not come up)
> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Goal: A box that adopts a named-unit build (today: the console) checks that the new process actually comes up. If it doesn't, the box rolls back, refuses that promotion until carlos promote --retry, and alerts.
Architecture: adoptHost (internal/release/adopt.go) already rolls back when its restart callback errors. This plan makes the named-unit restart return an honest error by settling the unit after try-restart, through a small unitSystem seam over systemctl. It adds a durable refusal record in the bake directory, scoped to the routes that share one process. It also adds a --retry promote, and an alert plus a status --live line fed from the refusal records.
Tech Stack: Go (stdlib, os/exec systemctl, net/http over TCP or Unix sockets), the existing release/registry/host/status packages.
Spec: docs/superpowers/specs/2026-10-06-console-release-defences-design.md (rev 5): §2 (D1) and §7's D1 and retry tests. Plans 2 (D2), 3 (D3) and 4 (D4, process and scripts) follow; this plan stands alone.
Global Constraints
- Settle window
CARLOS_UNIT_SETTLE: default 30 s, floor 5 s, poll every 500 ms (spec §2.3). - Readiness follows the effective upstream:
Socketif set, elseAddr(spec §2.4). - No-probe readiness: the unit stays
activewithin one invocation for 5 s (spec §2.4). - A unit not running before the restart is never checked or started (spec §2.3).
- A binding mismatch (
ExecStartnotlive/<host>) skips the settle check only; thetry-restarterror still propagates (spec §2.2, codex 12). - D1 rollback runs
reset-failedthenrestart; every othermakeStartRoutecaller keepsstart(spec §2.3, codex 1). - A rollback group is the bound route plus unstamped same-app, same-account routes with the same non-empty upstream; two empty upstreams never match (spec §2.3).
- A refusal blocks while the pointer's
VersionandPromotedAtequal the record's (spec §2.5). See Task 1's note on keying. - If
systemctl showfails (for example, the sandboxed agent may not query the unit), behave exactly as today: plaintry-restart, return its error. - Gate locally with
GOFLAGS=-buildvcs=false; in this sandbox setGOCACHE=$TMPDIR/gocacheper command (AGENTS.md path is read-only here). - Commits go on branch
design-console-release-defences's implementation branch (named at execution), as the agent identity, ending with the Co-Authored-By line.
Review Focus
- A console that takes longer than the window to start (cold registry, slow disk). Expect: it is rolled back only after the full window, the window is configurable, and the log says "not ready after 30s", not "crashed". Test in Task 4 (
TestSettleDeadlineSaysNotReady). - The sandboxed agent cannot run
systemctl show. Expect: today's behaviour exactly, with no false rollback. Test in Task 6 (TestNamedRestartShowFailsFallsBack). - An operator stopped the console on purpose during an adoption. Expect: no check, no start, and it stays stopped. Test in Task 4 (
TestSettleSkipsAUnitThatWasNotRunning). - A corrupt or half-written refusal file. Expect: treated as absent and logged, never a permanent block or a panic. Test in Task 1 (
TestReadRefusalCorruptIsAbsent). - Two instances of one app on different upstreams. Expect: a rollback of one never touches the other. Test in Task 3 (
TestRollbackGroupLeavesOtherUpstreamAlone).
---
Task 1: Refusal records
Files:
- Create:
internal/release/refusal.go - Test:
internal/release/refusal_test.go
Interfaces:
- Produces: -
type Refusal struct { Schema int; Host, App, Version, SHA256, Platform string; PointerPromotedAt, RefusedAt time.Time; Reason string }(JSON keys below) -func RefusalDir(bakeDir string) string-func WriteRefusal(bakeDir string, r Refusal) error-func ReadRefusal(bakeDir, host string) (*Refusal, error)—(nil, nil)when absent or unparseable -func ListRefusals(bakeDir string) ([]Refusal, error)-func ClearRefusal(bakeDir, host string) error— nil when absent -func (r *Refusal) Blocks(p *ChannelPointer) bool
Keying note (for the reviewer): spec §2.5 keys on the artifact sha256. This plan blocks on (Version, PointerPromotedAt) and records SHA256 for display. That is equivalent: a version's manifest is immutable while a pointer names it, and reusing a name for new bytes needs a new promotion, which changes PromotedAt and clears the block. It also avoids a manifest GET on every fast-loop tick for a refused route.
- [ ] Step 1: Write the failing tests
package release
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestRefusalRoundTripAndBlocks(t *testing.T) {
dir := t.TempDir()
at := time.Date(2026, 10, 6, 9, 30, 30, 0, time.UTC)
r := Refusal{Host: "console.example.com", App: "console", Version: "v2",
SHA256: "abc", Platform: "linux-arm64", PointerPromotedAt: at,
RefusedAt: at.Add(10 * time.Second), Reason: "exited: status=2"}
if err := WriteRefusal(dir, r); err != nil {
t.Fatal(err)
}
got, err := ReadRefusal(dir, "console.example.com")
if err != nil || got == nil {
t.Fatalf("ReadRefusal = %+v, %v", got, err)
}
if got.Schema != 1 || got.Version != "v2" || !got.PointerPromotedAt.Equal(at) {
t.Fatalf("round trip = %+v", got)
}
if !got.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at}) {
t.Fatal("the same promotion is not blocked")
}
if got.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at.Add(time.Second)}) {
t.Fatal("a re-signed pointer (retry) is still blocked")
}
if got.Blocks(&ChannelPointer{Version: "v3", PromotedAt: at}) {
t.Fatal("a different version is blocked")
}
var nilR *Refusal
if nilR.Blocks(&ChannelPointer{Version: "v2", PromotedAt: at}) {
t.Fatal("a nil refusal blocks")
}
}
func TestReadRefusalAbsentIsNil(t *testing.T) {
got, err := ReadRefusal(t.TempDir(), "nobody.example.com")
if got != nil || err != nil {
t.Fatalf("absent = %+v, %v; want nil, nil", got, err)
}
}
func TestReadRefusalCorruptIsAbsent(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(RefusalDir(dir), "console.example.com.json")
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(`{"schema":1,"host":`), 0o644); err != nil {
t.Fatal(err)
}
got, err := ReadRefusal(dir, "console.example.com")
if got != nil || err != nil {
t.Fatalf("corrupt = %+v, %v; want nil, nil (a torn file must never block)", got, err)
}
}
func TestRefusalRejectsAPathInTheHost(t *testing.T) {
if err := WriteRefusal(t.TempDir(), Refusal{Host: "../etc/x"}); err == nil {
t.Fatal("a host with a path separator was written")
}
}
func TestListAndClearRefusals(t *testing.T) {
dir := t.TempDir()
for _, h := range []string{"a.example.com", "b.example.com"} {
if err := WriteRefusal(dir, Refusal{Host: h, Version: "v2"}); err != nil {
t.Fatal(err)
}
}
if err := ClearRefusal(dir, "a.example.com"); err != nil {
t.Fatal(err)
}
if err := ClearRefusal(dir, "never.example.com"); err != nil {
t.Fatalf("clearing an absent record: %v", err)
}
all, err := ListRefusals(dir)
if err != nil || len(all) != 1 || all[0].Host != "b.example.com" {
t.Fatalf("ListRefusals = %+v, %v", all, err)
}
}- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'Refusal' -count=1 Expected: FAIL, undefined: Refusal.
- [ ] Step 3: Implement
// Refusal records — D1 of the console release defences (spec
// 2026-10-06-console-release-defences-design.md §2.5).
//
// When a named unit does not come up on a new build, the root pass rolls it
// back and writes one of these per host in the rolled-back process group.
// Every later pass, the agent's fast loop and root alike, reads it before
// staging and skips the promotion it names. Without it the very next pass
// would adopt the same build and bounce the service again.
//
// The record blocks a PROMOTION, not a version name: it matches while the
// channel pointer still has the version and PromotedAt it had when the build
// was refused. `carlos promote --retry` re-signs the pointer with a fresh
// PromotedAt, and so does any move to another version; either clears it.
//
// Root writes these under the bake directory, which the sandboxed agent can
// read. A file that does not parse reads as absent: a torn write must never
// become a permanent block.
package release
import (
"encoding/json"
"errors"
"fmt"
"log"
"os"
"path/filepath"
"sort"
"strings"
"time"
)
const refusalSchema = 1
type Refusal struct {
Schema int `json:"schema"`
Host string `json:"host"`
App string `json:"app"`
Version string `json:"version"`
SHA256 string `json:"sha256,omitempty"`
Platform string `json:"platform,omitempty"`
PointerPromotedAt time.Time `json:"pointer_promoted_at"`
RefusedAt time.Time `json:"refused_at"`
Reason string `json:"reason"`
}
func RefusalDir(bakeDir string) string { return filepath.Join(bakeDir, "refusals") }
func refusalPath(bakeDir, host string) (string, error) {
if host == "" || strings.ContainsAny(host, `/\`) || host == "." || host == ".." {
return "", fmt.Errorf("refusal: invalid host %q", host)
}
return filepath.Join(RefusalDir(bakeDir), host+".json"), nil
}
// WriteRefusal writes atomically (temp file + rename) and world-readable,
// because the reader is the sandboxed agent.
func WriteRefusal(bakeDir string, r Refusal) error {
p, err := refusalPath(bakeDir, r.Host)
if err != nil {
return err
}
r.Schema = refusalSchema
b, err := json.MarshalIndent(r, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(p), ".refusal-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(append(b, '\n')); err != nil {
tmp.Close()
return err
}
if err := tmp.Chmod(0o644); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), p)
}
func ReadRefusal(bakeDir, host string) (*Refusal, error) {
p, err := refusalPath(bakeDir, host)
if err != nil {
return nil, err
}
b, err := os.ReadFile(p)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
var r Refusal
if err := json.Unmarshal(b, &r); err != nil || r.Schema != refusalSchema {
log.Printf("refusal: %s does not parse (%v), treating it as absent", p, err)
return nil, nil
}
return &r, nil
}
func ListRefusals(bakeDir string) ([]Refusal, error) {
ents, err := os.ReadDir(RefusalDir(bakeDir))
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
var out []Refusal
for _, e := range ents {
name := e.Name()
if e.IsDir() || !strings.HasSuffix(name, ".json") || strings.HasPrefix(name, ".") {
continue
}
r, err := ReadRefusal(bakeDir, strings.TrimSuffix(name, ".json"))
if err != nil || r == nil {
continue
}
out = append(out, *r)
}
sort.Slice(out, func(i, j int) bool { return out[i].Host < out[j].Host })
return out, nil
}
func ClearRefusal(bakeDir, host string) error {
p, err := refusalPath(bakeDir, host)
if err != nil {
return err
}
if err := os.Remove(p); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
return nil
}
// Blocks reports whether this record still refuses the promotion p names.
func (r *Refusal) Blocks(p *ChannelPointer) bool {
return r != nil && p != nil && r.Version == p.Version && r.PointerPromotedAt.Equal(p.PromotedAt)
}- [ ] Step 4: Run the tests to verify they pass
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'Refusal' -count=1 Expected: PASS.
- [ ] Step 5: Commit
git add internal/release/refusal.go internal/release/refusal_test.go
git commit -m "release: refusal records for builds a box rolled back (console defences D1)"---
Task 2: adoptHost honours a refusal before it stages or relays
Files:
- Modify:
internal/release/adopt.go(inadoptHost, after thepoisoned[ver]check and beforeensureStaged; and after the finalSetVersionssuccess) - Test:
internal/release/adopt_test.go(append)
Interfaces:
- Consumes: Task 1's
ReadRefusal,ClearRefusal,(*Refusal).Blocks;passConfig.bakeDir(set byBakeDir(dir)). - Produces: nothing new; the printed line
refused: <ver> did not come up on this box at <time> (<reason>); retry with: carlos promote --retry <ver> <channel>is what Task 8 and operators grep.
- [ ] Step 1: Write the failing tests
// TestRefusedPromotionIsNeitherStagedNorRelayed: a refusal naming the current
// pointer stops adoptHost before ensureStaged, so the agent never relays a
// root pass for it and nothing restarts.
func TestRefusedPromotionIsNeitherStagedNorRelayed(t *testing.T) {
host := "console.example.com"
ctx, reg, s, pub, opt := startUnitFixture(t, host)
bake := t.TempDir()
ptr, err := Channel(ctx, s, pub, "bab", "console", "canary/x")
if err != nil {
t.Fatal(err)
}
if err := WriteRefusal(bake, Refusal{Host: host, App: "console", Version: ptr.Version,
PointerPromotedAt: ptr.PromotedAt, Reason: "exited: status=2"}); err != nil {
t.Fatal(err)
}
restart := func(h string) error { t.Fatalf("restart consulted for refused %s", h); return nil }
relayed := false
var out bytes.Buffer
withCapturedStdout(t, &out, func() {
if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake),
OnStageBlocked(func(string, string, error) { relayed = true })); err != nil {
t.Fatalf("Pass: %v", err)
}
})
if v, _ := LiveVersion(opt, host); v != "old" {
t.Fatalf("live = %q, want old", v)
}
if relayed {
t.Fatal("a refused promotion was relayed to root")
}
if !strings.Contains(out.String(), "promote --retry") {
t.Fatalf("printed %q, want the retry hint", out.String())
}
}
// TestRePromotionClearsTheBlock: a pointer re-signed for the same version
// (what promote --retry does) is adopted, and the stale record is removed.
func TestRePromotionClearsTheBlock(t *testing.T) {
host := "console.example.com"
ctx, reg, s, pub, opt := startUnitFixture(t, host)
bake := t.TempDir()
if err := WriteRefusal(bake, Refusal{Host: host, App: "console", Version: "new",
PointerPromotedAt: time.Unix(1, 0).UTC(), Reason: "older promotion"}); err != nil {
t.Fatal(err)
}
if err := Pass(ctx, reg, s, pub, opt, func(string) error { return nil }, BakeDir(bake)); err != nil {
t.Fatalf("Pass: %v", err)
}
if v, _ := LiveVersion(opt, host); v != "new" {
t.Fatalf("live = %q, want new (the record names an older promotion)", v)
}
if r, _ := ReadRefusal(bake, host); r != nil {
t.Fatalf("stale refusal survived a successful adoption: %+v", r)
}
}Channel (internal/release/release.go:1047) is the verified pointer read.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'TestRefusedPromotion|TestRePromotionClears' -count=1 Expected: FAIL. The first test's restart fatals, and the second still finds the record.
- [ ] Step 3: Implement the gate
In adoptHost, immediately after:
if poisoned[ver] {
fmt.Printf(" %s: skipping %s (failed on another host this pass)\n", rt.Host, ver)
return
}insert:
// D1's refusal (refusal.go): this box rolled this exact promotion back
// because the unit did not come up. Checked before ensureStaged so a
// refused build is neither downloaded nor relayed to a root pass.
if cfg.bakeDir != "" {
if r, err := ReadRefusal(cfg.bakeDir, rt.Host); err != nil {
fmt.Printf(" %s: reading refusal record: %v\n", rt.Host, err)
} else if r.Blocks(ptr) {
fmt.Printf(" %s: refused: %s did not come up on this box at %s (%s); retry with: carlos promote --retry %s %s\n",
rt.Host, ver, r.RefusedAt.Format(time.RFC3339), r.Reason, ver, channel)
return
}
}and immediately after the final successful reg.SetVersions(rt.Host, ver, cur) (the one followed by the updated %s -> %s print), insert:
// A record left from an older promotion is stale once a build is live.
// Best-effort: the agent cannot remove root's file, the root pass can.
if cfg.bakeDir != "" {
_ = ClearRefusal(cfg.bakeDir, rt.Host)
}- [ ] Step 4: Run the release package tests
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -count=1 Expected: PASS (new tests and every existing adopt test).
- [ ] Step 5: Commit
git add internal/release/adopt.go internal/release/adopt_test.go
git commit -m "release: adoptHost skips a refused promotion before staging or relaying"---
Task 3: The process-group rollback when a unit does not come up
Files:
- Create:
internal/release/rollback.go - Modify:
internal/release/adopt.go(passConfig, newWithRollbackRestart, thererr != nilbranch ofadoptHost) - Test:
internal/release/rollback_test.go
Interfaces:
- Produces: -
type UnitNotUpError struct { Unit, Reason, Journal string }withError() string. The host package's restart returns this (Task 6) and adoptHost tests it witherrors.As. -func upstream(rt registry.Route) string:rt.Socketif non-empty, elsert.Addr. -func processGroup(reg *registry.Registry, bound registry.Route) []registry.Route: excludesbound. -func WithRollbackRestart(fn func(host string) error) PassOption: replacesstartUniton the D1 path only. - Consumes: Task 1's
WriteRefusal;LiveVersion,PointLive,RouteAccount,reg.List,reg.SetVersions.
- [ ] Step 1: Write the failing tests
package release
import (
"bytes"
"testing"
"github.com/carlosframework/platform/internal/registry"
)
func groupFixtureRoute(t *testing.T, reg *registry.Registry, host, socket, addr, unit string) {
t.Helper()
if err := reg.Upsert(registry.Route{Host: host, Kind: "instance", App: "console",
Channel: "canary/x", Socket: socket, Addr: addr, Unit: unit}); err != nil {
t.Fatalf("upsert %s: %v", host, err)
}
}
func TestProcessGroupKeysOnTheEffectiveUpstream(t *testing.T) {
reg := testRegistry(t)
groupFixtureRoute(t, reg, "platform.example.com", "", "127.0.0.1:8090", "console.service")
groupFixtureRoute(t, reg, "console.example.com", "", "127.0.0.1:8090", "")
groupFixtureRoute(t, reg, "other.example.com", "", "127.0.0.1:9000", "")
groupFixtureRoute(t, reg, "sock.example.com", "/run/x.sock", "127.0.0.1:8090", "")
bound, _ := reg.Get("platform.example.com")
got := processGroup(reg, *bound)
if len(got) != 1 || got[0].Host != "console.example.com" {
t.Fatalf("group = %+v, want only console.example.com (socket wins over addr; 9000 is another process)", got)
}
}
func TestProcessGroupNeverMatchesEmptyUpstreams(t *testing.T) {
reg := testRegistry(t)
groupFixtureRoute(t, reg, "a.example.com", "", "", "a.service")
groupFixtureRoute(t, reg, "b.example.com", "", "", "")
bound, _ := reg.Get("a.example.com")
if got := processGroup(reg, *bound); len(got) != 0 {
t.Fatalf("group = %+v, want none: two empty upstreams are not one process", got)
}
}
// The flagship shape: three hosts, one process, the unit runs the LAST
// host's link. Hosts before it were repointed and stamped; the bound host's
// restart reports the unit did not come up.
func TestNotUpRollsTheWholeGroupBackAndRefusesEveryHost(t *testing.T) {
ctx, reg, s, pub, opt := startUnitFixture(t, "console.example.com")
if err := reg.SetUnit("console.example.com", ""); err != nil {
t.Fatal(err)
}
groupFixtureRoute(t, reg, "platform.example.com", "", "127.0.0.1:8090", "console.service")
if err := reg.SetUpstreams("console.example.com", "127.0.0.1:8090", "", ""); err != nil {
t.Fatal(err)
}
if err := PointLive(opt, "platform.example.com", "bab", "old", "console"); err != nil {
t.Fatal(err)
}
bake := t.TempDir()
restart := func(h string) error {
if h == "platform.example.com" {
return &UnitNotUpError{Unit: "console.service", Reason: "exited: status=2"}
}
return nil
}
var rolledBackHost string
rb := func(h string) error { rolledBackHost = h; return nil }
var out bytes.Buffer
withCapturedStdout(t, &out, func() {
if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake), WithRollbackRestart(rb)); err != nil {
t.Fatalf("Pass: %v", err)
}
})
for _, h := range []string{"console.example.com", "platform.example.com"} {
if v, _ := LiveVersion(opt, h); v != "old" {
t.Fatalf("%s live = %q, want old", h, v)
}
rt, _ := reg.Get(h)
if rt.Version == "new" {
t.Fatalf("%s still stamped new: %+v", h, rt)
}
if r, _ := ReadRefusal(bake, h); r == nil || r.Version != "new" {
t.Fatalf("%s refusal = %+v, want one naming new", h, r)
}
}
if rolledBackHost != "platform.example.com" {
t.Fatalf("rollback restart ran for %q, want the bound host", rolledBackHost)
}
}
func TestRollbackGroupLeavesOtherUpstreamAlone(t *testing.T) {
ctx, reg, s, pub, opt := startUnitFixture(t, "console.example.com")
if err := reg.SetUnit("console.example.com", "console.service"); err != nil {
t.Fatal(err)
}
if err := reg.SetUpstreams("console.example.com", "127.0.0.1:8090", "", ""); err != nil {
t.Fatal(err)
}
groupFixtureRoute(t, reg, "zz-other.example.com", "", "127.0.0.1:9000", "")
if err := StageArtifact(opt, "bab", "new", "console", []byte("bad build")); err != nil {
t.Fatal(err)
}
if err := PointLive(opt, "zz-other.example.com", "bab", "new", "console"); err != nil {
t.Fatal(err)
}
restart := func(h string) error {
if h == "console.example.com" {
return &UnitNotUpError{Unit: "console.service", Reason: "not ready after 30s"}
}
return nil
}
bake := t.TempDir()
if err := Pass(ctx, reg, s, pub, opt, restart, BakeDir(bake), WithRollbackRestart(func(string) error { return nil })); err != nil {
t.Fatalf("Pass: %v", err)
}
if v, _ := LiveVersion(opt, "zz-other.example.com"); v != "new" {
t.Fatalf("another process's route was rolled back to %q", v)
}
if r, _ := ReadRefusal(bake, "zz-other.example.com"); r != nil {
t.Fatalf("another process's route was refused: %+v", r)
}
}Check the fixture's assumptions before running: reg.SetUnit and reg.SetUpstreams(host, addr, addrs, health) exist at internal/registry/registry.go:1231 and :1315, and startUnitFixture's route starts with no Unit.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'ProcessGroup|NotUpRolls|RollbackGroup' -count=1 Expected: FAIL, undefined: UnitNotUpError.
- [ ] Step 3: Implement
rollback.go
// The D1 rollback (spec 2026-10-06-console-release-defences-design.md §2.3):
// what adoptHost does when a named unit's restart reports that the new build
// did not come up, as opposed to systemctl itself failing.
package release
import (
"fmt"
"strings"
"github.com/carlosframework/platform/internal/registry"
)
// UnitNotUpError is a named unit that was restarted onto a new build and did
// not settle: it failed, restarted again, or was not ready by the deadline.
// The host package's settle check returns it (internal/host/unitsettle.go).
type UnitNotUpError struct {
Unit string
Reason string
Journal string // the unit's last journal lines, for the rollback line and the alert
}
func (e *UnitNotUpError) Error() string {
return fmt.Sprintf("%s did not come up: %s", e.Unit, e.Reason)
}
// upstream is a route's effective upstream: Socket wins over Addr, as at the
// edge (registry.Route.Socket's doc).
func upstream(rt registry.Route) string {
if rt.Socket != "" {
return rt.Socket
}
return rt.Addr
}
// processGroup is every OTHER route served by bound's process: unstamped,
// same account and app, same non-empty upstream. A bound route with no
// upstream is a group of one.
func processGroup(reg *registry.Registry, bound registry.Route) []registry.Route {
up := upstream(bound)
if up == "" {
return nil
}
all, err := reg.List()
if err != nil {
fmt.Printf(" %s: listing routes for the rollback group: %v (rolling back this host only)\n", bound.Host, err)
return nil
}
var out []registry.Route
for _, r := range all {
if r.Host == bound.Host || r.Unit != "" || r.App != bound.App ||
RouteAccount(r) != RouteAccount(bound) || upstream(r) != up {
continue
}
out = append(out, r)
}
return out
}
// rollbackGroup puts every group route whose link names the rejected version
// onto restored (what bound now runs), stamps it to match, and writes a
// refusal for bound and every group host.
func rollbackGroup(reg *registry.Registry, optDir string, cfg passConfig, bound registry.Route,
ptr *ChannelPointer, rejected, restored, sha, platform string, nu *UnitNotUpError, now time.Time) {
hosts := []string{bound.Host}
for _, g := range processGroup(reg, bound) {
hosts = append(hosts, g.Host)
v, err := LiveVersion(optDir, g.Host)
if err != nil || v != rejected || restored == "" {
continue
}
if err := PointLive(optDir, g.Host, RouteAccount(g), restored, g.App); err != nil {
fmt.Printf(" %s: could not roll the symlink back to %s with its process: %v\n", g.Host, restored, err)
continue
}
if err := reg.SetVersions(g.Host, restored, g.PrevVersion); err != nil {
fmt.Printf(" %s: rolled back to %s but recording the version failed: %v\n", g.Host, restored, err)
}
}
if cfg.bakeDir == "" {
return
}
reason := nu.Reason
if j := strings.TrimSpace(nu.Journal); j != "" {
lines := strings.Split(j, "\n")
reason += " — " + lines[len(lines)-1]
}
for _, h := range hosts {
r := Refusal{Host: h, App: bound.App, Version: rejected, SHA256: sha, Platform: platform,
PointerPromotedAt: ptr.PromotedAt, RefusedAt: now.UTC(), Reason: reason}
if err := WriteRefusal(cfg.bakeDir, r); err != nil {
fmt.Printf(" %s: writing refusal record: %v\n", h, err)
}
}
}now is the now time.Time that adoptHost already receives. Add "time" to rollback.go's imports.
- [ ] Step 4: Wire it into
adoptHostandpassConfig
In passConfig (adopt.go:34) add:
// rollbackRestart replaces startUnit on D1's path only — a named unit
// whose restart returned *UnitNotUpError. It must REPLACE whatever is
// running (reset-failed + restart), because a deadline failure can leave
// the rejected build running unready and `start` is a no-op on it.
rollbackRestart func(host string) errorBelow WithStartUnit, add:
// WithRollbackRestart sets D1's rollback restart (see passConfig).
func WithRollbackRestart(fn func(host string) error) PassOption {
return func(c *passConfig) { c.rollbackRestart = fn }
}In the rerr != nil branch of adoptHost, right after poisoned[ver] = true, record whether this is D1's case:
var notUp *UnitNotUpError
isNotUp := errors.As(rerr, ¬Up)Replace the final start block
if cfg.startUnit != nil && rolledBack && !daemonRestart {
if serr := cfg.startUnit(rt.Host); serr != nil {with:
start := cfg.startUnit
if isNotUp && cfg.rollbackRestart != nil {
start = cfg.rollbackRestart
}
if isNotUp {
sha := ""
if m != nil {
sha = m.Artifacts[adoptPlatform].SHA256
}
rollbackGroup(reg, optDir, cfg, rt, ptr, ver, cur, sha, adoptPlatform, notUp, now)
if notUp.Journal != "" {
fmt.Printf(" %s: %s's last lines:\n%s\n", rt.Host, notUp.Unit, indent(notUp.Journal, " "))
}
}
if start != nil && rolledBack && !daemonRestart {
if serr := start(rt.Host); serr != nil {Keep the rest of that block (the "could not start it" print) unchanged. Add indent to rollback.go if the package has no such helper:
func indent(s, pre string) string {
return pre + strings.ReplaceAll(strings.TrimRight(s, "\n"), "\n", "\n"+pre)
}adoptPlatform (adopt.go:26, runtime.GOOS + "-" + runtime.GOARCH) is the box's own artifact key.
- [ ] Step 5: Run the release package tests
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -count=1 Expected: PASS, including every pre-existing restart-failure test. Those return plain errors, so isNotUp is false and their behaviour is unchanged.
- [ ] Step 6: Commit
git add internal/release/rollback.go internal/release/rollback_test.go internal/release/adopt.go
git commit -m "release: roll back a unit's whole process group and refuse the build when it does not come up"---
Task 4: The settle check
Files:
- Create:
internal/host/unitsettle.go - Test:
internal/host/unitsettle_test.go
Interfaces:
- Produces: -
type unitProps struct { ActiveState, InvocationID string; MainPID, NRestarts int; ExecPath string }-type unitSystem interface { Show(unit string) (unitProps, error); TryRestart(unit string) error; Restart(unit string) error; ResetFailed(unit string) error; JournalTail(unit string, n int) string; Exe(pid int) (string, error) }-type systemdUnits struct{}implementing it withsystemctl/journalctl/os.Readlink("/proc/<pid>/exe")-func parseShow(out string) unitProps-type settleOpts struct { Timeout, Poll, Hold time.Duration; WantDir string; Ready func() error; Now func() time.Time; Sleep func(time.Duration) }-func settleUnit(sys unitSystem, unit, beforeInvocation string, o settleOpts) error: nil or*release.UnitNotUpError-func unitSettleTimeout(getenv func(string) string) time.Duration
- [ ] Step 1: Write the failing tests
package host
import (
"errors"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/release"
)
type fakeUnits struct {
shows []unitProps // returned in order; the last repeats
n int
exe map[int]string
journal string
restarts []string
}
func (f *fakeUnits) Show(string) (unitProps, error) {
p := f.shows[min(f.n, len(f.shows)-1)]
f.n++
return p, nil
}
func (f *fakeUnits) TryRestart(u string) error { f.restarts = append(f.restarts, "try-restart "+u); return nil }
func (f *fakeUnits) Restart(u string) error { f.restarts = append(f.restarts, "restart "+u); return nil }
func (f *fakeUnits) ResetFailed(u string) error { f.restarts = append(f.restarts, "reset-failed "+u); return nil }
func (f *fakeUnits) JournalTail(string, int) string { return f.journal }
func (f *fakeUnits) Exe(pid int) (string, error) {
if e, ok := f.exe[pid]; ok {
return e, nil
}
return "", errors.New("no such pid")
}
// settleClock, not fakeClock: stage_relay_test.go already declares a
// fakeClock type in this package.
func settleClock() (func() time.Time, func(time.Duration)) {
t := time.Unix(0, 0)
return func() time.Time { return t }, func(d time.Duration) { t = t.Add(d) }
}
func opts(ready func() error) settleOpts {
now, sleep := settleClock()
return settleOpts{Timeout: 30 * time.Second, Poll: 500 * time.Millisecond, Hold: 5 * time.Second,
WantDir: "/opt/carlos/versions/new", Ready: ready, Now: now, Sleep: sleep}
}
func TestParseShow(t *testing.T) {
p := parseShow("ActiveState=active\nInvocationID=abc\nMainPID=42\nNRestarts=0\n" +
"ExecStart={ path=/opt/carlos/live/platform.example.com ; argv[]=/opt/carlos/live/platform.example.com serve ; ignore_errors=no }\n")
if p.ActiveState != "active" || p.InvocationID != "abc" || p.MainPID != 42 ||
p.ExecPath != "/opt/carlos/live/platform.example.com" {
t.Fatalf("parseShow = %+v", p)
}
}
func TestSettleHealthyNewInvocation(t *testing.T) {
f := &fakeUnits{shows: []unitProps{
{ActiveState: "activating", InvocationID: "new"},
{ActiveState: "active", InvocationID: "new", MainPID: 7},
}, exe: map[int]string{7: "/opt/carlos/versions/new/console"}}
if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); err != nil {
t.Fatalf("settle = %v, want healthy", err)
}
}
// systemd resets NRestarts on a manual start: a healthy unit that had
// auto-restarted before must still pass (spec §2.3, codex 4).
func TestSettleIgnoresThePreviousInvocationsCounter(t *testing.T) {
f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7, NRestarts: 0}},
exe: map[int]string{7: "/opt/carlos/versions/new/console"}}
if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); err != nil {
t.Fatalf("settle = %v", err)
}
}
// Today's outage: the process exits in 55 ms and systemd cycles it.
func TestSettleCatchesARestartLoop(t *testing.T) {
f := &fakeUnits{shows: []unitProps{
{ActiveState: "active", InvocationID: "new1", MainPID: 7},
{ActiveState: "activating", InvocationID: "new2", NRestarts: 1},
}, exe: map[int]string{7: "/opt/carlos/versions/new/console"}, journal: "usage: carlos <command> [flags]"}
err := settleUnit(f, "console.service", "old", opts(func() error { return errors.New("connection refused") }))
var nu *release.UnitNotUpError
if !errors.As(err, &nu) || !strings.Contains(nu.Journal, "usage: carlos") {
t.Fatalf("settle = %v, want UnitNotUpError carrying the journal", err)
}
}
func TestSettleCatchesFailed(t *testing.T) {
f := &fakeUnits{shows: []unitProps{{ActiveState: "failed", InvocationID: "new"}}}
var nu *release.UnitNotUpError
if err := settleUnit(f, "console.service", "old", opts(nil)); !errors.As(err, &nu) {
t.Fatalf("settle = %v, want UnitNotUpError", err)
}
}
func TestSettleDeadlineSaysNotReady(t *testing.T) {
f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}},
exe: map[int]string{7: "/opt/carlos/versions/new/console"}}
err := settleUnit(f, "console.service", "old", opts(func() error { return errors.New("dial: refused") }))
var nu *release.UnitNotUpError
if !errors.As(err, &nu) || !strings.Contains(nu.Reason, "not ready after 30s") {
t.Fatalf("settle = %v, want a not-ready deadline", err)
}
}
func TestSettleRejectsTheOldBinary(t *testing.T) {
f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}},
exe: map[int]string{7: "/opt/carlos/versions/old/console"}}
var nu *release.UnitNotUpError
if err := settleUnit(f, "console.service", "old", opts(func() error { return nil })); !errors.As(err, &nu) {
t.Fatalf("settle = %v, want not-up: the process runs the old build", err)
}
}
func TestSettleWithoutReadyHoldsFiveSeconds(t *testing.T) {
f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7}},
exe: map[int]string{7: "/opt/carlos/versions/new/console"}}
o := opts(nil)
start := o.Now()
if err := settleUnit(f, "console.service", "old", o); err != nil {
t.Fatalf("settle = %v", err)
}
if got := o.Now().Sub(start); got < 5*time.Second {
t.Fatalf("settled after %v, want at least the 5s hold", got)
}
}
func TestSettleSkipsAUnitThatWasNotRunning(t *testing.T) {
// beforeInvocation "" means the caller saw the unit inactive; settleUnit
// is not called at all in that case (Task 6). This pins the caller's
// contract from this side: an inactive unit that stays inactive after
// try-restart is "nothing started", not a failure.
f := &fakeUnits{shows: []unitProps{{ActiveState: "inactive"}}}
if err := settleUnit(f, "console.service", "", opts(nil)); err != nil {
t.Fatalf("settle on a stopped unit = %v, want nil", err)
}
}
func TestUnitSettleTimeoutFloor(t *testing.T) {
env := func(v string) func(string) string { return func(string) string { return v } }
for in, want := range map[string]time.Duration{"": 30 * time.Second, "1s": 5 * time.Second,
"45s": 45 * time.Second, "junk": 30 * time.Second} {
if got := unitSettleTimeout(env(in)); got != want {
t.Fatalf("CARLOS_UNIT_SETTLE=%q -> %v, want %v", in, got, want)
}
}
}- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'Settle|ParseShow' -count=1 Expected: FAIL, undefined: unitProps.
- [ ] Step 3: Implement
unitsettle.go
// The settle check — D1 of the console release defences (spec
// 2026-10-06-console-release-defences-design.md §2.2-§2.4).
//
// `systemctl try-restart` of a Type=simple unit returns 0 as soon as the start
// job is queued, so it says nothing about whether the new build runs. On
// 2026-10-06 the console's new process printed the CLI's usage and exited 2
// after 55 ms while try-restart reported success, and adoptHost's rollback
// never fired. settleUnit watches the unit after the restart and returns
// *release.UnitNotUpError when the new build does not come up, which drives
// that rollback.
package host
import (
"fmt"
"os"
"os/exec"
"regexp"
"strconv"
"strings"
"time"
"github.com/carlosframework/platform/internal/release"
)
type unitProps struct {
ActiveState string
InvocationID string
MainPID int
NRestarts int
ExecPath string // ExecStart's executable path
}
type unitSystem interface {
Show(unit string) (unitProps, error)
TryRestart(unit string) error
Restart(unit string) error
ResetFailed(unit string) error
JournalTail(unit string, n int) string
Exe(pid int) (string, error)
}
type systemdUnits struct{}
func (systemdUnits) Show(unit string) (unitProps, error) {
out, err := exec.Command("systemctl", "show", unit,
"-p", "ActiveState", "-p", "InvocationID", "-p", "MainPID", "-p", "NRestarts", "-p", "ExecStart").Output()
if err != nil {
return unitProps{}, fmt.Errorf("systemctl show %s: %w", unit, err)
}
return parseShow(string(out)), nil
}
func (systemdUnits) TryRestart(unit string) error { return systemctlTryRestartUnit(unit) }
func (systemdUnits) Restart(unit string) error { return systemctlVerb("restart", unit) }
func (systemdUnits) ResetFailed(unit string) error { return systemctlVerb("reset-failed", unit) }
func (systemdUnits) JournalTail(unit string, n int) string {
out, _ := exec.Command("journalctl", "-u", unit, "-o", "cat", "-n", strconv.Itoa(n), "--no-pager").Output()
return strings.TrimSpace(string(out))
}
func (systemdUnits) Exe(pid int) (string, error) { return os.Readlink(fmt.Sprintf("/proc/%d/exe", pid)) }
func systemctlVerb(verb, unit string) error {
out, err := exec.Command("systemctl", verb, unit).CombinedOutput()
if err != nil {
return fmt.Errorf("systemctl %s %s: %w: %s", verb, unit, err, strings.TrimSpace(string(out)))
}
return nil
}
var execPathRE = regexp.MustCompile(`path=([^ ;]+)`)
func parseShow(out string) unitProps {
var p unitProps
for _, line := range strings.Split(out, "\n") {
k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
if !ok {
continue
}
switch k {
case "ActiveState":
p.ActiveState = v
case "InvocationID":
p.InvocationID = v
case "MainPID":
p.MainPID, _ = strconv.Atoi(v)
case "NRestarts":
p.NRestarts, _ = strconv.Atoi(v)
case "ExecStart":
if m := execPathRE.FindStringSubmatch(v); m != nil {
p.ExecPath = m[1]
}
}
}
return p
}
type settleOpts struct {
Timeout time.Duration // the whole window (CARLOS_UNIT_SETTLE)
Poll time.Duration // 500ms
Hold time.Duration // no-probe readiness: active this long in one invocation
WantDir string // the new version's directory; MainPID's exe must be under it
Ready func() error // nil = no upstream to probe; Hold applies
Now func() time.Time
Sleep func(time.Duration)
}
// settleUnit watches unit after a try-restart. beforeInvocation is the
// InvocationID before the restart; "" means the unit was not running, and an
// inactive unit is then "nothing started", not a failure.
func settleUnit(sys unitSystem, unit, beforeInvocation string, o settleOpts) error {
deadline := o.Now().Add(o.Timeout)
var inv string // the new invocation, once seen active
var activeSince time.Time
var lastReady error
notUp := func(reason string) error {
return &release.UnitNotUpError{Unit: unit, Reason: reason, Journal: sys.JournalTail(unit, 20)}
}
for {
p, err := sys.Show(unit)
if err != nil {
return notUp(err.Error())
}
if beforeInvocation == "" && p.ActiveState == "inactive" {
return nil
}
switch {
case p.ActiveState == "failed":
return notUp("unit entered failed state")
case inv != "" && p.InvocationID != inv:
return notUp("systemd restarted it (new invocation)")
case inv != "" && p.NRestarts > 0:
return notUp(fmt.Sprintf("systemd restarted it %d time(s)", p.NRestarts))
case inv == "" && p.ActiveState == "active" && p.InvocationID != beforeInvocation:
inv, activeSince = p.InvocationID, o.Now()
}
if inv != "" && p.ActiveState == "active" {
exe, err := sys.Exe(p.MainPID)
if err == nil && o.WantDir != "" && !strings.HasPrefix(exe, strings.TrimSuffix(o.WantDir, "/")+"/") {
return notUp(fmt.Sprintf("running %s, not the new build in %s", exe, o.WantDir))
}
if o.Ready != nil {
if lastReady = o.Ready(); lastReady == nil {
return nil
}
} else if o.Now().Sub(activeSince) >= o.Hold {
return nil
}
}
if !o.Now().Before(deadline) {
if lastReady != nil {
return notUp(fmt.Sprintf("not ready after %s: %v", o.Timeout, lastReady))
}
return notUp(fmt.Sprintf("not ready after %s", o.Timeout))
}
o.Sleep(o.Poll)
}
}
func unitSettleTimeout(getenv func(string) string) time.Duration {
d, err := time.ParseDuration(strings.TrimSpace(getenv("CARLOS_UNIT_SETTLE")))
switch {
case err != nil || d <= 0:
return 30 * time.Second
case d < 5*time.Second:
return 5 * time.Second
}
return d
}systemctlTryRestartUnit already exists in agent.go and is reused, not duplicated.
- [ ] Step 4: Run the tests to verify they pass
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'Settle|ParseShow' -count=1 Expected: PASS.
- [ ] Step 5: Commit
git add internal/host/unitsettle.go internal/host/unitsettle_test.go
git commit -m "host: settle check for a named unit after try-restart"---
Task 5: Readiness on the effective upstream
Files:
- Modify:
internal/host/unitsettle.go(append) - Test:
internal/host/unitsettle_test.go(append)
Interfaces:
- Produces:
func routeReady(rt registry.Route, timeout time.Duration) func() error, which returns nil when the route has no upstream. - Consumes:
provision.Healthand.WithDefaults()(internal/provision/provision.go:189).
- [ ] Step 1: Write the failing tests
func TestRouteReadyProbesTheUnixSocket(t *testing.T) {
sock := filepath.Join(t.TempDir(), "s.sock")
ln, err := net.Listen("unix", sock)
if err != nil {
t.Fatal(err)
}
srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/healthz" && r.Host == "console.example.com" {
w.WriteHeader(200)
return
}
w.WriteHeader(500)
})}
go srv.Serve(ln)
t.Cleanup(func() { srv.Close() })
rt := registry.Route{Host: "console.example.com", Socket: sock, Addr: "127.0.0.1:1",
Health: `{"path":"/healthz","status":200}`}
if err := routeReady(rt, time.Second)(); err != nil {
t.Fatalf("ready over the socket = %v (Addr must be ignored when Socket is set)", err)
}
}
func TestRouteReadyWrongStatus(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(503) }))
t.Cleanup(srv.Close)
rt := registry.Route{Host: "x.example.com", Addr: strings.TrimPrefix(srv.URL, "http://"),
Health: `{"path":"/healthz","status":200}`}
if err := routeReady(rt, time.Second)(); err == nil || !strings.Contains(err.Error(), "503") {
t.Fatalf("ready = %v, want a 503 failure", err)
}
}
func TestRouteReadyNoProbeIsAConnect(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { ln.Close() })
if err := routeReady(registry.Route{Host: "x", Addr: ln.Addr().String()}, time.Second)(); err != nil {
t.Fatalf("connect readiness = %v", err)
}
}
func TestRouteReadyNoUpstreamIsNil(t *testing.T) {
if routeReady(registry.Route{Host: "x"}, time.Second) != nil {
t.Fatal("a route with no upstream got a ready check; settle must use the hold instead")
}
}Add net, net/http, net/http/httptest, path/filepath and github.com/carlosframework/platform/internal/registry to the test imports.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RouteReady' -count=1 Expected: FAIL, undefined: routeReady.
- [ ] Step 3: Implement
Append to unitsettle.go (add context, encoding/json, io, net, net/http, registry and provision imports):
// routeReady is the readiness check for rt's effective upstream (spec §2.4):
// Socket wins over Addr. With a configured probe it is an HTTP GET; without
// one, a connect. nil when the route has no upstream: settle then requires
// the unit to stay active for its Hold.
func routeReady(rt registry.Route, timeout time.Duration) func() error {
network, address := "unix", rt.Socket
if address == "" {
network, address = "tcp", rt.Addr
}
if address == "" {
return nil
}
if rt.Health == "" {
return func() error {
c, err := net.DialTimeout(network, address, timeout)
if err != nil {
return fmt.Errorf("connect %s: %w", address, err)
}
return c.Close()
}
}
var h provision.Health
if err := json.Unmarshal([]byte(rt.Health), &h); err != nil {
return func() error { return fmt.Errorf("route health probe does not parse: %w", err) }
}
h = h.WithDefaults()
hostHdr := h.HostHeader
if hostHdr == "" {
hostHdr = rt.Host
}
client := &http.Client{Timeout: timeout, Transport: &http.Transport{
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, network, address)
},
}}
return func() error {
req, err := http.NewRequest(http.MethodGet, "http://upstream"+h.Path, nil)
if err != nil {
return err
}
req.Host = hostHdr
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("dial: %v", err)
}
io.Copy(io.Discard, io.LimitReader(resp.Body, 4<<10))
resp.Body.Close()
if resp.StatusCode != h.Status {
return fmt.Errorf("status %d (want %d)", resp.StatusCode, h.Status)
}
return nil
}
}- [ ] Step 4: Run the tests to verify they pass
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RouteReady|Settle' -count=1 Expected: PASS.
- [ ] Step 5: Commit
git add internal/host/unitsettle.go internal/host/unitsettle_test.go
git commit -m "host: readiness on a route's effective upstream, socket before addr"---
Task 6: Wire the checked restart and the rollback restart
Files:
- Modify:
internal/host/agent.go:makeRestartRoute(3101) getstryRestartUnit func(rt registry.Route) error; newcheckedUnitRestartandrollbackUnitRestart; callers at 820, 1931, 3243;RunAdoptpass options at 3660; the agent loop options at 871 and 1500. - Modify:
internal/host/agent_test.go: the sevenmakeRestartRoute(call sites wrap their unit fake withunitOnly(...). - Test:
internal/host/unitsettle_test.go(append)
Interfaces:
- Consumes: Tasks 4 and 5 (
unitSystem,settleUnit,settleOpts,routeReady,unitSettleTimeout); Task 3'srelease.WithRollbackRestart. - Produces: -
func unitOnly(f func(unit string) error) func(rt registry.Route) error-func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string) func(rt registry.Route) error-func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error
- [ ] Step 1: Write the failing tests
func boundRoute() registry.Route {
return registry.Route{Host: "platform.example.com", App: "console", Unit: "console.service", Addr: "127.0.0.1:1"}
}
func liveTo(t *testing.T, opt, host, ver string) {
t.Helper()
dir := filepath.Join(opt, "versions", ver)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "console"), []byte("x"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(opt, "live"), 0o755); err != nil {
t.Fatal(err)
}
os.Remove(filepath.Join(opt, "live", host))
if err := os.Symlink(filepath.Join(dir, "console"), filepath.Join(opt, "live", host)); err != nil {
t.Fatal(err)
}
}
func TestNamedRestartBindingMismatchKeepsTheError(t *testing.T) {
opt := t.TempDir()
liveTo(t, opt, "platform.example.com", "new")
f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "old",
ExecPath: filepath.Join(opt, "live", "console.example.com")}}}
failing := &failingTry{fakeUnits: f}
err := checkedUnitRestart(failing, opt, func(string) string { return "" })(boundRoute())
if err == nil || !strings.Contains(err.Error(), "try-restart failed") {
t.Fatalf("err = %v, want the try-restart error propagated (codex 12)", err)
}
}
type failingTry struct{ *fakeUnits }
func (failingTry) TryRestart(string) error { return errors.New("try-restart failed") }
func TestNamedRestartShowFailsFallsBack(t *testing.T) {
opt := t.TempDir()
sys := &showFails{fakeUnits: &fakeUnits{}}
if err := checkedUnitRestart(sys, opt, func(string) string { return "" })(boundRoute()); err != nil {
t.Fatalf("err = %v, want today's behaviour (plain try-restart, its nil)", err)
}
if len(sys.restarts) != 1 || sys.restarts[0] != "try-restart console.service" {
t.Fatalf("restarts = %v", sys.restarts)
}
}
type showFails struct{ *fakeUnits }
func (showFails) Show(string) (unitProps, error) { return unitProps{}, errors.New("Access denied") }
func TestNamedRestartStoppedUnitIsNotChecked(t *testing.T) {
opt := t.TempDir()
liveTo(t, opt, "platform.example.com", "new")
f := &fakeUnits{shows: []unitProps{{ActiveState: "inactive", ExecPath: filepath.Join(opt, "live", "platform.example.com")}}}
if err := checkedUnitRestart(f, opt, func(string) string { return "" })(boundRoute()); err != nil {
t.Fatalf("err = %v, want nil for an operator-stopped unit", err)
}
}
func TestRollbackRestartResetsThenRestarts(t *testing.T) {
opt := t.TempDir()
liveTo(t, opt, "platform.example.com", "old")
reg := restartTestRegistry(t)
if err := reg.Upsert(boundRoute()); err != nil {
t.Fatal(err)
}
f := &fakeUnits{shows: []unitProps{
{ActiveState: "active", InvocationID: "bad"},
{ActiveState: "active", InvocationID: "restored", MainPID: 9},
}, exe: map[int]string{9: filepath.Join(opt, "versions", "old", "console")}}
rb := rollbackUnitRestart(reg, f, opt, func(string) string { return "" })
if err := rb("platform.example.com"); err != nil {
t.Fatalf("rollback = %v", err)
}
if strings.Join(f.restarts, ",") != "reset-failed console.service,restart console.service" {
t.Fatalf("restarts = %v, want reset-failed then restart (never start)", f.restarts)
}
}restartTestRegistry is agent_test.go's existing constructor (line 539). In TestRollbackRestartResetsThenRestarts, upsert boundRoute() with Addr cleared (r := boundRoute(); r.Addr = ""). With no upstream, routeReady returns nil and settle uses the hold instead of dialling. rollbackUnitRestart uses the real clock, so this test takes about 5 s, which is acceptable for one test. The binding test keeps Addr because it returns before readiness.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'NamedRestart|RollbackRestart' -count=1 Expected: FAIL, undefined: checkedUnitRestart.
- [ ] Step 3: Implement
In agent.go, change makeRestartRoute's signature and its first arm:
func makeRestartRoute(reg *registry.Registry, tryRestart func(host string), tryRestartUnit func(rt registry.Route) error, bounce func(host string) bool) func(host string) error {
return func(host string) error {
rt, err := reg.Get(host)
if err != nil {
log.Printf("adopt: registry lookup %s: %v (restarting as a unit)", host, err)
}
if err == nil && rt != nil && rt.Unit != "" {
return tryRestartUnit(*rt)
}(the rest unchanged). Add, next to systemctlTryRestartUnit:
// unitOnly adapts a unit-name restart to makeRestartRoute's route-aware arm.
func unitOnly(f func(unit string) error) func(rt registry.Route) error {
return func(rt registry.Route) error { return f(rt.Unit) }
}
// checkedUnitRestart is the named-unit arm with D1's settle check (spec
// §2.2-§2.3). The binding is verified first: the check judges a restart only
// when the unit's ExecStart runs THIS route's live link. Anything it cannot
// establish (show denied, wrong binding, unit stopped) falls back to today's
// plain try-restart and returns that command's own error.
func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string) func(rt registry.Route) error {
return func(rt registry.Route) error {
before, err := sys.Show(rt.Unit)
if err != nil {
log.Printf("adopt: %s: cannot read %s (%v); restart not checked", rt.Host, rt.Unit, err)
return sys.TryRestart(rt.Unit)
}
link := filepath.Join(optDir, "live", rt.Host)
if before.ExecPath != link {
log.Printf("adopt: binding: %s runs %s, but the unit stamp is on %s; restart not checked", rt.Unit, before.ExecPath, rt.Host)
return sys.TryRestart(rt.Unit)
}
if before.ActiveState != "active" {
return sys.TryRestart(rt.Unit) // stopped by an operator: stays a no-op
}
want, err := filepath.EvalSymlinks(link)
if err != nil {
return sys.TryRestart(rt.Unit)
}
if err := sys.TryRestart(rt.Unit); err != nil {
return err
}
timeout := unitSettleTimeout(getenv)
err = settleUnit(sys, rt.Unit, before.InvocationID, settleOpts{
Timeout: timeout, Poll: 500 * time.Millisecond, Hold: 5 * time.Second,
WantDir: filepath.Dir(want), Ready: routeReady(rt, 2*time.Second),
Now: time.Now, Sleep: time.Sleep})
if err == nil {
log.Printf("adopt: %s: %s settled on %s", rt.Host, rt.Unit, filepath.Dir(want))
}
return err
}
}
// rollbackUnitRestart is D1's rollback restart (release.WithRollbackRestart):
// the live link already names the restored build; replace whatever is
// running (a deadline failure can leave the rejected build up) and check
// the restored build comes up too.
func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error {
return func(host string) error {
rt, err := reg.Get(host)
if err != nil || rt == nil || rt.Unit == "" {
return nil
}
before, _ := sys.Show(rt.Unit)
if err := sys.ResetFailed(rt.Unit); err != nil {
log.Printf("adopt: %s: reset-failed %s: %v (continuing)", host, rt.Unit, err)
}
if err := sys.Restart(rt.Unit); err != nil {
return err
}
want, err := filepath.EvalSymlinks(filepath.Join(optDir, "live", host))
if err != nil {
return err
}
if err := settleUnit(sys, rt.Unit, before.InvocationID, settleOpts{
Timeout: unitSettleTimeout(getenv), Poll: 500 * time.Millisecond, Hold: 5 * time.Second,
WantDir: filepath.Dir(want), Ready: routeReady(*rt, 2*time.Second),
Now: time.Now, Sleep: time.Sleep}); err != nil {
return fmt.Errorf("rollback did not come up either: %w", err)
}
return nil
}
}settleUnit treats beforeInvocation == "" with an inactive unit as nothing started. In rollbackUnitRestart the unit has just been restarted, so before.InvocationID is whatever ran before. A failed unit keeps its last InvocationID, and a new one appears on restart, which is what settle keys on.
Callers:
- line 820 and line 1931: replace
systemctlTryRestartUnitwithcheckedUnitRestart(systemdUnits{}, optDir, os.Getenv), using whatever opt-dir variable is in scope there (d.optDiron the fast path). - line 3243
rootRestartRoute: the same, with the opt dir its caller uses (RunAdopt's). Add anoptDir stringparameter torootRestartRouteand pass it from its one production caller. Update its three test callers too:internal/host/agent_test.go:718,:836and:963, each passing that test's own opt dir (ort.TempDir()where the test has none). RunAdopt'spassOpts(3660): appendrelease.WithRollbackRestart(rollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv)).- the agent loop options at 871 and 1500: append the same option. The sandboxed agent's
reset-failed/restartmay be denied; that is logged and the rollback line says "could not start it", the same as today's start path there.
In agent_test.go, change each makeRestartRoute(reg, X, Y, Z) whose third argument is a func(string) error to makeRestartRoute(reg, X, unitOnly(Y), Z).
- [ ] Step 4: Run the host and release suites
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host ./internal/release -count=1 Expected: PASS. Per [[cpu-budget-docs-t1]], run them niced and one at a time if the box is busy: nice -n 10.
- [ ] Step 5: Commit
git add internal/host/agent.go internal/host/agent_test.go internal/host/unitsettle_test.go
git commit -m "host: named-unit restarts settle and roll back by restart (console defences D1)"---
Task 7: carlos promote --retry
Files:
- Modify:
internal/release/release.go:PromoteOptions(786) addsRetry bool;Promotion(144) addsRetry bool \json:"retry,omitempty"\`;PromoteOnLadder`'s repeat branch. - Modify:
cli.go(2356: flag; 2397 and 2417: pass it) - Modify:
apimode.go:947(promotepayload) - Modify:
internal/console/apicli.go(743: request field; 899: option) - Test:
internal/release/release_test.go(append),internal/console/apicli_test.go(append)
Interfaces:
- Produces:
PromoteOptions.Retry. With Retry, a repeat is re-signed with a freshPromotedAt. Without a repeat it is an error:promote --retry renews a promotion: <channel> names <cur>, not <version>.
- [ ] Step 1: Write the failing tests
func TestPromoteRetryRenewsPromotedAt(t *testing.T) {
s := NewDirStore(t.TempDir())
ctx := context.Background()
priv, pub := testKeys(t)
shipRelease(t, s, "bab", "console", "v1", adoptPlatform, []byte("b"))
if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{}); err != nil {
t.Fatal(err)
}
before, _ := Channel(ctx, s, pub, "bab", "console", "canary/x")
time.Sleep(1100 * time.Millisecond) // PromotedAt is truncated to the second
res, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{})
if err != nil || !res.NoOp {
t.Fatalf("plain repeat = %+v, %v; want the existing no-op", res, err)
}
same, _ := Channel(ctx, s, pub, "bab", "console", "canary/x")
if !same.PromotedAt.Equal(before.PromotedAt) {
t.Fatal("a plain repeat rewrote the pointer")
}
if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{Retry: true}); err != nil {
t.Fatalf("retry: %v", err)
}
after, _ := Channel(ctx, s, pub, "bab", "console", "canary/x")
if after.Version != "v1" || !after.PromotedAt.After(before.PromotedAt) {
t.Fatalf("after retry = %+v, want v1 with a later PromotedAt", after)
}
}
func TestPromoteRetryNeverChangesTheVersion(t *testing.T) {
s := NewDirStore(t.TempDir())
ctx := context.Background()
priv, _ := testKeys(t)
shipRelease(t, s, "bab", "console", "v1", adoptPlatform, []byte("a"))
shipRelease(t, s, "bab", "console", "v2", adoptPlatform, []byte("b"))
if _, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v1", PromoteOptions{}); err != nil {
t.Fatal(err)
}
_, err := PromoteOnLadder(ctx, s, priv, "bab", "console", "canary/x", "v2", PromoteOptions{Retry: true})
if err == nil || !strings.Contains(err.Error(), "renews a promotion") {
t.Fatalf("retry onto a different version = %v, want a refusal", err)
}
}For the API door, add one test beside the existing handleAPIPromote tests in internal/console/apicli_test.go. Copy the nearest existing promote test's setup verbatim, send {"version":"v1","channel":"canary/x","retry":true} after a first promote, and assert 200 and that the stored pointer's PromotedAt moved.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release -run 'PromoteRetry' -count=1 Expected: FAIL, unknown field Retry.
- [ ] Step 3: Implement
PromoteOptions: add
// Retry re-signs a pointer that already names this version, with a
// fresh PromotedAt: the way past a box's D1 refusal (refusal.go), which
// blocks one promotion, not a version. Never changes the version.
Retry boolPromotion: add Retry bool \json:"retry,omitempty"\``.
In PromoteOnLadder's repeat branch, before res.NoOp = true, insert:
if opt.Retry {
res.From = res.To
rec := Promotion{From: res.To, To: res.To, Actor: opt.Actor, Retry: true}
if !time.Now().UTC().Truncate(time.Second).After(p.PromotedAt) {
time.Sleep(time.Until(p.PromotedAt.Add(time.Second)))
}
if err := sg.promote(ctx, s, acct, app, channel, version, opt.Authorize, rec); err != nil {
return res, err
}
return res, nil
}and after the repeat branch's closing brace (so it is reached only when the pointer is absent or names another version), insert:
if opt.Retry {
cur := "nothing"
if c, _, err := s.Get(ctx, ChannelKey(acct, app, channel)); err == nil {
if p, verr := verifyPointer(pub, c, acct, app, channel); verr == nil {
cur = p.Version
}
}
return res, fmt.Errorf("promote --retry renews a promotion: %s names %s, not %s", channel, cur, version)
}cli.go: next to hotfix := fs.Bool("hotfix", ...) (2356) add
retry := fs.Bool("retry", false, "re-sign the channel's current promotion of this version with a fresh time, so a box that refused it tries again")and pass it: release.PromoteOptions{Hotfix: *hotfix, Retry: *retry, Actor: promoteActor()} (2417), and in API mode sess.promote(..., *hotfix, *retry, message).
apimode.go:947: add a retry bool parameter after hotfix, and a Retry bool \json:"retry,omitempty"\` field in the payload. Update its one other caller if any (grep sess.promote( and .promote(ctx`).
internal/console/apicli.go: add Retry bool \json:"retry"\` to the request struct (743), and Retry: req.Retry to the PromoteOptions at 899. The retry goes through the same Authorize` and scope checks as any promote.
- [ ] Step 4: Run the tests
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/release ./internal/console -run 'Promote' -count=1 && GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./... Expected: PASS, and the build succeeds.
- [ ] Step 5: Commit
git add internal/release/release.go internal/release/release_test.go cli.go apimode.go internal/console/apicli.go internal/console/apicli_test.go
git commit -m "promote --retry: re-sign a promotion so a box retries a refused build"---
Task 8: The alert and the status --live line
Files:
- Create:
internal/host/refusalwatch.go - Modify:
internal/host/alertsink.go(adddeliverRelease) - Modify:
internal/status/snapshot.go(ReleaseRefusaltype;Probes.ReleaseRefusals;BoxMetrics.ReleaseRefusals; the copy at 719 and the emptiness check at 700) - Modify:
internal/host/agent.go:2939(feed the probe), and the metrics loop (internal/host/metrics_loop.go:511neighbourhood) to call the watcher each tick - Modify:
cli.go:3418printLiveBoxes - Test:
internal/host/refusalwatch_test.go,statuslive_test.go(the existingprintLiveBoxestests, extended)
Interfaces:
- Consumes: Task 1's
release.ListRefusals. - Produces: -
type ReleaseRefusal struct { Host, Version, Reason string; RefusedAt time.Time }(status package, JSONhost,version,reason,refused_at) -type refusalWatch struct { seen map[string]time.Time }withfunc (w *refusalWatch) tick(ctx context.Context, bakeDir string, sinks *alertSinks, now time.Time) []status.ReleaseRefusal-func (a *alertSinks) deliverRelease(ctx context.Context, r release.Refusal)
- [ ] Step 1: Write the failing tests
package host
import (
"context"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/release"
)
func TestRefusalWatchAlertsOncePerRecord(t *testing.T) {
bake := t.TempDir()
now := time.Date(2026, 10, 6, 9, 31, 0, 0, time.UTC)
if err := release.WriteRefusal(bake, release.Refusal{Host: "console.example.com", App: "console",
Version: "v2", RefusedAt: now.Add(-time.Minute), Reason: "exited: status=2"}); err != nil {
t.Fatal(err)
}
var posts []string
sinks := newAlertSinks("https://hook.example", func(_ string, b []byte) error { posts = append(posts, string(b)); return nil }, "", nil, &alertCounters{})
w := &refusalWatch{}
got := w.tick(context.Background(), bake, sinks, now)
w.tick(context.Background(), bake, sinks, now.Add(time.Minute))
if len(posts) != 1 || !strings.Contains(posts[0], `"release_refused"`) || !strings.Contains(posts[0], "v2") {
t.Fatalf("posts = %v, want exactly one release_refused alert", posts)
}
if len(got) != 1 || got[0].Host != "console.example.com" {
t.Fatalf("status rows = %+v", got)
}
}
func TestRefusalWatchSkipsOldRecordsAtStartup(t *testing.T) {
bake := t.TempDir()
now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
if err := release.WriteRefusal(bake, release.Refusal{Host: "a.example.com", Version: "v2",
RefusedAt: now.Add(-2 * time.Hour)}); err != nil {
t.Fatal(err)
}
var posts int
sinks := newAlertSinks("https://hook.example", func(string, []byte) error { posts++; return nil }, "", nil, &alertCounters{})
(&refusalWatch{}).tick(context.Background(), bake, sinks, now)
if posts != 0 {
t.Fatalf("an agent restart re-alerted a refusal from two hours ago")
}
}In statuslive_test.go, extend the existing printLiveBoxes test with a box whose Metrics.ReleaseRefusals has one row. Assert that the output contains refused console.example.com v2 since 2026-10-06T09:30:40Z: exited: status=2.
- [ ] Step 2: Run them to verify they fail
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host -run 'RefusalWatch' -count=1 Expected: FAIL, undefined: refusalWatch.
- [ ] Step 3: Implement
internal/status/snapshot.go, beside ConfigRefusal:
// ReleaseRefusal is a build this box rolled back because its unit did not
// come up (D1, release.Refusal). PRIVATE like the rest of Box.Metrics: Host
// is a real route name.
type ReleaseRefusal struct {
Host string `json:"host"`
Version string `json:"version"`
Reason string `json:"reason"`
RefusedAt time.Time `json:"refused_at"`
}Add ReleaseRefusals []ReleaseRefusal to Probes and to the metrics struct that ConfigRefusals is copied into at line 719, with JSON key release_refusals,omitempty. Copy it there the same way, and add len(p.ReleaseRefusals) > 0 || to the emptiness check at 700.
internal/host/alertsink.go, after deliver:
// deliverRelease sends a D1 refusal to every configured sink. The webhook
// body is the record with kind "release_refused"; SNS gets a sentence.
func (a *alertSinks) deliverRelease(ctx context.Context, r release.Refusal) {
if a == nil {
return
}
if a.webhookURL != "" && a.post != nil {
body, err := json.Marshal(struct {
Kind string `json:"kind"`
release.Refusal
}{"release_refused", r})
if err == nil {
if err := a.post(a.webhookURL, body); err != nil {
a.counts.webhook.Add(1)
log.Printf("host: alert webhook post (release refusal %s): %v", r.Host, err)
}
}
}
if a.topicARN != "" && a.pub != nil {
subject := fmt.Sprintf("%s: %s did not start, rolled back", r.Host, r.Version)
if len(subject) > 100 { // SNS subject limit
subject = subject[:100]
}
msg := fmt.Sprintf("%s refused %s at %s: %s\n\nRetry with: carlos promote --retry %s <channel>",
r.Host, r.Version, r.RefusedAt.Format(time.RFC3339), r.Reason, r.Version)
pctx, cancel := context.WithTimeout(ctx, alertPublishTimeout)
defer cancel()
if _, err := a.pub.Publish(pctx, &sns.PublishInput{TopicArn: &a.topicARN, Subject: &subject, Message: &msg}); err != nil {
a.counts.sns.Add(1)
log.Printf("host: alert sns publish (release refusal %s): %v", r.Host, err)
}
}
}internal/host/refusalwatch.go:
// refusalWatch turns D1's refusal records (release.Refusal, written by the
// root pass) into one alert each and a status row each. It runs on the
// agent's metrics tick. A record older than refusalAlertWindow when first
// seen is not alerted, so an agent restart does not re-send old news.
package host
import (
"context"
"log"
"time"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/status"
)
const refusalAlertWindow = 30 * time.Minute
type refusalWatch struct{ seen map[string]time.Time }
func (w *refusalWatch) tick(ctx context.Context, bakeDir string, sinks *alertSinks, now time.Time) []status.ReleaseRefusal {
if w.seen == nil {
w.seen = map[string]time.Time{}
}
recs, err := release.ListRefusals(bakeDir)
if err != nil {
log.Printf("agent: refusals: %v", err)
return nil
}
var rows []status.ReleaseRefusal
for _, r := range recs {
rows = append(rows, status.ReleaseRefusal{Host: r.Host, Version: r.Version, Reason: r.Reason, RefusedAt: r.RefusedAt})
if last, ok := w.seen[r.Host]; ok && !r.RefusedAt.After(last) {
continue
}
w.seen[r.Host] = r.RefusedAt
if now.Sub(r.RefusedAt) <= refusalAlertWindow {
sinks.deliverRelease(ctx, r)
}
}
return rows
}Wire it. In the metrics loop, keep one refusalWatch for the life of the loop. Each tick, call rows := watch.tick(ctx, dataDir(), sinks, now) and store the rows where the probe at agent.go:2939 can read them. Follow the configRefusalProbe pattern: a package-level atomic.Pointer[[]status.ReleaseRefusal] written by the tick and read by a releaseRefusalProbe() that fills p.ReleaseRefusals.
cli.go printLiveBoxes: after the box's main Fprintf, add
for _, r := range m.ReleaseRefusals {
fmt.Fprintf(w, " refused %s %s since %s: %s\n", r.Host, r.Version, r.RefusedAt.UTC().Format(time.RFC3339), r.Reason)
}- [ ] Step 4: Run the tests
Run: GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go test ./internal/host ./internal/status . -run 'RefusalWatch|LiveBoxes|Snapshot' -count=1 Expected: PASS.
- [ ] Step 5: Commit
git add internal/host/refusalwatch.go internal/host/refusalwatch_test.go internal/host/alertsink.go internal/host/agent.go internal/host/metrics_loop.go internal/status/snapshot.go cli.go statuslive_test.go
git commit -m "host: alert on a refused build and show it in status --live"---
Task 9: Full gate and the README operator note
Files:
- Modify:
README.md: under "Releasing the platform", in the release-notes area that lists operator checks, add the note below.
- [ ] Step 1: Add the note
**Named units now settle (console release defences, D1).** After adopting a
build for a route with a named unit (the console), the box waits up to 30 s
(`CARLOS_UNIT_SETTLE`) for the unit to stay up on the new build and pass its
health probe. If it does not, the box rolls back, refuses that promotion and
alerts. Retry with `carlos promote --retry <version> <channel>`. The check
only runs when the unit's `ExecStart` names the stamped route's own live
link; `adopt` logs `binding: … restart not checked` where it does not, and
that box needs its stamp moved (2026-08-23 console design §4c).Keep its sentences plain and avoid em dashes in new prose ([[copy-voice-no-em-dashes]]). It is README prose, so invoke the copy-review skill on it before committing and apply Paul's edits verbatim.
- [ ] Step 2: Run the full gate
Run, niced and serial ([[cpu-budget-docs-t1]]):
GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false nice -n 10 go vet ./... && \
GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false nice -n 10 go test ./... -count=1 && \
GOOS=darwin GOARCH=arm64 GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./... && \
GOOS=windows GOARCH=amd64 GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false go build ./...Expected: all pass. unitsettle.go uses no linux-only syscalls (exec and /proc reads only), so it needs no build tag ([[cross-compile-gate]]). If the cross build fails on it anyway, split the systemdUnits methods into unitsettle_linux.go with a !linux stub returning an error from Show.
- [ ] Step 3: Commit
git add README.md
git commit -m "README: operator note for the named-unit settle check"