carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# `carlos system update` Implementation Plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** A fleet operator runs `carlos system update` and every box on the channel fetches, verifies, installs and health-gates the new platform binary itself, rolling back to the previous one if it does not come up.
6
7**Architecture:** A signed `SystemPointer` at `control/system/router/<channel>.json` names a version and its per-platform artifacts mirrored into the deployment bucket. A sandboxed agent leg notices the pointer on the serial poll, streams and verifies the artifact into `/data`, and touches a marker. A root oneshot (`carlos system apply`) re-verifies everything from root-owned copies, journals each step, swaps binary and units, restarts the edge, gates it, and rolls back to `carlos.prev` on failure; a boot-time unit recovers an interrupted apply. The console does the promote: fetch the public signed release, mirror, sign the pointer, bump the serial.
8
9**Tech Stack:** Go (stdlib, ed25519, net/http), systemd units embedded in the binary, S3/dir/fleet-door stores behind `release.Store`.
10
11**Spec:** `docs/superpowers/specs/2026-09-06-carlos-system-update-design.md` — read §3, §4, §6 before the task you are on. The spec wins where this plan and it disagree; say so in the commit if you find such a place.
12
13## Global Constraints
14
15- Go build cache is read-only in the sandbox: run every `go` command with `dangerouslyDisableSandbox: true`, and `export TMPDIR=/tmp/c` first (long socket paths break `internal/edge`/`internal/activator` tests).
16- Capture test exit codes explicitly: `out=$(go test ./pkg/... 2>&1); rc=$?; echo "$out" | tail -40; echo rc=$rc`. Never pipe `go test` into `grep`/`tail` and read the pipeline's status.
17- `-count=1` on every test run. `gofmt -l .` must print nothing before a commit.
18- Never run full `carlos bootstrap` on a live box path; the apply stages units into a temp root (`bootstrap -root <dir>`) and copies only `etc/`. Never copy a staging root's `data/`.
19- sysusers before tmpfiles, always, in every code path that installs units.
20- Root reads exactly one file under `/data` (`requested.json`), walks to it with `O_DIRECTORY|O_NOFOLLOW` per component, and derives every other path from a validated version string. Root never deletes under `/data`.
21- Apply never `systemctl start`s a `.service`; it calls `systemctlPlan`, which starts only `.timer`/`.path` units.
22- Every root state file is written temp → fsync → rename → directory fsync (`writeFileDurable`, Task 4).
23- The component name is `router`; channels are `canary` and `stable`. Platform strings are `<goos>-<goarch>` matching `^[a-z0-9]+-[a-z0-9]+$`.
24- Commit after every task with the attribution footer:
25 ```
26 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
27 Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
28 ```
29- Work on branch `carlos-system-update` from forge `main`; land with `amadan branch merge`, never `gh`.
30
31---
32
33## File Structure
34
35| File | Responsibility |
36|---|---|
37| `internal/release/system.go` (+`_test`) | `SystemPointer`, `SystemRelease`, `SystemArtifact`; key builders; validate/sign/verify (root key only); `GetSystemPointer`, `PromoteSystem`, `RollbackSystem` |
38| `internal/release/store.go`, `internal/fleetapi/fleetapi.go` (+tests) | `Opener` side-interface, `Open` on `DirStore`/`S3Store`/`prefixStore`/fleetapi, `OpenObject`, `FetchToFile` |
39| `internal/edge/storedoor.go` (+`_test`) | `aclRead` grants for the two new key shapes |
40| `internal/host/systemstate.go` (+`_test`) | On-box state types (request, result, seen, applied, journal), paths, `writeFileDurable`, outcome constants |
41| `internal/status/snapshot.go`, `internal/host/agent.go` | `Platform`, `SystemChannel`, `System` in `Probes`/`BoxMetrics`; filled in `statusLoop` |
42| `internal/host/units/carlos-system-update.path`, `carlos-system-update.service`, `carlos-system-recover.service`; `internal/host/bootstrap.go`; `internal/host/probes.go` | The three units, `enabledUnits`, `-plan-json`, the path-unit probe |
43| `internal/host/systemleg.go` (+`_test`) | `systemLeg`: `observe`/`tick`, fetch, request, marker re-touch; wired into `fastDeps` |
44| `internal/host/systemapply.go` (+`_test`) | `RunSystemApply`, `systemApplier` with an injectable `sysOps` seam: refuse path, install path, gate, rollback, recovery, lock |
45| `internal/host/agent.go` `RunAdopt` | The hourly belt: install and arm the three units if missing |
46| `internal/publicrelease/feed.go` (+`_test`) | `Feed`: latest tag, verified asset fetch — factored out of `update.go`, shared with the console |
47| `internal/console/apisystem.go` (+`_test`) | `/api/cli/system/{status,update,upload,jobs/{id},rollback}`; ops-owner gate; in-memory update job |
48| `system.go` (+`_test`, package main), `main.go` | `carlos system update|status|rollback|apply`, `carlos system router …` alias |
49| `docs/site/cli.md`, `AGENTS.md` | Operator docs; the env contract |
50
51Task order is dependency order. Tasks 1–3 are independent of each other and can run in parallel; 4 before 5/7/8; 6 before 8; 10 before 11; 11 before 12.
52
53---
54
55### Task 1: The system pointer (`internal/release/system.go`)
56
57**Files:**
58- Create: `internal/release/system.go`
59- Test: `internal/release/system_test.go`
60
61**Interfaces:**
62- Consumes: `signedWrapper`, `Signer` (fields `key`, `scope`, method `usable()`), `Promotion`, `historyMax`, `auditMax`, `prependCapped`, `ValidChannel`, `ValidVersion`, `ParsePublicKey`, `Store`, `ErrCASConflict` — all in package `release` (`release.go`, `signer.go`, `store.go`).
63- Produces (all exported, package `release`):
64 ```go
65 const SystemComponentRouter = "router"
66 func SystemPointerKey(component, channel string) string // control/system/<c>/<ch>.json
67 func SystemArtifactKey(version, platform string) string // bootstrap/router/<v>/carlos-<platform>
68 type SystemArtifact struct{ Key, SHA256 string }
69 type SystemRelease struct{ Version string; Artifacts map[string]SystemArtifact; Source string; ReleasedAt time.Time }
70 type SystemPointer struct{ Schema int; Component, Channel string; Current SystemRelease; History []SystemRelease; Spread string; PromotedAt time.Time; Audit []Promotion }
71 func (p *SystemPointer) SpreadDuration() time.Duration
72 func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error)
73 type SystemPromoteOptions struct{ Actor string; Spread time.Duration }
74 func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error)
75 func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error)
76 var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to")
77 ```
78
79- [ ] **Step 1: Write the failing tests**
80
81```go
82package release
83
84import (
85 "context"
86 "encoding/json"
87 "errors"
88 "os"
89 "strings"
90 "testing"
91 "time"
92)
93
94func testSystemRelease(v string) SystemRelease {
95 return SystemRelease{
96 Version: v,
97 Artifacts: map[string]SystemArtifact{
98 "linux-arm64": {Key: SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("ab", 32)},
99 },
100 Source: "release:" + v,
101 ReleasedAt: time.Date(2026, 9, 6, 8, 0, 0, 0, time.UTC),
102 }
103}
104
105func TestSystemPointerRoundTrip(t *testing.T) {
106 priv, pub := testKeys(t)
107 s := NewDirStore(t.TempDir())
108 ctx := context.Background()
109 if _, _, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable"); !errors.Is(err, os.ErrNotExist) {
110 t.Fatalf("missing pointer: err = %v, want ErrNotExist", err)
111 }
112 p, err := PromoteSystem(ctx, s, RootSigner(priv), SystemComponentRouter, "stable", testSystemRelease("v0.18.0"),
113 SystemPromoteOptions{Actor: "paul", Spread: 5 * time.Minute})
114 if err != nil {
115 t.Fatal(err)
116 }
117 if p.Current.Version != "v0.18.0" || p.Spread != "5m0s" || p.Schema != 1 || p.PromotedAt.IsZero() {
118 t.Fatalf("promoted pointer = %+v", p)
119 }
120 got, etag, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable")
121 if err != nil || etag == "" {
122 t.Fatalf("get: %v etag=%q", err, etag)
123 }
124 if got.Current.Artifacts["linux-arm64"].SHA256 != strings.Repeat("ab", 32) {
125 t.Fatalf("artifact lost: %+v", got.Current)
126 }
127 if got.SpreadDuration() != 5*time.Minute {
128 t.Fatalf("spread = %v", got.SpreadDuration())
129 }
130 if len(got.Audit) != 1 || got.Audit[0].Actor != "paul" || got.Audit[0].Version != "v0.18.0" {
131 t.Fatalf("audit = %+v", got.Audit)
132 }
133}
134
135func TestSystemPromoteKeepsWholeReleaseInHistoryAndRollbackRestoresIt(t *testing.T) {
136 priv, pub := testKeys(t)
137 s := NewDirStore(t.TempDir())
138 ctx := context.Background()
139 sg := RootSigner(priv)
140 if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.18.0"), SystemPromoteOptions{}); err != nil {
141 t.Fatal(err)
142 }
143 first, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary")
144 time.Sleep(1100 * time.Millisecond) // PromotedAt is second-truncated; the rollback must be NEWER
145 if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.19.0"), SystemPromoteOptions{}); err != nil {
146 t.Fatal(err)
147 }
148 p, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary")
149 if len(p.History) != 1 || p.History[0].Version != "v0.18.0" || p.History[0].Artifacts["linux-arm64"].Key == "" {
150 t.Fatalf("history must carry the whole release: %+v", p.History)
151 }
152 time.Sleep(1100 * time.Millisecond)
153 rb, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul")
154 if err != nil {
155 t.Fatal(err)
156 }
157 if rb.Current.Version != "v0.18.0" || rb.Current.Artifacts["linux-arm64"].Key != SystemArtifactKey("v0.18.0", "linux-arm64") {
158 t.Fatalf("rollback current = %+v", rb.Current)
159 }
160 if !rb.PromotedAt.After(first.PromotedAt) || !rb.PromotedAt.After(p.PromotedAt) {
161 t.Fatalf("a rollback is a NEW promote: promoted_at %v must be newer than %v and %v", rb.PromotedAt, first.PromotedAt, p.PromotedAt)
162 }
163 if len(rb.History) != 0 {
164 t.Fatalf("history after rollback = %+v", rb.History)
165 }
166 if _, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul"); !errors.Is(err, ErrNoSystemHistory) {
167 t.Fatalf("second rollback err = %v", err)
168 }
169}
170
171func TestSystemPromoteSameVersionDoesNotGrowHistory(t *testing.T) {
172 priv, pub := testKeys(t)
173 s := NewDirStore(t.TempDir())
174 ctx := context.Background()
175 sg := RootSigner(priv)
176 for i := 0; i < 2; i++ {
177 if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
178 t.Fatal(err)
179 }
180 }
181 p, _, _ := GetSystemPointer(ctx, s, pub, "router", "stable")
182 if len(p.History) != 0 {
183 t.Fatalf("re-promoting the running version must not push it into history: %+v", p.History)
184 }
185}
186
187func TestSystemPointerRefusesScopedSigner(t *testing.T) {
188 _, _, sg := scopedFixture(t, []string{RungCanary, RungEdge}, 30*24*time.Hour)
189 s := NewDirStore(t.TempDir())
190 _, err := PromoteSystem(context.Background(), s, sg, "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{})
191 if err == nil || !strings.Contains(err.Error(), "root key") {
192 t.Fatalf("scoped signer signed a system pointer: err = %v", err)
193 }
194}
195
196func TestSystemPointerRefusesAGrantOnTheWire(t *testing.T) {
197 // A wrapper carrying a grant is refused BEFORE the signature is checked,
198 // like the production flag (production.go) and the pipeline.
199 priv, pub := testKeys(t)
200 s := NewDirStore(t.TempDir())
201 ctx := context.Background()
202 if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
203 t.Fatal(err)
204 }
205 body, etag, _ := s.Get(ctx, SystemPointerKey("router", "stable"))
206 var sw signedWrapper
207 if err := json.Unmarshal(body, &sw); err != nil {
208 t.Fatal(err)
209 }
210 sw.Grant = &signedGrant{Payload: "x", Sig: "y"}
211 tampered, _ := json.Marshal(sw)
212 if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), tampered, etag); err != nil {
213 t.Fatal(err)
214 }
215 if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil || !strings.Contains(err.Error(), "root key") {
216 t.Fatalf("grant-bearing wrapper accepted: err = %v", err)
217 }
218}
219
220func TestSystemPointerTamperAndCrossChannelRefused(t *testing.T) {
221 priv, pub := testKeys(t)
222 s := NewDirStore(t.TempDir())
223 ctx := context.Background()
224 if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
225 t.Fatal(err)
226 }
227 body, _, _ := s.Get(ctx, SystemPointerKey("router", "canary"))
228 // Cross-channel replay: the canary wrapper planted at the stable key.
229 if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), body, ""); err != nil {
230 t.Fatal(err)
231 }
232 if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil {
233 t.Fatal("a canary pointer verified as stable")
234 }
235 // Tampered signature.
236 var sw signedWrapper
237 _ = json.Unmarshal(body, &sw)
238 sw.Sig = "AA" + sw.Sig[2:]
239 bad, _ := json.Marshal(sw)
240 _, etag, _ := s.Get(ctx, SystemPointerKey("router", "canary"))
241 if _, err := s.Put(ctx, SystemPointerKey("router", "canary"), bad, etag); err != nil {
242 t.Fatal(err)
243 }
244 if _, _, err := GetSystemPointer(ctx, s, pub, "router", "canary"); err == nil {
245 t.Fatal("tampered signature verified")
246 }
247}
248
249func TestSystemPointerValidation(t *testing.T) {
250 priv, _ := testKeys(t)
251 sg := RootSigner(priv)
252 ctx := context.Background()
253 cases := map[string]SystemRelease{
254 "dev version": {Version: "dev", Artifacts: testSystemRelease("x").Artifacts},
255 "no artifacts": {Version: "v1.0.0"},
256 "bad platform": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux_arm64": {Key: "bootstrap/router/v1.0.0/carlos-linux_arm64", SHA256: strings.Repeat("a", 64)}}},
257 "key not derived": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: "bootstrap/carlos", SHA256: strings.Repeat("a", 64)}}},
258 "short sha": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: SystemArtifactKey("v1.0.0", "linux-arm64"), SHA256: "abc"}}},
259 "version traversal": {Version: "../x", Artifacts: testSystemRelease("../x").Artifacts},
260 }
261 for name, rel := range cases {
262 s := NewDirStore(t.TempDir())
263 if _, err := PromoteSystem(ctx, s, sg, "router", "stable", rel, SystemPromoteOptions{}); err == nil {
264 t.Errorf("%s: accepted", name)
265 }
266 }
267 s := NewDirStore(t.TempDir())
268 if _, err := PromoteSystem(ctx, s, sg, "Router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err == nil {
269 t.Error("bad component accepted")
270 }
271 if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{Spread: 48 * time.Hour}); err == nil {
272 t.Error("48h spread accepted")
273 }
274}
275
276func TestSystemPromoteCASConflict(t *testing.T) {
277 priv, _ := testKeys(t)
278 ctx := context.Background()
279 inner := NewDirStore(t.TempDir())
280 // A store whose Put always says "someone else moved it".
281 s := conflictingStore{Store: inner}
282 if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); !errors.Is(err, ErrCASConflict) {
283 t.Fatalf("err = %v, want ErrCASConflict", err)
284 }
285}
286
287type conflictingStore struct{ Store }
288
289func (c conflictingStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
290 return "", ErrCASConflict
291}
292```
293
294- [ ] **Step 2: Run the tests to verify they fail**
295
296Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
297Expected: build failure naming `SystemPointer`, `PromoteSystem`, etc. as undefined.
298
299- [ ] **Step 3: Implement `internal/release/system.go`**
300
301```go
302package release
303
304import (
305 "context"
306 "crypto/ed25519"
307 "encoding/base64"
308 "encoding/json"
309 "errors"
310 "fmt"
311 "os"
312 "regexp"
313 "time"
314)
315
316// SystemPointer is the platform's own channel pointer: what a ChannelPointer
317// is for an app, for the thing that does the adopting. It lives under
318// control/, not apps/, because the substrate is not an app (the
319// console-as-an-app spec's dividing line), and it is ROOT-KEY-ONLY: a scoped
320// grant's rungs name app channels, and a key that can roll the platform is a
321// bigger key than any app promoter should hold. Spec §3.
322type SystemPointer struct {
323 Schema int `json:"schema"`
324 Component string `json:"component"`
325 Channel string `json:"channel"`
326 Current SystemRelease `json:"current"`
327 History []SystemRelease `json:"history,omitempty"`
328 Spread string `json:"spread"`
329 PromotedAt time.Time `json:"promoted_at"` // of THIS write; the box's replay guard reads only this
330 Audit []Promotion `json:"audit,omitempty"`
331}
332
333// SystemRelease is everything a box needs to install one version. It travels
334// whole into History so a rollback re-installs what was there, not a version
335// string with someone else's artifacts.
336type SystemRelease struct {
337 Version string `json:"version"`
338 Artifacts map[string]SystemArtifact `json:"artifacts"` // keyed by "<goos>-<goarch>"
339 Source string `json:"source,omitempty"`
340 ReleasedAt time.Time `json:"released_at"` // audit only; never compared
341}
342
343// SystemArtifact is one mirrored binary. Its sha in the signed pointer IS
344// the artifact's signature: boxes need no .sig object.
345type SystemArtifact struct {
346 Key string `json:"key"`
347 SHA256 string `json:"sha256"`
348}
349
350// SystemComponentRouter is the one component today: the substrate binary
351// (edge + activator + adoption, one process). A split adds a name here.
352const SystemComponentRouter = "router"
353
354// ErrNoSystemHistory is RollbackSystem's answer when the pointer has never
355// moved: there is nothing older to go back to.
356var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to")
357
358const (
359 systemSpreadMax = 24 * time.Hour
360 systemSpreadDefault = 5 * time.Minute
361)
362
363var (
364 systemComponentRe = regexp.MustCompile(`^[a-z][a-z0-9-]{0,31}$`)
365 systemPlatformRe = regexp.MustCompile(`^[a-z0-9]+-[a-z0-9]+$`)
366 systemSHA256Re = regexp.MustCompile(`^[0-9a-f]{64}$`)
367)
368
369// SystemPointerKey is control/system/<component>/<channel>.json.
370func SystemPointerKey(component, channel string) string {
371 return "control/system/" + component + "/" + channel + ".json"
372}
373
374// SystemArtifactKey is bootstrap/router/<version>/carlos-<platform>: under
375// bootstrap/ because the store door already means "the platform's own
376// binary" by that name, per-version because canary and stable boxes must be
377// able to run different binaries at once.
378func SystemArtifactKey(version, platform string) string {
379 return "bootstrap/router/" + version + "/carlos-" + platform
380}
381
382// SpreadDuration is Spread parsed, or the default when it is unparseable —
383// validation refuses the unparseable case on the way in, so this only
384// defaults for a hand-edited or pre-field pointer.
385func (p *SystemPointer) SpreadDuration() time.Duration {
386 d, err := time.ParseDuration(p.Spread)
387 if err != nil || d < 0 || d > systemSpreadMax {
388 return systemSpreadDefault
389 }
390 return d
391}
392
393func validateSystemRelease(r SystemRelease) error {
394 if !ValidVersion(r.Version) || r.Version == "dev" {
395 return fmt.Errorf("version %q is not a valid platform version", r.Version)
396 }
397 if len(r.Artifacts) == 0 {
398 return fmt.Errorf("version %s names no artifacts", r.Version)
399 }
400 for plat, a := range r.Artifacts {
401 if !systemPlatformRe.MatchString(plat) {
402 return fmt.Errorf("platform %q is not <goos>-<goarch>", plat)
403 }
404 if a.Key != SystemArtifactKey(r.Version, plat) {
405 return fmt.Errorf("artifact key %q is not %s", a.Key, SystemArtifactKey(r.Version, plat))
406 }
407 if !systemSHA256Re.MatchString(a.SHA256) {
408 return fmt.Errorf("artifact %s: sha256 %q is not 64 hex characters", a.Key, a.SHA256)
409 }
410 }
411 return nil
412}
413
414func validateSystemPointer(p *SystemPointer, wantComponent, wantChannel string) error {
415 if p.Schema != 1 {
416 return fmt.Errorf("system pointer schema %d unsupported", p.Schema)
417 }
418 if p.Component != wantComponent || p.Channel != wantChannel {
419 return fmt.Errorf("system pointer is for %s/%s, not %s/%s", p.Component, p.Channel, wantComponent, wantChannel)
420 }
421 if !systemComponentRe.MatchString(p.Component) || !ValidChannel(p.Channel) {
422 return fmt.Errorf("system pointer %s/%s: bad component or channel", p.Component, p.Channel)
423 }
424 if err := validateSystemRelease(p.Current); err != nil {
425 return fmt.Errorf("system pointer %s/%s: %w", p.Component, p.Channel, err)
426 }
427 if d, err := time.ParseDuration(p.Spread); err != nil || d < 0 || d > systemSpreadMax {
428 return fmt.Errorf("system pointer %s/%s: spread %q must be a duration between 0 and %s", p.Component, p.Channel, p.Spread, systemSpreadMax)
429 }
430 if p.PromotedAt.IsZero() {
431 return fmt.Errorf("system pointer %s/%s: promoted_at unset", p.Component, p.Channel)
432 }
433 if len(p.History) > historyMax {
434 return fmt.Errorf("system pointer %s/%s: history longer than %d", p.Component, p.Channel, historyMax)
435 }
436 for _, h := range p.History {
437 if err := validateSystemRelease(h); err != nil {
438 return fmt.Errorf("system pointer %s/%s: history: %w", p.Component, p.Channel, err)
439 }
440 }
441 if len(p.Audit) > auditMax {
442 return fmt.Errorf("system pointer %s/%s: audit longer than %d", p.Component, p.Channel, auditMax)
443 }
444 return nil
445}
446
447// signSystemPointer mirrors signPointer with two differences: the signer
448// must be the root key, and the wrapper never carries a grant.
449func signSystemPointer(sg Signer, p *SystemPointer) ([]byte, error) {
450 if err := sg.usable(); err != nil {
451 return nil, err
452 }
453 if sg.scope != nil {
454 return nil, errors.New("system pointer: signed by a scoped key — the platform takes the root key only")
455 }
456 p.Schema = 1
457 p.PromotedAt = time.Now().UTC().Truncate(time.Second)
458 if err := validateSystemPointer(p, p.Component, p.Channel); err != nil {
459 return nil, err
460 }
461 payload, err := json.MarshalIndent(p, "", " ")
462 if err != nil {
463 return nil, err
464 }
465 b64 := base64.StdEncoding.EncodeToString(payload)
466 sig := ed25519.Sign(sg.key, []byte(b64))
467 return json.MarshalIndent(signedWrapper{Payload: b64, Sig: base64.StdEncoding.EncodeToString(sig)}, "", " ")
468}
469
470// verifySystemPointer is the single parse site for system pointers. A grant
471// on the wire is refused before the signature is looked at (the production
472// flag's rule, production.go).
473func verifySystemPointer(pub ed25519.PublicKey, wrapper []byte, wantComponent, wantChannel string) (*SystemPointer, error) {
474 var sw signedWrapper
475 if err := json.Unmarshal(wrapper, &sw); err != nil {
476 return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err)
477 }
478 if sw.Grant != nil {
479 return nil, fmt.Errorf("system pointer %s/%s: signed by a scoped key — the platform takes the root key only", wantComponent, wantChannel)
480 }
481 sig, err := base64.StdEncoding.DecodeString(sw.Sig)
482 if err != nil {
483 return nil, fmt.Errorf("system pointer %s/%s: bad signature encoding", wantComponent, wantChannel)
484 }
485 if len(pub) != ed25519.PublicKeySize || !ed25519.Verify(pub, []byte(sw.Payload), sig) {
486 return nil, fmt.Errorf("system pointer %s/%s: signature does not verify", wantComponent, wantChannel)
487 }
488 payload, err := base64.StdEncoding.DecodeString(sw.Payload)
489 if err != nil {
490 return nil, fmt.Errorf("system pointer %s/%s: bad payload encoding", wantComponent, wantChannel)
491 }
492 var p SystemPointer
493 if err := json.Unmarshal(payload, &p); err != nil {
494 return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err)
495 }
496 if err := validateSystemPointer(&p, wantComponent, wantChannel); err != nil {
497 return nil, err
498 }
499 return &p, nil
500}
501
502// GetSystemPointer reads and verifies one channel's pointer. A missing key
503// is os.ErrNotExist, which is the normal state of every deployment before
504// its first `carlos system update`.
505func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error) {
506 body, etag, err := s.Get(ctx, SystemPointerKey(component, channel))
507 if err != nil {
508 return nil, "", err
509 }
510 p, err := verifySystemPointer(pub, body, component, channel)
511 if err != nil {
512 return nil, "", err
513 }
514 return p, etag, nil
515}
516
517// SystemPromoteOptions: who did it, and how long boxes spread their starts.
518type SystemPromoteOptions struct {
519 Actor string
520 Spread time.Duration // 0 = hotfix, every box at once; unset in the CLI means the default
521}
522
523// PromoteSystem writes the pointer for one channel, CAS-guarded. The
524// outgoing release goes whole into History unless it is the same version
525// (a re-promote of the running version is a no-op for boxes and must not
526// push that version into its own history).
527func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error) {
528 key := SystemPointerKey(component, channel)
529 var old *SystemPointer
530 body, etag, err := s.Get(ctx, key)
531 switch {
532 case err == nil:
533 if old, err = verifySystemPointer(sg.verifyPub(), body, component, channel); err != nil {
534 return nil, err
535 }
536 case errors.Is(err, os.ErrNotExist):
537 etag = ""
538 default:
539 return nil, err
540 }
541 next := &SystemPointer{Component: component, Channel: channel, Current: rel, Spread: o.Spread.String()}
542 if rel.ReleasedAt.IsZero() {
543 next.Current.ReleasedAt = time.Now().UTC().Truncate(time.Second)
544 }
545 from := ""
546 if old != nil {
547 next.History, next.Audit = old.History, old.Audit
548 from = old.Current.Version
549 if old.Current.Version != rel.Version {
550 next.History = prependCapped(old.Current, old.History, historyMax)
551 }
552 }
553 next.Audit = prependCapped(Promotion{Version: rel.Version, From: from, To: channel, Actor: o.Actor, At: time.Now().UTC()}, next.Audit, auditMax)
554 wrapper, err := signSystemPointer(sg, next)
555 if err != nil {
556 return nil, err
557 }
558 if _, err := s.Put(ctx, key, wrapper, etag); err != nil {
559 return nil, err
560 }
561 return next, nil
562}
563
564// RollbackSystem swaps Current with History[0] and writes a NEW pointer —
565// new PromotedAt — so every box's replay guard sees a promote, not a replay.
566func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error) {
567 key := SystemPointerKey(component, channel)
568 body, etag, err := s.Get(ctx, key)
569 if err != nil {
570 return nil, err
571 }
572 old, err := verifySystemPointer(sg.verifyPub(), body, component, channel)
573 if err != nil {
574 return nil, err
575 }
576 if len(old.History) == 0 {
577 return nil, ErrNoSystemHistory
578 }
579 next := &SystemPointer{
580 Component: component, Channel: channel,
581 Current: old.History[0], History: old.History[1:], Spread: old.Spread,
582 Audit: prependCapped(Promotion{Version: old.History[0].Version, From: old.Current.Version, To: channel, Actor: actor, At: time.Now().UTC()}, old.Audit, auditMax),
583 }
584 wrapper, err := signSystemPointer(sg, next)
585 if err != nil {
586 return nil, err
587 }
588 if _, err := s.Put(ctx, key, wrapper, etag); err != nil {
589 return nil, err
590 }
591 return next, nil
592}
593```
594
595Note `sg.verifyPub()` is the existing unexported method on `Signer` (`signer.go:66`); for a root signer it returns the root public key.
596
597- [ ] **Step 4: Run the tests**
598
599Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
600Expected: `ok`, rc=0. If `TestSystemPromoteCASConflict` fails because `conflictingStore` lacks a method, the `Store` interface grew — embed satisfies it; check the error text.
601
602- [ ] **Step 5: gofmt and commit**
603
604```bash
605gofmt -l internal/release/ && git add internal/release/system.go internal/release/system_test.go && git commit -q -F - <<'EOF'
606release: the platform's own channel pointer, root-key-only
607
608control/system/<component>/<channel>.json names a version and its
609per-platform artifacts. History carries whole releases so a rollback
610restores artifacts, and a rollback is a new promote (new PromotedAt).
611A grant on the wire is refused before the signature is read, like the
612production flag.
613
614Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
615Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
616EOF
617```
618
619---
620
621### Task 2: Streaming reads (`Opener`, `FetchToFile`)
622
623**Files:**
624- Modify: `internal/release/store.go` (after the `SizeLister` block, ~line 190)
625- Modify: `internal/fleetapi/fleetapi.go` (after `Get`, ~line 156)
626- Create: `internal/release/fetch.go`
627- Test: `internal/release/fetch_test.go`, `internal/fleetapi/fleetapi_test.go` (add one test)
628
629**Interfaces:**
630- Consumes: `Store`, `DirStore.path`, `S3Store.Client/Bucket`, `s3NotFound`, `prefixStore.full`; fleetapi `store.newRequest`, `objectURL`.
631- Produces:
632 ```go
633 // package release
634 type Opener interface{ Open(ctx context.Context, key string) (io.ReadCloser, string, error) }
635 func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error)
636 var ErrTooLarge = errors.New("store: object larger than the caller's limit")
637 func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) error
638 ```
639 `FetchToFile` writes `dst+".tmp"`, streams with a running sha, refuses more than `limit` bytes (`ErrTooLarge`), refuses a sha mismatch (error contains "sha256"), fsyncs, renames, fsyncs the directory, and removes the temp on any failure. Etag from `Open` may be `""` (DirStore).
640
641- [ ] **Step 1: Write the failing tests**
642
643`internal/release/fetch_test.go`:
644```go
645package release
646
647import (
648 "context"
649 "crypto/sha256"
650 "encoding/hex"
651 "errors"
652 "os"
653 "path/filepath"
654 "strings"
655 "testing"
656)
657
658func shaOf(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
659
660func TestFetchToFileStreamsAndVerifies(t *testing.T) {
661 ctx := context.Background()
662 body := []byte(strings.Repeat("carlos", 10000))
663 for name, s := range map[string]Store{
664 "dir": NewDirStore(t.TempDir()),
665 "prefix": NewPrefixStore(NewDirStore(t.TempDir()), "fleets/pi/"),
666 } {
667 t.Run(name, func(t *testing.T) {
668 if _, err := s.Put(ctx, "bootstrap/router/v1/carlos-linux-arm64", body, ""); err != nil {
669 t.Fatal(err)
670 }
671 dst := filepath.Join(t.TempDir(), "carlos")
672 if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst, int64(len(body)), shaOf(body)); err != nil {
673 t.Fatal(err)
674 }
675 got, _ := os.ReadFile(dst)
676 if string(got) != string(body) {
677 t.Fatal("content differs")
678 }
679 if _, err := os.Stat(dst + ".tmp"); !errors.Is(err, os.ErrNotExist) {
680 t.Fatal("temp file left behind")
681 }
682 // One byte over the limit is refused, and nothing is left on disk.
683 dst2 := filepath.Join(t.TempDir(), "carlos")
684 if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst2, int64(len(body))-1, shaOf(body)); !errors.Is(err, ErrTooLarge) {
685 t.Fatalf("over-limit err = %v", err)
686 }
687 if _, err := os.Stat(dst2); !errors.Is(err, os.ErrNotExist) {
688 t.Fatal("over-limit fetch left a file")
689 }
690 // A sha mismatch is refused and leaves nothing.
691 dst3 := filepath.Join(t.TempDir(), "carlos")
692 err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst3, int64(len(body)), strings.Repeat("0", 64))
693 if err == nil || !strings.Contains(err.Error(), "sha256") {
694 t.Fatalf("mismatch err = %v", err)
695 }
696 if _, err := os.Stat(dst3); !errors.Is(err, os.ErrNotExist) {
697 t.Fatal("mismatch fetch left a file")
698 }
699 // Missing key is ErrNotExist.
700 if err := FetchToFile(ctx, s, "bootstrap/router/v9/carlos-linux-arm64", dst3, 10, shaOf(nil)); !errors.Is(err, os.ErrNotExist) {
701 t.Fatalf("missing err = %v", err)
702 }
703 })
704 }
705}
706
707func TestPrefixStoreOpenRebasesTheKey(t *testing.T) {
708 // prefixStore embeds the Store INTERFACE, so a promoted Open would forward
709 // the raw key and silently escape the prefix. It must be written out.
710 ctx := context.Background()
711 inner := NewDirStore(t.TempDir())
712 if _, err := inner.Put(ctx, "fleets/pi/x", []byte("in"), ""); err != nil {
713 t.Fatal(err)
714 }
715 if _, err := inner.Put(ctx, "x", []byte("OUT"), ""); err != nil {
716 t.Fatal(err)
717 }
718 rc, _, err := OpenObject(ctx, NewPrefixStore(inner, "fleets/pi/"), "x")
719 if err != nil {
720 t.Fatal(err)
721 }
722 defer rc.Close()
723 b := make([]byte, 8)
724 n, _ := rc.Read(b)
725 if string(b[:n]) != "in" {
726 t.Fatalf("read %q through the prefix, want \"in\"", b[:n])
727 }
728}
729
730func TestOpenObjectFallsBackToGet(t *testing.T) {
731 // A Store that is not an Opener still works: OpenObject buffers via Get.
732 s := getOnly{NewDirStore(t.TempDir())}
733 if _, err := s.Put(context.Background(), "k", []byte("v"), ""); err != nil {
734 t.Fatal(err)
735 }
736 rc, _, err := OpenObject(context.Background(), s, "k")
737 if err != nil {
738 t.Fatal(err)
739 }
740 b := make([]byte, 4)
741 n, _ := rc.Read(b)
742 if string(b[:n]) != "v" {
743 t.Fatalf("got %q", b[:n])
744 }
745}
746
747type getOnly struct{ *DirStore }
748```
749(`getOnly` embeds `*DirStore`, so it has `Open` promoted — to make it a non-Opener, shadow it:)
750```go
751func (getOnly) Open() {} // different signature: getOnly is NOT an Opener
752```
753
754`internal/fleetapi/fleetapi_test.go`, add:
755```go
756func TestOpenStreamsThroughTheDoor(t *testing.T) {
757 d := newTestDoor(t)
758 body := bytes.Repeat([]byte("x"), 1<<20)
759 d.seed(t, "bootstrap/carlos", body)
760 c := d.box(t)
761 op, ok := c.(release.Opener)
762 if !ok {
763 t.Fatal("fleetapi store is not an Opener")
764 }
765 rc, _, err := op.Open(context.Background(), "bootstrap/carlos")
766 if err != nil {
767 t.Fatal(err)
768 }
769 defer rc.Close()
770 got, _ := io.ReadAll(rc)
771 if !bytes.Equal(got, body) {
772 t.Fatalf("read %d bytes, want %d", len(got), len(body))
773 }
774 if _, _, err := op.Open(context.Background(), "bootstrap/nothing"); !errors.Is(err, os.ErrNotExist) {
775 t.Fatalf("missing: %v", err)
776 }
777}
778```
779(`newTestDoor`, `seed`, `box` exist in that file, `fleetapi_test.go:52-105`; `bootstrap/carlos` is already readable by a box.)
780
781- [ ] **Step 2: Run to verify failure**
782
783Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ -run 'Fetch|Open' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
784Expected: undefined `Opener`, `OpenObject`, `FetchToFile`, `ErrTooLarge`.
785
786- [ ] **Step 3: Implement**
787
788`internal/release/store.go`, after the `SizeLister` assertions (~line 190):
789```go
790// Opener is the optional streaming read. Store.Get returns the whole object
791// in memory, which is fine for a pointer and wrong for a platform binary on
792// a box under MemoryHigh=70%. Implemented by every production store; a test
793// fake that only has Get still works through OpenObject. The etag may be ""
794// (DirStore cannot know it without reading the file).
795type Opener interface {
796 Open(ctx context.Context, key string) (io.ReadCloser, string, error)
797}
798
799var (
800 _ Opener = (*DirStore)(nil)
801 _ Opener = (*S3Store)(nil)
802 _ Opener = prefixStore{}
803)
804
805// OpenObject streams key from s, buffering through Get when s cannot stream.
806func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error) {
807 if o, ok := s.(Opener); ok {
808 return o.Open(ctx, key)
809 }
810 b, etag, err := s.Get(ctx, key)
811 if err != nil {
812 return nil, "", err
813 }
814 return io.NopCloser(bytes.NewReader(b)), etag, nil
815}
816```
817`DirStore`:
818```go
819func (d *DirStore) Open(_ context.Context, key string) (io.ReadCloser, string, error) {
820 p, err := d.path(key)
821 if err != nil {
822 return nil, "", err
823 }
824 f, err := os.Open(p)
825 if err != nil {
826 if errors.Is(err, os.ErrNotExist) {
827 return nil, "", fmt.Errorf("dirstore: %s: %w", key, os.ErrNotExist)
828 }
829 return nil, "", err
830 }
831 return f, "", nil
832}
833```
834`S3Store`:
835```go
836func (s *S3Store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
837 out, err := s.Client.GetObject(ctx, &awss3.GetObjectInput{Bucket: &s.Bucket, Key: &key})
838 if err != nil {
839 if s3NotFound(err) {
840 return nil, "", fmt.Errorf("s3store: %s: %w", key, os.ErrNotExist)
841 }
842 return nil, "", err
843 }
844 return out.Body, aws.ToString(out.ETag), nil
845}
846```
847`prefixStore` (explicit, next to its `Get`):
848```go
849// Open is written out, not promoted: the embedded field is the interface,
850// and a promoted Open would forward the UNREBASED key past the prefix.
851func (p prefixStore) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
852 return OpenObject(ctx, p.Store, p.full(key))
853}
854```
855Add `"bytes"` and `"io"` to the imports if absent.
856
857`internal/fleetapi/fleetapi.go`, after `Get`:
858```go
859// Open streams one object. Unlike Get it does not buffer the body, which
860// is what a 25 MiB platform binary on a Pi needs; the 404 mapping is the
861// same as Get's so callers can errors.Is(err, os.ErrNotExist).
862func (s *store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
863 req, err := s.newRequest(ctx, http.MethodGet, s.objectURL(key), nil)
864 if err != nil {
865 return nil, "", err
866 }
867 resp, err := s.client.Do(req)
868 if err != nil {
869 return nil, "", errorf("open %s: %w", key, err)
870 }
871 switch resp.StatusCode {
872 case http.StatusOK:
873 return resp.Body, resp.Header.Get("ETag"), nil
874 case http.StatusNotFound:
875 resp.Body.Close()
876 return nil, "", fmt.Errorf("fleetapi: %s: %w", key, os.ErrNotExist)
877 default:
878 resp.Body.Close()
879 return nil, "", errorf("open %s: unexpected status %d", key, resp.StatusCode)
880 }
881}
882
883var _ release.Opener = (*store)(nil)
884```
885
886`internal/release/fetch.go`:
887```go
888package release
889
890import (
891 "context"
892 "crypto/sha256"
893 "encoding/hex"
894 "errors"
895 "fmt"
896 "io"
897 "os"
898 "path/filepath"
899)
900
901// ErrTooLarge: the object exceeded the caller's limit. The reader is cut at
902// limit+1 bytes, so an object EXACTLY at the limit is told from one over it.
903var ErrTooLarge = errors.New("store: object larger than the caller's limit")
904
905// FetchToFile streams key into dst with a running sha256, never holding the
906// object in memory: temp file, fsync, rename, directory fsync. Any failure
907// removes the temp and leaves dst untouched.
908func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) (err error) {
909 rc, _, err := OpenObject(ctx, s, key)
910 if err != nil {
911 return err
912 }
913 defer rc.Close()
914 tmp := dst + ".tmp"
915 f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
916 if err != nil {
917 return err
918 }
919 defer func() {
920 if err != nil {
921 f.Close()
922 os.Remove(tmp)
923 }
924 }()
925 h := sha256.New()
926 n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, limit+1))
927 if err != nil {
928 return fmt.Errorf("fetch %s: %w", key, err)
929 }
930 if n > limit {
931 return fmt.Errorf("fetch %s: %w (limit %d)", key, ErrTooLarge, limit)
932 }
933 if got := hex.EncodeToString(h.Sum(nil)); got != wantSHA256 {
934 return fmt.Errorf("fetch %s: sha256 %s does not match the signed %s", key, got, wantSHA256)
935 }
936 if err = f.Sync(); err != nil {
937 return err
938 }
939 if err = f.Close(); err != nil {
940 return err
941 }
942 if err = os.Rename(tmp, dst); err != nil {
943 return err
944 }
945 return syncDir(filepath.Dir(dst))
946}
947
948// syncDir makes a rename durable: the directory entry is what fsync(file)
949// does not cover.
950func syncDir(dir string) error {
951 d, err := os.Open(dir)
952 if err != nil {
953 return err
954 }
955 defer d.Close()
956 return d.Sync()
957}
958```
959
960- [ ] **Step 4: Run the tests**
961
962Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
963Expected: all `ok`. (`internal/edge` is included because its test fakes assign to `release.Store`; the interface did not change, so they still compile.)
964
965- [ ] **Step 5: Commit**
966
967```bash
968gofmt -l internal/ && git add internal/release/store.go internal/release/fetch.go internal/release/fetch_test.go internal/fleetapi/fleetapi.go internal/fleetapi/fleetapi_test.go && git commit -q -F - <<'EOF'
969release: stream an object to a file with a running sha
970
971Opener is a side-interface (48 test fakes implement Store; none break),
972implemented by DirStore, S3Store, prefixStore and the fleet door client.
973FetchToFile is what a box uses for a platform binary: limit+1 read, sha
974checked before rename, directory fsync after.
975
976Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
977Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
978EOF
979```
980
981---
982
983### Task 3: Store-door grants for the pointer and artifact shapes
984
985**Files:**
986- Modify: `internal/edge/storedoor.go:383-397` (`aclRead`) and the doc comment above it (`:352-382`)
987- Modify: `internal/edge/storedoor_test.go` (`TestStoreDoorACLMatrix` rows at `:69+`, `TestStoreDoorGetServesOwnEstate` at `:365`, the prose in `TestDoneNeverServesCertMaterial`'s comment at `:519`)
988
989**Interfaces:**
990- Consumes: `aclRead(sqid, fleet, key string) bool`; `canonicalKey` already refuses any `..`/`.`/empty segment before the ACL runs, so traversal never reaches here.
991- Produces: `func systemReadKey(key string) bool` (unexported) — true for exactly `control/system/<c>/<ch>.json` and `bootstrap/router/<v>/<asset>`.
992
993- [ ] **Step 1: Add the failing matrix rows and GET cases**
994
995In `TestStoreDoorACLMatrix`'s table, after the `bootstrap/carlos.sig` row (`:84`):
996```go
997 {"control/system/router/stable.json", opRead, true, "the platform channel pointer (system update spec §3)"},
998 {"control/system/router/canary.json", opRead, true, "…and the canary one"},
999 {"control/system/router/stable.json", opWrite, false, "a box never writes a pointer"},
1000 {"control/system/router", opRead, false, "the pointer shape is exactly three segments ending .json"},
1001 {"control/system/router/stable.json/x", opRead, false, "…not four"},
1002 {"control/system/router/stable.txt", opRead, false, "…and .json"},
1003 {"bootstrap/router/v0.18.0/carlos-linux-arm64", opRead, true, "a mirrored platform artifact"},
1004 {"bootstrap/router/v0.18.0/carlos-linux-arm64", opWrite, false, "a box must never publish the platform binary"},
1005 {"bootstrap/router/v0.18.0", opRead, false, "the artifact shape is exactly three segments"},
1006 {"bootstrap/router/v0.18.0/a/b", opRead, false, "…not four"},
1007 {"bootstrap/router.sig", opRead, false, "nothing else under bootstrap/"},
1008 {"bootstrap/router/../../x", opRead, false, "traversal is refused by canonicalKey before the ACL"},
1009 {"control/system/../serial", opRead, false, "…on both prefixes"},
1010```
1011In `TestStoreDoorGetServesOwnEstate`, seed two more objects with `seedObject` and add their keys to the loop that expects 200:
1012```go
1013 seedObject(t, store, "control/system/router/stable.json", []byte(`{"payload":"x"}`))
1014 seedObject(t, store, "bootstrap/router/v1/carlos-linux-arm64", []byte("ELF"))
1015```
1016Update the comment at `storedoor_test.go:519` (the allowlist prose) to add "`control/system/<c>/<ch>.json`, `bootstrap/router/<v>/<asset>`".
1017
1018- [ ] **Step 2: Run to verify failure**
1019
1020Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -run 'TestStoreDoorACLMatrix|TestStoreDoorGetServesOwnEstate' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
1021Expected: FAIL on the four `want true` rows and the two new GET keys (404).
1022
1023- [ ] **Step 3: Implement**
1024
1025In `storedoor.go`, replace the `switch key` in `aclRead` with:
1026```go
1027 switch key {
1028 case "control/serial", "bootstrap/carlos":
1029 return true
1030 }
1031 if systemReadKey(key) {
1032 return true
1033 }
1034```
1035and add, below `aclRead`:
1036```go
1037// systemReadKey is the two shapes the system-update spec (§3) grants every
1038// fleet box: its platform channel pointer and a mirrored platform artifact.
1039// Shapes, not prefixes — exactly three segments each, so nothing else that
1040// ever lands under control/ or bootstrap/ is readable by a box. Listing is
1041// not granted (aclDir is unchanged), and canonicalKey has already refused
1042// any "." or ".." segment before this runs.
1043func systemReadKey(key string) bool {
1044 if rest, ok := strings.CutPrefix(key, "control/system/"); ok {
1045 comp, file, ok := strings.Cut(rest, "/")
1046 return ok && comp != "" && file != "" && !strings.Contains(file, "/") &&
1047 strings.HasSuffix(file, ".json") && len(file) > len(".json")
1048 }
1049 if rest, ok := strings.CutPrefix(key, "bootstrap/router/"); ok {
1050 ver, asset, ok := strings.Cut(rest, "/")
1051 return ok && ver != "" && asset != "" && !strings.Contains(asset, "/")
1052 }
1053 return false
1054}
1055```
1056Extend the `aclRead` doc comment's allowlist enumeration (`:352-382`) with the two shapes and the sentence: "`bootstrap/carlos` stays until `update-platform-binary.sh` is retired (spec §10)."
1057
1058- [ ] **Step 4: Run the edge tests**
1059
1060Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
1061Expected: `ok`.
1062
1063- [ ] **Step 5: Commit**
1064
1065```bash
1066gofmt -l internal/edge/ && git add internal/edge/storedoor.go internal/edge/storedoor_test.go && git commit -q -F - <<'EOF'
1067edge: the store door grants a box its platform pointer and artifact
1068
1069Two exact shapes under control/system/ and bootstrap/router/, no
1070listing, no writes. Without the pointer grant a door-backed fleet box
1071would never see a platform promote.
1072
1073Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1074Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
1075EOF
1076```
1077
1078---
1079### Task 4: On-box state files (`internal/host/systemstate.go`)
1080
1081**Files:**
1082- Create: `internal/host/systemstate.go`
1083- Test: `internal/host/systemstate_test.go`
1084
1085**Interfaces:**
1086- Consumes: nothing new.
1087- Produces (package `host`, unexported unless noted):
1088 ```go
1089 const (
1090 systemComponent = "router"
1091 systemRequestMarker = "system-update-requested" // touched under dataDir
1092 systemRequestEvery = time.Minute
1093 systemArtifactLimit = int64(96 << 20)
1094 systemOutcomeApplied = "applied"; systemOutcomeRolledBack = "rolled-back"; systemOutcomeRollbackFailed = "rollback-failed"
1095 systemOutcomeRefused = "refused"; systemOutcomeDeferred = "deferred"; systemOutcomeNoted = "noted"
1096 systemPhaseBackingUp = "backing-up"; systemPhaseInstalling = "installing"; systemPhaseInstalled = "installed"
1097 systemPhaseUnits = "units"; systemPhaseRestarted = "restarted"; systemPhaseGated = "gated"
1098 )
1099 type systemRequest struct{ Version string; PromotedAt time.Time; NoOp bool; SeenAt time.Time }
1100 type systemResult struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time }
1101 type systemApplied struct{ Version, Channel string; At time.Time }
1102 type systemSeen map[string]time.Time
1103 type systemJournal struct{ Phase, Target, PrevBuild string; UnitsAdded, UnitsRemoved []string; StartedAt time.Time }
1104 func systemDataDir(dd string) string // <dd>/system/router
1105 func systemRequestPath(dd string) string // <dd>/system/router/requested.json
1106 func systemCachePath(dd, version string) string // <dd>/system/router/<version>/carlos
1107 func systemRootDir(optDir string) string // <optDir>/system/router
1108 func systemPrevPath(optDir string) string // <optDir>/carlos.prev
1109 func systemBinaryPath(optDir string) string // <optDir>/carlos
1110 func writeFileDurable(path string, data []byte, mode os.FileMode) error
1111 func writeJSONDurable(path string, v any, mode os.FileMode) error
1112 func readJSONFile(path string, v any) error // os.ErrNotExist passthrough
1113 func readSystemResult(optDir string) (*systemResult, error)
1114 func readSystemSeen(optDir string) (systemSeen, error) // missing file = empty map, nil error
1115 func readSystemApplied(optDir string) (*systemApplied, error)
1116 func readSystemJournal(optDir string) (*systemJournal, error)
1117 func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool
1118 func (r *systemResult) terminal() bool // applied|rolled-back|rollback-failed|refused
1119 func validSystemVersion(v string) bool // ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$
1120 ```
1121 The root files are `result.json`, `seen.json`, `applied.json`, `journal.json` directly under `systemRootDir`.
1122
1123- [ ] **Step 1: Write the failing tests**
1124
1125```go
1126package host
1127
1128import (
1129 "errors"
1130 "os"
1131 "path/filepath"
1132 "testing"
1133 "time"
1134)
1135
1136func TestSystemPathsAreDerivedNotTrusted(t *testing.T) {
1137 if got := systemCachePath("/data", "v1.2.3"); got != "/data/system/router/v1.2.3/carlos" {
1138 t.Fatalf("cache path = %s", got)
1139 }
1140 if got := systemRequestPath("/data"); got != "/data/system/router/requested.json" {
1141 t.Fatalf("request path = %s", got)
1142 }
1143 if got := systemRootDir("/opt/carlos"); got != "/opt/carlos/system/router" {
1144 t.Fatalf("root dir = %s", got)
1145 }
1146 for _, bad := range []string{"", ".", "..", "../x", "a/b", ".hidden", "v1 2", string(make([]byte, 65))} {
1147 if validSystemVersion(bad) {
1148 t.Errorf("validSystemVersion(%q) = true", bad)
1149 }
1150 }
1151 for _, good := range []string{"v0.18.0", "66587fa", "a", "release-2026.09.06_rc1"} {
1152 if !validSystemVersion(good) {
1153 t.Errorf("validSystemVersion(%q) = false", good)
1154 }
1155 }
1156}
1157
1158func TestWriteFileDurableLeavesWholeFileOrNothing(t *testing.T) {
1159 dir := t.TempDir()
1160 p := filepath.Join(dir, "journal.json")
1161 if err := writeFileDurable(p, []byte("one"), 0o644); err != nil {
1162 t.Fatal(err)
1163 }
1164 if err := writeFileDurable(p, []byte("two"), 0o644); err != nil {
1165 t.Fatal(err)
1166 }
1167 b, _ := os.ReadFile(p)
1168 if string(b) != "two" {
1169 t.Fatalf("content = %q", b)
1170 }
1171 if _, err := os.Stat(p + ".tmp"); !errors.Is(err, os.ErrNotExist) {
1172 t.Fatal("temp left behind")
1173 }
1174 st, _ := os.Stat(p)
1175 if st.Mode().Perm() != 0o644 {
1176 t.Fatalf("mode = %v", st.Mode().Perm())
1177 }
1178 // A directory that does not exist is an error, not a silent no-op.
1179 if err := writeFileDurable(filepath.Join(dir, "no", "such", "f"), []byte("x"), 0o644); err == nil {
1180 t.Fatal("wrote into a missing directory")
1181 }
1182}
1183
1184func TestSystemStateReadersRoundTrip(t *testing.T) {
1185 opt := t.TempDir()
1186 root := systemRootDir(opt)
1187 if err := os.MkdirAll(root, 0o755); err != nil {
1188 t.Fatal(err)
1189 }
1190 if r, err := readSystemResult(opt); err != nil || r != nil {
1191 t.Fatalf("missing result: %v %v", r, err)
1192 }
1193 seen, err := readSystemSeen(opt)
1194 if err != nil || len(seen) != 0 {
1195 t.Fatalf("missing seen: %v %v", seen, err)
1196 }
1197 at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
1198 res := systemResult{Channel: "stable", Version: "v1", PromotedAt: at, Outcome: systemOutcomeRolledBack, Reason: "gate", At: at}
1199 if err := writeJSONDurable(filepath.Join(root, "result.json"), res, 0o644); err != nil {
1200 t.Fatal(err)
1201 }
1202 got, err := readSystemResult(opt)
1203 if err != nil || !got.matches("stable", "v1", at) || !got.terminal() {
1204 t.Fatalf("result = %+v err=%v", got, err)
1205 }
1206 if got.matches("canary", "v1", at) || got.matches("stable", "v2", at) || got.matches("stable", "v1", at.Add(time.Second)) {
1207 t.Fatal("matches ignores part of the tuple")
1208 }
1209 got.Outcome = systemOutcomeDeferred
1210 if got.terminal() {
1211 t.Fatal("deferred is not terminal")
1212 }
1213 got.Outcome = systemOutcomeNoted
1214 if got.terminal() {
1215 t.Fatal("noted is not terminal for the leg: the version may still differ next time")
1216 }
1217 seen["stable"] = at
1218 if err := writeJSONDurable(filepath.Join(root, "seen.json"), seen, 0o644); err != nil {
1219 t.Fatal(err)
1220 }
1221 seen2, _ := readSystemSeen(opt)
1222 if !seen2["stable"].Equal(at) {
1223 t.Fatalf("seen = %v", seen2)
1224 }
1225 j := systemJournal{Phase: systemPhaseInstalled, Target: "v1", PrevBuild: "v0", UnitsAdded: []string{"a.path"}, StartedAt: at}
1226 if err := writeJSONDurable(filepath.Join(root, "journal.json"), j, 0o600); err != nil {
1227 t.Fatal(err)
1228 }
1229 j2, err := readSystemJournal(opt)
1230 if err != nil || j2.Phase != systemPhaseInstalled || j2.UnitsAdded[0] != "a.path" {
1231 t.Fatalf("journal = %+v err=%v", j2, err)
1232 }
1233}
1234```
1235
1236- [ ] **Step 2: Run to verify failure**
1237
1238Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
1239Expected: undefined identifiers.
1240
1241- [ ] **Step 3: Implement `internal/host/systemstate.go`**
1242
1243```go
1244package host
1245
1246import (
1247 "encoding/json"
1248 "errors"
1249 "os"
1250 "path/filepath"
1251 "regexp"
1252 "time"
1253)
1254
1255// The on-box state of the platform self-update (spec §4, §6). Two homes:
1256//
1257// <data>/system/router/ — written by the sandboxed agent: the
1258// request and the artifact cache. Root
1259// reads exactly one file here and trusts
1260// nothing in it beyond a version string.
1261// <opt>/system/router/ — written by root: journal, seen (replay
1262// watermarks), applied (the .prev
1263// authority), result (what the agent
1264// publishes). Root-owned, world-readable.
1265const (
1266 systemComponent = "router"
1267 systemRequestMarker = "system-update-requested"
1268 systemRequestEvery = time.Minute
1269 systemArtifactLimit = int64(96 << 20)
1270
1271 systemOutcomeApplied = "applied"
1272 systemOutcomeRolledBack = "rolled-back"
1273 systemOutcomeRollbackFailed = "rollback-failed"
1274 systemOutcomeRefused = "refused"
1275 systemOutcomeDeferred = "deferred"
1276 systemOutcomeNoted = "noted"
1277
1278 systemPhaseBackingUp = "backing-up"
1279 systemPhaseInstalling = "installing"
1280 systemPhaseInstalled = "installed"
1281 systemPhaseUnits = "units"
1282 systemPhaseRestarted = "restarted"
1283 systemPhaseGated = "gated"
1284)
1285
1286// systemRequest is the agent's hint to root: which promote it saw. No
1287// channel (root's own host.env says), no key, no sha, no path (root derives
1288// the cache path from the version and takes key and sha from the pointer).
1289type systemRequest struct {
1290 Version string `json:"version"`
1291 PromotedAt time.Time `json:"promoted_at"`
1292 NoOp bool `json:"noop"`
1293 SeenAt time.Time `json:"seen_at"`
1294}
1295
1296// systemResult is root's answer to the last request, keyed by the REQUEST's
1297// tuple so a refusal of one promote never suppresses the next.
1298type systemResult struct {
1299 Channel string `json:"channel"`
1300 Version string `json:"version"`
1301 PromotedAt time.Time `json:"promoted_at"`
1302 Outcome string `json:"outcome"`
1303 Reason string `json:"reason,omitempty"`
1304 At time.Time `json:"at"`
1305}
1306
1307func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool {
1308 return r != nil && r.Channel == channel && r.Version == version && r.PromotedAt.Equal(promotedAt)
1309}
1310
1311// terminal: outcomes the leg must not retry for the same pointer.
1312func (r *systemResult) terminal() bool {
1313 switch r.Outcome {
1314 case systemOutcomeApplied, systemOutcomeRolledBack, systemOutcomeRollbackFailed, systemOutcomeRefused:
1315 return true
1316 }
1317 return false
1318}
1319
1320// systemApplied is the last build that PASSED A GATE on this box, forward or
1321// rollback. It is the only thing that may authorise refreshing carlos.prev.
1322type systemApplied struct {
1323 Version string `json:"version"`
1324 Channel string `json:"channel"`
1325 At time.Time `json:"at"`
1326}
1327
1328// systemSeen is the replay watermark per channel: the newest PromotedAt root
1329// has verified. Advanced on every verified pointer, including no-ops.
1330type systemSeen map[string]time.Time
1331
1332// systemJournal is the open transaction. Written before every box-changing
1333// step, removed last; a run that finds one recovers before anything else.
1334type systemJournal struct {
1335 Phase string `json:"phase"`
1336 Target string `json:"target"`
1337 PrevBuild string `json:"prev_build"`
1338 UnitsAdded []string `json:"units_added,omitempty"`
1339 UnitsRemoved []string `json:"units_removed,omitempty"`
1340 StartedAt time.Time `json:"started_at"`
1341}
1342
1343var systemVersionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`)
1344
1345// validSystemVersion is what makes the cache path derivable: no leading dot,
1346// no separator, nothing that could be a traversal.
1347func validSystemVersion(v string) bool { return systemVersionRe.MatchString(v) }
1348
1349func systemDataDir(dd string) string { return filepath.Join(dd, "system", systemComponent) }
1350func systemRequestPath(dd string) string {
1351 return filepath.Join(systemDataDir(dd), "requested.json")
1352}
1353func systemCachePath(dd, version string) string {
1354 return filepath.Join(systemDataDir(dd), version, "carlos")
1355}
1356func systemRootDir(optDir string) string { return filepath.Join(optDir, "system", systemComponent) }
1357func systemPrevPath(optDir string) string { return filepath.Join(optDir, "carlos.prev") }
1358func systemBinaryPath(optDir string) string { return filepath.Join(optDir, "carlos") }
1359
1360// writeFileDurable: temp, fsync, rename, directory fsync. A crash leaves the
1361// old file or the new one, never a torn one, and the rename itself is on
1362// disk before the caller's next step depends on it.
1363func writeFileDurable(path string, data []byte, mode os.FileMode) (err error) {
1364 tmp := path + ".tmp"
1365 f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode)
1366 if err != nil {
1367 return err
1368 }
1369 defer func() {
1370 if err != nil {
1371 f.Close()
1372 os.Remove(tmp)
1373 }
1374 }()
1375 if _, err = f.Write(data); err != nil {
1376 return err
1377 }
1378 if err = f.Chmod(mode); err != nil {
1379 return err
1380 }
1381 if err = f.Sync(); err != nil {
1382 return err
1383 }
1384 if err = f.Close(); err != nil {
1385 return err
1386 }
1387 if err = os.Rename(tmp, path); err != nil {
1388 return err
1389 }
1390 d, err := os.Open(filepath.Dir(path))
1391 if err != nil {
1392 return err
1393 }
1394 defer d.Close()
1395 return d.Sync()
1396}
1397
1398func writeJSONDurable(path string, v any, mode os.FileMode) error {
1399 b, err := json.MarshalIndent(v, "", " ")
1400 if err != nil {
1401 return err
1402 }
1403 return writeFileDurable(path, append(b, '\n'), mode)
1404}
1405
1406// readJSONFile leaves os.ErrNotExist intact so callers can treat "no file"
1407// as a state, not a failure.
1408func readJSONFile(path string, v any) error {
1409 b, err := os.ReadFile(path)
1410 if err != nil {
1411 return err
1412 }
1413 return json.Unmarshal(b, v)
1414}
1415
1416func readSystemResult(optDir string) (*systemResult, error) {
1417 var r systemResult
1418 err := readJSONFile(filepath.Join(systemRootDir(optDir), "result.json"), &r)
1419 if errors.Is(err, os.ErrNotExist) {
1420 return nil, nil
1421 }
1422 return &r, err
1423}
1424
1425func readSystemSeen(optDir string) (systemSeen, error) {
1426 s := systemSeen{}
1427 err := readJSONFile(filepath.Join(systemRootDir(optDir), "seen.json"), &s)
1428 if errors.Is(err, os.ErrNotExist) {
1429 return s, nil
1430 }
1431 return s, err
1432}
1433
1434func readSystemApplied(optDir string) (*systemApplied, error) {
1435 var a systemApplied
1436 err := readJSONFile(filepath.Join(systemRootDir(optDir), "applied.json"), &a)
1437 if errors.Is(err, os.ErrNotExist) {
1438 return nil, nil
1439 }
1440 return &a, err
1441}
1442
1443func readSystemJournal(optDir string) (*systemJournal, error) {
1444 var j systemJournal
1445 err := readJSONFile(filepath.Join(systemRootDir(optDir), "journal.json"), &j)
1446 if errors.Is(err, os.ErrNotExist) {
1447 return nil, nil
1448 }
1449 return &j, err
1450}
1451```
1452
1453- [ ] **Step 4: Run the tests**
1454
1455Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
1456Expected: PASS.
1457
1458- [ ] **Step 5: Commit**
1459
1460```bash
1461gofmt -l internal/host/ && git add internal/host/systemstate.go internal/host/systemstate_test.go && git commit -q -F - <<'EOF'
1462host: the on-box state of a platform self-update
1463
1464Request (agent-written hint), result, seen watermarks, applied, and
1465the journal, with durable writes: temp, fsync, rename, directory
1466fsync. Cache paths are derived from a validated version string.
1467
1468Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1469Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
1470EOF
1471```
1472
1473---
1474
1475### Task 5: The heartbeat learns platform, channel and last apply outcome
1476
1477**Files:**
1478- Modify: `internal/status/snapshot.go` — `Probes` (~`:98`, next to `Release`), `BoxMetrics` (~`:158`), `CollectBox` (~`:404`, where `Release: p.Release` is copied), the emptiness guard (~`:543`)
1479- Modify: `internal/host/agent.go` `statusLoop` (~`:2318`, the `Release: version.Build,` line)
1480- Create: `internal/host/systemprobe.go`
1481- Test: `internal/status/snapshot_test.go` (add), `internal/host/systemprobe_test.go`
1482
1483**Interfaces:**
1484- Consumes: `readSystemResult` (Task 4), `release.OptDir()`, `status.Probes`, `status.BoxMetrics`.
1485- Produces:
1486 ```go
1487 // package status
1488 type SystemState struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time }
1489 // Probes gains: Platform string; SystemChannel string; System *SystemState
1490 // BoxMetrics gains (json): platform, system_channel, system
1491 // package host
1492 func systemChannel(getenv func(string) string) string // CARLOS_SYSTEM_CHANNEL, default "stable"
1493 func systemStateProbe(optDir string) *status.SystemState // nil when no result.json
1494 const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL"
1495 ```
1496
1497- [ ] **Step 1: Write the failing tests**
1498
1499`internal/status/snapshot_test.go`, add:
1500```go
1501func TestBoxMetricsCarryThePlatformSelfUpdateState(t *testing.T) {
1502 at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
1503 p := Probes{Release: "v1", Platform: "linux-arm64", SystemChannel: "canary",
1504 System: &SystemState{Channel: "canary", Version: "v2", PromotedAt: at, Outcome: "rolled-back", Reason: "gate", At: at}}
1505 b := CollectBox(nil, p, at)
1506 if b.Metrics == nil || b.Metrics.Platform != "linux-arm64" || b.Metrics.SystemChannel != "canary" || b.Metrics.System == nil || b.Metrics.System.Outcome != "rolled-back" {
1507 t.Fatalf("metrics = %+v", b.Metrics)
1508 }
1509 out, _ := json.Marshal(b)
1510 for _, want := range []string{`"platform":"linux-arm64"`, `"system_channel":"canary"`, `"outcome":"rolled-back"`} {
1511 if !strings.Contains(string(out), want) {
1512 t.Errorf("json lacks %s: %s", want, out)
1513 }
1514 }
1515 // A box with only the new fields is not "empty".
1516 b2 := CollectBox(nil, Probes{Platform: "linux-arm64"}, at)
1517 if b2.Metrics == nil {
1518 t.Fatal("platform alone must produce a metrics block")
1519 }
1520 // The public fleet feed still never carries the metrics block.
1521 f := BuildFleet([]Box{b}, at)
1522 fb, _ := json.Marshal(f)
1523 if strings.Contains(string(fb), "system_channel") || strings.Contains(string(fb), "rolled-back") {
1524 t.Fatalf("fleet.json leaked the private metrics: %s", fb)
1525 }
1526}
1527```
1528(Check `BuildFleet`'s exact signature at `fleet.go:157` and adjust the call; it takes the box list and a time.)
1529
1530`internal/host/systemprobe_test.go`:
1531```go
1532package host
1533
1534import (
1535 "os"
1536 "path/filepath"
1537 "testing"
1538 "time"
1539)
1540
1541func TestSystemChannelDefaultsToStable(t *testing.T) {
1542 if got := systemChannel(func(string) string { return "" }); got != "stable" {
1543 t.Fatalf("default = %q", got)
1544 }
1545 if got := systemChannel(func(k string) string {
1546 if k == systemChannelEnv {
1547 return " canary "
1548 }
1549 return ""
1550 }); got != "canary" {
1551 t.Fatalf("env = %q", got)
1552 }
1553}
1554
1555func TestSystemStateProbeReadsRootsResult(t *testing.T) {
1556 opt := t.TempDir()
1557 if got := systemStateProbe(opt); got != nil {
1558 t.Fatalf("no result.json → %+v, want nil", got)
1559 }
1560 root := systemRootDir(opt)
1561 os.MkdirAll(root, 0o755)
1562 at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
1563 writeJSONDurable(filepath.Join(root, "result.json"), systemResult{Channel: "stable", Version: "v3", PromotedAt: at, Outcome: "applied", At: at}, 0o644)
1564 got := systemStateProbe(opt)
1565 if got == nil || got.Version != "v3" || got.Outcome != "applied" || !got.PromotedAt.Equal(at) {
1566 t.Fatalf("probe = %+v", got)
1567 }
1568}
1569```
1570
1571- [ ] **Step 2: Run to verify failure**
1572
1573Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -run 'TestBoxMetricsCarry|TestSystemChannel|TestSystemStateProbe' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
1574Expected: undefined fields/functions.
1575
1576- [ ] **Step 3: Implement**
1577
1578`internal/status/snapshot.go` — in `Probes`, after `Release string` (`:98`):
1579```go
1580 // Platform is runtime.GOOS-runtime.GOARCH, so the console knows which
1581 // artifacts a platform promote must mirror. SystemChannel is the box's
1582 // CARLOS_SYSTEM_CHANNEL, so the console knows which boxes a promote
1583 // reaches. System is root's last self-update outcome (spec §5, §6).
1584 Platform string
1585 SystemChannel string
1586 System *SystemState
1587```
1588Add the type near `BoxMetrics`:
1589```go
1590// SystemState is the last platform self-update outcome root recorded on the
1591// box, published so `carlos system status` can show a rolled-back box with
1592// its reason within one heartbeat.
1593type SystemState struct {
1594 Channel string `json:"channel"`
1595 Version string `json:"version"`
1596 PromotedAt time.Time `json:"promoted_at"`
1597 Outcome string `json:"outcome"`
1598 Reason string `json:"reason,omitempty"`
1599 At time.Time `json:"at"`
1600}
1601```
1602In `BoxMetrics`, after `Release` (`:158`):
1603```go
1604 Platform string `json:"platform,omitempty"` // runtime.GOOS-runtime.GOARCH
1605 SystemChannel string `json:"system_channel,omitempty"` // CARLOS_SYSTEM_CHANNEL
1606 System *SystemState `json:"system,omitempty"` // last self-update outcome
1607```
1608In `CollectBox`, where `Release: p.Release` is copied into the metrics block, add `Platform: p.Platform, SystemChannel: p.SystemChannel, System: p.System`. In the emptiness guard (`:543`), add `|| p.Platform != "" || p.SystemChannel != "" || p.System != nil`.
1609
1610`internal/host/systemprobe.go`:
1611```go
1612package host
1613
1614import (
1615 "strings"
1616
1617 "github.com/carlosframework/platform/internal/status"
1618)
1619
1620// systemChannelEnv is the box's platform channel: canary or stable. Set in
1621// host.env, read through the getenv seam like every other CARLOS_* value;
1622// the ROOT apply reads it from its own unit environment, never from the
1623// agent's request (spec §6 step 2).
1624const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL"
1625
1626func systemChannel(getenv func(string) string) string {
1627 if c := strings.TrimSpace(getenv(systemChannelEnv)); c != "" {
1628 return c
1629 }
1630 return "stable"
1631}
1632
1633// systemStateProbe is root's result.json as the heartbeat carries it, or nil
1634// when this box has never been asked to update.
1635func systemStateProbe(optDir string) *status.SystemState {
1636 r, err := readSystemResult(optDir)
1637 if err != nil || r == nil {
1638 return nil
1639 }
1640 return &status.SystemState{Channel: r.Channel, Version: r.Version, PromotedAt: r.PromotedAt, Outcome: r.Outcome, Reason: r.Reason, At: r.At}
1641}
1642```
1643`internal/host/agent.go` `statusLoop`, next to `Release: version.Build,` (`:2318`):
1644```go
1645 Platform: runtime.GOOS + "-" + runtime.GOARCH,
1646 SystemChannel: systemChannel(os.Getenv),
1647 System: systemStateProbe(release.OptDir()),
1648```
1649(add `"runtime"` to the imports). Add `CARLOS_SYSTEM_CHANNEL` to the env contract in the package doc comment at the top of `agent.go`, one line: `CARLOS_SYSTEM_CHANNEL — platform channel this box follows for self-update, canary|stable (default stable)`.
1650
1651- [ ] **Step 4: Run the tests**
1652
1653Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
1654Expected: `ok` for both.
1655
1656- [ ] **Step 5: Commit**
1657
1658```bash
1659gofmt -l internal/ && git add internal/status/snapshot.go internal/status/snapshot_test.go internal/host/agent.go internal/host/systemprobe.go internal/host/systemprobe_test.go && git commit -q -F - <<'EOF'
1660status: a box reports its platform, its system channel and its last self-update outcome
1661
1662Three optional fields on the private metrics block; the public fleet
1663feed does not carry them, as before.
1664
1665Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1666Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
1667EOF
1668```
1669
1670---
1671
1672### Task 6: The three units, `enabledUnits`, `-plan-json`, the path probe
1673
1674**Files:**
1675- Create: `internal/host/units/carlos-system-update.path`, `internal/host/units/carlos-system-update.service`, `internal/host/units/carlos-system-recover.service`
1676- Modify: `internal/host/bootstrap.go` (`enabledUnits` `:72`, `RunBootstrap` flags `:115-132`, the staging print block `:327-341`)
1677- Modify: `internal/host/probes.go` (add a fourth `*Active()`)
1678- Modify: `internal/host/host_test.go` (`expectedUnits` `:29-46`; `TestProbedUnitsAreTheOnesBootstrapArms` `:1129`)
1679- Test: `internal/host/host_test.go` (add `TestBootstrapPlanJSON`)
1680
1681**Interfaces:**
1682- Consumes: `unitsFS`, `unitNames`, `systemctlPlan`, `bootstrap(root, consumers)`, `parseIsActive`.
1683- Produces:
1684 ```go
1685 // package host
1686 var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"}
1687 type bootstrapPlan struct{ Enabled []string `json:"enabled"`; Inert []string `json:"inert"` }
1688 func currentBootstrapPlan() bootstrapPlan
1689 // `carlos bootstrap -root <dir> -plan-json` prints ONLY the JSON plan on stdout
1690 const systemUpdatePathUnit = "carlos-system-update.path"
1691 func systemUpdatePathActive() *bool
1692 ```
1693
1694- [ ] **Step 1: Add the failing tests**
1695
1696In `host_test.go`, add the three names to `expectedUnits` (keep it alphabetical):
1697```go
1698 "carlos-system-recover.service",
1699 "carlos-system-update.path",
1700 "carlos-system-update.service",
1701```
1702and add:
1703```go
1704func TestBootstrapPlanJSONNamesWhatALiveRunArms(t *testing.T) {
1705 p := currentBootstrapPlan()
1706 want := map[string]bool{"carlos-edge.service": true, "carlos-adopt.timer": true, "carlos-adopt.path": true,
1707 "carlos-stage.path": true, "carlos-restart.path": true, "carlos-system-update.path": true, "carlos-system-recover.service": true}
1708 got := map[string]bool{}
1709 for _, u := range p.Enabled {
1710 got[u] = true
1711 }
1712 for u := range want {
1713 if !got[u] {
1714 t.Errorf("plan lacks %s", u)
1715 }
1716 }
1717 if got["carlos-system-update.service"] {
1718 t.Error("the path-triggered oneshot must not be enabled: it is started by its .path")
1719 }
1720 inert := map[string]bool{}
1721 for _, u := range p.Inert {
1722 inert[u] = true
1723 }
1724 if !inert["carlos-restore-verify.timer"] || !inert["carlos-hibernate-verify.timer"] {
1725 t.Errorf("inert = %v", p.Inert)
1726 }
1727 // And systemctlPlan never starts a service, whatever is enabled.
1728 for _, args := range systemctlPlan(p.Enabled) {
1729 if args[0] == "start" {
1730 for _, u := range args[1:] {
1731 if strings.HasSuffix(u, ".service") {
1732 t.Errorf("plan starts a service: %v", args)
1733 }
1734 }
1735 }
1736 }
1737}
1738
1739func TestSystemUnitsAreShapedLikeTheStageRelay(t *testing.T) {
1740 read := func(name string) string {
1741 b, err := unitsFS.ReadFile("units/" + name)
1742 if err != nil {
1743 t.Fatal(err)
1744 }
1745 return string(b)
1746 }
1747 p := read("carlos-system-update.path")
1748 for _, want := range []string{"PathChanged=/data/system-update-requested", "Unit=carlos-system-update.service", "WantedBy=multi-user.target"} {
1749 if !strings.Contains(p, want) {
1750 t.Errorf("path unit lacks %q", want)
1751 }
1752 }
1753 u := read("carlos-system-update.service")
1754 for _, want := range []string{"Type=oneshot", "StartLimitIntervalSec=0", "TimeoutStartSec=15min", "OnFailure=carlos-system-recover.service",
1755 "EnvironmentFile=-/etc/carlos/host.env", "system apply", "carlos.prev", "MemoryMax=256M"} {
1756 if !strings.Contains(u, want) {
1757 t.Errorf("update unit lacks %q", want)
1758 }
1759 }
1760 for _, never := range []string{"ProtectSystem", "User="} {
1761 if strings.Contains(u, never) {
1762 t.Errorf("update unit must run as root with /opt writable; found %q", never)
1763 }
1764 }
1765 r := read("carlos-system-recover.service")
1766 for _, want := range []string{"Type=oneshot", "Before=carlos-edge.service", "WantedBy=multi-user.target", "TimeoutStartSec=10min", "system apply --recover", "carlos.prev"} {
1767 if !strings.Contains(r, want) {
1768 t.Errorf("recover unit lacks %q", want)
1769 }
1770 }
1771}
1772```
1773Update `TestProbedUnitsAreTheOnesBootstrapArms` so its probed set includes `systemUpdatePathUnit` (read the test; it compares probe constants against `enabledUnits`).
1774
1775- [ ] **Step 2: Run to verify failure**
1776
1777Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestBootstrap|TestSystemUnits|TestProbedUnits' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
1778Expected: FAIL (missing units, undefined `currentBootstrapPlan`).
1779
1780- [ ] **Step 3: Write the units**
1781
1782`internal/host/units/carlos-system-update.path`:
1783```ini
1784# The relay from the sandboxed agent to a root apply of a new platform
1785# binary (spec §4). The agent (User=carlos, ProtectSystem=strict) can fetch
1786# and verify the artifact but cannot write /opt or /etc; it touches this
1787# marker and root does the rest. Same shape as carlos-stage.path.
1788[Unit]
1789Description=CARLOS platform self-update on agent request
1790
1791[Path]
1792PathChanged=/data/system-update-requested
1793Unit=carlos-system-update.service
1794
1795[Install]
1796WantedBy=multi-user.target
1797```
1798
1799`internal/host/units/carlos-system-update.service`:
1800```ini
1801# Root apply of a platform self-update (spec §6): re-verify the pointer,
1802# copy the candidate into a root-owned dir, journal, swap the binary and
1803# units, restart the edge, gate it, roll back to carlos.prev on failure.
1804#
1805# ExecStart goes through one line of shell so that when a journal is open
1806# (a previous apply was cut off) and carlos.prev exists, the RECOVERY runs
1807# from carlos.prev rather than from the candidate that did not work.
1808#
1809# No ProtectSystem, no User=: it must write /opt/carlos and
1810# /etc/systemd/system. StartLimitIntervalSec=0 because the path unit can
1811# fire in bursts. TimeoutStartSec bounds a hung apply; OnFailure= makes a
1812# killed or timed-out apply recover without waiting for a touch.
1813[Unit]
1814Description=carlos platform self-update (root apply)
1815After=network-online.target
1816Wants=network-online.target
1817StartLimitIntervalSec=0
1818OnFailure=carlos-system-recover.service
1819
1820[Service]
1821Type=oneshot
1822TimeoutStartSec=15min
1823EnvironmentFile=-/etc/carlos/host.env
1824ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply; fi; exec /opt/carlos/carlos system apply'
1825
1826# --- hardening (the carlos-stage.service set) ---
1827ProtectHome=yes
1828PrivateTmp=yes
1829PrivateDevices=yes
1830ProtectClock=yes
1831ProtectHostname=yes
1832ProtectKernelTunables=yes
1833ProtectKernelModules=yes
1834ProtectKernelLogs=yes
1835RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
1836RestrictNamespaces=yes
1837RestrictRealtime=yes
1838LockPersonality=yes
1839MemoryDenyWriteExecute=yes
1840MemoryMax=256M
1841TasksMax=64
1842```
1843
1844`internal/host/units/carlos-system-recover.service`:
1845```ini
1846# Recovery of an interrupted platform self-update (spec §6, "every run
1847# begins with recovery"). Runs once per boot, BEFORE the edge, and on the
1848# update oneshot's failure. It restores files only — carlos.prev over
1849# carlos, the previous unit set — and never restarts or gates the edge
1850# from inside a unit the edge is ordered after: at boot the edge's own
1851# start IS the restart. At runtime (edge already started this boot) it
1852# issues a non-blocking restart and exits.
1853[Unit]
1854Description=carlos platform self-update recovery
1855Before=carlos-edge.service
1856DefaultDependencies=yes
1857
1858[Service]
1859Type=oneshot
1860TimeoutStartSec=10min
1861EnvironmentFile=-/etc/carlos/host.env
1862ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply --recover; fi; exec /opt/carlos/carlos system apply --recover'
1863
1864ProtectHome=yes
1865PrivateTmp=yes
1866PrivateDevices=yes
1867ProtectClock=yes
1868ProtectHostname=yes
1869ProtectKernelTunables=yes
1870ProtectKernelModules=yes
1871ProtectKernelLogs=yes
1872RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
1873RestrictNamespaces=yes
1874RestrictRealtime=yes
1875LockPersonality=yes
1876MemoryDenyWriteExecute=yes
1877MemoryMax=256M
1878TasksMax=64
1879
1880[Install]
1881WantedBy=multi-user.target
1882```
1883
1884- [ ] **Step 4: `bootstrap.go` changes**
1885
1886Replace `enabledUnits` (`:72`):
1887```go
1888var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path",
1889 "carlos-system-update.path", "carlos-system-recover.service"}
1890
1891// inertUnits are installed everywhere and deliberately never armed by
1892// bootstrap or by a self-update apply: restore-verify until an operator
1893// stages litestream, hibernate-verify by long-standing choice.
1894var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"}
1895
1896// bootstrapPlan is what `bootstrap -root <dir> -plan-json` prints: the
1897// arming list of THIS binary, so a self-update apply arms what the new
1898// binary's bootstrap would arm and never a list an older binary remembers.
1899type bootstrapPlan struct {
1900 Enabled []string `json:"enabled"`
1901 Inert []string `json:"inert"`
1902}
1903
1904func currentBootstrapPlan() bootstrapPlan {
1905 return bootstrapPlan{Enabled: append([]string(nil), enabledUnits...), Inert: append([]string(nil), inertUnits...)}
1906}
1907```
1908In `RunBootstrap`, add the flag after `offcloud`:
1909```go
1910 planJSON := fs.Bool("plan-json", false, "with -root: print the arming plan as JSON on stdout instead of the human summary")
1911```
1912and pass it through: change `bootstrap(*root, consumers)` to `bootstrapWith(*root, consumers, *planJSON)`, where `bootstrap(root, consumers)` becomes `func bootstrap(root string, consumers []string) error { return bootstrapWith(root, consumers, false) }` and `bootstrapWith` is the old body with the staging print block (`:327-341`) wrapped:
1913```go
1914 if !live {
1915 if planJSON {
1916 return json.NewEncoder(os.Stdout).Encode(currentBootstrapPlan())
1917 }
1918 fmt.Printf("bootstrap: staged under %s — on a real box (--root /) it would also:\n", root)
1919 …existing prints…
1920 return nil
1921 }
1922```
1923All the `fmt.Printf(" wrote %s\n", path)` / `" created %s\n"` lines in the staging path must go to **stderr** when `planJSON` is set, or the JSON is not alone on stdout: introduce `out := io.Writer(os.Stdout); if planJSON { out = os.Stderr }` at the top of `bootstrapWith` and change those `fmt.Printf` to `fmt.Fprintf(out, …)`. (`-plan-json` with `-root /` is refused: `log.Fatal("bootstrap: -plan-json is for a staging root")`.)
1924
1925`internal/host/probes.go`, alongside `restartPathActive`:
1926```go
1927const systemUpdatePathUnit = "carlos-system-update.path"
1928
1929// systemUpdatePathActive: is the self-update relay armed? nil = unknown.
1930func systemUpdatePathActive() *bool {
1931 out, _ := exec.Command("systemctl", "is-active", systemUpdatePathUnit).CombinedOutput()
1932 return parseIsActive(string(out))
1933}
1934```
1935(copy the exact shape of `restartPathActive` at `probes.go:82`, including how it runs `systemctl`).
1936
1937- [ ] **Step 5: Run the tests**
1938
1939Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
1940Expected: `ok`. Then, as a real check of the flag: `go run . bootstrap -root /tmp/c/stage -plan-json 2>/dev/null | head -3` prints one JSON object with `enabled` and `inert`.
1941
1942- [ ] **Step 6: Commit**
1943
1944```bash
1945gofmt -l internal/host/ && git add internal/host/units/carlos-system-update.path internal/host/units/carlos-system-update.service internal/host/units/carlos-system-recover.service internal/host/bootstrap.go internal/host/probes.go internal/host/host_test.go && git commit -q -F - <<'EOF'
1946host: the self-update relay, apply and recover units, and a JSON arming plan
1947
1948bootstrap -root <dir> -plan-json prints this binary's enabledUnits and
1949inert list, so an apply arms what the NEW binary's bootstrap would arm.
1950Both oneshots exec carlos.prev when a journal is open.
1951
1952Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1953Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
1954EOF
1955```
1956
1957---
1958### Task 7: The agent leg (`internal/host/systemleg.go`)
1959
1960**Files:**
1961- Create: `internal/host/systemleg.go`
1962- Modify: `internal/host/agent.go` — `fastDeps` (`:1077`), `fastTick` (`:1330`; the serial-move branch `:1352-1367` and the unconditional part before `fastMembers` `:1370`), `fastConvergeLoop` (`:1434-1459`)
1963- Test: `internal/host/systemleg_test.go`, `internal/host/agent_test.go` (`fastFixture.deps` at `:1098` may leave the new field nil)
1964
1965**Interfaces:**
1966- Consumes: `release.GetSystemPointer`, `release.FetchToFile`, `release.SystemComponentRouter`, `release.OptDir`, `systemChannel`, `systemRequest`, `readSystemResult`, `readSystemSeen`, `systemCachePath`, `systemRequestPath`, `systemRequestMarker`, `systemRequestEvery`, `systemArtifactLimit`, `validSystemVersion`, `version.Build`.
1967- Produces:
1968 ```go
1969 type systemLeg struct { … }
1970 func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg
1971 func (l *systemLeg) observe(ctx context.Context) // serial move / hourly: read pointer, decide, set pending
1972 func (l *systemLeg) tick(ctx context.Context, now time.Time) // every 2 s: fire pending, re-touch, hourly re-observe
1973 // fastDeps gains: systemLeg *systemLeg
1974 ```
1975 Test seams on the struct: `now func() time.Time`, `jitter func(time.Duration) time.Duration`, `build string`, `platform string`.
1976
1977- [ ] **Step 1: Write the failing tests**
1978
1979```go
1980package host
1981
1982import (
1983 "context"
1984 "crypto/sha256"
1985 "encoding/hex"
1986 "os"
1987 "path/filepath"
1988 "strings"
1989 "testing"
1990 "time"
1991
1992 "github.com/carlosframework/platform/internal/release"
1993)
1994
1995type legFixture struct {
1996 t *testing.T
1997 store *release.DirStore
1998 priv ed25519.PrivateKey
1999 leg *systemLeg
2000 dd string
2001 opt string
2002 now time.Time
2003}
2004
2005func newLegFixture(t *testing.T, channel, build string) *legFixture {
2006 t.Helper()
2007 priv, pub, _ := ed25519.GenerateKey(nil)
2008 f := &legFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, dd: t.TempDir(), opt: t.TempDir(),
2009 now: time.Date(2026, 9, 6, 10, 0, 0, 0, time.UTC)}
2010 getenv := func(k string) string {
2011 if k == systemChannelEnv {
2012 return channel
2013 }
2014 return ""
2015 }
2016 f.leg = newSystemLeg(f.store, pub, getenv, f.dd, f.opt)
2017 f.leg.build = build
2018 f.leg.platform = "linux-arm64"
2019 f.leg.now = func() time.Time { return f.now }
2020 f.leg.jitter = func(d time.Duration) time.Duration { return d / 2 } // deterministic: half the spread
2021 return f
2022}
2023
2024func (f *legFixture) promote(version string, body []byte, spread time.Duration) {
2025 f.t.Helper()
2026 sum := sha256.Sum256(body)
2027 rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{
2028 "linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}}
2029 if _, err := f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, ""); err != nil {
2030 f.t.Fatal(err)
2031 }
2032 if _, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.leg.channel, rel,
2033 release.SystemPromoteOptions{Spread: spread}); err != nil {
2034 f.t.Fatal(err)
2035 }
2036}
2037
2038func (f *legFixture) marker() (os.FileInfo, error) { return os.Stat(filepath.Join(f.dd, systemRequestMarker)) }
2039
2040func (f *legFixture) request() *systemRequest {
2041 var r systemRequest
2042 if err := readJSONFile(systemRequestPath(f.dd), &r); err != nil {
2043 return nil
2044 }
2045 return &r
2046}
2047
2048func TestSystemLegDoesNothingWithoutAPointer(t *testing.T) {
2049 f := newLegFixture(t, "stable", "v1")
2050 f.leg.observe(context.Background())
2051 f.leg.tick(context.Background(), f.now.Add(time.Hour))
2052 if _, err := f.marker(); err == nil {
2053 t.Fatal("marker touched with no pointer")
2054 }
2055}
2056
2057func TestSystemLegFetchesVerifiesAndRequestsAfterTheSpread(t *testing.T) {
2058 f := newLegFixture(t, "stable", "v1")
2059 body := []byte("ELF v2")
2060 f.promote("v2", body, 10*time.Minute)
2061 f.leg.observe(context.Background())
2062 if _, err := f.marker(); err == nil {
2063 t.Fatal("requested before the spread elapsed")
2064 }
2065 f.leg.tick(context.Background(), f.now.Add(4*time.Minute)) // jitter = 5m
2066 if _, err := f.marker(); err == nil {
2067 t.Fatal("requested before the deadline")
2068 }
2069 f.leg.tick(context.Background(), f.now.Add(5*time.Minute+time.Second))
2070 if _, err := f.marker(); err != nil {
2071 t.Fatal("marker not touched after the deadline")
2072 }
2073 got, _ := os.ReadFile(systemCachePath(f.dd, "v2"))
2074 if string(got) != "ELF v2" {
2075 t.Fatalf("cache = %q", got)
2076 }
2077 r := f.request()
2078 if r == nil || r.Version != "v2" || r.NoOp || r.PromotedAt.IsZero() {
2079 t.Fatalf("request = %+v", r)
2080 }
2081 raw, _ := os.ReadFile(systemRequestPath(f.dd))
2082 for _, never := range []string{"channel", "key", "sha", "path"} {
2083 if strings.Contains(string(raw), `"`+never+`"`) {
2084 t.Errorf("request carries %q — root must not take that from the sandbox", never)
2085 }
2086 }
2087}
2088
2089func TestSystemLegRetouchesUntilRootAnswers(t *testing.T) {
2090 f := newLegFixture(t, "stable", "v1")
2091 f.promote("v2", []byte("x"), 0)
2092 f.leg.observe(context.Background())
2093 f.leg.tick(context.Background(), f.now)
2094 first, _ := f.marker()
2095 f.leg.tick(context.Background(), f.now.Add(30*time.Second))
2096 same, _ := f.marker()
2097 if !same.ModTime().Equal(first.ModTime()) {
2098 t.Fatal("re-touched inside the minute throttle")
2099 }
2100 // The touch is a rewrite (PathChanged needs a change), so mtime moves.
2101 time.Sleep(20 * time.Millisecond)
2102 f.leg.tick(context.Background(), f.now.Add(61*time.Second))
2103 later, _ := f.marker()
2104 if !later.ModTime().After(first.ModTime()) {
2105 t.Fatal("not re-touched after a minute with no result")
2106 }
2107 // Root answers: a terminal result for this tuple stops the re-touch and
2108 // the leg deletes its own cache.
2109 r := f.request()
2110 os.MkdirAll(systemRootDir(f.opt), 0o755)
2111 writeJSONDurable(filepath.Join(systemRootDir(f.opt), "result.json"),
2112 systemResult{Channel: "stable", Version: "v2", PromotedAt: r.PromotedAt, Outcome: systemOutcomeRolledBack, Reason: "gate", At: f.now}, 0o644)
2113 time.Sleep(20 * time.Millisecond)
2114 f.leg.tick(context.Background(), f.now.Add(3*time.Minute))
2115 final, _ := f.marker()
2116 if !final.ModTime().Equal(later.ModTime()) {
2117 t.Fatal("re-touched after a terminal result")
2118 }
2119 if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
2120 t.Fatal("cache kept after a terminal result")
2121 }
2122 // And a later observe of the SAME pointer does nothing: refuse to repeat a failure.
2123 f.leg.observe(context.Background())
2124 f.leg.tick(context.Background(), f.now.Add(10*time.Minute))
2125 if f.leg.pending != nil {
2126 t.Fatal("re-requested a pointer root already rolled back")
2127 }
2128}
2129
2130func TestSystemLegNoOpRequestFetchesNothing(t *testing.T) {
2131 f := newLegFixture(t, "stable", "v2")
2132 f.promote("v2", []byte("same"), 0)
2133 f.leg.observe(context.Background())
2134 f.leg.tick(context.Background(), f.now)
2135 r := f.request()
2136 if r == nil || !r.NoOp {
2137 t.Fatalf("request = %+v, want a no-op", r)
2138 }
2139 if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
2140 t.Fatal("a no-op must not download the artifact")
2141 }
2142 // Once root has seen it (seen.json advanced to this PromotedAt), nothing more.
2143 os.MkdirAll(systemRootDir(f.opt), 0o755)
2144 writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": r.PromotedAt}, 0o644)
2145 f.leg.pending = nil
2146 os.Remove(filepath.Join(f.dd, systemRequestMarker))
2147 f.leg.observe(context.Background())
2148 f.leg.tick(context.Background(), f.now.Add(2*time.Minute))
2149 if _, err := f.marker(); err == nil {
2150 t.Fatal("requested again after root recorded the watermark")
2151 }
2152}
2153
2154func TestSystemLegIgnoresDevBuildEqualityAndOtherPlatforms(t *testing.T) {
2155 f := newLegFixture(t, "stable", "dev")
2156 f.promote("v2", []byte("x"), 0)
2157 f.leg.platform = "linux-amd64" // no artifact for me
2158 f.leg.observe(context.Background())
2159 f.leg.tick(context.Background(), f.now)
2160 if _, err := f.marker(); err == nil {
2161 t.Fatal("requested with no artifact for this platform")
2162 }
2163}
2164
2165func TestSystemLegRefusesShaMismatchAndOversize(t *testing.T) {
2166 f := newLegFixture(t, "stable", "v1")
2167 f.promote("v2", []byte("good"), 0)
2168 // Swap the object under the signed sha.
2169 f.store.Put(context.Background(), release.SystemArtifactKey("v2", "linux-arm64"), []byte("evil"), "")
2170 f.leg.observe(context.Background())
2171 f.leg.tick(context.Background(), f.now)
2172 if _, err := f.marker(); err == nil {
2173 t.Fatal("requested an artifact whose sha does not match")
2174 }
2175 if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
2176 t.Fatal("mismatched artifact left in the cache")
2177 }
2178}
2179
2180func TestSystemLegIgnoresAPointerOlderThanTheWatermark(t *testing.T) {
2181 f := newLegFixture(t, "stable", "v1")
2182 f.promote("v2", []byte("x"), 0)
2183 os.MkdirAll(systemRootDir(f.opt), 0o755)
2184 writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": time.Now().Add(time.Hour)}, 0o644)
2185 f.leg.observe(context.Background())
2186 f.leg.tick(context.Background(), f.now)
2187 if _, err := f.marker(); err == nil {
2188 t.Fatal("requested a pointer root would refuse as a replay")
2189 }
2190}
2191
2192func TestFastTickRunsTheSystemLeg(t *testing.T) {
2193 fx := newFastFixture(t)
2194 d := fx.deps(nil)
2195 observes := 0
2196 d.systemLeg = &systemLeg{observeHook: func() { observes++ }}
2197 ctx := context.Background()
2198 fastTick(ctx, d, time.Now()) // boot seed
2199 if observes != 0 {
2200 t.Fatalf("boot seed observed %d times", observes)
2201 }
2202 release.BumpSerial(ctx, fx.store, time.Now())
2203 fastTick(ctx, d, time.Now().Add(10*time.Second))
2204 if observes != 1 {
2205 t.Fatalf("one serial move → %d observes", observes)
2206 }
2207}
2208```
2209(`observeHook func()` is a test seam on `systemLeg`: when non-nil, `observe` calls it and returns. Add `"crypto/ed25519"` to the imports.)
2210
2211- [ ] **Step 2: Run to verify failure**
2212
2213Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemLeg|TestFastTickRunsTheSystemLeg' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
2214Expected: undefined `systemLeg`, `newSystemLeg`.
2215
2216- [ ] **Step 3: Implement `internal/host/systemleg.go`**
2217
2218```go
2219package host
2220
2221import (
2222 "context"
2223 "crypto/ed25519"
2224 "errors"
2225 "log"
2226 "math/rand/v2"
2227 "os"
2228 "path/filepath"
2229 "runtime"
2230 "time"
2231
2232 "github.com/carlosframework/platform/internal/release"
2233 "github.com/carlosframework/platform/internal/version"
2234)
2235
2236// systemLeg is the sandboxed half of the platform self-update (spec §4):
2237// notice the channel pointer, fetch and verify the artifact into /data, ask
2238// root. Two entry points so the spread has something to run it: observe on
2239// a serial move (and hourly), tick every 2 s with no store read.
2240type systemLeg struct {
2241 store release.Store
2242 pub ed25519.PublicKey
2243 channel string
2244 platform string
2245 build string
2246 dataDir string
2247 optDir string
2248
2249 now func() time.Time
2250 jitter func(time.Duration) time.Duration
2251
2252 pending *systemPending
2253 lastTouch time.Time
2254 lastObserve time.Time
2255 backoffUntil time.Time
2256 backoff time.Duration
2257
2258 observeHook func() // test seam
2259}
2260
2261type systemPending struct {
2262 version string
2263 promotedAt time.Time
2264 noop bool
2265 artifact release.SystemArtifact
2266 deadline time.Time
2267 requested bool
2268}
2269
2270const systemObserveEvery = time.Hour
2271
2272func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg {
2273 return &systemLeg{
2274 store: store, pub: pub, channel: systemChannel(getenv),
2275 platform: runtime.GOOS + "-" + runtime.GOARCH, build: version.Build,
2276 dataDir: dataDir, optDir: optDir,
2277 now: time.Now,
2278 jitter: func(spread time.Duration) time.Duration { return rand.N(spread + 1) },
2279 }
2280}
2281
2282// observe reads the pointer and decides. It never blocks on a download:
2283// that happens in tick, after the spread.
2284func (l *systemLeg) observe(ctx context.Context) {
2285 if l.observeHook != nil {
2286 l.observeHook()
2287 return
2288 }
2289 now := l.now()
2290 l.lastObserve = now
2291 p, _, err := release.GetSystemPointer(ctx, l.store, l.pub, release.SystemComponentRouter, l.channel)
2292 if err != nil {
2293 if !errors.Is(err, os.ErrNotExist) {
2294 log.Printf("agent: system pointer %s: %v", l.channel, err)
2295 }
2296 return
2297 }
2298 seen, _ := readSystemSeen(l.optDir)
2299 if wm, ok := seen[l.channel]; ok && p.PromotedAt.Before(wm) {
2300 return // root would refuse it as a replay
2301 }
2302 if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.Current.Version, p.PromotedAt) && res.terminal() {
2303 l.forget(p.Current.Version)
2304 return // refuse to repeat a failure
2305 }
2306 noop := p.Current.Version == l.build
2307 if noop {
2308 if wm, ok := seen[l.channel]; ok && !p.PromotedAt.After(wm) {
2309 return // nothing to do, and root has seen this promote
2310 }
2311 }
2312 art, ok := p.Current.Artifacts[l.platform]
2313 if !ok && !noop {
2314 if l.pending == nil || l.pending.version != p.Current.Version {
2315 log.Printf("agent: system pointer %s names %s with no artifact for %s — not for this box", l.channel, p.Current.Version, l.platform)
2316 }
2317 return
2318 }
2319 if l.pending != nil && l.pending.version == p.Current.Version && l.pending.promotedAt.Equal(p.PromotedAt) {
2320 return // already pending; keep its deadline
2321 }
2322 l.pending = &systemPending{version: p.Current.Version, promotedAt: p.PromotedAt, noop: noop, artifact: art,
2323 deadline: now.Add(l.jitter(p.SpreadDuration()))}
2324 if noop {
2325 l.pending.deadline = now
2326 }
2327}
2328
2329// tick fires a pending request whose deadline has passed and re-touches an
2330// unanswered one every minute. It reads no store except for the one fetch.
2331func (l *systemLeg) tick(ctx context.Context, now time.Time) {
2332 if now.Sub(l.lastObserve) >= systemObserveEvery {
2333 l.observe(ctx)
2334 }
2335 p := l.pending
2336 if p == nil {
2337 return
2338 }
2339 if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.version, p.promotedAt) {
2340 switch {
2341 case res.terminal():
2342 l.forget(p.version)
2343 return
2344 case res.Outcome == systemOutcomeNoted:
2345 l.pending = nil
2346 return
2347 case res.Outcome == systemOutcomeDeferred && res.At.After(p.deadline):
2348 // root could not reach the store; try again with backoff
2349 if l.backoff == 0 {
2350 l.backoff = fastBackoffBase
2351 } else if l.backoff < fastBackoffCap {
2352 l.backoff *= 2
2353 }
2354 p.deadline = now.Add(l.backoff)
2355 p.requested = false
2356 }
2357 }
2358 if now.Before(p.deadline) || now.Before(l.backoffUntil) {
2359 return
2360 }
2361 if !p.requested {
2362 if !p.noop {
2363 if err := l.fetch(ctx, p); err != nil {
2364 log.Printf("agent: system update %s: %v", p.version, err)
2365 os.Remove(systemCachePath(l.dataDir, p.version))
2366 l.backoffUntil = now.Add(fastBackoffBase)
2367 return
2368 }
2369 }
2370 if err := l.writeRequest(p, now); err != nil {
2371 log.Printf("agent: system update %s: request: %v", p.version, err)
2372 return
2373 }
2374 p.requested = true
2375 }
2376 if now.Sub(l.lastTouch) >= systemRequestEvery {
2377 if err := os.WriteFile(filepath.Join(l.dataDir, systemRequestMarker), []byte(now.UTC().Format(time.RFC3339Nano)+"\n"), 0o644); err != nil {
2378 log.Printf("agent: system update %s: cannot ask for a root apply: %v", p.version, err)
2379 return
2380 }
2381 l.lastTouch = now
2382 }
2383}
2384
2385func (l *systemLeg) fetch(ctx context.Context, p *systemPending) error {
2386 dst := systemCachePath(l.dataDir, p.version)
2387 if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
2388 return err
2389 }
2390 return release.FetchToFile(ctx, l.store, p.artifact.Key, dst, systemArtifactLimit, p.artifact.SHA256)
2391}
2392
2393func (l *systemLeg) writeRequest(p *systemPending, now time.Time) error {
2394 if err := os.MkdirAll(systemDataDir(l.dataDir), 0o755); err != nil {
2395 return err
2396 }
2397 return writeJSONDurable(systemRequestPath(l.dataDir), systemRequest{Version: p.version, PromotedAt: p.promotedAt, NoOp: p.noop, SeenAt: now.UTC()}, 0o644)
2398}
2399
2400// forget: root has answered with a terminal outcome. Drop the pending item
2401// and the cache — root never deletes under /data, the agent does.
2402func (l *systemLeg) forget(version string) {
2403 l.pending = nil
2404 l.backoff = 0
2405 if validSystemVersion(version) {
2406 os.RemoveAll(filepath.Dir(systemCachePath(l.dataDir, version)))
2407 }
2408}
2409```
2410
2411`agent.go` wiring:
2412- `fastDeps` gains `systemLeg *systemLeg // the platform self-update leg (spec §4); nil = not armed`.
2413- In `fastTick`, inside the serial-move branch after the `scheduleLeg` call (`:1365`): `if d.systemLeg != nil { d.systemLeg.observe(ctx) }`.
2414- Before `members := d.fastMembers(now)` (`:1370`), outside the switch: `if d.systemLeg != nil { d.systemLeg.tick(ctx, now) }`.
2415- In `fastConvergeLoop`, after `d.scheduleLeg = scheduleLegFor(sd)` (`:1506`): `d.systemLeg = newSystemLeg(store, pub, os.Getenv, dd, optDir)` and extend the "armed" log line with `+ system pointer`.
2416- The `adoptLoop` doc comment's DO-NOT list (`:566-600`) gets one sentence: the system leg lives in the fast loop and re-observes hourly on its own; nothing is added here.
2417
2418- [ ] **Step 4: Run the tests**
2419
2420Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
2421Expected: `ok`.
2422
2423- [ ] **Step 5: Commit**
2424
2425```bash
2426gofmt -l internal/host/ && git add internal/host/systemleg.go internal/host/systemleg_test.go internal/host/agent.go internal/host/agent_test.go && git commit -q -F - <<'EOF'
2427host: the agent notices a platform promote, fetches, verifies, and asks root
2428
2429observe on a serial move (and hourly), tick every 2 s: spread jitter,
2430streamed fetch with the signed sha, a request carrying only version
2431and promoted_at, and a marker re-touched every minute until root
2432answers. A pointer root already rolled back is never re-requested.
2433
2434Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2435Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
2436EOF
2437```
2438
2439---
2440### Task 8a: `carlos system apply` — the refuse path (steps 1–5, no-op, lock)
2441
2442**Files:**
2443- Create: `internal/host/systemapply.go`
2444- Test: `internal/host/systemapply_test.go`
2445
2446**Interfaces:**
2447- Consumes: Task 4 state helpers; `release.GetSystemPointer`; `release.SystemComponentRouter`; `systemChannel`.
2448- Produces (package `host`):
2449 ```go
2450 type sysOps struct {
2451 systemctl func(args ...string) (string, error)
2452 run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error)
2453 now func() time.Time
2454 sleep func(time.Duration)
2455 probeEdge func() error
2456 edgeStartedThisBoot func() bool
2457 logf func(format string, args ...any)
2458 }
2459 type systemApplier struct {
2460 ops sysOps
2461 store release.Store
2462 pub ed25519.PublicKey
2463 channel string
2464 optDir, dataDir string
2465 etcDir string // "/etc" in production; a temp dir in tests
2466 build string // version.Build of THIS process
2467 gateWait, soak time.Duration // 20s, 15s in production
2468 }
2469 func (a *systemApplier) Run(ctx context.Context, recover bool) error
2470 const (
2471 systemQuiesceUnits = "carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service"
2472 systemEdgeUnit = "carlos-edge"
2473 )
2474 func (a *systemApplier) rootDir() string; stagingDir() string; lockPath() string
2475 func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error
2476 func (a *systemApplier) readRequest() (systemRequest, error) // O_NOFOLLOW walk, 4 KiB cap, validSystemVersion
2477 func (a *systemApplier) verifyPointer(ctx) (*release.SystemPointer, error)
2478 func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) // path of staging/carlos
2479 func openBeneath(root string, parts ...string) (*os.File, error) // per-component O_DIRECTORY|O_NOFOLLOW, then O_NOFOLLOW
2480 var errSystemRefused = errors.New("refused"); var errSystemDeferred = errors.New("deferred")
2481 ```
2482 In this task `Run` handles: lock → (recovery: Task 8c, stub as "journal present → return error") → read request → verify pointer + replay guard → advance seen → no-op → stage candidate → **stop with `errNotYetImplemented`** (Task 8b continues). `Run` returns nil on `noted`/`refused`/`deferred` after writing `result.json`; exit codes are `RunSystemApply`'s business (8c).
2483
2484- [ ] **Step 1: Write the failing tests**
2485
2486```go
2487package host
2488
2489import (
2490 "context"
2491 "crypto/ed25519"
2492 "crypto/sha256"
2493 "encoding/hex"
2494 "errors"
2495 "os"
2496 "path/filepath"
2497 "strings"
2498 "testing"
2499 "time"
2500
2501 "github.com/carlosframework/platform/internal/release"
2502)
2503
2504// applyFixture is a fake box: temp opt/data/etc roots, a DirStore with a
2505// signed pointer, and a sysOps that records every call and answers like a
2506// healthy systemd. The candidate "binary" is answered by the fake run.
2507type applyFixture struct {
2508 t *testing.T
2509 a *systemApplier
2510 store *release.DirStore
2511 priv ed25519.PrivateKey
2512 calls []string // "systemctl is-active carlos-edge", "run <base> version", …
2513 answer map[string]string // "version:<path>" → stdout; "plan:<path>" → JSON plan
2514 now time.Time
2515 edgeUp bool
2516 nrestarts string
2517 activeEnter string
2518 probeErr error
2519 edgeStartedThisBoot bool
2520}
2521
2522func newApplyFixture(t *testing.T, channel, build string) *applyFixture {
2523 t.Helper()
2524 priv, pub, _ := ed25519.GenerateKey(nil)
2525 f := &applyFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, answer: map[string]string{},
2526 now: time.Date(2026, 9, 6, 11, 0, 0, 0, time.UTC), edgeUp: true, nrestarts: "0", activeEnter: "1000", edgeStartedThisBoot: true}
2527 f.a = &systemApplier{
2528 store: f.store, pub: pub, channel: channel,
2529 optDir: t.TempDir(), dataDir: t.TempDir(), etcDir: t.TempDir(), build: build,
2530 }
2531 f.a.ops = sysOps{
2532 systemctl: func(args ...string) (string, error) {
2533 f.calls = append(f.calls, "systemctl "+strings.Join(args, " "))
2534 switch {
2535 case args[0] == "is-active" && args[1] == systemEdgeUnit:
2536 if f.edgeUp {
2537 return "active\n", nil
2538 }
2539 return "failed\n", errors.New("inactive")
2540 case args[0] == "show":
2541 return f.nrestarts + "\n" + f.activeEnter + "\n", nil
2542 }
2543 return "", nil
2544 },
2545 run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) {
2546 f.calls = append(f.calls, "run "+filepath.Base(name)+" "+strings.Join(args, " "))
2547 if len(args) == 1 && args[0] == "version" {
2548 if v, ok := f.answer["version:"+name]; ok {
2549 return v, nil
2550 }
2551 return "", errors.New("no such binary")
2552 }
2553 if len(args) >= 3 && args[0] == "bootstrap" && args[1] == "-root" {
2554 plan, ok := f.answer["plan:"+name]
2555 if !ok {
2556 return "", errors.New("bootstrap failed")
2557 }
2558 units := strings.Split(f.answer["units:"+name], " ")
2559 dir := filepath.Join(args[2], "etc", "systemd", "system")
2560 os.MkdirAll(dir, 0o755)
2561 for _, u := range units {
2562 if u != "" {
2563 os.WriteFile(filepath.Join(dir, u), []byte("[Unit]\n# "+filepath.Base(name)+"\n"), 0o644)
2564 }
2565 }
2566 os.MkdirAll(filepath.Join(args[2], "etc", "sysusers.d"), 0o755)
2567 os.WriteFile(filepath.Join(args[2], "etc", "sysusers.d", "carlos.conf"), []byte("g carlos-sockets -\n"), 0o644)
2568 os.MkdirAll(filepath.Join(args[2], "etc", "tmpfiles.d"), 0o755)
2569 os.WriteFile(filepath.Join(args[2], "etc", "tmpfiles.d", "carlos.conf"), []byte("d /run/carlos\n"), 0o644)
2570 os.MkdirAll(filepath.Join(args[2], "data"), 0o755)
2571 os.WriteFile(filepath.Join(args[2], "data", "registry.db"), nil, 0o640)
2572 return plan, nil
2573 }
2574 return "", errors.New("unexpected command")
2575 },
2576 now: func() time.Time { return f.now },
2577 sleep: func(d time.Duration) { f.now = f.now.Add(d) },
2578 probeEdge: func() error { return f.probeErr },
2579 edgeStartedThisBoot: func() bool { return f.edgeStartedThisBoot },
2580 logf: t.Logf,
2581 }
2582 os.MkdirAll(systemRootDir(f.a.optDir), 0o755)
2583 os.MkdirAll(filepath.Join(f.a.etcDir, "systemd", "system"), 0o755)
2584 // The on-disk binary answers `version` with build.
2585 os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos "+build+"\n"), 0o755)
2586 f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos " + build + "\n"
2587 f.answer["plan:"+systemBinaryPath(f.a.optDir)] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service"],"inert":["carlos-restore-verify.timer"]}`
2588 f.answer["units:"+systemBinaryPath(f.a.optDir)] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-old-only.timer"
2589 return f
2590}
2591
2592// promote signs a pointer for f.a.channel naming version with body as the
2593// linux-arm64 artifact, stages body into the agent's cache, and writes the
2594// request the agent would have written. Returns the request.
2595func (f *applyFixture) promote(version string, body []byte, noop bool) systemRequest {
2596 f.t.Helper()
2597 sum := sha256.Sum256(body)
2598 rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{
2599 "linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}}
2600 f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, "")
2601 p, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.a.channel, rel, release.SystemPromoteOptions{})
2602 if err != nil {
2603 f.t.Fatal(err)
2604 }
2605 req := systemRequest{Version: version, PromotedAt: p.PromotedAt, NoOp: noop, SeenAt: f.now}
2606 os.MkdirAll(systemDataDir(f.a.dataDir), 0o755)
2607 writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644)
2608 if !noop {
2609 os.MkdirAll(filepath.Dir(systemCachePath(f.a.dataDir, version)), 0o755)
2610 os.WriteFile(systemCachePath(f.a.dataDir, version), body, 0o644)
2611 }
2612 return req
2613}
2614
2615func (f *applyFixture) result() *systemResult { r, _ := readSystemResult(f.a.optDir); return r }
2616
2617func (f *applyFixture) called(prefix string) bool {
2618 for _, c := range f.calls {
2619 if strings.HasPrefix(c, prefix) {
2620 return true
2621 }
2622 }
2623 return false
2624}
2625
2626func TestApplyNoRequestIsNothingToDo(t *testing.T) {
2627 f := newApplyFixture(t, "stable", "v1")
2628 if err := f.a.Run(context.Background(), false); err != nil {
2629 t.Fatal(err)
2630 }
2631 if f.result() != nil || len(f.calls) != 0 {
2632 t.Fatalf("result=%+v calls=%v", f.result(), f.calls)
2633 }
2634}
2635
2636func TestApplyLockKeepsTwoRunsApart(t *testing.T) {
2637 f := newApplyFixture(t, "stable", "v1")
2638 f.promote("v2", []byte("x"), false)
2639 // Hold the lock as another process would.
2640 held, err := os.OpenFile(f.a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600)
2641 if err != nil {
2642 t.Fatal(err)
2643 }
2644 defer held.Close()
2645 if err := flockExclusiveNonblock(held); err != nil {
2646 t.Fatal(err)
2647 }
2648 if err := f.a.Run(context.Background(), false); err != nil {
2649 t.Fatalf("a locked-out path run must exit 0 quietly: %v", err)
2650 }
2651 if f.result() != nil {
2652 t.Fatal("a locked-out run wrote a result")
2653 }
2654}
2655
2656func TestApplyRequestIsAHintNotAnInstruction(t *testing.T) {
2657 f := newApplyFixture(t, "stable", "v1")
2658 // A symlinked parent under /data.
2659 os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere"), 0o755)
2660 os.Symlink(filepath.Join(f.a.dataDir, "elsewhere"), filepath.Join(f.a.dataDir, "system"))
2661 os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere", "router"), 0o755)
2662 os.WriteFile(filepath.Join(f.a.dataDir, "elsewhere", "router", "requested.json"), []byte(`{"version":"v2"}`), 0o644)
2663 if _, err := f.a.readRequest(); err == nil {
2664 t.Fatal("followed a symlinked parent under /data")
2665 }
2666 os.Remove(filepath.Join(f.a.dataDir, "system"))
2667 // A symlinked request file.
2668 os.MkdirAll(systemDataDir(f.a.dataDir), 0o755)
2669 os.WriteFile(filepath.Join(f.a.dataDir, "real.json"), []byte(`{"version":"v2"}`), 0o644)
2670 os.Symlink(filepath.Join(f.a.dataDir, "real.json"), systemRequestPath(f.a.dataDir))
2671 if _, err := f.a.readRequest(); err == nil {
2672 t.Fatal("followed a symlinked request")
2673 }
2674 os.Remove(systemRequestPath(f.a.dataDir))
2675 // A traversal version.
2676 writeJSONDurable(systemRequestPath(f.a.dataDir), systemRequest{Version: "../../etc/passwd", PromotedAt: f.now}, 0o644)
2677 if _, err := f.a.readRequest(); err == nil {
2678 t.Fatal("accepted a traversal version")
2679 }
2680 // Oversized.
2681 os.WriteFile(systemRequestPath(f.a.dataDir), []byte(`{"version":"`+strings.Repeat("a", 5000)+`"}`), 0o644)
2682 if _, err := f.a.readRequest(); err == nil {
2683 t.Fatal("accepted a 5 KiB request")
2684 }
2685}
2686
2687func TestApplyChannelComesFromRootNotTheRequest(t *testing.T) {
2688 // The box is on stable; a valid CANARY pointer exists and the agent
2689 // requests its version. Root reads control/system/router/stable.json
2690 // (missing) and refuses — a signed canary promote does not authorise a
2691 // stable box.
2692 f := newApplyFixture(t, "stable", "v1")
2693 f.a.channel = "canary"
2694 req := f.promote("v2", []byte("x"), false) // signs the CANARY pointer
2695 f.a.channel = "stable"
2696 if err := f.a.Run(context.Background(), false); err != nil {
2697 t.Fatal(err)
2698 }
2699 r := f.result()
2700 if r == nil || r.Outcome != systemOutcomeRefused || !r.matches("stable", "v2", req.PromotedAt) {
2701 t.Fatalf("result = %+v", r)
2702 }
2703 if _, err := os.Stat(systemBinaryPath(f.a.optDir) + ".prev"); err == nil {
2704 t.Fatal("touched the box")
2705 }
2706}
2707
2708func TestApplyStaleTupleIsRefusedAgainstTheRequestsTuple(t *testing.T) {
2709 f := newApplyFixture(t, "stable", "v1")
2710 old := f.promote("v2", []byte("x"), false)
2711 time.Sleep(1100 * time.Millisecond)
2712 f.promote("v3", []byte("y"), false)
2713 // Put the OLD request back: the agent's spread started under v2.
2714 writeJSONDurable(systemRequestPath(f.a.dataDir), old, 0o644)
2715 if err := f.a.Run(context.Background(), false); err != nil {
2716 t.Fatal(err)
2717 }
2718 r := f.result()
2719 if r == nil || r.Outcome != systemOutcomeRefused || r.Version != "v2" || !r.PromotedAt.Equal(old.PromotedAt) {
2720 t.Fatalf("result = %+v (must name the REQUEST's tuple so v3 is not suppressed)", r)
2721 }
2722}
2723
2724func TestApplyReplayGuardIsPerChannelAndEqualIsAllowed(t *testing.T) {
2725 f := newApplyFixture(t, "stable", "v1")
2726 req := f.promote("v2", []byte("x"), true)
2727 // Older than the watermark → refused, seen unchanged.
2728 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt.Add(time.Hour), "canary": time.Time{}}, 0o644)
2729 if err := f.a.Run(context.Background(), false); err != nil {
2730 t.Fatal(err)
2731 }
2732 if r := f.result(); r == nil || r.Outcome != systemOutcomeRefused || !strings.Contains(r.Reason, "replay") {
2733 t.Fatalf("result = %+v", r)
2734 }
2735 // Equal → allowed (a retry of the same promote).
2736 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt}, 0o644)
2737 if err := f.a.Run(context.Background(), false); err != nil {
2738 t.Fatal(err)
2739 }
2740 if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
2741 t.Fatalf("result = %+v", r)
2742 }
2743 // Another channel's newer watermark does not block this channel.
2744 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"canary": req.PromotedAt.Add(time.Hour)}, 0o644)
2745 if err := f.a.Run(context.Background(), false); err != nil {
2746 t.Fatal(err)
2747 }
2748 if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
2749 t.Fatalf("result = %+v", r)
2750 }
2751 seen, _ := readSystemSeen(f.a.optDir)
2752 if !seen["stable"].Equal(req.PromotedAt) {
2753 t.Fatalf("seen not advanced: %v", seen)
2754 }
2755}
2756
2757func TestApplyNoOpNotesAndTouchesNothing(t *testing.T) {
2758 f := newApplyFixture(t, "stable", "v2")
2759 f.promote("v2", []byte("x"), true)
2760 if err := f.a.Run(context.Background(), false); err != nil {
2761 t.Fatal(err)
2762 }
2763 if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
2764 t.Fatalf("result = %+v", r)
2765 }
2766 if _, err := os.Stat(f.a.stagingDir()); err == nil {
2767 t.Fatal("a no-op created staging/")
2768 }
2769 if len(f.calls) != 0 {
2770 t.Fatalf("a no-op ran commands: %v", f.calls)
2771 }
2772}
2773
2774func TestApplyStoreErrorIsDeferredNotRefused(t *testing.T) {
2775 f := newApplyFixture(t, "stable", "v1")
2776 req := f.promote("v2", []byte("x"), false)
2777 f.a.store = failingStore{f.store}
2778 if err := f.a.Run(context.Background(), false); err != nil {
2779 t.Fatal(err)
2780 }
2781 if r := f.result(); r == nil || r.Outcome != systemOutcomeDeferred || !r.matches("stable", "v2", req.PromotedAt) {
2782 t.Fatalf("result = %+v", r)
2783 }
2784}
2785
2786type failingStore struct{ release.Store }
2787
2788func (failingStore) Get(context.Context, string) ([]byte, string, error) {
2789 return nil, "", errors.New("door restarting")
2790}
2791
2792func TestApplyStagesTheCandidateFromARootOwnedCopy(t *testing.T) {
2793 f := newApplyFixture(t, "stable", "v1")
2794 body := []byte("#!/bin/sh\necho carlos v2\n")
2795 f.promote("v2", body, false)
2796 f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v2\n"
2797 p, err := f.a.verifyPointer(context.Background())
2798 if err != nil {
2799 t.Fatal(err)
2800 }
2801 req, _ := f.a.readRequest()
2802 staged, err := f.a.stageCandidate(p, req)
2803 if err != nil {
2804 t.Fatal(err)
2805 }
2806 st, _ := os.Stat(staged)
2807 if st.Mode().Perm() != 0o755 {
2808 t.Fatalf("staged mode = %v after verification", st.Mode().Perm())
2809 }
2810 if dst, _ := os.Stat(f.a.stagingDir()); dst.Mode().Perm() != 0o700 {
2811 t.Fatalf("staging dir mode = %v", dst.Mode().Perm())
2812 }
2813 // Swapping the cache after staging changes nothing about the copy.
2814 os.WriteFile(systemCachePath(f.a.dataDir, "v2"), []byte("evil"), 0o644)
2815 got, _ := os.ReadFile(staged)
2816 if string(got) != string(body) {
2817 t.Fatal("staged copy follows the cache")
2818 }
2819 // A sha mismatch is refused BEFORE the copy is made executable and
2820 // staging/ is cleared.
2821 f2 := newApplyFixture(t, "stable", "v1")
2822 f2.promote("v2", body, false)
2823 os.WriteFile(systemCachePath(f2.a.dataDir, "v2"), []byte("evil"), 0o644)
2824 p2, _ := f2.a.verifyPointer(context.Background())
2825 req2, _ := f2.a.readRequest()
2826 if _, err := f2.a.stageCandidate(p2, req2); err == nil || !strings.Contains(err.Error(), "sha256") {
2827 t.Fatalf("err = %v", err)
2828 }
2829 // A candidate that lies about its version is refused.
2830 f3 := newApplyFixture(t, "stable", "v1")
2831 f3.promote("v2", body, false)
2832 f3.answer["version:"+filepath.Join(f3.a.stagingDir(), "carlos")] = "carlos v9\n"
2833 p3, _ := f3.a.verifyPointer(context.Background())
2834 req3, _ := f3.a.readRequest()
2835 if _, err := f3.a.stageCandidate(p3, req3); err == nil || !strings.Contains(err.Error(), "reports") {
2836 t.Fatalf("err = %v", err)
2837 }
2838 // A symlinked cache file is refused.
2839 f4 := newApplyFixture(t, "stable", "v1")
2840 f4.promote("v2", body, false)
2841 os.Remove(systemCachePath(f4.a.dataDir, "v2"))
2842 os.WriteFile(filepath.Join(f4.a.dataDir, "real"), body, 0o644)
2843 os.Symlink(filepath.Join(f4.a.dataDir, "real"), systemCachePath(f4.a.dataDir, "v2"))
2844 p4, _ := f4.a.verifyPointer(context.Background())
2845 req4, _ := f4.a.readRequest()
2846 if _, err := f4.a.stageCandidate(p4, req4); err == nil {
2847 t.Fatal("followed a symlinked cache file")
2848 }
2849}
2850```
2851
2852- [ ] **Step 2: Run to verify failure**
2853
2854Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
2855Expected: undefined `systemApplier`, `sysOps`, `flockExclusiveNonblock`.
2856
2857- [ ] **Step 3: Implement the refuse path in `internal/host/systemapply.go`**
2858
2859```go
2860package host
2861
2862import (
2863 "context"
2864 "crypto/ed25519"
2865 "crypto/sha256"
2866 "encoding/hex"
2867 "encoding/json"
2868 "errors"
2869 "fmt"
2870 "io"
2871 "os"
2872 "path/filepath"
2873 "strings"
2874 "syscall"
2875 "time"
2876
2877 "github.com/carlosframework/platform/internal/release"
2878)
2879
2880// The root half of the platform self-update (spec §6). Runs as root from
2881// carlos-system-update.service (or --recover from carlos-system-recover).
2882// Every box-changing step is journaled first; every run begins with
2883// recovery; root reads ONE file under /data and treats it as a hint.
2884
2885const (
2886 systemEdgeUnit = "carlos-edge"
2887 systemLockFile = "lock"
2888 systemMaxRequest = 4 << 10
2889 systemGateWait = 20 * time.Second
2890 systemGateSoak = 15 * time.Second
2891 systemVersionWait = 30 * time.Second
2892 systemBootstrapWait = 120 * time.Second
2893)
2894
2895// systemQuiesceUnits are the root units that exec /opt/carlos/carlos
2896// against the running edge. Stopped (services too — stopping a timer does
2897// not stop the pass it already started) before the rename, re-armed after
2898// the gate.
2899var systemQuiesceUnits = []string{
2900 "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path",
2901 "carlos-adopt.service", "carlos-config-sync.service", "carlos-stage.service", "carlos-restart.service",
2902}
2903
2904var (
2905 errSystemRefused = errors.New("refused")
2906 errSystemDeferred = errors.New("deferred")
2907 errNotYetImplemented = errors.New("system apply: install path not implemented")
2908)
2909
2910// sysOps is the seam between the apply and the box: systemctl, exec with a
2911// deadline, the clock, sleep, the edge probe. Tests record and answer.
2912type sysOps struct {
2913 systemctl func(args ...string) (string, error)
2914 run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error)
2915 now func() time.Time
2916 sleep func(time.Duration)
2917 probeEdge func() error
2918 edgeStartedThisBoot func() bool
2919 logf func(format string, args ...any)
2920}
2921
2922type systemApplier struct {
2923 ops sysOps
2924 store release.Store
2925 pub ed25519.PublicKey
2926 channel string
2927 optDir string
2928 dataDir string
2929 etcDir string
2930 build string
2931 gateWait, soak time.Duration
2932}
2933
2934func (a *systemApplier) rootDir() string { return systemRootDir(a.optDir) }
2935func (a *systemApplier) stagingDir() string { return filepath.Join(a.rootDir(), "staging") }
2936func (a *systemApplier) lockPath() string { return filepath.Join(a.rootDir(), systemLockFile) }
2937func (a *systemApplier) file(name string) string { return filepath.Join(a.rootDir(), name) }
2938
2939// Run is the whole oneshot. It returns an error only for a failure the
2940// caller should exit non-zero on (an apply that rolled back, a rollback
2941// that failed, a broken box); refusals and no-ops are recorded in
2942// result.json and return nil.
2943func (a *systemApplier) Run(ctx context.Context, recover bool) error {
2944 if err := os.MkdirAll(a.rootDir(), 0o755); err != nil {
2945 return err
2946 }
2947 lock, err := os.OpenFile(a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600)
2948 if err != nil {
2949 return err
2950 }
2951 defer lock.Close()
2952 if recover {
2953 if err := flockExclusive(lock); err != nil {
2954 return err
2955 }
2956 } else if err := flockExclusiveNonblock(lock); err != nil {
2957 a.ops.logf("system apply: another apply is running; leaving it to finish")
2958 return nil
2959 }
2960 defer os.RemoveAll(a.stagingDir())
2961
2962 if err := a.recoverIfNeeded(ctx); err != nil {
2963 return err
2964 }
2965 if recover {
2966 return nil
2967 }
2968 req, err := a.readRequest()
2969 if errors.Is(err, os.ErrNotExist) {
2970 return nil
2971 }
2972 if err != nil {
2973 a.ops.logf("system apply: request: %v", err)
2974 return nil // nothing to record against: the request itself is unreadable
2975 }
2976 p, err := a.verifyPointer(ctx)
2977 if err != nil {
2978 return a.record(req, err)
2979 }
2980 if req.Version != p.Current.Version || !req.PromotedAt.Equal(p.PromotedAt) {
2981 return a.record(req, fmt.Errorf("%w: request names %s@%s, pointer is %s@%s", errSystemRefused,
2982 req.Version, req.PromotedAt.Format(time.RFC3339), p.Current.Version, p.PromotedAt.Format(time.RFC3339)))
2983 }
2984 seen, err := readSystemSeen(a.optDir)
2985 if err != nil {
2986 return a.record(req, fmt.Errorf("%w: seen.json unreadable: %v", errSystemRefused, err))
2987 }
2988 if wm, ok := seen[a.channel]; ok && p.PromotedAt.Before(wm) {
2989 return a.record(req, fmt.Errorf("%w: replay — pointer promoted %s is older than the %s watermark %s", errSystemRefused,
2990 p.PromotedAt.Format(time.RFC3339), a.channel, wm.Format(time.RFC3339)))
2991 }
2992 seen[a.channel] = p.PromotedAt
2993 if err := writeJSONDurable(a.file("seen.json"), seen, 0o644); err != nil {
2994 return err
2995 }
2996 if req.NoOp || p.Current.Version == a.onDiskBuild(ctx) {
2997 return a.writeResult(req, systemOutcomeNoted, "")
2998 }
2999 staged, err := a.stageCandidate(p, req)
3000 if err != nil {
3001 return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err))
3002 }
3003 return a.install(ctx, req, p, staged) // Task 8b
3004}
3005
3006func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error {
3007 return errNotYetImplemented
3008}
3009
3010func (a *systemApplier) recoverIfNeeded(ctx context.Context) error {
3011 j, err := readSystemJournal(a.optDir)
3012 if err != nil || j == nil {
3013 return err
3014 }
3015 return fmt.Errorf("system apply: journal open at %s — recovery not implemented", j.Phase) // Task 8c
3016}
3017
3018// record turns a refusal or deferral into result.json and a nil return.
3019func (a *systemApplier) record(req systemRequest, err error) error {
3020 outcome := systemOutcomeRefused
3021 if errors.Is(err, errSystemDeferred) {
3022 outcome = systemOutcomeDeferred
3023 }
3024 a.ops.logf("system apply: %s %s: %s: %v", a.channel, req.Version, outcome, err)
3025 return a.writeResult(req, outcome, err.Error())
3026}
3027
3028func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error {
3029 return writeJSONDurable(a.file("result.json"), systemResult{
3030 Channel: a.channel, Version: req.Version, PromotedAt: req.PromotedAt,
3031 Outcome: outcome, Reason: reason, At: a.ops.now().UTC(),
3032 }, 0o644)
3033}
3034
3035// readRequest is the one read under /data: a fixed path reached without
3036// following a single symlink, a regular file under 4 KiB, a version that
3037// passes validSystemVersion. Anything else is an error, and root acts on
3038// nothing.
3039func (a *systemApplier) readRequest() (systemRequest, error) {
3040 f, err := openBeneath(a.dataDir, "system", systemComponent, "requested.json")
3041 if err != nil {
3042 return systemRequest{}, err
3043 }
3044 defer f.Close()
3045 st, err := f.Stat()
3046 if err != nil {
3047 return systemRequest{}, err
3048 }
3049 if !st.Mode().IsRegular() || st.Size() > systemMaxRequest {
3050 return systemRequest{}, fmt.Errorf("request: not a regular file under %d bytes", systemMaxRequest)
3051 }
3052 var req systemRequest
3053 if err := json.NewDecoder(io.LimitReader(f, systemMaxRequest)).Decode(&req); err != nil {
3054 return systemRequest{}, fmt.Errorf("request: %w", err)
3055 }
3056 if !validSystemVersion(req.Version) {
3057 return systemRequest{}, fmt.Errorf("request: version %q is not a version", req.Version)
3058 }
3059 if req.PromotedAt.IsZero() {
3060 return systemRequest{}, errors.New("request: promoted_at unset")
3061 }
3062 return req, nil
3063}
3064
3065// openBeneath opens root/parts... one component at a time with
3066// O_DIRECTORY|O_NOFOLLOW for every directory and O_NOFOLLOW for the leaf,
3067// so no component under a user-owned tree can be a symlink.
3068func openBeneath(root string, parts ...string) (*os.File, error) {
3069 fd, err := syscall.Open(root, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_CLOEXEC, 0)
3070 if err != nil {
3071 return nil, fmt.Errorf("open %s: %w", root, err)
3072 }
3073 for i, part := range parts {
3074 flags := syscall.O_RDONLY | syscall.O_NOFOLLOW | syscall.O_CLOEXEC
3075 if i < len(parts)-1 {
3076 flags |= syscall.O_DIRECTORY
3077 }
3078 next, err := syscall.Openat(fd, part, flags, 0)
3079 syscall.Close(fd)
3080 if err != nil {
3081 return nil, fmt.Errorf("open %s: %w", filepath.Join(append([]string{root}, parts[:i+1]...)...), err)
3082 }
3083 fd = next
3084 }
3085 return os.NewFile(uintptr(fd), filepath.Join(append([]string{root}, parts...)...)), nil
3086}
3087
3088// verifyPointer fetches THIS box's channel pointer and verifies it with
3089// the release public key. The channel is root's own (host.env), never the
3090// request's. A store error is deferred, a bad pointer refused.
3091func (a *systemApplier) verifyPointer(ctx context.Context) (*release.SystemPointer, error) {
3092 p, _, err := release.GetSystemPointer(ctx, a.store, a.pub, release.SystemComponentRouter, a.channel)
3093 switch {
3094 case err == nil:
3095 return p, nil
3096 case errors.Is(err, os.ErrNotExist):
3097 return nil, fmt.Errorf("%w: no %s pointer for this box's channel", errSystemRefused, a.channel)
3098 case strings.Contains(err.Error(), "system pointer"):
3099 return nil, fmt.Errorf("%w: %v", errSystemRefused, err)
3100 default:
3101 return nil, fmt.Errorf("%w: %v", errSystemDeferred, err)
3102 }
3103}
3104
3105// onDiskBuild asks the installed binary what it is, with a deadline. Not
3106// version.Build: after an interrupted apply the two can differ.
3107func (a *systemApplier) onDiskBuild(ctx context.Context) string {
3108 out, err := a.ops.run(ctx, systemVersionWait, systemBinaryPath(a.optDir), "version")
3109 if err != nil {
3110 return ""
3111 }
3112 return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos "))
3113}
3114
3115// stageCandidate copies the agent's cache into staging/ (0700) as 0600,
3116// verifies the copy's sha against the pointer, makes it executable, and
3117// runs its `version`. Nothing under /data is read after the copy.
3118func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) {
3119 art, ok := p.Current.Artifacts[a.platform()]
3120 if !ok {
3121 return "", fmt.Errorf("pointer names no artifact for %s", a.platform())
3122 }
3123 os.RemoveAll(a.stagingDir())
3124 if err := os.MkdirAll(a.stagingDir(), 0o700); err != nil {
3125 return "", err
3126 }
3127 if err := os.Chmod(a.stagingDir(), 0o700); err != nil {
3128 return "", err
3129 }
3130 src, err := openBeneath(a.dataDir, "system", systemComponent, req.Version, "carlos")
3131 if err != nil {
3132 return "", fmt.Errorf("candidate: %w", err)
3133 }
3134 defer src.Close()
3135 if st, err := src.Stat(); err != nil || !st.Mode().IsRegular() {
3136 return "", errors.New("candidate: not a regular file")
3137 }
3138 staged := filepath.Join(a.stagingDir(), "carlos")
3139 dst, err := os.OpenFile(staged, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
3140 if err != nil {
3141 return "", err
3142 }
3143 h := sha256.New()
3144 n, err := io.Copy(io.MultiWriter(dst, h), io.LimitReader(src, systemArtifactLimit+1))
3145 dst.Close()
3146 if err != nil {
3147 return "", err
3148 }
3149 if n > systemArtifactLimit {
3150 return "", fmt.Errorf("candidate: larger than %d bytes", systemArtifactLimit)
3151 }
3152 if got := hex.EncodeToString(h.Sum(nil)); got != art.SHA256 {
3153 return "", fmt.Errorf("candidate: sha256 %s does not match the signed %s", got, art.SHA256)
3154 }
3155 if err := os.Chmod(staged, 0o755); err != nil {
3156 return "", err
3157 }
3158 out, err := a.ops.run(context.Background(), systemVersionWait, staged, "version")
3159 if err != nil || !strings.Contains(out, "carlos "+p.Current.Version) {
3160 return "", fmt.Errorf("candidate reports %q, not %s", strings.TrimSpace(out), p.Current.Version)
3161 }
3162 return staged, nil
3163}
3164
3165func (a *systemApplier) platform() string { return runtime.GOOS + "-" + runtime.GOARCH }
3166
3167func flockExclusive(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX) }
3168func flockExclusiveNonblock(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) }
3169```
3170Add `"runtime"` to the imports. Tests that stage a candidate need `platform()` to be `linux-arm64`: make `platform` a field set in the constructor (`platform string`, default `runtime.GOOS+"-"+runtime.GOARCH`) rather than a method, and have the fixture set `f.a.platform = "linux-arm64"`. Do that; update the test fixture accordingly.
3171
3172- [ ] **Step 4: Run the tests**
3173
3174Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
3175Expected: PASS for every test in this task.
3176
3177- [ ] **Step 5: Commit**
3178
3179```bash
3180gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
3181host: carlos system apply — the refuse path
3182
3183Lock, one O_NOFOLLOW-walked request under /data, the channel from
3184root's own env, the pointer re-verified, the per-channel replay
3185watermark, the no-op, and the candidate copied into a 0700 staging
3186dir and verified there before it is made executable.
3187
3188Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3189Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
3190EOF
3191```
3192
3193---
3194
3195### Task 8b: `carlos system apply` — install, gate, arm, record (steps 6–12)
3196
3197**Files:**
3198- Modify: `internal/host/systemapply.go` (replace the `install` stub)
3199- Test: `internal/host/systemapply_test.go` (add)
3200
3201**Interfaces:**
3202- Consumes: `systemctlPlan`, `bootstrapPlan`, `systemQuiesceUnits`, the fixture from 8a.
3203- Produces:
3204 ```go
3205 func (a *systemApplier) install(ctx, req, p, staged) error
3206 func (a *systemApplier) decidePrev(ctx) (prevExists bool, err error) // journal backing-up, copy when authorised
3207 func (a *systemApplier) stagePlan(ctx, binary, dir string) (bootstrapPlan, []string, error) // runs bootstrap -root dir -plan-json; returns plan + unit file names
3208 func (a *systemApplier) journal(j systemJournal) error
3209 func (a *systemApplier) installUnits(stagedRoot string, remove []string) error // remove, copy units, sysusers, tmpfiles, daemon-reload
3210 func (a *systemApplier) restartEdge() error // reset-failed then restart
3211 func (a *systemApplier) gate(ctx, target string) error
3212 func (a *systemApplier) arm(plan bootstrapPlan) error // systemctlPlan verbatim
3213 func (a *systemApplier) rollback(ctx, req, reason string, j *systemJournal) error // Task 8c; stub returns the reason as an error
3214 ```
3215
3216- [ ] **Step 1: Write the failing tests**
3217
3218Add to `systemapply_test.go`:
3219```go
3220// applyHappy sets up a v1 box with a v2 candidate whose fake bootstrap
3221// plan adds carlos-new-only.path and drops carlos-old-only.timer.
3222func applyHappy(t *testing.T) (*applyFixture, systemRequest) {
3223 f := newApplyFixture(t, "stable", "v1")
3224 body := []byte("#!/bin/sh\necho carlos v2\n")
3225 req := f.promote("v2", body, false)
3226 staged := filepath.Join(f.a.stagingDir(), "carlos")
3227 f.answer["version:"+staged] = "carlos v2\n"
3228 f.answer["plan:"+staged] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service","carlos-new-only.path"],"inert":["carlos-restore-verify.timer"]}`
3229 f.answer["units:"+staged] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-new-only.path"
3230 // .prev, once made, answers as v1 and stages v1's units.
3231 prev := systemPrevPath(f.a.optDir)
3232 f.answer["version:"+prev] = "carlos v1\n"
3233 f.answer["plan:"+prev] = f.answer["plan:"+systemBinaryPath(f.a.optDir)]
3234 f.answer["units:"+prev] = f.answer["units:"+systemBinaryPath(f.a.optDir)]
3235 // After the install the on-disk binary answers v2.
3236 f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v2\n"
3237 return f, req
3238}
3239
3240func TestApplyInstallsGatesArmsAndRecords(t *testing.T) {
3241 f, req := applyHappy(t)
3242 if err := f.a.Run(context.Background(), false); err != nil {
3243 t.Fatal(err)
3244 }
3245 r := f.result()
3246 if r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) {
3247 t.Fatalf("result = %+v", r)
3248 }
3249 ap, _ := readSystemApplied(f.a.optDir)
3250 if ap == nil || ap.Version != "v2" || ap.Channel != "stable" {
3251 t.Fatalf("applied = %+v", ap)
3252 }
3253 if j, _ := readSystemJournal(f.a.optDir); j != nil {
3254 t.Fatalf("journal left open: %+v", j)
3255 }
3256 if _, err := os.Stat(f.a.stagingDir()); err == nil {
3257 t.Fatal("staging/ left behind")
3258 }
3259 bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
3260 if !strings.Contains(string(bin), "carlos v2") {
3261 t.Fatal("binary not swapped")
3262 }
3263 prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
3264 if !strings.Contains(string(prev), "carlos v1") {
3265 t.Fatal(".prev is not the old binary")
3266 }
3267 units := filepath.Join(f.a.etcDir, "systemd", "system")
3268 if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err != nil {
3269 t.Error("new unit not installed")
3270 }
3271 if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err == nil {
3272 t.Error("removed unit still installed")
3273 }
3274 if _, err := os.Stat(filepath.Join(f.a.etcDir, "sysusers.d", "carlos.conf")); err != nil {
3275 t.Error("sysusers not installed")
3276 }
3277 if _, err := os.Stat(filepath.Join(f.a.etcDir, "tmpfiles.d", "carlos.conf")); err != nil {
3278 t.Error("tmpfiles not installed")
3279 }
3280 if _, err := os.Stat(filepath.Join(f.a.etcDir, "registry.db")); err == nil {
3281 t.Error("staging data/ copied")
3282 }
3283 // Order of the systemctl calls is the spec.
3284 joined := strings.Join(f.calls, "\n")
3285 order := []string{
3286 "systemctl stop carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service",
3287 "systemctl disable --now carlos-old-only.timer",
3288 "run systemd-sysusers",
3289 "run systemd-tmpfiles --create",
3290 "systemctl daemon-reload",
3291 "systemctl reset-failed carlos-edge",
3292 "systemctl restart carlos-edge",
3293 "systemctl is-active carlos-edge",
3294 "systemctl daemon-reload",
3295 "systemctl enable carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service carlos-new-only.path",
3296 "systemctl start carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-new-only.path",
3297 }
3298 at := 0
3299 for _, want := range order {
3300 i := strings.Index(joined[at:], want)
3301 if i < 0 {
3302 t.Fatalf("call order broken: %q not found after position %d in:\n%s", want, at, joined)
3303 }
3304 at += i + len(want)
3305 }
3306 for _, c := range f.calls {
3307 if strings.HasPrefix(c, "systemctl start ") && strings.Contains(c, ".service") {
3308 t.Fatalf("apply started a service: %s", c)
3309 }
3310 }
3311 if strings.Contains(joined, "systemctl enable --now") {
3312 t.Fatal("enable --now is never used")
3313 }
3314}
3315
3316func TestApplyGateSoakCatchesAFlap(t *testing.T) {
3317 f, req := applyHappy(t)
3318 // Edge comes up, then restarts once during the soak.
3319 restarts := 0
3320 orig := f.a.ops.systemctl
3321 f.a.ops.systemctl = func(args ...string) (string, error) {
3322 if args[0] == "show" {
3323 restarts++
3324 if restarts > 1 {
3325 return "1\n2000\n", nil
3326 }
3327 }
3328 return orig(args...)
3329 }
3330 err := f.a.Run(context.Background(), false)
3331 if err == nil {
3332 t.Fatal("a flapping edge passed the gate")
3333 }
3334 r := f.result()
3335 if r == nil || !r.matches("stable", "v2", req.PromotedAt) || r.Outcome == systemOutcomeApplied {
3336 t.Fatalf("result = %+v", r)
3337 }
3338}
3339
3340func TestApplyPrevIsRefreshedOnlyWhenTheDiskIsKnownGood(t *testing.T) {
3341 // applied.json names v0 but the disk says v1 (a rollback-failed left a
3342 // bad v1 in place, .prev is the good v0): .prev must be left alone.
3343 f, _ := applyHappy(t)
3344 os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755)
3345 f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n"
3346 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644)
3347 _ = f.a.Run(context.Background(), false)
3348 prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
3349 if !strings.Contains(string(prev), "carlos v0") {
3350 t.Fatal(".prev was overwritten by a binary that never passed a gate")
3351 }
3352}
3353
3354func TestApplyPrevUnitsComeFromPrevNotFromDisk(t *testing.T) {
3355 f, _ := applyHappy(t)
3356 os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755)
3357 f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n"
3358 f.answer["plan:"+systemPrevPath(f.a.optDir)] = `{"enabled":["carlos-edge.service"],"inert":[]}`
3359 f.answer["units:"+systemPrevPath(f.a.optDir)] = "carlos-edge.service carlos-v0-only.timer"
3360 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644)
3361 // Inspect the journal mid-flight by failing the gate.
3362 f.edgeUp = false
3363 _ = f.a.Run(context.Background(), false)
3364 // units_removed must be computed against .prev's set (v0-only.timer),
3365 // not the on-disk v1's (old-only.timer).
3366 found := false
3367 for _, c := range f.calls {
3368 if c == "run carlos.prev bootstrap -root "+filepath.Join(f.a.stagingDir(), "prev")+" -plan-json" {
3369 found = true
3370 }
3371 }
3372 if !found {
3373 t.Fatalf("prev units not staged from carlos.prev:\n%s", strings.Join(f.calls, "\n"))
3374 }
3375}
3376```
3377
3378- [ ] **Step 2: Run to verify failure**
3379
3380Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyInstalls|TestApplyGate|TestApplyPrev' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
3381Expected: FAIL with `errNotYetImplemented`.
3382
3383- [ ] **Step 3: Implement the install path**
3384
3385Replace the `install` stub and add the helpers:
3386```go
3387func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error {
3388 target := p.Current.Version
3389 prevOK, err := a.decidePrev(ctx)
3390 if err != nil {
3391 return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err))
3392 }
3393 if !prevOK {
3394 return a.record(req, fmt.Errorf("%w: no known-good carlos.prev to fall back to", errSystemRefused))
3395 }
3396 newDir, prevDir := filepath.Join(a.stagingDir(), "new"), filepath.Join(a.stagingDir(), "prev")
3397 newPlan, newUnits, err := a.stagePlan(ctx, staged, newDir)
3398 if err != nil {
3399 return a.record(req, fmt.Errorf("%w: candidate bootstrap: %v", errSystemRefused, err))
3400 }
3401 prevPlan, prevUnits, err := a.stagePlan(ctx, systemPrevPath(a.optDir), prevDir)
3402 if err != nil {
3403 return a.record(req, fmt.Errorf("%w: carlos.prev bootstrap: %v", errSystemRefused, err))
3404 }
3405 j := systemJournal{Phase: systemPhaseInstalling, Target: target, PrevBuild: a.prevBuild(ctx),
3406 UnitsAdded: diff(newUnits, prevUnits), UnitsRemoved: diff(prevUnits, newUnits), StartedAt: a.ops.now().UTC()}
3407 if err := a.journal(j); err != nil {
3408 return err
3409 }
3410 fail := func(step string, err error) error {
3411 return a.rollback(ctx, req, fmt.Sprintf("%s: %v", step, err), &j, prevPlan)
3412 }
3413 if _, err := a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...); err != nil {
3414 a.ops.logf("system apply: quiesce: %v (continuing)", err)
3415 }
3416 if err := os.Rename(staged, systemBinaryPath(a.optDir)); err != nil {
3417 return fail("install", err)
3418 }
3419 if err := os.Chown(systemBinaryPath(a.optDir), 0, 0); err != nil && !errors.Is(err, os.ErrPermission) {
3420 a.ops.logf("system apply: chown root: %v", err)
3421 }
3422 j.Phase = systemPhaseInstalled
3423 if err := a.journal(j); err != nil {
3424 return fail("journal", err)
3425 }
3426 if err := a.installUnits(newDir, j.UnitsRemoved); err != nil {
3427 return fail("units", err)
3428 }
3429 j.Phase = systemPhaseUnits
3430 if err := a.journal(j); err != nil {
3431 return fail("journal", err)
3432 }
3433 if err := a.restartEdge(); err != nil {
3434 return fail("restart", err)
3435 }
3436 j.Phase = systemPhaseRestarted
3437 if err := a.journal(j); err != nil {
3438 return fail("journal", err)
3439 }
3440 if err := a.gate(ctx, target); err != nil {
3441 return fail("gate", err)
3442 }
3443 j.Phase = systemPhaseGated
3444 if err := a.journal(j); err != nil {
3445 return fail("journal", err)
3446 }
3447 return a.finish(req, target, newPlan)
3448}
3449
3450// finish is steps 11–12, and what recovery runs forward from phase gated.
3451func (a *systemApplier) finish(req systemRequest, target string, plan bootstrapPlan) error {
3452 if err := a.arm(plan); err != nil {
3453 a.ops.logf("system apply: arm: %v", err)
3454 }
3455 if err := writeJSONDurable(a.file("applied.json"), systemApplied{Version: target, Channel: a.channel, At: a.ops.now().UTC()}, 0o644); err != nil {
3456 return err
3457 }
3458 if err := a.writeResult(req, systemOutcomeApplied, ""); err != nil {
3459 return err
3460 }
3461 os.Remove(a.file("journal.json"))
3462 return syncDirBestEffort(a.rootDir())
3463}
3464
3465// decidePrev: refresh carlos.prev from the on-disk binary only when that
3466// binary is known good — applied.json names it, or nothing has ever been
3467// applied and there is no .prev (a hand-rolled box whose running binary
3468// passed the script's own post-condition). Otherwise leave .prev alone.
3469func (a *systemApplier) decidePrev(ctx context.Context) (bool, error) {
3470 prev := systemPrevPath(a.optDir)
3471 _, prevErr := os.Stat(prev)
3472 applied, err := readSystemApplied(a.optDir)
3473 if err != nil {
3474 return false, err
3475 }
3476 disk := a.onDiskBuild(ctx)
3477 known := (applied != nil && applied.Version == disk && disk != "") || (applied == nil && prevErr != nil)
3478 if !known {
3479 return prevErr == nil, nil
3480 }
3481 if err := a.journal(systemJournal{Phase: systemPhaseBackingUp, Target: "", PrevBuild: disk, StartedAt: a.ops.now().UTC()}); err != nil {
3482 return false, err
3483 }
3484 if err := copyFileDurable(systemBinaryPath(a.optDir), prev, 0o755); err != nil {
3485 return false, err
3486 }
3487 os.Remove(a.file("journal.json"))
3488 return true, nil
3489}
3490
3491func (a *systemApplier) prevBuild(ctx context.Context) string {
3492 out, err := a.ops.run(ctx, systemVersionWait, systemPrevPath(a.optDir), "version")
3493 if err != nil {
3494 return ""
3495 }
3496 return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos "))
3497}
3498
3499// stagePlan runs `<binary> bootstrap -root <dir> -plan-json` and returns
3500// that binary's arming plan and the unit files it staged.
3501func (a *systemApplier) stagePlan(ctx context.Context, binary, dir string) (bootstrapPlan, []string, error) {
3502 os.RemoveAll(dir)
3503 if err := os.MkdirAll(dir, 0o700); err != nil {
3504 return bootstrapPlan{}, nil, err
3505 }
3506 out, err := a.ops.run(ctx, systemBootstrapWait, binary, "bootstrap", "-root", dir, "-plan-json")
3507 if err != nil {
3508 return bootstrapPlan{}, nil, err
3509 }
3510 var plan bootstrapPlan
3511 if err := json.Unmarshal([]byte(out), &plan); err != nil {
3512 return bootstrapPlan{}, nil, fmt.Errorf("plan: %w", err)
3513 }
3514 entries, err := os.ReadDir(filepath.Join(dir, "etc", "systemd", "system"))
3515 if err != nil {
3516 return bootstrapPlan{}, nil, err
3517 }
3518 var units []string
3519 for _, e := range entries {
3520 if !e.IsDir() {
3521 units = append(units, e.Name())
3522 }
3523 }
3524 return plan, units, nil
3525}
3526
3527func (a *systemApplier) journal(j systemJournal) error {
3528 return writeJSONDurable(a.file("journal.json"), j, 0o600)
3529}
3530
3531// installUnits: remove what the target build dropped, copy its units in,
3532// then sysusers BEFORE tmpfiles, then daemon-reload. Never the staging
3533// root's data/.
3534func (a *systemApplier) installUnits(stagedRoot string, remove []string) error {
3535 unitDir := filepath.Join(a.etcDir, "systemd", "system")
3536 for _, u := range remove {
3537 if _, err := a.ops.systemctl("disable", "--now", u); err != nil {
3538 a.ops.logf("system apply: disable %s: %v", u, err)
3539 }
3540 os.Remove(filepath.Join(unitDir, u))
3541 }
3542 entries, err := os.ReadDir(filepath.Join(stagedRoot, "etc", "systemd", "system"))
3543 if err != nil {
3544 return err
3545 }
3546 for _, e := range entries {
3547 if e.IsDir() {
3548 continue
3549 }
3550 if err := copyFileDurable(filepath.Join(stagedRoot, "etc", "systemd", "system", e.Name()), filepath.Join(unitDir, e.Name()), 0o644); err != nil {
3551 return err
3552 }
3553 }
3554 sys := filepath.Join(stagedRoot, "etc", "sysusers.d", "carlos.conf")
3555 if _, err := os.Stat(sys); err == nil {
3556 if err := os.MkdirAll(filepath.Join(a.etcDir, "sysusers.d"), 0o755); err != nil {
3557 return err
3558 }
3559 dst := filepath.Join(a.etcDir, "sysusers.d", "carlos.conf")
3560 if err := copyFileDurable(sys, dst, 0o644); err != nil {
3561 return err
3562 }
3563 if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-sysusers", dst); err != nil {
3564 return fmt.Errorf("systemd-sysusers: %w", err)
3565 }
3566 }
3567 tmp := filepath.Join(stagedRoot, "etc", "tmpfiles.d", "carlos.conf")
3568 if err := os.MkdirAll(filepath.Join(a.etcDir, "tmpfiles.d"), 0o755); err != nil {
3569 return err
3570 }
3571 dst := filepath.Join(a.etcDir, "tmpfiles.d", "carlos.conf")
3572 if err := copyFileDurable(tmp, dst, 0o644); err != nil {
3573 return err
3574 }
3575 if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-tmpfiles", "--create", dst); err != nil {
3576 a.ops.logf("system apply: systemd-tmpfiles: %v (continuing, as the script does)", err)
3577 }
3578 _, err = a.ops.systemctl("daemon-reload")
3579 return err
3580}
3581
3582// restartEdge: reset-failed FIRST — a start-limited unit refuses restart —
3583// then restart.
3584func (a *systemApplier) restartEdge() error {
3585 a.ops.systemctl("reset-failed", systemEdgeUnit)
3586 _, err := a.ops.systemctl("restart", systemEdgeUnit)
3587 return err
3588}
3589
3590// gate: within gateWait the edge is active, answers on loopback, and the
3591// installed binary reports the target; then a soak during which neither
3592// NRestarts nor ActiveEnterTimestampMonotonic moves.
3593func (a *systemApplier) gate(ctx context.Context, target string) error {
3594 wait, soak := a.gateWait, a.soak
3595 deadline := a.ops.now().Add(wait)
3596 var lastErr error
3597 for {
3598 lastErr = a.gateOnce(ctx, target)
3599 if lastErr == nil {
3600 break
3601 }
3602 if !a.ops.now().Before(deadline) {
3603 return lastErr
3604 }
3605 a.ops.sleep(time.Second)
3606 }
3607 before, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit)
3608 if err != nil {
3609 return err
3610 }
3611 a.ops.sleep(soak)
3612 after, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit)
3613 if err != nil {
3614 return err
3615 }
3616 if strings.TrimSpace(before) != strings.TrimSpace(after) {
3617 return fmt.Errorf("edge restarted during the soak (%s → %s)", strings.ReplaceAll(strings.TrimSpace(before), "\n", "/"), strings.ReplaceAll(strings.TrimSpace(after), "\n", "/"))
3618 }
3619 if err := a.gateOnce(ctx, target); err != nil {
3620 return err
3621 }
3622 return nil
3623}
3624
3625func (a *systemApplier) gateOnce(ctx context.Context, target string) error {
3626 if out, err := a.ops.systemctl("is-active", systemEdgeUnit); err != nil || strings.TrimSpace(out) != "active" {
3627 return fmt.Errorf("%s is %s", systemEdgeUnit, strings.TrimSpace(out))
3628 }
3629 if err := a.ops.probeEdge(); err != nil {
3630 return fmt.Errorf("edge probe: %w", err)
3631 }
3632 if got := a.onDiskBuild(ctx); got != target {
3633 return fmt.Errorf("installed binary reports %q, not %s", got, target)
3634 }
3635 return nil
3636}
3637
3638// arm is systemctlPlan verbatim: enable all, start timers and paths, never
3639// a service — which is why apply can call it from inside itself.
3640func (a *systemApplier) arm(plan bootstrapPlan) error {
3641 for _, args := range systemctlPlan(plan.Enabled) {
3642 if _, err := a.ops.systemctl(args...); err != nil {
3643 return fmt.Errorf("systemctl %s: %w", strings.Join(args, " "), err)
3644 }
3645 }
3646 return nil
3647}
3648
3649func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
3650 return errors.New("rollback not implemented: " + reason) // Task 8c
3651}
3652
3653func diff(have, drop []string) []string {
3654 skip := map[string]bool{}
3655 for _, d := range drop {
3656 skip[d] = true
3657 }
3658 var out []string
3659 for _, h := range have {
3660 if !skip[h] {
3661 out = append(out, h)
3662 }
3663 }
3664 return out
3665}
3666
3667// copyFileDurable is writeFileDurable for a file too large to hold: temp,
3668// fsync, rename, directory fsync.
3669func copyFileDurable(src, dst string, mode os.FileMode) (err error) {
3670 in, err := os.Open(src)
3671 if err != nil {
3672 return err
3673 }
3674 defer in.Close()
3675 tmp := dst + ".tmp"
3676 out, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode)
3677 if err != nil {
3678 return err
3679 }
3680 defer func() {
3681 if err != nil {
3682 out.Close()
3683 os.Remove(tmp)
3684 }
3685 }()
3686 if _, err = io.Copy(out, in); err != nil {
3687 return err
3688 }
3689 if err = out.Chmod(mode); err != nil {
3690 return err
3691 }
3692 if err = out.Sync(); err != nil {
3693 return err
3694 }
3695 if err = out.Close(); err != nil {
3696 return err
3697 }
3698 if err = os.Rename(tmp, dst); err != nil {
3699 return err
3700 }
3701 return syncDirBestEffort(filepath.Dir(dst))
3702}
3703
3704func syncDirBestEffort(dir string) error {
3705 d, err := os.Open(dir)
3706 if err != nil {
3707 return err
3708 }
3709 defer d.Close()
3710 return d.Sync()
3711}
3712```
3713The test for the flap expects `TestApplyGateSoakCatchesAFlap` to end in a rollback; with the 8c stub it ends in an error from `rollback`, and `result.json` is not yet written by the stub. Make the stub write `result.json` with `outcome: rolled-back` and reason, then return the error — that is the 8c contract in miniature and keeps this task's tests honest:
3714```go
3715func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
3716 _ = a.writeResult(req, systemOutcomeRolledBack, reason)
3717 return errors.New("rollback not implemented: " + reason)
3718}
3719```
3720
3721- [ ] **Step 4: Run the tests**
3722
3723Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
3724Expected: PASS. `TestApplyInstallsGatesArmsAndRecords` checks the exact systemctl order; if a call is missing, fix the code, not the expected list, unless the spec says otherwise.
3725
3726- [ ] **Step 5: Commit**
3727
3728```bash
3729gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
3730host: carlos system apply — install, gate, arm, record
3731
3732.prev refreshed only when the disk is known good; both unit sets
3733staged from the binaries that own them; quiesce before the rename;
3734sysusers before tmpfiles; reset-failed before restart; a 15 s soak on
3735NRestarts; arming after the gate via systemctlPlan.
3736
3737Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3738Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
3739EOF
3740```
3741
3742---
3743
3744### Task 8c: `carlos system apply` — rollback, recovery, `--recover`, the real `sysOps`
3745
3746**Files:**
3747- Modify: `internal/host/systemapply.go` (replace `rollback` and `recoverIfNeeded`; add `RunSystemApply` and `productionSysOps`)
3748- Test: `internal/host/systemapply_test.go` (add)
3749
3750**Interfaces:**
3751- Produces:
3752 ```go
3753 func RunSystemApply(args []string) // flags: -recover; exits 1 on a non-nil Run error
3754 func productionSysOps() sysOps
3755 func (a *systemApplier) rollback(ctx, req, reason, j *systemJournal, prevPlan bootstrapPlan) error
3756 func (a *systemApplier) recoverIfNeeded(ctx) error
3757 func (a *systemApplier) restoreFiles(ctx, j *systemJournal) (bootstrapPlan, error) // files only: binary, units, sysusers, tmpfiles, daemon-reload, enable
3758 ```
3759
3760- [ ] **Step 1: Write the failing tests**
3761
3762```go
3763func TestApplyRollsBackWhenTheGateFails(t *testing.T) {
3764 f, req := applyHappy(t)
3765 f.edgeUp = false
3766 err := f.a.Run(context.Background(), false)
3767 if err == nil {
3768 t.Fatal("a failed gate must exit non-zero")
3769 }
3770 r := f.result()
3771 if r == nil || r.Outcome != systemOutcomeRolledBack || !r.matches("stable", "v2", req.PromotedAt) || !strings.Contains(r.Reason, "gate") {
3772 t.Fatalf("result = %+v", r)
3773 }
3774 bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
3775 if !strings.Contains(string(bin), "carlos v1") {
3776 t.Fatal("binary not restored")
3777 }
3778 if _, err := os.Stat(systemPrevPath(f.a.optDir)); err != nil {
3779 t.Fatal(".prev consumed by the rollback")
3780 }
3781 units := filepath.Join(f.a.etcDir, "systemd", "system")
3782 if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err == nil {
3783 t.Error("added unit survived the rollback")
3784 }
3785 if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err != nil {
3786 t.Error("removed unit not restored")
3787 }
3788 if ap, _ := readSystemApplied(f.a.optDir); ap != nil {
3789 t.Fatalf("applied.json written by a rollback: %+v", ap)
3790 }
3791 if j, _ := readSystemJournal(f.a.optDir); j != nil {
3792 t.Fatal("journal left open after rollback")
3793 }
3794 joined := strings.Join(f.calls, "\n")
3795 if !strings.Contains(joined, "systemctl disable --now carlos-new-only.path") {
3796 t.Error("added unit not disabled")
3797 }
3798 // reset-failed precedes the rollback's restart too, and the previous
3799 // plan is armed afterwards.
3800 i := strings.LastIndex(joined, "systemctl reset-failed carlos-edge")
3801 k := strings.LastIndex(joined, "systemctl restart carlos-edge")
3802 m := strings.LastIndex(joined, "systemctl start carlos-adopt.timer")
3803 if !(i < k && k < m) {
3804 t.Fatalf("rollback order wrong:\n%s", joined)
3805 }
3806}
3807
3808func TestApplyRollbackFailedLeavesPrevAndAppliedAlone(t *testing.T) {
3809 f, req := applyHappy(t)
3810 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v1", Channel: "stable", At: f.now}, 0o644)
3811 f.edgeUp = false // never comes up, on either binary
3812 if err := f.a.Run(context.Background(), false); err == nil {
3813 t.Fatal("must exit non-zero")
3814 }
3815 r := f.result()
3816 if r == nil || r.Outcome != systemOutcomeRollbackFailed || !r.matches("stable", "v2", req.PromotedAt) {
3817 t.Fatalf("result = %+v", r)
3818 }
3819 ap, _ := readSystemApplied(f.a.optDir)
3820 if ap == nil || ap.Version != "v1" {
3821 t.Fatalf("applied = %+v, must still name v1", ap)
3822 }
3823 prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
3824 if !strings.Contains(string(prev), "carlos v1") {
3825 t.Fatal(".prev damaged")
3826 }
3827 // A second apply of a NEW version must not refresh .prev from the
3828 // unproven disk binary.
3829 f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v1\n" // rollback restored v1 on disk
3830 os.Remove(systemRequestPath(f.a.dataDir))
3831 f.promote("v3", []byte("#!/bin/sh\necho carlos v3\n"), false)
3832 f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v3\n"
3833 f.answer["plan:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["plan:"+systemPrevPath(f.a.optDir)]
3834 f.answer["units:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["units:"+systemPrevPath(f.a.optDir)]
3835 _ = f.a.Run(context.Background(), false)
3836 prev2, _ := os.ReadFile(systemPrevPath(f.a.optDir))
3837 if string(prev2) != string(prev) {
3838 t.Fatal(".prev refreshed after rollback-failed")
3839 }
3840}
3841
3842func TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime(t *testing.T) {
3843 for _, phase := range []string{systemPhaseInstalling, systemPhaseInstalled, systemPhaseUnits, systemPhaseRestarted} {
3844 t.Run(phase, func(t *testing.T) {
3845 f, _ := applyHappy(t)
3846 // Simulate: v2 installed, journal open at phase, box rebooted.
3847 os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
3848 os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755)
3849 os.WriteFile(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path"), []byte("x"), 0o644)
3850 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
3851 systemJournal{Phase: phase, Target: "v2", PrevBuild: "v1", UnitsAdded: []string{"carlos-new-only.path"}, UnitsRemoved: []string{"carlos-old-only.timer"}, StartedAt: f.now}, 0o600)
3852 f.edgeStartedThisBoot = false
3853 if err := f.a.Run(context.Background(), true); err != nil {
3854 t.Fatal(err)
3855 }
3856 bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
3857 if !strings.Contains(string(bin), "carlos v1") {
3858 t.Fatal("binary not restored")
3859 }
3860 if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path")); err == nil {
3861 t.Error("added unit survived")
3862 }
3863 if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-old-only.timer")); err != nil {
3864 t.Error("removed unit not restored")
3865 }
3866 joined := strings.Join(f.calls, "\n")
3867 if strings.Contains(joined, "systemctl restart carlos-edge") || strings.Contains(joined, "systemctl start ") {
3868 t.Fatalf("boot recovery restarted or started something:\n%s", joined)
3869 }
3870 if !strings.Contains(joined, "systemctl enable carlos-edge.service") {
3871 t.Fatal("previous plan not enabled for the boot")
3872 }
3873 if r := f.result(); r == nil || r.Outcome != systemOutcomeRolledBack || !strings.Contains(r.Reason, phase) {
3874 t.Fatalf("result = %+v", r)
3875 }
3876 if j, _ := readSystemJournal(f.a.optDir); j != nil {
3877 t.Fatal("journal left open")
3878 }
3879 })
3880 }
3881 // Runtime (OnFailure=): same, plus a non-blocking restart and the
3882 // previous plan's timers/paths started.
3883 f, _ := applyHappy(t)
3884 os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
3885 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
3886 systemJournal{Phase: systemPhaseUnits, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
3887 f.edgeStartedThisBoot = true
3888 if err := f.a.Run(context.Background(), true); err != nil {
3889 t.Fatal(err)
3890 }
3891 joined := strings.Join(f.calls, "\n")
3892 for _, want := range []string{"systemctl stop carlos-adopt.timer", "systemctl reset-failed carlos-edge", "systemctl restart --no-block carlos-edge", "systemctl start carlos-adopt.timer"} {
3893 if !strings.Contains(joined, want) {
3894 t.Errorf("runtime recovery lacks %q:\n%s", want, joined)
3895 }
3896 }
3897}
3898
3899func TestRecoveryFinishesForwardFromGated(t *testing.T) {
3900 f, req := applyHappy(t)
3901 os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755)
3902 writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644)
3903 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
3904 systemJournal{Phase: systemPhaseGated, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
3905 if err := f.a.Run(context.Background(), true); err != nil {
3906 t.Fatal(err)
3907 }
3908 bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
3909 if !strings.Contains(string(bin), "carlos v2") {
3910 t.Fatal("a gated apply was undone")
3911 }
3912 ap, _ := readSystemApplied(f.a.optDir)
3913 if ap == nil || ap.Version != "v2" {
3914 t.Fatalf("applied = %+v", ap)
3915 }
3916 if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied {
3917 t.Fatalf("result = %+v", r)
3918 }
3919}
3920
3921func TestRecoveryRemovesTheJournalLast(t *testing.T) {
3922 // A restart that fails mid-recovery leaves the journal for the next run.
3923 f, _ := applyHappy(t)
3924 os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
3925 writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
3926 systemJournal{Phase: systemPhaseInstalled, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
3927 orig := f.a.ops.systemctl
3928 f.a.ops.systemctl = func(args ...string) (string, error) {
3929 if args[0] == "daemon-reload" {
3930 return "", errors.New("systemd busy")
3931 }
3932 return orig(args...)
3933 }
3934 if err := f.a.Run(context.Background(), true); err == nil {
3935 t.Fatal("a failed recovery must exit non-zero")
3936 }
3937 if j, _ := readSystemJournal(f.a.optDir); j == nil {
3938 t.Fatal("journal removed although recovery did not complete")
3939 }
3940}
3941```
3942
3943- [ ] **Step 2: Run to verify failure**
3944
3945Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyRolls|TestApplyRollback|TestRecovery' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
3946Expected: FAIL ("rollback not implemented", "recovery not implemented").
3947
3948- [ ] **Step 3: Implement**
3949
3950Replace `rollback` and `recoverIfNeeded`; add `restoreFiles`, `RunSystemApply`, `productionSysOps`:
3951```go
3952// rollback is step 13: back to carlos.prev and its units, restart, gate,
3953// arm the previous plan. applied.json is untouched — the .prev build is
3954// what passed a gate, and it is what applied.json already names.
3955func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
3956 a.ops.logf("system apply: %s %s: rolling back: %s", a.channel, j.Target, reason)
3957 plan, err := a.restoreFiles(ctx, j)
3958 if err != nil {
3959 _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restore: "+err.Error())
3960 os.Remove(a.file("journal.json"))
3961 return fmt.Errorf("system apply: rollback of %s failed: %v", j.Target, err)
3962 }
3963 if err := a.restartEdge(); err != nil {
3964 _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restart: "+err.Error())
3965 os.Remove(a.file("journal.json"))
3966 return fmt.Errorf("system apply: rollback restart failed: %v", err)
3967 }
3968 if err := a.gate(ctx, j.PrevBuild); err != nil {
3969 _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; previous build failed its gate too: "+err.Error())
3970 os.Remove(a.file("journal.json"))
3971 return fmt.Errorf("system apply: rollback gate failed: %v", err)
3972 }
3973 if err := a.arm(plan); err != nil {
3974 a.ops.logf("system apply: arm previous plan: %v", err)
3975 }
3976 if err := a.writeResult(req, systemOutcomeRolledBack, reason); err != nil {
3977 return err
3978 }
3979 os.Remove(a.file("journal.json"))
3980 return fmt.Errorf("system apply: %s rolled back: %s", j.Target, reason)
3981}
3982
3983// restoreFiles is the files-only half shared by rollback and recovery:
3984// carlos.prev over carlos (keeping .prev), units_added disabled and
3985// removed, the previous unit set copied back (bringing units_removed
3986// back), sysusers, tmpfiles, daemon-reload, and `enable` of the previous
3987// plan so it comes up at boot. No restart, no start.
3988func (a *systemApplier) restoreFiles(ctx context.Context, j *systemJournal) (bootstrapPlan, error) {
3989 prev := systemPrevPath(a.optDir)
3990 if _, err := os.Stat(prev); err != nil {
3991 return bootstrapPlan{}, fmt.Errorf("no carlos.prev: %w", err)
3992 }
3993 a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...)
3994 if err := copyFileDurable(prev, systemBinaryPath(a.optDir), 0o755); err != nil {
3995 return bootstrapPlan{}, err
3996 }
3997 prevDir := filepath.Join(a.stagingDir(), "prev")
3998 plan, _, err := a.stagePlan(ctx, prev, prevDir)
3999 if err != nil {
4000 return bootstrapPlan{}, fmt.Errorf("carlos.prev bootstrap: %w", err)
4001 }
4002 if err := a.installUnits(prevDir, j.UnitsAdded); err != nil {
4003 return bootstrapPlan{}, err
4004 }
4005 if _, err := a.ops.systemctl(append([]string{"enable"}, plan.Enabled...)...); err != nil {
4006 return bootstrapPlan{}, err
4007 }
4008 return plan, nil
4009}
4010
4011// recoverIfNeeded: an open journal means a previous run was cut off.
4012// Before gated → back, files only, restart only if the edge has started
4013// this boot; gated → forward. The journal is removed LAST.
4014func (a *systemApplier) recoverIfNeeded(ctx context.Context) error {
4015 j, err := readSystemJournal(a.optDir)
4016 if err != nil || j == nil {
4017 return err
4018 }
4019 req, _ := a.readRequest()
4020 if req.Version == "" {
4021 req = systemRequest{Version: j.Target, PromotedAt: j.StartedAt}
4022 }
4023 a.ops.logf("system apply: recovering an apply of %s interrupted at %s", j.Target, j.Phase)
4024 switch j.Phase {
4025 case systemPhaseGated:
4026 plan, _, err := a.stagePlan(ctx, systemBinaryPath(a.optDir), filepath.Join(a.stagingDir(), "new"))
4027 if err != nil {
4028 return fmt.Errorf("system apply: recovery: %w", err)
4029 }
4030 return a.finish(req, j.Target, plan)
4031 case systemPhaseBackingUp:
4032 os.Remove(systemPrevPath(a.optDir) + ".tmp")
4033 os.Remove(a.file("journal.json"))
4034 return nil
4035 }
4036 plan, err := a.restoreFiles(ctx, j)
4037 if err != nil {
4038 return fmt.Errorf("system apply: recovery: %w", err) // journal kept: the next run recovers again
4039 }
4040 if a.ops.edgeStartedThisBoot() {
4041 a.ops.systemctl("reset-failed", systemEdgeUnit)
4042 if _, err := a.ops.systemctl("restart", "--no-block", systemEdgeUnit); err != nil {
4043 return fmt.Errorf("system apply: recovery: restart: %w", err)
4044 }
4045 var armed []string
4046 for _, u := range plan.Enabled {
4047 if strings.HasSuffix(u, ".timer") || strings.HasSuffix(u, ".path") {
4048 armed = append(armed, u)
4049 }
4050 }
4051 if len(armed) > 0 {
4052 a.ops.systemctl(append([]string{"start"}, armed...)...)
4053 }
4054 }
4055 if err := a.writeResult(req, systemOutcomeRolledBack, "interrupted at "+j.Phase); err != nil {
4056 return err
4057 }
4058 os.Remove(a.file("journal.json"))
4059 return syncDirBestEffort(a.rootDir())
4060}
4061
4062// RunSystemApply is `carlos system apply [--recover]`: the root oneshot.
4063func RunSystemApply(args []string) {
4064 fs := flag.NewFlagSet("system apply", flag.ExitOnError)
4065 recover := fs.Bool("recover", false, "recover an interrupted apply (files only) and exit: the boot and OnFailure unit")
4066 fs.Usage = cliflag.Usage(fs, "")
4067 _ = fs.Parse(args)
4068 if os.Geteuid() != 0 {
4069 log.Fatal("carlos system apply is the root oneshot behind carlos-system-update.service; it is not for operators")
4070 }
4071 ctx := context.Background()
4072 store, err := openDeployStore(ctx)
4073 if err != nil {
4074 log.Fatalf("system apply: %v", err)
4075 }
4076 pub, err := release.ParsePublicKey(os.Getenv("CARLOS_RELEASE_PUBKEY"), "CARLOS_RELEASE_PUBKEY")
4077 if err != nil {
4078 log.Fatalf("system apply: %v", err)
4079 }
4080 a := &systemApplier{
4081 ops: productionSysOps(), store: store, pub: pub, channel: systemChannel(os.Getenv),
4082 optDir: release.OptDir(), dataDir: dataDir(), etcDir: "/etc", build: version.Build,
4083 platform: runtime.GOOS + "-" + runtime.GOARCH, gateWait: systemGateWait, soak: systemGateSoak,
4084 }
4085 if err := a.Run(ctx, *recover); err != nil {
4086 log.Fatalf("%v", err)
4087 }
4088}
4089
4090func productionSysOps() sysOps {
4091 return sysOps{
4092 systemctl: func(args ...string) (string, error) {
4093 out, err := exec.Command("systemctl", args...).CombinedOutput()
4094 if err != nil {
4095 return string(out), fmt.Errorf("systemctl %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out)))
4096 }
4097 return string(out), nil
4098 },
4099 run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) {
4100 ctx, cancel := context.WithTimeout(ctx, timeout)
4101 defer cancel()
4102 out, err := exec.CommandContext(ctx, name, args...).Output()
4103 if err != nil {
4104 return string(out), fmt.Errorf("%s %s: %w", name, strings.Join(args, " "), err)
4105 }
4106 return string(out), nil
4107 },
4108 now: time.Now,
4109 sleep: time.Sleep,
4110 probeEdge: func() error {
4111 // Any HTTP answer on loopback proves the process serves; a box
4112 // with no route yet must still pass (spec §6 step 10).
4113 c := &http.Client{Timeout: 3 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
4114 resp, err := c.Get("http://127.0.0.1/")
4115 if err != nil {
4116 return err
4117 }
4118 resp.Body.Close()
4119 return nil
4120 },
4121 edgeStartedThisBoot: func() bool {
4122 out, err := exec.Command("systemctl", "show", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit).Output()
4123 return err == nil && strings.TrimSpace(string(out)) != "" && strings.TrimSpace(string(out)) != "0"
4124 },
4125 logf: log.Printf,
4126 }
4127}
4128```
4129Imports to add: `"flag"`, `"log"`, `"net/http"`, `"os/exec"`, `"github.com/carlosframework/platform/internal/cliflag"`, `"github.com/carlosframework/platform/internal/version"`. In the fixture, set `f.a.platform = "linux-arm64"` (from 8a). The edge's plain-HTTP listener: confirm the port with `grep -n 'ListenAndServe\|":80"\|Listen(' internal/edge/edge.go | head`; if the edge only listens on 443, probe `https://127.0.0.1/` with `InsecureSkipVerify` and a `ServerName` of `localhost` instead, and say which in the commit.
4130
4131- [ ] **Step 4: Run the whole host package**
4132
4133Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
4134Expected: `ok`.
4135
4136- [ ] **Step 5: Commit**
4137
4138```bash
4139gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
4140host: carlos system apply — rollback, recovery, and the real box
4141
4142Rollback restores carlos.prev and its unit set, gates it, arms the
4143previous plan, and leaves applied.json alone. Recovery is files-only
4144and goes forward from phase gated; it restarts the edge only when the
4145edge has started this boot, and removes the journal last.
4146
4147Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4148Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
4149EOF
4150```
4151
4152---
4153### Task 9: `carlos system` dispatch and the `apply` verb
4154
4155**Files:**
4156- Create: `system.go` (package main), `system_test.go`
4157- Modify: `main.go` (the `switch cmd` at `:30-134`, `usage()` at `:139-233`), `update.go` (`noHintVerbs` at `:76-85`)
4158
4159**Interfaces:**
4160- Consumes: `host.RunSystemApply` (Task 8c), `main.go` dispatch pattern (`case "ops": ops.Run(args)`), `usage()` section "On the host".
4161- Produces:
4162 ```go
4163 func runSystem(args []string) // carlos system <verb> | carlos system router <verb>
4164 func systemVerb(args []string) (verb string, rest []string, component string) // strips an optional leading "router"
4165 ```
4166 Verbs in this task: `apply` → `host.RunSystemApply(rest)`; `update`/`status`/`rollback` → `log.Fatal("carlos system " + verb + " arrives in a later task")` placeholders that Task 12 replaces; `help`/none → usage, exit 2.
4167
4168- [ ] **Step 1: Write the failing test**
4169
4170```go
4171package main
4172
4173import "testing"
4174
4175func TestSystemVerbAcceptsTheComponentForm(t *testing.T) {
4176 cases := []struct {
4177 in []string
4178 verb, comp string
4179 rest int
4180 }{
4181 {[]string{"update", "--channel", "canary"}, "update", "router", 2},
4182 {[]string{"router", "update"}, "update", "router", 0},
4183 {[]string{"router", "status"}, "status", "router", 0},
4184 {[]string{"apply", "--recover"}, "apply", "router", 1},
4185 {[]string{}, "", "router", 0},
4186 {[]string{"router"}, "", "router", 0},
4187 }
4188 for _, c := range cases {
4189 verb, rest, comp := systemVerb(c.in)
4190 if verb != c.verb || comp != c.comp || len(rest) != c.rest {
4191 t.Errorf("systemVerb(%v) = %q %v %q", c.in, verb, rest, comp)
4192 }
4193 }
4194 if verb, _, _ := systemVerb([]string{"console", "update"}); verb != "" {
4195 t.Errorf("an unknown component must not be a verb: got %q", verb)
4196 }
4197}
4198```
4199
4200- [ ] **Step 2: Run to verify failure**
4201
4202Run: `export TMPDIR=/tmp/c; out=$(go test . -run TestSystemVerb -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc`
4203Expected: undefined `systemVerb`.
4204
4205- [ ] **Step 3: Implement `system.go`**
4206
4207```go
4208package main
4209
4210import (
4211 "fmt"
4212 "log"
4213 "os"
4214
4215 "github.com/carlosframework/platform/internal/host"
4216)
4217
4218// carlos system: what is not an app (spec §0, the console-as-an-app
4219// dividing line). One component today, the router — the substrate binary.
4220// `carlos system router update` is the same verb scoped to it, so the
4221// command Paul named works and a future split needs no new verb.
4222
4223const systemComponentRouter = "router"
4224
4225func systemVerb(args []string) (verb string, rest []string, component string) {
4226 component = systemComponentRouter
4227 if len(args) > 0 && args[0] == systemComponentRouter {
4228 args = args[1:]
4229 }
4230 if len(args) == 0 {
4231 return "", nil, component
4232 }
4233 switch args[0] {
4234 case "update", "status", "rollback", "apply", "help", "-h", "--help":
4235 return args[0], args[1:], component
4236 }
4237 return "", args, component
4238}
4239
4240func runSystem(args []string) {
4241 verb, rest, _ := systemVerb(args)
4242 switch verb {
4243 case "apply":
4244 host.RunSystemApply(rest)
4245 case "update":
4246 runSystemUpdate(rest)
4247 case "status":
4248 runSystemStatus(rest)
4249 case "rollback":
4250 runSystemRollback(rest)
4251 case "help", "-h", "--help":
4252 systemUsage()
4253 default:
4254 if len(args) > 0 && verb == "" {
4255 fmt.Fprintf(os.Stderr, "system: unknown verb %q\n\n", args[0])
4256 }
4257 systemUsage()
4258 os.Exit(2)
4259 }
4260}
4261
4262func systemUsage() {
4263 fmt.Fprint(os.Stderr, `usage: carlos system <verb> [flags]
4264 carlos system router <verb> [flags]
4265
4266 update promote a platform release to a channel; every box on it rolls itself
4267 status every box: platform, channel, running build, channel target, last apply outcome
4268 rollback point a channel back at the release before this one
4269 apply (root oneshot behind carlos-system-update.service; not for operators)
4270
4271The platform is not an app: the console is updated with carlos deploy --app console.
4272`)
4273}
4274
4275// Placeholders until Task 12.
4276func runSystemUpdate(args []string) { log.Fatal("carlos system update: not yet wired to the console") }
4277func runSystemStatus(args []string) { log.Fatal("carlos system status: not yet wired to the console") }
4278func runSystemRollback(args []string) { log.Fatal("carlos system rollback: not yet wired to the console") }
4279```
4280`main.go`: add `case "system": runSystem(args)` after the `ops` case; in `usage()` under "On the host" add ` system platform self-update: update | status | rollback (apply is the root oneshot)`; in `update.go` `noHintVerbs` add `"system": true`.
4281
4282- [ ] **Step 4: Run and check the binary**
4283
4284Run: `export TMPDIR=/tmp/c; out=$(go test . -run 'TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go build -o /tmp/c/carlos . && /tmp/c/carlos system help 2>&1 | head -3; /tmp/c/carlos system apply 2>&1 | head -1`
4285Expected: PASS; usage printed; `apply` as non-root prints "not for operators".
4286
4287- [ ] **Step 5: Commit**
4288
4289```bash
4290gofmt -l . && git add system.go system_test.go main.go update.go && git commit -q -F - <<'EOF'
4291cli: carlos system, with apply wired to the root oneshot
4292
4293Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4294Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
4295EOF
4296```
4297
4298---
4299
4300### Task 10: The public release feed (`internal/publicrelease`)
4301
4302**Files:**
4303- Create: `internal/publicrelease/feed.go`, `internal/publicrelease/feed_test.go`
4304- Modify: `update.go` (`fetchLatest` `:100-119`, `selfReplace` `:295-327`, `fetchUpdateFile` `:358-369`)
4305
4306**Interfaces:**
4307- Consumes: the package vars `updateLatestURL`, `updateDownloadBase`, `cliUpdatePubKey` in `update.go` (kept, since `update_test.go` repoints them).
4308- Produces:
4309 ```go
4310 package publicrelease
4311 type Feed struct {
4312 LatestURL string // https://apt.carlosframework.com/latest.txt
4313 DownloadBase string // https://github.com/carlosframework/releases/releases/download (tag-pinned; never latest/download)
4314 PubKey string // base64 ed25519 public key that signs SHA256SUMS.txt
4315 Client *http.Client // nil = a 5-minute client
4316 }
4317 func Default() Feed
4318 var TagRe = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+$`)
4319 func (f Feed) Latest(ctx context.Context) (string, error)
4320 func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) // asset → sha, signature verified
4321 func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) (data []byte, sha256hex string, err error) // verified against Sums
4322 func AssetName(platform string) string // "carlos-" + platform
4323 ```
4324
4325- [ ] **Step 1: Write the failing tests**
4326
4327```go
4328package publicrelease
4329
4330import (
4331 "context"
4332 "crypto/ed25519"
4333 "crypto/sha256"
4334 "encoding/base64"
4335 "encoding/hex"
4336 "fmt"
4337 "net/http"
4338 "net/http/httptest"
4339 "strings"
4340 "testing"
4341)
4342
4343func testFeed(t *testing.T, sign bool, assets map[string][]byte) (Feed, string) {
4344 t.Helper()
4345 pub, priv, _ := ed25519.GenerateKey(nil)
4346 var sums strings.Builder
4347 for name, body := range assets {
4348 s := sha256.Sum256(body)
4349 fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name)
4350 }
4351 sig := ed25519.Sign(priv, []byte(sums.String()))
4352 mux := http.NewServeMux()
4353 mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") })
4354 mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) })
4355 mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) {
4356 if sign {
4357 fmt.Fprintln(w, base64.StdEncoding.EncodeToString(sig))
4358 } else {
4359 fmt.Fprintln(w, base64.StdEncoding.EncodeToString(make([]byte, ed25519.SignatureSize)))
4360 }
4361 })
4362 for name, body := range assets {
4363 body := body
4364 mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) })
4365 }
4366 ts := httptest.NewServer(mux)
4367 t.Cleanup(ts.Close)
4368 return Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)}, ts.URL
4369}
4370
4371func TestFeedLatestAndVerifiedAsset(t *testing.T) {
4372 body := []byte("ELF arm64")
4373 f, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body, "carlos-linux-amd64": []byte("ELF amd64")})
4374 ctx := context.Background()
4375 tag, err := f.Latest(ctx)
4376 if err != nil || tag != "v0.18.0" {
4377 t.Fatalf("latest = %q %v", tag, err)
4378 }
4379 data, sha, err := f.Asset(ctx, tag, AssetName("linux-arm64"), 1<<20)
4380 if err != nil || string(data) != string(body) {
4381 t.Fatalf("asset: %v", err)
4382 }
4383 s := sha256.Sum256(body)
4384 if sha != hex.EncodeToString(s[:]) {
4385 t.Fatalf("sha = %s", sha)
4386 }
4387 if _, _, err := f.Asset(ctx, tag, "carlos-windows-amd64.zip", 1<<20); err == nil {
4388 t.Fatal("an asset absent from the sums was accepted")
4389 }
4390}
4391
4392func TestFeedRefusesABadSignatureAndATamperedAsset(t *testing.T) {
4393 body := []byte("ELF")
4394 f, _ := testFeed(t, false, map[string][]byte{"carlos-linux-arm64": body})
4395 if _, err := f.Sums(context.Background(), "v0.18.0"); err == nil || !strings.Contains(err.Error(), "signature") {
4396 t.Fatalf("bad signature: %v", err)
4397 }
4398 // Signed sums, but the served asset differs from the sum.
4399 f2, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body})
4400 f2.DownloadBase = f2.DownloadBase // same server
4401 pub, _ := base64.StdEncoding.DecodeString(f2.PubKey)
4402 _ = pub
4403 // Simulate tampering by asking for a limit smaller than the asset: the
4404 // truncated download cannot match its sum.
4405 if _, _, err := f2.Asset(context.Background(), "v0.18.0", "carlos-linux-arm64", 1); err == nil {
4406 t.Fatal("a truncated asset matched its sum")
4407 }
4408}
4409
4410func TestFeedLatestRequiresATag(t *testing.T) {
4411 ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "main") }))
4412 defer ts.Close()
4413 f := Feed{LatestURL: ts.URL, DownloadBase: ts.URL, PubKey: ""}
4414 if _, err := f.Latest(context.Background()); err == nil {
4415 t.Fatal("accepted a non-tag latest")
4416 }
4417}
4418```
4419
4420- [ ] **Step 2: Run to verify failure**
4421
4422Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc`
4423Expected: package does not exist.
4424
4425- [ ] **Step 3: Implement `feed.go`**
4426
4427```go
4428// Package publicrelease reads the vendor's public signed release feed: the
4429// latest tag, the signed checksum list, and a verified asset. Two callers
4430// share it — `carlos update` on a workstation and the console's `carlos
4431// system update` promote — so the verification chain is written once.
4432package publicrelease
4433
4434import (
4435 "context"
4436 "crypto/ed25519"
4437 "crypto/sha256"
4438 "encoding/base64"
4439 "encoding/hex"
4440 "errors"
4441 "fmt"
4442 "io"
4443 "net/http"
4444 "regexp"
4445 "strings"
4446 "time"
4447)
4448
4449type Feed struct {
4450 LatestURL string
4451 DownloadBase string // tag-pinned: <DownloadBase>/<tag>/<file>; never latest/download
4452 PubKey string // base64 ed25519 public key over the raw SHA256SUMS.txt bytes
4453 Client *http.Client
4454}
4455
4456// Default is the vendor feed. The strings are duplicated in update.go's
4457// package vars so update_test.go can repoint them; keep the two in step.
4458func Default() Feed {
4459 return Feed{
4460 LatestURL: "https://apt.carlosframework.com/latest.txt",
4461 DownloadBase: "https://github.com/carlosframework/releases/releases/download",
4462 PubKey: "R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=",
4463 }
4464}
4465
4466var TagRe = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+$`)
4467
4468func AssetName(platform string) string { return "carlos-" + platform }
4469
4470func (f Feed) client() *http.Client {
4471 if f.Client != nil {
4472 return f.Client
4473 }
4474 return &http.Client{Timeout: 5 * time.Minute}
4475}
4476
4477func (f Feed) get(ctx context.Context, url string, limit int64) ([]byte, error) {
4478 req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
4479 if err != nil {
4480 return nil, err
4481 }
4482 res, err := f.client().Do(req)
4483 if err != nil {
4484 return nil, err
4485 }
4486 defer res.Body.Close()
4487 if res.StatusCode != http.StatusOK {
4488 return nil, fmt.Errorf("%s: status %d", url, res.StatusCode)
4489 }
4490 return io.ReadAll(io.LimitReader(res.Body, limit))
4491}
4492
4493// Latest is the newest release tag, and only a tag.
4494func (f Feed) Latest(ctx context.Context) (string, error) {
4495 b, err := f.get(ctx, f.LatestURL, 128)
4496 if err != nil {
4497 return "", err
4498 }
4499 tag := strings.TrimSpace(string(b))
4500 if !TagRe.MatchString(tag) {
4501 return "", fmt.Errorf("%s: %q is not a release tag", f.LatestURL, tag)
4502 }
4503 return tag, nil
4504}
4505
4506// Sums fetches SHA256SUMS.txt and its .sig for tag, verifies the signature
4507// with PubKey, and returns asset → sha256 hex.
4508func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) {
4509 base := f.DownloadBase + "/" + tag + "/"
4510 sums, err := f.get(ctx, base+"SHA256SUMS.txt", 1<<20)
4511 if err != nil {
4512 return nil, err
4513 }
4514 sig, err := f.get(ctx, base+"SHA256SUMS.txt.sig", 4096)
4515 if err != nil {
4516 return nil, fmt.Errorf("%v (a release without a signed checksum list predates self-update)", err)
4517 }
4518 pub, err := base64.StdEncoding.DecodeString(f.PubKey)
4519 if err != nil || len(pub) != ed25519.PublicKeySize {
4520 return nil, errors.New("release feed public key is invalid")
4521 }
4522 rawSig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(string(sig)))
4523 if err != nil || !ed25519.Verify(ed25519.PublicKey(pub), sums, rawSig) {
4524 return nil, errors.New("SHA256SUMS.txt signature does not verify — refusing the release")
4525 }
4526 out := map[string]string{}
4527 for _, line := range strings.Split(string(sums), "\n") {
4528 fields := strings.Fields(line)
4529 if len(fields) == 2 {
4530 out[fields[1]] = fields[0]
4531 }
4532 }
4533 return out, nil
4534}
4535
4536// Asset fetches one named asset for tag and refuses it unless its sha256
4537// matches the signed checksum list.
4538func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) ([]byte, string, error) {
4539 sums, err := f.Sums(ctx, tag)
4540 if err != nil {
4541 return nil, "", err
4542 }
4543 want, ok := sums[asset]
4544 if !ok {
4545 return nil, "", fmt.Errorf("no %s in the release's checksum list", asset)
4546 }
4547 data, err := f.get(ctx, f.DownloadBase+"/"+tag+"/"+asset, limit)
4548 if err != nil {
4549 return nil, "", err
4550 }
4551 s := sha256.Sum256(data)
4552 got := hex.EncodeToString(s[:])
4553 if got != want {
4554 return nil, "", fmt.Errorf("%s does not match its signed checksum — refusing it", asset)
4555 }
4556 return data, got, nil
4557}
4558```
4559`update.go`: replace the bodies of `fetchLatest` and the fetch-and-verify half of `selfReplace` (`:296-327`) with calls through a feed built from the package vars, so tests that repoint the vars still work:
4560```go
4561func updateFeed() publicrelease.Feed {
4562 return publicrelease.Feed{LatestURL: updateLatestURL, DownloadBase: updateDownloadBase, PubKey: cliUpdatePubKey}
4563}
4564```
4565`fetchLatest(timeout)` → `updateFeed()` with `Client: &http.Client{Timeout: timeout}` and `.Latest(ctx)`; in `selfReplace`, `binData, _, err := updateFeed().Asset(ctx, latest, asset, 256<<20)`, keeping the workstation-only half (temp file, lie detector, rename) as is. Delete `fetchUpdateFile` if nothing else calls it. Run `go test . -run 'Update' -count=1` to prove `update_test.go` still passes.
4566
4567- [ ] **Step 4: Run the tests**
4568
4569Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ . -run 'Feed|Update' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc`
4570Expected: `ok` for both packages.
4571
4572- [ ] **Step 5: Commit**
4573
4574```bash
4575gofmt -l . internal/publicrelease && git add internal/publicrelease update.go && git commit -q -F - <<'EOF'
4576publicrelease: the vendor feed's verification chain, written once
4577
4578carlos update and the console's carlos system update share it: latest
4579tag, signed SHA256SUMS, per-asset sha.
4580
4581Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4582Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
4583EOF
4584```
4585
4586---
4587
4588### Task 11: The console doors (`internal/console/apisystem.go`)
4589
4590**Files:**
4591- Create: `internal/console/apisystem.go`, `internal/console/apisystem_test.go`
4592- Modify: `internal/console/authcli.go` `registerCLIRoutes()` (add five `cliRoute` lines), `internal/console/server.go` `Config` (add `SystemFeed publicrelease.Feed`)
4593
4594**Interfaces:**
4595- Consumes: `s.cliRoute(pattern, svcrole.Set{}, handler)`; `principalFrom(ctx)`, `PrincipalUser`; `s.store.Role(OpsAccountID, addr)`, `RoleOwner`; `s.cfg.Releases`, `s.cfg.Key`, `s.cfg.ReleasePub`; `s.bumpSerial(ctx)`; `writeJSON`, `jsonError`, `readJSONStrictLimit`; `status.LoadBoxes`, `release.NewPrefixStore`, `s.openPassRegistry().ListFleets()` (the `fleetMeters` loop, `economicspass.go:648-670`); `release.PromoteSystem`, `release.RollbackSystem`, `release.GetSystemPointer`, `release.RootSigner`; `publicrelease.Feed`; `apiShipMaxBytes`-style limits.
4596- Produces (routes, all person-only, ops-owner-only, bare 404 on refusal like `IsOperator` doors):
4597 ```
4598 GET /api/cli/system/status → {"pointers": {"canary": {...}|null, "stable": ...}, "boxes": [{label, platform, channel, release, target, system:{...}|null, stale}]}
4599 POST /api/cli/system/update → {"job": "<id>"}; body {"channel","version"?,"upload"?,"spread"?,"component"?}
4600 GET /api/cli/system/jobs/{id} → {"steps": ["resolved v0.18.0", ...], "done": bool, "error": ""}
4601 PUT /api/cli/system/upload → {"upload": "<handle>", "size": n}; multipart: meta {"version","platform"}, file
4602 POST /api/cli/system/rollback → {"channel","version"}; body {"channel","component"?}
4603 ```
4604 ```go
4605 func (s *Server) apiSystemOwner(w, r) (addr string, ok bool)
4606 type systemJob struct{ mu sync.Mutex; Steps []string; Done bool; Err string }
4607 func (s *Server) runSystemUpdate(ctx, job *systemJob, actor string, in systemUpdateRequest)
4608 ```
4609 Job and upload registries are fields on `Server`: `systemJobs map[string]*systemJob`, `systemUploads map[string]systemUpload`, guarded by `systemMu sync.Mutex`; one upload slot (128 MiB cap) held in memory, consumed by the job that names it.
4610
4611- [ ] **Step 1: Write the failing tests**
4612
4613```go
4614package console
4615
4616import (
4617 "bytes"
4618 "context"
4619 "crypto/ed25519"
4620 "crypto/sha256"
4621 "encoding/base64"
4622 "encoding/hex"
4623 "fmt"
4624 "mime/multipart"
4625 "net/http"
4626 "net/http/httptest"
4627 "strings"
4628 "testing"
4629 "time"
4630
4631 "github.com/carlosframework/platform/internal/publicrelease"
4632 "github.com/carlosframework/platform/internal/release"
4633 "github.com/carlosframework/platform/internal/status"
4634)
4635
4636// systemFeed serves a signed fake release from an httptest server and
4637// points the fixture's console at it.
4638func systemFeed(t *testing.T, fx *consoleFixture, assets map[string][]byte) {
4639 t.Helper()
4640 pub, priv, _ := ed25519.GenerateKey(nil)
4641 var sums strings.Builder
4642 for name, body := range assets {
4643 s := sha256.Sum256(body)
4644 fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name)
4645 }
4646 sig := base64.StdEncoding.EncodeToString(ed25519.Sign(priv, []byte(sums.String())))
4647 mux := http.NewServeMux()
4648 mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") })
4649 mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) })
4650 mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, sig) })
4651 for name, body := range assets {
4652 body := body
4653 mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) })
4654 }
4655 ts := httptest.NewServer(mux)
4656 t.Cleanup(ts.Close)
4657 fx.srv.cfg.SystemFeed = publicrelease.Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)}
4658}
4659
4660func waitJob(t *testing.T, fx *consoleFixture, token, id string) map[string]any {
4661 t.Helper()
4662 for i := 0; i < 100; i++ {
4663 st, out := getJSON(t, fx.ts.URL+"/api/cli/system/jobs/"+id, token)
4664 if st != http.StatusOK {
4665 t.Fatalf("job = %d %v", st, out)
4666 }
4667 if out["done"] == true {
4668 return out
4669 }
4670 time.Sleep(20 * time.Millisecond)
4671 }
4672 t.Fatal("job never finished")
4673 return nil
4674}
4675
4676func seedBox(t *testing.T, fx *consoleFixture, label, platform, channel, rel string) {
4677 t.Helper()
4678 b := status.CollectBox(nil, status.Probes{Release: rel, Platform: platform, SystemChannel: channel}, time.Now())
4679 b.Label = label
4680 if err := status.WriteBox(context.Background(), fx.rel, b); err != nil {
4681 t.Fatal(err)
4682 }
4683}
4684
4685func TestSystemDoorsAreForOpsOwnersOnly(t *testing.T) {
4686 fx := newConsoleFixture(t)
4687 if err := fx.st.AddMember(OpsAccountID, "member@keymail.dev", RoleMember, time.Now()); err != nil {
4688 t.Fatal(err)
4689 }
4690 member := cliToken(t, fx, "member@keymail.dev", "code-sys-member")
4691 for _, u := range []string{"/api/cli/system/status"} {
4692 if st, _ := getJSON(t, fx.ts.URL+u, member); st != http.StatusNotFound {
4693 t.Errorf("GET %s as a member = %d, want 404", u, st)
4694 }
4695 }
4696 if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound {
4697 t.Errorf("update as a member = %d, want 404", st)
4698 }
4699 if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound {
4700 t.Errorf("rollback as a member = %d, want 404", st)
4701 }
4702 if st, _ := getJSON(t, fx.ts.URL+"/api/cli/system/status", "not-a-token"); st == http.StatusOK {
4703 t.Error("unauthenticated status answered 200")
4704 }
4705}
4706
4707func TestSystemUpdateMirrorsSignsAndBumps(t *testing.T) {
4708 fx := newConsoleFixture(t)
4709 arm := []byte("ELF arm64")
4710 systemFeed(t, fx, map[string][]byte{"carlos-linux-arm64": arm, "carlos-linux-amd64": []byte("ELF amd64")})
4711 seedBox(t, fx, "ie", "linux-arm64", "canary", "v0.17.0")
4712 owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-owner")
4713 before, _, _ := release.ReadSerial(context.Background(), fx.rel)
4714 st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "spread": "2m"}, owner)
4715 if st != http.StatusOK {
4716 t.Fatalf("update = %d %v", st, out)
4717 }
4718 job := waitJob(t, fx, owner, str(t, out, "job"))
4719 if job["error"] != "" {
4720 t.Fatalf("job failed: %v", job)
4721 }
4722 p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary")
4723 if err != nil {
4724 t.Fatal(err)
4725 }
4726 if p.Current.Version != "v0.18.0" || p.Spread != "2m0s" || p.Current.Source != "release:v0.18.0" {
4727 t.Fatalf("pointer = %+v", p)
4728 }
4729 a, ok := p.Current.Artifacts["linux-arm64"]
4730 if !ok {
4731 t.Fatalf("no arm64 artifact: %+v", p.Current.Artifacts)
4732 }
4733 if _, ok := p.Current.Artifacts["linux-amd64"]; ok {
4734 t.Fatal("mirrored a platform no box reports")
4735 }
4736 body, _, err := fx.rel.Get(context.Background(), a.Key)
4737 if err != nil || !bytes.Equal(body, arm) {
4738 t.Fatalf("artifact not mirrored: %v", err)
4739 }
4740 s := sha256.Sum256(arm)
4741 if a.SHA256 != hex.EncodeToString(s[:]) {
4742 t.Fatalf("sha = %s", a.SHA256)
4743 }
4744 after, _, _ := release.ReadSerial(context.Background(), fx.rel)
4745 if after.Serial == before.Serial {
4746 t.Fatal("serial not bumped")
4747 }
4748 // A second promote of the same version to stable copies nothing and
4749 // refuses different bytes under the same version.
4750 st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "stable", "version": "v0.18.0"}, owner)
4751 if st != http.StatusOK {
4752 t.Fatalf("second update = %d %v", st, out)
4753 }
4754 if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" {
4755 t.Fatalf("second job: %v", job)
4756 }
4757 if _, err := fx.rel.Put(context.Background(), a.Key, []byte("different"), ""); err == nil {
4758 // Overwrote the object out from under the pointer; a re-promote must refuse.
4759 st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "version": "v0.18.0"}, owner)
4760 job := waitJob(t, fx, owner, str(t, out, "job"))
4761 if job["error"] == "" || !strings.Contains(job["error"].(string), "immutable") {
4762 t.Fatalf("re-promote over different bytes: %v", job)
4763 }
4764 }
4765}
4766
4767func TestSystemUpdateFromAnUpload(t *testing.T) {
4768 fx := newConsoleFixture(t)
4769 owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-upload")
4770 var buf bytes.Buffer
4771 mw := multipart.NewWriter(&buf)
4772 meta, _ := mw.CreateFormField("meta")
4773 meta.Write([]byte(`{"version":"66587fa","platform":"linux-arm64"}`))
4774 file, _ := mw.CreateFormFile("file", "carlos-linux-arm64")
4775 file.Write([]byte("ELF local"))
4776 mw.Close()
4777 req, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf)
4778 req.Header.Set("Content-Type", mw.FormDataContentType())
4779 req.Header.Set("Authorization", "Bearer "+owner)
4780 res, err := http.DefaultClient.Do(req)
4781 if err != nil {
4782 t.Fatal(err)
4783 }
4784 var up map[string]any
4785 if res.StatusCode != http.StatusOK {
4786 t.Fatalf("upload = %d", res.StatusCode)
4787 }
4788 if err := json.NewDecoder(res.Body).Decode(&up); err != nil {
4789 t.Fatal(err)
4790 }
4791 st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "upload": up["upload"]}, owner)
4792 if st != http.StatusOK {
4793 t.Fatalf("update = %d %v", st, out)
4794 }
4795 if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" {
4796 t.Fatalf("job: %v", job)
4797 }
4798 p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary")
4799 if err != nil || p.Current.Version != "66587fa" || p.Current.Source != "file" {
4800 t.Fatalf("pointer = %+v %v", p, err)
4801 }
4802 // Upload without a platform is refused at the door.
4803 var buf2 bytes.Buffer
4804 mw2 := multipart.NewWriter(&buf2)
4805 m2, _ := mw2.CreateFormField("meta")
4806 m2.Write([]byte(`{"version":"abc"}`))
4807 f2, _ := mw2.CreateFormFile("file", "x")
4808 f2.Write([]byte("x"))
4809 mw2.Close()
4810 req2, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf2)
4811 req2.Header.Set("Content-Type", mw2.FormDataContentType())
4812 req2.Header.Set("Authorization", "Bearer "+owner)
4813 res2, _ := http.DefaultClient.Do(req2)
4814 if res2.StatusCode != http.StatusBadRequest {
4815 t.Fatalf("upload without platform = %d", res2.StatusCode)
4816 }
4817}
4818
4819func TestSystemStatusAndRollback(t *testing.T) {
4820 fx := newConsoleFixture(t)
4821 owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-status")
4822 seedBox(t, fx, "ie", "linux-arm64", "canary", "v1")
4823 sg := release.RootSigner(fx.priv)
4824 rel := func(v string) release.SystemRelease {
4825 return release.SystemRelease{Version: v, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("a", 64)}}}
4826 }
4827 release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v1"), release.SystemPromoteOptions{})
4828 time.Sleep(1100 * time.Millisecond)
4829 release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v2"), release.SystemPromoteOptions{})
4830 st, out := getJSON(t, fx.ts.URL+"/api/cli/system/status", owner)
4831 if st != http.StatusOK {
4832 t.Fatalf("status = %d %v", st, out)
4833 }
4834 boxes := out["boxes"].([]any)
4835 if len(boxes) != 1 {
4836 t.Fatalf("boxes = %v", boxes)
4837 }
4838 b := boxes[0].(map[string]any)
4839 if b["label"] != "ie" || b["release"] != "v1" || b["target"] != "v2" || b["channel"] != "canary" {
4840 t.Fatalf("box row = %v", b)
4841 }
4842 time.Sleep(1100 * time.Millisecond)
4843 st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner)
4844 if st != http.StatusOK || str(t, out, "version") != "v1" {
4845 t.Fatalf("rollback = %d %v", st, out)
4846 }
4847 if st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner); st != http.StatusConflict {
4848 t.Fatalf("second rollback = %d %v (nothing left in history)", st, out)
4849 }
4850}
4851```
4852(add `"encoding/json"` to the imports.)
4853
4854- [ ] **Step 2: Run to verify failure**
4855
4856Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc`
4857Expected: build failure (`SystemFeed` undefined) or 404s.
4858
4859- [ ] **Step 3: Implement**
4860
4861`server.go` `Config` gains, next to `Releases`:
4862```go
4863 // SystemFeed is where `carlos system update` fetches a public release
4864 // from. Zero value = the vendor feed; tests point it at a stub.
4865 SystemFeed publicrelease.Feed
4866```
4867and `Server` gains `systemMu sync.Mutex; systemJobs map[string]*systemJob; systemUploads map[string]systemUpload`.
4868
4869`authcli.go` `registerCLIRoutes()`:
4870```go
4871 // carlos system (spec §5): ops-account owners, person sessions only.
4872 s.cliRoute("GET /api/cli/system/status", svcrole.Set{}, s.handleAPISystemStatus)
4873 s.cliRoute("POST /api/cli/system/update", svcrole.Set{}, s.handleAPISystemUpdate)
4874 s.cliRoute("GET /api/cli/system/jobs/{id}", svcrole.Set{}, s.handleAPISystemJob)
4875 s.cliRoute("PUT /api/cli/system/upload", svcrole.Set{}, s.handleAPISystemUpload)
4876 s.cliRoute("POST /api/cli/system/rollback", svcrole.Set{}, s.handleAPISystemRollback)
4877```
4878
4879`apisystem.go`:
4880```go
4881package console
4882
4883import (
4884 "context"
4885 "crypto/rand"
4886 "crypto/sha256"
4887 "encoding/hex"
4888 "encoding/json"
4889 "errors"
4890 "fmt"
4891 "io"
4892 "log"
4893 "net/http"
4894 "sort"
4895 "strings"
4896 "sync"
4897 "time"
4898
4899 "github.com/carlosframework/platform/internal/publicrelease"
4900 "github.com/carlosframework/platform/internal/release"
4901 "github.com/carlosframework/platform/internal/status"
4902)
4903
4904// The console half of `carlos system` (spec §5). Every door here is for an
4905// OWNER of the ops account on a PERSON session, checked on every call;
4906// refusal is a bare 404 like the operator doors, so a non-owner learns
4907// nothing. The console holds the bucket, the signing key and the egress;
4908// the operator's laptop holds none of them.
4909
4910const (
4911 systemUploadMaxBytes = int64(128 << 20)
4912 systemAssetMaxBytes = int64(256 << 20)
4913 systemJobTTL = time.Hour
4914)
4915
4916var systemPlatformRe = targetRe // apicli.go: ^[a-z0-9]+-[a-z0-9]+$
4917
4918type systemJob struct {
4919 mu sync.Mutex
4920 Steps []string
4921 Done bool
4922 Err string
4923 at time.Time
4924}
4925
4926func (j *systemJob) step(format string, args ...any) {
4927 j.mu.Lock()
4928 j.Steps = append(j.Steps, fmt.Sprintf(format, args...))
4929 j.mu.Unlock()
4930}
4931
4932func (j *systemJob) finish(err error) {
4933 j.mu.Lock()
4934 j.Done = true
4935 if err != nil {
4936 j.Err = err.Error()
4937 }
4938 j.mu.Unlock()
4939}
4940
4941type systemUpload struct {
4942 version, platform string
4943 data []byte
4944 at time.Time
4945}
4946
4947func (s *Server) apiSystemOwner(w http.ResponseWriter, r *http.Request) (string, bool) {
4948 p, ok := principalFrom(r.Context())
4949 if !ok || p.Kind != PrincipalUser {
4950 http.NotFound(w, r)
4951 return "", false
4952 }
4953 role, err := s.store.Role(OpsAccountID, p.Address)
4954 if err != nil {
4955 log.Printf("console: api system: role: %v", err)
4956 http.NotFound(w, r)
4957 return "", false
4958 }
4959 if role != RoleOwner {
4960 http.NotFound(w, r)
4961 return "", false
4962 }
4963 return p.Address, true
4964}
4965
4966func (s *Server) systemFeed() publicrelease.Feed {
4967 if s.cfg.SystemFeed.DownloadBase == "" {
4968 return publicrelease.Default()
4969 }
4970 return s.cfg.SystemFeed
4971}
4972
4973// systemBoxes reads every heartbeat: the platform prefix and each fleet's.
4974func (s *Server) systemBoxes(ctx context.Context) ([]status.Box, error) {
4975 boxes, err := status.LoadBoxes(ctx, s.cfg.Releases)
4976 if err != nil {
4977 return nil, err
4978 }
4979 reg, err := s.openPassRegistry()
4980 if err != nil {
4981 return boxes, nil // no registry: platform boxes only
4982 }
4983 fleets, err := reg.ListFleets()
4984 if err != nil {
4985 return boxes, nil
4986 }
4987 for _, f := range fleets {
4988 fb, err := status.LoadBoxes(ctx, release.NewPrefixStore(s.cfg.Releases, "fleets/"+f.Name+"/"))
4989 if err != nil {
4990 log.Printf("console: api system: fleet %s boxes: %v", f.Name, err)
4991 continue
4992 }
4993 boxes = append(boxes, fb...)
4994 }
4995 return boxes, nil
4996}
4997
4998type systemUpdateRequest struct {
4999 Component string `json:"component"`
5000 Channel string `json:"channel"`
5001 Version string `json:"version"`
5002 Upload string `json:"upload"`
5003 Spread string `json:"spread"`
5004}
5005
5006func (s *Server) handleAPISystemUpdate(w http.ResponseWriter, r *http.Request) {
5007 addr, ok := s.apiSystemOwner(w, r)
5008 if !ok {
5009 return
5010 }
5011 if s.cfg.Key == nil {
5012 jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only")
5013 return
5014 }
5015 var in systemUpdateRequest
5016 if !readJSONStrictLimit(w, r, &in, 4<<10) {
5017 return
5018 }
5019 if in.Component == "" {
5020 in.Component = release.SystemComponentRouter
5021 }
5022 if in.Component != release.SystemComponentRouter {
5023 jsonError(w, http.StatusBadRequest, "unknown component "+in.Component)
5024 return
5025 }
5026 if in.Channel != release.RungCanary && in.Channel != release.RungStable {
5027 jsonError(w, http.StatusBadRequest, "channel must be canary or stable")
5028 return
5029 }
5030 spread := 5 * time.Minute
5031 if in.Spread != "" {
5032 d, err := time.ParseDuration(in.Spread)
5033 if err != nil || d < 0 || d > 24*time.Hour {
5034 jsonError(w, http.StatusBadRequest, "spread must be a duration between 0 and 24h")
5035 return
5036 }
5037 spread = d
5038 }
5039 id := s.newSystemJob()
5040 job := s.systemJobs[id]
5041 go s.runSystemUpdate(context.WithoutCancel(r.Context()), job, addr, in, spread)
5042 writeJSON(w, http.StatusOK, map[string]string{"job": id})
5043}
5044
5045func (s *Server) newSystemJob() string {
5046 b := make([]byte, 8)
5047 rand.Read(b)
5048 id := hex.EncodeToString(b)
5049 s.systemMu.Lock()
5050 defer s.systemMu.Unlock()
5051 if s.systemJobs == nil {
5052 s.systemJobs = map[string]*systemJob{}
5053 }
5054 for k, j := range s.systemJobs {
5055 if time.Since(j.at) > systemJobTTL {
5056 delete(s.systemJobs, k)
5057 }
5058 }
5059 s.systemJobs[id] = &systemJob{at: time.Now()}
5060 return id
5061}
5062
5063// runSystemUpdate is the promote: resolve, ingest (or take the upload),
5064// mirror, sign, bump. Every step is idempotent, so a lost job is re-run.
5065func (s *Server) runSystemUpdate(ctx context.Context, job *systemJob, actor string, in systemUpdateRequest, spread time.Duration) {
5066 err := func() error {
5067 var rel release.SystemRelease
5068 if in.Upload != "" {
5069 s.systemMu.Lock()
5070 up, ok := s.systemUploads[in.Upload]
5071 delete(s.systemUploads, in.Upload)
5072 s.systemMu.Unlock()
5073 if !ok {
5074 return errors.New("upload handle unknown or already consumed")
5075 }
5076 if in.Version != "" && in.Version != up.version {
5077 return fmt.Errorf("upload is %s, request says %s", up.version, in.Version)
5078 }
5079 job.step("using uploaded %s for %s", up.version, up.platform)
5080 sum := sha256.Sum256(up.data)
5081 rel = release.SystemRelease{Version: up.version, Source: "file", Artifacts: map[string]release.SystemArtifact{}}
5082 if err := s.mirrorSystemArtifact(ctx, job, up.version, up.platform, up.data, hex.EncodeToString(sum[:])); err != nil {
5083 return err
5084 }
5085 rel.Artifacts[up.platform] = release.SystemArtifact{Key: release.SystemArtifactKey(up.version, up.platform), SHA256: hex.EncodeToString(sum[:])}
5086 } else {
5087 feed := s.systemFeed()
5088 tag := in.Version
5089 if tag == "" {
5090 var err error
5091 if tag, err = feed.Latest(ctx); err != nil {
5092 return err
5093 }
5094 }
5095 if !publicrelease.TagRe.MatchString(tag) {
5096 return fmt.Errorf("%q is not a release tag; use --file for a local build", tag)
5097 }
5098 job.step("resolved %s", tag)
5099 platforms, err := s.systemPlatforms(ctx)
5100 if err != nil {
5101 return err
5102 }
5103 rel = release.SystemRelease{Version: tag, Source: "release:" + tag, Artifacts: map[string]release.SystemArtifact{}}
5104 for _, plat := range platforms {
5105 data, sha, err := feed.Asset(ctx, tag, publicrelease.AssetName(plat), systemAssetMaxBytes)
5106 if err != nil {
5107 return fmt.Errorf("%s: %w", plat, err)
5108 }
5109 job.step("verified %s (%d bytes)", publicrelease.AssetName(plat), len(data))
5110 if err := s.mirrorSystemArtifact(ctx, job, tag, plat, data, sha); err != nil {
5111 return err
5112 }
5113 rel.Artifacts[plat] = release.SystemArtifact{Key: release.SystemArtifactKey(tag, plat), SHA256: sha}
5114 }
5115 }
5116 p, err := release.PromoteSystem(ctx, s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, rel,
5117 release.SystemPromoteOptions{Actor: actor, Spread: spread})
5118 if err != nil {
5119 return err
5120 }
5121 job.step("pointed %s/%s at %s (spread %s)", in.Component, in.Channel, p.Current.Version, p.Spread)
5122 s.audit(nil, "platform", actor, "system-update", "pointed "+in.Component+"/"+in.Channel+" at "+p.Current.Version)
5123 s.bumpSerial(ctx)
5124 job.step("serial bumped: boxes on %s start their spread", in.Channel)
5125 return nil
5126 }()
5127 job.finish(err)
5128}
5129
5130// systemPlatforms: the distinct platforms the fleet reports; linux-arm64
5131// when nothing has reported the field yet.
5132func (s *Server) systemPlatforms(ctx context.Context) ([]string, error) {
5133 boxes, err := s.systemBoxes(ctx)
5134 if err != nil {
5135 return nil, err
5136 }
5137 set := map[string]bool{}
5138 for _, b := range boxes {
5139 if b.Metrics != nil && b.Metrics.Platform != "" {
5140 set[b.Metrics.Platform] = true
5141 }
5142 }
5143 if len(set) == 0 {
5144 set["linux-arm64"] = true
5145 }
5146 var out []string
5147 for p := range set {
5148 out = append(out, p)
5149 }
5150 sort.Strings(out)
5151 return out, nil
5152}
5153
5154// mirrorSystemArtifact: present with the same sha → skip; present with a
5155// different sha → refuse (versions are immutable, spec §3).
5156func (s *Server) mirrorSystemArtifact(ctx context.Context, job *systemJob, version, platform string, data []byte, sha string) error {
5157 key := release.SystemArtifactKey(version, platform)
5158 existing, etag, err := s.cfg.Releases.Get(ctx, key)
5159 switch {
5160 case err == nil:
5161 sum := sha256.Sum256(existing)
5162 if hex.EncodeToString(sum[:]) == sha {
5163 job.step("%s already mirrored", key)
5164 return nil
5165 }
5166 return fmt.Errorf("%s exists with different bytes — versions are immutable; pick a new version", key)
5167 case errors.Is(err, os.ErrNotExist):
5168 etag = ""
5169 default:
5170 return err
5171 }
5172 if _, err := s.cfg.Releases.Put(ctx, key, data, etag); err != nil {
5173 return err
5174 }
5175 job.step("mirrored %s", key)
5176 return nil
5177}
5178
5179func (s *Server) handleAPISystemJob(w http.ResponseWriter, r *http.Request) {
5180 if _, ok := s.apiSystemOwner(w, r); !ok {
5181 return
5182 }
5183 s.systemMu.Lock()
5184 job, ok := s.systemJobs[r.PathValue("id")]
5185 s.systemMu.Unlock()
5186 if !ok {
5187 jsonError(w, http.StatusNotFound, "no such job — the console may have restarted; re-run, every step is idempotent")
5188 return
5189 }
5190 job.mu.Lock()
5191 defer job.mu.Unlock()
5192 writeJSON(w, http.StatusOK, map[string]any{"steps": append([]string(nil), job.Steps...), "done": job.Done, "error": job.Err})
5193}
5194
5195func (s *Server) handleAPISystemUpload(w http.ResponseWriter, r *http.Request) {
5196 if _, ok := s.apiSystemOwner(w, r); !ok {
5197 return
5198 }
5199 r.Body = http.MaxBytesReader(w, r.Body, systemUploadMaxBytes+64<<10)
5200 mr, err := r.MultipartReader()
5201 if err != nil {
5202 jsonError(w, http.StatusBadRequest, "multipart body required")
5203 return
5204 }
5205 var meta struct {
5206 Version string `json:"version"`
5207 Platform string `json:"platform"`
5208 }
5209 first, err := mr.NextPart()
5210 if err != nil || first.FormName() != "meta" {
5211 jsonError(w, http.StatusBadRequest, "first part must be meta")
5212 return
5213 }
5214 if err := json.NewDecoder(io.LimitReader(first, 4<<10)).Decode(&meta); err != nil {
5215 jsonError(w, http.StatusBadRequest, "meta: "+err.Error())
5216 return
5217 }
5218 if !release.ValidVersion(meta.Version) || meta.Version == "dev" {
5219 jsonError(w, http.StatusBadRequest, "meta.version is required and must be a version")
5220 return
5221 }
5222 if !systemPlatformRe.MatchString(meta.Platform) {
5223 jsonError(w, http.StatusBadRequest, "meta.platform is required (e.g. linux-arm64): the console cannot run a foreign binary to ask it")
5224 return
5225 }
5226 part, err := mr.NextPart()
5227 if err != nil || part.FormName() != "file" {
5228 jsonError(w, http.StatusBadRequest, "second part must be file")
5229 return
5230 }
5231 data, err := io.ReadAll(io.LimitReader(part, systemUploadMaxBytes+1))
5232 if err != nil {
5233 jsonError(w, http.StatusBadRequest, "file: "+err.Error())
5234 return
5235 }
5236 if int64(len(data)) > systemUploadMaxBytes {
5237 jsonError(w, http.StatusRequestEntityTooLarge, "file exceeds 128 MiB")
5238 return
5239 }
5240 b := make([]byte, 8)
5241 rand.Read(b)
5242 handle := hex.EncodeToString(b)
5243 s.systemMu.Lock()
5244 if s.systemUploads == nil {
5245 s.systemUploads = map[string]systemUpload{}
5246 }
5247 for k := range s.systemUploads { // one slot: a forgotten upload is not a memory leak
5248 delete(s.systemUploads, k)
5249 }
5250 s.systemUploads[handle] = systemUpload{version: meta.Version, platform: meta.Platform, data: data, at: time.Now()}
5251 s.systemMu.Unlock()
5252 writeJSON(w, http.StatusOK, map[string]any{"upload": handle, "size": len(data)})
5253}
5254
5255func (s *Server) handleAPISystemRollback(w http.ResponseWriter, r *http.Request) {
5256 addr, ok := s.apiSystemOwner(w, r)
5257 if !ok {
5258 return
5259 }
5260 if s.cfg.Key == nil {
5261 jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only")
5262 return
5263 }
5264 var in struct {
5265 Component string `json:"component"`
5266 Channel string `json:"channel"`
5267 }
5268 if !readJSONStrictLimit(w, r, &in, 1<<10) {
5269 return
5270 }
5271 if in.Component == "" {
5272 in.Component = release.SystemComponentRouter
5273 }
5274 p, err := release.RollbackSystem(r.Context(), s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, addr)
5275 switch {
5276 case errors.Is(err, release.ErrNoSystemHistory):
5277 jsonError(w, http.StatusConflict, err.Error())
5278 return
5279 case errors.Is(err, os.ErrNotExist):
5280 jsonError(w, http.StatusNotFound, "no "+in.Channel+" pointer to roll back")
5281 return
5282 case err != nil:
5283 jsonError(w, http.StatusInternalServerError, err.Error())
5284 return
5285 }
5286 s.audit(nil, "platform", addr, "system-rollback", "pointed "+in.Component+"/"+in.Channel+" back at "+p.Current.Version)
5287 s.bumpSerial(r.Context())
5288 writeJSON(w, http.StatusOK, map[string]string{"channel": in.Channel, "version": p.Current.Version})
5289}
5290
5291func (s *Server) handleAPISystemStatus(w http.ResponseWriter, r *http.Request) {
5292 if _, ok := s.apiSystemOwner(w, r); !ok {
5293 return
5294 }
5295 ctx := r.Context()
5296 pointers := map[string]any{}
5297 targets := map[string]string{}
5298 for _, ch := range []string{release.RungCanary, release.RungStable} {
5299 p, _, err := release.GetSystemPointer(ctx, s.cfg.Releases, s.cfg.ReleasePub, release.SystemComponentRouter, ch)
5300 if err != nil {
5301 pointers[ch] = nil
5302 continue
5303 }
5304 pointers[ch] = map[string]any{"version": p.Current.Version, "promoted_at": p.PromotedAt, "spread": p.Spread, "source": p.Current.Source,
5305 "history": len(p.History)}
5306 targets[ch] = p.Current.Version
5307 }
5308 boxes, err := s.systemBoxes(ctx)
5309 if err != nil {
5310 jsonError(w, http.StatusBadGateway, "box heartbeats unreadable: "+err.Error())
5311 return
5312 }
5313 now := time.Now()
5314 rows := make([]map[string]any, 0, len(boxes))
5315 for _, b := range boxes {
5316 row := map[string]any{"label": b.Label, "region": b.Region, "stale": status.BoxStale(b, now, status.SnapshotPeriod),
5317 "platform": "", "channel": "", "release": "", "target": "", "system": nil}
5318 if b.Metrics != nil {
5319 ch := b.Metrics.SystemChannel
5320 if ch == "" {
5321 ch = "stable"
5322 }
5323 row["platform"], row["channel"], row["release"] = b.Metrics.Platform, ch, b.Metrics.Release
5324 row["target"] = targets[ch]
5325 if b.Metrics.System != nil {
5326 row["system"] = b.Metrics.System
5327 }
5328 }
5329 rows = append(rows, row)
5330 }
5331 sort.Slice(rows, func(i, j int) bool { return rows[i]["label"].(string) < rows[j]["label"].(string) })
5332 writeJSON(w, http.StatusOK, map[string]any{"pointers": pointers, "boxes": rows})
5333}
5334```
5335Check three things against the code while writing: `s.audit`'s signature (`apicli.go:855` passes `(acct, app, addr, action, detail)`; if `acct` cannot be nil, use the ops account fetched by id, or add a `s.auditSystem(addr, action, detail)` writing an event row against `OpsAccountID`); `status.BoxStale`'s signature (`fleet.go:137`); `os` import. `readJSONStrictLimit` is at `authcli.go:849`.
5336
5337- [ ] **Step 4: Run the console tests**
5338
5339Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc`
5340Expected: `ok`. `TestEveryCLIDoorNamesItsRoles` and `TestEveryClosedDoorRefusesAServiceCredential` cover the five new doors automatically; if either fails, the route line is wrong, not the test.
5341
5342- [ ] **Step 5: Commit**
5343
5344```bash
5345gofmt -l internal/console/ && git add internal/console/apisystem.go internal/console/apisystem_test.go internal/console/authcli.go internal/console/server.go && git commit -q -F - <<'EOF'
5346console: the carlos system doors — status, update job, upload, rollback
5347
5348Ops-account owners on a person session, bare 404 otherwise. The
5349update is a job: resolve, ingest from the signed public feed (or an
5350upload), mirror per reported platform, sign the pointer, bump the
5351serial. Versions are immutable in the bucket.
5352
5353Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5354Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
5355EOF
5356```
5357
5358---
5359### Task 12: `carlos system update | status | rollback` (API mode)
5360
5361**Files:**
5362- Modify: `system.go` (replace the three placeholders), `system_test.go`
5363
5364**Interfaces:**
5365- Consumes: `requireDeployAPISession(consoleFlag) *apiSession` (`deploy.go:1151`); `(*apiSession).getJSON/postJSON` (`apimode.go:737,799`); `(*consoleClient).doMultipart` (`auth.go:485`) via `sess.c`; `writeFilePart(mw, name, path)` (`apimode.go:3009`); `interactiveTerminal`, `promptYes` (`account_choose.go`); `stdout`, `colorFor`, `mutedf` (`color.go`); `flagUsage`.
5366- Produces:
5367 ```go
5368 func runSystemUpdate(args []string) // --console --channel --version --file --platform --spread --yes
5369 func runSystemStatus(args []string) // --console --channel
5370 func runSystemRollback(args []string) // --console --channel --yes
5371 func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string)
5372 type systemStatusResponse struct{ Pointers map[string]*systemPointerRow; Boxes []systemBoxRow }
5373 ```
5374
5375- [ ] **Step 1: Write the failing test** (the pure renderer; the doors are covered in Task 11)
5376
5377```go
5378func TestPrintSystemStatusSaysWhatEachBoxIsDoing(t *testing.T) {
5379 resp := systemStatusResponse{
5380 Pointers: map[string]*systemPointerRow{"canary": {Version: "v2", Spread: "5m0s"}, "stable": nil},
5381 Boxes: []systemBoxRow{
5382 {Label: "ie", Platform: "linux-arm64", Channel: "canary", Release: "v2", Target: "v2"},
5383 {Label: "us", Platform: "linux-arm64", Channel: "canary", Release: "v1", Target: "v2",
5384 System: &systemStateRow{Version: "v2", Outcome: "rolled-back", Reason: "gate: carlos-edge is failed"}},
5385 {Label: "au", Platform: "", Channel: "stable", Release: "v1", Target: "", Stale: true},
5386 },
5387 }
5388 var b strings.Builder
5389 printSystemStatus(&b, resp, "")
5390 out := b.String()
5391 for _, want := range []string{"canary v2", "stable (no pointer)", "ie ", "current", "us ", "rolled-back", "gate: carlos-edge is failed", "au ", "UNREACHABLE", "pre-upgrade"} {
5392 if !strings.Contains(out, want) {
5393 t.Errorf("status output lacks %q:\n%s", want, out)
5394 }
5395 }
5396 var only strings.Builder
5397 printSystemStatus(&only, resp, "stable")
5398 if strings.Contains(only.String(), "ie ") {
5399 t.Error("--channel stable still listed a canary box")
5400 }
5401}
5402```
5403
5404- [ ] **Step 2: Run to verify failure**
5405
5406Run: `export TMPDIR=/tmp/c; out=$(go test . -run TestPrintSystemStatus -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc`
5407Expected: undefined types.
5408
5409- [ ] **Step 3: Implement**
5410
5411Replace the placeholders in `system.go`:
5412```go
5413type systemPointerRow struct {
5414 Version string `json:"version"`
5415 PromotedAt time.Time `json:"promoted_at"`
5416 Spread string `json:"spread"`
5417 Source string `json:"source"`
5418 History int `json:"history"`
5419}
5420
5421type systemStateRow struct {
5422 Channel string `json:"channel"`
5423 Version string `json:"version"`
5424 PromotedAt time.Time `json:"promoted_at"`
5425 Outcome string `json:"outcome"`
5426 Reason string `json:"reason"`
5427 At time.Time `json:"at"`
5428}
5429
5430type systemBoxRow struct {
5431 Label string `json:"label"`
5432 Region string `json:"region"`
5433 Platform string `json:"platform"`
5434 Channel string `json:"channel"`
5435 Release string `json:"release"`
5436 Target string `json:"target"`
5437 System *systemStateRow `json:"system"`
5438 Stale bool `json:"stale"`
5439}
5440
5441type systemStatusResponse struct {
5442 Pointers map[string]*systemPointerRow `json:"pointers"`
5443 Boxes []systemBoxRow `json:"boxes"`
5444}
5445
5446func runSystemUpdate(args []string) {
5447 fs := flag.NewFlagSet("system update", flag.ExitOnError)
5448 consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
5449 channel := fs.String("channel", "stable", "canary or stable")
5450 version := fs.String("version", "", "release tag to promote (default: the latest public release); with --file, the version the binary reports")
5451 file := fs.String("file", "", "a locally built binary instead of a public release (dev and pre-release)")
5452 platform := fs.String("platform", "linux-arm64", "with --file: the binary's <goos>-<goarch>")
5453 spread := fs.String("spread", "5m", "boxes jitter their restarts over this; 0 for a hotfix")
5454 yes := fs.Bool("yes", false, "do not ask")
5455 fs.Usage = flagUsage(fs, "")
5456 fs.Parse(args)
5457 if *file != "" && *version == "" {
5458 log.Fatal("--file needs --version: the console cannot run a foreign-arch binary to ask it")
5459 }
5460 sess := requireDeployAPISession(*consoleFlag)
5461 ctx := context.Background()
5462
5463 // What this reaches, before it is done.
5464 st, body, err := sess.getJSON(ctx, "/api/cli/system/status")
5465 if err != nil || st != http.StatusOK {
5466 log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err))
5467 }
5468 var cur systemStatusResponse
5469 _ = json.Unmarshal(body, &cur)
5470 var reach []string
5471 for _, b := range cur.Boxes {
5472 if b.Channel == *channel {
5473 reach = append(reach, fmt.Sprintf("%s (%s, running %s)", b.Label, b.Platform, orDash(b.Release)))
5474 }
5475 }
5476 what := "the latest public release"
5477 if *version != "" {
5478 what = *version
5479 }
5480 fmt.Fprintf(stdout, "promote %s to %s/%s — %d box(es) will roll over %s:\n", what, systemComponentRouter, *channel, len(reach), *spread)
5481 for _, r := range reach {
5482 fmt.Fprintf(stdout, " %s\n", r)
5483 }
5484 if len(reach) == 0 {
5485 mutedf(stdout, " (no box reports channel %s yet; boxes on an older build count as stable)\n", *channel)
5486 }
5487 if !*yes {
5488 if !interactiveTerminal() {
5489 log.Fatal("no terminal to confirm on; pass --yes")
5490 }
5491 if !promptYes(stdout, "go ahead? [y/N] ") {
5492 return
5493 }
5494 }
5495
5496 in := map[string]any{"channel": *channel, "spread": *spread}
5497 if *version != "" {
5498 in["version"] = *version
5499 }
5500 if *file != "" {
5501 handle, err := sess.uploadSystemBinary(ctx, *file, *version, *platform)
5502 if err != nil {
5503 log.Fatalf("upload: %v", err)
5504 }
5505 in["upload"] = handle
5506 }
5507 st, body, err = sess.postJSON(ctx, "/api/cli/system/update", in)
5508 if err != nil || st != http.StatusOK {
5509 log.Fatalf("carlos system update: %s", apiErrorFromBody(st, body, err))
5510 }
5511 var started struct{ Job string `json:"job"` }
5512 _ = json.Unmarshal(body, &started)
5513 if err := sess.watchSystemJob(ctx, started.Job, stdout); err != nil {
5514 log.Fatal(err)
5515 }
5516 fmt.Fprintf(stdout, "%s boxes on %s roll themselves; watch with: carlos system status --channel %s\n", colorFor(stdout).ok("promoted"), *channel, *channel)
5517}
5518
5519// watchSystemJob polls the job and prints each step once, as deploy's
5520// watchVersion does for a route.
5521func (s *apiSession) watchSystemJob(ctx context.Context, id string, w io.Writer) error {
5522 printed := 0
5523 deadline := time.Now().Add(15 * time.Minute)
5524 for {
5525 st, body, err := s.getJSON(ctx, "/api/cli/system/jobs/"+id)
5526 if err != nil || st != http.StatusOK {
5527 return fmt.Errorf("job %s: %s", id, apiErrorFromBody(st, body, err))
5528 }
5529 var j struct {
5530 Steps []string `json:"steps"`
5531 Done bool `json:"done"`
5532 Error string `json:"error"`
5533 }
5534 _ = json.Unmarshal(body, &j)
5535 for ; printed < len(j.Steps); printed++ {
5536 fmt.Fprintf(w, " %s\n", j.Steps[printed])
5537 }
5538 if j.Done {
5539 if j.Error != "" {
5540 return fmt.Errorf("carlos system update: %s", j.Error)
5541 }
5542 return nil
5543 }
5544 if time.Now().After(deadline) {
5545 return errors.New("carlos system update: the console's job did not finish in 15 minutes")
5546 }
5547 time.Sleep(time.Second)
5548 }
5549}
5550
5551func (s *apiSession) uploadSystemBinary(ctx context.Context, path, version, platform string) (string, error) {
5552 st, body, err := s.c.doMultipart(ctx, "/api/cli/system/upload", func(mw *multipart.Writer) error {
5553 meta, err := mw.CreateFormField("meta")
5554 if err != nil {
5555 return err
5556 }
5557 if err := json.NewEncoder(meta).Encode(map[string]string{"version": version, "platform": platform}); err != nil {
5558 return err
5559 }
5560 return writeFilePart(mw, "file", path)
5561 })
5562 if err != nil || st != http.StatusOK {
5563 return "", fmt.Errorf("%s", apiErrorFromBody(st, body, err))
5564 }
5565 var out struct{ Upload string `json:"upload"` }
5566 _ = json.Unmarshal(body, &out)
5567 return out.Upload, nil
5568}
5569
5570func runSystemStatus(args []string) {
5571 fs := flag.NewFlagSet("system status", flag.ExitOnError)
5572 consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
5573 channel := fs.String("channel", "", "only boxes on this channel")
5574 fs.Usage = flagUsage(fs, "")
5575 fs.Parse(args)
5576 sess := requireDeployAPISession(*consoleFlag)
5577 st, body, err := sess.getJSON(context.Background(), "/api/cli/system/status")
5578 if err != nil || st != http.StatusOK {
5579 log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err))
5580 }
5581 var resp systemStatusResponse
5582 if err := json.Unmarshal(body, &resp); err != nil {
5583 log.Fatalf("carlos system status: %v", err)
5584 }
5585 printSystemStatus(stdout, resp, *channel)
5586}
5587
5588func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string) {
5589 for _, ch := range []string{"canary", "stable"} {
5590 if channel != "" && ch != channel {
5591 continue
5592 }
5593 p := resp.Pointers[ch]
5594 if p == nil {
5595 fmt.Fprintf(w, "%-7s (no pointer)\n", ch)
5596 continue
5597 }
5598 fmt.Fprintf(w, "%-7s %s promoted %s spread %s %s\n", ch, p.Version, p.PromotedAt.Format(time.RFC3339), p.Spread, p.Source)
5599 }
5600 fmt.Fprintln(w)
5601 c := colorFor(w)
5602 for _, b := range resp.Boxes {
5603 if channel != "" && b.Channel != channel {
5604 continue
5605 }
5606 state := ""
5607 switch {
5608 case b.Stale:
5609 state = c.bad("UNREACHABLE")
5610 case b.Platform == "":
5611 state = c.warn("pre-upgrade (no platform reported; counts as stable)")
5612 case b.Target == "":
5613 state = "no target"
5614 case b.Release == b.Target:
5615 state = c.ok("current")
5616 default:
5617 state = c.warn("behind → " + b.Target)
5618 }
5619 fmt.Fprintf(w, "%-20s %-12s %-7s %-10s %s", b.Label, orDash(b.Platform), b.Channel, orDash(b.Release), state)
5620 if b.System != nil && b.System.Outcome != "applied" && b.System.Outcome != "noted" {
5621 fmt.Fprintf(w, " %s %s: %s", c.bad(b.System.Outcome), b.System.Version, b.System.Reason)
5622 }
5623 fmt.Fprintln(w)
5624 }
5625}
5626
5627func runSystemRollback(args []string) {
5628 fs := flag.NewFlagSet("system rollback", flag.ExitOnError)
5629 consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
5630 channel := fs.String("channel", "", "canary or stable (required)")
5631 yes := fs.Bool("yes", false, "do not ask")
5632 fs.Usage = flagUsage(fs, "")
5633 fs.Parse(args)
5634 if *channel == "" {
5635 log.Fatal("--channel is required")
5636 }
5637 sess := requireDeployAPISession(*consoleFlag)
5638 if !*yes {
5639 if !interactiveTerminal() {
5640 log.Fatal("no terminal to confirm on; pass --yes")
5641 }
5642 if !promptYes(stdout, fmt.Sprintf("point %s back at the release before this one? every box on it rolls back [y/N] ", *channel)) {
5643 return
5644 }
5645 }
5646 st, body, err := sess.postJSON(context.Background(), "/api/cli/system/rollback", map[string]string{"channel": *channel})
5647 if err != nil || st != http.StatusOK {
5648 log.Fatalf("carlos system rollback: %s", apiErrorFromBody(st, body, err))
5649 }
5650 var out struct{ Version string `json:"version"` }
5651 _ = json.Unmarshal(body, &out)
5652 fmt.Fprintf(stdout, "%s %s → %s\n", colorFor(stdout).ok("rolled back"), *channel, out.Version)
5653}
5654```
5655Imports: `context`, `encoding/json`, `errors`, `flag`, `io`, `mime/multipart`, `net/http`, `time`. `orDash` exists in `cli.go:3327+`; `apiErrorFromBody` exists in `apimode.go` (confirm its signature with `grep -n 'func apiErrorFromBody' *.go`; adapt the three-argument call if it takes fewer). `sess.c` is the `*consoleClient` field on `apiSession` (confirm with `grep -n 'type apiSession struct' -A 6 apimode.go`).
5656
5657- [ ] **Step 4: Run the tests and try the binary against a fixture console**
5658
5659Run: `export TMPDIR=/tmp/c; out=$(go test . -run 'TestPrintSystemStatus|TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go vet . `
5660Expected: PASS, vet clean.
5661
5662- [ ] **Step 5: Commit**
5663
5664```bash
5665gofmt -l . && git add system.go system_test.go && git commit -q -F - <<'EOF'
5666cli: carlos system update, status, rollback through the console
5667
5668Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5669Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
5670EOF
5671```
5672
5673---
5674
5675### Task 13: The hourly belt — `carlos adopt` arms the self-update units
5676
5677**Files:**
5678- Modify: `internal/host/agent.go` `RunAdopt` (`:2811`; insert before `openServiceRegistry` at `:2825`)
5679- Create: `internal/host/systemunits.go`, `internal/host/systemunits_test.go`
5680
5681**Interfaces:**
5682- Consumes: `unitsFS`, `systemctlPlan`, `copyFileDurable`, a `sysOps`-shaped `systemctl` func.
5683- Produces:
5684 ```go
5685 // ensureSystemUnits installs and arms carlos-system-update.path/.service
5686 // and carlos-system-recover.service when any is missing from etcDir or
5687 // the path unit is inactive / the recover unit not enabled. Non-fatal.
5688 func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any))
5689 ```
5690
5691- [ ] **Step 1: Write the failing test**
5692
5693```go
5694package host
5695
5696import (
5697 "os"
5698 "path/filepath"
5699 "strings"
5700 "testing"
5701)
5702
5703func TestEnsureSystemUnitsInstallsAndArmsWhatIsMissing(t *testing.T) {
5704 etc := t.TempDir()
5705 os.MkdirAll(filepath.Join(etc, "systemd", "system"), 0o755)
5706 var calls []string
5707 systemctl := func(args ...string) (string, error) {
5708 calls = append(calls, strings.Join(args, " "))
5709 switch args[0] {
5710 case "is-active":
5711 return "inactive\n", nil
5712 case "is-enabled":
5713 return "disabled\n", nil
5714 }
5715 return "", nil
5716 }
5717 ensureSystemUnits(etc, systemctl, t.Logf)
5718 for _, u := range []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"} {
5719 if _, err := os.Stat(filepath.Join(etc, "systemd", "system", u)); err != nil {
5720 t.Errorf("%s not installed", u)
5721 }
5722 }
5723 joined := strings.Join(calls, "\n")
5724 for _, want := range []string{"daemon-reload", "enable carlos-system-update.path carlos-system-recover.service", "start carlos-system-update.path"} {
5725 if !strings.Contains(joined, want) {
5726 t.Errorf("calls lack %q:\n%s", want, joined)
5727 }
5728 }
5729 if strings.Contains(joined, "start carlos-system-recover.service") || strings.Contains(joined, "start carlos-system-update.service") {
5730 t.Fatal("a service was started")
5731 }
5732 // All present and armed: no writes, no systemctl beyond the two checks.
5733 calls = nil
5734 systemctl2 := func(args ...string) (string, error) {
5735 calls = append(calls, strings.Join(args, " "))
5736 switch args[0] {
5737 case "is-active":
5738 return "active\n", nil
5739 case "is-enabled":
5740 return "enabled\n", nil
5741 }
5742 return "", nil
5743 }
5744 ensureSystemUnits(etc, systemctl2, t.Logf)
5745 if len(calls) != 2 {
5746 t.Fatalf("an armed box got %v", calls)
5747 }
5748}
5749```
5750
5751- [ ] **Step 2: Run to verify failure**
5752
5753Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run TestEnsureSystemUnits -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc`
5754
5755- [ ] **Step 3: Implement**
5756
5757```go
5758package host
5759
5760import (
5761 "os"
5762 "path/filepath"
5763 "strings"
5764)
5765
5766var systemUnitFiles = []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"}
5767
5768// ensureSystemUnits is the second belt of spec §7: a box the roll script
5769// forgot to arm is armed within the hour by the root adopt pass. It
5770// installs the embedded units when missing and arms them through
5771// systemctlPlan (never starting a service). Non-fatal: it runs before
5772// RunAdopt's registry and pubkey gates, and logs instead of exiting.
5773func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any)) {
5774 unitDir := filepath.Join(etcDir, "systemd", "system")
5775 missing := false
5776 for _, u := range systemUnitFiles {
5777 if _, err := os.Stat(filepath.Join(unitDir, u)); err != nil {
5778 missing = true
5779 }
5780 }
5781 active, _ := systemctl("is-active", systemUpdatePathUnit)
5782 enabled, _ := systemctl("is-enabled", "carlos-system-recover.service")
5783 if !missing && strings.TrimSpace(active) == "active" && strings.TrimSpace(enabled) == "enabled" {
5784 return
5785 }
5786 for _, u := range systemUnitFiles {
5787 b, err := unitsFS.ReadFile("units/" + u)
5788 if err != nil {
5789 logf("adopt: system units: %v", err)
5790 return
5791 }
5792 if err := writeFileDurable(filepath.Join(unitDir, u), b, 0o644); err != nil {
5793 logf("adopt: system units: %v", err)
5794 return
5795 }
5796 }
5797 for _, args := range systemctlPlan([]string{systemUpdatePathUnit, "carlos-system-recover.service"}) {
5798 if _, err := systemctl(args...); err != nil {
5799 logf("adopt: system units: systemctl %s: %v", strings.Join(args, " "), err)
5800 return
5801 }
5802 }
5803 logf("adopt: self-update units installed and armed")
5804}
5805```
5806In `RunAdopt`, immediately after `fs.Parse(args)` and before `openServiceRegistry`:
5807```go
5808 // Spec §7 belt 2: arm the self-update relay if the last hand roll did
5809 // not. Before the fatal gates below, and never fatal itself.
5810 if !*restartOnly {
5811 ensureSystemUnits("/etc", productionSysOps().systemctl, log.Printf)
5812 }
5813```
5814
5815- [ ] **Step 4: Run**
5816
5817Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc`
5818
5819- [ ] **Step 5: Commit**
5820
5821```bash
5822gofmt -l internal/host/ && git add internal/host/systemunits.go internal/host/systemunits_test.go internal/host/agent.go && git commit -q -F - <<'EOF'
5823host: the hourly adopt pass arms the self-update units if the roll script did not
5824
5825Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5826Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
5827EOF
5828```
5829
5830---
5831
5832### Task 14: Docs, the env contract, and the last two script lines
5833
5834**Files:**
5835- Modify: `docs/site/cli.md` (after `### carlos status`, ~line 930), `AGENTS.md` (the section that names `carlos-stage.path` / `update-platform-binary.sh`, ~lines 246-277)
5836- Out of repo, hand off (do not edit from this checkout): `platform-infrastructure/scripts/update-platform-binary.sh` and the Tito copy in `amadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh`
5837
5838- [ ] **Step 1: `docs/site/cli.md`** — add under `## Your deployment`, after `### carlos status`:
5839
5840````markdown
5841### carlos system
5842
5843The platform's own release channel. `update` promotes a public release (or,
5844with `--file`, a binary you built) to `canary` or `stable`; every box on that
5845channel fetches it through its normal store, verifies it against the signed
5846pointer, installs it, restarts its edge, and rolls back to the previous
5847binary if the new one does not come up and stay up. `status` shows each box's
5848platform, channel, running build, the channel's target, and the last apply
5849outcome with its reason. `rollback` points a channel back at the release
5850before this one, and every box on it rolls back the same way it rolled
5851forward. All three talk to your console; you need to be an owner of the ops
5852account.
5853
5854```sh
5855carlos system update --channel canary
5856carlos system status --channel canary
5857carlos system update --channel stable --spread 10m
5858carlos system rollback --channel stable
5859```
5860
5861A box is put on canary with `CARLOS_SYSTEM_CHANNEL=canary` in its host.env.
5862Boxes that have not reported a platform yet count as stable. `carlos system
5863apply` is the root oneshot behind `carlos-system-update.service`; it is not
5864for operators. The console is an app — update it with `carlos deploy --app
5865console`.
5866````
5867
5868- [ ] **Step 2: `AGENTS.md`** — in the section describing the root relays, add after the `carlos-restart.path` paragraph:
5869
5870```markdown
5871- `carlos-system-update.path` relays the agent's platform self-update request
5872 to `carlos system apply` (root; spec 2026-09-06-carlos-system-update-design.md
5873 §6). `carlos-system-recover.service` runs once per boot, before the edge,
5874 and on the apply's failure. Both must be armed; the last run of
5875 `update-platform-binary.sh` does it, and the hourly `carlos adopt` pass
5876 installs and arms them if it did not. A box follows `CARLOS_SYSTEM_CHANNEL`
5877 (canary|stable, default stable) in host.env.
5878```
5879
5880- [ ] **Step 3: The pinfra hand-off** — write `docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.md` with exactly this content, for whoever edits the two scripts (they are in other repos; this checkout does not touch them):
5881
5882```markdown
5883# update-platform-binary.sh: the last two lines it needs
5884
5885After the `carlos-restart.path` block, guarded the same way:
5886
5887 if [ -f /etc/systemd/system/carlos-system-update.path ]; then
5888 systemctl enable --now carlos-system-update.path
5889 systemctl enable carlos-system-recover.service
5890 fi
5891
5892And to the post-conditions at the end:
5893
5894 systemctl is-active carlos-system-update.path
5895 systemctl is-enabled carlos-system-recover.service
5896
5897Same change in amadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh.
5898After every deployment has been hand-rolled once onto a binary carrying the
5899system-update spec, the script is retired (spec §10).
5900```
5901
5902- [ ] **Step 4: Full gate**
5903
5904Run: `export TMPDIR=/tmp/c; out=$(make test 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc`
5905Expected: rc=0 (vet, gofmt, every package).
5906
5907- [ ] **Step 5: Commit, push, describe, land**
5908
5909```bash
5910git add docs/site/cli.md AGENTS.md docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.md && git commit -q -F - <<'EOF'
5911docs: carlos system, the env contract, and the roll script's last two lines
5912
5913Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5914Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
5915EOF
5916git push -u origin carlos-system-update
5917amadan branch describe carlos/platform carlos-system-update -body - <<'EOF'
5918carlos system update: the platform rolls itself. Spec: docs/superpowers/specs/2026-09-06-carlos-system-update-design.md. Plan: docs/superpowers/plans/2026-09-06-carlos-system-update.md.
5919EOF
5920# then: amadan ci status carlos/platform -branch carlos-system-update (until 0)
5921# then: amadan branch merge carlos/platform carlos-system-update -expect <full sha>
5922```
5923
5924---
5925
5926## Self-review
5927
5928**Spec coverage.** §3 pointer, keys, immutability, spread, history-of-releases, rollback-is-new-promote, door grants, streaming read → Tasks 1, 2, 3. §4 leg (observe/tick, re-touch, watermark check, suppression, jitter, streamed fetch, request shape, deferred retry), units, exec shim, `enabledUnits` → Tasks 6, 7. §5 authorization, job, upload, update/status/rollback, `router` alias, no console verb → Tasks 9, 11, 12. §6 root state files, durability, lock, recovery-first (forward from gated), request walk, channel from env, replay guard per channel, no-op, staging copy 0600→0755, `.prev` decision, both unit sets from their binaries, quiesce incl. services, removed units, sysusers→tmpfiles→daemon-reload, reset-failed, gate+soak, arm after gate via `systemctlPlan`, record, rollback, rollback-failed → Tasks 4, 8a–8c. §7 first-roll arming (script + hourly belt), env contract → Tasks 13, 14. §9/§10 nothing to build. §11 unit-level proof: every bullet maps to a test in Tasks 1–13; the live checks are the operator's after landing.
5929
5930**Gaps found and fixed in review:** the spec's `release.Artifact` name collides with an existing type — Task 1 uses `SystemArtifact`; the spec's "interface gains `Open`" would break 48 fakes — Task 2 uses an `Opener` side-interface; the spec's 403 on the system doors is a bare 404 here, matching the console's operator doors; the spec's "ops-account owner" is `Role(OpsAccountID) == RoleOwner`, not `IsOperator` — Task 11 says which.
5931
5932**Placeholder scan.** Tasks 9 and 8a/8b carry explicit stubs that the next task replaces (`errNotYetImplemented`, the placeholder verbs, the stub `rollback`); each is named and replaced in a listed task. No "TBD", no "similar to".
5933
5934**Type consistency.** `systemRequest{Version, PromotedAt, NoOp, SeenAt}` is written by Task 7 and read by 8a; `systemResult.matches/terminal` are used identically in 7 and 8a; `bootstrapPlan{Enabled, Inert}` is produced by Task 6 and consumed by 8b/8c/13; `sysOps` fields are the same in 8a's fixture and 8c's `productionSysOps`; `release.SystemPromoteOptions{Actor, Spread}` in 1, 7's fixture and 11; `systemStatusResponse` in 12 matches the JSON Task 11 writes (`pointers`, `boxes` with `label, region, platform, channel, release, target, system, stale`).
5935