| 1 | # `carlos system update` Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** A fleet operator runs `carlos system update` and every box on the channel fetches, verifies, installs and health-gates the new platform binary itself, rolling back to the previous one if it does not come up. |
| 6 | |
| 7 | **Architecture:** A signed `SystemPointer` at `control/system/router/<channel>.json` names a version and its per-platform artifacts mirrored into the deployment bucket. A sandboxed agent leg notices the pointer on the serial poll, streams and verifies the artifact into `/data`, and touches a marker. A root oneshot (`carlos system apply`) re-verifies everything from root-owned copies, journals each step, swaps binary and units, restarts the edge, gates it, and rolls back to `carlos.prev` on failure; a boot-time unit recovers an interrupted apply. The console does the promote: fetch the public signed release, mirror, sign the pointer, bump the serial. |
| 8 | |
| 9 | **Tech Stack:** Go (stdlib, ed25519, net/http), systemd units embedded in the binary, S3/dir/fleet-door stores behind `release.Store`. |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-09-06-carlos-system-update-design.md` — read §3, §4, §6 before the task you are on. The spec wins where this plan and it disagree; say so in the commit if you find such a place. |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - Go build cache is read-only in the sandbox: run every `go` command with `dangerouslyDisableSandbox: true`, and `export TMPDIR=/tmp/c` first (long socket paths break `internal/edge`/`internal/activator` tests). |
| 16 | - Capture test exit codes explicitly: `out=$(go test ./pkg/... 2>&1); rc=$?; echo "$out" | tail -40; echo rc=$rc`. Never pipe `go test` into `grep`/`tail` and read the pipeline's status. |
| 17 | - `-count=1` on every test run. `gofmt -l .` must print nothing before a commit. |
| 18 | - Never run full `carlos bootstrap` on a live box path; the apply stages units into a temp root (`bootstrap -root <dir>`) and copies only `etc/`. Never copy a staging root's `data/`. |
| 19 | - sysusers before tmpfiles, always, in every code path that installs units. |
| 20 | - Root reads exactly one file under `/data` (`requested.json`), walks to it with `O_DIRECTORY|O_NOFOLLOW` per component, and derives every other path from a validated version string. Root never deletes under `/data`. |
| 21 | - Apply never `systemctl start`s a `.service`; it calls `systemctlPlan`, which starts only `.timer`/`.path` units. |
| 22 | - Every root state file is written temp → fsync → rename → directory fsync (`writeFileDurable`, Task 4). |
| 23 | - The component name is `router`; channels are `canary` and `stable`. Platform strings are `<goos>-<goarch>` matching `^[a-z0-9]+-[a-z0-9]+$`. |
| 24 | - Commit after every task with the attribution footer: |
| 25 | ``` |
| 26 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 27 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 28 | ``` |
| 29 | - Work on branch `carlos-system-update` from forge `main`; land with `amadan branch merge`, never `gh`. |
| 30 | |
| 31 | --- |
| 32 | |
| 33 | ## File Structure |
| 34 | |
| 35 | | File | Responsibility | |
| 36 | |---|---| |
| 37 | | `internal/release/system.go` (+`_test`) | `SystemPointer`, `SystemRelease`, `SystemArtifact`; key builders; validate/sign/verify (root key only); `GetSystemPointer`, `PromoteSystem`, `RollbackSystem` | |
| 38 | | `internal/release/store.go`, `internal/fleetapi/fleetapi.go` (+tests) | `Opener` side-interface, `Open` on `DirStore`/`S3Store`/`prefixStore`/fleetapi, `OpenObject`, `FetchToFile` | |
| 39 | | `internal/edge/storedoor.go` (+`_test`) | `aclRead` grants for the two new key shapes | |
| 40 | | `internal/host/systemstate.go` (+`_test`) | On-box state types (request, result, seen, applied, journal), paths, `writeFileDurable`, outcome constants | |
| 41 | | `internal/status/snapshot.go`, `internal/host/agent.go` | `Platform`, `SystemChannel`, `System` in `Probes`/`BoxMetrics`; filled in `statusLoop` | |
| 42 | | `internal/host/units/carlos-system-update.path`, `carlos-system-update.service`, `carlos-system-recover.service`; `internal/host/bootstrap.go`; `internal/host/probes.go` | The three units, `enabledUnits`, `-plan-json`, the path-unit probe | |
| 43 | | `internal/host/systemleg.go` (+`_test`) | `systemLeg`: `observe`/`tick`, fetch, request, marker re-touch; wired into `fastDeps` | |
| 44 | | `internal/host/systemapply.go` (+`_test`) | `RunSystemApply`, `systemApplier` with an injectable `sysOps` seam: refuse path, install path, gate, rollback, recovery, lock | |
| 45 | | `internal/host/agent.go` `RunAdopt` | The hourly belt: install and arm the three units if missing | |
| 46 | | `internal/publicrelease/feed.go` (+`_test`) | `Feed`: latest tag, verified asset fetch — factored out of `update.go`, shared with the console | |
| 47 | | `internal/console/apisystem.go` (+`_test`) | `/api/cli/system/{status,update,upload,jobs/{id},rollback}`; ops-owner gate; in-memory update job | |
| 48 | | `system.go` (+`_test`, package main), `main.go` | `carlos system update|status|rollback|apply`, `carlos system router …` alias | |
| 49 | | `docs/site/cli.md`, `AGENTS.md` | Operator docs; the env contract | |
| 50 | |
| 51 | Task order is dependency order. Tasks 1–3 are independent of each other and can run in parallel; 4 before 5/7/8; 6 before 8; 10 before 11; 11 before 12. |
| 52 | |
| 53 | --- |
| 54 | |
| 55 | ### Task 1: The system pointer (`internal/release/system.go`) |
| 56 | |
| 57 | **Files:** |
| 58 | - Create: `internal/release/system.go` |
| 59 | - Test: `internal/release/system_test.go` |
| 60 | |
| 61 | **Interfaces:** |
| 62 | - Consumes: `signedWrapper`, `Signer` (fields `key`, `scope`, method `usable()`), `Promotion`, `historyMax`, `auditMax`, `prependCapped`, `ValidChannel`, `ValidVersion`, `ParsePublicKey`, `Store`, `ErrCASConflict` — all in package `release` (`release.go`, `signer.go`, `store.go`). |
| 63 | - Produces (all exported, package `release`): |
| 64 | ```go |
| 65 | const SystemComponentRouter = "router" |
| 66 | func SystemPointerKey(component, channel string) string // control/system/<c>/<ch>.json |
| 67 | func SystemArtifactKey(version, platform string) string // bootstrap/router/<v>/carlos-<platform> |
| 68 | type SystemArtifact struct{ Key, SHA256 string } |
| 69 | type SystemRelease struct{ Version string; Artifacts map[string]SystemArtifact; Source string; ReleasedAt time.Time } |
| 70 | type SystemPointer struct{ Schema int; Component, Channel string; Current SystemRelease; History []SystemRelease; Spread string; PromotedAt time.Time; Audit []Promotion } |
| 71 | func (p *SystemPointer) SpreadDuration() time.Duration |
| 72 | func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error) |
| 73 | type SystemPromoteOptions struct{ Actor string; Spread time.Duration } |
| 74 | func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error) |
| 75 | func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error) |
| 76 | var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to") |
| 77 | ``` |
| 78 | |
| 79 | - [ ] **Step 1: Write the failing tests** |
| 80 | |
| 81 | ```go |
| 82 | package release |
| 83 | |
| 84 | import ( |
| 85 | "context" |
| 86 | "encoding/json" |
| 87 | "errors" |
| 88 | "os" |
| 89 | "strings" |
| 90 | "testing" |
| 91 | "time" |
| 92 | ) |
| 93 | |
| 94 | func testSystemRelease(v string) SystemRelease { |
| 95 | return SystemRelease{ |
| 96 | Version: v, |
| 97 | Artifacts: map[string]SystemArtifact{ |
| 98 | "linux-arm64": {Key: SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("ab", 32)}, |
| 99 | }, |
| 100 | Source: "release:" + v, |
| 101 | ReleasedAt: time.Date(2026, 9, 6, 8, 0, 0, 0, time.UTC), |
| 102 | } |
| 103 | } |
| 104 | |
| 105 | func TestSystemPointerRoundTrip(t *testing.T) { |
| 106 | priv, pub := testKeys(t) |
| 107 | s := NewDirStore(t.TempDir()) |
| 108 | ctx := context.Background() |
| 109 | if _, _, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable"); !errors.Is(err, os.ErrNotExist) { |
| 110 | t.Fatalf("missing pointer: err = %v, want ErrNotExist", err) |
| 111 | } |
| 112 | p, err := PromoteSystem(ctx, s, RootSigner(priv), SystemComponentRouter, "stable", testSystemRelease("v0.18.0"), |
| 113 | SystemPromoteOptions{Actor: "paul", Spread: 5 * time.Minute}) |
| 114 | if err != nil { |
| 115 | t.Fatal(err) |
| 116 | } |
| 117 | if p.Current.Version != "v0.18.0" || p.Spread != "5m0s" || p.Schema != 1 || p.PromotedAt.IsZero() { |
| 118 | t.Fatalf("promoted pointer = %+v", p) |
| 119 | } |
| 120 | got, etag, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable") |
| 121 | if err != nil || etag == "" { |
| 122 | t.Fatalf("get: %v etag=%q", err, etag) |
| 123 | } |
| 124 | if got.Current.Artifacts["linux-arm64"].SHA256 != strings.Repeat("ab", 32) { |
| 125 | t.Fatalf("artifact lost: %+v", got.Current) |
| 126 | } |
| 127 | if got.SpreadDuration() != 5*time.Minute { |
| 128 | t.Fatalf("spread = %v", got.SpreadDuration()) |
| 129 | } |
| 130 | if len(got.Audit) != 1 || got.Audit[0].Actor != "paul" || got.Audit[0].Version != "v0.18.0" { |
| 131 | t.Fatalf("audit = %+v", got.Audit) |
| 132 | } |
| 133 | } |
| 134 | |
| 135 | func TestSystemPromoteKeepsWholeReleaseInHistoryAndRollbackRestoresIt(t *testing.T) { |
| 136 | priv, pub := testKeys(t) |
| 137 | s := NewDirStore(t.TempDir()) |
| 138 | ctx := context.Background() |
| 139 | sg := RootSigner(priv) |
| 140 | if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.18.0"), SystemPromoteOptions{}); err != nil { |
| 141 | t.Fatal(err) |
| 142 | } |
| 143 | first, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary") |
| 144 | time.Sleep(1100 * time.Millisecond) // PromotedAt is second-truncated; the rollback must be NEWER |
| 145 | if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.19.0"), SystemPromoteOptions{}); err != nil { |
| 146 | t.Fatal(err) |
| 147 | } |
| 148 | p, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary") |
| 149 | if len(p.History) != 1 || p.History[0].Version != "v0.18.0" || p.History[0].Artifacts["linux-arm64"].Key == "" { |
| 150 | t.Fatalf("history must carry the whole release: %+v", p.History) |
| 151 | } |
| 152 | time.Sleep(1100 * time.Millisecond) |
| 153 | rb, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul") |
| 154 | if err != nil { |
| 155 | t.Fatal(err) |
| 156 | } |
| 157 | if rb.Current.Version != "v0.18.0" || rb.Current.Artifacts["linux-arm64"].Key != SystemArtifactKey("v0.18.0", "linux-arm64") { |
| 158 | t.Fatalf("rollback current = %+v", rb.Current) |
| 159 | } |
| 160 | if !rb.PromotedAt.After(first.PromotedAt) || !rb.PromotedAt.After(p.PromotedAt) { |
| 161 | t.Fatalf("a rollback is a NEW promote: promoted_at %v must be newer than %v and %v", rb.PromotedAt, first.PromotedAt, p.PromotedAt) |
| 162 | } |
| 163 | if len(rb.History) != 0 { |
| 164 | t.Fatalf("history after rollback = %+v", rb.History) |
| 165 | } |
| 166 | if _, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul"); !errors.Is(err, ErrNoSystemHistory) { |
| 167 | t.Fatalf("second rollback err = %v", err) |
| 168 | } |
| 169 | } |
| 170 | |
| 171 | func TestSystemPromoteSameVersionDoesNotGrowHistory(t *testing.T) { |
| 172 | priv, pub := testKeys(t) |
| 173 | s := NewDirStore(t.TempDir()) |
| 174 | ctx := context.Background() |
| 175 | sg := RootSigner(priv) |
| 176 | for i := 0; i < 2; i++ { |
| 177 | if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil { |
| 178 | t.Fatal(err) |
| 179 | } |
| 180 | } |
| 181 | p, _, _ := GetSystemPointer(ctx, s, pub, "router", "stable") |
| 182 | if len(p.History) != 0 { |
| 183 | t.Fatalf("re-promoting the running version must not push it into history: %+v", p.History) |
| 184 | } |
| 185 | } |
| 186 | |
| 187 | func TestSystemPointerRefusesScopedSigner(t *testing.T) { |
| 188 | _, _, sg := scopedFixture(t, []string{RungCanary, RungEdge}, 30*24*time.Hour) |
| 189 | s := NewDirStore(t.TempDir()) |
| 190 | _, err := PromoteSystem(context.Background(), s, sg, "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{}) |
| 191 | if err == nil || !strings.Contains(err.Error(), "root key") { |
| 192 | t.Fatalf("scoped signer signed a system pointer: err = %v", err) |
| 193 | } |
| 194 | } |
| 195 | |
| 196 | func TestSystemPointerRefusesAGrantOnTheWire(t *testing.T) { |
| 197 | // A wrapper carrying a grant is refused BEFORE the signature is checked, |
| 198 | // like the production flag (production.go) and the pipeline. |
| 199 | priv, pub := testKeys(t) |
| 200 | s := NewDirStore(t.TempDir()) |
| 201 | ctx := context.Background() |
| 202 | if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil { |
| 203 | t.Fatal(err) |
| 204 | } |
| 205 | body, etag, _ := s.Get(ctx, SystemPointerKey("router", "stable")) |
| 206 | var sw signedWrapper |
| 207 | if err := json.Unmarshal(body, &sw); err != nil { |
| 208 | t.Fatal(err) |
| 209 | } |
| 210 | sw.Grant = &signedGrant{Payload: "x", Sig: "y"} |
| 211 | tampered, _ := json.Marshal(sw) |
| 212 | if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), tampered, etag); err != nil { |
| 213 | t.Fatal(err) |
| 214 | } |
| 215 | if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil || !strings.Contains(err.Error(), "root key") { |
| 216 | t.Fatalf("grant-bearing wrapper accepted: err = %v", err) |
| 217 | } |
| 218 | } |
| 219 | |
| 220 | func TestSystemPointerTamperAndCrossChannelRefused(t *testing.T) { |
| 221 | priv, pub := testKeys(t) |
| 222 | s := NewDirStore(t.TempDir()) |
| 223 | ctx := context.Background() |
| 224 | if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil { |
| 225 | t.Fatal(err) |
| 226 | } |
| 227 | body, _, _ := s.Get(ctx, SystemPointerKey("router", "canary")) |
| 228 | // Cross-channel replay: the canary wrapper planted at the stable key. |
| 229 | if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), body, ""); err != nil { |
| 230 | t.Fatal(err) |
| 231 | } |
| 232 | if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil { |
| 233 | t.Fatal("a canary pointer verified as stable") |
| 234 | } |
| 235 | // Tampered signature. |
| 236 | var sw signedWrapper |
| 237 | _ = json.Unmarshal(body, &sw) |
| 238 | sw.Sig = "AA" + sw.Sig[2:] |
| 239 | bad, _ := json.Marshal(sw) |
| 240 | _, etag, _ := s.Get(ctx, SystemPointerKey("router", "canary")) |
| 241 | if _, err := s.Put(ctx, SystemPointerKey("router", "canary"), bad, etag); err != nil { |
| 242 | t.Fatal(err) |
| 243 | } |
| 244 | if _, _, err := GetSystemPointer(ctx, s, pub, "router", "canary"); err == nil { |
| 245 | t.Fatal("tampered signature verified") |
| 246 | } |
| 247 | } |
| 248 | |
| 249 | func TestSystemPointerValidation(t *testing.T) { |
| 250 | priv, _ := testKeys(t) |
| 251 | sg := RootSigner(priv) |
| 252 | ctx := context.Background() |
| 253 | cases := map[string]SystemRelease{ |
| 254 | "dev version": {Version: "dev", Artifacts: testSystemRelease("x").Artifacts}, |
| 255 | "no artifacts": {Version: "v1.0.0"}, |
| 256 | "bad platform": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux_arm64": {Key: "bootstrap/router/v1.0.0/carlos-linux_arm64", SHA256: strings.Repeat("a", 64)}}}, |
| 257 | "key not derived": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: "bootstrap/carlos", SHA256: strings.Repeat("a", 64)}}}, |
| 258 | "short sha": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: SystemArtifactKey("v1.0.0", "linux-arm64"), SHA256: "abc"}}}, |
| 259 | "version traversal": {Version: "../x", Artifacts: testSystemRelease("../x").Artifacts}, |
| 260 | } |
| 261 | for name, rel := range cases { |
| 262 | s := NewDirStore(t.TempDir()) |
| 263 | if _, err := PromoteSystem(ctx, s, sg, "router", "stable", rel, SystemPromoteOptions{}); err == nil { |
| 264 | t.Errorf("%s: accepted", name) |
| 265 | } |
| 266 | } |
| 267 | s := NewDirStore(t.TempDir()) |
| 268 | if _, err := PromoteSystem(ctx, s, sg, "Router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err == nil { |
| 269 | t.Error("bad component accepted") |
| 270 | } |
| 271 | if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{Spread: 48 * time.Hour}); err == nil { |
| 272 | t.Error("48h spread accepted") |
| 273 | } |
| 274 | } |
| 275 | |
| 276 | func TestSystemPromoteCASConflict(t *testing.T) { |
| 277 | priv, _ := testKeys(t) |
| 278 | ctx := context.Background() |
| 279 | inner := NewDirStore(t.TempDir()) |
| 280 | // A store whose Put always says "someone else moved it". |
| 281 | s := conflictingStore{Store: inner} |
| 282 | if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); !errors.Is(err, ErrCASConflict) { |
| 283 | t.Fatalf("err = %v, want ErrCASConflict", err) |
| 284 | } |
| 285 | } |
| 286 | |
| 287 | type conflictingStore struct{ Store } |
| 288 | |
| 289 | func (c conflictingStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) { |
| 290 | return "", ErrCASConflict |
| 291 | } |
| 292 | ``` |
| 293 | |
| 294 | - [ ] **Step 2: Run the tests to verify they fail** |
| 295 | |
| 296 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 297 | Expected: build failure naming `SystemPointer`, `PromoteSystem`, etc. as undefined. |
| 298 | |
| 299 | - [ ] **Step 3: Implement `internal/release/system.go`** |
| 300 | |
| 301 | ```go |
| 302 | package release |
| 303 | |
| 304 | import ( |
| 305 | "context" |
| 306 | "crypto/ed25519" |
| 307 | "encoding/base64" |
| 308 | "encoding/json" |
| 309 | "errors" |
| 310 | "fmt" |
| 311 | "os" |
| 312 | "regexp" |
| 313 | "time" |
| 314 | ) |
| 315 | |
| 316 | // SystemPointer is the platform's own channel pointer: what a ChannelPointer |
| 317 | // is for an app, for the thing that does the adopting. It lives under |
| 318 | // control/, not apps/, because the substrate is not an app (the |
| 319 | // console-as-an-app spec's dividing line), and it is ROOT-KEY-ONLY: a scoped |
| 320 | // grant's rungs name app channels, and a key that can roll the platform is a |
| 321 | // bigger key than any app promoter should hold. Spec §3. |
| 322 | type SystemPointer struct { |
| 323 | Schema int `json:"schema"` |
| 324 | Component string `json:"component"` |
| 325 | Channel string `json:"channel"` |
| 326 | Current SystemRelease `json:"current"` |
| 327 | History []SystemRelease `json:"history,omitempty"` |
| 328 | Spread string `json:"spread"` |
| 329 | PromotedAt time.Time `json:"promoted_at"` // of THIS write; the box's replay guard reads only this |
| 330 | Audit []Promotion `json:"audit,omitempty"` |
| 331 | } |
| 332 | |
| 333 | // SystemRelease is everything a box needs to install one version. It travels |
| 334 | // whole into History so a rollback re-installs what was there, not a version |
| 335 | // string with someone else's artifacts. |
| 336 | type SystemRelease struct { |
| 337 | Version string `json:"version"` |
| 338 | Artifacts map[string]SystemArtifact `json:"artifacts"` // keyed by "<goos>-<goarch>" |
| 339 | Source string `json:"source,omitempty"` |
| 340 | ReleasedAt time.Time `json:"released_at"` // audit only; never compared |
| 341 | } |
| 342 | |
| 343 | // SystemArtifact is one mirrored binary. Its sha in the signed pointer IS |
| 344 | // the artifact's signature: boxes need no .sig object. |
| 345 | type SystemArtifact struct { |
| 346 | Key string `json:"key"` |
| 347 | SHA256 string `json:"sha256"` |
| 348 | } |
| 349 | |
| 350 | // SystemComponentRouter is the one component today: the substrate binary |
| 351 | // (edge + activator + adoption, one process). A split adds a name here. |
| 352 | const SystemComponentRouter = "router" |
| 353 | |
| 354 | // ErrNoSystemHistory is RollbackSystem's answer when the pointer has never |
| 355 | // moved: there is nothing older to go back to. |
| 356 | var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to") |
| 357 | |
| 358 | const ( |
| 359 | systemSpreadMax = 24 * time.Hour |
| 360 | systemSpreadDefault = 5 * time.Minute |
| 361 | ) |
| 362 | |
| 363 | var ( |
| 364 | systemComponentRe = regexp.MustCompile(`^[a-z][a-z0-9-]{0,31}$`) |
| 365 | systemPlatformRe = regexp.MustCompile(`^[a-z0-9]+-[a-z0-9]+$`) |
| 366 | systemSHA256Re = regexp.MustCompile(`^[0-9a-f]{64}$`) |
| 367 | ) |
| 368 | |
| 369 | // SystemPointerKey is control/system/<component>/<channel>.json. |
| 370 | func SystemPointerKey(component, channel string) string { |
| 371 | return "control/system/" + component + "/" + channel + ".json" |
| 372 | } |
| 373 | |
| 374 | // SystemArtifactKey is bootstrap/router/<version>/carlos-<platform>: under |
| 375 | // bootstrap/ because the store door already means "the platform's own |
| 376 | // binary" by that name, per-version because canary and stable boxes must be |
| 377 | // able to run different binaries at once. |
| 378 | func SystemArtifactKey(version, platform string) string { |
| 379 | return "bootstrap/router/" + version + "/carlos-" + platform |
| 380 | } |
| 381 | |
| 382 | // SpreadDuration is Spread parsed, or the default when it is unparseable — |
| 383 | // validation refuses the unparseable case on the way in, so this only |
| 384 | // defaults for a hand-edited or pre-field pointer. |
| 385 | func (p *SystemPointer) SpreadDuration() time.Duration { |
| 386 | d, err := time.ParseDuration(p.Spread) |
| 387 | if err != nil || d < 0 || d > systemSpreadMax { |
| 388 | return systemSpreadDefault |
| 389 | } |
| 390 | return d |
| 391 | } |
| 392 | |
| 393 | func validateSystemRelease(r SystemRelease) error { |
| 394 | if !ValidVersion(r.Version) || r.Version == "dev" { |
| 395 | return fmt.Errorf("version %q is not a valid platform version", r.Version) |
| 396 | } |
| 397 | if len(r.Artifacts) == 0 { |
| 398 | return fmt.Errorf("version %s names no artifacts", r.Version) |
| 399 | } |
| 400 | for plat, a := range r.Artifacts { |
| 401 | if !systemPlatformRe.MatchString(plat) { |
| 402 | return fmt.Errorf("platform %q is not <goos>-<goarch>", plat) |
| 403 | } |
| 404 | if a.Key != SystemArtifactKey(r.Version, plat) { |
| 405 | return fmt.Errorf("artifact key %q is not %s", a.Key, SystemArtifactKey(r.Version, plat)) |
| 406 | } |
| 407 | if !systemSHA256Re.MatchString(a.SHA256) { |
| 408 | return fmt.Errorf("artifact %s: sha256 %q is not 64 hex characters", a.Key, a.SHA256) |
| 409 | } |
| 410 | } |
| 411 | return nil |
| 412 | } |
| 413 | |
| 414 | func validateSystemPointer(p *SystemPointer, wantComponent, wantChannel string) error { |
| 415 | if p.Schema != 1 { |
| 416 | return fmt.Errorf("system pointer schema %d unsupported", p.Schema) |
| 417 | } |
| 418 | if p.Component != wantComponent || p.Channel != wantChannel { |
| 419 | return fmt.Errorf("system pointer is for %s/%s, not %s/%s", p.Component, p.Channel, wantComponent, wantChannel) |
| 420 | } |
| 421 | if !systemComponentRe.MatchString(p.Component) || !ValidChannel(p.Channel) { |
| 422 | return fmt.Errorf("system pointer %s/%s: bad component or channel", p.Component, p.Channel) |
| 423 | } |
| 424 | if err := validateSystemRelease(p.Current); err != nil { |
| 425 | return fmt.Errorf("system pointer %s/%s: %w", p.Component, p.Channel, err) |
| 426 | } |
| 427 | if d, err := time.ParseDuration(p.Spread); err != nil || d < 0 || d > systemSpreadMax { |
| 428 | return fmt.Errorf("system pointer %s/%s: spread %q must be a duration between 0 and %s", p.Component, p.Channel, p.Spread, systemSpreadMax) |
| 429 | } |
| 430 | if p.PromotedAt.IsZero() { |
| 431 | return fmt.Errorf("system pointer %s/%s: promoted_at unset", p.Component, p.Channel) |
| 432 | } |
| 433 | if len(p.History) > historyMax { |
| 434 | return fmt.Errorf("system pointer %s/%s: history longer than %d", p.Component, p.Channel, historyMax) |
| 435 | } |
| 436 | for _, h := range p.History { |
| 437 | if err := validateSystemRelease(h); err != nil { |
| 438 | return fmt.Errorf("system pointer %s/%s: history: %w", p.Component, p.Channel, err) |
| 439 | } |
| 440 | } |
| 441 | if len(p.Audit) > auditMax { |
| 442 | return fmt.Errorf("system pointer %s/%s: audit longer than %d", p.Component, p.Channel, auditMax) |
| 443 | } |
| 444 | return nil |
| 445 | } |
| 446 | |
| 447 | // signSystemPointer mirrors signPointer with two differences: the signer |
| 448 | // must be the root key, and the wrapper never carries a grant. |
| 449 | func signSystemPointer(sg Signer, p *SystemPointer) ([]byte, error) { |
| 450 | if err := sg.usable(); err != nil { |
| 451 | return nil, err |
| 452 | } |
| 453 | if sg.scope != nil { |
| 454 | return nil, errors.New("system pointer: signed by a scoped key — the platform takes the root key only") |
| 455 | } |
| 456 | p.Schema = 1 |
| 457 | p.PromotedAt = time.Now().UTC().Truncate(time.Second) |
| 458 | if err := validateSystemPointer(p, p.Component, p.Channel); err != nil { |
| 459 | return nil, err |
| 460 | } |
| 461 | payload, err := json.MarshalIndent(p, "", " ") |
| 462 | if err != nil { |
| 463 | return nil, err |
| 464 | } |
| 465 | b64 := base64.StdEncoding.EncodeToString(payload) |
| 466 | sig := ed25519.Sign(sg.key, []byte(b64)) |
| 467 | return json.MarshalIndent(signedWrapper{Payload: b64, Sig: base64.StdEncoding.EncodeToString(sig)}, "", " ") |
| 468 | } |
| 469 | |
| 470 | // verifySystemPointer is the single parse site for system pointers. A grant |
| 471 | // on the wire is refused before the signature is looked at (the production |
| 472 | // flag's rule, production.go). |
| 473 | func verifySystemPointer(pub ed25519.PublicKey, wrapper []byte, wantComponent, wantChannel string) (*SystemPointer, error) { |
| 474 | var sw signedWrapper |
| 475 | if err := json.Unmarshal(wrapper, &sw); err != nil { |
| 476 | return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err) |
| 477 | } |
| 478 | if sw.Grant != nil { |
| 479 | return nil, fmt.Errorf("system pointer %s/%s: signed by a scoped key — the platform takes the root key only", wantComponent, wantChannel) |
| 480 | } |
| 481 | sig, err := base64.StdEncoding.DecodeString(sw.Sig) |
| 482 | if err != nil { |
| 483 | return nil, fmt.Errorf("system pointer %s/%s: bad signature encoding", wantComponent, wantChannel) |
| 484 | } |
| 485 | if len(pub) != ed25519.PublicKeySize || !ed25519.Verify(pub, []byte(sw.Payload), sig) { |
| 486 | return nil, fmt.Errorf("system pointer %s/%s: signature does not verify", wantComponent, wantChannel) |
| 487 | } |
| 488 | payload, err := base64.StdEncoding.DecodeString(sw.Payload) |
| 489 | if err != nil { |
| 490 | return nil, fmt.Errorf("system pointer %s/%s: bad payload encoding", wantComponent, wantChannel) |
| 491 | } |
| 492 | var p SystemPointer |
| 493 | if err := json.Unmarshal(payload, &p); err != nil { |
| 494 | return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err) |
| 495 | } |
| 496 | if err := validateSystemPointer(&p, wantComponent, wantChannel); err != nil { |
| 497 | return nil, err |
| 498 | } |
| 499 | return &p, nil |
| 500 | } |
| 501 | |
| 502 | // GetSystemPointer reads and verifies one channel's pointer. A missing key |
| 503 | // is os.ErrNotExist, which is the normal state of every deployment before |
| 504 | // its first `carlos system update`. |
| 505 | func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error) { |
| 506 | body, etag, err := s.Get(ctx, SystemPointerKey(component, channel)) |
| 507 | if err != nil { |
| 508 | return nil, "", err |
| 509 | } |
| 510 | p, err := verifySystemPointer(pub, body, component, channel) |
| 511 | if err != nil { |
| 512 | return nil, "", err |
| 513 | } |
| 514 | return p, etag, nil |
| 515 | } |
| 516 | |
| 517 | // SystemPromoteOptions: who did it, and how long boxes spread their starts. |
| 518 | type SystemPromoteOptions struct { |
| 519 | Actor string |
| 520 | Spread time.Duration // 0 = hotfix, every box at once; unset in the CLI means the default |
| 521 | } |
| 522 | |
| 523 | // PromoteSystem writes the pointer for one channel, CAS-guarded. The |
| 524 | // outgoing release goes whole into History unless it is the same version |
| 525 | // (a re-promote of the running version is a no-op for boxes and must not |
| 526 | // push that version into its own history). |
| 527 | func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error) { |
| 528 | key := SystemPointerKey(component, channel) |
| 529 | var old *SystemPointer |
| 530 | body, etag, err := s.Get(ctx, key) |
| 531 | switch { |
| 532 | case err == nil: |
| 533 | if old, err = verifySystemPointer(sg.verifyPub(), body, component, channel); err != nil { |
| 534 | return nil, err |
| 535 | } |
| 536 | case errors.Is(err, os.ErrNotExist): |
| 537 | etag = "" |
| 538 | default: |
| 539 | return nil, err |
| 540 | } |
| 541 | next := &SystemPointer{Component: component, Channel: channel, Current: rel, Spread: o.Spread.String()} |
| 542 | if rel.ReleasedAt.IsZero() { |
| 543 | next.Current.ReleasedAt = time.Now().UTC().Truncate(time.Second) |
| 544 | } |
| 545 | from := "" |
| 546 | if old != nil { |
| 547 | next.History, next.Audit = old.History, old.Audit |
| 548 | from = old.Current.Version |
| 549 | if old.Current.Version != rel.Version { |
| 550 | next.History = prependCapped(old.Current, old.History, historyMax) |
| 551 | } |
| 552 | } |
| 553 | next.Audit = prependCapped(Promotion{Version: rel.Version, From: from, To: channel, Actor: o.Actor, At: time.Now().UTC()}, next.Audit, auditMax) |
| 554 | wrapper, err := signSystemPointer(sg, next) |
| 555 | if err != nil { |
| 556 | return nil, err |
| 557 | } |
| 558 | if _, err := s.Put(ctx, key, wrapper, etag); err != nil { |
| 559 | return nil, err |
| 560 | } |
| 561 | return next, nil |
| 562 | } |
| 563 | |
| 564 | // RollbackSystem swaps Current with History[0] and writes a NEW pointer — |
| 565 | // new PromotedAt — so every box's replay guard sees a promote, not a replay. |
| 566 | func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error) { |
| 567 | key := SystemPointerKey(component, channel) |
| 568 | body, etag, err := s.Get(ctx, key) |
| 569 | if err != nil { |
| 570 | return nil, err |
| 571 | } |
| 572 | old, err := verifySystemPointer(sg.verifyPub(), body, component, channel) |
| 573 | if err != nil { |
| 574 | return nil, err |
| 575 | } |
| 576 | if len(old.History) == 0 { |
| 577 | return nil, ErrNoSystemHistory |
| 578 | } |
| 579 | next := &SystemPointer{ |
| 580 | Component: component, Channel: channel, |
| 581 | Current: old.History[0], History: old.History[1:], Spread: old.Spread, |
| 582 | Audit: prependCapped(Promotion{Version: old.History[0].Version, From: old.Current.Version, To: channel, Actor: actor, At: time.Now().UTC()}, old.Audit, auditMax), |
| 583 | } |
| 584 | wrapper, err := signSystemPointer(sg, next) |
| 585 | if err != nil { |
| 586 | return nil, err |
| 587 | } |
| 588 | if _, err := s.Put(ctx, key, wrapper, etag); err != nil { |
| 589 | return nil, err |
| 590 | } |
| 591 | return next, nil |
| 592 | } |
| 593 | ``` |
| 594 | |
| 595 | Note `sg.verifyPub()` is the existing unexported method on `Signer` (`signer.go:66`); for a root signer it returns the root public key. |
| 596 | |
| 597 | - [ ] **Step 4: Run the tests** |
| 598 | |
| 599 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 600 | Expected: `ok`, rc=0. If `TestSystemPromoteCASConflict` fails because `conflictingStore` lacks a method, the `Store` interface grew — embed satisfies it; check the error text. |
| 601 | |
| 602 | - [ ] **Step 5: gofmt and commit** |
| 603 | |
| 604 | ```bash |
| 605 | gofmt -l internal/release/ && git add internal/release/system.go internal/release/system_test.go && git commit -q -F - <<'EOF' |
| 606 | release: the platform's own channel pointer, root-key-only |
| 607 | |
| 608 | control/system/<component>/<channel>.json names a version and its |
| 609 | per-platform artifacts. History carries whole releases so a rollback |
| 610 | restores artifacts, and a rollback is a new promote (new PromotedAt). |
| 611 | A grant on the wire is refused before the signature is read, like the |
| 612 | production flag. |
| 613 | |
| 614 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 615 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 616 | EOF |
| 617 | ``` |
| 618 | |
| 619 | --- |
| 620 | |
| 621 | ### Task 2: Streaming reads (`Opener`, `FetchToFile`) |
| 622 | |
| 623 | **Files:** |
| 624 | - Modify: `internal/release/store.go` (after the `SizeLister` block, ~line 190) |
| 625 | - Modify: `internal/fleetapi/fleetapi.go` (after `Get`, ~line 156) |
| 626 | - Create: `internal/release/fetch.go` |
| 627 | - Test: `internal/release/fetch_test.go`, `internal/fleetapi/fleetapi_test.go` (add one test) |
| 628 | |
| 629 | **Interfaces:** |
| 630 | - Consumes: `Store`, `DirStore.path`, `S3Store.Client/Bucket`, `s3NotFound`, `prefixStore.full`; fleetapi `store.newRequest`, `objectURL`. |
| 631 | - Produces: |
| 632 | ```go |
| 633 | // package release |
| 634 | type Opener interface{ Open(ctx context.Context, key string) (io.ReadCloser, string, error) } |
| 635 | func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error) |
| 636 | var ErrTooLarge = errors.New("store: object larger than the caller's limit") |
| 637 | func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) error |
| 638 | ``` |
| 639 | `FetchToFile` writes `dst+".tmp"`, streams with a running sha, refuses more than `limit` bytes (`ErrTooLarge`), refuses a sha mismatch (error contains "sha256"), fsyncs, renames, fsyncs the directory, and removes the temp on any failure. Etag from `Open` may be `""` (DirStore). |
| 640 | |
| 641 | - [ ] **Step 1: Write the failing tests** |
| 642 | |
| 643 | `internal/release/fetch_test.go`: |
| 644 | ```go |
| 645 | package release |
| 646 | |
| 647 | import ( |
| 648 | "context" |
| 649 | "crypto/sha256" |
| 650 | "encoding/hex" |
| 651 | "errors" |
| 652 | "os" |
| 653 | "path/filepath" |
| 654 | "strings" |
| 655 | "testing" |
| 656 | ) |
| 657 | |
| 658 | func shaOf(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) } |
| 659 | |
| 660 | func TestFetchToFileStreamsAndVerifies(t *testing.T) { |
| 661 | ctx := context.Background() |
| 662 | body := []byte(strings.Repeat("carlos", 10000)) |
| 663 | for name, s := range map[string]Store{ |
| 664 | "dir": NewDirStore(t.TempDir()), |
| 665 | "prefix": NewPrefixStore(NewDirStore(t.TempDir()), "fleets/pi/"), |
| 666 | } { |
| 667 | t.Run(name, func(t *testing.T) { |
| 668 | if _, err := s.Put(ctx, "bootstrap/router/v1/carlos-linux-arm64", body, ""); err != nil { |
| 669 | t.Fatal(err) |
| 670 | } |
| 671 | dst := filepath.Join(t.TempDir(), "carlos") |
| 672 | if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst, int64(len(body)), shaOf(body)); err != nil { |
| 673 | t.Fatal(err) |
| 674 | } |
| 675 | got, _ := os.ReadFile(dst) |
| 676 | if string(got) != string(body) { |
| 677 | t.Fatal("content differs") |
| 678 | } |
| 679 | if _, err := os.Stat(dst + ".tmp"); !errors.Is(err, os.ErrNotExist) { |
| 680 | t.Fatal("temp file left behind") |
| 681 | } |
| 682 | // One byte over the limit is refused, and nothing is left on disk. |
| 683 | dst2 := filepath.Join(t.TempDir(), "carlos") |
| 684 | if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst2, int64(len(body))-1, shaOf(body)); !errors.Is(err, ErrTooLarge) { |
| 685 | t.Fatalf("over-limit err = %v", err) |
| 686 | } |
| 687 | if _, err := os.Stat(dst2); !errors.Is(err, os.ErrNotExist) { |
| 688 | t.Fatal("over-limit fetch left a file") |
| 689 | } |
| 690 | // A sha mismatch is refused and leaves nothing. |
| 691 | dst3 := filepath.Join(t.TempDir(), "carlos") |
| 692 | err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst3, int64(len(body)), strings.Repeat("0", 64)) |
| 693 | if err == nil || !strings.Contains(err.Error(), "sha256") { |
| 694 | t.Fatalf("mismatch err = %v", err) |
| 695 | } |
| 696 | if _, err := os.Stat(dst3); !errors.Is(err, os.ErrNotExist) { |
| 697 | t.Fatal("mismatch fetch left a file") |
| 698 | } |
| 699 | // Missing key is ErrNotExist. |
| 700 | if err := FetchToFile(ctx, s, "bootstrap/router/v9/carlos-linux-arm64", dst3, 10, shaOf(nil)); !errors.Is(err, os.ErrNotExist) { |
| 701 | t.Fatalf("missing err = %v", err) |
| 702 | } |
| 703 | }) |
| 704 | } |
| 705 | } |
| 706 | |
| 707 | func TestPrefixStoreOpenRebasesTheKey(t *testing.T) { |
| 708 | // prefixStore embeds the Store INTERFACE, so a promoted Open would forward |
| 709 | // the raw key and silently escape the prefix. It must be written out. |
| 710 | ctx := context.Background() |
| 711 | inner := NewDirStore(t.TempDir()) |
| 712 | if _, err := inner.Put(ctx, "fleets/pi/x", []byte("in"), ""); err != nil { |
| 713 | t.Fatal(err) |
| 714 | } |
| 715 | if _, err := inner.Put(ctx, "x", []byte("OUT"), ""); err != nil { |
| 716 | t.Fatal(err) |
| 717 | } |
| 718 | rc, _, err := OpenObject(ctx, NewPrefixStore(inner, "fleets/pi/"), "x") |
| 719 | if err != nil { |
| 720 | t.Fatal(err) |
| 721 | } |
| 722 | defer rc.Close() |
| 723 | b := make([]byte, 8) |
| 724 | n, _ := rc.Read(b) |
| 725 | if string(b[:n]) != "in" { |
| 726 | t.Fatalf("read %q through the prefix, want \"in\"", b[:n]) |
| 727 | } |
| 728 | } |
| 729 | |
| 730 | func TestOpenObjectFallsBackToGet(t *testing.T) { |
| 731 | // A Store that is not an Opener still works: OpenObject buffers via Get. |
| 732 | s := getOnly{NewDirStore(t.TempDir())} |
| 733 | if _, err := s.Put(context.Background(), "k", []byte("v"), ""); err != nil { |
| 734 | t.Fatal(err) |
| 735 | } |
| 736 | rc, _, err := OpenObject(context.Background(), s, "k") |
| 737 | if err != nil { |
| 738 | t.Fatal(err) |
| 739 | } |
| 740 | b := make([]byte, 4) |
| 741 | n, _ := rc.Read(b) |
| 742 | if string(b[:n]) != "v" { |
| 743 | t.Fatalf("got %q", b[:n]) |
| 744 | } |
| 745 | } |
| 746 | |
| 747 | type getOnly struct{ *DirStore } |
| 748 | ``` |
| 749 | (`getOnly` embeds `*DirStore`, so it has `Open` promoted — to make it a non-Opener, shadow it:) |
| 750 | ```go |
| 751 | func (getOnly) Open() {} // different signature: getOnly is NOT an Opener |
| 752 | ``` |
| 753 | |
| 754 | `internal/fleetapi/fleetapi_test.go`, add: |
| 755 | ```go |
| 756 | func TestOpenStreamsThroughTheDoor(t *testing.T) { |
| 757 | d := newTestDoor(t) |
| 758 | body := bytes.Repeat([]byte("x"), 1<<20) |
| 759 | d.seed(t, "bootstrap/carlos", body) |
| 760 | c := d.box(t) |
| 761 | op, ok := c.(release.Opener) |
| 762 | if !ok { |
| 763 | t.Fatal("fleetapi store is not an Opener") |
| 764 | } |
| 765 | rc, _, err := op.Open(context.Background(), "bootstrap/carlos") |
| 766 | if err != nil { |
| 767 | t.Fatal(err) |
| 768 | } |
| 769 | defer rc.Close() |
| 770 | got, _ := io.ReadAll(rc) |
| 771 | if !bytes.Equal(got, body) { |
| 772 | t.Fatalf("read %d bytes, want %d", len(got), len(body)) |
| 773 | } |
| 774 | if _, _, err := op.Open(context.Background(), "bootstrap/nothing"); !errors.Is(err, os.ErrNotExist) { |
| 775 | t.Fatalf("missing: %v", err) |
| 776 | } |
| 777 | } |
| 778 | ``` |
| 779 | (`newTestDoor`, `seed`, `box` exist in that file, `fleetapi_test.go:52-105`; `bootstrap/carlos` is already readable by a box.) |
| 780 | |
| 781 | - [ ] **Step 2: Run to verify failure** |
| 782 | |
| 783 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ -run 'Fetch|Open' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 784 | Expected: undefined `Opener`, `OpenObject`, `FetchToFile`, `ErrTooLarge`. |
| 785 | |
| 786 | - [ ] **Step 3: Implement** |
| 787 | |
| 788 | `internal/release/store.go`, after the `SizeLister` assertions (~line 190): |
| 789 | ```go |
| 790 | // Opener is the optional streaming read. Store.Get returns the whole object |
| 791 | // in memory, which is fine for a pointer and wrong for a platform binary on |
| 792 | // a box under MemoryHigh=70%. Implemented by every production store; a test |
| 793 | // fake that only has Get still works through OpenObject. The etag may be "" |
| 794 | // (DirStore cannot know it without reading the file). |
| 795 | type Opener interface { |
| 796 | Open(ctx context.Context, key string) (io.ReadCloser, string, error) |
| 797 | } |
| 798 | |
| 799 | var ( |
| 800 | _ Opener = (*DirStore)(nil) |
| 801 | _ Opener = (*S3Store)(nil) |
| 802 | _ Opener = prefixStore{} |
| 803 | ) |
| 804 | |
| 805 | // OpenObject streams key from s, buffering through Get when s cannot stream. |
| 806 | func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error) { |
| 807 | if o, ok := s.(Opener); ok { |
| 808 | return o.Open(ctx, key) |
| 809 | } |
| 810 | b, etag, err := s.Get(ctx, key) |
| 811 | if err != nil { |
| 812 | return nil, "", err |
| 813 | } |
| 814 | return io.NopCloser(bytes.NewReader(b)), etag, nil |
| 815 | } |
| 816 | ``` |
| 817 | `DirStore`: |
| 818 | ```go |
| 819 | func (d *DirStore) Open(_ context.Context, key string) (io.ReadCloser, string, error) { |
| 820 | p, err := d.path(key) |
| 821 | if err != nil { |
| 822 | return nil, "", err |
| 823 | } |
| 824 | f, err := os.Open(p) |
| 825 | if err != nil { |
| 826 | if errors.Is(err, os.ErrNotExist) { |
| 827 | return nil, "", fmt.Errorf("dirstore: %s: %w", key, os.ErrNotExist) |
| 828 | } |
| 829 | return nil, "", err |
| 830 | } |
| 831 | return f, "", nil |
| 832 | } |
| 833 | ``` |
| 834 | `S3Store`: |
| 835 | ```go |
| 836 | func (s *S3Store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) { |
| 837 | out, err := s.Client.GetObject(ctx, &awss3.GetObjectInput{Bucket: &s.Bucket, Key: &key}) |
| 838 | if err != nil { |
| 839 | if s3NotFound(err) { |
| 840 | return nil, "", fmt.Errorf("s3store: %s: %w", key, os.ErrNotExist) |
| 841 | } |
| 842 | return nil, "", err |
| 843 | } |
| 844 | return out.Body, aws.ToString(out.ETag), nil |
| 845 | } |
| 846 | ``` |
| 847 | `prefixStore` (explicit, next to its `Get`): |
| 848 | ```go |
| 849 | // Open is written out, not promoted: the embedded field is the interface, |
| 850 | // and a promoted Open would forward the UNREBASED key past the prefix. |
| 851 | func (p prefixStore) Open(ctx context.Context, key string) (io.ReadCloser, string, error) { |
| 852 | return OpenObject(ctx, p.Store, p.full(key)) |
| 853 | } |
| 854 | ``` |
| 855 | Add `"bytes"` and `"io"` to the imports if absent. |
| 856 | |
| 857 | `internal/fleetapi/fleetapi.go`, after `Get`: |
| 858 | ```go |
| 859 | // Open streams one object. Unlike Get it does not buffer the body, which |
| 860 | // is what a 25 MiB platform binary on a Pi needs; the 404 mapping is the |
| 861 | // same as Get's so callers can errors.Is(err, os.ErrNotExist). |
| 862 | func (s *store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) { |
| 863 | req, err := s.newRequest(ctx, http.MethodGet, s.objectURL(key), nil) |
| 864 | if err != nil { |
| 865 | return nil, "", err |
| 866 | } |
| 867 | resp, err := s.client.Do(req) |
| 868 | if err != nil { |
| 869 | return nil, "", errorf("open %s: %w", key, err) |
| 870 | } |
| 871 | switch resp.StatusCode { |
| 872 | case http.StatusOK: |
| 873 | return resp.Body, resp.Header.Get("ETag"), nil |
| 874 | case http.StatusNotFound: |
| 875 | resp.Body.Close() |
| 876 | return nil, "", fmt.Errorf("fleetapi: %s: %w", key, os.ErrNotExist) |
| 877 | default: |
| 878 | resp.Body.Close() |
| 879 | return nil, "", errorf("open %s: unexpected status %d", key, resp.StatusCode) |
| 880 | } |
| 881 | } |
| 882 | |
| 883 | var _ release.Opener = (*store)(nil) |
| 884 | ``` |
| 885 | |
| 886 | `internal/release/fetch.go`: |
| 887 | ```go |
| 888 | package release |
| 889 | |
| 890 | import ( |
| 891 | "context" |
| 892 | "crypto/sha256" |
| 893 | "encoding/hex" |
| 894 | "errors" |
| 895 | "fmt" |
| 896 | "io" |
| 897 | "os" |
| 898 | "path/filepath" |
| 899 | ) |
| 900 | |
| 901 | // ErrTooLarge: the object exceeded the caller's limit. The reader is cut at |
| 902 | // limit+1 bytes, so an object EXACTLY at the limit is told from one over it. |
| 903 | var ErrTooLarge = errors.New("store: object larger than the caller's limit") |
| 904 | |
| 905 | // FetchToFile streams key into dst with a running sha256, never holding the |
| 906 | // object in memory: temp file, fsync, rename, directory fsync. Any failure |
| 907 | // removes the temp and leaves dst untouched. |
| 908 | func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) (err error) { |
| 909 | rc, _, err := OpenObject(ctx, s, key) |
| 910 | if err != nil { |
| 911 | return err |
| 912 | } |
| 913 | defer rc.Close() |
| 914 | tmp := dst + ".tmp" |
| 915 | f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600) |
| 916 | if err != nil { |
| 917 | return err |
| 918 | } |
| 919 | defer func() { |
| 920 | if err != nil { |
| 921 | f.Close() |
| 922 | os.Remove(tmp) |
| 923 | } |
| 924 | }() |
| 925 | h := sha256.New() |
| 926 | n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, limit+1)) |
| 927 | if err != nil { |
| 928 | return fmt.Errorf("fetch %s: %w", key, err) |
| 929 | } |
| 930 | if n > limit { |
| 931 | return fmt.Errorf("fetch %s: %w (limit %d)", key, ErrTooLarge, limit) |
| 932 | } |
| 933 | if got := hex.EncodeToString(h.Sum(nil)); got != wantSHA256 { |
| 934 | return fmt.Errorf("fetch %s: sha256 %s does not match the signed %s", key, got, wantSHA256) |
| 935 | } |
| 936 | if err = f.Sync(); err != nil { |
| 937 | return err |
| 938 | } |
| 939 | if err = f.Close(); err != nil { |
| 940 | return err |
| 941 | } |
| 942 | if err = os.Rename(tmp, dst); err != nil { |
| 943 | return err |
| 944 | } |
| 945 | return syncDir(filepath.Dir(dst)) |
| 946 | } |
| 947 | |
| 948 | // syncDir makes a rename durable: the directory entry is what fsync(file) |
| 949 | // does not cover. |
| 950 | func syncDir(dir string) error { |
| 951 | d, err := os.Open(dir) |
| 952 | if err != nil { |
| 953 | return err |
| 954 | } |
| 955 | defer d.Close() |
| 956 | return d.Sync() |
| 957 | } |
| 958 | ``` |
| 959 | |
| 960 | - [ ] **Step 4: Run the tests** |
| 961 | |
| 962 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 963 | Expected: all `ok`. (`internal/edge` is included because its test fakes assign to `release.Store`; the interface did not change, so they still compile.) |
| 964 | |
| 965 | - [ ] **Step 5: Commit** |
| 966 | |
| 967 | ```bash |
| 968 | gofmt -l internal/ && git add internal/release/store.go internal/release/fetch.go internal/release/fetch_test.go internal/fleetapi/fleetapi.go internal/fleetapi/fleetapi_test.go && git commit -q -F - <<'EOF' |
| 969 | release: stream an object to a file with a running sha |
| 970 | |
| 971 | Opener is a side-interface (48 test fakes implement Store; none break), |
| 972 | implemented by DirStore, S3Store, prefixStore and the fleet door client. |
| 973 | FetchToFile is what a box uses for a platform binary: limit+1 read, sha |
| 974 | checked before rename, directory fsync after. |
| 975 | |
| 976 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 977 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 978 | EOF |
| 979 | ``` |
| 980 | |
| 981 | --- |
| 982 | |
| 983 | ### Task 3: Store-door grants for the pointer and artifact shapes |
| 984 | |
| 985 | **Files:** |
| 986 | - Modify: `internal/edge/storedoor.go:383-397` (`aclRead`) and the doc comment above it (`:352-382`) |
| 987 | - Modify: `internal/edge/storedoor_test.go` (`TestStoreDoorACLMatrix` rows at `:69+`, `TestStoreDoorGetServesOwnEstate` at `:365`, the prose in `TestDoneNeverServesCertMaterial`'s comment at `:519`) |
| 988 | |
| 989 | **Interfaces:** |
| 990 | - Consumes: `aclRead(sqid, fleet, key string) bool`; `canonicalKey` already refuses any `..`/`.`/empty segment before the ACL runs, so traversal never reaches here. |
| 991 | - Produces: `func systemReadKey(key string) bool` (unexported) — true for exactly `control/system/<c>/<ch>.json` and `bootstrap/router/<v>/<asset>`. |
| 992 | |
| 993 | - [ ] **Step 1: Add the failing matrix rows and GET cases** |
| 994 | |
| 995 | In `TestStoreDoorACLMatrix`'s table, after the `bootstrap/carlos.sig` row (`:84`): |
| 996 | ```go |
| 997 | {"control/system/router/stable.json", opRead, true, "the platform channel pointer (system update spec §3)"}, |
| 998 | {"control/system/router/canary.json", opRead, true, "…and the canary one"}, |
| 999 | {"control/system/router/stable.json", opWrite, false, "a box never writes a pointer"}, |
| 1000 | {"control/system/router", opRead, false, "the pointer shape is exactly three segments ending .json"}, |
| 1001 | {"control/system/router/stable.json/x", opRead, false, "…not four"}, |
| 1002 | {"control/system/router/stable.txt", opRead, false, "…and .json"}, |
| 1003 | {"bootstrap/router/v0.18.0/carlos-linux-arm64", opRead, true, "a mirrored platform artifact"}, |
| 1004 | {"bootstrap/router/v0.18.0/carlos-linux-arm64", opWrite, false, "a box must never publish the platform binary"}, |
| 1005 | {"bootstrap/router/v0.18.0", opRead, false, "the artifact shape is exactly three segments"}, |
| 1006 | {"bootstrap/router/v0.18.0/a/b", opRead, false, "…not four"}, |
| 1007 | {"bootstrap/router.sig", opRead, false, "nothing else under bootstrap/"}, |
| 1008 | {"bootstrap/router/../../x", opRead, false, "traversal is refused by canonicalKey before the ACL"}, |
| 1009 | {"control/system/../serial", opRead, false, "…on both prefixes"}, |
| 1010 | ``` |
| 1011 | In `TestStoreDoorGetServesOwnEstate`, seed two more objects with `seedObject` and add their keys to the loop that expects 200: |
| 1012 | ```go |
| 1013 | seedObject(t, store, "control/system/router/stable.json", []byte(`{"payload":"x"}`)) |
| 1014 | seedObject(t, store, "bootstrap/router/v1/carlos-linux-arm64", []byte("ELF")) |
| 1015 | ``` |
| 1016 | Update the comment at `storedoor_test.go:519` (the allowlist prose) to add "`control/system/<c>/<ch>.json`, `bootstrap/router/<v>/<asset>`". |
| 1017 | |
| 1018 | - [ ] **Step 2: Run to verify failure** |
| 1019 | |
| 1020 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -run 'TestStoreDoorACLMatrix|TestStoreDoorGetServesOwnEstate' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 1021 | Expected: FAIL on the four `want true` rows and the two new GET keys (404). |
| 1022 | |
| 1023 | - [ ] **Step 3: Implement** |
| 1024 | |
| 1025 | In `storedoor.go`, replace the `switch key` in `aclRead` with: |
| 1026 | ```go |
| 1027 | switch key { |
| 1028 | case "control/serial", "bootstrap/carlos": |
| 1029 | return true |
| 1030 | } |
| 1031 | if systemReadKey(key) { |
| 1032 | return true |
| 1033 | } |
| 1034 | ``` |
| 1035 | and add, below `aclRead`: |
| 1036 | ```go |
| 1037 | // systemReadKey is the two shapes the system-update spec (§3) grants every |
| 1038 | // fleet box: its platform channel pointer and a mirrored platform artifact. |
| 1039 | // Shapes, not prefixes — exactly three segments each, so nothing else that |
| 1040 | // ever lands under control/ or bootstrap/ is readable by a box. Listing is |
| 1041 | // not granted (aclDir is unchanged), and canonicalKey has already refused |
| 1042 | // any "." or ".." segment before this runs. |
| 1043 | func systemReadKey(key string) bool { |
| 1044 | if rest, ok := strings.CutPrefix(key, "control/system/"); ok { |
| 1045 | comp, file, ok := strings.Cut(rest, "/") |
| 1046 | return ok && comp != "" && file != "" && !strings.Contains(file, "/") && |
| 1047 | strings.HasSuffix(file, ".json") && len(file) > len(".json") |
| 1048 | } |
| 1049 | if rest, ok := strings.CutPrefix(key, "bootstrap/router/"); ok { |
| 1050 | ver, asset, ok := strings.Cut(rest, "/") |
| 1051 | return ok && ver != "" && asset != "" && !strings.Contains(asset, "/") |
| 1052 | } |
| 1053 | return false |
| 1054 | } |
| 1055 | ``` |
| 1056 | Extend the `aclRead` doc comment's allowlist enumeration (`:352-382`) with the two shapes and the sentence: "`bootstrap/carlos` stays until `update-platform-binary.sh` is retired (spec §10)." |
| 1057 | |
| 1058 | - [ ] **Step 4: Run the edge tests** |
| 1059 | |
| 1060 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 1061 | Expected: `ok`. |
| 1062 | |
| 1063 | - [ ] **Step 5: Commit** |
| 1064 | |
| 1065 | ```bash |
| 1066 | gofmt -l internal/edge/ && git add internal/edge/storedoor.go internal/edge/storedoor_test.go && git commit -q -F - <<'EOF' |
| 1067 | edge: the store door grants a box its platform pointer and artifact |
| 1068 | |
| 1069 | Two exact shapes under control/system/ and bootstrap/router/, no |
| 1070 | listing, no writes. Without the pointer grant a door-backed fleet box |
| 1071 | would never see a platform promote. |
| 1072 | |
| 1073 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 1074 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 1075 | EOF |
| 1076 | ``` |
| 1077 | |
| 1078 | --- |
| 1079 | ### Task 4: On-box state files (`internal/host/systemstate.go`) |
| 1080 | |
| 1081 | **Files:** |
| 1082 | - Create: `internal/host/systemstate.go` |
| 1083 | - Test: `internal/host/systemstate_test.go` |
| 1084 | |
| 1085 | **Interfaces:** |
| 1086 | - Consumes: nothing new. |
| 1087 | - Produces (package `host`, unexported unless noted): |
| 1088 | ```go |
| 1089 | const ( |
| 1090 | systemComponent = "router" |
| 1091 | systemRequestMarker = "system-update-requested" // touched under dataDir |
| 1092 | systemRequestEvery = time.Minute |
| 1093 | systemArtifactLimit = int64(96 << 20) |
| 1094 | systemOutcomeApplied = "applied"; systemOutcomeRolledBack = "rolled-back"; systemOutcomeRollbackFailed = "rollback-failed" |
| 1095 | systemOutcomeRefused = "refused"; systemOutcomeDeferred = "deferred"; systemOutcomeNoted = "noted" |
| 1096 | systemPhaseBackingUp = "backing-up"; systemPhaseInstalling = "installing"; systemPhaseInstalled = "installed" |
| 1097 | systemPhaseUnits = "units"; systemPhaseRestarted = "restarted"; systemPhaseGated = "gated" |
| 1098 | ) |
| 1099 | type systemRequest struct{ Version string; PromotedAt time.Time; NoOp bool; SeenAt time.Time } |
| 1100 | type systemResult struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time } |
| 1101 | type systemApplied struct{ Version, Channel string; At time.Time } |
| 1102 | type systemSeen map[string]time.Time |
| 1103 | type systemJournal struct{ Phase, Target, PrevBuild string; UnitsAdded, UnitsRemoved []string; StartedAt time.Time } |
| 1104 | func systemDataDir(dd string) string // <dd>/system/router |
| 1105 | func systemRequestPath(dd string) string // <dd>/system/router/requested.json |
| 1106 | func systemCachePath(dd, version string) string // <dd>/system/router/<version>/carlos |
| 1107 | func systemRootDir(optDir string) string // <optDir>/system/router |
| 1108 | func systemPrevPath(optDir string) string // <optDir>/carlos.prev |
| 1109 | func systemBinaryPath(optDir string) string // <optDir>/carlos |
| 1110 | func writeFileDurable(path string, data []byte, mode os.FileMode) error |
| 1111 | func writeJSONDurable(path string, v any, mode os.FileMode) error |
| 1112 | func readJSONFile(path string, v any) error // os.ErrNotExist passthrough |
| 1113 | func readSystemResult(optDir string) (*systemResult, error) |
| 1114 | func readSystemSeen(optDir string) (systemSeen, error) // missing file = empty map, nil error |
| 1115 | func readSystemApplied(optDir string) (*systemApplied, error) |
| 1116 | func readSystemJournal(optDir string) (*systemJournal, error) |
| 1117 | func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool |
| 1118 | func (r *systemResult) terminal() bool // applied|rolled-back|rollback-failed|refused |
| 1119 | func validSystemVersion(v string) bool // ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ |
| 1120 | ``` |
| 1121 | The root files are `result.json`, `seen.json`, `applied.json`, `journal.json` directly under `systemRootDir`. |
| 1122 | |
| 1123 | - [ ] **Step 1: Write the failing tests** |
| 1124 | |
| 1125 | ```go |
| 1126 | package host |
| 1127 | |
| 1128 | import ( |
| 1129 | "errors" |
| 1130 | "os" |
| 1131 | "path/filepath" |
| 1132 | "testing" |
| 1133 | "time" |
| 1134 | ) |
| 1135 | |
| 1136 | func TestSystemPathsAreDerivedNotTrusted(t *testing.T) { |
| 1137 | if got := systemCachePath("/data", "v1.2.3"); got != "/data/system/router/v1.2.3/carlos" { |
| 1138 | t.Fatalf("cache path = %s", got) |
| 1139 | } |
| 1140 | if got := systemRequestPath("/data"); got != "/data/system/router/requested.json" { |
| 1141 | t.Fatalf("request path = %s", got) |
| 1142 | } |
| 1143 | if got := systemRootDir("/opt/carlos"); got != "/opt/carlos/system/router" { |
| 1144 | t.Fatalf("root dir = %s", got) |
| 1145 | } |
| 1146 | for _, bad := range []string{"", ".", "..", "../x", "a/b", ".hidden", "v1 2", string(make([]byte, 65))} { |
| 1147 | if validSystemVersion(bad) { |
| 1148 | t.Errorf("validSystemVersion(%q) = true", bad) |
| 1149 | } |
| 1150 | } |
| 1151 | for _, good := range []string{"v0.18.0", "66587fa", "a", "release-2026.09.06_rc1"} { |
| 1152 | if !validSystemVersion(good) { |
| 1153 | t.Errorf("validSystemVersion(%q) = false", good) |
| 1154 | } |
| 1155 | } |
| 1156 | } |
| 1157 | |
| 1158 | func TestWriteFileDurableLeavesWholeFileOrNothing(t *testing.T) { |
| 1159 | dir := t.TempDir() |
| 1160 | p := filepath.Join(dir, "journal.json") |
| 1161 | if err := writeFileDurable(p, []byte("one"), 0o644); err != nil { |
| 1162 | t.Fatal(err) |
| 1163 | } |
| 1164 | if err := writeFileDurable(p, []byte("two"), 0o644); err != nil { |
| 1165 | t.Fatal(err) |
| 1166 | } |
| 1167 | b, _ := os.ReadFile(p) |
| 1168 | if string(b) != "two" { |
| 1169 | t.Fatalf("content = %q", b) |
| 1170 | } |
| 1171 | if _, err := os.Stat(p + ".tmp"); !errors.Is(err, os.ErrNotExist) { |
| 1172 | t.Fatal("temp left behind") |
| 1173 | } |
| 1174 | st, _ := os.Stat(p) |
| 1175 | if st.Mode().Perm() != 0o644 { |
| 1176 | t.Fatalf("mode = %v", st.Mode().Perm()) |
| 1177 | } |
| 1178 | // A directory that does not exist is an error, not a silent no-op. |
| 1179 | if err := writeFileDurable(filepath.Join(dir, "no", "such", "f"), []byte("x"), 0o644); err == nil { |
| 1180 | t.Fatal("wrote into a missing directory") |
| 1181 | } |
| 1182 | } |
| 1183 | |
| 1184 | func TestSystemStateReadersRoundTrip(t *testing.T) { |
| 1185 | opt := t.TempDir() |
| 1186 | root := systemRootDir(opt) |
| 1187 | if err := os.MkdirAll(root, 0o755); err != nil { |
| 1188 | t.Fatal(err) |
| 1189 | } |
| 1190 | if r, err := readSystemResult(opt); err != nil || r != nil { |
| 1191 | t.Fatalf("missing result: %v %v", r, err) |
| 1192 | } |
| 1193 | seen, err := readSystemSeen(opt) |
| 1194 | if err != nil || len(seen) != 0 { |
| 1195 | t.Fatalf("missing seen: %v %v", seen, err) |
| 1196 | } |
| 1197 | at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC) |
| 1198 | res := systemResult{Channel: "stable", Version: "v1", PromotedAt: at, Outcome: systemOutcomeRolledBack, Reason: "gate", At: at} |
| 1199 | if err := writeJSONDurable(filepath.Join(root, "result.json"), res, 0o644); err != nil { |
| 1200 | t.Fatal(err) |
| 1201 | } |
| 1202 | got, err := readSystemResult(opt) |
| 1203 | if err != nil || !got.matches("stable", "v1", at) || !got.terminal() { |
| 1204 | t.Fatalf("result = %+v err=%v", got, err) |
| 1205 | } |
| 1206 | if got.matches("canary", "v1", at) || got.matches("stable", "v2", at) || got.matches("stable", "v1", at.Add(time.Second)) { |
| 1207 | t.Fatal("matches ignores part of the tuple") |
| 1208 | } |
| 1209 | got.Outcome = systemOutcomeDeferred |
| 1210 | if got.terminal() { |
| 1211 | t.Fatal("deferred is not terminal") |
| 1212 | } |
| 1213 | got.Outcome = systemOutcomeNoted |
| 1214 | if got.terminal() { |
| 1215 | t.Fatal("noted is not terminal for the leg: the version may still differ next time") |
| 1216 | } |
| 1217 | seen["stable"] = at |
| 1218 | if err := writeJSONDurable(filepath.Join(root, "seen.json"), seen, 0o644); err != nil { |
| 1219 | t.Fatal(err) |
| 1220 | } |
| 1221 | seen2, _ := readSystemSeen(opt) |
| 1222 | if !seen2["stable"].Equal(at) { |
| 1223 | t.Fatalf("seen = %v", seen2) |
| 1224 | } |
| 1225 | j := systemJournal{Phase: systemPhaseInstalled, Target: "v1", PrevBuild: "v0", UnitsAdded: []string{"a.path"}, StartedAt: at} |
| 1226 | if err := writeJSONDurable(filepath.Join(root, "journal.json"), j, 0o600); err != nil { |
| 1227 | t.Fatal(err) |
| 1228 | } |
| 1229 | j2, err := readSystemJournal(opt) |
| 1230 | if err != nil || j2.Phase != systemPhaseInstalled || j2.UnitsAdded[0] != "a.path" { |
| 1231 | t.Fatalf("journal = %+v err=%v", j2, err) |
| 1232 | } |
| 1233 | } |
| 1234 | ``` |
| 1235 | |
| 1236 | - [ ] **Step 2: Run to verify failure** |
| 1237 | |
| 1238 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 1239 | Expected: undefined identifiers. |
| 1240 | |
| 1241 | - [ ] **Step 3: Implement `internal/host/systemstate.go`** |
| 1242 | |
| 1243 | ```go |
| 1244 | package host |
| 1245 | |
| 1246 | import ( |
| 1247 | "encoding/json" |
| 1248 | "errors" |
| 1249 | "os" |
| 1250 | "path/filepath" |
| 1251 | "regexp" |
| 1252 | "time" |
| 1253 | ) |
| 1254 | |
| 1255 | // The on-box state of the platform self-update (spec §4, §6). Two homes: |
| 1256 | // |
| 1257 | // <data>/system/router/ — written by the sandboxed agent: the |
| 1258 | // request and the artifact cache. Root |
| 1259 | // reads exactly one file here and trusts |
| 1260 | // nothing in it beyond a version string. |
| 1261 | // <opt>/system/router/ — written by root: journal, seen (replay |
| 1262 | // watermarks), applied (the .prev |
| 1263 | // authority), result (what the agent |
| 1264 | // publishes). Root-owned, world-readable. |
| 1265 | const ( |
| 1266 | systemComponent = "router" |
| 1267 | systemRequestMarker = "system-update-requested" |
| 1268 | systemRequestEvery = time.Minute |
| 1269 | systemArtifactLimit = int64(96 << 20) |
| 1270 | |
| 1271 | systemOutcomeApplied = "applied" |
| 1272 | systemOutcomeRolledBack = "rolled-back" |
| 1273 | systemOutcomeRollbackFailed = "rollback-failed" |
| 1274 | systemOutcomeRefused = "refused" |
| 1275 | systemOutcomeDeferred = "deferred" |
| 1276 | systemOutcomeNoted = "noted" |
| 1277 | |
| 1278 | systemPhaseBackingUp = "backing-up" |
| 1279 | systemPhaseInstalling = "installing" |
| 1280 | systemPhaseInstalled = "installed" |
| 1281 | systemPhaseUnits = "units" |
| 1282 | systemPhaseRestarted = "restarted" |
| 1283 | systemPhaseGated = "gated" |
| 1284 | ) |
| 1285 | |
| 1286 | // systemRequest is the agent's hint to root: which promote it saw. No |
| 1287 | // channel (root's own host.env says), no key, no sha, no path (root derives |
| 1288 | // the cache path from the version and takes key and sha from the pointer). |
| 1289 | type systemRequest struct { |
| 1290 | Version string `json:"version"` |
| 1291 | PromotedAt time.Time `json:"promoted_at"` |
| 1292 | NoOp bool `json:"noop"` |
| 1293 | SeenAt time.Time `json:"seen_at"` |
| 1294 | } |
| 1295 | |
| 1296 | // systemResult is root's answer to the last request, keyed by the REQUEST's |
| 1297 | // tuple so a refusal of one promote never suppresses the next. |
| 1298 | type systemResult struct { |
| 1299 | Channel string `json:"channel"` |
| 1300 | Version string `json:"version"` |
| 1301 | PromotedAt time.Time `json:"promoted_at"` |
| 1302 | Outcome string `json:"outcome"` |
| 1303 | Reason string `json:"reason,omitempty"` |
| 1304 | At time.Time `json:"at"` |
| 1305 | } |
| 1306 | |
| 1307 | func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool { |
| 1308 | return r != nil && r.Channel == channel && r.Version == version && r.PromotedAt.Equal(promotedAt) |
| 1309 | } |
| 1310 | |
| 1311 | // terminal: outcomes the leg must not retry for the same pointer. |
| 1312 | func (r *systemResult) terminal() bool { |
| 1313 | switch r.Outcome { |
| 1314 | case systemOutcomeApplied, systemOutcomeRolledBack, systemOutcomeRollbackFailed, systemOutcomeRefused: |
| 1315 | return true |
| 1316 | } |
| 1317 | return false |
| 1318 | } |
| 1319 | |
| 1320 | // systemApplied is the last build that PASSED A GATE on this box, forward or |
| 1321 | // rollback. It is the only thing that may authorise refreshing carlos.prev. |
| 1322 | type systemApplied struct { |
| 1323 | Version string `json:"version"` |
| 1324 | Channel string `json:"channel"` |
| 1325 | At time.Time `json:"at"` |
| 1326 | } |
| 1327 | |
| 1328 | // systemSeen is the replay watermark per channel: the newest PromotedAt root |
| 1329 | // has verified. Advanced on every verified pointer, including no-ops. |
| 1330 | type systemSeen map[string]time.Time |
| 1331 | |
| 1332 | // systemJournal is the open transaction. Written before every box-changing |
| 1333 | // step, removed last; a run that finds one recovers before anything else. |
| 1334 | type systemJournal struct { |
| 1335 | Phase string `json:"phase"` |
| 1336 | Target string `json:"target"` |
| 1337 | PrevBuild string `json:"prev_build"` |
| 1338 | UnitsAdded []string `json:"units_added,omitempty"` |
| 1339 | UnitsRemoved []string `json:"units_removed,omitempty"` |
| 1340 | StartedAt time.Time `json:"started_at"` |
| 1341 | } |
| 1342 | |
| 1343 | var systemVersionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`) |
| 1344 | |
| 1345 | // validSystemVersion is what makes the cache path derivable: no leading dot, |
| 1346 | // no separator, nothing that could be a traversal. |
| 1347 | func validSystemVersion(v string) bool { return systemVersionRe.MatchString(v) } |
| 1348 | |
| 1349 | func systemDataDir(dd string) string { return filepath.Join(dd, "system", systemComponent) } |
| 1350 | func systemRequestPath(dd string) string { |
| 1351 | return filepath.Join(systemDataDir(dd), "requested.json") |
| 1352 | } |
| 1353 | func systemCachePath(dd, version string) string { |
| 1354 | return filepath.Join(systemDataDir(dd), version, "carlos") |
| 1355 | } |
| 1356 | func systemRootDir(optDir string) string { return filepath.Join(optDir, "system", systemComponent) } |
| 1357 | func systemPrevPath(optDir string) string { return filepath.Join(optDir, "carlos.prev") } |
| 1358 | func systemBinaryPath(optDir string) string { return filepath.Join(optDir, "carlos") } |
| 1359 | |
| 1360 | // writeFileDurable: temp, fsync, rename, directory fsync. A crash leaves the |
| 1361 | // old file or the new one, never a torn one, and the rename itself is on |
| 1362 | // disk before the caller's next step depends on it. |
| 1363 | func writeFileDurable(path string, data []byte, mode os.FileMode) (err error) { |
| 1364 | tmp := path + ".tmp" |
| 1365 | f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode) |
| 1366 | if err != nil { |
| 1367 | return err |
| 1368 | } |
| 1369 | defer func() { |
| 1370 | if err != nil { |
| 1371 | f.Close() |
| 1372 | os.Remove(tmp) |
| 1373 | } |
| 1374 | }() |
| 1375 | if _, err = f.Write(data); err != nil { |
| 1376 | return err |
| 1377 | } |
| 1378 | if err = f.Chmod(mode); err != nil { |
| 1379 | return err |
| 1380 | } |
| 1381 | if err = f.Sync(); err != nil { |
| 1382 | return err |
| 1383 | } |
| 1384 | if err = f.Close(); err != nil { |
| 1385 | return err |
| 1386 | } |
| 1387 | if err = os.Rename(tmp, path); err != nil { |
| 1388 | return err |
| 1389 | } |
| 1390 | d, err := os.Open(filepath.Dir(path)) |
| 1391 | if err != nil { |
| 1392 | return err |
| 1393 | } |
| 1394 | defer d.Close() |
| 1395 | return d.Sync() |
| 1396 | } |
| 1397 | |
| 1398 | func writeJSONDurable(path string, v any, mode os.FileMode) error { |
| 1399 | b, err := json.MarshalIndent(v, "", " ") |
| 1400 | if err != nil { |
| 1401 | return err |
| 1402 | } |
| 1403 | return writeFileDurable(path, append(b, '\n'), mode) |
| 1404 | } |
| 1405 | |
| 1406 | // readJSONFile leaves os.ErrNotExist intact so callers can treat "no file" |
| 1407 | // as a state, not a failure. |
| 1408 | func readJSONFile(path string, v any) error { |
| 1409 | b, err := os.ReadFile(path) |
| 1410 | if err != nil { |
| 1411 | return err |
| 1412 | } |
| 1413 | return json.Unmarshal(b, v) |
| 1414 | } |
| 1415 | |
| 1416 | func readSystemResult(optDir string) (*systemResult, error) { |
| 1417 | var r systemResult |
| 1418 | err := readJSONFile(filepath.Join(systemRootDir(optDir), "result.json"), &r) |
| 1419 | if errors.Is(err, os.ErrNotExist) { |
| 1420 | return nil, nil |
| 1421 | } |
| 1422 | return &r, err |
| 1423 | } |
| 1424 | |
| 1425 | func readSystemSeen(optDir string) (systemSeen, error) { |
| 1426 | s := systemSeen{} |
| 1427 | err := readJSONFile(filepath.Join(systemRootDir(optDir), "seen.json"), &s) |
| 1428 | if errors.Is(err, os.ErrNotExist) { |
| 1429 | return s, nil |
| 1430 | } |
| 1431 | return s, err |
| 1432 | } |
| 1433 | |
| 1434 | func readSystemApplied(optDir string) (*systemApplied, error) { |
| 1435 | var a systemApplied |
| 1436 | err := readJSONFile(filepath.Join(systemRootDir(optDir), "applied.json"), &a) |
| 1437 | if errors.Is(err, os.ErrNotExist) { |
| 1438 | return nil, nil |
| 1439 | } |
| 1440 | return &a, err |
| 1441 | } |
| 1442 | |
| 1443 | func readSystemJournal(optDir string) (*systemJournal, error) { |
| 1444 | var j systemJournal |
| 1445 | err := readJSONFile(filepath.Join(systemRootDir(optDir), "journal.json"), &j) |
| 1446 | if errors.Is(err, os.ErrNotExist) { |
| 1447 | return nil, nil |
| 1448 | } |
| 1449 | return &j, err |
| 1450 | } |
| 1451 | ``` |
| 1452 | |
| 1453 | - [ ] **Step 4: Run the tests** |
| 1454 | |
| 1455 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 1456 | Expected: PASS. |
| 1457 | |
| 1458 | - [ ] **Step 5: Commit** |
| 1459 | |
| 1460 | ```bash |
| 1461 | gofmt -l internal/host/ && git add internal/host/systemstate.go internal/host/systemstate_test.go && git commit -q -F - <<'EOF' |
| 1462 | host: the on-box state of a platform self-update |
| 1463 | |
| 1464 | Request (agent-written hint), result, seen watermarks, applied, and |
| 1465 | the journal, with durable writes: temp, fsync, rename, directory |
| 1466 | fsync. Cache paths are derived from a validated version string. |
| 1467 | |
| 1468 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 1469 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 1470 | EOF |
| 1471 | ``` |
| 1472 | |
| 1473 | --- |
| 1474 | |
| 1475 | ### Task 5: The heartbeat learns platform, channel and last apply outcome |
| 1476 | |
| 1477 | **Files:** |
| 1478 | - Modify: `internal/status/snapshot.go` — `Probes` (~`:98`, next to `Release`), `BoxMetrics` (~`:158`), `CollectBox` (~`:404`, where `Release: p.Release` is copied), the emptiness guard (~`:543`) |
| 1479 | - Modify: `internal/host/agent.go` `statusLoop` (~`:2318`, the `Release: version.Build,` line) |
| 1480 | - Create: `internal/host/systemprobe.go` |
| 1481 | - Test: `internal/status/snapshot_test.go` (add), `internal/host/systemprobe_test.go` |
| 1482 | |
| 1483 | **Interfaces:** |
| 1484 | - Consumes: `readSystemResult` (Task 4), `release.OptDir()`, `status.Probes`, `status.BoxMetrics`. |
| 1485 | - Produces: |
| 1486 | ```go |
| 1487 | // package status |
| 1488 | type SystemState struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time } |
| 1489 | // Probes gains: Platform string; SystemChannel string; System *SystemState |
| 1490 | // BoxMetrics gains (json): platform, system_channel, system |
| 1491 | // package host |
| 1492 | func systemChannel(getenv func(string) string) string // CARLOS_SYSTEM_CHANNEL, default "stable" |
| 1493 | func systemStateProbe(optDir string) *status.SystemState // nil when no result.json |
| 1494 | const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL" |
| 1495 | ``` |
| 1496 | |
| 1497 | - [ ] **Step 1: Write the failing tests** |
| 1498 | |
| 1499 | `internal/status/snapshot_test.go`, add: |
| 1500 | ```go |
| 1501 | func TestBoxMetricsCarryThePlatformSelfUpdateState(t *testing.T) { |
| 1502 | at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC) |
| 1503 | p := Probes{Release: "v1", Platform: "linux-arm64", SystemChannel: "canary", |
| 1504 | System: &SystemState{Channel: "canary", Version: "v2", PromotedAt: at, Outcome: "rolled-back", Reason: "gate", At: at}} |
| 1505 | b := CollectBox(nil, p, at) |
| 1506 | if b.Metrics == nil || b.Metrics.Platform != "linux-arm64" || b.Metrics.SystemChannel != "canary" || b.Metrics.System == nil || b.Metrics.System.Outcome != "rolled-back" { |
| 1507 | t.Fatalf("metrics = %+v", b.Metrics) |
| 1508 | } |
| 1509 | out, _ := json.Marshal(b) |
| 1510 | for _, want := range []string{`"platform":"linux-arm64"`, `"system_channel":"canary"`, `"outcome":"rolled-back"`} { |
| 1511 | if !strings.Contains(string(out), want) { |
| 1512 | t.Errorf("json lacks %s: %s", want, out) |
| 1513 | } |
| 1514 | } |
| 1515 | // A box with only the new fields is not "empty". |
| 1516 | b2 := CollectBox(nil, Probes{Platform: "linux-arm64"}, at) |
| 1517 | if b2.Metrics == nil { |
| 1518 | t.Fatal("platform alone must produce a metrics block") |
| 1519 | } |
| 1520 | // The public fleet feed still never carries the metrics block. |
| 1521 | f := BuildFleet([]Box{b}, at) |
| 1522 | fb, _ := json.Marshal(f) |
| 1523 | if strings.Contains(string(fb), "system_channel") || strings.Contains(string(fb), "rolled-back") { |
| 1524 | t.Fatalf("fleet.json leaked the private metrics: %s", fb) |
| 1525 | } |
| 1526 | } |
| 1527 | ``` |
| 1528 | (Check `BuildFleet`'s exact signature at `fleet.go:157` and adjust the call; it takes the box list and a time.) |
| 1529 | |
| 1530 | `internal/host/systemprobe_test.go`: |
| 1531 | ```go |
| 1532 | package host |
| 1533 | |
| 1534 | import ( |
| 1535 | "os" |
| 1536 | "path/filepath" |
| 1537 | "testing" |
| 1538 | "time" |
| 1539 | ) |
| 1540 | |
| 1541 | func TestSystemChannelDefaultsToStable(t *testing.T) { |
| 1542 | if got := systemChannel(func(string) string { return "" }); got != "stable" { |
| 1543 | t.Fatalf("default = %q", got) |
| 1544 | } |
| 1545 | if got := systemChannel(func(k string) string { |
| 1546 | if k == systemChannelEnv { |
| 1547 | return " canary " |
| 1548 | } |
| 1549 | return "" |
| 1550 | }); got != "canary" { |
| 1551 | t.Fatalf("env = %q", got) |
| 1552 | } |
| 1553 | } |
| 1554 | |
| 1555 | func TestSystemStateProbeReadsRootsResult(t *testing.T) { |
| 1556 | opt := t.TempDir() |
| 1557 | if got := systemStateProbe(opt); got != nil { |
| 1558 | t.Fatalf("no result.json → %+v, want nil", got) |
| 1559 | } |
| 1560 | root := systemRootDir(opt) |
| 1561 | os.MkdirAll(root, 0o755) |
| 1562 | at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC) |
| 1563 | writeJSONDurable(filepath.Join(root, "result.json"), systemResult{Channel: "stable", Version: "v3", PromotedAt: at, Outcome: "applied", At: at}, 0o644) |
| 1564 | got := systemStateProbe(opt) |
| 1565 | if got == nil || got.Version != "v3" || got.Outcome != "applied" || !got.PromotedAt.Equal(at) { |
| 1566 | t.Fatalf("probe = %+v", got) |
| 1567 | } |
| 1568 | } |
| 1569 | ``` |
| 1570 | |
| 1571 | - [ ] **Step 2: Run to verify failure** |
| 1572 | |
| 1573 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -run 'TestBoxMetricsCarry|TestSystemChannel|TestSystemStateProbe' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 1574 | Expected: undefined fields/functions. |
| 1575 | |
| 1576 | - [ ] **Step 3: Implement** |
| 1577 | |
| 1578 | `internal/status/snapshot.go` — in `Probes`, after `Release string` (`:98`): |
| 1579 | ```go |
| 1580 | // Platform is runtime.GOOS-runtime.GOARCH, so the console knows which |
| 1581 | // artifacts a platform promote must mirror. SystemChannel is the box's |
| 1582 | // CARLOS_SYSTEM_CHANNEL, so the console knows which boxes a promote |
| 1583 | // reaches. System is root's last self-update outcome (spec §5, §6). |
| 1584 | Platform string |
| 1585 | SystemChannel string |
| 1586 | System *SystemState |
| 1587 | ``` |
| 1588 | Add the type near `BoxMetrics`: |
| 1589 | ```go |
| 1590 | // SystemState is the last platform self-update outcome root recorded on the |
| 1591 | // box, published so `carlos system status` can show a rolled-back box with |
| 1592 | // its reason within one heartbeat. |
| 1593 | type SystemState struct { |
| 1594 | Channel string `json:"channel"` |
| 1595 | Version string `json:"version"` |
| 1596 | PromotedAt time.Time `json:"promoted_at"` |
| 1597 | Outcome string `json:"outcome"` |
| 1598 | Reason string `json:"reason,omitempty"` |
| 1599 | At time.Time `json:"at"` |
| 1600 | } |
| 1601 | ``` |
| 1602 | In `BoxMetrics`, after `Release` (`:158`): |
| 1603 | ```go |
| 1604 | Platform string `json:"platform,omitempty"` // runtime.GOOS-runtime.GOARCH |
| 1605 | SystemChannel string `json:"system_channel,omitempty"` // CARLOS_SYSTEM_CHANNEL |
| 1606 | System *SystemState `json:"system,omitempty"` // last self-update outcome |
| 1607 | ``` |
| 1608 | In `CollectBox`, where `Release: p.Release` is copied into the metrics block, add `Platform: p.Platform, SystemChannel: p.SystemChannel, System: p.System`. In the emptiness guard (`:543`), add `|| p.Platform != "" || p.SystemChannel != "" || p.System != nil`. |
| 1609 | |
| 1610 | `internal/host/systemprobe.go`: |
| 1611 | ```go |
| 1612 | package host |
| 1613 | |
| 1614 | import ( |
| 1615 | "strings" |
| 1616 | |
| 1617 | "github.com/carlosframework/platform/internal/status" |
| 1618 | ) |
| 1619 | |
| 1620 | // systemChannelEnv is the box's platform channel: canary or stable. Set in |
| 1621 | // host.env, read through the getenv seam like every other CARLOS_* value; |
| 1622 | // the ROOT apply reads it from its own unit environment, never from the |
| 1623 | // agent's request (spec §6 step 2). |
| 1624 | const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL" |
| 1625 | |
| 1626 | func systemChannel(getenv func(string) string) string { |
| 1627 | if c := strings.TrimSpace(getenv(systemChannelEnv)); c != "" { |
| 1628 | return c |
| 1629 | } |
| 1630 | return "stable" |
| 1631 | } |
| 1632 | |
| 1633 | // systemStateProbe is root's result.json as the heartbeat carries it, or nil |
| 1634 | // when this box has never been asked to update. |
| 1635 | func systemStateProbe(optDir string) *status.SystemState { |
| 1636 | r, err := readSystemResult(optDir) |
| 1637 | if err != nil || r == nil { |
| 1638 | return nil |
| 1639 | } |
| 1640 | return &status.SystemState{Channel: r.Channel, Version: r.Version, PromotedAt: r.PromotedAt, Outcome: r.Outcome, Reason: r.Reason, At: r.At} |
| 1641 | } |
| 1642 | ``` |
| 1643 | `internal/host/agent.go` `statusLoop`, next to `Release: version.Build,` (`:2318`): |
| 1644 | ```go |
| 1645 | Platform: runtime.GOOS + "-" + runtime.GOARCH, |
| 1646 | SystemChannel: systemChannel(os.Getenv), |
| 1647 | System: systemStateProbe(release.OptDir()), |
| 1648 | ``` |
| 1649 | (add `"runtime"` to the imports). Add `CARLOS_SYSTEM_CHANNEL` to the env contract in the package doc comment at the top of `agent.go`, one line: `CARLOS_SYSTEM_CHANNEL — platform channel this box follows for self-update, canary|stable (default stable)`. |
| 1650 | |
| 1651 | - [ ] **Step 4: Run the tests** |
| 1652 | |
| 1653 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 1654 | Expected: `ok` for both. |
| 1655 | |
| 1656 | - [ ] **Step 5: Commit** |
| 1657 | |
| 1658 | ```bash |
| 1659 | gofmt -l internal/ && git add internal/status/snapshot.go internal/status/snapshot_test.go internal/host/agent.go internal/host/systemprobe.go internal/host/systemprobe_test.go && git commit -q -F - <<'EOF' |
| 1660 | status: a box reports its platform, its system channel and its last self-update outcome |
| 1661 | |
| 1662 | Three optional fields on the private metrics block; the public fleet |
| 1663 | feed does not carry them, as before. |
| 1664 | |
| 1665 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 1666 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 1667 | EOF |
| 1668 | ``` |
| 1669 | |
| 1670 | --- |
| 1671 | |
| 1672 | ### Task 6: The three units, `enabledUnits`, `-plan-json`, the path probe |
| 1673 | |
| 1674 | **Files:** |
| 1675 | - Create: `internal/host/units/carlos-system-update.path`, `internal/host/units/carlos-system-update.service`, `internal/host/units/carlos-system-recover.service` |
| 1676 | - Modify: `internal/host/bootstrap.go` (`enabledUnits` `:72`, `RunBootstrap` flags `:115-132`, the staging print block `:327-341`) |
| 1677 | - Modify: `internal/host/probes.go` (add a fourth `*Active()`) |
| 1678 | - Modify: `internal/host/host_test.go` (`expectedUnits` `:29-46`; `TestProbedUnitsAreTheOnesBootstrapArms` `:1129`) |
| 1679 | - Test: `internal/host/host_test.go` (add `TestBootstrapPlanJSON`) |
| 1680 | |
| 1681 | **Interfaces:** |
| 1682 | - Consumes: `unitsFS`, `unitNames`, `systemctlPlan`, `bootstrap(root, consumers)`, `parseIsActive`. |
| 1683 | - Produces: |
| 1684 | ```go |
| 1685 | // package host |
| 1686 | var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"} |
| 1687 | type bootstrapPlan struct{ Enabled []string `json:"enabled"`; Inert []string `json:"inert"` } |
| 1688 | func currentBootstrapPlan() bootstrapPlan |
| 1689 | // `carlos bootstrap -root <dir> -plan-json` prints ONLY the JSON plan on stdout |
| 1690 | const systemUpdatePathUnit = "carlos-system-update.path" |
| 1691 | func systemUpdatePathActive() *bool |
| 1692 | ``` |
| 1693 | |
| 1694 | - [ ] **Step 1: Add the failing tests** |
| 1695 | |
| 1696 | In `host_test.go`, add the three names to `expectedUnits` (keep it alphabetical): |
| 1697 | ```go |
| 1698 | "carlos-system-recover.service", |
| 1699 | "carlos-system-update.path", |
| 1700 | "carlos-system-update.service", |
| 1701 | ``` |
| 1702 | and add: |
| 1703 | ```go |
| 1704 | func TestBootstrapPlanJSONNamesWhatALiveRunArms(t *testing.T) { |
| 1705 | p := currentBootstrapPlan() |
| 1706 | want := map[string]bool{"carlos-edge.service": true, "carlos-adopt.timer": true, "carlos-adopt.path": true, |
| 1707 | "carlos-stage.path": true, "carlos-restart.path": true, "carlos-system-update.path": true, "carlos-system-recover.service": true} |
| 1708 | got := map[string]bool{} |
| 1709 | for _, u := range p.Enabled { |
| 1710 | got[u] = true |
| 1711 | } |
| 1712 | for u := range want { |
| 1713 | if !got[u] { |
| 1714 | t.Errorf("plan lacks %s", u) |
| 1715 | } |
| 1716 | } |
| 1717 | if got["carlos-system-update.service"] { |
| 1718 | t.Error("the path-triggered oneshot must not be enabled: it is started by its .path") |
| 1719 | } |
| 1720 | inert := map[string]bool{} |
| 1721 | for _, u := range p.Inert { |
| 1722 | inert[u] = true |
| 1723 | } |
| 1724 | if !inert["carlos-restore-verify.timer"] || !inert["carlos-hibernate-verify.timer"] { |
| 1725 | t.Errorf("inert = %v", p.Inert) |
| 1726 | } |
| 1727 | // And systemctlPlan never starts a service, whatever is enabled. |
| 1728 | for _, args := range systemctlPlan(p.Enabled) { |
| 1729 | if args[0] == "start" { |
| 1730 | for _, u := range args[1:] { |
| 1731 | if strings.HasSuffix(u, ".service") { |
| 1732 | t.Errorf("plan starts a service: %v", args) |
| 1733 | } |
| 1734 | } |
| 1735 | } |
| 1736 | } |
| 1737 | } |
| 1738 | |
| 1739 | func TestSystemUnitsAreShapedLikeTheStageRelay(t *testing.T) { |
| 1740 | read := func(name string) string { |
| 1741 | b, err := unitsFS.ReadFile("units/" + name) |
| 1742 | if err != nil { |
| 1743 | t.Fatal(err) |
| 1744 | } |
| 1745 | return string(b) |
| 1746 | } |
| 1747 | p := read("carlos-system-update.path") |
| 1748 | for _, want := range []string{"PathChanged=/data/system-update-requested", "Unit=carlos-system-update.service", "WantedBy=multi-user.target"} { |
| 1749 | if !strings.Contains(p, want) { |
| 1750 | t.Errorf("path unit lacks %q", want) |
| 1751 | } |
| 1752 | } |
| 1753 | u := read("carlos-system-update.service") |
| 1754 | for _, want := range []string{"Type=oneshot", "StartLimitIntervalSec=0", "TimeoutStartSec=15min", "OnFailure=carlos-system-recover.service", |
| 1755 | "EnvironmentFile=-/etc/carlos/host.env", "system apply", "carlos.prev", "MemoryMax=256M"} { |
| 1756 | if !strings.Contains(u, want) { |
| 1757 | t.Errorf("update unit lacks %q", want) |
| 1758 | } |
| 1759 | } |
| 1760 | for _, never := range []string{"ProtectSystem", "User="} { |
| 1761 | if strings.Contains(u, never) { |
| 1762 | t.Errorf("update unit must run as root with /opt writable; found %q", never) |
| 1763 | } |
| 1764 | } |
| 1765 | r := read("carlos-system-recover.service") |
| 1766 | for _, want := range []string{"Type=oneshot", "Before=carlos-edge.service", "WantedBy=multi-user.target", "TimeoutStartSec=10min", "system apply --recover", "carlos.prev"} { |
| 1767 | if !strings.Contains(r, want) { |
| 1768 | t.Errorf("recover unit lacks %q", want) |
| 1769 | } |
| 1770 | } |
| 1771 | } |
| 1772 | ``` |
| 1773 | Update `TestProbedUnitsAreTheOnesBootstrapArms` so its probed set includes `systemUpdatePathUnit` (read the test; it compares probe constants against `enabledUnits`). |
| 1774 | |
| 1775 | - [ ] **Step 2: Run to verify failure** |
| 1776 | |
| 1777 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestBootstrap|TestSystemUnits|TestProbedUnits' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 1778 | Expected: FAIL (missing units, undefined `currentBootstrapPlan`). |
| 1779 | |
| 1780 | - [ ] **Step 3: Write the units** |
| 1781 | |
| 1782 | `internal/host/units/carlos-system-update.path`: |
| 1783 | ```ini |
| 1784 | # The relay from the sandboxed agent to a root apply of a new platform |
| 1785 | # binary (spec §4). The agent (User=carlos, ProtectSystem=strict) can fetch |
| 1786 | # and verify the artifact but cannot write /opt or /etc; it touches this |
| 1787 | # marker and root does the rest. Same shape as carlos-stage.path. |
| 1788 | [Unit] |
| 1789 | Description=CARLOS platform self-update on agent request |
| 1790 | |
| 1791 | [Path] |
| 1792 | PathChanged=/data/system-update-requested |
| 1793 | Unit=carlos-system-update.service |
| 1794 | |
| 1795 | [Install] |
| 1796 | WantedBy=multi-user.target |
| 1797 | ``` |
| 1798 | |
| 1799 | `internal/host/units/carlos-system-update.service`: |
| 1800 | ```ini |
| 1801 | # Root apply of a platform self-update (spec §6): re-verify the pointer, |
| 1802 | # copy the candidate into a root-owned dir, journal, swap the binary and |
| 1803 | # units, restart the edge, gate it, roll back to carlos.prev on failure. |
| 1804 | # |
| 1805 | # ExecStart goes through one line of shell so that when a journal is open |
| 1806 | # (a previous apply was cut off) and carlos.prev exists, the RECOVERY runs |
| 1807 | # from carlos.prev rather than from the candidate that did not work. |
| 1808 | # |
| 1809 | # No ProtectSystem, no User=: it must write /opt/carlos and |
| 1810 | # /etc/systemd/system. StartLimitIntervalSec=0 because the path unit can |
| 1811 | # fire in bursts. TimeoutStartSec bounds a hung apply; OnFailure= makes a |
| 1812 | # killed or timed-out apply recover without waiting for a touch. |
| 1813 | [Unit] |
| 1814 | Description=carlos platform self-update (root apply) |
| 1815 | After=network-online.target |
| 1816 | Wants=network-online.target |
| 1817 | StartLimitIntervalSec=0 |
| 1818 | OnFailure=carlos-system-recover.service |
| 1819 | |
| 1820 | [Service] |
| 1821 | Type=oneshot |
| 1822 | TimeoutStartSec=15min |
| 1823 | EnvironmentFile=-/etc/carlos/host.env |
| 1824 | ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply; fi; exec /opt/carlos/carlos system apply' |
| 1825 | |
| 1826 | # --- hardening (the carlos-stage.service set) --- |
| 1827 | ProtectHome=yes |
| 1828 | PrivateTmp=yes |
| 1829 | PrivateDevices=yes |
| 1830 | ProtectClock=yes |
| 1831 | ProtectHostname=yes |
| 1832 | ProtectKernelTunables=yes |
| 1833 | ProtectKernelModules=yes |
| 1834 | ProtectKernelLogs=yes |
| 1835 | RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX |
| 1836 | RestrictNamespaces=yes |
| 1837 | RestrictRealtime=yes |
| 1838 | LockPersonality=yes |
| 1839 | MemoryDenyWriteExecute=yes |
| 1840 | MemoryMax=256M |
| 1841 | TasksMax=64 |
| 1842 | ``` |
| 1843 | |
| 1844 | `internal/host/units/carlos-system-recover.service`: |
| 1845 | ```ini |
| 1846 | # Recovery of an interrupted platform self-update (spec §6, "every run |
| 1847 | # begins with recovery"). Runs once per boot, BEFORE the edge, and on the |
| 1848 | # update oneshot's failure. It restores files only — carlos.prev over |
| 1849 | # carlos, the previous unit set — and never restarts or gates the edge |
| 1850 | # from inside a unit the edge is ordered after: at boot the edge's own |
| 1851 | # start IS the restart. At runtime (edge already started this boot) it |
| 1852 | # issues a non-blocking restart and exits. |
| 1853 | [Unit] |
| 1854 | Description=carlos platform self-update recovery |
| 1855 | Before=carlos-edge.service |
| 1856 | DefaultDependencies=yes |
| 1857 | |
| 1858 | [Service] |
| 1859 | Type=oneshot |
| 1860 | TimeoutStartSec=10min |
| 1861 | EnvironmentFile=-/etc/carlos/host.env |
| 1862 | ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply --recover; fi; exec /opt/carlos/carlos system apply --recover' |
| 1863 | |
| 1864 | ProtectHome=yes |
| 1865 | PrivateTmp=yes |
| 1866 | PrivateDevices=yes |
| 1867 | ProtectClock=yes |
| 1868 | ProtectHostname=yes |
| 1869 | ProtectKernelTunables=yes |
| 1870 | ProtectKernelModules=yes |
| 1871 | ProtectKernelLogs=yes |
| 1872 | RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX |
| 1873 | RestrictNamespaces=yes |
| 1874 | RestrictRealtime=yes |
| 1875 | LockPersonality=yes |
| 1876 | MemoryDenyWriteExecute=yes |
| 1877 | MemoryMax=256M |
| 1878 | TasksMax=64 |
| 1879 | |
| 1880 | [Install] |
| 1881 | WantedBy=multi-user.target |
| 1882 | ``` |
| 1883 | |
| 1884 | - [ ] **Step 4: `bootstrap.go` changes** |
| 1885 | |
| 1886 | Replace `enabledUnits` (`:72`): |
| 1887 | ```go |
| 1888 | var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path", |
| 1889 | "carlos-system-update.path", "carlos-system-recover.service"} |
| 1890 | |
| 1891 | // inertUnits are installed everywhere and deliberately never armed by |
| 1892 | // bootstrap or by a self-update apply: restore-verify until an operator |
| 1893 | // stages litestream, hibernate-verify by long-standing choice. |
| 1894 | var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"} |
| 1895 | |
| 1896 | // bootstrapPlan is what `bootstrap -root <dir> -plan-json` prints: the |
| 1897 | // arming list of THIS binary, so a self-update apply arms what the new |
| 1898 | // binary's bootstrap would arm and never a list an older binary remembers. |
| 1899 | type bootstrapPlan struct { |
| 1900 | Enabled []string `json:"enabled"` |
| 1901 | Inert []string `json:"inert"` |
| 1902 | } |
| 1903 | |
| 1904 | func currentBootstrapPlan() bootstrapPlan { |
| 1905 | return bootstrapPlan{Enabled: append([]string(nil), enabledUnits...), Inert: append([]string(nil), inertUnits...)} |
| 1906 | } |
| 1907 | ``` |
| 1908 | In `RunBootstrap`, add the flag after `offcloud`: |
| 1909 | ```go |
| 1910 | planJSON := fs.Bool("plan-json", false, "with -root: print the arming plan as JSON on stdout instead of the human summary") |
| 1911 | ``` |
| 1912 | and pass it through: change `bootstrap(*root, consumers)` to `bootstrapWith(*root, consumers, *planJSON)`, where `bootstrap(root, consumers)` becomes `func bootstrap(root string, consumers []string) error { return bootstrapWith(root, consumers, false) }` and `bootstrapWith` is the old body with the staging print block (`:327-341`) wrapped: |
| 1913 | ```go |
| 1914 | if !live { |
| 1915 | if planJSON { |
| 1916 | return json.NewEncoder(os.Stdout).Encode(currentBootstrapPlan()) |
| 1917 | } |
| 1918 | fmt.Printf("bootstrap: staged under %s — on a real box (--root /) it would also:\n", root) |
| 1919 | …existing prints… |
| 1920 | return nil |
| 1921 | } |
| 1922 | ``` |
| 1923 | All the `fmt.Printf(" wrote %s\n", path)` / `" created %s\n"` lines in the staging path must go to **stderr** when `planJSON` is set, or the JSON is not alone on stdout: introduce `out := io.Writer(os.Stdout); if planJSON { out = os.Stderr }` at the top of `bootstrapWith` and change those `fmt.Printf` to `fmt.Fprintf(out, …)`. (`-plan-json` with `-root /` is refused: `log.Fatal("bootstrap: -plan-json is for a staging root")`.) |
| 1924 | |
| 1925 | `internal/host/probes.go`, alongside `restartPathActive`: |
| 1926 | ```go |
| 1927 | const systemUpdatePathUnit = "carlos-system-update.path" |
| 1928 | |
| 1929 | // systemUpdatePathActive: is the self-update relay armed? nil = unknown. |
| 1930 | func systemUpdatePathActive() *bool { |
| 1931 | out, _ := exec.Command("systemctl", "is-active", systemUpdatePathUnit).CombinedOutput() |
| 1932 | return parseIsActive(string(out)) |
| 1933 | } |
| 1934 | ``` |
| 1935 | (copy the exact shape of `restartPathActive` at `probes.go:82`, including how it runs `systemctl`). |
| 1936 | |
| 1937 | - [ ] **Step 5: Run the tests** |
| 1938 | |
| 1939 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 1940 | Expected: `ok`. Then, as a real check of the flag: `go run . bootstrap -root /tmp/c/stage -plan-json 2>/dev/null | head -3` prints one JSON object with `enabled` and `inert`. |
| 1941 | |
| 1942 | - [ ] **Step 6: Commit** |
| 1943 | |
| 1944 | ```bash |
| 1945 | gofmt -l internal/host/ && git add internal/host/units/carlos-system-update.path internal/host/units/carlos-system-update.service internal/host/units/carlos-system-recover.service internal/host/bootstrap.go internal/host/probes.go internal/host/host_test.go && git commit -q -F - <<'EOF' |
| 1946 | host: the self-update relay, apply and recover units, and a JSON arming plan |
| 1947 | |
| 1948 | bootstrap -root <dir> -plan-json prints this binary's enabledUnits and |
| 1949 | inert list, so an apply arms what the NEW binary's bootstrap would arm. |
| 1950 | Both oneshots exec carlos.prev when a journal is open. |
| 1951 | |
| 1952 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 1953 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 1954 | EOF |
| 1955 | ``` |
| 1956 | |
| 1957 | --- |
| 1958 | ### Task 7: The agent leg (`internal/host/systemleg.go`) |
| 1959 | |
| 1960 | **Files:** |
| 1961 | - Create: `internal/host/systemleg.go` |
| 1962 | - Modify: `internal/host/agent.go` — `fastDeps` (`:1077`), `fastTick` (`:1330`; the serial-move branch `:1352-1367` and the unconditional part before `fastMembers` `:1370`), `fastConvergeLoop` (`:1434-1459`) |
| 1963 | - Test: `internal/host/systemleg_test.go`, `internal/host/agent_test.go` (`fastFixture.deps` at `:1098` may leave the new field nil) |
| 1964 | |
| 1965 | **Interfaces:** |
| 1966 | - Consumes: `release.GetSystemPointer`, `release.FetchToFile`, `release.SystemComponentRouter`, `release.OptDir`, `systemChannel`, `systemRequest`, `readSystemResult`, `readSystemSeen`, `systemCachePath`, `systemRequestPath`, `systemRequestMarker`, `systemRequestEvery`, `systemArtifactLimit`, `validSystemVersion`, `version.Build`. |
| 1967 | - Produces: |
| 1968 | ```go |
| 1969 | type systemLeg struct { … } |
| 1970 | func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg |
| 1971 | func (l *systemLeg) observe(ctx context.Context) // serial move / hourly: read pointer, decide, set pending |
| 1972 | func (l *systemLeg) tick(ctx context.Context, now time.Time) // every 2 s: fire pending, re-touch, hourly re-observe |
| 1973 | // fastDeps gains: systemLeg *systemLeg |
| 1974 | ``` |
| 1975 | Test seams on the struct: `now func() time.Time`, `jitter func(time.Duration) time.Duration`, `build string`, `platform string`. |
| 1976 | |
| 1977 | - [ ] **Step 1: Write the failing tests** |
| 1978 | |
| 1979 | ```go |
| 1980 | package host |
| 1981 | |
| 1982 | import ( |
| 1983 | "context" |
| 1984 | "crypto/sha256" |
| 1985 | "encoding/hex" |
| 1986 | "os" |
| 1987 | "path/filepath" |
| 1988 | "strings" |
| 1989 | "testing" |
| 1990 | "time" |
| 1991 | |
| 1992 | "github.com/carlosframework/platform/internal/release" |
| 1993 | ) |
| 1994 | |
| 1995 | type legFixture struct { |
| 1996 | t *testing.T |
| 1997 | store *release.DirStore |
| 1998 | priv ed25519.PrivateKey |
| 1999 | leg *systemLeg |
| 2000 | dd string |
| 2001 | opt string |
| 2002 | now time.Time |
| 2003 | } |
| 2004 | |
| 2005 | func newLegFixture(t *testing.T, channel, build string) *legFixture { |
| 2006 | t.Helper() |
| 2007 | priv, pub, _ := ed25519.GenerateKey(nil) |
| 2008 | f := &legFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, dd: t.TempDir(), opt: t.TempDir(), |
| 2009 | now: time.Date(2026, 9, 6, 10, 0, 0, 0, time.UTC)} |
| 2010 | getenv := func(k string) string { |
| 2011 | if k == systemChannelEnv { |
| 2012 | return channel |
| 2013 | } |
| 2014 | return "" |
| 2015 | } |
| 2016 | f.leg = newSystemLeg(f.store, pub, getenv, f.dd, f.opt) |
| 2017 | f.leg.build = build |
| 2018 | f.leg.platform = "linux-arm64" |
| 2019 | f.leg.now = func() time.Time { return f.now } |
| 2020 | f.leg.jitter = func(d time.Duration) time.Duration { return d / 2 } // deterministic: half the spread |
| 2021 | return f |
| 2022 | } |
| 2023 | |
| 2024 | func (f *legFixture) promote(version string, body []byte, spread time.Duration) { |
| 2025 | f.t.Helper() |
| 2026 | sum := sha256.Sum256(body) |
| 2027 | rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{ |
| 2028 | "linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}} |
| 2029 | if _, err := f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, ""); err != nil { |
| 2030 | f.t.Fatal(err) |
| 2031 | } |
| 2032 | if _, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.leg.channel, rel, |
| 2033 | release.SystemPromoteOptions{Spread: spread}); err != nil { |
| 2034 | f.t.Fatal(err) |
| 2035 | } |
| 2036 | } |
| 2037 | |
| 2038 | func (f *legFixture) marker() (os.FileInfo, error) { return os.Stat(filepath.Join(f.dd, systemRequestMarker)) } |
| 2039 | |
| 2040 | func (f *legFixture) request() *systemRequest { |
| 2041 | var r systemRequest |
| 2042 | if err := readJSONFile(systemRequestPath(f.dd), &r); err != nil { |
| 2043 | return nil |
| 2044 | } |
| 2045 | return &r |
| 2046 | } |
| 2047 | |
| 2048 | func TestSystemLegDoesNothingWithoutAPointer(t *testing.T) { |
| 2049 | f := newLegFixture(t, "stable", "v1") |
| 2050 | f.leg.observe(context.Background()) |
| 2051 | f.leg.tick(context.Background(), f.now.Add(time.Hour)) |
| 2052 | if _, err := f.marker(); err == nil { |
| 2053 | t.Fatal("marker touched with no pointer") |
| 2054 | } |
| 2055 | } |
| 2056 | |
| 2057 | func TestSystemLegFetchesVerifiesAndRequestsAfterTheSpread(t *testing.T) { |
| 2058 | f := newLegFixture(t, "stable", "v1") |
| 2059 | body := []byte("ELF v2") |
| 2060 | f.promote("v2", body, 10*time.Minute) |
| 2061 | f.leg.observe(context.Background()) |
| 2062 | if _, err := f.marker(); err == nil { |
| 2063 | t.Fatal("requested before the spread elapsed") |
| 2064 | } |
| 2065 | f.leg.tick(context.Background(), f.now.Add(4*time.Minute)) // jitter = 5m |
| 2066 | if _, err := f.marker(); err == nil { |
| 2067 | t.Fatal("requested before the deadline") |
| 2068 | } |
| 2069 | f.leg.tick(context.Background(), f.now.Add(5*time.Minute+time.Second)) |
| 2070 | if _, err := f.marker(); err != nil { |
| 2071 | t.Fatal("marker not touched after the deadline") |
| 2072 | } |
| 2073 | got, _ := os.ReadFile(systemCachePath(f.dd, "v2")) |
| 2074 | if string(got) != "ELF v2" { |
| 2075 | t.Fatalf("cache = %q", got) |
| 2076 | } |
| 2077 | r := f.request() |
| 2078 | if r == nil || r.Version != "v2" || r.NoOp || r.PromotedAt.IsZero() { |
| 2079 | t.Fatalf("request = %+v", r) |
| 2080 | } |
| 2081 | raw, _ := os.ReadFile(systemRequestPath(f.dd)) |
| 2082 | for _, never := range []string{"channel", "key", "sha", "path"} { |
| 2083 | if strings.Contains(string(raw), `"`+never+`"`) { |
| 2084 | t.Errorf("request carries %q — root must not take that from the sandbox", never) |
| 2085 | } |
| 2086 | } |
| 2087 | } |
| 2088 | |
| 2089 | func TestSystemLegRetouchesUntilRootAnswers(t *testing.T) { |
| 2090 | f := newLegFixture(t, "stable", "v1") |
| 2091 | f.promote("v2", []byte("x"), 0) |
| 2092 | f.leg.observe(context.Background()) |
| 2093 | f.leg.tick(context.Background(), f.now) |
| 2094 | first, _ := f.marker() |
| 2095 | f.leg.tick(context.Background(), f.now.Add(30*time.Second)) |
| 2096 | same, _ := f.marker() |
| 2097 | if !same.ModTime().Equal(first.ModTime()) { |
| 2098 | t.Fatal("re-touched inside the minute throttle") |
| 2099 | } |
| 2100 | // The touch is a rewrite (PathChanged needs a change), so mtime moves. |
| 2101 | time.Sleep(20 * time.Millisecond) |
| 2102 | f.leg.tick(context.Background(), f.now.Add(61*time.Second)) |
| 2103 | later, _ := f.marker() |
| 2104 | if !later.ModTime().After(first.ModTime()) { |
| 2105 | t.Fatal("not re-touched after a minute with no result") |
| 2106 | } |
| 2107 | // Root answers: a terminal result for this tuple stops the re-touch and |
| 2108 | // the leg deletes its own cache. |
| 2109 | r := f.request() |
| 2110 | os.MkdirAll(systemRootDir(f.opt), 0o755) |
| 2111 | writeJSONDurable(filepath.Join(systemRootDir(f.opt), "result.json"), |
| 2112 | systemResult{Channel: "stable", Version: "v2", PromotedAt: r.PromotedAt, Outcome: systemOutcomeRolledBack, Reason: "gate", At: f.now}, 0o644) |
| 2113 | time.Sleep(20 * time.Millisecond) |
| 2114 | f.leg.tick(context.Background(), f.now.Add(3*time.Minute)) |
| 2115 | final, _ := f.marker() |
| 2116 | if !final.ModTime().Equal(later.ModTime()) { |
| 2117 | t.Fatal("re-touched after a terminal result") |
| 2118 | } |
| 2119 | if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil { |
| 2120 | t.Fatal("cache kept after a terminal result") |
| 2121 | } |
| 2122 | // And a later observe of the SAME pointer does nothing: refuse to repeat a failure. |
| 2123 | f.leg.observe(context.Background()) |
| 2124 | f.leg.tick(context.Background(), f.now.Add(10*time.Minute)) |
| 2125 | if f.leg.pending != nil { |
| 2126 | t.Fatal("re-requested a pointer root already rolled back") |
| 2127 | } |
| 2128 | } |
| 2129 | |
| 2130 | func TestSystemLegNoOpRequestFetchesNothing(t *testing.T) { |
| 2131 | f := newLegFixture(t, "stable", "v2") |
| 2132 | f.promote("v2", []byte("same"), 0) |
| 2133 | f.leg.observe(context.Background()) |
| 2134 | f.leg.tick(context.Background(), f.now) |
| 2135 | r := f.request() |
| 2136 | if r == nil || !r.NoOp { |
| 2137 | t.Fatalf("request = %+v, want a no-op", r) |
| 2138 | } |
| 2139 | if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil { |
| 2140 | t.Fatal("a no-op must not download the artifact") |
| 2141 | } |
| 2142 | // Once root has seen it (seen.json advanced to this PromotedAt), nothing more. |
| 2143 | os.MkdirAll(systemRootDir(f.opt), 0o755) |
| 2144 | writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": r.PromotedAt}, 0o644) |
| 2145 | f.leg.pending = nil |
| 2146 | os.Remove(filepath.Join(f.dd, systemRequestMarker)) |
| 2147 | f.leg.observe(context.Background()) |
| 2148 | f.leg.tick(context.Background(), f.now.Add(2*time.Minute)) |
| 2149 | if _, err := f.marker(); err == nil { |
| 2150 | t.Fatal("requested again after root recorded the watermark") |
| 2151 | } |
| 2152 | } |
| 2153 | |
| 2154 | func TestSystemLegIgnoresDevBuildEqualityAndOtherPlatforms(t *testing.T) { |
| 2155 | f := newLegFixture(t, "stable", "dev") |
| 2156 | f.promote("v2", []byte("x"), 0) |
| 2157 | f.leg.platform = "linux-amd64" // no artifact for me |
| 2158 | f.leg.observe(context.Background()) |
| 2159 | f.leg.tick(context.Background(), f.now) |
| 2160 | if _, err := f.marker(); err == nil { |
| 2161 | t.Fatal("requested with no artifact for this platform") |
| 2162 | } |
| 2163 | } |
| 2164 | |
| 2165 | func TestSystemLegRefusesShaMismatchAndOversize(t *testing.T) { |
| 2166 | f := newLegFixture(t, "stable", "v1") |
| 2167 | f.promote("v2", []byte("good"), 0) |
| 2168 | // Swap the object under the signed sha. |
| 2169 | f.store.Put(context.Background(), release.SystemArtifactKey("v2", "linux-arm64"), []byte("evil"), "") |
| 2170 | f.leg.observe(context.Background()) |
| 2171 | f.leg.tick(context.Background(), f.now) |
| 2172 | if _, err := f.marker(); err == nil { |
| 2173 | t.Fatal("requested an artifact whose sha does not match") |
| 2174 | } |
| 2175 | if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil { |
| 2176 | t.Fatal("mismatched artifact left in the cache") |
| 2177 | } |
| 2178 | } |
| 2179 | |
| 2180 | func TestSystemLegIgnoresAPointerOlderThanTheWatermark(t *testing.T) { |
| 2181 | f := newLegFixture(t, "stable", "v1") |
| 2182 | f.promote("v2", []byte("x"), 0) |
| 2183 | os.MkdirAll(systemRootDir(f.opt), 0o755) |
| 2184 | writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": time.Now().Add(time.Hour)}, 0o644) |
| 2185 | f.leg.observe(context.Background()) |
| 2186 | f.leg.tick(context.Background(), f.now) |
| 2187 | if _, err := f.marker(); err == nil { |
| 2188 | t.Fatal("requested a pointer root would refuse as a replay") |
| 2189 | } |
| 2190 | } |
| 2191 | |
| 2192 | func TestFastTickRunsTheSystemLeg(t *testing.T) { |
| 2193 | fx := newFastFixture(t) |
| 2194 | d := fx.deps(nil) |
| 2195 | observes := 0 |
| 2196 | d.systemLeg = &systemLeg{observeHook: func() { observes++ }} |
| 2197 | ctx := context.Background() |
| 2198 | fastTick(ctx, d, time.Now()) // boot seed |
| 2199 | if observes != 0 { |
| 2200 | t.Fatalf("boot seed observed %d times", observes) |
| 2201 | } |
| 2202 | release.BumpSerial(ctx, fx.store, time.Now()) |
| 2203 | fastTick(ctx, d, time.Now().Add(10*time.Second)) |
| 2204 | if observes != 1 { |
| 2205 | t.Fatalf("one serial move → %d observes", observes) |
| 2206 | } |
| 2207 | } |
| 2208 | ``` |
| 2209 | (`observeHook func()` is a test seam on `systemLeg`: when non-nil, `observe` calls it and returns. Add `"crypto/ed25519"` to the imports.) |
| 2210 | |
| 2211 | - [ ] **Step 2: Run to verify failure** |
| 2212 | |
| 2213 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemLeg|TestFastTickRunsTheSystemLeg' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 2214 | Expected: undefined `systemLeg`, `newSystemLeg`. |
| 2215 | |
| 2216 | - [ ] **Step 3: Implement `internal/host/systemleg.go`** |
| 2217 | |
| 2218 | ```go |
| 2219 | package host |
| 2220 | |
| 2221 | import ( |
| 2222 | "context" |
| 2223 | "crypto/ed25519" |
| 2224 | "errors" |
| 2225 | "log" |
| 2226 | "math/rand/v2" |
| 2227 | "os" |
| 2228 | "path/filepath" |
| 2229 | "runtime" |
| 2230 | "time" |
| 2231 | |
| 2232 | "github.com/carlosframework/platform/internal/release" |
| 2233 | "github.com/carlosframework/platform/internal/version" |
| 2234 | ) |
| 2235 | |
| 2236 | // systemLeg is the sandboxed half of the platform self-update (spec §4): |
| 2237 | // notice the channel pointer, fetch and verify the artifact into /data, ask |
| 2238 | // root. Two entry points so the spread has something to run it: observe on |
| 2239 | // a serial move (and hourly), tick every 2 s with no store read. |
| 2240 | type systemLeg struct { |
| 2241 | store release.Store |
| 2242 | pub ed25519.PublicKey |
| 2243 | channel string |
| 2244 | platform string |
| 2245 | build string |
| 2246 | dataDir string |
| 2247 | optDir string |
| 2248 | |
| 2249 | now func() time.Time |
| 2250 | jitter func(time.Duration) time.Duration |
| 2251 | |
| 2252 | pending *systemPending |
| 2253 | lastTouch time.Time |
| 2254 | lastObserve time.Time |
| 2255 | backoffUntil time.Time |
| 2256 | backoff time.Duration |
| 2257 | |
| 2258 | observeHook func() // test seam |
| 2259 | } |
| 2260 | |
| 2261 | type systemPending struct { |
| 2262 | version string |
| 2263 | promotedAt time.Time |
| 2264 | noop bool |
| 2265 | artifact release.SystemArtifact |
| 2266 | deadline time.Time |
| 2267 | requested bool |
| 2268 | } |
| 2269 | |
| 2270 | const systemObserveEvery = time.Hour |
| 2271 | |
| 2272 | func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg { |
| 2273 | return &systemLeg{ |
| 2274 | store: store, pub: pub, channel: systemChannel(getenv), |
| 2275 | platform: runtime.GOOS + "-" + runtime.GOARCH, build: version.Build, |
| 2276 | dataDir: dataDir, optDir: optDir, |
| 2277 | now: time.Now, |
| 2278 | jitter: func(spread time.Duration) time.Duration { return rand.N(spread + 1) }, |
| 2279 | } |
| 2280 | } |
| 2281 | |
| 2282 | // observe reads the pointer and decides. It never blocks on a download: |
| 2283 | // that happens in tick, after the spread. |
| 2284 | func (l *systemLeg) observe(ctx context.Context) { |
| 2285 | if l.observeHook != nil { |
| 2286 | l.observeHook() |
| 2287 | return |
| 2288 | } |
| 2289 | now := l.now() |
| 2290 | l.lastObserve = now |
| 2291 | p, _, err := release.GetSystemPointer(ctx, l.store, l.pub, release.SystemComponentRouter, l.channel) |
| 2292 | if err != nil { |
| 2293 | if !errors.Is(err, os.ErrNotExist) { |
| 2294 | log.Printf("agent: system pointer %s: %v", l.channel, err) |
| 2295 | } |
| 2296 | return |
| 2297 | } |
| 2298 | seen, _ := readSystemSeen(l.optDir) |
| 2299 | if wm, ok := seen[l.channel]; ok && p.PromotedAt.Before(wm) { |
| 2300 | return // root would refuse it as a replay |
| 2301 | } |
| 2302 | if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.Current.Version, p.PromotedAt) && res.terminal() { |
| 2303 | l.forget(p.Current.Version) |
| 2304 | return // refuse to repeat a failure |
| 2305 | } |
| 2306 | noop := p.Current.Version == l.build |
| 2307 | if noop { |
| 2308 | if wm, ok := seen[l.channel]; ok && !p.PromotedAt.After(wm) { |
| 2309 | return // nothing to do, and root has seen this promote |
| 2310 | } |
| 2311 | } |
| 2312 | art, ok := p.Current.Artifacts[l.platform] |
| 2313 | if !ok && !noop { |
| 2314 | if l.pending == nil || l.pending.version != p.Current.Version { |
| 2315 | log.Printf("agent: system pointer %s names %s with no artifact for %s — not for this box", l.channel, p.Current.Version, l.platform) |
| 2316 | } |
| 2317 | return |
| 2318 | } |
| 2319 | if l.pending != nil && l.pending.version == p.Current.Version && l.pending.promotedAt.Equal(p.PromotedAt) { |
| 2320 | return // already pending; keep its deadline |
| 2321 | } |
| 2322 | l.pending = &systemPending{version: p.Current.Version, promotedAt: p.PromotedAt, noop: noop, artifact: art, |
| 2323 | deadline: now.Add(l.jitter(p.SpreadDuration()))} |
| 2324 | if noop { |
| 2325 | l.pending.deadline = now |
| 2326 | } |
| 2327 | } |
| 2328 | |
| 2329 | // tick fires a pending request whose deadline has passed and re-touches an |
| 2330 | // unanswered one every minute. It reads no store except for the one fetch. |
| 2331 | func (l *systemLeg) tick(ctx context.Context, now time.Time) { |
| 2332 | if now.Sub(l.lastObserve) >= systemObserveEvery { |
| 2333 | l.observe(ctx) |
| 2334 | } |
| 2335 | p := l.pending |
| 2336 | if p == nil { |
| 2337 | return |
| 2338 | } |
| 2339 | if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.version, p.promotedAt) { |
| 2340 | switch { |
| 2341 | case res.terminal(): |
| 2342 | l.forget(p.version) |
| 2343 | return |
| 2344 | case res.Outcome == systemOutcomeNoted: |
| 2345 | l.pending = nil |
| 2346 | return |
| 2347 | case res.Outcome == systemOutcomeDeferred && res.At.After(p.deadline): |
| 2348 | // root could not reach the store; try again with backoff |
| 2349 | if l.backoff == 0 { |
| 2350 | l.backoff = fastBackoffBase |
| 2351 | } else if l.backoff < fastBackoffCap { |
| 2352 | l.backoff *= 2 |
| 2353 | } |
| 2354 | p.deadline = now.Add(l.backoff) |
| 2355 | p.requested = false |
| 2356 | } |
| 2357 | } |
| 2358 | if now.Before(p.deadline) || now.Before(l.backoffUntil) { |
| 2359 | return |
| 2360 | } |
| 2361 | if !p.requested { |
| 2362 | if !p.noop { |
| 2363 | if err := l.fetch(ctx, p); err != nil { |
| 2364 | log.Printf("agent: system update %s: %v", p.version, err) |
| 2365 | os.Remove(systemCachePath(l.dataDir, p.version)) |
| 2366 | l.backoffUntil = now.Add(fastBackoffBase) |
| 2367 | return |
| 2368 | } |
| 2369 | } |
| 2370 | if err := l.writeRequest(p, now); err != nil { |
| 2371 | log.Printf("agent: system update %s: request: %v", p.version, err) |
| 2372 | return |
| 2373 | } |
| 2374 | p.requested = true |
| 2375 | } |
| 2376 | if now.Sub(l.lastTouch) >= systemRequestEvery { |
| 2377 | if err := os.WriteFile(filepath.Join(l.dataDir, systemRequestMarker), []byte(now.UTC().Format(time.RFC3339Nano)+"\n"), 0o644); err != nil { |
| 2378 | log.Printf("agent: system update %s: cannot ask for a root apply: %v", p.version, err) |
| 2379 | return |
| 2380 | } |
| 2381 | l.lastTouch = now |
| 2382 | } |
| 2383 | } |
| 2384 | |
| 2385 | func (l *systemLeg) fetch(ctx context.Context, p *systemPending) error { |
| 2386 | dst := systemCachePath(l.dataDir, p.version) |
| 2387 | if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { |
| 2388 | return err |
| 2389 | } |
| 2390 | return release.FetchToFile(ctx, l.store, p.artifact.Key, dst, systemArtifactLimit, p.artifact.SHA256) |
| 2391 | } |
| 2392 | |
| 2393 | func (l *systemLeg) writeRequest(p *systemPending, now time.Time) error { |
| 2394 | if err := os.MkdirAll(systemDataDir(l.dataDir), 0o755); err != nil { |
| 2395 | return err |
| 2396 | } |
| 2397 | return writeJSONDurable(systemRequestPath(l.dataDir), systemRequest{Version: p.version, PromotedAt: p.promotedAt, NoOp: p.noop, SeenAt: now.UTC()}, 0o644) |
| 2398 | } |
| 2399 | |
| 2400 | // forget: root has answered with a terminal outcome. Drop the pending item |
| 2401 | // and the cache — root never deletes under /data, the agent does. |
| 2402 | func (l *systemLeg) forget(version string) { |
| 2403 | l.pending = nil |
| 2404 | l.backoff = 0 |
| 2405 | if validSystemVersion(version) { |
| 2406 | os.RemoveAll(filepath.Dir(systemCachePath(l.dataDir, version))) |
| 2407 | } |
| 2408 | } |
| 2409 | ``` |
| 2410 | |
| 2411 | `agent.go` wiring: |
| 2412 | - `fastDeps` gains `systemLeg *systemLeg // the platform self-update leg (spec §4); nil = not armed`. |
| 2413 | - In `fastTick`, inside the serial-move branch after the `scheduleLeg` call (`:1365`): `if d.systemLeg != nil { d.systemLeg.observe(ctx) }`. |
| 2414 | - Before `members := d.fastMembers(now)` (`:1370`), outside the switch: `if d.systemLeg != nil { d.systemLeg.tick(ctx, now) }`. |
| 2415 | - In `fastConvergeLoop`, after `d.scheduleLeg = scheduleLegFor(sd)` (`:1506`): `d.systemLeg = newSystemLeg(store, pub, os.Getenv, dd, optDir)` and extend the "armed" log line with `+ system pointer`. |
| 2416 | - The `adoptLoop` doc comment's DO-NOT list (`:566-600`) gets one sentence: the system leg lives in the fast loop and re-observes hourly on its own; nothing is added here. |
| 2417 | |
| 2418 | - [ ] **Step 4: Run the tests** |
| 2419 | |
| 2420 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 2421 | Expected: `ok`. |
| 2422 | |
| 2423 | - [ ] **Step 5: Commit** |
| 2424 | |
| 2425 | ```bash |
| 2426 | gofmt -l internal/host/ && git add internal/host/systemleg.go internal/host/systemleg_test.go internal/host/agent.go internal/host/agent_test.go && git commit -q -F - <<'EOF' |
| 2427 | host: the agent notices a platform promote, fetches, verifies, and asks root |
| 2428 | |
| 2429 | observe on a serial move (and hourly), tick every 2 s: spread jitter, |
| 2430 | streamed fetch with the signed sha, a request carrying only version |
| 2431 | and promoted_at, and a marker re-touched every minute until root |
| 2432 | answers. A pointer root already rolled back is never re-requested. |
| 2433 | |
| 2434 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 2435 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 2436 | EOF |
| 2437 | ``` |
| 2438 | |
| 2439 | --- |
| 2440 | ### Task 8a: `carlos system apply` — the refuse path (steps 1–5, no-op, lock) |
| 2441 | |
| 2442 | **Files:** |
| 2443 | - Create: `internal/host/systemapply.go` |
| 2444 | - Test: `internal/host/systemapply_test.go` |
| 2445 | |
| 2446 | **Interfaces:** |
| 2447 | - Consumes: Task 4 state helpers; `release.GetSystemPointer`; `release.SystemComponentRouter`; `systemChannel`. |
| 2448 | - Produces (package `host`): |
| 2449 | ```go |
| 2450 | type sysOps struct { |
| 2451 | systemctl func(args ...string) (string, error) |
| 2452 | run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) |
| 2453 | now func() time.Time |
| 2454 | sleep func(time.Duration) |
| 2455 | probeEdge func() error |
| 2456 | edgeStartedThisBoot func() bool |
| 2457 | logf func(format string, args ...any) |
| 2458 | } |
| 2459 | type systemApplier struct { |
| 2460 | ops sysOps |
| 2461 | store release.Store |
| 2462 | pub ed25519.PublicKey |
| 2463 | channel string |
| 2464 | optDir, dataDir string |
| 2465 | etcDir string // "/etc" in production; a temp dir in tests |
| 2466 | build string // version.Build of THIS process |
| 2467 | gateWait, soak time.Duration // 20s, 15s in production |
| 2468 | } |
| 2469 | func (a *systemApplier) Run(ctx context.Context, recover bool) error |
| 2470 | const ( |
| 2471 | systemQuiesceUnits = "carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service" |
| 2472 | systemEdgeUnit = "carlos-edge" |
| 2473 | ) |
| 2474 | func (a *systemApplier) rootDir() string; stagingDir() string; lockPath() string |
| 2475 | func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error |
| 2476 | func (a *systemApplier) readRequest() (systemRequest, error) // O_NOFOLLOW walk, 4 KiB cap, validSystemVersion |
| 2477 | func (a *systemApplier) verifyPointer(ctx) (*release.SystemPointer, error) |
| 2478 | func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) // path of staging/carlos |
| 2479 | func openBeneath(root string, parts ...string) (*os.File, error) // per-component O_DIRECTORY|O_NOFOLLOW, then O_NOFOLLOW |
| 2480 | var errSystemRefused = errors.New("refused"); var errSystemDeferred = errors.New("deferred") |
| 2481 | ``` |
| 2482 | In this task `Run` handles: lock → (recovery: Task 8c, stub as "journal present → return error") → read request → verify pointer + replay guard → advance seen → no-op → stage candidate → **stop with `errNotYetImplemented`** (Task 8b continues). `Run` returns nil on `noted`/`refused`/`deferred` after writing `result.json`; exit codes are `RunSystemApply`'s business (8c). |
| 2483 | |
| 2484 | - [ ] **Step 1: Write the failing tests** |
| 2485 | |
| 2486 | ```go |
| 2487 | package host |
| 2488 | |
| 2489 | import ( |
| 2490 | "context" |
| 2491 | "crypto/ed25519" |
| 2492 | "crypto/sha256" |
| 2493 | "encoding/hex" |
| 2494 | "errors" |
| 2495 | "os" |
| 2496 | "path/filepath" |
| 2497 | "strings" |
| 2498 | "testing" |
| 2499 | "time" |
| 2500 | |
| 2501 | "github.com/carlosframework/platform/internal/release" |
| 2502 | ) |
| 2503 | |
| 2504 | // applyFixture is a fake box: temp opt/data/etc roots, a DirStore with a |
| 2505 | // signed pointer, and a sysOps that records every call and answers like a |
| 2506 | // healthy systemd. The candidate "binary" is answered by the fake run. |
| 2507 | type applyFixture struct { |
| 2508 | t *testing.T |
| 2509 | a *systemApplier |
| 2510 | store *release.DirStore |
| 2511 | priv ed25519.PrivateKey |
| 2512 | calls []string // "systemctl is-active carlos-edge", "run <base> version", … |
| 2513 | answer map[string]string // "version:<path>" → stdout; "plan:<path>" → JSON plan |
| 2514 | now time.Time |
| 2515 | edgeUp bool |
| 2516 | nrestarts string |
| 2517 | activeEnter string |
| 2518 | probeErr error |
| 2519 | edgeStartedThisBoot bool |
| 2520 | } |
| 2521 | |
| 2522 | func newApplyFixture(t *testing.T, channel, build string) *applyFixture { |
| 2523 | t.Helper() |
| 2524 | priv, pub, _ := ed25519.GenerateKey(nil) |
| 2525 | f := &applyFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, answer: map[string]string{}, |
| 2526 | now: time.Date(2026, 9, 6, 11, 0, 0, 0, time.UTC), edgeUp: true, nrestarts: "0", activeEnter: "1000", edgeStartedThisBoot: true} |
| 2527 | f.a = &systemApplier{ |
| 2528 | store: f.store, pub: pub, channel: channel, |
| 2529 | optDir: t.TempDir(), dataDir: t.TempDir(), etcDir: t.TempDir(), build: build, |
| 2530 | } |
| 2531 | f.a.ops = sysOps{ |
| 2532 | systemctl: func(args ...string) (string, error) { |
| 2533 | f.calls = append(f.calls, "systemctl "+strings.Join(args, " ")) |
| 2534 | switch { |
| 2535 | case args[0] == "is-active" && args[1] == systemEdgeUnit: |
| 2536 | if f.edgeUp { |
| 2537 | return "active\n", nil |
| 2538 | } |
| 2539 | return "failed\n", errors.New("inactive") |
| 2540 | case args[0] == "show": |
| 2541 | return f.nrestarts + "\n" + f.activeEnter + "\n", nil |
| 2542 | } |
| 2543 | return "", nil |
| 2544 | }, |
| 2545 | run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) { |
| 2546 | f.calls = append(f.calls, "run "+filepath.Base(name)+" "+strings.Join(args, " ")) |
| 2547 | if len(args) == 1 && args[0] == "version" { |
| 2548 | if v, ok := f.answer["version:"+name]; ok { |
| 2549 | return v, nil |
| 2550 | } |
| 2551 | return "", errors.New("no such binary") |
| 2552 | } |
| 2553 | if len(args) >= 3 && args[0] == "bootstrap" && args[1] == "-root" { |
| 2554 | plan, ok := f.answer["plan:"+name] |
| 2555 | if !ok { |
| 2556 | return "", errors.New("bootstrap failed") |
| 2557 | } |
| 2558 | units := strings.Split(f.answer["units:"+name], " ") |
| 2559 | dir := filepath.Join(args[2], "etc", "systemd", "system") |
| 2560 | os.MkdirAll(dir, 0o755) |
| 2561 | for _, u := range units { |
| 2562 | if u != "" { |
| 2563 | os.WriteFile(filepath.Join(dir, u), []byte("[Unit]\n# "+filepath.Base(name)+"\n"), 0o644) |
| 2564 | } |
| 2565 | } |
| 2566 | os.MkdirAll(filepath.Join(args[2], "etc", "sysusers.d"), 0o755) |
| 2567 | os.WriteFile(filepath.Join(args[2], "etc", "sysusers.d", "carlos.conf"), []byte("g carlos-sockets -\n"), 0o644) |
| 2568 | os.MkdirAll(filepath.Join(args[2], "etc", "tmpfiles.d"), 0o755) |
| 2569 | os.WriteFile(filepath.Join(args[2], "etc", "tmpfiles.d", "carlos.conf"), []byte("d /run/carlos\n"), 0o644) |
| 2570 | os.MkdirAll(filepath.Join(args[2], "data"), 0o755) |
| 2571 | os.WriteFile(filepath.Join(args[2], "data", "registry.db"), nil, 0o640) |
| 2572 | return plan, nil |
| 2573 | } |
| 2574 | return "", errors.New("unexpected command") |
| 2575 | }, |
| 2576 | now: func() time.Time { return f.now }, |
| 2577 | sleep: func(d time.Duration) { f.now = f.now.Add(d) }, |
| 2578 | probeEdge: func() error { return f.probeErr }, |
| 2579 | edgeStartedThisBoot: func() bool { return f.edgeStartedThisBoot }, |
| 2580 | logf: t.Logf, |
| 2581 | } |
| 2582 | os.MkdirAll(systemRootDir(f.a.optDir), 0o755) |
| 2583 | os.MkdirAll(filepath.Join(f.a.etcDir, "systemd", "system"), 0o755) |
| 2584 | // The on-disk binary answers `version` with build. |
| 2585 | os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos "+build+"\n"), 0o755) |
| 2586 | f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos " + build + "\n" |
| 2587 | f.answer["plan:"+systemBinaryPath(f.a.optDir)] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service"],"inert":["carlos-restore-verify.timer"]}` |
| 2588 | f.answer["units:"+systemBinaryPath(f.a.optDir)] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-old-only.timer" |
| 2589 | return f |
| 2590 | } |
| 2591 | |
| 2592 | // promote signs a pointer for f.a.channel naming version with body as the |
| 2593 | // linux-arm64 artifact, stages body into the agent's cache, and writes the |
| 2594 | // request the agent would have written. Returns the request. |
| 2595 | func (f *applyFixture) promote(version string, body []byte, noop bool) systemRequest { |
| 2596 | f.t.Helper() |
| 2597 | sum := sha256.Sum256(body) |
| 2598 | rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{ |
| 2599 | "linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}} |
| 2600 | f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, "") |
| 2601 | p, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.a.channel, rel, release.SystemPromoteOptions{}) |
| 2602 | if err != nil { |
| 2603 | f.t.Fatal(err) |
| 2604 | } |
| 2605 | req := systemRequest{Version: version, PromotedAt: p.PromotedAt, NoOp: noop, SeenAt: f.now} |
| 2606 | os.MkdirAll(systemDataDir(f.a.dataDir), 0o755) |
| 2607 | writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644) |
| 2608 | if !noop { |
| 2609 | os.MkdirAll(filepath.Dir(systemCachePath(f.a.dataDir, version)), 0o755) |
| 2610 | os.WriteFile(systemCachePath(f.a.dataDir, version), body, 0o644) |
| 2611 | } |
| 2612 | return req |
| 2613 | } |
| 2614 | |
| 2615 | func (f *applyFixture) result() *systemResult { r, _ := readSystemResult(f.a.optDir); return r } |
| 2616 | |
| 2617 | func (f *applyFixture) called(prefix string) bool { |
| 2618 | for _, c := range f.calls { |
| 2619 | if strings.HasPrefix(c, prefix) { |
| 2620 | return true |
| 2621 | } |
| 2622 | } |
| 2623 | return false |
| 2624 | } |
| 2625 | |
| 2626 | func TestApplyNoRequestIsNothingToDo(t *testing.T) { |
| 2627 | f := newApplyFixture(t, "stable", "v1") |
| 2628 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2629 | t.Fatal(err) |
| 2630 | } |
| 2631 | if f.result() != nil || len(f.calls) != 0 { |
| 2632 | t.Fatalf("result=%+v calls=%v", f.result(), f.calls) |
| 2633 | } |
| 2634 | } |
| 2635 | |
| 2636 | func TestApplyLockKeepsTwoRunsApart(t *testing.T) { |
| 2637 | f := newApplyFixture(t, "stable", "v1") |
| 2638 | f.promote("v2", []byte("x"), false) |
| 2639 | // Hold the lock as another process would. |
| 2640 | held, err := os.OpenFile(f.a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600) |
| 2641 | if err != nil { |
| 2642 | t.Fatal(err) |
| 2643 | } |
| 2644 | defer held.Close() |
| 2645 | if err := flockExclusiveNonblock(held); err != nil { |
| 2646 | t.Fatal(err) |
| 2647 | } |
| 2648 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2649 | t.Fatalf("a locked-out path run must exit 0 quietly: %v", err) |
| 2650 | } |
| 2651 | if f.result() != nil { |
| 2652 | t.Fatal("a locked-out run wrote a result") |
| 2653 | } |
| 2654 | } |
| 2655 | |
| 2656 | func TestApplyRequestIsAHintNotAnInstruction(t *testing.T) { |
| 2657 | f := newApplyFixture(t, "stable", "v1") |
| 2658 | // A symlinked parent under /data. |
| 2659 | os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere"), 0o755) |
| 2660 | os.Symlink(filepath.Join(f.a.dataDir, "elsewhere"), filepath.Join(f.a.dataDir, "system")) |
| 2661 | os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere", "router"), 0o755) |
| 2662 | os.WriteFile(filepath.Join(f.a.dataDir, "elsewhere", "router", "requested.json"), []byte(`{"version":"v2"}`), 0o644) |
| 2663 | if _, err := f.a.readRequest(); err == nil { |
| 2664 | t.Fatal("followed a symlinked parent under /data") |
| 2665 | } |
| 2666 | os.Remove(filepath.Join(f.a.dataDir, "system")) |
| 2667 | // A symlinked request file. |
| 2668 | os.MkdirAll(systemDataDir(f.a.dataDir), 0o755) |
| 2669 | os.WriteFile(filepath.Join(f.a.dataDir, "real.json"), []byte(`{"version":"v2"}`), 0o644) |
| 2670 | os.Symlink(filepath.Join(f.a.dataDir, "real.json"), systemRequestPath(f.a.dataDir)) |
| 2671 | if _, err := f.a.readRequest(); err == nil { |
| 2672 | t.Fatal("followed a symlinked request") |
| 2673 | } |
| 2674 | os.Remove(systemRequestPath(f.a.dataDir)) |
| 2675 | // A traversal version. |
| 2676 | writeJSONDurable(systemRequestPath(f.a.dataDir), systemRequest{Version: "../../etc/passwd", PromotedAt: f.now}, 0o644) |
| 2677 | if _, err := f.a.readRequest(); err == nil { |
| 2678 | t.Fatal("accepted a traversal version") |
| 2679 | } |
| 2680 | // Oversized. |
| 2681 | os.WriteFile(systemRequestPath(f.a.dataDir), []byte(`{"version":"`+strings.Repeat("a", 5000)+`"}`), 0o644) |
| 2682 | if _, err := f.a.readRequest(); err == nil { |
| 2683 | t.Fatal("accepted a 5 KiB request") |
| 2684 | } |
| 2685 | } |
| 2686 | |
| 2687 | func TestApplyChannelComesFromRootNotTheRequest(t *testing.T) { |
| 2688 | // The box is on stable; a valid CANARY pointer exists and the agent |
| 2689 | // requests its version. Root reads control/system/router/stable.json |
| 2690 | // (missing) and refuses — a signed canary promote does not authorise a |
| 2691 | // stable box. |
| 2692 | f := newApplyFixture(t, "stable", "v1") |
| 2693 | f.a.channel = "canary" |
| 2694 | req := f.promote("v2", []byte("x"), false) // signs the CANARY pointer |
| 2695 | f.a.channel = "stable" |
| 2696 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2697 | t.Fatal(err) |
| 2698 | } |
| 2699 | r := f.result() |
| 2700 | if r == nil || r.Outcome != systemOutcomeRefused || !r.matches("stable", "v2", req.PromotedAt) { |
| 2701 | t.Fatalf("result = %+v", r) |
| 2702 | } |
| 2703 | if _, err := os.Stat(systemBinaryPath(f.a.optDir) + ".prev"); err == nil { |
| 2704 | t.Fatal("touched the box") |
| 2705 | } |
| 2706 | } |
| 2707 | |
| 2708 | func TestApplyStaleTupleIsRefusedAgainstTheRequestsTuple(t *testing.T) { |
| 2709 | f := newApplyFixture(t, "stable", "v1") |
| 2710 | old := f.promote("v2", []byte("x"), false) |
| 2711 | time.Sleep(1100 * time.Millisecond) |
| 2712 | f.promote("v3", []byte("y"), false) |
| 2713 | // Put the OLD request back: the agent's spread started under v2. |
| 2714 | writeJSONDurable(systemRequestPath(f.a.dataDir), old, 0o644) |
| 2715 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2716 | t.Fatal(err) |
| 2717 | } |
| 2718 | r := f.result() |
| 2719 | if r == nil || r.Outcome != systemOutcomeRefused || r.Version != "v2" || !r.PromotedAt.Equal(old.PromotedAt) { |
| 2720 | t.Fatalf("result = %+v (must name the REQUEST's tuple so v3 is not suppressed)", r) |
| 2721 | } |
| 2722 | } |
| 2723 | |
| 2724 | func TestApplyReplayGuardIsPerChannelAndEqualIsAllowed(t *testing.T) { |
| 2725 | f := newApplyFixture(t, "stable", "v1") |
| 2726 | req := f.promote("v2", []byte("x"), true) |
| 2727 | // Older than the watermark → refused, seen unchanged. |
| 2728 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt.Add(time.Hour), "canary": time.Time{}}, 0o644) |
| 2729 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2730 | t.Fatal(err) |
| 2731 | } |
| 2732 | if r := f.result(); r == nil || r.Outcome != systemOutcomeRefused || !strings.Contains(r.Reason, "replay") { |
| 2733 | t.Fatalf("result = %+v", r) |
| 2734 | } |
| 2735 | // Equal → allowed (a retry of the same promote). |
| 2736 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt}, 0o644) |
| 2737 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2738 | t.Fatal(err) |
| 2739 | } |
| 2740 | if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted { |
| 2741 | t.Fatalf("result = %+v", r) |
| 2742 | } |
| 2743 | // Another channel's newer watermark does not block this channel. |
| 2744 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"canary": req.PromotedAt.Add(time.Hour)}, 0o644) |
| 2745 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2746 | t.Fatal(err) |
| 2747 | } |
| 2748 | if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted { |
| 2749 | t.Fatalf("result = %+v", r) |
| 2750 | } |
| 2751 | seen, _ := readSystemSeen(f.a.optDir) |
| 2752 | if !seen["stable"].Equal(req.PromotedAt) { |
| 2753 | t.Fatalf("seen not advanced: %v", seen) |
| 2754 | } |
| 2755 | } |
| 2756 | |
| 2757 | func TestApplyNoOpNotesAndTouchesNothing(t *testing.T) { |
| 2758 | f := newApplyFixture(t, "stable", "v2") |
| 2759 | f.promote("v2", []byte("x"), true) |
| 2760 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2761 | t.Fatal(err) |
| 2762 | } |
| 2763 | if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted { |
| 2764 | t.Fatalf("result = %+v", r) |
| 2765 | } |
| 2766 | if _, err := os.Stat(f.a.stagingDir()); err == nil { |
| 2767 | t.Fatal("a no-op created staging/") |
| 2768 | } |
| 2769 | if len(f.calls) != 0 { |
| 2770 | t.Fatalf("a no-op ran commands: %v", f.calls) |
| 2771 | } |
| 2772 | } |
| 2773 | |
| 2774 | func TestApplyStoreErrorIsDeferredNotRefused(t *testing.T) { |
| 2775 | f := newApplyFixture(t, "stable", "v1") |
| 2776 | req := f.promote("v2", []byte("x"), false) |
| 2777 | f.a.store = failingStore{f.store} |
| 2778 | if err := f.a.Run(context.Background(), false); err != nil { |
| 2779 | t.Fatal(err) |
| 2780 | } |
| 2781 | if r := f.result(); r == nil || r.Outcome != systemOutcomeDeferred || !r.matches("stable", "v2", req.PromotedAt) { |
| 2782 | t.Fatalf("result = %+v", r) |
| 2783 | } |
| 2784 | } |
| 2785 | |
| 2786 | type failingStore struct{ release.Store } |
| 2787 | |
| 2788 | func (failingStore) Get(context.Context, string) ([]byte, string, error) { |
| 2789 | return nil, "", errors.New("door restarting") |
| 2790 | } |
| 2791 | |
| 2792 | func TestApplyStagesTheCandidateFromARootOwnedCopy(t *testing.T) { |
| 2793 | f := newApplyFixture(t, "stable", "v1") |
| 2794 | body := []byte("#!/bin/sh\necho carlos v2\n") |
| 2795 | f.promote("v2", body, false) |
| 2796 | f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v2\n" |
| 2797 | p, err := f.a.verifyPointer(context.Background()) |
| 2798 | if err != nil { |
| 2799 | t.Fatal(err) |
| 2800 | } |
| 2801 | req, _ := f.a.readRequest() |
| 2802 | staged, err := f.a.stageCandidate(p, req) |
| 2803 | if err != nil { |
| 2804 | t.Fatal(err) |
| 2805 | } |
| 2806 | st, _ := os.Stat(staged) |
| 2807 | if st.Mode().Perm() != 0o755 { |
| 2808 | t.Fatalf("staged mode = %v after verification", st.Mode().Perm()) |
| 2809 | } |
| 2810 | if dst, _ := os.Stat(f.a.stagingDir()); dst.Mode().Perm() != 0o700 { |
| 2811 | t.Fatalf("staging dir mode = %v", dst.Mode().Perm()) |
| 2812 | } |
| 2813 | // Swapping the cache after staging changes nothing about the copy. |
| 2814 | os.WriteFile(systemCachePath(f.a.dataDir, "v2"), []byte("evil"), 0o644) |
| 2815 | got, _ := os.ReadFile(staged) |
| 2816 | if string(got) != string(body) { |
| 2817 | t.Fatal("staged copy follows the cache") |
| 2818 | } |
| 2819 | // A sha mismatch is refused BEFORE the copy is made executable and |
| 2820 | // staging/ is cleared. |
| 2821 | f2 := newApplyFixture(t, "stable", "v1") |
| 2822 | f2.promote("v2", body, false) |
| 2823 | os.WriteFile(systemCachePath(f2.a.dataDir, "v2"), []byte("evil"), 0o644) |
| 2824 | p2, _ := f2.a.verifyPointer(context.Background()) |
| 2825 | req2, _ := f2.a.readRequest() |
| 2826 | if _, err := f2.a.stageCandidate(p2, req2); err == nil || !strings.Contains(err.Error(), "sha256") { |
| 2827 | t.Fatalf("err = %v", err) |
| 2828 | } |
| 2829 | // A candidate that lies about its version is refused. |
| 2830 | f3 := newApplyFixture(t, "stable", "v1") |
| 2831 | f3.promote("v2", body, false) |
| 2832 | f3.answer["version:"+filepath.Join(f3.a.stagingDir(), "carlos")] = "carlos v9\n" |
| 2833 | p3, _ := f3.a.verifyPointer(context.Background()) |
| 2834 | req3, _ := f3.a.readRequest() |
| 2835 | if _, err := f3.a.stageCandidate(p3, req3); err == nil || !strings.Contains(err.Error(), "reports") { |
| 2836 | t.Fatalf("err = %v", err) |
| 2837 | } |
| 2838 | // A symlinked cache file is refused. |
| 2839 | f4 := newApplyFixture(t, "stable", "v1") |
| 2840 | f4.promote("v2", body, false) |
| 2841 | os.Remove(systemCachePath(f4.a.dataDir, "v2")) |
| 2842 | os.WriteFile(filepath.Join(f4.a.dataDir, "real"), body, 0o644) |
| 2843 | os.Symlink(filepath.Join(f4.a.dataDir, "real"), systemCachePath(f4.a.dataDir, "v2")) |
| 2844 | p4, _ := f4.a.verifyPointer(context.Background()) |
| 2845 | req4, _ := f4.a.readRequest() |
| 2846 | if _, err := f4.a.stageCandidate(p4, req4); err == nil { |
| 2847 | t.Fatal("followed a symlinked cache file") |
| 2848 | } |
| 2849 | } |
| 2850 | ``` |
| 2851 | |
| 2852 | - [ ] **Step 2: Run to verify failure** |
| 2853 | |
| 2854 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 2855 | Expected: undefined `systemApplier`, `sysOps`, `flockExclusiveNonblock`. |
| 2856 | |
| 2857 | - [ ] **Step 3: Implement the refuse path in `internal/host/systemapply.go`** |
| 2858 | |
| 2859 | ```go |
| 2860 | package host |
| 2861 | |
| 2862 | import ( |
| 2863 | "context" |
| 2864 | "crypto/ed25519" |
| 2865 | "crypto/sha256" |
| 2866 | "encoding/hex" |
| 2867 | "encoding/json" |
| 2868 | "errors" |
| 2869 | "fmt" |
| 2870 | "io" |
| 2871 | "os" |
| 2872 | "path/filepath" |
| 2873 | "strings" |
| 2874 | "syscall" |
| 2875 | "time" |
| 2876 | |
| 2877 | "github.com/carlosframework/platform/internal/release" |
| 2878 | ) |
| 2879 | |
| 2880 | // The root half of the platform self-update (spec §6). Runs as root from |
| 2881 | // carlos-system-update.service (or --recover from carlos-system-recover). |
| 2882 | // Every box-changing step is journaled first; every run begins with |
| 2883 | // recovery; root reads ONE file under /data and treats it as a hint. |
| 2884 | |
| 2885 | const ( |
| 2886 | systemEdgeUnit = "carlos-edge" |
| 2887 | systemLockFile = "lock" |
| 2888 | systemMaxRequest = 4 << 10 |
| 2889 | systemGateWait = 20 * time.Second |
| 2890 | systemGateSoak = 15 * time.Second |
| 2891 | systemVersionWait = 30 * time.Second |
| 2892 | systemBootstrapWait = 120 * time.Second |
| 2893 | ) |
| 2894 | |
| 2895 | // systemQuiesceUnits are the root units that exec /opt/carlos/carlos |
| 2896 | // against the running edge. Stopped (services too — stopping a timer does |
| 2897 | // not stop the pass it already started) before the rename, re-armed after |
| 2898 | // the gate. |
| 2899 | var systemQuiesceUnits = []string{ |
| 2900 | "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path", |
| 2901 | "carlos-adopt.service", "carlos-config-sync.service", "carlos-stage.service", "carlos-restart.service", |
| 2902 | } |
| 2903 | |
| 2904 | var ( |
| 2905 | errSystemRefused = errors.New("refused") |
| 2906 | errSystemDeferred = errors.New("deferred") |
| 2907 | errNotYetImplemented = errors.New("system apply: install path not implemented") |
| 2908 | ) |
| 2909 | |
| 2910 | // sysOps is the seam between the apply and the box: systemctl, exec with a |
| 2911 | // deadline, the clock, sleep, the edge probe. Tests record and answer. |
| 2912 | type sysOps struct { |
| 2913 | systemctl func(args ...string) (string, error) |
| 2914 | run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) |
| 2915 | now func() time.Time |
| 2916 | sleep func(time.Duration) |
| 2917 | probeEdge func() error |
| 2918 | edgeStartedThisBoot func() bool |
| 2919 | logf func(format string, args ...any) |
| 2920 | } |
| 2921 | |
| 2922 | type systemApplier struct { |
| 2923 | ops sysOps |
| 2924 | store release.Store |
| 2925 | pub ed25519.PublicKey |
| 2926 | channel string |
| 2927 | optDir string |
| 2928 | dataDir string |
| 2929 | etcDir string |
| 2930 | build string |
| 2931 | gateWait, soak time.Duration |
| 2932 | } |
| 2933 | |
| 2934 | func (a *systemApplier) rootDir() string { return systemRootDir(a.optDir) } |
| 2935 | func (a *systemApplier) stagingDir() string { return filepath.Join(a.rootDir(), "staging") } |
| 2936 | func (a *systemApplier) lockPath() string { return filepath.Join(a.rootDir(), systemLockFile) } |
| 2937 | func (a *systemApplier) file(name string) string { return filepath.Join(a.rootDir(), name) } |
| 2938 | |
| 2939 | // Run is the whole oneshot. It returns an error only for a failure the |
| 2940 | // caller should exit non-zero on (an apply that rolled back, a rollback |
| 2941 | // that failed, a broken box); refusals and no-ops are recorded in |
| 2942 | // result.json and return nil. |
| 2943 | func (a *systemApplier) Run(ctx context.Context, recover bool) error { |
| 2944 | if err := os.MkdirAll(a.rootDir(), 0o755); err != nil { |
| 2945 | return err |
| 2946 | } |
| 2947 | lock, err := os.OpenFile(a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600) |
| 2948 | if err != nil { |
| 2949 | return err |
| 2950 | } |
| 2951 | defer lock.Close() |
| 2952 | if recover { |
| 2953 | if err := flockExclusive(lock); err != nil { |
| 2954 | return err |
| 2955 | } |
| 2956 | } else if err := flockExclusiveNonblock(lock); err != nil { |
| 2957 | a.ops.logf("system apply: another apply is running; leaving it to finish") |
| 2958 | return nil |
| 2959 | } |
| 2960 | defer os.RemoveAll(a.stagingDir()) |
| 2961 | |
| 2962 | if err := a.recoverIfNeeded(ctx); err != nil { |
| 2963 | return err |
| 2964 | } |
| 2965 | if recover { |
| 2966 | return nil |
| 2967 | } |
| 2968 | req, err := a.readRequest() |
| 2969 | if errors.Is(err, os.ErrNotExist) { |
| 2970 | return nil |
| 2971 | } |
| 2972 | if err != nil { |
| 2973 | a.ops.logf("system apply: request: %v", err) |
| 2974 | return nil // nothing to record against: the request itself is unreadable |
| 2975 | } |
| 2976 | p, err := a.verifyPointer(ctx) |
| 2977 | if err != nil { |
| 2978 | return a.record(req, err) |
| 2979 | } |
| 2980 | if req.Version != p.Current.Version || !req.PromotedAt.Equal(p.PromotedAt) { |
| 2981 | return a.record(req, fmt.Errorf("%w: request names %s@%s, pointer is %s@%s", errSystemRefused, |
| 2982 | req.Version, req.PromotedAt.Format(time.RFC3339), p.Current.Version, p.PromotedAt.Format(time.RFC3339))) |
| 2983 | } |
| 2984 | seen, err := readSystemSeen(a.optDir) |
| 2985 | if err != nil { |
| 2986 | return a.record(req, fmt.Errorf("%w: seen.json unreadable: %v", errSystemRefused, err)) |
| 2987 | } |
| 2988 | if wm, ok := seen[a.channel]; ok && p.PromotedAt.Before(wm) { |
| 2989 | return a.record(req, fmt.Errorf("%w: replay — pointer promoted %s is older than the %s watermark %s", errSystemRefused, |
| 2990 | p.PromotedAt.Format(time.RFC3339), a.channel, wm.Format(time.RFC3339))) |
| 2991 | } |
| 2992 | seen[a.channel] = p.PromotedAt |
| 2993 | if err := writeJSONDurable(a.file("seen.json"), seen, 0o644); err != nil { |
| 2994 | return err |
| 2995 | } |
| 2996 | if req.NoOp || p.Current.Version == a.onDiskBuild(ctx) { |
| 2997 | return a.writeResult(req, systemOutcomeNoted, "") |
| 2998 | } |
| 2999 | staged, err := a.stageCandidate(p, req) |
| 3000 | if err != nil { |
| 3001 | return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err)) |
| 3002 | } |
| 3003 | return a.install(ctx, req, p, staged) // Task 8b |
| 3004 | } |
| 3005 | |
| 3006 | func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error { |
| 3007 | return errNotYetImplemented |
| 3008 | } |
| 3009 | |
| 3010 | func (a *systemApplier) recoverIfNeeded(ctx context.Context) error { |
| 3011 | j, err := readSystemJournal(a.optDir) |
| 3012 | if err != nil || j == nil { |
| 3013 | return err |
| 3014 | } |
| 3015 | return fmt.Errorf("system apply: journal open at %s — recovery not implemented", j.Phase) // Task 8c |
| 3016 | } |
| 3017 | |
| 3018 | // record turns a refusal or deferral into result.json and a nil return. |
| 3019 | func (a *systemApplier) record(req systemRequest, err error) error { |
| 3020 | outcome := systemOutcomeRefused |
| 3021 | if errors.Is(err, errSystemDeferred) { |
| 3022 | outcome = systemOutcomeDeferred |
| 3023 | } |
| 3024 | a.ops.logf("system apply: %s %s: %s: %v", a.channel, req.Version, outcome, err) |
| 3025 | return a.writeResult(req, outcome, err.Error()) |
| 3026 | } |
| 3027 | |
| 3028 | func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error { |
| 3029 | return writeJSONDurable(a.file("result.json"), systemResult{ |
| 3030 | Channel: a.channel, Version: req.Version, PromotedAt: req.PromotedAt, |
| 3031 | Outcome: outcome, Reason: reason, At: a.ops.now().UTC(), |
| 3032 | }, 0o644) |
| 3033 | } |
| 3034 | |
| 3035 | // readRequest is the one read under /data: a fixed path reached without |
| 3036 | // following a single symlink, a regular file under 4 KiB, a version that |
| 3037 | // passes validSystemVersion. Anything else is an error, and root acts on |
| 3038 | // nothing. |
| 3039 | func (a *systemApplier) readRequest() (systemRequest, error) { |
| 3040 | f, err := openBeneath(a.dataDir, "system", systemComponent, "requested.json") |
| 3041 | if err != nil { |
| 3042 | return systemRequest{}, err |
| 3043 | } |
| 3044 | defer f.Close() |
| 3045 | st, err := f.Stat() |
| 3046 | if err != nil { |
| 3047 | return systemRequest{}, err |
| 3048 | } |
| 3049 | if !st.Mode().IsRegular() || st.Size() > systemMaxRequest { |
| 3050 | return systemRequest{}, fmt.Errorf("request: not a regular file under %d bytes", systemMaxRequest) |
| 3051 | } |
| 3052 | var req systemRequest |
| 3053 | if err := json.NewDecoder(io.LimitReader(f, systemMaxRequest)).Decode(&req); err != nil { |
| 3054 | return systemRequest{}, fmt.Errorf("request: %w", err) |
| 3055 | } |
| 3056 | if !validSystemVersion(req.Version) { |
| 3057 | return systemRequest{}, fmt.Errorf("request: version %q is not a version", req.Version) |
| 3058 | } |
| 3059 | if req.PromotedAt.IsZero() { |
| 3060 | return systemRequest{}, errors.New("request: promoted_at unset") |
| 3061 | } |
| 3062 | return req, nil |
| 3063 | } |
| 3064 | |
| 3065 | // openBeneath opens root/parts... one component at a time with |
| 3066 | // O_DIRECTORY|O_NOFOLLOW for every directory and O_NOFOLLOW for the leaf, |
| 3067 | // so no component under a user-owned tree can be a symlink. |
| 3068 | func openBeneath(root string, parts ...string) (*os.File, error) { |
| 3069 | fd, err := syscall.Open(root, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_CLOEXEC, 0) |
| 3070 | if err != nil { |
| 3071 | return nil, fmt.Errorf("open %s: %w", root, err) |
| 3072 | } |
| 3073 | for i, part := range parts { |
| 3074 | flags := syscall.O_RDONLY | syscall.O_NOFOLLOW | syscall.O_CLOEXEC |
| 3075 | if i < len(parts)-1 { |
| 3076 | flags |= syscall.O_DIRECTORY |
| 3077 | } |
| 3078 | next, err := syscall.Openat(fd, part, flags, 0) |
| 3079 | syscall.Close(fd) |
| 3080 | if err != nil { |
| 3081 | return nil, fmt.Errorf("open %s: %w", filepath.Join(append([]string{root}, parts[:i+1]...)...), err) |
| 3082 | } |
| 3083 | fd = next |
| 3084 | } |
| 3085 | return os.NewFile(uintptr(fd), filepath.Join(append([]string{root}, parts...)...)), nil |
| 3086 | } |
| 3087 | |
| 3088 | // verifyPointer fetches THIS box's channel pointer and verifies it with |
| 3089 | // the release public key. The channel is root's own (host.env), never the |
| 3090 | // request's. A store error is deferred, a bad pointer refused. |
| 3091 | func (a *systemApplier) verifyPointer(ctx context.Context) (*release.SystemPointer, error) { |
| 3092 | p, _, err := release.GetSystemPointer(ctx, a.store, a.pub, release.SystemComponentRouter, a.channel) |
| 3093 | switch { |
| 3094 | case err == nil: |
| 3095 | return p, nil |
| 3096 | case errors.Is(err, os.ErrNotExist): |
| 3097 | return nil, fmt.Errorf("%w: no %s pointer for this box's channel", errSystemRefused, a.channel) |
| 3098 | case strings.Contains(err.Error(), "system pointer"): |
| 3099 | return nil, fmt.Errorf("%w: %v", errSystemRefused, err) |
| 3100 | default: |
| 3101 | return nil, fmt.Errorf("%w: %v", errSystemDeferred, err) |
| 3102 | } |
| 3103 | } |
| 3104 | |
| 3105 | // onDiskBuild asks the installed binary what it is, with a deadline. Not |
| 3106 | // version.Build: after an interrupted apply the two can differ. |
| 3107 | func (a *systemApplier) onDiskBuild(ctx context.Context) string { |
| 3108 | out, err := a.ops.run(ctx, systemVersionWait, systemBinaryPath(a.optDir), "version") |
| 3109 | if err != nil { |
| 3110 | return "" |
| 3111 | } |
| 3112 | return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos ")) |
| 3113 | } |
| 3114 | |
| 3115 | // stageCandidate copies the agent's cache into staging/ (0700) as 0600, |
| 3116 | // verifies the copy's sha against the pointer, makes it executable, and |
| 3117 | // runs its `version`. Nothing under /data is read after the copy. |
| 3118 | func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) { |
| 3119 | art, ok := p.Current.Artifacts[a.platform()] |
| 3120 | if !ok { |
| 3121 | return "", fmt.Errorf("pointer names no artifact for %s", a.platform()) |
| 3122 | } |
| 3123 | os.RemoveAll(a.stagingDir()) |
| 3124 | if err := os.MkdirAll(a.stagingDir(), 0o700); err != nil { |
| 3125 | return "", err |
| 3126 | } |
| 3127 | if err := os.Chmod(a.stagingDir(), 0o700); err != nil { |
| 3128 | return "", err |
| 3129 | } |
| 3130 | src, err := openBeneath(a.dataDir, "system", systemComponent, req.Version, "carlos") |
| 3131 | if err != nil { |
| 3132 | return "", fmt.Errorf("candidate: %w", err) |
| 3133 | } |
| 3134 | defer src.Close() |
| 3135 | if st, err := src.Stat(); err != nil || !st.Mode().IsRegular() { |
| 3136 | return "", errors.New("candidate: not a regular file") |
| 3137 | } |
| 3138 | staged := filepath.Join(a.stagingDir(), "carlos") |
| 3139 | dst, err := os.OpenFile(staged, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600) |
| 3140 | if err != nil { |
| 3141 | return "", err |
| 3142 | } |
| 3143 | h := sha256.New() |
| 3144 | n, err := io.Copy(io.MultiWriter(dst, h), io.LimitReader(src, systemArtifactLimit+1)) |
| 3145 | dst.Close() |
| 3146 | if err != nil { |
| 3147 | return "", err |
| 3148 | } |
| 3149 | if n > systemArtifactLimit { |
| 3150 | return "", fmt.Errorf("candidate: larger than %d bytes", systemArtifactLimit) |
| 3151 | } |
| 3152 | if got := hex.EncodeToString(h.Sum(nil)); got != art.SHA256 { |
| 3153 | return "", fmt.Errorf("candidate: sha256 %s does not match the signed %s", got, art.SHA256) |
| 3154 | } |
| 3155 | if err := os.Chmod(staged, 0o755); err != nil { |
| 3156 | return "", err |
| 3157 | } |
| 3158 | out, err := a.ops.run(context.Background(), systemVersionWait, staged, "version") |
| 3159 | if err != nil || !strings.Contains(out, "carlos "+p.Current.Version) { |
| 3160 | return "", fmt.Errorf("candidate reports %q, not %s", strings.TrimSpace(out), p.Current.Version) |
| 3161 | } |
| 3162 | return staged, nil |
| 3163 | } |
| 3164 | |
| 3165 | func (a *systemApplier) platform() string { return runtime.GOOS + "-" + runtime.GOARCH } |
| 3166 | |
| 3167 | func flockExclusive(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX) } |
| 3168 | func flockExclusiveNonblock(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) } |
| 3169 | ``` |
| 3170 | Add `"runtime"` to the imports. Tests that stage a candidate need `platform()` to be `linux-arm64`: make `platform` a field set in the constructor (`platform string`, default `runtime.GOOS+"-"+runtime.GOARCH`) rather than a method, and have the fixture set `f.a.platform = "linux-arm64"`. Do that; update the test fixture accordingly. |
| 3171 | |
| 3172 | - [ ] **Step 4: Run the tests** |
| 3173 | |
| 3174 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 3175 | Expected: PASS for every test in this task. |
| 3176 | |
| 3177 | - [ ] **Step 5: Commit** |
| 3178 | |
| 3179 | ```bash |
| 3180 | gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF' |
| 3181 | host: carlos system apply — the refuse path |
| 3182 | |
| 3183 | Lock, one O_NOFOLLOW-walked request under /data, the channel from |
| 3184 | root's own env, the pointer re-verified, the per-channel replay |
| 3185 | watermark, the no-op, and the candidate copied into a 0700 staging |
| 3186 | dir and verified there before it is made executable. |
| 3187 | |
| 3188 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 3189 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 3190 | EOF |
| 3191 | ``` |
| 3192 | |
| 3193 | --- |
| 3194 | |
| 3195 | ### Task 8b: `carlos system apply` — install, gate, arm, record (steps 6–12) |
| 3196 | |
| 3197 | **Files:** |
| 3198 | - Modify: `internal/host/systemapply.go` (replace the `install` stub) |
| 3199 | - Test: `internal/host/systemapply_test.go` (add) |
| 3200 | |
| 3201 | **Interfaces:** |
| 3202 | - Consumes: `systemctlPlan`, `bootstrapPlan`, `systemQuiesceUnits`, the fixture from 8a. |
| 3203 | - Produces: |
| 3204 | ```go |
| 3205 | func (a *systemApplier) install(ctx, req, p, staged) error |
| 3206 | func (a *systemApplier) decidePrev(ctx) (prevExists bool, err error) // journal backing-up, copy when authorised |
| 3207 | func (a *systemApplier) stagePlan(ctx, binary, dir string) (bootstrapPlan, []string, error) // runs bootstrap -root dir -plan-json; returns plan + unit file names |
| 3208 | func (a *systemApplier) journal(j systemJournal) error |
| 3209 | func (a *systemApplier) installUnits(stagedRoot string, remove []string) error // remove, copy units, sysusers, tmpfiles, daemon-reload |
| 3210 | func (a *systemApplier) restartEdge() error // reset-failed then restart |
| 3211 | func (a *systemApplier) gate(ctx, target string) error |
| 3212 | func (a *systemApplier) arm(plan bootstrapPlan) error // systemctlPlan verbatim |
| 3213 | func (a *systemApplier) rollback(ctx, req, reason string, j *systemJournal) error // Task 8c; stub returns the reason as an error |
| 3214 | ``` |
| 3215 | |
| 3216 | - [ ] **Step 1: Write the failing tests** |
| 3217 | |
| 3218 | Add to `systemapply_test.go`: |
| 3219 | ```go |
| 3220 | // applyHappy sets up a v1 box with a v2 candidate whose fake bootstrap |
| 3221 | // plan adds carlos-new-only.path and drops carlos-old-only.timer. |
| 3222 | func applyHappy(t *testing.T) (*applyFixture, systemRequest) { |
| 3223 | f := newApplyFixture(t, "stable", "v1") |
| 3224 | body := []byte("#!/bin/sh\necho carlos v2\n") |
| 3225 | req := f.promote("v2", body, false) |
| 3226 | staged := filepath.Join(f.a.stagingDir(), "carlos") |
| 3227 | f.answer["version:"+staged] = "carlos v2\n" |
| 3228 | f.answer["plan:"+staged] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service","carlos-new-only.path"],"inert":["carlos-restore-verify.timer"]}` |
| 3229 | f.answer["units:"+staged] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-new-only.path" |
| 3230 | // .prev, once made, answers as v1 and stages v1's units. |
| 3231 | prev := systemPrevPath(f.a.optDir) |
| 3232 | f.answer["version:"+prev] = "carlos v1\n" |
| 3233 | f.answer["plan:"+prev] = f.answer["plan:"+systemBinaryPath(f.a.optDir)] |
| 3234 | f.answer["units:"+prev] = f.answer["units:"+systemBinaryPath(f.a.optDir)] |
| 3235 | // After the install the on-disk binary answers v2. |
| 3236 | f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v2\n" |
| 3237 | return f, req |
| 3238 | } |
| 3239 | |
| 3240 | func TestApplyInstallsGatesArmsAndRecords(t *testing.T) { |
| 3241 | f, req := applyHappy(t) |
| 3242 | if err := f.a.Run(context.Background(), false); err != nil { |
| 3243 | t.Fatal(err) |
| 3244 | } |
| 3245 | r := f.result() |
| 3246 | if r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) { |
| 3247 | t.Fatalf("result = %+v", r) |
| 3248 | } |
| 3249 | ap, _ := readSystemApplied(f.a.optDir) |
| 3250 | if ap == nil || ap.Version != "v2" || ap.Channel != "stable" { |
| 3251 | t.Fatalf("applied = %+v", ap) |
| 3252 | } |
| 3253 | if j, _ := readSystemJournal(f.a.optDir); j != nil { |
| 3254 | t.Fatalf("journal left open: %+v", j) |
| 3255 | } |
| 3256 | if _, err := os.Stat(f.a.stagingDir()); err == nil { |
| 3257 | t.Fatal("staging/ left behind") |
| 3258 | } |
| 3259 | bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)) |
| 3260 | if !strings.Contains(string(bin), "carlos v2") { |
| 3261 | t.Fatal("binary not swapped") |
| 3262 | } |
| 3263 | prev, _ := os.ReadFile(systemPrevPath(f.a.optDir)) |
| 3264 | if !strings.Contains(string(prev), "carlos v1") { |
| 3265 | t.Fatal(".prev is not the old binary") |
| 3266 | } |
| 3267 | units := filepath.Join(f.a.etcDir, "systemd", "system") |
| 3268 | if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err != nil { |
| 3269 | t.Error("new unit not installed") |
| 3270 | } |
| 3271 | if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err == nil { |
| 3272 | t.Error("removed unit still installed") |
| 3273 | } |
| 3274 | if _, err := os.Stat(filepath.Join(f.a.etcDir, "sysusers.d", "carlos.conf")); err != nil { |
| 3275 | t.Error("sysusers not installed") |
| 3276 | } |
| 3277 | if _, err := os.Stat(filepath.Join(f.a.etcDir, "tmpfiles.d", "carlos.conf")); err != nil { |
| 3278 | t.Error("tmpfiles not installed") |
| 3279 | } |
| 3280 | if _, err := os.Stat(filepath.Join(f.a.etcDir, "registry.db")); err == nil { |
| 3281 | t.Error("staging data/ copied") |
| 3282 | } |
| 3283 | // Order of the systemctl calls is the spec. |
| 3284 | joined := strings.Join(f.calls, "\n") |
| 3285 | order := []string{ |
| 3286 | "systemctl stop carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service", |
| 3287 | "systemctl disable --now carlos-old-only.timer", |
| 3288 | "run systemd-sysusers", |
| 3289 | "run systemd-tmpfiles --create", |
| 3290 | "systemctl daemon-reload", |
| 3291 | "systemctl reset-failed carlos-edge", |
| 3292 | "systemctl restart carlos-edge", |
| 3293 | "systemctl is-active carlos-edge", |
| 3294 | "systemctl daemon-reload", |
| 3295 | "systemctl enable carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service carlos-new-only.path", |
| 3296 | "systemctl start carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-new-only.path", |
| 3297 | } |
| 3298 | at := 0 |
| 3299 | for _, want := range order { |
| 3300 | i := strings.Index(joined[at:], want) |
| 3301 | if i < 0 { |
| 3302 | t.Fatalf("call order broken: %q not found after position %d in:\n%s", want, at, joined) |
| 3303 | } |
| 3304 | at += i + len(want) |
| 3305 | } |
| 3306 | for _, c := range f.calls { |
| 3307 | if strings.HasPrefix(c, "systemctl start ") && strings.Contains(c, ".service") { |
| 3308 | t.Fatalf("apply started a service: %s", c) |
| 3309 | } |
| 3310 | } |
| 3311 | if strings.Contains(joined, "systemctl enable --now") { |
| 3312 | t.Fatal("enable --now is never used") |
| 3313 | } |
| 3314 | } |
| 3315 | |
| 3316 | func TestApplyGateSoakCatchesAFlap(t *testing.T) { |
| 3317 | f, req := applyHappy(t) |
| 3318 | // Edge comes up, then restarts once during the soak. |
| 3319 | restarts := 0 |
| 3320 | orig := f.a.ops.systemctl |
| 3321 | f.a.ops.systemctl = func(args ...string) (string, error) { |
| 3322 | if args[0] == "show" { |
| 3323 | restarts++ |
| 3324 | if restarts > 1 { |
| 3325 | return "1\n2000\n", nil |
| 3326 | } |
| 3327 | } |
| 3328 | return orig(args...) |
| 3329 | } |
| 3330 | err := f.a.Run(context.Background(), false) |
| 3331 | if err == nil { |
| 3332 | t.Fatal("a flapping edge passed the gate") |
| 3333 | } |
| 3334 | r := f.result() |
| 3335 | if r == nil || !r.matches("stable", "v2", req.PromotedAt) || r.Outcome == systemOutcomeApplied { |
| 3336 | t.Fatalf("result = %+v", r) |
| 3337 | } |
| 3338 | } |
| 3339 | |
| 3340 | func TestApplyPrevIsRefreshedOnlyWhenTheDiskIsKnownGood(t *testing.T) { |
| 3341 | // applied.json names v0 but the disk says v1 (a rollback-failed left a |
| 3342 | // bad v1 in place, .prev is the good v0): .prev must be left alone. |
| 3343 | f, _ := applyHappy(t) |
| 3344 | os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755) |
| 3345 | f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n" |
| 3346 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644) |
| 3347 | _ = f.a.Run(context.Background(), false) |
| 3348 | prev, _ := os.ReadFile(systemPrevPath(f.a.optDir)) |
| 3349 | if !strings.Contains(string(prev), "carlos v0") { |
| 3350 | t.Fatal(".prev was overwritten by a binary that never passed a gate") |
| 3351 | } |
| 3352 | } |
| 3353 | |
| 3354 | func TestApplyPrevUnitsComeFromPrevNotFromDisk(t *testing.T) { |
| 3355 | f, _ := applyHappy(t) |
| 3356 | os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755) |
| 3357 | f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n" |
| 3358 | f.answer["plan:"+systemPrevPath(f.a.optDir)] = `{"enabled":["carlos-edge.service"],"inert":[]}` |
| 3359 | f.answer["units:"+systemPrevPath(f.a.optDir)] = "carlos-edge.service carlos-v0-only.timer" |
| 3360 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644) |
| 3361 | // Inspect the journal mid-flight by failing the gate. |
| 3362 | f.edgeUp = false |
| 3363 | _ = f.a.Run(context.Background(), false) |
| 3364 | // units_removed must be computed against .prev's set (v0-only.timer), |
| 3365 | // not the on-disk v1's (old-only.timer). |
| 3366 | found := false |
| 3367 | for _, c := range f.calls { |
| 3368 | if c == "run carlos.prev bootstrap -root "+filepath.Join(f.a.stagingDir(), "prev")+" -plan-json" { |
| 3369 | found = true |
| 3370 | } |
| 3371 | } |
| 3372 | if !found { |
| 3373 | t.Fatalf("prev units not staged from carlos.prev:\n%s", strings.Join(f.calls, "\n")) |
| 3374 | } |
| 3375 | } |
| 3376 | ``` |
| 3377 | |
| 3378 | - [ ] **Step 2: Run to verify failure** |
| 3379 | |
| 3380 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyInstalls|TestApplyGate|TestApplyPrev' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 3381 | Expected: FAIL with `errNotYetImplemented`. |
| 3382 | |
| 3383 | - [ ] **Step 3: Implement the install path** |
| 3384 | |
| 3385 | Replace the `install` stub and add the helpers: |
| 3386 | ```go |
| 3387 | func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error { |
| 3388 | target := p.Current.Version |
| 3389 | prevOK, err := a.decidePrev(ctx) |
| 3390 | if err != nil { |
| 3391 | return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err)) |
| 3392 | } |
| 3393 | if !prevOK { |
| 3394 | return a.record(req, fmt.Errorf("%w: no known-good carlos.prev to fall back to", errSystemRefused)) |
| 3395 | } |
| 3396 | newDir, prevDir := filepath.Join(a.stagingDir(), "new"), filepath.Join(a.stagingDir(), "prev") |
| 3397 | newPlan, newUnits, err := a.stagePlan(ctx, staged, newDir) |
| 3398 | if err != nil { |
| 3399 | return a.record(req, fmt.Errorf("%w: candidate bootstrap: %v", errSystemRefused, err)) |
| 3400 | } |
| 3401 | prevPlan, prevUnits, err := a.stagePlan(ctx, systemPrevPath(a.optDir), prevDir) |
| 3402 | if err != nil { |
| 3403 | return a.record(req, fmt.Errorf("%w: carlos.prev bootstrap: %v", errSystemRefused, err)) |
| 3404 | } |
| 3405 | j := systemJournal{Phase: systemPhaseInstalling, Target: target, PrevBuild: a.prevBuild(ctx), |
| 3406 | UnitsAdded: diff(newUnits, prevUnits), UnitsRemoved: diff(prevUnits, newUnits), StartedAt: a.ops.now().UTC()} |
| 3407 | if err := a.journal(j); err != nil { |
| 3408 | return err |
| 3409 | } |
| 3410 | fail := func(step string, err error) error { |
| 3411 | return a.rollback(ctx, req, fmt.Sprintf("%s: %v", step, err), &j, prevPlan) |
| 3412 | } |
| 3413 | if _, err := a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...); err != nil { |
| 3414 | a.ops.logf("system apply: quiesce: %v (continuing)", err) |
| 3415 | } |
| 3416 | if err := os.Rename(staged, systemBinaryPath(a.optDir)); err != nil { |
| 3417 | return fail("install", err) |
| 3418 | } |
| 3419 | if err := os.Chown(systemBinaryPath(a.optDir), 0, 0); err != nil && !errors.Is(err, os.ErrPermission) { |
| 3420 | a.ops.logf("system apply: chown root: %v", err) |
| 3421 | } |
| 3422 | j.Phase = systemPhaseInstalled |
| 3423 | if err := a.journal(j); err != nil { |
| 3424 | return fail("journal", err) |
| 3425 | } |
| 3426 | if err := a.installUnits(newDir, j.UnitsRemoved); err != nil { |
| 3427 | return fail("units", err) |
| 3428 | } |
| 3429 | j.Phase = systemPhaseUnits |
| 3430 | if err := a.journal(j); err != nil { |
| 3431 | return fail("journal", err) |
| 3432 | } |
| 3433 | if err := a.restartEdge(); err != nil { |
| 3434 | return fail("restart", err) |
| 3435 | } |
| 3436 | j.Phase = systemPhaseRestarted |
| 3437 | if err := a.journal(j); err != nil { |
| 3438 | return fail("journal", err) |
| 3439 | } |
| 3440 | if err := a.gate(ctx, target); err != nil { |
| 3441 | return fail("gate", err) |
| 3442 | } |
| 3443 | j.Phase = systemPhaseGated |
| 3444 | if err := a.journal(j); err != nil { |
| 3445 | return fail("journal", err) |
| 3446 | } |
| 3447 | return a.finish(req, target, newPlan) |
| 3448 | } |
| 3449 | |
| 3450 | // finish is steps 11–12, and what recovery runs forward from phase gated. |
| 3451 | func (a *systemApplier) finish(req systemRequest, target string, plan bootstrapPlan) error { |
| 3452 | if err := a.arm(plan); err != nil { |
| 3453 | a.ops.logf("system apply: arm: %v", err) |
| 3454 | } |
| 3455 | if err := writeJSONDurable(a.file("applied.json"), systemApplied{Version: target, Channel: a.channel, At: a.ops.now().UTC()}, 0o644); err != nil { |
| 3456 | return err |
| 3457 | } |
| 3458 | if err := a.writeResult(req, systemOutcomeApplied, ""); err != nil { |
| 3459 | return err |
| 3460 | } |
| 3461 | os.Remove(a.file("journal.json")) |
| 3462 | return syncDirBestEffort(a.rootDir()) |
| 3463 | } |
| 3464 | |
| 3465 | // decidePrev: refresh carlos.prev from the on-disk binary only when that |
| 3466 | // binary is known good — applied.json names it, or nothing has ever been |
| 3467 | // applied and there is no .prev (a hand-rolled box whose running binary |
| 3468 | // passed the script's own post-condition). Otherwise leave .prev alone. |
| 3469 | func (a *systemApplier) decidePrev(ctx context.Context) (bool, error) { |
| 3470 | prev := systemPrevPath(a.optDir) |
| 3471 | _, prevErr := os.Stat(prev) |
| 3472 | applied, err := readSystemApplied(a.optDir) |
| 3473 | if err != nil { |
| 3474 | return false, err |
| 3475 | } |
| 3476 | disk := a.onDiskBuild(ctx) |
| 3477 | known := (applied != nil && applied.Version == disk && disk != "") || (applied == nil && prevErr != nil) |
| 3478 | if !known { |
| 3479 | return prevErr == nil, nil |
| 3480 | } |
| 3481 | if err := a.journal(systemJournal{Phase: systemPhaseBackingUp, Target: "", PrevBuild: disk, StartedAt: a.ops.now().UTC()}); err != nil { |
| 3482 | return false, err |
| 3483 | } |
| 3484 | if err := copyFileDurable(systemBinaryPath(a.optDir), prev, 0o755); err != nil { |
| 3485 | return false, err |
| 3486 | } |
| 3487 | os.Remove(a.file("journal.json")) |
| 3488 | return true, nil |
| 3489 | } |
| 3490 | |
| 3491 | func (a *systemApplier) prevBuild(ctx context.Context) string { |
| 3492 | out, err := a.ops.run(ctx, systemVersionWait, systemPrevPath(a.optDir), "version") |
| 3493 | if err != nil { |
| 3494 | return "" |
| 3495 | } |
| 3496 | return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos ")) |
| 3497 | } |
| 3498 | |
| 3499 | // stagePlan runs `<binary> bootstrap -root <dir> -plan-json` and returns |
| 3500 | // that binary's arming plan and the unit files it staged. |
| 3501 | func (a *systemApplier) stagePlan(ctx context.Context, binary, dir string) (bootstrapPlan, []string, error) { |
| 3502 | os.RemoveAll(dir) |
| 3503 | if err := os.MkdirAll(dir, 0o700); err != nil { |
| 3504 | return bootstrapPlan{}, nil, err |
| 3505 | } |
| 3506 | out, err := a.ops.run(ctx, systemBootstrapWait, binary, "bootstrap", "-root", dir, "-plan-json") |
| 3507 | if err != nil { |
| 3508 | return bootstrapPlan{}, nil, err |
| 3509 | } |
| 3510 | var plan bootstrapPlan |
| 3511 | if err := json.Unmarshal([]byte(out), &plan); err != nil { |
| 3512 | return bootstrapPlan{}, nil, fmt.Errorf("plan: %w", err) |
| 3513 | } |
| 3514 | entries, err := os.ReadDir(filepath.Join(dir, "etc", "systemd", "system")) |
| 3515 | if err != nil { |
| 3516 | return bootstrapPlan{}, nil, err |
| 3517 | } |
| 3518 | var units []string |
| 3519 | for _, e := range entries { |
| 3520 | if !e.IsDir() { |
| 3521 | units = append(units, e.Name()) |
| 3522 | } |
| 3523 | } |
| 3524 | return plan, units, nil |
| 3525 | } |
| 3526 | |
| 3527 | func (a *systemApplier) journal(j systemJournal) error { |
| 3528 | return writeJSONDurable(a.file("journal.json"), j, 0o600) |
| 3529 | } |
| 3530 | |
| 3531 | // installUnits: remove what the target build dropped, copy its units in, |
| 3532 | // then sysusers BEFORE tmpfiles, then daemon-reload. Never the staging |
| 3533 | // root's data/. |
| 3534 | func (a *systemApplier) installUnits(stagedRoot string, remove []string) error { |
| 3535 | unitDir := filepath.Join(a.etcDir, "systemd", "system") |
| 3536 | for _, u := range remove { |
| 3537 | if _, err := a.ops.systemctl("disable", "--now", u); err != nil { |
| 3538 | a.ops.logf("system apply: disable %s: %v", u, err) |
| 3539 | } |
| 3540 | os.Remove(filepath.Join(unitDir, u)) |
| 3541 | } |
| 3542 | entries, err := os.ReadDir(filepath.Join(stagedRoot, "etc", "systemd", "system")) |
| 3543 | if err != nil { |
| 3544 | return err |
| 3545 | } |
| 3546 | for _, e := range entries { |
| 3547 | if e.IsDir() { |
| 3548 | continue |
| 3549 | } |
| 3550 | if err := copyFileDurable(filepath.Join(stagedRoot, "etc", "systemd", "system", e.Name()), filepath.Join(unitDir, e.Name()), 0o644); err != nil { |
| 3551 | return err |
| 3552 | } |
| 3553 | } |
| 3554 | sys := filepath.Join(stagedRoot, "etc", "sysusers.d", "carlos.conf") |
| 3555 | if _, err := os.Stat(sys); err == nil { |
| 3556 | if err := os.MkdirAll(filepath.Join(a.etcDir, "sysusers.d"), 0o755); err != nil { |
| 3557 | return err |
| 3558 | } |
| 3559 | dst := filepath.Join(a.etcDir, "sysusers.d", "carlos.conf") |
| 3560 | if err := copyFileDurable(sys, dst, 0o644); err != nil { |
| 3561 | return err |
| 3562 | } |
| 3563 | if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-sysusers", dst); err != nil { |
| 3564 | return fmt.Errorf("systemd-sysusers: %w", err) |
| 3565 | } |
| 3566 | } |
| 3567 | tmp := filepath.Join(stagedRoot, "etc", "tmpfiles.d", "carlos.conf") |
| 3568 | if err := os.MkdirAll(filepath.Join(a.etcDir, "tmpfiles.d"), 0o755); err != nil { |
| 3569 | return err |
| 3570 | } |
| 3571 | dst := filepath.Join(a.etcDir, "tmpfiles.d", "carlos.conf") |
| 3572 | if err := copyFileDurable(tmp, dst, 0o644); err != nil { |
| 3573 | return err |
| 3574 | } |
| 3575 | if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-tmpfiles", "--create", dst); err != nil { |
| 3576 | a.ops.logf("system apply: systemd-tmpfiles: %v (continuing, as the script does)", err) |
| 3577 | } |
| 3578 | _, err = a.ops.systemctl("daemon-reload") |
| 3579 | return err |
| 3580 | } |
| 3581 | |
| 3582 | // restartEdge: reset-failed FIRST — a start-limited unit refuses restart — |
| 3583 | // then restart. |
| 3584 | func (a *systemApplier) restartEdge() error { |
| 3585 | a.ops.systemctl("reset-failed", systemEdgeUnit) |
| 3586 | _, err := a.ops.systemctl("restart", systemEdgeUnit) |
| 3587 | return err |
| 3588 | } |
| 3589 | |
| 3590 | // gate: within gateWait the edge is active, answers on loopback, and the |
| 3591 | // installed binary reports the target; then a soak during which neither |
| 3592 | // NRestarts nor ActiveEnterTimestampMonotonic moves. |
| 3593 | func (a *systemApplier) gate(ctx context.Context, target string) error { |
| 3594 | wait, soak := a.gateWait, a.soak |
| 3595 | deadline := a.ops.now().Add(wait) |
| 3596 | var lastErr error |
| 3597 | for { |
| 3598 | lastErr = a.gateOnce(ctx, target) |
| 3599 | if lastErr == nil { |
| 3600 | break |
| 3601 | } |
| 3602 | if !a.ops.now().Before(deadline) { |
| 3603 | return lastErr |
| 3604 | } |
| 3605 | a.ops.sleep(time.Second) |
| 3606 | } |
| 3607 | before, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit) |
| 3608 | if err != nil { |
| 3609 | return err |
| 3610 | } |
| 3611 | a.ops.sleep(soak) |
| 3612 | after, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit) |
| 3613 | if err != nil { |
| 3614 | return err |
| 3615 | } |
| 3616 | if strings.TrimSpace(before) != strings.TrimSpace(after) { |
| 3617 | return fmt.Errorf("edge restarted during the soak (%s → %s)", strings.ReplaceAll(strings.TrimSpace(before), "\n", "/"), strings.ReplaceAll(strings.TrimSpace(after), "\n", "/")) |
| 3618 | } |
| 3619 | if err := a.gateOnce(ctx, target); err != nil { |
| 3620 | return err |
| 3621 | } |
| 3622 | return nil |
| 3623 | } |
| 3624 | |
| 3625 | func (a *systemApplier) gateOnce(ctx context.Context, target string) error { |
| 3626 | if out, err := a.ops.systemctl("is-active", systemEdgeUnit); err != nil || strings.TrimSpace(out) != "active" { |
| 3627 | return fmt.Errorf("%s is %s", systemEdgeUnit, strings.TrimSpace(out)) |
| 3628 | } |
| 3629 | if err := a.ops.probeEdge(); err != nil { |
| 3630 | return fmt.Errorf("edge probe: %w", err) |
| 3631 | } |
| 3632 | if got := a.onDiskBuild(ctx); got != target { |
| 3633 | return fmt.Errorf("installed binary reports %q, not %s", got, target) |
| 3634 | } |
| 3635 | return nil |
| 3636 | } |
| 3637 | |
| 3638 | // arm is systemctlPlan verbatim: enable all, start timers and paths, never |
| 3639 | // a service — which is why apply can call it from inside itself. |
| 3640 | func (a *systemApplier) arm(plan bootstrapPlan) error { |
| 3641 | for _, args := range systemctlPlan(plan.Enabled) { |
| 3642 | if _, err := a.ops.systemctl(args...); err != nil { |
| 3643 | return fmt.Errorf("systemctl %s: %w", strings.Join(args, " "), err) |
| 3644 | } |
| 3645 | } |
| 3646 | return nil |
| 3647 | } |
| 3648 | |
| 3649 | func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error { |
| 3650 | return errors.New("rollback not implemented: " + reason) // Task 8c |
| 3651 | } |
| 3652 | |
| 3653 | func diff(have, drop []string) []string { |
| 3654 | skip := map[string]bool{} |
| 3655 | for _, d := range drop { |
| 3656 | skip[d] = true |
| 3657 | } |
| 3658 | var out []string |
| 3659 | for _, h := range have { |
| 3660 | if !skip[h] { |
| 3661 | out = append(out, h) |
| 3662 | } |
| 3663 | } |
| 3664 | return out |
| 3665 | } |
| 3666 | |
| 3667 | // copyFileDurable is writeFileDurable for a file too large to hold: temp, |
| 3668 | // fsync, rename, directory fsync. |
| 3669 | func copyFileDurable(src, dst string, mode os.FileMode) (err error) { |
| 3670 | in, err := os.Open(src) |
| 3671 | if err != nil { |
| 3672 | return err |
| 3673 | } |
| 3674 | defer in.Close() |
| 3675 | tmp := dst + ".tmp" |
| 3676 | out, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode) |
| 3677 | if err != nil { |
| 3678 | return err |
| 3679 | } |
| 3680 | defer func() { |
| 3681 | if err != nil { |
| 3682 | out.Close() |
| 3683 | os.Remove(tmp) |
| 3684 | } |
| 3685 | }() |
| 3686 | if _, err = io.Copy(out, in); err != nil { |
| 3687 | return err |
| 3688 | } |
| 3689 | if err = out.Chmod(mode); err != nil { |
| 3690 | return err |
| 3691 | } |
| 3692 | if err = out.Sync(); err != nil { |
| 3693 | return err |
| 3694 | } |
| 3695 | if err = out.Close(); err != nil { |
| 3696 | return err |
| 3697 | } |
| 3698 | if err = os.Rename(tmp, dst); err != nil { |
| 3699 | return err |
| 3700 | } |
| 3701 | return syncDirBestEffort(filepath.Dir(dst)) |
| 3702 | } |
| 3703 | |
| 3704 | func syncDirBestEffort(dir string) error { |
| 3705 | d, err := os.Open(dir) |
| 3706 | if err != nil { |
| 3707 | return err |
| 3708 | } |
| 3709 | defer d.Close() |
| 3710 | return d.Sync() |
| 3711 | } |
| 3712 | ``` |
| 3713 | The test for the flap expects `TestApplyGateSoakCatchesAFlap` to end in a rollback; with the 8c stub it ends in an error from `rollback`, and `result.json` is not yet written by the stub. Make the stub write `result.json` with `outcome: rolled-back` and reason, then return the error — that is the 8c contract in miniature and keeps this task's tests honest: |
| 3714 | ```go |
| 3715 | func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error { |
| 3716 | _ = a.writeResult(req, systemOutcomeRolledBack, reason) |
| 3717 | return errors.New("rollback not implemented: " + reason) |
| 3718 | } |
| 3719 | ``` |
| 3720 | |
| 3721 | - [ ] **Step 4: Run the tests** |
| 3722 | |
| 3723 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 3724 | Expected: PASS. `TestApplyInstallsGatesArmsAndRecords` checks the exact systemctl order; if a call is missing, fix the code, not the expected list, unless the spec says otherwise. |
| 3725 | |
| 3726 | - [ ] **Step 5: Commit** |
| 3727 | |
| 3728 | ```bash |
| 3729 | gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF' |
| 3730 | host: carlos system apply — install, gate, arm, record |
| 3731 | |
| 3732 | .prev refreshed only when the disk is known good; both unit sets |
| 3733 | staged from the binaries that own them; quiesce before the rename; |
| 3734 | sysusers before tmpfiles; reset-failed before restart; a 15 s soak on |
| 3735 | NRestarts; arming after the gate via systemctlPlan. |
| 3736 | |
| 3737 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 3738 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 3739 | EOF |
| 3740 | ``` |
| 3741 | |
| 3742 | --- |
| 3743 | |
| 3744 | ### Task 8c: `carlos system apply` — rollback, recovery, `--recover`, the real `sysOps` |
| 3745 | |
| 3746 | **Files:** |
| 3747 | - Modify: `internal/host/systemapply.go` (replace `rollback` and `recoverIfNeeded`; add `RunSystemApply` and `productionSysOps`) |
| 3748 | - Test: `internal/host/systemapply_test.go` (add) |
| 3749 | |
| 3750 | **Interfaces:** |
| 3751 | - Produces: |
| 3752 | ```go |
| 3753 | func RunSystemApply(args []string) // flags: -recover; exits 1 on a non-nil Run error |
| 3754 | func productionSysOps() sysOps |
| 3755 | func (a *systemApplier) rollback(ctx, req, reason, j *systemJournal, prevPlan bootstrapPlan) error |
| 3756 | func (a *systemApplier) recoverIfNeeded(ctx) error |
| 3757 | func (a *systemApplier) restoreFiles(ctx, j *systemJournal) (bootstrapPlan, error) // files only: binary, units, sysusers, tmpfiles, daemon-reload, enable |
| 3758 | ``` |
| 3759 | |
| 3760 | - [ ] **Step 1: Write the failing tests** |
| 3761 | |
| 3762 | ```go |
| 3763 | func TestApplyRollsBackWhenTheGateFails(t *testing.T) { |
| 3764 | f, req := applyHappy(t) |
| 3765 | f.edgeUp = false |
| 3766 | err := f.a.Run(context.Background(), false) |
| 3767 | if err == nil { |
| 3768 | t.Fatal("a failed gate must exit non-zero") |
| 3769 | } |
| 3770 | r := f.result() |
| 3771 | if r == nil || r.Outcome != systemOutcomeRolledBack || !r.matches("stable", "v2", req.PromotedAt) || !strings.Contains(r.Reason, "gate") { |
| 3772 | t.Fatalf("result = %+v", r) |
| 3773 | } |
| 3774 | bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)) |
| 3775 | if !strings.Contains(string(bin), "carlos v1") { |
| 3776 | t.Fatal("binary not restored") |
| 3777 | } |
| 3778 | if _, err := os.Stat(systemPrevPath(f.a.optDir)); err != nil { |
| 3779 | t.Fatal(".prev consumed by the rollback") |
| 3780 | } |
| 3781 | units := filepath.Join(f.a.etcDir, "systemd", "system") |
| 3782 | if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err == nil { |
| 3783 | t.Error("added unit survived the rollback") |
| 3784 | } |
| 3785 | if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err != nil { |
| 3786 | t.Error("removed unit not restored") |
| 3787 | } |
| 3788 | if ap, _ := readSystemApplied(f.a.optDir); ap != nil { |
| 3789 | t.Fatalf("applied.json written by a rollback: %+v", ap) |
| 3790 | } |
| 3791 | if j, _ := readSystemJournal(f.a.optDir); j != nil { |
| 3792 | t.Fatal("journal left open after rollback") |
| 3793 | } |
| 3794 | joined := strings.Join(f.calls, "\n") |
| 3795 | if !strings.Contains(joined, "systemctl disable --now carlos-new-only.path") { |
| 3796 | t.Error("added unit not disabled") |
| 3797 | } |
| 3798 | // reset-failed precedes the rollback's restart too, and the previous |
| 3799 | // plan is armed afterwards. |
| 3800 | i := strings.LastIndex(joined, "systemctl reset-failed carlos-edge") |
| 3801 | k := strings.LastIndex(joined, "systemctl restart carlos-edge") |
| 3802 | m := strings.LastIndex(joined, "systemctl start carlos-adopt.timer") |
| 3803 | if !(i < k && k < m) { |
| 3804 | t.Fatalf("rollback order wrong:\n%s", joined) |
| 3805 | } |
| 3806 | } |
| 3807 | |
| 3808 | func TestApplyRollbackFailedLeavesPrevAndAppliedAlone(t *testing.T) { |
| 3809 | f, req := applyHappy(t) |
| 3810 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v1", Channel: "stable", At: f.now}, 0o644) |
| 3811 | f.edgeUp = false // never comes up, on either binary |
| 3812 | if err := f.a.Run(context.Background(), false); err == nil { |
| 3813 | t.Fatal("must exit non-zero") |
| 3814 | } |
| 3815 | r := f.result() |
| 3816 | if r == nil || r.Outcome != systemOutcomeRollbackFailed || !r.matches("stable", "v2", req.PromotedAt) { |
| 3817 | t.Fatalf("result = %+v", r) |
| 3818 | } |
| 3819 | ap, _ := readSystemApplied(f.a.optDir) |
| 3820 | if ap == nil || ap.Version != "v1" { |
| 3821 | t.Fatalf("applied = %+v, must still name v1", ap) |
| 3822 | } |
| 3823 | prev, _ := os.ReadFile(systemPrevPath(f.a.optDir)) |
| 3824 | if !strings.Contains(string(prev), "carlos v1") { |
| 3825 | t.Fatal(".prev damaged") |
| 3826 | } |
| 3827 | // A second apply of a NEW version must not refresh .prev from the |
| 3828 | // unproven disk binary. |
| 3829 | f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v1\n" // rollback restored v1 on disk |
| 3830 | os.Remove(systemRequestPath(f.a.dataDir)) |
| 3831 | f.promote("v3", []byte("#!/bin/sh\necho carlos v3\n"), false) |
| 3832 | f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v3\n" |
| 3833 | f.answer["plan:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["plan:"+systemPrevPath(f.a.optDir)] |
| 3834 | f.answer["units:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["units:"+systemPrevPath(f.a.optDir)] |
| 3835 | _ = f.a.Run(context.Background(), false) |
| 3836 | prev2, _ := os.ReadFile(systemPrevPath(f.a.optDir)) |
| 3837 | if string(prev2) != string(prev) { |
| 3838 | t.Fatal(".prev refreshed after rollback-failed") |
| 3839 | } |
| 3840 | } |
| 3841 | |
| 3842 | func TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime(t *testing.T) { |
| 3843 | for _, phase := range []string{systemPhaseInstalling, systemPhaseInstalled, systemPhaseUnits, systemPhaseRestarted} { |
| 3844 | t.Run(phase, func(t *testing.T) { |
| 3845 | f, _ := applyHappy(t) |
| 3846 | // Simulate: v2 installed, journal open at phase, box rebooted. |
| 3847 | os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755) |
| 3848 | os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755) |
| 3849 | os.WriteFile(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path"), []byte("x"), 0o644) |
| 3850 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"), |
| 3851 | systemJournal{Phase: phase, Target: "v2", PrevBuild: "v1", UnitsAdded: []string{"carlos-new-only.path"}, UnitsRemoved: []string{"carlos-old-only.timer"}, StartedAt: f.now}, 0o600) |
| 3852 | f.edgeStartedThisBoot = false |
| 3853 | if err := f.a.Run(context.Background(), true); err != nil { |
| 3854 | t.Fatal(err) |
| 3855 | } |
| 3856 | bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)) |
| 3857 | if !strings.Contains(string(bin), "carlos v1") { |
| 3858 | t.Fatal("binary not restored") |
| 3859 | } |
| 3860 | if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path")); err == nil { |
| 3861 | t.Error("added unit survived") |
| 3862 | } |
| 3863 | if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-old-only.timer")); err != nil { |
| 3864 | t.Error("removed unit not restored") |
| 3865 | } |
| 3866 | joined := strings.Join(f.calls, "\n") |
| 3867 | if strings.Contains(joined, "systemctl restart carlos-edge") || strings.Contains(joined, "systemctl start ") { |
| 3868 | t.Fatalf("boot recovery restarted or started something:\n%s", joined) |
| 3869 | } |
| 3870 | if !strings.Contains(joined, "systemctl enable carlos-edge.service") { |
| 3871 | t.Fatal("previous plan not enabled for the boot") |
| 3872 | } |
| 3873 | if r := f.result(); r == nil || r.Outcome != systemOutcomeRolledBack || !strings.Contains(r.Reason, phase) { |
| 3874 | t.Fatalf("result = %+v", r) |
| 3875 | } |
| 3876 | if j, _ := readSystemJournal(f.a.optDir); j != nil { |
| 3877 | t.Fatal("journal left open") |
| 3878 | } |
| 3879 | }) |
| 3880 | } |
| 3881 | // Runtime (OnFailure=): same, plus a non-blocking restart and the |
| 3882 | // previous plan's timers/paths started. |
| 3883 | f, _ := applyHappy(t) |
| 3884 | os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755) |
| 3885 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"), |
| 3886 | systemJournal{Phase: systemPhaseUnits, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600) |
| 3887 | f.edgeStartedThisBoot = true |
| 3888 | if err := f.a.Run(context.Background(), true); err != nil { |
| 3889 | t.Fatal(err) |
| 3890 | } |
| 3891 | joined := strings.Join(f.calls, "\n") |
| 3892 | for _, want := range []string{"systemctl stop carlos-adopt.timer", "systemctl reset-failed carlos-edge", "systemctl restart --no-block carlos-edge", "systemctl start carlos-adopt.timer"} { |
| 3893 | if !strings.Contains(joined, want) { |
| 3894 | t.Errorf("runtime recovery lacks %q:\n%s", want, joined) |
| 3895 | } |
| 3896 | } |
| 3897 | } |
| 3898 | |
| 3899 | func TestRecoveryFinishesForwardFromGated(t *testing.T) { |
| 3900 | f, req := applyHappy(t) |
| 3901 | os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755) |
| 3902 | writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644) |
| 3903 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"), |
| 3904 | systemJournal{Phase: systemPhaseGated, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600) |
| 3905 | if err := f.a.Run(context.Background(), true); err != nil { |
| 3906 | t.Fatal(err) |
| 3907 | } |
| 3908 | bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)) |
| 3909 | if !strings.Contains(string(bin), "carlos v2") { |
| 3910 | t.Fatal("a gated apply was undone") |
| 3911 | } |
| 3912 | ap, _ := readSystemApplied(f.a.optDir) |
| 3913 | if ap == nil || ap.Version != "v2" { |
| 3914 | t.Fatalf("applied = %+v", ap) |
| 3915 | } |
| 3916 | if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied { |
| 3917 | t.Fatalf("result = %+v", r) |
| 3918 | } |
| 3919 | } |
| 3920 | |
| 3921 | func TestRecoveryRemovesTheJournalLast(t *testing.T) { |
| 3922 | // A restart that fails mid-recovery leaves the journal for the next run. |
| 3923 | f, _ := applyHappy(t) |
| 3924 | os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755) |
| 3925 | writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"), |
| 3926 | systemJournal{Phase: systemPhaseInstalled, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600) |
| 3927 | orig := f.a.ops.systemctl |
| 3928 | f.a.ops.systemctl = func(args ...string) (string, error) { |
| 3929 | if args[0] == "daemon-reload" { |
| 3930 | return "", errors.New("systemd busy") |
| 3931 | } |
| 3932 | return orig(args...) |
| 3933 | } |
| 3934 | if err := f.a.Run(context.Background(), true); err == nil { |
| 3935 | t.Fatal("a failed recovery must exit non-zero") |
| 3936 | } |
| 3937 | if j, _ := readSystemJournal(f.a.optDir); j == nil { |
| 3938 | t.Fatal("journal removed although recovery did not complete") |
| 3939 | } |
| 3940 | } |
| 3941 | ``` |
| 3942 | |
| 3943 | - [ ] **Step 2: Run to verify failure** |
| 3944 | |
| 3945 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyRolls|TestApplyRollback|TestRecovery' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 3946 | Expected: FAIL ("rollback not implemented", "recovery not implemented"). |
| 3947 | |
| 3948 | - [ ] **Step 3: Implement** |
| 3949 | |
| 3950 | Replace `rollback` and `recoverIfNeeded`; add `restoreFiles`, `RunSystemApply`, `productionSysOps`: |
| 3951 | ```go |
| 3952 | // rollback is step 13: back to carlos.prev and its units, restart, gate, |
| 3953 | // arm the previous plan. applied.json is untouched — the .prev build is |
| 3954 | // what passed a gate, and it is what applied.json already names. |
| 3955 | func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error { |
| 3956 | a.ops.logf("system apply: %s %s: rolling back: %s", a.channel, j.Target, reason) |
| 3957 | plan, err := a.restoreFiles(ctx, j) |
| 3958 | if err != nil { |
| 3959 | _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restore: "+err.Error()) |
| 3960 | os.Remove(a.file("journal.json")) |
| 3961 | return fmt.Errorf("system apply: rollback of %s failed: %v", j.Target, err) |
| 3962 | } |
| 3963 | if err := a.restartEdge(); err != nil { |
| 3964 | _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restart: "+err.Error()) |
| 3965 | os.Remove(a.file("journal.json")) |
| 3966 | return fmt.Errorf("system apply: rollback restart failed: %v", err) |
| 3967 | } |
| 3968 | if err := a.gate(ctx, j.PrevBuild); err != nil { |
| 3969 | _ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; previous build failed its gate too: "+err.Error()) |
| 3970 | os.Remove(a.file("journal.json")) |
| 3971 | return fmt.Errorf("system apply: rollback gate failed: %v", err) |
| 3972 | } |
| 3973 | if err := a.arm(plan); err != nil { |
| 3974 | a.ops.logf("system apply: arm previous plan: %v", err) |
| 3975 | } |
| 3976 | if err := a.writeResult(req, systemOutcomeRolledBack, reason); err != nil { |
| 3977 | return err |
| 3978 | } |
| 3979 | os.Remove(a.file("journal.json")) |
| 3980 | return fmt.Errorf("system apply: %s rolled back: %s", j.Target, reason) |
| 3981 | } |
| 3982 | |
| 3983 | // restoreFiles is the files-only half shared by rollback and recovery: |
| 3984 | // carlos.prev over carlos (keeping .prev), units_added disabled and |
| 3985 | // removed, the previous unit set copied back (bringing units_removed |
| 3986 | // back), sysusers, tmpfiles, daemon-reload, and `enable` of the previous |
| 3987 | // plan so it comes up at boot. No restart, no start. |
| 3988 | func (a *systemApplier) restoreFiles(ctx context.Context, j *systemJournal) (bootstrapPlan, error) { |
| 3989 | prev := systemPrevPath(a.optDir) |
| 3990 | if _, err := os.Stat(prev); err != nil { |
| 3991 | return bootstrapPlan{}, fmt.Errorf("no carlos.prev: %w", err) |
| 3992 | } |
| 3993 | a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...) |
| 3994 | if err := copyFileDurable(prev, systemBinaryPath(a.optDir), 0o755); err != nil { |
| 3995 | return bootstrapPlan{}, err |
| 3996 | } |
| 3997 | prevDir := filepath.Join(a.stagingDir(), "prev") |
| 3998 | plan, _, err := a.stagePlan(ctx, prev, prevDir) |
| 3999 | if err != nil { |
| 4000 | return bootstrapPlan{}, fmt.Errorf("carlos.prev bootstrap: %w", err) |
| 4001 | } |
| 4002 | if err := a.installUnits(prevDir, j.UnitsAdded); err != nil { |
| 4003 | return bootstrapPlan{}, err |
| 4004 | } |
| 4005 | if _, err := a.ops.systemctl(append([]string{"enable"}, plan.Enabled...)...); err != nil { |
| 4006 | return bootstrapPlan{}, err |
| 4007 | } |
| 4008 | return plan, nil |
| 4009 | } |
| 4010 | |
| 4011 | // recoverIfNeeded: an open journal means a previous run was cut off. |
| 4012 | // Before gated → back, files only, restart only if the edge has started |
| 4013 | // this boot; gated → forward. The journal is removed LAST. |
| 4014 | func (a *systemApplier) recoverIfNeeded(ctx context.Context) error { |
| 4015 | j, err := readSystemJournal(a.optDir) |
| 4016 | if err != nil || j == nil { |
| 4017 | return err |
| 4018 | } |
| 4019 | req, _ := a.readRequest() |
| 4020 | if req.Version == "" { |
| 4021 | req = systemRequest{Version: j.Target, PromotedAt: j.StartedAt} |
| 4022 | } |
| 4023 | a.ops.logf("system apply: recovering an apply of %s interrupted at %s", j.Target, j.Phase) |
| 4024 | switch j.Phase { |
| 4025 | case systemPhaseGated: |
| 4026 | plan, _, err := a.stagePlan(ctx, systemBinaryPath(a.optDir), filepath.Join(a.stagingDir(), "new")) |
| 4027 | if err != nil { |
| 4028 | return fmt.Errorf("system apply: recovery: %w", err) |
| 4029 | } |
| 4030 | return a.finish(req, j.Target, plan) |
| 4031 | case systemPhaseBackingUp: |
| 4032 | os.Remove(systemPrevPath(a.optDir) + ".tmp") |
| 4033 | os.Remove(a.file("journal.json")) |
| 4034 | return nil |
| 4035 | } |
| 4036 | plan, err := a.restoreFiles(ctx, j) |
| 4037 | if err != nil { |
| 4038 | return fmt.Errorf("system apply: recovery: %w", err) // journal kept: the next run recovers again |
| 4039 | } |
| 4040 | if a.ops.edgeStartedThisBoot() { |
| 4041 | a.ops.systemctl("reset-failed", systemEdgeUnit) |
| 4042 | if _, err := a.ops.systemctl("restart", "--no-block", systemEdgeUnit); err != nil { |
| 4043 | return fmt.Errorf("system apply: recovery: restart: %w", err) |
| 4044 | } |
| 4045 | var armed []string |
| 4046 | for _, u := range plan.Enabled { |
| 4047 | if strings.HasSuffix(u, ".timer") || strings.HasSuffix(u, ".path") { |
| 4048 | armed = append(armed, u) |
| 4049 | } |
| 4050 | } |
| 4051 | if len(armed) > 0 { |
| 4052 | a.ops.systemctl(append([]string{"start"}, armed...)...) |
| 4053 | } |
| 4054 | } |
| 4055 | if err := a.writeResult(req, systemOutcomeRolledBack, "interrupted at "+j.Phase); err != nil { |
| 4056 | return err |
| 4057 | } |
| 4058 | os.Remove(a.file("journal.json")) |
| 4059 | return syncDirBestEffort(a.rootDir()) |
| 4060 | } |
| 4061 | |
| 4062 | // RunSystemApply is `carlos system apply [--recover]`: the root oneshot. |
| 4063 | func RunSystemApply(args []string) { |
| 4064 | fs := flag.NewFlagSet("system apply", flag.ExitOnError) |
| 4065 | recover := fs.Bool("recover", false, "recover an interrupted apply (files only) and exit: the boot and OnFailure unit") |
| 4066 | fs.Usage = cliflag.Usage(fs, "") |
| 4067 | _ = fs.Parse(args) |
| 4068 | if os.Geteuid() != 0 { |
| 4069 | log.Fatal("carlos system apply is the root oneshot behind carlos-system-update.service; it is not for operators") |
| 4070 | } |
| 4071 | ctx := context.Background() |
| 4072 | store, err := openDeployStore(ctx) |
| 4073 | if err != nil { |
| 4074 | log.Fatalf("system apply: %v", err) |
| 4075 | } |
| 4076 | pub, err := release.ParsePublicKey(os.Getenv("CARLOS_RELEASE_PUBKEY"), "CARLOS_RELEASE_PUBKEY") |
| 4077 | if err != nil { |
| 4078 | log.Fatalf("system apply: %v", err) |
| 4079 | } |
| 4080 | a := &systemApplier{ |
| 4081 | ops: productionSysOps(), store: store, pub: pub, channel: systemChannel(os.Getenv), |
| 4082 | optDir: release.OptDir(), dataDir: dataDir(), etcDir: "/etc", build: version.Build, |
| 4083 | platform: runtime.GOOS + "-" + runtime.GOARCH, gateWait: systemGateWait, soak: systemGateSoak, |
| 4084 | } |
| 4085 | if err := a.Run(ctx, *recover); err != nil { |
| 4086 | log.Fatalf("%v", err) |
| 4087 | } |
| 4088 | } |
| 4089 | |
| 4090 | func productionSysOps() sysOps { |
| 4091 | return sysOps{ |
| 4092 | systemctl: func(args ...string) (string, error) { |
| 4093 | out, err := exec.Command("systemctl", args...).CombinedOutput() |
| 4094 | if err != nil { |
| 4095 | return string(out), fmt.Errorf("systemctl %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out))) |
| 4096 | } |
| 4097 | return string(out), nil |
| 4098 | }, |
| 4099 | run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) { |
| 4100 | ctx, cancel := context.WithTimeout(ctx, timeout) |
| 4101 | defer cancel() |
| 4102 | out, err := exec.CommandContext(ctx, name, args...).Output() |
| 4103 | if err != nil { |
| 4104 | return string(out), fmt.Errorf("%s %s: %w", name, strings.Join(args, " "), err) |
| 4105 | } |
| 4106 | return string(out), nil |
| 4107 | }, |
| 4108 | now: time.Now, |
| 4109 | sleep: time.Sleep, |
| 4110 | probeEdge: func() error { |
| 4111 | // Any HTTP answer on loopback proves the process serves; a box |
| 4112 | // with no route yet must still pass (spec §6 step 10). |
| 4113 | c := &http.Client{Timeout: 3 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} |
| 4114 | resp, err := c.Get("http://127.0.0.1/") |
| 4115 | if err != nil { |
| 4116 | return err |
| 4117 | } |
| 4118 | resp.Body.Close() |
| 4119 | return nil |
| 4120 | }, |
| 4121 | edgeStartedThisBoot: func() bool { |
| 4122 | out, err := exec.Command("systemctl", "show", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit).Output() |
| 4123 | return err == nil && strings.TrimSpace(string(out)) != "" && strings.TrimSpace(string(out)) != "0" |
| 4124 | }, |
| 4125 | logf: log.Printf, |
| 4126 | } |
| 4127 | } |
| 4128 | ``` |
| 4129 | Imports to add: `"flag"`, `"log"`, `"net/http"`, `"os/exec"`, `"github.com/carlosframework/platform/internal/cliflag"`, `"github.com/carlosframework/platform/internal/version"`. In the fixture, set `f.a.platform = "linux-arm64"` (from 8a). The edge's plain-HTTP listener: confirm the port with `grep -n 'ListenAndServe\|":80"\|Listen(' internal/edge/edge.go | head`; if the edge only listens on 443, probe `https://127.0.0.1/` with `InsecureSkipVerify` and a `ServerName` of `localhost` instead, and say which in the commit. |
| 4130 | |
| 4131 | - [ ] **Step 4: Run the whole host package** |
| 4132 | |
| 4133 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 4134 | Expected: `ok`. |
| 4135 | |
| 4136 | - [ ] **Step 5: Commit** |
| 4137 | |
| 4138 | ```bash |
| 4139 | gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF' |
| 4140 | host: carlos system apply — rollback, recovery, and the real box |
| 4141 | |
| 4142 | Rollback restores carlos.prev and its unit set, gates it, arms the |
| 4143 | previous plan, and leaves applied.json alone. Recovery is files-only |
| 4144 | and goes forward from phase gated; it restarts the edge only when the |
| 4145 | edge has started this boot, and removes the journal last. |
| 4146 | |
| 4147 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 4148 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 4149 | EOF |
| 4150 | ``` |
| 4151 | |
| 4152 | --- |
| 4153 | ### Task 9: `carlos system` dispatch and the `apply` verb |
| 4154 | |
| 4155 | **Files:** |
| 4156 | - Create: `system.go` (package main), `system_test.go` |
| 4157 | - Modify: `main.go` (the `switch cmd` at `:30-134`, `usage()` at `:139-233`), `update.go` (`noHintVerbs` at `:76-85`) |
| 4158 | |
| 4159 | **Interfaces:** |
| 4160 | - Consumes: `host.RunSystemApply` (Task 8c), `main.go` dispatch pattern (`case "ops": ops.Run(args)`), `usage()` section "On the host". |
| 4161 | - Produces: |
| 4162 | ```go |
| 4163 | func runSystem(args []string) // carlos system <verb> | carlos system router <verb> |
| 4164 | func systemVerb(args []string) (verb string, rest []string, component string) // strips an optional leading "router" |
| 4165 | ``` |
| 4166 | Verbs in this task: `apply` → `host.RunSystemApply(rest)`; `update`/`status`/`rollback` → `log.Fatal("carlos system " + verb + " arrives in a later task")` placeholders that Task 12 replaces; `help`/none → usage, exit 2. |
| 4167 | |
| 4168 | - [ ] **Step 1: Write the failing test** |
| 4169 | |
| 4170 | ```go |
| 4171 | package main |
| 4172 | |
| 4173 | import "testing" |
| 4174 | |
| 4175 | func TestSystemVerbAcceptsTheComponentForm(t *testing.T) { |
| 4176 | cases := []struct { |
| 4177 | in []string |
| 4178 | verb, comp string |
| 4179 | rest int |
| 4180 | }{ |
| 4181 | {[]string{"update", "--channel", "canary"}, "update", "router", 2}, |
| 4182 | {[]string{"router", "update"}, "update", "router", 0}, |
| 4183 | {[]string{"router", "status"}, "status", "router", 0}, |
| 4184 | {[]string{"apply", "--recover"}, "apply", "router", 1}, |
| 4185 | {[]string{}, "", "router", 0}, |
| 4186 | {[]string{"router"}, "", "router", 0}, |
| 4187 | } |
| 4188 | for _, c := range cases { |
| 4189 | verb, rest, comp := systemVerb(c.in) |
| 4190 | if verb != c.verb || comp != c.comp || len(rest) != c.rest { |
| 4191 | t.Errorf("systemVerb(%v) = %q %v %q", c.in, verb, rest, comp) |
| 4192 | } |
| 4193 | } |
| 4194 | if verb, _, _ := systemVerb([]string{"console", "update"}); verb != "" { |
| 4195 | t.Errorf("an unknown component must not be a verb: got %q", verb) |
| 4196 | } |
| 4197 | } |
| 4198 | ``` |
| 4199 | |
| 4200 | - [ ] **Step 2: Run to verify failure** |
| 4201 | |
| 4202 | Run: `export TMPDIR=/tmp/c; out=$(go test . -run TestSystemVerb -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc` |
| 4203 | Expected: undefined `systemVerb`. |
| 4204 | |
| 4205 | - [ ] **Step 3: Implement `system.go`** |
| 4206 | |
| 4207 | ```go |
| 4208 | package main |
| 4209 | |
| 4210 | import ( |
| 4211 | "fmt" |
| 4212 | "log" |
| 4213 | "os" |
| 4214 | |
| 4215 | "github.com/carlosframework/platform/internal/host" |
| 4216 | ) |
| 4217 | |
| 4218 | // carlos system: what is not an app (spec §0, the console-as-an-app |
| 4219 | // dividing line). One component today, the router — the substrate binary. |
| 4220 | // `carlos system router update` is the same verb scoped to it, so the |
| 4221 | // command Paul named works and a future split needs no new verb. |
| 4222 | |
| 4223 | const systemComponentRouter = "router" |
| 4224 | |
| 4225 | func systemVerb(args []string) (verb string, rest []string, component string) { |
| 4226 | component = systemComponentRouter |
| 4227 | if len(args) > 0 && args[0] == systemComponentRouter { |
| 4228 | args = args[1:] |
| 4229 | } |
| 4230 | if len(args) == 0 { |
| 4231 | return "", nil, component |
| 4232 | } |
| 4233 | switch args[0] { |
| 4234 | case "update", "status", "rollback", "apply", "help", "-h", "--help": |
| 4235 | return args[0], args[1:], component |
| 4236 | } |
| 4237 | return "", args, component |
| 4238 | } |
| 4239 | |
| 4240 | func runSystem(args []string) { |
| 4241 | verb, rest, _ := systemVerb(args) |
| 4242 | switch verb { |
| 4243 | case "apply": |
| 4244 | host.RunSystemApply(rest) |
| 4245 | case "update": |
| 4246 | runSystemUpdate(rest) |
| 4247 | case "status": |
| 4248 | runSystemStatus(rest) |
| 4249 | case "rollback": |
| 4250 | runSystemRollback(rest) |
| 4251 | case "help", "-h", "--help": |
| 4252 | systemUsage() |
| 4253 | default: |
| 4254 | if len(args) > 0 && verb == "" { |
| 4255 | fmt.Fprintf(os.Stderr, "system: unknown verb %q\n\n", args[0]) |
| 4256 | } |
| 4257 | systemUsage() |
| 4258 | os.Exit(2) |
| 4259 | } |
| 4260 | } |
| 4261 | |
| 4262 | func systemUsage() { |
| 4263 | fmt.Fprint(os.Stderr, `usage: carlos system <verb> [flags] |
| 4264 | carlos system router <verb> [flags] |
| 4265 | |
| 4266 | update promote a platform release to a channel; every box on it rolls itself |
| 4267 | status every box: platform, channel, running build, channel target, last apply outcome |
| 4268 | rollback point a channel back at the release before this one |
| 4269 | apply (root oneshot behind carlos-system-update.service; not for operators) |
| 4270 | |
| 4271 | The platform is not an app: the console is updated with carlos deploy --app console. |
| 4272 | `) |
| 4273 | } |
| 4274 | |
| 4275 | // Placeholders until Task 12. |
| 4276 | func runSystemUpdate(args []string) { log.Fatal("carlos system update: not yet wired to the console") } |
| 4277 | func runSystemStatus(args []string) { log.Fatal("carlos system status: not yet wired to the console") } |
| 4278 | func runSystemRollback(args []string) { log.Fatal("carlos system rollback: not yet wired to the console") } |
| 4279 | ``` |
| 4280 | `main.go`: add `case "system": runSystem(args)` after the `ops` case; in `usage()` under "On the host" add ` system platform self-update: update | status | rollback (apply is the root oneshot)`; in `update.go` `noHintVerbs` add `"system": true`. |
| 4281 | |
| 4282 | - [ ] **Step 4: Run and check the binary** |
| 4283 | |
| 4284 | Run: `export TMPDIR=/tmp/c; out=$(go test . -run 'TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go build -o /tmp/c/carlos . && /tmp/c/carlos system help 2>&1 | head -3; /tmp/c/carlos system apply 2>&1 | head -1` |
| 4285 | Expected: PASS; usage printed; `apply` as non-root prints "not for operators". |
| 4286 | |
| 4287 | - [ ] **Step 5: Commit** |
| 4288 | |
| 4289 | ```bash |
| 4290 | gofmt -l . && git add system.go system_test.go main.go update.go && git commit -q -F - <<'EOF' |
| 4291 | cli: carlos system, with apply wired to the root oneshot |
| 4292 | |
| 4293 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 4294 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 4295 | EOF |
| 4296 | ``` |
| 4297 | |
| 4298 | --- |
| 4299 | |
| 4300 | ### Task 10: The public release feed (`internal/publicrelease`) |
| 4301 | |
| 4302 | **Files:** |
| 4303 | - Create: `internal/publicrelease/feed.go`, `internal/publicrelease/feed_test.go` |
| 4304 | - Modify: `update.go` (`fetchLatest` `:100-119`, `selfReplace` `:295-327`, `fetchUpdateFile` `:358-369`) |
| 4305 | |
| 4306 | **Interfaces:** |
| 4307 | - Consumes: the package vars `updateLatestURL`, `updateDownloadBase`, `cliUpdatePubKey` in `update.go` (kept, since `update_test.go` repoints them). |
| 4308 | - Produces: |
| 4309 | ```go |
| 4310 | package publicrelease |
| 4311 | type Feed struct { |
| 4312 | LatestURL string // https://apt.carlosframework.com/latest.txt |
| 4313 | DownloadBase string // https://github.com/carlosframework/releases/releases/download (tag-pinned; never latest/download) |
| 4314 | PubKey string // base64 ed25519 public key that signs SHA256SUMS.txt |
| 4315 | Client *http.Client // nil = a 5-minute client |
| 4316 | } |
| 4317 | func Default() Feed |
| 4318 | var TagRe = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+$`) |
| 4319 | func (f Feed) Latest(ctx context.Context) (string, error) |
| 4320 | func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) // asset → sha, signature verified |
| 4321 | func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) (data []byte, sha256hex string, err error) // verified against Sums |
| 4322 | func AssetName(platform string) string // "carlos-" + platform |
| 4323 | ``` |
| 4324 | |
| 4325 | - [ ] **Step 1: Write the failing tests** |
| 4326 | |
| 4327 | ```go |
| 4328 | package publicrelease |
| 4329 | |
| 4330 | import ( |
| 4331 | "context" |
| 4332 | "crypto/ed25519" |
| 4333 | "crypto/sha256" |
| 4334 | "encoding/base64" |
| 4335 | "encoding/hex" |
| 4336 | "fmt" |
| 4337 | "net/http" |
| 4338 | "net/http/httptest" |
| 4339 | "strings" |
| 4340 | "testing" |
| 4341 | ) |
| 4342 | |
| 4343 | func testFeed(t *testing.T, sign bool, assets map[string][]byte) (Feed, string) { |
| 4344 | t.Helper() |
| 4345 | pub, priv, _ := ed25519.GenerateKey(nil) |
| 4346 | var sums strings.Builder |
| 4347 | for name, body := range assets { |
| 4348 | s := sha256.Sum256(body) |
| 4349 | fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name) |
| 4350 | } |
| 4351 | sig := ed25519.Sign(priv, []byte(sums.String())) |
| 4352 | mux := http.NewServeMux() |
| 4353 | mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") }) |
| 4354 | mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) }) |
| 4355 | mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) { |
| 4356 | if sign { |
| 4357 | fmt.Fprintln(w, base64.StdEncoding.EncodeToString(sig)) |
| 4358 | } else { |
| 4359 | fmt.Fprintln(w, base64.StdEncoding.EncodeToString(make([]byte, ed25519.SignatureSize))) |
| 4360 | } |
| 4361 | }) |
| 4362 | for name, body := range assets { |
| 4363 | body := body |
| 4364 | mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) }) |
| 4365 | } |
| 4366 | ts := httptest.NewServer(mux) |
| 4367 | t.Cleanup(ts.Close) |
| 4368 | return Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)}, ts.URL |
| 4369 | } |
| 4370 | |
| 4371 | func TestFeedLatestAndVerifiedAsset(t *testing.T) { |
| 4372 | body := []byte("ELF arm64") |
| 4373 | f, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body, "carlos-linux-amd64": []byte("ELF amd64")}) |
| 4374 | ctx := context.Background() |
| 4375 | tag, err := f.Latest(ctx) |
| 4376 | if err != nil || tag != "v0.18.0" { |
| 4377 | t.Fatalf("latest = %q %v", tag, err) |
| 4378 | } |
| 4379 | data, sha, err := f.Asset(ctx, tag, AssetName("linux-arm64"), 1<<20) |
| 4380 | if err != nil || string(data) != string(body) { |
| 4381 | t.Fatalf("asset: %v", err) |
| 4382 | } |
| 4383 | s := sha256.Sum256(body) |
| 4384 | if sha != hex.EncodeToString(s[:]) { |
| 4385 | t.Fatalf("sha = %s", sha) |
| 4386 | } |
| 4387 | if _, _, err := f.Asset(ctx, tag, "carlos-windows-amd64.zip", 1<<20); err == nil { |
| 4388 | t.Fatal("an asset absent from the sums was accepted") |
| 4389 | } |
| 4390 | } |
| 4391 | |
| 4392 | func TestFeedRefusesABadSignatureAndATamperedAsset(t *testing.T) { |
| 4393 | body := []byte("ELF") |
| 4394 | f, _ := testFeed(t, false, map[string][]byte{"carlos-linux-arm64": body}) |
| 4395 | if _, err := f.Sums(context.Background(), "v0.18.0"); err == nil || !strings.Contains(err.Error(), "signature") { |
| 4396 | t.Fatalf("bad signature: %v", err) |
| 4397 | } |
| 4398 | // Signed sums, but the served asset differs from the sum. |
| 4399 | f2, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body}) |
| 4400 | f2.DownloadBase = f2.DownloadBase // same server |
| 4401 | pub, _ := base64.StdEncoding.DecodeString(f2.PubKey) |
| 4402 | _ = pub |
| 4403 | // Simulate tampering by asking for a limit smaller than the asset: the |
| 4404 | // truncated download cannot match its sum. |
| 4405 | if _, _, err := f2.Asset(context.Background(), "v0.18.0", "carlos-linux-arm64", 1); err == nil { |
| 4406 | t.Fatal("a truncated asset matched its sum") |
| 4407 | } |
| 4408 | } |
| 4409 | |
| 4410 | func TestFeedLatestRequiresATag(t *testing.T) { |
| 4411 | ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "main") })) |
| 4412 | defer ts.Close() |
| 4413 | f := Feed{LatestURL: ts.URL, DownloadBase: ts.URL, PubKey: ""} |
| 4414 | if _, err := f.Latest(context.Background()); err == nil { |
| 4415 | t.Fatal("accepted a non-tag latest") |
| 4416 | } |
| 4417 | } |
| 4418 | ``` |
| 4419 | |
| 4420 | - [ ] **Step 2: Run to verify failure** |
| 4421 | |
| 4422 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc` |
| 4423 | Expected: package does not exist. |
| 4424 | |
| 4425 | - [ ] **Step 3: Implement `feed.go`** |
| 4426 | |
| 4427 | ```go |
| 4428 | // Package publicrelease reads the vendor's public signed release feed: the |
| 4429 | // latest tag, the signed checksum list, and a verified asset. Two callers |
| 4430 | // share it — `carlos update` on a workstation and the console's `carlos |
| 4431 | // system update` promote — so the verification chain is written once. |
| 4432 | package publicrelease |
| 4433 | |
| 4434 | import ( |
| 4435 | "context" |
| 4436 | "crypto/ed25519" |
| 4437 | "crypto/sha256" |
| 4438 | "encoding/base64" |
| 4439 | "encoding/hex" |
| 4440 | "errors" |
| 4441 | "fmt" |
| 4442 | "io" |
| 4443 | "net/http" |
| 4444 | "regexp" |
| 4445 | "strings" |
| 4446 | "time" |
| 4447 | ) |
| 4448 | |
| 4449 | type Feed struct { |
| 4450 | LatestURL string |
| 4451 | DownloadBase string // tag-pinned: <DownloadBase>/<tag>/<file>; never latest/download |
| 4452 | PubKey string // base64 ed25519 public key over the raw SHA256SUMS.txt bytes |
| 4453 | Client *http.Client |
| 4454 | } |
| 4455 | |
| 4456 | // Default is the vendor feed. The strings are duplicated in update.go's |
| 4457 | // package vars so update_test.go can repoint them; keep the two in step. |
| 4458 | func Default() Feed { |
| 4459 | return Feed{ |
| 4460 | LatestURL: "https://apt.carlosframework.com/latest.txt", |
| 4461 | DownloadBase: "https://github.com/carlosframework/releases/releases/download", |
| 4462 | PubKey: "R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=", |
| 4463 | } |
| 4464 | } |
| 4465 | |
| 4466 | var TagRe = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+$`) |
| 4467 | |
| 4468 | func AssetName(platform string) string { return "carlos-" + platform } |
| 4469 | |
| 4470 | func (f Feed) client() *http.Client { |
| 4471 | if f.Client != nil { |
| 4472 | return f.Client |
| 4473 | } |
| 4474 | return &http.Client{Timeout: 5 * time.Minute} |
| 4475 | } |
| 4476 | |
| 4477 | func (f Feed) get(ctx context.Context, url string, limit int64) ([]byte, error) { |
| 4478 | req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) |
| 4479 | if err != nil { |
| 4480 | return nil, err |
| 4481 | } |
| 4482 | res, err := f.client().Do(req) |
| 4483 | if err != nil { |
| 4484 | return nil, err |
| 4485 | } |
| 4486 | defer res.Body.Close() |
| 4487 | if res.StatusCode != http.StatusOK { |
| 4488 | return nil, fmt.Errorf("%s: status %d", url, res.StatusCode) |
| 4489 | } |
| 4490 | return io.ReadAll(io.LimitReader(res.Body, limit)) |
| 4491 | } |
| 4492 | |
| 4493 | // Latest is the newest release tag, and only a tag. |
| 4494 | func (f Feed) Latest(ctx context.Context) (string, error) { |
| 4495 | b, err := f.get(ctx, f.LatestURL, 128) |
| 4496 | if err != nil { |
| 4497 | return "", err |
| 4498 | } |
| 4499 | tag := strings.TrimSpace(string(b)) |
| 4500 | if !TagRe.MatchString(tag) { |
| 4501 | return "", fmt.Errorf("%s: %q is not a release tag", f.LatestURL, tag) |
| 4502 | } |
| 4503 | return tag, nil |
| 4504 | } |
| 4505 | |
| 4506 | // Sums fetches SHA256SUMS.txt and its .sig for tag, verifies the signature |
| 4507 | // with PubKey, and returns asset → sha256 hex. |
| 4508 | func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) { |
| 4509 | base := f.DownloadBase + "/" + tag + "/" |
| 4510 | sums, err := f.get(ctx, base+"SHA256SUMS.txt", 1<<20) |
| 4511 | if err != nil { |
| 4512 | return nil, err |
| 4513 | } |
| 4514 | sig, err := f.get(ctx, base+"SHA256SUMS.txt.sig", 4096) |
| 4515 | if err != nil { |
| 4516 | return nil, fmt.Errorf("%v (a release without a signed checksum list predates self-update)", err) |
| 4517 | } |
| 4518 | pub, err := base64.StdEncoding.DecodeString(f.PubKey) |
| 4519 | if err != nil || len(pub) != ed25519.PublicKeySize { |
| 4520 | return nil, errors.New("release feed public key is invalid") |
| 4521 | } |
| 4522 | rawSig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(string(sig))) |
| 4523 | if err != nil || !ed25519.Verify(ed25519.PublicKey(pub), sums, rawSig) { |
| 4524 | return nil, errors.New("SHA256SUMS.txt signature does not verify — refusing the release") |
| 4525 | } |
| 4526 | out := map[string]string{} |
| 4527 | for _, line := range strings.Split(string(sums), "\n") { |
| 4528 | fields := strings.Fields(line) |
| 4529 | if len(fields) == 2 { |
| 4530 | out[fields[1]] = fields[0] |
| 4531 | } |
| 4532 | } |
| 4533 | return out, nil |
| 4534 | } |
| 4535 | |
| 4536 | // Asset fetches one named asset for tag and refuses it unless its sha256 |
| 4537 | // matches the signed checksum list. |
| 4538 | func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) ([]byte, string, error) { |
| 4539 | sums, err := f.Sums(ctx, tag) |
| 4540 | if err != nil { |
| 4541 | return nil, "", err |
| 4542 | } |
| 4543 | want, ok := sums[asset] |
| 4544 | if !ok { |
| 4545 | return nil, "", fmt.Errorf("no %s in the release's checksum list", asset) |
| 4546 | } |
| 4547 | data, err := f.get(ctx, f.DownloadBase+"/"+tag+"/"+asset, limit) |
| 4548 | if err != nil { |
| 4549 | return nil, "", err |
| 4550 | } |
| 4551 | s := sha256.Sum256(data) |
| 4552 | got := hex.EncodeToString(s[:]) |
| 4553 | if got != want { |
| 4554 | return nil, "", fmt.Errorf("%s does not match its signed checksum — refusing it", asset) |
| 4555 | } |
| 4556 | return data, got, nil |
| 4557 | } |
| 4558 | ``` |
| 4559 | `update.go`: replace the bodies of `fetchLatest` and the fetch-and-verify half of `selfReplace` (`:296-327`) with calls through a feed built from the package vars, so tests that repoint the vars still work: |
| 4560 | ```go |
| 4561 | func updateFeed() publicrelease.Feed { |
| 4562 | return publicrelease.Feed{LatestURL: updateLatestURL, DownloadBase: updateDownloadBase, PubKey: cliUpdatePubKey} |
| 4563 | } |
| 4564 | ``` |
| 4565 | `fetchLatest(timeout)` → `updateFeed()` with `Client: &http.Client{Timeout: timeout}` and `.Latest(ctx)`; in `selfReplace`, `binData, _, err := updateFeed().Asset(ctx, latest, asset, 256<<20)`, keeping the workstation-only half (temp file, lie detector, rename) as is. Delete `fetchUpdateFile` if nothing else calls it. Run `go test . -run 'Update' -count=1` to prove `update_test.go` still passes. |
| 4566 | |
| 4567 | - [ ] **Step 4: Run the tests** |
| 4568 | |
| 4569 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ . -run 'Feed|Update' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc` |
| 4570 | Expected: `ok` for both packages. |
| 4571 | |
| 4572 | - [ ] **Step 5: Commit** |
| 4573 | |
| 4574 | ```bash |
| 4575 | gofmt -l . internal/publicrelease && git add internal/publicrelease update.go && git commit -q -F - <<'EOF' |
| 4576 | publicrelease: the vendor feed's verification chain, written once |
| 4577 | |
| 4578 | carlos update and the console's carlos system update share it: latest |
| 4579 | tag, signed SHA256SUMS, per-asset sha. |
| 4580 | |
| 4581 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 4582 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 4583 | EOF |
| 4584 | ``` |
| 4585 | |
| 4586 | --- |
| 4587 | |
| 4588 | ### Task 11: The console doors (`internal/console/apisystem.go`) |
| 4589 | |
| 4590 | **Files:** |
| 4591 | - Create: `internal/console/apisystem.go`, `internal/console/apisystem_test.go` |
| 4592 | - Modify: `internal/console/authcli.go` `registerCLIRoutes()` (add five `cliRoute` lines), `internal/console/server.go` `Config` (add `SystemFeed publicrelease.Feed`) |
| 4593 | |
| 4594 | **Interfaces:** |
| 4595 | - Consumes: `s.cliRoute(pattern, svcrole.Set{}, handler)`; `principalFrom(ctx)`, `PrincipalUser`; `s.store.Role(OpsAccountID, addr)`, `RoleOwner`; `s.cfg.Releases`, `s.cfg.Key`, `s.cfg.ReleasePub`; `s.bumpSerial(ctx)`; `writeJSON`, `jsonError`, `readJSONStrictLimit`; `status.LoadBoxes`, `release.NewPrefixStore`, `s.openPassRegistry().ListFleets()` (the `fleetMeters` loop, `economicspass.go:648-670`); `release.PromoteSystem`, `release.RollbackSystem`, `release.GetSystemPointer`, `release.RootSigner`; `publicrelease.Feed`; `apiShipMaxBytes`-style limits. |
| 4596 | - Produces (routes, all person-only, ops-owner-only, bare 404 on refusal like `IsOperator` doors): |
| 4597 | ``` |
| 4598 | GET /api/cli/system/status → {"pointers": {"canary": {...}|null, "stable": ...}, "boxes": [{label, platform, channel, release, target, system:{...}|null, stale}]} |
| 4599 | POST /api/cli/system/update → {"job": "<id>"}; body {"channel","version"?,"upload"?,"spread"?,"component"?} |
| 4600 | GET /api/cli/system/jobs/{id} → {"steps": ["resolved v0.18.0", ...], "done": bool, "error": ""} |
| 4601 | PUT /api/cli/system/upload → {"upload": "<handle>", "size": n}; multipart: meta {"version","platform"}, file |
| 4602 | POST /api/cli/system/rollback → {"channel","version"}; body {"channel","component"?} |
| 4603 | ``` |
| 4604 | ```go |
| 4605 | func (s *Server) apiSystemOwner(w, r) (addr string, ok bool) |
| 4606 | type systemJob struct{ mu sync.Mutex; Steps []string; Done bool; Err string } |
| 4607 | func (s *Server) runSystemUpdate(ctx, job *systemJob, actor string, in systemUpdateRequest) |
| 4608 | ``` |
| 4609 | Job and upload registries are fields on `Server`: `systemJobs map[string]*systemJob`, `systemUploads map[string]systemUpload`, guarded by `systemMu sync.Mutex`; one upload slot (128 MiB cap) held in memory, consumed by the job that names it. |
| 4610 | |
| 4611 | - [ ] **Step 1: Write the failing tests** |
| 4612 | |
| 4613 | ```go |
| 4614 | package console |
| 4615 | |
| 4616 | import ( |
| 4617 | "bytes" |
| 4618 | "context" |
| 4619 | "crypto/ed25519" |
| 4620 | "crypto/sha256" |
| 4621 | "encoding/base64" |
| 4622 | "encoding/hex" |
| 4623 | "fmt" |
| 4624 | "mime/multipart" |
| 4625 | "net/http" |
| 4626 | "net/http/httptest" |
| 4627 | "strings" |
| 4628 | "testing" |
| 4629 | "time" |
| 4630 | |
| 4631 | "github.com/carlosframework/platform/internal/publicrelease" |
| 4632 | "github.com/carlosframework/platform/internal/release" |
| 4633 | "github.com/carlosframework/platform/internal/status" |
| 4634 | ) |
| 4635 | |
| 4636 | // systemFeed serves a signed fake release from an httptest server and |
| 4637 | // points the fixture's console at it. |
| 4638 | func systemFeed(t *testing.T, fx *consoleFixture, assets map[string][]byte) { |
| 4639 | t.Helper() |
| 4640 | pub, priv, _ := ed25519.GenerateKey(nil) |
| 4641 | var sums strings.Builder |
| 4642 | for name, body := range assets { |
| 4643 | s := sha256.Sum256(body) |
| 4644 | fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name) |
| 4645 | } |
| 4646 | sig := base64.StdEncoding.EncodeToString(ed25519.Sign(priv, []byte(sums.String()))) |
| 4647 | mux := http.NewServeMux() |
| 4648 | mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") }) |
| 4649 | mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) }) |
| 4650 | mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, sig) }) |
| 4651 | for name, body := range assets { |
| 4652 | body := body |
| 4653 | mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) }) |
| 4654 | } |
| 4655 | ts := httptest.NewServer(mux) |
| 4656 | t.Cleanup(ts.Close) |
| 4657 | fx.srv.cfg.SystemFeed = publicrelease.Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)} |
| 4658 | } |
| 4659 | |
| 4660 | func waitJob(t *testing.T, fx *consoleFixture, token, id string) map[string]any { |
| 4661 | t.Helper() |
| 4662 | for i := 0; i < 100; i++ { |
| 4663 | st, out := getJSON(t, fx.ts.URL+"/api/cli/system/jobs/"+id, token) |
| 4664 | if st != http.StatusOK { |
| 4665 | t.Fatalf("job = %d %v", st, out) |
| 4666 | } |
| 4667 | if out["done"] == true { |
| 4668 | return out |
| 4669 | } |
| 4670 | time.Sleep(20 * time.Millisecond) |
| 4671 | } |
| 4672 | t.Fatal("job never finished") |
| 4673 | return nil |
| 4674 | } |
| 4675 | |
| 4676 | func seedBox(t *testing.T, fx *consoleFixture, label, platform, channel, rel string) { |
| 4677 | t.Helper() |
| 4678 | b := status.CollectBox(nil, status.Probes{Release: rel, Platform: platform, SystemChannel: channel}, time.Now()) |
| 4679 | b.Label = label |
| 4680 | if err := status.WriteBox(context.Background(), fx.rel, b); err != nil { |
| 4681 | t.Fatal(err) |
| 4682 | } |
| 4683 | } |
| 4684 | |
| 4685 | func TestSystemDoorsAreForOpsOwnersOnly(t *testing.T) { |
| 4686 | fx := newConsoleFixture(t) |
| 4687 | if err := fx.st.AddMember(OpsAccountID, "member@keymail.dev", RoleMember, time.Now()); err != nil { |
| 4688 | t.Fatal(err) |
| 4689 | } |
| 4690 | member := cliToken(t, fx, "member@keymail.dev", "code-sys-member") |
| 4691 | for _, u := range []string{"/api/cli/system/status"} { |
| 4692 | if st, _ := getJSON(t, fx.ts.URL+u, member); st != http.StatusNotFound { |
| 4693 | t.Errorf("GET %s as a member = %d, want 404", u, st) |
| 4694 | } |
| 4695 | } |
| 4696 | if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound { |
| 4697 | t.Errorf("update as a member = %d, want 404", st) |
| 4698 | } |
| 4699 | if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound { |
| 4700 | t.Errorf("rollback as a member = %d, want 404", st) |
| 4701 | } |
| 4702 | if st, _ := getJSON(t, fx.ts.URL+"/api/cli/system/status", "not-a-token"); st == http.StatusOK { |
| 4703 | t.Error("unauthenticated status answered 200") |
| 4704 | } |
| 4705 | } |
| 4706 | |
| 4707 | func TestSystemUpdateMirrorsSignsAndBumps(t *testing.T) { |
| 4708 | fx := newConsoleFixture(t) |
| 4709 | arm := []byte("ELF arm64") |
| 4710 | systemFeed(t, fx, map[string][]byte{"carlos-linux-arm64": arm, "carlos-linux-amd64": []byte("ELF amd64")}) |
| 4711 | seedBox(t, fx, "ie", "linux-arm64", "canary", "v0.17.0") |
| 4712 | owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-owner") |
| 4713 | before, _, _ := release.ReadSerial(context.Background(), fx.rel) |
| 4714 | st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "spread": "2m"}, owner) |
| 4715 | if st != http.StatusOK { |
| 4716 | t.Fatalf("update = %d %v", st, out) |
| 4717 | } |
| 4718 | job := waitJob(t, fx, owner, str(t, out, "job")) |
| 4719 | if job["error"] != "" { |
| 4720 | t.Fatalf("job failed: %v", job) |
| 4721 | } |
| 4722 | p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary") |
| 4723 | if err != nil { |
| 4724 | t.Fatal(err) |
| 4725 | } |
| 4726 | if p.Current.Version != "v0.18.0" || p.Spread != "2m0s" || p.Current.Source != "release:v0.18.0" { |
| 4727 | t.Fatalf("pointer = %+v", p) |
| 4728 | } |
| 4729 | a, ok := p.Current.Artifacts["linux-arm64"] |
| 4730 | if !ok { |
| 4731 | t.Fatalf("no arm64 artifact: %+v", p.Current.Artifacts) |
| 4732 | } |
| 4733 | if _, ok := p.Current.Artifacts["linux-amd64"]; ok { |
| 4734 | t.Fatal("mirrored a platform no box reports") |
| 4735 | } |
| 4736 | body, _, err := fx.rel.Get(context.Background(), a.Key) |
| 4737 | if err != nil || !bytes.Equal(body, arm) { |
| 4738 | t.Fatalf("artifact not mirrored: %v", err) |
| 4739 | } |
| 4740 | s := sha256.Sum256(arm) |
| 4741 | if a.SHA256 != hex.EncodeToString(s[:]) { |
| 4742 | t.Fatalf("sha = %s", a.SHA256) |
| 4743 | } |
| 4744 | after, _, _ := release.ReadSerial(context.Background(), fx.rel) |
| 4745 | if after.Serial == before.Serial { |
| 4746 | t.Fatal("serial not bumped") |
| 4747 | } |
| 4748 | // A second promote of the same version to stable copies nothing and |
| 4749 | // refuses different bytes under the same version. |
| 4750 | st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "stable", "version": "v0.18.0"}, owner) |
| 4751 | if st != http.StatusOK { |
| 4752 | t.Fatalf("second update = %d %v", st, out) |
| 4753 | } |
| 4754 | if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" { |
| 4755 | t.Fatalf("second job: %v", job) |
| 4756 | } |
| 4757 | if _, err := fx.rel.Put(context.Background(), a.Key, []byte("different"), ""); err == nil { |
| 4758 | // Overwrote the object out from under the pointer; a re-promote must refuse. |
| 4759 | st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "version": "v0.18.0"}, owner) |
| 4760 | job := waitJob(t, fx, owner, str(t, out, "job")) |
| 4761 | if job["error"] == "" || !strings.Contains(job["error"].(string), "immutable") { |
| 4762 | t.Fatalf("re-promote over different bytes: %v", job) |
| 4763 | } |
| 4764 | } |
| 4765 | } |
| 4766 | |
| 4767 | func TestSystemUpdateFromAnUpload(t *testing.T) { |
| 4768 | fx := newConsoleFixture(t) |
| 4769 | owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-upload") |
| 4770 | var buf bytes.Buffer |
| 4771 | mw := multipart.NewWriter(&buf) |
| 4772 | meta, _ := mw.CreateFormField("meta") |
| 4773 | meta.Write([]byte(`{"version":"66587fa","platform":"linux-arm64"}`)) |
| 4774 | file, _ := mw.CreateFormFile("file", "carlos-linux-arm64") |
| 4775 | file.Write([]byte("ELF local")) |
| 4776 | mw.Close() |
| 4777 | req, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf) |
| 4778 | req.Header.Set("Content-Type", mw.FormDataContentType()) |
| 4779 | req.Header.Set("Authorization", "Bearer "+owner) |
| 4780 | res, err := http.DefaultClient.Do(req) |
| 4781 | if err != nil { |
| 4782 | t.Fatal(err) |
| 4783 | } |
| 4784 | var up map[string]any |
| 4785 | if res.StatusCode != http.StatusOK { |
| 4786 | t.Fatalf("upload = %d", res.StatusCode) |
| 4787 | } |
| 4788 | if err := json.NewDecoder(res.Body).Decode(&up); err != nil { |
| 4789 | t.Fatal(err) |
| 4790 | } |
| 4791 | st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "upload": up["upload"]}, owner) |
| 4792 | if st != http.StatusOK { |
| 4793 | t.Fatalf("update = %d %v", st, out) |
| 4794 | } |
| 4795 | if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" { |
| 4796 | t.Fatalf("job: %v", job) |
| 4797 | } |
| 4798 | p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary") |
| 4799 | if err != nil || p.Current.Version != "66587fa" || p.Current.Source != "file" { |
| 4800 | t.Fatalf("pointer = %+v %v", p, err) |
| 4801 | } |
| 4802 | // Upload without a platform is refused at the door. |
| 4803 | var buf2 bytes.Buffer |
| 4804 | mw2 := multipart.NewWriter(&buf2) |
| 4805 | m2, _ := mw2.CreateFormField("meta") |
| 4806 | m2.Write([]byte(`{"version":"abc"}`)) |
| 4807 | f2, _ := mw2.CreateFormFile("file", "x") |
| 4808 | f2.Write([]byte("x")) |
| 4809 | mw2.Close() |
| 4810 | req2, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf2) |
| 4811 | req2.Header.Set("Content-Type", mw2.FormDataContentType()) |
| 4812 | req2.Header.Set("Authorization", "Bearer "+owner) |
| 4813 | res2, _ := http.DefaultClient.Do(req2) |
| 4814 | if res2.StatusCode != http.StatusBadRequest { |
| 4815 | t.Fatalf("upload without platform = %d", res2.StatusCode) |
| 4816 | } |
| 4817 | } |
| 4818 | |
| 4819 | func TestSystemStatusAndRollback(t *testing.T) { |
| 4820 | fx := newConsoleFixture(t) |
| 4821 | owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-status") |
| 4822 | seedBox(t, fx, "ie", "linux-arm64", "canary", "v1") |
| 4823 | sg := release.RootSigner(fx.priv) |
| 4824 | rel := func(v string) release.SystemRelease { |
| 4825 | return release.SystemRelease{Version: v, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("a", 64)}}} |
| 4826 | } |
| 4827 | release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v1"), release.SystemPromoteOptions{}) |
| 4828 | time.Sleep(1100 * time.Millisecond) |
| 4829 | release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v2"), release.SystemPromoteOptions{}) |
| 4830 | st, out := getJSON(t, fx.ts.URL+"/api/cli/system/status", owner) |
| 4831 | if st != http.StatusOK { |
| 4832 | t.Fatalf("status = %d %v", st, out) |
| 4833 | } |
| 4834 | boxes := out["boxes"].([]any) |
| 4835 | if len(boxes) != 1 { |
| 4836 | t.Fatalf("boxes = %v", boxes) |
| 4837 | } |
| 4838 | b := boxes[0].(map[string]any) |
| 4839 | if b["label"] != "ie" || b["release"] != "v1" || b["target"] != "v2" || b["channel"] != "canary" { |
| 4840 | t.Fatalf("box row = %v", b) |
| 4841 | } |
| 4842 | time.Sleep(1100 * time.Millisecond) |
| 4843 | st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner) |
| 4844 | if st != http.StatusOK || str(t, out, "version") != "v1" { |
| 4845 | t.Fatalf("rollback = %d %v", st, out) |
| 4846 | } |
| 4847 | if st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner); st != http.StatusConflict { |
| 4848 | t.Fatalf("second rollback = %d %v (nothing left in history)", st, out) |
| 4849 | } |
| 4850 | } |
| 4851 | ``` |
| 4852 | (add `"encoding/json"` to the imports.) |
| 4853 | |
| 4854 | - [ ] **Step 2: Run to verify failure** |
| 4855 | |
| 4856 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc` |
| 4857 | Expected: build failure (`SystemFeed` undefined) or 404s. |
| 4858 | |
| 4859 | - [ ] **Step 3: Implement** |
| 4860 | |
| 4861 | `server.go` `Config` gains, next to `Releases`: |
| 4862 | ```go |
| 4863 | // SystemFeed is where `carlos system update` fetches a public release |
| 4864 | // from. Zero value = the vendor feed; tests point it at a stub. |
| 4865 | SystemFeed publicrelease.Feed |
| 4866 | ``` |
| 4867 | and `Server` gains `systemMu sync.Mutex; systemJobs map[string]*systemJob; systemUploads map[string]systemUpload`. |
| 4868 | |
| 4869 | `authcli.go` `registerCLIRoutes()`: |
| 4870 | ```go |
| 4871 | // carlos system (spec §5): ops-account owners, person sessions only. |
| 4872 | s.cliRoute("GET /api/cli/system/status", svcrole.Set{}, s.handleAPISystemStatus) |
| 4873 | s.cliRoute("POST /api/cli/system/update", svcrole.Set{}, s.handleAPISystemUpdate) |
| 4874 | s.cliRoute("GET /api/cli/system/jobs/{id}", svcrole.Set{}, s.handleAPISystemJob) |
| 4875 | s.cliRoute("PUT /api/cli/system/upload", svcrole.Set{}, s.handleAPISystemUpload) |
| 4876 | s.cliRoute("POST /api/cli/system/rollback", svcrole.Set{}, s.handleAPISystemRollback) |
| 4877 | ``` |
| 4878 | |
| 4879 | `apisystem.go`: |
| 4880 | ```go |
| 4881 | package console |
| 4882 | |
| 4883 | import ( |
| 4884 | "context" |
| 4885 | "crypto/rand" |
| 4886 | "crypto/sha256" |
| 4887 | "encoding/hex" |
| 4888 | "encoding/json" |
| 4889 | "errors" |
| 4890 | "fmt" |
| 4891 | "io" |
| 4892 | "log" |
| 4893 | "net/http" |
| 4894 | "sort" |
| 4895 | "strings" |
| 4896 | "sync" |
| 4897 | "time" |
| 4898 | |
| 4899 | "github.com/carlosframework/platform/internal/publicrelease" |
| 4900 | "github.com/carlosframework/platform/internal/release" |
| 4901 | "github.com/carlosframework/platform/internal/status" |
| 4902 | ) |
| 4903 | |
| 4904 | // The console half of `carlos system` (spec §5). Every door here is for an |
| 4905 | // OWNER of the ops account on a PERSON session, checked on every call; |
| 4906 | // refusal is a bare 404 like the operator doors, so a non-owner learns |
| 4907 | // nothing. The console holds the bucket, the signing key and the egress; |
| 4908 | // the operator's laptop holds none of them. |
| 4909 | |
| 4910 | const ( |
| 4911 | systemUploadMaxBytes = int64(128 << 20) |
| 4912 | systemAssetMaxBytes = int64(256 << 20) |
| 4913 | systemJobTTL = time.Hour |
| 4914 | ) |
| 4915 | |
| 4916 | var systemPlatformRe = targetRe // apicli.go: ^[a-z0-9]+-[a-z0-9]+$ |
| 4917 | |
| 4918 | type systemJob struct { |
| 4919 | mu sync.Mutex |
| 4920 | Steps []string |
| 4921 | Done bool |
| 4922 | Err string |
| 4923 | at time.Time |
| 4924 | } |
| 4925 | |
| 4926 | func (j *systemJob) step(format string, args ...any) { |
| 4927 | j.mu.Lock() |
| 4928 | j.Steps = append(j.Steps, fmt.Sprintf(format, args...)) |
| 4929 | j.mu.Unlock() |
| 4930 | } |
| 4931 | |
| 4932 | func (j *systemJob) finish(err error) { |
| 4933 | j.mu.Lock() |
| 4934 | j.Done = true |
| 4935 | if err != nil { |
| 4936 | j.Err = err.Error() |
| 4937 | } |
| 4938 | j.mu.Unlock() |
| 4939 | } |
| 4940 | |
| 4941 | type systemUpload struct { |
| 4942 | version, platform string |
| 4943 | data []byte |
| 4944 | at time.Time |
| 4945 | } |
| 4946 | |
| 4947 | func (s *Server) apiSystemOwner(w http.ResponseWriter, r *http.Request) (string, bool) { |
| 4948 | p, ok := principalFrom(r.Context()) |
| 4949 | if !ok || p.Kind != PrincipalUser { |
| 4950 | http.NotFound(w, r) |
| 4951 | return "", false |
| 4952 | } |
| 4953 | role, err := s.store.Role(OpsAccountID, p.Address) |
| 4954 | if err != nil { |
| 4955 | log.Printf("console: api system: role: %v", err) |
| 4956 | http.NotFound(w, r) |
| 4957 | return "", false |
| 4958 | } |
| 4959 | if role != RoleOwner { |
| 4960 | http.NotFound(w, r) |
| 4961 | return "", false |
| 4962 | } |
| 4963 | return p.Address, true |
| 4964 | } |
| 4965 | |
| 4966 | func (s *Server) systemFeed() publicrelease.Feed { |
| 4967 | if s.cfg.SystemFeed.DownloadBase == "" { |
| 4968 | return publicrelease.Default() |
| 4969 | } |
| 4970 | return s.cfg.SystemFeed |
| 4971 | } |
| 4972 | |
| 4973 | // systemBoxes reads every heartbeat: the platform prefix and each fleet's. |
| 4974 | func (s *Server) systemBoxes(ctx context.Context) ([]status.Box, error) { |
| 4975 | boxes, err := status.LoadBoxes(ctx, s.cfg.Releases) |
| 4976 | if err != nil { |
| 4977 | return nil, err |
| 4978 | } |
| 4979 | reg, err := s.openPassRegistry() |
| 4980 | if err != nil { |
| 4981 | return boxes, nil // no registry: platform boxes only |
| 4982 | } |
| 4983 | fleets, err := reg.ListFleets() |
| 4984 | if err != nil { |
| 4985 | return boxes, nil |
| 4986 | } |
| 4987 | for _, f := range fleets { |
| 4988 | fb, err := status.LoadBoxes(ctx, release.NewPrefixStore(s.cfg.Releases, "fleets/"+f.Name+"/")) |
| 4989 | if err != nil { |
| 4990 | log.Printf("console: api system: fleet %s boxes: %v", f.Name, err) |
| 4991 | continue |
| 4992 | } |
| 4993 | boxes = append(boxes, fb...) |
| 4994 | } |
| 4995 | return boxes, nil |
| 4996 | } |
| 4997 | |
| 4998 | type systemUpdateRequest struct { |
| 4999 | Component string `json:"component"` |
| 5000 | Channel string `json:"channel"` |
| 5001 | Version string `json:"version"` |
| 5002 | Upload string `json:"upload"` |
| 5003 | Spread string `json:"spread"` |
| 5004 | } |
| 5005 | |
| 5006 | func (s *Server) handleAPISystemUpdate(w http.ResponseWriter, r *http.Request) { |
| 5007 | addr, ok := s.apiSystemOwner(w, r) |
| 5008 | if !ok { |
| 5009 | return |
| 5010 | } |
| 5011 | if s.cfg.Key == nil { |
| 5012 | jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only") |
| 5013 | return |
| 5014 | } |
| 5015 | var in systemUpdateRequest |
| 5016 | if !readJSONStrictLimit(w, r, &in, 4<<10) { |
| 5017 | return |
| 5018 | } |
| 5019 | if in.Component == "" { |
| 5020 | in.Component = release.SystemComponentRouter |
| 5021 | } |
| 5022 | if in.Component != release.SystemComponentRouter { |
| 5023 | jsonError(w, http.StatusBadRequest, "unknown component "+in.Component) |
| 5024 | return |
| 5025 | } |
| 5026 | if in.Channel != release.RungCanary && in.Channel != release.RungStable { |
| 5027 | jsonError(w, http.StatusBadRequest, "channel must be canary or stable") |
| 5028 | return |
| 5029 | } |
| 5030 | spread := 5 * time.Minute |
| 5031 | if in.Spread != "" { |
| 5032 | d, err := time.ParseDuration(in.Spread) |
| 5033 | if err != nil || d < 0 || d > 24*time.Hour { |
| 5034 | jsonError(w, http.StatusBadRequest, "spread must be a duration between 0 and 24h") |
| 5035 | return |
| 5036 | } |
| 5037 | spread = d |
| 5038 | } |
| 5039 | id := s.newSystemJob() |
| 5040 | job := s.systemJobs[id] |
| 5041 | go s.runSystemUpdate(context.WithoutCancel(r.Context()), job, addr, in, spread) |
| 5042 | writeJSON(w, http.StatusOK, map[string]string{"job": id}) |
| 5043 | } |
| 5044 | |
| 5045 | func (s *Server) newSystemJob() string { |
| 5046 | b := make([]byte, 8) |
| 5047 | rand.Read(b) |
| 5048 | id := hex.EncodeToString(b) |
| 5049 | s.systemMu.Lock() |
| 5050 | defer s.systemMu.Unlock() |
| 5051 | if s.systemJobs == nil { |
| 5052 | s.systemJobs = map[string]*systemJob{} |
| 5053 | } |
| 5054 | for k, j := range s.systemJobs { |
| 5055 | if time.Since(j.at) > systemJobTTL { |
| 5056 | delete(s.systemJobs, k) |
| 5057 | } |
| 5058 | } |
| 5059 | s.systemJobs[id] = &systemJob{at: time.Now()} |
| 5060 | return id |
| 5061 | } |
| 5062 | |
| 5063 | // runSystemUpdate is the promote: resolve, ingest (or take the upload), |
| 5064 | // mirror, sign, bump. Every step is idempotent, so a lost job is re-run. |
| 5065 | func (s *Server) runSystemUpdate(ctx context.Context, job *systemJob, actor string, in systemUpdateRequest, spread time.Duration) { |
| 5066 | err := func() error { |
| 5067 | var rel release.SystemRelease |
| 5068 | if in.Upload != "" { |
| 5069 | s.systemMu.Lock() |
| 5070 | up, ok := s.systemUploads[in.Upload] |
| 5071 | delete(s.systemUploads, in.Upload) |
| 5072 | s.systemMu.Unlock() |
| 5073 | if !ok { |
| 5074 | return errors.New("upload handle unknown or already consumed") |
| 5075 | } |
| 5076 | if in.Version != "" && in.Version != up.version { |
| 5077 | return fmt.Errorf("upload is %s, request says %s", up.version, in.Version) |
| 5078 | } |
| 5079 | job.step("using uploaded %s for %s", up.version, up.platform) |
| 5080 | sum := sha256.Sum256(up.data) |
| 5081 | rel = release.SystemRelease{Version: up.version, Source: "file", Artifacts: map[string]release.SystemArtifact{}} |
| 5082 | if err := s.mirrorSystemArtifact(ctx, job, up.version, up.platform, up.data, hex.EncodeToString(sum[:])); err != nil { |
| 5083 | return err |
| 5084 | } |
| 5085 | rel.Artifacts[up.platform] = release.SystemArtifact{Key: release.SystemArtifactKey(up.version, up.platform), SHA256: hex.EncodeToString(sum[:])} |
| 5086 | } else { |
| 5087 | feed := s.systemFeed() |
| 5088 | tag := in.Version |
| 5089 | if tag == "" { |
| 5090 | var err error |
| 5091 | if tag, err = feed.Latest(ctx); err != nil { |
| 5092 | return err |
| 5093 | } |
| 5094 | } |
| 5095 | if !publicrelease.TagRe.MatchString(tag) { |
| 5096 | return fmt.Errorf("%q is not a release tag; use --file for a local build", tag) |
| 5097 | } |
| 5098 | job.step("resolved %s", tag) |
| 5099 | platforms, err := s.systemPlatforms(ctx) |
| 5100 | if err != nil { |
| 5101 | return err |
| 5102 | } |
| 5103 | rel = release.SystemRelease{Version: tag, Source: "release:" + tag, Artifacts: map[string]release.SystemArtifact{}} |
| 5104 | for _, plat := range platforms { |
| 5105 | data, sha, err := feed.Asset(ctx, tag, publicrelease.AssetName(plat), systemAssetMaxBytes) |
| 5106 | if err != nil { |
| 5107 | return fmt.Errorf("%s: %w", plat, err) |
| 5108 | } |
| 5109 | job.step("verified %s (%d bytes)", publicrelease.AssetName(plat), len(data)) |
| 5110 | if err := s.mirrorSystemArtifact(ctx, job, tag, plat, data, sha); err != nil { |
| 5111 | return err |
| 5112 | } |
| 5113 | rel.Artifacts[plat] = release.SystemArtifact{Key: release.SystemArtifactKey(tag, plat), SHA256: sha} |
| 5114 | } |
| 5115 | } |
| 5116 | p, err := release.PromoteSystem(ctx, s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, rel, |
| 5117 | release.SystemPromoteOptions{Actor: actor, Spread: spread}) |
| 5118 | if err != nil { |
| 5119 | return err |
| 5120 | } |
| 5121 | job.step("pointed %s/%s at %s (spread %s)", in.Component, in.Channel, p.Current.Version, p.Spread) |
| 5122 | s.audit(nil, "platform", actor, "system-update", "pointed "+in.Component+"/"+in.Channel+" at "+p.Current.Version) |
| 5123 | s.bumpSerial(ctx) |
| 5124 | job.step("serial bumped: boxes on %s start their spread", in.Channel) |
| 5125 | return nil |
| 5126 | }() |
| 5127 | job.finish(err) |
| 5128 | } |
| 5129 | |
| 5130 | // systemPlatforms: the distinct platforms the fleet reports; linux-arm64 |
| 5131 | // when nothing has reported the field yet. |
| 5132 | func (s *Server) systemPlatforms(ctx context.Context) ([]string, error) { |
| 5133 | boxes, err := s.systemBoxes(ctx) |
| 5134 | if err != nil { |
| 5135 | return nil, err |
| 5136 | } |
| 5137 | set := map[string]bool{} |
| 5138 | for _, b := range boxes { |
| 5139 | if b.Metrics != nil && b.Metrics.Platform != "" { |
| 5140 | set[b.Metrics.Platform] = true |
| 5141 | } |
| 5142 | } |
| 5143 | if len(set) == 0 { |
| 5144 | set["linux-arm64"] = true |
| 5145 | } |
| 5146 | var out []string |
| 5147 | for p := range set { |
| 5148 | out = append(out, p) |
| 5149 | } |
| 5150 | sort.Strings(out) |
| 5151 | return out, nil |
| 5152 | } |
| 5153 | |
| 5154 | // mirrorSystemArtifact: present with the same sha → skip; present with a |
| 5155 | // different sha → refuse (versions are immutable, spec §3). |
| 5156 | func (s *Server) mirrorSystemArtifact(ctx context.Context, job *systemJob, version, platform string, data []byte, sha string) error { |
| 5157 | key := release.SystemArtifactKey(version, platform) |
| 5158 | existing, etag, err := s.cfg.Releases.Get(ctx, key) |
| 5159 | switch { |
| 5160 | case err == nil: |
| 5161 | sum := sha256.Sum256(existing) |
| 5162 | if hex.EncodeToString(sum[:]) == sha { |
| 5163 | job.step("%s already mirrored", key) |
| 5164 | return nil |
| 5165 | } |
| 5166 | return fmt.Errorf("%s exists with different bytes — versions are immutable; pick a new version", key) |
| 5167 | case errors.Is(err, os.ErrNotExist): |
| 5168 | etag = "" |
| 5169 | default: |
| 5170 | return err |
| 5171 | } |
| 5172 | if _, err := s.cfg.Releases.Put(ctx, key, data, etag); err != nil { |
| 5173 | return err |
| 5174 | } |
| 5175 | job.step("mirrored %s", key) |
| 5176 | return nil |
| 5177 | } |
| 5178 | |
| 5179 | func (s *Server) handleAPISystemJob(w http.ResponseWriter, r *http.Request) { |
| 5180 | if _, ok := s.apiSystemOwner(w, r); !ok { |
| 5181 | return |
| 5182 | } |
| 5183 | s.systemMu.Lock() |
| 5184 | job, ok := s.systemJobs[r.PathValue("id")] |
| 5185 | s.systemMu.Unlock() |
| 5186 | if !ok { |
| 5187 | jsonError(w, http.StatusNotFound, "no such job — the console may have restarted; re-run, every step is idempotent") |
| 5188 | return |
| 5189 | } |
| 5190 | job.mu.Lock() |
| 5191 | defer job.mu.Unlock() |
| 5192 | writeJSON(w, http.StatusOK, map[string]any{"steps": append([]string(nil), job.Steps...), "done": job.Done, "error": job.Err}) |
| 5193 | } |
| 5194 | |
| 5195 | func (s *Server) handleAPISystemUpload(w http.ResponseWriter, r *http.Request) { |
| 5196 | if _, ok := s.apiSystemOwner(w, r); !ok { |
| 5197 | return |
| 5198 | } |
| 5199 | r.Body = http.MaxBytesReader(w, r.Body, systemUploadMaxBytes+64<<10) |
| 5200 | mr, err := r.MultipartReader() |
| 5201 | if err != nil { |
| 5202 | jsonError(w, http.StatusBadRequest, "multipart body required") |
| 5203 | return |
| 5204 | } |
| 5205 | var meta struct { |
| 5206 | Version string `json:"version"` |
| 5207 | Platform string `json:"platform"` |
| 5208 | } |
| 5209 | first, err := mr.NextPart() |
| 5210 | if err != nil || first.FormName() != "meta" { |
| 5211 | jsonError(w, http.StatusBadRequest, "first part must be meta") |
| 5212 | return |
| 5213 | } |
| 5214 | if err := json.NewDecoder(io.LimitReader(first, 4<<10)).Decode(&meta); err != nil { |
| 5215 | jsonError(w, http.StatusBadRequest, "meta: "+err.Error()) |
| 5216 | return |
| 5217 | } |
| 5218 | if !release.ValidVersion(meta.Version) || meta.Version == "dev" { |
| 5219 | jsonError(w, http.StatusBadRequest, "meta.version is required and must be a version") |
| 5220 | return |
| 5221 | } |
| 5222 | if !systemPlatformRe.MatchString(meta.Platform) { |
| 5223 | jsonError(w, http.StatusBadRequest, "meta.platform is required (e.g. linux-arm64): the console cannot run a foreign binary to ask it") |
| 5224 | return |
| 5225 | } |
| 5226 | part, err := mr.NextPart() |
| 5227 | if err != nil || part.FormName() != "file" { |
| 5228 | jsonError(w, http.StatusBadRequest, "second part must be file") |
| 5229 | return |
| 5230 | } |
| 5231 | data, err := io.ReadAll(io.LimitReader(part, systemUploadMaxBytes+1)) |
| 5232 | if err != nil { |
| 5233 | jsonError(w, http.StatusBadRequest, "file: "+err.Error()) |
| 5234 | return |
| 5235 | } |
| 5236 | if int64(len(data)) > systemUploadMaxBytes { |
| 5237 | jsonError(w, http.StatusRequestEntityTooLarge, "file exceeds 128 MiB") |
| 5238 | return |
| 5239 | } |
| 5240 | b := make([]byte, 8) |
| 5241 | rand.Read(b) |
| 5242 | handle := hex.EncodeToString(b) |
| 5243 | s.systemMu.Lock() |
| 5244 | if s.systemUploads == nil { |
| 5245 | s.systemUploads = map[string]systemUpload{} |
| 5246 | } |
| 5247 | for k := range s.systemUploads { // one slot: a forgotten upload is not a memory leak |
| 5248 | delete(s.systemUploads, k) |
| 5249 | } |
| 5250 | s.systemUploads[handle] = systemUpload{version: meta.Version, platform: meta.Platform, data: data, at: time.Now()} |
| 5251 | s.systemMu.Unlock() |
| 5252 | writeJSON(w, http.StatusOK, map[string]any{"upload": handle, "size": len(data)}) |
| 5253 | } |
| 5254 | |
| 5255 | func (s *Server) handleAPISystemRollback(w http.ResponseWriter, r *http.Request) { |
| 5256 | addr, ok := s.apiSystemOwner(w, r) |
| 5257 | if !ok { |
| 5258 | return |
| 5259 | } |
| 5260 | if s.cfg.Key == nil { |
| 5261 | jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only") |
| 5262 | return |
| 5263 | } |
| 5264 | var in struct { |
| 5265 | Component string `json:"component"` |
| 5266 | Channel string `json:"channel"` |
| 5267 | } |
| 5268 | if !readJSONStrictLimit(w, r, &in, 1<<10) { |
| 5269 | return |
| 5270 | } |
| 5271 | if in.Component == "" { |
| 5272 | in.Component = release.SystemComponentRouter |
| 5273 | } |
| 5274 | p, err := release.RollbackSystem(r.Context(), s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, addr) |
| 5275 | switch { |
| 5276 | case errors.Is(err, release.ErrNoSystemHistory): |
| 5277 | jsonError(w, http.StatusConflict, err.Error()) |
| 5278 | return |
| 5279 | case errors.Is(err, os.ErrNotExist): |
| 5280 | jsonError(w, http.StatusNotFound, "no "+in.Channel+" pointer to roll back") |
| 5281 | return |
| 5282 | case err != nil: |
| 5283 | jsonError(w, http.StatusInternalServerError, err.Error()) |
| 5284 | return |
| 5285 | } |
| 5286 | s.audit(nil, "platform", addr, "system-rollback", "pointed "+in.Component+"/"+in.Channel+" back at "+p.Current.Version) |
| 5287 | s.bumpSerial(r.Context()) |
| 5288 | writeJSON(w, http.StatusOK, map[string]string{"channel": in.Channel, "version": p.Current.Version}) |
| 5289 | } |
| 5290 | |
| 5291 | func (s *Server) handleAPISystemStatus(w http.ResponseWriter, r *http.Request) { |
| 5292 | if _, ok := s.apiSystemOwner(w, r); !ok { |
| 5293 | return |
| 5294 | } |
| 5295 | ctx := r.Context() |
| 5296 | pointers := map[string]any{} |
| 5297 | targets := map[string]string{} |
| 5298 | for _, ch := range []string{release.RungCanary, release.RungStable} { |
| 5299 | p, _, err := release.GetSystemPointer(ctx, s.cfg.Releases, s.cfg.ReleasePub, release.SystemComponentRouter, ch) |
| 5300 | if err != nil { |
| 5301 | pointers[ch] = nil |
| 5302 | continue |
| 5303 | } |
| 5304 | pointers[ch] = map[string]any{"version": p.Current.Version, "promoted_at": p.PromotedAt, "spread": p.Spread, "source": p.Current.Source, |
| 5305 | "history": len(p.History)} |
| 5306 | targets[ch] = p.Current.Version |
| 5307 | } |
| 5308 | boxes, err := s.systemBoxes(ctx) |
| 5309 | if err != nil { |
| 5310 | jsonError(w, http.StatusBadGateway, "box heartbeats unreadable: "+err.Error()) |
| 5311 | return |
| 5312 | } |
| 5313 | now := time.Now() |
| 5314 | rows := make([]map[string]any, 0, len(boxes)) |
| 5315 | for _, b := range boxes { |
| 5316 | row := map[string]any{"label": b.Label, "region": b.Region, "stale": status.BoxStale(b, now, status.SnapshotPeriod), |
| 5317 | "platform": "", "channel": "", "release": "", "target": "", "system": nil} |
| 5318 | if b.Metrics != nil { |
| 5319 | ch := b.Metrics.SystemChannel |
| 5320 | if ch == "" { |
| 5321 | ch = "stable" |
| 5322 | } |
| 5323 | row["platform"], row["channel"], row["release"] = b.Metrics.Platform, ch, b.Metrics.Release |
| 5324 | row["target"] = targets[ch] |
| 5325 | if b.Metrics.System != nil { |
| 5326 | row["system"] = b.Metrics.System |
| 5327 | } |
| 5328 | } |
| 5329 | rows = append(rows, row) |
| 5330 | } |
| 5331 | sort.Slice(rows, func(i, j int) bool { return rows[i]["label"].(string) < rows[j]["label"].(string) }) |
| 5332 | writeJSON(w, http.StatusOK, map[string]any{"pointers": pointers, "boxes": rows}) |
| 5333 | } |
| 5334 | ``` |
| 5335 | Check three things against the code while writing: `s.audit`'s signature (`apicli.go:855` passes `(acct, app, addr, action, detail)`; if `acct` cannot be nil, use the ops account fetched by id, or add a `s.auditSystem(addr, action, detail)` writing an event row against `OpsAccountID`); `status.BoxStale`'s signature (`fleet.go:137`); `os` import. `readJSONStrictLimit` is at `authcli.go:849`. |
| 5336 | |
| 5337 | - [ ] **Step 4: Run the console tests** |
| 5338 | |
| 5339 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc` |
| 5340 | Expected: `ok`. `TestEveryCLIDoorNamesItsRoles` and `TestEveryClosedDoorRefusesAServiceCredential` cover the five new doors automatically; if either fails, the route line is wrong, not the test. |
| 5341 | |
| 5342 | - [ ] **Step 5: Commit** |
| 5343 | |
| 5344 | ```bash |
| 5345 | gofmt -l internal/console/ && git add internal/console/apisystem.go internal/console/apisystem_test.go internal/console/authcli.go internal/console/server.go && git commit -q -F - <<'EOF' |
| 5346 | console: the carlos system doors — status, update job, upload, rollback |
| 5347 | |
| 5348 | Ops-account owners on a person session, bare 404 otherwise. The |
| 5349 | update is a job: resolve, ingest from the signed public feed (or an |
| 5350 | upload), mirror per reported platform, sign the pointer, bump the |
| 5351 | serial. Versions are immutable in the bucket. |
| 5352 | |
| 5353 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 5354 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 5355 | EOF |
| 5356 | ``` |
| 5357 | |
| 5358 | --- |
| 5359 | ### Task 12: `carlos system update | status | rollback` (API mode) |
| 5360 | |
| 5361 | **Files:** |
| 5362 | - Modify: `system.go` (replace the three placeholders), `system_test.go` |
| 5363 | |
| 5364 | **Interfaces:** |
| 5365 | - Consumes: `requireDeployAPISession(consoleFlag) *apiSession` (`deploy.go:1151`); `(*apiSession).getJSON/postJSON` (`apimode.go:737,799`); `(*consoleClient).doMultipart` (`auth.go:485`) via `sess.c`; `writeFilePart(mw, name, path)` (`apimode.go:3009`); `interactiveTerminal`, `promptYes` (`account_choose.go`); `stdout`, `colorFor`, `mutedf` (`color.go`); `flagUsage`. |
| 5366 | - Produces: |
| 5367 | ```go |
| 5368 | func runSystemUpdate(args []string) // --console --channel --version --file --platform --spread --yes |
| 5369 | func runSystemStatus(args []string) // --console --channel |
| 5370 | func runSystemRollback(args []string) // --console --channel --yes |
| 5371 | func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string) |
| 5372 | type systemStatusResponse struct{ Pointers map[string]*systemPointerRow; Boxes []systemBoxRow } |
| 5373 | ``` |
| 5374 | |
| 5375 | - [ ] **Step 1: Write the failing test** (the pure renderer; the doors are covered in Task 11) |
| 5376 | |
| 5377 | ```go |
| 5378 | func TestPrintSystemStatusSaysWhatEachBoxIsDoing(t *testing.T) { |
| 5379 | resp := systemStatusResponse{ |
| 5380 | Pointers: map[string]*systemPointerRow{"canary": {Version: "v2", Spread: "5m0s"}, "stable": nil}, |
| 5381 | Boxes: []systemBoxRow{ |
| 5382 | {Label: "ie", Platform: "linux-arm64", Channel: "canary", Release: "v2", Target: "v2"}, |
| 5383 | {Label: "us", Platform: "linux-arm64", Channel: "canary", Release: "v1", Target: "v2", |
| 5384 | System: &systemStateRow{Version: "v2", Outcome: "rolled-back", Reason: "gate: carlos-edge is failed"}}, |
| 5385 | {Label: "au", Platform: "", Channel: "stable", Release: "v1", Target: "", Stale: true}, |
| 5386 | }, |
| 5387 | } |
| 5388 | var b strings.Builder |
| 5389 | printSystemStatus(&b, resp, "") |
| 5390 | out := b.String() |
| 5391 | for _, want := range []string{"canary v2", "stable (no pointer)", "ie ", "current", "us ", "rolled-back", "gate: carlos-edge is failed", "au ", "UNREACHABLE", "pre-upgrade"} { |
| 5392 | if !strings.Contains(out, want) { |
| 5393 | t.Errorf("status output lacks %q:\n%s", want, out) |
| 5394 | } |
| 5395 | } |
| 5396 | var only strings.Builder |
| 5397 | printSystemStatus(&only, resp, "stable") |
| 5398 | if strings.Contains(only.String(), "ie ") { |
| 5399 | t.Error("--channel stable still listed a canary box") |
| 5400 | } |
| 5401 | } |
| 5402 | ``` |
| 5403 | |
| 5404 | - [ ] **Step 2: Run to verify failure** |
| 5405 | |
| 5406 | Run: `export TMPDIR=/tmp/c; out=$(go test . -run TestPrintSystemStatus -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc` |
| 5407 | Expected: undefined types. |
| 5408 | |
| 5409 | - [ ] **Step 3: Implement** |
| 5410 | |
| 5411 | Replace the placeholders in `system.go`: |
| 5412 | ```go |
| 5413 | type systemPointerRow struct { |
| 5414 | Version string `json:"version"` |
| 5415 | PromotedAt time.Time `json:"promoted_at"` |
| 5416 | Spread string `json:"spread"` |
| 5417 | Source string `json:"source"` |
| 5418 | History int `json:"history"` |
| 5419 | } |
| 5420 | |
| 5421 | type systemStateRow struct { |
| 5422 | Channel string `json:"channel"` |
| 5423 | Version string `json:"version"` |
| 5424 | PromotedAt time.Time `json:"promoted_at"` |
| 5425 | Outcome string `json:"outcome"` |
| 5426 | Reason string `json:"reason"` |
| 5427 | At time.Time `json:"at"` |
| 5428 | } |
| 5429 | |
| 5430 | type systemBoxRow struct { |
| 5431 | Label string `json:"label"` |
| 5432 | Region string `json:"region"` |
| 5433 | Platform string `json:"platform"` |
| 5434 | Channel string `json:"channel"` |
| 5435 | Release string `json:"release"` |
| 5436 | Target string `json:"target"` |
| 5437 | System *systemStateRow `json:"system"` |
| 5438 | Stale bool `json:"stale"` |
| 5439 | } |
| 5440 | |
| 5441 | type systemStatusResponse struct { |
| 5442 | Pointers map[string]*systemPointerRow `json:"pointers"` |
| 5443 | Boxes []systemBoxRow `json:"boxes"` |
| 5444 | } |
| 5445 | |
| 5446 | func runSystemUpdate(args []string) { |
| 5447 | fs := flag.NewFlagSet("system update", flag.ExitOnError) |
| 5448 | consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)") |
| 5449 | channel := fs.String("channel", "stable", "canary or stable") |
| 5450 | version := fs.String("version", "", "release tag to promote (default: the latest public release); with --file, the version the binary reports") |
| 5451 | file := fs.String("file", "", "a locally built binary instead of a public release (dev and pre-release)") |
| 5452 | platform := fs.String("platform", "linux-arm64", "with --file: the binary's <goos>-<goarch>") |
| 5453 | spread := fs.String("spread", "5m", "boxes jitter their restarts over this; 0 for a hotfix") |
| 5454 | yes := fs.Bool("yes", false, "do not ask") |
| 5455 | fs.Usage = flagUsage(fs, "") |
| 5456 | fs.Parse(args) |
| 5457 | if *file != "" && *version == "" { |
| 5458 | log.Fatal("--file needs --version: the console cannot run a foreign-arch binary to ask it") |
| 5459 | } |
| 5460 | sess := requireDeployAPISession(*consoleFlag) |
| 5461 | ctx := context.Background() |
| 5462 | |
| 5463 | // What this reaches, before it is done. |
| 5464 | st, body, err := sess.getJSON(ctx, "/api/cli/system/status") |
| 5465 | if err != nil || st != http.StatusOK { |
| 5466 | log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err)) |
| 5467 | } |
| 5468 | var cur systemStatusResponse |
| 5469 | _ = json.Unmarshal(body, &cur) |
| 5470 | var reach []string |
| 5471 | for _, b := range cur.Boxes { |
| 5472 | if b.Channel == *channel { |
| 5473 | reach = append(reach, fmt.Sprintf("%s (%s, running %s)", b.Label, b.Platform, orDash(b.Release))) |
| 5474 | } |
| 5475 | } |
| 5476 | what := "the latest public release" |
| 5477 | if *version != "" { |
| 5478 | what = *version |
| 5479 | } |
| 5480 | fmt.Fprintf(stdout, "promote %s to %s/%s — %d box(es) will roll over %s:\n", what, systemComponentRouter, *channel, len(reach), *spread) |
| 5481 | for _, r := range reach { |
| 5482 | fmt.Fprintf(stdout, " %s\n", r) |
| 5483 | } |
| 5484 | if len(reach) == 0 { |
| 5485 | mutedf(stdout, " (no box reports channel %s yet; boxes on an older build count as stable)\n", *channel) |
| 5486 | } |
| 5487 | if !*yes { |
| 5488 | if !interactiveTerminal() { |
| 5489 | log.Fatal("no terminal to confirm on; pass --yes") |
| 5490 | } |
| 5491 | if !promptYes(stdout, "go ahead? [y/N] ") { |
| 5492 | return |
| 5493 | } |
| 5494 | } |
| 5495 | |
| 5496 | in := map[string]any{"channel": *channel, "spread": *spread} |
| 5497 | if *version != "" { |
| 5498 | in["version"] = *version |
| 5499 | } |
| 5500 | if *file != "" { |
| 5501 | handle, err := sess.uploadSystemBinary(ctx, *file, *version, *platform) |
| 5502 | if err != nil { |
| 5503 | log.Fatalf("upload: %v", err) |
| 5504 | } |
| 5505 | in["upload"] = handle |
| 5506 | } |
| 5507 | st, body, err = sess.postJSON(ctx, "/api/cli/system/update", in) |
| 5508 | if err != nil || st != http.StatusOK { |
| 5509 | log.Fatalf("carlos system update: %s", apiErrorFromBody(st, body, err)) |
| 5510 | } |
| 5511 | var started struct{ Job string `json:"job"` } |
| 5512 | _ = json.Unmarshal(body, &started) |
| 5513 | if err := sess.watchSystemJob(ctx, started.Job, stdout); err != nil { |
| 5514 | log.Fatal(err) |
| 5515 | } |
| 5516 | fmt.Fprintf(stdout, "%s boxes on %s roll themselves; watch with: carlos system status --channel %s\n", colorFor(stdout).ok("promoted"), *channel, *channel) |
| 5517 | } |
| 5518 | |
| 5519 | // watchSystemJob polls the job and prints each step once, as deploy's |
| 5520 | // watchVersion does for a route. |
| 5521 | func (s *apiSession) watchSystemJob(ctx context.Context, id string, w io.Writer) error { |
| 5522 | printed := 0 |
| 5523 | deadline := time.Now().Add(15 * time.Minute) |
| 5524 | for { |
| 5525 | st, body, err := s.getJSON(ctx, "/api/cli/system/jobs/"+id) |
| 5526 | if err != nil || st != http.StatusOK { |
| 5527 | return fmt.Errorf("job %s: %s", id, apiErrorFromBody(st, body, err)) |
| 5528 | } |
| 5529 | var j struct { |
| 5530 | Steps []string `json:"steps"` |
| 5531 | Done bool `json:"done"` |
| 5532 | Error string `json:"error"` |
| 5533 | } |
| 5534 | _ = json.Unmarshal(body, &j) |
| 5535 | for ; printed < len(j.Steps); printed++ { |
| 5536 | fmt.Fprintf(w, " %s\n", j.Steps[printed]) |
| 5537 | } |
| 5538 | if j.Done { |
| 5539 | if j.Error != "" { |
| 5540 | return fmt.Errorf("carlos system update: %s", j.Error) |
| 5541 | } |
| 5542 | return nil |
| 5543 | } |
| 5544 | if time.Now().After(deadline) { |
| 5545 | return errors.New("carlos system update: the console's job did not finish in 15 minutes") |
| 5546 | } |
| 5547 | time.Sleep(time.Second) |
| 5548 | } |
| 5549 | } |
| 5550 | |
| 5551 | func (s *apiSession) uploadSystemBinary(ctx context.Context, path, version, platform string) (string, error) { |
| 5552 | st, body, err := s.c.doMultipart(ctx, "/api/cli/system/upload", func(mw *multipart.Writer) error { |
| 5553 | meta, err := mw.CreateFormField("meta") |
| 5554 | if err != nil { |
| 5555 | return err |
| 5556 | } |
| 5557 | if err := json.NewEncoder(meta).Encode(map[string]string{"version": version, "platform": platform}); err != nil { |
| 5558 | return err |
| 5559 | } |
| 5560 | return writeFilePart(mw, "file", path) |
| 5561 | }) |
| 5562 | if err != nil || st != http.StatusOK { |
| 5563 | return "", fmt.Errorf("%s", apiErrorFromBody(st, body, err)) |
| 5564 | } |
| 5565 | var out struct{ Upload string `json:"upload"` } |
| 5566 | _ = json.Unmarshal(body, &out) |
| 5567 | return out.Upload, nil |
| 5568 | } |
| 5569 | |
| 5570 | func runSystemStatus(args []string) { |
| 5571 | fs := flag.NewFlagSet("system status", flag.ExitOnError) |
| 5572 | consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)") |
| 5573 | channel := fs.String("channel", "", "only boxes on this channel") |
| 5574 | fs.Usage = flagUsage(fs, "") |
| 5575 | fs.Parse(args) |
| 5576 | sess := requireDeployAPISession(*consoleFlag) |
| 5577 | st, body, err := sess.getJSON(context.Background(), "/api/cli/system/status") |
| 5578 | if err != nil || st != http.StatusOK { |
| 5579 | log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err)) |
| 5580 | } |
| 5581 | var resp systemStatusResponse |
| 5582 | if err := json.Unmarshal(body, &resp); err != nil { |
| 5583 | log.Fatalf("carlos system status: %v", err) |
| 5584 | } |
| 5585 | printSystemStatus(stdout, resp, *channel) |
| 5586 | } |
| 5587 | |
| 5588 | func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string) { |
| 5589 | for _, ch := range []string{"canary", "stable"} { |
| 5590 | if channel != "" && ch != channel { |
| 5591 | continue |
| 5592 | } |
| 5593 | p := resp.Pointers[ch] |
| 5594 | if p == nil { |
| 5595 | fmt.Fprintf(w, "%-7s (no pointer)\n", ch) |
| 5596 | continue |
| 5597 | } |
| 5598 | fmt.Fprintf(w, "%-7s %s promoted %s spread %s %s\n", ch, p.Version, p.PromotedAt.Format(time.RFC3339), p.Spread, p.Source) |
| 5599 | } |
| 5600 | fmt.Fprintln(w) |
| 5601 | c := colorFor(w) |
| 5602 | for _, b := range resp.Boxes { |
| 5603 | if channel != "" && b.Channel != channel { |
| 5604 | continue |
| 5605 | } |
| 5606 | state := "" |
| 5607 | switch { |
| 5608 | case b.Stale: |
| 5609 | state = c.bad("UNREACHABLE") |
| 5610 | case b.Platform == "": |
| 5611 | state = c.warn("pre-upgrade (no platform reported; counts as stable)") |
| 5612 | case b.Target == "": |
| 5613 | state = "no target" |
| 5614 | case b.Release == b.Target: |
| 5615 | state = c.ok("current") |
| 5616 | default: |
| 5617 | state = c.warn("behind → " + b.Target) |
| 5618 | } |
| 5619 | fmt.Fprintf(w, "%-20s %-12s %-7s %-10s %s", b.Label, orDash(b.Platform), b.Channel, orDash(b.Release), state) |
| 5620 | if b.System != nil && b.System.Outcome != "applied" && b.System.Outcome != "noted" { |
| 5621 | fmt.Fprintf(w, " %s %s: %s", c.bad(b.System.Outcome), b.System.Version, b.System.Reason) |
| 5622 | } |
| 5623 | fmt.Fprintln(w) |
| 5624 | } |
| 5625 | } |
| 5626 | |
| 5627 | func runSystemRollback(args []string) { |
| 5628 | fs := flag.NewFlagSet("system rollback", flag.ExitOnError) |
| 5629 | consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)") |
| 5630 | channel := fs.String("channel", "", "canary or stable (required)") |
| 5631 | yes := fs.Bool("yes", false, "do not ask") |
| 5632 | fs.Usage = flagUsage(fs, "") |
| 5633 | fs.Parse(args) |
| 5634 | if *channel == "" { |
| 5635 | log.Fatal("--channel is required") |
| 5636 | } |
| 5637 | sess := requireDeployAPISession(*consoleFlag) |
| 5638 | if !*yes { |
| 5639 | if !interactiveTerminal() { |
| 5640 | log.Fatal("no terminal to confirm on; pass --yes") |
| 5641 | } |
| 5642 | if !promptYes(stdout, fmt.Sprintf("point %s back at the release before this one? every box on it rolls back [y/N] ", *channel)) { |
| 5643 | return |
| 5644 | } |
| 5645 | } |
| 5646 | st, body, err := sess.postJSON(context.Background(), "/api/cli/system/rollback", map[string]string{"channel": *channel}) |
| 5647 | if err != nil || st != http.StatusOK { |
| 5648 | log.Fatalf("carlos system rollback: %s", apiErrorFromBody(st, body, err)) |
| 5649 | } |
| 5650 | var out struct{ Version string `json:"version"` } |
| 5651 | _ = json.Unmarshal(body, &out) |
| 5652 | fmt.Fprintf(stdout, "%s %s → %s\n", colorFor(stdout).ok("rolled back"), *channel, out.Version) |
| 5653 | } |
| 5654 | ``` |
| 5655 | Imports: `context`, `encoding/json`, `errors`, `flag`, `io`, `mime/multipart`, `net/http`, `time`. `orDash` exists in `cli.go:3327+`; `apiErrorFromBody` exists in `apimode.go` (confirm its signature with `grep -n 'func apiErrorFromBody' *.go`; adapt the three-argument call if it takes fewer). `sess.c` is the `*consoleClient` field on `apiSession` (confirm with `grep -n 'type apiSession struct' -A 6 apimode.go`). |
| 5656 | |
| 5657 | - [ ] **Step 4: Run the tests and try the binary against a fixture console** |
| 5658 | |
| 5659 | Run: `export TMPDIR=/tmp/c; out=$(go test . -run 'TestPrintSystemStatus|TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go vet . ` |
| 5660 | Expected: PASS, vet clean. |
| 5661 | |
| 5662 | - [ ] **Step 5: Commit** |
| 5663 | |
| 5664 | ```bash |
| 5665 | gofmt -l . && git add system.go system_test.go && git commit -q -F - <<'EOF' |
| 5666 | cli: carlos system update, status, rollback through the console |
| 5667 | |
| 5668 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 5669 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 5670 | EOF |
| 5671 | ``` |
| 5672 | |
| 5673 | --- |
| 5674 | |
| 5675 | ### Task 13: The hourly belt — `carlos adopt` arms the self-update units |
| 5676 | |
| 5677 | **Files:** |
| 5678 | - Modify: `internal/host/agent.go` `RunAdopt` (`:2811`; insert before `openServiceRegistry` at `:2825`) |
| 5679 | - Create: `internal/host/systemunits.go`, `internal/host/systemunits_test.go` |
| 5680 | |
| 5681 | **Interfaces:** |
| 5682 | - Consumes: `unitsFS`, `systemctlPlan`, `copyFileDurable`, a `sysOps`-shaped `systemctl` func. |
| 5683 | - Produces: |
| 5684 | ```go |
| 5685 | // ensureSystemUnits installs and arms carlos-system-update.path/.service |
| 5686 | // and carlos-system-recover.service when any is missing from etcDir or |
| 5687 | // the path unit is inactive / the recover unit not enabled. Non-fatal. |
| 5688 | func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any)) |
| 5689 | ``` |
| 5690 | |
| 5691 | - [ ] **Step 1: Write the failing test** |
| 5692 | |
| 5693 | ```go |
| 5694 | package host |
| 5695 | |
| 5696 | import ( |
| 5697 | "os" |
| 5698 | "path/filepath" |
| 5699 | "strings" |
| 5700 | "testing" |
| 5701 | ) |
| 5702 | |
| 5703 | func TestEnsureSystemUnitsInstallsAndArmsWhatIsMissing(t *testing.T) { |
| 5704 | etc := t.TempDir() |
| 5705 | os.MkdirAll(filepath.Join(etc, "systemd", "system"), 0o755) |
| 5706 | var calls []string |
| 5707 | systemctl := func(args ...string) (string, error) { |
| 5708 | calls = append(calls, strings.Join(args, " ")) |
| 5709 | switch args[0] { |
| 5710 | case "is-active": |
| 5711 | return "inactive\n", nil |
| 5712 | case "is-enabled": |
| 5713 | return "disabled\n", nil |
| 5714 | } |
| 5715 | return "", nil |
| 5716 | } |
| 5717 | ensureSystemUnits(etc, systemctl, t.Logf) |
| 5718 | for _, u := range []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"} { |
| 5719 | if _, err := os.Stat(filepath.Join(etc, "systemd", "system", u)); err != nil { |
| 5720 | t.Errorf("%s not installed", u) |
| 5721 | } |
| 5722 | } |
| 5723 | joined := strings.Join(calls, "\n") |
| 5724 | for _, want := range []string{"daemon-reload", "enable carlos-system-update.path carlos-system-recover.service", "start carlos-system-update.path"} { |
| 5725 | if !strings.Contains(joined, want) { |
| 5726 | t.Errorf("calls lack %q:\n%s", want, joined) |
| 5727 | } |
| 5728 | } |
| 5729 | if strings.Contains(joined, "start carlos-system-recover.service") || strings.Contains(joined, "start carlos-system-update.service") { |
| 5730 | t.Fatal("a service was started") |
| 5731 | } |
| 5732 | // All present and armed: no writes, no systemctl beyond the two checks. |
| 5733 | calls = nil |
| 5734 | systemctl2 := func(args ...string) (string, error) { |
| 5735 | calls = append(calls, strings.Join(args, " ")) |
| 5736 | switch args[0] { |
| 5737 | case "is-active": |
| 5738 | return "active\n", nil |
| 5739 | case "is-enabled": |
| 5740 | return "enabled\n", nil |
| 5741 | } |
| 5742 | return "", nil |
| 5743 | } |
| 5744 | ensureSystemUnits(etc, systemctl2, t.Logf) |
| 5745 | if len(calls) != 2 { |
| 5746 | t.Fatalf("an armed box got %v", calls) |
| 5747 | } |
| 5748 | } |
| 5749 | ``` |
| 5750 | |
| 5751 | - [ ] **Step 2: Run to verify failure** |
| 5752 | |
| 5753 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run TestEnsureSystemUnits -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc` |
| 5754 | |
| 5755 | - [ ] **Step 3: Implement** |
| 5756 | |
| 5757 | ```go |
| 5758 | package host |
| 5759 | |
| 5760 | import ( |
| 5761 | "os" |
| 5762 | "path/filepath" |
| 5763 | "strings" |
| 5764 | ) |
| 5765 | |
| 5766 | var systemUnitFiles = []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"} |
| 5767 | |
| 5768 | // ensureSystemUnits is the second belt of spec §7: a box the roll script |
| 5769 | // forgot to arm is armed within the hour by the root adopt pass. It |
| 5770 | // installs the embedded units when missing and arms them through |
| 5771 | // systemctlPlan (never starting a service). Non-fatal: it runs before |
| 5772 | // RunAdopt's registry and pubkey gates, and logs instead of exiting. |
| 5773 | func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any)) { |
| 5774 | unitDir := filepath.Join(etcDir, "systemd", "system") |
| 5775 | missing := false |
| 5776 | for _, u := range systemUnitFiles { |
| 5777 | if _, err := os.Stat(filepath.Join(unitDir, u)); err != nil { |
| 5778 | missing = true |
| 5779 | } |
| 5780 | } |
| 5781 | active, _ := systemctl("is-active", systemUpdatePathUnit) |
| 5782 | enabled, _ := systemctl("is-enabled", "carlos-system-recover.service") |
| 5783 | if !missing && strings.TrimSpace(active) == "active" && strings.TrimSpace(enabled) == "enabled" { |
| 5784 | return |
| 5785 | } |
| 5786 | for _, u := range systemUnitFiles { |
| 5787 | b, err := unitsFS.ReadFile("units/" + u) |
| 5788 | if err != nil { |
| 5789 | logf("adopt: system units: %v", err) |
| 5790 | return |
| 5791 | } |
| 5792 | if err := writeFileDurable(filepath.Join(unitDir, u), b, 0o644); err != nil { |
| 5793 | logf("adopt: system units: %v", err) |
| 5794 | return |
| 5795 | } |
| 5796 | } |
| 5797 | for _, args := range systemctlPlan([]string{systemUpdatePathUnit, "carlos-system-recover.service"}) { |
| 5798 | if _, err := systemctl(args...); err != nil { |
| 5799 | logf("adopt: system units: systemctl %s: %v", strings.Join(args, " "), err) |
| 5800 | return |
| 5801 | } |
| 5802 | } |
| 5803 | logf("adopt: self-update units installed and armed") |
| 5804 | } |
| 5805 | ``` |
| 5806 | In `RunAdopt`, immediately after `fs.Parse(args)` and before `openServiceRegistry`: |
| 5807 | ```go |
| 5808 | // Spec §7 belt 2: arm the self-update relay if the last hand roll did |
| 5809 | // not. Before the fatal gates below, and never fatal itself. |
| 5810 | if !*restartOnly { |
| 5811 | ensureSystemUnits("/etc", productionSysOps().systemctl, log.Printf) |
| 5812 | } |
| 5813 | ``` |
| 5814 | |
| 5815 | - [ ] **Step 4: Run** |
| 5816 | |
| 5817 | Run: `export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc` |
| 5818 | |
| 5819 | - [ ] **Step 5: Commit** |
| 5820 | |
| 5821 | ```bash |
| 5822 | gofmt -l internal/host/ && git add internal/host/systemunits.go internal/host/systemunits_test.go internal/host/agent.go && git commit -q -F - <<'EOF' |
| 5823 | host: the hourly adopt pass arms the self-update units if the roll script did not |
| 5824 | |
| 5825 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 5826 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 5827 | EOF |
| 5828 | ``` |
| 5829 | |
| 5830 | --- |
| 5831 | |
| 5832 | ### Task 14: Docs, the env contract, and the last two script lines |
| 5833 | |
| 5834 | **Files:** |
| 5835 | - Modify: `docs/site/cli.md` (after `### carlos status`, ~line 930), `AGENTS.md` (the section that names `carlos-stage.path` / `update-platform-binary.sh`, ~lines 246-277) |
| 5836 | - Out of repo, hand off (do not edit from this checkout): `platform-infrastructure/scripts/update-platform-binary.sh` and the Tito copy in `amadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh` |
| 5837 | |
| 5838 | - [ ] **Step 1: `docs/site/cli.md`** — add under `## Your deployment`, after `### carlos status`: |
| 5839 | |
| 5840 | ````markdown |
| 5841 | ### carlos system |
| 5842 | |
| 5843 | The platform's own release channel. `update` promotes a public release (or, |
| 5844 | with `--file`, a binary you built) to `canary` or `stable`; every box on that |
| 5845 | channel fetches it through its normal store, verifies it against the signed |
| 5846 | pointer, installs it, restarts its edge, and rolls back to the previous |
| 5847 | binary if the new one does not come up and stay up. `status` shows each box's |
| 5848 | platform, channel, running build, the channel's target, and the last apply |
| 5849 | outcome with its reason. `rollback` points a channel back at the release |
| 5850 | before this one, and every box on it rolls back the same way it rolled |
| 5851 | forward. All three talk to your console; you need to be an owner of the ops |
| 5852 | account. |
| 5853 | |
| 5854 | ```sh |
| 5855 | carlos system update --channel canary |
| 5856 | carlos system status --channel canary |
| 5857 | carlos system update --channel stable --spread 10m |
| 5858 | carlos system rollback --channel stable |
| 5859 | ``` |
| 5860 | |
| 5861 | A box is put on canary with `CARLOS_SYSTEM_CHANNEL=canary` in its host.env. |
| 5862 | Boxes that have not reported a platform yet count as stable. `carlos system |
| 5863 | apply` is the root oneshot behind `carlos-system-update.service`; it is not |
| 5864 | for operators. The console is an app — update it with `carlos deploy --app |
| 5865 | console`. |
| 5866 | ```` |
| 5867 | |
| 5868 | - [ ] **Step 2: `AGENTS.md`** — in the section describing the root relays, add after the `carlos-restart.path` paragraph: |
| 5869 | |
| 5870 | ```markdown |
| 5871 | - `carlos-system-update.path` relays the agent's platform self-update request |
| 5872 | to `carlos system apply` (root; spec 2026-09-06-carlos-system-update-design.md |
| 5873 | §6). `carlos-system-recover.service` runs once per boot, before the edge, |
| 5874 | and on the apply's failure. Both must be armed; the last run of |
| 5875 | `update-platform-binary.sh` does it, and the hourly `carlos adopt` pass |
| 5876 | installs and arms them if it did not. A box follows `CARLOS_SYSTEM_CHANNEL` |
| 5877 | (canary|stable, default stable) in host.env. |
| 5878 | ``` |
| 5879 | |
| 5880 | - [ ] **Step 3: The pinfra hand-off** — write `docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.md` with exactly this content, for whoever edits the two scripts (they are in other repos; this checkout does not touch them): |
| 5881 | |
| 5882 | ```markdown |
| 5883 | # update-platform-binary.sh: the last two lines it needs |
| 5884 | |
| 5885 | After the `carlos-restart.path` block, guarded the same way: |
| 5886 | |
| 5887 | if [ -f /etc/systemd/system/carlos-system-update.path ]; then |
| 5888 | systemctl enable --now carlos-system-update.path |
| 5889 | systemctl enable carlos-system-recover.service |
| 5890 | fi |
| 5891 | |
| 5892 | And to the post-conditions at the end: |
| 5893 | |
| 5894 | systemctl is-active carlos-system-update.path |
| 5895 | systemctl is-enabled carlos-system-recover.service |
| 5896 | |
| 5897 | Same change in amadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh. |
| 5898 | After every deployment has been hand-rolled once onto a binary carrying the |
| 5899 | system-update spec, the script is retired (spec §10). |
| 5900 | ``` |
| 5901 | |
| 5902 | - [ ] **Step 4: Full gate** |
| 5903 | |
| 5904 | Run: `export TMPDIR=/tmp/c; out=$(make test 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc` |
| 5905 | Expected: rc=0 (vet, gofmt, every package). |
| 5906 | |
| 5907 | - [ ] **Step 5: Commit, push, describe, land** |
| 5908 | |
| 5909 | ```bash |
| 5910 | git add docs/site/cli.md AGENTS.md docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.md && git commit -q -F - <<'EOF' |
| 5911 | docs: carlos system, the env contract, and the roll script's last two lines |
| 5912 | |
| 5913 | Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
| 5914 | Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5 |
| 5915 | EOF |
| 5916 | git push -u origin carlos-system-update |
| 5917 | amadan branch describe carlos/platform carlos-system-update -body - <<'EOF' |
| 5918 | carlos system update: the platform rolls itself. Spec: docs/superpowers/specs/2026-09-06-carlos-system-update-design.md. Plan: docs/superpowers/plans/2026-09-06-carlos-system-update.md. |
| 5919 | EOF |
| 5920 | # then: amadan ci status carlos/platform -branch carlos-system-update (until 0) |
| 5921 | # then: amadan branch merge carlos/platform carlos-system-update -expect <full sha> |
| 5922 | ``` |
| 5923 | |
| 5924 | --- |
| 5925 | |
| 5926 | ## Self-review |
| 5927 | |
| 5928 | **Spec coverage.** §3 pointer, keys, immutability, spread, history-of-releases, rollback-is-new-promote, door grants, streaming read → Tasks 1, 2, 3. §4 leg (observe/tick, re-touch, watermark check, suppression, jitter, streamed fetch, request shape, deferred retry), units, exec shim, `enabledUnits` → Tasks 6, 7. §5 authorization, job, upload, update/status/rollback, `router` alias, no console verb → Tasks 9, 11, 12. §6 root state files, durability, lock, recovery-first (forward from gated), request walk, channel from env, replay guard per channel, no-op, staging copy 0600→0755, `.prev` decision, both unit sets from their binaries, quiesce incl. services, removed units, sysusers→tmpfiles→daemon-reload, reset-failed, gate+soak, arm after gate via `systemctlPlan`, record, rollback, rollback-failed → Tasks 4, 8a–8c. §7 first-roll arming (script + hourly belt), env contract → Tasks 13, 14. §9/§10 nothing to build. §11 unit-level proof: every bullet maps to a test in Tasks 1–13; the live checks are the operator's after landing. |
| 5929 | |
| 5930 | **Gaps found and fixed in review:** the spec's `release.Artifact` name collides with an existing type — Task 1 uses `SystemArtifact`; the spec's "interface gains `Open`" would break 48 fakes — Task 2 uses an `Opener` side-interface; the spec's 403 on the system doors is a bare 404 here, matching the console's operator doors; the spec's "ops-account owner" is `Role(OpsAccountID) == RoleOwner`, not `IsOperator` — Task 11 says which. |
| 5931 | |
| 5932 | **Placeholder scan.** Tasks 9 and 8a/8b carry explicit stubs that the next task replaces (`errNotYetImplemented`, the placeholder verbs, the stub `rollback`); each is named and replaced in a listed task. No "TBD", no "similar to". |
| 5933 | |
| 5934 | **Type consistency.** `systemRequest{Version, PromotedAt, NoOp, SeenAt}` is written by Task 7 and read by 8a; `systemResult.matches/terminal` are used identically in 7 and 8a; `bootstrapPlan{Enabled, Inert}` is produced by Task 6 and consumed by 8b/8c/13; `sysOps` fields are the same in 8a's fixture and 8c's `productionSysOps`; `release.SystemPromoteOptions{Actor, Spread}` in 1, 7's fixture and 11; `systemStatusResponse` in 12 matches the JSON Task 11 writes (`pointers`, `boxes` with `label, region, platform, channel, release, target, system, stale`). |
| 5935 | |