amadan
carlos / platform Public
carlos system update Implementation Plan
> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Goal: A fleet operator runs carlos system update and every box on the channel fetches, verifies, installs and health-gates the new platform binary itself, rolling back to the previous one if it does not come up.
Architecture: A signed SystemPointer at control/system/router/<channel>.json names a version and its per-platform artifacts mirrored into the deployment bucket. A sandboxed agent leg notices the pointer on the serial poll, streams and verifies the artifact into /data, and touches a marker. A root oneshot (carlos system apply) re-verifies everything from root-owned copies, journals each step, swaps binary and units, restarts the edge, gates it, and rolls back to carlos.prev on failure; a boot-time unit recovers an interrupted apply. The console does the promote: fetch the public signed release, mirror, sign the pointer, bump the serial.
Tech Stack: Go (stdlib, ed25519, net/http), systemd units embedded in the binary, S3/dir/fleet-door stores behind release.Store.
Spec: docs/superpowers/specs/2026-09-06-carlos-system-update-design.md — read §3, §4, §6 before the task you are on. The spec wins where this plan and it disagree; say so in the commit if you find such a place.
Global Constraints
- Go build cache is read-only in the sandbox: run every
gocommand withdangerouslyDisableSandbox: true, andexport TMPDIR=/tmp/cfirst (long socket paths breakinternal/edge/internal/activatortests). - Capture test exit codes explicitly:
out=$(go test ./pkg/... 2>&1); rc=$?; echo "$out" | tail -40; echo rc=$rc. Never pipego testintogrep/tailand read the pipeline's status. -count=1on every test run.gofmt -l .must print nothing before a commit.- Never run full
carlos bootstrapon a live box path; the apply stages units into a temp root (bootstrap -root <dir>) and copies onlyetc/. Never copy a staging root'sdata/. - sysusers before tmpfiles, always, in every code path that installs units.
- Root reads exactly one file under
/data(requested.json), walks to it withO_DIRECTORY|O_NOFOLLOWper component, and derives every other path from a validated version string. Root never deletes under/data. - Apply never
systemctl starts a.service; it callssystemctlPlan, which starts only.timer/.pathunits. - Every root state file is written temp → fsync → rename → directory fsync (
writeFileDurable, Task 4). - The component name is
router; channels arecanaryandstable. Platform strings are<goos>-<goarch>matching^[a-z0-9]+-[a-z0-9]+$. - Commit after every task with the attribution footer: ``
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5`` - Work on branch
carlos-system-updatefrom forgemain; land withamadan branch merge, nevergh.
---
File Structure
| File | Responsibility | |---|---| | internal/release/system.go (+_test) | SystemPointer, SystemRelease, SystemArtifact; key builders; validate/sign/verify (root key only); GetSystemPointer, PromoteSystem, RollbackSystem | | internal/release/store.go, internal/fleetapi/fleetapi.go (+tests) | Opener side-interface, Open on DirStore/S3Store/prefixStore/fleetapi, OpenObject, FetchToFile | | internal/edge/storedoor.go (+_test) | aclRead grants for the two new key shapes | | internal/host/systemstate.go (+_test) | On-box state types (request, result, seen, applied, journal), paths, writeFileDurable, outcome constants | | internal/status/snapshot.go, internal/host/agent.go | Platform, SystemChannel, System in Probes/BoxMetrics; filled in statusLoop | | internal/host/units/carlos-system-update.path, carlos-system-update.service, carlos-system-recover.service; internal/host/bootstrap.go; internal/host/probes.go | The three units, enabledUnits, -plan-json, the path-unit probe | | internal/host/systemleg.go (+_test) | systemLeg: observe/tick, fetch, request, marker re-touch; wired into fastDeps | | internal/host/systemapply.go (+_test) | RunSystemApply, systemApplier with an injectable sysOps seam: refuse path, install path, gate, rollback, recovery, lock | | internal/host/agent.go RunAdopt | The hourly belt: install and arm the three units if missing | | internal/publicrelease/feed.go (+_test) | Feed: latest tag, verified asset fetch — factored out of update.go, shared with the console | | internal/console/apisystem.go (+_test) | /api/cli/system/{status,update,upload,jobs/{id},rollback}; ops-owner gate; in-memory update job | | system.go (+_test, package main), main.go | carlos system update|status|rollback|apply, carlos system router … alias | | docs/site/cli.md, AGENTS.md | Operator docs; the env contract |
Task order is dependency order. Tasks 1–3 are independent of each other and can run in parallel; 4 before 5/7/8; 6 before 8; 10 before 11; 11 before 12.
---
Task 1: The system pointer (internal/release/system.go)
Files:
- Create:
internal/release/system.go - Test:
internal/release/system_test.go
Interfaces:
- Consumes:
signedWrapper,Signer(fieldskey,scope, methodusable()),Promotion,historyMax,auditMax,prependCapped,ValidChannel,ValidVersion,ParsePublicKey,Store,ErrCASConflict— all in packagerelease(release.go,signer.go,store.go). - Produces (all exported, package
release): ``go const SystemComponentRouter = "router" func SystemPointerKey(component, channel string) string // control/system/<c>/<ch>.json func SystemArtifactKey(version, platform string) string // bootstrap/router/<v>/carlos-<platform> type SystemArtifact struct{ Key, SHA256 string } type SystemRelease struct{ Version string; Artifacts map[string]SystemArtifact; Source string; ReleasedAt time.Time } type SystemPointer struct{ Schema int; Component, Channel string; Current SystemRelease; History []SystemRelease; Spread string; PromotedAt time.Time; Audit []Promotion } func (p *SystemPointer) SpreadDuration() time.Duration func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error) type SystemPromoteOptions struct{ Actor string; Spread time.Duration } func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error) func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error) var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to")``
- [ ] Step 1: Write the failing tests
package release
import (
"context"
"encoding/json"
"errors"
"os"
"strings"
"testing"
"time"
)
func testSystemRelease(v string) SystemRelease {
return SystemRelease{
Version: v,
Artifacts: map[string]SystemArtifact{
"linux-arm64": {Key: SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("ab", 32)},
},
Source: "release:" + v,
ReleasedAt: time.Date(2026, 9, 6, 8, 0, 0, 0, time.UTC),
}
}
func TestSystemPointerRoundTrip(t *testing.T) {
priv, pub := testKeys(t)
s := NewDirStore(t.TempDir())
ctx := context.Background()
if _, _, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable"); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("missing pointer: err = %v, want ErrNotExist", err)
}
p, err := PromoteSystem(ctx, s, RootSigner(priv), SystemComponentRouter, "stable", testSystemRelease("v0.18.0"),
SystemPromoteOptions{Actor: "paul", Spread: 5 * time.Minute})
if err != nil {
t.Fatal(err)
}
if p.Current.Version != "v0.18.0" || p.Spread != "5m0s" || p.Schema != 1 || p.PromotedAt.IsZero() {
t.Fatalf("promoted pointer = %+v", p)
}
got, etag, err := GetSystemPointer(ctx, s, pub, SystemComponentRouter, "stable")
if err != nil || etag == "" {
t.Fatalf("get: %v etag=%q", err, etag)
}
if got.Current.Artifacts["linux-arm64"].SHA256 != strings.Repeat("ab", 32) {
t.Fatalf("artifact lost: %+v", got.Current)
}
if got.SpreadDuration() != 5*time.Minute {
t.Fatalf("spread = %v", got.SpreadDuration())
}
if len(got.Audit) != 1 || got.Audit[0].Actor != "paul" || got.Audit[0].Version != "v0.18.0" {
t.Fatalf("audit = %+v", got.Audit)
}
}
func TestSystemPromoteKeepsWholeReleaseInHistoryAndRollbackRestoresIt(t *testing.T) {
priv, pub := testKeys(t)
s := NewDirStore(t.TempDir())
ctx := context.Background()
sg := RootSigner(priv)
if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.18.0"), SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
first, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary")
time.Sleep(1100 * time.Millisecond) // PromotedAt is second-truncated; the rollback must be NEWER
if _, err := PromoteSystem(ctx, s, sg, "router", "canary", testSystemRelease("v0.19.0"), SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
p, _, _ := GetSystemPointer(ctx, s, pub, "router", "canary")
if len(p.History) != 1 || p.History[0].Version != "v0.18.0" || p.History[0].Artifacts["linux-arm64"].Key == "" {
t.Fatalf("history must carry the whole release: %+v", p.History)
}
time.Sleep(1100 * time.Millisecond)
rb, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul")
if err != nil {
t.Fatal(err)
}
if rb.Current.Version != "v0.18.0" || rb.Current.Artifacts["linux-arm64"].Key != SystemArtifactKey("v0.18.0", "linux-arm64") {
t.Fatalf("rollback current = %+v", rb.Current)
}
if !rb.PromotedAt.After(first.PromotedAt) || !rb.PromotedAt.After(p.PromotedAt) {
t.Fatalf("a rollback is a NEW promote: promoted_at %v must be newer than %v and %v", rb.PromotedAt, first.PromotedAt, p.PromotedAt)
}
if len(rb.History) != 0 {
t.Fatalf("history after rollback = %+v", rb.History)
}
if _, err := RollbackSystem(ctx, s, sg, "router", "canary", "paul"); !errors.Is(err, ErrNoSystemHistory) {
t.Fatalf("second rollback err = %v", err)
}
}
func TestSystemPromoteSameVersionDoesNotGrowHistory(t *testing.T) {
priv, pub := testKeys(t)
s := NewDirStore(t.TempDir())
ctx := context.Background()
sg := RootSigner(priv)
for i := 0; i < 2; i++ {
if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
}
p, _, _ := GetSystemPointer(ctx, s, pub, "router", "stable")
if len(p.History) != 0 {
t.Fatalf("re-promoting the running version must not push it into history: %+v", p.History)
}
}
func TestSystemPointerRefusesScopedSigner(t *testing.T) {
_, _, sg := scopedFixture(t, []string{RungCanary, RungEdge}, 30*24*time.Hour)
s := NewDirStore(t.TempDir())
_, err := PromoteSystem(context.Background(), s, sg, "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{})
if err == nil || !strings.Contains(err.Error(), "root key") {
t.Fatalf("scoped signer signed a system pointer: err = %v", err)
}
}
func TestSystemPointerRefusesAGrantOnTheWire(t *testing.T) {
// A wrapper carrying a grant is refused BEFORE the signature is checked,
// like the production flag (production.go) and the pipeline.
priv, pub := testKeys(t)
s := NewDirStore(t.TempDir())
ctx := context.Background()
if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
body, etag, _ := s.Get(ctx, SystemPointerKey("router", "stable"))
var sw signedWrapper
if err := json.Unmarshal(body, &sw); err != nil {
t.Fatal(err)
}
sw.Grant = &signedGrant{Payload: "x", Sig: "y"}
tampered, _ := json.Marshal(sw)
if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), tampered, etag); err != nil {
t.Fatal(err)
}
if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil || !strings.Contains(err.Error(), "root key") {
t.Fatalf("grant-bearing wrapper accepted: err = %v", err)
}
}
func TestSystemPointerTamperAndCrossChannelRefused(t *testing.T) {
priv, pub := testKeys(t)
s := NewDirStore(t.TempDir())
ctx := context.Background()
if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "canary", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
body, _, _ := s.Get(ctx, SystemPointerKey("router", "canary"))
// Cross-channel replay: the canary wrapper planted at the stable key.
if _, err := s.Put(ctx, SystemPointerKey("router", "stable"), body, ""); err != nil {
t.Fatal(err)
}
if _, _, err := GetSystemPointer(ctx, s, pub, "router", "stable"); err == nil {
t.Fatal("a canary pointer verified as stable")
}
// Tampered signature.
var sw signedWrapper
_ = json.Unmarshal(body, &sw)
sw.Sig = "AA" + sw.Sig[2:]
bad, _ := json.Marshal(sw)
_, etag, _ := s.Get(ctx, SystemPointerKey("router", "canary"))
if _, err := s.Put(ctx, SystemPointerKey("router", "canary"), bad, etag); err != nil {
t.Fatal(err)
}
if _, _, err := GetSystemPointer(ctx, s, pub, "router", "canary"); err == nil {
t.Fatal("tampered signature verified")
}
}
func TestSystemPointerValidation(t *testing.T) {
priv, _ := testKeys(t)
sg := RootSigner(priv)
ctx := context.Background()
cases := map[string]SystemRelease{
"dev version": {Version: "dev", Artifacts: testSystemRelease("x").Artifacts},
"no artifacts": {Version: "v1.0.0"},
"bad platform": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux_arm64": {Key: "bootstrap/router/v1.0.0/carlos-linux_arm64", SHA256: strings.Repeat("a", 64)}}},
"key not derived": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: "bootstrap/carlos", SHA256: strings.Repeat("a", 64)}}},
"short sha": {Version: "v1.0.0", Artifacts: map[string]SystemArtifact{"linux-arm64": {Key: SystemArtifactKey("v1.0.0", "linux-arm64"), SHA256: "abc"}}},
"version traversal": {Version: "../x", Artifacts: testSystemRelease("../x").Artifacts},
}
for name, rel := range cases {
s := NewDirStore(t.TempDir())
if _, err := PromoteSystem(ctx, s, sg, "router", "stable", rel, SystemPromoteOptions{}); err == nil {
t.Errorf("%s: accepted", name)
}
}
s := NewDirStore(t.TempDir())
if _, err := PromoteSystem(ctx, s, sg, "Router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); err == nil {
t.Error("bad component accepted")
}
if _, err := PromoteSystem(ctx, s, sg, "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{Spread: 48 * time.Hour}); err == nil {
t.Error("48h spread accepted")
}
}
func TestSystemPromoteCASConflict(t *testing.T) {
priv, _ := testKeys(t)
ctx := context.Background()
inner := NewDirStore(t.TempDir())
// A store whose Put always says "someone else moved it".
s := conflictingStore{Store: inner}
if _, err := PromoteSystem(ctx, s, RootSigner(priv), "router", "stable", testSystemRelease("v1.0.0"), SystemPromoteOptions{}); !errors.Is(err, ErrCASConflict) {
t.Fatalf("err = %v, want ErrCASConflict", err)
}
}
type conflictingStore struct{ Store }
func (c conflictingStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
return "", ErrCASConflict
}- [ ] Step 2: Run the tests to verify they fail
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: build failure naming SystemPointer, PromoteSystem, etc. as undefined.
- [ ] Step 3: Implement
internal/release/system.go
package release
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"regexp"
"time"
)
// SystemPointer is the platform's own channel pointer: what a ChannelPointer
// is for an app, for the thing that does the adopting. It lives under
// control/, not apps/, because the substrate is not an app (the
// console-as-an-app spec's dividing line), and it is ROOT-KEY-ONLY: a scoped
// grant's rungs name app channels, and a key that can roll the platform is a
// bigger key than any app promoter should hold. Spec §3.
type SystemPointer struct {
Schema int `json:"schema"`
Component string `json:"component"`
Channel string `json:"channel"`
Current SystemRelease `json:"current"`
History []SystemRelease `json:"history,omitempty"`
Spread string `json:"spread"`
PromotedAt time.Time `json:"promoted_at"` // of THIS write; the box's replay guard reads only this
Audit []Promotion `json:"audit,omitempty"`
}
// SystemRelease is everything a box needs to install one version. It travels
// whole into History so a rollback re-installs what was there, not a version
// string with someone else's artifacts.
type SystemRelease struct {
Version string `json:"version"`
Artifacts map[string]SystemArtifact `json:"artifacts"` // keyed by "<goos>-<goarch>"
Source string `json:"source,omitempty"`
ReleasedAt time.Time `json:"released_at"` // audit only; never compared
}
// SystemArtifact is one mirrored binary. Its sha in the signed pointer IS
// the artifact's signature: boxes need no .sig object.
type SystemArtifact struct {
Key string `json:"key"`
SHA256 string `json:"sha256"`
}
// SystemComponentRouter is the one component today: the substrate binary
// (edge + activator + adoption, one process). A split adds a name here.
const SystemComponentRouter = "router"
// ErrNoSystemHistory is RollbackSystem's answer when the pointer has never
// moved: there is nothing older to go back to.
var ErrNoSystemHistory = errors.New("system pointer: nothing to roll back to")
const (
systemSpreadMax = 24 * time.Hour
systemSpreadDefault = 5 * time.Minute
)
var (
systemComponentRe = regexp.MustCompile(`^[a-z][a-z0-9-]{0,31}$`)
systemPlatformRe = regexp.MustCompile(`^[a-z0-9]+-[a-z0-9]+$`)
systemSHA256Re = regexp.MustCompile(`^[0-9a-f]{64}$`)
)
// SystemPointerKey is control/system/<component>/<channel>.json.
func SystemPointerKey(component, channel string) string {
return "control/system/" + component + "/" + channel + ".json"
}
// SystemArtifactKey is bootstrap/router/<version>/carlos-<platform>: under
// bootstrap/ because the store door already means "the platform's own
// binary" by that name, per-version because canary and stable boxes must be
// able to run different binaries at once.
func SystemArtifactKey(version, platform string) string {
return "bootstrap/router/" + version + "/carlos-" + platform
}
// SpreadDuration is Spread parsed, or the default when it is unparseable —
// validation refuses the unparseable case on the way in, so this only
// defaults for a hand-edited or pre-field pointer.
func (p *SystemPointer) SpreadDuration() time.Duration {
d, err := time.ParseDuration(p.Spread)
if err != nil || d < 0 || d > systemSpreadMax {
return systemSpreadDefault
}
return d
}
func validateSystemRelease(r SystemRelease) error {
if !ValidVersion(r.Version) || r.Version == "dev" {
return fmt.Errorf("version %q is not a valid platform version", r.Version)
}
if len(r.Artifacts) == 0 {
return fmt.Errorf("version %s names no artifacts", r.Version)
}
for plat, a := range r.Artifacts {
if !systemPlatformRe.MatchString(plat) {
return fmt.Errorf("platform %q is not <goos>-<goarch>", plat)
}
if a.Key != SystemArtifactKey(r.Version, plat) {
return fmt.Errorf("artifact key %q is not %s", a.Key, SystemArtifactKey(r.Version, plat))
}
if !systemSHA256Re.MatchString(a.SHA256) {
return fmt.Errorf("artifact %s: sha256 %q is not 64 hex characters", a.Key, a.SHA256)
}
}
return nil
}
func validateSystemPointer(p *SystemPointer, wantComponent, wantChannel string) error {
if p.Schema != 1 {
return fmt.Errorf("system pointer schema %d unsupported", p.Schema)
}
if p.Component != wantComponent || p.Channel != wantChannel {
return fmt.Errorf("system pointer is for %s/%s, not %s/%s", p.Component, p.Channel, wantComponent, wantChannel)
}
if !systemComponentRe.MatchString(p.Component) || !ValidChannel(p.Channel) {
return fmt.Errorf("system pointer %s/%s: bad component or channel", p.Component, p.Channel)
}
if err := validateSystemRelease(p.Current); err != nil {
return fmt.Errorf("system pointer %s/%s: %w", p.Component, p.Channel, err)
}
if d, err := time.ParseDuration(p.Spread); err != nil || d < 0 || d > systemSpreadMax {
return fmt.Errorf("system pointer %s/%s: spread %q must be a duration between 0 and %s", p.Component, p.Channel, p.Spread, systemSpreadMax)
}
if p.PromotedAt.IsZero() {
return fmt.Errorf("system pointer %s/%s: promoted_at unset", p.Component, p.Channel)
}
if len(p.History) > historyMax {
return fmt.Errorf("system pointer %s/%s: history longer than %d", p.Component, p.Channel, historyMax)
}
for _, h := range p.History {
if err := validateSystemRelease(h); err != nil {
return fmt.Errorf("system pointer %s/%s: history: %w", p.Component, p.Channel, err)
}
}
if len(p.Audit) > auditMax {
return fmt.Errorf("system pointer %s/%s: audit longer than %d", p.Component, p.Channel, auditMax)
}
return nil
}
// signSystemPointer mirrors signPointer with two differences: the signer
// must be the root key, and the wrapper never carries a grant.
func signSystemPointer(sg Signer, p *SystemPointer) ([]byte, error) {
if err := sg.usable(); err != nil {
return nil, err
}
if sg.scope != nil {
return nil, errors.New("system pointer: signed by a scoped key — the platform takes the root key only")
}
p.Schema = 1
p.PromotedAt = time.Now().UTC().Truncate(time.Second)
if err := validateSystemPointer(p, p.Component, p.Channel); err != nil {
return nil, err
}
payload, err := json.MarshalIndent(p, "", " ")
if err != nil {
return nil, err
}
b64 := base64.StdEncoding.EncodeToString(payload)
sig := ed25519.Sign(sg.key, []byte(b64))
return json.MarshalIndent(signedWrapper{Payload: b64, Sig: base64.StdEncoding.EncodeToString(sig)}, "", " ")
}
// verifySystemPointer is the single parse site for system pointers. A grant
// on the wire is refused before the signature is looked at (the production
// flag's rule, production.go).
func verifySystemPointer(pub ed25519.PublicKey, wrapper []byte, wantComponent, wantChannel string) (*SystemPointer, error) {
var sw signedWrapper
if err := json.Unmarshal(wrapper, &sw); err != nil {
return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err)
}
if sw.Grant != nil {
return nil, fmt.Errorf("system pointer %s/%s: signed by a scoped key — the platform takes the root key only", wantComponent, wantChannel)
}
sig, err := base64.StdEncoding.DecodeString(sw.Sig)
if err != nil {
return nil, fmt.Errorf("system pointer %s/%s: bad signature encoding", wantComponent, wantChannel)
}
if len(pub) != ed25519.PublicKeySize || !ed25519.Verify(pub, []byte(sw.Payload), sig) {
return nil, fmt.Errorf("system pointer %s/%s: signature does not verify", wantComponent, wantChannel)
}
payload, err := base64.StdEncoding.DecodeString(sw.Payload)
if err != nil {
return nil, fmt.Errorf("system pointer %s/%s: bad payload encoding", wantComponent, wantChannel)
}
var p SystemPointer
if err := json.Unmarshal(payload, &p); err != nil {
return nil, fmt.Errorf("system pointer %s/%s: %w", wantComponent, wantChannel, err)
}
if err := validateSystemPointer(&p, wantComponent, wantChannel); err != nil {
return nil, err
}
return &p, nil
}
// GetSystemPointer reads and verifies one channel's pointer. A missing key
// is os.ErrNotExist, which is the normal state of every deployment before
// its first `carlos system update`.
func GetSystemPointer(ctx context.Context, s Store, pub ed25519.PublicKey, component, channel string) (*SystemPointer, string, error) {
body, etag, err := s.Get(ctx, SystemPointerKey(component, channel))
if err != nil {
return nil, "", err
}
p, err := verifySystemPointer(pub, body, component, channel)
if err != nil {
return nil, "", err
}
return p, etag, nil
}
// SystemPromoteOptions: who did it, and how long boxes spread their starts.
type SystemPromoteOptions struct {
Actor string
Spread time.Duration // 0 = hotfix, every box at once; unset in the CLI means the default
}
// PromoteSystem writes the pointer for one channel, CAS-guarded. The
// outgoing release goes whole into History unless it is the same version
// (a re-promote of the running version is a no-op for boxes and must not
// push that version into its own history).
func PromoteSystem(ctx context.Context, s Store, sg Signer, component, channel string, rel SystemRelease, o SystemPromoteOptions) (*SystemPointer, error) {
key := SystemPointerKey(component, channel)
var old *SystemPointer
body, etag, err := s.Get(ctx, key)
switch {
case err == nil:
if old, err = verifySystemPointer(sg.verifyPub(), body, component, channel); err != nil {
return nil, err
}
case errors.Is(err, os.ErrNotExist):
etag = ""
default:
return nil, err
}
next := &SystemPointer{Component: component, Channel: channel, Current: rel, Spread: o.Spread.String()}
if rel.ReleasedAt.IsZero() {
next.Current.ReleasedAt = time.Now().UTC().Truncate(time.Second)
}
from := ""
if old != nil {
next.History, next.Audit = old.History, old.Audit
from = old.Current.Version
if old.Current.Version != rel.Version {
next.History = prependCapped(old.Current, old.History, historyMax)
}
}
next.Audit = prependCapped(Promotion{Version: rel.Version, From: from, To: channel, Actor: o.Actor, At: time.Now().UTC()}, next.Audit, auditMax)
wrapper, err := signSystemPointer(sg, next)
if err != nil {
return nil, err
}
if _, err := s.Put(ctx, key, wrapper, etag); err != nil {
return nil, err
}
return next, nil
}
// RollbackSystem swaps Current with History[0] and writes a NEW pointer —
// new PromotedAt — so every box's replay guard sees a promote, not a replay.
func RollbackSystem(ctx context.Context, s Store, sg Signer, component, channel, actor string) (*SystemPointer, error) {
key := SystemPointerKey(component, channel)
body, etag, err := s.Get(ctx, key)
if err != nil {
return nil, err
}
old, err := verifySystemPointer(sg.verifyPub(), body, component, channel)
if err != nil {
return nil, err
}
if len(old.History) == 0 {
return nil, ErrNoSystemHistory
}
next := &SystemPointer{
Component: component, Channel: channel,
Current: old.History[0], History: old.History[1:], Spread: old.Spread,
Audit: prependCapped(Promotion{Version: old.History[0].Version, From: old.Current.Version, To: channel, Actor: actor, At: time.Now().UTC()}, old.Audit, auditMax),
}
wrapper, err := signSystemPointer(sg, next)
if err != nil {
return nil, err
}
if _, err := s.Put(ctx, key, wrapper, etag); err != nil {
return nil, err
}
return next, nil
}Note sg.verifyPub() is the existing unexported method on Signer (signer.go:66); for a root signer it returns the root public key.
- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/release/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: ok, rc=0. If TestSystemPromoteCASConflict fails because conflictingStore lacks a method, the Store interface grew — embed satisfies it; check the error text.
- [ ] Step 5: gofmt and commit
gofmt -l internal/release/ && git add internal/release/system.go internal/release/system_test.go && git commit -q -F - <<'EOF'
release: the platform's own channel pointer, root-key-only
control/system/<component>/<channel>.json names a version and its
per-platform artifacts. History carries whole releases so a rollback
restores artifacts, and a rollback is a new promote (new PromotedAt).
A grant on the wire is refused before the signature is read, like the
production flag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 2: Streaming reads (Opener, FetchToFile)
Files:
- Modify:
internal/release/store.go(after theSizeListerblock, ~line 190) - Modify:
internal/fleetapi/fleetapi.go(afterGet, ~line 156) - Create:
internal/release/fetch.go - Test:
internal/release/fetch_test.go,internal/fleetapi/fleetapi_test.go(add one test)
Interfaces:
- Consumes:
Store,DirStore.path,S3Store.Client/Bucket,s3NotFound,prefixStore.full; fleetapistore.newRequest,objectURL. - Produces: ``
go // package release type Opener interface{ Open(ctx context.Context, key string) (io.ReadCloser, string, error) } func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error) var ErrTooLarge = errors.New("store: object larger than the caller's limit") func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) error`writesdst+".tmp", streams with a running sha, refuses more thanlimitbytes (ErrTooLarge), refuses a sha mismatch (error contains "sha256"), fsyncs, renames, fsyncs the directory, and removes the temp on any failure. Etag fromOpenmay be""` (DirStore).
- [ ] Step 1: Write the failing tests
internal/release/fetch_test.go:
package release
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
func shaOf(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
func TestFetchToFileStreamsAndVerifies(t *testing.T) {
ctx := context.Background()
body := []byte(strings.Repeat("carlos", 10000))
for name, s := range map[string]Store{
"dir": NewDirStore(t.TempDir()),
"prefix": NewPrefixStore(NewDirStore(t.TempDir()), "fleets/pi/"),
} {
t.Run(name, func(t *testing.T) {
if _, err := s.Put(ctx, "bootstrap/router/v1/carlos-linux-arm64", body, ""); err != nil {
t.Fatal(err)
}
dst := filepath.Join(t.TempDir(), "carlos")
if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst, int64(len(body)), shaOf(body)); err != nil {
t.Fatal(err)
}
got, _ := os.ReadFile(dst)
if string(got) != string(body) {
t.Fatal("content differs")
}
if _, err := os.Stat(dst + ".tmp"); !errors.Is(err, os.ErrNotExist) {
t.Fatal("temp file left behind")
}
// One byte over the limit is refused, and nothing is left on disk.
dst2 := filepath.Join(t.TempDir(), "carlos")
if err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst2, int64(len(body))-1, shaOf(body)); !errors.Is(err, ErrTooLarge) {
t.Fatalf("over-limit err = %v", err)
}
if _, err := os.Stat(dst2); !errors.Is(err, os.ErrNotExist) {
t.Fatal("over-limit fetch left a file")
}
// A sha mismatch is refused and leaves nothing.
dst3 := filepath.Join(t.TempDir(), "carlos")
err := FetchToFile(ctx, s, "bootstrap/router/v1/carlos-linux-arm64", dst3, int64(len(body)), strings.Repeat("0", 64))
if err == nil || !strings.Contains(err.Error(), "sha256") {
t.Fatalf("mismatch err = %v", err)
}
if _, err := os.Stat(dst3); !errors.Is(err, os.ErrNotExist) {
t.Fatal("mismatch fetch left a file")
}
// Missing key is ErrNotExist.
if err := FetchToFile(ctx, s, "bootstrap/router/v9/carlos-linux-arm64", dst3, 10, shaOf(nil)); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("missing err = %v", err)
}
})
}
}
func TestPrefixStoreOpenRebasesTheKey(t *testing.T) {
// prefixStore embeds the Store INTERFACE, so a promoted Open would forward
// the raw key and silently escape the prefix. It must be written out.
ctx := context.Background()
inner := NewDirStore(t.TempDir())
if _, err := inner.Put(ctx, "fleets/pi/x", []byte("in"), ""); err != nil {
t.Fatal(err)
}
if _, err := inner.Put(ctx, "x", []byte("OUT"), ""); err != nil {
t.Fatal(err)
}
rc, _, err := OpenObject(ctx, NewPrefixStore(inner, "fleets/pi/"), "x")
if err != nil {
t.Fatal(err)
}
defer rc.Close()
b := make([]byte, 8)
n, _ := rc.Read(b)
if string(b[:n]) != "in" {
t.Fatalf("read %q through the prefix, want \"in\"", b[:n])
}
}
func TestOpenObjectFallsBackToGet(t *testing.T) {
// A Store that is not an Opener still works: OpenObject buffers via Get.
s := getOnly{NewDirStore(t.TempDir())}
if _, err := s.Put(context.Background(), "k", []byte("v"), ""); err != nil {
t.Fatal(err)
}
rc, _, err := OpenObject(context.Background(), s, "k")
if err != nil {
t.Fatal(err)
}
b := make([]byte, 4)
n, _ := rc.Read(b)
if string(b[:n]) != "v" {
t.Fatalf("got %q", b[:n])
}
}
type getOnly struct{ *DirStore }(getOnly embeds *DirStore, so it has Open promoted — to make it a non-Opener, shadow it:)
func (getOnly) Open() {} // different signature: getOnly is NOT an Openerinternal/fleetapi/fleetapi_test.go, add:
func TestOpenStreamsThroughTheDoor(t *testing.T) {
d := newTestDoor(t)
body := bytes.Repeat([]byte("x"), 1<<20)
d.seed(t, "bootstrap/carlos", body)
c := d.box(t)
op, ok := c.(release.Opener)
if !ok {
t.Fatal("fleetapi store is not an Opener")
}
rc, _, err := op.Open(context.Background(), "bootstrap/carlos")
if err != nil {
t.Fatal(err)
}
defer rc.Close()
got, _ := io.ReadAll(rc)
if !bytes.Equal(got, body) {
t.Fatalf("read %d bytes, want %d", len(got), len(body))
}
if _, _, err := op.Open(context.Background(), "bootstrap/nothing"); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("missing: %v", err)
}
}(newTestDoor, seed, box exist in that file, fleetapi_test.go:52-105; bootstrap/carlos is already readable by a box.)
- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ -run 'Fetch|Open' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: undefined Opener, OpenObject, FetchToFile, ErrTooLarge.
- [ ] Step 3: Implement
internal/release/store.go, after the SizeLister assertions (~line 190):
// Opener is the optional streaming read. Store.Get returns the whole object
// in memory, which is fine for a pointer and wrong for a platform binary on
// a box under MemoryHigh=70%. Implemented by every production store; a test
// fake that only has Get still works through OpenObject. The etag may be ""
// (DirStore cannot know it without reading the file).
type Opener interface {
Open(ctx context.Context, key string) (io.ReadCloser, string, error)
}
var (
_ Opener = (*DirStore)(nil)
_ Opener = (*S3Store)(nil)
_ Opener = prefixStore{}
)
// OpenObject streams key from s, buffering through Get when s cannot stream.
func OpenObject(ctx context.Context, s Store, key string) (io.ReadCloser, string, error) {
if o, ok := s.(Opener); ok {
return o.Open(ctx, key)
}
b, etag, err := s.Get(ctx, key)
if err != nil {
return nil, "", err
}
return io.NopCloser(bytes.NewReader(b)), etag, nil
}DirStore:
func (d *DirStore) Open(_ context.Context, key string) (io.ReadCloser, string, error) {
p, err := d.path(key)
if err != nil {
return nil, "", err
}
f, err := os.Open(p)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, "", fmt.Errorf("dirstore: %s: %w", key, os.ErrNotExist)
}
return nil, "", err
}
return f, "", nil
}S3Store:
func (s *S3Store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
out, err := s.Client.GetObject(ctx, &awss3.GetObjectInput{Bucket: &s.Bucket, Key: &key})
if err != nil {
if s3NotFound(err) {
return nil, "", fmt.Errorf("s3store: %s: %w", key, os.ErrNotExist)
}
return nil, "", err
}
return out.Body, aws.ToString(out.ETag), nil
}prefixStore (explicit, next to its Get):
// Open is written out, not promoted: the embedded field is the interface,
// and a promoted Open would forward the UNREBASED key past the prefix.
func (p prefixStore) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
return OpenObject(ctx, p.Store, p.full(key))
}Add "bytes" and "io" to the imports if absent.
internal/fleetapi/fleetapi.go, after Get:
// Open streams one object. Unlike Get it does not buffer the body, which
// is what a 25 MiB platform binary on a Pi needs; the 404 mapping is the
// same as Get's so callers can errors.Is(err, os.ErrNotExist).
func (s *store) Open(ctx context.Context, key string) (io.ReadCloser, string, error) {
req, err := s.newRequest(ctx, http.MethodGet, s.objectURL(key), nil)
if err != nil {
return nil, "", err
}
resp, err := s.client.Do(req)
if err != nil {
return nil, "", errorf("open %s: %w", key, err)
}
switch resp.StatusCode {
case http.StatusOK:
return resp.Body, resp.Header.Get("ETag"), nil
case http.StatusNotFound:
resp.Body.Close()
return nil, "", fmt.Errorf("fleetapi: %s: %w", key, os.ErrNotExist)
default:
resp.Body.Close()
return nil, "", errorf("open %s: unexpected status %d", key, resp.StatusCode)
}
}
var _ release.Opener = (*store)(nil)internal/release/fetch.go:
package release
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"os"
"path/filepath"
)
// ErrTooLarge: the object exceeded the caller's limit. The reader is cut at
// limit+1 bytes, so an object EXACTLY at the limit is told from one over it.
var ErrTooLarge = errors.New("store: object larger than the caller's limit")
// FetchToFile streams key into dst with a running sha256, never holding the
// object in memory: temp file, fsync, rename, directory fsync. Any failure
// removes the temp and leaves dst untouched.
func FetchToFile(ctx context.Context, s Store, key, dst string, limit int64, wantSHA256 string) (err error) {
rc, _, err := OpenObject(ctx, s, key)
if err != nil {
return err
}
defer rc.Close()
tmp := dst + ".tmp"
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return err
}
defer func() {
if err != nil {
f.Close()
os.Remove(tmp)
}
}()
h := sha256.New()
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, limit+1))
if err != nil {
return fmt.Errorf("fetch %s: %w", key, err)
}
if n > limit {
return fmt.Errorf("fetch %s: %w (limit %d)", key, ErrTooLarge, limit)
}
if got := hex.EncodeToString(h.Sum(nil)); got != wantSHA256 {
return fmt.Errorf("fetch %s: sha256 %s does not match the signed %s", key, got, wantSHA256)
}
if err = f.Sync(); err != nil {
return err
}
if err = f.Close(); err != nil {
return err
}
if err = os.Rename(tmp, dst); err != nil {
return err
}
return syncDir(filepath.Dir(dst))
}
// syncDir makes a rename durable: the directory entry is what fsync(file)
// does not cover.
func syncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/release/ ./internal/fleetapi/ ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: all ok. (internal/edge is included because its test fakes assign to release.Store; the interface did not change, so they still compile.)
- [ ] Step 5: Commit
gofmt -l internal/ && git add internal/release/store.go internal/release/fetch.go internal/release/fetch_test.go internal/fleetapi/fleetapi.go internal/fleetapi/fleetapi_test.go && git commit -q -F - <<'EOF'
release: stream an object to a file with a running sha
Opener is a side-interface (48 test fakes implement Store; none break),
implemented by DirStore, S3Store, prefixStore and the fleet door client.
FetchToFile is what a box uses for a platform binary: limit+1 read, sha
checked before rename, directory fsync after.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 3: Store-door grants for the pointer and artifact shapes
Files:
- Modify:
internal/edge/storedoor.go:383-397(aclRead) and the doc comment above it (:352-382) - Modify:
internal/edge/storedoor_test.go(TestStoreDoorACLMatrixrows at:69+,TestStoreDoorGetServesOwnEstateat:365, the prose inTestDoneNeverServesCertMaterial's comment at:519)
Interfaces:
- Consumes:
aclRead(sqid, fleet, key string) bool;canonicalKeyalready refuses any.././empty segment before the ACL runs, so traversal never reaches here. - Produces:
func systemReadKey(key string) bool(unexported) — true for exactlycontrol/system/<c>/<ch>.jsonandbootstrap/router/<v>/<asset>.
- [ ] Step 1: Add the failing matrix rows and GET cases
In TestStoreDoorACLMatrix's table, after the bootstrap/carlos.sig row (:84):
{"control/system/router/stable.json", opRead, true, "the platform channel pointer (system update spec §3)"},
{"control/system/router/canary.json", opRead, true, "…and the canary one"},
{"control/system/router/stable.json", opWrite, false, "a box never writes a pointer"},
{"control/system/router", opRead, false, "the pointer shape is exactly three segments ending .json"},
{"control/system/router/stable.json/x", opRead, false, "…not four"},
{"control/system/router/stable.txt", opRead, false, "…and .json"},
{"bootstrap/router/v0.18.0/carlos-linux-arm64", opRead, true, "a mirrored platform artifact"},
{"bootstrap/router/v0.18.0/carlos-linux-arm64", opWrite, false, "a box must never publish the platform binary"},
{"bootstrap/router/v0.18.0", opRead, false, "the artifact shape is exactly three segments"},
{"bootstrap/router/v0.18.0/a/b", opRead, false, "…not four"},
{"bootstrap/router.sig", opRead, false, "nothing else under bootstrap/"},
{"bootstrap/router/../../x", opRead, false, "traversal is refused by canonicalKey before the ACL"},
{"control/system/../serial", opRead, false, "…on both prefixes"},In TestStoreDoorGetServesOwnEstate, seed two more objects with seedObject and add their keys to the loop that expects 200:
seedObject(t, store, "control/system/router/stable.json", []byte(`{"payload":"x"}`))
seedObject(t, store, "bootstrap/router/v1/carlos-linux-arm64", []byte("ELF"))Update the comment at storedoor_test.go:519 (the allowlist prose) to add "control/system/<c>/<ch>.json, bootstrap/router/<v>/<asset>".
- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -run 'TestStoreDoorACLMatrix|TestStoreDoorGetServesOwnEstate' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: FAIL on the four want true rows and the two new GET keys (404).
- [ ] Step 3: Implement
In storedoor.go, replace the switch key in aclRead with:
switch key {
case "control/serial", "bootstrap/carlos":
return true
}
if systemReadKey(key) {
return true
}and add, below aclRead:
// systemReadKey is the two shapes the system-update spec (§3) grants every
// fleet box: its platform channel pointer and a mirrored platform artifact.
// Shapes, not prefixes — exactly three segments each, so nothing else that
// ever lands under control/ or bootstrap/ is readable by a box. Listing is
// not granted (aclDir is unchanged), and canonicalKey has already refused
// any "." or ".." segment before this runs.
func systemReadKey(key string) bool {
if rest, ok := strings.CutPrefix(key, "control/system/"); ok {
comp, file, ok := strings.Cut(rest, "/")
return ok && comp != "" && file != "" && !strings.Contains(file, "/") &&
strings.HasSuffix(file, ".json") && len(file) > len(".json")
}
if rest, ok := strings.CutPrefix(key, "bootstrap/router/"); ok {
ver, asset, ok := strings.Cut(rest, "/")
return ok && ver != "" && asset != "" && !strings.Contains(asset, "/")
}
return false
}Extend the aclRead doc comment's allowlist enumeration (:352-382) with the two shapes and the sentence: "bootstrap/carlos stays until update-platform-binary.sh is retired (spec §10)."
- [ ] Step 4: Run the edge tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/edge/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: ok.
- [ ] Step 5: Commit
gofmt -l internal/edge/ && git add internal/edge/storedoor.go internal/edge/storedoor_test.go && git commit -q -F - <<'EOF'
edge: the store door grants a box its platform pointer and artifact
Two exact shapes under control/system/ and bootstrap/router/, no
listing, no writes. Without the pointer grant a door-backed fleet box
would never see a platform promote.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 4: On-box state files (internal/host/systemstate.go)
Files:
- Create:
internal/host/systemstate.go - Test:
internal/host/systemstate_test.go
Interfaces:
- Consumes: nothing new.
- Produces (package
host, unexported unless noted): ``go const ( systemComponent = "router" systemRequestMarker = "system-update-requested" // touched under dataDir systemRequestEvery = time.Minute systemArtifactLimit = int64(96 << 20) systemOutcomeApplied = "applied"; systemOutcomeRolledBack = "rolled-back"; systemOutcomeRollbackFailed = "rollback-failed" systemOutcomeRefused = "refused"; systemOutcomeDeferred = "deferred"; systemOutcomeNoted = "noted" systemPhaseBackingUp = "backing-up"; systemPhaseInstalling = "installing"; systemPhaseInstalled = "installed" systemPhaseUnits = "units"; systemPhaseRestarted = "restarted"; systemPhaseGated = "gated" ) type systemRequest struct{ Version string; PromotedAt time.Time; NoOp bool; SeenAt time.Time } type systemResult struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time } type systemApplied struct{ Version, Channel string; At time.Time } type systemSeen map[string]time.Time type systemJournal struct{ Phase, Target, PrevBuild string; UnitsAdded, UnitsRemoved []string; StartedAt time.Time } func systemDataDir(dd string) string // <dd>/system/router func systemRequestPath(dd string) string // <dd>/system/router/requested.json func systemCachePath(dd, version string) string // <dd>/system/router/<version>/carlos func systemRootDir(optDir string) string // <optDir>/system/router func systemPrevPath(optDir string) string // <optDir>/carlos.prev func systemBinaryPath(optDir string) string // <optDir>/carlos func writeFileDurable(path string, data []byte, mode os.FileMode) error func writeJSONDurable(path string, v any, mode os.FileMode) error func readJSONFile(path string, v any) error // os.ErrNotExist passthrough func readSystemResult(optDir string) (*systemResult, error) func readSystemSeen(optDir string) (systemSeen, error) // missing file = empty map, nil error func readSystemApplied(optDir string) (*systemApplied, error) func readSystemJournal(optDir string) (*systemJournal, error) func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool func (r *systemResult) terminal() bool // applied|rolled-back|rollback-failed|refused func validSystemVersion(v string) bool // ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`The root files areresult.json,seen.json,applied.json,journal.jsondirectly undersystemRootDir`.
- [ ] Step 1: Write the failing tests
package host
import (
"errors"
"os"
"path/filepath"
"testing"
"time"
)
func TestSystemPathsAreDerivedNotTrusted(t *testing.T) {
if got := systemCachePath("/data", "v1.2.3"); got != "/data/system/router/v1.2.3/carlos" {
t.Fatalf("cache path = %s", got)
}
if got := systemRequestPath("/data"); got != "/data/system/router/requested.json" {
t.Fatalf("request path = %s", got)
}
if got := systemRootDir("/opt/carlos"); got != "/opt/carlos/system/router" {
t.Fatalf("root dir = %s", got)
}
for _, bad := range []string{"", ".", "..", "../x", "a/b", ".hidden", "v1 2", string(make([]byte, 65))} {
if validSystemVersion(bad) {
t.Errorf("validSystemVersion(%q) = true", bad)
}
}
for _, good := range []string{"v0.18.0", "66587fa", "a", "release-2026.09.06_rc1"} {
if !validSystemVersion(good) {
t.Errorf("validSystemVersion(%q) = false", good)
}
}
}
func TestWriteFileDurableLeavesWholeFileOrNothing(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "journal.json")
if err := writeFileDurable(p, []byte("one"), 0o644); err != nil {
t.Fatal(err)
}
if err := writeFileDurable(p, []byte("two"), 0o644); err != nil {
t.Fatal(err)
}
b, _ := os.ReadFile(p)
if string(b) != "two" {
t.Fatalf("content = %q", b)
}
if _, err := os.Stat(p + ".tmp"); !errors.Is(err, os.ErrNotExist) {
t.Fatal("temp left behind")
}
st, _ := os.Stat(p)
if st.Mode().Perm() != 0o644 {
t.Fatalf("mode = %v", st.Mode().Perm())
}
// A directory that does not exist is an error, not a silent no-op.
if err := writeFileDurable(filepath.Join(dir, "no", "such", "f"), []byte("x"), 0o644); err == nil {
t.Fatal("wrote into a missing directory")
}
}
func TestSystemStateReadersRoundTrip(t *testing.T) {
opt := t.TempDir()
root := systemRootDir(opt)
if err := os.MkdirAll(root, 0o755); err != nil {
t.Fatal(err)
}
if r, err := readSystemResult(opt); err != nil || r != nil {
t.Fatalf("missing result: %v %v", r, err)
}
seen, err := readSystemSeen(opt)
if err != nil || len(seen) != 0 {
t.Fatalf("missing seen: %v %v", seen, err)
}
at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
res := systemResult{Channel: "stable", Version: "v1", PromotedAt: at, Outcome: systemOutcomeRolledBack, Reason: "gate", At: at}
if err := writeJSONDurable(filepath.Join(root, "result.json"), res, 0o644); err != nil {
t.Fatal(err)
}
got, err := readSystemResult(opt)
if err != nil || !got.matches("stable", "v1", at) || !got.terminal() {
t.Fatalf("result = %+v err=%v", got, err)
}
if got.matches("canary", "v1", at) || got.matches("stable", "v2", at) || got.matches("stable", "v1", at.Add(time.Second)) {
t.Fatal("matches ignores part of the tuple")
}
got.Outcome = systemOutcomeDeferred
if got.terminal() {
t.Fatal("deferred is not terminal")
}
got.Outcome = systemOutcomeNoted
if got.terminal() {
t.Fatal("noted is not terminal for the leg: the version may still differ next time")
}
seen["stable"] = at
if err := writeJSONDurable(filepath.Join(root, "seen.json"), seen, 0o644); err != nil {
t.Fatal(err)
}
seen2, _ := readSystemSeen(opt)
if !seen2["stable"].Equal(at) {
t.Fatalf("seen = %v", seen2)
}
j := systemJournal{Phase: systemPhaseInstalled, Target: "v1", PrevBuild: "v0", UnitsAdded: []string{"a.path"}, StartedAt: at}
if err := writeJSONDurable(filepath.Join(root, "journal.json"), j, 0o600); err != nil {
t.Fatal(err)
}
j2, err := readSystemJournal(opt)
if err != nil || j2.Phase != systemPhaseInstalled || j2.UnitsAdded[0] != "a.path" {
t.Fatalf("journal = %+v err=%v", j2, err)
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: undefined identifiers.
- [ ] Step 3: Implement
internal/host/systemstate.go
package host
import (
"encoding/json"
"errors"
"os"
"path/filepath"
"regexp"
"time"
)
// The on-box state of the platform self-update (spec §4, §6). Two homes:
//
// <data>/system/router/ — written by the sandboxed agent: the
// request and the artifact cache. Root
// reads exactly one file here and trusts
// nothing in it beyond a version string.
// <opt>/system/router/ — written by root: journal, seen (replay
// watermarks), applied (the .prev
// authority), result (what the agent
// publishes). Root-owned, world-readable.
const (
systemComponent = "router"
systemRequestMarker = "system-update-requested"
systemRequestEvery = time.Minute
systemArtifactLimit = int64(96 << 20)
systemOutcomeApplied = "applied"
systemOutcomeRolledBack = "rolled-back"
systemOutcomeRollbackFailed = "rollback-failed"
systemOutcomeRefused = "refused"
systemOutcomeDeferred = "deferred"
systemOutcomeNoted = "noted"
systemPhaseBackingUp = "backing-up"
systemPhaseInstalling = "installing"
systemPhaseInstalled = "installed"
systemPhaseUnits = "units"
systemPhaseRestarted = "restarted"
systemPhaseGated = "gated"
)
// systemRequest is the agent's hint to root: which promote it saw. No
// channel (root's own host.env says), no key, no sha, no path (root derives
// the cache path from the version and takes key and sha from the pointer).
type systemRequest struct {
Version string `json:"version"`
PromotedAt time.Time `json:"promoted_at"`
NoOp bool `json:"noop"`
SeenAt time.Time `json:"seen_at"`
}
// systemResult is root's answer to the last request, keyed by the REQUEST's
// tuple so a refusal of one promote never suppresses the next.
type systemResult struct {
Channel string `json:"channel"`
Version string `json:"version"`
PromotedAt time.Time `json:"promoted_at"`
Outcome string `json:"outcome"`
Reason string `json:"reason,omitempty"`
At time.Time `json:"at"`
}
func (r *systemResult) matches(channel, version string, promotedAt time.Time) bool {
return r != nil && r.Channel == channel && r.Version == version && r.PromotedAt.Equal(promotedAt)
}
// terminal: outcomes the leg must not retry for the same pointer.
func (r *systemResult) terminal() bool {
switch r.Outcome {
case systemOutcomeApplied, systemOutcomeRolledBack, systemOutcomeRollbackFailed, systemOutcomeRefused:
return true
}
return false
}
// systemApplied is the last build that PASSED A GATE on this box, forward or
// rollback. It is the only thing that may authorise refreshing carlos.prev.
type systemApplied struct {
Version string `json:"version"`
Channel string `json:"channel"`
At time.Time `json:"at"`
}
// systemSeen is the replay watermark per channel: the newest PromotedAt root
// has verified. Advanced on every verified pointer, including no-ops.
type systemSeen map[string]time.Time
// systemJournal is the open transaction. Written before every box-changing
// step, removed last; a run that finds one recovers before anything else.
type systemJournal struct {
Phase string `json:"phase"`
Target string `json:"target"`
PrevBuild string `json:"prev_build"`
UnitsAdded []string `json:"units_added,omitempty"`
UnitsRemoved []string `json:"units_removed,omitempty"`
StartedAt time.Time `json:"started_at"`
}
var systemVersionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`)
// validSystemVersion is what makes the cache path derivable: no leading dot,
// no separator, nothing that could be a traversal.
func validSystemVersion(v string) bool { return systemVersionRe.MatchString(v) }
func systemDataDir(dd string) string { return filepath.Join(dd, "system", systemComponent) }
func systemRequestPath(dd string) string {
return filepath.Join(systemDataDir(dd), "requested.json")
}
func systemCachePath(dd, version string) string {
return filepath.Join(systemDataDir(dd), version, "carlos")
}
func systemRootDir(optDir string) string { return filepath.Join(optDir, "system", systemComponent) }
func systemPrevPath(optDir string) string { return filepath.Join(optDir, "carlos.prev") }
func systemBinaryPath(optDir string) string { return filepath.Join(optDir, "carlos") }
// writeFileDurable: temp, fsync, rename, directory fsync. A crash leaves the
// old file or the new one, never a torn one, and the rename itself is on
// disk before the caller's next step depends on it.
func writeFileDurable(path string, data []byte, mode os.FileMode) (err error) {
tmp := path + ".tmp"
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode)
if err != nil {
return err
}
defer func() {
if err != nil {
f.Close()
os.Remove(tmp)
}
}()
if _, err = f.Write(data); err != nil {
return err
}
if err = f.Chmod(mode); err != nil {
return err
}
if err = f.Sync(); err != nil {
return err
}
if err = f.Close(); err != nil {
return err
}
if err = os.Rename(tmp, path); err != nil {
return err
}
d, err := os.Open(filepath.Dir(path))
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
func writeJSONDurable(path string, v any, mode os.FileMode) error {
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
return err
}
return writeFileDurable(path, append(b, '\n'), mode)
}
// readJSONFile leaves os.ErrNotExist intact so callers can treat "no file"
// as a state, not a failure.
func readJSONFile(path string, v any) error {
b, err := os.ReadFile(path)
if err != nil {
return err
}
return json.Unmarshal(b, v)
}
func readSystemResult(optDir string) (*systemResult, error) {
var r systemResult
err := readJSONFile(filepath.Join(systemRootDir(optDir), "result.json"), &r)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return &r, err
}
func readSystemSeen(optDir string) (systemSeen, error) {
s := systemSeen{}
err := readJSONFile(filepath.Join(systemRootDir(optDir), "seen.json"), &s)
if errors.Is(err, os.ErrNotExist) {
return s, nil
}
return s, err
}
func readSystemApplied(optDir string) (*systemApplied, error) {
var a systemApplied
err := readJSONFile(filepath.Join(systemRootDir(optDir), "applied.json"), &a)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return &a, err
}
func readSystemJournal(optDir string) (*systemJournal, error) {
var j systemJournal
err := readJSONFile(filepath.Join(systemRootDir(optDir), "journal.json"), &j)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return &j, err
}- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemPaths|TestWriteFileDurable|TestSystemStateReaders' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: PASS.
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemstate.go internal/host/systemstate_test.go && git commit -q -F - <<'EOF'
host: the on-box state of a platform self-update
Request (agent-written hint), result, seen watermarks, applied, and
the journal, with durable writes: temp, fsync, rename, directory
fsync. Cache paths are derived from a validated version string.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 5: The heartbeat learns platform, channel and last apply outcome
Files:
- Modify:
internal/status/snapshot.go—Probes(~:98, next toRelease),BoxMetrics(~:158),CollectBox(~:404, whereRelease: p.Releaseis copied), the emptiness guard (~:543) - Modify:
internal/host/agent.gostatusLoop(~:2318, theRelease: version.Build,line) - Create:
internal/host/systemprobe.go - Test:
internal/status/snapshot_test.go(add),internal/host/systemprobe_test.go
Interfaces:
- Consumes:
readSystemResult(Task 4),release.OptDir(),status.Probes,status.BoxMetrics. - Produces: ``
go // package status type SystemState struct{ Channel, Version string; PromotedAt time.Time; Outcome, Reason string; At time.Time } // Probes gains: Platform string; SystemChannel string; System *SystemState // BoxMetrics gains (json): platform, system_channel, system // package host func systemChannel(getenv func(string) string) string // CARLOS_SYSTEM_CHANNEL, default "stable" func systemStateProbe(optDir string) *status.SystemState // nil when no result.json const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL"``
- [ ] Step 1: Write the failing tests
internal/status/snapshot_test.go, add:
func TestBoxMetricsCarryThePlatformSelfUpdateState(t *testing.T) {
at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
p := Probes{Release: "v1", Platform: "linux-arm64", SystemChannel: "canary",
System: &SystemState{Channel: "canary", Version: "v2", PromotedAt: at, Outcome: "rolled-back", Reason: "gate", At: at}}
b := CollectBox(nil, p, at)
if b.Metrics == nil || b.Metrics.Platform != "linux-arm64" || b.Metrics.SystemChannel != "canary" || b.Metrics.System == nil || b.Metrics.System.Outcome != "rolled-back" {
t.Fatalf("metrics = %+v", b.Metrics)
}
out, _ := json.Marshal(b)
for _, want := range []string{`"platform":"linux-arm64"`, `"system_channel":"canary"`, `"outcome":"rolled-back"`} {
if !strings.Contains(string(out), want) {
t.Errorf("json lacks %s: %s", want, out)
}
}
// A box with only the new fields is not "empty".
b2 := CollectBox(nil, Probes{Platform: "linux-arm64"}, at)
if b2.Metrics == nil {
t.Fatal("platform alone must produce a metrics block")
}
// The public fleet feed still never carries the metrics block.
f := BuildFleet([]Box{b}, at)
fb, _ := json.Marshal(f)
if strings.Contains(string(fb), "system_channel") || strings.Contains(string(fb), "rolled-back") {
t.Fatalf("fleet.json leaked the private metrics: %s", fb)
}
}(Check BuildFleet's exact signature at fleet.go:157 and adjust the call; it takes the box list and a time.)
internal/host/systemprobe_test.go:
package host
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestSystemChannelDefaultsToStable(t *testing.T) {
if got := systemChannel(func(string) string { return "" }); got != "stable" {
t.Fatalf("default = %q", got)
}
if got := systemChannel(func(k string) string {
if k == systemChannelEnv {
return " canary "
}
return ""
}); got != "canary" {
t.Fatalf("env = %q", got)
}
}
func TestSystemStateProbeReadsRootsResult(t *testing.T) {
opt := t.TempDir()
if got := systemStateProbe(opt); got != nil {
t.Fatalf("no result.json → %+v, want nil", got)
}
root := systemRootDir(opt)
os.MkdirAll(root, 0o755)
at := time.Date(2026, 9, 6, 9, 0, 0, 0, time.UTC)
writeJSONDurable(filepath.Join(root, "result.json"), systemResult{Channel: "stable", Version: "v3", PromotedAt: at, Outcome: "applied", At: at}, 0o644)
got := systemStateProbe(opt)
if got == nil || got.Version != "v3" || got.Outcome != "applied" || !got.PromotedAt.Equal(at) {
t.Fatalf("probe = %+v", got)
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -run 'TestBoxMetricsCarry|TestSystemChannel|TestSystemStateProbe' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: undefined fields/functions.
- [ ] Step 3: Implement
internal/status/snapshot.go — in Probes, after Release string (:98):
// Platform is runtime.GOOS-runtime.GOARCH, so the console knows which
// artifacts a platform promote must mirror. SystemChannel is the box's
// CARLOS_SYSTEM_CHANNEL, so the console knows which boxes a promote
// reaches. System is root's last self-update outcome (spec §5, §6).
Platform string
SystemChannel string
System *SystemStateAdd the type near BoxMetrics:
// SystemState is the last platform self-update outcome root recorded on the
// box, published so `carlos system status` can show a rolled-back box with
// its reason within one heartbeat.
type SystemState struct {
Channel string `json:"channel"`
Version string `json:"version"`
PromotedAt time.Time `json:"promoted_at"`
Outcome string `json:"outcome"`
Reason string `json:"reason,omitempty"`
At time.Time `json:"at"`
}In BoxMetrics, after Release (:158):
Platform string `json:"platform,omitempty"` // runtime.GOOS-runtime.GOARCH
SystemChannel string `json:"system_channel,omitempty"` // CARLOS_SYSTEM_CHANNEL
System *SystemState `json:"system,omitempty"` // last self-update outcomeIn CollectBox, where Release: p.Release is copied into the metrics block, add Platform: p.Platform, SystemChannel: p.SystemChannel, System: p.System. In the emptiness guard (:543), add || p.Platform != "" || p.SystemChannel != "" || p.System != nil.
internal/host/systemprobe.go:
package host
import (
"strings"
"github.com/carlosframework/platform/internal/status"
)
// systemChannelEnv is the box's platform channel: canary or stable. Set in
// host.env, read through the getenv seam like every other CARLOS_* value;
// the ROOT apply reads it from its own unit environment, never from the
// agent's request (spec §6 step 2).
const systemChannelEnv = "CARLOS_SYSTEM_CHANNEL"
func systemChannel(getenv func(string) string) string {
if c := strings.TrimSpace(getenv(systemChannelEnv)); c != "" {
return c
}
return "stable"
}
// systemStateProbe is root's result.json as the heartbeat carries it, or nil
// when this box has never been asked to update.
func systemStateProbe(optDir string) *status.SystemState {
r, err := readSystemResult(optDir)
if err != nil || r == nil {
return nil
}
return &status.SystemState{Channel: r.Channel, Version: r.Version, PromotedAt: r.PromotedAt, Outcome: r.Outcome, Reason: r.Reason, At: r.At}
}internal/host/agent.go statusLoop, next to Release: version.Build, (:2318):
Platform: runtime.GOOS + "-" + runtime.GOARCH,
SystemChannel: systemChannel(os.Getenv),
System: systemStateProbe(release.OptDir()),(add "runtime" to the imports). Add CARLOS_SYSTEM_CHANNEL to the env contract in the package doc comment at the top of agent.go, one line: CARLOS_SYSTEM_CHANNEL — platform channel this box follows for self-update, canary|stable (default stable).
- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/status/ ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: ok for both.
- [ ] Step 5: Commit
gofmt -l internal/ && git add internal/status/snapshot.go internal/status/snapshot_test.go internal/host/agent.go internal/host/systemprobe.go internal/host/systemprobe_test.go && git commit -q -F - <<'EOF'
status: a box reports its platform, its system channel and its last self-update outcome
Three optional fields on the private metrics block; the public fleet
feed does not carry them, as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 6: The three units, enabledUnits, -plan-json, the path probe
Files:
- Create:
internal/host/units/carlos-system-update.path,internal/host/units/carlos-system-update.service,internal/host/units/carlos-system-recover.service - Modify:
internal/host/bootstrap.go(enabledUnits:72,RunBootstrapflags:115-132, the staging print block:327-341) - Modify:
internal/host/probes.go(add a fourth*Active()) - Modify:
internal/host/host_test.go(expectedUnits:29-46;TestProbedUnitsAreTheOnesBootstrapArms:1129) - Test:
internal/host/host_test.go(addTestBootstrapPlanJSON)
Interfaces:
- Consumes:
unitsFS,unitNames,systemctlPlan,bootstrap(root, consumers),parseIsActive. - Produces: ``
go // package host var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"} type bootstrapPlan struct{ Enabled []stringjson:"enabled"; Inert []stringjson:"inert"} func currentBootstrapPlan() bootstrapPlan //carlos bootstrap -root <dir> -plan-jsonprints ONLY the JSON plan on stdout const systemUpdatePathUnit = "carlos-system-update.path" func systemUpdatePathActive() *bool``
- [ ] Step 1: Add the failing tests
In host_test.go, add the three names to expectedUnits (keep it alphabetical):
"carlos-system-recover.service",
"carlos-system-update.path",
"carlos-system-update.service",and add:
func TestBootstrapPlanJSONNamesWhatALiveRunArms(t *testing.T) {
p := currentBootstrapPlan()
want := map[string]bool{"carlos-edge.service": true, "carlos-adopt.timer": true, "carlos-adopt.path": true,
"carlos-stage.path": true, "carlos-restart.path": true, "carlos-system-update.path": true, "carlos-system-recover.service": true}
got := map[string]bool{}
for _, u := range p.Enabled {
got[u] = true
}
for u := range want {
if !got[u] {
t.Errorf("plan lacks %s", u)
}
}
if got["carlos-system-update.service"] {
t.Error("the path-triggered oneshot must not be enabled: it is started by its .path")
}
inert := map[string]bool{}
for _, u := range p.Inert {
inert[u] = true
}
if !inert["carlos-restore-verify.timer"] || !inert["carlos-hibernate-verify.timer"] {
t.Errorf("inert = %v", p.Inert)
}
// And systemctlPlan never starts a service, whatever is enabled.
for _, args := range systemctlPlan(p.Enabled) {
if args[0] == "start" {
for _, u := range args[1:] {
if strings.HasSuffix(u, ".service") {
t.Errorf("plan starts a service: %v", args)
}
}
}
}
}
func TestSystemUnitsAreShapedLikeTheStageRelay(t *testing.T) {
read := func(name string) string {
b, err := unitsFS.ReadFile("units/" + name)
if err != nil {
t.Fatal(err)
}
return string(b)
}
p := read("carlos-system-update.path")
for _, want := range []string{"PathChanged=/data/system-update-requested", "Unit=carlos-system-update.service", "WantedBy=multi-user.target"} {
if !strings.Contains(p, want) {
t.Errorf("path unit lacks %q", want)
}
}
u := read("carlos-system-update.service")
for _, want := range []string{"Type=oneshot", "StartLimitIntervalSec=0", "TimeoutStartSec=15min", "OnFailure=carlos-system-recover.service",
"EnvironmentFile=-/etc/carlos/host.env", "system apply", "carlos.prev", "MemoryMax=256M"} {
if !strings.Contains(u, want) {
t.Errorf("update unit lacks %q", want)
}
}
for _, never := range []string{"ProtectSystem", "User="} {
if strings.Contains(u, never) {
t.Errorf("update unit must run as root with /opt writable; found %q", never)
}
}
r := read("carlos-system-recover.service")
for _, want := range []string{"Type=oneshot", "Before=carlos-edge.service", "WantedBy=multi-user.target", "TimeoutStartSec=10min", "system apply --recover", "carlos.prev"} {
if !strings.Contains(r, want) {
t.Errorf("recover unit lacks %q", want)
}
}
}Update TestProbedUnitsAreTheOnesBootstrapArms so its probed set includes systemUpdatePathUnit (read the test; it compares probe constants against enabledUnits).
- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestBootstrap|TestSystemUnits|TestProbedUnits' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: FAIL (missing units, undefined currentBootstrapPlan).
- [ ] Step 3: Write the units
internal/host/units/carlos-system-update.path:
# The relay from the sandboxed agent to a root apply of a new platform
# binary (spec §4). The agent (User=carlos, ProtectSystem=strict) can fetch
# and verify the artifact but cannot write /opt or /etc; it touches this
# marker and root does the rest. Same shape as carlos-stage.path.
[Unit]
Description=CARLOS platform self-update on agent request
[Path]
PathChanged=/data/system-update-requested
Unit=carlos-system-update.service
[Install]
WantedBy=multi-user.targetinternal/host/units/carlos-system-update.service:
# Root apply of a platform self-update (spec §6): re-verify the pointer,
# copy the candidate into a root-owned dir, journal, swap the binary and
# units, restart the edge, gate it, roll back to carlos.prev on failure.
#
# ExecStart goes through one line of shell so that when a journal is open
# (a previous apply was cut off) and carlos.prev exists, the RECOVERY runs
# from carlos.prev rather than from the candidate that did not work.
#
# No ProtectSystem, no User=: it must write /opt/carlos and
# /etc/systemd/system. StartLimitIntervalSec=0 because the path unit can
# fire in bursts. TimeoutStartSec bounds a hung apply; OnFailure= makes a
# killed or timed-out apply recover without waiting for a touch.
[Unit]
Description=carlos platform self-update (root apply)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0
OnFailure=carlos-system-recover.service
[Service]
Type=oneshot
TimeoutStartSec=15min
EnvironmentFile=-/etc/carlos/host.env
ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply; fi; exec /opt/carlos/carlos system apply'
# --- hardening (the carlos-stage.service set) ---
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectClock=yes
ProtectHostname=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
MemoryMax=256M
TasksMax=64internal/host/units/carlos-system-recover.service:
# Recovery of an interrupted platform self-update (spec §6, "every run
# begins with recovery"). Runs once per boot, BEFORE the edge, and on the
# update oneshot's failure. It restores files only — carlos.prev over
# carlos, the previous unit set — and never restarts or gates the edge
# from inside a unit the edge is ordered after: at boot the edge's own
# start IS the restart. At runtime (edge already started this boot) it
# issues a non-blocking restart and exits.
[Unit]
Description=carlos platform self-update recovery
Before=carlos-edge.service
DefaultDependencies=yes
[Service]
Type=oneshot
TimeoutStartSec=10min
EnvironmentFile=-/etc/carlos/host.env
ExecStart=/bin/sh -c 'if [ -f /opt/carlos/system/router/journal.json ] && [ -x /opt/carlos/carlos.prev ]; then exec /opt/carlos/carlos.prev system apply --recover; fi; exec /opt/carlos/carlos system apply --recover'
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectClock=yes
ProtectHostname=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
MemoryMax=256M
TasksMax=64
[Install]
WantedBy=multi-user.target- [ ] Step 4:
bootstrap.gochanges
Replace enabledUnits (:72):
var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path",
"carlos-system-update.path", "carlos-system-recover.service"}
// inertUnits are installed everywhere and deliberately never armed by
// bootstrap or by a self-update apply: restore-verify until an operator
// stages litestream, hibernate-verify by long-standing choice.
var inertUnits = []string{"carlos-restore-verify.timer", "carlos-hibernate-verify.timer"}
// bootstrapPlan is what `bootstrap -root <dir> -plan-json` prints: the
// arming list of THIS binary, so a self-update apply arms what the new
// binary's bootstrap would arm and never a list an older binary remembers.
type bootstrapPlan struct {
Enabled []string `json:"enabled"`
Inert []string `json:"inert"`
}
func currentBootstrapPlan() bootstrapPlan {
return bootstrapPlan{Enabled: append([]string(nil), enabledUnits...), Inert: append([]string(nil), inertUnits...)}
}In RunBootstrap, add the flag after offcloud:
planJSON := fs.Bool("plan-json", false, "with -root: print the arming plan as JSON on stdout instead of the human summary")and pass it through: change bootstrap(*root, consumers) to bootstrapWith(*root, consumers, *planJSON), where bootstrap(root, consumers) becomes func bootstrap(root string, consumers []string) error { return bootstrapWith(root, consumers, false) } and bootstrapWith is the old body with the staging print block (:327-341) wrapped:
if !live {
if planJSON {
return json.NewEncoder(os.Stdout).Encode(currentBootstrapPlan())
}
fmt.Printf("bootstrap: staged under %s — on a real box (--root /) it would also:\n", root)
…existing prints…
return nil
}All the fmt.Printf(" wrote %s\n", path) / " created %s\n" lines in the staging path must go to stderr when planJSON is set, or the JSON is not alone on stdout: introduce out := io.Writer(os.Stdout); if planJSON { out = os.Stderr } at the top of bootstrapWith and change those fmt.Printf to fmt.Fprintf(out, …). (-plan-json with -root / is refused: log.Fatal("bootstrap: -plan-json is for a staging root").)
internal/host/probes.go, alongside restartPathActive:
const systemUpdatePathUnit = "carlos-system-update.path"
// systemUpdatePathActive: is the self-update relay armed? nil = unknown.
func systemUpdatePathActive() *bool {
out, _ := exec.Command("systemctl", "is-active", systemUpdatePathUnit).CombinedOutput()
return parseIsActive(string(out))
}(copy the exact shape of restartPathActive at probes.go:82, including how it runs systemctl).
- [ ] Step 5: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: ok. Then, as a real check of the flag: go run . bootstrap -root /tmp/c/stage -plan-json 2>/dev/null | head -3 prints one JSON object with enabled and inert.
- [ ] Step 6: Commit
gofmt -l internal/host/ && git add internal/host/units/carlos-system-update.path internal/host/units/carlos-system-update.service internal/host/units/carlos-system-recover.service internal/host/bootstrap.go internal/host/probes.go internal/host/host_test.go && git commit -q -F - <<'EOF'
host: the self-update relay, apply and recover units, and a JSON arming plan
bootstrap -root <dir> -plan-json prints this binary's enabledUnits and
inert list, so an apply arms what the NEW binary's bootstrap would arm.
Both oneshots exec carlos.prev when a journal is open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 7: The agent leg (internal/host/systemleg.go)
Files:
- Create:
internal/host/systemleg.go - Modify:
internal/host/agent.go—fastDeps(:1077),fastTick(:1330; the serial-move branch:1352-1367and the unconditional part beforefastMembers:1370),fastConvergeLoop(:1434-1459) - Test:
internal/host/systemleg_test.go,internal/host/agent_test.go(fastFixture.depsat:1098may leave the new field nil)
Interfaces:
- Consumes:
release.GetSystemPointer,release.FetchToFile,release.SystemComponentRouter,release.OptDir,systemChannel,systemRequest,readSystemResult,readSystemSeen,systemCachePath,systemRequestPath,systemRequestMarker,systemRequestEvery,systemArtifactLimit,validSystemVersion,version.Build. - Produces: ``
go type systemLeg struct { … } func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg func (l *systemLeg) observe(ctx context.Context) // serial move / hourly: read pointer, decide, set pending func (l *systemLeg) tick(ctx context.Context, now time.Time) // every 2 s: fire pending, re-touch, hourly re-observe // fastDeps gains: systemLeg *systemLeg`Test seams on the struct:now func() time.Time,jitter func(time.Duration) time.Duration,build string,platform string`.
- [ ] Step 1: Write the failing tests
package host
import (
"context"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/release"
)
type legFixture struct {
t *testing.T
store *release.DirStore
priv ed25519.PrivateKey
leg *systemLeg
dd string
opt string
now time.Time
}
func newLegFixture(t *testing.T, channel, build string) *legFixture {
t.Helper()
priv, pub, _ := ed25519.GenerateKey(nil)
f := &legFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, dd: t.TempDir(), opt: t.TempDir(),
now: time.Date(2026, 9, 6, 10, 0, 0, 0, time.UTC)}
getenv := func(k string) string {
if k == systemChannelEnv {
return channel
}
return ""
}
f.leg = newSystemLeg(f.store, pub, getenv, f.dd, f.opt)
f.leg.build = build
f.leg.platform = "linux-arm64"
f.leg.now = func() time.Time { return f.now }
f.leg.jitter = func(d time.Duration) time.Duration { return d / 2 } // deterministic: half the spread
return f
}
func (f *legFixture) promote(version string, body []byte, spread time.Duration) {
f.t.Helper()
sum := sha256.Sum256(body)
rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{
"linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}}
if _, err := f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, ""); err != nil {
f.t.Fatal(err)
}
if _, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.leg.channel, rel,
release.SystemPromoteOptions{Spread: spread}); err != nil {
f.t.Fatal(err)
}
}
func (f *legFixture) marker() (os.FileInfo, error) { return os.Stat(filepath.Join(f.dd, systemRequestMarker)) }
func (f *legFixture) request() *systemRequest {
var r systemRequest
if err := readJSONFile(systemRequestPath(f.dd), &r); err != nil {
return nil
}
return &r
}
func TestSystemLegDoesNothingWithoutAPointer(t *testing.T) {
f := newLegFixture(t, "stable", "v1")
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now.Add(time.Hour))
if _, err := f.marker(); err == nil {
t.Fatal("marker touched with no pointer")
}
}
func TestSystemLegFetchesVerifiesAndRequestsAfterTheSpread(t *testing.T) {
f := newLegFixture(t, "stable", "v1")
body := []byte("ELF v2")
f.promote("v2", body, 10*time.Minute)
f.leg.observe(context.Background())
if _, err := f.marker(); err == nil {
t.Fatal("requested before the spread elapsed")
}
f.leg.tick(context.Background(), f.now.Add(4*time.Minute)) // jitter = 5m
if _, err := f.marker(); err == nil {
t.Fatal("requested before the deadline")
}
f.leg.tick(context.Background(), f.now.Add(5*time.Minute+time.Second))
if _, err := f.marker(); err != nil {
t.Fatal("marker not touched after the deadline")
}
got, _ := os.ReadFile(systemCachePath(f.dd, "v2"))
if string(got) != "ELF v2" {
t.Fatalf("cache = %q", got)
}
r := f.request()
if r == nil || r.Version != "v2" || r.NoOp || r.PromotedAt.IsZero() {
t.Fatalf("request = %+v", r)
}
raw, _ := os.ReadFile(systemRequestPath(f.dd))
for _, never := range []string{"channel", "key", "sha", "path"} {
if strings.Contains(string(raw), `"`+never+`"`) {
t.Errorf("request carries %q — root must not take that from the sandbox", never)
}
}
}
func TestSystemLegRetouchesUntilRootAnswers(t *testing.T) {
f := newLegFixture(t, "stable", "v1")
f.promote("v2", []byte("x"), 0)
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now)
first, _ := f.marker()
f.leg.tick(context.Background(), f.now.Add(30*time.Second))
same, _ := f.marker()
if !same.ModTime().Equal(first.ModTime()) {
t.Fatal("re-touched inside the minute throttle")
}
// The touch is a rewrite (PathChanged needs a change), so mtime moves.
time.Sleep(20 * time.Millisecond)
f.leg.tick(context.Background(), f.now.Add(61*time.Second))
later, _ := f.marker()
if !later.ModTime().After(first.ModTime()) {
t.Fatal("not re-touched after a minute with no result")
}
// Root answers: a terminal result for this tuple stops the re-touch and
// the leg deletes its own cache.
r := f.request()
os.MkdirAll(systemRootDir(f.opt), 0o755)
writeJSONDurable(filepath.Join(systemRootDir(f.opt), "result.json"),
systemResult{Channel: "stable", Version: "v2", PromotedAt: r.PromotedAt, Outcome: systemOutcomeRolledBack, Reason: "gate", At: f.now}, 0o644)
time.Sleep(20 * time.Millisecond)
f.leg.tick(context.Background(), f.now.Add(3*time.Minute))
final, _ := f.marker()
if !final.ModTime().Equal(later.ModTime()) {
t.Fatal("re-touched after a terminal result")
}
if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
t.Fatal("cache kept after a terminal result")
}
// And a later observe of the SAME pointer does nothing: refuse to repeat a failure.
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now.Add(10*time.Minute))
if f.leg.pending != nil {
t.Fatal("re-requested a pointer root already rolled back")
}
}
func TestSystemLegNoOpRequestFetchesNothing(t *testing.T) {
f := newLegFixture(t, "stable", "v2")
f.promote("v2", []byte("same"), 0)
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now)
r := f.request()
if r == nil || !r.NoOp {
t.Fatalf("request = %+v, want a no-op", r)
}
if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
t.Fatal("a no-op must not download the artifact")
}
// Once root has seen it (seen.json advanced to this PromotedAt), nothing more.
os.MkdirAll(systemRootDir(f.opt), 0o755)
writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": r.PromotedAt}, 0o644)
f.leg.pending = nil
os.Remove(filepath.Join(f.dd, systemRequestMarker))
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now.Add(2*time.Minute))
if _, err := f.marker(); err == nil {
t.Fatal("requested again after root recorded the watermark")
}
}
func TestSystemLegIgnoresDevBuildEqualityAndOtherPlatforms(t *testing.T) {
f := newLegFixture(t, "stable", "dev")
f.promote("v2", []byte("x"), 0)
f.leg.platform = "linux-amd64" // no artifact for me
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now)
if _, err := f.marker(); err == nil {
t.Fatal("requested with no artifact for this platform")
}
}
func TestSystemLegRefusesShaMismatchAndOversize(t *testing.T) {
f := newLegFixture(t, "stable", "v1")
f.promote("v2", []byte("good"), 0)
// Swap the object under the signed sha.
f.store.Put(context.Background(), release.SystemArtifactKey("v2", "linux-arm64"), []byte("evil"), "")
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now)
if _, err := f.marker(); err == nil {
t.Fatal("requested an artifact whose sha does not match")
}
if _, err := os.Stat(systemCachePath(f.dd, "v2")); err == nil {
t.Fatal("mismatched artifact left in the cache")
}
}
func TestSystemLegIgnoresAPointerOlderThanTheWatermark(t *testing.T) {
f := newLegFixture(t, "stable", "v1")
f.promote("v2", []byte("x"), 0)
os.MkdirAll(systemRootDir(f.opt), 0o755)
writeJSONDurable(filepath.Join(systemRootDir(f.opt), "seen.json"), systemSeen{"stable": time.Now().Add(time.Hour)}, 0o644)
f.leg.observe(context.Background())
f.leg.tick(context.Background(), f.now)
if _, err := f.marker(); err == nil {
t.Fatal("requested a pointer root would refuse as a replay")
}
}
func TestFastTickRunsTheSystemLeg(t *testing.T) {
fx := newFastFixture(t)
d := fx.deps(nil)
observes := 0
d.systemLeg = &systemLeg{observeHook: func() { observes++ }}
ctx := context.Background()
fastTick(ctx, d, time.Now()) // boot seed
if observes != 0 {
t.Fatalf("boot seed observed %d times", observes)
}
release.BumpSerial(ctx, fx.store, time.Now())
fastTick(ctx, d, time.Now().Add(10*time.Second))
if observes != 1 {
t.Fatalf("one serial move → %d observes", observes)
}
}(observeHook func() is a test seam on systemLeg: when non-nil, observe calls it and returns. Add "crypto/ed25519" to the imports.)
- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestSystemLeg|TestFastTickRunsTheSystemLeg' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: undefined systemLeg, newSystemLeg.
- [ ] Step 3: Implement
internal/host/systemleg.go
package host
import (
"context"
"crypto/ed25519"
"errors"
"log"
"math/rand/v2"
"os"
"path/filepath"
"runtime"
"time"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/version"
)
// systemLeg is the sandboxed half of the platform self-update (spec §4):
// notice the channel pointer, fetch and verify the artifact into /data, ask
// root. Two entry points so the spread has something to run it: observe on
// a serial move (and hourly), tick every 2 s with no store read.
type systemLeg struct {
store release.Store
pub ed25519.PublicKey
channel string
platform string
build string
dataDir string
optDir string
now func() time.Time
jitter func(time.Duration) time.Duration
pending *systemPending
lastTouch time.Time
lastObserve time.Time
backoffUntil time.Time
backoff time.Duration
observeHook func() // test seam
}
type systemPending struct {
version string
promotedAt time.Time
noop bool
artifact release.SystemArtifact
deadline time.Time
requested bool
}
const systemObserveEvery = time.Hour
func newSystemLeg(store release.Store, pub ed25519.PublicKey, getenv func(string) string, dataDir, optDir string) *systemLeg {
return &systemLeg{
store: store, pub: pub, channel: systemChannel(getenv),
platform: runtime.GOOS + "-" + runtime.GOARCH, build: version.Build,
dataDir: dataDir, optDir: optDir,
now: time.Now,
jitter: func(spread time.Duration) time.Duration { return rand.N(spread + 1) },
}
}
// observe reads the pointer and decides. It never blocks on a download:
// that happens in tick, after the spread.
func (l *systemLeg) observe(ctx context.Context) {
if l.observeHook != nil {
l.observeHook()
return
}
now := l.now()
l.lastObserve = now
p, _, err := release.GetSystemPointer(ctx, l.store, l.pub, release.SystemComponentRouter, l.channel)
if err != nil {
if !errors.Is(err, os.ErrNotExist) {
log.Printf("agent: system pointer %s: %v", l.channel, err)
}
return
}
seen, _ := readSystemSeen(l.optDir)
if wm, ok := seen[l.channel]; ok && p.PromotedAt.Before(wm) {
return // root would refuse it as a replay
}
if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.Current.Version, p.PromotedAt) && res.terminal() {
l.forget(p.Current.Version)
return // refuse to repeat a failure
}
noop := p.Current.Version == l.build
if noop {
if wm, ok := seen[l.channel]; ok && !p.PromotedAt.After(wm) {
return // nothing to do, and root has seen this promote
}
}
art, ok := p.Current.Artifacts[l.platform]
if !ok && !noop {
if l.pending == nil || l.pending.version != p.Current.Version {
log.Printf("agent: system pointer %s names %s with no artifact for %s — not for this box", l.channel, p.Current.Version, l.platform)
}
return
}
if l.pending != nil && l.pending.version == p.Current.Version && l.pending.promotedAt.Equal(p.PromotedAt) {
return // already pending; keep its deadline
}
l.pending = &systemPending{version: p.Current.Version, promotedAt: p.PromotedAt, noop: noop, artifact: art,
deadline: now.Add(l.jitter(p.SpreadDuration()))}
if noop {
l.pending.deadline = now
}
}
// tick fires a pending request whose deadline has passed and re-touches an
// unanswered one every minute. It reads no store except for the one fetch.
func (l *systemLeg) tick(ctx context.Context, now time.Time) {
if now.Sub(l.lastObserve) >= systemObserveEvery {
l.observe(ctx)
}
p := l.pending
if p == nil {
return
}
if res, _ := readSystemResult(l.optDir); res.matches(l.channel, p.version, p.promotedAt) {
switch {
case res.terminal():
l.forget(p.version)
return
case res.Outcome == systemOutcomeNoted:
l.pending = nil
return
case res.Outcome == systemOutcomeDeferred && res.At.After(p.deadline):
// root could not reach the store; try again with backoff
if l.backoff == 0 {
l.backoff = fastBackoffBase
} else if l.backoff < fastBackoffCap {
l.backoff *= 2
}
p.deadline = now.Add(l.backoff)
p.requested = false
}
}
if now.Before(p.deadline) || now.Before(l.backoffUntil) {
return
}
if !p.requested {
if !p.noop {
if err := l.fetch(ctx, p); err != nil {
log.Printf("agent: system update %s: %v", p.version, err)
os.Remove(systemCachePath(l.dataDir, p.version))
l.backoffUntil = now.Add(fastBackoffBase)
return
}
}
if err := l.writeRequest(p, now); err != nil {
log.Printf("agent: system update %s: request: %v", p.version, err)
return
}
p.requested = true
}
if now.Sub(l.lastTouch) >= systemRequestEvery {
if err := os.WriteFile(filepath.Join(l.dataDir, systemRequestMarker), []byte(now.UTC().Format(time.RFC3339Nano)+"\n"), 0o644); err != nil {
log.Printf("agent: system update %s: cannot ask for a root apply: %v", p.version, err)
return
}
l.lastTouch = now
}
}
func (l *systemLeg) fetch(ctx context.Context, p *systemPending) error {
dst := systemCachePath(l.dataDir, p.version)
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
return err
}
return release.FetchToFile(ctx, l.store, p.artifact.Key, dst, systemArtifactLimit, p.artifact.SHA256)
}
func (l *systemLeg) writeRequest(p *systemPending, now time.Time) error {
if err := os.MkdirAll(systemDataDir(l.dataDir), 0o755); err != nil {
return err
}
return writeJSONDurable(systemRequestPath(l.dataDir), systemRequest{Version: p.version, PromotedAt: p.promotedAt, NoOp: p.noop, SeenAt: now.UTC()}, 0o644)
}
// forget: root has answered with a terminal outcome. Drop the pending item
// and the cache — root never deletes under /data, the agent does.
func (l *systemLeg) forget(version string) {
l.pending = nil
l.backoff = 0
if validSystemVersion(version) {
os.RemoveAll(filepath.Dir(systemCachePath(l.dataDir, version)))
}
}agent.go wiring:
fastDepsgainssystemLeg *systemLeg // the platform self-update leg (spec §4); nil = not armed.- In
fastTick, inside the serial-move branch after thescheduleLegcall (:1365):if d.systemLeg != nil { d.systemLeg.observe(ctx) }. - Before
members := d.fastMembers(now)(:1370), outside the switch:if d.systemLeg != nil { d.systemLeg.tick(ctx, now) }. - In
fastConvergeLoop, afterd.scheduleLeg = scheduleLegFor(sd)(:1506):d.systemLeg = newSystemLeg(store, pub, os.Getenv, dd, optDir)and extend the "armed" log line with+ system pointer. - The
adoptLoopdoc comment's DO-NOT list (:566-600) gets one sentence: the system leg lives in the fast loop and re-observes hourly on its own; nothing is added here.
- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: ok.
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemleg.go internal/host/systemleg_test.go internal/host/agent.go internal/host/agent_test.go && git commit -q -F - <<'EOF'
host: the agent notices a platform promote, fetches, verifies, and asks root
observe on a serial move (and hourly), tick every 2 s: spread jitter,
streamed fetch with the signed sha, a request carrying only version
and promoted_at, and a marker re-touched every minute until root
answers. A pointer root already rolled back is never re-requested.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 8a: carlos system apply — the refuse path (steps 1–5, no-op, lock)
Files:
- Create:
internal/host/systemapply.go - Test:
internal/host/systemapply_test.go
Interfaces:
- Consumes: Task 4 state helpers;
release.GetSystemPointer;release.SystemComponentRouter;systemChannel. - Produces (package
host): ``go type sysOps struct { systemctl func(args ...string) (string, error) run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) now func() time.Time sleep func(time.Duration) probeEdge func() error edgeStartedThisBoot func() bool logf func(format string, args ...any) } type systemApplier struct { ops sysOps store release.Store pub ed25519.PublicKey channel string optDir, dataDir string etcDir string // "/etc" in production; a temp dir in tests build string // version.Build of THIS process gateWait, soak time.Duration // 20s, 15s in production } func (a *systemApplier) Run(ctx context.Context, recover bool) error const ( systemQuiesceUnits = "carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service" systemEdgeUnit = "carlos-edge" ) func (a *systemApplier) rootDir() string; stagingDir() string; lockPath() string func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error func (a *systemApplier) readRequest() (systemRequest, error) // O_NOFOLLOW walk, 4 KiB cap, validSystemVersion func (a *systemApplier) verifyPointer(ctx) (*release.SystemPointer, error) func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) // path of staging/carlos func openBeneath(root string, parts ...string) (*os.File, error) // per-component O_DIRECTORY|O_NOFOLLOW, then O_NOFOLLOW var errSystemRefused = errors.New("refused"); var errSystemDeferred = errors.New("deferred")`In this taskRunhandles: lock → (recovery: Task 8c, stub as "journal present → return error") → read request → verify pointer + replay guard → advance seen → no-op → stage candidate → **stop witherrNotYetImplemented** (Task 8b continues).Runreturns nil onnoted/refused/deferredafter writingresult.json; exit codes areRunSystemApply`'s business (8c).
- [ ] Step 1: Write the failing tests
package host
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"encoding/hex"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/release"
)
// applyFixture is a fake box: temp opt/data/etc roots, a DirStore with a
// signed pointer, and a sysOps that records every call and answers like a
// healthy systemd. The candidate "binary" is answered by the fake run.
type applyFixture struct {
t *testing.T
a *systemApplier
store *release.DirStore
priv ed25519.PrivateKey
calls []string // "systemctl is-active carlos-edge", "run <base> version", …
answer map[string]string // "version:<path>" → stdout; "plan:<path>" → JSON plan
now time.Time
edgeUp bool
nrestarts string
activeEnter string
probeErr error
edgeStartedThisBoot bool
}
func newApplyFixture(t *testing.T, channel, build string) *applyFixture {
t.Helper()
priv, pub, _ := ed25519.GenerateKey(nil)
f := &applyFixture{t: t, store: release.NewDirStore(t.TempDir()), priv: priv, answer: map[string]string{},
now: time.Date(2026, 9, 6, 11, 0, 0, 0, time.UTC), edgeUp: true, nrestarts: "0", activeEnter: "1000", edgeStartedThisBoot: true}
f.a = &systemApplier{
store: f.store, pub: pub, channel: channel,
optDir: t.TempDir(), dataDir: t.TempDir(), etcDir: t.TempDir(), build: build,
}
f.a.ops = sysOps{
systemctl: func(args ...string) (string, error) {
f.calls = append(f.calls, "systemctl "+strings.Join(args, " "))
switch {
case args[0] == "is-active" && args[1] == systemEdgeUnit:
if f.edgeUp {
return "active\n", nil
}
return "failed\n", errors.New("inactive")
case args[0] == "show":
return f.nrestarts + "\n" + f.activeEnter + "\n", nil
}
return "", nil
},
run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) {
f.calls = append(f.calls, "run "+filepath.Base(name)+" "+strings.Join(args, " "))
if len(args) == 1 && args[0] == "version" {
if v, ok := f.answer["version:"+name]; ok {
return v, nil
}
return "", errors.New("no such binary")
}
if len(args) >= 3 && args[0] == "bootstrap" && args[1] == "-root" {
plan, ok := f.answer["plan:"+name]
if !ok {
return "", errors.New("bootstrap failed")
}
units := strings.Split(f.answer["units:"+name], " ")
dir := filepath.Join(args[2], "etc", "systemd", "system")
os.MkdirAll(dir, 0o755)
for _, u := range units {
if u != "" {
os.WriteFile(filepath.Join(dir, u), []byte("[Unit]\n# "+filepath.Base(name)+"\n"), 0o644)
}
}
os.MkdirAll(filepath.Join(args[2], "etc", "sysusers.d"), 0o755)
os.WriteFile(filepath.Join(args[2], "etc", "sysusers.d", "carlos.conf"), []byte("g carlos-sockets -\n"), 0o644)
os.MkdirAll(filepath.Join(args[2], "etc", "tmpfiles.d"), 0o755)
os.WriteFile(filepath.Join(args[2], "etc", "tmpfiles.d", "carlos.conf"), []byte("d /run/carlos\n"), 0o644)
os.MkdirAll(filepath.Join(args[2], "data"), 0o755)
os.WriteFile(filepath.Join(args[2], "data", "registry.db"), nil, 0o640)
return plan, nil
}
return "", errors.New("unexpected command")
},
now: func() time.Time { return f.now },
sleep: func(d time.Duration) { f.now = f.now.Add(d) },
probeEdge: func() error { return f.probeErr },
edgeStartedThisBoot: func() bool { return f.edgeStartedThisBoot },
logf: t.Logf,
}
os.MkdirAll(systemRootDir(f.a.optDir), 0o755)
os.MkdirAll(filepath.Join(f.a.etcDir, "systemd", "system"), 0o755)
// The on-disk binary answers `version` with build.
os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos "+build+"\n"), 0o755)
f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos " + build + "\n"
f.answer["plan:"+systemBinaryPath(f.a.optDir)] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service"],"inert":["carlos-restore-verify.timer"]}`
f.answer["units:"+systemBinaryPath(f.a.optDir)] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-old-only.timer"
return f
}
// promote signs a pointer for f.a.channel naming version with body as the
// linux-arm64 artifact, stages body into the agent's cache, and writes the
// request the agent would have written. Returns the request.
func (f *applyFixture) promote(version string, body []byte, noop bool) systemRequest {
f.t.Helper()
sum := sha256.Sum256(body)
rel := release.SystemRelease{Version: version, Artifacts: map[string]release.SystemArtifact{
"linux-arm64": {Key: release.SystemArtifactKey(version, "linux-arm64"), SHA256: hex.EncodeToString(sum[:])}}}
f.store.Put(context.Background(), rel.Artifacts["linux-arm64"].Key, body, "")
p, err := release.PromoteSystem(context.Background(), f.store, release.RootSigner(f.priv), "router", f.a.channel, rel, release.SystemPromoteOptions{})
if err != nil {
f.t.Fatal(err)
}
req := systemRequest{Version: version, PromotedAt: p.PromotedAt, NoOp: noop, SeenAt: f.now}
os.MkdirAll(systemDataDir(f.a.dataDir), 0o755)
writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644)
if !noop {
os.MkdirAll(filepath.Dir(systemCachePath(f.a.dataDir, version)), 0o755)
os.WriteFile(systemCachePath(f.a.dataDir, version), body, 0o644)
}
return req
}
func (f *applyFixture) result() *systemResult { r, _ := readSystemResult(f.a.optDir); return r }
func (f *applyFixture) called(prefix string) bool {
for _, c := range f.calls {
if strings.HasPrefix(c, prefix) {
return true
}
}
return false
}
func TestApplyNoRequestIsNothingToDo(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if f.result() != nil || len(f.calls) != 0 {
t.Fatalf("result=%+v calls=%v", f.result(), f.calls)
}
}
func TestApplyLockKeepsTwoRunsApart(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
f.promote("v2", []byte("x"), false)
// Hold the lock as another process would.
held, err := os.OpenFile(f.a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
t.Fatal(err)
}
defer held.Close()
if err := flockExclusiveNonblock(held); err != nil {
t.Fatal(err)
}
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatalf("a locked-out path run must exit 0 quietly: %v", err)
}
if f.result() != nil {
t.Fatal("a locked-out run wrote a result")
}
}
func TestApplyRequestIsAHintNotAnInstruction(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
// A symlinked parent under /data.
os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere"), 0o755)
os.Symlink(filepath.Join(f.a.dataDir, "elsewhere"), filepath.Join(f.a.dataDir, "system"))
os.MkdirAll(filepath.Join(f.a.dataDir, "elsewhere", "router"), 0o755)
os.WriteFile(filepath.Join(f.a.dataDir, "elsewhere", "router", "requested.json"), []byte(`{"version":"v2"}`), 0o644)
if _, err := f.a.readRequest(); err == nil {
t.Fatal("followed a symlinked parent under /data")
}
os.Remove(filepath.Join(f.a.dataDir, "system"))
// A symlinked request file.
os.MkdirAll(systemDataDir(f.a.dataDir), 0o755)
os.WriteFile(filepath.Join(f.a.dataDir, "real.json"), []byte(`{"version":"v2"}`), 0o644)
os.Symlink(filepath.Join(f.a.dataDir, "real.json"), systemRequestPath(f.a.dataDir))
if _, err := f.a.readRequest(); err == nil {
t.Fatal("followed a symlinked request")
}
os.Remove(systemRequestPath(f.a.dataDir))
// A traversal version.
writeJSONDurable(systemRequestPath(f.a.dataDir), systemRequest{Version: "../../etc/passwd", PromotedAt: f.now}, 0o644)
if _, err := f.a.readRequest(); err == nil {
t.Fatal("accepted a traversal version")
}
// Oversized.
os.WriteFile(systemRequestPath(f.a.dataDir), []byte(`{"version":"`+strings.Repeat("a", 5000)+`"}`), 0o644)
if _, err := f.a.readRequest(); err == nil {
t.Fatal("accepted a 5 KiB request")
}
}
func TestApplyChannelComesFromRootNotTheRequest(t *testing.T) {
// The box is on stable; a valid CANARY pointer exists and the agent
// requests its version. Root reads control/system/router/stable.json
// (missing) and refuses — a signed canary promote does not authorise a
// stable box.
f := newApplyFixture(t, "stable", "v1")
f.a.channel = "canary"
req := f.promote("v2", []byte("x"), false) // signs the CANARY pointer
f.a.channel = "stable"
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
r := f.result()
if r == nil || r.Outcome != systemOutcomeRefused || !r.matches("stable", "v2", req.PromotedAt) {
t.Fatalf("result = %+v", r)
}
if _, err := os.Stat(systemBinaryPath(f.a.optDir) + ".prev"); err == nil {
t.Fatal("touched the box")
}
}
func TestApplyStaleTupleIsRefusedAgainstTheRequestsTuple(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
old := f.promote("v2", []byte("x"), false)
time.Sleep(1100 * time.Millisecond)
f.promote("v3", []byte("y"), false)
// Put the OLD request back: the agent's spread started under v2.
writeJSONDurable(systemRequestPath(f.a.dataDir), old, 0o644)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
r := f.result()
if r == nil || r.Outcome != systemOutcomeRefused || r.Version != "v2" || !r.PromotedAt.Equal(old.PromotedAt) {
t.Fatalf("result = %+v (must name the REQUEST's tuple so v3 is not suppressed)", r)
}
}
func TestApplyReplayGuardIsPerChannelAndEqualIsAllowed(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
req := f.promote("v2", []byte("x"), true)
// Older than the watermark → refused, seen unchanged.
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt.Add(time.Hour), "canary": time.Time{}}, 0o644)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeRefused || !strings.Contains(r.Reason, "replay") {
t.Fatalf("result = %+v", r)
}
// Equal → allowed (a retry of the same promote).
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"stable": req.PromotedAt}, 0o644)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
t.Fatalf("result = %+v", r)
}
// Another channel's newer watermark does not block this channel.
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "seen.json"), systemSeen{"canary": req.PromotedAt.Add(time.Hour)}, 0o644)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
t.Fatalf("result = %+v", r)
}
seen, _ := readSystemSeen(f.a.optDir)
if !seen["stable"].Equal(req.PromotedAt) {
t.Fatalf("seen not advanced: %v", seen)
}
}
func TestApplyNoOpNotesAndTouchesNothing(t *testing.T) {
f := newApplyFixture(t, "stable", "v2")
f.promote("v2", []byte("x"), true)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeNoted {
t.Fatalf("result = %+v", r)
}
if _, err := os.Stat(f.a.stagingDir()); err == nil {
t.Fatal("a no-op created staging/")
}
if len(f.calls) != 0 {
t.Fatalf("a no-op ran commands: %v", f.calls)
}
}
func TestApplyStoreErrorIsDeferredNotRefused(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
req := f.promote("v2", []byte("x"), false)
f.a.store = failingStore{f.store}
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeDeferred || !r.matches("stable", "v2", req.PromotedAt) {
t.Fatalf("result = %+v", r)
}
}
type failingStore struct{ release.Store }
func (failingStore) Get(context.Context, string) ([]byte, string, error) {
return nil, "", errors.New("door restarting")
}
func TestApplyStagesTheCandidateFromARootOwnedCopy(t *testing.T) {
f := newApplyFixture(t, "stable", "v1")
body := []byte("#!/bin/sh\necho carlos v2\n")
f.promote("v2", body, false)
f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v2\n"
p, err := f.a.verifyPointer(context.Background())
if err != nil {
t.Fatal(err)
}
req, _ := f.a.readRequest()
staged, err := f.a.stageCandidate(p, req)
if err != nil {
t.Fatal(err)
}
st, _ := os.Stat(staged)
if st.Mode().Perm() != 0o755 {
t.Fatalf("staged mode = %v after verification", st.Mode().Perm())
}
if dst, _ := os.Stat(f.a.stagingDir()); dst.Mode().Perm() != 0o700 {
t.Fatalf("staging dir mode = %v", dst.Mode().Perm())
}
// Swapping the cache after staging changes nothing about the copy.
os.WriteFile(systemCachePath(f.a.dataDir, "v2"), []byte("evil"), 0o644)
got, _ := os.ReadFile(staged)
if string(got) != string(body) {
t.Fatal("staged copy follows the cache")
}
// A sha mismatch is refused BEFORE the copy is made executable and
// staging/ is cleared.
f2 := newApplyFixture(t, "stable", "v1")
f2.promote("v2", body, false)
os.WriteFile(systemCachePath(f2.a.dataDir, "v2"), []byte("evil"), 0o644)
p2, _ := f2.a.verifyPointer(context.Background())
req2, _ := f2.a.readRequest()
if _, err := f2.a.stageCandidate(p2, req2); err == nil || !strings.Contains(err.Error(), "sha256") {
t.Fatalf("err = %v", err)
}
// A candidate that lies about its version is refused.
f3 := newApplyFixture(t, "stable", "v1")
f3.promote("v2", body, false)
f3.answer["version:"+filepath.Join(f3.a.stagingDir(), "carlos")] = "carlos v9\n"
p3, _ := f3.a.verifyPointer(context.Background())
req3, _ := f3.a.readRequest()
if _, err := f3.a.stageCandidate(p3, req3); err == nil || !strings.Contains(err.Error(), "reports") {
t.Fatalf("err = %v", err)
}
// A symlinked cache file is refused.
f4 := newApplyFixture(t, "stable", "v1")
f4.promote("v2", body, false)
os.Remove(systemCachePath(f4.a.dataDir, "v2"))
os.WriteFile(filepath.Join(f4.a.dataDir, "real"), body, 0o644)
os.Symlink(filepath.Join(f4.a.dataDir, "real"), systemCachePath(f4.a.dataDir, "v2"))
p4, _ := f4.a.verifyPointer(context.Background())
req4, _ := f4.a.readRequest()
if _, err := f4.a.stageCandidate(p4, req4); err == nil {
t.Fatal("followed a symlinked cache file")
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: undefined systemApplier, sysOps, flockExclusiveNonblock.
- [ ] Step 3: Implement the refuse path in
internal/host/systemapply.go
package host
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"syscall"
"time"
"github.com/carlosframework/platform/internal/release"
)
// The root half of the platform self-update (spec §6). Runs as root from
// carlos-system-update.service (or --recover from carlos-system-recover).
// Every box-changing step is journaled first; every run begins with
// recovery; root reads ONE file under /data and treats it as a hint.
const (
systemEdgeUnit = "carlos-edge"
systemLockFile = "lock"
systemMaxRequest = 4 << 10
systemGateWait = 20 * time.Second
systemGateSoak = 15 * time.Second
systemVersionWait = 30 * time.Second
systemBootstrapWait = 120 * time.Second
)
// systemQuiesceUnits are the root units that exec /opt/carlos/carlos
// against the running edge. Stopped (services too — stopping a timer does
// not stop the pass it already started) before the rename, re-armed after
// the gate.
var systemQuiesceUnits = []string{
"carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path", "carlos-restart.path",
"carlos-adopt.service", "carlos-config-sync.service", "carlos-stage.service", "carlos-restart.service",
}
var (
errSystemRefused = errors.New("refused")
errSystemDeferred = errors.New("deferred")
errNotYetImplemented = errors.New("system apply: install path not implemented")
)
// sysOps is the seam between the apply and the box: systemctl, exec with a
// deadline, the clock, sleep, the edge probe. Tests record and answer.
type sysOps struct {
systemctl func(args ...string) (string, error)
run func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error)
now func() time.Time
sleep func(time.Duration)
probeEdge func() error
edgeStartedThisBoot func() bool
logf func(format string, args ...any)
}
type systemApplier struct {
ops sysOps
store release.Store
pub ed25519.PublicKey
channel string
optDir string
dataDir string
etcDir string
build string
gateWait, soak time.Duration
}
func (a *systemApplier) rootDir() string { return systemRootDir(a.optDir) }
func (a *systemApplier) stagingDir() string { return filepath.Join(a.rootDir(), "staging") }
func (a *systemApplier) lockPath() string { return filepath.Join(a.rootDir(), systemLockFile) }
func (a *systemApplier) file(name string) string { return filepath.Join(a.rootDir(), name) }
// Run is the whole oneshot. It returns an error only for a failure the
// caller should exit non-zero on (an apply that rolled back, a rollback
// that failed, a broken box); refusals and no-ops are recorded in
// result.json and return nil.
func (a *systemApplier) Run(ctx context.Context, recover bool) error {
if err := os.MkdirAll(a.rootDir(), 0o755); err != nil {
return err
}
lock, err := os.OpenFile(a.lockPath(), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
return err
}
defer lock.Close()
if recover {
if err := flockExclusive(lock); err != nil {
return err
}
} else if err := flockExclusiveNonblock(lock); err != nil {
a.ops.logf("system apply: another apply is running; leaving it to finish")
return nil
}
defer os.RemoveAll(a.stagingDir())
if err := a.recoverIfNeeded(ctx); err != nil {
return err
}
if recover {
return nil
}
req, err := a.readRequest()
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
a.ops.logf("system apply: request: %v", err)
return nil // nothing to record against: the request itself is unreadable
}
p, err := a.verifyPointer(ctx)
if err != nil {
return a.record(req, err)
}
if req.Version != p.Current.Version || !req.PromotedAt.Equal(p.PromotedAt) {
return a.record(req, fmt.Errorf("%w: request names %s@%s, pointer is %s@%s", errSystemRefused,
req.Version, req.PromotedAt.Format(time.RFC3339), p.Current.Version, p.PromotedAt.Format(time.RFC3339)))
}
seen, err := readSystemSeen(a.optDir)
if err != nil {
return a.record(req, fmt.Errorf("%w: seen.json unreadable: %v", errSystemRefused, err))
}
if wm, ok := seen[a.channel]; ok && p.PromotedAt.Before(wm) {
return a.record(req, fmt.Errorf("%w: replay — pointer promoted %s is older than the %s watermark %s", errSystemRefused,
p.PromotedAt.Format(time.RFC3339), a.channel, wm.Format(time.RFC3339)))
}
seen[a.channel] = p.PromotedAt
if err := writeJSONDurable(a.file("seen.json"), seen, 0o644); err != nil {
return err
}
if req.NoOp || p.Current.Version == a.onDiskBuild(ctx) {
return a.writeResult(req, systemOutcomeNoted, "")
}
staged, err := a.stageCandidate(p, req)
if err != nil {
return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err))
}
return a.install(ctx, req, p, staged) // Task 8b
}
func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error {
return errNotYetImplemented
}
func (a *systemApplier) recoverIfNeeded(ctx context.Context) error {
j, err := readSystemJournal(a.optDir)
if err != nil || j == nil {
return err
}
return fmt.Errorf("system apply: journal open at %s — recovery not implemented", j.Phase) // Task 8c
}
// record turns a refusal or deferral into result.json and a nil return.
func (a *systemApplier) record(req systemRequest, err error) error {
outcome := systemOutcomeRefused
if errors.Is(err, errSystemDeferred) {
outcome = systemOutcomeDeferred
}
a.ops.logf("system apply: %s %s: %s: %v", a.channel, req.Version, outcome, err)
return a.writeResult(req, outcome, err.Error())
}
func (a *systemApplier) writeResult(req systemRequest, outcome, reason string) error {
return writeJSONDurable(a.file("result.json"), systemResult{
Channel: a.channel, Version: req.Version, PromotedAt: req.PromotedAt,
Outcome: outcome, Reason: reason, At: a.ops.now().UTC(),
}, 0o644)
}
// readRequest is the one read under /data: a fixed path reached without
// following a single symlink, a regular file under 4 KiB, a version that
// passes validSystemVersion. Anything else is an error, and root acts on
// nothing.
func (a *systemApplier) readRequest() (systemRequest, error) {
f, err := openBeneath(a.dataDir, "system", systemComponent, "requested.json")
if err != nil {
return systemRequest{}, err
}
defer f.Close()
st, err := f.Stat()
if err != nil {
return systemRequest{}, err
}
if !st.Mode().IsRegular() || st.Size() > systemMaxRequest {
return systemRequest{}, fmt.Errorf("request: not a regular file under %d bytes", systemMaxRequest)
}
var req systemRequest
if err := json.NewDecoder(io.LimitReader(f, systemMaxRequest)).Decode(&req); err != nil {
return systemRequest{}, fmt.Errorf("request: %w", err)
}
if !validSystemVersion(req.Version) {
return systemRequest{}, fmt.Errorf("request: version %q is not a version", req.Version)
}
if req.PromotedAt.IsZero() {
return systemRequest{}, errors.New("request: promoted_at unset")
}
return req, nil
}
// openBeneath opens root/parts... one component at a time with
// O_DIRECTORY|O_NOFOLLOW for every directory and O_NOFOLLOW for the leaf,
// so no component under a user-owned tree can be a symlink.
func openBeneath(root string, parts ...string) (*os.File, error) {
fd, err := syscall.Open(root, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, fmt.Errorf("open %s: %w", root, err)
}
for i, part := range parts {
flags := syscall.O_RDONLY | syscall.O_NOFOLLOW | syscall.O_CLOEXEC
if i < len(parts)-1 {
flags |= syscall.O_DIRECTORY
}
next, err := syscall.Openat(fd, part, flags, 0)
syscall.Close(fd)
if err != nil {
return nil, fmt.Errorf("open %s: %w", filepath.Join(append([]string{root}, parts[:i+1]...)...), err)
}
fd = next
}
return os.NewFile(uintptr(fd), filepath.Join(append([]string{root}, parts...)...)), nil
}
// verifyPointer fetches THIS box's channel pointer and verifies it with
// the release public key. The channel is root's own (host.env), never the
// request's. A store error is deferred, a bad pointer refused.
func (a *systemApplier) verifyPointer(ctx context.Context) (*release.SystemPointer, error) {
p, _, err := release.GetSystemPointer(ctx, a.store, a.pub, release.SystemComponentRouter, a.channel)
switch {
case err == nil:
return p, nil
case errors.Is(err, os.ErrNotExist):
return nil, fmt.Errorf("%w: no %s pointer for this box's channel", errSystemRefused, a.channel)
case strings.Contains(err.Error(), "system pointer"):
return nil, fmt.Errorf("%w: %v", errSystemRefused, err)
default:
return nil, fmt.Errorf("%w: %v", errSystemDeferred, err)
}
}
// onDiskBuild asks the installed binary what it is, with a deadline. Not
// version.Build: after an interrupted apply the two can differ.
func (a *systemApplier) onDiskBuild(ctx context.Context) string {
out, err := a.ops.run(ctx, systemVersionWait, systemBinaryPath(a.optDir), "version")
if err != nil {
return ""
}
return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos "))
}
// stageCandidate copies the agent's cache into staging/ (0700) as 0600,
// verifies the copy's sha against the pointer, makes it executable, and
// runs its `version`. Nothing under /data is read after the copy.
func (a *systemApplier) stageCandidate(p *release.SystemPointer, req systemRequest) (string, error) {
art, ok := p.Current.Artifacts[a.platform()]
if !ok {
return "", fmt.Errorf("pointer names no artifact for %s", a.platform())
}
os.RemoveAll(a.stagingDir())
if err := os.MkdirAll(a.stagingDir(), 0o700); err != nil {
return "", err
}
if err := os.Chmod(a.stagingDir(), 0o700); err != nil {
return "", err
}
src, err := openBeneath(a.dataDir, "system", systemComponent, req.Version, "carlos")
if err != nil {
return "", fmt.Errorf("candidate: %w", err)
}
defer src.Close()
if st, err := src.Stat(); err != nil || !st.Mode().IsRegular() {
return "", errors.New("candidate: not a regular file")
}
staged := filepath.Join(a.stagingDir(), "carlos")
dst, err := os.OpenFile(staged, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
if err != nil {
return "", err
}
h := sha256.New()
n, err := io.Copy(io.MultiWriter(dst, h), io.LimitReader(src, systemArtifactLimit+1))
dst.Close()
if err != nil {
return "", err
}
if n > systemArtifactLimit {
return "", fmt.Errorf("candidate: larger than %d bytes", systemArtifactLimit)
}
if got := hex.EncodeToString(h.Sum(nil)); got != art.SHA256 {
return "", fmt.Errorf("candidate: sha256 %s does not match the signed %s", got, art.SHA256)
}
if err := os.Chmod(staged, 0o755); err != nil {
return "", err
}
out, err := a.ops.run(context.Background(), systemVersionWait, staged, "version")
if err != nil || !strings.Contains(out, "carlos "+p.Current.Version) {
return "", fmt.Errorf("candidate reports %q, not %s", strings.TrimSpace(out), p.Current.Version)
}
return staged, nil
}
func (a *systemApplier) platform() string { return runtime.GOOS + "-" + runtime.GOARCH }
func flockExclusive(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX) }
func flockExclusiveNonblock(f *os.File) error { return syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) }Add "runtime" to the imports. Tests that stage a candidate need platform() to be linux-arm64: make platform a field set in the constructor (platform string, default runtime.GOOS+"-"+runtime.GOARCH) rather than a method, and have the fixture set f.a.platform = "linux-arm64". Do that; update the test fixture accordingly.
- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: PASS for every test in this task.
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
host: carlos system apply — the refuse path
Lock, one O_NOFOLLOW-walked request under /data, the channel from
root's own env, the pointer re-verified, the per-channel replay
watermark, the no-op, and the candidate copied into a 0700 staging
dir and verified there before it is made executable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 8b: carlos system apply — install, gate, arm, record (steps 6–12)
Files:
- Modify:
internal/host/systemapply.go(replace theinstallstub) - Test:
internal/host/systemapply_test.go(add)
Interfaces:
- Consumes:
systemctlPlan,bootstrapPlan,systemQuiesceUnits, the fixture from 8a. - Produces: ``
go func (a *systemApplier) install(ctx, req, p, staged) error func (a *systemApplier) decidePrev(ctx) (prevExists bool, err error) // journal backing-up, copy when authorised func (a *systemApplier) stagePlan(ctx, binary, dir string) (bootstrapPlan, []string, error) // runs bootstrap -root dir -plan-json; returns plan + unit file names func (a *systemApplier) journal(j systemJournal) error func (a *systemApplier) installUnits(stagedRoot string, remove []string) error // remove, copy units, sysusers, tmpfiles, daemon-reload func (a *systemApplier) restartEdge() error // reset-failed then restart func (a *systemApplier) gate(ctx, target string) error func (a *systemApplier) arm(plan bootstrapPlan) error // systemctlPlan verbatim func (a *systemApplier) rollback(ctx, req, reason string, j *systemJournal) error // Task 8c; stub returns the reason as an error``
- [ ] Step 1: Write the failing tests
Add to systemapply_test.go:
// applyHappy sets up a v1 box with a v2 candidate whose fake bootstrap
// plan adds carlos-new-only.path and drops carlos-old-only.timer.
func applyHappy(t *testing.T) (*applyFixture, systemRequest) {
f := newApplyFixture(t, "stable", "v1")
body := []byte("#!/bin/sh\necho carlos v2\n")
req := f.promote("v2", body, false)
staged := filepath.Join(f.a.stagingDir(), "carlos")
f.answer["version:"+staged] = "carlos v2\n"
f.answer["plan:"+staged] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service","carlos-new-only.path"],"inert":["carlos-restore-verify.timer"]}`
f.answer["units:"+staged] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-new-only.path"
// .prev, once made, answers as v1 and stages v1's units.
prev := systemPrevPath(f.a.optDir)
f.answer["version:"+prev] = "carlos v1\n"
f.answer["plan:"+prev] = f.answer["plan:"+systemBinaryPath(f.a.optDir)]
f.answer["units:"+prev] = f.answer["units:"+systemBinaryPath(f.a.optDir)]
// After the install the on-disk binary answers v2.
f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v2\n"
return f, req
}
func TestApplyInstallsGatesArmsAndRecords(t *testing.T) {
f, req := applyHappy(t)
if err := f.a.Run(context.Background(), false); err != nil {
t.Fatal(err)
}
r := f.result()
if r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) {
t.Fatalf("result = %+v", r)
}
ap, _ := readSystemApplied(f.a.optDir)
if ap == nil || ap.Version != "v2" || ap.Channel != "stable" {
t.Fatalf("applied = %+v", ap)
}
if j, _ := readSystemJournal(f.a.optDir); j != nil {
t.Fatalf("journal left open: %+v", j)
}
if _, err := os.Stat(f.a.stagingDir()); err == nil {
t.Fatal("staging/ left behind")
}
bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
if !strings.Contains(string(bin), "carlos v2") {
t.Fatal("binary not swapped")
}
prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
if !strings.Contains(string(prev), "carlos v1") {
t.Fatal(".prev is not the old binary")
}
units := filepath.Join(f.a.etcDir, "systemd", "system")
if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err != nil {
t.Error("new unit not installed")
}
if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err == nil {
t.Error("removed unit still installed")
}
if _, err := os.Stat(filepath.Join(f.a.etcDir, "sysusers.d", "carlos.conf")); err != nil {
t.Error("sysusers not installed")
}
if _, err := os.Stat(filepath.Join(f.a.etcDir, "tmpfiles.d", "carlos.conf")); err != nil {
t.Error("tmpfiles not installed")
}
if _, err := os.Stat(filepath.Join(f.a.etcDir, "registry.db")); err == nil {
t.Error("staging data/ copied")
}
// Order of the systemctl calls is the spec.
joined := strings.Join(f.calls, "\n")
order := []string{
"systemctl stop carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-adopt.service carlos-config-sync.service carlos-stage.service carlos-restart.service",
"systemctl disable --now carlos-old-only.timer",
"run systemd-sysusers",
"run systemd-tmpfiles --create",
"systemctl daemon-reload",
"systemctl reset-failed carlos-edge",
"systemctl restart carlos-edge",
"systemctl is-active carlos-edge",
"systemctl daemon-reload",
"systemctl enable carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service carlos-new-only.path",
"systemctl start carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-new-only.path",
}
at := 0
for _, want := range order {
i := strings.Index(joined[at:], want)
if i < 0 {
t.Fatalf("call order broken: %q not found after position %d in:\n%s", want, at, joined)
}
at += i + len(want)
}
for _, c := range f.calls {
if strings.HasPrefix(c, "systemctl start ") && strings.Contains(c, ".service") {
t.Fatalf("apply started a service: %s", c)
}
}
if strings.Contains(joined, "systemctl enable --now") {
t.Fatal("enable --now is never used")
}
}
func TestApplyGateSoakCatchesAFlap(t *testing.T) {
f, req := applyHappy(t)
// Edge comes up, then restarts once during the soak.
restarts := 0
orig := f.a.ops.systemctl
f.a.ops.systemctl = func(args ...string) (string, error) {
if args[0] == "show" {
restarts++
if restarts > 1 {
return "1\n2000\n", nil
}
}
return orig(args...)
}
err := f.a.Run(context.Background(), false)
if err == nil {
t.Fatal("a flapping edge passed the gate")
}
r := f.result()
if r == nil || !r.matches("stable", "v2", req.PromotedAt) || r.Outcome == systemOutcomeApplied {
t.Fatalf("result = %+v", r)
}
}
func TestApplyPrevIsRefreshedOnlyWhenTheDiskIsKnownGood(t *testing.T) {
// applied.json names v0 but the disk says v1 (a rollback-failed left a
// bad v1 in place, .prev is the good v0): .prev must be left alone.
f, _ := applyHappy(t)
os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755)
f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n"
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644)
_ = f.a.Run(context.Background(), false)
prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
if !strings.Contains(string(prev), "carlos v0") {
t.Fatal(".prev was overwritten by a binary that never passed a gate")
}
}
func TestApplyPrevUnitsComeFromPrevNotFromDisk(t *testing.T) {
f, _ := applyHappy(t)
os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v0\n"), 0o755)
f.answer["version:"+systemPrevPath(f.a.optDir)] = "carlos v0\n"
f.answer["plan:"+systemPrevPath(f.a.optDir)] = `{"enabled":["carlos-edge.service"],"inert":[]}`
f.answer["units:"+systemPrevPath(f.a.optDir)] = "carlos-edge.service carlos-v0-only.timer"
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v0", Channel: "stable", At: f.now}, 0o644)
// Inspect the journal mid-flight by failing the gate.
f.edgeUp = false
_ = f.a.Run(context.Background(), false)
// units_removed must be computed against .prev's set (v0-only.timer),
// not the on-disk v1's (old-only.timer).
found := false
for _, c := range f.calls {
if c == "run carlos.prev bootstrap -root "+filepath.Join(f.a.stagingDir(), "prev")+" -plan-json" {
found = true
}
}
if !found {
t.Fatalf("prev units not staged from carlos.prev:\n%s", strings.Join(f.calls, "\n"))
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyInstalls|TestApplyGate|TestApplyPrev' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: FAIL with errNotYetImplemented.
- [ ] Step 3: Implement the install path
Replace the install stub and add the helpers:
func (a *systemApplier) install(ctx context.Context, req systemRequest, p *release.SystemPointer, staged string) error {
target := p.Current.Version
prevOK, err := a.decidePrev(ctx)
if err != nil {
return a.record(req, fmt.Errorf("%w: %v", errSystemRefused, err))
}
if !prevOK {
return a.record(req, fmt.Errorf("%w: no known-good carlos.prev to fall back to", errSystemRefused))
}
newDir, prevDir := filepath.Join(a.stagingDir(), "new"), filepath.Join(a.stagingDir(), "prev")
newPlan, newUnits, err := a.stagePlan(ctx, staged, newDir)
if err != nil {
return a.record(req, fmt.Errorf("%w: candidate bootstrap: %v", errSystemRefused, err))
}
prevPlan, prevUnits, err := a.stagePlan(ctx, systemPrevPath(a.optDir), prevDir)
if err != nil {
return a.record(req, fmt.Errorf("%w: carlos.prev bootstrap: %v", errSystemRefused, err))
}
j := systemJournal{Phase: systemPhaseInstalling, Target: target, PrevBuild: a.prevBuild(ctx),
UnitsAdded: diff(newUnits, prevUnits), UnitsRemoved: diff(prevUnits, newUnits), StartedAt: a.ops.now().UTC()}
if err := a.journal(j); err != nil {
return err
}
fail := func(step string, err error) error {
return a.rollback(ctx, req, fmt.Sprintf("%s: %v", step, err), &j, prevPlan)
}
if _, err := a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...); err != nil {
a.ops.logf("system apply: quiesce: %v (continuing)", err)
}
if err := os.Rename(staged, systemBinaryPath(a.optDir)); err != nil {
return fail("install", err)
}
if err := os.Chown(systemBinaryPath(a.optDir), 0, 0); err != nil && !errors.Is(err, os.ErrPermission) {
a.ops.logf("system apply: chown root: %v", err)
}
j.Phase = systemPhaseInstalled
if err := a.journal(j); err != nil {
return fail("journal", err)
}
if err := a.installUnits(newDir, j.UnitsRemoved); err != nil {
return fail("units", err)
}
j.Phase = systemPhaseUnits
if err := a.journal(j); err != nil {
return fail("journal", err)
}
if err := a.restartEdge(); err != nil {
return fail("restart", err)
}
j.Phase = systemPhaseRestarted
if err := a.journal(j); err != nil {
return fail("journal", err)
}
if err := a.gate(ctx, target); err != nil {
return fail("gate", err)
}
j.Phase = systemPhaseGated
if err := a.journal(j); err != nil {
return fail("journal", err)
}
return a.finish(req, target, newPlan)
}
// finish is steps 11–12, and what recovery runs forward from phase gated.
func (a *systemApplier) finish(req systemRequest, target string, plan bootstrapPlan) error {
if err := a.arm(plan); err != nil {
a.ops.logf("system apply: arm: %v", err)
}
if err := writeJSONDurable(a.file("applied.json"), systemApplied{Version: target, Channel: a.channel, At: a.ops.now().UTC()}, 0o644); err != nil {
return err
}
if err := a.writeResult(req, systemOutcomeApplied, ""); err != nil {
return err
}
os.Remove(a.file("journal.json"))
return syncDirBestEffort(a.rootDir())
}
// decidePrev: refresh carlos.prev from the on-disk binary only when that
// binary is known good — applied.json names it, or nothing has ever been
// applied and there is no .prev (a hand-rolled box whose running binary
// passed the script's own post-condition). Otherwise leave .prev alone.
func (a *systemApplier) decidePrev(ctx context.Context) (bool, error) {
prev := systemPrevPath(a.optDir)
_, prevErr := os.Stat(prev)
applied, err := readSystemApplied(a.optDir)
if err != nil {
return false, err
}
disk := a.onDiskBuild(ctx)
known := (applied != nil && applied.Version == disk && disk != "") || (applied == nil && prevErr != nil)
if !known {
return prevErr == nil, nil
}
if err := a.journal(systemJournal{Phase: systemPhaseBackingUp, Target: "", PrevBuild: disk, StartedAt: a.ops.now().UTC()}); err != nil {
return false, err
}
if err := copyFileDurable(systemBinaryPath(a.optDir), prev, 0o755); err != nil {
return false, err
}
os.Remove(a.file("journal.json"))
return true, nil
}
func (a *systemApplier) prevBuild(ctx context.Context) string {
out, err := a.ops.run(ctx, systemVersionWait, systemPrevPath(a.optDir), "version")
if err != nil {
return ""
}
return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(out), "carlos "))
}
// stagePlan runs `<binary> bootstrap -root <dir> -plan-json` and returns
// that binary's arming plan and the unit files it staged.
func (a *systemApplier) stagePlan(ctx context.Context, binary, dir string) (bootstrapPlan, []string, error) {
os.RemoveAll(dir)
if err := os.MkdirAll(dir, 0o700); err != nil {
return bootstrapPlan{}, nil, err
}
out, err := a.ops.run(ctx, systemBootstrapWait, binary, "bootstrap", "-root", dir, "-plan-json")
if err != nil {
return bootstrapPlan{}, nil, err
}
var plan bootstrapPlan
if err := json.Unmarshal([]byte(out), &plan); err != nil {
return bootstrapPlan{}, nil, fmt.Errorf("plan: %w", err)
}
entries, err := os.ReadDir(filepath.Join(dir, "etc", "systemd", "system"))
if err != nil {
return bootstrapPlan{}, nil, err
}
var units []string
for _, e := range entries {
if !e.IsDir() {
units = append(units, e.Name())
}
}
return plan, units, nil
}
func (a *systemApplier) journal(j systemJournal) error {
return writeJSONDurable(a.file("journal.json"), j, 0o600)
}
// installUnits: remove what the target build dropped, copy its units in,
// then sysusers BEFORE tmpfiles, then daemon-reload. Never the staging
// root's data/.
func (a *systemApplier) installUnits(stagedRoot string, remove []string) error {
unitDir := filepath.Join(a.etcDir, "systemd", "system")
for _, u := range remove {
if _, err := a.ops.systemctl("disable", "--now", u); err != nil {
a.ops.logf("system apply: disable %s: %v", u, err)
}
os.Remove(filepath.Join(unitDir, u))
}
entries, err := os.ReadDir(filepath.Join(stagedRoot, "etc", "systemd", "system"))
if err != nil {
return err
}
for _, e := range entries {
if e.IsDir() {
continue
}
if err := copyFileDurable(filepath.Join(stagedRoot, "etc", "systemd", "system", e.Name()), filepath.Join(unitDir, e.Name()), 0o644); err != nil {
return err
}
}
sys := filepath.Join(stagedRoot, "etc", "sysusers.d", "carlos.conf")
if _, err := os.Stat(sys); err == nil {
if err := os.MkdirAll(filepath.Join(a.etcDir, "sysusers.d"), 0o755); err != nil {
return err
}
dst := filepath.Join(a.etcDir, "sysusers.d", "carlos.conf")
if err := copyFileDurable(sys, dst, 0o644); err != nil {
return err
}
if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-sysusers", dst); err != nil {
return fmt.Errorf("systemd-sysusers: %w", err)
}
}
tmp := filepath.Join(stagedRoot, "etc", "tmpfiles.d", "carlos.conf")
if err := os.MkdirAll(filepath.Join(a.etcDir, "tmpfiles.d"), 0o755); err != nil {
return err
}
dst := filepath.Join(a.etcDir, "tmpfiles.d", "carlos.conf")
if err := copyFileDurable(tmp, dst, 0o644); err != nil {
return err
}
if _, err := a.ops.run(context.Background(), systemVersionWait, "systemd-tmpfiles", "--create", dst); err != nil {
a.ops.logf("system apply: systemd-tmpfiles: %v (continuing, as the script does)", err)
}
_, err = a.ops.systemctl("daemon-reload")
return err
}
// restartEdge: reset-failed FIRST — a start-limited unit refuses restart —
// then restart.
func (a *systemApplier) restartEdge() error {
a.ops.systemctl("reset-failed", systemEdgeUnit)
_, err := a.ops.systemctl("restart", systemEdgeUnit)
return err
}
// gate: within gateWait the edge is active, answers on loopback, and the
// installed binary reports the target; then a soak during which neither
// NRestarts nor ActiveEnterTimestampMonotonic moves.
func (a *systemApplier) gate(ctx context.Context, target string) error {
wait, soak := a.gateWait, a.soak
deadline := a.ops.now().Add(wait)
var lastErr error
for {
lastErr = a.gateOnce(ctx, target)
if lastErr == nil {
break
}
if !a.ops.now().Before(deadline) {
return lastErr
}
a.ops.sleep(time.Second)
}
before, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit)
if err != nil {
return err
}
a.ops.sleep(soak)
after, err := a.ops.systemctl("show", "-p", "NRestarts", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit)
if err != nil {
return err
}
if strings.TrimSpace(before) != strings.TrimSpace(after) {
return fmt.Errorf("edge restarted during the soak (%s → %s)", strings.ReplaceAll(strings.TrimSpace(before), "\n", "/"), strings.ReplaceAll(strings.TrimSpace(after), "\n", "/"))
}
if err := a.gateOnce(ctx, target); err != nil {
return err
}
return nil
}
func (a *systemApplier) gateOnce(ctx context.Context, target string) error {
if out, err := a.ops.systemctl("is-active", systemEdgeUnit); err != nil || strings.TrimSpace(out) != "active" {
return fmt.Errorf("%s is %s", systemEdgeUnit, strings.TrimSpace(out))
}
if err := a.ops.probeEdge(); err != nil {
return fmt.Errorf("edge probe: %w", err)
}
if got := a.onDiskBuild(ctx); got != target {
return fmt.Errorf("installed binary reports %q, not %s", got, target)
}
return nil
}
// arm is systemctlPlan verbatim: enable all, start timers and paths, never
// a service — which is why apply can call it from inside itself.
func (a *systemApplier) arm(plan bootstrapPlan) error {
for _, args := range systemctlPlan(plan.Enabled) {
if _, err := a.ops.systemctl(args...); err != nil {
return fmt.Errorf("systemctl %s: %w", strings.Join(args, " "), err)
}
}
return nil
}
func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
return errors.New("rollback not implemented: " + reason) // Task 8c
}
func diff(have, drop []string) []string {
skip := map[string]bool{}
for _, d := range drop {
skip[d] = true
}
var out []string
for _, h := range have {
if !skip[h] {
out = append(out, h)
}
}
return out
}
// copyFileDurable is writeFileDurable for a file too large to hold: temp,
// fsync, rename, directory fsync.
func copyFileDurable(src, dst string, mode os.FileMode) (err error) {
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
tmp := dst + ".tmp"
out, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, mode)
if err != nil {
return err
}
defer func() {
if err != nil {
out.Close()
os.Remove(tmp)
}
}()
if _, err = io.Copy(out, in); err != nil {
return err
}
if err = out.Chmod(mode); err != nil {
return err
}
if err = out.Sync(); err != nil {
return err
}
if err = out.Close(); err != nil {
return err
}
if err = os.Rename(tmp, dst); err != nil {
return err
}
return syncDirBestEffort(filepath.Dir(dst))
}
func syncDirBestEffort(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}The test for the flap expects TestApplyGateSoakCatchesAFlap to end in a rollback; with the 8c stub it ends in an error from rollback, and result.json is not yet written by the stub. Make the stub write result.json with outcome: rolled-back and reason, then return the error — that is the 8c contract in miniature and keeps this task's tests honest:
func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
_ = a.writeResult(req, systemOutcomeRolledBack, reason)
return errors.New("rollback not implemented: " + reason)
}- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApply' -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: PASS. TestApplyInstallsGatesArmsAndRecords checks the exact systemctl order; if a call is missing, fix the code, not the expected list, unless the spec says otherwise.
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
host: carlos system apply — install, gate, arm, record
.prev refreshed only when the disk is known good; both unit sets
staged from the binaries that own them; quiesce before the rename;
sysusers before tmpfiles; reset-failed before restart; a 15 s soak on
NRestarts; arming after the gate via systemctlPlan.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 8c: carlos system apply — rollback, recovery, --recover, the real sysOps
Files:
- Modify:
internal/host/systemapply.go(replacerollbackandrecoverIfNeeded; addRunSystemApplyandproductionSysOps) - Test:
internal/host/systemapply_test.go(add)
Interfaces:
- Produces: ``
go func RunSystemApply(args []string) // flags: -recover; exits 1 on a non-nil Run error func productionSysOps() sysOps func (a *systemApplier) rollback(ctx, req, reason, j *systemJournal, prevPlan bootstrapPlan) error func (a *systemApplier) recoverIfNeeded(ctx) error func (a *systemApplier) restoreFiles(ctx, j *systemJournal) (bootstrapPlan, error) // files only: binary, units, sysusers, tmpfiles, daemon-reload, enable``
- [ ] Step 1: Write the failing tests
func TestApplyRollsBackWhenTheGateFails(t *testing.T) {
f, req := applyHappy(t)
f.edgeUp = false
err := f.a.Run(context.Background(), false)
if err == nil {
t.Fatal("a failed gate must exit non-zero")
}
r := f.result()
if r == nil || r.Outcome != systemOutcomeRolledBack || !r.matches("stable", "v2", req.PromotedAt) || !strings.Contains(r.Reason, "gate") {
t.Fatalf("result = %+v", r)
}
bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
if !strings.Contains(string(bin), "carlos v1") {
t.Fatal("binary not restored")
}
if _, err := os.Stat(systemPrevPath(f.a.optDir)); err != nil {
t.Fatal(".prev consumed by the rollback")
}
units := filepath.Join(f.a.etcDir, "systemd", "system")
if _, err := os.Stat(filepath.Join(units, "carlos-new-only.path")); err == nil {
t.Error("added unit survived the rollback")
}
if _, err := os.Stat(filepath.Join(units, "carlos-old-only.timer")); err != nil {
t.Error("removed unit not restored")
}
if ap, _ := readSystemApplied(f.a.optDir); ap != nil {
t.Fatalf("applied.json written by a rollback: %+v", ap)
}
if j, _ := readSystemJournal(f.a.optDir); j != nil {
t.Fatal("journal left open after rollback")
}
joined := strings.Join(f.calls, "\n")
if !strings.Contains(joined, "systemctl disable --now carlos-new-only.path") {
t.Error("added unit not disabled")
}
// reset-failed precedes the rollback's restart too, and the previous
// plan is armed afterwards.
i := strings.LastIndex(joined, "systemctl reset-failed carlos-edge")
k := strings.LastIndex(joined, "systemctl restart carlos-edge")
m := strings.LastIndex(joined, "systemctl start carlos-adopt.timer")
if !(i < k && k < m) {
t.Fatalf("rollback order wrong:\n%s", joined)
}
}
func TestApplyRollbackFailedLeavesPrevAndAppliedAlone(t *testing.T) {
f, req := applyHappy(t)
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "applied.json"), systemApplied{Version: "v1", Channel: "stable", At: f.now}, 0o644)
f.edgeUp = false // never comes up, on either binary
if err := f.a.Run(context.Background(), false); err == nil {
t.Fatal("must exit non-zero")
}
r := f.result()
if r == nil || r.Outcome != systemOutcomeRollbackFailed || !r.matches("stable", "v2", req.PromotedAt) {
t.Fatalf("result = %+v", r)
}
ap, _ := readSystemApplied(f.a.optDir)
if ap == nil || ap.Version != "v1" {
t.Fatalf("applied = %+v, must still name v1", ap)
}
prev, _ := os.ReadFile(systemPrevPath(f.a.optDir))
if !strings.Contains(string(prev), "carlos v1") {
t.Fatal(".prev damaged")
}
// A second apply of a NEW version must not refresh .prev from the
// unproven disk binary.
f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos v1\n" // rollback restored v1 on disk
os.Remove(systemRequestPath(f.a.dataDir))
f.promote("v3", []byte("#!/bin/sh\necho carlos v3\n"), false)
f.answer["version:"+filepath.Join(f.a.stagingDir(), "carlos")] = "carlos v3\n"
f.answer["plan:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["plan:"+systemPrevPath(f.a.optDir)]
f.answer["units:"+filepath.Join(f.a.stagingDir(), "carlos")] = f.answer["units:"+systemPrevPath(f.a.optDir)]
_ = f.a.Run(context.Background(), false)
prev2, _ := os.ReadFile(systemPrevPath(f.a.optDir))
if string(prev2) != string(prev) {
t.Fatal(".prev refreshed after rollback-failed")
}
}
func TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime(t *testing.T) {
for _, phase := range []string{systemPhaseInstalling, systemPhaseInstalled, systemPhaseUnits, systemPhaseRestarted} {
t.Run(phase, func(t *testing.T) {
f, _ := applyHappy(t)
// Simulate: v2 installed, journal open at phase, box rebooted.
os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755)
os.WriteFile(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path"), []byte("x"), 0o644)
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
systemJournal{Phase: phase, Target: "v2", PrevBuild: "v1", UnitsAdded: []string{"carlos-new-only.path"}, UnitsRemoved: []string{"carlos-old-only.timer"}, StartedAt: f.now}, 0o600)
f.edgeStartedThisBoot = false
if err := f.a.Run(context.Background(), true); err != nil {
t.Fatal(err)
}
bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
if !strings.Contains(string(bin), "carlos v1") {
t.Fatal("binary not restored")
}
if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-new-only.path")); err == nil {
t.Error("added unit survived")
}
if _, err := os.Stat(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-old-only.timer")); err != nil {
t.Error("removed unit not restored")
}
joined := strings.Join(f.calls, "\n")
if strings.Contains(joined, "systemctl restart carlos-edge") || strings.Contains(joined, "systemctl start ") {
t.Fatalf("boot recovery restarted or started something:\n%s", joined)
}
if !strings.Contains(joined, "systemctl enable carlos-edge.service") {
t.Fatal("previous plan not enabled for the boot")
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeRolledBack || !strings.Contains(r.Reason, phase) {
t.Fatalf("result = %+v", r)
}
if j, _ := readSystemJournal(f.a.optDir); j != nil {
t.Fatal("journal left open")
}
})
}
// Runtime (OnFailure=): same, plus a non-blocking restart and the
// previous plan's timers/paths started.
f, _ := applyHappy(t)
os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
systemJournal{Phase: systemPhaseUnits, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
f.edgeStartedThisBoot = true
if err := f.a.Run(context.Background(), true); err != nil {
t.Fatal(err)
}
joined := strings.Join(f.calls, "\n")
for _, want := range []string{"systemctl stop carlos-adopt.timer", "systemctl reset-failed carlos-edge", "systemctl restart --no-block carlos-edge", "systemctl start carlos-adopt.timer"} {
if !strings.Contains(joined, want) {
t.Errorf("runtime recovery lacks %q:\n%s", want, joined)
}
}
}
func TestRecoveryFinishesForwardFromGated(t *testing.T) {
f, req := applyHappy(t)
os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v2\n"), 0o755)
writeJSONDurable(systemRequestPath(f.a.dataDir), req, 0o644)
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
systemJournal{Phase: systemPhaseGated, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
if err := f.a.Run(context.Background(), true); err != nil {
t.Fatal(err)
}
bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir))
if !strings.Contains(string(bin), "carlos v2") {
t.Fatal("a gated apply was undone")
}
ap, _ := readSystemApplied(f.a.optDir)
if ap == nil || ap.Version != "v2" {
t.Fatalf("applied = %+v", ap)
}
if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied {
t.Fatalf("result = %+v", r)
}
}
func TestRecoveryRemovesTheJournalLast(t *testing.T) {
// A restart that fails mid-recovery leaves the journal for the next run.
f, _ := applyHappy(t)
os.WriteFile(systemPrevPath(f.a.optDir), []byte("#!/bin/sh\necho carlos v1\n"), 0o755)
writeJSONDurable(filepath.Join(systemRootDir(f.a.optDir), "journal.json"),
systemJournal{Phase: systemPhaseInstalled, Target: "v2", PrevBuild: "v1", StartedAt: f.now}, 0o600)
orig := f.a.ops.systemctl
f.a.ops.systemctl = func(args ...string) (string, error) {
if args[0] == "daemon-reload" {
return "", errors.New("systemd busy")
}
return orig(args...)
}
if err := f.a.Run(context.Background(), true); err == nil {
t.Fatal("a failed recovery must exit non-zero")
}
if j, _ := readSystemJournal(f.a.optDir); j == nil {
t.Fatal("journal removed although recovery did not complete")
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run 'TestApplyRolls|TestApplyRollback|TestRecovery' -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: FAIL ("rollback not implemented", "recovery not implemented").
- [ ] Step 3: Implement
Replace rollback and recoverIfNeeded; add restoreFiles, RunSystemApply, productionSysOps:
// rollback is step 13: back to carlos.prev and its units, restart, gate,
// arm the previous plan. applied.json is untouched — the .prev build is
// what passed a gate, and it is what applied.json already names.
func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason string, j *systemJournal, prevPlan bootstrapPlan) error {
a.ops.logf("system apply: %s %s: rolling back: %s", a.channel, j.Target, reason)
plan, err := a.restoreFiles(ctx, j)
if err != nil {
_ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restore: "+err.Error())
os.Remove(a.file("journal.json"))
return fmt.Errorf("system apply: rollback of %s failed: %v", j.Target, err)
}
if err := a.restartEdge(); err != nil {
_ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; restart: "+err.Error())
os.Remove(a.file("journal.json"))
return fmt.Errorf("system apply: rollback restart failed: %v", err)
}
if err := a.gate(ctx, j.PrevBuild); err != nil {
_ = a.writeResult(req, systemOutcomeRollbackFailed, reason+"; previous build failed its gate too: "+err.Error())
os.Remove(a.file("journal.json"))
return fmt.Errorf("system apply: rollback gate failed: %v", err)
}
if err := a.arm(plan); err != nil {
a.ops.logf("system apply: arm previous plan: %v", err)
}
if err := a.writeResult(req, systemOutcomeRolledBack, reason); err != nil {
return err
}
os.Remove(a.file("journal.json"))
return fmt.Errorf("system apply: %s rolled back: %s", j.Target, reason)
}
// restoreFiles is the files-only half shared by rollback and recovery:
// carlos.prev over carlos (keeping .prev), units_added disabled and
// removed, the previous unit set copied back (bringing units_removed
// back), sysusers, tmpfiles, daemon-reload, and `enable` of the previous
// plan so it comes up at boot. No restart, no start.
func (a *systemApplier) restoreFiles(ctx context.Context, j *systemJournal) (bootstrapPlan, error) {
prev := systemPrevPath(a.optDir)
if _, err := os.Stat(prev); err != nil {
return bootstrapPlan{}, fmt.Errorf("no carlos.prev: %w", err)
}
a.ops.systemctl(append([]string{"stop"}, systemQuiesceUnits...)...)
if err := copyFileDurable(prev, systemBinaryPath(a.optDir), 0o755); err != nil {
return bootstrapPlan{}, err
}
prevDir := filepath.Join(a.stagingDir(), "prev")
plan, _, err := a.stagePlan(ctx, prev, prevDir)
if err != nil {
return bootstrapPlan{}, fmt.Errorf("carlos.prev bootstrap: %w", err)
}
if err := a.installUnits(prevDir, j.UnitsAdded); err != nil {
return bootstrapPlan{}, err
}
if _, err := a.ops.systemctl(append([]string{"enable"}, plan.Enabled...)...); err != nil {
return bootstrapPlan{}, err
}
return plan, nil
}
// recoverIfNeeded: an open journal means a previous run was cut off.
// Before gated → back, files only, restart only if the edge has started
// this boot; gated → forward. The journal is removed LAST.
func (a *systemApplier) recoverIfNeeded(ctx context.Context) error {
j, err := readSystemJournal(a.optDir)
if err != nil || j == nil {
return err
}
req, _ := a.readRequest()
if req.Version == "" {
req = systemRequest{Version: j.Target, PromotedAt: j.StartedAt}
}
a.ops.logf("system apply: recovering an apply of %s interrupted at %s", j.Target, j.Phase)
switch j.Phase {
case systemPhaseGated:
plan, _, err := a.stagePlan(ctx, systemBinaryPath(a.optDir), filepath.Join(a.stagingDir(), "new"))
if err != nil {
return fmt.Errorf("system apply: recovery: %w", err)
}
return a.finish(req, j.Target, plan)
case systemPhaseBackingUp:
os.Remove(systemPrevPath(a.optDir) + ".tmp")
os.Remove(a.file("journal.json"))
return nil
}
plan, err := a.restoreFiles(ctx, j)
if err != nil {
return fmt.Errorf("system apply: recovery: %w", err) // journal kept: the next run recovers again
}
if a.ops.edgeStartedThisBoot() {
a.ops.systemctl("reset-failed", systemEdgeUnit)
if _, err := a.ops.systemctl("restart", "--no-block", systemEdgeUnit); err != nil {
return fmt.Errorf("system apply: recovery: restart: %w", err)
}
var armed []string
for _, u := range plan.Enabled {
if strings.HasSuffix(u, ".timer") || strings.HasSuffix(u, ".path") {
armed = append(armed, u)
}
}
if len(armed) > 0 {
a.ops.systemctl(append([]string{"start"}, armed...)...)
}
}
if err := a.writeResult(req, systemOutcomeRolledBack, "interrupted at "+j.Phase); err != nil {
return err
}
os.Remove(a.file("journal.json"))
return syncDirBestEffort(a.rootDir())
}
// RunSystemApply is `carlos system apply [--recover]`: the root oneshot.
func RunSystemApply(args []string) {
fs := flag.NewFlagSet("system apply", flag.ExitOnError)
recover := fs.Bool("recover", false, "recover an interrupted apply (files only) and exit: the boot and OnFailure unit")
fs.Usage = cliflag.Usage(fs, "")
_ = fs.Parse(args)
if os.Geteuid() != 0 {
log.Fatal("carlos system apply is the root oneshot behind carlos-system-update.service; it is not for operators")
}
ctx := context.Background()
store, err := openDeployStore(ctx)
if err != nil {
log.Fatalf("system apply: %v", err)
}
pub, err := release.ParsePublicKey(os.Getenv("CARLOS_RELEASE_PUBKEY"), "CARLOS_RELEASE_PUBKEY")
if err != nil {
log.Fatalf("system apply: %v", err)
}
a := &systemApplier{
ops: productionSysOps(), store: store, pub: pub, channel: systemChannel(os.Getenv),
optDir: release.OptDir(), dataDir: dataDir(), etcDir: "/etc", build: version.Build,
platform: runtime.GOOS + "-" + runtime.GOARCH, gateWait: systemGateWait, soak: systemGateSoak,
}
if err := a.Run(ctx, *recover); err != nil {
log.Fatalf("%v", err)
}
}
func productionSysOps() sysOps {
return sysOps{
systemctl: func(args ...string) (string, error) {
out, err := exec.Command("systemctl", args...).CombinedOutput()
if err != nil {
return string(out), fmt.Errorf("systemctl %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out)))
}
return string(out), nil
},
run: func(ctx context.Context, timeout time.Duration, name string, args ...string) (string, error) {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
out, err := exec.CommandContext(ctx, name, args...).Output()
if err != nil {
return string(out), fmt.Errorf("%s %s: %w", name, strings.Join(args, " "), err)
}
return string(out), nil
},
now: time.Now,
sleep: time.Sleep,
probeEdge: func() error {
// Any HTTP answer on loopback proves the process serves; a box
// with no route yet must still pass (spec §6 step 10).
c := &http.Client{Timeout: 3 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
resp, err := c.Get("http://127.0.0.1/")
if err != nil {
return err
}
resp.Body.Close()
return nil
},
edgeStartedThisBoot: func() bool {
out, err := exec.Command("systemctl", "show", "-p", "ActiveEnterTimestampMonotonic", "--value", systemEdgeUnit).Output()
return err == nil && strings.TrimSpace(string(out)) != "" && strings.TrimSpace(string(out)) != "0"
},
logf: log.Printf,
}
}Imports to add: "flag", "log", "net/http", "os/exec", "github.com/carlosframework/platform/internal/cliflag", "github.com/carlosframework/platform/internal/version". In the fixture, set f.a.platform = "linux-arm64" (from 8a). The edge's plain-HTTP listener: confirm the port with grep -n 'ListenAndServe\|":80"\|Listen(' internal/edge/edge.go | head; if the edge only listens on 443, probe https://127.0.0.1/ with InsecureSkipVerify and a ServerName of localhost instead, and say which in the commit.
- [ ] Step 4: Run the whole host package
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: ok.
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemapply.go internal/host/systemapply_test.go && git commit -q -F - <<'EOF'
host: carlos system apply — rollback, recovery, and the real box
Rollback restores carlos.prev and its unit set, gates it, arms the
previous plan, and leaves applied.json alone. Recovery is files-only
and goes forward from phase gated; it restarts the edge only when the
edge has started this boot, and removes the journal last.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 9: carlos system dispatch and the apply verb
Files:
- Create:
system.go(package main),system_test.go - Modify:
main.go(theswitch cmdat:30-134,usage()at:139-233),update.go(noHintVerbsat:76-85)
Interfaces:
- Consumes:
host.RunSystemApply(Task 8c),main.godispatch pattern (case "ops": ops.Run(args)),usage()section "On the host". - Produces: ``
go func runSystem(args []string) // carlos system <verb> | carlos system router <verb> func systemVerb(args []string) (verb string, rest []string, component string) // strips an optional leading "router"`Verbs in this task:apply→host.RunSystemApply(rest);update/status/rollback→log.Fatal("carlos system " + verb + " arrives in a later task")placeholders that Task 12 replaces;help`/none → usage, exit 2.
- [ ] Step 1: Write the failing test
package main
import "testing"
func TestSystemVerbAcceptsTheComponentForm(t *testing.T) {
cases := []struct {
in []string
verb, comp string
rest int
}{
{[]string{"update", "--channel", "canary"}, "update", "router", 2},
{[]string{"router", "update"}, "update", "router", 0},
{[]string{"router", "status"}, "status", "router", 0},
{[]string{"apply", "--recover"}, "apply", "router", 1},
{[]string{}, "", "router", 0},
{[]string{"router"}, "", "router", 0},
}
for _, c := range cases {
verb, rest, comp := systemVerb(c.in)
if verb != c.verb || comp != c.comp || len(rest) != c.rest {
t.Errorf("systemVerb(%v) = %q %v %q", c.in, verb, rest, comp)
}
}
if verb, _, _ := systemVerb([]string{"console", "update"}); verb != "" {
t.Errorf("an unknown component must not be a verb: got %q", verb)
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test . -run TestSystemVerb -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc Expected: undefined systemVerb.
- [ ] Step 3: Implement
system.go
package main
import (
"fmt"
"log"
"os"
"github.com/carlosframework/platform/internal/host"
)
// carlos system: what is not an app (spec §0, the console-as-an-app
// dividing line). One component today, the router — the substrate binary.
// `carlos system router update` is the same verb scoped to it, so the
// command Paul named works and a future split needs no new verb.
const systemComponentRouter = "router"
func systemVerb(args []string) (verb string, rest []string, component string) {
component = systemComponentRouter
if len(args) > 0 && args[0] == systemComponentRouter {
args = args[1:]
}
if len(args) == 0 {
return "", nil, component
}
switch args[0] {
case "update", "status", "rollback", "apply", "help", "-h", "--help":
return args[0], args[1:], component
}
return "", args, component
}
func runSystem(args []string) {
verb, rest, _ := systemVerb(args)
switch verb {
case "apply":
host.RunSystemApply(rest)
case "update":
runSystemUpdate(rest)
case "status":
runSystemStatus(rest)
case "rollback":
runSystemRollback(rest)
case "help", "-h", "--help":
systemUsage()
default:
if len(args) > 0 && verb == "" {
fmt.Fprintf(os.Stderr, "system: unknown verb %q\n\n", args[0])
}
systemUsage()
os.Exit(2)
}
}
func systemUsage() {
fmt.Fprint(os.Stderr, `usage: carlos system <verb> [flags]
carlos system router <verb> [flags]
update promote a platform release to a channel; every box on it rolls itself
status every box: platform, channel, running build, channel target, last apply outcome
rollback point a channel back at the release before this one
apply (root oneshot behind carlos-system-update.service; not for operators)
The platform is not an app: the console is updated with carlos deploy --app console.
`)
}
// Placeholders until Task 12.
func runSystemUpdate(args []string) { log.Fatal("carlos system update: not yet wired to the console") }
func runSystemStatus(args []string) { log.Fatal("carlos system status: not yet wired to the console") }
func runSystemRollback(args []string) { log.Fatal("carlos system rollback: not yet wired to the console") }main.go: add case "system": runSystem(args) after the ops case; in usage() under "On the host" add system platform self-update: update | status | rollback (apply is the root oneshot); in update.go noHintVerbs add "system": true.
- [ ] Step 4: Run and check the binary
Run: export TMPDIR=/tmp/c; out=$(go test . -run 'TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go build -o /tmp/c/carlos . && /tmp/c/carlos system help 2>&1 | head -3; /tmp/c/carlos system apply 2>&1 | head -1 Expected: PASS; usage printed; apply as non-root prints "not for operators".
- [ ] Step 5: Commit
gofmt -l . && git add system.go system_test.go main.go update.go && git commit -q -F - <<'EOF'
cli: carlos system, with apply wired to the root oneshot
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 10: The public release feed (internal/publicrelease)
Files:
- Create:
internal/publicrelease/feed.go,internal/publicrelease/feed_test.go - Modify:
update.go(fetchLatest:100-119,selfReplace:295-327,fetchUpdateFile:358-369)
Interfaces:
- Consumes: the package vars
updateLatestURL,updateDownloadBase,cliUpdatePubKeyinupdate.go(kept, sinceupdate_test.gorepoints them). - Produces: ``
go package publicrelease type Feed struct { LatestURL string // https://apt.carlosframework.com/latest.txt DownloadBase string // https://github.com/carlosframework/releases/releases/download (tag-pinned; never latest/download) PubKey string // base64 ed25519 public key that signs SHA256SUMS.txt Client *http.Client // nil = a 5-minute client } func Default() Feed var TagRe = regexp.MustCompile(^v[0-9]+\.[0-9]+\.[0-9]+$) func (f Feed) Latest(ctx context.Context) (string, error) func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) // asset → sha, signature verified func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) (data []byte, sha256hex string, err error) // verified against Sums func AssetName(platform string) string // "carlos-" + platform``
- [ ] Step 1: Write the failing tests
package publicrelease
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func testFeed(t *testing.T, sign bool, assets map[string][]byte) (Feed, string) {
t.Helper()
pub, priv, _ := ed25519.GenerateKey(nil)
var sums strings.Builder
for name, body := range assets {
s := sha256.Sum256(body)
fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name)
}
sig := ed25519.Sign(priv, []byte(sums.String()))
mux := http.NewServeMux()
mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") })
mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) })
mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) {
if sign {
fmt.Fprintln(w, base64.StdEncoding.EncodeToString(sig))
} else {
fmt.Fprintln(w, base64.StdEncoding.EncodeToString(make([]byte, ed25519.SignatureSize)))
}
})
for name, body := range assets {
body := body
mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) })
}
ts := httptest.NewServer(mux)
t.Cleanup(ts.Close)
return Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)}, ts.URL
}
func TestFeedLatestAndVerifiedAsset(t *testing.T) {
body := []byte("ELF arm64")
f, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body, "carlos-linux-amd64": []byte("ELF amd64")})
ctx := context.Background()
tag, err := f.Latest(ctx)
if err != nil || tag != "v0.18.0" {
t.Fatalf("latest = %q %v", tag, err)
}
data, sha, err := f.Asset(ctx, tag, AssetName("linux-arm64"), 1<<20)
if err != nil || string(data) != string(body) {
t.Fatalf("asset: %v", err)
}
s := sha256.Sum256(body)
if sha != hex.EncodeToString(s[:]) {
t.Fatalf("sha = %s", sha)
}
if _, _, err := f.Asset(ctx, tag, "carlos-windows-amd64.zip", 1<<20); err == nil {
t.Fatal("an asset absent from the sums was accepted")
}
}
func TestFeedRefusesABadSignatureAndATamperedAsset(t *testing.T) {
body := []byte("ELF")
f, _ := testFeed(t, false, map[string][]byte{"carlos-linux-arm64": body})
if _, err := f.Sums(context.Background(), "v0.18.0"); err == nil || !strings.Contains(err.Error(), "signature") {
t.Fatalf("bad signature: %v", err)
}
// Signed sums, but the served asset differs from the sum.
f2, _ := testFeed(t, true, map[string][]byte{"carlos-linux-arm64": body})
f2.DownloadBase = f2.DownloadBase // same server
pub, _ := base64.StdEncoding.DecodeString(f2.PubKey)
_ = pub
// Simulate tampering by asking for a limit smaller than the asset: the
// truncated download cannot match its sum.
if _, _, err := f2.Asset(context.Background(), "v0.18.0", "carlos-linux-arm64", 1); err == nil {
t.Fatal("a truncated asset matched its sum")
}
}
func TestFeedLatestRequiresATag(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "main") }))
defer ts.Close()
f := Feed{LatestURL: ts.URL, DownloadBase: ts.URL, PubKey: ""}
if _, err := f.Latest(context.Background()); err == nil {
t.Fatal("accepted a non-tag latest")
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc Expected: package does not exist.
- [ ] Step 3: Implement
feed.go
// Package publicrelease reads the vendor's public signed release feed: the
// latest tag, the signed checksum list, and a verified asset. Two callers
// share it — `carlos update` on a workstation and the console's `carlos
// system update` promote — so the verification chain is written once.
package publicrelease
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"regexp"
"strings"
"time"
)
type Feed struct {
LatestURL string
DownloadBase string // tag-pinned: <DownloadBase>/<tag>/<file>; never latest/download
PubKey string // base64 ed25519 public key over the raw SHA256SUMS.txt bytes
Client *http.Client
}
// Default is the vendor feed. The strings are duplicated in update.go's
// package vars so update_test.go can repoint them; keep the two in step.
func Default() Feed {
return Feed{
LatestURL: "https://apt.carlosframework.com/latest.txt",
DownloadBase: "https://github.com/carlosframework/releases/releases/download",
PubKey: "R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=",
}
}
var TagRe = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+$`)
func AssetName(platform string) string { return "carlos-" + platform }
func (f Feed) client() *http.Client {
if f.Client != nil {
return f.Client
}
return &http.Client{Timeout: 5 * time.Minute}
}
func (f Feed) get(ctx context.Context, url string, limit int64) ([]byte, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
res, err := f.client().Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%s: status %d", url, res.StatusCode)
}
return io.ReadAll(io.LimitReader(res.Body, limit))
}
// Latest is the newest release tag, and only a tag.
func (f Feed) Latest(ctx context.Context) (string, error) {
b, err := f.get(ctx, f.LatestURL, 128)
if err != nil {
return "", err
}
tag := strings.TrimSpace(string(b))
if !TagRe.MatchString(tag) {
return "", fmt.Errorf("%s: %q is not a release tag", f.LatestURL, tag)
}
return tag, nil
}
// Sums fetches SHA256SUMS.txt and its .sig for tag, verifies the signature
// with PubKey, and returns asset → sha256 hex.
func (f Feed) Sums(ctx context.Context, tag string) (map[string]string, error) {
base := f.DownloadBase + "/" + tag + "/"
sums, err := f.get(ctx, base+"SHA256SUMS.txt", 1<<20)
if err != nil {
return nil, err
}
sig, err := f.get(ctx, base+"SHA256SUMS.txt.sig", 4096)
if err != nil {
return nil, fmt.Errorf("%v (a release without a signed checksum list predates self-update)", err)
}
pub, err := base64.StdEncoding.DecodeString(f.PubKey)
if err != nil || len(pub) != ed25519.PublicKeySize {
return nil, errors.New("release feed public key is invalid")
}
rawSig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(string(sig)))
if err != nil || !ed25519.Verify(ed25519.PublicKey(pub), sums, rawSig) {
return nil, errors.New("SHA256SUMS.txt signature does not verify — refusing the release")
}
out := map[string]string{}
for _, line := range strings.Split(string(sums), "\n") {
fields := strings.Fields(line)
if len(fields) == 2 {
out[fields[1]] = fields[0]
}
}
return out, nil
}
// Asset fetches one named asset for tag and refuses it unless its sha256
// matches the signed checksum list.
func (f Feed) Asset(ctx context.Context, tag, asset string, limit int64) ([]byte, string, error) {
sums, err := f.Sums(ctx, tag)
if err != nil {
return nil, "", err
}
want, ok := sums[asset]
if !ok {
return nil, "", fmt.Errorf("no %s in the release's checksum list", asset)
}
data, err := f.get(ctx, f.DownloadBase+"/"+tag+"/"+asset, limit)
if err != nil {
return nil, "", err
}
s := sha256.Sum256(data)
got := hex.EncodeToString(s[:])
if got != want {
return nil, "", fmt.Errorf("%s does not match its signed checksum — refusing it", asset)
}
return data, got, nil
}update.go: replace the bodies of fetchLatest and the fetch-and-verify half of selfReplace (:296-327) with calls through a feed built from the package vars, so tests that repoint the vars still work:
func updateFeed() publicrelease.Feed {
return publicrelease.Feed{LatestURL: updateLatestURL, DownloadBase: updateDownloadBase, PubKey: cliUpdatePubKey}
}fetchLatest(timeout) → updateFeed() with Client: &http.Client{Timeout: timeout} and .Latest(ctx); in selfReplace, binData, _, err := updateFeed().Asset(ctx, latest, asset, 256<<20), keeping the workstation-only half (temp file, lie detector, rename) as is. Delete fetchUpdateFile if nothing else calls it. Run go test . -run 'Update' -count=1 to prove update_test.go still passes.
- [ ] Step 4: Run the tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/publicrelease/ . -run 'Feed|Update' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc Expected: ok for both packages.
- [ ] Step 5: Commit
gofmt -l . internal/publicrelease && git add internal/publicrelease update.go && git commit -q -F - <<'EOF'
publicrelease: the vendor feed's verification chain, written once
carlos update and the console's carlos system update share it: latest
tag, signed SHA256SUMS, per-asset sha.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 11: The console doors (internal/console/apisystem.go)
Files:
- Create:
internal/console/apisystem.go,internal/console/apisystem_test.go - Modify:
internal/console/authcli.goregisterCLIRoutes()(add fivecliRoutelines),internal/console/server.goConfig(addSystemFeed publicrelease.Feed)
Interfaces:
- Consumes:
s.cliRoute(pattern, svcrole.Set{}, handler);principalFrom(ctx),PrincipalUser;s.store.Role(OpsAccountID, addr),RoleOwner;s.cfg.Releases,s.cfg.Key,s.cfg.ReleasePub;s.bumpSerial(ctx);writeJSON,jsonError,readJSONStrictLimit;status.LoadBoxes,release.NewPrefixStore,s.openPassRegistry().ListFleets()(thefleetMetersloop,economicspass.go:648-670);release.PromoteSystem,release.RollbackSystem,release.GetSystemPointer,release.RootSigner;publicrelease.Feed;apiShipMaxBytes-style limits. - Produces (routes, all person-only, ops-owner-only, bare 404 on refusal like
IsOperatordoors): ``GET /api/cli/system/status → {"pointers": {"canary": {...}|null, "stable": ...}, "boxes": [{label, platform, channel, release, target, system:{...}|null, stale}]} POST /api/cli/system/update → {"job": "<id>"}; body {"channel","version"?,"upload"?,"spread"?,"component"?} GET /api/cli/system/jobs/{id} → {"steps": ["resolved v0.18.0", ...], "done": bool, "error": ""} PUT /api/cli/system/upload → {"upload": "<handle>", "size": n}; multipart: meta {"version","platform"}, file POST /api/cli/system/rollback → {"channel","version"}; body {"channel","component"?}`go func (s *Server) apiSystemOwner(w, r) (addr string, ok bool) type systemJob struct{ mu sync.Mutex; Steps []string; Done bool; Err string } func (s *Server) runSystemUpdate(ctx, job *systemJob, actor string, in systemUpdateRequest)`Job and upload registries are fields onServer:systemJobs map[string]*systemJob,systemUploads map[string]systemUpload, guarded bysystemMu sync.Mutex`; one upload slot (128 MiB cap) held in memory, consumed by the job that names it.
- [ ] Step 1: Write the failing tests
package console
import (
"bytes"
"context"
"crypto/ed25519"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"fmt"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/publicrelease"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/status"
)
// systemFeed serves a signed fake release from an httptest server and
// points the fixture's console at it.
func systemFeed(t *testing.T, fx *consoleFixture, assets map[string][]byte) {
t.Helper()
pub, priv, _ := ed25519.GenerateKey(nil)
var sums strings.Builder
for name, body := range assets {
s := sha256.Sum256(body)
fmt.Fprintf(&sums, "%s %s\n", hex.EncodeToString(s[:]), name)
}
sig := base64.StdEncoding.EncodeToString(ed25519.Sign(priv, []byte(sums.String())))
mux := http.NewServeMux()
mux.HandleFunc("/latest.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "v0.18.0") })
mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, sums.String()) })
mux.HandleFunc("/dl/v0.18.0/SHA256SUMS.txt.sig", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, sig) })
for name, body := range assets {
body := body
mux.HandleFunc("/dl/v0.18.0/"+name, func(w http.ResponseWriter, r *http.Request) { w.Write(body) })
}
ts := httptest.NewServer(mux)
t.Cleanup(ts.Close)
fx.srv.cfg.SystemFeed = publicrelease.Feed{LatestURL: ts.URL + "/latest.txt", DownloadBase: ts.URL + "/dl", PubKey: base64.StdEncoding.EncodeToString(pub)}
}
func waitJob(t *testing.T, fx *consoleFixture, token, id string) map[string]any {
t.Helper()
for i := 0; i < 100; i++ {
st, out := getJSON(t, fx.ts.URL+"/api/cli/system/jobs/"+id, token)
if st != http.StatusOK {
t.Fatalf("job = %d %v", st, out)
}
if out["done"] == true {
return out
}
time.Sleep(20 * time.Millisecond)
}
t.Fatal("job never finished")
return nil
}
func seedBox(t *testing.T, fx *consoleFixture, label, platform, channel, rel string) {
t.Helper()
b := status.CollectBox(nil, status.Probes{Release: rel, Platform: platform, SystemChannel: channel}, time.Now())
b.Label = label
if err := status.WriteBox(context.Background(), fx.rel, b); err != nil {
t.Fatal(err)
}
}
func TestSystemDoorsAreForOpsOwnersOnly(t *testing.T) {
fx := newConsoleFixture(t)
if err := fx.st.AddMember(OpsAccountID, "member@keymail.dev", RoleMember, time.Now()); err != nil {
t.Fatal(err)
}
member := cliToken(t, fx, "member@keymail.dev", "code-sys-member")
for _, u := range []string{"/api/cli/system/status"} {
if st, _ := getJSON(t, fx.ts.URL+u, member); st != http.StatusNotFound {
t.Errorf("GET %s as a member = %d, want 404", u, st)
}
}
if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound {
t.Errorf("update as a member = %d, want 404", st)
}
if st, _ := postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, member); st != http.StatusNotFound {
t.Errorf("rollback as a member = %d, want 404", st)
}
if st, _ := getJSON(t, fx.ts.URL+"/api/cli/system/status", "not-a-token"); st == http.StatusOK {
t.Error("unauthenticated status answered 200")
}
}
func TestSystemUpdateMirrorsSignsAndBumps(t *testing.T) {
fx := newConsoleFixture(t)
arm := []byte("ELF arm64")
systemFeed(t, fx, map[string][]byte{"carlos-linux-arm64": arm, "carlos-linux-amd64": []byte("ELF amd64")})
seedBox(t, fx, "ie", "linux-arm64", "canary", "v0.17.0")
owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-owner")
before, _, _ := release.ReadSerial(context.Background(), fx.rel)
st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "spread": "2m"}, owner)
if st != http.StatusOK {
t.Fatalf("update = %d %v", st, out)
}
job := waitJob(t, fx, owner, str(t, out, "job"))
if job["error"] != "" {
t.Fatalf("job failed: %v", job)
}
p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary")
if err != nil {
t.Fatal(err)
}
if p.Current.Version != "v0.18.0" || p.Spread != "2m0s" || p.Current.Source != "release:v0.18.0" {
t.Fatalf("pointer = %+v", p)
}
a, ok := p.Current.Artifacts["linux-arm64"]
if !ok {
t.Fatalf("no arm64 artifact: %+v", p.Current.Artifacts)
}
if _, ok := p.Current.Artifacts["linux-amd64"]; ok {
t.Fatal("mirrored a platform no box reports")
}
body, _, err := fx.rel.Get(context.Background(), a.Key)
if err != nil || !bytes.Equal(body, arm) {
t.Fatalf("artifact not mirrored: %v", err)
}
s := sha256.Sum256(arm)
if a.SHA256 != hex.EncodeToString(s[:]) {
t.Fatalf("sha = %s", a.SHA256)
}
after, _, _ := release.ReadSerial(context.Background(), fx.rel)
if after.Serial == before.Serial {
t.Fatal("serial not bumped")
}
// A second promote of the same version to stable copies nothing and
// refuses different bytes under the same version.
st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "stable", "version": "v0.18.0"}, owner)
if st != http.StatusOK {
t.Fatalf("second update = %d %v", st, out)
}
if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" {
t.Fatalf("second job: %v", job)
}
if _, err := fx.rel.Put(context.Background(), a.Key, []byte("different"), ""); err == nil {
// Overwrote the object out from under the pointer; a re-promote must refuse.
st, out = postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "version": "v0.18.0"}, owner)
job := waitJob(t, fx, owner, str(t, out, "job"))
if job["error"] == "" || !strings.Contains(job["error"].(string), "immutable") {
t.Fatalf("re-promote over different bytes: %v", job)
}
}
}
func TestSystemUpdateFromAnUpload(t *testing.T) {
fx := newConsoleFixture(t)
owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-upload")
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
meta, _ := mw.CreateFormField("meta")
meta.Write([]byte(`{"version":"66587fa","platform":"linux-arm64"}`))
file, _ := mw.CreateFormFile("file", "carlos-linux-arm64")
file.Write([]byte("ELF local"))
mw.Close()
req, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf)
req.Header.Set("Content-Type", mw.FormDataContentType())
req.Header.Set("Authorization", "Bearer "+owner)
res, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
var up map[string]any
if res.StatusCode != http.StatusOK {
t.Fatalf("upload = %d", res.StatusCode)
}
if err := json.NewDecoder(res.Body).Decode(&up); err != nil {
t.Fatal(err)
}
st, out := postJSON(t, fx.ts.URL+"/api/cli/system/update", map[string]any{"channel": "canary", "upload": up["upload"]}, owner)
if st != http.StatusOK {
t.Fatalf("update = %d %v", st, out)
}
if job := waitJob(t, fx, owner, str(t, out, "job")); job["error"] != "" {
t.Fatalf("job: %v", job)
}
p, _, err := release.GetSystemPointer(context.Background(), fx.rel, fx.pub, "router", "canary")
if err != nil || p.Current.Version != "66587fa" || p.Current.Source != "file" {
t.Fatalf("pointer = %+v %v", p, err)
}
// Upload without a platform is refused at the door.
var buf2 bytes.Buffer
mw2 := multipart.NewWriter(&buf2)
m2, _ := mw2.CreateFormField("meta")
m2.Write([]byte(`{"version":"abc"}`))
f2, _ := mw2.CreateFormFile("file", "x")
f2.Write([]byte("x"))
mw2.Close()
req2, _ := http.NewRequest(http.MethodPut, fx.ts.URL+"/api/cli/system/upload", &buf2)
req2.Header.Set("Content-Type", mw2.FormDataContentType())
req2.Header.Set("Authorization", "Bearer "+owner)
res2, _ := http.DefaultClient.Do(req2)
if res2.StatusCode != http.StatusBadRequest {
t.Fatalf("upload without platform = %d", res2.StatusCode)
}
}
func TestSystemStatusAndRollback(t *testing.T) {
fx := newConsoleFixture(t)
owner := cliToken(t, fx, "paul@keymail.dev", "code-sys-status")
seedBox(t, fx, "ie", "linux-arm64", "canary", "v1")
sg := release.RootSigner(fx.priv)
rel := func(v string) release.SystemRelease {
return release.SystemRelease{Version: v, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey(v, "linux-arm64"), SHA256: strings.Repeat("a", 64)}}}
}
release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v1"), release.SystemPromoteOptions{})
time.Sleep(1100 * time.Millisecond)
release.PromoteSystem(context.Background(), fx.rel, sg, "router", "canary", rel("v2"), release.SystemPromoteOptions{})
st, out := getJSON(t, fx.ts.URL+"/api/cli/system/status", owner)
if st != http.StatusOK {
t.Fatalf("status = %d %v", st, out)
}
boxes := out["boxes"].([]any)
if len(boxes) != 1 {
t.Fatalf("boxes = %v", boxes)
}
b := boxes[0].(map[string]any)
if b["label"] != "ie" || b["release"] != "v1" || b["target"] != "v2" || b["channel"] != "canary" {
t.Fatalf("box row = %v", b)
}
time.Sleep(1100 * time.Millisecond)
st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner)
if st != http.StatusOK || str(t, out, "version") != "v1" {
t.Fatalf("rollback = %d %v", st, out)
}
if st, out = postJSON(t, fx.ts.URL+"/api/cli/system/rollback", map[string]string{"channel": "canary"}, owner); st != http.StatusConflict {
t.Fatalf("second rollback = %d %v (nothing left in history)", st, out)
}
}(add "encoding/json" to the imports.)
- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -run 'TestSystem' -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc Expected: build failure (SystemFeed undefined) or 404s.
- [ ] Step 3: Implement
server.go Config gains, next to Releases:
// SystemFeed is where `carlos system update` fetches a public release
// from. Zero value = the vendor feed; tests point it at a stub.
SystemFeed publicrelease.Feedand Server gains systemMu sync.Mutex; systemJobs map[string]*systemJob; systemUploads map[string]systemUpload.
authcli.go registerCLIRoutes():
// carlos system (spec §5): ops-account owners, person sessions only.
s.cliRoute("GET /api/cli/system/status", svcrole.Set{}, s.handleAPISystemStatus)
s.cliRoute("POST /api/cli/system/update", svcrole.Set{}, s.handleAPISystemUpdate)
s.cliRoute("GET /api/cli/system/jobs/{id}", svcrole.Set{}, s.handleAPISystemJob)
s.cliRoute("PUT /api/cli/system/upload", svcrole.Set{}, s.handleAPISystemUpload)
s.cliRoute("POST /api/cli/system/rollback", svcrole.Set{}, s.handleAPISystemRollback)apisystem.go:
package console
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"sort"
"strings"
"sync"
"time"
"github.com/carlosframework/platform/internal/publicrelease"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/status"
)
// The console half of `carlos system` (spec §5). Every door here is for an
// OWNER of the ops account on a PERSON session, checked on every call;
// refusal is a bare 404 like the operator doors, so a non-owner learns
// nothing. The console holds the bucket, the signing key and the egress;
// the operator's laptop holds none of them.
const (
systemUploadMaxBytes = int64(128 << 20)
systemAssetMaxBytes = int64(256 << 20)
systemJobTTL = time.Hour
)
var systemPlatformRe = targetRe // apicli.go: ^[a-z0-9]+-[a-z0-9]+$
type systemJob struct {
mu sync.Mutex
Steps []string
Done bool
Err string
at time.Time
}
func (j *systemJob) step(format string, args ...any) {
j.mu.Lock()
j.Steps = append(j.Steps, fmt.Sprintf(format, args...))
j.mu.Unlock()
}
func (j *systemJob) finish(err error) {
j.mu.Lock()
j.Done = true
if err != nil {
j.Err = err.Error()
}
j.mu.Unlock()
}
type systemUpload struct {
version, platform string
data []byte
at time.Time
}
func (s *Server) apiSystemOwner(w http.ResponseWriter, r *http.Request) (string, bool) {
p, ok := principalFrom(r.Context())
if !ok || p.Kind != PrincipalUser {
http.NotFound(w, r)
return "", false
}
role, err := s.store.Role(OpsAccountID, p.Address)
if err != nil {
log.Printf("console: api system: role: %v", err)
http.NotFound(w, r)
return "", false
}
if role != RoleOwner {
http.NotFound(w, r)
return "", false
}
return p.Address, true
}
func (s *Server) systemFeed() publicrelease.Feed {
if s.cfg.SystemFeed.DownloadBase == "" {
return publicrelease.Default()
}
return s.cfg.SystemFeed
}
// systemBoxes reads every heartbeat: the platform prefix and each fleet's.
func (s *Server) systemBoxes(ctx context.Context) ([]status.Box, error) {
boxes, err := status.LoadBoxes(ctx, s.cfg.Releases)
if err != nil {
return nil, err
}
reg, err := s.openPassRegistry()
if err != nil {
return boxes, nil // no registry: platform boxes only
}
fleets, err := reg.ListFleets()
if err != nil {
return boxes, nil
}
for _, f := range fleets {
fb, err := status.LoadBoxes(ctx, release.NewPrefixStore(s.cfg.Releases, "fleets/"+f.Name+"/"))
if err != nil {
log.Printf("console: api system: fleet %s boxes: %v", f.Name, err)
continue
}
boxes = append(boxes, fb...)
}
return boxes, nil
}
type systemUpdateRequest struct {
Component string `json:"component"`
Channel string `json:"channel"`
Version string `json:"version"`
Upload string `json:"upload"`
Spread string `json:"spread"`
}
func (s *Server) handleAPISystemUpdate(w http.ResponseWriter, r *http.Request) {
addr, ok := s.apiSystemOwner(w, r)
if !ok {
return
}
if s.cfg.Key == nil {
jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only")
return
}
var in systemUpdateRequest
if !readJSONStrictLimit(w, r, &in, 4<<10) {
return
}
if in.Component == "" {
in.Component = release.SystemComponentRouter
}
if in.Component != release.SystemComponentRouter {
jsonError(w, http.StatusBadRequest, "unknown component "+in.Component)
return
}
if in.Channel != release.RungCanary && in.Channel != release.RungStable {
jsonError(w, http.StatusBadRequest, "channel must be canary or stable")
return
}
spread := 5 * time.Minute
if in.Spread != "" {
d, err := time.ParseDuration(in.Spread)
if err != nil || d < 0 || d > 24*time.Hour {
jsonError(w, http.StatusBadRequest, "spread must be a duration between 0 and 24h")
return
}
spread = d
}
id := s.newSystemJob()
job := s.systemJobs[id]
go s.runSystemUpdate(context.WithoutCancel(r.Context()), job, addr, in, spread)
writeJSON(w, http.StatusOK, map[string]string{"job": id})
}
func (s *Server) newSystemJob() string {
b := make([]byte, 8)
rand.Read(b)
id := hex.EncodeToString(b)
s.systemMu.Lock()
defer s.systemMu.Unlock()
if s.systemJobs == nil {
s.systemJobs = map[string]*systemJob{}
}
for k, j := range s.systemJobs {
if time.Since(j.at) > systemJobTTL {
delete(s.systemJobs, k)
}
}
s.systemJobs[id] = &systemJob{at: time.Now()}
return id
}
// runSystemUpdate is the promote: resolve, ingest (or take the upload),
// mirror, sign, bump. Every step is idempotent, so a lost job is re-run.
func (s *Server) runSystemUpdate(ctx context.Context, job *systemJob, actor string, in systemUpdateRequest, spread time.Duration) {
err := func() error {
var rel release.SystemRelease
if in.Upload != "" {
s.systemMu.Lock()
up, ok := s.systemUploads[in.Upload]
delete(s.systemUploads, in.Upload)
s.systemMu.Unlock()
if !ok {
return errors.New("upload handle unknown or already consumed")
}
if in.Version != "" && in.Version != up.version {
return fmt.Errorf("upload is %s, request says %s", up.version, in.Version)
}
job.step("using uploaded %s for %s", up.version, up.platform)
sum := sha256.Sum256(up.data)
rel = release.SystemRelease{Version: up.version, Source: "file", Artifacts: map[string]release.SystemArtifact{}}
if err := s.mirrorSystemArtifact(ctx, job, up.version, up.platform, up.data, hex.EncodeToString(sum[:])); err != nil {
return err
}
rel.Artifacts[up.platform] = release.SystemArtifact{Key: release.SystemArtifactKey(up.version, up.platform), SHA256: hex.EncodeToString(sum[:])}
} else {
feed := s.systemFeed()
tag := in.Version
if tag == "" {
var err error
if tag, err = feed.Latest(ctx); err != nil {
return err
}
}
if !publicrelease.TagRe.MatchString(tag) {
return fmt.Errorf("%q is not a release tag; use --file for a local build", tag)
}
job.step("resolved %s", tag)
platforms, err := s.systemPlatforms(ctx)
if err != nil {
return err
}
rel = release.SystemRelease{Version: tag, Source: "release:" + tag, Artifacts: map[string]release.SystemArtifact{}}
for _, plat := range platforms {
data, sha, err := feed.Asset(ctx, tag, publicrelease.AssetName(plat), systemAssetMaxBytes)
if err != nil {
return fmt.Errorf("%s: %w", plat, err)
}
job.step("verified %s (%d bytes)", publicrelease.AssetName(plat), len(data))
if err := s.mirrorSystemArtifact(ctx, job, tag, plat, data, sha); err != nil {
return err
}
rel.Artifacts[plat] = release.SystemArtifact{Key: release.SystemArtifactKey(tag, plat), SHA256: sha}
}
}
p, err := release.PromoteSystem(ctx, s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, rel,
release.SystemPromoteOptions{Actor: actor, Spread: spread})
if err != nil {
return err
}
job.step("pointed %s/%s at %s (spread %s)", in.Component, in.Channel, p.Current.Version, p.Spread)
s.audit(nil, "platform", actor, "system-update", "pointed "+in.Component+"/"+in.Channel+" at "+p.Current.Version)
s.bumpSerial(ctx)
job.step("serial bumped: boxes on %s start their spread", in.Channel)
return nil
}()
job.finish(err)
}
// systemPlatforms: the distinct platforms the fleet reports; linux-arm64
// when nothing has reported the field yet.
func (s *Server) systemPlatforms(ctx context.Context) ([]string, error) {
boxes, err := s.systemBoxes(ctx)
if err != nil {
return nil, err
}
set := map[string]bool{}
for _, b := range boxes {
if b.Metrics != nil && b.Metrics.Platform != "" {
set[b.Metrics.Platform] = true
}
}
if len(set) == 0 {
set["linux-arm64"] = true
}
var out []string
for p := range set {
out = append(out, p)
}
sort.Strings(out)
return out, nil
}
// mirrorSystemArtifact: present with the same sha → skip; present with a
// different sha → refuse (versions are immutable, spec §3).
func (s *Server) mirrorSystemArtifact(ctx context.Context, job *systemJob, version, platform string, data []byte, sha string) error {
key := release.SystemArtifactKey(version, platform)
existing, etag, err := s.cfg.Releases.Get(ctx, key)
switch {
case err == nil:
sum := sha256.Sum256(existing)
if hex.EncodeToString(sum[:]) == sha {
job.step("%s already mirrored", key)
return nil
}
return fmt.Errorf("%s exists with different bytes — versions are immutable; pick a new version", key)
case errors.Is(err, os.ErrNotExist):
etag = ""
default:
return err
}
if _, err := s.cfg.Releases.Put(ctx, key, data, etag); err != nil {
return err
}
job.step("mirrored %s", key)
return nil
}
func (s *Server) handleAPISystemJob(w http.ResponseWriter, r *http.Request) {
if _, ok := s.apiSystemOwner(w, r); !ok {
return
}
s.systemMu.Lock()
job, ok := s.systemJobs[r.PathValue("id")]
s.systemMu.Unlock()
if !ok {
jsonError(w, http.StatusNotFound, "no such job — the console may have restarted; re-run, every step is idempotent")
return
}
job.mu.Lock()
defer job.mu.Unlock()
writeJSON(w, http.StatusOK, map[string]any{"steps": append([]string(nil), job.Steps...), "done": job.Done, "error": job.Err})
}
func (s *Server) handleAPISystemUpload(w http.ResponseWriter, r *http.Request) {
if _, ok := s.apiSystemOwner(w, r); !ok {
return
}
r.Body = http.MaxBytesReader(w, r.Body, systemUploadMaxBytes+64<<10)
mr, err := r.MultipartReader()
if err != nil {
jsonError(w, http.StatusBadRequest, "multipart body required")
return
}
var meta struct {
Version string `json:"version"`
Platform string `json:"platform"`
}
first, err := mr.NextPart()
if err != nil || first.FormName() != "meta" {
jsonError(w, http.StatusBadRequest, "first part must be meta")
return
}
if err := json.NewDecoder(io.LimitReader(first, 4<<10)).Decode(&meta); err != nil {
jsonError(w, http.StatusBadRequest, "meta: "+err.Error())
return
}
if !release.ValidVersion(meta.Version) || meta.Version == "dev" {
jsonError(w, http.StatusBadRequest, "meta.version is required and must be a version")
return
}
if !systemPlatformRe.MatchString(meta.Platform) {
jsonError(w, http.StatusBadRequest, "meta.platform is required (e.g. linux-arm64): the console cannot run a foreign binary to ask it")
return
}
part, err := mr.NextPart()
if err != nil || part.FormName() != "file" {
jsonError(w, http.StatusBadRequest, "second part must be file")
return
}
data, err := io.ReadAll(io.LimitReader(part, systemUploadMaxBytes+1))
if err != nil {
jsonError(w, http.StatusBadRequest, "file: "+err.Error())
return
}
if int64(len(data)) > systemUploadMaxBytes {
jsonError(w, http.StatusRequestEntityTooLarge, "file exceeds 128 MiB")
return
}
b := make([]byte, 8)
rand.Read(b)
handle := hex.EncodeToString(b)
s.systemMu.Lock()
if s.systemUploads == nil {
s.systemUploads = map[string]systemUpload{}
}
for k := range s.systemUploads { // one slot: a forgotten upload is not a memory leak
delete(s.systemUploads, k)
}
s.systemUploads[handle] = systemUpload{version: meta.Version, platform: meta.Platform, data: data, at: time.Now()}
s.systemMu.Unlock()
writeJSON(w, http.StatusOK, map[string]any{"upload": handle, "size": len(data)})
}
func (s *Server) handleAPISystemRollback(w http.ResponseWriter, r *http.Request) {
addr, ok := s.apiSystemOwner(w, r)
if !ok {
return
}
if s.cfg.Key == nil {
jsonError(w, http.StatusServiceUnavailable, "no signing key on this host — the console is running read-only")
return
}
var in struct {
Component string `json:"component"`
Channel string `json:"channel"`
}
if !readJSONStrictLimit(w, r, &in, 1<<10) {
return
}
if in.Component == "" {
in.Component = release.SystemComponentRouter
}
p, err := release.RollbackSystem(r.Context(), s.cfg.Releases, release.RootSigner(s.cfg.Key), in.Component, in.Channel, addr)
switch {
case errors.Is(err, release.ErrNoSystemHistory):
jsonError(w, http.StatusConflict, err.Error())
return
case errors.Is(err, os.ErrNotExist):
jsonError(w, http.StatusNotFound, "no "+in.Channel+" pointer to roll back")
return
case err != nil:
jsonError(w, http.StatusInternalServerError, err.Error())
return
}
s.audit(nil, "platform", addr, "system-rollback", "pointed "+in.Component+"/"+in.Channel+" back at "+p.Current.Version)
s.bumpSerial(r.Context())
writeJSON(w, http.StatusOK, map[string]string{"channel": in.Channel, "version": p.Current.Version})
}
func (s *Server) handleAPISystemStatus(w http.ResponseWriter, r *http.Request) {
if _, ok := s.apiSystemOwner(w, r); !ok {
return
}
ctx := r.Context()
pointers := map[string]any{}
targets := map[string]string{}
for _, ch := range []string{release.RungCanary, release.RungStable} {
p, _, err := release.GetSystemPointer(ctx, s.cfg.Releases, s.cfg.ReleasePub, release.SystemComponentRouter, ch)
if err != nil {
pointers[ch] = nil
continue
}
pointers[ch] = map[string]any{"version": p.Current.Version, "promoted_at": p.PromotedAt, "spread": p.Spread, "source": p.Current.Source,
"history": len(p.History)}
targets[ch] = p.Current.Version
}
boxes, err := s.systemBoxes(ctx)
if err != nil {
jsonError(w, http.StatusBadGateway, "box heartbeats unreadable: "+err.Error())
return
}
now := time.Now()
rows := make([]map[string]any, 0, len(boxes))
for _, b := range boxes {
row := map[string]any{"label": b.Label, "region": b.Region, "stale": status.BoxStale(b, now, status.SnapshotPeriod),
"platform": "", "channel": "", "release": "", "target": "", "system": nil}
if b.Metrics != nil {
ch := b.Metrics.SystemChannel
if ch == "" {
ch = "stable"
}
row["platform"], row["channel"], row["release"] = b.Metrics.Platform, ch, b.Metrics.Release
row["target"] = targets[ch]
if b.Metrics.System != nil {
row["system"] = b.Metrics.System
}
}
rows = append(rows, row)
}
sort.Slice(rows, func(i, j int) bool { return rows[i]["label"].(string) < rows[j]["label"].(string) })
writeJSON(w, http.StatusOK, map[string]any{"pointers": pointers, "boxes": rows})
}Check three things against the code while writing: s.audit's signature (apicli.go:855 passes (acct, app, addr, action, detail); if acct cannot be nil, use the ops account fetched by id, or add a s.auditSystem(addr, action, detail) writing an event row against OpsAccountID); status.BoxStale's signature (fleet.go:137); os import. readJSONStrictLimit is at authcli.go:849.
- [ ] Step 4: Run the console tests
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/console/ -count=1 2>&1); rc=$?; echo "$out" | tail -20; echo rc=$rc Expected: ok. TestEveryCLIDoorNamesItsRoles and TestEveryClosedDoorRefusesAServiceCredential cover the five new doors automatically; if either fails, the route line is wrong, not the test.
- [ ] Step 5: Commit
gofmt -l internal/console/ && git add internal/console/apisystem.go internal/console/apisystem_test.go internal/console/authcli.go internal/console/server.go && git commit -q -F - <<'EOF'
console: the carlos system doors — status, update job, upload, rollback
Ops-account owners on a person session, bare 404 otherwise. The
update is a job: resolve, ingest from the signed public feed (or an
upload), mirror per reported platform, sign the pointer, bump the
serial. Versions are immutable in the bucket.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 12: carlos system update | status | rollback (API mode)
Files:
- Modify:
system.go(replace the three placeholders),system_test.go
Interfaces:
- Consumes:
requireDeployAPISession(consoleFlag) *apiSession(deploy.go:1151);(*apiSession).getJSON/postJSON(apimode.go:737,799);(*consoleClient).doMultipart(auth.go:485) viasess.c;writeFilePart(mw, name, path)(apimode.go:3009);interactiveTerminal,promptYes(account_choose.go);stdout,colorFor,mutedf(color.go);flagUsage. - Produces: ``
go func runSystemUpdate(args []string) // --console --channel --version --file --platform --spread --yes func runSystemStatus(args []string) // --console --channel func runSystemRollback(args []string) // --console --channel --yes func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string) type systemStatusResponse struct{ Pointers map[string]*systemPointerRow; Boxes []systemBoxRow }``
- [ ] Step 1: Write the failing test (the pure renderer; the doors are covered in Task 11)
func TestPrintSystemStatusSaysWhatEachBoxIsDoing(t *testing.T) {
resp := systemStatusResponse{
Pointers: map[string]*systemPointerRow{"canary": {Version: "v2", Spread: "5m0s"}, "stable": nil},
Boxes: []systemBoxRow{
{Label: "ie", Platform: "linux-arm64", Channel: "canary", Release: "v2", Target: "v2"},
{Label: "us", Platform: "linux-arm64", Channel: "canary", Release: "v1", Target: "v2",
System: &systemStateRow{Version: "v2", Outcome: "rolled-back", Reason: "gate: carlos-edge is failed"}},
{Label: "au", Platform: "", Channel: "stable", Release: "v1", Target: "", Stale: true},
},
}
var b strings.Builder
printSystemStatus(&b, resp, "")
out := b.String()
for _, want := range []string{"canary v2", "stable (no pointer)", "ie ", "current", "us ", "rolled-back", "gate: carlos-edge is failed", "au ", "UNREACHABLE", "pre-upgrade"} {
if !strings.Contains(out, want) {
t.Errorf("status output lacks %q:\n%s", want, out)
}
}
var only strings.Builder
printSystemStatus(&only, resp, "stable")
if strings.Contains(only.String(), "ie ") {
t.Error("--channel stable still listed a canary box")
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test . -run TestPrintSystemStatus -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc Expected: undefined types.
- [ ] Step 3: Implement
Replace the placeholders in system.go:
type systemPointerRow struct {
Version string `json:"version"`
PromotedAt time.Time `json:"promoted_at"`
Spread string `json:"spread"`
Source string `json:"source"`
History int `json:"history"`
}
type systemStateRow struct {
Channel string `json:"channel"`
Version string `json:"version"`
PromotedAt time.Time `json:"promoted_at"`
Outcome string `json:"outcome"`
Reason string `json:"reason"`
At time.Time `json:"at"`
}
type systemBoxRow struct {
Label string `json:"label"`
Region string `json:"region"`
Platform string `json:"platform"`
Channel string `json:"channel"`
Release string `json:"release"`
Target string `json:"target"`
System *systemStateRow `json:"system"`
Stale bool `json:"stale"`
}
type systemStatusResponse struct {
Pointers map[string]*systemPointerRow `json:"pointers"`
Boxes []systemBoxRow `json:"boxes"`
}
func runSystemUpdate(args []string) {
fs := flag.NewFlagSet("system update", flag.ExitOnError)
consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
channel := fs.String("channel", "stable", "canary or stable")
version := fs.String("version", "", "release tag to promote (default: the latest public release); with --file, the version the binary reports")
file := fs.String("file", "", "a locally built binary instead of a public release (dev and pre-release)")
platform := fs.String("platform", "linux-arm64", "with --file: the binary's <goos>-<goarch>")
spread := fs.String("spread", "5m", "boxes jitter their restarts over this; 0 for a hotfix")
yes := fs.Bool("yes", false, "do not ask")
fs.Usage = flagUsage(fs, "")
fs.Parse(args)
if *file != "" && *version == "" {
log.Fatal("--file needs --version: the console cannot run a foreign-arch binary to ask it")
}
sess := requireDeployAPISession(*consoleFlag)
ctx := context.Background()
// What this reaches, before it is done.
st, body, err := sess.getJSON(ctx, "/api/cli/system/status")
if err != nil || st != http.StatusOK {
log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err))
}
var cur systemStatusResponse
_ = json.Unmarshal(body, &cur)
var reach []string
for _, b := range cur.Boxes {
if b.Channel == *channel {
reach = append(reach, fmt.Sprintf("%s (%s, running %s)", b.Label, b.Platform, orDash(b.Release)))
}
}
what := "the latest public release"
if *version != "" {
what = *version
}
fmt.Fprintf(stdout, "promote %s to %s/%s — %d box(es) will roll over %s:\n", what, systemComponentRouter, *channel, len(reach), *spread)
for _, r := range reach {
fmt.Fprintf(stdout, " %s\n", r)
}
if len(reach) == 0 {
mutedf(stdout, " (no box reports channel %s yet; boxes on an older build count as stable)\n", *channel)
}
if !*yes {
if !interactiveTerminal() {
log.Fatal("no terminal to confirm on; pass --yes")
}
if !promptYes(stdout, "go ahead? [y/N] ") {
return
}
}
in := map[string]any{"channel": *channel, "spread": *spread}
if *version != "" {
in["version"] = *version
}
if *file != "" {
handle, err := sess.uploadSystemBinary(ctx, *file, *version, *platform)
if err != nil {
log.Fatalf("upload: %v", err)
}
in["upload"] = handle
}
st, body, err = sess.postJSON(ctx, "/api/cli/system/update", in)
if err != nil || st != http.StatusOK {
log.Fatalf("carlos system update: %s", apiErrorFromBody(st, body, err))
}
var started struct{ Job string `json:"job"` }
_ = json.Unmarshal(body, &started)
if err := sess.watchSystemJob(ctx, started.Job, stdout); err != nil {
log.Fatal(err)
}
fmt.Fprintf(stdout, "%s boxes on %s roll themselves; watch with: carlos system status --channel %s\n", colorFor(stdout).ok("promoted"), *channel, *channel)
}
// watchSystemJob polls the job and prints each step once, as deploy's
// watchVersion does for a route.
func (s *apiSession) watchSystemJob(ctx context.Context, id string, w io.Writer) error {
printed := 0
deadline := time.Now().Add(15 * time.Minute)
for {
st, body, err := s.getJSON(ctx, "/api/cli/system/jobs/"+id)
if err != nil || st != http.StatusOK {
return fmt.Errorf("job %s: %s", id, apiErrorFromBody(st, body, err))
}
var j struct {
Steps []string `json:"steps"`
Done bool `json:"done"`
Error string `json:"error"`
}
_ = json.Unmarshal(body, &j)
for ; printed < len(j.Steps); printed++ {
fmt.Fprintf(w, " %s\n", j.Steps[printed])
}
if j.Done {
if j.Error != "" {
return fmt.Errorf("carlos system update: %s", j.Error)
}
return nil
}
if time.Now().After(deadline) {
return errors.New("carlos system update: the console's job did not finish in 15 minutes")
}
time.Sleep(time.Second)
}
}
func (s *apiSession) uploadSystemBinary(ctx context.Context, path, version, platform string) (string, error) {
st, body, err := s.c.doMultipart(ctx, "/api/cli/system/upload", func(mw *multipart.Writer) error {
meta, err := mw.CreateFormField("meta")
if err != nil {
return err
}
if err := json.NewEncoder(meta).Encode(map[string]string{"version": version, "platform": platform}); err != nil {
return err
}
return writeFilePart(mw, "file", path)
})
if err != nil || st != http.StatusOK {
return "", fmt.Errorf("%s", apiErrorFromBody(st, body, err))
}
var out struct{ Upload string `json:"upload"` }
_ = json.Unmarshal(body, &out)
return out.Upload, nil
}
func runSystemStatus(args []string) {
fs := flag.NewFlagSet("system status", flag.ExitOnError)
consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
channel := fs.String("channel", "", "only boxes on this channel")
fs.Usage = flagUsage(fs, "")
fs.Parse(args)
sess := requireDeployAPISession(*consoleFlag)
st, body, err := sess.getJSON(context.Background(), "/api/cli/system/status")
if err != nil || st != http.StatusOK {
log.Fatalf("carlos system status: %s", apiErrorFromBody(st, body, err))
}
var resp systemStatusResponse
if err := json.Unmarshal(body, &resp); err != nil {
log.Fatalf("carlos system status: %v", err)
}
printSystemStatus(stdout, resp, *channel)
}
func printSystemStatus(w io.Writer, resp systemStatusResponse, channel string) {
for _, ch := range []string{"canary", "stable"} {
if channel != "" && ch != channel {
continue
}
p := resp.Pointers[ch]
if p == nil {
fmt.Fprintf(w, "%-7s (no pointer)\n", ch)
continue
}
fmt.Fprintf(w, "%-7s %s promoted %s spread %s %s\n", ch, p.Version, p.PromotedAt.Format(time.RFC3339), p.Spread, p.Source)
}
fmt.Fprintln(w)
c := colorFor(w)
for _, b := range resp.Boxes {
if channel != "" && b.Channel != channel {
continue
}
state := ""
switch {
case b.Stale:
state = c.bad("UNREACHABLE")
case b.Platform == "":
state = c.warn("pre-upgrade (no platform reported; counts as stable)")
case b.Target == "":
state = "no target"
case b.Release == b.Target:
state = c.ok("current")
default:
state = c.warn("behind → " + b.Target)
}
fmt.Fprintf(w, "%-20s %-12s %-7s %-10s %s", b.Label, orDash(b.Platform), b.Channel, orDash(b.Release), state)
if b.System != nil && b.System.Outcome != "applied" && b.System.Outcome != "noted" {
fmt.Fprintf(w, " %s %s: %s", c.bad(b.System.Outcome), b.System.Version, b.System.Reason)
}
fmt.Fprintln(w)
}
}
func runSystemRollback(args []string) {
fs := flag.NewFlagSet("system rollback", flag.ExitOnError)
consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
channel := fs.String("channel", "", "canary or stable (required)")
yes := fs.Bool("yes", false, "do not ask")
fs.Usage = flagUsage(fs, "")
fs.Parse(args)
if *channel == "" {
log.Fatal("--channel is required")
}
sess := requireDeployAPISession(*consoleFlag)
if !*yes {
if !interactiveTerminal() {
log.Fatal("no terminal to confirm on; pass --yes")
}
if !promptYes(stdout, fmt.Sprintf("point %s back at the release before this one? every box on it rolls back [y/N] ", *channel)) {
return
}
}
st, body, err := sess.postJSON(context.Background(), "/api/cli/system/rollback", map[string]string{"channel": *channel})
if err != nil || st != http.StatusOK {
log.Fatalf("carlos system rollback: %s", apiErrorFromBody(st, body, err))
}
var out struct{ Version string `json:"version"` }
_ = json.Unmarshal(body, &out)
fmt.Fprintf(stdout, "%s %s → %s\n", colorFor(stdout).ok("rolled back"), *channel, out.Version)
}Imports: context, encoding/json, errors, flag, io, mime/multipart, net/http, time. orDash exists in cli.go:3327+; apiErrorFromBody exists in apimode.go (confirm its signature with grep -n 'func apiErrorFromBody' *.go; adapt the three-argument call if it takes fewer). sess.c is the *consoleClient field on apiSession (confirm with grep -n 'type apiSession struct' -A 6 apimode.go).
- [ ] Step 4: Run the tests and try the binary against a fixture console
Run: export TMPDIR=/tmp/c; out=$(go test . -run 'TestPrintSystemStatus|TestSystemVerb' -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc; go vet . Expected: PASS, vet clean.
- [ ] Step 5: Commit
gofmt -l . && git add system.go system_test.go && git commit -q -F - <<'EOF'
cli: carlos system update, status, rollback through the console
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 13: The hourly belt — carlos adopt arms the self-update units
Files:
- Modify:
internal/host/agent.goRunAdopt(:2811; insert beforeopenServiceRegistryat:2825) - Create:
internal/host/systemunits.go,internal/host/systemunits_test.go
Interfaces:
- Consumes:
unitsFS,systemctlPlan,copyFileDurable, asysOps-shapedsystemctlfunc. - Produces: ``
go // ensureSystemUnits installs and arms carlos-system-update.path/.service // and carlos-system-recover.service when any is missing from etcDir or // the path unit is inactive / the recover unit not enabled. Non-fatal. func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any))``
- [ ] Step 1: Write the failing test
package host
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestEnsureSystemUnitsInstallsAndArmsWhatIsMissing(t *testing.T) {
etc := t.TempDir()
os.MkdirAll(filepath.Join(etc, "systemd", "system"), 0o755)
var calls []string
systemctl := func(args ...string) (string, error) {
calls = append(calls, strings.Join(args, " "))
switch args[0] {
case "is-active":
return "inactive\n", nil
case "is-enabled":
return "disabled\n", nil
}
return "", nil
}
ensureSystemUnits(etc, systemctl, t.Logf)
for _, u := range []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"} {
if _, err := os.Stat(filepath.Join(etc, "systemd", "system", u)); err != nil {
t.Errorf("%s not installed", u)
}
}
joined := strings.Join(calls, "\n")
for _, want := range []string{"daemon-reload", "enable carlos-system-update.path carlos-system-recover.service", "start carlos-system-update.path"} {
if !strings.Contains(joined, want) {
t.Errorf("calls lack %q:\n%s", want, joined)
}
}
if strings.Contains(joined, "start carlos-system-recover.service") || strings.Contains(joined, "start carlos-system-update.service") {
t.Fatal("a service was started")
}
// All present and armed: no writes, no systemctl beyond the two checks.
calls = nil
systemctl2 := func(args ...string) (string, error) {
calls = append(calls, strings.Join(args, " "))
switch args[0] {
case "is-active":
return "active\n", nil
case "is-enabled":
return "enabled\n", nil
}
return "", nil
}
ensureSystemUnits(etc, systemctl2, t.Logf)
if len(calls) != 2 {
t.Fatalf("an armed box got %v", calls)
}
}- [ ] Step 2: Run to verify failure
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -run TestEnsureSystemUnits -count=1 2>&1); rc=$?; echo "$out" | tail -5; echo rc=$rc
- [ ] Step 3: Implement
package host
import (
"os"
"path/filepath"
"strings"
)
var systemUnitFiles = []string{"carlos-system-update.path", "carlos-system-update.service", "carlos-system-recover.service"}
// ensureSystemUnits is the second belt of spec §7: a box the roll script
// forgot to arm is armed within the hour by the root adopt pass. It
// installs the embedded units when missing and arms them through
// systemctlPlan (never starting a service). Non-fatal: it runs before
// RunAdopt's registry and pubkey gates, and logs instead of exiting.
func ensureSystemUnits(etcDir string, systemctl func(args ...string) (string, error), logf func(string, ...any)) {
unitDir := filepath.Join(etcDir, "systemd", "system")
missing := false
for _, u := range systemUnitFiles {
if _, err := os.Stat(filepath.Join(unitDir, u)); err != nil {
missing = true
}
}
active, _ := systemctl("is-active", systemUpdatePathUnit)
enabled, _ := systemctl("is-enabled", "carlos-system-recover.service")
if !missing && strings.TrimSpace(active) == "active" && strings.TrimSpace(enabled) == "enabled" {
return
}
for _, u := range systemUnitFiles {
b, err := unitsFS.ReadFile("units/" + u)
if err != nil {
logf("adopt: system units: %v", err)
return
}
if err := writeFileDurable(filepath.Join(unitDir, u), b, 0o644); err != nil {
logf("adopt: system units: %v", err)
return
}
}
for _, args := range systemctlPlan([]string{systemUpdatePathUnit, "carlos-system-recover.service"}) {
if _, err := systemctl(args...); err != nil {
logf("adopt: system units: systemctl %s: %v", strings.Join(args, " "), err)
return
}
}
logf("adopt: self-update units installed and armed")
}In RunAdopt, immediately after fs.Parse(args) and before openServiceRegistry:
// Spec §7 belt 2: arm the self-update relay if the last hand roll did
// not. Before the fatal gates below, and never fatal itself.
if !*restartOnly {
ensureSystemUnits("/etc", productionSysOps().systemctl, log.Printf)
}- [ ] Step 4: Run
Run: export TMPDIR=/tmp/c; out=$(go test ./internal/host/ -count=1 2>&1); rc=$?; echo "$out" | tail -10; echo rc=$rc
- [ ] Step 5: Commit
gofmt -l internal/host/ && git add internal/host/systemunits.go internal/host/systemunits_test.go internal/host/agent.go && git commit -q -F - <<'EOF'
host: the hourly adopt pass arms the self-update units if the roll script did not
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF---
Task 14: Docs, the env contract, and the last two script lines
Files:
- Modify:
docs/site/cli.md(after### carlos status, ~line 930),AGENTS.md(the section that namescarlos-stage.path/update-platform-binary.sh, ~lines 246-277) - Out of repo, hand off (do not edit from this checkout):
platform-infrastructure/scripts/update-platform-binary.shand the Tito copy inamadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh
- [ ] Step 1:
docs/site/cli.md— add under## Your deployment, after### carlos status:
````markdown
carlos system
The platform's own release channel. update promotes a public release (or, with --file, a binary you built) to canary or stable; every box on that channel fetches it through its normal store, verifies it against the signed pointer, installs it, restarts its edge, and rolls back to the previous binary if the new one does not come up and stay up. status shows each box's platform, channel, running build, the channel's target, and the last apply outcome with its reason. rollback points a channel back at the release before this one, and every box on it rolls back the same way it rolled forward. All three talk to your console; you need to be an owner of the ops account.
carlos system update --channel canary
carlos system status --channel canary
carlos system update --channel stable --spread 10m
carlos system rollback --channel stableA box is put on canary with CARLOS_SYSTEM_CHANNEL=canary in its host.env. Boxes that have not reported a platform yet count as stable. carlos system apply is the root oneshot behind carlos-system-update.service; it is not for operators. The console is an app — update it with carlos deploy --app console. ````
- [ ] Step 2:
AGENTS.md— in the section describing the root relays, add after thecarlos-restart.pathparagraph:
- `carlos-system-update.path` relays the agent's platform self-update request
to `carlos system apply` (root; spec 2026-09-06-carlos-system-update-design.md
§6). `carlos-system-recover.service` runs once per boot, before the edge,
and on the apply's failure. Both must be armed; the last run of
`update-platform-binary.sh` does it, and the hourly `carlos adopt` pass
installs and arms them if it did not. A box follows `CARLOS_SYSTEM_CHANNEL`
(canary|stable, default stable) in host.env.- [ ] Step 3: The pinfra hand-off — write
docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.mdwith exactly this content, for whoever edits the two scripts (they are in other repos; this checkout does not touch them):
# update-platform-binary.sh: the last two lines it needs
After the `carlos-restart.path` block, guarded the same way:
if [ -f /etc/systemd/system/carlos-system-update.path ]; then
systemctl enable --now carlos-system-update.path
systemctl enable carlos-system-recover.service
fi
And to the post-conditions at the end:
systemctl is-active carlos-system-update.path
systemctl is-enabled carlos-system-recover.service
Same change in amadan.net/tito/carlos-infrastructure/scripts/update-platform-binary.sh.
After every deployment has been hand-rolled once onto a binary carrying the
system-update spec, the script is retired (spec §10).- [ ] Step 4: Full gate
Run: export TMPDIR=/tmp/c; out=$(make test 2>&1); rc=$?; echo "$out" | tail -30; echo rc=$rc Expected: rc=0 (vet, gofmt, every package).
- [ ] Step 5: Commit, push, describe, land
git add docs/site/cli.md AGENTS.md docs/superpowers/plans/2026-09-06-carlos-system-update-pinfra.md && git commit -q -F - <<'EOF'
docs: carlos system, the env contract, and the roll script's last two lines
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
EOF
git push -u origin carlos-system-update
amadan branch describe carlos/platform carlos-system-update -body - <<'EOF'
carlos system update: the platform rolls itself. Spec: docs/superpowers/specs/2026-09-06-carlos-system-update-design.md. Plan: docs/superpowers/plans/2026-09-06-carlos-system-update.md.
EOF
# then: amadan ci status carlos/platform -branch carlos-system-update (until 0)
# then: amadan branch merge carlos/platform carlos-system-update -expect <full sha>---
Self-review
Spec coverage. §3 pointer, keys, immutability, spread, history-of-releases, rollback-is-new-promote, door grants, streaming read → Tasks 1, 2, 3. §4 leg (observe/tick, re-touch, watermark check, suppression, jitter, streamed fetch, request shape, deferred retry), units, exec shim, enabledUnits → Tasks 6, 7. §5 authorization, job, upload, update/status/rollback, router alias, no console verb → Tasks 9, 11, 12. §6 root state files, durability, lock, recovery-first (forward from gated), request walk, channel from env, replay guard per channel, no-op, staging copy 0600→0755, .prev decision, both unit sets from their binaries, quiesce incl. services, removed units, sysusers→tmpfiles→daemon-reload, reset-failed, gate+soak, arm after gate via systemctlPlan, record, rollback, rollback-failed → Tasks 4, 8a–8c. §7 first-roll arming (script + hourly belt), env contract → Tasks 13, 14. §9/§10 nothing to build. §11 unit-level proof: every bullet maps to a test in Tasks 1–13; the live checks are the operator's after landing.
Gaps found and fixed in review: the spec's release.Artifact name collides with an existing type — Task 1 uses SystemArtifact; the spec's "interface gains Open" would break 48 fakes — Task 2 uses an Opener side-interface; the spec's 403 on the system doors is a bare 404 here, matching the console's operator doors; the spec's "ops-account owner" is Role(OpsAccountID) == RoleOwner, not IsOperator — Task 11 says which.
Placeholder scan. Tasks 9 and 8a/8b carry explicit stubs that the next task replaces (errNotYetImplemented, the placeholder verbs, the stub rollback); each is named and replaced in a listed task. No "TBD", no "similar to".
Type consistency. systemRequest{Version, PromotedAt, NoOp, SeenAt} is written by Task 7 and read by 8a; systemResult.matches/terminal are used identically in 7 and 8a; bootstrapPlan{Enabled, Inert} is produced by Task 6 and consumed by 8b/8c/13; sysOps fields are the same in 8a's fixture and 8c's productionSysOps; release.SystemPromoteOptions{Actor, Spread} in 1, 7's fixture and 11; systemStatusResponse in 12 matches the JSON Task 11 writes (pointers, boxes with label, region, platform, channel, release, target, system, stale).