| new file mode 100644 |
| index 000000000..2bd8e34b8 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-08-carlos-deploy.md |
| @@ -0,0 +1,894 @@ |
| +# carlos deploy Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** `carlos deploy` gets a build serving on its URL in single-digit seconds: ship → promote → watch, backed by a bucket deploy signal, a generalized fast-converge loop, on-demand cold-wake staging, and an `X-Carlos-Version` proof header. |
| + |
| +**Architecture:** Pull-only. The console bumps one bucket object (`control/serial`) on writes that create box work; boxes poll that single object every 2s and run the ordinary verified pass set on change. A fast set (canary/* by name ∪ unstaged routes, with per-host backoff) sweeps at 2s until drained. `ExecBackend.Materialize` stages the channel head at cold wake. The edge stamps `X-Carlos-Version` from the route table's `Version` column; the registry fingerprint becomes content-accurate so version swaps actually reload the edge (closes #15). |
| + |
| +**Tech Stack:** Go, sqlite registry, S3/dir release store, systemd units, console HTTP API. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-08-08-carlos-deploy-design.md` — read it before starting; it explains every "why" below. |
| + |
| +## Global Constraints |
| + |
| +- Build gate before every commit: `go build ./... && go vet ./... && go test ./... -count=1` — all packages clean; `gofmt -l` clean for files you touch (three pre-existing unformatted files elsewhere stay alone). |
| +- Sandbox: prefix go commands with `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; if the toolchain reports "read-only file system", run with the sandbox disabled. |
| +- `GOOS=darwin` must compile: `internal/host` links into the CLI people run on Macs — no Linux-only types on any path you touch (CI enforces). |
| +- Wire shapes are additive: live pointers/manifests in production buckets must still parse. The two NEW shapes (serial object, response header) are ignored-by-old-parsers by construction. Never change existing pointer/manifest fields. |
| +- `internal/release/laddercheck_test.go` pins production channel names and must stay green untouched. |
| +- Any test that shells the carlos binary MUST build env via `carlosTestEnv` (defined in `e2e_test.go:81`) AND set `cmd.Dir` to a scratch dir — the binary walks cwd ancestors for `.carlos/config`; a miss mints real accounts on the production console. |
| +- Console security-gate order (session → CSRF → owner 404 → freshAuth) is untouched by every task here; bump calls go AFTER a handler's successful write, never before its gates. |
| +- The in-process agent loops run inside `carlos-edge.service` (ProtectSystem=strict, ReadWritePaths=/data /run/carlos): never add `release.NagExpiredCanaries` or `release.WithCapabilities` to any in-process pass — carry the existing DO-NOT comments from `canaryAdoptLoop` into its successor verbatim. |
| +- Delegation note (CLAUDE.md): Tasks 1, 2, 7 are Sonnet-grade from this brief; Tasks 3, 4, 5 are converge-grade/root-adjacent (Opus/Fable per policy) and their review is Opus minimum; Task 9 is Sonnet with Opus review (it mints a user-facing verb). |
| + |
| +--- |
| + |
| +### Task 1: `control/serial` — the deploy signal object |
| + |
| +**Files:** |
| +- Create: `internal/release/serial.go` |
| +- Test: `internal/release/serial_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `release.Store` (Get/Put semantics: Get not-found is `os.ErrNotExist`-wrapped; Put ifMatch "" = create-only, non-empty = replace-only, race = `ErrCASConflict`). |
| +- Produces: `const SerialKey = "control/serial"`, `type SerialDoc struct { Schema int; Serial int64; BumpedAt string }` (JSON `schema`, `serial`, `bumped_at`), `func ReadSerial(ctx context.Context, s Store) (SerialDoc, string, error)` (doc, etag; missing object → zero doc, "" etag, nil error), `func BumpSerial(ctx context.Context, s Store, now time.Time) error`. Tasks 5 and 6 call these. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +package release |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + "time" |
| +) |
| + |
| +func TestSerialBumpFromMissing(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + doc, etag, err := ReadSerial(ctx, s) |
| + if err != nil || doc.Serial != 0 || etag != "" { |
| + t.Fatalf("missing serial should read as zero: %+v %q %v", doc, etag, err) |
| + } |
| + if err := BumpSerial(ctx, s, time.Unix(1754640000, 0).UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + doc, etag, err = ReadSerial(ctx, s) |
| + if err != nil || doc.Serial != 1 || doc.Schema != 1 || etag == "" { |
| + t.Fatalf("first bump: %+v %q %v", doc, etag, err) |
| + } |
| + if doc.BumpedAt == "" { |
| + t.Fatal("bumped_at not stamped") |
| + } |
| +} |
| + |
| +func TestSerialBumpIncrements(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + for i := 0; i < 3; i++ { |
| + if err := BumpSerial(ctx, s, time.Unix(1754640000, 0).UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + doc, _, _ := ReadSerial(ctx, s) |
| + if doc.Serial != 3 { |
| + t.Fatalf("serial = %d, want 3", doc.Serial) |
| + } |
| +} |
| + |
| +func TestSerialGarbageObjectStillBumps(t *testing.T) { |
| + // A bucket-writer can corrupt the object; a bump must recover, not wedge. |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + if _, err := s.Put(ctx, SerialKey, []byte("not json"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := BumpSerial(ctx, s, time.Unix(1754640000, 0).UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + doc, _, err := ReadSerial(ctx, s) |
| + if err != nil || doc.Serial != 1 { |
| + t.Fatalf("recovery bump: %+v %v", doc, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/release -run TestSerial -count=1` — expect compile failure (undefined: SerialKey). |
| + |
| +- [ ] **Step 3: Implement `internal/release/serial.go`** |
| + |
| +```go |
| +// Package-file doc: control/serial is the deploy signal (carlos-deploy spec |
| +// §3): an UNSIGNED counter the console bumps after any bucket write a box |
| +// loop consumes. It carries no authority — it can only trigger verified work |
| +// sooner — which is why it is the one object in the bucket outside the |
| +// Ed25519 envelope. Boxes react to any CHANGE, not only increments, so a |
| +// reset or a lost-increment race can never wedge a box into ignoring bumps. |
| +package release |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "time" |
| +) |
| + |
| +// SerialKey deliberately sits at the bucket top level: nothing that lists or |
| +// parses app shapes (ChannelsPrefix, appPrefix) ever sees it. |
| +const SerialKey = "control/serial" |
| + |
| +type SerialDoc struct { |
| + Schema int `json:"schema"` |
| + Serial int64 `json:"serial"` |
| + BumpedAt string `json:"bumped_at"` |
| +} |
| + |
| +// ReadSerial returns the current serial, its etag for CAS, and nil for a |
| +// missing object (zero doc, "" etag) — absence is the ordinary pre-first-bump |
| +// state, not an error. A present-but-unparseable object also reads as the |
| +// zero doc WITH its real etag, so the next bump overwrites the garbage. |
| +func ReadSerial(ctx context.Context, s Store) (SerialDoc, string, error) { |
| + body, etag, err := s.Get(ctx, SerialKey) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return SerialDoc{}, "", nil |
| + } |
| + if err != nil { |
| + return SerialDoc{}, "", err |
| + } |
| + var doc SerialDoc |
| + if json.Unmarshal(body, &doc) != nil { |
| + return SerialDoc{}, etag, nil // garbage: recoverable, next bump replaces it |
| + } |
| + return doc, etag, nil |
| +} |
| + |
| +// BumpSerial increments the serial with a small CAS retry loop. Callers must |
| +// treat failure as log-only: a lost bump costs latency (the hourly floor |
| +// still converges everything), never correctness, and must never fail the |
| +// user-facing write that triggered it. |
| +func BumpSerial(ctx context.Context, s Store, now time.Time) error { |
| + var lastErr error |
| + for attempt := 0; attempt < 4; attempt++ { |
| + doc, etag, err := ReadSerial(ctx, s) |
| + if err != nil { |
| + return err |
| + } |
| + next := SerialDoc{Schema: 1, Serial: doc.Serial + 1, BumpedAt: now.UTC().Format(time.RFC3339)} |
| + body, err := json.Marshal(next) |
| + if err != nil { |
| + return err |
| + } |
| + if _, err = s.Put(ctx, SerialKey, body, etag); err == nil { |
| + return nil |
| + } else if errors.Is(err, ErrCASConflict) { |
| + lastErr = err |
| + continue // someone else bumped — their change signal serves ours too, |
| + // but retry so OUR write's announcement isn't silently coalesced away |
| + } else { |
| + return err |
| + } |
| + } |
| + return fmt.Errorf("bump %s: lost CAS race %d times: %w", SerialKey, 4, lastErr) |
| +} |
| +``` |
| + |
| +(If `ErrCASConflict` has a different exported name in `internal/release`, use the store's actual sentinel — check `go doc ./internal/release | grep -i cas`.) |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/release -run TestSerial -count=1` — PASS. |
| +- [ ] **Step 5: Full gate, then commit** — `git add internal/release/serial.go internal/release/serial_test.go && git commit -m "release: control/serial — the deploy signal object (spec §3)"` |
| + |
| +--- |
| + |
| +### Task 2: content-accurate registry fingerprint (closes #15) |
| + |
| +**Files:** |
| +- Modify: `internal/registry/registry.go:986-1017` (the `Fingerprint` method) |
| +- Test: `internal/registry/registry_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Produces: same signature `func (r *Registry) Fingerprint() (string, error)` — an opaque string the edge watcher (`internal/edge/edge.go:822-831`, 2s poll) compares for change. Only the guarantee strengthens: any change to a reload-relevant column changes the string, including same-length rewrites. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +func TestFingerprintSeesSameLengthVersionSwap(t *testing.T) { |
| + reg := mustOpen(t) // use this file's existing open/seed helper naming |
| + rt := Route{Host: "fp.example.test", Kind: "instance", App: "echo", Status: "ready", Version: "abc1234"} |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, err := reg.Fingerprint() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetVersions("fp.example.test", "def5678", "abc1234"); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, err := reg.Fingerprint() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if before == after { |
| + t.Fatal("same-length version swap invisible to Fingerprint (issue #15)") |
| + } |
| +} |
| + |
| +func TestFingerprintSeesSameLengthChannelSwap(t *testing.T) { |
| + reg := mustOpen(t) |
| + rt := Route{Host: "fp2.example.test", Kind: "instance", App: "echo", Status: "ready", Channel: "canary/1"} |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, _ := reg.Fingerprint() |
| + rt.Channel = "canary/2" |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, _ := reg.Fingerprint() |
| + if before == after { |
| + t.Fatal("same-length channel swap invisible to Fingerprint (issue #15)") |
| + } |
| +} |
| +``` |
| + |
| +Adapt the registry-open helper to whatever `registry_test.go` actually uses (`New(t.TempDir()+"/r.db")` shape) — read the file's first test before writing. |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/registry -run TestFingerprintSees -count=1` — FAIL on `before == after`. |
| + |
| +- [ ] **Step 3: Reimplement Fingerprint** — same fresh-connection + busy_timeout preamble, then select values and hash in Go. Keep `grants` deliberately excluded (registry.go:93's rule: grants change unit shape, not routing). Replace the aggregate query with: |
| + |
| +```go |
| + rows, err := db.Query(` |
| +SELECT rowid, status, socket, addr, version, hibernate, channel, backing, tunnel |
| +FROM routes ORDER BY rowid`) |
| + if err != nil { |
| + return "", err |
| + } |
| + defer rows.Close() |
| + h := sha256.New() |
| + n := 0 |
| + for rows.Next() { |
| + var rowid, hibernate int64 |
| + var status, socket, addr, version, channel, backing, tunnel string |
| + if err := rows.Scan(&rowid, &status, &socket, &addr, &version, &hibernate, &channel, &backing, &tunnel); err != nil { |
| + return "", err |
| + } |
| + fmt.Fprintf(h, "%d\x1f%s\x1f%s\x1f%s\x1f%s\x1f%d\x1f%s\x1f%s\x1f%s\x1e", |
| + rowid, status, socket, addr, version, hibernate, channel, backing, tunnel) |
| + n++ |
| + } |
| + if err := rows.Err(); err != nil { |
| + return "", err |
| + } |
| + return fmt.Sprintf("%d:%x", n, h.Sum(nil)), nil |
| +``` |
| + |
| +Rewrite the doc comment: delete the "LENGTH … deliberate cost" paragraph and state the new contract — content-accurate over the reload-relevant columns, still a fresh connection per poll, row count folded in so an empty table and a never-opened one differ from populated ones. Note it closes #15 and why the deploy watch needs it (spec §7: the `X-Carlos-Version` header reads `Version`, and git short shas are same-length). |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/registry -count=1` — all PASS (existing fingerprint tests must still hold: insert/delete visibility). |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "registry: content-accurate Fingerprint — same-length rewrites reload the edge (closes #15)"` |
| + |
| +--- |
| + |
| +### Task 3: `release.MaterializeHost` + activator wiring — cold wake stages on demand |
| + |
| +**Files:** |
| +- Create: `internal/release/materialize.go` |
| +- Test: `internal/release/materialize_test.go` |
| +- Modify: `internal/host/agent.go:146-152` (the `ExecBackend` literal) and add `materializeHook` nearby |
| + |
| +**Interfaces:** |
| +- Consumes: `PassFilter(ctx, reg, s, pub, optDir, restart, extra, opts...)` (restart nil = symlink-only, never wakes), `LiveVersion(optDir, host)`, `BakeDir(dir) PassOption`, `activator.ExecBackend.Materialize func(ctx context.Context, host string) error` (called at wake when the live link is missing; backend_exec.go:364-379). |
| +- Produces: `func MaterializeHost(ctx context.Context, reg *registry.Registry, s Store, pub ed25519.PublicKey, optDir, dataDir, host string) error` — stages host's channel head via the REAL adopt pass scoped to one host, so every invariant (signature verification, bake hold, Version stamp) is the pass's own, not a second staging path. |
| + |
| +- [ ] **Step 1: Write the failing test** — build the fixture the way `internal/release/adopt_test.go` builds one (dir store seeded with a shipped version + promoted channel pointer, temp registry with one exec-backed ready route, temp optDir). Read that file first and reuse its helpers; the assertions: |
| + |
| +```go |
| +func TestMaterializeHostStagesChannelHead(t *testing.T) { |
| + // fixture: route "mat.test" on channel "canary/dev", version "abc1234" |
| + // shipped and promoted, nothing staged yet. |
| + env := newAdoptFixture(t) // whatever adopt_test.go's fixture builder is named |
| + err := MaterializeHost(context.Background(), env.reg, env.store, env.pub, env.optDir, env.dataDir, "mat.test") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + v, err := LiveVersion(env.optDir, "mat.test") |
| + if err != nil || v != "abc1234" { |
| + t.Fatalf("live = %q, %v; want abc1234", v, err) |
| + } |
| +} |
| + |
| +func TestMaterializeHostRefusesHeldHead(t *testing.T) { |
| + // fixture: production flag TRUE for the app, channel "beta" (24h hold), |
| + // version promoted moments ago → the pass must not stage it. |
| + env := newAdoptFixtureProductionBeta(t) |
| + err := MaterializeHost(context.Background(), env.reg, env.store, env.pub, env.optDir, env.dataDir, "mat.test") |
| + if err == nil { |
| + t.Fatal("materialize of a bake-held head must refuse") |
| + } |
| + if v, _ := LiveVersion(env.optDir, "mat.test"); v != "" { |
| + t.Fatalf("held head was staged anyway: %q", v) |
| + } |
| +} |
| + |
| +func TestMaterializeHostUnknownHost(t *testing.T) { |
| + env := newAdoptFixture(t) |
| + if err := MaterializeHost(context.Background(), env.reg, env.store, env.pub, env.optDir, env.dataDir, "nope.test"); err == nil { |
| + t.Fatal("unknown host must error") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/release -run TestMaterializeHost -count=1`. |
| + |
| +- [ ] **Step 3: Implement `materialize.go`** |
| + |
| +```go |
| +// MaterializeHost stages host's channel head and points its live symlink — |
| +// the cold-wake, on-demand form of the adopt pass (carlos-deploy spec §5). |
| +// It IS the adopt pass, filtered to one host with restart=nil (the host is |
| +// cold; the activator execing it is the "restart"): bake holds, signature |
| +// verification, staging, and the Version stamp are all the pass's own — |
| +// a second staging path is exactly the drift this refuses to create. |
| +// Success is judged by the symlink, not the pass error: Pass logs per-route |
| +// trouble rather than returning it, and "held by a bake window" and "nothing |
| +// promoted" both correctly leave the link absent. |
| +func MaterializeHost(ctx context.Context, reg *registry.Registry, s Store, pub ed25519.PublicKey, optDir, dataDir, host string) error { |
| + only := func(rt registry.Route) bool { return rt.Host == host } |
| + if err := PassFilter(ctx, reg, s, pub, optDir, nil, only, BakeDir(dataDir)); err != nil { |
| + return fmt.Errorf("materialize %s: %w", host, err) |
| + } |
| + v, err := LiveVersion(optDir, host) |
| + if err != nil { |
| + return fmt.Errorf("materialize %s: %w", host, err) |
| + } |
| + if v == "" { |
| + return fmt.Errorf("materialize %s: no adoptable head (nothing promoted for its channel, or the head is bake-held)", host) |
| + } |
| + return nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run to verify pass**, then wire the agent. In `agent.go`, add: |
| + |
| +```go |
| +// materializeHook builds the ExecBackend's cold-wake staging hook, or nil |
| +// when this box has no release store/pubkey — nil keeps today's behavior |
| +// ("no live binary" wake failure), the safe degrade. |
| +func materializeHook(reg *registry.Registry, dd string) func(context.Context, string) error { |
| + store, err := openDeployStore(context.Background()) |
| + if err != nil { |
| + return nil |
| + } |
| + pub, err := release.ParsePublicKey(os.Getenv("CARLOS_RELEASE_PUBKEY"), "CARLOS_RELEASE_PUBKEY") |
| + if err != nil { |
| + return nil |
| + } |
| + optDir := release.OptDir() |
| + return func(ctx context.Context, host string) error { |
| + return release.MaterializeHost(ctx, reg, store, pub, optDir, dd, host) |
| + } |
| +} |
| +``` |
| + |
| +and in the `ExecBackend` literal add `Materialize: materializeHook(reg, dd),`. |
| + |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "release+host: Materialize stages the channel head at cold wake (closes #77)"` |
| + |
| +--- |
| + |
| +### Task 4: backing-dispatched restart — warm exec children bounce on adopt |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go:1138-1144` (`restartRoute`) and the call sites `adoptLoop` / `canaryAdoptLoop` (which Task 5 replaces — coordinate: do this task first) |
| +- Test: `internal/host/agent_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: `registry.Registry.Get(host) (*Route, error)`, `registry.BackingUnit`, `activator.Service.Bounce(host string) bool`, `activator.UnitName(host)`. |
| +- Produces: `func makeRestartRoute(reg *registry.Registry, tryRestart func(host string), bounce func(host string) bool) func(host string) error` — injectable for tests; agent wires `systemctlTryRestart` and `svc.Bounce` (nil-safe). Task 5's loops take this closure. `adoptLoop` gains the `svc *activator.Service` parameter to build it. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +func TestMakeRestartRouteDispatchesOnBacking(t *testing.T) { |
| + reg := newTestRegistry(t) // agent_test.go's existing registry helper |
| + if err := reg.Upsert(registry.Route{Host: "unit.test", Kind: "instance", App: "a", Status: "ready", Backing: registry.BackingUnit}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.Upsert(registry.Route{Host: "exec.test", Kind: "instance", App: "a", Status: "ready"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + var restarted, bounced []string |
| + restart := makeRestartRoute(reg, |
| + func(h string) { restarted = append(restarted, h) }, |
| + func(h string) bool { bounced = append(bounced, h); return true }) |
| + if err := restart("unit.test"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := restart("exec.test"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(restarted) != 1 || restarted[0] != "unit.test" { |
| + t.Fatalf("unit path: %v", restarted) |
| + } |
| + if len(bounced) != 1 || bounced[0] != "exec.test" { |
| + t.Fatalf("exec path: %v", bounced) |
| + } |
| +} |
| + |
| +func TestMakeRestartRouteUnknownHostFallsBackToUnit(t *testing.T) { |
| + // A host the registry can't answer for gets the OLD behavior (try-restart, |
| + // a safe no-op for a nonexistent unit) — never a bounce it might not own. |
| + reg := newTestRegistry(t) |
| + var restarted []string |
| + restart := makeRestartRoute(reg, func(h string) { restarted = append(restarted, h) }, nil) |
| + if err := restart("ghost.test"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(restarted) != 1 { |
| + t.Fatalf("fallback: %v", restarted) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/host -run TestMakeRestartRoute -count=1`. |
| + |
| +- [ ] **Step 3: Implement.** Replace `restartRoute` with: |
| + |
| +```go |
| +// makeRestartRoute builds the adopt pass's restart callback, dispatched on |
| +// the route's backing (spec §4 correction): a unit tenant is try-restarted |
| +// (no-op when not running — hibernators are never woken), an exec child is |
| +// Bounced through the activator, the same graceful cycle config sync uses — |
| +// systemctl cannot reach a child of THIS process, which is why exec-backed |
| +// warm instances kept serving old binaries until they happened to hibernate. |
| +// Unknown-to-the-registry hosts take the unit path: try-restart of a |
| +// nonexistent unit is a logged no-op, a bounce of a process we might not own |
| +// is not. Errors are logged, never returned — a failed restart must not |
| +// poison the version for other hosts (adopt.go:352's rule). |
| +func makeRestartRoute(reg *registry.Registry, tryRestart func(host string), bounce func(host string) bool) func(host string) error { |
| + return func(host string) error { |
| + rt, err := reg.Get(host) |
| + if err == nil && rt != nil && rt.Backing != registry.BackingUnit { |
| + if bounce != nil && bounce(host) { |
| + log.Printf("adopt: bounced exec instance %s onto its new version", host) |
| + } |
| + return nil |
| + } |
| + tryRestart(host) |
| + return nil |
| + } |
| +} |
| + |
| +// systemctlTryRestart is makeRestartRoute's production unit path — the body |
| +// of the old restartRoute, unchanged. |
| +func systemctlTryRestart(host string) { |
| + out, err := exec.Command("systemctl", "try-restart", activator.UnitName(host)).CombinedOutput() |
| + if err != nil { |
| + log.Printf("adopt: try-restart %s: %v: %s (continuing)", activator.UnitName(host), err, strings.TrimSpace(string(out))) |
| + } |
| +} |
| +``` |
| + |
| +Wire: `adoptLoop(reg)` becomes `adoptLoop(reg, svc)`; inside, `restart := makeRestartRoute(reg, systemctlTryRestart, bounceFn(svc))` where `bounceFn(svc)` returns nil for nil svc, else `svc.Bounce`. Update the `go adoptLoop(reg)` call at agent.go:158 (and `canaryAdoptLoop` identically — Task 5 replaces it, but keep it compiling now). `RunAdopt` (root, separate process) is NOT changed — it keeps the pure systemctl restart: another process's children are not its to bounce. |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/host -count=1`. |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "host: adopt restart dispatches on backing — warm exec children bounce (spec §4)"` |
| + |
| +--- |
| + |
| +### Task 5: `fastConvergeLoop` — serial poll + fast set, replacing `canaryAdoptLoop` |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` — delete `canaryAdoptLoop` (lines ~565-613), add `fastConvergeLoop` + `fastTick`; change the spawn at line 160; keep `isCanaryRoute` (it stays, spec §4 correction) |
| +- Test: `internal/host/agent_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `release.ReadSerial`, Task 4's `makeRestartRoute`, `release.PassFilter`, `release.Pass`, `provision.Reconcile(ctx, store, reg, provisionTickConfig(os.Getenv))`, `syncExecSerialized(ctx, reg, store, priv, dd, svc)` (agent.go:304 — the ONLY way to run SyncExec, mutex inside), `release.LiveVersion(optDir, host)`. |
| +- Produces: `fastConvergeLoop(reg *registry.Registry, svc *activator.Service)` goroutine; `type fastDeps struct` + `func fastTick(ctx context.Context, d *fastDeps, now time.Time)` — the testable single tick. |
| + |
| +**Design (from spec §3-§4):** every 2s+jitter tick does two independent things: |
| +1. **Serial check** — one `ReadSerial`. First successful read only SEEDS `lastSerial` (boot must not replay history; `adoptLoop`'s 1-minute first pass covers boot). Afterward, any change → if ≥5s since last full pass-set: run **reconcile → adopt pass → exec config sync** in that order (mint routes, then stage them, then their config), record the serial as acted; if clamped, do NOT record — next tick re-fires. Config sync piece is skipped when `CARLOS_SECRETS_KEY` doesn't parse (same degrade as `configSyncLoop`). |
| +2. **Fast sweep** — membership per route: `isCanaryRoute(rt)` (always — direct-bucket operator promotes never bump, spec §4 correction) OR (`release.LiveVersion(optDir, rt.Host)` == "" AND host's backoff window has passed). Compute the member set FIRST; skip `PassFilter` entirely when empty (an empty sweep must cost zero store reads). After the sweep, hosts still unstaged double their backoff 2s→5min cap; staged hosts clear their backoff state. Held routes cost nothing here: the pass's own bake logic refuses them and the backoff keeps re-checks rare; the serial/hourly passes re-notice them when the window expires. |
| + |
| +- [ ] **Step 1: Write the failing tests** (dir store + temp registry, like existing agent tests; `fastDeps` carries every seam so no env vars needed): |
| + |
| +```go |
| +func TestFastTickActsOnSerialChangeNotBoot(t *testing.T) { |
| + env := newFastFixture(t) // store+reg+pub+optDir fixture; build like Task 3's |
| + var passes int |
| + d := env.deps(func() { passes++ }) // deps with the full-pass-set hook counted |
| + now := time.Unix(1754640000, 0) |
| + fastTick(context.Background(), d, now) // no serial object at all: seeds nothing, runs nothing |
| + release.BumpSerial(context.Background(), env.store, now) |
| + fastTick(context.Background(), d, now.Add(2*time.Second)) // first sight: SEED only |
| + if passes != 0 { |
| + t.Fatalf("boot seed must not run a pass, got %d", passes) |
| + } |
| + release.BumpSerial(context.Background(), env.store, now) |
| + fastTick(context.Background(), d, now.Add(4*time.Second)) |
| + if passes != 1 { |
| + t.Fatalf("serial change must run the pass set once, got %d", passes) |
| + } |
| + fastTick(context.Background(), d, now.Add(6*time.Second)) |
| + if passes != 1 { |
| + t.Fatalf("unchanged serial must not re-run, got %d", passes) |
| + } |
| +} |
| + |
| +func TestFastTickClampsFullPasses(t *testing.T) { |
| + env := newFastFixture(t) |
| + var passes int |
| + d := env.deps(func() { passes++ }) |
| + now := time.Unix(1754640000, 0) |
| + release.BumpSerial(context.Background(), env.store, now) |
| + fastTick(context.Background(), d, now) // seed |
| + release.BumpSerial(context.Background(), env.store, now) |
| + fastTick(context.Background(), d, now.Add(2*time.Second)) // pass 1 |
| + release.BumpSerial(context.Background(), env.store, now) |
| + fastTick(context.Background(), d, now.Add(4*time.Second)) // inside 5s clamp: deferred |
| + if passes != 1 { |
| + t.Fatalf("clamp: got %d passes", passes) |
| + } |
| + fastTick(context.Background(), d, now.Add(8*time.Second)) // clamp expired: fires |
| + if passes != 2 { |
| + t.Fatalf("deferred bump must fire after clamp, got %d", passes) |
| + } |
| +} |
| + |
| +func TestFastSweepMembership(t *testing.T) { |
| + env := newFastFixture(t) |
| + // three routes: a canary (always member), a staged edge route (not a |
| + // member), an unstaged edge route (member until staged, then drops out). |
| + env.addRoute(t, "canary.test", "canary/x") |
| + env.addRoute(t, "staged.test", "edge") |
| + env.stageLive(t, "staged.test", "abc1234") // mint its live symlink |
| + env.addRoute(t, "cold.test", "edge") |
| + members := env.deps(nil).fastMembers(time.Unix(1754640000, 0)) |
| + sort.Strings(members) |
| + if want := []string{"canary.test", "cold.test"}; !slices.Equal(members, want) { |
| + t.Fatalf("members = %v, want %v", members, want) |
| + } |
| +} |
| + |
| +func TestFastSweepBackoffDoubles(t *testing.T) { |
| + env := newFastFixture(t) |
| + env.addRoute(t, "broken.test", "edge") // nothing promoted: staging can never succeed |
| + d := env.deps(nil) |
| + now := time.Unix(1754640000, 0) |
| + d.noteUnstaged("broken.test", now) |
| + if m := d.fastMembers(now.Add(1 * time.Second)); len(m) != 0 { |
| + t.Fatalf("inside backoff window, want no members, got %v", m) |
| + } |
| + if m := d.fastMembers(now.Add(3 * time.Second)); len(m) != 1 { |
| + t.Fatalf("past first backoff, want member, got %v", m) |
| + } |
| + for i := 0; i < 20; i++ { // drive to the cap |
| + d.noteUnstaged("broken.test", now) |
| + } |
| + if d.backoffFor("broken.test") != 5*time.Minute { |
| + t.Fatalf("backoff cap = %v, want 5m", d.backoffFor("broken.test")) |
| + } |
| +} |
| +``` |
| + |
| +Shape `fastDeps`/`fastMembers`/`noteUnstaged`/`backoffFor` to make exactly these tests natural; the loop is a thin wrapper (sleep-jitter, build deps once, call `fastTick`). |
| + |
| +- [ ] **Step 2: Run to verify failure.** |
| +- [ ] **Step 3: Implement.** Structure: |
| + |
| +```go |
| +// fastDeps is one fastConvergeLoop's state and seams — a struct so the tick |
| +// is testable without goroutines, env vars, or sleeps. |
| +type fastDeps struct { |
| + reg *registry.Registry |
| + store release.Store |
| + pub ed25519.PublicKey |
| + optDir string |
| + restart func(string) error |
| + fullPass func(ctx context.Context) // reconcile → adopt → config sync, or a test hook |
| + lastSerial int64 |
| + seeded bool |
| + lastFull time.Time |
| + notBefore map[string]time.Time |
| + step map[string]time.Duration |
| +} |
| +``` |
| + |
| +`fastTick`: serial check per the design block above, then `members := d.fastMembers(now)`; if non-empty run `release.PassFilter(ctx, d.reg, d.store, d.pub, d.optDir, d.restart, memberPredicate, release.BakeDir(dataDir()))`, then re-scan members: still-unstaged → `d.noteUnstaged(host, now)`, staged → clear both maps for host. `fastConvergeLoop` mirrors `canaryAdoptLoop`'s setup (openDeployStore/ParsePublicKey silent-return degrade, 2s+jitter sleep) and its armed log line becomes `"agent: fast converge loop armed (2s + jitter: serial poll + canary/* + unstaged routes)"`. The production `fullPass` runs `provision.Reconcile(ctx, store, reg, provisionTickConfig(os.Getenv))`, then `release.Pass(ctx, reg, store, pub, optDir, restart, release.BakeDir(dataDir()))`, then — only if `CARLOS_SECRETS_KEY` parses — `syncExecSerialized(...)`; each error is one log line, never fatal. **Carry the two DO-NOT comment blocks (NagExpiredCanaries / WithCapabilities) from `canaryAdoptLoop` verbatim onto `fastConvergeLoop`** — same sandbox, same 2s cadence, same reasons. Update the spawn: `go fastConvergeLoop(reg, svc)`. |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/host -count=1`; also `go test ./internal/release -run TestLadder -count=1` (channel names untouched). |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "host: fastConvergeLoop — serial poll + outstanding-work sweep replaces canary-only fast adopt (#101)"` |
| + |
| +--- |
| + |
| +### Task 6: console bumps the serial; promote API says `held` |
| + |
| +**Files:** |
| +- Create: `internal/console/serialbump.go` |
| +- Modify: `internal/console/apicli.go` (`handleAPIPromote:289`, `handleAPIRollback:377`), `internal/console/apiinstances.go` (`handleAPIInstanceCreate:214`, `handleAPIInstanceEnable:440`, `handleAPIInstanceDelete:331`), `internal/console/apiconfig.go` (`handleAPIConfigPut:274`), `internal/console/server.go` (the web promote handler containing `s.audit(acct, app, addr, "promote", ...)` at :1740, the web rollback handler `handleRollback`, `handleProduction:1872`, and the fast-track write if it is a separate handler — find every successful `release.PromoteOnLadder` / `release.SetProduction` / instance-record / config write with `grep -n "PromoteOnLadder\|SetProduction\|instancerecords\.\|appconfig\." internal/console/*.go` and cover each) |
| +- Test: `internal/console/serialbump_test.go` + one assertion added to the existing promote-API test |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `release.BumpSerial`; `s.cfg.Releases` (the console's release store), `s.cfg.Key` (root signing key; its `.Public().(ed25519.PublicKey)` verifies production flags), `release.GetProduction(ctx, s, pub, acct, app) (*ProductionFlag, error)`, `release.Rung(channel)` / `release.RungBeta`. |
| +- Produces: `func (s *Server) bumpSerial(ctx context.Context)` — log-only on failure, 2s timeout, called after each successful box-consumed write; promote API response gains additive `"held": true|false`; client task 9 reads it. |
| + |
| +- [ ] **Step 1: Write the failing tests.** `serialbump_test.go`: build a `Server` the way the smallest existing console API test does (read one first — there is an established fixture), call the promote API against a seeded app/version, then `release.ReadSerial` on the test store and assert `Serial == 1`. Second test: flip the app to production (write the flag with `release.SetProduction` using the test key), promote to `beta`, assert response JSON carries `"held": true`; promote to a `canary/*` channel asserts `"held": false` (or absent — decode into `struct{ Held bool }`). |
| +- [ ] **Step 2: Run to verify failure.** |
| +- [ ] **Step 3: Implement.** `serialbump.go`: |
| + |
| +```go |
| +// bumpSerial announces "the bucket has news" (carlos-deploy spec §3) after a |
| +// write a box loop consumes. Log-only on failure and clamped to 2s: a lost |
| +// bump costs latency (the hourly floor still converges), and no user-facing |
| +// write may fail or stall because its announcement did. |
| +func (s *Server) bumpSerial(ctx context.Context) { |
| + ctx, cancel := context.WithTimeout(ctx, 2*time.Second) |
| + defer cancel() |
| + if err := release.BumpSerial(ctx, s.cfg.Releases, time.Now()); err != nil { |
| + log.Printf("console: serial bump: %v (boxes fall back to the hourly pass)", err) |
| + } |
| +} |
| +``` |
| + |
| +Call it right after each handler's successful write (next to the existing `s.audit` calls — after every gate, per the security-order constraint). In `handleAPIPromote`, before the final `writeJSON`, compute `held`: |
| + |
| +```go |
| + held := false |
| + if rung, _ := release.Rung(channel); rung == release.RungBeta { |
| + pub := s.cfg.Key.Public().(ed25519.PublicKey) |
| + if f, err := release.GetProduction(ctx, s.cfg.Releases, pub, acct.Sqid, app); err == nil && f != nil && f.Production { |
| + held = true |
| + } |
| + } |
| + writeJSON(w, http.StatusOK, map[string]any{ |
| + "result": "promoted", "from": res.From, "to": res.To, "noop": false, "held": held, |
| + }) |
| +``` |
| + |
| +(Stable never reaches here — the rung gate 403s it to the console; beta is the only bakeable rung this door can move. Verify `ProductionFlag`'s bool field name with `go doc ./internal/release ProductionFlag` and adjust.) |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/console -count=1`. |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "console: bump control/serial on box-consumed writes; promote API reports held"` |
| + |
| +--- |
| + |
| +### Task 7: `X-Carlos-Version` response header |
| + |
| +**Files:** |
| +- Modify: `internal/edge/edge.go` — in `Reload()`'s per-route switch (lines ~236-296), after `proxy` is chosen |
| +- Test: `internal/edge/edge_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: `registry.Route.Version` (stamped by the adopt pass after restart — serving truth; kept fresh by Task 2's fingerprint). |
| +- Produces: `X-Carlos-Version: <version>` on every response for routes with non-empty `Version` (all backings: proxied, static, status). Task 9's watch parses it; header absent = "cannot confirm", never a match. |
| + |
| +- [ ] **Step 1: Write the failing test** — follow the existing pattern in `edge_test.go` that builds an `Edge` over a registry and drives a request through it (find one with `httptest`); assert: |
| + |
| +```go |
| +func TestVersionHeaderOnVersionedRoute(t *testing.T) { |
| + // route with Version "abc1234" over a local httptest backend |
| + // → response carries X-Carlos-Version: abc1234 |
| +} |
| + |
| +func TestNoVersionHeaderOnUnversionedRoute(t *testing.T) { |
| + // route with Version "" → header absent |
| +} |
| +``` |
| + |
| +Write both bodies concretely against the fixture pattern the file already uses (do not invent a new harness). |
| + |
| +- [ ] **Step 2: Run to verify failure.** |
| +- [ ] **Step 3: Implement.** In `Reload()`, after the `switch` assigns `proxy` (and before the map insert), wrap: |
| + |
| +```go |
| + // The deploy watch's proof and the human "verify with the binary you |
| + // built" tell (carlos-deploy spec §7): the version this route is |
| + // serving, from the row the adopt pass stamps AFTER the restart. |
| + // Server:-header-grade disclosure, ruled public 2026-08-08. Reload |
| + // rebuilds these handlers on every registry change — Fingerprint is |
| + // content-accurate since #15 closed, so the header can't go stale. |
| + if t.Version != "" { |
| + proxy = versionHeader(t.Version, proxy) |
| + } |
| +``` |
| + |
| +with, near the other small handler builders: |
| + |
| +```go |
| +func versionHeader(ver string, next http.Handler) http.Handler { |
| + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("X-Carlos-Version", ver) |
| + next.ServeHTTP(w, r) |
| + }) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test ./internal/edge -count=1`. |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "edge: X-Carlos-Version on versioned routes — the deploy watch's proof (spec §7)"` |
| + |
| +--- |
| + |
| +### Task 8: `staged_version` in the instances API (chips at #78) |
| + |
| +**Files:** |
| +- Modify: `internal/status/status.go` (the raw `Metrics` block `CollectBox` fills — read the type first; it is the deliberate bucket-private exception to the banding rule), `internal/console/apiinstances.go:157-178` (`handleAPIInstanceList`), root `instances_cli.go:255-280` (`runInstancesList` gains a VERSION column), `apimode.go` (`apiInstanceRecord` gains `StagedVersion string \`json:"staged_version"\``) |
| +- Test: `internal/status/status_test.go`, `internal/console/apiinstances_test.go` (append to each) |
| + |
| +**Interfaces:** |
| +- Consumes: `registry.Route.Version`; `status/boxes/<label>.json` objects (bucket-private, raw Metrics allowed); the console's `s.cfg.Releases` store reads them via `List(ctx, "status/boxes/")` + `Get`. |
| +- Produces: `Metrics.RouteVersions map[string]string` (host → Version, non-platform routes with Version set) in the box snapshot; instances API records gain additive `staged_version` (empty when no box has reported — freshness is the 5-minute status cadence, explicitly best-effort; the deploy watch does NOT use it). |
| + |
| +- [ ] **Step 1: status side, TDD** — test that `CollectBox` over a registry with a versioned route lands `RouteVersions{"h.test": "abc1234"}` in the written `status/boxes/<label>.json`, and — critically — extend the existing `DryRun`/needle privacy test's expectations ONLY if it fails: `RouteVersions` lives in `Metrics`, the block `BuildFleet` never copies, so the public `fleet.json` must NOT contain any hostname from it (the existing needle test should already enforce this — run it and confirm it passes untouched; if it flags the new field, the field is in the wrong place, fix the placement, not the test). |
| +- [ ] **Step 2: console side, TDD** — seed the console fixture's store with a `status/boxes/box1.json` carrying `RouteVersions`, call the instances list API, assert the matching record has `staged_version: "abc1234"` and a record with no box entry has `""`. Missing/garbage box files: skipped with a log line, never a 500 — the list must serve without status objects at all. |
| +- [ ] **Step 3: CLI column** — `runInstancesList` header becomes `HOST\tREGION\tCHANNEL\tPOOL\tVERSION\tCREATED\tBY`, printing `r.StagedVersion`. No test beyond compile (table printing has no harness); the API-side test covers the data. |
| +- [ ] **Step 4: Full gate, then commit** — `git commit -m "status+console: staged_version surfaces through the instances API (#78)"` |
| + |
| +--- |
| + |
| +### Task 9: the `carlos deploy` verb |
| + |
| +**Files:** |
| +- Create: `deploy.go` (root package), `deploy_test.go` |
| +- Modify: `main.go` (dispatch case + usage line, in the "shipping your own app" group), `apimode.go` (`apiPromoteResult` gains `Held bool \`json:"held"\``) |
| + |
| +**Interfaces:** |
| +- Consumes: `resolveMode`/`requireAPISession("deploy", *consoleFlag)`, `accountFlags(fs)`, `apiAccount(acctRaw, acctIDRaw)`, `readCapabilities(*capsFile, *app)`, `runShipAPI(sess, acct, fs, app, ver, kind, target, capsSet) error` (cli.go:1592 — reuse, do not duplicate), `sess.promote(ctx, acct, app, version, channel) (apiPromoteResult, error)`, `sess.instanceList(ctx, acct, app) ([]apiInstanceRecord, bool, error)` (records carry `Host`, `Channel`), Task 6's `held` field, Task 7's header, `printUsing(stdout, sess)`, `colorFor(stdout)`/`mutedf`. |
| +- Produces: `carlos deploy` — see spec §6 for the exact contract. Helper seams for tests: `resolveDeployChannel(recs []apiInstanceRecord, hostFlag string) (channel string, hosts []string, err error)` and `watchVersion(w io.Writer, targets []watchTarget, ver string, timeout, interval time.Duration) error` with `type watchTarget struct{ Host, Origin string }`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestResolveDeployChannel(t *testing.T) { |
| + recs := []apiInstanceRecord{ |
| + {Host: "a.test", Channel: "canary/dev"}, |
| + {Host: "b.test", Channel: "canary/dev"}, |
| + } |
| + ch, hosts, err := resolveDeployChannel(recs, "") |
| + if err != nil || ch != "canary/dev" || len(hosts) != 2 { |
| + t.Fatalf("%q %v %v", ch, hosts, err) |
| + } |
| + recs[1].Channel = "edge" |
| + if _, _, err := resolveDeployChannel(recs, ""); err == nil { |
| + t.Fatal("two channels with no -channel/-host must error") |
| + } |
| + ch, hosts, err = resolveDeployChannel(recs, "b.test") |
| + if err != nil || ch != "edge" || len(hosts) != 1 || hosts[0] != "b.test" { |
| + t.Fatalf("host-scoped: %q %v %v", ch, hosts, err) |
| + } |
| + if _, _, err := resolveDeployChannel(nil, ""); err == nil { |
| + t.Fatal("no instances must error (hint: carlos instances create)") |
| + } |
| +} |
| + |
| +func TestWatchVersionSucceedsWhenHeaderMatches(t *testing.T) { |
| + var mu sync.Mutex |
| + ver := "old1234" |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + mu.Lock() |
| + w.Header().Set("X-Carlos-Version", ver) |
| + mu.Unlock() |
| + })) |
| + defer srv.Close() |
| + go func() { time.Sleep(30 * time.Millisecond); mu.Lock(); ver = "new5678"; mu.Unlock() }() |
| + var out bytes.Buffer |
| + err := watchVersion(&out, []watchTarget{{Host: "x.test", Origin: srv.URL}}, "new5678", |
| + 2*time.Second, 10*time.Millisecond) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !strings.Contains(out.String(), "x.test") { |
| + t.Fatalf("live line missing: %q", out.String()) |
| + } |
| +} |
| + |
| +func TestWatchVersionTimeoutDiagnostics(t *testing.T) { |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("X-Carlos-Version", "old1234") |
| + })) |
| + defer srv.Close() |
| + var out bytes.Buffer |
| + err := watchVersion(&out, []watchTarget{{Host: "x.test", Origin: srv.URL}}, "new5678", |
| + 50*time.Millisecond, 10*time.Millisecond) |
| + if err == nil { |
| + t.Fatal("must time out") |
| + } |
| + if !strings.Contains(err.Error(), "old1234") { |
| + t.Fatalf("diagnostic must name the version still serving: %v", err) |
| + } |
| +} |
| + |
| +func TestWatchVersionNoHeaderDiagnostic(t *testing.T) { |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})) |
| + defer srv.Close() |
| + var out bytes.Buffer |
| + err := watchVersion(&out, []watchTarget{{Host: "x.test", Origin: srv.URL}}, "new5678", |
| + 50*time.Millisecond, 10*time.Millisecond) |
| + if err == nil || !strings.Contains(err.Error(), "no X-Carlos-Version") { |
| + t.Fatalf("old-edge diagnostic: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test . -run 'TestResolveDeployChannel|TestWatchVersion' -count=1`. |
| + |
| +- [ ] **Step 3: Implement `deploy.go`.** Flags: `-app`, account flags, `-console`, `-kind binary|static`, `-version` (default: `git rev-parse --short HEAD` via a `gitShortHead()` helper that `log.Fatal`s with `"not in a git repository — pass -version"` on failure), `-target linux-arm64`, `-capabilities`, `-channel`, `-host`, `-timeout 120s`, positional files-or-site-dir (required). Flow, straight from spec §6: |
| + |
| +```go |
| + sess := requireAPISession("deploy", *consoleFlag) // API-mode only: member verb |
| + acct := apiAccount(acctRaw, acctIDRaw) |
| + printUsing(stdout, sess) |
| + ctx := context.Background() |
| + recs, _, err := sess.instanceList(ctx, acct, *app) |
| + if err != nil { |
| + log.Fatalf("deploy: list instances: %v", err) |
| + } |
| + var channel string |
| + var hosts []string |
| + if *channelFlag != "" { |
| + // Explicit channel: no ambiguity to resolve. Watch the instances that |
| + // follow it (none is fine — promote still lands; the watch is skipped |
| + // with a note, since there is nothing that will serve this channel). |
| + channel = *channelFlag |
| + for _, r := range recs { |
| + if r.Channel == channel && (*hostFlag == "" || r.Host == *hostFlag) { |
| + hosts = append(hosts, r.Host) |
| + } |
| + } |
| + } else { |
| + channel, hosts, err = resolveDeployChannel(recs, *hostFlag) |
| + if err != nil { |
| + log.Fatalf("deploy: %v", err) |
| + } |
| + } |
| + if err := runShipAPI(sess, acct, fs, *app, *ver, *kind, *target, capsSet); err != nil { |
| + log.Fatalf("deploy: ship: %v", err) |
| + } |
| + fmt.Fprintf(stdout, "%s %s %s\n", colorFor(stdout).ok("shipped"), *app, *ver) |
| + res, err := sess.promote(ctx, acct, *app, *ver, channel) |
| + if err != nil { |
| + log.Fatalf("deploy: promote: %v", err) // a stable 403's console_url surfaces in err |
| + } |
| + fmt.Fprintf(stdout, "%s %s %s -> %s\n", colorFor(stdout).ok("promoted"), *app, *ver, channel) |
| + if res.Held { |
| + mutedf(stdout, "%s is a production app: the %s bake window applies — the fleet adopts when it expires; console fast-track if this is an emergency\n", *app, channel) |
| + return // exit 0: the writes succeeded, the hold is policy (spec §6.6) |
| + } |
| + if len(hosts) == 0 { |
| + mutedf(stdout, "no instance follows %s — promoted, nothing to watch (carlos instances create to add one)\n", channel) |
| + return |
| + } |
| + targets := make([]watchTarget, 0, len(hosts)) |
| + for _, h := range hosts { |
| + targets = append(targets, watchTarget{Host: h, Origin: "https://" + h}) |
| + } |
| + if err := watchVersion(stdout, targets, *ver, *timeout, 2*time.Second); err != nil { |
| + log.Fatalf("deploy: %v", err) |
| + } |
| +``` |
| + |
| +`resolveDeployChannel`: error text for the ambiguous case lists each channel with its hosts and says `"pass -channel or -host"`; the empty case says `"no instances — create one: carlos instances create -app <a> ..."`. `watchVersion`: GET `Origin + "/"`, drain ≤4KiB and close, compare `X-Carlos-Version`; on match print `live https://<host>` (ok color) and drop the target; sleep `interval` between rounds; on deadline, one error naming each pending host's last observation — `still serving <v>`, `no X-Carlos-Version header (edge predates this feature — the hourly pass will adopt it)`, or `unreachable: <err>` (route not minted yet / wake failing; point at `carlos instances -app <a>`). Add the `main.go` case (`case "deploy": runDeploy(args)`) and a usage line directly under `ship`/`promote`: `deploy ship + promote + wait until the URL serves the new version`. |
| + |
| +- [ ] **Step 4: Run to verify pass** — `go test . -count=1`. This package's tests shell the binary in places — if you add any test that does, `carlosTestEnv` + scratch `cmd.Dir` are mandatory (Global Constraints). |
| +- [ ] **Step 5: Full gate, then commit** — `git commit -m "cli: carlos deploy — ship, promote, and watch until the URL serves it (spec §6)"` |
| + |
| +--- |
| + |
| +### Task 10: whole-branch verification |
| + |
| +- [ ] **Step 1:** `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && go vet ./... && go test ./... -count=1` — every package, clean. |
| +- [ ] **Step 2:** `GOOS=darwin go build ./...` — the Mac CLI gate. |
| +- [ ] **Step 3:** `gofmt -l .` — nothing new (the three known pre-existing files only). |
| +- [ ] **Step 4:** Spec sweep — reread `docs/superpowers/specs/2026-08-08-carlos-deploy-design.md` §2-§7 and confirm each shape landed: serial object + console bumps + `held`; fast set (canary ∪ unstaged, backoff, clamp, seed-not-replay); Materialize refusing held heads; header + fingerprint; deploy verb incl. bake-held exit and diagnostics; `staged_version`. Confirm `internal/release/laddercheck_test.go` is untouched (`git diff main --stat -- internal/release/laddercheck_test.go` is empty). |
| +- [ ] **Step 5:** Commit anything outstanding; the branch is ready for review (superpowers:requesting-code-review, then superpowers:finishing-a-development-branch). |
| + |
| +## Follow-ups deliberately NOT in this plan |
| + |
| +- pinfra `ship-app.sh`/`promote-app.sh` learning to bump the serial (spec §8 non-goal; flagship script deploys stay hourly-floor + canary-fast). |
| +- Website/tap batch for the new verb (standing rule: same-batch on release, owed at deploy time). |
| +- Deploy rollout to the boxes (spec §10: ship → adopt → restart `carlos-edge.service`, both boxes, verify via the header itself) — operator work after merge. |
| +- Issue #95 verification (serial bump on config writes may cover it — test live, then close or keep). |
amadan