amadan

carlos / platform Public

plan: sidecar members — 16 tasks, TDD, gate-per-task

Paul Campbell pushed by paul@keymail.dev ae1e462c831af388f8c2ea32b7fae9afb7a7708c parent 1c6faf8
1 file changed, +2101 −0
  • docs/superpowers/plans/2026-08-17-sidecar-members.md +2101 −0
diff --git a/docs/superpowers/plans/2026-08-17-sidecar-members.md b/docs/superpowers/plans/2026-08-17-sidecar-members.md
new file mode 100644
index 000000000..77e0f31fd
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-17-sidecar-members.md
@@ -0,0 +1,2101 @@
+# Sidecar Members Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make a sidecar a first-class platform member — provisioned through
+an API like a web app instance, unreachable from the internet by
+construction, hibernating with timer and command wakes, placeable on a box
+with no inbound path at all.
+
+**Architecture:** Mirror instance provisioning one directory over
+(`sidecars.json` manifest + `sidecars/<name>.json` create-only records +
+a reconcile pass + console verbs + an app-held token), key sidecars by a
+synthetic `<name>.<app>.<sqid>.sidecar.internal` host so every host-keyed
+subsystem works unchanged, and reuse `activator.Service` for the lifecycle
+while keeping sidecars entirely off its request path. A sidecar binds
+nothing; it reports readiness, next-due and idleness by dialing **out** to a
+new agent-side unix socket.
+
+**Tech Stack:** Go (stdlib only — no new dependencies), modernc sqlite
+(`internal/registry`), S3/dir-backed `release.Store`, litestream via
+`internal/lease`.
+
+**Spec:** `docs/superpowers/specs/2026-08-17-sidecar-members-design.md`
+
+## Global Constraints
+
+- **The gate, before every commit:** `go build ./...`, `go vet ./...`,
+ `go test ./... -count=1` all clean, and `gofmt -l` clean for files you
+ touched. Three pre-existing unformatted files elsewhere are left alone.
+- **Sandbox note:** use `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod
+ GOPROXY=off`. Go commands may need the sandbox disabled ("read-only file
+ system" from the toolchain is the tell).
+- **The gate is not Linux-only.** `internal/host` links into the CLI, which
+ people run on Macs. `GOOS=darwin go build ./...` must pass too.
+- **Wire-shape changes are additive.** Live pointers, manifests and registry
+ rows exist in production buckets and on production boxes; anything written
+ before your change must still parse. Never remove or repurpose a field.
+- **When a storage shape changes, the consumer list is everything that
+ PARSES the shape** — not everything that calls the API.
+- **No new dependencies.** Standard library only.
+- **Never touch a live box.** This branch is build-and-PR only. No deploys,
+ no `ssm`, no `aws` calls, no writes to any real bucket.
+- **Reserved suffix, exact string:** `.sidecar.internal`
+- **Synthetic host format, exact:** `<name>.<app>.<sqid>.sidecar.internal`
+- **Registry Kind value, exact:** `sidecar`
+- **Agent socket path, exact:** `/run/carlos/agent.sock`
+- **Env var names, exact:** `CARLOS_AGENT_SOCKET`, `CARLOS_SIDECAR_TOKEN`,
+ `CARLOS_SIDECAR_NAME`, `CARLOS_SIDECAR_HOST`, `CARLOS_INBOUND`
+- **Bucket keys, exact:** `apps/<sqid>/<app>/sidecars.json`,
+ `apps/<sqid>/<app>/sidecars/<name>.json`,
+ `apps/<sqid>/<app>/sidecar-wake/<name>.json`
+- **Defaults, exact:** idle `5m`, `max_run` `1h`, settle window `10s`,
+ hibernate `true`, placement `any`.
+- **Commit style:** one commit per task, imperative subject, body explaining
+ *why*. End every commit message with:
+ ```
+ Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
+ Claude-Session: https://claude.ai/code/session_014WvcQVWggXkE5BAkjRDgWF
+ ```
+
+---
+
+## File Structure
+
+**Created:**
+- `internal/provision/sidecar.go` — sidecar manifest + record wire shapes,
+ key helpers, validation, store operations. Sibling to `provision.go`,
+ same package: it is the same kind of object one directory over, and it
+ reuses `labelRe`, `ValidRegion`, `ValidPool` directly.
+- `internal/provision/sidecar_test.go`
+- `internal/provision/sidecar_reconcile.go` — the records→routes pass.
+- `internal/provision/sidecar_reconcile_test.go`
+- `internal/agentsock/agentsock.go` — the agent-side unix socket: listener,
+ per-wake token registry, the four verbs. Its own package because it is a
+ server with its own lifecycle, and because `internal/activator` must be
+ able to depend on it without a cycle.
+- `internal/agentsock/agentsock_test.go`
+- `internal/release/sidecarwake.go` — the wake generation object.
+- `internal/release/sidecarwake_test.go`
+- `internal/console/apisidecars.go` — the console CLI verbs.
+- `internal/console/apisidecars_test.go`
+- `sidecars_cli.go` (root package) — `carlos sidecars`.
+- `sidecars_cli_test.go`, `sidecars_e2e_test.go` (root package)
+- `infra/no-inbound-box.md` — the private-subnet runbook (Paul applies).
+
+**Modified:**
+- `internal/registry/registry.go` — `args` column (FOUR places), `SetArgs`,
+ `DueInstances` widening.
+- `internal/edge/edge.go` — structural sidecar refusal.
+- `internal/host/fleetbind.go` — never publish a sidecar binding.
+- `internal/host/restart.go` — cycle sidecars.
+- `internal/host/agent.go` — `CARLOS_INBOUND`, the sidecar reconcile leg,
+ the agent socket's lifecycle, the wake pass.
+- `internal/appconfig/syncexec.go` — deliver config to sidecars.
+- `internal/activator/activator.go` — `StartWorker` seam, `WakeWorker`,
+ socketless stop, `max_run`.
+- `internal/activator/backend_exec.go`, `backend_unit.go` — the seam's two
+ implementations.
+- `internal/provision/reconcile.go` — instance removal phase skips sidecars.
+- `internal/console/authcli.go` — route registration.
+- `internal/console/instances.go`, `overview.go` — Kind guards, sidecar words.
+- `internal/status/snapshot.go` — sidecars are not public Residents.
+- `cli.go` (root) — `carlos add` rails off `-kind sidecar` and the suffix.
+- `main.go` (root) — `sidecars` dispatch.
+- `apimode.go` (root) — the client methods.
+- `CLAUDE.md` — the co-spawn/native split.
+
+---
+
+## Task ordering and dependencies
+
+```
+1 registry ──┬── 3 edge/fleetbind ── 4 suffix rails
+ ├── 5 StartWorker ── 6 lifecycle ── 7 agentsock ── 8 wiring
+ ├── 11 config+restart
+ └── 15 console UI
+2 provision shapes ──┬── 9 reconciler ── 10 placement
+ ├── 12 console API ── 14 CLI
+ └── 13 wake object
+16 docs (last)
+```
+
+Tasks 1 and 2 are independent of each other and unblock everything else.
+
+---
+
+### Task 1: Registry — the `sidecar` kind, the `args` column, and the due-time widening
+
+**Files:**
+- Modify: `internal/registry/registry.go`
+- Test: `internal/registry/registry_test.go`
+
+**Interfaces:**
+- Consumes: nothing.
+- Produces: `Route.Args string` (JSON argv, `""` = none);
+ `Route.IdleSeconds int`, `Route.MaxRunSeconds int` (0 = use the platform
+ default); `(*Registry).SetArgs(host, args string) error`;
+ `(*Registry).SetWorkerLimits(host string, idle, maxRun int) error`;
+ `DueInstances` now returns `kind IN ('instance','sidecar')`.
+
+**Why the ceilings live on the row.** The sweeper decides on every tick
+whether a worker has outrun its ceiling, and it must do that from local
+state — a bucket read per wake would put the control plane in the hot path
+of a timer. So `idle_seconds`/`max_run_seconds` are materialized from the
+app's manifest by the reconcile pass exactly as `args` and `seed` are:
+truth stays in the bucket, the row is the box's local copy.
+
+**Why this is first and why it matters:** `DueInstances` is the sweeper's
+only source of timer wakes. Left filtering `kind = 'instance'`, a sidecar's
+`next_due` would be written, stored, and never once acted on — the schedule
+wake source would silently not exist, with no error anywhere.
+
+- [ ] **Step 1: Write the failing tests**
+
+Add to `internal/registry/registry_test.go`:
+
+```go
+func TestDueInstancesIncludesSidecars(t *testing.T) {
+ r := testRegistry(t) // use this file's existing helper for a temp registry
+ if err := r.Upsert(Route{
+ Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar",
+ DBPath: "/tmp/bot.db", Hibernate: true, Status: "ready",
+ }); err != nil {
+ t.Fatalf("upsert sidecar: %v", err)
+ }
+ if err := r.SetNextDue("bot.eleven.bdm.sidecar.internal", 100); err != nil {
+ t.Fatalf("SetNextDue: %v", err)
+ }
+
+ due, err := r.DueInstances(200)
+ if err != nil {
+ t.Fatalf("DueInstances: %v", err)
+ }
+ if len(due) != 1 || due[0].Host != "bot.eleven.bdm.sidecar.internal" {
+ t.Fatalf("DueInstances = %+v, want the due sidecar — a sidecar's timer wake is its ONLY schedule wake source", due)
+ }
+}
+
+func TestArgsRoundTrips(t *testing.T) {
+ r := testRegistry(t)
+ if err := r.Upsert(Route{Host: "bot.x.internal", Kind: "sidecar", Args: `["sidecar","run"]`}); err != nil {
+ t.Fatalf("upsert: %v", err)
+ }
+ got, err := r.Get("bot.x.internal")
+ if err != nil {
+ t.Fatalf("get: %v", err)
+ }
+ if got.Args != `["sidecar","run"]` {
+ t.Fatalf("Args = %q, want the argv back", got.Args)
+ }
+
+ if err := r.SetArgs("bot.x.internal", `["worker","run"]`); err != nil {
+ t.Fatalf("SetArgs: %v", err)
+ }
+ got, _ = r.Get("bot.x.internal")
+ if got.Args != `["worker","run"]` {
+ t.Fatalf("Args after SetArgs = %q", got.Args)
+ }
+}
+
+func TestWorkerLimitsRoundTrip(t *testing.T) {
+ r := testRegistry(t)
+ if err := r.Upsert(Route{Host: "bot.x.internal", Kind: "sidecar", IdleSeconds: 300, MaxRunSeconds: 3600}); err != nil {
+ t.Fatalf("upsert: %v", err)
+ }
+ got, _ := r.Get("bot.x.internal")
+ if got.IdleSeconds != 300 || got.MaxRunSeconds != 3600 {
+ t.Fatalf("limits = %d/%d, want 300/3600 — the sweeper reads these locally on every tick", got.IdleSeconds, got.MaxRunSeconds)
+ }
+ if err := r.SetWorkerLimits("bot.x.internal", 60, 120); err != nil {
+ t.Fatalf("SetWorkerLimits: %v", err)
+ }
+ got, _ = r.Get("bot.x.internal")
+ if got.IdleSeconds != 60 || got.MaxRunSeconds != 120 {
+ t.Fatalf("limits after set = %d/%d", got.IdleSeconds, got.MaxRunSeconds)
+ }
+}
+
+func TestRouteWrittenBeforeArgsStillParses(t *testing.T) {
+ // The additive rule: a row written by older code has no args value.
+ r := testRegistry(t)
+ if err := r.Upsert(Route{Host: "old.example.com", Kind: "instance"}); err != nil {
+ t.Fatalf("upsert: %v", err)
+ }
+ got, err := r.Get("old.example.com")
+ if err != nil {
+ t.Fatalf("get: %v", err)
+ }
+ if got.Args != "" || got.IdleSeconds != 0 || got.MaxRunSeconds != 0 {
+ t.Fatalf("row = %+v, want zero values for a pre-existing row", got)
+ }
+}
+```
+
+If `testRegistry` is not the helper's name in this file, use whatever the
+file already uses to open a registry in `t.TempDir()` — do not add a new
+helper.
+
+- [ ] **Step 2: Run the tests and watch them fail**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/registry/ -run 'DueInstancesIncludesSidecars|ArgsRoundTrips|BeforeArgs' -count=1
+```
+
+Expected: compile failure — `Route` has no field `Args`, `SetArgs` undefined.
+
+- [ ] **Step 3: Add the column in all FOUR places**
+
+`registry.go` warns at the `routeCols` comment that a column lives in four
+hand-maintained lists and that a miscount is a **silent column shift**, not
+an error. Make all four edits together:
+
+1. Struct field, beside `Seed` (`registry.go` ~line 127 area):
+ ```go
+ // Args is a sidecar's argv as a JSON array, materialized from the app's
+ // sidecars manifest on every reconcile pass (truth stays in the bucket).
+ // Empty for every other kind of route.
+ Args string
+ ```
+2. Migration — append to the `ALTER TABLE` list in `migrate()`:
+ ```go
+ `ALTER TABLE routes ADD COLUMN args TEXT NOT NULL DEFAULT ''`,
+ ```
+3. `routeCols` — append `args, idle_seconds, max_run_seconds` at the very
+ END of the list (after `created_at`). Position does not matter as long
+ as it is THE SAME position in all four lists; appending at the end is
+ the least error-prone choice.
+4. `scanRoute` — add `&t.Args, &t.IdleSeconds, &t.MaxRunSeconds` at the end
+ of the destination list, in that order.
+5. `Upsert` — add three `?` placeholders at the end of the placeholder row
+ and `t.Args, t.IdleSeconds, t.MaxRunSeconds` at the end of the argument
+ list, in that order.
+
+Then add the setter beside `SetSeed`:
+
+```go
+// SetArgs repoints a sidecar's argv — the narrow UPDATE the sidecar
+// reconcile pass uses when the app's manifest changed.
+func (r *Registry) SetArgs(host, args string) error {
+ _, err := r.sql.Exec(`UPDATE routes SET args = ? WHERE host = ?`, args, host)
+ return err
+}
+
+// SetWorkerLimits repoints a sidecar's idle window and run ceiling, both in
+// seconds, 0 meaning "use the platform default". Same narrow-UPDATE shape
+// and same reason as SetArgs: the manifest is the truth, this is the copy
+// the sweeper reads on every tick.
+func (r *Registry) SetWorkerLimits(host string, idle, maxRun int) error {
+ _, err := r.sql.Exec(
+ `UPDATE routes SET idle_seconds = ?, max_run_seconds = ? WHERE host = ?`,
+ idle, maxRun, host)
+ return err
+}
+```
+
+- [ ] **Step 4: Widen `DueInstances`**
+
+```go
+// DueInstances returns hibernatable instance AND sidecar routes whose
+// next_due has arrived (0 means no schedule and never matches). Sidecars
+// are included because the due-time sweep is their ONLY timer wake source:
+// nothing can dial a sidecar, so no request will ever wake one.
+func (r *Registry) DueInstances(now int64) ([]Route, error) {
+ rows, err := r.sql.Query(`SELECT `+routeCols+` FROM routes
+WHERE kind IN ('instance','sidecar') AND next_due > 0 AND next_due <= ? ORDER BY host`, now)
+```
+
+- [ ] **Step 5: Run the full registry suite**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/registry/ -count=1
+```
+
+Expected: PASS, including `TestUpsertRoundTripsAllColumns` — that test is
+the canary for a four-list miscount. If it fails with a value in the wrong
+field, you shifted a column: recount all four lists.
+
+- [ ] **Step 6: Full gate, then commit**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && \
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./... && \
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./... -count=1
+gofmt -l internal/registry/
+git add internal/registry/
+git commit -m "registry: sidecar kind, args column, and due-time widening
+
+DueInstances filtered kind='instance', which would have stored every
+sidecar's next_due and acted on none of them — the schedule wake source
+would silently not have existed. A sidecar cannot be dialed, so the
+due-time sweep is its only timer wake.
+
+args carries a sidecar's argv to the box, materialized from the app's
+manifest each pass exactly as seed is; truth stays in the bucket.
+[trailers]"
+```
+
+---
+
+### Task 2: Provision — sidecar manifest and record shapes
+
+**Files:**
+- Create: `internal/provision/sidecar.go`
+- Test: `internal/provision/sidecar_test.go`
+
+**Interfaces:**
+- Consumes: `release.Store`, and this package's existing `labelRe`,
+ `ValidRegion`, `ValidPool`, `DefaultDBPath`.
+- Produces:
+ ```go
+ const SidecarSuffix = ".sidecar.internal"
+ type SidecarManifest struct{ Schema int; App, Pool, Channel, Placement string
+ Regions []string; Args []string; Hibernate *bool; IdleSeconds, MaxRunSeconds int }
+ type SidecarRecord struct{ Schema int; Name, Host, App, Account, Pool, Region,
+ Channel, Placement, Environment, CreatedAt, CreatedBy string }
+ func SidecarHost(name, app, sqid string) string
+ func ValidSidecarName(name string) bool
+ func ValidSidecarHost(host, app, sqid string) bool
+ func SidecarManifestKey(acct, app string) string
+ func SidecarsPrefix(acct, app string) string
+ func SidecarRecordKey(acct, app, name string) string
+ func ParseSidecarKey(key string, known map[string]bool) (acct, app, name string, ok bool)
+ func ValidateSidecarShape(rec SidecarRecord) error
+ func GetSidecarManifest(ctx, s, acct, app) (*SidecarManifest, error)
+ func GetSidecarManifestETag(ctx, s, acct, app) (*SidecarManifest, string, error)
+ func PutSidecarManifest(ctx, s, acct, app, m) error
+ func PutSidecarManifestIfMatch(ctx, s, acct, app, m, etag string) error
+ func CreateSidecarRecord(ctx, s, rec) error
+ func GetSidecarRecord(ctx, s, acct, app, name) (*SidecarRecord, error)
+ func ListSidecarRecords(ctx, s, acct, app string, max int) ([]SidecarRecord, bool, error)
+ func DeleteSidecarRecord(ctx, s, acct, app, name string) error
+ func (m SidecarManifest) IdleOrDefault() time.Duration // 5m
+ func (m SidecarManifest) MaxRunOrDefault() time.Duration // 1h
+ func (m SidecarManifest) HibernateOrDefault() bool // true
+ func (m SidecarManifest) PlacementOrDefault() string // "any"
+ ```
+
+**Durations are integer seconds on the wire** (`idle_seconds`,
+`max_run_seconds`), not Go duration strings — the existing records use plain
+JSON scalars and a string would invite `"5 minutes"`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `internal/provision/sidecar_test.go`. Mirror the idiom in
+`provision_test.go` and `reconcile_test.go`: a real `release.DirStore` in
+`t.TempDir()`, no mocks.
+
+```go
+func TestSidecarHostAndValidation(t *testing.T) {
+ got := SidecarHost("bot-acme", "eleven", "bdm")
+ if want := "bot-acme.eleven.bdm.sidecar.internal"; got != want {
+ t.Fatalf("SidecarHost = %q, want %q", got, want)
+ }
+ if !ValidSidecarHost(got, "eleven", "bdm") {
+ t.Fatal("its own SidecarHost output must validate")
+ }
+ for _, bad := range []string{
+ "bot-acme.other.bdm.sidecar.internal", // wrong app
+ "bot-acme.eleven.bad.sidecar.internal", // wrong sqid
+ "bot-acme.eleven.bdm.example.com", // wrong suffix
+ "a.b.bot.eleven.bdm.sidecar.internal", // more than one label
+ } {
+ if ValidSidecarHost(bad, "eleven", "bdm") {
+ t.Fatalf("ValidSidecarHost(%q) = true, want false", bad)
+ }
+ }
+ for _, bad := range []string{"", "-lead", "trail-", "UPPER", "has.dot", "has/slash", "..", strings.Repeat("a", 64)} {
+ if ValidSidecarName(bad) {
+ t.Fatalf("ValidSidecarName(%q) = true — the name becomes a bucket key AND a filesystem path segment", bad)
+ }
+ }
+}
+
+func TestParseSidecarKey(t *testing.T) {
+ known := map[string]bool{"bdm": true}
+ acct, app, name, ok := ParseSidecarKey("apps/bdm/eleven/sidecars/bot-acme.json", known)
+ if !ok || acct != "bdm" || app != "eleven" || name != "bot-acme" {
+ t.Fatalf("ParseSidecarKey = %q/%q/%q ok=%v", acct, app, name, ok)
+ }
+ for _, bad := range []string{
+ "apps/bdm/eleven/sidecars.json", // the manifest, not a record
+ "apps/bdm/eleven/sidecars/nested/x.json", // nested
+ "apps/bdm/eleven/sidecars/.json", // empty name
+ "apps/unknown/eleven/sidecars/bot.json", // account not materialized
+ "apps/bdm/channels/sidecars/bot.json", // release-plane collision
+ "apps/bdm/eleven/instances/bot.json", // the other family
+ } {
+ if _, _, _, ok := ParseSidecarKey(bad, known); ok {
+ t.Fatalf("ParseSidecarKey(%q) parsed, want refusal", bad)
+ }
+ }
+}
+
+func TestCreateSidecarRecordIsCreateOnly(t *testing.T) {
+ ctx := context.Background()
+ s := dirStore(t) // the helper reconcile_test.go already uses
+ rec := SidecarRecord{
+ Schema: 1, Name: "bot-acme", Host: SidecarHost("bot-acme", "eleven", "bdm"),
+ App: "eleven", Account: "bdm", Channel: "stable",
+ }
+ if err := CreateSidecarRecord(ctx, s, rec); err != nil {
+ t.Fatalf("first create: %v", err)
+ }
+ err := CreateSidecarRecord(ctx, s, rec)
+ if !errors.Is(err, release.ErrCASConflict) {
+ t.Fatalf("second create err = %v, want ErrCASConflict — a lost race must never silently overwrite", err)
+ }
+}
+
+func TestGetSidecarRecordRefusesRecordThatLiesAboutItsPath(t *testing.T) {
+ ctx := context.Background()
+ s := dirStore(t)
+ body := []byte(`{"schema":1,"name":"bot-acme","app":"OTHER","account":"bdm"}`)
+ if _, err := s.Put(ctx, SidecarRecordKey("bdm", "eleven", "bot-acme"), body, ""); err != nil {
+ t.Fatalf("seed: %v", err)
+ }
+ if _, err := GetSidecarRecord(ctx, s, "bdm", "eleven", "bot-acme"); err == nil {
+ t.Fatal("a record disagreeing with its own key path must be refused, never guessed at")
+ }
+}
+
+func TestSidecarManifestDefaults(t *testing.T) {
+ var m SidecarManifest
+ if m.IdleOrDefault() != 5*time.Minute {
+ t.Fatalf("idle default = %v, want 5m", m.IdleOrDefault())
+ }
+ if m.MaxRunOrDefault() != time.Hour {
+ t.Fatalf("max_run default = %v, want 1h", m.MaxRunOrDefault())
+ }
+ if !m.HibernateOrDefault() {
+ t.Fatal("hibernate must default true")
+ }
+ if m.PlacementOrDefault() != "any" {
+ t.Fatalf("placement default = %q, want any", m.PlacementOrDefault())
+ }
+ no := false
+ m2 := SidecarManifest{Hibernate: &no}
+ if m2.HibernateOrDefault() {
+ t.Fatal("an explicit false must win — tri-state, like instances' Hibernate")
+ }
+}
+
+func TestSidecarManifestSchemaNewerIsRefused(t *testing.T) {
+ ctx := context.Background()
+ s := dirStore(t)
+ if _, err := s.Put(ctx, SidecarManifestKey("bdm", "eleven"), []byte(`{"schema":2}`), ""); err != nil {
+ t.Fatalf("seed: %v", err)
+ }
+ if _, err := GetSidecarManifest(ctx, s, "bdm", "eleven"); err == nil {
+ t.Fatal("a schema newer than this binary understands must be refused, not silently truncated")
+ }
+}
+
+func TestSidecarManifestAbsenceIsNotExist(t *testing.T) {
+ _, err := GetSidecarManifest(context.Background(), dirStore(t), "bdm", "eleven")
+ if !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("absence err = %v, want os.ErrNotExist — absence means the feature does not exist for that app", err)
+ }
+}
+
+func TestValidateSidecarShape(t *testing.T) {
+ ok := SidecarRecord{Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"),
+ App: "eleven", Account: "bdm", Channel: "stable", Placement: "no-inbound"}
+ if err := ValidateSidecarShape(ok); err != nil {
+ t.Fatalf("valid record refused: %v", err)
+ }
+ bad := ok
+ bad.Placement = "sideways"
+ if err := ValidateSidecarShape(bad); err == nil {
+ t.Fatal("placement vocabulary must be closed")
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch it fail**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/provision/ -run Sidecar -count=1
+```
+Expected: compile failure, everything undefined.
+
+- [ ] **Step 3: Implement `internal/provision/sidecar.go`**
+
+Write the file mirroring `provision.go` structurally. The store operations
+are **the same six functions with `Sidecar` in the name and the sidecar key
+helpers substituted** — copy the bodies and their doc comments' reasoning
+from these exact sites:
+
+- `GetSidecarManifest` / `GetSidecarManifestETag` ← `provision.go:508-529`
+ (schema>1 refusal included).
+- `PutSidecarManifest` ← `provision.go:535-557` (CAS upsert, one retry).
+- `PutSidecarManifestIfMatch` ← `provision.go:568-575` (no retry).
+- `CreateSidecarRecord` ← `provision.go:580-587` (`Put` with `ifMatch ""`).
+- `GetSidecarRecord` ← `provision.go:687-709`, **keeping the
+ identity-vs-path agreement check** (`rec.Account != acct || rec.App != app
+ || rec.Name != name` → error).
+- `ListSidecarRecords` ← `provision.go:726-760` (sorted, capped, `truncated`
+ reported on key count, malformed bodies skipped).
+- `DeleteSidecarRecord` ← `provision.go:641-643`.
+
+The new logic is only this:
+
+```go
+// SidecarSuffix is the reserved, never-resolvable suffix every sidecar host
+// carries. It is refused at every door where a name could become routable
+// (see cli.go's addRail, the route door, and domain attach) — reservation
+// is enforced, not documented.
+const SidecarSuffix = ".sidecar.internal"
+
+// SidecarHost builds a sidecar's synthetic host. It is the registry key, the
+// bucket key segment and the on-disk path segment, which is why every
+// component is validated before it is ever joined.
+func SidecarHost(name, app, sqid string) string {
+ return name + "." + app + "." + sqid + SidecarSuffix
+}
+
+// ValidSidecarName pins one DNS label exactly as ValidHost pins an
+// instance's: the name becomes a bucket key segment and the <host>.db /
+// cgroup path segments, so the label regex is what keeps a hostile name an
+// opaque, safe path component.
+func ValidSidecarName(name string) bool { return labelRe.MatchString(name) }
+
+func ValidSidecarHost(host, app, sqid string) bool {
+ rest, ok := strings.CutSuffix(host, SidecarSuffix)
+ if !ok {
+ return false
+ }
+ name, tail, ok := strings.Cut(rest, ".")
+ if !ok || tail != app+"."+sqid {
+ return false
+ }
+ return ValidSidecarName(name)
+}
+```
+
+Key helpers, beside the instance ones (`provision.go:155-159`):
+
+```go
+func SidecarManifestKey(acct, app string) string {
+ return release.AppPrefix(acct, app) + "sidecars.json"
+}
+func SidecarsPrefix(acct, app string) string {
+ return release.AppPrefix(acct, app) + "sidecars/"
+}
+func SidecarRecordKey(acct, app, name string) string {
+ return SidecarsPrefix(acct, app) + name + ".json"
+}
+```
+
+`ParseSidecarKey` is `ParseRecordKey` (`provision.go:171-196`) with
+`"instances/"` → `"sidecars/"`: keep the `known` account guard, the
+`channels`/`releases` collision refusal, and the exactly-one-non-empty-
+segment rule that makes `sidecars.json` and any nested path both refuse.
+
+Defaults:
+
+```go
+func (m SidecarManifest) IdleOrDefault() time.Duration {
+ if m.IdleSeconds > 0 {
+ return time.Duration(m.IdleSeconds) * time.Second
+ }
+ return 5 * time.Minute
+}
+func (m SidecarManifest) MaxRunOrDefault() time.Duration {
+ if m.MaxRunSeconds > 0 {
+ return time.Duration(m.MaxRunSeconds) * time.Second
+ }
+ return time.Hour
+}
+func (m SidecarManifest) HibernateOrDefault() bool {
+ if m.Hibernate != nil {
+ return *m.Hibernate
+ }
+ return true
+}
+func (m SidecarManifest) PlacementOrDefault() string {
+ if m.Placement == "" {
+ return PlacementAny
+ }
+ return m.Placement
+}
+```
+
+with `const ( PlacementAny = "any"; PlacementNoInbound = "no-inbound" )`.
+
+`ValidateSidecarShape` checks: schema ≤ 1; `ValidSidecarName(rec.Name)`;
+`rec.App`/`rec.Account` non-empty; placement in the closed vocabulary;
+`ValidRegion(rec.Region)` when non-empty; `ValidPool(rec.Pool)` when
+non-empty.
+
+- [ ] **Step 4: Run the tests**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/provision/ -count=1
+```
+Expected: PASS.
+
+- [ ] **Step 5: Full gate, then commit**
+
+```
+provision: sidecar manifest and record shapes
+
+Mirrors the instance record family one directory over, reusing its
+validation rails verbatim — the label regex, the identity-vs-path
+agreement check, create-only writes, and the absence-means-disabled
+convention. The name becomes a bucket key AND a filesystem path segment,
+which is why it is validated before it is ever joined into a host.
+```
+
+---
+
+### Task 3: The edge cannot bind a sidecar, and no box advertises one
+
+**Files:**
+- Modify: `internal/edge/edge.go` (the `Reload` dispatch switch, ~364-445)
+- Modify: `internal/host/fleetbind.go` (`publishResidentFleetBindings`, ~301-316)
+- Test: `internal/edge/edge_test.go`, `internal/host/fleetbind_test.go`
+
+**Interfaces:**
+- Consumes: Task 1's `Kind == "sidecar"`.
+- Produces: nothing other tasks call.
+
+**Why this is not paranoia.** A sidecar row is unroutable today only by
+*omission*: it has no Socket, Addr or Tunnel, so it falls to
+`default: continue` and never enters the routing map. But the dispatch
+switch has a `case isFleetPlaced && e.tunnels != nil` arm with no Kind
+check, and `publishResidentFleetBindings` iterates **every** route with no
+Kind filter. Eleven's sidecars are fleet-placed (`region: fleet/eleven`) —
+so the row that reaches that arm by shape is exactly the row this feature
+creates. Omission is not good enough; refuse explicitly at both ends.
+
+- [ ] **Step 1: Write the failing tests**
+
+In `internal/edge/edge_test.go` (follow the file's existing Reload/route
+fixtures):
+
+```go
+func TestSidecarRouteIsNeverServable(t *testing.T) {
+ // Adversarial on purpose: this row carries a socket, an addr, a tunnel
+ // AND a fleet placement. A sidecar must be refused on KIND, so that no
+ // present or future arm of the dispatch switch can ever bind one.
+ e := newTestEdge(t)
+ rt := registry.Route{
+ Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar",
+ Socket: "/tmp/whatever.sock", Addr: "127.0.0.1:9999",
+ Tunnel: "eleven/box1", Region: "fleet/eleven", FleetBox: "box1",
+ Status: "ready", App: "eleven",
+ }
+ if err := e.Reload([]registry.Route{rt}); err != nil {
+ t.Fatalf("Reload: %v", err)
+ }
+ if _, ok := e.current()["bot.eleven.bdm.sidecar.internal"]; ok {
+ t.Fatal("a sidecar row entered the routing map — it is reachable from the internet")
+ }
+ if e.Allowed("bot.eleven.bdm.sidecar.internal") == nil {
+ t.Fatal("a certificate must never be requested for a sidecar host")
+ }
+}
+```
+
+(Match `newTestEdge`/`current`/`Allowed` to whatever the file actually
+provides; if `Allowed` returns an error use `!= nil` as "refused".)
+
+In `internal/host/fleetbind_test.go`:
+
+```go
+func TestSidecarBindingIsNeverPublished(t *testing.T) {
+ routes := []registry.Route{
+ {Host: "app.example.com", Kind: "instance", Socket: "/run/carlos/a.sock"},
+ {Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", Region: "fleet/eleven"},
+ }
+ got := residentBindingHosts(routes) // the pure helper you extract in step 3
+ if len(got) != 1 || got[0] != "app.example.com" {
+ t.Fatalf("published %v — a sidecar must never be advertised as served by a box", got)
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch both fail**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/edge/ ./internal/host/ -run 'Sidecar' -count=1
+```
+Expected: the edge test FAILS (the row is served via the tunnel/fleet arm);
+the fleetbind test fails to compile (`residentBindingHosts` undefined).
+
+- [ ] **Step 3: Refuse in the edge, before the switch**
+
+In `Reload`'s per-route loop, as the FIRST thing inside the loop body:
+
+```go
+// A sidecar is a member that serves nothing: it binds no socket, and
+// nothing may ever dial it. Refuse it on KIND here, before the dispatch
+// switch, rather than relying on it having no socket/addr/tunnel — a
+// fleet-placed sidecar (which is Eleven's shape) reaches the fleet arm
+// below on shape alone. This refusal is the structural guarantee that
+// survives a future arm, column or placement finding the row.
+if t.Kind == "sidecar" {
+ continue
+}
+```
+
+- [ ] **Step 4: Refuse in the fleet-binding publisher**
+
+Extract the host selection into a pure, testable helper and filter there:
+
+```go
+// residentBindingHosts is the set of hosts this box advertises as served.
+// Sidecars are excluded: nothing routes to one, so a binding claiming a box
+// serves it is a lie that the platform-side reconciler would then act on
+// (SetFleetBox -> the edge's fleet arm -> a live proxy).
+func residentBindingHosts(routes []registry.Route) []string {
+ var out []string
+ for _, rt := range routes {
+ if rt.Host == "" || rt.Kind == "sidecar" {
+ continue
+ }
+ out = append(out, rt.Host)
+ }
+ return out
+}
+```
+
+and call it from `publishResidentFleetBindings` in place of its current
+inline loop.
+
+- [ ] **Step 5: Run both tests, then the full gate, then commit**
+
+```
+edge, host: a sidecar row is refused on kind, at both ends of the fleet chain
+
+A sidecar was unroutable only by omission — no socket, addr or tunnel, so
+it fell to the dispatch switch's default. But the switch's fleet-placed
+arm checks no kind, and the binding publisher filtered no kind, and
+Eleven's sidecars are fleet-placed. That is a live path from a resident
+sidecar row to a working internet-facing proxy, so both ends now refuse
+explicitly and the test asserts it against a row that carries every
+routable column at once.
+```
+
+---
+
+### Task 4: The reserved suffix is enforced at every minting door
+
+**Files:**
+- Modify: `cli.go` (root, `runAdd` ~68-260 and `runRoute`'s rails ~1197-1373)
+- Modify: `internal/domains/reconcile.go` (alias minting)
+- Modify: `internal/console/domainattach.go` (attach door)
+- Test: `cli_test.go`, `internal/domains/reconcile_test.go`,
+ `internal/console/domainattach_test.go`
+
+**Interfaces:**
+- Consumes: `provision.SidecarSuffix`.
+- Produces: nothing.
+
+`carlos add` today accepts `-kind sidecar` with **no validation at all**
+(`cli.go:75,194-199`), and any host string reaches `registry.Upsert`. These
+rails are defence in depth behind Task 3's structural refusal.
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestAddRefusesSidecarSuffixAndKind(t *testing.T) {
+ if err := addRail("bot.eleven.bdm.sidecar.internal", "instance"); err == nil {
+ t.Fatal("a host under the reserved suffix must be refused")
+ }
+ if err := addRail("app.example.com", "sidecar"); err == nil {
+ t.Fatal("carlos add must not mint a sidecar row — sidecars are provisioned, not added")
+ }
+ if err := addRail("app.example.com", "instance"); err != nil {
+ t.Fatalf("an ordinary add must still work: %v", err)
+ }
+}
+```
+
+Add matching cases asserting `domains` never mints an alias whose source or
+target carries the suffix, and that `domainattach` refuses to attach a
+custom domain to a host under it.
+
+- [ ] **Step 2: Run and watch it fail** (`addRail` undefined).
+
+- [ ] **Step 3: Implement the rail**
+
+In `cli.go`, beside `grantRail`/`backingRail`:
+
+```go
+// addRail refuses the two things `carlos add` must never mint: a host under
+// the reserved sidecar suffix, and a sidecar row itself. Sidecars are
+// provisioned through their own record family (carlos sidecars create), so
+// an operator minting one by hand would produce a row no reconcile pass
+// owns and no record explains.
+func addRail(host, kind string) error {
+ if strings.HasSuffix(host, provision.SidecarSuffix) {
+ return fmt.Errorf("%s is under the reserved suffix %s — sidecar hosts are minted by `carlos sidecars create`, never by hand", host, provision.SidecarSuffix)
+ }
+ if kind == "sidecar" {
+ return errors.New("carlos add does not mint sidecars — use `carlos sidecars create -app <app> <name>`")
+ }
+ return nil
+}
+```
+
+Call it at the top of `runAdd`, before any registry work. Apply the same
+`HasSuffix` refusal in the domains alias minting path and in the console's
+domain-attach handler (return the family's ordinary 400 there, not a panic).
+
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 5: `StartWorker` — spawning a child that binds nothing
+
+**Files:**
+- Modify: `internal/activator/activator.go` (the `Backend` interface, ~134-167)
+- Modify: `internal/activator/backend_exec.go`
+- Modify: `internal/activator/backend_unit.go`
+- Test: `internal/activator/backend_exec_test.go` (or the existing backend test file)
+
+**Interfaces:**
+- Consumes: Task 1's `Route.Args`.
+- Produces:
+ ```go
+ // on Backend:
+ StartWorker(ctx context.Context, rt registry.Route, env map[string]string) (Proc, string, error)
+ ```
+ Returns the `Proc` and the spawn build version, exactly like `Start`.
+
+**Why a new method rather than a flag on `Start`:** `ExecBackend.Start` ends
+with an unconditional `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)`
+(`backend_exec.go:601`). Handed `Socket ""` it dials `unix://""` every 150ms
+for the full 30s and then fails the wake. A worker has no socket by
+definition, so it needs a spawn path with no probe — which is very nearly
+what `StartSidecar` (`backend_exec.go:746-773`) already is.
+
+- [ ] **Step 1: Write the failing test**
+
+```go
+func TestStartWorkerSpawnsWithoutAnyHealthProbe(t *testing.T) {
+ dir := t.TempDir()
+ b := &ExecBackend{DataDir: dir, OptDir: dir, Cgroups: nil}
+ // A binary that runs until killed and never binds anything.
+ writeFakeLive(t, dir, "bot.x.internal", "#!/bin/sh\nsleep 30\n")
+
+ rt := registry.Route{Host: "bot.x.internal", Kind: "sidecar",
+ Args: `["sidecar","run"]`, DBPath: filepath.Join(dir, "bot.db")}
+
+ start := time.Now()
+ p, _, err := b.StartWorker(context.Background(), rt, map[string]string{"CARLOS_SIDECAR_NAME": "bot"})
+ if err != nil {
+ t.Fatalf("StartWorker: %v", err)
+ }
+ defer p.Stop(context.Background())
+ if d := time.Since(start); d > 5*time.Second {
+ t.Fatalf("StartWorker took %v — it waited for a health probe that can never answer", d)
+ }
+ if p.Exited() {
+ t.Fatal("worker exited immediately")
+ }
+}
+
+func TestStartWorkerRefusesEmptyArgs(t *testing.T) {
+ b := &ExecBackend{DataDir: t.TempDir()}
+ _, _, err := b.StartWorker(context.Background(), registry.Route{Host: "b.x.internal", Kind: "sidecar"}, nil)
+ if err == nil {
+ t.Fatal("a worker with no argv has nothing to run — refuse rather than exec the bare binary")
+ }
+}
+
+func TestUnitBackendRefusesWorkers(t *testing.T) {
+ var b UnitBackend
+ if _, _, err := b.StartWorker(context.Background(), registry.Route{Host: "x"}, nil); err == nil {
+ t.Fatal("unit-backed tenants have no agent-managed workers, exactly as they have no agent-managed sidecars")
+ }
+}
+```
+
+`writeFakeLive` — if the test file has no such helper, write one that
+creates the `live/<host>` symlink target as an executable shell script,
+matching however the existing exec-backend tests fake a binary.
+
+- [ ] **Step 2: Run and watch it fail.**
+
+- [ ] **Step 3: Add the method to the `Backend` interface**
+
+```go
+ // StartWorker spawns a child that binds NOTHING: no socket, no port.
+ // No health probe is possible or wanted — readiness is the caller's
+ // settle window, or the child's own call over the agent socket. env is
+ // merged over the route's own env layers and wins conflicts (it carries
+ // the per-wake sidecar token, which the tenant must never override).
+ StartWorker(ctx context.Context, rt registry.Route, env map[string]string) (Proc, string, error)
+```
+
+- [ ] **Step 4: Implement `ExecBackend.StartWorker`**
+
+Model it on `Start` (`backend_exec.go:476-636`) minus the probe and the
+socket-mode normalization, and on `StartSidecar` for the no-probe framing:
+
+```go
+func (b *ExecBackend) StartWorker(ctx context.Context, rt registry.Route, extra map[string]string) (Proc, string, error) {
+ var argv []string
+ if err := json.Unmarshal([]byte(rt.Args), &argv); err != nil || len(argv) == 0 {
+ return nil, "", fmt.Errorf("worker %s: no runnable argv in args %q", rt.Host, rt.Args)
+ }
+ if err := b.ensureLiveLink(rt.Host); err != nil {
+ return nil, "", err
+ }
+ spawnVer, _ := b.liveVersion(rt.Host) // same call Start uses; "" = unknown
+
+ fileEnv := b.instanceEnv(rt) // the same merged layers Start composes
+ for k, v := range extra {
+ fileEnv = withVar(fileEnv, k, v)
+ }
+
+ cmd, err := b.Cgroups.Start(rt.Host, func() *exec.Cmd {
+ c := exec.Command(b.liveLink(rt.Host), argv...)
+ c.Env = childEnv(fileEnv)
+ b.teeStd(c, rt)
+ return c
+ })
+ if err != nil {
+ return nil, "", err
+ }
+ p := &childProc{cmd: cmd, done: make(chan struct{})}
+ go func() {
+ p.err = cmd.Wait()
+ b.Cgroups.CleanupHost(rt.Host)
+ close(p.done)
+ }()
+ // Deliberately NO waitHealthy: a worker binds nothing, so there is
+ // nothing to probe. Readiness is the activator's settle window.
+ return p, spawnVer, nil
+}
+```
+
+Use the real names this file already has for the live-link, version and env
+helpers (`ensureLiveLink`, `liveLink`, `instanceEnv`, `childEnv`,
+`withVar`, `teeStd`); do not invent new ones. If `b.Cgroups` may be nil in
+tests, follow whatever nil-guard `Start` uses.
+
+`UnitBackend.StartWorker` and `SplitBackend.StartWorker` mirror
+`StartSidecar` (`backend_unit.go:186-188`, `:304-306`): refuse, and
+delegate by `b.pick(rt)` respectively.
+
+- [ ] **Step 5: Run the tests, full gate, commit.**
+
+---
+
+### Task 6: The standalone sidecar lifecycle inside the activator
+
+**Files:**
+- Modify: `internal/activator/activator.go`
+- Test: `internal/activator/activator_test.go`
+
+**Interfaces:**
+- Consumes: Task 5's `StartWorker`.
+- Produces:
+ ```go
+ func (a *Service) WakeWorker(rt registry.Route) error // idempotent; nil if already awake
+ func (a *Service) NoteWorkerContact(host string) // stamps lastActive
+ func (a *Service) SetWorkerNextDue(host string, at int64) error
+ func (a *Service) SleepWorker(host string) // /v1/idle
+ func (a *Service) WorkerToken(host string) string // "" unless awake
+ func (a *Service) NoteWorkerReady(host, version string)
+ ```
+ `WakeWorker` takes the route ALONE: the idle window and run ceiling ride
+ on the row (Task 1), so the sweeper never needs a bucket read to decide a
+ worker has outrun its ceiling.
+
+**Design constraint:** sidecars stay entirely off the request path.
+`beginRequest`, `BeginSidecarRequest` and `ServeHTTP` are not modified —
+`ServeHTTP` already gates on `rt.Kind == "instance" && rt.Hibernate`.
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestWorkerWakeUsesStartWorkerAndNoProbe(t *testing.T) {
+ a, b, _, _ := testActivator(t, 10*time.Second)
+ rt := testWorkerRoute("bot.x.internal")
+ if err := a.WakeWorker(rt); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ if got := b.events(); !contains(got, "startWorker:bot.x.internal") {
+ t.Fatalf("events %v — a worker must spawn through StartWorker, never Start", got)
+ }
+ if b.startCount() != 0 {
+ t.Fatal("Start (the socket path, with its health probe) must never be called for a worker")
+ }
+}
+
+func TestWorkerSleepsWhenIdleAndRecordsNextDue(t *testing.T) {
+ a, b, clk, reg := testActivator(t, 10*time.Second)
+ rt := testWorkerRoute("bot.x.internal")
+ if err := a.WakeWorker(rt); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ if err := a.SetWorkerNextDue("bot.x.internal", 4242); err != nil {
+ t.Fatalf("SetWorkerNextDue: %v", err)
+ }
+
+ clk.advance(11 * time.Second)
+ a.sweepOnce()
+ waitFor(t, "worker hibernation", func() bool { return a.entryCount() == 0 })
+
+ got, err := reg.Get("bot.x.internal")
+ if err != nil {
+ t.Fatalf("get: %v", err)
+ }
+ if got.NextDue != 4242 {
+ t.Fatalf("next_due = %d, want 4242 — a worker's next-due arrives over the agent socket while it is ALIVE, not from a probe of a process we are about to kill", got.NextDue)
+ }
+ if b.dueProbes() != 0 {
+ t.Fatal("a worker must never be probed for next-due: it has no socket to answer on")
+ }
+ if !b.lastRep().isClosed() {
+ t.Fatal("the lease must be released and the DB parked on a worker sleep, exactly as for an instance")
+ }
+}
+
+func TestWorkerContactDefersSleep(t *testing.T) {
+ a, _, clk, _ := testActivator(t, 10*time.Second)
+ if err := a.WakeWorker(testWorkerRoute("bot.x.internal")); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ clk.advance(9 * time.Second)
+ a.NoteWorkerContact("bot.x.internal")
+ clk.advance(9 * time.Second)
+ a.sweepOnce()
+ if a.entryCount() != 1 {
+ t.Fatal("agent-socket contact must defer the idle clock — it is a worker's only sign of life")
+ }
+}
+
+func TestWorkerMaxRunStopsALoopingWorker(t *testing.T) {
+ a, _, clk, _ := testActivator(t, time.Hour) // idle window never fires
+ rt := testWorkerRoute("bot.x.internal")
+ rt.MaxRunSeconds = 60
+ if err := a.WakeWorker(rt); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ for i := 0; i < 10; i++ { // a worker that keeps talking but never idles
+ clk.advance(10 * time.Second)
+ a.NoteWorkerContact("bot.x.internal")
+ }
+ a.sweepOnce()
+ waitFor(t, "max_run stop", func() bool { return a.entryCount() == 0 })
+}
+
+func TestSleepWorkerIsImmediate(t *testing.T) {
+ a, _, _, _ := testActivator(t, time.Hour)
+ if err := a.WakeWorker(testWorkerRoute("bot.x.internal")); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ a.SleepWorker("bot.x.internal")
+ waitFor(t, "idle-requested stop", func() bool { return a.entryCount() == 0 })
+}
+
+func TestWakeWorkerIsIdempotent(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ rt := testWorkerRoute("bot.x.internal")
+ for i := 0; i < 3; i++ {
+ if err := a.WakeWorker(rt); err != nil {
+ t.Fatalf("WakeWorker #%d: %v", i, err)
+ }
+ }
+ if n := b.workerStarts(); n != 1 {
+ t.Fatalf("workerStarts = %d, want 1 — a wake for an already-awake worker is a no-op, not a second child", n)
+ }
+}
+```
+
+Extend `fakeBackend` with `StartWorker` (recording `startWorker:<host>` in
+its `order` log and incrementing a `workerStarts` counter), and add
+`testWorkerRoute(host)` returning
+`registry.Route{Host: host, Kind: "sidecar", Args: `["sidecar","run"]`, DBPath: ..., Hibernate: true, Status: "ready"}`.
+Add the `dueProbes()` accessor if the fake does not already count `NextDue`
+calls.
+
+- [ ] **Step 2: Run and watch them fail.**
+
+- [ ] **Step 3: Implement**
+
+Add to `actEntry`:
+
+```go
+ worker bool // this entry is a sidecar: no socket, no requests
+ workerIdle time.Duration // 0 = fall back to cfg.Idle
+ workerMaxRun time.Duration // 0 = no ceiling
+ workerToken string // the per-wake agent-socket token
+ spawnedAt time.Time
+```
+
+populated from the route's `IdleSeconds`/`MaxRunSeconds` at wake time.
+
+```go
+// WakeWorker brings a sidecar up, or does nothing if it is already awake.
+// It is the request-less twin of beginRequest: a sidecar has no socket, so
+// no request can ever wake one — every wake arrives from the due-time
+// sweep, an operator command, or the agent's boot pass.
+func (a *Service) WakeWorker(rt registry.Route) error
+```
+
+Implementation:
+
+- Under `a.mu`: if an entry exists and is `actAwake` or `actWaking`, return
+ nil (idempotent). If `actStopping`, wait on `e.stopped` then retry.
+- Otherwise create the entry with `worker: true`, `workerMaxRun: maxRun`,
+ mint `workerToken` (32 bytes from `crypto/rand`, hex), launch
+ `go a.wake(e)` and **do not unlock before falling into the wait** — the
+ singleflight discipline PR #199 established; then wait on `e.ready`.
+- In `doWake`, branch step 4:
+ ```go
+ var (
+ proc Proc
+ spawnVer string
+ )
+ if e.worker {
+ proc, spawnVer, err = a.backend.StartWorker(ctx, rt, a.workerEnv(e))
+ } else {
+ proc, spawnVer, err = a.backend.Start(ctx, rt)
+ }
+ ```
+ and skip step 6 (sidecar co-spawn) entirely for a worker — a worker is
+ nobody's parent.
+- In `stop`, skip the next-due probe when `e.worker`: its next-due arrived
+ over the agent socket while it was alive. Everything after (stop the
+ proc, final checkpoint, lease release) is shared unchanged.
+- In `sweepOnce`, beside the existing idle check, add:
+ ```go
+ if e.worker && e.workerMaxRun > 0 && now.Sub(e.spawnedAt) >= e.workerMaxRun {
+ // A worker that never reports idle still has a ceiling: a wedged
+ // reconnect loop must not hold a lease and a box forever.
+ a.beginStopLocked(e)
+ continue
+ }
+ ```
+ and make the existing idle predicate use the entry's own window when set
+ (`e.workerIdle`), falling back to `a.cfg.Idle`. Otherwise it is unchanged:
+ `inflight` is always 0 for a worker, and `lastActive` is stamped by
+ agent-socket contact instead of by a released request.
+
+The four small public methods:
+
+```go
+func (a *Service) NoteWorkerContact(host string) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if e := a.entries[host]; e != nil && e.worker {
+ e.lastActive = a.now()
+ }
+}
+
+func (a *Service) SetWorkerNextDue(host string, at int64) error {
+ return a.due.SetNextDue(host, at)
+}
+
+func (a *Service) SleepWorker(host string) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if e := a.entries[host]; e != nil && e.worker && e.state == actAwake {
+ a.beginStopLocked(e)
+ }
+}
+
+// WorkerToken reports the live per-wake token for host, or "" — the agent
+// socket's authorization check. A token from a previous wake never matches.
+func (a *Service) WorkerToken(host string) string {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if e := a.entries[host]; e != nil && e.worker && e.state == actAwake {
+ return e.workerToken
+ }
+ return ""
+}
+```
+
+`a.workerEnv(e)` returns
+`{"CARLOS_AGENT_SOCKET": a.cfg.AgentSocket, "CARLOS_SIDECAR_TOKEN": e.workerToken, "CARLOS_SIDECAR_HOST": e.host, "CARLOS_SIDECAR_NAME": <first label of e.host>}`.
+Add `AgentSocket string` to `activator.Config`, defaulting to
+`/run/carlos/agent.sock`.
+
+- [ ] **Step 4: Run the activator suite in full**
+
+```bash
+GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \
+ go test ./internal/activator/ -count=1
+```
+Expected: PASS, **including every pre-existing instance test** — especially
+`TestActivatorFailedWakeIsSingleflight`, `TestActivatorCoStopOrdering` and
+`TestActivatorIdleSleep`. If any instance test changed behaviour, you put a
+worker branch on a shared path; move it.
+
+- [ ] **Step 5: Full gate, commit.**
+
+---
+
+### Task 7: The agent socket
+
+**Files:**
+- Create: `internal/agentsock/agentsock.go`, `internal/agentsock/agentsock_test.go`
+
+**Interfaces:**
+- Consumes: an interface it defines and Task 6 satisfies:
+ ```go
+ type Workers interface {
+ WorkerToken(host string) string
+ NoteWorkerContact(host string)
+ SetWorkerNextDue(host string, at int64) error
+ SleepWorker(host string)
+ NoteWorkerReady(host, version string)
+ }
+ ```
+- Produces:
+ ```go
+ func New(w Workers) *Server
+ func (s *Server) Serve(ctx context.Context, path string) error // listens, cleans up
+ func (s *Server) Handler() http.Handler // for tests
+ ```
+
+**The direction is the whole point:** the agent listens, children dial. The
+platform never dials a sidecar.
+
+Wire protocol — all `POST`, JSON in, `204` out, `Authorization: Bearer <token>`
+and `Carlos-Sidecar-Host: <host>`:
+
+| Path | Body | Effect |
+|---|---|---|
+| `/v1/ready` | `{"version":"<build>"}` | ends the settle window, records the running version |
+| `/v1/next-due` | `{"at": <unix seconds>}` | `SetWorkerNextDue` |
+| `/v1/idle` | — | `SleepWorker` |
+| `/v1/heartbeat` | — | `NoteWorkerContact` |
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+type fakeWorkers struct {
+ mu sync.Mutex
+ tokens map[string]string
+ contacts []string
+ due map[string]int64
+ slept []string
+ ready map[string]string
+}
+
+func (f *fakeWorkers) WorkerToken(host string) string { /* guarded read */ }
+// ... the rest recording into the slices/maps
+
+func TestAgentSockRejectsWrongToken(t *testing.T) {
+ f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}}
+ h := New(f).Handler()
+
+ for _, tc := range []struct{ name, host, tok string }{
+ {"wrong token", "bot.x.internal", "bad"},
+ {"no token", "bot.x.internal", ""},
+ {"unknown host", "nope.x.internal", "good"},
+ {"token for another host", "other.x.internal", "good"},
+ } {
+ rec := post(h, "/v1/heartbeat", tc.host, tc.tok, "")
+ if rec.Code != http.StatusUnauthorized {
+ t.Fatalf("%s: code = %d, want 401", tc.name, rec.Code)
+ }
+ }
+ if len(f.contacts) != 0 {
+ t.Fatal("a refused call must have no effect at all")
+ }
+}
+
+func TestAgentSockTokenIsScopedToOneHost(t *testing.T) {
+ // The security property: possession of A's token says nothing about B.
+ f := &fakeWorkers{tokens: map[string]string{"a.x.internal": "tok-a", "b.x.internal": "tok-b"}}
+ h := New(f).Handler()
+ if rec := post(h, "/v1/idle", "b.x.internal", "tok-a", ""); rec.Code != http.StatusUnauthorized {
+ t.Fatalf("A's token spoke about B: code = %d", rec.Code)
+ }
+ if len(f.slept) != 0 {
+ t.Fatal("B must not have been slept by A's token")
+ }
+}
+
+func TestAgentSockVerbs(t *testing.T) {
+ f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}}
+ h := New(f).Handler()
+
+ if rec := post(h, "/v1/ready", "bot.x.internal", "good", `{"version":"abc123"}`); rec.Code != http.StatusNoContent {
+ t.Fatalf("ready: %d", rec.Code)
+ }
+ if f.ready["bot.x.internal"] != "abc123" {
+ t.Fatal("ready must record the running version — it is a sidecar's only version signal")
+ }
+ if rec := post(h, "/v1/next-due", "bot.x.internal", "good", `{"at":1756}`); rec.Code != http.StatusNoContent {
+ t.Fatalf("next-due: %d", rec.Code)
+ }
+ if f.due["bot.x.internal"] != 1756 {
+ t.Fatalf("due = %d", f.due["bot.x.internal"])
+ }
+ if rec := post(h, "/v1/idle", "bot.x.internal", "good", ""); rec.Code != http.StatusNoContent {
+ t.Fatalf("idle: %d", rec.Code)
+ }
+ if len(f.slept) != 1 {
+ t.Fatal("idle must ask for a stop")
+ }
+}
+
+func TestAgentSockRefusesGarbage(t *testing.T) {
+ f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}}
+ h := New(f).Handler()
+ for _, body := range []string{"not json", `{"at":"soon"}`, strings.Repeat("x", 1<<20)} {
+ rec := post(h, "/v1/next-due", "bot.x.internal", "good", body)
+ if rec.Code != http.StatusBadRequest {
+ t.Fatalf("body %.20q: code = %d, want 400", body, rec.Code)
+ }
+ }
+ if rec := post(h, "/v1/nope", "bot.x.internal", "good", ""); rec.Code != http.StatusNotFound {
+ t.Fatalf("unknown verb: %d", rec.Code)
+ }
+ req := httptest.NewRequest("GET", "/v1/idle", nil)
+ // GET on a POST-only verb
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, req)
+ if rec.Code != http.StatusMethodNotAllowed && rec.Code != http.StatusUnauthorized {
+ t.Fatalf("GET: %d", rec.Code)
+ }
+}
+
+func TestAgentSockServeCleansUpItsSocket(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "agent.sock")
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan error, 1)
+ go func() { done <- New(&fakeWorkers{}).Serve(ctx, path) }()
+ waitFor(t, "socket to appear", func() bool { _, err := os.Stat(path); return err == nil })
+ cancel()
+ <-done
+ if _, err := os.Stat(path); !os.IsNotExist(err) {
+ t.Fatal("a stale socket file must not survive shutdown — the next start would fail to bind")
+ }
+}
+```
+
+Write `post(h http.Handler, path, host, tok, body string) *httptest.ResponseRecorder`
+as a small local helper.
+
+- [ ] **Step 2: Run and watch it fail.**
+
+- [ ] **Step 3: Implement**
+
+Package doc first, stating the direction:
+
+```go
+// Package agentsock is the agent's own listening socket — the ONE channel
+// between a sidecar and the platform, and it points inward: children dial
+// the agent, the agent never dials a child. That is what lets a sidecar
+// bind nothing, and what lets the identical artifact run off-platform
+// behind NAT with no protocol difference.
+//
+// Every exec child runs as the agent's user, so the socket's file mode is
+// not an identity boundary. Identity is the per-wake token the activator
+// mints and passes in the child's environment (CARLOS_SIDECAR_TOKEN),
+// scoped to one host for one wake: a token for A cannot speak about B, and
+// a token from a previous wake matches nothing.
+package agentsock
+```
+
+Implementation notes that the tests pin:
+
+- Compare tokens with `crypto/subtle.ConstantTimeCompare`, and refuse when
+ `WorkerToken(host)` returns `""` (asleep or unknown) — one 401 for every
+ failure, no distinguishing message.
+- Bound the body: `io.LimitReader(r.Body, 4096)`, the `NextDue` decode
+ precedent in `backend_exec.go:451`.
+- `Serve` should `os.Remove(path)` before binding (a stale socket from a
+ hard kill), `net.Listen("unix", path)`, `os.Chmod(path, 0o660)`, serve
+ with `http.Server`, and on `ctx.Done()` shut down and remove the file.
+- No timeouts longer than a few seconds; these are local calls.
+
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 8: Wire the socket, the boot pass and the ceilings into the agent
+
+**Files:**
+- Modify: `internal/host/agent.go`
+- Modify: `internal/activator/activator.go` (implement `NoteWorkerReady`)
+- Test: `internal/host/agent_test.go`
+
+**Interfaces:**
+- Consumes: Tasks 6 and 7.
+- Produces: a running agent socket on every box that runs the activator, and
+ a boot pass that starts non-hibernating sidecars.
+
+- [ ] **Step 1: Write the failing test**
+
+```go
+func TestWorkerBootPassStartsOnlyNonHibernatingSidecars(t *testing.T) {
+ routes := []registry.Route{
+ {Host: "a.x.sidecar.internal", Kind: "sidecar", Hibernate: false, Status: "ready"},
+ {Host: "b.x.sidecar.internal", Kind: "sidecar", Hibernate: true, Status: "ready"},
+ {Host: "c.example.com", Kind: "instance", Hibernate: false, Status: "ready"},
+ {Host: "d.x.sidecar.internal", Kind: "sidecar", Hibernate: false, Status: "draining"},
+ }
+ got := workersToStart(routes)
+ if len(got) != 1 || got[0].Host != "a.x.sidecar.internal" {
+ t.Fatalf("workersToStart = %v — only a ready, non-hibernating sidecar starts at boot; a hibernating one waits for its timer or a command", got)
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch it fail.**
+
+- [ ] **Step 3: Implement**
+
+```go
+// workersToStart is the boot pass's selection: a sidecar that does not
+// hibernate must be running whenever the agent is. A hibernating sidecar is
+// deliberately NOT started here — it waits for its due time, an operator
+// command, or (later) a delivery signal, which is the whole point of the
+// class.
+func workersToStart(routes []registry.Route) []registry.Route {
+ var out []registry.Route
+ for _, rt := range routes {
+ if rt.Kind == "sidecar" && !rt.Hibernate && (rt.Status == "" || rt.Status == "ready") {
+ out = append(out, rt)
+ }
+ }
+ return out
+}
+```
+
+Then, in the agent's startup block beside the other loops:
+
+- construct `agentsock.New(svc)` and `go srv.Serve(ctx, "/run/carlos/agent.sock")`
+ **only when the activator is enabled** (the same condition that
+ constructs the activator — with hibernation off there are no workers).
+- run the boot pass once after the first registry read, calling
+ `svc.WakeWorker(rt)` for each result.
+- The sweeper already handles the due-time wakes; make sure its
+ `DueInstances` results route to `WakeWorker` for `Kind == "sidecar"`
+ rather than the instance wake path.
+
+Add `NoteWorkerReady(host, version string)` to the activator: stamp
+`lastActive`, mark the settle window satisfied, and record the version via
+the registry's existing version setter so §9.2's honesty rule holds (a
+sidecar that never reports shows unknown, never "current").
+
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 9: The sidecar reconciler — and the removal-authority collision
+
+**Files:**
+- Create: `internal/provision/sidecar_reconcile.go`, `sidecar_reconcile_test.go`
+- Modify: `internal/provision/reconcile.go` (removal phase, ~651-673)
+- Modify: `internal/host/agent.go` (call it through `reconcileSerialized`)
+
+**Interfaces:**
+- Consumes: Task 2's shapes, Task 1's `Args`.
+- Produces: `func ReconcileSidecars(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig) error`
+
+**The collision, stated plainly:** the instance pass's removal phase is
+scoped by `Provisioned`, region, pool and absence-from-records, with **no
+Kind check**. Sidecar rows are `Provisioned` and can never appear in the
+instance record set, so the existing pass would delete every sidecar on
+every tick — minting and destroying forever, with no error anywhere. Each
+pass's removal authority must be scoped to its own Kind.
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestInstanceReconcileNeverRemovesSidecarRows(t *testing.T) {
+ f := setup(t) // reconcile_test.go's existing fixture
+ must(t, f.reg.Upsert(registry.Route{
+ Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar",
+ App: "eleven", Account: "bdm", Provisioned: true, Status: "ready",
+ }))
+ // No instance records at all: the pass's removal phase runs in full.
+ if err := Reconcile(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil {
+ t.Fatalf("Reconcile: %v", err)
+ }
+ if _, err := f.reg.Get("bot.eleven.bdm.sidecar.internal"); err != nil {
+ t.Fatal("the instance pass deleted a sidecar row — each pass's removal authority is scoped to its OWN kind, or the two passes destroy each other's work forever")
+ }
+}
+
+func TestSidecarReconcileMints(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{
+ Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, Channel: "stable",
+ })
+ writeSidecarRecord(t, f.store, SidecarRecord{
+ Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"),
+ App: "eleven", Account: "bdm", Channel: "stable",
+ })
+
+ if err := ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil {
+ t.Fatalf("ReconcileSidecars: %v", err)
+ }
+ rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm"))
+ if rt.Kind != "sidecar" {
+ t.Fatalf("Kind = %q", rt.Kind)
+ }
+ if rt.Socket != "" || rt.Addr != "" {
+ t.Fatalf("a sidecar row must carry NO socket and NO addr, got socket=%q addr=%q", rt.Socket, rt.Addr)
+ }
+ if rt.DBPath == "" || !rt.Provisioned || rt.Args != `["sidecar","run"]` {
+ t.Fatalf("route = %+v", rt)
+ }
+}
+
+func TestSidecarReconcileRemovesOnlyItsOwnKind(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ must(t, f.reg.Upsert(registry.Route{Host: "gone.eleven.bdm.sidecar.internal",
+ Kind: "sidecar", App: "eleven", Account: "bdm", Provisioned: true}))
+ must(t, f.reg.Upsert(registry.Route{Host: "app.example.com",
+ Kind: "instance", App: "eleven", Account: "bdm", Provisioned: true}))
+
+ if err := ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil {
+ t.Fatalf("ReconcileSidecars: %v", err)
+ }
+ if _, err := f.reg.Get("gone.eleven.bdm.sidecar.internal"); err == nil {
+ t.Fatal("a provisioned sidecar with no record must be removed")
+ }
+ if _, err := f.reg.Get("app.example.com"); err != nil {
+ t.Fatal("the sidecar pass must never touch an instance row")
+ }
+}
+
+func TestSidecarReconcileSkipsRemovalWhenAReadFailed(t *testing.T) {
+ // The fail-safe: a transient read failure must never mass-retire a
+ // tenant's fleet. Minting still proceeds; minting is additive.
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ must(t, f.reg.Upsert(registry.Route{Host: "keep.eleven.bdm.sidecar.internal",
+ Kind: "sidecar", App: "eleven", Account: "bdm", Provisioned: true}))
+ f.store.failReadsFor(SidecarsPrefix("bdm", "eleven")) // fixture hook
+
+ _ = ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})
+ if _, err := f.reg.Get("keep.eleven.bdm.sidecar.internal"); err != nil {
+ t.Fatal("a read failure skipped nothing — one bad list must not retire a fleet")
+ }
+}
+
+func TestSidecarReconcileRefusesHostWithAStagedCoSpawnFile(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ host := SidecarHost("bot", "eleven", "bdm")
+ must(t, os.WriteFile(filepath.Join(f.dir, host+"-sidecar.env"), []byte("X=1\n"), 0o600))
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}})
+ writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", Host: host, App: "eleven", Account: "bdm"})
+
+ _ = ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})
+ if _, err := f.reg.Get(host); err == nil {
+ t.Fatal("a host with a staged legacy co-spawn file must not also get a native sidecar row — one host, one mechanism")
+ }
+}
+
+func TestSidecarReconcileConvergesArgsFromTheManifest(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}})
+ writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm"})
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}))
+
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"worker", "run"}})
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}))
+
+ rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm"))
+ if rt.Args != `["worker","run"]` {
+ t.Fatalf("Args = %q — truth lives in the bucket and is re-stamped every pass, exactly as seed is", rt.Args)
+ }
+}
+
+func TestSidecarReconcileConvergesTheCeilings(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{
+ Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, IdleSeconds: 300, MaxRunSeconds: 3600})
+ writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot",
+ Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm"})
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}))
+
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{
+ Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, IdleSeconds: 60, MaxRunSeconds: 120})
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}))
+
+ rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm"))
+ if rt.IdleSeconds != 60 || rt.MaxRunSeconds != 120 {
+ t.Fatalf("ceilings = %d/%d, want 60/120 — the sweeper reads the row, so the row must follow the manifest", rt.IdleSeconds, rt.MaxRunSeconds)
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch them fail.**
+
+- [ ] **Step 3: Guard the instance pass's removal phase**
+
+In `reconcile.go`'s removal loop, in the same condition that already checks
+`rt.Provisioned` and scope:
+
+```go
+ // Removal authority is scoped to this pass's OWN kind. A sidecar row
+ // is Provisioned too, and can never appear in the instance record
+ // set — without this guard the two passes would delete each other's
+ // rows on every tick, forever, with no error anywhere.
+ if rt.Kind == "sidecar" {
+ continue
+ }
+```
+
+- [ ] **Step 4: Implement `ReconcileSidecars`**
+
+Mirror `Reconcile` (`reconcile.go:330-675`) exactly, with these
+substitutions and additions:
+
+- records come from `ParseSidecarKey` over `apps/`, manifests from
+ `GetSidecarManifest`.
+- scope is the same `rec.Region != cfg.Region || !poolServed(cfg.Pools, rec.Pool)` skip.
+- **placement gate**: skip a record whose `PlacementOrDefault()` is
+ `"no-inbound"` unless `cfg.Inbound == "none"` (Task 10 supplies the field).
+- mint:
+ ```go
+ func sidecarRouteFor(rec SidecarRecord, m SidecarManifest, dataDir string) registry.Route {
+ argv, _ := json.Marshal(m.Args)
+ return registry.Route{
+ Host: rec.Host, Slug: rec.Name, Kind: "sidecar",
+ App: rec.App, Account: rec.Account, Pool: rec.Pool,
+ Region: rec.Region, Channel: rec.Channel, Environment: rec.Environment,
+ DBPath: DefaultDBPath(dataDir, rec.Host),
+ Hibernate: m.HibernateOrDefault(),
+ Args: string(argv),
+ IdleSeconds: int(m.IdleOrDefault() / time.Second),
+ MaxRunSeconds: int(m.MaxRunOrDefault() / time.Second),
+ Provisioned: true,
+ Status: "ready",
+ // Socket and Addr are deliberately left empty: a sidecar binds
+ // nothing, and the edge refuses the kind outright.
+ }
+ }
+ ```
+- converge: re-stamp `Args` (narrow `SetArgs`), the ceilings (narrow
+ `SetWorkerLimits`), `Channel`, `Environment`, and `Hibernate` when they
+ differ.
+- removal: only `rt.Kind == "sidecar" && rt.Provisioned`, in scope, absent
+ from the record set, and not in `protect`; **skipped entirely when any
+ read failed**.
+- the co-spawn conflict: before minting, `os.Stat(filepath.Join(cfg.DataDir, rec.Host+"-sidecar.env"))`
+ — if it exists, log loudly and skip. One host, one mechanism.
+
+Finally, add the leg to `internal/host/agent.go` beside the instance one,
+**inside `reconcileSerialized`'s mutex** so it cannot race the instance pass
+(the comment at `agent.go:525-537` explains exactly why an overlapping pass
+is dangerous), and inside `fullPass` so a console serial bump converges it
+within ~2s.
+
+- [ ] **Step 5: Run the tests, full gate, commit.**
+
+---
+
+### Task 10: Placement — boxes that accept no-inbound work
+
+**Files:**
+- Modify: `internal/host/agent.go` (`provisionTickConfig`, ~1316-1334)
+- Modify: `internal/provision/reconcile.go` (mint gate)
+- Modify: `internal/provision/sidecar_reconcile.go` (placement gate)
+- Test: `internal/host/agent_test.go`, `internal/provision/reconcile_test.go`
+
+**Interfaces:**
+- Consumes: Task 9.
+- Produces: `ReconcileConfig.Inbound string` (`""`/`"any"` or `"none"`).
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestProvisionTickConfigReadsInbound(t *testing.T) {
+ env := map[string]string{"CARLOS_REGION": "ie", "CARLOS_POOLS": "eleven", "CARLOS_INBOUND": "none"}
+ cfg := provisionTickConfig(func(k string) string { return env[k] })
+ if cfg.Inbound != "none" {
+ t.Fatalf("Inbound = %q", cfg.Inbound)
+ }
+ cfg = provisionTickConfig(func(string) string { return "" })
+ if cfg.Inbound != "" {
+ t.Fatal("an unset CARLOS_INBOUND must be an inert default — today's behaviour exactly")
+ }
+}
+
+func TestNoInboundBoxRefusesRoutedInstances(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ writeManifest(t, f.store, "bdm", "tito", Manifest{Schema: 1, App: "tito", Domain: "go.tito.io"})
+ writeRecord(t, f.store, Record{Schema: 1, Host: "acme.go.tito.io", Slug: "acme", App: "tito", Account: "bdm", Channel: "stable"})
+
+ err := Reconcile(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir, Inbound: "none"})
+ if err != nil {
+ t.Fatalf("Reconcile: %v", err)
+ }
+ if _, err := f.reg.Get("acme.go.tito.io"); err == nil {
+ t.Fatal("a box that cannot serve must never be handed something to serve — a misfiled record is a log line, not a silent outage")
+ }
+}
+
+func TestNoInboundPlacementOnlyLandsOnMatchingBoxes(t *testing.T) {
+ f := setup(t)
+ seedAccount(t, f.store, "bdm")
+ writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{
+ Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, Placement: "no-inbound"})
+ writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot",
+ Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm", Placement: "no-inbound"})
+
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) // ordinary box
+ if _, err := f.reg.Get(SidecarHost("bot", "eleven", "bdm")); err == nil {
+ t.Fatal("a no-inbound workload must not land on a box that accepts inbound work")
+ }
+
+ must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir, Inbound: "none"}))
+ if _, err := f.reg.Get(SidecarHost("bot", "eleven", "bdm")); err != nil {
+ t.Fatal("it must land on a box that declares CARLOS_INBOUND=none")
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement** — add the field, read the env var, and add the
+ two gates with the comments the tests' failure messages quote.
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 11: Config delivery and restart reach sidecars
+
+**Files:**
+- Modify: `internal/appconfig/syncexec.go` (`execRouteBase`, ~398-407)
+- Modify: `internal/host/restart.go` (~77, 192, 545)
+- Test: `internal/appconfig/syncexec_test.go`, `internal/host/restart_test.go`
+
+**`release.Adoptable` is deliberately NOT changed.** It is the most reused
+predicate in the repo — adopt membership, capability convergence, socket and
+backing advisories, status pages — and every one of those is correct to
+exclude a sidecar. Each consumer that genuinely wants sidecars opts in by
+name. Two deliberate edits instead of one blind one.
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestSidecarGetsItsConfigEnvFile(t *testing.T) {
+ rt := registry.Route{Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar",
+ App: "eleven", Status: "ready"}
+ if !execRouteBase(rt) {
+ t.Fatal("a sidecar must receive config delivery — this is the direct replacement for the hand-staged <host>-sidecar.env, and the whole reason the file path existed")
+ }
+}
+
+func TestSidecarConfigStillExcludesPlatformAndUnitRoutes(t *testing.T) {
+ base := registry.Route{Host: "bot.x.sidecar.internal", Kind: "sidecar", App: "eleven", Status: "ready"}
+ plat := base
+ plat.Platform = true
+ if execRouteBase(plat) {
+ t.Fatal("a Platform alias is still excluded")
+ }
+ unit := base
+ unit.Backing = registry.BackingUnit
+ if execRouteBase(unit) {
+ t.Fatal("unit-backed routes are still excluded")
+ }
+}
+
+func TestRestartPassCyclesSidecars(t *testing.T) {
+ routes := []registry.Route{
+ {Host: "bot.x.sidecar.internal", Kind: "sidecar", App: "eleven", Account: "bdm", Status: "ready"},
+ }
+ if len(restartCandidates(routes)) != 1 {
+ t.Fatal("carlos restart -app X must cycle that app's sidecars too")
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement** — widen `execRouteBase`'s kind test to
+ `(rt.Kind == "instance" || rt.Kind == "sidecar")`, leaving every other
+ clause exactly as it is; and in `restart.go` accept sidecars alongside
+ `release.Adoptable` routes, cycling them via the activator's stop/wake
+ rather than the adopt path. Also confirm the reaper in
+ `syncexec.go:617` still treats `<host>-sidecar.env` as someone else's file
+ — the legacy path must stay untouched (Task 16 documents why).
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 12: Console API — the sidecar verbs
+
+**Files:**
+- Create: `internal/console/apisidecars.go`, `apisidecars_test.go`
+- Modify: `internal/console/authcli.go` (`registerCLIRoutes`, ~273-286)
+
+**Interfaces:**
+- Consumes: Task 2's shapes, Task 13's wake object (build 13 first if you
+ prefer; the wake handler can land in either order).
+- Produces the routes:
+ ```
+ PUT /api/cli/apps/{account}/{app}/sidecars-manifest
+ GET /api/cli/apps/{account}/{app}/sidecars
+ POST /api/cli/apps/{account}/{app}/sidecars
+ GET /api/cli/apps/{account}/{app}/sidecars/{name}
+ DELETE /api/cli/apps/{account}/{app}/sidecars/{name}
+ POST /api/cli/apps/{account}/{app}/sidecars/{name}/wake
+ ```
+
+**The app-held token works unchanged.** `CLIAppToken` binds a token to
+(account, app); these verbs authorize against the same principal, which is
+exactly the "apps can provision sidecar instances easily" requirement —
+Eleven's app process, holding no AWS credentials and no box access, creates
+a bot per enrolled member.
+
+Refusal order, copied from `apiinstances.go:474-478` and its tests:
+**principal gate → rate limit → releases configured → manifest present
+(409 naming `carlos sidecars enable`) → name shape (400) → region shape and
+allow-list (400) → channel (400) → shape validation (400) → create-only
+write (409 on a lost race) → audit + `bumpSerial` → 201.**
+
+- [ ] **Step 1: Write the failing tests**
+
+Mirror `apiinstances_test.go` exactly, reusing its fixture helpers
+(`rawReq`, `canonicalNotFound`, `assertNotFound`, `newConsoleFixture`):
+
+```go
+func TestAPISidecarRefusalMatrix(t *testing.T) {
+ // Every refusal before the principal is proven must be BYTE-IDENTICAL
+ // to http.NotFound's own output: a non-member must never learn that an
+ // account, an app, or a sidecar exists.
+ // ... one case per: no such account, no such app, not a member,
+ // app token bound to a different app, user token on an app verb.
+}
+
+func TestAPISidecarCreateRefusalOrder(t *testing.T) {
+ // Walk the order above with one token, asserting each stage's code and
+ // that the NEXT stage was not reached.
+}
+
+func TestAPISidecarAppTokenContainment(t *testing.T) {
+ // An app token creates/lists/deletes/wakes its OWN app's sidecars, and
+ // 401s on promote and on minting another token.
+}
+
+func TestAPISidecarCreateIsCreateOnly(t *testing.T) {
+ // Second POST for the same name -> 409, never a silent overwrite.
+}
+
+func TestAPISidecarDeleteIsIdempotent(t *testing.T) {
+ // DELETE of an absent name -> 200. The caller's goal is "not there".
+ // And it must work after the manifest is gone: deletion outlives policy.
+}
+
+func TestAPISidecarCreateWithoutManifestNamesTheEnableVerb(t *testing.T) {
+ // 409 whose body names `carlos sidecars enable` — membership is already
+ // proven at this point, so this is not a leak.
+}
+
+func TestAPISidecarWakeBumpsTheGeneration(t *testing.T) {
+ // Two wakes -> generation 1 then 2, and the serial moved both times.
+}
+```
+
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement** `apisidecars.go` following `apiinstances.go`'s
+ structure handler-for-handler, and register the routes. The enable handler
+ is gated by `apiAccountApp` **directly** (it is a human act), like
+ `handleAPIInstanceEnable`; the rest go through the shared gate.
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 13: The wake generation object and the box-side pass
+
+**Files:**
+- Create: `internal/release/sidecarwake.go`, `sidecarwake_test.go`
+- Modify: `internal/host/restart.go` or a sibling in `internal/host`
+- Test: `internal/host/*_test.go`
+
+**Interfaces:**
+- Produces:
+ ```go
+ func SidecarWakeKey(acct, app, name string) string // apps/<acct>/<app>/sidecar-wake/<name>.json
+ type SidecarWake struct{ Schema, Generation int; At, By, Reason string }
+ ```
+
+Copy `restartgen.go` (`internal/release/restartgen.go`) wholesale: a small
+CAS-guarded generation object whose **etag moving** is the entire signal.
+The box compares etags per (account, app, name) against a cache and wakes
+what moved — the `restartGenPass` shape (`internal/host/restart.go:128`).
+
+Nothing is pushed at a box. That is what lets the box live in a private
+subnet.
+
+- [ ] **Step 1: Write the failing tests** — an etag move wakes exactly the
+ named sidecar; an unchanged etag wakes nothing; a first sighting on an
+ unseeded cache does **not** replay a wake (the no-replay-on-absence rule
+ the serial watcher already uses, `agent.go:1032-1042`); a read error for
+ one app does not suppress the others.
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 14: `carlos sidecars`
+
+**Files:**
+- Create: `sidecars_cli.go`, `sidecars_cli_test.go`, `sidecars_e2e_test.go`
+- Modify: `main.go` (dispatch), `apimode.go` (client methods)
+
+**Interfaces:**
+- Consumes: Task 12's routes.
+- Produces the verbs:
+ ```
+ carlos sidecars enable -app <app> -args '<argv>' [-placement no-inbound]
+ [-regions ie,fleet/eleven] [-pool p] [-channel c]
+ [-idle 5m] [-max-run 1h] [-no-hibernate]
+ carlos sidecars create -app <app> <name>
+ carlos sidecars ls -app <app> [-long]
+ carlos sidecars wake -app <app> <name>
+ carlos sidecars rm -app <app> <name>
+ carlos sidecars logs -app <app> <name>
+ ```
+
+Follow `instances_cli.go` precisely: subcommand words dispatch positionally
+before flag parsing; `enable` works in **both** API and direct/store mode
+(operator-only flags refused in signed-in mode); `create`/`ls`/`wake`/`rm`
+are **API-mode only** via `requireAPISession`, whose refusal must name
+sidecars' own advice, never another verb's.
+
+`ls` prints the short name by default and the synthetic host only under
+`-long`. It shows **running version against desired channel version**, and a
+sidecar that has never called `/v1/ready` shows `unknown`, never the desired
+version — a stamp that overclaims is worse than an absent one.
+
+`logs` delegates to the existing `carlos logs` path for the sidecar's host.
+
+- [ ] **Step 1: Write the failing tests** — unit tests for flag parsing and
+ the duration flags (`-idle 5m` → `IdleSeconds: 300`), plus an e2e test in
+ the `instances_e2e_test.go` mould driving a real `console.Server` over
+ `httptest` end to end: enable → create → ls → wake → rm.
+
+ **Every test that shells out to the carlos binary must build its env
+ through `carlosTestEnv` and set `cmd.Dir` to a scratch dir.** The binary
+ walks cwd ancestors for `.carlos/config`; the fallback is the operator's
+ real login and default console. A logged-in laptop running
+ `go test ./...` once minted 40+ real accounts on the production console.
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 15: The console tells the truth about sidecars
+
+**Files:**
+- Modify: `internal/console/instances.go` (~88-97), `internal/console/overview.go`
+ (~129-144, ~237-264, ~286-306), `internal/status/snapshot.go` (~406-427)
+- Test: the matching `_test.go` files
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+func TestReportingHostIsNeverASidecar(t *testing.T) {
+ routes := []registry.Route{
+ {Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", Status: "ready"},
+ {Host: "eleven.example.com", Kind: "instance", Status: "ready"},
+ }
+ if got := reportingHost(routes); got != "eleven.example.com" {
+ t.Fatalf("reportingHost = %q — polling an unroutable sidecar makes a HEALTHY app read 'not reporting'", got)
+ }
+}
+
+func TestSidecarIsNotAPublicResident(t *testing.T) {
+ counts := CollectBoxCounts([]registry.Route{{Host: "bot.x.sidecar.internal", Kind: "sidecar"}})
+ if counts.Resident != 0 {
+ t.Fatal("a sidecar must not be counted as a public resident app instance")
+ }
+ if counts.Sidecars != 1 {
+ t.Fatal("it belongs in its own bucket")
+ }
+}
+
+func TestSidecarHealthWordIsNotServing(t *testing.T) {
+ if w := healthWord("sidecar", "ready", false, false, false); w == "Serving" {
+ t.Fatal("a sidecar never serves a request — 'Serving' is false")
+ }
+}
+```
+
+- [ ] **Step 2: Run and watch them fail.**
+- [ ] **Step 3: Implement** — filter the reporting-host picker by Kind, give
+ `CollectBox` an explicit sidecar branch and a `Sidecars` counter (additive
+ field), count sidecars separately in the overview, and give `healthWord`
+ the sidecar vocabulary: `Running`, `Asleep`, `Waking`, `Crashed`,
+ `Never started`. Sidecars are **not** on the public status page: their
+ existence is account-visible, not public.
+- [ ] **Step 4: Run the tests, full gate, commit.**
+
+---
+
+### Task 16: Documentation, the runbook, and the legacy split
+
+**Files:**
+- Create: `infra/no-inbound-box.md`
+- Modify: `CLAUDE.md`
+- Modify: `internal/activator/activator.go` (package doc), `backend_exec.go`
+ (the `HasSidecar`/`StartSidecar` comments)
+
+- [ ] **Step 1: Write `infra/no-inbound-box.md`** — the private-subnet
+ runbook, in `infra/eleven-fleet.md`'s voice: preconditions, the tofu
+ shape (private subnet, no IGW route, egress via NAT, no inbound SG rules,
+ no SSM), the box env contract (`CARLOS_INBOUND=none` plus the fleet
+ contract), verification, and an abort path at every step. Mark clearly:
+ **written for Paul's applies; nothing here has been applied.**
+
+ Note the easier posture too: fleet boxes are *already* outbound-only dial,
+ so Eleven's first sidecar box needs no new network shape at all — the
+ private-subnet variant is the harder posture, available when wanted.
+
+- [ ] **Step 2: Record the legacy split** in the activator's package doc and
+ beside `HasSidecar`: the file-staged `<host>-sidecar.env` co-spawn is
+ **legacy but live** (Eleven runs on it today), the record path is native,
+ a host may not have both, and retiring the file path is a follow-up gated
+ on Eleven's own migration.
+
+- [ ] **Step 3: Add a CLAUDE.md bullet** under "Rules this repo has paid
+ for":
+
+ ```markdown
+ - **A sidecar is a member that nothing can dial.** `Kind == "sidecar"`
+ rows carry no socket and no addr, are refused by the edge on kind (not
+ merely unmatched by it), and are never advertised in a fleet binding —
+ a fleet-placed row reaches the edge's fleet arm on shape alone. Each
+ reconcile pass's removal authority is scoped to its OWN kind: the
+ instance pass and the sidecar pass would otherwise delete each other's
+ rows on every tick, forever, with no error anywhere.
+ ```
+
+- [ ] **Step 4: Full gate, commit.**
+
+---
+
+## Final verification before the PR
+
+- [ ] `go build ./...`, `go vet ./...`, `go test ./... -count=1` — all clean.
+- [ ] `GOOS=darwin go build ./...` — clean (`internal/host` links into the
+ CLI, which people run on Macs).
+- [ ] `gofmt -l` clean for every file this branch touched.
+- [ ] `git log --oneline main..HEAD` reads as one coherent story.
+- [ ] No live box was touched; no real bucket was written.
+- [ ] Push the branch and open the PR. **Do not merge it** — this is a new
+ subsystem touching the activator, the edge and the console's auth
+ surface, which is exactly the diff CLAUDE.md says waits for a person.