| 1 | # Sidecar Members Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Make a sidecar a first-class platform member — provisioned through |
| 6 | an API like a web app instance, unreachable from the internet by |
| 7 | construction, hibernating with timer and command wakes, placeable on a box |
| 8 | with no inbound path at all. |
| 9 | |
| 10 | **Architecture:** Mirror instance provisioning one directory over |
| 11 | (`sidecars.json` manifest + `sidecars/<name>.json` create-only records + |
| 12 | a reconcile pass + console verbs + an app-held token), key sidecars by a |
| 13 | synthetic `<name>.<app>.<sqid>.sidecar.internal` host so every host-keyed |
| 14 | subsystem works unchanged, and reuse `activator.Service` for the lifecycle |
| 15 | while keeping sidecars entirely off its request path. A sidecar binds |
| 16 | nothing; it reports readiness, next-due and idleness by dialing **out** to a |
| 17 | new agent-side unix socket. |
| 18 | |
| 19 | **Tech Stack:** Go (stdlib only — no new dependencies), modernc sqlite |
| 20 | (`internal/registry`), S3/dir-backed `release.Store`, litestream via |
| 21 | `internal/lease`. |
| 22 | |
| 23 | **Spec:** `docs/superpowers/specs/2026-08-17-sidecar-members-design.md` |
| 24 | |
| 25 | ## Global Constraints |
| 26 | |
| 27 | - **The gate, before every commit:** `go build ./...`, `go vet ./...`, |
| 28 | `go test ./... -count=1` all clean, and `gofmt -l` clean for files you |
| 29 | touched. Three pre-existing unformatted files elsewhere are left alone. |
| 30 | - **Sandbox note:** use `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod |
| 31 | GOPROXY=off`. Go commands may need the sandbox disabled ("read-only file |
| 32 | system" from the toolchain is the tell). |
| 33 | - **The gate is not Linux-only.** `internal/host` links into the CLI, which |
| 34 | people run on Macs. `GOOS=darwin go build ./...` must pass too. |
| 35 | - **Wire-shape changes are additive.** Live pointers, manifests and registry |
| 36 | rows exist in production buckets and on production boxes; anything written |
| 37 | before your change must still parse. Never remove or repurpose a field. |
| 38 | - **When a storage shape changes, the consumer list is everything that |
| 39 | PARSES the shape** — not everything that calls the API. |
| 40 | - **No new dependencies.** Standard library only. |
| 41 | - **Never touch a live box.** This branch is build-and-PR only. No deploys, |
| 42 | no `ssm`, no `aws` calls, no writes to any real bucket. |
| 43 | - **Reserved suffix, exact string:** `.sidecar.internal` |
| 44 | - **Synthetic host format, exact:** `<name>.<app>.<sqid>.sidecar.internal` |
| 45 | - **Registry Kind value, exact:** `sidecar` |
| 46 | - **Agent socket path, exact:** `/run/carlos/agent.sock` |
| 47 | - **Env var names, exact:** `CARLOS_AGENT_SOCKET`, `CARLOS_SIDECAR_TOKEN`, |
| 48 | `CARLOS_SIDECAR_NAME`, `CARLOS_SIDECAR_HOST`, `CARLOS_INBOUND` |
| 49 | - **Bucket keys, exact:** `apps/<sqid>/<app>/sidecars.json`, |
| 50 | `apps/<sqid>/<app>/sidecars/<name>.json`, |
| 51 | `apps/<sqid>/<app>/sidecar-wake/<name>.json` |
| 52 | - **Defaults, exact:** idle `5m`, `max_run` `1h`, settle window `10s`, |
| 53 | hibernate `true`, placement `any`. |
| 54 | - **Commit style:** one commit per task, imperative subject, body explaining |
| 55 | *why*. End every commit message with: |
| 56 | ``` |
| 57 | Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
| 58 | Claude-Session: https://claude.ai/code/session_014WvcQVWggXkE5BAkjRDgWF |
| 59 | ``` |
| 60 | |
| 61 | --- |
| 62 | |
| 63 | ## File Structure |
| 64 | |
| 65 | **Created:** |
| 66 | - `internal/provision/sidecar.go` — sidecar manifest + record wire shapes, |
| 67 | key helpers, validation, store operations. Sibling to `provision.go`, |
| 68 | same package: it is the same kind of object one directory over, and it |
| 69 | reuses `labelRe`, `ValidRegion`, `ValidPool` directly. |
| 70 | - `internal/provision/sidecar_test.go` |
| 71 | - `internal/provision/sidecar_reconcile.go` — the records→routes pass. |
| 72 | - `internal/provision/sidecar_reconcile_test.go` |
| 73 | - `internal/agentsock/agentsock.go` — the agent-side unix socket: listener, |
| 74 | per-wake token registry, the four verbs. Its own package because it is a |
| 75 | server with its own lifecycle, and because `internal/activator` must be |
| 76 | able to depend on it without a cycle. |
| 77 | - `internal/agentsock/agentsock_test.go` |
| 78 | - `internal/release/sidecarwake.go` — the wake generation object. |
| 79 | - `internal/release/sidecarwake_test.go` |
| 80 | - `internal/console/apisidecars.go` — the console CLI verbs. |
| 81 | - `internal/console/apisidecars_test.go` |
| 82 | - `sidecars_cli.go` (root package) — `carlos sidecars`. |
| 83 | - `sidecars_cli_test.go`, `sidecars_e2e_test.go` (root package) |
| 84 | - `infra/no-inbound-box.md` — the private-subnet runbook (Paul applies). |
| 85 | |
| 86 | **Modified:** |
| 87 | - `internal/registry/registry.go` — `args` column (FOUR places), `SetArgs`, |
| 88 | `DueInstances` widening. |
| 89 | - `internal/edge/edge.go` — structural sidecar refusal. |
| 90 | - `internal/host/fleetbind.go` — never publish a sidecar binding. |
| 91 | - `internal/host/restart.go` — cycle sidecars. |
| 92 | - `internal/host/agent.go` — `CARLOS_INBOUND`, the sidecar reconcile leg, |
| 93 | the agent socket's lifecycle, the wake pass. |
| 94 | - `internal/appconfig/syncexec.go` — deliver config to sidecars. |
| 95 | - `internal/activator/activator.go` — `StartWorker` seam, `WakeWorker`, |
| 96 | socketless stop, `max_run`. |
| 97 | - `internal/activator/backend_exec.go`, `backend_unit.go` — the seam's two |
| 98 | implementations. |
| 99 | - `internal/provision/reconcile.go` — instance removal phase skips sidecars. |
| 100 | - `internal/console/authcli.go` — route registration. |
| 101 | - `internal/console/instances.go`, `overview.go` — Kind guards, sidecar words. |
| 102 | - `internal/status/snapshot.go` — sidecars are not public Residents. |
| 103 | - `cli.go` (root) — `carlos add` rails off `-kind sidecar` and the suffix. |
| 104 | - `main.go` (root) — `sidecars` dispatch. |
| 105 | - `apimode.go` (root) — the client methods. |
| 106 | - `CLAUDE.md` — the co-spawn/native split. |
| 107 | |
| 108 | --- |
| 109 | |
| 110 | ## Task ordering and dependencies |
| 111 | |
| 112 | ``` |
| 113 | 1 registry ──┬── 3 edge/fleetbind ── 4 suffix rails |
| 114 | ├── 5 StartWorker ── 6 lifecycle ── 7 agentsock ── 8 wiring |
| 115 | ├── 11 config+restart |
| 116 | └── 15 console UI |
| 117 | 2 provision shapes ──┬── 9 reconciler ── 10 placement |
| 118 | ├── 12 console API ── 14 CLI |
| 119 | └── 13 wake object |
| 120 | 16 docs (last) |
| 121 | ``` |
| 122 | |
| 123 | Tasks 1 and 2 are independent of each other and unblock everything else. |
| 124 | |
| 125 | --- |
| 126 | |
| 127 | ### Task 1: Registry — the `sidecar` kind, the `args` column, and the due-time widening |
| 128 | |
| 129 | **Files:** |
| 130 | - Modify: `internal/registry/registry.go` |
| 131 | - Test: `internal/registry/registry_test.go` |
| 132 | |
| 133 | **Interfaces:** |
| 134 | - Consumes: nothing. |
| 135 | - Produces: `Route.Args string` (JSON argv, `""` = none); |
| 136 | `Route.IdleSeconds int`, `Route.MaxRunSeconds int` (0 = use the platform |
| 137 | default); `(*Registry).SetArgs(host, args string) error`; |
| 138 | `(*Registry).SetWorkerLimits(host string, idle, maxRun int) error`; |
| 139 | `DueInstances` now returns `kind IN ('instance','sidecar')`. |
| 140 | |
| 141 | **Why the ceilings live on the row.** The sweeper decides on every tick |
| 142 | whether a worker has outrun its ceiling, and it must do that from local |
| 143 | state — a bucket read per wake would put the control plane in the hot path |
| 144 | of a timer. So `idle_seconds`/`max_run_seconds` are materialized from the |
| 145 | app's manifest by the reconcile pass exactly as `args` and `seed` are: |
| 146 | truth stays in the bucket, the row is the box's local copy. |
| 147 | |
| 148 | **Why this is first and why it matters:** `DueInstances` is the sweeper's |
| 149 | only source of timer wakes. Left filtering `kind = 'instance'`, a sidecar's |
| 150 | `next_due` would be written, stored, and never once acted on — the schedule |
| 151 | wake source would silently not exist, with no error anywhere. |
| 152 | |
| 153 | - [ ] **Step 1: Write the failing tests** |
| 154 | |
| 155 | Add to `internal/registry/registry_test.go`: |
| 156 | |
| 157 | ```go |
| 158 | func TestDueInstancesIncludesSidecars(t *testing.T) { |
| 159 | r := testRegistry(t) // use this file's existing helper for a temp registry |
| 160 | if err := r.Upsert(Route{ |
| 161 | Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", |
| 162 | DBPath: "/tmp/bot.db", Hibernate: true, Status: "ready", |
| 163 | }); err != nil { |
| 164 | t.Fatalf("upsert sidecar: %v", err) |
| 165 | } |
| 166 | if err := r.SetNextDue("bot.eleven.bdm.sidecar.internal", 100); err != nil { |
| 167 | t.Fatalf("SetNextDue: %v", err) |
| 168 | } |
| 169 | |
| 170 | due, err := r.DueInstances(200) |
| 171 | if err != nil { |
| 172 | t.Fatalf("DueInstances: %v", err) |
| 173 | } |
| 174 | if len(due) != 1 || due[0].Host != "bot.eleven.bdm.sidecar.internal" { |
| 175 | t.Fatalf("DueInstances = %+v, want the due sidecar — a sidecar's timer wake is its ONLY schedule wake source", due) |
| 176 | } |
| 177 | } |
| 178 | |
| 179 | func TestArgsRoundTrips(t *testing.T) { |
| 180 | r := testRegistry(t) |
| 181 | if err := r.Upsert(Route{Host: "bot.x.internal", Kind: "sidecar", Args: `["sidecar","run"]`}); err != nil { |
| 182 | t.Fatalf("upsert: %v", err) |
| 183 | } |
| 184 | got, err := r.Get("bot.x.internal") |
| 185 | if err != nil { |
| 186 | t.Fatalf("get: %v", err) |
| 187 | } |
| 188 | if got.Args != `["sidecar","run"]` { |
| 189 | t.Fatalf("Args = %q, want the argv back", got.Args) |
| 190 | } |
| 191 | |
| 192 | if err := r.SetArgs("bot.x.internal", `["worker","run"]`); err != nil { |
| 193 | t.Fatalf("SetArgs: %v", err) |
| 194 | } |
| 195 | got, _ = r.Get("bot.x.internal") |
| 196 | if got.Args != `["worker","run"]` { |
| 197 | t.Fatalf("Args after SetArgs = %q", got.Args) |
| 198 | } |
| 199 | } |
| 200 | |
| 201 | func TestWorkerLimitsRoundTrip(t *testing.T) { |
| 202 | r := testRegistry(t) |
| 203 | if err := r.Upsert(Route{Host: "bot.x.internal", Kind: "sidecar", IdleSeconds: 300, MaxRunSeconds: 3600}); err != nil { |
| 204 | t.Fatalf("upsert: %v", err) |
| 205 | } |
| 206 | got, _ := r.Get("bot.x.internal") |
| 207 | if got.IdleSeconds != 300 || got.MaxRunSeconds != 3600 { |
| 208 | t.Fatalf("limits = %d/%d, want 300/3600 — the sweeper reads these locally on every tick", got.IdleSeconds, got.MaxRunSeconds) |
| 209 | } |
| 210 | if err := r.SetWorkerLimits("bot.x.internal", 60, 120); err != nil { |
| 211 | t.Fatalf("SetWorkerLimits: %v", err) |
| 212 | } |
| 213 | got, _ = r.Get("bot.x.internal") |
| 214 | if got.IdleSeconds != 60 || got.MaxRunSeconds != 120 { |
| 215 | t.Fatalf("limits after set = %d/%d", got.IdleSeconds, got.MaxRunSeconds) |
| 216 | } |
| 217 | } |
| 218 | |
| 219 | func TestRouteWrittenBeforeArgsStillParses(t *testing.T) { |
| 220 | // The additive rule: a row written by older code has no args value. |
| 221 | r := testRegistry(t) |
| 222 | if err := r.Upsert(Route{Host: "old.example.com", Kind: "instance"}); err != nil { |
| 223 | t.Fatalf("upsert: %v", err) |
| 224 | } |
| 225 | got, err := r.Get("old.example.com") |
| 226 | if err != nil { |
| 227 | t.Fatalf("get: %v", err) |
| 228 | } |
| 229 | if got.Args != "" || got.IdleSeconds != 0 || got.MaxRunSeconds != 0 { |
| 230 | t.Fatalf("row = %+v, want zero values for a pre-existing row", got) |
| 231 | } |
| 232 | } |
| 233 | ``` |
| 234 | |
| 235 | If `testRegistry` is not the helper's name in this file, use whatever the |
| 236 | file already uses to open a registry in `t.TempDir()` — do not add a new |
| 237 | helper. |
| 238 | |
| 239 | - [ ] **Step 2: Run the tests and watch them fail** |
| 240 | |
| 241 | ```bash |
| 242 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 243 | go test ./internal/registry/ -run 'DueInstancesIncludesSidecars|ArgsRoundTrips|BeforeArgs' -count=1 |
| 244 | ``` |
| 245 | |
| 246 | Expected: compile failure — `Route` has no field `Args`, `SetArgs` undefined. |
| 247 | |
| 248 | - [ ] **Step 3: Add the column in all FOUR places** |
| 249 | |
| 250 | `registry.go` warns at the `routeCols` comment that a column lives in four |
| 251 | hand-maintained lists and that a miscount is a **silent column shift**, not |
| 252 | an error. Make all four edits together: |
| 253 | |
| 254 | 1. Struct field, beside `Seed` (`registry.go` ~line 127 area): |
| 255 | ```go |
| 256 | // Args is a sidecar's argv as a JSON array, materialized from the app's |
| 257 | // sidecars manifest on every reconcile pass (truth stays in the bucket). |
| 258 | // Empty for every other kind of route. |
| 259 | Args string |
| 260 | ``` |
| 261 | 2. Migration — append to the `ALTER TABLE` list in `migrate()`: |
| 262 | ```go |
| 263 | `ALTER TABLE routes ADD COLUMN args TEXT NOT NULL DEFAULT ''`, |
| 264 | ``` |
| 265 | 3. `routeCols` — append `args, idle_seconds, max_run_seconds` at the very |
| 266 | END of the list (after `created_at`). Position does not matter as long |
| 267 | as it is THE SAME position in all four lists; appending at the end is |
| 268 | the least error-prone choice. |
| 269 | 4. `scanRoute` — add `&t.Args, &t.IdleSeconds, &t.MaxRunSeconds` at the end |
| 270 | of the destination list, in that order. |
| 271 | 5. `Upsert` — add three `?` placeholders at the end of the placeholder row |
| 272 | and `t.Args, t.IdleSeconds, t.MaxRunSeconds` at the end of the argument |
| 273 | list, in that order. |
| 274 | |
| 275 | Then add the setter beside `SetSeed`: |
| 276 | |
| 277 | ```go |
| 278 | // SetArgs repoints a sidecar's argv — the narrow UPDATE the sidecar |
| 279 | // reconcile pass uses when the app's manifest changed. |
| 280 | func (r *Registry) SetArgs(host, args string) error { |
| 281 | _, err := r.sql.Exec(`UPDATE routes SET args = ? WHERE host = ?`, args, host) |
| 282 | return err |
| 283 | } |
| 284 | |
| 285 | // SetWorkerLimits repoints a sidecar's idle window and run ceiling, both in |
| 286 | // seconds, 0 meaning "use the platform default". Same narrow-UPDATE shape |
| 287 | // and same reason as SetArgs: the manifest is the truth, this is the copy |
| 288 | // the sweeper reads on every tick. |
| 289 | func (r *Registry) SetWorkerLimits(host string, idle, maxRun int) error { |
| 290 | _, err := r.sql.Exec( |
| 291 | `UPDATE routes SET idle_seconds = ?, max_run_seconds = ? WHERE host = ?`, |
| 292 | idle, maxRun, host) |
| 293 | return err |
| 294 | } |
| 295 | ``` |
| 296 | |
| 297 | - [ ] **Step 4: Widen `DueInstances`** |
| 298 | |
| 299 | ```go |
| 300 | // DueInstances returns hibernatable instance AND sidecar routes whose |
| 301 | // next_due has arrived (0 means no schedule and never matches). Sidecars |
| 302 | // are included because the due-time sweep is their ONLY timer wake source: |
| 303 | // nothing can dial a sidecar, so no request will ever wake one. |
| 304 | func (r *Registry) DueInstances(now int64) ([]Route, error) { |
| 305 | rows, err := r.sql.Query(`SELECT `+routeCols+` FROM routes |
| 306 | WHERE kind IN ('instance','sidecar') AND next_due > 0 AND next_due <= ? ORDER BY host`, now) |
| 307 | ``` |
| 308 | |
| 309 | - [ ] **Step 5: Run the full registry suite** |
| 310 | |
| 311 | ```bash |
| 312 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 313 | go test ./internal/registry/ -count=1 |
| 314 | ``` |
| 315 | |
| 316 | Expected: PASS, including `TestUpsertRoundTripsAllColumns` — that test is |
| 317 | the canary for a four-list miscount. If it fails with a value in the wrong |
| 318 | field, you shifted a column: recount all four lists. |
| 319 | |
| 320 | - [ ] **Step 6: Full gate, then commit** |
| 321 | |
| 322 | ```bash |
| 323 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && \ |
| 324 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./... && \ |
| 325 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./... -count=1 |
| 326 | gofmt -l internal/registry/ |
| 327 | git add internal/registry/ |
| 328 | git commit -m "registry: sidecar kind, args column, and due-time widening |
| 329 | |
| 330 | DueInstances filtered kind='instance', which would have stored every |
| 331 | sidecar's next_due and acted on none of them — the schedule wake source |
| 332 | would silently not have existed. A sidecar cannot be dialed, so the |
| 333 | due-time sweep is its only timer wake. |
| 334 | |
| 335 | args carries a sidecar's argv to the box, materialized from the app's |
| 336 | manifest each pass exactly as seed is; truth stays in the bucket. |
| 337 | [trailers]" |
| 338 | ``` |
| 339 | |
| 340 | --- |
| 341 | |
| 342 | ### Task 2: Provision — sidecar manifest and record shapes |
| 343 | |
| 344 | **Files:** |
| 345 | - Create: `internal/provision/sidecar.go` |
| 346 | - Test: `internal/provision/sidecar_test.go` |
| 347 | |
| 348 | **Interfaces:** |
| 349 | - Consumes: `release.Store`, and this package's existing `labelRe`, |
| 350 | `ValidRegion`, `ValidPool`, `DefaultDBPath`. |
| 351 | - Produces: |
| 352 | ```go |
| 353 | const SidecarSuffix = ".sidecar.internal" |
| 354 | type SidecarManifest struct{ Schema int; App, Pool, Channel, Placement string |
| 355 | Regions []string; Args []string; Hibernate *bool; IdleSeconds, MaxRunSeconds int } |
| 356 | type SidecarRecord struct{ Schema int; Name, Host, App, Account, Pool, Region, |
| 357 | Channel, Placement, Environment, CreatedAt, CreatedBy string } |
| 358 | func SidecarHost(name, app, sqid string) string |
| 359 | func ValidSidecarName(name string) bool |
| 360 | func ValidSidecarHost(host, app, sqid string) bool |
| 361 | func SidecarManifestKey(acct, app string) string |
| 362 | func SidecarsPrefix(acct, app string) string |
| 363 | func SidecarRecordKey(acct, app, name string) string |
| 364 | func ParseSidecarKey(key string, known map[string]bool) (acct, app, name string, ok bool) |
| 365 | func ValidateSidecarShape(rec SidecarRecord) error |
| 366 | func GetSidecarManifest(ctx, s, acct, app) (*SidecarManifest, error) |
| 367 | func GetSidecarManifestETag(ctx, s, acct, app) (*SidecarManifest, string, error) |
| 368 | func PutSidecarManifest(ctx, s, acct, app, m) error |
| 369 | func PutSidecarManifestIfMatch(ctx, s, acct, app, m, etag string) error |
| 370 | func CreateSidecarRecord(ctx, s, rec) error |
| 371 | func GetSidecarRecord(ctx, s, acct, app, name) (*SidecarRecord, error) |
| 372 | func ListSidecarRecords(ctx, s, acct, app string, max int) ([]SidecarRecord, bool, error) |
| 373 | func DeleteSidecarRecord(ctx, s, acct, app, name string) error |
| 374 | func (m SidecarManifest) IdleOrDefault() time.Duration // 5m |
| 375 | func (m SidecarManifest) MaxRunOrDefault() time.Duration // 1h |
| 376 | func (m SidecarManifest) HibernateOrDefault() bool // true |
| 377 | func (m SidecarManifest) PlacementOrDefault() string // "any" |
| 378 | ``` |
| 379 | |
| 380 | **Durations are integer seconds on the wire** (`idle_seconds`, |
| 381 | `max_run_seconds`), not Go duration strings — the existing records use plain |
| 382 | JSON scalars and a string would invite `"5 minutes"`. |
| 383 | |
| 384 | - [ ] **Step 1: Write the failing tests** |
| 385 | |
| 386 | Create `internal/provision/sidecar_test.go`. Mirror the idiom in |
| 387 | `provision_test.go` and `reconcile_test.go`: a real `release.DirStore` in |
| 388 | `t.TempDir()`, no mocks. |
| 389 | |
| 390 | ```go |
| 391 | func TestSidecarHostAndValidation(t *testing.T) { |
| 392 | got := SidecarHost("bot-acme", "eleven", "bdm") |
| 393 | if want := "bot-acme.eleven.bdm.sidecar.internal"; got != want { |
| 394 | t.Fatalf("SidecarHost = %q, want %q", got, want) |
| 395 | } |
| 396 | if !ValidSidecarHost(got, "eleven", "bdm") { |
| 397 | t.Fatal("its own SidecarHost output must validate") |
| 398 | } |
| 399 | for _, bad := range []string{ |
| 400 | "bot-acme.other.bdm.sidecar.internal", // wrong app |
| 401 | "bot-acme.eleven.bad.sidecar.internal", // wrong sqid |
| 402 | "bot-acme.eleven.bdm.example.com", // wrong suffix |
| 403 | "a.b.bot.eleven.bdm.sidecar.internal", // more than one label |
| 404 | } { |
| 405 | if ValidSidecarHost(bad, "eleven", "bdm") { |
| 406 | t.Fatalf("ValidSidecarHost(%q) = true, want false", bad) |
| 407 | } |
| 408 | } |
| 409 | for _, bad := range []string{"", "-lead", "trail-", "UPPER", "has.dot", "has/slash", "..", strings.Repeat("a", 64)} { |
| 410 | if ValidSidecarName(bad) { |
| 411 | t.Fatalf("ValidSidecarName(%q) = true — the name becomes a bucket key AND a filesystem path segment", bad) |
| 412 | } |
| 413 | } |
| 414 | } |
| 415 | |
| 416 | func TestParseSidecarKey(t *testing.T) { |
| 417 | known := map[string]bool{"bdm": true} |
| 418 | acct, app, name, ok := ParseSidecarKey("apps/bdm/eleven/sidecars/bot-acme.json", known) |
| 419 | if !ok || acct != "bdm" || app != "eleven" || name != "bot-acme" { |
| 420 | t.Fatalf("ParseSidecarKey = %q/%q/%q ok=%v", acct, app, name, ok) |
| 421 | } |
| 422 | for _, bad := range []string{ |
| 423 | "apps/bdm/eleven/sidecars.json", // the manifest, not a record |
| 424 | "apps/bdm/eleven/sidecars/nested/x.json", // nested |
| 425 | "apps/bdm/eleven/sidecars/.json", // empty name |
| 426 | "apps/unknown/eleven/sidecars/bot.json", // account not materialized |
| 427 | "apps/bdm/channels/sidecars/bot.json", // release-plane collision |
| 428 | "apps/bdm/eleven/instances/bot.json", // the other family |
| 429 | } { |
| 430 | if _, _, _, ok := ParseSidecarKey(bad, known); ok { |
| 431 | t.Fatalf("ParseSidecarKey(%q) parsed, want refusal", bad) |
| 432 | } |
| 433 | } |
| 434 | } |
| 435 | |
| 436 | func TestCreateSidecarRecordIsCreateOnly(t *testing.T) { |
| 437 | ctx := context.Background() |
| 438 | s := dirStore(t) // the helper reconcile_test.go already uses |
| 439 | rec := SidecarRecord{ |
| 440 | Schema: 1, Name: "bot-acme", Host: SidecarHost("bot-acme", "eleven", "bdm"), |
| 441 | App: "eleven", Account: "bdm", Channel: "stable", |
| 442 | } |
| 443 | if err := CreateSidecarRecord(ctx, s, rec); err != nil { |
| 444 | t.Fatalf("first create: %v", err) |
| 445 | } |
| 446 | err := CreateSidecarRecord(ctx, s, rec) |
| 447 | if !errors.Is(err, release.ErrCASConflict) { |
| 448 | t.Fatalf("second create err = %v, want ErrCASConflict — a lost race must never silently overwrite", err) |
| 449 | } |
| 450 | } |
| 451 | |
| 452 | func TestGetSidecarRecordRefusesRecordThatLiesAboutItsPath(t *testing.T) { |
| 453 | ctx := context.Background() |
| 454 | s := dirStore(t) |
| 455 | body := []byte(`{"schema":1,"name":"bot-acme","app":"OTHER","account":"bdm"}`) |
| 456 | if _, err := s.Put(ctx, SidecarRecordKey("bdm", "eleven", "bot-acme"), body, ""); err != nil { |
| 457 | t.Fatalf("seed: %v", err) |
| 458 | } |
| 459 | if _, err := GetSidecarRecord(ctx, s, "bdm", "eleven", "bot-acme"); err == nil { |
| 460 | t.Fatal("a record disagreeing with its own key path must be refused, never guessed at") |
| 461 | } |
| 462 | } |
| 463 | |
| 464 | func TestSidecarManifestDefaults(t *testing.T) { |
| 465 | var m SidecarManifest |
| 466 | if m.IdleOrDefault() != 5*time.Minute { |
| 467 | t.Fatalf("idle default = %v, want 5m", m.IdleOrDefault()) |
| 468 | } |
| 469 | if m.MaxRunOrDefault() != time.Hour { |
| 470 | t.Fatalf("max_run default = %v, want 1h", m.MaxRunOrDefault()) |
| 471 | } |
| 472 | if !m.HibernateOrDefault() { |
| 473 | t.Fatal("hibernate must default true") |
| 474 | } |
| 475 | if m.PlacementOrDefault() != "any" { |
| 476 | t.Fatalf("placement default = %q, want any", m.PlacementOrDefault()) |
| 477 | } |
| 478 | no := false |
| 479 | m2 := SidecarManifest{Hibernate: &no} |
| 480 | if m2.HibernateOrDefault() { |
| 481 | t.Fatal("an explicit false must win — tri-state, like instances' Hibernate") |
| 482 | } |
| 483 | } |
| 484 | |
| 485 | func TestSidecarManifestSchemaNewerIsRefused(t *testing.T) { |
| 486 | ctx := context.Background() |
| 487 | s := dirStore(t) |
| 488 | if _, err := s.Put(ctx, SidecarManifestKey("bdm", "eleven"), []byte(`{"schema":2}`), ""); err != nil { |
| 489 | t.Fatalf("seed: %v", err) |
| 490 | } |
| 491 | if _, err := GetSidecarManifest(ctx, s, "bdm", "eleven"); err == nil { |
| 492 | t.Fatal("a schema newer than this binary understands must be refused, not silently truncated") |
| 493 | } |
| 494 | } |
| 495 | |
| 496 | func TestSidecarManifestAbsenceIsNotExist(t *testing.T) { |
| 497 | _, err := GetSidecarManifest(context.Background(), dirStore(t), "bdm", "eleven") |
| 498 | if !errors.Is(err, os.ErrNotExist) { |
| 499 | t.Fatalf("absence err = %v, want os.ErrNotExist — absence means the feature does not exist for that app", err) |
| 500 | } |
| 501 | } |
| 502 | |
| 503 | func TestValidateSidecarShape(t *testing.T) { |
| 504 | ok := SidecarRecord{Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"), |
| 505 | App: "eleven", Account: "bdm", Channel: "stable", Placement: "no-inbound"} |
| 506 | if err := ValidateSidecarShape(ok); err != nil { |
| 507 | t.Fatalf("valid record refused: %v", err) |
| 508 | } |
| 509 | bad := ok |
| 510 | bad.Placement = "sideways" |
| 511 | if err := ValidateSidecarShape(bad); err == nil { |
| 512 | t.Fatal("placement vocabulary must be closed") |
| 513 | } |
| 514 | } |
| 515 | ``` |
| 516 | |
| 517 | - [ ] **Step 2: Run and watch it fail** |
| 518 | |
| 519 | ```bash |
| 520 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 521 | go test ./internal/provision/ -run Sidecar -count=1 |
| 522 | ``` |
| 523 | Expected: compile failure, everything undefined. |
| 524 | |
| 525 | - [ ] **Step 3: Implement `internal/provision/sidecar.go`** |
| 526 | |
| 527 | Write the file mirroring `provision.go` structurally. The store operations |
| 528 | are **the same six functions with `Sidecar` in the name and the sidecar key |
| 529 | helpers substituted** — copy the bodies and their doc comments' reasoning |
| 530 | from these exact sites: |
| 531 | |
| 532 | - `GetSidecarManifest` / `GetSidecarManifestETag` ← `provision.go:508-529` |
| 533 | (schema>1 refusal included). |
| 534 | - `PutSidecarManifest` ← `provision.go:535-557` (CAS upsert, one retry). |
| 535 | - `PutSidecarManifestIfMatch` ← `provision.go:568-575` (no retry). |
| 536 | - `CreateSidecarRecord` ← `provision.go:580-587` (`Put` with `ifMatch ""`). |
| 537 | - `GetSidecarRecord` ← `provision.go:687-709`, **keeping the |
| 538 | identity-vs-path agreement check** (`rec.Account != acct || rec.App != app |
| 539 | || rec.Name != name` → error). |
| 540 | - `ListSidecarRecords` ← `provision.go:726-760` (sorted, capped, `truncated` |
| 541 | reported on key count, malformed bodies skipped). |
| 542 | - `DeleteSidecarRecord` ← `provision.go:641-643`. |
| 543 | |
| 544 | The new logic is only this: |
| 545 | |
| 546 | ```go |
| 547 | // SidecarSuffix is the reserved, never-resolvable suffix every sidecar host |
| 548 | // carries. It is refused at every door where a name could become routable |
| 549 | // (see cli.go's addRail, the route door, and domain attach) — reservation |
| 550 | // is enforced, not documented. |
| 551 | const SidecarSuffix = ".sidecar.internal" |
| 552 | |
| 553 | // SidecarHost builds a sidecar's synthetic host. It is the registry key, the |
| 554 | // bucket key segment and the on-disk path segment, which is why every |
| 555 | // component is validated before it is ever joined. |
| 556 | func SidecarHost(name, app, sqid string) string { |
| 557 | return name + "." + app + "." + sqid + SidecarSuffix |
| 558 | } |
| 559 | |
| 560 | // ValidSidecarName pins one DNS label exactly as ValidHost pins an |
| 561 | // instance's: the name becomes a bucket key segment and the <host>.db / |
| 562 | // cgroup path segments, so the label regex is what keeps a hostile name an |
| 563 | // opaque, safe path component. |
| 564 | func ValidSidecarName(name string) bool { return labelRe.MatchString(name) } |
| 565 | |
| 566 | func ValidSidecarHost(host, app, sqid string) bool { |
| 567 | rest, ok := strings.CutSuffix(host, SidecarSuffix) |
| 568 | if !ok { |
| 569 | return false |
| 570 | } |
| 571 | name, tail, ok := strings.Cut(rest, ".") |
| 572 | if !ok || tail != app+"."+sqid { |
| 573 | return false |
| 574 | } |
| 575 | return ValidSidecarName(name) |
| 576 | } |
| 577 | ``` |
| 578 | |
| 579 | Key helpers, beside the instance ones (`provision.go:155-159`): |
| 580 | |
| 581 | ```go |
| 582 | func SidecarManifestKey(acct, app string) string { |
| 583 | return release.AppPrefix(acct, app) + "sidecars.json" |
| 584 | } |
| 585 | func SidecarsPrefix(acct, app string) string { |
| 586 | return release.AppPrefix(acct, app) + "sidecars/" |
| 587 | } |
| 588 | func SidecarRecordKey(acct, app, name string) string { |
| 589 | return SidecarsPrefix(acct, app) + name + ".json" |
| 590 | } |
| 591 | ``` |
| 592 | |
| 593 | `ParseSidecarKey` is `ParseRecordKey` (`provision.go:171-196`) with |
| 594 | `"instances/"` → `"sidecars/"`: keep the `known` account guard, the |
| 595 | `channels`/`releases` collision refusal, and the exactly-one-non-empty- |
| 596 | segment rule that makes `sidecars.json` and any nested path both refuse. |
| 597 | |
| 598 | Defaults: |
| 599 | |
| 600 | ```go |
| 601 | func (m SidecarManifest) IdleOrDefault() time.Duration { |
| 602 | if m.IdleSeconds > 0 { |
| 603 | return time.Duration(m.IdleSeconds) * time.Second |
| 604 | } |
| 605 | return 5 * time.Minute |
| 606 | } |
| 607 | func (m SidecarManifest) MaxRunOrDefault() time.Duration { |
| 608 | if m.MaxRunSeconds > 0 { |
| 609 | return time.Duration(m.MaxRunSeconds) * time.Second |
| 610 | } |
| 611 | return time.Hour |
| 612 | } |
| 613 | func (m SidecarManifest) HibernateOrDefault() bool { |
| 614 | if m.Hibernate != nil { |
| 615 | return *m.Hibernate |
| 616 | } |
| 617 | return true |
| 618 | } |
| 619 | func (m SidecarManifest) PlacementOrDefault() string { |
| 620 | if m.Placement == "" { |
| 621 | return PlacementAny |
| 622 | } |
| 623 | return m.Placement |
| 624 | } |
| 625 | ``` |
| 626 | |
| 627 | with `const ( PlacementAny = "any"; PlacementNoInbound = "no-inbound" )`. |
| 628 | |
| 629 | `ValidateSidecarShape` checks: schema ≤ 1; `ValidSidecarName(rec.Name)`; |
| 630 | `rec.App`/`rec.Account` non-empty; placement in the closed vocabulary; |
| 631 | `ValidRegion(rec.Region)` when non-empty; `ValidPool(rec.Pool)` when |
| 632 | non-empty. |
| 633 | |
| 634 | - [ ] **Step 4: Run the tests** |
| 635 | |
| 636 | ```bash |
| 637 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 638 | go test ./internal/provision/ -count=1 |
| 639 | ``` |
| 640 | Expected: PASS. |
| 641 | |
| 642 | - [ ] **Step 5: Full gate, then commit** |
| 643 | |
| 644 | ``` |
| 645 | provision: sidecar manifest and record shapes |
| 646 | |
| 647 | Mirrors the instance record family one directory over, reusing its |
| 648 | validation rails verbatim — the label regex, the identity-vs-path |
| 649 | agreement check, create-only writes, and the absence-means-disabled |
| 650 | convention. The name becomes a bucket key AND a filesystem path segment, |
| 651 | which is why it is validated before it is ever joined into a host. |
| 652 | ``` |
| 653 | |
| 654 | --- |
| 655 | |
| 656 | ### Task 3: The edge cannot bind a sidecar, and no box advertises one |
| 657 | |
| 658 | **Files:** |
| 659 | - Modify: `internal/edge/edge.go` (the `Reload` dispatch switch, ~364-445) |
| 660 | - Modify: `internal/host/fleetbind.go` (`publishResidentFleetBindings`, ~301-316) |
| 661 | - Test: `internal/edge/edge_test.go`, `internal/host/fleetbind_test.go` |
| 662 | |
| 663 | **Interfaces:** |
| 664 | - Consumes: Task 1's `Kind == "sidecar"`. |
| 665 | - Produces: nothing other tasks call. |
| 666 | |
| 667 | **Why this is not paranoia.** A sidecar row is unroutable today only by |
| 668 | *omission*: it has no Socket, Addr or Tunnel, so it falls to |
| 669 | `default: continue` and never enters the routing map. But the dispatch |
| 670 | switch has a `case isFleetPlaced && e.tunnels != nil` arm with no Kind |
| 671 | check, and `publishResidentFleetBindings` iterates **every** route with no |
| 672 | Kind filter. Eleven's sidecars are fleet-placed (`region: fleet/eleven`) — |
| 673 | so the row that reaches that arm by shape is exactly the row this feature |
| 674 | creates. Omission is not good enough; refuse explicitly at both ends. |
| 675 | |
| 676 | - [ ] **Step 1: Write the failing tests** |
| 677 | |
| 678 | In `internal/edge/edge_test.go` (follow the file's existing Reload/route |
| 679 | fixtures): |
| 680 | |
| 681 | ```go |
| 682 | func TestSidecarRouteIsNeverServable(t *testing.T) { |
| 683 | // Adversarial on purpose: this row carries a socket, an addr, a tunnel |
| 684 | // AND a fleet placement. A sidecar must be refused on KIND, so that no |
| 685 | // present or future arm of the dispatch switch can ever bind one. |
| 686 | e := newTestEdge(t) |
| 687 | rt := registry.Route{ |
| 688 | Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", |
| 689 | Socket: "/tmp/whatever.sock", Addr: "127.0.0.1:9999", |
| 690 | Tunnel: "eleven/box1", Region: "fleet/eleven", FleetBox: "box1", |
| 691 | Status: "ready", App: "eleven", |
| 692 | } |
| 693 | if err := e.Reload([]registry.Route{rt}); err != nil { |
| 694 | t.Fatalf("Reload: %v", err) |
| 695 | } |
| 696 | if _, ok := e.current()["bot.eleven.bdm.sidecar.internal"]; ok { |
| 697 | t.Fatal("a sidecar row entered the routing map — it is reachable from the internet") |
| 698 | } |
| 699 | if e.Allowed("bot.eleven.bdm.sidecar.internal") == nil { |
| 700 | t.Fatal("a certificate must never be requested for a sidecar host") |
| 701 | } |
| 702 | } |
| 703 | ``` |
| 704 | |
| 705 | (Match `newTestEdge`/`current`/`Allowed` to whatever the file actually |
| 706 | provides; if `Allowed` returns an error use `!= nil` as "refused".) |
| 707 | |
| 708 | In `internal/host/fleetbind_test.go`: |
| 709 | |
| 710 | ```go |
| 711 | func TestSidecarBindingIsNeverPublished(t *testing.T) { |
| 712 | routes := []registry.Route{ |
| 713 | {Host: "app.example.com", Kind: "instance", Socket: "/run/carlos/a.sock"}, |
| 714 | {Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", Region: "fleet/eleven"}, |
| 715 | } |
| 716 | got := residentBindingHosts(routes) // the pure helper you extract in step 3 |
| 717 | if len(got) != 1 || got[0] != "app.example.com" { |
| 718 | t.Fatalf("published %v — a sidecar must never be advertised as served by a box", got) |
| 719 | } |
| 720 | } |
| 721 | ``` |
| 722 | |
| 723 | - [ ] **Step 2: Run and watch both fail** |
| 724 | |
| 725 | ```bash |
| 726 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 727 | go test ./internal/edge/ ./internal/host/ -run 'Sidecar' -count=1 |
| 728 | ``` |
| 729 | Expected: the edge test FAILS (the row is served via the tunnel/fleet arm); |
| 730 | the fleetbind test fails to compile (`residentBindingHosts` undefined). |
| 731 | |
| 732 | - [ ] **Step 3: Refuse in the edge, before the switch** |
| 733 | |
| 734 | In `Reload`'s per-route loop, as the FIRST thing inside the loop body: |
| 735 | |
| 736 | ```go |
| 737 | // A sidecar is a member that serves nothing: it binds no socket, and |
| 738 | // nothing may ever dial it. Refuse it on KIND here, before the dispatch |
| 739 | // switch, rather than relying on it having no socket/addr/tunnel — a |
| 740 | // fleet-placed sidecar (which is Eleven's shape) reaches the fleet arm |
| 741 | // below on shape alone. This refusal is the structural guarantee that |
| 742 | // survives a future arm, column or placement finding the row. |
| 743 | if t.Kind == "sidecar" { |
| 744 | continue |
| 745 | } |
| 746 | ``` |
| 747 | |
| 748 | - [ ] **Step 4: Refuse in the fleet-binding publisher** |
| 749 | |
| 750 | Extract the host selection into a pure, testable helper and filter there: |
| 751 | |
| 752 | ```go |
| 753 | // residentBindingHosts is the set of hosts this box advertises as served. |
| 754 | // Sidecars are excluded: nothing routes to one, so a binding claiming a box |
| 755 | // serves it is a lie that the platform-side reconciler would then act on |
| 756 | // (SetFleetBox -> the edge's fleet arm -> a live proxy). |
| 757 | func residentBindingHosts(routes []registry.Route) []string { |
| 758 | var out []string |
| 759 | for _, rt := range routes { |
| 760 | if rt.Host == "" || rt.Kind == "sidecar" { |
| 761 | continue |
| 762 | } |
| 763 | out = append(out, rt.Host) |
| 764 | } |
| 765 | return out |
| 766 | } |
| 767 | ``` |
| 768 | |
| 769 | and call it from `publishResidentFleetBindings` in place of its current |
| 770 | inline loop. |
| 771 | |
| 772 | - [ ] **Step 5: Run both tests, then the full gate, then commit** |
| 773 | |
| 774 | ``` |
| 775 | edge, host: a sidecar row is refused on kind, at both ends of the fleet chain |
| 776 | |
| 777 | A sidecar was unroutable only by omission — no socket, addr or tunnel, so |
| 778 | it fell to the dispatch switch's default. But the switch's fleet-placed |
| 779 | arm checks no kind, and the binding publisher filtered no kind, and |
| 780 | Eleven's sidecars are fleet-placed. That is a live path from a resident |
| 781 | sidecar row to a working internet-facing proxy, so both ends now refuse |
| 782 | explicitly and the test asserts it against a row that carries every |
| 783 | routable column at once. |
| 784 | ``` |
| 785 | |
| 786 | --- |
| 787 | |
| 788 | ### Task 4: The reserved suffix is enforced at every minting door |
| 789 | |
| 790 | **Files:** |
| 791 | - Modify: `cli.go` (root, `runAdd` ~68-260 and `runRoute`'s rails ~1197-1373) |
| 792 | - Modify: `internal/domains/reconcile.go` (alias minting) |
| 793 | - Modify: `internal/console/domainattach.go` (attach door) |
| 794 | - Test: `cli_test.go`, `internal/domains/reconcile_test.go`, |
| 795 | `internal/console/domainattach_test.go` |
| 796 | |
| 797 | **Interfaces:** |
| 798 | - Consumes: `provision.SidecarSuffix`. |
| 799 | - Produces: nothing. |
| 800 | |
| 801 | `carlos add` today accepts `-kind sidecar` with **no validation at all** |
| 802 | (`cli.go:75,194-199`), and any host string reaches `registry.Upsert`. These |
| 803 | rails are defence in depth behind Task 3's structural refusal. |
| 804 | |
| 805 | - [ ] **Step 1: Write the failing tests** |
| 806 | |
| 807 | ```go |
| 808 | func TestAddRefusesSidecarSuffixAndKind(t *testing.T) { |
| 809 | if err := addRail("bot.eleven.bdm.sidecar.internal", "instance"); err == nil { |
| 810 | t.Fatal("a host under the reserved suffix must be refused") |
| 811 | } |
| 812 | if err := addRail("app.example.com", "sidecar"); err == nil { |
| 813 | t.Fatal("carlos add must not mint a sidecar row — sidecars are provisioned, not added") |
| 814 | } |
| 815 | if err := addRail("app.example.com", "instance"); err != nil { |
| 816 | t.Fatalf("an ordinary add must still work: %v", err) |
| 817 | } |
| 818 | } |
| 819 | ``` |
| 820 | |
| 821 | Add matching cases asserting `domains` never mints an alias whose source or |
| 822 | target carries the suffix, and that `domainattach` refuses to attach a |
| 823 | custom domain to a host under it. |
| 824 | |
| 825 | - [ ] **Step 2: Run and watch it fail** (`addRail` undefined). |
| 826 | |
| 827 | - [ ] **Step 3: Implement the rail** |
| 828 | |
| 829 | In `cli.go`, beside `grantRail`/`backingRail`: |
| 830 | |
| 831 | ```go |
| 832 | // addRail refuses the two things `carlos add` must never mint: a host under |
| 833 | // the reserved sidecar suffix, and a sidecar row itself. Sidecars are |
| 834 | // provisioned through their own record family (carlos sidecars create), so |
| 835 | // an operator minting one by hand would produce a row no reconcile pass |
| 836 | // owns and no record explains. |
| 837 | func addRail(host, kind string) error { |
| 838 | if strings.HasSuffix(host, provision.SidecarSuffix) { |
| 839 | return fmt.Errorf("%s is under the reserved suffix %s — sidecar hosts are minted by `carlos sidecars create`, never by hand", host, provision.SidecarSuffix) |
| 840 | } |
| 841 | if kind == "sidecar" { |
| 842 | return errors.New("carlos add does not mint sidecars — use `carlos sidecars create -app <app> <name>`") |
| 843 | } |
| 844 | return nil |
| 845 | } |
| 846 | ``` |
| 847 | |
| 848 | Call it at the top of `runAdd`, before any registry work. Apply the same |
| 849 | `HasSuffix` refusal in the domains alias minting path and in the console's |
| 850 | domain-attach handler (return the family's ordinary 400 there, not a panic). |
| 851 | |
| 852 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 853 | |
| 854 | --- |
| 855 | |
| 856 | ### Task 5: `StartWorker` — spawning a child that binds nothing |
| 857 | |
| 858 | **Files:** |
| 859 | - Modify: `internal/activator/activator.go` (the `Backend` interface, ~134-167) |
| 860 | - Modify: `internal/activator/backend_exec.go` |
| 861 | - Modify: `internal/activator/backend_unit.go` |
| 862 | - Test: `internal/activator/backend_exec_test.go` (or the existing backend test file) |
| 863 | |
| 864 | **Interfaces:** |
| 865 | - Consumes: Task 1's `Route.Args`. |
| 866 | - Produces: |
| 867 | ```go |
| 868 | // on Backend: |
| 869 | StartWorker(ctx context.Context, rt registry.Route, env map[string]string) (Proc, string, error) |
| 870 | ``` |
| 871 | Returns the `Proc` and the spawn build version, exactly like `Start`. |
| 872 | |
| 873 | **Why a new method rather than a flag on `Start`:** `ExecBackend.Start` ends |
| 874 | with an unconditional `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` |
| 875 | (`backend_exec.go:601`). Handed `Socket ""` it dials `unix://""` every 150ms |
| 876 | for the full 30s and then fails the wake. A worker has no socket by |
| 877 | definition, so it needs a spawn path with no probe — which is very nearly |
| 878 | what `StartSidecar` (`backend_exec.go:746-773`) already is. |
| 879 | |
| 880 | - [ ] **Step 1: Write the failing test** |
| 881 | |
| 882 | ```go |
| 883 | func TestStartWorkerSpawnsWithoutAnyHealthProbe(t *testing.T) { |
| 884 | dir := t.TempDir() |
| 885 | b := &ExecBackend{DataDir: dir, OptDir: dir, Cgroups: nil} |
| 886 | // A binary that runs until killed and never binds anything. |
| 887 | writeFakeLive(t, dir, "bot.x.internal", "#!/bin/sh\nsleep 30\n") |
| 888 | |
| 889 | rt := registry.Route{Host: "bot.x.internal", Kind: "sidecar", |
| 890 | Args: `["sidecar","run"]`, DBPath: filepath.Join(dir, "bot.db")} |
| 891 | |
| 892 | start := time.Now() |
| 893 | p, _, err := b.StartWorker(context.Background(), rt, map[string]string{"CARLOS_SIDECAR_NAME": "bot"}) |
| 894 | if err != nil { |
| 895 | t.Fatalf("StartWorker: %v", err) |
| 896 | } |
| 897 | defer p.Stop(context.Background()) |
| 898 | if d := time.Since(start); d > 5*time.Second { |
| 899 | t.Fatalf("StartWorker took %v — it waited for a health probe that can never answer", d) |
| 900 | } |
| 901 | if p.Exited() { |
| 902 | t.Fatal("worker exited immediately") |
| 903 | } |
| 904 | } |
| 905 | |
| 906 | func TestStartWorkerRefusesEmptyArgs(t *testing.T) { |
| 907 | b := &ExecBackend{DataDir: t.TempDir()} |
| 908 | _, _, err := b.StartWorker(context.Background(), registry.Route{Host: "b.x.internal", Kind: "sidecar"}, nil) |
| 909 | if err == nil { |
| 910 | t.Fatal("a worker with no argv has nothing to run — refuse rather than exec the bare binary") |
| 911 | } |
| 912 | } |
| 913 | |
| 914 | func TestUnitBackendRefusesWorkers(t *testing.T) { |
| 915 | var b UnitBackend |
| 916 | if _, _, err := b.StartWorker(context.Background(), registry.Route{Host: "x"}, nil); err == nil { |
| 917 | t.Fatal("unit-backed tenants have no agent-managed workers, exactly as they have no agent-managed sidecars") |
| 918 | } |
| 919 | } |
| 920 | ``` |
| 921 | |
| 922 | `writeFakeLive` — if the test file has no such helper, write one that |
| 923 | creates the `live/<host>` symlink target as an executable shell script, |
| 924 | matching however the existing exec-backend tests fake a binary. |
| 925 | |
| 926 | - [ ] **Step 2: Run and watch it fail.** |
| 927 | |
| 928 | - [ ] **Step 3: Add the method to the `Backend` interface** |
| 929 | |
| 930 | ```go |
| 931 | // StartWorker spawns a child that binds NOTHING: no socket, no port. |
| 932 | // No health probe is possible or wanted — readiness is the caller's |
| 933 | // settle window, or the child's own call over the agent socket. env is |
| 934 | // merged over the route's own env layers and wins conflicts (it carries |
| 935 | // the per-wake sidecar token, which the tenant must never override). |
| 936 | StartWorker(ctx context.Context, rt registry.Route, env map[string]string) (Proc, string, error) |
| 937 | ``` |
| 938 | |
| 939 | - [ ] **Step 4: Implement `ExecBackend.StartWorker`** |
| 940 | |
| 941 | Model it on `Start` (`backend_exec.go:476-636`) minus the probe and the |
| 942 | socket-mode normalization, and on `StartSidecar` for the no-probe framing: |
| 943 | |
| 944 | ```go |
| 945 | func (b *ExecBackend) StartWorker(ctx context.Context, rt registry.Route, extra map[string]string) (Proc, string, error) { |
| 946 | var argv []string |
| 947 | if err := json.Unmarshal([]byte(rt.Args), &argv); err != nil || len(argv) == 0 { |
| 948 | return nil, "", fmt.Errorf("worker %s: no runnable argv in args %q", rt.Host, rt.Args) |
| 949 | } |
| 950 | if err := b.ensureLiveLink(rt.Host); err != nil { |
| 951 | return nil, "", err |
| 952 | } |
| 953 | spawnVer, _ := b.liveVersion(rt.Host) // same call Start uses; "" = unknown |
| 954 | |
| 955 | fileEnv := b.instanceEnv(rt) // the same merged layers Start composes |
| 956 | for k, v := range extra { |
| 957 | fileEnv = withVar(fileEnv, k, v) |
| 958 | } |
| 959 | |
| 960 | cmd, err := b.Cgroups.Start(rt.Host, func() *exec.Cmd { |
| 961 | c := exec.Command(b.liveLink(rt.Host), argv...) |
| 962 | c.Env = childEnv(fileEnv) |
| 963 | b.teeStd(c, rt) |
| 964 | return c |
| 965 | }) |
| 966 | if err != nil { |
| 967 | return nil, "", err |
| 968 | } |
| 969 | p := &childProc{cmd: cmd, done: make(chan struct{})} |
| 970 | go func() { |
| 971 | p.err = cmd.Wait() |
| 972 | b.Cgroups.CleanupHost(rt.Host) |
| 973 | close(p.done) |
| 974 | }() |
| 975 | // Deliberately NO waitHealthy: a worker binds nothing, so there is |
| 976 | // nothing to probe. Readiness is the activator's settle window. |
| 977 | return p, spawnVer, nil |
| 978 | } |
| 979 | ``` |
| 980 | |
| 981 | Use the real names this file already has for the live-link, version and env |
| 982 | helpers (`ensureLiveLink`, `liveLink`, `instanceEnv`, `childEnv`, |
| 983 | `withVar`, `teeStd`); do not invent new ones. If `b.Cgroups` may be nil in |
| 984 | tests, follow whatever nil-guard `Start` uses. |
| 985 | |
| 986 | `UnitBackend.StartWorker` and `SplitBackend.StartWorker` mirror |
| 987 | `StartSidecar` (`backend_unit.go:186-188`, `:304-306`): refuse, and |
| 988 | delegate by `b.pick(rt)` respectively. |
| 989 | |
| 990 | - [ ] **Step 5: Run the tests, full gate, commit.** |
| 991 | |
| 992 | --- |
| 993 | |
| 994 | ### Task 6: The standalone sidecar lifecycle inside the activator |
| 995 | |
| 996 | **Files:** |
| 997 | - Modify: `internal/activator/activator.go` |
| 998 | - Test: `internal/activator/activator_test.go` |
| 999 | |
| 1000 | **Interfaces:** |
| 1001 | - Consumes: Task 5's `StartWorker`. |
| 1002 | - Produces: |
| 1003 | ```go |
| 1004 | func (a *Service) WakeWorker(rt registry.Route) error // idempotent; nil if already awake |
| 1005 | func (a *Service) NoteWorkerContact(host string) // stamps lastActive |
| 1006 | func (a *Service) SetWorkerNextDue(host string, at int64) error |
| 1007 | func (a *Service) SleepWorker(host string) // /v1/idle |
| 1008 | func (a *Service) WorkerToken(host string) string // "" unless awake |
| 1009 | func (a *Service) NoteWorkerReady(host, version string) |
| 1010 | ``` |
| 1011 | `WakeWorker` takes the route ALONE: the idle window and run ceiling ride |
| 1012 | on the row (Task 1), so the sweeper never needs a bucket read to decide a |
| 1013 | worker has outrun its ceiling. |
| 1014 | |
| 1015 | **Design constraint:** sidecars stay entirely off the request path. |
| 1016 | `beginRequest`, `BeginSidecarRequest` and `ServeHTTP` are not modified — |
| 1017 | `ServeHTTP` already gates on `rt.Kind == "instance" && rt.Hibernate`. |
| 1018 | |
| 1019 | - [ ] **Step 1: Write the failing tests** |
| 1020 | |
| 1021 | ```go |
| 1022 | func TestWorkerWakeUsesStartWorkerAndNoProbe(t *testing.T) { |
| 1023 | a, b, _, _ := testActivator(t, 10*time.Second) |
| 1024 | rt := testWorkerRoute("bot.x.internal") |
| 1025 | if err := a.WakeWorker(rt); err != nil { |
| 1026 | t.Fatalf("WakeWorker: %v", err) |
| 1027 | } |
| 1028 | if got := b.events(); !contains(got, "startWorker:bot.x.internal") { |
| 1029 | t.Fatalf("events %v — a worker must spawn through StartWorker, never Start", got) |
| 1030 | } |
| 1031 | if b.startCount() != 0 { |
| 1032 | t.Fatal("Start (the socket path, with its health probe) must never be called for a worker") |
| 1033 | } |
| 1034 | } |
| 1035 | |
| 1036 | func TestWorkerSleepsWhenIdleAndRecordsNextDue(t *testing.T) { |
| 1037 | a, b, clk, reg := testActivator(t, 10*time.Second) |
| 1038 | rt := testWorkerRoute("bot.x.internal") |
| 1039 | if err := a.WakeWorker(rt); err != nil { |
| 1040 | t.Fatalf("WakeWorker: %v", err) |
| 1041 | } |
| 1042 | if err := a.SetWorkerNextDue("bot.x.internal", 4242); err != nil { |
| 1043 | t.Fatalf("SetWorkerNextDue: %v", err) |
| 1044 | } |
| 1045 | |
| 1046 | clk.advance(11 * time.Second) |
| 1047 | a.sweepOnce() |
| 1048 | waitFor(t, "worker hibernation", func() bool { return a.entryCount() == 0 }) |
| 1049 | |
| 1050 | got, err := reg.Get("bot.x.internal") |
| 1051 | if err != nil { |
| 1052 | t.Fatalf("get: %v", err) |
| 1053 | } |
| 1054 | if got.NextDue != 4242 { |
| 1055 | t.Fatalf("next_due = %d, want 4242 — a worker's next-due arrives over the agent socket while it is ALIVE, not from a probe of a process we are about to kill", got.NextDue) |
| 1056 | } |
| 1057 | if b.dueProbes() != 0 { |
| 1058 | t.Fatal("a worker must never be probed for next-due: it has no socket to answer on") |
| 1059 | } |
| 1060 | if !b.lastRep().isClosed() { |
| 1061 | t.Fatal("the lease must be released and the DB parked on a worker sleep, exactly as for an instance") |
| 1062 | } |
| 1063 | } |
| 1064 | |
| 1065 | func TestWorkerContactDefersSleep(t *testing.T) { |
| 1066 | a, _, clk, _ := testActivator(t, 10*time.Second) |
| 1067 | if err := a.WakeWorker(testWorkerRoute("bot.x.internal")); err != nil { |
| 1068 | t.Fatalf("WakeWorker: %v", err) |
| 1069 | } |
| 1070 | clk.advance(9 * time.Second) |
| 1071 | a.NoteWorkerContact("bot.x.internal") |
| 1072 | clk.advance(9 * time.Second) |
| 1073 | a.sweepOnce() |
| 1074 | if a.entryCount() != 1 { |
| 1075 | t.Fatal("agent-socket contact must defer the idle clock — it is a worker's only sign of life") |
| 1076 | } |
| 1077 | } |
| 1078 | |
| 1079 | func TestWorkerMaxRunStopsALoopingWorker(t *testing.T) { |
| 1080 | a, _, clk, _ := testActivator(t, time.Hour) // idle window never fires |
| 1081 | rt := testWorkerRoute("bot.x.internal") |
| 1082 | rt.MaxRunSeconds = 60 |
| 1083 | if err := a.WakeWorker(rt); err != nil { |
| 1084 | t.Fatalf("WakeWorker: %v", err) |
| 1085 | } |
| 1086 | for i := 0; i < 10; i++ { // a worker that keeps talking but never idles |
| 1087 | clk.advance(10 * time.Second) |
| 1088 | a.NoteWorkerContact("bot.x.internal") |
| 1089 | } |
| 1090 | a.sweepOnce() |
| 1091 | waitFor(t, "max_run stop", func() bool { return a.entryCount() == 0 }) |
| 1092 | } |
| 1093 | |
| 1094 | func TestSleepWorkerIsImmediate(t *testing.T) { |
| 1095 | a, _, _, _ := testActivator(t, time.Hour) |
| 1096 | if err := a.WakeWorker(testWorkerRoute("bot.x.internal")); err != nil { |
| 1097 | t.Fatalf("WakeWorker: %v", err) |
| 1098 | } |
| 1099 | a.SleepWorker("bot.x.internal") |
| 1100 | waitFor(t, "idle-requested stop", func() bool { return a.entryCount() == 0 }) |
| 1101 | } |
| 1102 | |
| 1103 | func TestWakeWorkerIsIdempotent(t *testing.T) { |
| 1104 | a, b, _, _ := testActivator(t, time.Hour) |
| 1105 | rt := testWorkerRoute("bot.x.internal") |
| 1106 | for i := 0; i < 3; i++ { |
| 1107 | if err := a.WakeWorker(rt); err != nil { |
| 1108 | t.Fatalf("WakeWorker #%d: %v", i, err) |
| 1109 | } |
| 1110 | } |
| 1111 | if n := b.workerStarts(); n != 1 { |
| 1112 | t.Fatalf("workerStarts = %d, want 1 — a wake for an already-awake worker is a no-op, not a second child", n) |
| 1113 | } |
| 1114 | } |
| 1115 | ``` |
| 1116 | |
| 1117 | Extend `fakeBackend` with `StartWorker` (recording `startWorker:<host>` in |
| 1118 | its `order` log and incrementing a `workerStarts` counter), and add |
| 1119 | `testWorkerRoute(host)` returning |
| 1120 | `registry.Route{Host: host, Kind: "sidecar", Args: `["sidecar","run"]`, DBPath: ..., Hibernate: true, Status: "ready"}`. |
| 1121 | Add the `dueProbes()` accessor if the fake does not already count `NextDue` |
| 1122 | calls. |
| 1123 | |
| 1124 | - [ ] **Step 2: Run and watch them fail.** |
| 1125 | |
| 1126 | - [ ] **Step 3: Implement** |
| 1127 | |
| 1128 | Add to `actEntry`: |
| 1129 | |
| 1130 | ```go |
| 1131 | worker bool // this entry is a sidecar: no socket, no requests |
| 1132 | workerIdle time.Duration // 0 = fall back to cfg.Idle |
| 1133 | workerMaxRun time.Duration // 0 = no ceiling |
| 1134 | workerToken string // the per-wake agent-socket token |
| 1135 | spawnedAt time.Time |
| 1136 | ``` |
| 1137 | |
| 1138 | populated from the route's `IdleSeconds`/`MaxRunSeconds` at wake time. |
| 1139 | |
| 1140 | ```go |
| 1141 | // WakeWorker brings a sidecar up, or does nothing if it is already awake. |
| 1142 | // It is the request-less twin of beginRequest: a sidecar has no socket, so |
| 1143 | // no request can ever wake one — every wake arrives from the due-time |
| 1144 | // sweep, an operator command, or the agent's boot pass. |
| 1145 | func (a *Service) WakeWorker(rt registry.Route) error |
| 1146 | ``` |
| 1147 | |
| 1148 | Implementation: |
| 1149 | |
| 1150 | - Under `a.mu`: if an entry exists and is `actAwake` or `actWaking`, return |
| 1151 | nil (idempotent). If `actStopping`, wait on `e.stopped` then retry. |
| 1152 | - Otherwise create the entry with `worker: true`, `workerMaxRun: maxRun`, |
| 1153 | mint `workerToken` (32 bytes from `crypto/rand`, hex), launch |
| 1154 | `go a.wake(e)` and **do not unlock before falling into the wait** — the |
| 1155 | singleflight discipline PR #199 established; then wait on `e.ready`. |
| 1156 | - In `doWake`, branch step 4: |
| 1157 | ```go |
| 1158 | var ( |
| 1159 | proc Proc |
| 1160 | spawnVer string |
| 1161 | ) |
| 1162 | if e.worker { |
| 1163 | proc, spawnVer, err = a.backend.StartWorker(ctx, rt, a.workerEnv(e)) |
| 1164 | } else { |
| 1165 | proc, spawnVer, err = a.backend.Start(ctx, rt) |
| 1166 | } |
| 1167 | ``` |
| 1168 | and skip step 6 (sidecar co-spawn) entirely for a worker — a worker is |
| 1169 | nobody's parent. |
| 1170 | - In `stop`, skip the next-due probe when `e.worker`: its next-due arrived |
| 1171 | over the agent socket while it was alive. Everything after (stop the |
| 1172 | proc, final checkpoint, lease release) is shared unchanged. |
| 1173 | - In `sweepOnce`, beside the existing idle check, add: |
| 1174 | ```go |
| 1175 | if e.worker && e.workerMaxRun > 0 && now.Sub(e.spawnedAt) >= e.workerMaxRun { |
| 1176 | // A worker that never reports idle still has a ceiling: a wedged |
| 1177 | // reconnect loop must not hold a lease and a box forever. |
| 1178 | a.beginStopLocked(e) |
| 1179 | continue |
| 1180 | } |
| 1181 | ``` |
| 1182 | and make the existing idle predicate use the entry's own window when set |
| 1183 | (`e.workerIdle`), falling back to `a.cfg.Idle`. Otherwise it is unchanged: |
| 1184 | `inflight` is always 0 for a worker, and `lastActive` is stamped by |
| 1185 | agent-socket contact instead of by a released request. |
| 1186 | |
| 1187 | The four small public methods: |
| 1188 | |
| 1189 | ```go |
| 1190 | func (a *Service) NoteWorkerContact(host string) { |
| 1191 | a.mu.Lock() |
| 1192 | defer a.mu.Unlock() |
| 1193 | if e := a.entries[host]; e != nil && e.worker { |
| 1194 | e.lastActive = a.now() |
| 1195 | } |
| 1196 | } |
| 1197 | |
| 1198 | func (a *Service) SetWorkerNextDue(host string, at int64) error { |
| 1199 | return a.due.SetNextDue(host, at) |
| 1200 | } |
| 1201 | |
| 1202 | func (a *Service) SleepWorker(host string) { |
| 1203 | a.mu.Lock() |
| 1204 | defer a.mu.Unlock() |
| 1205 | if e := a.entries[host]; e != nil && e.worker && e.state == actAwake { |
| 1206 | a.beginStopLocked(e) |
| 1207 | } |
| 1208 | } |
| 1209 | |
| 1210 | // WorkerToken reports the live per-wake token for host, or "" — the agent |
| 1211 | // socket's authorization check. A token from a previous wake never matches. |
| 1212 | func (a *Service) WorkerToken(host string) string { |
| 1213 | a.mu.Lock() |
| 1214 | defer a.mu.Unlock() |
| 1215 | if e := a.entries[host]; e != nil && e.worker && e.state == actAwake { |
| 1216 | return e.workerToken |
| 1217 | } |
| 1218 | return "" |
| 1219 | } |
| 1220 | ``` |
| 1221 | |
| 1222 | `a.workerEnv(e)` returns |
| 1223 | `{"CARLOS_AGENT_SOCKET": a.cfg.AgentSocket, "CARLOS_SIDECAR_TOKEN": e.workerToken, "CARLOS_SIDECAR_HOST": e.host, "CARLOS_SIDECAR_NAME": <first label of e.host>}`. |
| 1224 | Add `AgentSocket string` to `activator.Config`, defaulting to |
| 1225 | `/run/carlos/agent.sock`. |
| 1226 | |
| 1227 | - [ ] **Step 4: Run the activator suite in full** |
| 1228 | |
| 1229 | ```bash |
| 1230 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1231 | go test ./internal/activator/ -count=1 |
| 1232 | ``` |
| 1233 | Expected: PASS, **including every pre-existing instance test** — especially |
| 1234 | `TestActivatorFailedWakeIsSingleflight`, `TestActivatorCoStopOrdering` and |
| 1235 | `TestActivatorIdleSleep`. If any instance test changed behaviour, you put a |
| 1236 | worker branch on a shared path; move it. |
| 1237 | |
| 1238 | - [ ] **Step 5: Full gate, commit.** |
| 1239 | |
| 1240 | --- |
| 1241 | |
| 1242 | ### Task 7: The agent socket |
| 1243 | |
| 1244 | **Files:** |
| 1245 | - Create: `internal/agentsock/agentsock.go`, `internal/agentsock/agentsock_test.go` |
| 1246 | |
| 1247 | **Interfaces:** |
| 1248 | - Consumes: an interface it defines and Task 6 satisfies: |
| 1249 | ```go |
| 1250 | type Workers interface { |
| 1251 | WorkerToken(host string) string |
| 1252 | NoteWorkerContact(host string) |
| 1253 | SetWorkerNextDue(host string, at int64) error |
| 1254 | SleepWorker(host string) |
| 1255 | NoteWorkerReady(host, version string) |
| 1256 | } |
| 1257 | ``` |
| 1258 | - Produces: |
| 1259 | ```go |
| 1260 | func New(w Workers) *Server |
| 1261 | func (s *Server) Serve(ctx context.Context, path string) error // listens, cleans up |
| 1262 | func (s *Server) Handler() http.Handler // for tests |
| 1263 | ``` |
| 1264 | |
| 1265 | **The direction is the whole point:** the agent listens, children dial. The |
| 1266 | platform never dials a sidecar. |
| 1267 | |
| 1268 | Wire protocol — all `POST`, JSON in, `204` out, `Authorization: Bearer <token>` |
| 1269 | and `Carlos-Sidecar-Host: <host>`: |
| 1270 | |
| 1271 | | Path | Body | Effect | |
| 1272 | |---|---|---| |
| 1273 | | `/v1/ready` | `{"version":"<build>"}` | ends the settle window, records the running version | |
| 1274 | | `/v1/next-due` | `{"at": <unix seconds>}` | `SetWorkerNextDue` | |
| 1275 | | `/v1/idle` | — | `SleepWorker` | |
| 1276 | | `/v1/heartbeat` | — | `NoteWorkerContact` | |
| 1277 | |
| 1278 | - [ ] **Step 1: Write the failing tests** |
| 1279 | |
| 1280 | ```go |
| 1281 | type fakeWorkers struct { |
| 1282 | mu sync.Mutex |
| 1283 | tokens map[string]string |
| 1284 | contacts []string |
| 1285 | due map[string]int64 |
| 1286 | slept []string |
| 1287 | ready map[string]string |
| 1288 | } |
| 1289 | |
| 1290 | func (f *fakeWorkers) WorkerToken(host string) string { /* guarded read */ } |
| 1291 | // ... the rest recording into the slices/maps |
| 1292 | |
| 1293 | func TestAgentSockRejectsWrongToken(t *testing.T) { |
| 1294 | f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}} |
| 1295 | h := New(f).Handler() |
| 1296 | |
| 1297 | for _, tc := range []struct{ name, host, tok string }{ |
| 1298 | {"wrong token", "bot.x.internal", "bad"}, |
| 1299 | {"no token", "bot.x.internal", ""}, |
| 1300 | {"unknown host", "nope.x.internal", "good"}, |
| 1301 | {"token for another host", "other.x.internal", "good"}, |
| 1302 | } { |
| 1303 | rec := post(h, "/v1/heartbeat", tc.host, tc.tok, "") |
| 1304 | if rec.Code != http.StatusUnauthorized { |
| 1305 | t.Fatalf("%s: code = %d, want 401", tc.name, rec.Code) |
| 1306 | } |
| 1307 | } |
| 1308 | if len(f.contacts) != 0 { |
| 1309 | t.Fatal("a refused call must have no effect at all") |
| 1310 | } |
| 1311 | } |
| 1312 | |
| 1313 | func TestAgentSockTokenIsScopedToOneHost(t *testing.T) { |
| 1314 | // The security property: possession of A's token says nothing about B. |
| 1315 | f := &fakeWorkers{tokens: map[string]string{"a.x.internal": "tok-a", "b.x.internal": "tok-b"}} |
| 1316 | h := New(f).Handler() |
| 1317 | if rec := post(h, "/v1/idle", "b.x.internal", "tok-a", ""); rec.Code != http.StatusUnauthorized { |
| 1318 | t.Fatalf("A's token spoke about B: code = %d", rec.Code) |
| 1319 | } |
| 1320 | if len(f.slept) != 0 { |
| 1321 | t.Fatal("B must not have been slept by A's token") |
| 1322 | } |
| 1323 | } |
| 1324 | |
| 1325 | func TestAgentSockVerbs(t *testing.T) { |
| 1326 | f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}} |
| 1327 | h := New(f).Handler() |
| 1328 | |
| 1329 | if rec := post(h, "/v1/ready", "bot.x.internal", "good", `{"version":"abc123"}`); rec.Code != http.StatusNoContent { |
| 1330 | t.Fatalf("ready: %d", rec.Code) |
| 1331 | } |
| 1332 | if f.ready["bot.x.internal"] != "abc123" { |
| 1333 | t.Fatal("ready must record the running version — it is a sidecar's only version signal") |
| 1334 | } |
| 1335 | if rec := post(h, "/v1/next-due", "bot.x.internal", "good", `{"at":1756}`); rec.Code != http.StatusNoContent { |
| 1336 | t.Fatalf("next-due: %d", rec.Code) |
| 1337 | } |
| 1338 | if f.due["bot.x.internal"] != 1756 { |
| 1339 | t.Fatalf("due = %d", f.due["bot.x.internal"]) |
| 1340 | } |
| 1341 | if rec := post(h, "/v1/idle", "bot.x.internal", "good", ""); rec.Code != http.StatusNoContent { |
| 1342 | t.Fatalf("idle: %d", rec.Code) |
| 1343 | } |
| 1344 | if len(f.slept) != 1 { |
| 1345 | t.Fatal("idle must ask for a stop") |
| 1346 | } |
| 1347 | } |
| 1348 | |
| 1349 | func TestAgentSockRefusesGarbage(t *testing.T) { |
| 1350 | f := &fakeWorkers{tokens: map[string]string{"bot.x.internal": "good"}} |
| 1351 | h := New(f).Handler() |
| 1352 | for _, body := range []string{"not json", `{"at":"soon"}`, strings.Repeat("x", 1<<20)} { |
| 1353 | rec := post(h, "/v1/next-due", "bot.x.internal", "good", body) |
| 1354 | if rec.Code != http.StatusBadRequest { |
| 1355 | t.Fatalf("body %.20q: code = %d, want 400", body, rec.Code) |
| 1356 | } |
| 1357 | } |
| 1358 | if rec := post(h, "/v1/nope", "bot.x.internal", "good", ""); rec.Code != http.StatusNotFound { |
| 1359 | t.Fatalf("unknown verb: %d", rec.Code) |
| 1360 | } |
| 1361 | req := httptest.NewRequest("GET", "/v1/idle", nil) |
| 1362 | // GET on a POST-only verb |
| 1363 | rec := httptest.NewRecorder() |
| 1364 | h.ServeHTTP(rec, req) |
| 1365 | if rec.Code != http.StatusMethodNotAllowed && rec.Code != http.StatusUnauthorized { |
| 1366 | t.Fatalf("GET: %d", rec.Code) |
| 1367 | } |
| 1368 | } |
| 1369 | |
| 1370 | func TestAgentSockServeCleansUpItsSocket(t *testing.T) { |
| 1371 | dir := t.TempDir() |
| 1372 | path := filepath.Join(dir, "agent.sock") |
| 1373 | ctx, cancel := context.WithCancel(context.Background()) |
| 1374 | done := make(chan error, 1) |
| 1375 | go func() { done <- New(&fakeWorkers{}).Serve(ctx, path) }() |
| 1376 | waitFor(t, "socket to appear", func() bool { _, err := os.Stat(path); return err == nil }) |
| 1377 | cancel() |
| 1378 | <-done |
| 1379 | if _, err := os.Stat(path); !os.IsNotExist(err) { |
| 1380 | t.Fatal("a stale socket file must not survive shutdown — the next start would fail to bind") |
| 1381 | } |
| 1382 | } |
| 1383 | ``` |
| 1384 | |
| 1385 | Write `post(h http.Handler, path, host, tok, body string) *httptest.ResponseRecorder` |
| 1386 | as a small local helper. |
| 1387 | |
| 1388 | - [ ] **Step 2: Run and watch it fail.** |
| 1389 | |
| 1390 | - [ ] **Step 3: Implement** |
| 1391 | |
| 1392 | Package doc first, stating the direction: |
| 1393 | |
| 1394 | ```go |
| 1395 | // Package agentsock is the agent's own listening socket — the ONE channel |
| 1396 | // between a sidecar and the platform, and it points inward: children dial |
| 1397 | // the agent, the agent never dials a child. That is what lets a sidecar |
| 1398 | // bind nothing, and what lets the identical artifact run off-platform |
| 1399 | // behind NAT with no protocol difference. |
| 1400 | // |
| 1401 | // Every exec child runs as the agent's user, so the socket's file mode is |
| 1402 | // not an identity boundary. Identity is the per-wake token the activator |
| 1403 | // mints and passes in the child's environment (CARLOS_SIDECAR_TOKEN), |
| 1404 | // scoped to one host for one wake: a token for A cannot speak about B, and |
| 1405 | // a token from a previous wake matches nothing. |
| 1406 | package agentsock |
| 1407 | ``` |
| 1408 | |
| 1409 | Implementation notes that the tests pin: |
| 1410 | |
| 1411 | - Compare tokens with `crypto/subtle.ConstantTimeCompare`, and refuse when |
| 1412 | `WorkerToken(host)` returns `""` (asleep or unknown) — one 401 for every |
| 1413 | failure, no distinguishing message. |
| 1414 | - Bound the body: `io.LimitReader(r.Body, 4096)`, the `NextDue` decode |
| 1415 | precedent in `backend_exec.go:451`. |
| 1416 | - `Serve` should `os.Remove(path)` before binding (a stale socket from a |
| 1417 | hard kill), `net.Listen("unix", path)`, `os.Chmod(path, 0o660)`, serve |
| 1418 | with `http.Server`, and on `ctx.Done()` shut down and remove the file. |
| 1419 | - No timeouts longer than a few seconds; these are local calls. |
| 1420 | |
| 1421 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1422 | |
| 1423 | --- |
| 1424 | |
| 1425 | ### Task 8: Wire the socket, the boot pass and the ceilings into the agent |
| 1426 | |
| 1427 | **Files:** |
| 1428 | - Modify: `internal/host/agent.go` |
| 1429 | - Modify: `internal/activator/activator.go` (implement `NoteWorkerReady`) |
| 1430 | - Test: `internal/host/agent_test.go` |
| 1431 | |
| 1432 | **Interfaces:** |
| 1433 | - Consumes: Tasks 6 and 7. |
| 1434 | - Produces: a running agent socket on every box that runs the activator, and |
| 1435 | a boot pass that starts non-hibernating sidecars. |
| 1436 | |
| 1437 | - [ ] **Step 1: Write the failing test** |
| 1438 | |
| 1439 | ```go |
| 1440 | func TestWorkerBootPassStartsOnlyNonHibernatingSidecars(t *testing.T) { |
| 1441 | routes := []registry.Route{ |
| 1442 | {Host: "a.x.sidecar.internal", Kind: "sidecar", Hibernate: false, Status: "ready"}, |
| 1443 | {Host: "b.x.sidecar.internal", Kind: "sidecar", Hibernate: true, Status: "ready"}, |
| 1444 | {Host: "c.example.com", Kind: "instance", Hibernate: false, Status: "ready"}, |
| 1445 | {Host: "d.x.sidecar.internal", Kind: "sidecar", Hibernate: false, Status: "draining"}, |
| 1446 | } |
| 1447 | got := workersToStart(routes) |
| 1448 | if len(got) != 1 || got[0].Host != "a.x.sidecar.internal" { |
| 1449 | t.Fatalf("workersToStart = %v — only a ready, non-hibernating sidecar starts at boot; a hibernating one waits for its timer or a command", got) |
| 1450 | } |
| 1451 | } |
| 1452 | ``` |
| 1453 | |
| 1454 | - [ ] **Step 2: Run and watch it fail.** |
| 1455 | |
| 1456 | - [ ] **Step 3: Implement** |
| 1457 | |
| 1458 | ```go |
| 1459 | // workersToStart is the boot pass's selection: a sidecar that does not |
| 1460 | // hibernate must be running whenever the agent is. A hibernating sidecar is |
| 1461 | // deliberately NOT started here — it waits for its due time, an operator |
| 1462 | // command, or (later) a delivery signal, which is the whole point of the |
| 1463 | // class. |
| 1464 | func workersToStart(routes []registry.Route) []registry.Route { |
| 1465 | var out []registry.Route |
| 1466 | for _, rt := range routes { |
| 1467 | if rt.Kind == "sidecar" && !rt.Hibernate && (rt.Status == "" || rt.Status == "ready") { |
| 1468 | out = append(out, rt) |
| 1469 | } |
| 1470 | } |
| 1471 | return out |
| 1472 | } |
| 1473 | ``` |
| 1474 | |
| 1475 | Then, in the agent's startup block beside the other loops: |
| 1476 | |
| 1477 | - construct `agentsock.New(svc)` and `go srv.Serve(ctx, "/run/carlos/agent.sock")` |
| 1478 | **only when the activator is enabled** (the same condition that |
| 1479 | constructs the activator — with hibernation off there are no workers). |
| 1480 | - run the boot pass once after the first registry read, calling |
| 1481 | `svc.WakeWorker(rt)` for each result. |
| 1482 | - The sweeper already handles the due-time wakes; make sure its |
| 1483 | `DueInstances` results route to `WakeWorker` for `Kind == "sidecar"` |
| 1484 | rather than the instance wake path. |
| 1485 | |
| 1486 | Add `NoteWorkerReady(host, version string)` to the activator: stamp |
| 1487 | `lastActive`, mark the settle window satisfied, and record the version via |
| 1488 | the registry's existing version setter so §9.2's honesty rule holds (a |
| 1489 | sidecar that never reports shows unknown, never "current"). |
| 1490 | |
| 1491 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1492 | |
| 1493 | --- |
| 1494 | |
| 1495 | ### Task 9: The sidecar reconciler — and the removal-authority collision |
| 1496 | |
| 1497 | **Files:** |
| 1498 | - Create: `internal/provision/sidecar_reconcile.go`, `sidecar_reconcile_test.go` |
| 1499 | - Modify: `internal/provision/reconcile.go` (removal phase, ~651-673) |
| 1500 | - Modify: `internal/host/agent.go` (call it through `reconcileSerialized`) |
| 1501 | |
| 1502 | **Interfaces:** |
| 1503 | - Consumes: Task 2's shapes, Task 1's `Args`. |
| 1504 | - Produces: `func ReconcileSidecars(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig) error` |
| 1505 | |
| 1506 | **The collision, stated plainly:** the instance pass's removal phase is |
| 1507 | scoped by `Provisioned`, region, pool and absence-from-records, with **no |
| 1508 | Kind check**. Sidecar rows are `Provisioned` and can never appear in the |
| 1509 | instance record set, so the existing pass would delete every sidecar on |
| 1510 | every tick — minting and destroying forever, with no error anywhere. Each |
| 1511 | pass's removal authority must be scoped to its own Kind. |
| 1512 | |
| 1513 | - [ ] **Step 1: Write the failing tests** |
| 1514 | |
| 1515 | ```go |
| 1516 | func TestInstanceReconcileNeverRemovesSidecarRows(t *testing.T) { |
| 1517 | f := setup(t) // reconcile_test.go's existing fixture |
| 1518 | must(t, f.reg.Upsert(registry.Route{ |
| 1519 | Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", |
| 1520 | App: "eleven", Account: "bdm", Provisioned: true, Status: "ready", |
| 1521 | })) |
| 1522 | // No instance records at all: the pass's removal phase runs in full. |
| 1523 | if err := Reconcile(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil { |
| 1524 | t.Fatalf("Reconcile: %v", err) |
| 1525 | } |
| 1526 | if _, err := f.reg.Get("bot.eleven.bdm.sidecar.internal"); err != nil { |
| 1527 | t.Fatal("the instance pass deleted a sidecar row — each pass's removal authority is scoped to its OWN kind, or the two passes destroy each other's work forever") |
| 1528 | } |
| 1529 | } |
| 1530 | |
| 1531 | func TestSidecarReconcileMints(t *testing.T) { |
| 1532 | f := setup(t) |
| 1533 | seedAccount(t, f.store, "bdm") |
| 1534 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{ |
| 1535 | Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, Channel: "stable", |
| 1536 | }) |
| 1537 | writeSidecarRecord(t, f.store, SidecarRecord{ |
| 1538 | Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"), |
| 1539 | App: "eleven", Account: "bdm", Channel: "stable", |
| 1540 | }) |
| 1541 | |
| 1542 | if err := ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil { |
| 1543 | t.Fatalf("ReconcileSidecars: %v", err) |
| 1544 | } |
| 1545 | rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm")) |
| 1546 | if rt.Kind != "sidecar" { |
| 1547 | t.Fatalf("Kind = %q", rt.Kind) |
| 1548 | } |
| 1549 | if rt.Socket != "" || rt.Addr != "" { |
| 1550 | t.Fatalf("a sidecar row must carry NO socket and NO addr, got socket=%q addr=%q", rt.Socket, rt.Addr) |
| 1551 | } |
| 1552 | if rt.DBPath == "" || !rt.Provisioned || rt.Args != `["sidecar","run"]` { |
| 1553 | t.Fatalf("route = %+v", rt) |
| 1554 | } |
| 1555 | } |
| 1556 | |
| 1557 | func TestSidecarReconcileRemovesOnlyItsOwnKind(t *testing.T) { |
| 1558 | f := setup(t) |
| 1559 | seedAccount(t, f.store, "bdm") |
| 1560 | must(t, f.reg.Upsert(registry.Route{Host: "gone.eleven.bdm.sidecar.internal", |
| 1561 | Kind: "sidecar", App: "eleven", Account: "bdm", Provisioned: true})) |
| 1562 | must(t, f.reg.Upsert(registry.Route{Host: "app.example.com", |
| 1563 | Kind: "instance", App: "eleven", Account: "bdm", Provisioned: true})) |
| 1564 | |
| 1565 | if err := ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}); err != nil { |
| 1566 | t.Fatalf("ReconcileSidecars: %v", err) |
| 1567 | } |
| 1568 | if _, err := f.reg.Get("gone.eleven.bdm.sidecar.internal"); err == nil { |
| 1569 | t.Fatal("a provisioned sidecar with no record must be removed") |
| 1570 | } |
| 1571 | if _, err := f.reg.Get("app.example.com"); err != nil { |
| 1572 | t.Fatal("the sidecar pass must never touch an instance row") |
| 1573 | } |
| 1574 | } |
| 1575 | |
| 1576 | func TestSidecarReconcileSkipsRemovalWhenAReadFailed(t *testing.T) { |
| 1577 | // The fail-safe: a transient read failure must never mass-retire a |
| 1578 | // tenant's fleet. Minting still proceeds; minting is additive. |
| 1579 | f := setup(t) |
| 1580 | seedAccount(t, f.store, "bdm") |
| 1581 | must(t, f.reg.Upsert(registry.Route{Host: "keep.eleven.bdm.sidecar.internal", |
| 1582 | Kind: "sidecar", App: "eleven", Account: "bdm", Provisioned: true})) |
| 1583 | f.store.failReadsFor(SidecarsPrefix("bdm", "eleven")) // fixture hook |
| 1584 | |
| 1585 | _ = ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}) |
| 1586 | if _, err := f.reg.Get("keep.eleven.bdm.sidecar.internal"); err != nil { |
| 1587 | t.Fatal("a read failure skipped nothing — one bad list must not retire a fleet") |
| 1588 | } |
| 1589 | } |
| 1590 | |
| 1591 | func TestSidecarReconcileRefusesHostWithAStagedCoSpawnFile(t *testing.T) { |
| 1592 | f := setup(t) |
| 1593 | seedAccount(t, f.store, "bdm") |
| 1594 | host := SidecarHost("bot", "eleven", "bdm") |
| 1595 | must(t, os.WriteFile(filepath.Join(f.dir, host+"-sidecar.env"), []byte("X=1\n"), 0o600)) |
| 1596 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}}) |
| 1597 | writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", Host: host, App: "eleven", Account: "bdm"}) |
| 1598 | |
| 1599 | _ = ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir}) |
| 1600 | if _, err := f.reg.Get(host); err == nil { |
| 1601 | t.Fatal("a host with a staged legacy co-spawn file must not also get a native sidecar row — one host, one mechanism") |
| 1602 | } |
| 1603 | } |
| 1604 | |
| 1605 | func TestSidecarReconcileConvergesArgsFromTheManifest(t *testing.T) { |
| 1606 | f := setup(t) |
| 1607 | seedAccount(t, f.store, "bdm") |
| 1608 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}}) |
| 1609 | writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm"}) |
| 1610 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) |
| 1611 | |
| 1612 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"worker", "run"}}) |
| 1613 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) |
| 1614 | |
| 1615 | rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm")) |
| 1616 | if rt.Args != `["worker","run"]` { |
| 1617 | t.Fatalf("Args = %q — truth lives in the bucket and is re-stamped every pass, exactly as seed is", rt.Args) |
| 1618 | } |
| 1619 | } |
| 1620 | |
| 1621 | func TestSidecarReconcileConvergesTheCeilings(t *testing.T) { |
| 1622 | f := setup(t) |
| 1623 | seedAccount(t, f.store, "bdm") |
| 1624 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{ |
| 1625 | Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, IdleSeconds: 300, MaxRunSeconds: 3600}) |
| 1626 | writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", |
| 1627 | Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm"}) |
| 1628 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) |
| 1629 | |
| 1630 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{ |
| 1631 | Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, IdleSeconds: 60, MaxRunSeconds: 120}) |
| 1632 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) |
| 1633 | |
| 1634 | rt := mustGet(t, f.reg, SidecarHost("bot", "eleven", "bdm")) |
| 1635 | if rt.IdleSeconds != 60 || rt.MaxRunSeconds != 120 { |
| 1636 | t.Fatalf("ceilings = %d/%d, want 60/120 — the sweeper reads the row, so the row must follow the manifest", rt.IdleSeconds, rt.MaxRunSeconds) |
| 1637 | } |
| 1638 | } |
| 1639 | ``` |
| 1640 | |
| 1641 | - [ ] **Step 2: Run and watch them fail.** |
| 1642 | |
| 1643 | - [ ] **Step 3: Guard the instance pass's removal phase** |
| 1644 | |
| 1645 | In `reconcile.go`'s removal loop, in the same condition that already checks |
| 1646 | `rt.Provisioned` and scope: |
| 1647 | |
| 1648 | ```go |
| 1649 | // Removal authority is scoped to this pass's OWN kind. A sidecar row |
| 1650 | // is Provisioned too, and can never appear in the instance record |
| 1651 | // set — without this guard the two passes would delete each other's |
| 1652 | // rows on every tick, forever, with no error anywhere. |
| 1653 | if rt.Kind == "sidecar" { |
| 1654 | continue |
| 1655 | } |
| 1656 | ``` |
| 1657 | |
| 1658 | - [ ] **Step 4: Implement `ReconcileSidecars`** |
| 1659 | |
| 1660 | Mirror `Reconcile` (`reconcile.go:330-675`) exactly, with these |
| 1661 | substitutions and additions: |
| 1662 | |
| 1663 | - records come from `ParseSidecarKey` over `apps/`, manifests from |
| 1664 | `GetSidecarManifest`. |
| 1665 | - scope is the same `rec.Region != cfg.Region || !poolServed(cfg.Pools, rec.Pool)` skip. |
| 1666 | - **placement gate**: skip a record whose `PlacementOrDefault()` is |
| 1667 | `"no-inbound"` unless `cfg.Inbound == "none"` (Task 10 supplies the field). |
| 1668 | - mint: |
| 1669 | ```go |
| 1670 | func sidecarRouteFor(rec SidecarRecord, m SidecarManifest, dataDir string) registry.Route { |
| 1671 | argv, _ := json.Marshal(m.Args) |
| 1672 | return registry.Route{ |
| 1673 | Host: rec.Host, Slug: rec.Name, Kind: "sidecar", |
| 1674 | App: rec.App, Account: rec.Account, Pool: rec.Pool, |
| 1675 | Region: rec.Region, Channel: rec.Channel, Environment: rec.Environment, |
| 1676 | DBPath: DefaultDBPath(dataDir, rec.Host), |
| 1677 | Hibernate: m.HibernateOrDefault(), |
| 1678 | Args: string(argv), |
| 1679 | IdleSeconds: int(m.IdleOrDefault() / time.Second), |
| 1680 | MaxRunSeconds: int(m.MaxRunOrDefault() / time.Second), |
| 1681 | Provisioned: true, |
| 1682 | Status: "ready", |
| 1683 | // Socket and Addr are deliberately left empty: a sidecar binds |
| 1684 | // nothing, and the edge refuses the kind outright. |
| 1685 | } |
| 1686 | } |
| 1687 | ``` |
| 1688 | - converge: re-stamp `Args` (narrow `SetArgs`), the ceilings (narrow |
| 1689 | `SetWorkerLimits`), `Channel`, `Environment`, and `Hibernate` when they |
| 1690 | differ. |
| 1691 | - removal: only `rt.Kind == "sidecar" && rt.Provisioned`, in scope, absent |
| 1692 | from the record set, and not in `protect`; **skipped entirely when any |
| 1693 | read failed**. |
| 1694 | - the co-spawn conflict: before minting, `os.Stat(filepath.Join(cfg.DataDir, rec.Host+"-sidecar.env"))` |
| 1695 | — if it exists, log loudly and skip. One host, one mechanism. |
| 1696 | |
| 1697 | Finally, add the leg to `internal/host/agent.go` beside the instance one, |
| 1698 | **inside `reconcileSerialized`'s mutex** so it cannot race the instance pass |
| 1699 | (the comment at `agent.go:525-537` explains exactly why an overlapping pass |
| 1700 | is dangerous), and inside `fullPass` so a console serial bump converges it |
| 1701 | within ~2s. |
| 1702 | |
| 1703 | - [ ] **Step 5: Run the tests, full gate, commit.** |
| 1704 | |
| 1705 | --- |
| 1706 | |
| 1707 | ### Task 10: Placement — boxes that accept no-inbound work |
| 1708 | |
| 1709 | **Files:** |
| 1710 | - Modify: `internal/host/agent.go` (`provisionTickConfig`, ~1316-1334) |
| 1711 | - Modify: `internal/provision/reconcile.go` (mint gate) |
| 1712 | - Modify: `internal/provision/sidecar_reconcile.go` (placement gate) |
| 1713 | - Test: `internal/host/agent_test.go`, `internal/provision/reconcile_test.go` |
| 1714 | |
| 1715 | **Interfaces:** |
| 1716 | - Consumes: Task 9. |
| 1717 | - Produces: `ReconcileConfig.Inbound string` (`""`/`"any"` or `"none"`). |
| 1718 | |
| 1719 | - [ ] **Step 1: Write the failing tests** |
| 1720 | |
| 1721 | ```go |
| 1722 | func TestProvisionTickConfigReadsInbound(t *testing.T) { |
| 1723 | env := map[string]string{"CARLOS_REGION": "ie", "CARLOS_POOLS": "eleven", "CARLOS_INBOUND": "none"} |
| 1724 | cfg := provisionTickConfig(func(k string) string { return env[k] }) |
| 1725 | if cfg.Inbound != "none" { |
| 1726 | t.Fatalf("Inbound = %q", cfg.Inbound) |
| 1727 | } |
| 1728 | cfg = provisionTickConfig(func(string) string { return "" }) |
| 1729 | if cfg.Inbound != "" { |
| 1730 | t.Fatal("an unset CARLOS_INBOUND must be an inert default — today's behaviour exactly") |
| 1731 | } |
| 1732 | } |
| 1733 | |
| 1734 | func TestNoInboundBoxRefusesRoutedInstances(t *testing.T) { |
| 1735 | f := setup(t) |
| 1736 | seedAccount(t, f.store, "bdm") |
| 1737 | writeManifest(t, f.store, "bdm", "tito", Manifest{Schema: 1, App: "tito", Domain: "go.tito.io"}) |
| 1738 | writeRecord(t, f.store, Record{Schema: 1, Host: "acme.go.tito.io", Slug: "acme", App: "tito", Account: "bdm", Channel: "stable"}) |
| 1739 | |
| 1740 | err := Reconcile(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir, Inbound: "none"}) |
| 1741 | if err != nil { |
| 1742 | t.Fatalf("Reconcile: %v", err) |
| 1743 | } |
| 1744 | if _, err := f.reg.Get("acme.go.tito.io"); err == nil { |
| 1745 | t.Fatal("a box that cannot serve must never be handed something to serve — a misfiled record is a log line, not a silent outage") |
| 1746 | } |
| 1747 | } |
| 1748 | |
| 1749 | func TestNoInboundPlacementOnlyLandsOnMatchingBoxes(t *testing.T) { |
| 1750 | f := setup(t) |
| 1751 | seedAccount(t, f.store, "bdm") |
| 1752 | writeSidecarManifest(t, f.store, "bdm", "eleven", SidecarManifest{ |
| 1753 | Schema: 1, App: "eleven", Args: []string{"sidecar", "run"}, Placement: "no-inbound"}) |
| 1754 | writeSidecarRecord(t, f.store, SidecarRecord{Schema: 1, Name: "bot", |
| 1755 | Host: SidecarHost("bot", "eleven", "bdm"), App: "eleven", Account: "bdm", Placement: "no-inbound"}) |
| 1756 | |
| 1757 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir})) // ordinary box |
| 1758 | if _, err := f.reg.Get(SidecarHost("bot", "eleven", "bdm")); err == nil { |
| 1759 | t.Fatal("a no-inbound workload must not land on a box that accepts inbound work") |
| 1760 | } |
| 1761 | |
| 1762 | must(t, ReconcileSidecars(context.Background(), f.store, f.reg, ReconcileConfig{DataDir: f.dir, Inbound: "none"})) |
| 1763 | if _, err := f.reg.Get(SidecarHost("bot", "eleven", "bdm")); err != nil { |
| 1764 | t.Fatal("it must land on a box that declares CARLOS_INBOUND=none") |
| 1765 | } |
| 1766 | } |
| 1767 | ``` |
| 1768 | |
| 1769 | - [ ] **Step 2: Run and watch them fail.** |
| 1770 | - [ ] **Step 3: Implement** — add the field, read the env var, and add the |
| 1771 | two gates with the comments the tests' failure messages quote. |
| 1772 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1773 | |
| 1774 | --- |
| 1775 | |
| 1776 | ### Task 11: Config delivery and restart reach sidecars |
| 1777 | |
| 1778 | **Files:** |
| 1779 | - Modify: `internal/appconfig/syncexec.go` (`execRouteBase`, ~398-407) |
| 1780 | - Modify: `internal/host/restart.go` (~77, 192, 545) |
| 1781 | - Test: `internal/appconfig/syncexec_test.go`, `internal/host/restart_test.go` |
| 1782 | |
| 1783 | **`release.Adoptable` is deliberately NOT changed.** It is the most reused |
| 1784 | predicate in the repo — adopt membership, capability convergence, socket and |
| 1785 | backing advisories, status pages — and every one of those is correct to |
| 1786 | exclude a sidecar. Each consumer that genuinely wants sidecars opts in by |
| 1787 | name. Two deliberate edits instead of one blind one. |
| 1788 | |
| 1789 | - [ ] **Step 1: Write the failing tests** |
| 1790 | |
| 1791 | ```go |
| 1792 | func TestSidecarGetsItsConfigEnvFile(t *testing.T) { |
| 1793 | rt := registry.Route{Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", |
| 1794 | App: "eleven", Status: "ready"} |
| 1795 | if !execRouteBase(rt) { |
| 1796 | t.Fatal("a sidecar must receive config delivery — this is the direct replacement for the hand-staged <host>-sidecar.env, and the whole reason the file path existed") |
| 1797 | } |
| 1798 | } |
| 1799 | |
| 1800 | func TestSidecarConfigStillExcludesPlatformAndUnitRoutes(t *testing.T) { |
| 1801 | base := registry.Route{Host: "bot.x.sidecar.internal", Kind: "sidecar", App: "eleven", Status: "ready"} |
| 1802 | plat := base |
| 1803 | plat.Platform = true |
| 1804 | if execRouteBase(plat) { |
| 1805 | t.Fatal("a Platform alias is still excluded") |
| 1806 | } |
| 1807 | unit := base |
| 1808 | unit.Backing = registry.BackingUnit |
| 1809 | if execRouteBase(unit) { |
| 1810 | t.Fatal("unit-backed routes are still excluded") |
| 1811 | } |
| 1812 | } |
| 1813 | |
| 1814 | func TestRestartPassCyclesSidecars(t *testing.T) { |
| 1815 | routes := []registry.Route{ |
| 1816 | {Host: "bot.x.sidecar.internal", Kind: "sidecar", App: "eleven", Account: "bdm", Status: "ready"}, |
| 1817 | } |
| 1818 | if len(restartCandidates(routes)) != 1 { |
| 1819 | t.Fatal("carlos restart -app X must cycle that app's sidecars too") |
| 1820 | } |
| 1821 | } |
| 1822 | ``` |
| 1823 | |
| 1824 | - [ ] **Step 2: Run and watch them fail.** |
| 1825 | - [ ] **Step 3: Implement** — widen `execRouteBase`'s kind test to |
| 1826 | `(rt.Kind == "instance" || rt.Kind == "sidecar")`, leaving every other |
| 1827 | clause exactly as it is; and in `restart.go` accept sidecars alongside |
| 1828 | `release.Adoptable` routes, cycling them via the activator's stop/wake |
| 1829 | rather than the adopt path. Also confirm the reaper in |
| 1830 | `syncexec.go:617` still treats `<host>-sidecar.env` as someone else's file |
| 1831 | — the legacy path must stay untouched (Task 16 documents why). |
| 1832 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1833 | |
| 1834 | --- |
| 1835 | |
| 1836 | ### Task 12: Console API — the sidecar verbs |
| 1837 | |
| 1838 | **Files:** |
| 1839 | - Create: `internal/console/apisidecars.go`, `apisidecars_test.go` |
| 1840 | - Modify: `internal/console/authcli.go` (`registerCLIRoutes`, ~273-286) |
| 1841 | |
| 1842 | **Interfaces:** |
| 1843 | - Consumes: Task 2's shapes, Task 13's wake object (build 13 first if you |
| 1844 | prefer; the wake handler can land in either order). |
| 1845 | - Produces the routes: |
| 1846 | ``` |
| 1847 | PUT /api/cli/apps/{account}/{app}/sidecars-manifest |
| 1848 | GET /api/cli/apps/{account}/{app}/sidecars |
| 1849 | POST /api/cli/apps/{account}/{app}/sidecars |
| 1850 | GET /api/cli/apps/{account}/{app}/sidecars/{name} |
| 1851 | DELETE /api/cli/apps/{account}/{app}/sidecars/{name} |
| 1852 | POST /api/cli/apps/{account}/{app}/sidecars/{name}/wake |
| 1853 | ``` |
| 1854 | |
| 1855 | **The app-held token works unchanged.** `CLIAppToken` binds a token to |
| 1856 | (account, app); these verbs authorize against the same principal, which is |
| 1857 | exactly the "apps can provision sidecar instances easily" requirement — |
| 1858 | Eleven's app process, holding no AWS credentials and no box access, creates |
| 1859 | a bot per enrolled member. |
| 1860 | |
| 1861 | Refusal order, copied from `apiinstances.go:474-478` and its tests: |
| 1862 | **principal gate → rate limit → releases configured → manifest present |
| 1863 | (409 naming `carlos sidecars enable`) → name shape (400) → region shape and |
| 1864 | allow-list (400) → channel (400) → shape validation (400) → create-only |
| 1865 | write (409 on a lost race) → audit + `bumpSerial` → 201.** |
| 1866 | |
| 1867 | - [ ] **Step 1: Write the failing tests** |
| 1868 | |
| 1869 | Mirror `apiinstances_test.go` exactly, reusing its fixture helpers |
| 1870 | (`rawReq`, `canonicalNotFound`, `assertNotFound`, `newConsoleFixture`): |
| 1871 | |
| 1872 | ```go |
| 1873 | func TestAPISidecarRefusalMatrix(t *testing.T) { |
| 1874 | // Every refusal before the principal is proven must be BYTE-IDENTICAL |
| 1875 | // to http.NotFound's own output: a non-member must never learn that an |
| 1876 | // account, an app, or a sidecar exists. |
| 1877 | // ... one case per: no such account, no such app, not a member, |
| 1878 | // app token bound to a different app, user token on an app verb. |
| 1879 | } |
| 1880 | |
| 1881 | func TestAPISidecarCreateRefusalOrder(t *testing.T) { |
| 1882 | // Walk the order above with one token, asserting each stage's code and |
| 1883 | // that the NEXT stage was not reached. |
| 1884 | } |
| 1885 | |
| 1886 | func TestAPISidecarAppTokenContainment(t *testing.T) { |
| 1887 | // An app token creates/lists/deletes/wakes its OWN app's sidecars, and |
| 1888 | // 401s on promote and on minting another token. |
| 1889 | } |
| 1890 | |
| 1891 | func TestAPISidecarCreateIsCreateOnly(t *testing.T) { |
| 1892 | // Second POST for the same name -> 409, never a silent overwrite. |
| 1893 | } |
| 1894 | |
| 1895 | func TestAPISidecarDeleteIsIdempotent(t *testing.T) { |
| 1896 | // DELETE of an absent name -> 200. The caller's goal is "not there". |
| 1897 | // And it must work after the manifest is gone: deletion outlives policy. |
| 1898 | } |
| 1899 | |
| 1900 | func TestAPISidecarCreateWithoutManifestNamesTheEnableVerb(t *testing.T) { |
| 1901 | // 409 whose body names `carlos sidecars enable` — membership is already |
| 1902 | // proven at this point, so this is not a leak. |
| 1903 | } |
| 1904 | |
| 1905 | func TestAPISidecarWakeBumpsTheGeneration(t *testing.T) { |
| 1906 | // Two wakes -> generation 1 then 2, and the serial moved both times. |
| 1907 | } |
| 1908 | ``` |
| 1909 | |
| 1910 | - [ ] **Step 2: Run and watch them fail.** |
| 1911 | - [ ] **Step 3: Implement** `apisidecars.go` following `apiinstances.go`'s |
| 1912 | structure handler-for-handler, and register the routes. The enable handler |
| 1913 | is gated by `apiAccountApp` **directly** (it is a human act), like |
| 1914 | `handleAPIInstanceEnable`; the rest go through the shared gate. |
| 1915 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1916 | |
| 1917 | --- |
| 1918 | |
| 1919 | ### Task 13: The wake generation object and the box-side pass |
| 1920 | |
| 1921 | **Files:** |
| 1922 | - Create: `internal/release/sidecarwake.go`, `sidecarwake_test.go` |
| 1923 | - Modify: `internal/host/restart.go` or a sibling in `internal/host` |
| 1924 | - Test: `internal/host/*_test.go` |
| 1925 | |
| 1926 | **Interfaces:** |
| 1927 | - Produces: |
| 1928 | ```go |
| 1929 | func SidecarWakeKey(acct, app, name string) string // apps/<acct>/<app>/sidecar-wake/<name>.json |
| 1930 | type SidecarWake struct{ Schema, Generation int; At, By, Reason string } |
| 1931 | ``` |
| 1932 | |
| 1933 | Copy `restartgen.go` (`internal/release/restartgen.go`) wholesale: a small |
| 1934 | CAS-guarded generation object whose **etag moving** is the entire signal. |
| 1935 | The box compares etags per (account, app, name) against a cache and wakes |
| 1936 | what moved — the `restartGenPass` shape (`internal/host/restart.go:128`). |
| 1937 | |
| 1938 | Nothing is pushed at a box. That is what lets the box live in a private |
| 1939 | subnet. |
| 1940 | |
| 1941 | - [ ] **Step 1: Write the failing tests** — an etag move wakes exactly the |
| 1942 | named sidecar; an unchanged etag wakes nothing; a first sighting on an |
| 1943 | unseeded cache does **not** replay a wake (the no-replay-on-absence rule |
| 1944 | the serial watcher already uses, `agent.go:1032-1042`); a read error for |
| 1945 | one app does not suppress the others. |
| 1946 | - [ ] **Step 2: Run and watch them fail.** |
| 1947 | - [ ] **Step 3: Implement.** |
| 1948 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1949 | |
| 1950 | --- |
| 1951 | |
| 1952 | ### Task 14: `carlos sidecars` |
| 1953 | |
| 1954 | **Files:** |
| 1955 | - Create: `sidecars_cli.go`, `sidecars_cli_test.go`, `sidecars_e2e_test.go` |
| 1956 | - Modify: `main.go` (dispatch), `apimode.go` (client methods) |
| 1957 | |
| 1958 | **Interfaces:** |
| 1959 | - Consumes: Task 12's routes. |
| 1960 | - Produces the verbs: |
| 1961 | ``` |
| 1962 | carlos sidecars enable -app <app> -args '<argv>' [-placement no-inbound] |
| 1963 | [-regions ie,fleet/eleven] [-pool p] [-channel c] |
| 1964 | [-idle 5m] [-max-run 1h] [-no-hibernate] |
| 1965 | carlos sidecars create -app <app> <name> |
| 1966 | carlos sidecars ls -app <app> [-long] |
| 1967 | carlos sidecars wake -app <app> <name> |
| 1968 | carlos sidecars rm -app <app> <name> |
| 1969 | carlos sidecars logs -app <app> <name> |
| 1970 | ``` |
| 1971 | |
| 1972 | Follow `instances_cli.go` precisely: subcommand words dispatch positionally |
| 1973 | before flag parsing; `enable` works in **both** API and direct/store mode |
| 1974 | (operator-only flags refused in signed-in mode); `create`/`ls`/`wake`/`rm` |
| 1975 | are **API-mode only** via `requireAPISession`, whose refusal must name |
| 1976 | sidecars' own advice, never another verb's. |
| 1977 | |
| 1978 | `ls` prints the short name by default and the synthetic host only under |
| 1979 | `-long`. It shows **running version against desired channel version**, and a |
| 1980 | sidecar that has never called `/v1/ready` shows `unknown`, never the desired |
| 1981 | version — a stamp that overclaims is worse than an absent one. |
| 1982 | |
| 1983 | `logs` delegates to the existing `carlos logs` path for the sidecar's host. |
| 1984 | |
| 1985 | - [ ] **Step 1: Write the failing tests** — unit tests for flag parsing and |
| 1986 | the duration flags (`-idle 5m` → `IdleSeconds: 300`), plus an e2e test in |
| 1987 | the `instances_e2e_test.go` mould driving a real `console.Server` over |
| 1988 | `httptest` end to end: enable → create → ls → wake → rm. |
| 1989 | |
| 1990 | **Every test that shells out to the carlos binary must build its env |
| 1991 | through `carlosTestEnv` and set `cmd.Dir` to a scratch dir.** The binary |
| 1992 | walks cwd ancestors for `.carlos/config`; the fallback is the operator's |
| 1993 | real login and default console. A logged-in laptop running |
| 1994 | `go test ./...` once minted 40+ real accounts on the production console. |
| 1995 | - [ ] **Step 2: Run and watch them fail.** |
| 1996 | - [ ] **Step 3: Implement.** |
| 1997 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 1998 | |
| 1999 | --- |
| 2000 | |
| 2001 | ### Task 15: The console tells the truth about sidecars |
| 2002 | |
| 2003 | **Files:** |
| 2004 | - Modify: `internal/console/instances.go` (~88-97), `internal/console/overview.go` |
| 2005 | (~129-144, ~237-264, ~286-306), `internal/status/snapshot.go` (~406-427) |
| 2006 | - Test: the matching `_test.go` files |
| 2007 | |
| 2008 | - [ ] **Step 1: Write the failing tests** |
| 2009 | |
| 2010 | ```go |
| 2011 | func TestReportingHostIsNeverASidecar(t *testing.T) { |
| 2012 | routes := []registry.Route{ |
| 2013 | {Host: "bot.eleven.bdm.sidecar.internal", Kind: "sidecar", Status: "ready"}, |
| 2014 | {Host: "eleven.example.com", Kind: "instance", Status: "ready"}, |
| 2015 | } |
| 2016 | if got := reportingHost(routes); got != "eleven.example.com" { |
| 2017 | t.Fatalf("reportingHost = %q — polling an unroutable sidecar makes a HEALTHY app read 'not reporting'", got) |
| 2018 | } |
| 2019 | } |
| 2020 | |
| 2021 | func TestSidecarIsNotAPublicResident(t *testing.T) { |
| 2022 | counts := CollectBoxCounts([]registry.Route{{Host: "bot.x.sidecar.internal", Kind: "sidecar"}}) |
| 2023 | if counts.Resident != 0 { |
| 2024 | t.Fatal("a sidecar must not be counted as a public resident app instance") |
| 2025 | } |
| 2026 | if counts.Sidecars != 1 { |
| 2027 | t.Fatal("it belongs in its own bucket") |
| 2028 | } |
| 2029 | } |
| 2030 | |
| 2031 | func TestSidecarHealthWordIsNotServing(t *testing.T) { |
| 2032 | if w := healthWord("sidecar", "ready", false, false, false); w == "Serving" { |
| 2033 | t.Fatal("a sidecar never serves a request — 'Serving' is false") |
| 2034 | } |
| 2035 | } |
| 2036 | ``` |
| 2037 | |
| 2038 | - [ ] **Step 2: Run and watch them fail.** |
| 2039 | - [ ] **Step 3: Implement** — filter the reporting-host picker by Kind, give |
| 2040 | `CollectBox` an explicit sidecar branch and a `Sidecars` counter (additive |
| 2041 | field), count sidecars separately in the overview, and give `healthWord` |
| 2042 | the sidecar vocabulary: `Running`, `Asleep`, `Waking`, `Crashed`, |
| 2043 | `Never started`. Sidecars are **not** on the public status page: their |
| 2044 | existence is account-visible, not public. |
| 2045 | - [ ] **Step 4: Run the tests, full gate, commit.** |
| 2046 | |
| 2047 | --- |
| 2048 | |
| 2049 | ### Task 16: Documentation, the runbook, and the legacy split |
| 2050 | |
| 2051 | **Files:** |
| 2052 | - Create: `infra/no-inbound-box.md` |
| 2053 | - Modify: `AGENTS.md` (was `CLAUDE.md` — see Step 3) |
| 2054 | - Modify: `internal/activator/activator.go` (package doc), `backend_exec.go` |
| 2055 | (the `HasSidecar`/`StartSidecar` comments) |
| 2056 | |
| 2057 | - [ ] **Step 1: Write `infra/no-inbound-box.md`** — the private-subnet |
| 2058 | runbook, in `infra/eleven-fleet.md`'s voice: preconditions, the tofu |
| 2059 | shape (private subnet, no IGW route, egress via NAT, no inbound SG rules, |
| 2060 | no SSM), the box env contract (`CARLOS_INBOUND=none` plus the fleet |
| 2061 | contract), verification, and an abort path at every step. Mark clearly: |
| 2062 | **written for Paul's applies; nothing here has been applied.** |
| 2063 | |
| 2064 | Note the easier posture too: fleet boxes are *already* outbound-only dial, |
| 2065 | so Eleven's first sidecar box needs no new network shape at all — the |
| 2066 | private-subnet variant is the harder posture, available when wanted. |
| 2067 | |
| 2068 | - [ ] **Step 2: Record the legacy split** in the activator's package doc and |
| 2069 | beside `HasSidecar`: the file-staged `<host>-sidecar.env` co-spawn is |
| 2070 | **legacy but live** (Eleven runs on it today), the record path is native, |
| 2071 | a host may not have both, and retiring the file path is a follow-up gated |
| 2072 | on Eleven's own migration. |
| 2073 | |
| 2074 | - [ ] **Step 3: Add an AGENTS.md bullet** under "Rules this repo has paid |
| 2075 | for". (Amended 2026-09-05: this said CLAUDE.md, and it was right when it |
| 2076 | was written. Main has since reduced CLAUDE.md to a pointer and moved the |
| 2077 | guidance — that section included — into AGENTS.md, which is where the |
| 2078 | bullet now lives.): |
| 2079 | |
| 2080 | ```markdown |
| 2081 | - **A sidecar is a member that nothing can dial.** `Kind == "sidecar"` |
| 2082 | rows carry no socket and no addr, are refused by the edge on kind (not |
| 2083 | merely unmatched by it), and are never advertised in a fleet binding — |
| 2084 | a fleet-placed row reaches the edge's fleet arm on shape alone. Each |
| 2085 | reconcile pass's removal authority is scoped to its OWN kind: the |
| 2086 | instance pass and the sidecar pass would otherwise delete each other's |
| 2087 | rows on every tick, forever, with no error anywhere. |
| 2088 | ``` |
| 2089 | |
| 2090 | - [ ] **Step 4: Full gate, commit.** |
| 2091 | |
| 2092 | --- |
| 2093 | |
| 2094 | ## Final verification before the PR |
| 2095 | |
| 2096 | - [ ] `go build ./...`, `go vet ./...`, `go test ./... -count=1` — all clean. |
| 2097 | - [ ] `GOOS=darwin go build ./...` — clean (`internal/host` links into the |
| 2098 | CLI, which people run on Macs). |
| 2099 | - [ ] `gofmt -l` clean for every file this branch touched. |
| 2100 | - [ ] `git log --oneline main..HEAD` reads as one coherent story. |
| 2101 | - [ ] No live box was touched; no real bucket was written. |
| 2102 | - [ ] Push the branch and open the PR. **Do not merge it** — this is a new |
| 2103 | subsystem touching the activator, the edge and the console's auth |
| 2104 | surface, which is exactly the diff CLAUDE.md says waits for a person. |
| 2105 | |