amadan
spec: sidecar members — a serving class that nothing can dial
1 file changed,
+625
−0
docs/superpowers/specs/2026-08-17-sidecar-members-design.md+625 −0
diff --git a/docs/superpowers/specs/2026-08-17-sidecar-members-design.md b/docs/superpowers/specs/2026-08-17-sidecar-members-design.md| new file mode 100644 |
| index 000000000..0064c4889 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-17-sidecar-members-design.md |
| @@ -0,0 +1,625 @@ |
| +# Sidecar members: a serving class that nothing can dial |
| + |
| +**Date:** 2026-08-17 · **Status:** approved design (Paul, via brainstorming |
| +session, 2026-08-17) · **Requested by:** Paul, for Eleven. |
| + |
| +**Closes:** the gap between the platform spec's `sidecar` serving class |
| +(`2026-08-01-carlos-platform-design.md` §5.4, §4 manifest) and what the |
| +platform can actually be asked to run. The runtime half has existed since |
| +Plan A; the declaration half has never existed. |
| + |
| +**Extends:** instance provisioning (`2026-08-04-instance-provisioning-design.md` |
| +— the record/manifest/reconciler/API pattern this mirrors), exec config |
| +delivery (`2026-08-02-exec-config-delivery-design.md` §"Per-sidecar config … |
| +deferred", which this closes), customer fleets |
| +(`2026-08-05-customer-fleets-design.md` — outbound-only boxes), |
| +`carlos restart` (#113 — the pull-based generation-object idiom the wake |
| +verb copies). |
| + |
| +**Deliberately does not depend on** the unmerged trust-classes and |
| +private-instances specs (PR #183). Those name the *vocabulary* this class |
| +will eventually be placed by (an A-class sidecar holding a member key is |
| +exactly their subject); nothing here waits on them. |
| + |
| +## 1. Why |
| + |
| +A sidecar today is a file on a box. `HasSidecar` is `os.Stat` of |
| +`/data/<host>-sidecar.env` (`internal/activator/backend_exec.go:732`), and |
| +nothing in the platform writes that file — config delivery explicitly |
| +disowns it (`internal/appconfig/syncexec.go:617`). So the only way to run |
| +one is to get root on a box and place a file, which is precisely the |
| +product gap CLAUDE.md names: *every `carlos` command is designed for a user |
| +with no access to the underlying infra.* |
| + |
| +The runtime that exists is also the wrong shape for what Eleven needs. It |
| +is a **co-spawn**: one sidecar bound to one routed instance, alive exactly |
| +while that instance is awake (`internal/activator/activator.go:17-24`). A |
| +bot that must run on a box with no routed app on it — eventually a box in a |
| +private subnet with no inbound path from the internet at all — cannot be |
| +expressed that way, because there is no parent instance to hang it from. |
| + |
| +## 2. Rulings (Paul, 2026-08-17) |
| + |
| +1. **Standalone members.** A sidecar gets its own record, its own registry |
| + row, its own identity and its own lifecycle. It can be the only thing on |
| + a box. The existing co-spawn path keeps working, untouched, for Eleven's |
| + current shape; provisioned sidecars never use it (§10). |
| +2. **Hibernating**, with schedule and operator-command wakes. Delivery-wake |
| + (the edge signalling that ciphertext is queued for a sidecar's identity) |
| + is designed here but **not built** — it needs edge work this does not |
| + include (§7.5). |
| +3. **Synthetic internal host** as the identity: `<name>.<app>.<sqid>.sidecar.internal`, |
| + never resolvable, refused by the edge, so every host-keyed subsystem |
| + works unchanged (§3). |
| +4. **Placement axis in the platform now; infra spec'd, not applied.** Boxes |
| + declare that they accept no-inbound work; the reconciler honours it; the |
| + tofu for a real private-subnet box is a runbook for Paul's applies (§8). |
| + |
| +## 3. Identity: a host that cannot be a host |
| + |
| +A sidecar is keyed by a **synthetic host**: |
| + |
| + <name>.<app>.<sqid>.sidecar.internal |
| + bot-acme.eleven.bdm.sidecar.internal |
| + |
| +`name` is one DNS label, validated by the same regex instance hosts use |
| +(`provision.labelRe`) and for the same stated reason: the name becomes a |
| +bucket key segment and a filesystem path segment (`<host>.db`, the cgroup |
| +directory, the log sink file), and the label regex is what keeps a hostile |
| +name an opaque, safe path component. |
| + |
| +**Why a synthetic host and not a new identity axis.** The registry, config |
| +delivery, cgroups, the S3 lease, the DB path, the log sink, metrics |
| +attribution, `carlos logs`, `carlos restart` and the release ladder are all |
| +keyed by host. A second identity axis would mean touching every consumer |
| +that assumes host is the key — the wire-shape rule this repo has paid for |
| +three times. A synthetic host costs one reserved suffix and buys every one |
| +of those subsystems working on day one. |
| + |
| +**`.sidecar.internal` is reserved, and reservation is enforced, not |
| +documented.** The suffix is refused at every door where a name becomes |
| +routable: |
| + |
| +- the edge never binds a `Kind == "sidecar"` row at all (§6.1), which is |
| + the structural guarantee; the suffix refusals below are defence in depth |
| + against a *different* row type wearing the name. |
| +- `carlos add` / the route door / domain attach refuse a host ending in |
| + `.sidecar.internal`. |
| +- the domains reconciler refuses to mint an alias for one. |
| +- a custom domain may not be attached to a sidecar. |
| + |
| +The CLI speaks the short name (`carlos sidecars ls` prints `bot-acme`); the |
| +synthetic host is the internal key, shown only by `-long`. |
| + |
| +## 4. Bucket shapes (all additive) |
| + |
| +Mirrors instance provisioning exactly, one directory over. |
| + |
| +### 4.1 `apps/<sqid>/<app>/sidecars.json` — the app's opt-in and policy |
| + |
| +```json |
| +{ |
| + "schema": 1, |
| + "app": "eleven", |
| + "pool": "eleven", |
| + "regions": ["fleet/eleven"], |
| + "channel": "stable", |
| + "args": ["sidecar", "run"], |
| + "placement": "no-inbound", |
| + "hibernate": true, |
| + "idle": "5m", |
| + "max_run": "1h" |
| +} |
| +``` |
| + |
| +- **Absence means the feature does not exist for that app** — the |
| + litestream/instances convention. The API refuses creates with a message |
| + naming `carlos sidecars enable`, only after membership is proven. |
| +- `args` (required): the argv the platform execs after the release binary. |
| + App-level, not per-record — like `seed`, it is a property of the app's |
| + current release contract. `["sidecar", "run"]` is the shape the existing |
| + co-spawn hard-codes; making it declared is what lets an app ship more |
| + than one kind of worker later. |
| + |
| + **One binary, many sidecars.** Args being app-level is what makes |
| + Eleven's shape — a bot per enrolled member — work: the same release runs |
| + every sidecar, and each child learns *which* one it is from its |
| + environment, not its argv: |
| + |
| + CARLOS_SIDECAR_NAME=bot-acme |
| + CARLOS_SIDECAR_HOST=bot-acme.eleven.bdm.sidecar.internal |
| + |
| + Per-sidecar configuration beyond that is the config plane's job, not the |
| + manifest's: a record may carry `environment`, and route-scoped config |
| + delivery (merged, `#206`) then gives that sidecar its own variables and |
| + sealed secrets. There is deliberately no per-record `args`. |
| +- `placement`: `"any"` (default) or `"no-inbound"` (§8). |
| +- `hibernate` (default true), `idle` (default 5m), `max_run` (default 1h) — |
| + the lifecycle knobs (§7). |
| +- `regions`, `pool`, `channel`: as instances. `regions: ["fleet/eleven"]` |
| + is how Eleven's sidecars land only on Eleven's own boxes. |
| + |
| +Written by the operator or the app owner — `carlos sidecars enable`, |
| +API and direct/store mode both, the `carlos instances enable` pattern. |
| + |
| +### 4.2 `apps/<sqid>/<app>/sidecars/<name>.json` — the record |
| + |
| +```json |
| +{ |
| + "schema": 1, |
| + "name": "bot-acme", |
| + "host": "bot-acme.eleven.bdm.sidecar.internal", |
| + "app": "eleven", |
| + "account": "bdm", |
| + "pool": "eleven", |
| + "region": "fleet/eleven", |
| + "channel": "stable", |
| + "placement": "no-inbound", |
| + "created_at": "2026-08-17T23:00:00Z", |
| + "created_by": "app:bdm/eleven" |
| +} |
| +``` |
| + |
| +Create-only (`If-None-Match: *`) — a lost race is a 409, never a silent |
| +overwrite. The body must agree with its own key path (name/app/account vs |
| +the key's segments) or the reconciler skips it loudly. No tombstone: |
| +deletion deletes the object; the parked DB and WAL stream under the |
| +regional bucket are deliberately untouched, the same data-safety contract |
| +`registry.Delete` and `carlos remove` hold. |
| + |
| +`ParseRecordKey`'s discipline is inherited whole: exactly one non-empty |
| +segment after `sidecars/`, so `sidecars.json` (the manifest) and any nested |
| +path both refuse to parse as a record. |
| + |
| +### 4.3 `apps/<sqid>/<app>/sidecar-wake/<name>.json` — the wake generation |
| + |
| +The operator/app wake source (§7.4), copied from `restart-gen.json` |
| +(`internal/release/restartgen.go`): a small CAS-guarded generation object |
| +whose **etag moving** is the whole signal. A sibling prefix rather than a |
| +key under `sidecars/` so that §4.2's one-segment rule stays exactly as |
| +strict as it reads. |
| + |
| +```json |
| +{"schema": 1, "generation": 3, "at": "2026-08-17T23:05:00Z", |
| + "by": "member:bdm", "reason": "operator wake"} |
| +``` |
| + |
| +## 5. Registry: `Kind = "sidecar"` |
| + |
| +A sidecar row is a route row that is not a route: `Kind "sidecar"`, |
| +**`Socket ""`, `Addr ""`** — the two columns that make a row servable are |
| +empty by construction, and §6.1 makes that structural rather than |
| +incidental. |
| + |
| +Set: `App`, `Account`, `Pool`, `Region`, `Channel`, `Slug` (the short |
| +name), `DBPath` (`<DataDir>/<host>.db`), `Hibernate`, `Provisioned true`, |
| +`NextDue`, and one additive column: |
| + |
| +- **`args`** (new, additive TEXT, JSON argv) — materialized from the |
| + manifest on every reconcile pass, exactly as `seed` is. Truth stays in |
| + the bucket; the column is the box's local copy. |
| + |
| +`Provisioned` carries removal authority as it does for instances: the flag, |
| +inside the agent's own region/pool scope, with the record gone, is what |
| +authorizes the one sanctioned destructive move. |
| + |
| +**Additive-only, per the wire-shape rule:** every route written before this |
| +change parses unchanged (`Kind` defaults to `"instance"` at |
| +`internal/registry/registry.go:704`; an absent `args` is the empty argv). |
| + |
| +**`DueInstances` must widen — this is load-bearing.** |
| +`internal/registry/registry.go:1341-1343` reads |
| + |
| +```sql |
| +SELECT ... FROM routes WHERE kind = 'instance' AND next_due > 0 AND next_due <= ? |
| +``` |
| + |
| +The sweeper's timer wake is driven entirely by that query. Left alone, a |
| +sidecar's `next_due` would be written, stored, and **never once acted on** — |
| +the schedule wake source would silently not exist, with no error anywhere. |
| +It widens to `kind IN ('instance','sidecar')`, and the test asserts a due |
| +sidecar row comes back. |
| + |
| +This is the shape of bug the wire-shape rule exists to catch: the consumer |
| +that breaks is a `WHERE` clause, not a caller. |
| + |
| +## 6. What makes a sidecar unreachable |
| + |
| +Three independent mechanisms, in order of how much they are relied on: |
| + |
| +### 6.1 The edge cannot bind it (structural) |
| + |
| +The edge's route→servable path refuses `Kind == "sidecar"` at the point a |
| +host is resolved to a route, before any upstream selection. Not "has no |
| +socket so nothing happens" — an explicit refusal with a test that asserts a |
| +sidecar row is never servable even if a `Socket` value is somehow present. |
| +The test is the point: it is the assertion that survives someone later |
| +adding a socket column to a sidecar row for an unrelated reason. |
| + |
| +Likewise the sidecar row is excluded from: upstream selection |
| +(`firstReadyUpstream`), alias minting (`candidateApps`), the apex/canary |
| +domain reconciler, custom domain attach, and the status page's public |
| +surface (a sidecar's existence is not public information). |
| + |
| +### 6.2 It has no listener (contract) |
| + |
| +A sidecar binds nothing. The platform never dials it — there is no health |
| +probe, no admin socket, no next-due probe. Everything the platform learns |
| +from a sidecar, the sidecar tells it by dialing **out** (§7.1). This is the |
| +property that lets the identical artifact run off-platform behind NAT, |
| +which the platform spec §5.4 names as the whole point of the class. |
| + |
| +### 6.2a The fleet-binding chain must be closed — this one is not theoretical |
| + |
| +A Kind-blind chain exists today from a resident route to a servable edge |
| +upstream, and **Eleven's sidecars sit exactly in it**, because they are |
| +fleet-placed (`region: "fleet/eleven"`): |
| + |
| +1. `publishResidentFleetBindings` (`internal/host/fleetbind.go:301-316`) |
| + iterates **every** route with no Kind filter and publishes a fleet |
| + binding claiming this box serves that host. |
| +2. `ReconcileFleetBindings` (`fleetbind.go:210-255`) reads it back, checks |
| + `registry.FleetPlacement(rt.Region)` — again no Kind filter — and stamps |
| + `SetFleetBox`. |
| +3. The edge's dispatch switch has a `case isFleetPlaced && e.tunnels != nil` |
| + arm (`internal/edge/edge.go:379-383`) with **no Kind check**, which |
| + builds a live tunnel proxy. |
| + |
| +A sidecar row is unroutable today only because it has no Socket, no Addr |
| +and no Tunnel — it falls to `default: continue` (`edge.go:445`) and is |
| +never inserted into the routing map. That is omission, not refusal. Put a |
| +fleet placement on it, which is precisely what Eleven's records carry, and |
| +step 3's arm is reached on shape alone. |
| + |
| +So the guard is added at **both** ends, and both are tested: |
| + |
| +- `publishResidentFleetBindings` skips `Kind == "sidecar"` — a sidecar is |
| + never advertised as served by any box. |
| +- the edge refuses `Kind == "sidecar"` **before** the dispatch switch, so |
| + no present or future arm of it can bind one. |
| + |
| +The edge-side refusal is the one that must hold on its own: it is the |
| +assertion that survives someone later adding a column, an arm, or a |
| +placement that reaches a sidecar row by shape. |
| + |
| +### 6.3 The box need not be reachable at all (placement) |
| + |
| +§8. A box that accepts only sidecar work needs no certs, no inbound ports, |
| +no tunnel, and can sit in a private subnet with egress-only networking. |
| + |
| +## 7. Lifecycle |
| + |
| +A sidecar is an activator-managed child, like a hibernating instance, with |
| +the probe-shaped parts replaced by an outbound protocol. |
| + |
| +### 7.1 The agent socket — the only channel, and it points outward |
| + |
| +The agent listens on a unix socket; children dial it. **Nothing ever dials |
| +a sidecar.** |
| + |
| + /run/carlos/agent.sock 0660 carlos:carlos |
| + |
| +Exec children all run as the agent's user, so the socket's file mode is not |
| +an identity boundary. Identity is a **per-wake token** the agent mints and |
| +passes in the child's environment — the `CARLOS_ADMIN_TOKEN` idiom |
| +inverted: |
| + |
| + CARLOS_AGENT_SOCKET=/run/carlos/agent.sock |
| + CARLOS_SIDECAR_TOKEN=<per-wake, per-sidecar, never persisted> |
| + |
| +The token is scoped to one sidecar for one wake: it authorizes calls about |
| +that host only, and it dies when the child is stopped. A stale token from a |
| +previous wake is refused. |
| + |
| +Verbs (all `POST`, all JSON, all tiny): |
| + |
| +| Verb | Meaning | |
| +|---|---| |
| +| `/v1/ready` | "I am up." Ends the settle window; carries the running version (§9.2). | |
| +| `/v1/next-due` `{"at": <unix>}` | "Wake me at T." Recorded in `routes.next_due`. | |
| +| `/v1/idle` | "I have nothing to do." The agent stops me. | |
| +| `/v1/heartbeat` | Optional liveness for a long-running sidecar. | |
| + |
| +An app that implements none of them still works: it runs until `max_run` |
| +or until the record is deleted. `idle` is what makes hibernation actually |
| +cheap, and `next-due` is what makes a schedule survive sleep. |
| + |
| +### 7.2 Wake, run, sleep |
| + |
| +**Wake** (any source): claim the per-host S3 lease → restore the DB from |
| +the bucket (`fresh` computed as today) → seed if fresh and a seed is |
| +declared → mint the wake token → spawn `live/<host> <args...>` in the |
| +host's cgroup, output teed to the log sink → **settle window** instead of a |
| +health probe: the child is considered up when it either calls `/v1/ready` |
| +or survives `sidecarSettle` (10s) without exiting. A child that exits |
| +non-zero inside the settle window is a failed wake, released and backed off |
| +like a failed instance start. |
| + |
| +**The seam this needs.** `doWake`'s step 4 calls `Backend.Start`, and |
| +`ExecBackend.Start` ends with an **unconditional** |
| +`waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` |
| +(`internal/activator/backend_exec.go:601`) — there is no branch for a route |
| +with no socket. Handed `Socket ""` it would dial `unix://""` every 150ms for |
| +the full 30s and then fail the wake. So the `Backend` seam gains one method |
| +rather than the existing one growing a mode flag: |
| + |
| +```go |
| +// StartWorker spawns a child that binds nothing. No health probe is |
| +// possible or wanted: readiness is the settle window, or the child's own |
| +// call to /v1/ready over the agent socket. |
| +StartWorker(ctx context.Context, rt registry.Route, tok string) (Proc, string, error) |
| +``` |
| + |
| +`ExecBackend.StartWorker` is `Start`'s body with the health wait and the |
| +socket-mode normalization removed and `args` in place of |
| +`--socket/--db` — which is very nearly what `StartSidecar` already is |
| +(`backend_exec.go:746-773`), the one spawn path in the package that |
| +deliberately has no health wait. `UnitBackend.StartWorker` refuses, exactly |
| +as it refuses `StartSidecar` today. |
| + |
| +**Run**: supervised. A crash restarts with the existing exponential |
| +backoff (2s → 1m, reset after a healthy minute) — the constants the |
| +co-spawn path already uses, now applied to a standalone member. |
| + |
| +**Sleep**, triggered by `/v1/idle`, by `max_run`, or by the idle timer |
| +(no `/v1/heartbeat` and no agent-socket call within `idle`): |
| +SIGTERM (5s grace, then SIGKILL) → record `next_due` from whatever the |
| +sidecar last declared → park the DB to the bucket → release the lease. |
| +The ordering is the instance stop path's, minus the next-due probe: a |
| +sidecar's next-due arrived over the socket while it was alive, which is |
| +strictly better than asking a process we are about to kill. |
| + |
| +**Always-on** is `hibernate: false`: the record exists ⇒ the process runs, |
| +same supervision, no idle clock, no lease churn. |
| + |
| +### 7.3 Where this lives: inside the activator, off its request path |
| + |
| +A sidecar's lifecycle is ~80% of `doWake`/`stop` already: acquire the lease, |
| +restore, seed, spawn, replicate; then stop, park, release. Reimplementing |
| +that in a sibling package would duplicate exactly the code where bugs are |
| +expensive — the error paths that must release a lease. So sidecars reuse |
| +`activator.Service` and its `actEntry`, with the request path left alone: |
| + |
| +- **Nothing routes to a sidecar, so `beginRequest`, `BeginSidecarRequest` |
| + and `ServeHTTP` never see one.** The edge refuses the row first (§6.1), |
| + and `ServeHTTP` already gates on `rt.Kind == "instance" && rt.Hibernate` |
| + (`activator.go:1128`), so it is unreachable there twice over. A new |
| + `WakeWorker(host)` entry point creates the entry and runs the same |
| + `wake`/`doWake` machinery the sweeper's request-less due-time wake uses. |
| +- **The idle clock is reused verbatim, with a different input.** For a |
| + sidecar `inflight` is always 0 and `lastActive` is stamped by **agent |
| + socket contact** instead of by a released HTTP request. The existing |
| + predicate — `inflight == 0 && now-lastActive >= idle` |
| + (`activator.go:927`) — then means exactly the right thing with no new |
| + branch, and `/v1/idle` is a stop requested immediately rather than a |
| + timestamp nudged backwards. |
| +- **`stop` skips only the next-due probe** (`activator.go:820-845`): a |
| + sidecar's next-due arrived over the socket while it was alive, which is |
| + strictly better than asking a process we are about to kill. Everything |
| + after it — stop, final checkpoint, lease release — is shared unchanged. |
| +- **`max_run`** is one more sweep predicate beside the idle one. |
| + |
| +The co-spawn fields on `actEntry` (`sidecar`, `sidecarWanted`, …) are |
| +untouched: a standalone sidecar is the entry's own `proc`, not a passenger |
| +on someone else's. |
| + |
| +### 7.4 Wake sources built here |
| + |
| +1. **Timer** — the sweeper's existing due-time wake, unchanged: a sleeping |
| + host whose `next_due` has arrived is woken with no client request. The |
| + machinery already exists for instances; a sidecar is simply a host that |
| + only ever wakes this way. |
| +2. **Operator/app command** — `carlos sidecars wake`, pull-based: the |
| + console bumps the wake generation (§4.3) and the serial; every box |
| + serving that app notices within seconds (the `carlos restart` pattern |
| + verbatim) and wakes the named sidecar. Nothing is pushed at a box — |
| + which is what lets the box live in a private subnet. |
| +3. **Boot** — a non-hibernating sidecar starts with the agent. |
| + |
| +### 7.5 Delivery wake — designed, not built |
| + |
| +The spec's third wake source is the edge observing (metadata only) that |
| +ciphertext is queued for a sidecar's identity and signalling the host |
| +agent. It is out of scope here and needs: an identity→sidecar mapping the |
| +edge can consult without learning content, a signalling path to a box that |
| +may have no inbound port (so: the same pull-based generation object as §7.4, |
| +written by the edge instead of the console), and a rate discipline so a |
| +chatty correspondent cannot bill a member for continuous wakes. Recorded so |
| +the shape is not invented differently later. |
| + |
| +## 8. Placement: boxes that accept no-inbound work |
| + |
| +Two halves that must agree. |
| + |
| +**The box declares.** A new env var, read where `CARLOS_REGION`/ |
| +`CARLOS_POOLS` are read (`internal/host/agent.go:1317`): |
| + |
| + CARLOS_INBOUND=none # this box serves sidecar work only |
| + |
| +Unset/`any` is today's behaviour exactly — an inert default, the |
| +apps-domain-style gate this repo already uses. On a `none` box: |
| + |
| +- the provisioning reconciler **refuses to mint routed instance rows**, |
| + loudly. A box that cannot serve must never be handed something to serve; |
| + this is the guard that turns a misfiled record into a log line instead of |
| + a silent outage. |
| +- the edge and cert machinery are not required to be running at all. |
| + |
| +**The record asks.** `placement: "no-inbound"` on the manifest/record makes |
| +the sidecar reconciler mint **only** on boxes declaring `CARLOS_INBOUND=none`. |
| +`placement: "any"` (default) places by pool/region alone. |
| + |
| +The two are independent on purpose: `none` is a property of the box's |
| +network, `no-inbound` is a requirement of the workload. A member who wants |
| +"this bot must only ever run somewhere with no inbound path" gets a |
| +mechanical guarantee rather than an operator's promise. |
| + |
| +**Infra is written, not applied** (Paul's ruling): a runbook for a |
| +private-subnet box — no IGW route, egress via NAT, no inbound SG rules, no |
| +SSM — lands beside `infra/eleven-fleet.md` for Paul's own applies. Note |
| +that fleet boxes are *already* outbound-only dial (`infra/eleven-fleet.md`), |
| +so Eleven's first sidecar box needs no new network shape at all: the |
| +private-subnet variant is the harder posture, available when wanted. |
| + |
| +## 9. The rest of the platform |
| + |
| +### 9.1 Config delivery |
| + |
| +Sidecars get their env files the way instances do — `execRouteBase` |
| +(`internal/appconfig/syncexec.go:404`) widens to include `Kind == "sidecar"`. |
| +This is the direct replacement for the hand-staged `<host>-sidecar.env`: |
| +`carlos env set` reaches a sidecar, secrets are sealed and delivered the |
| +same way, and the 2s serial-driven convergence applies unchanged. |
| + |
| +### 9.2 Version visibility without a header |
| + |
| +`carlos deploy` verifies a roll by reading `X-Carlos-Version` off the URL. |
| +A sidecar has no URL. The substitute is the `/v1/ready` call: the sidecar |
| +reports the version it is running, the agent records it on the row, and |
| +`carlos sidecars ls` shows *running* version against *desired* channel |
| +version. A sidecar that never calls `/v1/ready` shows its version as |
| +unknown rather than as current — the honesty rule this repo has paid for: |
| +a stamp that overclaims is worse than an absent one. |
| + |
| +### 9.3 Logs, metrics, restart |
| + |
| +**Correct for free**, because they key on host or on the cgroup name rather |
| +than on Kind: the log sink names its file by host; `internal/cgroup`'s |
| +`carlos-exec@<host>` convention means `listExecCgroups` |
| +(`internal/host/metrics_probes.go:240-261`) samples a sidecar's CPU and |
| +memory the moment it spawns into that cgroup, with no code change; and |
| +`ops.FleetDBs`'s `claimed` map (`internal/ops/litestream.go:371-379`) is |
| +Kind-agnostic, so a sidecar's `DBPath` is replicated like any other. |
| + |
| +**Needs widening**: `carlos restart -app X` is gated on |
| +`release.Adoptable` (`internal/release/adopt.go:518-521`, via |
| +`internal/host/restart.go:77,192,545`), which requires `Kind == "instance"`. |
| +Sidecars are cycled by the restart pass too, so `restart.go` widens. |
| + |
| +**`release.Adoptable` itself does not change.** It is the most reused |
| +predicate in the repo — adopt membership, capability convergence, socket |
| +and backing advisories, status pages all inherit it, and every one of those |
| +is *correct* to exclude a sidecar. Widening it centrally would silently |
| +enrol sidecars in capability drop-ins and adopt bookkeeping that make no |
| +sense for them. Each consumer that genuinely wants sidecars opts in by |
| +name: the config-delivery predicate (§9.1) and the restart pass. That is |
| +two deliberate edits instead of one blind one. |
| + |
| +### 9.4 Consumers that need a guard, and why |
| + |
| +The exhaustive list, from a sweep of every `.Kind` branch and every route |
| +iteration in the tree. Each is a test in this branch. |
| + |
| +| Site | Today with a sidecar row | Change | |
| +|---|---|---| |
| +| `edge.go:364-445` dispatch | unroutable by omission (`default: continue`) | explicit Kind refusal before the switch (§6.1) | |
| +| `fleetbind.go:301-316` publish | advertises a sidecar as served | skip `Kind == "sidecar"` (§6.2a) | |
| +| `provision/reconcile.go:651-673` removal | **deletes every sidecar row, every pass** | see below | |
| +| `registry.go:1341` `DueInstances` | timer wakes never fire | `kind IN ('instance','sidecar')` (§5) | |
| +| `appconfig/syncexec.go:404` | no env file ever written | include sidecars (§9.1) | |
| +| `host/restart.go` (via `Adoptable`) | never cycled | include sidecars (§9.3) | |
| +| `console/instances.go:88-97` | can pick a sidecar as an app's *reporting host*, making a healthy app read "not reporting" | filter by Kind | |
| +| `status/snapshot.go:406-427` `CollectBox` | falls to `default:` → counted as a public **Resident** instance | explicit branch; sidecars are not public | |
| +| `console/overview.go:286-306` `healthWord` | renders "Serving", which is false | a sidecar's own words (§9.5) | |
| +| `console/overview.go:129-144` | `InstanceTotal` counts sidecars | count separately | |
| +| `cli.go:75,194-199` `carlos add -kind` | **no validation at all** — accepts `-kind sidecar` today | rail it off (§3) | |
| + |
| +**The removal-phase collision is the sharpest of these.** The instance |
| +reconciler's removal phase (`reconcile.go:651-673`) is scoped by |
| +`Provisioned`, region, pool and absence-from-records — with **no Kind |
| +check**. Sidecar rows are `Provisioned` (§5) and can never appear in the |
| +instance record set, so the existing pass would delete every sidecar on |
| +every tick: a silent, self-healing-looking loop where sidecars are minted |
| +and destroyed forever, with no error anywhere. |
| + |
| +The fix is that each pass's removal authority is scoped to **its own Kind**: |
| +the instance pass skips `Kind == "sidecar"`, the sidecar pass touches only |
| +sidecar rows. Both halves are asserted directly, because this is the |
| +failure mode with no symptom other than work quietly disappearing. |
| + |
| +### 9.5 What a sidecar's status reads |
| + |
| +A sidecar never serves a request, so the vocabulary the console uses for |
| +routes is wrong for it. It gets its own words — `Running`, `Asleep`, |
| +`Waking`, `Crashed`, `Never started` — and is counted in its own bucket |
| +rather than folded into an instance count. Sidecars are **not** on the |
| +public status page: their existence is account-visible, not public. |
| + |
| +### 9.6 Object storage, ledgers, economics |
| + |
| +A sidecar is an ordinary tenant for the object-store primitive and accrues |
| +usage like any exec child. No new meters. |
| + |
| +## 10. The existing co-spawn stays |
| + |
| +`HasSidecar`/`StartSidecar` on the file-staged path are **not removed**. |
| +Eleven runs on that shape today, and this branch does not touch a live box. |
| +The relationship is stated in code and docs: the file path is legacy, the |
| +record path is native, and a host may not have both (the reconciler refuses |
| +to mint a sidecar row for a host that already has a staged |
| +`<host>-sidecar.env`, naming the conflict). Retiring the file path is a |
| +follow-up issue, gated on Eleven's own migration. |
| + |
| +## 11. Console API and CLI |
| + |
| +Verbs under `registerCLIRoutes`, bearer, JSON, mirroring the instances |
| +family including its refusal ordering (principal gate → rate limit → |
| +manifest present → name shape → region shape/allow-list → channel → the |
| +create-only write) and its byte-identical-404 discipline: |
| + |
| + GET /api/cli/apps/{account}/{app}/sidecars |
| + GET /api/cli/apps/{account}/{app}/sidecars/{name} |
| + POST /api/cli/apps/{account}/{app}/sidecars |
| + DELETE /api/cli/apps/{account}/{app}/sidecars/{name} |
| + POST /api/cli/apps/{account}/{app}/sidecars/{name}/wake |
| + |
| +The **app-held token** kind that authorizes an app's own instance verbs |
| +extends to its own sidecar verbs — this is the "apps can provision sidecar |
| +instances easily" requirement: Eleven's app process, holding no AWS |
| +credentials and no box access, creates a bot per enrolled member. |
| + |
| +CLI, the `instances` family's shape: |
| + |
| + carlos sidecars enable -app eleven -args 'sidecar run' [-placement no-inbound] |
| + carlos sidecars create -app eleven bot-acme |
| + carlos sidecars ls -app eleven |
| + carlos sidecars wake -app eleven bot-acme |
| + carlos sidecars rm -app eleven bot-acme |
| + carlos sidecars logs -app eleven bot-acme |
| + |
| +Every one of them completes its job with no box access and verifies its own |
| +result, per the no-infra rule. |
| + |
| +## 12. Testing |
| + |
| +- **Unreachability is a test, not a claim**: a `Kind == "sidecar"` row is |
| + never servable by the edge, never selected as an upstream, never aliased, |
| + never attachable to a domain, never on the status page — asserted |
| + directly, including the adversarial case of a sidecar row carrying a |
| + socket. |
| +- **Reserved suffix**: every door that mints or accepts a host refuses |
| + `.sidecar.internal`. |
| +- **Lifecycle**: wake → ready → next-due → idle → sleep → timer wake, on |
| + the activator's existing fake clock/fake backend harness; crash-backoff; |
| + settle-window failure; `max_run`; lease release on every exit path. |
| +- **Agent socket**: token scoping (a token for host A cannot speak about |
| + host B), stale-token refusal after a stop, malformed input, and the |
| + socket's absence being survivable. |
| +- **Reconciler**: mint/converge/remove with the fail-safe (a read failure |
| + skips the removal phase), placement matching, and the co-spawn conflict |
| + refusal. |
| +- **Additive parse**: a route row and a bucket record written before this |
| + change still parse. |
| +- The gate: `go build ./...`, `go vet ./...`, `go test ./... -count=1`, |
| + `gofmt -l` clean, plus the `GOOS=darwin` compile. |
| + |
| +## 13. Explicitly out of scope |
| + |
| +- Delivery-wake (§7.5). |
| +- Retiring the file-staged co-spawn (§10). |
| +- Applying any infra (§8) or touching any live box. |
| +- Sidecars for unit-backed tenants — `UnitBackend` refuses agent-managed |
| + sidecars today and continues to; the capabilities spec already names this |
| + a non-goal. |
| +- Multi-box placement/failover for a sidecar: one record, one lease, one |
| + box at a time, exactly as instances work. |