carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Sidecar members: a serving class that nothing can dial

Date: 2026-08-17 · Status: approved design (Paul, via brainstorming session, 2026-08-17) · Requested by: Paul, for Eleven.

Closes: the gap between the platform spec's sidecar serving class (2026-08-01-carlos-platform-design.md §5.4, §4 manifest) and what the platform can actually be asked to run. The runtime half has existed since Plan A; the declaration half has never existed.

Extends: instance provisioning (2026-08-04-instance-provisioning-design.md — the record/manifest/reconciler/API pattern this mirrors), exec config delivery (2026-08-02-exec-config-delivery-design.md §"Per-sidecar config … deferred", which this closes), customer fleets (2026-08-05-customer-fleets-design.md — outbound-only boxes), carlos restart (#113 — the pull-based generation-object idiom the wake verb copies).

Deliberately does not depend on the unmerged trust-classes and private-instances specs (PR #183). Those name the vocabulary this class will eventually be placed by (an A-class sidecar holding a member key is exactly their subject); nothing here waits on them.

1. Why

A sidecar today is a file on a box. HasSidecar is os.Stat of /data/<host>-sidecar.env (internal/activator/backend_exec.go:732), and nothing in the platform writes that file — config delivery explicitly disowns it (internal/appconfig/syncexec.go:617). So the only way to run one is to get root on a box and place a file, which is precisely the product gap CLAUDE.md names: every carlos command is designed for a user with no access to the underlying infra.

The runtime that exists is also the wrong shape for what Eleven needs. It is a co-spawn: one sidecar bound to one routed instance, alive exactly while that instance is awake (internal/activator/activator.go:17-24). A bot that must run on a box with no routed app on it — eventually a box in a private subnet with no inbound path from the internet at all — cannot be expressed that way, because there is no parent instance to hang it from.

2. Rulings (Paul, 2026-08-17)

  1. Standalone members. A sidecar gets its own record, its own registry row, its own identity and its own lifecycle. It can be the only thing on a box. The existing co-spawn path keeps working, untouched, for Eleven's current shape; provisioned sidecars never use it (§10).
  2. Hibernating, with schedule and operator-command wakes. Delivery-wake (the edge signalling that ciphertext is queued for a sidecar's identity) is designed here but not built — it needs edge work this does not include (§7.5).
  3. Synthetic internal host as the identity: <name>.<app>.<sqid>.sidecar.internal, never resolvable, refused by the edge, so every host-keyed subsystem works unchanged (§3).
  4. Placement axis in the platform now; infra spec'd, not applied. Boxes declare that they accept no-inbound work; the reconciler honours it; the tofu for a real private-subnet box is a runbook for Paul's applies (§8).

3. Identity: a host that cannot be a host

A sidecar is keyed by a synthetic host:

<name>.<app>.<sqid>.sidecar.internal bot-acme.eleven.bdm.sidecar.internal

name is one DNS label, validated by the same regex instance hosts use (provision.labelRe) and for the same stated reason: the name becomes a bucket key segment and a filesystem path segment (<host>.db, the cgroup directory, the log sink file), and the label regex is what keeps a hostile name an opaque, safe path component.

Why a synthetic host and not a new identity axis. The registry, config delivery, cgroups, the S3 lease, the DB path, the log sink, metrics attribution, carlos logs, carlos restart and the release ladder are all keyed by host. A second identity axis would mean touching every consumer that assumes host is the key — the wire-shape rule this repo has paid for three times. A synthetic host costs one reserved suffix and buys every one of those subsystems working on day one.

.sidecar.internal is reserved, and reservation is enforced, not documented. The suffix is refused at every door where a name becomes routable:

  • the edge never binds a Kind == "sidecar" row at all (§6.1), which is the structural guarantee; the suffix refusals below are defence in depth against a different row type wearing the name.
  • carlos add / the route door / domain attach refuse a host ending in .sidecar.internal.
  • the domains reconciler refuses to mint an alias for one.
  • a custom domain may not be attached to a sidecar.

The CLI speaks the short name (carlos sidecars ls prints bot-acme); the synthetic host is the internal key, shown only by -long.

4. Bucket shapes (all additive)

Mirrors instance provisioning exactly, one directory over.

4.1 apps/<sqid>/<app>/sidecars.json — the app's opt-in and policy

{
  "schema": 1,
  "app": "eleven",
  "pool": "eleven",
  "regions": ["fleet/eleven"],
  "channel": "stable",
  "args": ["sidecar", "run"],
  "placement": "no-inbound",
  "hibernate": true,
  "idle": "5m",
  "max_run": "1h"
}
  • Absence means the feature does not exist for that app — the litestream/instances convention. The API refuses creates with a message naming carlos sidecars enable, only after membership is proven.
  • args (required): the argv the platform execs after the release binary. App-level, not per-record — like seed, it is a property of the app's current release contract. ["sidecar", "run"] is the shape the existing co-spawn hard-codes; making it declared is what lets an app ship more than one kind of worker later.

One binary, many sidecars. Args being app-level is what makes Eleven's shape — a bot per enrolled member — work: the same release runs every sidecar, and each child learns which one it is from its environment, not its argv:

CARLOSSIDECARNAME=bot-acme CARLOSSIDECARHOST=bot-acme.eleven.bdm.sidecar.internal

Per-sidecar configuration beyond that is the config plane's job, not the manifest's: a record may carry environment, and route-scoped config delivery (merged, #206) then gives that sidecar its own variables and sealed secrets. There is deliberately no per-record args.

  • placement: "any" (default) or "no-inbound" (§8).
  • hibernate (default true), idle (default 5m), max_run (default 1h) — the lifecycle knobs (§7).
  • regions, pool, channel: as instances. regions: ["fleet/eleven"] is how Eleven's sidecars land only on Eleven's own boxes.

Written by the operator or the app owner — carlos sidecars enable, API and direct/store mode both, the carlos instances enable pattern.

4.2 apps/<sqid>/<app>/sidecars/<name>.json — the record

{
  "schema": 1,
  "name": "bot-acme",
  "host": "bot-acme.eleven.bdm.sidecar.internal",
  "app": "eleven",
  "account": "bdm",
  "pool": "eleven",
  "region": "fleet/eleven",
  "channel": "stable",
  "placement": "no-inbound",
  "created_at": "2026-08-17T23:00:00Z",
  "created_by": "app:bdm/eleven"
}

Create-only (If-None-Match: *) — a lost race is a 409, never a silent overwrite. The body must agree with its own key path (name/app/account vs the key's segments) or the reconciler skips it loudly. No tombstone: deletion deletes the object; the parked DB and WAL stream under the regional bucket are deliberately untouched, the same data-safety contract registry.Delete and carlos remove hold.

ParseRecordKey's discipline is inherited whole: exactly one non-empty segment after sidecars/, so sidecars.json (the manifest) and any nested path both refuse to parse as a record.

4.3 apps/<sqid>/<app>/sidecar-wake/<name>.json — the wake generation

The operator/app wake source (§7.4), copied from restart-gen.json (internal/release/restartgen.go): a small CAS-guarded generation object whose etag moving is the whole signal. A sibling prefix rather than a key under sidecars/ so that §4.2's one-segment rule stays exactly as strict as it reads.

{"schema": 1, "generation": 3, "at": "2026-08-17T23:05:00Z",
 "by": "member:bdm", "reason": "operator wake"}

5. Registry: Kind = "sidecar"

A sidecar row is a route row that is not a route: Kind "sidecar", Socket "", Addr "" — the two columns that make a row servable are empty by construction, and §6.1 makes that structural rather than incidental.

Set: App, Account, Pool, Region, Channel, Slug (the short name), DBPath (<DataDir>/<host>.db), Hibernate, Provisioned true, NextDue, and one additive column:

  • args (new, additive TEXT, JSON argv) — materialized from the manifest on every reconcile pass, exactly as seed is. Truth stays in the bucket; the column is the box's local copy.

Provisioned carries removal authority as it does for instances: the flag, inside the agent's own region/pool scope, with the record gone, is what authorizes the one sanctioned destructive move.

Additive-only, per the wire-shape rule: every route written before this change parses unchanged (Kind defaults to "instance" at internal/registry/registry.go:704; an absent args is the empty argv).

DueInstances must widen — this is load-bearing. internal/registry/registry.go:1341-1343 reads

SELECT ... FROM routes WHERE kind = 'instance' AND next_due > 0 AND next_due <= ?

The sweeper's timer wake is driven entirely by that query. Left alone, a sidecar's next_due would be written, stored, and never once acted on — the schedule wake source would silently not exist, with no error anywhere. It widens to kind IN ('instance','sidecar'), and the test asserts a due sidecar row comes back.

This is the shape of bug the wire-shape rule exists to catch: the consumer that breaks is a WHERE clause, not a caller.

6. What makes a sidecar unreachable

Three independent mechanisms, in order of how much they are relied on:

6.1 The edge cannot bind it (structural)

The edge's route→servable path refuses Kind == "sidecar" at the point a host is resolved to a route, before any upstream selection. Not "has no socket so nothing happens" — an explicit refusal with a test that asserts a sidecar row is never servable even if a Socket value is somehow present. The test is the point: it is the assertion that survives someone later adding a socket column to a sidecar row for an unrelated reason.

Likewise the sidecar row is excluded from: upstream selection (firstReadyUpstream), alias minting (candidateApps), the apex/canary domain reconciler, custom domain attach, and the status page's public surface (a sidecar's existence is not public information).

6.2 It has no listener (contract)

A sidecar binds nothing. The platform never dials it — there is no health probe, no admin socket, no next-due probe. Everything the platform learns from a sidecar, the sidecar tells it by dialing out (§7.1). This is the property that lets the identical artifact run off-platform behind NAT, which the platform spec §5.4 names as the whole point of the class.

6.2a The fleet-binding chain must be closed — this one is not theoretical

A Kind-blind chain exists today from a resident route to a servable edge upstream, and Eleven's sidecars sit exactly in it, because they are fleet-placed (region: "fleet/eleven"):

  1. publishResidentFleetBindings (internal/host/fleetbind.go:301-316) iterates every route with no Kind filter and publishes a fleet binding claiming this box serves that host.
  2. ReconcileFleetBindings (fleetbind.go:210-255) reads it back, checks registry.FleetPlacement(rt.Region) — again no Kind filter — and stamps SetFleetBox.
  3. The edge's dispatch switch has a case isFleetPlaced && e.tunnels != nil arm (internal/edge/edge.go:379-383) with no Kind check, which builds a live tunnel proxy.

A sidecar row is unroutable today only because it has no Socket, no Addr and no Tunnel — it falls to default: continue (edge.go:445) and is never inserted into the routing map. That is omission, not refusal. Put a fleet placement on it, which is precisely what Eleven's records carry, and step 3's arm is reached on shape alone.

So the guard is added at both ends, and both are tested:

  • publishResidentFleetBindings skips Kind == "sidecar" — a sidecar is never advertised as served by any box.
  • the edge refuses Kind == "sidecar" before the dispatch switch, so no present or future arm of it can bind one.

The edge-side refusal is the one that must hold on its own: it is the assertion that survives someone later adding a column, an arm, or a placement that reaches a sidecar row by shape.

6.3 The box need not be reachable at all (placement)

§8. A box that accepts only sidecar work needs no certs, no inbound ports, no tunnel, and can sit in a private subnet with egress-only networking.

7. Lifecycle

A sidecar is an activator-managed child, like a hibernating instance, with the probe-shaped parts replaced by an outbound protocol.

7.1 The agent socket — the only channel, and it points outward

The agent listens on a unix socket; children dial it. Nothing ever dials a sidecar.

/run/carlos/agent.sock 0660 carlos:carlos

Exec children all run as the agent's user, so the socket's file mode is not an identity boundary. Identity is a per-wake token the agent mints and passes in the child's environment — the CARLOS_ADMIN_TOKEN idiom inverted:

CARLOSAGENTSOCKET=/run/carlos/agent.sock CARLOSSIDECARTOKEN=<per-wake, per-sidecar, never persisted>

The token is scoped to one sidecar for one wake: it authorizes calls about that host only, and it dies when the child is stopped. A stale token from a previous wake is refused.

Verbs (all POST, all JSON, all tiny):

| Verb | Meaning | |---|---| | /v1/ready | "I am up." Ends the settle window; carries the running version (§9.2). | | /v1/next-due {"at": <unix>} | "Wake me at T." Recorded in routes.next_due. | | /v1/idle | "I have nothing to do." The agent stops me. | | /v1/heartbeat | Optional liveness for a long-running sidecar. |

An app that implements none of them still works: it runs until max_run or until the record is deleted. idle is what makes hibernation actually cheap, and next-due is what makes a schedule survive sleep.

7.2 Wake, run, sleep

Wake (any source): claim the per-host S3 lease → restore the DB from the bucket (fresh computed as today) → seed if fresh and a seed is declared → mint the wake token → spawn live/<host> <args...> in the host's cgroup, output teed to the log sink → settle window instead of a health probe: the child is considered up when it either calls /v1/ready or survives sidecarSettle (10s) without exiting. A child that exits non-zero inside the settle window is a failed wake, released and backed off like a failed instance start.

The seam this needs. doWake's step 4 calls Backend.Start, and ExecBackend.Start ends with an unconditional waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout) (internal/activator/backend_exec.go:601) — there is no branch for a route with no socket. Handed Socket "" it would dial unix://"" every 150ms for the full 30s and then fail the wake. So the Backend seam gains one method rather than the existing one growing a mode flag:

// StartWorker spawns a child that binds nothing. No health probe is
// possible or wanted: readiness is the settle window, or the child's own
// call to /v1/ready over the agent socket.
StartWorker(ctx context.Context, rt registry.Route, tok string) (Proc, string, error)

ExecBackend.StartWorker is Start's body with the health wait and the socket-mode normalization removed and args in place of --socket/--db — which is very nearly what StartSidecar already is (backend_exec.go:746-773), the one spawn path in the package that deliberately has no health wait. UnitBackend.StartWorker refuses, exactly as it refuses StartSidecar today.

Run: supervised. A crash restarts with the existing exponential backoff (2s → 1m, reset after a healthy minute) — the constants the co-spawn path already uses, now applied to a standalone member.

A hibernating worker's exit is read by exit status, the only thing that separates the two meanings it can carry (final review, 2026-08-17): a clean exit is "my work is done" and parks; a non-zero exit is a crash and restarts with that same backoff, exactly as an always-on worker's does. Without the distinction, a bot that fell over before setting a next-due slept forever — nothing can dial one.

A fired due time is consumed by the wake it causes, never by the sleep that follows (final review, 2026-08-17): a worker's stop writes no next_due at all, so otherwise the passed value is re-served on every sweep and the sidecar wakes, idles, parks and re-wakes forever. The consumption is conditional on the row still holding the value that fired, so a next-due the child posted during its own wake survives — and a worker that sleeps BEFORE its timer fires keeps its schedule untouched.

Sleep, triggered by /v1/idle, by max_run, or by the idle timer (no /v1/heartbeat and no agent-socket call within idle): SIGTERM (5s grace, then SIGKILL) → record next_due from whatever the sidecar last declared → park the DB to the bucket → release the lease. The ordering is the instance stop path's, minus the next-due probe: a sidecar's next-due arrived over the socket while it was alive, which is strictly better than asking a process we are about to kill.

Always-on is hibernate: false: the record exists ⇒ the process runs, same supervision, no idle clock, no lease churn. Two consequences of that rule, both from the final review (2026-08-17): the sweep starts an always-on worker that has no live entry — otherwise one created while the agent was running never ran at all, since the boot pass runs once, at arming — and max_run applies to it too (it is the wedged-reconnect-loop guard, and a long-lived worker is exactly that shape), but is spent there as a RECYCLE of its child on the same entry, keeping the lease, because stopping an always-on worker leaves nothing to start it again.

7.3 Where this lives: inside the activator, off its request path

A sidecar's lifecycle is ~80% of doWake/stop already: acquire the lease, restore, seed, spawn, replicate; then stop, park, release. Reimplementing that in a sibling package would duplicate exactly the code where bugs are expensive — the error paths that must release a lease. So sidecars reuse activator.Service and its actEntry, with the request path left alone:

  • Nothing routes to a sidecar, so beginRequest, BeginSidecarRequest and ServeHTTP never see one. The edge refuses the row first (§6.1), and ServeHTTP already gates on rt.Kind == "instance" && rt.Hibernate (activator.go:1128), so it is unreachable there twice over. A new WakeWorker(host) entry point creates the entry and runs the same wake/doWake machinery the sweeper's request-less due-time wake uses.
  • The idle clock is reused verbatim, with a different input. For a sidecar inflight is always 0 and lastActive is stamped by agent socket contact instead of by a released HTTP request. The existing predicate — inflight == 0 && now-lastActive >= idle (activator.go:927) — then means exactly the right thing with no new branch, and /v1/idle is a stop requested immediately rather than a timestamp nudged backwards.
  • stop skips only the next-due probe (activator.go:820-845): a sidecar's next-due arrived over the socket while it was alive, which is strictly better than asking a process we are about to kill. Everything after it — stop, final checkpoint, lease release — is shared unchanged.
  • max_run is one more sweep predicate beside the idle one.

The co-spawn fields on actEntry (sidecar, sidecarWanted, …) are untouched: a standalone sidecar is the entry's own proc, not a passenger on someone else's.

7.4 Wake sources built here

  1. Timer — the sweeper's existing due-time wake, unchanged: a sleeping host whose next_due has arrived is woken with no client request. The machinery already exists for instances; a sidecar is simply a host that only ever wakes this way.
  2. Operator/app command — carlos sidecars wake, pull-based: the console bumps the wake generation (§4.3) and the serial; every box serving that app notices within seconds (the carlos restart pattern verbatim) and wakes the named sidecar. Nothing is pushed at a box — which is what lets the box live in a private subnet.

The box's etag memory is seeded once, at arming, and that seeding is what keeps an agent restart from replaying every wake in history. It is scoped to arming precisely (final review, 2026-08-17): a sidecar route seen for the first time AFTER that baseline has just been minted, and a wake object already standing at it is a wake somebody asked for and the CLI already reported as accepted — the create-then-wake flow, which is the feature's primary one, since an operator wake is a hibernating sidecar's only first-run mechanism. A spurious extra wake is idempotent and cheap; a lost one is a bot that never runs.

  1. Boot — a non-hibernating sidecar starts with the agent.

7.5 Delivery wake — designed, not built

The spec's third wake source is the edge observing (metadata only) that ciphertext is queued for a sidecar's identity and signalling the host agent. It is out of scope here and needs: an identity→sidecar mapping the edge can consult without learning content, a signalling path to a box that may have no inbound port (so: the same pull-based generation object as §7.4, written by the edge instead of the console), and a rate discipline so a chatty correspondent cannot bill a member for continuous wakes. Recorded so the shape is not invented differently later.

8. Placement: boxes that accept no-inbound work

Two halves that must agree.

The box declares. A new env var, read where CARLOS_REGION/ CARLOS_POOLS are read (internal/host/agent.go:1317):

CARLOS_INBOUND=none # this box serves sidecar work only

Unset/any is today's behaviour exactly — an inert default, the apps-domain-style gate this repo already uses. On a none box:

  • the provisioning reconciler refuses to mint routed instance rows, loudly. A box that cannot serve must never be handed something to serve; this is the guard that turns a misfiled record into a log line instead of a silent outage.
  • the edge and cert machinery are not required to be running at all.

The record asks. placement: "no-inbound" on the manifest/record makes the sidecar reconciler mint only on boxes declaring CARLOS_INBOUND=none. placement: "any" (default) places by pool/region alone. Where the two disagree, the most restrictive wins (final review, 2026-08-17): a record may narrow an app that asks for nothing, but a record stating "any" under a no-inbound manifest does not widen it — that is the direction the console door already refuses, and a direct bucket write must not reach around it.

The two are independent on purpose: none is a property of the box's network, no-inbound is a requirement of the workload. A member who wants "this bot must only ever run somewhere with no inbound path" gets a mechanical guarantee rather than an operator's promise.

Infra is written, not applied (Paul's ruling): a runbook for a private-subnet box — no IGW route, egress via NAT, no inbound SG rules, no SSM — lands beside infra/eleven-fleet.md for Paul's own applies. Note that fleet boxes are already outbound-only dial (infra/eleven-fleet.md), so Eleven's first sidecar box needs no new network shape at all: the private-subnet variant is the harder posture, available when wanted.

9. The rest of the platform

9.1 Config delivery

Sidecars get their env files the way instances do — execRouteBase (internal/appconfig/syncexec.go:404) widens to include Kind == "sidecar". This is the direct replacement for the hand-staged <host>-sidecar.env: carlos env set reaches a sidecar, secrets are sealed and delivered the same way, and the 2s serial-driven convergence applies unchanged.

9.2 Version visibility without a header

carlos deploy verifies a roll by reading X-Carlos-Version off the URL. A sidecar has no URL. The substitute is the /v1/ready call: the sidecar reports the version it is running, the agent records it on the row, and carlos sidecars ls shows running version against desired channel version. A sidecar that never calls /v1/ready shows its version as unknown rather than as current — the honesty rule this repo has paid for: a stamp that overclaims is worse than an absent one.

9.3 Logs, metrics, restart

Correct for free, because they key on host or on the cgroup name rather than on Kind: the log sink names its file by host; internal/cgroup's carlos-exec@<host> convention means listExecCgroups (internal/host/metrics_probes.go:240-261) samples a sidecar's CPU and memory the moment it spawns into that cgroup, with no code change; and ops.FleetDBs's claimed map (internal/ops/litestream.go:371-379) is Kind-agnostic, so a sidecar's DBPath is replicated like any other.

Needs widening: carlos restart -app X is gated on release.Adoptable (internal/release/adopt.go:518-521, via internal/host/restart.go:77,192,545), which requires Kind == "instance". Sidecars are cycled by the restart pass too, so restart.go widens.

release.Adoptable itself does not change. It is the most reused predicate in the repo — adopt membership, capability convergence, socket and backing advisories, status pages all inherit it, and every one of those is correct to exclude a sidecar. Widening it centrally would silently enrol sidecars in capability drop-ins and adopt bookkeeping that make no sense for them. Each consumer that genuinely wants sidecars opts in by name: the config-delivery predicate (§9.1) and the restart pass. That is two deliberate edits instead of one blind one.

9.4 Consumers that need a guard, and why

The exhaustive list, from a sweep of every .Kind branch and every route iteration in the tree. Each is a test in this branch.

| Site | Today with a sidecar row | Change | |---|---|---| | edge.go:364-445 dispatch | unroutable by omission (default: continue) | explicit Kind refusal before the switch (§6.1) | | fleetbind.go:301-316 publish | advertises a sidecar as served | skip Kind == "sidecar" (§6.2a) | | provision/reconcile.go:651-673 removal | deletes every sidecar row, every pass | see below | | registry.go:1341 DueInstances | timer wakes never fire | kind IN ('instance','sidecar') (§5) | | appconfig/syncexec.go:404 | no env file ever written | include sidecars (§9.1) | | host/restart.go (via Adoptable) | never cycled | include sidecars (§9.3) | | console/instances.go:88-97 | can pick a sidecar as an app's reporting host, making a healthy app read "not reporting" | filter by Kind | | status/snapshot.go:406-427 CollectBox | falls to default: → counted as a public Resident instance | explicit branch; sidecars are not public | | console/overview.go:286-306 healthWord | renders "Serving", which is false | a sidecar's own words (§9.5) | | console/overview.go:129-144 | InstanceTotal counts sidecars | count separately | | cli.go:75,194-199 carlos add -kind | no validation at all — accepts -kind sidecar today | rail it off (§3) |

The removal-phase collision is the sharpest of these. The instance reconciler's removal phase (reconcile.go:651-673) is scoped by Provisioned, region, pool and absence-from-records — with no Kind check. Sidecar rows are Provisioned (§5) and can never appear in the instance record set, so the existing pass would delete every sidecar on every tick: a silent, self-healing-looking loop where sidecars are minted and destroyed forever, with no error anywhere.

The fix is that each pass's removal authority is scoped to its own Kind: the instance pass skips Kind == "sidecar", the sidecar pass touches only sidecar rows. Both halves are asserted directly, because this is the failure mode with no symptom other than work quietly disappearing.

9.5 What a sidecar's status reads

A sidecar never serves a request, so the vocabulary the console uses for routes is wrong for it. It gets its own words — Running, Asleep, Waking, Crashed, Never started — and is counted in its own bucket rather than folded into an instance count. Sidecars are not on the public status page: their existence is account-visible, not public.

9.6 Object storage, ledgers, economics

A sidecar is an ordinary tenant for the object-store primitive and accrues usage like any exec child. No new meters.

10. The existing co-spawn stays

HasSidecar/StartSidecar on the file-staged path are not removed. Eleven runs on that shape today, and this branch does not touch a live box. The relationship is stated in code and docs: the file path is legacy, the record path is native, and a host may not have both (the reconciler refuses to mint a sidecar row for a host that already has a staged <host>-sidecar.env, naming the conflict). Retiring the file path is a follow-up issue, gated on Eleven's own migration.

11. Console API and CLI

Verbs under registerCLIRoutes, bearer, JSON, mirroring the instances family including its refusal ordering (principal gate → rate limit → manifest present → name shape → region shape/allow-list → channel → the create-only write) and its byte-identical-404 discipline:

GET /api/cli/apps/{account}/{app}/sidecars GET /api/cli/apps/{account}/{app}/sidecars/{name} POST /api/cli/apps/{account}/{app}/sidecars DELETE /api/cli/apps/{account}/{app}/sidecars/{name} POST /api/cli/apps/{account}/{app}/sidecars/{name}/wake

The app-held token kind that authorizes an app's own instance verbs extends to its own sidecar verbs — this is the "apps can provision sidecar instances easily" requirement: Eleven's app process, holding no AWS credentials and no box access, creates a bot per enrolled member.

CLI, the instances family's shape:

carlos sidecars enable -app eleven -args 'sidecar run' [-placement no-inbound] carlos sidecars create -app eleven bot-acme carlos sidecars ls -app eleven carlos sidecars wake -app eleven bot-acme carlos sidecars rm -app eleven bot-acme carlos sidecars logs -app eleven bot-acme

Every one of them completes its job with no box access and verifies its own result, per the no-infra rule.

12. Testing

  • Unreachability is a test, not a claim: a Kind == "sidecar" row is never servable by the edge, never selected as an upstream, never aliased, never attachable to a domain, never on the status page — asserted directly, including the adversarial case of a sidecar row carrying a socket.
  • Reserved suffix: every door that mints or accepts a host refuses .sidecar.internal.
  • Lifecycle: wake → ready → next-due → idle → sleep → timer wake, on the activator's existing fake clock/fake backend harness; crash-backoff; settle-window failure; max_run; lease release on every exit path.
  • Agent socket: token scoping (a token for host A cannot speak about host B), stale-token refusal after a stop, malformed input, and the socket's absence being survivable.
  • Reconciler: mint/converge/remove with the fail-safe (a read failure skips the removal phase), placement matching, and the co-spawn conflict refusal.
  • Additive parse: a route row and a bucket record written before this change still parse.
  • The gate: go build ./..., go vet ./..., go test ./... -count=1, gofmt -l clean, plus the GOOS=darwin compile.

13. Explicitly out of scope

  • Delivery-wake (§7.5).
  • Retiring the file-staged co-spawn (§10).
  • Applying any infra (§8) or touching any live box.
  • Sidecars for unit-backed tenants — UnitBackend refuses agent-managed sidecars today and continues to; the capabilities spec already names this a non-goal.
  • Multi-box placement/failover for a sidecar: one record, one lease, one box at a time, exactly as instances work.

14. Amended 2026-09-05 — the sweep, redone against 600 commits of main

§9.4's consumer sweep was exhaustive when it was written, against main at d7077d1. This branch then sat unmerged from 2026-08-18 while main moved 600 commits, and the sweep was re-run on the merge. It is recorded here rather than folded into §9.4 because the lesson is the interesting part: the consumer list is not a fact about the tree, it is a fact about the tree ON A DATE. Two of the three new consumers were built by people who had no reason to know an unroutable serving class was coming.

14.1 The edges file publishes every served row, and that is now wrong

internal/host/edgesfile.go was added to main after this branch. Every box publishes control/edges/<box>.json naming the hosts it homes, and its own comment states the rule it was built to: every SERVED row on this box, ALL kinds — the serving-class split belongs to the reader, not to this writer.

That rule was coherent when every kind was servable. A sidecar row is Status: "ready" — that is what makes it runnable — so it passed the only filter and was published as role: "home", which is the assertion this box serves that host. Three consumers act on it:

  • edge.backhaul (internal/edge/edge.go, via homeMap). This is the sharp one, and it is sharp because the guarantee this design leans on cannot reach it. §6.1's edge refusal lives in Reload, which builds THIS box's routing map out of rows THIS box holds. A remote edge holds no sidecar row and never sees a Kind at all. It misses on its own upstream lookup, reads this file, and dials the home box with the sidecar's own name as SNI — a live proxy into the one member the platform guarantees nothing can dial.
  • deploymentHost = Allowed(host) || homes.KnownHost(host), which is what admits a name to the cert machinery. A published sidecar becomes orderable deployment-wide.
  • edgesfile.DeploymentRoutes, which feeds the alias pass's "routes always win" rule fleet-wide.

The guard therefore belongs to the WRITER, exactly as §6.2a puts publishResidentFleetBindings' there: a sidecar is never advertised as served, by this box or any other. Pinned by TestEdgesFileNeverPublishesASidecar, which fails with the guard removed — the row comes back {Role: home}.

14.2 Path-based routing mints into a third table

ReconcileAliases derived two tables when §3's reserved-suffix guard was written. Main added route_paths, from the same claims, through a branch the guard never sees. Both ends are now refused, and they are not the same finding twice:

  • the host end is the alias guard's argument verbatim: a path row keyed on a reserved-suffix name puts that name in the routing map, and edge.go's Kind check never sees it, because the row it reads is a path row.
  • the target end has nothing else standing in front of it. The cross-tenant check passes a sidecar, because a sidecar belongs to the claim's own account — that is the ordinary case, not an edge one. The readiness check passes it, because a live sidecar row is ready. So the target field is the one place in this pass where an account can route real traffic into its own undialable member, and only a check on the name closes it. Removing the guard mints bot -> bot.eleven.bdm.sidecar.internal.

14.3 Two gaps recorded, deliberately not closed here

Both are in the safe direction — a sidecar is left alone rather than acted on wrongly — and both are follow-ups rather than merge work.

  • Narrowed restarts reach no sidecar. carlos restart --host (main, after this branch) resolves hosts through scopedHosts, which gates on release.Adoptable, which is Kind == "instance" and stays that way for §9.3's reason. So a narrowed restart naming a sidecar cycles nothing. That is the right failure — a scoped bounce with no wake beside it would strand a worker, which is precisely what cycleWorkers exists to prevent — but reaching a sidecar by name wants a selector of its own.
  • A trashed app's sidecar rows are not retired. #340 gave the instance pass a second answer off the trash marker (removable, guarded by the production flag) and moved retirement onto every box. The sidecar pass keeps the mint-only predicate: retirement is not a question it can answer without inheriting an authority its removal phase was not designed around. A trashed app's sidecars therefore linger until their records go — the pre-#340 behaviour for everything.

14.4 Checked and correct, for the record

internal/schedule gates on Kind != "instance". internal/steering selects on the Steering column, which no sidecar record can set. movespass is instance-only. status/fleet.go filters by kind. Metrics and the economics pass attribute a sidecar's cgroup like any other exec child, which is §9.6's stated intent, not an oversight.

One vocabulary inconsistency is left standing: carlos status --instances matches units to routes through routeForUnit, which matches a sidecar's carlos-exec@<host> cgroup, so a running sidecar is listed there under "instances" rather than in the separate bucket §9.5 gives it. The door is authenticated and account-scoped, so nothing leaks; it is a naming fix, and changing the shape of a status API is not merge work.

15. Amended 2026-09-15 — the sweep, redone AGAIN, 405 commits later

§14 was written against main at 66587fa8 on 2026-09-05 and was exhaustive then. Main moved 405 commits in the ten days that followed, and the branch went conflicted rather than merely stale a second time. The sweep was rerun from scratch, on the principle §14 had already arrived at: a consumer list is not a fact about the tree, it is a fact about the tree ON A DATE, and the only way to keep one true is to stop trusting it and look again.

One new consumer reaches a sidecar. It is worth reading the shape of it, because it is the THIRD instance of one pattern and that pattern is now the thing to watch for rather than any of its three instances.

15.1 The domain-attach door aliases onto a sidecar

internal/console/domainattach.go's readyRouteHost picks the route a newly-attached custom domain is aliased onto. Its own doc says the selection is domains.firstReadyUpstream's "minus the upstream copy" — and that is accurate about the intent and wrong about the consequence.

firstReadyUpstream ends with a clause this one drops:

if rt.Socket != "" || rt.Addr != "" || (rt.Kind == "static" && rt.Channel != "")

That clause is doing two jobs. It selects a row with something to copy — the job it was written for, and the job readyRouteHost genuinely does not need, since it copies nothing and returns only the host. It is ALSO the question "is this row dialable at all", and that job nobody assigned it is the entire reason a sidecar never appears in firstReadyUpstream's answer: a sidecar is precisely the row with no socket and no addr.

So dropping the clause dropped the guard with it, and every test that remains passes a sidecar. It belongs to the app. It belongs to the account. It is not Platform. It is ready — because ready is what makes it runnable. Selection is in reg.List's host order, so bot.<app>... sorts ahead of most hosts an app would really serve from: the sidecar is not a straggler that wins a tie, it is the first candidate.

The result is not a routing breach — ReconcileAliases refuses a claim whose Route carries the reserved suffix (the guard at aliases.go's alias branch, §14's sibling). It is worse in a quieter way: the member's attach returns 200, the claim is written naming a sidecar, and the box then refuses that claim on every pass, forever. A custom domain that never serves and nothing anywhere says why.

Guard added at the door, so the console agrees with the box. Verified by neutralising it: the attach aliases echo.example.com onto bot.story.bab.sidecar.internal.

15.2 The pattern, which is the actual finding

Three sweeps have now found the same thing three times:

  • §14.1, the edges file: a writer that published "every served row, ALL kinds", true only while every kind was servable.
  • §14.2, path routing: a third table derived from the same claims, through a branch the reserved-suffix guard never saw.
  • §15.1, domain attach: a selection copied from a guarded one, minus the clause that was the guard.

None of the three was written carelessly and none is a sidecar bug. Each is code written before an unroutable serving class existed, by someone with no reason to imagine one. The common shape is a predicate that EXCLUDED sidecars as a side effect of asking something else — "has an upstream to copy", "is a row this box serves" — being narrowed, copied or re-derived by someone who correctly identified the stated purpose and could not see the unstated one.

The practical consequence for whoever sweeps next: grep for registry.Route is not enough, and neither is grepping for Kind. What finds these is asking, of every consumer, "what in this predicate happens to require a socket or an address?" — and then checking whether that clause is load-bearing somewhere it was never documented to be.

15.3 Checked this pass and correct

  • internal/gate (new): the password curtain is applied at the edge, in front of a route serving requests. The edge refuses a sidecar on Kind before dispatch, so a gate hash on a sidecar row is inert.
  • apexHolder (internal/console/apex.go, new): reads reg.Get at the platform apex <name>.<sqid>.<domain>. A sidecar host is under .sidecar.internal, a different suffix entirely, so the lookup can never land on one.
  • edgesfile.DeploymentRoutes: reads the published edges files, which §14.1's writer-side guard has already emptied of sidecars. That guard covers this consumer too, and is the reason it needs none of its own.
  • warnExternalSockets (internal/host/socket_advice.go): explicit Kind != "instance".
  • versionsweep: pins a sidecar's Version/PrevVersion against staged reaping. Correct and wanted — a sidecar runs the app's binary and the build it is running must not be swept out from under it.
  • schedule's prune: marks a sidecar's host alive, so its schedule rows survive. Correct and load-bearing — a hibernating sidecar's schedule is its only way back up.
  • ops/litestream.FleetDBs: an always-on sidecar's database joins the box-wide replication set like any other always-on row, which is right; a hibernating one is excluded there because the activator replicates it per-wake.
  • None of the brand-new packages this pass (analytics, clamd, maintenance*, objstore, privatefile, publicrelease, relay, telemetry, usage) reads routes at all.

15.4 A gap this merge opened, recorded not closed

Isolated exec (§8.5/§8.6 of its own design) gave StartSidecar a layout dispatch: a nested box starts a sidecar as a unit through the control channel, a fenced one refuses, and only the default falls through to the child spawn. StartWorker — the NATIVE path, which is what a standalone Kind == "sidecar" row uses — has no such dispatch and always spawns a bare child.

On a nested-layout box a native sidecar would therefore come up outside the isolation every other child gets. It is not wrong today, because no box runs both native sidecars and the nested layout, but it is exactly the kind of agreement that stops being true quietly. The fix is StartWorker learning the same dispatch, and it wants its own review rather than a merge's.