amadan
spec: remote host-agent (#53) + Raspberry Pi instance host design
Two stacked PRs: C6 Step 3 machinery (tunnel backing via reverse HTTP/2, heartbeat metrics banded-on-publish, failover verification, off-cloud bootstrap), then Pi enablement as dev-platform region 'pi'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+239
−0
docs/superpowers/specs/2026-08-04-remote-host-pi-design.md+239 −0
diff --git a/docs/superpowers/specs/2026-08-04-remote-host-pi-design.md b/docs/superpowers/specs/2026-08-04-remote-host-pi-design.md| new file mode 100644 |
| index 000000000..5b06fb4f0 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-04-remote-host-pi-design.md |
| @@ -0,0 +1,239 @@ |
| +# Remote host-agent + Raspberry Pi instance host — design |
| + |
| +**Date:** 2026-08-04 |
| +**Charter:** [platform#53](https://github.com/carlosframework/platform/issues/53) |
| +(C6 Step 3 of `docs/superpowers/plans/2026-08-01-plan-c-tito-selfhosted.md`, |
| +PR #49) plus Raspberry Pi enablement on the dev platform. |
| +**Settled upstream, not relitigated here:** the control plane is the |
| +deployment bucket; a remote host is outbound-only (443 to S3 and the edge — |
| +no inbound port, no SSH path, no push channel); this is the existing |
| +`carlos` binary in host-agent role, not a new binary. |
| + |
| +## Goal |
| + |
| +Two deliverables, two stacked PRs: |
| + |
| +1. **PR 1 — remote host-agent machinery (#53 verbatim):** tunnel route |
| + backing on the edge, host metrics in heartbeats, placement/failover |
| + verification, off-cloud bootstrap. Deployment-agnostic. |
| +2. **PR 2 — Pi enablement:** a Raspberry Pi 4 (4 GB) joins the **dev |
| + platform** (Kass account, 640012953448) as instance host for region |
| + `pi`, hosting a demo fleet — proving "a CARLOS host is anything that |
| + can reach the deployment bucket" on a $60 board, and rehearsing |
| + exactly the machinery tito's `hq` office box needs. |
| + |
| +Demo blast radius stays off production; the pocket-Pi demo only needs |
| +outbound 443, which survives venue Wi-Fi. |
| + |
| +## PR 1 — remote host-agent machinery |
| + |
| +### Tunnel route backing |
| + |
| +- `registry.Route` gains an additive `Tunnel` column: the name of the |
| + reverse channel backing the route. Channel name = the owning host's |
| + box label. Wire/DB shape additive; routes written before this change |
| + parse unchanged. |
| +- The dispatch switch in `internal/edge/edge.go` `Reload()` (today: |
| + socket → addr → static → status) gains a fourth backing: |
| + `t.Tunnel != ""` → `tunnelProxy`, the same `httputil.ReverseProxy` |
| + shape as `tcpProxy` but with a RoundTripper bound to the named |
| + channel. Header discipline is inherited because it is the same proxy |
| + path: `SetXForwarded`, `scrubInboundPurgeHeader` inbound, |
| + `applyPurgeHeader` outbound. **A tunnel-backed host is exactly as |
| + untrusted as an addr-backed one.** |
| + |
| +### Transport: reverse HTTP/2 over a host-dialed TLS connection |
| + |
| +Decision (the in-repo judgment call #53 asks for, recorded): we port the |
| +*idea* of titogo's `internal/tunnel` — persistent authenticated dial-out |
| +— but not its websocket transport. |
| + |
| +- The host dials the edge's **existing 443 listener** — no new port — |
| + with an upgrade request to a reserved path carrying |
| + `Authorization: Bearer <token>` and its channel name. |
| +- The edge verifies the token (constant-time; the token is bound to the |
| + channel name it may claim — dev configures one `name=token` pair in |
| + the edge environment, hq can list several), hijacks the connection, |
| + and **inverts** it: the edge runs an `http2.Transport` client over |
| + the hijacked conn; the host runs an `http2.Server` whose handler is |
| + its own local edge routing. |
| +- HTTP/2 supplies stream multiplexing, flow control, and pings. |
| + A failed ping tears the channel down and flips its state to |
| + `reconnecting`. `golang.org/x/net` is already in the module graph; |
| + no websocket dependency is added. |
| +- New `internal/edge/tunnel.go`: the channel hub (name → live channel), |
| + registration handler, token check, lifecycle. |
| + |
| +Rejected: porting the websocket transport (vendors a dependency and |
| +hand-rolls multiplexing h2 gives us); a dedicated listener port (venue |
| +and office networks may only pass 443). |
| + |
| +### Host dial-out |
| + |
| +- New `tunnelLoop` goroutine in `RunAgent` (`internal/host/agent.go`), |
| + beside `statusLoop`/`configSyncLoop`; active only when |
| + `CARLOS_TUNNEL_URL` + `CARLOS_TUNNEL_TOKEN` are configured. |
| +- Jittered exponential backoff **capped low**: the acceptance test |
| + asserts a reconnect bound in seconds. hq's dual-WAN failover is a |
| + re-dial over the surviving link; the Pi's Wi-Fi blip is the same |
| + event. Reconnect latency is user-visible. |
| +- While a channel is down the edge answers its routes 502; the live |
| + edge cache keeps public pages serving through a blip. |
| + |
| +### Host metrics in heartbeats |
| + |
| +The heartbeat record is the existing fleet status Box snapshot |
| +(`status/boxes/<label>.json`, writer `internal/status/snapshot.go` |
| +`WriteBox`, driven by `statusLoop`). Decision: **extend Box with an |
| +optional additive `metrics` block; band or strip before anything |
| +public.** |
| + |
| +- New fields: memory total/available, load, disk free, resident and |
| + parked instance counts, current release + channel, tunnel/channel |
| + state (connected/reconnecting; egress link hint if knowable). |
| +- The public fleet merger (`internal/status/fleet.go` `BuildFleet` → |
| + `PublishFleet`) continues to publish **only the existing banded |
| + words**. Raw numbers never reach the publicly served `fleet.json` — |
| + the privacy banding in `CollectBox` was deliberate and survives. |
| +- Consumers of the raw block (consumer list = everything that PARSES |
| + the shape): the private, session-gated console fleet view renders raw |
| + metrics from an in-process unbanded merge; `carlos status` gains a |
| + live mode reading box snapshots from the bucket (operator |
| + credentials). Nothing else parses Box today (verified: edge status |
| + serving is byte-passthrough; activator and ops do not parse it). |
| +- **Additive:** Boxes written by old hosts must still parse — pinned by |
| + a back-compat test fixture (yesterday's shape parses today). |
| +- Staleness is the one existing rule: a Box older than 3× the publish |
| + period renders **unreachable** — never its last happy numbers — in |
| + both console fleet view and CLI live status. |
| + |
| +### Placement + failover: verify, don't build |
| + |
| +- No pool machinery. `Route.Pool`/`Route.Region` remain dormant |
| + columns. Placement stays IAM + topology: one region, one data |
| + bucket, one host that can reach it. |
| +- New tests prove the dark-then-returns sequence end to end: host goes |
| + dark (stale heartbeat + lease expiry) → instance wakes on a host |
| + with fallback eligibility from the regional bucket → the returning |
| + host **loses** the conditional-write race and does not reclaim. |
| +- No redirect verb exists or is added. |
| + |
| +### Off-cloud bootstrap |
| + |
| +- `carlos bootstrap -offcloud`: identical structural work (units, dirs, |
| + service user, timers — Raspberry Pi OS and the office box are both |
| + systemd Debian), but expects a **staged credentials file** (0600, in |
| + the credentials dir, owned by the service user) instead of an EC2 |
| + instance profile. The flag is explicit; no IMDS sniffing. |
| +- Credential scope: the deployment bucket + the host's own region data |
| + bucket and **nothing else** — wrong-region-fails-at-IAM holds |
| + off-cloud. IAM Roles Anywhere is documented as the hq-grade |
| + alternative to a long-lived key. |
| +- The existing IMDS public-IP probe is already gated off for non-EC2. |
| +- Deliverable includes the one-page **software-half host runbook** |
| + shape (the hq physical runbook lands later in |
| + `infra/tito/region-hq.md`). |
| + |
| +### Acceptance (tests, not narration) |
| + |
| +- E2E in-repo: edge + fake remote host over a real reverse channel; |
| + route resolves, request round-trips, response headers survive. |
| +- Kill the channel mid-flight → host re-dials → next request succeeds; |
| + measure and assert a reconnect bound. |
| +- Stale-heartbeat host renders unreachable; its instance wakes on a |
| + fallback host; the returning host loses the lease race. |
| +- Heartbeat parse test: yesterday's shape still parses today. |
| +- All 14 packages: `go build`, `go vet`, `go test -count=1`, `gofmt` |
| + clean on touched files; pinned production channel names in |
| + `internal/release/laddercheck_test.go` keep passing. |
| + |
| +Per CLAUDE.md delegation policy this is production-path work: **Opus |
| +reviews** the edge listener, auth, and lease/failover pieces. |
| + |
| +## PR 2 — Raspberry Pi enablement (dev platform) |
| + |
| +### Infra |
| + |
| +- New reusable tofu module `infra/modules/region-remote-host`: the |
| + region's data bucket + an off-cloud IAM principal scoped to exactly |
| + the deployment bucket and that region's bucket. Pi uses an IAM user |
| + + access key; the module shape leaves room for an IAM Roles Anywhere |
| + trust later. This is deliberately the module tito's hq Step 1 reuses. |
| +- The dev root module (platform-infrastructure checkout, like other |
| + roots) instantiates it as region `pi`. |
| + |
| +### Region + placement |
| + |
| +- Dev gets a `pi` region entry. Demo instances are created with |
| + residency `pi`; the Pi's activator — the only one pointed at that |
| + bucket — wakes them. |
| +- **No live fallback** (decision): Pi instances are Pi-resident, |
| + period. A dark Pi renders unreachable; it never silently migrates the |
| + demo to the cloud host mid-blip. Failover semantics are proven by |
| + PR 1's tests, not wired live here. |
| +- No tier field, no provisioner changes — that is tito's C6 Step 2. |
| + |
| +### The Pi itself |
| + |
| +Hardware: Raspberry Pi 4, 4 GB. Runbook `infra/dev/region-pi.md` |
| +(software half, mirroring the hq runbook shape): |
| + |
| +- **64-bit Raspberry Pi OS Lite is mandatory** — hosts pick artifacts |
| + by `runtime.GOOS + "-" + runtime.GOARCH`; a 32-bit OS resolves no |
| + artifact. |
| +- SSD over USB3 recommended; high-endurance SD acceptable for the demo. |
| + SQLite + WAL streaming is the I/O load and SD I/O is the honest |
| + constraint on a Pi 4. |
| +- Stage the `linux-arm64` binary + `host.env` + credentials file, run |
| + `carlos bootstrap -offcloud`, verify the dial-out. |
| +- **Zero build-pipeline work:** ship's default target is already |
| + `linux-arm64` and the fleet is arm64 Graviton; the Pi eats the same |
| + artifact. |
| + |
| +### Demo seeding + success criteria |
| + |
| +A small script creates the demo fleet via `carlos apps create` (the |
| +PR #52 console-API path) against dev: **≥5 resident + ≥50 parked** |
| +instances (hibernation makes parked ~free; resident count is bounded by |
| +per-app footprint in 4 GB). The demo proves: |
| + |
| +1. The Pi appears in the fleet view with real metrics and its channel |
| + state. |
| +2. Instances serve through the dev edge over the reverse channel with |
| + correct headers. |
| +3. A parked instance wakes on first request through the tunnel. |
| +4. Pull the Wi-Fi → the next request after re-dial succeeds within the |
| + asserted reconnect bound. |
| +5. Power the Pi off → it renders unreachable; power it on → it resumes |
| + serving. Nothing migrated. |
| + |
| +That is the viability claim demonstrated rather than asserted: *a |
| +CARLOS host is anything that can reach the deployment bucket.* |
| + |
| +## Error handling (both PRs) |
| + |
| +- Token mismatch → 401; the host backs off with a cap (no thundering |
| + re-dial) and logs. |
| +- Half-dead channel → h2 ping timeout closes it; the host re-dials. |
| +- Bucket unreachable from the host → heartbeats stop → unreachable in |
| + fleet after 3× period. |
| +- Channel down → 502 from the edge; edge cache covers public pages. |
| + |
| +## Sequencing |
| + |
| +1. PR 1 in its own worktree/branch/draft PR; Opus review on edge |
| + listener, auth, lease/failover. On merge: tick C6 Step 3 in Plan C |
| + (once PR #49's plan text is on main) and record divergences. |
| +2. PR 2 stacks on PR 1: tofu module, dev `pi` region, runbook, demo |
| + seeding. |
| +3. Website-feature-sync rule applies at merge: "run CARLOS on a |
| + Raspberry Pi" is website material. |
| + |
| +## Out of scope |
| + |
| +- C6 Steps 1, 2, 4–7 (tito hq bucket/IAM tofu, tier/provisioner, |
| + hardware, relocation verb, canary, default flip). |
| +- Pool/eligibility machinery (dormant columns stay dormant). |
| +- Live fallback for the Pi region. |
| +- Multi-arch CI (manual `make build GOARCH=...` remains the build |
| + path). |