| new file mode 100644 |
| index 000000000..fe9befe08 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-04-remote-host-agent.md |
| @@ -0,0 +1,1259 @@ |
| +# Remote Host-Agent (issue #53, C6 Step 3) Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** The existing `carlos` binary gains a remote host-agent role: tunnel-backed routes on the edge (reverse HTTP/2 over a host-dialed connection), raw host metrics in the heartbeat Box snapshot (banded before anything public), verified dark-then-returns lease semantics, and an off-cloud bootstrap path. |
| + |
| +**Architecture:** The host dials the edge's existing 443 listener on a reserved hostname, the connection inverts (edge = h2 client, host = h2 server), and tunnel-backed routes proxy over it via the same `httputil.ReverseProxy` discipline as addr-backed routes. Spec: `docs/superpowers/specs/2026-08-04-remote-host-pi-design.md`. Charter: issue #53; settled architecture is NOT relitigated (outbound-only, no new binary). |
| + |
| +**Tech Stack:** Go 1.26, `golang.org/x/net/http2` (already in module graph as indirect — promoting to direct needs no network), stdlib `httputil`, existing S3/lease/status machinery. |
| + |
| +## Global Constraints |
| + |
| +- Wire-shape changes are ADDITIVE: Box snapshots, registry rows, and route JSON written before this change must still parse (repo rule, learned three times). |
| +- Consumer list for any shape change = everything that PARSES the shape, not everything that calls the API. |
| +- All 14 packages: `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l` clean on touched files, before any commit lands. |
| +- Sandbox: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; disable the tool sandbox if the toolchain reports "read-only file system". |
| +- Production channel names pinned in `internal/release/laddercheck_test.go` must keep passing. |
| +- A tunnel-backed host is exactly as untrusted as an addr-backed one: `SetXForwarded` + `scrubInboundPurgeHeader` inbound, `applyPurgeHeader`/`purgeModifier` outbound, unchanged. |
| +- Raw metrics NEVER reach the publicly served `status/fleet.json` — the privacy banding in `CollectBox`/`BuildFleet` is a public commitment. |
| +- Per CLAUDE.md delegation policy: Opus implements/reviews the edge listener, auth, and lease/failover tasks (1, 2, 3, 4, 9, 10); Sonnet fits tasks 5–8; the reviewer for the security-adjacent tasks is Opus. |
| +- Work happens on the existing worktree branch `worktree-remote-host-agent`; draft PR when Task 1 lands. |
| + |
| +**Recorded divergences from issue #53 wording** (carry into the PR description and the C6 Step 3 tick): |
| +1. "current release + channel" per box: the Box heartbeat carries the running binary's release (`version.Build`). There is no box-level platform channel (channels are per-route; verified by grep — no `CARLOS_PLATFORM_CHANNEL` exists), so no channel field is invented. |
| +2. Resident/parked instance counts already exist on the Box (`RouteCounts.Resident`/`Hibernating`) — not duplicated into the new metrics block. |
| +3. Transport is reverse HTTP/2, not a websocket port of titogo's tunnel (the in-repo judgment call the issue delegates; rationale in the spec). |
| + |
| +--- |
| + |
| +### Task 1: Registry `Tunnel` column + `carlos route -tunnel` setter |
| + |
| +**Files:** |
| +- Modify: `internal/registry/registry.go` (Route struct ~line 30–103, migrate ALTER list ~line 163, scan ~line 353, Upsert ~line 460, narrow setters near `SetEnvironment`) |
| +- Modify: `cli.go` (runRoute — follow the existing `-environment` flag shape) |
| +- Test: `internal/registry/registry_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: existing `Route`, `Registry.Upsert`, `SetEnvironment` precedent. |
| +- Produces: `Route.Tunnel string` (channel name backing this route; `""` = not tunnel-backed), `(*Registry).SetTunnel(host, tunnel string) error`, CLI `carlos route -host <h> -tunnel <name|->`. Upsert PRESERVES a non-empty existing Tunnel (same rationale as Environment: full-row re-Upserts must not silently unbind). |
| + |
| +- [ ] **Step 1: Write the failing tests** (model on the existing Environment preservation tests in `registry_test.go` — read them first, mirror their harness): |
| + |
| +```go |
| +func TestTunnelColumnRoundTrips(t *testing.T) { |
| + r := testRegistry(t) // whatever helper the file already uses to open a temp registry |
| + must(t, r.Upsert(registry.Route{Host: "demo.example", Kind: "instance", Status: "ready"})) |
| + must(t, r.SetTunnel("demo.example", "pi-box")) |
| + rt := mustGet(t, r, "demo.example") |
| + if rt.Tunnel != "pi-box" { |
| + t.Fatalf("Tunnel = %q, want pi-box", rt.Tunnel) |
| + } |
| +} |
| + |
| +func TestUpsertPreservesTunnel(t *testing.T) { |
| + r := testRegistry(t) |
| + must(t, r.Upsert(registry.Route{Host: "demo.example", Kind: "instance", Status: "ready"})) |
| + must(t, r.SetTunnel("demo.example", "pi-box")) |
| + // A full-row re-Upsert with zero-value Tunnel must not unbind. |
| + must(t, r.Upsert(registry.Route{Host: "demo.example", Kind: "instance", Status: "ready"})) |
| + if rt := mustGet(t, r, "demo.example"); rt.Tunnel != "pi-box" { |
| + t.Fatalf("Upsert clobbered Tunnel: %q", rt.Tunnel) |
| + } |
| + // Clearing has exactly one door: SetTunnel(host, ""). |
| + must(t, r.SetTunnel("demo.example", "")) |
| + if rt := mustGet(t, r, "demo.example"); rt.Tunnel != "" { |
| + t.Fatalf("SetTunnel clear failed: %q", rt.Tunnel) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/registry/ -run Tunnel -count=1` → FAIL (`rt.Tunnel undefined`). |
| +- [ ] **Step 3: Implement.** Add `Tunnel string` to `Route` (doc comment: "name of the reverse channel backing this route at the edge; written only by SetTunnel / carlos route -tunnel; Upsert preserves non-empty values — see Environment's comment for why"). Add `` `ALTER TABLE routes ADD COLUMN tunnel TEXT NOT NULL DEFAULT ''` `` to the migrate ALTER list. Add `tunnel` to the SELECT column list + Scan targets + Upsert (INSERT column, and the preserve-non-empty UPDATE clause exactly as Environment does — copy its SQL shape). Add `SetTunnel` modeled byte-for-byte on `SetEnvironment`. |
| +- [ ] **Step 4: Run tests** — `go test ./internal/registry/ -count=1` → PASS (all, not just new). |
| +- [ ] **Step 5: CLI flag.** In `cli.go` runRoute, add `-tunnel` string flag following `-environment`'s pattern exactly (including `-` meaning clear and the platform/canary-owned-row refusal if `-environment` has one — mirror it). Manual check: `go run . route -h` shows the flag. |
| +- [ ] **Step 6: Full gate + commit** — `go build ./... && go vet ./... && go test ./... -count=1 && gofmt -l internal/registry/registry.go cli.go` (expect no output from gofmt). Commit: `git add -A && git commit -m "registry: additive Tunnel column, preserved on Upsert, set via carlos route -tunnel"`. |
| + |
| +--- |
| + |
| +### Task 2: Edge `TunnelHub` — authenticated reverse-channel registration |
| + |
| +**Files:** |
| +- Create: `internal/edge/tunnel.go` |
| +- Test: `internal/edge/tunnel_test.go` |
| +- Modify: `go.mod` (promote `golang.org/x/net` to direct require — `-mod=mod` does this automatically on build; no network needed, v0.55.0 is in the module cache) |
| + |
| +**Interfaces:** |
| +- Produces (consumed by Tasks 3, 4, 10): |
| + - `type TunnelHub struct{...}` with `NewTunnelHub(host string, clients map[string]string) *TunnelHub` (host = reserved hostname; clients = channel name → bearer token) |
| + - `(*TunnelHub).ServeHTTP(w, r)` — the registration endpoint (only ever invoked for the reserved hostname) |
| + - `(*TunnelHub).transportFor(name string) http.RoundTripper` — errors when the channel is down (ReverseProxy turns that into 502) |
| + - `(*TunnelHub).State(name string) string` — `"connected"` or `""` |
| + - `(*TunnelHub).Host() string` |
| + - Wire protocol: `GET /register`, headers `Authorization: Bearer <token>`, `Carlos-Tunnel-Name: <name>`, `Connection: Upgrade`, `Upgrade: carlos-tunnel/1` → `101 Switching Protocols` → connection inverts (edge h2 client / host h2 server). |
| + |
| +- [ ] **Step 1: Write the failing tests:** |
| + |
| +```go |
| +package edge |
| + |
| +import ( |
| + "net/http" |
| + "net/http/httptest" |
| + "testing" |
| +) |
| + |
| +func TestTunnelRegisterRejectsBadToken(t *testing.T) { |
| + hub := NewTunnelHub("tunnel.test", map[string]string{"pi-box": "sekrit"}) |
| + for name, hdr := range map[string]http.Header{ |
| + "no auth": {}, |
| + "wrong token": {"Authorization": {"Bearer nope"}, "Carlos-Tunnel-Name": {"pi-box"}}, |
| + "unknown name": {"Authorization": {"Bearer sekrit"}, "Carlos-Tunnel-Name": {"other"}}, |
| + // A valid token may only claim ITS OWN channel name. |
| + "cross-name": {"Authorization": {"Bearer sekrit"}, "Carlos-Tunnel-Name": {""}}, |
| + } { |
| + req := httptest.NewRequest("GET", "/register", nil) |
| + req.Header = hdr |
| + rec := httptest.NewRecorder() |
| + hub.ServeHTTP(rec, req) |
| + if rec.Code != http.StatusUnauthorized { |
| + t.Errorf("%s: got %d, want 401", name, rec.Code) |
| + } |
| + } |
| +} |
| + |
| +func TestTunnelRegisterRejectsUnknownPath(t *testing.T) { |
| + hub := NewTunnelHub("tunnel.test", map[string]string{"pi-box": "sekrit"}) |
| + req := httptest.NewRequest("GET", "/anything-else", nil) |
| + rec := httptest.NewRecorder() |
| + hub.ServeHTTP(rec, req) |
| + if rec.Code != http.StatusNotFound { |
| + t.Fatalf("got %d, want 404", rec.Code) |
| + } |
| +} |
| + |
| +func TestParseTunnelClients(t *testing.T) { |
| + got, err := parseTunnelClients("pi-box=sekrit,hq=other") |
| + if err != nil || got["pi-box"] != "sekrit" || got["hq"] != "other" { |
| + t.Fatalf("parse: %v %v", got, err) |
| + } |
| + if _, err := parseTunnelClients("malformed"); err == nil { |
| + t.Fatal("want error on entry without '='") |
| + } |
| +} |
| +``` |
| + |
| +(The happy-path 101 + round-trip is Task 10's e2e — it needs a real hijackable conn, which httptest.NewRecorder can't provide.) |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/edge/ -run Tunnel -count=1` → FAIL (undefined: NewTunnelHub). |
| +- [ ] **Step 3: Implement `internal/edge/tunnel.go`:** |
| + |
| +```go |
| +// Package-internal reverse-channel hub: remote hosts dial OUT to the edge's |
| +// existing 443 listener on a reserved hostname, authenticate with a bearer |
| +// token bound to their channel name, and the connection inverts — the edge |
| +// becomes an HTTP/2 client over the hijacked conn, the host an HTTP/2 server. |
| +// Nothing ever connects TO a remote host (issue #53's locked-down property); |
| +// this file is the edge half of that bargain. Transport judgment call |
| +// (spec 2026-08-04): reverse h2 rather than a websocket port of titogo's |
| +// tunnel — x/net/http2 already supplies mux, flow control, and pings. |
| +package edge |
| + |
| +import ( |
| + "crypto/subtle" |
| + "fmt" |
| + "log" |
| + "net" |
| + "net/http" |
| + "strings" |
| + "sync" |
| + "time" |
| + |
| + "golang.org/x/net/http2" |
| +) |
| + |
| +const ( |
| + tunnelRegisterPath = "/register" |
| + tunnelUpgradeProto = "carlos-tunnel/1" |
| + // tunnelPingEvery / tunnelPingTimeout bound how long a half-dead channel |
| + // can look connected: a failed or timed-out ping tears it down, the host |
| + // notices its conn die and re-dials (seconds — the bound Task 10 asserts). |
| + tunnelPingEvery = 15 * time.Second |
| + tunnelPingTimeout = 10 * time.Second |
| +) |
| + |
| +type tunnelConn struct { |
| + cc *http2.ClientConn |
| + raw net.Conn |
| +} |
| + |
| +type TunnelHub struct { |
| + host string // reserved hostname this hub answers on |
| + clients map[string]string // channel name -> bearer token |
| + mu sync.Mutex |
| + conns map[string]*tunnelConn |
| +} |
| + |
| +func NewTunnelHub(host string, clients map[string]string) *TunnelHub { |
| + return &TunnelHub{host: host, clients: clients, conns: make(map[string]*tunnelConn)} |
| +} |
| + |
| +func (h *TunnelHub) Host() string { return h.host } |
| + |
| +// parseTunnelClients parses CARLOS_TUNNEL_CLIENTS ("name=token[,name=token]"). |
| +func parseTunnelClients(s string) (map[string]string, error) { |
| + out := make(map[string]string) |
| + for _, pair := range strings.Split(s, ",") { |
| + pair = strings.TrimSpace(pair) |
| + if pair == "" { |
| + continue |
| + } |
| + name, token, ok := strings.Cut(pair, "=") |
| + if !ok || name == "" || token == "" { |
| + return nil, fmt.Errorf("tunnel clients: malformed entry %q (want name=token)", pair) |
| + } |
| + out[name] = token |
| + } |
| + return out, nil |
| +} |
| + |
| +// ServeHTTP handles ONLY the registration handshake; the Edge routes requests |
| +// here when r.Host equals the reserved hostname, so ordinary app traffic can |
| +// never reach it and this handler never sees ordinary traffic. |
| +func (h *TunnelHub) ServeHTTP(w http.ResponseWriter, r *http.Request) { |
| + if r.URL.Path != tunnelRegisterPath { |
| + http.NotFound(w, r) |
| + return |
| + } |
| + name := r.Header.Get("Carlos-Tunnel-Name") |
| + token := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ") |
| + want, ok := h.clients[name] |
| + // Constant-time compare; the name lookup happening first is fine — the |
| + // name is not the secret, the token is. A token is bound to exactly one |
| + // channel name, so a leaked token cannot impersonate a different host. |
| + if name == "" || !ok || subtle.ConstantTimeCompare([]byte(token), []byte(want)) != 1 { |
| + http.Error(w, "unauthorized", http.StatusUnauthorized) |
| + return |
| + } |
| + hj, ok := w.(http.Hijacker) |
| + if !ok { |
| + http.Error(w, "hijack unsupported", http.StatusInternalServerError) |
| + return |
| + } |
| + conn, bufrw, err := hj.Hijack() |
| + if err != nil { |
| + log.Printf("edge: tunnel %s: hijack: %v", name, err) |
| + return |
| + } |
| + // Manual 101 on the hijacked conn (the ResponseWriter is spent once hijacked). |
| + fmt.Fprintf(bufrw, "HTTP/1.1 101 Switching Protocols\r\nUpgrade: %s\r\nConnection: Upgrade\r\n\r\n", tunnelUpgradeProto) |
| + if err := bufrw.Flush(); err != nil { |
| + conn.Close() |
| + return |
| + } |
| + tr := &http2.Transport{} |
| + cc, err := tr.NewClientConn(conn) |
| + if err != nil { |
| + log.Printf("edge: tunnel %s: h2 client conn: %v", name, err) |
| + conn.Close() |
| + return |
| + } |
| + h.mu.Lock() |
| + if old := h.conns[name]; old != nil { |
| + old.raw.Close() // a re-register replaces the previous channel |
| + } |
| + h.conns[name] = &tunnelConn{cc: cc, raw: conn} |
| + h.mu.Unlock() |
| + log.Printf("edge: tunnel %s: connected from %s", name, conn.RemoteAddr()) |
| + go h.pingLoop(name, cc, conn) |
| +} |
| + |
| +// pingLoop is the edge-side liveness check: a dead ping closes the raw conn |
| +// and unregisters the channel, so requests fail fast to 502 instead of |
| +// hanging on a black-holed TCP session (the venue-Wi-Fi failure mode). |
| +func (h *TunnelHub) pingLoop(name string, cc *http2.ClientConn, raw net.Conn) { |
| + for { |
| + time.Sleep(tunnelPingEvery) |
| + ctx, cancel := context.WithTimeout(context.Background(), tunnelPingTimeout) |
| + err := cc.Ping(ctx) |
| + cancel() |
| + if err != nil { |
| + log.Printf("edge: tunnel %s: ping failed (%v) — dropping channel", name, err) |
| + raw.Close() |
| + h.mu.Lock() |
| + if cur := h.conns[name]; cur != nil && cur.raw == raw { |
| + delete(h.conns, name) |
| + } |
| + h.mu.Unlock() |
| + return |
| + } |
| + } |
| +} |
| + |
| +func (h *TunnelHub) conn(name string) *http2.ClientConn { |
| + h.mu.Lock() |
| + defer h.mu.Unlock() |
| + if c := h.conns[name]; c != nil { |
| + return c.cc |
| + } |
| + return nil |
| +} |
| + |
| +// State reports "connected" or "" — consumed by fleet status rendering. |
| +func (h *TunnelHub) State(name string) string { |
| + if h.conn(name) != nil { |
| + return "connected" |
| + } |
| + return "" |
| +} |
| + |
| +type roundTripperFunc func(*http.Request) (*http.Response, error) |
| + |
| +func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } |
| + |
| +// transportFor resolves the channel PER REQUEST, so a proxy built before a |
| +// reconnect uses the fresh channel afterwards with no reload needed. |
| +func (h *TunnelHub) transportFor(name string) http.RoundTripper { |
| + return roundTripperFunc(func(req *http.Request) (*http.Response, error) { |
| + cc := h.conn(name) |
| + if cc == nil { |
| + return nil, fmt.Errorf("tunnel %q not connected", name) |
| + } |
| + return cc.RoundTrip(req) |
| + }) |
| +} |
| +``` |
| + |
| +(Add `"context"` to imports. `go build ./internal/edge/` promotes `golang.org/x/net` to a direct require in go.mod under `-mod=mod` — commit the go.mod/go.sum change with this task.) |
| + |
| +- [ ] **Step 4: Run tests** — `go test ./internal/edge/ -run Tunnel -count=1` → PASS. |
| +- [ ] **Step 5: Full gate + commit** — all four commands clean. Commit: `edge: TunnelHub — authenticated reverse-channel registration (reverse h2 over hijacked 443)`. |
| + |
| +--- |
| + |
| +### Task 3: Edge dispatch — tunnel-backed routes + hub wiring |
| + |
| +**Files:** |
| +- Modify: `internal/edge/edge.go` (Edge struct ~line 77, `Allowed` ~line 282, `ServeHTTP` ~line 287, `Reload` switch ~line 234–248, proxy constructors ~line 760–803, and `RunWithActivator` for env wiring) |
| +- Test: `internal/edge/tunnel_test.go` (extend) |
| + |
| +**Interfaces:** |
| +- Consumes: `TunnelHub` (Task 2), `Route.Tunnel` (Task 1), `purgeModifier`, `scrubInboundPurgeHeader`. |
| +- Produces: `(*Edge).EnableTunnels(hub *TunnelHub)`; env contract `CARLOS_TUNNEL_HOST` (reserved hostname) + `CARLOS_TUNNEL_CLIENTS` (name=token list) parsed in `RunWithActivator`; ready-hook seam `RunWithActivatorReady(args, act, cache, ready func(*Edge))` (consumed by Task 4). |
| + |
| +- [ ] **Step 1: Write the failing test** (extend tunnel_test.go): |
| + |
| +```go |
| +// testRegistryWithRoute opens a temp registry and upserts one route. If |
| +// edge_test.go already has an equivalent helper, use that instead — one |
| +// harness, not two. |
| +func testRegistryWithRoute(t *testing.T, rt registry.Route) *registry.Registry { |
| + t.Helper() |
| + r, err := registry.Open(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { r.Close() }) |
| + if err := r.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + return r |
| +} |
| + |
| +func TestReloadBuildsTunnelBackedRoute(t *testing.T) { |
| + reg := testRegistryWithRoute(t, registry.Route{ |
| + Host: "demo.example", Kind: "instance", Status: "ready", Tunnel: "pi-box", |
| + }) |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + hub := NewTunnelHub("tunnel.test", map[string]string{"pi-box": "sekrit"}) |
| + e.EnableTunnels(hub) |
| + if err := e.Reload(); err != nil { |
| + t.Fatal(err) |
| + } |
| + // Without EnableTunnels the route must be skipped (default: continue); |
| + // with it, the host must be served and Allowed, and the reserved |
| + // hostname must be Allowed for cert minting. |
| + if !e.Allowed("demo.example") { |
| + t.Fatal("tunnel-backed route not served after EnableTunnels+Reload") |
| + } |
| + if !e.Allowed("tunnel.test") { |
| + t.Fatal("reserved tunnel hostname must be Allowed (autocert HostPolicy)") |
| + } |
| + // Channel down -> 502, not a hang. |
| + req := httptest.NewRequest("GET", "http://demo.example/", nil) |
| + req.Host = "demo.example" |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, req) |
| + if rec.Code != http.StatusBadGateway { |
| + t.Fatalf("disconnected tunnel: got %d, want 502", rec.Code) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/edge/ -run TestReloadBuildsTunnelBackedRoute -count=1` → FAIL (undefined: EnableTunnels). |
| +- [ ] **Step 3: Implement.** |
| + - Edge struct: add `tunnels *TunnelHub` field with doc comment ("nil = tunnel serving not configured; routes with Tunnel set are skipped, same nil-means-disabled convention as act/cache"). |
| + - `func (e *Edge) EnableTunnels(hub *TunnelHub) { e.tunnels = hub }` — call before Reload, same contract as EnableStatic. |
| + - `Allowed`: `if e.tunnels != nil && host == e.tunnels.Host() { return true }` before the map lookup. |
| + - `ServeHTTP`: immediately after the host split (~line 291), before the upstream lookup: `if e.tunnels != nil && host == e.tunnels.Host() { e.tunnels.ServeHTTP(w, r); return }`. |
| + - `Reload` switch — new case after `t.Addr != ""` (a tunnel route has no socket/addr, so order vs. those is moot, but keep it before the static/status kinds so an instance-kind tunnel route can't fall through): |
| + |
| +```go |
| +case t.Tunnel != "": |
| + if e.tunnels == nil { |
| + continue // tunnel serving not configured on this box |
| + } |
| + p := tunnelProxy(t.Tunnel, e.tunnels) |
| + p.ModifyResponse = e.purgeModifier(t) |
| + proxy = p |
| +``` |
| + |
| + - Proxy constructor beside tcpProxy: |
| + |
| +```go |
| +// tunnelProxy reverse-proxies over a named reverse channel (TunnelHub). |
| +// Same Host preservation and purge-header discipline as tcpProxy — a |
| +// tunnel-backed host is exactly as untrusted as an addr-backed one. The |
| +// scheme is https because the h2 ClientConn requires it; no dial happens |
| +// (the channel already exists), so the value is otherwise inert. |
| +func tunnelProxy(name string, hub *TunnelHub) *httputil.ReverseProxy { |
| + return &httputil.ReverseProxy{ |
| + Rewrite: func(pr *httputil.ProxyRequest) { |
| + pr.SetXForwarded() |
| + pr.Out.URL.Scheme = "https" |
| + pr.Out.URL.Host = pr.In.Host // h2 :authority = the real hostname |
| + pr.Out.Host = pr.In.Host |
| + scrubInboundPurgeHeader(pr) |
| + }, |
| + Transport: hub.transportFor(name), |
| + } |
| +} |
| +``` |
| + |
| + - `RunWithActivator` wiring: rename the body to `RunWithActivatorReady(args []string, act Activator, cache *Cache, ready func(*Edge))`; keep `RunWithActivator` as a wrapper passing `nil`. Inside, after the Edge is fully configured (after EnableCache/EnableStatic and before the blocking serve/listen), add: |
| + |
| +```go |
| +if th := os.Getenv("CARLOS_TUNNEL_HOST"); th != "" { |
| + clients, err := parseTunnelClients(os.Getenv("CARLOS_TUNNEL_CLIENTS")) |
| + if err != nil { |
| + log.Fatalf("edge: %v", err) |
| + } |
| + if len(clients) == 0 { |
| + log.Fatalf("edge: CARLOS_TUNNEL_HOST set but CARLOS_TUNNEL_CLIENTS empty") |
| + } |
| + e.EnableTunnels(NewTunnelHub(th, clients)) |
| + if err := e.Reload(); err != nil { |
| + log.Fatalf("edge: reload after EnableTunnels: %v", err) |
| + } |
| + log.Printf("edge: tunnel hub armed on %s (%d client(s))", th, len(clients)) |
| +} |
| +if ready != nil { |
| + ready(e) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests** — `go test ./internal/edge/ -count=1` → PASS (whole package — the Reload/ServeHTTP changes must not break socket/addr/static/status tests). |
| +- [ ] **Step 5: Full gate + commit** — `edge: tunnel-backed route dispatch + hub wiring on the existing 443 listener`. |
| + |
| +--- |
| + |
| +### Task 4: Host dial-out client (`tunnelLoop`) |
| + |
| +**Files:** |
| +- Create: `internal/host/tunnel_client.go` |
| +- Test: `internal/host/tunnel_client_test.go` |
| +- Modify: `internal/host/agent.go` (RunAgent ~line 146–157) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 3's wire protocol and `RunWithActivatorReady` seam. |
| +- Produces: `tunnelLoop(handler http.Handler)` goroutine; `tunnelState() string` (`""` disabled / `"connected"` / `"reconnecting"`) consumed by Task 6's heartbeat probe; env contract `CARLOS_TUNNEL_URL`, `CARLOS_TUNNEL_TOKEN`, `CARLOS_TUNNEL_NAME`. |
| + |
| +- [ ] **Step 1: Write the failing test** — dial + serve against a hand-rolled upgrade server (no Edge involvement; the full loop is Task 10): |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "bufio" |
| + "fmt" |
| + "io" |
| + "net" |
| + "net/http" |
| + "net/http/httptest" |
| + "testing" |
| + |
| + "golang.org/x/net/http2" |
| +) |
| + |
| +// fakeTunnelEdge accepts one registration, answers 101, then drives ONE h2 |
| +// request over the inverted conn and returns the response body it read. |
| +func fakeTunnelEdge(t *testing.T, wantToken, wantName string) (*httptest.Server, chan string) { |
| + t.Helper() |
| + got := make(chan string, 1) |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + if r.Header.Get("Authorization") != "Bearer "+wantToken || |
| + r.Header.Get("Carlos-Tunnel-Name") != wantName { |
| + http.Error(w, "unauthorized", http.StatusUnauthorized) |
| + return |
| + } |
| + conn, bufrw, err := w.(http.Hijacker).Hijack() |
| + if err != nil { |
| + t.Errorf("hijack: %v", err) |
| + return |
| + } |
| + fmt.Fprintf(bufrw, "HTTP/1.1 101 Switching Protocols\r\nUpgrade: carlos-tunnel/1\r\nConnection: Upgrade\r\n\r\n") |
| + bufrw.Flush() |
| + cc, err := (&http2.Transport{}).NewClientConn(conn) |
| + if err != nil { |
| + t.Errorf("h2 client: %v", err) |
| + return |
| + } |
| + req, _ := http.NewRequest("GET", "https://demo.example/hello", nil) |
| + resp, err := cc.RoundTrip(req) |
| + if err != nil { |
| + t.Errorf("roundtrip over tunnel: %v", err) |
| + got <- "" |
| + return |
| + } |
| + body, _ := io.ReadAll(resp.Body) |
| + got <- resp.Header.Get("X-From-Host") + ":" + string(body) |
| + })) |
| + return srv, got |
| +} |
| + |
| +func TestDialAndServeInvertsConnection(t *testing.T) { |
| + srv, got := fakeTunnelEdge(t, "sekrit", "pi-box") |
| + defer srv.Close() |
| + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("X-From-Host", r.Host) |
| + fmt.Fprint(w, "pong") |
| + }) |
| + conn, err := dialTunnel(srv.URL, "pi-box", "sekrit") |
| + if err != nil { |
| + t.Fatalf("dialTunnel: %v", err) |
| + } |
| + go serveTunnel(conn, handler) |
| + if v := <-got; v != "demo.example:pong" { |
| + t.Fatalf("over-tunnel response = %q, want demo.example:pong", v) |
| + } |
| +} |
| + |
| +func TestDialTunnelRejectsNon101(t *testing.T) { |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + http.Error(w, "unauthorized", http.StatusUnauthorized) |
| + })) |
| + defer srv.Close() |
| + if _, err := dialTunnel(srv.URL, "pi-box", "wrong"); err == nil { |
| + t.Fatal("want error on 401 handshake") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/host/ -run Tunnel -count=1` → FAIL (undefined: dialTunnel). |
| +- [ ] **Step 3: Implement `internal/host/tunnel_client.go`:** |
| + |
| +```go |
| +// The host half of the reverse channel (issue #53 / C6 Step 3): dial OUT to |
| +// the edge, invert the connection, and serve our own local edge routing over |
| +// HTTP/2. Reconnect fast — hq's dual-WAN failover and a Pi's Wi-Fi blip are |
| +// both "the conn died, dial again", and that latency is user-visible. |
| +package host |
| + |
| +import ( |
| + "bufio" |
| + "crypto/tls" |
| + "fmt" |
| + "io" |
| + "log" |
| + "math/rand/v2" |
| + "net" |
| + "net/http" |
| + "net/url" |
| + "os" |
| + "sync/atomic" |
| + "time" |
| + |
| + "golang.org/x/net/http2" |
| +) |
| + |
| +// tunnelStateVal is "" (disabled/never started), "connected", or |
| +// "reconnecting" — read by statusLoop's heartbeat probe (probes must fail |
| +// toward honest values; "" means the loop isn't running at all). |
| +var tunnelStateVal atomic.Value |
| + |
| +func tunnelState() string { |
| + if v, ok := tunnelStateVal.Load().(string); ok { |
| + return v |
| + } |
| + return "" |
| +} |
| + |
| +const ( |
| + tunnelBackoffFloor = 250 * time.Millisecond |
| + tunnelBackoffCap = 2 * time.Second |
| +) |
| + |
| +// tunnelLoop dials CARLOS_TUNNEL_URL and serves handler over the inverted |
| +// conn, forever, with jittered capped backoff. Inert unless all three env |
| +// vars are set — the same convention as every other optional agent loop. |
| +func tunnelLoop(handler http.Handler) { |
| + rawURL := os.Getenv("CARLOS_TUNNEL_URL") |
| + token := os.Getenv("CARLOS_TUNNEL_TOKEN") |
| + name := os.Getenv("CARLOS_TUNNEL_NAME") |
| + if rawURL == "" || token == "" || name == "" { |
| + if rawURL != "" || token != "" || name != "" { |
| + log.Printf("agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME") |
| + } |
| + return |
| + } |
| + log.Printf("agent: tunnel loop armed (%s as %q)", rawURL, name) |
| + backoff := tunnelBackoffFloor |
| + for { |
| + conn, err := dialTunnel(rawURL, name, token) |
| + if err != nil { |
| + tunnelStateVal.Store("reconnecting") |
| + log.Printf("agent: tunnel dial: %v (retry in %s)", err, backoff) |
| + time.Sleep(backoff + rand.N(backoff/2)) |
| + if backoff *= 2; backoff > tunnelBackoffCap { |
| + backoff = tunnelBackoffCap |
| + } |
| + continue |
| + } |
| + backoff = tunnelBackoffFloor |
| + tunnelStateVal.Store("connected") |
| + log.Printf("agent: tunnel connected as %q", name) |
| + serveTunnel(conn, handler) // blocks until the conn dies |
| + tunnelStateVal.Store("reconnecting") |
| + log.Printf("agent: tunnel lost — re-dialing") |
| + } |
| +} |
| + |
| +// bufferedConn replays bytes the handshake's bufio.Reader over-read (the h2 |
| +// preface can arrive in the same packet as the 101) before the raw conn. |
| +type bufferedConn struct { |
| + r io.Reader |
| + net.Conn |
| +} |
| + |
| +func (b bufferedConn) Read(p []byte) (int, error) { return b.r.Read(p) } |
| + |
| +// dialTunnel performs the registration handshake and returns the raw conn, |
| +// ready to serve h2 on. https URLs use real TLS (the edge's public cert — |
| +// server auth by PKI, client auth by bearer token); http is for tests. |
| +func dialTunnel(rawURL, name, token string) (net.Conn, error) { |
| + u, err := url.Parse(rawURL) |
| + if err != nil { |
| + return nil, fmt.Errorf("tunnel url: %w", err) |
| + } |
| + addr := u.Host |
| + var conn net.Conn |
| + switch u.Scheme { |
| + case "https": |
| + if u.Port() == "" { |
| + addr = net.JoinHostPort(u.Hostname(), "443") |
| + } |
| + conn, err = tls.Dial("tcp", addr, &tls.Config{ |
| + ServerName: u.Hostname(), |
| + NextProtos: []string{"http/1.1"}, // the handshake is 1.1; h2 starts after the 101 |
| + }) |
| + case "http": |
| + conn, err = net.Dial("tcp", addr) |
| + default: |
| + return nil, fmt.Errorf("tunnel url: unsupported scheme %q", u.Scheme) |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + req := &http.Request{ |
| + Method: "GET", |
| + URL: &url.URL{Path: "/register"}, |
| + Host: u.Hostname(), |
| + Header: http.Header{ |
| + "Authorization": {"Bearer " + token}, |
| + "Carlos-Tunnel-Name": {name}, |
| + "Connection": {"Upgrade"}, |
| + "Upgrade": {"carlos-tunnel/1"}, |
| + }, |
| + } |
| + if err := req.Write(conn); err != nil { |
| + conn.Close() |
| + return nil, err |
| + } |
| + br := bufio.NewReader(conn) |
| + resp, err := http.ReadResponse(br, req) |
| + if err != nil { |
| + conn.Close() |
| + return nil, err |
| + } |
| + resp.Body.Close() |
| + if resp.StatusCode != http.StatusSwitchingProtocols { |
| + conn.Close() |
| + return nil, fmt.Errorf("tunnel register: %s", resp.Status) |
| + } |
| + if br.Buffered() > 0 { |
| + return bufferedConn{r: io.MultiReader(br, conn), Conn: conn}, nil |
| + } |
| + return conn, nil |
| +} |
| + |
| +// serveTunnel serves handler over the inverted conn until it dies. |
| +func serveTunnel(conn net.Conn, handler http.Handler) { |
| + (&http2.Server{}).ServeConn(conn, &http2.ServeConnOpts{Handler: handler}) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests** — `go test ./internal/host/ -run Tunnel -count=1` → PASS. |
| +- [ ] **Step 5: Wire into RunAgent.** In `internal/host/agent.go` replace the final `edge.RunWithActivator(args, act, cache)` call with: |
| + |
| +```go |
| +edge.RunWithActivatorReady(args, act, cache, func(e *edge.Edge) { |
| + go tunnelLoop(e) |
| +}) |
| +``` |
| + |
| +(`*edge.Edge` implements `http.Handler`; the tunnel serves the host's own local routing, so a request arriving over the channel takes exactly the path a local request would — activator wake included.) |
| + |
| +- [ ] **Step 6: Full gate + commit** — `host: tunnel dial-out loop — reverse channel client with capped jittered backoff`. |
| + |
| +--- |
| + |
| +### Task 5: Status shape — additive `Metrics` block on Box + back-compat + dry-run leak scope |
| + |
| +**Files:** |
| +- Modify: `internal/status/snapshot.go` (package doc lines 1–7, `Probes` ~line 22, `Box` ~line 72, `CollectBox` ~line 184) |
| +- Modify: `internal/status/dryrun.go` (read it FIRST — it publishes a fixture fleet and asserts no fixture leaks into published objects) |
| +- Test: `internal/status/snapshot_test.go` (or the file the package's existing CollectBox tests live in — read and extend, don't create a parallel file) |
| + |
| +**Interfaces:** |
| +- Consumes: nothing new. |
| +- Produces (consumed by Tasks 6, 7): |
| + |
| +```go |
| +// BoxMetrics is the raw, PRIVATE block: real numbers, never banded. It lives |
| +// in status/boxes/<label>.json — which only fleet-status machinery and |
| +// operators read — and is deliberately NOT copied into FleetBox, so the |
| +// publicly served fleet.json keeps the banded-words-only commitment. |
| +type BoxMetrics struct { |
| + MemTotalBytes *int64 `json:"mem_total_bytes,omitempty"` |
| + MemAvailBytes *int64 `json:"mem_avail_bytes,omitempty"` |
| + Load1 *float64 `json:"load1,omitempty"` |
| + DiskFreeBytes *int64 `json:"disk_free_bytes,omitempty"` |
| + Release string `json:"release,omitempty"` // running binary's version.Build |
| + Tunnel string `json:"tunnel,omitempty"` // ""|"connected"|"reconnecting" |
| +} |
| +``` |
| + |
| + plus `Box.Metrics *BoxMetrics` with `json:"metrics,omitempty"`, and `Probes` gains the same six fields (`MemTotalBytes`, `MemAvailBytes`, `Load1`, `DiskFreeBytes *int64/*float64`, `Release`, `Tunnel string`). Resident/parked counts are NOT duplicated — `RouteCounts` already carries them (recorded divergence 2). |
| + |
| +- [ ] **Step 1: Write the failing tests:** |
| + |
| +```go |
| +func TestCollectBoxCarriesRawMetrics(t *testing.T) { |
| + mt, ma, df := int64(4056547328), int64(2028273664), int64(9999999) |
| + l1 := 0.42 |
| + b := CollectBox(nil, Probes{ |
| + MemTotalBytes: &mt, MemAvailBytes: &ma, Load1: &l1, DiskFreeBytes: &df, |
| + Release: "abc1234", Tunnel: "connected", |
| + }, time.Now()) |
| + m := b.Metrics |
| + if m == nil || *m.MemTotalBytes != mt || *m.Load1 != l1 || m.Release != "abc1234" || m.Tunnel != "connected" { |
| + t.Fatalf("metrics block wrong: %+v", m) |
| + } |
| +} |
| + |
| +// Yesterday's Box shape (no metrics key) must parse today, and a FUTURE |
| +// box's unknown keys must not break today's parse either. |
| +func TestBoxBackCompatParse(t *testing.T) { |
| + yesterday := []byte(`{"generated_at":"2026-08-03T10:00:00Z","label":"ada-stargazing", |
| + "routes":{"static":1,"resident":2,"hibernating":3,"awake":null,"asleep":null}, |
| + "load":"comfortable","capacity":"plenty", |
| + "replication":{"healthy":true,"lag_seconds":5}, |
| + "adoption_running":true,"expired_canaries":0}`) |
| + var b Box |
| + if err := json.Unmarshal(yesterday, &b); err != nil { |
| + t.Fatalf("yesterday's shape failed to parse: %v", err) |
| + } |
| + if b.Metrics != nil { |
| + t.Fatal("absent metrics must decode as nil, not zero-value") |
| + } |
| + tomorrow := []byte(`{"generated_at":"2026-08-05T10:00:00Z","label":"ada-stargazing", |
| + "routes":{"static":0,"resident":0,"hibernating":0,"awake":null,"asleep":null}, |
| + "load":null,"capacity":null,"replication":{"healthy":null,"lag_seconds":null}, |
| + "adoption_running":null,"expired_canaries":null, |
| + "metrics":{"mem_total_bytes":1,"some_future_field":true}}`) |
| + if err := json.Unmarshal(tomorrow, &b); err != nil { |
| + t.Fatalf("tomorrow's shape failed to parse: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/status/ -run 'CollectBoxCarriesRaw|BackCompat' -count=1` → FAIL. |
| +- [ ] **Step 3: Implement.** Add the fields; in `CollectBox`, after the existing Box literal is built, attach the block only when something is known (all-unknown publishes no block — additive for old parsers, quiet for boxes that can't answer): |
| + |
| +```go |
| +if p.MemTotalBytes != nil || p.MemAvailBytes != nil || p.Load1 != nil || |
| + p.DiskFreeBytes != nil || p.Release != "" || p.Tunnel != "" { |
| + box.Metrics = &BoxMetrics{ |
| + MemTotalBytes: p.MemTotalBytes, MemAvailBytes: p.MemAvailBytes, |
| + Load1: p.Load1, DiskFreeBytes: p.DiskFreeBytes, |
| + Release: p.Release, Tunnel: p.Tunnel, |
| + } |
| +} |
| +``` |
| + |
| + Update the package doc comment (lines 1–7) and Box's doc: banding is still enforced here for every PUBLIC field; the Metrics block is the deliberate raw/private exception, excluded from fleet.json by construction (BuildFleet copies fields explicitly into FleetBox and does not copy Metrics — verify by reading `fleet.go:116-124`, and say so in the comment). |
| +- [ ] **Step 4: Dry-run leak scope.** Read `internal/status/dryrun.go` end to end. Extend its fixture Probes with distinctive raw values (e.g. `MemTotalBytes = 4242424242`). Keep/extend its leak assertions so that: (a) `status/fleet.json` and `status/index.html` must NOT contain `4242424242` (raw never public); (b) the box object MAY contain it. If the existing leak check sweeps ALL published objects for fraction-like fixtures, scope that sweep to fleet.json + index.html for the new raw fixtures only — do not weaken the existing hostname/machine-id assertions for box objects. |
| +- [ ] **Step 5: Run tests** — `go test ./internal/status/ -count=1` → PASS (including DryRun tests). |
| +- [ ] **Step 6: Full gate + commit** — `status: additive raw Metrics block on Box — private to the bucket, never published to fleet.json`. |
| + |
| +--- |
| + |
| +### Task 6: Host probes for the metrics block + heartbeat wiring |
| + |
| +**Files:** |
| +- Modify: `internal/host/probes.go`, `internal/host/agent.go` (statusLoop's Probes literal ~line 622) |
| +- Test: `internal/host/probes_test.go` (or wherever `parseIsActive`-style probe tests live — read first, extend in place) |
| + |
| +**Interfaces:** |
| +- Consumes: `Probes` fields (Task 5), `tunnelState()` (Task 4), `version.Build` (`internal/version`). |
| +- Produces: `readMemBytes() (total, avail *int64)`, `readLoad1() *float64`, `readDiskFreeBytes() *int64` — all fail toward nil, never a guessed zero (the file's contract). |
| + |
| +- [ ] **Step 1: Write the failing tests** — pure parsers, no /proc in tests (follow the file's design note): |
| + |
| +```go |
| +func TestParseMeminfoBytes(t *testing.T) { |
| + total, avail := parseMeminfoBytes(strings.NewReader( |
| + "MemTotal: 3961472 kB\nMemFree: 100 kB\nMemAvailable: 1980736 kB\n")) |
| + if total == nil || *total != 3961472*1024 || avail == nil || *avail != 1980736*1024 { |
| + t.Fatalf("got %v %v", total, avail) |
| + } |
| + if tt, aa := parseMeminfoBytes(strings.NewReader("garbage")); tt != nil || aa != nil { |
| + t.Fatal("unparseable meminfo must yield nil, not zero") |
| + } |
| +} |
| + |
| +func TestParseLoad1(t *testing.T) { |
| + if l := parseLoad1("0.42 0.36 0.30 1/234 5678\n"); l == nil || *l != 0.42 { |
| + t.Fatalf("got %v", l) |
| + } |
| + if l := parseLoad1(""); l != nil { |
| + t.Fatal("empty loadavg must yield nil") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/host/ -run 'Meminfo|Load1' -count=1` → FAIL. |
| +- [ ] **Step 3: Implement** in probes.go: `parseMeminfoBytes(r io.Reader)` (same scanner shape as readMemFrac, returning bytes = kB×1024), `readMemBytes()` opening `/proc/meminfo` and delegating; `parseLoad1(s string)` (first field of `/proc/loadavg` via `strconv.ParseFloat`), `readLoad1()` reading the file; `readDiskFreeBytes()` via `syscall.Statfs(dataDir(), &st)` returning `int64(st.Bavail) * st.Bsize` (nil on error). Every failure → nil. |
| +- [ ] **Step 4: Wire into statusLoop's Probes literal** (agent.go ~line 622), after the existing fields: |
| + |
| +```go |
| +memTotal, memAvail := readMemBytes() |
| +p := status.Probes{ |
| + // ... existing fields unchanged ... |
| + MemTotalBytes: memTotal, |
| + MemAvailBytes: memAvail, |
| + Load1: readLoad1(), |
| + DiskFreeBytes: readDiskFreeBytes(), |
| + Release: version.Build, |
| + Tunnel: tunnelState(), |
| +} |
| +``` |
| + |
| +(Import `"github.com/carlosframework/platform/internal/version"`.) |
| +- [ ] **Step 5: Run tests** — `go test ./internal/host/ -count=1` → PASS. |
| +- [ ] **Step 6: Full gate + commit** — `host: raw metric probes (mem/load/disk/release/tunnel) into the heartbeat Box`. |
| + |
| +---### Task 7: Render metrics + staleness — console fleet view and `carlos status -live` |
| + |
| +**Files:** |
| +- Modify: `internal/status/fleet.go` (export a box loader), `internal/console/view.go` + `internal/console/server.go` (fleet view, anchor `view.go:344` FleetView / `server.go:961` handleFleet — read both first), `cli.go` (RunStatus ~line 1587, statusUsage ~line 1568) |
| +- Test: `internal/status/fleet_test.go`, plus the console package's existing fleet-view test file (read and extend in place) |
| + |
| +**Interfaces:** |
| +- Consumes: `Box.Metrics` (Task 5), existing `loadBoxes` (`fleet.go:145`), staleness rule (`now - GeneratedAt > 3×period`, `fleet.go:115`). |
| +- Produces: `status.LoadBoxes(ctx, s) ([]Box, error)` (exported wrapper over loadBoxes — one loader, one skip-corrupt rule, no second parser); `status.BoxStale(b Box, now time.Time) bool` using the same 3× constant; CLI `carlos status -live` reading `CARLOS_STATUS_BUCKET`/`CARLOS_DEPLOYMENT_BUCKET` via the same `resolveStatusBucket` helper agent.go's statusLoop uses. |
| + |
| +- [ ] **Step 1: Write the failing test** (fleet_test.go): |
| + |
| +```go |
| +func TestLoadBoxesAndStaleness(t *testing.T) { |
| + s := testDirStore(t) // the DirStore helper the package's tests already use |
| + fresh := Box{GeneratedAt: time.Now(), Label: "ada-stargazing", |
| + Metrics: &BoxMetrics{Release: "abc1234", Tunnel: "connected"}} |
| + dark := Box{GeneratedAt: time.Now().Add(-time.Hour), Label: "grace-kayaking"} |
| + must(t, WriteBox(context.Background(), s, fresh)) |
| + must(t, WriteBox(context.Background(), s, dark)) |
| + boxes, err := LoadBoxes(context.Background(), s) |
| + if err != nil || len(boxes) != 2 { |
| + t.Fatalf("LoadBoxes: %v, %d boxes", err, len(boxes)) |
| + } |
| + for _, b := range boxes { |
| + wantStale := b.Label == "grace-kayaking" |
| + if BoxStale(b, time.Now(), 5*time.Minute) != wantStale { |
| + t.Fatalf("%s: stale = %v, want %v", b.Label, !wantStale, wantStale) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** → FAIL (undefined: LoadBoxes). |
| +- [ ] **Step 3: Implement** in fleet.go: `func LoadBoxes(ctx context.Context, s release.Store) ([]Box, error) { return loadBoxes(ctx, s) }`; hoist the 3× rule into `const staleFactor = 3` + `func BoxStale(b Box, now time.Time, period time.Duration) bool { return now.Sub(b.GeneratedAt) > staleFactor*period }`. BuildFleet's inline computation at `fleet.go:115` switches to calling it so the two can never drift. |
| +- [ ] **Step 4: Console.** In the fleet view (FleetView/handleFleet), add a per-box "Hosts" detail: call `status.LoadBoxes` against the store handle the console already holds for fleet.json, render per box — label, region, release, tunnel state, `mem avail/total` (humanized), load1, disk free — and when `BoxStale`: render the box as **unreachable** (distinct style + the age of the last heartbeat) and suppress the metric numbers entirely (an unreachable host must never show its last happy numbers — spec). Follow the surrounding template/view conventions exactly (vendored Lucide icons only, no CDN). Extend the existing console fleet-view test with: a stale box renders "unreachable" and does NOT render its stale metric values; a fresh box renders them. |
| +- [ ] **Step 5: CLI.** Replace RunStatus's dry-run-only shape with two modes (keep `-dry-run` byte-identical; update statusUsage — it currently *documents* having no live mode, and that sentence must go): |
| + |
| +```go |
| +liveFlag := fs.Bool("live", false, "read status/boxes/*.json from the status bucket and print per-host metrics") |
| +// ... |
| +if *liveFlag { |
| + if err := statusLive(os.Stdout); err != nil { |
| + log.Fatal(err) |
| + } |
| + return |
| +} |
| +``` |
| + |
| +```go |
| +// statusLive prints one line per box: label, region, release, tunnel state, |
| +// memory, load, disk — or UNREACHABLE (+heartbeat age) for a stale box, |
| +// never its last happy numbers. Operator credentials; same bucket |
| +// resolution as the agent's own status loop. |
| +func statusLive(w io.Writer) error { |
| + ctx := context.Background() |
| + bucket := host.ResolveStatusBucket(os.Getenv) // export the existing helper from internal/host |
| + if bucket == "" { |
| + return fmt.Errorf("set CARLOS_STATUS_BUCKET or CARLOS_DEPLOYMENT_BUCKET") |
| + } |
| + s, err := release.NewS3Store(ctx, bucket) |
| + if err != nil { |
| + return err |
| + } |
| + boxes, err := status.LoadBoxes(ctx, s) |
| + if err != nil { |
| + return err |
| + } |
| + now := time.Now() |
| + for _, b := range boxes { |
| + if status.BoxStale(b, now, 5*time.Minute) { |
| + fmt.Fprintf(w, "%-24s %-12s UNREACHABLE (last heartbeat %s ago)\n", |
| + b.Label, b.Region, now.Sub(b.GeneratedAt).Round(time.Minute)) |
| + continue |
| + } |
| + m := b.Metrics |
| + if m == nil { |
| + fmt.Fprintf(w, "%-24s %-12s ok (no metrics block — pre-upgrade host)\n", b.Label, b.Region) |
| + continue |
| + } |
| + fmt.Fprintf(w, "%-24s %-12s release=%s tunnel=%s mem=%s/%s load=%s disk-free=%s\n", |
| + b.Label, b.Region, orDash(m.Release), orDash(m.Tunnel), |
| + humanBytes(m.MemAvailBytes), humanBytes(m.MemTotalBytes), |
| + humanFloat(m.Load1), humanBytes(m.DiskFreeBytes)) |
| + } |
| + return nil |
| +} |
| +``` |
| + |
| +(Write the tiny `orDash`/`humanBytes`/`humanFloat` helpers beside it — nil renders as `-`. Rename/export `resolveStatusBucket` → `host.ResolveStatusBucket` with a wrapper keeping the internal callsite compiling.) |
| +- [ ] **Step 6: Run everything** — `go test ./internal/status/ ./internal/console/ ./... -count=1` → PASS. |
| +- [ ] **Step 7: Full gate + commit** — `status/console/cli: render raw host metrics privately, stale hosts as unreachable`. |
| + |
| +--- |
| + |
| +### Task 8: Off-cloud bootstrap (`carlos bootstrap -offcloud`) + runbook |
| + |
| +**Files:** |
| +- Modify: `internal/host/bootstrap.go` (RunBootstrap ~line 89, bootstrap ~line 104) |
| +- Create: `infra/remote-host.md` |
| +- Test: `internal/host/bootstrap_test.go` (read the existing staging-root tests first, extend in place) |
| + |
| +**Interfaces:** |
| +- Consumes: existing `bootstrap(root)` idempotent flow. |
| +- Produces: `-offcloud` flag; contract: credentials file at `<root>/etc/carlos/credentials` (mode 0600, owned by the service user on a live run) must exist BEFORE bootstrap proceeds; host.env must carry `AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials` (checked and printed, not silently assumed). |
| + |
| +- [ ] **Step 1: Write the failing test** (staging-root path — no root privileges in tests): |
| + |
| +```go |
| +func TestBootstrapOffcloudRequiresCredentialsFile(t *testing.T) { |
| + root := t.TempDir() |
| + if err := bootstrapOffcloud(root); err == nil { |
| + t.Fatal("want error when <root>/etc/carlos/credentials is absent") |
| + } |
| + credDir := filepath.Join(root, "etc", "carlos") |
| + os.MkdirAll(credDir, 0o755) |
| + credFile := filepath.Join(credDir, "credentials") |
| + os.WriteFile(credFile, []byte("[default]\naws_access_key_id=X\naws_secret_access_key=Y\n"), 0o644) |
| + if err := bootstrapOffcloud(root); err == nil { |
| + t.Fatal("want error on 0644 credentials (must be 0600)") |
| + } |
| + os.Chmod(credFile, 0o600) |
| + if err := bootstrapOffcloud(root); err != nil { |
| + t.Fatalf("offcloud bootstrap with staged 0600 credentials: %v", err) |
| + } |
| + // The normal bootstrap artifacts must exist too — offcloud is a superset. |
| + if _, err := os.Stat(filepath.Join(root, "etc", "systemd", "system", "carlos-edge.service")); err != nil { |
| + t.Fatalf("units not installed: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** → FAIL (undefined: bootstrapOffcloud). |
| +- [ ] **Step 3: Implement.** |
| + |
| +```go |
| +// bootstrapOffcloud is bootstrap for a box with no EC2 instance profile |
| +// (issue #53 point 4): identical structural work, plus the credential |
| +// preconditions — a staged 0600 credentials file scoped (at IAM) to the |
| +// deployment bucket + this host's own region bucket and nothing else, so |
| +// wrong-region-fails-at-IAM holds off-cloud too. The flag is explicit; no |
| +// IMDS sniffing (the agent's one IMDS probe is already gated for non-EC2). |
| +func bootstrapOffcloud(root string) error { |
| + cred := filepath.Join(root, "etc", "carlos", "credentials") |
| + fi, err := os.Stat(cred) |
| + if err != nil { |
| + return fmt.Errorf("offcloud: stage the scoped credentials file first: %s (0600, see infra/remote-host.md): %w", cred, err) |
| + } |
| + if fi.Mode().Perm() != 0o600 { |
| + return fmt.Errorf("offcloud: %s must be mode 0600, is %o", cred, fi.Mode().Perm()) |
| + } |
| + if err := bootstrap(root); err != nil { |
| + return err |
| + } |
| + if filepath.Clean(root) == "/" { |
| + if err := chownToService(cred); err != nil { // same helper the data dirs use |
| + return err |
| + } |
| + } |
| + fmt.Printf("offcloud bootstrap complete.\n ensure /etc/carlos/host.env sets:\n AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials\n then stage /opt/carlos/carlos and start carlos-edge.service (see infra/remote-host.md)\n") |
| + return nil |
| +} |
| +``` |
| + |
| + Wire the flag in RunBootstrap: `offcloud := fs.Bool("offcloud", false, "off-cloud box: require a staged 0600 credentials file instead of an EC2 instance profile")`, dispatching to `bootstrapOffcloud(*root)` when set. If `chownToService`'s signature differs (read it at ~line 230), adapt the call — the intent is: service user owns the file, still 0600. |
| +- [ ] **Step 4: Run tests** — `go test ./internal/host/ -run Bootstrap -count=1` → PASS (and the pre-existing bootstrap tests still pass). |
| +- [ ] **Step 5: Write `infra/remote-host.md`** — the one-page software-half runbook (issue #53's deliverable; the hq PHYSICAL half lands later in `infra/tito/region-hq.md`): |
| + |
| +```markdown |
| +# Remote host runbook (software half) |
| + |
| +A CARLOS host is anything that can reach the deployment bucket. A remote |
| +host is OUTBOUND-ONLY: its whole network surface is outbound 443 to S3 and |
| +the edge. No inbound port, no SSH path required by the platform. |
| + |
| +Prereqs: systemd Debian-family OS, 64-bit (the artifact key is |
| +GOOS-GOARCH — linux-arm64 or linux-amd64; a 32-bit OS resolves nothing). |
| + |
| +1. Provision the IAM principal + region data bucket |
| + (infra/modules/region-remote-host). Scope: deployment bucket + this |
| + host's own region bucket, NOTHING else. IAM Roles Anywhere is the |
| + long-term option; a scoped access key is acceptable to start. |
| +2. Stage /etc/carlos/credentials (0600): |
| + [default] |
| + aws_access_key_id=... |
| + aws_secret_access_key=... |
| +3. Stage /etc/carlos/host.env — the usual agent env plus: |
| + AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials |
| + CARLOS_TUNNEL_URL=https://<reserved-tunnel-hostname> |
| + CARLOS_TUNNEL_TOKEN=<this host's token> |
| + CARLOS_TUNNEL_NAME=<this host's channel name> |
| +4. Stage the binary at /opt/carlos/carlos (matching GOOS-GOARCH). |
| +5. Run: carlos bootstrap -offcloud |
| +6. systemctl start carlos-edge.service |
| +7. Verify: journalctl -u carlos-edge shows "tunnel connected"; the box |
| + appears in `carlos status -live` within one 5-minute heartbeat. |
| + |
| +Edge side: set CARLOS_TUNNEL_HOST + CARLOS_TUNNEL_CLIENTS |
| +(name=token,...) in the edge box's host.env; back routes with |
| +`carlos route -host <h> -tunnel <name>`. |
| +``` |
| + |
| +- [ ] **Step 6: Full gate + commit** — `host: carlos bootstrap -offcloud + remote host runbook (software half)`. |
| + |
| +--- |
| + |
| +### Task 9: Failover semantics — the dark-then-returns lease test |
| + |
| +**Files:** |
| +- Modify: `internal/lease/lease.go` (Acquire ~line 177 — extract a TTL seam) |
| +- Test: `internal/lease/lease_test.go` (live-gated harness: `liveConfig(t)` skips without `CARLOS_TEST_S3_BUCKET`) |
| + |
| +**Interfaces:** |
| +- Consumes: existing `Acquire`, `HeldError`, `liveConfig` harness. |
| +- Produces: unexported `acquireTTL(ctx, bucket, prefix, region string, ttl time.Duration) (*Held, error)`; `Acquire` becomes `return acquireTTL(ctx, bucket, prefix, region, LeaseTTL)`. No exported surface changes. |
| + |
| +Scope note: the wake-elsewhere half of failover (a fallback host's activator waking the instance from the regional bucket on request) is the EXISTING activator wake path, already exercised by the activator's own tests — nothing new to build or test there (issue #53: "verify it rather than build it"). What is genuinely new to prove is the returning-dark-host-must-lose property below. |
| + |
| +- [ ] **Step 1: Extract the seam** — mechanical: move Acquire's body into `acquireTTL` with `leaser.TTL = ttl`; Acquire delegates with `LeaseTTL`. Run `go test ./internal/lease/ -count=1` → PASS (everything skips or passes as before). |
| +- [ ] **Step 2: Write the dark-then-returns test** (proves issue #53 point 3: a returning host must NOT reclaim a lease now held elsewhere — the conditional-write race guarantees it; this test is the proof for the dark-then-returns sequence specifically): |
| + |
| +```go |
| +// TestDarkHostCannotReclaimAfterFailover: host A holds the lease and goes |
| +// dark (stops renewing, never releases). After TTL expiry, host B claims it |
| +// (the failover wake). A returns and tries to re-acquire: it MUST lose with |
| +// *HeldError naming B's ownership — never silently reclaim. |
| +func TestDarkHostCannotReclaimAfterFailover(t *testing.T) { |
| + bucket, prefix, region := liveConfig(t) |
| + ctx := context.Background() |
| + const shortTTL = 15 * time.Second |
| + |
| + a, err := acquireTTL(ctx, bucket, prefix, region, shortTTL) |
| + if err != nil { |
| + t.Fatalf("A initial acquire: %v", err) |
| + } |
| + // A goes dark: stop renewal WITHOUT releasing (the crash/power-cut shape). |
| + // rn.stop() halts the renewer but does not touch the lock object in S3. |
| + a.rn.stop() |
| + |
| + // While A's lease is still live, B must be refused — dark ≠ expired. |
| + if _, err := acquireTTL(ctx, bucket, prefix, region, shortTTL); err == nil { |
| + t.Fatal("B acquired while A's un-expired lease still held") |
| + } |
| + |
| + time.Sleep(shortTTL + 5*time.Second) // let A's lease expire |
| + |
| + b, err := acquireTTL(ctx, bucket, prefix, region, shortTTL) |
| + if err != nil { |
| + t.Fatalf("B acquire after expiry (the failover wake): %v", err) |
| + } |
| + defer b.Release(ctx) |
| + |
| + // A returns from the dead and tries to pick up where it left off. |
| + _, err = acquireTTL(ctx, bucket, prefix, region, shortTTL) |
| + var held *HeldError |
| + if !errors.As(err, &held) { |
| + t.Fatalf("returning dark host got %v, want *HeldError (must lose the race)", err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 3: Run gated** — without `CARLOS_TEST_S3_BUCKET`: `go test ./internal/lease/ -count=1` → SKIP cleanly. With it set (if operator credentials are available in the session): the test passes in ~30s. Record in the PR description whether the live run was executed. |
| +- [ ] **Step 4: Full gate + commit** — `lease: prove the dark-then-returns failover race (returning host must lose)`. |
| + |
| +--- |
| + |
| +### Task 10: E2E — edge + fake remote host over a real reverse channel, reconnect bound |
| + |
| +**Files:** |
| +- Create: `internal/edge/tunnel_e2e_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: everything from Tasks 1–4. This is issue #53's acceptance test, in-repo, no network beyond loopback. |
| + |
| +- [ ] **Step 1: Write the e2e test:** |
| + |
| +```go |
| +package edge |
| + |
| +// The acceptance test issue #53 names: edge + fake remote host over a REAL |
| +// reverse channel on loopback. Round-trip, header survival, kill-mid-flight, |
| +// re-dial, and the reconnect bound (asserted, not narrated). |
| + |
| +import ( |
| + "fmt" |
| + "io" |
| + "net/http" |
| + "net/http/httptest" |
| + "strings" |
| + "testing" |
| + "time" |
| +) |
| + |
| +const e2eReconnectBound = 5 * time.Second |
| + |
| +func TestTunnelEndToEnd(t *testing.T) { |
| + reg := testRegistryWithRoute(t, registry.Route{ |
| + Host: "demo.example", Kind: "instance", Status: "ready", Tunnel: "pi-box", |
| + }) |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + hub := NewTunnelHub("tunnel.test", map[string]string{"pi-box": "sekrit"}) |
| + e.EnableTunnels(hub) |
| + if err := e.Reload(); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The edge listens on loopback; both app traffic and the tunnel |
| + // registration ride this one listener — the no-new-port property. |
| + srv := httptest.NewServer(e) |
| + defer srv.Close() |
| + |
| + // Fake remote host: its "local edge" answers with a header + body, and |
| + // echoes X-Forwarded-For so the test can prove header discipline held. |
| + appHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("X-App-Header", "survived") |
| + w.Header().Set("Carlos-Purge", "/should-be-stripped") // edge must strip |
| + fmt.Fprintf(w, "host=%s xff=%s", r.Host, r.Header.Get("X-Forwarded-For")) |
| + }) |
| + stop := startFakeRemoteHost(t, srv.URL, "pi-box", "sekrit", appHandler) |
| + defer stop() |
| + |
| + waitForState(t, hub, "pi-box", "connected", e2eReconnectBound) |
| + |
| + do := func() *http.Response { |
| + req, _ := http.NewRequest("GET", srv.URL+"/", nil) |
| + req.Host = "demo.example" |
| + resp, err := http.DefaultClient.Do(req) |
| + if err != nil { |
| + t.Fatalf("request through tunnel: %v", err) |
| + } |
| + return resp |
| + } |
| + resp := do() |
| + body, _ := io.ReadAll(resp.Body) |
| + resp.Body.Close() |
| + if resp.StatusCode != 200 || resp.Header.Get("X-App-Header") != "survived" { |
| + t.Fatalf("round-trip: %d %v", resp.StatusCode, resp.Header) |
| + } |
| + if resp.Header.Get("Carlos-Purge") != "" { |
| + t.Fatal("Carlos-Purge leaked to the client — purge discipline broken over tunnel") |
| + } |
| + if !strings.Contains(string(body), "host=demo.example") || !strings.Contains(string(body), "xff=127.") { |
| + t.Fatalf("host/XFF discipline broken: %q", body) |
| + } |
| + |
| + // Kill the channel out from under the host (edge side drops it), then |
| + // assert the host re-dials and the NEXT request succeeds within bound. |
| + hub.mu.Lock() |
| + hub.conns["pi-box"].raw.Close() |
| + delete(hub.conns, "pi-box") |
| + hub.mu.Unlock() |
| + |
| + start := time.Now() |
| + waitForState(t, hub, "pi-box", "connected", e2eReconnectBound) |
| + if resp := do(); resp.StatusCode != 200 { |
| + t.Fatalf("post-reconnect request: %d", resp.StatusCode) |
| + } |
| + if d := time.Since(start); d > e2eReconnectBound { |
| + t.Fatalf("reconnect took %s, bound is %s", d, e2eReconnectBound) |
| + } |
| +} |
| + |
| +func waitForState(t *testing.T, hub *TunnelHub, name, want string, within time.Duration) { |
| + t.Helper() |
| + deadline := time.Now().Add(within) |
| + for time.Now().Before(deadline) { |
| + if hub.State(name) == want { |
| + return |
| + } |
| + time.Sleep(20 * time.Millisecond) |
| + } |
| + t.Fatalf("tunnel %s never reached state %q within %s", name, want, within) |
| +} |
| +``` |
| + |
| +`startFakeRemoteHost` reuses the HOST-side client for real: it must call `internal/host`'s actual dial+serve+backoff loop, not a reimplementation — otherwise the reconnect bound proves nothing. `internal/host` already imports `internal/edge`, so the edge test can't import host (cycle). Resolve it the repo-idiomatic way: extract the host's dial/serve/backoff core (`dialTunnel`, `serveTunnel`, `bufferedConn`, the retry loop, backoff constants) into a tiny new leaf package `internal/tunnelclient` (no imports beyond stdlib + x/net), with `internal/host/tunnel_client.go` becoming a thin wrapper that reads env, stores state via a `func(string)` callback, and calls it. Then `startFakeRemoteHost` runs `tunnelclient.Loop(ctx, url, name, token, handler, onState)` in a goroutine and returns a cancel func. Update Task 4's files accordingly when executing this task (the tests written there keep passing — they exercise the same functions through the new package). |
| +- [ ] **Step 2: Run it** — `go test ./internal/edge/ -run TestTunnelEndToEnd -count=1 -v` → PASS, reconnect measured under 5s. |
| +- [ ] **Step 3: Race check** — `go test ./internal/edge/ ./internal/tunnelclient/ -count=1 -race` → PASS. |
| +- [ ] **Step 4: Full gate + commit** — `edge: e2e reverse-channel acceptance — round-trip, header discipline, reconnect bound`. |
| + |
| +--- |
| + |
| +### Task 11: Final verification + draft PR |
| + |
| +- [ ] **Step 1:** `go build ./... && go vet ./... && go test ./... -count=1` — all packages (now 15+ with tunnelclient; CLAUDE.md's "14 packages" line is stale the moment tunnelclient lands — update CLAUDE.md's count in this commit). `gofmt -l` on every touched file → empty. |
| +- [ ] **Step 2:** `go test ./internal/release/ -run Ladder -count=1` → the pinned production channel names still pass. |
| +- [ ] **Step 3:** Push branch, open DRAFT PR titled "Remote host-agent: tunnel route backing, heartbeat metrics, off-cloud bootstrap (#53)". Body: link issue #53 + the spec; list the three recorded divergences (top of this plan); state which acceptance tests ran live vs gated (Task 9). End with the Claude Code attribution footer. |
| +- [ ] **Step 4:** Request Opus review (per CLAUDE.md) on: `internal/edge/tunnel.go` (auth, hijack lifecycle), the Reload/ServeHTTP changes, `internal/tunnelclient`, and the lease test. |
| +- [ ] **Step 5:** Once PR #49 (plan-c update) is merged to main: tick C6 Step 3 in `docs/superpowers/plans/2026-08-01-plan-c-tito-selfhosted.md` and record the divergences there. If #49 is still open at merge time, leave a comment on #49 instead so the tick isn't lost. |
amadan