amadan
spec+plan: fleet API surface — the deployment bucket behind the edge (#73)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 files changed,
+208
−0
docs/superpowers/plans/2026-08-11-fleet-api-surface.md+97 −0docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md+111 −0
diff --git a/docs/superpowers/plans/2026-08-11-fleet-api-surface.md b/docs/superpowers/plans/2026-08-11-fleet-api-surface.md| new file mode 100644 |
| index 000000000..ff4d3a856 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-11-fleet-api-surface.md |
| @@ -0,0 +1,97 @@ |
| +# Fleet API Surface Implementation Plan (issue #73) |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Fleet boxes lose direct platform-bucket S3: a `release.Store` spoken over HTTPS through the edge's reserved hostname, fleet-token-authed, account-scoped server-side — per `docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md` (F1–F8). |
| + |
| +**Architecture:** New edge handler beside the tunnel `/register` reusing the dynamic-auth table; new `internal/fleetapi` client implementing `release.Store`; env-driven store selection on the box (bucket set wins = fallback). No loop changes. |
| + |
| +**Tech Stack:** Go stdlib HTTP; no new deps. |
| + |
| +## Global Constraints |
| + |
| +- Branch `worktree-fleet-api` from main 79f6fc2. Gate per commit batch: `go build ./... && go vet ./... && go test ./... -count=1` (all packages), `gofmt -l` clean on touched files, `GOOS=darwin go build ./...`. Env `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; sandbox off only on "read-only file system". Known flake: internal/activator TestUnitTenantFailedHealthStopsUnit — re-run once. |
| +- Security invariants from the spec §Security are binding on every task: byte-identical 404s (no oracle), constant-time token compare, static names never authenticate the door, canonicalization before ACL, no token logging. |
| +- Edge test suite runs -race in CI: run `go test ./internal/edge/ -count=1 -race` in any task touching internal/edge. |
| +- Wire shape additive; platform boxes' S3 behavior byte-identical (regression-pinned in Task 4). |
| + |
| +--- |
| + |
| +### Task 1: `TunnelHub.VerifyDynamic` — auth extraction |
| + |
| +**Files:** Modify `internal/edge/tunnel.go` (ServeHTTP dynamic branch ~L129-138); Test `internal/edge/tunnel_test.go`. |
| + |
| +**Interfaces — Produces:** `func (h *TunnelHub) VerifyDynamic(name, token string) (fleet string, ok bool)` — ok only when `name` contains "/" AND is present in h.dynamicAuth AND the sha256-hex of token constant-time-matches; fleet = the segment before the first "/". ServeHTTP's dynamic branch calls it (behavior byte-identical: same single-401, viaDynamic = ok). |
| + |
| +Steps (TDD): failing tests — static name refused even with correct plaintext in h.clients; unknown dynamic name refused; wrong token refused; right token → (fleet, true); after SetDynamicAuth swap removing the name, refused. Then extract, re-run edge suite with -race, commit `edge: VerifyDynamic — the dynamic-auth check as a reusable gate`. |
| + |
| +### Task 2: the store door — edge handler + canonicalization + ACL |
| + |
| +**Files:** Create `internal/edge/storedoor.go`, `internal/edge/storedoor_test.go`. Modify `internal/edge/edge.go` (reserved-host routing: today it hands the reserved host to TunnelHub — route `/store/` paths to the new handler first, `/register` to the hub unchanged) and `internal/edge/tunnel.go` only if the hub needs a getter for the door. |
| + |
| +**Interfaces — Consumes:** Task 1's `VerifyDynamic`; the edge's existing deployment store handle (find how edge.go builds/holds its release store — reuse, do not open a second S3 client) and its registry handle (`e.reg.GetFleet(name)` for fleet→account sqid). **Produces:** |
| + |
| +```go |
| +// storeDoor serves release.Store over HTTPS for fleet boxes (spec F1-F4). |
| +type storeDoor struct { |
| + hub *TunnelHub // VerifyDynamic |
| + store release.Store // the deployment bucket, edge's own handle |
| + fleetAccount func(fleet string) (sqid string, ok bool) // registry lookup, cached per reload like FleetChannelAuth consumers |
| +} |
| +// Routes (reserved host only): |
| +// GET /store/o/<key> → Get (200 body + ETag; 404) |
| +// PUT /store/o/<key> → Put (If-Match or If-None-Match:*; 200 new ETag; 412) |
| +// DELETE /store/o/<key> → Delete (204 always for in-scope keys) |
| +// GET /store/list?prefix=<p> → List (200 JSON array of keys) |
| +// GET /store/dirs?prefix=<p> → ListDirs (200 JSON array) |
| +func canonicalKey(raw string) (string, bool) // refuses "..", "//", leading "/", empty, >1024 |
| +func aclRead(sqid, fleet, key string) bool // control/serial | bootstrap/carlos | accounts/<sqid>.json | apps/<sqid>/** |
| +func aclWrite(fleet, key string) bool // fleets/<fleet>/** only (write AND delete) |
| +func aclListScope(sqid, fleet, prefix string) (string, bool) // maps "apps/"→"apps/<sqid>/", "accounts/"→allowed-with-filter, "fleets/<fleet>/..."→itself; else false |
| +``` |
| + |
| +Every refusal — bad auth, bad key, out-of-scope, missing object — is `http.NotFound(w, r)` after auth ran; auth runs exactly once per request; the handler on any NON-reserved host is unreachable (assert in test via the edge's routing, mirroring how tunnel registration is host-gated). |
| + |
| +Steps (TDD): the ACL matrix table test FIRST (every inventory key class from the spec × op × own/other → allow/404 — write ~25 rows, exact keys like `apps/ab12/app/channels/stable.json`, `apps/OTHER/app/...`, `fleets/pi/bindings/h.json`, `fleets/other/...`, `control/serial`, `bootstrap/carlos`, `accounts/ab12.json`, `accounts/other.json`, `../etc`, `apps/ab12/../OTHER/x`); handler tests over httptest with a DirStore: CAS round-trip (Put no-precondition → 400; If-Match stale → 412; fresh → 200 new etag), Get missing → 404 byte-identical to out-of-scope 404, list scoping strips other accounts, static-name auth → 404s identical too. Then implement; `-race` suite; commit `edge: the store door — release.Store over HTTPS, account-scoped (spec F1-F4)`. |
| + |
| +### Task 3: `internal/fleetapi` — the client store |
| + |
| +**Files:** Create `internal/fleetapi/fleetapi.go`, `internal/fleetapi/fleetapi_test.go`. |
| + |
| +**Interfaces — Consumes:** Task 2's wire (test against the real storeDoor over httptest, not a hand-rolled fake). **Produces:** |
| + |
| +```go |
| +// New builds a release.Store speaking the edge store door. |
| +// base = CARLOS_TUNNEL_URL's origin, name/token = the box's channel identity. |
| +func New(base, name, token string) (release.Store, error) |
| +``` |
| + |
| +Semantics: Get returns (body, etag); Put sends If-Match (or If-None-Match:* when ifMatch==""... CHECK: read release.Store's Put contract for the empty-ifMatch meaning in DirStore/S3Store first and mirror it EXACTLY); 412 → `release.ErrCASConflict`; 404 on Get → the same not-found error S3Store returns (`os.ErrNotExist`-wrapped — check and mirror); Delete idempotent; List/ListDirs decode. Timeouts: 30s per call; artifact GETs must stream (return the body reader... CHECK Store.Get returns []byte — it does; keep []byte, buffered — fine for binaries the box already downloads whole today). No token in error strings. |
| + |
| +Steps: failing round-trip tests through a real storeDoor+DirStore; implement; commit `fleetapi: release.Store over the edge store door`. |
| + |
| +### Task 4: box-side selection + retention skip |
| + |
| +**Files:** Modify `internal/host/agent.go` (`openDeployStore` ~L1926; status-bucket resolution `resolveStatusBucket` consumers in statusLoop ~L1582 and metrics_loop.go ~L748; retentionLoop arming ~L276/1456); Test `internal/host/agent_test.go` (+ existing fixtures). |
| + |
| +**Interfaces — Consumes:** `fleetapi.New`, `resolveFleetIdentity` (fleetidentity.go). **Produces:** selection rule (spec F5): when `resolveFleetIdentity` succeeds AND the relevant bucket env (`CARLOS_DEPLOYMENT_BUCKET`/`_DIR` for openDeployStore; `CARLOS_STATUS_BUCKET` else deployment for status/metrics) is UNSET → `fleetapi.New(os.Getenv("CARLOS_TUNNEL_URL"), identity string, token)`; bucket set → S3/dir exactly as today. retentionLoop: fleet identity present → log once `agent: retention loop is a platform-box job — skipped on a fleet box (spec F6)` and return. |
| + |
| +Tests: table over env permutations pinning which store type each path builds (export a tiny seam if needed — prefer a package-level `deployStoreForTest` accessor pattern ONLY if an existing test seam convention exists; otherwise test via behavior: fake edge httptest server + env pointing at it, assert a serial read hits the door). Regression pin: bucket-set env yields S3-backed behavior with zero requests to the door. Retention skip test. Commit `host: fleet boxes speak the store door; retention stays a platform job (spec F5/F6)`. |
| + |
| +### Task 5: e2e — the fleet loop through the door |
| + |
| +**Files:** Modify/extend `internal/edge/fleet_e2e_test.go` (or a new `fleetapi_e2e_test.go` beside it — read the existing harness first and extend, don't duplicate). |
| + |
| +Re-run the existing two-fake-box fleet e2e with the boxes' deployment store swapped for `fleetapi.New` pointed at the test edge: adoption converges, serial poll works, heartbeat lands under `fleets/<name>/status/...` via the door, binding takeover works, and the ACL blocks box A's fleet from reading a second account's keys planted in the store. Commit `e2e: fleet boxes converge with no bucket credentials (spec proof)`. |
| + |
| +### Task 6: docs + module + BYO contract |
| + |
| +**Files:** Modify `internal/console/templates/fleet_byo.html` (env contract: the no-bucket form becomes primary — CARLOS_TUNNEL_URL/NAME/TOKEN + hibernate bucket creds only; deployment-bucket form demoted to a "legacy direct-S3" note), its test assertion in `internal/console/accountfleet_test.go` if strings pinned; `infra/dev/region-pi.md` (flip note); `infra/modules/region-remote-host/{variables.tf,main.tf,README.md}` (`deployment_bucket_arn` optional — default `""`, IAM statement emitted only when non-empty; README reframes the customer artifact); spec status line. |
| + |
| +Gate + commit `docs+module: fleet boxes without platform-bucket credentials`. |
| + |
| +--- |
| + |
| +## Self-review notes |
| + |
| +Spec coverage: F1→T2 routing; F2→T1; F3→T2 ACL; F4→T2+T3 CAS both ends; F5→T4; F6→T4; F7→T2 allowlist row; F8 is a non-feature (streaming = []byte Get, noted T3). Security §: every invariant appears as a named test in T1/T2. Inventory classes all appear in the T2 matrix. Types: `VerifyDynamic` (T1) consumed T2; `storeDoor` internal; `fleetapi.New` (T3) consumed T4/T5. Open checks flagged inline (Put empty-ifMatch contract, not-found error shape) — implementers resolve against DirStore/S3Store as instructed, never invent. |
diff --git a/docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md b/docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md| new file mode 100644 |
| index 000000000..270e39971 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md |
| @@ -0,0 +1,111 @@ |
| +# Fleet API Surface: the deployment bucket behind the edge (issue #73) |
| + |
| +**Date:** 2026-08-11 |
| +**Status:** Draft — designed autonomously in the unattended fleets session; |
| +Paul pre-approved the direction in issue #73 (2026-08-07: "don't paper over |
| +it with a bucket policy — abstract the bucket behind the carlos API"). |
| +**Depends on:** customer fleets v1+v2 (merged, main 00c3344); the tunnel |
| +dynamic-auth table (edge/tunnel.go). |
| +**Closes:** #73. **Unblocks:** untrusted-customer fleets, the Amadan |
| +cross-account fleet, the Eleven→Carloku flip without platform-bucket creds. |
| + |
| +## The problem (from the #73 inventory, verified at 00c3344) |
| + |
| +A fleet box holds AWS credentials with RW on the PLATFORM's deployment |
| +bucket. Every consumer flows through `release.Store` handles built in |
| +exactly three places on the box: `openDeployStore` (agent.go, 10 call |
| +sites), and `release.NewS3Store` in statusLoop + metricsLoop (status-bucket |
| +resolution). What those handles actually touch, complete inventory: |
| + |
| +- **Reads**: `control/serial` (every 2s); `apps/<sqid>/<app>/…` channel |
| + pointers, production flags, release manifests + artifacts, restart-gen, |
| + instance records, config bundles + environments + config-gen, trash |
| + markers; `accounts/` list + records; `bootstrap/carlos` has NO in-repo |
| + fetch path today (pinfra pushes over SSM — a customer box has no update |
| + path at all). |
| +- **Writes (CAS)**: heartbeats/metrics/alerts under the |
| + `fleets/<name>/…` prefix rebase; fleet bindings at |
| + `fleets/<name>/bindings/<host>.json` (unprefixed handle, same keyspace). |
| +- **Deletes**: metrics 48h prune under own prefix; `retentionLoop` would |
| + delete RELEASE artifacts platform-wide — armed on any box with the |
| + pubkey, which is exactly the tamper surface to close. |
| +- **Not on fleet boxes** (verified gates): status-lease election |
| + (`shouldSkipPublisherElection`), `ReconcileFleetBindings`, domainsLoop |
| + (no `PLATFORM_APPS_DOMAIN`). Instance leases + litestream live entirely |
| + in the customer's own hibernate bucket — untouched by this spec. |
| + |
| +## Design: `release.Store` over HTTPS, ACL'd server-side |
| + |
| +One new mechanism, not N endpoints: an edge-hosted **store door** that |
| +speaks the five `release.Store` operations over plain HTTPS, authenticated |
| +per-request by the credential the box already holds (its fleet token), with |
| +a server-side ACL derived from the fleet's identity. The box swaps its S3 |
| +store handle for an HTTP one; every loop — adoption, fast-converge, |
| +reconcile, config, status, metrics, bindings — runs unchanged. |
| + |
| +### Decisions |
| + |
| +| # | Decision | Why | |
| +|---|---|---| |
| +| F1 | **Transport is the edge's reserved tunnel hostname**, new paths beside `/register`: `GET/PUT/DELETE /store/o/<key>`, `GET /store/list?prefix=`, `GET /store/dirs?prefix=`. Same headers as the tunnel dial: `Authorization: Bearer <token>` + `Carlos-Tunnel-Name: <fleet>/<label>`. | The reserved host is already TLS'd, already routed, already carries the only box-shaped credential the platform can verify. No new hostname, cert, or credential mint. | |
| +| F2 | **Auth = the dynamic-auth table**, via a new `TunnelHub.VerifyDynamic(name, token) (fleet string, ok bool)` refactored from ServeHTTP's dynamic branch (constant-time hash compare, single-401 shape preserved; static `CARLOS_TUNNEL_CLIENTS` names never authenticate here — this door is for fleet boxes only). | One credential, one hash table, one revocation path: detach/rotate already converge to the edge in seconds and now revoke API access with the same motion. | |
| +| F3 | **ACL is account-scoped + own-prefix**, derived server-side from the authenticated fleet's registry row (`fleets.account` → sqid): READ allows `control/serial`, `bootstrap/carlos`, `accounts/<own-sqid>.json`, and `apps/<own-sqid>/**`; list of `accounts/` and `apps/` answers only the own-account subset. WRITE/DELETE allow `fleets/<own-fleet>/**` only. Everything else: byte-identical 404 (never 403 — the keyspace must not be an oracle). | This is the issue's "account-scoped surface" literally: a fleet reads only its owning account's estate and writes only its own corner. A compromised customer box can no longer read other tenants or touch platform releases. | |
| +| F4 | **CAS passes through**: Get returns the backing etag in `ETag`; Put requires `If-Match` (or `If-None-Match: *` for create), maps conflict to 412; the client maps 412 back to `release.ErrCASConflict`. Deletes are idempotent 204. | The lease-adjacent writers (heartbeat CAS, binding takeover guard) keep their exact semantics through the proxy. | |
| +| F5 | **Client = `internal/fleetapi.Store`**, implementing `release.Store` over the F1 wire with the box's existing `CARLOS_TUNNEL_URL`/`_NAME`/`_TOKEN` env. Selection: `openDeployStore` (and the status/metrics bucket resolution) returns the API store **when fleet identity resolves AND `CARLOS_DEPLOYMENT_BUCKET`/`CARLOS_STATUS_BUCKET` is unset**; a bucket still set wins (the issue's migration fallback — platform boxes and grandfathered fleet boxes keep direct S3, byte-for-byte unchanged). | Env-driven cutover, box by box, no flag day: drop the bucket var + AWS creds from a box's env and it is on the API; put them back and it is off. | |
| +| F6 | **retentionLoop does not arm on a fleet box** (fleet identity present → skip with one log line). | A customer box scheduling deletes of platform-bucket releases was wrong under direct S3 too; the ACL would refuse it anyway — skipping avoids an hourly error loop. Platform boxes keep retention. | |
| +| F7 | **`bootstrap/carlos` is readable through the door** (F3 allowlist). | Gives customer boxes their first sanctioned self-update path (#73's §9 gap): customer tooling can fetch the platform binary with the fleet token; sha256 discipline stays the operator's, as with pinfra's script. | |
| +| F8 | **Artifact reads stream** (no presigned-URL redirect in v1). | Presigning re-exposes the bucket identity this spec exists to hide; edge egress for one fleet's artifacts is small, and the edge already streams tunnel traffic. Revisit only if a real fleet's adoption traffic hurts. | |
| + |
| +### What this deletes (when a box flips) |
| + |
| +`CARLOS_DEPLOYMENT_BUCKET` and the platform-bucket half of |
| +`/etc/carlos/credentials` on fleet boxes; the cross-account bucket-policy |
| +requirement (Amadan); the "trusted customers only" caveat on fleets. The |
| +`region-remote-host` module's `deployment_bucket_arn` input becomes |
| +optional (omitted = fleet-API consumer; the grant is not emitted). |
| + |
| +## Security invariants (test-pinned) |
| + |
| +- The door answers ONLY on the reserved hostname; every other host 404s it. |
| +- Auth failures, unknown keys, and out-of-scope keys are byte-identical |
| + 404s after a single constant-time verify; no response distinguishes |
| + "wrong token" / "no such key" / "someone else's key". |
| +- Static tunnel names (no `/`) never authenticate; a detached box's token |
| + fails from the moment `SetDynamicAuth` swaps the table (same bound as |
| + tunnel revocation). |
| +- Key canonicalization refuses `..`, empty segments, leading `/`, and |
| + keys >1024 bytes BEFORE the ACL runs; the ACL is prefix-string exact. |
| +- The proxy never logs tokens, and never forwards client-supplied AWS |
| + headers; it holds the deployment store handle the edge already has. |
| + |
| +## Testing |
| + |
| +Unit: ACL matrix table (every §inventory key class × read/write/delete × |
| +own/other account → allow/404); VerifyDynamic refutes static names, wrong |
| +tokens, detached boxes; canonicalization rejects. Integration: fleetapi |
| +client against a real edge handler over `httptest` backed by a DirStore — |
| +Get/Put CAS round-trip incl. 412→ErrCASConflict, List/ListDirs prefix |
| +strip, artifact streaming; then the FULL existing fleet e2e (two fake |
| +boxes) re-run with the boxes' deploy store swapped to the API client — |
| +adoption, serial poll, reconcile, heartbeat, binding takeover all through |
| +the door. Fallback: bucket-set env still builds S3 stores (platform boxes |
| +regression-pinned). |
| + |
| +## Rollout |
| + |
| +1. This branch: edge door + client + env switch + F6 + docs (BYO page env |
| + contract gains the no-bucket form; region-pi.md notes the flip). |
| +2. Platform binary roll (edge gains the door; no behavior change for any |
| + existing box). |
| +3. Pi fleet flips first (drop bucket env, keep hibernate creds), then |
| + Amadan joins API-native (no cross-account policy ever), then Eleven per |
| + its runbook. |
| +4. `region-remote-host` module: `deployment_bucket_arn` optional, README |
| + reframed (customer artifact = hibernate-bucket-only IAM). |
| + |
| +## Out of scope |
| + |
| +Presigned redirects (F8); per-app (finer than account) ACL; device tokens |
| +for non-fleet boxes; moving platform boxes off S3; hibernate-bucket |
| +abstraction (the customer owns that bucket by design); revoking the pi |
| +fleet's existing IAM user (operator step after its flip). |