| new file mode 100644 |
| index 000000000..42453daff |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-04-instance-provisioning.md |
| @@ -0,0 +1,661 @@ |
| +# Instance Provisioning Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Provisioning an instance becomes a control-plane data write: a |
| +bucket record, minted into a registry route by the host agent's reconcile |
| +pass, created/deleted through authenticated console API verbs an app's own |
| +process can call — per `docs/superpowers/specs/2026-08-04-instance-provisioning-design.md`. |
| + |
| +**Architecture:** New `internal/provision` package owns record/manifest |
| +shapes, validation, store ops, route conventions, and the reconcile pass. |
| +Registry gains two additive columns (`provisioned`, `seed`). The activator's |
| +`Backend` seam widens so restore-on-miss reports "fresh" and a declared seed |
| +argv runs before the first health probe. The console gains the instance verb |
| +family plus an app-bound `cli_tokens` kind. |
| + |
| +**Tech Stack:** Go 1.x, modernc.org/sqlite, S3/dir `release.Store`, stdlib |
| +`net/http` mux. |
| + |
| +## Global Constraints |
| + |
| +- `go build ./... && go vet ./... && go test ./... -count=1` clean before EVERY commit; `gofmt -l` clean for files touched. |
| +- Sandbox: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; go commands may need sandbox disabled ("read-only file system" is the tell). |
| +- Wire shapes are ADDITIVE: every object written before this change must still parse; registry migration is `ALTER TABLE ... ADD COLUMN` with duplicate-column swallowed. |
| +- Refusal-order discipline: unknown account / someone else's account / unknown app / foreign app-token path are byte-identical 404s (`http.NotFound`), asserted byte-for-byte in tests. |
| +- Bucket-side failures answer 502 and are never described in bucket-side terms. |
| +- Production channel names are pinned in `internal/release/laddercheck_test.go` — nothing here may change channel semantics. |
| +- Comment density/style: match the repo (explanatory doc comments stating the WHY and the invariant). |
| +- Delegation (CLAUDE.md): Tasks 1–4, 8 → Sonnet subagents from this brief; Tasks 5–7 touch security/production paths → implement with Opus (or Sonnet + mandatory Opus review before commit). Name the model explicitly on every dispatch. |
| + |
| +--- |
| + |
| +### Task 1: `internal/provision` — records, manifest, validation, store ops, route conventions |
| + |
| +**Files:** |
| +- Create: `internal/provision/provision.go` |
| +- Create: `internal/provision/provision_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `release.Store`, `release.AppPrefix`, `release.ErrCASConflict`, `release.ValidChannel`. |
| +- Produces (later tasks rely on these exact names): |
| + - `type Record struct` / `type Manifest struct` (JSON tags below) |
| + - `func ManifestKey(acct, app string) string` |
| + - `func RecordKey(acct, app, host string) string` |
| + - `func RecordsPrefix(acct, app string) string` |
| + - `func ParseRecordKey(key string, known map[string]bool) (acct, app, host string, ok bool)` |
| + - `func ValidHost(host, domain string) bool` |
| + - `func ValidHostShape(host string) bool` (no-domain form: length ≤ 253, every dot-split label matches the label regex — for doors that must validate a host after its app's manifest is gone) |
| + - `func ValidRegion(region string) bool` |
| + - `func ValidPool(pool string) bool` |
| + - `func GetManifest(ctx context.Context, s release.Store, acct, app string) (*Manifest, error)` |
| + - `func PutManifest(ctx context.Context, s release.Store, acct, app string, m Manifest) error` |
| + - `func CreateRecord(ctx context.Context, s release.Store, rec Record) error` |
| + - `func DeleteRecord(ctx context.Context, s release.Store, acct, app, host string) error` |
| + - `func ListRecords(ctx context.Context, s release.Store, acct, app string, max int) (recs []Record, truncated bool, err error)` |
| + - `func GetRecord(ctx context.Context, s release.Store, acct, app, host string) (*Record, error)` |
| + - `const SocketDir = "/run/carlos"` |
| + - `func DefaultSocket(host string) string` |
| + - `func DefaultDBPath(dataDir, host string) string` |
| + |
| +- [ ] **Step 1: Write failing tests** (`internal/provision/provision_test.go`) |
| + |
| +Cover, with table tests: |
| + |
| +```go |
| +func TestValidHost(t *testing.T) { |
| + cases := []struct { |
| + host, domain string |
| + want bool |
| + }{ |
| + {"acme.go.tito.io", "go.tito.io", true}, |
| + {"a.go.tito.io", "go.tito.io", true}, |
| + {"acme.go.tito.io", "other.io", false}, // wrong suffix |
| + {"go.tito.io", "go.tito.io", false}, // no label |
| + {"a.b.go.tito.io", "go.tito.io", false}, // two labels |
| + {"Acme.go.tito.io", "go.tito.io", false}, // uppercase refused, not folded |
| + {"-a.go.tito.io", "go.tito.io", false}, // leading hyphen |
| + {"a-.go.tito.io", "go.tito.io", false}, // trailing hyphen |
| + {"../x.go.tito.io", "go.tito.io", false}, // traversal shapes |
| + {"a/b.go.tito.io", "go.tito.io", false}, |
| + {"acme.go.tito.io", "", false}, // empty domain never validates |
| + {strings.Repeat("a", 64) + ".go.tito.io", "go.tito.io", false}, // label > 63 |
| + } |
| + // plus: total host length > 253 refused |
| +} |
| +``` |
| + |
| +`TestValidRegion`: `"ie"` ok, `""` ok (unconstrained), `"IE"` refused, |
| +`"x/y"` refused, 17 chars refused. `TestValidPool`: same charset rule. |
| +`TestValidHostShape`: `"acme.go.tito.io"` ok, `"a"` ok, `"a..b"` refused, |
| +`"a/b"` refused, `".a"` refused, 254 chars refused, uppercase refused. |
| + |
| +`TestRecordKeyRoundTrip`: `RecordKey("bad","tito","acme.go.tito.io")` == |
| +`"apps/bad/tito/instances/acme.go.tito.io.json"`; `ParseRecordKey` on that |
| +key with `known={"bad":true}` returns `("bad","tito","acme.go.tito.io",true)`; |
| +unknown acct → `ok=false`; `apps/bad/tito/instances.json` (the manifest) → |
| +`ok=false`; `apps/bad/tito/channels/stable.json` → `ok=false`; a key with a |
| +nested slash after `instances/` → `ok=false`. |
| + |
| +`TestCreateRecordCreateOnly`: against `release.NewDirStore(t.TempDir())`, |
| +first `CreateRecord` succeeds, second returns error wrapping |
| +`release.ErrCASConflict` (`errors.Is`). |
| + |
| +`TestManifestRoundTrip`: `PutManifest` then `GetManifest` returns equal |
| +values; `PutManifest` twice (update) succeeds (CAS upsert); `GetManifest` |
| +on absence returns error satisfying `errors.Is(err, os.ErrNotExist)`; |
| +a manifest body with `schema: 2` refused on Get. |
| + |
| +`TestListRecords`: three records + the manifest object at the app prefix → |
| +list returns exactly the three, sorted by host; `max=2` → 2 records, |
| +`truncated=true`; a malformed record body is skipped, not fatal. |
| + |
| +`TestDefaults`: `DefaultSocket("h") == "/run/carlos/h.sock"`, |
| +`DefaultDBPath("/data","h") == "/data/h.db"`. |
| + |
| +- [ ] **Step 2: Run to verify failure** — `go test ./internal/provision/ -count=1` fails to compile (package absent). |
| + |
| +- [ ] **Step 3: Implement** (`internal/provision/provision.go`) |
| + |
| +```go |
| +// Package provision owns instance provisioning as a control-plane data |
| +// write: the bucket record and app manifest shapes, their validation, the |
| +// store operations the console's instance verbs use, and the route |
| +// conventions shared with `carlos add`. The reconcile pass that turns |
| +// records into routes lives in reconcile.go (Task 3). Spec: |
| +// docs/superpowers/specs/2026-08-04-instance-provisioning-design.md. |
| +package provision |
| + |
| +type Record struct { |
| + Schema int `json:"schema"` // 1 |
| + Host string `json:"host"` |
| + Slug string `json:"slug"` |
| + App string `json:"app"` |
| + Account string `json:"account"` |
| + Pool string `json:"pool,omitempty"` |
| + Region string `json:"region,omitempty"` |
| + Channel string `json:"channel"` |
| + CreatedAt time.Time `json:"created_at"` |
| + CreatedBy string `json:"created_by,omitempty"` |
| +} |
| + |
| +type Manifest struct { |
| + Schema int `json:"schema"` // 1 |
| + App string `json:"app"` |
| + Domain string `json:"domain"` |
| + Pool string `json:"pool,omitempty"` |
| + Channel string `json:"channel,omitempty"` |
| + Regions []string `json:"regions,omitempty"` |
| + Seed []string `json:"seed,omitempty"` |
| +} |
| + |
| +func ManifestKey(acct, app string) string { return release.AppPrefix(acct, app) + "instances.json" } |
| +func RecordsPrefix(acct, app string) string { return release.AppPrefix(acct, app) + "instances/" } |
| +func RecordKey(acct, app, host string) string { return RecordsPrefix(acct, app) + host + ".json" } |
| +``` |
| + |
| +`ParseRecordKey`: `strings.CutPrefix(key, "apps/")`, cut acct (must be in |
| +`known`), cut app (refuse `channels`/`releases`, the reconcile.go collision |
| +rule), require the next segment to be exactly `instances`, then |
| +`host.json` with no further `/` and non-empty host. Mirror the shape of |
| +`domains.appKeyOwner` (reconcile.go:375) in miniature. |
| + |
| +`ValidHost(host, domain)`: `domain != ""`, `len(host) <= 253`, |
| +`label, ok := strings.CutSuffix(host, "."+domain)`, label matches |
| +`labelRe = regexp.MustCompile(`+"`"+`^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$`+"`"+`)` |
| +(the `domains.accountNameRe` shape — comment: this is path-traversal |
| +defense stated as such; the host becomes a bucket key segment and the |
| +box-side `<host>.sock`/`<host>.db` path segments). |
| + |
| +`ValidRegion` / `ValidPool`: `^[a-z0-9-]{0,16}$` (empty allowed — "" is |
| +the unconstrained region / shared pool). |
| + |
| +Store ops: `GetManifest` — `s.Get(ManifestKey(...))`, pass through |
| +not-found, `json.Unmarshal`, refuse `m.Schema > 1` with a "newer than this |
| +binary" error (appconfig's store.go:69 precedent). `PutManifest` — |
| +read-etag-then-Put with one retry on `release.ErrCASConflict` (the |
| +`WriteAccountManifest` shape, console/account.go:45). `CreateRecord` — |
| +`s.Put(RecordKey(rec.Account, rec.App, rec.Host), body, "")`, error |
| +passthrough so callers can `errors.Is(err, release.ErrCASConflict)`. |
| +`DeleteRecord` — `s.Delete` (idempotent by Store contract). `GetRecord` — |
| +Get + unmarshal + verify body host/app/account agree with the key path |
| +(skip = error "record disagrees with its own path"). `ListRecords` — |
| +`s.List(ctx, RecordsPrefix(...))`, parse each key's host via a local |
| +suffix/prefix cut, Get+unmarshal each, skip malformed or disagreeing |
| +bodies, stop collecting at `max` and set `truncated` (but still count |
| +keys, not bodies, toward the cap decision), sort by Host. |
| + |
| +- [ ] **Step 4: Run tests to green** — `go test ./internal/provision/ -count=1` |
| +- [ ] **Step 5: gofmt + vet + full test sweep; commit** `feat(provision): instance records, app instances manifest, validation and store ops` |
| + |
| +--- |
| + |
| +### Task 2: registry — `provisioned` + `seed` columns |
| + |
| +**Files:** |
| +- Modify: `internal/registry/registry.go` (Route struct ~:100, migrate ALTER list :163-182, routeCols :344, scanRoute :349, Upsert :443-464) |
| +- Test: `internal/registry/registry_test.go` (add cases to existing file) |
| + |
| +**Interfaces:** |
| +- Produces: `Route.Provisioned bool`, `Route.Seed string` (JSON argv, e.g. `["seed"]`; `""` = none), `func (r *Registry) SetSeed(host, seed string) error`. |
| + |
| +- [ ] **Step 1: Failing test** |
| + |
| +```go |
| +func TestProvisionedSeedRoundTrip(t *testing.T) { |
| + r := openTemp(t) // follow the file's existing helper pattern |
| + if err := r.Upsert(Route{Host: "a.go.tito.io", Provisioned: true, Seed: `["seed"]`}); err != nil { t.Fatal(err) } |
| + got, err := r.Get("a.go.tito.io") |
| + // assert got.Provisioned && got.Seed == `["seed"]` |
| + if err := r.SetSeed("a.go.tito.io", `["seed","-v"]`); err != nil { t.Fatal(err) } |
| + // re-Get, assert new seed; SetSeed on unknown host returns error naming the host |
| +} |
| +``` |
| + |
| +Also: open an existing pre-change DB fixture (create a Registry, close it, |
| +reopen) to prove the ALTER path is idempotent — the file's migrate test |
| +pattern. |
| + |
| +- [ ] **Step 2: Run to verify failure** (unknown fields). |
| +- [ ] **Step 3: Implement** |
| + - Route struct: after `Platform`, add |
| + ```go |
| + // Provisioned marks a route the instance-provisioning reconcile pass |
| + // (internal/provision) minted from a bucket instance record. Only |
| + // provisioned routes are ever auto-removed by that pass — the same |
| + // authority rule Platform carries for the domains pass, kept as a |
| + // SEPARATE flag because Platform means "alias to another route's |
| + // upstream" to three consumers (config sync, upstream picking, apex |
| + // candidacy) and a provisioned instance is the opposite: it owns a |
| + // process. |
| + Provisioned bool |
| + // Seed is the app's first-boot init argv as a JSON array (e.g. |
| + // ["seed"]), materialized from the app's instances.json by the |
| + // provision pass; "" = no seed. Run by the activator only on a wake |
| + // where restore-on-miss found no replica (backend_exec.go). |
| + Seed string |
| + ``` |
| + - ALTER list: `provisioned INTEGER NOT NULL DEFAULT 0`, `seed TEXT NOT NULL DEFAULT ''`. |
| + - `routeCols` + `scanRoute` + `Upsert` column lists (27 → 29 placeholders; scan `prov int`). |
| + - `SetSeed`: narrow UPDATE, error on zero rows (the `SetChannel` contract — the reconciler re-stamps it and must notice a vanished row). |
| +- [ ] **Step 4: green**; **Step 5: sweep + commit** `feat(registry): provisioned + seed route columns` |
| + |
| +--- |
| + |
| +### Task 3: `provision.Reconcile` — records → routes |
| + |
| +**Files:** |
| +- Create: `internal/provision/reconcile.go` |
| +- Create: `internal/provision/reconcile_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1 types/helpers; Task 2 `Route.Provisioned/Seed`, `SetSeed`; `domains.AccountManifestSqid`, `domains.AccountManifest`, `domains.Sqid`, `domains.OpsSqid` (all exported today). |
| +- Produces: |
| + ```go |
| + type ReconcileConfig struct { |
| + Region string // agent's region code; "" serves unconstrained records only |
| + Pools []string // pools this box serves; empty = the shared "" pool only |
| + DataDir string // for DBPath defaults ("/data" in production) |
| + Now func() time.Time // nil = time.Now |
| + } |
| + func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig) error |
| + ``` |
| + |
| +- [ ] **Step 1: Failing tests** (dir store + temp registry, the domains reconcile_test pattern). Seed the store with a valid account manifest (`accounts/bad/account.json` with a REAL agreeing sqid: use `domains.Sqid(id)` to compute one, id 2 → whatever it encodes — compute in the test, never hardcode) and `apps/<sqid>/tito/instances.json` (`domain go.tito.io, pool tito, seed ["seed"]`). |
| + |
| +Cases, each its own test function: |
| +1. **Mint**: record `acme.go.tito.io` (pool tito, region ie), cfg |
| + {Region: "ie", Pools: []string{"tito"}, DataDir: "/data"} → route exists |
| + with Host, Kind "instance", Socket "/run/carlos/acme.go.tito.io.sock", |
| + DBPath "/data/acme.go.tito.io.db", Hibernate true, Provisioned true, |
| + Status "ready", App "tito", Account <sqid>, Pool "tito", Region "ie", |
| + Channel "stable", Seed `["seed"]`, Slug "acme". |
| +2. **Scope**: same record, cfg Region "us" → NOT minted. cfg Region "ie" |
| + Pools nil → NOT minted (pool "tito" not served). Record with empty |
| + pool+region, cfg zero-value → minted. |
| +3. **Idempotent**: second pass changes nothing (compare full route rows). |
| +4. **Converge seed**: change manifest seed to `["seed","-v"]` → next pass |
| + re-stamps route Seed narrowly (and Version/Status untouched). |
| +5. **Conflict**: pre-existing hand-added route (`Provisioned false`) at the |
| + same host → pass leaves it byte-identical, mints nothing, logs. |
| +6. **Remove**: delete the record object → route deleted; a neighboring |
| + hand-added route and a foreign-scope provisioned route (region "us") |
| + survive. |
| +7. **Fail-safe**: break an account manifest (write garbage over it) → |
| + pass still mints nothing new for it AND does not remove existing |
| + provisioned routes (removal skipped); fix the manifest → next pass |
| + converges again. |
| +8. **Malformed record**: body host disagrees with key path → skipped, no |
| + route. |
| + |
| +- [ ] **Step 2: verify failure.** |
| +- [ ] **Step 3: Implement** `Reconcile`: |
| + |
| +```go |
| +// Reconcile materializes bucket instance records into registry routes for |
| +// THIS agent's region and pools: mint on appearance, converge seed, remove |
| +// on record deletion — removal only for routes this pass minted |
| +// (Provisioned), the same authority discipline as the domains pass's |
| +// canary removal. Per-item failures log and skip; a failed account |
| +// manifest read additionally disables the removal phase for the whole |
| +// pass — a transient read failure must never mass-retire a fleet. |
| +func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig) error |
| +``` |
| + |
| +Flow (mirror domains.Reconcile's structure and logging voice): |
| +1. `known, readFailed := knownAccounts(ctx, s)` — list `accounts/`, parse |
| + with `domains.AccountManifestSqid`, Get+unmarshal each |
| + `domains.AccountManifest`, require the three-way sqid agreement |
| + (reconcile.go:106 rule), skip tombstoned (`m.Deleted()`); any Get/parse |
| + failure sets `readFailed` (and logs). Listing failure returns error. |
| + Seed `known` with `domains.OpsSqid`. |
| +2. `keys := s.List(ctx, "apps/")` (error returns). Collect |
| + `recs map[string]Record` (by host) for keys matching |
| + `ParseRecordKey(key, known)`: Get + unmarshal + agreement check |
| + (body.Host == path host && body.Account == acct && body.App == app; |
| + skip loudly otherwise). A record Get failure sets `readFailed` too |
| + (its absence from the diff set must not retire its route). |
| + In-scope filter: `rec.Region == cfg.Region && poolServed(cfg.Pools, rec.Pool)` |
| + where `poolServed` treats empty cfg.Pools as `[""]`. |
| +3. Manifest cache per (acct,app): `GetManifest`; absent manifest → |
| + records still mint with seed "" (the manifest governs the API door; |
| + a record already written is truth). Manifest read FAILURE (non-ENOENT) |
| + sets `readFailed` and skips seed stamping for that app this pass. |
| + `seedJSON(m)`: `""` when nil/empty else `json.Marshal(m.Seed)`. |
| +4. `routes := reg.List()`; index by host. For each in-scope record: |
| + - existing route, `Provisioned` and same App+Account → converge: if |
| + seed differs, `reg.SetSeed(host, seedJSON)`. |
| + - existing route otherwise → `log.Printf("provision: %s exists but was not minted by this pass (app %q acct %q) — leaving it alone", ...)`. |
| + - absent → `reg.Upsert(mintRoute(rec, seedJSON, cfg))` with |
| + ```go |
| + registry.Route{Host: rec.Host, Slug: rec.Slug, Kind: "instance", |
| + Socket: DefaultSocket(rec.Host), DBPath: DefaultDBPath(cfg.DataDir, rec.Host), |
| + App: rec.App, Account: rec.Account, Pool: rec.Pool, Region: rec.Region, |
| + Channel: rec.Channel, Hibernate: true, Provisioned: true, |
| + Seed: seedJSON, Status: "ready"} |
| + ``` |
| + (comment: Hibernate true is the activator model — and what hands the |
| + DB to the lease replicator from birth; DBPath explicit is what keeps |
| + ops.FleetDBs' hibernating-exclusion correct.) |
| +5. Removal, only when `!readFailed`: every registry route with |
| + `Provisioned && rt.Region == cfg.Region && poolServed(cfg.Pools, rt.Pool)` |
| + and no record in the collected set → `reg.Delete(rt.Host)`, log. |
| + Comment the three-conditions discipline explicitly (flag = authority, |
| + scope = this box's jurisdiction, absent record = the bucket's decision). |
| +6. `// DNS seam (spec §10): multi-region deployments write the per-slug |
| + CNAME at provision time in the console's create handler, not here — |
| + the reconcile pass must stay DNS-free so a box never needs Route53 |
| + credentials.` (comment only, in the file header.) |
| + |
| +- [ ] **Step 4: green.** **Step 5: sweep + commit** `feat(provision): reconcile instance records into routes` |
| + |
| +--- |
| + |
| +### Task 4: host agent wiring — the instances loop |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (goroutine block :146-152; add loop near `domainsLoop` :483) |
| +- Test: `internal/host/agent_test.go` (config parsing only) |
| + |
| +**Interfaces:** |
| +- Consumes: `provision.Reconcile`, `openDeployStore` (agent.go:724). |
| +- Produces: `func provisionTickConfig(getenv func(string) string) provision.ReconcileConfig` (exported to tests the way `domainsTickConfig` is). |
| + |
| +- [ ] **Step 1: Failing test** — `provisionTickConfig`: `CARLOS_REGION=ie`, |
| + `CARLOS_POOLS=tito, shared` → `{Region: "ie", Pools: ["tito","shared"]}`; |
| + unset → zero values; whitespace trimmed, empties dropped; |
| + `CARLOS_DATA_DIR` honored for DataDir (default `/data`). |
| +- [ ] **Step 2: verify failure.** |
| +- [ ] **Step 3: Implement** `instancesLoop(reg *registry.Registry)`: |
| + mirror `domainsLoop` exactly — `openDeployStore` failure disables with a |
| + log; armed log line naming region/pools; first tick after 1 minute, then |
| + `5*time.Minute + rand.N(time.Minute)`; per-pass errors logged, loop |
| + lives on. No apps-domain gate: this loop is armed whenever the store is |
| + (an unconfigured region deliberately still serves ""-scoped records). |
| + Add `go instancesLoop(reg)` beside `go domainsLoop(...)`. |
| +- [ ] **Step 4: green.** **Step 5: sweep + commit** `feat(host): instance-provisioning reconcile loop (CARLOS_REGION/CARLOS_POOLS)` |
| + |
| +--- |
| + |
| +### Task 5: activator — fresh-boot seam and seed *(security/production path: Opus)* |
| + |
| +**Files:** |
| +- Modify: `internal/activator/activator.go` (Backend interface :125-151, doWake :604-673) |
| +- Modify: `internal/activator/backend_exec.go` (Restore :124, new Seed, extract env composition from Start) |
| +- Modify: `internal/activator/backend_unit.go` (Restore :114, no-op Seed, SplitBackend dispatch) |
| +- Test: `internal/activator/activator_test.go`, `backend_exec_test.go` (existing fakes gain the new methods) |
| + |
| +**Interfaces:** |
| +- Consumes: `registry.Route.Seed` (Task 2). |
| +- Produces (exact): |
| + ```go |
| + Restore(ctx context.Context, rt registry.Route) (fresh bool, err error) // Backend |
| + Seed(ctx context.Context, rt registry.Route) error // Backend, new |
| + var seedTimeout = 2 * time.Minute // backend_exec.go, var for tests |
| + ``` |
| + |
| +- [ ] **Step 1: Failing tests** |
| + - Backend-fake matrix in activator_test.go: doWake calls Seed exactly |
| + when `Restore` reported fresh AND `rt.Seed != ""`; never on |
| + fresh=false; a Seed error fails the wake and the fake lease records a |
| + Release call (the Start-failure precedent, asserted the same way the |
| + existing start-failure test does); Seed is called BEFORE Start (record |
| + call order in the fake). |
| + - ExecBackend.Restore matrix (dir-store/temp-dir harness the file's |
| + existing tests use): DB present → `(false, nil)`; DB absent + replica |
| + restores → `(false, nil)`; DB absent + no replica → `(true, nil)`. |
| + - ExecBackend.Seed: with a stub "binary" (a shell script the test stages |
| + at `live/<host>` that records its argv+env to a file and exits 0) — |
| + argv is `<seed args...> --db <dbPath>`; env contains the merged |
| + `.config.env`/`.env` layers minus `CARLOS_*` agent vars; a script |
| + exiting 1 returns an error containing its stderr; `rt.Seed` that fails |
| + `json.Unmarshal` returns a clear error (never a silent skip). |
| +- [ ] **Step 2: verify failure** (interface change breaks fakes first — update fakes as part of this step so the assertions, not the compile, are what fails). |
| +- [ ] **Step 3: Implement** |
| + - Widen `Backend.Restore` to `(bool, error)`; doc comment gains: "fresh |
| + reports that the local DB was absent AND no replica existed — a first |
| + boot, never a restore. The activator runs the route's Seed exactly |
| + then." |
| + - Add `Seed(ctx, rt) error` to Backend with doc comment; `UnitBackend`: |
| + `Restore` returns `(false, nil)` (unit tenants manage their own state |
| + dir), `Seed` returns nil. `SplitBackend`: dispatch both by |
| + `rt.Backing` like every other method. |
| + - `ExecBackend.Restore`: return `(false, nil)` on warm cache; |
| + `(false, err)` on stat/restore errors; `(!restored, nil)` after |
| + `lease.Restore` — keep both log lines. |
| + - `ExecBackend.Seed`: parse `rt.Seed` as `[]string` (empty → nil error, |
| + defensive); `ensureLiveLink`; compose env exactly as Start does — |
| + extract Start's cfgEnv/ownEnv/mergeEnv block into |
| + `func (b *ExecBackend) instanceEnv(host string) (map[string]string, error)` |
| + and call it from both (Start keeps its cache-var layering on top); |
| + `exec.CommandContext` with `context.WithTimeout(ctx, seedTimeout)`, |
| + argv `append(seedArgs, "--db", dbPath)`, `cmd.Stdout/Stderr` to the |
| + agent's streams like Start; non-zero exit → error naming host and exit |
| + state. Comment the app-author contract line from spec §6. |
| + - `doWake`: step 2 becomes |
| + ```go |
| + fresh, err := a.backend.Restore(ctx, rt) |
| + if err != nil { _ = lease.Release(ctx); return nil, fmt.Errorf("restore %s: %w", host, err) } |
| + if fresh && rt.Seed != "" { |
| + if err := a.backend.Seed(ctx, rt); err != nil { |
| + _ = lease.Release(ctx) |
| + return nil, fmt.Errorf("seed %s: %w", host, err) |
| + } |
| + } |
| + ``` |
| + with a comment: seeded strictly before Start's first health probe, so |
| + a brand-new instance is born seeded and a restore is never mistaken |
| + for a first boot. |
| +- [ ] **Step 4: green** (whole `./internal/activator/`). |
| +- [ ] **Step 5: sweep + commit** `feat(activator): report fresh boots and run the route's seed before first health probe` |
| + |
| +--- |
| + |
| +### Task 6: console store — the app token kind *(security path: Opus)* |
| + |
| +**Files:** |
| +- Modify: `internal/console/authstore.go` (schema :103-134, CLITokenAddress :370, new funcs) |
| +- Test: `internal/console/authstore_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, now time.Time) (raw string, err error) |
| + func (s *Store) CLIAppToken(rawToken string, now time.Time) (accountID int64, app string, ok bool) |
| + ``` |
| + and the containment change: `CLITokenAddress` answers ONLY `kind = ''` tokens. |
| + |
| +- [ ] **Step 1: Failing tests** |
| + - `NewAppToken` mints; `CLIAppToken` resolves it to (accountID, app); |
| + `CLITokenAddress` on the SAME raw token returns ok=false (containment: |
| + an app token is invisible to every user-token door). |
| + - A user token (existing `PollCLIAuth` path helper in tests) resolves via |
| + `CLITokenAddress` but `CLIAppToken` returns ok=false (containment, other |
| + direction). |
| + - App tokens never expire: with `now` 10 years out, `CLIAppToken` still |
| + ok. User tokens still expire at `CLITokenTTL` (existing behavior, |
| + re-asserted). |
| + - Revocation: `RevokeCLIToken(mintedBy, id)` kills an app token (it lists |
| + under the minter's address via `CLITokens`). |
| + - Migration idempotence: open, close, reopen the store — no error |
| + (duplicate-column swallow), old rows read as kind ''. |
| +- [ ] **Step 2: verify failure.** |
| +- [ ] **Step 3: Implement** |
| + - `migrateCLIAuth` gains an ALTER list (the registry migrate pattern — |
| + swallow "duplicate column"): |
| + `kind TEXT NOT NULL DEFAULT ''`, `account_id INTEGER NOT NULL DEFAULT 0`, |
| + `app TEXT NOT NULL DEFAULT ''`. |
| + - `NewAppToken`: `randomToken()`; INSERT with kind 'app', |
| + address=mintedBy (the audit/listing/revocation principal), label, |
| + `expires_at = 0`. Doc comment records the no-expiry decision verbatim |
| + from spec §8 (production signup must not die on day 91; revocation + |
| + config rotation are the control). |
| + - `CLIAppToken`: SELECT `token_hash, kind, account_id, app, expires_at` |
| + by hash; `subtle.ConstantTimeCompare`; require `kind == "app"`; |
| + expiry rule `expires == 0 || now.Unix() <= expires`; best-effort |
| + last_used update (same comment as CLITokenAddress). |
| + - `CLITokenAddress`: add `kind` to the SELECT and return false unless |
| + `kind == ""` — with the comment: "an app token presented to a user |
| + door must look exactly like a token that does not exist." |
| +- [ ] **Step 4: green.** **Step 5: sweep + commit** `feat(console): app-bound cli_tokens kind — invisible to user-token doors` |
| + |
| +--- |
| + |
| +### Task 7: console API — instance verbs + token mint *(security path: Opus)* |
| + |
| +**Files:** |
| +- Create: `internal/console/apiinstances.go` |
| +- Modify: `internal/console/authcli.go` (limits consts ~:40-79, cliLimits struct :142, newCLILimits :146, registerCLIRoutes ~:180) |
| +- Test: `internal/console/apiinstances_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1 provision store ops/validation; Task 6 token funcs; `apiAccountApp`, `cliUser`, `bearer`, `hashToken`, `jsonError`, `writeJSON`, `readJSON`, `s.audit`, `s.store.AccountByID`, `s.cfg.Releases` (the store), `newRateLimiter`. |
| +- Produces routes: |
| + ``` |
| + GET /api/cli/apps/{account}/{app}/instances |
| + POST /api/cli/apps/{account}/{app}/instances |
| + GET /api/cli/apps/{account}/{app}/instances/{host} |
| + DELETE /api/cli/apps/{account}/{app}/instances/{host} |
| + POST /api/cli/apps/{account}/{app}/instance-token |
| + ``` |
| + and `cliLimits.instances *rateLimiter` with |
| + `cliInstanceBurst = 60` / `cliRateWindow` (comment: keyed by TOKEN hash — |
| + the app token is not a person, and a signup burst is the legitimate |
| + workload; the address-keyed write budget would let one stolen user token |
| + starve the app's own signups and vice versa). |
| + |
| +- [ ] **Step 1: Failing tests** (the `auth_e2e_test.go` httptest + dir-store pattern; mint accounts/apps through the store test helpers already used by apicli tests) |
| + - **Byte-identical 404s**, asserted byte-for-byte against a canonical |
| + `http.NotFound` response: non-member user token on create/list/delete; |
| + app token whose binding is (acctA, tito) probing |
| + `/apps/<acctB>/tito/instances` and `/apps/<acctA>/other/instances`; |
| + unknown account; unknown app. |
| + - **Containment**: app token on `/promote` → 401 (Task 6 makes it |
| + unknown); app token on `POST .../instance-token` → 404 (only user |
| + tokens mint); user member token on instance verbs → works. |
| + - **Refusal order on create** (member user token): no manifest → 409 |
| + naming `carlos instances enable`; bad host shape → 400; wrong suffix → |
| + 400; region outside manifest allow-list → 400; bad channel → 400; |
| + duplicate create → 409 "already exists"; success → 201 with the |
| + record JSON; the record then exists in the dir store at the exact |
| + `provision.RecordKey` with `created_by` = the principal. |
| + - **Delete**: 200; record gone; second delete → 200 (idempotent). |
| + - **List**: returns created records; `truncated` false. |
| + - **Token mint**: member mints → 201 `{token, account, app}`; raw token |
| + then creates instances (the app-token happy path, e2e); audit rows |
| + exist for create/delete/mint with detail suffix `" (cli)"`. |
| + - **Rate limit**: 61st create in the window → 429 (shrink via the |
| + limiter's fields or a test hook consistent with existing limiter |
| + tests). |
| +- [ ] **Step 2: verify failure.** |
| +- [ ] **Step 3: Implement** `apiinstances.go`: |
| + |
| +```go |
| +// apiInstancePrincipal authorizes one instance-verb request. Two shapes of |
| +// bearer may pass: a USER token (resolved through the same |
| +// membership-proving gate every CLI verb uses) or an APP token, which must |
| +// match the path's account (exact sqid — an app token holds no memberships |
| +// to resolve a name against) and app. Every failure before that proof is |
| +// the family's byte-identical 404. The returned principal is the audit |
| +// actor: the user's address, or "app:<sqid>/<app>". |
| +func (s *Server) apiInstancePrincipal(w http.ResponseWriter, r *http.Request) (principal string, acct *Account, app string, ok bool) |
| +``` |
| +- `raw := bearer(r)`; try `s.store.CLIAppToken(raw, time.Now())` FIRST |
| + (cheap, and a user token returns !ok): on ok, load |
| + `s.store.AccountByID(accountID)`; 404 unless the path's `{account}` |
| + equals the account's Sqid AND `{app}` equals the bound app (comment: the |
| + sqid-only rule — no name resolution, nothing to leak); principal |
| + `"app:" + sqid + "/" + app`. |
| +- else fall through to `s.apiAccountApp(w, r)` (which 401s/404s itself); |
| + principal = addr. |
| + |
| +Handlers, each: principal gate → `s.cli.instances.allow(hashToken(bearer(r)), time.Now())` |
| +→ 429 → verb body. Create follows spec §7's order exactly; use |
| +`provision.GetManifest` (absent → 409 |
| +`"instances are not enabled for <app> — an operator writes apps/<sqid>/<app>/instances.json (carlos instances enable)"`, |
| +non-ENOENT error → 502 "could not read instance settings"), then |
| +`provision.ValidHost(host, m.Domain)` → 400 naming the domain rule, |
| +region/channel checks, then |
| + |
| +```go |
| +rec := provision.Record{Schema: 1, Host: host, |
| + Slug: strings.TrimSuffix(host, "."+m.Domain), |
| + App: app, Account: acct.Sqid, Pool: m.Pool, Region: req.Region, |
| + Channel: channel, CreatedAt: time.Now().UTC(), CreatedBy: principal} |
| +err := provision.CreateRecord(ctx, s.cfg.Releases, rec) |
| +``` |
| +`errors.Is(release.ErrCASConflict)` → 409; other → 502. 15s ctx like the |
| +family. Audit `"instance-create"` / `"instance-delete"` / `"instance-token"`. |
| +Delete: validate host SHAPE only when the manifest is absent (deletion must |
| +outlive a retired manifest) — a hostile `{host}` never reaches the key |
| +builder unvalidated: `provision.ValidHost(host, m.Domain)` when manifest |
| +present, else `provision.ValidHostShape(host)` (Task 1). |
| +GET one: `provision.GetRecord`, ENOENT → the byte-identical 404. |
| +List: `provision.ListRecords(ctx, store, sqid, app, 500)` → |
| +`{"instances": [...], "truncated": bool}`. |
| + |
| +`handleAPIInstanceToken`: `apiAccountApp` gate directly (user tokens only — |
| +an app token cannot mint tokens, and apiAccountApp can never see one); |
| +`s.cli.write.allow(addr, ...)` (address-keyed — minting is a human act); |
| +body `{label?}`; `NewAppToken(acct.ID, app, addr, label, now)`; audit; 201 |
| +`{"token": raw, "account": sqid, "app": app}` with the comment: shown |
| +exactly once; delivery is the config-env layer |
| +(`carlos env set -app <app> -secret ...`). |
| + |
| +`// DNS seam (spec §10):` comment in the create handler after the record |
| +write. |
| + |
| +Register routes + limiter field/const. |
| + |
| +- [ ] **Step 4: green** (`./internal/console/`). |
| +- [ ] **Step 5: sweep + commit** `feat(console): instance provisioning API verbs + app-held instance token` |
| + |
| +--- |
| + |
| +### Task 8: CLI — `carlos instances enable`, shared route defaults |
| + |
| +**Files:** |
| +- Modify: `cli.go` (runAdd :90-109 socket/db defaults → provision helpers; runInstances :157 grows subcommand dispatch) |
| +- Create: `instances_cli.go` (package main — the appconfig_cli.go pattern) |
| +- Test: `instances_cli_test.go` + extend `cli_test.go`'s runAdd coverage if present (check first; add minimal direct tests of the helper substitution otherwise via a build-and-run e2e only if the file already has one — do NOT invent a new e2e harness) |
| + |
| +**Interfaces:** |
| +- Consumes: `provision.PutManifest/GetManifest/Manifest`, `provision.DefaultSocket/DefaultDBPath`, `openStore()`, `accountFlags(fs)` (3-value form on main), `release.ValidChannel`. |
| +- Produces: `carlos instances enable -app <a> [-account ...] -domain <d> [-pool p] [-channel c] [-regions ie,us] [-seed "seed args"]`; bare `carlos instances` unchanged. |
| + |
| +- [ ] **Step 1: Failing test** — `runInstancesEnable` writes the manifest: |
| + against a `CARLOS_DEPLOYMENT_DIR` temp store (the env-verb test pattern in |
| + the repo — follow `appconfig_cli`'s tests if they exist, else test the |
| + extracted core `func instancesEnable(ctx, s release.Store, acct, app string, m provision.Manifest) error` directly), assert the object lands at |
| + `provision.ManifestKey` and re-running with a changed pool updates it |
| + (CAS upsert). Seed flag parsing: `-seed "seed -v"` → |
| + `[]string{"seed","-v"}` (strings.Fields — document: seed args are |
| + shell-word-free by contract; anything richer belongs in the app). |
| +- [ ] **Step 2: verify failure.** |
| +- [ ] **Step 3: Implement** |
| + - `instances_cli.go`: flagset per the env-verb shape; require `-app` and |
| + `-domain` non-empty; `release.ValidApp(*app)`; validate `-regions` |
| + entries via `provision.ValidRegion`, `-pool` via `provision.ValidPool`, |
| + `-channel` via `release.ValidChannel`; build Manifest{Schema: 1, ...}; |
| + `PutManifest`; print `enabled instances for <app> under <domain>`. |
| + - `runInstances`: `if len(args) > 0 && args[0] == "enable" { runInstancesEnable(args[1:]); return }` — bare form untouched. |
| + - `runAdd`: replace the two literals with `provision.DefaultSocket(host)` |
| + and `provision.DefaultDBPath(dataDir(), host)`; the unit-tenant socket |
| + literal (`/run/carlos/apps`) stays as-is (not a provision concern). |
| + Keep behavior byte-identical — this is the "shared provisioner core |
| + where honest" move; the why-not-further is recorded in spec §9. |
| +- [ ] **Step 4: green.** **Step 5: sweep + commit** `feat(cli): carlos instances enable; add shares provision's route conventions` |
| + |
| +--- |
| + |
| +### Task 9: full verification + spec self-review |
| + |
| +- [ ] `gofmt -l .` (only the three pre-existing unformatted files elsewhere may appear; none of ours). |
| +- [ ] `go build ./... && go vet ./... && go test ./... -count=1` — all packages, from the worktree root, sandbox env per Global Constraints. |
| +- [ ] Re-read the spec + the original brief end to end against the diff: |
| + every numbered build slice (records, reconciler, verbs, credential, |
| + seed, DNS seam) has landed or is explicitly recorded as a comment/spec |
| + note; the acceptance path (create → materialize → wake+seed → replicate |
| + → delete) is covered by tests at each hop; grep the diff for TODO/XXX. |
| +- [ ] Confirm nothing changed under `internal/release/laddercheck_test.go` semantics. |
| +- [ ] Commit any review fixes; the branch is PR-ready. |
amadan