| 1 | # Console as an App — Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Make the CARLOS console an ordinary app deployed by `carlos deploy`, retiring `roll-console.sh` to break-glass. |
| 6 | |
| 7 | **Architecture:** The console already ships as a release, rides a channel and is adopted through a route — what it lacks is an app *row* (blocked by #79), an honest way to verify which build is running (the edge's `X-Carlos-Version` is stamped from the route row and lies), and a rollback that leaves the box running. Tasks 1–4 give the app a row and a control-plane flag that gates destruction; tasks 5–6 replace header-watching with a probe the console process answers itself; tasks 7–10 close three failure modes the pinfra script has been papering over. Then a runbook rolls it out, tito first. |
| 8 | |
| 9 | **Tech Stack:** Go 1.24, SQLite (modernc driver), systemd, AWS SSM (runbook only, never from `carlos`). |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-08-23-console-as-an-app-design.md` — read it first; this plan argues from it. |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - `go build ./...`, `go vet ./...`, `go test ./... -count=1` clean, **every package**, before any commit lands. |
| 16 | - `gofmt -l` clean for files you touch. Three pre-existing unformatted files elsewhere are left alone. |
| 17 | - `GOOS=darwin go build ./...` must pass — `internal/host` links into the CLI, which people run on Macs. |
| 18 | - Sandbox: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`. |
| 19 | - **Wire-shape changes are additive.** Anything written before your change must still parse. |
| 20 | - **Console gate order never moves**: session → CSRF → owner (byte-identical 404) → `requireFreshAuth` LAST. |
| 21 | - A test that shells out to the `carlos` binary must build its env through `carlosTestEnv` and set `cmd.Dir` to a scratch dir. |
| 22 | - Every change lands through a PR. Nothing merges to `main` locally. |
| 23 | - Branch: `console-as-an-app` (already exists, carries the spec). |
| 24 | |
| 25 | --- |
| 26 | |
| 27 | ### Task 1: Two columns on `apps`, and the migration trap |
| 28 | |
| 29 | **Files:** |
| 30 | - Modify: `internal/console/store.go:189-231` (the additive ALTER block), `internal/console/store.go:266-288` (`migrateAppsKey`'s `apps_new`) |
| 31 | - Modify: `internal/console/store.go` (new accessors near `SetAppPlacement`, ~line 1170) |
| 32 | - Test: `internal/console/migrate_test.go` |
| 33 | |
| 34 | **Interfaces:** |
| 35 | - Consumes: nothing. |
| 36 | - Produces: |
| 37 | - `func (s *Store) AppFlags(accountID int64, app string) (controlPlane bool, buildProbe string, err error)` |
| 38 | - `func (s *Store) SetAppControlPlane(accountID int64, app string, on bool) error` |
| 39 | - `func (s *Store) SetAppBuildProbe(accountID int64, app, path string) error` |
| 40 | |
| 41 | **Why both columns land together:** they are one migration, and splitting them means touching `migrateAppsKey`'s copy list twice. |
| 42 | |
| 43 | **The trap** (spec §8): the ALTER block at `store.go:189` runs **before** `migrateAppsKey()` at line 231, and that rebuild copies an **enumerated** column list into `apps_new`. A column added only to the ALTER block is silently dropped for any database that has not yet been rebuilt — a self-hoster jumping versions. `deleted_at` and `placement` are already in both places for exactly this reason; the comment at `store.go:280` says so. Follow that pattern: **add to both.** |
| 44 | |
| 45 | - [ ] **Step 1: Write the failing test** |
| 46 | |
| 47 | Add to `internal/console/migrate_test.go`: |
| 48 | |
| 49 | ```go |
| 50 | // TestAppFlagsSurviveTheAppsKeyRebuild: control_plane and build_probe are |
| 51 | // added by the ALTER block, which runs BEFORE migrateAppsKey's table rebuild |
| 52 | // (store.go:189 vs :231). The rebuild copies an ENUMERATED column list, so a |
| 53 | // column missing from apps_new is silently dropped on any database that has |
| 54 | // not been rebuilt yet — a self-hoster jumping versions. deleted_at and |
| 55 | // placement are in both places for this reason (store.go:280); these must be |
| 56 | // too, and this test is what proves it rather than a comment. |
| 57 | func TestAppFlagsSurviveTheAppsKeyRebuild(t *testing.T) { |
| 58 | dir := t.TempDir() |
| 59 | path := filepath.Join(dir, "old.db") |
| 60 | |
| 61 | // A pre-rebuild database: PRIMARY KEY(app), the shape migrateAppsKey exists |
| 62 | // to replace. Built by hand because no released binary produces it any more. |
| 63 | raw, err := sql.Open("sqlite", path) |
| 64 | if err != nil { |
| 65 | t.Fatal(err) |
| 66 | } |
| 67 | if _, err := raw.Exec(`CREATE TABLE apps ( |
| 68 | app TEXT PRIMARY KEY, |
| 69 | account_id INTEGER NOT NULL DEFAULT 1, |
| 70 | created_by TEXT NOT NULL, |
| 71 | created_at INTEGER NOT NULL, |
| 72 | deleted_at INTEGER NOT NULL DEFAULT 0, |
| 73 | placement TEXT NOT NULL DEFAULT '' |
| 74 | );`); err != nil { |
| 75 | t.Fatal(err) |
| 76 | } |
| 77 | if _, err := raw.Exec(`INSERT INTO apps (app, account_id, created_by, created_at) |
| 78 | VALUES ('console', 1, 'paul@keymail.dev', 1)`); err != nil { |
| 79 | t.Fatal(err) |
| 80 | } |
| 81 | if err := raw.Close(); err != nil { |
| 82 | t.Fatal(err) |
| 83 | } |
| 84 | |
| 85 | st, err := Open(path) |
| 86 | if err != nil { |
| 87 | t.Fatalf("Open (which runs migrate) on a pre-rebuild database: %v", err) |
| 88 | } |
| 89 | defer st.Close() |
| 90 | |
| 91 | if err := st.SetAppControlPlane(OpsAccountID, "console", true); err != nil { |
| 92 | t.Fatalf("SetAppControlPlane: %v — the column did not survive the rebuild", err) |
| 93 | } |
| 94 | if err := st.SetAppBuildProbe(OpsAccountID, "console", "/healthz"); err != nil { |
| 95 | t.Fatalf("SetAppBuildProbe: %v", err) |
| 96 | } |
| 97 | cp, probe, err := st.AppFlags(OpsAccountID, "console") |
| 98 | if err != nil { |
| 99 | t.Fatalf("AppFlags: %v", err) |
| 100 | } |
| 101 | if !cp || probe != "/healthz" { |
| 102 | t.Fatalf("AppFlags = %v %q, want true \"/healthz\"", cp, probe) |
| 103 | } |
| 104 | } |
| 105 | |
| 106 | // TestAppFlagsDefaultOff: every row that existed before this change means |
| 107 | // "not the control plane, no probe" — the additive-default idiom the kind and |
| 108 | // service-principal columns used. |
| 109 | func TestAppFlagsDefaultOff(t *testing.T) { |
| 110 | fx := newConsoleFixture(t) |
| 111 | if _, err := fx.st.CreateAppIn(OpsAccountID, "titogo", "paul@keymail.dev", time.Now()); err != nil { |
| 112 | t.Fatal(err) |
| 113 | } |
| 114 | cp, probe, err := fx.st.AppFlags(OpsAccountID, "titogo") |
| 115 | if err != nil { |
| 116 | t.Fatal(err) |
| 117 | } |
| 118 | if cp || probe != "" { |
| 119 | t.Fatalf("a freshly claimed app = %v %q, want false \"\"", cp, probe) |
| 120 | } |
| 121 | } |
| 122 | ``` |
| 123 | |
| 124 | Check `migrate_test.go`'s existing imports; add `database/sql`, `path/filepath`, `time` if absent, and confirm the sqlite driver import matches what `store.go` uses (`_ "modernc.org/sqlite"` or the package's existing alias). |
| 125 | |
| 126 | - [ ] **Step 2: Run the test to verify it fails** |
| 127 | |
| 128 | ``` |
| 129 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 130 | go test ./internal/console/ -run 'TestAppFlags' -count=1 |
| 131 | ``` |
| 132 | |
| 133 | Expected: FAIL — `st.SetAppControlPlane undefined`. |
| 134 | |
| 135 | - [ ] **Step 3: Add the columns to the ALTER block** |
| 136 | |
| 137 | In `internal/console/store.go`, append to the additive slice that ends with the `accounts … tier` statement (~line 226): |
| 138 | |
| 139 | ```go |
| 140 | // Control-plane apps (console-as-an-app spec §6): the flag REFUSES |
| 141 | // destruction — trash, purge, rename, transfer — and refuses a |
| 142 | // service principal's promote. It deliberately does NOT gate a |
| 143 | // person's promote: `stable`'s fresh-auth 403 has no API-side |
| 144 | // step-up and `carlos deploy` log.Fatalfs on a failed promote AFTER |
| 145 | // the ship has landed (deploy.go:206-208), so gating promotion would |
| 146 | // break `carlos deploy -app console` — the command the flag exists to |
| 147 | // make safe. Deploying the console should be easy; destroying it |
| 148 | // should be hard. |
| 149 | `ALTER TABLE apps ADD COLUMN control_plane INTEGER NOT NULL DEFAULT 0;`, |
| 150 | // The path `carlos deploy` polls for the build the app's own PROCESS |
| 151 | // reports, instead of X-Carlos-Version — which the edge stamps from |
| 152 | // the route row (edge.go:1336) and which therefore lies from the |
| 153 | // moment adoption swaps the symlink until something restarts the |
| 154 | // process. "" keeps today's header behaviour byte for byte. |
| 155 | `ALTER TABLE apps ADD COLUMN build_probe TEXT NOT NULL DEFAULT '';`, |
| 156 | ``` |
| 157 | |
| 158 | - [ ] **Step 4: Add the same columns to `migrateAppsKey`'s rebuild** |
| 159 | |
| 160 | In `internal/console/store.go`, `migrateAppsKey`, extend `apps_new` and the copy — both, or the rebuild drops them: |
| 161 | |
| 162 | ```go |
| 163 | `CREATE TABLE apps_new ( |
| 164 | app TEXT NOT NULL, |
| 165 | account_id INTEGER NOT NULL DEFAULT 1, |
| 166 | created_by TEXT NOT NULL, |
| 167 | created_at INTEGER NOT NULL, |
| 168 | deleted_at INTEGER NOT NULL DEFAULT 0, |
| 169 | placement TEXT NOT NULL DEFAULT '', |
| 170 | control_plane INTEGER NOT NULL DEFAULT 0, |
| 171 | build_probe TEXT NOT NULL DEFAULT '', |
| 172 | PRIMARY KEY (account_id, app) |
| 173 | );`, |
| 174 | `INSERT INTO apps_new (app, account_id, created_by, created_at, deleted_at, placement, control_plane, build_probe) |
| 175 | SELECT app, account_id, created_by, created_at, deleted_at, placement, control_plane, build_probe FROM apps;`, |
| 176 | ``` |
| 177 | |
| 178 | Extend the existing comment above the INSERT to name the two new columns alongside `deleted_at` and `placement`. |
| 179 | |
| 180 | - [ ] **Step 5: Add the accessors** |
| 181 | |
| 182 | In `internal/console/store.go`, after `SetAppPlacement`: |
| 183 | |
| 184 | ```go |
| 185 | // AppFlags reads the two per-app deployment facts the control-plane work |
| 186 | // added: whether this app IS the control plane, and the path `carlos deploy` |
| 187 | // should poll for a build the app's own process reports. |
| 188 | // |
| 189 | // One query for both because both doors that read them read them together |
| 190 | // (the ship gate and the deploy watch), and a second round trip to the same |
| 191 | // row buys nothing. |
| 192 | func (s *Store) AppFlags(accountID int64, app string) (controlPlane bool, buildProbe string, err error) { |
| 193 | err = s.db.QueryRow(`SELECT control_plane, build_probe FROM apps WHERE account_id = ? AND app = ?`, |
| 194 | accountID, app).Scan(&controlPlane, &buildProbe) |
| 195 | if errors.Is(err, sql.ErrNoRows) { |
| 196 | // No row is not an error here: every caller has already been through |
| 197 | // apiAccountApp's byte-identical 404, so reaching this with no row |
| 198 | // means a race with a purge, and "not the control plane, no probe" is |
| 199 | // the right reading of an app that no longer exists. |
| 200 | return false, "", nil |
| 201 | } |
| 202 | return controlPlane, buildProbe, err |
| 203 | } |
| 204 | |
| 205 | // SetAppControlPlane marks (or unmarks) an app as the control plane. The row |
| 206 | // is guaranteed to exist: both callers reach this straight after a successful |
| 207 | // CreateAppIn or behind apiAccountApp's gate. |
| 208 | func (s *Store) SetAppControlPlane(accountID int64, app string, on bool) error { |
| 209 | _, err := s.db.Exec(`UPDATE apps SET control_plane = ? WHERE account_id = ? AND app = ?`, |
| 210 | on, accountID, app) |
| 211 | return err |
| 212 | } |
| 213 | |
| 214 | // SetAppBuildProbe stamps the deploy watch's probe path. "" clears it back to |
| 215 | // header-watching. |
| 216 | func (s *Store) SetAppBuildProbe(accountID int64, app, path string) error { |
| 217 | _, err := s.db.Exec(`UPDATE apps SET build_probe = ? WHERE account_id = ? AND app = ?`, |
| 218 | path, accountID, app) |
| 219 | return err |
| 220 | } |
| 221 | ``` |
| 222 | |
| 223 | Confirm `errors` and `database/sql` are already imported in `store.go`; add if not. |
| 224 | |
| 225 | - [ ] **Step 6: Run the tests to verify they pass** |
| 226 | |
| 227 | ``` |
| 228 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 229 | go test ./internal/console/ -run 'TestAppFlags' -count=1 -v |
| 230 | ``` |
| 231 | |
| 232 | Expected: both PASS. |
| 233 | |
| 234 | - [ ] **Step 7: Full gate** |
| 235 | |
| 236 | ``` |
| 237 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 238 | go build ./... && go vet ./... && GOOS=darwin go build ./... && |
| 239 | go test ./... -count=1' |
| 240 | gofmt -l internal/console/store.go internal/console/migrate_test.go |
| 241 | ``` |
| 242 | |
| 243 | Expected: no failures, no gofmt output. |
| 244 | |
| 245 | - [ ] **Step 8: Commit** |
| 246 | |
| 247 | ```bash |
| 248 | git add internal/console/store.go internal/console/migrate_test.go |
| 249 | git commit -m "console: control_plane and build_probe on the app row |
| 250 | |
| 251 | Both in the ALTER block AND in migrateAppsKey's apps_new copy list. The |
| 252 | rebuild copies an enumerated column list, so a column in only the first |
| 253 | place is silently dropped on a database that has not been rebuilt yet — |
| 254 | which is a self-hoster jumping versions, not a hypothetical. deleted_at |
| 255 | and placement are already in both for this reason (store.go:280); the new |
| 256 | test builds a pre-rebuild database by hand and proves it." |
| 257 | ``` |
| 258 | |
| 259 | --- |
| 260 | |
| 261 | ### Task 2: The #79 owner exemption |
| 262 | |
| 263 | **Files:** |
| 264 | - Modify: `internal/console/server.go:2426` (`claimApp`), `internal/console/server.go:2518-2545` (`handleNewApp`), `internal/console/apicli.go:394` (`handleAPICreateApp`) |
| 265 | - Test: `internal/console/apps_claim_test.go` (create) |
| 266 | |
| 267 | **Interfaces:** |
| 268 | - Consumes: nothing from Task 1. |
| 269 | - Produces: `claimApp` gains a final parameter — `func (s *Server) claimApp(ctx context.Context, acct *Account, name, addr, placement string, ownerClaim bool) (outcome claimOutcome, placed string)`. |
| 270 | |
| 271 | **The reasoning** (spec §2): `apps/<sqid>/<app>/` is namespaced **by account**, so objects under `apps/bab/` can only be `ops`'s. An owner of `bab` claiming a name under `bab`'s own prefix is adopting their own objects, not inheriting a stranger's. The guard is right for a stranger and wrong for the owner. |
| 272 | |
| 273 | - [ ] **Step 1: Write the failing tests** |
| 274 | |
| 275 | Create `internal/console/apps_claim_test.go`: |
| 276 | |
| 277 | ```go |
| 278 | package console |
| 279 | |
| 280 | import ( |
| 281 | "context" |
| 282 | "net/http" |
| 283 | "testing" |
| 284 | "time" |
| 285 | |
| 286 | "github.com/carlosframework/platform/internal/svcrole" |
| 287 | ) |
| 288 | |
| 289 | // seedBucketApp puts an object under an account's own app prefix WITHOUT a |
| 290 | // row, which is exactly what a direct-bucket ship does (direct mode skips the |
| 291 | // claim) and exactly the state that makes #79 unrecoverable: the objects |
| 292 | // block the claim, and no pass can discover a row that does not exist. |
| 293 | func seedBucketApp(t *testing.T, fx *consoleFixture, sqid, app string) { |
| 294 | t.Helper() |
| 295 | key := "apps/" + sqid + "/" + app + "/channels/canary-console.json" |
| 296 | if err := fx.rel.Put(context.Background(), key, []byte(`{}`)); err != nil { |
| 297 | t.Fatalf("seed %s: %v", key, err) |
| 298 | } |
| 299 | } |
| 300 | |
| 301 | // TestOwnerClaimsAppWhoseObjectsExist is #79's fix. Before it, an app first |
| 302 | // shipped direct-to-bucket could never be claimed and so was permanently |
| 303 | // invisible to the console, with no recovery but deleting its release history. |
| 304 | func TestOwnerClaimsAppWhoseObjectsExist(t *testing.T) { |
| 305 | fx := newConsoleFixture(t) |
| 306 | seedBucketApp(t, fx, "bab", "console") |
| 307 | acct, err := fx.st.Account(OpsAccountID) |
| 308 | if err != nil { |
| 309 | t.Fatal(err) |
| 310 | } |
| 311 | outcome, _ := fx.srv.claimApp(context.Background(), acct, "console", "paul@keymail.dev", "", true) |
| 312 | if outcome != claimOK { |
| 313 | t.Fatalf("owner claim over existing objects = %v, want claimOK", outcome) |
| 314 | } |
| 315 | has, err := fx.st.HasAppIn(OpsAccountID, "console") |
| 316 | if err != nil || !has { |
| 317 | t.Fatalf("HasAppIn after claim = %v %v, want true", has, err) |
| 318 | } |
| 319 | } |
| 320 | |
| 321 | // TestNonOwnerStillRefusedOverExistingObjects: the exemption is OWNERSHIP, not |
| 322 | // membership. Adopting existing release history is an ownership act. |
| 323 | func TestNonOwnerStillRefusedOverExistingObjects(t *testing.T) { |
| 324 | fx := newConsoleFixture(t) |
| 325 | seedBucketApp(t, fx, "bab", "console") |
| 326 | acct, err := fx.st.Account(OpsAccountID) |
| 327 | if err != nil { |
| 328 | t.Fatal(err) |
| 329 | } |
| 330 | outcome, _ := fx.srv.claimApp(context.Background(), acct, "console", "member@keymail.dev", "", false) |
| 331 | if outcome != claimTaken { |
| 332 | t.Fatalf("non-owner claim over existing objects = %v, want claimTaken", outcome) |
| 333 | } |
| 334 | } |
| 335 | |
| 336 | // TestLegacyPrefixStillBlocksAnOwner: the exemption covers the ACCOUNT- |
| 337 | // QUALIFIED prefix only. Legacy objects at apps/<app>/ predate account scoping |
| 338 | // and are owed an `accounts migrate -from legacy`; handing the name out would |
| 339 | // put an app in the way of that migration. |
| 340 | func TestLegacyPrefixStillBlocksAnOwner(t *testing.T) { |
| 341 | fx := newConsoleFixture(t) |
| 342 | if err := fx.rel.Put(context.Background(), "apps/oldsite/channels/stable.json", []byte(`{}`)); err != nil { |
| 343 | t.Fatal(err) |
| 344 | } |
| 345 | acct, err := fx.st.Account(OpsAccountID) |
| 346 | if err != nil { |
| 347 | t.Fatal(err) |
| 348 | } |
| 349 | outcome, _ := fx.srv.claimApp(context.Background(), acct, "oldsite", "paul@keymail.dev", "", true) |
| 350 | if outcome != claimTaken { |
| 351 | t.Fatalf("owner claim over a LEGACY prefix = %v, want claimTaken — those objects are owed a migrate", outcome) |
| 352 | } |
| 353 | } |
| 354 | |
| 355 | // TestGrantStillBlocksAnOwner: a CLI-granted app in ops stays claim-proof. |
| 356 | // Verified live on flagship 2026-08-23: derp, meep and platform-site hold |
| 357 | // grants; console does not, which is why the exemption is sufficient for it. |
| 358 | func TestGrantStillBlocksAnOwner(t *testing.T) { |
| 359 | fx := newConsoleFixture(t) |
| 360 | if err := fx.st.Grant("someone@keymail.dev", "derp"); err != nil { |
| 361 | t.Fatal(err) |
| 362 | } |
| 363 | acct, err := fx.st.Account(OpsAccountID) |
| 364 | if err != nil { |
| 365 | t.Fatal(err) |
| 366 | } |
| 367 | outcome, _ := fx.srv.claimApp(context.Background(), acct, "derp", "paul@keymail.dev", "", true) |
| 368 | if outcome != claimTaken { |
| 369 | t.Fatalf("owner claim over a GRANTED name = %v, want claimTaken", outcome) |
| 370 | } |
| 371 | } |
| 372 | |
| 373 | // TestServicePrincipalNeverClaimsOverObjects: a service has no role at all, so |
| 374 | // ownerClaim must be pinned FALSE for it. An account-scoped PrincipalService |
| 375 | // legitimately reaches claimApp (apicli.go blocks only app-scoped ones), so |
| 376 | // without the pin the exemption would silently hand every CI credential the |
| 377 | // power to adopt release history. |
| 378 | func TestServicePrincipalNeverClaimsOverObjects(t *testing.T) { |
| 379 | fx := newConsoleFixture(t) |
| 380 | seedBucketApp(t, fx, "bab", "console") |
| 381 | svc := serviceToken(t, fx.st, OpsAccountID, "hub", "", svcrole.Publish) |
| 382 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab", map[string]string{"name": "console"}, svc) |
| 383 | if status != http.StatusConflict { |
| 384 | t.Fatalf("service claim over existing objects = %d %v, want 409 (taken)", status, out) |
| 385 | } |
| 386 | has, err := fx.st.HasAppIn(OpsAccountID, "console") |
| 387 | if err != nil || has { |
| 388 | t.Fatalf("HasAppIn after a service claim = %v %v, want false — a service must never adopt release history", has, err) |
| 389 | } |
| 390 | } |
| 391 | ``` |
| 392 | |
| 393 | Two things to check against the codebase before running, and fix in the test if they differ: the release-store `Put` helper's name on `fx.rel` (grep `fx.rel.` in existing tests), and `handleAPICreateApp`'s success/failure status codes and request path (read `apicli.go` around line 394 and the route registration in `authcli.go`). Use whatever the existing create tests use. |
| 394 | |
| 395 | - [ ] **Step 2: Run to verify they fail** |
| 396 | |
| 397 | ``` |
| 398 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 399 | go test ./internal/console/ -run 'Claim|ClaimsApp|Refused|Blocks|ServicePrincipalNever' -count=1 |
| 400 | ``` |
| 401 | |
| 402 | Expected: FAIL — `too many arguments in call to fx.srv.claimApp`. |
| 403 | |
| 404 | - [ ] **Step 3: Add the parameter and the exemption** |
| 405 | |
| 406 | In `internal/console/server.go`, change the signature and the bucket guard. Extend `claimApp`'s doc comment with the new paragraph: |
| 407 | |
| 408 | ```go |
| 409 | // ownerClaim exempts the ACCOUNT-QUALIFIED prefix check below, and nothing |
| 410 | // else. The guard exists so a stranger cannot inherit objects somebody else |
| 411 | // shipped — but apps/<sqid>/<app>/ is namespaced BY ACCOUNT, so objects under |
| 412 | // a given sqid can only be that account's. An OWNER of that account claiming |
| 413 | // a name under its own prefix is adopting their own release history, which is |
| 414 | // the whole of issue #79: an app first shipped direct-to-bucket (direct mode |
| 415 | // skips the claim) could otherwise never be given a row, and so stayed |
| 416 | // invisible to the console forever with no recovery but deleting its history. |
| 417 | // |
| 418 | // It is OWNER, not member: adopting existing release history is an ownership |
| 419 | // act. It is FALSE for every service principal — a service has no role at |
| 420 | // all, and an account-scoped one legitimately reaches this function, so |
| 421 | // without an explicit pin the exemption would hand every CI credential the |
| 422 | // same power. |
| 423 | func (s *Server) claimApp(ctx context.Context, acct *Account, name, addr, placement string, ownerClaim bool) (outcome claimOutcome, placed string) { |
| 424 | ``` |
| 425 | |
| 426 | Then the prefix loop. The legacy prefix must keep refusing, so the exemption applies per-prefix, not to the loop: |
| 427 | |
| 428 | ```go |
| 429 | for _, prefix := range prefixes { |
| 430 | keys, err := s.cfg.Releases.List(ctx, prefix) |
| 431 | if err != nil { |
| 432 | return claimStoreFailed, "" |
| 433 | } |
| 434 | if len(keys) == 0 { |
| 435 | continue |
| 436 | } |
| 437 | // The owner exemption covers the account's OWN prefix only. The |
| 438 | // legacy unqualified one (ops only) still refuses whoever is asking: |
| 439 | // those objects are owed an `accounts migrate -from legacy`, and a |
| 440 | // name handed out over them puts an app in that migration's way. |
| 441 | if ownerClaim && prefix != release.LegacyAppPrefix(name) { |
| 442 | continue |
| 443 | } |
| 444 | return claimTaken, "" |
| 445 | } |
| 446 | ``` |
| 447 | |
| 448 | - [ ] **Step 4: Update both call sites** |
| 449 | |
| 450 | `internal/console/apicli.go:394` — a service principal is pinned false: |
| 451 | |
| 452 | ```go |
| 453 | // ownerClaim: a PERSON who owns this account may adopt objects already |
| 454 | // under its own prefix (#79). A service principal never may — it holds no |
| 455 | // account role, and handing a CI credential the power to adopt release |
| 456 | // history is exactly the widening svcrole exists to prevent. |
| 457 | ownerClaim := p.Kind == PrincipalUser && acct.Role == RoleOwner |
| 458 | outcome, placed := s.claimApp(r.Context(), acct, name, addr, place, ownerClaim) |
| 459 | ``` |
| 460 | |
| 461 | `internal/console/server.go:2542` — the browser door already has the role and was discarding it: |
| 462 | |
| 463 | ```go |
| 464 | addr, raw, acct, role, ok := s.account(w, r) |
| 465 | ``` |
| 466 | |
| 467 | ```go |
| 468 | outcome, _ := s.claimApp(r.Context(), acct, name, addr, "", role == RoleOwner) |
| 469 | ``` |
| 470 | |
| 471 | And correct `handleNewApp`'s doc comment, whose stated policy this changes: |
| 472 | |
| 473 | ```go |
| 474 | // handleNewApp is the browser door onto claimApp: the account page's New app |
| 475 | // form. Any member may claim a FREE name — creation grants nothing promotion |
| 476 | // doesn't already gate — and unlike the CLI door it carries no rate limit: a |
| 477 | // session behind CSRF is not the stolen-credential shape the API budget |
| 478 | // exists for, and this form is the escape hatch when that budget is honestly |
| 479 | // spent. |
| 480 | // |
| 481 | // An OWNER may additionally claim a name whose objects already exist under |
| 482 | // this account's own prefix (#79, claimApp's ownerClaim). A plain member |
| 483 | // cannot: adopting existing release history is an ownership act. |
| 484 | ``` |
| 485 | |
| 486 | - [ ] **Step 5: Run the tests to verify they pass** |
| 487 | |
| 488 | ``` |
| 489 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 490 | go test ./internal/console/ -run 'Claim|ClaimsApp|Refused|Blocks|ServicePrincipalNever' -count=1 -v |
| 491 | ``` |
| 492 | |
| 493 | Expected: all five PASS. |
| 494 | |
| 495 | - [ ] **Step 6: Full gate, then commit** |
| 496 | |
| 497 | ```bash |
| 498 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 499 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 500 | gofmt -l internal/console/server.go internal/console/apicli.go internal/console/apps_claim_test.go |
| 501 | git add internal/console/server.go internal/console/apicli.go internal/console/apps_claim_test.go |
| 502 | git commit -m "console: an owner may claim a name whose own-account objects exist (#79) |
| 503 | |
| 504 | apps/<sqid>/<app>/ is namespaced BY ACCOUNT, so objects under a sqid can |
| 505 | only be that account's — an owner claiming there is adopting their own |
| 506 | release history, not inheriting a stranger's. That is the whole of #79: an |
| 507 | app first shipped direct-to-bucket was otherwise unclaimable forever, with |
| 508 | no recovery but deleting its history. |
| 509 | |
| 510 | Owner, not member. Pinned FALSE for every service principal: an |
| 511 | account-scoped one reaches claimApp and holds no account role, so without |
| 512 | the pin every CI credential could adopt release history. The legacy |
| 513 | unqualified prefix still refuses everyone — those objects are owed a |
| 514 | migrate and the name must stay out of its way." |
| 515 | ``` |
| 516 | |
| 517 | --- |
| 518 | |
| 519 | ### Task 3: Set the flags at claim time |
| 520 | |
| 521 | **Files:** |
| 522 | - Modify: `internal/console/apicli.go` (`handleAPICreateApp` request struct and post-claim stamping) |
| 523 | - Modify: `apps.go:379` (`runAppsCreate`) |
| 524 | - Test: `internal/console/apps_claim_test.go`, `apps_test.go` |
| 525 | |
| 526 | **Interfaces:** |
| 527 | - Consumes: `Store.SetAppControlPlane`, `Store.SetAppBuildProbe` (Task 1); `claimApp(..., ownerClaim)` (Task 2). |
| 528 | - Produces: `carlos apps create` accepts `-control-plane` and `-build-probe <path>`; the API accepts `"control_plane": bool` and `"build_probe": string`. |
| 529 | |
| 530 | **Why at claim time, not a settings page:** the runbook claims `ops/console` **with the flag already set** (spec §10 step 10). A row that exists without the flag is a window in which any ops member or `Publish` service principal can roll the production console. One step, no window. A settings-page toggle is deferred. |
| 531 | |
| 532 | - [ ] **Step 1: Write the failing tests** |
| 533 | |
| 534 | Add to `internal/console/apps_claim_test.go`: |
| 535 | |
| 536 | ```go |
| 537 | // TestCreateAppStampsControlPlaneAndProbe: the flags are set in the SAME call |
| 538 | // that claims the name. Claiming first and flagging second would leave a |
| 539 | // window in which the production console is an ordinary app any ops member |
| 540 | // could roll. |
| 541 | func TestCreateAppStampsControlPlaneAndProbe(t *testing.T) { |
| 542 | fx := newConsoleFixture(t) |
| 543 | seedBucketApp(t, fx, "bab", "console") |
| 544 | token := cliToken(t, fx, "paul@keymail.dev", "code-create-cp") |
| 545 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab", map[string]any{ |
| 546 | "name": "console", "control_plane": true, "build_probe": "/healthz", |
| 547 | }, token) |
| 548 | if status != http.StatusCreated { |
| 549 | t.Fatalf("create = %d %v, want 201", status, out) |
| 550 | } |
| 551 | cp, probe, err := fx.st.AppFlags(OpsAccountID, "console") |
| 552 | if err != nil { |
| 553 | t.Fatal(err) |
| 554 | } |
| 555 | if !cp || probe != "/healthz" { |
| 556 | t.Fatalf("flags after create = %v %q, want true \"/healthz\"", cp, probe) |
| 557 | } |
| 558 | } |
| 559 | |
| 560 | // TestOnlyAnOwnerSetsControlPlane: the flag refuses destruction, so setting it |
| 561 | // is an ownership act — a member who could set it could also make an app |
| 562 | // undeletable by its owner. |
| 563 | func TestOnlyAnOwnerSetsControlPlane(t *testing.T) { |
| 564 | fx := newConsoleFixture(t) |
| 565 | if err := fx.st.AddMember(OpsAccountID, "member@keymail.dev", RoleMember, time.Now()); err != nil { |
| 566 | t.Fatal(err) |
| 567 | } |
| 568 | token := cliToken(t, fx, "member@keymail.dev", "code-create-cp-member") |
| 569 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab", map[string]any{ |
| 570 | "name": "notyours", "control_plane": true, |
| 571 | }, token) |
| 572 | if status != http.StatusForbidden { |
| 573 | t.Fatalf("member setting control_plane = %d %v, want 403", status, out) |
| 574 | } |
| 575 | has, err := fx.st.HasAppIn(OpsAccountID, "notyours") |
| 576 | if err != nil || has { |
| 577 | t.Fatalf("the app was claimed anyway (%v) — a refused flag must refuse the WHOLE create, like -place does", has) |
| 578 | } |
| 579 | } |
| 580 | |
| 581 | // TestBuildProbeMustBeAnAbsolutePath: the probe is a URL path the deploy watch |
| 582 | // dials. Anything else is a typo that would otherwise surface as a deploy that |
| 583 | // times out for reasons nobody can see. |
| 584 | func TestBuildProbeMustBeAnAbsolutePath(t *testing.T) { |
| 585 | fx := newConsoleFixture(t) |
| 586 | token := cliToken(t, fx, "paul@keymail.dev", "code-create-probe-bad") |
| 587 | for _, bad := range []string{"healthz", "http://x/healthz", "/healthz?x=1", "/health z"} { |
| 588 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab", map[string]any{ |
| 589 | "name": "probe" + bad[:1], "build_probe": bad, |
| 590 | }, token) |
| 591 | if status != http.StatusBadRequest { |
| 592 | t.Fatalf("build_probe %q = %d %v, want 400", bad, status, out) |
| 593 | } |
| 594 | } |
| 595 | } |
| 596 | ``` |
| 597 | |
| 598 | Confirm `RoleMember`'s actual identifier by grepping `RoleOwner` in `internal/console`; use whatever the member constant is really called. |
| 599 | |
| 600 | - [ ] **Step 2: Run to verify they fail** |
| 601 | |
| 602 | ``` |
| 603 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 604 | go test ./internal/console/ -run 'CreateAppStamps|OnlyAnOwnerSets|BuildProbeMust' -count=1 |
| 605 | ``` |
| 606 | |
| 607 | Expected: FAIL — flags ignored, 201 where 403/400 expected. |
| 608 | |
| 609 | - [ ] **Step 3: Extend the API door** |
| 610 | |
| 611 | In `internal/console/apicli.go`, `handleAPICreateApp`, extend the request struct and validate **before** `claimApp` — the create limiter charges attempts, so a shape the server is certain to refuse must not cost a claim: |
| 612 | |
| 613 | ```go |
| 614 | var req struct { |
| 615 | Name string `json:"name"` |
| 616 | Place string `json:"place"` |
| 617 | ControlPlane bool `json:"control_plane"` |
| 618 | BuildProbe string `json:"build_probe"` |
| 619 | } |
| 620 | ``` |
| 621 | |
| 622 | ```go |
| 623 | // Validated BEFORE claimApp, like place: a refused flag refuses the WHOLE |
| 624 | // create rather than leaving a claimed name with the wrong shape, and a |
| 625 | // shape the server is certain to refuse costs no claim attempt. |
| 626 | if req.ControlPlane && acct.Role != RoleOwner { |
| 627 | jsonError(w, http.StatusForbidden, |
| 628 | "only an owner can mark an app as the control plane") |
| 629 | return |
| 630 | } |
| 631 | probe := strings.TrimSpace(req.BuildProbe) |
| 632 | if probe != "" && !validBuildProbe(probe) { |
| 633 | jsonError(w, http.StatusBadRequest, |
| 634 | `build_probe must be an absolute URL path with no query, e.g. "/healthz"`) |
| 635 | return |
| 636 | } |
| 637 | ``` |
| 638 | |
| 639 | Add the validator near `normalizeAppName` in `server.go`: |
| 640 | |
| 641 | ```go |
| 642 | // validBuildProbe bounds the deploy watch's probe path to what it actually |
| 643 | // dials: an absolute path, no query, no spaces. The watch appends its own |
| 644 | // cache-busting query parameter (deploy.go), so a probe carrying one would |
| 645 | // produce a URL with two — and a probe that is a full URL would silently |
| 646 | // point the watch at another host entirely, which is the one mistake here |
| 647 | // that could report a DIFFERENT deployment's build as this one's. |
| 648 | func validBuildProbe(p string) bool { |
| 649 | if !strings.HasPrefix(p, "/") || strings.ContainsAny(p, "?# \t") { |
| 650 | return false |
| 651 | } |
| 652 | return len(p) <= 256 |
| 653 | } |
| 654 | ``` |
| 655 | |
| 656 | Then stamp after a successful claim, beside the existing placement stamp: |
| 657 | |
| 658 | ```go |
| 659 | if req.ControlPlane { |
| 660 | if err := s.store.SetAppControlPlane(acct.ID, name, true); err != nil { |
| 661 | log.Printf("console: claim %s/%s: set control_plane: %v", acct.Sqid, name, err) |
| 662 | } |
| 663 | } |
| 664 | if probe != "" { |
| 665 | if err := s.store.SetAppBuildProbe(acct.ID, name, probe); err != nil { |
| 666 | log.Printf("console: claim %s/%s: set build_probe: %v", acct.Sqid, name, err) |
| 667 | } |
| 668 | } |
| 669 | ``` |
| 670 | |
| 671 | Echo both in the response body alongside `placement`, for the same reason `placed` is echoed: a caller that cannot see what was actually stored cannot tell "claimed and flagged" from "claimed, flag silently didn't". |
| 672 | |
| 673 | - [ ] **Step 4: Extend the CLI** |
| 674 | |
| 675 | In `apps.go`, `runAppsCreate`: |
| 676 | |
| 677 | ```go |
| 678 | controlPlane := fs.Bool("control-plane", false, |
| 679 | "mark the app as the control plane: it refuses trash, purge, rename and transfer, "+ |
| 680 | "and refuses a service credential's promote (owner only)") |
| 681 | buildProbe := fs.String("build-probe", "", |
| 682 | `path whose response body carries the running build, e.g. "/healthz" — `+ |
| 683 | "`carlos deploy` polls it instead of the X-Carlos-Version header, which the edge "+ |
| 684 | "stamps from the route row and which therefore lies until the process restarts") |
| 685 | ``` |
| 686 | |
| 687 | Extend the usage string with `[-control-plane] [-build-probe /path]`, send both fields in the request body, and print what came back in the echo so the operator sees the stored truth rather than what they asked for. |
| 688 | |
| 689 | - [ ] **Step 5: Run the tests to verify they pass** |
| 690 | |
| 691 | ``` |
| 692 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 693 | go test ./internal/console/ ./ -run 'CreateAppStamps|OnlyAnOwnerSets|BuildProbeMust|AppsCreate' -count=1 -v |
| 694 | ``` |
| 695 | |
| 696 | Expected: PASS. |
| 697 | |
| 698 | - [ ] **Step 6: Full gate, then commit** |
| 699 | |
| 700 | ```bash |
| 701 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 702 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 703 | gofmt -l internal/console/apicli.go internal/console/server.go apps.go internal/console/apps_claim_test.go |
| 704 | git add -A |
| 705 | git commit -m "apps create: -control-plane and -build-probe, stamped with the claim |
| 706 | |
| 707 | Set in the SAME call that claims the name. Claiming first and flagging |
| 708 | second leaves a window in which the production console is an ordinary app |
| 709 | any ops member — or any Publish service credential — could roll. |
| 710 | |
| 711 | Both validated before claimApp, like -place: a refused flag refuses the |
| 712 | whole create rather than leaving a claimed name with the wrong shape, and |
| 713 | costs no attempt against the per-day create budget. Both echoed in the |
| 714 | response, so a caller can tell 'claimed and flagged' from 'claimed, flag |
| 715 | silently didn't'." |
| 716 | ``` |
| 717 | |
| 718 | --- |
| 719 | |
| 720 | ### Task 4: The flag refuses destruction, and refuses a service's promote |
| 721 | |
| 722 | **Files:** |
| 723 | - Modify: `internal/console/ownership.go:293` (`handlePurgeApp`), `internal/console/appsettings.go` (`handleAppRename`, `handleAppDelete`, `handleAppMoveAccount`) |
| 724 | - Modify: `internal/console/apicli.go` (`handleAPITrashApp`, `handleAPIPromote`, `handleAPIRollback`) |
| 725 | - Test: `internal/console/controlplane_test.go` (create) |
| 726 | |
| 727 | **Interfaces:** |
| 728 | - Consumes: `Store.AppFlags` (Task 1). |
| 729 | - Produces: `func (s *Server) refuseControlPlane(w http.ResponseWriter, accountID int64, app, verb string) bool` — returns true when it wrote a refusal and the caller must stop. |
| 730 | |
| 731 | **The inversion** (spec §6): an earlier draft gated *promotion* behind fresh auth. That would have made `carlos deploy -app console` ship and then die every time, because `stable`'s 403 has no API-side step-up and `deploy.go:206-208` `log.Fatalf`s after the ship lands. Deploying the console should be easy; destroying it should be hard — and `handlePurgeApp` can currently delete `apps/bab/console/**`, the release history the fleet is *running*, behind an ordinary confirmation. |
| 732 | |
| 733 | - [ ] **Step 1: Write the failing tests** |
| 734 | |
| 735 | Create `internal/console/controlplane_test.go`: |
| 736 | |
| 737 | ```go |
| 738 | package console |
| 739 | |
| 740 | import ( |
| 741 | "net/http" |
| 742 | "testing" |
| 743 | "time" |
| 744 | |
| 745 | "github.com/carlosframework/platform/internal/svcrole" |
| 746 | ) |
| 747 | |
| 748 | func controlPlaneFixture(t *testing.T, label string) (*consoleFixture, string) { |
| 749 | t.Helper() |
| 750 | fx := newConsoleFixture(t) |
| 751 | if _, err := fx.st.CreateAppIn(OpsAccountID, "console", "paul@keymail.dev", time.Now()); err != nil { |
| 752 | t.Fatal(err) |
| 753 | } |
| 754 | if err := fx.st.SetAppControlPlane(OpsAccountID, "console", true); err != nil { |
| 755 | t.Fatal(err) |
| 756 | } |
| 757 | return fx, cliToken(t, fx, "paul@keymail.dev", label) |
| 758 | } |
| 759 | |
| 760 | // TestControlPlaneAppRefusesTrash: the console's release history is not |
| 761 | // something to delete behind a confirmation dialog. Refused outright, not |
| 762 | // "with fresh auth" — a gesture is not the guard here, the flag is. |
| 763 | func TestControlPlaneAppRefusesTrash(t *testing.T) { |
| 764 | fx, token := controlPlaneFixture(t, "code-cp-trash") |
| 765 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab/console/delete", map[string]string{}, token) |
| 766 | if status != http.StatusForbidden { |
| 767 | t.Fatalf("trash a control-plane app = %d %v, want 403", status, out) |
| 768 | } |
| 769 | if msg := str(t, out, "error"); !strings.Contains(msg, "control plane") { |
| 770 | t.Fatalf("refusal %q does not name the flag the operator has to clear", msg) |
| 771 | } |
| 772 | trashed, err := fx.st.HasAppIn(OpsAccountID, "console") |
| 773 | if err != nil || !trashed { |
| 774 | t.Fatalf("app row after a refused trash: HasAppIn = %v %v, want true (untouched)", trashed, err) |
| 775 | } |
| 776 | } |
| 777 | |
| 778 | // TestControlPlaneAppAllowsAPersonToPromote is the other half, and the one |
| 779 | // that matters most: gating promotion is what an earlier draft got wrong. |
| 780 | // `stable`'s fresh-auth 403 has no API-side step-up and carlos deploy |
| 781 | // log.Fatalfs on a failed promote AFTER the ship has landed, so a promote gate |
| 782 | // would break the very command this work exists to create. |
| 783 | func TestControlPlaneAppAllowsAPersonToPromote(t *testing.T) { |
| 784 | fx, token := controlPlaneFixture(t, "code-cp-promote") |
| 785 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/console/releases", token, |
| 786 | []shipPart{metaPart(t, "cp00001", "binary"), {name: "linux-arm64", body: []byte("x")}}) |
| 787 | if status != http.StatusCreated { |
| 788 | t.Fatalf("ship = %d %v", status, out) |
| 789 | } |
| 790 | status, out = postJSON(t, fx.ts.URL+"/api/cli/apps/bab/console/promote", |
| 791 | map[string]string{"version": "cp00001", "channel": "canary/console"}, token) |
| 792 | if status != http.StatusOK { |
| 793 | t.Fatalf("a PERSON promoting a control-plane app = %d %v, want 200 — gating this breaks carlos deploy", status, out) |
| 794 | } |
| 795 | } |
| 796 | |
| 797 | // TestControlPlaneAppRefusesAServicePromote: a person with ops membership may |
| 798 | // roll the console; a CI credential may not. This is what the flag buys on the |
| 799 | // promote path instead of a fresh-auth gate. |
| 800 | func TestControlPlaneAppRefusesAServicePromote(t *testing.T) { |
| 801 | fx, token := controlPlaneFixture(t, "code-cp-svc") |
| 802 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/console/releases", token, |
| 803 | []shipPart{metaPart(t, "cp00002", "binary"), {name: "linux-arm64", body: []byte("x")}}) |
| 804 | if status != http.StatusCreated { |
| 805 | t.Fatalf("ship = %d %v", status, out) |
| 806 | } |
| 807 | svc := serviceToken(t, fx.st, OpsAccountID, "ci", "console", svcrole.Publish) |
| 808 | status, out = postJSON(t, fx.ts.URL+"/api/cli/apps/bab/console/promote", |
| 809 | map[string]string{"version": "cp00002", "channel": "canary/console"}, svc) |
| 810 | if status != http.StatusForbidden { |
| 811 | t.Fatalf("a SERVICE promoting a control-plane app = %d %v, want 403", status, out) |
| 812 | } |
| 813 | } |
| 814 | |
| 815 | // TestOrdinaryAppUnaffected: every refusal above is scoped to the flag. |
| 816 | func TestOrdinaryAppUnaffected(t *testing.T) { |
| 817 | fx := newConsoleFixture(t) |
| 818 | if _, err := fx.st.CreateAppIn(OpsAccountID, "titogo", "paul@keymail.dev", time.Now()); err != nil { |
| 819 | t.Fatal(err) |
| 820 | } |
| 821 | token := cliToken(t, fx, "paul@keymail.dev", "code-cp-ordinary") |
| 822 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab/titogo/delete", map[string]string{}, token) |
| 823 | if status == http.StatusForbidden { |
| 824 | t.Fatalf("trash an ORDINARY app = %d %v, want it to work as before", status, out) |
| 825 | } |
| 826 | } |
| 827 | |
| 828 | // TestControlPlaneRefusalIsNotAMembershipOracle: a non-member must meet the |
| 829 | // byte-identical 404 in apiAccountApp before the flag is ever consulted, or |
| 830 | // the refusal becomes a way to learn which accounts hold a control-plane app. |
| 831 | func TestControlPlaneRefusalIsNotAMembershipOracle(t *testing.T) { |
| 832 | fx, _ := controlPlaneFixture(t, "code-cp-oracle") |
| 833 | stranger := cliToken(t, fx, "eve@keymail.dev", "code-cp-stranger") |
| 834 | status, out := postJSON(t, fx.ts.URL+"/api/cli/apps/bab/console/delete", map[string]string{}, stranger) |
| 835 | if status != http.StatusNotFound { |
| 836 | t.Fatalf("a non-member = %d %v, want the byte-identical 404", status, out) |
| 837 | } |
| 838 | } |
| 839 | ``` |
| 840 | |
| 841 | Add `strings` to the imports. Check `handleAPIPromote`'s real request field names (read `apicli.go` around the promote handler) and use them; the ones above are a guess and must be corrected to match. |
| 842 | |
| 843 | - [ ] **Step 2: Run to verify they fail** |
| 844 | |
| 845 | ``` |
| 846 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 847 | go test ./internal/console/ -run 'ControlPlane|OrdinaryAppUnaffected' -count=1 |
| 848 | ``` |
| 849 | |
| 850 | Expected: FAIL — trash succeeds, service promote succeeds. |
| 851 | |
| 852 | - [ ] **Step 3: Add the shared refusal** |
| 853 | |
| 854 | In `internal/console/controlplane.go` (create): |
| 855 | |
| 856 | ```go |
| 857 | // Control-plane apps: the flag that says "this app IS the platform". |
| 858 | // |
| 859 | // It gates DESTRUCTION, not promotion, and the asymmetry is the whole design. |
| 860 | // Deploying the console should be as ordinary as deploying any app — that is |
| 861 | // what this slice exists to make true — so a promote gate would defeat it: |
| 862 | // `stable`'s fresh-auth 403 has no API-side step-up, and `carlos deploy` |
| 863 | // log.Fatalfs on a failed promote AFTER the ship has already landed |
| 864 | // (deploy.go:206-208), so any gate here would leave every console deploy |
| 865 | // half-done. Destroying the console is the act that deserves a wall, and it |
| 866 | // had none: handlePurgeApp will delete apps/<sqid>/console/** — the release |
| 867 | // history the fleet is currently running — behind an owner check, a typed |
| 868 | // name and a passkey, none of which knows what this app is. |
| 869 | // |
| 870 | // Refused OUTRIGHT rather than behind fresh auth. A gesture proves intent, |
| 871 | // and intent is not the thing in doubt: an operator who means to delete the |
| 872 | // console can clear the flag first, which is a separate, deliberate act with |
| 873 | // its own record. |
| 874 | package console |
| 875 | |
| 876 | import ( |
| 877 | "log" |
| 878 | "net/http" |
| 879 | ) |
| 880 | |
| 881 | // refuseControlPlane writes a refusal and returns true when app is the control |
| 882 | // plane. Callers MUST be past their own membership gate first — the |
| 883 | // byte-identical 404 protects account membership, and consulting the flag |
| 884 | // before it would turn this refusal into an oracle for which accounts hold a |
| 885 | // control-plane app. |
| 886 | // |
| 887 | // A lookup failure refuses too. The alternative — treating an unreadable row |
| 888 | // as "not the control plane" — makes a database blip the way past the guard. |
| 889 | func (s *Server) refuseControlPlane(w http.ResponseWriter, accountID int64, app, verb string) bool { |
| 890 | cp, _, err := s.store.AppFlags(accountID, app) |
| 891 | if err != nil { |
| 892 | log.Printf("console: control-plane check %s/%d: %v", app, accountID, err) |
| 893 | jsonError(w, http.StatusServiceUnavailable, "could not check this app's control-plane status — nothing was changed") |
| 894 | return true |
| 895 | } |
| 896 | if !cp { |
| 897 | return false |
| 898 | } |
| 899 | jsonError(w, http.StatusForbidden, app+" is marked as the control plane, so it cannot be "+verb+ |
| 900 | " — clear the control-plane flag first if you really mean to") |
| 901 | return true |
| 902 | } |
| 903 | ``` |
| 904 | |
| 905 | For the browser doors, which render pages rather than JSON, add a sibling that sets a flash and redirects, following whatever pattern `handleAppRename` already uses for a refusal. |
| 906 | |
| 907 | - [ ] **Step 4: Wire the four destructive doors and the promote pair** |
| 908 | |
| 909 | Each call goes **after** the existing membership/owner gate and **before** any write or any fresh-auth prompt — refusing before a passkey round trip is both faster and clearer than asking for a gesture that cannot help: |
| 910 | |
| 911 | - `internal/console/apicli.go`, `handleAPITrashApp`: `if s.refuseControlPlane(w, acct.ID, app, "trashed") { return }` |
| 912 | - `internal/console/appsettings.go`, `handleAppDelete`: refuse with `"trashed"` |
| 913 | - `internal/console/appsettings.go`, `handleAppRename`: refuse with `"renamed"` |
| 914 | - `internal/console/appsettings.go`, `handleAppMoveAccount`: refuse with `"moved to another account"` |
| 915 | - `internal/console/ownership.go`, `handlePurgeApp`: refuse with `"deleted"` — this door takes the app from the trash form, so read the name it resolved |
| 916 | |
| 917 | For the promote pair, in `handleAPIPromote` and `handleAPIRollback`, after the existing gates and before `PromoteOnLadder`: |
| 918 | |
| 919 | ```go |
| 920 | // A person with ops membership may roll the control plane; a service |
| 921 | // credential may not. This is the flag's promote-side job — the whole of |
| 922 | // it. Gating a PERSON here would break `carlos deploy -app console`, the |
| 923 | // command this exists to make ordinary. |
| 924 | if p.Kind != PrincipalUser { |
| 925 | cp, _, err := s.store.AppFlags(acct.ID, app) |
| 926 | if err != nil { |
| 927 | jsonError(w, http.StatusServiceUnavailable, "could not check this app's control-plane status — nothing was changed") |
| 928 | return |
| 929 | } |
| 930 | if cp { |
| 931 | jsonError(w, http.StatusForbidden, |
| 932 | app+" is the control plane — a service credential cannot promote it; a person must") |
| 933 | return |
| 934 | } |
| 935 | } |
| 936 | ``` |
| 937 | |
| 938 | - [ ] **Step 5: Run the tests to verify they pass** |
| 939 | |
| 940 | ``` |
| 941 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 942 | go test ./internal/console/ -run 'ControlPlane|OrdinaryAppUnaffected' -count=1 -v |
| 943 | ``` |
| 944 | |
| 945 | Expected: all five PASS. |
| 946 | |
| 947 | - [ ] **Step 6: Full gate, then commit** |
| 948 | |
| 949 | ```bash |
| 950 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 951 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 952 | gofmt -l internal/console/controlplane.go internal/console/apicli.go internal/console/appsettings.go internal/console/ownership.go internal/console/controlplane_test.go |
| 953 | git add -A |
| 954 | git commit -m "console: the control-plane flag refuses destruction, not promotion |
| 955 | |
| 956 | Trash, purge, rename and transfer are refused outright for a flagged app; |
| 957 | a service credential cannot promote one; a PERSON promotes it with |
| 958 | ordinary ops membership. |
| 959 | |
| 960 | The asymmetry is the design. Gating promotion would have broken |
| 961 | carlos deploy -app console — stable's fresh-auth 403 has no API-side |
| 962 | step-up and deploy log.Fatalfs on a failed promote after the ship has |
| 963 | landed. Meanwhile purge could delete the release history the fleet is |
| 964 | running, behind a confirmation that had no idea what the app was." |
| 965 | ``` |
| 966 | |
| 967 | --- |
| 968 | |
| 969 | ### Task 5: `/healthz` reports the build |
| 970 | |
| 971 | **Files:** |
| 972 | - Modify: `internal/console/server.go:794-796` |
| 973 | - Test: `internal/console/server_test.go:212` (extend the existing assertion) |
| 974 | |
| 975 | **Interfaces:** |
| 976 | - Consumes: nothing. |
| 977 | - Produces: `GET /healthz` → `ok <build>\n`, from the console process. |
| 978 | |
| 979 | **Existing consumers checked** (CLAUDE.md's parse-the-shape rule): `internal/vet` and the edge health probe assert **status only**, so appending is additive-safe. Re-verify with `grep -rn '/healthz' --include=*.go .` before changing it, and if any consumer compares the body exactly, fix that consumer in this task. |
| 980 | |
| 981 | - [ ] **Step 1: Write the failing test** |
| 982 | |
| 983 | Add to `internal/console/server_test.go`: |
| 984 | |
| 985 | ```go |
| 986 | // TestHealthzCarriesTheBuild: the console's own build, from the PROCESS. |
| 987 | // |
| 988 | // This is the deploy watch's exit criterion, and it exists because both cheap |
| 989 | // answers lie. X-Carlos-Version is stamped by the EDGE from the route row |
| 990 | // (edge.go:1336), so adoption sets it while the old process keeps serving; |
| 991 | // `systemctl is-active` says "active" for a process started from the previous |
| 992 | // version just as happily (tito, 2026-08-09, seconds after adopt and restart |
| 993 | // both reported success). Only the running process can answer this. |
| 994 | func TestHealthzCarriesTheBuild(t *testing.T) { |
| 995 | fx := newConsoleFixture(t) |
| 996 | status, _, body := get(t, nil, fx.ts.URL+"/healthz") |
| 997 | if status != 200 { |
| 998 | t.Fatalf("/healthz = %d, want 200", status) |
| 999 | } |
| 1000 | if !strings.Contains(body, "ok") { |
| 1001 | t.Fatalf("/healthz body = %q, want it to still start with ok (vet and the edge probe read this)", body) |
| 1002 | } |
| 1003 | fields := strings.Fields(body) |
| 1004 | if len(fields) != 2 || fields[1] != version.Build { |
| 1005 | t.Fatalf("/healthz body = %q, want exactly \"ok %s\" — the watch matches a whitespace-delimited token", body, version.Build) |
| 1006 | } |
| 1007 | } |
| 1008 | ``` |
| 1009 | |
| 1010 | Match `get`'s real signature from the neighbouring test at `server_test.go:212`, and import `internal/version` and `strings`. |
| 1011 | |
| 1012 | - [ ] **Step 2: Run to verify it fails** |
| 1013 | |
| 1014 | ``` |
| 1015 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1016 | go test ./internal/console/ -run TestHealthzCarriesTheBuild -count=1 |
| 1017 | ``` |
| 1018 | |
| 1019 | Expected: FAIL — body is `"ok\n"`, two fields wanted. |
| 1020 | |
| 1021 | - [ ] **Step 3: Implement** |
| 1022 | |
| 1023 | `internal/console/server.go`: |
| 1024 | |
| 1025 | ```go |
| 1026 | // "ok <build>" — the build comes from THIS PROCESS (ldflags, at link |
| 1027 | // time), which is the whole point: X-Carlos-Version is stamped by the |
| 1028 | // edge from the route row, so it reports the version adoption INTENDED |
| 1029 | // from the moment the symlink moves, whether or not anything restarted. |
| 1030 | // `carlos deploy` polls this instead for any app declaring a build probe. |
| 1031 | // |
| 1032 | // "ok" stays the first token: internal/vet and the edge health probe read |
| 1033 | // this endpoint and check status, not body, but keeping the prefix means |
| 1034 | // a human or a script eyeballing it sees what it always saw. |
| 1035 | s.mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { |
| 1036 | fmt.Fprintln(w, "ok", version.Build) |
| 1037 | }) |
| 1038 | ``` |
| 1039 | |
| 1040 | Import `github.com/carlosframework/platform/internal/version` in `server.go` if not present. |
| 1041 | |
| 1042 | - [ ] **Step 4: Run the tests to verify they pass** |
| 1043 | |
| 1044 | ``` |
| 1045 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1046 | go test ./internal/console/ -run 'Healthz|TestServer' -count=1 |
| 1047 | ``` |
| 1048 | |
| 1049 | Expected: PASS, including the pre-existing `/healthz` assertion at line 212. |
| 1050 | |
| 1051 | - [ ] **Step 5: Full gate, then commit** |
| 1052 | |
| 1053 | ```bash |
| 1054 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1055 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1056 | gofmt -l internal/console/server.go internal/console/server_test.go |
| 1057 | git add internal/console/server.go internal/console/server_test.go |
| 1058 | git commit -m "console: /healthz reports the build the PROCESS is running |
| 1059 | |
| 1060 | X-Carlos-Version is stamped by the edge from the route row, so it reports |
| 1061 | what adoption intended from the moment the symlink moves — whether or not |
| 1062 | anything restarted. systemctl is-active is no better. Only the process can |
| 1063 | answer this honestly, and now it does. |
| 1064 | |
| 1065 | 'ok' stays the first token: vet and the edge probe read status not body, |
| 1066 | but a human eyeballing it should see what they always saw." |
| 1067 | ``` |
| 1068 | |
| 1069 | --- |
| 1070 | |
| 1071 | ### Task 6: `carlos deploy` polls the probe instead of the header |
| 1072 | |
| 1073 | **Files:** |
| 1074 | - Modify: `deploy.go` (`watchTarget`, `watchVersion`, and the call site that builds targets) |
| 1075 | - Modify: `internal/console/apicli.go` (serve `build_probe` on an existing per-app GET the deploy path already calls) |
| 1076 | - Test: `deploy_test.go` |
| 1077 | |
| 1078 | **Interfaces:** |
| 1079 | - Consumes: `Store.AppFlags` (Task 1); `/healthz` shape (Task 5). |
| 1080 | - Produces: `watchTarget` gains `Probe string`; when non-empty the watch polls `Origin + Probe` and matches the body, otherwise it polls `/` and matches `X-Carlos-Version` exactly as today. |
| 1081 | |
| 1082 | **Two lies this must not reintroduce** (spec §3b): the ldflags trap (a binary built without `-X …version.Build` reports `dev` forever, so a healthy deploy reports failure) — closed by naming the body in the timeout, and at the door in Task 8; and the edge cache — `watchVersion` already busts every poll because the edge caches a proxied 2xx for up to five minutes, and the probe leg **must inherit that**. |
| 1083 | |
| 1084 | - [ ] **Step 1: Write the failing test** |
| 1085 | |
| 1086 | Add to `deploy_test.go`: |
| 1087 | |
| 1088 | ```go |
| 1089 | // TestWatchProbeMatchesAWholeToken: the probe body must carry the build as a |
| 1090 | // whitespace-delimited token. Substring matching would let abc123 satisfy a |
| 1091 | // watch for abc1234 — and short git shas make that a real collision, not a |
| 1092 | // theoretical one. |
| 1093 | func TestWatchProbeMatchesAWholeToken(t *testing.T) { |
| 1094 | var body atomic.Value |
| 1095 | body.Store("ok abc123") |
| 1096 | srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| 1097 | if r.URL.Path != "/healthz" { |
| 1098 | t.Errorf("watch dialled %q, want /healthz", r.URL.Path) |
| 1099 | } |
| 1100 | if r.URL.Query().Get("carlos-deploy-probe") == "" { |
| 1101 | t.Error("probe poll carried no cache-buster — the edge caches a proxied 2xx for up to 5 minutes") |
| 1102 | } |
| 1103 | fmt.Fprintln(w, body.Load().(string)) |
| 1104 | })) |
| 1105 | defer srv.Close() |
| 1106 | old := watchOrigin |
| 1107 | watchOrigin = func(host string) string { return srv.URL } |
| 1108 | defer func() { watchOrigin = old }() |
| 1109 | |
| 1110 | targets := []watchTarget{{Host: "console.example", Origin: srv.URL, Probe: "/healthz"}} |
| 1111 | err := watchVersion(io.Discard, targets, "abc1234", 500*time.Millisecond, 50*time.Millisecond, nil) |
| 1112 | if err == nil { |
| 1113 | t.Fatal("watch for abc1234 succeeded against a body of \"ok abc123\" — substring match, not token match") |
| 1114 | } |
| 1115 | |
| 1116 | body.Store("ok abc1234") |
| 1117 | if err := watchVersion(io.Discard, targets, "abc1234", 2*time.Second, 50*time.Millisecond, nil); err != nil { |
| 1118 | t.Fatalf("watch against the matching build = %v, want success", err) |
| 1119 | } |
| 1120 | } |
| 1121 | |
| 1122 | // TestWatchProbeTimeoutNamesTheBody: a binary built without the version |
| 1123 | // ldflag reports "dev" forever, so the deploy fails while the roll was |
| 1124 | // perfectly healthy. The report has to make that self-diagnosing rather than |
| 1125 | // saying "timed out". |
| 1126 | func TestWatchProbeTimeoutNamesTheBody(t *testing.T) { |
| 1127 | srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| 1128 | fmt.Fprintln(w, "ok dev") |
| 1129 | })) |
| 1130 | defer srv.Close() |
| 1131 | var buf bytes.Buffer |
| 1132 | targets := []watchTarget{{Host: "console.example", Origin: srv.URL, Probe: "/healthz"}} |
| 1133 | err := watchVersion(&buf, targets, "abc1234", 300*time.Millisecond, 50*time.Millisecond, nil) |
| 1134 | if err == nil { |
| 1135 | t.Fatal("want a timeout") |
| 1136 | } |
| 1137 | out := buf.String() + err.Error() |
| 1138 | if !strings.Contains(out, "dev") { |
| 1139 | t.Fatalf("timeout report %q does not name what the probe actually said — the ldflags trap stays invisible", out) |
| 1140 | } |
| 1141 | } |
| 1142 | |
| 1143 | // TestWatchWithoutAProbeStillReadsTheHeader: an app that declares no probe |
| 1144 | // keeps today's behaviour byte for byte. |
| 1145 | func TestWatchWithoutAProbeStillReadsTheHeader(t *testing.T) { |
| 1146 | srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| 1147 | w.Header().Set("X-Carlos-Version", "abc1234") |
| 1148 | w.WriteHeader(200) |
| 1149 | })) |
| 1150 | defer srv.Close() |
| 1151 | targets := []watchTarget{{Host: "app.example", Origin: srv.URL}} |
| 1152 | if err := watchVersion(io.Discard, targets, "abc1234", 2*time.Second, 50*time.Millisecond, nil); err != nil { |
| 1153 | t.Fatalf("header watch = %v, want success", err) |
| 1154 | } |
| 1155 | } |
| 1156 | ``` |
| 1157 | |
| 1158 | Read `watchTarget`'s real fields and `watchVersion`'s signature in `deploy.go` first and adjust; add `sync/atomic`, `bytes`, `strings` to imports as needed. |
| 1159 | |
| 1160 | - [ ] **Step 2: Run to verify they fail** |
| 1161 | |
| 1162 | ``` |
| 1163 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1164 | go test . -run 'TestWatch(Probe|Without)' -count=1 |
| 1165 | ``` |
| 1166 | |
| 1167 | Expected: FAIL — `unknown field Probe`. |
| 1168 | |
| 1169 | - [ ] **Step 3: Implement the probe leg** |
| 1170 | |
| 1171 | Add `Probe string` to `watchTarget` with a comment naming why it exists. In `watchVersion`'s per-target goroutine, branch on it, keeping the cache-buster on both paths: |
| 1172 | |
| 1173 | ```go |
| 1174 | n := attempt.Add(1) |
| 1175 | path := "/" |
| 1176 | if t.Probe != "" { |
| 1177 | path = t.Probe |
| 1178 | } |
| 1179 | url := fmt.Sprintf("%s%s?carlos-deploy-probe=%d", t.Origin, path, n) |
| 1180 | resp, err := client.Get(url) |
| 1181 | if err != nil { |
| 1182 | results[i] = result{diag: "unreachable: " + err.Error()} |
| 1183 | return |
| 1184 | } |
| 1185 | if t.Probe != "" { |
| 1186 | // The app's own process answers here, so the body IS the |
| 1187 | // truth — no status/header disagreement to reconcile the |
| 1188 | // way the header path has to. Read a bounded amount: this |
| 1189 | // is a health endpoint, and a target that answers with a |
| 1190 | // stream is misconfigured, not something to buffer. |
| 1191 | b, rerr := io.ReadAll(io.LimitReader(resp.Body, 4096)) |
| 1192 | resp.Body.Close() |
| 1193 | if rerr != nil { |
| 1194 | results[i] = result{diag: "probe unreadable: " + rerr.Error()} |
| 1195 | return |
| 1196 | } |
| 1197 | got := strings.TrimSpace(string(b)) |
| 1198 | if resp.StatusCode/100 != 2 { |
| 1199 | results[i] = result{diag: fmt.Sprintf("%s answered HTTP %d", path, resp.StatusCode)} |
| 1200 | return |
| 1201 | } |
| 1202 | // Whole-token, never substring: abc123 must not satisfy a |
| 1203 | // watch for abc1234, and short git shas collide like that |
| 1204 | // in practice. |
| 1205 | for _, f := range strings.Fields(got) { |
| 1206 | if f == ver { |
| 1207 | results[i] = result{live: true} |
| 1208 | return |
| 1209 | } |
| 1210 | } |
| 1211 | // Naming the body is what makes the ldflags trap |
| 1212 | // self-diagnosing: a binary built without |
| 1213 | // -X …version.Build reports "dev" forever, and "timed |
| 1214 | // out" would send the operator hunting the wrong thing. |
| 1215 | results[i] = result{diag: fmt.Sprintf("%s reports %q, waiting for %s", path, got, ver)} |
| 1216 | return |
| 1217 | } |
| 1218 | io.CopyN(io.Discard, resp.Body, 4096) |
| 1219 | resp.Body.Close() |
| 1220 | got := resp.Header.Get("X-Carlos-Version") |
| 1221 | // ... existing header switch unchanged ... |
| 1222 | ``` |
| 1223 | |
| 1224 | - [ ] **Step 4: Serve and consume the probe** |
| 1225 | |
| 1226 | Server side: include `"build_probe"` in the JSON of the per-app GET the deploy path already calls. Read `deploy.go` to find which call that is (the instance list, or channels) and extend that handler and its client struct rather than adding a door. State in a comment that the probe is app-level and read from the app ROW — not from `provision.Record`, whose `handleAPIInstanceUpdate` is a read-modify-write that would silently drop an unknown field. |
| 1227 | |
| 1228 | Client side: set `Probe` on every `watchTarget` the deploy builds from that response. |
| 1229 | |
| 1230 | - [ ] **Step 5: Run the tests** |
| 1231 | |
| 1232 | ``` |
| 1233 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1234 | go test . ./internal/console/ -run 'Watch|Deploy' -count=1 -v |
| 1235 | ``` |
| 1236 | |
| 1237 | Expected: PASS, existing deploy tests included. |
| 1238 | |
| 1239 | - [ ] **Step 6: Full gate, then commit** |
| 1240 | |
| 1241 | ```bash |
| 1242 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1243 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1244 | gofmt -l deploy.go deploy_test.go internal/console/apicli.go |
| 1245 | git add -A |
| 1246 | git commit -m "carlos deploy: watch the app's own build report, not the edge's header |
| 1247 | |
| 1248 | An app declaring a build probe is watched by polling that path and |
| 1249 | matching the build as a WHOLE TOKEN in the body — substring matching would |
| 1250 | let abc123 satisfy a watch for abc1234, which short git shas make a real |
| 1251 | collision. No probe declared keeps the header path byte for byte. |
| 1252 | |
| 1253 | The probe leg inherits the cache-buster: the edge caches a proxied 2xx for |
| 1254 | up to five minutes, longer than the watch's own timeout. And the timeout |
| 1255 | NAMES what the probe said, so a binary built without -X version.Build |
| 1256 | diagnoses itself instead of looking like a failed deploy." |
| 1257 | ``` |
| 1258 | |
| 1259 | --- |
| 1260 | |
| 1261 | ### Task 7: Adoption starts the unit again after a rolled-back restart |
| 1262 | |
| 1263 | **Files:** |
| 1264 | - Modify: `internal/release/adopt.go:781-815` (the restart-failure block) |
| 1265 | - Modify: `internal/host/agent.go` (`makeRestartRoute`, to expose a start) |
| 1266 | - Test: `internal/release/adopt_test.go` |
| 1267 | |
| 1268 | **Interfaces:** |
| 1269 | - Consumes: nothing. |
| 1270 | - Produces: the restart callback type gains a start capability. Prefer the smallest change that keeps `Pass`'s signature stable — e.g. a `PassOption` carrying `startUnit func(host string) error`, so callers that pass no start keep today's behaviour. |
| 1271 | |
| 1272 | **This is a tenant fix and the most important defect in the slice** (spec §4b). Verified: |
| 1273 | |
| 1274 | ``` |
| 1275 | pass 1: PointLive(new) → try-restart FAILS → poison, PointLive(old), no stamp |
| 1276 | unit: failed. console: DOWN. header: honest (old). |
| 1277 | pass 2: PointLive(new) → try-restart on a NOT-RUNNING unit → NO-OP, exit 0 |
| 1278 | → SetVersions(new) |
| 1279 | unit: still failed. console: STILL DOWN. header: claims the new build. |
| 1280 | ``` |
| 1281 | |
| 1282 | `try-restart` being a no-op on a stopped unit is deliberate — it is what lets a deliberately-stopped console stay stopped (`makeRestartRoute`'s comment). The bug is that the rollback leaves the unit stopped in the first place. |
| 1283 | |
| 1284 | - [ ] **Step 1: Write the failing test** |
| 1285 | |
| 1286 | Add to `internal/release/adopt_test.go`, following the file's existing fixture style for a registry + store + route: |
| 1287 | |
| 1288 | ```go |
| 1289 | // TestFailedRestartLeavesTheUnitRUNNINGOnTheOldVersion. |
| 1290 | // |
| 1291 | // The rollback repointed the symlink and stopped there, which left the box |
| 1292 | // with a correct symlink and no process. Worse, try-restart is a NO-OP on a |
| 1293 | // unit that is not running — so the NEXT pass exited 0, stamped SetVersions, |
| 1294 | // and produced exactly the lying header the whole mechanism exists to |
| 1295 | // prevent, over a service that was not running at all. |
| 1296 | func TestFailedRestartLeavesTheUnitRUNNINGOnTheOldVersion(t *testing.T) { |
| 1297 | // Build the fixture the way the neighbouring adopt tests do: a registry |
| 1298 | // with one route on a channel, a store holding two versions, "old" |
| 1299 | // already live and recorded. |
| 1300 | fx := newAdoptFixture(t) |
| 1301 | fx.setLive("console.example", "old") |
| 1302 | fx.promote("console.example", "new") |
| 1303 | |
| 1304 | var started []string |
| 1305 | restart := func(host string) error { return errors.New("Job for console.service failed") } |
| 1306 | start := func(host string) error { started = append(started, host); return nil } |
| 1307 | |
| 1308 | if err := Pass(context.Background(), fx.reg, fx.store, fx.pub, fx.optDir, restart, |
| 1309 | WithStartUnit(start)); err != nil { |
| 1310 | t.Fatalf("pass: %v", err) |
| 1311 | } |
| 1312 | |
| 1313 | if got := fx.liveVersion("console.example"); got != "old" { |
| 1314 | t.Fatalf("symlink = %q, want it rolled back to old", got) |
| 1315 | } |
| 1316 | if len(started) != 1 || started[0] != "console.example" { |
| 1317 | t.Fatalf("started = %v, want the unit started once on the restored version — a rollback that leaves the service down is not a rollback", started) |
| 1318 | } |
| 1319 | if got := fx.recordedVersion("console.example"); got == "new" { |
| 1320 | t.Fatal("SetVersions stamped the failed build") |
| 1321 | } |
| 1322 | } |
| 1323 | |
| 1324 | // TestSecondPassOverAStillBrokenBuildDoesNotStamp: the second-order failure. |
| 1325 | // With the unit running again after pass 1, pass 2's try-restart is a real |
| 1326 | // restart that fails again — so it rolls back again and still does not stamp. |
| 1327 | func TestSecondPassOverAStillBrokenBuildDoesNotStamp(t *testing.T) { |
| 1328 | fx := newAdoptFixture(t) |
| 1329 | fx.setLive("console.example", "old") |
| 1330 | fx.promote("console.example", "new") |
| 1331 | |
| 1332 | restart := func(host string) error { return errors.New("still broken") } |
| 1333 | start := func(host string) error { return nil } |
| 1334 | for i := 0; i < 2; i++ { |
| 1335 | if err := Pass(context.Background(), fx.reg, fx.store, fx.pub, fx.optDir, restart, |
| 1336 | WithStartUnit(start)); err != nil { |
| 1337 | t.Fatalf("pass %d: %v", i, err) |
| 1338 | } |
| 1339 | } |
| 1340 | if got := fx.recordedVersion("console.example"); got == "new" { |
| 1341 | t.Fatal("the second pass stamped a build that has never run") |
| 1342 | } |
| 1343 | if got := fx.liveVersion("console.example"); got != "old" { |
| 1344 | t.Fatalf("symlink after two passes = %q, want old", got) |
| 1345 | } |
| 1346 | } |
| 1347 | ``` |
| 1348 | |
| 1349 | `newAdoptFixture` and its helpers are illustrative — read `internal/release/adopt_test.go` and use the fixture that file already has, or add these helpers to it if none fits. |
| 1350 | |
| 1351 | - [ ] **Step 2: Run to verify they fail** |
| 1352 | |
| 1353 | ``` |
| 1354 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1355 | go test ./internal/release/ -run 'FailedRestartLeaves|SecondPassOver' -count=1 |
| 1356 | ``` |
| 1357 | |
| 1358 | Expected: FAIL — `undefined: WithStartUnit`. |
| 1359 | |
| 1360 | - [ ] **Step 3: Add the option and use it in the rollback** |
| 1361 | |
| 1362 | Add `WithStartUnit` alongside the existing `PassOption`s, then in `adoptHost`'s restart-failure block, after the symlink has gone back and any capability restore has run: |
| 1363 | |
| 1364 | ```go |
| 1365 | // Start the service again on the version the symlink NOW names. |
| 1366 | // The rollback above put the bytes back; without this the box is |
| 1367 | // left with a correct symlink and no process — and because |
| 1368 | // try-restart is a no-op on a unit that is not running (which is |
| 1369 | // what lets a deliberately-stopped service stay stopped), the |
| 1370 | // NEXT pass's restart would exit 0, SetVersions would stamp the |
| 1371 | // build that has never run, and the header would claim it over a |
| 1372 | // service that is down. Start, not try-restart: the whole premise |
| 1373 | // here is that the unit is not running. |
| 1374 | // |
| 1375 | // A start that also fails changes nothing about the refusal — the |
| 1376 | // version stays poisoned and unstamped — but it is printed, |
| 1377 | // because at that point the box is genuinely broken and the |
| 1378 | // operator needs to know that rather than reading a rollback |
| 1379 | // message as "recovered". |
| 1380 | if cfg.startUnit != nil { |
| 1381 | if serr := cfg.startUnit(rt.Host); serr != nil { |
| 1382 | fmt.Printf(" %s: rolled back to %s but could not start it: %v — this host is serving nothing\n", |
| 1383 | rt.Host, cur, serr) |
| 1384 | } |
| 1385 | } |
| 1386 | ``` |
| 1387 | |
| 1388 | Then wire it at the agent: `makeRestartRoute`'s neighbours already have `tryRestartUnit`; add a `startUnit` sibling that resolves the route's `Unit` and runs `systemctl start`, and pass `WithStartUnit` where the agent builds its pass options. A route with no `Unit` has nothing to start — return nil, matching arm 1's "this arm takes no part in the fallthrough" rule. |
| 1389 | |
| 1390 | - [ ] **Step 4: Run the tests** |
| 1391 | |
| 1392 | ``` |
| 1393 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1394 | go test ./internal/release/ ./internal/host/ -count=1 |
| 1395 | ``` |
| 1396 | |
| 1397 | Expected: PASS. |
| 1398 | |
| 1399 | - [ ] **Step 5: Full gate, then commit** |
| 1400 | |
| 1401 | ```bash |
| 1402 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1403 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1404 | gofmt -l internal/release/adopt.go internal/host/agent.go internal/release/adopt_test.go |
| 1405 | git add -A |
| 1406 | git commit -m "adopt: a rolled-back restart starts the service again |
| 1407 | |
| 1408 | The rollback repointed the symlink and stopped, leaving a correct symlink |
| 1409 | and no process. And because try-restart is a no-op on a unit that is not |
| 1410 | running — the property that lets a deliberately-stopped service stay |
| 1411 | stopped — the NEXT pass exited 0 and stamped SetVersions over a service |
| 1412 | that was not running at all: the exact lying header the mechanism exists |
| 1413 | to prevent, now with nothing behind it. |
| 1414 | |
| 1415 | Every unit-backed route has this. It stayed hidden because the pinfra |
| 1416 | roll script's explicit post-pass restart papers over it on the only |
| 1417 | unit-stamped route in the fleet." |
| 1418 | ``` |
| 1419 | |
| 1420 | --- |
| 1421 | |
| 1422 | ### Task 8: Ship-time refusals for a control-plane app |
| 1423 | |
| 1424 | **Files:** |
| 1425 | - Modify: `internal/console/apicli.go` (`handleAPIShip`, after the artifact loop) |
| 1426 | - Test: `internal/console/apicli_test.go` |
| 1427 | |
| 1428 | **Interfaces:** |
| 1429 | - Consumes: `Store.AppFlags` (Task 1). |
| 1430 | - Produces: `func checkControlPlaneArtifacts(files map[string][]byte, version string) error` in `internal/console`. |
| 1431 | |
| 1432 | **Both checks verified against a real arm64 console build** — stdlib only, and `-s -w` does not strip either: |
| 1433 | |
| 1434 | ``` |
| 1435 | debug/buildinfo → Settings["-ldflags"] = "-s -w -X …/internal/version.Build=deccd44" |
| 1436 | debug/elf → Machine = EM_AARCH64 |
| 1437 | ``` |
| 1438 | |
| 1439 | Scoped to control-plane apps deliberately: a tenant may legitimately ship an artifact we would guess wrong about, and refusing their ship on our inference is not our call. |
| 1440 | |
| 1441 | - [ ] **Step 1: Write the failing test** |
| 1442 | |
| 1443 | Add to `internal/console/apicli_test.go`: |
| 1444 | |
| 1445 | ```go |
| 1446 | // TestControlPlaneShipRefusesTheWrongArchitecture: failure mode #1, an amd64 |
| 1447 | // binary shipped to Graviton boxes. -target linux-arm64 is carlos deploy's |
| 1448 | // default, which protects the careful caller; this closes the path a hand-run |
| 1449 | // ship, a CI job or a future script takes into the same channel. |
| 1450 | func TestControlPlaneShipRefusesTheWrongArchitecture(t *testing.T) { |
| 1451 | fx, token := controlPlaneFixture(t, "code-cp-arch") |
| 1452 | liftShipBudget(fx) |
| 1453 | amd64ELF := buildTestELF(t, "linux", "amd64", "cp00003") |
| 1454 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/console/releases", token, |
| 1455 | []shipPart{metaPart(t, "cp00003", "binary"), {name: "linux-arm64", body: amd64ELF}}) |
| 1456 | if status != http.StatusBadRequest { |
| 1457 | t.Fatalf("amd64 ELF as linux-arm64 = %d %v, want 400", status, out) |
| 1458 | } |
| 1459 | msg := str(t, out, "error") |
| 1460 | if !strings.Contains(msg, "linux-arm64") || !strings.Contains(msg, "amd64") { |
| 1461 | t.Fatalf("refusal %q must name both the declared target and what was found", msg) |
| 1462 | } |
| 1463 | } |
| 1464 | |
| 1465 | // TestControlPlaneShipRefusesAMismatchedEmbeddedBuild: closes the ldflags trap |
| 1466 | // at the door. Build without -X …version.Build and the roll succeeds while the |
| 1467 | // probe reports "dev" forever — deploy reports failure on a healthy deploy, |
| 1468 | // the exact inverse of the lie the probe exists to kill. |
| 1469 | func TestControlPlaneShipRefusesAMismatchedEmbeddedBuild(t *testing.T) { |
| 1470 | fx, token := controlPlaneFixture(t, "code-cp-embed") |
| 1471 | liftShipBudget(fx) |
| 1472 | wrong := buildTestELF(t, "linux", "arm64", "notthesha") |
| 1473 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/console/releases", token, |
| 1474 | []shipPart{metaPart(t, "cp00004", "binary"), {name: "linux-arm64", body: wrong}}) |
| 1475 | if status != http.StatusBadRequest { |
| 1476 | t.Fatalf("embedded build != -version = %d %v, want 400", status, out) |
| 1477 | } |
| 1478 | if msg := str(t, out, "error"); !strings.Contains(msg, "notthesha") || !strings.Contains(msg, "cp00004") { |
| 1479 | t.Fatalf("refusal %q must name both builds", msg) |
| 1480 | } |
| 1481 | } |
| 1482 | |
| 1483 | // TestControlPlaneShipAcceptsAMatchingArtifact. |
| 1484 | func TestControlPlaneShipAcceptsAMatchingArtifact(t *testing.T) { |
| 1485 | fx, token := controlPlaneFixture(t, "code-cp-ok") |
| 1486 | liftShipBudget(fx) |
| 1487 | good := buildTestELF(t, "linux", "arm64", "cp00005") |
| 1488 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/console/releases", token, |
| 1489 | []shipPart{metaPart(t, "cp00005", "binary"), {name: "linux-arm64", body: good}}) |
| 1490 | if status != http.StatusCreated { |
| 1491 | t.Fatalf("matching artifact = %d %v, want 201", status, out) |
| 1492 | } |
| 1493 | } |
| 1494 | |
| 1495 | // TestOrdinaryAppShipIsNotArchChecked: a tenant may legitimately ship an |
| 1496 | // artifact we would guess wrong about; refusing on our inference is not our |
| 1497 | // call. Only the app that IS the control plane, on boxes we own and know are |
| 1498 | // arm64, gets the check. |
| 1499 | func TestOrdinaryAppShipIsNotArchChecked(t *testing.T) { |
| 1500 | fx := newConsoleFixture(t) |
| 1501 | if _, err := fx.st.CreateAppIn(OpsAccountID, "titogo", "paul@keymail.dev", time.Now()); err != nil { |
| 1502 | t.Fatal(err) |
| 1503 | } |
| 1504 | token := cliToken(t, fx, "paul@keymail.dev", "code-cp-tenant") |
| 1505 | liftShipBudget(fx) |
| 1506 | status, out := postShip(t, fx.ts.URL+"/api/cli/apps/bab/titogo/releases", token, |
| 1507 | []shipPart{metaPart(t, "t0000001", "binary"), {name: "linux-arm64", body: []byte("not an ELF at all")}}) |
| 1508 | if status != http.StatusCreated { |
| 1509 | t.Fatalf("ordinary app shipping a non-ELF = %d %v, want 201 (unaffected)", status, out) |
| 1510 | } |
| 1511 | } |
| 1512 | ``` |
| 1513 | |
| 1514 | `buildTestELF` compiles a tiny program with `go build` at the requested `GOOS`/`GOARCH` and ldflags, caching per (goos, goarch, build) in a package-level `sync.Map` so the suite compiles each combination once. Write it in this file, `t.Skip` if the toolchain is unavailable, and give it a comment saying why a real binary is used rather than a handcrafted ELF header (the point is to prove `debug/buildinfo` reads what the real build recipe writes). |
| 1515 | |
| 1516 | - [ ] **Step 2: Run to verify they fail** |
| 1517 | |
| 1518 | ``` |
| 1519 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1520 | go test ./internal/console/ -run 'ControlPlaneShip|OrdinaryAppShip' -count=1 |
| 1521 | ``` |
| 1522 | |
| 1523 | Expected: FAIL — all ships accepted. |
| 1524 | |
| 1525 | - [ ] **Step 3: Implement the checks** |
| 1526 | |
| 1527 | Add to `internal/console/controlplane.go`: |
| 1528 | |
| 1529 | ```go |
| 1530 | // targetMachine maps an artifact part name — which IS the declared target, |
| 1531 | // "linux-arm64" and friends — to the ELF machine a binary for it must carry. |
| 1532 | // Unknown targets are not refused: this guard exists to catch a wrong build |
| 1533 | // of a target we know, not to bound the target vocabulary, which targetRe |
| 1534 | // already does. |
| 1535 | var targetMachine = map[string]elf.Machine{ |
| 1536 | "linux-arm64": elf.EM_AARCH64, |
| 1537 | "linux-amd64": elf.EM_X86_64, |
| 1538 | } |
| 1539 | |
| 1540 | // checkControlPlaneArtifacts refuses two ways a control-plane ship can be |
| 1541 | // wrong in a manner nothing downstream would catch. |
| 1542 | // |
| 1543 | // ARCHITECTURE. An amd64 binary on a Graviton box has happened and briefly |
| 1544 | // broke one. `carlos deploy` defaults -target to linux-arm64, which protects |
| 1545 | // the careful caller and nobody else; this closes the same door against a |
| 1546 | // hand-run ship, a CI job, or a script written next year. |
| 1547 | // |
| 1548 | // EMBEDDED BUILD. version.Build is set by ldflags the operator has to |
| 1549 | // remember. Forget it and the artifact says "dev" forever: the roll SUCCEEDS, |
| 1550 | // the deploy watch's probe never matches, and a perfectly healthy deploy |
| 1551 | // reports failure — the exact inverse of the lie the probe exists to kill. |
| 1552 | // Caught here it is one sentence at the door instead of a timeout nobody can |
| 1553 | // explain. |
| 1554 | // |
| 1555 | // Both read the bytes that actually arrived, and both are stdlib. Verified |
| 1556 | // against a real arm64 console build that -s -w strips neither the ELF header |
| 1557 | // nor the build info. |
| 1558 | func checkControlPlaneArtifacts(files map[string][]byte, version string) error { |
| 1559 | for name, b := range files { |
| 1560 | want, known := targetMachine[name] |
| 1561 | if !known { |
| 1562 | continue |
| 1563 | } |
| 1564 | f, err := elf.NewFile(bytes.NewReader(b)) |
| 1565 | if err != nil { |
| 1566 | return fmt.Errorf("artifact %s is not a readable ELF binary: %w", name, err) |
| 1567 | } |
| 1568 | if f.Machine != want { |
| 1569 | return fmt.Errorf("artifact %s declares target %s but is a %s binary", |
| 1570 | name, name, machineName(f.Machine)) |
| 1571 | } |
| 1572 | bi, err := buildinfo.Read(bytes.NewReader(b)) |
| 1573 | if err != nil { |
| 1574 | return fmt.Errorf("artifact %s carries no Go build info: %w", name, err) |
| 1575 | } |
| 1576 | got := embeddedBuild(bi) |
| 1577 | if got != version { |
| 1578 | return fmt.Errorf("artifact %s was built with version.Build=%q but is being shipped as %q — "+ |
| 1579 | `rebuild with -ldflags "-X github.com/carlosframework/platform/internal/version.Build=%s"`, |
| 1580 | name, got, version, version) |
| 1581 | } |
| 1582 | } |
| 1583 | return nil |
| 1584 | } |
| 1585 | ``` |
| 1586 | |
| 1587 | Write `embeddedBuild(bi *debug.BuildInfo) string` to find the `-ldflags` setting and extract the value after `internal/version.Build=`, returning `""` when absent — and refuse `""` with a message naming the missing ldflag specifically, because that is the common case and deserves its own sentence. Write `machineName(elf.Machine) string` mapping the two known machines to `amd64`/`arm64` and falling back to the `elf.Machine` string. |
| 1588 | |
| 1589 | Call it in `handleAPIShip` after the artifact loop and before `release.Ship`: |
| 1590 | |
| 1591 | ```go |
| 1592 | // After the parts are read (the bytes have to exist to be inspected) and |
| 1593 | // before anything is written. A control-plane app only. |
| 1594 | if cp, _, err := s.store.AppFlags(acct.ID, app); err != nil { |
| 1595 | jsonError(w, http.StatusServiceUnavailable, "could not check this app's control-plane status — nothing was shipped") |
| 1596 | return |
| 1597 | } else if cp && meta.Kind != "static" { |
| 1598 | if err := checkControlPlaneArtifacts(files, version); err != nil { |
| 1599 | jsonError(w, http.StatusBadRequest, err.Error()) |
| 1600 | return |
| 1601 | } |
| 1602 | } |
| 1603 | ``` |
| 1604 | |
| 1605 | `version` here must be the **resolved** version (after minting), not `meta.Version` — read the surrounding code and use whatever holds the final value. |
| 1606 | |
| 1607 | - [ ] **Step 4: Run the tests, then the full gate, then commit** |
| 1608 | |
| 1609 | ```bash |
| 1610 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1611 | go test ./internal/console/ -run 'ControlPlaneShip|OrdinaryAppShip' -count=1 -v |
| 1612 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1613 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1614 | gofmt -l internal/console/controlplane.go internal/console/apicli.go internal/console/apicli_test.go |
| 1615 | git add -A |
| 1616 | git commit -m "ship: a control-plane artifact must match its target and its version |
| 1617 | |
| 1618 | Two refusals read from the bytes that arrived, stdlib only. debug/elf |
| 1619 | catches an amd64 binary declared linux-arm64 — which has happened and |
| 1620 | briefly broke a box, and which -target only protects the careful caller |
| 1621 | from. debug/buildinfo catches a binary whose embedded version.Build is not |
| 1622 | the version being shipped, which is the ldflags trap: forget the flag and |
| 1623 | the roll succeeds while the probe reports 'dev' forever, so a healthy |
| 1624 | deploy reports failure. |
| 1625 | |
| 1626 | Control-plane apps only. A tenant may legitimately ship an artifact we |
| 1627 | would guess wrong about, and refusing their ship on our inference is not |
| 1628 | our call." |
| 1629 | ``` |
| 1630 | |
| 1631 | --- |
| 1632 | |
| 1633 | ### Task 9: `carlos deploy` refuses a derived version from a dirty tree |
| 1634 | |
| 1635 | **Files:** |
| 1636 | - Modify: `deploy.go:52-54` |
| 1637 | - Test: `deploy_test.go` |
| 1638 | |
| 1639 | **Interfaces:** |
| 1640 | - Consumes: nothing. |
| 1641 | - Produces: nothing new; `runDeploy` gains a refusal. |
| 1642 | |
| 1643 | `roll-console.sh` refuses a dirty tree because the stamp is the git short sha, and a stamp naming a commit the binary does not match is how a false "verified" happens. An explicit `-version` is the operator saying they mean it and is not second-guessed. This is a general tenant fix. |
| 1644 | |
| 1645 | - [ ] **Step 1: Write the failing test** |
| 1646 | |
| 1647 | Add to `deploy_test.go`, following the file's existing pattern for driving `runDeploy` against a fake console mux, and using a `t.TempDir()` git repo so the working tree state is controlled: |
| 1648 | |
| 1649 | ```go |
| 1650 | // TestDeployRefusesADerivedVersionFromADirtyTree: the default -version is the |
| 1651 | // git short sha, so a dirty tree ships a stamp naming a commit the binary was |
| 1652 | // not built from — which is precisely how a false "verified" happens, and why |
| 1653 | // roll-console.sh refused it. An EXPLICIT -version is the operator saying they |
| 1654 | // mean it and is not second-guessed. |
| 1655 | func TestDeployRefusesADerivedVersionFromADirtyTree(t *testing.T) { |
| 1656 | dir := initDirtyGitRepo(t) |
| 1657 | err := deployPreflight(dir, "" /* no explicit -version */) |
| 1658 | if err == nil { |
| 1659 | t.Fatal("a dirty tree with a derived version was accepted") |
| 1660 | } |
| 1661 | if !strings.Contains(err.Error(), "-version") { |
| 1662 | t.Fatalf("refusal %q should name the flag that overrides it", err) |
| 1663 | } |
| 1664 | if err := deployPreflight(dir, "abc1234"); err != nil { |
| 1665 | t.Fatalf("an explicit -version from a dirty tree = %v, want accepted", err) |
| 1666 | } |
| 1667 | } |
| 1668 | ``` |
| 1669 | |
| 1670 | Factor the check into a small testable `deployPreflight(dir, explicitVersion string) error` rather than testing through `runDeploy`, which `log.Fatal`s. `initDirtyGitRepo` runs `git init`, commits a file, then modifies it; skip the test if `git` is unavailable. |
| 1671 | |
| 1672 | - [ ] **Step 2: Run to verify it fails** |
| 1673 | |
| 1674 | ``` |
| 1675 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1676 | go test . -run TestDeployRefusesADerivedVersion -count=1 |
| 1677 | ``` |
| 1678 | |
| 1679 | Expected: FAIL — `undefined: deployPreflight`. |
| 1680 | |
| 1681 | - [ ] **Step 3: Implement** |
| 1682 | |
| 1683 | ```go |
| 1684 | // deployPreflight refuses the one combination that produces a release whose |
| 1685 | // name is a lie: a version DERIVED from git (the -version default) taken from |
| 1686 | // a tree with uncommitted changes. The sha then names a commit the artifact |
| 1687 | // was not built from, and every later "which build is this" answer — |
| 1688 | // including the honest ones — inherits the lie. |
| 1689 | // |
| 1690 | // An explicit -version is untouched: that is the operator naming the release |
| 1691 | // themselves, and second-guessing it would break every workflow that versions |
| 1692 | // by something other than a sha. |
| 1693 | func deployPreflight(dir, explicitVersion string) error { |
| 1694 | if explicitVersion != "" { |
| 1695 | return nil |
| 1696 | } |
| 1697 | out, err := exec.Command("git", "-C", dir, "status", "--porcelain").Output() |
| 1698 | if err != nil { |
| 1699 | return nil // not a git checkout, or no git: nothing to be dirty |
| 1700 | } |
| 1701 | if len(bytes.TrimSpace(out)) == 0 { |
| 1702 | return nil |
| 1703 | } |
| 1704 | return errors.New("this tree has uncommitted changes and -version defaults to the git short sha, " + |
| 1705 | "which would name a commit this build does not match — commit, or pass -version explicitly") |
| 1706 | } |
| 1707 | ``` |
| 1708 | |
| 1709 | Call it in `runDeploy` immediately after `-version` resolution, before anything ships. Track whether `-version` was explicitly set by testing the flag's value **before** the `gitShortHead()` default is applied. |
| 1710 | |
| 1711 | - [ ] **Step 4: Run, gate, commit** |
| 1712 | |
| 1713 | ```bash |
| 1714 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test . -run Deploy -count=1 |
| 1715 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1716 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1717 | gofmt -l deploy.go deploy_test.go |
| 1718 | git add deploy.go deploy_test.go |
| 1719 | git commit -m "carlos deploy: refuse a git-derived version from a dirty tree |
| 1720 | |
| 1721 | -version defaults to the git short sha, so a dirty tree ships a stamp |
| 1722 | naming a commit the artifact was not built from — the false-verified shape |
| 1723 | roll-console.sh has always refused. An explicit -version is the operator |
| 1724 | naming the release themselves and is left alone." |
| 1725 | ``` |
| 1726 | |
| 1727 | --- |
| 1728 | |
| 1729 | ### Task 10: The console signals readiness |
| 1730 | |
| 1731 | **Files:** |
| 1732 | - Modify: `cmd/console/main.go:274-276` |
| 1733 | - Create: `internal/host/sdnotify.go` (or `cmd/console/sdnotify.go` if nothing else will use it) |
| 1734 | - Test: `internal/host/sdnotify_test.go` |
| 1735 | |
| 1736 | **Interfaces:** |
| 1737 | - Consumes: nothing. |
| 1738 | - Produces: `func NotifyReady() error` — writes `READY=1` to `$NOTIFY_SOCKET`, a no-op returning nil when the variable is unset. |
| 1739 | |
| 1740 | **Why** (spec §4a): `systemctl try-restart` on a `Type=simple` unit succeeds the moment the process is exec'd, so a console that starts and then panics on its first request exits 0 — `SetVersions` stamps the new build and nothing rolls back. Readiness signalled *after* the listener is open converts that into a failed start, which Task 7's machinery already handles. The unit-file half (`Type=notify`) is a **pinfra** change in the runbook; this task is the code half and is inert until then. |
| 1741 | |
| 1742 | - [ ] **Step 1: Write the failing test** |
| 1743 | |
| 1744 | ```go |
| 1745 | // TestNotifyReadyWritesToTheSocket: the whole contract. systemd hands the |
| 1746 | // process a unixgram socket path in NOTIFY_SOCKET and waits for READY=1 |
| 1747 | // before calling the unit started — which is what turns "exec'd" into |
| 1748 | // "actually serving" for the adopt pass's restart check. |
| 1749 | func TestNotifyReadyWritesToTheSocket(t *testing.T) { |
| 1750 | dir := t.TempDir() |
| 1751 | sock := filepath.Join(dir, "notify.sock") |
| 1752 | conn, err := net.ListenPacket("unixgram", sock) |
| 1753 | if err != nil { |
| 1754 | t.Skipf("unixgram unavailable here: %v", err) |
| 1755 | } |
| 1756 | defer conn.Close() |
| 1757 | t.Setenv("NOTIFY_SOCKET", sock) |
| 1758 | |
| 1759 | if err := NotifyReady(); err != nil { |
| 1760 | t.Fatalf("NotifyReady: %v", err) |
| 1761 | } |
| 1762 | buf := make([]byte, 64) |
| 1763 | conn.SetReadDeadline(time.Now().Add(2 * time.Second)) |
| 1764 | n, _, err := conn.ReadFrom(buf) |
| 1765 | if err != nil { |
| 1766 | t.Fatalf("nothing arrived on the notify socket: %v", err) |
| 1767 | } |
| 1768 | if got := string(buf[:n]); got != "READY=1" { |
| 1769 | t.Fatalf("sent %q, want READY=1", got) |
| 1770 | } |
| 1771 | } |
| 1772 | |
| 1773 | // TestNotifyReadyIsANoOpOutsideSystemd: the console runs under `go run`, in |
| 1774 | // tests and on a laptop, and none of those set NOTIFY_SOCKET. Absent is |
| 1775 | // normal, not an error. |
| 1776 | func TestNotifyReadyIsANoOpOutsideSystemd(t *testing.T) { |
| 1777 | t.Setenv("NOTIFY_SOCKET", "") |
| 1778 | if err := NotifyReady(); err != nil { |
| 1779 | t.Fatalf("NotifyReady with no socket = %v, want nil", err) |
| 1780 | } |
| 1781 | } |
| 1782 | ``` |
| 1783 | |
| 1784 | - [ ] **Step 2: Run to verify it fails** |
| 1785 | |
| 1786 | ``` |
| 1787 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1788 | go test ./internal/host/ -run TestNotifyReady -count=1 |
| 1789 | ``` |
| 1790 | |
| 1791 | Expected: FAIL — `undefined: NotifyReady`. |
| 1792 | |
| 1793 | - [ ] **Step 3: Implement** |
| 1794 | |
| 1795 | ```go |
| 1796 | // NotifyReady tells systemd this process is ready to serve, for a unit |
| 1797 | // declared Type=notify. |
| 1798 | // |
| 1799 | // It exists so that "the process was exec'd" and "the process is serving" |
| 1800 | // stop being the same fact to systemd. With Type=simple, `systemctl restart` |
| 1801 | // succeeds the instant the binary is exec'd — so a console that starts and |
| 1802 | // then dies on its first request, or half-applies a migration, restarts |
| 1803 | // SUCCESSFULLY as far as the adopt pass can tell, and the pass then stamps a |
| 1804 | // version nothing is really running. Signalled after the listener is open, |
| 1805 | // a broken build fails its restart instead, and adoption's existing rollback |
| 1806 | // takes over. |
| 1807 | // |
| 1808 | // No dependency: the protocol is one datagram on the socket systemd names in |
| 1809 | // NOTIFY_SOCKET. An unset variable means "not under systemd" — a laptop, a |
| 1810 | // test, `go run` — and is not an error. |
| 1811 | // |
| 1812 | // An "@" prefix denotes a Linux abstract socket, which net.Dial expects as a |
| 1813 | // leading NUL. |
| 1814 | func NotifyReady() error { |
| 1815 | sock := os.Getenv("NOTIFY_SOCKET") |
| 1816 | if sock == "" { |
| 1817 | return nil |
| 1818 | } |
| 1819 | if strings.HasPrefix(sock, "@") { |
| 1820 | sock = "\x00" + sock[1:] |
| 1821 | } |
| 1822 | conn, err := net.Dial("unixgram", sock) |
| 1823 | if err != nil { |
| 1824 | return err |
| 1825 | } |
| 1826 | defer conn.Close() |
| 1827 | _, err = conn.Write([]byte("READY=1")) |
| 1828 | return err |
| 1829 | } |
| 1830 | ``` |
| 1831 | |
| 1832 | - [ ] **Step 4: Call it after the listener is open** |
| 1833 | |
| 1834 | `cmd/console/main.go` — the current last line is `log.Fatal(http.ListenAndServe(addr, srv))`, which cannot signal readiness because it never returns. Split it: |
| 1835 | |
| 1836 | ```go |
| 1837 | addr := envOr("CONSOLE_ADDR", "127.0.0.1:8090") |
| 1838 | ln, err := net.Listen("tcp", addr) |
| 1839 | if err != nil { |
| 1840 | log.Fatal(err) |
| 1841 | } |
| 1842 | log.Printf("console: %s listening on %s", origin, addr) |
| 1843 | // AFTER the listener is open and the store has migrated (openConsoleDB |
| 1844 | // and NewServer both ran above), so readiness means what it says. With |
| 1845 | // console.service declared Type=notify, a build that dies before this |
| 1846 | // line fails its restart — and the adopt pass rolls the symlink back and |
| 1847 | // starts the previous version, instead of stamping a build that never |
| 1848 | // served (internal/release/adopt.go). |
| 1849 | if err := host.NotifyReady(); err != nil { |
| 1850 | log.Printf("console: notifying systemd of readiness: %v", err) |
| 1851 | } |
| 1852 | log.Fatal(http.Serve(ln, srv)) |
| 1853 | ``` |
| 1854 | |
| 1855 | A failed notify is logged, not fatal: under `Type=notify` systemd will time the start out anyway, and under any other `Type=` the console must keep serving. |
| 1856 | |
| 1857 | - [ ] **Step 5: Run, gate, commit** |
| 1858 | |
| 1859 | ```bash |
| 1860 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off \ |
| 1861 | go test ./internal/host/ -run TestNotifyReady -count=1 -v |
| 1862 | GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off sh -c ' |
| 1863 | go build ./... && go vet ./... && GOOS=darwin go build ./... && go test ./... -count=1' |
| 1864 | gofmt -l cmd/console/main.go internal/host/sdnotify.go internal/host/sdnotify_test.go |
| 1865 | git add -A |
| 1866 | git commit -m "console: signal readiness to systemd after the listener is open |
| 1867 | |
| 1868 | With Type=simple, systemctl restart succeeds the instant the binary is |
| 1869 | exec'd — so a console that dies on its first request restarts |
| 1870 | 'successfully' as far as the adopt pass can tell, and the pass stamps a |
| 1871 | version nothing is running. Readiness signalled after migrate + listen |
| 1872 | makes a broken build fail its restart, which adoption's rollback already |
| 1873 | handles. |
| 1874 | |
| 1875 | Inert until console.service is declared Type=notify, which is the pinfra |
| 1876 | half of this change." |
| 1877 | ``` |
| 1878 | |
| 1879 | --- |
| 1880 | |
| 1881 | ## Rollout runbook |
| 1882 | |
| 1883 | Not TDD tasks — an ordered operational sequence. **Do not start it until tasks 1–10 are merged.** Each step states how to know it worked. Tito first throughout: smaller blast radius, and its console going down does not take carloku with it. |
| 1884 | |
| 1885 | ### R0 — Survey both boxes (read-only) |
| 1886 | |
| 1887 | Flagship's readings are in the spec's appendix. **Tito's are not, and it is deployed first.** For each box collect: `systemctl show -p ExecStart -p Type --value console`; every `console` route with its channel and `unit` (`sqlite3 /data/registry.db`); `sqlite3 /data/console/console.db "select count(*) from owners where app='console'"`; whether `apps/console/` (legacy, unqualified) exists in the bucket; the bucket production flag on `console`. |
| 1888 | |
| 1889 | **Stop and re-plan if:** `owners` has a `console` row (the grant refuses the claim before the owner exemption is reached), or a legacy `apps/console/` prefix exists. |
| 1890 | |
| 1891 | ### R1 — Deliver the code by the legacy path, one last time |
| 1892 | |
| 1893 | **THREE deliveries, not one.** Corrected 2026-08-28, after the branch merged and |
| 1894 | setting up the roll exposed the gap: `roll-console.sh` builds `./cmd/console` for |
| 1895 | the box and a *native* `carlos` used only to drive ship/promote locally (its lines |
| 1896 | 52–53). **It never touches the box's root `/opt/carlos/carlos` binary.** |
| 1897 | |
| 1898 | | Delivery | Carries | How | |
| 1899 | |---|---|---| |
| 1900 | | console | the control-plane flag and its refusal doors, #79, the claim flags, `/healthz`, the ship-door checks, the probe echo | `roll-console.sh <clean-checkout> <box>` | |
| 1901 | | **root binary** | **§4b's restart recovery and its flap bound** — they live in `internal/release/adopt.go` and `internal/host/agent.go` | build `linux/arm64` with the version ldflag, upload to `s3://<deployment-bucket>/bootstrap/carlos`, then `update-platform-binary.sh <profile> <instance> <bucket> <sha256>` | |
| 1902 | | CLI | the build-probe watch, the dirty-tree guard | a release, or a locally built `carlos` for R5–R6 | |
| 1903 | |
| 1904 | Skipping the root-binary delivery is not cosmetic. R5's first |
| 1905 | `carlos deploy -app console` would run against the OLD adoption pass — so if that |
| 1906 | deploy failed, the recovery path would be the exact defect §4b exists to fix: the |
| 1907 | rollback leaving the console down and the next pass stamping over it. Deliver all |
| 1908 | three before R5. |
| 1909 | |
| 1910 | `update-platform-binary.sh` **restarts `carlos-edge`**, the front door for every |
| 1911 | tenant on the box — the one tenant-visible step in this runbook. Do one box at a |
| 1912 | time and verify the edge is serving a real tenant route before moving on. |
| 1913 | |
| 1914 | Verify: `/proc/<pid>/exe` names the sha for the console; `carlos version` on the box |
| 1915 | names it for the root binary; `systemctl is-active carlos-edge carlos-adopt.timer`; |
| 1916 | a real tenant hostname still answers 200; and `curl -s https://<origin>/healthz` |
| 1917 | returns `ok <sha>` — the first proof Task 5 is live. |
| 1918 | |
| 1919 | ### R2 — pinfra: `Type=notify` on `console.service` |
| 1920 | |
| 1921 | **DO NOT flip this at the first opportunity.** Found in the Task 9/10 review, and it |
| 1922 | inverts the naive ordering: |
| 1923 | |
| 1924 | Once the unit is `Type=notify`, a start only succeeds if the started binary sends |
| 1925 | `READY=1`. Adoption's restart-failure path (§4b, `WithStartUnit`) rolls the symlink |
| 1926 | back and starts the **previous** version — and if that previous version predates the |
| 1927 | readiness commit, it never signals, so the rollback's start TIMES OUT and fails. The |
| 1928 | box is then left with no console AND a poisoned new version: the §4b hole again, but |
| 1929 | worse, because the recovery path is now the thing that fails. |
| 1930 | |
| 1931 | So the precondition is **two consecutive readiness-capable console versions on disk** |
| 1932 | under `/opt/carlos/versions/` — i.e. R2 happens only after a SECOND post-R1 roll, so |
| 1933 | that whatever the rollback lands on can also signal. Until then `console.service` |
| 1934 | stays `Type=simple` and §4a's residual (a build that starts and then misbehaves) |
| 1935 | stays open, which is the state we are in today anyway. |
| 1936 | |
| 1937 | Then: add `Type=notify` and a `TimeoutStartSec` comfortably exceeding console |
| 1938 | startup to `console.service` in the pinfra repo; apply to both boxes; |
| 1939 | `systemctl daemon-reload` and restart once. |
| 1940 | |
| 1941 | Verify: `systemctl show -p Type --value console` says `notify`; the console is |
| 1942 | active and serving; and `ls /opt/carlos/versions/` shows at least two |
| 1943 | readiness-capable builds. |
| 1944 | |
| 1945 | ### R3 — Stamp the right route |
| 1946 | |
| 1947 | From R0: the `Unit` stamp must be on **the host named in `ExecStart`**, and on exactly one route. Flagship today has these disagreeing — `ExecStart` names `platform.carlosframework.com`, the stamp is on `console.carloku.com`. |
| 1948 | |
| 1949 | ``` |
| 1950 | carlos route -registry /data/registry.db -host <execstart-host> -unit console.service |
| 1951 | carlos route -registry /data/registry.db -host <previously-stamped-host> -unit "" |
| 1952 | ``` |
| 1953 | |
| 1954 | Run as `carlos`, not root. Verify by reading the routes back, then run one `carlos adopt` and confirm it is a no-op. |
| 1955 | |
| 1956 | ### R4 — Claim `ops/console` with the flag, in one step |
| 1957 | |
| 1958 | ``` |
| 1959 | carlos apps create -app console -account ops -control-plane -build-probe /healthz \ |
| 1960 | -console https://carlos.tito.io |
| 1961 | ``` |
| 1962 | |
| 1963 | then the same against `https://platform.carlosframework.com`. One step, not two: a row without the flag is a window in which any ops member or `Publish` service credential can roll the production console. |
| 1964 | |
| 1965 | Verify: `carlos channels -app console -account ops -console <origin>` lists the existing history — proof the claim adopted the objects rather than creating an empty app. |
| 1966 | |
| 1967 | **Expect a pipeline object to appear.** Since #254, every fresh claim stamps a default one-channel `edge` pipeline, so this claim writes `apps/bab/console/…` pipeline state the earlier steps never mention. That is the normal shape of a claim now, not a sign the adoption went wrong — do not investigate it mid-runbook. |
| 1968 | |
| 1969 | It does not gate anything this runbook does: `canary/*` sits outside every pipeline by design, so `canary/console` promotes and the adoptions in R5–R7 are ungated by it, and `stable` keeps its passkey 403 with or without the pipeline. |
| 1970 | |
| 1971 | ### R5 — First `carlos deploy -app console`, on tito |
| 1972 | |
| 1973 | Build and deploy in the recipe the spec's failure-mode table demands — **`cmd/console`, not the root binary** (the 8-minute outage of 2026-08-08): |
| 1974 | |
| 1975 | ``` |
| 1976 | GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build \ |
| 1977 | -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=$(git rev-parse --short HEAD)" \ |
| 1978 | -o /tmp/console ./cmd/console |
| 1979 | |
| 1980 | carlos deploy -app console -account ops -console https://carlos.tito.io \ |
| 1981 | -channel canary/console -host <tito-execstart-host> -target linux-arm64 /tmp/console |
| 1982 | ``` |
| 1983 | |
| 1984 | Verify: the command reports live off the **probe**, and independently `verify-console.sh tito <sha>` agrees via `/proc`. Both, this once — that agreement is what retires the script. |
| 1985 | |
| 1986 | **If it fails:** `roll-console.sh <checkout> tito` is the break-glass path. |
| 1987 | |
| 1988 | ### R6 — Flagship |
| 1989 | |
| 1990 | The same, against `https://platform.carlosframework.com`. Verify identically. |
| 1991 | |
| 1992 | ### R7 — Prove the ordinary apps |
| 1993 | |
| 1994 | For `kass`, `keymail`, `platform-site`: apply R3's precondition **first** (their `Unit` stamps are unsurveyed, and `kass`/`keymail` are addr-backed bespoke services). An unstamped `keymail` "proof" run is the #110 shape — symlink swapped, nothing restarted, header lies — and would certify a deploy that did not happen. Then `carlos deploy -app <name> -account <acct>` for each. |
| 1995 | |
| 1996 | ### R8 — Documentation |
| 1997 | |
| 1998 | Update CLAUDE.md's "Deploy reality", which currently sends the reader to the pinfra scripts for all four apps. Reclassify `roll-console.sh` and `verify-console.sh` as **break-glass** in their header comments. File the two follow-ups: adoption deduping named-unit restarts to the end of a pass (spec §4c), and how a **fresh deployment** gets its `ops/console` row (spec §11). |
| 1999 | |
| 2000 | --- |
| 2001 | |
| 2002 | ## Self-review |
| 2003 | |
| 2004 | **Spec coverage.** §2 → Task 2. §3/§3a/§3b → Tasks 5, 6, and 8 (the ldflags trap is closed at the door as well as in the timeout). §4a → Task 10 + R2. §4b → Task 7. §4c → R0/R3. §5 → Task 8. §6 → Tasks 1, 3, 4. §7 → Task 9. §8 → Task 1. §9 → R8. §10 → the runbook, same order. §11's three open questions are carried into R0 (stop conditions) and R8 (filed follow-ups). |
| 2005 | |
| 2006 | **Placeholders.** None. Four places name a verification the implementer must do against the codebase rather than a value I could read: `handleAPIPromote`'s request field names (Task 4 step 1), the release-store `Put` helper and create-door status codes (Task 2 step 1), the member-role constant (Task 3 step 1), and `internal/release/adopt_test.go`'s existing fixture (Task 7 step 1). Each says what to check and what to do with the answer — these are "match the surrounding code", not "decide the design". |
| 2007 | |
| 2008 | **Type consistency.** `AppFlags(accountID int64, app string) (bool, string, error)` is used identically in Tasks 3, 4, 6 and 8. `refuseControlPlane(w, accountID, app, verb) bool` is defined in Task 4 and used only there. `watchTarget.Probe` is added in Task 6 and read only there. `NotifyReady() error` is defined in Task 10 and called once. `claimApp`'s new trailing `ownerClaim bool` is added in Task 2 and both call sites are updated in the same task. |
| 2009 | |
| 2010 | **One deliberate deviation from the spec**, flagged for the reviewer: §3a argues the build probe onto "the app row" without naming a column; this plan adds `build_probe TEXT` in the same migration as `control_plane` rather than in a separate task, because splitting them means touching `migrateAppsKey`'s copy list twice and that list is exactly where the spec's §8 trap lives. |
| 2011 | |