| 1 | # AGENTS.md |
| 2 | |
| 3 | Guidance for AI coding agents working in this repository. This is the |
| 4 | canonical agent-instructions file — tool-specific files (`CLAUDE.md`) |
| 5 | are pointers to it, so keep it current *here*. |
| 6 | |
| 7 | CARLOS platform — bucket-as-control-plane deployment platform. Design specs and plans |
| 8 | (`docs/superpowers/{specs,plans}/`, referenced by path throughout the code |
| 9 | comments) live in the private `platform-infrastructure` repo alongside the |
| 10 | deployment roots; read the spec behind whatever you touch. |
| 11 | |
| 12 | ## Delegation policy (Paul's rule, 2026-08-02) |
| 13 | |
| 14 | Delegate mechanical work to subagents on the cheapest model that fits, and |
| 15 | have Opus review anything that grants authority or touches production paths: |
| 16 | |
| 17 | - **Haiku** — transcription-grade work: the plan contains the exact code, |
| 18 | single-file mechanical fixes, doc edits. |
| 19 | - **Sonnet** — implementation from a precise brief: 1-3 files, clear |
| 20 | interfaces, established conventions to follow. |
| 21 | - **Opus** — design judgment, multi-file integration, anything security- or |
| 22 | production-touching; and the **reviewer** for work in those areas. |
| 23 | - **Fable** — where the tier changes the outcome, not the speed (Paul, |
| 24 | 2026-08-03): implementation whose correctness rests on several interacting |
| 25 | invariants at once (converge-grade, root-side, fleet-wide blast radius), |
| 26 | whole-branch final reviews, and deep work that wants a fresh |
| 27 | full-capability context window. It burns usage budget fastest — never |
| 28 | spend it where a precise Sonnet brief or an Opus review already fits. |
| 29 | |
| 30 | Always name the model explicitly when dispatching; an omitted model silently |
| 31 | inherits the session's (usually the most expensive). A precise brief beats a |
| 32 | capable model — exact values, file paths and interfaces go in the brief. |
| 33 | |
| 34 | ## Operate via API and CLI (Paul's rule, 2026-08-06) |
| 35 | |
| 36 | Day-to-day CARLOS administration goes through the console API and the |
| 37 | `carlos` CLI — listing routes, inspecting apps, checking channels, all of it. |
| 38 | Direct AWS access (SSM sessions, S3 console, aws CLI) is reserved for work on |
| 39 | the platform itself: building CARLOS internals, provisioning infra, debugging |
| 40 | below the API surface. |
| 41 | |
| 42 | If a day-to-day task has no API/CLI command, that is a product gap, not a |
| 43 | license to drop to AWS: plan and build the command (or file the gap) instead |
| 44 | of working around it. Box-local-only counts as a gap too. (2026-08-06: |
| 45 | `carlos instances` lists routes but only by opening the registry DB on local |
| 46 | disk, so an agent went to Systems Manager to run it — the missing API-backed |
| 47 | remote form was the bug.) |
| 48 | |
| 49 | The design-side corollary (Paul, 2026-08-08): **every `carlos` command is |
| 50 | designed for a user with no access to the underlying infra and no box |
| 51 | commands.** The command alone must complete the job — no follow-up SSM |
| 52 | session, AWS console visit, or on-box verb may be required for it to |
| 53 | deliver or verify its result. |
| 54 | |
| 55 | ## Production infrastructure needs scoped approval (Paul's rule, 2026-09-15) |
| 56 | |
| 57 | Before creating, changing, or deleting infrastructure resources or records in |
| 58 | production or shared accounts, or under a real company or customer domain, |
| 59 | get explicit user approval for the scoped work. This includes temporary |
| 60 | experiments and applies through every interface: DNS, SES identities, |
| 61 | configuration and tenants, IAM users, roles, policies and credentials, |
| 62 | queues, topics, buckets, compute, and related resources. |
| 63 | |
| 64 | Prepare a concrete plan before asking for approval. State the account and |
| 65 | region, domains, exact resource changes, purpose, permissions and traffic |
| 66 | effects, expected cost, duration, and cleanup and verification steps. One |
| 67 | explicit approval covers the complete scoped experiment and its cleanup; do |
| 68 | not ask repeatedly for each API call. Existing explicit scoped authorization |
| 69 | and routine, preauthorized deployment workflows remain valid. Ask again if |
| 70 | the work materially expands the approved account, region, domain, resource |
| 71 | type, permissions, traffic, cost, or duration. |
| 72 | |
| 73 | Read-only investigation and local mocks may proceed without approval. Live |
| 74 | tests may run unattended within an explicitly designated sandbox scope with |
| 75 | restricted permissions and automatic expiry. A sandbox domain alone does |
| 76 | not authorize IAM or other account-level changes in a production or shared |
| 77 | account; those still need scoped approval. A name beginning |
| 78 | with `spike-`, a planned cleanup, working credentials, or a generic instruction |
| 79 | to implement or test does not constitute approval or designate a sandbox. |
| 80 | |
| 81 | This rule supplements the AWS access rule above and applies to subagents. |
| 82 | Pass the approved scope to delegates so they can check their work against it. |
| 83 | |
| 84 | The rationale is concrete: on September 15, an SES spike created |
| 85 | `spike.teamtito.com`, DNS records, IAM credentials, and messaging in Tito's |
| 86 | production account. It was removed minutes later, but cleanup does not |
| 87 | substitute for approval. |
| 88 | |
| 89 | ## Every change lands through a reviewed branch on amadan (Paul's rule, 2026-08-10; amadan made the source of truth by Bill, 2026-09-24) |
| 90 | |
| 91 | **Nothing reaches `main` except through a branch landed on amadan.** No |
| 92 | local merge to `main`, no direct commit to `main`, no fast-forward "it's |
| 93 | only a docs change" — every change, however small, goes up as a branch |
| 94 | and lands through `amadan branch merge`. That includes doc-only edits, |
| 95 | one-line fixes, and revert commits. |
| 96 | |
| 97 | **amadan (`carlos/platform` on amadan.net) is the source of truth.** Its |
| 98 | `main` is the real one, its CI is the only CI, and its branch description |
| 99 | is where a change's context lives. GitHub is a mirror of it: do not open |
| 100 | pull requests there, and never merge there. (Until 2026-09-24 this rule |
| 101 | said "through a PR" and pointed at GitHub, while the issues rule below |
| 102 | already said `main` lands through `amadan branch merge`. Both were |
| 103 | followed, and GitHub-first merges ran CI only after the merge. That |
| 104 | contradiction is what this rewrite removes.) |
| 105 | |
| 106 | The steps, for agents and people alike: |
| 107 | |
| 108 | 1. **Push the branch to amadan.** The push is what triggers CI. The remote |
| 109 | may be named `amadan` or `origin` depending on the checkout; it is |
| 110 | whichever one points at `amadan.net`. |
| 111 | 2. **Describe it** — what changes, why, how it was tested and reviewed. |
| 112 | This is the pull request's job and it carries the same weight: |
| 113 | |
| 114 | amadan branch describe carlos/platform <branch> -body - |
| 115 | |
| 116 | 3. **Squash it to one commit** before landing, with a message that says |
| 117 | what the change is and why. `amadan branch merge` has no squash mode, |
| 118 | so this is done on the branch: `main` then gets the hub's merge commit |
| 119 | and exactly one change commit under it, not the branch's iterations. |
| 120 | The iterations are not lost: the description and the reviews record |
| 121 | them. Squashing moves the tip, so CI runs again on it. |
| 122 | 4. **Land it once CI is green on that tip:** |
| 123 | |
| 124 | amadan ci status carlos/platform -branch <branch> |
| 125 | amadan branch merge carlos/platform <branch> -expect <tip> -message "..." |
| 126 | |
| 127 | The merge refuses a branch whose CI is not green (exit 2) and, with |
| 128 | `-expect`, one that moved after you checked it. That ordering is the |
| 129 | point: a red run stops the merge instead of landing on `main` as an |
| 130 | incident to clean up. |
| 131 | |
| 132 | **Merging is fine** (Paul, 2026-08-16, clarifying an earlier wording that |
| 133 | read as a blanket ban and cost a session a round trip). The rule is about |
| 134 | how work REACHES `main`, not about whose hand lands it: the described |
| 135 | branch has to exist, so that it can carry the context, the iterations and |
| 136 | the review. Once it is green, `amadan branch merge` is an ordinary step. |
| 137 | |
| 138 | Use the judgement the rest of this file asks for: a branch that is still |
| 139 | collecting review, that is gated on something, or whose diff you would |
| 140 | want a second pair of eyes on, waits for a person. |
| 141 | |
| 142 | ## Work is tracked on amadan, not GitHub (Paul's rule, 2026-09-05) |
| 143 | |
| 144 | **Do not open issues on github.com/carlosframework/platform.** Not for a |
| 145 | bug you hit, not for a follow-up you noticed while doing something else, |
| 146 | not for a "just so it's written down somewhere" note. That tracker is |
| 147 | closed for new work. |
| 148 | |
| 149 | Everything goes on amadan instead, as a discussion on `carlos/platform`: |
| 150 | |
| 151 | amadan discuss new carlos/platform -title "..." -body - |
| 152 | amadan discuss list carlos/platform # what's already there |
| 153 | amadan discuss show carlos/platform <id> |
| 154 | amadan discuss reply carlos/platform <id> -body - |
| 155 | |
| 156 | Give it a type — `Issue`, `Feature Request` or `Idea` — with |
| 157 | `amadan discuss type`, or leave it untyped and let a person label it. |
| 158 | `amadan discuss types` lists what this repo uses. |
| 159 | |
| 160 | The reason is that a tracker nobody reads is worse than no tracker. CI |
| 161 | moved here on 2026-08-24 and `main` lands here through `amadan branch |
| 162 | merge`; GitHub issues were the last thing left pointing somewhere the |
| 163 | daily loop had stopped looking. 110 open ones had accumulated by |
| 164 | 2026-09-05, and triaging every one against `main` found 20 already |
| 165 | fixed — several of them by work that cited the issue number in its own |
| 166 | comments and still never closed it — plus one duplicate. The remaining |
| 167 | 89 were migrated to discussions that day, each carrying its full |
| 168 | original text, and all 110 were closed on GitHub with a comment saying |
| 169 | where it went and why. |
| 170 | |
| 171 | That ratio is the argument. Roughly one issue in six was answered |
| 172 | without anyone noticing, which is what a tracker looks like when it has |
| 173 | stopped being part of the loop. |
| 174 | |
| 175 | Reading the old ones is still fine and often useful — code comments cite |
| 176 | them by number constantly (`issue #195`, `(#152)`), and those citations |
| 177 | are historical references, not an invitation to reopen anything. When a |
| 178 | comment cites a GitHub issue that is still real outstanding work, it now |
| 179 | has a discussion too; cite that alongside it rather than renumbering |
| 180 | anything. |
| 181 | |
| 182 | ## Build & test |
| 183 | |
| 184 | - `go build ./...`, `go vet ./...`, `go test ./... -count=1` — all clean, |
| 185 | every package (21 as of 2026-08-08; the count grows), before any commit |
| 186 | lands. |
| 187 | - `gofmt -l .` clean over the whole tree — `make test` checks it, and the |
| 188 | three pre-existing unformatted files that used to be exempted are gone. |
| 189 | - **`.amadan/ci.d/` is the only CI this repo has.** The GitHub workflow |
| 190 | that used to run this gate was deleted on 2026-08-24, once the amadan |
| 191 | steps covered everything it did and more; `.amadan/README.md` records |
| 192 | what each step descends from, and `.amadan/ci.d/10-gate`'s header |
| 193 | records why the switch was safe. `10-gate` is `go build ./...` beside |
| 194 | `make test`, so CI and what you run locally are one definition rather |
| 195 | than two that can drift. |
| 196 | - **`make test` splits the slowest packages' tests across processes** |
| 197 | (`.amadan/test-gate.sh`, `TEST_SPLIT`). Each shard is its own process, |
| 198 | so a test must not depend on another top-level test having run first |
| 199 | in the same process. Shards are dealt by `.amadan/test-weights.tsv`; |
| 200 | refresh it when the gate's shard times in the CI log drift far apart |
| 201 | (its header says how). A test missing from it still runs. |
| 202 | - **CI is triggered by a push to amadan, not by GitHub.** Nothing on |
| 203 | GitHub runs a build. A branch that exists only on GitHub has been gated |
| 204 | by exactly nothing. Read a branch's verdict with |
| 205 | `amadan ci status carlos/platform -branch <branch>` (exit 0 pass, 1 |
| 206 | fail, 2 pending, 3 no status yet) before calling it green. |
| 207 | - **After a merge, read `main`'s run too.** Each branch was green against |
| 208 | the base it was cut from, not against what landed beside it. A red |
| 209 | `main` is worth knowing about the same day. On 2026-08-31, when merges |
| 210 | still happened on GitHub, amadan's `main` was 86 commits and three days |
| 211 | behind, and the gate had not run on `main` once in that window; landing |
| 212 | on amadan is what ends that. |
| 213 | - **GitHub follows amadan.** If GitHub's `main` is behind amadan's after a |
| 214 | merge, fast-forward it from amadan; never the other way round, and |
| 215 | never with anything that is not already on amadan's `main`. |
| 216 | - `20-cross@5` adds `GOOS=darwin`/`windows`/`linux-arm64` compiles, one |
| 217 | per shard. **The gate is not Linux-only**: `internal/host` is linked |
| 218 | into the CLI, which people run on Macs, so a Linux-only type assumption |
| 219 | breaks `go build .` for them. Cross-compiling is enough to catch it — |
| 220 | no Mac required. |
| 221 | - Sandbox note: use `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod |
| 222 | GOPROXY=off`; go commands may need the sandbox disabled ("read-only file |
| 223 | system" from the toolchain is the tell). |
| 224 | |
| 225 | ## The census: an approval, not a decision (Paul, 2026-10-24, #369) |
| 226 | |
| 227 | `TestPlan14RootReadCensus` (`internal/provision`) fails when a branch |
| 228 | changes a unit that reads root-owned state, or a package inside the |
| 229 | census envelope. Every such change needs the census owner's approval on |
| 230 | #280 before the merge. Most are approvals: the same reads in an edited |
| 231 | body, or an envelope fingerprint that moved. Get them approved like this: |
| 232 | |
| 233 | 1. `make census-card` prints what changed and writes nothing. Each change |
| 234 | has a class: |
| 235 | - `body`: the digest moved and the reads did not; |
| 236 | - `sites`: more or fewer reads of kinds the unit already makes; |
| 237 | - `envelope`: a package was re-pinned; |
| 238 | - `reads` (a kind of read the unit did not make before), `new`, |
| 239 | `removed` or `other` (any census discrepancy none of these explains, |
| 240 | shown with the census's own message): a decision. |
| 241 | 2. Write a CLAIMS file with one line per changed unit: |
| 242 | `<file>:<unit><TAB><what changed, and that no read was added or which one was>`. |
| 243 | 3. `make census-propose NAME=<branch> CLAIMS=<file>` writes the rest: |
| 244 | - the overlay (`testdata/plan14-root-reads-<NAME>.tsv`), each unit |
| 245 | re-frozen at its existing disposition with the claim appended; |
| 246 | - the `@legacy` fingerprints, the envelope re-pins and the new seal; |
| 247 | - its line in `testdata/plan14-root-reads-chain.tsv`; |
| 248 | - the approval card, `docs/superpowers/notes/census-proposal-<NAME>.md`. |
| 249 | |
| 250 | It checks that the overlay leaves the census clean, and writes nothing |
| 251 | if any step fails. If the code changes again, rerun it with the same |
| 252 | NAME: it replaces its own overlay (only one it wrote itself), which must |
| 253 | still be the last line of the chain. |
| 254 | 4. Show the card to the census owner and ask for an approval. Ask in the |
| 255 | session when they are there (one question: approve or hold); otherwise |
| 256 | post the card on #280. Record the approval on #280: who, when, and the |
| 257 | tip. Rows carry `rev3=census-ruling-<NAME>` from the start, so there is |
| 258 | nothing to flip afterwards and no second CI run. |
| 259 | |
| 260 | The tool refuses decisions (`reads`, `new`, `removed`, `other`): a new |
| 261 | kind of read can make a unit's disposition wrong, so it is the owner's |
| 262 | call, not an approval. Write those rows by hand, following the overlays in the |
| 263 | chain, and ask on #280 with the card attached; the census owner decides |
| 264 | those. Nothing about the census's strength changes: every changed read is |
| 265 | still named, digested and seen by a person before merge. The record-write |
| 266 | census (9B8, the `*_census_test.go` inventories) still takes hand-written |
| 267 | entries. |
| 268 | |
| 269 | ## Rules this repo has paid for |
| 270 | |
| 271 | - **A rule stated as an absence can only be checked by looking for the absent |
| 272 | thing, and cannot see a second door.** (2026-09-04, object-storage scanning.) |
| 273 | Every scan verdict rests on the app being unable to tag its own object, and |
| 274 | that was enforced by keeping `s3:PutObjectTagging` out of the app's Allow — |
| 275 | stated in capitals in `storePolicyJSON`, and asserted by a helper that |
| 276 | searched the policy for the string `"s3:PutObjectTagging"`. Correct, enforced, |
| 277 | and insufficient: **there are two ways to tag an object**, and the second — |
| 278 | `x-amz-tagging` on `PutObject` — names no action at all, so no search for an |
| 279 | action name could ever have found it. Against MinIO the pre-tagged upload |
| 280 | succeeded, and an explicit `Deny` on `s3:PutObjectTagging` did not close it; |
| 281 | only a `Deny` on `s3:PutObject` conditioned on `s3:RequestObjectTagKeys` did. |
| 282 | A second hole in the same property survived that fix: an app filling the |
| 283 | ten-tag budget with its own tags made its objects permanently *unmarkable*, |
| 284 | so the `enforce` Deny — which keys on the tag being present — never engaged. |
| 285 | So: **`s3:PutObjectTagging` must never be in the app's Allow, and its absence |
| 286 | is not the property.** The property is "this credential cannot put this tag, |
| 287 | by any route", it is asserted against a real server in |
| 288 | `TestDeliveredContractMinio`, and a policy change is not done until that test |
| 289 | has been watched to fail. |
| 290 | - **When a storage shape changes, the consumer list is everything that PARSES |
| 291 | the shape** — not everything that calls the API. (Learned three times.) |
| 292 | - **A new `carlos` command is not done until `docs/site/cli.md` names it.** |
| 293 | `internal/docsite` reads main.go's dispatch switch and fails the gate over a |
| 294 | command with no heading on the CLI page, over a fenced example running a verb |
| 295 | that no longer exists, and over a `/docs` link whose page or `#fragment` does |
| 296 | not resolve. The check is a heading and not a substring on purpose: `route` is |
| 297 | a prefix of `routes`, and a substring check let a deleted section pass. The |
| 298 | corpus is rendered by `carlosframework/website`, which vendors a copy — so a |
| 299 | change here reaches the site only after that repo re-syncs and ships. |
| 300 | - **Wire-shape changes are additive**: live pointers/manifests exist in |
| 301 | production buckets; anything written before your change must still parse. |
| 302 | - **Production channel names are pinned in tests** |
| 303 | (`internal/release/laddercheck_test.go`) — the fleet runs on `canary/*` |
| 304 | channels today; a ladder change that refuses them stops deploys. |
| 305 | - **Adoption swaps a symlink; it does not restart processes.** Order is: |
| 306 | promote → adopt → restart. Long-running units (console, keymail relay) need |
| 307 | the restart; hibernating tenants pick it up on wake. |
| 308 | - **A route's Channel and Environment belong to its instance record** once the |
| 309 | route is `Provisioned` (main, 2026-08-10). The provisioning reconciler |
| 310 | converges both from `apps/<sqid>/<app>/instances/<host>.json` on every pass, |
| 311 | so a `carlos route -channel` on the box lands and is reverted within seconds |
| 312 | of the next serial bump — `carlos route` refuses both axes on such a row and |
| 313 | names the record. Reconcile also ADOPTS an existing route whose shape matches |
| 314 | an appearing record, which is how the fleet gets backfilled with no box |
| 315 | commands. Adoption stamps `Provisioned`, and `Provisioned` is the one |
| 316 | condition that GRANTS the removal phase authority over a row (the loop also |
| 317 | gates on region/pool scope and on the record being gone) — so adoption is an |
| 318 | authority transfer, |
| 319 | and the shape match is what stands between a backfill and a retired |
| 320 | production route. It refuses any row carrying a column the record cannot |
| 321 | express (`Backing`, `Tunnel`, `Caps`) or owned by another pass (`Platform`, |
| 322 | `Canary`). |
| 323 | - **A ResponseWriter wrapper that edits headers needs its own header map.** |
| 324 | Wrappers in `internal/edge` embed the writer below, so `Header()` is ONE |
| 325 | shared map from handler to connection — and `maybeStore` caches |
| 326 | `rec.Header().Clone()`. The gzip wrapper (`internal/edge/compress.go`) keeps |
| 327 | a private map and writes `Content-Encoding` only to the real writer's; on the |
| 328 | shared map it would have stored an identity body labelled gzip and served it |
| 329 | to every later visitor. Two corollaries it paid for: a header set AFTER |
| 330 | `WriteHeader` is a TRAILER, so the wrapper must sync late additions down or |
| 331 | it silently eats them; and canonical for `ETag` is `Etag`, so a raw map-key |
| 332 | comparison spelled `"ETag"` matches nothing. Ordering is load-bearing too: |
| 333 | recorder ABOVE the compressor (cache stores identity), usage counter BELOW it |
| 334 | (tenants are billed for wire bytes), wrap site AFTER the upgrade branch. |
| 335 | - **Console security gates are ordered**: session → CSRF → owner (byte-identical |
| 336 | 404) → `requireFreshAuth` LAST. A non-member must never get a redirect that |
| 337 | confirms an account exists. |
| 338 | - **Verify against the thing you changed, with the binary you built.** Stale |
| 339 | DNS answers and stale `CARLOS_BIN` binaries have both produced false |
| 340 | "verified" reports in one day. |
| 341 | - Icons are vendored Lucide (in-code SVG constants, never CDN); emoji is |
| 342 | content, icons are chrome. |
| 343 | - **A test that shells out to the carlos binary must build its env through |
| 344 | `carlosTestEnv`** (masks `CARLOS_CREDENTIALS`, `HOME`, `CARLOS_CONSOLE`, |
| 345 | `CARLOS_DESTINATION`) |
| 346 | **and set `cmd.Dir` to a scratch dir** — the binary walks cwd ancestors for |
| 347 | `.carlos/config`. The fallback is the operator's real login and default |
| 348 | console; a logged-in laptop running `go test ./...` minted 40+ real accounts |
| 349 | on the production console before this was isolated. A `t.TempDir()` is |
| 350 | scratch in that sense only because the root package's `TestMain` |
| 351 | (`isolateTempRoot`, main_test.go) checks both temp roots, TMPDIR and |
| 352 | GOTMPDIR (`t.TempDir()` uses GOTMPDIR when it is set), and moves whichever |
| 353 | has a `.carlos/config` above it, which is the case wherever one is under a |
| 354 | logged-in home (#226, #258). The move is to a new |
| 355 | `/var/tmp/carlos-test-*` directory, named on stderr and removed when the |
| 356 | run ends. A run that is aborted (`log.Fatal`, a panic, a timeout) leaves |
| 357 | that one directory behind: remove it by the name that was printed, never |
| 358 | by pattern. `/tmp` is never used, and with `/var/tmp` unavailable the |
| 359 | binary stops and says which variable to set. `TestMain` also drops an |
| 360 | exported `CARLOS_CONFIG` and `CARLOS_DESTINATION`, which no shared helper |
| 361 | pins. A test that leaves its cwd at the checkout is not covered by any of |
| 362 | that. |
| 363 | - **A sidecar is a member that nothing can dial.** `Kind == "sidecar"` rows |
| 364 | carry no socket and no addr, and are refused by the edge ON KIND — not |
| 365 | merely unmatched by it — before dispatch ever looks at shape. That refusal |
| 366 | has to be explicit: a fleet-placed sidecar row reaches the edge's fleet |
| 367 | arm on shape alone (a `Tunnel`/`Region` it never should have earned), so |
| 368 | `publishResidentFleetBindings` also skips it — a sidecar is never |
| 369 | advertised as served in a fleet binding. Separately, each reconcile pass's |
| 370 | removal authority is scoped to its OWN kind: without that scoping the |
| 371 | instance pass and the sidecar pass delete each other's rows on every |
| 372 | tick, forever, with no error anywhere. Use `registry.Route.IsSidecar()` in |
| 373 | any code that needs the check — never the bare `"sidecar"` literal. |
| 374 | - **A known-failing test is a defect in its own right.** `log.Fatal` inside |
| 375 | an in-process test aborts the whole binary rather than failing the one |
| 376 | test, so everything after it silently never runs — and what gets read |
| 377 | afterward is the failure COUNT, not the names. A red root package sat |
| 378 | "known-failing" for a week while `TestStoreVerbDispatchAndRefusals` hid a |
| 379 | stale usage assertion behind it, and CI stayed red on every push with |
| 380 | nobody the wiser (discussion #17). |
| 381 | - **Absence of `apps/<acct>/<app>/pipeline.json` = the legacy ladder**, |
| 382 | byte-for-byte; new apps get a one-channel `edge` pipeline at create; |
| 383 | per-channel rules are console-enforced, bake windows box-enforced through |
| 384 | the `FirstSeen.Holds` ratchet. |
| 385 | |
| 386 | ## Deploy reality (2026-08-02; convergence rewritten 2026-08-08) |
| 387 | |
| 388 | - Deployment-specific facts (which apps live where, box names, accounts, |
| 389 | ship/promote scripts) live in private deployment repos, not here: |
| 390 | `carlosframework/carloku-infrastructure` for the flagship (carloku.com, |
| 391 | split out 2026-10-01; the tofu root is the repo root) and |
| 392 | `carlosframework/platform-infrastructure` for dev and the design docs. |
| 393 | The signing key comes from the deployment's secret store inside those |
| 394 | scripts, never a terminal. |
| 395 | - **The flagship consumes the platform modules at a release tag** |
| 396 | (`?ref=vX.Y.Z` of the GitHub mirror), not `?ref=main`, the same way Tito |
| 397 | and any self-hoster do. A module change on main does not reach carloku |
| 398 | until a release is cut, its tag is pushed to the GitHub mirror |
| 399 | (`git push https://github.com/carlosframework/platform.git |
| 400 | refs/tags/vX.Y.Z`; the mirror had no tags past v0.9.0 until v0.21.0 went |
| 401 | up on 2026-10-01), every `?ref=` in carloku-infrastructure is bumped, and |
| 402 | `tofu init -upgrade` plus a read plan pass. Dev still tracks `main`. |
| 403 | - keymail.dev deploys via `carlos ship`/`promote`, NOT via its repo's legacy |
| 404 | `deploy/deploy.sh` — see that repo's CLAUDE.md. |
| 405 | - **Since main 1fba18f (PR #108, live both boxes 2026-08-08): convergence is |
| 406 | seconds, not ticks.** Console-mediated writes bump `control/serial`; boxes |
| 407 | poll it at 2s and run the full pass set. Direct-bucket promotes (pinfra |
| 408 | scripts) don't bump but canary/* channels are name-scoped fast anyway. |
| 409 | Blocked stages relay to a root pass (`carlos-stage.path` — must be armed; |
| 410 | update-platform-binary.sh does it). That root pass stages, repoints and |
| 411 | stamps in ~8s but cannot cycle the agent's exec children, so the agent's |
| 412 | 2s tick finishes the job: a warm instance whose route is stamped with the |
| 413 | version its symlink names is cycled at once (`convergeAdopted`, #231). The |
| 414 | activator sweep's 90s grace is only the backstop. `carlos deploy` is the |
| 415 | member one-command |
| 416 | form: ship → promote → wait until `X-Carlos-Version` on the URL reports the |
| 417 | shipped build over 2xx. Verify deploys with that header (`curl -sI |
| 418 | --resolve`), never a cached answer. `-I` is fine for the VERSION header, but |
| 419 | it sends a HEAD — and HEAD is excluded from edge compression, so anything |
| 420 | about `Content-Encoding` needs a GET (`curl -s -D - -o /tmp/b`) instead. |
| 421 | - **Console restarts: manual until the route is unit-stamped** (#110, fixed |
| 422 | in main 916cd6b, not yet rolled). Routes can now name the systemd unit that |
| 423 | owns their process (`carlos route -registry /data/registry.db -host |
| 424 | <console-host> -unit console.service`, run as carlos, not root); a stamped |
| 425 | route is cycled by adoption itself, and a failed named-unit restart rolls |
| 426 | the adoption back instead of stamping a lying header. Until a box runs |
| 427 | 916cd6b+ AND its console row is stamped, the old rule stands: promote → |
| 428 | adopt → `systemctl restart console.service`, and the header overclaims |
| 429 | until the restart. Ordinary unit/exec tenants restart themselves either way. |
| 430 | - **`carlos restart -app X`** (#113, merged 916cd6b, live after the same |
| 431 | roll): member-facing process cycle with no version or config change — |
| 432 | console door touches `apps/<acct>/<app>/restart-gen.json` + bumps the |
| 433 | serial; boxes act on the serial move (exec children bounced in-process, |
| 434 | unit-shaped routes via root `carlos adopt -restart-only` through |
| 435 | `carlos-restart.path`, which update-platform-binary.sh arms since pinfra |
| 436 | PR #10). Hibernating instances are never woken. Static routes carry |
| 437 | `X-Carlos-Version` from the same roll (#112), so `carlos deploy -kind |
| 438 | static -host <site>` verifies static sites too. |
| 439 | - `carlos-system-update.path` relays the agent's platform self-update request |
| 440 | to `carlos system apply` (root; spec 2026-09-06-carlos-system-update-design.md |
| 441 | §6). `carlos-system-recover.service` runs once per boot, before the edge, |
| 442 | and on the apply's failure. Both must be armed; the last run of |
| 443 | `update-platform-binary.sh` does it, and the hourly `carlos adopt` pass |
| 444 | installs and arms them if it did not. A box follows `CARLOS_SYSTEM_CHANNEL` |
| 445 | (canary|stable, default stable) in host.env. |
| 446 | - **`carlos restart --host` / `--environment` / `--dry-run`** (discussion |
| 447 | #16): the same verb narrowed to one route, or to everything reading one |
| 448 | config bundle — the shape that follows `carlos secrets set --environment |
| 449 | relay`, since a bundle is delivered to an ENVIRONMENT. `--host` wins over |
| 450 | `--environment`, and the console refuses the pair when that host does not |
| 451 | read that environment; a selector matching nothing is refused with what |
| 452 | the app actually has, never widened. The narrowed request is a SECOND |
| 453 | bucket object (`apps/<acct>/<app>/restart-scope.json`, an append log of |
| 454 | `{id, requested_at, scope}`) reached through a SECOND console path |
| 455 | (`POST .../restart/scope`), because both old readers act on the app-wide |
| 456 | object alone: an old box never reads the log and an old console answers |
| 457 | 404, so neither can turn a narrowed request into a whole-app restart. A |
| 458 | box steps OVER a selector `kind` it does not know rather than falling |
| 459 | back to the app — never widen on ignorance. Boxes remember the last |
| 460 | request id they acted on (`scopes` in `restart-marks.json`, additive in |
| 461 | both directions); a mark that has rolled off the log re-baselines and |
| 462 | cycles nothing. |
| 463 | |
| 464 | ## Short platform aliases (Paul's request, 2026-09-15) |
| 465 | |
| 466 | Tito may delegate direct app-domain aliases to named accounts with |
| 467 | CONSOLE_APPS_DOMAIN_ALIAS_ACCOUNTS (comma-separated sqids; empty by default). |
| 468 | A listed account can claim an unoccupied single-label hostname below |
| 469 | PLATFORM_APPS_DOMAIN through the existing domain attach API. This is explicit |
| 470 | operator delegation of namespace authority, not DNS ownership evidence. |
| 471 | The exception supersedes blanket ancestor deny entries only for these names; |
| 472 | exact denied hosts, Origin, RedirectHosts, SteeringHost, account-sqid labels, |
| 473 | the domain apex and nested account namespaces remain protected. Alias targets |
| 474 | still require an existing app route. First-route creation and wildcard or |
| 475 | catchall capabilities are refused. Global claims and deployment-route conflict |
| 476 | checks remain authoritative. Tito's initial approved account is bap (Pullup). |
| 477 | |
| 478 | The delegation only means something on a deployment that sets |
| 479 | CONSOLE_RESERVE_PLATFORM_DOMAINS=true (below). Without that setting, the |
| 480 | apps domain is not reserved, so every account can already attach names under |
| 481 | it, and the delegated-alias limits do not apply. |
| 482 | |
| 483 | ## Reserving platform domains is opt-in (James, 2026-09-17) |
| 484 | |
| 485 | `carlos domains attach` refuses everything under PLATFORM_APPS_DOMAIN and |
| 486 | under the console's own hostnames ONLY when the console sets |
| 487 | CONSOLE_RESERVE_PLATFORM_DOMAINS=true. That rule protects tenants from each |
| 488 | other, so a deployment that hosts apps for other people (the flagship, |
| 489 | carloku, Tito) must set it. The default is off: a self-hosted deployment |
| 490 | usually keeps its console, its apps and its custom domains in one zone that |
| 491 | one person owns, and the rule refused ordinary names there. Every deployment |
| 492 | still refuses the console's own hostnames by exact name, the steering host, |
| 493 | CONSOLE_DENY_DOMAINS, and a hostname another app's route holds. |
| 494 | |
| 495 | Two refusals do NOT depend on the setting (#215): every deployment refuses a |
| 496 | name in another live account's `<sqid>.<apps-domain>` namespace, and a first |
| 497 | route under a sqid-shaped namespace that is not the caller's own. |
| 498 | |
| 499 | ## The operator's own hostnames (#209, Paul, 2026-10-03) |
| 500 | |
| 501 | The deny list has two tiers (domainHostPolicy, internal/console/domainpolicy.go). |
| 502 | **Platform** names are CONSOLE_DENY_DOMAINS entries and, with |
| 503 | CONSOLE_RESERVE_PLATFORM_DOMAINS, the names under the console's own hostnames. |
| 504 | Members are refused them. A platform operator (an ops account owner, or an |
| 505 | address on CARLOS_SYSTEM_OPERATORS) may attach one to an app in an account they |
| 506 | are a member of, with `carlos domains attach --platform`. That is how carloku.com |
| 507 | runs as an ordinary console app. **Reserved** names are refused to everyone, |
| 508 | operators included: the console's own hostnames by exact name, the steering |
| 509 | host, the tunnel host (Config.TunnelHost, kept out of DenyDomains so the door |
| 510 | cannot reach it) and, with the reserve setting, the apps domain. |
| 511 | |
| 512 | Once a platform name is claimed, every later write to that claim is the |
| 513 | operator's alone: a member of the holding account cannot repoint it, flip |
| 514 | wildcard or catchall, publish its paths or detach it. These checks read the |
| 515 | hostname, never the claim, so they reveal nothing about who holds it. The |
| 516 | browser form has no platform door. Its attach refuses platform names to |
| 517 | everyone, and its detach admits the operator. |
| 518 | |
| 519 | ## Instance catch-alls (James, 2026-10-05) |
| 520 | |
| 521 | A claim on `*.<parent>` (`carlos domains attach --app <a> '*.<parent>'`) |
| 522 | serves each of the app's instances at `<label>.<parent>`. Every box derives |
| 523 | ordinary `route_aliases` rows from it for its own instance routes |
| 524 | (`domains.ReconcileAliases`). The design is |
| 525 | `docs/superpowers/specs/2026-10-05-instance-catchall-design.md`. Four rules |
| 526 | hold it together; keep all of them: |
| 527 | |
| 528 | - **Only a person, in the account that holds `<parent>`'s own claim, may |
| 529 | attach one.** The box re-checks the parent on every pass, and a catch-all |
| 530 | whose parent has left the account is inert. |
| 531 | - **An exact claim on a derived name always wins, whoever holds it.** A |
| 532 | real route also wins over a derived name. |
| 533 | - **While a catch-all is live, the attach doors refuse other accounts the |
| 534 | names it reserves**, with the neutral 409. Without that, any account |
| 535 | could claim a team's name, and its exact claim would win. |
| 536 | - It reserves **every** one-label name under the parent once the account |
| 537 | proves the parent in DNS: TXT `_carlos-catchall.<parent>` = |
| 538 | `carlos-catchall-account=<sqid>`, checked live. |
| 539 | - Otherwise it reserves only the labels of the app's existing instances, |
| 540 | under any instance domain, so an unproven apex claim cannot block a |
| 541 | whole domain. |
| 542 | - It never refuses the caller's own existing claim. |
| 543 | - The box decides "claimed" from the bucket listing, never from the |
| 544 | bodies that decoded. |
| 545 | - A label two instances share derives nothing. |
| 546 | - **Derivation takes only provisioned, non-sidecar, non-replica routes of |
| 547 | the claim's own account and app.** |
| 548 | |
| 549 | ## Proof of control for custom domains is opt-in (#215, 2026-10-04) |
| 550 | |
| 551 | With CONSOLE_DOMAIN_PROOF=true, an attached hostname is stored pending and is |
| 552 | neither routed nor given a certificate until a TXT record at |
| 553 | `_carlos-challenge.<host>` carries the account's token (`carlos domains |
| 554 | verify`). A pending claim yields to a proven attach from another account, and |
| 555 | lapses after seven days. Without it, attach is first-writer-wins exactly as |
| 556 | before: an account can take over a name whose DNS already points at the |
| 557 | edge, or hold any name it never controlled. A deployment that hosts apps for |
| 558 | other people must set it. Roll the fleet first: a box from before #215 reads |
| 559 | a pending claim as live and routes it. The default is off because the |
| 560 | 2026-08-26/27 scope ruling kept first-writer-wins; turning it on per |
| 561 | deployment is the operator's decision. A pending claim does not hold a |
| 562 | parent for an instance catch-all. |
| 563 | |
amadan