| 1 | # Fleet architecture diagram — `/fleet/architecture` |
| 2 | |
| 3 | **Status:** approved in conversation 2026-08-19 (Paul). |
| 4 | **Scope:** one new operator page on the console, one new provider-neutral |
| 5 | infrastructure-inventory layer behind it, and the IAM grant that lets a |
| 6 | deployment see its own cloud estate. |
| 7 | |
| 8 | ## 1. Why |
| 9 | |
| 10 | The console can already answer every question about the platform's own |
| 11 | model of itself — which boxes report, what they run, what they cost under |
| 12 | the active scenario — and cannot answer a single question about the |
| 13 | machine the model sits on. Nothing in CARLOS knows that a box is an EC2 |
| 14 | instance, that a hostname resolves through an Elastic IP, that six |
| 15 | buckets exist, or that an auto-recover alarm is armed. That knowledge |
| 16 | lives in an OpenTofu checkout in another repo and in the AWS console, |
| 17 | which is exactly where CLAUDE.md says day-to-day operation must NOT go. |
| 18 | |
| 19 | The result is that no page anywhere answers "what is actually stood up, |
| 20 | and how does a request get from DNS to a process?". This spec builds that |
| 21 | page: DNS at the entry, routers, boxes with their size, cost, apps and |
| 22 | instance counts, and every other resource the deployment's cloud account |
| 23 | is carrying — including resources CARLOS itself knows nothing about, |
| 24 | which are precisely the ones worth seeing. |
| 25 | |
| 26 | Four rulings shape it (Paul, 2026-08-19): |
| 27 | |
| 28 | - **Live inventory plus CARLOS overlay.** Not a hand-drawn diagram, not a |
| 29 | picture assembled only from what CARLOS already models. The cloud |
| 30 | estate is discovered, and the platform's own knowledge is layered onto |
| 31 | it. |
| 32 | - **Whichever account the console runs in.** No cross-account roles. The |
| 33 | flagship console draws the flagship estate; tito is a separate deploy |
| 34 | and gets its own version of this page by running the same code, as any |
| 35 | third-party hoster would. |
| 36 | - **Rate card prices the boxes.** The discovered instance shape is shown |
| 37 | beside the price and used to flag drift, never to invent a second cost |
| 38 | model that can disagree with `/fleet/economics`. |
| 39 | - **Do not couple the page to AWS.** AWS is where we are, not what the |
| 40 | page is about. One provider interface, one implementation, no plugin |
| 41 | system built speculatively. |
| 42 | |
| 43 | ## 2. Shape of the answer |
| 44 | |
| 45 | ``` |
| 46 | GET /fleet/architecture operator-only page: graph + resource cards |
| 47 | POST /fleet/architecture/refresh run the inventory pass once, now |
| 48 | |
| 49 | infra/inventory.json bucket object, written hourly by the pass |
| 50 | internal/infraview provider-neutral inventory types + Provider |
| 51 | internal/awsinv the AWS implementation (only AWS SDK importer) |
| 52 | internal/console/archgraph.go join + layout, emits SVG |
| 53 | ``` |
| 54 | |
| 55 | ## 3. The provider seam |
| 56 | |
| 57 | `internal/infraview` defines what an inventory IS, in terms no cloud owns: |
| 58 | |
| 59 | ```go |
| 60 | type Resource struct { |
| 61 | Provider string // "aws" | "hetzner" | "self-hosted" |
| 62 | Kind string // see the kind vocabulary below |
| 63 | NativeType string // "ec2:instance", "hetzner:server" — display only |
| 64 | ID string // provider-stable identity (instance id, ARN, bucket name) |
| 65 | Name string // human name (Name tag, bucket name, zone name) |
| 66 | Region string // "" for a global resource |
| 67 | Attrs map[string]string // size, state, ip, bytes, … stringly and additive |
| 68 | Tags map[string]string |
| 69 | } |
| 70 | |
| 71 | type Inventory struct { |
| 72 | Provider string |
| 73 | Account string // account id / project id, for display |
| 74 | TakenAt time.Time |
| 75 | Resources []Resource |
| 76 | Partial []string // human-readable list of reads that failed |
| 77 | } |
| 78 | |
| 79 | type Provider interface { |
| 80 | Inventory(ctx context.Context) (Inventory, error) |
| 81 | } |
| 82 | ``` |
| 83 | |
| 84 | **Kind vocabulary** — deliberately small and neutral: `machine`, `ip`, |
| 85 | `firewall`, `network`, `disk`, `bucket`, `dns_zone`, `dns_record`, |
| 86 | `identity`, `alarm`, `certificate`, `other`. A provider maps its own |
| 87 | types into these; `NativeType` carries the truth for display. `other` is |
| 88 | not a failure — it is the honest home for anything discovered that the |
| 89 | graph has no opinion about, and it must render, not be dropped. |
| 90 | |
| 91 | **`Attrs` is stringly on purpose.** A typed struct per kind would need a |
| 92 | migration every time a provider exposes a new field, and this object is |
| 93 | written to a bucket that must stay parseable by older readers (the |
| 94 | repo's wire-shape rule). Keys the renderer knows about are documented |
| 95 | where it reads them; keys it does not know are shown verbatim in the |
| 96 | resource card. Attr keys used by the joins and the cards: |
| 97 | |
| 98 | | Kind | Attrs | |
| 99 | |---|---| |
| 100 | | `machine` | `type`, `state`, `az`, `public_ip`, `private_ip`, `launched` | |
| 101 | | `ip` | `address`, `attached_to` | |
| 102 | | `disk` | `size_gb`, `volume_type`, `attached_to` | |
| 103 | | `bucket` | `versioning`, `encryption`, `public_access_block` | |
| 104 | | `dns_zone` | `record_count`, `private` | |
| 105 | | `dns_record` | `zone`, `type`, `value` | |
| 106 | | `firewall` | `ingress`, `egress` (rendered summaries, never raw JSON) | |
| 107 | | `alarm` | `state`, `metric`, `target` | |
| 108 | | `identity` | `kind` (`role`/`user`), `attached_policies` (count) | |
| 109 | | `certificate` | `domain`, `status`, `expires` | |
| 110 | |
| 111 | **`internal/awsinv` is the only package in the tree that imports an AWS |
| 112 | SDK.** `archgraph.go`, the handler and the template consume `infraview` |
| 113 | types and never see an AWS type. That is the whole seam: a second |
| 114 | provider is a new file implementing `Provider`, not a rewrite of the |
| 115 | page. |
| 116 | |
| 117 | **No abstraction beyond that.** No provider registry, no config-driven |
| 118 | selection, no speculative second implementation. The console constructs |
| 119 | `awsinv.New()` directly. The cost of adding a provider later is one |
| 120 | package; the cost of a plugin system now is paid every day it is unused. |
| 121 | |
| 122 | ## 4. AWS discovery (`internal/awsinv`) |
| 123 | |
| 124 | Two tiers, both strictly read-only. **No call in this package mutates |
| 125 | anything**, and the IAM grant in §8 contains no write action, so that |
| 126 | property is enforced by the credential and not only by the code. |
| 127 | |
| 128 | **Tier 1 — catch-all.** `resourcegroupstaggingapi:GetResources` per |
| 129 | region returns every taggable resource in the account. This is what makes |
| 130 | "any and all resources" tractable instead of a service-by-service |
| 131 | checklist that silently misses whatever was stood up last week. Regions |
| 132 | come from `ec2:DescribeRegions` (opted-in regions only). Everything it |
| 133 | returns that no typed handler claims becomes a `Kind: "other"` Resource, |
| 134 | grouped by ARN service in the page's "other resources" section. |
| 135 | |
| 136 | **Tier 2 — typed detail** for what the graph draws: |
| 137 | |
| 138 | | Call | Produces | |
| 139 | |---|---| |
| 140 | | `ec2:DescribeInstances` | `machine` (+ type, state, AZ, IPs, tags) | |
| 141 | | `ec2:DescribeVolumes` | `disk` | |
| 142 | | `ec2:DescribeAddresses` | `ip` | |
| 143 | | `ec2:DescribeSecurityGroups` | `firewall` | |
| 144 | | `ec2:DescribeVpcs`, `DescribeSubnets` | `network` | |
| 145 | | `s3:ListBuckets` + `GetBucketVersioning`/`GetBucketEncryption`/`GetPublicAccessBlock`/`GetBucketLocation` | `bucket` | |
| 146 | | `route53:ListHostedZones` + `ListResourceRecordSets` | `dns_zone`, `dns_record` | |
| 147 | | `iam:ListRoles`, `ListUsers`, `ListAttachedRolePolicies` | `identity` | |
| 148 | | `cloudwatch:DescribeAlarms` | `alarm` | |
| 149 | | `acm:ListCertificates` | `certificate` | |
| 150 | |
| 151 | **What is never read and never rendered:** access key material, IAM |
| 152 | policy documents, bucket contents, instance user-data, SSM parameters, |
| 153 | secrets of any kind. Identities appear as name, kind and a count of |
| 154 | attached policies. An operator who wants a policy document goes to AWS; |
| 155 | this page is a map, not a credential viewer. |
| 156 | |
| 157 | **Per-call soft failure.** Each read that fails appends one line to |
| 158 | `Inventory.Partial` and the rest continue. A page that can show eleven |
| 159 | of twelve resource classes shows eleven and says which one is missing — |
| 160 | it never renders "no security groups" because a single call was |
| 161 | throttled. This is `accountfleet.go`'s known-versus-none rule applied to |
| 162 | a dozen reads instead of three. |
| 163 | |
| 164 | **Record-set volume.** `ListResourceRecordSets` is paginated per zone and |
| 165 | a zone can be large. `dns_record` Resources are materialised only for |
| 166 | records that participate in a join (A/AAAA/ALIAS and the apex/wildcard |
| 167 | records); every other record is counted into the zone's `record_count` |
| 168 | and not carried individually. The inventory object stays small enough to |
| 169 | read on every page load. |
| 170 | |
| 171 | ## 5. The pass |
| 172 | |
| 173 | `StartInfraPass(ctx)` in `internal/console/infrapass.go`, modelled line |
| 174 | for line on `StartEconomicsPass` (`economicspass.go:38`): run once at |
| 175 | start, then hourly, log and continue on error. Started from |
| 176 | `cmd/console/main.go` beside the economics pass. |
| 177 | |
| 178 | Each tick calls the provider, marshals the `Inventory`, and CAS-writes |
| 179 | `infra/inventory.json` to `s.cfg.Releases` using the same |
| 180 | read-etag-then-put helper `upsertBill` uses. A console with no bucket |
| 181 | handle configured starts no pass. |
| 182 | |
| 183 | **Failure posture is `cepull.go`'s, verbatim and for the same reasons.** |
| 184 | Construction-class failures — AWS config never resolves, no usable |
| 185 | credential in the chain, an `AccessDenied`-shaped error from the first |
| 186 | call — latch the pass off for the process after a single log line |
| 187 | (`isAuthShapedError` is reused, not reimplemented). There is no "the IAM |
| 188 | grant might appear later" retry worth making, and a deployment with no |
| 189 | grant must show "not granted", never a recurring error. Everything else |
| 190 | — throttling, a 5xx, a network blip — logs and the next tick tries |
| 191 | again. |
| 192 | |
| 193 | `POST /fleet/architecture/refresh` runs one pass synchronously (bounded |
| 194 | by a context timeout), then redirects back to the page with a flash. It |
| 195 | is CSRF-protected, operator-gated, and single-flighted so a leaned-on |
| 196 | button cannot stack passes. It deliberately does NOT clear the latch: a |
| 197 | deployment without the grant gets the same honest answer however many |
| 198 | times the button is pressed. |
| 199 | |
| 200 | ## 6. The graph (`internal/console/archgraph.go`) |
| 201 | |
| 202 | ### 6.1 Nodes and edges |
| 203 | |
| 204 | Typed nodes — `zone`, `record`, `ip`, `router`, `box`, `app`, |
| 205 | `instances`, `bucket`, `alarm`, `identity`, `other` — and typed edges |
| 206 | (`resolves`, `routes`, `runs`, `stores`, `watches`). The builder takes |
| 207 | three inputs and owns every join: |
| 208 | |
| 209 | 1. `infraview.Inventory` (the cloud estate), |
| 210 | 2. `FleetView` + `HostRows` (`view.go` — boxes, staleness, per-box |
| 211 | `Metrics.RouteVersions` and `RouteCounts`), |
| 212 | 3. the registry's routes and the active economics scenario. |
| 213 | |
| 214 | ### 6.2 The joins |
| 215 | |
| 216 | - **DNS → machine**: `dns_record` value → `ip.address` → |
| 217 | `machine.public_ip`. This needs no tags and no naming convention, and |
| 218 | it is the join that makes the picture true rather than decorative. |
| 219 | - **machine ↔ CARLOS box**: `machine.Name` (the `Name` tag, which |
| 220 | `infra/modules/region-host/main.tf:188` sets to `var.name`) matched |
| 221 | against the box label, which is the box's hostname |
| 222 | (`internal/ops/ops.go:319`). Case-insensitive, exact. |
| 223 | - **box → apps and instances**: the box's own |
| 224 | `Metrics.RouteVersions` keys are its hostnames; each is looked up in |
| 225 | the registry for app, channel, version and status. Counts come from the |
| 226 | box's `RouteCounts` (`static`/`resident`/`hibernating`, plus |
| 227 | `awake`/`asleep` which are `*int` and render "unknown" when null). |
| 228 | - **box → cost and size**: the economics rate card's `BoxSpec` for that |
| 229 | box label (falling back to the `""` default spec, exactly as |
| 230 | `AllocateBox` does), priced `HourUSD × 730 + EBSGB × EBSGBMonthUSD`. |
| 231 | The discovered `machine.type` and summed `disk.size_gb` are shown |
| 232 | beside it. A box with **no cloud machine behind it** — a customer's |
| 233 | fleet box, a Pi on a desk — is not priced at all (amended 2026-08-27): |
| 234 | the rate card prices what this platform pays for, so a monthly figure |
| 235 | there would be a claim about someone else's bill, and a "priced by the |
| 236 | default spec" chip would warn about a pricing that should not be |
| 237 | happening. The card still says plainly that there is no cloud machine. |
| 238 | - **box → bucket**: every box that reports edges to the deployment bucket |
| 239 | the console itself is configured with (`s.cfg.Releases`) — the one |
| 240 | bucket relationship the console can state as fact. Per-box replication |
| 241 | targets are not drawn: the console cannot read a box's litestream |
| 242 | config, and a guessed edge on this page would be worse than no edge. |
| 243 | - **alarm → machine**: alarm dimensions naming an instance id. |
| 244 | |
| 245 | ### 6.3 Unjoined is shown, never hidden |
| 246 | |
| 247 | This is the rule that makes the page worth building: |
| 248 | |
| 249 | - A `machine` no box reports is **drift** — an instance nobody is running |
| 250 | CARLOS on, or a box that has gone dark. Drawn, marked "no box |
| 251 | reporting". |
| 252 | - A box with no `machine` is a **customer or BYO box** (the Pi, a fleet |
| 253 | box on someone else's hardware). Drawn outside the provider estate, and |
| 254 | correct — not an error. This is also why the non-cloud path is |
| 255 | exercised on flagship from day one and cannot rot: it is on the page |
| 256 | today. |
| 257 | - A `dns_record` pointing at an address the estate does not own is drawn |
| 258 | as leaving the diagram, not dropped. |
| 259 | - Anything in `other` is listed with its native type and ARN. |
| 260 | |
| 261 | ### 6.4 Layout |
| 262 | |
| 263 | A deterministic layered pass in Go, no JavaScript library, no CDN |
| 264 | (the vendored-icons rule): |
| 265 | |
| 266 | 1. **Layer by kind** — left to right: `zone`, `record`, `ip`, |
| 267 | `router`/`external`, `box`/`machine`, `apps`, `bucket`. `bucket` is the |
| 268 | one satellite drawn as a node; `identity`, `firewall`, `network`, |
| 269 | `disk` and `certificate` are listed in the resource tables below the |
| 270 | diagram instead, because a graph that draws every IAM role is a |
| 271 | hairball. |
| 272 | |
| 273 | **Alarms are drawn ON the node they watch, not as nodes of their own** |
| 274 | (amended 2026-08-27, after seeing the page rendered). They were first |
| 275 | given the column immediately left of the boxes — the only gap in the |
| 276 | flow — and that column is the one every IP-to-box edge crosses. Nodes |
| 277 | paint after edges, so each alarm covered the line running beneath it |
| 278 | and the picture read as a request reaching its box *through* an alarm. |
| 279 | The deeper problem was the modelling: an alarm is not a hop a request |
| 280 | takes, it is a property of the thing it watches. Each alarm is now a |
| 281 | mark on its subject's node, coloured when firing, with its name and |
| 282 | state in the node's hover title — which is what "anchored to its owner" |
| 283 | should have meant all along, and which no layer can put in the way of a |
| 284 | line. The Alarms table still lists every one in full. |
| 285 | |
| 286 | The invariant this cost us is now enforced directly: **no node may sit |
| 287 | on an edge's path** unless it is one of that edge's endpoints. |
| 288 | 2. **Order within a layer** to reduce crossings: one barycentre sweep |
| 289 | down and one up, then a stable sort by name so the same fleet always |
| 290 | produces the same picture. A diagram that reshuffles between refreshes |
| 291 | cannot be read for change. |
| 292 | 3. **Fixed node sizes**, orthogonal edges, emitted as inline SVG with a |
| 293 | `viewBox` so it scales without script. |
| 294 | |
| 295 | Legibility as the fleet grows: instances are drawn as one `instances` |
| 296 | node per box carrying a count, never one node per instance. |
| 297 | |
| 298 | **Not delivered (follow-up, 2026-08-19):** nodes were to link by anchor to |
| 299 | their detail cards below the SVG. They do not; the page needs no JS |
| 300 | regardless, because everything on it is server-rendered, but clicking a |
| 301 | node does nothing and the reader scrolls to find the card. Recorded here |
| 302 | rather than left as a promise the page does not keep — see §10. |
| 303 | |
| 304 | ## 7. The page |
| 305 | |
| 306 | `GET /fleet/architecture`, gated by `handleFleet`'s exact shape |
| 307 | (`server.go:1720`): session → `IsOperator` → bare `http.NotFound` on |
| 308 | every failure including the store error, so the route's existence stays |
| 309 | unprobeable. Operator-only, not member-visible: it names hostnames, |
| 310 | instance ids and the estate's shape. Nav entry beside Status, Metrics and |
| 311 | Economics. |
| 312 | |
| 313 | Body, in order: |
| 314 | |
| 315 | 1. **The graph**, full width. |
| 316 | 2. **Boxes**, one card each: label, region, instance type, cores/memory, |
| 317 | disk, monthly cost, apps, instance counts, release, load, staleness, |
| 318 | and a **drift chip** when the discovered shape contradicts the rate |
| 319 | card entry (or when no rate card entry exists for a reporting box) — |
| 320 | which makes this page a rate-card drift detector as a side effect. |
| 321 | 3. **Resource groups** — compute, network, storage, DNS, identity, |
| 322 | alarms, certificates, other — as tables of `Name / NativeType / |
| 323 | Region / key attrs`. |
| 324 | 4. **Footer**: the snapshot's `TakenAt` as an age, the account id, every |
| 325 | line in `Partial`, and the refresh button. |
| 326 | |
| 327 | **Every section soft-fails independently.** A missing inventory renders |
| 328 | the CARLOS half of the page with an "inventory not available" panel, |
| 329 | naming which of the three states applies: never taken, not granted |
| 330 | (latched), or unreadable — the snapshot object exists but could not be |
| 331 | fetched or parsed. ("Unreadable" rather than "stale", which an earlier |
| 332 | draft of this section said: the snapshot's age is already on the page |
| 333 | beside the account id, so staleness is a number the operator reads, not a |
| 334 | fourth state, and a snapshot that will not parse is the failure actually |
| 335 | worth naming.) A failed registry read costs the overlay, not the page. |
| 336 | Nothing renders a confident "none" for a read that did not happen. |
| 337 | |
| 338 | ## 8. The IAM grant (precondition) |
| 339 | |
| 340 | The inventory needs read-only grants the box role does not have: |
| 341 | `ec2:Describe*`, `tag:GetResources`, `s3:ListAllMyBuckets` and the |
| 342 | per-bucket `Get*Configuration`/`GetBucketLocation` reads, |
| 343 | `route53:List*`, `iam:List*`, `cloudwatch:DescribeAlarms`, |
| 344 | `acm:ListCertificates`. No write action, no `iam:Get*Policy*` (policy |
| 345 | documents are out of scope by §4), no `s3:GetObject`. |
| 346 | |
| 347 | Two places, because the module in this repo is not what provisioned the |
| 348 | flagship: |
| 349 | |
| 350 | 1. `infra/modules/region-host/main.tf` — the module's own |
| 351 | `aws_iam_role_policy`, so any future box gets it. |
| 352 | 2. A matching pinfra change and apply on the flagship box, which is |
| 353 | Paul's to land. |
| 354 | |
| 355 | Until (2) lands, the page renders the CARLOS half and the panel says the |
| 356 | inventory is not granted. **The page must be built and merged without |
| 357 | depending on the grant existing** — the ungranted state is a first-class |
| 358 | rendering, not a broken one. |
| 359 | |
| 360 | **Status (2026-08-19):** the module statement is in this repo as of this |
| 361 | branch. The flagship box runs a role provisioned from pinfra, so the grant |
| 362 | reaches production only when the matching pinfra change is applied — until |
| 363 | then `/fleet/architecture` renders the CARLOS half and reports the estate |
| 364 | as ungranted, which is the designed state and not a fault. |
| 365 | |
| 366 | ## 9. Testing |
| 367 | |
| 368 | - `internal/infraview`: shape round-trip, and the additive rule — an |
| 369 | inventory object written before a field existed still parses, and an |
| 370 | unknown `Kind` renders as `other` rather than being dropped. |
| 371 | - `internal/awsinv`: the AWS calls sit behind narrow interfaces so the |
| 372 | mapping is tested against fakes — instance/EIP/record fixtures in, |
| 373 | `Resource`s out. One test per typed mapping, one for the catch-all |
| 374 | producing `other`, one asserting a failing call lands in `Partial` and |
| 375 | costs only its own resources. No test touches a real AWS endpoint. |
| 376 | - `archgraph`: table-driven join tests — the DNS→IP→machine chain, the |
| 377 | name↔label match, the unmatched-machine and box-without-machine cases, |
| 378 | and the cost/drift computation against a fixture rate card. Layout is |
| 379 | tested for determinism: the same input twice yields byte-identical SVG. |
| 380 | - Handler: the four-way gate (signed out, signed in non-operator, |
| 381 | store error, operator) with the first three byte-identical to a missing |
| 382 | route — the `economicspage_test.go` pattern. Refresh door: CSRF |
| 383 | required, operator required, single-flight. |
| 384 | - Pass: an auth-shaped failure latches after one attempt; a transient |
| 385 | failure does not. |
| 386 | |
| 387 | ## 10. Out of scope |
| 388 | |
| 389 | - Cross-account inventory (tito, or any future estate). Each deploy sees |
| 390 | its own account; that was ruled on. |
| 391 | - A second provider implementation. |
| 392 | - Any write, tag, or remediation action from this page. |
| 393 | - A `carlos` CLI view of the inventory. `infra/inventory.json` is |
| 394 | deliberately a durable bucket object rather than a page-local cache so |
| 395 | that a CLI door is a small follow-up, but it is not this spec. |
| 396 | - Cost Explorer actuals on this page. `/fleet/economics` owns money; |
| 397 | this page owns shape, and shows the modelled per-box figure only. |
| 398 | - **Deferred, not dropped:** anchor-linking each diagram node to its detail |
| 399 | card below the SVG (§6.4). The page ships without it; every fact the |
| 400 | anchors would have jumped to is on the page already. |
| 401 | |
amadan