Fleet architecture diagram — /fleet/architecture

Status: approved in conversation 2026-08-19 (Paul). Scope: one new operator page on the console, one new provider-neutral infrastructure-inventory layer behind it, and the IAM grant that lets a deployment see its own cloud estate.

1. Why

The console can already answer every question about the platform's own model of itself — which boxes report, what they run, what they cost under the active scenario — and cannot answer a single question about the machine the model sits on. Nothing in CARLOS knows that a box is an EC2 instance, that a hostname resolves through an Elastic IP, that six buckets exist, or that an auto-recover alarm is armed. That knowledge lives in an OpenTofu checkout in another repo and in the AWS console, which is exactly where CLAUDE.md says day-to-day operation must NOT go.

The result is that no page anywhere answers "what is actually stood up, and how does a request get from DNS to a process?". This spec builds that page: DNS at the entry, routers, boxes with their size, cost, apps and instance counts, and every other resource the deployment's cloud account is carrying — including resources CARLOS itself knows nothing about, which are precisely the ones worth seeing.

Four rulings shape it (Paul, 2026-08-19):

• Live inventory plus CARLOS overlay. Not a hand-drawn diagram, not a picture assembled only from what CARLOS already models. The cloud estate is discovered, and the platform's own knowledge is layered onto it.
• Whichever account the console runs in. No cross-account roles. The flagship console draws the flagship estate; tito is a separate deploy and gets its own version of this page by running the same code, as any third-party hoster would.
• Rate card prices the boxes. The discovered instance shape is shown beside the price and used to flag drift, never to invent a second cost model that can disagree with /fleet/economics.
• Do not couple the page to AWS. AWS is where we are, not what the page is about. One provider interface, one implementation, no plugin system built speculatively.

2. Shape of the answer

GET /fleet/architecture      operator-only page: graph + resource cards
POST /fleet/architecture/refresh   run the inventory pass once, now

infra/inventory.json         bucket object, written hourly by the pass
internal/infraview           provider-neutral inventory types + Provider
internal/awsinv              the AWS implementation (only AWS SDK importer)
internal/console/archgraph.go   join + layout, emits SVG

3. The provider seam

internal/infraview defines what an inventory IS, in terms no cloud owns:

type Resource struct {
	Provider   string            // "aws" | "hetzner" | "self-hosted"
	Kind       string            // see the kind vocabulary below
	NativeType string            // "ec2:instance", "hetzner:server" — display only
	ID         string            // provider-stable identity (instance id, ARN, bucket name)
	Name       string            // human name (Name tag, bucket name, zone name)
	Region     string            // "" for a global resource
	Attrs      map[string]string // size, state, ip, bytes, … stringly and additive
	Tags       map[string]string
}

type Inventory struct {
	Provider  string
	Account   string    // account id / project id, for display
	TakenAt   time.Time
	Resources []Resource
	Partial   []string  // human-readable list of reads that failed
}

type Provider interface {
	Inventory(ctx context.Context) (Inventory, error)
}

Kind vocabulary — deliberately small and neutral: machine, ip, firewall, network, disk, bucket, dns_zone, dns_record, identity, alarm, certificate, other. A provider maps its own types into these; NativeType carries the truth for display. other is not a failure — it is the honest home for anything discovered that the graph has no opinion about, and it must render, not be dropped.

Attrs is stringly on purpose. A typed struct per kind would need a migration every time a provider exposes a new field, and this object is written to a bucket that must stay parseable by older readers (the repo's wire-shape rule). Keys the renderer knows about are documented where it reads them; keys it does not know are shown verbatim in the resource card. Attr keys used by the joins and the cards:

| Kind | Attrs | |---|---| | machine | type, state, az, public_ip, private_ip, launched | | ip | address, attached_to | | disk | size_gb, volume_type, attached_to | | bucket | versioning, encryption, public_access_block | | dns_zone | record_count, private | | dns_record | zone, type, value | | firewall | ingress, egress (rendered summaries, never raw JSON) | | alarm | state, metric, target | | identity | kind (role/user), attached_policies (count) | | certificate | domain, status, expires |

internal/awsinv is the only package in the tree that imports an AWS SDK. archgraph.go, the handler and the template consume infraview types and never see an AWS type. That is the whole seam: a second provider is a new file implementing Provider, not a rewrite of the page.

No abstraction beyond that. No provider registry, no config-driven selection, no speculative second implementation. The console constructs awsinv.New() directly. The cost of adding a provider later is one package; the cost of a plugin system now is paid every day it is unused.

4. AWS discovery (internal/awsinv)

Two tiers, both strictly read-only. No call in this package mutates anything, and the IAM grant in §8 contains no write action, so that property is enforced by the credential and not only by the code.

Tier 1 — catch-all. resourcegroupstaggingapi:GetResources per region returns every taggable resource in the account. This is what makes "any and all resources" tractable instead of a service-by-service checklist that silently misses whatever was stood up last week. Regions come from ec2:DescribeRegions (opted-in regions only). Everything it returns that no typed handler claims becomes a Kind: "other" Resource, grouped by ARN service in the page's "other resources" section.

Tier 2 — typed detail for what the graph draws:

| Call | Produces | |---|---| | ec2:DescribeInstances | machine (+ type, state, AZ, IPs, tags) | | ec2:DescribeVolumes | disk | | ec2:DescribeAddresses | ip | | ec2:DescribeSecurityGroups | firewall | | ec2:DescribeVpcs, DescribeSubnets | network | | s3:ListBuckets + GetBucketVersioning/GetBucketEncryption/GetPublicAccessBlock/GetBucketLocation | bucket | | route53:ListHostedZones + ListResourceRecordSets | dns_zone, dns_record | | iam:ListRoles, ListUsers, ListAttachedRolePolicies | identity | | cloudwatch:DescribeAlarms | alarm | | acm:ListCertificates | certificate |

What is never read and never rendered: access key material, IAM policy documents, bucket contents, instance user-data, SSM parameters, secrets of any kind. Identities appear as name, kind and a count of attached policies. An operator who wants a policy document goes to AWS; this page is a map, not a credential viewer.

Per-call soft failure. Each read that fails appends one line to Inventory.Partial and the rest continue. A page that can show eleven of twelve resource classes shows eleven and says which one is missing — it never renders "no security groups" because a single call was throttled. This is accountfleet.go's known-versus-none rule applied to a dozen reads instead of three.

Record-set volume. ListResourceRecordSets is paginated per zone and a zone can be large. dns_record Resources are materialised only for records that participate in a join (A/AAAA/ALIAS and the apex/wildcard records); every other record is counted into the zone's record_count and not carried individually. The inventory object stays small enough to read on every page load.

5. The pass

StartInfraPass(ctx) in internal/console/infrapass.go, modelled line for line on StartEconomicsPass (economicspass.go:38): run once at start, then hourly, log and continue on error. Started from cmd/console/main.go beside the economics pass.

Each tick calls the provider, marshals the Inventory, and CAS-writes infra/inventory.json to s.cfg.Releases using the same read-etag-then-put helper upsertBill uses. A console with no bucket handle configured starts no pass.

Failure posture is cepull.go's, verbatim and for the same reasons. Construction-class failures — AWS config never resolves, no usable credential in the chain, an AccessDenied-shaped error from the first call — latch the pass off for the process after a single log line (isAuthShapedError is reused, not reimplemented). There is no "the IAM grant might appear later" retry worth making, and a deployment with no grant must show "not granted", never a recurring error. Everything else — throttling, a 5xx, a network blip — logs and the next tick tries again.

POST /fleet/architecture/refresh runs one pass synchronously (bounded by a context timeout), then redirects back to the page with a flash. It is CSRF-protected, operator-gated, and single-flighted so a leaned-on button cannot stack passes. It deliberately does NOT clear the latch: a deployment without the grant gets the same honest answer however many times the button is pressed.

6. The graph (internal/console/archgraph.go)

6.1 Nodes and edges

Typed nodes — zone, record, ip, router, box, app, instances, bucket, alarm, identity, other — and typed edges (resolves, routes, runs, stores, watches). The builder takes three inputs and owns every join:

1. infraview.Inventory (the cloud estate),
2. FleetView + HostRows (view.go — boxes, staleness, per-box Metrics.RouteVersions and RouteCounts),
3. the registry's routes and the active economics scenario.

6.2 The joins

• DNS → machine: dns_record value → ip.address → machine.public_ip. This needs no tags and no naming convention, and it is the join that makes the picture true rather than decorative.
• machine ↔ CARLOS box: machine.Name (the Name tag, which infra/modules/region-host/main.tf:188 sets to var.name) matched against the box label, which is the box's hostname (internal/ops/ops.go:319). Case-insensitive, exact.
• box → apps and instances: the box's own Metrics.RouteVersions keys are its hostnames; each is looked up in the registry for app, channel, version and status. Counts come from the box's RouteCounts (static/resident/hibernating, plus awake/asleep which are *int and render "unknown" when null).
• box → cost and size: the economics rate card's BoxSpec for that box label (falling back to the "" default spec, exactly as AllocateBox does), priced HourUSD × 730 + EBSGB × EBSGBMonthUSD. The discovered machine.type and summed disk.size_gb are shown beside it. A box with no cloud machine behind it — a customer's fleet box, a Pi on a desk — is not priced at all (amended 2026-08-27): the rate card prices what this platform pays for, so a monthly figure there would be a claim about someone else's bill, and a "priced by the default spec" chip would warn about a pricing that should not be happening. The card still says plainly that there is no cloud machine.
• box → bucket: every box that reports edges to the deployment bucket the console itself is configured with (s.cfg.Releases) — the one bucket relationship the console can state as fact. Per-box replication targets are not drawn: the console cannot read a box's litestream config, and a guessed edge on this page would be worse than no edge.
• alarm → machine: alarm dimensions naming an instance id.

6.3 Unjoined is shown, never hidden

This is the rule that makes the page worth building:

• A machine no box reports is drift — an instance nobody is running CARLOS on, or a box that has gone dark. Drawn, marked "no box reporting".
• A box with no machine is a customer or BYO box (the Pi, a fleet box on someone else's hardware). Drawn outside the provider estate, and correct — not an error. This is also why the non-cloud path is exercised on flagship from day one and cannot rot: it is on the page today.
• A dns_record pointing at an address the estate does not own is drawn as leaving the diagram, not dropped.
• Anything in other is listed with its native type and ARN.

6.4 Layout

A deterministic layered pass in Go, no JavaScript library, no CDN (the vendored-icons rule):

1. Layer by kind — left to right: zone, record, ip, router/external, box/machine, apps, bucket. bucket is the one satellite drawn as a node; identity, firewall, network, disk and certificate are listed in the resource tables below the diagram instead, because a graph that draws every IAM role is a hairball.

Alarms are drawn ON the node they watch, not as nodes of their own (amended 2026-08-27, after seeing the page rendered). They were first given the column immediately left of the boxes — the only gap in the flow — and that column is the one every IP-to-box edge crosses. Nodes paint after edges, so each alarm covered the line running beneath it and the picture read as a request reaching its box through an alarm. The deeper problem was the modelling: an alarm is not a hop a request takes, it is a property of the thing it watches. Each alarm is now a mark on its subject's node, coloured when firing, with its name and state in the node's hover title — which is what "anchored to its owner" should have meant all along, and which no layer can put in the way of a line. The Alarms table still lists every one in full.

The invariant this cost us is now enforced directly: no node may sit on an edge's path unless it is one of that edge's endpoints.

1. Order within a layer to reduce crossings: one barycentre sweep down and one up, then a stable sort by name so the same fleet always produces the same picture. A diagram that reshuffles between refreshes cannot be read for change.
2. Fixed node sizes, orthogonal edges, emitted as inline SVG with a viewBox so it scales without script.

Legibility as the fleet grows: instances are drawn as one instances node per box carrying a count, never one node per instance.

Not delivered (follow-up, 2026-08-19): nodes were to link by anchor to their detail cards below the SVG. They do not; the page needs no JS regardless, because everything on it is server-rendered, but clicking a node does nothing and the reader scrolls to find the card. Recorded here rather than left as a promise the page does not keep — see §10.

7. The page

GET /fleet/architecture, gated by handleFleet's exact shape (server.go:1720): session → IsOperator → bare http.NotFound on every failure including the store error, so the route's existence stays unprobeable. Operator-only, not member-visible: it names hostnames, instance ids and the estate's shape. Nav entry beside Status, Metrics and Economics.

Body, in order:

1. The graph, full width.
2. Boxes, one card each: label, region, instance type, cores/memory, disk, monthly cost, apps, instance counts, release, load, staleness, and a drift chip when the discovered shape contradicts the rate card entry (or when no rate card entry exists for a reporting box) — which makes this page a rate-card drift detector as a side effect.
3. Resource groups — compute, network, storage, DNS, identity, alarms, certificates, other — as tables of Name / NativeType / Region / key attrs.
4. Footer: the snapshot's TakenAt as an age, the account id, every line in Partial, and the refresh button.

Every section soft-fails independently. A missing inventory renders the CARLOS half of the page with an "inventory not available" panel, naming which of the three states applies: never taken, not granted (latched), or unreadable — the snapshot object exists but could not be fetched or parsed. ("Unreadable" rather than "stale", which an earlier draft of this section said: the snapshot's age is already on the page beside the account id, so staleness is a number the operator reads, not a fourth state, and a snapshot that will not parse is the failure actually worth naming.) A failed registry read costs the overlay, not the page. Nothing renders a confident "none" for a read that did not happen.

8. The IAM grant (precondition)

The inventory needs read-only grants the box role does not have: ec2:Describe*, tag:GetResources, s3:ListAllMyBuckets and the per-bucket Get*Configuration/GetBucketLocation reads, route53:List*, iam:List*, cloudwatch:DescribeAlarms, acm:ListCertificates. No write action, no iam:Get*Policy* (policy documents are out of scope by §4), no s3:GetObject.

Two places, because the module in this repo is not what provisioned the flagship:

1. infra/modules/region-host/main.tf — the module's own aws_iam_role_policy, so any future box gets it.
2. A matching pinfra change and apply on the flagship box, which is Paul's to land.

Until (2) lands, the page renders the CARLOS half and the panel says the inventory is not granted. The page must be built and merged without depending on the grant existing — the ungranted state is a first-class rendering, not a broken one.

Status (2026-08-19): the module statement is in this repo as of this branch. The flagship box runs a role provisioned from pinfra, so the grant reaches production only when the matching pinfra change is applied — until then /fleet/architecture renders the CARLOS half and reports the estate as ungranted, which is the designed state and not a fault.

9. Testing

• internal/infraview: shape round-trip, and the additive rule — an inventory object written before a field existed still parses, and an unknown Kind renders as other rather than being dropped.
• internal/awsinv: the AWS calls sit behind narrow interfaces so the mapping is tested against fakes — instance/EIP/record fixtures in, Resources out. One test per typed mapping, one for the catch-all producing other, one asserting a failing call lands in Partial and costs only its own resources. No test touches a real AWS endpoint.
• archgraph: table-driven join tests — the DNS→IP→machine chain, the name↔label match, the unmatched-machine and box-without-machine cases, and the cost/drift computation against a fixture rate card. Layout is tested for determinism: the same input twice yields byte-identical SVG.
• Handler: the four-way gate (signed out, signed in non-operator, store error, operator) with the first three byte-identical to a missing route — the economicspage_test.go pattern. Refresh door: CSRF required, operator required, single-flight.
• Pass: an auth-shaped failure latches after one attempt; a transient failure does not.

10. Out of scope

• Cross-account inventory (tito, or any future estate). Each deploy sees its own account; that was ruled on.
• A second provider implementation.
• Any write, tag, or remediation action from this page.
• A carlos CLI view of the inventory. infra/inventory.json is deliberately a durable bucket object rather than a page-local cache so that a CLI door is a small follow-up, but it is not this spec.
• Cost Explorer actuals on this page. /fleet/economics owns money; this page owns shape, and shows the modelled per-box figure only.
• Deferred, not dropped: anchor-linking each diagram node to its detail card below the SVG (§6.4). The page ships without it; every fact the anchors would have jumped to is on the page already.