Fleet status: say which box makes the fleet at-capacity Feature Requestopen
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 57e315fe · claude-code/2.1.284 · DOCS-T1
On 2026-10-01 the carloku fleet status page read CAPACITY "at-capacity" while every row in the Boxes table read LOAD "comfortable". Both readings were true, but the page gave no way to see that.
Capacity bands the worse of DiskFrac and MemFrac, and the fleet card takes the worst box (worstCapacity in fleet.go).
FleetBox carries capacity, but the Boxes table has no column for it.
The cause was one box (Ireland, annie-welding) at 91% disk: 1.9 GB free of 20 GB. Finding that took the private per-box snapshots in the bucket plus an SSM df. An operator reading the public page sees a contradiction instead.
Ask:
Add a Capacity column to the Boxes table, using the band word that is already in fleet.json. This publishes nothing new.
Optional: a Disk/Memory hint, so a reader can tell which resource drove the band. That would be new public information, so it needs a privacy call first.
Related question: internal/metrics/alert.go alerts at DiskFrac > 0.90. Did that alert reach anyone this time? If it didn't, the outage in #111 and this one share an alert-delivery gap, not a probe gap.
What was done: the Ireland root volume was grown from 20 to 60 GiB online, with no restart (platform-infrastructure #38). It now reads 30% used.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 57e315fe · claude-code/2.1.284 · DOCS-T1
Answer to the alert question: no. Nothing could have been notified, on two separate paths.
The platform's own disk rule (metrics.Evaluate, DiskFrac > 0.90) delivers only via CARLOS_ALERT_WEBHOOK, and that variable isn't in /etc/carlos/host.env on the Ireland box. The event has nowhere to go.
The CloudWatch memory alarms publish to per-region SNS topics (carlos-flagship-ie-alerts, carlos-flagship-us-alerts, ...) that have zero subscriptions. A full flagship plan wants to create alerts_email[0] on all four hosts plus alerts_us_email. The likely cause is that the email subscriptions were never confirmed and AWS expired them.
Fixing (2) is a targeted apply in the flagship root, now carlosframework/carloku-infrastructure, followed by Paul confirming four or five emails. Fixing (1) needs a webhook target. Does the platform have a preferred sink, or should an unset webhook be a visible state on the console (alerts: not delivered) instead of silent? I'd suggest the second either way: a box that can't deliver alerts should say so, the same way adoption_running does.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 57e315fe · claude-code/2.1.284 · DOCS-T1
Update 2026-10-01: SNS half fixed. The five alert email subscriptions were re-created with a targeted apply (carloku-infrastructure #1) and Paul confirmed all of them, so CloudWatch memory alarms reach paul@rslw.com on all four regions again.
Still open:
CARLOS_ALERT_WEBHOOK is unset on the boxes, so the platform's own disk/mem/cpu rules still deliver nowhere.
The per-box Capacity column.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 57e315fe · claude-code/2.1.284 · DOCS-T1
Update 2026-10-05. Paul chose email via SNS for the platform's own alerts.
Platform: feat/alerts-sns (0f73987f) adds an SNS sink. CARLOSALERTSNSTOPICARN is published with the box's instance role, and the region is taken from the ARN. The webhook is unchanged. A missing sink is now visible: the private snapshot has metrics.alerts, carlos status --live shows alerts=not-delivered, and the agent logs at start.
Infra: carlosframework/carloku-infrastructure feat/alerts-sns (8602bfa7). It adds four sns:Publish policies, each scoped to its region's topic, and scripts/set-alert-sns-topic.sh, which writes the host.env line over SSM without a restart. A targeted apply of the four policies is 4 to add, with no restarts. A whole-root apply would restart every box (user_data drift), so don't run it.
Correction to ask 1 above: FleetBox does NOT carry capacity. Per-box capacity is deliberately absent from the public feed (contract_test allowlist), so a Capacity column would publish new information and needs a privacy call first. It's left open.
Order: apply the IAM policies, set the env line on the four platform boxes only (customer boxes would page the platform topic), then roll a binary with the sink.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 57e315fe · claude-code/2.1.284 · DOCS-T1
Live on the flagship as of 2026-10-06 09:30 UTC. v0.24.2 is rolled to all four boxes, the four sns:Publish policies are applied (targeted, with no restarts), and CARLOSALERTSNSTOPICARN is set on each box over SSM. Every box's private snapshot reads metrics.alerts {webhook: false, sns: true}, and carlos status --live shows alerts=sns on all four. Platform disk, memory and cpu alerts now reach the per-region alert topics by email. Still open on this thread: the per-box Capacity column, which needs a privacy call because per-box capacity isn't published today.
amadan