carlos / platform Public

The amadan-side multiplier (eager, permanent, unrecoverable repo instances) is amadan/amadan discussion #19: https://amadan.net/amadan/amadan/discussions/19

Built, reviewed, not yet merged or applied — platform PR carlosframework/platform#336 and platform-infrastructure#28. Spec and plan: platform-design/superpowers/{specs,plans}/2026-08-28-memory-pressure-containment* on the pinfra branch.

What the box says now (read-only SSM, after the resize) — four corrections to the write-up above:

• carlos-exec@amadan.net is 257 MB anonymous across the 25 repo children (595 MB with page cache). The 47 MB "each" is RSS, which counts shared pages 25 times.
• The agent's own leaf (supervise/) is charged 1.2 GB, 1.12 GB of it page cache from release-store reads — reclaimable, and the first thing any ceiling on the tree reclaims.
• keymail is keymail-relay.service (343 MB), an ordinary unit; it is not under the edge tree. The tree has two leaves.
• flagship/region-ie.tf still said t4g.small: a plain tofu apply would have resized the box back down. Fixed on the branch.

What's built (all four mitigations):

1. zram via scripts/harden-box.sh <profile> <instance> (idempotent, SSM) + the same lines in every host-init.sh / .tftpl. ram/2, zstd, swappiness 180, page-cluster 0.
2. carlos-edge.service: MemoryHigh=70%, MemoryMax=80% — in the embedded unit file, not a drop-in, so it rolls with the binary. Plus OOMPolicy=continue, which the proposal misses: systemd's default stop turns one tenant's OOM kill into a whole-edge restart. And, from review: the memcg OOM killer scores per process by RSS, so the outer wall's first victim would have been the agent (110 MB vs 20–60 MB tenant processes) — every exec child now gets oom_score_adj=+500 at spawn.
3. cgroup.Manager.Start writes memory.max/memory.high/memory.oom.group per carlos-exec@<host>: 512M default, 1G when the manifest declares capabilities.children. amadan-hub declares nothing today, so amadan needs "capabilities": {"children": true} in its manifest before the platform rolls, or it lands on 512M — the cap bounds total charge (anon + cache), and 595 MB under a 512M cap is permanent reclaim.
4. Detection: Route53 checks on amadan.net + console.carloku.com → alarms → email; and a memwatch agent duty (not a shell timer) publishing MemAvailablePercent + PSI some/avg60 to CloudWatch Carlos/Host once a minute, MemAvailablePercent < 15% for 3 min alarm per box, missing data = breaching. Alarms and their SNS topics are per region (an alarm cannot notify a cross-region topic), and the memory alarms are wired in a second apply once every box publishes.

Rollout order is spec §4 and is load-bearing; nothing has touched the live box.

Deployed 2026-08-28 (Paul: "merge and deploy"). platform #336 + #342 + #343 merged, binary 0ac8c98 rolled to all four flagship boxes and both dev boxes; pinfra #28 merged and applied (A then B).

Live on the flagship now: zram (1.9 GB on IE, ~930 MB on the t4g.smalls), carlos-edge at MemoryHigh 70% / MemoryMax 80% / OOMPolicy=continue, every exec child at oomscoreadj +500, carlos-exec@amadan.net capped at 1G (amadan-hub 8a866cd-mem ships capabilities.children), memwatch publishing Carlos/Host per region, memory-low alarms OK on IE/US/AU (UK still evaluating), reachability alarms on amadan.net + console.carloku.com OK.

Three things the rollout taught, none in the spec's first draft:

• With zram on the box, memory.max alone is escapable — a 200 MB allocator under a 64M cap just swapped. memory.swap.max (= max) is now written alongside it (#342), proven with systemd-run -p MemoryMax=64M -p MemorySwapMax=64M → killed.
• host.env's fleet-wide AWS_REGION=eu-west-1 sent every box's metrics to Ireland while each box's alarm lives in its own region. memwatch now publishes to the IMDS placement region (#343).
• kill -9 of an amadan repo child leaves the edge untouched (NRestarts 0, amadan.net 200) — and the child stays dead, which is amadan #19's half.

Remaining: dev's tofu root can't be applied (DNSimple record drift, #319), so dev boxes have zram + the binary but no IAM grant/alarm; tito boxes are tito's roll.

Status check, three days on. Everything in the write-up above is merged and applied; what is left is two box classes that the flagship roll could not reach.

Verified now:

• platform #336 (2026-08-28 16:48), #342 (17:10), #343 (17:19) — all MERGED.
• platform-infrastructure #28 "Memory pressure: zram, alerts, memory alarms, region-ie.tf truth" — MERGED 17:24.
• The flagship serves x-carlos-version: 5ef69c6, which is well after 0ac8c98, so the binary-embedded half is live there: carlos-edge at MemoryHigh 70% / MemoryMax 80% / OOMPolicy=continue, exec children at oomscoreadj +500, per-carlos-exec@<host> memory.max/high/oom.group and memory.swap.max, memwatch publishing to Carlos/Host.
• Tito serves x-carlos-version: c3f021d (2026-08-29, PR #355), and 0ac8c98 is an ancestor of it. So the tito boxes have the binary half too, without anyone having to do anything for it — which is the point of putting those limits in the embedded unit file rather than a drop-in.

What is NOT done, and the split matters because the two halves protect against different things:

1. dev boxes have zram and the binary but no IAM grant and no alarm — dev's tofu root cannot be applied because of DNSimple record drift. Tracked as platform #319, still OPEN.
2. tito boxes have the binary half and, as far as I can tell, none of the account-side half: no zram, no Carlos/Host alarms, no Route53 reachability checks. Those live in the tito account's own tofu and in scripts/harden-box.sh, so a flagship apply cannot reach them; it needs a tito-side apply. I can find nothing tracking it — not in teamtito/carlos-infrastructure (one open issue, #8, about the region fleet), not in platform, not in platform-infrastructure.

Item 2 is the one worth a decision. The binary half caps a tenant and stops the OOM killer picking the agent, which is most of the original failure. But zram is the change that turns "kernel thrashes forever with nothing to reclaim" into "things get slow, then something gets killed" — the actual mechanism that took the flagship dark for 15 minutes — and the alarms are the reason anyone would find out. A tito box under the same pressure today would still go quiet without paging anyone.

Happy to file that against teamtito/carlos-infrastructure and write the tito profile into harden-box.sh, but it is a tito-side apply and by the standing rule that roll is tito's to ask for, so I have not.

Leaving this open on account of the two above rather than closing it as deployed.

Triage, 2026-09-05. Nothing here has moved since the three-days-on status check, and the two remaining items are both blocked on somebody else's apply rather than on any work in this repo.

Restating what is left, because the split is what makes this still worth having open:

1. dev boxes — zram and the binary half, no IAM grant and no alarm, because dev's tofu root cannot be applied over DNSimple record drift.
2. tito boxes — the binary half only. No zram, no Carlos/Host alarms, no Route 53 reachability checks. Those live in the tito account's own tofu and in scripts/harden-box.sh, so no flagship apply reaches them.

Item 2 is the one that matters and the reasoning above is right: the binary half caps a tenant and stops the OOM killer picking the agent, which is most of the original failure, but zram is the change that addresses the actual mechanism that took the flagship dark for fifteen minutes. With no swap the kernel had nothing to evict but page cache and flushed it in a loop instead of killing anything; the OOM killer never fired. A tito box under the same pressure today still has nothing to reclaim, and no alarm to say so.

My recommendation is to take the offer that was already made. Writing the tito profile into harden-box.sh and filing against teamtito/carlos-infrastructure costs little and is not a roll — it puts a reviewed, idempotent script and a tracked issue in front of the people whose apply it is. Holding back the preparation because the apply is theirs is stricter than the standing rule needs to be, and the current state is that nothing anywhere tracks it: not in teamtito/carlos-infrastructure, not in platform, not in platform-infrastructure. An untracked gap on a box class that can go dark is worse than an unwelcome issue.

Item 1 stays parked behind the DNSimple drift, which is its own piece of work.

Keeping open. The incident is closed and the flagship is fixed; this is now a coverage gap on two box classes, and it would be reasonable to retitle it as such so the open item is not mistaken for an unresolved outage.

Blocked on: your say-so to prepare the tito-side change and file it.