amadan
Recovery update: queue visibility returned around 10:01 UTC and the Tito CI-status endpoint is responding again. The edge error view reports no errors in the latest three-minute window. This reduces urgency but does not prove the resource-pressure hypothesis; keep the diagnostic/capability request open. We used only supported Carlos controls (logs, status, errors and host-scoped restart).
Expose effective app memory limits and OOM kills through supported operations Issue open
Amadan CI incident: https://amadan.net/amadan/amadan/-/discussion/60 . carlos status --instances reports amadan.net (account bah / app amadan-hub / host annie-welding) around 500–506 MB; its process was killed again at 10:00:55 UTC after a supported restart at 09:59. Git operations and Tito CI claim/report/status endpoints time out. Memory pressure near the default 512 MiB tier is a hypothesis, not confirmed OOM. We need a supported way to inspect the effective memory tier, cgroup OOM events, and the deployed release capability block, and to correct the resource allocation if needed. The app spawns repository children; confirm whether its deployed manifest declares children and receives the 1 GiB child-process tier. Current CLI offers usage and restart, but not this diagnostic detail. Please provide the platform capability; do not use SSH or hand-edit cgroups as a workaround.
Recurrence on 2026-09-28. Supported Carlos logs for bah/amadan-hub on annie-welding record process exits with "signal: killed" at 16:14:44 and 20:36:21 UTC, followed by automatic wakes of release 3173d79. The 20:36 restart was followed by 45 repository-server starts in that minute and repeated idle evictions. Across 20:32–20:44 UTC, Git locks were held for tens of seconds (one reached 93s), and browse, pushes, CI claims/heartbeats/reports and the queue census returned 5xx. The instance-usage snapshots read 478 MB and later 483 MB; these snapshots do not establish the memory peak, effective cap, or OOM cause.
We reproduced an Amadan cancellation bug locally: a canceled browse request retains its Git lock and causes the next request to return 503. Work is underway on request-aware Git cancellation, bounded repository starts with active-request leases, and coalesced queue observations. Those app fixes do not resolve the missing platform evidence. Please expose effective memory.high/memory.max and swap limits, memory.events OOM counters, pressure readings with timestamps, the deployed capabilities block, and the recorded process-exit reason through supported CLI/API operations. In particular, we still need to verify whether this child-spawning app is actually receiving the children tier. No SSH or direct cgroup changes were used.
The October 1 recurrence happened on the deployed 67745cd app fix. Around 17:07–17:22 UTC, CI census/status/claims timed out, the homepage returned 502, and Git discovery held a repository lock for over nine minutes. All three AWS CI workers were healthy. Supported instance snapshots reported 473–498 MB, but we still cannot see the effective memory limit or OOM counters.
A supported host-scoped restart began at 17:22:14. The old process exited with signal killed at 17:22:42 during that requested stop (not proof of OOM). Subsequent startups were terminated after the 30-second health check could not connect to amadan.net.sock. The app log stream had no startup explanation. Service is responding again at 22:51 UTC; the exact recovery cause is not established.
Please include startup-stage diagnostics and process-tree cleanup status in the supported operations surface, alongside the memory/pressure/capabilities data already requested. We need to distinguish host pressure, an app stuck before listening, and surviving child processes. Repeatedly restarting without those facts is not a lasting mitigation.
Application safeguards are being tested on https://amadan.net/amadan/amadan/-/branch/fix/overload-mitigation : a total preparation/read budget, one-pack-at-a-time rebuilds that release the SQLite connection before indexing, a limit on simultaneous cold starts, and Linux parent-death handling for repository processes. These reduce known failure paths but do not establish the cause of the platform startup failures.
October 3 recurrence on 9da96f1: the sentinel began timing out on the queue at 13:08:29 UTC while worker desired capacity was zero. At 13:17:15 the app reported a ten-minute pack-objects --all timeout while persisting a push to oficina/native-releases. At 13:24:25 the platform recorded both instance stopping and instance exited: signal killed; at 13:24:26 it woke the same version in 247 ms. This investigation did not request that restart, and the available events do not distinguish an operator-requested stop from resource enforcement. Please expose the stop initiator/reason alongside effective limits, pressure and OOM counters.
CARLOS v0.22 availability now usefully separates failure classes: in the incident window it reported 106 app-generated 502s, 31 visitor-abandoned requests, 13 mid-request disconnects, four app-generated 503s and one app-generated 504. App-generated 500s are separate. Current recovery snapshots were 323 MB and then 289 MB; neither reveals the peak or limit.
Application follow-ups are concrete: https://amadan.net/amadan/amadan/-/discussion/62 removes synchronous whole-repository repacking from durable push acknowledgement; https://amadan.net/amadan/amadan/-/discussion/63 addresses queue polling and 494 repo evictions in a five-minute recovery window. These are triaged, not deployed. Do not infer OOM or increase runners from this evidence. The supported diagnostics requested here are still needed to attribute the cross-repository outage and validate resource isolation.