carlos / platform Public

Migrated from github.com/carlosframework/platform issue #236, opened 2026-08-19. GitHub issues are closed for this repo; see AGENTS.md, "Work is tracked on amadan, not GitHub".

Still outstanding: An exec app whose socket path is deleted stays "healthy" and unreachable, with no detection or self-heal.

---

What happened

2026-08-19, flagship, amadan.net. A stale systemd-run transient unit (amadan-stopgap.service, from an 2026-08-08 deploy) was running an old build of the same app, bound to the same unix socket path as the live exec child.

Stopping the stale unit took amadan.net down — not immediately, which is what made it nasty:

| time | event | |---|---| | 21:39:15 | stale unit stopped; its Go net.UnixListener unlinked /run/carlos/amadan.net.sock on close — the path the live hub had since re-created | | 21:39–21:47 | site still served 200s on the edge's already-pooled connections | | 21:47:07 | first http: proxy error: dial unix /run/carlos/amadan.net.sock: connect: no such file or directory | | 21:54 | recovered by carlos restart -app amadan-hub |

Go's UnixListener unlinks the socket path on Close(). Two processes can hold the same path: the second bind() replaces the directory entry, and the first one's close then deletes the current entry — the one belonging to the live process.

The platform gap

Throughout, carlos considered the app healthy. The exec child was alive, the supervisor was content, and nothing anywhere said the app had become unreachable:

1. No detection. The agent never checks that its exec child's socket path still exists and still resolves to that child. ss -lxp kept listing the live listener by inode, so even a manual check looks fine — the path is only a directory entry, and it was gone.
2. No self-heal. A missing socket path is trivially recoverable (restart the child, which rebinds), but nothing did.
3. No alarm. Seven minutes of dial unix ... no such file or directory in the agent's own log produced no alert. The proxy error is logged at the point where it is already user-visible.

Suggested fix

• Agent-side liveness for exec tenants should stat the socket path (and ideally confirm it is the same socket the child is bound to), not just check the process is alive. Recreate/restart on mismatch.
• Consider refusing to start an exec child when the socket path is already bound by a process that is not this supervisor's child — that is the condition that makes the shared-path deletion possible at all.
• The repeated dial unix ...: no such file or directory is a strong, unambiguous signal and deserves to be surfaced rather than only logged.

Verification trap worth recording

Checking immediately after killing a process that shares a socket path passes on pooled connections while the path is already gone. Three consecutive curl 200s at 21:40 were a false negative. Verify with a fresh connection, or by stat-ing the socket path.

Related

Found while recovering the amadan wedge (amadannet/amadan#108 and follow-up). The stale unit itself was an orphan from the 2026-08-08 deploy, which is its own smell: a hand-rolled systemd-run stopgap outlived the deploy that created it by 11 days, running an 11-day-old binary with 10 stale child processes.