carlos / platform Public

What happened

On 2026-09-17 a Tito organizer switching accounts on go.tito.io got a blank white page. carlos-edge logged, three times:

http: proxy error: dial tcp 127.0.0.1:8444: connect: connection refused

The upstream (titogo's router) was restarting during a deploy; controld restarts it on every fleet retarget and it took ~4s to bind its port again. The edge answered each request with a bare 502 and no body, which Safari renders as an empty page. Nothing told the person to try again, and nothing in the page said what had happened.

Two asks

1. An unreachable upstream should render an error page, not an empty 502.

Go's httputil.ReverseProxy with no ErrorHandler writes a 502 with an empty body. titogo's own router has the opposite behaviour in the same situation (proxyUnreachable in internal/carlos/router.go renders a page with art, a plain sentence and a meta-refresh at the document's own URL), because a person staring at a blank tab has no idea whether to wait or to call support. The edge is the outermost hop and the one most likely to be the only thing between a person and a blank screen.

2. A refused dial should be retried briefly before it becomes an error at all.

titogo dials its instance sockets through dialUnixWithRetry, which masks exactly this gap: a restart is a few seconds, and a request that waits is better than a request that fails. The edge needs the same for its upstreams, and its dial and response timeouts have to allow the wait — we measure ~4s for a router restart and ~80s for an instance roll.

What we did on our side, and why it is not enough

titogo #2760 (merged 2026-09-18) adds systemd socket activation to its router, so systemd holds the listening socket and connections queue in the backlog across a restart instead of being refused. That closes the refused-dial case for this one upstream, once a socket unit is installed on the box.

It does not close two others:

• Any upstream without a socket unit still refuses during a restart, and the edge still answers blank.
• A keep-alive connection the edge holds to an upstream that then shuts down gives an EOF or reset rather than a refusal. Whether the edge retries an idempotent request on that connection is the same question as above, and we cannot see the answer from outside.

Reported by

Tito (titogo), paul@tito.io. Happy to test a build on carlos-tito-ie — the failure is reproducible there with systemctl restart titogo-router and a request timed into the window.