carlos / platform Public

Changing a box's CARLOS_POOLS leaves orphan routes that keep serving (and can serve stale data) open

Found in the dev cohort-1 claim trial on 2026-10-18 (#250; runbook claim-dev-cohort1-runbook, step 4).

What happened:

  • leo-climbing (carlos-dev-ie-b) got CARLOS_POOLS=cohort1 and its edge was restarted. The edge logged instances loop armed ... pools [cohort1] and published serves_shared_pool: false.
  • wakecheck's existing route (provisioned=1, pool '') stayed in leo's registry through 10 minutes of provision passes. Under the code it never leaves.

Why: internal/provision/reconcile.go checks each row in the removal loop with if !rt.Provisioned || !cfg.inScope(rt.Region, rt.Pool, ...) { continue }. Once the pool change puts a row out of scope, the loop SKIPS it rather than retiring it. Removal only fires for in-scope rows whose record is gone or whose app is in the trash.

Impact:

  • An orphan route keeps answering any request that reaches the box directly.
  • With two boxes for one host, it can serve a stale local database. This is the #286 hazard: a wake on the orphan box restores its old local copy.
  • It also leaves a route the claim's prechecks assume isn't there.

Trial workaround (Paul-approved): a one-off box-local carlos remove --registry /data/registry.db <host> on leo. That removes only the local row, and touches no DB or record.

Fix wanted: when a row falls out of scope because of the box's own scope change (pools or region), retire it with the same evidence a normal removal leaves: the same drain, journal and parks path. Never delete it silently, and never leave it in place. Add a test for a box whose pools change between passes.

Related: #286. Unclaimed.