| index d4973e2d9..ccc34b4c2 100644 |
| --- a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md |
| +++ b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md |
| @@ -1,7 +1,7 @@ |
| # Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint |
| |
| **Date:** 2026-10-06 |
| -**Revision:** 3 |
| +**Revision:** 4 |
| **Discussion:** amadan carlos/platform #241 (quoted in Appendix A) |
| **Branch:** `geoip-for-apps` |
| |
| @@ -26,6 +26,18 @@ |
| no app ever sees it. |
| - It withholds geo on a rejected hop and on a plaintext tunnel. |
| - It adds a bucket lifecycle rule for GeoLite2's destroy duty. |
| +- Rev 4 answers Astra's round 3 (`CODEX-FINDINGS-GEOIP-R3.md`). That round |
| + resolved all 9 of round 2's findings, left 2 earlier findings partly |
| + resolved, and raised 4 new ones, 1 of them P1. Every one was accepted, |
| + and changes are marked "(codex R3-N)". |
| + - Rev 4 also folds in a code survey made for plan 1, marked "(survey)": |
| + - the console's unit isn't in the repo; |
| + - nothing creates `/data/carlos` yet; |
| + - the existing staleness rule; |
| + - the console's 5-minute read timeout; |
| + - which boxes hold a store; |
| + - there is no in-memory store type; |
| + - there is no in-repo page listing the headers apps receive. |
| |
| ## 0. Rulings and decisions |
| |
| @@ -202,6 +214,13 @@ An edge that read the old pointer just before a publish can find its blob |
| already deleted. It keeps what it has loaded and retries on the next poll, |
| when it reads the new pointer. |
| |
| +**The orphan sweep** (codex R3 on R1-4): a console that crashed after |
| +uploading a blob but before writing the pointer leaves a blob that is |
| +still a current version, which the noncurrent-version lifecycle rule |
| +(§15) doesn't touch. So the console runs the §3 sweep, holding the |
| +mutex, at startup and once a day. Deleting the orphan turns it into a |
| +noncurrent version, and the lifecycle rule then removes it. |
| + |
| ## 4. Publishing |
| |
| ### 4.1 The console's publish path |
| @@ -269,6 +288,15 @@ Under `carlos system`, through the console API: |
| instead, so they are not the precedent for this gate. |
| - `carlos system geoip publish <file.mmdb> [-force]` streams the file to |
| the console, which runs §4.1. |
| + - The CLI sends a raw `application/octet-stream` body with a |
| + `Content-Length`, through `PUT /api/cli/system/geoip`. |
| + - The console's server-wide `ReadTimeout` is 5 minutes. So the handler |
| + extends its own read deadline to 15 minutes with |
| + `http.NewResponseController`, the way the ship door does |
| + (`handleAPIShip`, survey), and wraps the body in `MaxBytesReader` at |
| + 1 GiB. |
| + - The existing `PUT /api/cli/system/upload` holds the whole body in |
| + memory, so it isn't reused. |
| - `carlos system geoip status` shows: |
| - the pointer (type, edition, build date, age, origin, expiry for |
| GeoLite2); |
| @@ -285,14 +313,26 @@ Under `carlos system`, through the console API: |
| |
| ### 5.1 Where and who |
| |
| -- The mirror runs inside `carlos agent`'s edge on every box that reads the |
| - deployment bucket directly. |
| +- The mirror runs inside `carlos agent`'s edge when the edge has a store |
| + (`e.staticStore`, set from `CARLOS_DEPLOYMENT_BUCKET` or |
| + `CARLOS_DEPLOYMENT_DIR`) and no tunnel client is configured. Its mode is |
| + `mirror` (survey). |
| - **It does not run on boxes that use the tunnel client** (`CARLOS_TUNNEL_*` |
| - set). Those boxes take the hub's headers (§8). |
| -- Files live under `/data/carlos/geoip/`, which already sits inside the |
| - edge unit's `ReadWritePaths`. The directory is mode 0755 and the files |
| - are 0644, so the console (user `carlos-console`, on the console box) can |
| - read them (§11.3). The data isn't secret. |
| + set). Those boxes take the hub's headers (§8), and their mode is |
| + `tunnel`. |
| +- **A box with no store and no tunnel** has mode `none`. It strips forged |
| + headers, as every edge does, and sets none. |
| +- **Files** live under `/data/carlos/geoip/`, inside the edge unit's |
| + `ReadWritePaths=/data`. |
| + - Nothing creates `/data/carlos` today (survey). The mirror creates the |
| + directory itself at mode 0755, and the files are 0644. The data isn't |
| + secret. |
| + - Bootstrap makes `/data` mode 0750, owned `carlos:carlos`. The console |
| + already writes `/data/registry.db` and `/data/console/console.db`, so |
| + whatever user its hand-installed unit runs as can already get into |
| + `/data` (the unit isn't in the repo, survey). Step 2 of the rollout |
| + (§15) checks on the console box that the console can read |
| + `/data/carlos/geoip/current`. |
| |
| ### 5.2 Local layout (codex R1-5) |
| |
| @@ -323,12 +363,27 @@ old generation, or the new `current` with the new one. |
| logged, and the edge starts with no database. |
| 5. A GeoLite2 generation that has passed its expiry (D1) is deleted, not |
| opened. |
| +- **One lock for selection** (codex R3-1). Choosing the generation, |
| + expiring it and cleaning up the local files all happen under one mutex, |
| + `genMu`. That covers writing `current`, `Swap`, and deleting |
| + generations. Network reads and downloads happen outside it, so a stuck |
| + download never holds it. |
| - **Every 10 minutes,** with up to 1 minute of jitter, read the pointer |
| (bounded read, at most 64 KiB). Each pass has a 10-minute deadline, |
| covering the pointer read and any blob download. A pass that misses it |
| is abandoned and its `.incoming-*` deleted (codex R2 on R1-7). The |
| GeoLite2 expiry check runs on its own one-minute ticker, with no network |
| access, so a stuck download can't delay it. |
| +- **An expired candidate is never installed.** |
| + - A pointer whose `build_epoch` is past D1's expiry (it carries |
| + `database_type` and `build_epoch`) is treated as an expired pointer |
| + before anything is downloaded. |
| + - The check runs again under `genMu`, immediately before installing, |
| + against the downloaded file's own metadata. A download that crossed |
| + its expiry is deleted, not installed. |
| + - The mirror remembers the expired sha and doesn't fetch it again until |
| + the pointer names another one. An expired pointer the console couldn't |
| + mark can therefore never come back. |
| - **The pointer names the sha already loaded:** nothing. |
| - **It names a new sha:** |
| 1. Stream the blob through `OpenObject` into `.incoming-<sha>`. Stop and |
| @@ -345,9 +400,16 @@ old generation, or the new `current` with the new one. |
| once an hour while it keeps failing. |
| - **A blob whose digest doesn't match** is deleted, logged once per sha, |
| and not retried until the pointer changes. |
| -- **GeoLite2 expiry (D1):** the expiry ticker checks the loaded |
| - generation's `build_epoch`. Past expiry it swaps in nil and deletes the |
| - generation, whatever the pointer says. |
| +- **GeoLite2 expiry (D1):** the expiry ticker takes `genMu`, then checks |
| + the **currently loaded** generation's `build_epoch`. Past expiry it swaps |
| + in nil and deletes that generation, whatever the pointer says. |
| + - Because it checks after taking the lock, it acts only on the |
| + generation that is current at that moment. It can never inspect an |
| + expired A and then remove a freshly installed B. |
| + - Tests: |
| + - a download crossing expiry; |
| + - repeated polls against an expired pointer that was never marked; |
| + - an install racing the ticker. |
| |
| ### 5.4 Hot swap |
| |
| @@ -366,8 +428,9 @@ type Snapshot struct { |
| Source string // empty only when Loaded is false |
| Loaded bool |
| } |
| -func (s *Swappable) Lookup(ip net.IP) Snapshot // one RLock covers lookup and source |
| -func (s *Swappable) Swap(next *MMDB) // Lock, replace, Unlock, then Close the old one |
| +func (s *Swappable) Lookup(ip net.IP) Snapshot // one RLock covers lookup and source |
| +func (s *Swappable) Locate(ip net.IP) (Point, bool) // geo.Locator; RLock held through MMDB.Locate; a miss when empty |
| +func (s *Swappable) Swap(next *MMDB) // Lock, replace, Unlock, then Close the old one |
| ``` |
| |
| The headers, analytics and the endpoint's attribution all come from one |
| @@ -375,6 +438,12 @@ The headers, analytics and the endpoint's attribution all come from one |
| DB-IP data as GeoLite2. There is deliberately no separate `Source()` on |
| `Swappable`. |
| |
| +`Swappable` implements `geo.Locator`, so the console's `Config.GeoIP` holds |
| +the `Swappable` itself (codex R3-2). Region guesses |
| +(`internal/console/regionguess.go`) never see the `*MMDB` underneath and |
| +can't outlive a swap. The `-race` test runs concurrent `Locate` calls as |
| +well as `Lookup` calls. |
| + |
| - The old mapping is closed only after the write lock has been taken and |
| released. No lookup can be in flight on an unmapped file. |
| - A lookup holds the read lock for a few microseconds, so a swap waits that |
| @@ -385,29 +454,39 @@ DB-IP data as GeoLite2. There is deliberately no separate `Source()` on |
| |
| ### 5.5 Reporting (codex R1-16) |
| |
| -Each box's status heartbeat (`status/boxes/<label>.json`) gains a block on |
| -**every** box running a binary with this feature: |
| +Each box's status heartbeat gains a block on **every** box running a |
| +binary with this feature: a new `GeoIP` field in `status.BoxMetrics` |
| +(`internal/status/snapshot.go`), which `BuildFleet` never copies into the |
| +public `fleet.json`. Platform boxes write `status/boxes/<label>.json`, and |
| +fleet boxes write under `fleets/<fleet>/status/boxes/`: |
| |
| ```json |
| "geoip": {"mode": "mirror", "sha256": "9f2c…", "type": "DBIP-City-Lite", "error": ""} |
| "geoip": {"mode": "tunnel"} |
| +"geoip": {"mode": "none"} |
| ``` |
| |
| -`carlos system geoip status` lists every box record under |
| -`status/boxes/` written in the last 7 days. It reports: |
| +`carlos system geoip status` lists boxes from the same inventory |
| +`carlos system status` uses: the console's `systemBoxes`, which includes |
| +the fleet prefixes (codex R3 on R1-16). Boxes whose last heartbeat is more |
| +than 7 days old are left out as retired. It reports: |
| |
| -- **heartbeat older than 5 minutes:** "stale". The verdict can't vouch for |
| - a box it can't currently see. |
| +- **stale,** by the shared rule `status.BoxStale` (3 heartbeat periods, |
| + 15 minutes), not a threshold of its own (codex R3-4). The verdict can't |
| + vouch for a box it can't currently see. |
| - **record unreadable:** "unreadable". |
| - **no block:** "old binary". The headers can't be trusted while any such |
| box is serving. |
| - **`mode: tunnel`:** takes the hub's headers. |
| +- **`mode: none`:** no store and no tunnel, so no headers. Safe, because |
| + it strips forged ones. |
| - **`mode: mirror`:** shows the box's sha against the pointer's. |
| |
| It ends with a one-line verdict. **"Ready"** means every listed box is |
| fresh, readable and GeoIP-aware, and every mirroring box has loaded the |
| current sha. Anything else names the boxes in the way (codex R2 on R1-16). |
| -A box retired more than 7 days ago drops out of the list. |
| +A test sets a box's heartbeat just inside the next jittered tick and |
| +checks it still reads as fresh. |
| |
| ## 6. `internal/geo` changes |
| |
| @@ -512,9 +591,14 @@ X-Carlos-Hop-Geo: off |
| under the same trust as the app headers. |
| - `setGeo` deletes it from every outbound request to an app, so it never |
| reaches app code. |
| -- An adopted `hit` without a valid country, or a missing or malformed |
| - `X-Carlos-Hop-Geo` on a trusted transport, is treated as `off`. Fewer |
| - countries are counted, but none is made up. |
| +- **An adopted `hit` needs a valid country or a valid coordinate pair** |
| + (codex R3-3), matching `Lookup`'s own success rule (§6). A `hit` with |
| + neither is treated as `off`, and so is a missing or malformed |
| + `X-Carlos-Hop-Geo` on a trusted transport. Fewer countries are counted, |
| + but none is made up. |
| +- A test checks that identical app headers and analytics come out of a |
| + local route, a backhaul and the tunnel, including for a coordinate-only |
| + result. |
| |
| ### 8.2 Computing: only at ingress |
| |
| @@ -617,7 +701,8 @@ show that, and only then publishes a database and the app docs. |
| `observeAnalytics` reads from the context's `geoResult`. It never looks |
| up again. |
| - **By state:** |
| - - `hit` counts the country. |
| + - `hit` counts the country, or `"_unknown"` for a coordinate-only result |
| + (codex R3-3). |
| - `miss` counts `"_unknown"`. |
| - `off` (no database, a door call, a rejected hop, a plaintext tunnel) |
| isn't counted in `Countries` at all, so "no GeoIP" never looks like |
| @@ -859,14 +944,20 @@ MaxMind key**. |
| - **The operator owns compliance.** The operator docs say that plainly, |
| next to the publish verb and the feed settings, and link both providers' |
| terms. |
| -- **The app docs:** |
| - - list the headers and explain that they may be absent and why; |
| - - cover `Vary`, with the cache cost (§7); |
| - - say that coordinates are an approximate area; |
| - - spell out the duty: an app that shows visitors anything derived from |
| - these headers or the endpoint must carry the attribution that |
| - `X-Carlos-Geo-Source` names. The endpoint returns the attribution |
| - text, so an app can render it. |
| +- **The app docs** go in `docs/site/cli.md`, the public reference, as a |
| + new section, "Headers your app receives". No page lists them today |
| + (survey). |
| + - App skills are served from carlosframework.com, outside this repo. Plan |
| + 1 files an amadan discussion asking for the skill text to pick up the |
| + section. |
| + - The section must: |
| + - list the headers and explain that they may be absent and why; |
| + - cover `Vary`, with the cache cost (§7); |
| + - say that coordinates are an approximate area; |
| + - spell out the duty: an app that shows visitors anything derived from |
| + these headers or the endpoint must carry the attribution that |
| + `X-Carlos-Geo-Source` names. The endpoint returns the attribution |
| + text, so an app can render it. |
| - **The platform's own pages** that use the data carry the attribution: |
| the analytics Countries table, per §9, and the new-app region guess if |
| it ever shows the visitor where it thinks they are. |
| @@ -883,7 +974,7 @@ MaxMind key**. |
| - `OpenMMDBTrusted`. |
| - **Swap:** `-race`, with 32 goroutines looking up while 100 swaps happen, |
| and a `Close` spy proving no close happens while a read lock is held. |
| -- **Mirror,** against the in-memory store: |
| +- **Mirror,** against `release.NewDirStore(t.TempDir())`. There is no in-memory store type (survey): |
| - first publish, a new sha, the pointer removed, the pointer |
| unreadable; |
| - a digest mismatch, an oversize blob, a disk write failure; |
| @@ -1025,6 +1116,17 @@ Round 2 (`CODEX-FINDINGS-GEOIP-R2.md`): every finding accepted. |
| | 8 | P3 | Accepted. Immediate clearing, stated (§10.4) | |
| | 9 | P3 | Accepted. The off bound is one edge poll plus one console poll (§4.2) | |
| |
| +Round 3 (`CODEX-FINDINGS-GEOIP-R3.md`): every finding accepted. |
| + |
| +| # | Sev | Disposition | |
| +|---|---|---| |
| +| R1-4 (partly) | | The orphan sweep runs at console startup and daily (§3) | |
| +| R1-16 (partly) | | The `systemBoxes` inventory, including fleet prefixes (§5.5) | |
| +| 1 | P1 | Accepted. `genMu` serializes selection, expiry and cleanup; expiry is checked before download and before install; expired shas are remembered (§5.3) | |
| +| 2 | P2 | Accepted. `Swappable.Locate`, and `Config.GeoIP` holds the `Swappable` (§5.4) | |
| +| 3 | P2 | Accepted. A coordinate-only `hit` is adopted, and counted as `_unknown` (§8.1a, §9) | |
| +| 4 | P2 | Accepted. `status.BoxStale` (§5.5) | |
| + |
| ## Appendix A. The request (#241, verbatim) |
| |
| > **Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint)** |
amadan