| index 78091617f..57dadd4fe 100644 |
| --- a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md |
| +++ b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md |
| @@ -1,14 +1,24 @@ |
| # Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint |
| |
| **Date:** 2026-10-06 |
| -**Revision:** 1 |
| -**Discussion:** amadan carlos/platform #241 |
| +**Revision:** 2 |
| +**Discussion:** amadan carlos/platform #241 (quoted in Appendix A) |
| **Branch:** `geoip-for-apps` |
| |
| -Rev 1 records the design Paul approved in conversation on 2026-10-06. It has |
| -not been reviewed by a second model yet. |
| - |
| -## 0. Rulings |
| +- Rev 1 (5e1129be) records the design Paul approved in conversation on |
| + 2026-10-06. |
| +- Rev 1 went to Astra (GPT-6, via codex). The findings are in |
| + `CODEX-FINDINGS-GEOIP-R1.md`: 22 findings, 4 of them P1. Every one was |
| + checked against the code and accepted. Changes are marked |
| + "(codex R1-N)", and §18 records each disposition. |
| +- **Rev 2 changes the trust model (codex R1-3, R1-10).** The edge that |
| + accepts the visitor's connection computes the headers. Downstream edges |
| + trust them only over a verified hop or the tunnel. Rev 1 had the home edge |
| + recompute, which located the sending box for a backhauled door request. |
| +- Rev 2 makes one decision Paul has not seen, **D1** in §0: GeoLite2 |
| + databases expire 30 days after they were built. |
| + |
| +## 0. Rulings and decisions |
| |
| Paul's answers during brainstorming. Do not reopen them without him. |
| |
| @@ -20,7 +30,7 @@ Paul's answers during brainstorming. Do not reopen them without him. |
| - **R2. Four headers:** country, subdivision, coordinates and source (§7). |
| - **R3. Tunnel and fleet boxes trust the hub.** The hub computes the |
| headers. The remote edge keeps them only on requests that arrived over |
| - the authenticated tunnel (§8.3). Remote boxes hold no database. |
| + the authenticated tunnel (§8). Remote boxes hold no database. |
| - **R4. No per-app switch.** Headers go to every app whenever a database |
| is loaded. A switch can ride the telemetry settings pipeline later if |
| someone asks for one. |
| @@ -30,13 +40,34 @@ Paul's answers during brainstorming. Do not reopen them without him. |
| DB-IP Lite or GeoLite2 and publishes them through the same path as a |
| manual publish (§10). |
| |
| +Decisions taken during review. Paul hasn't seen these yet; each is easy to |
| +reverse. |
| + |
| +- **D1 (codex R1-4). GeoLite2 databases expire 30 days after their |
| + `build_epoch`.** |
| + - **Why:** MaxMind's GeoLite EULA requires the licensee to stop using a |
| + superseded edition within 30 days of an update's release. MaxMind |
| + updates GeoLite2 twice a week. A database 30 days past its build has |
| + certainly been superseded, and counting from the build rather than |
| + from the release of its successor errs on the strict side. |
| + - **What happens:** |
| + - At expiry every edge and the console drop the reader and delete |
| + their local copies. |
| + - The console deletes the pointer and the blob. |
| + - The box alert (§10.4) warns from day 21. |
| + - **When it matters:** with the feed working, a GeoLite2 database never |
| + gets near 30 days. Only a dead feed or a manual publish ever reaches |
| + the limit. |
| + - **DB-IP Lite has no such duty** (CC BY 4.0) and never expires. It |
| + only alerts as stale. |
| + |
| ## 1. Why, and what this is not |
| |
| The nearest-region work gave the console a GeoIP reader (`internal/geo`, |
| `CONSOLE_GEOIP_DB`). It serves one caller: guessing a new app's region. |
| Once the platform carries the database, every app should be able to ask |
| "roughly where is this visitor" without shipping its own `.mmdb`, handling |
| -attribution, or tracking updates. |
| +attribution, or tracking updates. Appendix A quotes the request. |
| |
| Goals: |
| |
| @@ -54,7 +85,8 @@ Not goals: |
| - **Header injection for traffic that never crosses the edge.** Scheduler |
| calls go straight to the socket. Sidecars reach their instance through |
| loopback. |
| -- **ASN, ISP, timezone or city-name headers.** Additive later if asked for. |
| +- **ASN, ISP, timezone, city-name or accuracy-radius headers.** Additive |
| + later if asked for. |
| - **A second reader.** Everything builds on `internal/geo`. |
| |
| ## 2. Overview |
| @@ -62,20 +94,22 @@ Not goals: |
| ``` |
| operator ──carlos system geoip publish──┐ |
| ▼ |
| -feed (console, daily) ──────────► console publish path |
| +feed (console, daily) ──────────► console publish path (one at a time) |
| │ verify, upload blob, flip pointer |
| ▼ |
| bucket: control/geoip/current.json → control/geoip/<sha256>.mmdb |
| │ |
| - every edge polls the pointer (10 min) |
| + every mirroring edge polls the pointer (10 min) |
| ▼ |
| - /data/carlos/geoip/current.mmdb → geo.Swappable (hot swap) |
| + /data/carlos/geoip/<sha256>.mmdb + current → geo.Swappable (hot swap) |
| │ |
| - ServeHTTP: strip X-Carlos-Geo-*, look up clientIP(r), set headers |
| - │ │ |
| - ▼ ▼ |
| - proxied request to the app analytics Obs.Country |
| - → Day.Countries |
| + ingress edge: strip X-Carlos-Geo-*, look up the visitor, keep the result |
| + on the request; downstream edges (verified hop, tunnel) |
| + adopt the validated headers instead of looking up |
| + │ │ |
| + ▼ ▼ |
| + every proxy's Rewrite sets the headers analytics Obs.Country |
| + on the outbound request → Day.Countries |
| console: reads the same mirrored file → GET /api/cli/apps/{a}/{app}/geo?ip= |
| carlos geo <ip> |
| ``` |
| @@ -83,6 +117,9 @@ console: reads the same mirrored file → GET /api/cli/apps/{a}/{app}/geo?ip= |
| ## 3. Bucket records |
| |
| The console is the **only writer** of `control/geoip/`. Edges only read it. |
| +A deployment runs one console, and inside it a single mutex serializes |
| +every publish, `off` and expiry through to the end of its cleanup |
| +(codex R1-2). |
| |
| **The blob** is `control/geoip/<sha256>.mmdb`. Its name is the hex SHA-256 |
| of its bytes, so a blob never changes after it is written. |
| @@ -97,63 +134,118 @@ of its bytes, so a blob never changes after it is written. |
| "size": 131072000, |
| "database_type": "DBIP-City-Lite", |
| "build_epoch": 1790812800, |
| + "node_count": 7342011, |
| "published": "2026-10-06T21:00:00Z", |
| "origin": "feed:dbip-city-lite", |
| "edition": "2026-10" |
| } |
| ``` |
| |
| -- `database_type` and `build_epoch` come from the file's own MMDB metadata. |
| +- `database_type`, `build_epoch` and `node_count` come from the file's own |
| + MMDB metadata. `node_count` is recorded so the shrink check (§4.1) can |
| + compare against the current pointer without reading the current blob |
| + (codex R1-14). |
| - `origin` is `manual` or `feed:<name>`. `edition` is the feed's own edition |
| marker: the `YYYY-MM` for DB-IP, the `Last-Modified` value for MaxMind, |
| and empty for a manual publish. |
| - **No pointer means the feature is off.** An unknown `v` is treated as no |
| pointer, logged once. |
| |
| -**Publish order:** the blob first, then the pointer, written with `ifMatch` |
| -on the pointer's previous etag so two publishers can't interleave. The |
| -previous blob is deleted only after the new pointer is written. An edge |
| -that read the old pointer moments before can find its blob gone. It |
| -retries on the next poll and reads the new pointer then. |
| +**Publishing,** holding the mutex: |
| + |
| +1. Read the pointer and its etag. Every comparison in §4.1 and §10.3 is |
| + against this read. |
| +2. Upload the blob, create-only. If a blob with that name already exists |
| + and its size matches, it is the same bytes, so carry on. Publishing the |
| + same file twice is not an error. |
| +3. Write the pointer with `ifMatch` set to the etag from step 1. A refusal |
| + means something outside this console wrote the pointer. Stop, and leave |
| + the new blob for the sweep. |
| +4. **Sweep:** delete every `control/geoip/*.mmdb` other than the one the new |
| + pointer names. This removes the previous blob, the leftovers of any |
| + earlier failure, and never the current one, even in an X to Y to X |
| + sequence. |
| + |
| +`off` and expiry take the same mutex. They delete the pointer (with |
| +`ifMatch`), then sweep every blob. |
| + |
| +An edge that read the old pointer just before a publish can find its blob |
| +already deleted. It keeps what it has loaded and retries on the next poll, |
| +when it reads the new pointer. |
| |
| ## 4. Publishing |
| |
| ### 4.1 The console's publish path |
| |
| -`geoipPublish(ctx, src io.Reader, origin, edition string)`, used by both |
| -the CLI verb and the feed: |
| - |
| -1. Stream `src` to a temporary file under the console's data directory, |
| - hashing as it goes. Refuse anything over 1 GiB. |
| -2. Open it with `geo.OpenMMDB`, which verifies it. |
| -3. Refuse a database with no country data. Probe: the reader's metadata |
| - lists a `country` field in its first record, or a lookup of a fixed set |
| - of well-known public addresses yields a country for at least one of |
| - them. The plan settles which check, against the real files. |
| -4. Refuse a database whose search tree has fewer than half the nodes of |
| - the current one (`metadata.node_count`). This guards against a truncated |
| - or wrong-product file. A manual publish can override it with `-force`. |
| - The feed can't. |
| -5. Upload the blob, write the pointer, delete the old blob (§3). |
| -6. Delete the temporary file. |
| +`geoipPublish(ctx, src io.Reader, origin, edition string, force bool)`, used |
| +by both the CLI verb and the feed: |
| + |
| +1. Refuse with 409 if another publish holds the mutex. Don't queue. |
| +2. Stream `src` to a fresh temporary directory under the console's data |
| + directory, hashing as it goes. |
| + - Refuse once 1 GiB has been read. The limit is enforced during the |
| + copy, not afterwards (codex R1-7). |
| + - The whole operation has a 15-minute deadline. |
| + - The temporary directory is removed on every exit path. |
| +3. Open it with `geo.OpenMMDB`, which verifies it. |
| +4. **Country probe** (codex R1-21): look up `8.8.8.8`, `1.1.1.1`, `9.9.9.9`, |
| + `208.67.222.222` and `2001:4860:4860::8888`. Refuse the database unless |
| + at least three of them return a country. That catches an ASN database or |
| + a schema the decoder can't read. |
| +5. **Shrink check:** refuse a database with fewer than half the pointer's |
| + `node_count`. This guards against a truncated or wrong-product file. A |
| + manual publish can override it with `force`. The feed can't. |
| +6. Publish (§3). |
| + |
| +Plan 1, task 1 measures `Verify`'s peak memory on a real DB-IP City Lite |
| +file and a real GeoLite2 City file, and records the result in the plan. |
| +`Verify` builds a map of data offsets, so it can cost much more than the |
| +file's size. If the peak would push the console box past its memory |
| +headroom, the plan says so and proposes a limit (codex R1-7). |
| + |
| +**Streaming upload** (codex R1-8). `Store.Put` takes the whole body as |
| +`[]byte`. Plan 1 adds one optional interface: |
| + |
| +```go |
| +// release.PutStreamer is implemented by stores that can upload from a reader. |
| +type PutStreamer interface { |
| + PutFrom(ctx context.Context, key string, body io.Reader, size int64, ifMatch string) (etag string, err error) |
| +} |
| +``` |
| + |
| +- It is implemented by `S3Store` (PutObject with a reader body and a |
| + content length), `DirStore` (temp file and rename) and `prefixStore` |
| + (rebase, then forward). |
| +- Plan 1 lists every other wrapper around the console's store and makes |
| + each one forward `PutFrom` the way it forwards `Opener`. |
| +- A store without `PutFrom` takes the `Put` fallback only for files up to |
| + 64 MiB. Anything larger is refused with a clear error. |
| + |
| +**Streaming reads** use the existing `release.Opener` / `OpenObject`, which |
| +S3, `DirStore` and `prefixStore` already implement. No new read interface is |
| +added. |
| |
| ### 4.2 Operator verbs |
| |
| -Under `carlos system`, through the console API like `carlos system |
| -antivirus`, for system operators only (`CARLOS_SYSTEM_OPERATORS`): |
| +Under `carlos system`, through the console API: |
| |
| -- `carlos system geoip publish <file.mmdb> [-force]` uploads the file to |
| - the console, which runs §4.1. The body is streamed, not buffered. |
| +- **Who may use them** (codex R1-13): a person, not a token, who |
| + `apiSystemRoller` (`internal/console/apisystemupdate.go`) admits. That is |
| + an owner of the ops account, or an address on `CARLOS_SYSTEM_OPERATORS`. |
| + These are the people who can already roll the platform. The antivirus |
| + doors use `apiSystemOperator`, which checks `CARLOS_STORE_OPERATORS` |
| + instead, so they are not the precedent for this gate. |
| +- `carlos system geoip publish <file.mmdb> [-force]` streams the file to |
| + the console, which runs §4.1. |
| - `carlos system geoip status` shows: |
| - - the pointer (type, edition, build date, age, origin); |
| + - the pointer (type, edition, build date, age, origin, expiry for |
| + GeoLite2); |
| - the feed's configuration and its last result; |
| - - the sha each box reports having loaded (§5.4), so "has every edge |
| - picked it up" is one command. |
| -- `carlos system geoip off` deletes the pointer, then the blob. Every edge |
| - drops its reader within one poll. The feed (§10) does not republish |
| - while the pointer is absent and the feed is unset. If the feed is set, |
| - `off` warns that the feed will publish again tomorrow and says to unset |
| - `CONSOLE_GEOIP_FEED`. |
| + - the per-box report from §5.4. |
| +- `carlos system geoip off` deletes the pointer, then sweeps the blobs |
| + (§3). Every edge drops its reader within one poll, and the console within |
| + a minute (§11.3). If `CONSOLE_GEOIP_FEED` is set, `off` warns that the |
| + feed will publish again within a day and says to unset it. |
| |
| ## 5. The edge mirror |
| |
| @@ -162,46 +254,64 @@ antivirus`, for system operators only (`CARLOS_SYSTEM_OPERATORS`): |
| - The mirror runs inside `carlos agent`'s edge on every box that reads the |
| deployment bucket directly. |
| - **It does not run on boxes that use the tunnel client** (`CARLOS_TUNNEL_*` |
| - set). Those get their headers from the hub (§8.3). |
| + set). Those boxes take the hub's headers (§8). |
| - Files live under `/data/carlos/geoip/`, which already sits inside the |
| - edge unit's `ReadWritePaths`. The directory is mode 0755 and |
| - `current.mmdb` is 0644, so the console (user `carlos-console`, on the |
| - console box) can read the same file (§11.3). The data isn't secret. |
| - |
| -### 5.2 The loop |
| - |
| -- **At boot:** if `current.mmdb` exists, open it straight away. Serving |
| - never waits on the bucket. The sha it was published under is kept in |
| - `current.sha256` next to it. |
| -- **Every 10 minutes**, with jitter, read the pointer: |
| - - **Same sha as loaded:** nothing. |
| - - **New sha:** |
| - 1. Stream the blob to `.incoming` and check its sha and size against |
| - the pointer. |
| - 2. Open it with `geo.OpenMMDB`, off the request path. |
| - 3. Rename it to `current.mmdb`, write `current.sha256`, and swap the |
| - reader (§5.3). |
| - - **No pointer:** swap in a nil reader, then delete the local files. |
| - - **The read fails** (network, permissions): keep what is loaded. Log |
| - once per hour while it keeps failing. |
| -- **A blob that fails its sha or verification** is logged once per sha and |
| - not retried until the pointer changes. |
| - |
| -The current `release.Store.Get` returns the whole body as `[]byte`. For a |
| -DB-IP City file (about 130 MB) that's too much to hold on a small box, so |
| -plan 1 adds an optional streaming read: |
| + edge unit's `ReadWritePaths`. The directory is mode 0755 and the files |
| + are 0644, so the console (user `carlos-console`, on the console box) can |
| + read them (§11.3). The data isn't secret. |
| |
| -```go |
| -// release.StreamGetter is implemented by stores that can stream a body. |
| -type StreamGetter interface { |
| - GetTo(ctx context.Context, key string, w io.Writer) (etag string, err error) |
| -} |
| -``` |
| +### 5.2 Local layout (codex R1-5) |
| + |
| +Each generation is immutable and named by its digest: |
| |
| -The S3 store and the directory store implement it. The mirror uses it when |
| -it's there and falls back to `Get` when it isn't. |
| +``` |
| +/data/carlos/geoip/<sha256>.mmdb one per generation, written once |
| +/data/carlos/geoip/current {"sha256": "...", "database_type": "...", "build_epoch": ...} |
| +/data/carlos/geoip/.incoming-* downloads in progress |
| +``` |
| |
| -### 5.3 Hot swap |
| +`current` is the only file that changes. It is written to a temporary file, |
| +fsynced, renamed over `current`, and then the directory is fsynced. A |
| +reader that sees `current` naming a sha always finds that generation |
| +complete, because the generation was renamed into place before `current` |
| +was written. A crash at any step leaves either the old `current` with the |
| +old generation, or the new `current` with the new one. |
| + |
| +### 5.3 The loop |
| + |
| +- **At boot:** |
| + 1. Read `current` and open `<sha>.mmdb` with `geo.OpenMMDBTrusted`. That |
| + opens the file without `Verify` (§6), because the console verified |
| + those exact bytes and the name proves they are the same. |
| + 2. Delete any other `*.mmdb` and any `.incoming-*`. |
| + 3. Serving never waits on the bucket. |
| + 4. A `current` that names a missing or unopenable file is removed and |
| + logged, and the edge starts with no database. |
| + 5. A GeoLite2 generation that has passed its expiry (D1) is deleted, not |
| + opened. |
| +- **Every 10 minutes,** with jitter, read the pointer (bounded read, at most |
| + 64 KiB): |
| + - **The pointer names the sha already loaded:** nothing. |
| + - **It names a new sha:** |
| + 1. Stream the blob through `OpenObject` into `.incoming-<sha>`. Stop and |
| + delete it if it exceeds the pointer's `size` or 1 GiB, or if the disk |
| + write fails. |
| + 2. Check the digest. |
| + 3. Fsync and rename it to `<sha>.mmdb`. |
| + 4. Write `current` (§5.2), open the generation and swap the reader |
| + (§5.4). |
| + 5. Delete the previous generation. |
| + - **No pointer:** write the empty state to `current`, swap in a nil |
| + reader, then delete the generations. |
| + - **The read fails** (network, permissions): keep what is loaded, and log |
| + once an hour while it keeps failing. |
| +- **A blob whose digest doesn't match** is deleted, logged once per sha, |
| + and not retried until the pointer changes. |
| +- **GeoLite2 expiry (D1):** the loop also checks the loaded generation's |
| + `build_epoch`. Past expiry it swaps in nil and deletes the generation, |
| + whatever the pointer says. |
| + |
| +### 5.4 Hot swap |
| |
| `geo.Swappable` wraps the reader: |
| |
| @@ -211,29 +321,44 @@ type Swappable struct { |
| cur *MMDB |
| } |
| func (s *Swappable) Lookup(ip net.IP) (Result, bool) // RLock for the lookup |
| +func (s *Swappable) Source() string |
| func (s *Swappable) Swap(next *MMDB) // Lock, replace, Unlock, then Close the old one |
| ``` |
| |
| -The old mapping is closed only after the write lock has been taken and |
| -released. No lookup is ever in flight on an unmapped file. A lookup holds |
| -the read lock for a few microseconds, so a swap waits that long at most. |
| -The swap needs a `-race` test with concurrent lookups (§14). |
| +- The old mapping is closed only after the write lock has been taken and |
| + released. No lookup can be in flight on an unmapped file. |
| +- A lookup holds the read lock for a few microseconds, so a swap waits that |
| + long at most. |
| +- Go's `RWMutex` blocks new readers while a writer waits, so a steady |
| + stream of lookups can't starve the swap. |
| +- The swap gets a `-race` test with concurrent lookups (§14). |
| |
| -### 5.4 Reporting |
| +### 5.5 Reporting (codex R1-16) |
| |
| -Each box's status heartbeat (`status/boxes/<label>.json`) gains: |
| +Each box's status heartbeat (`status/boxes/<label>.json`) gains a block on |
| +**every** box running a binary with this feature: |
| |
| ```json |
| -"geoip": {"sha256": "9f2c…", "type": "DBIP-City-Lite", "error": ""} |
| +"geoip": {"mode": "mirror", "sha256": "9f2c…", "type": "DBIP-City-Lite", "error": ""} |
| +"geoip": {"mode": "tunnel"} |
| ``` |
| |
| -- The block is omitted entirely on tunnel boxes. |
| -- `carlos system geoip status` reads it (§4.2). |
| +`carlos system geoip status` lists every box with a fresh heartbeat (newer |
| +than 5 minutes) from the existing fleet listing. It reports: |
| + |
| +- **no block:** "old binary". The headers can't be trusted while any such |
| + box is serving. |
| +- **`mode: tunnel`:** takes the hub's headers. |
| +- **`mode: mirror`:** shows the box's sha against the pointer's. |
| + |
| +It ends with a one-line verdict: whether every fresh box runs a |
| +GeoIP-aware binary, and whether every mirroring box has loaded the current |
| +sha. |
| |
| ## 6. `internal/geo` changes |
| |
| -`Locate` stays as it is for the console's region guess. Lookups that need |
| -more get a new method: |
| +`Locate` keeps its exact behaviour (codex R1-9). Both methods share one |
| +decode and validation step: |
| |
| ```go |
| type Result struct { |
| @@ -243,21 +368,27 @@ type Result struct { |
| HasPoint bool // true only for a real location record, never a centroid |
| } |
| func (m *MMDB) Lookup(ip net.IP) (Result, bool) |
| -func (m *MMDB) Source() string // metadata.DatabaseType, reduced to [A-Za-z0-9._-] |
| +func (m *MMDB) Source() string // metadata.DatabaseType, reduced to [A-Za-z0-9._-], at most 64 bytes |
| +func (m *MMDB) BuildEpoch() uint |
| +func (m *MMDB) NodeCount() uint |
| +func OpenMMDBTrusted(path string) (*MMDB, error) // opens without Verify |
| ``` |
| |
| -- `Lookup` applies the same `Locatable` filter, `To4` normalisation, |
| - `onTheGlobe` check and fixed-category error log as `Locate`. |
| -- A malformed location makes the whole record a miss, as it does for |
| - `Locate`. A record wrong in one field isn't trusted on the next. |
| -- `ok` is false when no country can be found. |
| -- `Subdivision` is `country + "-" + subdivisions[0].iso_code`, only when |
| - both are present and the code is 1 to 3 `[A-Z0-9]` characters. |
| -- **DB-IP Lite has subdivision names but no ISO codes**, so with DB-IP the |
| - subdivision is always empty. With GeoLite2 City it is usually filled. |
| -- `Locate` is reimplemented on top of `Lookup` so the two can't drift. A |
| - record with no point still falls back to the country's representative |
| - point for `Locate`, as today. |
| +- **Shared:** `decode(ip) (mmdbRecord, bool)` applies the `Locatable` |
| + filter, `To4` normalisation and the fixed-category error log, and treats |
| + a malformed location as a miss of the whole record. |
| +- **`Locate`:** unchanged. A valid point succeeds even without a country. |
| + Otherwise it tries the country's representative point, then the |
| + registered country's. Today's tests stay, and plan 1 adds regression |
| + cases for "point with no country" and "country with no representative |
| + point but a registered country that has one". |
| +- **`Lookup`:** succeeds when it finds a country, or a point, or both. |
| + - `Country` is the country's ISO code, else the registered country's. |
| + - `Subdivision` is `country + "-" + subdivisions[0].iso_code`, only when |
| + both exist and the code is 1 to 3 `[A-Z0-9]` characters. |
| + - **DB-IP Lite has subdivision names but no ISO codes**, so with DB-IP |
| + the subdivision is always empty. With GeoLite2 City it is usually |
| + filled. |
| |
| ## 7. The header contract |
| |
| @@ -269,87 +400,141 @@ func (m *MMDB) Source() string // metadata.DatabaseType, reduced to [A-Za-z0-9._ |
| | `X-Carlos-Geo-Lon` | decimal degrees, 2 places | the same | |
| | `X-Carlos-Geo-Source` | the database type, e.g. `DBIP-City-Lite`, `GeoLite2-City` | any geo header is present | |
| |
| -- All five are absent on a miss, on an address that isn't locatable, or |
| - when no database is loaded. The platform never guesses. |
| -- Two decimal places is about 1 km, roughly the precision of these |
| - databases anyway. The edge rounds before setting the header. |
| +- **All five are absent** on a miss, on an address that isn't locatable, |
| + when no database is loaded, and on a door request (§8). The platform |
| + never guesses. |
| +- **Coordinates are an approximate area, not a position** (codex R1-20). |
| + The two decimal places are formatting only. The database's own accuracy |
| + can be tens or hundreds of kilometres, and the app docs say so. |
| - Apps can already see the client IP in `X-Forwarded-For`, so these headers |
| tell an app nothing it couldn't work out for itself. |
| -- Apps that vary a cached response on a geo header must send `Vary` on it. |
| - The edge cache honours `Vary` with at most 8 variants per URL |
| - (`internal/edge/cache.go`). Country-level variation fits that budget. |
| - Coordinate-level variation does not, and the docs say so. |
| +- **Caching** (codex R1-19). An app that varies a cached response on a geo |
| + header must send `Vary` on it. The edge cache keeps at most 8 variants |
| + per URL across every `Vary` combination |
| + (`internal/edge/cache.go`). A ninth country evicts one, and other |
| + `Vary` headers multiply the combinations. The app docs say plainly that |
| + varying on country means more cache misses for a page with a worldwide |
| + audience. Varying on coordinates makes the cache almost useless for that |
| + page. |
| |
| ## 8. Trust: who may set the headers |
| |
| ### 8.1 The strip |
| |
| -`ServeHTTP` deletes every `X-Carlos-Geo-*` header from the inbound request |
| -on **every** request: every route, every proxy type, before any branch. |
| -The one exception is §8.3. A test sends forged headers through each of the |
| -six proxy types (unix, tcp, multi, tunnel, fleet, backhaul) and asserts the |
| -app never sees them. |
| +At the top of `ServeHTTP`, before any branch, the edge removes every |
| +`X-Carlos-Geo-*` header from the inbound request on every request. If the |
| +request is a verified hop or came over the tunnel (§8.3), it first parses |
| +and validates them (§8.4). It then writes the request's own geo result |
| +back onto `r.Header` before the cache lookup, so `Vary` matching sees |
| +exactly the values the app will get. |
| |
| -### 8.2 Setting them |
| +### 8.2 Computing: only at ingress |
| |
| -After the strip, the edge looks up `clientIP(r)` and sets the headers. This |
| -happens after `acceptHop`, the route lookup and the gate, and **before** |
| -the upgrade branch and the cache lookup, so WebSocket upgrades carry them |
| -and `Vary` matching sees the right values. The result is also put on the |
| -request context so analytics reuses it (§9) instead of looking up twice. |
| +The **ingress edge** is the one that accepted the visitor's connection. |
| +Only it looks the visitor up, using `clientIP(r)`. Every other edge adopts |
| +the validated headers it received. |
| |
| -| Path | What happens | |
| +| Request | Geo result | |
| |---|---| |
| -| Direct request | `clientIP` is `RemoteAddr`. Look up. | |
| -| Verified backhaul hop (home edge) | `clientIP` is the last XFF element, trusted only under `hopVerifiedKey`. The home edge strips and recomputes. It doesn't trust the sending edge's copy. | |
| -| Backhaul sender | Strips like any edge, then forwards. It doesn't compute, because the home edge will. | |
| -| Unverified hop | Treated as a direct request from the sender's address, as today. | |
| -| Door (edge-local socket) | **No geo headers.** The door replaces `RemoteAddr` with the box's own public address, so a lookup would locate the box. | |
| -| Sidecar through loopback | The address isn't locatable, so there are no headers. | |
| -| Static sites served by the edge | No proxied request, so nothing to set. | |
| - |
| -### 8.3 Tunnel and fleet boxes |
| - |
| -The hub edge strips and sets the headers like any edge, then `tunnelProxy` |
| -or `fleetProxy` forwards them. On the remote box: |
| - |
| -- `tunnelclient` serves the edge over the inverted HTTP/2 connection. |
| - Plan 1 wraps that handler so every request it serves carries a new |
| - context mark, `tunnelKey{}`. Nothing else may set that mark, the same |
| - property `hopVerifiedKey` rests on. |
| -- The remote edge's strip skips requests that carry `tunnelKey`, and it |
| - doesn't look up for them either (its `RemoteAddr` is the hub, which would |
| - locate the hub). The hub's headers reach the app unchanged. |
| -- A remote box has no public listener for app traffic. If an operator ever |
| - configures one, requests through it don't carry `tunnelKey` and get |
| - stripped, so a forged header still can't get through. |
| - |
| -A test drives a request through a real `tunnelclient` connection and checks |
| -that the hub's headers arrive. Another test sends forged headers to the |
| -remote edge's own listener and checks they are stripped. |
| - |
| -### 8.4 Mixed versions during a roll |
| - |
| -An edge still on an old binary doesn't strip `X-Carlos-Geo-*`. That is |
| -true today for every edge. The contract is not real until every edge runs |
| -the new binary. So the rollout order (§15) is: roll every box, **then** |
| -publish a database, **then** publish the app docs. |
| - |
| -## 9. Analytics countries |
| - |
| -- `analytics.Obs` gains `Country string`. `observeAnalytics` reads it from |
| - the request context (§8.2) and doesn't look up again. |
| -- `Delta` and the flush fold `Country` into `Day.Countries`. An empty |
| - country counts as `"_unknown"`. The map is bounded like the other |
| - summaries: there are fewer than 260 ISO codes, so in practice it never |
| - hits the `_other` overflow, but the bound is enforced anyway. |
| -- The console's analytics page (`app_analytics.html`) gains a Countries |
| - table: the top 10, then "other". |
| -- **The page shows the attribution its database requires** (§13), chosen |
| - from the pointer's `database_type`. The new strings go through |
| - copy-review before they're written. |
| -- Days before plan 1 rolls have an empty map. The table says "no country |
| - data for this period" rather than showing nothing. |
| +| Direct request | Look up `RemoteAddr` | |
| +| Door request (edge-local socket) | **None.** The door replaces `RemoteAddr` with the box's own public address, so a lookup would locate the box. | |
| +| Sidecar through loopback | None, because the address isn't locatable | |
| +| Verified backhaul hop (home edge) | Adopt the sending edge's validated headers. No lookup. | |
| +| Request over the tunnel (remote box) | Adopt the hub's validated headers. No lookup. | |
| +| Unverified hop | Treated as a direct request from the sender's address, as today. The headers were stripped. | |
| + |
| +- **The sending edge** of a backhaul computes the result like any ingress |
| + edge, then forwards it. A door request that is backhauled therefore |
| + carries no geo headers. The home edge adopts that absence and never |
| + locates the sending box (codex R1-3). |
| +- **The result goes on the request context** as `geoResult`. Analytics |
| + (§9) reads it from there, so an adopted result counts the same as a |
| + computed one (codex R1-10). |
| + |
| +### 8.3 Setting them on the outbound request |
| + |
| +Every proxy's `Rewrite` (unix, tcp, multi, tunnel, fleet, backhaul) calls |
| +one helper, `setGeo(pr)`, after `setForwarded(pr)`. It: |
| + |
| +- deletes every `X-Carlos-Geo-*` header from `pr.Out`; |
| +- writes them back from the context's `geoResult`. |
| + |
| +`ReverseProxy` removes headers named in `Connection` before calling |
| +`Rewrite`. So a client sending `Connection: X-Carlos-Geo-Country` can no |
| +longer strip the country from the app's request while the cache files the |
| +response under that country (codex R1-1). The cache stores the response |
| +under the same values `setGeo` writes, because both come from `geoResult`. |
| + |
| +### 8.4 Who is trusted, and how much |
| + |
| +- **Verified hop:** `hopVerifiedKey`, which only `acceptHop` sets, after an |
| + HMAC check against the mesh key. |
| +- **Tunnel:** a new context mark, `tunnelKey{}`. |
| + - `tunnelclient` wraps the handler it gives `http2.Server.ServeConn`, so |
| + every request it serves carries the mark, and nothing else sets it. |
| + - Tunnel boxes do run the ordinary edge listeners on :80 and :443 |
| + alongside the tunnel (`internal/host/agent.go`, `tunnelLoop`). A |
| + request through those listeners doesn't carry the mark, so it is |
| + stripped. A tunnel box holds no database, so it gets no headers either |
| + (codex R1-17). |
| +- **Validation before adopting a header** (anything invalid is dropped on |
| + its own): |
| + - country `^[A-Z]{2}$`; |
| + - subdivision `^[A-Z]{2}-[A-Z0-9]{1,3}$`, whose prefix matches the |
| + country; |
| + - lat/lon decimal numbers in range, re-rounded to 2 places, and kept |
| + only as a pair; |
| + - source `^[A-Za-z0-9._-]{1,64}$`, kept only if some other field |
| + survived. |
| +- **Tests:** |
| + - a forged header on every proxy type, with and without a `Connection` |
| + nomination, including a cache replay; |
| + - a forged header on a tunnel box's public listener; |
| + - a request through a real `tunnelclient` connection that adopts the |
| + hub's headers and counts the country in the remote box's analytics; |
| + - a door call to an app whose home is another box, which reaches the app |
| + with no geo headers. |
| + |
| +### 8.5 Mixed versions during a roll |
| + |
| +An old edge doesn't strip `X-Carlos-Geo-*`, so an old sending edge passes a |
| +forged header across a verified hop, and a new home edge adopts it. The |
| +contract therefore holds only once every edge runs a GeoIP-aware binary. |
| +The rollout (§15) uses `carlos system geoip status`'s verdict (§5.5) to |
| +show that, and only then publishes a database and the app docs. |
| + |
| +## 9. Analytics countries (codex R1-11, R1-12) |
| + |
| +- **Counting unit:** pageviews. A request counts toward `Countries` exactly |
| + when it counts toward `pages`, so the two tables add up the same way. |
| +- `analytics.Obs` gains `Country` and `GeoSource`, which `observeAnalytics` |
| + reads from the context's `geoResult`. It never looks up again. |
| +- A pageview with no country counts as `"_unknown"` when a database was |
| + loaded. With no database loaded it isn't counted in `Countries` at all, |
| + so "no GeoIP" never looks like "visitors from nowhere". |
| +- `Delta`, the flush and `Day` gain `GeoSources map[string]int64` |
| + (`json:"geo_sources"`): pageviews counted per database type. This is the |
| + provenance the attribution needs. A day's data can come from two sources |
| + if a publish changed the type mid-day. |
| +- **Merging:** |
| + - `Day.Countries` and `Day.GeoSources` merge additively across flushes; |
| + - the read path (`ReadRange`, `Summary`) sums them across boxes and |
| + days; |
| + - the analytics API response carries `countries` (top 10 plus `_other`) |
| + and `geo_sources`. |
| + - Both maps are bounded: countries by the ISO list plus `_unknown` and |
| + `_other`, sources at 8 entries plus `_other`. |
| +- **Old and new days:** days written before plan 1 have no `geo_sources` |
| + and an empty `countries`. They contribute nothing, and the read path |
| + treats the missing field as empty. A mixed range tests this. |
| +- **The console's analytics page** (`app_analytics.html`) gains a Countries |
| + table: the top 10, then "other". With no country data in the range it |
| + says so instead. |
| +- **Attribution:** the page shows one attribution line for every source in |
| + the range's `geo_sources`, using the table in §13. Historical DB-IP |
| + counts keep their DB-IP credit after a switch to GeoLite2, and the credit |
| + stays after `off`. |
| +- The new strings go through copy-review before they are written. |
| |
| ## 10. The feed (plan 2) |
| |
| @@ -364,8 +549,10 @@ process per deployment, so there is only ever one fetcher. |
| | `CONSOLE_GEOIP_MAXMIND_ACCOUNT` | MaxMind account ID, for `geolite2-*` | |
| | `CONSOLE_GEOIP_MAXMIND_KEY` | MaxMind license key, for `geolite2-*` | |
| |
| -An unknown feed name, or a GeoLite2 feed with no credentials, is logged |
| -once at startup. The feed stays off and the console still starts. |
| +- An unknown feed name, or a GeoLite2 feed with no credentials, is logged |
| + once at startup. The feed stays off and the console still starts. |
| +- The key is never logged and never sent anywhere but MaxMind's download |
| + host. It goes in Basic auth, never in the URL. |
| |
| ### 10.2 Checking for a new edition |
| |
| @@ -374,44 +561,88 @@ it starts. |
| |
| - **DB-IP:** the edition is `YYYY-MM`, from |
| `https://download.db-ip.com/free/dbip-<city|country>-lite-YYYY-MM.mmdb.gz`. |
| - If the pointer's `edition` isn't this month, it tries this month's URL. A |
| - 404 just means it isn't out yet, so it tries again tomorrow. When nothing |
| - has been published yet, it tries this month and then last month. |
| + - If the pointer's `edition` isn't this month, it tries this month's URL. |
| + A 404 means it isn't out yet, so it tries again tomorrow. |
| + - With no pointer, or a pointer from another origin, it tries this month |
| + and then last month. |
| - **MaxMind:** it HEADs |
| `https://download.maxmind.com/geoip/databases/GeoLite2-<City|Country>/download?suffix=tar.gz` |
| with Basic auth and compares `Last-Modified` with the pointer's |
| `edition`. MaxMind documents HEAD as not counting against the download |
| limit. When the edition differs, it fetches `suffix=tar.gz.sha256` and |
| then the archive. |
| +- **Redirects:** both providers redirect downloads (MaxMind to R2 presigned |
| + URLs). The client follows at most 3 redirects, HTTPS only. On a |
| + cross-host redirect it drops the Basic auth header. |
| +- A URL that starts answering 404 or 410 for more than 7 consecutive daily |
| + checks raises the box alert's feed reason (§10.4). That is how a provider |
| + changing its URLs shows up. |
| |
| ### 10.3 Integrity and safety |
| |
| -- The download streams to disk and is decompressed: `.gz` for DB-IP, the |
| - single `.mmdb` member of the `.tar.gz` for MaxMind. Decompressed output |
| - is capped at 1 GiB, so a decompression bomb can't fill the disk. |
| -- MaxMind's archive is checked against its published SHA-256. DB-IP |
| - publishes no checksum that the console can fetch, so it relies on TLS, |
| - `Verify` and §4.1's checks. |
| -- The feed publishes only when the new database's `build_epoch` is |
| - **newer** than the current pointer's and its `database_type` **matches** |
| - the current one. A feed can't replace a City database with a Country one |
| - or roll back to an older edition. The one exception is when the current |
| - pointer is a manual publish of a different type: the feed logs that |
| - it's replacing it and goes ahead, because configuring a feed is the |
| - operator saying which database they want. |
| -- If the pointer is absent and the feed is set, the feed publishes. That |
| - is how the first publish happens on a fresh deployment. |
| -- On any failure the current database stays. Each edition's failure is |
| - logged once. |
| - |
| -### 10.4 Staleness |
| - |
| -When the pointer's `build_epoch` is more than 45 days old, the console |
| -raises a deployment-level alert, `geoip_stale`, through its existing alert |
| -watcher (`internal/console/alertwatch.go`). This happens whether or not a |
| -feed is set. The geo data keeps being served. Email to the grant |
| -recipients works. `CARLOS_ALERT_WEBHOOK` is still unset on the flagship |
| -(#192), so the webhook leg delivers nowhere until that is fixed. |
| +- **Bounded input** (codex R1-7). The download streams to the publish |
| + path's temporary directory. |
| + - **The compressed download** is capped at 512 MiB while it is being |
| + copied. |
| + - **Decompression:** `.gz` for DB-IP. For MaxMind, the single regular |
| + `.mmdb` member of the `.tar.gz`. Every byte the tar reader expands |
| + counts toward a 1.5 GiB cap, including members it skips, and the |
| + `.mmdb` itself is capped at 1 GiB. |
| + - Each check runs under the 15-minute deadline. The directory is removed |
| + on every exit path. |
| +- **Checksums** (codex R1-18). MaxMind's archive is checked against its |
| + published `.sha256`. DB-IP shows MD5 and SHA-1 checksums on its HTML |
| + download page but offers no machine-readable checksum endpoint. The feed |
| + doesn't scrape the page. It relies on TLS to `download.db-ip.com`, then |
| + `Verify`, the country probe and the shrink check (§4.1). |
| +- **Type check** (codex R1-14). The database's `database_type` must match |
| + the configured feed, every time, including the first publish: |
| + - `dbip-city-lite` → `DBIP-City-Lite` |
| + - `dbip-country-lite` → `DBIP-Country-Lite` |
| + - `geolite2-city` → `GeoLite2-City` |
| + - `geolite2-country` → `GeoLite2-Country` |
| +- **Monotonic.** When the current pointer came from the same feed, the new |
| + `build_epoch` must be newer. A pointer from a manual publish or another |
| + feed is replaced by any database that passes the type check, and the |
| + feed logs that it replaced it. The feed never uses `force`, so the |
| + shrink check applies whenever the pointer came from a feed with the same |
| + type. |
| +- **First publish:** if the pointer is absent and the feed is set, the |
| + feed publishes. That is how a fresh deployment gets its first database. |
| +- **On any failure** the current database stays, and each edition's failure |
| + is logged once. |
| +- **GeoLite2 freshness:** MaxMind updates twice a week and the feed checks |
| + daily, so a working feed keeps the database well inside D1's 30 days. |
| + |
| +### 10.4 Alerts (codex R1-15) |
| + |
| +Today the console's alert watcher only reads per-box records under |
| +`alerts/boxes/`, and webhooks are delivered by the host process. A |
| +deployment-level console alert would need a new pipeline. Instead, GeoIP |
| +becomes a **box rule**, `geoip`, alongside `disk`, `mem`, `cpu` and `lag`. |
| + |
| +- **Evaluation:** each mirroring edge evaluates it on its own state, using |
| + `metrics.RuleState`'s existing sustain and hysteresis. The rule fires |
| + when any of these holds for 30 minutes: |
| + - the loaded database is older than 45 days, for any type; |
| + - a GeoLite2 database is within 9 days of its D1 expiry (21 days old); |
| + - a pointer exists but this box hasn't loaded it for over an hour (a bad |
| + blob or a full disk); |
| + - the feed reason: the console's last feed result, published in the |
| + pointer's sibling `control/geoip/feed.json`, has failed 7 times in a |
| + row. Only the console box evaluates this, so it fires once. |
| +- **Delivery:** through the existing paths. The host's alert sink handles |
| + the webhook (when `CARLOS_ALERT_WEBHOOK` is set, which on the flagship it |
| + isn't yet, #192) and the console's alert watcher sends email. |
| + `internal/host/alertsink.go` gains the `geoip` cases for its message |
| + text. |
| +- **Clearing** follows the rule's hysteresis: the condition must be false |
| + for 30 minutes, for example after a publish, after `off` (no pointer |
| + means no rule), or when the feed recovers. |
| +- **Tests:** |
| + - it fires with no other box alert present; |
| + - it clears after a publish; |
| + - it clears after `off`. |
| |
| ## 11. The lookup endpoint and CLI (plan 3) |
| |
| @@ -425,9 +656,11 @@ recipients works. `CARLOS_ALERT_WEBHOOK` is still unset on the flagship |
| - **Rate limit:** the existing per-caller limiter at 60 a minute. The |
| intended use is occasional lookups, not bulk enrichment; a bulk form can |
| come later if anyone needs it. |
| -- **200** `{"found": true, "country": "IE", "subdivision": "", "lat": 53.34, "lon": -6.27, "source": "DBIP-City-Lite", "attribution": "IP Geolocation by DB-IP", "attribution_url": "https://db-ip.com"}`. |
| - The fields follow §7: absent values are omitted, and the coordinates |
| - are rounded the same way. |
| +- **200** `{"found": true, "country": "IE", "lat": 53.34, "lon": -6.27, "source": "DBIP-City-Lite", "attribution": "IP Geolocation by DB-IP", "attribution_url": "https://db-ip.com"}`. |
| + - The fields follow §7: absent values are omitted, and the coordinates |
| + are rounded the same way. |
| + - The source and attribution come from the reader that answered, which |
| + may be the `CONSOLE_GEOIP_DB` override. |
| - **200** `{"found": false}` for an address that isn't locatable or isn't |
| in the database. |
| - **400** for a missing or malformed `ip`. |
| @@ -440,120 +673,244 @@ recipients works. `CARLOS_ALERT_WEBHOOK` is still unset on the flagship |
| and the attribution. `--json` prints the response as is. With no app |
| context it uses the current app, like the other app verbs. |
| |
| -### 11.3 The console's reader |
| +### 11.3 The console's reader (codex R1-6) |
| |
| -The console reads `/data/carlos/geoip/current.mmdb`, the file the console |
| -box's own edge mirrors. Every minute it checks the file's inode and swaps |
| -the reader (`geo.Swappable`) when the inode changes. Region guesses and the |
| -endpoint share that one reader. |
| +The console reads the generation that the console box's own edge mirrors. |
| +Once a minute it reads `/data/carlos/geoip/current` (§5.2): |
| |
| -`CONSOLE_GEOIP_DB`, when set, overrides the mirrored path and keeps its |
| -current rules: it's opened once, and replacing it means a rename plus a |
| -restart. That keeps existing deployments working unchanged. |
| +| What it finds | What it does | |
| +|---|---| |
| +| a new sha | open that generation with `OpenMMDBTrusted` and swap | |
| +| the same sha | nothing | |
| +| `current` missing, or naming no database | `Swap(nil)` | |
| +| any other read error | keep what's loaded, log once an hour | |
| + |
| +- **Starting with no file** is the normal case before the first publish. |
| + The reader starts empty and picks up the first generation within a |
| + minute. |
| +- **GeoLite2 expiry (D1)** is checked on the same tick, independently of |
| + the edge. |
| +- **Region guesses and the endpoint** share that one reader. |
| +- **`CONSOLE_GEOIP_DB`**, when set, overrides the mirrored path and keeps |
| + its current rules: it's opened once at start, with `Verify`, and |
| + replacing it means a rename plus a restart. Existing deployments keep |
| + working unchanged. D1 applies to the override too: a GeoLite2 override |
| + past expiry is closed and logged, and the operator replaces it. |
| |
| ## 12. Logging |
| |
| -Every message is a fixed category and **never includes an IP address**. |
| +Every message is a fixed category and **never includes an IP address or the |
| +MaxMind key**. |
| |
| | Where | Message | Rate | |
| |---|---|---| |
| | edge mirror | `edge: geoip database <sha8> (<type>, built <date>) loaded` | per swap | |
| | edge mirror | `edge: geoip turned off; no database published` | per transition | |
| +| edge mirror | `edge: geoip database <sha8> expired (GeoLite2, 30 days after build) and was removed` | per generation | |
| | edge mirror | `edge: geoip blob <sha8> rejected: <category>` | once per sha | |
| | edge mirror | `edge: geoip pointer unreadable; keeping the loaded database` | hourly while failing | |
| | geo | `geo: a GeoIP lookup failed and was treated as a miss` | once a minute, process-wide (existing) | |
| | console | `console: geoip published <sha8> (<type>, <origin>)` | per publish | |
| +| console | `console: geoip publish refused: <category>` | per attempt | |
| | console feed | `console: geoip feed <name>: edition <e> rejected: <category>` | once per edition | |
| | console feed | `console: geoip feed misconfigured: <category>` | at startup | |
| |
| ## 13. Licensing and docs |
| |
| -- **DB-IP Lite** is licensed CC BY 4.0. Pages that show or use its data |
| - need `IP Geolocation by DB-IP` with a link to `https://db-ip.com`. |
| -- **GeoLite2** is covered by MaxMind's EULA. It requires attribution |
| - (`This product includes GeoLite2 data created by MaxMind, available from |
| - https://www.maxmind.com`) and keeping the data current, which the feed |
| - does. |
| +| `database_type` prefix | Attribution the platform shows | Other duties (operator's) | |
| +|---|---|---| |
| +| `DBIP-` | `IP Geolocation by DB-IP`, linked to `https://db-ip.com` | CC BY 4.0: attribution on every page that shows or uses the data | |
| +| `GeoLite2-` | `This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com` | The GeoLite EULA, including the 30-day update duty the platform enforces through D1, and terms that pass on to the operator's own users. The docs link the EULA and don't paraphrase all of it. | |
| +| anything else | none known | The operator's licence for that database | |
| + |
| - **The operator owns compliance.** The operator docs say that plainly, |
| - next to the publish verb and the feed settings. |
| -- **The app docs** list the headers, explain that they may be absent and |
| - why, cover `Vary` and its limit, and spell out the duty: an app that |
| - shows visitors anything derived from these headers or the endpoint must |
| - carry the attribution `X-Carlos-Geo-Source` names. The endpoint returns |
| - the attribution text so an app can render it. |
| + next to the publish verb and the feed settings, and link both providers' |
| + terms. |
| +- **The app docs:** |
| + - list the headers and explain that they may be absent and why; |
| + - cover `Vary`, with the cache cost (§7); |
| + - say that coordinates are an approximate area; |
| + - spell out the duty: an app that shows visitors anything derived from |
| + these headers or the endpoint must carry the attribution that |
| + `X-Carlos-Geo-Source` names. The endpoint returns the attribution |
| + text, so an app can render it. |
| - **The platform's own pages** that use the data carry the attribution: |
| - the analytics Countries table, and the new-app region guess if it ever |
| - shows the visitor where it thinks they are. |
| + the analytics Countries table, per §9, and the new-app region guess if |
| + it ever shows the visitor where it thinks they are. |
| |
| ## 14. Testing |
| |
| - **`internal/geo`:** |
| - - `Lookup` against the MaxMind fixtures already in `testdata` |
| - (country, registered-country fallback, subdivision codes from |
| + - `Lookup` against the MaxMind fixtures already in `testdata` (country, |
| + registered-country fallback, subdivision codes from |
| `GeoIP2-City-Test.mmdb`, no point from a Country database); |
| - malformed locations; |
| - - `Locate` returning exactly what it did before (the existing tests |
| - stay); |
| - - `Source` sanitising. |
| + - `Locate` regressions (§6); |
| + - `Source` sanitising; |
| + - `OpenMMDBTrusted`. |
| - **Swap:** `-race`, with 32 goroutines looking up while 100 swaps happen, |
| and a `Close` spy proving no close happens while a read lock is held. |
| -- **Mirror**, against the in-memory store: |
| +- **Mirror,** against the in-memory store: |
| - first publish, a new sha, the pointer removed, the pointer |
| unreadable; |
| - - a blob with a sha mismatch, a blob that fails `Verify`; |
| - - boot with a local file and no bucket; |
| + - a digest mismatch, an oversize blob, a disk write failure; |
| + - boot with a local generation and no bucket; |
| + - boot with `current` naming a missing file; |
| + - a crash injected between each step of §5.3; |
| - the old blob deleted between reading the pointer and fetching the |
| - blob. |
| + blob; |
| + - GeoLite2 expiry at boot and while running. |
| - **Edge:** |
| - - the strip test across all six proxy types; |
| - - setting on a direct request; |
| - - recomputing on a verified hop, never trusting a sender's copy; |
| - - none on the door, none for loopback; |
| - - the tunnel mark passing the hub's headers; |
| - - a forged header on a tunnel box's own listener being stripped; |
| + - the forgery tests in §8.4; |
| + - a direct request getting headers; |
| + - a verified hop adopting the sender's headers with no lookup (a lookup |
| + spy proves it); |
| + - none on the door, locally or backhauled; |
| + - none for loopback; |
| - headers present on a WebSocket upgrade; |
| - the cache's `Vary` variant selected by country; |
| - - analytics `Countries` filled from the same lookup. |
| -- **Publish path:** each refusal in §4.1, the `ifMatch` race, the |
| - old-blob delete. |
| -- **Feed**, against an `httptest` server standing in for DB-IP and MaxMind: |
| + - analytics `Countries` and `GeoSources` filled from `geoResult`. |
| +- **Publish path:** |
| + - each refusal in §4.1; |
| + - publishing the same file twice; |
| + - X to Y to X; |
| + - publish racing `off` (the mutex); |
| + - a foreign pointer write (the `ifMatch` refusal); |
| + - the sweep; |
| + - the `PutFrom` fallback size limit. |
| +- **Feed,** against an `httptest` server standing in for DB-IP and MaxMind: |
| - a new edition; |
| - a 404 before the month's file appears; |
| - - an older `build_epoch`, a type mismatch; |
| - - a bad SHA-256, a decompression bomb; |
| + - an older `build_epoch`; |
| + - a type mismatch, including on the first publish; |
| + - a bad SHA-256; |
| + - a decompression bomb, and a tar with a huge skipped member; |
| + - a cross-host redirect dropping auth; |
| - HEAD-only when `Last-Modified` hasn't changed; |
| - - the staleness alert. |
| -- **Endpoint:** each status code, role refusals, the rate limit, and that |
| - no IP ever reaches the log. |
| + - 7 consecutive 404s raising the feed reason. |
| +- **Analytics:** |
| + - merge across boxes and days; |
| + - a mixed range of old and new days; |
| + - attribution per source. |
| +- **Endpoint:** |
| + - each status code; |
| + - role refusals; |
| + - the rate limit; |
| + - the override reader's attribution; |
| + - that no IP ever reaches the log. |
| |
| ## 15. Rollout |
| |
| -1. Merge plan 1 and roll every box. There is no pointer yet, so nothing |
| - changes. Strips happen, but there is nothing to strip. |
| -2. Confirm with `carlos system geoip status` that every box reports a |
| - mirror with no database. |
| -3. Publish a database by hand (plan 1), or wait for plan 2's feed. Confirm |
| - every box reports the sha. |
| +1. Merge plan 1 and roll every box. There is no pointer yet, so no box |
| + sets any header. Every edge strips forged ones. |
| +2. Run `carlos system geoip status` until its verdict says every fresh box |
| + runs a GeoIP-aware binary (§5.5). |
| +3. Publish a database by hand (plan 1), or set the feed (plan 2). Check |
| + that every mirroring box reports the pointer's sha. |
| 4. Check a request to a test app carries the headers. Check a forged header |
| - from outside is replaced. |
| + from outside is replaced, and a door call carries none. |
| 5. Publish the app docs. |
| |
| -Plans 2 and 3 roll the console only. The feed does nothing until |
| -`CONSOLE_GEOIP_FEED` is set. |
| +Plans 2 and 3 roll the console, and plan 2 also rolls the boxes for the |
| +`geoip` rule. The feed does nothing until `CONSOLE_GEOIP_FEED` is set. |
| |
| ## 16. Plans |
| |
| | Plan | Contents | |
| |---|---| |
| -| 1 | `geo.Lookup`, `geo.Swappable`, `release.StreamGetter`, the console publish path and `carlos system geoip publish|status|off`, the edge mirror and box status, strip and set, the tunnel mark, analytics `Countries` and the console table, operator and app docs | |
| -| 2 | The feed: DB-IP and MaxMind sources, integrity checks, monotonic publish, the `geoip_stale` alert | |
| -| 3 | The console reader on the mirrored file, the endpoint, `carlos geo` | |
| +| 1 | `geo.Lookup`/`OpenMMDBTrusted`/`Swappable`; `release.PutStreamer` and its wrappers; the console publish path and `carlos system geoip publish/status/off`; the edge mirror, its local layout and box status; ingress computing, adopting, `setGeo` and the tunnel mark; GeoLite2 expiry; analytics `Countries`/`GeoSources` through to the console table; operator and app docs | |
| +| 2 | The feed (DB-IP and MaxMind sources, bounded input, integrity, type and monotonic checks, `feed.json`) and the `geoip` box rule | |
| +| 3 | The console reader on the mirrored generation, the endpoint, `carlos geo` | |
| |
| ## 17. Open items for the plans |
| |
| -- **The country-data probe in §4.1.3** (metadata versus known addresses) |
| - is decided against a real DB-IP Lite and GeoLite2 file in plan 1, task 1. |
| - That task also confirms that DB-IP Lite has no subdivision ISO codes. |
| +- **Plan 1, task 1** confirms against real DB-IP Lite and GeoLite2 City |
| + files: |
| + - that DB-IP Lite has no subdivision ISO codes; |
| + - that the country probe's five addresses resolve in both; |
| + - `Verify`'s peak memory (§4.1). |
| - **DB-IP's site says nothing on automated downloads.** Plan 2 fetches at |
| most once a day, and only when a new month is due. If DB-IP objects, the |
| feed falls back to manual publishing with no code change. |
| + |
| +## 18. Review dispositions |
| + |
| +Round 1 (`CODEX-FINDINGS-GEOIP-R1.md`): every finding accepted. |
| + |
| +| # | Sev | Disposition | |
| +|---|---|---| |
| +| 1 | P1 | Accepted. `setGeo` in every `Rewrite`, after hop-header removal, from the same `geoResult` the cache uses (§8.3) | |
| +| 2 | P1 | Accepted. One mutex through cleanup, idempotent blobs, a sweep that keeps the current blob (§3) | |
| +| 3 | P1 | Accepted. The trust model is now ingress-computes, adopt-downstream, so a backhauled door call carries no headers (§8.2) | |
| +| 4 | P1 | Accepted. D1: GeoLite2 expiry at 30 days, enforced by the edges, console and override; alert at 21 days (§0, §5.3, §10.4, §11.3) | |
| +| 5 | P2 | Accepted. Immutable generations named by digest, plus an fsynced `current` (§5.2) | |
| +| 6 | P2 | Accepted. The console's off transition and empty start (§11.3) | |
| +| 7 | P2 | Accepted. Bounds during the copy, a 409 for a concurrent publish, deadlines, cleanup, a `Verify` memory measurement, and no `Verify` on edges (§4.1, §5.3, §10.3) | |
| +| 8 | P2 | Accepted. `Opener`/`OpenObject` for reads; `PutStreamer` for uploads, forwarded by every wrapper (§4.1) | |
| +| 9 | P2 | Accepted. Shared decode; `Locate` unchanged, with regression cases (§6) | |
| +| 10 | P2 | Accepted. Adopted headers fill `geoResult`, so tunnel analytics count countries (§8.2) | |
| +| 11 | P2 | Accepted. Pageviews; the read path, API and table are in plan 1; mixed-range test (§9) | |
| +| 12 | P2 | Accepted. Per-day `GeoSources` provenance; live attribution from the answering reader; the EULA linked, not paraphrased (§9, §11.1, §13) | |
| +| 13 | P2 | Accepted. `apiSystemRoller`, person-only (§4.2) | |
| +| 14 | P2 | Accepted. `node_count` in the pointer, comparisons against the etag read, type checked against the feed always (§3, §4.1, §10.3) | |
| +| 15 | P2 | Accepted. A box rule through the existing pipelines instead of a new console alert (§10.4) | |
| +| 16 | P2 | Accepted. A status block on every new binary, with a mode and a fleet verdict (§5.5) | |
| +| 17 | P3 | Accepted. Tunnel boxes have public listeners; the strip covers them (§8.4) | |
| +| 18 | P3 | Accepted. DB-IP's HTML checksums named and deliberately not scraped (§10.3) | |
| +| 19 | P3 | Accepted. The cache-eviction cost is documented (§7) | |
| +| 20 | P3 | Accepted. Coordinates described as an area (§7) | |
| +| 21 | P3 | Accepted. Lookup probes instead of metadata (§4.1) | |
| +| 22 | P3 | Accepted. #241 quoted in Appendix A | |
| + |
| +## Appendix A. The request (#241, verbatim) |
| + |
| +> **Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint)** |
| +> |
| +> ## Why |
| +> |
| +> The nearest-region-default spec (`docs/superpowers/specs/2026-10-01-nearest-region-default-design.md`, §6, on branch `edge-replica-placements`) gives the console an optional GeoIP reader: `CONSOLE_GEOIP_DB` names an operator-supplied `.mmdb` (DB-IP Lite or MaxMind GeoLite2), read through `github.com/oschwald/maxminddb-golang`, mapped and opened once per process. Today it serves one caller: guessing a new app's region from the requester's IP. |
| +> |
| +> Once the platform carries that database, every app on it is one lookup away from "roughly where is this visitor", and none of them should have to ship their own `.mmdb`, handle attribution, or track database updates. The platform should expose the lookup to apps. |
| +> |
| +> ## What |
| +> |
| +> A platform geolocation facility that any app can use, backed by the same database and reader the console already has. Two shapes are worth weighing; the agent picking this up should choose one (or both) and say why in the spec: |
| +> |
| +> 1. **Router-injected request headers.** The edge/router does the lookup on the client IP and sets headers on the proxied request, e.g. `X-Carlos-Geo-Country`, `X-Carlos-Geo-Region`, `X-Carlos-Geo-Lat` / `-Lon`, `X-Carlos-Geo-Source`. Zero integration work for apps; works in every language; covers the common case (country/currency/locale defaults, "nearest region" hints). Cost: a lookup on every request unless cached per connection or per IP. |
| +> 2. **A lookup endpoint.** A platform-internal HTTP endpoint apps can call with an arbitrary IP (`GET /geo?ip=…` on the sidecar/hop path, authenticated the way other platform-internal calls are). Needed for anything that isn't "the current request": batch jobs, IPs from logs, webhooks carrying a client IP. |
| +> |
| +> Headers alone probably cover 90% of app needs; the endpoint covers the rest. Shipping headers first and the endpoint as a follow-on is a reasonable sequence. |
| +> |
| +> ## Constraints to carry over from the spec |
| +> |
| +> - Unset `CONSOLE_GEOIP_DB` means the feature is off: headers absent / endpoint returns "unavailable", never a guess. Unopenable database: logged once, feature off, platform still starts. |
| +> - Never look up loopback, private, link-local, CGNAT or unspecified addresses. |
| +> - Lookup errors are a miss, logged at most once a minute. |
| +> - Database replacement is an atomic rename plus restart; whatever process owns the lookup needs the same lifecycle. |
| +> - Attribution: DB-IP Lite (CC BY 4.0) requires "IP Geolocation by DB-IP" with a link on pages that show the data. If apps surface the data, the platform docs must tell them about that obligation, and the platform's own pages that use it must carry it. GeoLite2 has its own EULA terms (attribution, keeping the data updated). The operator owns compliance; the docs should say so plainly. |
| +> - The reader lives in (or is being added to) the console. If the router or a sidecar needs it, the reader should move to a shared internal package rather than being duplicated. |
| +> |
| +> ## Open questions |
| +> |
| +> - Where does the lookup physically run: router, console, sidecar? Which one actually sees the client IP, and which one can memory-map the database? Only one place should open the file. |
| +> - Header naming and whether to include coordinates or just country/subdivision. Coordinates make "nearest X" trivial for apps but are more than most need. |
| +> - Should apps be able to opt out (an app config flag) to avoid the per-request cost when they don't care? |
| +> - Does `carlos` CLI need anything (e.g. `carlos geo <ip>` for debugging what the platform would return)? |
| +> |
| +> ## Related |
| +> |
| +> - Nearest-region default spec, §6 (IP lookup), §14 (region table). Build on its reader; do not add a second one. |
| + |
| +How rev 2 answers the request's constraints: |
| + |
| +- **"Unset means off":** the off switch is now an absent pointer (R1). |
| + `CONSOLE_GEOIP_DB` keeps its meaning for the console. |
| +- **"Atomic rename plus restart":** replaced by the hot swap for the edge |
| + and console mirror (R1). The override keeps the rename-plus-restart rule. |
| +- **"Only one place should open the file":** each process that serves |
| + lookups opens it once (one edge per box, plus the console). One file |
| + per box is mapped by every reader on that box, and the bucket holds the |
| + single source. |
| +- **"Header naming":** `X-Carlos-Geo-Subdivision` replaces the request's |
| + example `-Region`, because "region" already means an AWS region on this |
| + platform. |
amadan