| 1 | # Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint |
| 2 | |
| 3 | **Date:** 2026-10-06 |
| 4 | **Revision:** 13 |
| 5 | **Discussion:** amadan carlos/platform #241 (quoted in Appendix A) |
| 6 | **Branch:** `geoip-for-apps` |
| 7 | |
| 8 | - Rev 1 (5e1129be) records the design Paul approved in conversation on |
| 9 | 2026-10-06. |
| 10 | - Rev 1 went to Astra (GPT-6, via codex). The findings are in |
| 11 | `CODEX-FINDINGS-GEOIP-R1.md`: 22 findings, 4 of them P1. Every one was |
| 12 | checked against the code and accepted. Changes are marked |
| 13 | "(codex R1-N)", and §18 records each disposition. |
| 14 | - **Rev 2 changes the trust model (codex R1-3, R1-10).** The edge that |
| 15 | accepts the visitor's connection computes the headers. Downstream edges |
| 16 | trust them only over a verified hop or the tunnel. Rev 1 had the home edge |
| 17 | recompute, which located the sending box for a backhauled door request. |
| 18 | - Rev 2 makes one decision Paul has not seen, **D1** in §0: GeoLite2 |
| 19 | databases expire 30 days after they were built. |
| 20 | - Rev 3 answers Astra's round 2 (`CODEX-FINDINGS-GEOIP-R2.md`). Of round |
| 21 | 1's findings, 18 were resolved and 4 partly resolved. Round 2 adds 9 new |
| 22 | findings, 2 of them P1. Every one was checked and accepted, and changes |
| 23 | are marked "(codex R2-N)". |
| 24 | - Rev 3 adds an internal trust header, `X-Carlos-Hop-Geo`. It carries the |
| 25 | ingress edge's state (hit, miss, off) across a hop or the tunnel, and |
| 26 | no app ever sees it. |
| 27 | - It withholds geo on a rejected hop (and, until rev 7, on a plaintext tunnel; see D2). |
| 28 | - It adds a bucket lifecycle rule for GeoLite2's destroy duty. |
| 29 | - Rev 4 answers Astra's round 3 (`CODEX-FINDINGS-GEOIP-R3.md`). That round |
| 30 | resolved all 9 of round 2's findings, left 2 earlier findings partly |
| 31 | resolved, and raised 4 new ones, 1 of them P1. Every one was accepted, |
| 32 | and changes are marked "(codex R3-N)". |
| 33 | - Rev 4 also folds in a code survey made for plan 1, marked "(survey)": |
| 34 | - the console's unit isn't in the repo; |
| 35 | - nothing creates `/data/carlos` yet; |
| 36 | - the existing staleness rule; |
| 37 | - the console's 5-minute read timeout; |
| 38 | - which boxes hold a store; |
| 39 | - there is no in-memory store type; |
| 40 | - there is no in-repo page listing the headers apps receive. |
| 41 | - Rev 5 answers Astra's round 4 (`CODEX-FINDINGS-GEOIP-R4.md`). That round |
| 42 | resolved all of round 3, left 1 earlier finding partly resolved, and |
| 43 | raised 2 new P2s. All were accepted, marked "(codex R4-N)". |
| 44 | - Publishing becomes an upload followed by a job. |
| 45 | - A tunnelled request is counted once, by the hub. |
| 46 | - The rollout verdict reconciles against the edges records. |
| 47 | - Rev 6 answers Astra's round 5 (`CODEX-FINDINGS-GEOIP-R5.md`): 1 new P2 |
| 48 | and 2 partly resolved findings, all accepted, marked "(codex R5-N)". |
| 49 | - The upload slot is reserved before any body is read. |
| 50 | - Fleet boxes are reconciled through the registry. |
| 51 | - The tunnel analytics test is corrected. |
| 52 | - Rev 7 answers Astra's round 6 (`CODEX-FINDINGS-GEOIP-R6.md`), marked |
| 53 | "(codex R6-N)". |
| 54 | - It takes **D2**, which reverses round 2's TLS-only tunnel trust. |
| 55 | - Uploads stage in a directory the console clears at startup. |
| 56 | - A box is retired explicitly, by deleting its edges record. |
| 57 | - Rev 8 answers Astra's round 7 (`CODEX-FINDINGS-GEOIP-R7.md`), marked |
| 58 | "(codex R7-N)": |
| 59 | - an expired database is refused at publish; |
| 60 | - a source can never sanitise to empty; |
| 61 | - attributed sources are never folded into `_other`; |
| 62 | - the spec's signatures now match the committed code. |
| 63 | - Rev 9 answers Astra's round 8 (`CODEX-FINDINGS-GEOIP-R8.md`), marked |
| 64 | "(codex R8-N)": |
| 65 | - box identity is (namespace, label); |
| 66 | - a download superseded mid-flight is never installed, and the poll |
| 67 | clock runs from the start of each pass; |
| 68 | - §3 says digest, not size; |
| 69 | - only expected boxes can block the verdict. |
| 70 | - Rev 10 answers Astra's round 9 (`CODEX-FINDINGS-GEOIP-R9.md`), marked |
| 71 | "(codex R9-N)": |
| 72 | - the verdict knows four pointer states; |
| 73 | - `off` answers 409 during a publish instead of waiting; |
| 74 | - pointer reads are bounded at 30 seconds, and the off bound includes |
| 75 | them; |
| 76 | - an unreadable heartbeat blocks only through the expected box it hides. |
| 77 | - Rev 11 answers Astra's round 10 (`CODEX-FINDINGS-GEOIP-R10.md`): |
| 78 | - the `CONSOLE_GEOIP_DB` override's D1 expiry moves into plan 1; |
| 79 | - the off bounds read 11.5 and 12.5 minutes (user copy rounds them up to |
| 80 | 12 and 13); |
| 81 | - the verdict judges expiry by the clock. |
| 82 | - Rev 12 answers Astra's round 11 (`CODEX-FINDINGS-GEOIP-R11.md`). The |
| 83 | alert's expired reason now matches §5.5 (marked, or past D1 by the |
| 84 | clock), so expiry never clears an alert while the bucket is still |
| 85 | unmarked. |
| 86 | - **Astra's round 12 (`CODEX-FINDINGS-GEOIP-R12.md`) accepted rev 12 with no |
| 87 | findings.** |
| 88 | - **Rev 13 records Paul's rulings of 2026-10-07:** |
| 89 | - he accepts D1 and D2 (now R7 and R8); |
| 90 | - the free DB-IP Lite is the default, and operators (the flagship |
| 91 | included) may upgrade to paid MaxMind through two more feeds, |
| 92 | `geoip2-city` and `geoip2-country` (R9). |
| 93 | |
| 94 | ## 0. Rulings and decisions |
| 95 | |
| 96 | Paul's answers during brainstorming. Do not reopen them without him. |
| 97 | |
| 98 | - **R1. The database reaches the edges from the bucket.** The operator (or |
| 99 | the feed, R6) publishes once. Every edge mirrors it and swaps its reader |
| 100 | in place, with no restart. This replaces #241's "atomic rename plus |
| 101 | restart" rule for the edge. `CONSOLE_GEOIP_DB` keeps that rule as an |
| 102 | override (§11.3). |
| 103 | - **R2. Four headers:** country, subdivision, coordinates and source (§7). |
| 104 | - **R3. Tunnel and fleet boxes trust the hub.** The hub computes the |
| 105 | headers. The remote edge keeps them only on requests that arrived over |
| 106 | the authenticated tunnel (§8). Remote boxes hold no database. |
| 107 | - **R4. No per-app switch.** Headers go to every app whenever a database |
| 108 | is loaded. A switch can ride the telemetry settings pipeline later if |
| 109 | someone asks for one. |
| 110 | - **R5. One spec, three plans:** headers first, then the feed, then the |
| 111 | endpoint (§16). Analytics countries ship in plan 1. |
| 112 | - **R6. The database updates itself.** The console fetches new editions of |
| 113 | DB-IP Lite or GeoLite2 and publishes them through the same path as a |
| 114 | manual publish (§10). |
| 115 | |
| 116 | Decisions taken during review. Paul hasn't seen these yet; each is easy to |
| 117 | reverse. |
| 118 | |
| 119 | - **D1 (codex R1-4). GeoLite2 databases expire 30 days after their |
| 120 | `build_epoch`.** |
| 121 | - **Why:** MaxMind's GeoLite EULA requires the licensee to stop using a |
| 122 | superseded edition within 30 days of an update's release. MaxMind |
| 123 | updates GeoLite2 twice a week. A database 30 days past its build has |
| 124 | certainly been superseded, and counting from the build rather than |
| 125 | from the release of its successor errs on the strict side. |
| 126 | - **What happens:** |
| 127 | - At expiry every edge and the console drop the reader and delete |
| 128 | their local copies. |
| 129 | - The console deletes the blob and marks the pointer |
| 130 | `"expired": true` (§3). |
| 131 | - The deployment bucket is versioned, so a deleted blob survives as a |
| 132 | noncurrent version. A lifecycle rule on `control/geoip/` expires |
| 133 | noncurrent versions after 1 day (§15, codex R2 on R1-4). |
| 134 | - A `CONSOLE_GEOIP_DB` override file belongs to the operator. The |
| 135 | platform closes it at expiry and logs that it must be replaced, but |
| 136 | never deletes it. |
| 137 | - The box alert (§10.4) warns from day 21. |
| 138 | - **When it matters:** with the feed working, a GeoLite2 database never |
| 139 | gets near 30 days. Only a dead feed or a manual publish ever reaches |
| 140 | the limit. |
| 141 | - **DB-IP Lite has no such duty** (CC BY 4.0) and never expires. It |
| 142 | only alerts as stale. |
| 143 | |
| 144 | - **D2 (codex R6-1, reversing the R2-2 answer). Every request served over |
| 145 | the tunnel carries the hub's answer, TLS or not.** |
| 146 | - Round 2 withheld geo trust from plaintext tunnels, because nothing |
| 147 | authenticates the hub to the box there. |
| 148 | - That protects nothing. An on-path attacker who can pose as the hub on |
| 149 | a plaintext tunnel can already inject any request at all: method, |
| 150 | path, body, cookies, `X-Forwarded-For`. Geo headers add nothing to |
| 151 | what such an attacker controls. |
| 152 | - Withholding them did cost something. The hub's cache stored variants |
| 153 | under a country the app never received, and those variants were |
| 154 | replayed after the box reconnected over TLS. The hub's analytics |
| 155 | counted a country that §9's plaintext rule said shouldn't exist. |
| 156 | - So one mark, set on every request `tunnelclient` serves, means "the |
| 157 | hub computed this request's answer". The remote adopts that answer |
| 158 | and doesn't observe the request for analytics. |
| 159 | - Plaintext tunnels remain what `tunnelclient.Dial` already calls them: |
| 160 | "for tests". The operator docs say production tunnels use `https://` |
| 161 | (`infra/remote-host.md` already does). |
| 162 | - Because the hub's answer always reaches the app, the variant the hub |
| 163 | caches always matches what the app saw, whatever the transport, on |
| 164 | reconnect and on fleet fallback. |
| 165 | |
| 166 | - **R7, R8 (Paul, 2026-10-07):** D1 and D2 above are accepted as they |
| 167 | stand. |
| 168 | - **R9 (Paul, 2026-10-07): free by default, paid by choice.** |
| 169 | - The documented default feed is `dbip-city-lite`. DB-IP Lite is free, |
| 170 | CC BY 4.0, and allows commercial use with the attribution the |
| 171 | platform already shows. |
| 172 | - Operators, the flagship included, may upgrade to MaxMind's paid |
| 173 | databases with `geoip2-city` or `geoip2-country`. These use the same |
| 174 | download API and credentials as GeoLite2. |
| 175 | - Paid editions are not GeoLite2, so D1's expiry does not apply, and |
| 176 | the platform shows no attribution for them: the operator's MaxMind |
| 177 | licence governs. |
| 178 | - The feed stays off until `CONSOLE_GEOIP_FEED` is set, so no |
| 179 | deployment starts downloading, or takes on a licence, without its |
| 180 | operator choosing to. |
| 181 | - GeoLite2 remains supported, but the operator docs note that MaxMind |
| 182 | calls it not recommended for commercial use and that its EULA limits |
| 183 | redistribution, which serving answers to customer apps may engage. |
| 184 | |
| 185 | ## 1. Why, and what this is not |
| 186 | |
| 187 | The nearest-region work gave the console a GeoIP reader (`internal/geo`, |
| 188 | `CONSOLE_GEOIP_DB`). It serves one caller: guessing a new app's region. |
| 189 | Once the platform carries the database, every app should be able to ask |
| 190 | "roughly where is this visitor" without shipping its own `.mmdb`, handling |
| 191 | attribution, or tracking updates. Appendix A quotes the request. |
| 192 | |
| 193 | Goals: |
| 194 | |
| 195 | - Every proxied request to an app carries the visitor's country, and where |
| 196 | the database has them, subdivision and coordinates. |
| 197 | - Apps can look up an arbitrary IP (batch jobs, logs, webhooks). |
| 198 | - The platform's own analytics fill the `countries` map the analytics spec |
| 199 | reserved for this (`2026-09-07-errors-and-analytics-design.md`). |
| 200 | - The operator publishes or configures a feed once per deployment, not once |
| 201 | per box. |
| 202 | |
| 203 | Not goals: |
| 204 | |
| 205 | - **Accuracy beyond the database.** The headers say what the database says. |
| 206 | - **Header injection for traffic that never crosses the edge.** Scheduler |
| 207 | calls go straight to the socket. Sidecars reach their instance through |
| 208 | loopback. |
| 209 | - **ASN, ISP, timezone, city-name or accuracy-radius headers.** Additive |
| 210 | later if asked for. |
| 211 | - **A second reader.** Everything builds on `internal/geo`. |
| 212 | |
| 213 | ## 2. Overview |
| 214 | |
| 215 | ``` |
| 216 | operator ──carlos system geoip publish──┐ |
| 217 | ▼ |
| 218 | feed (console, daily) ──────────► console publish path (one at a time) |
| 219 | │ verify, upload blob, flip pointer |
| 220 | ▼ |
| 221 | bucket: control/geoip/current.json → control/geoip/<sha256>.mmdb |
| 222 | │ |
| 223 | every mirroring edge polls the pointer (10 min) |
| 224 | ▼ |
| 225 | /data/carlos/geoip/<sha256>.mmdb + current → geo.Swappable (hot swap) |
| 226 | │ |
| 227 | ingress edge: strip X-Carlos-Geo-*, look up the visitor, keep the result |
| 228 | on the request; downstream edges (verified hop, tunnel) |
| 229 | adopt the validated headers instead of looking up |
| 230 | │ │ |
| 231 | ▼ ▼ |
| 232 | every proxy's Rewrite sets the headers analytics Obs.Country |
| 233 | on the outbound request → Day.Countries |
| 234 | console: reads the same mirrored file → GET /api/cli/apps/{a}/{app}/geo?ip= |
| 235 | carlos geo <ip> |
| 236 | ``` |
| 237 | |
| 238 | ## 3. Bucket records |
| 239 | |
| 240 | The console is the **only writer** of `control/geoip/`. Edges only read it. |
| 241 | A deployment runs one console, and inside it a single mutex serializes |
| 242 | every publish, `off` and expiry through to the end of its cleanup |
| 243 | (codex R1-2). |
| 244 | |
| 245 | **The blob** is `control/geoip/<sha256>.mmdb`. Its name is the hex SHA-256 |
| 246 | of its bytes, so a blob never changes after it is written. |
| 247 | |
| 248 | **The pointer** is `control/geoip/current.json`: |
| 249 | |
| 250 | ```json |
| 251 | { |
| 252 | "v": 1, |
| 253 | "sha256": "9f2c…", |
| 254 | "key": "control/geoip/9f2c….mmdb", |
| 255 | "size": 131072000, |
| 256 | "database_type": "DBIP-City-Lite", |
| 257 | "build_epoch": 1790812800, |
| 258 | "node_count": 7342011, |
| 259 | "published": "2026-10-06T21:00:00Z", |
| 260 | "origin": "feed:dbip-city-lite", |
| 261 | "edition": "2026-10" |
| 262 | } |
| 263 | ``` |
| 264 | |
| 265 | - `database_type`, `build_epoch` and `node_count` come from the file's own |
| 266 | MMDB metadata. `node_count` is recorded so the shrink check (§4.1) can |
| 267 | compare against the current pointer without reading the current blob |
| 268 | (codex R1-14). |
| 269 | - `origin` is `manual` or `feed:<name>`. `edition` is the feed's own edition |
| 270 | marker: the `YYYY-MM` for DB-IP, the `Last-Modified` value for MaxMind, |
| 271 | and empty for a manual publish. |
| 272 | - **No pointer means the feature is off.** An unknown `v` is treated as no |
| 273 | pointer, logged once. |
| 274 | - **An expired pointer** (`"expired": true`, set only by D1) means "a |
| 275 | database was published and has run out". Edges load nothing, and the |
| 276 | `geoip` rule keeps firing (§10.4) until a fresh publish or an explicit |
| 277 | `off`. Expiry is not the same as off, so it never clears an alert |
| 278 | (codex R2-7). |
| 279 | |
| 280 | **Publishing,** holding the mutex: |
| 281 | |
| 282 | 1. Read the pointer and its etag. Every comparison in §4.1 and §10.3 is |
| 283 | against this read. |
| 284 | 2. Upload the blob, create-only. If an object with that name already |
| 285 | exists, stream it and check its SHA-256 and size. Only if both match is |
| 286 | it the same bytes, and publishing carries on: publishing the same file |
| 287 | twice is not an error. Anything else is refused, and the previous |
| 288 | pointer and blob are left untouched (codex R8-3). |
| 289 | 3. Write the pointer with `ifMatch` set to the etag from step 1. A refusal |
| 290 | means something outside this console wrote the pointer. Stop, and leave |
| 291 | the new blob for the sweep. |
| 292 | 4. **Sweep:** delete every `control/geoip/*.mmdb` other than the one the new |
| 293 | pointer names. This removes the previous blob, the leftovers of any |
| 294 | earlier failure, and never the current one, even in an X to Y to X |
| 295 | sequence. |
| 296 | |
| 297 | `off` and expiry take the same mutex. |
| 298 | |
| 299 | - **`off`** deletes the pointer, then sweeps every blob. |
| 300 | - **Expiry** rewrites the pointer with `"expired": true` (with `ifMatch`), |
| 301 | then sweeps every blob. |
| 302 | |
| 303 | `Store.Delete` has no condition. The pointer delete is therefore |
| 304 | unconditional, and its safety rests on the single-writer rule above: one |
| 305 | console, one mutex. Rev 3 drops the conditional delete that rev 2 asked |
| 306 | for, rather than invent a store operation (codex R2-6). |
| 307 | |
| 308 | An edge that read the old pointer just before a publish can find its blob |
| 309 | already deleted. It keeps what it has loaded and retries on the next poll, |
| 310 | when it reads the new pointer. |
| 311 | |
| 312 | **The orphan sweep** (codex R3 on R1-4): a console that crashed after |
| 313 | uploading a blob but before writing the pointer leaves a blob that is |
| 314 | still a current version, which the noncurrent-version lifecycle rule |
| 315 | (§15) doesn't touch. So the console runs the §3 sweep, holding the |
| 316 | mutex, at startup and once a day. Deleting the orphan turns it into a |
| 317 | noncurrent version, and the lifecycle rule then removes it. |
| 318 | |
| 319 | ## 4. Publishing |
| 320 | |
| 321 | ### 4.1 The console's publish path |
| 322 | |
| 323 | `geoipPublish(ctx, src io.Reader, origin, edition string, force bool)`, used |
| 324 | by both the CLI verb and the feed: |
| 325 | |
| 326 | 1. Refuse with 409 if another publish holds the mutex. Don't queue. |
| 327 | 2. Stream `src` to a fresh temporary directory under the console's data |
| 328 | directory, hashing as it goes. |
| 329 | - Refuse once 1 GiB has been read. The limit is enforced during the |
| 330 | copy, not afterwards (codex R1-7). |
| 331 | - The whole operation has a 15-minute deadline. |
| 332 | - The temporary directory is removed on every exit path. |
| 333 | 3. Open it with `geo.OpenMMDB`, which verifies it. |
| 334 | 4. **Country probe** (codex R1-21): look up `8.8.8.8`, `1.1.1.1`, `9.9.9.9`, |
| 335 | `208.67.222.222` and `2001:4860:4860::8888`. Refuse the database unless |
| 336 | at least three of them return a country. That catches an ASN database or |
| 337 | a schema the decoder can't read. |
| 338 | 5. **Expiry check** (codex R7-1): refuse a database that D1 already |
| 339 | counts as expired. `force` never overrides this. The check runs again |
| 340 | immediately before the pointer is written, because a long upload can |
| 341 | cross the expiry. A refusal leaves the previous pointer and blob |
| 342 | untouched. |
| 343 | 6. **Shrink check:** when the new database has the same `database_type` |
| 344 | as the current pointer's, refuse it if it has fewer than half that |
| 345 | pointer's `node_count`. This guards against a truncated file. The check |
| 346 | is skipped when the types differ, because node counts across products |
| 347 | aren't comparable. A manual publish can override the check with |
| 348 | `force`. The feed can't (codex R2 on R1-14). |
| 349 | 7. Publish (§3). |
| 350 | |
| 351 | Plan 1, task 1 measures `Verify`'s peak memory on a real DB-IP City Lite |
| 352 | file and a real GeoLite2 City file, and records the result in the plan. |
| 353 | `Verify` builds a map of data offsets, so it can cost much more than the |
| 354 | file's size. If the peak would push the console box past its memory |
| 355 | headroom, the plan says so and proposes a limit (codex R1-7). |
| 356 | |
| 357 | **Streaming upload** (codex R1-8). `Store.Put` takes the whole body as |
| 358 | `[]byte`. Plan 1 adds one optional interface. Its body is an |
| 359 | `io.ReadSeeker`, because the publish path always streams from its staged |
| 360 | file, and the AWS SDK signs a seekable body without buffering it (codex |
| 361 | R7-4): |
| 362 | |
| 363 | ```go |
| 364 | // release.PutStreamer is implemented by stores that can upload from a seekable reader. |
| 365 | type PutStreamer interface { |
| 366 | PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (etag string, err error) |
| 367 | } |
| 368 | ``` |
| 369 | |
| 370 | - It is implemented by `S3Store` (PutObject with a reader body and a |
| 371 | content length), `DirStore` (temp file and rename) and `prefixStore` |
| 372 | (rebase, then forward). |
| 373 | - Plan 1 lists every other wrapper around the console's store and makes |
| 374 | each one forward `PutFrom` the way it forwards `Opener`. |
| 375 | - A store without `PutFrom` takes the `Put` fallback only for files up to |
| 376 | 64 MiB. Anything larger is refused with a clear error. |
| 377 | |
| 378 | **Streaming reads** use the existing `release.Opener` / `OpenObject`, which |
| 379 | S3, `DirStore` and `prefixStore` already implement. No new read interface is |
| 380 | added. |
| 381 | |
| 382 | ### 4.2 Operator verbs |
| 383 | |
| 384 | Under `carlos system`, through the console API: |
| 385 | |
| 386 | - **Who may use them** (codex R1-13): a person, not a token, who |
| 387 | `apiSystemRoller` (`internal/console/apisystemupdate.go`) admits. That is |
| 388 | an owner of the ops account, or an address on `CARLOS_SYSTEM_OPERATORS`. |
| 389 | These are the people who can already roll the platform. The antivirus |
| 390 | doors use `apiSystemOperator`, which checks `CARLOS_STORE_OPERATORS` |
| 391 | instead, so they are not the precedent for this gate. |
| 392 | - `carlos system geoip publish <file.mmdb> [-force]` streams the file to |
| 393 | the console, which runs §4.1. |
| 394 | - **Upload, then a job** (codex R4-2): |
| 395 | - The CLI sends a raw `application/octet-stream` body with a |
| 396 | `Content-Length`, through `PUT /api/cli/system/geoip`. |
| 397 | - The console's server-wide `ReadTimeout` is 5 minutes. So the handler |
| 398 | extends its own read deadline to 15 minutes with |
| 399 | `http.NewResponseController`, the way the ship door does |
| 400 | (`handleAPIShip`, survey), and wraps the body in `MaxBytesReader` at |
| 401 | 1 GiB. |
| 402 | - **One slot, reserved first** (codex R5-1). The handler takes the |
| 403 | publish mutex (`TryLock`) **before reading any of the body**. If it |
| 404 | is held, by another upload still streaming or by a job, the answer |
| 405 | is 409 straight away. So at most one upload is ever staged on the |
| 406 | console's disk. |
| 407 | - The mutex and the temporary directory pass from the handler to |
| 408 | the job. The job releases the mutex only after removing the |
| 409 | directory. |
| 410 | - If the upload fails or is cut off before the job starts, the |
| 411 | handler removes the directory and releases the mutex itself. |
| 412 | - A test stalls a first upload's body and checks that a second |
| 413 | upload gets 409 without any of its body being read. |
| 414 | - The handler only stages the upload: it streams it to the publish |
| 415 | path's temporary file and hashes it. |
| 416 | - **The staging directory** is `<console data dir>/geoip-staging/`, |
| 417 | next to `CONSOLE_DB`, and used for nothing else. The console clears |
| 418 | it at startup, before it admits any upload or starts the feed, so a |
| 419 | crash during staging or during a job never leaves its 1 GiB behind |
| 420 | for good (codex R6-2). A test kills the publisher mid-stage and |
| 421 | mid-job, then checks that a fresh publisher on the same directory |
| 422 | starts empty and accepts a retry. |
| 423 | - It then answers **202 with a job id** straight away. A request that |
| 424 | came through another edge's backhaul has 30 seconds after its body |
| 425 | to see response headers (`backhaulHeaderTimeout`), and verifying |
| 426 | plus uploading can take longer. |
| 427 | - The job runs §4.1 steps 3 to 6 under the publish mutex and the |
| 428 | 15-minute deadline. |
| 429 | - It reuses the console's existing system job registry |
| 430 | (`s.systemJobs`, polled at `GET /api/cli/system/jobs/{id}`, the door |
| 431 | `carlos system update` already uses). |
| 432 | - The CLI polls until the job is done and prints the result. |
| 433 | - A second upload while a job holds the mutex is answered 409 before |
| 434 | its body is read. |
| 435 | - A test publishes through a real backhaul, with the job delayed past |
| 436 | the header timeout. |
| 437 | - The existing `PUT /api/cli/system/upload` holds the whole body in |
| 438 | memory, so it isn't reused. |
| 439 | - `carlos system geoip status` shows: |
| 440 | - the pointer (type, edition, build date, age, origin, expiry for |
| 441 | GeoLite2); |
| 442 | - the feed's configuration and its last result; |
| 443 | - the per-box report from §5.4. |
| 444 | - `carlos system geoip off` deletes the pointer, then sweeps the blobs |
| 445 | (§3). While a publish holds the slot (an upload streaming in, or its |
| 446 | job), `off` answers **409 straight away** rather than waiting behind it |
| 447 | (codex R9-2). The operator runs it again once the publish finishes. Every edge drops its reader within one poll: at most 11.5 minutes, |
| 448 | with jitter and one bounded pointer read, while pointer reads succeed |
| 449 | (§5.3). The console follows its own box's edge within a further minute, |
| 450 | so it takes at most 12.5 minutes (§11.3, codex R2-9, R10). User-facing |
| 451 | copy rounds these up to 12 and 13 minutes. If |
| 452 | `CONSOLE_GEOIP_FEED` is set, `off` warns that the feed will publish |
| 453 | again within a day and says to unset it. |
| 454 | |
| 455 | ## 5. The edge mirror |
| 456 | |
| 457 | ### 5.1 Where and who |
| 458 | |
| 459 | - The mirror runs inside `carlos agent`'s edge when the edge has a store |
| 460 | (`e.staticStore`, set from `CARLOS_DEPLOYMENT_BUCKET` or |
| 461 | `CARLOS_DEPLOYMENT_DIR`) and no tunnel client is configured. Its mode is |
| 462 | `mirror` (survey). |
| 463 | - **It does not run on boxes that use the tunnel client** (`CARLOS_TUNNEL_*` |
| 464 | set). Those boxes take the hub's headers (§8), and their mode is |
| 465 | `tunnel`. |
| 466 | - **A box with no store and no tunnel** has mode `none`. It strips forged |
| 467 | headers, as every edge does, and sets none. |
| 468 | - **Files** live under `/data/carlos/geoip/`, inside the edge unit's |
| 469 | `ReadWritePaths=/data`. |
| 470 | - Nothing creates `/data/carlos` today (survey). The mirror creates the |
| 471 | directory itself at mode 0755, and the files are 0644. The data isn't |
| 472 | secret. |
| 473 | - Bootstrap makes `/data` mode 0750, owned `carlos:carlos`. The console |
| 474 | already writes `/data/registry.db` and `/data/console/console.db`, so |
| 475 | whatever user its hand-installed unit runs as can already get into |
| 476 | `/data` (the unit isn't in the repo, survey). Step 2 of the rollout |
| 477 | (§15) checks on the console box that the console can read |
| 478 | `/data/carlos/geoip/current`. |
| 479 | |
| 480 | ### 5.2 Local layout (codex R1-5) |
| 481 | |
| 482 | Each generation is immutable and named by its digest: |
| 483 | |
| 484 | ``` |
| 485 | /data/carlos/geoip/<sha256>.mmdb one per generation, written once |
| 486 | /data/carlos/geoip/current {"sha256": "...", "database_type": "...", "build_epoch": ...} |
| 487 | /data/carlos/geoip/.incoming-* downloads in progress |
| 488 | ``` |
| 489 | |
| 490 | `current` is the only file that changes. It is written to a temporary file, |
| 491 | fsynced, renamed over `current`, and then the directory is fsynced. A |
| 492 | reader that sees `current` naming a sha always finds that generation |
| 493 | complete, because the generation was renamed into place before `current` |
| 494 | was written. A crash at any step leaves either the old `current` with the |
| 495 | old generation, or the new `current` with the new one. |
| 496 | |
| 497 | ### 5.3 The loop |
| 498 | |
| 499 | - **At boot:** |
| 500 | 1. Read `current` and open `<sha>.mmdb` with `geo.OpenMMDBTrusted`. That |
| 501 | opens the file without `Verify` (§6), because the console verified |
| 502 | those exact bytes and the name proves they are the same. |
| 503 | 2. Delete any other `*.mmdb` and any `.incoming-*`. |
| 504 | 3. Serving never waits on the bucket. |
| 505 | 4. A `current` that names a missing or unopenable file is removed and |
| 506 | logged, and the edge starts with no database. |
| 507 | 5. A GeoLite2 generation that has passed its expiry (D1) is deleted, not |
| 508 | opened. |
| 509 | - **One lock for selection** (codex R3-1). Choosing the generation, |
| 510 | expiring it and cleaning up the local files all happen under one mutex, |
| 511 | `genMu`. That covers writing `current`, `Swap`, and deleting |
| 512 | generations. Network reads and downloads happen outside it, so a stuck |
| 513 | download never holds it. |
| 514 | - **A finished download is installed only if the pointer still names it** |
| 515 | (codex R8-2). |
| 516 | - After the download the mirror re-reads the pointer. If the pointer now |
| 517 | names something else, is gone, or has expired, the download is |
| 518 | deleted and the pass re-runs once at once. So an `off` or a newer |
| 519 | publish that landed mid-download takes effect at the end of that pass. |
| 520 | - If the re-read fails, nothing is installed until the next pass. |
| 521 | - Each pass is due 10 minutes plus jitter after the previous one |
| 522 | **started**, and a pass is bounded at 10 minutes. |
| 523 | - Every pointer read has its own 30-second timeout (codex R9 on R8-2). |
| 524 | A change to the pointer is therefore acted on within 11 minutes plus |
| 525 | one pointer read (at most 11.5 minutes), even with a long download in |
| 526 | flight. That is the bound §4.2 quotes. |
| 527 | - While pointer reads keep failing, the edge keeps what it has loaded, |
| 528 | and its heartbeat reports the error, so the verdict shows the box as |
| 529 | not caught up. |
| 530 | - **Every 10 minutes,** with up to 1 minute of jitter, read the pointer |
| 531 | (bounded read, at most 64 KiB). Each pass has a 10-minute deadline, |
| 532 | covering the pointer read and any blob download. A pass that misses it |
| 533 | is abandoned and its `.incoming-*` deleted (codex R2 on R1-7). The |
| 534 | GeoLite2 expiry check runs on its own one-minute ticker, with no network |
| 535 | access, so a stuck download can't delay it. |
| 536 | - **An expired candidate is never installed.** |
| 537 | - A pointer whose `build_epoch` is past D1's expiry (it carries |
| 538 | `database_type` and `build_epoch`) is treated as an expired pointer |
| 539 | before anything is downloaded. |
| 540 | - The check runs again under `genMu`, immediately before installing, |
| 541 | against the downloaded file's own metadata. A download that crossed |
| 542 | its expiry is deleted, not installed. |
| 543 | - The mirror remembers the expired sha and doesn't fetch it again until |
| 544 | the pointer names another one. An expired pointer the console couldn't |
| 545 | mark can therefore never come back. |
| 546 | - **The pointer names the sha already loaded:** nothing. |
| 547 | - **It names a new sha:** |
| 548 | 1. Stream the blob through `OpenObject` into `.incoming-<sha>`. Stop and |
| 549 | delete it if it exceeds the pointer's `size` or 1 GiB, or if the disk |
| 550 | write fails. |
| 551 | 2. Check the digest. |
| 552 | 3. Fsync and rename it to `<sha>.mmdb`. |
| 553 | 4. Write `current` (§5.2), open the generation and swap the reader |
| 554 | (§5.4). |
| 555 | 5. Delete the previous generation. |
| 556 | - **No pointer, or an expired one:** write the empty state to |
| 557 | `current`, swap in a nil reader, then delete the generations. |
| 558 | - **The read fails** (network, permissions): keep what is loaded, and log |
| 559 | once an hour while it keeps failing. |
| 560 | - **A blob whose digest doesn't match** is deleted, logged once per sha, |
| 561 | and not retried until the pointer changes. |
| 562 | - **GeoLite2 expiry (D1):** the expiry ticker takes `genMu`, then checks |
| 563 | the **currently loaded** generation's `build_epoch`. Past expiry it swaps |
| 564 | in nil and deletes that generation, whatever the pointer says. |
| 565 | - Because it checks after taking the lock, it acts only on the |
| 566 | generation that is current at that moment. It can never inspect an |
| 567 | expired A and then remove a freshly installed B. |
| 568 | - Tests: |
| 569 | - a download crossing expiry; |
| 570 | - repeated polls against an expired pointer that was never marked; |
| 571 | - an install racing the ticker. |
| 572 | |
| 573 | ### 5.4 Hot swap |
| 574 | |
| 575 | `geo.Swappable` wraps the reader: |
| 576 | |
| 577 | ```go |
| 578 | type Swappable struct { |
| 579 | mu sync.RWMutex |
| 580 | cur *MMDB |
| 581 | } |
| 582 | // Snapshot is one answer from one reader: the result, that reader's |
| 583 | // source, and whether any reader was loaded at all (codex R2-4). |
| 584 | type Snapshot struct { |
| 585 | Result Result |
| 586 | Found bool |
| 587 | Source string // empty only when Loaded is false |
| 588 | Loaded bool |
| 589 | } |
| 590 | func (s *Swappable) Lookup(ip net.IP) Snapshot // one RLock covers lookup and source |
| 591 | func (s *Swappable) Locate(ip net.IP) (Point, bool) // geo.Locator; RLock held through MMDB.Locate; a miss when empty |
| 592 | func (s *Swappable) Swap(next *MMDB) // Lock, replace, Unlock, then Close the old one |
| 593 | ``` |
| 594 | |
| 595 | The headers, analytics and the endpoint's attribution all come from one |
| 596 | `Snapshot`, so a swap between the lookup and the response can't label |
| 597 | DB-IP data as GeoLite2. There is deliberately no separate `Source()` on |
| 598 | `Swappable`. |
| 599 | |
| 600 | `Swappable` implements `geo.Locator`, so the console's `Config.GeoIP` holds |
| 601 | the `Swappable` itself (codex R3-2). Region guesses |
| 602 | (`internal/console/regionguess.go`) never see the `*MMDB` underneath and |
| 603 | can't outlive a swap. The `-race` test runs concurrent `Locate` calls as |
| 604 | well as `Lookup` calls. |
| 605 | |
| 606 | - The old mapping is closed only after the write lock has been taken and |
| 607 | released. No lookup can be in flight on an unmapped file. |
| 608 | - A lookup holds the read lock for a few microseconds, so a swap waits that |
| 609 | long at most. |
| 610 | - Go's `RWMutex` blocks new readers while a writer waits, so a steady |
| 611 | stream of lookups can't starve the swap. |
| 612 | - The swap gets a `-race` test with concurrent lookups (§14). |
| 613 | |
| 614 | ### 5.5 Reporting (codex R1-16) |
| 615 | |
| 616 | Each box's status heartbeat gains a block on **every** box running a |
| 617 | binary with this feature: a new `GeoIP` field in `status.BoxMetrics` |
| 618 | (`internal/status/snapshot.go`), which `BuildFleet` never copies into the |
| 619 | public `fleet.json`. Platform boxes write `status/boxes/<label>.json`, and |
| 620 | fleet boxes write under `fleets/<fleet>/status/boxes/`: |
| 621 | |
| 622 | ```json |
| 623 | "geoip": {"mode": "mirror", "sha256": "9f2c…", "type": "DBIP-City-Lite", "error": ""} |
| 624 | "geoip": {"mode": "tunnel"} |
| 625 | "geoip": {"mode": "none"} |
| 626 | ``` |
| 627 | |
| 628 | `carlos system geoip status` reconciles two lists (codex R4 on R1-16). |
| 629 | |
| 630 | **Box identity is (namespace, label)** (codex R8-1). The namespace is the |
| 631 | platform (`""`) or a fleet's name. Fleet labels repeat across fleets and |
| 632 | can equal a platform label, so every match, every expectation and every |
| 633 | blocking line uses the pair. Lines name a fleet box as `<fleet>/<label>`. |
| 634 | |
| 635 | **What it reads:** |
| 636 | - **Expected boxes:** |
| 637 | - every platform box that publishes an edges record |
| 638 | (`control/edges/<box>.json`, what the home map reads); |
| 639 | - every active fleet box the registry lists (`ListFleetBoxes`). Fleet |
| 640 | boxes don't publish edges records (`internal/host/edgesfile.go`), so |
| 641 | they are reconciled against the registry instead (codex R5 on R1-16). |
| 642 | - **Heartbeats:** every prefix (the platform's and each fleet's |
| 643 | `fleets/<name>/status/boxes/`) is read with `status.LoadBoxesComplete`, |
| 644 | which says when any record couldn't be read. Neither `LoadBoxes` nor |
| 645 | `systemBoxes`'s strict mode is used, because both skip unreadable |
| 646 | records silently. |
| 647 | |
| 648 | **What the pointer asks of every mirror** (codex R9-1): |
| 649 | |
| 650 | | Pointer | Each expected mirroring box must report | |
| 651 | |---|---| |
| 652 | | live | the pointer's sha | |
| 653 | | absent (off), or expired (marked, or past D1 by the clock, codex R10) | no generation loaded | |
| 654 | | unreadable | nothing satisfies this: the verdict is never ready while the pointer can't be read | |
| 655 | |
| 656 | **Which boxes are listed, and which can block:** |
| 657 | - Every expected box is listed whatever its age, and **only expected |
| 658 | boxes can block** the verdict (codex R8 on R1-16). |
| 659 | - A box with a heartbeat but no edges record or fleet registration is |
| 660 | listed, for information, unless its last heartbeat is more than 7 days |
| 661 | old. It never blocks. So deleting a retired box's edges record (or |
| 662 | detaching a fleet box) unblocks the verdict at once. |
| 663 | |
| 664 | It reports: |
| 665 | |
| 666 | - **stale,** by the shared rule `status.BoxStale` (3 heartbeat periods, |
| 667 | 15 minutes), not a threshold of its own (codex R3-4). The verdict can't |
| 668 | vouch for a box it can't currently see. |
| 669 | - **record unreadable:** the box is missing from what was read, so if it |
| 670 | is expected it blocks as "no heartbeat". An unreadable record for a box |
| 671 | nobody expects blocks nothing (codex R9 on R1-16). If the list of |
| 672 | expected fleet boxes itself can't be read, that always blocks. |
| 673 | - **expected box with no heartbeat at all:** "no heartbeat". |
| 674 | - **no block:** "old binary". The headers can't be trusted while any such |
| 675 | box is serving. |
| 676 | - **`mode: tunnel`:** takes the hub's headers. |
| 677 | - **`mode: none`:** no store and no tunnel, so no headers. Safe, because |
| 678 | it strips forged ones. |
| 679 | - **`mode: mirror`:** shows the box's sha against the pointer's. |
| 680 | |
| 681 | **Retiring a box** (codex R6 on R1-16). Nothing deletes a retired box's |
| 682 | edges record today (`internal/console/instancesplacement.go`, "nothing |
| 683 | deletes a retired box's file"). A stale heartbeat is never taken as |
| 684 | retirement. |
| 685 | - An expected box whose heartbeat is stale blocks the verdict, with the |
| 686 | instruction: "no heartbeat since <date>. if this box is retired, delete |
| 687 | its edges record (control/edges/<label>.json)". |
| 688 | - Deleting that record is the explicit retirement. It also stops the home |
| 689 | map backhauling traffic to a dead box. |
| 690 | - A `carlos system` verb to retire a box belongs to the platform, not to |
| 691 | GeoIP. Plan 1 files it as a discussion rather than working around its |
| 692 | absence. |
| 693 | |
| 694 | It ends with a one-line verdict. **"Ready"** means the pointer was read, |
| 695 | the expected-box list was complete, and every **expected** box is fresh, |
| 696 | readable and GeoIP-aware, with every expected mirroring box holding what |
| 697 | the pointer asks for (table above). Anything else names the boxes in the way (codex R2 on R1-16). |
| 698 | A test sets a box's heartbeat just inside the next jittered tick and |
| 699 | checks it still reads as fresh. |
| 700 | |
| 701 | ## 6. `internal/geo` changes |
| 702 | |
| 703 | `Locate` keeps its exact behaviour (codex R1-9). Both methods share one |
| 704 | decode and validation step: |
| 705 | |
| 706 | ```go |
| 707 | type Result struct { |
| 708 | Country string // ISO 3166-1 alpha-2; country, else registered country |
| 709 | Subdivision string // ISO 3166-2 like "GB-SCT"; empty unless the database has an ISO code |
| 710 | Point Point |
| 711 | HasPoint bool // true only for a real location record, never a centroid |
| 712 | } |
| 713 | func (m *MMDB) Lookup(ip net.IP) (Result, bool) |
| 714 | func (m *MMDB) Source() string // metadata.DatabaseType, reduced to [A-Za-z0-9._-], at most 64 bytes; "mmdb" if that leaves nothing |
| 715 | func (m *MMDB) BuildEpoch() time.Time // UTC; the pointer stores BuildEpoch().Unix() |
| 716 | func (m *MMDB) NodeCount() uint64 |
| 717 | func OpenMMDBTrusted(path string) (*MMDB, error) // opens without Verify |
| 718 | ``` |
| 719 | |
| 720 | - **The source is never empty** (codex R7-2). A database type that |
| 721 | sanitises to nothing becomes `mmdb`, which satisfies the header grammar |
| 722 | (§8.4). A loaded reader therefore always has a source that crosses a |
| 723 | hop or the tunnel intact, and a local hit is never turned into `off` |
| 724 | downstream. |
| 725 | - **Shared:** `decode(ip) (mmdbRecord, bool)` applies the `Locatable` |
| 726 | filter, `To4` normalisation and the fixed-category error log, and treats |
| 727 | a malformed location as a miss of the whole record. |
| 728 | - **`Locate`:** unchanged. A valid point succeeds even without a country. |
| 729 | Otherwise it tries the country's representative point, then the |
| 730 | registered country's. Today's tests stay, and plan 1 adds regression |
| 731 | cases for "point with no country" and "country with no representative |
| 732 | point but a registered country that has one". |
| 733 | - **`Lookup`:** succeeds when it finds a country, or a point, or both. |
| 734 | - `Country` is the country's ISO code, else the registered country's. |
| 735 | - `Subdivision` is `country + "-" + subdivisions[0].iso_code`, only when |
| 736 | both exist and the code is 1 to 3 `[A-Z0-9]` characters. |
| 737 | - **DB-IP Lite has subdivision names but no ISO codes**, so with DB-IP |
| 738 | the subdivision is always empty. With GeoLite2 City it is usually |
| 739 | filled. |
| 740 | |
| 741 | ## 7. The header contract |
| 742 | |
| 743 | | Header | Value | Present when | |
| 744 | |---|---|---| |
| 745 | | `X-Carlos-Geo-Country` | ISO 3166-1 alpha-2, upper case | a country was found | |
| 746 | | `X-Carlos-Geo-Subdivision` | ISO 3166-2, e.g. `GB-SCT` | the database has an ISO subdivision code (GeoLite2, not DB-IP Lite) | |
| 747 | | `X-Carlos-Geo-Lat` | decimal degrees, 2 places | the record has a real location | |
| 748 | | `X-Carlos-Geo-Lon` | decimal degrees, 2 places | the same | |
| 749 | | `X-Carlos-Geo-Source` | the database type, e.g. `DBIP-City-Lite`, `GeoLite2-City` | any geo header is present | |
| 750 | |
| 751 | - **All five are absent** on a miss, on an address that isn't locatable, |
| 752 | when no database is loaded, and on a door request (§8). The platform |
| 753 | never guesses. |
| 754 | - **Coordinates are an approximate area, not a position** (codex R1-20). |
| 755 | The two decimal places are formatting only. The database's own accuracy |
| 756 | can be tens or hundreds of kilometres, and the app docs say so. |
| 757 | - Apps can already see the client IP in `X-Forwarded-For`, so these headers |
| 758 | tell an app nothing it couldn't work out for itself. |
| 759 | - **Caching** (codex R1-19). An app that varies a cached response on a geo |
| 760 | header must send `Vary` on it. The edge cache keeps at most 8 variants |
| 761 | per URL across every `Vary` combination |
| 762 | (`internal/edge/cache.go`). A ninth country evicts one, and other |
| 763 | `Vary` headers multiply the combinations. The app docs say plainly that |
| 764 | varying on country means more cache misses for a page with a worldwide |
| 765 | audience. Varying on coordinates makes the cache almost useless for that |
| 766 | page. |
| 767 | |
| 768 | ## 8. Trust: who may set the headers |
| 769 | |
| 770 | ### 8.1 Order of operations (codex R2-3) |
| 771 | |
| 772 | 1. **Capture and strip,** at the top of `ServeHTTP`, before any branch. The |
| 773 | edge copies `X-Carlos-Hop-Geo` and every `X-Carlos-Geo-*` header into an |
| 774 | untrusted `capturedGeo` value, then deletes them all from `r.Header`. |
| 775 | Nothing reads `capturedGeo` yet. |
| 776 | 2. **Authenticate:** `acceptHop` runs where it does today, and so does the |
| 777 | tunnel mark, which comes from the handler wrapper. |
| 778 | 3. **Decide,** after `acceptHop` and before either backhaul branch, the |
| 779 | door, the gate, analytics, the upgrade branch and the cache lookup. The |
| 780 | edge sets `geoResult` once, from the table in §8.2. Captured values are |
| 781 | adopted only when step 2 trusted the transport. Otherwise they are |
| 782 | discarded. |
| 783 | 4. **Expose:** the edge writes the app-visible headers from `geoResult` |
| 784 | back onto `r.Header`, so `Vary` matching sees exactly the values the app |
| 785 | will get. `X-Carlos-Hop-Geo` is never written back to `r.Header`. |
| 786 | |
| 787 | ### 8.1a The internal trust header (codex R2-5) |
| 788 | |
| 789 | Without more information a downstream edge can't tell "the ingress looked |
| 790 | up and missed" from "the ingress had no database" or "this was a door |
| 791 | call", since all three send no app headers. So the edge that forwards over |
| 792 | a backhaul or the tunnel adds one internal header: |
| 793 | |
| 794 | ``` |
| 795 | X-Carlos-Hop-Geo: hit;src=DBIP-City-Lite |
| 796 | X-Carlos-Hop-Geo: miss;src=DBIP-City-Lite |
| 797 | X-Carlos-Hop-Geo: off |
| 798 | ``` |
| 799 | |
| 800 | - `hit` and `miss` mean a reader was loaded and answered. `off` means no |
| 801 | lookup was attempted: no database, a door request, or a rejected |
| 802 | hop. |
| 803 | - It is set only by `backhaulProxy`, `tunnelProxy` and `fleetProxy`, and |
| 804 | only from `geoResult`. |
| 805 | - It is stripped at capture (step 1) on every request, and adopted only |
| 806 | under the same trust as the app headers. |
| 807 | - `setGeo` deletes it from every outbound request to an app, so it never |
| 808 | reaches app code. |
| 809 | - **An adopted `hit` needs a valid country or a valid coordinate pair** |
| 810 | (codex R3-3), matching `Lookup`'s own success rule (§6). A `hit` with |
| 811 | neither is treated as `off`, and so is a missing or malformed |
| 812 | `X-Carlos-Hop-Geo` on a trusted transport. Fewer countries are counted, |
| 813 | but none is made up. |
| 814 | - A test checks that identical app headers and analytics come out of a |
| 815 | local route, a backhaul and the tunnel, including for a coordinate-only |
| 816 | result. |
| 817 | |
| 818 | ### 8.2 Computing: only at ingress |
| 819 | |
| 820 | The **ingress edge** is the one that accepted the visitor's connection. |
| 821 | Only it looks the visitor up, using `clientIP(r)`. Every other edge adopts |
| 822 | the validated headers it received. |
| 823 | |
| 824 | | Request | Geo result | |
| 825 | |---|---| |
| 826 | | Direct request | Look up `RemoteAddr`: `hit` or `miss`, or `off` with no database | |
| 827 | | Door request (edge-local socket) | **`off`.** The door replaces `RemoteAddr` with the box's own public address, so a lookup would locate the box. | |
| 828 | | Sidecar through loopback | `miss`, because the address isn't locatable | |
| 829 | | Verified backhaul hop (home edge) | Adopt the sending edge's validated headers and state. No lookup. | |
| 830 | | Request over the tunnel (remote box), TLS or not | Adopt the hub's validated headers and state. No lookup (D2). | |
| 831 | | **Rejected hop:** it carries `X-Carlos-Hop`, but `acceptHop` didn't verify it | **`off`** (codex R2-1). The request is served as today, as an ordinary client with its forwarded headers stripped, but geo is withheld, because its `RemoteAddr` is the sending edge. A client that sends a bogus `X-Carlos-Hop` only suppresses its own geo. | |
| 832 | | Any other request | The same as a direct request | |
| 833 | |
| 834 | - **The sending edge** of a backhaul computes the result like any ingress |
| 835 | edge, then forwards it. A door request that is backhauled therefore |
| 836 | carries no geo headers. The home edge adopts that absence and never |
| 837 | locates the sending box (codex R1-3). |
| 838 | - **The result goes on the request context** as `geoResult`. Analytics |
| 839 | (§9) reads it from there, so an adopted result counts the same as a |
| 840 | computed one (codex R1-10). |
| 841 | |
| 842 | ### 8.3 Setting them on the outbound request |
| 843 | |
| 844 | Every proxy's `Rewrite` (unix, tcp, multi, tunnel, fleet, backhaul) calls |
| 845 | one helper, `setGeo(pr)`, after `setForwarded(pr)`. It: |
| 846 | |
| 847 | - deletes every `X-Carlos-Geo-*` header and `X-Carlos-Hop-Geo` from |
| 848 | `pr.Out`; |
| 849 | - writes the app headers back from the context's `geoResult`; |
| 850 | - writes `X-Carlos-Hop-Geo` too, but only when the proxy forwards to |
| 851 | another edge (`backhaulProxy`, `tunnelProxy`, `fleetProxy`, §8.1a). |
| 852 | |
| 853 | `ReverseProxy` removes headers named in `Connection` before calling |
| 854 | `Rewrite`. So a client sending `Connection: X-Carlos-Geo-Country` can no |
| 855 | longer strip the country from the app's request while the cache files the |
| 856 | response under that country (codex R1-1). The cache stores the response |
| 857 | under the same values `setGeo` writes, because both come from `geoResult`. |
| 858 | |
| 859 | ### 8.4 Who is trusted, and how much |
| 860 | |
| 861 | - **Verified hop:** `hopVerifiedKey`, which only `acceptHop` sets, after an |
| 862 | HMAC check against the mesh key. |
| 863 | - **Tunnel:** a new context mark, `tunnelKey{}`. |
| 864 | - `tunnelclient` wraps the handler it gives `http2.Server.ServeConn`, so |
| 865 | every request it serves carries the mark, and nothing else sets it. |
| 866 | - The mark is set whatever the tunnel's scheme (D2). |
| 867 | - The same mark tells the remote edge not to observe the request for |
| 868 | analytics (§9). |
| 869 | - Tunnel boxes do run the ordinary edge listeners on :80 and :443 |
| 870 | alongside the tunnel (`internal/host/agent.go`, `tunnelLoop`). A |
| 871 | request through those listeners doesn't carry the mark, so it is |
| 872 | stripped. A tunnel box holds no database, so it gets no headers either |
| 873 | (codex R1-17). |
| 874 | - **Validation before adopting a header** (anything invalid is dropped on |
| 875 | its own): |
| 876 | - country `^[A-Z]{2}$`; |
| 877 | - subdivision `^[A-Z]{2}-[A-Z0-9]{1,3}$`, whose prefix matches the |
| 878 | country; |
| 879 | - lat/lon decimal numbers in range, re-rounded to 2 places, and kept |
| 880 | only as a pair; |
| 881 | - source `^[A-Za-z0-9._-]{1,64}$`, kept only if some other field |
| 882 | survived. |
| 883 | - **Tests:** |
| 884 | - a forged header on every proxy type, with and without a `Connection` |
| 885 | nomination, including a cache replay; |
| 886 | - a forged header on a tunnel box's public listener; |
| 887 | - a request through a real `tunnelclient` connection that adopts the |
| 888 | hub's headers. The remote box makes **zero** analytics observations, |
| 889 | and the hub exactly one (codex R5 on R4-1, §9); |
| 890 | - a door call to an app whose home is another box, which reaches the app |
| 891 | with no geo headers; |
| 892 | - a hop signed with a key the verifier doesn't have yet (the rotation |
| 893 | window), and a door call backhauled in that window. Both are served, |
| 894 | with no geo; |
| 895 | - miss, off and door states crossing both a backhaul and the tunnel, and |
| 896 | counted correctly at the far side. |
| 897 | |
| 898 | ### 8.5 Mixed versions during a roll |
| 899 | |
| 900 | An old edge doesn't strip `X-Carlos-Geo-*`, so an old sending edge passes a |
| 901 | forged header across a verified hop, and a new home edge adopts it. The |
| 902 | contract therefore holds only once every edge runs a GeoIP-aware binary. |
| 903 | The rollout (§15) uses `carlos system geoip status`'s verdict (§5.5) to |
| 904 | show that, and only then publishes a database and the app docs. |
| 905 | |
| 906 | ## 9. Analytics countries (codex R1-11, R1-12) |
| 907 | |
| 908 | - **Counting unit:** pageviews. A pageview counts toward `Countries` when |
| 909 | its `geoResult` state is `hit` or `miss`. |
| 910 | - `analytics.Obs` gains `Country`, `GeoSource` and `GeoState`, which |
| 911 | `observeAnalytics` reads from the context's `geoResult`. It never looks |
| 912 | up again. |
| 913 | - **By state:** |
| 914 | - `hit` counts the country, or `"_unknown"` for a coordinate-only result |
| 915 | (codex R3-3). |
| 916 | - `miss` counts `"_unknown"`. |
| 917 | - `off` (no database, a door call, a rejected hop) isn't counted in `Countries` at all, so "no GeoIP" never looks like |
| 918 | "visitors from nowhere". |
| 919 | - The state crosses hops and the tunnel in `X-Carlos-Hop-Geo` (§8.1a), so |
| 920 | the edge that observes the pageview counts it the way the ingress edge |
| 921 | saw it. |
| 922 | - **One observation per served response** (codex R4-1). |
| 923 | - Today a request to a tunnel or fleet box is observed twice: once by |
| 924 | the hub's observer, which runs for every proxied route, and once by |
| 925 | the remote edge. Both flush to the same unprefixed |
| 926 | `apps/<sqid>/<app>/analytics/`. That already double-counts pageviews, |
| 927 | and countries would inherit the same double count. |
| 928 | - **Rule:** the remote edge doesn't observe a request that arrived over |
| 929 | the tunnel. The hub has already observed it, including when the hub |
| 930 | served it from its cache, where the remote never sees it. |
| 931 | - **Mark:** the tunnel mark, `tunnelKey`, which `tunnelclient` puts on |
| 932 | every request it serves (§8.4, D2). |
| 933 | - **Scope:** usage accounting (`usageMeter`) is a separate meter, and |
| 934 | this doesn't change it. |
| 935 | - **Test:** a real hub and remote edge with both meters on, both |
| 936 | flushed. Pageviews, countries and sources each go up by exactly one, |
| 937 | for a fresh request and for a hub cache hit. |
| 938 | - The sending edge of a backhaul does no metering today (§8.2), so a |
| 939 | backhauled request is already observed once, at the home edge. |
| 940 | - `Delta`, the flush and `Day` gain `GeoSources map[string]int64` |
| 941 | (`json:"geo_sources"`): pageviews counted per database type. This is the |
| 942 | provenance the attribution needs. A day's data can come from two sources |
| 943 | if a publish changed the type mid-day. |
| 944 | - **Merging:** |
| 945 | - `Day.Countries` and `Day.GeoSources` merge additively across flushes; |
| 946 | - the read path (`ReadRange`, `Summary`) sums them across boxes and |
| 947 | days; |
| 948 | - the analytics API response carries `countries` (top 10 plus `_other`) |
| 949 | and `geo_sources`. |
| 950 | - Both maps are bounded: countries by the ISO list plus `_unknown` and |
| 951 | `_other`, sources at 8 entries plus `_other`. |
| 952 | - **A source with a known attribution** (§13: the `DBIP-` and |
| 953 | `GeoLite2-` prefixes) is never folded into `_other`, whatever the cap. |
| 954 | Only unattributed types fold (codex R7-3). Those prefixes cover a |
| 955 | handful of product names, so the map stays small, and every credit a |
| 956 | retained range owes stays readable. |
| 957 | - **Old and new days:** days written before plan 1 have no `geo_sources` |
| 958 | and an empty `countries`. They contribute nothing, and the read path |
| 959 | treats the missing field as empty. A mixed range tests this. |
| 960 | - **The console's analytics page** (`app_analytics.html`) gains a Countries |
| 961 | table: the top 10, then "other". With no country data in the range it |
| 962 | says so instead. |
| 963 | - **Attribution:** the page shows one attribution line for every source in |
| 964 | the range's `geo_sources`, using the table in §13. Historical DB-IP |
| 965 | counts keep their DB-IP credit after a switch to GeoLite2, and the credit |
| 966 | stays after `off`. |
| 967 | - The new strings go through copy-review before they are written. |
| 968 | |
| 969 | ## 10. The feed (plan 2) |
| 970 | |
| 971 | The console fetches new editions itself. It is the one control-plane |
| 972 | process per deployment, so there is only ever one fetcher. |
| 973 | |
| 974 | ### 10.1 Configuration (console env) |
| 975 | |
| 976 | | Variable | Meaning | |
| 977 | |---|---| |
| 978 | | `CONSOLE_GEOIP_FEED` | `dbip-city-lite` (the documented default, R9), `dbip-country-lite`, `geolite2-city`, `geolite2-country`, and the paid `geoip2-city` and `geoip2-country`; unset means no feed | |
| 979 | | `CONSOLE_GEOIP_MAXMIND_ACCOUNT` | MaxMind account ID, for `geolite2-*` and `geoip2-*` | |
| 980 | | `CONSOLE_GEOIP_MAXMIND_KEY` | MaxMind license key, for `geolite2-*` and `geoip2-*` | |
| 981 | |
| 982 | - An unknown feed name, or a GeoLite2 feed with no credentials, is logged |
| 983 | once at startup. The feed stays off and the console still starts. |
| 984 | - The key is never logged and never sent anywhere but MaxMind's download |
| 985 | host. It goes in Basic auth, never in the URL. |
| 986 | |
| 987 | ### 10.2 Checking for a new edition |
| 988 | |
| 989 | The console checks once a day, at a jittered time, and once 5 minutes after |
| 990 | it starts. |
| 991 | |
| 992 | - **DB-IP:** the edition is `YYYY-MM`, from |
| 993 | `https://download.db-ip.com/free/dbip-<city|country>-lite-YYYY-MM.mmdb.gz`. |
| 994 | - If the pointer's `edition` isn't this month, it tries this month's URL. |
| 995 | A 404 means it isn't out yet, so it tries again tomorrow. |
| 996 | - With no pointer, or a pointer from another origin, it tries this month |
| 997 | and then last month. |
| 998 | - **MaxMind:** it HEADs |
| 999 | `https://download.maxmind.com/geoip/databases/GeoLite2-<City|Country>/download?suffix=tar.gz` |
| 1000 | with Basic auth and compares `Last-Modified` with the pointer's |
| 1001 | `edition`. MaxMind documents HEAD as not counting against the download |
| 1002 | limit. When the edition differs, it fetches `suffix=tar.gz.sha256` and |
| 1003 | then the archive. |
| 1004 | - **Redirects:** both providers redirect downloads (MaxMind to R2 presigned |
| 1005 | URLs). The client follows at most 3 redirects, HTTPS only. On a |
| 1006 | cross-host redirect it drops the Basic auth header. |
| 1007 | - A URL that starts answering 404 or 410 for more than 7 consecutive daily |
| 1008 | checks raises the box alert's feed reason (§10.4). That is how a provider |
| 1009 | changing its URLs shows up. |
| 1010 | |
| 1011 | ### 10.3 Integrity and safety |
| 1012 | |
| 1013 | - **Bounded input** (codex R1-7). The download streams to the publish |
| 1014 | path's temporary directory. |
| 1015 | - **The compressed download** is capped at 512 MiB while it is being |
| 1016 | copied. |
| 1017 | - **Decompression:** `.gz` for DB-IP. For MaxMind, the single regular |
| 1018 | `.mmdb` member of the `.tar.gz`. Every byte the tar reader expands |
| 1019 | counts toward a 1.5 GiB cap, including members it skips, and the |
| 1020 | `.mmdb` itself is capped at 1 GiB. |
| 1021 | - Each check runs under the 15-minute deadline. The directory is removed |
| 1022 | on every exit path. |
| 1023 | - **Checksums** (codex R1-18). MaxMind's archive is checked against its |
| 1024 | published `.sha256`. DB-IP shows MD5 and SHA-1 checksums on its HTML |
| 1025 | download page but offers no machine-readable checksum endpoint. The feed |
| 1026 | doesn't scrape the page. It relies on TLS to `download.db-ip.com`, then |
| 1027 | `Verify`, the country probe and the shrink check (§4.1). |
| 1028 | - **Type check** (codex R1-14). The database's `database_type` must match |
| 1029 | the configured feed, every time, including the first publish: |
| 1030 | - `dbip-city-lite` → `DBIP-City-Lite` |
| 1031 | - `dbip-country-lite` → `DBIP-Country-Lite` |
| 1032 | - `geolite2-city` → `GeoLite2-City` |
| 1033 | - `geolite2-country` → `GeoLite2-Country` |
| 1034 | - **Monotonic.** When the current pointer came from the same feed, the new |
| 1035 | `build_epoch` must be newer. A pointer from a manual publish or another |
| 1036 | feed is replaced by any database that passes the type check, and the |
| 1037 | feed logs that it replaced it. The feed never uses `force`, so the |
| 1038 | shrink check applies whenever the pointer came from a feed with the same |
| 1039 | type. |
| 1040 | - **First publish:** if the pointer is absent and the feed is set, the |
| 1041 | feed publishes. That is how a fresh deployment gets its first database. |
| 1042 | - **On any failure** the current database stays, and each edition's failure |
| 1043 | is logged once. |
| 1044 | - **GeoLite2 freshness:** MaxMind updates twice a week and the feed checks |
| 1045 | daily, so a working feed keeps the database well inside D1's 30 days. |
| 1046 | |
| 1047 | ### 10.4 Alerts (codex R1-15) |
| 1048 | |
| 1049 | Today the console's alert watcher only reads per-box records under |
| 1050 | `alerts/boxes/`, and webhooks are delivered by the host process. A |
| 1051 | deployment-level console alert would need a new pipeline. Instead, GeoIP |
| 1052 | becomes a **box rule**, `geoip`, alongside `disk`, `mem`, `cpu` and `lag`. |
| 1053 | |
| 1054 | - **Evaluation:** each mirroring edge evaluates it on its own state. The |
| 1055 | rule fires once any of these has held for 30 minutes, using |
| 1056 | `metrics.RuleState`'s existing sustain: |
| 1057 | - the loaded database is older than 45 days, for any type; |
| 1058 | - a GeoLite2 database is within 9 days of its D1 expiry (21 days old); |
| 1059 | - the pointer is expired: marked (§3), **or past D1 by the clock**. The |
| 1060 | mirror keeps the observed pointer's type and build after it drops the |
| 1061 | reader, so expiry never clears the alert before the console's daily |
| 1062 | pass marks the bucket (codex R11-1); |
| 1063 | - a live pointer exists but this box hasn't loaded it for over an hour |
| 1064 | (a bad blob or a full disk); |
| 1065 | - **the feed reason** (codex R2-7), evaluated only by the console box's |
| 1066 | edge, so it fires once: |
| 1067 | - the console's feed result, published at `control/geoip/feed.json`, |
| 1068 | shows 7 consecutive failed checks. This reason is independent of |
| 1069 | the pointer, so a fresh deployment whose feed has never succeeded |
| 1070 | still alerts. |
| 1071 | - `feed.json` is written by the console after every check (configured |
| 1072 | feed, last success, consecutive failures, last failure category). It |
| 1073 | is deleted when the feed is unset. |
| 1074 | - **Delivery:** through the existing paths. The host's alert sink handles |
| 1075 | the webhook (when `CARLOS_ALERT_WEBHOOK` is set, which on the flagship it |
| 1076 | isn't yet, #192) and the console's alert watcher sends email. |
| 1077 | `internal/host/alertsink.go` gains the `geoip` cases for its message |
| 1078 | text. |
| 1079 | - **Clearing is immediate** (codex R2-8): the first evaluation in which no |
| 1080 | reason holds clears the rule. That matches what the existing machinery |
| 1081 | does (`internal/metrics/alert.go`, which clears on the first reading |
| 1082 | below the line). None of these conditions flaps, so no sustained-clear |
| 1083 | bookkeeping is added. |
| 1084 | - The pointer-related reasons clear on a fresh publish, or on an |
| 1085 | explicit `off`, which deletes the pointer. |
| 1086 | - The feed reason clears on the feed's next success, or when the feed |
| 1087 | is unset. |
| 1088 | - Expiry alone never clears anything. |
| 1089 | - **Tests:** |
| 1090 | - it fires with no other box alert present; |
| 1091 | - it clears after a publish; |
| 1092 | - it clears after `off`; |
| 1093 | - it stays firing through expiry; |
| 1094 | - the feed reason fires with no pointer at all; |
| 1095 | - it restarts with the rule firing. |
| 1096 | |
| 1097 | ## 11. The lookup endpoint and CLI (plan 3) |
| 1098 | |
| 1099 | ### 11.1 Endpoint |
| 1100 | |
| 1101 | `GET /api/cli/apps/{account}/{app}/geo?ip=<addr>`, registered through |
| 1102 | `cliRoute`: |
| 1103 | |
| 1104 | - **Allowed:** the instance token (`svcrole.Instance`), and the member and |
| 1105 | app tokens that can read the app's analytics. |
| 1106 | - **Rate limit:** the existing per-caller limiter at 60 a minute. The |
| 1107 | intended use is occasional lookups, not bulk enrichment; a bulk form can |
| 1108 | come later if anyone needs it. |
| 1109 | - **200** `{"found": true, "country": "IE", "lat": 53.34, "lon": -6.27, "source": "DBIP-City-Lite", "attribution": "IP Geolocation by DB-IP", "attribution_url": "https://db-ip.com"}`. |
| 1110 | - The fields follow §7: absent values are omitted, and the coordinates |
| 1111 | are rounded the same way. |
| 1112 | - The source and attribution come from the same `Snapshot` as the |
| 1113 | result (§5.4), so they always describe the reader that answered, |
| 1114 | which may be the `CONSOLE_GEOIP_DB` override. |
| 1115 | - **200** `{"found": false}` for an address that isn't locatable or isn't |
| 1116 | in the database. |
| 1117 | - **400** for a missing or malformed `ip`. |
| 1118 | - **503** `geo unavailable` when no database is loaded. |
| 1119 | - The console never logs the queried IP. |
| 1120 | |
| 1121 | ### 11.2 CLI |
| 1122 | |
| 1123 | `carlos geo <ip> [--app <app>]` calls the endpoint and prints the fields |
| 1124 | and the attribution. `--json` prints the response as is. With no app |
| 1125 | context it uses the current app, like the other app verbs. |
| 1126 | |
| 1127 | ### 11.3 The console's reader (codex R1-6) |
| 1128 | |
| 1129 | The console reads the generation that the console box's own edge mirrors. |
| 1130 | Once a minute it reads `/data/carlos/geoip/current` (§5.2): |
| 1131 | |
| 1132 | | What it finds | What it does | |
| 1133 | |---|---| |
| 1134 | | a new sha | open that generation with `OpenMMDBTrusted` and swap | |
| 1135 | | the same sha | nothing | |
| 1136 | | `current` missing, or naming no database | `Swap(nil)`. The console therefore follows `off` within one edge poll plus one console poll (§4.2). | |
| 1137 | | any other read error | keep what's loaded, log once an hour | |
| 1138 | |
| 1139 | - **Starting with no file** is the normal case before the first publish. |
| 1140 | The reader starts empty and picks up the first generation within a |
| 1141 | minute. |
| 1142 | - **GeoLite2 expiry (D1)** is checked on the same tick, independently of |
| 1143 | the edge. |
| 1144 | - **Region guesses and the endpoint** share that one reader. |
| 1145 | - **`CONSOLE_GEOIP_DB`**, when set, overrides the mirrored path and keeps |
| 1146 | its current rules: it's opened once at start, with `Verify`, and |
| 1147 | replacing it means a rename plus a restart. Existing deployments keep |
| 1148 | working unchanged. D1 applies to the override too: a GeoLite2 override |
| 1149 | past expiry is closed and logged. The file is the operator's, so the |
| 1150 | platform never deletes it, and the log line says it must be replaced |
| 1151 | and the old one destroyed. |
| 1152 | |
| 1153 | ## 12. Logging |
| 1154 | |
| 1155 | Every message is a fixed category and **never includes an IP address or the |
| 1156 | MaxMind key**. |
| 1157 | |
| 1158 | | Where | Message | Rate | |
| 1159 | |---|---|---| |
| 1160 | | edge mirror | `edge: geoip database <sha8> (<type>, built <date>) loaded` | per swap | |
| 1161 | | edge mirror | `edge: geoip turned off; no database published` | per transition | |
| 1162 | | edge mirror | `edge: geoip database <sha8> expired (GeoLite2, 30 days after build) and was removed` | per generation | |
| 1163 | | edge mirror | `edge: geoip blob <sha8> rejected: <category>` | once per sha | |
| 1164 | | edge mirror | `edge: geoip pointer unreadable; keeping the loaded database` | hourly while failing | |
| 1165 | | geo | `geo: a GeoIP lookup failed and was treated as a miss` | once a minute, process-wide (existing) | |
| 1166 | | console | `console: geoip published <sha8> (<type>, <origin>)` | per publish | |
| 1167 | | console | `console: geoip publish refused: <category>` | per attempt | |
| 1168 | | console feed | `console: geoip feed <name>: edition <e> rejected: <category>` | once per edition | |
| 1169 | | console feed | `console: geoip feed misconfigured: <category>` | at startup | |
| 1170 | |
| 1171 | ## 13. Licensing and docs |
| 1172 | |
| 1173 | | `database_type` prefix | Attribution the platform shows | Other duties (operator's) | |
| 1174 | |---|---|---| |
| 1175 | | `DBIP-` | `IP Geolocation by DB-IP`, linked to `https://db-ip.com` | CC BY 4.0: attribution on every page that shows or uses the data | |
| 1176 | | `GeoLite2-` | `This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com` | The GeoLite EULA, including the 30-day update duty the platform enforces through D1, and terms that pass on to the operator's own users. The docs link the EULA and don't paraphrase all of it. | |
| 1177 | | anything else | none known | The operator's licence for that database | |
| 1178 | |
| 1179 | - **The operator owns compliance.** The operator docs say that plainly, |
| 1180 | next to the publish verb and the feed settings, and link both providers' |
| 1181 | terms. |
| 1182 | - **The app docs** go in `docs/site/cli.md`, the public reference, as a |
| 1183 | new section, "Headers your app receives". No page lists them today |
| 1184 | (survey). |
| 1185 | - App skills are served from carlosframework.com, outside this repo. Plan |
| 1186 | 1 files an amadan discussion asking for the skill text to pick up the |
| 1187 | section. |
| 1188 | - The section must: |
| 1189 | - list the headers and explain that they may be absent and why; |
| 1190 | - cover `Vary`, with the cache cost (§7); |
| 1191 | - say that coordinates are an approximate area; |
| 1192 | - spell out the duty: an app that shows visitors anything derived from |
| 1193 | these headers or the endpoint must carry the attribution that |
| 1194 | `X-Carlos-Geo-Source` names. The endpoint returns the attribution |
| 1195 | text, so an app can render it. |
| 1196 | - **The platform's own pages** that use the data carry the attribution: |
| 1197 | the analytics Countries table, per §9, and the new-app region guess if |
| 1198 | it ever shows the visitor where it thinks they are. |
| 1199 | |
| 1200 | ## 14. Testing |
| 1201 | |
| 1202 | - **`internal/geo`:** |
| 1203 | - `Lookup` against the MaxMind fixtures already in `testdata` (country, |
| 1204 | registered-country fallback, subdivision codes from |
| 1205 | `GeoIP2-City-Test.mmdb`, no point from a Country database); |
| 1206 | - malformed locations; |
| 1207 | - `Locate` regressions (§6); |
| 1208 | - `Source` sanitising; |
| 1209 | - `OpenMMDBTrusted`. |
| 1210 | - **Swap:** `-race`, with 32 goroutines looking up while 100 swaps happen, |
| 1211 | and a `Close` spy proving no close happens while a read lock is held. |
| 1212 | - **Mirror,** against `release.NewDirStore(t.TempDir())`. There is no in-memory store type (survey): |
| 1213 | - first publish, a new sha, the pointer removed, the pointer |
| 1214 | unreadable; |
| 1215 | - a digest mismatch, an oversize blob, a disk write failure; |
| 1216 | - boot with a local generation and no bucket; |
| 1217 | - boot with `current` naming a missing file; |
| 1218 | - a crash injected between each step of §5.3; |
| 1219 | - the old blob deleted between reading the pointer and fetching the |
| 1220 | blob; |
| 1221 | - GeoLite2 expiry at boot and while running. |
| 1222 | - **Edge:** |
| 1223 | - the forgery tests in §8.4; |
| 1224 | - a direct request getting headers; |
| 1225 | - a verified hop adopting the sender's headers with no lookup (a lookup |
| 1226 | spy proves it); |
| 1227 | - none on the door, locally or backhauled; |
| 1228 | - none for loopback; |
| 1229 | - headers present on a WebSocket upgrade; |
| 1230 | - the cache's `Vary` variant selected by country; |
| 1231 | - analytics `Countries` and `GeoSources` filled from `geoResult`. |
| 1232 | - **Publish path:** |
| 1233 | - each refusal in §4.1; |
| 1234 | - publishing the same file twice; |
| 1235 | - X to Y to X; |
| 1236 | - publish racing `off` (the mutex); |
| 1237 | - a foreign pointer write (the `ifMatch` refusal); |
| 1238 | - the sweep; |
| 1239 | - the `PutFrom` fallback size limit. |
| 1240 | - **Feed,** against an `httptest` server standing in for DB-IP and MaxMind: |
| 1241 | - a new edition; |
| 1242 | - a 404 before the month's file appears; |
| 1243 | - an older `build_epoch`; |
| 1244 | - a type mismatch, including on the first publish; |
| 1245 | - a bad SHA-256; |
| 1246 | - a decompression bomb, and a tar with a huge skipped member; |
| 1247 | - a cross-host redirect dropping auth; |
| 1248 | - HEAD-only when `Last-Modified` hasn't changed; |
| 1249 | - 7 consecutive 404s raising the feed reason. |
| 1250 | - **Analytics:** |
| 1251 | - merge across boxes and days; |
| 1252 | - a mixed range of old and new days; |
| 1253 | - attribution per source. |
| 1254 | - **Endpoint:** |
| 1255 | - each status code; |
| 1256 | - role refusals; |
| 1257 | - the rate limit; |
| 1258 | - the override reader's attribution; |
| 1259 | - that no IP ever reaches the log. |
| 1260 | |
| 1261 | ## 15. Rollout |
| 1262 | |
| 1263 | 0. **The bucket lifecycle rule:** |
| 1264 | - `infra/modules/deployment-buckets` gains an |
| 1265 | `aws_s3_bucket_lifecycle_configuration` on the deployment bucket, the |
| 1266 | first one it has. |
| 1267 | - Its single rule is filtered to the `control/geoip/` prefix. It |
| 1268 | expires noncurrent versions after 1 day and aborts incomplete |
| 1269 | multipart uploads after 1 day. |
| 1270 | - It is applied (targeted) before the first GeoLite2 publish. DB-IP |
| 1271 | Lite has no destroy duty, so a DB-IP-only deployment can apply it |
| 1272 | later. |
| 1273 | - For the flagship that means bumping the module ref in |
| 1274 | `carlosframework/carloku-infrastructure`, which needs Paul's go-ahead. |
| 1275 | 1. Merge plan 1 and roll every box. There is no pointer yet, so no box |
| 1276 | sets any header. Every edge strips forged ones. |
| 1277 | 2. Run `carlos system geoip status` until its verdict says every fresh box |
| 1278 | runs a GeoIP-aware binary (§5.5). |
| 1279 | 3. Publish a database by hand (plan 1), or set the feed (plan 2). Check |
| 1280 | that every mirroring box reports the pointer's sha. |
| 1281 | 4. Check a request to a test app carries the headers. Check a forged header |
| 1282 | from outside is replaced, and a door call carries none. |
| 1283 | 5. Publish the app docs. |
| 1284 | |
| 1285 | Plans 2 and 3 roll the console, and plan 2 also rolls the boxes for the |
| 1286 | `geoip` rule. The feed does nothing until `CONSOLE_GEOIP_FEED` is set. |
| 1287 | |
| 1288 | ## 16. Plans |
| 1289 | |
| 1290 | | Plan | Contents | |
| 1291 | |---|---| |
| 1292 | | 1 | The `control/geoip/` lifecycle rule in `infra/modules/deployment-buckets` (applied later, §15); `geo.Lookup`/`OpenMMDBTrusted`/`Swappable`/`Snapshot`; `release.PutStreamer` and its wrappers; the console publish path and `carlos system geoip publish/status/off`; the edge mirror, its local layout and box status; capture/decide/expose ordering, ingress computing, adopting, `X-Carlos-Hop-Geo`, `setGeo`, the rejected-hop rule and the transport-independent tunnel mark (D2); GeoLite2 expiry, including the `CONSOLE_GEOIP_DB` override's (it moves behind a `geo.Swappable`, is refused at startup when already expired, and is closed at expiry, codex R10-1); analytics `Countries`/`GeoSources` through to the console table; operator and app docs | |
| 1293 | | 2 | The feed (DB-IP and MaxMind sources, bounded input, integrity, type and monotonic checks, `feed.json`) and the `geoip` box rule | |
| 1294 | | 3 | The console reader on the mirrored generation, the endpoint, `carlos geo` | |
| 1295 | |
| 1296 | ## 17. Open items for the plans |
| 1297 | |
| 1298 | - **Plan 1, task 1** confirms against real DB-IP Lite and GeoLite2 City |
| 1299 | files: |
| 1300 | - that DB-IP Lite has no subdivision ISO codes; |
| 1301 | - that the country probe's five addresses resolve in both; |
| 1302 | - `Verify`'s peak memory (§4.1). |
| 1303 | - **DB-IP's site says nothing on automated downloads.** Plan 2 fetches at |
| 1304 | most once a day, and only when a new month is due. If DB-IP objects, the |
| 1305 | feed falls back to manual publishing with no code change. |
| 1306 | |
| 1307 | ## 18. Review dispositions |
| 1308 | |
| 1309 | Round 1 (`CODEX-FINDINGS-GEOIP-R1.md`): every finding accepted. |
| 1310 | |
| 1311 | | # | Sev | Disposition | |
| 1312 | |---|---|---| |
| 1313 | | 1 | P1 | Accepted. `setGeo` in every `Rewrite`, after hop-header removal, from the same `geoResult` the cache uses (§8.3) | |
| 1314 | | 2 | P1 | Accepted. One mutex through cleanup, idempotent blobs, a sweep that keeps the current blob (§3) | |
| 1315 | | 3 | P1 | Accepted. The trust model is now ingress-computes, adopt-downstream, so a backhauled door call carries no headers (§8.2) | |
| 1316 | | 4 | P1 | Accepted. D1: GeoLite2 expiry at 30 days, enforced by the edges, console and override; alert at 21 days (§0, §5.3, §10.4, §11.3) | |
| 1317 | | 5 | P2 | Accepted. Immutable generations named by digest, plus an fsynced `current` (§5.2) | |
| 1318 | | 6 | P2 | Accepted. The console's off transition and empty start (§11.3) | |
| 1319 | | 7 | P2 | Accepted. Bounds during the copy, a 409 for a concurrent publish, deadlines, cleanup, a `Verify` memory measurement, and no `Verify` on edges (§4.1, §5.3, §10.3) | |
| 1320 | | 8 | P2 | Accepted. `Opener`/`OpenObject` for reads; `PutStreamer` for uploads, forwarded by every wrapper (§4.1) | |
| 1321 | | 9 | P2 | Accepted. Shared decode; `Locate` unchanged, with regression cases (§6) | |
| 1322 | | 10 | P2 | Accepted. Adopted headers fill `geoResult`, so tunnel analytics count countries (§8.2) | |
| 1323 | | 11 | P2 | Accepted. Pageviews; the read path, API and table are in plan 1; mixed-range test (§9) | |
| 1324 | | 12 | P2 | Accepted. Per-day `GeoSources` provenance; live attribution from the answering reader; the EULA linked, not paraphrased (§9, §11.1, §13) | |
| 1325 | | 13 | P2 | Accepted. `apiSystemRoller`, person-only (§4.2) | |
| 1326 | | 14 | P2 | Accepted. `node_count` in the pointer, comparisons against the etag read, type checked against the feed always (§3, §4.1, §10.3) | |
| 1327 | | 15 | P2 | Accepted. A box rule through the existing pipelines instead of a new console alert (§10.4) | |
| 1328 | | 16 | P2 | Accepted. A status block on every new binary, with a mode and a fleet verdict (§5.5) | |
| 1329 | | 17 | P3 | Accepted. Tunnel boxes have public listeners; the strip covers them (§8.4) | |
| 1330 | | 18 | P3 | Accepted. DB-IP's HTML checksums named and deliberately not scraped (§10.3) | |
| 1331 | | 19 | P3 | Accepted. The cache-eviction cost is documented (§7) | |
| 1332 | | 20 | P3 | Accepted. Coordinates described as an area (§7) | |
| 1333 | | 21 | P3 | Accepted. Lookup probes instead of metadata (§4.1) | |
| 1334 | | 22 | P3 | Accepted. #241 quoted in Appendix A | |
| 1335 | |
| 1336 | Round 2 (`CODEX-FINDINGS-GEOIP-R2.md`): every finding accepted. |
| 1337 | |
| 1338 | | # | Sev | Disposition | |
| 1339 | |---|---|---| |
| 1340 | | R1-4 (partly) | | The S3 noncurrent-version lifecycle rule (§15 step 0); override files are the operator's to destroy (§0, §11.3) | |
| 1341 | | R1-7 (partly) | | A 10-minute deadline on mirror passes; expiry on its own ticker (§5.3) | |
| 1342 | | R1-14 (partly) | | The shrink check applies on a type match, whatever the origin (§4.1) | |
| 1343 | | R1-16 (partly) | | The verdict covers stale and unreadable boxes from 7 days of records (§5.5) | |
| 1344 | | 1 | P1 | Accepted. A rejected hop gets geo `off` (§8.2) | |
| 1345 | | 2 | P1 | Accepted. The tunnel mark needs a TLS dial (§8.4) | |
| 1346 | | 3 | P2 | Accepted. Capture, authenticate, decide, expose (§8.1) | |
| 1347 | | 4 | P2 | Accepted. `Snapshot` under one read lock (§5.4) | |
| 1348 | | 5 | P2 | Accepted. `X-Carlos-Hop-Geo` carries hit/miss/off (§8.1a, §9) | |
| 1349 | | 6 | P2 | Accepted. The pointer delete is unconditional under the single-writer mutex (§3) | |
| 1350 | | 7 | P2 | Accepted. The feed reason is independent of the pointer; an expired pointer is distinct from off (§3, §10.4) | |
| 1351 | | 8 | P3 | Accepted. Immediate clearing, stated (§10.4) | |
| 1352 | | 9 | P3 | Accepted. The off bound is one edge poll plus one console poll (§4.2) | |
| 1353 | |
| 1354 | Round 3 (`CODEX-FINDINGS-GEOIP-R3.md`): every finding accepted. |
| 1355 | |
| 1356 | | # | Sev | Disposition | |
| 1357 | |---|---|---| |
| 1358 | | R1-4 (partly) | | The orphan sweep runs at console startup and daily (§3) | |
| 1359 | | R1-16 (partly) | | The `systemBoxes` inventory, including fleet prefixes (§5.5) | |
| 1360 | | 1 | P1 | Accepted. `genMu` serializes selection, expiry and cleanup; expiry is checked before download and before install; expired shas are remembered (§5.3) | |
| 1361 | | 2 | P2 | Accepted. `Swappable.Locate`, and `Config.GeoIP` holds the `Swappable` (§5.4) | |
| 1362 | | 3 | P2 | Accepted. A coordinate-only `hit` is adopted, and counted as `_unknown` (§8.1a, §9) | |
| 1363 | | 4 | P2 | Accepted. `status.BoxStale` (§5.5) | |
| 1364 | |
| 1365 | Round 4 (`CODEX-FINDINGS-GEOIP-R4.md`): every finding accepted. |
| 1366 | |
| 1367 | | # | Sev | Disposition | |
| 1368 | |---|---|---| |
| 1369 | | R1-16 (partly) | | Reconciles against `control/edges/`; `LoadBoxesComplete` and strict fleet reads; an incomplete read is never ready (§5.5) | |
| 1370 | | 1 | P2 | Accepted. The remote edge skips analytics for tunnelled requests, via a trust-free `viaTunnelKey`; this fixes the existing pageview double count too (§9) | |
| 1371 | | 2 | P2 | Accepted. Upload then a job on the existing system job registry; 202, then the CLI polls (§4.2) | |
| 1372 | |
| 1373 | Round 5 (`CODEX-FINDINGS-GEOIP-R5.md`): every finding accepted. |
| 1374 | |
| 1375 | | # | Sev | Disposition | |
| 1376 | |---|---|---| |
| 1377 | | R1-16 (partly) | | Fleet boxes reconciled via `ListFleetBoxes`; `LoadBoxesComplete` on every prefix (§5.5) | |
| 1378 | | R4-1 (partly) | | The §8.4 test now asserts zero remote observations and one at the hub | |
| 1379 | | 1 | P2 | Accepted. The publish slot is reserved before the body is read, and handed to the job (§4.2) | |
| 1380 | |
| 1381 | Round 6 (`CODEX-FINDINGS-GEOIP-R6.md`): every finding accepted. |
| 1382 | |
| 1383 | | # | Sev | Disposition | |
| 1384 | |---|---|---| |
| 1385 | | R1-16 (partly) | | Explicit retirement by deleting the edges record; a stale expected box blocks, with that instruction; a platform retire verb is filed as a discussion (§5.5) | |
| 1386 | | 1 | P2 | Accepted, through D2: the tunnel mark is transport-independent, so the hub's cache, the app's headers and the analytics always agree (§0, §8.2, §8.4, §9) | |
| 1387 | | 2 | P2 | Accepted. A dedicated staging directory, cleared at console startup (§4.2) | |
| 1388 | |
| 1389 | Round 7 (`CODEX-FINDINGS-GEOIP-R7.md`): every finding accepted. |
| 1390 | |
| 1391 | | # | Sev | Disposition | |
| 1392 | |---|---|---| |
| 1393 | | R6-1 (partly) | | §16 now says the tunnel mark is transport-independent (D2) | |
| 1394 | | 1 | P2 | Accepted. An expired candidate is refused, with no force override, and rechecked before the pointer write (§4.1 step 5) | |
| 1395 | | 2 | P2 | Accepted. `Source` falls back to `mmdb` and is never empty (§6) | |
| 1396 | | 3 | P2 | Accepted. Attributed sources are never folded into `_other` (§9) | |
| 1397 | | 4 | P3 | Accepted. Signatures reconciled with the committed code (§4.1, §6) | |
| 1398 | |
| 1399 | Round 8 (`CODEX-FINDINGS-GEOIP-R8.md`): every finding accepted. |
| 1400 | |
| 1401 | | # | Sev | Disposition | |
| 1402 | |---|---|---| |
| 1403 | | R1-16 (partly) | | Only expected boxes block; a retired box unblocks as soon as its edges record is deleted. The retire verb stays a filed platform discussion, not part of GeoIP (§5.5) | |
| 1404 | | 1 | P2 | Accepted. (namespace, label) identity throughout the verdict (§5.5) | |
| 1405 | | 2 | P2 | Accepted. A re-read of the pointer before install, one immediate re-run, and pass-start timing, so the 11-minute bound holds (§5.3) | |
| 1406 | | 3 | P3 | Accepted. §3 step 2 now requires digest and size | |
| 1407 | |
| 1408 | Round 9 (`CODEX-FINDINGS-GEOIP-R9.md`): every finding accepted. |
| 1409 | |
| 1410 | | # | Sev | Disposition | |
| 1411 | |---|---|---| |
| 1412 | | R1-16 (partly) | | Unreadable records block only through expected identities; the final requirement says "every expected box" (§5.5) | |
| 1413 | | R8-2 (partly) | | 30-second bound on each pointer read; the off bound states it; read failures keep the loaded database and surface in the heartbeat (§5.3) | |
| 1414 | | 1 | P2 | Accepted. Four pointer states: unknown never ready, off/expired requires empty mirrors, live requires the sha (§5.5) | |
| 1415 | | 2 | P2 | Accepted. `off` answers 409 while a publish holds the slot (§4.2) | |
| 1416 | |
| 1417 | Round 10 (`CODEX-FINDINGS-GEOIP-R10.md`): every finding accepted. |
| 1418 | |
| 1419 | | # | Sev | Disposition | |
| 1420 | |---|---|---| |
| 1421 | | R8-2 (partly) | | The bounds read 11.5 and 12.5 minutes, given successful reads (§4.2) | |
| 1422 | | R9-1 (partly) | | The verdict treats a pointer past D1 by the clock as expired (§5.5) | |
| 1423 | | 1 | P2 | Accepted. Override expiry is in plan 1: a `Swappable`, refused at startup when expired, closed by a ticker (§16) | |
| 1424 | |
| 1425 | Round 11 (`CODEX-FINDINGS-GEOIP-R11.md`): accepted. |
| 1426 | |
| 1427 | | # | Sev | Disposition | |
| 1428 | |---|---|---| |
| 1429 | | 1 | P2 | Accepted. The alert's expired reason is marked or past D1 by the clock, using pointer metadata the mirror keeps after dropping the reader; tests cover delayed marking, restart, fresh publish and off (§10.4; plan 2 Task 4) | |
| 1430 | |
| 1431 | ## Appendix A. The request (#241, verbatim) |
| 1432 | |
| 1433 | > **Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint)** |
| 1434 | > |
| 1435 | > ## Why |
| 1436 | > |
| 1437 | > The nearest-region-default spec (`docs/superpowers/specs/2026-10-01-nearest-region-default-design.md`, §6, on branch `edge-replica-placements`) gives the console an optional GeoIP reader: `CONSOLE_GEOIP_DB` names an operator-supplied `.mmdb` (DB-IP Lite or MaxMind GeoLite2), read through `github.com/oschwald/maxminddb-golang`, mapped and opened once per process. Today it serves one caller: guessing a new app's region from the requester's IP. |
| 1438 | > |
| 1439 | > Once the platform carries that database, every app on it is one lookup away from "roughly where is this visitor", and none of them should have to ship their own `.mmdb`, handle attribution, or track database updates. The platform should expose the lookup to apps. |
| 1440 | > |
| 1441 | > ## What |
| 1442 | > |
| 1443 | > A platform geolocation facility that any app can use, backed by the same database and reader the console already has. Two shapes are worth weighing; the agent picking this up should choose one (or both) and say why in the spec: |
| 1444 | > |
| 1445 | > 1. **Router-injected request headers.** The edge/router does the lookup on the client IP and sets headers on the proxied request, e.g. `X-Carlos-Geo-Country`, `X-Carlos-Geo-Region`, `X-Carlos-Geo-Lat` / `-Lon`, `X-Carlos-Geo-Source`. Zero integration work for apps; works in every language; covers the common case (country/currency/locale defaults, "nearest region" hints). Cost: a lookup on every request unless cached per connection or per IP. |
| 1446 | > 2. **A lookup endpoint.** A platform-internal HTTP endpoint apps can call with an arbitrary IP (`GET /geo?ip=…` on the sidecar/hop path, authenticated the way other platform-internal calls are). Needed for anything that isn't "the current request": batch jobs, IPs from logs, webhooks carrying a client IP. |
| 1447 | > |
| 1448 | > Headers alone probably cover 90% of app needs; the endpoint covers the rest. Shipping headers first and the endpoint as a follow-on is a reasonable sequence. |
| 1449 | > |
| 1450 | > ## Constraints to carry over from the spec |
| 1451 | > |
| 1452 | > - Unset `CONSOLE_GEOIP_DB` means the feature is off: headers absent / endpoint returns "unavailable", never a guess. Unopenable database: logged once, feature off, platform still starts. |
| 1453 | > - Never look up loopback, private, link-local, CGNAT or unspecified addresses. |
| 1454 | > - Lookup errors are a miss, logged at most once a minute. |
| 1455 | > - Database replacement is an atomic rename plus restart; whatever process owns the lookup needs the same lifecycle. |
| 1456 | > - Attribution: DB-IP Lite (CC BY 4.0) requires "IP Geolocation by DB-IP" with a link on pages that show the data. If apps surface the data, the platform docs must tell them about that obligation, and the platform's own pages that use it must carry it. GeoLite2 has its own EULA terms (attribution, keeping the data updated). The operator owns compliance; the docs should say so plainly. |
| 1457 | > - The reader lives in (or is being added to) the console. If the router or a sidecar needs it, the reader should move to a shared internal package rather than being duplicated. |
| 1458 | > |
| 1459 | > ## Open questions |
| 1460 | > |
| 1461 | > - Where does the lookup physically run: router, console, sidecar? Which one actually sees the client IP, and which one can memory-map the database? Only one place should open the file. |
| 1462 | > - Header naming and whether to include coordinates or just country/subdivision. Coordinates make "nearest X" trivial for apps but are more than most need. |
| 1463 | > - Should apps be able to opt out (an app config flag) to avoid the per-request cost when they don't care? |
| 1464 | > - Does `carlos` CLI need anything (e.g. `carlos geo <ip>` for debugging what the platform would return)? |
| 1465 | > |
| 1466 | > ## Related |
| 1467 | > |
| 1468 | > - Nearest-region default spec, §6 (IP lookup), §14 (region table). Build on its reader; do not add a second one. |
| 1469 | |
| 1470 | How rev 2 answers the request's constraints: |
| 1471 | |
| 1472 | - **"Unset means off":** the off switch is now an absent pointer (R1). |
| 1473 | `CONSOLE_GEOIP_DB` keeps its meaning for the console. |
| 1474 | - **"Atomic rename plus restart":** replaced by the hot swap for the edge |
| 1475 | and console mirror (R1). The override keeps the rename-plus-restart rule. |
| 1476 | - **"Only one place should open the file":** each process that serves |
| 1477 | lookups opens it once (one edge per box, plus the console). One file |
| 1478 | per box is mapped by every reader on that box, and the bucket holds the |
| 1479 | single source. |
| 1480 | - **"Header naming":** `X-Carlos-Geo-Subdivision` replaces the request's |
| 1481 | example `-Region`, because "region" already means an AWS region on this |
| 1482 | platform. |
| 1483 | |
amadan