amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint

Date: 2026-10-06 Revision: 13 Discussion: amadan carlos/platform #241 (quoted in Appendix A) Branch: geoip-for-apps

  • Rev 1 (5e1129be) records the design Paul approved in conversation on 2026-10-06.
  • Rev 1 went to Astra (GPT-6, via codex). The findings are in CODEX-FINDINGS-GEOIP-R1.md: 22 findings, 4 of them P1. Every one was checked against the code and accepted. Changes are marked "(codex R1-N)", and §18 records each disposition.
  • Rev 2 changes the trust model (codex R1-3, R1-10). The edge that accepts the visitor's connection computes the headers. Downstream edges trust them only over a verified hop or the tunnel. Rev 1 had the home edge recompute, which located the sending box for a backhauled door request.
  • Rev 2 makes one decision Paul has not seen, D1 in §0: GeoLite2 databases expire 30 days after they were built.
  • Rev 3 answers Astra's round 2 (CODEX-FINDINGS-GEOIP-R2.md). Of round 1's findings, 18 were resolved and 4 partly resolved. Round 2 adds 9 new findings, 2 of them P1. Every one was checked and accepted, and changes are marked "(codex R2-N)". - Rev 3 adds an internal trust header, X-Carlos-Hop-Geo. It carries the ingress edge's state (hit, miss, off) across a hop or the tunnel, and no app ever sees it. - It withholds geo on a rejected hop (and, until rev 7, on a plaintext tunnel; see D2). - It adds a bucket lifecycle rule for GeoLite2's destroy duty.
  • Rev 4 answers Astra's round 3 (CODEX-FINDINGS-GEOIP-R3.md). That round resolved all 9 of round 2's findings, left 2 earlier findings partly resolved, and raised 4 new ones, 1 of them P1. Every one was accepted, and changes are marked "(codex R3-N)". - Rev 4 also folds in a code survey made for plan 1, marked "(survey)": - the console's unit isn't in the repo; - nothing creates /data/carlos yet; - the existing staleness rule; - the console's 5-minute read timeout; - which boxes hold a store; - there is no in-memory store type; - there is no in-repo page listing the headers apps receive.
  • Rev 5 answers Astra's round 4 (CODEX-FINDINGS-GEOIP-R4.md). That round resolved all of round 3, left 1 earlier finding partly resolved, and raised 2 new P2s. All were accepted, marked "(codex R4-N)". - Publishing becomes an upload followed by a job. - A tunnelled request is counted once, by the hub. - The rollout verdict reconciles against the edges records.
  • Rev 6 answers Astra's round 5 (CODEX-FINDINGS-GEOIP-R5.md): 1 new P2 and 2 partly resolved findings, all accepted, marked "(codex R5-N)". - The upload slot is reserved before any body is read. - Fleet boxes are reconciled through the registry. - The tunnel analytics test is corrected.
  • Rev 7 answers Astra's round 6 (CODEX-FINDINGS-GEOIP-R6.md), marked "(codex R6-N)". - It takes D2, which reverses round 2's TLS-only tunnel trust. - Uploads stage in a directory the console clears at startup. - A box is retired explicitly, by deleting its edges record.
  • Rev 8 answers Astra's round 7 (CODEX-FINDINGS-GEOIP-R7.md), marked "(codex R7-N)": - an expired database is refused at publish; - a source can never sanitise to empty; - attributed sources are never folded into _other; - the spec's signatures now match the committed code.
  • Rev 9 answers Astra's round 8 (CODEX-FINDINGS-GEOIP-R8.md), marked "(codex R8-N)": - box identity is (namespace, label); - a download superseded mid-flight is never installed, and the poll clock runs from the start of each pass; - §3 says digest, not size; - only expected boxes can block the verdict.
  • Rev 10 answers Astra's round 9 (CODEX-FINDINGS-GEOIP-R9.md), marked "(codex R9-N)": - the verdict knows four pointer states; - off answers 409 during a publish instead of waiting; - pointer reads are bounded at 30 seconds, and the off bound includes them; - an unreadable heartbeat blocks only through the expected box it hides.
  • Rev 11 answers Astra's round 10 (CODEX-FINDINGS-GEOIP-R10.md): - the CONSOLE_GEOIP_DB override's D1 expiry moves into plan 1; - the off bounds read 11.5 and 12.5 minutes (user copy rounds them up to 12 and 13); - the verdict judges expiry by the clock.
  • Rev 12 answers Astra's round 11 (CODEX-FINDINGS-GEOIP-R11.md). The alert's expired reason now matches §5.5 (marked, or past D1 by the clock), so expiry never clears an alert while the bucket is still unmarked.
  • Astra's round 12 (CODEX-FINDINGS-GEOIP-R12.md) accepted rev 12 with no findings.
  • Rev 13 records Paul's rulings of 2026-10-07: - he accepts D1 and D2 (now R7 and R8); - the free DB-IP Lite is the default, and operators (the flagship included) may upgrade to paid MaxMind through two more feeds, geoip2-city and geoip2-country (R9).

0. Rulings and decisions

Paul's answers during brainstorming. Do not reopen them without him.

  • R1. The database reaches the edges from the bucket. The operator (or the feed, R6) publishes once. Every edge mirrors it and swaps its reader in place, with no restart. This replaces #241's "atomic rename plus restart" rule for the edge. CONSOLE_GEOIP_DB keeps that rule as an override (§11.3).
  • R2. Four headers: country, subdivision, coordinates and source (§7).
  • R3. Tunnel and fleet boxes trust the hub. The hub computes the headers. The remote edge keeps them only on requests that arrived over the authenticated tunnel (§8). Remote boxes hold no database.
  • R4. No per-app switch. Headers go to every app whenever a database is loaded. A switch can ride the telemetry settings pipeline later if someone asks for one.
  • R5. One spec, three plans: headers first, then the feed, then the endpoint (§16). Analytics countries ship in plan 1.
  • R6. The database updates itself. The console fetches new editions of DB-IP Lite or GeoLite2 and publishes them through the same path as a manual publish (§10).

Decisions taken during review. Paul hasn't seen these yet; each is easy to reverse.

  • D1 (codex R1-4). GeoLite2 databases expire 30 days after their build_epoch. - Why: MaxMind's GeoLite EULA requires the licensee to stop using a superseded edition within 30 days of an update's release. MaxMind updates GeoLite2 twice a week. A database 30 days past its build has certainly been superseded, and counting from the build rather than from the release of its successor errs on the strict side. - What happens: - At expiry every edge and the console drop the reader and delete their local copies. - The console deletes the blob and marks the pointer "expired": true (§3). - The deployment bucket is versioned, so a deleted blob survives as a noncurrent version. A lifecycle rule on control/geoip/ expires noncurrent versions after 1 day (§15, codex R2 on R1-4). - A CONSOLE_GEOIP_DB override file belongs to the operator. The platform closes it at expiry and logs that it must be replaced, but never deletes it. - The box alert (§10.4) warns from day 21. - When it matters: with the feed working, a GeoLite2 database never gets near 30 days. Only a dead feed or a manual publish ever reaches the limit. - DB-IP Lite has no such duty (CC BY 4.0) and never expires. It only alerts as stale.
  • D2 (codex R6-1, reversing the R2-2 answer). Every request served over the tunnel carries the hub's answer, TLS or not. - Round 2 withheld geo trust from plaintext tunnels, because nothing authenticates the hub to the box there. - That protects nothing. An on-path attacker who can pose as the hub on a plaintext tunnel can already inject any request at all: method, path, body, cookies, X-Forwarded-For. Geo headers add nothing to what such an attacker controls. - Withholding them did cost something. The hub's cache stored variants under a country the app never received, and those variants were replayed after the box reconnected over TLS. The hub's analytics counted a country that §9's plaintext rule said shouldn't exist. - So one mark, set on every request tunnelclient serves, means "the hub computed this request's answer". The remote adopts that answer and doesn't observe the request for analytics. - Plaintext tunnels remain what tunnelclient.Dial already calls them: "for tests". The operator docs say production tunnels use https:// (infra/remote-host.md already does). - Because the hub's answer always reaches the app, the variant the hub caches always matches what the app saw, whatever the transport, on reconnect and on fleet fallback.
  • R7, R8 (Paul, 2026-10-07): D1 and D2 above are accepted as they stand.
  • R9 (Paul, 2026-10-07): free by default, paid by choice. - The documented default feed is dbip-city-lite. DB-IP Lite is free, CC BY 4.0, and allows commercial use with the attribution the platform already shows. - Operators, the flagship included, may upgrade to MaxMind's paid databases with geoip2-city or geoip2-country. These use the same download API and credentials as GeoLite2. - Paid editions are not GeoLite2, so D1's expiry does not apply, and the platform shows no attribution for them: the operator's MaxMind licence governs. - The feed stays off until CONSOLE_GEOIP_FEED is set, so no deployment starts downloading, or takes on a licence, without its operator choosing to. - GeoLite2 remains supported, but the operator docs note that MaxMind calls it not recommended for commercial use and that its EULA limits redistribution, which serving answers to customer apps may engage.

1. Why, and what this is not

The nearest-region work gave the console a GeoIP reader (internal/geo, CONSOLE_GEOIP_DB). It serves one caller: guessing a new app's region. Once the platform carries the database, every app should be able to ask "roughly where is this visitor" without shipping its own .mmdb, handling attribution, or tracking updates. Appendix A quotes the request.

Goals:

  • Every proxied request to an app carries the visitor's country, and where the database has them, subdivision and coordinates.
  • Apps can look up an arbitrary IP (batch jobs, logs, webhooks).
  • The platform's own analytics fill the countries map the analytics spec reserved for this (2026-09-07-errors-and-analytics-design.md).
  • The operator publishes or configures a feed once per deployment, not once per box.

Not goals:

  • Accuracy beyond the database. The headers say what the database says.
  • Header injection for traffic that never crosses the edge. Scheduler calls go straight to the socket. Sidecars reach their instance through loopback.
  • ASN, ISP, timezone, city-name or accuracy-radius headers. Additive later if asked for.
  • A second reader. Everything builds on internal/geo.

2. Overview

operator ──carlos system geoip publish──┐
                                        ▼
feed (console, daily) ──────────► console publish path (one at a time)
                                        │  verify, upload blob, flip pointer
                                        ▼
                 bucket: control/geoip/current.json  →  control/geoip/<sha256>.mmdb
                                        │
                     every mirroring edge polls the pointer (10 min)
                                        ▼
        /data/carlos/geoip/<sha256>.mmdb + current  →  geo.Swappable (hot swap)
                                        │
   ingress edge: strip X-Carlos-Geo-*, look up the visitor, keep the result
                 on the request; downstream edges (verified hop, tunnel)
                 adopt the validated headers instead of looking up
                         │                                  │
                         ▼                                  ▼
       every proxy's Rewrite sets the headers      analytics Obs.Country
       on the outbound request                     → Day.Countries
console: reads the same mirrored file  →  GET /api/cli/apps/{a}/{app}/geo?ip=
                                         carlos geo <ip>

3. Bucket records

The console is the only writer of control/geoip/. Edges only read it. A deployment runs one console, and inside it a single mutex serializes every publish, off and expiry through to the end of its cleanup (codex R1-2).

The blob is control/geoip/<sha256>.mmdb. Its name is the hex SHA-256 of its bytes, so a blob never changes after it is written.

The pointer is control/geoip/current.json:

{
  "v": 1,
  "sha256": "9f2c…",
  "key": "control/geoip/9f2c….mmdb",
  "size": 131072000,
  "database_type": "DBIP-City-Lite",
  "build_epoch": 1790812800,
  "node_count": 7342011,
  "published": "2026-10-06T21:00:00Z",
  "origin": "feed:dbip-city-lite",
  "edition": "2026-10"
}
  • database_type, build_epoch and node_count come from the file's own MMDB metadata. node_count is recorded so the shrink check (§4.1) can compare against the current pointer without reading the current blob (codex R1-14).
  • origin is manual or feed:<name>. edition is the feed's own edition marker: the YYYY-MM for DB-IP, the Last-Modified value for MaxMind, and empty for a manual publish.
  • No pointer means the feature is off. An unknown v is treated as no pointer, logged once.
  • An expired pointer ("expired": true, set only by D1) means "a database was published and has run out". Edges load nothing, and the geoip rule keeps firing (§10.4) until a fresh publish or an explicit off. Expiry is not the same as off, so it never clears an alert (codex R2-7).

Publishing, holding the mutex:

  1. Read the pointer and its etag. Every comparison in §4.1 and §10.3 is against this read.
  2. Upload the blob, create-only. If an object with that name already exists, stream it and check its SHA-256 and size. Only if both match is it the same bytes, and publishing carries on: publishing the same file twice is not an error. Anything else is refused, and the previous pointer and blob are left untouched (codex R8-3).
  3. Write the pointer with ifMatch set to the etag from step 1. A refusal means something outside this console wrote the pointer. Stop, and leave the new blob for the sweep.
  4. Sweep: delete every control/geoip/*.mmdb other than the one the new pointer names. This removes the previous blob, the leftovers of any earlier failure, and never the current one, even in an X to Y to X sequence.

off and expiry take the same mutex.

  • off deletes the pointer, then sweeps every blob.
  • Expiry rewrites the pointer with "expired": true (with ifMatch), then sweeps every blob.

Store.Delete has no condition. The pointer delete is therefore unconditional, and its safety rests on the single-writer rule above: one console, one mutex. Rev 3 drops the conditional delete that rev 2 asked for, rather than invent a store operation (codex R2-6).

An edge that read the old pointer just before a publish can find its blob already deleted. It keeps what it has loaded and retries on the next poll, when it reads the new pointer.

The orphan sweep (codex R3 on R1-4): a console that crashed after uploading a blob but before writing the pointer leaves a blob that is still a current version, which the noncurrent-version lifecycle rule (§15) doesn't touch. So the console runs the §3 sweep, holding the mutex, at startup and once a day. Deleting the orphan turns it into a noncurrent version, and the lifecycle rule then removes it.

4. Publishing

4.1 The console's publish path

geoipPublish(ctx, src io.Reader, origin, edition string, force bool), used by both the CLI verb and the feed:

  1. Refuse with 409 if another publish holds the mutex. Don't queue.
  2. Stream src to a fresh temporary directory under the console's data directory, hashing as it goes. - Refuse once 1 GiB has been read. The limit is enforced during the copy, not afterwards (codex R1-7). - The whole operation has a 15-minute deadline. - The temporary directory is removed on every exit path.
  3. Open it with geo.OpenMMDB, which verifies it.
  4. Country probe (codex R1-21): look up 8.8.8.8, 1.1.1.1, 9.9.9.9, 208.67.222.222 and 2001:4860:4860::8888. Refuse the database unless at least three of them return a country. That catches an ASN database or a schema the decoder can't read.
  5. Expiry check (codex R7-1): refuse a database that D1 already counts as expired. force never overrides this. The check runs again immediately before the pointer is written, because a long upload can cross the expiry. A refusal leaves the previous pointer and blob untouched.
  6. Shrink check: when the new database has the same database_type as the current pointer's, refuse it if it has fewer than half that pointer's node_count. This guards against a truncated file. The check is skipped when the types differ, because node counts across products aren't comparable. A manual publish can override the check with force. The feed can't (codex R2 on R1-14).
  7. Publish (§3).

Plan 1, task 1 measures Verify's peak memory on a real DB-IP City Lite file and a real GeoLite2 City file, and records the result in the plan. Verify builds a map of data offsets, so it can cost much more than the file's size. If the peak would push the console box past its memory headroom, the plan says so and proposes a limit (codex R1-7).

Streaming upload (codex R1-8). Store.Put takes the whole body as []byte. Plan 1 adds one optional interface. Its body is an io.ReadSeeker, because the publish path always streams from its staged file, and the AWS SDK signs a seekable body without buffering it (codex R7-4):

// release.PutStreamer is implemented by stores that can upload from a seekable reader.
type PutStreamer interface {
	PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (etag string, err error)
}
  • It is implemented by S3Store (PutObject with a reader body and a content length), DirStore (temp file and rename) and prefixStore (rebase, then forward).
  • Plan 1 lists every other wrapper around the console's store and makes each one forward PutFrom the way it forwards Opener.
  • A store without PutFrom takes the Put fallback only for files up to 64 MiB. Anything larger is refused with a clear error.

Streaming reads use the existing release.Opener / OpenObject, which S3, DirStore and prefixStore already implement. No new read interface is added.

4.2 Operator verbs

Under carlos system, through the console API:

  • Who may use them (codex R1-13): a person, not a token, who apiSystemRoller (internal/console/apisystemupdate.go) admits. That is an owner of the ops account, or an address on CARLOS_SYSTEM_OPERATORS. These are the people who can already roll the platform. The antivirus doors use apiSystemOperator, which checks CARLOS_STORE_OPERATORS instead, so they are not the precedent for this gate.
  • carlos system geoip publish <file.mmdb> [-force] streams the file to the console, which runs §4.1. - Upload, then a job (codex R4-2): - The CLI sends a raw application/octet-stream body with a Content-Length, through PUT /api/cli/system/geoip. - The console's server-wide ReadTimeout is 5 minutes. So the handler extends its own read deadline to 15 minutes with http.NewResponseController, the way the ship door does (handleAPIShip, survey), and wraps the body in MaxBytesReader at 1 GiB. - One slot, reserved first (codex R5-1). The handler takes the publish mutex (TryLock) before reading any of the body. If it is held, by another upload still streaming or by a job, the answer is 409 straight away. So at most one upload is ever staged on the console's disk. - The mutex and the temporary directory pass from the handler to the job. The job releases the mutex only after removing the directory. - If the upload fails or is cut off before the job starts, the handler removes the directory and releases the mutex itself. - A test stalls a first upload's body and checks that a second upload gets 409 without any of its body being read. - The handler only stages the upload: it streams it to the publish path's temporary file and hashes it. - The staging directory is <console data dir>/geoip-staging/, next to CONSOLE_DB, and used for nothing else. The console clears it at startup, before it admits any upload or starts the feed, so a crash during staging or during a job never leaves its 1 GiB behind for good (codex R6-2). A test kills the publisher mid-stage and mid-job, then checks that a fresh publisher on the same directory starts empty and accepts a retry. - It then answers 202 with a job id straight away. A request that came through another edge's backhaul has 30 seconds after its body to see response headers (backhaulHeaderTimeout), and verifying plus uploading can take longer. - The job runs §4.1 steps 3 to 6 under the publish mutex and the 15-minute deadline. - It reuses the console's existing system job registry (s.systemJobs, polled at GET /api/cli/system/jobs/{id}, the door carlos system update already uses). - The CLI polls until the job is done and prints the result. - A second upload while a job holds the mutex is answered 409 before its body is read. - A test publishes through a real backhaul, with the job delayed past the header timeout. - The existing PUT /api/cli/system/upload holds the whole body in memory, so it isn't reused.
  • carlos system geoip status shows: - the pointer (type, edition, build date, age, origin, expiry for GeoLite2); - the feed's configuration and its last result; - the per-box report from §5.4.
  • carlos system geoip off deletes the pointer, then sweeps the blobs (§3). While a publish holds the slot (an upload streaming in, or its job), off answers 409 straight away rather than waiting behind it (codex R9-2). The operator runs it again once the publish finishes. Every edge drops its reader within one poll: at most 11.5 minutes, with jitter and one bounded pointer read, while pointer reads succeed (§5.3). The console follows its own box's edge within a further minute, so it takes at most 12.5 minutes (§11.3, codex R2-9, R10). User-facing copy rounds these up to 12 and 13 minutes. If CONSOLE_GEOIP_FEED is set, off warns that the feed will publish again within a day and says to unset it.

5. The edge mirror

5.1 Where and who

  • The mirror runs inside carlos agent's edge when the edge has a store (e.staticStore, set from CARLOS_DEPLOYMENT_BUCKET or CARLOS_DEPLOYMENT_DIR) and no tunnel client is configured. Its mode is mirror (survey).
  • It does not run on boxes that use the tunnel client (CARLOS_TUNNEL_* set). Those boxes take the hub's headers (§8), and their mode is tunnel.
  • A box with no store and no tunnel has mode none. It strips forged headers, as every edge does, and sets none.
  • Files live under /data/carlos/geoip/, inside the edge unit's ReadWritePaths=/data. - Nothing creates /data/carlos today (survey). The mirror creates the directory itself at mode 0755, and the files are 0644. The data isn't secret. - Bootstrap makes /data mode 0750, owned carlos:carlos. The console already writes /data/registry.db and /data/console/console.db, so whatever user its hand-installed unit runs as can already get into /data (the unit isn't in the repo, survey). Step 2 of the rollout (§15) checks on the console box that the console can read /data/carlos/geoip/current.

5.2 Local layout (codex R1-5)

Each generation is immutable and named by its digest:

/data/carlos/geoip/<sha256>.mmdb    one per generation, written once
/data/carlos/geoip/current          {"sha256": "...", "database_type": "...", "build_epoch": ...}
/data/carlos/geoip/.incoming-*      downloads in progress

current is the only file that changes. It is written to a temporary file, fsynced, renamed over current, and then the directory is fsynced. A reader that sees current naming a sha always finds that generation complete, because the generation was renamed into place before current was written. A crash at any step leaves either the old current with the old generation, or the new current with the new one.

5.3 The loop

  • At boot: 1. Read current and open <sha>.mmdb with geo.OpenMMDBTrusted. That opens the file without Verify (§6), because the console verified those exact bytes and the name proves they are the same. 2. Delete any other *.mmdb and any .incoming-*. 3. Serving never waits on the bucket. 4. A current that names a missing or unopenable file is removed and logged, and the edge starts with no database. 5. A GeoLite2 generation that has passed its expiry (D1) is deleted, not opened.
  • One lock for selection (codex R3-1). Choosing the generation, expiring it and cleaning up the local files all happen under one mutex, genMu. That covers writing current, Swap, and deleting generations. Network reads and downloads happen outside it, so a stuck download never holds it.
  • A finished download is installed only if the pointer still names it (codex R8-2). - After the download the mirror re-reads the pointer. If the pointer now names something else, is gone, or has expired, the download is deleted and the pass re-runs once at once. So an off or a newer publish that landed mid-download takes effect at the end of that pass. - If the re-read fails, nothing is installed until the next pass. - Each pass is due 10 minutes plus jitter after the previous one started, and a pass is bounded at 10 minutes. - Every pointer read has its own 30-second timeout (codex R9 on R8-2). A change to the pointer is therefore acted on within 11 minutes plus one pointer read (at most 11.5 minutes), even with a long download in flight. That is the bound §4.2 quotes. - While pointer reads keep failing, the edge keeps what it has loaded, and its heartbeat reports the error, so the verdict shows the box as not caught up.
  • Every 10 minutes, with up to 1 minute of jitter, read the pointer (bounded read, at most 64 KiB). Each pass has a 10-minute deadline, covering the pointer read and any blob download. A pass that misses it is abandoned and its .incoming-* deleted (codex R2 on R1-7). The GeoLite2 expiry check runs on its own one-minute ticker, with no network access, so a stuck download can't delay it.
  • An expired candidate is never installed. - A pointer whose build_epoch is past D1's expiry (it carries database_type and build_epoch) is treated as an expired pointer before anything is downloaded. - The check runs again under genMu, immediately before installing, against the downloaded file's own metadata. A download that crossed its expiry is deleted, not installed. - The mirror remembers the expired sha and doesn't fetch it again until the pointer names another one. An expired pointer the console couldn't mark can therefore never come back. - The pointer names the sha already loaded: nothing. - It names a new sha: 1. Stream the blob through OpenObject into .incoming-<sha>. Stop and delete it if it exceeds the pointer's size or 1 GiB, or if the disk write fails. 2. Check the digest. 3. Fsync and rename it to <sha>.mmdb. 4. Write current (§5.2), open the generation and swap the reader (§5.4). 5. Delete the previous generation. - No pointer, or an expired one: write the empty state to current, swap in a nil reader, then delete the generations. - The read fails (network, permissions): keep what is loaded, and log once an hour while it keeps failing.
  • A blob whose digest doesn't match is deleted, logged once per sha, and not retried until the pointer changes.
  • GeoLite2 expiry (D1): the expiry ticker takes genMu, then checks the currently loaded generation's build_epoch. Past expiry it swaps in nil and deletes that generation, whatever the pointer says. - Because it checks after taking the lock, it acts only on the generation that is current at that moment. It can never inspect an expired A and then remove a freshly installed B. - Tests: - a download crossing expiry; - repeated polls against an expired pointer that was never marked; - an install racing the ticker.

5.4 Hot swap

geo.Swappable wraps the reader:

type Swappable struct {
	mu  sync.RWMutex
	cur *MMDB
}
// Snapshot is one answer from one reader: the result, that reader's
// source, and whether any reader was loaded at all (codex R2-4).
type Snapshot struct {
	Result Result
	Found  bool
	Source string // empty only when Loaded is false
	Loaded bool
}
func (s *Swappable) Lookup(ip net.IP) Snapshot        // one RLock covers lookup and source
func (s *Swappable) Locate(ip net.IP) (Point, bool)    // geo.Locator; RLock held through MMDB.Locate; a miss when empty
func (s *Swappable) Swap(next *MMDB)                  // Lock, replace, Unlock, then Close the old one

The headers, analytics and the endpoint's attribution all come from one Snapshot, so a swap between the lookup and the response can't label DB-IP data as GeoLite2. There is deliberately no separate Source() on Swappable.

Swappable implements geo.Locator, so the console's Config.GeoIP holds the Swappable itself (codex R3-2). Region guesses (internal/console/regionguess.go) never see the *MMDB underneath and can't outlive a swap. The -race test runs concurrent Locate calls as well as Lookup calls.

  • The old mapping is closed only after the write lock has been taken and released. No lookup can be in flight on an unmapped file.
  • A lookup holds the read lock for a few microseconds, so a swap waits that long at most.
  • Go's RWMutex blocks new readers while a writer waits, so a steady stream of lookups can't starve the swap.
  • The swap gets a -race test with concurrent lookups (§14).

5.5 Reporting (codex R1-16)

Each box's status heartbeat gains a block on every box running a binary with this feature: a new GeoIP field in status.BoxMetrics (internal/status/snapshot.go), which BuildFleet never copies into the public fleet.json. Platform boxes write status/boxes/<label>.json, and fleet boxes write under fleets/<fleet>/status/boxes/:

"geoip": {"mode": "mirror", "sha256": "9f2c…", "type": "DBIP-City-Lite", "error": ""}
"geoip": {"mode": "tunnel"}
"geoip": {"mode": "none"}

carlos system geoip status reconciles two lists (codex R4 on R1-16).

Box identity is (namespace, label) (codex R8-1). The namespace is the platform ("") or a fleet's name. Fleet labels repeat across fleets and can equal a platform label, so every match, every expectation and every blocking line uses the pair. Lines name a fleet box as <fleet>/<label>.

What it reads:

  • Expected boxes: - every platform box that publishes an edges record (control/edges/<box>.json, what the home map reads); - every active fleet box the registry lists (ListFleetBoxes). Fleet boxes don't publish edges records (internal/host/edgesfile.go), so they are reconciled against the registry instead (codex R5 on R1-16).
  • Heartbeats: every prefix (the platform's and each fleet's fleets/<name>/status/boxes/) is read with status.LoadBoxesComplete, which says when any record couldn't be read. Neither LoadBoxes nor systemBoxes's strict mode is used, because both skip unreadable records silently.

What the pointer asks of every mirror (codex R9-1):

| Pointer | Each expected mirroring box must report | |---|---| | live | the pointer's sha | | absent (off), or expired (marked, or past D1 by the clock, codex R10) | no generation loaded | | unreadable | nothing satisfies this: the verdict is never ready while the pointer can't be read |

Which boxes are listed, and which can block:

  • Every expected box is listed whatever its age, and only expected boxes can block the verdict (codex R8 on R1-16).
  • A box with a heartbeat but no edges record or fleet registration is listed, for information, unless its last heartbeat is more than 7 days old. It never blocks. So deleting a retired box's edges record (or detaching a fleet box) unblocks the verdict at once.

It reports:

  • stale, by the shared rule status.BoxStale (3 heartbeat periods, 15 minutes), not a threshold of its own (codex R3-4). The verdict can't vouch for a box it can't currently see.
  • record unreadable: the box is missing from what was read, so if it is expected it blocks as "no heartbeat". An unreadable record for a box nobody expects blocks nothing (codex R9 on R1-16). If the list of expected fleet boxes itself can't be read, that always blocks.
  • expected box with no heartbeat at all: "no heartbeat".
  • no block: "old binary". The headers can't be trusted while any such box is serving.
  • mode: tunnel: takes the hub's headers.
  • mode: none: no store and no tunnel, so no headers. Safe, because it strips forged ones.
  • mode: mirror: shows the box's sha against the pointer's.

Retiring a box (codex R6 on R1-16). Nothing deletes a retired box's edges record today (internal/console/instancesplacement.go, "nothing deletes a retired box's file"). A stale heartbeat is never taken as retirement.

  • An expected box whose heartbeat is stale blocks the verdict, with the instruction: "no heartbeat since <date>. if this box is retired, delete its edges record (control/edges/<label>.json)".
  • Deleting that record is the explicit retirement. It also stops the home map backhauling traffic to a dead box.
  • A carlos system verb to retire a box belongs to the platform, not to GeoIP. Plan 1 files it as a discussion rather than working around its absence.

It ends with a one-line verdict. "Ready" means the pointer was read, the expected-box list was complete, and every expected box is fresh, readable and GeoIP-aware, with every expected mirroring box holding what the pointer asks for (table above). Anything else names the boxes in the way (codex R2 on R1-16). A test sets a box's heartbeat just inside the next jittered tick and checks it still reads as fresh.

6. internal/geo changes

Locate keeps its exact behaviour (codex R1-9). Both methods share one decode and validation step:

type Result struct {
	Country     string // ISO 3166-1 alpha-2; country, else registered country
	Subdivision string // ISO 3166-2 like "GB-SCT"; empty unless the database has an ISO code
	Point       Point
	HasPoint    bool   // true only for a real location record, never a centroid
}
func (m *MMDB) Lookup(ip net.IP) (Result, bool)
func (m *MMDB) Source() string            // metadata.DatabaseType, reduced to [A-Za-z0-9._-], at most 64 bytes; "mmdb" if that leaves nothing
func (m *MMDB) BuildEpoch() time.Time     // UTC; the pointer stores BuildEpoch().Unix()
func (m *MMDB) NodeCount() uint64
func OpenMMDBTrusted(path string) (*MMDB, error) // opens without Verify
  • The source is never empty (codex R7-2). A database type that sanitises to nothing becomes mmdb, which satisfies the header grammar (§8.4). A loaded reader therefore always has a source that crosses a hop or the tunnel intact, and a local hit is never turned into off downstream.
  • Shared: decode(ip) (mmdbRecord, bool) applies the Locatable filter, To4 normalisation and the fixed-category error log, and treats a malformed location as a miss of the whole record.
  • Locate: unchanged. A valid point succeeds even without a country. Otherwise it tries the country's representative point, then the registered country's. Today's tests stay, and plan 1 adds regression cases for "point with no country" and "country with no representative point but a registered country that has one".
  • Lookup: succeeds when it finds a country, or a point, or both. - Country is the country's ISO code, else the registered country's. - Subdivision is country + "-" + subdivisions[0].iso_code, only when both exist and the code is 1 to 3 [A-Z0-9] characters. - DB-IP Lite has subdivision names but no ISO codes, so with DB-IP the subdivision is always empty. With GeoLite2 City it is usually filled.

7. The header contract

| Header | Value | Present when | |---|---|---| | X-Carlos-Geo-Country | ISO 3166-1 alpha-2, upper case | a country was found | | X-Carlos-Geo-Subdivision | ISO 3166-2, e.g. GB-SCT | the database has an ISO subdivision code (GeoLite2, not DB-IP Lite) | | X-Carlos-Geo-Lat | decimal degrees, 2 places | the record has a real location | | X-Carlos-Geo-Lon | decimal degrees, 2 places | the same | | X-Carlos-Geo-Source | the database type, e.g. DBIP-City-Lite, GeoLite2-City | any geo header is present |

  • All five are absent on a miss, on an address that isn't locatable, when no database is loaded, and on a door request (§8). The platform never guesses.
  • Coordinates are an approximate area, not a position (codex R1-20). The two decimal places are formatting only. The database's own accuracy can be tens or hundreds of kilometres, and the app docs say so.
  • Apps can already see the client IP in X-Forwarded-For, so these headers tell an app nothing it couldn't work out for itself.
  • Caching (codex R1-19). An app that varies a cached response on a geo header must send Vary on it. The edge cache keeps at most 8 variants per URL across every Vary combination (internal/edge/cache.go). A ninth country evicts one, and other Vary headers multiply the combinations. The app docs say plainly that varying on country means more cache misses for a page with a worldwide audience. Varying on coordinates makes the cache almost useless for that page.

8. Trust: who may set the headers

8.1 Order of operations (codex R2-3)

  1. Capture and strip, at the top of ServeHTTP, before any branch. The edge copies X-Carlos-Hop-Geo and every X-Carlos-Geo-* header into an untrusted capturedGeo value, then deletes them all from r.Header. Nothing reads capturedGeo yet.
  2. Authenticate: acceptHop runs where it does today, and so does the tunnel mark, which comes from the handler wrapper.
  3. Decide, after acceptHop and before either backhaul branch, the door, the gate, analytics, the upgrade branch and the cache lookup. The edge sets geoResult once, from the table in §8.2. Captured values are adopted only when step 2 trusted the transport. Otherwise they are discarded.
  4. Expose: the edge writes the app-visible headers from geoResult back onto r.Header, so Vary matching sees exactly the values the app will get. X-Carlos-Hop-Geo is never written back to r.Header.

8.1a The internal trust header (codex R2-5)

Without more information a downstream edge can't tell "the ingress looked up and missed" from "the ingress had no database" or "this was a door call", since all three send no app headers. So the edge that forwards over a backhaul or the tunnel adds one internal header:

X-Carlos-Hop-Geo: hit;src=DBIP-City-Lite
X-Carlos-Hop-Geo: miss;src=DBIP-City-Lite
X-Carlos-Hop-Geo: off
  • hit and miss mean a reader was loaded and answered. off means no lookup was attempted: no database, a door request, or a rejected hop.
  • It is set only by backhaulProxy, tunnelProxy and fleetProxy, and only from geoResult.
  • It is stripped at capture (step 1) on every request, and adopted only under the same trust as the app headers.
  • setGeo deletes it from every outbound request to an app, so it never reaches app code.
  • An adopted hit needs a valid country or a valid coordinate pair (codex R3-3), matching Lookup's own success rule (§6). A hit with neither is treated as off, and so is a missing or malformed X-Carlos-Hop-Geo on a trusted transport. Fewer countries are counted, but none is made up.
  • A test checks that identical app headers and analytics come out of a local route, a backhaul and the tunnel, including for a coordinate-only result.

8.2 Computing: only at ingress

The ingress edge is the one that accepted the visitor's connection. Only it looks the visitor up, using clientIP(r). Every other edge adopts the validated headers it received.

| Request | Geo result | |---|---| | Direct request | Look up RemoteAddr: hit or miss, or off with no database | | Door request (edge-local socket) | off. The door replaces RemoteAddr with the box's own public address, so a lookup would locate the box. | | Sidecar through loopback | miss, because the address isn't locatable | | Verified backhaul hop (home edge) | Adopt the sending edge's validated headers and state. No lookup. | | Request over the tunnel (remote box), TLS or not | Adopt the hub's validated headers and state. No lookup (D2). | | Rejected hop: it carries X-Carlos-Hop, but acceptHop didn't verify it | off (codex R2-1). The request is served as today, as an ordinary client with its forwarded headers stripped, but geo is withheld, because its RemoteAddr is the sending edge. A client that sends a bogus X-Carlos-Hop only suppresses its own geo. | | Any other request | The same as a direct request |

  • The sending edge of a backhaul computes the result like any ingress edge, then forwards it. A door request that is backhauled therefore carries no geo headers. The home edge adopts that absence and never locates the sending box (codex R1-3).
  • The result goes on the request context as geoResult. Analytics (§9) reads it from there, so an adopted result counts the same as a computed one (codex R1-10).

8.3 Setting them on the outbound request

Every proxy's Rewrite (unix, tcp, multi, tunnel, fleet, backhaul) calls one helper, setGeo(pr), after setForwarded(pr). It:

  • deletes every X-Carlos-Geo-* header and X-Carlos-Hop-Geo from pr.Out;
  • writes the app headers back from the context's geoResult;
  • writes X-Carlos-Hop-Geo too, but only when the proxy forwards to another edge (backhaulProxy, tunnelProxy, fleetProxy, §8.1a).

ReverseProxy removes headers named in Connection before calling Rewrite. So a client sending Connection: X-Carlos-Geo-Country can no longer strip the country from the app's request while the cache files the response under that country (codex R1-1). The cache stores the response under the same values setGeo writes, because both come from geoResult.

8.4 Who is trusted, and how much

  • Verified hop: hopVerifiedKey, which only acceptHop sets, after an HMAC check against the mesh key.
  • Tunnel: a new context mark, tunnelKey{}. - tunnelclient wraps the handler it gives http2.Server.ServeConn, so every request it serves carries the mark, and nothing else sets it. - The mark is set whatever the tunnel's scheme (D2). - The same mark tells the remote edge not to observe the request for analytics (§9). - Tunnel boxes do run the ordinary edge listeners on :80 and :443 alongside the tunnel (internal/host/agent.go, tunnelLoop). A request through those listeners doesn't carry the mark, so it is stripped. A tunnel box holds no database, so it gets no headers either (codex R1-17).
  • Validation before adopting a header (anything invalid is dropped on its own): - country ^[A-Z]{2}$; - subdivision ^[A-Z]{2}-[A-Z0-9]{1,3}$, whose prefix matches the country; - lat/lon decimal numbers in range, re-rounded to 2 places, and kept only as a pair; - source ^[A-Za-z0-9._-]{1,64}$, kept only if some other field survived.
  • Tests: - a forged header on every proxy type, with and without a Connection nomination, including a cache replay; - a forged header on a tunnel box's public listener; - a request through a real tunnelclient connection that adopts the hub's headers. The remote box makes zero analytics observations, and the hub exactly one (codex R5 on R4-1, §9); - a door call to an app whose home is another box, which reaches the app with no geo headers; - a hop signed with a key the verifier doesn't have yet (the rotation window), and a door call backhauled in that window. Both are served, with no geo; - miss, off and door states crossing both a backhaul and the tunnel, and counted correctly at the far side.

8.5 Mixed versions during a roll

An old edge doesn't strip X-Carlos-Geo-*, so an old sending edge passes a forged header across a verified hop, and a new home edge adopts it. The contract therefore holds only once every edge runs a GeoIP-aware binary. The rollout (§15) uses carlos system geoip status's verdict (§5.5) to show that, and only then publishes a database and the app docs.

9. Analytics countries (codex R1-11, R1-12)

  • Counting unit: pageviews. A pageview counts toward Countries when its geoResult state is hit or miss.
  • analytics.Obs gains Country, GeoSource and GeoState, which observeAnalytics reads from the context's geoResult. It never looks up again.
  • By state: - hit counts the country, or "_unknown" for a coordinate-only result (codex R3-3). - miss counts "_unknown". - off (no database, a door call, a rejected hop) isn't counted in Countries at all, so "no GeoIP" never looks like "visitors from nowhere".
  • The state crosses hops and the tunnel in X-Carlos-Hop-Geo (§8.1a), so the edge that observes the pageview counts it the way the ingress edge saw it.
  • One observation per served response (codex R4-1). - Today a request to a tunnel or fleet box is observed twice: once by the hub's observer, which runs for every proxied route, and once by the remote edge. Both flush to the same unprefixed apps/<sqid>/<app>/analytics/. That already double-counts pageviews, and countries would inherit the same double count. - Rule: the remote edge doesn't observe a request that arrived over the tunnel. The hub has already observed it, including when the hub served it from its cache, where the remote never sees it. - Mark: the tunnel mark, tunnelKey, which tunnelclient puts on every request it serves (§8.4, D2). - Scope: usage accounting (usageMeter) is a separate meter, and this doesn't change it. - Test: a real hub and remote edge with both meters on, both flushed. Pageviews, countries and sources each go up by exactly one, for a fresh request and for a hub cache hit.
  • The sending edge of a backhaul does no metering today (§8.2), so a backhauled request is already observed once, at the home edge.
  • Delta, the flush and Day gain GeoSources map[string]int64 (json:"geo_sources"): pageviews counted per database type. This is the provenance the attribution needs. A day's data can come from two sources if a publish changed the type mid-day.
  • Merging: - Day.Countries and Day.GeoSources merge additively across flushes; - the read path (ReadRange, Summary) sums them across boxes and days; - the analytics API response carries countries (top 10 plus _other) and geo_sources. - Both maps are bounded: countries by the ISO list plus _unknown and _other, sources at 8 entries plus _other. - A source with a known attribution (§13: the DBIP- and GeoLite2- prefixes) is never folded into _other, whatever the cap. Only unattributed types fold (codex R7-3). Those prefixes cover a handful of product names, so the map stays small, and every credit a retained range owes stays readable.
  • Old and new days: days written before plan 1 have no geo_sources and an empty countries. They contribute nothing, and the read path treats the missing field as empty. A mixed range tests this.
  • The console's analytics page (app_analytics.html) gains a Countries table: the top 10, then "other". With no country data in the range it says so instead.
  • Attribution: the page shows one attribution line for every source in the range's geo_sources, using the table in §13. Historical DB-IP counts keep their DB-IP credit after a switch to GeoLite2, and the credit stays after off.
  • The new strings go through copy-review before they are written.

10. The feed (plan 2)

The console fetches new editions itself. It is the one control-plane process per deployment, so there is only ever one fetcher.

10.1 Configuration (console env)

| Variable | Meaning | |---|---| | CONSOLE_GEOIP_FEED | dbip-city-lite (the documented default, R9), dbip-country-lite, geolite2-city, geolite2-country, and the paid geoip2-city and geoip2-country; unset means no feed | | CONSOLE_GEOIP_MAXMIND_ACCOUNT | MaxMind account ID, for geolite2-* and geoip2-* | | CONSOLE_GEOIP_MAXMIND_KEY | MaxMind license key, for geolite2-* and geoip2-* |

  • An unknown feed name, or a GeoLite2 feed with no credentials, is logged once at startup. The feed stays off and the console still starts.
  • The key is never logged and never sent anywhere but MaxMind's download host. It goes in Basic auth, never in the URL.

10.2 Checking for a new edition

The console checks once a day, at a jittered time, and once 5 minutes after it starts.

  • DB-IP: the edition is YYYY-MM, from https://download.db-ip.com/free/dbip-<city|country>-lite-YYYY-MM.mmdb.gz. - If the pointer's edition isn't this month, it tries this month's URL. A 404 means it isn't out yet, so it tries again tomorrow. - With no pointer, or a pointer from another origin, it tries this month and then last month.
  • MaxMind: it HEADs https://download.maxmind.com/geoip/databases/GeoLite2-<City|Country>/download?suffix=tar.gz with Basic auth and compares Last-Modified with the pointer's edition. MaxMind documents HEAD as not counting against the download limit. When the edition differs, it fetches suffix=tar.gz.sha256 and then the archive.
  • Redirects: both providers redirect downloads (MaxMind to R2 presigned URLs). The client follows at most 3 redirects, HTTPS only. On a cross-host redirect it drops the Basic auth header.
  • A URL that starts answering 404 or 410 for more than 7 consecutive daily checks raises the box alert's feed reason (§10.4). That is how a provider changing its URLs shows up.

10.3 Integrity and safety

  • Bounded input (codex R1-7). The download streams to the publish path's temporary directory. - The compressed download is capped at 512 MiB while it is being copied. - Decompression: .gz for DB-IP. For MaxMind, the single regular .mmdb member of the .tar.gz. Every byte the tar reader expands counts toward a 1.5 GiB cap, including members it skips, and the .mmdb itself is capped at 1 GiB. - Each check runs under the 15-minute deadline. The directory is removed on every exit path.
  • Checksums (codex R1-18). MaxMind's archive is checked against its published .sha256. DB-IP shows MD5 and SHA-1 checksums on its HTML download page but offers no machine-readable checksum endpoint. The feed doesn't scrape the page. It relies on TLS to download.db-ip.com, then Verify, the country probe and the shrink check (§4.1).
  • Type check (codex R1-14). The database's database_type must match the configured feed, every time, including the first publish: - dbip-city-lite → DBIP-City-Lite - dbip-country-lite → DBIP-Country-Lite - geolite2-city → GeoLite2-City - geolite2-country → GeoLite2-Country
  • Monotonic. When the current pointer came from the same feed, the new build_epoch must be newer. A pointer from a manual publish or another feed is replaced by any database that passes the type check, and the feed logs that it replaced it. The feed never uses force, so the shrink check applies whenever the pointer came from a feed with the same type.
  • First publish: if the pointer is absent and the feed is set, the feed publishes. That is how a fresh deployment gets its first database.
  • On any failure the current database stays, and each edition's failure is logged once.
  • GeoLite2 freshness: MaxMind updates twice a week and the feed checks daily, so a working feed keeps the database well inside D1's 30 days.

10.4 Alerts (codex R1-15)

Today the console's alert watcher only reads per-box records under alerts/boxes/, and webhooks are delivered by the host process. A deployment-level console alert would need a new pipeline. Instead, GeoIP becomes a box rule, geoip, alongside disk, mem, cpu and lag.

  • Evaluation: each mirroring edge evaluates it on its own state. The rule fires once any of these has held for 30 minutes, using metrics.RuleState's existing sustain: - the loaded database is older than 45 days, for any type; - a GeoLite2 database is within 9 days of its D1 expiry (21 days old); - the pointer is expired: marked (§3), or past D1 by the clock. The mirror keeps the observed pointer's type and build after it drops the reader, so expiry never clears the alert before the console's daily pass marks the bucket (codex R11-1); - a live pointer exists but this box hasn't loaded it for over an hour (a bad blob or a full disk); - the feed reason (codex R2-7), evaluated only by the console box's edge, so it fires once: - the console's feed result, published at control/geoip/feed.json, shows 7 consecutive failed checks. This reason is independent of the pointer, so a fresh deployment whose feed has never succeeded still alerts. - feed.json is written by the console after every check (configured feed, last success, consecutive failures, last failure category). It is deleted when the feed is unset.
  • Delivery: through the existing paths. The host's alert sink handles the webhook (when CARLOS_ALERT_WEBHOOK is set, which on the flagship it isn't yet, #192) and the console's alert watcher sends email. internal/host/alertsink.go gains the geoip cases for its message text.
  • Clearing is immediate (codex R2-8): the first evaluation in which no reason holds clears the rule. That matches what the existing machinery does (internal/metrics/alert.go, which clears on the first reading below the line). None of these conditions flaps, so no sustained-clear bookkeeping is added. - The pointer-related reasons clear on a fresh publish, or on an explicit off, which deletes the pointer. - The feed reason clears on the feed's next success, or when the feed is unset. - Expiry alone never clears anything.
  • Tests: - it fires with no other box alert present; - it clears after a publish; - it clears after off; - it stays firing through expiry; - the feed reason fires with no pointer at all; - it restarts with the rule firing.

11. The lookup endpoint and CLI (plan 3)

11.1 Endpoint

GET /api/cli/apps/{account}/{app}/geo?ip=<addr>, registered through cliRoute:

  • Allowed: the instance token (svcrole.Instance), and the member and app tokens that can read the app's analytics.
  • Rate limit: the existing per-caller limiter at 60 a minute. The intended use is occasional lookups, not bulk enrichment; a bulk form can come later if anyone needs it.
  • 200 {"found": true, "country": "IE", "lat": 53.34, "lon": -6.27, "source": "DBIP-City-Lite", "attribution": "IP Geolocation by DB-IP", "attribution_url": "https://db-ip.com"}. - The fields follow §7: absent values are omitted, and the coordinates are rounded the same way. - The source and attribution come from the same Snapshot as the result (§5.4), so they always describe the reader that answered, which may be the CONSOLE_GEOIP_DB override.
  • 200 {"found": false} for an address that isn't locatable or isn't in the database.
  • 400 for a missing or malformed ip.
  • 503 geo unavailable when no database is loaded.
  • The console never logs the queried IP.

11.2 CLI

carlos geo <ip> [--app <app>] calls the endpoint and prints the fields and the attribution. --json prints the response as is. With no app context it uses the current app, like the other app verbs.

11.3 The console's reader (codex R1-6)

The console reads the generation that the console box's own edge mirrors. Once a minute it reads /data/carlos/geoip/current (§5.2):

| What it finds | What it does | |---|---| | a new sha | open that generation with OpenMMDBTrusted and swap | | the same sha | nothing | | current missing, or naming no database | Swap(nil). The console therefore follows off within one edge poll plus one console poll (§4.2). | | any other read error | keep what's loaded, log once an hour |

  • Starting with no file is the normal case before the first publish. The reader starts empty and picks up the first generation within a minute.
  • GeoLite2 expiry (D1) is checked on the same tick, independently of the edge.
  • Region guesses and the endpoint share that one reader.
  • CONSOLE_GEOIP_DB, when set, overrides the mirrored path and keeps its current rules: it's opened once at start, with Verify, and replacing it means a rename plus a restart. Existing deployments keep working unchanged. D1 applies to the override too: a GeoLite2 override past expiry is closed and logged. The file is the operator's, so the platform never deletes it, and the log line says it must be replaced and the old one destroyed.

12. Logging

Every message is a fixed category and never includes an IP address or the MaxMind key.

| Where | Message | Rate | |---|---|---| | edge mirror | edge: geoip database <sha8> (<type>, built <date>) loaded | per swap | | edge mirror | edge: geoip turned off; no database published | per transition | | edge mirror | edge: geoip database <sha8> expired (GeoLite2, 30 days after build) and was removed | per generation | | edge mirror | edge: geoip blob <sha8> rejected: <category> | once per sha | | edge mirror | edge: geoip pointer unreadable; keeping the loaded database | hourly while failing | | geo | geo: a GeoIP lookup failed and was treated as a miss | once a minute, process-wide (existing) | | console | console: geoip published <sha8> (<type>, <origin>) | per publish | | console | console: geoip publish refused: <category> | per attempt | | console feed | console: geoip feed <name>: edition <e> rejected: <category> | once per edition | | console feed | console: geoip feed misconfigured: <category> | at startup |

13. Licensing and docs

| database_type prefix | Attribution the platform shows | Other duties (operator's) | |---|---|---| | DBIP- | IP Geolocation by DB-IP, linked to https://db-ip.com | CC BY 4.0: attribution on every page that shows or uses the data | | GeoLite2- | This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com | The GeoLite EULA, including the 30-day update duty the platform enforces through D1, and terms that pass on to the operator's own users. The docs link the EULA and don't paraphrase all of it. | | anything else | none known | The operator's licence for that database |

  • The operator owns compliance. The operator docs say that plainly, next to the publish verb and the feed settings, and link both providers' terms.
  • The app docs go in docs/site/cli.md, the public reference, as a new section, "Headers your app receives". No page lists them today (survey). - App skills are served from carlosframework.com, outside this repo. Plan 1 files an amadan discussion asking for the skill text to pick up the section. - The section must: - list the headers and explain that they may be absent and why; - cover Vary, with the cache cost (§7); - say that coordinates are an approximate area; - spell out the duty: an app that shows visitors anything derived from these headers or the endpoint must carry the attribution that X-Carlos-Geo-Source names. The endpoint returns the attribution text, so an app can render it.
  • The platform's own pages that use the data carry the attribution: the analytics Countries table, per §9, and the new-app region guess if it ever shows the visitor where it thinks they are.

14. Testing

  • internal/geo: - Lookup against the MaxMind fixtures already in testdata (country, registered-country fallback, subdivision codes from GeoIP2-City-Test.mmdb, no point from a Country database); - malformed locations; - Locate regressions (§6); - Source sanitising; - OpenMMDBTrusted.
  • Swap: -race, with 32 goroutines looking up while 100 swaps happen, and a Close spy proving no close happens while a read lock is held.
  • Mirror, against release.NewDirStore(t.TempDir()). There is no in-memory store type (survey): - first publish, a new sha, the pointer removed, the pointer unreadable; - a digest mismatch, an oversize blob, a disk write failure; - boot with a local generation and no bucket; - boot with current naming a missing file; - a crash injected between each step of §5.3; - the old blob deleted between reading the pointer and fetching the blob; - GeoLite2 expiry at boot and while running.
  • Edge: - the forgery tests in §8.4; - a direct request getting headers; - a verified hop adopting the sender's headers with no lookup (a lookup spy proves it); - none on the door, locally or backhauled; - none for loopback; - headers present on a WebSocket upgrade; - the cache's Vary variant selected by country; - analytics Countries and GeoSources filled from geoResult.
  • Publish path: - each refusal in §4.1; - publishing the same file twice; - X to Y to X; - publish racing off (the mutex); - a foreign pointer write (the ifMatch refusal); - the sweep; - the PutFrom fallback size limit.
  • Feed, against an httptest server standing in for DB-IP and MaxMind: - a new edition; - a 404 before the month's file appears; - an older build_epoch; - a type mismatch, including on the first publish; - a bad SHA-256; - a decompression bomb, and a tar with a huge skipped member; - a cross-host redirect dropping auth; - HEAD-only when Last-Modified hasn't changed; - 7 consecutive 404s raising the feed reason.
  • Analytics: - merge across boxes and days; - a mixed range of old and new days; - attribution per source.
  • Endpoint: - each status code; - role refusals; - the rate limit; - the override reader's attribution; - that no IP ever reaches the log.

15. Rollout

  1. The bucket lifecycle rule: - infra/modules/deployment-buckets gains an aws_s3_bucket_lifecycle_configuration on the deployment bucket, the first one it has. - Its single rule is filtered to the control/geoip/ prefix. It expires noncurrent versions after 1 day and aborts incomplete multipart uploads after 1 day. - It is applied (targeted) before the first GeoLite2 publish. DB-IP Lite has no destroy duty, so a DB-IP-only deployment can apply it later. - For the flagship that means bumping the module ref in carlosframework/carloku-infrastructure, which needs Paul's go-ahead.
  2. Merge plan 1 and roll every box. There is no pointer yet, so no box sets any header. Every edge strips forged ones.
  3. Run carlos system geoip status until its verdict says every fresh box runs a GeoIP-aware binary (§5.5).
  4. Publish a database by hand (plan 1), or set the feed (plan 2). Check that every mirroring box reports the pointer's sha.
  5. Check a request to a test app carries the headers. Check a forged header from outside is replaced, and a door call carries none.
  6. Publish the app docs.

Plans 2 and 3 roll the console, and plan 2 also rolls the boxes for the geoip rule. The feed does nothing until CONSOLE_GEOIP_FEED is set.

16. Plans

| Plan | Contents | |---|---| | 1 | The control/geoip/ lifecycle rule in infra/modules/deployment-buckets (applied later, §15); geo.Lookup/OpenMMDBTrusted/Swappable/Snapshot; release.PutStreamer and its wrappers; the console publish path and carlos system geoip publish/status/off; the edge mirror, its local layout and box status; capture/decide/expose ordering, ingress computing, adopting, X-Carlos-Hop-Geo, setGeo, the rejected-hop rule and the transport-independent tunnel mark (D2); GeoLite2 expiry, including the CONSOLE_GEOIP_DB override's (it moves behind a geo.Swappable, is refused at startup when already expired, and is closed at expiry, codex R10-1); analytics Countries/GeoSources through to the console table; operator and app docs | | 2 | The feed (DB-IP and MaxMind sources, bounded input, integrity, type and monotonic checks, feed.json) and the geoip box rule | | 3 | The console reader on the mirrored generation, the endpoint, carlos geo |

17. Open items for the plans

  • Plan 1, task 1 confirms against real DB-IP Lite and GeoLite2 City files: - that DB-IP Lite has no subdivision ISO codes; - that the country probe's five addresses resolve in both; - Verify's peak memory (§4.1).
  • DB-IP's site says nothing on automated downloads. Plan 2 fetches at most once a day, and only when a new month is due. If DB-IP objects, the feed falls back to manual publishing with no code change.

18. Review dispositions

Round 1 (CODEX-FINDINGS-GEOIP-R1.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | 1 | P1 | Accepted. setGeo in every Rewrite, after hop-header removal, from the same geoResult the cache uses (§8.3) | | 2 | P1 | Accepted. One mutex through cleanup, idempotent blobs, a sweep that keeps the current blob (§3) | | 3 | P1 | Accepted. The trust model is now ingress-computes, adopt-downstream, so a backhauled door call carries no headers (§8.2) | | 4 | P1 | Accepted. D1: GeoLite2 expiry at 30 days, enforced by the edges, console and override; alert at 21 days (§0, §5.3, §10.4, §11.3) | | 5 | P2 | Accepted. Immutable generations named by digest, plus an fsynced current (§5.2) | | 6 | P2 | Accepted. The console's off transition and empty start (§11.3) | | 7 | P2 | Accepted. Bounds during the copy, a 409 for a concurrent publish, deadlines, cleanup, a Verify memory measurement, and no Verify on edges (§4.1, §5.3, §10.3) | | 8 | P2 | Accepted. Opener/OpenObject for reads; PutStreamer for uploads, forwarded by every wrapper (§4.1) | | 9 | P2 | Accepted. Shared decode; Locate unchanged, with regression cases (§6) | | 10 | P2 | Accepted. Adopted headers fill geoResult, so tunnel analytics count countries (§8.2) | | 11 | P2 | Accepted. Pageviews; the read path, API and table are in plan 1; mixed-range test (§9) | | 12 | P2 | Accepted. Per-day GeoSources provenance; live attribution from the answering reader; the EULA linked, not paraphrased (§9, §11.1, §13) | | 13 | P2 | Accepted. apiSystemRoller, person-only (§4.2) | | 14 | P2 | Accepted. node_count in the pointer, comparisons against the etag read, type checked against the feed always (§3, §4.1, §10.3) | | 15 | P2 | Accepted. A box rule through the existing pipelines instead of a new console alert (§10.4) | | 16 | P2 | Accepted. A status block on every new binary, with a mode and a fleet verdict (§5.5) | | 17 | P3 | Accepted. Tunnel boxes have public listeners; the strip covers them (§8.4) | | 18 | P3 | Accepted. DB-IP's HTML checksums named and deliberately not scraped (§10.3) | | 19 | P3 | Accepted. The cache-eviction cost is documented (§7) | | 20 | P3 | Accepted. Coordinates described as an area (§7) | | 21 | P3 | Accepted. Lookup probes instead of metadata (§4.1) | | 22 | P3 | Accepted. #241 quoted in Appendix A |

Round 2 (CODEX-FINDINGS-GEOIP-R2.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-4 (partly) | | The S3 noncurrent-version lifecycle rule (§15 step 0); override files are the operator's to destroy (§0, §11.3) | | R1-7 (partly) | | A 10-minute deadline on mirror passes; expiry on its own ticker (§5.3) | | R1-14 (partly) | | The shrink check applies on a type match, whatever the origin (§4.1) | | R1-16 (partly) | | The verdict covers stale and unreadable boxes from 7 days of records (§5.5) | | 1 | P1 | Accepted. A rejected hop gets geo off (§8.2) | | 2 | P1 | Accepted. The tunnel mark needs a TLS dial (§8.4) | | 3 | P2 | Accepted. Capture, authenticate, decide, expose (§8.1) | | 4 | P2 | Accepted. Snapshot under one read lock (§5.4) | | 5 | P2 | Accepted. X-Carlos-Hop-Geo carries hit/miss/off (§8.1a, §9) | | 6 | P2 | Accepted. The pointer delete is unconditional under the single-writer mutex (§3) | | 7 | P2 | Accepted. The feed reason is independent of the pointer; an expired pointer is distinct from off (§3, §10.4) | | 8 | P3 | Accepted. Immediate clearing, stated (§10.4) | | 9 | P3 | Accepted. The off bound is one edge poll plus one console poll (§4.2) |

Round 3 (CODEX-FINDINGS-GEOIP-R3.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-4 (partly) | | The orphan sweep runs at console startup and daily (§3) | | R1-16 (partly) | | The systemBoxes inventory, including fleet prefixes (§5.5) | | 1 | P1 | Accepted. genMu serializes selection, expiry and cleanup; expiry is checked before download and before install; expired shas are remembered (§5.3) | | 2 | P2 | Accepted. Swappable.Locate, and Config.GeoIP holds the Swappable (§5.4) | | 3 | P2 | Accepted. A coordinate-only hit is adopted, and counted as _unknown (§8.1a, §9) | | 4 | P2 | Accepted. status.BoxStale (§5.5) |

Round 4 (CODEX-FINDINGS-GEOIP-R4.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-16 (partly) | | Reconciles against control/edges/; LoadBoxesComplete and strict fleet reads; an incomplete read is never ready (§5.5) | | 1 | P2 | Accepted. The remote edge skips analytics for tunnelled requests, via a trust-free viaTunnelKey; this fixes the existing pageview double count too (§9) | | 2 | P2 | Accepted. Upload then a job on the existing system job registry; 202, then the CLI polls (§4.2) |

Round 5 (CODEX-FINDINGS-GEOIP-R5.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-16 (partly) | | Fleet boxes reconciled via ListFleetBoxes; LoadBoxesComplete on every prefix (§5.5) | | R4-1 (partly) | | The §8.4 test now asserts zero remote observations and one at the hub | | 1 | P2 | Accepted. The publish slot is reserved before the body is read, and handed to the job (§4.2) |

Round 6 (CODEX-FINDINGS-GEOIP-R6.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-16 (partly) | | Explicit retirement by deleting the edges record; a stale expected box blocks, with that instruction; a platform retire verb is filed as a discussion (§5.5) | | 1 | P2 | Accepted, through D2: the tunnel mark is transport-independent, so the hub's cache, the app's headers and the analytics always agree (§0, §8.2, §8.4, §9) | | 2 | P2 | Accepted. A dedicated staging directory, cleared at console startup (§4.2) |

Round 7 (CODEX-FINDINGS-GEOIP-R7.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R6-1 (partly) | | §16 now says the tunnel mark is transport-independent (D2) | | 1 | P2 | Accepted. An expired candidate is refused, with no force override, and rechecked before the pointer write (§4.1 step 5) | | 2 | P2 | Accepted. Source falls back to mmdb and is never empty (§6) | | 3 | P2 | Accepted. Attributed sources are never folded into _other (§9) | | 4 | P3 | Accepted. Signatures reconciled with the committed code (§4.1, §6) |

Round 8 (CODEX-FINDINGS-GEOIP-R8.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-16 (partly) | | Only expected boxes block; a retired box unblocks as soon as its edges record is deleted. The retire verb stays a filed platform discussion, not part of GeoIP (§5.5) | | 1 | P2 | Accepted. (namespace, label) identity throughout the verdict (§5.5) | | 2 | P2 | Accepted. A re-read of the pointer before install, one immediate re-run, and pass-start timing, so the 11-minute bound holds (§5.3) | | 3 | P3 | Accepted. §3 step 2 now requires digest and size |

Round 9 (CODEX-FINDINGS-GEOIP-R9.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R1-16 (partly) | | Unreadable records block only through expected identities; the final requirement says "every expected box" (§5.5) | | R8-2 (partly) | | 30-second bound on each pointer read; the off bound states it; read failures keep the loaded database and surface in the heartbeat (§5.3) | | 1 | P2 | Accepted. Four pointer states: unknown never ready, off/expired requires empty mirrors, live requires the sha (§5.5) | | 2 | P2 | Accepted. off answers 409 while a publish holds the slot (§4.2) |

Round 10 (CODEX-FINDINGS-GEOIP-R10.md): every finding accepted.

| # | Sev | Disposition | |---|---|---| | R8-2 (partly) | | The bounds read 11.5 and 12.5 minutes, given successful reads (§4.2) | | R9-1 (partly) | | The verdict treats a pointer past D1 by the clock as expired (§5.5) | | 1 | P2 | Accepted. Override expiry is in plan 1: a Swappable, refused at startup when expired, closed by a ticker (§16) |

Round 11 (CODEX-FINDINGS-GEOIP-R11.md): accepted.

| # | Sev | Disposition | |---|---|---| | 1 | P2 | Accepted. The alert's expired reason is marked or past D1 by the clock, using pointer metadata the mirror keeps after dropping the reader; tests cover delayed marking, restart, fresh publish and off (§10.4; plan 2 Task 4) |

Appendix A. The request (#241, verbatim)

> Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint) > > ## Why > > The nearest-region-default spec (docs/superpowers/specs/2026-10-01-nearest-region-default-design.md, §6, on branch edge-replica-placements) gives the console an optional GeoIP reader: CONSOLE_GEOIP_DB names an operator-supplied .mmdb (DB-IP Lite or MaxMind GeoLite2), read through github.com/oschwald/maxminddb-golang, mapped and opened once per process. Today it serves one caller: guessing a new app's region from the requester's IP. > > Once the platform carries that database, every app on it is one lookup away from "roughly where is this visitor", and none of them should have to ship their own .mmdb, handle attribution, or track database updates. The platform should expose the lookup to apps. > > ## What > > A platform geolocation facility that any app can use, backed by the same database and reader the console already has. Two shapes are worth weighing; the agent picking this up should choose one (or both) and say why in the spec: > > 1. Router-injected request headers. The edge/router does the lookup on the client IP and sets headers on the proxied request, e.g. X-Carlos-Geo-Country, X-Carlos-Geo-Region, X-Carlos-Geo-Lat / -Lon, X-Carlos-Geo-Source. Zero integration work for apps; works in every language; covers the common case (country/currency/locale defaults, "nearest region" hints). Cost: a lookup on every request unless cached per connection or per IP. > 2. A lookup endpoint. A platform-internal HTTP endpoint apps can call with an arbitrary IP (GET /geo?ip=… on the sidecar/hop path, authenticated the way other platform-internal calls are). Needed for anything that isn't "the current request": batch jobs, IPs from logs, webhooks carrying a client IP. > > Headers alone probably cover 90% of app needs; the endpoint covers the rest. Shipping headers first and the endpoint as a follow-on is a reasonable sequence. > > ## Constraints to carry over from the spec > > - Unset CONSOLE_GEOIP_DB means the feature is off: headers absent / endpoint returns "unavailable", never a guess. Unopenable database: logged once, feature off, platform still starts. > - Never look up loopback, private, link-local, CGNAT or unspecified addresses. > - Lookup errors are a miss, logged at most once a minute. > - Database replacement is an atomic rename plus restart; whatever process owns the lookup needs the same lifecycle. > - Attribution: DB-IP Lite (CC BY 4.0) requires "IP Geolocation by DB-IP" with a link on pages that show the data. If apps surface the data, the platform docs must tell them about that obligation, and the platform's own pages that use it must carry it. GeoLite2 has its own EULA terms (attribution, keeping the data updated). The operator owns compliance; the docs should say so plainly. > - The reader lives in (or is being added to) the console. If the router or a sidecar needs it, the reader should move to a shared internal package rather than being duplicated. > > ## Open questions > > - Where does the lookup physically run: router, console, sidecar? Which one actually sees the client IP, and which one can memory-map the database? Only one place should open the file. > - Header naming and whether to include coordinates or just country/subdivision. Coordinates make "nearest X" trivial for apps but are more than most need. > - Should apps be able to opt out (an app config flag) to avoid the per-request cost when they don't care? > - Does carlos CLI need anything (e.g. carlos geo <ip> for debugging what the platform would return)? > > ## Related > > - Nearest-region default spec, §6 (IP lookup), §14 (region table). Build on its reader; do not add a second one.

How rev 2 answers the request's constraints:

  • "Unset means off": the off switch is now an absent pointer (R1). CONSOLE_GEOIP_DB keeps its meaning for the console.
  • "Atomic rename plus restart": replaced by the hot swap for the edge and console mirror (R1). The override keeps the rename-plus-restart rule.
  • "Only one place should open the file": each process that serves lookups opens it once (one edge per box, plus the console). One file per box is mapped by every reader on that box, and the bucket holds the single source.
  • "Header naming": X-Carlos-Geo-Subdivision replaces the request's example -Region, because "region" already means an AWS region on this platform.