amadan

carlos / platform Public

geoip spec rev 5 (Astra round 4: upload+job, single analytics observation, edges-record reconciliation) and plan 1 draft

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude feef9a5c49227c969049d8e31b0dffefdfc8f9c8 parent d06821c
3 files changed, +4080 −13
  • docs/superpowers/plans/2026-10-06-platform-geoip-plan1.md +3976 −0
  • docs/superpowers/specs/2026-10-06-platform-geoip-design.md +80 −13
  • docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R4.md +24 −0

CI cancelled — superseded by a newer push — run details

diff --git a/docs/superpowers/plans/2026-10-06-platform-geoip-plan1.md b/docs/superpowers/plans/2026-10-06-platform-geoip-plan1.md
new file mode 100644
index 000000000..49d83f51e
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-06-platform-geoip-plan1.md
@@ -0,0 +1,3976 @@
+# Platform GeoIP, plan 1: mirror, headers and analytics countries
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Every proxied request to an app carries `X-Carlos-Geo-*` headers. They are computed by the ingress edge from a bucket-published `.mmdb` that every edge mirrors and hot-swaps. Analytics count pageviews by country, and operators publish with `carlos system geoip`.
+
+**Architecture:**
+- A shared vocabulary package, `internal/geoip`, holds the pointer record,
+ the expiry rule, attribution, and header encoding and validation.
+- `internal/geo` gains a richer `Lookup` and a lock-guarded `Swappable`
+ reader.
+- The console publishes content-addressed blobs and a pointer under
+ `control/geoip/`. Each edge's mirror loop installs immutable generations
+ under `/data/carlos/geoip/`.
+- `ServeHTTP` captures and strips inbound geo headers, authenticates, then
+ computes or adopts one result. Every proxy's `Rewrite` writes that result
+ out.
+- Analytics carry the result's country and source into `Day`.
+
+**Tech Stack:** Go, `github.com/oschwald/maxminddb-golang` v1.13.1 (already in `go.mod`), `net/http/httputil.ReverseProxy`, AWS SDK v2 S3, OpenTofu (`infra/modules/deployment-buckets`).
+
+**Spec:** `docs/superpowers/specs/2026-10-06-platform-geoip-design.md` (rev 5). Read §0 rulings first. Measured facts: `docs/superpowers/plans/2026-10-06-platform-geoip-FACTS.md`.
+
+## Global Constraints
+
+- **Header names**, exactly:
+ - `X-Carlos-Geo-Country`, `X-Carlos-Geo-Subdivision`, `X-Carlos-Geo-Lat`,
+ `X-Carlos-Geo-Lon`, `X-Carlos-Geo-Source`;
+ - the internal one, `X-Carlos-Hop-Geo`.
+- **Bucket keys:**
+ - pointer `control/geoip/current.json`;
+ - blob `control/geoip/<sha256 hex>.mmdb`;
+ - feed status `control/geoip/feed.json` (plan 2 writes it; plan 1 only
+ reserves the name).
+- **Local layout:** `/data/carlos/geoip/<sha256>.mmdb`, `/data/carlos/geoip/current`, and `/data/carlos/geoip/.incoming-*`. The directory is 0755 and the files 0644.
+- **Limits:**
+ - blob at most 1 GiB (`1 << 30`); pointer read at most 64 KiB;
+ - `PutFrom` fallback to `Put` only up to 64 MiB;
+ - publish deadline 15 minutes; mirror pass deadline 10 minutes;
+ - mirror poll every 10 minutes plus up to 1 minute of jitter; expiry
+ ticker every 1 minute.
+- **D1:** a `database_type` with prefix `GeoLite2-` expires 30 days after
+ `build_epoch`. The warning starts at 21 days. Other types never expire.
+- **Coordinates** are formatted with exactly 2 decimal places
+ (`strconv.FormatFloat(v, 'f', 2, 64)`) after
+ `math.Round(v*100)/100`.
+- **Validation:**
+ - country `^[A-Z]{2}$`;
+ - subdivision `^[A-Z]{2}-[A-Z0-9]{1,3}$`, whose prefix equals the
+ country;
+ - source `^[A-Za-z0-9._-]{1,64}$`;
+ - lat in [-90, 90] and lon in [-180, 180], finite.
+- **Logs** never contain an IP address and never contain database bytes.
+ Use fixed categories; the spec §12 table is the wording.
+- **Operator gate:** a person admitted by `apiSystemRoller`
+ (`internal/console/apisystemupdate.go:106`). The routes are registered
+ with `svcrole.Set{}` and added to `closedDoors`
+ (`internal/console/svcroutes_test.go`).
+- **Copy:** no em dashes in user-facing strings. Use a full stop and a new
+ lower-case sentence (memory: copy-voice). Every new user-facing string
+ (CLI output, console page, docs) goes through the `copy-review` skill
+ before it lands. The executor collects them in Task 13.
+- **Cross-compile:** `GOOS=darwin GOARCH=arm64 go build ./...` and
+ `GOOS=windows go build ./...` must pass. Nothing new here is
+ Linux-syscall-specific, but `fsync` of a directory uses `os.File.Sync`,
+ which is portable.
+- **Build environment, inside the sandbox:** `export GOCACHE=$TMPDIR/gocache GOFLAGS="-mod=mod -buildvcs=false" GOPROXY=off`. Packages that bind unix sockets (`internal/edge`, `internal/host`) run outside the sandbox with `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false`, after `mkdir -p /tmp/g`. Run tests `nice -n 19`, one package at a time (CPU budget).
+
+## Review Focus
+
+1. **A client-sent `X-Carlos-Geo-*` or `X-Carlos-Hop-Geo` on any path,**
+ including a `Connection:` nomination. The app must see only the edge's
+ values, or none. Tests are in Task 9.
+2. **An edge locating something that isn't the visitor:** the box (door),
+ the sender edge (rejected hop), or the hub (tunnel). It must send no
+ headers instead. Tests are in Tasks 9 and 10.
+3. **A swap while lookups run.** No lookup may run against a closed
+ mapping, and a source label must never come from a different reader
+ than the result. Tests are in Task 3.
+4. **A crash or a full disk mid-install,** or an expired GeoLite2
+ database arriving. The edge must come back to a consistent generation
+ or to none, never to a half file or an expired one. Tests are in
+ Task 7.
+5. **Old days mixed with new days** in one analytics range, and a switch
+ of database type mid-range. The country totals must add up, and every
+ source must be credited. Tests are in Task 11.
+
+---
+
+## File structure
+
+| File | Responsibility |
+|---|---|
+| `infra/modules/deployment-buckets/main.tf` | lifecycle rule for `control/geoip/` |
+| `internal/geo/mmdb.go` | `decode`, `Lookup`, `Result`, metadata accessors, `OpenMMDBTrusted`; `Locate` unchanged in behaviour |
+| `internal/geo/swappable.go` (new) | `Swappable`, `Snapshot` |
+| `internal/release/store.go` | `PutStreamer`, implemented by S3, Dir and prefix stores; `PutObjectFrom` helper |
+| `internal/geoip/pointer.go` (new) | keys, `Pointer`, `ParsePointer`, D1 expiry, attribution |
+| `internal/geoip/answer.go` (new) | `Answer` (state, country, ...), app header write and strip, hop value encode, adopt and validate |
+| `internal/console/geoippublish.go` (new) | `geoipPublisher`: publish, off, sweep, expire; probes; shrink check |
+| `internal/console/apisystemgeoip.go` (new) | `PUT/GET/DELETE /api/cli/system/geoip` handlers |
+| `system_geoip.go` (new, package main) | `carlos system geoip publish|status|off` |
+| `internal/edge/geoipmirror.go` (new) | the mirror: local layout, boot, poll, expiry ticker, state |
+| `internal/edge/geoipheaders.go` (new) | `geoResult` context, capture, decide, `setGeo` |
+| `internal/edge/edge.go` | `ServeHTTP` wiring, six `Rewrite`s, the `Edge.geo` field, run wiring |
+| `internal/tunnelclient/tunnelclient.go` | TLS-only tunnel mark, `Config.TLSConfig` test seam |
+| `internal/status/snapshot.go` | `GeoIPState` in `BoxMetrics` and `Probes` |
+| `internal/host/agent.go`, `internal/host/tunnel_client.go` | publish the edge's geo state to the heartbeat; `WithGeoTunnelClient` |
+| `internal/analytics/{meter,day,read}.go` | `Obs.Country/GeoSource/GeoState`, `Delta`, `Day.GeoSources`, merge, read |
+| `internal/console/{apianalytics.go,analyticsview.go,templates/app_analytics.html}` | countries and attribution |
+| `apimode.go`, `analytics_cmd.go` | the CLI prints countries |
+| `docs/site/cli.md`, `README.md` | app and operator docs |
+
+---
+
+### Task 1: Bucket lifecycle rule for `control/geoip/`
+
+**Files:**
+- Modify: `infra/modules/deployment-buckets/main.tf` (after the `aws_s3_bucket_versioning.deployment` resource, ~line 26)
+
+**Interfaces:** none.
+
+- [ ] **Step 1: Add the resource**
+
+```hcl
+# GeoIP databases (platform spec 2026-10-06 §15 step 0, D1): a deleted or
+# superseded blob under control/geoip/ must not live on as a noncurrent
+# version — MaxMind's GeoLite EULA requires destroying superseded editions.
+# Scoped to that prefix only; nothing else in this bucket expires.
+resource "aws_s3_bucket_lifecycle_configuration" "deployment" {
+ bucket = aws_s3_bucket.deployment.id
+
+ rule {
+ id = "geoip-noncurrent"
+ status = "Enabled"
+ filter {
+ prefix = "control/geoip/"
+ }
+ noncurrent_version_expiration {
+ noncurrent_days = 1
+ }
+ abort_incomplete_multipart_upload {
+ days_after_initiation = 1
+ }
+ }
+
+ depends_on = [aws_s3_bucket_versioning.deployment]
+}
+```
+
+- [ ] **Step 2: Validate**
+
+Run: `cd infra/modules/deployment-buckets && tofu fmt -check && tofu init -backend=false >/dev/null && tofu validate`
+Expected: `Success! The configuration is valid.` If `tofu` isn't installed, run `terraform` with the same arguments. If neither is installed, record that in the task report and leave the check to CI.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add infra/modules/deployment-buckets/main.tf
+git commit -m "infra: expire noncurrent control/geoip/ versions after a day (geoip spec §15)"
+```
+
+The rule is **applied later, by Paul**: a targeted apply through `carlosframework/carloku-infrastructure` with the module ref bumped. Don't apply it.
+
+---
+
+### Task 2: `internal/geo` — `Lookup`, metadata, `OpenMMDBTrusted`
+
+**Files:**
+- Modify: `internal/geo/mmdb.go`
+- Test: `internal/geo/mmdb_test.go`
+
+**Interfaces:**
+- Produces:
+ ```go
+ type Result struct { Country, Subdivision string; Point Point; HasPoint bool }
+ func (m *MMDB) Lookup(ip net.IP) (Result, bool)
+ func (m *MMDB) Source() string
+ func (m *MMDB) BuildEpoch() time.Time
+ func (m *MMDB) NodeCount() uint64
+ func OpenMMDBTrusted(path string) (*MMDB, error)
+ ```
+ `Locate` keeps its signature and exact behaviour.
+
+- [ ] **Step 1: Write the failing tests** (append to `mmdb_test.go`; add `"time"` to its imports)
+
+```go
+func TestLookupCityGivesCountrySubdivisionAndPoint(t *testing.T) {
+ m, err := OpenMMDB("testdata/GeoIP2-City-Test.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer m.Close()
+ cases := map[string]Result{
+ "81.2.69.142": {Country: "GB", Subdivision: "GB-ENG", HasPoint: true},
+ "216.160.83.56": {Country: "US", Subdivision: "US-WA", HasPoint: true},
+ "89.160.20.112": {Country: "SE", Subdivision: "SE-E", HasPoint: true},
+ "2001:218::1": {Country: "JP", HasPoint: true}, // no subdivision in the fixture
+ }
+ for ip, want := range cases {
+ got, ok := m.Lookup(net.ParseIP(ip))
+ if !ok || got.Country != want.Country || got.Subdivision != want.Subdivision || got.HasPoint != want.HasPoint {
+ t.Errorf("Lookup(%s) = (%+v, %v), want %+v", ip, got, ok, want)
+ }
+ }
+ if got, _ := m.Lookup(net.ParseIP("81.2.69.142")); got.Point.Lat < 51 || got.Point.Lat > 52 {
+ t.Errorf("London point = %v", got.Point)
+ }
+ if m.Source() != "GeoIP2-City" {
+ t.Errorf("Source = %q", m.Source())
+ }
+ if m.NodeCount() != 1547 {
+ t.Errorf("NodeCount = %d, want 1547 (FACTS.md)", m.NodeCount())
+ }
+ if m.BuildEpoch().Year() < 2020 {
+ t.Errorf("BuildEpoch = %v", m.BuildEpoch())
+ }
+}
+
+func TestLookupCountryDatabaseHasNoPoint(t *testing.T) {
+ m, err := OpenMMDB("testdata/GeoIP2-Country-Test.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer m.Close()
+ got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
+ if !ok || got.Country != "GB" || got.HasPoint || got.Subdivision != "" {
+ t.Fatalf("country-only = (%+v, %v), want GB with no point (never a centroid)", got, ok)
+ }
+}
+
+func TestLookupNoCountryDatabaseMisses(t *testing.T) {
+ m, err := OpenMMDB("testdata/MaxMind-DB-test-ipv4-24.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer m.Close()
+ if got, ok := m.Lookup(net.ParseIP("81.2.69.142")); ok {
+ t.Fatalf("a database with no country or location answered %+v", got)
+ }
+}
+
+func TestLookupFallsBackToRegisteredCountryButNotItsSubdivision(t *testing.T) {
+ m := &MMDB{lookup: func(_ net.IP, result any) error {
+ rec := result.(*mmdbRecord)
+ rec.RegisteredCountry.ISOCode = "IE"
+ rec.Subdivisions = append(rec.Subdivisions, struct {
+ ISOCode string `maxminddb:"iso_code"`
+ }{"L"})
+ return nil
+ }}
+ got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
+ if !ok || got.Country != "IE" || got.Subdivision != "" {
+ t.Fatalf("registered fallback = (%+v, %v), want IE with no subdivision", got, ok)
+ }
+}
+
+func TestLookupPointWithoutCountrySucceeds(t *testing.T) {
+ lat, lon := 10.0, 20.0
+ m := &MMDB{lookup: func(_ net.IP, result any) error {
+ rec := result.(*mmdbRecord)
+ rec.Location.Latitude, rec.Location.Longitude = &lat, &lon
+ return nil
+ }}
+ got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
+ if !ok || got.Country != "" || !got.HasPoint || got.Point != (Point{10, 20}) {
+ t.Fatalf("point-only = (%+v, %v)", got, ok)
+ }
+ // Locate's existing rule: a valid point succeeds without a country.
+ if p, ok := m.Locate(net.ParseIP("81.2.69.142")); !ok || p != (Point{10, 20}) {
+ t.Fatalf("Locate point-only = (%v, %v)", p, ok)
+ }
+}
+
+func TestLocateRegisteredCountryFallbackUnchanged(t *testing.T) {
+ // Codex R1-9: a country with no representative point falls back to the
+ // registered country's point. "XK" has none in the table; "GB" does.
+ if _, ok := CountryPoint("XK"); ok {
+ t.Skip("XK gained a representative point; pick another code absent from the table")
+ }
+ m := &MMDB{lookup: func(_ net.IP, result any) error {
+ rec := result.(*mmdbRecord)
+ rec.Country.ISOCode, rec.RegisteredCountry.ISOCode = "XK", "GB"
+ return nil
+ }}
+ gb, _ := CountryPoint("GB")
+ if p, ok := m.Locate(net.ParseIP("81.2.69.142")); !ok || p != gb {
+ t.Fatalf("Locate = (%v, %v), want GB's point", p, ok)
+ }
+}
+
+func TestLookupRejectsMalformedCodes(t *testing.T) {
+ m := &MMDB{lookup: func(_ net.IP, result any) error {
+ rec := result.(*mmdbRecord)
+ rec.Country.ISOCode = "gb" // lower case is not ISO
+ rec.Subdivisions = append(rec.Subdivisions, struct {
+ ISOCode string `maxminddb:"iso_code"`
+ }{"TOOLONG"})
+ return nil
+ }}
+ if got, ok := m.Lookup(net.ParseIP("81.2.69.142")); ok {
+ t.Fatalf("malformed codes answered %+v", got)
+ }
+}
+
+func TestOpenMMDBTrustedOpensWithoutVerify(t *testing.T) {
+ m, err := OpenMMDBTrusted("testdata/GeoIP2-City-Test.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer m.Close()
+ if r, ok := m.Lookup(net.ParseIP("81.2.69.142")); !ok || r.Country != "GB" {
+ t.Fatalf("trusted open lookup = (%+v, %v)", r, ok)
+ }
+ if _, err := OpenMMDBTrusted(filepath.Join(t.TempDir(), "missing.mmdb")); err == nil {
+ t.Fatal("a missing file opened")
+ }
+}
+
+func TestSourceIsSanitised(t *testing.T) {
+ m := &MMDB{source: sanitizeSource("GeoLite2-City\n<script>" + strings.Repeat("x", 100))}
+ if s := m.Source(); strings.ContainsAny(s, "\n<>") || len(s) > 64 {
+ t.Fatalf("Source = %q", s)
+ }
+ var nilM *MMDB
+ if nilM.Source() != "" || nilM.NodeCount() != 0 || !nilM.BuildEpoch().IsZero() {
+ t.Fatal("a nil MMDB must answer zero values")
+ }
+ _ = time.Now
+}
+```
+
+Add `"strings"` to the test imports.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/geo -run 'Lookup|OpenMMDBTrusted|SourceIs|LocateRegistered' -count=1`
+Expected: build failure. `Lookup`, `Result`, `Source`, `NodeCount`, `BuildEpoch`, `OpenMMDBTrusted`, `sanitizeSource` and `mmdbRecord.Subdivisions` are all undefined.
+
+- [ ] **Step 3: Implement it.** Replace `mmdb.go`'s `MMDB`, `OpenMMDB`, `mmdbRecord` and `Locate` with the following, and keep `Close`, `onTheGlobe`, `cgnat` and `Locatable` as they are:
+
+```go
+// MMDB is an operator-supplied MaxMind-format database. lookup is the
+// reader's Lookup, a field so a test can prove which addresses ever reach
+// it; source, build and nodes are copied from the metadata at open so a
+// test fake can set them too.
+type MMDB struct {
+ r *maxminddb.Reader
+ lookup func(ip net.IP, result any) error
+ source string
+ build time.Time
+ nodes uint64
+}
+
+// OpenMMDB opens and verifies the file. An error means "IP step off"; the
+// console logs it once and starts anyway.
+func OpenMMDB(path string) (*MMDB, error) {
+ r, err := maxminddb.Open(path)
+ if err != nil {
+ return nil, err
+ }
+ if err := r.Verify(); err != nil {
+ r.Close()
+ return nil, err
+ }
+ return newMMDB(r), nil
+}
+
+// OpenMMDBTrusted opens without Verify. Only for a file whose bytes were
+// verified elsewhere and proven identical by digest (geoip spec §5.3):
+// Verify decodes the whole data section, which costs seconds and tens of
+// MiB on every edge for no new information.
+func OpenMMDBTrusted(path string) (*MMDB, error) {
+ r, err := maxminddb.Open(path)
+ if err != nil {
+ return nil, err
+ }
+ return newMMDB(r), nil
+}
+
+func newMMDB(r *maxminddb.Reader) *MMDB {
+ md := r.Metadata
+ return &MMDB{
+ r: r, lookup: r.Lookup,
+ source: sanitizeSource(md.DatabaseType),
+ build: time.Unix(int64(md.BuildEpoch), 0).UTC(),
+ nodes: uint64(md.NodeCount),
+ }
+}
+
+// sanitizeSource reduces a metadata database_type to [A-Za-z0-9._-],
+// at most 64 bytes: it is echoed into a request header and a log line.
+func sanitizeSource(s string) string {
+ var b strings.Builder
+ for _, c := range s {
+ if b.Len() >= 64 {
+ break
+ }
+ if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '.' || c == '_' || c == '-' {
+ b.WriteRune(c)
+ }
+ }
+ return b.String()
+}
+
+// Source is the database's type ("DBIP-City-Lite", "GeoLite2-City"), sanitised.
+func (m *MMDB) Source() string {
+ if m == nil {
+ return ""
+ }
+ return m.source
+}
+
+// BuildEpoch is when the database was built (UTC); zero for a nil MMDB.
+func (m *MMDB) BuildEpoch() time.Time {
+ if m == nil {
+ return time.Time{}
+ }
+ return m.build
+}
+
+// NodeCount is the search tree's size, the shrink check's yardstick.
+func (m *MMDB) NodeCount() uint64 {
+ if m == nil {
+ return 0
+ }
+ return m.nodes
+}
+
+type mmdbRecord struct {
+ Location struct {
+ Latitude *float64 `maxminddb:"latitude"`
+ Longitude *float64 `maxminddb:"longitude"`
+ } `maxminddb:"location"`
+ Country struct {
+ ISOCode string `maxminddb:"iso_code"`
+ } `maxminddb:"country"`
+ RegisteredCountry struct {
+ ISOCode string `maxminddb:"iso_code"`
+ } `maxminddb:"registered_country"`
+ Subdivisions []struct {
+ ISOCode string `maxminddb:"iso_code"`
+ } `maxminddb:"subdivisions"`
+}
+
+var (
+ lookupErrMu sync.Mutex
+ lookupErrLast time.Time
+)
+
+// decode is the one place a record is read and judged (codex R1-9): the
+// Locatable filter, IPv4 normalisation, the fixed-category error log, and
+// the rule that a malformed location makes the WHOLE record a miss.
+func (m *MMDB) decode(ip net.IP) (mmdbRecord, bool) {
+ var rec mmdbRecord
+ if m == nil || m.lookup == nil || !Locatable(ip) {
+ return rec, false
+ }
+ if v4 := ip.To4(); v4 != nil {
+ ip = v4
+ }
+ if err := m.lookup(ip, &rec); err != nil {
+ lookupErrMu.Lock()
+ if time.Since(lookupErrLast) > time.Minute {
+ lookupErrLast = time.Now()
+ log.Print("geo: a GeoIP lookup failed and was treated as a miss")
+ }
+ lookupErrMu.Unlock()
+ return mmdbRecord{}, false
+ }
+ if rec.Location.Latitude != nil && rec.Location.Longitude != nil &&
+ !onTheGlobe(Point{*rec.Location.Latitude, *rec.Location.Longitude}) {
+ // A structurally valid database can still carry a malformed
+ // location (final review). A record wrong in one field is not
+ // trusted on the next.
+ return mmdbRecord{}, false
+ }
+ return rec, true
+}
+
+// Locate returns the record's location (City databases), else its
+// country's point, else its registered country's. Behaviour is exactly
+// what it was before Lookup existed (codex R1-9).
+func (m *MMDB) Locate(ip net.IP) (Point, bool) {
+ rec, ok := m.decode(ip)
+ if !ok {
+ return Point{}, false
+ }
+ if rec.Location.Latitude != nil && rec.Location.Longitude != nil {
+ return Point{*rec.Location.Latitude, *rec.Location.Longitude}, true
+ }
+ if p, ok := CountryPoint(rec.Country.ISOCode); ok {
+ return p, true
+ }
+ return CountryPoint(rec.RegisteredCountry.ISOCode)
+}
+
+// Result is what Lookup found. Country is ISO 3166-1 alpha-2 (the
+// record's country, else its registered country); Subdivision is
+// "CC-XXX" only when the database carries an ISO code for it (GeoLite2
+// does, DB-IP Lite does not, FACTS.md); HasPoint is true only for a real
+// location record, never a country centroid.
+type Result struct {
+ Country string
+ Subdivision string
+ Point Point
+ HasPoint bool
+}
+
+// Lookup answers with whatever the record holds. ok is true when it found
+// a country, a point, or both (geoip spec §6).
+func (m *MMDB) Lookup(ip net.IP) (Result, bool) {
+ rec, ok := m.decode(ip)
+ if !ok {
+ return Result{}, false
+ }
+ var r Result
+ if isISO2(rec.Country.ISOCode) {
+ r.Country = rec.Country.ISOCode
+ if len(rec.Subdivisions) > 0 && isSubdivisionCode(rec.Subdivisions[0].ISOCode) {
+ r.Subdivision = r.Country + "-" + rec.Subdivisions[0].ISOCode
+ }
+ } else if isISO2(rec.RegisteredCountry.ISOCode) {
+ // The registered country says where the block is registered, not
+ // which subdivision of it the visitor is in: no subdivision.
+ r.Country = rec.RegisteredCountry.ISOCode
+ }
+ if rec.Location.Latitude != nil && rec.Location.Longitude != nil {
+ r.Point, r.HasPoint = Point{*rec.Location.Latitude, *rec.Location.Longitude}, true
+ }
+ return r, r.Country != "" || r.HasPoint
+}
+
+func isISO2(s string) bool {
+ return len(s) == 2 && s[0] >= 'A' && s[0] <= 'Z' && s[1] >= 'A' && s[1] <= 'Z'
+}
+
+func isSubdivisionCode(s string) bool {
+ if len(s) < 1 || len(s) > 3 {
+ return false
+ }
+ for i := 0; i < len(s); i++ {
+ if !(s[i] >= 'A' && s[i] <= 'Z' || s[i] >= '0' && s[i] <= '9') {
+ return false
+ }
+ }
+ return true
+}
+```
+
+Add `"strings"` to `mmdb.go`'s imports. `math` is still used by `onTheGlobe`.
+
+- [ ] **Step 4: Run the package's tests**
+
+Run: `nice -n 19 go test ./internal/geo -count=1`
+Expected: PASS, including every test that existed before (`TestLocateRejectsMalformedCoordinates`, `TestLocateNeverLooksUpNonPublicAddresses`, `TestMMDBLookupErrorsNeverLogTheAddress`).
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/geo/mmdb.go internal/geo/mmdb_test.go
+git commit -m "geo: Lookup with country, subdivision and point; metadata accessors; OpenMMDBTrusted (geoip spec §6)"
+```
+
+---
+
+### Task 3: `geo.Swappable`
+
+**Files:**
+- Create: `internal/geo/swappable.go`
+- Test: `internal/geo/swappable_test.go`
+
+**Interfaces:**
+- Consumes: Task 2's `MMDB.Lookup`, `Source`, `BuildEpoch`, `Locate`, `Close`.
+- Produces:
+ ```go
+ type Snapshot struct { Result Result; Found bool; Source string; Loaded bool }
+ func NewSwappable() *Swappable
+ func (s *Swappable) Lookup(ip net.IP) Snapshot
+ func (s *Swappable) Locate(ip net.IP) (Point, bool) // implements Locator
+ func (s *Swappable) Swap(next *MMDB)
+ func (s *Swappable) Loaded() (source string, build time.Time, ok bool)
+ ```
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+package geo
+
+import (
+ "net"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "time"
+)
+
+func TestSwappableEmptyAnswersNotLoaded(t *testing.T) {
+ s := NewSwappable()
+ if snap := s.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded || snap.Found || snap.Source != "" {
+ t.Fatalf("empty swappable = %+v", snap)
+ }
+ if _, ok := s.Locate(net.ParseIP("81.2.69.142")); ok {
+ t.Fatal("empty swappable located")
+ }
+ var _ Locator = s
+}
+
+func TestSwappableSnapshotCarriesItsReadersSource(t *testing.T) {
+ s := NewSwappable()
+ m, err := OpenMMDB("testdata/GeoIP2-City-Test.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ s.Swap(m)
+ snap := s.Lookup(net.ParseIP("81.2.69.142"))
+ if !snap.Loaded || !snap.Found || snap.Source != "GeoIP2-City" || snap.Result.Country != "GB" {
+ t.Fatalf("snapshot = %+v", snap)
+ }
+ miss := s.Lookup(net.ParseIP("10.0.0.1"))
+ if !miss.Loaded || miss.Found || miss.Source != "GeoIP2-City" {
+ t.Fatalf("a miss with a loaded reader must say Loaded and name the source: %+v", miss)
+ }
+ s.Swap(nil)
+ if snap := s.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded {
+ t.Fatalf("after Swap(nil) = %+v", snap)
+ }
+}
+
+// Review Focus 3: no lookup ever runs on a closed reader, and a snapshot's
+// source always belongs to the reader that answered.
+func TestSwappableNeverLooksUpAClosedReader(t *testing.T) {
+ s := NewSwappable()
+ var closedWhileInUse atomic.Bool
+ mk := func(src string) *MMDB {
+ var inUse atomic.Int32
+ var closed atomic.Bool
+ m := &MMDB{source: src}
+ m.lookup = func(_ net.IP, result any) error {
+ inUse.Add(1)
+ defer inUse.Add(-1)
+ if closed.Load() {
+ closedWhileInUse.Store(true)
+ }
+ rec := result.(*mmdbRecord)
+ rec.Country.ISOCode = src[:2]
+ time.Sleep(time.Microsecond)
+ return nil
+ }
+ m.closeHook = func() {
+ if inUse.Load() != 0 {
+ closedWhileInUse.Store(true)
+ }
+ closed.Store(true)
+ }
+ return m
+ }
+ s.Swap(mk("AAsrc"))
+ var wg sync.WaitGroup
+ stop := make(chan struct{})
+ var mismatched atomic.Bool
+ for i := 0; i < 32; i++ {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ for {
+ select {
+ case <-stop:
+ return
+ default:
+ }
+ snap := s.Lookup(net.ParseIP("81.2.69.142"))
+ if snap.Found && snap.Result.Country != snap.Source[:2] {
+ mismatched.Store(true)
+ }
+ s.Locate(net.ParseIP("81.2.69.142"))
+ }
+ }()
+ }
+ for i := 0; i < 100; i++ {
+ if i%2 == 0 {
+ s.Swap(mk("BBsrc"))
+ } else {
+ s.Swap(mk("AAsrc"))
+ }
+ }
+ close(stop)
+ wg.Wait()
+ if closedWhileInUse.Load() {
+ t.Fatal("a reader was closed while a lookup was using it")
+ }
+ if mismatched.Load() {
+ t.Fatal("a snapshot carried a different reader's source")
+ }
+}
+```
+
+- [ ] **Step 2: Run it to see it fail**
+
+Run: `nice -n 19 go test ./internal/geo -run Swappable -race -count=1`
+Expected: build failure. `NewSwappable` and `closeHook` are undefined.
+
+- [ ] **Step 3: Implement it.** Add the `closeHook func()` field to `MMDB` (a test seam, nil in production), and make `Close` call it:
+
+```go
+// in mmdb.go, MMDB struct:
+ closeHook func() // test seam: called by Close before the mapping is released
+
+// in mmdb.go, Close:
+func (m *MMDB) Close() error {
+ if m == nil {
+ return nil
+ }
+ if m.closeHook != nil {
+ m.closeHook()
+ }
+ if m.r == nil {
+ return nil
+ }
+ return m.r.Close()
+}
+```
+
+Then create `internal/geo/swappable.go`:
+
+```go
+package geo
+
+import (
+ "net"
+ "sync"
+ "time"
+)
+
+// Snapshot is one answer from one reader: the result, that reader's source,
+// and whether any reader was loaded at all (geoip spec §5.4, codex R2-4).
+// Headers, analytics and attribution all come from one Snapshot, so a swap
+// between the lookup and the response can never relabel the data.
+type Snapshot struct {
+ Result Result
+ Found bool
+ Source string // empty only when Loaded is false
+ Loaded bool
+}
+
+// Swappable is a reader that can be replaced while lookups run. The old
+// mapping is closed only after the write lock has been taken and released,
+// so no lookup is ever in flight on an unmapped file. There is deliberately
+// no Source() method: a source read apart from its lookup is the race this
+// type exists to prevent.
+type Swappable struct {
+ mu sync.RWMutex
+ cur *MMDB
+}
+
+func NewSwappable() *Swappable { return &Swappable{} }
+
+// Lookup answers under one read lock.
+func (s *Swappable) Lookup(ip net.IP) Snapshot {
+ s.mu.RLock()
+ defer s.mu.RUnlock()
+ if s.cur == nil {
+ return Snapshot{}
+ }
+ r, ok := s.cur.Lookup(ip)
+ return Snapshot{Result: r, Found: ok, Source: s.cur.Source(), Loaded: true}
+}
+
+// Locate makes Swappable a Locator, so the console's region guess can hold
+// it without ever seeing the *MMDB underneath (codex R3-2).
+func (s *Swappable) Locate(ip net.IP) (Point, bool) {
+ s.mu.RLock()
+ defer s.mu.RUnlock()
+ if s.cur == nil {
+ return Point{}, false
+ }
+ return s.cur.Locate(ip)
+}
+
+// Loaded reports the current reader's source and build time.
+func (s *Swappable) Loaded() (source string, build time.Time, ok bool) {
+ s.mu.RLock()
+ defer s.mu.RUnlock()
+ if s.cur == nil {
+ return "", time.Time{}, false
+ }
+ return s.cur.Source(), s.cur.BuildEpoch(), true
+}
+
+// Swap installs next (nil = none) and closes the previous reader after
+// every lookup that could see it has finished. Go's RWMutex blocks new
+// readers while a writer waits, so lookups cannot starve a swap.
+func (s *Swappable) Swap(next *MMDB) {
+ s.mu.Lock()
+ old := s.cur
+ s.cur = next
+ s.mu.Unlock()
+ if old != nil && old != next {
+ old.Close()
+ }
+}
+```
+
+- [ ] **Step 4: Run the package's tests under the race detector**
+
+Run: `nice -n 19 go test ./internal/geo -race -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/geo/swappable.go internal/geo/swappable_test.go internal/geo/mmdb.go
+git commit -m "geo: Swappable reader with one-lock snapshots (geoip spec §5.4)"
+```
+
+---
+
+### Task 4: `release.PutStreamer`
+
+**Files:**
+- Modify: `internal/release/store.go` (the interface beside `Opener` at ~:226; the prefixStore methods at ~:133; the DirStore methods after `Put` at ~:374; the S3Store methods after `Put` at ~:667)
+- Test: `internal/release/putfrom_test.go`
+
+**Interfaces:**
+- Produces:
+ ```go
+ type PutStreamer interface {
+ PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
+ }
+ const PutFromFallbackMax = 64 << 20
+ func PutObjectFrom(ctx context.Context, s Store, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
+ ```
+ `PutObjectFrom` uses `PutFrom` when the store has it. Otherwise it reads up to `PutFromFallbackMax` into memory and calls `Put`, and refuses anything larger with `ErrTooLargeForPut`.
+
+The body is an `io.ReadSeeker` on purpose. The publish path always streams from a temporary file (`*os.File`), and the AWS SDK v2 signs or checksums a seekable body without buffering it.
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+package release
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "io"
+ "strings"
+ "testing"
+)
+
+func TestDirStorePutFromMatchesPutSemantics(t *testing.T) {
+ ctx := context.Background()
+ d := NewDirStore(t.TempDir())
+ var _ PutStreamer = d
+ etag, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("hello"), 5, "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ got, getEtag, err := d.Get(ctx, "a/b.bin")
+ if err != nil || string(got) != "hello" || getEtag != etag {
+ t.Fatalf("Get = (%q, %q, %v), want hello with etag %q", got, getEtag, err, etag)
+ }
+ if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("again"), 5, ""); !errors.Is(err, ErrCASConflict) {
+ t.Fatalf("create-only over an existing key = %v, want ErrCASConflict", err)
+ }
+ if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("next!"), 5, etag); err != nil {
+ t.Fatalf("replace with the right etag: %v", err)
+ }
+ if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("short"), 9, ""); err == nil {
+ t.Fatal("a body shorter than size must fail")
+ }
+}
+
+func TestPrefixStorePutFromRebases(t *testing.T) {
+ ctx := context.Background()
+ d := NewDirStore(t.TempDir())
+ p := NewPrefixStore(d, "fleets/x/")
+ ps, ok := p.(PutStreamer)
+ if !ok {
+ t.Fatal("prefixStore must forward PutFrom (codex R1-8)")
+ }
+ if _, err := ps.PutFrom(ctx, "k", strings.NewReader("v"), 1, ""); err != nil {
+ t.Fatal(err)
+ }
+ if b, _, err := d.Get(ctx, "fleets/x/k"); err != nil || string(b) != "v" {
+ t.Fatalf("rebased object = (%q, %v)", b, err)
+ }
+}
+
+type putOnlyStore struct{ Store }
+
+func TestPutObjectFromFallsBackOnlyForSmallBodies(t *testing.T) {
+ ctx := context.Background()
+ s := putOnlyStore{NewDirStore(t.TempDir())}
+ if _, err := PutObjectFrom(ctx, s, "small", bytes.NewReader([]byte("abc")), 3, ""); err != nil {
+ t.Fatalf("small fallback: %v", err)
+ }
+ big := io.NewSectionReader(zeroReaderAt{}, 0, PutFromFallbackMax+1)
+ if _, err := PutObjectFrom(ctx, s, "big", big, PutFromFallbackMax+1, ""); !errors.Is(err, ErrTooLargeForPut) {
+ t.Fatalf("big fallback = %v, want ErrTooLargeForPut", err)
+ }
+}
+
+type zeroReaderAt struct{}
+
+func (zeroReaderAt) ReadAt(p []byte, _ int64) (int, error) {
+ for i := range p {
+ p[i] = 0
+ }
+ return len(p), nil
+}
+```
+
+- [ ] **Step 2: Run it to see it fail**
+
+Run: `nice -n 19 go test ./internal/release -run 'PutFrom|PutObjectFrom' -count=1`
+Expected: build failure. `PutStreamer`, `PutFrom`, `PutObjectFrom`, `PutFromFallbackMax` and `ErrTooLargeForPut` are undefined.
+
+- [ ] **Step 3: Implement it.** Next to `Opener`:
+
+```go
+// PutStreamer is the optional streaming write: Store.Put takes the whole
+// body as []byte, which is wrong for a 130 MB GeoIP database on a box
+// under MemoryHigh. Implemented by every production store; a fake with only
+// Put still works through PutObjectFrom for bodies up to PutFromFallbackMax.
+// ifMatch has Put's exact semantics ("" = create-only).
+type PutStreamer interface {
+ PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
+}
+
+var (
+ _ PutStreamer = (*DirStore)(nil)
+ _ PutStreamer = (*S3Store)(nil)
+ _ PutStreamer = prefixStore{}
+)
+
+// PutFromFallbackMax bounds how much PutObjectFrom will buffer for a store
+// with no PutFrom.
+const PutFromFallbackMax = 64 << 20
+
+// ErrTooLargeForPut is PutObjectFrom refusing to buffer a large body.
+var ErrTooLargeForPut = errors.New("store cannot stream and the body is too large to buffer")
+
+// PutObjectFrom streams body into s at key.
+func PutObjectFrom(ctx context.Context, s Store, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
+ if ps, ok := s.(PutStreamer); ok {
+ return ps.PutFrom(ctx, key, body, size, ifMatch)
+ }
+ if size > PutFromFallbackMax {
+ return "", fmt.Errorf("%s: %w", key, ErrTooLargeForPut)
+ }
+ b, err := io.ReadAll(io.LimitReader(body, size+1))
+ if err != nil {
+ return "", err
+ }
+ if int64(len(b)) != size {
+ return "", fmt.Errorf("%s: body is %d bytes, want %d", key, len(b), size)
+ }
+ return s.Put(ctx, key, b, ifMatch)
+}
+```
+
+prefixStore (written out for the same reason as `Open`):
+
+```go
+// PutFrom is written out, not promoted, for the reason Open is: a promoted
+// method would forward the UNREBASED key.
+func (p prefixStore) PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
+ return PutObjectFrom(ctx, p.Store, p.full(key), body, size, ifMatch)
+}
+```
+
+DirStore:
+
+```go
+// PutFrom is Put streamed: the body goes to a temp file (hashed as it
+// goes, so the etag is the same content sha256 Put would give) and is
+// renamed into place under d.mu after the same CAS gate.
+func (d *DirStore) PutFrom(_ context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
+ p, err := d.path(key)
+ if err != nil {
+ return "", err
+ }
+ if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+ return "", err
+ }
+ tmp, err := os.CreateTemp(filepath.Dir(p), filepath.Base(p)+".*.tmp")
+ if err != nil {
+ return "", err
+ }
+ defer os.Remove(tmp.Name())
+ h := sha256.New()
+ n, err := io.Copy(io.MultiWriter(tmp, h), io.LimitReader(body, size+1))
+ if cerr := tmp.Close(); err == nil {
+ err = cerr
+ }
+ if err != nil {
+ return "", err
+ }
+ if n != size {
+ return "", fmt.Errorf("dirstore: %s: body is %d bytes, want %d", key, n, size)
+ }
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ cur, err := os.ReadFile(p)
+ exists := err == nil
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return "", err
+ }
+ if ifMatch == "" && exists {
+ return "", fmt.Errorf("dirstore: %s exists: %w", key, ErrCASConflict)
+ }
+ if ifMatch != "" && (!exists || etagOf(cur) != ifMatch) {
+ return "", fmt.Errorf("dirstore: %s moved on: %w", key, ErrCASConflict)
+ }
+ if err := os.Chmod(tmp.Name(), 0o644); err != nil {
+ return "", err
+ }
+ if err := os.Rename(tmp.Name(), p); err != nil {
+ return "", err
+ }
+ return hex.EncodeToString(h.Sum(nil)), nil
+}
+```
+
+`ListMod` already excludes `*.tmp` names, so the `.<rand>.tmp` temp file is never listed. Confirm that by reading the comment at ~:376, and if the filter matches only the `.tmp` suffix, the name above already satisfies it.
+
+S3Store:
+
+```go
+// PutFrom is Put with a streamed, seekable body and an explicit length.
+func (s *S3Store) PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
+ in := &awss3.PutObjectInput{
+ Bucket: &s.Bucket,
+ Key: &key,
+ Body: body,
+ ContentLength: aws.Int64(size),
+ }
+ if ifMatch == "" {
+ in.IfNoneMatch = aws.String("*")
+ } else {
+ in.IfMatch = aws.String(ifMatch)
+ }
+ out, err := s.Client.PutObject(ctx, in)
+ if err != nil {
+ if s3CASConflict(err) || (ifMatch != "" && s3NotFound(err)) {
+ return "", fmt.Errorf("s3store: %s: %w", key, ErrCASConflict)
+ }
+ return "", err
+ }
+ return aws.ToString(out.ETag), nil
+}
+```
+
+`internal/fleetapi.store` deliberately gets no `PutFrom`. The console never uses it, and it would need a server side in `internal/edge/storedoor.go`. Add a one-line comment saying so beside its `Opener` assert (`internal/fleetapi/fleetapi.go:179`).
+
+- [ ] **Step 4: Run the package's tests**
+
+Run: `nice -n 19 go test ./internal/release -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/release/store.go internal/release/putfrom_test.go internal/fleetapi/fleetapi.go
+git commit -m "release: PutStreamer for S3, dir and prefix stores (geoip spec §4.1)"
+```
+
+---
+
+### Task 5: `internal/geoip` — the shared vocabulary
+
+**Files:**
+- Create: `internal/geoip/pointer.go`, `internal/geoip/answer.go`
+- Test: `internal/geoip/pointer_test.go`, `internal/geoip/answer_test.go`
+
+**Interfaces:**
+- Consumes: `geo.Snapshot` (Task 3).
+- Produces:
+ ```go
+ const Prefix = "control/geoip/"; PointerKey; FeedKey; MaxBlobBytes int64 = 1<<30; MaxPointerBytes = 64<<10
+ func BlobKey(sha string) string
+ type Pointer struct{ V int; SHA256, Key string; Size int64; DatabaseType string; BuildEpoch int64; NodeCount uint64; Published time.Time; Origin, Edition string; Expired bool }
+ func ParsePointer(raw []byte) (Pointer, error)
+ func (p Pointer) Build() time.Time
+ func ExpiresAt(dbType string, build time.Time) (time.Time, bool)
+ func Expired(dbType string, build, now time.Time) bool
+ type Attribution struct{ Text, URL string }
+ func AttributionFor(dbType string) (Attribution, bool)
+ type State string; const StateHit, StateMiss, StateOff
+ type Answer struct{ State State; Country, Subdivision, Source string; Lat, Lon float64; HasPoint bool }
+ func FromSnapshot(s geo.Snapshot) Answer
+ func Off() Answer
+ const HeaderCountry, HeaderSubdivision, HeaderLat, HeaderLon, HeaderSource, HeaderHop
+ func StripAll(h http.Header)
+ func (a Answer) WriteApp(h http.Header) // deletes, then writes the app headers
+ func (a Answer) HopValue() string // "hit;src=…", "miss;src=…", "off"
+ type Captured struct{ Hop string; Country, Subdivision, Lat, Lon, Source string }
+ func Capture(h http.Header) Captured // copies the values, then StripAll
+ func (c Captured) Adopt() Answer // validates; anything invalid becomes Off or drops the field
+ ```
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/geoip/pointer_test.go`:
+
+```go
+package geoip
+
+import (
+ "strings"
+ "testing"
+ "time"
+)
+
+const sha = "9f2c000000000000000000000000000000000000000000000000000000000001"
+
+func TestParsePointerAcceptsTheSpecShape(t *testing.T) {
+ raw := `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":10,
+ "database_type":"DBIP-City-Lite","build_epoch":1790812800,"node_count":7,
+ "published":"2026-10-06T21:00:00Z","origin":"manual","edition":""}`
+ p, err := ParsePointer([]byte(raw))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if p.Key != BlobKey(sha) || p.Size != 10 || p.DatabaseType != "DBIP-City-Lite" || p.Build().Unix() != 1790812800 {
+ t.Fatalf("pointer = %+v", p)
+ }
+}
+
+func TestParsePointerRefusesWhatAnEdgeMustNotFollow(t *testing.T) {
+ good := `"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":10,"database_type":"X","build_epoch":1`
+ bad := map[string]string{
+ "unknown version": `{"v":2,` + good + `}`,
+ "short sha": `{"v":1,"sha256":"9f","key":"control/geoip/9f.mmdb","size":1,"database_type":"X","build_epoch":1}`,
+ "key not the blob key": `{"v":1,"sha256":"` + sha + `","key":"apps/x/secret","size":10,"database_type":"X","build_epoch":1}`,
+ "zero size": `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":0,"database_type":"X","build_epoch":1}`,
+ "oversize": `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":1073741825,"database_type":"X","build_epoch":1}`,
+ "not json": `nope`,
+ }
+ for name, raw := range bad {
+ if _, err := ParsePointer([]byte(raw)); err == nil {
+ t.Errorf("%s: accepted", name)
+ }
+ }
+}
+
+func TestGeoLiteExpiresThirtyDaysAfterBuild(t *testing.T) {
+ build := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+ at, ok := ExpiresAt("GeoLite2-City", build)
+ if !ok || !at.Equal(build.Add(30*24*time.Hour)) {
+ t.Fatalf("ExpiresAt = (%v, %v)", at, ok)
+ }
+ if Expired("GeoLite2-City", build, build.Add(30*24*time.Hour-time.Second)) {
+ t.Fatal("expired a second early")
+ }
+ if !Expired("GeoLite2-Country", build, build.Add(30*24*time.Hour)) {
+ t.Fatal("not expired at 30 days")
+ }
+ if Expired("DBIP-City-Lite", build, build.Add(400*24*time.Hour)) {
+ t.Fatal("DB-IP must never expire (D1)")
+ }
+ if _, ok := ExpiresAt("GeoIP2-City", build); ok {
+ t.Fatal("only GeoLite2- expires")
+ }
+}
+
+func TestAttributionTable(t *testing.T) {
+ if a, ok := AttributionFor("DBIP-City-Lite"); !ok || a.Text != "IP Geolocation by DB-IP" || a.URL != "https://db-ip.com" {
+ t.Fatalf("dbip = %+v %v", a, ok)
+ }
+ if a, ok := AttributionFor("GeoLite2-Country"); !ok || !strings.Contains(a.Text, "GeoLite2 data created by MaxMind") {
+ t.Fatalf("geolite = %+v %v", a, ok)
+ }
+ if _, ok := AttributionFor("GeoIP2-City"); ok {
+ t.Fatal("no attribution is known for commercial GeoIP2; the operator's licence governs")
+ }
+}
+```
+
+`internal/geoip/answer_test.go`:
+
+```go
+package geoip
+
+import (
+ "net/http"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/geo"
+)
+
+func TestFromSnapshotStates(t *testing.T) {
+ if a := FromSnapshot(geo.Snapshot{}); a.State != StateOff {
+ t.Fatalf("not loaded = %+v, want off", a)
+ }
+ if a := FromSnapshot(geo.Snapshot{Loaded: true, Source: "DBIP-City-Lite"}); a.State != StateMiss || a.Source != "DBIP-City-Lite" {
+ t.Fatalf("loaded miss = %+v", a)
+ }
+ hit := FromSnapshot(geo.Snapshot{Loaded: true, Found: true, Source: "GeoLite2-City",
+ Result: geo.Result{Country: "GB", Subdivision: "GB-ENG", Point: geo.Point{Lat: 51.5074, Lon: -0.1278}, HasPoint: true}})
+ if hit.State != StateHit || hit.Lat != 51.51 || hit.Lon != -0.13 {
+ t.Fatalf("hit = %+v (coordinates must be rounded to 2 places)", hit)
+ }
+}
+
+func TestWriteAppHeaders(t *testing.T) {
+ h := http.Header{}
+ h.Set(HeaderCountry, "XX") // stale value must go
+ Answer{State: StateHit, Country: "IE", Source: "DBIP-City-Lite", Lat: 53.34, Lon: -6.27, HasPoint: true}.WriteApp(h)
+ if h.Get(HeaderCountry) != "IE" || h.Get(HeaderLat) != "53.34" || h.Get(HeaderLon) != "-6.27" ||
+ h.Get(HeaderSource) != "DBIP-City-Lite" || h.Get(HeaderSubdivision) != "" || h.Get(HeaderHop) != "" {
+ t.Fatalf("headers = %v", h)
+ }
+ h2 := http.Header{}
+ h2.Set(HeaderSource, "forged")
+ Answer{State: StateMiss, Source: "DBIP-City-Lite"}.WriteApp(h2)
+ if len(h2) != 0 {
+ t.Fatalf("a miss sends no app headers, not even Source: %v", h2)
+ }
+}
+
+func TestHopValueRoundTrip(t *testing.T) {
+ for _, a := range []Answer{
+ {State: StateHit, Country: "GB", Subdivision: "GB-ENG", Source: "GeoLite2-City", Lat: 51.51, Lon: -0.09, HasPoint: true},
+ {State: StateHit, Source: "GeoLite2-City", Lat: 10, Lon: 20, HasPoint: true}, // coordinate-only (codex R3-3)
+ {State: StateMiss, Source: "DBIP-City-Lite"},
+ Off(),
+ } {
+ h := http.Header{}
+ a.WriteApp(h)
+ h.Set(HeaderHop, a.HopValue())
+ got := Capture(h).Adopt()
+ if got != a {
+ t.Errorf("round trip %+v -> %+v", a, got)
+ }
+ if len(h) != 0 {
+ t.Errorf("Capture must strip every geo header, left %v", h)
+ }
+ }
+}
+
+func TestAdoptRefusesWhatItCannotTrust(t *testing.T) {
+ cases := map[string]http.Header{
+ "no hop value": {HeaderCountry: {"GB"}, HeaderSource: {"x"}},
+ "malformed hop value": {HeaderHop: {"maybe"}},
+ "hit with nothing": {HeaderHop: {"hit;src=x"}},
+ "hit with bad country": {HeaderHop: {"hit;src=x"}, HeaderCountry: {"gb"}},
+ "bad source": {HeaderHop: {"hit;src=a b"}, HeaderCountry: {"GB"}},
+ }
+ for name, h := range cases {
+ if got := Capture(h).Adopt(); got.State != StateOff {
+ t.Errorf("%s: adopted %+v, want off", name, got)
+ }
+ }
+ // Field-level: an invalid subdivision or half a coordinate pair is
+ // dropped on its own, the rest survives.
+ h := http.Header{HeaderHop: {"hit;src=GeoLite2-City"}, HeaderCountry: {"GB"},
+ HeaderSubdivision: {"FR-IDF"}, HeaderLat: {"91"}, HeaderLon: {"0"}}
+ got := Capture(h).Adopt()
+ if got.State != StateHit || got.Country != "GB" || got.Subdivision != "" || got.HasPoint {
+ t.Fatalf("partial = %+v", got)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/geoip -count=1`
+Expected: build failure, because the package doesn't exist yet.
+
+- [ ] **Step 3: Implement it.** `internal/geoip/pointer.go`:
+
+```go
+// Package geoip is the platform's GeoIP vocabulary (spec
+// docs/superpowers/specs/2026-10-06-platform-geoip-design.md): the bucket
+// records the console writes and every edge reads, the GeoLite2 expiry rule
+// (D1), attribution, and the request headers. No I/O lives here.
+package geoip
+
+import (
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "strings"
+ "time"
+)
+
+const (
+ Prefix = "control/geoip/"
+ PointerKey = Prefix + "current.json"
+ FeedKey = Prefix + "feed.json"
+ MaxBlobBytes = int64(1 << 30)
+ MaxPointerBytes = 64 << 10
+
+ geoLiteTTL = 30 * 24 * time.Hour
+ GeoLiteWarn = 21 * 24 * time.Hour
+ StaleAfter = 45 * 24 * time.Hour
+)
+
+// BlobKey is the content-addressed object for a database whose bytes hash
+// to sha (lower-case hex).
+func BlobKey(sha string) string { return Prefix + sha + ".mmdb" }
+
+// Pointer is control/geoip/current.json (spec §3).
+type Pointer struct {
+ V int `json:"v"`
+ SHA256 string `json:"sha256"`
+ Key string `json:"key"`
+ Size int64 `json:"size"`
+ DatabaseType string `json:"database_type"`
+ BuildEpoch int64 `json:"build_epoch"`
+ NodeCount uint64 `json:"node_count"`
+ Published time.Time `json:"published"`
+ Origin string `json:"origin"`
+ Edition string `json:"edition"`
+ Expired bool `json:"expired,omitempty"`
+}
+
+// Build is BuildEpoch as a time.
+func (p Pointer) Build() time.Time { return time.Unix(p.BuildEpoch, 0).UTC() }
+
+// ParsePointer decodes and checks a pointer. An unknown version, a key that
+// is not this sha's blob key, or an impossible size is an error: an edge
+// never follows a pointer it cannot fully vouch for.
+func ParsePointer(raw []byte) (Pointer, error) {
+ var p Pointer
+ if err := json.Unmarshal(raw, &p); err != nil {
+ return Pointer{}, fmt.Errorf("geoip pointer: %w", err)
+ }
+ if p.V != 1 {
+ return Pointer{}, fmt.Errorf("geoip pointer: unknown version %d", p.V)
+ }
+ if !IsSHA256Hex(p.SHA256) {
+ return Pointer{}, errors.New("geoip pointer: sha256 is not 64 lower-case hex characters")
+ }
+ if p.Key != BlobKey(p.SHA256) {
+ return Pointer{}, errors.New("geoip pointer: key is not the sha's blob key")
+ }
+ if p.Size <= 0 || p.Size > MaxBlobBytes {
+ return Pointer{}, fmt.Errorf("geoip pointer: size %d out of range", p.Size)
+ }
+ return p, nil
+}
+
+// IsSHA256Hex is 64 lower-case hex characters.
+func IsSHA256Hex(s string) bool {
+ if len(s) != 64 || strings.ToLower(s) != s {
+ return false
+ }
+ _, err := hex.DecodeString(s)
+ return err == nil
+}
+
+// ExpiresAt is D1: a GeoLite2 database expires 30 days after it was built.
+// ok is false for every other type, which never expires.
+func ExpiresAt(dbType string, build time.Time) (time.Time, bool) {
+ if !strings.HasPrefix(dbType, "GeoLite2-") {
+ return time.Time{}, false
+ }
+ return build.Add(geoLiteTTL), true
+}
+
+// Expired reports whether now is at or past ExpiresAt.
+func Expired(dbType string, build, now time.Time) bool {
+ at, ok := ExpiresAt(dbType, build)
+ return ok && !now.Before(at)
+}
+
+// Attribution is the credit a page showing the data must carry (spec §13).
+type Attribution struct{ Text, URL string }
+
+// AttributionFor names the attribution for a database type. ok is false
+// when none is known: the operator's own licence governs.
+func AttributionFor(dbType string) (Attribution, bool) {
+ switch {
+ case strings.HasPrefix(dbType, "DBIP-"):
+ return Attribution{"IP Geolocation by DB-IP", "https://db-ip.com"}, true
+ case strings.HasPrefix(dbType, "GeoLite2-"):
+ return Attribution{"This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com", "https://www.maxmind.com"}, true
+ }
+ return Attribution{}, false
+}
+```
+
+`internal/geoip/answer.go`:
+
+```go
+package geoip
+
+import (
+ "math"
+ "net/http"
+ "regexp"
+ "strconv"
+ "strings"
+
+ "github.com/carlosframework/platform/internal/geo"
+)
+
+const (
+ HeaderCountry = "X-Carlos-Geo-Country"
+ HeaderSubdivision = "X-Carlos-Geo-Subdivision"
+ HeaderLat = "X-Carlos-Geo-Lat"
+ HeaderLon = "X-Carlos-Geo-Lon"
+ HeaderSource = "X-Carlos-Geo-Source"
+ // HeaderHop is edge-to-edge only (spec §8.1a): never written to a
+ // request an app receives.
+ HeaderHop = "X-Carlos-Hop-Geo"
+)
+
+var appHeaders = []string{HeaderCountry, HeaderSubdivision, HeaderLat, HeaderLon, HeaderSource}
+
+// State is what the ingress edge did: hit (a reader answered), miss (a
+// reader was loaded and found nothing, or the address is not locatable),
+// off (no lookup was attempted).
+type State string
+
+const (
+ StateHit State = "hit"
+ StateMiss State = "miss"
+ StateOff State = "off"
+)
+
+// Answer is one request's geo result, computed once at ingress or adopted
+// from a trusted transport. Lat/Lon are already rounded to 2 places.
+type Answer struct {
+ State State
+ Country string
+ Subdivision string
+ Source string
+ Lat, Lon float64
+ HasPoint bool
+}
+
+// Off is the answer for a request no lookup was attempted for.
+func Off() Answer { return Answer{State: StateOff} }
+
+func round2(v float64) float64 { return math.Round(v*100) / 100 }
+
+// FromSnapshot turns one reader's answer into a request's answer.
+func FromSnapshot(s geo.Snapshot) Answer {
+ switch {
+ case !s.Loaded:
+ return Off()
+ case !s.Found:
+ return Answer{State: StateMiss, Source: s.Source}
+ }
+ a := Answer{State: StateHit, Country: s.Result.Country, Subdivision: s.Result.Subdivision, Source: s.Source}
+ if s.Result.HasPoint {
+ a.Lat, a.Lon, a.HasPoint = round2(s.Result.Point.Lat), round2(s.Result.Point.Lon), true
+ }
+ return a
+}
+
+// StripAll deletes every geo header an app or an edge could receive.
+func StripAll(h http.Header) {
+ for _, k := range appHeaders {
+ h.Del(k)
+ }
+ h.Del(HeaderHop)
+}
+
+// WriteApp deletes the app headers and writes this answer's. A miss and
+// off write nothing at all; Source is written only beside another field.
+func (a Answer) WriteApp(h http.Header) {
+ for _, k := range appHeaders {
+ h.Del(k)
+ }
+ if a.State != StateHit {
+ return
+ }
+ wrote := false
+ if a.Country != "" {
+ h.Set(HeaderCountry, a.Country)
+ wrote = true
+ if a.Subdivision != "" {
+ h.Set(HeaderSubdivision, a.Subdivision)
+ }
+ }
+ if a.HasPoint {
+ h.Set(HeaderLat, strconv.FormatFloat(a.Lat, 'f', 2, 64))
+ h.Set(HeaderLon, strconv.FormatFloat(a.Lon, 'f', 2, 64))
+ wrote = true
+ }
+ if wrote && a.Source != "" {
+ h.Set(HeaderSource, a.Source)
+ }
+}
+
+// HopValue is X-Carlos-Hop-Geo for this answer.
+func (a Answer) HopValue() string {
+ if a.State == StateOff || !sourceRE.MatchString(a.Source) {
+ return string(StateOff)
+ }
+ return string(a.State) + ";src=" + a.Source
+}
+
+// Captured is what a request arrived carrying, before anyone trusts it.
+type Captured struct {
+ Hop, Country, Subdivision, Lat, Lon, Source string
+}
+
+// Capture copies the geo headers out of h and strips them all (spec §8.1
+// step 1). Nothing reads the copy until the transport is authenticated.
+func Capture(h http.Header) Captured {
+ c := Captured{
+ Hop: h.Get(HeaderHop), Country: h.Get(HeaderCountry), Subdivision: h.Get(HeaderSubdivision),
+ Lat: h.Get(HeaderLat), Lon: h.Get(HeaderLon), Source: h.Get(HeaderSource),
+ }
+ StripAll(h)
+ return c
+}
+
+var (
+ countryRE = regexp.MustCompile(`^[A-Z]{2}$`)
+ subdivisionRE = regexp.MustCompile(`^[A-Z]{2}-[A-Z0-9]{1,3}$`)
+ sourceRE = regexp.MustCompile(`^[A-Za-z0-9._-]{1,64}$`)
+)
+
+// Adopt validates what a TRUSTED transport carried (spec §8.4). Call it only
+// for a verified hop or a TLS tunnel. Invalid fields are dropped one by one;
+// a hit left with neither a country nor a coordinate pair, or a missing or
+// malformed hop value, is off.
+func (c Captured) Adopt() Answer {
+ state, src, ok := strings.Cut(c.Hop, ";src=")
+ switch {
+ case c.Hop == string(StateOff):
+ return Off()
+ case !ok || !sourceRE.MatchString(src):
+ return Off()
+ case State(state) == StateMiss:
+ return Answer{State: StateMiss, Source: src}
+ case State(state) != StateHit:
+ return Off()
+ }
+ a := Answer{State: StateHit, Source: src}
+ if countryRE.MatchString(c.Country) {
+ a.Country = c.Country
+ if subdivisionRE.MatchString(c.Subdivision) && strings.HasPrefix(c.Subdivision, c.Country+"-") {
+ a.Subdivision = c.Subdivision
+ }
+ }
+ if lat, lon, ok := parsePair(c.Lat, c.Lon); ok {
+ a.Lat, a.Lon, a.HasPoint = lat, lon, true
+ }
+ if a.Country == "" && !a.HasPoint {
+ return Off()
+ }
+ return a
+}
+
+func parsePair(latS, lonS string) (float64, float64, bool) {
+ lat, err1 := strconv.ParseFloat(latS, 64)
+ lon, err2 := strconv.ParseFloat(lonS, 64)
+ if err1 != nil || err2 != nil || math.IsNaN(lat) || math.IsNaN(lon) || math.IsInf(lat, 0) || math.IsInf(lon, 0) ||
+ lat < -90 || lat > 90 || lon < -180 || lon > 180 {
+ return 0, 0, false
+ }
+ return round2(lat), round2(lon), true
+}
+```
+
+- [ ] **Step 4: Run the package's tests**
+
+Run: `nice -n 19 go test ./internal/geoip -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/geoip/
+git commit -m "geoip: pointer, D1 expiry, attribution and the header vocabulary (geoip spec §3, §7, §8.1a)"
+```
+
+---
+
+### Task 6: The console's publish path
+
+**Files:**
+- Create: `internal/console/geoippublish.go`
+- Test: `internal/console/geoippublish_test.go`
+
+**Interfaces:**
+- Consumes:
+ - `geo.OpenMMDB`, `(*geo.MMDB).Lookup/Source/BuildEpoch/NodeCount/Close` (Task 2);
+ - `release.PutObjectFrom`, `release.OpenObject` (Task 4);
+ - `geoip.Pointer/ParsePointer/BlobKey/PointerKey/Prefix/MaxBlobBytes/Expired/IsSHA256Hex` (Task 5).
+- Produces:
+ ```go
+ type geoipPublisher struct{ ... }
+ func newGeoipPublisher(store release.Store, tmpRoot string) *geoipPublisher
+ type geoipPublishOpts struct{ Origin, Edition string; Force bool }
+ var errGeoipBusy, errGeoipRefused // errors.Is-able; refusals wrap errGeoipRefused with a category
+ func (p *geoipPublisher) Publish(ctx context.Context, src io.Reader, o geoipPublishOpts) (geoip.Pointer, error)
+ func (p *geoipPublisher) Off(ctx context.Context) error
+ func (p *geoipPublisher) Sweep(ctx context.Context) error
+ func (p *geoipPublisher) ExpireIfDue(ctx context.Context, now time.Time) (bool, error)
+ func (p *geoipPublisher) Current(ctx context.Context) (geoip.Pointer, string, bool, error) // pointer, etag, present
+ var geoipProbeAddrs = []string{"8.8.8.8","1.1.1.1","9.9.9.9","208.67.222.222","2001:4860:4860::8888"}
+ ```
+
+- [ ] **Step 1: Write the failing tests**
+
+```go
+package console
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "os"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The repo's fixtures hold none of the production probe addresses
+// (FACTS.md), so tests swap in addresses the fixtures do resolve.
+func withFixtureProbes(t *testing.T) {
+ prev := geoipProbeAddrs
+ geoipProbeAddrs = []string{"81.2.69.142", "216.160.83.56", "89.160.20.112", "2001:218::1", "67.43.156.1"}
+ t.Cleanup(func() { geoipProbeAddrs = prev })
+}
+
+func fixture(t *testing.T, name string) []byte {
+ b, err := os.ReadFile("../geo/testdata/" + name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return b
+}
+
+func newTestPublisher(t *testing.T) (*geoipPublisher, release.Store) {
+ s := release.NewDirStore(t.TempDir())
+ return newGeoipPublisher(s, t.TempDir()), s
+}
+
+func blobKeys(t *testing.T, s release.Store) []string {
+ keys, err := s.List(context.Background(), geoip.Prefix)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var out []string
+ for _, k := range keys {
+ if strings.HasSuffix(k, ".mmdb") {
+ out = append(out, k)
+ }
+ }
+ return out
+}
+
+func TestPublishWritesBlobThenPointer(t *testing.T) {
+ withFixtureProbes(t)
+ p, s := newTestPublisher(t)
+ ptr, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if ptr.DatabaseType != "GeoIP2-City" || ptr.NodeCount != 1547 || ptr.Origin != "manual" || ptr.Key != geoip.BlobKey(ptr.SHA256) {
+ t.Fatalf("pointer = %+v", ptr)
+ }
+ raw, _, err := s.Get(context.Background(), geoip.PointerKey)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got, err := geoip.ParsePointer(raw); err != nil || got.SHA256 != ptr.SHA256 {
+ t.Fatalf("stored pointer = (%+v, %v)", got, err)
+ }
+ if keys := blobKeys(t, s); len(keys) != 1 || keys[0] != ptr.Key {
+ t.Fatalf("blobs = %v", keys)
+ }
+}
+
+func TestPublishRefusals(t *testing.T) {
+ withFixtureProbes(t)
+ ctx := context.Background()
+ p, _ := newTestPublisher(t)
+ for name, body := range map[string][]byte{
+ "not a database": []byte("junk"),
+ "no country data": fixture(t, "MaxMind-DB-test-ipv4-24.mmdb"),
+ } {
+ if _, err := p.Publish(ctx, bytes.NewReader(body), geoipPublishOpts{Origin: "manual"}); !errors.Is(err, errGeoipRefused) {
+ t.Errorf("%s: err = %v, want errGeoipRefused", name, err)
+ }
+ }
+}
+
+func TestPublishShrinkCheckOnlyForTheSameType(t *testing.T) {
+ withFixtureProbes(t)
+ ctx := context.Background()
+ p, s := newTestPublisher(t)
+ if _, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-Country-Test.mmdb")), geoipPublishOpts{Origin: "manual"}); err != nil {
+ t.Fatal(err)
+ }
+ // Inflate the recorded node_count so a same-type republish looks like a
+ // truncation (fewer than half).
+ raw, etag, _ := s.Get(ctx, geoip.PointerKey)
+ inflated := bytes.Replace(raw, []byte(`"node_count": 1704`), []byte(`"node_count": 99999`), 1)
+ if bytes.Equal(inflated, raw) {
+ t.Fatalf("pointer JSON shape changed; adjust the test: %s", raw)
+ }
+ if _, err := s.Put(ctx, geoip.PointerKey, inflated, etag); err != nil {
+ t.Fatal(err)
+ }
+ same := fixture(t, "GeoIP2-Country-Test.mmdb")
+ if _, err := p.Publish(ctx, bytes.NewReader(same), geoipPublishOpts{Origin: "manual"}); !errors.Is(err, errGeoipRefused) {
+ t.Fatalf("same-type shrink = %v, want refused", err)
+ }
+ if _, err := p.Publish(ctx, bytes.NewReader(same), geoipPublishOpts{Origin: "manual", Force: true}); err != nil {
+ t.Fatalf("forced = %v", err)
+ }
+ // A different type skips the check entirely.
+ if _, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"}); err != nil {
+ t.Fatalf("different type = %v", err)
+ }
+}
+
+func TestPublishSameFileTwiceAndXYXKeepsTheCurrentBlob(t *testing.T) {
+ withFixtureProbes(t)
+ ctx := context.Background()
+ p, s := newTestPublisher(t)
+ x, y := fixture(t, "GeoIP2-City-Test.mmdb"), fixture(t, "GeoIP2-Country-Test.mmdb")
+ for i, body := range [][]byte{x, x, y, x} {
+ ptr, err := p.Publish(ctx, bytes.NewReader(body), geoipPublishOpts{Origin: "manual", Force: true})
+ if err != nil {
+ t.Fatalf("publish %d: %v", i, err)
+ }
+ keys := blobKeys(t, s)
+ if len(keys) != 1 || keys[0] != ptr.Key {
+ t.Fatalf("after publish %d blobs = %v, want only %s", i, keys, ptr.Key)
+ }
+ }
+}
+
+func TestPublishIsOneAtATime(t *testing.T) {
+ withFixtureProbes(t)
+ p, _ := newTestPublisher(t)
+ p.mu.Lock() // a publish in progress
+ _, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
+ p.mu.Unlock()
+ if !errors.Is(err, errGeoipBusy) {
+ t.Fatalf("concurrent publish = %v, want errGeoipBusy", err)
+ }
+}
+
+func TestPublishRefusesOversizeWhileCopying(t *testing.T) {
+ withFixtureProbes(t)
+ p, _ := newTestPublisher(t)
+ p.maxBytes = 1024
+ _, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
+ if !errors.Is(err, errGeoipRefused) {
+ t.Fatalf("oversize = %v", err)
+ }
+ if entries, _ := os.ReadDir(p.tmpRoot); len(entries) != 0 {
+ t.Fatalf("temp left behind: %v", entries)
+ }
+}
+
+func TestOffAndSweep(t *testing.T) {
+ withFixtureProbes(t)
+ ctx := context.Background()
+ p, s := newTestPublisher(t)
+ ptr, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ // An orphan from a crashed publish (spec §3 orphan sweep).
+ orphan := geoip.BlobKey(strings.Repeat("a", 64))
+ if _, err := s.Put(ctx, orphan, []byte("x"), ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.Sweep(ctx); err != nil {
+ t.Fatal(err)
+ }
+ if keys := blobKeys(t, s); len(keys) != 1 || keys[0] != ptr.Key {
+ t.Fatalf("after sweep = %v", keys)
+ }
+ if err := p.Off(ctx); err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := s.Get(ctx, geoip.PointerKey); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("pointer after off: %v", err)
+ }
+ if keys := blobKeys(t, s); len(keys) != 0 {
+ t.Fatalf("blobs after off = %v", keys)
+ }
+}
+
+func TestExpireIfDueMarksGeoLiteExpired(t *testing.T) {
+ ctx := context.Background()
+ p, s := newTestPublisher(t)
+ sha := strings.Repeat("b", 64)
+ ptr := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: 1, DatabaseType: "GeoLite2-City",
+ BuildEpoch: time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC).Unix(), Origin: "manual"}
+ putPointer(t, s, ptr)
+ if _, err := s.Put(ctx, ptr.Key, []byte("x"), ""); err != nil {
+ t.Fatal(err)
+ }
+ if did, err := p.ExpireIfDue(ctx, time.Date(2026, 9, 20, 0, 0, 0, 0, time.UTC)); did || err != nil {
+ t.Fatalf("early = (%v, %v)", did, err)
+ }
+ if did, err := p.ExpireIfDue(ctx, time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)); !did || err != nil {
+ t.Fatalf("due = (%v, %v)", did, err)
+ }
+ got, _, present, err := p.Current(ctx)
+ if err != nil || !present || !got.Expired {
+ t.Fatalf("pointer = (%+v, %v, %v)", got, present, err)
+ }
+ if keys := blobKeys(t, s); len(keys) != 0 {
+ t.Fatalf("blob not destroyed: %v", keys)
+ }
+}
+
+func putPointer(t *testing.T, s release.Store, p geoip.Pointer) {
+ t.Helper()
+ b, err := marshalPointer(p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Put(context.Background(), geoip.PointerKey, b, ""); err != nil {
+ t.Fatal(err)
+ }
+}
+
+var _ = sync.Mutex{}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/console -run 'Publish|OffAndSweep|ExpireIfDue' -count=1`
+Expected: build failure, because the publisher is undefined.
+
+- [ ] **Step 3: Implement it.** `internal/console/geoippublish.go`:
+
+```go
+package console
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "log"
+ "net"
+ "os"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geo"
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// geoipProbeAddrs is the country probe (spec §4.1 step 4): at least three
+// must answer with a country. A variable so tests can use addresses the
+// MaxMind fixtures hold (FACTS.md).
+var geoipProbeAddrs = []string{"8.8.8.8", "1.1.1.1", "9.9.9.9", "208.67.222.222", "2001:4860:4860::8888"}
+
+const geoipProbeNeed = 3
+
+var (
+ errGeoipBusy = errors.New("a geoip publish is already in progress")
+ errGeoipRefused = errors.New("geoip database refused")
+)
+
+func refused(category string) error { return fmt.Errorf("%w: %s", errGeoipRefused, category) }
+
+// geoipPublisher is the console's single writer of control/geoip/ (spec §3).
+// mu is held through cleanup by every publish, off, sweep and expiry, and a
+// publish that finds it held answers errGeoipBusy rather than queueing.
+type geoipPublisher struct {
+ mu sync.Mutex
+ store release.Store
+ tmpRoot string
+ maxBytes int64
+ deadline time.Duration
+ now func() time.Time
+}
+
+type geoipPublishOpts struct {
+ Origin, Edition string
+ Force bool
+}
+
+func newGeoipPublisher(store release.Store, tmpRoot string) *geoipPublisher {
+ return &geoipPublisher{store: store, tmpRoot: tmpRoot, maxBytes: geoip.MaxBlobBytes,
+ deadline: 15 * time.Minute, now: time.Now}
+}
+
+func marshalPointer(p geoip.Pointer) ([]byte, error) { return json.MarshalIndent(p, "", " ") }
+
+// Current reads the pointer. present=false means no pointer (feature off).
+func (p *geoipPublisher) Current(ctx context.Context) (geoip.Pointer, string, bool, error) {
+ raw, etag, err := p.store.Get(ctx, geoip.PointerKey)
+ if errors.Is(err, os.ErrNotExist) {
+ return geoip.Pointer{}, "", false, nil
+ }
+ if err != nil {
+ return geoip.Pointer{}, "", false, err
+ }
+ ptr, err := geoip.ParsePointer(raw)
+ if err != nil {
+ return geoip.Pointer{}, etag, true, err
+ }
+ return ptr, etag, true, nil
+}
+
+// Publish streams src to a temp file, verifies it, and publishes it.
+func (p *geoipPublisher) Publish(ctx context.Context, src io.Reader, o geoipPublishOpts) (geoip.Pointer, error) {
+ if !p.mu.TryLock() {
+ return geoip.Pointer{}, errGeoipBusy
+ }
+ defer p.mu.Unlock()
+ ctx, cancel := context.WithTimeout(ctx, p.deadline)
+ defer cancel()
+
+ dir, err := os.MkdirTemp(p.tmpRoot, "geoip-publish-*")
+ if err != nil {
+ return geoip.Pointer{}, err
+ }
+ defer os.RemoveAll(dir)
+ path := dir + "/db.mmdb"
+ sha, size, err := copyCapped(ctx, path, src, p.maxBytes)
+ if err != nil {
+ return geoip.Pointer{}, err
+ }
+ db, err := geo.OpenMMDB(path)
+ if err != nil {
+ return geoip.Pointer{}, refused("not a valid MaxMind-format database")
+ }
+ defer db.Close()
+ if !hasCountryData(db) {
+ return geoip.Pointer{}, refused("the database has no country data")
+ }
+ cur, etag, present, err := p.Current(ctx)
+ if err != nil && present {
+ log.Printf("console: geoip publish: the current pointer is unreadable and will be replaced")
+ } else if err != nil {
+ return geoip.Pointer{}, err
+ }
+ if present && err == nil && !o.Force && cur.DatabaseType == db.Source() && db.NodeCount() < cur.NodeCount/2 {
+ return geoip.Pointer{}, refused("fewer than half the current database's nodes (use -force if intended)")
+ }
+ next := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: size,
+ DatabaseType: db.Source(), BuildEpoch: db.BuildEpoch().Unix(), NodeCount: db.NodeCount(),
+ Published: p.now().UTC().Truncate(time.Second), Origin: o.Origin, Edition: o.Edition}
+ f, err := os.Open(path)
+ if err != nil {
+ return geoip.Pointer{}, err
+ }
+ defer f.Close()
+ if _, err := release.PutObjectFrom(ctx, p.store, next.Key, f, size, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
+ return geoip.Pointer{}, err
+ }
+ // A CAS conflict on the blob means a blob with this exact name (so
+ // these exact bytes) already exists: carry on (spec §3 step 2).
+ body, err := marshalPointer(next)
+ if err != nil {
+ return geoip.Pointer{}, err
+ }
+ if !present {
+ etag = ""
+ }
+ if _, err := p.store.Put(ctx, geoip.PointerKey, body, etag); err != nil {
+ return geoip.Pointer{}, fmt.Errorf("pointer write: %w", err)
+ }
+ if err := p.sweepLocked(ctx, next.Key); err != nil {
+ log.Printf("console: geoip sweep after publish: %v", err)
+ }
+ log.Printf("console: geoip published %s (%s, %s)", sha[:8], next.DatabaseType, next.Origin)
+ return next, nil
+}
+
+// copyCapped streams src into path, hashing, refusing more than max bytes
+// as it copies (codex R1-7).
+func copyCapped(ctx context.Context, path string, src io.Reader, max int64) (string, int64, error) {
+ f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+ if err != nil {
+ return "", 0, err
+ }
+ defer f.Close()
+ h := sha256.New()
+ n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(ctxReader{ctx, src}, max+1))
+ if err != nil {
+ return "", 0, err
+ }
+ if n > max {
+ return "", 0, refused(fmt.Sprintf("larger than %d bytes", max))
+ }
+ if n == 0 {
+ return "", 0, refused("empty upload")
+ }
+ if err := f.Sync(); err != nil {
+ return "", 0, err
+ }
+ return hex.EncodeToString(h.Sum(nil)), n, nil
+}
+
+type ctxReader struct {
+ ctx context.Context
+ r io.Reader
+}
+
+func (c ctxReader) Read(b []byte) (int, error) {
+ if err := c.ctx.Err(); err != nil {
+ return 0, err
+ }
+ return c.r.Read(b)
+}
+
+func hasCountryData(db *geo.MMDB) bool {
+ n := 0
+ for _, a := range geoipProbeAddrs {
+ if r, ok := db.Lookup(net.ParseIP(a)); ok && r.Country != "" {
+ n++
+ }
+ }
+ return n >= geoipProbeNeed
+}
+
+// Off deletes the pointer, then every blob (spec §3). Unconditional delete:
+// safe under the single-writer mutex (codex R2-6).
+func (p *geoipPublisher) Off(ctx context.Context) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if err := p.store.Delete(ctx, geoip.PointerKey); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ return p.sweepLocked(ctx, "")
+}
+
+// Sweep deletes every blob the pointer does not name (orphans from a crash
+// between blob upload and pointer write). Run at startup and daily.
+func (p *geoipPublisher) Sweep(ctx context.Context) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ cur, _, present, err := p.Current(ctx)
+ if err != nil {
+ return err
+ }
+ keep := ""
+ if present && !cur.Expired {
+ keep = cur.Key
+ }
+ return p.sweepLocked(ctx, keep)
+}
+
+func (p *geoipPublisher) sweepLocked(ctx context.Context, keep string) error {
+ keys, err := p.store.List(ctx, geoip.Prefix)
+ if err != nil {
+ return err
+ }
+ var firstErr error
+ for _, k := range keys {
+ if !strings.HasSuffix(k, ".mmdb") || k == keep {
+ continue
+ }
+ if err := p.store.Delete(ctx, k); err != nil && !errors.Is(err, os.ErrNotExist) && firstErr == nil {
+ firstErr = err
+ }
+ }
+ return firstErr
+}
+
+// ExpireIfDue applies D1 to the bucket: a GeoLite2 pointer past expiry is
+// rewritten with expired=true and its blob destroyed.
+func (p *geoipPublisher) ExpireIfDue(ctx context.Context, now time.Time) (bool, error) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ cur, etag, present, err := p.Current(ctx)
+ if err != nil || !present || cur.Expired || !geoip.Expired(cur.DatabaseType, cur.Build(), now) {
+ return false, err
+ }
+ cur.Expired = true
+ body, err := marshalPointer(cur)
+ if err != nil {
+ return false, err
+ }
+ if _, err := p.store.Put(ctx, geoip.PointerKey, body, etag); err != nil {
+ return false, err
+ }
+ log.Printf("console: geoip database %s expired (GeoLite2, 30 days after build) and was removed", cur.SHA256[:8])
+ return true, p.sweepLocked(ctx, "")
+}
+```
+
+`json.MarshalIndent` writes `"node_count": 1704` with a space, which is what the shrink test edits. If you change the marshaller, update that test.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/console -run 'Publish|OffAndSweep|ExpireIfDue' -count=1`
+Expected: PASS. If the console package's other tests need a socket TMPDIR, run them later in Task 12 outside the sandbox. These ones don't bind sockets.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/console/geoippublish.go internal/console/geoippublish_test.go
+git commit -m "console: geoip publish path (verify, country probe, shrink check, blob+pointer, sweep, off, D1 expiry) (geoip spec §3, §4.1)"
+```
+
+---
+
+### Task 7: The edge mirror
+
+**Files:**
+- Create: `internal/edge/geoipmirror.go`
+- Test: `internal/edge/geoipmirror_test.go`
+- Modify: `internal/edge/edge.go`:
+ - the `Edge` struct (new fields next to `analyticsMeter`, ~:158);
+ - `RunWithActivatorReady`: arm the mirror after the home map (~:3088), and start the loop next to `go mirror.run` (~:3171).
+
+**Interfaces:**
+- Consumes: `geo.OpenMMDBTrusted`, `geo.Swappable` (Tasks 2 and 3); `geoip.*` (Task 5); `release.OpenObject`.
+- Produces:
+ ```go
+ // in package edge
+ type GeoIPState struct{ Mode, SHA256, Type, Error string } // Mode: "mirror"|"tunnel"|"none"
+ func (e *Edge) GeoIPState() GeoIPState
+ func WithGeoTunnelClient() RunOption // agent passes it when CARLOS_TUNNEL_* are all set
+ // fields: e.geo *geo.Swappable (always non-nil after New), e.geoMirror *geoMirror (nil unless mode mirror), e.geoMode string
+ ```
+
+- [ ] **Step 1: Write the failing tests.** They drive a `geoMirror` directly over a `DirStore` and a temp data dir. Publish with the real Task 6 shape: write the blob and pointer by hand.
+
+```go
+package edge
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "net"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geo"
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func geoFixture(t *testing.T, name string) []byte {
+ b, err := os.ReadFile(filepath.Join("..", "geo", "testdata", name))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return b
+}
+
+// publishFixture writes blob + pointer the way the console does.
+func publishFixture(t *testing.T, s release.Store, body []byte, dbType string, build time.Time) geoip.Pointer {
+ t.Helper()
+ sum := sha256.Sum256(body)
+ sha := hex.EncodeToString(sum[:])
+ ctx := context.Background()
+ if _, err := s.Put(ctx, geoip.BlobKey(sha), body, ""); err != nil {
+ t.Fatal(err)
+ }
+ p := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: int64(len(body)),
+ DatabaseType: dbType, BuildEpoch: build.Unix(), Origin: "manual"}
+ raw, _ := json.Marshal(p)
+ _, etag, _ := s.Get(ctx, geoip.PointerKey)
+ if _, err := s.Put(ctx, geoip.PointerKey, raw, etag); err != nil {
+ t.Fatal(err)
+ }
+ return p
+}
+
+func newTestMirror(t *testing.T) (*geoMirror, release.Store, string) {
+ s := release.NewDirStore(t.TempDir())
+ dir := filepath.Join(t.TempDir(), "carlos", "geoip")
+ return newGeoMirror(s, dir, geo.NewSwappable()), s, dir
+}
+
+func lookupGB(m *geoMirror) bool {
+ snap := m.sw.Lookup(net.ParseIP("81.2.69.142"))
+ return snap.Found && snap.Result.Country == "GB"
+}
+
+func TestMirrorInstallsAPublishedDatabase(t *testing.T) {
+ m, s, dir := newTestMirror(t)
+ p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if !lookupGB(m) {
+ t.Fatal("not serving after poll")
+ }
+ if _, err := os.Stat(filepath.Join(dir, p.SHA256+".mmdb")); err != nil {
+ t.Fatal(err)
+ }
+ if st := m.State(); st.SHA256 != p.SHA256 || st.Type != "GeoIP2-City" {
+ t.Fatalf("state = %+v", st)
+ }
+ fi, _ := os.Stat(dir)
+ if fi.Mode().Perm() != 0o755 {
+ t.Fatalf("dir mode = %v, want 0755 (the console reads it)", fi.Mode().Perm())
+ }
+}
+
+func TestMirrorBootsFromTheLocalGenerationWithoutTheBucket(t *testing.T) {
+ m, s, dir := newTestMirror(t)
+ publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ // A fresh process, a dead bucket.
+ m2 := newGeoMirror(release.NewDirStore(filepath.Join(t.TempDir(), "empty")), dir, geo.NewSwappable())
+ m2.Boot()
+ if !lookupGB(m2) {
+ t.Fatal("boot did not serve the local generation")
+ }
+}
+
+func TestMirrorBootWithCurrentNamingAMissingFile(t *testing.T) {
+ m, _, dir := newTestMirror(t)
+ os.MkdirAll(dir, 0o755)
+ os.WriteFile(filepath.Join(dir, "current"), []byte(`{"sha256":"`+string(make([]byte, 0))+`deadbeef","database_type":"X","build_epoch":1}`), 0o644)
+ m.Boot()
+ if snap := m.sw.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded {
+ t.Fatal("loaded something from a dangling current")
+ }
+ if _, err := os.Stat(filepath.Join(dir, "current")); !os.IsNotExist(err) {
+ t.Fatal("dangling current was not removed")
+ }
+}
+
+func TestMirrorRejectsADigestMismatchOnce(t *testing.T) {
+ m, s, _ := newTestMirror(t)
+ p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
+ // Corrupt the blob after publishing.
+ os.WriteFile(filepath.Join(s.(*release.DirStore).Root(), p.Key), append(geoFixture(t, "GeoIP2-City-Test.mmdb")[:p.Size-1], 'x'), 0o644)
+ if err := m.Poll(context.Background()); err == nil {
+ t.Fatal("a corrupt blob installed")
+ }
+ if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
+ t.Fatal("serving a corrupt blob")
+ }
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatalf("second poll of the same rejected sha should be a quiet no-op, got %v", err)
+ }
+}
+
+func TestMirrorOffClearsEverything(t *testing.T) {
+ m, s, dir := newTestMirror(t)
+ publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
+ m.Poll(context.Background())
+ s.Delete(context.Background(), geoip.PointerKey)
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
+ t.Fatal("still serving after off")
+ }
+ if matches, _ := filepath.Glob(filepath.Join(dir, "*.mmdb")); len(matches) != 0 {
+ t.Fatalf("generations left: %v", matches)
+ }
+}
+
+func TestMirrorNeverInstallsAnExpiredGeoLite(t *testing.T) {
+ m, s, _ := newTestMirror(t)
+ now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
+ m.now = func() time.Time { return now }
+ // The fixture's own metadata says GeoIP2-City (no expiry), so make the
+ // POINTER claim GeoLite2 with an old build: expired before download.
+ publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoLite2-City", now.Add(-31*24*time.Hour))
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
+ t.Fatal("installed an expired pointer")
+ }
+ // Repeated polls of the same unmarked expired pointer stay quiet.
+ if err := m.Poll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestMirrorExpiryTickRemovesOnlyTheCurrentExpiredGeneration(t *testing.T) {
+ m, s, _ := newTestMirror(t)
+ now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
+ m.now = func() time.Time { return now }
+ publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", now)
+ m.Poll(context.Background())
+ // Pretend what is loaded is a GeoLite2 built 31 days ago.
+ m.genMu.Lock()
+ m.loadedType, m.loadedBuild = "GeoLite2-City", now.Add(-31*24*time.Hour)
+ m.genMu.Unlock()
+ m.ExpireTick()
+ if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
+ t.Fatal("expired generation still serving")
+ }
+ // A fresh, non-expiring install afterwards is untouched by later ticks.
+ publishFixture(t, s, geoFixture(t, "GeoIP2-Country-Test.mmdb"), "GeoIP2-Country", now)
+ m.Poll(context.Background())
+ m.ExpireTick()
+ if !lookupGB(m) {
+ t.Fatal("a later tick removed a fresh generation")
+ }
+}
+
+func TestMirrorRefusesAnOversizeBlob(t *testing.T) {
+ m, s, dir := newTestMirror(t)
+ p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
+ m.maxBytes = p.Size - 1
+ if err := m.Poll(context.Background()); err == nil {
+ t.Fatal("oversize installed")
+ }
+ if matches, _ := filepath.Glob(filepath.Join(dir, ".incoming-*")); len(matches) != 0 {
+ t.Fatalf("incoming left: %v", matches)
+ }
+}
+```
+
+The test calls `(*release.DirStore).Root()`. If DirStore has no such accessor, add one in this task (`func (d *DirStore) Root() string { return d.root }`, using whatever its field is called), or have the test keep the directory it passed to `NewDirStore` and corrupt `filepath.Join(storeDir, p.Key)` instead. Prefer the second: it doesn't touch `release`.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run (outside the sandbox, socket package): `mkdir -p /tmp/g && TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run Mirror -count=1`
+Expected: build failure, because `newGeoMirror` is undefined.
+
+- [ ] **Step 3: Implement `internal/edge/geoipmirror.go`**
+
+```go
+package edge
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "log"
+ "math/rand/v2"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geo"
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The GeoIP mirror (geoip spec §5): poll control/geoip/current.json, fetch
+// the content-addressed blob it names into an immutable local generation,
+// and hot-swap the edge's reader. genMu serialises everything that chooses,
+// expires or deletes a generation (codex R3-1); network I/O stays outside it.
+
+var (
+ geoMirrorInterval = 10 * time.Minute
+ geoMirrorJitter = time.Minute
+ geoMirrorDeadline = 10 * time.Minute
+ geoExpiryInterval = time.Minute
+)
+
+type geoCurrent struct {
+ SHA256 string `json:"sha256,omitempty"`
+ DatabaseType string `json:"database_type,omitempty"`
+ BuildEpoch int64 `json:"build_epoch,omitempty"`
+}
+
+type geoMirror struct {
+ store release.Store
+ dir string
+ sw *geo.Swappable
+ now func() time.Time
+ maxBytes int64
+
+ genMu sync.Mutex
+ loadedSHA string
+ loadedType string
+ loadedBuild time.Time
+ skipSHA map[string]string // sha -> why (rejected | expired); cleared when the pointer moves on
+ lastErr string // last poll's error category, "" when healthy
+ failingAt time.Time
+ lastFailLog time.Time
+}
+
+func newGeoMirror(store release.Store, dir string, sw *geo.Swappable) *geoMirror {
+ return &geoMirror{store: store, dir: dir, sw: sw, now: time.Now, maxBytes: geoip.MaxBlobBytes, skipSHA: map[string]string{}}
+}
+
+// GeoIPState is the heartbeat's view of this edge's GeoIP (spec §5.5).
+type GeoIPState struct {
+ Mode string `json:"mode"` // mirror | tunnel | none
+ SHA256 string `json:"sha256,omitempty"`
+ Type string `json:"type,omitempty"`
+ Error string `json:"error,omitempty"`
+}
+
+func (m *geoMirror) State() GeoIPState {
+ m.genMu.Lock()
+ defer m.genMu.Unlock()
+ return GeoIPState{Mode: "mirror", SHA256: m.loadedSHA, Type: m.loadedType, Error: m.lastErr}
+}
+
+func (m *geoMirror) path(sha string) string { return filepath.Join(m.dir, sha+".mmdb") }
+
+// Boot installs the local generation `current` names, if any, without
+// touching the bucket, and clears everything else (spec §5.3 "At boot").
+func (m *geoMirror) Boot() {
+ if err := os.MkdirAll(m.dir, 0o755); err != nil {
+ log.Printf("edge: geoip directory unusable; GeoIP stays off until it is: %v", err)
+ return
+ }
+ os.Chmod(m.dir, 0o755)
+ m.genMu.Lock()
+ defer m.genMu.Unlock()
+ cur, err := m.readCurrent()
+ keep := ""
+ switch {
+ case err != nil && !errors.Is(err, os.ErrNotExist):
+ log.Printf("edge: geoip local state unreadable; starting with no database")
+ os.Remove(filepath.Join(m.dir, "current"))
+ case cur.SHA256 == "":
+ case geoip.Expired(cur.DatabaseType, time.Unix(cur.BuildEpoch, 0), m.now()):
+ log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(cur.SHA256))
+ m.writeCurrentLocked(geoCurrent{})
+ default:
+ db, err := geo.OpenMMDBTrusted(m.path(cur.SHA256))
+ if err != nil {
+ log.Printf("edge: geoip local generation %s unopenable; starting with no database", short(cur.SHA256))
+ os.Remove(filepath.Join(m.dir, "current"))
+ break
+ }
+ m.sw.Swap(db)
+ m.loadedSHA, m.loadedType, m.loadedBuild = cur.SHA256, db.Source(), db.BuildEpoch()
+ keep = cur.SHA256
+ log.Printf("edge: geoip database %s (%s, built %s) loaded", short(cur.SHA256), db.Source(), db.BuildEpoch().Format("2006-01-02"))
+ }
+ m.cleanLocked(keep)
+}
+
+func short(sha string) string {
+ if len(sha) > 8 {
+ return sha[:8]
+ }
+ return sha
+}
+
+func (m *geoMirror) readCurrent() (geoCurrent, error) {
+ var c geoCurrent
+ b, err := os.ReadFile(filepath.Join(m.dir, "current"))
+ if err != nil {
+ return c, err
+ }
+ if err := json.Unmarshal(b, &c); err != nil {
+ return c, err
+ }
+ if c.SHA256 != "" && !geoip.IsSHA256Hex(c.SHA256) {
+ return geoCurrent{}, errors.New("bad sha")
+ }
+ return c, nil
+}
+
+// writeCurrentLocked replaces `current` atomically: temp, fsync, rename,
+// fsync the directory (spec §5.2).
+func (m *geoMirror) writeCurrentLocked(c geoCurrent) error {
+ b, _ := json.Marshal(c)
+ tmp, err := os.CreateTemp(m.dir, ".current-*")
+ if err != nil {
+ return err
+ }
+ defer os.Remove(tmp.Name())
+ if _, err := tmp.Write(b); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Sync(); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Close(); err != nil {
+ return err
+ }
+ if err := os.Chmod(tmp.Name(), 0o644); err != nil {
+ return err
+ }
+ if err := os.Rename(tmp.Name(), filepath.Join(m.dir, "current")); err != nil {
+ return err
+ }
+ return syncDir(m.dir)
+}
+
+func syncDir(dir string) error {
+ d, err := os.Open(dir)
+ if err != nil {
+ return err
+ }
+ defer d.Close()
+ return d.Sync()
+}
+
+// cleanLocked deletes every generation but keep, every .incoming-* and
+// every stray .current-*.
+func (m *geoMirror) cleanLocked(keep string) {
+ entries, _ := os.ReadDir(m.dir)
+ for _, e := range entries {
+ n := e.Name()
+ if n == "current" || (keep != "" && n == keep+".mmdb") {
+ continue
+ }
+ if strings.HasSuffix(n, ".mmdb") || strings.HasPrefix(n, ".incoming-") || strings.HasPrefix(n, ".current-") {
+ os.Remove(filepath.Join(m.dir, n))
+ }
+ }
+}
+
+// clearLocked turns GeoIP off on this box: empty current, nil reader, no
+// generations.
+func (m *geoMirror) clearLocked() {
+ if err := m.writeCurrentLocked(geoCurrent{}); err != nil {
+ os.Remove(filepath.Join(m.dir, "current"))
+ }
+ m.sw.Swap(nil)
+ m.loadedSHA, m.loadedType, m.loadedBuild = "", "", time.Time{}
+ m.cleanLocked("")
+}
+
+// Poll is one pass (spec §5.3). The caller bounds ctx by geoMirrorDeadline.
+func (m *geoMirror) Poll(ctx context.Context) error {
+ if err := os.MkdirAll(m.dir, 0o755); err != nil {
+ return m.fail("local directory unusable", err)
+ }
+ raw, err := readCapped(ctx, m.store, geoip.PointerKey, geoip.MaxPointerBytes)
+ if errors.Is(err, os.ErrNotExist) {
+ m.genMu.Lock()
+ if m.loadedSHA != "" {
+ log.Printf("edge: geoip turned off; no database published")
+ }
+ m.clearLocked()
+ m.lastErr = ""
+ m.genMu.Unlock()
+ return nil
+ }
+ if err != nil {
+ return m.fail("pointer unreadable", err)
+ }
+ p, err := geoip.ParsePointer(raw)
+ if err != nil {
+ return m.fail("pointer invalid", err)
+ }
+ m.genMu.Lock()
+ if p.Expired || geoip.Expired(p.DatabaseType, p.Build(), m.now()) {
+ if m.skipSHA[p.SHA256] != "expired" {
+ log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(p.SHA256))
+ }
+ m.skipSHA = map[string]string{p.SHA256: "expired"}
+ m.clearLocked()
+ m.lastErr = ""
+ m.genMu.Unlock()
+ return nil
+ }
+ if p.SHA256 == m.loadedSHA {
+ m.lastErr = ""
+ m.genMu.Unlock()
+ return nil
+ }
+ if why, ok := m.skipSHA[p.SHA256]; ok {
+ m.genMu.Unlock()
+ _ = why
+ return nil
+ }
+ m.skipSHA = map[string]string{} // the pointer moved on
+ m.genMu.Unlock()
+
+ tmp, err := m.download(ctx, p)
+ if err != nil {
+ if errors.Is(err, errGeoDigest) || errors.Is(err, errGeoSize) {
+ m.genMu.Lock()
+ m.skipSHA[p.SHA256] = "rejected"
+ m.genMu.Unlock()
+ log.Printf("edge: geoip blob %s rejected: %v", short(p.SHA256), err)
+ }
+ return m.fail("blob unavailable", err)
+ }
+ return m.install(tmp, p)
+}
+
+var (
+ errGeoDigest = errors.New("digest mismatch")
+ errGeoSize = errors.New("size mismatch")
+)
+
+// download streams the blob to .incoming-<sha>, checks size and digest,
+// fsyncs, and returns its path. Outside genMu.
+func (m *geoMirror) download(ctx context.Context, p geoip.Pointer) (string, error) {
+ rc, _, err := release.OpenObject(ctx, m.store, p.Key)
+ if err != nil {
+ return "", err
+ }
+ defer rc.Close()
+ tmp := filepath.Join(m.dir, ".incoming-"+p.SHA256)
+ f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
+ if err != nil {
+ return "", err
+ }
+ limit := p.Size
+ if m.maxBytes < limit {
+ limit = m.maxBytes
+ }
+ h := sha256.New()
+ n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, limit+1))
+ if err == nil {
+ err = f.Sync()
+ }
+ if cerr := f.Close(); err == nil {
+ err = cerr
+ }
+ switch {
+ case err != nil:
+ os.Remove(tmp)
+ return "", err
+ case n != p.Size:
+ os.Remove(tmp)
+ return "", errGeoSize
+ case hex.EncodeToString(h.Sum(nil)) != p.SHA256:
+ os.Remove(tmp)
+ return "", errGeoDigest
+ }
+ return tmp, nil
+}
+
+// install commits a verified download under genMu (spec §5.3 steps 3-5),
+// re-checking expiry against the file's own metadata first.
+func (m *geoMirror) install(tmp string, p geoip.Pointer) error {
+ m.genMu.Lock()
+ defer m.genMu.Unlock()
+ final := m.path(p.SHA256)
+ if err := os.Rename(tmp, final); err != nil {
+ os.Remove(tmp)
+ return m.failLocked("install failed", err)
+ }
+ syncDir(m.dir)
+ db, err := geo.OpenMMDBTrusted(final)
+ if err != nil {
+ os.Remove(final)
+ m.skipSHA[p.SHA256] = "rejected"
+ log.Printf("edge: geoip blob %s rejected: unopenable", short(p.SHA256))
+ return m.failLocked("blob unopenable", err)
+ }
+ if geoip.Expired(db.Source(), db.BuildEpoch(), m.now()) {
+ db.Close()
+ os.Remove(final)
+ m.skipSHA[p.SHA256] = "expired"
+ log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(p.SHA256))
+ return nil
+ }
+ if err := m.writeCurrentLocked(geoCurrent{SHA256: p.SHA256, DatabaseType: db.Source(), BuildEpoch: db.BuildEpoch().Unix()}); err != nil {
+ db.Close()
+ os.Remove(final)
+ return m.failLocked("install failed", err)
+ }
+ m.sw.Swap(db)
+ m.loadedSHA, m.loadedType, m.loadedBuild = p.SHA256, db.Source(), db.BuildEpoch()
+ m.cleanLocked(p.SHA256)
+ m.lastErr, m.failingAt = "", time.Time{}
+ log.Printf("edge: geoip database %s (%s, built %s) loaded", short(p.SHA256), db.Source(), db.BuildEpoch().Format("2006-01-02"))
+ return nil
+}
+
+// ExpireTick is D1 on the loaded generation, under genMu, so it can only
+// ever remove the generation that is current at that moment (codex R3-1).
+func (m *geoMirror) ExpireTick() {
+ m.genMu.Lock()
+ defer m.genMu.Unlock()
+ if m.loadedSHA == "" || !geoip.Expired(m.loadedType, m.loadedBuild, m.now()) {
+ return
+ }
+ log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(m.loadedSHA))
+ m.skipSHA[m.loadedSHA] = "expired"
+ m.clearLocked()
+}
+
+func (m *geoMirror) fail(category string, err error) error {
+ m.genMu.Lock()
+ defer m.genMu.Unlock()
+ return m.failLocked(category, err)
+}
+
+// failLocked records the category for the heartbeat and logs at most hourly
+// while failing. The error text is never logged: it may carry a key or
+// bytes; the category is the message (spec §12).
+func (m *geoMirror) failLocked(category string, err error) error {
+ now := m.now()
+ if m.lastErr == "" {
+ m.failingAt = now
+ }
+ m.lastErr = category
+ if now.Sub(m.lastFailLog) >= time.Hour {
+ m.lastFailLog = now
+ log.Printf("edge: geoip %s; keeping the loaded database", category)
+ }
+ return fmt.Errorf("%s: %w", category, err)
+}
+
+// readCapped reads key through OpenObject, refusing more than max bytes.
+func readCapped(ctx context.Context, s release.Store, key string, max int) ([]byte, error) {
+ rc, _, err := release.OpenObject(ctx, s, key)
+ if err != nil {
+ return nil, err
+ }
+ defer rc.Close()
+ b, err := io.ReadAll(io.LimitReader(rc, int64(max)+1))
+ if err != nil {
+ return nil, err
+ }
+ if len(b) > max {
+ return nil, fmt.Errorf("%s larger than %d bytes", key, max)
+ }
+ return b, nil
+}
+
+// run is the process-lifetime loop: a poll every interval+jitter, each
+// bounded by geoMirrorDeadline, and an independent expiry ticker that never
+// touches the network.
+func (m *geoMirror) run(stop <-chan struct{}) {
+ go func() {
+ t := time.NewTicker(geoExpiryInterval)
+ defer t.Stop()
+ for {
+ select {
+ case <-stop:
+ return
+ case <-t.C:
+ m.ExpireTick()
+ }
+ }
+ }()
+ for {
+ ctx, cancel := context.WithTimeout(context.Background(), geoMirrorDeadline)
+ m.Poll(ctx)
+ cancel()
+ select {
+ case <-stop:
+ return
+ case <-time.After(geoMirrorInterval + rand.N(geoMirrorJitter)):
+ }
+ }
+}
+```
+
+Then wire it into `Edge`:
+
+```go
+// Edge struct, beside analyticsMeter:
+ // geo is the GeoIP reader every request's lookup goes through (geoip
+ // spec §5.4); never nil after New, empty until a database is installed.
+ // geoMirror fills it when this edge has a deployment store and is not a
+ // tunnel client; geoMode is what the heartbeat reports.
+ geo *geo.Swappable
+ geoMirror *geoMirror
+ geoMode string
+ geoTunnelClient bool
+```
+
+In `New(...)`, initialise `e.geo = geo.NewSwappable()`. Add the option and the accessor:
+
+```go
+// WithGeoTunnelClient marks this edge as a tunnel client's: it never mirrors
+// the GeoIP database and takes the hub's headers instead (spec §5.1).
+func WithGeoTunnelClient() RunOption { return func(e *Edge) { e.geoTunnelClient = true } }
+
+// GeoIPState is the heartbeat block (spec §5.5).
+func (e *Edge) GeoIPState() GeoIPState {
+ if e.geoMirror != nil {
+ return e.geoMirror.State()
+ }
+ if e.geoMode == "" {
+ return GeoIPState{Mode: "none"}
+ }
+ return GeoIPState{Mode: e.geoMode}
+}
+```
+
+In `RunWithActivatorReady`, right after the home-map block (~:3088):
+
+```go
+ // GeoIP (geoip spec §5): a tunnel client takes the hub's headers and
+ // mirrors nothing; an edge with the deployment store mirrors; anything
+ // else has no database and only strips.
+ switch {
+ case e.geoTunnelClient:
+ e.geoMode = "tunnel"
+ case e.staticStore != nil:
+ e.geoMode = "mirror"
+ e.geoMirror = newGeoMirror(e.staticStore, filepath.Join(dataDir, "carlos", "geoip"), e.geo)
+ e.geoMirror.Boot()
+ default:
+ e.geoMode = "none"
+ }
+```
+
+And next to `go mirror.run(...)` (~:3171):
+
+```go
+ if e.geoMirror != nil {
+ go e.geoMirror.run(make(chan struct{})) // process-lifetime, like the cert mirror
+ }
+```
+
+`dataDir` is the existing `envOr("CARLOS_DATA_DIR", "/data")` at ~:2948. Use the existing variable; don't re-read the env.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Mirror' -race -count=1`
+Expected: PASS. Then run the whole edge package once: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -count=1`. Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/edge/geoipmirror.go internal/edge/geoipmirror_test.go internal/edge/edge.go
+git commit -m "edge: GeoIP mirror with immutable generations, genMu, D1 expiry ticker (geoip spec §5)"
+```
+
+---
+
+### Task 8: The heartbeat block
+
+**Files:**
+- Modify:
+ - `internal/status/snapshot.go` (`Probes` ~:101; `BoxMetrics` ~:282; `CollectBox` ~:739-770);
+ - `internal/host/agent.go`:
+ - a source var beside `upstreamSource` (~:2781);
+ - the ready hook (~:576-585): publish it;
+ - `edgeOpts` (~:550-575): `WithGeoTunnelClient` when the tunnel env is complete;
+ - `statusLoop`'s `Probes` literal (~:2938-3003).
+- Test: `internal/status/snapshot_test.go` (add one), `internal/host/agent_geoip_test.go` (new)
+
+**Interfaces:**
+- Consumes: `edge.GeoIPState`, `(*edge.Edge).GeoIPState()`, `edge.WithGeoTunnelClient()` (Task 7).
+- Produces:
+ ```go
+ // package status
+ type GeoIPState struct { Mode string `json:"mode"`; SHA256 string `json:"sha256,omitempty"`; Type string `json:"type,omitempty"`; Error string `json:"error,omitempty"` }
+ // Probes.GeoIP *GeoIPState ; BoxMetrics.GeoIP *GeoIPState `json:"geoip,omitempty"`
+ // package host
+ func tunnelEnvComplete(getenv func(string) string) bool
+ ```
+
+- [ ] **Step 1: Write the failing tests.** In `internal/status/snapshot_test.go`:
+
+```go
+func TestCollectBoxCarriesGeoIP(t *testing.T) {
+ box := CollectBox(Probes{Label: "b", GeoIP: &GeoIPState{Mode: "mirror", SHA256: "abc", Type: "DBIP-City-Lite"}} /* fill whatever other required fields the other CollectBox tests pass */)
+ if box.Metrics == nil || box.Metrics.GeoIP == nil || box.Metrics.GeoIP.SHA256 != "abc" {
+ t.Fatalf("metrics = %+v", box.Metrics)
+ }
+ raw, _ := json.Marshal(box)
+ if !bytes.Contains(raw, []byte(`"geoip":{"mode":"mirror"`)) {
+ t.Fatalf("json = %s", raw)
+ }
+}
+```
+
+Copy the `CollectBox` call shape from the nearest existing test in that file. `CollectBox`'s signature may take more than `Probes`.
+
+In `internal/host/agent_geoip_test.go`:
+
+```go
+package host
+
+import "testing"
+
+func TestTunnelEnvComplete(t *testing.T) {
+ env := map[string]string{"CARLOS_TUNNEL_URL": "https://t", "CARLOS_TUNNEL_TOKEN": "x", "CARLOS_TUNNEL_NAME": "n"}
+ get := func(k string) string { return env[k] }
+ if !tunnelEnvComplete(get) {
+ t.Fatal("complete env")
+ }
+ delete(env, "CARLOS_TUNNEL_NAME")
+ if tunnelEnvComplete(get) {
+ t.Fatal("incomplete env must not make a tunnel client (tunnelLoop would not run)")
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/status -run GeoIP -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/host -run TunnelEnvComplete -count=1`
+Expected: build failures.
+
+- [ ] **Step 3: Implement it**
+
+`internal/status/snapshot.go`:
+
+```go
+// GeoIPState is one box's GeoIP mirror (geoip spec §5.5). Present on every
+// box running a GeoIP-aware binary; absent means an older binary, which the
+// rollout verdict treats as "cannot be trusted to strip forged headers".
+type GeoIPState struct {
+ Mode string `json:"mode"` // mirror | tunnel | none
+ SHA256 string `json:"sha256,omitempty"`
+ Type string `json:"type,omitempty"`
+ Error string `json:"error,omitempty"`
+}
+```
+
+- Add `GeoIP *GeoIPState` to `Probes`, beside `Tunnel`.
+- Add ``GeoIP *GeoIPState `json:"geoip,omitempty"` `` to `BoxMetrics`, beside `Tunnel`.
+- In `CollectBox`, add `|| p.GeoIP != nil` to the "anything known" condition, and `GeoIP: p.GeoIP,` to the `BoxMetrics` literal.
+
+`internal/host/agent.go`:
+
+```go
+// geoIPSource is the edge's GeoIP state for the heartbeat, published from
+// the ready hook (same pattern as upstreamSource).
+var geoIPSource atomic.Value // func() edge.GeoIPState
+
+func geoIPState() *status.GeoIPState {
+ fn, _ := geoIPSource.Load().(func() edge.GeoIPState)
+ if fn == nil {
+ return nil
+ }
+ s := fn()
+ return &status.GeoIPState{Mode: s.Mode, SHA256: s.SHA256, Type: s.Type, Error: s.Error}
+}
+
+// tunnelEnvComplete is tunnelLoop's own arming rule: all three set.
+func tunnelEnvComplete(getenv func(string) string) bool {
+ return getenv("CARLOS_TUNNEL_URL") != "" && getenv("CARLOS_TUNNEL_TOKEN") != "" && getenv("CARLOS_TUNNEL_NAME") != ""
+}
+```
+
+- In `edgeOpts`, append `edge.WithGeoTunnelClient()` when `tunnelEnvComplete(os.Getenv)`.
+- In the ready hook, after `setUpstreamSource(e.UpstreamStates)`, add `geoIPSource.Store(e.GeoIPState)`.
+- In `statusLoop`'s `Probes` literal, beside `Tunnel: tunnelState()`, add `GeoIP: geoIPState()`.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/status -count=1`, then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/host -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/status/snapshot.go internal/status/snapshot_test.go internal/host/agent.go internal/host/agent_geoip_test.go
+git commit -m "status: GeoIP block on every box heartbeat; agent marks tunnel clients (geoip spec §5.5)"
+```
+
+---
+
+### Task 9: Capture, decide, expose; `setGeo` on every proxy
+
+**Files:**
+- Create: `internal/edge/geoipheaders.go`
+- Modify: `internal/edge/edge.go`:
+ - `ServeHTTP` (~:1343-1404);
+ - the six `Rewrite` funcs: `unixProxy` :2233, `tcpProxy` :2259, `multiProxy` :2347, `tunnelProxy` :2434, `fleetProxy` :2489, `backhaulProxy` :2863.
+- Test: `internal/edge/geoipheaders_test.go`
+
+**Interfaces:**
+- Consumes: `geoip.Capture/Adopt/FromSnapshot/Off/WriteApp/HopValue/StripAll/HeaderHop` (Task 5); `e.geo` (Task 7); `hopVerified`, `doorCallerFrom`, `clientIP`; `tunnelclient.FromTunnel(ctx) bool` (Task 10; until then the stub `fromTunnel` below returns false).
+- Produces:
+ ```go
+ type geoResultKey struct{}
+ func geoAnswerFrom(ctx context.Context) (geoip.Answer, bool)
+ func (e *Edge) decideGeo(r *http.Request, c geoip.Captured, hopPresent, hopped, isDoor bool) geoip.Answer
+ func setGeo(pr *httputil.ProxyRequest, forwardsToEdge bool)
+ var fromTunnel = func(ctx context.Context) bool { return false } // Task 10 points it at tunnelclient.FromTunnel
+ ```
+
+- [ ] **Step 1: Write the failing tests.** Use `testEdgeWithStore`-style construction with a header-echo backend, and drive it with `httptest.NewRecorder` so `RemoteAddr` can be a public IP. Read `edge_test.go:274` (`testEdgeWithStore`) and `homemap_test.go:322-387` (`hopEdge`, `hopRequest`) first, and reuse them.
+
+```go
+package edge
+
+import (
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/geo"
+ "github.com/carlosframework/platform/internal/geoip"
+)
+
+// geoEchoEdge returns an edge serving app.test from a backend that echoes
+// the geo headers it received, with the City fixture loaded.
+func geoEchoEdge(t *testing.T) (*Edge, func(*http.Request) http.Header) {
+ t.Helper()
+ got := make(chan http.Header, 1)
+ backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ h := http.Header{}
+ for _, k := range []string{geoip.HeaderCountry, geoip.HeaderSubdivision, geoip.HeaderLat, geoip.HeaderLon, geoip.HeaderSource, geoip.HeaderHop} {
+ if v, ok := r.Header[http.CanonicalHeaderKey(k)]; ok {
+ h[k] = v
+ }
+ }
+ select {
+ case got <- h:
+ default:
+ }
+ w.Header().Set("Content-Type", "text/plain")
+ w.Write([]byte("ok"))
+ }))
+ t.Cleanup(backend.Close)
+ e := newTCPRouteEdge(t, "app.test", backend.Listener.Addr().String()) // build from the helpers edge_test.go uses for a tcp upstream
+ db, err := geo.OpenMMDB("../geo/testdata/GeoIP2-City-Test.mmdb")
+ if err != nil {
+ t.Fatal(err)
+ }
+ e.geo.Swap(db)
+ return e, func(r *http.Request) http.Header {
+ w := httptest.NewRecorder()
+ e.ServeHTTP(w, r)
+ select {
+ case h := <-got:
+ return h
+ default:
+ t.Fatalf("backend not reached: %d %s", w.Code, w.Body.String())
+ return nil
+ }
+ }
+}
+
+func publicReq(host, from string) *http.Request {
+ r := httptest.NewRequest("GET", "http://"+host+"/", nil)
+ r.RemoteAddr = net.JoinHostPort(from, "4444")
+ return r
+}
+
+func TestDirectRequestGetsGeoHeaders(t *testing.T) {
+ _, serve := geoEchoEdge(t)
+ h := serve(publicReq("app.test", "81.2.69.142"))
+ if h.Get(geoip.HeaderCountry) != "GB" || h.Get(geoip.HeaderSubdivision) != "GB-ENG" ||
+ h.Get(geoip.HeaderSource) != "GeoIP2-City" || h.Get(geoip.HeaderLat) == "" {
+ t.Fatalf("headers = %v", h)
+ }
+ if h.Get(geoip.HeaderHop) != "" {
+ t.Fatal("the internal hop header reached the app")
+ }
+}
+
+func TestForgedGeoHeadersNeverReachTheApp(t *testing.T) {
+ _, serve := geoEchoEdge(t)
+ for name, mutate := range map[string]func(*http.Request){
+ "plain forgery": func(r *http.Request) {
+ r.Header.Set(geoip.HeaderCountry, "KP")
+ r.Header.Set(geoip.HeaderHop, "hit;src=x")
+ },
+ "connection nomination": func(r *http.Request) {
+ r.Header.Set("Connection", "X-Carlos-Geo-Country, X-Carlos-Geo-Source")
+ },
+ "forgery from a private address": func(r *http.Request) {
+ r.RemoteAddr = "10.0.0.1:1"
+ r.Header.Set(geoip.HeaderCountry, "KP")
+ },
+ } {
+ r := publicReq("app.test", "81.2.69.142")
+ mutate(r)
+ h := serve(r)
+ if c := h.Get(geoip.HeaderCountry); c == "KP" {
+ t.Errorf("%s: forged country reached the app", name)
+ }
+ if name == "connection nomination" && h.Get(geoip.HeaderCountry) != "GB" {
+ t.Errorf("%s: the edge's own country was removed: %v", name, h)
+ }
+ if h.Get(geoip.HeaderHop) != "" {
+ t.Errorf("%s: hop header reached the app", name)
+ }
+ }
+}
+
+func TestVerifiedHopAdoptsWithoutLookingUp(t *testing.T) {
+ // Use hopEdge/hopRequest (homemap_test.go). Send hop-signed headers
+ // carrying X-Carlos-Hop-Geo "hit;src=GeoLite2-City" and a country "FR";
+ // with a counting lookup installed in e.geo, assert the app sees FR and
+ // the lookup count is 0.
+}
+
+func TestRejectedHopGetsNoGeo(t *testing.T) {
+ // hopRequest with a WRONG signature (signHop under a different key) from
+ // RemoteAddr 81.2.69.142: served (status 200, as today), and the app
+ // sees no X-Carlos-Geo-* at all (spec §8.2 rejected hop, codex R2-1).
+}
+
+func TestDoorRequestGetsNoGeo(t *testing.T) {
+ // Build a door request the way door_test.go does (doorRequest / the
+ // door context mark) with the box's public address 81.2.69.142: the app
+ // sees no geo headers.
+}
+
+func TestBackhaulCarriesHopGeoAndNeverAppHeadersWithoutIt(t *testing.T) {
+ // Using newBackhaulScene (backhaul_e2e_test.go:373): a request from a
+ // public client address to the sending edge reaches the home edge's app
+ // with the sender's country; a door call backhauled reaches it with
+ // none. Use e2eClientAddr to set the client address.
+}
+
+func TestGeoSetBeforeCacheVaryMatch(t *testing.T) {
+ // Using varyBackendEdge (edge_test.go:590) with Vary: X-Carlos-Geo-Country:
+ // two requests from GB and US addresses produce two variants; a third
+ // from GB is a hit on the GB variant (cachedVariants helper).
+}
+
+func TestUpgradeRequestCarriesGeo(t *testing.T) {
+ // Using upgradeEchoBackend (edge_test.go:1991): the upgrade request the
+ // backend sees carries X-Carlos-Geo-Country.
+}
+```
+
+**The executor writes the bodies of the six stub tests,** using the named helpers. The plan names each helper's location rather than guessing its signature. Every stub must assert exactly what its comment says, and each must fail before Step 3. If a helper turns out not to exist under that name, find its replacement with `grep -n "func newBackhaulScene\|func hopEdge\|func varyBackendEdge\|func upgradeEchoBackend" internal/edge/*_test.go`. Do not invent a new harness when one exists. `newTCPRouteEdge` is likewise a stand-in for whichever helper `edge_test.go` uses to route a host to a TCP backend.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Geo|Forged|Hop.*Geo|DoorRequestGetsNoGeo' -count=1`
+Expected: FAIL, because no geo headers are set.
+
+- [ ] **Step 3: Implement it.** `internal/edge/geoipheaders.go`:
+
+```go
+package edge
+
+import (
+ "context"
+ "net"
+ "net/http"
+ "net/http/httputil"
+
+ "github.com/carlosframework/platform/internal/geoip"
+)
+
+// geoResultKey carries the request's one geo answer (spec §8.2): computed
+// at ingress or adopted from a trusted transport, read by setGeo, the cache
+// path (through r.Header) and analytics.
+type geoResultKey struct{}
+
+func geoAnswerFrom(ctx context.Context) (geoip.Answer, bool) {
+ a, ok := ctx.Value(geoResultKey{}).(geoip.Answer)
+ return a, ok
+}
+
+// fromTunnel reports the tunnel client's mark (Task 10 sets it to
+// tunnelclient.FromTunnel). A var so this file builds before Task 10.
+var fromTunnel = func(ctx context.Context) bool { return false }
+
+// decideGeo is step 3 of spec §8.1. hopPresent says the request ARRIVED
+// carrying X-Carlos-Hop (captured before acceptHop strips it); hopped says
+// acceptHop verified it.
+func (e *Edge) decideGeo(r *http.Request, c geoip.Captured, hopPresent, hopped, isDoor bool) geoip.Answer {
+ switch {
+ case isDoor:
+ return geoip.Off() // RemoteAddr is this box (door.go)
+ case hopped:
+ return c.Adopt() // the sending edge looked up; never trust our own peer
+ case hopPresent:
+ return geoip.Off() // rejected hop: RemoteAddr is the sending edge (codex R2-1)
+ case fromTunnel(r.Context()):
+ return c.Adopt() // the hub looked up (spec §8.4)
+ }
+ ip := net.ParseIP(clientIP(r))
+ return geoip.FromSnapshot(e.geo.Lookup(ip))
+}
+
+// setGeo is every Rewrite's geo step (spec §8.3): after hop-by-hop removal,
+// delete every geo header from the outbound request and write the answer
+// back from the context. forwardsToEdge adds X-Carlos-Hop-Geo for the next
+// edge; an app never receives it.
+func setGeo(pr *httputil.ProxyRequest, forwardsToEdge bool) {
+ geoip.StripAll(pr.Out.Header)
+ a, ok := geoAnswerFrom(pr.In.Context())
+ if !ok {
+ a = geoip.Off() // scheduler calls through UnixProxy and anything else that never passed ServeHTTP
+ }
+ a.WriteApp(pr.Out.Header)
+ if forwardsToEdge {
+ pr.Out.Header.Set(geoip.HeaderHop, a.HopValue())
+ }
+}
+```
+
+Edit `ServeHTTP`:
+
+```go
+ // (immediately after the host split, before doorCallerFrom)
+ // GeoIP step 1 (spec §8.1): capture and strip before ANY branch. Nothing
+ // trusts these values until step 3.
+ capturedGeo := geoip.Capture(r.Header)
+ hopPresent := r.Header.Get(hopHeader) != ""
+```
+
+```go
+ // (immediately after `r, hopped := e.acceptHop(r, host)`)
+ // GeoIP steps 3-4: decide once, put it on the context, and expose the
+ // app-visible headers on r.Header so Vary matching sees the values the
+ // app will get.
+ geoAns := e.decideGeo(r, capturedGeo, hopPresent, hopped, isDoor)
+ r = r.WithContext(context.WithValue(r.Context(), geoResultKey{}, geoAns))
+ geoAns.WriteApp(r.Header)
+```
+
+In each of the six `Rewrite` funcs, add the line right after `setForwarded(pr)`:
+
+| Proxy | Line to add |
+|---|---|
+| `unixProxy`, `tcpProxy`, `multiProxy` | `setGeo(pr, false)` |
+| `tunnelProxy`, `fleetProxy`, `backhaulProxy` | `setGeo(pr, true)` |
+
+- [ ] **Step 4: Run the tests**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -count=1`
+Expected: PASS, the whole package. In particular the existing hop, door, cache and purge tests still pass.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/edge/geoipheaders.go internal/edge/geoipheaders_test.go internal/edge/edge.go
+git commit -m "edge: capture/decide/expose geo headers; setGeo on every proxy; door and rejected hop get none (geoip spec §8)"
+```
+
+---
+
+### Task 10: The TLS-only tunnel mark
+
+**Files:**
+- Modify: `internal/tunnelclient/tunnelclient.go` (`Config` :66; `Loop` :87; `serveUntilDead` :158; `Dial` :191; `Serve` :265)
+- Modify: `internal/edge/geoipheaders.go` (point `fromTunnel` at `tunnelclient.FromTunnel`)
+- Test: `internal/tunnelclient/mark_test.go`, `internal/edge/tunnel_geo_e2e_test.go`
+
+**Interfaces:**
+- Produces:
+ ```go
+ // package tunnelclient
+ func FromTunnel(ctx context.Context) bool
+ // Config gains: TLSConfig *tls.Config // nil = system roots (production); tests pass the test server's pool
+ func Mark(h http.Handler) http.Handler // wraps h so every request carries the TLS trust mark
+ func ViaTunnel(ctx context.Context) bool // true for EVERY request served over a tunnel, TLS or not (spec §9, codex R4-1)
+ ```
+ `Loop` always puts the via-tunnel mark on requests. It adds the trust mark (`Mark`) only when `cfg.URL`'s scheme is `https`.
+
+- [ ] **Step 1: Write the failing tests.** `internal/tunnelclient/mark_test.go`:
+
+```go
+package tunnelclient
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "testing"
+)
+
+func TestMarkIsOnlyOnMarkedHandlers(t *testing.T) {
+ var saw bool
+ h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { saw = FromTunnel(r.Context()) })
+ h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
+ if saw {
+ t.Fatal("an unmarked handler saw the mark")
+ }
+ Mark(h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
+ if !saw {
+ t.Fatal("Mark did not mark")
+ }
+}
+
+func TestServeHandlerMarksTrustOnlyOverTLSButViaAlways(t *testing.T) {
+ var trusted, via bool
+ h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ trusted, via = FromTunnel(r.Context()), ViaTunnel(r.Context())
+ })
+ servedHandler(Config{URL: "http://edge.test/register"}, h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
+ if trusted || !via {
+ t.Fatalf("plaintext: trusted=%v via=%v, want false/true (codex R2-2, R4-1)", trusted, via)
+ }
+ servedHandler(Config{URL: "https://edge.test/register"}, h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
+ if !trusted || !via {
+ t.Fatalf("tls: trusted=%v via=%v, want true/true", trusted, via)
+ }
+}
+```
+
+`internal/edge/tunnel_geo_e2e_test.go` follows `TestTunnelEndToEnd` (`tunnel_e2e_test.go:37`), but:
+
+- the hub is `httptest.NewTLSServer(hubEdge)`;
+- the remote host's `tunnelclient.Loop` gets `Config{URL: "https://…", TLSConfig: &tls.Config{RootCAs: pool, ServerName: …}}`, where `pool` holds `srv.Certificate()`;
+- the remote handler is a second `Edge` with an echo backend route.
+
+Assert:
+
+- a request to the hub from `RemoteAddr` 81.2.69.142 (City fixture loaded in the hub's `e.geo`) reaches the remote app with `X-Carlos-Geo-Country: GB` and no `X-Carlos-Hop-Geo`;
+- the remote edge's `e.geo` is empty and was never consulted;
+- the remote edge's analytics meter records `Country: "GB"` (this assertion lands in Task 11; leave a `// Task 11` marker here);
+- the same scene over plain `http://` delivers **no** geo headers.
+
+Factor `startFakeRemoteHost` (`tunnel_e2e_test.go:190`) so it takes a `tunnelclient.Config` mutator, and don't copy it.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/tunnelclient -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run TunnelGeo -count=1`
+Expected: build failures.
+
+- [ ] **Step 3: Implement it**
+
+```go
+// internal/tunnelclient/tunnelclient.go
+
+type tunnelKey struct{}
+
+// FromTunnel reports whether a request arrived over a TLS tunnel this
+// client dialled: the edge's grant to adopt the hub's geo headers (geoip
+// spec §8.4). Nothing outside this package can set it.
+func FromTunnel(ctx context.Context) bool {
+ v, _ := ctx.Value(tunnelKey{}).(bool)
+ return v
+}
+
+type markedHandler struct{ h http.Handler }
+
+func (m markedHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ m.h.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), tunnelKey{}, true)))
+}
+
+// Mark wraps h so every request it serves carries the tunnel mark.
+func Mark(h http.Handler) http.Handler { return markedHandler{h} }
+
+type viaTunnelKey struct{}
+
+// ViaTunnel reports that a request was served over a tunnel at all, TLS or
+// not. It grants nothing; the edge uses it only to NOT count a request the
+// hub already counted (geoip spec §9, codex R4-1).
+func ViaTunnel(ctx context.Context) bool {
+ v, _ := ctx.Value(viaTunnelKey{}).(bool)
+ return v
+}
+
+type viaHandler struct{ h http.Handler }
+
+func (v viaHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ v.h.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), viaTunnelKey{}, true)))
+}
+
+// servedHandler is what Loop serves: always via-marked; trust-marked only
+// over https, because the registration token authenticates the box to the
+// hub, not the hub to the box (codex R2-2).
+func servedHandler(cfg Config, h http.Handler) http.Handler {
+ if u, err := url.Parse(cfg.URL); err == nil && u.Scheme == "https" {
+ h = Mark(h)
+ }
+ return viaHandler{h}
+}
+```
+
+- In `Config`, add `TLSConfig *tls.Config` with the comment: `// nil = system roots and ServerName from the URL; tests pass a pool`.
+- Thread it into `Dial`: add `DialConfig(ctx, cfg Config)`, have the existing `Dial(ctx, url, name, token)` call it, and in the `https` branch clone `cfg.TLSConfig` when it is set, then fill in `ServerName` and `NextProtos`.
+- In `Loop`, compute `handler := servedHandler(cfg, cfg.Handler)` once, and pass `handler` to `serveUntilDead`.
+
+`internal/edge/geoipheaders.go`: replace the stub with:
+
+```go
+var fromTunnel = tunnelclient.FromTunnel
+```
+
+and import `internal/tunnelclient`. The edge's tests already import it. Check that `go list -deps ./internal/tunnelclient` doesn't include `internal/edge`, so there's no import cycle.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/tunnelclient -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Tunnel' -count=1`
+Expected: PASS, including the existing `TestTunnelEndToEnd`.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/tunnelclient/ internal/edge/geoipheaders.go internal/edge/tunnel_geo_e2e_test.go internal/edge/tunnel_e2e_test.go
+git commit -m "tunnelclient: TLS-only tunnel mark; the remote edge adopts the hub's geo (geoip spec §8.4)"
+```
+
+---
+
+### Task 11: Analytics countries, end to end
+
+**Files:**
+- Modify:
+ - `internal/analytics/meter.go` (`Obs`; `delta`; `Observe` pageview block);
+ - `internal/analytics/day.go` (`Delta`, `Day.GeoSources`, `Merge`);
+ - `internal/analytics/read.go` (`Summary`, `ReadRange`);
+ - `internal/edge/analytics.go` (`observeAnalytics`);
+ - `internal/console/apianalytics.go`;
+ - `apimode.go` (`apiAnalyticsResponse`);
+ - `analytics_cmd.go`.
+- Test: `internal/analytics/countries_test.go`, plus an assertion added to `internal/edge/analytics_test.go`.
+
+**Interfaces:**
+- Consumes: `geoAnswerFrom` (Task 9).
+- Produces:
+ - `Obs.Country`, `Obs.GeoSource string`, `Obs.GeoState string` (`"hit"|"miss"|"off"|""`);
+ - `Delta.Countries`, `Delta.GeoSources map[string]int64`;
+ - ``Day.GeoSources map[string]int64 `json:"geo_sources,omitempty"` ``;
+ - `Summary.Countries`, `Summary.GeoSources map[string]int64`;
+ - the API's `countries` and `geo_sources`.
+
+Counting rules (spec §9):
+
+- **Counted only for pageviews.**
+- **By state:**
+ - `hit` with a country counts that country;
+ - `hit` with only coordinates, or `miss`, counts `"_unknown"`;
+ - `off` or `""` counts nothing.
+- **`GeoSources[source]++`** whenever the state is `hit` or `miss`.
+- **Bounds:**
+ - `Countries` keeps at most 300 keys, then `_other`;
+ - `GeoSources` keeps at most 8 keys, then `_other`;
+ - the API's `countries` is the top 10 plus `_other` (the sum of the rest).
+
+- [ ] **Step 1: Write the failing tests.** `internal/analytics/countries_test.go`:
+
+```go
+package analytics
+
+import (
+ "context"
+ "encoding/json"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func pv(country, src, state string) Obs {
+ return Obs{Acct: "a", App: "x", Host: "x.test", Method: "GET", Status: 200, ContentType: "text/html",
+ Path: "/", UA: "Mozilla/5.0", ClientIP: "203.0.113.1", When: time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC),
+ Country: country, GeoSource: src, GeoState: state}
+}
+
+func TestMeterCountsCountriesByState(t *testing.T) {
+ m := newTestMeter(t) // use the constructor the existing meter tests use
+ m.Observe(pv("GB", "DBIP-City-Lite", "hit"))
+ m.Observe(pv("", "DBIP-City-Lite", "hit")) // coordinate-only
+ m.Observe(pv("", "DBIP-City-Lite", "miss"))
+ m.Observe(pv("", "", "off"))
+ nonPV := pv("FR", "DBIP-City-Lite", "hit")
+ nonPV.Status = 404
+ m.Observe(nonPV)
+ d := onlyDelta(t, m) // drain helper from the existing tests
+ if d.Countries["GB"] != 1 || d.Countries["_unknown"] != 2 || len(d.Countries) != 2 {
+ t.Fatalf("countries = %v", d.Countries)
+ }
+ if d.GeoSources["DBIP-City-Lite"] != 3 || len(d.GeoSources) != 1 {
+ t.Fatalf("sources = %v", d.GeoSources)
+ }
+}
+
+func TestReadRangeMergesCountriesAcrossBoxesDaysAndOldDays(t *testing.T) {
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ put := func(box, date string, d Day) {
+ raw, _ := json.Marshal(d)
+ if _, err := s.Put(ctx, DayKey("a", "x", box, date), raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ // An old day: no geo_sources, empty countries.
+ put("b1", "2026-10-04", Day{V: 1, Date: "2026-10-04", Countries: map[string]int64{}})
+ put("b1", "2026-10-05", Day{V: 1, Date: "2026-10-05", Countries: map[string]int64{"GB": 3}, GeoSources: map[string]int64{"DBIP-City-Lite": 3}})
+ put("b2", "2026-10-05", Day{V: 1, Date: "2026-10-05", Countries: map[string]int64{"GB": 1, "IE": 2}, GeoSources: map[string]int64{"GeoLite2-City": 3}})
+ sum, err := ReadRange(ctx, s, "a", "x", time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC), time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if sum.Countries["GB"] != 4 || sum.Countries["IE"] != 2 {
+ t.Fatalf("countries = %v", sum.Countries)
+ }
+ if sum.GeoSources["DBIP-City-Lite"] != 3 || sum.GeoSources["GeoLite2-City"] != 3 {
+ t.Fatalf("both sources must be credited (spec §9): %v", sum.GeoSources)
+ }
+}
+
+func TestDayMergeAddsCountries(t *testing.T) {
+ var d Day
+ d.Merge(Delta{Box: "b", Seq: 1, Countries: map[string]int64{"GB": 1}, GeoSources: map[string]int64{"s": 1}})
+ d.Merge(Delta{Box: "b", Seq: 2, Countries: map[string]int64{"GB": 2, "IE": 1}, GeoSources: map[string]int64{"s": 3}})
+ if d.Countries["GB"] != 3 || d.Countries["IE"] != 1 || d.GeoSources["s"] != 4 {
+ t.Fatalf("day = %v %v", d.Countries, d.GeoSources)
+ }
+}
+
+func TestBoundedAddFoldsIntoOther(t *testing.T) {
+ m := map[string]int64{}
+ for i := 0; i < 12; i++ {
+ boundedAdd(m, string(rune('A'+i))+"X", 1, 8)
+ }
+ if len(m) != 9 || m["_other"] != 4 {
+ t.Fatalf("bounded = %v", m)
+ }
+}
+```
+
+Replace `newTestMeter` and `onlyDelta` with the helpers `meter_test.go` already uses to build a `Meter` and drain one delta (`grep -n "func new.*Meter\|Drain()" internal/analytics/*_test.go`).
+
+In `internal/edge/tunnel_geo_e2e_test.go`, give the hub and the remote edge an analytics meter each (`EnableAnalytics` with the app switched on through `SetAnalyticsOn`), send one fresh request and one that the hub answers from its cache, then drain both meters. Across the two meters, pageviews, `Countries["GB"]` and `GeoSources` must each total exactly 2: one per served response, never 3 or 4. The same must hold over a plaintext tunnel, where pageviews total 2 and countries 0.
+
+In `internal/edge/analytics_test.go`, extend `TestEdgeObservesOnlyWhenSwitchedOn` (or add a sibling test). With the City fixture in `e.geo`, a pageview from 81.2.69.142 records `Obs.Country == "GB"`, `GeoState == "hit"` and `GeoSource == "GeoIP2-City"`. Also fill in the Task 10 marker in `tunnel_geo_e2e_test.go`.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/analytics -count=1`
+Expected: build failures.
+
+- [ ] **Step 3: Implement it**
+
+`meter.go`:
+- Add to `Obs`: `Country string`, `GeoSource string`, `GeoState string` (comment: `// geoip spec §9: hit | miss | off`).
+- In `delta(...)`'s literal, add `Countries: map[string]int64{}, GeoSources: map[string]int64{}`.
+- In `Observe`'s `if pv {` block, after the referer:
+
+```go
+ switch o.GeoState {
+ case "hit", "miss":
+ c := o.Country
+ if o.GeoState == "miss" || c == "" {
+ c = "_unknown"
+ }
+ boundedAdd(d.Countries, c, 1, maxCountryKeys)
+ if o.GeoSource != "" {
+ boundedAdd(d.GeoSources, o.GeoSource, 1, maxGeoSourceKeys)
+ }
+ }
+```
+
+`day.go`:
+
+```go
+const (
+ maxCountryKeys = 300
+ maxGeoSourceKeys = 8
+)
+
+// boundedAdd adds n to m[k], folding new keys past max into OtherKey.
+func boundedAdd(m map[string]int64, k string, n int64, max int) {
+ if _, ok := m[k]; ok || k == OtherKey || len(m) < max {
+ m[k] += n
+ return
+ }
+ m[OtherKey] += n
+}
+
+func mergeCounts(dst, src map[string]int64, max int) map[string]int64 {
+ if dst == nil {
+ dst = map[string]int64{}
+ }
+ for k, n := range src {
+ boundedAdd(dst, k, n, max)
+ }
+ return dst
+}
+```
+
+- Add `Countries map[string]int64` and `GeoSources map[string]int64` to `Delta`.
+- Add ``GeoSources map[string]int64 `json:"geo_sources,omitempty"` `` to `Day`.
+- In `Merge`, replace the `if d.Countries == nil { … }` block with:
+ `d.Countries = mergeCounts(d.Countries, x.Countries, maxCountryKeys)` and `d.GeoSources = mergeCounts(d.GeoSources, x.GeoSources, maxGeoSourceKeys)`.
+- `Requeue` and `Drain` copy deltas by pointer, so the new maps travel with them. Read `Requeue` (meter.go:252) to confirm it doesn't rebuild a `Delta` field by field. If it does, add the two maps there too.
+
+`boundedAdd(m, k, n, 8)` keeps 8 real keys plus `_other`, so the map has len 9, which matches the test. With `len(m) < max` and `_other` itself counted in `len(m)`, the 9th distinct key lands in `_other` only once `_other` exists. Walk through the test's 12 inserts:
+
+| Inserts | Effect | `len(m)` after |
+|---|---|---|
+| keys 1-8 | each enters as a real key | 8 |
+| key 9 | goes to `_other` | 9 |
+| keys 10-12 | go to `_other` | 9 |
+
+`_other` ends at 4. That matches the test's assertion.
+
+`read.go`:
+- Add `Countries map[string]int64` and `GeoSources map[string]int64` to `Summary`.
+- In `ReadRange`, keep `countries, sources := map[string]int64{}, map[string]int64{}` beside `pages, refs, events`. For each day, add `mergeCounts(countries, day.Countries, maxCountryKeys)` and `mergeCounts(sources, day.GeoSources, maxGeoSourceKeys)`.
+- At the end, set `sum.Countries, sum.GeoSources = countries, sources`.
+
+`internal/edge/analytics.go` `observeAnalytics`: first, right after the meter/switch guard,
+
+```go
+ // One observation per served response (geoip spec §9, codex R4-1): the
+ // hub that forwarded this over the tunnel already observed it.
+ if tunnelclient.ViaTunnel(r.Context()) {
+ return
+ }
+```
+
+then, before `ref.Observe(...)`,
+
+```go
+ geoAns, _ := geoAnswerFrom(r.Context())
+```
+
+and in the `analytics.Obs{...}` literal: `Country: geoAns.Country, GeoSource: geoAns.Source, GeoState: string(geoAns.State),`.
+
+The `r` the defer sees is the reassigned one from Task 9, because the closure captures the variable. Confirm that `observeAnalytics` is called with that same `r` (edge.go ~:1567).
+
+`internal/console/apianalytics.go`:
+- Add ``Countries map[string]int64 `json:"countries"` `` and ``GeoSources map[string]int64 `json:"geo_sources"` `` to `analyticsResponse`.
+- Fill them with `topWithOther(sum.Countries, 10)` and `sum.GeoSources`.
+- Default nil to `{}`, like the others at :84-89.
+
+Add:
+
+```go
+// topWithOther keeps the n largest keys (ties by key) and sums the rest
+// into "_other" (geoip spec §9: top 10 plus other).
+func topWithOther(m map[string]int64, n int) map[string]int64 {
+ type kv struct {
+ k string
+ v int64
+ }
+ var all []kv
+ for k, v := range m {
+ if k != "_other" {
+ all = append(all, kv{k, v})
+ }
+ }
+ sort.Slice(all, func(i, j int) bool {
+ if all[i].v != all[j].v {
+ return all[i].v > all[j].v
+ }
+ return all[i].k < all[j].k
+ })
+ out := map[string]int64{}
+ other := m["_other"]
+ for i, e := range all {
+ if i < n {
+ out[e.k] = e.v
+ } else {
+ other += e.v
+ }
+ }
+ if other > 0 {
+ out["_other"] = other
+ }
+ return out
+}
+```
+
+with a table test in `internal/console/apianalytics_test.go`: 12 countries, then `_other` holds the sum of the smallest two plus any existing `_other`.
+
+- `apimode.go` `apiAnalyticsResponse`: add the same two fields.
+- `analytics_cmd.go`: after the referers table, call `printAnalyticsTable(w, "Countries", resp.Countries)`, with the exact helper signature at `analytics_cmd.go:157`. **The heading string is copy:** write it as `Countries`, and record it for Task 13's copy review.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/analytics ./internal/console -run 'Countr|Analytics|TopWithOther' -count=1`, then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Analytics|TunnelGeo' -count=1`, then `nice -n 19 go test . -run Analytics -count=1`.
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/analytics/ internal/edge/analytics.go internal/edge/analytics_test.go internal/edge/tunnel_geo_e2e_test.go internal/console/apianalytics.go internal/console/apianalytics_test.go apimode.go analytics_cmd.go
+git commit -m "analytics: count pageviews by country with per-day source provenance (geoip spec §9)"
+```
+
+---
+
+### Task 12: Console doors, startup jobs and `carlos system geoip`
+
+**Files:**
+- Create: `internal/console/apisystemgeoip.go`, `internal/console/apisystemgeoip_test.go`, `system_geoip.go`, `system_geoip_test.go`
+- Modify:
+ - `internal/console/authcli.go` (routes ~:883);
+ - `internal/console/svcroutes_test.go` (`closedDoors`);
+ - `internal/console/server.go` (a `geoip *geoipPublisher` field, built where `Releases` is known; start the daily job);
+ - `system.go` (verb allowlist :50 and dispatch :56-70; usage text);
+ - `apimode.go` (session methods).
+
+**Interfaces:**
+- Consumes: `geoipPublisher` (Task 6), `status.BoxStale`, `s.systemBoxes` (`apisystemupdate.go:232`), `apiSystemRoller`, `systemBucket`.
+- Produces:
+ - `PUT /api/cli/system/geoip?force=1` with a raw body. It stages the upload, starts a job, and answers 202 `{"job": "<id>"}` (spec §4.2, codex R4-2):
+
+ | Status | Meaning |
+ |---|---|
+ | 202 | job started |
+ | 409 | busy, answered before the body is read |
+ | 413 | over 1 GiB |
+ | 503 | no bucket |
+
+ The job's result (the pointer, or the refusal category for a 422-class refusal) is read at the existing `GET /api/cli/system/jobs/{id}`.
+
+ - `GET /api/cli/system/geoip`, answering:
+ ```json
+ {"pointer": {...} | null, "expires_at": "...", "boxes": [{"label","region","stale","state","geoip":{...}}], "verdict": "ready"|"not ready", "blocking": ["label: reason"], "partial": bool}
+ ```
+ - `DELETE /api/cli/system/geoip`, answering 200 `{"off": true, "feed_set": bool}`.
+ - The CLI: `carlos system geoip publish <file> [-force]`, `carlos system geoip status`, `carlos system geoip off`.
+
+**The verdict** (spec §5.5, codex R4 on R1-16):
+
+- **Expected boxes:** the box names under `control/edges/` (list the prefix; strip `.json`).
+- **Heartbeats:** `status.LoadBoxesComplete` on the platform prefix, plus each fleet prefix read the way `systemBoxes(ctx, true)` reads them.
+- **Listed boxes:** every expected box, plus any heartbeat box whose `Updated` is within 7 days.
+- An expected box with no heartbeat blocks, as `"<label>: no heartbeat"`.
+- An incomplete read blocks, as `"heartbeats: unreadable"`.
+
+**Ready** only when, for every listed box:
+
+- it isn't `BoxStale`;
+- its `Metrics.GeoIP != nil`;
+- if its mode is `mirror`, its `SHA256` equals the pointer's (when a live pointer exists).
+
+`partial` (a fleet's records unreadable) also makes it not ready. `blocking` lists one string per box in the way:
+
+- `"<label>: stale"`
+- `"<label>: old binary"`
+- `"<label>: sha <8> not <8>"`
+- `"<label>: heartbeat unreadable"`
+
+- [ ] **Step 1: Write the failing tests.** In `internal/console/apisystemgeoip_test.go`, use whatever helper the existing `apisystemupdate` tests use to stand up a server with a `DirStore` and a logged-in ops owner (`grep -n "func.*systemTestServer\|apiSystemRoller" internal/console/*_test.go`). Tests:
+
+1. A PUT of the City fixture as an ops owner gives 200, and the pointer exists. Use `withFixtureProbes(t)` from Task 6.
+2. A PUT by a non-operator person gives 404. A service token gives 404, through `cliRoute`'s zero `Set`.
+3. A PUT of junk gives 422, and the body names the category.
+4. A GET with two box records gives the verdict:
+ - one mirror box at the pointer's sha, fresh;
+ - one box with no `geoip` block, which reads as not ready, with `blocking` naming it as "old binary".
+ Write the records with `status.WriteBox`-shaped JSON under `status/boxes/`.
+5. A GET where every box is fresh, aware and at the sha gives `ready`. A heartbeat stamped `SnapshotPeriod + 59s` ago still counts as fresh (codex R3-4).
+6. A DELETE removes the pointer and the blobs.
+7. `closedDoors` gains the three new patterns. The existing equality test fails until it's updated.
+
+In `system_geoip_test.go` (package main): follow `system_test.go`'s pattern (it fakes the console with `httptest`). Check that:
+
+- `publish` streams the file body with `Content-Type: application/octet-stream` and the right `Content-Length`;
+- `status` prints the verdict line;
+- `off` prints the feed warning when `feed_set` is true.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -run 'SystemGeoip|SvcRoutes|ClosedDoors' -count=1` and `nice -n 19 go test . -run SystemGeoip -count=1`
+Expected: FAIL.
+
+- [ ] **Step 3: Implement it**
+
+**Publish as upload then job** (spec §4.2, codex R4-2). Split `geoipPublisher.Publish` into two methods. The tests in Task 6 keep calling `Publish`, which becomes `Stage` followed by `Commit`.
+
+- `Stage(ctx, src) (staged, error)` copies the upload to the temp dir under `copyCapped`. Before copying it takes the mutex with `TryLock`, and returns `errGeoipBusy` if it can't. On success the mutex stays held and passes to the job, which releases it.
+- `Commit(ctx, staged, opts) (geoip.Pointer, error)` runs §4.1 steps 3 to 6, then removes the temp dir and unlocks.
+
+The handler:
+1. calls `Stage` within the request;
+2. registers a job in `s.systemJobs` (read how `handleAPISystemUpdate`, `apisystemupdate.go:345`, creates and records one, and use the same type);
+3. runs `Commit` in a goroutine under a fresh `context.WithTimeout(context.Background(), 15*time.Minute)`, because the request's context dies with the 202;
+4. answers 202.
+
+**Test:** stand up the console behind a real backhaul (`newBackhaulScene`), or, if that harness can't host a console handler, an `httputil.ReverseProxy` with `Transport.ResponseHeaderTimeout = 50ms`. Delay `Commit` by 200ms through a test hook (a package var `geoipCommitDelay`). The PUT must get its 202 through the proxy, and polling the job must end in done with the pointer.
+
+The handler sketch below shows the gate, the deadline and the error mapping. Its synchronous `Publish` call becomes `Stage`, then job registration, then a 202, as described above. The `errGeoipRefused` mapping moves into the job's error message, and the CLI prints it.
+
+**The handlers** (`apisystemgeoip.go`):
+
+```go
+func (s *Server) handleAPISystemGeoipPut(w http.ResponseWriter, r *http.Request) {
+ if _, _, ok := s.apiSystemRoller(w, r); !ok {
+ return
+ }
+ if !s.systemBucket(w) {
+ return
+ }
+ // The server-wide ReadTimeout is 5 minutes (cmd/console/main.go); a
+ // 1 GiB upload needs its own deadline, the way the ship door does it.
+ rc := http.NewResponseController(w)
+ _ = rc.SetReadDeadline(time.Now().Add(15 * time.Minute))
+ defer rc.SetReadDeadline(time.Time{})
+ body := http.MaxBytesReader(w, r.Body, geoip.MaxBlobBytes)
+ ptr, err := s.geoip.Publish(r.Context(), body, geoipPublishOpts{Origin: "manual", Force: r.URL.Query().Get("force") == "1"})
+ var mbe *http.MaxBytesError
+ switch {
+ case errors.As(err, &mbe):
+ jsonError(w, http.StatusRequestEntityTooLarge, "the database is larger than 1 GiB")
+ case errors.Is(err, errGeoipBusy):
+ jsonError(w, http.StatusConflict, "another geoip publish is in progress. try again when it finishes")
+ case errors.Is(err, errGeoipRefused):
+ log.Printf("console: geoip publish refused: %v", err)
+ jsonError(w, http.StatusUnprocessableEntity, err.Error())
+ case err != nil:
+ log.Printf("console: geoip publish: %v", err)
+ jsonError(w, http.StatusBadGateway, "could not publish the database")
+ default:
+ s.auditSystem(r, "geoip publish "+ptr.SHA256[:8]) // match auditSystem's real signature (apisystemupdate.go:159)
+ writeJSON(w, http.StatusOK, ptr)
+ }
+}
+```
+
+`handleAPISystemGeoipGet` and `handleAPISystemGeoipDelete` follow the same gate. Build the GET's body from:
+
+- `s.geoip.Current`, plus `geoip.ExpiresAt`;
+- `s.systemBoxes(ctx, false)`;
+- a pure function `geoipVerdict(ptr *geoip.Pointer, boxes []status.Box, partial bool, now time.Time) (ready bool, blocking []string)`.
+
+Unit-test `geoipVerdict` directly with a table, as well as through the handler.
+
+The DELETE calls `s.geoip.Off`, then answers `feed_set: os.Getenv("CONSOLE_GEOIP_FEED") != ""`. Plan 2 replaces that with config.
+
+**Routes** (authcli.go, beside antivirus), and add the three patterns to `closedDoors`:
+
+```go
+ s.cliRoute("PUT /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipPut)
+ s.cliRoute("GET /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipGet)
+ s.cliRoute("DELETE /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipDelete)
+```
+
+Update the comment block above them to say the geoip doors use `apiSystemRoller`.
+
+**Server wiring:**
+
+- Where `Server` is constructed and `cfg.Releases` is known, set:
+ `s.geoip = newGeoipPublisher(cfg.Releases, geoipTmpRoot(cfg))`.
+ `geoipTmpRoot` is `filepath.Dir(<the console DB path>)`. Find how the server learns `CONSOLE_DB`, falling back to `os.TempDir()`. The temp root must be on disk, not tmpfs, because the upload can be 1 GiB.
+- If `cfg.Releases` is nil, `s.geoip` stays nil, and the handlers' `systemBucket` check already answers 503 first.
+- Start a goroutine from the same place the other console background loops start (`grep -n "go s\.\|StartAlertWatcher" cmd/console/main.go internal/console/server.go`):
+
+```go
+// StartGeoipJobs runs the orphan sweep and D1 expiry at startup and daily
+// (geoip spec §3, D1).
+func (s *Server) StartGeoipJobs(ctx context.Context) {
+ if s.geoip == nil {
+ return
+ }
+ go func() {
+ for {
+ if err := s.geoip.Sweep(ctx); err != nil {
+ log.Printf("console: geoip sweep: %v", err)
+ }
+ if _, err := s.geoip.ExpireIfDue(ctx, time.Now()); err != nil {
+ log.Printf("console: geoip expiry check: %v", err)
+ }
+ select {
+ case <-ctx.Done():
+ return
+ case <-time.After(24*time.Hour + rand.N(time.Hour)):
+ }
+ }
+ }()
+}
+```
+
+**The CLI** (`system_geoip.go`, package main):
+
+- Add `"geoip"` to the verb allowlist at `system.go:50`, and dispatch it to `runSystemGeoipVerb(rest)`, the way `antivirus` dispatches.
+- Use `requireDeployAPISession` (the session `carlos system status` uses).
+- `publish` opens the file, `Stat`s it, and does a `PUT` with the `*os.File` as the body and `ContentLength` set. On a 202 it polls `GET /api/cli/system/jobs/{id}` every 2 seconds, the way `carlos system update` polls its job (find that loop in `system.go` and reuse it), until `done`, then prints the pointer or the error. Add a session method next to `doMultipartMethod` (auth.go:496) that sends a raw streamed body with a zero client timeout, like `shipHTTPClientTimeout`.
+- **Output copy, recorded for Task 13:**
+ - publish:
+ ```
+ published <sha8>: <type>, built <date>. every edge picks it up within 11 minutes
+ ```
+ - status: the pointer block, one line per box, then
+ ```
+ ready: every box strips forged headers and has the current database
+ ```
+ or
+ ```
+ not ready: <n> boxes in the way
+ ```
+ followed by the list;
+ - off:
+ ```
+ geoip is off. edges drop the database within 11 minutes
+ ```
+ plus, when the feed is set:
+ ```
+ the feed is still set and will publish again within a day. unset CONSOLE_GEOIP_FEED to keep it off
+ ```
+
+- [ ] **Step 4: Run the tests**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -count=1` (the whole package, because `closedDoors` is package-wide), then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test . -run 'System' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/console/apisystemgeoip.go internal/console/apisystemgeoip_test.go internal/console/authcli.go internal/console/svcroutes_test.go internal/console/server.go cmd/console/main.go system.go system_geoip.go system_geoip_test.go apimode.go auth.go
+git commit -m "console+cli: carlos system geoip publish/status/off; daily sweep and D1 expiry (geoip spec §4.2, §5.5)"
+```
+
+---
+
+### Task 13: Countries on the analytics page, docs, and the copy review
+
+**Files:**
+- Modify: `internal/console/analyticsview.go`, `internal/console/templates/app_analytics.html`, `docs/site/cli.md`, `README.md`
+- Test: the existing analytics view test file (`grep -ln "app_analytics" internal/console/*_test.go`)
+
+**Interfaces:**
+- Consumes: `Summary.Countries/GeoSources` (Task 11), `geoip.AttributionFor` (Task 5), `topWithOther` (Task 11).
+
+- [ ] **Step 1: Gather every new user-facing string from Tasks 11-13 and run the `copy-review` skill on them** before writing them into templates or docs. The strings:
+
+ - **CLI** (Task 11): the `Countries` table heading.
+ - **CLI** (Task 12): the publish, status, verdict and off lines, and every `jsonError` message in `apisystemgeoip.go`.
+ - **Console page:**
+ - the heading `Countries`;
+ - the empty state `no country data for this period`;
+ - the `_unknown` row label `unknown`;
+ - the `_other` row label `other`;
+ - each attribution line from `geoip.AttributionFor`, rendered as a link.
+ - **Docs:** the new `docs/site/cli.md` section "Headers your app receives", and the `carlos system geoip` subsection under `### carlos system`. The README operator paragraph goes next to `CONSOLE_GEOIP_DB` (README.md:560).
+
+ The docs content follows spec §13. App docs:
+ - the five headers;
+ - why each may be absent;
+ - `Vary` and the 8-variant cache cost;
+ - that coordinates are an area;
+ - the attribution duty, by source.
+
+ Operator docs:
+ - publish, status and off;
+ - the rollout order from §15, including "roll every box before publishing";
+ - "the operator owns licence compliance", with links to DB-IP's licence page and MaxMind's GeoLite EULA;
+ - D1;
+ - the lifecycle rule (Task 1) being required before a GeoLite2 publish.
+
+ **The skill is a gate:** apply Paul's edits verbatim. If this runs while Paul is away, launch the review, commit everything else, and leave the task at "awaiting copy review" in the branch tasks and the handoff. **Do not write unreviewed copy into the files.**
+
+- [ ] **Step 2: Write the failing test** for the page:
+ - a summary with `Countries{"GB":3,"_unknown":1}` and `GeoSources{"DBIP-City-Lite":4}` renders a Countries table containing `GB`, the unknown label and the DB-IP attribution link;
+ - a summary with no countries renders the empty-state string;
+ - a summary with two sources renders both attributions.
+
+- [ ] **Step 3: Implement it.**
+ - In `analyticsview.go`, pass to the template:
+ - `Countries` (through `topWithOther(sum.Countries, 10)`, with `_unknown` and `_other` relabelled by the approved strings);
+ - `GeoAttributions []geoip.Attribution`: for each key of `sum.GeoSources` in sorted order whose `AttributionFor` answers, de-duplicated by text.
+ - In `app_analytics.html`, after the Named events block:
+
+```html
+<h3>{{/* approved heading */}}</h3>
+{{if .Countries}}{{template "kvtable" .Countries}}{{else}}<p class="small">{{/* approved empty state */}}</p>{{end}}
+{{range .GeoAttributions}}<p class="small"><a href="{{.URL}}">{{.Text}}</a></p>{{end}}
+```
+
+ Then write the approved docs text into `docs/site/cli.md` and `README.md`.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -run Analytics -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: File the skills discussion and commit**
+
+```bash
+amadan discuss new carlos/platform -branch geoip-for-apps -title "Agent skills: pick up 'Headers your app receives' (GeoIP headers)" -body - <<'EOF'
+App skills are served from carlosframework.com, outside this repo. docs/site/cli.md now documents the X-Carlos-Geo-* headers (geoip spec §7, §13), including the attribution duty. The skill text apps' agents read should pick that section up once it is published.
+EOF
+git add internal/console/analyticsview.go internal/console/templates/app_analytics.html internal/console/*analytics*_test.go docs/site/cli.md README.md
+git commit -m "console+docs: countries on the analytics page with attribution; headers and operator docs (geoip spec §9, §13)"
+```
+
+---
+
+## Final gate (after Task 13)
+
+- [ ] `gofmt -l .` prints nothing. Run `go vet ./...`.
+- [ ] Cross-compile: `GOOS=darwin GOARCH=arm64 go build ./...` and `GOOS=windows GOARCH=amd64 go build ./...`.
+- [ ] The full gate, per memory `gate-tmp-exec-wrapper` (outside the sandbox, with the `-exec` wrapper), `nice -n 19`.
+- [ ] Push, then check `amadan ci status carlos/platform -branch geoip-for-apps`.
+- [ ] A whole-branch review by Astra (codex) against the spec, from a committed brief, before merge.
+
+## Not in this plan
+
+- **Plan 2:** the feed and the `geoip` box alert rule.
+- **Plan 3:** the console's mirrored reader, `/geo`, and `carlos geo`.
+- **Rolling, and applying Task 1's tofu:** Paul's go-ahead.
diff --git a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
index ccc34b4c2..b87d042e2 100644
--- a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
+++ b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
@@ -1,7 +1,7 @@
# Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint
**Date:** 2026-10-06
-**Revision:** 4
+**Revision:** 5
**Discussion:** amadan carlos/platform #241 (quoted in Appendix A)
**Branch:** `geoip-for-apps`
@@ -38,6 +38,12 @@
- which boxes hold a store;
- there is no in-memory store type;
- there is no in-repo page listing the headers apps receive.
+- Rev 5 answers Astra's round 4 (`CODEX-FINDINGS-GEOIP-R4.md`). That round
+ resolved all of round 3, left 1 earlier finding partly resolved, and
+ raised 2 new P2s. All were accepted, marked "(codex R4-N)".
+ - Publishing becomes an upload followed by a job.
+ - A tunnelled request is counted once, by the hub.
+ - The rollout verdict reconciles against the edges records.
## 0. Rulings and decisions
@@ -288,13 +294,30 @@ Under `carlos system`, through the console API:
instead, so they are not the precedent for this gate.
- `carlos system geoip publish <file.mmdb> [-force]` streams the file to
the console, which runs §4.1.
- - The CLI sends a raw `application/octet-stream` body with a
- `Content-Length`, through `PUT /api/cli/system/geoip`.
- - The console's server-wide `ReadTimeout` is 5 minutes. So the handler
- extends its own read deadline to 15 minutes with
- `http.NewResponseController`, the way the ship door does
- (`handleAPIShip`, survey), and wraps the body in `MaxBytesReader` at
- 1 GiB.
+ - **Upload, then a job** (codex R4-2):
+ - The CLI sends a raw `application/octet-stream` body with a
+ `Content-Length`, through `PUT /api/cli/system/geoip`.
+ - The console's server-wide `ReadTimeout` is 5 minutes. So the handler
+ extends its own read deadline to 15 minutes with
+ `http.NewResponseController`, the way the ship door does
+ (`handleAPIShip`, survey), and wraps the body in `MaxBytesReader` at
+ 1 GiB.
+ - The handler only stages the upload: it streams it to the publish
+ path's temporary file and hashes it.
+ - It then answers **202 with a job id** straight away. A request that
+ came through another edge's backhaul has 30 seconds after its body
+ to see response headers (`backhaulHeaderTimeout`), and verifying
+ plus uploading can take longer.
+ - The job runs §4.1 steps 3 to 6 under the publish mutex and the
+ 15-minute deadline.
+ - It reuses the console's existing system job registry
+ (`s.systemJobs`, polled at `GET /api/cli/system/jobs/{id}`, the door
+ `carlos system update` already uses).
+ - The CLI polls until the job is done and prints the result.
+ - A second upload while a job holds the mutex is answered 409 before
+ its body is read.
+ - A test publishes through a real backhaul, with the job delayed past
+ the header timeout.
- The existing `PUT /api/cli/system/upload` holds the whole body in
memory, so it isn't reused.
- `carlos system geoip status` shows:
@@ -466,15 +489,31 @@ fleet boxes write under `fleets/<fleet>/status/boxes/`:
"geoip": {"mode": "none"}
```
-`carlos system geoip status` lists boxes from the same inventory
-`carlos system status` uses: the console's `systemBoxes`, which includes
-the fleet prefixes (codex R3 on R1-16). Boxes whose last heartbeat is more
-than 7 days old are left out as retired. It reports:
+`carlos system geoip status` reconciles two lists (codex R4 on R1-16).
+
+**What it reads:**
+- **Expected boxes:** every box that publishes an edges record
+ (`control/edges/<box>.json`, what the home map reads). These boxes serve
+ traffic.
+- **Heartbeats:**
+ - the platform prefix, read with `status.LoadBoxesComplete`, which says
+ when any record couldn't be read, unlike `LoadBoxes`, which skips
+ unreadable records silently;
+ - every fleet prefix, the way `systemBoxes` reads them, in strict mode.
+
+**Which boxes are listed:**
+- Every expected box is listed whatever its age.
+- A box with a heartbeat but no edges record is listed unless its last
+ heartbeat is more than 7 days old, in which case it is taken as retired.
+
+It reports:
- **stale,** by the shared rule `status.BoxStale` (3 heartbeat periods,
15 minutes), not a threshold of its own (codex R3-4). The verdict can't
vouch for a box it can't currently see.
-- **record unreadable:** "unreadable".
+- **record unreadable**, or an incomplete read of any prefix:
+ "unreadable". An incomplete read is never "ready".
+- **expected box with no heartbeat at all:** "no heartbeat".
- **no block:** "old binary". The headers can't be trusted while any such
box is serving.
- **`mode: tunnel`:** takes the hub's headers.
@@ -710,6 +749,26 @@ show that, and only then publishes a database and the app docs.
- The state crosses hops and the tunnel in `X-Carlos-Hop-Geo` (§8.1a), so
the edge that observes the pageview counts it the way the ingress edge
saw it.
+- **One observation per served response** (codex R4-1).
+ - Today a request to a tunnel or fleet box is observed twice: once by
+ the hub's observer, which runs for every proxied route, and once by
+ the remote edge. Both flush to the same unprefixed
+ `apps/<sqid>/<app>/analytics/`. That already double-counts pageviews,
+ and countries would inherit the same double count.
+ - **Rule:** the remote edge doesn't observe a request that arrived over
+ the tunnel. The hub has already observed it, including when the hub
+ served it from its cache, where the remote never sees it.
+ - **Mark:** `tunnelclient` puts a second context mark, `viaTunnelKey`,
+ on every request it serves, TLS or not. Suppressing a count needs no
+ trust, so it covers plaintext tunnels too. The TLS-only `tunnelKey`
+ stays the geo-trust mark.
+ - **Scope:** usage accounting (`usageMeter`) is a separate meter, and
+ this doesn't change it.
+ - **Test:** a real hub and remote edge with both meters on, both
+ flushed. Pageviews, countries and sources each go up by exactly one,
+ for a fresh request and for a hub cache hit.
+- The sending edge of a backhaul does no metering today (§8.2), so a
+ backhauled request is already observed once, at the home edge.
- `Delta`, the flush and `Day` gain `GeoSources map[string]int64`
(`json:"geo_sources"`): pageviews counted per database type. This is the
provenance the attribution needs. A day's data can come from two sources
@@ -1127,6 +1186,14 @@ Round 3 (`CODEX-FINDINGS-GEOIP-R3.md`): every finding accepted.
| 3 | P2 | Accepted. A coordinate-only `hit` is adopted, and counted as `_unknown` (§8.1a, §9) |
| 4 | P2 | Accepted. `status.BoxStale` (§5.5) |
+Round 4 (`CODEX-FINDINGS-GEOIP-R4.md`): every finding accepted.
+
+| # | Sev | Disposition |
+|---|---|---|
+| R1-16 (partly) | | Reconciles against `control/edges/`; `LoadBoxesComplete` and strict fleet reads; an incomplete read is never ready (§5.5) |
+| 1 | P2 | Accepted. The remote edge skips analytics for tunnelled requests, via a trust-free `viaTunnelKey`; this fixes the existing pageview double count too (§9) |
+| 2 | P2 | Accepted. Upload then a job on the existing system job registry; 202, then the CLI polls (§4.2) |
+
## Appendix A. The request (#241, verbatim)
> **Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint)**
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R4.md b/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R4.md
new file mode 100644
index 000000000..b02124a28
--- /dev/null
+++ b/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R4.md
@@ -0,0 +1,24 @@
+Round 4 review of `d06821cb`.
+
+| Previous finding | Disposition | Evidence from revision 4 |
+|---|---|---|
+| R3-1. Expiry/install races | Resolved | §5.3 checks expiry before download and installation; `genMu` serializes selection, expiry and cleanup. |
+| R3-2. Unprotected console `Locate` | Resolved | §5.4 adds locked `Swappable.Locate`, puts the wrapper in `Config.GeoIP`, and includes concurrent region guesses in testing. |
+| R3-3. Coordinate-only results | Resolved | §8.1a accepts a valid coordinate pair without country; §9 counts that result as `_unknown`. |
+| R3-4. Heartbeat freshness | Resolved | §5.5 uses `status.BoxStale` and tests the normal jittered heartbeat interval. |
+| R1-4, carried forward: orphan destruction | Resolved | §3 adds startup and daily orphan sweeps, making orphaned uploads eligible for §15’s noncurrent-version expiration. |
+| R1-16, carried forward: rollout verification | Partly resolved | §5.5 includes fleet prefixes through `systemBoxes`, but still excludes records older than seven days and has no expected-inventory comparison. Moreover, `systemBoxes` calls `LoadBoxes`, which discards unreadable records: [apisystemupdate.go:233](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/console/apisystemupdate.go:233), [fleet.go:219](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/status/fleet.go:219). “Ready” still needs expected-box reconciliation and explicit handling of incomplete reads. |
+
+1. **P2 — §§8.1a, 9: tunneled pageviews acquire two country counts.** The spec adds country observations at every existing analytics observer and sums them across boxes. A successful tunneled request passes through both the hub’s observer and the remote edge’s observer. With analytics enabled on both, one pageview contributes twice to `Countries` and `GeoSources`. A hub cache hit contributes once, so the distortion depends on cache behavior.
+
+ **Evidence:** the observer runs for every non-status route, including tunnel and fleet routes, at [edge.go:1551](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/edge/edge.go:1551). Analytics switches do not exclude forwarding routes: [telemetry_pass.go:35](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/host/telemetry_pass.go:35). Both boxes publish app analytics through [usage_loop.go:30](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/host/usage_loop.go:30).
+
+ **Fix:** define one analytics observation per served response. Suppress the forwarding hub’s observation for responses served through the tunnel, while retaining observations for responses answered at the hub, including cache hits. Keep usage accounting separate. Test a real hub and remote edge with both meters enabled, flush both, and assert that the aggregated pageview, country and source counts each increase once.
+
+2. **P2 — §§4.1–4.2: the publication deadline does not survive backhaul.** Extending the console’s request-body deadline to fifteen minutes does not extend the sending edge’s response-header deadline. After the upload finishes, verification and bucket publication can legitimately take more than thirty seconds. If the request arrived through another edge, that edge returns 502 before the console’s allowed operation finishes.
+
+ **Evidence:** [edge.go:2674](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/edge/edge.go:2674) sets `backhaulHeaderTimeout` to thirty seconds, and [edge.go:2805](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/edge/edge.go:2805) applies it as `ResponseHeaderTimeout`. The platform-update API already moves lengthy processing into a job at [apisystemupdate.go:345](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/console/apisystemupdate.go:345).
+
+ **Fix:** specify an upload-and-job protocol: stage the bounded upload, return a job identifier promptly, then verify and publish under the existing publication mutex and operation deadline. Have the CLI wait for the job’s result. Test publication through a backhaul with processing delayed beyond its response-header timeout.
+
+Verdict: revise.
\ No newline at end of file