carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Platform GeoIP, plan 1: mirror, headers and analytics countries
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Every proxied request to an app carries `X-Carlos-Geo-*` headers. They are computed by the ingress edge from a bucket-published `.mmdb` that every edge mirrors and hot-swaps. Analytics count pageviews by country, and operators publish with `carlos system geoip`.
6
7**Architecture:**
8- A shared vocabulary package, `internal/geoip`, holds the pointer record,
9 the expiry rule, attribution, and header encoding and validation.
10- `internal/geo` gains a richer `Lookup` and a lock-guarded `Swappable`
11 reader.
12- The console publishes content-addressed blobs and a pointer under
13 `control/geoip/`. Each edge's mirror loop installs immutable generations
14 under `/data/carlos/geoip/`.
15- `ServeHTTP` captures and strips inbound geo headers, authenticates, then
16 computes or adopts one result. Every proxy's `Rewrite` writes that result
17 out.
18- Analytics carry the result's country and source into `Day`.
19
20**Tech Stack:** Go, `github.com/oschwald/maxminddb-golang` v1.13.1 (already in `go.mod`), `net/http/httputil.ReverseProxy`, AWS SDK v2 S3, OpenTofu (`infra/modules/deployment-buckets`).
21
22**Spec:** `docs/superpowers/specs/2026-10-06-platform-geoip-design.md` (rev 7). Read §0 rulings first. Measured facts: `docs/superpowers/plans/2026-10-06-platform-geoip-FACTS.md`.
23
24## Global Constraints
25
26- **Header names**, exactly:
27 - `X-Carlos-Geo-Country`, `X-Carlos-Geo-Subdivision`, `X-Carlos-Geo-Lat`,
28 `X-Carlos-Geo-Lon`, `X-Carlos-Geo-Source`;
29 - the internal one, `X-Carlos-Hop-Geo`.
30- **Bucket keys:**
31 - pointer `control/geoip/current.json`;
32 - blob `control/geoip/<sha256 hex>.mmdb`;
33 - feed status `control/geoip/feed.json` (plan 2 writes it; plan 1 only
34 reserves the name).
35- **Local layout:** `/data/carlos/geoip/<sha256>.mmdb`, `/data/carlos/geoip/current`, and `/data/carlos/geoip/.incoming-*`. The directory is 0755 and the files 0644.
36- **Limits:**
37 - blob at most 1 GiB (`1 << 30`); pointer read at most 64 KiB;
38 - `PutFrom` fallback to `Put` only up to 64 MiB;
39 - publish deadline 15 minutes; mirror pass deadline 10 minutes;
40 - mirror poll every 10 minutes plus up to 1 minute of jitter; expiry
41 ticker every 1 minute.
42- **D1:** a `database_type` with prefix `GeoLite2-` expires 30 days after
43 `build_epoch`. The warning starts at 21 days. Other types never expire.
44- **Coordinates** are formatted with exactly 2 decimal places
45 (`strconv.FormatFloat(v, 'f', 2, 64)`) after
46 `math.Round(v*100)/100`.
47- **Validation:**
48 - country `^[A-Z]{2}$`;
49 - subdivision `^[A-Z]{2}-[A-Z0-9]{1,3}$`, whose prefix equals the
50 country;
51 - source `^[A-Za-z0-9._-]{1,64}$`;
52 - lat in [-90, 90] and lon in [-180, 180], finite.
53- **Logs** never contain an IP address and never contain database bytes.
54 Use fixed categories; the spec §12 table is the wording.
55- **Operator gate:** a person admitted by `apiSystemRoller`
56 (`internal/console/apisystemupdate.go:106`). The routes are registered
57 with `svcrole.Set{}` and added to `closedDoors`
58 (`internal/console/svcroutes_test.go`).
59- **Copy:** no em dashes in user-facing strings. Use a full stop and a new
60 lower-case sentence (memory: copy-voice). Every new user-facing string
61 (CLI output, console page, docs) goes through the `copy-review` skill
62 before it lands. The executor collects them in Task 13.
63- **Cross-compile:** `GOOS=darwin GOARCH=arm64 go build ./...` and
64 `GOOS=windows go build ./...` must pass. Nothing new here is
65 Linux-syscall-specific, but `fsync` of a directory uses `os.File.Sync`,
66 which is portable.
67- **Build environment, inside the sandbox:** `export GOCACHE=$TMPDIR/gocache GOFLAGS="-mod=mod -buildvcs=false" GOPROXY=off`. Packages that bind unix sockets (`internal/edge`, `internal/host`) run outside the sandbox with `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false`, after `mkdir -p /tmp/g`. Run tests `nice -n 19`, one package at a time (CPU budget).
68
69## Review Focus
70
711. **A client-sent `X-Carlos-Geo-*` or `X-Carlos-Hop-Geo` on any path,**
72 including a `Connection:` nomination. The app must see only the edge's
73 values, or none. Tests are in Task 9.
742. **An edge locating something that isn't the visitor:** the box (door),
75 the sender edge (rejected hop), or the hub (tunnel). It must send no
76 headers instead. Tests are in Tasks 9 and 10.
773. **A swap while lookups run.** No lookup may run against a closed
78 mapping, and a source label must never come from a different reader
79 than the result. Tests are in Task 3.
804. **A crash or a full disk mid-install,** or an expired GeoLite2
81 database arriving. The edge must come back to a consistent generation
82 or to none, never to a half file or an expired one. Tests are in
83 Task 7.
845. **Old days mixed with new days** in one analytics range, and a switch
85 of database type mid-range. The country totals must add up, and every
86 source must be credited. Tests are in Task 11.
87
88---
89
90## File structure
91
92| File | Responsibility |
93|---|---|
94| `infra/modules/deployment-buckets/main.tf` | lifecycle rule for `control/geoip/` |
95| `internal/geo/mmdb.go` | `decode`, `Lookup`, `Result`, metadata accessors, `OpenMMDBTrusted`; `Locate` unchanged in behaviour |
96| `internal/geo/swappable.go` (new) | `Swappable`, `Snapshot` |
97| `internal/release/store.go` | `PutStreamer`, implemented by S3, Dir and prefix stores; `PutObjectFrom` helper |
98| `internal/geoip/pointer.go` (new) | keys, `Pointer`, `ParsePointer`, D1 expiry, attribution |
99| `internal/geoip/answer.go` (new) | `Answer` (state, country, ...), app header write and strip, hop value encode, adopt and validate |
100| `internal/console/geoippublish.go` (new) | `geoipPublisher`: publish, off, sweep, expire; probes; shrink check |
101| `internal/console/apisystemgeoip.go` (new) | `PUT/GET/DELETE /api/cli/system/geoip` handlers |
102| `system_geoip.go` (new, package main) | `carlos system geoip publish|status|off` |
103| `internal/edge/geoipmirror.go` (new) | the mirror: local layout, boot, poll, expiry ticker, state |
104| `internal/edge/geoipheaders.go` (new) | `geoResult` context, capture, decide, `setGeo` |
105| `internal/edge/edge.go` | `ServeHTTP` wiring, six `Rewrite`s, the `Edge.geo` field, run wiring |
106| `internal/tunnelclient/tunnelclient.go` | TLS-only tunnel mark, `Config.TLSConfig` test seam |
107| `internal/status/snapshot.go` | `GeoIPState` in `BoxMetrics` and `Probes` |
108| `internal/host/agent.go`, `internal/host/tunnel_client.go` | publish the edge's geo state to the heartbeat; `WithGeoTunnelClient` |
109| `internal/analytics/{meter,day,read}.go` | `Obs.Country/GeoSource/GeoState`, `Delta`, `Day.GeoSources`, merge, read |
110| `internal/console/{apianalytics.go,analyticsview.go,templates/app_analytics.html}` | countries and attribution |
111| `apimode.go`, `analytics_cmd.go` | the CLI prints countries |
112| `docs/site/cli.md`, `README.md` | app and operator docs |
113
114---
115
116### Task 1: Bucket lifecycle rule for `control/geoip/`
117
118**Files:**
119- Modify: `infra/modules/deployment-buckets/main.tf` (after the `aws_s3_bucket_versioning.deployment` resource, ~line 26)
120
121**Interfaces:** none.
122
123- [ ] **Step 1: Add the resource**
124
125```hcl
126# GeoIP databases (platform spec 2026-10-06 §15 step 0, D1): a deleted or
127# superseded blob under control/geoip/ must not live on as a noncurrent
128# version — MaxMind's GeoLite EULA requires destroying superseded editions.
129# Scoped to that prefix only; nothing else in this bucket expires.
130resource "aws_s3_bucket_lifecycle_configuration" "deployment" {
131 bucket = aws_s3_bucket.deployment.id
132
133 rule {
134 id = "geoip-noncurrent"
135 status = "Enabled"
136 filter {
137 prefix = "control/geoip/"
138 }
139 noncurrent_version_expiration {
140 noncurrent_days = 1
141 }
142 abort_incomplete_multipart_upload {
143 days_after_initiation = 1
144 }
145 }
146
147 depends_on = [aws_s3_bucket_versioning.deployment]
148}
149```
150
151- [ ] **Step 2: Validate**
152
153Run: `cd infra/modules/deployment-buckets && tofu fmt -check && tofu init -backend=false >/dev/null && tofu validate`
154Expected: `Success! The configuration is valid.` If `tofu` isn't installed, run `terraform` with the same arguments. If neither is installed, record that in the task report and leave the check to CI.
155
156- [ ] **Step 3: Commit**
157
158```bash
159git add infra/modules/deployment-buckets/main.tf
160git commit -m "infra: expire noncurrent control/geoip/ versions after a day (geoip spec §15)"
161```
162
163The rule is **applied later, by Paul**: a targeted apply through `carlosframework/carloku-infrastructure` with the module ref bumped. Don't apply it.
164
165---
166
167### Task 2: `internal/geo` — `Lookup`, metadata, `OpenMMDBTrusted`
168
169**Files:**
170- Modify: `internal/geo/mmdb.go`
171- Test: `internal/geo/mmdb_test.go`
172
173**Interfaces:**
174- Produces:
175 ```go
176 type Result struct { Country, Subdivision string; Point Point; HasPoint bool }
177 func (m *MMDB) Lookup(ip net.IP) (Result, bool)
178 func (m *MMDB) Source() string
179 func (m *MMDB) BuildEpoch() time.Time
180 func (m *MMDB) NodeCount() uint64
181 func OpenMMDBTrusted(path string) (*MMDB, error)
182 ```
183 `Locate` keeps its signature and exact behaviour.
184
185- [ ] **Step 1: Write the failing tests** (append to `mmdb_test.go`; add `"time"` to its imports)
186
187```go
188func TestLookupCityGivesCountrySubdivisionAndPoint(t *testing.T) {
189 m, err := OpenMMDB("testdata/GeoIP2-City-Test.mmdb")
190 if err != nil {
191 t.Fatal(err)
192 }
193 defer m.Close()
194 cases := map[string]Result{
195 "81.2.69.142": {Country: "GB", Subdivision: "GB-ENG", HasPoint: true},
196 "216.160.83.56": {Country: "US", Subdivision: "US-WA", HasPoint: true},
197 "89.160.20.112": {Country: "SE", Subdivision: "SE-E", HasPoint: true},
198 "2001:218::1": {Country: "JP", HasPoint: true}, // no subdivision in the fixture
199 }
200 for ip, want := range cases {
201 got, ok := m.Lookup(net.ParseIP(ip))
202 if !ok || got.Country != want.Country || got.Subdivision != want.Subdivision || got.HasPoint != want.HasPoint {
203 t.Errorf("Lookup(%s) = (%+v, %v), want %+v", ip, got, ok, want)
204 }
205 }
206 if got, _ := m.Lookup(net.ParseIP("81.2.69.142")); got.Point.Lat < 51 || got.Point.Lat > 52 {
207 t.Errorf("London point = %v", got.Point)
208 }
209 if m.Source() != "GeoIP2-City" {
210 t.Errorf("Source = %q", m.Source())
211 }
212 if m.NodeCount() != 1547 {
213 t.Errorf("NodeCount = %d, want 1547 (FACTS.md)", m.NodeCount())
214 }
215 if m.BuildEpoch().Year() < 2020 {
216 t.Errorf("BuildEpoch = %v", m.BuildEpoch())
217 }
218}
219
220func TestLookupCountryDatabaseHasNoPoint(t *testing.T) {
221 m, err := OpenMMDB("testdata/GeoIP2-Country-Test.mmdb")
222 if err != nil {
223 t.Fatal(err)
224 }
225 defer m.Close()
226 got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
227 if !ok || got.Country != "GB" || got.HasPoint || got.Subdivision != "" {
228 t.Fatalf("country-only = (%+v, %v), want GB with no point (never a centroid)", got, ok)
229 }
230}
231
232func TestLookupNoCountryDatabaseMisses(t *testing.T) {
233 m, err := OpenMMDB("testdata/MaxMind-DB-test-ipv4-24.mmdb")
234 if err != nil {
235 t.Fatal(err)
236 }
237 defer m.Close()
238 if got, ok := m.Lookup(net.ParseIP("81.2.69.142")); ok {
239 t.Fatalf("a database with no country or location answered %+v", got)
240 }
241}
242
243func TestLookupFallsBackToRegisteredCountryButNotItsSubdivision(t *testing.T) {
244 m := &MMDB{lookup: func(_ net.IP, result any) error {
245 rec := result.(*mmdbRecord)
246 rec.RegisteredCountry.ISOCode = "IE"
247 rec.Subdivisions = append(rec.Subdivisions, struct {
248 ISOCode string `maxminddb:"iso_code"`
249 }{"L"})
250 return nil
251 }}
252 got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
253 if !ok || got.Country != "IE" || got.Subdivision != "" {
254 t.Fatalf("registered fallback = (%+v, %v), want IE with no subdivision", got, ok)
255 }
256}
257
258func TestLookupPointWithoutCountrySucceeds(t *testing.T) {
259 lat, lon := 10.0, 20.0
260 m := &MMDB{lookup: func(_ net.IP, result any) error {
261 rec := result.(*mmdbRecord)
262 rec.Location.Latitude, rec.Location.Longitude = &lat, &lon
263 return nil
264 }}
265 got, ok := m.Lookup(net.ParseIP("81.2.69.142"))
266 if !ok || got.Country != "" || !got.HasPoint || got.Point != (Point{10, 20}) {
267 t.Fatalf("point-only = (%+v, %v)", got, ok)
268 }
269 // Locate's existing rule: a valid point succeeds without a country.
270 if p, ok := m.Locate(net.ParseIP("81.2.69.142")); !ok || p != (Point{10, 20}) {
271 t.Fatalf("Locate point-only = (%v, %v)", p, ok)
272 }
273}
274
275func TestLocateRegisteredCountryFallbackUnchanged(t *testing.T) {
276 // Codex R1-9: a country with no representative point falls back to the
277 // registered country's point. "XK" has none in the table; "GB" does.
278 if _, ok := CountryPoint("XK"); ok {
279 t.Skip("XK gained a representative point; pick another code absent from the table")
280 }
281 m := &MMDB{lookup: func(_ net.IP, result any) error {
282 rec := result.(*mmdbRecord)
283 rec.Country.ISOCode, rec.RegisteredCountry.ISOCode = "XK", "GB"
284 return nil
285 }}
286 gb, _ := CountryPoint("GB")
287 if p, ok := m.Locate(net.ParseIP("81.2.69.142")); !ok || p != gb {
288 t.Fatalf("Locate = (%v, %v), want GB's point", p, ok)
289 }
290}
291
292func TestLookupRejectsMalformedCodes(t *testing.T) {
293 m := &MMDB{lookup: func(_ net.IP, result any) error {
294 rec := result.(*mmdbRecord)
295 rec.Country.ISOCode = "gb" // lower case is not ISO
296 rec.Subdivisions = append(rec.Subdivisions, struct {
297 ISOCode string `maxminddb:"iso_code"`
298 }{"TOOLONG"})
299 return nil
300 }}
301 if got, ok := m.Lookup(net.ParseIP("81.2.69.142")); ok {
302 t.Fatalf("malformed codes answered %+v", got)
303 }
304}
305
306func TestOpenMMDBTrustedOpensWithoutVerify(t *testing.T) {
307 m, err := OpenMMDBTrusted("testdata/GeoIP2-City-Test.mmdb")
308 if err != nil {
309 t.Fatal(err)
310 }
311 defer m.Close()
312 if r, ok := m.Lookup(net.ParseIP("81.2.69.142")); !ok || r.Country != "GB" {
313 t.Fatalf("trusted open lookup = (%+v, %v)", r, ok)
314 }
315 if _, err := OpenMMDBTrusted(filepath.Join(t.TempDir(), "missing.mmdb")); err == nil {
316 t.Fatal("a missing file opened")
317 }
318}
319
320func TestSourceIsSanitised(t *testing.T) {
321 m := &MMDB{source: sanitizeSource("GeoLite2-City\n<script>" + strings.Repeat("x", 100))}
322 if s := m.Source(); strings.ContainsAny(s, "\n<>") || len(s) > 64 {
323 t.Fatalf("Source = %q", s)
324 }
325 var nilM *MMDB
326 if nilM.Source() != "" || nilM.NodeCount() != 0 || !nilM.BuildEpoch().IsZero() {
327 t.Fatal("a nil MMDB must answer zero values")
328 }
329 _ = time.Now
330}
331```
332
333Add `"strings"` to the test imports.
334
335- [ ] **Step 2: Run them to see them fail**
336
337Run: `nice -n 19 go test ./internal/geo -run 'Lookup|OpenMMDBTrusted|SourceIs|LocateRegistered' -count=1`
338Expected: build failure. `Lookup`, `Result`, `Source`, `NodeCount`, `BuildEpoch`, `OpenMMDBTrusted`, `sanitizeSource` and `mmdbRecord.Subdivisions` are all undefined.
339
340- [ ] **Step 3: Implement it.** Replace `mmdb.go`'s `MMDB`, `OpenMMDB`, `mmdbRecord` and `Locate` with the following, and keep `Close`, `onTheGlobe`, `cgnat` and `Locatable` as they are:
341
342```go
343// MMDB is an operator-supplied MaxMind-format database. lookup is the
344// reader's Lookup, a field so a test can prove which addresses ever reach
345// it; source, build and nodes are copied from the metadata at open so a
346// test fake can set them too.
347type MMDB struct {
348 r *maxminddb.Reader
349 lookup func(ip net.IP, result any) error
350 source string
351 build time.Time
352 nodes uint64
353}
354
355// OpenMMDB opens and verifies the file. An error means "IP step off"; the
356// console logs it once and starts anyway.
357func OpenMMDB(path string) (*MMDB, error) {
358 r, err := maxminddb.Open(path)
359 if err != nil {
360 return nil, err
361 }
362 if err := r.Verify(); err != nil {
363 r.Close()
364 return nil, err
365 }
366 return newMMDB(r), nil
367}
368
369// OpenMMDBTrusted opens without Verify. Only for a file whose bytes were
370// verified elsewhere and proven identical by digest (geoip spec §5.3):
371// Verify decodes the whole data section, which costs seconds and tens of
372// MiB on every edge for no new information.
373func OpenMMDBTrusted(path string) (*MMDB, error) {
374 r, err := maxminddb.Open(path)
375 if err != nil {
376 return nil, err
377 }
378 return newMMDB(r), nil
379}
380
381func newMMDB(r *maxminddb.Reader) *MMDB {
382 md := r.Metadata
383 return &MMDB{
384 r: r, lookup: r.Lookup,
385 source: sanitizeSource(md.DatabaseType),
386 build: time.Unix(int64(md.BuildEpoch), 0).UTC(),
387 nodes: uint64(md.NodeCount),
388 }
389}
390
391// sanitizeSource reduces a metadata database_type to [A-Za-z0-9._-],
392// at most 64 bytes: it is echoed into a request header and a log line.
393func sanitizeSource(s string) string {
394 var b strings.Builder
395 for _, c := range s {
396 if b.Len() >= 64 {
397 break
398 }
399 if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '.' || c == '_' || c == '-' {
400 b.WriteRune(c)
401 }
402 }
403 return b.String()
404}
405
406// Source is the database's type ("DBIP-City-Lite", "GeoLite2-City"), sanitised.
407func (m *MMDB) Source() string {
408 if m == nil {
409 return ""
410 }
411 return m.source
412}
413
414// BuildEpoch is when the database was built (UTC); zero for a nil MMDB.
415func (m *MMDB) BuildEpoch() time.Time {
416 if m == nil {
417 return time.Time{}
418 }
419 return m.build
420}
421
422// NodeCount is the search tree's size, the shrink check's yardstick.
423func (m *MMDB) NodeCount() uint64 {
424 if m == nil {
425 return 0
426 }
427 return m.nodes
428}
429
430type mmdbRecord struct {
431 Location struct {
432 Latitude *float64 `maxminddb:"latitude"`
433 Longitude *float64 `maxminddb:"longitude"`
434 } `maxminddb:"location"`
435 Country struct {
436 ISOCode string `maxminddb:"iso_code"`
437 } `maxminddb:"country"`
438 RegisteredCountry struct {
439 ISOCode string `maxminddb:"iso_code"`
440 } `maxminddb:"registered_country"`
441 Subdivisions []struct {
442 ISOCode string `maxminddb:"iso_code"`
443 } `maxminddb:"subdivisions"`
444}
445
446var (
447 lookupErrMu sync.Mutex
448 lookupErrLast time.Time
449)
450
451// decode is the one place a record is read and judged (codex R1-9): the
452// Locatable filter, IPv4 normalisation, the fixed-category error log, and
453// the rule that a malformed location makes the WHOLE record a miss.
454func (m *MMDB) decode(ip net.IP) (mmdbRecord, bool) {
455 var rec mmdbRecord
456 if m == nil || m.lookup == nil || !Locatable(ip) {
457 return rec, false
458 }
459 if v4 := ip.To4(); v4 != nil {
460 ip = v4
461 }
462 if err := m.lookup(ip, &rec); err != nil {
463 lookupErrMu.Lock()
464 if time.Since(lookupErrLast) > time.Minute {
465 lookupErrLast = time.Now()
466 log.Print("geo: a GeoIP lookup failed and was treated as a miss")
467 }
468 lookupErrMu.Unlock()
469 return mmdbRecord{}, false
470 }
471 if rec.Location.Latitude != nil && rec.Location.Longitude != nil &&
472 !onTheGlobe(Point{*rec.Location.Latitude, *rec.Location.Longitude}) {
473 // A structurally valid database can still carry a malformed
474 // location (final review). A record wrong in one field is not
475 // trusted on the next.
476 return mmdbRecord{}, false
477 }
478 return rec, true
479}
480
481// Locate returns the record's location (City databases), else its
482// country's point, else its registered country's. Behaviour is exactly
483// what it was before Lookup existed (codex R1-9).
484func (m *MMDB) Locate(ip net.IP) (Point, bool) {
485 rec, ok := m.decode(ip)
486 if !ok {
487 return Point{}, false
488 }
489 if rec.Location.Latitude != nil && rec.Location.Longitude != nil {
490 return Point{*rec.Location.Latitude, *rec.Location.Longitude}, true
491 }
492 if p, ok := CountryPoint(rec.Country.ISOCode); ok {
493 return p, true
494 }
495 return CountryPoint(rec.RegisteredCountry.ISOCode)
496}
497
498// Result is what Lookup found. Country is ISO 3166-1 alpha-2 (the
499// record's country, else its registered country); Subdivision is
500// "CC-XXX" only when the database carries an ISO code for it (GeoLite2
501// does, DB-IP Lite does not, FACTS.md); HasPoint is true only for a real
502// location record, never a country centroid.
503type Result struct {
504 Country string
505 Subdivision string
506 Point Point
507 HasPoint bool
508}
509
510// Lookup answers with whatever the record holds. ok is true when it found
511// a country, a point, or both (geoip spec §6).
512func (m *MMDB) Lookup(ip net.IP) (Result, bool) {
513 rec, ok := m.decode(ip)
514 if !ok {
515 return Result{}, false
516 }
517 var r Result
518 if isISO2(rec.Country.ISOCode) {
519 r.Country = rec.Country.ISOCode
520 if len(rec.Subdivisions) > 0 && isSubdivisionCode(rec.Subdivisions[0].ISOCode) {
521 r.Subdivision = r.Country + "-" + rec.Subdivisions[0].ISOCode
522 }
523 } else if isISO2(rec.RegisteredCountry.ISOCode) {
524 // The registered country says where the block is registered, not
525 // which subdivision of it the visitor is in: no subdivision.
526 r.Country = rec.RegisteredCountry.ISOCode
527 }
528 if rec.Location.Latitude != nil && rec.Location.Longitude != nil {
529 r.Point, r.HasPoint = Point{*rec.Location.Latitude, *rec.Location.Longitude}, true
530 }
531 return r, r.Country != "" || r.HasPoint
532}
533
534func isISO2(s string) bool {
535 return len(s) == 2 && s[0] >= 'A' && s[0] <= 'Z' && s[1] >= 'A' && s[1] <= 'Z'
536}
537
538func isSubdivisionCode(s string) bool {
539 if len(s) < 1 || len(s) > 3 {
540 return false
541 }
542 for i := 0; i < len(s); i++ {
543 if !(s[i] >= 'A' && s[i] <= 'Z' || s[i] >= '0' && s[i] <= '9') {
544 return false
545 }
546 }
547 return true
548}
549```
550
551Add `"strings"` to `mmdb.go`'s imports. `math` is still used by `onTheGlobe`.
552
553- [ ] **Step 4: Run the package's tests**
554
555Run: `nice -n 19 go test ./internal/geo -count=1`
556Expected: PASS, including every test that existed before (`TestLocateRejectsMalformedCoordinates`, `TestLocateNeverLooksUpNonPublicAddresses`, `TestMMDBLookupErrorsNeverLogTheAddress`).
557
558- [ ] **Step 5: Commit**
559
560```bash
561git add internal/geo/mmdb.go internal/geo/mmdb_test.go
562git commit -m "geo: Lookup with country, subdivision and point; metadata accessors; OpenMMDBTrusted (geoip spec §6)"
563```
564
565---
566
567### Task 3: `geo.Swappable`
568
569**Files:**
570- Create: `internal/geo/swappable.go`
571- Test: `internal/geo/swappable_test.go`
572
573**Interfaces:**
574- Consumes: Task 2's `MMDB.Lookup`, `Source`, `BuildEpoch`, `Locate`, `Close`.
575- Produces:
576 ```go
577 type Snapshot struct { Result Result; Found bool; Source string; Loaded bool }
578 func NewSwappable() *Swappable
579 func (s *Swappable) Lookup(ip net.IP) Snapshot
580 func (s *Swappable) Locate(ip net.IP) (Point, bool) // implements Locator
581 func (s *Swappable) Swap(next *MMDB)
582 func (s *Swappable) Loaded() (source string, build time.Time, ok bool)
583 ```
584
585- [ ] **Step 1: Write the failing tests**
586
587```go
588package geo
589
590import (
591 "net"
592 "sync"
593 "sync/atomic"
594 "testing"
595 "time"
596)
597
598func TestSwappableEmptyAnswersNotLoaded(t *testing.T) {
599 s := NewSwappable()
600 if snap := s.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded || snap.Found || snap.Source != "" {
601 t.Fatalf("empty swappable = %+v", snap)
602 }
603 if _, ok := s.Locate(net.ParseIP("81.2.69.142")); ok {
604 t.Fatal("empty swappable located")
605 }
606 var _ Locator = s
607}
608
609func TestSwappableSnapshotCarriesItsReadersSource(t *testing.T) {
610 s := NewSwappable()
611 m, err := OpenMMDB("testdata/GeoIP2-City-Test.mmdb")
612 if err != nil {
613 t.Fatal(err)
614 }
615 s.Swap(m)
616 snap := s.Lookup(net.ParseIP("81.2.69.142"))
617 if !snap.Loaded || !snap.Found || snap.Source != "GeoIP2-City" || snap.Result.Country != "GB" {
618 t.Fatalf("snapshot = %+v", snap)
619 }
620 miss := s.Lookup(net.ParseIP("10.0.0.1"))
621 if !miss.Loaded || miss.Found || miss.Source != "GeoIP2-City" {
622 t.Fatalf("a miss with a loaded reader must say Loaded and name the source: %+v", miss)
623 }
624 s.Swap(nil)
625 if snap := s.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded {
626 t.Fatalf("after Swap(nil) = %+v", snap)
627 }
628}
629
630// Review Focus 3: no lookup ever runs on a closed reader, and a snapshot's
631// source always belongs to the reader that answered.
632func TestSwappableNeverLooksUpAClosedReader(t *testing.T) {
633 s := NewSwappable()
634 var closedWhileInUse atomic.Bool
635 mk := func(src string) *MMDB {
636 var inUse atomic.Int32
637 var closed atomic.Bool
638 m := &MMDB{source: src}
639 m.lookup = func(_ net.IP, result any) error {
640 inUse.Add(1)
641 defer inUse.Add(-1)
642 if closed.Load() {
643 closedWhileInUse.Store(true)
644 }
645 rec := result.(*mmdbRecord)
646 rec.Country.ISOCode = src[:2]
647 time.Sleep(time.Microsecond)
648 return nil
649 }
650 m.closeHook = func() {
651 if inUse.Load() != 0 {
652 closedWhileInUse.Store(true)
653 }
654 closed.Store(true)
655 }
656 return m
657 }
658 s.Swap(mk("AAsrc"))
659 var wg sync.WaitGroup
660 stop := make(chan struct{})
661 var mismatched atomic.Bool
662 for i := 0; i < 32; i++ {
663 wg.Add(1)
664 go func() {
665 defer wg.Done()
666 for {
667 select {
668 case <-stop:
669 return
670 default:
671 }
672 snap := s.Lookup(net.ParseIP("81.2.69.142"))
673 if snap.Found && snap.Result.Country != snap.Source[:2] {
674 mismatched.Store(true)
675 }
676 s.Locate(net.ParseIP("81.2.69.142"))
677 }
678 }()
679 }
680 for i := 0; i < 100; i++ {
681 if i%2 == 0 {
682 s.Swap(mk("BBsrc"))
683 } else {
684 s.Swap(mk("AAsrc"))
685 }
686 }
687 close(stop)
688 wg.Wait()
689 if closedWhileInUse.Load() {
690 t.Fatal("a reader was closed while a lookup was using it")
691 }
692 if mismatched.Load() {
693 t.Fatal("a snapshot carried a different reader's source")
694 }
695}
696```
697
698- [ ] **Step 2: Run it to see it fail**
699
700Run: `nice -n 19 go test ./internal/geo -run Swappable -race -count=1`
701Expected: build failure. `NewSwappable` and `closeHook` are undefined.
702
703- [ ] **Step 3: Implement it.** Add the `closeHook func()` field to `MMDB` (a test seam, nil in production), and make `Close` call it:
704
705```go
706// in mmdb.go, MMDB struct:
707 closeHook func() // test seam: called by Close before the mapping is released
708
709// in mmdb.go, Close:
710func (m *MMDB) Close() error {
711 if m == nil {
712 return nil
713 }
714 if m.closeHook != nil {
715 m.closeHook()
716 }
717 if m.r == nil {
718 return nil
719 }
720 return m.r.Close()
721}
722```
723
724Then create `internal/geo/swappable.go`:
725
726```go
727package geo
728
729import (
730 "net"
731 "sync"
732 "time"
733)
734
735// Snapshot is one answer from one reader: the result, that reader's source,
736// and whether any reader was loaded at all (geoip spec §5.4, codex R2-4).
737// Headers, analytics and attribution all come from one Snapshot, so a swap
738// between the lookup and the response can never relabel the data.
739type Snapshot struct {
740 Result Result
741 Found bool
742 Source string // empty only when Loaded is false
743 Loaded bool
744}
745
746// Swappable is a reader that can be replaced while lookups run. The old
747// mapping is closed only after the write lock has been taken and released,
748// so no lookup is ever in flight on an unmapped file. There is deliberately
749// no Source() method: a source read apart from its lookup is the race this
750// type exists to prevent.
751type Swappable struct {
752 mu sync.RWMutex
753 cur *MMDB
754}
755
756func NewSwappable() *Swappable { return &Swappable{} }
757
758// Lookup answers under one read lock.
759func (s *Swappable) Lookup(ip net.IP) Snapshot {
760 s.mu.RLock()
761 defer s.mu.RUnlock()
762 if s.cur == nil {
763 return Snapshot{}
764 }
765 r, ok := s.cur.Lookup(ip)
766 return Snapshot{Result: r, Found: ok, Source: s.cur.Source(), Loaded: true}
767}
768
769// Locate makes Swappable a Locator, so the console's region guess can hold
770// it without ever seeing the *MMDB underneath (codex R3-2).
771func (s *Swappable) Locate(ip net.IP) (Point, bool) {
772 s.mu.RLock()
773 defer s.mu.RUnlock()
774 if s.cur == nil {
775 return Point{}, false
776 }
777 return s.cur.Locate(ip)
778}
779
780// Loaded reports the current reader's source and build time.
781func (s *Swappable) Loaded() (source string, build time.Time, ok bool) {
782 s.mu.RLock()
783 defer s.mu.RUnlock()
784 if s.cur == nil {
785 return "", time.Time{}, false
786 }
787 return s.cur.Source(), s.cur.BuildEpoch(), true
788}
789
790// Swap installs next (nil = none) and closes the previous reader after
791// every lookup that could see it has finished. Go's RWMutex blocks new
792// readers while a writer waits, so lookups cannot starve a swap.
793func (s *Swappable) Swap(next *MMDB) {
794 s.mu.Lock()
795 old := s.cur
796 s.cur = next
797 s.mu.Unlock()
798 if old != nil && old != next {
799 old.Close()
800 }
801}
802```
803
804- [ ] **Step 4: Run the package's tests under the race detector**
805
806Run: `nice -n 19 go test ./internal/geo -race -count=1`
807Expected: PASS.
808
809- [ ] **Step 5: Commit**
810
811```bash
812git add internal/geo/swappable.go internal/geo/swappable_test.go internal/geo/mmdb.go
813git commit -m "geo: Swappable reader with one-lock snapshots (geoip spec §5.4)"
814```
815
816---
817
818### Task 4: `release.PutStreamer`
819
820**Files:**
821- Modify: `internal/release/store.go` (the interface beside `Opener` at ~:226; the prefixStore methods at ~:133; the DirStore methods after `Put` at ~:374; the S3Store methods after `Put` at ~:667)
822- Test: `internal/release/putfrom_test.go`
823
824**Interfaces:**
825- Produces:
826 ```go
827 type PutStreamer interface {
828 PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
829 }
830 const PutFromFallbackMax = 64 << 20
831 func PutObjectFrom(ctx context.Context, s Store, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
832 ```
833 `PutObjectFrom` uses `PutFrom` when the store has it. Otherwise it reads up to `PutFromFallbackMax` into memory and calls `Put`, and refuses anything larger with `ErrTooLargeForPut`.
834
835The body is an `io.ReadSeeker` on purpose. The publish path always streams from a temporary file (`*os.File`), and the AWS SDK v2 signs or checksums a seekable body without buffering it.
836
837- [ ] **Step 1: Write the failing tests**
838
839```go
840package release
841
842import (
843 "bytes"
844 "context"
845 "errors"
846 "io"
847 "strings"
848 "testing"
849)
850
851func TestDirStorePutFromMatchesPutSemantics(t *testing.T) {
852 ctx := context.Background()
853 d := NewDirStore(t.TempDir())
854 var _ PutStreamer = d
855 etag, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("hello"), 5, "")
856 if err != nil {
857 t.Fatal(err)
858 }
859 got, getEtag, err := d.Get(ctx, "a/b.bin")
860 if err != nil || string(got) != "hello" || getEtag != etag {
861 t.Fatalf("Get = (%q, %q, %v), want hello with etag %q", got, getEtag, err, etag)
862 }
863 if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("again"), 5, ""); !errors.Is(err, ErrCASConflict) {
864 t.Fatalf("create-only over an existing key = %v, want ErrCASConflict", err)
865 }
866 if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("next!"), 5, etag); err != nil {
867 t.Fatalf("replace with the right etag: %v", err)
868 }
869 if _, err := d.PutFrom(ctx, "a/b.bin", strings.NewReader("short"), 9, ""); err == nil {
870 t.Fatal("a body shorter than size must fail")
871 }
872}
873
874func TestPrefixStorePutFromRebases(t *testing.T) {
875 ctx := context.Background()
876 d := NewDirStore(t.TempDir())
877 p := NewPrefixStore(d, "fleets/x/")
878 ps, ok := p.(PutStreamer)
879 if !ok {
880 t.Fatal("prefixStore must forward PutFrom (codex R1-8)")
881 }
882 if _, err := ps.PutFrom(ctx, "k", strings.NewReader("v"), 1, ""); err != nil {
883 t.Fatal(err)
884 }
885 if b, _, err := d.Get(ctx, "fleets/x/k"); err != nil || string(b) != "v" {
886 t.Fatalf("rebased object = (%q, %v)", b, err)
887 }
888}
889
890type putOnlyStore struct{ Store }
891
892func TestPutObjectFromFallsBackOnlyForSmallBodies(t *testing.T) {
893 ctx := context.Background()
894 s := putOnlyStore{NewDirStore(t.TempDir())}
895 if _, err := PutObjectFrom(ctx, s, "small", bytes.NewReader([]byte("abc")), 3, ""); err != nil {
896 t.Fatalf("small fallback: %v", err)
897 }
898 big := io.NewSectionReader(zeroReaderAt{}, 0, PutFromFallbackMax+1)
899 if _, err := PutObjectFrom(ctx, s, "big", big, PutFromFallbackMax+1, ""); !errors.Is(err, ErrTooLargeForPut) {
900 t.Fatalf("big fallback = %v, want ErrTooLargeForPut", err)
901 }
902}
903
904type zeroReaderAt struct{}
905
906func (zeroReaderAt) ReadAt(p []byte, _ int64) (int, error) {
907 for i := range p {
908 p[i] = 0
909 }
910 return len(p), nil
911}
912```
913
914- [ ] **Step 2: Run it to see it fail**
915
916Run: `nice -n 19 go test ./internal/release -run 'PutFrom|PutObjectFrom' -count=1`
917Expected: build failure. `PutStreamer`, `PutFrom`, `PutObjectFrom`, `PutFromFallbackMax` and `ErrTooLargeForPut` are undefined.
918
919- [ ] **Step 3: Implement it.** Next to `Opener`:
920
921```go
922// PutStreamer is the optional streaming write: Store.Put takes the whole
923// body as []byte, which is wrong for a 130 MB GeoIP database on a box
924// under MemoryHigh. Implemented by every production store; a fake with only
925// Put still works through PutObjectFrom for bodies up to PutFromFallbackMax.
926// ifMatch has Put's exact semantics ("" = create-only).
927type PutStreamer interface {
928 PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error)
929}
930
931var (
932 _ PutStreamer = (*DirStore)(nil)
933 _ PutStreamer = (*S3Store)(nil)
934 _ PutStreamer = prefixStore{}
935)
936
937// PutFromFallbackMax bounds how much PutObjectFrom will buffer for a store
938// with no PutFrom.
939const PutFromFallbackMax = 64 << 20
940
941// ErrTooLargeForPut is PutObjectFrom refusing to buffer a large body.
942var ErrTooLargeForPut = errors.New("store cannot stream and the body is too large to buffer")
943
944// PutObjectFrom streams body into s at key.
945func PutObjectFrom(ctx context.Context, s Store, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
946 if ps, ok := s.(PutStreamer); ok {
947 return ps.PutFrom(ctx, key, body, size, ifMatch)
948 }
949 if size > PutFromFallbackMax {
950 return "", fmt.Errorf("%s: %w", key, ErrTooLargeForPut)
951 }
952 b, err := io.ReadAll(io.LimitReader(body, size+1))
953 if err != nil {
954 return "", err
955 }
956 if int64(len(b)) != size {
957 return "", fmt.Errorf("%s: body is %d bytes, want %d", key, len(b), size)
958 }
959 return s.Put(ctx, key, b, ifMatch)
960}
961```
962
963prefixStore (written out for the same reason as `Open`):
964
965```go
966// PutFrom is written out, not promoted, for the reason Open is: a promoted
967// method would forward the UNREBASED key.
968func (p prefixStore) PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
969 return PutObjectFrom(ctx, p.Store, p.full(key), body, size, ifMatch)
970}
971```
972
973DirStore:
974
975```go
976// PutFrom is Put streamed: the body goes to a temp file (hashed as it
977// goes, so the etag is the same content sha256 Put would give) and is
978// renamed into place under d.mu after the same CAS gate.
979func (d *DirStore) PutFrom(_ context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
980 p, err := d.path(key)
981 if err != nil {
982 return "", err
983 }
984 if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
985 return "", err
986 }
987 tmp, err := os.CreateTemp(filepath.Dir(p), filepath.Base(p)+".*.tmp")
988 if err != nil {
989 return "", err
990 }
991 defer os.Remove(tmp.Name())
992 h := sha256.New()
993 n, err := io.Copy(io.MultiWriter(tmp, h), io.LimitReader(body, size+1))
994 if cerr := tmp.Close(); err == nil {
995 err = cerr
996 }
997 if err != nil {
998 return "", err
999 }
1000 if n != size {
1001 return "", fmt.Errorf("dirstore: %s: body is %d bytes, want %d", key, n, size)
1002 }
1003 d.mu.Lock()
1004 defer d.mu.Unlock()
1005 cur, err := os.ReadFile(p)
1006 exists := err == nil
1007 if err != nil && !errors.Is(err, fs.ErrNotExist) {
1008 return "", err
1009 }
1010 if ifMatch == "" && exists {
1011 return "", fmt.Errorf("dirstore: %s exists: %w", key, ErrCASConflict)
1012 }
1013 if ifMatch != "" && (!exists || etagOf(cur) != ifMatch) {
1014 return "", fmt.Errorf("dirstore: %s moved on: %w", key, ErrCASConflict)
1015 }
1016 if err := os.Chmod(tmp.Name(), 0o644); err != nil {
1017 return "", err
1018 }
1019 if err := os.Rename(tmp.Name(), p); err != nil {
1020 return "", err
1021 }
1022 return hex.EncodeToString(h.Sum(nil)), nil
1023}
1024```
1025
1026`ListMod` already excludes `*.tmp` names, so the `.<rand>.tmp` temp file is never listed. Confirm that by reading the comment at ~:376, and if the filter matches only the `.tmp` suffix, the name above already satisfies it.
1027
1028S3Store:
1029
1030```go
1031// PutFrom is Put with a streamed, seekable body and an explicit length.
1032func (s *S3Store) PutFrom(ctx context.Context, key string, body io.ReadSeeker, size int64, ifMatch string) (string, error) {
1033 in := &awss3.PutObjectInput{
1034 Bucket: &s.Bucket,
1035 Key: &key,
1036 Body: body,
1037 ContentLength: aws.Int64(size),
1038 }
1039 if ifMatch == "" {
1040 in.IfNoneMatch = aws.String("*")
1041 } else {
1042 in.IfMatch = aws.String(ifMatch)
1043 }
1044 out, err := s.Client.PutObject(ctx, in)
1045 if err != nil {
1046 if s3CASConflict(err) || (ifMatch != "" && s3NotFound(err)) {
1047 return "", fmt.Errorf("s3store: %s: %w", key, ErrCASConflict)
1048 }
1049 return "", err
1050 }
1051 return aws.ToString(out.ETag), nil
1052}
1053```
1054
1055`internal/fleetapi.store` deliberately gets no `PutFrom`. The console never uses it, and it would need a server side in `internal/edge/storedoor.go`. Add a one-line comment saying so beside its `Opener` assert (`internal/fleetapi/fleetapi.go:179`).
1056
1057- [ ] **Step 4: Run the package's tests**
1058
1059Run: `nice -n 19 go test ./internal/release -count=1`
1060Expected: PASS.
1061
1062- [ ] **Step 5: Commit**
1063
1064```bash
1065git add internal/release/store.go internal/release/putfrom_test.go internal/fleetapi/fleetapi.go
1066git commit -m "release: PutStreamer for S3, dir and prefix stores (geoip spec §4.1)"
1067```
1068
1069---
1070
1071### Task 5: `internal/geoip` — the shared vocabulary
1072
1073**Files:**
1074- Create: `internal/geoip/pointer.go`, `internal/geoip/answer.go`
1075- Test: `internal/geoip/pointer_test.go`, `internal/geoip/answer_test.go`
1076
1077**Interfaces:**
1078- Consumes: `geo.Snapshot` (Task 3).
1079- Produces:
1080 ```go
1081 const Prefix = "control/geoip/"; PointerKey; FeedKey; MaxBlobBytes int64 = 1<<30; MaxPointerBytes = 64<<10
1082 func BlobKey(sha string) string
1083 type Pointer struct{ V int; SHA256, Key string; Size int64; DatabaseType string; BuildEpoch int64; NodeCount uint64; Published time.Time; Origin, Edition string; Expired bool }
1084 func ParsePointer(raw []byte) (Pointer, error)
1085 func (p Pointer) Build() time.Time
1086 func ExpiresAt(dbType string, build time.Time) (time.Time, bool)
1087 func Expired(dbType string, build, now time.Time) bool
1088 type Attribution struct{ Text, URL string }
1089 func AttributionFor(dbType string) (Attribution, bool)
1090 type State string; const StateHit, StateMiss, StateOff
1091 type Answer struct{ State State; Country, Subdivision, Source string; Lat, Lon float64; HasPoint bool }
1092 func FromSnapshot(s geo.Snapshot) Answer
1093 func Off() Answer
1094 const HeaderCountry, HeaderSubdivision, HeaderLat, HeaderLon, HeaderSource, HeaderHop
1095 func StripAll(h http.Header)
1096 func (a Answer) WriteApp(h http.Header) // deletes, then writes the app headers
1097 func (a Answer) HopValue() string // "hit;src=…", "miss;src=…", "off"
1098 type Captured struct{ Hop string; Country, Subdivision, Lat, Lon, Source string }
1099 func Capture(h http.Header) Captured // copies the values, then StripAll
1100 func (c Captured) Adopt() Answer // validates; anything invalid becomes Off or drops the field
1101 ```
1102
1103- [ ] **Step 1: Write the failing tests**
1104
1105`internal/geoip/pointer_test.go`:
1106
1107```go
1108package geoip
1109
1110import (
1111 "strings"
1112 "testing"
1113 "time"
1114)
1115
1116const sha = "9f2c000000000000000000000000000000000000000000000000000000000001"
1117
1118func TestParsePointerAcceptsTheSpecShape(t *testing.T) {
1119 raw := `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":10,
1120 "database_type":"DBIP-City-Lite","build_epoch":1790812800,"node_count":7,
1121 "published":"2026-10-06T21:00:00Z","origin":"manual","edition":""}`
1122 p, err := ParsePointer([]byte(raw))
1123 if err != nil {
1124 t.Fatal(err)
1125 }
1126 if p.Key != BlobKey(sha) || p.Size != 10 || p.DatabaseType != "DBIP-City-Lite" || p.Build().Unix() != 1790812800 {
1127 t.Fatalf("pointer = %+v", p)
1128 }
1129}
1130
1131func TestParsePointerRefusesWhatAnEdgeMustNotFollow(t *testing.T) {
1132 good := `"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":10,"database_type":"X","build_epoch":1`
1133 bad := map[string]string{
1134 "unknown version": `{"v":2,` + good + `}`,
1135 "short sha": `{"v":1,"sha256":"9f","key":"control/geoip/9f.mmdb","size":1,"database_type":"X","build_epoch":1}`,
1136 "key not the blob key": `{"v":1,"sha256":"` + sha + `","key":"apps/x/secret","size":10,"database_type":"X","build_epoch":1}`,
1137 "zero size": `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":0,"database_type":"X","build_epoch":1}`,
1138 "oversize": `{"v":1,"sha256":"` + sha + `","key":"control/geoip/` + sha + `.mmdb","size":1073741825,"database_type":"X","build_epoch":1}`,
1139 "not json": `nope`,
1140 }
1141 for name, raw := range bad {
1142 if _, err := ParsePointer([]byte(raw)); err == nil {
1143 t.Errorf("%s: accepted", name)
1144 }
1145 }
1146}
1147
1148func TestGeoLiteExpiresThirtyDaysAfterBuild(t *testing.T) {
1149 build := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
1150 at, ok := ExpiresAt("GeoLite2-City", build)
1151 if !ok || !at.Equal(build.Add(30*24*time.Hour)) {
1152 t.Fatalf("ExpiresAt = (%v, %v)", at, ok)
1153 }
1154 if Expired("GeoLite2-City", build, build.Add(30*24*time.Hour-time.Second)) {
1155 t.Fatal("expired a second early")
1156 }
1157 if !Expired("GeoLite2-Country", build, build.Add(30*24*time.Hour)) {
1158 t.Fatal("not expired at 30 days")
1159 }
1160 if Expired("DBIP-City-Lite", build, build.Add(400*24*time.Hour)) {
1161 t.Fatal("DB-IP must never expire (D1)")
1162 }
1163 if _, ok := ExpiresAt("GeoIP2-City", build); ok {
1164 t.Fatal("only GeoLite2- expires")
1165 }
1166}
1167
1168func TestAttributionTable(t *testing.T) {
1169 if a, ok := AttributionFor("DBIP-City-Lite"); !ok || a.Text != "IP Geolocation by DB-IP" || a.URL != "https://db-ip.com" {
1170 t.Fatalf("dbip = %+v %v", a, ok)
1171 }
1172 if a, ok := AttributionFor("GeoLite2-Country"); !ok || !strings.Contains(a.Text, "GeoLite2 data created by MaxMind") {
1173 t.Fatalf("geolite = %+v %v", a, ok)
1174 }
1175 if _, ok := AttributionFor("GeoIP2-City"); ok {
1176 t.Fatal("no attribution is known for commercial GeoIP2; the operator's licence governs")
1177 }
1178}
1179```
1180
1181`internal/geoip/answer_test.go`:
1182
1183```go
1184package geoip
1185
1186import (
1187 "net/http"
1188 "testing"
1189
1190 "github.com/carlosframework/platform/internal/geo"
1191)
1192
1193func TestFromSnapshotStates(t *testing.T) {
1194 if a := FromSnapshot(geo.Snapshot{}); a.State != StateOff {
1195 t.Fatalf("not loaded = %+v, want off", a)
1196 }
1197 if a := FromSnapshot(geo.Snapshot{Loaded: true, Source: "DBIP-City-Lite"}); a.State != StateMiss || a.Source != "DBIP-City-Lite" {
1198 t.Fatalf("loaded miss = %+v", a)
1199 }
1200 hit := FromSnapshot(geo.Snapshot{Loaded: true, Found: true, Source: "GeoLite2-City",
1201 Result: geo.Result{Country: "GB", Subdivision: "GB-ENG", Point: geo.Point{Lat: 51.5074, Lon: -0.1278}, HasPoint: true}})
1202 if hit.State != StateHit || hit.Lat != 51.51 || hit.Lon != -0.13 {
1203 t.Fatalf("hit = %+v (coordinates must be rounded to 2 places)", hit)
1204 }
1205}
1206
1207func TestWriteAppHeaders(t *testing.T) {
1208 h := http.Header{}
1209 h.Set(HeaderCountry, "XX") // stale value must go
1210 Answer{State: StateHit, Country: "IE", Source: "DBIP-City-Lite", Lat: 53.34, Lon: -6.27, HasPoint: true}.WriteApp(h)
1211 if h.Get(HeaderCountry) != "IE" || h.Get(HeaderLat) != "53.34" || h.Get(HeaderLon) != "-6.27" ||
1212 h.Get(HeaderSource) != "DBIP-City-Lite" || h.Get(HeaderSubdivision) != "" || h.Get(HeaderHop) != "" {
1213 t.Fatalf("headers = %v", h)
1214 }
1215 h2 := http.Header{}
1216 h2.Set(HeaderSource, "forged")
1217 Answer{State: StateMiss, Source: "DBIP-City-Lite"}.WriteApp(h2)
1218 if len(h2) != 0 {
1219 t.Fatalf("a miss sends no app headers, not even Source: %v", h2)
1220 }
1221}
1222
1223func TestHopValueRoundTrip(t *testing.T) {
1224 for _, a := range []Answer{
1225 {State: StateHit, Country: "GB", Subdivision: "GB-ENG", Source: "GeoLite2-City", Lat: 51.51, Lon: -0.09, HasPoint: true},
1226 {State: StateHit, Source: "GeoLite2-City", Lat: 10, Lon: 20, HasPoint: true}, // coordinate-only (codex R3-3)
1227 {State: StateMiss, Source: "DBIP-City-Lite"},
1228 Off(),
1229 } {
1230 h := http.Header{}
1231 a.WriteApp(h)
1232 h.Set(HeaderHop, a.HopValue())
1233 got := Capture(h).Adopt()
1234 if got != a {
1235 t.Errorf("round trip %+v -> %+v", a, got)
1236 }
1237 if len(h) != 0 {
1238 t.Errorf("Capture must strip every geo header, left %v", h)
1239 }
1240 }
1241}
1242
1243func TestAdoptRefusesWhatItCannotTrust(t *testing.T) {
1244 cases := map[string]http.Header{
1245 "no hop value": {HeaderCountry: {"GB"}, HeaderSource: {"x"}},
1246 "malformed hop value": {HeaderHop: {"maybe"}},
1247 "hit with nothing": {HeaderHop: {"hit;src=x"}},
1248 "hit with bad country": {HeaderHop: {"hit;src=x"}, HeaderCountry: {"gb"}},
1249 "bad source": {HeaderHop: {"hit;src=a b"}, HeaderCountry: {"GB"}},
1250 }
1251 for name, h := range cases {
1252 if got := Capture(h).Adopt(); got.State != StateOff {
1253 t.Errorf("%s: adopted %+v, want off", name, got)
1254 }
1255 }
1256 // Field-level: an invalid subdivision or half a coordinate pair is
1257 // dropped on its own, the rest survives.
1258 h := http.Header{HeaderHop: {"hit;src=GeoLite2-City"}, HeaderCountry: {"GB"},
1259 HeaderSubdivision: {"FR-IDF"}, HeaderLat: {"91"}, HeaderLon: {"0"}}
1260 got := Capture(h).Adopt()
1261 if got.State != StateHit || got.Country != "GB" || got.Subdivision != "" || got.HasPoint {
1262 t.Fatalf("partial = %+v", got)
1263 }
1264}
1265```
1266
1267- [ ] **Step 2: Run them to see them fail**
1268
1269Run: `nice -n 19 go test ./internal/geoip -count=1`
1270Expected: build failure, because the package doesn't exist yet.
1271
1272- [ ] **Step 3: Implement it.** `internal/geoip/pointer.go`:
1273
1274```go
1275// Package geoip is the platform's GeoIP vocabulary (spec
1276// docs/superpowers/specs/2026-10-06-platform-geoip-design.md): the bucket
1277// records the console writes and every edge reads, the GeoLite2 expiry rule
1278// (D1), attribution, and the request headers. No I/O lives here.
1279package geoip
1280
1281import (
1282 "encoding/hex"
1283 "encoding/json"
1284 "errors"
1285 "fmt"
1286 "strings"
1287 "time"
1288)
1289
1290const (
1291 Prefix = "control/geoip/"
1292 PointerKey = Prefix + "current.json"
1293 FeedKey = Prefix + "feed.json"
1294 MaxBlobBytes = int64(1 << 30)
1295 MaxPointerBytes = 64 << 10
1296
1297 geoLiteTTL = 30 * 24 * time.Hour
1298 GeoLiteWarn = 21 * 24 * time.Hour
1299 StaleAfter = 45 * 24 * time.Hour
1300)
1301
1302// BlobKey is the content-addressed object for a database whose bytes hash
1303// to sha (lower-case hex).
1304func BlobKey(sha string) string { return Prefix + sha + ".mmdb" }
1305
1306// Pointer is control/geoip/current.json (spec §3).
1307type Pointer struct {
1308 V int `json:"v"`
1309 SHA256 string `json:"sha256"`
1310 Key string `json:"key"`
1311 Size int64 `json:"size"`
1312 DatabaseType string `json:"database_type"`
1313 BuildEpoch int64 `json:"build_epoch"`
1314 NodeCount uint64 `json:"node_count"`
1315 Published time.Time `json:"published"`
1316 Origin string `json:"origin"`
1317 Edition string `json:"edition"`
1318 Expired bool `json:"expired,omitempty"`
1319}
1320
1321// Build is BuildEpoch as a time.
1322func (p Pointer) Build() time.Time { return time.Unix(p.BuildEpoch, 0).UTC() }
1323
1324// ParsePointer decodes and checks a pointer. An unknown version, a key that
1325// is not this sha's blob key, or an impossible size is an error: an edge
1326// never follows a pointer it cannot fully vouch for.
1327func ParsePointer(raw []byte) (Pointer, error) {
1328 var p Pointer
1329 if err := json.Unmarshal(raw, &p); err != nil {
1330 return Pointer{}, fmt.Errorf("geoip pointer: %w", err)
1331 }
1332 if p.V != 1 {
1333 return Pointer{}, fmt.Errorf("geoip pointer: unknown version %d", p.V)
1334 }
1335 if !IsSHA256Hex(p.SHA256) {
1336 return Pointer{}, errors.New("geoip pointer: sha256 is not 64 lower-case hex characters")
1337 }
1338 if p.Key != BlobKey(p.SHA256) {
1339 return Pointer{}, errors.New("geoip pointer: key is not the sha's blob key")
1340 }
1341 if p.Size <= 0 || p.Size > MaxBlobBytes {
1342 return Pointer{}, fmt.Errorf("geoip pointer: size %d out of range", p.Size)
1343 }
1344 return p, nil
1345}
1346
1347// IsSHA256Hex is 64 lower-case hex characters.
1348func IsSHA256Hex(s string) bool {
1349 if len(s) != 64 || strings.ToLower(s) != s {
1350 return false
1351 }
1352 _, err := hex.DecodeString(s)
1353 return err == nil
1354}
1355
1356// ExpiresAt is D1: a GeoLite2 database expires 30 days after it was built.
1357// ok is false for every other type, which never expires.
1358func ExpiresAt(dbType string, build time.Time) (time.Time, bool) {
1359 if !strings.HasPrefix(dbType, "GeoLite2-") {
1360 return time.Time{}, false
1361 }
1362 return build.Add(geoLiteTTL), true
1363}
1364
1365// Expired reports whether now is at or past ExpiresAt.
1366func Expired(dbType string, build, now time.Time) bool {
1367 at, ok := ExpiresAt(dbType, build)
1368 return ok && !now.Before(at)
1369}
1370
1371// Attribution is the credit a page showing the data must carry (spec §13).
1372type Attribution struct{ Text, URL string }
1373
1374// AttributionFor names the attribution for a database type. ok is false
1375// when none is known: the operator's own licence governs.
1376func AttributionFor(dbType string) (Attribution, bool) {
1377 switch {
1378 case strings.HasPrefix(dbType, "DBIP-"):
1379 return Attribution{"IP Geolocation by DB-IP", "https://db-ip.com"}, true
1380 case strings.HasPrefix(dbType, "GeoLite2-"):
1381 return Attribution{"This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com", "https://www.maxmind.com"}, true
1382 }
1383 return Attribution{}, false
1384}
1385```
1386
1387`internal/geoip/answer.go`:
1388
1389```go
1390package geoip
1391
1392import (
1393 "math"
1394 "net/http"
1395 "regexp"
1396 "strconv"
1397 "strings"
1398
1399 "github.com/carlosframework/platform/internal/geo"
1400)
1401
1402const (
1403 HeaderCountry = "X-Carlos-Geo-Country"
1404 HeaderSubdivision = "X-Carlos-Geo-Subdivision"
1405 HeaderLat = "X-Carlos-Geo-Lat"
1406 HeaderLon = "X-Carlos-Geo-Lon"
1407 HeaderSource = "X-Carlos-Geo-Source"
1408 // HeaderHop is edge-to-edge only (spec §8.1a): never written to a
1409 // request an app receives.
1410 HeaderHop = "X-Carlos-Hop-Geo"
1411)
1412
1413var appHeaders = []string{HeaderCountry, HeaderSubdivision, HeaderLat, HeaderLon, HeaderSource}
1414
1415// State is what the ingress edge did: hit (a reader answered), miss (a
1416// reader was loaded and found nothing, or the address is not locatable),
1417// off (no lookup was attempted).
1418type State string
1419
1420const (
1421 StateHit State = "hit"
1422 StateMiss State = "miss"
1423 StateOff State = "off"
1424)
1425
1426// Answer is one request's geo result, computed once at ingress or adopted
1427// from a trusted transport. Lat/Lon are already rounded to 2 places.
1428type Answer struct {
1429 State State
1430 Country string
1431 Subdivision string
1432 Source string
1433 Lat, Lon float64
1434 HasPoint bool
1435}
1436
1437// Off is the answer for a request no lookup was attempted for.
1438func Off() Answer { return Answer{State: StateOff} }
1439
1440func round2(v float64) float64 { return math.Round(v*100) / 100 }
1441
1442// FromSnapshot turns one reader's answer into a request's answer.
1443func FromSnapshot(s geo.Snapshot) Answer {
1444 switch {
1445 case !s.Loaded:
1446 return Off()
1447 case !s.Found:
1448 return Answer{State: StateMiss, Source: s.Source}
1449 }
1450 a := Answer{State: StateHit, Country: s.Result.Country, Subdivision: s.Result.Subdivision, Source: s.Source}
1451 if s.Result.HasPoint {
1452 a.Lat, a.Lon, a.HasPoint = round2(s.Result.Point.Lat), round2(s.Result.Point.Lon), true
1453 }
1454 return a
1455}
1456
1457// StripAll deletes every geo header an app or an edge could receive.
1458func StripAll(h http.Header) {
1459 for _, k := range appHeaders {
1460 h.Del(k)
1461 }
1462 h.Del(HeaderHop)
1463}
1464
1465// WriteApp deletes the app headers and writes this answer's. A miss and
1466// off write nothing at all; Source is written only beside another field.
1467func (a Answer) WriteApp(h http.Header) {
1468 for _, k := range appHeaders {
1469 h.Del(k)
1470 }
1471 if a.State != StateHit {
1472 return
1473 }
1474 wrote := false
1475 if a.Country != "" {
1476 h.Set(HeaderCountry, a.Country)
1477 wrote = true
1478 if a.Subdivision != "" {
1479 h.Set(HeaderSubdivision, a.Subdivision)
1480 }
1481 }
1482 if a.HasPoint {
1483 h.Set(HeaderLat, strconv.FormatFloat(a.Lat, 'f', 2, 64))
1484 h.Set(HeaderLon, strconv.FormatFloat(a.Lon, 'f', 2, 64))
1485 wrote = true
1486 }
1487 if wrote && a.Source != "" {
1488 h.Set(HeaderSource, a.Source)
1489 }
1490}
1491
1492// HopValue is X-Carlos-Hop-Geo for this answer.
1493func (a Answer) HopValue() string {
1494 if a.State == StateOff || !sourceRE.MatchString(a.Source) {
1495 return string(StateOff)
1496 }
1497 return string(a.State) + ";src=" + a.Source
1498}
1499
1500// Captured is what a request arrived carrying, before anyone trusts it.
1501type Captured struct {
1502 Hop, Country, Subdivision, Lat, Lon, Source string
1503}
1504
1505// Capture copies the geo headers out of h and strips them all (spec §8.1
1506// step 1). Nothing reads the copy until the transport is authenticated.
1507func Capture(h http.Header) Captured {
1508 c := Captured{
1509 Hop: h.Get(HeaderHop), Country: h.Get(HeaderCountry), Subdivision: h.Get(HeaderSubdivision),
1510 Lat: h.Get(HeaderLat), Lon: h.Get(HeaderLon), Source: h.Get(HeaderSource),
1511 }
1512 StripAll(h)
1513 return c
1514}
1515
1516var (
1517 countryRE = regexp.MustCompile(`^[A-Z]{2}$`)
1518 subdivisionRE = regexp.MustCompile(`^[A-Z]{2}-[A-Z0-9]{1,3}$`)
1519 sourceRE = regexp.MustCompile(`^[A-Za-z0-9._-]{1,64}$`)
1520)
1521
1522// Adopt validates what a TRUSTED transport carried (spec §8.4). Call it only
1523// for a verified hop or a TLS tunnel. Invalid fields are dropped one by one;
1524// a hit left with neither a country nor a coordinate pair, or a missing or
1525// malformed hop value, is off.
1526func (c Captured) Adopt() Answer {
1527 state, src, ok := strings.Cut(c.Hop, ";src=")
1528 switch {
1529 case c.Hop == string(StateOff):
1530 return Off()
1531 case !ok || !sourceRE.MatchString(src):
1532 return Off()
1533 case State(state) == StateMiss:
1534 return Answer{State: StateMiss, Source: src}
1535 case State(state) != StateHit:
1536 return Off()
1537 }
1538 a := Answer{State: StateHit, Source: src}
1539 if countryRE.MatchString(c.Country) {
1540 a.Country = c.Country
1541 if subdivisionRE.MatchString(c.Subdivision) && strings.HasPrefix(c.Subdivision, c.Country+"-") {
1542 a.Subdivision = c.Subdivision
1543 }
1544 }
1545 if lat, lon, ok := parsePair(c.Lat, c.Lon); ok {
1546 a.Lat, a.Lon, a.HasPoint = lat, lon, true
1547 }
1548 if a.Country == "" && !a.HasPoint {
1549 return Off()
1550 }
1551 return a
1552}
1553
1554func parsePair(latS, lonS string) (float64, float64, bool) {
1555 lat, err1 := strconv.ParseFloat(latS, 64)
1556 lon, err2 := strconv.ParseFloat(lonS, 64)
1557 if err1 != nil || err2 != nil || math.IsNaN(lat) || math.IsNaN(lon) || math.IsInf(lat, 0) || math.IsInf(lon, 0) ||
1558 lat < -90 || lat > 90 || lon < -180 || lon > 180 {
1559 return 0, 0, false
1560 }
1561 return round2(lat), round2(lon), true
1562}
1563```
1564
1565- [ ] **Step 4: Run the package's tests**
1566
1567Run: `nice -n 19 go test ./internal/geoip -count=1`
1568Expected: PASS.
1569
1570- [ ] **Step 5: Commit**
1571
1572```bash
1573git add internal/geoip/
1574git commit -m "geoip: pointer, D1 expiry, attribution and the header vocabulary (geoip spec §3, §7, §8.1a)"
1575```
1576
1577---
1578
1579### Task 6: The console's publish path
1580
1581**Files:**
1582- Create: `internal/console/geoippublish.go`
1583- Test: `internal/console/geoippublish_test.go`
1584
1585**Interfaces:**
1586- Consumes:
1587 - `geo.OpenMMDB`, `(*geo.MMDB).Lookup/Source/BuildEpoch/NodeCount/Close` (Task 2);
1588 - `release.PutObjectFrom`, `release.OpenObject` (Task 4);
1589 - `geoip.Pointer/ParsePointer/BlobKey/PointerKey/Prefix/MaxBlobBytes/Expired/IsSHA256Hex` (Task 5).
1590- Produces:
1591 ```go
1592 type geoipPublisher struct{ ... }
1593 func newGeoipPublisher(store release.Store, tmpRoot string) *geoipPublisher
1594 type geoipPublishOpts struct{ Origin, Edition string; Force bool }
1595 var errGeoipBusy, errGeoipRefused // errors.Is-able; refusals wrap errGeoipRefused with a category
1596 func (p *geoipPublisher) Publish(ctx context.Context, src io.Reader, o geoipPublishOpts) (geoip.Pointer, error)
1597 func (p *geoipPublisher) Off(ctx context.Context) error
1598 func (p *geoipPublisher) Sweep(ctx context.Context) error
1599 func (p *geoipPublisher) ExpireIfDue(ctx context.Context, now time.Time) (bool, error)
1600 func (p *geoipPublisher) Current(ctx context.Context) (geoip.Pointer, string, bool, error) // pointer, etag, present
1601 var geoipProbeAddrs = []string{"8.8.8.8","1.1.1.1","9.9.9.9","208.67.222.222","2001:4860:4860::8888"}
1602 ```
1603
1604- [ ] **Step 1: Write the failing tests**
1605
1606```go
1607package console
1608
1609import (
1610 "bytes"
1611 "context"
1612 "errors"
1613 "os"
1614 "strings"
1615 "sync"
1616 "testing"
1617 "time"
1618
1619 "github.com/carlosframework/platform/internal/geoip"
1620 "github.com/carlosframework/platform/internal/release"
1621)
1622
1623// The repo's fixtures hold none of the production probe addresses
1624// (FACTS.md), so tests swap in addresses the fixtures do resolve.
1625func withFixtureProbes(t *testing.T) {
1626 prev := geoipProbeAddrs
1627 geoipProbeAddrs = []string{"81.2.69.142", "216.160.83.56", "89.160.20.112", "2001:218::1", "67.43.156.1"}
1628 t.Cleanup(func() { geoipProbeAddrs = prev })
1629}
1630
1631func fixture(t *testing.T, name string) []byte {
1632 b, err := os.ReadFile("../geo/testdata/" + name)
1633 if err != nil {
1634 t.Fatal(err)
1635 }
1636 return b
1637}
1638
1639func newTestPublisher(t *testing.T) (*geoipPublisher, release.Store) {
1640 s := release.NewDirStore(t.TempDir())
1641 return newGeoipPublisher(s, t.TempDir()), s
1642}
1643
1644func blobKeys(t *testing.T, s release.Store) []string {
1645 keys, err := s.List(context.Background(), geoip.Prefix)
1646 if err != nil {
1647 t.Fatal(err)
1648 }
1649 var out []string
1650 for _, k := range keys {
1651 if strings.HasSuffix(k, ".mmdb") {
1652 out = append(out, k)
1653 }
1654 }
1655 return out
1656}
1657
1658func TestPublishWritesBlobThenPointer(t *testing.T) {
1659 withFixtureProbes(t)
1660 p, s := newTestPublisher(t)
1661 ptr, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
1662 if err != nil {
1663 t.Fatal(err)
1664 }
1665 if ptr.DatabaseType != "GeoIP2-City" || ptr.NodeCount != 1547 || ptr.Origin != "manual" || ptr.Key != geoip.BlobKey(ptr.SHA256) {
1666 t.Fatalf("pointer = %+v", ptr)
1667 }
1668 raw, _, err := s.Get(context.Background(), geoip.PointerKey)
1669 if err != nil {
1670 t.Fatal(err)
1671 }
1672 if got, err := geoip.ParsePointer(raw); err != nil || got.SHA256 != ptr.SHA256 {
1673 t.Fatalf("stored pointer = (%+v, %v)", got, err)
1674 }
1675 if keys := blobKeys(t, s); len(keys) != 1 || keys[0] != ptr.Key {
1676 t.Fatalf("blobs = %v", keys)
1677 }
1678}
1679
1680func TestPublishRefusals(t *testing.T) {
1681 withFixtureProbes(t)
1682 ctx := context.Background()
1683 p, _ := newTestPublisher(t)
1684 for name, body := range map[string][]byte{
1685 "not a database": []byte("junk"),
1686 "no country data": fixture(t, "MaxMind-DB-test-ipv4-24.mmdb"),
1687 } {
1688 if _, err := p.Publish(ctx, bytes.NewReader(body), geoipPublishOpts{Origin: "manual"}); !errors.Is(err, errGeoipRefused) {
1689 t.Errorf("%s: err = %v, want errGeoipRefused", name, err)
1690 }
1691 }
1692}
1693
1694func TestPublishShrinkCheckOnlyForTheSameType(t *testing.T) {
1695 withFixtureProbes(t)
1696 ctx := context.Background()
1697 p, s := newTestPublisher(t)
1698 if _, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-Country-Test.mmdb")), geoipPublishOpts{Origin: "manual"}); err != nil {
1699 t.Fatal(err)
1700 }
1701 // Inflate the recorded node_count so a same-type republish looks like a
1702 // truncation (fewer than half).
1703 raw, etag, _ := s.Get(ctx, geoip.PointerKey)
1704 inflated := bytes.Replace(raw, []byte(`"node_count": 1704`), []byte(`"node_count": 99999`), 1)
1705 if bytes.Equal(inflated, raw) {
1706 t.Fatalf("pointer JSON shape changed; adjust the test: %s", raw)
1707 }
1708 if _, err := s.Put(ctx, geoip.PointerKey, inflated, etag); err != nil {
1709 t.Fatal(err)
1710 }
1711 same := fixture(t, "GeoIP2-Country-Test.mmdb")
1712 if _, err := p.Publish(ctx, bytes.NewReader(same), geoipPublishOpts{Origin: "manual"}); !errors.Is(err, errGeoipRefused) {
1713 t.Fatalf("same-type shrink = %v, want refused", err)
1714 }
1715 if _, err := p.Publish(ctx, bytes.NewReader(same), geoipPublishOpts{Origin: "manual", Force: true}); err != nil {
1716 t.Fatalf("forced = %v", err)
1717 }
1718 // A different type skips the check entirely.
1719 if _, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"}); err != nil {
1720 t.Fatalf("different type = %v", err)
1721 }
1722}
1723
1724func TestPublishSameFileTwiceAndXYXKeepsTheCurrentBlob(t *testing.T) {
1725 withFixtureProbes(t)
1726 ctx := context.Background()
1727 p, s := newTestPublisher(t)
1728 x, y := fixture(t, "GeoIP2-City-Test.mmdb"), fixture(t, "GeoIP2-Country-Test.mmdb")
1729 for i, body := range [][]byte{x, x, y, x} {
1730 ptr, err := p.Publish(ctx, bytes.NewReader(body), geoipPublishOpts{Origin: "manual", Force: true})
1731 if err != nil {
1732 t.Fatalf("publish %d: %v", i, err)
1733 }
1734 keys := blobKeys(t, s)
1735 if len(keys) != 1 || keys[0] != ptr.Key {
1736 t.Fatalf("after publish %d blobs = %v, want only %s", i, keys, ptr.Key)
1737 }
1738 }
1739}
1740
1741func TestPublishIsOneAtATime(t *testing.T) {
1742 withFixtureProbes(t)
1743 p, _ := newTestPublisher(t)
1744 p.mu.Lock() // a publish in progress
1745 _, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
1746 p.mu.Unlock()
1747 if !errors.Is(err, errGeoipBusy) {
1748 t.Fatalf("concurrent publish = %v, want errGeoipBusy", err)
1749 }
1750}
1751
1752func TestPublishRefusesOversizeWhileCopying(t *testing.T) {
1753 withFixtureProbes(t)
1754 p, _ := newTestPublisher(t)
1755 p.maxBytes = 1024
1756 _, err := p.Publish(context.Background(), bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
1757 if !errors.Is(err, errGeoipRefused) {
1758 t.Fatalf("oversize = %v", err)
1759 }
1760 if entries, _ := os.ReadDir(p.tmpRoot); len(entries) != 0 {
1761 t.Fatalf("temp left behind: %v", entries)
1762 }
1763}
1764
1765func TestOffAndSweep(t *testing.T) {
1766 withFixtureProbes(t)
1767 ctx := context.Background()
1768 p, s := newTestPublisher(t)
1769 ptr, err := p.Publish(ctx, bytes.NewReader(fixture(t, "GeoIP2-City-Test.mmdb")), geoipPublishOpts{Origin: "manual"})
1770 if err != nil {
1771 t.Fatal(err)
1772 }
1773 // An orphan from a crashed publish (spec §3 orphan sweep).
1774 orphan := geoip.BlobKey(strings.Repeat("a", 64))
1775 if _, err := s.Put(ctx, orphan, []byte("x"), ""); err != nil {
1776 t.Fatal(err)
1777 }
1778 if err := p.Sweep(ctx); err != nil {
1779 t.Fatal(err)
1780 }
1781 if keys := blobKeys(t, s); len(keys) != 1 || keys[0] != ptr.Key {
1782 t.Fatalf("after sweep = %v", keys)
1783 }
1784 if err := p.Off(ctx); err != nil {
1785 t.Fatal(err)
1786 }
1787 if _, _, err := s.Get(ctx, geoip.PointerKey); !errors.Is(err, os.ErrNotExist) {
1788 t.Fatalf("pointer after off: %v", err)
1789 }
1790 if keys := blobKeys(t, s); len(keys) != 0 {
1791 t.Fatalf("blobs after off = %v", keys)
1792 }
1793}
1794
1795func TestExpireIfDueMarksGeoLiteExpired(t *testing.T) {
1796 ctx := context.Background()
1797 p, s := newTestPublisher(t)
1798 sha := strings.Repeat("b", 64)
1799 ptr := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: 1, DatabaseType: "GeoLite2-City",
1800 BuildEpoch: time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC).Unix(), Origin: "manual"}
1801 putPointer(t, s, ptr)
1802 if _, err := s.Put(ctx, ptr.Key, []byte("x"), ""); err != nil {
1803 t.Fatal(err)
1804 }
1805 if did, err := p.ExpireIfDue(ctx, time.Date(2026, 9, 20, 0, 0, 0, 0, time.UTC)); did || err != nil {
1806 t.Fatalf("early = (%v, %v)", did, err)
1807 }
1808 if did, err := p.ExpireIfDue(ctx, time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)); !did || err != nil {
1809 t.Fatalf("due = (%v, %v)", did, err)
1810 }
1811 got, _, present, err := p.Current(ctx)
1812 if err != nil || !present || !got.Expired {
1813 t.Fatalf("pointer = (%+v, %v, %v)", got, present, err)
1814 }
1815 if keys := blobKeys(t, s); len(keys) != 0 {
1816 t.Fatalf("blob not destroyed: %v", keys)
1817 }
1818}
1819
1820func putPointer(t *testing.T, s release.Store, p geoip.Pointer) {
1821 t.Helper()
1822 b, err := marshalPointer(p)
1823 if err != nil {
1824 t.Fatal(err)
1825 }
1826 if _, err := s.Put(context.Background(), geoip.PointerKey, b, ""); err != nil {
1827 t.Fatal(err)
1828 }
1829}
1830
1831var _ = sync.Mutex{}
1832```
1833
1834- [ ] **Step 2: Run them to see them fail**
1835
1836Run: `nice -n 19 go test ./internal/console -run 'Publish|OffAndSweep|ExpireIfDue' -count=1`
1837Expected: build failure, because the publisher is undefined.
1838
1839- [ ] **Step 3: Implement it.** `internal/console/geoippublish.go`:
1840
1841```go
1842package console
1843
1844import (
1845 "context"
1846 "crypto/sha256"
1847 "encoding/hex"
1848 "encoding/json"
1849 "errors"
1850 "fmt"
1851 "io"
1852 "log"
1853 "net"
1854 "os"
1855 "strings"
1856 "sync"
1857 "time"
1858
1859 "github.com/carlosframework/platform/internal/geo"
1860 "github.com/carlosframework/platform/internal/geoip"
1861 "github.com/carlosframework/platform/internal/release"
1862)
1863
1864// geoipProbeAddrs is the country probe (spec §4.1 step 4): at least three
1865// must answer with a country. A variable so tests can use addresses the
1866// MaxMind fixtures hold (FACTS.md).
1867var geoipProbeAddrs = []string{"8.8.8.8", "1.1.1.1", "9.9.9.9", "208.67.222.222", "2001:4860:4860::8888"}
1868
1869const geoipProbeNeed = 3
1870
1871var (
1872 errGeoipBusy = errors.New("a geoip publish is already in progress")
1873 errGeoipRefused = errors.New("geoip database refused")
1874)
1875
1876func refused(category string) error { return fmt.Errorf("%w: %s", errGeoipRefused, category) }
1877
1878// geoipPublisher is the console's single writer of control/geoip/ (spec §3).
1879// mu is held through cleanup by every publish, off, sweep and expiry, and a
1880// publish that finds it held answers errGeoipBusy rather than queueing.
1881type geoipPublisher struct {
1882 mu sync.Mutex
1883 store release.Store
1884 tmpRoot string
1885 maxBytes int64
1886 deadline time.Duration
1887 now func() time.Time
1888}
1889
1890type geoipPublishOpts struct {
1891 Origin, Edition string
1892 Force bool
1893}
1894
1895func newGeoipPublisher(store release.Store, tmpRoot string) *geoipPublisher {
1896 return &geoipPublisher{store: store, tmpRoot: tmpRoot, maxBytes: geoip.MaxBlobBytes,
1897 deadline: 15 * time.Minute, now: time.Now}
1898}
1899
1900func marshalPointer(p geoip.Pointer) ([]byte, error) { return json.MarshalIndent(p, "", " ") }
1901
1902// Current reads the pointer. present=false means no pointer (feature off).
1903func (p *geoipPublisher) Current(ctx context.Context) (geoip.Pointer, string, bool, error) {
1904 raw, etag, err := p.store.Get(ctx, geoip.PointerKey)
1905 if errors.Is(err, os.ErrNotExist) {
1906 return geoip.Pointer{}, "", false, nil
1907 }
1908 if err != nil {
1909 return geoip.Pointer{}, "", false, err
1910 }
1911 ptr, err := geoip.ParsePointer(raw)
1912 if err != nil {
1913 return geoip.Pointer{}, etag, true, err
1914 }
1915 return ptr, etag, true, nil
1916}
1917
1918// Publish streams src to a temp file, verifies it, and publishes it.
1919func (p *geoipPublisher) Publish(ctx context.Context, src io.Reader, o geoipPublishOpts) (geoip.Pointer, error) {
1920 if !p.mu.TryLock() {
1921 return geoip.Pointer{}, errGeoipBusy
1922 }
1923 defer p.mu.Unlock()
1924 ctx, cancel := context.WithTimeout(ctx, p.deadline)
1925 defer cancel()
1926
1927 dir, err := os.MkdirTemp(p.tmpRoot, "geoip-publish-*")
1928 if err != nil {
1929 return geoip.Pointer{}, err
1930 }
1931 defer os.RemoveAll(dir)
1932 path := dir + "/db.mmdb"
1933 sha, size, err := copyCapped(ctx, path, src, p.maxBytes)
1934 if err != nil {
1935 return geoip.Pointer{}, err
1936 }
1937 db, err := geo.OpenMMDB(path)
1938 if err != nil {
1939 return geoip.Pointer{}, refused("not a valid MaxMind-format database")
1940 }
1941 defer db.Close()
1942 if !hasCountryData(db) {
1943 return geoip.Pointer{}, refused("the database has no country data")
1944 }
1945 cur, etag, present, err := p.Current(ctx)
1946 if err != nil && present {
1947 log.Printf("console: geoip publish: the current pointer is unreadable and will be replaced")
1948 } else if err != nil {
1949 return geoip.Pointer{}, err
1950 }
1951 if present && err == nil && !o.Force && cur.DatabaseType == db.Source() && db.NodeCount() < cur.NodeCount/2 {
1952 return geoip.Pointer{}, refused("fewer than half the current database's nodes (use -force if intended)")
1953 }
1954 next := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: size,
1955 DatabaseType: db.Source(), BuildEpoch: db.BuildEpoch().Unix(), NodeCount: db.NodeCount(),
1956 Published: p.now().UTC().Truncate(time.Second), Origin: o.Origin, Edition: o.Edition}
1957 f, err := os.Open(path)
1958 if err != nil {
1959 return geoip.Pointer{}, err
1960 }
1961 defer f.Close()
1962 if _, err := release.PutObjectFrom(ctx, p.store, next.Key, f, size, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
1963 return geoip.Pointer{}, err
1964 }
1965 // A CAS conflict on the blob means a blob with this exact name (so
1966 // these exact bytes) already exists: carry on (spec §3 step 2).
1967 body, err := marshalPointer(next)
1968 if err != nil {
1969 return geoip.Pointer{}, err
1970 }
1971 if !present {
1972 etag = ""
1973 }
1974 if _, err := p.store.Put(ctx, geoip.PointerKey, body, etag); err != nil {
1975 return geoip.Pointer{}, fmt.Errorf("pointer write: %w", err)
1976 }
1977 if err := p.sweepLocked(ctx, next.Key); err != nil {
1978 log.Printf("console: geoip sweep after publish: %v", err)
1979 }
1980 log.Printf("console: geoip published %s (%s, %s)", sha[:8], next.DatabaseType, next.Origin)
1981 return next, nil
1982}
1983
1984// copyCapped streams src into path, hashing, refusing more than max bytes
1985// as it copies (codex R1-7).
1986func copyCapped(ctx context.Context, path string, src io.Reader, max int64) (string, int64, error) {
1987 f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
1988 if err != nil {
1989 return "", 0, err
1990 }
1991 defer f.Close()
1992 h := sha256.New()
1993 n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(ctxReader{ctx, src}, max+1))
1994 if err != nil {
1995 return "", 0, err
1996 }
1997 if n > max {
1998 return "", 0, refused(fmt.Sprintf("larger than %d bytes", max))
1999 }
2000 if n == 0 {
2001 return "", 0, refused("empty upload")
2002 }
2003 if err := f.Sync(); err != nil {
2004 return "", 0, err
2005 }
2006 return hex.EncodeToString(h.Sum(nil)), n, nil
2007}
2008
2009type ctxReader struct {
2010 ctx context.Context
2011 r io.Reader
2012}
2013
2014func (c ctxReader) Read(b []byte) (int, error) {
2015 if err := c.ctx.Err(); err != nil {
2016 return 0, err
2017 }
2018 return c.r.Read(b)
2019}
2020
2021func hasCountryData(db *geo.MMDB) bool {
2022 n := 0
2023 for _, a := range geoipProbeAddrs {
2024 if r, ok := db.Lookup(net.ParseIP(a)); ok && r.Country != "" {
2025 n++
2026 }
2027 }
2028 return n >= geoipProbeNeed
2029}
2030
2031// Off deletes the pointer, then every blob (spec §3). Unconditional delete:
2032// safe under the single-writer mutex (codex R2-6).
2033func (p *geoipPublisher) Off(ctx context.Context) error {
2034 p.mu.Lock()
2035 defer p.mu.Unlock()
2036 if err := p.store.Delete(ctx, geoip.PointerKey); err != nil && !errors.Is(err, os.ErrNotExist) {
2037 return err
2038 }
2039 return p.sweepLocked(ctx, "")
2040}
2041
2042// Sweep deletes every blob the pointer does not name (orphans from a crash
2043// between blob upload and pointer write). Run at startup and daily.
2044func (p *geoipPublisher) Sweep(ctx context.Context) error {
2045 p.mu.Lock()
2046 defer p.mu.Unlock()
2047 cur, _, present, err := p.Current(ctx)
2048 if err != nil {
2049 return err
2050 }
2051 keep := ""
2052 if present && !cur.Expired {
2053 keep = cur.Key
2054 }
2055 return p.sweepLocked(ctx, keep)
2056}
2057
2058func (p *geoipPublisher) sweepLocked(ctx context.Context, keep string) error {
2059 keys, err := p.store.List(ctx, geoip.Prefix)
2060 if err != nil {
2061 return err
2062 }
2063 var firstErr error
2064 for _, k := range keys {
2065 if !strings.HasSuffix(k, ".mmdb") || k == keep {
2066 continue
2067 }
2068 if err := p.store.Delete(ctx, k); err != nil && !errors.Is(err, os.ErrNotExist) && firstErr == nil {
2069 firstErr = err
2070 }
2071 }
2072 return firstErr
2073}
2074
2075// ExpireIfDue applies D1 to the bucket: a GeoLite2 pointer past expiry is
2076// rewritten with expired=true and its blob destroyed.
2077func (p *geoipPublisher) ExpireIfDue(ctx context.Context, now time.Time) (bool, error) {
2078 p.mu.Lock()
2079 defer p.mu.Unlock()
2080 cur, etag, present, err := p.Current(ctx)
2081 if err != nil || !present || cur.Expired || !geoip.Expired(cur.DatabaseType, cur.Build(), now) {
2082 return false, err
2083 }
2084 cur.Expired = true
2085 body, err := marshalPointer(cur)
2086 if err != nil {
2087 return false, err
2088 }
2089 if _, err := p.store.Put(ctx, geoip.PointerKey, body, etag); err != nil {
2090 return false, err
2091 }
2092 log.Printf("console: geoip database %s expired (GeoLite2, 30 days after build) and was removed", cur.SHA256[:8])
2093 return true, p.sweepLocked(ctx, "")
2094}
2095```
2096
2097`json.MarshalIndent` writes `"node_count": 1704` with a space, which is what the shrink test edits. If you change the marshaller, update that test.
2098
2099- [ ] **Step 4: Run the tests**
2100
2101Run: `nice -n 19 go test ./internal/console -run 'Publish|OffAndSweep|ExpireIfDue' -count=1`
2102Expected: PASS. If the console package's other tests need a socket TMPDIR, run them later in Task 12 outside the sandbox. These ones don't bind sockets.
2103
2104- [ ] **Step 5: Commit**
2105
2106```bash
2107git add internal/console/geoippublish.go internal/console/geoippublish_test.go
2108git commit -m "console: geoip publish path (verify, country probe, shrink check, blob+pointer, sweep, off, D1 expiry) (geoip spec §3, §4.1)"
2109```
2110
2111---
2112
2113### Task 7: The edge mirror
2114
2115**Files:**
2116- Create: `internal/edge/geoipmirror.go`
2117- Test: `internal/edge/geoipmirror_test.go`
2118- Modify: `internal/edge/edge.go`:
2119 - the `Edge` struct (new fields next to `analyticsMeter`, ~:158);
2120 - `RunWithActivatorReady`: arm the mirror after the home map (~:3088), and start the loop next to `go mirror.run` (~:3171).
2121
2122**Interfaces:**
2123- Consumes: `geo.OpenMMDBTrusted`, `geo.Swappable` (Tasks 2 and 3); `geoip.*` (Task 5); `release.OpenObject`.
2124- Produces:
2125 ```go
2126 // in package edge
2127 type GeoIPState struct{ Mode, SHA256, Type, Error string } // Mode: "mirror"|"tunnel"|"none"
2128 func (e *Edge) GeoIPState() GeoIPState
2129 func WithGeoTunnelClient() RunOption // agent passes it when CARLOS_TUNNEL_* are all set
2130 // fields: e.geo *geo.Swappable (always non-nil after New), e.geoMirror *geoMirror (nil unless mode mirror), e.geoMode string
2131 ```
2132
2133- [ ] **Step 1: Write the failing tests.** They drive a `geoMirror` directly over a `DirStore` and a temp data dir. Publish with the real Task 6 shape: write the blob and pointer by hand.
2134
2135```go
2136package edge
2137
2138import (
2139 "context"
2140 "crypto/sha256"
2141 "encoding/hex"
2142 "encoding/json"
2143 "net"
2144 "os"
2145 "path/filepath"
2146 "testing"
2147 "time"
2148
2149 "github.com/carlosframework/platform/internal/geo"
2150 "github.com/carlosframework/platform/internal/geoip"
2151 "github.com/carlosframework/platform/internal/release"
2152)
2153
2154func geoFixture(t *testing.T, name string) []byte {
2155 b, err := os.ReadFile(filepath.Join("..", "geo", "testdata", name))
2156 if err != nil {
2157 t.Fatal(err)
2158 }
2159 return b
2160}
2161
2162// publishFixture writes blob + pointer the way the console does.
2163func publishFixture(t *testing.T, s release.Store, body []byte, dbType string, build time.Time) geoip.Pointer {
2164 t.Helper()
2165 sum := sha256.Sum256(body)
2166 sha := hex.EncodeToString(sum[:])
2167 ctx := context.Background()
2168 if _, err := s.Put(ctx, geoip.BlobKey(sha), body, ""); err != nil {
2169 t.Fatal(err)
2170 }
2171 p := geoip.Pointer{V: 1, SHA256: sha, Key: geoip.BlobKey(sha), Size: int64(len(body)),
2172 DatabaseType: dbType, BuildEpoch: build.Unix(), Origin: "manual"}
2173 raw, _ := json.Marshal(p)
2174 _, etag, _ := s.Get(ctx, geoip.PointerKey)
2175 if _, err := s.Put(ctx, geoip.PointerKey, raw, etag); err != nil {
2176 t.Fatal(err)
2177 }
2178 return p
2179}
2180
2181func newTestMirror(t *testing.T) (*geoMirror, release.Store, string) {
2182 s := release.NewDirStore(t.TempDir())
2183 dir := filepath.Join(t.TempDir(), "carlos", "geoip")
2184 return newGeoMirror(s, dir, geo.NewSwappable()), s, dir
2185}
2186
2187func lookupGB(m *geoMirror) bool {
2188 snap := m.sw.Lookup(net.ParseIP("81.2.69.142"))
2189 return snap.Found && snap.Result.Country == "GB"
2190}
2191
2192func TestMirrorInstallsAPublishedDatabase(t *testing.T) {
2193 m, s, dir := newTestMirror(t)
2194 p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
2195 if err := m.Poll(context.Background()); err != nil {
2196 t.Fatal(err)
2197 }
2198 if !lookupGB(m) {
2199 t.Fatal("not serving after poll")
2200 }
2201 if _, err := os.Stat(filepath.Join(dir, p.SHA256+".mmdb")); err != nil {
2202 t.Fatal(err)
2203 }
2204 if st := m.State(); st.SHA256 != p.SHA256 || st.Type != "GeoIP2-City" {
2205 t.Fatalf("state = %+v", st)
2206 }
2207 fi, _ := os.Stat(dir)
2208 if fi.Mode().Perm() != 0o755 {
2209 t.Fatalf("dir mode = %v, want 0755 (the console reads it)", fi.Mode().Perm())
2210 }
2211}
2212
2213func TestMirrorBootsFromTheLocalGenerationWithoutTheBucket(t *testing.T) {
2214 m, s, dir := newTestMirror(t)
2215 publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
2216 if err := m.Poll(context.Background()); err != nil {
2217 t.Fatal(err)
2218 }
2219 // A fresh process, a dead bucket.
2220 m2 := newGeoMirror(release.NewDirStore(filepath.Join(t.TempDir(), "empty")), dir, geo.NewSwappable())
2221 m2.Boot()
2222 if !lookupGB(m2) {
2223 t.Fatal("boot did not serve the local generation")
2224 }
2225}
2226
2227func TestMirrorBootWithCurrentNamingAMissingFile(t *testing.T) {
2228 m, _, dir := newTestMirror(t)
2229 os.MkdirAll(dir, 0o755)
2230 os.WriteFile(filepath.Join(dir, "current"), []byte(`{"sha256":"`+string(make([]byte, 0))+`deadbeef","database_type":"X","build_epoch":1}`), 0o644)
2231 m.Boot()
2232 if snap := m.sw.Lookup(net.ParseIP("81.2.69.142")); snap.Loaded {
2233 t.Fatal("loaded something from a dangling current")
2234 }
2235 if _, err := os.Stat(filepath.Join(dir, "current")); !os.IsNotExist(err) {
2236 t.Fatal("dangling current was not removed")
2237 }
2238}
2239
2240func TestMirrorRejectsADigestMismatchOnce(t *testing.T) {
2241 m, s, _ := newTestMirror(t)
2242 p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
2243 // Corrupt the blob after publishing.
2244 os.WriteFile(filepath.Join(s.(*release.DirStore).Root(), p.Key), append(geoFixture(t, "GeoIP2-City-Test.mmdb")[:p.Size-1], 'x'), 0o644)
2245 if err := m.Poll(context.Background()); err == nil {
2246 t.Fatal("a corrupt blob installed")
2247 }
2248 if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
2249 t.Fatal("serving a corrupt blob")
2250 }
2251 if err := m.Poll(context.Background()); err != nil {
2252 t.Fatalf("second poll of the same rejected sha should be a quiet no-op, got %v", err)
2253 }
2254}
2255
2256func TestMirrorOffClearsEverything(t *testing.T) {
2257 m, s, dir := newTestMirror(t)
2258 publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
2259 m.Poll(context.Background())
2260 s.Delete(context.Background(), geoip.PointerKey)
2261 if err := m.Poll(context.Background()); err != nil {
2262 t.Fatal(err)
2263 }
2264 if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
2265 t.Fatal("still serving after off")
2266 }
2267 if matches, _ := filepath.Glob(filepath.Join(dir, "*.mmdb")); len(matches) != 0 {
2268 t.Fatalf("generations left: %v", matches)
2269 }
2270}
2271
2272func TestMirrorNeverInstallsAnExpiredGeoLite(t *testing.T) {
2273 m, s, _ := newTestMirror(t)
2274 now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
2275 m.now = func() time.Time { return now }
2276 // The fixture's own metadata says GeoIP2-City (no expiry), so make the
2277 // POINTER claim GeoLite2 with an old build: expired before download.
2278 publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoLite2-City", now.Add(-31*24*time.Hour))
2279 if err := m.Poll(context.Background()); err != nil {
2280 t.Fatal(err)
2281 }
2282 if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
2283 t.Fatal("installed an expired pointer")
2284 }
2285 // Repeated polls of the same unmarked expired pointer stay quiet.
2286 if err := m.Poll(context.Background()); err != nil {
2287 t.Fatal(err)
2288 }
2289}
2290
2291func TestMirrorExpiryTickRemovesOnlyTheCurrentExpiredGeneration(t *testing.T) {
2292 m, s, _ := newTestMirror(t)
2293 now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
2294 m.now = func() time.Time { return now }
2295 publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", now)
2296 m.Poll(context.Background())
2297 // Pretend what is loaded is a GeoLite2 built 31 days ago.
2298 m.genMu.Lock()
2299 m.loadedType, m.loadedBuild = "GeoLite2-City", now.Add(-31*24*time.Hour)
2300 m.genMu.Unlock()
2301 m.ExpireTick()
2302 if m.sw.Lookup(net.ParseIP("81.2.69.142")).Loaded {
2303 t.Fatal("expired generation still serving")
2304 }
2305 // A fresh, non-expiring install afterwards is untouched by later ticks.
2306 publishFixture(t, s, geoFixture(t, "GeoIP2-Country-Test.mmdb"), "GeoIP2-Country", now)
2307 m.Poll(context.Background())
2308 m.ExpireTick()
2309 if !lookupGB(m) {
2310 t.Fatal("a later tick removed a fresh generation")
2311 }
2312}
2313
2314func TestMirrorRefusesAnOversizeBlob(t *testing.T) {
2315 m, s, dir := newTestMirror(t)
2316 p := publishFixture(t, s, geoFixture(t, "GeoIP2-City-Test.mmdb"), "GeoIP2-City", time.Now())
2317 m.maxBytes = p.Size - 1
2318 if err := m.Poll(context.Background()); err == nil {
2319 t.Fatal("oversize installed")
2320 }
2321 if matches, _ := filepath.Glob(filepath.Join(dir, ".incoming-*")); len(matches) != 0 {
2322 t.Fatalf("incoming left: %v", matches)
2323 }
2324}
2325```
2326
2327The test calls `(*release.DirStore).Root()`. If DirStore has no such accessor, add one in this task (`func (d *DirStore) Root() string { return d.root }`, using whatever its field is called), or have the test keep the directory it passed to `NewDirStore` and corrupt `filepath.Join(storeDir, p.Key)` instead. Prefer the second: it doesn't touch `release`.
2328
2329- [ ] **Step 2: Run them to see them fail**
2330
2331Run (outside the sandbox, socket package): `mkdir -p /tmp/g && TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run Mirror -count=1`
2332Expected: build failure, because `newGeoMirror` is undefined.
2333
2334- [ ] **Step 3: Implement `internal/edge/geoipmirror.go`**
2335
2336```go
2337package edge
2338
2339import (
2340 "context"
2341 "crypto/sha256"
2342 "encoding/hex"
2343 "encoding/json"
2344 "errors"
2345 "fmt"
2346 "io"
2347 "log"
2348 "math/rand/v2"
2349 "os"
2350 "path/filepath"
2351 "strings"
2352 "sync"
2353 "time"
2354
2355 "github.com/carlosframework/platform/internal/geo"
2356 "github.com/carlosframework/platform/internal/geoip"
2357 "github.com/carlosframework/platform/internal/release"
2358)
2359
2360// The GeoIP mirror (geoip spec §5): poll control/geoip/current.json, fetch
2361// the content-addressed blob it names into an immutable local generation,
2362// and hot-swap the edge's reader. genMu serialises everything that chooses,
2363// expires or deletes a generation (codex R3-1); network I/O stays outside it.
2364
2365var (
2366 geoMirrorInterval = 10 * time.Minute
2367 geoMirrorJitter = time.Minute
2368 geoMirrorDeadline = 10 * time.Minute
2369 geoExpiryInterval = time.Minute
2370)
2371
2372type geoCurrent struct {
2373 SHA256 string `json:"sha256,omitempty"`
2374 DatabaseType string `json:"database_type,omitempty"`
2375 BuildEpoch int64 `json:"build_epoch,omitempty"`
2376}
2377
2378type geoMirror struct {
2379 store release.Store
2380 dir string
2381 sw *geo.Swappable
2382 now func() time.Time
2383 maxBytes int64
2384
2385 genMu sync.Mutex
2386 loadedSHA string
2387 loadedType string
2388 loadedBuild time.Time
2389 skipSHA map[string]string // sha -> why (rejected | expired); cleared when the pointer moves on
2390 lastErr string // last poll's error category, "" when healthy
2391 failingAt time.Time
2392 lastFailLog time.Time
2393}
2394
2395func newGeoMirror(store release.Store, dir string, sw *geo.Swappable) *geoMirror {
2396 return &geoMirror{store: store, dir: dir, sw: sw, now: time.Now, maxBytes: geoip.MaxBlobBytes, skipSHA: map[string]string{}}
2397}
2398
2399// GeoIPState is the heartbeat's view of this edge's GeoIP (spec §5.5).
2400type GeoIPState struct {
2401 Mode string `json:"mode"` // mirror | tunnel | none
2402 SHA256 string `json:"sha256,omitempty"`
2403 Type string `json:"type,omitempty"`
2404 Error string `json:"error,omitempty"`
2405}
2406
2407func (m *geoMirror) State() GeoIPState {
2408 m.genMu.Lock()
2409 defer m.genMu.Unlock()
2410 return GeoIPState{Mode: "mirror", SHA256: m.loadedSHA, Type: m.loadedType, Error: m.lastErr}
2411}
2412
2413func (m *geoMirror) path(sha string) string { return filepath.Join(m.dir, sha+".mmdb") }
2414
2415// Boot installs the local generation `current` names, if any, without
2416// touching the bucket, and clears everything else (spec §5.3 "At boot").
2417func (m *geoMirror) Boot() {
2418 if err := os.MkdirAll(m.dir, 0o755); err != nil {
2419 log.Printf("edge: geoip directory unusable; GeoIP stays off until it is: %v", err)
2420 return
2421 }
2422 os.Chmod(m.dir, 0o755)
2423 m.genMu.Lock()
2424 defer m.genMu.Unlock()
2425 cur, err := m.readCurrent()
2426 keep := ""
2427 switch {
2428 case err != nil && !errors.Is(err, os.ErrNotExist):
2429 log.Printf("edge: geoip local state unreadable; starting with no database")
2430 os.Remove(filepath.Join(m.dir, "current"))
2431 case cur.SHA256 == "":
2432 case geoip.Expired(cur.DatabaseType, time.Unix(cur.BuildEpoch, 0), m.now()):
2433 log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(cur.SHA256))
2434 m.writeCurrentLocked(geoCurrent{})
2435 default:
2436 db, err := geo.OpenMMDBTrusted(m.path(cur.SHA256))
2437 if err != nil {
2438 log.Printf("edge: geoip local generation %s unopenable; starting with no database", short(cur.SHA256))
2439 os.Remove(filepath.Join(m.dir, "current"))
2440 break
2441 }
2442 m.sw.Swap(db)
2443 m.loadedSHA, m.loadedType, m.loadedBuild = cur.SHA256, db.Source(), db.BuildEpoch()
2444 keep = cur.SHA256
2445 log.Printf("edge: geoip database %s (%s, built %s) loaded", short(cur.SHA256), db.Source(), db.BuildEpoch().Format("2006-01-02"))
2446 }
2447 m.cleanLocked(keep)
2448}
2449
2450func short(sha string) string {
2451 if len(sha) > 8 {
2452 return sha[:8]
2453 }
2454 return sha
2455}
2456
2457func (m *geoMirror) readCurrent() (geoCurrent, error) {
2458 var c geoCurrent
2459 b, err := os.ReadFile(filepath.Join(m.dir, "current"))
2460 if err != nil {
2461 return c, err
2462 }
2463 if err := json.Unmarshal(b, &c); err != nil {
2464 return c, err
2465 }
2466 if c.SHA256 != "" && !geoip.IsSHA256Hex(c.SHA256) {
2467 return geoCurrent{}, errors.New("bad sha")
2468 }
2469 return c, nil
2470}
2471
2472// writeCurrentLocked replaces `current` atomically: temp, fsync, rename,
2473// fsync the directory (spec §5.2).
2474func (m *geoMirror) writeCurrentLocked(c geoCurrent) error {
2475 b, _ := json.Marshal(c)
2476 tmp, err := os.CreateTemp(m.dir, ".current-*")
2477 if err != nil {
2478 return err
2479 }
2480 defer os.Remove(tmp.Name())
2481 if _, err := tmp.Write(b); err != nil {
2482 tmp.Close()
2483 return err
2484 }
2485 if err := tmp.Sync(); err != nil {
2486 tmp.Close()
2487 return err
2488 }
2489 if err := tmp.Close(); err != nil {
2490 return err
2491 }
2492 if err := os.Chmod(tmp.Name(), 0o644); err != nil {
2493 return err
2494 }
2495 if err := os.Rename(tmp.Name(), filepath.Join(m.dir, "current")); err != nil {
2496 return err
2497 }
2498 return syncDir(m.dir)
2499}
2500
2501func syncDir(dir string) error {
2502 d, err := os.Open(dir)
2503 if err != nil {
2504 return err
2505 }
2506 defer d.Close()
2507 return d.Sync()
2508}
2509
2510// cleanLocked deletes every generation but keep, every .incoming-* and
2511// every stray .current-*.
2512func (m *geoMirror) cleanLocked(keep string) {
2513 entries, _ := os.ReadDir(m.dir)
2514 for _, e := range entries {
2515 n := e.Name()
2516 if n == "current" || (keep != "" && n == keep+".mmdb") {
2517 continue
2518 }
2519 if strings.HasSuffix(n, ".mmdb") || strings.HasPrefix(n, ".incoming-") || strings.HasPrefix(n, ".current-") {
2520 os.Remove(filepath.Join(m.dir, n))
2521 }
2522 }
2523}
2524
2525// clearLocked turns GeoIP off on this box: empty current, nil reader, no
2526// generations.
2527func (m *geoMirror) clearLocked() {
2528 if err := m.writeCurrentLocked(geoCurrent{}); err != nil {
2529 os.Remove(filepath.Join(m.dir, "current"))
2530 }
2531 m.sw.Swap(nil)
2532 m.loadedSHA, m.loadedType, m.loadedBuild = "", "", time.Time{}
2533 m.cleanLocked("")
2534}
2535
2536// Poll is one pass (spec §5.3). The caller bounds ctx by geoMirrorDeadline.
2537func (m *geoMirror) Poll(ctx context.Context) error {
2538 if err := os.MkdirAll(m.dir, 0o755); err != nil {
2539 return m.fail("local directory unusable", err)
2540 }
2541 raw, err := readCapped(ctx, m.store, geoip.PointerKey, geoip.MaxPointerBytes)
2542 if errors.Is(err, os.ErrNotExist) {
2543 m.genMu.Lock()
2544 if m.loadedSHA != "" {
2545 log.Printf("edge: geoip turned off; no database published")
2546 }
2547 m.clearLocked()
2548 m.lastErr = ""
2549 m.genMu.Unlock()
2550 return nil
2551 }
2552 if err != nil {
2553 return m.fail("pointer unreadable", err)
2554 }
2555 p, err := geoip.ParsePointer(raw)
2556 if err != nil {
2557 return m.fail("pointer invalid", err)
2558 }
2559 m.genMu.Lock()
2560 if p.Expired || geoip.Expired(p.DatabaseType, p.Build(), m.now()) {
2561 if m.skipSHA[p.SHA256] != "expired" {
2562 log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(p.SHA256))
2563 }
2564 m.skipSHA = map[string]string{p.SHA256: "expired"}
2565 m.clearLocked()
2566 m.lastErr = ""
2567 m.genMu.Unlock()
2568 return nil
2569 }
2570 if p.SHA256 == m.loadedSHA {
2571 m.lastErr = ""
2572 m.genMu.Unlock()
2573 return nil
2574 }
2575 if why, ok := m.skipSHA[p.SHA256]; ok {
2576 m.genMu.Unlock()
2577 _ = why
2578 return nil
2579 }
2580 m.skipSHA = map[string]string{} // the pointer moved on
2581 m.genMu.Unlock()
2582
2583 tmp, err := m.download(ctx, p)
2584 if err != nil {
2585 if errors.Is(err, errGeoDigest) || errors.Is(err, errGeoSize) {
2586 m.genMu.Lock()
2587 m.skipSHA[p.SHA256] = "rejected"
2588 m.genMu.Unlock()
2589 log.Printf("edge: geoip blob %s rejected: %v", short(p.SHA256), err)
2590 }
2591 return m.fail("blob unavailable", err)
2592 }
2593 return m.install(tmp, p)
2594}
2595
2596var (
2597 errGeoDigest = errors.New("digest mismatch")
2598 errGeoSize = errors.New("size mismatch")
2599)
2600
2601// download streams the blob to .incoming-<sha>, checks size and digest,
2602// fsyncs, and returns its path. Outside genMu.
2603func (m *geoMirror) download(ctx context.Context, p geoip.Pointer) (string, error) {
2604 rc, _, err := release.OpenObject(ctx, m.store, p.Key)
2605 if err != nil {
2606 return "", err
2607 }
2608 defer rc.Close()
2609 tmp := filepath.Join(m.dir, ".incoming-"+p.SHA256)
2610 f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
2611 if err != nil {
2612 return "", err
2613 }
2614 limit := p.Size
2615 if m.maxBytes < limit {
2616 limit = m.maxBytes
2617 }
2618 h := sha256.New()
2619 n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, limit+1))
2620 if err == nil {
2621 err = f.Sync()
2622 }
2623 if cerr := f.Close(); err == nil {
2624 err = cerr
2625 }
2626 switch {
2627 case err != nil:
2628 os.Remove(tmp)
2629 return "", err
2630 case n > limit || n != p.Size:
2631 // n > limit: the blob ran past the cap (the box's maxBytes may be
2632 // below what the pointer claims), however the counts compare.
2633 os.Remove(tmp)
2634 return "", errGeoSize
2635 case hex.EncodeToString(h.Sum(nil)) != p.SHA256:
2636 os.Remove(tmp)
2637 return "", errGeoDigest
2638 }
2639 return tmp, nil
2640}
2641
2642// install commits a verified download under genMu (spec §5.3 steps 3-5),
2643// re-checking expiry against the file's own metadata first.
2644func (m *geoMirror) install(tmp string, p geoip.Pointer) error {
2645 m.genMu.Lock()
2646 defer m.genMu.Unlock()
2647 final := m.path(p.SHA256)
2648 if err := os.Rename(tmp, final); err != nil {
2649 os.Remove(tmp)
2650 return m.failLocked("install failed", err)
2651 }
2652 syncDir(m.dir)
2653 db, err := geo.OpenMMDBTrusted(final)
2654 if err != nil {
2655 os.Remove(final)
2656 m.skipSHA[p.SHA256] = "rejected"
2657 log.Printf("edge: geoip blob %s rejected: unopenable", short(p.SHA256))
2658 return m.failLocked("blob unopenable", err)
2659 }
2660 if geoip.Expired(db.Source(), db.BuildEpoch(), m.now()) {
2661 db.Close()
2662 os.Remove(final)
2663 m.skipSHA[p.SHA256] = "expired"
2664 log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(p.SHA256))
2665 return nil
2666 }
2667 if err := m.writeCurrentLocked(geoCurrent{SHA256: p.SHA256, DatabaseType: db.Source(), BuildEpoch: db.BuildEpoch().Unix()}); err != nil {
2668 db.Close()
2669 os.Remove(final)
2670 return m.failLocked("install failed", err)
2671 }
2672 m.sw.Swap(db)
2673 m.loadedSHA, m.loadedType, m.loadedBuild = p.SHA256, db.Source(), db.BuildEpoch()
2674 m.cleanLocked(p.SHA256)
2675 m.lastErr, m.failingAt = "", time.Time{}
2676 log.Printf("edge: geoip database %s (%s, built %s) loaded", short(p.SHA256), db.Source(), db.BuildEpoch().Format("2006-01-02"))
2677 return nil
2678}
2679
2680// ExpireTick is D1 on the loaded generation, under genMu, so it can only
2681// ever remove the generation that is current at that moment (codex R3-1).
2682func (m *geoMirror) ExpireTick() {
2683 m.genMu.Lock()
2684 defer m.genMu.Unlock()
2685 if m.loadedSHA == "" || !geoip.Expired(m.loadedType, m.loadedBuild, m.now()) {
2686 return
2687 }
2688 log.Printf("edge: geoip database %s expired (GeoLite2, 30 days after build) and was removed", short(m.loadedSHA))
2689 m.skipSHA[m.loadedSHA] = "expired"
2690 m.clearLocked()
2691}
2692
2693func (m *geoMirror) fail(category string, err error) error {
2694 m.genMu.Lock()
2695 defer m.genMu.Unlock()
2696 return m.failLocked(category, err)
2697}
2698
2699// failLocked records the category for the heartbeat and logs at most hourly
2700// while failing. The error text is never logged: it may carry a key or
2701// bytes; the category is the message (spec §12).
2702func (m *geoMirror) failLocked(category string, err error) error {
2703 now := m.now()
2704 if m.lastErr == "" {
2705 m.failingAt = now
2706 }
2707 m.lastErr = category
2708 if now.Sub(m.lastFailLog) >= time.Hour {
2709 m.lastFailLog = now
2710 log.Printf("edge: geoip %s; keeping the loaded database", category)
2711 }
2712 return fmt.Errorf("%s: %w", category, err)
2713}
2714
2715// readCapped reads key through OpenObject, refusing more than max bytes.
2716func readCapped(ctx context.Context, s release.Store, key string, max int) ([]byte, error) {
2717 rc, _, err := release.OpenObject(ctx, s, key)
2718 if err != nil {
2719 return nil, err
2720 }
2721 defer rc.Close()
2722 b, err := io.ReadAll(io.LimitReader(rc, int64(max)+1))
2723 if err != nil {
2724 return nil, err
2725 }
2726 if len(b) > max {
2727 return nil, fmt.Errorf("%s larger than %d bytes", key, max)
2728 }
2729 return b, nil
2730}
2731
2732// run is the process-lifetime loop: a poll every interval+jitter, each
2733// bounded by geoMirrorDeadline, and an independent expiry ticker that never
2734// touches the network.
2735func (m *geoMirror) run(stop <-chan struct{}) {
2736 go func() {
2737 t := time.NewTicker(geoExpiryInterval)
2738 defer t.Stop()
2739 for {
2740 select {
2741 case <-stop:
2742 return
2743 case <-t.C:
2744 m.ExpireTick()
2745 }
2746 }
2747 }()
2748 for {
2749 ctx, cancel := context.WithTimeout(context.Background(), geoMirrorDeadline)
2750 m.Poll(ctx)
2751 cancel()
2752 select {
2753 case <-stop:
2754 return
2755 case <-time.After(geoMirrorInterval + rand.N(geoMirrorJitter)):
2756 }
2757 }
2758}
2759```
2760
2761Then wire it into `Edge`:
2762
2763```go
2764// Edge struct, beside analyticsMeter:
2765 // geo is the GeoIP reader every request's lookup goes through (geoip
2766 // spec §5.4); never nil after New, empty until a database is installed.
2767 // geoMirror fills it when this edge has a deployment store and is not a
2768 // tunnel client; geoMode is what the heartbeat reports.
2769 geo *geo.Swappable
2770 geoMirror *geoMirror
2771 geoMode string
2772 geoTunnelClient bool
2773```
2774
2775In `New(...)`, initialise `e.geo = geo.NewSwappable()` and `e.geoLookup = e.geo.Lookup`. Add the field `geoLookup func(net.IP) geo.Snapshot` beside `geo`, commented `// the lookup decideGeo calls; e.geo.Lookup in production, a spy in tests (loopback test traffic is never Locatable)`. Add the option and the accessor:
2776
2777```go
2778// WithGeoTunnelClient marks this edge as a tunnel client's: it never mirrors
2779// the GeoIP database and takes the hub's headers instead (spec §5.1).
2780func WithGeoTunnelClient() RunOption { return func(e *Edge) { e.geoTunnelClient = true } }
2781
2782// GeoIPState is the heartbeat block (spec §5.5).
2783func (e *Edge) GeoIPState() GeoIPState {
2784 if e.geoMirror != nil {
2785 return e.geoMirror.State()
2786 }
2787 if e.geoMode == "" {
2788 return GeoIPState{Mode: "none"}
2789 }
2790 return GeoIPState{Mode: e.geoMode}
2791}
2792```
2793
2794In `RunWithActivatorReady`, right after the home-map block (~:3088):
2795
2796```go
2797 // GeoIP (geoip spec §5): a tunnel client takes the hub's headers and
2798 // mirrors nothing; an edge with the deployment store mirrors; anything
2799 // else has no database and only strips.
2800 switch {
2801 case e.geoTunnelClient:
2802 e.geoMode = "tunnel"
2803 case e.staticStore != nil:
2804 e.geoMode = "mirror"
2805 e.geoMirror = newGeoMirror(e.staticStore, filepath.Join(dataDir, "carlos", "geoip"), e.geo)
2806 e.geoMirror.Boot()
2807 default:
2808 e.geoMode = "none"
2809 }
2810```
2811
2812And next to `go mirror.run(...)` (~:3171):
2813
2814```go
2815 if e.geoMirror != nil {
2816 go e.geoMirror.run(make(chan struct{})) // process-lifetime, like the cert mirror
2817 }
2818```
2819
2820`dataDir` is the existing `envOr("CARLOS_DATA_DIR", "/data")` at ~:2948. Use the existing variable; don't re-read the env.
2821
2822- [ ] **Step 4: Run the tests**
2823
2824Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Mirror' -race -count=1`
2825Expected: PASS. Then run the whole edge package once: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -count=1`. Expected: PASS.
2826
2827- [ ] **Step 5: Commit**
2828
2829```bash
2830git add internal/edge/geoipmirror.go internal/edge/geoipmirror_test.go internal/edge/edge.go
2831git commit -m "edge: GeoIP mirror with immutable generations, genMu, D1 expiry ticker (geoip spec §5)"
2832```
2833
2834---
2835
2836### Task 8: The heartbeat block
2837
2838**Files:**
2839- Modify:
2840 - `internal/status/snapshot.go` (`Probes` ~:101; `BoxMetrics` ~:282; `CollectBox` ~:739-770);
2841 - `internal/host/agent.go`:
2842 - a source var beside `upstreamSource` (~:2781);
2843 - the ready hook (~:576-585): publish it;
2844 - `edgeOpts` (~:550-575): `WithGeoTunnelClient` when the tunnel env is complete;
2845 - `statusLoop`'s `Probes` literal (~:2938-3003).
2846- Test: `internal/status/snapshot_test.go` (add one), `internal/host/agent_geoip_test.go` (new)
2847
2848**Interfaces:**
2849- Consumes: `edge.GeoIPState`, `(*edge.Edge).GeoIPState()`, `edge.WithGeoTunnelClient()` (Task 7).
2850- Produces:
2851 ```go
2852 // package status
2853 type GeoIPState struct { Mode string `json:"mode"`; SHA256 string `json:"sha256,omitempty"`; Type string `json:"type,omitempty"`; Error string `json:"error,omitempty"` }
2854 // Probes.GeoIP *GeoIPState ; BoxMetrics.GeoIP *GeoIPState `json:"geoip,omitempty"`
2855 // package host
2856 func tunnelEnvComplete(getenv func(string) string) bool
2857 ```
2858
2859- [ ] **Step 1: Write the failing tests.** In `internal/status/snapshot_test.go`:
2860
2861```go
2862func TestCollectBoxCarriesGeoIP(t *testing.T) {
2863 box := CollectBox(Probes{Label: "b", GeoIP: &GeoIPState{Mode: "mirror", SHA256: "abc", Type: "DBIP-City-Lite"}} /* fill whatever other required fields the other CollectBox tests pass */)
2864 if box.Metrics == nil || box.Metrics.GeoIP == nil || box.Metrics.GeoIP.SHA256 != "abc" {
2865 t.Fatalf("metrics = %+v", box.Metrics)
2866 }
2867 raw, _ := json.Marshal(box)
2868 if !bytes.Contains(raw, []byte(`"geoip":{"mode":"mirror"`)) {
2869 t.Fatalf("json = %s", raw)
2870 }
2871}
2872```
2873
2874Copy the `CollectBox` call shape from the nearest existing test in that file. `CollectBox`'s signature may take more than `Probes`.
2875
2876In `internal/host/agent_geoip_test.go`:
2877
2878```go
2879package host
2880
2881import "testing"
2882
2883func TestTunnelEnvComplete(t *testing.T) {
2884 env := map[string]string{"CARLOS_TUNNEL_URL": "https://t", "CARLOS_TUNNEL_TOKEN": "x", "CARLOS_TUNNEL_NAME": "n"}
2885 get := func(k string) string { return env[k] }
2886 if !tunnelEnvComplete(get) {
2887 t.Fatal("complete env")
2888 }
2889 delete(env, "CARLOS_TUNNEL_NAME")
2890 if tunnelEnvComplete(get) {
2891 t.Fatal("incomplete env must not make a tunnel client (tunnelLoop would not run)")
2892 }
2893}
2894```
2895
2896- [ ] **Step 2: Run them to see them fail**
2897
2898Run: `nice -n 19 go test ./internal/status -run GeoIP -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/host -run TunnelEnvComplete -count=1`
2899Expected: build failures.
2900
2901- [ ] **Step 3: Implement it**
2902
2903`internal/status/snapshot.go`:
2904
2905```go
2906// GeoIPState is one box's GeoIP mirror (geoip spec §5.5). Present on every
2907// box running a GeoIP-aware binary; absent means an older binary, which the
2908// rollout verdict treats as "cannot be trusted to strip forged headers".
2909type GeoIPState struct {
2910 Mode string `json:"mode"` // mirror | tunnel | none
2911 SHA256 string `json:"sha256,omitempty"`
2912 Type string `json:"type,omitempty"`
2913 Error string `json:"error,omitempty"`
2914}
2915```
2916
2917- Add `GeoIP *GeoIPState` to `Probes`, beside `Tunnel`.
2918- Add ``GeoIP *GeoIPState `json:"geoip,omitempty"` `` to `BoxMetrics`, beside `Tunnel`.
2919- In `CollectBox`, add `|| p.GeoIP != nil` to the "anything known" condition, and `GeoIP: p.GeoIP,` to the `BoxMetrics` literal.
2920
2921`internal/host/agent.go`:
2922
2923```go
2924// geoIPSource is the edge's GeoIP state for the heartbeat, published from
2925// the ready hook (same pattern as upstreamSource).
2926var geoIPSource atomic.Value // func() edge.GeoIPState
2927
2928func geoIPState() *status.GeoIPState {
2929 fn, _ := geoIPSource.Load().(func() edge.GeoIPState)
2930 if fn == nil {
2931 return nil
2932 }
2933 s := fn()
2934 return &status.GeoIPState{Mode: s.Mode, SHA256: s.SHA256, Type: s.Type, Error: s.Error}
2935}
2936
2937// tunnelEnvComplete is tunnelLoop's own arming rule: all three set.
2938func tunnelEnvComplete(getenv func(string) string) bool {
2939 return getenv("CARLOS_TUNNEL_URL") != "" && getenv("CARLOS_TUNNEL_TOKEN") != "" && getenv("CARLOS_TUNNEL_NAME") != ""
2940}
2941```
2942
2943- In `edgeOpts`, append `edge.WithGeoTunnelClient()` when `tunnelEnvComplete(os.Getenv)`.
2944- In the ready hook, after `setUpstreamSource(e.UpstreamStates)`, add `geoIPSource.Store(e.GeoIPState)`.
2945- In `statusLoop`'s `Probes` literal, beside `Tunnel: tunnelState()`, add `GeoIP: geoIPState()`.
2946
2947- [ ] **Step 4: Run the tests**
2948
2949Run: `nice -n 19 go test ./internal/status -count=1`, then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/host -count=1`
2950Expected: PASS.
2951
2952- [ ] **Step 5: Commit**
2953
2954```bash
2955git add internal/status/snapshot.go internal/status/snapshot_test.go internal/host/agent.go internal/host/agent_geoip_test.go
2956git commit -m "status: GeoIP block on every box heartbeat; agent marks tunnel clients (geoip spec §5.5)"
2957```
2958
2959---
2960
2961### Task 9: Capture, decide, expose; `setGeo` on every proxy
2962
2963**Files:**
2964- Create: `internal/edge/geoipheaders.go`
2965- Modify: `internal/edge/edge.go`:
2966 - `ServeHTTP` (~:1343-1404);
2967 - the six `Rewrite` funcs: `unixProxy` :2233, `tcpProxy` :2259, `multiProxy` :2347, `tunnelProxy` :2434, `fleetProxy` :2489, `backhaulProxy` :2863.
2968- Test: `internal/edge/geoipheaders_test.go`
2969
2970**Interfaces:**
2971- Consumes: `geoip.Capture/Adopt/FromSnapshot/Off/WriteApp/HopValue/StripAll/HeaderHop` (Task 5); `e.geo` (Task 7); `hopVerified`, `doorCallerFrom`, `clientIP`; `tunnelclient.FromTunnel(ctx) bool` (Task 10; until then the stub `fromTunnel` below returns false).
2972- Produces:
2973 ```go
2974 type geoResultKey struct{}
2975 func geoAnswerFrom(ctx context.Context) (geoip.Answer, bool)
2976 func (e *Edge) decideGeo(r *http.Request, c geoip.Captured, hopPresent, hopped, isDoor bool) geoip.Answer
2977 func setGeo(pr *httputil.ProxyRequest, forwardsToEdge bool)
2978 var fromTunnel = func(ctx context.Context) bool { return false } // Task 10 points it at tunnelclient.FromTunnel
2979 ```
2980
2981- [ ] **Step 1: Write the failing tests.** `internal/edge/geoipheaders_test.go`:
2982
2983```go
2984package edge
2985
2986import (
2987 "context"
2988 "encoding/json"
2989 "net"
2990 "net/http"
2991 "net/http/httptest"
2992 "net/http/httputil"
2993 "sync/atomic"
2994 "testing"
2995 "time"
2996
2997 "github.com/carlosframework/platform/internal/geo"
2998 "github.com/carlosframework/platform/internal/geoip"
2999 "github.com/carlosframework/platform/internal/registry"
3000)
3001
3002var geoHeaderNames = []string{geoip.HeaderCountry, geoip.HeaderSubdivision, geoip.HeaderLat, geoip.HeaderLon, geoip.HeaderSource, geoip.HeaderHop}
3003
3004// geoEchoBackend answers with the geo headers it received, as JSON.
3005func geoEchoBackend(t *testing.T) *httptest.Server {
3006 t.Helper()
3007 ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
3008 got := map[string]string{}
3009 for _, k := range geoHeaderNames {
3010 if v := r.Header.Get(k); v != "" {
3011 got[k] = v
3012 }
3013 }
3014 w.Header().Set("Content-Type", "application/json")
3015 json.NewEncoder(w).Encode(got)
3016 }))
3017 t.Cleanup(ts.Close)
3018 return ts
3019}
3020
3021// geoEdge serves host from a geo-echo backend.
3022func geoEdge(t *testing.T, host string) *Edge {
3023 t.Helper()
3024 backend := geoEchoBackend(t)
3025 reg := openReg(t)
3026 if err := reg.Upsert(registry.Route{Host: host, Kind: "service", Addr: backend.Listener.Addr().String(), Status: "ready"}); err != nil {
3027 t.Fatal(err)
3028 }
3029 e, err := New(reg, nil)
3030 if err != nil {
3031 t.Fatal(err)
3032 }
3033 t.Cleanup(e.probers.Stop)
3034 return e
3035}
3036
3037// spyGeo makes every lookup a hit for country and counts the calls.
3038func spyGeo(e *Edge, country string) *atomic.Int64 {
3039 var n atomic.Int64
3040 e.geoLookup = func(net.IP) geo.Snapshot {
3041 n.Add(1)
3042 return geo.Snapshot{Loaded: true, Found: true, Source: "TestDB-" + country,
3043 Result: geo.Result{Country: country, Point: geo.Point{Lat: 1.234, Lon: 5.678}, HasPoint: true}}
3044 }
3045 return &n
3046}
3047
3048// serveGeo drives one request through ServeHTTP and returns the headers the
3049// app saw.
3050func serveGeo(t *testing.T, e *Edge, r *http.Request) (int, map[string]string) {
3051 t.Helper()
3052 w := httptest.NewRecorder()
3053 e.ServeHTTP(w, r)
3054 got := map[string]string{}
3055 if w.Code == http.StatusOK {
3056 if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
3057 t.Fatalf("backend body %q: %v", w.Body.String(), err)
3058 }
3059 }
3060 return w.Code, got
3061}
3062
3063func geoReq(host, from string) *http.Request {
3064 r := httptest.NewRequest("GET", "http://"+host+"/", nil)
3065 r.RemoteAddr = net.JoinHostPort(from, "4444")
3066 return r
3067}
3068
3069func TestDirectRequestGetsGeoHeadersFromTheRealReader(t *testing.T) {
3070 e := geoEdge(t, "app.test")
3071 db, err := geo.OpenMMDB("../geo/testdata/GeoIP2-City-Test.mmdb")
3072 if err != nil {
3073 t.Fatal(err)
3074 }
3075 e.geo.Swap(db)
3076 code, h := serveGeo(t, e, geoReq("app.test", "81.2.69.142"))
3077 if code != 200 || h[geoip.HeaderCountry] != "GB" || h[geoip.HeaderSubdivision] != "GB-ENG" ||
3078 h[geoip.HeaderSource] != "GeoIP2-City" || h[geoip.HeaderLat] != "51.51" || h[geoip.HeaderLon] != "-0.09" {
3079 t.Fatalf("app saw %v", h)
3080 }
3081 if _, leaked := h[geoip.HeaderHop]; leaked {
3082 t.Fatal("X-Carlos-Hop-Geo reached the app")
3083 }
3084}
3085
3086func TestNoDatabaseMeansNoHeaders(t *testing.T) {
3087 e := geoEdge(t, "app.test")
3088 r := geoReq("app.test", "81.2.69.142")
3089 r.Header.Set(geoip.HeaderCountry, "KP")
3090 if _, h := serveGeo(t, e, r); len(h) != 0 {
3091 t.Fatalf("app saw %v with no database loaded", h)
3092 }
3093}
3094
3095// Review Focus 1.
3096func TestForgedGeoHeadersNeverReachTheApp(t *testing.T) {
3097 e := geoEdge(t, "app.test")
3098 spyGeo(e, "GB")
3099 for name, mutate := range map[string]func(*http.Request){
3100 "plain forgery": func(r *http.Request) {
3101 r.Header.Set(geoip.HeaderCountry, "KP")
3102 r.Header.Set(geoip.HeaderSource, "forged")
3103 r.Header.Set(geoip.HeaderHop, "hit;src=forged")
3104 },
3105 "connection nomination": func(r *http.Request) {
3106 r.Header.Set("Connection", "X-Carlos-Geo-Country, X-Carlos-Geo-Source")
3107 },
3108 } {
3109 r := geoReq("app.test", "81.2.69.142")
3110 mutate(r)
3111 _, h := serveGeo(t, e, r)
3112 if h[geoip.HeaderCountry] != "GB" || h[geoip.HeaderSource] != "TestDB-GB" {
3113 t.Errorf("%s: app saw %v, want the edge's own GB answer", name, h)
3114 }
3115 if _, leaked := h[geoip.HeaderHop]; leaked {
3116 t.Errorf("%s: hop header reached the app", name)
3117 }
3118 }
3119}
3120
3121// meshGeoEdge is geoEdge plus a minted mesh key, so hop requests can be
3122// signed (homemap_test.go's hopEdge, with a geo-echo backend).
3123func meshGeoEdge(t *testing.T, host string) (*Edge, []byte) {
3124 t.Helper()
3125 e := geoEdge(t, host)
3126 s := meshTestStore(t)
3127 if err := mintMeshKey(context.Background(), s); err != nil {
3128 t.Fatal(err)
3129 }
3130 m := newHomeMap(s, "home")
3131 refreshHomeMap(t, m)
3132 e.homes.Store(m)
3133 key, _ := m.meshKeys()
3134 return e, key
3135}
3136
3137func TestVerifiedHopAdoptsTheSendersAnswerWithoutLookingUp(t *testing.T) {
3138 e, key := meshGeoEdge(t, "app.test")
3139 spy := spyGeo(e, "XX") // a lookup here would show as XX
3140 r := geoReq("app.test", "198.51.100.7") // the sending edge's address
3141 r.Header.Set(hopHeader, "1")
3142 r.Header.Set(hopAuthHeader, signHop(key, "app.test", time.Now()))
3143 r.Header.Set("X-Forwarded-For", "81.2.69.142")
3144 r.Header.Set(geoip.HeaderHop, "hit;src=GeoLite2-City")
3145 r.Header.Set(geoip.HeaderCountry, "FR")
3146 code, h := serveGeo(t, e, r)
3147 if code != 200 || h[geoip.HeaderCountry] != "FR" || h[geoip.HeaderSource] != "GeoLite2-City" {
3148 t.Fatalf("app saw %d %v, want the adopted FR", code, h)
3149 }
3150 if spy.Load() != 0 {
3151 t.Fatal("the home edge looked the visitor up instead of adopting")
3152 }
3153}
3154
3155// Review Focus 2, codex R2-1.
3156func TestRejectedHopIsServedWithNoGeo(t *testing.T) {
3157 e, _ := meshGeoEdge(t, "app.test")
3158 spy := spyGeo(e, "XX")
3159 r := geoReq("app.test", "81.2.69.142")
3160 r.Header.Set(hopHeader, "1")
3161 r.Header.Set(hopAuthHeader, signHop([]byte("not-the-mesh-key-not-the-mesh-key"), "app.test", time.Now()))
3162 r.Header.Set(geoip.HeaderHop, "hit;src=GeoLite2-City")
3163 r.Header.Set(geoip.HeaderCountry, "FR")
3164 code, h := serveGeo(t, e, r)
3165 if code != 200 {
3166 t.Fatalf("a rejected hop must still be served (as today), got %d", code)
3167 }
3168 if len(h) != 0 || spy.Load() != 0 {
3169 t.Fatalf("rejected hop: app saw %v, lookups %d; want nothing and no lookup", h, spy.Load())
3170 }
3171}
3172
3173func TestDecideGeoDoorIsOffWithoutALookup(t *testing.T) {
3174 e := geoEdge(t, "app.test")
3175 spy := spyGeo(e, "GB")
3176 a := e.decideGeo(geoReq("app.test", "81.2.69.142"), geoip.Captured{}, false, false, true)
3177 if a.State != geoip.StateOff || spy.Load() != 0 {
3178 t.Fatalf("door = %+v, lookups %d", a, spy.Load())
3179 }
3180}
3181
3182func TestDoorCallReachesTheAppWithNoGeo(t *testing.T) {
3183 s := newDoorScene(t)
3184 spy := spyGeo(s.e, "GB")
3185 rec := s.do(t, "GET", doorOwnHost, "/", s.key, map[string]string{geoip.HeaderCountry: "KP"}, nil)
3186 if rec.Code != http.StatusOK {
3187 t.Fatalf("door call = %d %s", rec.Code, rec.Body.String())
3188 }
3189 if spy.Load() != 0 {
3190 t.Fatal("a door call was looked up: it would locate the box itself")
3191 }
3192 // The own upstream (newDoorUpstream) records what it received; assert it
3193 // saw no X-Carlos-Geo-* header. Read newDoorUpstream for its accessor
3194 // (door_test.go) and assert through it.
3195}
3196
3197func TestSetGeoForBackhaulCarriesStateAndAppsNeverSeeIt(t *testing.T) {
3198 in := httptest.NewRequest("GET", "http://app.test/", nil)
3199 for _, c := range []struct {
3200 a geoip.Answer
3201 forward bool
3202 hop string
3203 }{
3204 {geoip.Answer{State: geoip.StateHit, Country: "GB", Source: "S"}, true, "hit;src=S"},
3205 {geoip.Off(), true, "off"}, // a door call being backhauled
3206 {geoip.Answer{State: geoip.StateHit, Country: "GB", Source: "S"}, false, ""},
3207 } {
3208 pr := &httputil.ProxyRequest{In: in.WithContext(context.WithValue(in.Context(), geoResultKey{}, c.a)), Out: in.Clone(context.Background())}
3209 pr.Out.Header.Set(geoip.HeaderHop, "hit;src=forged")
3210 pr.Out.Header.Set(geoip.HeaderCountry, "KP")
3211 setGeo(pr, c.forward)
3212 if got := pr.Out.Header.Get(geoip.HeaderHop); got != c.hop {
3213 t.Errorf("%+v forward=%v: hop = %q, want %q", c.a, c.forward, got, c.hop)
3214 }
3215 if got := pr.Out.Header.Get(geoip.HeaderCountry); got == "KP" {
3216 t.Errorf("%+v: forged country survived setGeo", c.a)
3217 }
3218 }
3219 // A request that never passed ServeHTTP (the scheduler's UnixProxy) gets nothing.
3220 pr := &httputil.ProxyRequest{In: in, Out: in.Clone(context.Background())}
3221 pr.Out.Header.Set(geoip.HeaderCountry, "KP")
3222 setGeo(pr, false)
3223 if pr.Out.Header.Get(geoip.HeaderCountry) != "" {
3224 t.Fatal("no context answer must mean no geo headers")
3225 }
3226}
3227
3228func TestBackhaulDeliversTheSendersAnswer(t *testing.T) {
3229 s := newBackhaulScene(t)
3230 s.converge(t)
3231 homeSpy, senderSpy := spyGeo(s.a.edge, "XX"), spyGeo(s.b.edge, "GB")
3232 _, body := s.get(t, s.b, "https://"+e2eHomedHost+"/", nil, true)
3233 seen := e2eParseApp(t, body)
3234 if seen.geoCountry != "GB" || seen.hopGeo != "" {
3235 t.Fatalf("home app saw country=%q hopGeo=%q, want GB and no hop header", seen.geoCountry, seen.hopGeo)
3236 }
3237 if homeSpy.Load() != 0 || senderSpy.Load() != 1 {
3238 t.Fatalf("lookups home=%d sender=%d, want 0 and 1 (ingress computes, spec §8.2)", homeSpy.Load(), senderSpy.Load())
3239 }
3240}
3241
3242// Review Focus 1: Vary on the geo header picks the right variant.
3243func TestCacheVariesOnTheEdgesCountry(t *testing.T) {
3244 var hits atomic.Int64
3245 backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
3246 hits.Add(1)
3247 w.Header().Set("Cache-Control", "max-age=60")
3248 w.Header().Set("Vary", geoip.HeaderCountry)
3249 w.Write([]byte("country=" + r.Header.Get(geoip.HeaderCountry)))
3250 }))
3251 t.Cleanup(backend.Close)
3252 reg := openReg(t)
3253 reg.Upsert(registry.Route{Host: "app.test", Kind: "instance", Addr: backend.Listener.Addr().String(), Status: "ready"})
3254 e, err := New(reg, nil)
3255 if err != nil {
3256 t.Fatal(err)
3257 }
3258 t.Cleanup(e.probers.Stop)
3259 e.EnableCache(NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes))
3260 db, _ := geo.OpenMMDB("../geo/testdata/GeoIP2-City-Test.mmdb")
3261 e.geo.Swap(db)
3262 get := func(from string, forged string) string {
3263 r := geoReq("app.test", from)
3264 if forged != "" {
3265 r.Header.Set(geoip.HeaderCountry, forged)
3266 }
3267 w := httptest.NewRecorder()
3268 e.ServeHTTP(w, r)
3269 return w.Body.String()
3270 }
3271 if b := get("81.2.69.142", ""); b != "country=GB" {
3272 t.Fatalf("GB = %q", b)
3273 }
3274 if b := get("216.160.83.56", ""); b != "country=US" {
3275 t.Fatalf("US = %q", b)
3276 }
3277 if b := get("81.2.69.142", "US"); b != "country=GB" {
3278 t.Fatalf("a forged header picked another country's variant: %q", b)
3279 }
3280 if hits.Load() != 2 || cachedVariants(e, "app.test", "/") != 2 {
3281 t.Fatalf("hits=%d variants=%d, want 2 and 2", hits.Load(), cachedVariants(e, "app.test", "/"))
3282 }
3283}
3284
3285func TestUpgradeRequestCarriesGeo(t *testing.T) {
3286 // upgradeEchoBackend (edge_test.go ~:1991) echoes the request headers it
3287 // received on the upgraded connection; route app.test to it, spyGeo(e,
3288 // "GB"), send a WebSocket upgrade from 81.2.69.142 through ServeHTTP on a
3289 // real httptest server (upgrades need a hijackable writer), and assert
3290 // the echoed headers contain X-Carlos-Geo-Country: GB. Read
3291 // TestServeHTTPUpgradeRequestBypassesRecording (~:1301) for the exact
3292 // client handshake this package's tests use, and copy it.
3293}
3294```
3295
3296For `TestBackhaulDeliversTheSendersAnswer`, extend `e2eApp` (`backhaul_e2e_test.go:101`) so it also prints `|geo=<X-Carlos-Geo-Country>|hopgeo=<X-Carlos-Hop-Geo>`. Extend `e2eSeen` and `e2eParseApp` with `geoCountry` and `hopGeo`. Every existing test that parses `e2eApp` output must still pass, so add the new fields at the end.
3297
3298Two bodies above are sketched as comments: `TestDoorCallReachesTheAppWithNoGeo`'s final assertion, and `TestUpgradeRequestCarriesGeo`. Both depend on helpers whose exact accessors the executor must read first. Write them from the named helpers, and make sure each fails before Step 3.
3299
3300- [ ] **Step 2: Run them to see them fail**
3301
3302Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Geo|Forged|Hop|Door.*Geo|Backhaul|Cache.*Country|Upgrade.*Geo' -count=1`
3303Expected: build failure (`decideGeo`, `setGeo`, `geoResultKey`, `geoLookup` undefined), then FAIL until Step 3 is complete.
3304
3305- [ ] **Step 3: Implement it.** `internal/edge/geoipheaders.go`:
3306
3307```go
3308package edge
3309
3310import (
3311 "context"
3312 "net"
3313 "net/http"
3314 "net/http/httputil"
3315
3316 "github.com/carlosframework/platform/internal/geoip"
3317)
3318
3319// geoResultKey carries the request's one geo answer (spec §8.2): computed
3320// at ingress or adopted from a trusted transport, read by setGeo, the cache
3321// path (through r.Header) and analytics.
3322type geoResultKey struct{}
3323
3324func geoAnswerFrom(ctx context.Context) (geoip.Answer, bool) {
3325 a, ok := ctx.Value(geoResultKey{}).(geoip.Answer)
3326 return a, ok
3327}
3328
3329// fromTunnel reports the tunnel client's mark (Task 10 sets it to
3330// tunnelclient.FromTunnel). A var so this file builds before Task 10.
3331var fromTunnel = func(ctx context.Context) bool { return false }
3332
3333// decideGeo is step 3 of spec §8.1. hopPresent says the request ARRIVED
3334// carrying X-Carlos-Hop (captured before acceptHop strips it); hopped says
3335// acceptHop verified it.
3336func (e *Edge) decideGeo(r *http.Request, c geoip.Captured, hopPresent, hopped, isDoor bool) geoip.Answer {
3337 switch {
3338 case isDoor:
3339 return geoip.Off() // RemoteAddr is this box (door.go)
3340 case hopped:
3341 return c.Adopt() // the sending edge looked up; never trust our own peer
3342 case hopPresent:
3343 return geoip.Off() // rejected hop: RemoteAddr is the sending edge (codex R2-1)
3344 case fromTunnel(r.Context()):
3345 return c.Adopt() // the hub looked up (spec §8.4)
3346 }
3347 ip := net.ParseIP(clientIP(r))
3348 return geoip.FromSnapshot(e.geoLookup(ip))
3349}
3350
3351// setGeo is every Rewrite's geo step (spec §8.3): after hop-by-hop removal,
3352// delete every geo header from the outbound request and write the answer
3353// back from the context. forwardsToEdge adds X-Carlos-Hop-Geo for the next
3354// edge; an app never receives it.
3355func setGeo(pr *httputil.ProxyRequest, forwardsToEdge bool) {
3356 geoip.StripAll(pr.Out.Header)
3357 a, ok := geoAnswerFrom(pr.In.Context())
3358 if !ok {
3359 a = geoip.Off() // scheduler calls through UnixProxy and anything else that never passed ServeHTTP
3360 }
3361 a.WriteApp(pr.Out.Header)
3362 if forwardsToEdge {
3363 pr.Out.Header.Set(geoip.HeaderHop, a.HopValue())
3364 }
3365}
3366```
3367
3368Edit `ServeHTTP`:
3369
3370```go
3371 // (immediately after the host split, before doorCallerFrom)
3372 // GeoIP step 1 (spec §8.1): capture and strip before ANY branch. Nothing
3373 // trusts these values until step 3.
3374 capturedGeo := geoip.Capture(r.Header)
3375 hopPresent := r.Header.Get(hopHeader) != ""
3376```
3377
3378```go
3379 // (immediately after `r, hopped := e.acceptHop(r, host)`)
3380 // GeoIP steps 3-4: decide once, put it on the context, and expose the
3381 // app-visible headers on r.Header so Vary matching sees the values the
3382 // app will get.
3383 geoAns := e.decideGeo(r, capturedGeo, hopPresent, hopped, isDoor)
3384 r = r.WithContext(context.WithValue(r.Context(), geoResultKey{}, geoAns))
3385 geoAns.WriteApp(r.Header)
3386```
3387
3388In each of the six `Rewrite` funcs, add the line right after `setForwarded(pr)`:
3389
3390| Proxy | Line to add |
3391|---|---|
3392| `unixProxy`, `tcpProxy`, `multiProxy` | `setGeo(pr, false)` |
3393| `tunnelProxy`, `fleetProxy`, `backhaulProxy` | `setGeo(pr, true)` |
3394
3395- [ ] **Step 4: Run the tests**
3396
3397Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -count=1`
3398Expected: PASS, the whole package. In particular the existing hop, door, cache and purge tests still pass.
3399
3400- [ ] **Step 5: Commit**
3401
3402```bash
3403git add internal/edge/geoipheaders.go internal/edge/geoipheaders_test.go internal/edge/edge.go
3404git commit -m "edge: capture/decide/expose geo headers; setGeo on every proxy; door and rejected hop get none (geoip spec §8)"
3405```
3406
3407---
3408
3409### Task 10: The tunnel mark (spec D2: every tunnelled request, whatever the scheme)
3410
3411**Files:**
3412- Modify: `internal/tunnelclient/tunnelclient.go` (`Config` :66; `Loop` :87; `serveUntilDead` :158; `Dial` :191; `Serve` :265)
3413- Modify: `internal/edge/geoipheaders.go` (point `fromTunnel` at `tunnelclient.FromTunnel`)
3414- Test: `internal/tunnelclient/mark_test.go`, `internal/edge/tunnel_geo_e2e_test.go`
3415
3416**Interfaces:**
3417- Produces:
3418 ```go
3419 // package tunnelclient
3420 func FromTunnel(ctx context.Context) bool
3421 // Config gains: TLSConfig *tls.Config // nil = system roots (production); tests pass the test server's pool
3422 func Mark(h http.Handler) http.Handler // wraps h so every request carries the mark
3423 ```
3424 `Loop` serves `Mark(cfg.Handler)` whatever `cfg.URL`'s scheme (spec D2). The mark means the hub computed this request's geo answer. The remote adopts that answer and does not observe the request for analytics.
3425
3426 **Ignore** every mention below of `ViaTunnel`, `viaTunnelKey` or `viaHandler`, and of "TLS only" or "plaintext gets no geo". Rev 7's D2 removed them. There is one mark, `FromTunnel`, and `servedHandler(cfg, h)` returns `Mark(h)` unconditionally. `TestServeHandlerMarksTrustOnlyOverTLSButViaAlways` becomes `TestServeHandlerAlwaysMarks`: plain and TLS both see `FromTunnel == true`. The `Config.TLSConfig` seam is still added, so the e2e can run over real TLS. `observeAnalytics` (Task 11) checks `tunnelclient.FromTunnel`.
3427
3428- [ ] **Step 1: Write the failing tests.** `internal/tunnelclient/mark_test.go`:
3429
3430```go
3431package tunnelclient
3432
3433import (
3434 "net/http"
3435 "net/http/httptest"
3436 "testing"
3437)
3438
3439func TestMarkIsOnlyOnMarkedHandlers(t *testing.T) {
3440 var saw bool
3441 h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { saw = FromTunnel(r.Context()) })
3442 h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
3443 if saw {
3444 t.Fatal("an unmarked handler saw the mark")
3445 }
3446 Mark(h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
3447 if !saw {
3448 t.Fatal("Mark did not mark")
3449 }
3450}
3451
3452func TestServeHandlerMarksTrustOnlyOverTLSButViaAlways(t *testing.T) {
3453 var trusted, via bool
3454 h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
3455 trusted, via = FromTunnel(r.Context()), ViaTunnel(r.Context())
3456 })
3457 servedHandler(Config{URL: "http://edge.test/register"}, h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
3458 if trusted || !via {
3459 t.Fatalf("plaintext: trusted=%v via=%v, want false/true (codex R2-2, R4-1)", trusted, via)
3460 }
3461 servedHandler(Config{URL: "https://edge.test/register"}, h).ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
3462 if !trusted || !via {
3463 t.Fatalf("tls: trusted=%v via=%v, want true/true", trusted, via)
3464 }
3465}
3466```
3467
3468`internal/edge/tunnel_geo_e2e_test.go` follows `TestTunnelEndToEnd` (`tunnel_e2e_test.go:37`), but:
3469
3470- the hub is `httptest.NewTLSServer(hubEdge)`;
3471- the remote host's `tunnelclient.Loop` gets `Config{URL: "https://…", TLSConfig: &tls.Config{RootCAs: pool, ServerName: …}}`, where `pool` holds `srv.Certificate()`;
3472- the remote handler is a second `Edge` with an echo backend route.
3473
3474Assert:
3475
3476- a request to the hub from `RemoteAddr` 81.2.69.142 (City fixture loaded in the hub's `e.geo`) reaches the remote app with `X-Carlos-Geo-Country: GB` and no `X-Carlos-Hop-Geo`;
3477- the remote edge's `e.geo` is empty and was never consulted;
3478- the remote edge's analytics meter records `Country: "GB"` (this assertion lands in Task 11; leave a `// Task 11` marker here);
3479- the same scene over plain `http://` delivers the **same** geo headers (spec D2). Cover both schemes in one table-driven test.
3480
3481Factor `startFakeRemoteHost` (`tunnel_e2e_test.go:190`) so it takes a `tunnelclient.Config` mutator, and don't copy it.
3482
3483- [ ] **Step 2: Run them to see them fail**
3484
3485Run: `nice -n 19 go test ./internal/tunnelclient -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run TunnelGeo -count=1`
3486Expected: build failures.
3487
3488- [ ] **Step 3: Implement it**
3489
3490```go
3491// internal/tunnelclient/tunnelclient.go
3492
3493type tunnelKey struct{}
3494
3495// FromTunnel reports whether a request arrived over a TLS tunnel this
3496// client dialled: the edge's grant to adopt the hub's geo headers (geoip
3497// spec §8.4). Nothing outside this package can set it.
3498func FromTunnel(ctx context.Context) bool {
3499 v, _ := ctx.Value(tunnelKey{}).(bool)
3500 return v
3501}
3502
3503type markedHandler struct{ h http.Handler }
3504
3505func (m markedHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
3506 m.h.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), tunnelKey{}, true)))
3507}
3508
3509// Mark wraps h so every request it serves carries the tunnel mark.
3510func Mark(h http.Handler) http.Handler { return markedHandler{h} }
3511
3512type viaTunnelKey struct{}
3513
3514// ViaTunnel reports that a request was served over a tunnel at all, TLS or
3515// not. It grants nothing; the edge uses it only to NOT count a request the
3516// hub already counted (geoip spec §9, codex R4-1).
3517func ViaTunnel(ctx context.Context) bool {
3518 v, _ := ctx.Value(viaTunnelKey{}).(bool)
3519 return v
3520}
3521
3522type viaHandler struct{ h http.Handler }
3523
3524func (v viaHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
3525 v.h.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), viaTunnelKey{}, true)))
3526}
3527
3528// servedHandler is what Loop serves: always via-marked; trust-marked only
3529// over https, because the registration token authenticates the box to the
3530// hub, not the hub to the box (codex R2-2).
3531func servedHandler(cfg Config, h http.Handler) http.Handler {
3532 if u, err := url.Parse(cfg.URL); err == nil && u.Scheme == "https" {
3533 h = Mark(h)
3534 }
3535 return viaHandler{h}
3536}
3537```
3538
3539- In `Config`, add `TLSConfig *tls.Config` with the comment: `// nil = system roots and ServerName from the URL; tests pass a pool`.
3540- Thread it into `Dial`: add `DialConfig(ctx, cfg Config)`, have the existing `Dial(ctx, url, name, token)` call it, and in the `https` branch clone `cfg.TLSConfig` when it is set, then fill in `ServerName` and `NextProtos`.
3541- In `Loop`, compute `handler := servedHandler(cfg, cfg.Handler)` once, and pass `handler` to `serveUntilDead`.
3542
3543`internal/edge/geoipheaders.go`: replace the stub with:
3544
3545```go
3546var fromTunnel = tunnelclient.FromTunnel
3547```
3548
3549and import `internal/tunnelclient`. The edge's tests already import it. Check that `go list -deps ./internal/tunnelclient` doesn't include `internal/edge`, so there's no import cycle.
3550
3551- [ ] **Step 4: Run the tests**
3552
3553Run: `nice -n 19 go test ./internal/tunnelclient -count=1` and `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Tunnel' -count=1`
3554Expected: PASS, including the existing `TestTunnelEndToEnd`.
3555
3556- [ ] **Step 5: Commit**
3557
3558```bash
3559git add internal/tunnelclient/ internal/edge/geoipheaders.go internal/edge/tunnel_geo_e2e_test.go internal/edge/tunnel_e2e_test.go
3560git commit -m "tunnelclient: TLS-only tunnel mark; the remote edge adopts the hub's geo (geoip spec §8.4)"
3561```
3562
3563---
3564
3565### Task 11: Analytics countries, end to end
3566
3567**Files:**
3568- Modify:
3569 - `internal/analytics/meter.go` (`Obs`; `delta`; `Observe` pageview block);
3570 - `internal/analytics/day.go` (`Delta`, `Day.GeoSources`, `Merge`);
3571 - `internal/analytics/read.go` (`Summary`, `ReadRange`);
3572 - `internal/edge/analytics.go` (`observeAnalytics`);
3573 - `internal/console/apianalytics.go`;
3574 - `apimode.go` (`apiAnalyticsResponse`);
3575 - `analytics_cmd.go`.
3576- Test: `internal/analytics/countries_test.go`, plus an assertion added to `internal/edge/analytics_test.go`.
3577
3578**Interfaces:**
3579- Consumes: `geoAnswerFrom` (Task 9).
3580- Produces:
3581 - `Obs.Country`, `Obs.GeoSource string`, `Obs.GeoState string` (`"hit"|"miss"|"off"|""`);
3582 - `Delta.Countries`, `Delta.GeoSources map[string]int64`;
3583 - ``Day.GeoSources map[string]int64 `json:"geo_sources,omitempty"` ``;
3584 - `Summary.Countries`, `Summary.GeoSources map[string]int64`;
3585 - the API's `countries` and `geo_sources`.
3586
3587Counting rules (spec §9):
3588
3589- **Counted only for pageviews.**
3590- **By state:**
3591 - `hit` with a country counts that country;
3592 - `hit` with only coordinates, or `miss`, counts `"_unknown"`;
3593 - `off` or `""` counts nothing.
3594- **`GeoSources[source]++`** whenever the state is `hit` or `miss`.
3595- **Bounds:**
3596 - `Countries` keeps at most 300 keys, then `_other`;
3597 - `GeoSources` keeps at most 8 keys, then `_other`;
3598 - the API's `countries` is the top 10 plus `_other` (the sum of the rest).
3599
3600- [ ] **Step 1: Write the failing tests.** `internal/analytics/countries_test.go`:
3601
3602```go
3603package analytics
3604
3605import (
3606 "context"
3607 "encoding/json"
3608 "testing"
3609 "time"
3610
3611 "github.com/carlosframework/platform/internal/release"
3612)
3613
3614func pv(country, src, state string) Obs {
3615 return Obs{Acct: "a", App: "x", Host: "x.test", Method: "GET", Status: 200, ContentType: "text/html",
3616 Path: "/", UA: "Mozilla/5.0", ClientIP: "203.0.113.1", When: time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC),
3617 Country: country, GeoSource: src, GeoState: state}
3618}
3619
3620func TestMeterCountsCountriesByState(t *testing.T) {
3621 m := newTestMeter(t) // use the constructor the existing meter tests use
3622 m.Observe(pv("GB", "DBIP-City-Lite", "hit"))
3623 m.Observe(pv("", "DBIP-City-Lite", "hit")) // coordinate-only
3624 m.Observe(pv("", "DBIP-City-Lite", "miss"))
3625 m.Observe(pv("", "", "off"))
3626 nonPV := pv("FR", "DBIP-City-Lite", "hit")
3627 nonPV.Status = 404
3628 m.Observe(nonPV)
3629 d := onlyDelta(t, m) // drain helper from the existing tests
3630 if d.Countries["GB"] != 1 || d.Countries["_unknown"] != 2 || len(d.Countries) != 2 {
3631 t.Fatalf("countries = %v", d.Countries)
3632 }
3633 if d.GeoSources["DBIP-City-Lite"] != 3 || len(d.GeoSources) != 1 {
3634 t.Fatalf("sources = %v", d.GeoSources)
3635 }
3636}
3637
3638func TestReadRangeMergesCountriesAcrossBoxesDaysAndOldDays(t *testing.T) {
3639 ctx := context.Background()
3640 s := release.NewDirStore(t.TempDir())
3641 put := func(box, date string, d Day) {
3642 raw, _ := json.Marshal(d)
3643 if _, err := s.Put(ctx, DayKey("a", "x", box, date), raw, ""); err != nil {
3644 t.Fatal(err)
3645 }
3646 }
3647 // An old day: no geo_sources, empty countries.
3648 put("b1", "2026-10-04", Day{V: 1, Date: "2026-10-04", Countries: map[string]int64{}})
3649 put("b1", "2026-10-05", Day{V: 1, Date: "2026-10-05", Countries: map[string]int64{"GB": 3}, GeoSources: map[string]int64{"DBIP-City-Lite": 3}})
3650 put("b2", "2026-10-05", Day{V: 1, Date: "2026-10-05", Countries: map[string]int64{"GB": 1, "IE": 2}, GeoSources: map[string]int64{"GeoLite2-City": 3}})
3651 sum, err := ReadRange(ctx, s, "a", "x", time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC), time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC))
3652 if err != nil {
3653 t.Fatal(err)
3654 }
3655 if sum.Countries["GB"] != 4 || sum.Countries["IE"] != 2 {
3656 t.Fatalf("countries = %v", sum.Countries)
3657 }
3658 if sum.GeoSources["DBIP-City-Lite"] != 3 || sum.GeoSources["GeoLite2-City"] != 3 {
3659 t.Fatalf("both sources must be credited (spec §9): %v", sum.GeoSources)
3660 }
3661}
3662
3663func TestDayMergeAddsCountries(t *testing.T) {
3664 var d Day
3665 d.Merge(Delta{Box: "b", Seq: 1, Countries: map[string]int64{"GB": 1}, GeoSources: map[string]int64{"s": 1}})
3666 d.Merge(Delta{Box: "b", Seq: 2, Countries: map[string]int64{"GB": 2, "IE": 1}, GeoSources: map[string]int64{"s": 3}})
3667 if d.Countries["GB"] != 3 || d.Countries["IE"] != 1 || d.GeoSources["s"] != 4 {
3668 t.Fatalf("day = %v %v", d.Countries, d.GeoSources)
3669 }
3670}
3671
3672func TestBoundedAddFoldsIntoOther(t *testing.T) {
3673 m := map[string]int64{}
3674 for i := 0; i < 12; i++ {
3675 boundedAdd(m, string(rune('A'+i))+"X", 1, 8)
3676 }
3677 if len(m) != 9 || m["_other"] != 4 {
3678 t.Fatalf("bounded = %v", m)
3679 }
3680}
3681```
3682
3683Replace `newTestMeter` and `onlyDelta` with the helpers `meter_test.go` already uses to build a `Meter` and drain one delta (`grep -n "func new.*Meter\|Drain()" internal/analytics/*_test.go`).
3684
3685In `internal/edge/tunnel_geo_e2e_test.go`, give the hub and the remote edge an analytics meter each (`EnableAnalytics` with the app switched on through `SetAnalyticsOn`), send one fresh request and one that the hub answers from its cache, then drain both meters. The remote meter must record **zero** observations. The hub's meter must record pageviews, `Countries["GB"]` and `GeoSources` of exactly 2 each, one per served response (spec §8.4, §9). The same must hold over a plaintext tunnel (D2): the remote records zero, and the hub records 2 pageviews and 2 for `Countries["GB"]`.
3686
3687In `internal/edge/analytics_test.go`, extend `TestEdgeObservesOnlyWhenSwitchedOn` (or add a sibling test). With the City fixture in `e.geo`, a pageview from 81.2.69.142 records `Obs.Country == "GB"`, `GeoState == "hit"` and `GeoSource == "GeoIP2-City"`. Also fill in the Task 10 marker in `tunnel_geo_e2e_test.go`.
3688
3689- [ ] **Step 2: Run them to see them fail**
3690
3691Run: `nice -n 19 go test ./internal/analytics -count=1`
3692Expected: build failures.
3693
3694- [ ] **Step 3: Implement it**
3695
3696`meter.go`:
3697- Add to `Obs`: `Country string`, `GeoSource string`, `GeoState string` (comment: `// geoip spec §9: hit | miss | off`).
3698- In `delta(...)`'s literal, add `Countries: map[string]int64{}, GeoSources: map[string]int64{}`.
3699- In `Observe`'s `if pv {` block, after the referer:
3700
3701```go
3702 switch o.GeoState {
3703 case "hit", "miss":
3704 c := o.Country
3705 if o.GeoState == "miss" || c == "" {
3706 c = "_unknown"
3707 }
3708 boundedAdd(d.Countries, c, 1, maxCountryKeys)
3709 if o.GeoSource != "" {
3710 boundedAdd(d.GeoSources, o.GeoSource, 1, maxGeoSourceKeys)
3711 }
3712 }
3713```
3714
3715`day.go`:
3716
3717```go
3718const (
3719 maxCountryKeys = 300
3720 maxGeoSourceKeys = 8
3721)
3722
3723// boundedAdd adds n to m[k], folding new keys past max into OtherKey.
3724func boundedAdd(m map[string]int64, k string, n int64, max int) {
3725 if _, ok := m[k]; ok || k == OtherKey || len(m) < max {
3726 m[k] += n
3727 return
3728 }
3729 m[OtherKey] += n
3730}
3731
3732func mergeCounts(dst, src map[string]int64, max int) map[string]int64 {
3733 if dst == nil {
3734 dst = map[string]int64{}
3735 }
3736 for k, n := range src {
3737 boundedAdd(dst, k, n, max)
3738 }
3739 return dst
3740}
3741```
3742
3743- Add `Countries map[string]int64` and `GeoSources map[string]int64` to `Delta`.
3744- Add ``GeoSources map[string]int64 `json:"geo_sources,omitempty"` `` to `Day`.
3745- In `Merge`, replace the `if d.Countries == nil { … }` block with:
3746 `d.Countries = mergeCounts(d.Countries, x.Countries, maxCountryKeys)` and `d.GeoSources = mergeCounts(d.GeoSources, x.GeoSources, maxGeoSourceKeys)`.
3747- `Requeue` and `Drain` copy deltas by pointer, so the new maps travel with them. Read `Requeue` (meter.go:252) to confirm it doesn't rebuild a `Delta` field by field. If it does, add the two maps there too.
3748
3749`boundedAdd(m, k, n, 8)` keeps 8 real keys plus `_other`, so the map has len 9, which matches the test. With `len(m) < max` and `_other` itself counted in `len(m)`, the 9th distinct key lands in `_other` only once `_other` exists. Walk through the test's 12 inserts:
3750
3751| Inserts | Effect | `len(m)` after |
3752|---|---|---|
3753| keys 1-8 | each enters as a real key | 8 |
3754| key 9 | goes to `_other` | 9 |
3755| keys 10-12 | go to `_other` | 9 |
3756
3757`_other` ends at 4. That matches the test's assertion.
3758
3759`read.go`:
3760- Add `Countries map[string]int64` and `GeoSources map[string]int64` to `Summary`.
3761- In `ReadRange`, keep `countries, sources := map[string]int64{}, map[string]int64{}` beside `pages, refs, events`. For each day, add `mergeCounts(countries, day.Countries, maxCountryKeys)` and `mergeCounts(sources, day.GeoSources, maxGeoSourceKeys)`.
3762- At the end, set `sum.Countries, sum.GeoSources = countries, sources`.
3763
3764`internal/edge/analytics.go` `observeAnalytics`: first, right after the meter/switch guard,
3765
3766```go
3767 // One observation per served response (geoip spec §9, codex R4-1): the
3768 // hub that forwarded this over the tunnel already observed it.
3769 if tunnelclient.FromTunnel(r.Context()) {
3770 return
3771 }
3772```
3773
3774then, before `ref.Observe(...)`,
3775
3776```go
3777 geoAns, _ := geoAnswerFrom(r.Context())
3778```
3779
3780and in the `analytics.Obs{...}` literal: `Country: geoAns.Country, GeoSource: geoAns.Source, GeoState: string(geoAns.State),`.
3781
3782The `r` the defer sees is the reassigned one from Task 9, because the closure captures the variable. Confirm that `observeAnalytics` is called with that same `r` (edge.go ~:1567).
3783
3784`internal/console/apianalytics.go`:
3785- Add ``Countries map[string]int64 `json:"countries"` `` and ``GeoSources map[string]int64 `json:"geo_sources"` `` to `analyticsResponse`.
3786- Fill them with `topWithOther(sum.Countries, 10)` and `sum.GeoSources`.
3787- Default nil to `{}`, like the others at :84-89.
3788
3789Add:
3790
3791```go
3792// topWithOther keeps the n largest keys (ties by key) and sums the rest
3793// into "_other" (geoip spec §9: top 10 plus other).
3794func topWithOther(m map[string]int64, n int) map[string]int64 {
3795 type kv struct {
3796 k string
3797 v int64
3798 }
3799 var all []kv
3800 for k, v := range m {
3801 if k != "_other" {
3802 all = append(all, kv{k, v})
3803 }
3804 }
3805 sort.Slice(all, func(i, j int) bool {
3806 if all[i].v != all[j].v {
3807 return all[i].v > all[j].v
3808 }
3809 return all[i].k < all[j].k
3810 })
3811 out := map[string]int64{}
3812 other := m["_other"]
3813 for i, e := range all {
3814 if i < n {
3815 out[e.k] = e.v
3816 } else {
3817 other += e.v
3818 }
3819 }
3820 if other > 0 {
3821 out["_other"] = other
3822 }
3823 return out
3824}
3825```
3826
3827with a table test in `internal/console/apianalytics_test.go`: 12 countries, then `_other` holds the sum of the smallest two plus any existing `_other`.
3828
3829- `apimode.go` `apiAnalyticsResponse`: add the same two fields.
3830- `analytics_cmd.go`: after the referers table, call `printAnalyticsTable(w, "Countries", resp.Countries)`, with the exact helper signature at `analytics_cmd.go:157`. **The heading string is copy:** write it as `Countries`, and record it for Task 13's copy review.
3831
3832- [ ] **Step 4: Run the tests**
3833
3834Run: `nice -n 19 go test ./internal/analytics ./internal/console -run 'Countr|Analytics|TopWithOther' -count=1`, then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/edge -run 'Analytics|TunnelGeo' -count=1`, then `nice -n 19 go test . -run Analytics -count=1`.
3835Expected: PASS.
3836
3837- [ ] **Step 5: Commit**
3838
3839```bash
3840git add internal/analytics/ internal/edge/analytics.go internal/edge/analytics_test.go internal/edge/tunnel_geo_e2e_test.go internal/console/apianalytics.go internal/console/apianalytics_test.go apimode.go analytics_cmd.go
3841git commit -m "analytics: count pageviews by country with per-day source provenance (geoip spec §9)"
3842```
3843
3844---
3845
3846### Task 12: Console doors, startup jobs and `carlos system geoip`
3847
3848**Files:**
3849- Create: `internal/console/apisystemgeoip.go`, `internal/console/apisystemgeoip_test.go`, `system_geoip.go`, `system_geoip_test.go`
3850- Modify:
3851 - `internal/console/authcli.go` (routes ~:883);
3852 - `internal/console/svcroutes_test.go` (`closedDoors`);
3853 - `internal/console/server.go` (a `geoip *geoipPublisher` field, built where `Releases` is known; start the daily job);
3854 - `system.go` (verb allowlist :50 and dispatch :56-70; usage text);
3855 - `apimode.go` (session methods).
3856
3857**Interfaces:**
3858- Consumes: `geoipPublisher` (Task 6), `status.BoxStale`, `s.systemBoxes` (`apisystemupdate.go:232`), `apiSystemRoller`, `systemBucket`.
3859- Produces:
3860 - `PUT /api/cli/system/geoip?force=1` with a raw body. It stages the upload, starts a job, and answers 202 `{"job": "<id>"}` (spec §4.2, codex R4-2):
3861
3862 | Status | Meaning |
3863 |---|---|
3864 | 202 | job started |
3865 | 409 | busy, answered before the body is read |
3866 | 413 | over 1 GiB |
3867 | 503 | no bucket |
3868
3869 The job's result (the pointer, or the refusal category for a 422-class refusal) is read at the existing `GET /api/cli/system/jobs/{id}`.
3870
3871 - `GET /api/cli/system/geoip`, answering:
3872 ```json
3873 {"pointer": {...} | null, "expires_at": "...", "boxes": [{"label","region","stale","state","geoip":{...}}], "verdict": "ready"|"not ready", "blocking": ["label: reason"], "partial": bool}
3874 ```
3875 - `DELETE /api/cli/system/geoip`, answering 200 `{"off": true, "feed_set": bool}`.
3876 - The CLI: `carlos system geoip publish <file> [-force]`, `carlos system geoip status`, `carlos system geoip off`.
3877
3878**The verdict** (spec §5.5, codex R4 on R1-16):
3879
3880- **Expected boxes:** the box names under `control/edges/` (list the prefix; strip `.json`).
3881- **Expected fleet boxes:** for each fleet from the registry (`reg.ListFleets()`), its active boxes from `reg.ListFleetBoxes(name)`. Open the registry the way `systemBoxes` does (`s.openPassRegistry()`). Fleet boxes publish no edges record (spec §5.5, codex R5 on R1-16).
3882- **Heartbeats:** `status.LoadBoxesComplete` on the platform prefix and on every `release.NewPrefixStore(s.cfg.Releases, "fleets/"+name+"/")`. Never `LoadBoxes` or `systemBoxes`, which skip unreadable records.
3883- **Listed boxes:** every expected box, plus any heartbeat box whose `Updated` is within 7 days.
3884- An expected box with no heartbeat blocks, as `"<label>: no heartbeat"`.
3885- An incomplete read blocks, as `"heartbeats: unreadable"`.
3886
3887**Ready** only when, for every listed box:
3888
3889- it isn't `BoxStale`;
3890- its `Metrics.GeoIP != nil`;
3891- if its mode is `mirror`, its `SHA256` equals the pointer's (when a live pointer exists).
3892
3893`partial` (a fleet's records unreadable) also makes it not ready. `blocking` lists one string per box in the way:
3894
3895- `"<label>: stale"`
3896- `"<label>: old binary"`
3897- `"<label>: sha <8> not <8>"`
3898- `"<label>: heartbeat unreadable"`
3899
3900- [ ] **Step 1: Write the failing tests.** In `internal/console/apisystemgeoip_test.go`, use whatever helper the existing `apisystemupdate` tests use to stand up a server with a `DirStore` and a logged-in ops owner (`grep -n "func.*systemTestServer\|apiSystemRoller" internal/console/*_test.go`). Tests:
3901
39021. A PUT of the City fixture as an ops owner gives 200, and the pointer exists. Use `withFixtureProbes(t)` from Task 6.
39032. A PUT by a non-operator person gives 404. A service token gives 404, through `cliRoute`'s zero `Set`.
39043. A PUT of junk gives 422, and the body names the category.
39054. A GET with two box records gives the verdict:
3906 - one mirror box at the pointer's sha, fresh;
3907 - one box with no `geoip` block, which reads as not ready, with `blocking` naming it as "old binary".
3908 Write the records with `status.WriteBox`-shaped JSON under `status/boxes/`.
39095. A GET where every box is fresh, aware and at the sha gives `ready`. A heartbeat stamped `SnapshotPeriod + 59s` ago still counts as fresh (codex R3-4).
39106. A DELETE removes the pointer and the blobs.
39117. `closedDoors` gains the three new patterns. The existing equality test fails until it's updated.
3912
3913In `system_geoip_test.go` (package main): follow `system_test.go`'s pattern (it fakes the console with `httptest`). Check that:
3914
3915- `publish` streams the file body with `Content-Type: application/octet-stream` and the right `Content-Length`;
3916- `status` prints the verdict line;
3917- `off` prints the feed warning when `feed_set` is true.
3918
3919- [ ] **Step 2: Run them to see them fail**
3920
3921Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -run 'SystemGeoip|SvcRoutes|ClosedDoors' -count=1` and `nice -n 19 go test . -run SystemGeoip -count=1`
3922Expected: FAIL.
3923
3924- [ ] **Step 3: Implement it**
3925
3926**Publish as upload then job** (spec §4.2, codex R4-2). Split `geoipPublisher.Publish` into two methods. The tests in Task 6 keep calling `Publish`, which becomes `Stage` followed by `Commit`.
3927
3928- `Stage(ctx, src) (staged, error)` copies the upload to the temp dir under `copyCapped`. Before reading **any** of the body it takes the mutex with `TryLock`, and returns `errGeoipBusy` if it can't. That reservation is spec §4.2's one slot (codex R5-1). On success the mutex stays held and passes to the job, which owns the temp dir from then on and releases the mutex after removing it. On failure, `Stage` removes the dir and unlocks.
3929- Add `TestSecondUploadIsRefusedBeforeItsBodyIsRead` in `apisystemgeoip_test.go`:
3930 - start PUT #1 with an `io.Pipe` body that writes 1 KiB and then blocks;
3931 - wait until the handler holds the mutex (poll `p.mu.TryLock()`, which fails while it's held);
3932 - send PUT #2 with a body that records whether `Read` was ever called;
3933 - PUT #2 must get 409, and its body's `Read` must never be called.
3934- `Commit(ctx, staged, opts) (geoip.Pointer, error)` runs §4.1 steps 3 to 6, then removes the temp dir and unlocks.
3935
3936The handler:
39371. calls `Stage` within the request;
39382. registers a job in `s.systemJobs` (read how `handleAPISystemUpdate`, `apisystemupdate.go:345`, creates and records one, and use the same type);
39393. runs `Commit` in a goroutine under a fresh `context.WithTimeout(context.Background(), 15*time.Minute)`, because the request's context dies with the 202;
39404. answers 202.
3941
3942**Test:** stand up the console behind a real backhaul (`newBackhaulScene`), or, if that harness can't host a console handler, an `httputil.ReverseProxy` with `Transport.ResponseHeaderTimeout = 50ms`. Delay `Commit` by 200ms through a test hook (a package var `geoipCommitDelay`). The PUT must get its 202 through the proxy, and polling the job must end in done with the pointer.
3943
3944The handler sketch below shows the gate, the deadline and the error mapping. Its synchronous `Publish` call becomes `Stage`, then job registration, then a 202, as described above. The `errGeoipRefused` mapping moves into the job's error message, and the CLI prints it.
3945
3946**The handlers** (`apisystemgeoip.go`):
3947
3948```go
3949func (s *Server) handleAPISystemGeoipPut(w http.ResponseWriter, r *http.Request) {
3950 if _, _, ok := s.apiSystemRoller(w, r); !ok {
3951 return
3952 }
3953 if !s.systemBucket(w) {
3954 return
3955 }
3956 // The server-wide ReadTimeout is 5 minutes (cmd/console/main.go); a
3957 // 1 GiB upload needs its own deadline, the way the ship door does it.
3958 rc := http.NewResponseController(w)
3959 _ = rc.SetReadDeadline(time.Now().Add(15 * time.Minute))
3960 defer rc.SetReadDeadline(time.Time{})
3961 body := http.MaxBytesReader(w, r.Body, geoip.MaxBlobBytes)
3962 ptr, err := s.geoip.Publish(r.Context(), body, geoipPublishOpts{Origin: "manual", Force: r.URL.Query().Get("force") == "1"})
3963 var mbe *http.MaxBytesError
3964 switch {
3965 case errors.As(err, &mbe):
3966 jsonError(w, http.StatusRequestEntityTooLarge, "the database is larger than 1 GiB")
3967 case errors.Is(err, errGeoipBusy):
3968 jsonError(w, http.StatusConflict, "another geoip publish is in progress. try again when it finishes")
3969 case errors.Is(err, errGeoipRefused):
3970 log.Printf("console: geoip publish refused: %v", err)
3971 jsonError(w, http.StatusUnprocessableEntity, err.Error())
3972 case err != nil:
3973 log.Printf("console: geoip publish: %v", err)
3974 jsonError(w, http.StatusBadGateway, "could not publish the database")
3975 default:
3976 s.auditSystem(r, "geoip publish "+ptr.SHA256[:8]) // match auditSystem's real signature (apisystemupdate.go:159)
3977 writeJSON(w, http.StatusOK, ptr)
3978 }
3979}
3980```
3981
3982`handleAPISystemGeoipGet` and `handleAPISystemGeoipDelete` follow the same gate. Build the GET's body from:
3983
3984- `s.geoip.Current`, plus `geoip.ExpiresAt`;
3985- `s.systemBoxes(ctx, false)`;
3986- a pure function `geoipVerdict(ptr *geoip.Pointer, boxes []status.Box, partial bool, now time.Time) (ready bool, blocking []string)`.
3987
3988Unit-test `geoipVerdict` directly with a table, as well as through the handler.
3989
3990The DELETE calls `s.geoip.Off`, then answers `feed_set: os.Getenv("CONSOLE_GEOIP_FEED") != ""`. Plan 2 replaces that with config.
3991
3992**Routes** (authcli.go, beside antivirus), and add the three patterns to `closedDoors`:
3993
3994```go
3995 s.cliRoute("PUT /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipPut)
3996 s.cliRoute("GET /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipGet)
3997 s.cliRoute("DELETE /api/cli/system/geoip", svcrole.Set{}, s.handleAPISystemGeoipDelete)
3998```
3999
4000Update the comment block above them to say the geoip doors use `apiSystemRoller`.
4001
4002**Staging directory** (spec §4.2, codex R6-2): `geoipTmpRoot(cfg)` is `filepath.Join(filepath.Dir(<CONSOLE_DB path>), "geoip-staging")`. `newGeoipPublisher` does two things at construction, before the server admits any request:
4003- `os.RemoveAll(tmpRoot)`;
4004- `os.MkdirAll(tmpRoot, 0o700)`.
4005
4006Add `TestPublisherStartupClearsAbandonedStaging` to `geoippublish_test.go`:
4007- create `tmpRoot/geoip-publish-123/db.mmdb` by hand, standing in for a crash mid-stage;
4008- construct a publisher on that root, and check the directory is now empty;
4009- check that a `Publish` then succeeds.
4010
4011**Retirement instruction** (spec §5.5, codex R6 on R1-16): an expected box whose heartbeat is stale gets a `blocking` entry of the form `"<label>: no heartbeat since <RFC3339 date>. if this box is retired, delete its edges record (control/edges/<label>.json)"`. That string is copy, so add it to Task 13's list. Also file the discussion:
4012
4013```bash
4014amadan discuss new carlos/platform -branch geoip-for-apps -title "A carlos system verb to retire a box (delete its edges record)" -body - <<'BODY'
4015Nothing deletes a retired box's control/edges/<label>.json (instancesplacement.go says so). The geoip rollout verdict (geoip spec §5.5) blocks on an expected box with a stale heartbeat and tells the operator to delete the record by hand. A first-class verb would also stop the home map backhauling to a dead box. Filed rather than worked around.
4016BODY
4017```
4018
4019**Server wiring:**
4020
4021- Where `Server` is constructed and `cfg.Releases` is known, set:
4022 `s.geoip = newGeoipPublisher(cfg.Releases, geoipTmpRoot(cfg))`.
4023 `geoipTmpRoot` is `filepath.Dir(<the console DB path>)`. Find how the server learns `CONSOLE_DB`, falling back to `os.TempDir()`. The temp root must be on disk, not tmpfs, because the upload can be 1 GiB.
4024- If `cfg.Releases` is nil, `s.geoip` stays nil, and the handlers' `systemBucket` check already answers 503 first.
4025- Start a goroutine from the same place the other console background loops start (`grep -n "go s\.\|StartAlertWatcher" cmd/console/main.go internal/console/server.go`):
4026
4027```go
4028// StartGeoipJobs runs the orphan sweep and D1 expiry at startup and daily
4029// (geoip spec §3, D1).
4030func (s *Server) StartGeoipJobs(ctx context.Context) {
4031 if s.geoip == nil {
4032 return
4033 }
4034 go func() {
4035 for {
4036 if err := s.geoip.Sweep(ctx); err != nil {
4037 log.Printf("console: geoip sweep: %v", err)
4038 }
4039 if _, err := s.geoip.ExpireIfDue(ctx, time.Now()); err != nil {
4040 log.Printf("console: geoip expiry check: %v", err)
4041 }
4042 select {
4043 case <-ctx.Done():
4044 return
4045 case <-time.After(24*time.Hour + rand.N(time.Hour)):
4046 }
4047 }
4048 }()
4049}
4050```
4051
4052**The CLI** (`system_geoip.go`, package main):
4053
4054- Add `"geoip"` to the verb allowlist at `system.go:50`, and dispatch it to `runSystemGeoipVerb(rest)`, the way `antivirus` dispatches.
4055- Use `requireDeployAPISession` (the session `carlos system status` uses).
4056- `publish` opens the file, `Stat`s it, and does a `PUT` with the `*os.File` as the body and `ContentLength` set. On a 202 it polls `GET /api/cli/system/jobs/{id}` every 2 seconds, the way `carlos system update` polls its job (find that loop in `system.go` and reuse it), until `done`, then prints the pointer or the error. Add a session method next to `doMultipartMethod` (auth.go:496) that sends a raw streamed body with a zero client timeout, like `shipHTTPClientTimeout`.
4057- **Output copy, recorded for Task 13:**
4058 - publish:
4059 ```
4060 published <sha8>: <type>, built <date>. every edge picks it up within 11 minutes
4061 ```
4062 - status: the pointer block, one line per box, then
4063 ```
4064 ready: every box strips forged headers and has the current database
4065 ```
4066 or
4067 ```
4068 not ready: <n> boxes in the way
4069 ```
4070 followed by the list;
4071 - off:
4072 ```
4073 geoip is off. edges drop the database within 11 minutes
4074 ```
4075 plus, when the feed is set:
4076 ```
4077 the feed is still set and will publish again within a day. unset CONSOLE_GEOIP_FEED to keep it off
4078 ```
4079
4080- [ ] **Step 4: Run the tests**
4081
4082Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -count=1` (the whole package, because `closedDoors` is package-wide), then `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test . -run 'System' -count=1`
4083Expected: PASS.
4084
4085- [ ] **Step 5: Commit**
4086
4087```bash
4088git add internal/console/apisystemgeoip.go internal/console/apisystemgeoip_test.go internal/console/authcli.go internal/console/svcroutes_test.go internal/console/server.go cmd/console/main.go system.go system_geoip.go system_geoip_test.go apimode.go auth.go
4089git commit -m "console+cli: carlos system geoip publish/status/off; daily sweep and D1 expiry (geoip spec §4.2, §5.5)"
4090```
4091
4092---
4093
4094### Task 13: Countries on the analytics page, docs, and the copy review
4095
4096**Files:**
4097- Modify: `internal/console/analyticsview.go`, `internal/console/templates/app_analytics.html`, `docs/site/cli.md`, `README.md`
4098- Test: the existing analytics view test file (`grep -ln "app_analytics" internal/console/*_test.go`)
4099
4100**Interfaces:**
4101- Consumes: `Summary.Countries/GeoSources` (Task 11), `geoip.AttributionFor` (Task 5), `topWithOther` (Task 11).
4102
4103- [ ] **Step 1: Gather every new user-facing string from Tasks 11-13 and run the `copy-review` skill on them** before writing them into templates or docs. The strings:
4104
4105 - **CLI** (Task 11): the `Countries` table heading.
4106 - **CLI** (Task 12): the publish, status, verdict and off lines, and every `jsonError` message in `apisystemgeoip.go`.
4107 - **Console page:**
4108 - the heading `Countries`;
4109 - the empty state `no country data for this period`;
4110 - the `_unknown` row label `unknown`;
4111 - the `_other` row label `other`;
4112 - each attribution line from `geoip.AttributionFor`, rendered as a link.
4113 - **Docs:** the new `docs/site/cli.md` section "Headers your app receives", and the `carlos system geoip` subsection under `### carlos system`. The README operator paragraph goes next to `CONSOLE_GEOIP_DB` (README.md:560).
4114
4115 The docs content follows spec §13. App docs:
4116 - the five headers;
4117 - why each may be absent;
4118 - `Vary` and the 8-variant cache cost;
4119 - that coordinates are an area;
4120 - the attribution duty, by source.
4121
4122 Operator docs:
4123 - publish, status and off;
4124 - the rollout order from §15, including "roll every box before publishing";
4125 - "the operator owns licence compliance", with links to DB-IP's licence page and MaxMind's GeoLite EULA;
4126 - D1;
4127 - the lifecycle rule (Task 1) being required before a GeoLite2 publish.
4128
4129 **The skill is a gate:** apply Paul's edits verbatim. If this runs while Paul is away, launch the review, commit everything else, and leave the task at "awaiting copy review" in the branch tasks and the handoff. **Do not write unreviewed copy into the files.**
4130
4131- [ ] **Step 2: Write the failing test** for the page:
4132 - a summary with `Countries{"GB":3,"_unknown":1}` and `GeoSources{"DBIP-City-Lite":4}` renders a Countries table containing `GB`, the unknown label and the DB-IP attribution link;
4133 - a summary with no countries renders the empty-state string;
4134 - a summary with two sources renders both attributions.
4135
4136- [ ] **Step 3: Implement it.**
4137 - In `analyticsview.go`, pass to the template:
4138 - `Countries` (through `topWithOther(sum.Countries, 10)`, with `_unknown` and `_other` relabelled by the approved strings);
4139 - `GeoAttributions []geoip.Attribution`: for each key of `sum.GeoSources` in sorted order whose `AttributionFor` answers, de-duplicated by text.
4140 - In `app_analytics.html`, after the Named events block:
4141
4142```html
4143<h3>{{/* approved heading */}}</h3>
4144{{if .Countries}}{{template "kvtable" .Countries}}{{else}}<p class="small">{{/* approved empty state */}}</p>{{end}}
4145{{range .GeoAttributions}}<p class="small"><a href="{{.URL}}">{{.Text}}</a></p>{{end}}
4146```
4147
4148 Then write the approved docs text into `docs/site/cli.md` and `README.md`.
4149
4150- [ ] **Step 4: Run the tests**
4151
4152Run: `TMPDIR=/tmp/g GOFLAGS=-buildvcs=false nice -n 19 go test ./internal/console -run Analytics -count=1`
4153Expected: PASS.
4154
4155- [ ] **Step 5: File the skills discussion and commit**
4156
4157```bash
4158amadan discuss new carlos/platform -branch geoip-for-apps -title "Agent skills: pick up 'Headers your app receives' (GeoIP headers)" -body - <<'EOF'
4159App skills are served from carlosframework.com, outside this repo. docs/site/cli.md now documents the X-Carlos-Geo-* headers (geoip spec §7, §13), including the attribution duty. The skill text apps' agents read should pick that section up once it is published.
4160EOF
4161git add internal/console/analyticsview.go internal/console/templates/app_analytics.html internal/console/*analytics*_test.go docs/site/cli.md README.md
4162git commit -m "console+docs: countries on the analytics page with attribution; headers and operator docs (geoip spec §9, §13)"
4163```
4164
4165---
4166
4167## Final gate (after Task 13)
4168
4169- [ ] `gofmt -l .` prints nothing. Run `go vet ./...`.
4170- [ ] Cross-compile: `GOOS=darwin GOARCH=arm64 go build ./...` and `GOOS=windows GOARCH=amd64 go build ./...`.
4171- [ ] The full gate, per memory `gate-tmp-exec-wrapper` (outside the sandbox, with the `-exec` wrapper), `nice -n 19`.
4172- [ ] Push, then check `amadan ci status carlos/platform -branch geoip-for-apps`.
4173- [ ] A whole-branch review by Astra (codex) against the spec, from a committed brief, before merge.
4174
4175## Not in this plan
4176
4177- **Plan 2:** the feed and the `geoip` box alert rule.
4178- **Plan 3:** the console's mirrored reader, `/geo`, and `carlos geo`.
4179- **Rolling, and applying Task 1's tofu:** Paul's go-ahead.
4180
4181---
4182
4183## Review dispositions (Astra plan round 1, `CODEX-FINDINGS-GEOIP-PLAN1-R1.md`)
4184
4185The plan was executed natively, and each finding was fixed in the code as it was built. Where the code and the plan text above differ, the code and its tests on the branch are authoritative.
4186
4187| # | Disposition |
4188|---|---|
4189| 1 | Fixed: `geoip.StripAll` removes the whole `X-Carlos-Geo-` namespace, case-insensitively (b00c58bd); the edge forgery matrix sends `X-Carlos-Geo-City` |
4190| 2 | Fixed: the verdict uses `LoadBoxesComplete` on every prefix, reconciled against edges records and active fleet boxes, keyed by (namespace, label), with `GeneratedAt` |
4191| 3 | Fixed: `.incoming-*` is cleaned only at boot; test `TestMirrorExpiryDuringADownloadKeepsTheDownload`, which was checked by mutation |
4192| 4 | Fixed: a sync failure after the rename is logged, not returned; test `TestMirrorSyncFailureAfterRenameKeepsTheGeneration` |
4193| 5 | Fixed: GeoIP is booted and its loop started before `ready(e)` and before the dev-mode return |
4194| 6 | Fixed: a blob CAS conflict streams the existing object and checks sha+size; test `TestPublishRefusesAWrongObjectAtTheBlobKey` |
4195| 7 | Fixed: `geoip.ErrUnknownVersion` means off; test `TestMirrorUnknownPointerVersionTurnsOff` |
4196| 8 | Fixed: rejections reset whenever the observed pointer changes, including on removal; test `TestMirrorRetriesARejectedShaAfterThePointerMoves` |
4197| 9 | Fixed: `DirStore.PutFrom` uses Stat for create-only and a streamed hash for replace-only |
4198| 10 | Fixed: the helper is named `geoipCtxReader` |
4199| 11 | Fixed: `n > limit` is refused |
4200| 12 | Fixed: `res.Countries` |
4201| 13 | Moot under native execution; every commit was built and tested from the working tree, and CI checks the pushed tip |
4202| 14 | Fixed: the PUT answers 202, a refusal is the job's error, and a member gets the bare 404. Service credentials get the closed-door 403 (closedDoors) |
4203| 15 | Fixed: `systemJob.Result`, returned by the jobs door, read by the CLI |
4204| 16 | Moot: D2 (rev 7) made the tunnel mark transport-independent, so no TLS seam was needed |
4205| 17 | Fixed: one consistent rule. The remote makes zero observations, the hub one per served response, with the hub's answer (`TestTunnelledPageviewsAreCountedOnceAtTheHub`, including a hub cache hit) |
4206| 18 | Fixed: `TestForgeryMatrixAcrossLocalProxyTypes` (unix, tcp, multi, with Connection nomination and unknown suffixes); backhaul and tunnel delivery and miss tests; rejected hop; door |
4207| 19 | Fixed: a valid missing-generation sha; injected sync failure; concurrent expiry during download; off and newer publish during download |
4208| 20 | Fixed: the node count is re-inflated before the different-type case; foreign-pointer CAS and publish/off race tests added; the pointer is now read at Stage (a real bug the new test found) |
4209| 21 | Fixed: an upgrade test with its own header-reporting backend |
4210| 22 | **Outstanding:** GeoLite2 City was not measured (it needs a MaxMind account). See FACTS.md |
4211