amadan

carlos / platform Public

console: the geoip feed (DB-IP and MaxMind, capped streams, checksum, redirect discipline, feed.json); spec rev 12 (Astra round 11)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 67e010eb06cfb32d10cb2154a0f9f2549074717d parent 40c8f3a
6 files changed, +709 −2
  • cmd/console/main.go +18 −0
  • docs/superpowers/specs/2026-10-06-platform-geoip-design.md +15 −2
  • docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R11.md +31 −0
  • internal/console/geoipfeed.go +349 −0
  • internal/console/geoipfeed_test.go +282 −0
  • internal/console/server.go +14 −0
diff --git a/cmd/console/main.go b/cmd/console/main.go
index 2da97bb5e..da6753b32 100644
--- a/cmd/console/main.go
+++ b/cmd/console/main.go
@@ -95,6 +95,7 @@ import (
"github.com/carlosframework/platform/internal/objstore"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/status"
"github.com/carlosframework/platform/internal/steering"
)
@@ -298,6 +299,11 @@ func serve() {
// Next to CONSOLE_DB, on disk (an upload can be 1 GiB), and used
// for nothing else: NewServer clears it (geoip spec §4.2).
GeoipStagingDir: filepath.Join(filepath.Dir(envOr("CONSOLE_DB", "/data/console/console.db")), "geoip-staging"),
+ // The feed (geoip spec §10.1). The key goes nowhere but MaxMind.
+ GeoipFeed: os.Getenv("CONSOLE_GEOIP_FEED"),
+ GeoipMaxMindAccount: strings.TrimSpace(os.Getenv("CONSOLE_GEOIP_MAXMIND_ACCOUNT")),
+ GeoipMaxMindKey: strings.TrimSpace(os.Getenv("CONSOLE_GEOIP_MAXMIND_KEY")),
+ BoxLabel: consoleBoxLabel(),
// The carlos db door's data bucket and this box's placement
// (dbDoorWiring).
HostClaim: hostClaim,
@@ -1267,3 +1273,15 @@ func dbDoorWiring() (bucket, region, prefix, placement string) {
func hostClaimEnabled(getenv func(string) string) bool {
return strings.TrimSpace(getenv("CARLOS_HOST_CLAIM")) != ""
}
+
+// consoleBoxLabel is the label this box's agent publishes its heartbeat
+// and alerts under (status.Label of /etc/machine-id, as internal/host does
+// for a platform box): the console names it in the geoip feed's status so
+// exactly one box, this one, alerts on feed failure.
+func consoleBoxLabel() string {
+ b, err := os.ReadFile("/etc/machine-id")
+ if err != nil {
+ return ""
+ }
+ return status.Label(strings.TrimSpace(string(b)))
+}
diff --git a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
index 3173235d0..c9ef59574 100644
--- a/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
+++ b/docs/superpowers/specs/2026-10-06-platform-geoip-design.md
@@ -1,7 +1,7 @@
# Platform GeoIP for apps: headers, a mirrored database, a lookup endpoint
**Date:** 2026-10-06
-**Revision:** 11
+**Revision:** 12
**Discussion:** amadan carlos/platform #241 (quoted in Appendix A)
**Branch:** `geoip-for-apps`
@@ -79,6 +79,10 @@
- the off bounds read 11.5 and 12.5 minutes (user copy rounds them up to
12 and 13);
- the verdict judges expiry by the clock.
+- Rev 12 answers Astra's round 11 (`CODEX-FINDINGS-GEOIP-R11.md`). The
+ alert's expired reason now matches §5.5 (marked, or past D1 by the
+ clock), so expiry never clears an alert while the bucket is still
+ unmarked.
## 0. Rulings and decisions
@@ -1026,7 +1030,10 @@ becomes a **box rule**, `geoip`, alongside `disk`, `mem`, `cpu` and `lag`.
`metrics.RuleState`'s existing sustain:
- the loaded database is older than 45 days, for any type;
- a GeoLite2 database is within 9 days of its D1 expiry (21 days old);
- - the pointer is marked expired (§3);
+ - the pointer is expired: marked (§3), **or past D1 by the clock**. The
+ mirror keeps the observed pointer's type and build after it drops the
+ reader, so expiry never clears the alert before the console's daily
+ pass marks the bucket (codex R11-1);
- a live pointer exists but this box hasn't loaded it for over an hour
(a bad blob or a full disk);
- **the feed reason** (codex R2-7), evaluated only by the console box's
@@ -1389,6 +1396,12 @@ Round 10 (`CODEX-FINDINGS-GEOIP-R10.md`): every finding accepted.
| R9-1 (partly) | | The verdict treats a pointer past D1 by the clock as expired (§5.5) |
| 1 | P2 | Accepted. Override expiry is in plan 1: a `Swappable`, refused at startup when expired, closed by a ticker (§16) |
+Round 11 (`CODEX-FINDINGS-GEOIP-R11.md`): accepted.
+
+| # | Sev | Disposition |
+|---|---|---|
+| 1 | P2 | Accepted. The alert's expired reason is marked or past D1 by the clock, using pointer metadata the mirror keeps after dropping the reader; tests cover delayed marking, restart, fresh publish and off (§10.4; plan 2 Task 4) |
+
## Appendix A. The request (#241, verbatim)
> **Expose the platform's GeoIP lookup to apps (headers and/or lookup endpoint)**
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R11.md b/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R11.md
new file mode 100644
index 000000000..2443e65d5
--- /dev/null
+++ b/docs/superpowers/specs/CODEX-FINDINGS-GEOIP-R11.md
@@ -0,0 +1,31 @@
+| Previous finding | Disposition | Evidence from revision 11 at `9146bd06` |
+|---|---|---|
+| R3-1. Expiry/install races | Resolved | §5.3 retains expiry checks before download and installation, with `genMu` serializing selection, expiry and cleanup. |
+| R3-2. Unprotected console `Locate` | Resolved | §5.4 requires locked `Swappable.Locate` and assigns the wrapper to `Config.GeoIP`. |
+| R3-3. Coordinate-only results | Resolved | §8.1a accepts coordinate-only hits; §9 counts their country as `_unknown`. |
+| R3-4. Heartbeat freshness | Resolved | §5.5 retains `status.BoxStale` and the jittered-heartbeat test. |
+| R1-4, carried forward: orphan destruction | Resolved | §3 retains startup and daily orphan sweeps, followed by §15’s noncurrent-version expiration. |
+| R1-16, carried forward: rollout verification | Resolved | §5.5 requires every expected box to satisfy readiness; unreadable heartbeats block through missing expected identities. |
+| R4-1. Duplicate tunnel analytics | Resolved | §§8.4 and 9 require zero remote observations and exactly one at the hub. |
+| R4-2. Publication deadline across backhaul | Resolved | §4.2 retains staging followed by a 202 response and a polled publication job. |
+| R5-1. Concurrent upload staging | Resolved | §4.2 reserves the mutex before reading the body and retains ownership through job cleanup. |
+| R6-1. Plaintext tunnel cache mismatch | Resolved | D2 and §§8–9, 16 consistently require the transport-independent tunnel mark. |
+| R6-2. Abandoned upload staging | Resolved | §4.2 retains the dedicated staging directory, startup cleanup and restart tests. |
+| R7-1. Publication of expired candidates | Resolved | §4.1 rejects expired candidates, rechecks before pointer replacement and forbids a force override. |
+| R7-2. Empty sanitized source | Resolved | §6 retains the valid fallback identifier `mmdb`. |
+| R7-3. Attribution lost through overflow | Resolved | §9 exempts attributed sources from folding into `_other`. |
+| R7-4. Interface discrepancies | Resolved | §§4.1 and 6 retain the corrected signatures and Unix-seconds conversion. |
+| R8-1. Box identity namespaces | Resolved | §5.5 requires `(namespace, label)` throughout inventory, heartbeat matching and blockers. |
+| R8-2. Shutdown propagation bound | Resolved | §4.2 now specifies 11.5 minutes for edges and 12.5 for the console, conditional on successful pointer reads. |
+| R8-3. Existing-blob verification | Resolved | §3 requires both SHA-256 and size to match before replacing the pointer. |
+| R9-1. Readiness without database convergence | Resolved | §5.5 now treats a pointer past D1’s deadline as expired even before it is marked; the status handler checks the clock. |
+| R9-2. `off` waiting behind publication | Resolved | §4.2 requires immediate 409 while an upload or publication job holds the slot. |
+| R10-1. Console-override expiry deferred past rollout | Resolved | §16 explicitly places protected override loading, startup expiry refusal and periodic closure in plan 1; `openGeoIP` implements that lifecycle independently of the deployment store. |
+
+1. **P2 — §§5.3, 10.4: expiry can clear the freshness alert before the console marks the pointer.** Consider a manually published GeoLite2 database with no feed. Its age alert is already firing when the edge’s expiry ticker closes the reader. The loaded-database reasons then disappear, but §10.4’s expired-pointer reason requires `"expired": true`. The console may not write that flag until its next daily maintenance pass. Following these predicates can therefore report recovery precisely when GeoIP becomes unavailable, contrary to the requirement that expiry never clears an alert. Treating the unmarked pointer as live merely substitutes the delayed missing-generation alert and still permits a clearing interval.
+
+ **Evidence:** [geoipmirror.go:431](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/edge/geoipmirror.go:431) expires the reader locally; [geoipmirror.go:225](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/edge/geoipmirror.go:225) clears its loaded metadata. Bucket expiry runs only every 24–25 hours at [server.go:1422](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/console/server.go:1422). The existing alert machinery clears on the first below-threshold reading at [alert.go:172](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/geoip-for-apps/internal/metrics/alert.go:172).
+
+ **Fix:** make the alert’s expiry predicate match §5.5: marked expired **or** past D1 by the clock. Retain the observed pointer’s metadata after dropping the reader. Require the existing alert to remain firing across that transition, and test delayed bucket marking, restart with an expired unmarked pointer, fresh publication and explicit `off`.
+
+Verdict: revise.
\ No newline at end of file
diff --git a/internal/console/geoipfeed.go b/internal/console/geoipfeed.go
new file mode 100644
index 000000000..1abeb6bf6
--- /dev/null
+++ b/internal/console/geoipfeed.go
@@ -0,0 +1,349 @@
+package console
+
+// The GeoIP feed (geoip spec §10): the console fetches new editions of
+// DB-IP Lite or GeoLite2 and publishes them through the same slot as
+// `carlos system geoip publish`, with two extra rules the feed may never
+// skip: the database's type must be the configured feed's, and an edition
+// must be newer than the last one this feed published.
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "io"
+ "log"
+ mrand "math/rand/v2"
+ "net/http"
+ "os"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+const (
+ geoipFeedMaxCompressed = int64(512 << 20)
+ geoipFeedMaxExpanded = int64(3 << 29) // 1.5 GiB
+ geoipFeedMaxChecksum = 4 << 10
+)
+
+type geoipFeeder struct {
+ pub *geoipPublisher
+ store release.Store
+ spec geoip.FeedSpec
+ account string
+ key string
+ box string
+ client *http.Client
+ now func() time.Time
+ maxExpanded int64
+ // Test seams; "" means the real hosts.
+ dbipBase, maxmindBase string
+}
+
+// newGeoipFeeder reads the feed configuration. false means no feed: unset,
+// or misconfigured (logged once; the console starts either way).
+func newGeoipFeeder(cfg Config, pub *geoipPublisher) (*geoipFeeder, bool) {
+ name := strings.TrimSpace(cfg.GeoipFeed)
+ if name == "" || pub == nil {
+ return nil, false
+ }
+ spec, ok := geoip.ParseFeed(name)
+ if !ok {
+ log.Printf("console: geoip feed misconfigured: CONSOLE_GEOIP_FEED %q is not one of dbip-city-lite, dbip-country-lite, geolite2-city, geolite2-country; the feed is off", name)
+ return nil, false
+ }
+ if spec.Provider == "maxmind" && (cfg.GeoipMaxMindAccount == "" || cfg.GeoipMaxMindKey == "") {
+ log.Printf("console: geoip feed misconfigured: %s needs CONSOLE_GEOIP_MAXMIND_ACCOUNT and CONSOLE_GEOIP_MAXMIND_KEY; the feed is off", name)
+ return nil, false
+ }
+ return &geoipFeeder{
+ pub: pub, store: pub.store, spec: spec,
+ account: cfg.GeoipMaxMindAccount, key: cfg.GeoipMaxMindKey, box: cfg.BoxLabel,
+ client: &http.Client{Timeout: 15 * time.Minute, CheckRedirect: geoipFeedRedirect},
+ now: time.Now,
+ maxExpanded: geoipFeedMaxExpanded,
+ }, true
+}
+
+// geoipFeedRedirect follows at most 3 redirects, https only, and drops the
+// Basic credentials on any cross-host hop (MaxMind redirects downloads to
+// presigned R2 URLs; spec §10.2).
+func geoipFeedRedirect(req *http.Request, via []*http.Request) error {
+ if len(via) > 3 {
+ return errors.New("geoip feed: too many redirects")
+ }
+ if req.URL.Scheme != "https" {
+ return errors.New("geoip feed: refusing a redirect away from https")
+ }
+ if len(via) > 0 && req.URL.Host != via[0].URL.Host {
+ req.Header.Del("Authorization")
+ }
+ return nil
+}
+
+// geoipFeedFailure is a check that failed, with a fixed category for
+// feed.json and the log (never an error text: it could carry a URL).
+type geoipFeedFailure struct{ category string }
+
+func (f geoipFeedFailure) Error() string { return "geoip feed: " + f.category }
+
+func feedFail(category string) error { return geoipFeedFailure{category} }
+
+var errFeedNotFound = errors.New("geoip feed: no edition available yet")
+
+// Check is one feed check. It always writes feed.json.
+func (f *geoipFeeder) Check(ctx context.Context) error {
+ st := f.readStatus(ctx)
+ st.V, st.Feed, st.Box, st.LastCheck = 1, f.spec.Name, f.box, f.now().UTC().Truncate(time.Second)
+ edition, err := f.check(ctx)
+ var ff geoipFeedFailure
+ switch {
+ case err == nil && edition != "":
+ st.LastSuccess, st.LastEdition = st.LastCheck, edition
+ st.ConsecutiveFailures, st.ConsecutiveNotFound, st.LastFailure = 0, 0, ""
+ case err == nil:
+ // Nothing new: a success with nothing to do.
+ st.ConsecutiveFailures, st.ConsecutiveNotFound, st.LastFailure = 0, 0, ""
+ case errors.Is(err, errGeoipBusy):
+ // Someone is publishing by hand: not a failure, try tomorrow.
+ case errors.Is(err, errFeedNotFound):
+ st.ConsecutiveNotFound++
+ st.LastFailure = "not found"
+ case errors.As(err, &ff):
+ st.ConsecutiveFailures++
+ st.LastFailure = ff.category
+ log.Printf("console: geoip feed %s: edition %s rejected: %s", f.spec.Name, orNone(edition), ff.category)
+ default:
+ st.ConsecutiveFailures++
+ st.LastFailure = "store"
+ log.Printf("console: geoip feed %s: check failed: store", f.spec.Name)
+ }
+ f.writeStatus(ctx, st)
+ return err
+}
+
+func orNone(s string) string {
+ if s == "" {
+ return "(none)"
+ }
+ return s
+}
+
+func (f *geoipFeeder) readStatus(ctx context.Context) geoip.FeedStatus {
+ rc, _, err := release.OpenObject(ctx, f.store, geoip.FeedKey)
+ if err != nil {
+ return geoip.FeedStatus{}
+ }
+ defer rc.Close()
+ b, err := io.ReadAll(io.LimitReader(rc, geoip.MaxPointerBytes+1))
+ if err != nil || len(b) > geoip.MaxPointerBytes {
+ return geoip.FeedStatus{}
+ }
+ s, err := geoip.ParseFeedStatus(b)
+ if err != nil || s.Feed != f.spec.Name {
+ return geoip.FeedStatus{} // another feed's history is not this one's
+ }
+ return s
+}
+
+func (f *geoipFeeder) writeStatus(ctx context.Context, st geoip.FeedStatus) {
+ b, _ := json.MarshalIndent(st, "", " ")
+ _, etag, err := f.store.Get(ctx, geoip.FeedKey)
+ if errors.Is(err, os.ErrNotExist) {
+ etag = ""
+ } else if err != nil {
+ log.Printf("console: geoip feed status unreadable; not written")
+ return
+ }
+ if _, err := f.store.Put(ctx, geoip.FeedKey, b, etag); err != nil {
+ log.Printf("console: geoip feed status not written")
+ }
+}
+
+// check returns the edition published ("" when there was nothing new).
+func (f *geoipFeeder) check(ctx context.Context) (string, error) {
+ cur, _, present, curErr := f.pub.Current(ctx)
+ ours := present && curErr == nil && cur.Origin == "feed:"+f.spec.Name
+ if f.spec.Provider == "dbip" {
+ return f.checkDBIP(ctx, cur, ours)
+ }
+ return f.checkMaxMind(ctx, cur, ours)
+}
+
+func (f *geoipFeeder) checkDBIP(ctx context.Context, cur geoip.Pointer, ours bool) (string, error) {
+ now := f.now().UTC()
+ this := time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC)
+ if ours && cur.Edition == this.Format("2006-01") {
+ return "", nil
+ }
+ months := []time.Time{this}
+ if !ours {
+ // No pointer of ours yet: last month is the best there is until
+ // this month's appears. A pointer of ours already holds it.
+ months = append(months, this.AddDate(0, -1, 0))
+ }
+ for _, m := range months {
+ res, err := f.get(ctx, geoip.DBIPURL(f.dbipBase, f.spec, m), false)
+ if err != nil {
+ return "", feedFail("download")
+ }
+ if res.StatusCode == http.StatusNotFound || res.StatusCode == http.StatusGone {
+ res.Body.Close()
+ continue
+ }
+ if res.StatusCode != http.StatusOK {
+ res.Body.Close()
+ return "", feedFail("download")
+ }
+ edition := m.Format("2006-01")
+ err = f.publish(ctx, res.Body, edition, "", func(r io.Reader) (io.Reader, error) {
+ return geoip.GunzipReader(r, f.maxExpanded)
+ })
+ res.Body.Close()
+ return edition, err
+ }
+ return "", errFeedNotFound
+}
+
+func (f *geoipFeeder) checkMaxMind(ctx context.Context, cur geoip.Pointer, ours bool) (string, error) {
+ head, err := f.request(ctx, http.MethodHead, geoip.MaxMindURL(f.maxmindBase, f.spec, "tar.gz"), true)
+ if err != nil {
+ return "", feedFail("download")
+ }
+ head.Body.Close()
+ if head.StatusCode == http.StatusNotFound || head.StatusCode == http.StatusGone {
+ return "", errFeedNotFound
+ }
+ if head.StatusCode != http.StatusOK {
+ return "", feedFail("download")
+ }
+ edition := head.Header.Get("Last-Modified")
+ if edition == "" {
+ return "", feedFail("download")
+ }
+ if ours && cur.Edition == edition {
+ return "", nil
+ }
+ sumRes, err := f.get(ctx, geoip.MaxMindURL(f.maxmindBase, f.spec, "tar.gz.sha256"), true)
+ if err != nil {
+ return edition, feedFail("download")
+ }
+ sumBody, err := io.ReadAll(io.LimitReader(sumRes.Body, geoipFeedMaxChecksum+1))
+ sumRes.Body.Close()
+ if err != nil || sumRes.StatusCode != http.StatusOK || len(sumBody) > geoipFeedMaxChecksum {
+ return edition, feedFail("checksum")
+ }
+ want := strings.ToLower(strings.TrimSpace(strings.SplitN(string(sumBody), " ", 2)[0]))
+ if !geoip.IsSHA256Hex(want) {
+ return edition, feedFail("checksum")
+ }
+ res, err := f.get(ctx, geoip.MaxMindURL(f.maxmindBase, f.spec, "tar.gz"), true)
+ if err != nil || res.StatusCode != http.StatusOK {
+ if res != nil {
+ res.Body.Close()
+ }
+ return edition, feedFail("download")
+ }
+ defer res.Body.Close()
+ return edition, f.publish(ctx, res.Body, edition, want, func(r io.Reader) (io.Reader, error) {
+ return geoip.TarGzMMDB(r, f.maxExpanded)
+ })
+}
+
+// publish streams body (capped, hashed) through open into the publish
+// slot. wantSHA, when set, is the archive's published SHA-256, checked
+// after the whole archive has been read and before anything is committed.
+func (f *geoipFeeder) publish(ctx context.Context, body io.Reader, edition, wantSHA string, open func(io.Reader) (io.Reader, error)) error {
+ h := sha256.New()
+ archive := io.TeeReader(geoip.CappedReader(body, geoipFeedMaxCompressed), h)
+ db, err := open(archive)
+ if err != nil {
+ return classifyFeedErr(err, "invalid")
+ }
+ st, err := f.pub.Stage(ctx, db)
+ if err != nil {
+ if errors.Is(err, errGeoipBusy) {
+ return err
+ }
+ return classifyFeedErr(err, "download")
+ }
+ if wantSHA != "" {
+ // The tar walk stopped at the database; the checksum covers the
+ // whole archive, so read the rest through the same capped tee.
+ if _, err := io.Copy(io.Discard, archive); err != nil {
+ f.pub.Abandon(st)
+ return classifyFeedErr(err, "download")
+ }
+ if hex.EncodeToString(h.Sum(nil)) != wantSHA {
+ f.pub.Abandon(st)
+ return feedFail("checksum")
+ }
+ }
+ _, err = f.pub.Commit(ctx, st, geoipPublishOpts{Origin: "feed:" + f.spec.Name, Edition: edition, RequireType: f.spec.Type, Monotonic: true})
+ if err == nil {
+ return nil
+ }
+ msg := err.Error()
+ switch {
+ case strings.Contains(msg, "not newer"):
+ return feedFail("not newer")
+ case strings.Contains(msg, "the feed returned"):
+ return feedFail("wrong type")
+ case strings.Contains(msg, "expired"):
+ return feedFail("expired")
+ case errors.Is(err, errGeoipRefused):
+ return feedFail("invalid")
+ }
+ return feedFail("store")
+}
+
+func classifyFeedErr(err error, fallback string) error {
+ if errors.Is(err, geoip.ErrTooLarge) || errors.Is(err, errGeoipRefused) && strings.Contains(err.Error(), "larger than") {
+ return feedFail("too large")
+ }
+ return feedFail(fallback)
+}
+
+func (f *geoipFeeder) get(ctx context.Context, url string, auth bool) (*http.Response, error) {
+ return f.request(ctx, http.MethodGet, url, auth)
+}
+
+func (f *geoipFeeder) request(ctx context.Context, method, url string, auth bool) (*http.Response, error) {
+ req, err := http.NewRequestWithContext(ctx, method, url, nil)
+ if err != nil {
+ return nil, err
+ }
+ if auth {
+ // Basic auth in a header, never in the URL (spec §10.1).
+ req.SetBasicAuth(f.account, f.key)
+ }
+ return f.client.Do(req)
+}
+
+// run is the feed's schedule: 5 minutes after start, then daily.
+func (f *geoipFeeder) run(ctx context.Context) {
+ wait := 5 * time.Minute
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-time.After(wait):
+ }
+ cctx, cancel := context.WithTimeout(ctx, f.pub.deadline)
+ f.Check(cctx)
+ cancel()
+ wait = 24*time.Hour + mrand.N(time.Hour)
+ }
+}
+
+// clearFeedStatus removes feed.json when no feed is configured, so a feed
+// switched off never alerts anyone.
+func clearFeedStatus(ctx context.Context, s release.Store) {
+ if err := s.Delete(ctx, geoip.FeedKey); err != nil && !errors.Is(err, os.ErrNotExist) {
+ log.Printf("console: geoip feed status could not be cleared")
+ }
+}
diff --git a/internal/console/geoipfeed_test.go b/internal/console/geoipfeed_test.go
new file mode 100644
index 000000000..094de712d
--- /dev/null
+++ b/internal/console/geoipfeed_test.go
@@ -0,0 +1,282 @@
+package console
+
+import (
+ "archive/tar"
+ "bytes"
+ "compress/gzip"
+ "context"
+ "crypto/sha256"
+ "crypto/tls"
+ "encoding/hex"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/geoip"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func gzipBytes(b []byte) []byte {
+ var buf bytes.Buffer
+ w := gzip.NewWriter(&buf)
+ w.Write(b)
+ w.Close()
+ return buf.Bytes()
+}
+
+func tarGzBytes(name string, body []byte) []byte {
+ var buf bytes.Buffer
+ zw := gzip.NewWriter(&buf)
+ tw := tar.NewWriter(zw)
+ tw.WriteHeader(&tar.Header{Name: "GeoLite2-City_20261006/COPYRIGHT.txt", Mode: 0o644, Size: 5, Typeflag: tar.TypeReg})
+ tw.Write([]byte("(c) x"))
+ tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg})
+ tw.Write(body)
+ tw.Close()
+ zw.Close()
+ return buf.Bytes()
+}
+
+var feedNow = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
+
+func newTestFeeder(t *testing.T, provider string, h http.Handler) (*geoipFeeder, release.Store) {
+ t.Helper()
+ srv := httptest.NewTLSServer(h)
+ t.Cleanup(srv.Close)
+ p, s := newTestPublisher(t)
+ f := &geoipFeeder{
+ pub: p, store: s, spec: geoip.FeedSpec{Name: "test-feed", Type: "GeoIP2-City", Provider: provider},
+ account: "acct", key: "s3cret-key", box: "console-box",
+ client: &http.Client{Timeout: time.Minute, CheckRedirect: geoipFeedRedirect,
+ Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}},
+ now: func() time.Time { return feedNow }, maxExpanded: geoipFeedMaxExpanded,
+ dbipBase: srv.URL, maxmindBase: srv.URL,
+ }
+ return f, s
+}
+
+func feedStatus(t *testing.T, s release.Store) geoip.FeedStatus {
+ t.Helper()
+ b, _, err := s.Get(context.Background(), geoip.FeedKey)
+ if err != nil {
+ t.Fatalf("feed.json: %v", err)
+ }
+ st, err := geoip.ParseFeedStatus(b)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return st
+}
+
+func TestFeedDBIPPublishesThisMonthThenRests(t *testing.T) {
+ withFixtureProbes(t)
+ body := gzipBytes(fixture(t, "GeoIP2-City-Test.mmdb"))
+ var gets atomic.Int32
+ f, s := newTestFeeder(t, "dbip", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ gets.Add(1)
+ if r.URL.Path != "/free/dbip-test-feed-lite-2026-10.mmdb.gz" && !strings.HasSuffix(r.URL.Path, "-2026-10.mmdb.gz") {
+ http.NotFound(w, r)
+ return
+ }
+ w.Write(body)
+ }))
+ if err := f.Check(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ cur, _, present, _ := f.pub.Current(context.Background())
+ if !present || cur.Origin != "feed:test-feed" || cur.Edition != "2026-10" {
+ t.Fatalf("pointer = %+v", cur)
+ }
+ st := feedStatus(t, s)
+ if st.LastEdition != "2026-10" || st.ConsecutiveFailures != 0 || st.Box != "console-box" {
+ t.Fatalf("feed.json = %+v", st)
+ }
+ before := gets.Load()
+ if err := f.Check(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if gets.Load() != before {
+ t.Fatal("a second check in the same month downloaded again")
+ }
+}
+
+func TestFeedDBIPFallsBackToLastMonthOnlyWithoutAPointerOfItsOwn(t *testing.T) {
+ withFixtureProbes(t)
+ body := gzipBytes(fixture(t, "GeoIP2-City-Test.mmdb"))
+ f, _ := newTestFeeder(t, "dbip", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if strings.HasSuffix(r.URL.Path, "-2026-09.mmdb.gz") {
+ w.Write(body)
+ return
+ }
+ http.NotFound(w, r)
+ }))
+ if err := f.Check(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if cur, _, _, _ := f.pub.Current(context.Background()); cur.Edition != "2026-09" {
+ t.Fatalf("edition = %q, want last month's", cur.Edition)
+ }
+}
+
+func TestFeedNotFoundCountsTowardTheAlert(t *testing.T) {
+ f, s := newTestFeeder(t, "dbip", http.NotFoundHandler())
+ for i := 1; i <= 7; i++ {
+ if err := f.Check(context.Background()); !errors.Is(err, errFeedNotFound) {
+ t.Fatalf("check %d = %v", i, err)
+ }
+ st := feedStatus(t, s)
+ if st.ConsecutiveNotFound != i || st.Failing() != (i >= 7) {
+ t.Fatalf("after %d checks feed.json = %+v", i, st)
+ }
+ }
+}
+
+func maxmindHandler(t *testing.T, archive []byte, sum string, lastModified string, gets *atomic.Int32) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ user, pass, ok := r.BasicAuth()
+ if !ok || user != "acct" || pass != "s3cret-key" {
+ http.Error(w, "no auth", http.StatusUnauthorized)
+ return
+ }
+ if strings.Contains(r.URL.String(), "s3cret") {
+ t.Error("the key appeared in a URL")
+ }
+ w.Header().Set("Last-Modified", lastModified)
+ if r.Method == http.MethodHead {
+ return
+ }
+ gets.Add(1)
+ switch r.URL.Query().Get("suffix") {
+ case "tar.gz.sha256":
+ w.Write([]byte(sum + " GeoLite2-City_20261006.tar.gz\n"))
+ case "tar.gz":
+ w.Write(archive)
+ default:
+ http.NotFound(w, r)
+ }
+ })
+}
+
+func TestFeedMaxMindHeadThenChecksumThenArchive(t *testing.T) {
+ withFixtureProbes(t)
+ archive := tarGzBytes("GeoLite2-City_20261006/GeoLite2-City.mmdb", fixture(t, "GeoIP2-City-Test.mmdb"))
+ sum := sha256.Sum256(archive)
+ var gets atomic.Int32
+ f, _ := newTestFeeder(t, "maxmind", maxmindHandler(t, archive, hex.EncodeToString(sum[:]), "Tue, 06 Oct 2026 00:00:00 GMT", &gets))
+ if err := f.Check(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ cur, _, _, _ := f.pub.Current(context.Background())
+ if cur.Edition != "Tue, 06 Oct 2026 00:00:00 GMT" {
+ t.Fatalf("edition = %q", cur.Edition)
+ }
+ before := gets.Load()
+ if err := f.Check(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if gets.Load() != before {
+ t.Fatal("an unchanged Last-Modified still downloaded")
+ }
+}
+
+func TestFeedMaxMindWrongChecksumPublishesNothing(t *testing.T) {
+ withFixtureProbes(t)
+ archive := tarGzBytes("x/GeoLite2-City.mmdb", fixture(t, "GeoIP2-City-Test.mmdb"))
+ var gets atomic.Int32
+ f, s := newTestFeeder(t, "maxmind", maxmindHandler(t, archive, strings.Repeat("0", 64), "Tue, 06 Oct 2026 00:00:00 GMT", &gets))
+ if err := f.Check(context.Background()); err == nil {
+ t.Fatal("a bad checksum published")
+ }
+ if _, _, present, _ := f.pub.Current(context.Background()); present {
+ t.Fatal("pointer written despite the checksum")
+ }
+ if st := feedStatus(t, s); st.LastFailure != "checksum" || st.ConsecutiveFailures != 1 {
+ t.Fatalf("feed.json = %+v", st)
+ }
+ if entries, _ := os.ReadDir(f.pub.tmpRoot); len(entries) != 0 {
+ t.Fatalf("staging left behind: %v", entries)
+ }
+}
+
+func TestFeedRedirectDropsCredentialsAcrossHosts(t *testing.T) {
+ var sawAuth atomic.Bool
+ other := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.Header.Get("Authorization") != "" {
+ sawAuth.Store(true)
+ }
+ w.WriteHeader(http.StatusTeapot)
+ }))
+ t.Cleanup(other.Close)
+ f, _ := newTestFeeder(t, "maxmind", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ http.Redirect(w, r, other.URL+"/presigned", http.StatusFound)
+ }))
+ res, err := f.get(context.Background(), f.maxmindBase+"/x", true)
+ if err != nil {
+ t.Fatal(err)
+ }
+ res.Body.Close()
+ if sawAuth.Load() {
+ t.Fatal("the MaxMind credentials followed a redirect to another host")
+ }
+ loop := 0
+ f2, _ := newTestFeeder(t, "maxmind", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ loop++
+ http.Redirect(w, r, "/again", http.StatusFound)
+ }))
+ if _, err := f2.get(context.Background(), f2.maxmindBase+"/x", true); err == nil {
+ t.Fatal("an endless redirect was followed")
+ }
+}
+
+func TestFeedRefusesADecompressionBomb(t *testing.T) {
+ var buf bytes.Buffer
+ zw := gzip.NewWriter(&buf)
+ zero := make([]byte, 1<<20)
+ for i := 0; i < 8; i++ {
+ zw.Write(zero)
+ }
+ zw.Close()
+ bomb := buf.Bytes()
+ f, s := newTestFeeder(t, "dbip", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write(bomb) }))
+ f.maxExpanded = 1 << 20
+ if err := f.Check(context.Background()); err == nil {
+ t.Fatal("a bomb was accepted")
+ }
+ if st := feedStatus(t, s); st.LastFailure != "too large" {
+ t.Fatalf("feed.json = %+v", st)
+ }
+ if entries, _ := os.ReadDir(f.pub.tmpRoot); len(entries) != 0 {
+ t.Fatalf("staging left behind: %v", entries)
+ }
+}
+
+func TestFeedRefusesTheWrongProductEvenFirstTime(t *testing.T) {
+ withFixtureProbes(t)
+ body := gzipBytes(fixture(t, "GeoIP2-Country-Test.mmdb")) // feed says City
+ f, s := newTestFeeder(t, "dbip", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
+ if err := f.Check(context.Background()); err == nil {
+ t.Fatal("a Country database was published by a City feed")
+ }
+ if st := feedStatus(t, s); st.LastFailure != "wrong type" {
+ t.Fatalf("feed.json = %+v", st)
+ }
+}
+
+func TestFeedSkipsWhileSomeoneIsPublishing(t *testing.T) {
+ body := gzipBytes(fixture(t, "GeoIP2-City-Test.mmdb"))
+ f, s := newTestFeeder(t, "dbip", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write(body) }))
+ f.pub.mu.Lock()
+ err := f.Check(context.Background())
+ f.pub.mu.Unlock()
+ if !errors.Is(err, errGeoipBusy) {
+ t.Fatalf("check while busy = %v", err)
+ }
+ if st := feedStatus(t, s); st.ConsecutiveFailures != 0 {
+ t.Fatalf("a busy slot counted as a failure: %+v", st)
+ }
+}
diff --git a/internal/console/server.go b/internal/console/server.go
index cae913f1b..b747cd8f9 100644
--- a/internal/console/server.go
+++ b/internal/console/server.go
@@ -502,6 +502,14 @@ type Config struct {
// crash never leaves an upload behind for good. "" = the OS temp dir,
// not cleared (tests).
GeoipStagingDir string
+ // The GeoIP feed (geoip spec §10.1): CONSOLE_GEOIP_FEED and, for
+ // GeoLite2, the MaxMind account and key. BoxLabel is this console's
+ // box, named in feed.json so only that box's mirror alerts on feed
+ // failure.
+ GeoipFeed string
+ GeoipMaxMindAccount string
+ GeoipMaxMindKey string
+ BoxLabel string
// HibernateBucket/Region/Prefix name the region data bucket the
// `carlos db` door reads (apidb.go; spec 2026-09-02 §6, Option A): the
// same CARLOS_HIBERNATE_* triple the activator on this box writes
@@ -1423,6 +1431,12 @@ func (s *Server) StartGeoipJobs(ctx context.Context) {
if s.geoip == nil {
return
}
+ if f, ok := newGeoipFeeder(s.cfg, s.geoip); ok {
+ log.Printf("console: geoip feed %s armed (first check in 5 minutes, then daily)", f.spec.Name)
+ go f.run(ctx)
+ } else {
+ clearFeedStatus(ctx, s.cfg.Releases)
+ }
go func() {
for {
if err := s.geoip.Sweep(ctx); err != nil {