amadan
geoip plan 2: the feed and the geoip alert
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+242
−0
docs/superpowers/plans/2026-10-07-platform-geoip-plan2.md+242 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-07-platform-geoip-plan2.md b/docs/superpowers/plans/2026-10-07-platform-geoip-plan2.md| new file mode 100644 |
| index 000000000..6c649a60e |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-07-platform-geoip-plan2.md |
| @@ -0,0 +1,242 @@ |
| +# Platform GeoIP, plan 2: the feed and the `geoip` alert |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (this plan is executed natively, as plan 1 was). Steps use checkbox (`- [ ]`) syntax. |
| + |
| +**Goal:** The console keeps the published database current by itself, fetching DB-IP Lite or GeoLite2. Every box raises a `geoip` alert, through the existing box-alert pipeline, when its database is stale, expiring, expired or not loaded, or (on the console's box only) when the feed keeps failing. |
| + |
| +**Architecture:** |
| +- **The fetcher** lives in the console (`internal/console/geoipfeed.go`). It is the one control-plane process, so there is only one fetcher. It reuses plan 1's publish slot (`geoipPublisher.Stage`/`Commit`), adding two feed-only rules: the type must match the configured feed, and an edition must be newer than the last one from the same feed. |
| +- **Archives** are opened by pure, capped readers in `internal/geoip/archive.go`. |
| +- **The feed's state** goes to `control/geoip/feed.json` after every check, naming the console's box. |
| +- **The alert** is a fifth box rule beside disk, mem, cpu and lag. Each mirror computes a reason code. `metrics.Evaluate` applies its sustain, and the host's alert sink and the console's alert watcher deliver it, unchanged. |
| + |
| +**Tech Stack:** Go stdlib (`compress/gzip`, `archive/tar`, `net/http`), plan 1's packages. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-06-platform-geoip-design.md`, §10 (rev 11 or later). Plan 1 is built on this branch; its code is the base. |
| + |
| +## Global Constraints |
| + |
| +- **Console env:** |
| + - `CONSOLE_GEOIP_FEED`: one of `dbip-city-lite`, `dbip-country-lite`, `geolite2-city`, `geolite2-country`; |
| + - `CONSOLE_GEOIP_MAXMIND_ACCOUNT` and `CONSOLE_GEOIP_MAXMIND_KEY`. |
| + - An unknown name, or GeoLite2 without both credentials, is logged once at startup, and the feed stays off. |
| +- **Feed name to type:** |
| + - `dbip-city-lite` → `DBIP-City-Lite` |
| + - `dbip-country-lite` → `DBIP-Country-Lite` |
| + - `geolite2-city` → `GeoLite2-City` |
| + - `geolite2-country` → `GeoLite2-Country` |
| +- **URLs:** |
| + - DB-IP: `https://download.db-ip.com/free/dbip-<city|country>-lite-YYYY-MM.mmdb.gz` |
| + - MaxMind: `https://download.maxmind.com/geoip/databases/GeoLite2-<City|Country>/download?suffix=tar.gz`, with `suffix=tar.gz.sha256` for the checksum. |
| +- **Caps:** |
| + - compressed download 512 MiB, enforced while copying; |
| + - all expanded tar bytes 1.5 GiB, skipped members included; |
| + - the `.mmdb` itself 1 GiB (plan 1's `copyCapped`); |
| + - checksum file 4 KiB; |
| + - `feed.json` 64 KiB; |
| + - each check under the publisher's 15-minute deadline. |
| +- **Redirects:** at most 3, `https` only. The Basic `Authorization` header is dropped on a cross-host redirect. The key never appears in a URL or a log line. |
| +- **Schedule:** a first check 5 minutes after start, then daily with up to an hour of jitter. |
| +- **Seven** consecutive 404/410 answers raise the alert's feed reason. |
| +- **Alert codes:** the rule's `Value` is the reason code; 0 = none. |
| + |
| + | Code | Reason | |
| + |---|---| |
| + | 1 | loaded database older than 45 days | |
| + | 2 | GeoLite2 within 9 days of D1 expiry (21 days old) | |
| + | 3 | the pointer is expired, marked or by the clock | |
| + | 4 | a live pointer has not been loaded for over an hour | |
| + | 5 | the feed has failed 7 times in a row (console box only) | |
| + |
| + When several hold at once, the most severe wins: 3, then 4, then 2, then 1, then 5. |
| +- **The rule** fires after the reason has held for 30 minutes (`Rules.GeoIPSustain`). It clears at once when the code returns to 0. |
| +- **Copy:** every new alert sentence and log line follows the house voice (no em dashes). The alert sentences join plan 1's copy review. |
| + |
| +## Review Focus |
| + |
| +1. **A feed that serves a different product, an older edition, or a truncated file** must leave the published database untouched. |
| +2. **A redirect** to another host must never carry the MaxMind credentials. |
| +3. **A decompression bomb, or a tar padded with junk members,** must stop at the caps without filling the disk. |
| +4. **The feed reason** must fire on exactly one box, the console's, and must still fire on a fresh deployment where the feed has never succeeded. |
| +5. **`off`** must clear every pointer-related reason; expiry must not. |
| + |
| +--- |
| + |
| +### Task 1: `internal/geoip/archive.go` and `feed.go` |
| + |
| +**Files:** Create `internal/geoip/archive.go`, `internal/geoip/feed.go`, `internal/geoip/archive_test.go`, `internal/geoip/feed_test.go`. |
| + |
| +**Produces:** |
| + |
| +```go |
| +// feed.go |
| +type FeedSpec struct{ Name, Type, Provider string } // Provider: "dbip" | "maxmind" |
| +func ParseFeed(name string) (FeedSpec, bool) |
| +func DBIPURL(spec FeedSpec, month time.Time) string |
| +func MaxMindURL(spec FeedSpec, suffix string) string |
| +type FeedStatus struct { |
| + V int `json:"v"` |
| + Feed string `json:"feed"` |
| + Box string `json:"box"` // the console's box label: the one box that alerts on feed failure |
| + LastCheck time.Time `json:"last_check"` |
| + LastSuccess time.Time `json:"last_success,omitempty"` |
| + LastEdition string `json:"last_edition,omitempty"` |
| + ConsecutiveFailures int `json:"consecutive_failures"` |
| + LastFailure string `json:"last_failure,omitempty"` // a fixed category, never an error text |
| + ConsecutiveNotFound int `json:"consecutive_not_found"` |
| +} |
| +const FeedNotFoundAlert = 7 |
| +func ParseFeedStatus(raw []byte) (FeedStatus, error) |
| +func (s FeedStatus) Failing() bool // ConsecutiveFailures >= 7 || ConsecutiveNotFound >= 7 |
| + |
| +// archive.go |
| +var ErrTooLarge = errors.New("geoip archive: larger than its cap") |
| +func CappedReader(r io.Reader, max int64) io.Reader // returns ErrTooLarge once more than max bytes are read |
| +func GunzipReader(r io.Reader, maxExpanded int64) (io.Reader, error) |
| +func TarGzMMDB(r io.Reader, maxExpanded int64) (io.Reader, error) // the first regular *.mmdb member; every expanded byte counts |
| +``` |
| + |
| +- [ ] **Step 1: Tests.** |
| + - `ParseFeed` returns the four names with their types and providers, and rejects anything else. |
| + - `DBIPURL` for `dbip-city-lite` in 2026-10 is `https://download.db-ip.com/free/dbip-city-lite-2026-10.mmdb.gz`. |
| + - `CappedReader` errors with `ErrTooLarge` at max+1 and passes exactly max. |
| + - `GunzipReader` on a 10 MiB run of zeros gzipped, with a cap of 1 MiB, gives `ErrTooLarge` after reading about 1 MiB, never the full expansion. |
| + - `TarGzMMDB` returns the bytes of `db.mmdb` from an archive whose first member is a 2 MiB `README`, and errors with `ErrTooLarge` when the cap is 1 MiB. |
| + - `TarGzMMDB` on an archive with no `.mmdb` member gives a clear error. |
| + - `FeedStatus` round-trips through JSON. |
| + - `Failing` crosses at exactly 7. |
| + - Build the test archives in memory with `archive/tar` and `compress/gzip`. |
| +- [ ] **Step 2: Implement.** |
| + - `CappedReader` counts bytes, and once the count exceeds max it returns `ErrTooLarge`. |
| + - `GunzipReader` is `CappedReader(gzip.NewReader(r), maxExpanded)`. |
| + - `TarGzMMDB` wraps the gunzipped stream in one `CappedReader(…, maxExpanded)`, so skipped members count. It walks `tar.NewReader` to the first `TypeReg` member whose name ends `.mmdb` (path base) and returns that member's reader. |
| +- [ ] **Step 3:** `go test ./internal/geoip`, then commit. |
| + |
| +### Task 2: publish rules for the feed |
| + |
| +**Files:** Modify `internal/console/geoippublish.go`, `geoippublish_test.go`. |
| + |
| +**Produces:** `geoipPublishOpts` gains `RequireType string` (refuse unless `db.Source()` equals it) and `Monotonic bool`. With `Monotonic` set, the publish is refused when the current pointer has the same `Origin` and the candidate's `BuildEpoch` is not after the current one's. The feed sets both and never sets `Force`. |
| + |
| +- [ ] **Step 1: Tests.** |
| + - `RequireType: "GeoIP2-Country"` refuses the City fixture with the category "the feed returned a GeoIP2-City database, not GeoIP2-Country". |
| + - With `Monotonic`, `Origin: "feed:x"`, publishing the City fixture twice refuses the second as "not newer than the published edition". The pointer is unchanged. |
| + - With `Monotonic` against a current pointer whose `Origin` is `manual`, the publish proceeds. |
| +- [ ] **Step 2: Implement it in `Commit`,** after the country probe and before the shrink check. |
| +- [ ] **Step 3:** Run the tests, then commit. |
| + |
| +### Task 3: the feed |
| + |
| +**Files:** Create `internal/console/geoipfeed.go` and `geoipfeed_test.go`. Modify `internal/console/server.go` (Config `GeoipFeed`, `GeoipMaxMindAccount`, `GeoipMaxMindKey` and `BoxLabel`; start the feed from `StartGeoipJobs`), `cmd/console/main.go` (read the env; `BoxLabel: status.Label(<machine id>)`, reading `/etc/machine-id` the way `internal/host/probes.go` does). |
| + |
| +**Produces:** |
| + |
| +```go |
| +type geoipFeeder struct { |
| + pub *geoipPublisher |
| + store release.Store |
| + spec geoip.FeedSpec |
| + account, key string |
| + box string |
| + client *http.Client // CheckRedirect: geoipFeedRedirect |
| + now func() time.Time |
| + dbipBase, maxmindBase string // test seams; "" = the real hosts |
| +} |
| +func newGeoipFeeder(cfg Config, pub *geoipPublisher) (*geoipFeeder, bool) // false = no feed (unset or misconfigured, logged once) |
| +func (f *geoipFeeder) Check(ctx context.Context) error // one check; always writes feed.json |
| +func (f *geoipFeeder) run(ctx context.Context) |
| +func geoipFeedRedirect(req *http.Request, via []*http.Request) error // ≤3, https only, drops Authorization cross-host |
| +``` |
| + |
| +`Check`: |
| +1. Read the pointer and `feed.json` (capped). |
| +2. **DB-IP.** |
| + - The edition is this month's `YYYY-MM`. If the pointer's `Origin` is this feed and its `Edition` is this month, the check is done (a success, nothing to do). |
| + - Otherwise GET this month's URL, and on a 404 last month's. Only when there is no pointer, or the pointer came from elsewhere, is last month tried, because a pointer from this feed already holds last month. |
| + - A 404 or 410 on every candidate is a "not out yet" check. `ConsecutiveNotFound` goes up by one; it is not a failure. |
| +3. **MaxMind.** |
| + - HEAD the archive URL with Basic auth. If `Last-Modified` equals the pointer's `Edition` (same feed origin), the check is done. |
| + - Otherwise GET `suffix=tar.gz.sha256`: the first 64 hex characters, capped at 4 KiB. |
| + - Then GET the archive. |
| +4. **Stream the body:** |
| + - `CappedReader(body, 512 MiB)`, teed into a sha256 hasher; |
| + - then `GunzipReader` (DB-IP) or `TarGzMMDB` (MaxMind), capped at 1.5 GiB; |
| + - then `pub.Stage`. `errGeoipBusy` skips this check; it is not a failure. |
| +5. **MaxMind checksum:** drain the rest of the archive through the tee (capped), and compare the archive's sha256 with the published one. On a mismatch, `Abandon` and record the failure "checksum". |
| +6. **Commit** with `geoipPublishOpts{Origin: "feed:" + name, Edition: edition, RequireType: spec.Type, Monotonic: true}`. |
| +7. **Record the outcome.** Success sets `LastSuccess` and `LastEdition`, and zeroes both counters. A refusal or error increments `ConsecutiveFailures` and sets `LastFailure` to a fixed category: |
| + - `download` / `not found` / `too large` / `checksum` / `invalid` / `wrong type` / `not newer` / `expired` / `store`. |
| +8. **Write `feed.json`** with `Box: f.box`, on every path. |
| + |
| +`run`: wait 5 minutes, then check; then wait 24 hours plus up to 1 hour of jitter, and check again. If `CONSOLE_GEOIP_FEED` is unset at startup, delete any `feed.json`, so a switched-off feed alerts nobody. |
| + |
| +- [ ] **Step 1: Tests,** against an `httptest.NewTLSServer` standing in for both providers. Its client is the server's client, wrapped with `geoipFeedRedirect`. Use `dbipBase`/`maxmindBase` pointing at it, and the gzip/tar.gz of the GeoIP2 fixtures. Feed types are matched through `RequireType`, so use a `FeedSpec{Type: "GeoIP2-City"}` built by hand: |
| + - **DB-IP:** a new edition is published, `feed.json` says success, and the edition is this month. A second check makes no GET at all, because it's the same month and the same origin. |
| + - **404 handling:** this month 404s and last month serves; with no pointer, last month is published. Every candidate 404s and `ConsecutiveNotFound` goes up by one each check; after 7, `FeedStatus.Failing()` is true. |
| + - **MaxMind:** |
| + - The HEAD with an unchanged `Last-Modified` makes no GET. |
| + - A changed one fetches `.sha256` and then the archive, and publishes. |
| + - A wrong `.sha256` records "checksum", and the pointer is unchanged. |
| + - The test server asserts that every request carries Basic auth, and that the key appears in no URL. |
| + - **Redirect:** the archive URL redirects to a second `httptest` TLS server, on another host. That server sees no `Authorization` header. A 4th redirect is refused. |
| + - **Bomb:** a gzip of 2 GiB of zeros (generate it streamed) is refused as "too large", with the staging directory empty afterwards. For speed, make the expanded cap a field set to 1 MiB in the test. |
| + - **A feed type mismatch** is refused as "wrong type", even with no pointer at all (the first publish). |
| + - **A busy slot** (`pub.mu` held) skips the check without counting a failure. |
| +- [ ] **Step 2:** Implement it. |
| +- [ ] **Step 3:** Run the tests, then commit. |
| + |
| +### Task 4: the mirror's alert code and `feed.json` |
| + |
| +**Files:** Modify `internal/edge/geoipmirror.go`, `internal/edge/geoipmirror_test.go`. |
| + |
| +**Produces:** |
| +- `geoMirror` tracks `wantSince` (when the pointer first named a sha that isn't loaded) and `pointerExpired`. It also reads `feed.json` (capped, best effort) on each poll, keeping `feedFailing` and `feedBox`. |
| +- `func (m *geoMirror) AlertCode(now time.Time, box string) float64` implements the priority above. |
| +- `func (e *Edge) GeoIPAlertCode(now time.Time) float64` uses the edge's box label (`WithBoxLabel`). It returns 0 when there is no mirror. |
| + |
| +- [ ] **Step 1: Tests:** |
| + - each code from a constructed state; |
| + - code 4 only after an hour; |
| + - code 5 only when `feed.json`'s box equals the edge's label, and with no pointer at all; |
| + - code 3 for an expired pointer; |
| + - `off` (no pointer) gives 0 even with a feed failing elsewhere; |
| + - the priority when 2 and 4 both hold. |
| +- [ ] **Step 2:** Implement it. |
| +- [ ] **Step 3:** Run the tests, then commit. |
| + |
| +### Task 5: the `geoip` rule |
| + |
| +**Files:** Modify: |
| +- `internal/metrics/alert.go`: `Observation.GeoIP *float64`; `Rules.GeoIPSustain` (default 30 minutes); `evalFrac("geoip", obs.GeoIP, 0.5, r.GeoIPSustain)`; |
| +- `internal/host/metrics_loop.go`: fill `obs.GeoIP` from a published source, the same pattern as `geoIPSource`; |
| +- `internal/host/agent.go`: publish `e.GeoIPAlertCode`; |
| +- `internal/host/alertsink.go`: the `geoip` cases. |
| + |
| +Alert sentences, sent to copy review with plan 1's batch: |
| + |
| +| Code | Firing | Clearing | |
| +|---|---|---| |
| +| 1 | `the geoip database on %s is more than 45 days old` | `geoip on %s is back to normal` | |
| +| 2 | `the GeoLite2 database on %s expires within 9 days` | the same | |
| +| 3 | `the geoip database on %s has expired and was removed` | the same | |
| +| 4 | `%s has not loaded the published geoip database for over an hour` | the same | |
| +| 5 | `the geoip feed has failed 7 times in a row (seen from %s)` | the same | |
| + |
| +`alertLevel` has no level for `geoip`, and `alertSustain` returns `GeoIPSustain`. |
| + |
| +- [ ] **Step 1: Tests.** |
| + - `metrics.Evaluate` fires geoip after 30 minutes at code 1 and not before. It clears on the first 0. A nil reading changes nothing. |
| + - The alert sink renders each code's sentence. |
| + - The heartbeat-adjacent test: with no edge published, `obs.GeoIP` is nil. |
| +- [ ] **Step 2:** Implement it. |
| +- [ ] **Step 3:** Run the tests outside the sandbox for `internal/host`, then commit. |
| + |
| +### Task 6: feed status in `carlos system geoip status` |
| + |
| +**Files:** Modify `internal/console/apisystemgeoip.go` (add `feed` from `feed.json`: name, last check, last success, edition, failures, last failure category), `system_geoip.go` (print one line: `feed: <name>, last success <date>` or `feed: <name>, failing (<n> in a row, last: <category>)`), plus tests. |
| + |
| +- [ ] **Steps:** test, implement, then commit. |
| + |
| +### Final gate |
| + |
| +Run the full gate per memory, check the cross-compile, push, and wait for CI. Then an Astra code review of plan 2's diff. Add the new strings to the copy review. |