amadan
plan: edge rate limit, eleven tasks (#358)
Summary
Implementation plan for the approved #358 design. New leaf package internal/ratelimit, the gate's record-to-edge path, plan ceilings, edge enforcement observe-by-default, door mark across hops, backhaul fair share, console doors, carlos ratelimit, settings page, docs. Copy drafts held for copy review before the tasks that write them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+427
−0
docs/superpowers/plans/2026-10-24-edge-rate-limit.md+427 −0
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (18 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (173 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 4.045s
ok github.com/carlosframework/platform/cmd/moveprobe 12.015s
ok github.com/carlosframework/platform/cmd/zddprobe 0.784s
ok github.com/carlosframework/platform/internal/activator 57.686s
ok github.com/carlosframework/platform/internal/agentsock 0.045s
ok github.com/carlosframework/platform/internal/alerting 0.006s
ok github.com/carlosframework/platform/internal/analytics 0.022s
ok github.com/carlosframework/platform/internal/appconfig 3.001s
ok github.com/carlosframework/platform/internal/apperr 0.124s
ok github.com/carlosframework/platform/internal/applog 0.260s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.005s
ok github.com/carlosframework/platform/internal/boxid 1.036s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.013s
ok github.com/carlosframework/platform/internal/caps 0.021s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.807s
ok github.com/carlosframework/platform/internal/changelog 0.535s
ok github.com/carlosframework/platform/internal/claimactivation 1.453s
ok github.com/carlosframework/platform/internal/clamd 0.072s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 8.355s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.023s
ok github.com/carlosframework/platform/internal/dbquery 0.006s
ok github.com/carlosframework/platform/internal/dbreads 2.212s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.011s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.018s
ok github.com/carlosframework/platform/internal/domains 2.934s
ok github.com/carlosframework/platform/internal/doorkey 0.013s
ok github.com/carlosframework/platform/internal/economics 0.009s
ok github.com/carlosframework/platform/internal/edge 45.663s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.133s
ok github.com/carlosframework/platform/internal/gate 1.655s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.030s
ok github.com/carlosframework/platform/internal/historyreadback 0.160s
ok github.com/carlosframework/platform/internal/host 143.269s
ok github.com/carlosframework/platform/internal/hostactors 6.111s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.008s
ok github.com/carlosframework/platform/internal/hostbox 3.085s
ok github.com/carlosframework/platform/internal/hostclaim 0.784s
ok github.com/carlosframework/platform/internal/hostenv 0.008s
ok github.com/carlosframework/platform/internal/hostevidence 0.007s
ok github.com/carlosframework/platform/internal/hostexec 5.341s
ok github.com/carlosframework/platform/internal/hostfence 0.064s
ok github.com/carlosframework/platform/internal/hostfiles 10.380s
ok github.com/carlosframework/platform/internal/hostfloor 0.021s
ok github.com/carlosframework/platform/internal/hostguard 1.084s
ok github.com/carlosframework/platform/internal/hostops 1.990s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.003s
ok github.com/carlosframework/platform/internal/hostquarantine 0.007s
ok github.com/carlosframework/platform/internal/hostrelocate 0.680s
ok github.com/carlosframework/platform/internal/hostresolve 0.233s
ok github.com/carlosframework/platform/internal/hostwrite 0.003s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.235s
ok github.com/carlosframework/platform/internal/instanceexec 2.844s
ok github.com/carlosframework/platform/internal/instancefs 0.006s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.015s
ok github.com/carlosframework/platform/internal/layout 1.793s
ok github.com/carlosframework/platform/internal/lbhealth 0.006s
ok github.com/carlosframework/platform/internal/lease 10.433s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.630s
ok github.com/carlosframework/platform/internal/ledger 0.024s
ok github.com/carlosframework/platform/internal/mail 0.216s
ok github.com/carlosframework/platform/internal/maintenance 0.089s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.591s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.103s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 22.478s
ok github.com/carlosframework/platform/internal/moves/archive 0.982s
ok github.com/carlosframework/platform/internal/objstore 0.322s
ok github.com/carlosframework/platform/internal/ops 0.278s
ok github.com/carlosframework/platform/internal/park 0.313s
ok github.com/carlosframework/platform/internal/plans 0.006s
ok github.com/carlosframework/platform/internal/privatefile 0.006s
ok github.com/carlosframework/platform/internal/provision 24.038s
ok github.com/carlosframework/platform/internal/publicrelease 0.016s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.727s
ok github.com/carlosframework/platform/internal/relay 7.247s
ok github.com/carlosframework/platform/internal/release 26.235s
ok github.com/carlosframework/platform/internal/rollout 3.846s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.010s
ok github.com/carlosframework/platform/internal/rootparticipation 0.065s
ok github.com/carlosframework/platform/internal/rootstate 0.030s
ok github.com/carlosframework/platform/internal/runs 0.560s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.007s
ok github.com/carlosframework/platform/internal/schedule 2.712s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/engines 2.594s
ok github.com/carlosframework/platform/internal/sim/hostlife 10.908s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.823s
ok github.com/carlosframework/platform/internal/status 0.079s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.406s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.008s
ok github.com/carlosframework/platform/internal/usage 0.010s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.672s
=== internal_console@0 (85 s)
ok github.com/carlosframework/platform/internal/console 82.868s
=== internal_console@1 (44 s)
ok github.com/carlosframework/platform/internal/console 42.405s
=== internal_console@2 (42 s)
ok github.com/carlosframework/platform/internal/console 40.143s
=== internal_console@3 (41 s)
ok github.com/carlosframework/platform/internal/console 39.442s
=== internal_console@4 (42 s)
ok github.com/carlosframework/platform/internal/console 40.399s
=== internal_console@5 (42 s)
ok github.com/carlosframework/platform/internal/console 39.649s
=== internal_console@6 (38 s)
ok github.com/carlosframework/platform/internal/console 35.773s
=== internal_console@7 (46 s)
ok github.com/carlosframework/platform/internal/console 43.747s
=== root@0 (110 s)
ok github.com/carlosframework/platform 107.814s
=== root@1 (84 s)
ok github.com/carlosframework/platform 83.346s
=== root@2 (92 s)
ok github.com/carlosframework/platform 90.762s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50659490 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.024s
=== RUN TestT4StorageProbeMinio
2026/10/24 20:34:08 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E19172BC463156, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.024s
=== RUN TestS3StoreMinioContract
=== RUN TestS3StoreMinioContract/conditional_get_and_etag_listing
=== RUN TestS3StoreMinioContract/bump_through_every_mutation_path
--- PASS: TestS3StoreMinioContract (0.03s)
--- PASS: TestS3StoreMinioContract/conditional_get_and_etag_listing (0.01s)
--- PASS: TestS3StoreMinioContract/bump_through_every_mutation_path (0.02s)
PASS
ok github.com/carlosframework/platform/internal/release 0.032s
Bucket created successfully `local/dbf-deploy`.
Bucket created successfully `local/dbf-data-ie`.
Bucket created successfully `local/dbf-data-uk`.
local/dbf-deploy versioning is enabled
Created policy `dbf-console` successfully.
Created policy `dbf-ukbox` successfully.
Added user `dbfconsole` successfully.
Added user `dbfukbox` successfully.
Attached Policies: [dbf-console]
To User: dbfconsole
Attached Policies: [dbf-ukbox]
To User: dbfukbox
=== RUN TestDBForwardMinio
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:34:11 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:34:11 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 20:34:11 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:34:11 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:34:11 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:34:11 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/24 20:34:11 console: could not notify member@example.com of a CLI token issue: keymail: lookup member@example.com: keymail: GET /api/federation/lookup?addr=member%40example.com&from=console%40127.0.0.1%3A43601: could not determine target mailbox
agent.go:135: dbreads: dbr_wvu5jfhi5kwtaoigzhxlkpyrfy: planned 3 files (1544 bytes) and restored to 3 in 10ms
agent.go:135: dbreads: dbr_wvu5jfhi5kwtaoigzhxlkpyrfy: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/24 20:34:12 console: db read dbr_wvu5jfhi5kwtaoigzhxlkpyrfy: relayed 1 chunks (102 bytes) in 1.004s, ended done
agent.go:135: dbreads: dbr_fi3m7ulbd34gxziopp3lw3yipa: planned 1 files (720 bytes) and restored to 1 in 8ms
agent.go:135: dbreads: dbr_fi3m7ulbd34gxziopp3lw3yipa: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/24 20:34:13 console: db read dbr_fi3m7ulbd34gxziopp3lw3yipa: relayed 1 chunks (102 bytes) in 1.004s, ended done
agent.go:135: dbreads: dbr_avdno4sp5faqmappvlx2etdmgq: planned 3 files (1544 bytes) and held through 3 in 9ms
agent.go:135: dbreads: dbr_avdno4sp5faqmappvlx2etdmgq: output 1544 bytes in 1 confirmed chunks in 1ms
2026/10/24 20:34:14 console: db read dbr_avdno4sp5faqmappvlx2etdmgq: relayed 1 chunks (1641 bytes) in 1.004s, ended done
=== RUN TestDBForwardMinio/by_disconnect
=== NAME TestDBForwardMinio
agent.go:135: dbreads: dbr_qts44r5dtwyga5spoebin35hbe: planned 3 files (1544 bytes) and restored to 3 in 9ms
2026/10/24 20:34:14 console: db read dbr_qts44r5dtwyga5spoebin35hbe: relayed 0 chunks (0 bytes) in 13ms, ended client gone
agent.go:135: dbreads: dbr_qts44r5dtwyga5spoebin35hbe: output 5 bytes in 1 confirmed chunks in 38ms
=== RUN TestDBForwardMinio/by_timeout
=== NAME TestDBForwardMinio
agent.go:135: dbreads: dbr_mwrmv67bzv5bm2xenrnqiklivy: planned 3 files (1544 bytes) and restored to 3 in 9ms
2026/10/24 20:34:34 console: db read dbr_mwrmv67bzv5bm2xenrnqiklivy: relayed 0 chunks (0 bytes) in 19.747s, ended no-result
agent.go:135: dbreads: dbr_mwrmv67bzv5bm2xenrnqiklivy: output 5 bytes in 1 confirmed chunks in 19.721s
--- PASS: TestDBForwardMinio (23.12s)
--- PASS: TestDBForwardMinio/by_disconnect (0.16s)
--- PASS: TestDBForwardMinio/by_timeout (19.86s)
PASS
ok github.com/carlosframework/platform/internal/console 23.140s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-24-edge-rate-limit.md b/docs/superpowers/plans/2026-10-24-edge-rate-limit.md| new file mode 100644 |
| index 000000000..dc998edb8 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-24-edge-rate-limit.md |
| @@ -0,0 +1,427 @@ |
| +# Edge Rate Limit Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** A per-client limit at the edge, set per instance, that sheds the heaviest clients only while an instance is busy, plus a high always-on backstop, with an optional plan ceiling. |
| + |
| +**Architecture:** A new leaf package `internal/ratelimit` owns the settings type, the defaults, the resolution against a plan ceiling, and the in-memory limiter. Settings travel the password gate's path (instance record → registry column → edge reload). The edge admits each request that would reach an instance just before the activator or proxy, holding a slot until response headers. Plans add three optional ceilings, and the console and the `carlos ratelimit` verb read and write it all. |
| + |
| +**Tech Stack:** Go (stdlib `net/netip`, `sync`, `net/http`), SQLite registry, the console's JSON API, no new modules. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-24-edge-rate-limit-design.md` (approved 2026-10-24). Read it first; section references below (§n) are to it. |
| + |
| +## Global Constraints |
| + |
| +- Defaults, verbatim from §4.6: `busy_at` 64, `client_share` 16, `rate` 300 req/s, `burst` 1000. |
| +- At most 8 path rules per instance; each prefix starts with `/`; longest prefix wins; byte-for-byte, case-sensitive match on `r.URL.Path` (§4.5). |
| +- Client key: `clientIP(r)` truncated to /32 (IPv4, including IPv4-mapped IPv6) or /64 (IPv6) (§4.1). Never any other header. |
| +- Fair-share refusal: `429`, `Retry-After: 1`. Backstop refusal: `429`, `Retry-After` = whole seconds until the next token, at least 1. Both carry `Cache-Control: no-store` and the approved plain-text body (Copy section). Never stored in the response cache. |
| +- A refused request never reaches the activator or the proxy, so it never wakes an instance. |
| +- Mode env `CARLOS_RATE_LIMIT` = `off | observe | enforce`, default `observe`; anything else logs and means `observe` (§7). |
| +- A plan's `rate` or `burst` ceiling below the platform default is refused when the catalog loads (§9 ruling 1). No app-wide default layer (§9 ruling 2). |
| +- Only `Kind == "instance"` routes are limited. Edge-door callers are never limited, locally or after a hop. |
| +- Self-hosted with no plan catalog: zero behaviour difference apart from the limiter itself, and no extra store reads. |
| +- Must cross-compile for darwin and windows (CI builds both); no linux-only syscalls. |
| +- No tenant names in code, tests or docs: use `shop.example.com`-style examples. |
| +- Local runs: only the touched packages' tests, with `-run`. CI runs the full suite. Leave `GOCACHE` at its default. |
| +- User-facing copy: only the approved strings in the Copy section below. Error fragments and diagnostics (`Validate` errors) are not copy. |
| +- Every new console route gets the census treatment of its `/move` GET sibling: `internal/console/testdata/claim-routes.txt`, `svcroutes_test.go`, `claim_invariant_fixtures_test.go`, `claim_invariant_probes_read_test.go`. |
| +- Record-write census: editing `handleAPIInstanceUpdate` changes its pinned digest in `internal/provision/plan9_b8_census_test.go`. Update the digest in the same commit and say so in the commit message. The #280 census ruling request is the controller's job, not the task's. |
| + |
| +## Copy (drafts; replaced by copy-review output before Tasks 5, 8, 9 and 10 start) |
| + |
| +The controller runs the copy-review skill on this list with Paul before dispatching any task that writes these, and pastes the approved text here. Implementers use exactly what is here. |
| + |
| +| Key | Where | Draft | |
| +| --- | --- | --- | |
| +| `body429` | edge 429 body (both reasons, backhaul too) | `Too many requests. Try again in a moment.` | |
| +| `overCeiling` | console 400 above a plan ceiling | `%s can be at most %d on this account's plan.` (arg 1 is `rate`, `burst` or `client share`) | |
| +| `consoleTooOld` | CLI, 400 "nothing to update" | `This console does not know about rate limits yet. It needs a newer carlos.` | |
| +| `usage` | `carlos ratelimit` usage text | see Task 9 | |
| +| `show*` | `carlos ratelimit show` labels | `busy at`, `per client while busy`, `backstop`, `plan ceiling`, `last 24 hours`, `(default)`, `%d refused while busy, %d over the backstop` | |
| +| `set*` | `carlos ratelimit set/clear` result lines | `rate limit set on`, `rate limit not set on`, `rate limit back to defaults on`, `rate limit not cleared on` | |
| +| `ui*` | console settings page | field labels mirroring `show*`, a `Reset to defaults` button, one help sentence: `Applies only when this instance is busy, except the backstop, which stops floods from one address.` | |
| + |
| +## Review Focus |
| + |
| +1. **Aliases and custom domains.** A request to a customer domain is limited against the primary instance's host, not the alias name. The test goes in Task 5. |
| +2. **A sleeping instance hit by a flood.** Requests over the backstop are refused without a wake, and the waiting requests hold slots so a wake storm reads as busy. The test goes in Task 5. |
| +3. **A client spraying addresses** (many distinct /64s). The limiter's memory stays bounded and never evicts a key that holds slots. The test goes in Task 2. |
| +4. **A plan downgraded after an owner set a higher value.** The box clamps at apply time even though the record still says the old number. The test goes in Task 5. |
| +5. **An old box reading a new record, or a new box reading an old record.** An unknown or absent `rate_limit` decodes to defaults, and a malformed column logs and uses defaults rather than failing the reload. The tests go in Tasks 3 and 5. |
| + |
| +--- |
| + |
| +### Task 1: `internal/ratelimit` settings, defaults, resolution and client key |
| + |
| +**Files:** |
| +- Create: `internal/ratelimit/settings.go`, `internal/ratelimit/clientkey.go` |
| +- Test: `internal/ratelimit/settings_test.go`, `internal/ratelimit/clientkey_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + type Rule struct { |
| + ClientShare int `json:"client_share,omitempty"` // in-flight per client while busy |
| + Rate int `json:"rate,omitempty"` // backstop, requests per second |
| + Burst int `json:"burst,omitempty"` // backstop bucket size |
| + } |
| + type PathRule struct { Prefix string `json:"prefix"`; Rule } |
| + type Settings struct { |
| + BusyAt int `json:"busy_at,omitempty"` |
| + Rule |
| + Paths []PathRule `json:"paths,omitempty"` |
| + } |
| + const MaxPaths = 8 |
| + const MaxValue = 1_000_000 |
| + var Defaults = Settings{BusyAt: 64, Rule: Rule{ClientShare: 16, Rate: 300, Burst: 1000}} |
| + func (s Settings) Validate() error |
| + func (s Settings) Canonical() string // "" for the zero Settings; else json.Marshal with Paths sorted by Prefix |
| + func Parse(col string) (Settings, error) // "" → zero Settings, nil |
| + type Ceiling struct{ ClientShare, Rate, Burst int } // 0 = no ceiling |
| + type Effective struct { BusyAt int; Base Rule; Paths []PathRule } // Paths sorted longest prefix first |
| + func Resolve(s Settings, c Ceiling) Effective |
| + func (e Effective) RuleFor(path string) (idx int, r Rule) // idx -1 = Base |
| + func ClientKey(ip string) (netip.Prefix, bool) |
| + ``` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| + - `TestValidateRefuses`, table-driven. Each of these returns an error: a negative field; a field above `MaxValue`; 9 path rules; a prefix `"api/"` (no leading slash); duplicate prefixes `"/api/"` twice; an empty prefix. `Settings{}` and `Defaults` both pass. |
| + - `TestCanonicalRoundTrip`: `Parse(s.Canonical())` equals `s` with Paths sorted; `Settings{}.Canonical() == ""`; `Parse("")` gives zero Settings and a nil error; `Parse("{bad")` returns an error. |
| + - `TestResolveFillsDefaults`: `Resolve(Settings{}, Ceiling{})` gives `Effective{BusyAt: 64, Base: Rule{16, 300, 1000}}`. |
| + - `TestResolvePathInheritsBase`: `Settings{Rule: Rule{Rate: 500}, Paths: []PathRule{{Prefix: "/api/", Rule: Rule{ClientShare: 32}}}}` resolves `/api/` to `Rule{32, 500, 1000}`. |
| + - `TestResolveClampsToCeiling`: owner `Rate: 5000`, `Burst: 9000`, path `/api/` `Rate: 8000`, with `Ceiling{Rate: 2000}`. Base rate and the `/api/` rate both resolve to 2000, burst stays 9000, and `BusyAt` is never clamped. |
| + - `TestRuleForLongestPrefix`: rules `/api/` and `/api/v2/`. `/api/v2/x` gives the `/api/v2/` rule, `/api/x` gives `/api/`, `/api` gives Base (idx -1), and `/API/x` gives Base. |
| + - `TestClientKey`: `"203.0.113.9"` gives `203.0.113.9/32`; `"::ffff:203.0.113.9"` gives `203.0.113.9/32`; `"2001:db8:1:2:3:4:5:6"` gives `2001:db8:1:2::/64`; `"not-an-ip"` gives false. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail** |
| + |
| + Run: `go test ./internal/ratelimit/ -run 'TestValidate|TestCanonical|TestResolve|TestRuleFor|TestClientKey'` |
| + Expected: build failure (package does not exist). |
| + |
| +- [ ] **Step 3: Implement the interfaces above.** Resolution order: start from `Defaults`, overlay the non-zero owner fields, give each path rule the resolved base values for any zero field, then clamp `ClientShare`, `Rate` and `Burst` (base and every path) to the ceiling's non-zero values. `ClientKey` uses `netip.ParseAddr`, then `Unmap`, then `Prefix(32)` or `Prefix(64)`. The package imports nothing from this repo. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. Also run `GOOS=darwin go build ./internal/ratelimit/ && GOOS=windows go build ./internal/ratelimit/`. |
| + |
| +- [ ] **Step 5: Commit** — `ratelimit: settings, defaults, plan-ceiling resolution, client key (#358)` |
| + |
| +### Task 2: `internal/ratelimit` limiter |
| + |
| +**Files:** |
| +- Create: `internal/ratelimit/limiter.go`, `internal/ratelimit/shares.go` |
| +- Test: `internal/ratelimit/limiter_test.go`, `internal/ratelimit/shares_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `Effective`, `Rule`, `RuleFor` (Task 1). |
| +- Produces: |
| + ```go |
| + type Decision int |
| + const ( Allow Decision = iota; Shed; OverCeiling ) |
| + type Ticket struct{ /* opaque */ } |
| + func (t *Ticket) Release() // idempotent, safe on a nil *Ticket |
| + type Limiter struct{ /* opaque */ } |
| + func NewLimiter(now func() time.Time, maxKeys int) *Limiter |
| + // Admit decides one request. holdSlot false (upgrades) judges without taking a slot. |
| + // On Shed/OverCeiling the returned Ticket is nil unless observe is true, in which |
| + // case the request is let through and the Ticket holds a slot like Allow. |
| + func (l *Limiter) Admit(host string, eff Effective, client netip.Prefix, path string, holdSlot, observe bool) (t *Ticket, d Decision, retryAfter time.Duration) |
| + func (l *Limiter) InFlight(host string) int |
| + // Shares is the concurrency-only half, also used by the backhaul (Task 7). |
| + type Shares struct{ /* opaque */ } |
| + func NewShares(maxKeys int) *Shares |
| + // Acquire refuses when busy is true and key already holds >= share slots. |
| + func (s *Shares) Acquire(key string, busy bool, share int) (release func(), ok bool) |
| + func (s *Shares) Held(key string) int |
| + ``` |
| + |
| +- [ ] **Step 1: Write the failing tests** (fake clock throughout) |
| + |
| + - `TestAdmitQuietInstanceNeverSheds`: `busy_at` 64. One client holds 63 slots (Admit 63 times, never Release); the 64th Admit from the same client is Allow. Anti-goal: never refuse a shared address on a quiet instance. |
| + - `TestAdmitBusyShedsHeaviestOnly`: `busy_at` 4, `client_share` 2. Client A holds 2 slots and B holds 2, so the host is busy. A's next Admit is Shed with retryAfter 1s, a fresh client C's Admit is Allow, and after A releases one slot, A's next Admit is Allow. |
| + - `TestBackstopBucket`: `rate` 10, `burst` 20. 20 Admits at t0 are Allow and the 21st is OverCeiling with retryAfter 1s. At t0+100ms one more is Allow. Release each ticket immediately so busy never applies. |
| + - `TestBackstopRetryAfterRoundsUp`: `rate` 1, `burst` 1. The second Admit at t0 gives retryAfter 1s; at t0+200ms it gives 1s; never below 1s. |
| + - `TestPathRuleSeparateAllowance`: base `rate` 1 `burst` 1, `/api/` `rate` 100 `burst` 100. Exhausting the base bucket leaves `/api/x` admitted, and `/api/` usage never drains the base bucket. |
| + - `TestHoldSlotFalseTakesNoSlot`: an Allow with holdSlot false leaves `InFlight` unchanged. |
| + - `TestObserveLetsThroughAndHolds`: an over-ceiling Admit with observe true returns OverCeiling and a non-nil Ticket, and `InFlight` goes up by one. |
| + - `TestReleaseIdempotent`: Release twice gives `InFlight` back to exactly the prior value; Release on a nil `*Ticket` does not panic. |
| + - `TestMemoryBoundNeverEvictsHolders` (Review Focus 3): `maxKeys` 100. 50 clients each hold one slot; then 10,000 distinct clients each Admit and Release. The tracked key count never exceeds 100, all 50 holders still count in `InFlight`, and each holder's Release returns `InFlight` to 0. When the map is full of holders, new clients are admitted untracked (Allow) rather than refused. |
| + - `TestSharesAcquire`: `busy` false always ok; `busy` true refuses a key at `share`; release frees it. |
| + - `go test -race` on a test that runs 64 goroutines × 1000 Admit/Release on one host and ends with `InFlight == 0`. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/ratelimit/ -run 'TestAdmit|TestBackstop|TestPathRule|TestHoldSlot|TestObserve|TestRelease|TestMemoryBound|TestShares'` gives a build failure. |
| + |
| +- [ ] **Step 3: Implement.** One mutex-guarded map per Limiter is fine at this scale; per-host sharding is optional. Order inside Admit: (1) take a backstop token from the `(host, ruleIdx, client)` bucket, or return OverCeiling; (2) if `InFlight(host) >= BusyAt` and the `(host, ruleIdx, client)` holder count is `>= ClientShare`, return Shed; (3) otherwise Allow, taking a slot if holdSlot. Eviction removes entries with zero holders and a full bucket first, then entries with zero holders, oldest first. The bucket stores a token count and a last-refill time; refill is `rate × elapsed`, capped at `burst`. |
| + |
| +- [ ] **Step 4: Run the tests**, then `go test -race ./internal/ratelimit/`. Expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `ratelimit: limiter with fair share under pressure and backstop buckets (#358)` |
| + |
| +### Task 3: Settings on the record and the registry, converged by reconcile |
| + |
| +**Files:** |
| +- Modify: `internal/provision/provision.go` (Record near `Gate`, line ~173; `ValidateShape` near 591) |
| +- Modify: `internal/provision/reconcile.go` (`RouteFor` ~445; owned-converge arm after the gate block ~1366-1374) |
| +- Modify: `internal/registry/registry.go` (Route near `Gate` ~182; migration ~597; select list ~897 and scan ~918; Upsert ~1203; new `SetRateLimit` beside `SetGate` ~2090; `Fingerprint` ~3070 and the snapshot fingerprint ~3369) |
| +- Modify: `internal/registry/remote_dispatch.go` (~252) and the remote client's request struct |
| +- Test: `internal/provision/ratelimit_record_test.go`, `internal/registry/ratelimit_column_test.go`, `internal/provision/reconcile_ratelimit_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `ratelimit.Settings`, `Validate`, `Canonical`, `Parse` (Task 1). |
| +- Produces: |
| + - `provision.Record.RateLimit *ratelimit.Settings` with JSON tag `json:"rate_limit,omitempty"`. Nil means defaults. |
| + - `func (rec Record) RateLimitColumn() string` returns `rec.RateLimit.Canonical()`, or `""` when nil. |
| + - `registry.Route.RateLimit string`, the canonical JSON (`""` means defaults), included in `Fingerprint()`. |
| + - `func (r *Registry) SetRateLimit(host, canonical string) error`, with the same contract as `SetGate`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + - `TestValidateShapeRateLimit`: a record with an invalid `RateLimit` (9 paths) fails `ValidateShape` with an error naming `rate_limit`; a nil or valid one passes. |
| + - `TestRecordWithoutRateLimitDecodes`: the JSON of a record with no `rate_limit` key decodes with nil `RateLimit` and `RateLimitColumn() == ""` (Review Focus 5). |
| + - `TestRateLimitColumnRoundTrip`: `SetRateLimit(host, c)` then `Get(host)` gives `.RateLimit == c`; an unmatched host errors. |
| + - `TestUpsertPreservesRateLimit`: a full-row Upsert with `RateLimit ""` keeps a set value; `SetRateLimit(host, "")` clears it. |
| + - `TestFingerprintMovesOnRateLimit`: the fingerprint changes when the column is set and again when it is cleared. A registry with no rate-limited rows has a fingerprint byte-identical to the one computed with the pre-change encoding: fold the column in only when non-empty, the `import_gate` rule. |
| + - `TestRemoteSetRateLimit`: the remote-dispatch twin round-trips, mirroring the existing `SetGate` remote test. |
| + - `TestReconcileConvergesRateLimit`: through the real owned-converge arm, as the gate converge test is wired. A record whose `RateLimit` differs from the row results in `SetRateLimit` with the canonical value. A new record mints a row carrying it (`RouteFor`). |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/provision/ ./internal/registry/ -run 'RateLimit'` |
| + |
| +- [ ] **Step 3: Implement**, copying every gate touchpoint listed under Files. The column is `rate_limit TEXT NOT NULL DEFAULT ''`. The converge log line names the host and says `rate limit custom -> default` or `default -> custom`. It never prints the JSON. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `provision, registry: rate_limit on the instance record and route, converged like the gate (#358)` |
| + |
| +### Task 4: Plan ceilings and the box hook |
| + |
| +**Files:** |
| +- Modify: `internal/plans/catalog.go` (`Limits` 26-40, `validate` 122, `nonNegative` 167, `Effective` clamp block ~226-232, `generous` 247) |
| +- Modify: `internal/host/plans_pass.go` (beside `planIdleFor`, ~124-131) |
| +- Test: `internal/plans/catalog_ratelimit_test.go`, `internal/host/plans_pass_ratelimit_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `ratelimit.Defaults`, `ratelimit.Ceiling` (Task 1). |
| +- Produces: |
| + - `plans.Limits` fields `RateClientShare int` (`json:"rate_client_share,omitempty"`), `RatePerSecond int` (`json:"rate_per_second,omitempty"`) and `RateBurst int` (`json:"rate_burst,omitempty"`). Zero means no ceiling. |
| + - `func (l Limits) RateCeiling() ratelimit.Ceiling` |
| + - `func planRateCeilingFor(rt registry.Route) ratelimit.Ceiling` in `internal/host`. Zero Ceiling when plans are off or there is no record. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + - `TestCatalogRefusesRateCeilingBelowDefault`: a plan with `rate_per_second: 100` (default 300) fails `ParseCatalog` with an error naming the plan and the field; the same for `rate_burst: 10`, and for `Ceilings`. `rate_client_share: 4` is accepted. A negative value is refused. |
| + - `TestPackCannotAddRateFields`: a pack whose `adds` sets `rate_per_second` is refused (the existing `additive()` rule; this pins it). |
| + - `TestEffectiveClampsRateFields`: plan `rate_per_second: 1000`, catalog ceiling 500, gives 500. A sponsored override of 5000 gives 5000 (`generous`). |
| + - `TestPlanRateCeilingForOffIsZero`: with plans off, `planRateCeilingFor` returns `Ceiling{}`. With a stored limits snapshot carrying `RatePerSecond: 400`, it returns `Ceiling{Rate: 400}`. Use the same snapshot set-up the existing `planIdleFor` test uses. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/plans/ ./internal/host/ -run 'RateCeiling|RateFields|PlanRateCeiling'` |
| + |
| +- [ ] **Step 3: Implement.** The `internal/plans` → `internal/ratelimit` import is allowed, since ratelimit imports nothing from the repo. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `plans: optional rate ceilings, never below the platform default; box hook (#358)` |
| + |
| +### Task 5: Edge enforcement |
| + |
| +**Files:** |
| +- Create: `internal/edge/ratelimit.go` (mode, admission helper, slot writer, 429 writer) |
| +- Modify: `internal/edge/edge.go` (`upstream` struct ~357; `Reload` map build ~906; `ServeHTTP` upgrade branch ~1721 and the stretch between the cache check and `if up.hibernating` ~1784; Edge fields and a setter beside `EnableUsage` ~458) |
| +- Modify: `internal/edge/usagemeter.go`, `internal/usage/edge.go` (`EdgeRow`, `AddDeltas`) |
| +- Modify: `internal/host/agent.go` (inside the `RunWithActivatorReady` callback ~926-958) |
| +- Test: `internal/edge/ratelimit_test.go` (model it on `internal/edge/gate_test.go`; every request goes through `Edge.ServeHTTP`), `internal/usage/edge_ratelimit_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1-4. |
| +- Produces: |
| + - `type RateMode int` with constants `RateOff`, `RateObserve` and `RateEnforce`, and `func rateModeFromEnv() RateMode`, which reads `CARLOS_RATE_LIMIT` once in the constructor. |
| + - `func (e *Edge) SetRateCeilingFor(f func(registry.Route) ratelimit.Ceiling)`, stored atomically; nil means no ceiling. |
| + - `upstream.rate ratelimit.Settings`, parsed at Reload. A parse error logs through `logThrottled` and uses zero Settings (defaults). |
| + - `usage.EdgeRow.RateShed int64` (`json:"rate_shed,omitempty"`) and `RateCeiling int64` (`json:"rate_ceiling,omitempty"`), summed in `AddDeltas`. |
| + - `func (u *UsageMeter) RecordRateLimited(rt registry.Route, d ratelimit.Decision)`. |
| + - `func (e *Edge) rateAdmit(w http.ResponseWriter, r *http.Request, up upstream, holdSlot bool) (release func(), ok bool)`. When ok is false it has already written the 429. |
| + |
| +- [ ] **Step 1: Write the failing tests**, all through `ServeHTTP` with a real upstream (`httptest` or a unix socket, as `gate_test.go` does): |
| + - `TestRateEnforceShedsWhenBusy`: a route with `RateLimit` `{"busy_at":2,"client_share":1}` and an upstream that blocks until released. Client A sends 1 request and B sends 1 (busy). A's second request gets 429 with `Retry-After: 1`, `Cache-Control: no-store` and the `body429` text. B's upstream count did not change. |
| + - `TestRateQuietNeverRefuses`: defaults, 200 sequential requests from one address all 200. |
| + - `TestRateBackstopRefusesWithoutWake` (Review Focus 2): a hibernating route with `{"rate":1,"burst":1}` and a sleeping instance. The first request wakes it (activator wake count 1). The second, while the first is still waiting, gets 429, and the wake count stays 1. With the instance asleep again and its bucket empty, a request gets 429 and no wake. |
| + - `TestRateSlotReleasedAtHeaders`: `{"busy_at":1,"client_share":1}` and an upstream that writes headers then streams a body for 2s. While the body streams, a request from a second client is 200, not shed, because the first slot was released at headers. |
| + - `TestRateUpgradeJudgedNotHeld`: an upgrade over the backstop gets 429. Ten admitted upgrades leave `InFlight` 0. |
| + - `TestRateAliasUsesPrimaryHost` (Review Focus 1): requests to an alias and to the primary share one bucket. |
| + - `TestRateObserveCountsButServes`: in observe mode an over-backstop request is 200, and the meter's drained row has `RateCeiling == 1`. |
| + - `TestRateOffDoesNothing`: in off mode no limiter state is touched (`InFlight` 0 throughout). |
| + - `TestRateCacheHitNeverLimited`: a cacheable GET served from cache is 200 even with the bucket empty. |
| + - `TestRateDoorCallerExempt`: a request through `serveDoor` with the bucket empty is 200. |
| + - `TestRateHopKeyedByClient`: a verified hop carrying `X-Forwarded-For: 203.0.113.9` and one carrying `198.51.100.7` get separate buckets. Use the hop signing from `backhaul_e2e_test.go`. |
| + - `TestRateCeilingClampsAtApply` (Review Focus 4): a route with `{"rate":1000,"burst":1}` and `SetRateCeilingFor` returning `Ceiling{Rate: 2}`. The third request in one second is 429. |
| + - `TestRateMalformedColumnUsesDefaults` (Review Focus 5): a route row whose column is `"{bad"` still serves, and a log line names the host. |
| + - `TestRateTunnelSides`: on the hub (`viaTunnel` true) the limiter applies. On the fleet box, a request arriving over the tunnel is never limited. Find the tunnel mark through `internal/tunnelclient`. |
| + - `TestRateModeFromEnv`: `""` gives observe, `"enforce"` enforce, `"off"` off, `"bogus"` observe. |
| + - In `internal/usage`, `TestAddDeltasSumsRateCounters`. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/edge/ ./internal/usage/ -run 'TestRate|TestAddDeltasSumsRate'` |
| + |
| +- [ ] **Step 3: Implement.** |
| + - **Placement:** the upgrade branch calls `rateAdmit(..., holdSlot=false)` first. The ordinary path calls `rateAdmit(..., holdSlot=true)` after the cache check and before `if up.hibernating`, wraps `w` in the slot writer before the recorders are built, and defers `release`. |
| + - **Slot writer:** it releases on the first `WriteHeader`, `Write` or `Flush`, and implements `Unwrap() http.ResponseWriter` and `Flush()`. |
| + - **Exemptions:** `Kind != "instance"`, door callers (`doorCallerFrom`), and fleet-side tunnel arrivals are exempt. |
| + - **Limiter key:** use `up.route.Host`, never the request host. |
| + - **Counting:** in observe and enforce modes every non-Allow decision calls `RecordRateLimited`. `logThrottled` uses the key `"ratelimit "+host+" "+reason`, and the message says `refused` or `would refuse` and names the host, the reason and the client prefix. |
| + - **Wiring:** `agent.go` calls `e.SetRateCeilingFor(planRateCeilingFor)` beside `EnableUsage`. |
| + |
| +- [ ] **Step 4: Run the tests**, then `go test -race ./internal/edge/ -run TestRate`. Expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `edge: pressure-gated fair share and backstop per instance, observe by default (#358)` |
| + |
| +### Task 6: The door mark survives a hop |
| + |
| +**Files:** |
| +- Modify: `internal/edge/mesh.go` (beside `signHop` 188, `verifyHopEitherKey` 228) |
| +- Modify: `internal/edge/homemap.go` (`acceptHop` 1069-1101, the backhaul send that sets hop headers ~2920) |
| +- Modify: `internal/edge/door.go` (header comment lines 25-26: "Nothing carries the mark across a hop" is no longer true; rewrite it) |
| +- Modify: `internal/edge/ratelimit.go` (exemption) |
| +- Test: `internal/edge/hopdoor_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `const hopDoorHeader = "X-Carlos-Hop-Door"` |
| + - `func signHopDoor(key []byte, host string, now time.Time) string` |
| + - `func verifyHopDoorEitherKey(current, next []byte, host, header string, now time.Time) bool` |
| + - A context key `hopDoorKey{}` with `func hopDoorFrom(ctx context.Context) bool` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + - `TestHopDoorDomainSeparated`: a `signHop` value presented as the door header fails `verifyHopDoorEitherKey`, and a `signHopDoor` value presented as the hop auth fails `verifyHopEitherKey`. |
| + - `TestAcceptHopSetsDoorMark`: a verified hop with a valid door header gives `hopDoorFrom` true. A door header without a verified hop gives false, and the header is deleted in every case (the "deleted in all hop cases" invariant). |
| + - `TestBackhauledDoorCallExempt`: end to end with `newBackhaulEdge` (`backhaul_e2e_test.go`). A door caller on box A calls a host homed on box B whose bucket is empty, and gets 200. An ordinary client on the same path gets 429. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/edge/ -run 'HopDoor|BackhauledDoor'` |
| + |
| +- [ ] **Step 3: Implement.** The MAC uses the same key and timestamp format as `signHop`, over a byte string prefixed with `door\x00` before the canonical host, so the two can never collide. The sender sets the header only when `doorCallerFrom(r.Context())` is ok. The limiter's exemption becomes `doorCallerFrom || hopDoorFrom`. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `edge: carry the door mark across a signed hop so door callers stay unlimited (#358)` |
| + |
| +### Task 7: Fair share over the sending edge's backhaul slots |
| + |
| +**Files:** |
| +- Modify: `internal/edge/edge.go` (`backhaulAcquire` 2795-2814, its caller ~2903-2919) |
| +- Test: `internal/edge/backhaul_share_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `ratelimit.Shares` (Task 2), `ratelimit.ClientKey` (Task 1), the rate mode (Task 5). |
| +- Produces: `func (e *Edge) backhaulAcquire(box, clientKey string) (releaseSlot func(), ok bool, shed bool)`. `shed` true means a fair-share refusal, not saturation. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + - `TestBackhaulShareShedsHeavyClient`: cap 64. Client A holds 32 slots, the box is half full, so it is busy. A's 33rd request is 429 `Retry-After: 1` with `body429`, while client B's request is admitted. |
| + - `TestBackhaulShareQuietAdmitsHeavyClient`: with 31 slots held in total, A's next request is admitted. |
| + - `TestBackhaulSaturationUnchanged`: at 64 slots the answer is still the existing 503 `Retry-After: 2`. |
| + - `TestBackhaulShareObserveOnlyLogs`: in observe mode, A's 33rd request is admitted and a log line says `would refuse`. |
| + - `TestBackhaulShareDoorExempt`: a door caller is never shed. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/edge/ -run TestBackhaulShare` |
| + |
| +- [ ] **Step 3: Implement.** Keep one `Shares` per destination box. Busy is `n >= backhaulMaxInflight/2`, the share is a constant 8, and the key is `ClientKey(clientIP(r))`. A refusal rolls back the box counter it incremented. There is no meter row, since the sending edge has no route. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `edge: fair share over backhaul slots so one client cannot fill a box's hop (#358)` |
| + |
| +### Task 8: Console doors |
| + |
| +**Files:** |
| +- Modify: `internal/console/apiinstances.go` (`handleAPIInstanceUpdate` ~1990-2090; a new GET handler beside it) |
| +- Modify: `internal/console/authcli.go` (route registration ~702) |
| +- Modify: the census files named in Global Constraints, plus `internal/provision/plan9_b8_census_test.go` (the digest) |
| +- Test: `internal/console/apiinstances_ratelimit_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1, 3, 4; `s.EffectiveLimits(acct)` (`internal/console/planlimits.go:118`). |
| +- Produces: |
| + - In the PATCH request struct, `RateLimit *ratelimit.Settings` with JSON tag `json:"rate_limit"`. Absent leaves it alone; `{}` clears to defaults; anything else replaces it whole. Add `rate_limit` to the "nothing to update" check and its message. |
| + - `GET /api/cli/apps/{account}/{app}/instances/{host}/rate-limit`, role `svcrole.Instance`, handled by `handleAPIInstanceRateLimitGet`. It returns: |
| + ```json |
| + {"host":"...","settings":{...},"defaults":{...},"ceiling":{...}|null, |
| + "effective":{...},"counts":{"shed":0,"ceiling":0,"since":"RFC3339"}} |
| + ``` |
| + `ceiling` is null when there is no plan catalog. `counts` sums `RateShed` and `RateCeiling` for the host over every box's edge day rows in the last 24 hours. Enumerate boxes the way `economicspass.go`'s sweep does (~400-415). |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + - `TestPatchRateLimitSetsAndClears`: PATCH `{"rate_limit":{"rate":500}}` stores it on the record, then `{"rate_limit":{}}` makes it nil. |
| + - `TestPatchRateLimitInvalid400`: 9 paths gives 400, and the record is unchanged. |
| + - `TestPatchRateLimitOverPlanCeiling`: with a catalog whose plan has `rate_per_second: 400`, PATCH `{"rate_limit":{"rate":500}}` gives 400 with the `overCeiling` text, and `{"rate":400}` gives 200. With no catalog, `{"rate":500000}` gives 200. |
| + - `TestRateLimitGet`: the response for a host with settings, a ceiling and two boxes' usage rows sums the counts across boxes within 24 h and excludes a row 25 h old. |
| + - Census: the existing census tests pass after their entries are added, mirroring the `/move` GET. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test ./internal/console/ -run 'RateLimit'` |
| + |
| +- [ ] **Step 3: Implement.** The ceiling check runs before `ValidateShape` and compares the base and every path rule. Then update the census digest and entries (run `go test ./internal/console/ ./internal/provision/ -run 'Census|ClaimInvariant|SvcRoutes'` and follow the failures). |
| + |
| +- [ ] **Step 4: Run** `go test ./internal/console/ ./internal/provision/ -run 'RateLimit|Census|ClaimInvariant|SvcRoutes'`; expected PASS. |
| + |
| +- [ ] **Step 5: Commit**, naming the census digest change in the message — `console: rate_limit on the instance PATCH, plan-ceiling refusal, rate-limit GET (#358)` |
| + |
| +### Task 9: `carlos ratelimit` |
| + |
| +**Files:** |
| +- Create: `ratelimit_cli.go`, `ratelimit_cli_test.go` (repo root, beside `gate_cli.go`) |
| +- Modify: the verb dispatch where `runGate` is registered; `apimode.go` (beside `instanceSetGate` ~3604); `docs/site/cli.md` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 8's doors. |
| +- Produces: |
| + - `func (s *apiSession) instanceRateLimit(ctx context.Context, a apiAcct, app, host string) (apiRateLimit, error)` |
| + - `func (s *apiSession) instanceSetRateLimit(ctx context.Context, a apiAcct, app, host string, set ratelimit.Settings) (apiInstanceRecord, error)` |
| + - The verb shape (usage text from the Copy section once approved): |
| + ``` |
| + carlos ratelimit show -app <a> [-host <h>] |
| + carlos ratelimit set -app <a> [-host <h>] [-busy-at N] [-share N] [-rate N] [-burst N] |
| + [-path /prefix:share=N,rate=N,burst=N]... [-remove-path /prefix]... |
| + carlos ratelimit clear -app <a> [-host <h>] |
| + -account, -account-id, -console as carlos gate |
| + ``` |
| + Without `-host`, the verb fans out over every instance of the app, one line per host, and exits 1 if any fail (the `gateApply` pattern). `set` reads the current settings, overlays only the flags given, and PATCHes the whole object. |
| + |
| +- [ ] **Step 1: Write the failing tests** against a fake console handler, as the gate CLI tests do: |
| + - `TestRatelimitSetOverlaysFlags`: current `{"rate":500}` plus `-burst 2000` PATCHes `{"rate":500,"burst":2000}`. |
| + - `TestRatelimitPathFlagParse`: `-path /api/:share=32,rate=1000` adds that rule, and `-remove-path /api/` removes it. A malformed `-path` exits 2 before any request. |
| + - `TestRatelimitClearSendsEmptyObject`. |
| + - `TestRatelimitFanOutReportsEach`: three hosts with one failing gives three lines and exit 1. |
| + - `TestRatelimitShowMarksDefaultsAndCeiling`: the output marks defaulted values with `(default)`, shows the plan-ceiling line only when `ceiling` is non-null, and prints the counts line. |
| + - `TestRatelimitOldConsole`: a 400 "nothing to update" gives the `consoleTooOld` text. |
| + |
| +- [ ] **Step 2: Run them to make sure they fail.** `go test . -run TestRatelimit` |
| + |
| +- [ ] **Step 3: Implement.** Add the verb to `docs/site/cli.md` with the approved usage text. |
| + |
| +- [ ] **Step 4: Run the tests.** Same command; expected PASS. Also run `GOOS=windows go build . && GOOS=darwin go build .` |
| + |
| +- [ ] **Step 5: Commit** — `carlos ratelimit: show, set and clear per-instance rate limits (#358)` |
| + |
| +### Task 10: Console settings page fields |
| + |
| +**Files:** |
| +- Modify: `internal/console/appsettings.go` (the gate writer ~982 and its form), with its template |
| +- Test: the existing app-settings test file, extended with a rate-limit case |
| + |
| +**Interfaces:** |
| +- Consumes: Task 8's validation and ceiling check. Factor that check into `func (s *Server) checkRateLimit(acct *Account, set ratelimit.Settings) error` in Task 8 if it is not already shared, and call it from both writers. |
| + |
| +- [ ] **Step 1: Write the failing test.** `TestAppSettingsRateLimitForm`: posting the form with `rate=500` stores it; `Reset to defaults` clears it; a value over the plan ceiling re-renders the form with the `overCeiling` text and stores nothing. Path rules are shown read-only, with a note that the CLI edits them (*my call:* no path-rule editor in the web UI in v1). |
| + |
| +- [ ] **Step 2: Run it to make sure it fails.** `go test ./internal/console/ -run TestAppSettingsRateLimit` |
| + |
| +- [ ] **Step 3: Implement**, following the gate's form, CSRF handling and audit row exactly. |
| + |
| +- [ ] **Step 4: Run the test.** Same command; expected PASS. |
| + |
| +- [ ] **Step 5: Commit** — `console: rate limit fields on the instance settings page (#358)` |
| + |
| +### Task 11: Operator docs |
| + |
| +**Files:** |
| +- Modify: the docs/site page that documents the password gate (find it with `grep -rl "carlos gate" docs/site`), or a new `docs/site/rate-limits.md` linked from `docs/site/nav.json` if the gate has no page |
| +- Modify: `docs/site/plans.md` (the three ceiling fields and the below-default refusal) |
| + |
| +- [ ] **Step 1: Write the section.** It covers what is limited and when (busy versus quiet), the defaults, path rules, the three modes and `CARLOS_RATE_LIMIT`, how to read the counts, and that fleet-tunnelled routes are limited on the hub only. Every sentence here is copy, so the controller sends this text through copy review before the commit. |
| + |
| +- [ ] **Step 2: Build the docs site check** the repo runs in CI for `docs/site` (find it in `.amadan/` CI config). It is expected to pass. |
| + |
| +- [ ] **Step 3: Commit** — `docs: rate limits per instance, modes and plan ceilings (#358)` |