amadan
design: edge rate limit, fair share under pressure, per instance (#358)
Human
Prompt
Can we take carlos/platform#358 and make sure it's configurable per instance?
From this commit's own message. The prompt ledger has no entries for it — which is what a squashed, rebased or amended commit looks like, and what every commit looks like in a repo without a ledger.
Summary
Draft design for #358. Pressure-gated fair share plus a high backstop, set per instance on the record (the password gate's path), with path-prefix exceptions and an optional plan ceiling where a catalog is loaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+338
−0
docs/superpowers/specs/2026-10-24-edge-rate-limit-design.md+338 −0
CI failed — run details
CI log
=== gate ===
.amadan/test-gate.sh
go: downloading golang.org/x/net v0.60.0
go: downloading golang.org/x/sys v0.48.0
go: downloading golang.org/x/crypto v0.57.0
go: downloading github.com/pion/stun/v3 v3.1.5
go: downloading golang.org/x/sync v0.23.0
go: downloading github.com/pion/transport/v4 v4.0.2
go: downloading github.com/pion/dtls/v3 v3.1.4
go: downloading github.com/pion/dtls/v3 v3.1.4
go: downloading golang.org/x/text v0.42.0
go: downloading golang.org/x/text v0.42.0
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (26 s)
go: downloading golang.org/x/tools v0.49.0
go: downloading github.com/pion/dtls/v3 v3.1.4
go: downloading golang.org/x/text v0.42.0
go: downloading golang.org/x/mod v0.41.0
=== gofmt (1 s)
gofmt -l .: clean
=== rest (178 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.739s
ok github.com/carlosframework/platform/cmd/moveprobe 11.991s
ok github.com/carlosframework/platform/cmd/zddprobe 0.799s
ok github.com/carlosframework/platform/internal/activator 57.885s
ok github.com/carlosframework/platform/internal/agentsock 0.050s
ok github.com/carlosframework/platform/internal/alerting 0.007s
ok github.com/carlosframework/platform/internal/analytics 0.055s
ok github.com/carlosframework/platform/internal/appconfig 2.892s
ok github.com/carlosframework/platform/internal/apperr 0.149s
ok github.com/carlosframework/platform/internal/applog 0.250s
ok github.com/carlosframework/platform/internal/awsinv 0.013s
ok github.com/carlosframework/platform/internal/boxchoice 0.014s
ok github.com/carlosframework/platform/internal/boxid 0.867s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.026s
ok github.com/carlosframework/platform/internal/caps 0.035s
ok github.com/carlosframework/platform/internal/certwarm 0.010s
ok github.com/carlosframework/platform/internal/cgroup 0.830s
ok github.com/carlosframework/platform/internal/changelog 0.577s
ok github.com/carlosframework/platform/internal/claimactivation 1.336s
ok github.com/carlosframework/platform/internal/clamd 0.078s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 8.342s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.029s
ok github.com/carlosframework/platform/internal/dbquery 0.016s
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000004
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000004 maxTXID=0000000000000004 size=537
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000004 txid.db=0000000000000004
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
--- FAIL: TestADrainingBoxDoesNotPick (0.23s)
agent.go:135: dbreads: dbr_atddiu25mtxrqz2hqijz3omkga: planned 1 files (720 bytes) and restored to 1 in 16ms
agent.go:135: dbreads: dbr_atddiu25mtxrqz2hqijz3omkga: output 4 bytes in 1 confirmed chunks in 1ms
agent.go:135: dbreads: dbr_qiv4jhhgyx2fiwhuitz2q2zoqq: planned 1 files (720 bytes) and restored to 1 in 2ms
agent.go:135: dbreads: dbr_qiv4jhhgyx2fiwhuitz2q2zoqq: output 4 bytes in 1 confirmed chunks in 33ms
agent.go:135: dbreads: dbr_zxoqkc3qn7fner5zfpnetvqble: planned 1 files (720 bytes) and restored to 1 in 27ms
agent.go:135: dbreads: dbr_zxoqkc3qn7fner5zfpnetvqble: output 4 bytes in 1 confirmed chunks in 5ms
loop_test.go:170: no result for dbr_n6evhd4ufeqjlqvinxfqm3wmhq: dirstore: db-reads/dbr_n6evhd4ufeqjlqvinxfqm3wmhq/result.json: file does not exist
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 20:05:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 20:05:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
FAIL
FAIL github.com/carlosframework/platform/internal/dbreads 2.098s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.012s
ok github.com/carlosframework/platform/internal/dnsrec 0.013s
ok github.com/carlosframework/platform/internal/docsite 0.020s
ok github.com/carlosframework/platform/internal/domains 3.002s
ok github.com/carlosframework/platform/internal/doorkey 0.024s
ok github.com/carlosframework/platform/internal/economics 0.019s
ok github.com/carlosframework/platform/internal/edge 46.030s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.135s
ok github.com/carlosframework/platform/internal/gate 1.655s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.019s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/historyreadback 0.158s
ok github.com/carlosframework/platform/internal/host 143.073s
ok github.com/carlosframework/platform/internal/hostactors 7.399s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.005s
ok github.com/carlosframework/platform/internal/hostbox 3.014s
ok github.com/carlosframework/platform/internal/hostclaim 0.848s
ok github.com/carlosframework/platform/internal/hostenv 0.011s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 5.275s
ok github.com/carlosframework/platform/internal/hostfence 0.059s
ok github.com/carlosframework/platform/internal/hostfiles 10.375s
ok github.com/carlosframework/platform/internal/hostfloor 0.032s
ok github.com/carlosframework/platform/internal/hostguard 1.028s
ok github.com/carlosframework/platform/internal/hostops 1.989s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.013s
ok github.com/carlosframework/platform/internal/hostquarantine 0.005s
ok github.com/carlosframework/platform/internal/hostrelocate 0.633s
ok github.com/carlosframework/platform/internal/hostresolve 0.225s
ok github.com/carlosframework/platform/internal/hostwrite 0.008s
ok github.com/carlosframework/platform/internal/infraview 0.004s
ok github.com/carlosframework/platform/internal/instancectl 5.259s
ok github.com/carlosframework/platform/internal/instanceexec 2.815s
ok github.com/carlosframework/platform/internal/instancefs 0.006s
ok github.com/carlosframework/platform/internal/instancekey 0.008s
ok github.com/carlosframework/platform/internal/instanceupdates 0.015s
ok github.com/carlosframework/platform/internal/layout 1.715s
ok github.com/carlosframework/platform/internal/lbhealth 0.007s
ok github.com/carlosframework/platform/internal/lease 10.115s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.489s
ok github.com/carlosframework/platform/internal/ledger 0.036s
ok github.com/carlosframework/platform/internal/mail 0.264s
ok github.com/carlosframework/platform/internal/maintenance 0.096s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.578s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.125s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 22.203s
ok github.com/carlosframework/platform/internal/moves/archive 1.067s
ok github.com/carlosframework/platform/internal/objstore 0.281s
ok github.com/carlosframework/platform/internal/ops 0.236s
ok github.com/carlosframework/platform/internal/park 0.288s
ok github.com/carlosframework/platform/internal/plans 0.007s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 31.356s
ok github.com/carlosframework/platform/internal/publicrelease 0.012s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.733s
ok github.com/carlosframework/platform/internal/relay 7.247s
ok github.com/carlosframework/platform/internal/release 26.329s
ok github.com/carlosframework/platform/internal/rollout 3.878s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.063s
ok github.com/carlosframework/platform/internal/rootstate 0.038s
ok github.com/carlosframework/platform/internal/runs 0.605s
ok github.com/carlosframework/platform/internal/s3compat 0.007s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.690s
ok github.com/carlosframework/platform/internal/seeds 0.025s
ok github.com/carlosframework/platform/internal/sim/engines 2.553s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.488s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.189s
ok github.com/carlosframework/platform/internal/status 0.080s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.319s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.007s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.007s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.007s
ok github.com/carlosframework/platform/internal/vet 1.670s
FAIL
=== internal_console@0 (79 s)
ok github.com/carlosframework/platform/internal/console 77.795s
=== internal_console@1 (44 s)
ok github.com/carlosframework/platform/internal/console 42.977s
=== internal_console@2 (41 s)
ok github.com/carlosframework/platform/internal/console 39.843s
=== internal_console@3 (41 s)
ok github.com/carlosframework/platform/internal/console 39.378s
=== internal_console@4 (44 s)
ok github.com/carlosframework/platform/internal/console 42.300s
=== internal_console@5 (42 s)
ok github.com/carlosframework/platform/internal/console 40.228s
=== internal_console@6 (38 s)
ok github.com/carlosframework/platform/internal/console 36.340s
=== internal_console@7 (47 s)
ok github.com/carlosframework/platform/internal/console 45.353s
=== root@0 (110 s)
ok github.com/carlosframework/platform 108.365s
=== root@1 (84 s)
ok github.com/carlosframework/platform 82.856s
=== root@2 (92 s)
ok github.com/carlosframework/platform 90.403s
FAILED: rest
make: *** [Makefile:19: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/specs/2026-10-24-edge-rate-limit-design.md b/docs/superpowers/specs/2026-10-24-edge-rate-limit-design.md| new file mode 100644 |
| index 000000000..aa695675d |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-24-edge-rate-limit-design.md |
| @@ -0,0 +1,338 @@ |
| +# Edge rate limit: fair share under pressure, set per instance |
| + |
| +**Date:** 2026-10-24 · **Status:** draft, awaiting Paul's read · **Requested |
| +by:** Paul, from amadan carlos/platform#358 ("Can we take |
| +carlos/platform#358 and make sure it's configurable per instance?"). |
| + |
| +**Builds on:** the password gate's route-property path |
| +(`docs/superpowers/specs/2026-09-07-password-gate-design.md` §2), the edge's |
| +trusted client address (`clientIP`, `internal/edge/analytics.go:76`), the |
| +plan limits box hook (`internal/host/plans_pass.go`, `SetIdleFor`), and the |
| +usage meter (`internal/edge/usagemeter.go`, `usage.EdgeRow`). |
| + |
| +Facts below about current code were checked on origin/main 9ccf6b4f. |
| + |
| +## 1. What this is for |
| + |
| +One client should never be able to monopolise an app instance. A busy |
| +instance should share its capacity fairly between the clients hitting it, |
| +and an outright flood from one address should be turned away at the edge |
| +before it reaches the instance or wakes it. |
| + |
| +It is **not** a cap on legitimate traffic. A real on-sale rush, where |
| +thousands of buyers each load a page and some assets, must reach the instance |
| +at full speed. So must many customers behind one shared address: a school, an |
| +office, a mobile carrier's NAT, a partner's server calling an API on behalf |
| +of its users. |
| + |
| +Why now (from #358): Tito is raising per-instance throughput on event pages |
| +(about 46 req/s today, target 160–320). As the instance gets faster, the gap |
| +between what one client can push and what the instance can serve is the |
| +thing protecting everyone else on it. That protection belongs in the |
| +platform, not in each app. |
| + |
| +## 2. Decisions Paul made (2026-10-23/24, in conversation) |
| + |
| +1. **Configurable per instance.** Each instance (each route host) carries its |
| + own settings. One setting per instance, not per route rule in general. |
| +2. **Different limits for some paths, e.g. API requests.** An instance's |
| + settings may carry a few path-prefix exceptions (such as `/api/`) with |
| + their own numbers. The reason: integrations and scripts legitimately call |
| + an API harder than a person browses. |
| +3. **Both kinds of paid tier.** The app owner sets per-instance numbers to |
| + match its own tiers (Tito moving a customer to its top tier). Where the |
| + platform runs a plan catalog (carloku.com), the account's plan also |
| + bounds what the owner can set. "Tito runs its own platform, so it is in |
| + full control": a self-hosted platform has no plan layer and the owner |
| + sets any value. |
| +4. **Never penalise many customers behind one IP pre-emptively.** The |
| + per-client limit only bites when the instance is actually under |
| + pressure. When the instance is quiet, a shared address sends as much as |
| + it likes. Under pressure, the heaviest clients are shed first. A very |
| + high fixed ceiling sits underneath as a backstop against outright floods. |
| +5. **Not keyed by API key or any other request header.** The edge cannot |
| + verify an app's credentials, and a client could rotate fake ones. The |
| + key is the client address only (from the decision in 4, and #358). |
| + |
| +From #358 itself, unchanged: over the limit the edge answers `429` with |
| +`Retry-After` itself; IPv6 is keyed by its /64; the client address is the |
| +one the edge already trusts, never a header a client can set; rejections are |
| +counted per host somewhere readable. |
| + |
| +## 3. Goals, non-goals, anti-goals |
| + |
| +**Goals** |
| + |
| +- Fair sharing of a busy instance between client addresses. |
| +- A backstop that stops a single-address flood whether or not the instance |
| + is busy. |
| +- Per-instance settings, with path-prefix exceptions, written through a |
| + carlos verb and the console API, the way `carlos gate` writes the gate. |
| +- An optional plan ceiling that only exists where a plan catalog is loaded. |
| +- Per-host counts of what was refused, split by why. |
| + |
| +**Non-goals (v1)** |
| + |
| +- Distributed floods from many addresses. Fair share per address cannot stop |
| + them and this design does not try. That is DDoS mitigation. |
| +- State shared between boxes. Each instance has one home box and every |
| + request for it ends on that box's edge (§5.1), so local state is complete. |
| +- Queueing requests instead of refusing them. |
| +- An app-wide default that instances inherit (§9, question for Paul). |
| +- Keying by credential, cookie or session. |
| + |
| +**Anti-goals: it fails if any of these happen, however well it works |
| +otherwise** |
| + |
| +- A shared address is refused while the instance it is calling is idle or |
| + comfortably within capacity (outside the backstop, which is set far above |
| + anything a shared address does in practice). |
| +- A refused request wakes a sleeping instance or reaches it at all. |
| +- A client can choose its own key by setting a header. |
| +- A self-hosted platform needs a plan catalog, an AWS service or any |
| + carloku.com-specific piece to use it (provider-neutral core). |
| +- Tito, or any app, ends up building its own capacity limiter because this |
| + one could not be configured to fit. |
| + |
| +## 4. How it behaves |
| + |
| +### 4.1 The client key |
| + |
| +`clientIP(r)` (`internal/edge/analytics.go:76-91`) is already the trusted |
| +address: the last `X-Forwarded-For` element only on a signed cross-box hop |
| +(`hopVerified`), otherwise `RemoteAddr`'s host. Unverified forwarding |
| +headers are stripped (`homemap.go:1110-1115`). The key is that address, |
| +truncated to /64 for IPv6, /32 for IPv4. |
| + |
| +Edge-local door callers (`door.go:124`, whose address is the box itself) are |
| +never limited. |
| + |
| +### 4.2 Busy |
| + |
| +The edge counts, per host, the requests it currently has outstanding to the |
| +instance: forwarded and not yet answered, including requests waiting on a |
| +wake. Today only the activator counts in-flight requests, privately, and only |
| +for hibernating routes (`activator.go:518-525`, `1541`, `1704-1718`); always-on |
| +routes go straight to the proxy uncounted. So the edge needs its own per-host |
| +counter on the path to both. |
| + |
| +An instance is **busy** while that count is at or above its `busy_at` |
| +setting. |
| + |
| +Upgraded connections (WebSocket) are judged when the handshake arrives but |
| +do not hold an in-flight slot once upgraded. Otherwise a page with a few |
| +hundred open sockets would read as permanently busy. |
| + |
| +### 4.3 Fair share (only while busy) |
| + |
| +While the instance is busy, a new request from a client that already holds |
| +`client_share` or more of the instance's in-flight slots is refused with |
| +`429` and `Retry-After: 1`. Clients holding fewer slots go through. |
| + |
| +The effect: the heaviest clients are refused first, and only while the |
| +instance is actually struggling. A shared address with real users is |
| +refused only if it is itself one of the clients crowding the instance at |
| +that moment. |
| + |
| +*My call:* fair share counts concurrent requests, not a request rate. |
| +Concurrency is what the instance actually runs out of. It also catches slow |
| +requests that a rate counter would wave through, and it needs no window or |
| +clock to tune. If the builder finds a reason concurrency cannot express what |
| +§2 asks for, raise it before switching to a rate. |
| + |
| +### 4.4 Backstop (always) |
| + |
| +Each client key has a token bucket per host: `rate` requests per second |
| +sustained, `burst` on top. A request with no token is refused with `429` |
| +and a `Retry-After` of the seconds until the next token, at least 1. This |
| +applies whether or not the instance is busy, and it applies to requests for |
| +a sleeping instance, which are refused without a wake. |
| + |
| +The defaults are set high enough that only a flood reaches them (§4.6). |
| + |
| +### 4.5 Path exceptions |
| + |
| +An instance's settings may carry up to 8 path rules, each a prefix starting |
| +with `/` and its own `client_share`, `rate` and `burst`. The longest |
| +matching prefix wins; no match uses the instance's own values. `busy_at` is |
| +instance-wide, since the instance has one capacity whichever path is |
| +called. |
| + |
| +A request is counted against its rule's bucket and share only, so heavy API |
| +use does not use up a client's page-browsing allowance. |
| + |
| +*My call:* prefixes match on the URL path before any rewriting, compared |
| +byte for byte, case-sensitive. A prefix of `/api/` does not match `/api`. |
| + |
| +### 4.6 Defaults |
| + |
| +*My call, all to be checked by the builder against a benchmark of a real |
| +instance before they ship:* |
| + |
| +| Setting | Default | Reasoning | |
| +| --- | --- | --- | |
| +| `busy_at` | 64 in-flight | well above what a healthy instance holds at Tito's target (300 req/s × ~50 ms ≈ 15) | |
| +| `client_share` | 16 in-flight | a browser opens at most ~6 connections per origin; 16 leaves room for a NAT with several heavy users | |
| +| `rate` | 300 req/s | only a flood sustains this from one address | |
| +| `burst` | 1,000 | a busy office loading a page with assets all at once | |
| + |
| +### 4.7 Where in the edge |
| + |
| +The check runs after the cache lookup, immediately before the request would |
| +go to the activator or the proxy: `edge.go` around 1784-1807, and on the |
| +upgrade path at 1721-1729, which skips the cache. So: |
| + |
| +- Cache hits are never limited. They cost the instance nothing, and #358 is |
| + about protecting the instance. |
| +- The password gate's own pages are never limited by this. The gate keeps |
| + its own login limiter (`internal/gate/limiter.go`). |
| +- The refusal is inside the meter wrapper (1665-1686), so it is counted in |
| + usage and seen by analytics. It is not an edge error (those are ≥ 500). |
| +- Nothing before that point wakes the instance (the gate comment at 1703 and |
| + the activator call sites at 1722 and 1784 confirm this). |
| + |
| +A `429` from this limiter is never stored in the cache. |
| + |
| +### 4.8 What the client sees |
| + |
| +A `429` with `Retry-After`, `Cache-Control: no-store`, and a short plain-text |
| +body. The body is user-facing copy and goes through copy review before it is |
| +written. No HTML page: callers may be scripts. |
| + |
| +## 5. Where the settings live and how they travel |
| + |
| +### 5.1 Home box only |
| + |
| +One edge runs per box, inside the agent (`internal/host/agent.go:926`). Each |
| +instance host has one home box (`homemap.go:700-705`). Requests that land on |
| +another box, through the region load balancer or cross-region DNS, are |
| +forwarded once to the home box over a signed hop that carries the client |
| +address (`edge.go:1526-1530`, `1561`; `homemap.go:1196-1232`). The sending |
| +edge has no route row for the host (`edge.go:1660-1661`). |
| + |
| +So the limiter runs on the home box's edge only, keyed by the hop-carried |
| +address. Its state is in memory and is lost on restart, which is acceptable: |
| +both the busy count and the buckets refill within seconds anyway. |
| + |
| +**Fleet tunnels are the open case.** A fleet-tunnelled request passes the hub |
| +edge and then the fleet box's edge, and on the fleet side `clientIP` is the |
| +tunnel peer (the hub), not the client (`FromTunnel`; |
| +`internal/tunnelclient/tunnelclient.go:291-293`). The builder must establish |
| +which edge has both the route row and the real client address, and limit |
| +there. If neither does, v1 does not limit fleet-tunnelled routes, and the |
| +spec says so in the docs, rather than keying every client as the hub. |
| + |
| +### 5.2 The path, copying the password gate |
| + |
| + carlos ratelimit (CLI) / console API |
| + → PATCH /api/cli/apps/{account}/{app}/instances/{host} on the instance record |
| + → reconcile converges it into a registry column |
| + → the edge reads it at reload and enforces it per request |
| + |
| +- **Record:** a new `RateLimit` field on `provision.Record` |
| + (`internal/provision/provision.go:38`), JSON `rate_limit`, `omitempty`. |
| + Empty means platform defaults. Shape checked in `ValidateShape`. |
| +- **Console door:** `handleAPIInstanceUpdate` |
| + (`internal/console/apiinstances.go:1939`) takes it as a pointer field, the |
| + way it takes `Gate`, with a way to clear back to defaults. Who may set it: |
| + whoever may set the gate today. |
| +- **Registry:** a new `routes` column holding the canonical JSON, with the |
| + gate column's full treatment: migration, select and scan, a narrow |
| + `SetRateLimit` setter plus its remote-dispatch twin, `RouteFor` copying it |
| + at mint, the owned-converge arm calling the setter on a difference, and |
| + **inclusion in `Fingerprint()`** so the edge reloads within its 2 s watch |
| + (`edge.go:2315-2369`). |
| +- **Census:** editing `handleAPIInstanceUpdate` changes its pinned digest in |
| + `internal/provision/plan9_b8_census_test.go`, and a new record field is a |
| + record-write change, so the branch carries the usual census overlay and |
| + #280 ruling request. |
| + |
| +### 5.3 The plan ceiling (carloku.com only) |
| + |
| +`plans.Limits` (`internal/plans/catalog.go:26-40`) gains optional ceilings |
| +for `client_share`, `rate` and `burst`. Missing means no ceiling. They are |
| +**not** pack-additive. |
| + |
| +- **At the console door:** a value above the account's effective ceiling is |
| + refused with a message naming the ceiling. The message is copy and goes |
| + through review. |
| +- **On the box:** the edge clamps each value to the ceiling at the moment it |
| + applies it, through a hook backed by `planLimitsFor`, the same pattern as |
| + `SetIdleFor` (`internal/host/plans_pass.go:52-59`, `124-131`; |
| + `agent.go:502`). That covers a plan that drops after the value was set. |
| +- **With no catalog** (`control/plans.json` absent, or |
| + `CARLOS_PLAN_LIMITS=off`), there is no ceiling, and the owner's values |
| + apply as written. This is the self-hosted case, and it must not change |
| + behaviour or cost more than one marker read per pass, which `plansPass` |
| + already pays. |
| + |
| +`busy_at` has no plan ceiling: it decides when the instance's own capacity |
| +is shared, not how much the box gives it. |
| + |
| +*My call:* a system operator (#176) gets no override past a plan in v1. A |
| +sponsored account already gets generous limits through `generous()`. |
| + |
| +### 5.4 The verb |
| + |
| +A carlos verb shows and sets an instance's settings, and clears them back to |
| +defaults. Shape it after `carlos gate` (`gate_cli.go`, `apimode.go:3604`), |
| +including the "console too old" mapping for a 400 "nothing to update". |
| +Showing an instance's settings also shows the effective values after the |
| +plan ceiling, so an owner can tell their number from the one being applied. |
| + |
| +*Name:* `carlos ratelimit` is a suggestion. The verb name, its flags and its |
| +help text are user-facing copy, so they go through copy review. |
| + |
| +The console web UI's instance settings page gets the same fields, following |
| +the gate's writer there (`internal/console/appsettings.go:982`). |
| + |
| +## 6. Observability |
| + |
| +- **Usage counters:** `usage.EdgeRow` gains `RateShed` (refused by fair |
| + share) and `RateCeiling` (refused by the backstop), additive and |
| + `omitempty`, drained per host per day with the rest of the edge row |
| + (`internal/usage/edge.go:15-34`). Two counters, because they mean |
| + different things: shedding is the design working under load, while |
| + backstop refusals on a real customer's address mean the defaults are |
| + wrong. |
| +- **Log:** one throttled line per host and reason through `logThrottled` |
| + (`edge.go:1046`), naming the host, the reason and the client key, so an |
| + operator can tell when a real customer is being limited. |
| +- **Where an owner reads it:** the verb from §5.4 shows the last day's two |
| + counts for the instance, read from the usage files the console already |
| + reads. |
| + |
| +## 7. Rollout |
| + |
| +The limiter ships behind a box setting, `off | observe | enforce`. |
| + |
| +- `observe` evaluates every request and counts and logs what it would |
| + refuse, without refusing anything. |
| +- The first release ships with `observe` as the default. Paul switches a |
| + platform to `enforce` after reading the counts, dev first and then |
| + flagship (the usual roll order). |
| + |
| +This is a kill switch and a rollout mode. It is not where the numbers live: |
| +those stay on the instance record, as #358 asked. |
| + |
| +## 8. What's left to the builder |
| + |
| +- Where exactly the per-host in-flight counter sits so it covers the |
| + activator path and the proxy path without double-counting a woken request. |
| +- The limiter's data structure and its memory bound. It must survive an |
| + address-spraying client without unbounded growth, the way the gate's |
| + limiter prunes at 10,000 keys and the CLI limiter caps at 8,192. |
| + Evicting idle keys first is fine; evicting a key that holds slots is not. |
| +- Whether `golang.org/x/time/rate` is worth adding or the repo's existing |
| + token buckets (`internal/gate/limiter.go`, `certsync.go:1943`) cover it. |
| + It is not in go.mod today. |
| +- The fleet-tunnel case (§5.1). |
| +- Benchmarking the defaults (§4.6) on a real instance. |
| +- Tests through real wiring: a request travelling the actual `ServeHTTP` |
| + path, a backhauled request keyed by its hop address, a sleeping instance |
| + that is not woken by a refused request, and a converge that changes a |
| + setting and is enforced after the reload. |
| + |
| +## 9. Open for Paul |
| + |
| +(Filled in after the builder check.) |