amadan

carlos / platform Public

reqrec: mask paths and query names by an allowlist of harmless segment shapes, input bounded to 2048 bytes (masking round, Paul's ruling)

Token detection kept slash-split base64 about a third of the time and
allocated per segment of an unbounded path (88 MB per 1 MiB). Stored
paths now keep only short segments and lower-case words, with three
neighbour guards for a token's short pieces; query names take the same
shape test. Masking reads at most 2048 bytes and walks the path in place
(576 bytes per call for a 1 MiB path). A property test over 10,000
random tokens per kind measures what survives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude e48b542fa6a23d44354ee3b0905d70ad9765d308 parent 09f6ae7
6 files changed, +533 −47
  • docs/superpowers/plans/2026-10-24-edge-protection-1-request-records.md +3 −0
  • docs/superpowers/specs/2026-10-24-edge-request-protection-design.md +12 −5
  • internal/edge/reqrecords_test.go +0 −1
  • internal/host/reqrec_e2e_test.go +6 −3
  • internal/reqrec/redact.go +315 −14
  • internal/reqrec/redact_test.go +197 −24
diff --git a/docs/superpowers/plans/2026-10-24-edge-protection-1-request-records.md b/docs/superpowers/plans/2026-10-24-edge-protection-1-request-records.md
index 52ed35dfa..a195e6f5f 100644
--- a/docs/superpowers/plans/2026-10-24-edge-protection-1-request-records.md
+++ b/docs/superpowers/plans/2026-10-24-edge-protection-1-request-records.md
@@ -26,6 +26,9 @@ code follows this list where a task says otherwise:
platform lines.
- The stored method is `GET`..`TRACE` or `OTHER`; request text has controls
escaped; records carry `env`; QueryNames drops components with no `=`.
+- Paths and query names are masked by an allowlist of harmless segment
+ shapes (short, or lower-case words) with neighbour guards, not by token
+ detection; masking reads at most 2048 bytes of input (`MaxMaskInput`).
- The edge leaves `Response.When` zero and the recorder stamps it;
`newRequestRecorder` takes a `now` clock.
- The edge-refs answer always carries `record_status`: `found`, `absent` or
diff --git a/docs/superpowers/specs/2026-10-24-edge-request-protection-design.md b/docs/superpowers/specs/2026-10-24-edge-request-protection-design.md
index 32a601414..ae50a178f 100644
--- a/docs/superpowers/specs/2026-10-24-edge-request-protection-design.md
+++ b/docs/superpowers/specs/2026-10-24-edge-request-protection-design.md
@@ -487,9 +487,15 @@ backtested.
URLs carry their secret in exactly that shape.
- What can't be classified is masked, not kept. Calibration loses a little
evidence; nobody's secret lands in a log. In practice (final review,
- 2026-10-25): any run of 24 or more such characters is masked whatever it
- holds, a standard-base64 token split into path segments at `/` is judged
- joined, and a query component with no `=` is left out of the names.
+ 2026-10-25, Paul's ruling): stored paths and query names keep only
+ segments of a plainly harmless shape and mask every other segment whole.
+ The harmless shapes are a segment of at most 8 bytes, or lower-case words
+ (`[a-z0-9._~@+-]`, at most 64 bytes, each word letters only up to 20,
+ digits only up to 8, or a mix up to 6). Guards mask short pieces that
+ sit beside a token's masked pieces, since a standard-base64 token is cut
+ into segments by its own `/`. The input is cut to 2048 bytes before any
+ of this. A query component with no `=` is left out of the names.
+ Fragments keep the run masking described above.
- Control characters in recorded request text (path, query names,
fragment) are stored escaped as `\xNN`, so a record cannot forge a log
line or drive the terminal it is printed on. The method is stored as one
@@ -504,8 +510,9 @@ backtested.
watches cannot use up the blocks' allowance. Minute records carry the
status classes, per-method counts (`m_get`, `m_head`, `m_post`, `m_put`,
`m_patch`, `m_delete`, `m_options`, `m_other`), wakes, blocks, watches and
- the dropped counts. Every record carries the route's environment (`env`)
- when it has one.
+ the dropped counts. Every not-found, block and watch record carries the
+ route's environment (`env`) when it has one. Minute records carry none:
+ their counts are per app, across its environments.
- **Shipping** uses the existing per-app log pipeline (`internal/applog`,
`StreamEdge`) and its budgets, so they are readable with `carlos logs` and in
the console. Retention is the logs family's 7 days (`logshipMaxAge`, `internal/host/logship.go`), changed at plan time from 14 because retention is per family and the calibration report can read two 7-day windows. Edge records have their own rate budget in the sink so they never crowd out the app's own lines, and in the sink's buffer they only ever evict each other, never an app or platform line. Minute records bypass the edge rate budget, and the buffer evicts request records first, then blocks, then minute records.
diff --git a/internal/edge/reqrecords_test.go b/internal/edge/reqrecords_test.go
index fe73b612c..16ad3ef9b 100644
--- a/internal/edge/reqrecords_test.go
+++ b/internal/edge/reqrecords_test.go
@@ -66,7 +66,6 @@ func TestRecordMatchReturnsSealedCode(t *testing.T) {
e.EnableRequestRecords(rec)
rt := registry.Route{Host: "app.test", App: "shop"}
rt.Environment = "staging"
- rt.Environment = "staging"
code := e.recordMatch(rt, httptest.NewRequest("GET", "http://app.test/.env", nil), reqrec.KindBlock, reqrec.TierHard, "dotenv", "", 404)
r, err := reqrec.ParseRef(code, key)
if err != nil || !r.Verified || len(rec.match) != 1 || rec.match[0].Ref != code {
diff --git a/internal/host/reqrec_e2e_test.go b/internal/host/reqrec_e2e_test.go
index e083f9989..3a10edf3d 100644
--- a/internal/host/reqrec_e2e_test.go
+++ b/internal/host/reqrec_e2e_test.go
@@ -220,14 +220,17 @@ func TestRequestRecordsHostileInputEndToEnd(t *testing.T) {
secrets := []string{"rKpWmTzQvLsNxYhJbCfDaGeS", "Xq3vB9kLmN2pQr7s", "T1uVwYz4aBcDeFg", "abcdefabcdefabcdefabcdefabcdefab", "hunter2"}
paths := map[string]string{
"/reset/rKpWmTzQvLsNxYhJbCfDaGeS": "/reset/[masked]",
- "/reset/Xq3vB9kLmN2pQr7s/T1uVwYz4aBcDeFg": "/reset/[masked]",
+ "/reset/Xq3vB9kLmN2pQr7s/T1uVwYz4aBcDeFg": "/reset/[masked]/[masked]",
"/zabcdefabcdefabcdefabcdefabcdefab": "/[masked]",
}
for p := range paths {
x.serve("GET", p, client)
}
x.serve("GET", "/login?hunter2", client)
- x.serve("GET", "/x%1b%5b2J%0aforged?a%0d=1", client)
+ // Short segments survive masking, so their controls must be escaped;
+ // a long one with controls in it is masked whole.
+ x.serve("GET", "/x%1b/p/%0a?a%0d=1", client)
+ x.serve("GET", "/forged%0a2026-10-24T21:07:30Z%20info%20login%20ok", client)
huge := strings.Repeat("&", 100000)
x.serve(huge, "/m", client)
x.flush()
@@ -274,7 +277,7 @@ func TestRequestRecordsHostileInputEndToEnd(t *testing.T) {
if r, ok := seen["/login"]; !ok || r.Fields["qnames"] != "" {
t.Errorf("bare query component kept: %+v", r.Fields)
}
- if r, ok := seen[`/x\x1b[2J\x0aforged`]; !ok || r.Fields["qnames"] != `a\x0d` {
+ if r, ok := seen[`/x\x1b/p/\x0a`]; !ok || r.Fields["qnames"] != `a\x0d` {
t.Errorf("control path not escaped: have %v", mapKeys(seen))
}
if r, ok := seen["/m"]; !ok || r.Fields["method"] != "OTHER" || r.Msg != "404 OTHER /m" {
diff --git a/internal/reqrec/redact.go b/internal/reqrec/redact.go
index 661260c7c..d1f89471e 100644
--- a/internal/reqrec/redact.go
+++ b/internal/reqrec/redact.go
@@ -23,6 +23,11 @@ const (
MaxQueryNames = 256
// MaxFragment is the most bytes of one evidence fragment a record keeps.
MaxFragment = 64
+ // MaxMaskInput is the most bytes of a path, query string or fragment
+ // value that masking reads. Request text can be any length; cutting it
+ // first bounds the work and memory of every record, and whatever lies
+ // past the cut is never stored.
+ MaxMaskInput = 2048
)
// minRun is the shortest run that can be a token or an id. Hyphenated slugs
@@ -246,31 +251,324 @@ func truncate(s string, n int) string {
}
// clean makes s valid UTF-8, masks secret-looking runs, escapes controls and
-// caps the length. The cap comes last so escaping cannot push s past it.
+// caps the length. Fragments use it. The input is cut to MaxMaskInput first,
+// and the cap comes last so escaping cannot push s past it.
func clean(s string, max int) string {
- s = strings.ToValidUTF8(s, "�")
+ s = strings.ToValidUTF8(truncate(s, MaxMaskInput), "�")
return truncate(escapeControls(maskRuns(s)), max)
}
+// Stored paths and query names do not try to detect tokens: every segment
+// whose shape is not plainly harmless is masked whole (spec §7, "what can't
+// be classified is masked, not kept"). Detection kept a split base64 token
+// about a third of the time, because no length or character-class test can
+// tell a short random piece from a word. The harmless shapes:
+//
+// - short: at most maxShortSegment bytes, whatever it holds (after
+// control escaping). A piece that short cannot carry a usable secret on
+// its own, and probes are full of such names (.env, HNAP1, GponForm).
+// - lower-case words: only [a-z0-9._~@+-], at most maxWordSegment bytes,
+// and every word between '.', '-', '_' and '~' is letters only (at most
+// 20), digits only (at most 8), or a mix of at most 6 (v2, mp4, h264).
+// That is a slug, a file name or a version. A word holding '@' or '+'
+// is none of these, so such a segment is kept only when it is short.
+//
+// A short piece is harmless alone, not beside the rest of its token, so
+// three guards look at a short segment's neighbours. They only ever mask
+// segments of at most maxShortSegment bytes; the first two only those kept
+// for being short rather than word-shaped:
+//
+// 1. Two or more consecutive short segments that each mix letters with
+// digits or upper with lower case, with minSplitToken or more bytes
+// between them, are masked together: a token split at '/'.
+// 2. A short segment that is opaque (anything but letters in one case or
+// Title Case: PHP and Util are words, e6Hzgpqw, ON2++A== and HNAP1 are
+// not) is masked when a neighbour is masked, or is short and not a
+// lower-case or Title Case word (opaque, ALL CAPS, or with digits:
+// r4v/UxSrhmBt/LNR is a token, @fs/etc a probe). A base64
+// token split at '/' leaves its short pieces beside its masked long
+// ones; without this, "[masked]/ON2++A==" kept eight characters of the
+// token and its '/'. A probe name alone (/HNAP1, /GponForm) is kept.
+// 3. After a masked segment, each following segment of at most
+// maxShortSegment bytes is masked too, until one that is a plain word
+// (lower-case words of letters only, three letters or more: items,
+// x.png). A token's later pieces follow its first masked piece through
+// short bridges such as "c", "K" or "m3y6"; a path's own words after a
+// masked id are kept. Backwards from a masked segment the same runs,
+// but it also stops at any lower-case word (the u in /u/<id>) or Title
+// Case word, so /Account/[masked] keeps Account, while /oIddqq/8/[masked]
+// and /d7cG/DOPC/[masked] lose both pieces.
+//
+// Measured on 10,000 random tokens of each kind (TestMaskPathTokensProperty):
+// the guards bring the share of standard-base64 tokens leaving a 9-character
+// piece of themselves from 4-5% (the shapes alone) to 3-12 in 10,000, every
+// one of them through a piece that is itself a word (vqk, Wwhjj). Hex,
+// UUID and base64url tokens leave none.
+const (
+ maxShortSegment = 8
+ maxWordSegment = 64
+ maxWordLetters = 20
+ maxWordDigits = 8
+ maxWordMixed = 6
+ minSplitToken = 16
+)
+
+// wordShaped reports whether seg is the lower-case word shape above.
+func wordShaped(seg string) bool {
+ if len(seg) > maxWordSegment {
+ return false
+ }
+ letters, digits, other := 0, 0, false
+ word := func() bool {
+ n := letters + digits
+ ok := n == 0 || !other && (digits == 0 && letters <= maxWordLetters ||
+ letters == 0 && digits <= maxWordDigits || n <= maxWordMixed)
+ letters, digits, other = 0, 0, false
+ return ok
+ }
+ for i := 0; i < len(seg); i++ {
+ switch c := seg[i]; {
+ case c >= 'a' && c <= 'z':
+ letters++
+ case c >= '0' && c <= '9':
+ digits++
+ case c == '.' || c == '-' || c == '_' || c == '~':
+ if !word() {
+ return false
+ }
+ case c == '@' || c == '+':
+ other = true
+ default:
+ return false
+ }
+ }
+ return word()
+}
+
+// mixedShort reports whether seg mixes letters with digits or upper case
+// with lower case: guard 1's test.
+func mixedShort(seg string) bool {
+ c := classesOf(seg)
+ return c&classDigit != 0 && c&(classUpper|classLower) != 0 || c&classUpper != 0 && c&classLower != 0
+}
+
+// opaqueShort reports whether a non-empty seg is anything but ASCII letters
+// in one case or in Title Case: guard 2's test.
+func opaqueShort(seg string) bool {
+ if seg == "" {
+ return false
+ }
+ upperAfterFirst, lower := false, false
+ for i := 0; i < len(seg); i++ {
+ switch c := seg[i]; {
+ case c >= 'a' && c <= 'z':
+ lower = true
+ case c >= 'A' && c <= 'Z':
+ if i > 0 {
+ upperAfterFirst = true
+ }
+ default:
+ return true
+ }
+ }
+ // One case throughout, or one capital then lower case.
+ return upperAfterFirst && lower
+}
+
+// keepSegment reports whether one path segment or query name has a harmless
+// shape on its own. "", "." and ".." are short, so they are kept.
+func keepSegment(seg string) bool { return len(seg) <= maxShortSegment || wordShaped(seg) }
+
+// Per-segment marks for MaskPath's walk.
+const (
+ segMask = 1 << iota // masked
+ segMixed // kept for being short, and guard 1's mix
+ segOpaque // kept for being short, and guard 2's opaque
+ segPlain // a plain word: guard 3 stops there
+ segTitle // a lower-case or Title Case word: guard 3 also stops there going back
+)
+
+// lowerWord reports whether seg is word-shaped and holds lower-case letters
+// but no digits (c, mh, x.png).
+func lowerWord(seg string) bool {
+ return wordShaped(seg) && strings.ContainsAny(seg, "abcdefghijklmnopqrstuvwxyz") && !strings.ContainsAny(seg, "0123456789")
+}
+
+// titleWord reports whether seg is one capital and two or more lower-case
+// letters (Account, Util).
+func titleWord(seg string) bool {
+ if len(seg) < 3 || seg[0] < 'A' || seg[0] > 'Z' {
+ return false
+ }
+ for i := 1; i < len(seg); i++ {
+ if seg[i] < 'a' || seg[i] > 'z' {
+ return false
+ }
+ }
+ return true
+}
+
+// plainWord reports whether seg is word-shaped with letters only and at
+// least three of them: guard 3's stop.
+func plainWord(seg string) bool {
+ letters := 0
+ for i := 0; i < len(seg); i++ {
+ switch c := seg[i]; {
+ case c >= 'a' && c <= 'z':
+ letters++
+ case c == '.' || c == '-' || c == '_' || c == '~':
+ default:
+ return false
+ }
+ }
+ return letters >= 3 && wordShaped(seg)
+}
+
// MaskPath returns a request path that is safe to store: valid UTF-8, with
-// token-like runs masked and controls escaped, and at most MaxPath bytes.
+// every segment of an unclassified shape masked whole (see wordShaped),
+// controls escaped, and at most MaxPath bytes. Only the first MaxMaskInput
+// bytes are read. The walk marks each segment in a fixed array (the cut
+// bounds how many there can be), applies the guards, then writes the
+// result once; nothing is allocated per segment.
func MaskPath(p string) string {
- return clean(p, MaxPath)
+ p = escapeControls(strings.ToValidUTF8(truncate(p, MaxMaskInput), "\ufffd"))
+ if len(p) > 4*MaxMaskInput {
+ // Escaping can quadruple the bytes; the marks array covers that.
+ p = truncate(p, 4*MaxMaskInput)
+ }
+ var marks [4*MaxMaskInput + 1]uint8
+ var lens [4*MaxMaskInput + 1]uint16
+ n := 0
+ for start, i := 0, 0; i <= len(p); i++ {
+ if i < len(p) && p[i] != '/' {
+ continue
+ }
+ seg := p[start:i]
+ var m uint8
+ switch {
+ case wordShaped(seg):
+ if plainWord(seg) {
+ m = segPlain
+ }
+ case len(seg) <= maxShortSegment:
+ if mixedShort(seg) {
+ m |= segMixed
+ }
+ if opaqueShort(seg) {
+ m |= segOpaque
+ }
+ default:
+ m = segMask
+ }
+ if titleWord(seg) || lowerWord(seg) {
+ m |= segTitle
+ }
+ marks[n], lens[n] = m, uint16(len(seg))
+ n++
+ start = i + 1
+ }
+ // Guard 1: runs of mixed short segments.
+ for i := 0; i < n; {
+ if marks[i]&segMixed == 0 {
+ i++
+ continue
+ }
+ j, total := i, 0
+ for j < n && marks[j]&segMixed != 0 {
+ total += int(lens[j])
+ j++
+ }
+ if j-i >= 2 && total >= minSplitToken {
+ for k := i; k < j; k++ {
+ marks[k] |= segMask
+ }
+ }
+ i = j
+ }
+ // Guard 2: an opaque short segment beside a masked one or a weak one: a
+ // non-empty segment of at most maxShortSegment bytes that is not a
+ // lower-case or Title Case word (opaque, ALL CAPS, or holding digits).
+ // It reads the marks as guard 1 left them, so it does not cascade.
+ weak := func(k int) bool {
+ return lens[k] > 0 && lens[k] <= maxShortSegment && marks[k]&segTitle == 0
+ }
+ var opaqueMasked [4*MaxMaskInput + 1]bool
+ for i := 0; i < n; i++ {
+ if marks[i]&segOpaque == 0 || marks[i]&segMask != 0 {
+ continue
+ }
+ for _, k := range [2]int{i - 1, i + 1} {
+ if k >= 0 && k < n && (marks[k]&segMask != 0 || weak(k)) {
+ opaqueMasked[i] = true
+ }
+ }
+ }
+ // Guard 3: forwards from a masked segment through short segments that
+ // are not plain words, then backwards, where lower-case and Title Case
+ // words (u, Account) also stop.
+ // Empty segments are passed over and kept: base64 can hold "//".
+ masked := func(i int) bool { return marks[i]&segMask != 0 || opaqueMasked[i] }
+ carry := false
+ for i := 0; i < n; i++ {
+ if lens[i] == 0 {
+ continue
+ }
+ if carry && lens[i] <= maxShortSegment && marks[i]&segPlain == 0 {
+ marks[i] |= segMask
+ }
+ carry = masked(i)
+ }
+ carry = false
+ for i := n - 1; i >= 0; i-- {
+ if lens[i] == 0 {
+ continue
+ }
+ if carry && lens[i] <= maxShortSegment && marks[i]&(segPlain|segTitle) == 0 {
+ marks[i] |= segMask
+ }
+ carry = masked(i)
+ }
+ var b strings.Builder
+ b.Grow(min(len(p), MaxPath) + len(MaskToken))
+ for k, start, i := 0, 0, 0; i <= len(p) && b.Len() <= MaxPath; i++ {
+ if i < len(p) && p[i] != '/' {
+ continue
+ }
+ if k > 0 {
+ b.WriteByte('/')
+ }
+ if marks[k]&segMask != 0 || opaqueMasked[k] {
+ b.WriteString(MaskToken)
+ } else {
+ b.WriteString(p[start:i])
+ }
+ k++
+ start = i + 1
+ }
+ return truncate(b.String(), MaxPath)
}
// QueryNames returns the parameter names of a raw query string, comma-joined,
// in first-seen order and without repeats. Values are never kept. A component
// with no '=' (?hunter2) is left out: it has no name, and what it carries may
-// be the value itself. Names are masked and their controls escaped like a
-// path. A name that would push the result past MaxQueryNames bytes is dropped
-// whole.
+// be the value itself. Each name has its controls escaped and is kept only
+// when it has a path segment's harmless shape (keepSegment), else it becomes
+// [masked]. Only the first MaxMaskInput bytes of the query are read. A name
+// that would push the result past MaxQueryNames bytes is dropped whole.
func QueryNames(rawQuery string) string {
if rawQuery == "" {
return ""
}
+ rest := truncate(rawQuery, MaxMaskInput)
seen := map[string]bool{}
var out strings.Builder
- for _, pair := range strings.FieldsFunc(rawQuery, func(r rune) bool { return r == '&' || r == ';' }) {
+ for rest != "" && out.Len() < MaxQueryNames {
+ var pair string
+ if i := strings.IndexAny(rest, "&;"); i >= 0 {
+ pair, rest = rest[:i], rest[i+1:]
+ } else {
+ pair, rest = rest, ""
+ }
name, _, ok := strings.Cut(pair, "=")
if !ok {
continue
@@ -278,8 +576,10 @@ func QueryNames(rawQuery string) string {
if un, err := url.QueryUnescape(name); err == nil {
name = un
}
- name = strings.ToValidUTF8(name, "�")
- name = escapeControls(maskRuns(name))
+ name = escapeControls(strings.ToValidUTF8(name, "�"))
+ if !keepSegment(name) {
+ name = MaskToken
+ }
if name == "" || seen[name] {
continue
}
@@ -370,10 +670,11 @@ func Fragment(src Source, name, value string) (string, bool) {
// referer reduces a Referer value to its scheme, host and masked path, with
// the query cut down to parameter names: a referring page's query string often
-// holds a password or an OAuth code. Userinfo and fragment are dropped. A value
-// that does not parse is omitted.
+// holds a password or an OAuth code. The path goes through MaskPath and the
+// names through QueryNames, as a request's own would. Userinfo and fragment
+// are dropped. A value that does not parse is omitted.
func referer(value string) (string, bool) {
- u, err := url.Parse(strings.TrimSpace(value))
+ u, err := url.Parse(strings.TrimSpace(truncate(value, MaxMaskInput)))
if err != nil {
return "", false
}
@@ -386,7 +687,7 @@ func referer(value string) (string, bool) {
b.WriteString("//")
b.WriteString(u.Host)
}
- b.WriteString(u.Path)
+ b.WriteString(MaskPath(u.Path))
if names := QueryNames(u.RawQuery); names != "" {
b.WriteString("?")
b.WriteString(names)
diff --git a/internal/reqrec/redact_test.go b/internal/reqrec/redact_test.go
index ea6bc90f2..3b664a0d0 100644
--- a/internal/reqrec/redact_test.go
+++ b/internal/reqrec/redact_test.go
@@ -1,41 +1,84 @@
package reqrec
import (
+ "encoding/base64"
+ "encoding/hex"
+ "math/rand/v2"
"strings"
"testing"
"unicode/utf8"
)
+// TestMaskPath: a stored path keeps only segments of a plainly harmless shape
+// (short, or lower-case words) and masks every other segment whole.
func TestMaskPath(t *testing.T) {
- cases := map[string]string{
- "/en/articles/how-do-i-access-the-admin-pane-2023": "/en/articles/how-do-i-access-the-admin-pane-2023",
- "/reset/3f9a0c1d2e4b5a6978877665544332211": "/reset/[masked]",
- "/assets/admin/application-0123456789abcdef0123456789abcdef.js": "/assets/admin/application-[masked].js",
- "/i/eyJhbGciOiJIUzI1NiJ9xQ2w9pLmNb7Ks4Ua/x.png": "/i/[masked]/x.png",
- "/wp-login.php": "/wp-login.php",
- "/short1234abcd": "/short1234abcd",
- // Compounds: runs joined by - or _ are judged as one token.
- "/r/Xq3vB9kLmN2pQr7sT1uVw-Yz4aBcDeFgHiJkLmNoPq": "/r/[masked]",
- "/r/Xq3vB9kLmN2pQr7sT1uV_wYz4aBcDeFgHiJkLmNoPq": "/r/[masked]",
+ kept := []string{
+ "/wp-login.php",
+ "/.env",
+ "/.git/config",
+ "/phpinfo.php",
+ "/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php",
+ "/_ignition/execute-solution",
+ "/@fs/etc/passwd",
+ "/HNAP1",
+ "/GponForm",
+ "/en/articles/how-do-i-access-the-admin-pane-2023",
+ "/export-attendees-from-all-events-2023",
+ "/blog/export-attendees-from-all-events-2023",
+ "/api/v2/customers",
+ "/api/v2/users/12345/orders/67890",
+ "/cgi-bin/luci",
+ "/server-status",
+ "/actuator/health",
+ "/wp-content/uploads/2023/08/logo-gold.png",
+ "/static/../.env", // /static/..%2f.env, decoded
+ "/a//b/./c/",
+ "/video/h264/utf8/mp4",
+ }
+ for _, in := range kept {
+ if got := MaskPath(in); got != in {
+ t.Errorf("MaskPath(%q) = %q, want it kept", in, got)
+ }
+ }
+ masked := map[string]string{
+ // The final review's C1 cases.
+ "/reset/rKpWmTzQvLsNxYhJbCfDaGeS": "/reset/[masked]",
+ "/reset/Xq3vB9kLmN2pQr7s/T1uVwYz4aBcDeFg": "/reset/[masked]/[masked]",
+ "/zabcdefabcdefabcdefabcdefabcdefab": "/[masked]",
+ // The re-review's: the long pieces go, a short piece may stay.
+ "/reset/xRL0/6P75ck6doc6ArQGG1qK": "/reset/[masked]/[masked]",
+ "/reset/YwvwSAJbdGtF/r/qYePZZbJxpVBrDWtE": "/reset/[masked]/[masked]/[masked]",
+ // Hex, UUID, base32, and the original reset link.
+ "/t/0123456789abcdef0123456789abcdef": "/t/[masked]",
"/reset/3f9a0c1d-2e4b-5a69-7887-766554433221": "/reset/[masked]",
- "/f/3f9a0c1d-2e4b-5a69-7887-766554433221.png": "/f/[masked].png",
- "/blog/export-attendees-from-all-events-2023": "/blog/export-attendees-from-all-events-2023",
- "/blog/Export-Attendees-From-All-Events": "/blog/Export-Attendees-From-All-Events",
- // Final review C1: a long run with no digit, standard base64 split
- // into short pieces at '/', and long hex beside a non-hex letter.
- "/reset/rKpWmTzQvLsNxYhJbCfDaGeS": "/reset/[masked]",
- "/reset/Xq3vB9kLmN2pQr7s/T1uVwYz4aBcDeFg": "/reset/[masked]",
- "/zabcdefabcdefabcdefabcdefabcdefab": "/[masked]",
- "/a/Xq3vB9kL+mN2pQr7s/T1uVwYz4aBcDeFg=/b.js": "/a/[masked]/b.js",
- // Title-case segments are words, not a split token.
- "/en/Products/iPhone/Accessories/Cases": "/en/Products/iPhone/Accessories/Cases",
- "/api/v2/users/12345/orders/67890": "/api/v2/users/12345/orders/67890",
+ "/f/3f9a0c1d-2e4b-5a69-7887-766554433221.png": "/f/[masked]",
+ "/c/mfrggzdfmztwq2lknnwg23tpoa": "/c/[masked]",
+ "/reset/3f9a0c1d2e4b5a6978877665544332211": "/reset/[masked]",
+ "/i/eyJhbGciOiJIUzI1NiJ9xQ2w9pLmNb7Ks4Ua/x.png": "/i/[masked]/x.png",
+ "/a/Xq3vB9kL+mN2pQr7s/T1uVwYz4aBcDeFg=/b.js": "/a/[masked]/[masked]/b.js",
+ "/r/Xq3vB9kLmN2pQr7sT1uVw-Yz4aBcDeFgHiJkLmNoPq": "/r/[masked]",
+ // Short pieces that only together could be a token (rule 5).
+ "/s/aB3dE6/f6GhI7/j9KlM8/x": "/s/[masked]/[masked]/[masked]/[masked]",
}
- for in, want := range cases {
+ for in, want := range masked {
if got := MaskPath(in); got != want {
t.Errorf("MaskPath(%q) = %q, want %q", in, got, want)
}
}
+ // Over-masking the rule accepts on purpose: no shape test can tell these
+ // from tokens, so they cost their evidence.
+ over := map[string]string{
+ "/Account/ForgotPasswordConfirmation": "/Account/[masked]",
+ "/Dockerfile": "/[masked]",
+ "/short1234abcd": "/[masked]",
+ "/blog/Export-Attendees-From-All-Events": "/blog/[masked]",
+ "/assets/admin/application-0123456789abcdef0123456789abcdef.js": "/assets/admin/[masked]",
+ }
+ for in, want := range over {
+ if got := MaskPath(in); got != want {
+ t.Errorf("MaskPath(%q) = %q, want %q (accepted over-masking)", in, got, want)
+ }
+ }
long := "/" + strings.Repeat("a/", 600)
if got := MaskPath(long); len(got) > MaxPath || !utf8.ValidString(got) {
t.Errorf("long path not truncated to a valid string of at most %d bytes", MaxPath)
@@ -43,6 +86,122 @@ func TestMaskPath(t *testing.T) {
if got := MaskPath("/\xff\xfe/x"); !utf8.ValidString(got) {
t.Errorf("invalid UTF-8 survived: %q", got)
}
+ // The input is cut to 2048 bytes before masking: what lies past that is
+ // never judged, so it cannot reappear.
+ if got := MaskPath(strings.Repeat("/a", 1024) + "/tail"); strings.Contains(got, "tail") {
+ t.Errorf("text past the input bound reached the output: %q", got[len(got)-20:])
+ }
+}
+
+// TestMaskPathTokensProperty: no 9-character piece of a random token survives
+// MaskPath, for the token shapes that carry secrets in URLs, with one
+// measured exception. Standard base64 is cut into path segments by its own
+// '/', and a piece that happens to be a word ("aeaucltfe", "vqk", "Wwhjj")
+// is exactly what the allowlist keeps and the guards stop at, so a few
+// tokens in ten thousand keep a window through such a piece. The test
+// allows those (at most maxWordLeaks per 10,000 per kind, each with such a
+// word piece kept) and fails any other leak. The counts are logged.
+func TestMaskPathTokensProperty(t *testing.T) {
+ // Over four seeds (this one, 7, 99, 123456) the word-bridge leaks ran 3
+ // to 12 per 10,000 per kind; the ceiling is about twice the worst, so
+ // a guard that stops working fails here while the seed stays fixed.
+ const n, maxWordLeaks = 10000, 20
+ rng := rand.New(rand.NewPCG(20261025, 1))
+ raw := func(k int) []byte {
+ b := make([]byte, k)
+ for i := range b {
+ b[i] = byte(rng.Uint32())
+ }
+ return b
+ }
+ uuid := func() string {
+ h := hex.EncodeToString(raw(16))
+ return h[:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:]
+ }
+ kinds := []struct {
+ name string
+ split bool // the alphabet has '/', so the token arrives as segments
+ tok func() string
+ }{
+ {"base64-16", true, func() string { return base64.StdEncoding.EncodeToString(raw(16)) }},
+ {"base64-18", true, func() string { return base64.StdEncoding.EncodeToString(raw(18)) }},
+ {"base64-24", true, func() string { return base64.StdEncoding.EncodeToString(raw(24)) }},
+ {"base64-32", true, func() string { return base64.StdEncoding.EncodeToString(raw(32)) }},
+ {"base64url-32", false, func() string { return base64.RawURLEncoding.EncodeToString(raw(32)) }},
+ {"hex-16", false, func() string { return hex.EncodeToString(raw(16)) }},
+ {"uuid", false, uuid},
+ }
+ // throughWord reports whether a token kept a piece of itself that is a
+ // word to the guards (lower-case letters, or Title Case like Account)
+ // whole in out: the bridge they stop at, which lets a window survive.
+ throughWord := func(tok, out string) bool {
+ for _, piece := range strings.Split(tok, "/") {
+ word := piece
+ if len(word) >= 3 && word[0] >= 'A' && word[0] <= 'Z' {
+ word = word[1:]
+ }
+ if word != "" && strings.Trim(word, "abcdefghijklmnopqrstuvwxyz") == "" &&
+ (strings.Contains(out, "/"+piece+"/") || strings.HasSuffix(out, "/"+piece)) {
+ return true
+ }
+ }
+ return false
+ }
+ for _, k := range kinds {
+ leaks, slashed := 0, 0
+ var example string
+ for i := 0; i < n; i++ {
+ tok := k.tok()
+ if strings.Contains(tok, "/") {
+ slashed++
+ }
+ out := MaskPath("/reset/" + tok)
+ for j := 0; j+9 <= len(tok); j++ {
+ w := tok[j : j+9]
+ if !strings.Contains(out, w) {
+ continue
+ }
+ leaks++
+ if example == "" {
+ example = tok + " -> " + out
+ }
+ if !k.split || !throughWord(tok, out) {
+ t.Errorf("%s: %q kept the window %q, not through a lower-case word: %s", k.name, tok, w, out)
+ }
+ break
+ }
+ }
+ t.Logf("%s: %d of %d tokens kept a 9-character window (%d contained '/'); first: %s", k.name, leaks, n, slashed, example)
+ if leaks > maxWordLeaks {
+ t.Errorf("%s: %d of %d tokens kept a 9-character window, want at most %d", k.name, leaks, n, maxWordLeaks)
+ }
+ }
+}
+
+// TestMaskBoundedAllocation: an unbounded request path, query or fragment
+// costs a bounded amount of memory, because the input is cut before it is
+// walked.
+func TestMaskBoundedAllocation(t *testing.T) {
+ huge := strings.Repeat("/", 1<<20)
+ hugeQuery := strings.Repeat("a=1&", 1<<18)
+ hugeFrag := strings.Repeat("ab/", 1<<18)
+ for what, f := range map[string]func(){
+ "MaskPath": func() { MaskPath(huge) },
+ "QueryNames": func() { QueryNames(hugeQuery) },
+ "Fragment": func() { Fragment(SourceQuery, "q", hugeFrag) },
+ } {
+ r := testing.Benchmark(func(b *testing.B) {
+ b.ReportAllocs()
+ for i := 0; i < b.N; i++ {
+ f()
+ }
+ })
+ if got := r.AllocedBytesPerOp(); got >= 64<<10 {
+ t.Errorf("%s of a 1 MiB input allocated %d bytes per call, want under 64 KiB", what, got)
+ } else {
+ t.Logf("%s of a 1 MiB input: %d bytes, %d allocations per call", what, got, r.AllocsPerOp())
+ }
+ }
}
func TestQueryNames(t *testing.T) {
@@ -52,6 +211,16 @@ func TestQueryNames(t *testing.T) {
if got := QueryNames(""); got != "" {
t.Errorf("got %q", got)
}
+ // A name is kept on the same shapes as a path segment, else masked.
+ for in, want := range map[string]string{
+ "utm_source=x&page=2": "utm_source,page",
+ "reset_token_3f9a0c1d2e4b5a69788776655=1&q=2": "[masked],q",
+ "Filter%5BName%5D=a&FilterName=b&x=1": "[masked],x",
+ } {
+ if got := QueryNames(in); got != want {
+ t.Errorf("QueryNames(%q) = %q, want %q", in, got, want)
+ }
+ }
// A bare component has no name to keep: it may be the value itself.
for in, want := range map[string]string{"hunter2": "", "a=1&hunter2&b=": "a,b", "hunter2;x=1": "x"} {
if got := QueryNames(in); got != want {
@@ -151,9 +320,13 @@ func TestControlsEscaped(t *testing.T) {
}
return false
}
- if got, want := MaskPath("/x\x1b[2J\nforged\r\x00\x7f\u009b"), `/x\x1b[2J\x0aforged\x0d\x00\x7f\x9b`; got != want {
+ if got, want := MaskPath("/x\x1b[2J/p/\n/q/f\r/r/\x7f\u009b"), `/x\x1b[2J/p/\x0a/q/f\x0d/r/\x7f\x9b`; got != want {
t.Errorf("MaskPath = %q, want %q", got, want)
}
+ // A long segment with controls in it is not word-shaped: masked whole.
+ if got := MaskPath("/x\x1b[2J\nforged-log-line"); got != "/[masked]" {
+ t.Errorf("MaskPath = %q, want /[masked]", got)
+ }
if got, want := QueryNames("a%0Ab=1&c%1b=2"), `a\x0ab,c\x1b`; got != want {
t.Errorf("QueryNames = %q, want %q", got, want)
}