amadan

carlos / platform Public

docs: receiving scan results, the page an app author needs

The tenant-facing half of scan on demand, which the final review argued
should land WITH the merge rather than after it. Once this branch merges
the enqueue door accepts callback_path from any Service credential and
the console starts POSTing signed bodies to apps. Without this page those
apps have no documented way to verify one.

Drafted, revised through the plain-writing and humane-docs passes, and
approved in review with zero edits across both rounds.

WHAT IT HAS TO GET ACROSS, in order of how much damage it prevents:

The hex trap. The secret arrives in the app's environment hex-encoded and
the MAC is keyed on the 32 raw bytes it decodes to. Feed the environment
string straight into HMAC and every callback fails, which looks exactly
like a platform fault. The page says so twice and shows the one line, in
Go and in Python, because the reader most likely to hit it is not writing
Go.

That none of `skipped`, `gone` or `unavailable` is `clean`. An app
treating them as safe to serve is claiming coverage it does not have,
which is the whole failure this feature exists to avoid.

That `recorded: false` means the verdict is not on the object, so the
enforcement deny does not match it and nothing is blocking reads of that
file. The app has to decide.

That delivery is at most once, so the callback is a shortcut and the
status door is the delivery mechanism. With the idempotency tuple, and
the instruction to poll when nothing arrives.

FIVE FENCES AND ONE LANGUAGE. The page's code blocks are labelled http,
text, json, go and python. `python` is new to knownFenceLangs, which
that test calls a deliberate act — an undeclared fence is invisible to
every content gate that reads fences, so it is a way to smuggle a wrong
command past review. It is declared here with the reason: this page ships
a second implementation on purpose, and the mistake it prevents is one
people make in whatever language they are in.

The page is registered in nav.json. Its label and blurb are the only two
strings on it that have not been through copy review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HfXevdxQwRxX3pGwwMCVNE
Paul Campbell pushed by paul@keymail.dev fc549d170125664896dfad2b174b5294ec7bec0c parent f64485d
3 files changed, +256 −0
  • docs/site/nav.json +5 −0
  • docs/site/scan-callbacks.md +245 −0
  • internal/docsite/fences_test.go +6 −0

CI passed — run details

diff --git a/docs/site/nav.json b/docs/site/nav.json
index 616eb25f0..495c0168d 100644
--- a/docs/site/nav.json
+++ b/docs/site/nav.json
@@ -7,6 +7,11 @@
"slug": "cli",
"label": "The CLI",
"blurb": "Every carlos command, grouped by whether you are shipping an app, running a deployment, or on a box."
+ },
+ {
+ "slug": "scan-callbacks",
+ "label": "Receiving scan results",
+ "blurb": "Ask for one object to be scanned, read the verdict, and verify the callback that tells you it is done."
}
]
}
diff --git a/docs/site/scan-callbacks.md b/docs/site/scan-callbacks.md
new file mode 100644
index 000000000..fc4e43048
--- /dev/null
+++ b/docs/site/scan-callbacks.md
@@ -0,0 +1,245 @@
+# Receiving scan results
+
+Your app accepted an upload and needs to know whether the file is safe before
+it serves it. Two endpoints answer that, and a callback can save you the poll.
+
+This assumes your app's store already has virus scanning granted. If it does
+not, see `carlos store scan` in [the CLI page](/docs/cli#carlos-store-scan).
+
+## Asking for a scan
+
+Both endpoints take a Service-role credential and reach only your own app.
+
+```http
+POST /api/cli/apps/{account}/{app}/store/scan/object
+{"key": "uploads/x.pdf", "callback_path": "/hooks/scan"}
+```
+
+`callback_path` is optional. You get back `202` and `{"state":"pending"}`. The
+scan is queued, so there is no verdict in that response.
+
+```http
+GET /api/cli/apps/{account}/{app}/store/scan/object?key=uploads/x.pdf&wait=15s
+```
+
+This reads what is known about one object. `wait` long-polls: the request holds
+open until the state changes or `wait` elapses, capped at 30 seconds. Ask for
+longer and you get 30 seconds. Leave `wait` off and you get an immediate answer.
+
+## The states
+
+```text
+{"state":"clean"}
+{"state":"infected"}
+{"state":"skipped"}
+{"state":"failed"}
+{"state":"denied"}
+{"state":"pending"}
+{"state":"gone"}
+{"state":"unavailable","reason":"...","error":"..."}
+```
+
+Only `clean` and `infected` are verdicts on the contents. The rest mean
+something else:
+
+- `skipped`: the engine did not judge the file. Usually it was too large, or
+ its type is not one the engine reads.
+- `failed`: the engine read the file and could not reach a verdict.
+- `denied`: the console could not read the object.
+- `pending`: no answer yet. Ask again, or wait longer.
+- `gone`: there is no object at this key.
+- `unavailable`: scanning is not available for this store right now. `reason`
+ is one of `not_granted`, `not_requested`, `encrypted`, `not_enabled`,
+ `disabled`, `no_engine`, `callbacks_not_configured`. `error` is a sentence
+ for whoever reads the log.
+
+If your app treats `skipped` or `gone` as safe to serve, it is claiming
+coverage it does not have.
+
+## Callbacks take a path
+
+`callback_path` names a path on your app. The console supplies the host
+itself: your app's platform-assigned name, `<app>.<account>.<apps-domain>`. It
+will not post to a custom domain, so the path has to be served on the platform
+host. Your app serves both, so usually it already is.
+
+The rules are checked when you enqueue, so a bad value comes back as a `400`
+and is never stored: one leading slash, no `//`, no backslash, no `#`, no `..`
+segment, no `;`, no `@`, printable ASCII only, and 1KB at most. A query string
+is fine and is preserved.
+
+An app with no ready route has no platform host yet, so it receives no
+callbacks even if you pass `callback_path`. Poll until it has one.
+
+## Verifying the signature
+
+Every callback carries a header:
+
+```http
+Carlos-Signature: t=1800000000,v1=ccd4f98ffb69f5dd:1c24c96520fc34a1...
+```
+
+`t` is the send time in unix seconds. Each `v1=` field is
+`<key-id>:<hex-mac>`, and there can be two of them during a key rotation, one
+under the old key and one under the new. Verify against each and accept the
+callback if any checks out. Ignore key ids you do not recognise, and ignore
+fields you do not recognise, because the platform may add more later and your
+verifier should survive that.
+
+The MAC is HMAC-SHA256 over `t`, then a literal `.`, then the raw request body.
+Verify the bytes as they arrived. If you parse the JSON and re-serialise it
+first you get a different byte string, and a genuine callback will not verify.
+Compare in constant time, and reject the callback if `t` is more than five
+minutes from your own clock in either direction.
+
+### The hex trap
+
+This one is boring but read it before you write the verifier.
+
+Your app receives its callback secret in an environment variable named
+`<YOUR_PREFIX>_SCAN_CALLBACK_SECRET`. That value is hex-encoded text. The
+signing key is the 32 raw bytes it decodes to, not the string itself.
+
+Hand the environment variable straight to your HMAC function and every callback
+fails to verify, which looks exactly like a platform fault. It is usually this
+one line:
+
+```python
+key = bytes.fromhex(secret_env_var) # not secret_env_var.encode()
+```
+
+### Go
+
+```go
+func verify(secretHex string, header string, body []byte) error {
+ key, err := hex.DecodeString(secretHex)
+ if err != nil {
+ return fmt.Errorf("callback secret is not valid hex: %w", err)
+ }
+
+ var timestamp string
+ var macs [][]byte
+ for _, field := range strings.Split(header, ",") {
+ switch {
+ case strings.HasPrefix(field, "t="):
+ timestamp = field[2:]
+ case strings.HasPrefix(field, "v1="):
+ parts := strings.SplitN(field[3:], ":", 2)
+ if len(parts) != 2 {
+ continue
+ }
+ mac, err := hex.DecodeString(parts[1])
+ if err == nil {
+ macs = append(macs, mac)
+ }
+ }
+ }
+ if timestamp == "" || len(macs) == 0 {
+ return errors.New("missing t or v1")
+ }
+
+ unix, err := strconv.ParseInt(timestamp, 10, 64)
+ if err != nil {
+ return errors.New("bad timestamp")
+ }
+ if age := time.Since(time.Unix(unix, 0)); age < -5*time.Minute || age > 5*time.Minute {
+ return errors.New("timestamp outside five-minute window")
+ }
+
+ h := hmac.New(sha256.New, key)
+ h.Write([]byte(timestamp))
+ h.Write([]byte{'.'})
+ h.Write(body)
+ want := h.Sum(nil)
+
+ for _, got := range macs {
+ if hmac.Equal(got, want) {
+ return nil
+ }
+ }
+ return errors.New("signature does not verify")
+}
+```
+
+### Python
+
+```python
+import hashlib
+import hmac
+import time
+
+def verify(secret_hex: str, header: str, body: bytes) -> bool:
+ key = bytes.fromhex(secret_hex) # the hex trap: decode before using
+
+ timestamp = None
+ macs = []
+ for field in header.split(","):
+ if field.startswith("t="):
+ timestamp = field[2:]
+ elif field.startswith("v1="):
+ _, _, mac_hex = field.partition(":")
+ try:
+ macs.append(bytes.fromhex(mac_hex))
+ except ValueError:
+ continue
+
+ if timestamp is None or not macs:
+ return False
+ if abs(time.time() - int(timestamp)) > 300:
+ return False
+
+ want = hmac.new(key, timestamp.encode() + b"." + body, hashlib.sha256).digest()
+ return any(hmac.compare_digest(mac, want) for mac in macs)
+```
+
+In both examples, `body` is the raw bytes read off the request before any JSON
+parsing touches them.
+
+## The callback body
+
+```json
+{"account":"bab","app":"titogo","key":"uploads/x.pdf","etag":"...",
+ "verdict":"clean","recorded":true,"engine":"clamav","signature":"..."}
+```
+
+Match a callback to your own upload on `etag`, the bare ETag your `PUT`
+returned, and not on the key alone: the same key can be overwritten. There is
+no version token in the body. The platform's internal version format is not
+something your app should compare against, and getting that comparison wrong is
+a security bug.
+
+`signature` is the malware signature name, present when `verdict` is
+`infected`. It has nothing to do with the request signature in the
+`Carlos-Signature` header. Both `engine` and `signature` can be empty, which
+happens when the verdict came from a tag already on the object instead of a
+scan just run. Empty means the platform does not know what judged it. That is
+an honest answer, not an error.
+
+Watch `recorded`. When it is `false` the verdict could not be written onto the
+object itself, so an `infected` object with `recorded: false` is not covered by
+the platform's enforcement deny. Nothing is blocking reads of that file. Your
+app decides what to do about it.
+
+## Delivery
+
+Callbacks are delivered at most once. A console restart at the wrong moment
+loses one and nothing re-sends it, and duplicates are possible in the other
+direction. So make your handler idempotent on
+`(account, app, key, etag, verdict)`, and if no callback arrives within
+whatever timeout your app sets, ask the `GET` endpoint. The callback is a
+shortcut. The verdict is stored on the object, and the `GET` always has an
+answer.
+
+The console POSTs with `Content-Type: application/json`, and any `2xx` counts
+as delivered. A redirect is a refusal and is not retried, so answer at
+`/hooks/scan` directly instead of redirecting to `/hooks/scan/`. A `4xx` is not
+retried either. A `5xx`, a `429` or a timeout is retried three times, at 1
+second, 4 seconds and 16 seconds. The console reads at most 4KB of your
+response and ignores it.
+
+## Rate limits
+
+Both endpoints share your app's write budget, about 120 requests a minute. A
+long-poll `GET` costs two of those a minute for as long as it is open. Polling
+in a tight loop with no `wait` will meet the limit, so use `wait` and keep
+polling as the backstop.
diff --git a/internal/docsite/fences_test.go b/internal/docsite/fences_test.go
index f0469239b..11f1e7c80 100644
--- a/internal/docsite/fences_test.go
+++ b/internal/docsite/fences_test.go
@@ -13,6 +13,12 @@ var knownFenceLangs = map[string]bool{
"sh": true, "bash": true, "console": true, "json": true,
"go": true, "yaml": true, "toml": true, "ini": true,
"text": true, "http": true,
+ // python, for the callback verifier on scan-callbacks.md. The page ships
+ // a second implementation on purpose: most people writing a webhook
+ // verifier against this platform are not writing Go, and the one mistake
+ // that page exists to prevent (hex secret, raw-byte key) is the one they
+ // make in whatever language they are in.
+ "python": true,
}
func TestFenceLanguagesAreDeclared(t *testing.T) {