| 1 | # Virus scanning for the object storage primitive |
| 2 | |
| 3 | **Date:** 2026-09-01 |
| 4 | **Status:** approved in principle (Paul, 2026-09-01). Items marked **(ruling)** |
| 5 | were decided in conversation; items marked **(decision)** are this spec's |
| 6 | reasoning and can be argued with. |
| 7 | |
| 8 | **Revision history.** Three review rounds, fourteen blockers, all fixed. |
| 9 | Draft 3 carried Paul's direction change — **ClamAV primary, GuardDuty an |
| 10 | option, self-hosters included** — after GuardDuty's 25-bucket cap surfaced. |
| 11 | Draft 4 (this one) closes review 3: the enforcement deadlock, verdicts not |
| 12 | bound to the bytes they judged, the untrue self-hoster claim, an audit |
| 13 | sentence that hid unscannable objects, and the ClamAV engine's missing |
| 14 | protocol detail. |
| 15 | |
| 16 | ## 1. What is being asked, and what can honestly be claimed |
| 17 | |
| 18 | Tony asked whether CARLOS can scan objects for viruses. Asked whose problem |
| 19 | that solves, the answer was a **compliance checkbox** — someone wants to know |
| 20 | whether stored files are scanned and the deployment needs a defensible yes. |
| 21 | Coverage and evidence matter; detection latency and enforcement strength do |
| 22 | not. **(ruling)** |
| 23 | |
| 24 | **The three sentences an audit response may use, and no stronger ones:** |
| 25 | |
| 26 | > As of the last completed sweep, which finished at time T, N objects in this |
| 27 | > bucket carried no recorded verdict, and a further S were recorded as not |
| 28 | > scannable — too large, unreadable, or the engine failed on them. |
| 29 | |
| 30 | > Objects scanned clean were scanned by ENGINE against signature database |
| 31 | > VERSION. |
| 32 | |
| 33 | > For apps whose objects are encrypted before they reach the store, scanning |
| 34 | > is not applicable and no coverage is claimed — see §2.1. |
| 35 | |
| 36 | The first sentence names **two** numbers. Draft 3 named only the unscanned |
| 37 | count, which let a bucket of 12,000 over-size objects report "0 objects |
| 38 | carried no recorded verdict" — technically true under §3's taxonomy and |
| 39 | thoroughly misleading. *Completed* matters too: a sweep in progress is not an |
| 40 | answer, which is why §5 keeps the last complete one separately. And T is as |
| 41 | wide as a sweep takes, so §7.1 gives it a budget and `scan status` prints the |
| 42 | projection when it spans ticks. |
| 43 | |
| 44 | This is a point-in-time reconciliation, not a continuous guarantee. It is |
| 45 | still worth more than a scan-on-upload pipeline, whose misses are invisible |
| 46 | while a reconciler's are counted. |
| 47 | |
| 48 | ### 1.1 Why a reconciler and not a pipeline |
| 49 | |
| 50 | CARLOS never sees the PUT. `internal/objstore` is a control plane and says so: |
| 51 | |
| 52 | > It never touches the data plane — apps talk to their bucket directly with |
| 53 | > the delivered credentials. |
| 54 | |
| 55 | It mints a bucket, creates one IAM user per app whose inline policy is exactly |
| 56 | two statements (`aws.go:129-130`), seals the secret into the app's config, and |
| 57 | steps out. There is no request path to hook because there is no request path. |
| 58 | |
| 59 | The lever that survives is that **CARLOS writes the app's IAM policy**, so the |
| 60 | control plane can gate reads on a verdict without being in the data path (§8). |
| 61 | |
| 62 | ## 2. Scope, and who this is for |
| 63 | |
| 64 | **The primary engine is ClamAV on the deployment's own infrastructure.** No |
| 65 | bucket cap, any S3-compatible store, no cloud service to buy. **(ruling)** |
| 66 | |
| 67 | **GuardDuty is an option for AWS deployments that would rather buy it.** Not |
| 68 | the default: AWS caps Malware Protection at **25 protected buckets per account |
| 69 | per Region, not adjustable**, and CARLOS mints one bucket per app |
| 70 | (`MintedBucket`, `walk.go:62`), so it runs out at 25 apps. |
| 71 | |
| 72 | ### 2.1 Who can actually turn this on today |
| 73 | |
| 74 | Draft 3 said a self-hoster "sets `CARLOS_STORE_SCANNER=clamav` and runs a |
| 75 | clamd unit — no AWS account." **That is not true today and the correction |
| 76 | matters.** |
| 77 | |
| 78 | The store primitive itself is AWS-bound: `cmd/console/main.go:497-498` fatals |
| 79 | on any `CARLOS_STORE_BACKEND` but `"aws"`, and `aws.go:73-75` records that a |
| 80 | non-AWS endpoint "still means IAM-against-real-AWS in v1 — there is nothing |
| 81 | else for it to point at." There is no store for a scanner to scan without AWS |
| 82 | IAM. |
| 83 | |
| 84 | So the honest statement: **scanning is available to any deployment that has |
| 85 | the store primitive, and today that still needs an AWS account for IAM.** The |
| 86 | scanner itself is backend-agnostic and will follow a non-AWS provisioner the |
| 87 | moment one lands — that provisioner, promised in `provisioner.go`'s doc |
| 88 | comment, is the actual dependency for a fully self-hosted story, and this spec |
| 89 | does not deliver it. What draft 4 does deliver is a scanner that will not have |
| 90 | to change when it arrives. **(decision)** |
| 91 | |
| 92 | ### 2.2 Scanning ciphertext is meaningless |
| 93 | |
| 94 | CARLOS's stated family default is server-blindness — *"If the server is |
| 95 | compromised, the attacker learns nothing"* (`building-carlos-apps/SKILL.md` |
| 96 | line 60). An app following it encrypts objects **client-side**, before they |
| 97 | reach the store. |
| 98 | |
| 99 | **A scanner cannot scan ciphertext.** Every object in such a bucket scans |
| 100 | clean forever, whatever it contains. |
| 101 | |
| 102 | Note the distinction draft 3 blurred: this is about **client-side** |
| 103 | encryption. Server-side encryption (SSE-S3, SSE-KMS — on by default for every |
| 104 | AWS bucket since 2023) is transparent to a reader holding the right |
| 105 | credentials and scans perfectly well. Only client-side encryption defeats a |
| 106 | scanner. |
| 107 | |
| 108 | **The platform cannot detect which it is, so it asks.** `scan request` takes |
| 109 | an explicit attestation, recorded on the record: |
| 110 | |
| 111 | - `--plaintext` — objects are readable by the deployment; scanning is |
| 112 | meaningful; the §1 sentences apply. |
| 113 | - `--client-encrypted` — objects are ciphertext. The sweep marks the sidecar |
| 114 | `not_applicable` and does no work, `scan status` prints §1's third sentence |
| 115 | and **refuses to print the first**, and `enforce` is rejected as inert |
| 116 | (nothing will ever be tagged infected). |
| 117 | |
| 118 | Without a recorded attestation, §1's third sentence has no data behind it and |
| 119 | `scan status` cannot know which answer to give. **(decision)** |
| 120 | |
| 121 | ### 2.3 What the member is actually consenting to |
| 122 | |
| 123 | The ClamAV engine reads tenant object bytes, and the per-app opt-in is where |
| 124 | the app owner consents to that. One honest qualification, because the consent |
| 125 | story would otherwise overclaim: |
| 126 | |
| 127 | **The console can already read every store.** It holds `PutUserPolicy` and |
| 128 | `CreateAccessKey` over every store user (`aws.go:282-311`), so it can mint |
| 129 | itself a credential to any tenant bucket at will. §10's `GetObject` grant |
| 130 | makes an existing capability explicit rather than creating a new one. What the |
| 131 | opt-in governs is what the platform **does**, not what it **could** do. Say |
| 132 | that plainly rather than implying scanning is what first gives the platform |
| 133 | reach into a member's bucket. **(decision)** |
| 134 | |
| 135 | ## 3. The Scanner seam |
| 136 | |
| 137 | ```go |
| 138 | // Scanner is the scanning-engine seam. Like Provisioner, every method is |
| 139 | // idempotent — "already enabled" and "already gone" are successes. |
| 140 | type Scanner interface { |
| 141 | EnsureEnabled(ctx context.Context, bucket string) error |
| 142 | // EnsureDisabled releases what EnsureEnabled took. Teardown calls it |
| 143 | // BEFORE deleting the bucket (§7.3). |
| 144 | EnsureDisabled(ctx context.Context, bucket string) error |
| 145 | // Verdict answers the recorded verdict for one object AT etag. A verdict |
| 146 | // recorded against a different etag is (VerdictUnknown, nil): the object |
| 147 | // has been rewritten since it was judged (§3.1). |
| 148 | Verdict(ctx context.Context, bucket, key, etag string) (Verdict, error) |
| 149 | // Scan scans one object and records its verdict against its etag. It |
| 150 | // returns the verdict when the engine is synchronous (ClamAV) and |
| 151 | // VerdictUnknown when the engine scans asynchronously (GuardDuty). |
| 152 | Scan(ctx context.Context, bucket, key, etag string) (Verdict, error) |
| 153 | // Enforcement names the tag this engine writes, so the policy writer can |
| 154 | // deny reads on it (§8). |
| 155 | Enforcement() DenyTag |
| 156 | // Capacity reports buckets still protectable and the cap; (-1,-1) is |
| 157 | // uncapped. ClamAV is uncapped; GuardDuty is not. |
| 158 | Capacity(ctx context.Context) (remaining, cap int, err error) |
| 159 | // Provenance names what did the scanning, for the evidence record. |
| 160 | Provenance(ctx context.Context) (engine, version string, err error) |
| 161 | } |
| 162 | |
| 163 | type Verdict string |
| 164 | |
| 165 | const ( |
| 166 | VerdictUnknown Verdict = "" // no recorded verdict for these bytes |
| 167 | VerdictClean Verdict = "clean" |
| 168 | VerdictInfected Verdict = "infected" |
| 169 | VerdictSkipped Verdict = "skipped" // too large, unsupported, unextractable |
| 170 | VerdictDenied Verdict = "denied" // the engine could not read the object |
| 171 | VerdictFailed Verdict = "failed" // the engine tried and failed |
| 172 | ) |
| 173 | |
| 174 | type DenyTag struct{ Key, Value string } |
| 175 | ``` |
| 176 | |
| 177 | `Scan` returning a `Verdict` is draft 4's correction to a subtle counting bug. |
| 178 | ClamAV is synchronous — fetch, stream, tag — so the verdict exists the moment |
| 179 | `Scan` returns. Draft 3's "the next sweep reads the result" was GuardDuty's |
| 180 | shape applied to both, which would have made the **first complete sweep over |
| 181 | every bucket report every object as unscanned**, and with a 24-hour re-sweep |
| 182 | floor the audit sentence would only become true two days in. The sweep counts |
| 183 | a verdict `Scan` hands back. |
| 184 | |
| 185 | ### 3.1 Verdicts are bound to the bytes they judged |
| 186 | |
| 187 | `PutObjectTagging` replaces a tag set unconditionally and knows nothing about |
| 188 | versions, so a verdict tag alone says "some bytes under this key were clean |
| 189 | once." The window: `Scan` fetches v1, the app overwrites with v2, `Scan` tags |
| 190 | clean — and v2 is marked clean having never been read. Narrow, but it is |
| 191 | precisely the window worth timing. |
| 192 | |
| 193 | So the engine writes the scanned object's **version** into the tag set |
| 194 | alongside the verdict, and `Verdict` compares it against what `ListObjectsV2` |
| 195 | already returns for free on the page being swept. A mismatch is |
| 196 | `VerdictUnknown` — the object gets rescanned, which is the correct outcome for |
| 197 | bytes nobody has looked at. This also closes the forgery story properly: a |
| 198 | re-uploaded object cannot inherit its predecessor's verdict. **(decision — |
| 199 | draft 3 had this only as a Hetzner fallback; it is the primary design.)** |
| 200 | |
| 201 | **The version is the ETag AND `LastModified`, not the ETag alone |
| 202 | (implementation review).** For a single-part upload the ETag is the content |
| 203 | MD5, and chosen-prefix MD5 collisions are practical on ordinary hardware. An |
| 204 | attacker who controls both uploads — which is exactly the case enforcement |
| 205 | exists for, an app accepting uploads and serving them back — could give a |
| 206 | payload the ETag of the clean object it replaces and inherit its verdict, |
| 207 | walking through the Deny. `LastModified` moves on every overwrite and is not |
| 208 | the uploader's to choose. Both halves are on the listing page already, so the |
| 209 | stronger binding costs nothing. The tag is `carlos-scan-version`. |
| 210 | |
| 211 | ## 4. The engines |
| 212 | |
| 213 | ### 4.1 `clamavScanner` — primary, shipped in v1 |
| 214 | |
| 215 | Fetch the object, stream it to `clamd`, write `carlos-scan`, |
| 216 | `carlos-scan-version` and `carlos-scan-at` tags — preserving any tag the |
| 217 | scanner does not own, because an ADOPTED bucket's objects can carry tags from |
| 218 | tooling that predates CARLOS. `Capacity` is uncapped. |
| 219 | |
| 220 | **Unreadable and vanished objects are per-object verdicts, not pass failures |
| 221 | (implementation review).** An `AccessDenied` on `GetObject` — an SSE-KMS object |
| 222 | outside the console's key policy, a foreign-owned object in an adopted bucket — |
| 223 | returned as an error would abandon the page unwritten, and the next tick would |
| 224 | re-list the same page and fail identically: the bucket's sweep would never |
| 225 | advance past it, forever, while every surface reported a sweep merely "in |
| 226 | progress". It is `VerdictDenied`, which is what that verdict was declared for. |
| 227 | A `NoSuchKey` is "gone": dropped rather than counted, since an object deleted |
| 228 | between the listing and the fetch is neither covered nor a gap. |
| 229 | |
| 230 | **The read path already exists.** `sizeProvisionedStore` |
| 231 | (`economicspass.go:990`) lists a tenant's provisioned bucket with |
| 232 | `objstore.S3ClientFor(ctx, p.Region, p.Endpoint)` (`aws.go:95`) and the |
| 233 | console's ambient credentials, from a scheduled pass, with no IAM involved. |
| 234 | Scanning adds `GetObject`, `GetObjectTagging` and `PutObjectTagging` to that |
| 235 | same principal. Stream `GetObject.Body` straight into the socket; never buffer |
| 236 | an object in memory. |
| 237 | |
| 238 | **Transport.** `clamd`'s INSTREAM protocol is unauthenticated, so it is never |
| 239 | exposed to a network. The console reaches it over a **unix socket** |
| 240 | (`CARLOS_CLAMD_SOCKET`, default `/run/clamav/clamd.ctl`). No TCP option in v1: |
| 241 | an unauthenticated scanning service reachable over a network is a worse |
| 242 | problem than the one being solved. **(decision)** |
| 243 | |
| 244 | **Protocol, stated so it is not invented twice.** Send `zINSTREAM\0`, then |
| 245 | chunks of a 4-byte big-endian length followed by that many bytes, terminated |
| 246 | by a zero-length chunk. Replies are null-terminated: `stream: OK\0` → clean, |
| 247 | `stream: <Signature> FOUND\0` → infected, `INSTREAM size limit exceeded. |
| 248 | ERROR\0` → skipped. The `z` (null-terminated) command form is used throughout, |
| 249 | not the newline form. One connection per object; `IDSESSION` is not used in |
| 250 | v1, and §11 measures whether that costs enough to revisit. |
| 251 | |
| 252 | **Size ceiling, checked before the fetch.** `ListObjectsV2` already returns |
| 253 | `Size`, so an object above `CARLOS_CLAMD_MAX_BYTES` is tagged `skipped` |
| 254 | without a `GetObject` — paying egress to fetch bytes clamd will refuse is |
| 255 | pure waste. If the variable is set above clamd's own `StreamMaxLength`, clamd |
| 256 | answers `size limit exceeded ... ERROR` mid-stream; that classifies as |
| 257 | **skipped, not failed**, or the counters lie about what happened. |
| 258 | |
| 259 | **Timeouts.** Dial deadline, per-chunk write deadline and reply read deadline |
| 260 | on our side; clamd's `ReadTimeout` and `CommandReadTimeout` on its side. All |
| 261 | configurable, all defaulted, none infinite. |
| 262 | |
| 263 | **When clamd is down.** A dial failure aborts that app's pass **without |
| 264 | writing any verdict** and is reported as a pass error to retry next tick. It |
| 265 | must never stamp thousands of objects `failed`: "the engine is unreachable" and |
| 266 | "the engine read this object and failed on it" are different facts and only |
| 267 | the second belongs in a tag. |
| 268 | |
| 269 | **Re-scan policy.** `failed` and `denied` are re-issued every sweep — both are |
| 270 | usually transient or a permissions fix away. `skipped` is not re-issued unless |
| 271 | the ETag changed or the ceiling moved. `clean` and `infected` are re-issued |
| 272 | only on ETag mismatch (§3.1). |
| 273 | |
| 274 | **Provenance.** `zVERSION\0` answers e.g. `ClamAV 1.4.1/27500/Mon Sep 1 |
| 275 | 08:00:00 2026` — engine version, signature database version, database date. |
| 276 | `Sweep.Signature` records all three verbatim. `scan status` warns when the |
| 277 | database date is older than `CARLOS_CLAMD_MAX_SIG_AGE` (default 7 days), |
| 278 | because a clean result from stale signatures should not read like a fresh one. |
| 279 | |
| 280 | **Operational facts self-hosters need up front**, and §10 must carry: clamd |
| 281 | holds roughly 1–1.5 GB of signatures resident; `freshclam` must run on a timer |
| 282 | or the database rots silently; and the console's unit user must be in the |
| 283 | socket's group (`/run/clamav/clamd.ctl` is group `clamav`). |
| 284 | |
| 285 | ### 4.2 `guarddutyScanner` — optional, AWS only, gated |
| 286 | |
| 287 | `EnsureEnabled` creates a Malware Protection plan with tagging on; `Verdict` |
| 288 | reads `GuardDutyMalwareScanStatus`; `Capacity` reports against the cap. |
| 289 | |
| 290 | Verified against AWS documentation rather than assumed: |
| 291 | |
| 292 | - **`SendObjectMalwareScan` is real**, takes an S3 object path (optionally a |
| 293 | version), and is the documented on-demand path for objects that predate the |
| 294 | plan. It needs `AmazonGuardDutyFullAccess_v2` on the caller and is excluded |
| 295 | from the free tier, so the sweep pays per unknown object per pass. |
| 296 | - **The object ceiling is 100 GB** and the bucket cap is 25, both from the |
| 297 | quotas table, both non-adjustable. |
| 298 | |
| 299 | **The gate that must be measured before any GuardDuty work starts.** AWS |
| 300 | states tagging "must be enabled before an object gets uploaded" or GuardDuty |
| 301 | "will not be able to add tags to the scanned object." Every object in an |
| 302 | existing store predates its plan. If an on-demand scan of such an object is |
| 303 | not tagged, `Verdict` never converges on any existing store, and this engine |
| 304 | needs an EventBridge consumer reading scan results instead of tags — a |
| 305 | different architecture. **Measure it against a real bucket first.** ClamAV is |
| 306 | unaffected either way, which is why it is the primary engine. |
| 307 | |
| 308 | ## 5. Evidence: a sidecar, not the store record |
| 309 | |
| 310 | Two objects, two writers, no contention. **(decision — draft 2 put everything |
| 311 | in `store.json` and needed a page of CAS choreography to survive it.)** |
| 312 | |
| 313 | **`store.json` gains a `scanning` block, written only by the doors:** |
| 314 | |
| 315 | ```go |
| 316 | type ScanRequested struct { |
| 317 | At time.Time `json:"at"` |
| 318 | By string `json:"by"` |
| 319 | Encrypted bool `json:"encrypted"` // §2.2 attestation |
| 320 | } |
| 321 | |
| 322 | type ScanEnabled struct { |
| 323 | Engine string `json:"engine"` // "clamav" | "guardduty" |
| 324 | At time.Time `json:"at"` |
| 325 | By string `json:"by"` |
| 326 | } |
| 327 | |
| 328 | type ScanBlock struct { |
| 329 | Requested *ScanRequested `json:"requested,omitempty"` |
| 330 | Enabled *ScanEnabled `json:"enabled,omitempty"` |
| 331 | // Enforce holds the tag itself, not a bool: Grant takes a Provisioner |
| 332 | // and has no Scanner to ask, so the record carries what the policy |
| 333 | // writer needs (§8). |
| 334 | Enforce *DenyTag `json:"enforce,omitempty"` |
| 335 | } |
| 336 | ``` |
| 337 | |
| 338 | **`apps/<acct>/<app>/scan.json` holds the rollups, written only by the sweep:** |
| 339 | |
| 340 | ```go |
| 341 | type ScanSidecar struct { |
| 342 | Schema int `json:"schema"` |
| 343 | Account string `json:"account"` // §5.1 |
| 344 | App string `json:"app"` |
| 345 | NotApplicable bool `json:"not_applicable,omitempty"` // §2.2 |
| 346 | Current *Sweep `json:"current,omitempty"` |
| 347 | LastComplete *Sweep `json:"last_complete,omitempty"` // the audit answer |
| 348 | } |
| 349 | |
| 350 | type Sweep struct { |
| 351 | StartedAt time.Time `json:"started_at"` |
| 352 | At time.Time `json:"at"` |
| 353 | Engine string `json:"engine"` |
| 354 | Signature string `json:"signature,omitempty"` |
| 355 | Objects int64 `json:"objects"` |
| 356 | Scanned int64 `json:"scanned"` |
| 357 | Infected int64 `json:"infected"` |
| 358 | Skipped int64 `json:"skipped"` |
| 359 | Denied int64 `json:"denied"` |
| 360 | Failed int64 `json:"failed"` |
| 361 | Unscanned int64 `json:"unscanned"` |
| 362 | Bytes int64 `json:"bytes"` |
| 363 | After string `json:"after,omitempty"` |
| 364 | } |
| 365 | ``` |
| 366 | |
| 367 | `Current` and `LastComplete` are separate because draft 2's single `*Sweep` |
| 368 | overloaded `After == ""` to mean both "complete" and "start fresh", so the |
| 369 | tick after a sweep finished overwrote the audit answer with page-one partials. |
| 370 | A sweep accumulates in `Current`; on its last page it moves to `LastComplete` |
| 371 | and `Current` clears. A completed sweep does not restart before |
| 372 | `CARLOS_STORE_SCAN_INTERVAL` (default 24h), so `LastComplete` stays stable |
| 373 | long enough to quote. |
| 374 | |
| 375 | `Schema` is refused when unknown, the rule `ReadAppTombstone` already applies |
| 376 | (`trash.go:76-80`). |
| 377 | |
| 378 | ### 5.1 Why the sidecar carries its own account |
| 379 | |
| 380 | `accounts migrate` copies every key under the source prefix |
| 381 | (`accounts.go:587-597`), so a sidecar would silently follow an app to a new |
| 382 | account carrying rollups for a bucket named after the old one. The reader |
| 383 | refuses a body whose `Account`/`App` disagree with its path, exactly as |
| 384 | `AppTombstone` does. |
| 385 | |
| 386 | (`store.json` has the same latent problem — its bucket and IAM user are named |
| 387 | after the original account — but that predates this spec and is noted, not |
| 388 | fixed here.) |
| 389 | |
| 390 | ### 5.2 The PARSER LIST hazard |
| 391 | |
| 392 | `objstore.go` records that `release.PurgeApp` parses `grant` and `provisioned` |
| 393 | with its own local struct and refuses box-side auto-purge when either is |
| 394 | present. Additive changes are safe (`trash.go:214-217` ignores unknown keys), |
| 395 | so `scanning` is fine, but those two names must not move. The full consumer |
| 396 | list for this shape is `objstore.Record`, `PurgeApp`'s local struct, and |
| 397 | `handleAPIStoreStatus`'s map (`apistore.go:130-141`). §13 adds the additive |
| 398 | subtest, which `TestPurgeGuardSeesOurRecord` does not cover today. |
| 399 | |
| 400 | ## 6. Who turns it on |
| 401 | |
| 402 | - **The member requests** — `carlos store scan request --plaintext` or |
| 403 | `--client-encrypted` (§2.2). Records the ask and the attestation, spends |
| 404 | nothing. |
| 405 | - **An operator grants** — `carlos store scan grant`. Calls the cloud account |
| 406 | and starts a meter, so it sits behind `CARLOS_STORE_OPERATORS`, the gate |
| 407 | `apistore.go` applies to store grants for the same reason, checked **after** |
| 408 | the member gate so a 403 never confirms which apps exist. |
| 409 | |
| 410 | `CARLOS_STORE_SCANNER` unset means no engine is built and `grant`/`sweep` 503 |
| 411 | naming the variable. The **member's** request door still records, following |
| 412 | the create door's deliberate precedent (`apistore.go:145-152`): a member |
| 413 | cannot see whether their deployment has a backend, and refusing them teaches |
| 414 | them nothing they can act on. |
| 415 | |
| 416 | **`revoke` and `enforce off` never 503**, and this is draft 4's fix for a |
| 417 | deadlock. Draft 3 had §8 refuse a repair when the scanner was nil and |
| 418 | `Enforce` was set, while §6 had the operator doors 503 without an engine — so |
| 419 | unsetting `CARLOS_STORE_SCANNER` on a deployment with any enforcing record |
| 420 | left the store unrepairable and the deny unclearable without hand-editing the |
| 421 | bucket. Both are **record-only doors that work with no engine built**, which |
| 422 | is the whole point of keeping the `DenyTag` in the record. |
| 423 | |
| 424 | ## 7. The sweep |
| 425 | |
| 426 | 1. Read `store.json`. No `scanning.enabled`, or `Requested.Encrypted` — write |
| 427 | the sidecar `not_applicable` if needed and return. Skip trashed apps |
| 428 | (`deleted.json`, `trash.go:34-36`): a trashed store is destroyed within 30 |
| 429 | days and scanning it bills for data on its way out. |
| 430 | 2. Read `scan.json`; resume from `Current.After` or start fresh. |
| 431 | 3. List a page with `ListObjectsV2` using `StartAfter` — not a |
| 432 | `ContinuationToken`, whose cross-process lifetime is unspecified and, on |
| 433 | Ceph, unmeasured. A key is a portable resume point. The page gives `Size` |
| 434 | and `ETag` for free, which §3.1 and §4.1 both need. |
| 435 | 4. For each key, `Verdict(key, etag)`; count it; `Scan` when unknown and count |
| 436 | the verdict it returns. |
| 437 | 5. **Re-read `store.json`, then write the sidecar** (§7.2), setting `After` to |
| 438 | the page's last key. On the final page move `Current` to `LastComplete`. |
| 439 | |
| 440 | **Honest semantics.** A key added lexically before the resume point mid-sweep |
| 441 | is missed by that sweep and caught by the next. A key deleted after being |
| 442 | counted inflates `Objects` for that pass. Neither breaks a point-in-time |
| 443 | reconciliation. |
| 444 | |
| 445 | ### 7.1 The driver and its budget |
| 446 | |
| 447 | Its own goroutine on the `StartInfraPass` shape (`infrapass.go:49`), started |
| 448 | from `serve()` in `cmd/console/main.go` alongside `StartEconomicsPass` and |
| 449 | `StartInfraPass` (`main.go:326-330`) — **never from `NewServer`**, the rule |
| 450 | `economicspass.go:35-38` states. Not inside `economicsPass`, whose single |
| 451 | goroutine already runs census, statements and billing and already worries |
| 452 | about a slow step delaying the others. |
| 453 | |
| 454 | Draft 2 wanted to ride `storagePass`; that was wrong. `storagePass` returns at |
| 455 | `economicspass.go:137-140` once the day's object exists, so it iterates once |
| 456 | per UTC day. |
| 457 | |
| 458 | **Enumeration.** The pass finds enabled stores the way `storagePass` does — |
| 459 | `ListDirs("apps/")`, per-account `ListDirs`, per-app `objstore.Get` |
| 460 | (`economicspass.go:150-181`). Hourly, that is 24× that pass's listing cost. |
| 461 | Accepted for v1 and named here so it is a known cost rather than a surprise; |
| 462 | an index is the fix if it bites. |
| 463 | |
| 464 | **Budget, with the arithmetic stated.** `CARLOS_STORE_SCAN_BUDGET` bounds work |
| 465 | per app per tick by time (default 60s). An hourly tick at 60s is 24 minutes of |
| 466 | scanning per app per day; at a few hundred objects a minute over INSTREAM, |
| 467 | a million-object bucket takes months, not days. **So the default is honest |
| 468 | only for small and medium stores**, and `scan status` prints the projected |
| 469 | completion whenever a sweep spans more than one tick. A deployment with large |
| 470 | stores raises the budget or shortens the tick, and §1's T widens accordingly. |
| 471 | Apps are swept sequentially in the one goroutine; when N apps × budget exceeds |
| 472 | the interval the ticker simply runs back to back, which is correct behaviour |
| 473 | and not an error. |
| 474 | |
| 475 | **Two drivers, one recorder.** The `scan sweep` verb and the ticker can both |
| 476 | list a page before either writes. For ClamAV that costs a duplicate fetch; for |
| 477 | GuardDuty on-demand it is double-billed. The guarantee is **at most one driver |
| 478 | records**, not that two never run — draft 3 overstated it. |
| 479 | |
| 480 | ### 7.2 The pre-write re-read |
| 481 | |
| 482 | Before every sidecar write, re-read `store.json` and stand down if it is gone |
| 483 | or `scanning.enabled` is nil. |
| 484 | |
| 485 | This closes a hole the retry guard alone does not. A **fresh** sweep's first |
| 486 | write is create-only by construction — no sidecar exists, etag `""`. Sequence: |
| 487 | the sweep reads an enabled `store.json`, lists a page over some seconds, |
| 488 | `Teardown` runs and `PurgeApp` deletes the whole prefix |
| 489 | (`trash.go:229-238`), and the sweep then writes `scan.json` create-only, which |
| 490 | succeeds. The result is an orphan object under a purged app, and that orphan |
| 491 | is expensive: the app-name claim gate refuses a name whose prefix holds any |
| 492 | key (`server.go:2876-2890`), rename is refused (`appsettings.go:468-479`), and |
| 493 | box-side auto-purge never reaches it because the tombstone went last. |
| 494 | |
| 495 | A one-round-trip tolerance is the same one `deleteRecordIfUnchanged` |
| 496 | (`walk.go:519-540`) already accepts. |
| 497 | |
| 498 | **The retry guard still applies to replaces:** on a lost CAS, continue only if |
| 499 | `Current` still exists and its `After` matches where this page started; |
| 500 | otherwise stand down. A conditional write against a deleted key reads as |
| 501 | `ErrCASConflict` (`release/store.go:521-534`; `DirStore` matches at `:260`), |
| 502 | so a blind retry would write create-only and resurrect the sidecar. |
| 503 | |
| 504 | ### 7.3 Teardown disables first |
| 505 | |
| 506 | `Teardown` (`walk.go:390-538`) empties the bucket, deletes it, deletes the |
| 507 | user, unsets config, then deletes the record. Nothing today disables a |
| 508 | scanner, so a GuardDuty plan and its managed EventBridge rule would outlive |
| 509 | the bucket, counting against the §2 cap, with the record naming them gone. |
| 510 | |
| 511 | `Teardown` takes a `Scanner`, calls `EnsureDisabled` before `DeleteBucket`, |
| 512 | and deletes the sidecar. A nil scanner with an enabled record refuses, as a |
| 513 | nil provisioner already does (`walk.go:417`). The destroy door's pre-check |
| 514 | (`appsettings.go:827-833`) tests only `StoreProvisioner` and needs the |
| 515 | matching scanner check, or the refusal surfaces as a generic "Could not |
| 516 | destroy" flash. Both callers (`appsettings.go:845`, `ownership.go:417`) update |
| 517 | with the signature. |
| 518 | |
| 519 | ### 7.4 Engine drift |
| 520 | |
| 521 | Both cases refuse rather than guess. A record naming another engine: the sweep |
| 522 | refuses for that app, because verdicts in the bucket were written under a tag |
| 523 | key the current engine does not read, and recounting them as unscanned would |
| 524 | destroy the coverage history — an operator revokes and re-grants deliberately. |
| 525 | Engine `guardduty` with `Deployment.Endpoint != ""`: refused at boot, since |
| 526 | GuardDuty cannot protect a non-S3 bucket. |
| 527 | |
| 528 | ## 8. Enforcement |
| 529 | |
| 530 | `ScanBlock.Enforce` — nil by default — adds a Deny statement: deny |
| 531 | `s3:GetObject` where `s3:ExistingObjectTag/<Key>` equals `<Value>`. It stores |
| 532 | the tag rather than a bool because `Grant` (`walk.go:133`) takes a |
| 533 | `Provisioner` and has no `Scanner` to ask. |
| 534 | |
| 535 | **Grant writes the deny from the record whatever the scanner is.** A nil |
| 536 | scanner never blocks a repair and never silently drops the deny — the record |
| 537 | holds everything the policy writer needs, which is why it holds a tag. |
| 538 | |
| 539 | **How the policy gets rewritten — the bug drafts 1 and 2 both had.** |
| 540 | `PutUserPolicy` is reachable only through `EnsureUser` (`aws.go:302`), called |
| 541 | from one line (`walk.go:184`), and `Grant` returns early before it for any |
| 542 | healthy store (`walk.go:169-177`). `Rotate` never writes a policy. A flag flip |
| 543 | would have been permanently inert. |
| 544 | |
| 545 | **This repo already fixed this exact bug once.** `internal/mail/walk.go:653-671` |
| 546 | ("C2 — review follow-up") hoisted `EnsureUser` above the idempotence |
| 547 | short-circuit, with a comment recording that the short-circuit had left the |
| 548 | policy naming only the first domain forever. The same move here: |
| 549 | |
| 550 | - `EnsureUser` gains `deny *DenyTag` and stays the single policy writer. |
| 551 | - `Grant` calls it **before** the short-circuit, passing the record's |
| 552 | `Enforce`. It needs only the bucket ARN string, resolved at `walk.go:143-151` |
| 553 | before the guard, so `EnsureBucket` stays behind the guard and the |
| 554 | record-before-create ordering survives (the grant block Put at |
| 555 | `walk.go:160-165` still precedes it). |
| 556 | - `scan enforce on|off` and `scan revoke` write the record and re-run the |
| 557 | grant walk. One convergence path, no second policy writer. Clearing the |
| 558 | record alone would leave the deny live until something re-walked. |
| 559 | |
| 560 | This makes every `store grant` re-run do three IAM calls on a healthy store |
| 561 | where it previously did none. That is the intended trade and the plan should |
| 562 | expect it. |
| 563 | |
| 564 | `policyStatement` (`aws.go:112-116`) has no `Condition` field today; it gains |
| 565 | one, marshalled `omitempty`, so a nil deny emits no `Condition` key at all. |
| 566 | |
| 567 | **The document does change once, estate-wide, and pretending otherwise was a |
| 568 | contradiction in draft 4.** Adding `s3:GetObjectTagging` to the Allow — which |
| 569 | apps need to read their own verdicts — changes the bytes by definition, so |
| 570 | "identical byte for byte when the deny is nil" cannot also be true. The |
| 571 | correct requirement, and what §13 tests, is narrower: **with a nil deny the |
| 572 | document has exactly two statements and no `Condition`**, so an |
| 573 | enforcement-off store never carries a Deny. Every store's policy is rewritten |
| 574 | on its next grant re-walk to gain the new action. That is a one-time, |
| 575 | idempotent change, it costs three IAM calls per re-walk (already accepted |
| 576 | above), and `assertStorePolicyDocument` in `aws_test.go` must be updated to |
| 577 | expect four actions rather than three — it asserts the count, so it fails as a |
| 578 | logic error rather than a compile error and is easy to miss. |
| 579 | |
| 580 | `MemProvisioner` mirrors the signature and gains policy state, which `MemUser` |
| 581 | lacks (`memprovisioner.go:12-15`). Note its `EnsureUser` early-returns for an |
| 582 | existing user, so the policy must be written on **both** branches or |
| 583 | enforcement never reaches a store that already has its principal — which is |
| 584 | every store this feature will ever be turned on for. |
| 585 | |
| 586 | **Anti-forgery, stated at its real strength.** AWS's documented tag-based |
| 587 | access control denies reads *unless* the tag says clean, so unscanned objects |
| 588 | are blocked too. **This design denies only known-infected objects; unscanned |
| 589 | ones stay readable.** That is the right trade under §1's ruling and it must be |
| 590 | said, not implied, to anyone quoting this in a compliance answer. It is safe |
| 591 | only because the app user has no `s3:PutObjectTagging` (`aws.go:130`) and so |
| 592 | cannot mark its own object clean — **that must never be added to the Allow |
| 593 | statement.** The app gains `s3:GetObjectTagging` only, so it can read its own |
| 594 | verdicts; the console gains `GetObject`, `GetObjectTagging` and |
| 595 | `PutObjectTagging` (§10). |
| 596 | |
| 597 | For a `--client-encrypted` store, `enforce` is rejected: nothing will ever be |
| 598 | tagged infected, so it would be theatre. |
| 599 | |
| 600 | ## 9. Surface |
| 601 | |
| 602 | | command | actor | effect | |
| 603 | |---|---|---| |
| 604 | | `carlos store scan request --plaintext\|--client-encrypted` | member | records the ask and the §2.2 attestation | |
| 605 | | `carlos store scan status` | member | the §1 sentences, engine, signature, projection | |
| 606 | | `carlos store scan grant` | operator | enables the engine on the bucket | |
| 607 | | `carlos store scan revoke` | operator | disables it, clears `Enforce`, re-walks | |
| 608 | | `carlos store scan sweep` | operator | advances a sweep by one budget | |
| 609 | | `carlos store scan enforce on\|off` | operator | flips the §8 Deny, re-walks | |
| 610 | |
| 611 | `request`/`grant` mirrors `create`/`grant` on the store itself. API mode only, |
| 612 | like the rest of `store.go`. |
| 613 | |
| 614 | `store status` gains the scan line. `docs/site/cli.md` gains the six verbs |
| 615 | under `### carlos store` (line 393) — the docsite gate reads only the dispatch |
| 616 | switch (`cliscan.go:22`) and cannot see sub-verbs, so this is a house-rule |
| 617 | obligation. |
| 618 | |
| 619 | ## 10. The infra half (`tito/carlos-infrastructure`) |
| 620 | |
| 621 | - **clamd**, per console box: a systemd unit, `freshclam` on a timer, a unix |
| 622 | socket the console's unit user can reach (group `clamav`), |
| 623 | `StreamMaxLength` matched to `CARLOS_CLAMD_MAX_BYTES`, and ~1–1.5 GB RAM for |
| 624 | resident signatures. This is the primary path and all a self-hoster needs |
| 625 | beyond the store primitive itself. |
| 626 | - **Console S3 permissions**: `GetObject`, `GetObjectTagging`, |
| 627 | `PutObjectTagging` on `arn:aws:s3:::<BucketPrefix>*`, alongside the listing |
| 628 | it already does. |
| 629 | - **GuardDuty, only if used**: a scanning role GuardDuty assumes (EventBridge |
| 630 | managed rule, `GetObject`, `PutObjectTagging`, KMS where encrypted), passed |
| 631 | at plan creation — so the console needs `iam:PassRole` on it plus |
| 632 | `guardduty:SendObjectMalwareScan`. Scope the role to |
| 633 | `arn:aws:s3:::<BucketPrefix>*` so it needs no per-app edit. |
| 634 | |
| 635 | ## 11. Gates and risks |
| 636 | |
| 637 | **Gate — measure before building the GuardDuty engine**: whether |
| 638 | `SendObjectMalwareScan` tags an object uploaded before tagging was enabled |
| 639 | (§4.2). Blocks GuardDuty only. |
| 640 | |
| 641 | **Risks to verify during implementation:** |
| 642 | |
| 643 | - **Hetzner object tagging.** `s3compat`'s quirks table exists because Ceph |
| 644 | RGW already differed from S3 in a way that broke every conditional write. |
| 645 | Measure `PutObjectTagging`/`GetObjectTagging` before relying on them; the |
| 646 | fallback is per-object verdicts in the sidecar keyed by key and ETag. |
| 647 | - **clamd throughput.** One connection per object with a size ceiling sets the |
| 648 | real budget in §7.1. Measure before promising any completion window, and |
| 649 | revisit `IDSESSION` if per-connection cost dominates. |
| 650 | - **Enumeration cost** at hourly cadence (§7.1). |
| 651 | |
| 652 | ## 12. Out of scope |
| 653 | |
| 654 | - Scanning the deployment's control-plane bucket, release artifacts, or |
| 655 | instance replicas. App stores only. |
| 656 | - A non-AWS store provisioner (§2.1) — the dependency for a fully self-hosted |
| 657 | story, not delivered here. |
| 658 | - Quarantine as movement. Verdicts are tags; nothing is copied aside or |
| 659 | deleted. Deleting a member's data on a scanner's say-so is a policy decision |
| 660 | nobody has made. |
| 661 | - Scan-on-upload latency guarantees. |
| 662 | - Notifying members of an infected object. |
| 663 | |
| 664 | ## 13. Testing |
| 665 | |
| 666 | - **`Scanner` contract test** over `MemScanner` and each real engine: enable |
| 667 | twice, disable twice, scan twice, verdict on an absent object, verdict on a |
| 668 | changed ETag. Note `contract_test.go` today is `TestDeliveredContractMinio`, |
| 669 | a delivered-credential wire test — there is no `Provisioner` idempotence |
| 670 | contract to sit beside, so this is a new file. |
| 671 | - **Version binding.** An object rewritten after being tagged reads |
| 672 | `VerdictUnknown` and is rescanned (§3.1). |
| 673 | - **Synchronous counting.** A ClamAV sweep's *first* pass reports objects as |
| 674 | scanned, not unscanned. |
| 675 | - **`LastComplete` survives** the tick after completion — draft 2's bug. |
| 676 | - **Resurrection, both paths.** Fresh sweep with no sidecar, `Teardown` plus |
| 677 | `PurgeApp` between list and write: no `scan.json` remains and the app name |
| 678 | is claimable. And the retry path on a replace. |
| 679 | - **Migration.** A sidecar whose `Account` disagrees with its path is refused. |
| 680 | - **Purge guard, additive.** A record carrying `scanning` still makes |
| 681 | `PurgeApp` refuse on `grant`/`provisioned` and ignore the new key. |
| 682 | - **Policy round-trip.** `EnsureUser` with `deny == nil` produces exactly two |
| 683 | statements and no `Condition` key; with a `DenyTag`, three statements, a |
| 684 | `Condition` on the verdict tag, and no `PutObjectTagging` in any of them. |
| 685 | - **Enforcement converges** on a healthy provisioned store, and `enforce off` |
| 686 | works with no engine built (§6's deadlock, asserted). |
| 687 | - **Teardown** disables first, deletes the sidecar, refuses on a nil scanner. |
| 688 | - **Encrypted attestation.** A `--client-encrypted` store is swept |
| 689 | `not_applicable`, `scan status` refuses §1's first sentence, `enforce` is |
| 690 | rejected. |
| 691 | - **clamd down** produces a pass error and zero written verdicts. |
| 692 | - **Skipped is not clean**, and an over-ceiling object is tagged without a |
| 693 | `GetObject`. |
| 694 | - **Trashed apps are skipped.** |
| 695 | - **Gate order and inert deployment.** `grant`/`sweep` 503 without an engine; |
| 696 | `request`, `revoke` and `enforce off` still work. |
| 697 | |
amadan