carlos / platform Public

s3compat: one home for S3-compatible quirks; fix CAS on Ceph RGW (#92)

Hetzner Object Storage compares If-Match against its UNQUOTED internal
etag, so the quoted entity-tag form S3 returns — and that the AWS SDK
faithfully round-trips — never matches. Measured against the carlos
bucket, a single writer lost compare-and-swap 20 times out of 20:

  PUT If-None-Match: *  (absent)        -> OK
  PUT If-None-Match: *  (exists)        -> 412   correctly rejected
  PUT If-Match: "<correct etag>"        -> 412   <-- the bug
  PUT If-Match:  <correct etag>         -> OK
  PUT If-Match:  <wrong etag, unquoted> -> 412   correctly rejected
  DELETE If-Match: "<etag>"             -> OK    (quoted is fine here)

The fifth line is what makes the fix safe rather than reckless: the
condition IS evaluated, only the quoting form differs, so stripping the
quotes repairs compare-and-swap instead of silently disabling it. RFC 9110
entity-tags are quoted, so AWS is right and RGW is wrong; CARLOS still has
to interoperate.

Because create-only writes work, a fresh deployment looks healthy and only
REPLACES fail — which is every CAS in CARLOS: release pointers and
manifests, the ledger head, appconfig generations, provision records,
status snapshots, metrics rollups. On the private CARLOS box this showed up
as the box reporting itself UNREACHABLE on its own status page, ~68 CAS
conflicts every three hours.

It also defeats hibernation deterministically, which is worse than the
known spurious-403 wart. litestream's leaser renews at TTL/3 with
PUT If-Match, so a woken instance lost its lease at the FIRST renewal and
the activator stood it back down — every five minutes, not occasionally.
Confirmed by driving lss3.Leaser directly: Acquire OK (If-None-Match:*),
Renew FAILS ("lease not held"), Release OK.

New internal/s3compat owns both halves of "this store is not AWS":

- the universal settings (endpoint, path-style, when-required checksums),
  which release.NewS3Store and internal/lease had each duplicated;
- a QUIRKS TABLE, so adding a provider is one entry naming what it matches,
  what it changes, and the measurement that justifies it.

The If-Match fix is an SDK middleware in the BUILD step — after
serialization writes the header, before Finalize signs it. If-Match is a
signed header, so a later rewrite would produce SignatureDoesNotMatch
rather than a fix; a test asserts if-match is still in SignedHeaders.

Scoped deliberately:
- PutObject only. Conditional DELETE accepts the quoted form on the same
  store (it is how the lease releases today, and it works); a working path
  is not worth touching on an untested guess.
- Matched by hostname to the provider actually measured. CARLOS_S3_QUIRKS
  force-enables a quirk by name, so a new store can be tried without a
  rebuild.
- s3compat.Options() is a NO-OP without CARLOS_S3_ENDPOINT, so the
  flagship's stock-AWS path is untouched — including registering no
  middleware at all. release.NewS3Store keeps applying its checksum
  settings unconditionally, as it always has, rather than quietly changing
  behaviour on the one deployment that cannot be tested from here.

This is also the only way the fix reaches litestream: its leaser takes a
ready-made client via SetClient and has no provider handling of its own.
Its replica client needs nothing here (LTX writes are unconditional).
Worth reporting upstream separately.

Verified against the real Hetzner bucket, quirk firing on hostname alone:
- CAS replaces: 20/20 conflicts before -> 0/20 after, data landed.
- Leaser: Acquire + 3 consecutive renewals + Release all clean.

Tests: the RGW fake reproduces the exact wire behaviour, with the quirk-off
subtest asserting the failure so the quirk-on pass cannot be vacuous, plus
a stale-etag test proving CAS still rejects a genuine conflict.

go build/vet/test ./... clean, 21 packages; gofmt clean. go.mod: credentials
promoted indirect -> direct (a test imports it), same version, go.sum unchanged.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>