amadan

carlos / platform Public

Count failures arriving during durability snapshots without retrying

Paul Campbell pushed by paul@keymail.dev 665bcb7cff602a64ab2f54a49fbc290d57e50eae parent eea00ee
3 files changed, +39 −11
  • implementation/carloku-paid-launch/AGENT_HANDOFF.md +4 −3
  • internal/usage/durabilityfailures_test.go +27 −0
  • internal/usage/durabilityledger.go +8 −8

CI passed — run details

diff --git a/implementation/carloku-paid-launch/AGENT_HANDOFF.md b/implementation/carloku-paid-launch/AGENT_HANDOFF.md
index b9b786568..ffb65a7d4 100644
--- a/implementation/carloku-paid-launch/AGENT_HANDOFF.md
+++ b/implementation/carloku-paid-launch/AGENT_HANDOFF.md
@@ -1,6 +1,6 @@
# Agent pickup instructions — Carloku paid launch
-CURRENT GATE: local build, vet and `make test` GREEN after fixing the intermittent durability snapshot failure. Amadan CI must run on the new commit. This remains a checkpoint of unfinished work.
+CURRENT GATE: local build, vet and `make test` GREEN after fixing the intermittent durability snapshot failure. Amadan CI passed on the first fix commit; the reviewed amendment still needs its own CI run. This remains a checkpoint of unfinished work.
This file is context and instructions for the next coding agent. The branch is a work-in-progress checkpoint, not a release candidate. Do not merge, deploy, enable billing, or infer that the overnight launch goal was achieved.
@@ -40,7 +40,8 @@ Read `AGENTS.md` and `docs/superpowers/specs/2026-09-07-carloku-pricing-plans-de
- Quantity/checkout/membership audit tests passed `-race` (49.015s), before the final invoice lock changes. Log: `quantity-complete-root-race.txt`.
- New host ownership, locked-ledger timeout, shutdown/publication and actual fake-HTTP SDK List/Copy/Delete-to-ledger tests passed `-race` (1.551s). Log: `host-durability-sdk-root-race.txt`.
- New activator meter selection plus existing staging-restore tests passed `-race` (1.032s). Log: `activator-metering-root-race.txt`.
-- The prior intermittent failure was real: a snapshot could read SQLite before a background failure drain, then read the latch after that drain cleared it. The fix retries when the latch changes during the read and checks the final latch state under the writer mutex, which spans commit and subtraction. A forced-interleaving regression and the original late-failure test passed `-race -count=20`; the original passed `-count=30` without race instrumentation. The complete local gate (`go build ./...`, `go vet ./...`, `make test` with JS parity) passed on 2026-09-18. Amadan CI at the previous handoff commit `099586ea` also passed, demonstrating why a single green run did not settle this intermittent bug. New-commit CI is pending.
+- The prior intermittent failure was real: a snapshot could read SQLite before a background failure drain, then read the latch after that drain cleared it. The fix retries when a committed drain crosses the read and checks the final latch state under the writer mutex, which spans commit and subtraction. A forced-interleaving regression, an arrival-during-read regression, and the original late-failure test passed `-race -count=20`; the original also passed `-count=30` without race instrumentation. The complete local gate (`go build ./...`, `go vet ./...`, `make test` with JS parity) passed twice on 2026-09-18. Amadan CI passed on both the previous handoff commit `099586ea` and the first fix commit `eea00ee`, demonstrating why a single green run did not settle this intermittent bug. The reviewed amendment needs a fresh CI run.
+- An Opus review found that retrying on every arriving failure could starve publication during a recording outage. The amendment retries only when a committed drain clears the latch; a second Opus review confirmed the missed-failure race remains closed. Remaining risk: eight consecutive drain commits during one snapshot still return an error. Because that can leave an older complete-looking publication in the bucket, production billing needs a safe publication/freeze response before attestation or charges turn on. The reviewer also noted a pre-existing unsynchronized read of `l.state.epoch` in the snapshot; it has not been addressed here.
- CURRENT tree passed all five cross-builds (Darwin arm64/amd64, Linux arm64, Windows amd64/arm64). Log: `handoff-cross.txt`.
- Final whole-branch Fable review and production/authority Opus review remain outstanding. All three Claude agents stopped at the shared usage limit, which reports reset at05:50 Europe/Dublin on2026-09-17 (04:50UTC). No agents remain running. That reset time has now passed; availability has not been rechecked for this handoff. Relevant task briefs/reports are in scratch, especially `durability-host-integration-brief.txt`, `opus-durability-integration-design.txt`, `quantity-history-hardening-brief.txt` and the invoice reports. The older design's arbitrary five-minute late-callback slack is NOT accepted proof.
@@ -78,7 +79,7 @@ A scratch SQLite experiment was also found replacing local root main.go. It was
## Next agent actions
1. Read Amadan CI for the new durability-fix commit and inspect the complete branch diff as WIP. The local gate is green, but that does not make the launch ready.
-2. Finish/review newest host wiring: full per-route move/checkpoint tests and client inventory, source completeness folding, scoped external-client exclusion, crash/late-callback/restart proof, bounded ledger retention and real-disk latency measurement. Keep unattested usage unbillable until proved.
+2. Resolve the sustained-drain publication error and review the host wiring: `bindingFailed` is set on unmetered fallbacks but currently only the permanently false attestation prevents billing. Finish per-route move/checkpoint tests and client inventory, source completeness folding, scoped external-client exclusion, crash/late-callback/restart proof, bounded ledger retention and real-disk latency measurement. Keep unattested usage unbillable until proved.
3. Implement P5 collector with durable deduplication, immutable historical bucket ownership and a settled transfer contract. Prepare exact scoped logging activation before seeking production approval.
4. Complete partial-month billing, quantity/proration collection, operator review/house doors, Free enrollment/caps/idle/flush-only, dormancy warnings/archive and revenue-pool accounting required by the spec. Use the requirements audit rather than treating these as optional follow-ups.
5. After external prerequisites and copy approvals, provision/verify Stripe catalog, run the real sandbox purchase/webhook/portal flow, obtain independent reviews, pass Amadan CI and release gates, then land/deploy through the required reviewed path and verify live behaviour. No main merge or deployment is part of this handoff.
diff --git a/internal/usage/durabilityfailures_test.go b/internal/usage/durabilityfailures_test.go
index b2b63445b..06d3c5119 100644
--- a/internal/usage/durabilityfailures_test.go
+++ b/internal/usage/durabilityfailures_test.go
@@ -68,6 +68,33 @@ func TestDurabilitySnapshotConcurrentFailureDrain(t *testing.T) {
}
}
+func TestDurabilitySnapshotCountsFailureArrivingDuringRead(t *testing.T) {
+ clock := newLedgerClock(ledgerDay0)
+ l := openTestLedger(t, t.TempDir(), "box1", clock, true)
+ defer l.Close(context.Background(), nil)
+ date := DayOf(ledgerDay0)
+ // Hold the drainer idle so this test isolates arrival from subtraction.
+ l.latch.mu.Lock()
+ l.latch.draining = true
+ l.latch.mu.Unlock()
+ defer func() {
+ l.latch.mu.Lock()
+ l.latch.draining = false
+ l.latch.mu.Unlock()
+ }()
+ reads := 0
+ l.snapshotPause = func(string) {
+ reads++
+ if reads == 1 {
+ l.latchFailure(ledgerDay0)
+ }
+ }
+ content, err := l.Snapshot(date)
+ if err != nil || content.RecordingFailures != 1 || reads != 1 {
+ t.Fatalf("failure during read: failures=%d reads=%d err=%v", content.RecordingFailures, reads, err)
+ }
+}
+
// ---- Recording failures: the non-blocking latch and its proofs ------------
//
// Incomplete is called from the request path with a 25 ms budget around
diff --git a/internal/usage/durabilityledger.go b/internal/usage/durabilityledger.go
index 96c3e07f4..b8891a258 100644
--- a/internal/usage/durabilityledger.go
+++ b/internal/usage/durabilityledger.go
@@ -130,8 +130,9 @@ type DurabilityLedger struct {
// covers every day a further failure touched.
type failureLatch struct {
mu sync.Mutex
- // generation changes whenever a failure arrives or a committed drain
- // removes one. A snapshot retries if either happens during its read.
+ // generation changes when a committed drain removes failures. A new
+ // failure is counted from the latch at the end of a snapshot; a drain
+ // needs a retry because its row may be newer than the read transaction.
generation uint64
days map[string]int64 // date -> failures not yet in the database
// overflow holds failures past the per-date bound: n of them, spread
@@ -1236,7 +1237,6 @@ func (l *DurabilityLedger) latchFailure(at time.Time) {
}
la := &l.latch
la.mu.Lock()
- la.generation++
if la.closed {
la.late++
if la.lateFrom.IsZero() || at.Before(la.lateFrom) {
@@ -1633,9 +1633,9 @@ func (l *DurabilityLedger) Snapshot(date string) (DurabilityContent, error) {
return c, err
}
-// snapshot retries when a failure enters or leaves the latch during its
-// database read. Otherwise a read may precede a drain's commit while the
-// final latch check follows its subtraction, making a failed day look clean.
+// snapshot retries when a committed drain clears failures from the latch
+// during its database read. The read may predate the commit while the
+// final latch check follows subtraction, making a failed day look clean.
func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (DurabilityContent, dayLineage, error) {
for range 8 {
_, before := l.latch.pendingAt(date)
@@ -1648,10 +1648,10 @@ func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (Durabilit
}
return content, lineage, nil
}
- return DurabilityContent{}, dayLineage{}, errors.New("usage ledger: recording failures changed during snapshot")
+ return DurabilityContent{}, dayLineage{}, errors.New("usage ledger: failure drains crossed eight snapshot attempts")
}
-var errSnapshotLatchChanged = errors.New("usage ledger: recording failures changed during snapshot")
+var errSnapshotLatchChanged = errors.New("usage ledger: failure drain crossed snapshot")
// snapshotOnce builds the content and reads the day's publication lineage in
// ONE read transaction on the reader connection: the watermark is the