carlos / platform Public

litestreamfacts gate: snapshot retention is awaited at the final snapshot (#266)

TestGate_AnchoredLineageThroughStore failed once under load with
"snapshot retention did not leave exactly the newest snapshot: L9 [1,17]
[1,18]". The round waited for level 9 to hold exactly one snapshot, then
closed the store and asserted the same.

One snapshot is not a state that lasts. The store's snapshot monitor
snapshots every interval (300 ms here) and deletes snapshots older than
the retention (400 ms), so it normally keeps the previous snapshot beside
the new one. A tick that runs 100 ms late finds the previous snapshot past
retention and leaves one, which is behind the position if a write follows.
The wait saw that, and the next tick wrote the final snapshot and kept the
other, with the store closed before a third tick could remove it.

The wait is now for one snapshot at the round's final position. After
that a tick can only rewrite the same file, and retention never deletes
the newest. The deadline is unchanged at 5 s.

f36718c5 (#263) does not widen the window: it sets the snapshot interval
and retention in internal/lease and internal/ops, and this test builds its
own litestream.Store with its own intervals and imports neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 1 week ago ce7b944113afd1e9dddf7a98a91c0d9fd223c842 parent 8fef823

1 file changed, +13 −1

internal/sim/litestreamfacts/facts_test.go +13 −1

CI passed — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	334.105s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.212s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.271s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.381s
ok  	github.com/carlosframework/platform/internal/activator	226.422s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.004s
ok  	github.com/carlosframework/platform/internal/analytics	0.018s
ok  	github.com/carlosframework/platform/internal/appconfig	2.329s
ok  	github.com/carlosframework/platform/internal/apperr	0.076s
ok  	github.com/carlosframework/platform/internal/applog	0.126s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.006s
ok  	github.com/carlosframework/platform/internal/boxid	0.507s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.009s
ok  	github.com/carlosframework/platform/internal/caps	0.011s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.692s
ok  	github.com/carlosframework/platform/internal/changelog	0.297s
ok  	github.com/carlosframework/platform/internal/clamd	0.065s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	361.520s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.841s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.113s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	26.137s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.007s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.122s
ok  	github.com/carlosframework/platform/internal/gate	1.659s
ok  	github.com/carlosframework/platform/internal/genselect	0.014s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.030s
ok  	github.com/carlosframework/platform/internal/host	95.272s
ok  	github.com/carlosframework/platform/internal/hostactors	0.535s
ok  	github.com/carlosframework/platform/internal/hostbox	2.113s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.491s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.012s
ok  	github.com/carlosframework/platform/internal/hostexec	4.985s
ok  	github.com/carlosframework/platform/internal/hostfence	0.033s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.529s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.013s
ok  	github.com/carlosframework/platform/internal/hostguard	0.966s
ok  	github.com/carlosframework/platform/internal/hostops	1.497s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.012s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.456s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.133s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.012s
ok  	github.com/carlosframework/platform/internal/infraview	0.012s
ok  	github.com/carlosframework/platform/internal/instancectl	1.834s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.725s
ok  	github.com/carlosframework/platform/internal/instancefs	0.016s
ok  	github.com/carlosframework/platform/internal/instancekey	0.013s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.017s
ok  	github.com/carlosframework/platform/internal/layout	1.439s
ok  	github.com/carlosframework/platform/internal/lease	9.447s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.005s
ok  	github.com/carlosframework/platform/internal/ledger	0.010s
ok  	github.com/carlosframework/platform/internal/mail	0.160s
ok  	github.com/carlosframework/platform/internal/maintenance	0.077s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.534s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.076s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	20.990s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.778s
ok  	github.com/carlosframework/platform/internal/objstore	0.171s
ok  	github.com/carlosframework/platform/internal/ops	0.160s
ok  	github.com/carlosframework/platform/internal/park	0.225s
ok  	github.com/carlosframework/platform/internal/privatefile	0.003s
ok  	github.com/carlosframework/platform/internal/provision	13.192s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.006s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.022s
ok  	github.com/carlosframework/platform/internal/relay	7.230s
ok  	github.com/carlosframework/platform/internal/release	19.269s
ok  	github.com/carlosframework/platform/internal/rollout	3.764s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.006s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.024s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.003s
ok  	github.com/carlosframework/platform/internal/schedule	2.324s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.725s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.704s
ok  	github.com/carlosframework/platform/internal/status	0.042s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.212s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.006s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.585s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
  minio: downloaded, digest verified
  mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
  mc: downloaded, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.022s
=== RUN   TestT4StorageProbeMinio
2026/10/13 22:53:52 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DE38B075401D7B, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.020s
minio contract: PASS

diff --git a/internal/sim/litestreamfacts/facts_test.go b/internal/sim/litestreamfacts/facts_test.go

index 2f76c6e5c..f66168944 100644

--- a/internal/sim/litestreamfacts/facts_test.go

+++ b/internal/sim/litestreamfacts/facts_test.go

@@ -817,7 +817,19 @@ func TestGate_AnchoredLineageThroughStore(t *testing.T) {

for _, lvl := range []int{1, 2, 3} {

waitFor(t, fmt.Sprintf("a level-%d compaction", lvl), 5*time.Second, func() bool { return len(files(t, gen, lvl)) > 0 })

}

- waitFor(t, "snapshot retention", 5*time.Second, func() bool { return len(files(t, gen, 9)) == 1 })

+ // Retention is awaited at the state that lasts: one snapshot, at the

+ // round's final position. One snapshot alone can be a moment in

+ // passing. The monitor snapshots every interval and keeps the previous

+ // snapshot while it is younger than the retention, so a tick that runs

+ // 100 ms late removes the previous one and leaves a single snapshot

+ // that is behind the position; the next tick adds the final snapshot

+ // beside it, and a round that had stopped waiting closed the store

+ // with two (#266). Once the only snapshot is at the final position a

+ // later tick can only rewrite that same file.

+ waitFor(t, "snapshot retention", 5*time.Second, func() bool {

+ l9 := files(t, gen, 9)

+ return len(l9) == 1 && int(l9[0].MaxTXID) == pos

+ })

waitFor(t, "level-0 expiry", 5*time.Second, func() bool { return len(files(t, gen, 0)) < l0Before+8 })

if err := store.Close(ctx); err != nil {

t.Fatal(err)