amadan

carlos / platform Public

Revive on-demand object scanning with guarded queue workers

claude c0040c1e74de12752f7ac0ac43d3cab670d8fd7f parent ce8c425
61 files changed, +8559 −234
  • .gitignore +1 −0
  • apimode.go +31 −5
  • cmd/console/main.go +15 −0
  • docs/site/cli.md +49 −0
  • docs/site/nav.json +5 −0
  • docs/site/scan-callbacks.md +233 −0
  • docs/superpowers/notes/census-proposal-scan-demand-revival.md +42 −0
  • docs/superpowers/plans/2026-09-02-scan-on-demand.md +227 −49
  • docs/superpowers/specs/2026-09-02-scan-on-demand-design.md +143 −16
  • internal/console/apiscan.go +372 −0
  • internal/console/apiscan_keys_test.go +244 −0
  • internal/console/apiscan_revival_test.go +61 −0
  • internal/console/apiscan_rotation_race_test.go +185 −0
  • internal/console/apiscan_test.go +19 −2
  • internal/console/apiscanqueue_test.go +433 −0
  • internal/console/apiscanstatus.go +286 −0
  • internal/console/apiscanstatus_revival_test.go +202 −0
  • internal/console/apiscanstatus_test.go +556 −0
  • internal/console/apistore_test.go +1 −0
  • internal/console/apisystem.go +108 −2
  • internal/console/apisystem_test.go +14 −0
  • internal/console/authcli.go +14 −0
  • internal/console/claim_invariant_fixtures_test.go +82 −31
  • internal/console/claim_invariant_probes_read_test.go +4 −0
  • internal/console/claim_invariant_world_test.go +22 −20
  • internal/console/claim_scan_revival_test.go +592 −0
  • internal/console/claim_workers.go +1 −0
  • internal/console/scan_group_permission_revival_test.go +73 −0
  • internal/console/scan_review_regression_test.go +173 −0
  • internal/console/scandeliver.go +307 −0
  • internal/console/scandeliver_test.go +427 −0
  • internal/console/scandrain.go +386 −0
  • internal/console/scandrain_recovery_test.go +137 −0
  • internal/console/scandrain_test.go +737 −0
  • internal/console/scanhook.go +180 −0
  • internal/console/scanhook_test.go +145 −0
  • internal/console/scankeysync.go +105 −0
  • internal/console/scankeysync_test.go +120 −0
  • internal/console/scanresult_collection_revival_test.go +84 −0
  • internal/console/scansign.go +170 −0
  • internal/console/scansign_test.go +228 −0
  • internal/console/server.go +28 −0
  • internal/console/svcroutes_test.go +3 −2
  • internal/console/testdata/claim-routes.txt +5 −0
  • internal/docsite/fences_test.go +6 −0
  • internal/hostops/storage_classifier.go +19 −0
  • internal/hostops/storage_classifier_test.go +40 −2
  • internal/objstore/memscanner.go +24 −0
  • internal/objstore/objstore.go +6 −3
  • internal/objstore/queue.go +438 −66
  • internal/objstore/queue_test.go +472 −13
  • internal/objstore/scanresult.go +61 −19
  • internal/provision/plan14_root_reads_test.go +10 −0
  • internal/provision/plan9_b8_census_test.go +1 −1
  • internal/provision/scan_demand_census_test.go +27 −0
  • internal/provision/testdata/plan14-root-reads-scan-demand-revival.tsv +18 −0
  • internal/usage/scan.go +77 −0
  • internal/usage/scan_test.go +72 −0
  • internal/usage/usage.go +5 −0
  • scan.go +9 −1
  • system.go +24 −2

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	330.609s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.891s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.288s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.713s
ok  	github.com/carlosframework/platform/internal/activator	251.196s
ok  	github.com/carlosframework/platform/internal/agentsock	0.015s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.019s
ok  	github.com/carlosframework/platform/internal/appconfig	2.517s
ok  	github.com/carlosframework/platform/internal/apperr	0.073s
ok  	github.com/carlosframework/platform/internal/applog	0.131s
ok  	github.com/carlosframework/platform/internal/awsinv	0.005s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	1.072s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.012s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.724s
ok  	github.com/carlosframework/platform/internal/changelog	0.339s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.862s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	436.350s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.931s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.016s
ok  	github.com/carlosframework/platform/internal/dbverify	0.009s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.010s
ok  	github.com/carlosframework/platform/internal/docsite	0.011s
ok  	github.com/carlosframework/platform/internal/domains	2.424s
ok  	github.com/carlosframework/platform/internal/doorkey	0.014s
ok  	github.com/carlosframework/platform/internal/economics	0.006s
ok  	github.com/carlosframework/platform/internal/edge	45.197s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.013s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.170s
ok  	github.com/carlosframework/platform/internal/gate	1.685s
ok  	github.com/carlosframework/platform/internal/genselect	0.038s
ok  	github.com/carlosframework/platform/internal/geo	0.044s
ok  	github.com/carlosframework/platform/internal/geoip	0.041s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.216s
ok  	github.com/carlosframework/platform/internal/host	192.067s
ok  	github.com/carlosframework/platform/internal/hostactors	5.367s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.015s
ok  	github.com/carlosframework/platform/internal/hostbox	2.752s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.620s
ok  	github.com/carlosframework/platform/internal/hostenv	0.015s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.015s
ok  	github.com/carlosframework/platform/internal/hostexec	5.105s
ok  	github.com/carlosframework/platform/internal/hostfence	0.038s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.478s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.015s
ok  	github.com/carlosframework/platform/internal/hostguard	1.057s
ok  	github.com/carlosframework/platform/internal/hostops	1.667s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.015s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.015s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.471s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.155s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.015s
ok  	github.com/carlosframework/platform/internal/infraview	0.014s
ok  	github.com/carlosframework/platform/internal/instancectl	5.255s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.767s
ok  	github.com/carlosframework/platform/internal/instancefs	0.028s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.023s
ok  	github.com/carlosframework/platform/internal/layout	1.602s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.002s
ok  	github.com/carlosframework/platform/internal/lease	9.956s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.374s
ok  	github.com/carlosframework/platform/internal/ledger	0.014s
ok  	github.com/carlosframework/platform/internal/mail	0.176s
ok  	github.com/carlosframework/platform/internal/maintenance	0.087s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.538s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.069s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.186s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.840s
ok  	github.com/carlosframework/platform/internal/objstore	0.220s
ok  	github.com/carlosframework/platform/internal/ops	0.184s
ok  	github.com/carlosframework/platform/internal/park	0.228s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	39.869s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.008s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.488s
ok  	github.com/carlosframework/platform/internal/relay	7.231s
ok  	github.com/carlosframework/platform/internal/release	25.271s
ok  	github.com/carlosframework/platform/internal/rollout	3.840s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.058s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.579s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.618s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.371s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.199s
ok  	github.com/carlosframework/platform/internal/status	0.064s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.246s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.005s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.613s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 49807522 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
  minio: downloaded, digest verified
  mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
  mc: downloaded, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.025s
=== RUN   TestT4StorageProbeMinio
2026/10/21 07:40:51 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E07B824BDC4FFF, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.022s
minio contract: PASS
.gitignore

Loading diff…

apimode.go

Loading diff…

cmd/console/main.go

Loading diff…

docs/site/cli.md

Loading diff…

docs/site/nav.json

Loading diff…

docs/site/scan-callbacks.md

Loading diff…

docs/superpowers/notes/census-proposal-scan-demand-revival.md

Loading diff…

docs/superpowers/plans/2026-09-02-scan-on-demand.md

Loading diff…

docs/superpowers/specs/2026-09-02-scan-on-demand-design.md

Loading diff…

internal/console/apiscan.go

Loading diff…

internal/console/apiscan_keys_test.go

Loading diff…

internal/console/apiscan_revival_test.go

Loading diff…

internal/console/apiscan_rotation_race_test.go

Loading diff…

internal/console/apiscan_test.go

Loading diff…

internal/console/apiscanqueue_test.go

Loading diff…

internal/console/apiscanstatus.go

Loading diff…

internal/console/apiscanstatus_revival_test.go

Loading diff…

internal/console/apiscanstatus_test.go

Loading diff…

internal/console/apistore_test.go

Loading diff…

internal/console/apisystem.go

Loading diff…

internal/console/apisystem_test.go

Loading diff…

internal/console/authcli.go

Loading diff…

internal/console/claim_invariant_fixtures_test.go

Loading diff…

internal/console/claim_invariant_probes_read_test.go

Loading diff…

internal/console/claim_invariant_world_test.go

Loading diff…

internal/console/claim_scan_revival_test.go

Loading diff…

internal/console/claim_workers.go

Loading diff…

internal/console/scan_group_permission_revival_test.go

Loading diff…

internal/console/scan_review_regression_test.go

Loading diff…

internal/console/scandeliver.go

Loading diff…

internal/console/scandeliver_test.go

Loading diff…

internal/console/scandrain.go

Loading diff…

internal/console/scandrain_recovery_test.go

Loading diff…

internal/console/scandrain_test.go

Loading diff…

internal/console/scanhook.go

Loading diff…

internal/console/scanhook_test.go

Loading diff…

internal/console/scankeysync.go

Loading diff…

internal/console/scankeysync_test.go

Loading diff…

internal/console/scanresult_collection_revival_test.go

Loading diff…

internal/console/scansign.go

Loading diff…

internal/console/scansign_test.go

Loading diff…

internal/console/server.go

Loading diff…

internal/console/svcroutes_test.go

Loading diff…

internal/console/testdata/claim-routes.txt

Loading diff…

internal/docsite/fences_test.go

Loading diff…

internal/hostops/storage_classifier.go

Loading diff…

internal/hostops/storage_classifier_test.go

Loading diff…

internal/objstore/memscanner.go

Loading diff…

internal/objstore/objstore.go

Loading diff…

internal/objstore/queue.go

Loading diff…

internal/objstore/queue_test.go

Loading diff…

internal/objstore/scanresult.go

Loading diff…

internal/provision/plan14_root_reads_test.go

Loading diff…

internal/provision/plan9_b8_census_test.go

Loading diff…

internal/provision/scan_demand_census_test.go

Loading diff…

internal/provision/testdata/plan14-root-reads-scan-demand-revival.tsv

Loading diff…

internal/usage/scan.go

Loading diff…

internal/usage/scan_test.go

Loading diff…

internal/usage/usage.go

Loading diff…

scan.go

Loading diff…

system.go

Loading diff…