amadan

carlos / platform Public

moves: into-home revert, rollback, abort, loss and conversion witnesses (home part 2)

Task 4's witnesses for a move ca -> home, each from clean claim state:

- TestMoveIntoHomeAbortParksTheHomeLane: Abort's lane park in @home;
  home-box's engine archives, its hostbox.Regional acknowledges, ca's
  refuses; the record clears only after.
- TestMoveIntoHomePreFlipRevert: Revert's lane park in @home; the
  revert probe re-admits on ca-box through the production adapter and
  recovery owners, uncovered.
- TestMoveIntoHomeCleanRollback: the probe admitted on home-box (ie)
  through the production owners fails; TargetReleased names @home; the
  source's recovery admission is clean-return in ca, its predecessor the
  home target's admission in ie.
- TestMoveIntoHomeDirtyReturn: the return lane in data-ca under ca's
  lease; the return generation admitted sealed and materialized with the
  real C8; the return lane park in ca; pre-move parks after the return.
- TestHomeRegionalAcknowledgesBothSpellings, and
  TestCommittedLandingIntoHomeConvertsOnTargetFence.
- hostops: TestMoveIntoHomeDeclareLost and
  TestMoveIntoHomeFencedTargetRelease.

The call-site inventory's 14 pending rows now name written witnesses.

Fix found by the revert witness: the revert's probing step reused its
pre-probe claim read for the probed CAS, which a real recovery probe
wake (it admits a writer) always made stale, so the revert never left
probing and each pass admitted another fresh generation. It now reads
the claim again (TestRevertProbedAfterARealAdmission, red before).
Census: revertRun.probing frozen at its sibling units' disposition in
the home-part2 overlay (ruling owed); the B8 row re-digested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 17438031b72322ef8ca07f8d9e27bd3ff00644de parent 6e5e4f6
10 files changed, +1344 −129
  • home_region_args_test.go +14 −15
  • internal/host/moveshome_target_test.go +997 −99
  • internal/host/movesrecover_test.go +19 −7
  • internal/host/movesrevert.go +7 −0
  • internal/host/movesrevert_test.go +31 −0
  • internal/host/movesrollback_test.go +2 −2
  • internal/hostops/move_home_target_test.go +263 −0
  • internal/provision/plan14_root_reads_test.go +4 −3
  • internal/provision/plan9_b8_census_test.go +2 −1
  • internal/provision/testdata/plan14-root-reads-home-part2.tsv +5 −2

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	315.562s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.808s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.254s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.711s
ok  	github.com/carlosframework/platform/internal/activator	251.467s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.005s
ok  	github.com/carlosframework/platform/internal/analytics	0.021s
ok  	github.com/carlosframework/platform/internal/appconfig	2.499s
ok  	github.com/carlosframework/platform/internal/apperr	0.095s
ok  	github.com/carlosframework/platform/internal/applog	0.147s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.007s
ok  	github.com/carlosframework/platform/internal/boxid	0.479s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.009s
ok  	github.com/carlosframework/platform/internal/caps	0.014s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.708s
ok  	github.com/carlosframework/platform/internal/changelog	0.318s
ok  	github.com/carlosframework/platform/internal/clamd	0.067s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	392.970s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.017s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.005s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.004s
ok  	github.com/carlosframework/platform/internal/docsite	0.004s
ok  	github.com/carlosframework/platform/internal/domains	2.404s
ok  	github.com/carlosframework/platform/internal/doorkey	0.008s
ok  	github.com/carlosframework/platform/internal/economics	0.006s
ok  	github.com/carlosframework/platform/internal/edge	26.656s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.147s
ok  	github.com/carlosframework/platform/internal/gate	1.666s
ok  	github.com/carlosframework/platform/internal/genselect	0.020s
ok  	github.com/carlosframework/platform/internal/geo	0.030s
ok  	github.com/carlosframework/platform/internal/geoip	0.033s
ok  	github.com/carlosframework/platform/internal/host	138.730s
ok  	github.com/carlosframework/platform/internal/hostactors	0.297s
ok  	github.com/carlosframework/platform/internal/hostbox	2.238s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.515s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.008s
ok  	github.com/carlosframework/platform/internal/hostexec	4.874s
ok  	github.com/carlosframework/platform/internal/hostfence	0.033s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.521s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.014s
ok  	github.com/carlosframework/platform/internal/hostguard	1.017s
ok  	github.com/carlosframework/platform/internal/hostops	1.603s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.002s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.002s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.489s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.151s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.003s
ok  	github.com/carlosframework/platform/internal/infraview	0.002s
ok  	github.com/carlosframework/platform/internal/instancectl	4.169s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.716s
ok  	github.com/carlosframework/platform/internal/instancefs	0.020s
ok  	github.com/carlosframework/platform/internal/instancekey	0.002s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.023s
ok  	github.com/carlosframework/platform/internal/layout	1.546s
ok  	github.com/carlosframework/platform/internal/lease	9.832s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.024s
ok  	github.com/carlosframework/platform/internal/ledger	0.017s
ok  	github.com/carlosframework/platform/internal/mail	0.186s
ok  	github.com/carlosframework/platform/internal/maintenance	0.082s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.531s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.091s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.147s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.747s
ok  	github.com/carlosframework/platform/internal/objstore	0.184s
ok  	github.com/carlosframework/platform/internal/ops	0.181s
ok  	github.com/carlosframework/platform/internal/park	0.209s
ok  	github.com/carlosframework/platform/internal/privatefile	0.005s
ok  	github.com/carlosframework/platform/internal/provision	15.324s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.009s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.960s
ok  	github.com/carlosframework/platform/internal/relay	7.232s
ok  	github.com/carlosframework/platform/internal/release	20.097s
ok  	github.com/carlosframework/platform/internal/rollout	3.655s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.010s
ok  	github.com/carlosframework/platform/internal/rootstate	0.025s
ok  	github.com/carlosframework/platform/internal/runs	0.574s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.483s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.435s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.938s
ok  	github.com/carlosframework/platform/internal/status	0.042s
ok  	github.com/carlosframework/platform/internal/steering	0.007s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.296s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.549s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 46399650 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.024s
=== RUN   TestT4StorageProbeMinio
2026/10/15 19:31:41 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DECAD11DFBB43B, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
minio contract: PASS
diff --git a/home_region_args_test.go b/home_region_args_test.go
index bf2ab90d1..9a5e6ab6e 100644
--- a/home_region_args_test.go
+++ b/home_region_args_test.go
@@ -55,7 +55,6 @@ const (
wDirty = "TestMoveIntoHomeDirtyReturn"
wFenced = "TestMoveIntoHomeFencedTargetRelease"
wLost = "TestMoveIntoHomeDeclareLost"
- task4 = "Task 4"
lossNote = "retained, validated intent: claimLossIntent copies TargetRegion from op.intent.to (@home, written by the start CAS) and the loss record signs it; SourceRegion is the record's From through park.PlacementRegion"
)
@@ -64,8 +63,8 @@ const (
var homeRegionCallSites = []homeRegionSite{
{File: "internal/console/apiinstancemove.go", Func: "Server.handleAPIInstanceMoveRequest", Callee: "hostclaim.BeginMove(targetRegion)", Arg: "to",
Witness: "named-only: the door refuses a home target before the plain path's BeginMove (MV-N21 today; MV-N510 under host claims, Task 5)"},
- {File: "internal/host/movesabort.go", Func: "movesWorker.abortClaim", Callee: "hostclaim.Abort(targetRegion)", Arg: "park.PlacementRegion(rec.Move.To)", Witness: wAbort, Pending: task4},
- {File: "internal/host/movescut.go", Func: "movesWorker.revertClaim", Callee: "hostclaim.Revert(targetRegion)", Arg: "park.PlacementRegion(rec.Move.To)", Witness: wRevert, Pending: task4},
+ {File: "internal/host/movesabort.go", Func: "movesWorker.abortClaim", Callee: "hostclaim.Abort(targetRegion)", Arg: "park.PlacementRegion(rec.Move.To)", Witness: wAbort},
+ {File: "internal/host/movescut.go", Func: "movesWorker.revertClaim", Callee: "hostclaim.Revert(targetRegion)", Arg: "park.PlacementRegion(rec.Move.To)", Witness: wRevert},
{File: "internal/host/movesinstall.go", Func: "installRun.parking", Callee: "hostclaim.QueueResidueParks(region)", Arg: "cfg.parkRegion()", Witness: wFresh},
{File: "internal/host/movesland.go", Func: "landingRun.checking", Callee: "hostclaim.Touch(regions)", Arg: "nil", Witness: wFresh,
Note: "the landing touches its own box only; the source round touched the region"},
@@ -74,21 +73,21 @@ var homeRegionCallSites = []homeRegionSite{
{File: "internal/host/movespass.go", Func: "stagingLeaseKey", Callee: "moves.StagingLeasePrefix(targetRegion)", Arg: "r", Witness: wFresh,
Note: "r is park.PlacementRegion(targetRegion), the lane's record code"},
{File: "internal/host/movesphase1.go", Func: "movesWorker.sourceRound", Callee: "hostclaim.Touch(regions)", Arg: "[]string{park.PlacementRegion(rec.Move.To)}", Witness: wFresh},
- {File: "internal/host/movesrecover.go", Func: "recoverRun.probed", Callee: "hostclaim.QueueReturnLanePark(sourceRegion)", Arg: "cfg.parkRegion()", Witness: wDirty, Pending: task4,
+ {File: "internal/host/movesrecover.go", Func: "recoverRun.probed", Callee: "hostclaim.QueueReturnLanePark(sourceRegion)", Arg: "cfg.parkRegion()", Witness: wDirty,
Note: "runs on the named source; its region is the source's"},
- {File: "internal/host/movesrollback.go", Func: "rollbackRun.claimReleased", Callee: "hostclaim.TargetReleased(targetRegion)", Arg: "cfg.parkRegion()", Witness: wClean, Pending: task4},
- {File: "internal/host/movesrollback.go", Func: "rollbackRun.remote", Callee: "hostclaim.QueueStagingPark(region)", Arg: "cfg.parkRegion()", Witness: wClean, Pending: task4},
+ {File: "internal/host/movesrollback.go", Func: "rollbackRun.claimReleased", Callee: "hostclaim.TargetReleased(targetRegion)", Arg: "cfg.parkRegion()", Witness: wClean},
+ {File: "internal/host/movesrollback.go", Func: "rollbackRun.remote", Callee: "hostclaim.QueueStagingPark(region)", Arg: "cfg.parkRegion()", Witness: wClean},
{File: "internal/hostops/move_enrollment.go", Func: "BeginClaimMove", Callee: "hostclaim.BeginAdmittedMove(targetRegion)", Arg: "park.PlacementRegion(to)", Witness: wBegin},
- {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.FencedTargetRelease{SourceRegion}", Arg: "park.PlacementRegion(m.From)", Witness: wFenced, Pending: task4},
- {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.FencedTargetRelease{TargetRegion}", Arg: "park.PlacementRegion(m.To)", Witness: wFenced, Pending: task4},
- {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.ReleaseFencedTarget(r)", Arg: "r", Witness: wFenced, Pending: task4},
- {File: "internal/hostops/move_loss_grant.go", Func: "lossScopedStore.claim", Callee: "hostclaim.ReleaseLostTarget(r)", Arg: "rel", Witness: wLost, Pending: task4,
+ {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.FencedTargetRelease{SourceRegion}", Arg: "park.PlacementRegion(m.From)", Witness: wFenced},
+ {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.FencedTargetRelease{TargetRegion}", Arg: "park.PlacementRegion(m.To)", Witness: wFenced},
+ {File: "internal/hostops/move_fence_release.go", Func: "MoveRecoveryGrant.releaseFencedTarget", Callee: "hostclaim.ReleaseFencedTarget(r)", Arg: "r", Witness: wFenced},
+ {File: "internal/hostops/move_loss_grant.go", Func: "lossScopedStore.claim", Callee: "hostclaim.ReleaseLostTarget(r)", Arg: "rel", Witness: wLost,
Note: "rel is moveLossRelease over the intent read back from the signed loss record (lossIntentOf), not a new derivation"},
- {File: "internal/hostops/move_loss_owner.go", Func: "claimLossIntent", Callee: "moveLossIntent{TargetRegion}", Arg: `op.Intent["to"]`, Witness: wLost, Pending: task4, Note: lossNote},
- {File: "internal/hostops/move_loss_owner.go", Func: "moveLossOwner.FinishLoss", Callee: "hostclaim.ReleaseLostTarget(r)", Arg: "moveLossRelease(in, durable, c.Holder, loss.Host)", Witness: wLost, Pending: task4},
- {File: "internal/hostops/move_loss_owner.go", Func: "moveLossRelease", Callee: "hostclaim.LostTargetRelease{SourceRegion}", Arg: "in.SourceRegion", Witness: wLost, Pending: task4, Note: lossNote},
- {File: "internal/hostops/move_loss_owner.go", Func: "moveLossRelease", Callee: "hostclaim.LostTargetRelease{TargetRegion}", Arg: "in.TargetRegion", Witness: wLost, Pending: task4, Note: lossNote},
- {File: "internal/hostops/move_loss_owner.go", Func: "recordLossIntent", Callee: "in.SourceRegion =", Arg: "park.PlacementRegion(m.From)", Witness: wLost, Pending: task4, Note: lossNote},
+ {File: "internal/hostops/move_loss_owner.go", Func: "claimLossIntent", Callee: "moveLossIntent{TargetRegion}", Arg: `op.Intent["to"]`, Witness: wLost, Note: lossNote},
+ {File: "internal/hostops/move_loss_owner.go", Func: "moveLossOwner.FinishLoss", Callee: "hostclaim.ReleaseLostTarget(r)", Arg: "moveLossRelease(in, durable, c.Holder, loss.Host)", Witness: wLost},
+ {File: "internal/hostops/move_loss_owner.go", Func: "moveLossRelease", Callee: "hostclaim.LostTargetRelease{SourceRegion}", Arg: "in.SourceRegion", Witness: wLost, Note: lossNote},
+ {File: "internal/hostops/move_loss_owner.go", Func: "moveLossRelease", Callee: "hostclaim.LostTargetRelease{TargetRegion}", Arg: "in.TargetRegion", Witness: wLost, Note: lossNote},
+ {File: "internal/hostops/move_loss_owner.go", Func: "recordLossIntent", Callee: "in.SourceRegion =", Arg: "park.PlacementRegion(m.From)", Witness: wLost, Note: lossNote},
{File: "internal/hostops/move_recovery_grant.go", Func: "MoveRecoveryGrant.Claim", Callee: "hostclaim.Commit(sourceRegion)", Arg: "park.PlacementRegion(m.From)", Witness: wConsoleCommt},
{File: "internal/hostops/move_recovery_grant.go", Func: "MoveRecoveryGrant.Claim", Callee: "hostclaim.Commit(targetRegion)", Arg: "park.PlacementRegion(m.To)", Witness: wConsoleCommt},
}
diff --git a/internal/host/moveshome_target_test.go b/internal/host/moveshome_target_test.go
index 1500c0f7b..ee06c87f8 100644
--- a/internal/host/moveshome_target_test.go
+++ b/internal/host/moveshome_target_test.go
@@ -14,17 +14,21 @@ import (
"time"
"github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/appconfig"
"github.com/carlosframework/platform/internal/genselect"
"github.com/carlosframework/platform/internal/hostactors"
"github.com/carlosframework/platform/internal/hostbox"
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/hostguard"
"github.com/carlosframework/platform/internal/hostops"
+ "github.com/carlosframework/platform/internal/hostwrite"
"github.com/carlosframework/platform/internal/instancekey"
"github.com/carlosframework/platform/internal/lease"
"github.com/carlosframework/platform/internal/moves"
"github.com/carlosframework/platform/internal/park"
"github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
"github.com/superfly/ltx"
)
@@ -589,8 +593,16 @@ func TestMoveIntoHomeCommitsAndCloses(t *testing.T) {
// @home too), on data-home, with the real moves park view seen from ie. The
// claim's authorizer and the regional owner check stand in as allowing.
func homeRegional(h *finalHarness) hostbox.Regional {
- return hostbox.Regional{Engine: park.Engine{HomeRegion: "ie", Store: h.vs, Claims: h.s, Bucket: "data-home", Region: "ie", Leases: claimParkLeases(h.leases),
- Now: h.clock.Now, NewID: obligationID, View: movesParkView(MovesRecordsFrom(h.s), h.vs, "data-home", "ie"), Authorize: allowParkAuthorize},
+ return regionalRunner(h.phase1Harness, h.leases, "ie", "ie", "data-home")
+}
+
+// regionalRunner is a box's regional runner as the claim assembly builds
+// it: hostbox.Regional over a park engine in the box's roster region with
+// the box's verified home binding (home, "" for a named box), on its data
+// bucket, with the real moves park view seen from its region.
+func regionalRunner(h *phase1Harness, leases park.Leases, region, home, bucket string) hostbox.Regional {
+ return hostbox.Regional{Engine: park.Engine{HomeRegion: home, Store: h.vs, Claims: h.s, Bucket: bucket, Region: region, Leases: claimParkLeases(leases),
+ Now: h.clock.Now, NewID: obligationID, View: movesParkView(MovesRecordsFrom(h.s), h.vs, bucket, region), Authorize: allowParkAuthorize},
Owner: func(context.Context, hostclaim.Observed) error { return nil }}
}
@@ -966,21 +978,31 @@ func homeTokenSeen(writes []hostclaim.Claim) (chores int, touched bool) {
}
// TestHomeTargetFreshStateShowsOnlyTheHomeToken (Review Focus 1, supplied
-// regions are @home; the runtime proof): each into-home witness of Task 3
-// runs once from clean claim state, where a wrong spelling cannot hide. A
+// regions are @home; the runtime proof): each box-side into-home witness of
+// Tasks 3 and 4 runs once from clean claim state, where a wrong spelling
+// cannot hide (the hostops witnesses, TestMoveIntoHomeDeclareLost and
+// TestMoveIntoHomeFencedTargetRelease, assert the same from their own clean
+// state with onlyHomeToken). A
// chore queued with ie would be a new obligation named ie; a Touch with ie
// would add ie to touched; a staging lease keyed with ie would show in the
-// recorded keys; an exact comparison with intent.to (Commit's target and
-// the release paths) would refuse and stop the witness. After each, every
+// recorded keys; an exact comparison with intent.to (Abort, Revert,
+// TargetReleased and the fenced and lost releases) would refuse and stop
+// the witness. hostclaim.Commit compares nothing with intent.to: given ie
+// it queues the lane park under ie, and only this check's chore assertion
+// catches it, as it does QueueStagingPark's. After each, every
// target-side chore the run ever created is @home and owned by
// region:@home, touched gained only @home and home-box, quiet_after never
-// named ie or "", and every home lane lease is keyed @home. Task 4 adds its
-// witnesses to this table.
+// named ie or "", and every home lane lease is keyed @home (a dirty
+// return's lane is the source's, under ca's lease).
func TestHomeTargetFreshStateShowsOnlyTheHomeToken(t *testing.T) {
type run struct {
writes []hostclaim.Claim
leases []string
chores int // the least number of @home chores the witness creates
+ // noHomeLease: the witness's recorder starts after the cut, so it
+ // sees no lease of the home lane; returnLane: the witness writes a
+ // dirty return's lane into the source's region, under ca's lease.
+ noHomeLease, returnLane bool
}
for name, witness := range map[string]func(t *testing.T) run{
"the source round stages the home lane": func(t *testing.T) run {
@@ -1017,6 +1039,24 @@ func TestHomeTargetFreshStateShowsOnlyTheHomeToken(t *testing.T) {
intoHomeCommitsAndCloses(t, r, 2)
return run{writes: r.watch.since(0), leases: r.leases.all(), chores: 1}
},
+ // Task 4.
+ "the abort": func(t *testing.T) run {
+ writes, leases := intoHomeAbort(t)
+ return run{writes: writes, leases: leases, chores: 1}
+ },
+ "the pre-flip revert": func(t *testing.T) run {
+ writes, leases := intoHomePreFlipRevert(t)
+ return run{writes: writes, leases: leases, chores: 1}
+ },
+ "the clean rollback": func(t *testing.T) run {
+ writes, leases := intoHomeCleanRollback(t)
+ // The home location's cleaning park and the lane park.
+ return run{writes: writes, leases: leases, chores: 2, noHomeLease: true}
+ },
+ "the dirty return": func(t *testing.T) run {
+ writes, leases := intoHomeDirtyReturn(t)
+ return run{writes: writes, leases: leases, chores: 1, noHomeLease: true, returnLane: true}
+ },
} {
t.Run(name, func(t *testing.T) {
got := witness(t)
@@ -1033,14 +1073,19 @@ func TestHomeTargetFreshStateShowsOnlyTheHomeToken(t *testing.T) {
if chores < got.chores || !touched {
t.Fatalf("the witness created %d chores in %s (want at least %d), touched held it %v", chores, park.HomeRegion, got.chores, touched)
}
- if len(got.leases) == 0 {
- t.Fatal("the witness took no staging lease")
- }
+ home := 0
for _, k := range got.leases {
- if k != homeLeaseKey {
+ switch {
+ case k == homeLeaseKey:
+ home++
+ case got.returnLane && k == "control/staging-leases/ca/"+p1Host:
+ default:
t.Fatalf("a staging lease keyed %q", k)
}
}
+ if home == 0 && !got.noHomeLease {
+ t.Fatal("the witness took no staging lease of the home lane")
+ }
})
}
}
@@ -1082,73 +1127,8 @@ func TestHomeTargetRetainedChoresKeepTheirSpelling(t *testing.T) {
for _, spelling := range []string{"ie", park.HomeRegion} {
t.Run(spelling, func(t *testing.T) {
ctx := context.Background()
- ids := map[string]hostclaim.Obligation{}
- var seeded, atQueue string
- r := intoHomeCloseWith(t, true, func(lh *landingHarness) {
- // Owed before the landing's parking step: one park-generation
- // per residue generation, with the landing's own scope.
- rec := lh.read(t, p1Host)
- rewriteP1Claim(t, lh.s, func(c *hostclaim.Claim) {
- scope := moves.ResidueScope(*c.Holder, *rec.Move.FlippedAt)
- for i, g := range []string{hostclaim.RootGeneration, "old"} {
- s := scope
- o := hostclaim.Obligation{ID: retainedID(2 * i), Op: p1ID, Kind: "park-generation", Region: spelling, Generation: g, Retirement: retainedID(2*i + 1), Owner: "region:" + spelling, Park: &s}
- ids[g] = o
- c.Obligations = append(c.Obligations, o)
- }
- })
- seeded = obligationsJSON(t, lh.claim(t).Obligations)
- lh.landFault = func(point string) error {
- if point == "after-residue-queued" && atQueue == "" {
- atQueue = obligationsJSON(t, lh.claim(t).Obligations)
- }
- return nil
- }
- })
- h, p := r.h, r.p
- if atQueue == "" {
- t.Fatal("the landing never queued its residue parks")
- }
- if atQueue != seeded {
- t.Fatalf("QueueResidueParks(%s) over chores owed under %s changed the obligations:\n%s\nwas\n%s", park.HomeRegion, spelling, atQueue, seeded)
- }
-
- // Owed before the commit: the lane's park with the commit's own
- // scope.
- var beforeCommit, afterCommit []hostclaim.Obligation
- h.landFault = func(point string) error {
- switch point {
- case "before-commit":
- if beforeCommit != nil {
- return nil
- }
- rec := h.read(t, p1Host)
- rewriteP1Claim(t, h.s, func(c *hostclaim.Claim) {
- _, staging, err := moves.CommitScopes(c, rec.Move.FlippedAt)
- if err != nil {
- t.Fatal(err)
- }
- o := hostclaim.Obligation{ID: retainedID(10), Op: p1ID, Kind: "park-replica", Region: spelling, Retirement: retainedID(11), Owner: "region:" + spelling, Park: &staging}
- ids["lane"] = o
- c.Obligations = append(c.Obligations, o)
- })
- beforeCommit = h.claim(t).Obligations
- case "after-claim-committed":
- afterCommit = h.claim(t).Obligations
- }
- return nil
- }
- h.land(t)
- h.committedAndOpen(t)
- if len(p.admissions) != 1 || afterCommit == nil {
- t.Fatalf("the commit: admissions %+v, claim captured %v", p.admissions, afterCommit != nil)
- }
- if got, want := obligationsJSON(t, otherThan(afterCommit, "ca")), obligationsJSON(t, beforeCommit); got != want {
- t.Fatalf("the commit over a lane park owed under %s changed the home chores:\n%s\nwas\n%s", spelling, got, want)
- }
- if src := afterCommit[len(afterCommit)-1]; len(afterCommit) != len(beforeCommit)+1 || src.Region != "ca" || src.Kind != "park-replica" || src.Park.Location != cutRoot {
- t.Fatalf("the commit queued %+v beside the retained chores", afterCommit[len(beforeCommit):])
- }
+ r, ids := retainedChoresRun(t, spelling)
+ h := r.h
// Each retained chore is acknowledged once, in its spelling: the
// lane by the home box's moves engine, the residue by its
@@ -1161,35 +1141,953 @@ func TestHomeTargetRetainedChoresKeepTheirSpelling(t *testing.T) {
t.Fatalf("the regional runner on the residue park of %s owed under %s: %v", g, spelling, err)
}
}
- c := h.claim(t)
+ ackedOnce(t, h, ids, spelling)
+ // A second run of an acknowledged chore writes nothing.
+ gen := h.claim(t).Gen
+ if err := reg.Run(ctx, park.Work{Host: p1Host, ID: ids["old"].ID, Op: p1ID}); err != nil || h.claim(t).Gen != gen {
+ t.Fatalf("a second acknowledgement: %v (claim gen %d to %d)", err, gen, h.claim(t).Gen)
+ }
+ h.sourcePass(t)
+ h.done(t)
+ })
+ }
+}
+
+// ackedOnce asserts each chore in ids is no longer owed and its retirement
+// holds exactly one park manifest in data-home, and that no home chore is
+// still owed.
+func ackedOnce(t *testing.T, h *finalHarness, ids map[string]hostclaim.Obligation, spelling string) {
+ t.Helper()
+ c := h.claim(t)
+ for name, o := range ids {
+ for _, owed := range c.Obligations {
+ if owed.ID == o.ID {
+ t.Fatalf("the %s chore owed under %s was not acknowledged: %+v", name, spelling, owed)
+ }
+ }
+ n := 0
+ for k := range h.f.buckets["data-home"] {
+ if strings.HasPrefix(k, "parked/"+h.inc+"/"+p1Host+"/"+o.Retirement+"/") && strings.HasSuffix(k, "/manifest.json") {
+ n++
+ }
+ }
+ if n != 1 {
+ t.Fatalf("the %s chore's retirement %s holds %d manifests, want 1", name, o.Retirement, n)
+ }
+ }
+ for _, o := range c.Obligations {
+ if o.Region == park.HomeRegion || o.Region == "ie" {
+ t.Fatalf("a home chore is still owed: %+v", o)
+ }
+ }
+}
+
+// retainedChoresRun is an into-home run whose residue chores and lane
+// chore are already owed under spelling when the landing and the commit
+// queue theirs; it asserts both queues leave them as they are, and returns
+// at landing/committed with the seeded chores by name (@root, old, lane).
+func retainedChoresRun(t *testing.T, spelling string) (*intoHomeRun, map[string]hostclaim.Obligation) {
+ t.Helper()
+ ids := map[string]hostclaim.Obligation{}
+ var seeded, atQueue string
+ r := intoHomeCloseWith(t, true, func(lh *landingHarness) {
+ // Owed before the landing's parking step: one park-generation
+ // per residue generation, with the landing's own scope.
+ rec := lh.read(t, p1Host)
+ rewriteP1Claim(t, lh.s, func(c *hostclaim.Claim) {
+ scope := moves.ResidueScope(*c.Holder, *rec.Move.FlippedAt)
+ for i, g := range []string{hostclaim.RootGeneration, "old"} {
+ s := scope
+ o := hostclaim.Obligation{ID: retainedID(2 * i), Op: p1ID, Kind: "park-generation", Region: spelling, Generation: g, Retirement: retainedID(2*i + 1), Owner: "region:" + spelling, Park: &s}
+ ids[g] = o
+ c.Obligations = append(c.Obligations, o)
+ }
+ })
+ seeded = obligationsJSON(t, lh.claim(t).Obligations)
+ lh.landFault = func(point string) error {
+ if point == "after-residue-queued" && atQueue == "" {
+ atQueue = obligationsJSON(t, lh.claim(t).Obligations)
+ }
+ return nil
+ }
+ })
+ h, p := r.h, r.p
+ if atQueue == "" {
+ t.Fatal("the landing never queued its residue parks")
+ }
+ if atQueue != seeded {
+ t.Fatalf("QueueResidueParks(%s) over chores owed under %s changed the obligations:\n%s\nwas\n%s", park.HomeRegion, spelling, atQueue, seeded)
+ }
+
+ // Owed before the commit: the lane's park with the commit's own
+ // scope.
+ var beforeCommit, afterCommit []hostclaim.Obligation
+ h.landFault = func(point string) error {
+ switch point {
+ case "before-commit":
+ if beforeCommit != nil {
+ return nil
+ }
+ rec := h.read(t, p1Host)
+ rewriteP1Claim(t, h.s, func(c *hostclaim.Claim) {
+ _, staging, err := moves.CommitScopes(c, rec.Move.FlippedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o := hostclaim.Obligation{ID: retainedID(10), Op: p1ID, Kind: "park-replica", Region: spelling, Retirement: retainedID(11), Owner: "region:" + spelling, Park: &staging}
+ ids["lane"] = o
+ c.Obligations = append(c.Obligations, o)
+ })
+ beforeCommit = h.claim(t).Obligations
+ case "after-claim-committed":
+ afterCommit = h.claim(t).Obligations
+ }
+ return nil
+ }
+ h.land(t)
+ h.committedAndOpen(t)
+ if len(p.admissions) != 1 || afterCommit == nil {
+ t.Fatalf("the commit: admissions %+v, claim captured %v", p.admissions, afterCommit != nil)
+ }
+ if got, want := obligationsJSON(t, otherThan(afterCommit, "ca")), obligationsJSON(t, beforeCommit); got != want {
+ t.Fatalf("the commit over a lane park owed under %s changed the home chores:\n%s\nwas\n%s", spelling, got, want)
+ }
+ if src := afterCommit[len(afterCommit)-1]; len(afterCommit) != len(beforeCommit)+1 || src.Region != "ca" || src.Kind != "park-replica" || src.Park.Location != cutRoot {
+ t.Fatalf("the commit queued %+v beside the retained chores", afterCommit[len(beforeCommit):])
+ }
+ return r, ids
+}
+
+// TestHomeRegionalAcknowledgesBothSpellings (Ask A2's witness): the real
+// hostbox.Regional on home-box (roster region ie, home ie) acknowledges a
+// lane park-replica and the residue park-generations owed under @home, and
+// the same owed under ie, each exactly once; ca-box's runner (region ca, no
+// home) acknowledges none of them.
+func TestHomeRegionalAcknowledgesBothSpellings(t *testing.T) {
+ for _, spelling := range []string{park.HomeRegion, "ie"} {
+ t.Run(spelling, func(t *testing.T) {
+ ctx := context.Background()
+ r, ids := retainedChoresRun(t, spelling)
+ h := r.h
+ h.clock.Advance(24 * time.Hour)
+ ca := regionalRunner(h.phase1Harness, h.leases, "ca", "", "data-ca")
+ home := homeRegional(h)
for name, o := range ids {
- for _, owed := range c.Obligations {
- if owed.ID == o.ID {
- t.Fatalf("the %s chore owed under %s was not acknowledged: %+v", name, spelling, owed)
+ gen := h.claim(t).Gen
+ if err := ca.Run(ctx, park.Work{Host: p1Host, ID: o.ID, Op: p1ID}); !errors.Is(err, hostbox.ErrUnsupported) || h.claim(t).Gen != gen {
+ t.Fatalf("ca's runner on the %s chore owed under %s: %v (claim gen %d to %d)", name, spelling, err, gen, h.claim(t).Gen)
+ }
+ }
+ for name, o := range ids {
+ if err := home.Run(ctx, park.Work{Host: p1Host, ID: o.ID, Op: p1ID}); err != nil {
+ t.Fatalf("home-box's runner on the %s chore owed under %s: %v", name, spelling, err)
+ }
+ gen := h.claim(t).Gen
+ if err := home.Run(ctx, park.Work{Host: p1Host, ID: o.ID, Op: p1ID}); err != nil || h.claim(t).Gen != gen {
+ t.Fatalf("a second acknowledgement of the %s chore: %v (claim gen %d to %d)", name, err, gen, h.claim(t).Gen)
+ }
+ }
+ ackedOnce(t, h, ids, spelling)
+ })
+ }
+}
+
+// Home part 2, Task 4: revert, rollback, abort, loss and conversion with a
+// home target. Each witness also runs from clean claim state in
+// TestHomeTargetFreshStateShowsOnlyTheHomeToken.
+
+// homeLaneCfg is home-box's configuration with its moves park engine armed
+// (region @home, ParkHomeRegion ie) over its own data directory and lease
+// table.
+func homeLaneCfg(h *phase1Harness, c moveCfg, dir string, leases *fakeParkLeases) moveCfg {
+ c.DataDir = dir
+ armParkFakes(&c, h.s, leases, park.PlacementRegion(c.Region), roleTarget)
+ return c
+}
+
+// homeLaneChore is the op's lane park owed in @home.
+func homeLaneChore(c *hostclaim.Claim) *hostclaim.Obligation {
+ for _, o := range c.Obligations {
+ if o.Op == p1ID && o.Kind == "park-replica" && o.Park != nil && o.Park.Location == moves.ScopeLocation(p1Host, p1ID) && o.Region != "ca" {
+ return &o
+ }
+ }
+ return nil
+}
+
+// TestMoveIntoHomeAbortParksTheHomeLane (Ask A2's witness): an abort before
+// the cut. Abort queues the lane park in @home, owned by region:@home;
+// home-box's moves engine archives the lane in data-home; past the horizon
+// a named box's regional runner (ca) refuses the chore and home-box's
+// hostbox.Regional (roster region ie, home ie) acknowledges it; the record
+// clears only after that.
+func TestMoveIntoHomeAbortParksTheHomeLane(t *testing.T) {
+ intoHomeAbort(t)
+}
+
+func intoHomeAbort(t *testing.T) ([]hostclaim.Claim, []string) {
+ t.Helper()
+ ctx := context.Background()
+ var watch *claimWatch
+ h := newPhase1Harness(t, intoHome, watchClaims(&watch))
+ leases := &stagingLeaseKeys{}
+ leases.wrap(&h.cfg)
+ h.start(t, "manual")
+ cleanHomeStart(t, h.claim(t))
+ h.pass(t)
+ staged := laneKeysIn(h, "data-home", p1ID)
+ if len(staged) == 0 {
+ t.Fatal("fixture: the round staged nothing in data-home")
+ }
+ g := provision.MoveGuard{ID: p1ID, Incarnation: h.inc}
+ if _, err := provision.SetRecordMove(ctx, h.s, "bab", "app", p1Host, provision.MoveGuardedAbort(g)); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t)
+ c := h.claim(t)
+ if c.Op.Kind != hostclaim.KindMoving || c.Op.Phase != hostclaim.PhaseAborting || c.Op.Exec != "ca-box" || c.Op.Intent["to"] != park.HomeRegion {
+ t.Fatalf("the claim after the abort: %+v", c.Op)
+ }
+ o := homeLaneChore(c)
+ if o == nil || o.Region != park.HomeRegion || o.Owner != "region:"+park.HomeRegion || o.Park.Incarnation != h.inc {
+ t.Fatalf("the abort's lane park: %+v (%+v)", o, c.Obligations)
+ }
+ waiting := func(why string) {
+ t.Helper()
+ if m := h.read(t, p1Host).Move; m == nil || m.State != provision.MoveAborting {
+ t.Fatalf("%s: the record cleared: %+v", why, m)
+ }
+ }
+ // home-box's moves engine archives the lane; the horizon has not passed.
+ tleases := &fakeParkLeases{}
+ tc := homeLaneCfg(h, h.targetCfg(), t.TempDir(), tleases)
+ h.targetPass(t, tc)
+ if left := laneKeysIn(h, "data-home", p1ID); len(left) != 0 {
+ t.Fatalf("the home lane still holds %v", left)
+ }
+ if homeLaneChore(h.claim(t)) == nil {
+ t.Fatal("the lane park was acknowledged before its horizon")
+ }
+ h.pass(t)
+ waiting("before the horizon")
+ // Past the horizon: a named box's runner refuses the chore, the home
+ // box's acknowledges it.
+ until := o.Park.EvidenceAt.Add(park.SignatureAge + o.Park.RequestLimit)
+ h.clock.Advance(until.Sub(h.clock.now) + time.Second)
+ work := park.Work{Host: p1Host, ID: o.ID, Op: p1ID}
+ if err := regionalRunner(h, tleases, "ca", "", "data-ca").Run(ctx, work); !errors.Is(err, hostbox.ErrUnsupported) {
+ t.Fatalf("ca's regional runner on the home lane park: %v", err)
+ }
+ if homeLaneChore(h.claim(t)) == nil {
+ t.Fatal("ca's runner acknowledged the home lane park")
+ }
+ h.pass(t)
+ waiting("ca's runner refused")
+ if err := regionalRunner(h, tleases, "ie", "ie", "data-home").Run(ctx, work); err != nil {
+ t.Fatalf("home-box's regional runner on the lane park: %v", err)
+ }
+ if homeLaneChore(h.claim(t)) != nil {
+ t.Fatal("the lane park was not acknowledged")
+ }
+ if q := h.claim(t).QuietAfter; q["ie"] != (time.Time{}) || q[""] != (time.Time{}) {
+ t.Fatalf("quiet_after %v", q)
+ }
+ // Only now does the record clear.
+ h.pass(t)
+ rec := h.read(t, p1Host)
+ if rec.Move != nil || rec.Region != "ca" || rec.LastMove == nil || rec.LastMove.Outcome != "aborted" {
+ t.Fatalf("the record after the abort: region %q move %+v last %+v", rec.Region, rec.Move, rec.LastMove)
+ }
+ if c := h.claim(t); c.Op.Kind != hostclaim.KindServing || c.Op.Exec != "ca-box" || len(c.Obligations) != 0 {
+ t.Fatalf("the claim after the abort: %+v %+v", c.Op, c.Obligations)
+ }
+ for k := range h.f.buckets["data-ca"] {
+ if strings.HasPrefix(k, "parked/") {
+ t.Fatalf("the abort parked in the source's bucket: %s", k)
+ }
+ }
+ return watch.since(0), leases.all()
+}
+
+// prodRecovery is the named source's recovery wake through the production
+// recovery owners (movesRecoveryOwners, dispatched by movesProbeOwners on
+// the op's kind) over the harness's claim, records and buckets: the
+// activator's recovery probe entry as far as the claim runs on main. Each
+// wake selects the probe's inputs with the owners' Select and admits through
+// the claim's writer with the owners' AdmissionEvidence bound, as the claim
+// runtime's Admit does. The source is ca-box in ca, a named box: no home
+// alias, the fleet's designation only. The owners compare the launch
+// environment by the full local digest (localEnvDigest), the recovery's
+// same-box rule.
+type prodRecovery struct {
+ h *phase1Harness
+ owners *movesProbeOwners
+ // observed, selected and admitted are each wake's claim before the
+ // admission, its inputs and the claim after it.
+ observed []hostclaim.Observed
+ selected []ClaimLaunchInputs
+ admitted []hostclaim.Observed
+ // before, when set, runs after Select and before the admission.
+ before func(o hostclaim.Observed, in ClaimLaunchInputs)
+ // work, when set, is the start's selected work Materialize runs under.
+ work hostwrite.Work
+ seq int
+}
+
+func newProdRecovery(t *testing.T, h *phase1Harness, dataDir string, route func() registry.Route) *prodRecovery {
+ t.Helper()
+ key, err := moves.ReportKeyFromSecrets(h.cfg.SecretsKey)
+ if err != nil {
+ t.Fatal(err)
+ }
+ r := &prodRecovery{h: h}
+ actor := h.roster[h.srcBox]
+ filesDir := func(host string) (string, string) {
+ return filepath.Join(dataDir, host), filepath.Join(dataDir, host+".env")
+ }
+ recorded := func() *provision.MoveInputs {
+ m := h.read(t, p1Host).Move
+ if m.Rollback != nil {
+ return m.Rollback.RecoveryInputs
+ }
+ return m.RecoveryInputs
+ }
+ owners := &movesProbeOwners{HomeRegion: actor.HomeRegion, FleetHomeRegion: actor.FleetHomeRegion, Claims: h.s, Records: MovesRecordsFrom(h.s),
+ ReadyPath: movesReadyPath(h.s), Data: h.vs, Bucket: h.srcBucket, Region: actor.Region, Box: actor.Box, DataDir: dataDir, FilesDir: filesDir,
+ Markers: p1Markers{h.vs},
+ Route: func(host string) (*registry.Route, error) { rt := route(); return &rt, nil },
+ // What this box would launch: its own composition over its data
+ // directory with its own region, the live release.
+ Launch: func(_ context.Context, rt registry.Route) (ClaimServingLaunch, moves.EnvReport, error) {
+ composed, supplied, _, err := (&activator.ExecBackend{DataDir: dataDir, Region: h.src}).LaunchLayers(rt)
+ if err != nil {
+ return ClaimServingLaunch{}, moves.EnvReport{}, err
+ }
+ delete(composed, "CARLOS_MOVE_PROBE")
+ env := make([]string, 0, len(composed))
+ for k, v := range composed {
+ env = append(env, k+"="+v)
+ }
+ sort.Strings(env)
+ return ClaimServingLaunch{Executable: "/opt/carlos/live/" + p1Host, ExecutableSHA256: strings.Repeat("a", 64), Env: env, Version: "v7"}, moves.ReportLaunch(key, composed, supplied), nil
+ },
+ EnvKey: key,
+ Admitted: MovesRecordAdmittedAttempts,
+ Works: func(context.Context, string, string, hostguard.Captured) (hostwrite.Work, error) {
+ return p1Work{order: new([]string), name: "held"}, nil
+ },
+ Render: func(_ context.Context, _ registry.Route, _ string, work func(appconfig.HeldRender) (hostwrite.Work, error)) (appconfig.HeldRender, error) {
+ in := recorded()
+ hr := appconfig.HeldRender{BundleETags: in.BundleETags, OverlayETag: in.OverlayETag}
+ if _, err := work(hr); err != nil {
+ return appconfig.HeldRender{}, err
+ }
+ return hr, nil
+ },
+ Release: func(ctx context.Context, _ registry.Route, version string, work hostwrite.Work) error {
+ return work.With(ctx, hostwrite.Identity{Release: version}, func() error { return nil })
+ },
+ NewID: func() string { r.seq++; return fmt.Sprintf("fresh%d", r.seq) },
+ Now: h.clock.Now}
+ owners.Files = &movesFiles{Files: p1FilesStore{vs: h.vs, bucket: h.srcBucket}, Data: h.vs, Bucket: h.srcBucket, HomeRegion: actor.FleetHomeRegion, Region: actor.Region, Box: actor.Box,
+ DataDir: dataDir, EnvFile: envFileOf(filesDir), Proof: ClaimOriginProof{Claims: h.s, HomeRegion: actor.FleetHomeRegion, Region: actor.Region}}
+ if !owners.valid() {
+ t.Fatal("fixture: the source's recovery owners are not valid")
+ }
+ r.owners = owners
+ return r
+}
+
+// materializeWith makes each wake materialize the admitted inputs too,
+// under work, after the restore stand-in (the activator's restore rewrites
+// <dbPath>.gen to the admitted generation): the owners' held render and
+// release, the launch environment's full local digest, the start gate, the
+// C8 establishment and the start check.
+func (r *prodRecovery) materializeWith(work hostwrite.Work) { r.work = work }
+
+// wake is one recovery probe wake: Select, then the writer's admission.
+func (r *prodRecovery) wake(ctx context.Context, rt registry.Route, p activator.StartPolicy) error {
+ h := r.h
+ o, err := hostclaim.Read(ctx, h.s, p1Host)
+ if err != nil {
+ return err
+ }
+ in, err := r.owners.Select(ctx, rt, o, p)
+ if err != nil {
+ return err
+ }
+ r.observed, r.selected = append(r.observed, o), append(r.selected, in)
+ if r.before != nil {
+ r.before(o, in)
+ }
+ w := hostbox.Writer{Claims: h.s, Data: h.vs, Bucket: h.srcBucket, Region: r.owners.Region, Box: r.owners.Box, AdmissionEvidence: r.owners.AdmissionEvidence}
+ admitted, err := w.Admit(ctx, o, in.Admission, in.Baseline)
+ if err != nil {
+ return err
+ }
+ r.admitted = append(r.admitted, admitted)
+ if r.work == nil {
+ return nil
+ }
+ gen := admitted.Claim.Holder.Generation
+ if cur, err := hostbox.ReadGenerationMarker(rt.DBPath + ".gen"); err != nil || cur != gen {
+ if err := hostbox.WriteGenerationMarker(rt.DBPath+".gen", gen); err != nil {
+ return err
+ }
+ }
+ return r.owners.Materialize(ctx, admitted, in, r.work)
+}
+
+// TestMoveIntoHomeCleanRollback: the home target's probe, through the
+// production probe owners, is admitted on home-box and fails before any
+// write, until its attempts are spent. The rollback releases clean:
+// TargetReleased names @home, which matches op.intent.to exactly, and the
+// cleaning park of the home location is queued in @home. The source then
+// reopens on ca-box through the production recovery owners, in two stages:
+// before the admission the clean ReturnToSource has put the holder back on
+// src_gen and the owners' baseline restores from src_gen; after it the
+// holder is on the fresh generation the admission named, the admission is
+// clean-return with no writes lost, in ca, and its predecessor is the failed
+// home target's last admission, in the spelling that admission recorded:
+// ie. The target's cleaning then queues the lane park in @home, home-box's
+// regional runner acknowledges both after the horizon, and the record ends
+// rolled_back in ca with nothing parked in data-ca.
+func TestMoveIntoHomeCleanRollback(t *testing.T) {
+ intoHomeCleanRollback(t)
+}
+
+func intoHomeCleanRollback(t *testing.T) ([]hostclaim.Claim, []string) {
+ t.Helper()
+ ctx := context.Background()
+ var watch *claimWatch
+ h := newRecoverHarness(t, false, intoHome, realLaunchEnv, watchClaims(&watch))
+ leases := &stagingLeaseKeys{}
+ leases.wrap(&h.cfg)
+ cleanHomeStart(t, firstMoving(t, watch.since(0)))
+ // One attempt: the probe's prepared admission keeps the flip's stamp,
+ // the stamp the target's installation is registered at.
+ if _, err := provision.SetRecordMove(ctx, h.s, "bab", "app", p1Host, func(rec *provision.Record) error {
+ rec.Move.CutAttempts = 1
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ // The probe through the production owners, admitted and failing before
+ // any write.
+ p := h.productionProbe(t)
+ p.afterMaterialize = func(int) error { return errors.New("test: the probe failed") }
+ for i := 0; i < 6 && h.read(t, p1Host).Move.State != provision.MoveRollingBack; i++ {
+ h.rollbackPass(t)
+ }
+ if len(p.admissions) == 0 {
+ t.Fatal("the probe was never admitted")
+ }
+ failed := p.admissions[len(p.admissions)-1]
+ if failed.Box != "home-box" || failed.Region != "ie" {
+ t.Fatalf("the home target's last admission: %+v", failed)
+ }
+ for i := 0; i < 4 && h.rb(t).TargetPhase != provision.RbTargetReleased; i++ {
+ h.rollbackPass(t)
+ }
+ rb := h.rb(t)
+ if rb.TargetPhase != provision.RbTargetReleased || rb.Dirty {
+ t.Fatalf("the target did not release clean: %+v", rb)
+ }
+ c := h.claim(t)
+ if c.Op.Kind != hostclaim.KindRestoring || c.Op.Phase != hostclaim.PhaseReleased || c.Op.Intent["to"] != park.HomeRegion || c.Op.Intent["rollback"] != rb.ID {
+ t.Fatalf("the claim after TargetReleased: %+v", c.Op)
+ }
+ location := false
+ for _, o := range c.Obligations {
+ if o.Op == p1ID && o.Kind == "park-replica" && o.Park != nil && o.Park.Location == cutRoot {
+ if o.Region != park.HomeRegion || o.Owner != "region:"+park.HomeRegion {
+ t.Fatalf("the cleaning park of the home location: %+v", o)
+ }
+ location = true
+ }
+ }
+ if !location {
+ t.Fatalf("TargetReleased queued no park of the home location: %+v", c.Obligations)
+ }
+
+ // The source's recovery through the production recovery owners. Its
+ // bucket is versioned, as production's is: the harness stored the
+ // source's replica directly, so each object gets its version here.
+ sourceMarker(t, h.phase1Harness)
+ srcGen := c.Op.Journal[hostclaim.JournalSrcGen]
+ rec := newProdRecovery(t, h.phase1Harness, h.sdir, func() registry.Route { return h.srcRoute })
+ rec.before = func(o hostclaim.Observed, in ClaimLaunchInputs) {
+ // Stage one: the clean ReturnToSource put the holder back on
+ // src_gen, and the owners restore from src_gen.
+ if o.Claim.Holder.Generation != srcGen || srcGen != "cur" || in.Baseline == nil || in.Baseline.Generation != srcGen || in.Prepared {
+ t.Fatalf("before the admission: holder at %q (src_gen %q), baseline %+v", o.Claim.Holder.Generation, srcGen, in.Baseline)
+ }
+ if a := in.Admission; a.Coverage != "clean-return" || a.Box != "ca-box" || a.Region != "ca" || a.Mode != "cold" || a.PredecessorRegion != failed.Region {
+ t.Fatalf("the selected admission: %+v", a)
+ }
+ }
+ h.recoveryWake = rec.wake
+ h.srcPass(t)
+ if len(rec.admitted) != 1 {
+ t.Fatalf("the recovery admitted %d times", len(rec.admitted))
+ }
+ // Stage two: the holder is on the fresh generation; the admission is
+ // clean-return, in ca, after the home target's last admission in ie.
+ ah := rec.admitted[0].Claim.Holder
+ last := ah.Admissions[len(ah.Admissions)-1]
+ if ah.Generation != rec.selected[0].Admission.NewGeneration || ah.Generation == srcGen || ah.WriterBox != "ca-box" {
+ t.Fatalf("the holder after the admission: %+v", ah)
+ }
+ if last.Coverage != "clean-return" || last.WritesLost != nil || len(last.AdditionalWritesLost) != 0 || last.Region != "ca" || last.Box != "ca-box" {
+ t.Fatalf("the clean return's admission: %+v", last)
+ }
+ if last.Predecessor != failed.Generation || last.PredecessorRegion != "ie" {
+ t.Fatalf("the clean return's predecessor %q in %q, want the home target's %q in ie", last.Predecessor, last.PredecessorRegion, failed.Generation)
+ }
+ h.opened(t)
+
+ // The target's cleaning: the lane park in @home, then both parks
+ // acknowledged by home-box's regional runner after their horizon.
+ h.rollbackPass(t)
+ if rb := h.rb(t); rb.TargetPhase != provision.RbTargetCleaned {
+ t.Fatalf("the target did not clean: %+v", rb)
+ }
+ owed := h.claim(t).Obligations
+ var lane *hostclaim.Obligation
+ for i, o := range owed {
+ if o.Op == p1ID && o.Park != nil && o.Park.Location == moves.ScopeLocation(p1Host, p1ID) {
+ lane = &owed[i]
+ }
+ if o.Op == p1ID && (o.Region != park.HomeRegion || o.Owner != "region:"+park.HomeRegion) {
+ t.Fatalf("a chore of the rollback outside @home: %+v", o)
+ }
+ }
+ if lane == nil {
+ t.Fatalf("no lane park: %+v", owed)
+ }
+ h.clock.Advance(24 * time.Hour)
+ home := homeRegionalOn(h.phase1Harness, h.leases)
+ for _, o := range owed {
+ if o.Op != p1ID {
+ continue
+ }
+ if err := home.Run(ctx, park.Work{Host: p1Host, ID: o.ID, Op: p1ID}); err != nil {
+ t.Fatalf("home-box's regional runner on %+v: %v", o, err)
+ }
+ }
+ h.srcPass(t)
+ final := h.read(t, p1Host)
+ if final.Move != nil || final.Region != "ca" || final.LastMove == nil || final.LastMove.Outcome != "rolled_back" {
+ t.Fatalf("the record after the rollback: region %q move %+v last %+v", final.Region, final.Move, final.LastMove)
+ }
+ if c := h.claim(t); c.Op.Kind != hostclaim.KindServing || c.Op.Exec != "ca-box" || len(c.Obligations) != 0 {
+ t.Fatalf("the claim after the rollback: %+v %+v", c.Op, c.Obligations)
+ }
+ // Nothing the source serves was parked: data-ca holds no archive, and
+ // src_gen's objects are where they were.
+ cur := 0
+ for k := range h.f.buckets["data-ca"] {
+ if strings.HasPrefix(k, "parked/") {
+ t.Fatalf("the rollback parked in the source's bucket: %s", k)
+ }
+ if strings.HasPrefix(k, cutLineage) {
+ cur++
+ }
+ }
+ if cur == 0 {
+ t.Fatal("src_gen's objects are gone from data-ca")
+ }
+ return watch.since(0), leases.all()
+}
+
+// TestMoveIntoHomeDirtyReturn: the home target's probe, through the
+// production probe owners, is admitted on home-box, writes, and fails. The
+// rollback releases dirty: home-box seals its installation and writes the
+// return lane into data-ca under the staging lease
+// control/staging-leases/ca/<host>. The source installs the return through
+// the shared installer, and the recovery owners admit it into the return
+// generation, sealed, establishing its files from the return's prepared
+// manifest through the real C8 provider. probed queues the return lane's
+// park in ca. The pre-move generations park after the return (D6): no
+// park-generation is owed before the target has returned, and the source's
+// install queues them in ca.
+func TestMoveIntoHomeDirtyReturn(t *testing.T) {
+ intoHomeDirtyReturn(t)
+}
+
+func intoHomeDirtyReturn(t *testing.T) ([]hostclaim.Claim, []string) {
+ t.Helper()
+ ctx := context.Background()
+ var watch *claimWatch
+ h := newRecoverHarness(t, false, intoHome, realLaunchEnv, watchClaims(&watch))
+ leases := &stagingLeaseKeys{}
+ leases.wrap(&h.cfg)
+ cleanHomeStart(t, firstMoving(t, watch.since(0)))
+ if _, err := provision.SetRecordMove(ctx, h.s, "bab", "app", p1Host, func(rec *provision.Record) error {
+ rec.Move.CutAttempts = 1
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ // The probe is admitted, the app writes, and the probe fails.
+ p := h.productionProbe(t)
+ p.afterMaterialize = func(int) error {
+ h.targetChanged(t)
+ return errors.New("test: the probe failed after a write")
+ }
+ residueBefore := func(why string) {
+ t.Helper()
+ for _, o := range h.claim(t).Obligations {
+ if o.Kind == "park-generation" {
+ t.Fatalf("%s: a pre-move generation's park is owed already: %+v", why, o)
+ }
+ }
+ }
+ for i := 0; i < 6 && h.read(t, p1Host).Move.State != provision.MoveRollingBack; i++ {
+ h.rollbackPass(t)
+ }
+ if len(p.admissions) != 1 || p.admissions[0].Box != "home-box" || p.admissions[0].Region != "ie" {
+ t.Fatalf("the home target's admission: %+v", p.admissions)
+ }
+ before := len(leases.all())
+ for i := 0; i < 6 && h.rb(t).TargetPhase != provision.RbTargetReturned; i++ {
+ h.rollbackPass(t)
+ residueBefore("the target's release")
+ }
+ rb := h.rb(t)
+ if !rb.Dirty || rb.TargetPhase != provision.RbTargetReturned {
+ t.Fatalf("the target did not return dirty: %+v", rb)
+ }
+ // The return lane in data-ca, under ca's staging lease.
+ lane := moves.ScopeLocation(p1Host, rb.ID) + "/"
+ inCA := 0
+ for k := range h.f.buckets["data-ca"] {
+ if strings.HasPrefix(k, lane) {
+ inCA++
+ }
+ }
+ for k := range h.f.buckets["data-home"] {
+ if strings.HasPrefix(k, lane) {
+ t.Fatalf("the return lane was written into data-home: %s", k)
+ }
+ }
+ if inCA == 0 {
+ t.Fatal("the return lane holds nothing in data-ca")
+ }
+ if got := leases.all()[before:]; len(got) == 0 || slicesAll(got, "control/staging-leases/ca/"+p1Host) != len(got) {
+ t.Fatalf("the return lane's staging leases: %v", got)
+ }
+
+ // The source's recovery through the production recovery owners, the
+ // real C8 provider establishing from the return's prepared manifest.
+ sourceMarker(t, h.phase1Harness)
+ rec := newProdRecovery(t, h.phase1Harness, h.sdir, func() registry.Route { return h.srcRoute })
+ rec.materializeWith(p1Work{order: new([]string), name: "spawn"})
+ // Under CARLOS_HOST_CLAIM (the probe mode sets it) the source's
+ // configured step publishes the live release through the held release;
+ // the stand-in answers as that publication would.
+ h.releaseLive = func(context.Context, release.Store, provision.Record, string) moves.Check {
+ return moves.Check{OK: true}
+ }
+ rec.before = func(o hostclaim.Observed, in ClaimLaunchInputs) {
+ residueQueued := false
+ for _, ob := range o.Claim.Obligations {
+ if ob.Kind == "park-generation" && ob.Region == "ca" && ob.Owner == "region:ca" {
+ residueQueued = true
+ }
+ }
+ if !residueQueued {
+ t.Fatalf("the source's install queued no park of the pre-move generations: %+v", o.Claim.Obligations)
+ }
+ if a := in.Admission; !in.Prepared || a.NewGeneration != rb.ID || a.Coverage != "sealed" || a.Box != "ca-box" || a.Region != "ca" || a.PredecessorRegion != "ie" {
+ t.Fatalf("the dirty return's selected admission: %+v prepared %v", a, in.Prepared)
+ }
+ }
+ h.recoveryWake = rec.wake
+ h.srcPass(t)
+ if len(rec.admitted) != 1 {
+ t.Fatalf("the recovery admitted %d times", len(rec.admitted))
+ }
+ got := h.opened(t)
+ if got.ID != rb.ID {
+ t.Fatalf("opened %+v", got)
+ }
+ // C8: the return generation's prepared manifest in data-ca is the one
+ // the establishment opened.
+ m, found, err := hostfiles.Load(ctx, p1FilesStore{vs: h.vs, bucket: "data-ca"}, moves.GenerationFilesPrefix(cutRoot, rb.ID))
+ if err != nil || !found {
+ t.Fatalf("the return generation's files manifest in data-ca: %v %v", found, err)
+ }
+ if _, ok := m.Entries["upload.txt"]; !ok || m.Platform[hostfiles.EnvItem].SHA256 == "" {
+ t.Fatalf("the return's files manifest: %+v", m)
+ }
+ // probed queued the return lane's park in ca.
+ returnLane := 0
+ for _, o := range h.claim(t).Obligations {
+ if o.Kind == "park-replica" && o.Park != nil && o.Park.Location == moves.ScopeLocation(p1Host, rb.ID) {
+ if o.Region != "ca" || o.Owner != "region:ca" {
+ t.Fatalf("the return lane's park: %+v", o)
+ }
+ returnLane++
+ }
+ }
+ if returnLane != 1 {
+ t.Fatalf("return lane parks: %d", returnLane)
+ }
+ return watch.since(0), leases.all()
+}
+
+// slicesAll counts the entries of xs equal to x.
+func slicesAll(xs []string, x string) int {
+ n := 0
+ for _, s := range xs {
+ if s == x {
+ n++
+ }
+ }
+ return n
+}
+
+// TestMoveIntoHomePreFlipRevert: the cut stops the source on ca-box and
+// fails before the flip (TestCutDeadlinePassedReverts' shape, stopped with
+// execution given up). Revert(id, ca-box, @home, &laneScope) matches
+// op.intent.to exactly and queues the lane park in @home, owned by
+// region:@home. The source's revert probe re-admits on ca-box through the
+// production recovery owners, uncovered (TestRevertProbeAdmitsUncovered's
+// shape), in ca, after the source's own generation in ca. home-box's moves
+// engine archives the lane in data-home; past the horizon home-box's
+// hostbox.Regional acknowledges it, and the revert closes. Nothing in data-ca
+// is parked.
+func TestMoveIntoHomePreFlipRevert(t *testing.T) {
+ intoHomePreFlipRevert(t)
+}
+
+func intoHomePreFlipRevert(t *testing.T) ([]hostclaim.Claim, []string) {
+ t.Helper()
+ ctx := context.Background()
+ var watch *claimWatch
+ h := newRevertHarness(t, intoHome, realLaunchEnv, watchClaims(&watch))
+ leases := &stagingLeaseKeys{}
+ leases.wrap(&h.cfg)
+ h.fromStopped(t)
+ cleanHomeStart(t, firstMoving(t, watch.since(0)))
+ if m := h.move(t); m.FlippedAt != nil || m.To != "" {
+ t.Fatalf("the fixture flipped: %+v", m)
+ }
+ c := h.claim(t)
+ if c.Op.Kind != hostclaim.KindReverting || c.Op.Exec != "ca-box" || c.Op.Intent["to"] != park.HomeRegion {
+ t.Fatalf("the revert's claim: %+v", c.Op)
+ }
+ lane := homeLaneChore(c)
+ if lane == nil || lane.Region != park.HomeRegion || lane.Owner != "region:"+park.HomeRegion {
+ t.Fatalf("Revert's lane park: %+v (%+v)", lane, c.Obligations)
+ }
+ staged := laneKeysIn(h.phase1Harness, "data-home", p1ID)
+ if len(staged) == 0 {
+ t.Fatal("fixture: the home lane is empty")
+ }
+
+ // The source's revert probe through the production recovery owners.
+ h.arm(t)
+ sourceMarker(t, h.phase1Harness)
+ rec := newProdRecovery(t, h.phase1Harness, h.cfg.DataDir, func() registry.Route { return h.route })
+ srcGen := c.Holder.Generation
+ rec.before = func(o hostclaim.Observed, in ClaimLaunchInputs) {
+ if a := in.Admission; a.Box != "ca-box" || a.Region != "ca" || a.PredecessorRegion != "ca" || a.Coverage != "" || a.Mode != "cold" || a.ObservedGeneration != srcGen ||
+ in.Baseline == nil || in.Baseline.Generation != srcGen {
+ t.Fatalf("the revert's selected admission: %+v baseline %+v", a, in.Baseline)
+ }
+ }
+ h.cfg.recoveryProbe = func(ctx context.Context, s release.Store, r provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ h.probes = append(h.probes, req)
+ w := h.w()
+ a := recoveryProbeAdapter{Box: w.cfg.Box, Claims: true, Route: w.cfg.Route, Current: w.recoveryInputs, Wake: rec.wake}
+ return a.probe(ctx, s, r, req)
+ }
+ h.pass(t)
+ h.reopened(t)
+ if len(rec.admitted) != 1 {
+ t.Fatalf("the revert probe admitted %d times", len(rec.admitted))
+ }
+ ah := rec.admitted[0].Claim.Holder
+ last := ah.Admissions[len(ah.Admissions)-1]
+ if ah.Generation == srcGen || ah.WriterBox != "ca-box" || last.Region != "ca" || last.PredecessorRegion != "ca" || last.Predecessor != srcGen || last.Coverage != "" {
+ t.Fatalf("the revert's admission: holder %+v last %+v", ah, last)
+ }
+ // Uncovered: the source's own writes are recorded as possibly lost when
+ // a writer session held them (the harness's serving holder never had
+ // one, so there is nothing to name).
+ if prior := rec.observed[0].Claim.Holder; prior.Session != 0 && (last.WritesLost == nil || last.WritesLost.Box != prior.WriterBox || last.WritesLost.Session != prior.Session) {
+ t.Fatalf("the revert's writes lost: %+v (prior holder %+v)", last.WritesLost, prior)
+ }
+
+ // home-box's moves engine archives the lane; the horizon has not passed.
+ h.tPass(t)
+ if left := laneKeysIn(h.phase1Harness, "data-home", p1ID); len(left) != 0 {
+ t.Fatalf("the home lane still holds %v", left)
+ }
+ if homeLaneChore(h.claim(t)) == nil {
+ t.Fatal("the lane park was acknowledged before its horizon")
+ }
+ h.pass(t)
+ h.reopened(t)
+ // Past the horizon home-box's regional runner acknowledges it, and the
+ // revert closes.
+ h.pastHorizon(t, lane)
+ if err := homeRegionalOn(h.phase1Harness, h.tleases).Run(ctx, park.Work{Host: p1Host, ID: lane.ID, Op: p1ID}); err != nil {
+ t.Fatalf("home-box's regional runner on the lane park: %v", err)
+ }
+ if homeLaneChore(h.claim(t)) != nil {
+ t.Fatal("the lane park was not acknowledged")
+ }
+ h.pass(t)
+ h.closed(t)
+ if rec := h.read(t, p1Host); rec.Region != "ca" {
+ t.Fatalf("the record after the revert: %+v", rec)
+ }
+ for k := range h.f.buckets["data-ca"] {
+ if strings.HasPrefix(k, "parked/") {
+ t.Fatalf("the revert parked in the source's bucket: %s", k)
+ }
+ }
+ return watch.since(0), leases.all()
+}
+
+// firstMoving is the first claim written with the move's op: the start
+// CAS's.
+func firstMoving(t *testing.T, writes []hostclaim.Claim) *hostclaim.Claim {
+ t.Helper()
+ for i := range writes {
+ if writes[i].Op.Kind == hostclaim.KindMoving {
+ return &writes[i]
+ }
+ }
+ t.Fatal("no claim was written with the move's op")
+ return nil
+}
+
+// sourceMarker makes the source's bucket what a source that served the
+// instance leaves: every object versioned (the harness stored the replica
+// directly), and the location's generation.json a marker naming src_gen,
+// cur, on ca-box (the harness's placeholder is not one).
+func sourceMarker(t *testing.T, h *phase1Harness) {
+ t.Helper()
+ body, err := json.Marshal(genselect.Marker{Op: "cur", Incarnation: h.inc, Host: p1Host, Box: h.srcBox, InstalledAt: p1Created})
+ if err != nil {
+ t.Fatal(err)
+ }
+ h.f.store(h.srcBucket, cutRoot+"/generation.json", fakeMoveObject{lease.MirrorObject{ETag: "marker", Size: int64(len(body))}, string(body)})
+ h.f.setVersion(h.srcBucket, cutRoot+"/generation.json", h.f.nextVersion())
+ for k := range h.f.buckets[h.srcBucket] {
+ if h.f.versionOf(h.srcBucket, k) == "" {
+ h.f.setVersion(h.srcBucket, k, h.f.nextVersion())
+ }
+ }
+}
+
+// homeRegionalOn is home-box's regional runner over a harness's store and
+// lease table.
+func homeRegionalOn(h *phase1Harness, leases park.Leases) hostbox.Regional {
+ return regionalRunner(h, leases, "ie", "ie", "data-home")
+}
+
+// TestCommittedLandingIntoHomeConvertsOnTargetFence (Ask H1's witness):
+// after the box's commit of a move into home (uk -> home, the home box dst
+// in roster region ie, the fleet's home), dst is fenced as terminated, and
+// region HA's committed-landing conversion turns the claim into a failover
+// relocation: intent.region is the roster name ie, move_to_region the
+// move's @home, the home binding kept. The commit's chores (the lane park,
+// owed under @home or, retained, under ie; the source's location park in
+// uk) come through byte-identical, each in the spelling it was owed in.
+func TestCommittedLandingIntoHomeConvertsOnTargetFence(t *testing.T) {
+ for _, lane := range []string{park.HomeRegion, "ie"} {
+ t.Run("lane owed under "+lane, func(t *testing.T) {
+ ctx := t.Context()
+ w := newLandingWorldHome(t, true, nil, "ie")
+ rec, err := provision.GetRecord(ctx, w.store, "bab", "jam", w.obs.Host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err := hostclaim.Read(ctx, w.store, w.obs.Host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ source, staging, err := moves.CommitScopes(o.Claim, rec.Move.FlippedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ id := o.Claim.Op.ID
+ if lane != park.HomeRegion {
+ // The lane park already owed in its retained spelling.
+ if o, err = hostclaim.Update(ctx, w.store, o, func(c *hostclaim.Claim) error {
+ s := staging
+ c.Obligations = append(c.Obligations, hostclaim.Obligation{ID: retainedID(20), Op: id, Kind: "park-replica", Region: lane, Retirement: retainedID(21), Owner: "region:" + lane, Park: &s})
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ // The box's commit, as commitClaim writes it: the record's codes
+ // through park.PlacementRegion, the target @home.
+ if o, err = hostclaim.Update(ctx, w.store, o, hostclaim.KeepWriterSession(id, hostclaim.Commit(id, park.PlacementRegion(rec.Move.From), "src", park.PlacementRegion(rec.Move.To), source, staging, newClaimID))); err != nil {
+ t.Fatal(err)
+ }
+ if o.Claim.Op.Phase != hostclaim.PhaseCommitted || o.Claim.Op.Intent["to"] != park.HomeRegion {
+ t.Fatalf("the commit: %+v", o.Claim.Op)
+ }
+ committed := o.Claim.Obligations
+ lanes := 0
+ for _, ob := range committed {
+ if ob.Park != nil && ob.Park.Location == moves.ScopeLocation(w.obs.Host, id) {
+ lanes++
+ if ob.Region != lane || ob.Owner != "region:"+lane {
+ t.Fatalf("the lane park after the commit: %+v", ob)
}
}
- n := 0
- for k := range h.f.buckets["data-home"] {
- if strings.HasPrefix(k, "parked/"+h.inc+"/"+p1Host+"/"+o.Retirement+"/") && strings.HasSuffix(k, "/manifest.json") {
- n++
+ }
+ if lanes != 1 {
+ t.Fatalf("the commit owes %d lane parks: %+v", lanes, committed)
+ }
+
+ if err := convertFencedLanding(w.homeWorker(w.enrollment["wrk"], "ie", "ie"), w.store)(ctx, w.store, o); err != nil {
+ t.Fatalf("convert: %v", err)
+ }
+ after, err := hostclaim.Read(ctx, w.store, w.obs.Host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ c := after.Claim
+ if c.Op.Kind != hostclaim.KindRelocating || c.Op.Journal["conversion"] != "committed-landing" || c.Op.Journal["conversion_landing_phase"] != hostclaim.PhaseCommitted {
+ t.Fatalf("not converted from committed: %+v", c.Op)
+ }
+ if c.Op.Intent["region"] != "ie" || c.Op.Intent["move_to_region"] != park.HomeRegion || c.Op.Intent["home_region"] != "ie" || c.Op.Journal["move_from_region"] != "uk" {
+ t.Fatalf("intent %v journal from %q", c.Op.Intent, c.Op.Journal["move_from_region"])
+ }
+ for _, before := range committed {
+ kept := false
+ for _, now := range c.Obligations {
+ if now.ID == before.ID {
+ if obligationsJSON(t, []hostclaim.Obligation{now}) != obligationsJSON(t, []hostclaim.Obligation{before}) {
+ t.Fatalf("the conversion rewrote %+v as %+v", before, now)
+ }
+ kept = true
}
}
- if n != 1 {
- t.Fatalf("the %s chore's retirement %s holds %d manifests, want 1", name, o.Retirement, n)
+ if !kept {
+ t.Fatalf("the conversion dropped %+v", before)
}
}
- for _, o := range c.Obligations {
- if o.Region == park.HomeRegion || o.Region == "ie" {
- t.Fatalf("a home chore is still owed: %+v", o)
+ for _, now := range c.Obligations {
+ if now.Region == "" {
+ t.Fatalf("a chore in the unset region: %+v", now)
+ }
+ // From clean state (the lane not retained), every chore is
+ // the source's (uk) or @home: nothing supplied the roster
+ // name.
+ if lane == park.HomeRegion && now.Region != "uk" && (now.Region != park.HomeRegion || now.Owner != "region:"+park.HomeRegion) {
+ t.Fatalf("a chore %+v after a commit from clean state", now)
}
}
- // A second run of an acknowledged chore writes nothing.
- gen := c.Gen
- if err := reg.Run(ctx, park.Work{Host: p1Host, ID: ids["old"].ID, Op: p1ID}); err != nil || h.claim(t).Gen != gen {
- t.Fatalf("a second acknowledgement: %v (claim gen %d to %d)", err, gen, h.claim(t).Gen)
- }
- h.sourcePass(t)
- h.done(t)
})
}
}
diff --git a/internal/host/movesrecover_test.go b/internal/host/movesrecover_test.go
index 847b0d27b..9c2669289 100644
--- a/internal/host/movesrecover_test.go
+++ b/internal/host/movesrecover_test.go
@@ -46,6 +46,10 @@ type recoverHarness struct {
// and answers.
probe func(req recoveryProbeRequest) (recoveryProbeOutcome, error)
probes []recoveryProbeRequest
+ // recoveryWake, when set, replaces the fake: the production adapter
+ // runs with it as the activator's recovery probe entry (home part 2,
+ // Task 4).
+ recoveryWake ClaimRecoveryProbeWake
// srcPage, when set, is the source's pager instead of pages.hook().
srcPage pageFunc
rcAt string
@@ -58,9 +62,9 @@ type recoverHarness struct {
// release: dirty makes the target's state differ from what it installed, so
// its release is dirty and its return a snapshot. The caller rolls back and
// runs the target's pass.
-func newRecoverHarness(t *testing.T, dirty bool) *recoverHarness {
+func newRecoverHarness(t *testing.T, dirty bool, opts ...p1Option) *recoverHarness {
t.Helper()
- h := &recoverHarness{rollbackHarness: newRollbackHarness(t)}
+ h := &recoverHarness{rollbackHarness: newRollbackHarness(t, opts...)}
if dirty {
h.targetChanged(t)
}
@@ -102,7 +106,8 @@ func (h *recoverHarness) released(t *testing.T) *provision.MoveRollback {
func (h *recoverHarness) sourceCfg(t *testing.T) moveCfg {
t.Helper()
c := h.landingCfg(t)
- c.Region, c.Box, c.OwnBucket, c.OwnBucketRegion = "ie", "ie-box", "data-ie", "eu-west-1"
+ c.Region, c.Box, c.OwnBucket, c.OwnBucketRegion = h.src, h.srcBox, h.srcBucket, h.srcAWS
+ c.ParkHomeRegion = ""
c.DataDir = h.sdir
c.OptDir = h.optDir
c.OwnsRoute = func(host string) bool { return host == p1Host }
@@ -117,8 +122,8 @@ func (h *recoverHarness) sourceCfg(t *testing.T) moveCfg {
return filepath.Join(h.sdir, host), filepath.Join(h.sdir, host+".env")
}
c.Guard = hostguard.Guard{DataDir: h.sdir, LockDir: h.lockDir}
- c.Local = park.Local{DataDir: h.sdir, Box: "ie-box"}
- c.ParkView = recordView(h.s, "ie", roleSource)
+ c.Local = park.Local{DataDir: h.sdir, Box: h.srcBox}
+ c.ParkView = recordView(h.s, park.PlacementRegion(h.src), roleSource)
c.RenderHeld = func(_ context.Context, _ release.Store, rt registry.Route, op string, _ *provision.MoveInputs, before func(string) error) (appconfig.HeldRender, error) {
h.srcRends++
if err := before(h.srcRender.SHA256); err != nil {
@@ -164,8 +169,15 @@ func (h *recoverHarness) sourceCfg(t *testing.T) moveCfg {
if h.srcPage != nil {
c.Page = h.srcPage
}
- c.recoveryProbe = func(ctx context.Context, _ release.Store, rec provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ c.recoveryProbe = func(ctx context.Context, s release.Store, rec provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
h.probes = append(h.probes, req)
+ if h.recoveryWake != nil {
+ // The production adapter (armRecoveryProbe's) over this worker:
+ // the box's current inputs recorded, then the wake.
+ w := h.sw
+ a := recoveryProbeAdapter{Box: w.cfg.Box, Claims: true, Route: w.cfg.Route, Current: w.recoveryInputs, Wake: h.recoveryWake}
+ return a.probe(ctx, s, rec, req)
+ }
if h.probe != nil {
return h.probe(req)
}
@@ -232,7 +244,7 @@ func (h *recoverHarness) opened(t *testing.T) *provision.MoveRollback {
t.Fatalf("opened without its stamps: %+v", rb)
}
c := h.claim(t)
- if c.Op.Kind != hostclaim.KindRestoring || c.Op.Phase != hostclaim.PhaseOpened || c.Op.Exec != "ie-box" || c.Op.Intent["rollback"] != rb.ID {
+ if c.Op.Kind != hostclaim.KindRestoring || c.Op.Phase != hostclaim.PhaseOpened || c.Op.Exec != h.srcBox || c.Op.Intent["rollback"] != rb.ID {
t.Fatalf("claim: %+v", c.Op)
}
if h.srcRoute.Status != "ready" || h.srcRoute.HoldOp != "" {
diff --git a/internal/host/movesrevert.go b/internal/host/movesrevert.go
index 471c6e8fa..f91f25cfb 100644
--- a/internal/host/movesrevert.go
+++ b/internal/host/movesrevert.go
@@ -594,6 +594,13 @@ func (r *revertRun) probing(ctx context.Context, obs hostclaim.Observed) (bool,
out = recoveryProbeWaiting
}
if out == recoveryProbeAnswered {
+ // The probe's wake wrote the claim (its admission and attempt
+ // reservation), so the step's own read is stale: read it again, as
+ // the rollback's probed does. SetRevPhase holds the op, the kind and
+ // rev_phase probing, so nothing else is taken on trust.
+ if obs, err = hostclaim.Read(ctx, r.s, host); err != nil {
+ return false, fmt.Errorf("re-read the host claim: %w", err)
+ }
if err := r.update(ctx, obs, "the revert's probed", hostclaim.SetRevPhase(id, hostclaim.PhaseProbing, hostclaim.PhaseProbed, r.rec.Move.Source != nil)); err != nil {
return false, err
}
diff --git a/internal/host/movesrevert_test.go b/internal/host/movesrevert_test.go
index dc2c99ddf..9f839a2ca 100644
--- a/internal/host/movesrevert_test.go
+++ b/internal/host/movesrevert_test.go
@@ -646,6 +646,37 @@ func countString(xs []string, s string) int {
return n
}
+// Home part 2, Task 4 (found by TestMoveIntoHomePreFlipRevert): a real
+// recovery probe wake admits a writer, so the claim the revert's step read
+// before the probe is stale when the probe answers. The probed step reads
+// the claim again; with the step's own read it lost the CAS every pass, and
+// each pass's probe admitted one more fresh generation until the attempts
+// were spent. Here, on the default named move, the probe runs through the
+// production adapter and recovery owners and admits once; the source then
+// reopens in the same pass.
+func TestRevertProbedAfterARealAdmission(t *testing.T) {
+ h := newRevertHarness(t, realLaunchEnv)
+ h.fromStopped(t)
+ h.arm(t)
+ sourceMarker(t, h.phase1Harness)
+ rec := newProdRecovery(t, h.phase1Harness, h.cfg.DataDir, func() registry.Route { return h.route })
+ h.cfg.recoveryProbe = func(ctx context.Context, s release.Store, r provision.Record, req recoveryProbeRequest) (recoveryProbeOutcome, error) {
+ w := h.w()
+ a := recoveryProbeAdapter{Box: w.cfg.Box, Claims: true, Route: w.cfg.Route, Current: w.recoveryInputs, Wake: rec.wake}
+ return a.probe(ctx, s, r, req)
+ }
+ h.pass(t)
+ h.reopened(t)
+ if len(rec.admitted) != 1 {
+ t.Fatalf("the revert probe admitted %d times", len(rec.admitted))
+ }
+ h.pass(t)
+ h.reopened(t)
+ if len(rec.admitted) != 1 {
+ t.Fatalf("a pass after the reopen probed again: %d admissions", len(rec.admitted))
+ }
+}
+
// TestRevertReplays: a crash after any of the revert's side effects, then a
// restarted agent, still reaches the close, with the evidence asked once it
// is journalled, the hold only ever restoring under the move id, and the
diff --git a/internal/host/movesrollback_test.go b/internal/host/movesrollback_test.go
index 6c3e9c380..7daf5b272 100644
--- a/internal/host/movesrollback_test.go
+++ b/internal/host/movesrollback_test.go
@@ -53,9 +53,9 @@ type rollbackHarness struct {
beforeHand func()
}
-func newRollbackHarness(t *testing.T) *rollbackHarness {
+func newRollbackHarness(t *testing.T, opts ...p1Option) *rollbackHarness {
t.Helper()
- h := &rollbackHarness{landingHarness: newLandingHarness(t)}
+ h := &rollbackHarness{landingHarness: newLandingHarness(t, opts...)}
h.flip(t)
h.land(t)
h.installed(t)
diff --git a/internal/hostops/move_home_target_test.go b/internal/hostops/move_home_target_test.go
new file mode 100644
index 000000000..c457a6e5f
--- /dev/null
+++ b/internal/hostops/move_home_target_test.go
@@ -0,0 +1,263 @@
+package hostops
+
+import (
+ "context"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/park"
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// Moves into the home region, part 2, Task 4: the loss and the fenced
+// release of a home target, from clean claim state. The fixtures are the
+// loss world and Plan 13's fence world, each turned into a move into home:
+// the record's To is the home's code "", the op's to is @home, written with
+// home_region set to the target box's roster name, and the console names
+// the same home. The home's roster name may appear only in what the claim
+// recorded about the box itself (its admissions, the loss record's write
+// sources); every region the moves code supplies is @home.
+
+// onlyHomeToken asserts a claim that started clean: every chore that is
+// not the named source's is in @home, owned by region:@home; nothing names
+// the home's roster name or the unset region in a chore, touched or
+// quiet_after; and the op's to is still @home.
+func onlyHomeToken(t *testing.T, c *hostclaim.Claim, source, rosterHome string) {
+ t.Helper()
+ for _, o := range c.Obligations {
+ switch {
+ case o.Kind == "park-local":
+ case o.Region == source:
+ if o.Owner != "region:"+source {
+ t.Fatalf("a source chore %+v", o)
+ }
+ case o.Region != park.HomeRegion || o.Owner != "region:"+park.HomeRegion:
+ t.Fatalf("a target chore not named %s: %+v", park.HomeRegion, o)
+ }
+ }
+ for _, r := range c.Touched.Regions {
+ if r == rosterHome || r == "" {
+ t.Fatalf("touched names %q: %+v", r, c.Touched)
+ }
+ }
+ for k := range c.QuietAfter {
+ if k == rosterHome || k == "" {
+ t.Fatalf("quiet_after names %q: %v", k, c.QuietAfter)
+ }
+ }
+ if to := c.Op.Intent["to"]; to != "" && to != park.HomeRegion {
+ t.Fatalf("the op's to is %q", to)
+ }
+}
+
+// TestMoveIntoHomeDeclareLost: a home target (uk-box, roster region uk,
+// the fleet's home) stopped mid-landing and declared lost through Plan 4
+// Task 10's door. The signed loss intent's TargetRegion is @home, copied
+// from op.intent.to, and its SourceRegion is ie; ReleaseLostTarget succeeds
+// with TargetRegion @home against intent.to @home and queues the target's
+// location and lane parks in @home; the record cites the loss. The source
+// is then re-admitted with the loss through AdmitWriterWithLoss and the
+// moves loss owner, its predecessor the lost target's last admission in
+// the spelling it recorded, uk.
+func TestMoveIntoHomeDeclareLost(t *testing.T) {
+ ctx := context.Background()
+ w := newLossWorld(t, "landing", func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) {
+ c.Op.Intent["to"], c.Op.Intent["home_region"] = park.HomeRegion, "uk"
+ r.Move.To, r.Region = "", ""
+ })
+ w.control.homeRegion = "uk"
+ before := w.claim(t)
+ if len(before.Claim.Obligations) != 0 || len(before.Claim.QuietAfter) != 0 {
+ t.Fatalf("fixture: the claim is not clean: %+v", before.Claim)
+ }
+ onlyHomeToken(t, before.Claim, "ie", "uk")
+
+ in := w.inspect(t, "uk-label")
+ if in.Details.Box != "uk-box" || in.Details.Op.Kind != hostclaim.KindLanding {
+ t.Fatalf("inspection = %+v", in.Details)
+ }
+ after, err := w.declare(w.submission(in))
+ if err != nil {
+ t.Fatal(err)
+ }
+ id := in.Details.ID
+ loss, err := hostclaim.ReadLossRecord(ctx, w.raw, w.pub, id)
+ if err != nil || loss.Record.State != "committed" {
+ t.Fatalf("loss record = %+v %v", loss.Record, err)
+ }
+ intent, _, err := lossIntentOf(loss.Record)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if intent.TargetRegion != park.HomeRegion || intent.SourceRegion != "ie" || intent.Target != "uk-box" || intent.Source != "ie-box" {
+ t.Fatalf("the signed loss intent: %+v", intent)
+ }
+ // The write sources are what the claim recorded about the box: its own
+ // region, uk.
+ for _, s := range loss.Record.WriteSources {
+ if s.Reason == "target" && s.Region != "uk" {
+ t.Fatalf("a target write source %+v", s)
+ }
+ }
+ c := w.claim(t)
+ if c.ETag != after.ETag || c.Claim.Op.Kind != hostclaim.KindRestoring || c.Claim.Op.Phase != hostclaim.PhaseReleased || c.Claim.Op.Exec != "none" ||
+ !hostclaim.LostTargetReleased(c.Claim, id) || c.Claim.Op.Intent["to"] != park.HomeRegion {
+ t.Fatalf("the claim after the declaration: %+v", c.Claim.Op)
+ }
+ parks := map[string]string{}
+ for _, o := range c.Claim.Obligations {
+ parks[o.Region+"|"+o.Park.Location] = o.Owner
+ }
+ if want := map[string]string{park.HomeRegion + "|instances/" + lossHost: "region:" + park.HomeRegion, park.HomeRegion + "|moves/" + lossHost + "/" + lossOp: "region:" + park.HomeRegion}; !reflect.DeepEqual(parks, want) {
+ t.Fatalf("parks = %v, want %v", parks, want)
+ }
+ onlyHomeToken(t, c.Claim, "ie", "uk")
+ lossRef, _ := hostclaim.LossRecordKey(id)
+ rec, _ := w.record(t)
+ if rb := rec.Move.Rollback; rec.Region != "ie" || rec.Move.To != "" || rb == nil || rb.LossRef != lossRef || rb.Reason != provision.RollbackDeclaredLost {
+ t.Fatalf("the record: region %q move %+v rollback %+v", rec.Region, rec.Move, rb)
+ }
+
+ // The source takes execution back on the declaration's authority and
+ // probes; the recovery's admission carries the loss.
+ g := provision.MoveGuard{ID: lossOp, Incarnation: w.inc}
+ for _, step := range [][2]string{{provision.RbSourceWaiting, provision.RbSourceClaimed}, {provision.RbSourceClaimed, provision.RbSourceRestoring},
+ {provision.RbSourceRestoring, provision.RbSourceVerified}, {provision.RbSourceVerified, provision.RbSourceProbing}} {
+ if _, err := provision.SetRecordMove(ctx, w.raw, "bab", "app", lossHost, provision.MoveRollbackSourcePhase(g, step[0], step[1], nil)); err != nil {
+ t.Fatalf("%s to %s: %v", step[0], step[1], err)
+ }
+ }
+ o := w.claim(t)
+ if o, err = hostclaim.Update(ctx, w.raw, o, hostclaim.ReturnToSource(lossOp, "ie-box", true, hostclaim.ReturnFence{})); err != nil {
+ t.Fatalf("the source's claimed: %v", err)
+ }
+ if o, err = hostclaim.Update(ctx, w.raw, o, hostclaim.SetMovePhase(lossOp, hostclaim.KindRestoring, hostclaim.PhaseClaimed, hostclaim.PhaseProbing)); err != nil {
+ t.Fatalf("the source's probing: %v", err)
+ }
+ h := o.Claim.Holder
+ last := h.Admissions[len(h.Admissions)-1]
+ if last.Box != "uk-box" || last.Region != "uk" {
+ t.Fatalf("fixture: the target's last admission %+v", last)
+ }
+ validated := make([]hostclaim.ValidatedWriteSource, 0, len(loss.Record.WriteSources))
+ var more []hostclaim.WritesLost
+ for _, s := range loss.Record.WriteSources {
+ v := hostclaim.ValidatedWriteSource{Reason: s.Reason, Box: s.Box, Generation: s.Generation, Session: s.Session, Region: s.Region, Baseline: s.Baseline}
+ if s.Reason == "source-cut" {
+ v.RemoteMaxTXID = 42
+ }
+ validated = append(validated, v)
+ if s.Box != last.Box || s.Generation != last.Generation || s.Session != last.Session {
+ more = append(more, hostclaim.WritesLost{Box: s.Box, Generation: s.Generation, Session: s.Session, RemoteMaxTXID: v.RemoteMaxTXID})
+ }
+ }
+ admission := hostclaim.Admission{ObservedOpID: lossOp, ObservedPhase: hostclaim.PhaseProbing, ObservedSession: h.Session, ObservedGeneration: h.Generation,
+ Box: "ie-box", Mode: "cold", NewGeneration: "fresh1", Region: "ie", PredecessorRegion: last.Region, AdditionalWritesLost: more}
+ if err := MoveLossRecoveryOwner().ValidateLossRecovery(*o.Claim, loss.Record, admission); err != nil {
+ t.Fatalf("the moves loss owner on the signed intent: %v (claim %+v holder stamp %d)", err, o.Claim.Op, h.Stamp)
+ }
+ admitted, err := hostclaim.AdmitWriterWithLoss(ctx, w.raw, w.pub, o, admission, id, validated, MoveLossRecoveryOwner())
+ if err != nil {
+ t.Fatalf("the source's admission with the loss: %v", err)
+ }
+ a := admitted.Claim.Holder.Admissions[len(admitted.Claim.Holder.Admissions)-1]
+ if a.LossID != id || a.Box != "ie-box" || a.Region != "ie" || a.Predecessor != last.Generation || a.PredecessorRegion != "uk" || a.WritesLost == nil || a.WritesLost.Box != "uk-box" {
+ t.Fatalf("the loss admission: %+v", a)
+ }
+ if !hostclaim.OperationLossConsumed(admitted.Claim, id) || admitted.Claim.Holder.WriterBox != "ie-box" {
+ t.Fatalf("the loss was not consumed: %+v", admitted.Claim.Holder)
+ }
+ onlyHomeToken(t, admitted.Claim, "ie", "uk")
+
+ // The moves loss owner holds the loss intent's @home to op.intent.to
+ // exactly: a loss record whose intent names the roster name is refused.
+ bad := loss
+ named := intent
+ named.TargetRegion = "uk"
+ raw, err := named.encode()
+ if err != nil {
+ t.Fatal(err)
+ }
+ bad.Record.ConfirmedConsequences.RecoveryIntent = raw
+ if err := MoveLossRecoveryOwner().ValidateLossRecovery(*o.Claim, bad.Record, admission); err == nil {
+ t.Fatal("a loss intent naming uk was accepted against intent.to @home")
+ }
+ if err := MoveLossRecoveryOwner().ValidateLossRecovery(*o.Claim, loss.Record, admission); err != nil {
+ t.Fatalf("the signed intent: %v", err)
+ }
+}
+
+// TestMoveIntoHomeFencedTargetRelease: a home target terminated after it
+// released, from clean claim state, on Plan 13's fence world. The recovery
+// grant's ReleaseFencedTarget builds FencedTargetRelease with TargetRegion
+// park.PlacementRegion(m.To), @home, and SourceRegion uk (the world's
+// named source); hostclaim.ReleaseFencedTarget accepts it against
+// intent.to @home; the location and lane parks it queues are in @home,
+// owned by region:@home; the record keeps the home's code.
+func TestMoveIntoHomeFencedTargetRelease(t *testing.T) {
+ w := newC7World(t, func(c *hostclaim.Claim, r *provision.Record, _ *AppAdmission) {
+ c.Op.Kind, c.Op.Phase, c.Op.Exec = hostclaim.KindLanding, hostclaim.PhaseProbing, "source"
+ c.Op.Intent["source"], c.Op.Intent["target"], c.Op.Intent["target_enrollment"] = "target", "source", strings.Repeat("a", 32)
+ c.Op.Intent["to"], c.Op.Intent["home_region"] = park.HomeRegion, "ie"
+ c.Op.Participants = []string{"target", "source"}
+ c.Obligations = nil
+ c.Op.Journal = map[string]string{hostclaim.JournalSourceStamp: "1", hostclaim.JournalSrcGen: "src"}
+ r.Region = "uk"
+ r.Move.From, r.Move.To, r.Move.State = "uk", "", provision.MoveRollingBack
+ at := r.CreatedAt
+ key := "control/box-fences/" + strings.Repeat("a", 32) + ".json"
+ r.Move.Rollback = &provision.MoveRollback{ID: "rb_abcdefghijklmnopqrstuvwxyz", Reason: provision.RollbackTargetFenced, Evidence: key, RequestedAt: &at, TargetPhase: provision.RbTargetLost, LossRef: key, LostAt: &at, SourcePhase: provision.RbSourceWaiting}
+ })
+ w.authority.home = "ie"
+ w.control.homeRegion = "ie"
+ before := w.claim(t).Claim
+ if len(before.Obligations) != 0 {
+ t.Fatalf("fixture: the claim owes %+v", before.Obligations)
+ }
+ onlyHomeToken(t, before, "uk", "ie")
+ run := func() error {
+ return w.control.RunRecovery(t.Context(), func(ctx context.Context) error {
+ return runMoveRecovery(ctx, w.claim(t), func(ctx context.Context, g *MoveRecoveryGrant) error {
+ _, err := g.ReleaseFencedTarget(ctx, w.ref)
+ return err
+ }, w.authority)
+ })
+ }
+ if err := run(); err != nil {
+ t.Fatal(err)
+ }
+ c := w.claim(t).Claim
+ if c.Op.Phase != hostclaim.PhaseReleased || c.Op.Intent["to"] != park.HomeRegion || c.Op.Journal["fence.source"] == "" {
+ t.Fatalf("the claim after the fenced release: %+v", c.Op)
+ }
+ parks := map[string]string{}
+ for _, o := range c.Obligations {
+ if o.Park == nil {
+ t.Fatalf("a chore without a scope: %+v", o)
+ }
+ parks[o.Region+"|"+o.Park.Location] = o.Owner
+ }
+ wantLocation, wantLane := false, false
+ for k, owner := range parks {
+ region, location, _ := strings.Cut(k, "|")
+ if region != park.HomeRegion || owner != "region:"+park.HomeRegion {
+ t.Fatalf("a park in %q owned by %q: %v", region, owner, parks)
+ }
+ wantLocation = wantLocation || location == strings.TrimSuffix(c.Holder.Location, "/")
+ wantLane = wantLane || strings.HasPrefix(location, "moves/"+w.host+"/")
+ }
+ if !wantLocation || !wantLane {
+ t.Fatalf("the fenced release's parks: %v", parks)
+ }
+ onlyHomeToken(t, c, "uk", "ie")
+ // The record's completion keeps the home's code.
+ if err := run(); err != nil {
+ t.Fatal("record completion", err)
+ }
+ rec, err := provision.GetRecord(t.Context(), w.raw, "bab", "app", w.host)
+ if err != nil || rec.Move.To != "" {
+ t.Fatalf("the record's target placement: %+v %v", rec.Move, err)
+ }
+}
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index f45ad18eb..f67841e41 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -745,12 +745,13 @@ func TestPlan14RootReadCensus(t *testing.T) {
// home rule (moves.CompareEnv's targetHome) and the move-portable env
// pin (moves.Pinned, ReportLaunch, LocalDigest; ExecBackend.LaunchLayers).
// Twelve changed or new units re-frozen at main's dispositions, ten of
- // them legacy, add ten frozen declarations. Ruling owed: the Plan 14
- // owner has not yet approved it.
+ // them legacy, add ten frozen declarations; Task 4's revert fix adds
+ // revertRun.probing, frozen at its sibling units' disposition, an
+ // eleventh. Ruling owed: the Plan 14 owner has not yet approved it.
if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-home-part2.tsv"), &inv, true); err != nil {
t.Fatal(err)
}
- seal = "4332d54efb67cd04114b6c52f11cdac5c28b554dc56860d3cb7c91167343efc5"
+ seal = "c0acbd4ad21d26fa77a7b456242bedc6c2d3e2c4a956f05360f9fc5ee874b9b2"
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go
index e2b6e39b1..4809da1ab 100644
--- a/internal/provision/plan9_b8_census_test.go
+++ b/internal/provision/plan9_b8_census_test.go
@@ -606,7 +606,8 @@ var plan9B8CensusInventory = []censusEntry{
// Marks recovery stuck through SetRecordMove; event and serial writes are not record objects.
{"internal/host/movesrecover.go", "recoverRun.stuck", "record-writer", "key:SetRecordMove", "d634bbd18794aedf081c51f6e3c16b62a9b6b82b21b24f636b949c840c8e5776"},
// Records revert inputs through SetRecordMove; stuck uses the reviewed record helper and journals are local. Plan 6 fix round 2: its journal write is a Commit through moves.UpdateJournal, and its two step removals are Unmarks.
- {"internal/host/movesrevert.go", "revertRun.probing", "record-writer", "key:SetRecordMove", "a3751bbc49cfc694236c446473a4bca6c55c2152736dd05f0cdbc1d684379a2d"},
+ // Re-reviewed for home part 2, Task 4 (ruling owed): after the probe answers it reads the host claim again for its probed CAS; its record writes are unchanged.
+ {"internal/host/movesrevert.go", "revertRun.probing", "record-writer", "key:SetRecordMove", "2eb21ddc4f49a08292f687f6f6e22b1c70facaf3a71a2d7a118583fbb278e5e3"},
// Applies revert record mutations only through SetRecordMove and its sequenced writer.
{"internal/host/movesrevert.go", "revertRun.record", "record-writer", "key:SetRecordMove", "f42100f118eb62ceece23b49423bda2ef357f1aa065760b288bf20340e36a920"},
// Starts rollback through SetRecordMove; the companion claim update targets the separate claim namespace.
diff --git a/internal/provision/testdata/plan14-root-reads-home-part2.tsv b/internal/provision/testdata/plan14-root-reads-home-part2.tsv
index 1e3d5358b..8c5914385 100644
--- a/internal/provision/testdata/plan14-root-reads-home-part2.tsv
+++ b/internal/provision/testdata/plan14-root-reads-home-part2.tsv
@@ -1,8 +1,9 @@
-# Plan 14 census overlay home-part2 (amadan #236, moves into home part 2, Tasks 1 and 2b with its fix rounds 2 to 4; loaded after hostfiles-cap-236). RULING OWED: the Plan 14 owner has not yet approved this overlay.
+# Plan 14 census overlay home-part2 (amadan #236, moves into home part 2, Tasks 1, 2b (with its fix rounds 2 to 5) and 4; loaded after hostfiles-cap-236). RULING OWED: the Plan 14 owner has not yet approved this overlay.
# Task 1: moves.CompareEnv takes targetHome, and T3 (internal/host checkEnv) passes rec.Move.To == "" for it: a home target must set no CARLOS_REGION and fails region-present when it does. Task 2b: internal/moves owns the move-portable env rule once: a name is left out of the pin only when it is box-owned (AgentOwnedVars, BoxVars) and the instance's own layers did not supply its running value (Pinned); ReportLaunch and PortableDigest make the v2 report and pin (EnvReportVersion, "env2:sha256:"), and LocalDigest the whole env for a check made on the box that recorded it. The composer reports the instance's layer names (activator ExecBackend.LaunchLayers, LaunchEnvironment's composition moved into its own unit), never a name the agent decides (agentOwnedNames). The moves launch source (claimMoveLaunchSource) makes the env report from that one composition and returns it with the launch; the probe owners take its digest as the launch's pin (movesReadLaunch passes it through) and keep it in memory for Materialize, bound to the exact env. No layer names cross into the claim: ClaimServingLaunch, ClaimLaunchInputs and ClaimLaunchDigest are main's. T3 refuses a v1 source report and the probe owners a v1 pin, fail closed. The recovery records recoveryInputs and checks localEnvDigest, the whole local env.
# Env digests, by where they are compared. Across boxes, the move-portable v2 pin: the source pin writeSourceReport sr.Env.Digest (host/movesphase1.go) copied to the target report's Inputs.EnvReport (host/movesreport.go) and checked on the target against the launch's own pin, the digest of the env report claimMoveLaunchSource makes from the same composition as the launch's env (host/claimmoveaccess.go, passed through by movesReadLaunch, host/moveswakesource.go): the probe owners' Select and the committed wake's selectCommitted compare it, and Materialize and materializeCommitted check the pin Select captured in memory for exactly that env (host/movesprobeowners.go captureEnvPin); the cut's currentInputs (host/movescut.go), compared on the source with that pin. Same box, the whole local env: the recovery's recoveryInputs (host/movescut.go, recorded through host/movesrecoveryprobe.go:139-146 on the recovering box) checked by the recovery owners' Select and Materialize (host/movesrecoveryowners.go localEnvDigest). Same box, main's full composed env, unchanged: ClaimLaunchDigest (host/claimoperations.go, byte for byte main's; it is root's LaunchSpec hash), compared only with the attempt's own box, boot and controller (host/movesprobeattempts.go, host/relocationattempts.go, host/claimlaunch_authority.go, host/claimwakestop.go, activator/servingretry.go, activator/rolelaunch_linux.go, activator/rolelaunch_root_linux.go); snapshot.Identity.Config (host/claimmoveaccess.go), which the moves owners never read (they set their own identity from the pins).
# Re-frozen units, every one at main's disposition (class, task and basis unchanged; kinds and digest refreshed); these rows replace this overlay's earlier, unruled rows. Authority rows, read set unchanged (only source offsets moved): host/claimmoveaccess.go claimMoveLaunchSource (authority C,G,J1; it calls LaunchLayers and reports the env from that composition) and host/movespass.go armPhase1 (authority B2,C; its LaunchEnv hook returns the layer names). Legacy rows: activator ExecBackend.LaunchEnvironment (a wrapper of LaunchLayers) and the new ExecBackend.LaunchLayers (LaunchEnvironment's composition and reads, moved; LaunchEnvironment's disposition); host movesWorker.writeSourceReport (ReportLaunch with the layer names); movesProbeOwners.Select, Materialize, selectCommitted and materializeCommitted (the captured report's pin, v2 only); movesRecoveryOwners.Select and Materialize (localEnvDigest); moveswakesource.go movesReadLaunch (passes the access's report through). No new read and no root decision. Ten frozen legacy declarations are added.
# Fix round 5: the probe owners' captured pins live as long as their wake (holdEnvPin from Materialize, under the start loop's context) and never past the wake's deadline (captureEnvPin, pruneEnvPins); movesProbeOwners.Select, Materialize and selectCommitted re-frozen again, read sets unchanged.
+# Task 4 (a defect its revert witness found): the revert's probing step reads the host claim again after its probe answers, because a real recovery probe wake writes the claim (its admission and attempt reservation) and the step's own read went stale, so its probed CAS lost every pass. revertRun.probing gains that one read (file:Read, hostclaim.Read of the control store) and is new to the census; it is frozen at the disposition main gives the same read in its sibling units (revertRun.step and recoverRun.claim: legacy, accepted-legacy;rev3=J4;basis=root-input). One frozen declaration is added; the seal moves.
@legacy 849ce111c61c3dcedd80595348d0a26c67fb47462d497fba63d52d2c94e1d5d0 internal/activator/backend_exec.go ExecBackend.LaunchEnvironment legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route whole:Route@139 e8853e15b427b80a701b5e10b565f7d4f0b778e0c8185da63266e2408f6b4113
internal/activator/backend_exec.go ExecBackend.LaunchEnvironment legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route whole:Route@139 e8853e15b427b80a701b5e10b565f7d4f0b778e0c8185da63266e2408f6b4113
@legacy 1680e7269c40013e72c2f0a39abc13ee75cecadf2b3b8aa3d3a0278c060458ce internal/activator/backend_exec.go ExecBackend.LaunchLayers legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Socket@751,whole:Route@179,whole:Route@747 66a14ac6d0a4c2c9333adf45a792407d8cc4fa389416feb31fcf58616fc5b611
@@ -25,6 +26,8 @@ internal/host/movesrecoveryowners.go movesRecoveryOwners.Materialize legacy acce
internal/host/movesrecoveryowners.go movesRecoveryOwners.Select legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:DBPath@1756,registry:DBPath@792,whole:Route@1011,whole:Route@514 746a4c5f4a87e6b5210ae12e986af05ea8bf3849d940a4fec6598000b555cd46
@legacy 1a77e1c538d55076eb81dbe54e0c110803dbafd56a79990246430473fca9a16d internal/host/moveswakesource.go movesReadLaunch legacy accepted-legacy accepted-legacy;rev3=merge-ruling-2;basis=Moves launch-input adapter selects by host through existing read interface; generic launch payload selection remains legacy. registry:Host@258 780a60ccf1035f2c7d64fdc804a867b64d380b05a51b151155ca6efe70b01767
internal/host/moveswakesource.go movesReadLaunch legacy accepted-legacy accepted-legacy;rev3=merge-ruling-2;basis=Moves launch-input adapter selects by host through existing read interface; generic launch payload selection remains legacy. registry:Host@258 780a60ccf1035f2c7d64fdc804a867b64d380b05a51b151155ca6efe70b01767
+@legacy d0a0ae52a0459fe62264eeb3b0cf906548b4afe6a8f23a4d1bae562b3ca286c0 internal/host/movesrevert.go revertRun.probing legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Read@2524 2eb21ddc4f49a08292f687f6f6e22b1c70facaf3a71a2d7a118583fbb278e5e3
+internal/host/movesrevert.go revertRun.probing legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Read@2524 2eb21ddc4f49a08292f687f6f6e22b1c70facaf3a71a2d7a118583fbb278e5e3
@coverage internal/activator 3e4fae49c0669c0524ec2cadcedf5d86de1bc10df1ebe4ad370d99b5712819f1
-@coverage internal/host c23c90cb6dedd6da24099f75d3af31f2e0e2f8ad685df4ad1cd5891aac219360
+@coverage internal/host fa686f59b817d104447d32ab419a98ebb44e30d8c91880bdb8475c9dcdba71f4
@coverage internal/moves 51ae19338bf9f92d45cc166f9fb71cdd3280c617892f71bb9ce67b2b00a7ae27