carlos / platform Public

Bind maintenance fences outside purge and preserve serving contracts

Paul Campbell pushed by paul@keymail.dev 1 month ago 0c59ea20c93a3d102fd34951845c718a1ecd0c2a parent 69b8aa9

1 file changed, +18 −2

implementation/maintenance-exec.md +18 −2

CI passed — run details

diff --git a/implementation/maintenance-exec.md b/implementation/maintenance-exec.md

index 5bd6721b6..727f32660 100644

--- a/implementation/maintenance-exec.md

+++ b/implementation/maintenance-exec.md

@@ -22,7 +22,7 @@ The public request record holds random request/attempt IDs, binding digests, bou

## Durable state and all consumers

-Use an additive optional maintenance object on provision.Record, mutated through narrow compare-and-swap helpers. It points to immutable request/attempt envelopes and records a monotonically increasing fence generation, primary host, owning box identity, expected record/config/artifact binding and current phase. Permanent operation receipts live in a separate account/app/host namespace; completing one does not erase its generation or history. No unbounded list is embedded in the instance record. Old records with no maintenance object retain byte-identical behavior.

+Use a durable guard at control/maintenance/<account>/<app>/<primary-host>/guard.json as the denial authority, updated by conditional writes. It survives ordinary apps/<account>/<app>/ deletion and pins the original record identity. Add an optional maintenance reference on provision.Record as a CAS-bound projection and conflict guard, not the sole barrier to a wake. The guard points to immutable request/attempt envelopes and records a monotonically increasing fence generation, primary host, owning box identity, expected record/config/artifact binding and phase. Permanent operation receipts live beside the guard, outside the app-purge prefix; completing one does not erase its generation or history. No unbounded list is embedded in the instance record. Old records with no maintenance history retain their existing behavior.

Phases are awaiting-approval (console-private intent, not yet a route fence), requested, quiescing, fenced, executing, succeeded, failed, ambiguous, resume-requested and resumed. requested through resume-requested all deny serving. A successful command stays fenced. Terminal failure never auto-clears. `executing` is durably committed before spawning; after restart an executing attempt becomes ambiguous and is NEVER automatically repeated. CAS binds phase+attempt+generation+owner; stale workers cannot overwrite new attempts or clear a later fence. Control serial nudges convergence, never supplies authority.

@@ -74,7 +74,23 @@ Expose a conservative monotonic safe-until deadline from lease acquisition/renew

The maintenance capability preflight also requires the platform-managed agent unit to explicitly enforce KillMode=control-group, SendSIGKILL=yes and a bounded TimeoutStopSec. The checked effective unit, not an assumed systemd default, is part of the deployed capability receipt. Parent-death protection is set on the direct command, but is not claimed to cover grandchildren. Agent main-process death must cause systemd to kill the entire delegated subtree before restart; test actual systemd behavior on the isolated Linux platform canary, including a descendant holding the output pipe. Without that deployed proof this class remains disabled. OOM/forced-kill and agent-restart tests must prove no surviving writer is mistaken for a clean stop. This is containment against process lifecycle failure under the existing trusted app/shared-uid model, not protection against a malicious instance intentionally migrating out of its delegated cgroup.

-**Resume must use the retained lease and a scoped internal permit.** The proposed seam is `Service.ResumeFenced` taking an already-held lease and a worker-created opaque permit bound to record/attempt/generation, approved serving artifact and environment snapshot. Ordinary callers cannot construct the permit. It starts replication and the exact pinned serving process while all public admission remains fenced; no Restore, Seed or live-link resolution is performed. It checks health and the app's actual reported version before the worker CASes the matching durable resume outcome. Failed CAS, version mismatch or process/replication failure invokes checked stop and retains the durable fence. The remaining design question is how the per-instance approved serving pin stays authoritative through concurrent channel promotion and the durable-clear/local-ready race; a bare release-and-next-wake or transient local pin is expressly rejected. This must be resolved in the final review, not deferred into a future rollout caveat.

+**Resume must use the retained lease and a scoped internal permit.** The proposed seam is `Service.ResumeFenced` taking an already-held lease and a worker-created opaque permit bound to record/attempt/generation, approved serving artifact and environment snapshot. Ordinary callers cannot construct the permit. It starts replication and the exact pinned serving process while all public admission remains fenced; no Restore, Seed or live-link resolution is performed. It checks health and the app's actual reported version before the worker CASes the matching durable resume outcome. Failed CAS, version mismatch or process/replication failure invokes checked stop and retains the durable fence. The proposed durable serving contract below closes the concurrent-promotion and durable-clear/local-ready race; a bare release-and-next-wake or transient local pin is rejected. Its full integration remains subject to the final review.

+## Durable guard and serving compatibility proposal

+The source audit found two reasons a record field alone is insufficient. provision.DeleteRecord currently performs an unconditional Store.Delete, and console/ownership.go purges the app prefix. A prior read followed by that delete can erase a newly written record fence. Therefore every maintenance-aware primary wake checks the outside-app guard, including after lease acquisition and before any Restore/Seed. A missing/malformed/mismatched record while a guard is active denies admission; a recreated record with the same hostname cannot claim the old database. Guard creation is create-only for the first operation and CAS on the monotonic generation thereafter. The worker may proceed only when the matching reference is also CAS-installed on the expected record revision. Failure between guard creation and record projection remains fenced/failed; it never clears the guard to repair availability automatically. Denial requires either barrier; execution/resume authority requires both matching barriers plus the approved request. This is a deliberate fail-closed multi-object protocol, not a claim to cross-object atomicity.

+Extend the store's conditional operation support with DeleteIfMatch, including directory/S3/prefix implementations. The pinned AWS S3 SDKv1.105.0 already exposes DeleteObjectInput.IfMatch. GetRecord with its etag, active-guard refusal and conditional delete close the old read/check/delete race for every instance delete caller. Unsupported store implementations refuse this operation; never fall back to unconditional delete. App purge and automated trash sweeps must check outside-app guards before ANY destructive step and must route instance-record deletion through this guarded primitive. Maintenance may start only on an active, untrashed app, and rechecks that lifecycle state after installing its durable guard and before executing. A concurrently removed/trashed/replaced app makes the operation fail closed; the external guard and receipts survive. Tests must interleave trash/purge/restore/guard creation at these boundaries; merely checking a list of guards once is not sufficient evidence that the race is closed.

+Writer inventory at dcf0cad: provision.CreateRecord, UpdateRecord, SetRecordRegion, SetRecordMove and DeleteRecord; console/apiinstances.go (create/shape/delete), channelmove.go, fleets.go, domainattach.go and apidomains.go (detach/delete); host/movespass.go; console/ownership.go and release trash/purge paths. Readers/admission projections: provision.Reconcile/ListAllRecords, host/steering_loop.go and agent.go, edge/edge.go and alias resolution, activator.begin/BeginSidecarRequest/ServeScheduled/sweeper, host restart/restartscope, config SyncExec/fingerprints/materialization, release/root adoption and system rollback. The implementation inventory must also search raw prefix deletes and JSON decoders, not only named helpers. Tests cover both named record calls and low-level consumer fixtures; new guard objects cannot be made reachable through edge/storedoor's app store namespace.

+To preserve normal continuous app deployments after maintenance, add a signed release-manifest declaration of supported runtime contracts and a bounded monotonic required-contract set in the outside-app guard. First resume specifies the required app contract (for example, a Home operator-schema contract or a Docs TeamMode authorization contract). The approved resume artifact must declare it; future normal adoption/wake may change artifact versions only if the verified signed manifest declares every required contract. Known older releases without the declaration refuse before starting. Ordinary record/config changes cannot remove a required contract; changing/removing a floor is a new explicit maintenance operation with its own reviewed compatibility checker and owner approval. Limit the set to16 canonical names; overflow refuses instead of dropping old requirements.

+This declaration is trusted release-author intent, not proof of code behavior. Real consumer compatibility and authorization tests plus independent review remain required for each claimed contract. It enforces rejection of known incompatible rollback artifacts while letting newly reviewed compatible releases follow the usual deployment path. It does not protect against a malicious trusted release author falsely declaring compatibility, which ordinary artifact signing also cannot do.

+The guard remains present after resume. Its resumed state binds the successful request/attempt, approved initial artifact/config, required contracts and last record identity. Service.ResumeFenced starts the exact approved initial artifact and replication under the existing lease with all admission closed. After its health/version checks, the worker CASes only that guard generation to resumed and updates the matching record projection. If either write fails, local admission remains closed and the started process is stopped through the checked path. If the guard write succeeded before the record projection and the agent crashes, a future wake requires both objects to agree; it does not infer completion. Recovery may finish that specific recorded resume only after rechecking exact pins/health under a lease, or remain fenced; it must never rerun the maintenance command.

+Only once both durable references agree may local admission open. Concurrent normal adoption continues to honor the active guard until then; after resumed it checks the required-contract set before a new release can replace the already-verified initial artifact. A process death after completed resume is an ordinary wake of a compatible signed release under the normal exclusive lease, not permission to return to an old incompatible artifact. A startup failure before completed resume stays fenced and has no successful-resume receipt. The exact CAS boundaries, crashes between them, channel promotion during startup, stale record objects and incompatible old manifests are required deterministic tests and deployed rehearsal cases.

## Implementation and release gates